[I 2022-10-08 09:45:01] ...ytorch/bootstrap/run.py.73: Saving environment info [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: options: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dataset: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: aux_kw_path: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: batch_sampler: triplet_classif [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: batch_size: 100 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: database: lmdb [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: eval_split: val [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: freq_mismatch: 0.0 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: image_from: database [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: import: recipe1m.datasets.factory [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: kw_path: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: recipe1m [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_classes: 1048 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_threads: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: randkw_p: 0.3 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: randkw_p_aux: 0.5 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: tokenized_raw_text: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: train_split: train [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: use_vcs: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: engine: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: logger [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_epochs: 120 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: print_freq: 10 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max'] [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: exp: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: checkpoint: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_recipe1m_13m_vlp/ckpt_last_model.pth.tar [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: resume: None [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: misc: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cuda: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: data_parrallel: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: logs_name: None [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: overrite: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: seed: 1338 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: model: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: criterion: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: tri_cross_encoder: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: itm_loss_weight: 1 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: keep_background: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: retrieval_strategy: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: aggregation: valid [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: margin: 0.05 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: margin_params: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: increment: 0.005 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: increment_margin: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: max_margin: 0.3 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: triplet [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_samples: 9999 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: sampling: max_negative [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: substrategy: ['IRR', 'RII', 'SIRR', 'SRII'] [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: substrategy_weights: [1.0, 1.0, 0.1, 0.1] [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: import: recipe1m.models.factory [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: metric: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: k_test: 10 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_bags: 10 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: nb_matchs_per_bag: 1000 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: trijoint: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: network: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: activations: ['tanh', 'normalize'] [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: aux_kwords: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: aux_kwords_encoder: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: avg_concat: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cat_pos: 0 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: class_attention_fus: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cls_token_fus: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_decoder_image: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_decoder_recipe: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_attention: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_decoder: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: double_decoder: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: get_tokens: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: load_backbones: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_layers: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: query_img: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: cross_transformer: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dim_emb: 1024 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dim_image_out: 2048 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dim_ingr_out: 300 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dim_instr_in: 1024 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: dim_instr_out: 1024 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: embedding_names: ['proj_image'] [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: freeze_im: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: freeze_rec: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: get_tokens_cross_decoder_recipe: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: hidden_size: 512 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: image_backbone_name: clip [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: image_res: 224 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: kwords_same_level: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: last_cat_aux: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_heads_cross: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_heads_cross_image: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_heads_single: 4 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_layers: 2 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_layers_cross: 2 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_layers_cross_image: 1 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: n_layers_single: 2 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: num_hidden_layers_kw: 2 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: old: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: recipe_encoder: tfoodvicha-transformer [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: text_encoder: bert-base-uncased [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: vision_width: 768 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: vit_all_tokens: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: with_ingrs: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: with_instrs: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: with_titles: True [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: with_classif: False [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: optimizer: [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: batch_size_factor: None [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: clip_grad: 8.0 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: import: recipe1m.optimizers.factory [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: lr: 1e-05 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: lr_img: 1e-06 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: name: trijoint_fixed_fine_tune [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: switch_epoch: 20 [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml [S 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.75: view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm'] [I 2022-10-08 09:45:02] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-a101', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64') [I 2022-10-08 09:45:03] ...ytorch/bootstrap/run.py.80: Available GPUs: [0, 1] [I 2022-10-08 09:45:03] ...strap/engines/factory.py.9: Creating engine... [I 2022-10-08 09:45:03] ...trap/datasets/factory.py.8: Creating dataset... [I 2022-10-08 09:45:06] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-08 09:45:10] ...m/datasets/recipe1m.py.165: randkw_p... 0.3 [I 2022-10-08 09:46:17] ...m/datasets/recipe1m.py.370: Calculate indices by class... [I 2022-10-08 09:46:19] ...m/datasets/recipe1m.py.375: Done! [I 2022-10-08 09:46:20] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-08 09:46:20] ...m/datasets/recipe1m.py.165: randkw_p... None [I 2022-10-08 09:47:32] ...rap/datasets/factory.py.18: Training will take place on trainset (238399 items) [I 2022-10-08 09:47:32] ...rap/datasets/factory.py.21: Evaluation will take place on valset (51119 items) [I 2022-10-08 09:47:32] ...strap/models/factory.py.21: Creating model... [I 2022-10-08 09:47:35] ...e_networks/networks.py.203: Loading CLIP Visual model... [I 2022-10-08 09:47:38] ...e_networks/networks.py.145: cuda [I 2022-10-08 09:47:39] ...networks/tfood_vicha.py.49: Load VCs encoder... [I 2022-10-08 09:47:41] .../criterions/trijoint.py.30: ignore_index=0 [I 2022-10-08 09:47:41] ...strap/models/factory.py.44: Enabling CUDA mode... [I 2022-10-08 09:47:43] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-08 09:47:43] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-08 09:47:43] ...otstrap/views/factory.py.8: Creating views... [I 2022-10-08 09:47:44] ...trap/engines/engine.py.407: Loading model... /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_recipe1m_13m_vlp/ckpt_last_model.pth.tar [I 2022-10-08 09:47:48] ...trap/engines/engine.py.125: Launching training procedures [I 2022-10-08 09:47:48] ...trap/engines/engine.py.163: Training model on trainset for epoch 0 [I 2022-10-08 09:47:48] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 09:47:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 0/2304 [I 2022-10-08 09:47:56] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 5:08:23 [I 2022-10-08 09:47:56] ...trap/engines/engine.py.229: process: 8.03104 | load: 3.10424 [I 2022-10-08 09:47:56] ...trap/engines/engine.py.230: loss: 2.68502 [I 2022-10-08 09:47:56] ...optimizers/trijoint.py.151: total_norm: 26.460121 [I 2022-10-08 09:48:05] ...trap/engines/engine.py.224: train: epoch 0 | batch 10/2304 [I 2022-10-08 09:48:05] ...trap/engines/engine.py.228: elapsed: 0:00:16 | left: 1:00:05 [I 2022-10-08 09:48:05] ...trap/engines/engine.py.229: process: 0.76243 | load: 0.00627 [I 2022-10-08 09:48:05] ...trap/engines/engine.py.230: loss: 1.32469 [I 2022-10-08 09:48:05] ...optimizers/trijoint.py.151: total_norm: 7.077985 [I 2022-10-08 09:48:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 20/2304 [I 2022-10-08 09:48:12] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:32:36 [I 2022-10-08 09:48:12] ...trap/engines/engine.py.229: process: 0.77054 | load: 0.00468 [I 2022-10-08 09:48:12] ...trap/engines/engine.py.230: loss: 1.27337 [I 2022-10-08 09:48:12] ...optimizers/trijoint.py.151: total_norm: 1.578444 [I 2022-10-08 09:48:20] ...trap/engines/engine.py.224: train: epoch 0 | batch 30/2304 [I 2022-10-08 09:48:20] ...trap/engines/engine.py.228: elapsed: 0:00:31 | left: 0:29:43 [I 2022-10-08 09:48:20] ...trap/engines/engine.py.229: process: 0.76202 | load: 0.00434 [I 2022-10-08 09:48:20] ...trap/engines/engine.py.230: loss: 1.20846 [I 2022-10-08 09:48:20] ...optimizers/trijoint.py.151: total_norm: 3.696662 [I 2022-10-08 09:48:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 40/2304 [I 2022-10-08 09:48:28] ...trap/engines/engine.py.228: elapsed: 0:00:39 | left: 0:29:03 [I 2022-10-08 09:48:28] ...trap/engines/engine.py.229: process: 0.76341 | load: 0.00347 [I 2022-10-08 09:48:28] ...trap/engines/engine.py.230: loss: 1.11490 [I 2022-10-08 09:48:28] ...optimizers/trijoint.py.151: total_norm: 3.483087 [I 2022-10-08 09:48:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 50/2304 [I 2022-10-08 09:48:35] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:28:49 [I 2022-10-08 09:48:35] ...trap/engines/engine.py.229: process: 0.76970 | load: 0.00345 [I 2022-10-08 09:48:35] ...trap/engines/engine.py.230: loss: 0.91966 [I 2022-10-08 09:48:35] ...optimizers/trijoint.py.151: total_norm: 2.847035 [I 2022-10-08 09:48:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 60/2304 [I 2022-10-08 09:48:43] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:28:56 [I 2022-10-08 09:48:43] ...trap/engines/engine.py.229: process: 0.78461 | load: 0.00305 [I 2022-10-08 09:48:43] ...trap/engines/engine.py.230: loss: 0.89513 [I 2022-10-08 09:48:43] ...optimizers/trijoint.py.151: total_norm: 5.393028 [I 2022-10-08 09:48:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 70/2304 [I 2022-10-08 09:48:51] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:28:46 [I 2022-10-08 09:48:51] ...trap/engines/engine.py.229: process: 0.77197 | load: 0.00339 [I 2022-10-08 09:48:51] ...trap/engines/engine.py.230: loss: 0.80835 [I 2022-10-08 09:48:51] ...optimizers/trijoint.py.151: total_norm: 4.851835 [I 2022-10-08 09:48:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 80/2304 [I 2022-10-08 09:48:59] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:28:24 [I 2022-10-08 09:48:59] ...trap/engines/engine.py.229: process: 0.76652 | load: 0.00326 [I 2022-10-08 09:48:59] ...trap/engines/engine.py.230: loss: 0.69455 [I 2022-10-08 09:48:59] ...optimizers/trijoint.py.151: total_norm: 6.142384 [I 2022-10-08 09:49:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 90/2304 [I 2022-10-08 09:49:06] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:28:17 [I 2022-10-08 09:49:06] ...trap/engines/engine.py.229: process: 0.76444 | load: 0.00350 [I 2022-10-08 09:49:06] ...trap/engines/engine.py.230: loss: 0.73139 [I 2022-10-08 09:49:06] ...optimizers/trijoint.py.151: total_norm: 4.323180 [I 2022-10-08 09:49:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 100/2304 [I 2022-10-08 09:49:14] ...trap/engines/engine.py.228: elapsed: 0:01:25 | left: 0:28:05 [I 2022-10-08 09:49:14] ...trap/engines/engine.py.229: process: 0.75333 | load: 0.00312 [I 2022-10-08 09:49:14] ...trap/engines/engine.py.230: loss: 0.59268 [I 2022-10-08 09:49:14] ...optimizers/trijoint.py.151: total_norm: 4.296107 [I 2022-10-08 09:49:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 110/2304 [I 2022-10-08 09:49:22] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:28:06 [I 2022-10-08 09:49:22] ...trap/engines/engine.py.229: process: 0.76838 | load: 0.00330 [I 2022-10-08 09:49:22] ...trap/engines/engine.py.230: loss: 0.65594 [I 2022-10-08 09:49:22] ...optimizers/trijoint.py.151: total_norm: 8.751902 [I 2022-10-08 09:49:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 120/2304 [I 2022-10-08 09:49:29] ...trap/engines/engine.py.228: elapsed: 0:01:41 | left: 0:28:05 [I 2022-10-08 09:49:29] ...trap/engines/engine.py.229: process: 0.76959 | load: 0.00314 [I 2022-10-08 09:49:29] ...trap/engines/engine.py.230: loss: 0.53476 [I 2022-10-08 09:49:29] ...optimizers/trijoint.py.151: total_norm: 4.462085 [I 2022-10-08 09:49:37] ...trap/engines/engine.py.224: train: epoch 0 | batch 130/2304 [I 2022-10-08 09:49:37] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:27:56 [I 2022-10-08 09:49:37] ...trap/engines/engine.py.229: process: 0.78109 | load: 0.00311 [I 2022-10-08 09:49:37] ...trap/engines/engine.py.230: loss: 0.54175 [I 2022-10-08 09:49:37] ...optimizers/trijoint.py.151: total_norm: 4.552520 [I 2022-10-08 09:49:45] ...trap/engines/engine.py.224: train: epoch 0 | batch 140/2304 [I 2022-10-08 09:49:45] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:27:51 [I 2022-10-08 09:49:45] ...trap/engines/engine.py.229: process: 0.77696 | load: 0.00308 [I 2022-10-08 09:49:45] ...trap/engines/engine.py.230: loss: 0.49542 [I 2022-10-08 09:49:45] ...optimizers/trijoint.py.151: total_norm: 3.984805 [I 2022-10-08 09:49:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 150/2304 [I 2022-10-08 09:49:53] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:27:46 [I 2022-10-08 09:49:53] ...trap/engines/engine.py.229: process: 0.77661 | load: 0.00315 [I 2022-10-08 09:49:53] ...trap/engines/engine.py.230: loss: 0.64409 [I 2022-10-08 09:49:53] ...optimizers/trijoint.py.151: total_norm: 5.815474 [I 2022-10-08 09:50:00] ...trap/engines/engine.py.224: train: epoch 0 | batch 160/2304 [I 2022-10-08 09:50:00] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:27:27 [I 2022-10-08 09:50:00] ...trap/engines/engine.py.229: process: 0.76309 | load: 0.00304 [I 2022-10-08 09:50:00] ...trap/engines/engine.py.230: loss: 0.54296 [I 2022-10-08 09:50:00] ...optimizers/trijoint.py.151: total_norm: 5.385842 [I 2022-10-08 09:50:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 170/2304 [I 2022-10-08 09:50:08] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:27:22 [I 2022-10-08 09:50:08] ...trap/engines/engine.py.229: process: 0.77145 | load: 0.00352 [I 2022-10-08 09:50:08] ...trap/engines/engine.py.230: loss: 0.53861 [I 2022-10-08 09:50:08] ...optimizers/trijoint.py.151: total_norm: 5.441092 [I 2022-10-08 09:50:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 180/2304 [I 2022-10-08 09:50:16] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:27:20 [I 2022-10-08 09:50:16] ...trap/engines/engine.py.229: process: 0.76811 | load: 0.00311 [I 2022-10-08 09:50:16] ...trap/engines/engine.py.230: loss: 0.62778 [I 2022-10-08 09:50:16] ...optimizers/trijoint.py.151: total_norm: 6.709436 [I 2022-10-08 09:50:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 190/2304 [I 2022-10-08 09:50:23] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:27:15 [I 2022-10-08 09:50:23] ...trap/engines/engine.py.229: process: 0.77382 | load: 0.00318 [I 2022-10-08 09:50:23] ...trap/engines/engine.py.230: loss: 0.52277 [I 2022-10-08 09:50:23] ...optimizers/trijoint.py.151: total_norm: 3.714696 [I 2022-10-08 09:50:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 200/2304 [I 2022-10-08 09:50:31] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:27:11 [I 2022-10-08 09:50:31] ...trap/engines/engine.py.229: process: 0.77529 | load: 0.00301 [I 2022-10-08 09:50:31] ...trap/engines/engine.py.230: loss: 0.52429 [I 2022-10-08 09:50:31] ...optimizers/trijoint.py.151: total_norm: 4.882494 [I 2022-10-08 09:50:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 210/2304 [I 2022-10-08 09:50:39] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:26:55 [I 2022-10-08 09:50:39] ...trap/engines/engine.py.229: process: 0.76132 | load: 0.00305 [I 2022-10-08 09:50:39] ...trap/engines/engine.py.230: loss: 0.54040 [I 2022-10-08 09:50:39] ...optimizers/trijoint.py.151: total_norm: 5.023267 [I 2022-10-08 09:50:47] ...trap/engines/engine.py.224: train: epoch 0 | batch 220/2304 [I 2022-10-08 09:50:47] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:26:47 [I 2022-10-08 09:50:47] ...trap/engines/engine.py.229: process: 0.77275 | load: 0.00302 [I 2022-10-08 09:50:47] ...trap/engines/engine.py.230: loss: 0.53995 [I 2022-10-08 09:50:47] ...optimizers/trijoint.py.151: total_norm: 6.898900 [I 2022-10-08 09:50:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 230/2304 [I 2022-10-08 09:50:54] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:26:36 [I 2022-10-08 09:50:54] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00302 [I 2022-10-08 09:50:54] ...trap/engines/engine.py.230: loss: 0.55494 [I 2022-10-08 09:50:54] ...optimizers/trijoint.py.151: total_norm: 7.219706 [I 2022-10-08 09:51:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 240/2304 [I 2022-10-08 09:51:02] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:26:31 [I 2022-10-08 09:51:02] ...trap/engines/engine.py.229: process: 0.76921 | load: 0.00295 [I 2022-10-08 09:51:02] ...trap/engines/engine.py.230: loss: 0.42971 [I 2022-10-08 09:51:02] ...optimizers/trijoint.py.151: total_norm: 3.582031 [I 2022-10-08 09:51:10] ...trap/engines/engine.py.224: train: epoch 0 | batch 250/2304 [I 2022-10-08 09:51:10] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:26:30 [I 2022-10-08 09:51:10] ...trap/engines/engine.py.229: process: 0.78341 | load: 0.00308 [I 2022-10-08 09:51:10] ...trap/engines/engine.py.230: loss: 0.57058 [I 2022-10-08 09:51:10] ...optimizers/trijoint.py.151: total_norm: 5.791222 [I 2022-10-08 09:51:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 260/2304 [I 2022-10-08 09:51:18] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:26:07 [I 2022-10-08 09:51:18] ...trap/engines/engine.py.229: process: 0.76220 | load: 0.00302 [I 2022-10-08 09:51:18] ...trap/engines/engine.py.230: loss: 0.46012 [I 2022-10-08 09:51:18] ...optimizers/trijoint.py.151: total_norm: 4.710969 [I 2022-10-08 09:51:25] ...trap/engines/engine.py.224: train: epoch 0 | batch 270/2304 [I 2022-10-08 09:51:25] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:26:13 [I 2022-10-08 09:51:25] ...trap/engines/engine.py.229: process: 0.77885 | load: 0.00294 [I 2022-10-08 09:51:25] ...trap/engines/engine.py.230: loss: 0.52310 [I 2022-10-08 09:51:25] ...optimizers/trijoint.py.151: total_norm: 4.932429 [I 2022-10-08 09:51:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 280/2304 [I 2022-10-08 09:51:33] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:26:00 [I 2022-10-08 09:51:33] ...trap/engines/engine.py.229: process: 0.76143 | load: 0.00302 [I 2022-10-08 09:51:33] ...trap/engines/engine.py.230: loss: 0.51748 [I 2022-10-08 09:51:33] ...optimizers/trijoint.py.151: total_norm: 5.429494 [I 2022-10-08 09:51:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 290/2304 [I 2022-10-08 09:51:41] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:25:48 [I 2022-10-08 09:51:41] ...trap/engines/engine.py.229: process: 0.75772 | load: 0.00292 [I 2022-10-08 09:51:41] ...trap/engines/engine.py.230: loss: 0.56953 [I 2022-10-08 09:51:41] ...optimizers/trijoint.py.151: total_norm: 4.691332 [I 2022-10-08 09:51:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 300/2304 [I 2022-10-08 09:51:48] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:25:49 [I 2022-10-08 09:51:48] ...trap/engines/engine.py.229: process: 0.77838 | load: 0.00294 [I 2022-10-08 09:51:48] ...trap/engines/engine.py.230: loss: 0.47201 [I 2022-10-08 09:51:48] ...optimizers/trijoint.py.151: total_norm: 4.939233 [I 2022-10-08 09:51:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 310/2304 [I 2022-10-08 09:51:56] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:25:36 [I 2022-10-08 09:51:56] ...trap/engines/engine.py.229: process: 0.76449 | load: 0.00293 [I 2022-10-08 09:51:56] ...trap/engines/engine.py.230: loss: 0.47476 [I 2022-10-08 09:51:56] ...optimizers/trijoint.py.151: total_norm: 4.916444 [I 2022-10-08 09:52:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 320/2304 [I 2022-10-08 09:52:04] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:25:18 [I 2022-10-08 09:52:04] ...trap/engines/engine.py.229: process: 0.76762 | load: 0.00296 [I 2022-10-08 09:52:04] ...trap/engines/engine.py.230: loss: 0.52932 [I 2022-10-08 09:52:04] ...optimizers/trijoint.py.151: total_norm: 4.811840 [I 2022-10-08 09:52:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 330/2304 [I 2022-10-08 09:52:12] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:25:19 [I 2022-10-08 09:52:12] ...trap/engines/engine.py.229: process: 0.78248 | load: 0.00294 [I 2022-10-08 09:52:12] ...trap/engines/engine.py.230: loss: 0.41475 [I 2022-10-08 09:52:12] ...optimizers/trijoint.py.151: total_norm: 4.840747 [I 2022-10-08 09:52:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 340/2304 [I 2022-10-08 09:52:19] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:25:08 [I 2022-10-08 09:52:19] ...trap/engines/engine.py.229: process: 0.76554 | load: 0.00295 [I 2022-10-08 09:52:19] ...trap/engines/engine.py.230: loss: 0.42495 [I 2022-10-08 09:52:19] ...optimizers/trijoint.py.151: total_norm: 4.675657 [I 2022-10-08 09:52:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 350/2304 [I 2022-10-08 09:52:27] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:25:07 [I 2022-10-08 09:52:27] ...trap/engines/engine.py.229: process: 0.76967 | load: 0.00294 [I 2022-10-08 09:52:27] ...trap/engines/engine.py.230: loss: 0.38122 [I 2022-10-08 09:52:27] ...optimizers/trijoint.py.151: total_norm: 4.087114 [I 2022-10-08 09:52:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 360/2304 [I 2022-10-08 09:52:35] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:24:59 [I 2022-10-08 09:52:35] ...trap/engines/engine.py.229: process: 0.77554 | load: 0.00294 [I 2022-10-08 09:52:35] ...trap/engines/engine.py.230: loss: 0.42170 [I 2022-10-08 09:52:35] ...optimizers/trijoint.py.151: total_norm: 3.748222 [I 2022-10-08 09:52:42] ...trap/engines/engine.py.224: train: epoch 0 | batch 370/2304 [I 2022-10-08 09:52:42] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:25:02 [I 2022-10-08 09:52:42] ...trap/engines/engine.py.229: process: 0.77089 | load: 0.00411 [I 2022-10-08 09:52:42] ...trap/engines/engine.py.230: loss: 0.35491 [I 2022-10-08 09:52:42] ...optimizers/trijoint.py.151: total_norm: 4.416185 [I 2022-10-08 09:52:50] ...trap/engines/engine.py.224: train: epoch 0 | batch 380/2304 [I 2022-10-08 09:52:50] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:24:42 [I 2022-10-08 09:52:50] ...trap/engines/engine.py.229: process: 0.76202 | load: 0.00295 [I 2022-10-08 09:52:50] ...trap/engines/engine.py.230: loss: 0.40300 [I 2022-10-08 09:52:50] ...optimizers/trijoint.py.151: total_norm: 4.051401 [I 2022-10-08 09:52:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 390/2304 [I 2022-10-08 09:52:58] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:24:41 [I 2022-10-08 09:52:58] ...trap/engines/engine.py.229: process: 0.76289 | load: 0.00295 [I 2022-10-08 09:52:58] ...trap/engines/engine.py.230: loss: 0.48765 [I 2022-10-08 09:52:58] ...optimizers/trijoint.py.151: total_norm: 5.566400 [I 2022-10-08 09:53:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 400/2304 [I 2022-10-08 09:53:06] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:24:27 [I 2022-10-08 09:53:06] ...trap/engines/engine.py.229: process: 0.75749 | load: 0.00309 [I 2022-10-08 09:53:06] ...trap/engines/engine.py.230: loss: 0.49588 [I 2022-10-08 09:53:06] ...optimizers/trijoint.py.151: total_norm: 7.238812 [I 2022-10-08 09:53:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 410/2304 [I 2022-10-08 09:53:13] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:24:10 [I 2022-10-08 09:53:13] ...trap/engines/engine.py.229: process: 0.76298 | load: 0.00293 [I 2022-10-08 09:53:13] ...trap/engines/engine.py.230: loss: 0.45459 [I 2022-10-08 09:53:13] ...optimizers/trijoint.py.151: total_norm: 4.496752 [I 2022-10-08 09:53:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 420/2304 [I 2022-10-08 09:53:21] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:24:10 [I 2022-10-08 09:53:21] ...trap/engines/engine.py.229: process: 0.78475 | load: 0.00295 [I 2022-10-08 09:53:21] ...trap/engines/engine.py.230: loss: 0.40666 [I 2022-10-08 09:53:21] ...optimizers/trijoint.py.151: total_norm: 3.797040 [I 2022-10-08 09:53:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 430/2304 [I 2022-10-08 09:53:29] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:24:02 [I 2022-10-08 09:53:29] ...trap/engines/engine.py.229: process: 0.76317 | load: 0.00293 [I 2022-10-08 09:53:29] ...trap/engines/engine.py.230: loss: 0.48494 [I 2022-10-08 09:53:29] ...optimizers/trijoint.py.151: total_norm: 4.845950 [I 2022-10-08 09:53:36] ...trap/engines/engine.py.224: train: epoch 0 | batch 440/2304 [I 2022-10-08 09:53:36] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:23:47 [I 2022-10-08 09:53:36] ...trap/engines/engine.py.229: process: 0.76049 | load: 0.00296 [I 2022-10-08 09:53:36] ...trap/engines/engine.py.230: loss: 0.52401 [I 2022-10-08 09:53:36] ...optimizers/trijoint.py.151: total_norm: 4.092865 [I 2022-10-08 09:53:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 450/2304 [I 2022-10-08 09:53:44] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:23:42 [I 2022-10-08 09:53:44] ...trap/engines/engine.py.229: process: 0.76243 | load: 0.00304 [I 2022-10-08 09:53:44] ...trap/engines/engine.py.230: loss: 0.39643 [I 2022-10-08 09:53:44] ...optimizers/trijoint.py.151: total_norm: 4.077346 [I 2022-10-08 09:53:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 460/2304 [I 2022-10-08 09:53:52] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:23:43 [I 2022-10-08 09:53:52] ...trap/engines/engine.py.229: process: 0.76579 | load: 0.00296 [I 2022-10-08 09:53:52] ...trap/engines/engine.py.230: loss: 0.32348 [I 2022-10-08 09:53:52] ...optimizers/trijoint.py.151: total_norm: 3.909770 [I 2022-10-08 09:54:00] ...trap/engines/engine.py.224: train: epoch 0 | batch 470/2304 [I 2022-10-08 09:54:00] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:23:27 [I 2022-10-08 09:54:00] ...trap/engines/engine.py.229: process: 0.76040 | load: 0.00295 [I 2022-10-08 09:54:00] ...trap/engines/engine.py.230: loss: 0.35724 [I 2022-10-08 09:54:00] ...optimizers/trijoint.py.151: total_norm: 4.042919 [I 2022-10-08 09:54:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 480/2304 [I 2022-10-08 09:54:07] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:24 [I 2022-10-08 09:54:07] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00300 [I 2022-10-08 09:54:07] ...trap/engines/engine.py.230: loss: 0.43289 [I 2022-10-08 09:54:07] ...optimizers/trijoint.py.151: total_norm: 5.065338 [I 2022-10-08 09:54:15] ...trap/engines/engine.py.224: train: epoch 0 | batch 490/2304 [I 2022-10-08 09:54:15] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:21 [I 2022-10-08 09:54:15] ...trap/engines/engine.py.229: process: 0.78439 | load: 0.00302 [I 2022-10-08 09:54:15] ...trap/engines/engine.py.230: loss: 0.37531 [I 2022-10-08 09:54:15] ...optimizers/trijoint.py.151: total_norm: 3.841912 [I 2022-10-08 09:54:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 500/2304 [I 2022-10-08 09:54:23] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:23:16 [I 2022-10-08 09:54:23] ...trap/engines/engine.py.229: process: 0.78148 | load: 0.00292 [I 2022-10-08 09:54:23] ...trap/engines/engine.py.230: loss: 0.44370 [I 2022-10-08 09:54:23] ...optimizers/trijoint.py.151: total_norm: 4.010732 [I 2022-10-08 09:54:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 510/2304 [I 2022-10-08 09:54:30] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:23:03 [I 2022-10-08 09:54:30] ...trap/engines/engine.py.229: process: 0.76925 | load: 0.00292 [I 2022-10-08 09:54:30] ...trap/engines/engine.py.230: loss: 0.54684 [I 2022-10-08 09:54:30] ...optimizers/trijoint.py.151: total_norm: 4.727596 [I 2022-10-08 09:54:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 520/2304 [I 2022-10-08 09:54:38] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:22:56 [I 2022-10-08 09:54:38] ...trap/engines/engine.py.229: process: 0.78054 | load: 0.00295 [I 2022-10-08 09:54:38] ...trap/engines/engine.py.230: loss: 0.42913 [I 2022-10-08 09:54:38] ...optimizers/trijoint.py.151: total_norm: 4.083988 [I 2022-10-08 09:54:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 530/2304 [I 2022-10-08 09:54:46] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:22:40 [I 2022-10-08 09:54:46] ...trap/engines/engine.py.229: process: 0.76451 | load: 0.00295 [I 2022-10-08 09:54:46] ...trap/engines/engine.py.230: loss: 0.45406 [I 2022-10-08 09:54:46] ...optimizers/trijoint.py.151: total_norm: 4.672812 [I 2022-10-08 09:54:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 540/2304 [I 2022-10-08 09:54:54] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:36 [I 2022-10-08 09:54:54] ...trap/engines/engine.py.229: process: 0.77086 | load: 0.00293 [I 2022-10-08 09:54:54] ...trap/engines/engine.py.230: loss: 0.40021 [I 2022-10-08 09:54:54] ...optimizers/trijoint.py.151: total_norm: 3.698544 [I 2022-10-08 09:55:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 550/2304 [I 2022-10-08 09:55:19] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:50:06 [I 2022-10-08 09:55:19] ...trap/engines/engine.py.229: process: 0.77004 | load: 0.00294 [I 2022-10-08 09:55:19] ...trap/engines/engine.py.230: loss: 0.38261 [I 2022-10-08 09:55:19] ...optimizers/trijoint.py.151: total_norm: 3.502150 [I 2022-10-08 09:55:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 560/2304 [I 2022-10-08 09:55:27] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:25:13 [I 2022-10-08 09:55:27] ...trap/engines/engine.py.229: process: 0.77669 | load: 0.00295 [I 2022-10-08 09:55:27] ...trap/engines/engine.py.230: loss: 0.53035 [I 2022-10-08 09:55:27] ...optimizers/trijoint.py.151: total_norm: 4.205788 [I 2022-10-08 09:55:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 570/2304 [I 2022-10-08 09:55:35] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:22:24 [I 2022-10-08 09:55:35] ...trap/engines/engine.py.229: process: 0.76350 | load: 0.00301 [I 2022-10-08 09:55:35] ...trap/engines/engine.py.230: loss: 0.41375 [I 2022-10-08 09:55:35] ...optimizers/trijoint.py.151: total_norm: 4.464187 [I 2022-10-08 09:55:42] ...trap/engines/engine.py.224: train: epoch 0 | batch 580/2304 [I 2022-10-08 09:55:42] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:22:06 [I 2022-10-08 09:55:42] ...trap/engines/engine.py.229: process: 0.77034 | load: 0.00294 [I 2022-10-08 09:55:42] ...trap/engines/engine.py.230: loss: 0.35052 [I 2022-10-08 09:55:42] ...optimizers/trijoint.py.151: total_norm: 4.654062 [I 2022-10-08 09:55:50] ...trap/engines/engine.py.224: train: epoch 0 | batch 590/2304 [I 2022-10-08 09:55:50] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:21:59 [I 2022-10-08 09:55:50] ...trap/engines/engine.py.229: process: 0.75864 | load: 0.00295 [I 2022-10-08 09:55:50] ...trap/engines/engine.py.230: loss: 0.41604 [I 2022-10-08 09:55:50] ...optimizers/trijoint.py.151: total_norm: 3.799495 [I 2022-10-08 09:55:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 600/2304 [I 2022-10-08 09:55:58] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:21:59 [I 2022-10-08 09:55:58] ...trap/engines/engine.py.229: process: 0.77505 | load: 0.00295 [I 2022-10-08 09:55:58] ...trap/engines/engine.py.230: loss: 0.42661 [I 2022-10-08 09:55:58] ...optimizers/trijoint.py.151: total_norm: 4.478862 [I 2022-10-08 09:56:05] ...trap/engines/engine.py.224: train: epoch 0 | batch 610/2304 [I 2022-10-08 09:56:05] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:21:42 [I 2022-10-08 09:56:05] ...trap/engines/engine.py.229: process: 0.76683 | load: 0.00293 [I 2022-10-08 09:56:05] ...trap/engines/engine.py.230: loss: 0.47443 [I 2022-10-08 09:56:05] ...optimizers/trijoint.py.151: total_norm: 5.562593 [I 2022-10-08 09:56:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 620/2304 [I 2022-10-08 09:56:13] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:21:38 [I 2022-10-08 09:56:13] ...trap/engines/engine.py.229: process: 0.78398 | load: 0.00292 [I 2022-10-08 09:56:13] ...trap/engines/engine.py.230: loss: 0.31803 [I 2022-10-08 09:56:13] ...optimizers/trijoint.py.151: total_norm: 3.605104 [I 2022-10-08 09:56:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 630/2304 [I 2022-10-08 09:56:21] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:21:24 [I 2022-10-08 09:56:21] ...trap/engines/engine.py.229: process: 0.76836 | load: 0.00294 [I 2022-10-08 09:56:21] ...trap/engines/engine.py.230: loss: 0.57248 [I 2022-10-08 09:56:21] ...optimizers/trijoint.py.151: total_norm: 6.019256 [I 2022-10-08 09:56:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 640/2304 [I 2022-10-08 09:56:29] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:21:13 [I 2022-10-08 09:56:29] ...trap/engines/engine.py.229: process: 0.76197 | load: 0.00295 [I 2022-10-08 09:56:29] ...trap/engines/engine.py.230: loss: 0.32228 [I 2022-10-08 09:56:29] ...optimizers/trijoint.py.151: total_norm: 3.935518 [I 2022-10-08 09:56:36] ...trap/engines/engine.py.224: train: epoch 0 | batch 650/2304 [I 2022-10-08 09:56:36] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:21:10 [I 2022-10-08 09:56:36] ...trap/engines/engine.py.229: process: 0.77232 | load: 0.00294 [I 2022-10-08 09:56:36] ...trap/engines/engine.py.230: loss: 0.47746 [I 2022-10-08 09:56:36] ...optimizers/trijoint.py.151: total_norm: 4.463667 [I 2022-10-08 09:56:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 660/2304 [I 2022-10-08 09:56:44] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:21:07 [I 2022-10-08 09:56:44] ...trap/engines/engine.py.229: process: 0.76793 | load: 0.00294 [I 2022-10-08 09:56:44] ...trap/engines/engine.py.230: loss: 0.49130 [I 2022-10-08 09:56:44] ...optimizers/trijoint.py.151: total_norm: 5.091074 [I 2022-10-08 09:56:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 670/2304 [I 2022-10-08 09:56:52] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:20:55 [I 2022-10-08 09:56:52] ...trap/engines/engine.py.229: process: 0.76680 | load: 0.00296 [I 2022-10-08 09:56:52] ...trap/engines/engine.py.230: loss: 0.46152 [I 2022-10-08 09:56:52] ...optimizers/trijoint.py.151: total_norm: 4.314686 [I 2022-10-08 09:56:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 680/2304 [I 2022-10-08 09:56:59] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:20:55 [I 2022-10-08 09:56:59] ...trap/engines/engine.py.229: process: 0.77747 | load: 0.00294 [I 2022-10-08 09:56:59] ...trap/engines/engine.py.230: loss: 0.45826 [I 2022-10-08 09:56:59] ...optimizers/trijoint.py.151: total_norm: 4.082682 [I 2022-10-08 09:57:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 690/2304 [I 2022-10-08 09:57:07] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:20:42 [I 2022-10-08 09:57:07] ...trap/engines/engine.py.229: process: 0.77148 | load: 0.00294 [I 2022-10-08 09:57:07] ...trap/engines/engine.py.230: loss: 0.34120 [I 2022-10-08 09:57:07] ...optimizers/trijoint.py.151: total_norm: 3.832869 [I 2022-10-08 09:57:15] ...trap/engines/engine.py.224: train: epoch 0 | batch 700/2304 [I 2022-10-08 09:57:15] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:20:37 [I 2022-10-08 09:57:15] ...trap/engines/engine.py.229: process: 0.76710 | load: 0.00292 [I 2022-10-08 09:57:15] ...trap/engines/engine.py.230: loss: 0.39548 [I 2022-10-08 09:57:15] ...optimizers/trijoint.py.151: total_norm: 6.034530 [I 2022-10-08 09:57:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 710/2304 [I 2022-10-08 09:57:22] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:20:31 [I 2022-10-08 09:57:23] ...trap/engines/engine.py.229: process: 0.77525 | load: 0.00294 [I 2022-10-08 09:57:23] ...trap/engines/engine.py.230: loss: 0.43975 [I 2022-10-08 09:57:23] ...optimizers/trijoint.py.151: total_norm: 3.708538 [I 2022-10-08 09:57:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 720/2304 [I 2022-10-08 09:57:30] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:24 [I 2022-10-08 09:57:30] ...trap/engines/engine.py.229: process: 0.76694 | load: 0.00291 [I 2022-10-08 09:57:30] ...trap/engines/engine.py.230: loss: 0.39595 [I 2022-10-08 09:57:30] ...optimizers/trijoint.py.151: total_norm: 3.462618 [I 2022-10-08 09:57:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 730/2304 [I 2022-10-08 09:57:38] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:14 [I 2022-10-08 09:57:38] ...trap/engines/engine.py.229: process: 0.78390 | load: 0.00295 [I 2022-10-08 09:57:38] ...trap/engines/engine.py.230: loss: 0.45857 [I 2022-10-08 09:57:38] ...optimizers/trijoint.py.151: total_norm: 3.824741 [I 2022-10-08 09:57:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 740/2304 [I 2022-10-08 09:57:46] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:19:55 [I 2022-10-08 09:57:46] ...trap/engines/engine.py.229: process: 0.76032 | load: 0.00291 [I 2022-10-08 09:57:46] ...trap/engines/engine.py.230: loss: 0.35184 [I 2022-10-08 09:57:46] ...optimizers/trijoint.py.151: total_norm: 3.683121 [I 2022-10-08 09:57:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 750/2304 [I 2022-10-08 09:57:53] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:57 [I 2022-10-08 09:57:53] ...trap/engines/engine.py.229: process: 0.77258 | load: 0.00293 [I 2022-10-08 09:57:53] ...trap/engines/engine.py.230: loss: 0.46030 [I 2022-10-08 09:57:53] ...optimizers/trijoint.py.151: total_norm: 4.780547 [I 2022-10-08 09:58:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 760/2304 [I 2022-10-08 09:58:01] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:51 [I 2022-10-08 09:58:01] ...trap/engines/engine.py.229: process: 0.78087 | load: 0.00296 [I 2022-10-08 09:58:01] ...trap/engines/engine.py.230: loss: 0.39391 [I 2022-10-08 09:58:01] ...optimizers/trijoint.py.151: total_norm: 4.095356 [I 2022-10-08 09:58:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 770/2304 [I 2022-10-08 09:58:09] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:50 [I 2022-10-08 09:58:09] ...trap/engines/engine.py.229: process: 0.77574 | load: 0.00296 [I 2022-10-08 09:58:09] ...trap/engines/engine.py.230: loss: 0.40515 [I 2022-10-08 09:58:09] ...optimizers/trijoint.py.151: total_norm: 4.197729 [I 2022-10-08 09:58:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 780/2304 [I 2022-10-08 09:58:17] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:38 [I 2022-10-08 09:58:17] ...trap/engines/engine.py.229: process: 0.76846 | load: 0.00294 [I 2022-10-08 09:58:17] ...trap/engines/engine.py.230: loss: 0.41250 [I 2022-10-08 09:58:17] ...optimizers/trijoint.py.151: total_norm: 4.353516 [I 2022-10-08 09:58:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 790/2304 [I 2022-10-08 09:58:24] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:31 [I 2022-10-08 09:58:24] ...trap/engines/engine.py.229: process: 0.78620 | load: 0.00293 [I 2022-10-08 09:58:24] ...trap/engines/engine.py.230: loss: 0.44548 [I 2022-10-08 09:58:24] ...optimizers/trijoint.py.151: total_norm: 4.059465 [I 2022-10-08 09:58:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 800/2304 [I 2022-10-08 09:58:32] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:22 [I 2022-10-08 09:58:32] ...trap/engines/engine.py.229: process: 0.77299 | load: 0.00293 [I 2022-10-08 09:58:32] ...trap/engines/engine.py.230: loss: 0.36032 [I 2022-10-08 09:58:32] ...optimizers/trijoint.py.151: total_norm: 4.206333 [I 2022-10-08 09:58:40] ...trap/engines/engine.py.224: train: epoch 0 | batch 810/2304 [I 2022-10-08 09:58:40] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:11 [I 2022-10-08 09:58:40] ...trap/engines/engine.py.229: process: 0.77287 | load: 0.00293 [I 2022-10-08 09:58:40] ...trap/engines/engine.py.230: loss: 0.26045 [I 2022-10-08 09:58:40] ...optimizers/trijoint.py.151: total_norm: 3.699981 [I 2022-10-08 09:58:47] ...trap/engines/engine.py.224: train: epoch 0 | batch 820/2304 [I 2022-10-08 09:58:47] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:19:05 [I 2022-10-08 09:58:47] ...trap/engines/engine.py.229: process: 0.77406 | load: 0.00300 [I 2022-10-08 09:58:47] ...trap/engines/engine.py.230: loss: 0.35583 [I 2022-10-08 09:58:47] ...optimizers/trijoint.py.151: total_norm: 3.214323 [I 2022-10-08 09:58:55] ...trap/engines/engine.py.224: train: epoch 0 | batch 830/2304 [I 2022-10-08 09:58:55] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:57 [I 2022-10-08 09:58:55] ...trap/engines/engine.py.229: process: 0.77223 | load: 0.00298 [I 2022-10-08 09:58:55] ...trap/engines/engine.py.230: loss: 0.38099 [I 2022-10-08 09:58:55] ...optimizers/trijoint.py.151: total_norm: 3.659186 [I 2022-10-08 09:59:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 840/2304 [I 2022-10-08 09:59:03] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:43 [I 2022-10-08 09:59:03] ...trap/engines/engine.py.229: process: 0.75749 | load: 0.00294 [I 2022-10-08 09:59:03] ...trap/engines/engine.py.230: loss: 0.36060 [I 2022-10-08 09:59:03] ...optimizers/trijoint.py.151: total_norm: 4.191692 [I 2022-10-08 09:59:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 850/2304 [I 2022-10-08 09:59:11] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:46 [I 2022-10-08 09:59:11] ...trap/engines/engine.py.229: process: 0.78518 | load: 0.00295 [I 2022-10-08 09:59:11] ...trap/engines/engine.py.230: loss: 0.40915 [I 2022-10-08 09:59:11] ...optimizers/trijoint.py.151: total_norm: 3.909260 [I 2022-10-08 09:59:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 860/2304 [I 2022-10-08 09:59:18] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:18:35 [I 2022-10-08 09:59:18] ...trap/engines/engine.py.229: process: 0.77325 | load: 0.00292 [I 2022-10-08 09:59:18] ...trap/engines/engine.py.230: loss: 0.41456 [I 2022-10-08 09:59:18] ...optimizers/trijoint.py.151: total_norm: 4.209318 [I 2022-10-08 09:59:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 870/2304 [I 2022-10-08 09:59:26] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:26 [I 2022-10-08 09:59:26] ...trap/engines/engine.py.229: process: 0.77087 | load: 0.00296 [I 2022-10-08 09:59:26] ...trap/engines/engine.py.230: loss: 0.33422 [I 2022-10-08 09:59:26] ...optimizers/trijoint.py.151: total_norm: 3.536204 [I 2022-10-08 09:59:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 880/2304 [I 2022-10-08 09:59:34] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:17 [I 2022-10-08 09:59:34] ...trap/engines/engine.py.229: process: 0.77133 | load: 0.00295 [I 2022-10-08 09:59:34] ...trap/engines/engine.py.230: loss: 0.32118 [I 2022-10-08 09:59:34] ...optimizers/trijoint.py.151: total_norm: 3.190287 [I 2022-10-08 09:59:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 890/2304 [I 2022-10-08 09:59:41] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:06 [I 2022-10-08 09:59:41] ...trap/engines/engine.py.229: process: 0.76041 | load: 0.00294 [I 2022-10-08 09:59:41] ...trap/engines/engine.py.230: loss: 0.36369 [I 2022-10-08 09:59:41] ...optimizers/trijoint.py.151: total_norm: 4.039547 [I 2022-10-08 09:59:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 900/2304 [I 2022-10-08 09:59:49] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:18:00 [I 2022-10-08 09:59:49] ...trap/engines/engine.py.229: process: 0.77241 | load: 0.00293 [I 2022-10-08 09:59:49] ...trap/engines/engine.py.230: loss: 0.37470 [I 2022-10-08 09:59:49] ...optimizers/trijoint.py.151: total_norm: 3.753491 [I 2022-10-08 09:59:57] ...trap/engines/engine.py.224: train: epoch 0 | batch 910/2304 [I 2022-10-08 09:59:57] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:53 [I 2022-10-08 09:59:57] ...trap/engines/engine.py.229: process: 0.77270 | load: 0.00295 [I 2022-10-08 09:59:57] ...trap/engines/engine.py.230: loss: 0.32900 [I 2022-10-08 09:59:57] ...optimizers/trijoint.py.151: total_norm: 3.384919 [I 2022-10-08 10:00:05] ...trap/engines/engine.py.224: train: epoch 0 | batch 920/2304 [I 2022-10-08 10:00:05] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:47 [I 2022-10-08 10:00:05] ...trap/engines/engine.py.229: process: 0.78310 | load: 0.00292 [I 2022-10-08 10:00:05] ...trap/engines/engine.py.230: loss: 0.32249 [I 2022-10-08 10:00:05] ...optimizers/trijoint.py.151: total_norm: 3.115732 [I 2022-10-08 10:00:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 930/2304 [I 2022-10-08 10:00:12] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:17:38 [I 2022-10-08 10:00:12] ...trap/engines/engine.py.229: process: 0.77349 | load: 0.00296 [I 2022-10-08 10:00:12] ...trap/engines/engine.py.230: loss: 0.33650 [I 2022-10-08 10:00:12] ...optimizers/trijoint.py.151: total_norm: 4.004681 [I 2022-10-08 10:00:20] ...trap/engines/engine.py.224: train: epoch 0 | batch 940/2304 [I 2022-10-08 10:00:20] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:17:28 [I 2022-10-08 10:00:20] ...trap/engines/engine.py.229: process: 0.76486 | load: 0.00295 [I 2022-10-08 10:00:20] ...trap/engines/engine.py.230: loss: 0.46098 [I 2022-10-08 10:00:20] ...optimizers/trijoint.py.151: total_norm: 4.260686 [I 2022-10-08 10:00:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 950/2304 [I 2022-10-08 10:00:28] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:17:23 [I 2022-10-08 10:00:28] ...trap/engines/engine.py.229: process: 0.76714 | load: 0.00293 [I 2022-10-08 10:00:28] ...trap/engines/engine.py.230: loss: 0.29921 [I 2022-10-08 10:00:28] ...optimizers/trijoint.py.151: total_norm: 3.116700 [I 2022-10-08 10:00:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 960/2304 [I 2022-10-08 10:00:35] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:17:12 [I 2022-10-08 10:00:35] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00304 [I 2022-10-08 10:00:35] ...trap/engines/engine.py.230: loss: 0.36725 [I 2022-10-08 10:00:35] ...optimizers/trijoint.py.151: total_norm: 4.627252 [I 2022-10-08 10:00:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 970/2304 [I 2022-10-08 10:00:43] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:17:07 [I 2022-10-08 10:00:43] ...trap/engines/engine.py.229: process: 0.77472 | load: 0.00300 [I 2022-10-08 10:00:43] ...trap/engines/engine.py.230: loss: 0.32459 [I 2022-10-08 10:00:43] ...optimizers/trijoint.py.151: total_norm: 3.583543 [I 2022-10-08 10:00:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 980/2304 [I 2022-10-08 10:00:51] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:17:00 [I 2022-10-08 10:00:51] ...trap/engines/engine.py.229: process: 0.77192 | load: 0.00294 [I 2022-10-08 10:00:51] ...trap/engines/engine.py.230: loss: 0.26608 [I 2022-10-08 10:00:51] ...optimizers/trijoint.py.151: total_norm: 3.288204 [I 2022-10-08 10:00:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 990/2304 [I 2022-10-08 10:00:59] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:16:55 [I 2022-10-08 10:00:59] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00295 [I 2022-10-08 10:00:59] ...trap/engines/engine.py.230: loss: 0.46056 [I 2022-10-08 10:00:59] ...optimizers/trijoint.py.151: total_norm: 4.388260 [I 2022-10-08 10:01:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 1000/2304 [I 2022-10-08 10:01:06] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:16:45 [I 2022-10-08 10:01:06] ...trap/engines/engine.py.229: process: 0.76816 | load: 0.00295 [I 2022-10-08 10:01:06] ...trap/engines/engine.py.230: loss: 0.31657 [I 2022-10-08 10:01:06] ...optimizers/trijoint.py.151: total_norm: 3.212867 [I 2022-10-08 10:01:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 1010/2304 [I 2022-10-08 10:01:14] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:16:32 [I 2022-10-08 10:01:14] ...trap/engines/engine.py.229: process: 0.75820 | load: 0.00295 [I 2022-10-08 10:01:14] ...trap/engines/engine.py.230: loss: 0.37076 [I 2022-10-08 10:01:14] ...optimizers/trijoint.py.151: total_norm: 3.797583 [I 2022-10-08 10:01:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 1020/2304 [I 2022-10-08 10:01:22] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:17:02 [I 2022-10-08 10:01:22] ...trap/engines/engine.py.229: process: 0.76960 | load: 0.00294 [I 2022-10-08 10:01:22] ...trap/engines/engine.py.230: loss: 0.29826 [I 2022-10-08 10:01:22] ...optimizers/trijoint.py.151: total_norm: 3.685013 [I 2022-10-08 10:01:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 1030/2304 [I 2022-10-08 10:01:30] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:16:23 [I 2022-10-08 10:01:30] ...trap/engines/engine.py.229: process: 0.75462 | load: 0.00295 [I 2022-10-08 10:01:30] ...trap/engines/engine.py.230: loss: 0.39976 [I 2022-10-08 10:01:30] ...optimizers/trijoint.py.151: total_norm: 4.232572 [I 2022-10-08 10:01:37] ...trap/engines/engine.py.224: train: epoch 0 | batch 1040/2304 [I 2022-10-08 10:01:37] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:16:11 [I 2022-10-08 10:01:37] ...trap/engines/engine.py.229: process: 0.76809 | load: 0.00295 [I 2022-10-08 10:01:37] ...trap/engines/engine.py.230: loss: 0.42978 [I 2022-10-08 10:01:37] ...optimizers/trijoint.py.151: total_norm: 4.574654 [I 2022-10-08 10:01:45] ...trap/engines/engine.py.224: train: epoch 0 | batch 1050/2304 [I 2022-10-08 10:01:45] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:16:09 [I 2022-10-08 10:01:45] ...trap/engines/engine.py.229: process: 0.78746 | load: 0.00296 [I 2022-10-08 10:01:45] ...trap/engines/engine.py.230: loss: 0.35867 [I 2022-10-08 10:01:45] ...optimizers/trijoint.py.151: total_norm: 3.300922 [I 2022-10-08 10:01:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1060/2304 [I 2022-10-08 10:01:53] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:16:00 [I 2022-10-08 10:01:53] ...trap/engines/engine.py.229: process: 0.77458 | load: 0.00296 [I 2022-10-08 10:01:53] ...trap/engines/engine.py.230: loss: 0.33976 [I 2022-10-08 10:01:53] ...optimizers/trijoint.py.151: total_norm: 3.437161 [I 2022-10-08 10:02:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 1070/2304 [I 2022-10-08 10:02:01] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:15:56 [I 2022-10-08 10:02:01] ...trap/engines/engine.py.229: process: 0.77857 | load: 0.00296 [I 2022-10-08 10:02:01] ...trap/engines/engine.py.230: loss: 0.34413 [I 2022-10-08 10:02:01] ...optimizers/trijoint.py.151: total_norm: 3.296316 [I 2022-10-08 10:02:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1080/2304 [I 2022-10-08 10:02:08] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:15:45 [I 2022-10-08 10:02:08] ...trap/engines/engine.py.229: process: 0.77785 | load: 0.00295 [I 2022-10-08 10:02:08] ...trap/engines/engine.py.230: loss: 0.35869 [I 2022-10-08 10:02:08] ...optimizers/trijoint.py.151: total_norm: 3.647161 [I 2022-10-08 10:02:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 1090/2304 [I 2022-10-08 10:02:16] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:15:33 [I 2022-10-08 10:02:16] ...trap/engines/engine.py.229: process: 0.76645 | load: 0.00293 [I 2022-10-08 10:02:16] ...trap/engines/engine.py.230: loss: 0.37223 [I 2022-10-08 10:02:16] ...optimizers/trijoint.py.151: total_norm: 4.081326 [I 2022-10-08 10:02:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 1100/2304 [I 2022-10-08 10:02:24] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:15:27 [I 2022-10-08 10:02:24] ...trap/engines/engine.py.229: process: 0.76744 | load: 0.00293 [I 2022-10-08 10:02:24] ...trap/engines/engine.py.230: loss: 0.34516 [I 2022-10-08 10:02:24] ...optimizers/trijoint.py.151: total_norm: 3.395481 [I 2022-10-08 10:02:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 1110/2304 [I 2022-10-08 10:02:31] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:15:21 [I 2022-10-08 10:02:31] ...trap/engines/engine.py.229: process: 0.77475 | load: 0.00295 [I 2022-10-08 10:02:31] ...trap/engines/engine.py.230: loss: 0.36644 [I 2022-10-08 10:02:31] ...optimizers/trijoint.py.151: total_norm: 3.995083 [I 2022-10-08 10:02:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 1120/2304 [I 2022-10-08 10:02:39] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:15:11 [I 2022-10-08 10:02:39] ...trap/engines/engine.py.229: process: 0.78065 | load: 0.00294 [I 2022-10-08 10:02:39] ...trap/engines/engine.py.230: loss: 0.36460 [I 2022-10-08 10:02:39] ...optimizers/trijoint.py.151: total_norm: 3.230664 [I 2022-10-08 10:02:47] ...trap/engines/engine.py.224: train: epoch 0 | batch 1130/2304 [I 2022-10-08 10:02:47] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:15:06 [I 2022-10-08 10:02:47] ...trap/engines/engine.py.229: process: 0.77315 | load: 0.00294 [I 2022-10-08 10:02:47] ...trap/engines/engine.py.230: loss: 0.23101 [I 2022-10-08 10:02:47] ...optimizers/trijoint.py.151: total_norm: 2.618243 [I 2022-10-08 10:02:55] ...trap/engines/engine.py.224: train: epoch 0 | batch 1140/2304 [I 2022-10-08 10:02:55] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:15:02 [I 2022-10-08 10:02:55] ...trap/engines/engine.py.229: process: 0.77852 | load: 0.00295 [I 2022-10-08 10:02:55] ...trap/engines/engine.py.230: loss: 0.41368 [I 2022-10-08 10:02:55] ...optimizers/trijoint.py.151: total_norm: 3.766073 [I 2022-10-08 10:03:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 1150/2304 [I 2022-10-08 10:03:02] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:14:47 [I 2022-10-08 10:03:02] ...trap/engines/engine.py.229: process: 0.76614 | load: 0.00294 [I 2022-10-08 10:03:02] ...trap/engines/engine.py.230: loss: 0.26793 [I 2022-10-08 10:03:02] ...optimizers/trijoint.py.151: total_norm: 3.193289 [I 2022-10-08 10:03:10] ...trap/engines/engine.py.224: train: epoch 0 | batch 1160/2304 [I 2022-10-08 10:03:10] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:14:44 [I 2022-10-08 10:03:10] ...trap/engines/engine.py.229: process: 0.78280 | load: 0.00293 [I 2022-10-08 10:03:10] ...trap/engines/engine.py.230: loss: 0.35347 [I 2022-10-08 10:03:10] ...optimizers/trijoint.py.151: total_norm: 3.403328 [I 2022-10-08 10:03:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1170/2304 [I 2022-10-08 10:03:18] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:14:35 [I 2022-10-08 10:03:18] ...trap/engines/engine.py.229: process: 0.77605 | load: 0.00296 [I 2022-10-08 10:03:18] ...trap/engines/engine.py.230: loss: 0.38989 [I 2022-10-08 10:03:18] ...optimizers/trijoint.py.151: total_norm: 3.848635 [I 2022-10-08 10:03:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 1180/2304 [I 2022-10-08 10:03:26] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:14:25 [I 2022-10-08 10:03:26] ...trap/engines/engine.py.229: process: 0.78105 | load: 0.00293 [I 2022-10-08 10:03:26] ...trap/engines/engine.py.230: loss: 0.31739 [I 2022-10-08 10:03:26] ...optimizers/trijoint.py.151: total_norm: 4.057147 [I 2022-10-08 10:03:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1190/2304 [I 2022-10-08 10:03:33] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:14:17 [I 2022-10-08 10:03:33] ...trap/engines/engine.py.229: process: 0.78194 | load: 0.00296 [I 2022-10-08 10:03:33] ...trap/engines/engine.py.230: loss: 0.33206 [I 2022-10-08 10:03:33] ...optimizers/trijoint.py.151: total_norm: 3.757220 [I 2022-10-08 10:03:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 1200/2304 [I 2022-10-08 10:03:41] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:14:13 [I 2022-10-08 10:03:41] ...trap/engines/engine.py.229: process: 0.77831 | load: 0.00292 [I 2022-10-08 10:03:41] ...trap/engines/engine.py.230: loss: 0.33650 [I 2022-10-08 10:03:41] ...optimizers/trijoint.py.151: total_norm: 3.474744 [I 2022-10-08 10:03:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 1210/2304 [I 2022-10-08 10:03:49] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:14:02 [I 2022-10-08 10:03:49] ...trap/engines/engine.py.229: process: 0.76741 | load: 0.00292 [I 2022-10-08 10:03:49] ...trap/engines/engine.py.230: loss: 0.26823 [I 2022-10-08 10:03:49] ...optimizers/trijoint.py.151: total_norm: 2.982683 [I 2022-10-08 10:03:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 1220/2304 [I 2022-10-08 10:03:56] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:14:00 [I 2022-10-08 10:03:56] ...trap/engines/engine.py.229: process: 0.78147 | load: 0.00294 [I 2022-10-08 10:03:56] ...trap/engines/engine.py.230: loss: 0.27914 [I 2022-10-08 10:03:56] ...optimizers/trijoint.py.151: total_norm: 2.844740 [I 2022-10-08 10:04:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 1230/2304 [I 2022-10-08 10:04:04] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:13:46 [I 2022-10-08 10:04:04] ...trap/engines/engine.py.229: process: 0.77793 | load: 0.00294 [I 2022-10-08 10:04:04] ...trap/engines/engine.py.230: loss: 0.31653 [I 2022-10-08 10:04:04] ...optimizers/trijoint.py.151: total_norm: 3.445713 [I 2022-10-08 10:04:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 1240/2304 [I 2022-10-08 10:04:12] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:13:35 [I 2022-10-08 10:04:12] ...trap/engines/engine.py.229: process: 0.76123 | load: 0.00293 [I 2022-10-08 10:04:12] ...trap/engines/engine.py.230: loss: 0.37861 [I 2022-10-08 10:04:12] ...optimizers/trijoint.py.151: total_norm: 4.409808 [I 2022-10-08 10:04:20] ...trap/engines/engine.py.224: train: epoch 0 | batch 1250/2304 [I 2022-10-08 10:04:20] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:13:38 [I 2022-10-08 10:04:20] ...trap/engines/engine.py.229: process: 0.77213 | load: 0.00293 [I 2022-10-08 10:04:20] ...trap/engines/engine.py.230: loss: 0.38588 [I 2022-10-08 10:04:20] ...optimizers/trijoint.py.151: total_norm: 4.085744 [I 2022-10-08 10:04:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 1260/2304 [I 2022-10-08 10:04:27] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:13:30 [I 2022-10-08 10:04:27] ...trap/engines/engine.py.229: process: 0.79018 | load: 0.00295 [I 2022-10-08 10:04:27] ...trap/engines/engine.py.230: loss: 0.29988 [I 2022-10-08 10:04:27] ...optimizers/trijoint.py.151: total_norm: 2.856035 [I 2022-10-08 10:04:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 1270/2304 [I 2022-10-08 10:04:35] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:13:16 [I 2022-10-08 10:04:35] ...trap/engines/engine.py.229: process: 0.76704 | load: 0.00294 [I 2022-10-08 10:04:35] ...trap/engines/engine.py.230: loss: 0.29889 [I 2022-10-08 10:04:35] ...optimizers/trijoint.py.151: total_norm: 3.425593 [I 2022-10-08 10:04:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1280/2304 [I 2022-10-08 10:04:43] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:13:09 [I 2022-10-08 10:04:43] ...trap/engines/engine.py.229: process: 0.77132 | load: 0.00294 [I 2022-10-08 10:04:43] ...trap/engines/engine.py.230: loss: 0.36090 [I 2022-10-08 10:04:43] ...optimizers/trijoint.py.151: total_norm: 3.259166 [I 2022-10-08 10:04:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 1290/2304 [I 2022-10-08 10:04:51] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:12:59 [I 2022-10-08 10:04:51] ...trap/engines/engine.py.229: process: 0.75765 | load: 0.00293 [I 2022-10-08 10:04:51] ...trap/engines/engine.py.230: loss: 0.42876 [I 2022-10-08 10:04:51] ...optimizers/trijoint.py.151: total_norm: 4.176527 [I 2022-10-08 10:04:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1300/2304 [I 2022-10-08 10:04:58] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:12:48 [I 2022-10-08 10:04:58] ...trap/engines/engine.py.229: process: 0.76417 | load: 0.00294 [I 2022-10-08 10:04:58] ...trap/engines/engine.py.230: loss: 0.35193 [I 2022-10-08 10:04:58] ...optimizers/trijoint.py.151: total_norm: 4.641404 [I 2022-10-08 10:05:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 1310/2304 [I 2022-10-08 10:05:06] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:12:41 [I 2022-10-08 10:05:06] ...trap/engines/engine.py.229: process: 0.76380 | load: 0.00295 [I 2022-10-08 10:05:06] ...trap/engines/engine.py.230: loss: 0.32781 [I 2022-10-08 10:05:06] ...optimizers/trijoint.py.151: total_norm: 3.123924 [I 2022-10-08 10:05:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 1320/2304 [I 2022-10-08 10:05:14] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:12:39 [I 2022-10-08 10:05:14] ...trap/engines/engine.py.229: process: 0.76442 | load: 0.00293 [I 2022-10-08 10:05:14] ...trap/engines/engine.py.230: loss: 0.32958 [I 2022-10-08 10:05:14] ...optimizers/trijoint.py.151: total_norm: 3.774244 [I 2022-10-08 10:05:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 1330/2304 [I 2022-10-08 10:05:21] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:12:32 [I 2022-10-08 10:05:21] ...trap/engines/engine.py.229: process: 0.77691 | load: 0.00296 [I 2022-10-08 10:05:21] ...trap/engines/engine.py.230: loss: 0.40819 [I 2022-10-08 10:05:21] ...optimizers/trijoint.py.151: total_norm: 3.744395 [I 2022-10-08 10:05:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 1340/2304 [I 2022-10-08 10:05:29] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:12:22 [I 2022-10-08 10:05:29] ...trap/engines/engine.py.229: process: 0.76703 | load: 0.00295 [I 2022-10-08 10:05:29] ...trap/engines/engine.py.230: loss: 0.36603 [I 2022-10-08 10:05:29] ...optimizers/trijoint.py.151: total_norm: 3.927079 [I 2022-10-08 10:05:37] ...trap/engines/engine.py.224: train: epoch 0 | batch 1350/2304 [I 2022-10-08 10:05:37] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:12:10 [I 2022-10-08 10:05:37] ...trap/engines/engine.py.229: process: 0.76361 | load: 0.00294 [I 2022-10-08 10:05:37] ...trap/engines/engine.py.230: loss: 0.40419 [I 2022-10-08 10:05:37] ...optimizers/trijoint.py.151: total_norm: 3.445321 [I 2022-10-08 10:05:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 1360/2304 [I 2022-10-08 10:05:44] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:12:11 [I 2022-10-08 10:05:44] ...trap/engines/engine.py.229: process: 0.77887 | load: 0.00294 [I 2022-10-08 10:05:44] ...trap/engines/engine.py.230: loss: 0.30664 [I 2022-10-08 10:05:44] ...optimizers/trijoint.py.151: total_norm: 3.173346 [I 2022-10-08 10:05:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 1370/2304 [I 2022-10-08 10:05:52] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:11:59 [I 2022-10-08 10:05:52] ...trap/engines/engine.py.229: process: 0.76923 | load: 0.00296 [I 2022-10-08 10:05:52] ...trap/engines/engine.py.230: loss: 0.34051 [I 2022-10-08 10:05:52] ...optimizers/trijoint.py.151: total_norm: 3.680816 [I 2022-10-08 10:06:00] ...trap/engines/engine.py.224: train: epoch 0 | batch 1380/2304 [I 2022-10-08 10:06:00] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:11:50 [I 2022-10-08 10:06:00] ...trap/engines/engine.py.229: process: 0.76895 | load: 0.00293 [I 2022-10-08 10:06:00] ...trap/engines/engine.py.230: loss: 0.35509 [I 2022-10-08 10:06:00] ...optimizers/trijoint.py.151: total_norm: 4.596752 [I 2022-10-08 10:06:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1390/2304 [I 2022-10-08 10:06:08] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:11:40 [I 2022-10-08 10:06:08] ...trap/engines/engine.py.229: process: 0.75336 | load: 0.00294 [I 2022-10-08 10:06:08] ...trap/engines/engine.py.230: loss: 0.38252 [I 2022-10-08 10:06:08] ...optimizers/trijoint.py.151: total_norm: 4.653232 [I 2022-10-08 10:06:15] ...trap/engines/engine.py.224: train: epoch 0 | batch 1400/2304 [I 2022-10-08 10:06:15] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:11:35 [I 2022-10-08 10:06:15] ...trap/engines/engine.py.229: process: 0.77332 | load: 0.00293 [I 2022-10-08 10:06:15] ...trap/engines/engine.py.230: loss: 0.29908 [I 2022-10-08 10:06:15] ...optimizers/trijoint.py.151: total_norm: 3.275329 [I 2022-10-08 10:06:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1410/2304 [I 2022-10-08 10:06:23] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:11:27 [I 2022-10-08 10:06:23] ...trap/engines/engine.py.229: process: 0.77636 | load: 0.00297 [I 2022-10-08 10:06:23] ...trap/engines/engine.py.230: loss: 0.39188 [I 2022-10-08 10:06:23] ...optimizers/trijoint.py.151: total_norm: 3.516314 [I 2022-10-08 10:06:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 1420/2304 [I 2022-10-08 10:06:31] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:11:21 [I 2022-10-08 10:06:31] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00295 [I 2022-10-08 10:06:31] ...trap/engines/engine.py.230: loss: 0.37921 [I 2022-10-08 10:06:31] ...optimizers/trijoint.py.151: total_norm: 4.276746 [I 2022-10-08 10:06:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1430/2304 [I 2022-10-08 10:06:38] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:11:08 [I 2022-10-08 10:06:38] ...trap/engines/engine.py.229: process: 0.76285 | load: 0.00296 [I 2022-10-08 10:06:38] ...trap/engines/engine.py.230: loss: 0.38012 [I 2022-10-08 10:06:38] ...optimizers/trijoint.py.151: total_norm: 4.170098 [I 2022-10-08 10:06:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 1440/2304 [I 2022-10-08 10:06:46] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:11:07 [I 2022-10-08 10:06:46] ...trap/engines/engine.py.229: process: 0.77422 | load: 0.00293 [I 2022-10-08 10:06:46] ...trap/engines/engine.py.230: loss: 0.33791 [I 2022-10-08 10:06:46] ...optimizers/trijoint.py.151: total_norm: 3.560058 [I 2022-10-08 10:06:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 1450/2304 [I 2022-10-08 10:06:54] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:11:00 [I 2022-10-08 10:06:54] ...trap/engines/engine.py.229: process: 0.77703 | load: 0.00295 [I 2022-10-08 10:06:54] ...trap/engines/engine.py.230: loss: 0.29517 [I 2022-10-08 10:06:54] ...optimizers/trijoint.py.151: total_norm: 3.330743 [I 2022-10-08 10:07:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 1460/2304 [I 2022-10-08 10:07:02] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:10:50 [I 2022-10-08 10:07:02] ...trap/engines/engine.py.229: process: 0.76287 | load: 0.00294 [I 2022-10-08 10:07:02] ...trap/engines/engine.py.230: loss: 0.37106 [I 2022-10-08 10:07:02] ...optimizers/trijoint.py.151: total_norm: 3.705077 [I 2022-10-08 10:07:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 1470/2304 [I 2022-10-08 10:07:09] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:10:42 [I 2022-10-08 10:07:09] ...trap/engines/engine.py.229: process: 0.77119 | load: 0.00293 [I 2022-10-08 10:07:09] ...trap/engines/engine.py.230: loss: 0.30468 [I 2022-10-08 10:07:09] ...optimizers/trijoint.py.151: total_norm: 3.988825 [I 2022-10-08 10:07:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 1480/2304 [I 2022-10-08 10:07:17] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:10:33 [I 2022-10-08 10:07:17] ...trap/engines/engine.py.229: process: 0.75722 | load: 0.00294 [I 2022-10-08 10:07:17] ...trap/engines/engine.py.230: loss: 0.34471 [I 2022-10-08 10:07:17] ...optimizers/trijoint.py.151: total_norm: 5.076855 [I 2022-10-08 10:07:25] ...trap/engines/engine.py.224: train: epoch 0 | batch 1490/2304 [I 2022-10-08 10:07:25] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:10:25 [I 2022-10-08 10:07:25] ...trap/engines/engine.py.229: process: 0.76954 | load: 0.00296 [I 2022-10-08 10:07:25] ...trap/engines/engine.py.230: loss: 0.24911 [I 2022-10-08 10:07:25] ...optimizers/trijoint.py.151: total_norm: 3.522743 [I 2022-10-08 10:07:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 1500/2304 [I 2022-10-08 10:07:32] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:10:20 [I 2022-10-08 10:07:32] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00295 [I 2022-10-08 10:07:32] ...trap/engines/engine.py.230: loss: 0.32214 [I 2022-10-08 10:07:32] ...optimizers/trijoint.py.151: total_norm: 3.010885 [I 2022-10-08 10:07:40] ...trap/engines/engine.py.224: train: epoch 0 | batch 1510/2304 [I 2022-10-08 10:07:40] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:13 [I 2022-10-08 10:07:40] ...trap/engines/engine.py.229: process: 0.77465 | load: 0.00297 [I 2022-10-08 10:07:40] ...trap/engines/engine.py.230: loss: 0.33154 [I 2022-10-08 10:07:40] ...optimizers/trijoint.py.151: total_norm: 3.775820 [I 2022-10-08 10:07:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1520/2304 [I 2022-10-08 10:07:48] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:10:11 [I 2022-10-08 10:07:48] ...trap/engines/engine.py.229: process: 0.82046 | load: 0.04895 [I 2022-10-08 10:07:48] ...trap/engines/engine.py.230: loss: 0.42205 [I 2022-10-08 10:07:48] ...optimizers/trijoint.py.151: total_norm: 5.004021 [I 2022-10-08 10:07:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 1530/2304 [I 2022-10-08 10:07:56] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:09:57 [I 2022-10-08 10:07:56] ...trap/engines/engine.py.229: process: 0.76407 | load: 0.00295 [I 2022-10-08 10:07:56] ...trap/engines/engine.py.230: loss: 0.38687 [I 2022-10-08 10:07:56] ...optimizers/trijoint.py.151: total_norm: 3.682928 [I 2022-10-08 10:08:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1540/2304 [I 2022-10-08 10:08:03] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:09:47 [I 2022-10-08 10:08:03] ...trap/engines/engine.py.229: process: 0.76590 | load: 0.00293 [I 2022-10-08 10:08:03] ...trap/engines/engine.py.230: loss: 0.28223 [I 2022-10-08 10:08:03] ...optimizers/trijoint.py.151: total_norm: 3.096337 [I 2022-10-08 10:08:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 1550/2304 [I 2022-10-08 10:08:11] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:39 [I 2022-10-08 10:08:11] ...trap/engines/engine.py.229: process: 0.76456 | load: 0.00294 [I 2022-10-08 10:08:11] ...trap/engines/engine.py.230: loss: 0.38978 [I 2022-10-08 10:08:11] ...optimizers/trijoint.py.151: total_norm: 4.384639 [I 2022-10-08 10:08:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 1560/2304 [I 2022-10-08 10:08:19] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:09:33 [I 2022-10-08 10:08:19] ...trap/engines/engine.py.229: process: 0.76561 | load: 0.00293 [I 2022-10-08 10:08:19] ...trap/engines/engine.py.230: loss: 0.32151 [I 2022-10-08 10:08:19] ...optimizers/trijoint.py.151: total_norm: 3.313281 [I 2022-10-08 10:08:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 1570/2304 [I 2022-10-08 10:08:27] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:09:23 [I 2022-10-08 10:08:27] ...trap/engines/engine.py.229: process: 0.77093 | load: 0.00295 [I 2022-10-08 10:08:27] ...trap/engines/engine.py.230: loss: 0.41670 [I 2022-10-08 10:08:27] ...optimizers/trijoint.py.151: total_norm: 4.069077 [I 2022-10-08 10:08:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 1580/2304 [I 2022-10-08 10:08:34] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:17 [I 2022-10-08 10:08:34] ...trap/engines/engine.py.229: process: 0.78564 | load: 0.00293 [I 2022-10-08 10:08:34] ...trap/engines/engine.py.230: loss: 0.29486 [I 2022-10-08 10:08:34] ...optimizers/trijoint.py.151: total_norm: 3.268868 [I 2022-10-08 10:08:42] ...trap/engines/engine.py.224: train: epoch 0 | batch 1590/2304 [I 2022-10-08 10:08:42] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:10 [I 2022-10-08 10:08:42] ...trap/engines/engine.py.229: process: 0.78228 | load: 0.00295 [I 2022-10-08 10:08:42] ...trap/engines/engine.py.230: loss: 0.28157 [I 2022-10-08 10:08:42] ...optimizers/trijoint.py.151: total_norm: 3.287538 [I 2022-10-08 10:08:50] ...trap/engines/engine.py.224: train: epoch 0 | batch 1600/2304 [I 2022-10-08 10:08:50] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:08:58 [I 2022-10-08 10:08:50] ...trap/engines/engine.py.229: process: 0.76025 | load: 0.00292 [I 2022-10-08 10:08:50] ...trap/engines/engine.py.230: loss: 0.40187 [I 2022-10-08 10:08:50] ...optimizers/trijoint.py.151: total_norm: 3.430529 [I 2022-10-08 10:08:57] ...trap/engines/engine.py.224: train: epoch 0 | batch 1610/2304 [I 2022-10-08 10:08:57] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:08:56 [I 2022-10-08 10:08:57] ...trap/engines/engine.py.229: process: 0.77694 | load: 0.00292 [I 2022-10-08 10:08:57] ...trap/engines/engine.py.230: loss: 0.36961 [I 2022-10-08 10:08:57] ...optimizers/trijoint.py.151: total_norm: 3.841044 [I 2022-10-08 10:09:05] ...trap/engines/engine.py.224: train: epoch 0 | batch 1620/2304 [I 2022-10-08 10:09:05] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:46 [I 2022-10-08 10:09:05] ...trap/engines/engine.py.229: process: 0.75537 | load: 0.00294 [I 2022-10-08 10:09:05] ...trap/engines/engine.py.230: loss: 0.39683 [I 2022-10-08 10:09:05] ...optimizers/trijoint.py.151: total_norm: 5.036132 [I 2022-10-08 10:09:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1630/2304 [I 2022-10-08 10:09:13] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:42 [I 2022-10-08 10:09:13] ...trap/engines/engine.py.229: process: 0.77432 | load: 0.00297 [I 2022-10-08 10:09:13] ...trap/engines/engine.py.230: loss: 0.37438 [I 2022-10-08 10:09:13] ...optimizers/trijoint.py.151: total_norm: 3.087934 [I 2022-10-08 10:09:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 1640/2304 [I 2022-10-08 10:09:21] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:33 [I 2022-10-08 10:09:21] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00297 [I 2022-10-08 10:09:21] ...trap/engines/engine.py.230: loss: 0.34583 [I 2022-10-08 10:09:21] ...optimizers/trijoint.py.151: total_norm: 3.677470 [I 2022-10-08 10:09:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1650/2304 [I 2022-10-08 10:09:28] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:21 [I 2022-10-08 10:09:28] ...trap/engines/engine.py.229: process: 0.76773 | load: 0.00294 [I 2022-10-08 10:09:28] ...trap/engines/engine.py.230: loss: 0.36207 [I 2022-10-08 10:09:28] ...optimizers/trijoint.py.151: total_norm: 3.452876 [I 2022-10-08 10:09:36] ...trap/engines/engine.py.224: train: epoch 0 | batch 1660/2304 [I 2022-10-08 10:09:36] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:08:21 [I 2022-10-08 10:09:36] ...trap/engines/engine.py.229: process: 0.79011 | load: 0.00297 [I 2022-10-08 10:09:36] ...trap/engines/engine.py.230: loss: 0.33347 [I 2022-10-08 10:09:36] ...optimizers/trijoint.py.151: total_norm: 3.587088 [I 2022-10-08 10:09:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 1670/2304 [I 2022-10-08 10:09:44] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:07 [I 2022-10-08 10:09:44] ...trap/engines/engine.py.229: process: 0.76315 | load: 0.00294 [I 2022-10-08 10:09:44] ...trap/engines/engine.py.230: loss: 0.40164 [I 2022-10-08 10:09:44] ...optimizers/trijoint.py.151: total_norm: 3.618401 [I 2022-10-08 10:09:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 1680/2304 [I 2022-10-08 10:09:52] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:08:02 [I 2022-10-08 10:09:52] ...trap/engines/engine.py.229: process: 0.78880 | load: 0.00312 [I 2022-10-08 10:09:52] ...trap/engines/engine.py.230: loss: 0.38226 [I 2022-10-08 10:09:52] ...optimizers/trijoint.py.151: total_norm: 3.428425 [I 2022-10-08 10:09:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 1690/2304 [I 2022-10-08 10:09:59] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:07:51 [I 2022-10-08 10:09:59] ...trap/engines/engine.py.229: process: 0.76292 | load: 0.00294 [I 2022-10-08 10:09:59] ...trap/engines/engine.py.230: loss: 0.38796 [I 2022-10-08 10:09:59] ...optimizers/trijoint.py.151: total_norm: 4.377201 [I 2022-10-08 10:10:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 1700/2304 [I 2022-10-08 10:10:07] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:07:45 [I 2022-10-08 10:10:07] ...trap/engines/engine.py.229: process: 0.76099 | load: 0.00292 [I 2022-10-08 10:10:07] ...trap/engines/engine.py.230: loss: 0.36060 [I 2022-10-08 10:10:07] ...optimizers/trijoint.py.151: total_norm: 3.832796 [I 2022-10-08 10:10:15] ...trap/engines/engine.py.224: train: epoch 0 | batch 1710/2304 [I 2022-10-08 10:10:15] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:07:38 [I 2022-10-08 10:10:15] ...trap/engines/engine.py.229: process: 0.76868 | load: 0.00291 [I 2022-10-08 10:10:15] ...trap/engines/engine.py.230: loss: 0.34151 [I 2022-10-08 10:10:15] ...optimizers/trijoint.py.151: total_norm: 3.556606 [I 2022-10-08 10:10:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 1720/2304 [I 2022-10-08 10:10:22] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:33 [I 2022-10-08 10:10:22] ...trap/engines/engine.py.229: process: 0.78434 | load: 0.00300 [I 2022-10-08 10:10:23] ...trap/engines/engine.py.230: loss: 0.35602 [I 2022-10-08 10:10:23] ...optimizers/trijoint.py.151: total_norm: 3.616595 [I 2022-10-08 10:10:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 1730/2304 [I 2022-10-08 10:10:30] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:07:22 [I 2022-10-08 10:10:30] ...trap/engines/engine.py.229: process: 0.76708 | load: 0.00303 [I 2022-10-08 10:10:30] ...trap/engines/engine.py.230: loss: 0.33265 [I 2022-10-08 10:10:30] ...optimizers/trijoint.py.151: total_norm: 3.576402 [I 2022-10-08 10:10:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1740/2304 [I 2022-10-08 10:10:38] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:07:15 [I 2022-10-08 10:10:38] ...trap/engines/engine.py.229: process: 0.77709 | load: 0.00303 [I 2022-10-08 10:10:38] ...trap/engines/engine.py.230: loss: 0.33630 [I 2022-10-08 10:10:38] ...optimizers/trijoint.py.151: total_norm: 3.891572 [I 2022-10-08 10:10:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 1750/2304 [I 2022-10-08 10:10:46] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:07:05 [I 2022-10-08 10:10:46] ...trap/engines/engine.py.229: process: 0.76242 | load: 0.00291 [I 2022-10-08 10:10:46] ...trap/engines/engine.py.230: loss: 0.26386 [I 2022-10-08 10:10:46] ...optimizers/trijoint.py.151: total_norm: 2.965699 [I 2022-10-08 10:10:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1760/2304 [I 2022-10-08 10:10:53] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:57 [I 2022-10-08 10:10:53] ...trap/engines/engine.py.229: process: 0.77146 | load: 0.00295 [I 2022-10-08 10:10:53] ...trap/engines/engine.py.230: loss: 0.34004 [I 2022-10-08 10:10:53] ...optimizers/trijoint.py.151: total_norm: 3.568938 [I 2022-10-08 10:11:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 1770/2304 [I 2022-10-08 10:11:01] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:06:52 [I 2022-10-08 10:11:01] ...trap/engines/engine.py.229: process: 0.77221 | load: 0.00295 [I 2022-10-08 10:11:01] ...trap/engines/engine.py.230: loss: 0.38752 [I 2022-10-08 10:11:01] ...optimizers/trijoint.py.151: total_norm: 4.618095 [I 2022-10-08 10:11:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 1780/2304 [I 2022-10-08 10:11:09] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:06:43 [I 2022-10-08 10:11:09] ...trap/engines/engine.py.229: process: 0.77352 | load: 0.00295 [I 2022-10-08 10:11:09] ...trap/engines/engine.py.230: loss: 0.33876 [I 2022-10-08 10:11:09] ...optimizers/trijoint.py.151: total_norm: 4.690897 [I 2022-10-08 10:11:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 1790/2304 [I 2022-10-08 10:11:16] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:06:34 [I 2022-10-08 10:11:16] ...trap/engines/engine.py.229: process: 0.76030 | load: 0.00294 [I 2022-10-08 10:11:16] ...trap/engines/engine.py.230: loss: 0.33372 [I 2022-10-08 10:11:16] ...optimizers/trijoint.py.151: total_norm: 3.062438 [I 2022-10-08 10:11:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 1800/2304 [I 2022-10-08 10:11:24] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:06:26 [I 2022-10-08 10:11:24] ...trap/engines/engine.py.229: process: 0.77142 | load: 0.00295 [I 2022-10-08 10:11:24] ...trap/engines/engine.py.230: loss: 0.30305 [I 2022-10-08 10:11:24] ...optimizers/trijoint.py.151: total_norm: 3.075769 [I 2022-10-08 10:11:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 1810/2304 [I 2022-10-08 10:11:32] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:06:20 [I 2022-10-08 10:11:32] ...trap/engines/engine.py.229: process: 0.77305 | load: 0.00293 [I 2022-10-08 10:11:32] ...trap/engines/engine.py.230: loss: 0.27355 [I 2022-10-08 10:11:32] ...optimizers/trijoint.py.151: total_norm: 3.644370 [I 2022-10-08 10:11:40] ...trap/engines/engine.py.224: train: epoch 0 | batch 1820/2304 [I 2022-10-08 10:11:40] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:06:09 [I 2022-10-08 10:11:40] ...trap/engines/engine.py.229: process: 0.76189 | load: 0.00296 [I 2022-10-08 10:11:40] ...trap/engines/engine.py.230: loss: 0.30014 [I 2022-10-08 10:11:40] ...optimizers/trijoint.py.151: total_norm: 3.695169 [I 2022-10-08 10:11:47] ...trap/engines/engine.py.224: train: epoch 0 | batch 1830/2304 [I 2022-10-08 10:11:47] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:06:04 [I 2022-10-08 10:11:47] ...trap/engines/engine.py.229: process: 0.77198 | load: 0.00294 [I 2022-10-08 10:11:47] ...trap/engines/engine.py.230: loss: 0.32165 [I 2022-10-08 10:11:47] ...optimizers/trijoint.py.151: total_norm: 3.279722 [I 2022-10-08 10:11:55] ...trap/engines/engine.py.224: train: epoch 0 | batch 1840/2304 [I 2022-10-08 10:11:55] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:05:57 [I 2022-10-08 10:11:55] ...trap/engines/engine.py.229: process: 0.77124 | load: 0.00294 [I 2022-10-08 10:11:55] ...trap/engines/engine.py.230: loss: 0.47569 [I 2022-10-08 10:11:55] ...optimizers/trijoint.py.151: total_norm: 4.890207 [I 2022-10-08 10:12:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1850/2304 [I 2022-10-08 10:12:03] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:05:51 [I 2022-10-08 10:12:03] ...trap/engines/engine.py.229: process: 0.78063 | load: 0.00294 [I 2022-10-08 10:12:03] ...trap/engines/engine.py.230: loss: 0.32351 [I 2022-10-08 10:12:03] ...optimizers/trijoint.py.151: total_norm: 3.134176 [I 2022-10-08 10:12:10] ...trap/engines/engine.py.224: train: epoch 0 | batch 1860/2304 [I 2022-10-08 10:12:10] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:05:43 [I 2022-10-08 10:12:10] ...trap/engines/engine.py.229: process: 0.76740 | load: 0.00291 [I 2022-10-08 10:12:10] ...trap/engines/engine.py.230: loss: 0.31981 [I 2022-10-08 10:12:10] ...optimizers/trijoint.py.151: total_norm: 2.667709 [I 2022-10-08 10:12:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1870/2304 [I 2022-10-08 10:12:18] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:05:34 [I 2022-10-08 10:12:18] ...trap/engines/engine.py.229: process: 0.76378 | load: 0.00291 [I 2022-10-08 10:12:18] ...trap/engines/engine.py.230: loss: 0.30293 [I 2022-10-08 10:12:18] ...optimizers/trijoint.py.151: total_norm: 3.544697 [I 2022-10-08 10:12:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 1880/2304 [I 2022-10-08 10:12:26] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:05:28 [I 2022-10-08 10:12:26] ...trap/engines/engine.py.229: process: 0.76642 | load: 0.00296 [I 2022-10-08 10:12:26] ...trap/engines/engine.py.230: loss: 0.34419 [I 2022-10-08 10:12:26] ...optimizers/trijoint.py.151: total_norm: 3.452512 [I 2022-10-08 10:12:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 1890/2304 [I 2022-10-08 10:12:34] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:05:19 [I 2022-10-08 10:12:34] ...trap/engines/engine.py.229: process: 0.77040 | load: 0.00294 [I 2022-10-08 10:12:34] ...trap/engines/engine.py.230: loss: 0.26311 [I 2022-10-08 10:12:34] ...optimizers/trijoint.py.151: total_norm: 3.665986 [I 2022-10-08 10:12:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 1900/2304 [I 2022-10-08 10:12:41] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:05:10 [I 2022-10-08 10:12:41] ...trap/engines/engine.py.229: process: 0.76754 | load: 0.00294 [I 2022-10-08 10:12:41] ...trap/engines/engine.py.230: loss: 0.36290 [I 2022-10-08 10:12:41] ...optimizers/trijoint.py.151: total_norm: 3.295468 [I 2022-10-08 10:12:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 1910/2304 [I 2022-10-08 10:12:49] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:05:02 [I 2022-10-08 10:12:49] ...trap/engines/engine.py.229: process: 0.76353 | load: 0.00293 [I 2022-10-08 10:12:49] ...trap/engines/engine.py.230: loss: 0.25334 [I 2022-10-08 10:12:49] ...optimizers/trijoint.py.151: total_norm: 4.155717 [I 2022-10-08 10:12:57] ...trap/engines/engine.py.224: train: epoch 0 | batch 1920/2304 [I 2022-10-08 10:12:57] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:04:54 [I 2022-10-08 10:12:57] ...trap/engines/engine.py.229: process: 0.77891 | load: 0.00293 [I 2022-10-08 10:12:57] ...trap/engines/engine.py.230: loss: 0.39045 [I 2022-10-08 10:12:57] ...optimizers/trijoint.py.151: total_norm: 4.354039 [I 2022-10-08 10:13:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 1930/2304 [I 2022-10-08 10:13:04] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:04:47 [I 2022-10-08 10:13:04] ...trap/engines/engine.py.229: process: 0.76449 | load: 0.00292 [I 2022-10-08 10:13:04] ...trap/engines/engine.py.230: loss: 0.29368 [I 2022-10-08 10:13:04] ...optimizers/trijoint.py.151: total_norm: 2.780620 [I 2022-10-08 10:13:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 1940/2304 [I 2022-10-08 10:13:12] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:04:41 [I 2022-10-08 10:13:12] ...trap/engines/engine.py.229: process: 0.77722 | load: 0.00292 [I 2022-10-08 10:13:12] ...trap/engines/engine.py.230: loss: 0.43741 [I 2022-10-08 10:13:12] ...optimizers/trijoint.py.151: total_norm: 4.583695 [I 2022-10-08 10:13:20] ...trap/engines/engine.py.224: train: epoch 0 | batch 1950/2304 [I 2022-10-08 10:13:20] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:04:34 [I 2022-10-08 10:13:20] ...trap/engines/engine.py.229: process: 0.78074 | load: 0.00295 [I 2022-10-08 10:13:20] ...trap/engines/engine.py.230: loss: 0.33247 [I 2022-10-08 10:13:20] ...optimizers/trijoint.py.151: total_norm: 3.877881 [I 2022-10-08 10:13:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1960/2304 [I 2022-10-08 10:13:28] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:04:24 [I 2022-10-08 10:13:28] ...trap/engines/engine.py.229: process: 0.74978 | load: 0.00294 [I 2022-10-08 10:13:28] ...trap/engines/engine.py.230: loss: 0.33798 [I 2022-10-08 10:13:28] ...optimizers/trijoint.py.151: total_norm: 2.932323 [I 2022-10-08 10:13:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 1970/2304 [I 2022-10-08 10:13:35] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:04:16 [I 2022-10-08 10:13:35] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00299 [I 2022-10-08 10:13:35] ...trap/engines/engine.py.230: loss: 0.32457 [I 2022-10-08 10:13:35] ...optimizers/trijoint.py.151: total_norm: 3.500933 [I 2022-10-08 10:13:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1980/2304 [I 2022-10-08 10:13:43] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:04:11 [I 2022-10-08 10:13:43] ...trap/engines/engine.py.229: process: 0.79180 | load: 0.00294 [I 2022-10-08 10:13:43] ...trap/engines/engine.py.230: loss: 0.23576 [I 2022-10-08 10:13:43] ...optimizers/trijoint.py.151: total_norm: 2.743170 [I 2022-10-08 10:13:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 1990/2304 [I 2022-10-08 10:13:51] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:04:01 [I 2022-10-08 10:13:51] ...trap/engines/engine.py.229: process: 0.77699 | load: 0.00300 [I 2022-10-08 10:13:51] ...trap/engines/engine.py.230: loss: 0.31933 [I 2022-10-08 10:13:51] ...optimizers/trijoint.py.151: total_norm: 2.729761 [I 2022-10-08 10:13:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 2000/2304 [I 2022-10-08 10:13:58] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:03:54 [I 2022-10-08 10:13:58] ...trap/engines/engine.py.229: process: 0.76255 | load: 0.00296 [I 2022-10-08 10:13:58] ...trap/engines/engine.py.230: loss: 0.31288 [I 2022-10-08 10:13:58] ...optimizers/trijoint.py.151: total_norm: 3.578893 [I 2022-10-08 10:14:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 2010/2304 [I 2022-10-08 10:14:06] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:03:46 [I 2022-10-08 10:14:06] ...trap/engines/engine.py.229: process: 0.76815 | load: 0.00294 [I 2022-10-08 10:14:06] ...trap/engines/engine.py.230: loss: 0.36930 [I 2022-10-08 10:14:06] ...optimizers/trijoint.py.151: total_norm: 3.828518 [I 2022-10-08 10:14:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 2020/2304 [I 2022-10-08 10:14:14] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:03:39 [I 2022-10-08 10:14:14] ...trap/engines/engine.py.229: process: 0.77209 | load: 0.00297 [I 2022-10-08 10:14:14] ...trap/engines/engine.py.230: loss: 0.29684 [I 2022-10-08 10:14:14] ...optimizers/trijoint.py.151: total_norm: 3.214138 [I 2022-10-08 10:14:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 2030/2304 [I 2022-10-08 10:14:22] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:03:31 [I 2022-10-08 10:14:22] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00297 [I 2022-10-08 10:14:22] ...trap/engines/engine.py.230: loss: 0.29931 [I 2022-10-08 10:14:22] ...optimizers/trijoint.py.151: total_norm: 3.621768 [I 2022-10-08 10:14:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 2040/2304 [I 2022-10-08 10:14:29] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:03:23 [I 2022-10-08 10:14:29] ...trap/engines/engine.py.229: process: 0.77729 | load: 0.00295 [I 2022-10-08 10:14:29] ...trap/engines/engine.py.230: loss: 0.32128 [I 2022-10-08 10:14:29] ...optimizers/trijoint.py.151: total_norm: 3.614037 [I 2022-10-08 10:14:37] ...trap/engines/engine.py.224: train: epoch 0 | batch 2050/2304 [I 2022-10-08 10:14:37] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:03:15 [I 2022-10-08 10:14:37] ...trap/engines/engine.py.229: process: 0.76193 | load: 0.00301 [I 2022-10-08 10:14:37] ...trap/engines/engine.py.230: loss: 0.36042 [I 2022-10-08 10:14:37] ...optimizers/trijoint.py.151: total_norm: 3.719482 [I 2022-10-08 10:14:45] ...trap/engines/engine.py.224: train: epoch 0 | batch 2060/2304 [I 2022-10-08 10:14:45] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:03:08 [I 2022-10-08 10:14:45] ...trap/engines/engine.py.229: process: 0.78347 | load: 0.00302 [I 2022-10-08 10:14:45] ...trap/engines/engine.py.230: loss: 0.33120 [I 2022-10-08 10:14:45] ...optimizers/trijoint.py.151: total_norm: 4.007756 [I 2022-10-08 10:14:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 2070/2304 [I 2022-10-08 10:14:52] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:02:59 [I 2022-10-08 10:14:52] ...trap/engines/engine.py.229: process: 0.76012 | load: 0.00294 [I 2022-10-08 10:14:52] ...trap/engines/engine.py.230: loss: 0.25855 [I 2022-10-08 10:14:52] ...optimizers/trijoint.py.151: total_norm: 2.706890 [I 2022-10-08 10:15:00] ...trap/engines/engine.py.224: train: epoch 0 | batch 2080/2304 [I 2022-10-08 10:15:00] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:02:51 [I 2022-10-08 10:15:00] ...trap/engines/engine.py.229: process: 0.75328 | load: 0.00294 [I 2022-10-08 10:15:00] ...trap/engines/engine.py.230: loss: 0.20263 [I 2022-10-08 10:15:00] ...optimizers/trijoint.py.151: total_norm: 2.664605 [I 2022-10-08 10:15:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 2090/2304 [I 2022-10-08 10:15:08] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:02:44 [I 2022-10-08 10:15:08] ...trap/engines/engine.py.229: process: 0.76705 | load: 0.00294 [I 2022-10-08 10:15:08] ...trap/engines/engine.py.230: loss: 0.24144 [I 2022-10-08 10:15:08] ...optimizers/trijoint.py.151: total_norm: 3.209842 [I 2022-10-08 10:15:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 2100/2304 [I 2022-10-08 10:15:16] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:02:38 [I 2022-10-08 10:15:16] ...trap/engines/engine.py.229: process: 0.78168 | load: 0.00294 [I 2022-10-08 10:15:16] ...trap/engines/engine.py.230: loss: 0.25543 [I 2022-10-08 10:15:16] ...optimizers/trijoint.py.151: total_norm: 2.516937 [I 2022-10-08 10:15:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 2110/2304 [I 2022-10-08 10:15:23] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:02:28 [I 2022-10-08 10:15:23] ...trap/engines/engine.py.229: process: 0.76283 | load: 0.00295 [I 2022-10-08 10:15:23] ...trap/engines/engine.py.230: loss: 0.39419 [I 2022-10-08 10:15:23] ...optimizers/trijoint.py.151: total_norm: 4.329075 [I 2022-10-08 10:15:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 2120/2304 [I 2022-10-08 10:15:31] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:02:21 [I 2022-10-08 10:15:31] ...trap/engines/engine.py.229: process: 0.77024 | load: 0.00294 [I 2022-10-08 10:15:31] ...trap/engines/engine.py.230: loss: 0.30620 [I 2022-10-08 10:15:31] ...optimizers/trijoint.py.151: total_norm: 2.988231 [I 2022-10-08 10:15:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 2130/2304 [I 2022-10-08 10:15:39] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:02:12 [I 2022-10-08 10:15:39] ...trap/engines/engine.py.229: process: 0.75777 | load: 0.00295 [I 2022-10-08 10:15:39] ...trap/engines/engine.py.230: loss: 0.28268 [I 2022-10-08 10:15:39] ...optimizers/trijoint.py.151: total_norm: 3.119411 [I 2022-10-08 10:15:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 2140/2304 [I 2022-10-08 10:15:46] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:02:05 [I 2022-10-08 10:15:46] ...trap/engines/engine.py.229: process: 0.76972 | load: 0.00296 [I 2022-10-08 10:15:46] ...trap/engines/engine.py.230: loss: 0.28957 [I 2022-10-08 10:15:46] ...optimizers/trijoint.py.151: total_norm: 3.944265 [I 2022-10-08 10:15:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 2150/2304 [I 2022-10-08 10:15:54] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:01:58 [I 2022-10-08 10:15:54] ...trap/engines/engine.py.229: process: 0.77211 | load: 0.00295 [I 2022-10-08 10:15:54] ...trap/engines/engine.py.230: loss: 0.28451 [I 2022-10-08 10:15:54] ...optimizers/trijoint.py.151: total_norm: 3.195586 [I 2022-10-08 10:16:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 2160/2304 [I 2022-10-08 10:16:02] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:01:50 [I 2022-10-08 10:16:02] ...trap/engines/engine.py.229: process: 0.76981 | load: 0.00294 [I 2022-10-08 10:16:02] ...trap/engines/engine.py.230: loss: 0.24743 [I 2022-10-08 10:16:02] ...optimizers/trijoint.py.151: total_norm: 3.116599 [I 2022-10-08 10:16:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 2170/2304 [I 2022-10-08 10:16:09] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:01:43 [I 2022-10-08 10:16:09] ...trap/engines/engine.py.229: process: 0.75853 | load: 0.00293 [I 2022-10-08 10:16:09] ...trap/engines/engine.py.230: loss: 0.35123 [I 2022-10-08 10:16:09] ...optimizers/trijoint.py.151: total_norm: 3.730060 [I 2022-10-08 10:16:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 2180/2304 [I 2022-10-08 10:16:17] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:01:34 [I 2022-10-08 10:16:17] ...trap/engines/engine.py.229: process: 0.76124 | load: 0.00296 [I 2022-10-08 10:16:17] ...trap/engines/engine.py.230: loss: 0.18056 [I 2022-10-08 10:16:17] ...optimizers/trijoint.py.151: total_norm: 2.796315 [I 2022-10-08 10:16:25] ...trap/engines/engine.py.224: train: epoch 0 | batch 2190/2304 [I 2022-10-08 10:16:25] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:01:27 [I 2022-10-08 10:16:25] ...trap/engines/engine.py.229: process: 0.77936 | load: 0.00294 [I 2022-10-08 10:16:25] ...trap/engines/engine.py.230: loss: 0.29989 [I 2022-10-08 10:16:25] ...optimizers/trijoint.py.151: total_norm: 3.118837 [I 2022-10-08 10:16:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 2200/2304 [I 2022-10-08 10:16:32] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:01:20 [I 2022-10-08 10:16:32] ...trap/engines/engine.py.229: process: 0.77030 | load: 0.00294 [I 2022-10-08 10:16:32] ...trap/engines/engine.py.230: loss: 0.37300 [I 2022-10-08 10:16:32] ...optimizers/trijoint.py.151: total_norm: 3.608846 [I 2022-10-08 10:16:40] ...trap/engines/engine.py.224: train: epoch 0 | batch 2210/2304 [I 2022-10-08 10:16:40] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:01:12 [I 2022-10-08 10:16:40] ...trap/engines/engine.py.229: process: 0.76531 | load: 0.00295 [I 2022-10-08 10:16:40] ...trap/engines/engine.py.230: loss: 0.39002 [I 2022-10-08 10:16:40] ...optimizers/trijoint.py.151: total_norm: 4.058537 [I 2022-10-08 10:16:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 2220/2304 [I 2022-10-08 10:16:48] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:01:04 [I 2022-10-08 10:16:48] ...trap/engines/engine.py.229: process: 0.75978 | load: 0.00294 [I 2022-10-08 10:16:48] ...trap/engines/engine.py.230: loss: 0.38278 [I 2022-10-08 10:16:48] ...optimizers/trijoint.py.151: total_norm: 3.821580 [I 2022-10-08 10:16:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 2230/2304 [I 2022-10-08 10:16:56] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:00:56 [I 2022-10-08 10:16:56] ...trap/engines/engine.py.229: process: 0.76927 | load: 0.00293 [I 2022-10-08 10:16:56] ...trap/engines/engine.py.230: loss: 0.38433 [I 2022-10-08 10:16:56] ...optimizers/trijoint.py.151: total_norm: 2.925704 [I 2022-10-08 10:17:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 2240/2304 [I 2022-10-08 10:17:03] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:00:49 [I 2022-10-08 10:17:03] ...trap/engines/engine.py.229: process: 0.76238 | load: 0.00293 [I 2022-10-08 10:17:03] ...trap/engines/engine.py.230: loss: 0.28590 [I 2022-10-08 10:17:03] ...optimizers/trijoint.py.151: total_norm: 2.980029 [I 2022-10-08 10:17:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 2250/2304 [I 2022-10-08 10:17:11] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:00:41 [I 2022-10-08 10:17:11] ...trap/engines/engine.py.229: process: 0.77699 | load: 0.00294 [I 2022-10-08 10:17:11] ...trap/engines/engine.py.230: loss: 0.28889 [I 2022-10-08 10:17:11] ...optimizers/trijoint.py.151: total_norm: 3.122552 [I 2022-10-08 10:17:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 2260/2304 [I 2022-10-08 10:17:19] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:00:33 [I 2022-10-08 10:17:19] ...trap/engines/engine.py.229: process: 0.78233 | load: 0.00294 [I 2022-10-08 10:17:19] ...trap/engines/engine.py.230: loss: 0.36643 [I 2022-10-08 10:17:19] ...optimizers/trijoint.py.151: total_norm: 3.875395 [I 2022-10-08 10:17:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 2270/2304 [I 2022-10-08 10:17:26] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:00:25 [I 2022-10-08 10:17:26] ...trap/engines/engine.py.229: process: 0.75780 | load: 0.00295 [I 2022-10-08 10:17:26] ...trap/engines/engine.py.230: loss: 0.30706 [I 2022-10-08 10:17:26] ...optimizers/trijoint.py.151: total_norm: 3.143955 [I 2022-10-08 10:17:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 2280/2304 [I 2022-10-08 10:17:34] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:00:18 [I 2022-10-08 10:17:34] ...trap/engines/engine.py.229: process: 0.77537 | load: 0.00293 [I 2022-10-08 10:17:34] ...trap/engines/engine.py.230: loss: 0.42301 [I 2022-10-08 10:17:34] ...optimizers/trijoint.py.151: total_norm: 3.785534 [I 2022-10-08 10:17:42] ...trap/engines/engine.py.224: train: epoch 0 | batch 2290/2304 [I 2022-10-08 10:17:42] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:00:10 [I 2022-10-08 10:17:42] ...trap/engines/engine.py.229: process: 0.77285 | load: 0.00291 [I 2022-10-08 10:17:42] ...trap/engines/engine.py.230: loss: 0.31375 [I 2022-10-08 10:17:42] ...optimizers/trijoint.py.151: total_norm: 3.392708 [I 2022-10-08 10:17:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 2300/2304 [I 2022-10-08 10:17:49] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:00:03 [I 2022-10-08 10:17:49] ...trap/engines/engine.py.229: process: 0.75691 | load: 0.00012 [I 2022-10-08 10:17:49] ...trap/engines/engine.py.230: loss: 0.33445 [I 2022-10-08 10:17:49] ...optimizers/trijoint.py.151: total_norm: 3.281989 [I 2022-10-08 10:17:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 2304/2304 [I 2022-10-08 10:17:52] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:00:00 [I 2022-10-08 10:17:52] ...trap/engines/engine.py.229: process: 0.76510 | load: 0.00011 [I 2022-10-08 10:17:52] ...trap/engines/engine.py.230: loss: 0.38544 [I 2022-10-08 10:17:52] ...optimizers/trijoint.py.151: total_norm: 3.315595 [S 2022-10-08 10:17:54] ...trap/engines/engine.py.129: train_epoch.epoch: 0 [S 2022-10-08 10:17:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.4081830799385679 [I 2022-10-08 10:17:54] ...s/criterions/triplet.py.84: margin = 0.055 [I 2022-10-08 10:17:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 0 [I 2022-10-08 10:17:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.med_im2recipe_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_1_im2recipe_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_5_im2recipe_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_10_im2recipe_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.med_recipe2im_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_1_recipe2im_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_5_recipe2im_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_10_recipe2im_mean' not in 'logs.json' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 10:17:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 10:17:58] ...trap/engines/engine.py.321: eval: epoch 0 | batch 0/510 [I 2022-10-08 10:17:58] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:58 [I 2022-10-08 10:17:58] ...trap/engines/engine.py.326: process: 4.58574 | load: 3.43926 [I 2022-10-08 10:18:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 10:18:04] ...trap/engines/engine.py.321: eval: epoch 0 | batch 10/510 [I 2022-10-08 10:18:04] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:31 [I 2022-10-08 10:18:04] ...trap/engines/engine.py.326: process: 0.59039 | load: 0.00021 [I 2022-10-08 10:18:09] ...trap/engines/engine.py.321: eval: epoch 0 | batch 20/510 [I 2022-10-08 10:18:09] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:50 [I 2022-10-08 10:18:09] ...trap/engines/engine.py.326: process: 0.50037 | load: 0.00025 [I 2022-10-08 10:18:16] ...trap/engines/engine.py.321: eval: epoch 0 | batch 30/510 [I 2022-10-08 10:18:16] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:29 [I 2022-10-08 10:18:16] ...trap/engines/engine.py.326: process: 0.77433 | load: 0.00017 [I 2022-10-08 10:18:22] ...trap/engines/engine.py.321: eval: epoch 0 | batch 40/510 [I 2022-10-08 10:18:22] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:04:57 [I 2022-10-08 10:18:22] ...trap/engines/engine.py.326: process: 0.55352 | load: 0.00017 [I 2022-10-08 10:18:27] ...trap/engines/engine.py.321: eval: epoch 0 | batch 50/510 [I 2022-10-08 10:18:27] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:43 [I 2022-10-08 10:18:27] ...trap/engines/engine.py.326: process: 0.49337 | load: 0.00019 [I 2022-10-08 10:18:31] ...trap/engines/engine.py.321: eval: epoch 0 | batch 60/510 [I 2022-10-08 10:18:31] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:29 [I 2022-10-08 10:18:31] ...trap/engines/engine.py.326: process: 0.43338 | load: 0.00017 [I 2022-10-08 10:18:37] ...trap/engines/engine.py.321: eval: epoch 0 | batch 70/510 [I 2022-10-08 10:18:37] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:57 [I 2022-10-08 10:18:37] ...trap/engines/engine.py.326: process: 0.50464 | load: 0.00018 [I 2022-10-08 10:18:43] ...trap/engines/engine.py.321: eval: epoch 0 | batch 80/510 [I 2022-10-08 10:18:43] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:04:07 [I 2022-10-08 10:18:43] ...trap/engines/engine.py.326: process: 0.54861 | load: 0.00018 [I 2022-10-08 10:18:47] ...trap/engines/engine.py.321: eval: epoch 0 | batch 90/510 [I 2022-10-08 10:18:47] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:04 [I 2022-10-08 10:18:47] ...trap/engines/engine.py.326: process: 0.41159 | load: 0.00018 [I 2022-10-08 10:18:51] ...trap/engines/engine.py.321: eval: epoch 0 | batch 100/510 [I 2022-10-08 10:18:51] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:52 [I 2022-10-08 10:18:51] ...trap/engines/engine.py.326: process: 0.31746 | load: 0.00018 [I 2022-10-08 10:18:54] ...trap/engines/engine.py.321: eval: epoch 0 | batch 110/510 [I 2022-10-08 10:18:54] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:11 [I 2022-10-08 10:18:54] ...trap/engines/engine.py.326: process: 0.31833 | load: 0.00019 [I 2022-10-08 10:18:58] ...trap/engines/engine.py.321: eval: epoch 0 | batch 120/510 [I 2022-10-08 10:18:58] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-08 10:18:58] ...trap/engines/engine.py.326: process: 0.31437 | load: 0.00019 [I 2022-10-08 10:19:01] ...trap/engines/engine.py.321: eval: epoch 0 | batch 130/510 [I 2022-10-08 10:19:01] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:01 [I 2022-10-08 10:19:01] ...trap/engines/engine.py.326: process: 0.32163 | load: 0.00020 [I 2022-10-08 10:19:04] ...trap/engines/engine.py.321: eval: epoch 0 | batch 140/510 [I 2022-10-08 10:19:04] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:58 [I 2022-10-08 10:19:04] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00019 [I 2022-10-08 10:19:07] ...trap/engines/engine.py.321: eval: epoch 0 | batch 150/510 [I 2022-10-08 10:19:07] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:55 [I 2022-10-08 10:19:07] ...trap/engines/engine.py.326: process: 0.32073 | load: 0.00019 [I 2022-10-08 10:19:10] ...trap/engines/engine.py.321: eval: epoch 0 | batch 160/510 [I 2022-10-08 10:19:10] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:51 [I 2022-10-08 10:19:10] ...trap/engines/engine.py.326: process: 0.31532 | load: 0.00019 [I 2022-10-08 10:19:13] ...trap/engines/engine.py.321: eval: epoch 0 | batch 170/510 [I 2022-10-08 10:19:13] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:47 [I 2022-10-08 10:19:13] ...trap/engines/engine.py.326: process: 0.32050 | load: 0.00019 [I 2022-10-08 10:19:17] ...trap/engines/engine.py.321: eval: epoch 0 | batch 180/510 [I 2022-10-08 10:19:17] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:46 [I 2022-10-08 10:19:17] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00019 [I 2022-10-08 10:19:20] ...trap/engines/engine.py.321: eval: epoch 0 | batch 190/510 [I 2022-10-08 10:19:20] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:41 [I 2022-10-08 10:19:20] ...trap/engines/engine.py.326: process: 0.31988 | load: 0.00019 [I 2022-10-08 10:19:23] ...trap/engines/engine.py.321: eval: epoch 0 | batch 200/510 [I 2022-10-08 10:19:23] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:38 [I 2022-10-08 10:19:23] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00020 [I 2022-10-08 10:19:26] ...trap/engines/engine.py.321: eval: epoch 0 | batch 210/510 [I 2022-10-08 10:19:26] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:35 [I 2022-10-08 10:19:26] ...trap/engines/engine.py.326: process: 0.31562 | load: 0.00018 [I 2022-10-08 10:19:29] ...trap/engines/engine.py.321: eval: epoch 0 | batch 220/510 [I 2022-10-08 10:19:29] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:31 [I 2022-10-08 10:19:29] ...trap/engines/engine.py.326: process: 0.31880 | load: 0.00020 [I 2022-10-08 10:19:33] ...trap/engines/engine.py.321: eval: epoch 0 | batch 230/510 [I 2022-10-08 10:19:33] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-08 10:19:33] ...trap/engines/engine.py.326: process: 0.31362 | load: 0.00018 [I 2022-10-08 10:19:36] ...trap/engines/engine.py.321: eval: epoch 0 | batch 240/510 [I 2022-10-08 10:19:36] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:25 [I 2022-10-08 10:19:36] ...trap/engines/engine.py.326: process: 0.31496 | load: 0.00018 [I 2022-10-08 10:19:39] ...trap/engines/engine.py.321: eval: epoch 0 | batch 250/510 [I 2022-10-08 10:19:39] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:22 [I 2022-10-08 10:19:39] ...trap/engines/engine.py.326: process: 0.31374 | load: 0.00019 [I 2022-10-08 10:19:42] ...trap/engines/engine.py.321: eval: epoch 0 | batch 260/510 [I 2022-10-08 10:19:42] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:19 [I 2022-10-08 10:19:42] ...trap/engines/engine.py.326: process: 0.32139 | load: 0.00019 [I 2022-10-08 10:19:45] ...trap/engines/engine.py.321: eval: epoch 0 | batch 270/510 [I 2022-10-08 10:19:45] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-08 10:19:45] ...trap/engines/engine.py.326: process: 0.31599 | load: 0.00019 [I 2022-10-08 10:19:49] ...trap/engines/engine.py.321: eval: epoch 0 | batch 280/510 [I 2022-10-08 10:19:49] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:12 [I 2022-10-08 10:19:49] ...trap/engines/engine.py.326: process: 0.31587 | load: 0.00019 [I 2022-10-08 10:19:52] ...trap/engines/engine.py.321: eval: epoch 0 | batch 290/510 [I 2022-10-08 10:19:52] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:09 [I 2022-10-08 10:19:52] ...trap/engines/engine.py.326: process: 0.31130 | load: 0.00019 [I 2022-10-08 10:19:55] ...trap/engines/engine.py.321: eval: epoch 0 | batch 300/510 [I 2022-10-08 10:19:55] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:06 [I 2022-10-08 10:19:55] ...trap/engines/engine.py.326: process: 0.31419 | load: 0.00019 [I 2022-10-08 10:19:58] ...trap/engines/engine.py.321: eval: epoch 0 | batch 310/510 [I 2022-10-08 10:19:58] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-08 10:19:58] ...trap/engines/engine.py.326: process: 0.32046 | load: 0.00019 [I 2022-10-08 10:20:01] ...trap/engines/engine.py.321: eval: epoch 0 | batch 320/510 [I 2022-10-08 10:20:01] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-08 10:20:01] ...trap/engines/engine.py.326: process: 0.32512 | load: 0.00018 [I 2022-10-08 10:20:04] ...trap/engines/engine.py.321: eval: epoch 0 | batch 330/510 [I 2022-10-08 10:20:04] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-08 10:20:04] ...trap/engines/engine.py.326: process: 0.32310 | load: 0.00018 [I 2022-10-08 10:20:08] ...trap/engines/engine.py.321: eval: epoch 0 | batch 340/510 [I 2022-10-08 10:20:08] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:53 [I 2022-10-08 10:20:08] ...trap/engines/engine.py.326: process: 0.30890 | load: 0.00018 [I 2022-10-08 10:20:11] ...trap/engines/engine.py.321: eval: epoch 0 | batch 350/510 [I 2022-10-08 10:20:11] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:50 [I 2022-10-08 10:20:11] ...trap/engines/engine.py.326: process: 0.32323 | load: 0.00019 [I 2022-10-08 10:20:14] ...trap/engines/engine.py.321: eval: epoch 0 | batch 360/510 [I 2022-10-08 10:20:14] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:47 [I 2022-10-08 10:20:14] ...trap/engines/engine.py.326: process: 0.31683 | load: 0.00019 [I 2022-10-08 10:20:17] ...trap/engines/engine.py.321: eval: epoch 0 | batch 370/510 [I 2022-10-08 10:20:17] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-08 10:20:17] ...trap/engines/engine.py.326: process: 0.31559 | load: 0.00019 [I 2022-10-08 10:20:20] ...trap/engines/engine.py.321: eval: epoch 0 | batch 380/510 [I 2022-10-08 10:20:20] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-08 10:20:20] ...trap/engines/engine.py.326: process: 0.31846 | load: 0.00019 [I 2022-10-08 10:20:23] ...trap/engines/engine.py.321: eval: epoch 0 | batch 390/510 [I 2022-10-08 10:20:23] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:37 [I 2022-10-08 10:20:23] ...trap/engines/engine.py.326: process: 0.31067 | load: 0.00018 [I 2022-10-08 10:20:27] ...trap/engines/engine.py.321: eval: epoch 0 | batch 400/510 [I 2022-10-08 10:20:27] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:34 [I 2022-10-08 10:20:27] ...trap/engines/engine.py.326: process: 0.31162 | load: 0.00019 [I 2022-10-08 10:20:30] ...trap/engines/engine.py.321: eval: epoch 0 | batch 410/510 [I 2022-10-08 10:20:30] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:31 [I 2022-10-08 10:20:30] ...trap/engines/engine.py.326: process: 0.31887 | load: 0.00020 [I 2022-10-08 10:20:33] ...trap/engines/engine.py.321: eval: epoch 0 | batch 420/510 [I 2022-10-08 10:20:33] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:28 [I 2022-10-08 10:20:33] ...trap/engines/engine.py.326: process: 0.31306 | load: 0.00019 [I 2022-10-08 10:20:36] ...trap/engines/engine.py.321: eval: epoch 0 | batch 430/510 [I 2022-10-08 10:20:36] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-08 10:20:36] ...trap/engines/engine.py.326: process: 0.31770 | load: 0.00019 [I 2022-10-08 10:20:39] ...trap/engines/engine.py.321: eval: epoch 0 | batch 440/510 [I 2022-10-08 10:20:39] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-08 10:20:39] ...trap/engines/engine.py.326: process: 0.31697 | load: 0.00018 [I 2022-10-08 10:20:43] ...trap/engines/engine.py.321: eval: epoch 0 | batch 450/510 [I 2022-10-08 10:20:43] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-08 10:20:43] ...trap/engines/engine.py.326: process: 0.31705 | load: 0.00019 [I 2022-10-08 10:20:46] ...trap/engines/engine.py.321: eval: epoch 0 | batch 460/510 [I 2022-10-08 10:20:46] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-08 10:20:46] ...trap/engines/engine.py.326: process: 0.31554 | load: 0.00018 [I 2022-10-08 10:20:49] ...trap/engines/engine.py.321: eval: epoch 0 | batch 470/510 [I 2022-10-08 10:20:49] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-08 10:20:49] ...trap/engines/engine.py.326: process: 0.32736 | load: 0.00019 [I 2022-10-08 10:20:52] ...trap/engines/engine.py.321: eval: epoch 0 | batch 480/510 [I 2022-10-08 10:20:52] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-08 10:20:52] ...trap/engines/engine.py.326: process: 0.31812 | load: 0.00018 [I 2022-10-08 10:20:55] ...trap/engines/engine.py.321: eval: epoch 0 | batch 490/510 [I 2022-10-08 10:20:55] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-08 10:20:55] ...trap/engines/engine.py.326: process: 0.31114 | load: 0.00019 [I 2022-10-08 10:20:58] ...trap/engines/engine.py.321: eval: epoch 0 | batch 500/510 [I 2022-10-08 10:20:58] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-08 10:20:58] ...trap/engines/engine.py.326: process: 0.31546 | load: 0.00018 [I 2022-10-08 10:21:02] ...trap/engines/engine.py.321: eval: epoch 0 | batch 510/510 [I 2022-10-08 10:21:02] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-08 10:21:02] ...trap/engines/engine.py.326: process: 0.31173 | load: 0.00010 [S 2022-10-08 10:21:02] ...trap/engines/engine.py.132: eval_epoch.epoch: 0 [I 2022-10-08 10:21:02] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.393800 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.725400 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.827400 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.439200 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.754900 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.843900 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011746 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008902 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.012035 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012734 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.014258 [S 2022-10-08 10:21:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.014815 [I 2022-10-08 10:21:32] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 10:21:32] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 10:21:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 10:21:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 10:21:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 10:21:32] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 10:21:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 10:21:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 10:21:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 1 [I 2022-10-08 10:21:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 10:21:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 0/2304 [I 2022-10-08 10:21:43] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:05:41 [I 2022-10-08 10:21:43] ...trap/engines/engine.py.229: process: 3.27303 | load: 2.16733 [I 2022-10-08 10:21:43] ...trap/engines/engine.py.230: loss: 0.32351 [I 2022-10-08 10:21:43] ...optimizers/trijoint.py.151: total_norm: 3.309561 [I 2022-10-08 10:21:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 10/2304 [I 2022-10-08 10:21:51] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:47 [I 2022-10-08 10:21:51] ...trap/engines/engine.py.229: process: 0.76735 | load: 0.00631 [I 2022-10-08 10:21:51] ...trap/engines/engine.py.230: loss: 0.32844 [I 2022-10-08 10:21:51] ...optimizers/trijoint.py.151: total_norm: 3.909953 [I 2022-10-08 10:21:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 20/2304 [I 2022-10-08 10:21:58] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:42 [I 2022-10-08 10:21:58] ...trap/engines/engine.py.229: process: 0.78284 | load: 0.00390 [I 2022-10-08 10:21:58] ...trap/engines/engine.py.230: loss: 0.26481 [I 2022-10-08 10:21:58] ...optimizers/trijoint.py.151: total_norm: 2.341547 [I 2022-10-08 10:22:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 30/2304 [I 2022-10-08 10:22:06] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:11 [I 2022-10-08 10:22:06] ...trap/engines/engine.py.229: process: 0.76220 | load: 0.00400 [I 2022-10-08 10:22:06] ...trap/engines/engine.py.230: loss: 0.39734 [I 2022-10-08 10:22:06] ...optimizers/trijoint.py.151: total_norm: 3.346632 [I 2022-10-08 10:22:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 40/2304 [I 2022-10-08 10:22:14] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:12 [I 2022-10-08 10:22:14] ...trap/engines/engine.py.229: process: 0.77858 | load: 0.00385 [I 2022-10-08 10:22:14] ...trap/engines/engine.py.230: loss: 0.28805 [I 2022-10-08 10:22:14] ...optimizers/trijoint.py.151: total_norm: 2.773293 [I 2022-10-08 10:22:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 50/2304 [I 2022-10-08 10:22:22] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:00 [I 2022-10-08 10:22:22] ...trap/engines/engine.py.229: process: 0.76789 | load: 0.00390 [I 2022-10-08 10:22:22] ...trap/engines/engine.py.230: loss: 0.31164 [I 2022-10-08 10:22:22] ...optimizers/trijoint.py.151: total_norm: 3.477290 [I 2022-10-08 10:22:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 60/2304 [I 2022-10-08 10:22:29] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:28:54 [I 2022-10-08 10:22:29] ...trap/engines/engine.py.229: process: 0.76778 | load: 0.00321 [I 2022-10-08 10:22:29] ...trap/engines/engine.py.230: loss: 0.28243 [I 2022-10-08 10:22:29] ...optimizers/trijoint.py.151: total_norm: 2.459738 [I 2022-10-08 10:22:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 70/2304 [I 2022-10-08 10:22:37] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:39 [I 2022-10-08 10:22:37] ...trap/engines/engine.py.229: process: 0.76673 | load: 0.00315 [I 2022-10-08 10:22:37] ...trap/engines/engine.py.230: loss: 0.34978 [I 2022-10-08 10:22:37] ...optimizers/trijoint.py.151: total_norm: 3.520514 [I 2022-10-08 10:22:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 80/2304 [I 2022-10-08 10:22:45] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:25 [I 2022-10-08 10:22:45] ...trap/engines/engine.py.229: process: 0.76238 | load: 0.00293 [I 2022-10-08 10:22:45] ...trap/engines/engine.py.230: loss: 0.35799 [I 2022-10-08 10:22:45] ...optimizers/trijoint.py.151: total_norm: 4.484284 [I 2022-10-08 10:22:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 90/2304 [I 2022-10-08 10:22:52] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:28:18 [I 2022-10-08 10:22:52] ...trap/engines/engine.py.229: process: 0.77270 | load: 0.00321 [I 2022-10-08 10:22:52] ...trap/engines/engine.py.230: loss: 0.30790 [I 2022-10-08 10:22:52] ...optimizers/trijoint.py.151: total_norm: 3.228985 [I 2022-10-08 10:23:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 100/2304 [I 2022-10-08 10:23:00] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:23 [I 2022-10-08 10:23:00] ...trap/engines/engine.py.229: process: 0.78107 | load: 0.00342 [I 2022-10-08 10:23:00] ...trap/engines/engine.py.230: loss: 0.30913 [I 2022-10-08 10:23:00] ...optimizers/trijoint.py.151: total_norm: 3.187296 [I 2022-10-08 10:23:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 110/2304 [I 2022-10-08 10:23:08] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:00 [I 2022-10-08 10:23:08] ...trap/engines/engine.py.229: process: 0.75845 | load: 0.00314 [I 2022-10-08 10:23:08] ...trap/engines/engine.py.230: loss: 0.27921 [I 2022-10-08 10:23:08] ...optimizers/trijoint.py.151: total_norm: 3.124875 [I 2022-10-08 10:23:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 120/2304 [I 2022-10-08 10:23:16] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:15 [I 2022-10-08 10:23:16] ...trap/engines/engine.py.229: process: 0.76333 | load: 0.00318 [I 2022-10-08 10:23:16] ...trap/engines/engine.py.230: loss: 0.34882 [I 2022-10-08 10:23:16] ...optimizers/trijoint.py.151: total_norm: 3.495369 [I 2022-10-08 10:23:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 130/2304 [I 2022-10-08 10:23:23] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:28:03 [I 2022-10-08 10:23:23] ...trap/engines/engine.py.229: process: 0.78405 | load: 0.00293 [I 2022-10-08 10:23:23] ...trap/engines/engine.py.230: loss: 0.40497 [I 2022-10-08 10:23:23] ...optimizers/trijoint.py.151: total_norm: 3.398452 [I 2022-10-08 10:23:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 140/2304 [I 2022-10-08 10:23:31] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:27:50 [I 2022-10-08 10:23:31] ...trap/engines/engine.py.229: process: 0.76458 | load: 0.00322 [I 2022-10-08 10:23:31] ...trap/engines/engine.py.230: loss: 0.46933 [I 2022-10-08 10:23:31] ...optimizers/trijoint.py.151: total_norm: 4.564661 [I 2022-10-08 10:23:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 150/2304 [I 2022-10-08 10:23:39] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:27:36 [I 2022-10-08 10:23:39] ...trap/engines/engine.py.229: process: 0.75923 | load: 0.00292 [I 2022-10-08 10:23:39] ...trap/engines/engine.py.230: loss: 0.26672 [I 2022-10-08 10:23:39] ...optimizers/trijoint.py.151: total_norm: 2.589882 [I 2022-10-08 10:23:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 160/2304 [I 2022-10-08 10:23:47] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:27:39 [I 2022-10-08 10:23:47] ...trap/engines/engine.py.229: process: 0.79579 | load: 0.00301 [I 2022-10-08 10:23:47] ...trap/engines/engine.py.230: loss: 0.29642 [I 2022-10-08 10:23:47] ...optimizers/trijoint.py.151: total_norm: 2.821038 [I 2022-10-08 10:23:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 170/2304 [I 2022-10-08 10:23:54] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:27:22 [I 2022-10-08 10:23:54] ...trap/engines/engine.py.229: process: 0.78016 | load: 0.00323 [I 2022-10-08 10:23:54] ...trap/engines/engine.py.230: loss: 0.28821 [I 2022-10-08 10:23:54] ...optimizers/trijoint.py.151: total_norm: 3.445343 [I 2022-10-08 10:24:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 180/2304 [I 2022-10-08 10:24:02] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:27:10 [I 2022-10-08 10:24:02] ...trap/engines/engine.py.229: process: 0.77790 | load: 0.00306 [I 2022-10-08 10:24:02] ...trap/engines/engine.py.230: loss: 0.31054 [I 2022-10-08 10:24:02] ...optimizers/trijoint.py.151: total_norm: 3.455919 [I 2022-10-08 10:24:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 190/2304 [I 2022-10-08 10:24:10] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:27:01 [I 2022-10-08 10:24:10] ...trap/engines/engine.py.229: process: 0.76428 | load: 0.00304 [I 2022-10-08 10:24:10] ...trap/engines/engine.py.230: loss: 0.23391 [I 2022-10-08 10:24:10] ...optimizers/trijoint.py.151: total_norm: 3.574417 [I 2022-10-08 10:24:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 200/2304 [I 2022-10-08 10:24:17] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:27:04 [I 2022-10-08 10:24:17] ...trap/engines/engine.py.229: process: 0.77384 | load: 0.00317 [I 2022-10-08 10:24:17] ...trap/engines/engine.py.230: loss: 0.51372 [I 2022-10-08 10:24:17] ...optimizers/trijoint.py.151: total_norm: 4.018907 [I 2022-10-08 10:24:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 210/2304 [I 2022-10-08 10:24:25] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:26:50 [I 2022-10-08 10:24:25] ...trap/engines/engine.py.229: process: 0.78125 | load: 0.00319 [I 2022-10-08 10:24:25] ...trap/engines/engine.py.230: loss: 0.32298 [I 2022-10-08 10:24:25] ...optimizers/trijoint.py.151: total_norm: 2.582639 [I 2022-10-08 10:24:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 220/2304 [I 2022-10-08 10:24:33] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:26:45 [I 2022-10-08 10:24:33] ...trap/engines/engine.py.229: process: 0.77137 | load: 0.00305 [I 2022-10-08 10:24:33] ...trap/engines/engine.py.230: loss: 0.22921 [I 2022-10-08 10:24:33] ...optimizers/trijoint.py.151: total_norm: 2.835063 [I 2022-10-08 10:24:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 230/2304 [I 2022-10-08 10:24:41] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:26:40 [I 2022-10-08 10:24:41] ...trap/engines/engine.py.229: process: 0.77251 | load: 0.00307 [I 2022-10-08 10:24:41] ...trap/engines/engine.py.230: loss: 0.36056 [I 2022-10-08 10:24:41] ...optimizers/trijoint.py.151: total_norm: 3.413689 [I 2022-10-08 10:24:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 240/2304 [I 2022-10-08 10:24:48] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:26:36 [I 2022-10-08 10:24:48] ...trap/engines/engine.py.229: process: 0.77173 | load: 0.00304 [I 2022-10-08 10:24:48] ...trap/engines/engine.py.230: loss: 0.27448 [I 2022-10-08 10:24:48] ...optimizers/trijoint.py.151: total_norm: 3.150742 [I 2022-10-08 10:24:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 250/2304 [I 2022-10-08 10:24:56] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:26:14 [I 2022-10-08 10:24:56] ...trap/engines/engine.py.229: process: 0.76827 | load: 0.00312 [I 2022-10-08 10:24:56] ...trap/engines/engine.py.230: loss: 0.22907 [I 2022-10-08 10:24:56] ...optimizers/trijoint.py.151: total_norm: 3.030925 [I 2022-10-08 10:25:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 260/2304 [I 2022-10-08 10:25:04] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:26:13 [I 2022-10-08 10:25:04] ...trap/engines/engine.py.229: process: 0.77445 | load: 0.00297 [I 2022-10-08 10:25:04] ...trap/engines/engine.py.230: loss: 0.22379 [I 2022-10-08 10:25:04] ...optimizers/trijoint.py.151: total_norm: 2.324344 [I 2022-10-08 10:25:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 270/2304 [I 2022-10-08 10:25:11] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:25:53 [I 2022-10-08 10:25:11] ...trap/engines/engine.py.229: process: 0.74807 | load: 0.00302 [I 2022-10-08 10:25:11] ...trap/engines/engine.py.230: loss: 0.29074 [I 2022-10-08 10:25:11] ...optimizers/trijoint.py.151: total_norm: 3.502635 [I 2022-10-08 10:25:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 280/2304 [I 2022-10-08 10:25:19] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:26:01 [I 2022-10-08 10:25:19] ...trap/engines/engine.py.229: process: 0.76983 | load: 0.00293 [I 2022-10-08 10:25:19] ...trap/engines/engine.py.230: loss: 0.26802 [I 2022-10-08 10:25:19] ...optimizers/trijoint.py.151: total_norm: 3.231905 [I 2022-10-08 10:25:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 290/2304 [I 2022-10-08 10:25:27] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:25:56 [I 2022-10-08 10:25:27] ...trap/engines/engine.py.229: process: 0.78905 | load: 0.00294 [I 2022-10-08 10:25:27] ...trap/engines/engine.py.230: loss: 0.34592 [I 2022-10-08 10:25:27] ...optimizers/trijoint.py.151: total_norm: 3.126140 [I 2022-10-08 10:25:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 300/2304 [I 2022-10-08 10:25:35] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:25:49 [I 2022-10-08 10:25:35] ...trap/engines/engine.py.229: process: 0.78434 | load: 0.00294 [I 2022-10-08 10:25:35] ...trap/engines/engine.py.230: loss: 0.32745 [I 2022-10-08 10:25:35] ...optimizers/trijoint.py.151: total_norm: 3.411682 [I 2022-10-08 10:25:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 310/2304 [I 2022-10-08 10:25:42] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:25:40 [I 2022-10-08 10:25:42] ...trap/engines/engine.py.229: process: 0.76475 | load: 0.00304 [I 2022-10-08 10:25:42] ...trap/engines/engine.py.230: loss: 0.23878 [I 2022-10-08 10:25:42] ...optimizers/trijoint.py.151: total_norm: 2.234203 [I 2022-10-08 10:25:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 320/2304 [I 2022-10-08 10:25:50] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:25:27 [I 2022-10-08 10:25:50] ...trap/engines/engine.py.229: process: 0.76267 | load: 0.00294 [I 2022-10-08 10:25:50] ...trap/engines/engine.py.230: loss: 0.24509 [I 2022-10-08 10:25:50] ...optimizers/trijoint.py.151: total_norm: 3.516987 [I 2022-10-08 10:25:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 330/2304 [I 2022-10-08 10:25:58] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:25:21 [I 2022-10-08 10:25:58] ...trap/engines/engine.py.229: process: 0.77165 | load: 0.00301 [I 2022-10-08 10:25:58] ...trap/engines/engine.py.230: loss: 0.36957 [I 2022-10-08 10:25:58] ...optimizers/trijoint.py.151: total_norm: 3.539139 [I 2022-10-08 10:26:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 340/2304 [I 2022-10-08 10:26:05] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:25:05 [I 2022-10-08 10:26:05] ...trap/engines/engine.py.229: process: 0.76695 | load: 0.00303 [I 2022-10-08 10:26:05] ...trap/engines/engine.py.230: loss: 0.27850 [I 2022-10-08 10:26:05] ...optimizers/trijoint.py.151: total_norm: 3.575745 [I 2022-10-08 10:26:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 350/2304 [I 2022-10-08 10:26:13] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:24:59 [I 2022-10-08 10:26:13] ...trap/engines/engine.py.229: process: 0.76487 | load: 0.00293 [I 2022-10-08 10:26:13] ...trap/engines/engine.py.230: loss: 0.26663 [I 2022-10-08 10:26:13] ...optimizers/trijoint.py.151: total_norm: 3.163070 [I 2022-10-08 10:26:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 360/2304 [I 2022-10-08 10:26:21] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:24:51 [I 2022-10-08 10:26:21] ...trap/engines/engine.py.229: process: 0.78087 | load: 0.00294 [I 2022-10-08 10:26:21] ...trap/engines/engine.py.230: loss: 0.31815 [I 2022-10-08 10:26:21] ...optimizers/trijoint.py.151: total_norm: 4.160487 [I 2022-10-08 10:26:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 370/2304 [I 2022-10-08 10:26:28] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:24:47 [I 2022-10-08 10:26:28] ...trap/engines/engine.py.229: process: 0.76830 | load: 0.00292 [I 2022-10-08 10:26:28] ...trap/engines/engine.py.230: loss: 0.23980 [I 2022-10-08 10:26:28] ...optimizers/trijoint.py.151: total_norm: 2.942652 [I 2022-10-08 10:26:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 380/2304 [I 2022-10-08 10:26:36] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:24:47 [I 2022-10-08 10:26:36] ...trap/engines/engine.py.229: process: 0.76944 | load: 0.00295 [I 2022-10-08 10:26:36] ...trap/engines/engine.py.230: loss: 0.33209 [I 2022-10-08 10:26:36] ...optimizers/trijoint.py.151: total_norm: 3.835265 [I 2022-10-08 10:26:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 390/2304 [I 2022-10-08 10:26:44] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:24:52 [I 2022-10-08 10:26:44] ...trap/engines/engine.py.229: process: 0.78291 | load: 0.00295 [I 2022-10-08 10:26:44] ...trap/engines/engine.py.230: loss: 0.20698 [I 2022-10-08 10:26:44] ...optimizers/trijoint.py.151: total_norm: 2.409822 [I 2022-10-08 10:26:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 400/2304 [I 2022-10-08 10:26:52] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:24:39 [I 2022-10-08 10:26:52] ...trap/engines/engine.py.229: process: 0.79288 | load: 0.00298 [I 2022-10-08 10:26:52] ...trap/engines/engine.py.230: loss: 0.38875 [I 2022-10-08 10:26:52] ...optimizers/trijoint.py.151: total_norm: 3.125459 [I 2022-10-08 10:26:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 410/2304 [I 2022-10-08 10:26:59] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:24:17 [I 2022-10-08 10:26:59] ...trap/engines/engine.py.229: process: 0.75600 | load: 0.00304 [I 2022-10-08 10:26:59] ...trap/engines/engine.py.230: loss: 0.29043 [I 2022-10-08 10:26:59] ...optimizers/trijoint.py.151: total_norm: 3.194574 [I 2022-10-08 10:27:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 420/2304 [I 2022-10-08 10:27:07] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:24:08 [I 2022-10-08 10:27:07] ...trap/engines/engine.py.229: process: 0.76828 | load: 0.00303 [I 2022-10-08 10:27:07] ...trap/engines/engine.py.230: loss: 0.28293 [I 2022-10-08 10:27:07] ...optimizers/trijoint.py.151: total_norm: 2.787881 [I 2022-10-08 10:27:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 430/2304 [I 2022-10-08 10:27:15] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:23:59 [I 2022-10-08 10:27:15] ...trap/engines/engine.py.229: process: 0.77096 | load: 0.00294 [I 2022-10-08 10:27:15] ...trap/engines/engine.py.230: loss: 0.34685 [I 2022-10-08 10:27:15] ...optimizers/trijoint.py.151: total_norm: 4.060167 [I 2022-10-08 10:27:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 440/2304 [I 2022-10-08 10:27:23] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:23:57 [I 2022-10-08 10:27:23] ...trap/engines/engine.py.229: process: 0.77206 | load: 0.00304 [I 2022-10-08 10:27:23] ...trap/engines/engine.py.230: loss: 0.32278 [I 2022-10-08 10:27:23] ...optimizers/trijoint.py.151: total_norm: 3.768149 [I 2022-10-08 10:27:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 450/2304 [I 2022-10-08 10:27:30] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:23:44 [I 2022-10-08 10:27:30] ...trap/engines/engine.py.229: process: 0.77187 | load: 0.00295 [I 2022-10-08 10:27:30] ...trap/engines/engine.py.230: loss: 0.37468 [I 2022-10-08 10:27:30] ...optimizers/trijoint.py.151: total_norm: 3.992104 [I 2022-10-08 10:27:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 460/2304 [I 2022-10-08 10:27:38] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:23:48 [I 2022-10-08 10:27:38] ...trap/engines/engine.py.229: process: 0.79530 | load: 0.00293 [I 2022-10-08 10:27:38] ...trap/engines/engine.py.230: loss: 0.39186 [I 2022-10-08 10:27:38] ...optimizers/trijoint.py.151: total_norm: 3.733776 [I 2022-10-08 10:27:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 470/2304 [I 2022-10-08 10:27:46] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:23:32 [I 2022-10-08 10:27:46] ...trap/engines/engine.py.229: process: 0.75987 | load: 0.00295 [I 2022-10-08 10:27:46] ...trap/engines/engine.py.230: loss: 0.32871 [I 2022-10-08 10:27:46] ...optimizers/trijoint.py.151: total_norm: 3.512969 [I 2022-10-08 10:27:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 480/2304 [I 2022-10-08 10:27:54] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:23:41 [I 2022-10-08 10:27:54] ...trap/engines/engine.py.229: process: 0.78208 | load: 0.00292 [I 2022-10-08 10:27:54] ...trap/engines/engine.py.230: loss: 0.26489 [I 2022-10-08 10:27:54] ...optimizers/trijoint.py.151: total_norm: 2.781201 [I 2022-10-08 10:28:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 490/2304 [I 2022-10-08 10:28:01] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:23:26 [I 2022-10-08 10:28:01] ...trap/engines/engine.py.229: process: 0.76857 | load: 0.00302 [I 2022-10-08 10:28:01] ...trap/engines/engine.py.230: loss: 0.33187 [I 2022-10-08 10:28:01] ...optimizers/trijoint.py.151: total_norm: 2.912277 [I 2022-10-08 10:28:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 500/2304 [I 2022-10-08 10:28:09] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:23:09 [I 2022-10-08 10:28:09] ...trap/engines/engine.py.229: process: 0.76907 | load: 0.00294 [I 2022-10-08 10:28:09] ...trap/engines/engine.py.230: loss: 0.26435 [I 2022-10-08 10:28:09] ...optimizers/trijoint.py.151: total_norm: 2.818991 [I 2022-10-08 10:28:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 510/2304 [I 2022-10-08 10:28:17] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:23:02 [I 2022-10-08 10:28:17] ...trap/engines/engine.py.229: process: 0.78119 | load: 0.00294 [I 2022-10-08 10:28:17] ...trap/engines/engine.py.230: loss: 0.28183 [I 2022-10-08 10:28:17] ...optimizers/trijoint.py.151: total_norm: 3.292988 [I 2022-10-08 10:28:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 520/2304 [I 2022-10-08 10:28:24] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:22:54 [I 2022-10-08 10:28:24] ...trap/engines/engine.py.229: process: 0.77329 | load: 0.00307 [I 2022-10-08 10:28:24] ...trap/engines/engine.py.230: loss: 0.30820 [I 2022-10-08 10:28:24] ...optimizers/trijoint.py.151: total_norm: 4.455946 [I 2022-10-08 10:28:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 530/2304 [I 2022-10-08 10:28:32] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:22:44 [I 2022-10-08 10:28:32] ...trap/engines/engine.py.229: process: 0.76484 | load: 0.00292 [I 2022-10-08 10:28:32] ...trap/engines/engine.py.230: loss: 0.23456 [I 2022-10-08 10:28:32] ...optimizers/trijoint.py.151: total_norm: 2.941012 [I 2022-10-08 10:28:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 540/2304 [I 2022-10-08 10:28:40] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:22:47 [I 2022-10-08 10:28:40] ...trap/engines/engine.py.229: process: 0.78837 | load: 0.00294 [I 2022-10-08 10:28:40] ...trap/engines/engine.py.230: loss: 0.33294 [I 2022-10-08 10:28:40] ...optimizers/trijoint.py.151: total_norm: 3.595216 [I 2022-10-08 10:28:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 550/2304 [I 2022-10-08 10:28:48] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:22:23 [I 2022-10-08 10:28:48] ...trap/engines/engine.py.229: process: 0.77360 | load: 0.00294 [I 2022-10-08 10:28:48] ...trap/engines/engine.py.230: loss: 0.28171 [I 2022-10-08 10:28:48] ...optimizers/trijoint.py.151: total_norm: 2.998051 [I 2022-10-08 10:28:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 560/2304 [I 2022-10-08 10:28:55] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:22:17 [I 2022-10-08 10:28:55] ...trap/engines/engine.py.229: process: 0.76433 | load: 0.00295 [I 2022-10-08 10:28:55] ...trap/engines/engine.py.230: loss: 0.23654 [I 2022-10-08 10:28:55] ...optimizers/trijoint.py.151: total_norm: 3.049349 [I 2022-10-08 10:29:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 570/2304 [I 2022-10-08 10:29:03] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:22:10 [I 2022-10-08 10:29:03] ...trap/engines/engine.py.229: process: 0.76290 | load: 0.00294 [I 2022-10-08 10:29:03] ...trap/engines/engine.py.230: loss: 0.29006 [I 2022-10-08 10:29:03] ...optimizers/trijoint.py.151: total_norm: 3.833478 [I 2022-10-08 10:29:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 580/2304 [I 2022-10-08 10:29:11] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:22:12 [I 2022-10-08 10:29:11] ...trap/engines/engine.py.229: process: 0.76707 | load: 0.00292 [I 2022-10-08 10:29:11] ...trap/engines/engine.py.230: loss: 0.33172 [I 2022-10-08 10:29:11] ...optimizers/trijoint.py.151: total_norm: 3.631773 [I 2022-10-08 10:29:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 590/2304 [I 2022-10-08 10:29:18] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:22:04 [I 2022-10-08 10:29:18] ...trap/engines/engine.py.229: process: 0.78192 | load: 0.00295 [I 2022-10-08 10:29:18] ...trap/engines/engine.py.230: loss: 0.28770 [I 2022-10-08 10:29:18] ...optimizers/trijoint.py.151: total_norm: 2.893305 [I 2022-10-08 10:29:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 600/2304 [I 2022-10-08 10:29:26] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:21:53 [I 2022-10-08 10:29:26] ...trap/engines/engine.py.229: process: 0.76637 | load: 0.00294 [I 2022-10-08 10:29:26] ...trap/engines/engine.py.230: loss: 0.25903 [I 2022-10-08 10:29:26] ...optimizers/trijoint.py.151: total_norm: 2.445360 [I 2022-10-08 10:29:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 610/2304 [I 2022-10-08 10:29:34] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:21:44 [I 2022-10-08 10:29:34] ...trap/engines/engine.py.229: process: 0.77236 | load: 0.00293 [I 2022-10-08 10:29:34] ...trap/engines/engine.py.230: loss: 0.32038 [I 2022-10-08 10:29:34] ...optimizers/trijoint.py.151: total_norm: 3.367367 [I 2022-10-08 10:29:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 620/2304 [I 2022-10-08 10:29:42] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:21:28 [I 2022-10-08 10:29:42] ...trap/engines/engine.py.229: process: 0.75358 | load: 0.00295 [I 2022-10-08 10:29:42] ...trap/engines/engine.py.230: loss: 0.37227 [I 2022-10-08 10:29:42] ...optimizers/trijoint.py.151: total_norm: 3.180700 [I 2022-10-08 10:29:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 630/2304 [I 2022-10-08 10:29:49] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:21:19 [I 2022-10-08 10:29:49] ...trap/engines/engine.py.229: process: 0.75721 | load: 0.00295 [I 2022-10-08 10:29:49] ...trap/engines/engine.py.230: loss: 0.23316 [I 2022-10-08 10:29:49] ...optimizers/trijoint.py.151: total_norm: 2.895997 [I 2022-10-08 10:29:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 640/2304 [I 2022-10-08 10:29:57] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:21:22 [I 2022-10-08 10:29:57] ...trap/engines/engine.py.229: process: 0.77151 | load: 0.00294 [I 2022-10-08 10:29:57] ...trap/engines/engine.py.230: loss: 0.30462 [I 2022-10-08 10:29:57] ...optimizers/trijoint.py.151: total_norm: 3.649654 [I 2022-10-08 10:30:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 650/2304 [I 2022-10-08 10:30:05] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:21:14 [I 2022-10-08 10:30:05] ...trap/engines/engine.py.229: process: 0.77313 | load: 0.00295 [I 2022-10-08 10:30:05] ...trap/engines/engine.py.230: loss: 0.34942 [I 2022-10-08 10:30:05] ...optimizers/trijoint.py.151: total_norm: 3.319076 [I 2022-10-08 10:30:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 660/2304 [I 2022-10-08 10:30:12] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:21:04 [I 2022-10-08 10:30:12] ...trap/engines/engine.py.229: process: 0.76256 | load: 0.00304 [I 2022-10-08 10:30:12] ...trap/engines/engine.py.230: loss: 0.33421 [I 2022-10-08 10:30:12] ...optimizers/trijoint.py.151: total_norm: 4.047742 [I 2022-10-08 10:30:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 670/2304 [I 2022-10-08 10:30:20] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:20:53 [I 2022-10-08 10:30:20] ...trap/engines/engine.py.229: process: 0.77122 | load: 0.00295 [I 2022-10-08 10:30:20] ...trap/engines/engine.py.230: loss: 0.31156 [I 2022-10-08 10:30:20] ...optimizers/trijoint.py.151: total_norm: 3.816348 [I 2022-10-08 10:30:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 680/2304 [I 2022-10-08 10:30:28] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:20:53 [I 2022-10-08 10:30:28] ...trap/engines/engine.py.229: process: 0.77450 | load: 0.00292 [I 2022-10-08 10:30:28] ...trap/engines/engine.py.230: loss: 0.33971 [I 2022-10-08 10:30:28] ...optimizers/trijoint.py.151: total_norm: 2.936539 [I 2022-10-08 10:30:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 690/2304 [I 2022-10-08 10:30:35] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:20:45 [I 2022-10-08 10:30:35] ...trap/engines/engine.py.229: process: 0.76890 | load: 0.00295 [I 2022-10-08 10:30:35] ...trap/engines/engine.py.230: loss: 0.22961 [I 2022-10-08 10:30:35] ...optimizers/trijoint.py.151: total_norm: 2.505200 [I 2022-10-08 10:30:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 700/2304 [I 2022-10-08 10:30:43] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:20:37 [I 2022-10-08 10:30:43] ...trap/engines/engine.py.229: process: 0.76771 | load: 0.00294 [I 2022-10-08 10:30:43] ...trap/engines/engine.py.230: loss: 0.24105 [I 2022-10-08 10:30:43] ...optimizers/trijoint.py.151: total_norm: 2.465634 [I 2022-10-08 10:30:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 710/2304 [I 2022-10-08 10:30:51] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:20:28 [I 2022-10-08 10:30:51] ...trap/engines/engine.py.229: process: 0.75282 | load: 0.00293 [I 2022-10-08 10:30:51] ...trap/engines/engine.py.230: loss: 0.26121 [I 2022-10-08 10:30:51] ...optimizers/trijoint.py.151: total_norm: 2.424437 [I 2022-10-08 10:30:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 720/2304 [I 2022-10-08 10:30:59] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:20:21 [I 2022-10-08 10:30:59] ...trap/engines/engine.py.229: process: 0.77320 | load: 0.00305 [I 2022-10-08 10:30:59] ...trap/engines/engine.py.230: loss: 0.27313 [I 2022-10-08 10:30:59] ...optimizers/trijoint.py.151: total_norm: 2.955386 [I 2022-10-08 10:31:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 730/2304 [I 2022-10-08 10:31:06] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:20:06 [I 2022-10-08 10:31:06] ...trap/engines/engine.py.229: process: 0.75448 | load: 0.00296 [I 2022-10-08 10:31:06] ...trap/engines/engine.py.230: loss: 0.23377 [I 2022-10-08 10:31:06] ...optimizers/trijoint.py.151: total_norm: 2.536735 [I 2022-10-08 10:31:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 740/2304 [I 2022-10-08 10:31:14] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:20:01 [I 2022-10-08 10:31:14] ...trap/engines/engine.py.229: process: 0.76704 | load: 0.00295 [I 2022-10-08 10:31:14] ...trap/engines/engine.py.230: loss: 0.25678 [I 2022-10-08 10:31:14] ...optimizers/trijoint.py.151: total_norm: 2.954327 [I 2022-10-08 10:31:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 750/2304 [I 2022-10-08 10:31:22] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:20:01 [I 2022-10-08 10:31:22] ...trap/engines/engine.py.229: process: 0.77263 | load: 0.00296 [I 2022-10-08 10:31:22] ...trap/engines/engine.py.230: loss: 0.24700 [I 2022-10-08 10:31:22] ...optimizers/trijoint.py.151: total_norm: 2.829702 [I 2022-10-08 10:31:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 760/2304 [I 2022-10-08 10:31:30] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:19:48 [I 2022-10-08 10:31:30] ...trap/engines/engine.py.229: process: 0.76358 | load: 0.00295 [I 2022-10-08 10:31:30] ...trap/engines/engine.py.230: loss: 0.31519 [I 2022-10-08 10:31:30] ...optimizers/trijoint.py.151: total_norm: 3.543931 [I 2022-10-08 10:31:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 770/2304 [I 2022-10-08 10:31:37] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:19:40 [I 2022-10-08 10:31:37] ...trap/engines/engine.py.229: process: 0.77085 | load: 0.00293 [I 2022-10-08 10:31:37] ...trap/engines/engine.py.230: loss: 0.26074 [I 2022-10-08 10:31:37] ...optimizers/trijoint.py.151: total_norm: 3.081377 [I 2022-10-08 10:31:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 780/2304 [I 2022-10-08 10:31:45] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:19:34 [I 2022-10-08 10:31:45] ...trap/engines/engine.py.229: process: 0.77279 | load: 0.00296 [I 2022-10-08 10:31:45] ...trap/engines/engine.py.230: loss: 0.36374 [I 2022-10-08 10:31:45] ...optimizers/trijoint.py.151: total_norm: 4.009770 [I 2022-10-08 10:31:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 790/2304 [I 2022-10-08 10:31:53] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:19:25 [I 2022-10-08 10:31:53] ...trap/engines/engine.py.229: process: 0.76135 | load: 0.00295 [I 2022-10-08 10:31:53] ...trap/engines/engine.py.230: loss: 0.25446 [I 2022-10-08 10:31:53] ...optimizers/trijoint.py.151: total_norm: 3.181690 [I 2022-10-08 10:32:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 800/2304 [I 2022-10-08 10:32:00] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:21 [I 2022-10-08 10:32:00] ...trap/engines/engine.py.229: process: 0.76929 | load: 0.00293 [I 2022-10-08 10:32:00] ...trap/engines/engine.py.230: loss: 0.30726 [I 2022-10-08 10:32:00] ...optimizers/trijoint.py.151: total_norm: 3.027997 [I 2022-10-08 10:32:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 810/2304 [I 2022-10-08 10:32:08] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:07 [I 2022-10-08 10:32:08] ...trap/engines/engine.py.229: process: 0.76560 | load: 0.00295 [I 2022-10-08 10:32:08] ...trap/engines/engine.py.230: loss: 0.30073 [I 2022-10-08 10:32:08] ...optimizers/trijoint.py.151: total_norm: 3.721605 [I 2022-10-08 10:32:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 820/2304 [I 2022-10-08 10:32:16] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:02 [I 2022-10-08 10:32:16] ...trap/engines/engine.py.229: process: 0.76484 | load: 0.00295 [I 2022-10-08 10:32:16] ...trap/engines/engine.py.230: loss: 0.21217 [I 2022-10-08 10:32:16] ...optimizers/trijoint.py.151: total_norm: 3.033257 [I 2022-10-08 10:32:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 830/2304 [I 2022-10-08 10:32:24] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:07 [I 2022-10-08 10:32:24] ...trap/engines/engine.py.229: process: 0.78383 | load: 0.00295 [I 2022-10-08 10:32:24] ...trap/engines/engine.py.230: loss: 0.31149 [I 2022-10-08 10:32:24] ...optimizers/trijoint.py.151: total_norm: 3.237354 [I 2022-10-08 10:32:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 840/2304 [I 2022-10-08 10:32:31] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:18:52 [I 2022-10-08 10:32:31] ...trap/engines/engine.py.229: process: 0.78263 | load: 0.00296 [I 2022-10-08 10:32:31] ...trap/engines/engine.py.230: loss: 0.30292 [I 2022-10-08 10:32:31] ...optimizers/trijoint.py.151: total_norm: 3.238127 [I 2022-10-08 10:32:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 850/2304 [I 2022-10-08 10:32:39] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:18:37 [I 2022-10-08 10:32:39] ...trap/engines/engine.py.229: process: 0.77885 | load: 0.00295 [I 2022-10-08 10:32:39] ...trap/engines/engine.py.230: loss: 0.26211 [I 2022-10-08 10:32:39] ...optimizers/trijoint.py.151: total_norm: 3.327394 [I 2022-10-08 10:32:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 860/2304 [I 2022-10-08 10:32:47] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:18:36 [I 2022-10-08 10:32:47] ...trap/engines/engine.py.229: process: 0.78215 | load: 0.00295 [I 2022-10-08 10:32:47] ...trap/engines/engine.py.230: loss: 0.44211 [I 2022-10-08 10:32:47] ...optimizers/trijoint.py.151: total_norm: 3.868508 [I 2022-10-08 10:32:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 870/2304 [I 2022-10-08 10:32:54] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:29 [I 2022-10-08 10:32:54] ...trap/engines/engine.py.229: process: 0.77572 | load: 0.00294 [I 2022-10-08 10:32:54] ...trap/engines/engine.py.230: loss: 0.29484 [I 2022-10-08 10:32:54] ...optimizers/trijoint.py.151: total_norm: 2.935230 [I 2022-10-08 10:33:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 880/2304 [I 2022-10-08 10:33:02] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:21 [I 2022-10-08 10:33:02] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00294 [I 2022-10-08 10:33:02] ...trap/engines/engine.py.230: loss: 0.19125 [I 2022-10-08 10:33:02] ...optimizers/trijoint.py.151: total_norm: 1.852929 [I 2022-10-08 10:33:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 890/2304 [I 2022-10-08 10:33:10] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:11 [I 2022-10-08 10:33:10] ...trap/engines/engine.py.229: process: 0.76616 | load: 0.00294 [I 2022-10-08 10:33:10] ...trap/engines/engine.py.230: loss: 0.28822 [I 2022-10-08 10:33:10] ...optimizers/trijoint.py.151: total_norm: 2.839080 [I 2022-10-08 10:33:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 900/2304 [I 2022-10-08 10:33:18] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:08 [I 2022-10-08 10:33:18] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00294 [I 2022-10-08 10:33:18] ...trap/engines/engine.py.230: loss: 0.43069 [I 2022-10-08 10:33:18] ...optimizers/trijoint.py.151: total_norm: 4.519439 [I 2022-10-08 10:33:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 910/2304 [I 2022-10-08 10:33:25] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:17:50 [I 2022-10-08 10:33:25] ...trap/engines/engine.py.229: process: 0.76242 | load: 0.00294 [I 2022-10-08 10:33:25] ...trap/engines/engine.py.230: loss: 0.29237 [I 2022-10-08 10:33:25] ...optimizers/trijoint.py.151: total_norm: 3.648695 [I 2022-10-08 10:33:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 920/2304 [I 2022-10-08 10:33:33] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:17:44 [I 2022-10-08 10:33:33] ...trap/engines/engine.py.229: process: 0.76755 | load: 0.00296 [I 2022-10-08 10:33:33] ...trap/engines/engine.py.230: loss: 0.28664 [I 2022-10-08 10:33:33] ...optimizers/trijoint.py.151: total_norm: 3.240514 [I 2022-10-08 10:33:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 930/2304 [I 2022-10-08 10:33:41] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:17:34 [I 2022-10-08 10:33:41] ...trap/engines/engine.py.229: process: 0.76451 | load: 0.00293 [I 2022-10-08 10:33:41] ...trap/engines/engine.py.230: loss: 0.41511 [I 2022-10-08 10:33:41] ...optimizers/trijoint.py.151: total_norm: 5.084932 [I 2022-10-08 10:33:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 940/2304 [I 2022-10-08 10:33:49] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:32 [I 2022-10-08 10:33:49] ...trap/engines/engine.py.229: process: 0.76878 | load: 0.00297 [I 2022-10-08 10:33:49] ...trap/engines/engine.py.230: loss: 0.31898 [I 2022-10-08 10:33:49] ...optimizers/trijoint.py.151: total_norm: 3.337356 [I 2022-10-08 10:33:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 950/2304 [I 2022-10-08 10:33:56] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:24 [I 2022-10-08 10:33:56] ...trap/engines/engine.py.229: process: 0.76759 | load: 0.00300 [I 2022-10-08 10:33:56] ...trap/engines/engine.py.230: loss: 0.21167 [I 2022-10-08 10:33:56] ...optimizers/trijoint.py.151: total_norm: 3.034372 [I 2022-10-08 10:34:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 960/2304 [I 2022-10-08 10:34:04] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:15 [I 2022-10-08 10:34:04] ...trap/engines/engine.py.229: process: 0.78496 | load: 0.00293 [I 2022-10-08 10:34:04] ...trap/engines/engine.py.230: loss: 0.29471 [I 2022-10-08 10:34:04] ...optimizers/trijoint.py.151: total_norm: 2.996418 [I 2022-10-08 10:34:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 970/2304 [I 2022-10-08 10:34:12] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:17:07 [I 2022-10-08 10:34:12] ...trap/engines/engine.py.229: process: 0.78605 | load: 0.00293 [I 2022-10-08 10:34:12] ...trap/engines/engine.py.230: loss: 0.24865 [I 2022-10-08 10:34:12] ...optimizers/trijoint.py.151: total_norm: 3.003724 [I 2022-10-08 10:34:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 980/2304 [I 2022-10-08 10:34:19] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:16:56 [I 2022-10-08 10:34:19] ...trap/engines/engine.py.229: process: 0.76095 | load: 0.00293 [I 2022-10-08 10:34:19] ...trap/engines/engine.py.230: loss: 0.40731 [I 2022-10-08 10:34:19] ...optimizers/trijoint.py.151: total_norm: 3.894351 [I 2022-10-08 10:34:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 990/2304 [I 2022-10-08 10:34:27] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:16:52 [I 2022-10-08 10:34:27] ...trap/engines/engine.py.229: process: 0.77510 | load: 0.00294 [I 2022-10-08 10:34:27] ...trap/engines/engine.py.230: loss: 0.31946 [I 2022-10-08 10:34:27] ...optimizers/trijoint.py.151: total_norm: 3.846825 [I 2022-10-08 10:34:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 1000/2304 [I 2022-10-08 10:34:35] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:16:44 [I 2022-10-08 10:34:35] ...trap/engines/engine.py.229: process: 0.78192 | load: 0.00292 [I 2022-10-08 10:34:35] ...trap/engines/engine.py.230: loss: 0.37036 [I 2022-10-08 10:34:35] ...optimizers/trijoint.py.151: total_norm: 4.974123 [I 2022-10-08 10:34:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 1010/2304 [I 2022-10-08 10:34:42] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:16:43 [I 2022-10-08 10:34:42] ...trap/engines/engine.py.229: process: 0.78270 | load: 0.00294 [I 2022-10-08 10:34:42] ...trap/engines/engine.py.230: loss: 0.29736 [I 2022-10-08 10:34:42] ...optimizers/trijoint.py.151: total_norm: 3.411525 [I 2022-10-08 10:34:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 1020/2304 [I 2022-10-08 10:34:50] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:16:32 [I 2022-10-08 10:34:50] ...trap/engines/engine.py.229: process: 0.77543 | load: 0.00291 [I 2022-10-08 10:34:50] ...trap/engines/engine.py.230: loss: 0.24038 [I 2022-10-08 10:34:50] ...optimizers/trijoint.py.151: total_norm: 2.735224 [I 2022-10-08 10:34:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 1030/2304 [I 2022-10-08 10:34:58] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:16:17 [I 2022-10-08 10:34:58] ...trap/engines/engine.py.229: process: 0.75712 | load: 0.00296 [I 2022-10-08 10:34:58] ...trap/engines/engine.py.230: loss: 0.21349 [I 2022-10-08 10:34:58] ...optimizers/trijoint.py.151: total_norm: 3.000143 [I 2022-10-08 10:35:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 1040/2304 [I 2022-10-08 10:35:06] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:16:16 [I 2022-10-08 10:35:06] ...trap/engines/engine.py.229: process: 0.77922 | load: 0.00295 [I 2022-10-08 10:35:06] ...trap/engines/engine.py.230: loss: 0.26947 [I 2022-10-08 10:35:06] ...optimizers/trijoint.py.151: total_norm: 3.512119 [I 2022-10-08 10:35:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 1050/2304 [I 2022-10-08 10:35:13] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:16:13 [I 2022-10-08 10:35:13] ...trap/engines/engine.py.229: process: 0.79302 | load: 0.00296 [I 2022-10-08 10:35:13] ...trap/engines/engine.py.230: loss: 0.33551 [I 2022-10-08 10:35:13] ...optimizers/trijoint.py.151: total_norm: 4.325740 [I 2022-10-08 10:35:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 1060/2304 [I 2022-10-08 10:35:21] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:16:01 [I 2022-10-08 10:35:21] ...trap/engines/engine.py.229: process: 0.77498 | load: 0.00294 [I 2022-10-08 10:35:21] ...trap/engines/engine.py.230: loss: 0.25154 [I 2022-10-08 10:35:21] ...optimizers/trijoint.py.151: total_norm: 3.480131 [I 2022-10-08 10:35:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 1070/2304 [I 2022-10-08 10:35:29] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:15:46 [I 2022-10-08 10:35:29] ...trap/engines/engine.py.229: process: 0.75818 | load: 0.00295 [I 2022-10-08 10:35:29] ...trap/engines/engine.py.230: loss: 0.28638 [I 2022-10-08 10:35:29] ...optimizers/trijoint.py.151: total_norm: 2.612115 [I 2022-10-08 10:35:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 1080/2304 [I 2022-10-08 10:35:37] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:15:38 [I 2022-10-08 10:35:37] ...trap/engines/engine.py.229: process: 0.76881 | load: 0.00294 [I 2022-10-08 10:35:37] ...trap/engines/engine.py.230: loss: 0.17340 [I 2022-10-08 10:35:37] ...optimizers/trijoint.py.151: total_norm: 2.932139 [I 2022-10-08 10:35:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 1090/2304 [I 2022-10-08 10:35:44] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:15:35 [I 2022-10-08 10:35:44] ...trap/engines/engine.py.229: process: 0.76034 | load: 0.00295 [I 2022-10-08 10:35:44] ...trap/engines/engine.py.230: loss: 0.22917 [I 2022-10-08 10:35:44] ...optimizers/trijoint.py.151: total_norm: 2.491840 [I 2022-10-08 10:35:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 1100/2304 [I 2022-10-08 10:35:52] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:15:32 [I 2022-10-08 10:35:52] ...trap/engines/engine.py.229: process: 0.76110 | load: 0.00294 [I 2022-10-08 10:35:52] ...trap/engines/engine.py.230: loss: 0.28510 [I 2022-10-08 10:35:52] ...optimizers/trijoint.py.151: total_norm: 3.693839 [I 2022-10-08 10:36:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 1110/2304 [I 2022-10-08 10:36:00] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:15:19 [I 2022-10-08 10:36:00] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00295 [I 2022-10-08 10:36:00] ...trap/engines/engine.py.230: loss: 0.34810 [I 2022-10-08 10:36:00] ...optimizers/trijoint.py.151: total_norm: 3.734190 [I 2022-10-08 10:36:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 1120/2304 [I 2022-10-08 10:36:07] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:15:12 [I 2022-10-08 10:36:07] ...trap/engines/engine.py.229: process: 0.77721 | load: 0.00296 [I 2022-10-08 10:36:07] ...trap/engines/engine.py.230: loss: 0.31267 [I 2022-10-08 10:36:07] ...optimizers/trijoint.py.151: total_norm: 4.067457 [I 2022-10-08 10:36:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 1130/2304 [I 2022-10-08 10:36:15] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:15:05 [I 2022-10-08 10:36:15] ...trap/engines/engine.py.229: process: 0.77576 | load: 0.00297 [I 2022-10-08 10:36:15] ...trap/engines/engine.py.230: loss: 0.29540 [I 2022-10-08 10:36:15] ...optimizers/trijoint.py.151: total_norm: 3.203165 [I 2022-10-08 10:36:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 1140/2304 [I 2022-10-08 10:36:23] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:14:57 [I 2022-10-08 10:36:23] ...trap/engines/engine.py.229: process: 0.76748 | load: 0.00304 [I 2022-10-08 10:36:23] ...trap/engines/engine.py.230: loss: 0.30046 [I 2022-10-08 10:36:23] ...optimizers/trijoint.py.151: total_norm: 2.533010 [I 2022-10-08 10:36:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 1150/2304 [I 2022-10-08 10:36:31] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:14:46 [I 2022-10-08 10:36:31] ...trap/engines/engine.py.229: process: 0.76590 | load: 0.00299 [I 2022-10-08 10:36:31] ...trap/engines/engine.py.230: loss: 0.26503 [I 2022-10-08 10:36:31] ...optimizers/trijoint.py.151: total_norm: 2.874099 [I 2022-10-08 10:36:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 1160/2304 [I 2022-10-08 10:36:38] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:14:48 [I 2022-10-08 10:36:38] ...trap/engines/engine.py.229: process: 0.77926 | load: 0.00295 [I 2022-10-08 10:36:38] ...trap/engines/engine.py.230: loss: 0.21830 [I 2022-10-08 10:36:38] ...optimizers/trijoint.py.151: total_norm: 2.192898 [I 2022-10-08 10:36:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 1170/2304 [I 2022-10-08 10:36:46] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:14:27 [I 2022-10-08 10:36:46] ...trap/engines/engine.py.229: process: 0.76519 | load: 0.00303 [I 2022-10-08 10:36:46] ...trap/engines/engine.py.230: loss: 0.30583 [I 2022-10-08 10:36:46] ...optimizers/trijoint.py.151: total_norm: 3.618263 [I 2022-10-08 10:36:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 1180/2304 [I 2022-10-08 10:36:54] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:14:21 [I 2022-10-08 10:36:54] ...trap/engines/engine.py.229: process: 0.75945 | load: 0.00299 [I 2022-10-08 10:36:54] ...trap/engines/engine.py.230: loss: 0.39074 [I 2022-10-08 10:36:54] ...optimizers/trijoint.py.151: total_norm: 3.941577 [I 2022-10-08 10:37:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 1190/2304 [I 2022-10-08 10:37:01] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:14:19 [I 2022-10-08 10:37:01] ...trap/engines/engine.py.229: process: 0.77003 | load: 0.00294 [I 2022-10-08 10:37:01] ...trap/engines/engine.py.230: loss: 0.23810 [I 2022-10-08 10:37:01] ...optimizers/trijoint.py.151: total_norm: 2.757597 [I 2022-10-08 10:37:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 1200/2304 [I 2022-10-08 10:37:09] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:14:15 [I 2022-10-08 10:37:09] ...trap/engines/engine.py.229: process: 0.77698 | load: 0.00293 [I 2022-10-08 10:37:09] ...trap/engines/engine.py.230: loss: 0.28600 [I 2022-10-08 10:37:09] ...optimizers/trijoint.py.151: total_norm: 3.361140 [I 2022-10-08 10:37:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 1210/2304 [I 2022-10-08 10:37:17] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:14:03 [I 2022-10-08 10:37:17] ...trap/engines/engine.py.229: process: 0.76288 | load: 0.00295 [I 2022-10-08 10:37:17] ...trap/engines/engine.py.230: loss: 0.37311 [I 2022-10-08 10:37:17] ...optimizers/trijoint.py.151: total_norm: 3.521835 [I 2022-10-08 10:37:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 1220/2304 [I 2022-10-08 10:37:25] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:13:57 [I 2022-10-08 10:37:25] ...trap/engines/engine.py.229: process: 0.76955 | load: 0.00293 [I 2022-10-08 10:37:25] ...trap/engines/engine.py.230: loss: 0.27846 [I 2022-10-08 10:37:25] ...optimizers/trijoint.py.151: total_norm: 2.924130 [I 2022-10-08 10:37:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 1230/2304 [I 2022-10-08 10:37:32] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:13:49 [I 2022-10-08 10:37:32] ...trap/engines/engine.py.229: process: 0.77521 | load: 0.00294 [I 2022-10-08 10:37:32] ...trap/engines/engine.py.230: loss: 0.20367 [I 2022-10-08 10:37:32] ...optimizers/trijoint.py.151: total_norm: 2.586586 [I 2022-10-08 10:37:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 1240/2304 [I 2022-10-08 10:37:40] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:13:40 [I 2022-10-08 10:37:40] ...trap/engines/engine.py.229: process: 0.76361 | load: 0.00294 [I 2022-10-08 10:37:40] ...trap/engines/engine.py.230: loss: 0.27888 [I 2022-10-08 10:37:40] ...optimizers/trijoint.py.151: total_norm: 3.296984 [I 2022-10-08 10:37:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 1250/2304 [I 2022-10-08 10:37:48] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:13:30 [I 2022-10-08 10:37:48] ...trap/engines/engine.py.229: process: 0.77722 | load: 0.00296 [I 2022-10-08 10:37:48] ...trap/engines/engine.py.230: loss: 0.28708 [I 2022-10-08 10:37:48] ...optimizers/trijoint.py.151: total_norm: 2.705948 [I 2022-10-08 10:37:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 1260/2304 [I 2022-10-08 10:37:55] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:13:20 [I 2022-10-08 10:37:55] ...trap/engines/engine.py.229: process: 0.75107 | load: 0.00294 [I 2022-10-08 10:37:55] ...trap/engines/engine.py.230: loss: 0.25396 [I 2022-10-08 10:37:55] ...optimizers/trijoint.py.151: total_norm: 3.133997 [I 2022-10-08 10:38:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 1270/2304 [I 2022-10-08 10:38:03] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:13:11 [I 2022-10-08 10:38:03] ...trap/engines/engine.py.229: process: 0.76248 | load: 0.00293 [I 2022-10-08 10:38:03] ...trap/engines/engine.py.230: loss: 0.27249 [I 2022-10-08 10:38:03] ...optimizers/trijoint.py.151: total_norm: 2.820421 [I 2022-10-08 10:38:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 1280/2304 [I 2022-10-08 10:38:11] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:13:04 [I 2022-10-08 10:38:11] ...trap/engines/engine.py.229: process: 0.75963 | load: 0.00292 [I 2022-10-08 10:38:11] ...trap/engines/engine.py.230: loss: 0.23444 [I 2022-10-08 10:38:11] ...optimizers/trijoint.py.151: total_norm: 3.044686 [I 2022-10-08 10:38:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 1290/2304 [I 2022-10-08 10:38:19] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:12:58 [I 2022-10-08 10:38:19] ...trap/engines/engine.py.229: process: 0.77274 | load: 0.00294 [I 2022-10-08 10:38:19] ...trap/engines/engine.py.230: loss: 0.27431 [I 2022-10-08 10:38:19] ...optimizers/trijoint.py.151: total_norm: 2.784405 [I 2022-10-08 10:38:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 1300/2304 [I 2022-10-08 10:38:26] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:12:56 [I 2022-10-08 10:38:26] ...trap/engines/engine.py.229: process: 0.78299 | load: 0.00294 [I 2022-10-08 10:38:26] ...trap/engines/engine.py.230: loss: 0.33074 [I 2022-10-08 10:38:26] ...optimizers/trijoint.py.151: total_norm: 3.838249 [I 2022-10-08 10:38:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 1310/2304 [I 2022-10-08 10:38:34] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:12:48 [I 2022-10-08 10:38:34] ...trap/engines/engine.py.229: process: 0.77279 | load: 0.00295 [I 2022-10-08 10:38:34] ...trap/engines/engine.py.230: loss: 0.33612 [I 2022-10-08 10:38:34] ...optimizers/trijoint.py.151: total_norm: 2.699928 [I 2022-10-08 10:38:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 1320/2304 [I 2022-10-08 10:38:42] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:12:38 [I 2022-10-08 10:38:42] ...trap/engines/engine.py.229: process: 0.77124 | load: 0.00295 [I 2022-10-08 10:38:42] ...trap/engines/engine.py.230: loss: 0.35475 [I 2022-10-08 10:38:42] ...optimizers/trijoint.py.151: total_norm: 3.331674 [I 2022-10-08 10:38:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 1330/2304 [I 2022-10-08 10:38:49] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:12:29 [I 2022-10-08 10:38:49] ...trap/engines/engine.py.229: process: 0.76309 | load: 0.00293 [I 2022-10-08 10:38:49] ...trap/engines/engine.py.230: loss: 0.31257 [I 2022-10-08 10:38:49] ...optimizers/trijoint.py.151: total_norm: 2.617516 [I 2022-10-08 10:38:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 1340/2304 [I 2022-10-08 10:38:57] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:12:21 [I 2022-10-08 10:38:57] ...trap/engines/engine.py.229: process: 0.75915 | load: 0.00293 [I 2022-10-08 10:38:57] ...trap/engines/engine.py.230: loss: 0.25082 [I 2022-10-08 10:38:57] ...optimizers/trijoint.py.151: total_norm: 3.065739 [I 2022-10-08 10:39:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 1350/2304 [I 2022-10-08 10:39:05] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:12:11 [I 2022-10-08 10:39:05] ...trap/engines/engine.py.229: process: 0.75906 | load: 0.00293 [I 2022-10-08 10:39:05] ...trap/engines/engine.py.230: loss: 0.29075 [I 2022-10-08 10:39:05] ...optimizers/trijoint.py.151: total_norm: 3.468368 [I 2022-10-08 10:39:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 1360/2304 [I 2022-10-08 10:39:13] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:12:03 [I 2022-10-08 10:39:13] ...trap/engines/engine.py.229: process: 0.75737 | load: 0.00294 [I 2022-10-08 10:39:13] ...trap/engines/engine.py.230: loss: 0.28219 [I 2022-10-08 10:39:13] ...optimizers/trijoint.py.151: total_norm: 2.725930 [I 2022-10-08 10:39:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 1370/2304 [I 2022-10-08 10:39:20] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:11:56 [I 2022-10-08 10:39:20] ...trap/engines/engine.py.229: process: 0.75694 | load: 0.00296 [I 2022-10-08 10:39:20] ...trap/engines/engine.py.230: loss: 0.27528 [I 2022-10-08 10:39:20] ...optimizers/trijoint.py.151: total_norm: 3.141222 [I 2022-10-08 10:39:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 1380/2304 [I 2022-10-08 10:39:28] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:11:52 [I 2022-10-08 10:39:28] ...trap/engines/engine.py.229: process: 0.77581 | load: 0.00293 [I 2022-10-08 10:39:28] ...trap/engines/engine.py.230: loss: 0.23752 [I 2022-10-08 10:39:28] ...optimizers/trijoint.py.151: total_norm: 2.793729 [I 2022-10-08 10:39:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 1390/2304 [I 2022-10-08 10:39:36] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:11:39 [I 2022-10-08 10:39:36] ...trap/engines/engine.py.229: process: 0.75932 | load: 0.00292 [I 2022-10-08 10:39:36] ...trap/engines/engine.py.230: loss: 0.24259 [I 2022-10-08 10:39:36] ...optimizers/trijoint.py.151: total_norm: 2.596452 [I 2022-10-08 10:39:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 1400/2304 [I 2022-10-08 10:39:43] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:11:33 [I 2022-10-08 10:39:43] ...trap/engines/engine.py.229: process: 0.76370 | load: 0.00292 [I 2022-10-08 10:39:43] ...trap/engines/engine.py.230: loss: 0.29255 [I 2022-10-08 10:39:43] ...optimizers/trijoint.py.151: total_norm: 2.798543 [I 2022-10-08 10:39:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 1410/2304 [I 2022-10-08 10:39:51] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:11:31 [I 2022-10-08 10:39:51] ...trap/engines/engine.py.229: process: 0.77805 | load: 0.00294 [I 2022-10-08 10:39:51] ...trap/engines/engine.py.230: loss: 0.22772 [I 2022-10-08 10:39:51] ...optimizers/trijoint.py.151: total_norm: 3.054455 [I 2022-10-08 10:39:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 1420/2304 [I 2022-10-08 10:39:59] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:11:21 [I 2022-10-08 10:39:59] ...trap/engines/engine.py.229: process: 0.76576 | load: 0.00292 [I 2022-10-08 10:39:59] ...trap/engines/engine.py.230: loss: 0.28780 [I 2022-10-08 10:39:59] ...optimizers/trijoint.py.151: total_norm: 3.179541 [I 2022-10-08 10:40:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 1430/2304 [I 2022-10-08 10:40:06] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:11:09 [I 2022-10-08 10:40:06] ...trap/engines/engine.py.229: process: 0.76214 | load: 0.00292 [I 2022-10-08 10:40:06] ...trap/engines/engine.py.230: loss: 0.35387 [I 2022-10-08 10:40:06] ...optimizers/trijoint.py.151: total_norm: 3.842828 [I 2022-10-08 10:40:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 1440/2304 [I 2022-10-08 10:40:14] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:11:04 [I 2022-10-08 10:40:14] ...trap/engines/engine.py.229: process: 0.76782 | load: 0.00293 [I 2022-10-08 10:40:14] ...trap/engines/engine.py.230: loss: 0.34446 [I 2022-10-08 10:40:14] ...optimizers/trijoint.py.151: total_norm: 3.763189 [I 2022-10-08 10:40:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 1450/2304 [I 2022-10-08 10:40:22] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:10:58 [I 2022-10-08 10:40:22] ...trap/engines/engine.py.229: process: 0.77155 | load: 0.00295 [I 2022-10-08 10:40:22] ...trap/engines/engine.py.230: loss: 0.24713 [I 2022-10-08 10:40:22] ...optimizers/trijoint.py.151: total_norm: 2.680558 [I 2022-10-08 10:40:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 1460/2304 [I 2022-10-08 10:40:30] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:10:44 [I 2022-10-08 10:40:30] ...trap/engines/engine.py.229: process: 0.76546 | load: 0.00291 [I 2022-10-08 10:40:30] ...trap/engines/engine.py.230: loss: 0.37057 [I 2022-10-08 10:40:30] ...optimizers/trijoint.py.151: total_norm: 3.308382 [I 2022-10-08 10:40:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 1470/2304 [I 2022-10-08 10:40:37] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:10:39 [I 2022-10-08 10:40:37] ...trap/engines/engine.py.229: process: 0.77368 | load: 0.00293 [I 2022-10-08 10:40:37] ...trap/engines/engine.py.230: loss: 0.24241 [I 2022-10-08 10:40:37] ...optimizers/trijoint.py.151: total_norm: 2.671613 [I 2022-10-08 10:40:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 1480/2304 [I 2022-10-08 10:40:45] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:10:34 [I 2022-10-08 10:40:45] ...trap/engines/engine.py.229: process: 0.77138 | load: 0.00293 [I 2022-10-08 10:40:45] ...trap/engines/engine.py.230: loss: 0.38949 [I 2022-10-08 10:40:45] ...optimizers/trijoint.py.151: total_norm: 3.747764 [I 2022-10-08 10:40:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 1490/2304 [I 2022-10-08 10:40:53] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:10:25 [I 2022-10-08 10:40:53] ...trap/engines/engine.py.229: process: 0.76283 | load: 0.00293 [I 2022-10-08 10:40:53] ...trap/engines/engine.py.230: loss: 0.34841 [I 2022-10-08 10:40:53] ...optimizers/trijoint.py.151: total_norm: 3.833492 [I 2022-10-08 10:41:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 1500/2304 [I 2022-10-08 10:41:00] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:10:18 [I 2022-10-08 10:41:00] ...trap/engines/engine.py.229: process: 0.77831 | load: 0.00295 [I 2022-10-08 10:41:00] ...trap/engines/engine.py.230: loss: 0.21737 [I 2022-10-08 10:41:00] ...optimizers/trijoint.py.151: total_norm: 3.152021 [I 2022-10-08 10:41:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 1510/2304 [I 2022-10-08 10:41:08] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:10:10 [I 2022-10-08 10:41:08] ...trap/engines/engine.py.229: process: 0.77342 | load: 0.00294 [I 2022-10-08 10:41:08] ...trap/engines/engine.py.230: loss: 0.32892 [I 2022-10-08 10:41:08] ...optimizers/trijoint.py.151: total_norm: 3.604665 [I 2022-10-08 10:41:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 1520/2304 [I 2022-10-08 10:41:16] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:10:06 [I 2022-10-08 10:41:16] ...trap/engines/engine.py.229: process: 0.78207 | load: 0.00298 [I 2022-10-08 10:41:16] ...trap/engines/engine.py.230: loss: 0.27549 [I 2022-10-08 10:41:16] ...optimizers/trijoint.py.151: total_norm: 2.662281 [I 2022-10-08 10:41:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 1530/2304 [I 2022-10-08 10:41:23] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:09:56 [I 2022-10-08 10:41:23] ...trap/engines/engine.py.229: process: 0.77021 | load: 0.00294 [I 2022-10-08 10:41:23] ...trap/engines/engine.py.230: loss: 0.35349 [I 2022-10-08 10:41:23] ...optimizers/trijoint.py.151: total_norm: 3.928022 [I 2022-10-08 10:41:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 1540/2304 [I 2022-10-08 10:41:31] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:09:52 [I 2022-10-08 10:41:31] ...trap/engines/engine.py.229: process: 0.76642 | load: 0.00296 [I 2022-10-08 10:41:31] ...trap/engines/engine.py.230: loss: 0.26954 [I 2022-10-08 10:41:31] ...optimizers/trijoint.py.151: total_norm: 2.717101 [I 2022-10-08 10:41:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 1550/2304 [I 2022-10-08 10:41:39] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:09:39 [I 2022-10-08 10:41:39] ...trap/engines/engine.py.229: process: 0.77198 | load: 0.00295 [I 2022-10-08 10:41:39] ...trap/engines/engine.py.230: loss: 0.36693 [I 2022-10-08 10:41:39] ...optimizers/trijoint.py.151: total_norm: 3.172035 [I 2022-10-08 10:41:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 1560/2304 [I 2022-10-08 10:41:47] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:31 [I 2022-10-08 10:41:47] ...trap/engines/engine.py.229: process: 0.77342 | load: 0.00295 [I 2022-10-08 10:41:47] ...trap/engines/engine.py.230: loss: 0.31154 [I 2022-10-08 10:41:47] ...optimizers/trijoint.py.151: total_norm: 3.886287 [I 2022-10-08 10:41:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 1570/2304 [I 2022-10-08 10:41:54] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:24 [I 2022-10-08 10:41:54] ...trap/engines/engine.py.229: process: 0.77169 | load: 0.00295 [I 2022-10-08 10:41:54] ...trap/engines/engine.py.230: loss: 0.20980 [I 2022-10-08 10:41:54] ...optimizers/trijoint.py.151: total_norm: 3.012768 [I 2022-10-08 10:42:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 1580/2304 [I 2022-10-08 10:42:02] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:24 [I 2022-10-08 10:42:02] ...trap/engines/engine.py.229: process: 0.79862 | load: 0.00295 [I 2022-10-08 10:42:02] ...trap/engines/engine.py.230: loss: 0.34838 [I 2022-10-08 10:42:02] ...optimizers/trijoint.py.151: total_norm: 3.364692 [I 2022-10-08 10:42:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 1590/2304 [I 2022-10-08 10:42:10] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:09:11 [I 2022-10-08 10:42:10] ...trap/engines/engine.py.229: process: 0.76022 | load: 0.00293 [I 2022-10-08 10:42:10] ...trap/engines/engine.py.230: loss: 0.37723 [I 2022-10-08 10:42:10] ...optimizers/trijoint.py.151: total_norm: 4.054229 [I 2022-10-08 10:42:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 1600/2304 [I 2022-10-08 10:42:18] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:02 [I 2022-10-08 10:42:18] ...trap/engines/engine.py.229: process: 0.77253 | load: 0.00294 [I 2022-10-08 10:42:18] ...trap/engines/engine.py.230: loss: 0.31470 [I 2022-10-08 10:42:18] ...optimizers/trijoint.py.151: total_norm: 3.409723 [I 2022-10-08 10:42:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 1610/2304 [I 2022-10-08 10:42:25] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:08:52 [I 2022-10-08 10:42:25] ...trap/engines/engine.py.229: process: 0.77182 | load: 0.00294 [I 2022-10-08 10:42:25] ...trap/engines/engine.py.230: loss: 0.41360 [I 2022-10-08 10:42:25] ...optimizers/trijoint.py.151: total_norm: 3.936529 [I 2022-10-08 10:42:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 1620/2304 [I 2022-10-08 10:42:33] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:08:52 [I 2022-10-08 10:42:33] ...trap/engines/engine.py.229: process: 0.80987 | load: 0.00295 [I 2022-10-08 10:42:33] ...trap/engines/engine.py.230: loss: 0.31648 [I 2022-10-08 10:42:33] ...optimizers/trijoint.py.151: total_norm: 3.886834 [I 2022-10-08 10:42:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 1630/2304 [I 2022-10-08 10:42:41] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:08:37 [I 2022-10-08 10:42:41] ...trap/engines/engine.py.229: process: 0.77306 | load: 0.00294 [I 2022-10-08 10:42:41] ...trap/engines/engine.py.230: loss: 0.37462 [I 2022-10-08 10:42:41] ...optimizers/trijoint.py.151: total_norm: 3.125357 [I 2022-10-08 10:42:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 1640/2304 [I 2022-10-08 10:42:48] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:30 [I 2022-10-08 10:42:48] ...trap/engines/engine.py.229: process: 0.77086 | load: 0.00294 [I 2022-10-08 10:42:48] ...trap/engines/engine.py.230: loss: 0.41848 [I 2022-10-08 10:42:48] ...optimizers/trijoint.py.151: total_norm: 4.024831 [I 2022-10-08 10:42:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 1650/2304 [I 2022-10-08 10:42:56] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:25 [I 2022-10-08 10:42:56] ...trap/engines/engine.py.229: process: 0.77392 | load: 0.00293 [I 2022-10-08 10:42:56] ...trap/engines/engine.py.230: loss: 0.31000 [I 2022-10-08 10:42:56] ...optimizers/trijoint.py.151: total_norm: 3.161975 [I 2022-10-08 10:43:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 1660/2304 [I 2022-10-08 10:43:04] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:14 [I 2022-10-08 10:43:04] ...trap/engines/engine.py.229: process: 0.76786 | load: 0.00295 [I 2022-10-08 10:43:04] ...trap/engines/engine.py.230: loss: 0.24477 [I 2022-10-08 10:43:04] ...optimizers/trijoint.py.151: total_norm: 2.666275 [I 2022-10-08 10:43:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 1670/2304 [I 2022-10-08 10:43:12] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:08:07 [I 2022-10-08 10:43:12] ...trap/engines/engine.py.229: process: 0.76483 | load: 0.00294 [I 2022-10-08 10:43:12] ...trap/engines/engine.py.230: loss: 0.25990 [I 2022-10-08 10:43:12] ...optimizers/trijoint.py.151: total_norm: 2.622109 [I 2022-10-08 10:43:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 1680/2304 [I 2022-10-08 10:43:19] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:08:01 [I 2022-10-08 10:43:19] ...trap/engines/engine.py.229: process: 0.78219 | load: 0.00294 [I 2022-10-08 10:43:19] ...trap/engines/engine.py.230: loss: 0.21231 [I 2022-10-08 10:43:19] ...optimizers/trijoint.py.151: total_norm: 2.572509 [I 2022-10-08 10:43:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 1690/2304 [I 2022-10-08 10:43:27] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:07:53 [I 2022-10-08 10:43:27] ...trap/engines/engine.py.229: process: 0.77892 | load: 0.00295 [I 2022-10-08 10:43:27] ...trap/engines/engine.py.230: loss: 0.19973 [I 2022-10-08 10:43:27] ...optimizers/trijoint.py.151: total_norm: 2.735552 [I 2022-10-08 10:43:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 1700/2304 [I 2022-10-08 10:43:35] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:07:42 [I 2022-10-08 10:43:35] ...trap/engines/engine.py.229: process: 0.76813 | load: 0.00293 [I 2022-10-08 10:43:35] ...trap/engines/engine.py.230: loss: 0.21916 [I 2022-10-08 10:43:35] ...optimizers/trijoint.py.151: total_norm: 2.332360 [I 2022-10-08 10:43:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 1710/2304 [I 2022-10-08 10:43:42] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:07:38 [I 2022-10-08 10:43:42] ...trap/engines/engine.py.229: process: 0.77296 | load: 0.00293 [I 2022-10-08 10:43:42] ...trap/engines/engine.py.230: loss: 0.36374 [I 2022-10-08 10:43:42] ...optimizers/trijoint.py.151: total_norm: 3.769745 [I 2022-10-08 10:43:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 1720/2304 [I 2022-10-08 10:43:50] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:07:27 [I 2022-10-08 10:43:50] ...trap/engines/engine.py.229: process: 0.77161 | load: 0.00292 [I 2022-10-08 10:43:50] ...trap/engines/engine.py.230: loss: 0.26789 [I 2022-10-08 10:43:50] ...optimizers/trijoint.py.151: total_norm: 2.941178 [I 2022-10-08 10:43:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 1730/2304 [I 2022-10-08 10:43:58] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:07:21 [I 2022-10-08 10:43:58] ...trap/engines/engine.py.229: process: 0.77942 | load: 0.00296 [I 2022-10-08 10:43:58] ...trap/engines/engine.py.230: loss: 0.20680 [I 2022-10-08 10:43:58] ...optimizers/trijoint.py.151: total_norm: 2.226389 [I 2022-10-08 10:44:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 1740/2304 [I 2022-10-08 10:44:05] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:07:12 [I 2022-10-08 10:44:05] ...trap/engines/engine.py.229: process: 0.75281 | load: 0.00295 [I 2022-10-08 10:44:05] ...trap/engines/engine.py.230: loss: 0.28743 [I 2022-10-08 10:44:05] ...optimizers/trijoint.py.151: total_norm: 2.518680 [I 2022-10-08 10:44:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 1750/2304 [I 2022-10-08 10:44:13] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:07:02 [I 2022-10-08 10:44:13] ...trap/engines/engine.py.229: process: 0.76318 | load: 0.00296 [I 2022-10-08 10:44:13] ...trap/engines/engine.py.230: loss: 0.21410 [I 2022-10-08 10:44:13] ...optimizers/trijoint.py.151: total_norm: 3.166307 [I 2022-10-08 10:44:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 1760/2304 [I 2022-10-08 10:44:21] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:07:00 [I 2022-10-08 10:44:21] ...trap/engines/engine.py.229: process: 0.77726 | load: 0.00296 [I 2022-10-08 10:44:21] ...trap/engines/engine.py.230: loss: 0.34129 [I 2022-10-08 10:44:21] ...optimizers/trijoint.py.151: total_norm: 2.794661 [I 2022-10-08 10:44:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 1770/2304 [I 2022-10-08 10:44:29] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:06:49 [I 2022-10-08 10:44:29] ...trap/engines/engine.py.229: process: 0.76143 | load: 0.00295 [I 2022-10-08 10:44:29] ...trap/engines/engine.py.230: loss: 0.25873 [I 2022-10-08 10:44:29] ...optimizers/trijoint.py.151: total_norm: 2.871672 [I 2022-10-08 10:44:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 1780/2304 [I 2022-10-08 10:44:36] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:06:44 [I 2022-10-08 10:44:36] ...trap/engines/engine.py.229: process: 0.76923 | load: 0.00299 [I 2022-10-08 10:44:36] ...trap/engines/engine.py.230: loss: 0.44069 [I 2022-10-08 10:44:36] ...optimizers/trijoint.py.151: total_norm: 4.522678 [I 2022-10-08 10:44:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 1790/2304 [I 2022-10-08 10:44:44] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:06:34 [I 2022-10-08 10:44:44] ...trap/engines/engine.py.229: process: 0.77229 | load: 0.00295 [I 2022-10-08 10:44:44] ...trap/engines/engine.py.230: loss: 0.21887 [I 2022-10-08 10:44:44] ...optimizers/trijoint.py.151: total_norm: 2.911670 [I 2022-10-08 10:44:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 1800/2304 [I 2022-10-08 10:44:52] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:06:25 [I 2022-10-08 10:44:52] ...trap/engines/engine.py.229: process: 0.75999 | load: 0.00295 [I 2022-10-08 10:44:52] ...trap/engines/engine.py.230: loss: 0.32117 [I 2022-10-08 10:44:52] ...optimizers/trijoint.py.151: total_norm: 3.440552 [I 2022-10-08 10:44:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 1810/2304 [I 2022-10-08 10:44:59] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:06:20 [I 2022-10-08 10:44:59] ...trap/engines/engine.py.229: process: 0.76466 | load: 0.00293 [I 2022-10-08 10:44:59] ...trap/engines/engine.py.230: loss: 0.24080 [I 2022-10-08 10:44:59] ...optimizers/trijoint.py.151: total_norm: 2.254980 [I 2022-10-08 10:45:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 1820/2304 [I 2022-10-08 10:45:07] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:06:13 [I 2022-10-08 10:45:07] ...trap/engines/engine.py.229: process: 0.76737 | load: 0.00297 [I 2022-10-08 10:45:07] ...trap/engines/engine.py.230: loss: 0.22838 [I 2022-10-08 10:45:07] ...optimizers/trijoint.py.151: total_norm: 2.293022 [I 2022-10-08 10:45:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 1830/2304 [I 2022-10-08 10:45:15] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:06:05 [I 2022-10-08 10:45:15] ...trap/engines/engine.py.229: process: 0.77116 | load: 0.00295 [I 2022-10-08 10:45:15] ...trap/engines/engine.py.230: loss: 0.23944 [I 2022-10-08 10:45:15] ...optimizers/trijoint.py.151: total_norm: 3.280290 [I 2022-10-08 10:45:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 1840/2304 [I 2022-10-08 10:45:23] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:05:57 [I 2022-10-08 10:45:23] ...trap/engines/engine.py.229: process: 0.77243 | load: 0.00294 [I 2022-10-08 10:45:23] ...trap/engines/engine.py.230: loss: 0.27505 [I 2022-10-08 10:45:23] ...optimizers/trijoint.py.151: total_norm: 3.756894 [I 2022-10-08 10:45:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 1850/2304 [I 2022-10-08 10:45:30] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:05:50 [I 2022-10-08 10:45:30] ...trap/engines/engine.py.229: process: 0.77141 | load: 0.00296 [I 2022-10-08 10:45:30] ...trap/engines/engine.py.230: loss: 0.32315 [I 2022-10-08 10:45:30] ...optimizers/trijoint.py.151: total_norm: 4.169483 [I 2022-10-08 10:45:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 1860/2304 [I 2022-10-08 10:45:38] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:05:41 [I 2022-10-08 10:45:38] ...trap/engines/engine.py.229: process: 0.77393 | load: 0.00295 [I 2022-10-08 10:45:38] ...trap/engines/engine.py.230: loss: 0.36228 [I 2022-10-08 10:45:38] ...optimizers/trijoint.py.151: total_norm: 3.894237 [I 2022-10-08 10:45:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 1870/2304 [I 2022-10-08 10:45:46] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:05:34 [I 2022-10-08 10:45:46] ...trap/engines/engine.py.229: process: 0.77112 | load: 0.00296 [I 2022-10-08 10:45:46] ...trap/engines/engine.py.230: loss: 0.32521 [I 2022-10-08 10:45:46] ...optimizers/trijoint.py.151: total_norm: 3.406369 [I 2022-10-08 10:45:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 1880/2304 [I 2022-10-08 10:45:53] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:05:26 [I 2022-10-08 10:45:53] ...trap/engines/engine.py.229: process: 0.77303 | load: 0.00296 [I 2022-10-08 10:45:53] ...trap/engines/engine.py.230: loss: 0.24797 [I 2022-10-08 10:45:53] ...optimizers/trijoint.py.151: total_norm: 3.197085 [I 2022-10-08 10:46:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 1890/2304 [I 2022-10-08 10:46:01] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:05:18 [I 2022-10-08 10:46:01] ...trap/engines/engine.py.229: process: 0.77730 | load: 0.00295 [I 2022-10-08 10:46:01] ...trap/engines/engine.py.230: loss: 0.24969 [I 2022-10-08 10:46:01] ...optimizers/trijoint.py.151: total_norm: 2.996069 [I 2022-10-08 10:46:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 1900/2304 [I 2022-10-08 10:46:09] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:05:10 [I 2022-10-08 10:46:09] ...trap/engines/engine.py.229: process: 0.78031 | load: 0.00295 [I 2022-10-08 10:46:09] ...trap/engines/engine.py.230: loss: 0.30705 [I 2022-10-08 10:46:09] ...optimizers/trijoint.py.151: total_norm: 2.985791 [I 2022-10-08 10:46:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 1910/2304 [I 2022-10-08 10:46:17] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:05:03 [I 2022-10-08 10:46:17] ...trap/engines/engine.py.229: process: 0.78247 | load: 0.00295 [I 2022-10-08 10:46:17] ...trap/engines/engine.py.230: loss: 0.36959 [I 2022-10-08 10:46:17] ...optimizers/trijoint.py.151: total_norm: 3.190763 [I 2022-10-08 10:46:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 1920/2304 [I 2022-10-08 10:46:24] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:04:54 [I 2022-10-08 10:46:24] ...trap/engines/engine.py.229: process: 0.76725 | load: 0.00297 [I 2022-10-08 10:46:24] ...trap/engines/engine.py.230: loss: 0.27278 [I 2022-10-08 10:46:24] ...optimizers/trijoint.py.151: total_norm: 2.404143 [I 2022-10-08 10:46:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 1930/2304 [I 2022-10-08 10:46:32] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:04:48 [I 2022-10-08 10:46:32] ...trap/engines/engine.py.229: process: 0.78007 | load: 0.00296 [I 2022-10-08 10:46:32] ...trap/engines/engine.py.230: loss: 0.28497 [I 2022-10-08 10:46:32] ...optimizers/trijoint.py.151: total_norm: 5.098506 [I 2022-10-08 10:46:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 1940/2304 [I 2022-10-08 10:46:40] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:04:41 [I 2022-10-08 10:46:40] ...trap/engines/engine.py.229: process: 0.77127 | load: 0.00301 [I 2022-10-08 10:46:40] ...trap/engines/engine.py.230: loss: 0.39450 [I 2022-10-08 10:46:40] ...optimizers/trijoint.py.151: total_norm: 3.445476 [I 2022-10-08 10:46:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 1950/2304 [I 2022-10-08 10:46:47] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:04:32 [I 2022-10-08 10:46:47] ...trap/engines/engine.py.229: process: 0.76217 | load: 0.00293 [I 2022-10-08 10:46:47] ...trap/engines/engine.py.230: loss: 0.34898 [I 2022-10-08 10:46:47] ...optimizers/trijoint.py.151: total_norm: 2.869695 [I 2022-10-08 10:46:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 1960/2304 [I 2022-10-08 10:46:55] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:04:23 [I 2022-10-08 10:46:55] ...trap/engines/engine.py.229: process: 0.76712 | load: 0.00295 [I 2022-10-08 10:46:55] ...trap/engines/engine.py.230: loss: 0.30023 [I 2022-10-08 10:46:55] ...optimizers/trijoint.py.151: total_norm: 2.744782 [I 2022-10-08 10:47:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 1970/2304 [I 2022-10-08 10:47:03] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:04:18 [I 2022-10-08 10:47:03] ...trap/engines/engine.py.229: process: 0.78199 | load: 0.00292 [I 2022-10-08 10:47:03] ...trap/engines/engine.py.230: loss: 0.23738 [I 2022-10-08 10:47:03] ...optimizers/trijoint.py.151: total_norm: 2.919185 [I 2022-10-08 10:47:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 1980/2304 [I 2022-10-08 10:47:11] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:04:09 [I 2022-10-08 10:47:11] ...trap/engines/engine.py.229: process: 0.75155 | load: 0.00292 [I 2022-10-08 10:47:11] ...trap/engines/engine.py.230: loss: 0.20629 [I 2022-10-08 10:47:11] ...optimizers/trijoint.py.151: total_norm: 2.642969 [I 2022-10-08 10:47:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 1990/2304 [I 2022-10-08 10:47:18] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:04:01 [I 2022-10-08 10:47:18] ...trap/engines/engine.py.229: process: 0.77753 | load: 0.00294 [I 2022-10-08 10:47:18] ...trap/engines/engine.py.230: loss: 0.28503 [I 2022-10-08 10:47:18] ...optimizers/trijoint.py.151: total_norm: 2.892048 [I 2022-10-08 10:47:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 2000/2304 [I 2022-10-08 10:47:26] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:03:53 [I 2022-10-08 10:47:26] ...trap/engines/engine.py.229: process: 0.76201 | load: 0.00295 [I 2022-10-08 10:47:26] ...trap/engines/engine.py.230: loss: 0.28835 [I 2022-10-08 10:47:26] ...optimizers/trijoint.py.151: total_norm: 2.713345 [I 2022-10-08 10:47:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 2010/2304 [I 2022-10-08 10:47:34] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:03:46 [I 2022-10-08 10:47:34] ...trap/engines/engine.py.229: process: 0.77344 | load: 0.00296 [I 2022-10-08 10:47:34] ...trap/engines/engine.py.230: loss: 0.24726 [I 2022-10-08 10:47:34] ...optimizers/trijoint.py.151: total_norm: 3.135718 [I 2022-10-08 10:47:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 2020/2304 [I 2022-10-08 10:47:41] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:03:38 [I 2022-10-08 10:47:41] ...trap/engines/engine.py.229: process: 0.77001 | load: 0.00296 [I 2022-10-08 10:47:41] ...trap/engines/engine.py.230: loss: 0.26469 [I 2022-10-08 10:47:41] ...optimizers/trijoint.py.151: total_norm: 2.985326 [I 2022-10-08 10:47:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 2030/2304 [I 2022-10-08 10:47:49] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:03:31 [I 2022-10-08 10:47:49] ...trap/engines/engine.py.229: process: 0.77299 | load: 0.00295 [I 2022-10-08 10:47:49] ...trap/engines/engine.py.230: loss: 0.40112 [I 2022-10-08 10:47:49] ...optimizers/trijoint.py.151: total_norm: 3.095164 [I 2022-10-08 10:47:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 2040/2304 [I 2022-10-08 10:47:57] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:03:23 [I 2022-10-08 10:47:57] ...trap/engines/engine.py.229: process: 0.76831 | load: 0.00293 [I 2022-10-08 10:47:57] ...trap/engines/engine.py.230: loss: 0.31428 [I 2022-10-08 10:47:57] ...optimizers/trijoint.py.151: total_norm: 3.453939 [I 2022-10-08 10:48:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 2050/2304 [I 2022-10-08 10:48:05] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:03:16 [I 2022-10-08 10:48:05] ...trap/engines/engine.py.229: process: 0.77432 | load: 0.00295 [I 2022-10-08 10:48:05] ...trap/engines/engine.py.230: loss: 0.23679 [I 2022-10-08 10:48:05] ...optimizers/trijoint.py.151: total_norm: 2.589715 [I 2022-10-08 10:48:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 2060/2304 [I 2022-10-08 10:48:12] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:03:07 [I 2022-10-08 10:48:12] ...trap/engines/engine.py.229: process: 0.75756 | load: 0.00293 [I 2022-10-08 10:48:12] ...trap/engines/engine.py.230: loss: 0.33877 [I 2022-10-08 10:48:12] ...optimizers/trijoint.py.151: total_norm: 3.419848 [I 2022-10-08 10:48:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 2070/2304 [I 2022-10-08 10:48:20] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:02:58 [I 2022-10-08 10:48:20] ...trap/engines/engine.py.229: process: 0.75746 | load: 0.00295 [I 2022-10-08 10:48:20] ...trap/engines/engine.py.230: loss: 0.34754 [I 2022-10-08 10:48:20] ...optimizers/trijoint.py.151: total_norm: 3.167019 [I 2022-10-08 10:48:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 2080/2304 [I 2022-10-08 10:48:28] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:02:53 [I 2022-10-08 10:48:28] ...trap/engines/engine.py.229: process: 0.77553 | load: 0.00296 [I 2022-10-08 10:48:28] ...trap/engines/engine.py.230: loss: 0.32724 [I 2022-10-08 10:48:28] ...optimizers/trijoint.py.151: total_norm: 3.083451 [I 2022-10-08 10:48:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 2090/2304 [I 2022-10-08 10:48:35] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:02:44 [I 2022-10-08 10:48:35] ...trap/engines/engine.py.229: process: 0.76110 | load: 0.00295 [I 2022-10-08 10:48:35] ...trap/engines/engine.py.230: loss: 0.25790 [I 2022-10-08 10:48:35] ...optimizers/trijoint.py.151: total_norm: 2.525218 [I 2022-10-08 10:48:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 2100/2304 [I 2022-10-08 10:48:43] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:02:36 [I 2022-10-08 10:48:43] ...trap/engines/engine.py.229: process: 0.76855 | load: 0.00295 [I 2022-10-08 10:48:43] ...trap/engines/engine.py.230: loss: 0.37008 [I 2022-10-08 10:48:43] ...optimizers/trijoint.py.151: total_norm: 3.395500 [I 2022-10-08 10:48:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 2110/2304 [I 2022-10-08 10:48:51] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:02:28 [I 2022-10-08 10:48:51] ...trap/engines/engine.py.229: process: 0.78119 | load: 0.00296 [I 2022-10-08 10:48:51] ...trap/engines/engine.py.230: loss: 0.25933 [I 2022-10-08 10:48:51] ...optimizers/trijoint.py.151: total_norm: 2.727230 [I 2022-10-08 10:48:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 2120/2304 [I 2022-10-08 10:48:58] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:02:21 [I 2022-10-08 10:48:58] ...trap/engines/engine.py.229: process: 0.76242 | load: 0.00293 [I 2022-10-08 10:48:58] ...trap/engines/engine.py.230: loss: 0.35910 [I 2022-10-08 10:48:58] ...optimizers/trijoint.py.151: total_norm: 3.393069 [I 2022-10-08 10:49:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 2130/2304 [I 2022-10-08 10:49:06] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:02:15 [I 2022-10-08 10:49:06] ...trap/engines/engine.py.229: process: 0.79041 | load: 0.00295 [I 2022-10-08 10:49:06] ...trap/engines/engine.py.230: loss: 0.29751 [I 2022-10-08 10:49:06] ...optimizers/trijoint.py.151: total_norm: 3.281058 [I 2022-10-08 10:49:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 2140/2304 [I 2022-10-08 10:49:14] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:02:06 [I 2022-10-08 10:49:14] ...trap/engines/engine.py.229: process: 0.77502 | load: 0.00294 [I 2022-10-08 10:49:14] ...trap/engines/engine.py.230: loss: 0.23676 [I 2022-10-08 10:49:14] ...optimizers/trijoint.py.151: total_norm: 3.178521 [I 2022-10-08 10:49:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 2150/2304 [I 2022-10-08 10:49:22] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:01:59 [I 2022-10-08 10:49:22] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00294 [I 2022-10-08 10:49:22] ...trap/engines/engine.py.230: loss: 0.24756 [I 2022-10-08 10:49:22] ...optimizers/trijoint.py.151: total_norm: 2.770838 [I 2022-10-08 10:49:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 2160/2304 [I 2022-10-08 10:49:30] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:01:51 [I 2022-10-08 10:49:30] ...trap/engines/engine.py.229: process: 0.76838 | load: 0.00305 [I 2022-10-08 10:49:30] ...trap/engines/engine.py.230: loss: 0.33230 [I 2022-10-08 10:49:30] ...optimizers/trijoint.py.151: total_norm: 3.411541 [I 2022-10-08 10:49:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 2170/2304 [I 2022-10-08 10:49:37] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:01:43 [I 2022-10-08 10:49:37] ...trap/engines/engine.py.229: process: 0.77528 | load: 0.00302 [I 2022-10-08 10:49:37] ...trap/engines/engine.py.230: loss: 0.29031 [I 2022-10-08 10:49:37] ...optimizers/trijoint.py.151: total_norm: 3.013365 [I 2022-10-08 10:49:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 2180/2304 [I 2022-10-08 10:49:45] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:01:36 [I 2022-10-08 10:49:45] ...trap/engines/engine.py.229: process: 0.77996 | load: 0.00301 [I 2022-10-08 10:49:45] ...trap/engines/engine.py.230: loss: 0.30488 [I 2022-10-08 10:49:45] ...optimizers/trijoint.py.151: total_norm: 2.853091 [I 2022-10-08 10:49:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 2190/2304 [I 2022-10-08 10:49:53] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:01:28 [I 2022-10-08 10:49:53] ...trap/engines/engine.py.229: process: 0.79080 | load: 0.00300 [I 2022-10-08 10:49:53] ...trap/engines/engine.py.230: loss: 0.25743 [I 2022-10-08 10:49:53] ...optimizers/trijoint.py.151: total_norm: 2.854118 [I 2022-10-08 10:50:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 2200/2304 [I 2022-10-08 10:50:01] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:01:20 [I 2022-10-08 10:50:01] ...trap/engines/engine.py.229: process: 0.77267 | load: 0.00301 [I 2022-10-08 10:50:01] ...trap/engines/engine.py.230: loss: 0.24441 [I 2022-10-08 10:50:01] ...optimizers/trijoint.py.151: total_norm: 2.609282 [I 2022-10-08 10:50:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 2210/2304 [I 2022-10-08 10:50:08] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:01:12 [I 2022-10-08 10:50:08] ...trap/engines/engine.py.229: process: 0.77185 | load: 0.00298 [I 2022-10-08 10:50:08] ...trap/engines/engine.py.230: loss: 0.27521 [I 2022-10-08 10:50:08] ...optimizers/trijoint.py.151: total_norm: 2.982449 [I 2022-10-08 10:50:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 2220/2304 [I 2022-10-08 10:50:16] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:01:04 [I 2022-10-08 10:50:16] ...trap/engines/engine.py.229: process: 0.75610 | load: 0.00302 [I 2022-10-08 10:50:16] ...trap/engines/engine.py.230: loss: 0.31420 [I 2022-10-08 10:50:16] ...optimizers/trijoint.py.151: total_norm: 3.166276 [I 2022-10-08 10:50:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 2230/2304 [I 2022-10-08 10:50:24] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:00:56 [I 2022-10-08 10:50:24] ...trap/engines/engine.py.229: process: 0.76515 | load: 0.00304 [I 2022-10-08 10:50:24] ...trap/engines/engine.py.230: loss: 0.26293 [I 2022-10-08 10:50:24] ...optimizers/trijoint.py.151: total_norm: 2.868428 [I 2022-10-08 10:50:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 2240/2304 [I 2022-10-08 10:50:31] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:00:49 [I 2022-10-08 10:50:31] ...trap/engines/engine.py.229: process: 0.78282 | load: 0.00299 [I 2022-10-08 10:50:31] ...trap/engines/engine.py.230: loss: 0.25097 [I 2022-10-08 10:50:31] ...optimizers/trijoint.py.151: total_norm: 3.038688 [I 2022-10-08 10:50:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 2250/2304 [I 2022-10-08 10:50:39] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:00:41 [I 2022-10-08 10:50:39] ...trap/engines/engine.py.229: process: 0.76086 | load: 0.00299 [I 2022-10-08 10:50:39] ...trap/engines/engine.py.230: loss: 0.30710 [I 2022-10-08 10:50:39] ...optimizers/trijoint.py.151: total_norm: 2.902583 [I 2022-10-08 10:50:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 2260/2304 [I 2022-10-08 10:50:47] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:00:33 [I 2022-10-08 10:50:47] ...trap/engines/engine.py.229: process: 0.75746 | load: 0.00296 [I 2022-10-08 10:50:47] ...trap/engines/engine.py.230: loss: 0.34364 [I 2022-10-08 10:50:47] ...optimizers/trijoint.py.151: total_norm: 3.476581 [I 2022-10-08 10:50:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 2270/2304 [I 2022-10-08 10:50:55] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:00:26 [I 2022-10-08 10:50:55] ...trap/engines/engine.py.229: process: 0.78896 | load: 0.00304 [I 2022-10-08 10:50:55] ...trap/engines/engine.py.230: loss: 0.34193 [I 2022-10-08 10:50:55] ...optimizers/trijoint.py.151: total_norm: 3.122626 [I 2022-10-08 10:51:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 2280/2304 [I 2022-10-08 10:51:02] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:00:18 [I 2022-10-08 10:51:02] ...trap/engines/engine.py.229: process: 0.77358 | load: 0.00303 [I 2022-10-08 10:51:02] ...trap/engines/engine.py.230: loss: 0.28540 [I 2022-10-08 10:51:02] ...optimizers/trijoint.py.151: total_norm: 3.229288 [I 2022-10-08 10:51:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 2290/2304 [I 2022-10-08 10:51:10] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:00:10 [I 2022-10-08 10:51:10] ...trap/engines/engine.py.229: process: 0.75789 | load: 0.00307 [I 2022-10-08 10:51:10] ...trap/engines/engine.py.230: loss: 0.26767 [I 2022-10-08 10:51:10] ...optimizers/trijoint.py.151: total_norm: 3.164715 [I 2022-10-08 10:51:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 2300/2304 [I 2022-10-08 10:51:18] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:00:03 [I 2022-10-08 10:51:18] ...trap/engines/engine.py.229: process: 0.75128 | load: 0.00016 [I 2022-10-08 10:51:18] ...trap/engines/engine.py.230: loss: 0.32844 [I 2022-10-08 10:51:18] ...optimizers/trijoint.py.151: total_norm: 3.946430 [I 2022-10-08 10:51:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 2304/2304 [I 2022-10-08 10:51:21] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:00:00 [I 2022-10-08 10:51:21] ...trap/engines/engine.py.229: process: 0.76275 | load: 0.00017 [I 2022-10-08 10:51:21] ...trap/engines/engine.py.230: loss: 0.31368 [I 2022-10-08 10:51:21] ...optimizers/trijoint.py.151: total_norm: 2.701435 [S 2022-10-08 10:51:22] ...trap/engines/engine.py.129: train_epoch.epoch: 1 [S 2022-10-08 10:51:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.2979231657795689 [I 2022-10-08 10:51:22] ...s/criterions/triplet.py.84: margin = 0.06 [I 2022-10-08 10:51:22] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 1 [I 2022-10-08 10:51:22] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 10:51:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 10:51:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 10:51:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 10:51:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 10:51:25] ...trap/engines/engine.py.321: eval: epoch 1 | batch 0/510 [I 2022-10-08 10:51:25] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:14 [I 2022-10-08 10:51:25] ...trap/engines/engine.py.326: process: 2.61736 | load: 2.08798 [I 2022-10-08 10:51:30] ...trap/engines/engine.py.321: eval: epoch 1 | batch 10/510 [I 2022-10-08 10:51:30] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:23 [I 2022-10-08 10:51:30] ...trap/engines/engine.py.326: process: 0.51796 | load: 0.00017 [I 2022-10-08 10:51:36] ...trap/engines/engine.py.321: eval: epoch 1 | batch 20/510 [I 2022-10-08 10:51:36] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:24 [I 2022-10-08 10:51:36] ...trap/engines/engine.py.326: process: 0.65847 | load: 0.00018 [I 2022-10-08 10:51:42] ...trap/engines/engine.py.321: eval: epoch 1 | batch 30/510 [I 2022-10-08 10:51:42] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:55 [I 2022-10-08 10:51:42] ...trap/engines/engine.py.326: process: 0.75293 | load: 0.00018 [I 2022-10-08 10:51:47] ...trap/engines/engine.py.321: eval: epoch 1 | batch 40/510 [I 2022-10-08 10:51:47] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:47 [I 2022-10-08 10:51:47] ...trap/engines/engine.py.326: process: 0.41097 | load: 0.00018 [I 2022-10-08 10:51:52] ...trap/engines/engine.py.321: eval: epoch 1 | batch 50/510 [I 2022-10-08 10:51:52] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:47 [I 2022-10-08 10:51:52] ...trap/engines/engine.py.326: process: 0.51669 | load: 0.00017 [I 2022-10-08 10:51:57] ...trap/engines/engine.py.321: eval: epoch 1 | batch 60/510 [I 2022-10-08 10:51:57] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:24 [I 2022-10-08 10:51:57] ...trap/engines/engine.py.326: process: 0.46679 | load: 0.00018 [I 2022-10-08 10:52:02] ...trap/engines/engine.py.321: eval: epoch 1 | batch 70/510 [I 2022-10-08 10:52:02] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:44 [I 2022-10-08 10:52:02] ...trap/engines/engine.py.326: process: 0.56081 | load: 0.00018 [I 2022-10-08 10:52:07] ...trap/engines/engine.py.321: eval: epoch 1 | batch 80/510 [I 2022-10-08 10:52:07] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:42 [I 2022-10-08 10:52:07] ...trap/engines/engine.py.326: process: 0.50161 | load: 0.00018 [I 2022-10-08 10:52:11] ...trap/engines/engine.py.321: eval: epoch 1 | batch 90/510 [I 2022-10-08 10:52:11] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:00 [I 2022-10-08 10:52:11] ...trap/engines/engine.py.326: process: 0.41705 | load: 0.00018 [I 2022-10-08 10:52:15] ...trap/engines/engine.py.321: eval: epoch 1 | batch 100/510 [I 2022-10-08 10:52:15] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:43 [I 2022-10-08 10:52:15] ...trap/engines/engine.py.326: process: 0.31180 | load: 0.00019 [I 2022-10-08 10:52:18] ...trap/engines/engine.py.321: eval: epoch 1 | batch 110/510 [I 2022-10-08 10:52:18] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:08 [I 2022-10-08 10:52:18] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00018 [I 2022-10-08 10:52:21] ...trap/engines/engine.py.321: eval: epoch 1 | batch 120/510 [I 2022-10-08 10:52:21] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:03 [I 2022-10-08 10:52:21] ...trap/engines/engine.py.326: process: 0.31799 | load: 0.00019 [I 2022-10-08 10:52:25] ...trap/engines/engine.py.321: eval: epoch 1 | batch 130/510 [I 2022-10-08 10:52:25] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:00 [I 2022-10-08 10:52:25] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00019 [I 2022-10-08 10:52:28] ...trap/engines/engine.py.321: eval: epoch 1 | batch 140/510 [I 2022-10-08 10:52:28] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:56 [I 2022-10-08 10:52:28] ...trap/engines/engine.py.326: process: 0.31287 | load: 0.00019 [I 2022-10-08 10:52:31] ...trap/engines/engine.py.321: eval: epoch 1 | batch 150/510 [I 2022-10-08 10:52:31] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:52 [I 2022-10-08 10:52:31] ...trap/engines/engine.py.326: process: 0.31313 | load: 0.00018 [I 2022-10-08 10:52:34] ...trap/engines/engine.py.321: eval: epoch 1 | batch 160/510 [I 2022-10-08 10:52:34] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:49 [I 2022-10-08 10:52:34] ...trap/engines/engine.py.326: process: 0.31360 | load: 0.00018 [I 2022-10-08 10:52:37] ...trap/engines/engine.py.321: eval: epoch 1 | batch 170/510 [I 2022-10-08 10:52:37] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:47 [I 2022-10-08 10:52:37] ...trap/engines/engine.py.326: process: 0.31658 | load: 0.00019 [I 2022-10-08 10:52:40] ...trap/engines/engine.py.321: eval: epoch 1 | batch 180/510 [I 2022-10-08 10:52:40] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:44 [I 2022-10-08 10:52:40] ...trap/engines/engine.py.326: process: 0.31488 | load: 0.00019 [I 2022-10-08 10:52:43] ...trap/engines/engine.py.321: eval: epoch 1 | batch 190/510 [I 2022-10-08 10:52:43] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:40 [I 2022-10-08 10:52:43] ...trap/engines/engine.py.326: process: 0.30957 | load: 0.00018 [I 2022-10-08 10:52:47] ...trap/engines/engine.py.321: eval: epoch 1 | batch 200/510 [I 2022-10-08 10:52:47] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:37 [I 2022-10-08 10:52:47] ...trap/engines/engine.py.326: process: 0.31420 | load: 0.00019 [I 2022-10-08 10:52:50] ...trap/engines/engine.py.321: eval: epoch 1 | batch 210/510 [I 2022-10-08 10:52:50] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:34 [I 2022-10-08 10:52:50] ...trap/engines/engine.py.326: process: 0.30977 | load: 0.00019 [I 2022-10-08 10:52:53] ...trap/engines/engine.py.321: eval: epoch 1 | batch 220/510 [I 2022-10-08 10:52:53] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:30 [I 2022-10-08 10:52:53] ...trap/engines/engine.py.326: process: 0.31453 | load: 0.00018 [I 2022-10-08 10:52:56] ...trap/engines/engine.py.321: eval: epoch 1 | batch 230/510 [I 2022-10-08 10:52:56] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:28 [I 2022-10-08 10:52:56] ...trap/engines/engine.py.326: process: 0.31490 | load: 0.00019 [I 2022-10-08 10:52:59] ...trap/engines/engine.py.321: eval: epoch 1 | batch 240/510 [I 2022-10-08 10:52:59] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:25 [I 2022-10-08 10:52:59] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00019 [I 2022-10-08 10:53:02] ...trap/engines/engine.py.321: eval: epoch 1 | batch 250/510 [I 2022-10-08 10:53:02] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:21 [I 2022-10-08 10:53:02] ...trap/engines/engine.py.326: process: 0.30818 | load: 0.00018 [I 2022-10-08 10:53:05] ...trap/engines/engine.py.321: eval: epoch 1 | batch 260/510 [I 2022-10-08 10:53:05] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:18 [I 2022-10-08 10:53:05] ...trap/engines/engine.py.326: process: 0.31578 | load: 0.00019 [I 2022-10-08 10:53:09] ...trap/engines/engine.py.321: eval: epoch 1 | batch 270/510 [I 2022-10-08 10:53:09] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:15 [I 2022-10-08 10:53:09] ...trap/engines/engine.py.326: process: 0.31614 | load: 0.00019 [I 2022-10-08 10:53:12] ...trap/engines/engine.py.321: eval: epoch 1 | batch 280/510 [I 2022-10-08 10:53:12] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:12 [I 2022-10-08 10:53:12] ...trap/engines/engine.py.326: process: 0.31543 | load: 0.00018 [I 2022-10-08 10:53:15] ...trap/engines/engine.py.321: eval: epoch 1 | batch 290/510 [I 2022-10-08 10:53:15] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:09 [I 2022-10-08 10:53:15] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00019 [I 2022-10-08 10:53:18] ...trap/engines/engine.py.321: eval: epoch 1 | batch 300/510 [I 2022-10-08 10:53:18] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:06 [I 2022-10-08 10:53:18] ...trap/engines/engine.py.326: process: 0.31492 | load: 0.00019 [I 2022-10-08 10:53:21] ...trap/engines/engine.py.321: eval: epoch 1 | batch 310/510 [I 2022-10-08 10:53:21] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:03 [I 2022-10-08 10:53:21] ...trap/engines/engine.py.326: process: 0.31852 | load: 0.00019 [I 2022-10-08 10:53:24] ...trap/engines/engine.py.321: eval: epoch 1 | batch 320/510 [I 2022-10-08 10:53:24] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-08 10:53:24] ...trap/engines/engine.py.326: process: 0.31592 | load: 0.00019 [I 2022-10-08 10:53:28] ...trap/engines/engine.py.321: eval: epoch 1 | batch 330/510 [I 2022-10-08 10:53:28] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-08 10:53:28] ...trap/engines/engine.py.326: process: 0.31853 | load: 0.00020 [I 2022-10-08 10:53:31] ...trap/engines/engine.py.321: eval: epoch 1 | batch 340/510 [I 2022-10-08 10:53:31] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-08 10:53:31] ...trap/engines/engine.py.326: process: 0.31602 | load: 0.00018 [I 2022-10-08 10:53:34] ...trap/engines/engine.py.321: eval: epoch 1 | batch 350/510 [I 2022-10-08 10:53:34] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:50 [I 2022-10-08 10:53:34] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00020 [I 2022-10-08 10:53:37] ...trap/engines/engine.py.321: eval: epoch 1 | batch 360/510 [I 2022-10-08 10:53:37] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-08 10:53:37] ...trap/engines/engine.py.326: process: 0.32487 | load: 0.00019 [I 2022-10-08 10:53:40] ...trap/engines/engine.py.321: eval: epoch 1 | batch 370/510 [I 2022-10-08 10:53:40] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-08 10:53:40] ...trap/engines/engine.py.326: process: 0.31518 | load: 0.00019 [I 2022-10-08 10:53:43] ...trap/engines/engine.py.321: eval: epoch 1 | batch 380/510 [I 2022-10-08 10:53:43] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-08 10:53:43] ...trap/engines/engine.py.326: process: 0.31791 | load: 0.00019 [I 2022-10-08 10:53:47] ...trap/engines/engine.py.321: eval: epoch 1 | batch 390/510 [I 2022-10-08 10:53:47] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-08 10:53:47] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00019 [I 2022-10-08 10:53:50] ...trap/engines/engine.py.321: eval: epoch 1 | batch 400/510 [I 2022-10-08 10:53:50] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-08 10:53:50] ...trap/engines/engine.py.326: process: 0.31368 | load: 0.00019 [I 2022-10-08 10:53:53] ...trap/engines/engine.py.321: eval: epoch 1 | batch 410/510 [I 2022-10-08 10:53:53] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:31 [I 2022-10-08 10:53:53] ...trap/engines/engine.py.326: process: 0.31887 | load: 0.00019 [I 2022-10-08 10:53:56] ...trap/engines/engine.py.321: eval: epoch 1 | batch 420/510 [I 2022-10-08 10:53:56] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-08 10:53:56] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00019 [I 2022-10-08 10:53:59] ...trap/engines/engine.py.321: eval: epoch 1 | batch 430/510 [I 2022-10-08 10:53:59] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-08 10:53:59] ...trap/engines/engine.py.326: process: 0.32167 | load: 0.00019 [I 2022-10-08 10:54:03] ...trap/engines/engine.py.321: eval: epoch 1 | batch 440/510 [I 2022-10-08 10:54:03] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:21 [I 2022-10-08 10:54:03] ...trap/engines/engine.py.326: process: 0.30775 | load: 0.00019 [I 2022-10-08 10:54:06] ...trap/engines/engine.py.321: eval: epoch 1 | batch 450/510 [I 2022-10-08 10:54:06] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-08 10:54:06] ...trap/engines/engine.py.326: process: 0.31987 | load: 0.00020 [I 2022-10-08 10:54:09] ...trap/engines/engine.py.321: eval: epoch 1 | batch 460/510 [I 2022-10-08 10:54:09] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-08 10:54:09] ...trap/engines/engine.py.326: process: 0.31881 | load: 0.00019 [I 2022-10-08 10:54:12] ...trap/engines/engine.py.321: eval: epoch 1 | batch 470/510 [I 2022-10-08 10:54:12] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-08 10:54:12] ...trap/engines/engine.py.326: process: 0.31630 | load: 0.00019 [I 2022-10-08 10:54:15] ...trap/engines/engine.py.321: eval: epoch 1 | batch 480/510 [I 2022-10-08 10:54:15] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-08 10:54:15] ...trap/engines/engine.py.326: process: 0.31155 | load: 0.00019 [I 2022-10-08 10:54:18] ...trap/engines/engine.py.321: eval: epoch 1 | batch 490/510 [I 2022-10-08 10:54:18] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-08 10:54:18] ...trap/engines/engine.py.326: process: 0.31590 | load: 0.00020 [I 2022-10-08 10:54:21] ...trap/engines/engine.py.321: eval: epoch 1 | batch 500/510 [I 2022-10-08 10:54:21] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-08 10:54:21] ...trap/engines/engine.py.326: process: 0.31236 | load: 0.00020 [I 2022-10-08 10:54:25] ...trap/engines/engine.py.321: eval: epoch 1 | batch 510/510 [I 2022-10-08 10:54:25] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-08 10:54:25] ...trap/engines/engine.py.326: process: 0.31867 | load: 0.00015 [S 2022-10-08 10:54:25] ...trap/engines/engine.py.132: eval_epoch.epoch: 1 [I 2022-10-08 10:54:25] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.55 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.458700 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.766700 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.855600 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.497700 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.783300 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.862000 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.471699 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010140 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007128 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010336 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015957 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009477 [S 2022-10-08 10:54:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009306 [I 2022-10-08 10:54:57] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 10:54:57] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 10:54:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 10:54:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 10:54:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 10:54:57] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 10:54:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 10:55:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 10:55:04] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 10:55:04] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 10:55:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 10:55:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 10:55:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 2 [I 2022-10-08 10:55:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 10:55:14] ...trap/engines/engine.py.224: train: epoch 2 | batch 0/2304 [I 2022-10-08 10:55:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:51:28 [I 2022-10-08 10:55:14] ...trap/engines/engine.py.229: process: 2.90320 | load: 2.05236 [I 2022-10-08 10:55:14] ...trap/engines/engine.py.230: loss: 0.32432 [I 2022-10-08 10:55:14] ...optimizers/trijoint.py.151: total_norm: 3.048230 [I 2022-10-08 10:55:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 10/2304 [I 2022-10-08 10:55:22] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:26 [I 2022-10-08 10:55:22] ...trap/engines/engine.py.229: process: 0.76685 | load: 0.00600 [I 2022-10-08 10:55:22] ...trap/engines/engine.py.230: loss: 0.30773 [I 2022-10-08 10:55:22] ...optimizers/trijoint.py.151: total_norm: 3.282992 [I 2022-10-08 10:55:30] ...trap/engines/engine.py.224: train: epoch 2 | batch 20/2304 [I 2022-10-08 10:55:30] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:34 [I 2022-10-08 10:55:30] ...trap/engines/engine.py.229: process: 0.77704 | load: 0.00471 [I 2022-10-08 10:55:30] ...trap/engines/engine.py.230: loss: 0.24211 [I 2022-10-08 10:55:30] ...optimizers/trijoint.py.151: total_norm: 2.313577 [I 2022-10-08 10:55:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 30/2304 [I 2022-10-08 10:55:38] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:41 [I 2022-10-08 10:55:38] ...trap/engines/engine.py.229: process: 0.77339 | load: 0.00402 [I 2022-10-08 10:55:38] ...trap/engines/engine.py.230: loss: 0.31296 [I 2022-10-08 10:55:38] ...optimizers/trijoint.py.151: total_norm: 3.154673 [I 2022-10-08 10:55:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 40/2304 [I 2022-10-08 10:55:46] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:23 [I 2022-10-08 10:55:46] ...trap/engines/engine.py.229: process: 0.77577 | load: 0.00398 [I 2022-10-08 10:55:46] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-08 10:55:46] ...optimizers/trijoint.py.151: total_norm: 2.569711 [I 2022-10-08 10:55:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 50/2304 [I 2022-10-08 10:55:53] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:09 [I 2022-10-08 10:55:53] ...trap/engines/engine.py.229: process: 0.78305 | load: 0.00407 [I 2022-10-08 10:55:53] ...trap/engines/engine.py.230: loss: 0.26979 [I 2022-10-08 10:55:53] ...optimizers/trijoint.py.151: total_norm: 2.062364 [I 2022-10-08 10:56:01] ...trap/engines/engine.py.224: train: epoch 2 | batch 60/2304 [I 2022-10-08 10:56:01] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:04 [I 2022-10-08 10:56:01] ...trap/engines/engine.py.229: process: 0.78415 | load: 0.00352 [I 2022-10-08 10:56:01] ...trap/engines/engine.py.230: loss: 0.26002 [I 2022-10-08 10:56:01] ...optimizers/trijoint.py.151: total_norm: 2.572462 [I 2022-10-08 10:56:09] ...trap/engines/engine.py.224: train: epoch 2 | batch 70/2304 [I 2022-10-08 10:56:09] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:42 [I 2022-10-08 10:56:09] ...trap/engines/engine.py.229: process: 0.76770 | load: 0.00341 [I 2022-10-08 10:56:09] ...trap/engines/engine.py.230: loss: 0.36517 [I 2022-10-08 10:56:09] ...optimizers/trijoint.py.151: total_norm: 3.708642 [I 2022-10-08 10:56:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 80/2304 [I 2022-10-08 10:56:17] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:44 [I 2022-10-08 10:56:17] ...trap/engines/engine.py.229: process: 0.77583 | load: 0.00350 [I 2022-10-08 10:56:17] ...trap/engines/engine.py.230: loss: 0.29758 [I 2022-10-08 10:56:17] ...optimizers/trijoint.py.151: total_norm: 3.514128 [I 2022-10-08 10:56:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 90/2304 [I 2022-10-08 10:56:24] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:58 [I 2022-10-08 10:56:24] ...trap/engines/engine.py.229: process: 0.79562 | load: 0.00314 [I 2022-10-08 10:56:24] ...trap/engines/engine.py.230: loss: 0.27178 [I 2022-10-08 10:56:24] ...optimizers/trijoint.py.151: total_norm: 2.975285 [I 2022-10-08 10:56:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 100/2304 [I 2022-10-08 10:56:32] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:35 [I 2022-10-08 10:56:32] ...trap/engines/engine.py.229: process: 0.77904 | load: 0.00328 [I 2022-10-08 10:56:32] ...trap/engines/engine.py.230: loss: 0.38258 [I 2022-10-08 10:56:32] ...optimizers/trijoint.py.151: total_norm: 3.515096 [I 2022-10-08 10:56:40] ...trap/engines/engine.py.224: train: epoch 2 | batch 110/2304 [I 2022-10-08 10:56:40] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:33 [I 2022-10-08 10:56:40] ...trap/engines/engine.py.229: process: 0.78508 | load: 0.00329 [I 2022-10-08 10:56:40] ...trap/engines/engine.py.230: loss: 0.35368 [I 2022-10-08 10:56:40] ...optimizers/trijoint.py.151: total_norm: 3.906603 [I 2022-10-08 10:56:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 120/2304 [I 2022-10-08 10:56:48] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:19 [I 2022-10-08 10:56:48] ...trap/engines/engine.py.229: process: 0.78506 | load: 0.00340 [I 2022-10-08 10:56:48] ...trap/engines/engine.py.230: loss: 0.23894 [I 2022-10-08 10:56:48] ...optimizers/trijoint.py.151: total_norm: 3.230876 [I 2022-10-08 10:56:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 130/2304 [I 2022-10-08 10:56:56] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:02 [I 2022-10-08 10:56:56] ...trap/engines/engine.py.229: process: 0.76364 | load: 0.00312 [I 2022-10-08 10:56:56] ...trap/engines/engine.py.230: loss: 0.26249 [I 2022-10-08 10:56:56] ...optimizers/trijoint.py.151: total_norm: 2.949741 [I 2022-10-08 10:57:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 140/2304 [I 2022-10-08 10:57:03] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:10 [I 2022-10-08 10:57:03] ...trap/engines/engine.py.229: process: 0.79189 | load: 0.00312 [I 2022-10-08 10:57:03] ...trap/engines/engine.py.230: loss: 0.24875 [I 2022-10-08 10:57:03] ...optimizers/trijoint.py.151: total_norm: 2.873821 [I 2022-10-08 10:57:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 150/2304 [I 2022-10-08 10:57:11] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:27:47 [I 2022-10-08 10:57:11] ...trap/engines/engine.py.229: process: 0.77309 | load: 0.00325 [I 2022-10-08 10:57:11] ...trap/engines/engine.py.230: loss: 0.30320 [I 2022-10-08 10:57:11] ...optimizers/trijoint.py.151: total_norm: 2.966443 [I 2022-10-08 10:57:19] ...trap/engines/engine.py.224: train: epoch 2 | batch 160/2304 [I 2022-10-08 10:57:19] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:47 [I 2022-10-08 10:57:19] ...trap/engines/engine.py.229: process: 0.77695 | load: 0.00326 [I 2022-10-08 10:57:19] ...trap/engines/engine.py.230: loss: 0.33835 [I 2022-10-08 10:57:19] ...optimizers/trijoint.py.151: total_norm: 3.849745 [I 2022-10-08 10:57:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 170/2304 [I 2022-10-08 10:57:27] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:36 [I 2022-10-08 10:57:27] ...trap/engines/engine.py.229: process: 0.77480 | load: 0.00320 [I 2022-10-08 10:57:27] ...trap/engines/engine.py.230: loss: 0.24957 [I 2022-10-08 10:57:27] ...optimizers/trijoint.py.151: total_norm: 2.801817 [I 2022-10-08 10:57:35] ...trap/engines/engine.py.224: train: epoch 2 | batch 180/2304 [I 2022-10-08 10:57:35] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:36 [I 2022-10-08 10:57:35] ...trap/engines/engine.py.229: process: 0.78021 | load: 0.00312 [I 2022-10-08 10:57:35] ...trap/engines/engine.py.230: loss: 0.23600 [I 2022-10-08 10:57:35] ...optimizers/trijoint.py.151: total_norm: 3.659652 [I 2022-10-08 10:57:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 190/2304 [I 2022-10-08 10:57:42] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:24 [I 2022-10-08 10:57:42] ...trap/engines/engine.py.229: process: 0.78163 | load: 0.00313 [I 2022-10-08 10:57:42] ...trap/engines/engine.py.230: loss: 0.25848 [I 2022-10-08 10:57:42] ...optimizers/trijoint.py.151: total_norm: 2.669446 [I 2022-10-08 10:57:50] ...trap/engines/engine.py.224: train: epoch 2 | batch 200/2304 [I 2022-10-08 10:57:50] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:27:14 [I 2022-10-08 10:57:50] ...trap/engines/engine.py.229: process: 0.78327 | load: 0.00300 [I 2022-10-08 10:57:50] ...trap/engines/engine.py.230: loss: 0.27053 [I 2022-10-08 10:57:50] ...optimizers/trijoint.py.151: total_norm: 3.731961 [I 2022-10-08 10:57:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 210/2304 [I 2022-10-08 10:57:58] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:03 [I 2022-10-08 10:57:58] ...trap/engines/engine.py.229: process: 0.77244 | load: 0.00300 [I 2022-10-08 10:57:58] ...trap/engines/engine.py.230: loss: 0.25234 [I 2022-10-08 10:57:58] ...optimizers/trijoint.py.151: total_norm: 2.546610 [I 2022-10-08 10:58:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 220/2304 [I 2022-10-08 10:58:06] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:26:57 [I 2022-10-08 10:58:06] ...trap/engines/engine.py.229: process: 0.77581 | load: 0.00305 [I 2022-10-08 10:58:06] ...trap/engines/engine.py.230: loss: 0.23845 [I 2022-10-08 10:58:06] ...optimizers/trijoint.py.151: total_norm: 3.385564 [I 2022-10-08 10:58:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 230/2304 [I 2022-10-08 10:58:13] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:45 [I 2022-10-08 10:58:13] ...trap/engines/engine.py.229: process: 0.76316 | load: 0.00297 [I 2022-10-08 10:58:13] ...trap/engines/engine.py.230: loss: 0.17553 [I 2022-10-08 10:58:13] ...optimizers/trijoint.py.151: total_norm: 2.361920 [I 2022-10-08 10:58:21] ...trap/engines/engine.py.224: train: epoch 2 | batch 240/2304 [I 2022-10-08 10:58:21] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:26:50 [I 2022-10-08 10:58:21] ...trap/engines/engine.py.229: process: 0.78345 | load: 0.00351 [I 2022-10-08 10:58:21] ...trap/engines/engine.py.230: loss: 0.27232 [I 2022-10-08 10:58:21] ...optimizers/trijoint.py.151: total_norm: 2.971915 [I 2022-10-08 10:58:29] ...trap/engines/engine.py.224: train: epoch 2 | batch 250/2304 [I 2022-10-08 10:58:29] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:26:34 [I 2022-10-08 10:58:29] ...trap/engines/engine.py.229: process: 0.78178 | load: 0.00297 [I 2022-10-08 10:58:29] ...trap/engines/engine.py.230: loss: 0.29330 [I 2022-10-08 10:58:29] ...optimizers/trijoint.py.151: total_norm: 2.964204 [I 2022-10-08 10:58:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 260/2304 [I 2022-10-08 10:58:37] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:33 [I 2022-10-08 10:58:37] ...trap/engines/engine.py.229: process: 0.78322 | load: 0.00303 [I 2022-10-08 10:58:37] ...trap/engines/engine.py.230: loss: 0.22978 [I 2022-10-08 10:58:37] ...optimizers/trijoint.py.151: total_norm: 3.012487 [I 2022-10-08 10:58:45] ...trap/engines/engine.py.224: train: epoch 2 | batch 270/2304 [I 2022-10-08 10:58:45] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:21 [I 2022-10-08 10:58:45] ...trap/engines/engine.py.229: process: 0.77419 | load: 0.00301 [I 2022-10-08 10:58:45] ...trap/engines/engine.py.230: loss: 0.30993 [I 2022-10-08 10:58:45] ...optimizers/trijoint.py.151: total_norm: 3.460207 [I 2022-10-08 10:58:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 280/2304 [I 2022-10-08 10:58:52] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:20 [I 2022-10-08 10:58:52] ...trap/engines/engine.py.229: process: 0.78650 | load: 0.00297 [I 2022-10-08 10:58:52] ...trap/engines/engine.py.230: loss: 0.26372 [I 2022-10-08 10:58:52] ...optimizers/trijoint.py.151: total_norm: 3.710381 [I 2022-10-08 10:59:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 290/2304 [I 2022-10-08 10:59:00] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:26:16 [I 2022-10-08 10:59:00] ...trap/engines/engine.py.229: process: 0.78477 | load: 0.00301 [I 2022-10-08 10:59:00] ...trap/engines/engine.py.230: loss: 0.30286 [I 2022-10-08 10:59:00] ...optimizers/trijoint.py.151: total_norm: 3.579139 [I 2022-10-08 10:59:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 300/2304 [I 2022-10-08 10:59:08] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:26:01 [I 2022-10-08 10:59:08] ...trap/engines/engine.py.229: process: 0.76932 | load: 0.00315 [I 2022-10-08 10:59:08] ...trap/engines/engine.py.230: loss: 0.28082 [I 2022-10-08 10:59:08] ...optimizers/trijoint.py.151: total_norm: 3.430783 [I 2022-10-08 10:59:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 310/2304 [I 2022-10-08 10:59:16] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:25:56 [I 2022-10-08 10:59:16] ...trap/engines/engine.py.229: process: 0.78343 | load: 0.00303 [I 2022-10-08 10:59:16] ...trap/engines/engine.py.230: loss: 0.29589 [I 2022-10-08 10:59:16] ...optimizers/trijoint.py.151: total_norm: 3.450540 [I 2022-10-08 10:59:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 320/2304 [I 2022-10-08 10:59:24] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:48 [I 2022-10-08 10:59:24] ...trap/engines/engine.py.229: process: 0.78548 | load: 0.00304 [I 2022-10-08 10:59:24] ...trap/engines/engine.py.230: loss: 0.23279 [I 2022-10-08 10:59:24] ...optimizers/trijoint.py.151: total_norm: 2.744827 [I 2022-10-08 10:59:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 330/2304 [I 2022-10-08 10:59:32] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:35 [I 2022-10-08 10:59:32] ...trap/engines/engine.py.229: process: 0.77122 | load: 0.00312 [I 2022-10-08 10:59:32] ...trap/engines/engine.py.230: loss: 0.30714 [I 2022-10-08 10:59:32] ...optimizers/trijoint.py.151: total_norm: 3.679484 [I 2022-10-08 10:59:39] ...trap/engines/engine.py.224: train: epoch 2 | batch 340/2304 [I 2022-10-08 10:59:39] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:25:30 [I 2022-10-08 10:59:39] ...trap/engines/engine.py.229: process: 0.78125 | load: 0.00302 [I 2022-10-08 10:59:39] ...trap/engines/engine.py.230: loss: 0.33776 [I 2022-10-08 10:59:39] ...optimizers/trijoint.py.151: total_norm: 3.067071 [I 2022-10-08 10:59:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 350/2304 [I 2022-10-08 10:59:47] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:25:26 [I 2022-10-08 10:59:47] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00299 [I 2022-10-08 10:59:47] ...trap/engines/engine.py.230: loss: 0.25700 [I 2022-10-08 10:59:47] ...optimizers/trijoint.py.151: total_norm: 2.346005 [I 2022-10-08 10:59:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 360/2304 [I 2022-10-08 10:59:55] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:11 [I 2022-10-08 10:59:55] ...trap/engines/engine.py.229: process: 0.77297 | load: 0.00300 [I 2022-10-08 10:59:55] ...trap/engines/engine.py.230: loss: 0.33499 [I 2022-10-08 10:59:55] ...optimizers/trijoint.py.151: total_norm: 2.709525 [I 2022-10-08 11:00:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 370/2304 [I 2022-10-08 11:00:03] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:09 [I 2022-10-08 11:00:03] ...trap/engines/engine.py.229: process: 0.77592 | load: 0.00301 [I 2022-10-08 11:00:03] ...trap/engines/engine.py.230: loss: 0.19964 [I 2022-10-08 11:00:03] ...optimizers/trijoint.py.151: total_norm: 2.683015 [I 2022-10-08 11:00:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 380/2304 [I 2022-10-08 11:00:11] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:04 [I 2022-10-08 11:00:11] ...trap/engines/engine.py.229: process: 0.78932 | load: 0.00307 [I 2022-10-08 11:00:11] ...trap/engines/engine.py.230: loss: 0.24967 [I 2022-10-08 11:00:11] ...optimizers/trijoint.py.151: total_norm: 3.156252 [I 2022-10-08 11:00:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 390/2304 [I 2022-10-08 11:00:18] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:47 [I 2022-10-08 11:00:18] ...trap/engines/engine.py.229: process: 0.78860 | load: 0.00305 [I 2022-10-08 11:00:18] ...trap/engines/engine.py.230: loss: 0.17963 [I 2022-10-08 11:00:18] ...optimizers/trijoint.py.151: total_norm: 3.273209 [I 2022-10-08 11:00:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 400/2304 [I 2022-10-08 11:00:26] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:24:41 [I 2022-10-08 11:00:26] ...trap/engines/engine.py.229: process: 0.77958 | load: 0.00308 [I 2022-10-08 11:00:26] ...trap/engines/engine.py.230: loss: 0.29610 [I 2022-10-08 11:00:26] ...optimizers/trijoint.py.151: total_norm: 3.413190 [I 2022-10-08 11:00:34] ...trap/engines/engine.py.224: train: epoch 2 | batch 410/2304 [I 2022-10-08 11:00:34] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:24:19 [I 2022-10-08 11:00:34] ...trap/engines/engine.py.229: process: 0.76526 | load: 0.00300 [I 2022-10-08 11:00:34] ...trap/engines/engine.py.230: loss: 0.24875 [I 2022-10-08 11:00:34] ...optimizers/trijoint.py.151: total_norm: 3.540856 [I 2022-10-08 11:00:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 420/2304 [I 2022-10-08 11:00:42] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:21 [I 2022-10-08 11:00:42] ...trap/engines/engine.py.229: process: 0.78282 | load: 0.00312 [I 2022-10-08 11:00:42] ...trap/engines/engine.py.230: loss: 0.25193 [I 2022-10-08 11:00:42] ...optimizers/trijoint.py.151: total_norm: 2.273386 [I 2022-10-08 11:00:49] ...trap/engines/engine.py.224: train: epoch 2 | batch 430/2304 [I 2022-10-08 11:00:49] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:19 [I 2022-10-08 11:00:49] ...trap/engines/engine.py.229: process: 0.78549 | load: 0.00302 [I 2022-10-08 11:00:49] ...trap/engines/engine.py.230: loss: 0.18498 [I 2022-10-08 11:00:49] ...optimizers/trijoint.py.151: total_norm: 2.131913 [I 2022-10-08 11:00:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 440/2304 [I 2022-10-08 11:00:57] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:24:10 [I 2022-10-08 11:00:57] ...trap/engines/engine.py.229: process: 0.77180 | load: 0.00301 [I 2022-10-08 11:00:57] ...trap/engines/engine.py.230: loss: 0.21380 [I 2022-10-08 11:00:57] ...optimizers/trijoint.py.151: total_norm: 2.644191 [I 2022-10-08 11:01:05] ...trap/engines/engine.py.224: train: epoch 2 | batch 450/2304 [I 2022-10-08 11:01:05] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:23:56 [I 2022-10-08 11:01:05] ...trap/engines/engine.py.229: process: 0.78292 | load: 0.00304 [I 2022-10-08 11:01:05] ...trap/engines/engine.py.230: loss: 0.21222 [I 2022-10-08 11:01:05] ...optimizers/trijoint.py.151: total_norm: 3.122092 [I 2022-10-08 11:01:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 460/2304 [I 2022-10-08 11:01:13] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:23:50 [I 2022-10-08 11:01:13] ...trap/engines/engine.py.229: process: 0.77252 | load: 0.00301 [I 2022-10-08 11:01:13] ...trap/engines/engine.py.230: loss: 0.23208 [I 2022-10-08 11:01:13] ...optimizers/trijoint.py.151: total_norm: 2.950330 [I 2022-10-08 11:01:21] ...trap/engines/engine.py.224: train: epoch 2 | batch 470/2304 [I 2022-10-08 11:01:21] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:48 [I 2022-10-08 11:01:21] ...trap/engines/engine.py.229: process: 0.76730 | load: 0.00303 [I 2022-10-08 11:01:21] ...trap/engines/engine.py.230: loss: 0.26986 [I 2022-10-08 11:01:21] ...optimizers/trijoint.py.151: total_norm: 3.265139 [I 2022-10-08 11:01:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 480/2304 [I 2022-10-08 11:01:28] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:40 [I 2022-10-08 11:01:28] ...trap/engines/engine.py.229: process: 0.78177 | load: 0.00301 [I 2022-10-08 11:01:28] ...trap/engines/engine.py.230: loss: 0.30991 [I 2022-10-08 11:01:28] ...optimizers/trijoint.py.151: total_norm: 2.523873 [I 2022-10-08 11:01:36] ...trap/engines/engine.py.224: train: epoch 2 | batch 490/2304 [I 2022-10-08 11:01:36] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:23:33 [I 2022-10-08 11:01:36] ...trap/engines/engine.py.229: process: 0.78660 | load: 0.00304 [I 2022-10-08 11:01:36] ...trap/engines/engine.py.230: loss: 0.40683 [I 2022-10-08 11:01:36] ...optimizers/trijoint.py.151: total_norm: 2.853521 [I 2022-10-08 11:02:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 500/2304 [I 2022-10-08 11:02:02] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:59:26 [I 2022-10-08 11:02:02] ...trap/engines/engine.py.229: process: 0.78593 | load: 0.00305 [I 2022-10-08 11:02:02] ...trap/engines/engine.py.230: loss: 0.22033 [I 2022-10-08 11:02:02] ...optimizers/trijoint.py.151: total_norm: 2.521300 [I 2022-10-08 11:02:10] ...trap/engines/engine.py.224: train: epoch 2 | batch 510/2304 [I 2022-10-08 11:02:10] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:26:55 [I 2022-10-08 11:02:10] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00299 [I 2022-10-08 11:02:10] ...trap/engines/engine.py.230: loss: 0.35065 [I 2022-10-08 11:02:10] ...optimizers/trijoint.py.151: total_norm: 3.692525 [I 2022-10-08 11:02:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 520/2304 [I 2022-10-08 11:02:18] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:23:32 [I 2022-10-08 11:02:18] ...trap/engines/engine.py.229: process: 0.78635 | load: 0.00307 [I 2022-10-08 11:02:18] ...trap/engines/engine.py.230: loss: 0.24504 [I 2022-10-08 11:02:18] ...optimizers/trijoint.py.151: total_norm: 2.828028 [I 2022-10-08 11:02:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 530/2304 [I 2022-10-08 11:02:26] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:23:03 [I 2022-10-08 11:02:26] ...trap/engines/engine.py.229: process: 0.78033 | load: 0.00302 [I 2022-10-08 11:02:26] ...trap/engines/engine.py.230: loss: 0.33135 [I 2022-10-08 11:02:26] ...optimizers/trijoint.py.151: total_norm: 3.032760 [I 2022-10-08 11:02:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 540/2304 [I 2022-10-08 11:02:33] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:22:52 [I 2022-10-08 11:02:33] ...trap/engines/engine.py.229: process: 0.77496 | load: 0.00307 [I 2022-10-08 11:02:33] ...trap/engines/engine.py.230: loss: 0.24746 [I 2022-10-08 11:02:33] ...optimizers/trijoint.py.151: total_norm: 2.858909 [I 2022-10-08 11:02:41] ...trap/engines/engine.py.224: train: epoch 2 | batch 550/2304 [I 2022-10-08 11:02:41] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:22:51 [I 2022-10-08 11:02:41] ...trap/engines/engine.py.229: process: 0.78250 | load: 0.00303 [I 2022-10-08 11:02:41] ...trap/engines/engine.py.230: loss: 0.22826 [I 2022-10-08 11:02:41] ...optimizers/trijoint.py.151: total_norm: 2.841447 [I 2022-10-08 11:02:49] ...trap/engines/engine.py.224: train: epoch 2 | batch 560/2304 [I 2022-10-08 11:02:49] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:22:38 [I 2022-10-08 11:02:49] ...trap/engines/engine.py.229: process: 0.77580 | load: 0.00305 [I 2022-10-08 11:02:49] ...trap/engines/engine.py.230: loss: 0.27353 [I 2022-10-08 11:02:49] ...optimizers/trijoint.py.151: total_norm: 2.507111 [I 2022-10-08 11:02:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 570/2304 [I 2022-10-08 11:02:57] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:22:22 [I 2022-10-08 11:02:57] ...trap/engines/engine.py.229: process: 0.77423 | load: 0.00300 [I 2022-10-08 11:02:57] ...trap/engines/engine.py.230: loss: 0.29279 [I 2022-10-08 11:02:57] ...optimizers/trijoint.py.151: total_norm: 3.491528 [I 2022-10-08 11:03:05] ...trap/engines/engine.py.224: train: epoch 2 | batch 580/2304 [I 2022-10-08 11:03:05] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:22:23 [I 2022-10-08 11:03:05] ...trap/engines/engine.py.229: process: 0.79858 | load: 0.00306 [I 2022-10-08 11:03:05] ...trap/engines/engine.py.230: loss: 0.25871 [I 2022-10-08 11:03:05] ...optimizers/trijoint.py.151: total_norm: 2.985526 [I 2022-10-08 11:03:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 590/2304 [I 2022-10-08 11:03:12] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:22:16 [I 2022-10-08 11:03:12] ...trap/engines/engine.py.229: process: 0.77641 | load: 0.00303 [I 2022-10-08 11:03:12] ...trap/engines/engine.py.230: loss: 0.30987 [I 2022-10-08 11:03:12] ...optimizers/trijoint.py.151: total_norm: 3.356577 [I 2022-10-08 11:03:20] ...trap/engines/engine.py.224: train: epoch 2 | batch 600/2304 [I 2022-10-08 11:03:20] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:22:16 [I 2022-10-08 11:03:20] ...trap/engines/engine.py.229: process: 0.79247 | load: 0.00298 [I 2022-10-08 11:03:20] ...trap/engines/engine.py.230: loss: 0.21775 [I 2022-10-08 11:03:20] ...optimizers/trijoint.py.151: total_norm: 3.123562 [I 2022-10-08 11:03:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 610/2304 [I 2022-10-08 11:03:28] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:22:02 [I 2022-10-08 11:03:28] ...trap/engines/engine.py.229: process: 0.77754 | load: 0.00308 [I 2022-10-08 11:03:28] ...trap/engines/engine.py.230: loss: 0.23575 [I 2022-10-08 11:03:28] ...optimizers/trijoint.py.151: total_norm: 3.514812 [I 2022-10-08 11:03:36] ...trap/engines/engine.py.224: train: epoch 2 | batch 620/2304 [I 2022-10-08 11:03:36] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:21:53 [I 2022-10-08 11:03:36] ...trap/engines/engine.py.229: process: 0.78410 | load: 0.00302 [I 2022-10-08 11:03:36] ...trap/engines/engine.py.230: loss: 0.33069 [I 2022-10-08 11:03:36] ...optimizers/trijoint.py.151: total_norm: 3.133210 [I 2022-10-08 11:03:44] ...trap/engines/engine.py.224: train: epoch 2 | batch 630/2304 [I 2022-10-08 11:03:44] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:21:48 [I 2022-10-08 11:03:44] ...trap/engines/engine.py.229: process: 0.79195 | load: 0.00301 [I 2022-10-08 11:03:44] ...trap/engines/engine.py.230: loss: 0.29545 [I 2022-10-08 11:03:44] ...optimizers/trijoint.py.151: total_norm: 3.125196 [I 2022-10-08 11:03:51] ...trap/engines/engine.py.224: train: epoch 2 | batch 640/2304 [I 2022-10-08 11:03:51] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:21:31 [I 2022-10-08 11:03:51] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00304 [I 2022-10-08 11:03:51] ...trap/engines/engine.py.230: loss: 0.26042 [I 2022-10-08 11:03:51] ...optimizers/trijoint.py.151: total_norm: 2.933143 [I 2022-10-08 11:03:59] ...trap/engines/engine.py.224: train: epoch 2 | batch 650/2304 [I 2022-10-08 11:03:59] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:21:23 [I 2022-10-08 11:03:59] ...trap/engines/engine.py.229: process: 0.78345 | load: 0.00308 [I 2022-10-08 11:03:59] ...trap/engines/engine.py.230: loss: 0.23800 [I 2022-10-08 11:03:59] ...optimizers/trijoint.py.151: total_norm: 3.607974 [I 2022-10-08 11:04:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 660/2304 [I 2022-10-08 11:04:07] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:21:19 [I 2022-10-08 11:04:07] ...trap/engines/engine.py.229: process: 0.78756 | load: 0.00302 [I 2022-10-08 11:04:07] ...trap/engines/engine.py.230: loss: 0.37652 [I 2022-10-08 11:04:07] ...optimizers/trijoint.py.151: total_norm: 3.540528 [I 2022-10-08 11:04:15] ...trap/engines/engine.py.224: train: epoch 2 | batch 670/2304 [I 2022-10-08 11:04:15] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:21:12 [I 2022-10-08 11:04:15] ...trap/engines/engine.py.229: process: 0.78006 | load: 0.00304 [I 2022-10-08 11:04:15] ...trap/engines/engine.py.230: loss: 0.20797 [I 2022-10-08 11:04:15] ...optimizers/trijoint.py.151: total_norm: 2.993209 [I 2022-10-08 11:04:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 680/2304 [I 2022-10-08 11:04:23] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:21:03 [I 2022-10-08 11:04:23] ...trap/engines/engine.py.229: process: 0.77591 | load: 0.00304 [I 2022-10-08 11:04:23] ...trap/engines/engine.py.230: loss: 0.34041 [I 2022-10-08 11:04:23] ...optimizers/trijoint.py.151: total_norm: 3.422576 [I 2022-10-08 11:04:30] ...trap/engines/engine.py.224: train: epoch 2 | batch 690/2304 [I 2022-10-08 11:04:30] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:20:53 [I 2022-10-08 11:04:30] ...trap/engines/engine.py.229: process: 0.76451 | load: 0.00304 [I 2022-10-08 11:04:30] ...trap/engines/engine.py.230: loss: 0.21664 [I 2022-10-08 11:04:30] ...optimizers/trijoint.py.151: total_norm: 3.161287 [I 2022-10-08 11:04:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 700/2304 [I 2022-10-08 11:04:38] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:20:49 [I 2022-10-08 11:04:38] ...trap/engines/engine.py.229: process: 0.76739 | load: 0.00300 [I 2022-10-08 11:04:38] ...trap/engines/engine.py.230: loss: 0.35603 [I 2022-10-08 11:04:38] ...optimizers/trijoint.py.151: total_norm: 3.168080 [I 2022-10-08 11:04:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 710/2304 [I 2022-10-08 11:04:46] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:20:51 [I 2022-10-08 11:04:46] ...trap/engines/engine.py.229: process: 0.77942 | load: 0.00309 [I 2022-10-08 11:04:46] ...trap/engines/engine.py.230: loss: 0.34547 [I 2022-10-08 11:04:46] ...optimizers/trijoint.py.151: total_norm: 2.801568 [I 2022-10-08 11:04:54] ...trap/engines/engine.py.224: train: epoch 2 | batch 720/2304 [I 2022-10-08 11:04:54] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:20:23 [I 2022-10-08 11:04:54] ...trap/engines/engine.py.229: process: 0.75858 | load: 0.00302 [I 2022-10-08 11:04:54] ...trap/engines/engine.py.230: loss: 0.32459 [I 2022-10-08 11:04:54] ...optimizers/trijoint.py.151: total_norm: 3.699387 [I 2022-10-08 11:05:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 730/2304 [I 2022-10-08 11:05:02] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:20:29 [I 2022-10-08 11:05:02] ...trap/engines/engine.py.229: process: 0.79150 | load: 0.00308 [I 2022-10-08 11:05:02] ...trap/engines/engine.py.230: loss: 0.26948 [I 2022-10-08 11:05:02] ...optimizers/trijoint.py.151: total_norm: 3.060169 [I 2022-10-08 11:05:09] ...trap/engines/engine.py.224: train: epoch 2 | batch 740/2304 [I 2022-10-08 11:05:09] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:20:19 [I 2022-10-08 11:05:09] ...trap/engines/engine.py.229: process: 0.79130 | load: 0.00313 [I 2022-10-08 11:05:09] ...trap/engines/engine.py.230: loss: 0.19928 [I 2022-10-08 11:05:09] ...optimizers/trijoint.py.151: total_norm: 2.624084 [I 2022-10-08 11:05:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 750/2304 [I 2022-10-08 11:05:17] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:20:15 [I 2022-10-08 11:05:17] ...trap/engines/engine.py.229: process: 0.77643 | load: 0.00307 [I 2022-10-08 11:05:17] ...trap/engines/engine.py.230: loss: 0.21081 [I 2022-10-08 11:05:17] ...optimizers/trijoint.py.151: total_norm: 2.765651 [I 2022-10-08 11:05:25] ...trap/engines/engine.py.224: train: epoch 2 | batch 760/2304 [I 2022-10-08 11:05:25] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:20:05 [I 2022-10-08 11:05:25] ...trap/engines/engine.py.229: process: 0.77903 | load: 0.00302 [I 2022-10-08 11:05:25] ...trap/engines/engine.py.230: loss: 0.38852 [I 2022-10-08 11:05:25] ...optimizers/trijoint.py.151: total_norm: 2.967351 [I 2022-10-08 11:05:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 770/2304 [I 2022-10-08 11:05:33] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:19:53 [I 2022-10-08 11:05:33] ...trap/engines/engine.py.229: process: 0.77915 | load: 0.00302 [I 2022-10-08 11:05:33] ...trap/engines/engine.py.230: loss: 0.34182 [I 2022-10-08 11:05:33] ...optimizers/trijoint.py.151: total_norm: 3.028812 [I 2022-10-08 11:05:41] ...trap/engines/engine.py.224: train: epoch 2 | batch 780/2304 [I 2022-10-08 11:05:41] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:19:40 [I 2022-10-08 11:05:41] ...trap/engines/engine.py.229: process: 0.77967 | load: 0.00302 [I 2022-10-08 11:05:41] ...trap/engines/engine.py.230: loss: 0.33143 [I 2022-10-08 11:05:41] ...optimizers/trijoint.py.151: total_norm: 2.797189 [I 2022-10-08 11:05:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 790/2304 [I 2022-10-08 11:05:48] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:19:35 [I 2022-10-08 11:05:48] ...trap/engines/engine.py.229: process: 0.77248 | load: 0.00301 [I 2022-10-08 11:05:48] ...trap/engines/engine.py.230: loss: 0.32545 [I 2022-10-08 11:05:48] ...optimizers/trijoint.py.151: total_norm: 3.880552 [I 2022-10-08 11:05:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 800/2304 [I 2022-10-08 11:05:56] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:19:33 [I 2022-10-08 11:05:56] ...trap/engines/engine.py.229: process: 0.77538 | load: 0.00301 [I 2022-10-08 11:05:56] ...trap/engines/engine.py.230: loss: 0.20609 [I 2022-10-08 11:05:56] ...optimizers/trijoint.py.151: total_norm: 3.607396 [I 2022-10-08 11:06:04] ...trap/engines/engine.py.224: train: epoch 2 | batch 810/2304 [I 2022-10-08 11:06:04] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:19:22 [I 2022-10-08 11:06:04] ...trap/engines/engine.py.229: process: 0.77845 | load: 0.00298 [I 2022-10-08 11:06:04] ...trap/engines/engine.py.230: loss: 0.28948 [I 2022-10-08 11:06:04] ...optimizers/trijoint.py.151: total_norm: 3.278060 [I 2022-10-08 11:06:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 820/2304 [I 2022-10-08 11:06:12] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:19:10 [I 2022-10-08 11:06:12] ...trap/engines/engine.py.229: process: 0.76589 | load: 0.00300 [I 2022-10-08 11:06:12] ...trap/engines/engine.py.230: loss: 0.37535 [I 2022-10-08 11:06:12] ...optimizers/trijoint.py.151: total_norm: 4.372564 [I 2022-10-08 11:06:20] ...trap/engines/engine.py.224: train: epoch 2 | batch 830/2304 [I 2022-10-08 11:06:20] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:19:10 [I 2022-10-08 11:06:20] ...trap/engines/engine.py.229: process: 0.78029 | load: 0.00308 [I 2022-10-08 11:06:20] ...trap/engines/engine.py.230: loss: 0.29474 [I 2022-10-08 11:06:20] ...optimizers/trijoint.py.151: total_norm: 4.018648 [I 2022-10-08 11:06:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 840/2304 [I 2022-10-08 11:06:28] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:19:03 [I 2022-10-08 11:06:28] ...trap/engines/engine.py.229: process: 0.77642 | load: 0.00302 [I 2022-10-08 11:06:28] ...trap/engines/engine.py.230: loss: 0.23314 [I 2022-10-08 11:06:28] ...optimizers/trijoint.py.151: total_norm: 2.760553 [I 2022-10-08 11:06:35] ...trap/engines/engine.py.224: train: epoch 2 | batch 850/2304 [I 2022-10-08 11:06:35] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:48 [I 2022-10-08 11:06:35] ...trap/engines/engine.py.229: process: 0.77452 | load: 0.00301 [I 2022-10-08 11:06:35] ...trap/engines/engine.py.230: loss: 0.45291 [I 2022-10-08 11:06:35] ...optimizers/trijoint.py.151: total_norm: 4.229190 [I 2022-10-08 11:06:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 860/2304 [I 2022-10-08 11:06:43] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:18:44 [I 2022-10-08 11:06:43] ...trap/engines/engine.py.229: process: 0.78274 | load: 0.00305 [I 2022-10-08 11:06:43] ...trap/engines/engine.py.230: loss: 0.31280 [I 2022-10-08 11:06:43] ...optimizers/trijoint.py.151: total_norm: 2.888524 [I 2022-10-08 11:06:51] ...trap/engines/engine.py.224: train: epoch 2 | batch 870/2304 [I 2022-10-08 11:06:51] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:18:38 [I 2022-10-08 11:06:51] ...trap/engines/engine.py.229: process: 0.78676 | load: 0.00302 [I 2022-10-08 11:06:51] ...trap/engines/engine.py.230: loss: 0.26099 [I 2022-10-08 11:06:51] ...optimizers/trijoint.py.151: total_norm: 2.575719 [I 2022-10-08 11:06:59] ...trap/engines/engine.py.224: train: epoch 2 | batch 880/2304 [I 2022-10-08 11:06:59] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:18:24 [I 2022-10-08 11:06:59] ...trap/engines/engine.py.229: process: 0.78406 | load: 0.00300 [I 2022-10-08 11:06:59] ...trap/engines/engine.py.230: loss: 0.26376 [I 2022-10-08 11:06:59] ...optimizers/trijoint.py.151: total_norm: 2.645228 [I 2022-10-08 11:07:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 890/2304 [I 2022-10-08 11:07:06] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:18:14 [I 2022-10-08 11:07:06] ...trap/engines/engine.py.229: process: 0.77202 | load: 0.00298 [I 2022-10-08 11:07:06] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-08 11:07:06] ...optimizers/trijoint.py.151: total_norm: 2.529153 [I 2022-10-08 11:07:14] ...trap/engines/engine.py.224: train: epoch 2 | batch 900/2304 [I 2022-10-08 11:07:14] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:18:11 [I 2022-10-08 11:07:14] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00302 [I 2022-10-08 11:07:14] ...trap/engines/engine.py.230: loss: 0.26377 [I 2022-10-08 11:07:14] ...optimizers/trijoint.py.151: total_norm: 2.937170 [I 2022-10-08 11:07:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 910/2304 [I 2022-10-08 11:07:22] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:17:59 [I 2022-10-08 11:07:22] ...trap/engines/engine.py.229: process: 0.77513 | load: 0.00303 [I 2022-10-08 11:07:22] ...trap/engines/engine.py.230: loss: 0.27905 [I 2022-10-08 11:07:22] ...optimizers/trijoint.py.151: total_norm: 2.771277 [I 2022-10-08 11:07:30] ...trap/engines/engine.py.224: train: epoch 2 | batch 920/2304 [I 2022-10-08 11:07:30] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:17:58 [I 2022-10-08 11:07:30] ...trap/engines/engine.py.229: process: 0.77640 | load: 0.00300 [I 2022-10-08 11:07:30] ...trap/engines/engine.py.230: loss: 0.24360 [I 2022-10-08 11:07:30] ...optimizers/trijoint.py.151: total_norm: 3.302421 [I 2022-10-08 11:07:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 930/2304 [I 2022-10-08 11:07:38] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:17:49 [I 2022-10-08 11:07:38] ...trap/engines/engine.py.229: process: 0.78130 | load: 0.00303 [I 2022-10-08 11:07:38] ...trap/engines/engine.py.230: loss: 0.35787 [I 2022-10-08 11:07:38] ...optimizers/trijoint.py.151: total_norm: 4.511183 [I 2022-10-08 11:07:45] ...trap/engines/engine.py.224: train: epoch 2 | batch 940/2304 [I 2022-10-08 11:07:45] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:17:38 [I 2022-10-08 11:07:45] ...trap/engines/engine.py.229: process: 0.78806 | load: 0.00303 [I 2022-10-08 11:07:45] ...trap/engines/engine.py.230: loss: 0.29573 [I 2022-10-08 11:07:45] ...optimizers/trijoint.py.151: total_norm: 3.272716 [I 2022-10-08 11:07:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 950/2304 [I 2022-10-08 11:07:53] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:31 [I 2022-10-08 11:07:53] ...trap/engines/engine.py.229: process: 0.78837 | load: 0.00303 [I 2022-10-08 11:07:53] ...trap/engines/engine.py.230: loss: 0.25698 [I 2022-10-08 11:07:53] ...optimizers/trijoint.py.151: total_norm: 3.067811 [I 2022-10-08 11:08:01] ...trap/engines/engine.py.224: train: epoch 2 | batch 960/2304 [I 2022-10-08 11:08:01] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:17:15 [I 2022-10-08 11:08:01] ...trap/engines/engine.py.229: process: 0.75960 | load: 0.00304 [I 2022-10-08 11:08:01] ...trap/engines/engine.py.230: loss: 0.22776 [I 2022-10-08 11:08:01] ...optimizers/trijoint.py.151: total_norm: 2.665797 [I 2022-10-08 11:08:09] ...trap/engines/engine.py.224: train: epoch 2 | batch 970/2304 [I 2022-10-08 11:08:09] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:17:27 [I 2022-10-08 11:08:09] ...trap/engines/engine.py.229: process: 0.79400 | load: 0.00302 [I 2022-10-08 11:08:09] ...trap/engines/engine.py.230: loss: 0.30382 [I 2022-10-08 11:08:09] ...optimizers/trijoint.py.151: total_norm: 4.243268 [I 2022-10-08 11:08:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 980/2304 [I 2022-10-08 11:08:16] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:17:08 [I 2022-10-08 11:08:16] ...trap/engines/engine.py.229: process: 0.78535 | load: 0.00304 [I 2022-10-08 11:08:16] ...trap/engines/engine.py.230: loss: 0.25984 [I 2022-10-08 11:08:16] ...optimizers/trijoint.py.151: total_norm: 3.371410 [I 2022-10-08 11:08:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 990/2304 [I 2022-10-08 11:08:24] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:58 [I 2022-10-08 11:08:24] ...trap/engines/engine.py.229: process: 0.77268 | load: 0.00299 [I 2022-10-08 11:08:24] ...trap/engines/engine.py.230: loss: 0.24682 [I 2022-10-08 11:08:24] ...optimizers/trijoint.py.151: total_norm: 2.752000 [I 2022-10-08 11:08:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 1000/2304 [I 2022-10-08 11:08:32] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:54 [I 2022-10-08 11:08:32] ...trap/engines/engine.py.229: process: 0.77233 | load: 0.00299 [I 2022-10-08 11:08:32] ...trap/engines/engine.py.230: loss: 0.21834 [I 2022-10-08 11:08:32] ...optimizers/trijoint.py.151: total_norm: 2.481551 [I 2022-10-08 11:08:40] ...trap/engines/engine.py.224: train: epoch 2 | batch 1010/2304 [I 2022-10-08 11:08:40] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:16:45 [I 2022-10-08 11:08:40] ...trap/engines/engine.py.229: process: 0.76496 | load: 0.00302 [I 2022-10-08 11:08:40] ...trap/engines/engine.py.230: loss: 0.24329 [I 2022-10-08 11:08:40] ...optimizers/trijoint.py.151: total_norm: 3.148221 [I 2022-10-08 11:08:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1020/2304 [I 2022-10-08 11:08:48] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:16:33 [I 2022-10-08 11:08:48] ...trap/engines/engine.py.229: process: 0.77135 | load: 0.00306 [I 2022-10-08 11:08:48] ...trap/engines/engine.py.230: loss: 0.22716 [I 2022-10-08 11:08:48] ...optimizers/trijoint.py.151: total_norm: 2.669248 [I 2022-10-08 11:08:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 1030/2304 [I 2022-10-08 11:08:55] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:16:27 [I 2022-10-08 11:08:55] ...trap/engines/engine.py.229: process: 0.77170 | load: 0.00301 [I 2022-10-08 11:08:55] ...trap/engines/engine.py.230: loss: 0.24196 [I 2022-10-08 11:08:55] ...optimizers/trijoint.py.151: total_norm: 2.660264 [I 2022-10-08 11:09:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1040/2304 [I 2022-10-08 11:09:03] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:27 [I 2022-10-08 11:09:03] ...trap/engines/engine.py.229: process: 0.77143 | load: 0.00302 [I 2022-10-08 11:09:03] ...trap/engines/engine.py.230: loss: 0.29152 [I 2022-10-08 11:09:03] ...optimizers/trijoint.py.151: total_norm: 3.041441 [I 2022-10-08 11:09:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 1050/2304 [I 2022-10-08 11:09:11] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:16:18 [I 2022-10-08 11:09:11] ...trap/engines/engine.py.229: process: 0.77511 | load: 0.00301 [I 2022-10-08 11:09:11] ...trap/engines/engine.py.230: loss: 0.28281 [I 2022-10-08 11:09:11] ...optimizers/trijoint.py.151: total_norm: 3.214426 [I 2022-10-08 11:09:19] ...trap/engines/engine.py.224: train: epoch 2 | batch 1060/2304 [I 2022-10-08 11:09:19] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:16:09 [I 2022-10-08 11:09:19] ...trap/engines/engine.py.229: process: 0.77693 | load: 0.00303 [I 2022-10-08 11:09:19] ...trap/engines/engine.py.230: loss: 0.28794 [I 2022-10-08 11:09:19] ...optimizers/trijoint.py.151: total_norm: 3.716643 [I 2022-10-08 11:09:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 1070/2304 [I 2022-10-08 11:09:27] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:15:58 [I 2022-10-08 11:09:27] ...trap/engines/engine.py.229: process: 0.77350 | load: 0.00301 [I 2022-10-08 11:09:27] ...trap/engines/engine.py.230: loss: 0.24999 [I 2022-10-08 11:09:27] ...optimizers/trijoint.py.151: total_norm: 2.631810 [I 2022-10-08 11:09:34] ...trap/engines/engine.py.224: train: epoch 2 | batch 1080/2304 [I 2022-10-08 11:09:34] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:15:48 [I 2022-10-08 11:09:34] ...trap/engines/engine.py.229: process: 0.77445 | load: 0.00300 [I 2022-10-08 11:09:34] ...trap/engines/engine.py.230: loss: 0.24482 [I 2022-10-08 11:09:34] ...optimizers/trijoint.py.151: total_norm: 3.073625 [I 2022-10-08 11:09:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 1090/2304 [I 2022-10-08 11:09:42] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:15:48 [I 2022-10-08 11:09:42] ...trap/engines/engine.py.229: process: 0.78912 | load: 0.00304 [I 2022-10-08 11:09:42] ...trap/engines/engine.py.230: loss: 0.21559 [I 2022-10-08 11:09:42] ...optimizers/trijoint.py.151: total_norm: 2.222490 [I 2022-10-08 11:09:50] ...trap/engines/engine.py.224: train: epoch 2 | batch 1100/2304 [I 2022-10-08 11:09:50] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:42 [I 2022-10-08 11:09:50] ...trap/engines/engine.py.229: process: 0.78272 | load: 0.00302 [I 2022-10-08 11:09:50] ...trap/engines/engine.py.230: loss: 0.30017 [I 2022-10-08 11:09:50] ...optimizers/trijoint.py.151: total_norm: 3.191501 [I 2022-10-08 11:09:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1110/2304 [I 2022-10-08 11:09:58] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:32 [I 2022-10-08 11:09:58] ...trap/engines/engine.py.229: process: 0.78426 | load: 0.00301 [I 2022-10-08 11:09:58] ...trap/engines/engine.py.230: loss: 0.35274 [I 2022-10-08 11:09:58] ...optimizers/trijoint.py.151: total_norm: 3.966016 [I 2022-10-08 11:10:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 1120/2304 [I 2022-10-08 11:10:06] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:15:26 [I 2022-10-08 11:10:06] ...trap/engines/engine.py.229: process: 0.79237 | load: 0.00303 [I 2022-10-08 11:10:06] ...trap/engines/engine.py.230: loss: 0.18939 [I 2022-10-08 11:10:06] ...optimizers/trijoint.py.151: total_norm: 3.584695 [I 2022-10-08 11:10:14] ...trap/engines/engine.py.224: train: epoch 2 | batch 1130/2304 [I 2022-10-08 11:10:14] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:15:13 [I 2022-10-08 11:10:14] ...trap/engines/engine.py.229: process: 0.77998 | load: 0.00302 [I 2022-10-08 11:10:14] ...trap/engines/engine.py.230: loss: 0.24122 [I 2022-10-08 11:10:14] ...optimizers/trijoint.py.151: total_norm: 3.865848 [I 2022-10-08 11:10:21] ...trap/engines/engine.py.224: train: epoch 2 | batch 1140/2304 [I 2022-10-08 11:10:21] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:15:06 [I 2022-10-08 11:10:21] ...trap/engines/engine.py.229: process: 0.77605 | load: 0.00314 [I 2022-10-08 11:10:21] ...trap/engines/engine.py.230: loss: 0.19945 [I 2022-10-08 11:10:21] ...optimizers/trijoint.py.151: total_norm: 2.899944 [I 2022-10-08 11:10:29] ...trap/engines/engine.py.224: train: epoch 2 | batch 1150/2304 [I 2022-10-08 11:10:29] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:59 [I 2022-10-08 11:10:29] ...trap/engines/engine.py.229: process: 0.78256 | load: 0.00302 [I 2022-10-08 11:10:29] ...trap/engines/engine.py.230: loss: 0.22930 [I 2022-10-08 11:10:29] ...optimizers/trijoint.py.151: total_norm: 3.434195 [I 2022-10-08 11:10:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 1160/2304 [I 2022-10-08 11:10:37] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:14:49 [I 2022-10-08 11:10:37] ...trap/engines/engine.py.229: process: 0.77024 | load: 0.00305 [I 2022-10-08 11:10:37] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-08 11:10:37] ...optimizers/trijoint.py.151: total_norm: 2.595710 [I 2022-10-08 11:10:45] ...trap/engines/engine.py.224: train: epoch 2 | batch 1170/2304 [I 2022-10-08 11:10:45] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:14:46 [I 2022-10-08 11:10:45] ...trap/engines/engine.py.229: process: 0.78931 | load: 0.00311 [I 2022-10-08 11:10:45] ...trap/engines/engine.py.230: loss: 0.30899 [I 2022-10-08 11:10:45] ...optimizers/trijoint.py.151: total_norm: 3.280843 [I 2022-10-08 11:10:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1180/2304 [I 2022-10-08 11:10:53] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:14:39 [I 2022-10-08 11:10:53] ...trap/engines/engine.py.229: process: 0.78209 | load: 0.00305 [I 2022-10-08 11:10:53] ...trap/engines/engine.py.230: loss: 0.27153 [I 2022-10-08 11:10:53] ...optimizers/trijoint.py.151: total_norm: 2.978572 [I 2022-10-08 11:11:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 1190/2304 [I 2022-10-08 11:11:01] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:14:32 [I 2022-10-08 11:11:01] ...trap/engines/engine.py.229: process: 0.78656 | load: 0.00304 [I 2022-10-08 11:11:01] ...trap/engines/engine.py.230: loss: 0.21651 [I 2022-10-08 11:11:01] ...optimizers/trijoint.py.151: total_norm: 3.175461 [I 2022-10-08 11:11:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1200/2304 [I 2022-10-08 11:11:08] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:14:24 [I 2022-10-08 11:11:08] ...trap/engines/engine.py.229: process: 0.78740 | load: 0.00301 [I 2022-10-08 11:11:08] ...trap/engines/engine.py.230: loss: 0.37558 [I 2022-10-08 11:11:08] ...optimizers/trijoint.py.151: total_norm: 3.632921 [I 2022-10-08 11:11:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 1210/2304 [I 2022-10-08 11:11:16] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:14:16 [I 2022-10-08 11:11:16] ...trap/engines/engine.py.229: process: 0.78132 | load: 0.00303 [I 2022-10-08 11:11:16] ...trap/engines/engine.py.230: loss: 0.21509 [I 2022-10-08 11:11:16] ...optimizers/trijoint.py.151: total_norm: 3.048653 [I 2022-10-08 11:11:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 1220/2304 [I 2022-10-08 11:11:24] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:14:05 [I 2022-10-08 11:11:24] ...trap/engines/engine.py.229: process: 0.77872 | load: 0.00306 [I 2022-10-08 11:11:24] ...trap/engines/engine.py.230: loss: 0.27405 [I 2022-10-08 11:11:24] ...optimizers/trijoint.py.151: total_norm: 3.668587 [I 2022-10-08 11:11:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 1230/2304 [I 2022-10-08 11:11:32] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:14:00 [I 2022-10-08 11:11:32] ...trap/engines/engine.py.229: process: 0.78328 | load: 0.00303 [I 2022-10-08 11:11:32] ...trap/engines/engine.py.230: loss: 0.34453 [I 2022-10-08 11:11:32] ...optimizers/trijoint.py.151: total_norm: 3.114943 [I 2022-10-08 11:11:40] ...trap/engines/engine.py.224: train: epoch 2 | batch 1240/2304 [I 2022-10-08 11:11:40] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:13:45 [I 2022-10-08 11:11:40] ...trap/engines/engine.py.229: process: 0.77377 | load: 0.00303 [I 2022-10-08 11:11:40] ...trap/engines/engine.py.230: loss: 0.28745 [I 2022-10-08 11:11:40] ...optimizers/trijoint.py.151: total_norm: 2.953001 [I 2022-10-08 11:11:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 1250/2304 [I 2022-10-08 11:11:47] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:13:37 [I 2022-10-08 11:11:47] ...trap/engines/engine.py.229: process: 0.77738 | load: 0.00309 [I 2022-10-08 11:11:47] ...trap/engines/engine.py.230: loss: 0.31178 [I 2022-10-08 11:11:47] ...optimizers/trijoint.py.151: total_norm: 2.952569 [I 2022-10-08 11:11:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 1260/2304 [I 2022-10-08 11:11:55] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:13:34 [I 2022-10-08 11:11:55] ...trap/engines/engine.py.229: process: 0.77688 | load: 0.00307 [I 2022-10-08 11:11:55] ...trap/engines/engine.py.230: loss: 0.23322 [I 2022-10-08 11:11:55] ...optimizers/trijoint.py.151: total_norm: 3.353498 [I 2022-10-08 11:12:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1270/2304 [I 2022-10-08 11:12:03] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:13:29 [I 2022-10-08 11:12:03] ...trap/engines/engine.py.229: process: 0.77798 | load: 0.00313 [I 2022-10-08 11:12:03] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-08 11:12:03] ...optimizers/trijoint.py.151: total_norm: 2.107598 [I 2022-10-08 11:12:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 1280/2304 [I 2022-10-08 11:12:11] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:13:13 [I 2022-10-08 11:12:11] ...trap/engines/engine.py.229: process: 0.78467 | load: 0.00314 [I 2022-10-08 11:12:11] ...trap/engines/engine.py.230: loss: 0.27466 [I 2022-10-08 11:12:11] ...optimizers/trijoint.py.151: total_norm: 3.234024 [I 2022-10-08 11:12:19] ...trap/engines/engine.py.224: train: epoch 2 | batch 1290/2304 [I 2022-10-08 11:12:19] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:13:12 [I 2022-10-08 11:12:19] ...trap/engines/engine.py.229: process: 0.78243 | load: 0.00302 [I 2022-10-08 11:12:19] ...trap/engines/engine.py.230: loss: 0.35655 [I 2022-10-08 11:12:19] ...optimizers/trijoint.py.151: total_norm: 4.089187 [I 2022-10-08 11:12:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 1300/2304 [I 2022-10-08 11:12:26] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:13:00 [I 2022-10-08 11:12:26] ...trap/engines/engine.py.229: process: 0.76483 | load: 0.00300 [I 2022-10-08 11:12:26] ...trap/engines/engine.py.230: loss: 0.21502 [I 2022-10-08 11:12:26] ...optimizers/trijoint.py.151: total_norm: 3.481872 [I 2022-10-08 11:12:34] ...trap/engines/engine.py.224: train: epoch 2 | batch 1310/2304 [I 2022-10-08 11:12:34] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:50 [I 2022-10-08 11:12:34] ...trap/engines/engine.py.229: process: 0.76924 | load: 0.00302 [I 2022-10-08 11:12:34] ...trap/engines/engine.py.230: loss: 0.21743 [I 2022-10-08 11:12:34] ...optimizers/trijoint.py.151: total_norm: 2.441091 [I 2022-10-08 11:12:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 1320/2304 [I 2022-10-08 11:12:42] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:12:47 [I 2022-10-08 11:12:42] ...trap/engines/engine.py.229: process: 0.76750 | load: 0.00305 [I 2022-10-08 11:12:42] ...trap/engines/engine.py.230: loss: 0.34527 [I 2022-10-08 11:12:42] ...optimizers/trijoint.py.151: total_norm: 3.640834 [I 2022-10-08 11:12:50] ...trap/engines/engine.py.224: train: epoch 2 | batch 1330/2304 [I 2022-10-08 11:12:50] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:12:35 [I 2022-10-08 11:12:50] ...trap/engines/engine.py.229: process: 0.77134 | load: 0.00304 [I 2022-10-08 11:12:50] ...trap/engines/engine.py.230: loss: 0.28024 [I 2022-10-08 11:12:50] ...optimizers/trijoint.py.151: total_norm: 2.620898 [I 2022-10-08 11:12:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1340/2304 [I 2022-10-08 11:12:58] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:12:27 [I 2022-10-08 11:12:58] ...trap/engines/engine.py.229: process: 0.77255 | load: 0.00301 [I 2022-10-08 11:12:58] ...trap/engines/engine.py.230: loss: 0.17719 [I 2022-10-08 11:12:58] ...optimizers/trijoint.py.151: total_norm: 2.562634 [I 2022-10-08 11:13:05] ...trap/engines/engine.py.224: train: epoch 2 | batch 1350/2304 [I 2022-10-08 11:13:05] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:12:19 [I 2022-10-08 11:13:05] ...trap/engines/engine.py.229: process: 0.77045 | load: 0.00302 [I 2022-10-08 11:13:05] ...trap/engines/engine.py.230: loss: 0.34136 [I 2022-10-08 11:13:05] ...optimizers/trijoint.py.151: total_norm: 3.824076 [I 2022-10-08 11:13:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1360/2304 [I 2022-10-08 11:13:13] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:10 [I 2022-10-08 11:13:13] ...trap/engines/engine.py.229: process: 0.76901 | load: 0.00302 [I 2022-10-08 11:13:13] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-08 11:13:13] ...optimizers/trijoint.py.151: total_norm: 2.057183 [I 2022-10-08 11:13:21] ...trap/engines/engine.py.224: train: epoch 2 | batch 1370/2304 [I 2022-10-08 11:13:21] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:12:07 [I 2022-10-08 11:13:21] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00310 [I 2022-10-08 11:13:21] ...trap/engines/engine.py.230: loss: 0.38579 [I 2022-10-08 11:13:21] ...optimizers/trijoint.py.151: total_norm: 3.875633 [I 2022-10-08 11:13:29] ...trap/engines/engine.py.224: train: epoch 2 | batch 1380/2304 [I 2022-10-08 11:13:29] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:12:01 [I 2022-10-08 11:13:29] ...trap/engines/engine.py.229: process: 0.77197 | load: 0.00303 [I 2022-10-08 11:13:29] ...trap/engines/engine.py.230: loss: 0.31528 [I 2022-10-08 11:13:29] ...optimizers/trijoint.py.151: total_norm: 3.888536 [I 2022-10-08 11:13:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 1390/2304 [I 2022-10-08 11:13:37] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:11:53 [I 2022-10-08 11:13:37] ...trap/engines/engine.py.229: process: 0.77607 | load: 0.00300 [I 2022-10-08 11:13:37] ...trap/engines/engine.py.230: loss: 0.26086 [I 2022-10-08 11:13:37] ...optimizers/trijoint.py.151: total_norm: 3.022710 [I 2022-10-08 11:13:44] ...trap/engines/engine.py.224: train: epoch 2 | batch 1400/2304 [I 2022-10-08 11:13:44] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:11:42 [I 2022-10-08 11:13:44] ...trap/engines/engine.py.229: process: 0.77650 | load: 0.00300 [I 2022-10-08 11:13:44] ...trap/engines/engine.py.230: loss: 0.31353 [I 2022-10-08 11:13:44] ...optimizers/trijoint.py.151: total_norm: 3.327775 [I 2022-10-08 11:13:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 1410/2304 [I 2022-10-08 11:13:52] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:33 [I 2022-10-08 11:13:52] ...trap/engines/engine.py.229: process: 0.76907 | load: 0.00299 [I 2022-10-08 11:13:52] ...trap/engines/engine.py.230: loss: 0.23081 [I 2022-10-08 11:13:52] ...optimizers/trijoint.py.151: total_norm: 2.095259 [I 2022-10-08 11:14:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 1420/2304 [I 2022-10-08 11:14:00] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:27 [I 2022-10-08 11:14:00] ...trap/engines/engine.py.229: process: 0.77803 | load: 0.00302 [I 2022-10-08 11:14:00] ...trap/engines/engine.py.230: loss: 0.22319 [I 2022-10-08 11:14:00] ...optimizers/trijoint.py.151: total_norm: 2.609342 [I 2022-10-08 11:14:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1430/2304 [I 2022-10-08 11:14:08] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:13 [I 2022-10-08 11:14:08] ...trap/engines/engine.py.229: process: 0.76043 | load: 0.00301 [I 2022-10-08 11:14:08] ...trap/engines/engine.py.230: loss: 0.23685 [I 2022-10-08 11:14:08] ...optimizers/trijoint.py.151: total_norm: 2.901147 [I 2022-10-08 11:14:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 1440/2304 [I 2022-10-08 11:14:16] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:11:11 [I 2022-10-08 11:14:16] ...trap/engines/engine.py.229: process: 0.78473 | load: 0.00301 [I 2022-10-08 11:14:16] ...trap/engines/engine.py.230: loss: 0.26938 [I 2022-10-08 11:14:16] ...optimizers/trijoint.py.151: total_norm: 3.815273 [I 2022-10-08 11:14:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1450/2304 [I 2022-10-08 11:14:23] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:11:00 [I 2022-10-08 11:14:23] ...trap/engines/engine.py.229: process: 0.76647 | load: 0.00306 [I 2022-10-08 11:14:23] ...trap/engines/engine.py.230: loss: 0.34490 [I 2022-10-08 11:14:23] ...optimizers/trijoint.py.151: total_norm: 3.116537 [I 2022-10-08 11:14:31] ...trap/engines/engine.py.224: train: epoch 2 | batch 1460/2304 [I 2022-10-08 11:14:31] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:56 [I 2022-10-08 11:14:31] ...trap/engines/engine.py.229: process: 0.77988 | load: 0.00303 [I 2022-10-08 11:14:31] ...trap/engines/engine.py.230: loss: 0.37486 [I 2022-10-08 11:14:31] ...optimizers/trijoint.py.151: total_norm: 3.578404 [I 2022-10-08 11:14:39] ...trap/engines/engine.py.224: train: epoch 2 | batch 1470/2304 [I 2022-10-08 11:14:39] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:49 [I 2022-10-08 11:14:39] ...trap/engines/engine.py.229: process: 0.78830 | load: 0.00299 [I 2022-10-08 11:14:39] ...trap/engines/engine.py.230: loss: 0.32525 [I 2022-10-08 11:14:39] ...optimizers/trijoint.py.151: total_norm: 3.400290 [I 2022-10-08 11:14:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 1480/2304 [I 2022-10-08 11:14:47] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:44 [I 2022-10-08 11:14:47] ...trap/engines/engine.py.229: process: 0.79461 | load: 0.00303 [I 2022-10-08 11:14:47] ...trap/engines/engine.py.230: loss: 0.20605 [I 2022-10-08 11:14:47] ...optimizers/trijoint.py.151: total_norm: 3.096778 [I 2022-10-08 11:14:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 1490/2304 [I 2022-10-08 11:14:55] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:36 [I 2022-10-08 11:14:55] ...trap/engines/engine.py.229: process: 0.78154 | load: 0.00304 [I 2022-10-08 11:14:55] ...trap/engines/engine.py.230: loss: 0.36063 [I 2022-10-08 11:14:55] ...optimizers/trijoint.py.151: total_norm: 3.356728 [I 2022-10-08 11:15:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 1500/2304 [I 2022-10-08 11:15:02] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:27 [I 2022-10-08 11:15:02] ...trap/engines/engine.py.229: process: 0.78558 | load: 0.00304 [I 2022-10-08 11:15:02] ...trap/engines/engine.py.230: loss: 0.21487 [I 2022-10-08 11:15:02] ...optimizers/trijoint.py.151: total_norm: 3.053267 [I 2022-10-08 11:15:10] ...trap/engines/engine.py.224: train: epoch 2 | batch 1510/2304 [I 2022-10-08 11:15:10] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:20 [I 2022-10-08 11:15:10] ...trap/engines/engine.py.229: process: 0.79100 | load: 0.00302 [I 2022-10-08 11:15:10] ...trap/engines/engine.py.230: loss: 0.26812 [I 2022-10-08 11:15:10] ...optimizers/trijoint.py.151: total_norm: 3.001114 [I 2022-10-08 11:15:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1520/2304 [I 2022-10-08 11:15:18] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:10:10 [I 2022-10-08 11:15:18] ...trap/engines/engine.py.229: process: 0.76816 | load: 0.00302 [I 2022-10-08 11:15:18] ...trap/engines/engine.py.230: loss: 0.23294 [I 2022-10-08 11:15:18] ...optimizers/trijoint.py.151: total_norm: 2.544926 [I 2022-10-08 11:15:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 1530/2304 [I 2022-10-08 11:15:26] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:10:00 [I 2022-10-08 11:15:26] ...trap/engines/engine.py.229: process: 0.77144 | load: 0.00313 [I 2022-10-08 11:15:26] ...trap/engines/engine.py.230: loss: 0.29360 [I 2022-10-08 11:15:26] ...optimizers/trijoint.py.151: total_norm: 4.324871 [I 2022-10-08 11:15:34] ...trap/engines/engine.py.224: train: epoch 2 | batch 1540/2304 [I 2022-10-08 11:15:34] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:50 [I 2022-10-08 11:15:34] ...trap/engines/engine.py.229: process: 0.75931 | load: 0.00309 [I 2022-10-08 11:15:34] ...trap/engines/engine.py.230: loss: 0.30374 [I 2022-10-08 11:15:34] ...optimizers/trijoint.py.151: total_norm: 3.029193 [I 2022-10-08 11:15:41] ...trap/engines/engine.py.224: train: epoch 2 | batch 1550/2304 [I 2022-10-08 11:15:41] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:47 [I 2022-10-08 11:15:41] ...trap/engines/engine.py.229: process: 0.78379 | load: 0.00308 [I 2022-10-08 11:15:41] ...trap/engines/engine.py.230: loss: 0.34844 [I 2022-10-08 11:15:41] ...optimizers/trijoint.py.151: total_norm: 3.304117 [I 2022-10-08 11:15:49] ...trap/engines/engine.py.224: train: epoch 2 | batch 1560/2304 [I 2022-10-08 11:15:49] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:40 [I 2022-10-08 11:15:49] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00313 [I 2022-10-08 11:15:49] ...trap/engines/engine.py.230: loss: 0.22881 [I 2022-10-08 11:15:49] ...optimizers/trijoint.py.151: total_norm: 2.753980 [I 2022-10-08 11:15:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 1570/2304 [I 2022-10-08 11:15:57] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:30 [I 2022-10-08 11:15:57] ...trap/engines/engine.py.229: process: 0.78553 | load: 0.00315 [I 2022-10-08 11:15:57] ...trap/engines/engine.py.230: loss: 0.21971 [I 2022-10-08 11:15:57] ...optimizers/trijoint.py.151: total_norm: 2.833839 [I 2022-10-08 11:16:05] ...trap/engines/engine.py.224: train: epoch 2 | batch 1580/2304 [I 2022-10-08 11:16:05] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:26 [I 2022-10-08 11:16:05] ...trap/engines/engine.py.229: process: 0.77961 | load: 0.00309 [I 2022-10-08 11:16:05] ...trap/engines/engine.py.230: loss: 0.18728 [I 2022-10-08 11:16:05] ...optimizers/trijoint.py.151: total_norm: 2.438218 [I 2022-10-08 11:16:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1590/2304 [I 2022-10-08 11:16:13] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:09:21 [I 2022-10-08 11:16:13] ...trap/engines/engine.py.229: process: 0.77948 | load: 0.00313 [I 2022-10-08 11:16:13] ...trap/engines/engine.py.230: loss: 0.24113 [I 2022-10-08 11:16:13] ...optimizers/trijoint.py.151: total_norm: 2.940458 [I 2022-10-08 11:16:20] ...trap/engines/engine.py.224: train: epoch 2 | batch 1600/2304 [I 2022-10-08 11:16:20] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:09:09 [I 2022-10-08 11:16:20] ...trap/engines/engine.py.229: process: 0.78806 | load: 0.00308 [I 2022-10-08 11:16:20] ...trap/engines/engine.py.230: loss: 0.29604 [I 2022-10-08 11:16:20] ...optimizers/trijoint.py.151: total_norm: 4.207033 [I 2022-10-08 11:16:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1610/2304 [I 2022-10-08 11:16:28] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:09:03 [I 2022-10-08 11:16:28] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00317 [I 2022-10-08 11:16:28] ...trap/engines/engine.py.230: loss: 0.22805 [I 2022-10-08 11:16:28] ...optimizers/trijoint.py.151: total_norm: 2.621467 [I 2022-10-08 11:16:36] ...trap/engines/engine.py.224: train: epoch 2 | batch 1620/2304 [I 2022-10-08 11:16:36] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:53 [I 2022-10-08 11:16:36] ...trap/engines/engine.py.229: process: 0.78476 | load: 0.00312 [I 2022-10-08 11:16:36] ...trap/engines/engine.py.230: loss: 0.36317 [I 2022-10-08 11:16:36] ...optimizers/trijoint.py.151: total_norm: 2.914503 [I 2022-10-08 11:16:44] ...trap/engines/engine.py.224: train: epoch 2 | batch 1630/2304 [I 2022-10-08 11:16:44] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:44 [I 2022-10-08 11:16:44] ...trap/engines/engine.py.229: process: 0.77534 | load: 0.00310 [I 2022-10-08 11:16:44] ...trap/engines/engine.py.230: loss: 0.21407 [I 2022-10-08 11:16:44] ...optimizers/trijoint.py.151: total_norm: 2.755667 [I 2022-10-08 11:16:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 1640/2304 [I 2022-10-08 11:16:52] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:37 [I 2022-10-08 11:16:52] ...trap/engines/engine.py.229: process: 0.77250 | load: 0.00316 [I 2022-10-08 11:16:52] ...trap/engines/engine.py.230: loss: 0.27067 [I 2022-10-08 11:16:52] ...optimizers/trijoint.py.151: total_norm: 2.360603 [I 2022-10-08 11:17:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 1650/2304 [I 2022-10-08 11:17:00] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:31 [I 2022-10-08 11:17:00] ...trap/engines/engine.py.229: process: 0.78001 | load: 0.00303 [I 2022-10-08 11:17:00] ...trap/engines/engine.py.230: loss: 0.27077 [I 2022-10-08 11:17:00] ...optimizers/trijoint.py.151: total_norm: 2.916369 [I 2022-10-08 11:17:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1660/2304 [I 2022-10-08 11:17:07] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:08:22 [I 2022-10-08 11:17:07] ...trap/engines/engine.py.229: process: 0.77579 | load: 0.00301 [I 2022-10-08 11:17:07] ...trap/engines/engine.py.230: loss: 0.28258 [I 2022-10-08 11:17:07] ...optimizers/trijoint.py.151: total_norm: 3.184163 [I 2022-10-08 11:17:15] ...trap/engines/engine.py.224: train: epoch 2 | batch 1670/2304 [I 2022-10-08 11:17:15] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:08:13 [I 2022-10-08 11:17:15] ...trap/engines/engine.py.229: process: 0.78491 | load: 0.00304 [I 2022-10-08 11:17:15] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-08 11:17:15] ...optimizers/trijoint.py.151: total_norm: 2.600616 [I 2022-10-08 11:17:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1680/2304 [I 2022-10-08 11:17:23] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:08:07 [I 2022-10-08 11:17:23] ...trap/engines/engine.py.229: process: 0.78860 | load: 0.00305 [I 2022-10-08 11:17:23] ...trap/engines/engine.py.230: loss: 0.32360 [I 2022-10-08 11:17:23] ...optimizers/trijoint.py.151: total_norm: 3.214609 [I 2022-10-08 11:17:31] ...trap/engines/engine.py.224: train: epoch 2 | batch 1690/2304 [I 2022-10-08 11:17:31] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:54 [I 2022-10-08 11:17:31] ...trap/engines/engine.py.229: process: 0.77255 | load: 0.00302 [I 2022-10-08 11:17:31] ...trap/engines/engine.py.230: loss: 0.17492 [I 2022-10-08 11:17:31] ...optimizers/trijoint.py.151: total_norm: 2.390772 [I 2022-10-08 11:17:39] ...trap/engines/engine.py.224: train: epoch 2 | batch 1700/2304 [I 2022-10-08 11:17:39] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:07:48 [I 2022-10-08 11:17:39] ...trap/engines/engine.py.229: process: 0.77210 | load: 0.00301 [I 2022-10-08 11:17:39] ...trap/engines/engine.py.230: loss: 0.29019 [I 2022-10-08 11:17:39] ...optimizers/trijoint.py.151: total_norm: 3.341916 [I 2022-10-08 11:17:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 1710/2304 [I 2022-10-08 11:17:46] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:07:43 [I 2022-10-08 11:17:46] ...trap/engines/engine.py.229: process: 0.77781 | load: 0.00298 [I 2022-10-08 11:17:46] ...trap/engines/engine.py.230: loss: 0.28764 [I 2022-10-08 11:17:46] ...optimizers/trijoint.py.151: total_norm: 3.361688 [I 2022-10-08 11:17:54] ...trap/engines/engine.py.224: train: epoch 2 | batch 1720/2304 [I 2022-10-08 11:17:54] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:34 [I 2022-10-08 11:17:54] ...trap/engines/engine.py.229: process: 0.78457 | load: 0.00301 [I 2022-10-08 11:17:54] ...trap/engines/engine.py.230: loss: 0.33872 [I 2022-10-08 11:17:54] ...optimizers/trijoint.py.151: total_norm: 3.630428 [I 2022-10-08 11:18:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 1730/2304 [I 2022-10-08 11:18:02] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:27 [I 2022-10-08 11:18:02] ...trap/engines/engine.py.229: process: 0.78902 | load: 0.00301 [I 2022-10-08 11:18:02] ...trap/engines/engine.py.230: loss: 0.31694 [I 2022-10-08 11:18:02] ...optimizers/trijoint.py.151: total_norm: 2.936587 [I 2022-10-08 11:18:10] ...trap/engines/engine.py.224: train: epoch 2 | batch 1740/2304 [I 2022-10-08 11:18:10] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:17 [I 2022-10-08 11:18:10] ...trap/engines/engine.py.229: process: 0.78050 | load: 0.00301 [I 2022-10-08 11:18:10] ...trap/engines/engine.py.230: loss: 0.34828 [I 2022-10-08 11:18:10] ...optimizers/trijoint.py.151: total_norm: 4.186541 [I 2022-10-08 11:18:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 1750/2304 [I 2022-10-08 11:18:17] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:07:07 [I 2022-10-08 11:18:17] ...trap/engines/engine.py.229: process: 0.77319 | load: 0.00298 [I 2022-10-08 11:18:17] ...trap/engines/engine.py.230: loss: 0.24718 [I 2022-10-08 11:18:17] ...optimizers/trijoint.py.151: total_norm: 2.510036 [I 2022-10-08 11:18:25] ...trap/engines/engine.py.224: train: epoch 2 | batch 1760/2304 [I 2022-10-08 11:18:25] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:58 [I 2022-10-08 11:18:25] ...trap/engines/engine.py.229: process: 0.75983 | load: 0.00306 [I 2022-10-08 11:18:25] ...trap/engines/engine.py.230: loss: 0.29634 [I 2022-10-08 11:18:25] ...optimizers/trijoint.py.151: total_norm: 3.072548 [I 2022-10-08 11:18:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1770/2304 [I 2022-10-08 11:18:33] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:56 [I 2022-10-08 11:18:33] ...trap/engines/engine.py.229: process: 0.75902 | load: 0.00303 [I 2022-10-08 11:18:33] ...trap/engines/engine.py.230: loss: 0.16774 [I 2022-10-08 11:18:33] ...optimizers/trijoint.py.151: total_norm: 1.939248 [I 2022-10-08 11:18:41] ...trap/engines/engine.py.224: train: epoch 2 | batch 1780/2304 [I 2022-10-08 11:18:41] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:47 [I 2022-10-08 11:18:41] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00297 [I 2022-10-08 11:18:41] ...trap/engines/engine.py.230: loss: 0.23603 [I 2022-10-08 11:18:41] ...optimizers/trijoint.py.151: total_norm: 2.523983 [I 2022-10-08 11:18:49] ...trap/engines/engine.py.224: train: epoch 2 | batch 1790/2304 [I 2022-10-08 11:18:49] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:38 [I 2022-10-08 11:18:49] ...trap/engines/engine.py.229: process: 0.77352 | load: 0.00302 [I 2022-10-08 11:18:49] ...trap/engines/engine.py.230: loss: 0.18464 [I 2022-10-08 11:18:49] ...optimizers/trijoint.py.151: total_norm: 2.547827 [I 2022-10-08 11:18:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 1800/2304 [I 2022-10-08 11:18:56] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:06:31 [I 2022-10-08 11:18:56] ...trap/engines/engine.py.229: process: 0.77331 | load: 0.00303 [I 2022-10-08 11:18:56] ...trap/engines/engine.py.230: loss: 0.30441 [I 2022-10-08 11:18:56] ...optimizers/trijoint.py.151: total_norm: 3.742946 [I 2022-10-08 11:19:04] ...trap/engines/engine.py.224: train: epoch 2 | batch 1810/2304 [I 2022-10-08 11:19:04] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:23 [I 2022-10-08 11:19:04] ...trap/engines/engine.py.229: process: 0.77837 | load: 0.00302 [I 2022-10-08 11:19:04] ...trap/engines/engine.py.230: loss: 0.21113 [I 2022-10-08 11:19:04] ...optimizers/trijoint.py.151: total_norm: 3.004072 [I 2022-10-08 11:19:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 1820/2304 [I 2022-10-08 11:19:12] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:15 [I 2022-10-08 11:19:12] ...trap/engines/engine.py.229: process: 0.77522 | load: 0.00304 [I 2022-10-08 11:19:12] ...trap/engines/engine.py.230: loss: 0.27139 [I 2022-10-08 11:19:12] ...optimizers/trijoint.py.151: total_norm: 3.144392 [I 2022-10-08 11:19:20] ...trap/engines/engine.py.224: train: epoch 2 | batch 1830/2304 [I 2022-10-08 11:19:20] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:06:06 [I 2022-10-08 11:19:20] ...trap/engines/engine.py.229: process: 0.76192 | load: 0.00302 [I 2022-10-08 11:19:20] ...trap/engines/engine.py.230: loss: 0.31221 [I 2022-10-08 11:19:20] ...optimizers/trijoint.py.151: total_norm: 3.266989 [I 2022-10-08 11:19:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 1840/2304 [I 2022-10-08 11:19:27] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:06:01 [I 2022-10-08 11:19:27] ...trap/engines/engine.py.229: process: 0.78714 | load: 0.00299 [I 2022-10-08 11:19:27] ...trap/engines/engine.py.230: loss: 0.31169 [I 2022-10-08 11:19:27] ...optimizers/trijoint.py.151: total_norm: 3.626409 [I 2022-10-08 11:19:35] ...trap/engines/engine.py.224: train: epoch 2 | batch 1850/2304 [I 2022-10-08 11:19:35] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:53 [I 2022-10-08 11:19:35] ...trap/engines/engine.py.229: process: 0.76915 | load: 0.00300 [I 2022-10-08 11:19:35] ...trap/engines/engine.py.230: loss: 0.29961 [I 2022-10-08 11:19:35] ...optimizers/trijoint.py.151: total_norm: 3.192373 [I 2022-10-08 11:19:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1860/2304 [I 2022-10-08 11:19:43] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:05:46 [I 2022-10-08 11:19:43] ...trap/engines/engine.py.229: process: 0.78506 | load: 0.00305 [I 2022-10-08 11:19:43] ...trap/engines/engine.py.230: loss: 0.32227 [I 2022-10-08 11:19:43] ...optimizers/trijoint.py.151: total_norm: 3.245965 [I 2022-10-08 11:19:51] ...trap/engines/engine.py.224: train: epoch 2 | batch 1870/2304 [I 2022-10-08 11:19:51] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:05:37 [I 2022-10-08 11:19:51] ...trap/engines/engine.py.229: process: 0.76953 | load: 0.00302 [I 2022-10-08 11:19:51] ...trap/engines/engine.py.230: loss: 0.29871 [I 2022-10-08 11:19:51] ...optimizers/trijoint.py.151: total_norm: 3.320848 [I 2022-10-08 11:19:59] ...trap/engines/engine.py.224: train: epoch 2 | batch 1880/2304 [I 2022-10-08 11:19:59] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:29 [I 2022-10-08 11:19:59] ...trap/engines/engine.py.229: process: 0.77125 | load: 0.00303 [I 2022-10-08 11:19:59] ...trap/engines/engine.py.230: loss: 0.27844 [I 2022-10-08 11:19:59] ...optimizers/trijoint.py.151: total_norm: 2.804447 [I 2022-10-08 11:20:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1890/2304 [I 2022-10-08 11:20:07] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:23 [I 2022-10-08 11:20:07] ...trap/engines/engine.py.229: process: 0.78272 | load: 0.00304 [I 2022-10-08 11:20:07] ...trap/engines/engine.py.230: loss: 0.22807 [I 2022-10-08 11:20:07] ...optimizers/trijoint.py.151: total_norm: 3.023243 [I 2022-10-08 11:20:14] ...trap/engines/engine.py.224: train: epoch 2 | batch 1900/2304 [I 2022-10-08 11:20:14] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:05:16 [I 2022-10-08 11:20:14] ...trap/engines/engine.py.229: process: 0.78486 | load: 0.00304 [I 2022-10-08 11:20:14] ...trap/engines/engine.py.230: loss: 0.17298 [I 2022-10-08 11:20:14] ...optimizers/trijoint.py.151: total_norm: 2.798175 [I 2022-10-08 11:20:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 1910/2304 [I 2022-10-08 11:20:22] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:05:06 [I 2022-10-08 11:20:22] ...trap/engines/engine.py.229: process: 0.77411 | load: 0.00303 [I 2022-10-08 11:20:22] ...trap/engines/engine.py.230: loss: 0.17662 [I 2022-10-08 11:20:22] ...optimizers/trijoint.py.151: total_norm: 3.020373 [I 2022-10-08 11:20:30] ...trap/engines/engine.py.224: train: epoch 2 | batch 1920/2304 [I 2022-10-08 11:20:30] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:04:58 [I 2022-10-08 11:20:30] ...trap/engines/engine.py.229: process: 0.77300 | load: 0.00302 [I 2022-10-08 11:20:30] ...trap/engines/engine.py.230: loss: 0.30789 [I 2022-10-08 11:20:30] ...optimizers/trijoint.py.151: total_norm: 3.295826 [I 2022-10-08 11:20:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1930/2304 [I 2022-10-08 11:20:38] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:51 [I 2022-10-08 11:20:38] ...trap/engines/engine.py.229: process: 0.77769 | load: 0.00298 [I 2022-10-08 11:20:38] ...trap/engines/engine.py.230: loss: 0.19682 [I 2022-10-08 11:20:38] ...optimizers/trijoint.py.151: total_norm: 2.299650 [I 2022-10-08 11:20:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 1940/2304 [I 2022-10-08 11:20:46] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:04:43 [I 2022-10-08 11:20:46] ...trap/engines/engine.py.229: process: 0.77806 | load: 0.00307 [I 2022-10-08 11:20:46] ...trap/engines/engine.py.230: loss: 0.22883 [I 2022-10-08 11:20:46] ...optimizers/trijoint.py.151: total_norm: 2.679304 [I 2022-10-08 11:20:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1950/2304 [I 2022-10-08 11:20:53] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:04:34 [I 2022-10-08 11:20:53] ...trap/engines/engine.py.229: process: 0.78166 | load: 0.00301 [I 2022-10-08 11:20:53] ...trap/engines/engine.py.230: loss: 0.26968 [I 2022-10-08 11:20:53] ...optimizers/trijoint.py.151: total_norm: 3.083073 [I 2022-10-08 11:21:01] ...trap/engines/engine.py.224: train: epoch 2 | batch 1960/2304 [I 2022-10-08 11:21:01] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:04:27 [I 2022-10-08 11:21:01] ...trap/engines/engine.py.229: process: 0.76992 | load: 0.00301 [I 2022-10-08 11:21:01] ...trap/engines/engine.py.230: loss: 0.24679 [I 2022-10-08 11:21:01] ...optimizers/trijoint.py.151: total_norm: 3.023032 [I 2022-10-08 11:21:09] ...trap/engines/engine.py.224: train: epoch 2 | batch 1970/2304 [I 2022-10-08 11:21:09] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:21 [I 2022-10-08 11:21:09] ...trap/engines/engine.py.229: process: 0.78411 | load: 0.00301 [I 2022-10-08 11:21:09] ...trap/engines/engine.py.230: loss: 0.23872 [I 2022-10-08 11:21:09] ...optimizers/trijoint.py.151: total_norm: 3.655685 [I 2022-10-08 11:21:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 1980/2304 [I 2022-10-08 11:21:17] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:04:13 [I 2022-10-08 11:21:17] ...trap/engines/engine.py.229: process: 0.77917 | load: 0.00300 [I 2022-10-08 11:21:17] ...trap/engines/engine.py.230: loss: 0.21698 [I 2022-10-08 11:21:17] ...optimizers/trijoint.py.151: total_norm: 2.281249 [I 2022-10-08 11:21:25] ...trap/engines/engine.py.224: train: epoch 2 | batch 1990/2304 [I 2022-10-08 11:21:25] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:04:05 [I 2022-10-08 11:21:25] ...trap/engines/engine.py.229: process: 0.78238 | load: 0.00301 [I 2022-10-08 11:21:25] ...trap/engines/engine.py.230: loss: 0.21195 [I 2022-10-08 11:21:25] ...optimizers/trijoint.py.151: total_norm: 2.788482 [I 2022-10-08 11:21:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 2000/2304 [I 2022-10-08 11:21:32] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:03:54 [I 2022-10-08 11:21:32] ...trap/engines/engine.py.229: process: 0.76982 | load: 0.00301 [I 2022-10-08 11:21:32] ...trap/engines/engine.py.230: loss: 0.34800 [I 2022-10-08 11:21:32] ...optimizers/trijoint.py.151: total_norm: 3.435223 [I 2022-10-08 11:21:40] ...trap/engines/engine.py.224: train: epoch 2 | batch 2010/2304 [I 2022-10-08 11:21:40] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:03:48 [I 2022-10-08 11:21:40] ...trap/engines/engine.py.229: process: 0.78249 | load: 0.00311 [I 2022-10-08 11:21:40] ...trap/engines/engine.py.230: loss: 0.36709 [I 2022-10-08 11:21:40] ...optimizers/trijoint.py.151: total_norm: 3.485604 [I 2022-10-08 11:21:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 2020/2304 [I 2022-10-08 11:21:48] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:03:41 [I 2022-10-08 11:21:48] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00301 [I 2022-10-08 11:21:48] ...trap/engines/engine.py.230: loss: 0.20353 [I 2022-10-08 11:21:48] ...optimizers/trijoint.py.151: total_norm: 2.500122 [I 2022-10-08 11:21:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 2030/2304 [I 2022-10-08 11:21:56] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:03:34 [I 2022-10-08 11:21:56] ...trap/engines/engine.py.229: process: 0.78241 | load: 0.00301 [I 2022-10-08 11:21:56] ...trap/engines/engine.py.230: loss: 0.21304 [I 2022-10-08 11:21:56] ...optimizers/trijoint.py.151: total_norm: 2.536446 [I 2022-10-08 11:22:04] ...trap/engines/engine.py.224: train: epoch 2 | batch 2040/2304 [I 2022-10-08 11:22:04] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:03:25 [I 2022-10-08 11:22:04] ...trap/engines/engine.py.229: process: 0.78984 | load: 0.00303 [I 2022-10-08 11:22:04] ...trap/engines/engine.py.230: loss: 0.18215 [I 2022-10-08 11:22:04] ...optimizers/trijoint.py.151: total_norm: 2.820770 [I 2022-10-08 11:22:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 2050/2304 [I 2022-10-08 11:22:11] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:03:17 [I 2022-10-08 11:22:11] ...trap/engines/engine.py.229: process: 0.77111 | load: 0.00305 [I 2022-10-08 11:22:11] ...trap/engines/engine.py.230: loss: 0.24384 [I 2022-10-08 11:22:11] ...optimizers/trijoint.py.151: total_norm: 2.414963 [I 2022-10-08 11:22:19] ...trap/engines/engine.py.224: train: epoch 2 | batch 2060/2304 [I 2022-10-08 11:22:19] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:03:07 [I 2022-10-08 11:22:19] ...trap/engines/engine.py.229: process: 0.76861 | load: 0.00300 [I 2022-10-08 11:22:19] ...trap/engines/engine.py.230: loss: 0.22391 [I 2022-10-08 11:22:19] ...optimizers/trijoint.py.151: total_norm: 3.494437 [I 2022-10-08 11:22:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 2070/2304 [I 2022-10-08 11:22:27] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:03:00 [I 2022-10-08 11:22:27] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00301 [I 2022-10-08 11:22:27] ...trap/engines/engine.py.230: loss: 0.29952 [I 2022-10-08 11:22:27] ...optimizers/trijoint.py.151: total_norm: 2.787125 [I 2022-10-08 11:22:35] ...trap/engines/engine.py.224: train: epoch 2 | batch 2080/2304 [I 2022-10-08 11:22:35] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:54 [I 2022-10-08 11:22:35] ...trap/engines/engine.py.229: process: 0.78786 | load: 0.00303 [I 2022-10-08 11:22:35] ...trap/engines/engine.py.230: loss: 0.23421 [I 2022-10-08 11:22:35] ...optimizers/trijoint.py.151: total_norm: 2.709201 [I 2022-10-08 11:22:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 2090/2304 [I 2022-10-08 11:22:43] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:47 [I 2022-10-08 11:22:43] ...trap/engines/engine.py.229: process: 0.77805 | load: 0.00301 [I 2022-10-08 11:22:43] ...trap/engines/engine.py.230: loss: 0.26838 [I 2022-10-08 11:22:43] ...optimizers/trijoint.py.151: total_norm: 3.304010 [I 2022-10-08 11:22:50] ...trap/engines/engine.py.224: train: epoch 2 | batch 2100/2304 [I 2022-10-08 11:22:50] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:02:38 [I 2022-10-08 11:22:50] ...trap/engines/engine.py.229: process: 0.77479 | load: 0.00309 [I 2022-10-08 11:22:50] ...trap/engines/engine.py.230: loss: 0.27578 [I 2022-10-08 11:22:50] ...optimizers/trijoint.py.151: total_norm: 3.517637 [I 2022-10-08 11:22:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 2110/2304 [I 2022-10-08 11:22:58] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:02:31 [I 2022-10-08 11:22:58] ...trap/engines/engine.py.229: process: 0.78595 | load: 0.00305 [I 2022-10-08 11:22:58] ...trap/engines/engine.py.230: loss: 0.23070 [I 2022-10-08 11:22:58] ...optimizers/trijoint.py.151: total_norm: 2.642065 [I 2022-10-08 11:23:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 2120/2304 [I 2022-10-08 11:23:06] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:02:23 [I 2022-10-08 11:23:06] ...trap/engines/engine.py.229: process: 0.76595 | load: 0.00299 [I 2022-10-08 11:23:06] ...trap/engines/engine.py.230: loss: 0.30683 [I 2022-10-08 11:23:06] ...optimizers/trijoint.py.151: total_norm: 3.615993 [I 2022-10-08 11:23:14] ...trap/engines/engine.py.224: train: epoch 2 | batch 2130/2304 [I 2022-10-08 11:23:14] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:02:15 [I 2022-10-08 11:23:14] ...trap/engines/engine.py.229: process: 0.77047 | load: 0.00301 [I 2022-10-08 11:23:14] ...trap/engines/engine.py.230: loss: 0.22662 [I 2022-10-08 11:23:14] ...optimizers/trijoint.py.151: total_norm: 3.006694 [I 2022-10-08 11:23:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 2140/2304 [I 2022-10-08 11:23:22] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:02:08 [I 2022-10-08 11:23:22] ...trap/engines/engine.py.229: process: 0.78868 | load: 0.00307 [I 2022-10-08 11:23:22] ...trap/engines/engine.py.230: loss: 0.23916 [I 2022-10-08 11:23:22] ...optimizers/trijoint.py.151: total_norm: 2.774974 [I 2022-10-08 11:23:29] ...trap/engines/engine.py.224: train: epoch 2 | batch 2150/2304 [I 2022-10-08 11:23:29] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:01:59 [I 2022-10-08 11:23:29] ...trap/engines/engine.py.229: process: 0.77195 | load: 0.00305 [I 2022-10-08 11:23:29] ...trap/engines/engine.py.230: loss: 0.29691 [I 2022-10-08 11:23:29] ...optimizers/trijoint.py.151: total_norm: 3.043871 [I 2022-10-08 11:23:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 2160/2304 [I 2022-10-08 11:23:37] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:01:51 [I 2022-10-08 11:23:37] ...trap/engines/engine.py.229: process: 0.77328 | load: 0.00302 [I 2022-10-08 11:23:37] ...trap/engines/engine.py.230: loss: 0.21810 [I 2022-10-08 11:23:37] ...optimizers/trijoint.py.151: total_norm: 2.724983 [I 2022-10-08 11:23:45] ...trap/engines/engine.py.224: train: epoch 2 | batch 2170/2304 [I 2022-10-08 11:23:45] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:01:44 [I 2022-10-08 11:23:45] ...trap/engines/engine.py.229: process: 0.78969 | load: 0.00300 [I 2022-10-08 11:23:45] ...trap/engines/engine.py.230: loss: 0.29558 [I 2022-10-08 11:23:45] ...optimizers/trijoint.py.151: total_norm: 3.462281 [I 2022-10-08 11:23:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 2180/2304 [I 2022-10-08 11:23:53] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:01:35 [I 2022-10-08 11:23:53] ...trap/engines/engine.py.229: process: 0.76643 | load: 0.00302 [I 2022-10-08 11:23:53] ...trap/engines/engine.py.230: loss: 0.28162 [I 2022-10-08 11:23:53] ...optimizers/trijoint.py.151: total_norm: 3.044779 [I 2022-10-08 11:24:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 2190/2304 [I 2022-10-08 11:24:00] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:01:28 [I 2022-10-08 11:24:00] ...trap/engines/engine.py.229: process: 0.77221 | load: 0.00301 [I 2022-10-08 11:24:01] ...trap/engines/engine.py.230: loss: 0.36311 [I 2022-10-08 11:24:01] ...optimizers/trijoint.py.151: total_norm: 3.352859 [I 2022-10-08 11:24:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 2200/2304 [I 2022-10-08 11:24:08] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:01:20 [I 2022-10-08 11:24:08] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00302 [I 2022-10-08 11:24:08] ...trap/engines/engine.py.230: loss: 0.29858 [I 2022-10-08 11:24:08] ...optimizers/trijoint.py.151: total_norm: 3.175842 [I 2022-10-08 11:24:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 2210/2304 [I 2022-10-08 11:24:16] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:01:13 [I 2022-10-08 11:24:16] ...trap/engines/engine.py.229: process: 0.77924 | load: 0.00303 [I 2022-10-08 11:24:16] ...trap/engines/engine.py.230: loss: 0.28970 [I 2022-10-08 11:24:16] ...optimizers/trijoint.py.151: total_norm: 3.147356 [I 2022-10-08 11:24:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 2220/2304 [I 2022-10-08 11:24:24] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:01:04 [I 2022-10-08 11:24:24] ...trap/engines/engine.py.229: process: 0.77973 | load: 0.00303 [I 2022-10-08 11:24:24] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-08 11:24:24] ...optimizers/trijoint.py.151: total_norm: 2.697587 [I 2022-10-08 11:24:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 2230/2304 [I 2022-10-08 11:24:32] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:00:57 [I 2022-10-08 11:24:32] ...trap/engines/engine.py.229: process: 0.77249 | load: 0.00301 [I 2022-10-08 11:24:32] ...trap/engines/engine.py.230: loss: 0.17822 [I 2022-10-08 11:24:32] ...optimizers/trijoint.py.151: total_norm: 2.673111 [I 2022-10-08 11:24:39] ...trap/engines/engine.py.224: train: epoch 2 | batch 2240/2304 [I 2022-10-08 11:24:39] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:00:49 [I 2022-10-08 11:24:39] ...trap/engines/engine.py.229: process: 0.77597 | load: 0.00303 [I 2022-10-08 11:24:39] ...trap/engines/engine.py.230: loss: 0.28740 [I 2022-10-08 11:24:39] ...optimizers/trijoint.py.151: total_norm: 2.779307 [I 2022-10-08 11:24:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 2250/2304 [I 2022-10-08 11:24:47] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:00:42 [I 2022-10-08 11:24:47] ...trap/engines/engine.py.229: process: 0.78972 | load: 0.00305 [I 2022-10-08 11:24:47] ...trap/engines/engine.py.230: loss: 0.23867 [I 2022-10-08 11:24:47] ...optimizers/trijoint.py.151: total_norm: 3.160202 [I 2022-10-08 11:24:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 2260/2304 [I 2022-10-08 11:24:55] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:00:34 [I 2022-10-08 11:24:55] ...trap/engines/engine.py.229: process: 0.77606 | load: 0.00305 [I 2022-10-08 11:24:55] ...trap/engines/engine.py.230: loss: 0.20701 [I 2022-10-08 11:24:55] ...optimizers/trijoint.py.151: total_norm: 2.529128 [I 2022-10-08 11:25:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 2270/2304 [I 2022-10-08 11:25:03] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:00:26 [I 2022-10-08 11:25:03] ...trap/engines/engine.py.229: process: 0.78867 | load: 0.00299 [I 2022-10-08 11:25:03] ...trap/engines/engine.py.230: loss: 0.29361 [I 2022-10-08 11:25:03] ...optimizers/trijoint.py.151: total_norm: 2.874711 [I 2022-10-08 11:25:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 2280/2304 [I 2022-10-08 11:25:11] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:00:18 [I 2022-10-08 11:25:11] ...trap/engines/engine.py.229: process: 0.77428 | load: 0.00307 [I 2022-10-08 11:25:11] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-08 11:25:11] ...optimizers/trijoint.py.151: total_norm: 2.376002 [I 2022-10-08 11:25:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 2290/2304 [I 2022-10-08 11:25:18] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:00:10 [I 2022-10-08 11:25:18] ...trap/engines/engine.py.229: process: 0.77144 | load: 0.00306 [I 2022-10-08 11:25:18] ...trap/engines/engine.py.230: loss: 0.38022 [I 2022-10-08 11:25:18] ...optimizers/trijoint.py.151: total_norm: 3.347399 [I 2022-10-08 11:25:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 2300/2304 [I 2022-10-08 11:25:26] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:00:03 [I 2022-10-08 11:25:26] ...trap/engines/engine.py.229: process: 0.76853 | load: 0.00017 [I 2022-10-08 11:25:26] ...trap/engines/engine.py.230: loss: 0.18228 [I 2022-10-08 11:25:26] ...optimizers/trijoint.py.151: total_norm: 2.547690 [I 2022-10-08 11:25:29] ...trap/engines/engine.py.224: train: epoch 2 | batch 2304/2304 [I 2022-10-08 11:25:29] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:00:00 [I 2022-10-08 11:25:29] ...trap/engines/engine.py.229: process: 0.76742 | load: 0.00016 [I 2022-10-08 11:25:29] ...trap/engines/engine.py.230: loss: 0.28349 [I 2022-10-08 11:25:29] ...optimizers/trijoint.py.151: total_norm: 3.359319 [S 2022-10-08 11:25:30] ...trap/engines/engine.py.129: train_epoch.epoch: 2 [S 2022-10-08 11:25:30] ...trap/engines/engine.py.129: train_epoch.loss: 0.26635745685555673 [I 2022-10-08 11:25:31] ...s/criterions/triplet.py.84: margin = 0.065 [I 2022-10-08 11:25:31] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 2 [I 2022-10-08 11:25:31] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 11:25:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 11:25:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 11:25:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 11:25:32] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 11:25:33] ...trap/engines/engine.py.321: eval: epoch 2 | batch 0/510 [I 2022-10-08 11:25:33] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:21 [I 2022-10-08 11:25:33] ...trap/engines/engine.py.326: process: 2.51362 | load: 2.01407 [I 2022-10-08 11:25:37] ...trap/engines/engine.py.321: eval: epoch 2 | batch 10/510 [I 2022-10-08 11:25:37] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:13 [I 2022-10-08 11:25:37] ...trap/engines/engine.py.326: process: 0.39808 | load: 0.00018 [I 2022-10-08 11:25:41] ...trap/engines/engine.py.321: eval: epoch 2 | batch 20/510 [I 2022-10-08 11:25:41] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:33 [I 2022-10-08 11:25:41] ...trap/engines/engine.py.326: process: 0.40252 | load: 0.00019 [I 2022-10-08 11:25:47] ...trap/engines/engine.py.321: eval: epoch 2 | batch 30/510 [I 2022-10-08 11:25:47] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:04 [I 2022-10-08 11:25:47] ...trap/engines/engine.py.326: process: 0.65497 | load: 0.00018 [I 2022-10-08 11:25:52] ...trap/engines/engine.py.321: eval: epoch 2 | batch 40/510 [I 2022-10-08 11:25:52] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:00 [I 2022-10-08 11:25:52] ...trap/engines/engine.py.326: process: 0.41462 | load: 0.00018 [I 2022-10-08 11:25:57] ...trap/engines/engine.py.321: eval: epoch 2 | batch 50/510 [I 2022-10-08 11:25:57] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:25 [I 2022-10-08 11:25:57] ...trap/engines/engine.py.326: process: 0.47970 | load: 0.00019 [I 2022-10-08 11:26:02] ...trap/engines/engine.py.321: eval: epoch 2 | batch 60/510 [I 2022-10-08 11:26:02] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:46 [I 2022-10-08 11:26:02] ...trap/engines/engine.py.326: process: 0.40607 | load: 0.00018 [I 2022-10-08 11:26:07] ...trap/engines/engine.py.321: eval: epoch 2 | batch 70/510 [I 2022-10-08 11:26:07] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:35 [I 2022-10-08 11:26:07] ...trap/engines/engine.py.326: process: 0.47718 | load: 0.00018 [I 2022-10-08 11:26:12] ...trap/engines/engine.py.321: eval: epoch 2 | batch 80/510 [I 2022-10-08 11:26:12] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:16 [I 2022-10-08 11:26:12] ...trap/engines/engine.py.326: process: 0.48776 | load: 0.00019 [I 2022-10-08 11:26:16] ...trap/engines/engine.py.321: eval: epoch 2 | batch 90/510 [I 2022-10-08 11:26:16] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:06 [I 2022-10-08 11:26:16] ...trap/engines/engine.py.326: process: 0.44300 | load: 0.00018 [I 2022-10-08 11:26:20] ...trap/engines/engine.py.321: eval: epoch 2 | batch 100/510 [I 2022-10-08 11:26:20] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:02:50 [I 2022-10-08 11:26:20] ...trap/engines/engine.py.326: process: 0.31015 | load: 0.00018 [I 2022-10-08 11:26:24] ...trap/engines/engine.py.321: eval: epoch 2 | batch 110/510 [I 2022-10-08 11:26:24] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:10 [I 2022-10-08 11:26:24] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00018 [I 2022-10-08 11:26:27] ...trap/engines/engine.py.321: eval: epoch 2 | batch 120/510 [I 2022-10-08 11:26:27] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:03 [I 2022-10-08 11:26:27] ...trap/engines/engine.py.326: process: 0.31681 | load: 0.00019 [I 2022-10-08 11:26:30] ...trap/engines/engine.py.321: eval: epoch 2 | batch 130/510 [I 2022-10-08 11:26:30] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:00 [I 2022-10-08 11:26:30] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00020 [I 2022-10-08 11:26:33] ...trap/engines/engine.py.321: eval: epoch 2 | batch 140/510 [I 2022-10-08 11:26:33] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:56 [I 2022-10-08 11:26:33] ...trap/engines/engine.py.326: process: 0.31715 | load: 0.00019 [I 2022-10-08 11:26:36] ...trap/engines/engine.py.321: eval: epoch 2 | batch 150/510 [I 2022-10-08 11:26:36] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:53 [I 2022-10-08 11:26:36] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00019 [I 2022-10-08 11:26:39] ...trap/engines/engine.py.321: eval: epoch 2 | batch 160/510 [I 2022-10-08 11:26:39] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:50 [I 2022-10-08 11:26:39] ...trap/engines/engine.py.326: process: 0.31639 | load: 0.00019 [I 2022-10-08 11:26:43] ...trap/engines/engine.py.321: eval: epoch 2 | batch 170/510 [I 2022-10-08 11:26:43] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:48 [I 2022-10-08 11:26:43] ...trap/engines/engine.py.326: process: 0.31874 | load: 0.00019 [I 2022-10-08 11:26:46] ...trap/engines/engine.py.321: eval: epoch 2 | batch 180/510 [I 2022-10-08 11:26:46] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:44 [I 2022-10-08 11:26:46] ...trap/engines/engine.py.326: process: 0.31497 | load: 0.00019 [I 2022-10-08 11:26:49] ...trap/engines/engine.py.321: eval: epoch 2 | batch 190/510 [I 2022-10-08 11:26:49] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:41 [I 2022-10-08 11:26:49] ...trap/engines/engine.py.326: process: 0.31940 | load: 0.00019 [I 2022-10-08 11:26:52] ...trap/engines/engine.py.321: eval: epoch 2 | batch 200/510 [I 2022-10-08 11:26:52] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:38 [I 2022-10-08 11:26:52] ...trap/engines/engine.py.326: process: 0.31502 | load: 0.00020 [I 2022-10-08 11:26:55] ...trap/engines/engine.py.321: eval: epoch 2 | batch 210/510 [I 2022-10-08 11:26:55] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:34 [I 2022-10-08 11:26:55] ...trap/engines/engine.py.326: process: 0.31319 | load: 0.00019 [I 2022-10-08 11:26:58] ...trap/engines/engine.py.321: eval: epoch 2 | batch 220/510 [I 2022-10-08 11:26:58] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:31 [I 2022-10-08 11:26:58] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00019 [I 2022-10-08 11:27:02] ...trap/engines/engine.py.321: eval: epoch 2 | batch 230/510 [I 2022-10-08 11:27:02] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:28 [I 2022-10-08 11:27:02] ...trap/engines/engine.py.326: process: 0.31531 | load: 0.00020 [I 2022-10-08 11:27:05] ...trap/engines/engine.py.321: eval: epoch 2 | batch 240/510 [I 2022-10-08 11:27:05] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:25 [I 2022-10-08 11:27:05] ...trap/engines/engine.py.326: process: 0.31814 | load: 0.00020 [I 2022-10-08 11:27:08] ...trap/engines/engine.py.321: eval: epoch 2 | batch 250/510 [I 2022-10-08 11:27:08] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:22 [I 2022-10-08 11:27:08] ...trap/engines/engine.py.326: process: 0.31236 | load: 0.00019 [I 2022-10-08 11:27:11] ...trap/engines/engine.py.321: eval: epoch 2 | batch 260/510 [I 2022-10-08 11:27:11] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:19 [I 2022-10-08 11:27:11] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00020 [I 2022-10-08 11:27:14] ...trap/engines/engine.py.321: eval: epoch 2 | batch 270/510 [I 2022-10-08 11:27:14] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:16 [I 2022-10-08 11:27:14] ...trap/engines/engine.py.326: process: 0.32143 | load: 0.00020 [I 2022-10-08 11:27:17] ...trap/engines/engine.py.321: eval: epoch 2 | batch 280/510 [I 2022-10-08 11:27:17] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:12 [I 2022-10-08 11:27:17] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00020 [I 2022-10-08 11:27:21] ...trap/engines/engine.py.321: eval: epoch 2 | batch 290/510 [I 2022-10-08 11:27:21] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:09 [I 2022-10-08 11:27:21] ...trap/engines/engine.py.326: process: 0.31669 | load: 0.00019 [I 2022-10-08 11:27:24] ...trap/engines/engine.py.321: eval: epoch 2 | batch 300/510 [I 2022-10-08 11:27:24] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:06 [I 2022-10-08 11:27:24] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00019 [I 2022-10-08 11:27:27] ...trap/engines/engine.py.321: eval: epoch 2 | batch 310/510 [I 2022-10-08 11:27:27] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:03 [I 2022-10-08 11:27:27] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00018 [I 2022-10-08 11:27:30] ...trap/engines/engine.py.321: eval: epoch 2 | batch 320/510 [I 2022-10-08 11:27:30] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:00 [I 2022-10-08 11:27:30] ...trap/engines/engine.py.326: process: 0.32309 | load: 0.00019 [I 2022-10-08 11:27:33] ...trap/engines/engine.py.321: eval: epoch 2 | batch 330/510 [I 2022-10-08 11:27:33] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:57 [I 2022-10-08 11:27:33] ...trap/engines/engine.py.326: process: 0.32378 | load: 0.00019 [I 2022-10-08 11:27:36] ...trap/engines/engine.py.321: eval: epoch 2 | batch 340/510 [I 2022-10-08 11:27:36] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:54 [I 2022-10-08 11:27:36] ...trap/engines/engine.py.326: process: 0.32019 | load: 0.00019 [I 2022-10-08 11:27:40] ...trap/engines/engine.py.321: eval: epoch 2 | batch 350/510 [I 2022-10-08 11:27:40] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:50 [I 2022-10-08 11:27:40] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00019 [I 2022-10-08 11:27:43] ...trap/engines/engine.py.321: eval: epoch 2 | batch 360/510 [I 2022-10-08 11:27:43] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:47 [I 2022-10-08 11:27:43] ...trap/engines/engine.py.326: process: 0.32032 | load: 0.00020 [I 2022-10-08 11:27:46] ...trap/engines/engine.py.321: eval: epoch 2 | batch 370/510 [I 2022-10-08 11:27:46] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:44 [I 2022-10-08 11:27:46] ...trap/engines/engine.py.326: process: 0.31990 | load: 0.00020 [I 2022-10-08 11:27:49] ...trap/engines/engine.py.321: eval: epoch 2 | batch 380/510 [I 2022-10-08 11:27:49] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:41 [I 2022-10-08 11:27:49] ...trap/engines/engine.py.326: process: 0.32187 | load: 0.00019 [I 2022-10-08 11:27:52] ...trap/engines/engine.py.321: eval: epoch 2 | batch 390/510 [I 2022-10-08 11:27:52] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:38 [I 2022-10-08 11:27:52] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00019 [I 2022-10-08 11:27:56] ...trap/engines/engine.py.321: eval: epoch 2 | batch 400/510 [I 2022-10-08 11:27:56] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:35 [I 2022-10-08 11:27:56] ...trap/engines/engine.py.326: process: 0.31912 | load: 0.00019 [I 2022-10-08 11:27:59] ...trap/engines/engine.py.321: eval: epoch 2 | batch 410/510 [I 2022-10-08 11:27:59] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:31 [I 2022-10-08 11:27:59] ...trap/engines/engine.py.326: process: 0.31871 | load: 0.00020 [I 2022-10-08 11:28:02] ...trap/engines/engine.py.321: eval: epoch 2 | batch 420/510 [I 2022-10-08 11:28:02] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:28 [I 2022-10-08 11:28:02] ...trap/engines/engine.py.326: process: 0.31724 | load: 0.00019 [I 2022-10-08 11:28:05] ...trap/engines/engine.py.321: eval: epoch 2 | batch 430/510 [I 2022-10-08 11:28:05] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:25 [I 2022-10-08 11:28:05] ...trap/engines/engine.py.326: process: 0.31403 | load: 0.00020 [I 2022-10-08 11:28:08] ...trap/engines/engine.py.321: eval: epoch 2 | batch 440/510 [I 2022-10-08 11:28:08] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:22 [I 2022-10-08 11:28:08] ...trap/engines/engine.py.326: process: 0.31291 | load: 0.00020 [I 2022-10-08 11:28:12] ...trap/engines/engine.py.321: eval: epoch 2 | batch 450/510 [I 2022-10-08 11:28:12] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:19 [I 2022-10-08 11:28:12] ...trap/engines/engine.py.326: process: 0.31754 | load: 0.00020 [I 2022-10-08 11:28:15] ...trap/engines/engine.py.321: eval: epoch 2 | batch 460/510 [I 2022-10-08 11:28:15] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:15 [I 2022-10-08 11:28:15] ...trap/engines/engine.py.326: process: 0.31082 | load: 0.00019 [I 2022-10-08 11:28:18] ...trap/engines/engine.py.321: eval: epoch 2 | batch 470/510 [I 2022-10-08 11:28:18] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:12 [I 2022-10-08 11:28:18] ...trap/engines/engine.py.326: process: 0.31844 | load: 0.00019 [I 2022-10-08 11:28:21] ...trap/engines/engine.py.321: eval: epoch 2 | batch 480/510 [I 2022-10-08 11:28:21] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:09 [I 2022-10-08 11:28:21] ...trap/engines/engine.py.326: process: 0.31922 | load: 0.00019 [I 2022-10-08 11:28:24] ...trap/engines/engine.py.321: eval: epoch 2 | batch 490/510 [I 2022-10-08 11:28:24] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:06 [I 2022-10-08 11:28:24] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00020 [I 2022-10-08 11:28:27] ...trap/engines/engine.py.321: eval: epoch 2 | batch 500/510 [I 2022-10-08 11:28:27] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:03 [I 2022-10-08 11:28:27] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-08 11:28:31] ...trap/engines/engine.py.321: eval: epoch 2 | batch 510/510 [I 2022-10-08 11:28:31] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:00 [I 2022-10-08 11:28:31] ...trap/engines/engine.py.326: process: 0.31355 | load: 0.00015 [S 2022-10-08 11:28:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 2 [I 2022-10-08 11:28:31] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.60 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.499800 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.793000 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.877200 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.523900 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.808800 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.877500 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.489898 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011259 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.014057 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.014449 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013095 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012424 [S 2022-10-08 11:28:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011935 [I 2022-10-08 11:29:03] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-08 11:29:03] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 11:29:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 11:29:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 11:29:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 11:29:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 11:29:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 11:29:10] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 11:29:10] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 11:29:10] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 11:29:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 11:29:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 11:29:18] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 11:29:18] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 11:29:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 11:29:25] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 11:29:25] ...trap/engines/engine.py.163: Training model on trainset for epoch 3 [I 2022-10-08 11:29:25] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 11:29:28] ...trap/engines/engine.py.224: train: epoch 3 | batch 0/2304 [I 2022-10-08 11:29:28] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:15 [I 2022-10-08 11:29:28] ...trap/engines/engine.py.229: process: 2.97552 | load: 2.15222 [I 2022-10-08 11:29:28] ...trap/engines/engine.py.230: loss: 0.19087 [I 2022-10-08 11:29:28] ...optimizers/trijoint.py.151: total_norm: 2.180360 [I 2022-10-08 11:29:36] ...trap/engines/engine.py.224: train: epoch 3 | batch 10/2304 [I 2022-10-08 11:29:36] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:39 [I 2022-10-08 11:29:36] ...trap/engines/engine.py.229: process: 0.76621 | load: 0.00594 [I 2022-10-08 11:29:36] ...trap/engines/engine.py.230: loss: 0.32495 [I 2022-10-08 11:29:36] ...optimizers/trijoint.py.151: total_norm: 3.227221 [I 2022-10-08 11:29:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 20/2304 [I 2022-10-08 11:29:44] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:52 [I 2022-10-08 11:29:44] ...trap/engines/engine.py.229: process: 0.80595 | load: 0.00478 [I 2022-10-08 11:29:44] ...trap/engines/engine.py.230: loss: 0.23118 [I 2022-10-08 11:29:44] ...optimizers/trijoint.py.151: total_norm: 2.725675 [I 2022-10-08 11:29:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 30/2304 [I 2022-10-08 11:29:51] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:43 [I 2022-10-08 11:29:51] ...trap/engines/engine.py.229: process: 0.78433 | load: 0.00413 [I 2022-10-08 11:29:51] ...trap/engines/engine.py.230: loss: 0.22987 [I 2022-10-08 11:29:51] ...optimizers/trijoint.py.151: total_norm: 2.873151 [I 2022-10-08 11:29:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 40/2304 [I 2022-10-08 11:29:59] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:19 [I 2022-10-08 11:29:59] ...trap/engines/engine.py.229: process: 0.75933 | load: 0.00358 [I 2022-10-08 11:29:59] ...trap/engines/engine.py.230: loss: 0.31100 [I 2022-10-08 11:29:59] ...optimizers/trijoint.py.151: total_norm: 4.009901 [I 2022-10-08 11:30:07] ...trap/engines/engine.py.224: train: epoch 3 | batch 50/2304 [I 2022-10-08 11:30:07] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:22 [I 2022-10-08 11:30:07] ...trap/engines/engine.py.229: process: 0.77349 | load: 0.00335 [I 2022-10-08 11:30:07] ...trap/engines/engine.py.230: loss: 0.23383 [I 2022-10-08 11:30:07] ...optimizers/trijoint.py.151: total_norm: 3.405691 [I 2022-10-08 11:30:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 60/2304 [I 2022-10-08 11:30:15] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:24 [I 2022-10-08 11:30:15] ...trap/engines/engine.py.229: process: 0.79413 | load: 0.00340 [I 2022-10-08 11:30:15] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-08 11:30:15] ...optimizers/trijoint.py.151: total_norm: 2.203027 [I 2022-10-08 11:30:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 70/2304 [I 2022-10-08 11:30:23] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:08 [I 2022-10-08 11:30:23] ...trap/engines/engine.py.229: process: 0.78054 | load: 0.00347 [I 2022-10-08 11:30:23] ...trap/engines/engine.py.230: loss: 0.29754 [I 2022-10-08 11:30:23] ...optimizers/trijoint.py.151: total_norm: 3.150953 [I 2022-10-08 11:30:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 80/2304 [I 2022-10-08 11:30:31] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:58 [I 2022-10-08 11:30:31] ...trap/engines/engine.py.229: process: 0.80184 | load: 0.00406 [I 2022-10-08 11:30:31] ...trap/engines/engine.py.230: loss: 0.19441 [I 2022-10-08 11:30:31] ...optimizers/trijoint.py.151: total_norm: 2.832110 [I 2022-10-08 11:30:38] ...trap/engines/engine.py.224: train: epoch 3 | batch 90/2304 [I 2022-10-08 11:30:38] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:48 [I 2022-10-08 11:30:38] ...trap/engines/engine.py.229: process: 0.76643 | load: 0.00308 [I 2022-10-08 11:30:38] ...trap/engines/engine.py.230: loss: 0.28877 [I 2022-10-08 11:30:38] ...optimizers/trijoint.py.151: total_norm: 3.380809 [I 2022-10-08 11:30:46] ...trap/engines/engine.py.224: train: epoch 3 | batch 100/2304 [I 2022-10-08 11:30:46] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:43 [I 2022-10-08 11:30:46] ...trap/engines/engine.py.229: process: 0.78816 | load: 0.00318 [I 2022-10-08 11:30:46] ...trap/engines/engine.py.230: loss: 0.36220 [I 2022-10-08 11:30:46] ...optimizers/trijoint.py.151: total_norm: 3.996629 [I 2022-10-08 11:30:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 110/2304 [I 2022-10-08 11:30:54] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:30 [I 2022-10-08 11:30:54] ...trap/engines/engine.py.229: process: 0.77917 | load: 0.00318 [I 2022-10-08 11:30:54] ...trap/engines/engine.py.230: loss: 0.21945 [I 2022-10-08 11:30:54] ...optimizers/trijoint.py.151: total_norm: 2.968516 [I 2022-10-08 11:31:02] ...trap/engines/engine.py.224: train: epoch 3 | batch 120/2304 [I 2022-10-08 11:31:02] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:31 [I 2022-10-08 11:31:02] ...trap/engines/engine.py.229: process: 0.78606 | load: 0.00325 [I 2022-10-08 11:31:02] ...trap/engines/engine.py.230: loss: 0.22315 [I 2022-10-08 11:31:02] ...optimizers/trijoint.py.151: total_norm: 2.699594 [I 2022-10-08 11:31:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 130/2304 [I 2022-10-08 11:31:10] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:20 [I 2022-10-08 11:31:10] ...trap/engines/engine.py.229: process: 0.78734 | load: 0.00362 [I 2022-10-08 11:31:10] ...trap/engines/engine.py.230: loss: 0.23590 [I 2022-10-08 11:31:10] ...optimizers/trijoint.py.151: total_norm: 2.718491 [I 2022-10-08 11:31:18] ...trap/engines/engine.py.224: train: epoch 3 | batch 140/2304 [I 2022-10-08 11:31:18] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:08 [I 2022-10-08 11:31:18] ...trap/engines/engine.py.229: process: 0.78332 | load: 0.00340 [I 2022-10-08 11:31:18] ...trap/engines/engine.py.230: loss: 0.17335 [I 2022-10-08 11:31:18] ...optimizers/trijoint.py.151: total_norm: 2.571766 [I 2022-10-08 11:31:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 150/2304 [I 2022-10-08 11:31:25] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:00 [I 2022-10-08 11:31:25] ...trap/engines/engine.py.229: process: 0.77906 | load: 0.00315 [I 2022-10-08 11:31:25] ...trap/engines/engine.py.230: loss: 0.23454 [I 2022-10-08 11:31:25] ...optimizers/trijoint.py.151: total_norm: 2.273641 [I 2022-10-08 11:31:33] ...trap/engines/engine.py.224: train: epoch 3 | batch 160/2304 [I 2022-10-08 11:31:33] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:48 [I 2022-10-08 11:31:33] ...trap/engines/engine.py.229: process: 0.76199 | load: 0.00308 [I 2022-10-08 11:31:33] ...trap/engines/engine.py.230: loss: 0.25746 [I 2022-10-08 11:31:33] ...optimizers/trijoint.py.151: total_norm: 3.330590 [I 2022-10-08 11:31:41] ...trap/engines/engine.py.224: train: epoch 3 | batch 170/2304 [I 2022-10-08 11:31:41] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:27:37 [I 2022-10-08 11:31:41] ...trap/engines/engine.py.229: process: 0.77679 | load: 0.00313 [I 2022-10-08 11:31:41] ...trap/engines/engine.py.230: loss: 0.31078 [I 2022-10-08 11:31:41] ...optimizers/trijoint.py.151: total_norm: 2.553160 [I 2022-10-08 11:31:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 180/2304 [I 2022-10-08 11:31:49] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:31 [I 2022-10-08 11:31:49] ...trap/engines/engine.py.229: process: 0.79010 | load: 0.00311 [I 2022-10-08 11:31:49] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-08 11:31:49] ...optimizers/trijoint.py.151: total_norm: 2.532912 [I 2022-10-08 11:31:57] ...trap/engines/engine.py.224: train: epoch 3 | batch 190/2304 [I 2022-10-08 11:31:57] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:24 [I 2022-10-08 11:31:57] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00321 [I 2022-10-08 11:31:57] ...trap/engines/engine.py.230: loss: 0.30150 [I 2022-10-08 11:31:57] ...optimizers/trijoint.py.151: total_norm: 3.511612 [I 2022-10-08 11:32:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 200/2304 [I 2022-10-08 11:32:04] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:11 [I 2022-10-08 11:32:04] ...trap/engines/engine.py.229: process: 0.76746 | load: 0.00324 [I 2022-10-08 11:32:04] ...trap/engines/engine.py.230: loss: 0.28910 [I 2022-10-08 11:32:04] ...optimizers/trijoint.py.151: total_norm: 3.557625 [I 2022-10-08 11:32:12] ...trap/engines/engine.py.224: train: epoch 3 | batch 210/2304 [I 2022-10-08 11:32:12] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:26:59 [I 2022-10-08 11:32:12] ...trap/engines/engine.py.229: process: 0.76126 | load: 0.00334 [I 2022-10-08 11:32:12] ...trap/engines/engine.py.230: loss: 0.20623 [I 2022-10-08 11:32:12] ...optimizers/trijoint.py.151: total_norm: 2.663720 [I 2022-10-08 11:32:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 220/2304 [I 2022-10-08 11:32:20] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:06 [I 2022-10-08 11:32:20] ...trap/engines/engine.py.229: process: 0.78025 | load: 0.00300 [I 2022-10-08 11:32:20] ...trap/engines/engine.py.230: loss: 0.23368 [I 2022-10-08 11:32:20] ...optimizers/trijoint.py.151: total_norm: 3.000216 [I 2022-10-08 11:32:28] ...trap/engines/engine.py.224: train: epoch 3 | batch 230/2304 [I 2022-10-08 11:32:28] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:53 [I 2022-10-08 11:32:28] ...trap/engines/engine.py.229: process: 0.78001 | load: 0.00323 [I 2022-10-08 11:32:28] ...trap/engines/engine.py.230: loss: 0.28360 [I 2022-10-08 11:32:28] ...optimizers/trijoint.py.151: total_norm: 3.529586 [I 2022-10-08 11:32:36] ...trap/engines/engine.py.224: train: epoch 3 | batch 240/2304 [I 2022-10-08 11:32:36] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:43 [I 2022-10-08 11:32:36] ...trap/engines/engine.py.229: process: 0.78031 | load: 0.00302 [I 2022-10-08 11:32:36] ...trap/engines/engine.py.230: loss: 0.27336 [I 2022-10-08 11:32:36] ...optimizers/trijoint.py.151: total_norm: 3.508359 [I 2022-10-08 11:32:43] ...trap/engines/engine.py.224: train: epoch 3 | batch 250/2304 [I 2022-10-08 11:32:43] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:36 [I 2022-10-08 11:32:43] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00298 [I 2022-10-08 11:32:43] ...trap/engines/engine.py.230: loss: 0.21013 [I 2022-10-08 11:32:43] ...optimizers/trijoint.py.151: total_norm: 3.215239 [I 2022-10-08 11:32:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 260/2304 [I 2022-10-08 11:32:51] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:35 [I 2022-10-08 11:32:51] ...trap/engines/engine.py.229: process: 0.78920 | load: 0.00301 [I 2022-10-08 11:32:51] ...trap/engines/engine.py.230: loss: 0.23929 [I 2022-10-08 11:32:51] ...optimizers/trijoint.py.151: total_norm: 2.877984 [I 2022-10-08 11:32:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 270/2304 [I 2022-10-08 11:32:59] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:23 [I 2022-10-08 11:32:59] ...trap/engines/engine.py.229: process: 0.78165 | load: 0.00315 [I 2022-10-08 11:32:59] ...trap/engines/engine.py.230: loss: 0.21723 [I 2022-10-08 11:32:59] ...optimizers/trijoint.py.151: total_norm: 2.780590 [I 2022-10-08 11:33:07] ...trap/engines/engine.py.224: train: epoch 3 | batch 280/2304 [I 2022-10-08 11:33:07] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:20 [I 2022-10-08 11:33:07] ...trap/engines/engine.py.229: process: 0.79881 | load: 0.00312 [I 2022-10-08 11:33:07] ...trap/engines/engine.py.230: loss: 0.23973 [I 2022-10-08 11:33:07] ...optimizers/trijoint.py.151: total_norm: 2.241926 [I 2022-10-08 11:33:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 290/2304 [I 2022-10-08 11:33:15] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:16 [I 2022-10-08 11:33:15] ...trap/engines/engine.py.229: process: 0.78010 | load: 0.00309 [I 2022-10-08 11:33:15] ...trap/engines/engine.py.230: loss: 0.14938 [I 2022-10-08 11:33:15] ...optimizers/trijoint.py.151: total_norm: 1.924459 [I 2022-10-08 11:33:22] ...trap/engines/engine.py.224: train: epoch 3 | batch 300/2304 [I 2022-10-08 11:33:22] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:26:01 [I 2022-10-08 11:33:22] ...trap/engines/engine.py.229: process: 0.77628 | load: 0.00306 [I 2022-10-08 11:33:22] ...trap/engines/engine.py.230: loss: 0.27316 [I 2022-10-08 11:33:22] ...optimizers/trijoint.py.151: total_norm: 3.289500 [I 2022-10-08 11:33:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 310/2304 [I 2022-10-08 11:33:30] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:57 [I 2022-10-08 11:33:30] ...trap/engines/engine.py.229: process: 0.78427 | load: 0.00316 [I 2022-10-08 11:33:30] ...trap/engines/engine.py.230: loss: 0.27528 [I 2022-10-08 11:33:30] ...optimizers/trijoint.py.151: total_norm: 3.023850 [I 2022-10-08 11:33:38] ...trap/engines/engine.py.224: train: epoch 3 | batch 320/2304 [I 2022-10-08 11:33:38] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:45 [I 2022-10-08 11:33:38] ...trap/engines/engine.py.229: process: 0.78736 | load: 0.00303 [I 2022-10-08 11:33:38] ...trap/engines/engine.py.230: loss: 0.21730 [I 2022-10-08 11:33:38] ...optimizers/trijoint.py.151: total_norm: 2.859881 [I 2022-10-08 11:33:46] ...trap/engines/engine.py.224: train: epoch 3 | batch 330/2304 [I 2022-10-08 11:33:46] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:42 [I 2022-10-08 11:33:46] ...trap/engines/engine.py.229: process: 0.78627 | load: 0.00299 [I 2022-10-08 11:33:46] ...trap/engines/engine.py.230: loss: 0.21533 [I 2022-10-08 11:33:46] ...optimizers/trijoint.py.151: total_norm: 2.839717 [I 2022-10-08 11:33:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 340/2304 [I 2022-10-08 11:33:54] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:35 [I 2022-10-08 11:33:54] ...trap/engines/engine.py.229: process: 0.78980 | load: 0.00305 [I 2022-10-08 11:33:54] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-08 11:33:54] ...optimizers/trijoint.py.151: total_norm: 2.777355 [I 2022-10-08 11:34:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 350/2304 [I 2022-10-08 11:34:01] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:27 [I 2022-10-08 11:34:01] ...trap/engines/engine.py.229: process: 0.78350 | load: 0.00301 [I 2022-10-08 11:34:01] ...trap/engines/engine.py.230: loss: 0.24152 [I 2022-10-08 11:34:01] ...optimizers/trijoint.py.151: total_norm: 2.995445 [I 2022-10-08 11:34:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 360/2304 [I 2022-10-08 11:34:09] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:16 [I 2022-10-08 11:34:09] ...trap/engines/engine.py.229: process: 0.78376 | load: 0.00302 [I 2022-10-08 11:34:09] ...trap/engines/engine.py.230: loss: 0.23594 [I 2022-10-08 11:34:09] ...optimizers/trijoint.py.151: total_norm: 3.102102 [I 2022-10-08 11:34:17] ...trap/engines/engine.py.224: train: epoch 3 | batch 370/2304 [I 2022-10-08 11:34:17] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:24:54 [I 2022-10-08 11:34:17] ...trap/engines/engine.py.229: process: 0.77495 | load: 0.00313 [I 2022-10-08 11:34:17] ...trap/engines/engine.py.230: loss: 0.21734 [I 2022-10-08 11:34:17] ...optimizers/trijoint.py.151: total_norm: 3.250327 [I 2022-10-08 11:34:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 380/2304 [I 2022-10-08 11:34:25] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:01 [I 2022-10-08 11:34:25] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00313 [I 2022-10-08 11:34:25] ...trap/engines/engine.py.230: loss: 0.24350 [I 2022-10-08 11:34:25] ...optimizers/trijoint.py.151: total_norm: 2.578411 [I 2022-10-08 11:34:33] ...trap/engines/engine.py.224: train: epoch 3 | batch 390/2304 [I 2022-10-08 11:34:33] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:50 [I 2022-10-08 11:34:33] ...trap/engines/engine.py.229: process: 0.77738 | load: 0.00315 [I 2022-10-08 11:34:33] ...trap/engines/engine.py.230: loss: 0.27682 [I 2022-10-08 11:34:33] ...optimizers/trijoint.py.151: total_norm: 3.264866 [I 2022-10-08 11:34:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 400/2304 [I 2022-10-08 11:34:40] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:51 [I 2022-10-08 11:34:40] ...trap/engines/engine.py.229: process: 0.79696 | load: 0.00310 [I 2022-10-08 11:34:40] ...trap/engines/engine.py.230: loss: 0.24897 [I 2022-10-08 11:34:40] ...optimizers/trijoint.py.151: total_norm: 2.618780 [I 2022-10-08 11:34:48] ...trap/engines/engine.py.224: train: epoch 3 | batch 410/2304 [I 2022-10-08 11:34:48] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:42 [I 2022-10-08 11:34:48] ...trap/engines/engine.py.229: process: 0.79053 | load: 0.00308 [I 2022-10-08 11:34:48] ...trap/engines/engine.py.230: loss: 0.22502 [I 2022-10-08 11:34:48] ...optimizers/trijoint.py.151: total_norm: 3.246143 [I 2022-10-08 11:34:56] ...trap/engines/engine.py.224: train: epoch 3 | batch 420/2304 [I 2022-10-08 11:34:56] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:29 [I 2022-10-08 11:34:56] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00307 [I 2022-10-08 11:34:56] ...trap/engines/engine.py.230: loss: 0.30026 [I 2022-10-08 11:34:56] ...optimizers/trijoint.py.151: total_norm: 2.766592 [I 2022-10-08 11:35:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 430/2304 [I 2022-10-08 11:35:04] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:20 [I 2022-10-08 11:35:04] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00312 [I 2022-10-08 11:35:04] ...trap/engines/engine.py.230: loss: 0.24157 [I 2022-10-08 11:35:04] ...optimizers/trijoint.py.151: total_norm: 2.838817 [I 2022-10-08 11:35:12] ...trap/engines/engine.py.224: train: epoch 3 | batch 440/2304 [I 2022-10-08 11:35:12] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:13 [I 2022-10-08 11:35:12] ...trap/engines/engine.py.229: process: 0.78534 | load: 0.00309 [I 2022-10-08 11:35:12] ...trap/engines/engine.py.230: loss: 0.28747 [I 2022-10-08 11:35:12] ...optimizers/trijoint.py.151: total_norm: 2.952208 [I 2022-10-08 11:35:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 450/2304 [I 2022-10-08 11:35:19] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:04 [I 2022-10-08 11:35:19] ...trap/engines/engine.py.229: process: 0.78041 | load: 0.00313 [I 2022-10-08 11:35:19] ...trap/engines/engine.py.230: loss: 0.29323 [I 2022-10-08 11:35:19] ...optimizers/trijoint.py.151: total_norm: 3.813285 [I 2022-10-08 11:35:27] ...trap/engines/engine.py.224: train: epoch 3 | batch 460/2304 [I 2022-10-08 11:35:27] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:52 [I 2022-10-08 11:35:27] ...trap/engines/engine.py.229: process: 0.78586 | load: 0.00344 [I 2022-10-08 11:35:27] ...trap/engines/engine.py.230: loss: 0.24996 [I 2022-10-08 11:35:27] ...optimizers/trijoint.py.151: total_norm: 2.619382 [I 2022-10-08 11:35:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 470/2304 [I 2022-10-08 11:35:35] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:44 [I 2022-10-08 11:35:35] ...trap/engines/engine.py.229: process: 0.78250 | load: 0.00312 [I 2022-10-08 11:35:35] ...trap/engines/engine.py.230: loss: 0.22208 [I 2022-10-08 11:35:35] ...optimizers/trijoint.py.151: total_norm: 2.421796 [I 2022-10-08 11:35:43] ...trap/engines/engine.py.224: train: epoch 3 | batch 480/2304 [I 2022-10-08 11:35:43] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:22 [I 2022-10-08 11:35:43] ...trap/engines/engine.py.229: process: 0.75653 | load: 0.00316 [I 2022-10-08 11:35:43] ...trap/engines/engine.py.230: loss: 0.27964 [I 2022-10-08 11:35:43] ...optimizers/trijoint.py.151: total_norm: 2.560191 [I 2022-10-08 11:35:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 490/2304 [I 2022-10-08 11:35:51] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:26 [I 2022-10-08 11:35:51] ...trap/engines/engine.py.229: process: 0.77487 | load: 0.00301 [I 2022-10-08 11:35:51] ...trap/engines/engine.py.230: loss: 0.21234 [I 2022-10-08 11:35:51] ...optimizers/trijoint.py.151: total_norm: 2.642673 [I 2022-10-08 11:35:58] ...trap/engines/engine.py.224: train: epoch 3 | batch 500/2304 [I 2022-10-08 11:35:58] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:21 [I 2022-10-08 11:35:58] ...trap/engines/engine.py.229: process: 0.77482 | load: 0.00301 [I 2022-10-08 11:35:58] ...trap/engines/engine.py.230: loss: 0.17164 [I 2022-10-08 11:35:58] ...optimizers/trijoint.py.151: total_norm: 2.544570 [I 2022-10-08 11:36:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 510/2304 [I 2022-10-08 11:36:06] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:18 [I 2022-10-08 11:36:06] ...trap/engines/engine.py.229: process: 0.79384 | load: 0.00301 [I 2022-10-08 11:36:06] ...trap/engines/engine.py.230: loss: 0.15907 [I 2022-10-08 11:36:06] ...optimizers/trijoint.py.151: total_norm: 2.170603 [I 2022-10-08 11:36:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 520/2304 [I 2022-10-08 11:36:14] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:23:07 [I 2022-10-08 11:36:14] ...trap/engines/engine.py.229: process: 0.77949 | load: 0.00313 [I 2022-10-08 11:36:14] ...trap/engines/engine.py.230: loss: 0.31125 [I 2022-10-08 11:36:14] ...optimizers/trijoint.py.151: total_norm: 3.355775 [I 2022-10-08 11:36:22] ...trap/engines/engine.py.224: train: epoch 3 | batch 530/2304 [I 2022-10-08 11:36:22] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:02 [I 2022-10-08 11:36:22] ...trap/engines/engine.py.229: process: 0.78068 | load: 0.00306 [I 2022-10-08 11:36:22] ...trap/engines/engine.py.230: loss: 0.21781 [I 2022-10-08 11:36:22] ...optimizers/trijoint.py.151: total_norm: 2.965653 [I 2022-10-08 11:36:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 540/2304 [I 2022-10-08 11:36:29] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:47 [I 2022-10-08 11:36:29] ...trap/engines/engine.py.229: process: 0.77984 | load: 0.00300 [I 2022-10-08 11:36:29] ...trap/engines/engine.py.230: loss: 0.28278 [I 2022-10-08 11:36:29] ...optimizers/trijoint.py.151: total_norm: 2.940090 [I 2022-10-08 11:36:37] ...trap/engines/engine.py.224: train: epoch 3 | batch 550/2304 [I 2022-10-08 11:36:37] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:39 [I 2022-10-08 11:36:37] ...trap/engines/engine.py.229: process: 0.76004 | load: 0.00304 [I 2022-10-08 11:36:37] ...trap/engines/engine.py.230: loss: 0.26962 [I 2022-10-08 11:36:37] ...optimizers/trijoint.py.151: total_norm: 3.756159 [I 2022-10-08 11:36:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 560/2304 [I 2022-10-08 11:36:45] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:37 [I 2022-10-08 11:36:45] ...trap/engines/engine.py.229: process: 0.78356 | load: 0.00300 [I 2022-10-08 11:36:45] ...trap/engines/engine.py.230: loss: 0.23367 [I 2022-10-08 11:36:45] ...optimizers/trijoint.py.151: total_norm: 3.583904 [I 2022-10-08 11:36:53] ...trap/engines/engine.py.224: train: epoch 3 | batch 570/2304 [I 2022-10-08 11:36:53] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:22:18 [I 2022-10-08 11:36:53] ...trap/engines/engine.py.229: process: 0.76474 | load: 0.00303 [I 2022-10-08 11:36:53] ...trap/engines/engine.py.230: loss: 0.21581 [I 2022-10-08 11:36:53] ...optimizers/trijoint.py.151: total_norm: 2.628543 [I 2022-10-08 11:37:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 580/2304 [I 2022-10-08 11:37:01] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:17 [I 2022-10-08 11:37:01] ...trap/engines/engine.py.229: process: 0.78395 | load: 0.00302 [I 2022-10-08 11:37:01] ...trap/engines/engine.py.230: loss: 0.16250 [I 2022-10-08 11:37:01] ...optimizers/trijoint.py.151: total_norm: 2.796673 [I 2022-10-08 11:37:08] ...trap/engines/engine.py.224: train: epoch 3 | batch 590/2304 [I 2022-10-08 11:37:08] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:09 [I 2022-10-08 11:37:08] ...trap/engines/engine.py.229: process: 0.76984 | load: 0.00300 [I 2022-10-08 11:37:08] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-08 11:37:08] ...optimizers/trijoint.py.151: total_norm: 2.180057 [I 2022-10-08 11:37:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 600/2304 [I 2022-10-08 11:37:16] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:06 [I 2022-10-08 11:37:16] ...trap/engines/engine.py.229: process: 0.77447 | load: 0.00307 [I 2022-10-08 11:37:16] ...trap/engines/engine.py.230: loss: 0.26164 [I 2022-10-08 11:37:16] ...optimizers/trijoint.py.151: total_norm: 2.819777 [I 2022-10-08 11:37:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 610/2304 [I 2022-10-08 11:37:24] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:04 [I 2022-10-08 11:37:24] ...trap/engines/engine.py.229: process: 0.78669 | load: 0.00301 [I 2022-10-08 11:37:24] ...trap/engines/engine.py.230: loss: 0.22207 [I 2022-10-08 11:37:24] ...optimizers/trijoint.py.151: total_norm: 2.663307 [I 2022-10-08 11:37:32] ...trap/engines/engine.py.224: train: epoch 3 | batch 620/2304 [I 2022-10-08 11:37:32] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:21:49 [I 2022-10-08 11:37:32] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00301 [I 2022-10-08 11:37:32] ...trap/engines/engine.py.230: loss: 0.19818 [I 2022-10-08 11:37:32] ...optimizers/trijoint.py.151: total_norm: 2.726389 [I 2022-10-08 11:37:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 630/2304 [I 2022-10-08 11:37:40] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:21:44 [I 2022-10-08 11:37:40] ...trap/engines/engine.py.229: process: 0.78028 | load: 0.00307 [I 2022-10-08 11:37:40] ...trap/engines/engine.py.230: loss: 0.27955 [I 2022-10-08 11:37:40] ...optimizers/trijoint.py.151: total_norm: 3.068879 [I 2022-10-08 11:37:47] ...trap/engines/engine.py.224: train: epoch 3 | batch 640/2304 [I 2022-10-08 11:37:47] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:43 [I 2022-10-08 11:37:47] ...trap/engines/engine.py.229: process: 0.77820 | load: 0.00304 [I 2022-10-08 11:37:47] ...trap/engines/engine.py.230: loss: 0.20386 [I 2022-10-08 11:37:47] ...optimizers/trijoint.py.151: total_norm: 2.072464 [I 2022-10-08 11:37:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 650/2304 [I 2022-10-08 11:37:55] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:23 [I 2022-10-08 11:37:55] ...trap/engines/engine.py.229: process: 0.76627 | load: 0.00302 [I 2022-10-08 11:37:55] ...trap/engines/engine.py.230: loss: 0.23069 [I 2022-10-08 11:37:55] ...optimizers/trijoint.py.151: total_norm: 2.282360 [I 2022-10-08 11:38:03] ...trap/engines/engine.py.224: train: epoch 3 | batch 660/2304 [I 2022-10-08 11:38:03] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:11 [I 2022-10-08 11:38:03] ...trap/engines/engine.py.229: process: 0.76029 | load: 0.00302 [I 2022-10-08 11:38:03] ...trap/engines/engine.py.230: loss: 0.28094 [I 2022-10-08 11:38:03] ...optimizers/trijoint.py.151: total_norm: 3.691540 [I 2022-10-08 11:38:11] ...trap/engines/engine.py.224: train: epoch 3 | batch 670/2304 [I 2022-10-08 11:38:11] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:21:04 [I 2022-10-08 11:38:11] ...trap/engines/engine.py.229: process: 0.77174 | load: 0.00302 [I 2022-10-08 11:38:11] ...trap/engines/engine.py.230: loss: 0.28631 [I 2022-10-08 11:38:11] ...optimizers/trijoint.py.151: total_norm: 2.973366 [I 2022-10-08 11:38:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 680/2304 [I 2022-10-08 11:38:19] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:21:08 [I 2022-10-08 11:38:19] ...trap/engines/engine.py.229: process: 0.78693 | load: 0.00304 [I 2022-10-08 11:38:19] ...trap/engines/engine.py.230: loss: 0.22030 [I 2022-10-08 11:38:19] ...optimizers/trijoint.py.151: total_norm: 2.843821 [I 2022-10-08 11:38:26] ...trap/engines/engine.py.224: train: epoch 3 | batch 690/2304 [I 2022-10-08 11:38:26] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:55 [I 2022-10-08 11:38:26] ...trap/engines/engine.py.229: process: 0.77608 | load: 0.00300 [I 2022-10-08 11:38:26] ...trap/engines/engine.py.230: loss: 0.19219 [I 2022-10-08 11:38:26] ...optimizers/trijoint.py.151: total_norm: 2.335890 [I 2022-10-08 11:38:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 700/2304 [I 2022-10-08 11:38:34] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:50 [I 2022-10-08 11:38:34] ...trap/engines/engine.py.229: process: 0.77117 | load: 0.00302 [I 2022-10-08 11:38:34] ...trap/engines/engine.py.230: loss: 0.29752 [I 2022-10-08 11:38:34] ...optimizers/trijoint.py.151: total_norm: 3.430203 [I 2022-10-08 11:38:42] ...trap/engines/engine.py.224: train: epoch 3 | batch 710/2304 [I 2022-10-08 11:38:42] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:38 [I 2022-10-08 11:38:42] ...trap/engines/engine.py.229: process: 0.76412 | load: 0.00307 [I 2022-10-08 11:38:42] ...trap/engines/engine.py.230: loss: 0.18002 [I 2022-10-08 11:38:42] ...optimizers/trijoint.py.151: total_norm: 2.845105 [I 2022-10-08 11:38:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 720/2304 [I 2022-10-08 11:38:50] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:20:32 [I 2022-10-08 11:38:50] ...trap/engines/engine.py.229: process: 0.77437 | load: 0.00305 [I 2022-10-08 11:38:50] ...trap/engines/engine.py.230: loss: 0.26834 [I 2022-10-08 11:38:50] ...optimizers/trijoint.py.151: total_norm: 3.304523 [I 2022-10-08 11:38:58] ...trap/engines/engine.py.224: train: epoch 3 | batch 730/2304 [I 2022-10-08 11:38:58] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:20:23 [I 2022-10-08 11:38:58] ...trap/engines/engine.py.229: process: 0.76605 | load: 0.00304 [I 2022-10-08 11:38:58] ...trap/engines/engine.py.230: loss: 0.25021 [I 2022-10-08 11:38:58] ...optimizers/trijoint.py.151: total_norm: 2.762577 [I 2022-10-08 11:39:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 740/2304 [I 2022-10-08 11:39:06] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:19 [I 2022-10-08 11:39:06] ...trap/engines/engine.py.229: process: 0.78146 | load: 0.00298 [I 2022-10-08 11:39:06] ...trap/engines/engine.py.230: loss: 0.19740 [I 2022-10-08 11:39:06] ...optimizers/trijoint.py.151: total_norm: 2.759790 [I 2022-10-08 11:39:13] ...trap/engines/engine.py.224: train: epoch 3 | batch 750/2304 [I 2022-10-08 11:39:13] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:06 [I 2022-10-08 11:39:13] ...trap/engines/engine.py.229: process: 0.76789 | load: 0.00313 [I 2022-10-08 11:39:13] ...trap/engines/engine.py.230: loss: 0.22481 [I 2022-10-08 11:39:13] ...optimizers/trijoint.py.151: total_norm: 2.980605 [I 2022-10-08 11:39:21] ...trap/engines/engine.py.224: train: epoch 3 | batch 760/2304 [I 2022-10-08 11:39:21] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:03 [I 2022-10-08 11:39:21] ...trap/engines/engine.py.229: process: 0.78467 | load: 0.00311 [I 2022-10-08 11:39:21] ...trap/engines/engine.py.230: loss: 0.22661 [I 2022-10-08 11:39:21] ...optimizers/trijoint.py.151: total_norm: 2.435722 [I 2022-10-08 11:39:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 770/2304 [I 2022-10-08 11:39:29] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:20:00 [I 2022-10-08 11:39:29] ...trap/engines/engine.py.229: process: 0.77856 | load: 0.00303 [I 2022-10-08 11:39:29] ...trap/engines/engine.py.230: loss: 0.31296 [I 2022-10-08 11:39:29] ...optimizers/trijoint.py.151: total_norm: 3.394604 [I 2022-10-08 11:39:37] ...trap/engines/engine.py.224: train: epoch 3 | batch 780/2304 [I 2022-10-08 11:39:37] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:19:47 [I 2022-10-08 11:39:37] ...trap/engines/engine.py.229: process: 0.77856 | load: 0.00304 [I 2022-10-08 11:39:37] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-08 11:39:37] ...optimizers/trijoint.py.151: total_norm: 1.812102 [I 2022-10-08 11:39:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 790/2304 [I 2022-10-08 11:39:44] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:27 [I 2022-10-08 11:39:44] ...trap/engines/engine.py.229: process: 0.76764 | load: 0.00302 [I 2022-10-08 11:39:44] ...trap/engines/engine.py.230: loss: 0.23170 [I 2022-10-08 11:39:44] ...optimizers/trijoint.py.151: total_norm: 2.780804 [I 2022-10-08 11:39:52] ...trap/engines/engine.py.224: train: epoch 3 | batch 800/2304 [I 2022-10-08 11:39:52] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:27 [I 2022-10-08 11:39:52] ...trap/engines/engine.py.229: process: 0.77743 | load: 0.00302 [I 2022-10-08 11:39:52] ...trap/engines/engine.py.230: loss: 0.31365 [I 2022-10-08 11:39:52] ...optimizers/trijoint.py.151: total_norm: 3.022609 [I 2022-10-08 11:40:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 810/2304 [I 2022-10-08 11:40:00] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:22 [I 2022-10-08 11:40:00] ...trap/engines/engine.py.229: process: 0.78340 | load: 0.00306 [I 2022-10-08 11:40:00] ...trap/engines/engine.py.230: loss: 0.22133 [I 2022-10-08 11:40:00] ...optimizers/trijoint.py.151: total_norm: 2.913204 [I 2022-10-08 11:40:08] ...trap/engines/engine.py.224: train: epoch 3 | batch 820/2304 [I 2022-10-08 11:40:08] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:19:14 [I 2022-10-08 11:40:08] ...trap/engines/engine.py.229: process: 0.78661 | load: 0.00302 [I 2022-10-08 11:40:08] ...trap/engines/engine.py.230: loss: 0.18711 [I 2022-10-08 11:40:08] ...optimizers/trijoint.py.151: total_norm: 2.437818 [I 2022-10-08 11:40:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 830/2304 [I 2022-10-08 11:40:16] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:18:59 [I 2022-10-08 11:40:16] ...trap/engines/engine.py.229: process: 0.76633 | load: 0.00301 [I 2022-10-08 11:40:16] ...trap/engines/engine.py.230: loss: 0.21676 [I 2022-10-08 11:40:16] ...optimizers/trijoint.py.151: total_norm: 2.796024 [I 2022-10-08 11:40:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 840/2304 [I 2022-10-08 11:40:23] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:18:57 [I 2022-10-08 11:40:23] ...trap/engines/engine.py.229: process: 0.76894 | load: 0.00305 [I 2022-10-08 11:40:23] ...trap/engines/engine.py.230: loss: 0.20149 [I 2022-10-08 11:40:23] ...optimizers/trijoint.py.151: total_norm: 2.754092 [I 2022-10-08 11:40:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 850/2304 [I 2022-10-08 11:40:31] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:45 [I 2022-10-08 11:40:31] ...trap/engines/engine.py.229: process: 0.78069 | load: 0.00300 [I 2022-10-08 11:40:31] ...trap/engines/engine.py.230: loss: 0.24145 [I 2022-10-08 11:40:31] ...optimizers/trijoint.py.151: total_norm: 2.683677 [I 2022-10-08 11:40:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 860/2304 [I 2022-10-08 11:40:39] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:40 [I 2022-10-08 11:40:39] ...trap/engines/engine.py.229: process: 0.76887 | load: 0.00302 [I 2022-10-08 11:40:39] ...trap/engines/engine.py.230: loss: 0.29697 [I 2022-10-08 11:40:39] ...optimizers/trijoint.py.151: total_norm: 3.150833 [I 2022-10-08 11:40:47] ...trap/engines/engine.py.224: train: epoch 3 | batch 870/2304 [I 2022-10-08 11:40:47] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:18:38 [I 2022-10-08 11:40:47] ...trap/engines/engine.py.229: process: 0.77131 | load: 0.00314 [I 2022-10-08 11:40:47] ...trap/engines/engine.py.230: loss: 0.18291 [I 2022-10-08 11:40:47] ...optimizers/trijoint.py.151: total_norm: 2.473726 [I 2022-10-08 11:40:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 880/2304 [I 2022-10-08 11:40:55] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:18:30 [I 2022-10-08 11:40:55] ...trap/engines/engine.py.229: process: 0.78026 | load: 0.00312 [I 2022-10-08 11:40:55] ...trap/engines/engine.py.230: loss: 0.20704 [I 2022-10-08 11:40:55] ...optimizers/trijoint.py.151: total_norm: 2.598332 [I 2022-10-08 11:41:02] ...trap/engines/engine.py.224: train: epoch 3 | batch 890/2304 [I 2022-10-08 11:41:02] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:14 [I 2022-10-08 11:41:02] ...trap/engines/engine.py.229: process: 0.77385 | load: 0.00311 [I 2022-10-08 11:41:02] ...trap/engines/engine.py.230: loss: 0.19269 [I 2022-10-08 11:41:02] ...optimizers/trijoint.py.151: total_norm: 1.865794 [I 2022-10-08 11:41:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 900/2304 [I 2022-10-08 11:41:10] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:16 [I 2022-10-08 11:41:10] ...trap/engines/engine.py.229: process: 0.78384 | load: 0.00311 [I 2022-10-08 11:41:10] ...trap/engines/engine.py.230: loss: 0.29044 [I 2022-10-08 11:41:10] ...optimizers/trijoint.py.151: total_norm: 2.563915 [I 2022-10-08 11:41:18] ...trap/engines/engine.py.224: train: epoch 3 | batch 910/2304 [I 2022-10-08 11:41:18] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:05 [I 2022-10-08 11:41:18] ...trap/engines/engine.py.229: process: 0.78663 | load: 0.00310 [I 2022-10-08 11:41:18] ...trap/engines/engine.py.230: loss: 0.27564 [I 2022-10-08 11:41:18] ...optimizers/trijoint.py.151: total_norm: 3.137227 [I 2022-10-08 11:41:26] ...trap/engines/engine.py.224: train: epoch 3 | batch 920/2304 [I 2022-10-08 11:41:26] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:17:57 [I 2022-10-08 11:41:26] ...trap/engines/engine.py.229: process: 0.77351 | load: 0.00310 [I 2022-10-08 11:41:26] ...trap/engines/engine.py.230: loss: 0.25420 [I 2022-10-08 11:41:26] ...optimizers/trijoint.py.151: total_norm: 2.765030 [I 2022-10-08 11:41:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 930/2304 [I 2022-10-08 11:41:34] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:51 [I 2022-10-08 11:41:34] ...trap/engines/engine.py.229: process: 0.79298 | load: 0.00299 [I 2022-10-08 11:41:34] ...trap/engines/engine.py.230: loss: 0.15231 [I 2022-10-08 11:41:34] ...optimizers/trijoint.py.151: total_norm: 2.026065 [I 2022-10-08 11:41:41] ...trap/engines/engine.py.224: train: epoch 3 | batch 940/2304 [I 2022-10-08 11:41:41] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:39 [I 2022-10-08 11:41:41] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00300 [I 2022-10-08 11:41:41] ...trap/engines/engine.py.230: loss: 0.22378 [I 2022-10-08 11:41:41] ...optimizers/trijoint.py.151: total_norm: 2.375635 [I 2022-10-08 11:41:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 950/2304 [I 2022-10-08 11:41:49] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:27 [I 2022-10-08 11:41:49] ...trap/engines/engine.py.229: process: 0.75512 | load: 0.00303 [I 2022-10-08 11:41:49] ...trap/engines/engine.py.230: loss: 0.17892 [I 2022-10-08 11:41:49] ...optimizers/trijoint.py.151: total_norm: 2.620193 [I 2022-10-08 11:41:57] ...trap/engines/engine.py.224: train: epoch 3 | batch 960/2304 [I 2022-10-08 11:41:57] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:17:27 [I 2022-10-08 11:41:57] ...trap/engines/engine.py.229: process: 0.78693 | load: 0.00304 [I 2022-10-08 11:41:57] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-08 11:41:57] ...optimizers/trijoint.py.151: total_norm: 2.257748 [I 2022-10-08 11:42:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 970/2304 [I 2022-10-08 11:42:05] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:17:16 [I 2022-10-08 11:42:05] ...trap/engines/engine.py.229: process: 0.77359 | load: 0.00305 [I 2022-10-08 11:42:05] ...trap/engines/engine.py.230: loss: 0.24997 [I 2022-10-08 11:42:05] ...optimizers/trijoint.py.151: total_norm: 3.060587 [I 2022-10-08 11:42:13] ...trap/engines/engine.py.224: train: epoch 3 | batch 980/2304 [I 2022-10-08 11:42:13] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:17:10 [I 2022-10-08 11:42:13] ...trap/engines/engine.py.229: process: 0.78191 | load: 0.00301 [I 2022-10-08 11:42:13] ...trap/engines/engine.py.230: loss: 0.21022 [I 2022-10-08 11:42:13] ...optimizers/trijoint.py.151: total_norm: 2.298160 [I 2022-10-08 11:42:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 990/2304 [I 2022-10-08 11:42:20] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:17:03 [I 2022-10-08 11:42:20] ...trap/engines/engine.py.229: process: 0.77573 | load: 0.00300 [I 2022-10-08 11:42:20] ...trap/engines/engine.py.230: loss: 0.22739 [I 2022-10-08 11:42:20] ...optimizers/trijoint.py.151: total_norm: 2.915817 [I 2022-10-08 11:42:28] ...trap/engines/engine.py.224: train: epoch 3 | batch 1000/2304 [I 2022-10-08 11:42:28] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:16:58 [I 2022-10-08 11:42:28] ...trap/engines/engine.py.229: process: 0.78273 | load: 0.00301 [I 2022-10-08 11:42:28] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-08 11:42:28] ...optimizers/trijoint.py.151: total_norm: 1.905737 [I 2022-10-08 11:42:36] ...trap/engines/engine.py.224: train: epoch 3 | batch 1010/2304 [I 2022-10-08 11:42:36] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:16:45 [I 2022-10-08 11:42:36] ...trap/engines/engine.py.229: process: 0.77242 | load: 0.00298 [I 2022-10-08 11:42:36] ...trap/engines/engine.py.230: loss: 0.24046 [I 2022-10-08 11:42:36] ...optimizers/trijoint.py.151: total_norm: 3.099742 [I 2022-10-08 11:42:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1020/2304 [I 2022-10-08 11:42:44] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:16:39 [I 2022-10-08 11:42:44] ...trap/engines/engine.py.229: process: 0.77683 | load: 0.00306 [I 2022-10-08 11:42:44] ...trap/engines/engine.py.230: loss: 0.20985 [I 2022-10-08 11:42:44] ...optimizers/trijoint.py.151: total_norm: 2.950999 [I 2022-10-08 11:42:52] ...trap/engines/engine.py.224: train: epoch 3 | batch 1030/2304 [I 2022-10-08 11:42:52] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:16:34 [I 2022-10-08 11:42:52] ...trap/engines/engine.py.229: process: 0.77098 | load: 0.00302 [I 2022-10-08 11:42:52] ...trap/engines/engine.py.230: loss: 0.21037 [I 2022-10-08 11:42:52] ...optimizers/trijoint.py.151: total_norm: 3.130946 [I 2022-10-08 11:42:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1040/2304 [I 2022-10-08 11:42:59] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:16:30 [I 2022-10-08 11:42:59] ...trap/engines/engine.py.229: process: 0.77770 | load: 0.00302 [I 2022-10-08 11:42:59] ...trap/engines/engine.py.230: loss: 0.24788 [I 2022-10-08 11:42:59] ...optimizers/trijoint.py.151: total_norm: 2.293205 [I 2022-10-08 11:43:07] ...trap/engines/engine.py.224: train: epoch 3 | batch 1050/2304 [I 2022-10-08 11:43:07] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:16:18 [I 2022-10-08 11:43:07] ...trap/engines/engine.py.229: process: 0.77835 | load: 0.00304 [I 2022-10-08 11:43:07] ...trap/engines/engine.py.230: loss: 0.19651 [I 2022-10-08 11:43:07] ...optimizers/trijoint.py.151: total_norm: 2.219217 [I 2022-10-08 11:43:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1060/2304 [I 2022-10-08 11:43:15] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:16:06 [I 2022-10-08 11:43:15] ...trap/engines/engine.py.229: process: 0.78079 | load: 0.00308 [I 2022-10-08 11:43:15] ...trap/engines/engine.py.230: loss: 0.23924 [I 2022-10-08 11:43:15] ...optimizers/trijoint.py.151: total_norm: 3.004742 [I 2022-10-08 11:43:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 1070/2304 [I 2022-10-08 11:43:23] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:16:01 [I 2022-10-08 11:43:23] ...trap/engines/engine.py.229: process: 0.78274 | load: 0.00305 [I 2022-10-08 11:43:23] ...trap/engines/engine.py.230: loss: 0.21559 [I 2022-10-08 11:43:23] ...optimizers/trijoint.py.151: total_norm: 2.993252 [I 2022-10-08 11:43:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 1080/2304 [I 2022-10-08 11:43:31] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:15:52 [I 2022-10-08 11:43:31] ...trap/engines/engine.py.229: process: 0.78044 | load: 0.00304 [I 2022-10-08 11:43:31] ...trap/engines/engine.py.230: loss: 0.20803 [I 2022-10-08 11:43:31] ...optimizers/trijoint.py.151: total_norm: 2.476555 [I 2022-10-08 11:43:38] ...trap/engines/engine.py.224: train: epoch 3 | batch 1090/2304 [I 2022-10-08 11:43:38] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:15:39 [I 2022-10-08 11:43:38] ...trap/engines/engine.py.229: process: 0.78204 | load: 0.00301 [I 2022-10-08 11:43:38] ...trap/engines/engine.py.230: loss: 0.27442 [I 2022-10-08 11:43:38] ...optimizers/trijoint.py.151: total_norm: 2.534036 [I 2022-10-08 11:43:46] ...trap/engines/engine.py.224: train: epoch 3 | batch 1100/2304 [I 2022-10-08 11:43:46] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:15:39 [I 2022-10-08 11:43:46] ...trap/engines/engine.py.229: process: 0.77899 | load: 0.00302 [I 2022-10-08 11:43:46] ...trap/engines/engine.py.230: loss: 0.29822 [I 2022-10-08 11:43:46] ...optimizers/trijoint.py.151: total_norm: 3.549709 [I 2022-10-08 11:43:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1110/2304 [I 2022-10-08 11:43:54] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:15:29 [I 2022-10-08 11:43:54] ...trap/engines/engine.py.229: process: 0.78054 | load: 0.00312 [I 2022-10-08 11:43:54] ...trap/engines/engine.py.230: loss: 0.17554 [I 2022-10-08 11:43:54] ...optimizers/trijoint.py.151: total_norm: 2.363776 [I 2022-10-08 11:44:02] ...trap/engines/engine.py.224: train: epoch 3 | batch 1120/2304 [I 2022-10-08 11:44:02] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:15:18 [I 2022-10-08 11:44:02] ...trap/engines/engine.py.229: process: 0.77590 | load: 0.00311 [I 2022-10-08 11:44:02] ...trap/engines/engine.py.230: loss: 0.26224 [I 2022-10-08 11:44:02] ...optimizers/trijoint.py.151: total_norm: 2.750905 [I 2022-10-08 11:44:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 1130/2304 [I 2022-10-08 11:44:10] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:15:09 [I 2022-10-08 11:44:10] ...trap/engines/engine.py.229: process: 0.77137 | load: 0.00315 [I 2022-10-08 11:44:10] ...trap/engines/engine.py.230: loss: 0.18054 [I 2022-10-08 11:44:10] ...optimizers/trijoint.py.151: total_norm: 2.767632 [I 2022-10-08 11:44:18] ...trap/engines/engine.py.224: train: epoch 3 | batch 1140/2304 [I 2022-10-08 11:44:18] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:15:11 [I 2022-10-08 11:44:18] ...trap/engines/engine.py.229: process: 0.78599 | load: 0.00311 [I 2022-10-08 11:44:18] ...trap/engines/engine.py.230: loss: 0.18572 [I 2022-10-08 11:44:18] ...optimizers/trijoint.py.151: total_norm: 2.545597 [I 2022-10-08 11:44:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 1150/2304 [I 2022-10-08 11:44:25] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:14:55 [I 2022-10-08 11:44:25] ...trap/engines/engine.py.229: process: 0.77143 | load: 0.00300 [I 2022-10-08 11:44:25] ...trap/engines/engine.py.230: loss: 0.24672 [I 2022-10-08 11:44:25] ...optimizers/trijoint.py.151: total_norm: 2.793993 [I 2022-10-08 11:44:33] ...trap/engines/engine.py.224: train: epoch 3 | batch 1160/2304 [I 2022-10-08 11:44:33] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:14:46 [I 2022-10-08 11:44:33] ...trap/engines/engine.py.229: process: 0.77630 | load: 0.00300 [I 2022-10-08 11:44:33] ...trap/engines/engine.py.230: loss: 0.39347 [I 2022-10-08 11:44:33] ...optimizers/trijoint.py.151: total_norm: 3.811207 [I 2022-10-08 11:44:41] ...trap/engines/engine.py.224: train: epoch 3 | batch 1170/2304 [I 2022-10-08 11:44:41] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:14:40 [I 2022-10-08 11:44:41] ...trap/engines/engine.py.229: process: 0.78036 | load: 0.00300 [I 2022-10-08 11:44:41] ...trap/engines/engine.py.230: loss: 0.30901 [I 2022-10-08 11:44:41] ...optimizers/trijoint.py.151: total_norm: 3.500813 [I 2022-10-08 11:44:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1180/2304 [I 2022-10-08 11:44:49] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:14:36 [I 2022-10-08 11:44:49] ...trap/engines/engine.py.229: process: 0.77973 | load: 0.00305 [I 2022-10-08 11:44:49] ...trap/engines/engine.py.230: loss: 0.31390 [I 2022-10-08 11:44:49] ...optimizers/trijoint.py.151: total_norm: 2.705990 [I 2022-10-08 11:44:56] ...trap/engines/engine.py.224: train: epoch 3 | batch 1190/2304 [I 2022-10-08 11:44:56] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:14:23 [I 2022-10-08 11:44:56] ...trap/engines/engine.py.229: process: 0.76764 | load: 0.00303 [I 2022-10-08 11:44:56] ...trap/engines/engine.py.230: loss: 0.24911 [I 2022-10-08 11:44:56] ...optimizers/trijoint.py.151: total_norm: 2.931592 [I 2022-10-08 11:45:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 1200/2304 [I 2022-10-08 11:45:04] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:14:16 [I 2022-10-08 11:45:04] ...trap/engines/engine.py.229: process: 0.77911 | load: 0.00304 [I 2022-10-08 11:45:04] ...trap/engines/engine.py.230: loss: 0.25242 [I 2022-10-08 11:45:04] ...optimizers/trijoint.py.151: total_norm: 2.178616 [I 2022-10-08 11:45:12] ...trap/engines/engine.py.224: train: epoch 3 | batch 1210/2304 [I 2022-10-08 11:45:12] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:14:06 [I 2022-10-08 11:45:12] ...trap/engines/engine.py.229: process: 0.76082 | load: 0.00302 [I 2022-10-08 11:45:12] ...trap/engines/engine.py.230: loss: 0.28203 [I 2022-10-08 11:45:12] ...optimizers/trijoint.py.151: total_norm: 3.195095 [I 2022-10-08 11:45:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 1220/2304 [I 2022-10-08 11:45:20] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:14:05 [I 2022-10-08 11:45:20] ...trap/engines/engine.py.229: process: 0.77569 | load: 0.00303 [I 2022-10-08 11:45:20] ...trap/engines/engine.py.230: loss: 0.21665 [I 2022-10-08 11:45:20] ...optimizers/trijoint.py.151: total_norm: 2.623744 [I 2022-10-08 11:45:28] ...trap/engines/engine.py.224: train: epoch 3 | batch 1230/2304 [I 2022-10-08 11:45:28] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:13:50 [I 2022-10-08 11:45:28] ...trap/engines/engine.py.229: process: 0.76840 | load: 0.00303 [I 2022-10-08 11:45:28] ...trap/engines/engine.py.230: loss: 0.21869 [I 2022-10-08 11:45:28] ...optimizers/trijoint.py.151: total_norm: 2.635515 [I 2022-10-08 11:45:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 1240/2304 [I 2022-10-08 11:45:35] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:13:41 [I 2022-10-08 11:45:35] ...trap/engines/engine.py.229: process: 0.76851 | load: 0.00302 [I 2022-10-08 11:45:35] ...trap/engines/engine.py.230: loss: 0.20807 [I 2022-10-08 11:45:35] ...optimizers/trijoint.py.151: total_norm: 2.223972 [I 2022-10-08 11:45:43] ...trap/engines/engine.py.224: train: epoch 3 | batch 1250/2304 [I 2022-10-08 11:45:43] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:13:38 [I 2022-10-08 11:45:43] ...trap/engines/engine.py.229: process: 0.76150 | load: 0.00305 [I 2022-10-08 11:45:43] ...trap/engines/engine.py.230: loss: 0.13161 [I 2022-10-08 11:45:43] ...optimizers/trijoint.py.151: total_norm: 2.117028 [I 2022-10-08 11:45:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 1260/2304 [I 2022-10-08 11:45:51] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:13:31 [I 2022-10-08 11:45:51] ...trap/engines/engine.py.229: process: 0.79079 | load: 0.00299 [I 2022-10-08 11:45:51] ...trap/engines/engine.py.230: loss: 0.27919 [I 2022-10-08 11:45:51] ...optimizers/trijoint.py.151: total_norm: 3.210231 [I 2022-10-08 11:45:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1270/2304 [I 2022-10-08 11:45:59] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:13:18 [I 2022-10-08 11:45:59] ...trap/engines/engine.py.229: process: 0.76267 | load: 0.00301 [I 2022-10-08 11:45:59] ...trap/engines/engine.py.230: loss: 0.20004 [I 2022-10-08 11:45:59] ...optimizers/trijoint.py.151: total_norm: 3.258150 [I 2022-10-08 11:46:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 1280/2304 [I 2022-10-08 11:46:06] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:13:14 [I 2022-10-08 11:46:06] ...trap/engines/engine.py.229: process: 0.78081 | load: 0.00300 [I 2022-10-08 11:46:06] ...trap/engines/engine.py.230: loss: 0.20288 [I 2022-10-08 11:46:06] ...optimizers/trijoint.py.151: total_norm: 2.461068 [I 2022-10-08 11:46:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 1290/2304 [I 2022-10-08 11:46:14] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:13:04 [I 2022-10-08 11:46:14] ...trap/engines/engine.py.229: process: 0.76920 | load: 0.00300 [I 2022-10-08 11:46:14] ...trap/engines/engine.py.230: loss: 0.31660 [I 2022-10-08 11:46:14] ...optimizers/trijoint.py.151: total_norm: 3.515652 [I 2022-10-08 11:46:22] ...trap/engines/engine.py.224: train: epoch 3 | batch 1300/2304 [I 2022-10-08 11:46:22] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:12:59 [I 2022-10-08 11:46:22] ...trap/engines/engine.py.229: process: 0.77918 | load: 0.00302 [I 2022-10-08 11:46:22] ...trap/engines/engine.py.230: loss: 0.19847 [I 2022-10-08 11:46:22] ...optimizers/trijoint.py.151: total_norm: 2.415719 [I 2022-10-08 11:46:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1310/2304 [I 2022-10-08 11:46:30] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:12:50 [I 2022-10-08 11:46:30] ...trap/engines/engine.py.229: process: 0.76716 | load: 0.00302 [I 2022-10-08 11:46:30] ...trap/engines/engine.py.230: loss: 0.17684 [I 2022-10-08 11:46:30] ...optimizers/trijoint.py.151: total_norm: 2.809196 [I 2022-10-08 11:46:37] ...trap/engines/engine.py.224: train: epoch 3 | batch 1320/2304 [I 2022-10-08 11:46:37] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:12:50 [I 2022-10-08 11:46:37] ...trap/engines/engine.py.229: process: 0.77339 | load: 0.00300 [I 2022-10-08 11:46:37] ...trap/engines/engine.py.230: loss: 0.24380 [I 2022-10-08 11:46:37] ...optimizers/trijoint.py.151: total_norm: 2.430993 [I 2022-10-08 11:46:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 1330/2304 [I 2022-10-08 11:46:45] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:12:39 [I 2022-10-08 11:46:45] ...trap/engines/engine.py.229: process: 0.78068 | load: 0.00301 [I 2022-10-08 11:46:45] ...trap/engines/engine.py.230: loss: 0.25115 [I 2022-10-08 11:46:45] ...optimizers/trijoint.py.151: total_norm: 3.140545 [I 2022-10-08 11:46:53] ...trap/engines/engine.py.224: train: epoch 3 | batch 1340/2304 [I 2022-10-08 11:46:53] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:12:29 [I 2022-10-08 11:46:53] ...trap/engines/engine.py.229: process: 0.76949 | load: 0.00301 [I 2022-10-08 11:46:53] ...trap/engines/engine.py.230: loss: 0.18842 [I 2022-10-08 11:46:53] ...optimizers/trijoint.py.151: total_norm: 2.690629 [I 2022-10-08 11:47:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 1350/2304 [I 2022-10-08 11:47:01] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:12:21 [I 2022-10-08 11:47:01] ...trap/engines/engine.py.229: process: 0.77851 | load: 0.00302 [I 2022-10-08 11:47:01] ...trap/engines/engine.py.230: loss: 0.26478 [I 2022-10-08 11:47:01] ...optimizers/trijoint.py.151: total_norm: 2.920476 [I 2022-10-08 11:47:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 1360/2304 [I 2022-10-08 11:47:09] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:12:10 [I 2022-10-08 11:47:09] ...trap/engines/engine.py.229: process: 0.76937 | load: 0.00302 [I 2022-10-08 11:47:09] ...trap/engines/engine.py.230: loss: 0.30124 [I 2022-10-08 11:47:09] ...optimizers/trijoint.py.151: total_norm: 2.908751 [I 2022-10-08 11:47:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 1370/2304 [I 2022-10-08 11:47:16] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:12:07 [I 2022-10-08 11:47:16] ...trap/engines/engine.py.229: process: 0.78138 | load: 0.00300 [I 2022-10-08 11:47:16] ...trap/engines/engine.py.230: loss: 0.24111 [I 2022-10-08 11:47:16] ...optimizers/trijoint.py.151: total_norm: 2.480369 [I 2022-10-08 11:47:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 1380/2304 [I 2022-10-08 11:47:24] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:12:04 [I 2022-10-08 11:47:24] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00300 [I 2022-10-08 11:47:24] ...trap/engines/engine.py.230: loss: 0.31102 [I 2022-10-08 11:47:24] ...optimizers/trijoint.py.151: total_norm: 3.037112 [I 2022-10-08 11:47:32] ...trap/engines/engine.py.224: train: epoch 3 | batch 1390/2304 [I 2022-10-08 11:47:32] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:11:55 [I 2022-10-08 11:47:32] ...trap/engines/engine.py.229: process: 0.78356 | load: 0.00305 [I 2022-10-08 11:47:32] ...trap/engines/engine.py.230: loss: 0.24263 [I 2022-10-08 11:47:32] ...optimizers/trijoint.py.151: total_norm: 3.034852 [I 2022-10-08 11:47:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 1400/2304 [I 2022-10-08 11:47:40] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:11:42 [I 2022-10-08 11:47:40] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00302 [I 2022-10-08 11:47:40] ...trap/engines/engine.py.230: loss: 0.18188 [I 2022-10-08 11:47:40] ...optimizers/trijoint.py.151: total_norm: 2.887458 [I 2022-10-08 11:47:48] ...trap/engines/engine.py.224: train: epoch 3 | batch 1410/2304 [I 2022-10-08 11:47:48] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:11:37 [I 2022-10-08 11:47:48] ...trap/engines/engine.py.229: process: 0.77075 | load: 0.00304 [I 2022-10-08 11:47:48] ...trap/engines/engine.py.230: loss: 0.19278 [I 2022-10-08 11:47:48] ...optimizers/trijoint.py.151: total_norm: 2.258298 [I 2022-10-08 11:47:56] ...trap/engines/engine.py.224: train: epoch 3 | batch 1420/2304 [I 2022-10-08 11:47:56] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:11:25 [I 2022-10-08 11:47:56] ...trap/engines/engine.py.229: process: 0.77652 | load: 0.00299 [I 2022-10-08 11:47:56] ...trap/engines/engine.py.230: loss: 0.19059 [I 2022-10-08 11:47:56] ...optimizers/trijoint.py.151: total_norm: 2.444891 [I 2022-10-08 11:48:03] ...trap/engines/engine.py.224: train: epoch 3 | batch 1430/2304 [I 2022-10-08 11:48:03] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:11:15 [I 2022-10-08 11:48:03] ...trap/engines/engine.py.229: process: 0.76858 | load: 0.00303 [I 2022-10-08 11:48:03] ...trap/engines/engine.py.230: loss: 0.24036 [I 2022-10-08 11:48:03] ...optimizers/trijoint.py.151: total_norm: 3.189651 [I 2022-10-08 11:48:11] ...trap/engines/engine.py.224: train: epoch 3 | batch 1440/2304 [I 2022-10-08 11:48:11] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:11:07 [I 2022-10-08 11:48:11] ...trap/engines/engine.py.229: process: 0.77623 | load: 0.00297 [I 2022-10-08 11:48:11] ...trap/engines/engine.py.230: loss: 0.16324 [I 2022-10-08 11:48:11] ...optimizers/trijoint.py.151: total_norm: 2.458990 [I 2022-10-08 11:48:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 1450/2304 [I 2022-10-08 11:48:19] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:11:04 [I 2022-10-08 11:48:19] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00300 [I 2022-10-08 11:48:19] ...trap/engines/engine.py.230: loss: 0.27365 [I 2022-10-08 11:48:19] ...optimizers/trijoint.py.151: total_norm: 2.831047 [I 2022-10-08 11:48:27] ...trap/engines/engine.py.224: train: epoch 3 | batch 1460/2304 [I 2022-10-08 11:48:27] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:10:57 [I 2022-10-08 11:48:27] ...trap/engines/engine.py.229: process: 0.78376 | load: 0.00304 [I 2022-10-08 11:48:27] ...trap/engines/engine.py.230: loss: 0.24097 [I 2022-10-08 11:48:27] ...optimizers/trijoint.py.151: total_norm: 2.524712 [I 2022-10-08 11:48:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1470/2304 [I 2022-10-08 11:48:34] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:10:47 [I 2022-10-08 11:48:34] ...trap/engines/engine.py.229: process: 0.77562 | load: 0.00301 [I 2022-10-08 11:48:34] ...trap/engines/engine.py.230: loss: 0.21775 [I 2022-10-08 11:48:34] ...optimizers/trijoint.py.151: total_norm: 2.888353 [I 2022-10-08 11:48:42] ...trap/engines/engine.py.224: train: epoch 3 | batch 1480/2304 [I 2022-10-08 11:48:42] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:10:39 [I 2022-10-08 11:48:42] ...trap/engines/engine.py.229: process: 0.77709 | load: 0.00305 [I 2022-10-08 11:48:42] ...trap/engines/engine.py.230: loss: 0.20616 [I 2022-10-08 11:48:42] ...optimizers/trijoint.py.151: total_norm: 3.064951 [I 2022-10-08 11:48:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 1490/2304 [I 2022-10-08 11:48:50] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:10:31 [I 2022-10-08 11:48:50] ...trap/engines/engine.py.229: process: 0.77728 | load: 0.00302 [I 2022-10-08 11:48:50] ...trap/engines/engine.py.230: loss: 0.32001 [I 2022-10-08 11:48:50] ...optimizers/trijoint.py.151: total_norm: 3.598293 [I 2022-10-08 11:48:58] ...trap/engines/engine.py.224: train: epoch 3 | batch 1500/2304 [I 2022-10-08 11:48:58] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:10:19 [I 2022-10-08 11:48:58] ...trap/engines/engine.py.229: process: 0.76422 | load: 0.00299 [I 2022-10-08 11:48:58] ...trap/engines/engine.py.230: loss: 0.24344 [I 2022-10-08 11:48:58] ...optimizers/trijoint.py.151: total_norm: 2.601335 [I 2022-10-08 11:49:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 1510/2304 [I 2022-10-08 11:49:05] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:10:15 [I 2022-10-08 11:49:05] ...trap/engines/engine.py.229: process: 0.77901 | load: 0.00306 [I 2022-10-08 11:49:05] ...trap/engines/engine.py.230: loss: 0.27713 [I 2022-10-08 11:49:05] ...optimizers/trijoint.py.151: total_norm: 3.118859 [I 2022-10-08 11:49:13] ...trap/engines/engine.py.224: train: epoch 3 | batch 1520/2304 [I 2022-10-08 11:49:13] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:10:08 [I 2022-10-08 11:49:13] ...trap/engines/engine.py.229: process: 0.76455 | load: 0.00303 [I 2022-10-08 11:49:13] ...trap/engines/engine.py.230: loss: 0.27454 [I 2022-10-08 11:49:13] ...optimizers/trijoint.py.151: total_norm: 2.948475 [I 2022-10-08 11:49:21] ...trap/engines/engine.py.224: train: epoch 3 | batch 1530/2304 [I 2022-10-08 11:49:21] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:10:03 [I 2022-10-08 11:49:21] ...trap/engines/engine.py.229: process: 0.78449 | load: 0.00303 [I 2022-10-08 11:49:21] ...trap/engines/engine.py.230: loss: 0.35306 [I 2022-10-08 11:49:21] ...optimizers/trijoint.py.151: total_norm: 3.632803 [I 2022-10-08 11:49:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 1540/2304 [I 2022-10-08 11:49:29] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:09:49 [I 2022-10-08 11:49:29] ...trap/engines/engine.py.229: process: 0.75797 | load: 0.00313 [I 2022-10-08 11:49:29] ...trap/engines/engine.py.230: loss: 0.18868 [I 2022-10-08 11:49:29] ...optimizers/trijoint.py.151: total_norm: 2.494064 [I 2022-10-08 11:49:37] ...trap/engines/engine.py.224: train: epoch 3 | batch 1550/2304 [I 2022-10-08 11:49:37] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:09:44 [I 2022-10-08 11:49:37] ...trap/engines/engine.py.229: process: 0.77683 | load: 0.00305 [I 2022-10-08 11:49:37] ...trap/engines/engine.py.230: loss: 0.25982 [I 2022-10-08 11:49:37] ...optimizers/trijoint.py.151: total_norm: 2.864882 [I 2022-10-08 11:49:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1560/2304 [I 2022-10-08 11:49:44] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:09:41 [I 2022-10-08 11:49:44] ...trap/engines/engine.py.229: process: 0.78999 | load: 0.00302 [I 2022-10-08 11:49:44] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-08 11:49:44] ...optimizers/trijoint.py.151: total_norm: 2.528814 [I 2022-10-08 11:49:52] ...trap/engines/engine.py.224: train: epoch 3 | batch 1570/2304 [I 2022-10-08 11:49:52] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:09:33 [I 2022-10-08 11:49:52] ...trap/engines/engine.py.229: process: 0.78073 | load: 0.00309 [I 2022-10-08 11:49:52] ...trap/engines/engine.py.230: loss: 0.32692 [I 2022-10-08 11:49:52] ...optimizers/trijoint.py.151: total_norm: 3.032014 [I 2022-10-08 11:50:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 1580/2304 [I 2022-10-08 11:50:00] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:09:21 [I 2022-10-08 11:50:00] ...trap/engines/engine.py.229: process: 0.77298 | load: 0.00305 [I 2022-10-08 11:50:00] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-08 11:50:00] ...optimizers/trijoint.py.151: total_norm: 2.107788 [I 2022-10-08 11:50:08] ...trap/engines/engine.py.224: train: epoch 3 | batch 1590/2304 [I 2022-10-08 11:50:08] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:09:12 [I 2022-10-08 11:50:08] ...trap/engines/engine.py.229: process: 0.77746 | load: 0.00303 [I 2022-10-08 11:50:08] ...trap/engines/engine.py.230: loss: 0.16225 [I 2022-10-08 11:50:08] ...optimizers/trijoint.py.151: total_norm: 2.084982 [I 2022-10-08 11:50:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 1600/2304 [I 2022-10-08 11:50:16] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:09:09 [I 2022-10-08 11:50:16] ...trap/engines/engine.py.229: process: 0.78618 | load: 0.00300 [I 2022-10-08 11:50:16] ...trap/engines/engine.py.230: loss: 0.23511 [I 2022-10-08 11:50:16] ...optimizers/trijoint.py.151: total_norm: 3.234127 [I 2022-10-08 11:50:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 1610/2304 [I 2022-10-08 11:50:23] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:09:00 [I 2022-10-08 11:50:23] ...trap/engines/engine.py.229: process: 0.78015 | load: 0.00303 [I 2022-10-08 11:50:23] ...trap/engines/engine.py.230: loss: 0.30699 [I 2022-10-08 11:50:23] ...optimizers/trijoint.py.151: total_norm: 3.694848 [I 2022-10-08 11:50:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 1620/2304 [I 2022-10-08 11:50:31] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:08:51 [I 2022-10-08 11:50:31] ...trap/engines/engine.py.229: process: 0.77282 | load: 0.00303 [I 2022-10-08 11:50:31] ...trap/engines/engine.py.230: loss: 0.23131 [I 2022-10-08 11:50:31] ...optimizers/trijoint.py.151: total_norm: 4.162251 [I 2022-10-08 11:50:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 1630/2304 [I 2022-10-08 11:50:39] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:08:42 [I 2022-10-08 11:50:39] ...trap/engines/engine.py.229: process: 0.76615 | load: 0.00302 [I 2022-10-08 11:50:39] ...trap/engines/engine.py.230: loss: 0.17036 [I 2022-10-08 11:50:39] ...optimizers/trijoint.py.151: total_norm: 2.636870 [I 2022-10-08 11:50:47] ...trap/engines/engine.py.224: train: epoch 3 | batch 1640/2304 [I 2022-10-08 11:50:47] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:08:36 [I 2022-10-08 11:50:47] ...trap/engines/engine.py.229: process: 0.78493 | load: 0.00308 [I 2022-10-08 11:50:47] ...trap/engines/engine.py.230: loss: 0.25715 [I 2022-10-08 11:50:47] ...optimizers/trijoint.py.151: total_norm: 3.519303 [I 2022-10-08 11:50:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1650/2304 [I 2022-10-08 11:50:54] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:08:25 [I 2022-10-08 11:50:54] ...trap/engines/engine.py.229: process: 0.77990 | load: 0.00300 [I 2022-10-08 11:50:54] ...trap/engines/engine.py.230: loss: 0.18738 [I 2022-10-08 11:50:54] ...optimizers/trijoint.py.151: total_norm: 3.311386 [I 2022-10-08 11:51:02] ...trap/engines/engine.py.224: train: epoch 3 | batch 1660/2304 [I 2022-10-08 11:51:02] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:08:20 [I 2022-10-08 11:51:02] ...trap/engines/engine.py.229: process: 0.76224 | load: 0.00311 [I 2022-10-08 11:51:02] ...trap/engines/engine.py.230: loss: 0.20672 [I 2022-10-08 11:51:02] ...optimizers/trijoint.py.151: total_norm: 2.217316 [I 2022-10-08 11:51:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 1670/2304 [I 2022-10-08 11:51:10] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:08:12 [I 2022-10-08 11:51:10] ...trap/engines/engine.py.229: process: 0.77176 | load: 0.00305 [I 2022-10-08 11:51:10] ...trap/engines/engine.py.230: loss: 0.28653 [I 2022-10-08 11:51:10] ...optimizers/trijoint.py.151: total_norm: 2.967114 [I 2022-10-08 11:51:18] ...trap/engines/engine.py.224: train: epoch 3 | batch 1680/2304 [I 2022-10-08 11:51:18] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:08:05 [I 2022-10-08 11:51:18] ...trap/engines/engine.py.229: process: 0.76670 | load: 0.00304 [I 2022-10-08 11:51:18] ...trap/engines/engine.py.230: loss: 0.19370 [I 2022-10-08 11:51:18] ...optimizers/trijoint.py.151: total_norm: 2.662863 [I 2022-10-08 11:51:26] ...trap/engines/engine.py.224: train: epoch 3 | batch 1690/2304 [I 2022-10-08 11:51:26] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:07:59 [I 2022-10-08 11:51:26] ...trap/engines/engine.py.229: process: 0.77797 | load: 0.00300 [I 2022-10-08 11:51:26] ...trap/engines/engine.py.230: loss: 0.23046 [I 2022-10-08 11:51:26] ...optimizers/trijoint.py.151: total_norm: 2.878570 [I 2022-10-08 11:51:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1700/2304 [I 2022-10-08 11:51:34] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:07:48 [I 2022-10-08 11:51:34] ...trap/engines/engine.py.229: process: 0.76602 | load: 0.00304 [I 2022-10-08 11:51:34] ...trap/engines/engine.py.230: loss: 0.20028 [I 2022-10-08 11:51:34] ...optimizers/trijoint.py.151: total_norm: 2.459833 [I 2022-10-08 11:51:41] ...trap/engines/engine.py.224: train: epoch 3 | batch 1710/2304 [I 2022-10-08 11:51:41] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:07:42 [I 2022-10-08 11:51:41] ...trap/engines/engine.py.229: process: 0.77655 | load: 0.00304 [I 2022-10-08 11:51:41] ...trap/engines/engine.py.230: loss: 0.29320 [I 2022-10-08 11:51:41] ...optimizers/trijoint.py.151: total_norm: 2.929155 [I 2022-10-08 11:51:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1720/2304 [I 2022-10-08 11:51:49] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:07:34 [I 2022-10-08 11:51:49] ...trap/engines/engine.py.229: process: 0.77368 | load: 0.00304 [I 2022-10-08 11:51:49] ...trap/engines/engine.py.230: loss: 0.22249 [I 2022-10-08 11:51:49] ...optimizers/trijoint.py.151: total_norm: 2.767480 [I 2022-10-08 11:51:57] ...trap/engines/engine.py.224: train: epoch 3 | batch 1730/2304 [I 2022-10-08 11:51:57] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:07:27 [I 2022-10-08 11:51:57] ...trap/engines/engine.py.229: process: 0.79327 | load: 0.00301 [I 2022-10-08 11:51:57] ...trap/engines/engine.py.230: loss: 0.20429 [I 2022-10-08 11:51:57] ...optimizers/trijoint.py.151: total_norm: 2.792870 [I 2022-10-08 11:52:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 1740/2304 [I 2022-10-08 11:52:05] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:07:17 [I 2022-10-08 11:52:05] ...trap/engines/engine.py.229: process: 0.76910 | load: 0.00301 [I 2022-10-08 11:52:05] ...trap/engines/engine.py.230: loss: 0.23495 [I 2022-10-08 11:52:05] ...optimizers/trijoint.py.151: total_norm: 3.291929 [I 2022-10-08 11:52:12] ...trap/engines/engine.py.224: train: epoch 3 | batch 1750/2304 [I 2022-10-08 11:52:12] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:07:10 [I 2022-10-08 11:52:12] ...trap/engines/engine.py.229: process: 0.76865 | load: 0.00302 [I 2022-10-08 11:52:12] ...trap/engines/engine.py.230: loss: 0.32688 [I 2022-10-08 11:52:12] ...optimizers/trijoint.py.151: total_norm: 3.691999 [I 2022-10-08 11:52:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 1760/2304 [I 2022-10-08 11:52:20] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:07:00 [I 2022-10-08 11:52:20] ...trap/engines/engine.py.229: process: 0.76637 | load: 0.00303 [I 2022-10-08 11:52:20] ...trap/engines/engine.py.230: loss: 0.24012 [I 2022-10-08 11:52:20] ...optimizers/trijoint.py.151: total_norm: 3.450841 [I 2022-10-08 11:52:28] ...trap/engines/engine.py.224: train: epoch 3 | batch 1770/2304 [I 2022-10-08 11:52:28] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:06:54 [I 2022-10-08 11:52:28] ...trap/engines/engine.py.229: process: 0.78624 | load: 0.00302 [I 2022-10-08 11:52:28] ...trap/engines/engine.py.230: loss: 0.21996 [I 2022-10-08 11:52:28] ...optimizers/trijoint.py.151: total_norm: 2.756431 [I 2022-10-08 11:52:36] ...trap/engines/engine.py.224: train: epoch 3 | batch 1780/2304 [I 2022-10-08 11:52:36] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:06:47 [I 2022-10-08 11:52:36] ...trap/engines/engine.py.229: process: 0.77607 | load: 0.00301 [I 2022-10-08 11:52:36] ...trap/engines/engine.py.230: loss: 0.19605 [I 2022-10-08 11:52:36] ...optimizers/trijoint.py.151: total_norm: 2.652695 [I 2022-10-08 11:52:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1790/2304 [I 2022-10-08 11:52:44] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:06:39 [I 2022-10-08 11:52:44] ...trap/engines/engine.py.229: process: 0.76552 | load: 0.00303 [I 2022-10-08 11:52:44] ...trap/engines/engine.py.230: loss: 0.28681 [I 2022-10-08 11:52:44] ...optimizers/trijoint.py.151: total_norm: 3.555712 [I 2022-10-08 11:52:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 1800/2304 [I 2022-10-08 11:52:51] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:06:32 [I 2022-10-08 11:52:51] ...trap/engines/engine.py.229: process: 0.78481 | load: 0.00299 [I 2022-10-08 11:52:51] ...trap/engines/engine.py.230: loss: 0.24759 [I 2022-10-08 11:52:51] ...optimizers/trijoint.py.151: total_norm: 2.379188 [I 2022-10-08 11:52:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1810/2304 [I 2022-10-08 11:52:59] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:06:24 [I 2022-10-08 11:52:59] ...trap/engines/engine.py.229: process: 0.77977 | load: 0.00300 [I 2022-10-08 11:52:59] ...trap/engines/engine.py.230: loss: 0.23811 [I 2022-10-08 11:52:59] ...optimizers/trijoint.py.151: total_norm: 3.072818 [I 2022-10-08 11:53:07] ...trap/engines/engine.py.224: train: epoch 3 | batch 1820/2304 [I 2022-10-08 11:53:07] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:06:16 [I 2022-10-08 11:53:07] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00299 [I 2022-10-08 11:53:07] ...trap/engines/engine.py.230: loss: 0.34069 [I 2022-10-08 11:53:07] ...optimizers/trijoint.py.151: total_norm: 3.904815 [I 2022-10-08 11:53:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1830/2304 [I 2022-10-08 11:53:15] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:06:07 [I 2022-10-08 11:53:15] ...trap/engines/engine.py.229: process: 0.77698 | load: 0.00300 [I 2022-10-08 11:53:15] ...trap/engines/engine.py.230: loss: 0.24560 [I 2022-10-08 11:53:15] ...optimizers/trijoint.py.151: total_norm: 2.667872 [I 2022-10-08 11:53:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 1840/2304 [I 2022-10-08 11:53:23] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:06:01 [I 2022-10-08 11:53:23] ...trap/engines/engine.py.229: process: 0.78824 | load: 0.00305 [I 2022-10-08 11:53:23] ...trap/engines/engine.py.230: loss: 0.19870 [I 2022-10-08 11:53:23] ...optimizers/trijoint.py.151: total_norm: 2.875470 [I 2022-10-08 11:53:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1850/2304 [I 2022-10-08 11:53:30] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:05:51 [I 2022-10-08 11:53:30] ...trap/engines/engine.py.229: process: 0.77882 | load: 0.00302 [I 2022-10-08 11:53:30] ...trap/engines/engine.py.230: loss: 0.22506 [I 2022-10-08 11:53:30] ...optimizers/trijoint.py.151: total_norm: 2.514989 [I 2022-10-08 11:53:38] ...trap/engines/engine.py.224: train: epoch 3 | batch 1860/2304 [I 2022-10-08 11:53:38] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:05:46 [I 2022-10-08 11:53:38] ...trap/engines/engine.py.229: process: 0.78957 | load: 0.00300 [I 2022-10-08 11:53:38] ...trap/engines/engine.py.230: loss: 0.19523 [I 2022-10-08 11:53:38] ...optimizers/trijoint.py.151: total_norm: 2.365752 [I 2022-10-08 11:53:46] ...trap/engines/engine.py.224: train: epoch 3 | batch 1870/2304 [I 2022-10-08 11:53:46] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:05:36 [I 2022-10-08 11:53:46] ...trap/engines/engine.py.229: process: 0.77465 | load: 0.00301 [I 2022-10-08 11:53:46] ...trap/engines/engine.py.230: loss: 0.24733 [I 2022-10-08 11:53:46] ...optimizers/trijoint.py.151: total_norm: 2.928938 [I 2022-10-08 11:53:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1880/2304 [I 2022-10-08 11:53:54] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:05:30 [I 2022-10-08 11:53:54] ...trap/engines/engine.py.229: process: 0.78106 | load: 0.00301 [I 2022-10-08 11:53:54] ...trap/engines/engine.py.230: loss: 0.27783 [I 2022-10-08 11:53:54] ...optimizers/trijoint.py.151: total_norm: 2.960213 [I 2022-10-08 11:54:02] ...trap/engines/engine.py.224: train: epoch 3 | batch 1890/2304 [I 2022-10-08 11:54:02] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:05:20 [I 2022-10-08 11:54:02] ...trap/engines/engine.py.229: process: 0.76279 | load: 0.00300 [I 2022-10-08 11:54:02] ...trap/engines/engine.py.230: loss: 0.20898 [I 2022-10-08 11:54:02] ...optimizers/trijoint.py.151: total_norm: 3.208219 [I 2022-10-08 11:54:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 1900/2304 [I 2022-10-08 11:54:09] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:05:12 [I 2022-10-08 11:54:09] ...trap/engines/engine.py.229: process: 0.77497 | load: 0.00305 [I 2022-10-08 11:54:09] ...trap/engines/engine.py.230: loss: 0.23005 [I 2022-10-08 11:54:09] ...optimizers/trijoint.py.151: total_norm: 2.516956 [I 2022-10-08 11:54:17] ...trap/engines/engine.py.224: train: epoch 3 | batch 1910/2304 [I 2022-10-08 11:54:17] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:05:08 [I 2022-10-08 11:54:17] ...trap/engines/engine.py.229: process: 0.79493 | load: 0.00309 [I 2022-10-08 11:54:17] ...trap/engines/engine.py.230: loss: 0.19321 [I 2022-10-08 11:54:17] ...optimizers/trijoint.py.151: total_norm: 2.661381 [I 2022-10-08 11:54:43] ...trap/engines/engine.py.224: train: epoch 3 | batch 1920/2304 [I 2022-10-08 11:54:43] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:20:00 [I 2022-10-08 11:54:43] ...trap/engines/engine.py.229: process: 0.78198 | load: 0.00314 [I 2022-10-08 11:54:43] ...trap/engines/engine.py.230: loss: 0.29282 [I 2022-10-08 11:54:43] ...optimizers/trijoint.py.151: total_norm: 3.396373 [I 2022-10-08 11:54:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 1930/2304 [I 2022-10-08 11:54:51] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:06:23 [I 2022-10-08 11:54:51] ...trap/engines/engine.py.229: process: 0.76351 | load: 0.00302 [I 2022-10-08 11:54:51] ...trap/engines/engine.py.230: loss: 0.25001 [I 2022-10-08 11:54:51] ...optimizers/trijoint.py.151: total_norm: 2.579132 [I 2022-10-08 11:54:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1940/2304 [I 2022-10-08 11:54:59] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:04:53 [I 2022-10-08 11:54:59] ...trap/engines/engine.py.229: process: 0.77543 | load: 0.00300 [I 2022-10-08 11:54:59] ...trap/engines/engine.py.230: loss: 0.20721 [I 2022-10-08 11:54:59] ...optimizers/trijoint.py.151: total_norm: 3.042233 [I 2022-10-08 11:55:07] ...trap/engines/engine.py.224: train: epoch 3 | batch 1950/2304 [I 2022-10-08 11:55:07] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:04:35 [I 2022-10-08 11:55:07] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00303 [I 2022-10-08 11:55:07] ...trap/engines/engine.py.230: loss: 0.35019 [I 2022-10-08 11:55:07] ...optimizers/trijoint.py.151: total_norm: 2.873187 [I 2022-10-08 11:55:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 1960/2304 [I 2022-10-08 11:55:14] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:04:25 [I 2022-10-08 11:55:14] ...trap/engines/engine.py.229: process: 0.77823 | load: 0.00299 [I 2022-10-08 11:55:14] ...trap/engines/engine.py.230: loss: 0.17342 [I 2022-10-08 11:55:14] ...optimizers/trijoint.py.151: total_norm: 3.029426 [I 2022-10-08 11:55:22] ...trap/engines/engine.py.224: train: epoch 3 | batch 1970/2304 [I 2022-10-08 11:55:22] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:17 [I 2022-10-08 11:55:22] ...trap/engines/engine.py.229: process: 0.76897 | load: 0.00301 [I 2022-10-08 11:55:22] ...trap/engines/engine.py.230: loss: 0.25617 [I 2022-10-08 11:55:22] ...optimizers/trijoint.py.151: total_norm: 3.954613 [I 2022-10-08 11:55:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1980/2304 [I 2022-10-08 11:55:30] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:04:10 [I 2022-10-08 11:55:30] ...trap/engines/engine.py.229: process: 0.77032 | load: 0.00300 [I 2022-10-08 11:55:30] ...trap/engines/engine.py.230: loss: 0.20244 [I 2022-10-08 11:55:30] ...optimizers/trijoint.py.151: total_norm: 2.404150 [I 2022-10-08 11:55:38] ...trap/engines/engine.py.224: train: epoch 3 | batch 1990/2304 [I 2022-10-08 11:55:38] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:04:03 [I 2022-10-08 11:55:38] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00303 [I 2022-10-08 11:55:38] ...trap/engines/engine.py.230: loss: 0.31150 [I 2022-10-08 11:55:38] ...optimizers/trijoint.py.151: total_norm: 2.786892 [I 2022-10-08 11:55:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 2000/2304 [I 2022-10-08 11:55:45] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:03:56 [I 2022-10-08 11:55:45] ...trap/engines/engine.py.229: process: 0.77832 | load: 0.00303 [I 2022-10-08 11:55:45] ...trap/engines/engine.py.230: loss: 0.24407 [I 2022-10-08 11:55:45] ...optimizers/trijoint.py.151: total_norm: 2.805864 [I 2022-10-08 11:55:53] ...trap/engines/engine.py.224: train: epoch 3 | batch 2010/2304 [I 2022-10-08 11:55:53] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:03:46 [I 2022-10-08 11:55:53] ...trap/engines/engine.py.229: process: 0.77308 | load: 0.00305 [I 2022-10-08 11:55:53] ...trap/engines/engine.py.230: loss: 0.26025 [I 2022-10-08 11:55:53] ...optimizers/trijoint.py.151: total_norm: 2.547219 [I 2022-10-08 11:56:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 2020/2304 [I 2022-10-08 11:56:01] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:03:40 [I 2022-10-08 11:56:01] ...trap/engines/engine.py.229: process: 0.76808 | load: 0.00298 [I 2022-10-08 11:56:01] ...trap/engines/engine.py.230: loss: 0.22663 [I 2022-10-08 11:56:01] ...optimizers/trijoint.py.151: total_norm: 2.404054 [I 2022-10-08 11:56:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 2030/2304 [I 2022-10-08 11:56:09] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:03:32 [I 2022-10-08 11:56:09] ...trap/engines/engine.py.229: process: 0.77183 | load: 0.00301 [I 2022-10-08 11:56:09] ...trap/engines/engine.py.230: loss: 0.22959 [I 2022-10-08 11:56:09] ...optimizers/trijoint.py.151: total_norm: 2.758178 [I 2022-10-08 11:56:17] ...trap/engines/engine.py.224: train: epoch 3 | batch 2040/2304 [I 2022-10-08 11:56:17] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:03:24 [I 2022-10-08 11:56:17] ...trap/engines/engine.py.229: process: 0.77256 | load: 0.00301 [I 2022-10-08 11:56:17] ...trap/engines/engine.py.230: loss: 0.17069 [I 2022-10-08 11:56:17] ...optimizers/trijoint.py.151: total_norm: 2.385997 [I 2022-10-08 11:56:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 2050/2304 [I 2022-10-08 11:56:24] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:03:17 [I 2022-10-08 11:56:24] ...trap/engines/engine.py.229: process: 0.78108 | load: 0.00303 [I 2022-10-08 11:56:24] ...trap/engines/engine.py.230: loss: 0.23579 [I 2022-10-08 11:56:24] ...optimizers/trijoint.py.151: total_norm: 3.361578 [I 2022-10-08 11:56:32] ...trap/engines/engine.py.224: train: epoch 3 | batch 2060/2304 [I 2022-10-08 11:56:32] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:03:10 [I 2022-10-08 11:56:32] ...trap/engines/engine.py.229: process: 0.78709 | load: 0.00303 [I 2022-10-08 11:56:32] ...trap/engines/engine.py.230: loss: 0.26353 [I 2022-10-08 11:56:32] ...optimizers/trijoint.py.151: total_norm: 3.064482 [I 2022-10-08 11:56:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 2070/2304 [I 2022-10-08 11:56:40] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:03:01 [I 2022-10-08 11:56:40] ...trap/engines/engine.py.229: process: 0.78402 | load: 0.00305 [I 2022-10-08 11:56:40] ...trap/engines/engine.py.230: loss: 0.19701 [I 2022-10-08 11:56:40] ...optimizers/trijoint.py.151: total_norm: 2.505786 [I 2022-10-08 11:56:48] ...trap/engines/engine.py.224: train: epoch 3 | batch 2080/2304 [I 2022-10-08 11:56:48] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:02:52 [I 2022-10-08 11:56:48] ...trap/engines/engine.py.229: process: 0.76709 | load: 0.00302 [I 2022-10-08 11:56:48] ...trap/engines/engine.py.230: loss: 0.25448 [I 2022-10-08 11:56:48] ...optimizers/trijoint.py.151: total_norm: 3.247035 [I 2022-10-08 11:56:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 2090/2304 [I 2022-10-08 11:56:55] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:02:45 [I 2022-10-08 11:56:55] ...trap/engines/engine.py.229: process: 0.78076 | load: 0.00317 [I 2022-10-08 11:56:55] ...trap/engines/engine.py.230: loss: 0.27105 [I 2022-10-08 11:56:55] ...optimizers/trijoint.py.151: total_norm: 2.742540 [I 2022-10-08 11:57:03] ...trap/engines/engine.py.224: train: epoch 3 | batch 2100/2304 [I 2022-10-08 11:57:03] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:02:40 [I 2022-10-08 11:57:03] ...trap/engines/engine.py.229: process: 0.79764 | load: 0.00305 [I 2022-10-08 11:57:03] ...trap/engines/engine.py.230: loss: 0.14617 [I 2022-10-08 11:57:03] ...optimizers/trijoint.py.151: total_norm: 2.274838 [I 2022-10-08 11:57:11] ...trap/engines/engine.py.224: train: epoch 3 | batch 2110/2304 [I 2022-10-08 11:57:11] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:02:30 [I 2022-10-08 11:57:11] ...trap/engines/engine.py.229: process: 0.76889 | load: 0.00306 [I 2022-10-08 11:57:11] ...trap/engines/engine.py.230: loss: 0.28099 [I 2022-10-08 11:57:11] ...optimizers/trijoint.py.151: total_norm: 2.666374 [I 2022-10-08 11:57:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 2120/2304 [I 2022-10-08 11:57:19] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:02:23 [I 2022-10-08 11:57:19] ...trap/engines/engine.py.229: process: 0.78260 | load: 0.00303 [I 2022-10-08 11:57:19] ...trap/engines/engine.py.230: loss: 0.28391 [I 2022-10-08 11:57:19] ...optimizers/trijoint.py.151: total_norm: 3.331900 [I 2022-10-08 11:57:27] ...trap/engines/engine.py.224: train: epoch 3 | batch 2130/2304 [I 2022-10-08 11:57:27] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:02:14 [I 2022-10-08 11:57:27] ...trap/engines/engine.py.229: process: 0.76240 | load: 0.00309 [I 2022-10-08 11:57:27] ...trap/engines/engine.py.230: loss: 0.33038 [I 2022-10-08 11:57:27] ...optimizers/trijoint.py.151: total_norm: 3.009705 [I 2022-10-08 11:57:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 2140/2304 [I 2022-10-08 11:57:34] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:02:06 [I 2022-10-08 11:57:34] ...trap/engines/engine.py.229: process: 0.77601 | load: 0.00305 [I 2022-10-08 11:57:34] ...trap/engines/engine.py.230: loss: 0.19108 [I 2022-10-08 11:57:34] ...optimizers/trijoint.py.151: total_norm: 2.857376 [I 2022-10-08 11:57:42] ...trap/engines/engine.py.224: train: epoch 3 | batch 2150/2304 [I 2022-10-08 11:57:42] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:01:58 [I 2022-10-08 11:57:42] ...trap/engines/engine.py.229: process: 0.76277 | load: 0.00301 [I 2022-10-08 11:57:42] ...trap/engines/engine.py.230: loss: 0.31370 [I 2022-10-08 11:57:42] ...optimizers/trijoint.py.151: total_norm: 3.572970 [I 2022-10-08 11:57:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 2160/2304 [I 2022-10-08 11:57:50] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:01:52 [I 2022-10-08 11:57:50] ...trap/engines/engine.py.229: process: 0.77601 | load: 0.00302 [I 2022-10-08 11:57:50] ...trap/engines/engine.py.230: loss: 0.19567 [I 2022-10-08 11:57:50] ...optimizers/trijoint.py.151: total_norm: 2.610209 [I 2022-10-08 11:57:58] ...trap/engines/engine.py.224: train: epoch 3 | batch 2170/2304 [I 2022-10-08 11:57:58] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:01:44 [I 2022-10-08 11:57:58] ...trap/engines/engine.py.229: process: 0.77806 | load: 0.00303 [I 2022-10-08 11:57:58] ...trap/engines/engine.py.230: loss: 0.34748 [I 2022-10-08 11:57:58] ...optimizers/trijoint.py.151: total_norm: 3.707639 [I 2022-10-08 11:58:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 2180/2304 [I 2022-10-08 11:58:06] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:01:36 [I 2022-10-08 11:58:06] ...trap/engines/engine.py.229: process: 0.76052 | load: 0.00301 [I 2022-10-08 11:58:06] ...trap/engines/engine.py.230: loss: 0.30959 [I 2022-10-08 11:58:06] ...optimizers/trijoint.py.151: total_norm: 3.332397 [I 2022-10-08 11:58:13] ...trap/engines/engine.py.224: train: epoch 3 | batch 2190/2304 [I 2022-10-08 11:58:13] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:01:28 [I 2022-10-08 11:58:13] ...trap/engines/engine.py.229: process: 0.77614 | load: 0.00302 [I 2022-10-08 11:58:13] ...trap/engines/engine.py.230: loss: 0.22630 [I 2022-10-08 11:58:13] ...optimizers/trijoint.py.151: total_norm: 3.459969 [I 2022-10-08 11:58:21] ...trap/engines/engine.py.224: train: epoch 3 | batch 2200/2304 [I 2022-10-08 11:58:21] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:01:20 [I 2022-10-08 11:58:21] ...trap/engines/engine.py.229: process: 0.78857 | load: 0.00298 [I 2022-10-08 11:58:21] ...trap/engines/engine.py.230: loss: 0.28448 [I 2022-10-08 11:58:21] ...optimizers/trijoint.py.151: total_norm: 2.822098 [I 2022-10-08 11:58:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 2210/2304 [I 2022-10-08 11:58:29] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:01:13 [I 2022-10-08 11:58:29] ...trap/engines/engine.py.229: process: 0.78442 | load: 0.00309 [I 2022-10-08 11:58:29] ...trap/engines/engine.py.230: loss: 0.23484 [I 2022-10-08 11:58:29] ...optimizers/trijoint.py.151: total_norm: 2.974580 [I 2022-10-08 11:58:37] ...trap/engines/engine.py.224: train: epoch 3 | batch 2220/2304 [I 2022-10-08 11:58:37] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:01:05 [I 2022-10-08 11:58:37] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00309 [I 2022-10-08 11:58:37] ...trap/engines/engine.py.230: loss: 0.22191 [I 2022-10-08 11:58:37] ...optimizers/trijoint.py.151: total_norm: 2.763657 [I 2022-10-08 11:58:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 2230/2304 [I 2022-10-08 11:58:44] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:00:57 [I 2022-10-08 11:58:44] ...trap/engines/engine.py.229: process: 0.78058 | load: 0.00302 [I 2022-10-08 11:58:44] ...trap/engines/engine.py.230: loss: 0.18156 [I 2022-10-08 11:58:44] ...optimizers/trijoint.py.151: total_norm: 3.524923 [I 2022-10-08 11:58:52] ...trap/engines/engine.py.224: train: epoch 3 | batch 2240/2304 [I 2022-10-08 11:58:52] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:00:49 [I 2022-10-08 11:58:52] ...trap/engines/engine.py.229: process: 0.76805 | load: 0.00300 [I 2022-10-08 11:58:52] ...trap/engines/engine.py.230: loss: 0.15481 [I 2022-10-08 11:58:52] ...optimizers/trijoint.py.151: total_norm: 2.205279 [I 2022-10-08 11:59:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 2250/2304 [I 2022-10-08 11:59:00] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:00:41 [I 2022-10-08 11:59:00] ...trap/engines/engine.py.229: process: 0.76355 | load: 0.00304 [I 2022-10-08 11:59:00] ...trap/engines/engine.py.230: loss: 0.29860 [I 2022-10-08 11:59:00] ...optimizers/trijoint.py.151: total_norm: 2.475605 [I 2022-10-08 11:59:08] ...trap/engines/engine.py.224: train: epoch 3 | batch 2260/2304 [I 2022-10-08 11:59:08] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:00:34 [I 2022-10-08 11:59:08] ...trap/engines/engine.py.229: process: 0.78058 | load: 0.00301 [I 2022-10-08 11:59:08] ...trap/engines/engine.py.230: loss: 0.20865 [I 2022-10-08 11:59:08] ...optimizers/trijoint.py.151: total_norm: 2.594521 [I 2022-10-08 11:59:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 2270/2304 [I 2022-10-08 11:59:16] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:00:26 [I 2022-10-08 11:59:16] ...trap/engines/engine.py.229: process: 0.78819 | load: 0.00306 [I 2022-10-08 11:59:16] ...trap/engines/engine.py.230: loss: 0.23319 [I 2022-10-08 11:59:16] ...optimizers/trijoint.py.151: total_norm: 2.569611 [I 2022-10-08 11:59:23] ...trap/engines/engine.py.224: train: epoch 3 | batch 2280/2304 [I 2022-10-08 11:59:23] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:00:18 [I 2022-10-08 11:59:23] ...trap/engines/engine.py.229: process: 0.78092 | load: 0.00303 [I 2022-10-08 11:59:23] ...trap/engines/engine.py.230: loss: 0.32650 [I 2022-10-08 11:59:23] ...optimizers/trijoint.py.151: total_norm: 3.241509 [I 2022-10-08 11:59:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 2290/2304 [I 2022-10-08 11:59:31] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:00:10 [I 2022-10-08 11:59:31] ...trap/engines/engine.py.229: process: 0.76625 | load: 0.00302 [I 2022-10-08 11:59:31] ...trap/engines/engine.py.230: loss: 0.25879 [I 2022-10-08 11:59:31] ...optimizers/trijoint.py.151: total_norm: 2.260659 [I 2022-10-08 11:59:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 2300/2304 [I 2022-10-08 11:59:39] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:00:03 [I 2022-10-08 11:59:39] ...trap/engines/engine.py.229: process: 0.76081 | load: 0.00016 [I 2022-10-08 11:59:39] ...trap/engines/engine.py.230: loss: 0.20299 [I 2022-10-08 11:59:39] ...optimizers/trijoint.py.151: total_norm: 2.481849 [I 2022-10-08 11:59:42] ...trap/engines/engine.py.224: train: epoch 3 | batch 2304/2304 [I 2022-10-08 11:59:42] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:00:00 [I 2022-10-08 11:59:42] ...trap/engines/engine.py.229: process: 0.76202 | load: 0.00017 [I 2022-10-08 11:59:42] ...trap/engines/engine.py.230: loss: 0.16849 [I 2022-10-08 11:59:42] ...optimizers/trijoint.py.151: total_norm: 2.445273 [S 2022-10-08 11:59:43] ...trap/engines/engine.py.129: train_epoch.epoch: 3 [S 2022-10-08 11:59:43] ...trap/engines/engine.py.129: train_epoch.loss: 0.2452389497689725 [I 2022-10-08 11:59:43] ...s/criterions/triplet.py.84: margin = 0.07 [I 2022-10-08 11:59:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 3 [I 2022-10-08 11:59:43] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 11:59:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 11:59:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 11:59:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 11:59:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 11:59:46] ...trap/engines/engine.py.321: eval: epoch 3 | batch 0/510 [I 2022-10-08 11:59:46] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:19 [I 2022-10-08 11:59:46] ...trap/engines/engine.py.326: process: 2.62644 | load: 2.12969 [I 2022-10-08 11:59:50] ...trap/engines/engine.py.321: eval: epoch 3 | batch 10/510 [I 2022-10-08 11:59:50] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:30 [I 2022-10-08 11:59:50] ...trap/engines/engine.py.326: process: 0.41658 | load: 0.00017 [I 2022-10-08 11:59:54] ...trap/engines/engine.py.321: eval: epoch 3 | batch 20/510 [I 2022-10-08 11:59:54] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:44 [I 2022-10-08 11:59:54] ...trap/engines/engine.py.326: process: 0.44876 | load: 0.00018 [I 2022-10-08 12:00:00] ...trap/engines/engine.py.321: eval: epoch 3 | batch 30/510 [I 2022-10-08 12:00:00] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:41 [I 2022-10-08 12:00:00] ...trap/engines/engine.py.326: process: 0.66654 | load: 0.00019 [I 2022-10-08 12:00:05] ...trap/engines/engine.py.321: eval: epoch 3 | batch 40/510 [I 2022-10-08 12:00:05] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:03:48 [I 2022-10-08 12:00:05] ...trap/engines/engine.py.326: process: 0.42768 | load: 0.00019 [I 2022-10-08 12:00:10] ...trap/engines/engine.py.321: eval: epoch 3 | batch 50/510 [I 2022-10-08 12:00:10] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:40 [I 2022-10-08 12:00:10] ...trap/engines/engine.py.326: process: 0.46567 | load: 0.00019 [I 2022-10-08 12:00:14] ...trap/engines/engine.py.321: eval: epoch 3 | batch 60/510 [I 2022-10-08 12:00:14] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:33 [I 2022-10-08 12:00:14] ...trap/engines/engine.py.326: process: 0.55575 | load: 0.00018 [I 2022-10-08 12:00:19] ...trap/engines/engine.py.321: eval: epoch 3 | batch 70/510 [I 2022-10-08 12:00:19] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:30 [I 2022-10-08 12:00:19] ...trap/engines/engine.py.326: process: 0.56592 | load: 0.00017 [I 2022-10-08 12:00:24] ...trap/engines/engine.py.321: eval: epoch 3 | batch 80/510 [I 2022-10-08 12:00:24] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:50 [I 2022-10-08 12:00:24] ...trap/engines/engine.py.326: process: 0.51384 | load: 0.00019 [I 2022-10-08 12:00:29] ...trap/engines/engine.py.321: eval: epoch 3 | batch 90/510 [I 2022-10-08 12:00:29] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:15 [I 2022-10-08 12:00:29] ...trap/engines/engine.py.326: process: 0.48323 | load: 0.00018 [I 2022-10-08 12:00:33] ...trap/engines/engine.py.321: eval: epoch 3 | batch 100/510 [I 2022-10-08 12:00:33] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:02:49 [I 2022-10-08 12:00:33] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00022 [I 2022-10-08 12:00:36] ...trap/engines/engine.py.321: eval: epoch 3 | batch 110/510 [I 2022-10-08 12:00:36] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:11 [I 2022-10-08 12:00:36] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00019 [I 2022-10-08 12:00:40] ...trap/engines/engine.py.321: eval: epoch 3 | batch 120/510 [I 2022-10-08 12:00:40] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:04 [I 2022-10-08 12:00:40] ...trap/engines/engine.py.326: process: 0.31872 | load: 0.00019 [I 2022-10-08 12:00:43] ...trap/engines/engine.py.321: eval: epoch 3 | batch 130/510 [I 2022-10-08 12:00:43] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:00 [I 2022-10-08 12:00:43] ...trap/engines/engine.py.326: process: 0.31578 | load: 0.00018 [I 2022-10-08 12:00:46] ...trap/engines/engine.py.321: eval: epoch 3 | batch 140/510 [I 2022-10-08 12:00:46] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:57 [I 2022-10-08 12:00:46] ...trap/engines/engine.py.326: process: 0.31932 | load: 0.00019 [I 2022-10-08 12:00:49] ...trap/engines/engine.py.321: eval: epoch 3 | batch 150/510 [I 2022-10-08 12:00:49] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:54 [I 2022-10-08 12:00:49] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00019 [I 2022-10-08 12:00:52] ...trap/engines/engine.py.321: eval: epoch 3 | batch 160/510 [I 2022-10-08 12:00:52] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:51 [I 2022-10-08 12:00:52] ...trap/engines/engine.py.326: process: 0.32287 | load: 0.00021 [I 2022-10-08 12:00:56] ...trap/engines/engine.py.321: eval: epoch 3 | batch 170/510 [I 2022-10-08 12:00:56] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:48 [I 2022-10-08 12:00:56] ...trap/engines/engine.py.326: process: 0.32074 | load: 0.00019 [I 2022-10-08 12:00:59] ...trap/engines/engine.py.321: eval: epoch 3 | batch 180/510 [I 2022-10-08 12:00:59] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:45 [I 2022-10-08 12:00:59] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00019 [I 2022-10-08 12:01:02] ...trap/engines/engine.py.321: eval: epoch 3 | batch 190/510 [I 2022-10-08 12:01:02] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:41 [I 2022-10-08 12:01:02] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00019 [I 2022-10-08 12:01:05] ...trap/engines/engine.py.321: eval: epoch 3 | batch 200/510 [I 2022-10-08 12:01:05] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:38 [I 2022-10-08 12:01:05] ...trap/engines/engine.py.326: process: 0.31456 | load: 0.00019 [I 2022-10-08 12:01:08] ...trap/engines/engine.py.321: eval: epoch 3 | batch 210/510 [I 2022-10-08 12:01:08] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:35 [I 2022-10-08 12:01:08] ...trap/engines/engine.py.326: process: 0.31680 | load: 0.00019 [I 2022-10-08 12:01:12] ...trap/engines/engine.py.321: eval: epoch 3 | batch 220/510 [I 2022-10-08 12:01:12] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:31 [I 2022-10-08 12:01:12] ...trap/engines/engine.py.326: process: 0.31597 | load: 0.00020 [I 2022-10-08 12:01:15] ...trap/engines/engine.py.321: eval: epoch 3 | batch 230/510 [I 2022-10-08 12:01:15] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:29 [I 2022-10-08 12:01:15] ...trap/engines/engine.py.326: process: 0.32184 | load: 0.00020 [I 2022-10-08 12:01:18] ...trap/engines/engine.py.321: eval: epoch 3 | batch 240/510 [I 2022-10-08 12:01:18] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:26 [I 2022-10-08 12:01:18] ...trap/engines/engine.py.326: process: 0.32107 | load: 0.00019 [I 2022-10-08 12:01:21] ...trap/engines/engine.py.321: eval: epoch 3 | batch 250/510 [I 2022-10-08 12:01:21] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:23 [I 2022-10-08 12:01:21] ...trap/engines/engine.py.326: process: 0.32249 | load: 0.00019 [I 2022-10-08 12:01:24] ...trap/engines/engine.py.321: eval: epoch 3 | batch 260/510 [I 2022-10-08 12:01:24] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:20 [I 2022-10-08 12:01:24] ...trap/engines/engine.py.326: process: 0.31960 | load: 0.00020 [I 2022-10-08 12:01:28] ...trap/engines/engine.py.321: eval: epoch 3 | batch 270/510 [I 2022-10-08 12:01:28] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:16 [I 2022-10-08 12:01:28] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00020 [I 2022-10-08 12:01:31] ...trap/engines/engine.py.321: eval: epoch 3 | batch 280/510 [I 2022-10-08 12:01:31] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:13 [I 2022-10-08 12:01:31] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00020 [I 2022-10-08 12:01:34] ...trap/engines/engine.py.321: eval: epoch 3 | batch 290/510 [I 2022-10-08 12:01:34] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:10 [I 2022-10-08 12:01:34] ...trap/engines/engine.py.326: process: 0.31667 | load: 0.00019 [I 2022-10-08 12:01:37] ...trap/engines/engine.py.321: eval: epoch 3 | batch 300/510 [I 2022-10-08 12:01:37] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:06 [I 2022-10-08 12:01:37] ...trap/engines/engine.py.326: process: 0.31420 | load: 0.00019 [I 2022-10-08 12:01:40] ...trap/engines/engine.py.321: eval: epoch 3 | batch 310/510 [I 2022-10-08 12:01:40] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:03 [I 2022-10-08 12:01:40] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00019 [I 2022-10-08 12:01:43] ...trap/engines/engine.py.321: eval: epoch 3 | batch 320/510 [I 2022-10-08 12:01:43] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:00 [I 2022-10-08 12:01:43] ...trap/engines/engine.py.326: process: 0.32104 | load: 0.00019 [I 2022-10-08 12:01:47] ...trap/engines/engine.py.321: eval: epoch 3 | batch 330/510 [I 2022-10-08 12:01:47] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:57 [I 2022-10-08 12:01:47] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00019 [I 2022-10-08 12:01:50] ...trap/engines/engine.py.321: eval: epoch 3 | batch 340/510 [I 2022-10-08 12:01:50] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:54 [I 2022-10-08 12:01:50] ...trap/engines/engine.py.326: process: 0.32018 | load: 0.00021 [I 2022-10-08 12:01:53] ...trap/engines/engine.py.321: eval: epoch 3 | batch 350/510 [I 2022-10-08 12:01:53] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:50 [I 2022-10-08 12:01:53] ...trap/engines/engine.py.326: process: 0.31103 | load: 0.00021 [I 2022-10-08 12:01:56] ...trap/engines/engine.py.321: eval: epoch 3 | batch 360/510 [I 2022-10-08 12:01:56] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:47 [I 2022-10-08 12:01:56] ...trap/engines/engine.py.326: process: 0.31447 | load: 0.00019 [I 2022-10-08 12:01:59] ...trap/engines/engine.py.321: eval: epoch 3 | batch 370/510 [I 2022-10-08 12:01:59] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:44 [I 2022-10-08 12:01:59] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00022 [I 2022-10-08 12:02:03] ...trap/engines/engine.py.321: eval: epoch 3 | batch 380/510 [I 2022-10-08 12:02:03] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:41 [I 2022-10-08 12:02:03] ...trap/engines/engine.py.326: process: 0.31430 | load: 0.00020 [I 2022-10-08 12:02:06] ...trap/engines/engine.py.321: eval: epoch 3 | batch 390/510 [I 2022-10-08 12:02:06] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:38 [I 2022-10-08 12:02:06] ...trap/engines/engine.py.326: process: 0.32001 | load: 0.00020 [I 2022-10-08 12:02:09] ...trap/engines/engine.py.321: eval: epoch 3 | batch 400/510 [I 2022-10-08 12:02:09] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:35 [I 2022-10-08 12:02:09] ...trap/engines/engine.py.326: process: 0.31815 | load: 0.00020 [I 2022-10-08 12:02:12] ...trap/engines/engine.py.321: eval: epoch 3 | batch 410/510 [I 2022-10-08 12:02:12] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:31 [I 2022-10-08 12:02:12] ...trap/engines/engine.py.326: process: 0.31477 | load: 0.00019 [I 2022-10-08 12:02:15] ...trap/engines/engine.py.321: eval: epoch 3 | batch 420/510 [I 2022-10-08 12:02:15] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:28 [I 2022-10-08 12:02:15] ...trap/engines/engine.py.326: process: 0.32156 | load: 0.00019 [I 2022-10-08 12:02:19] ...trap/engines/engine.py.321: eval: epoch 3 | batch 430/510 [I 2022-10-08 12:02:19] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:25 [I 2022-10-08 12:02:19] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00020 [I 2022-10-08 12:02:22] ...trap/engines/engine.py.321: eval: epoch 3 | batch 440/510 [I 2022-10-08 12:02:22] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:22 [I 2022-10-08 12:02:22] ...trap/engines/engine.py.326: process: 0.32551 | load: 0.00019 [I 2022-10-08 12:02:25] ...trap/engines/engine.py.321: eval: epoch 3 | batch 450/510 [I 2022-10-08 12:02:25] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:19 [I 2022-10-08 12:02:25] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00019 [I 2022-10-08 12:02:28] ...trap/engines/engine.py.321: eval: epoch 3 | batch 460/510 [I 2022-10-08 12:02:28] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:15 [I 2022-10-08 12:02:28] ...trap/engines/engine.py.326: process: 0.31298 | load: 0.00020 [I 2022-10-08 12:02:31] ...trap/engines/engine.py.321: eval: epoch 3 | batch 470/510 [I 2022-10-08 12:02:31] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:12 [I 2022-10-08 12:02:31] ...trap/engines/engine.py.326: process: 0.32254 | load: 0.00020 [I 2022-10-08 12:02:35] ...trap/engines/engine.py.321: eval: epoch 3 | batch 480/510 [I 2022-10-08 12:02:35] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:09 [I 2022-10-08 12:02:35] ...trap/engines/engine.py.326: process: 0.31881 | load: 0.00019 [I 2022-10-08 12:02:38] ...trap/engines/engine.py.321: eval: epoch 3 | batch 490/510 [I 2022-10-08 12:02:38] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:06 [I 2022-10-08 12:02:38] ...trap/engines/engine.py.326: process: 0.31690 | load: 0.00020 [I 2022-10-08 12:02:41] ...trap/engines/engine.py.321: eval: epoch 3 | batch 500/510 [I 2022-10-08 12:02:41] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:03 [I 2022-10-08 12:02:41] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00019 [I 2022-10-08 12:02:44] ...trap/engines/engine.py.321: eval: epoch 3 | batch 510/510 [I 2022-10-08 12:02:44] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:00 [I 2022-10-08 12:02:44] ...trap/engines/engine.py.326: process: 0.31868 | load: 0.00015 [S 2022-10-08 12:02:45] ...trap/engines/engine.py.132: eval_epoch.epoch: 3 [I 2022-10-08 12:02:45] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.10 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.532100 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.818300 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.886900 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.551300 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.824100 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.887600 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.300000 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014909 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013115 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011122 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015297 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011554 [S 2022-10-08 12:03:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010763 [I 2022-10-08 12:03:15] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-08 12:03:15] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 12:03:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 12:03:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 12:03:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 12:03:16] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 12:03:19] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:03:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:03:23] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 12:03:23] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 12:03:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:03:31] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:03:31] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 12:03:31] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 12:03:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:03:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:03:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 4 [I 2022-10-08 12:03:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 12:03:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 0/2304 [I 2022-10-08 12:03:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:51:58 [I 2022-10-08 12:03:42] ...trap/engines/engine.py.229: process: 2.91618 | load: 2.06277 [I 2022-10-08 12:03:42] ...trap/engines/engine.py.230: loss: 0.24147 [I 2022-10-08 12:03:42] ...optimizers/trijoint.py.151: total_norm: 2.318232 [I 2022-10-08 12:03:50] ...trap/engines/engine.py.224: train: epoch 4 | batch 10/2304 [I 2022-10-08 12:03:50] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:41 [I 2022-10-08 12:03:50] ...trap/engines/engine.py.229: process: 0.77647 | load: 0.00631 [I 2022-10-08 12:03:50] ...trap/engines/engine.py.230: loss: 0.18894 [I 2022-10-08 12:03:50] ...optimizers/trijoint.py.151: total_norm: 2.222259 [I 2022-10-08 12:03:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 20/2304 [I 2022-10-08 12:03:58] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:47 [I 2022-10-08 12:03:58] ...trap/engines/engine.py.229: process: 0.78131 | load: 0.00471 [I 2022-10-08 12:03:58] ...trap/engines/engine.py.230: loss: 0.21717 [I 2022-10-08 12:03:58] ...optimizers/trijoint.py.151: total_norm: 2.721473 [I 2022-10-08 12:04:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 30/2304 [I 2022-10-08 12:04:06] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:42 [I 2022-10-08 12:04:06] ...trap/engines/engine.py.229: process: 0.76832 | load: 0.00371 [I 2022-10-08 12:04:06] ...trap/engines/engine.py.230: loss: 0.22553 [I 2022-10-08 12:04:06] ...optimizers/trijoint.py.151: total_norm: 2.737350 [I 2022-10-08 12:04:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 40/2304 [I 2022-10-08 12:04:13] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:32 [I 2022-10-08 12:04:13] ...trap/engines/engine.py.229: process: 0.78260 | load: 0.00396 [I 2022-10-08 12:04:13] ...trap/engines/engine.py.230: loss: 0.25880 [I 2022-10-08 12:04:13] ...optimizers/trijoint.py.151: total_norm: 2.959010 [I 2022-10-08 12:04:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 50/2304 [I 2022-10-08 12:04:21] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:19 [I 2022-10-08 12:04:21] ...trap/engines/engine.py.229: process: 0.78405 | load: 0.00386 [I 2022-10-08 12:04:21] ...trap/engines/engine.py.230: loss: 0.26665 [I 2022-10-08 12:04:21] ...optimizers/trijoint.py.151: total_norm: 3.216466 [I 2022-10-08 12:04:29] ...trap/engines/engine.py.224: train: epoch 4 | batch 60/2304 [I 2022-10-08 12:04:29] ...trap/engines/engine.py.228: elapsed: 0:00:50 | left: 0:29:18 [I 2022-10-08 12:04:29] ...trap/engines/engine.py.229: process: 0.78285 | load: 0.00437 [I 2022-10-08 12:04:29] ...trap/engines/engine.py.230: loss: 0.28239 [I 2022-10-08 12:04:29] ...optimizers/trijoint.py.151: total_norm: 3.125301 [I 2022-10-08 12:04:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 70/2304 [I 2022-10-08 12:04:37] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:14 [I 2022-10-08 12:04:37] ...trap/engines/engine.py.229: process: 0.79348 | load: 0.00353 [I 2022-10-08 12:04:37] ...trap/engines/engine.py.230: loss: 0.34259 [I 2022-10-08 12:04:37] ...optimizers/trijoint.py.151: total_norm: 3.291262 [I 2022-10-08 12:04:45] ...trap/engines/engine.py.224: train: epoch 4 | batch 80/2304 [I 2022-10-08 12:04:45] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:29:01 [I 2022-10-08 12:04:45] ...trap/engines/engine.py.229: process: 0.78907 | load: 0.00337 [I 2022-10-08 12:04:45] ...trap/engines/engine.py.230: loss: 0.23373 [I 2022-10-08 12:04:45] ...optimizers/trijoint.py.151: total_norm: 1.866739 [I 2022-10-08 12:04:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 90/2304 [I 2022-10-08 12:04:53] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:50 [I 2022-10-08 12:04:53] ...trap/engines/engine.py.229: process: 0.79817 | load: 0.00330 [I 2022-10-08 12:04:53] ...trap/engines/engine.py.230: loss: 0.38151 [I 2022-10-08 12:04:53] ...optimizers/trijoint.py.151: total_norm: 3.460093 [I 2022-10-08 12:05:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 100/2304 [I 2022-10-08 12:05:00] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:38 [I 2022-10-08 12:05:00] ...trap/engines/engine.py.229: process: 0.77643 | load: 0.00354 [I 2022-10-08 12:05:00] ...trap/engines/engine.py.230: loss: 0.19114 [I 2022-10-08 12:05:00] ...optimizers/trijoint.py.151: total_norm: 2.148734 [I 2022-10-08 12:05:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 110/2304 [I 2022-10-08 12:05:08] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:35 [I 2022-10-08 12:05:08] ...trap/engines/engine.py.229: process: 0.77922 | load: 0.00324 [I 2022-10-08 12:05:08] ...trap/engines/engine.py.230: loss: 0.29067 [I 2022-10-08 12:05:08] ...optimizers/trijoint.py.151: total_norm: 2.775532 [I 2022-10-08 12:05:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 120/2304 [I 2022-10-08 12:05:16] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:27 [I 2022-10-08 12:05:16] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00323 [I 2022-10-08 12:05:16] ...trap/engines/engine.py.230: loss: 0.25986 [I 2022-10-08 12:05:16] ...optimizers/trijoint.py.151: total_norm: 2.692651 [I 2022-10-08 12:05:24] ...trap/engines/engine.py.224: train: epoch 4 | batch 130/2304 [I 2022-10-08 12:05:24] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:12 [I 2022-10-08 12:05:24] ...trap/engines/engine.py.229: process: 0.78003 | load: 0.00316 [I 2022-10-08 12:05:24] ...trap/engines/engine.py.230: loss: 0.17968 [I 2022-10-08 12:05:24] ...optimizers/trijoint.py.151: total_norm: 2.581163 [I 2022-10-08 12:05:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 140/2304 [I 2022-10-08 12:05:32] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:27:59 [I 2022-10-08 12:05:32] ...trap/engines/engine.py.229: process: 0.77165 | load: 0.00333 [I 2022-10-08 12:05:32] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-08 12:05:32] ...optimizers/trijoint.py.151: total_norm: 2.504557 [I 2022-10-08 12:05:40] ...trap/engines/engine.py.224: train: epoch 4 | batch 150/2304 [I 2022-10-08 12:05:40] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:58 [I 2022-10-08 12:05:40] ...trap/engines/engine.py.229: process: 0.78598 | load: 0.00345 [I 2022-10-08 12:05:40] ...trap/engines/engine.py.230: loss: 0.16265 [I 2022-10-08 12:05:40] ...optimizers/trijoint.py.151: total_norm: 2.489509 [I 2022-10-08 12:05:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 160/2304 [I 2022-10-08 12:05:47] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:40 [I 2022-10-08 12:05:47] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00328 [I 2022-10-08 12:05:47] ...trap/engines/engine.py.230: loss: 0.19932 [I 2022-10-08 12:05:47] ...optimizers/trijoint.py.151: total_norm: 2.421804 [I 2022-10-08 12:05:55] ...trap/engines/engine.py.224: train: epoch 4 | batch 170/2304 [I 2022-10-08 12:05:55] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:33 [I 2022-10-08 12:05:55] ...trap/engines/engine.py.229: process: 0.78530 | load: 0.00326 [I 2022-10-08 12:05:55] ...trap/engines/engine.py.230: loss: 0.23815 [I 2022-10-08 12:05:55] ...optimizers/trijoint.py.151: total_norm: 3.372078 [I 2022-10-08 12:06:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 180/2304 [I 2022-10-08 12:06:03] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:37 [I 2022-10-08 12:06:03] ...trap/engines/engine.py.229: process: 0.79373 | load: 0.00325 [I 2022-10-08 12:06:03] ...trap/engines/engine.py.230: loss: 0.18447 [I 2022-10-08 12:06:03] ...optimizers/trijoint.py.151: total_norm: 3.197690 [I 2022-10-08 12:06:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 190/2304 [I 2022-10-08 12:06:11] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:17 [I 2022-10-08 12:06:11] ...trap/engines/engine.py.229: process: 0.77297 | load: 0.00302 [I 2022-10-08 12:06:11] ...trap/engines/engine.py.230: loss: 0.21214 [I 2022-10-08 12:06:11] ...optimizers/trijoint.py.151: total_norm: 2.459668 [I 2022-10-08 12:06:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 200/2304 [I 2022-10-08 12:06:18] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:18 [I 2022-10-08 12:06:18] ...trap/engines/engine.py.229: process: 0.77788 | load: 0.00302 [I 2022-10-08 12:06:18] ...trap/engines/engine.py.230: loss: 0.30462 [I 2022-10-08 12:06:18] ...optimizers/trijoint.py.151: total_norm: 3.275379 [I 2022-10-08 12:06:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 210/2304 [I 2022-10-08 12:06:26] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:10 [I 2022-10-08 12:06:26] ...trap/engines/engine.py.229: process: 0.78138 | load: 0.00320 [I 2022-10-08 12:06:26] ...trap/engines/engine.py.230: loss: 0.23087 [I 2022-10-08 12:06:26] ...optimizers/trijoint.py.151: total_norm: 2.618094 [I 2022-10-08 12:06:34] ...trap/engines/engine.py.224: train: epoch 4 | batch 220/2304 [I 2022-10-08 12:06:34] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:11 [I 2022-10-08 12:06:34] ...trap/engines/engine.py.229: process: 0.79523 | load: 0.00301 [I 2022-10-08 12:06:34] ...trap/engines/engine.py.230: loss: 0.21870 [I 2022-10-08 12:06:34] ...optimizers/trijoint.py.151: total_norm: 3.761907 [I 2022-10-08 12:06:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 230/2304 [I 2022-10-08 12:06:42] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:27:04 [I 2022-10-08 12:06:42] ...trap/engines/engine.py.229: process: 0.79917 | load: 0.00303 [I 2022-10-08 12:06:42] ...trap/engines/engine.py.230: loss: 0.20554 [I 2022-10-08 12:06:42] ...optimizers/trijoint.py.151: total_norm: 2.728656 [I 2022-10-08 12:06:50] ...trap/engines/engine.py.224: train: epoch 4 | batch 240/2304 [I 2022-10-08 12:06:50] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:45 [I 2022-10-08 12:06:50] ...trap/engines/engine.py.229: process: 0.77404 | load: 0.00334 [I 2022-10-08 12:06:50] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-08 12:06:50] ...optimizers/trijoint.py.151: total_norm: 2.525175 [I 2022-10-08 12:06:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 250/2304 [I 2022-10-08 12:06:58] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:40 [I 2022-10-08 12:06:58] ...trap/engines/engine.py.229: process: 0.77478 | load: 0.00304 [I 2022-10-08 12:06:58] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-08 12:06:58] ...optimizers/trijoint.py.151: total_norm: 2.741415 [I 2022-10-08 12:07:05] ...trap/engines/engine.py.224: train: epoch 4 | batch 260/2304 [I 2022-10-08 12:07:05] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:32 [I 2022-10-08 12:07:05] ...trap/engines/engine.py.229: process: 0.77791 | load: 0.00306 [I 2022-10-08 12:07:05] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-08 12:07:05] ...optimizers/trijoint.py.151: total_norm: 3.025335 [I 2022-10-08 12:07:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 270/2304 [I 2022-10-08 12:07:13] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:24 [I 2022-10-08 12:07:13] ...trap/engines/engine.py.229: process: 0.78018 | load: 0.00299 [I 2022-10-08 12:07:13] ...trap/engines/engine.py.230: loss: 0.25412 [I 2022-10-08 12:07:13] ...optimizers/trijoint.py.151: total_norm: 3.227066 [I 2022-10-08 12:07:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 280/2304 [I 2022-10-08 12:07:21] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:09 [I 2022-10-08 12:07:21] ...trap/engines/engine.py.229: process: 0.76424 | load: 0.00335 [I 2022-10-08 12:07:21] ...trap/engines/engine.py.230: loss: 0.19232 [I 2022-10-08 12:07:21] ...optimizers/trijoint.py.151: total_norm: 2.504284 [I 2022-10-08 12:07:29] ...trap/engines/engine.py.224: train: epoch 4 | batch 290/2304 [I 2022-10-08 12:07:29] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:09 [I 2022-10-08 12:07:29] ...trap/engines/engine.py.229: process: 0.77670 | load: 0.00314 [I 2022-10-08 12:07:29] ...trap/engines/engine.py.230: loss: 0.26875 [I 2022-10-08 12:07:29] ...optimizers/trijoint.py.151: total_norm: 3.519792 [I 2022-10-08 12:07:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 300/2304 [I 2022-10-08 12:07:37] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:26:11 [I 2022-10-08 12:07:37] ...trap/engines/engine.py.229: process: 0.78526 | load: 0.00302 [I 2022-10-08 12:07:37] ...trap/engines/engine.py.230: loss: 0.20558 [I 2022-10-08 12:07:37] ...optimizers/trijoint.py.151: total_norm: 2.516086 [I 2022-10-08 12:07:44] ...trap/engines/engine.py.224: train: epoch 4 | batch 310/2304 [I 2022-10-08 12:07:44] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:54 [I 2022-10-08 12:07:44] ...trap/engines/engine.py.229: process: 0.78367 | load: 0.00303 [I 2022-10-08 12:07:44] ...trap/engines/engine.py.230: loss: 0.28082 [I 2022-10-08 12:07:44] ...optimizers/trijoint.py.151: total_norm: 3.311183 [I 2022-10-08 12:07:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 320/2304 [I 2022-10-08 12:07:52] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:45 [I 2022-10-08 12:07:52] ...trap/engines/engine.py.229: process: 0.77386 | load: 0.00308 [I 2022-10-08 12:07:52] ...trap/engines/engine.py.230: loss: 0.25406 [I 2022-10-08 12:07:52] ...optimizers/trijoint.py.151: total_norm: 4.175597 [I 2022-10-08 12:08:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 330/2304 [I 2022-10-08 12:08:00] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:44 [I 2022-10-08 12:08:00] ...trap/engines/engine.py.229: process: 0.78560 | load: 0.00311 [I 2022-10-08 12:08:00] ...trap/engines/engine.py.230: loss: 0.23317 [I 2022-10-08 12:08:00] ...optimizers/trijoint.py.151: total_norm: 2.489822 [I 2022-10-08 12:08:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 340/2304 [I 2022-10-08 12:08:08] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:26 [I 2022-10-08 12:08:08] ...trap/engines/engine.py.229: process: 0.77588 | load: 0.00312 [I 2022-10-08 12:08:08] ...trap/engines/engine.py.230: loss: 0.15787 [I 2022-10-08 12:08:08] ...optimizers/trijoint.py.151: total_norm: 1.787259 [I 2022-10-08 12:08:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 350/2304 [I 2022-10-08 12:08:16] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:22 [I 2022-10-08 12:08:16] ...trap/engines/engine.py.229: process: 0.78244 | load: 0.00323 [I 2022-10-08 12:08:16] ...trap/engines/engine.py.230: loss: 0.22645 [I 2022-10-08 12:08:16] ...optimizers/trijoint.py.151: total_norm: 2.792053 [I 2022-10-08 12:08:24] ...trap/engines/engine.py.224: train: epoch 4 | batch 360/2304 [I 2022-10-08 12:08:24] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:18 [I 2022-10-08 12:08:24] ...trap/engines/engine.py.229: process: 0.77943 | load: 0.00311 [I 2022-10-08 12:08:24] ...trap/engines/engine.py.230: loss: 0.19716 [I 2022-10-08 12:08:24] ...optimizers/trijoint.py.151: total_norm: 3.048234 [I 2022-10-08 12:08:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 370/2304 [I 2022-10-08 12:08:31] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:25:16 [I 2022-10-08 12:08:31] ...trap/engines/engine.py.229: process: 0.77951 | load: 0.00311 [I 2022-10-08 12:08:31] ...trap/engines/engine.py.230: loss: 0.25727 [I 2022-10-08 12:08:31] ...optimizers/trijoint.py.151: total_norm: 2.662272 [I 2022-10-08 12:08:39] ...trap/engines/engine.py.224: train: epoch 4 | batch 380/2304 [I 2022-10-08 12:08:39] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:58 [I 2022-10-08 12:08:39] ...trap/engines/engine.py.229: process: 0.78888 | load: 0.00307 [I 2022-10-08 12:08:39] ...trap/engines/engine.py.230: loss: 0.20365 [I 2022-10-08 12:08:39] ...optimizers/trijoint.py.151: total_norm: 2.125664 [I 2022-10-08 12:08:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 390/2304 [I 2022-10-08 12:08:47] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:50 [I 2022-10-08 12:08:47] ...trap/engines/engine.py.229: process: 0.77555 | load: 0.00308 [I 2022-10-08 12:08:47] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-08 12:08:47] ...optimizers/trijoint.py.151: total_norm: 2.512182 [I 2022-10-08 12:08:55] ...trap/engines/engine.py.224: train: epoch 4 | batch 400/2304 [I 2022-10-08 12:08:55] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:51 [I 2022-10-08 12:08:55] ...trap/engines/engine.py.229: process: 0.78632 | load: 0.00312 [I 2022-10-08 12:08:55] ...trap/engines/engine.py.230: loss: 0.25709 [I 2022-10-08 12:08:55] ...optimizers/trijoint.py.151: total_norm: 3.711460 [I 2022-10-08 12:09:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 410/2304 [I 2022-10-08 12:09:03] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:42 [I 2022-10-08 12:09:03] ...trap/engines/engine.py.229: process: 0.78656 | load: 0.00321 [I 2022-10-08 12:09:03] ...trap/engines/engine.py.230: loss: 0.23164 [I 2022-10-08 12:09:03] ...optimizers/trijoint.py.151: total_norm: 2.476252 [I 2022-10-08 12:09:10] ...trap/engines/engine.py.224: train: epoch 4 | batch 420/2304 [I 2022-10-08 12:09:10] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:24 [I 2022-10-08 12:09:10] ...trap/engines/engine.py.229: process: 0.78257 | load: 0.00314 [I 2022-10-08 12:09:10] ...trap/engines/engine.py.230: loss: 0.24835 [I 2022-10-08 12:09:10] ...optimizers/trijoint.py.151: total_norm: 2.879402 [I 2022-10-08 12:09:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 430/2304 [I 2022-10-08 12:09:18] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:18 [I 2022-10-08 12:09:18] ...trap/engines/engine.py.229: process: 0.78282 | load: 0.00310 [I 2022-10-08 12:09:18] ...trap/engines/engine.py.230: loss: 0.29592 [I 2022-10-08 12:09:18] ...optimizers/trijoint.py.151: total_norm: 3.877699 [I 2022-10-08 12:09:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 440/2304 [I 2022-10-08 12:09:26] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:00 [I 2022-10-08 12:09:26] ...trap/engines/engine.py.229: process: 0.76933 | load: 0.00311 [I 2022-10-08 12:09:26] ...trap/engines/engine.py.230: loss: 0.19715 [I 2022-10-08 12:09:26] ...optimizers/trijoint.py.151: total_norm: 2.684541 [I 2022-10-08 12:09:34] ...trap/engines/engine.py.224: train: epoch 4 | batch 450/2304 [I 2022-10-08 12:09:34] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:02 [I 2022-10-08 12:09:34] ...trap/engines/engine.py.229: process: 0.77362 | load: 0.00309 [I 2022-10-08 12:09:34] ...trap/engines/engine.py.230: loss: 0.24703 [I 2022-10-08 12:09:34] ...optimizers/trijoint.py.151: total_norm: 2.867636 [I 2022-10-08 12:09:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 460/2304 [I 2022-10-08 12:09:42] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:52 [I 2022-10-08 12:09:42] ...trap/engines/engine.py.229: process: 0.78263 | load: 0.00311 [I 2022-10-08 12:09:42] ...trap/engines/engine.py.230: loss: 0.19214 [I 2022-10-08 12:09:42] ...optimizers/trijoint.py.151: total_norm: 2.507061 [I 2022-10-08 12:09:49] ...trap/engines/engine.py.224: train: epoch 4 | batch 470/2304 [I 2022-10-08 12:09:49] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:51 [I 2022-10-08 12:09:49] ...trap/engines/engine.py.229: process: 0.77050 | load: 0.00310 [I 2022-10-08 12:09:49] ...trap/engines/engine.py.230: loss: 0.23073 [I 2022-10-08 12:09:49] ...optimizers/trijoint.py.151: total_norm: 2.762076 [I 2022-10-08 12:09:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 480/2304 [I 2022-10-08 12:09:57] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:31 [I 2022-10-08 12:09:57] ...trap/engines/engine.py.229: process: 0.78322 | load: 0.00302 [I 2022-10-08 12:09:57] ...trap/engines/engine.py.230: loss: 0.23090 [I 2022-10-08 12:09:57] ...optimizers/trijoint.py.151: total_norm: 2.673214 [I 2022-10-08 12:10:05] ...trap/engines/engine.py.224: train: epoch 4 | batch 490/2304 [I 2022-10-08 12:10:05] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:32 [I 2022-10-08 12:10:05] ...trap/engines/engine.py.229: process: 0.78998 | load: 0.00303 [I 2022-10-08 12:10:05] ...trap/engines/engine.py.230: loss: 0.18484 [I 2022-10-08 12:10:05] ...optimizers/trijoint.py.151: total_norm: 3.138898 [I 2022-10-08 12:10:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 500/2304 [I 2022-10-08 12:10:13] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:24 [I 2022-10-08 12:10:13] ...trap/engines/engine.py.229: process: 0.77003 | load: 0.00304 [I 2022-10-08 12:10:13] ...trap/engines/engine.py.230: loss: 0.27038 [I 2022-10-08 12:10:13] ...optimizers/trijoint.py.151: total_norm: 3.253463 [I 2022-10-08 12:10:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 510/2304 [I 2022-10-08 12:10:21] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:20 [I 2022-10-08 12:10:21] ...trap/engines/engine.py.229: process: 0.78402 | load: 0.00303 [I 2022-10-08 12:10:21] ...trap/engines/engine.py.230: loss: 0.18710 [I 2022-10-08 12:10:21] ...optimizers/trijoint.py.151: total_norm: 2.303468 [I 2022-10-08 12:10:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 520/2304 [I 2022-10-08 12:10:28] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:21 [I 2022-10-08 12:10:28] ...trap/engines/engine.py.229: process: 0.79251 | load: 0.00299 [I 2022-10-08 12:10:28] ...trap/engines/engine.py.230: loss: 0.23681 [I 2022-10-08 12:10:28] ...optimizers/trijoint.py.151: total_norm: 2.740011 [I 2022-10-08 12:10:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 530/2304 [I 2022-10-08 12:10:36] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:22:50 [I 2022-10-08 12:10:36] ...trap/engines/engine.py.229: process: 0.76811 | load: 0.00319 [I 2022-10-08 12:10:36] ...trap/engines/engine.py.230: loss: 0.29739 [I 2022-10-08 12:10:36] ...optimizers/trijoint.py.151: total_norm: 3.476439 [I 2022-10-08 12:10:44] ...trap/engines/engine.py.224: train: epoch 4 | batch 540/2304 [I 2022-10-08 12:10:44] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:50 [I 2022-10-08 12:10:44] ...trap/engines/engine.py.229: process: 0.77092 | load: 0.00304 [I 2022-10-08 12:10:44] ...trap/engines/engine.py.230: loss: 0.18653 [I 2022-10-08 12:10:44] ...optimizers/trijoint.py.151: total_norm: 3.189531 [I 2022-10-08 12:10:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 550/2304 [I 2022-10-08 12:10:52] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:41 [I 2022-10-08 12:10:52] ...trap/engines/engine.py.229: process: 0.77685 | load: 0.00306 [I 2022-10-08 12:10:52] ...trap/engines/engine.py.230: loss: 0.29694 [I 2022-10-08 12:10:52] ...optimizers/trijoint.py.151: total_norm: 3.566898 [I 2022-10-08 12:11:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 560/2304 [I 2022-10-08 12:11:00] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:43 [I 2022-10-08 12:11:00] ...trap/engines/engine.py.229: process: 0.78896 | load: 0.00300 [I 2022-10-08 12:11:00] ...trap/engines/engine.py.230: loss: 0.20404 [I 2022-10-08 12:11:00] ...optimizers/trijoint.py.151: total_norm: 2.330154 [I 2022-10-08 12:11:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 570/2304 [I 2022-10-08 12:11:07] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:33 [I 2022-10-08 12:11:07] ...trap/engines/engine.py.229: process: 0.78215 | load: 0.00303 [I 2022-10-08 12:11:07] ...trap/engines/engine.py.230: loss: 0.20758 [I 2022-10-08 12:11:07] ...optimizers/trijoint.py.151: total_norm: 2.755829 [I 2022-10-08 12:11:15] ...trap/engines/engine.py.224: train: epoch 4 | batch 580/2304 [I 2022-10-08 12:11:15] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:15 [I 2022-10-08 12:11:15] ...trap/engines/engine.py.229: process: 0.77366 | load: 0.00302 [I 2022-10-08 12:11:15] ...trap/engines/engine.py.230: loss: 0.19349 [I 2022-10-08 12:11:15] ...optimizers/trijoint.py.151: total_norm: 3.276478 [I 2022-10-08 12:11:23] ...trap/engines/engine.py.224: train: epoch 4 | batch 590/2304 [I 2022-10-08 12:11:23] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:15 [I 2022-10-08 12:11:23] ...trap/engines/engine.py.229: process: 0.79257 | load: 0.00301 [I 2022-10-08 12:11:23] ...trap/engines/engine.py.230: loss: 0.28808 [I 2022-10-08 12:11:23] ...optimizers/trijoint.py.151: total_norm: 2.904355 [I 2022-10-08 12:11:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 600/2304 [I 2022-10-08 12:11:31] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:05 [I 2022-10-08 12:11:31] ...trap/engines/engine.py.229: process: 0.77887 | load: 0.00302 [I 2022-10-08 12:11:31] ...trap/engines/engine.py.230: loss: 0.21113 [I 2022-10-08 12:11:31] ...optimizers/trijoint.py.151: total_norm: 2.374966 [I 2022-10-08 12:11:39] ...trap/engines/engine.py.224: train: epoch 4 | batch 610/2304 [I 2022-10-08 12:11:39] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:00 [I 2022-10-08 12:11:39] ...trap/engines/engine.py.229: process: 0.78216 | load: 0.00306 [I 2022-10-08 12:11:39] ...trap/engines/engine.py.230: loss: 0.23300 [I 2022-10-08 12:11:39] ...optimizers/trijoint.py.151: total_norm: 2.724107 [I 2022-10-08 12:11:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 620/2304 [I 2022-10-08 12:11:46] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:48 [I 2022-10-08 12:11:46] ...trap/engines/engine.py.229: process: 0.77853 | load: 0.00303 [I 2022-10-08 12:11:46] ...trap/engines/engine.py.230: loss: 0.22415 [I 2022-10-08 12:11:46] ...optimizers/trijoint.py.151: total_norm: 3.325292 [I 2022-10-08 12:11:54] ...trap/engines/engine.py.224: train: epoch 4 | batch 630/2304 [I 2022-10-08 12:11:54] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:38 [I 2022-10-08 12:11:54] ...trap/engines/engine.py.229: process: 0.78642 | load: 0.00305 [I 2022-10-08 12:11:54] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-08 12:11:54] ...optimizers/trijoint.py.151: total_norm: 2.529877 [I 2022-10-08 12:12:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 640/2304 [I 2022-10-08 12:12:02] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:32 [I 2022-10-08 12:12:02] ...trap/engines/engine.py.229: process: 0.77969 | load: 0.00300 [I 2022-10-08 12:12:02] ...trap/engines/engine.py.230: loss: 0.21738 [I 2022-10-08 12:12:02] ...optimizers/trijoint.py.151: total_norm: 2.930052 [I 2022-10-08 12:12:10] ...trap/engines/engine.py.224: train: epoch 4 | batch 650/2304 [I 2022-10-08 12:12:10] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:22 [I 2022-10-08 12:12:10] ...trap/engines/engine.py.229: process: 0.76687 | load: 0.00306 [I 2022-10-08 12:12:10] ...trap/engines/engine.py.230: loss: 0.25675 [I 2022-10-08 12:12:10] ...optimizers/trijoint.py.151: total_norm: 3.093059 [I 2022-10-08 12:12:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 660/2304 [I 2022-10-08 12:12:18] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:16 [I 2022-10-08 12:12:18] ...trap/engines/engine.py.229: process: 0.78776 | load: 0.00306 [I 2022-10-08 12:12:18] ...trap/engines/engine.py.230: loss: 0.20739 [I 2022-10-08 12:12:18] ...optimizers/trijoint.py.151: total_norm: 2.438730 [I 2022-10-08 12:12:25] ...trap/engines/engine.py.224: train: epoch 4 | batch 670/2304 [I 2022-10-08 12:12:25] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:14 [I 2022-10-08 12:12:25] ...trap/engines/engine.py.229: process: 0.78538 | load: 0.00302 [I 2022-10-08 12:12:25] ...trap/engines/engine.py.230: loss: 0.28593 [I 2022-10-08 12:12:25] ...optimizers/trijoint.py.151: total_norm: 2.527625 [I 2022-10-08 12:12:33] ...trap/engines/engine.py.224: train: epoch 4 | batch 680/2304 [I 2022-10-08 12:12:33] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:00 [I 2022-10-08 12:12:33] ...trap/engines/engine.py.229: process: 0.77530 | load: 0.00308 [I 2022-10-08 12:12:33] ...trap/engines/engine.py.230: loss: 0.27874 [I 2022-10-08 12:12:33] ...optimizers/trijoint.py.151: total_norm: 4.081851 [I 2022-10-08 12:12:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 690/2304 [I 2022-10-08 12:12:41] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:51 [I 2022-10-08 12:12:41] ...trap/engines/engine.py.229: process: 0.77411 | load: 0.00304 [I 2022-10-08 12:12:41] ...trap/engines/engine.py.230: loss: 0.21471 [I 2022-10-08 12:12:41] ...optimizers/trijoint.py.151: total_norm: 2.580939 [I 2022-10-08 12:12:49] ...trap/engines/engine.py.224: train: epoch 4 | batch 700/2304 [I 2022-10-08 12:12:49] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:44 [I 2022-10-08 12:12:49] ...trap/engines/engine.py.229: process: 0.77088 | load: 0.00299 [I 2022-10-08 12:12:49] ...trap/engines/engine.py.230: loss: 0.30980 [I 2022-10-08 12:12:49] ...optimizers/trijoint.py.151: total_norm: 3.189635 [I 2022-10-08 12:12:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 710/2304 [I 2022-10-08 12:12:56] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:32 [I 2022-10-08 12:12:56] ...trap/engines/engine.py.229: process: 0.75456 | load: 0.00303 [I 2022-10-08 12:12:56] ...trap/engines/engine.py.230: loss: 0.17995 [I 2022-10-08 12:12:56] ...optimizers/trijoint.py.151: total_norm: 3.209745 [I 2022-10-08 12:13:04] ...trap/engines/engine.py.224: train: epoch 4 | batch 720/2304 [I 2022-10-08 12:13:04] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:29 [I 2022-10-08 12:13:04] ...trap/engines/engine.py.229: process: 0.77775 | load: 0.00304 [I 2022-10-08 12:13:04] ...trap/engines/engine.py.230: loss: 0.21529 [I 2022-10-08 12:13:04] ...optimizers/trijoint.py.151: total_norm: 2.407980 [I 2022-10-08 12:13:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 730/2304 [I 2022-10-08 12:13:12] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:20:22 [I 2022-10-08 12:13:12] ...trap/engines/engine.py.229: process: 0.77907 | load: 0.00303 [I 2022-10-08 12:13:12] ...trap/engines/engine.py.230: loss: 0.22349 [I 2022-10-08 12:13:12] ...optimizers/trijoint.py.151: total_norm: 3.528314 [I 2022-10-08 12:13:20] ...trap/engines/engine.py.224: train: epoch 4 | batch 740/2304 [I 2022-10-08 12:13:20] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:14 [I 2022-10-08 12:13:20] ...trap/engines/engine.py.229: process: 0.77819 | load: 0.00305 [I 2022-10-08 12:13:20] ...trap/engines/engine.py.230: loss: 0.18476 [I 2022-10-08 12:13:20] ...optimizers/trijoint.py.151: total_norm: 2.247172 [I 2022-10-08 12:13:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 750/2304 [I 2022-10-08 12:13:28] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:05 [I 2022-10-08 12:13:28] ...trap/engines/engine.py.229: process: 0.76789 | load: 0.00302 [I 2022-10-08 12:13:28] ...trap/engines/engine.py.230: loss: 0.20343 [I 2022-10-08 12:13:28] ...optimizers/trijoint.py.151: total_norm: 3.430994 [I 2022-10-08 12:13:35] ...trap/engines/engine.py.224: train: epoch 4 | batch 760/2304 [I 2022-10-08 12:13:35] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:19:53 [I 2022-10-08 12:13:35] ...trap/engines/engine.py.229: process: 0.76976 | load: 0.00301 [I 2022-10-08 12:13:35] ...trap/engines/engine.py.230: loss: 0.20991 [I 2022-10-08 12:13:35] ...optimizers/trijoint.py.151: total_norm: 2.549147 [I 2022-10-08 12:13:43] ...trap/engines/engine.py.224: train: epoch 4 | batch 770/2304 [I 2022-10-08 12:13:43] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:45 [I 2022-10-08 12:13:43] ...trap/engines/engine.py.229: process: 0.76360 | load: 0.00303 [I 2022-10-08 12:13:43] ...trap/engines/engine.py.230: loss: 0.26553 [I 2022-10-08 12:13:43] ...optimizers/trijoint.py.151: total_norm: 3.159016 [I 2022-10-08 12:13:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 780/2304 [I 2022-10-08 12:13:51] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:19:36 [I 2022-10-08 12:13:51] ...trap/engines/engine.py.229: process: 0.75708 | load: 0.00305 [I 2022-10-08 12:13:51] ...trap/engines/engine.py.230: loss: 0.21605 [I 2022-10-08 12:13:51] ...optimizers/trijoint.py.151: total_norm: 2.281487 [I 2022-10-08 12:13:59] ...trap/engines/engine.py.224: train: epoch 4 | batch 790/2304 [I 2022-10-08 12:13:59] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:32 [I 2022-10-08 12:13:59] ...trap/engines/engine.py.229: process: 0.77563 | load: 0.00300 [I 2022-10-08 12:13:59] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-08 12:13:59] ...optimizers/trijoint.py.151: total_norm: 2.512951 [I 2022-10-08 12:14:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 800/2304 [I 2022-10-08 12:14:06] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:28 [I 2022-10-08 12:14:06] ...trap/engines/engine.py.229: process: 0.78293 | load: 0.00318 [I 2022-10-08 12:14:06] ...trap/engines/engine.py.230: loss: 0.25881 [I 2022-10-08 12:14:06] ...optimizers/trijoint.py.151: total_norm: 3.187053 [I 2022-10-08 12:14:14] ...trap/engines/engine.py.224: train: epoch 4 | batch 810/2304 [I 2022-10-08 12:14:14] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:12 [I 2022-10-08 12:14:14] ...trap/engines/engine.py.229: process: 0.77181 | load: 0.00302 [I 2022-10-08 12:14:14] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-08 12:14:14] ...optimizers/trijoint.py.151: total_norm: 2.265553 [I 2022-10-08 12:14:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 820/2304 [I 2022-10-08 12:14:22] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:19:10 [I 2022-10-08 12:14:22] ...trap/engines/engine.py.229: process: 0.76940 | load: 0.00303 [I 2022-10-08 12:14:22] ...trap/engines/engine.py.230: loss: 0.22152 [I 2022-10-08 12:14:22] ...optimizers/trijoint.py.151: total_norm: 2.333243 [I 2022-10-08 12:14:30] ...trap/engines/engine.py.224: train: epoch 4 | batch 830/2304 [I 2022-10-08 12:14:30] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:18:58 [I 2022-10-08 12:14:30] ...trap/engines/engine.py.229: process: 0.76707 | load: 0.00307 [I 2022-10-08 12:14:30] ...trap/engines/engine.py.230: loss: 0.24612 [I 2022-10-08 12:14:30] ...optimizers/trijoint.py.151: total_norm: 3.395090 [I 2022-10-08 12:14:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 840/2304 [I 2022-10-08 12:14:37] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:18:54 [I 2022-10-08 12:14:37] ...trap/engines/engine.py.229: process: 0.77417 | load: 0.00309 [I 2022-10-08 12:14:37] ...trap/engines/engine.py.230: loss: 0.28588 [I 2022-10-08 12:14:37] ...optimizers/trijoint.py.151: total_norm: 3.144491 [I 2022-10-08 12:14:45] ...trap/engines/engine.py.224: train: epoch 4 | batch 850/2304 [I 2022-10-08 12:14:45] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:45 [I 2022-10-08 12:14:45] ...trap/engines/engine.py.229: process: 0.78349 | load: 0.00301 [I 2022-10-08 12:14:45] ...trap/engines/engine.py.230: loss: 0.20238 [I 2022-10-08 12:14:45] ...optimizers/trijoint.py.151: total_norm: 2.728533 [I 2022-10-08 12:14:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 860/2304 [I 2022-10-08 12:14:53] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:18:49 [I 2022-10-08 12:14:53] ...trap/engines/engine.py.229: process: 0.78790 | load: 0.00303 [I 2022-10-08 12:14:53] ...trap/engines/engine.py.230: loss: 0.29809 [I 2022-10-08 12:14:53] ...optimizers/trijoint.py.151: total_norm: 3.163252 [I 2022-10-08 12:15:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 870/2304 [I 2022-10-08 12:15:01] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:18:34 [I 2022-10-08 12:15:01] ...trap/engines/engine.py.229: process: 0.78384 | load: 0.00305 [I 2022-10-08 12:15:01] ...trap/engines/engine.py.230: loss: 0.32537 [I 2022-10-08 12:15:01] ...optimizers/trijoint.py.151: total_norm: 3.574535 [I 2022-10-08 12:15:09] ...trap/engines/engine.py.224: train: epoch 4 | batch 880/2304 [I 2022-10-08 12:15:09] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:18:18 [I 2022-10-08 12:15:09] ...trap/engines/engine.py.229: process: 0.76465 | load: 0.00304 [I 2022-10-08 12:15:09] ...trap/engines/engine.py.230: loss: 0.15294 [I 2022-10-08 12:15:09] ...optimizers/trijoint.py.151: total_norm: 2.590007 [I 2022-10-08 12:15:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 890/2304 [I 2022-10-08 12:15:16] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:15 [I 2022-10-08 12:15:16] ...trap/engines/engine.py.229: process: 0.77072 | load: 0.00301 [I 2022-10-08 12:15:16] ...trap/engines/engine.py.230: loss: 0.29595 [I 2022-10-08 12:15:16] ...optimizers/trijoint.py.151: total_norm: 3.000413 [I 2022-10-08 12:15:24] ...trap/engines/engine.py.224: train: epoch 4 | batch 900/2304 [I 2022-10-08 12:15:24] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:12 [I 2022-10-08 12:15:24] ...trap/engines/engine.py.229: process: 0.77818 | load: 0.00301 [I 2022-10-08 12:15:24] ...trap/engines/engine.py.230: loss: 0.20487 [I 2022-10-08 12:15:24] ...optimizers/trijoint.py.151: total_norm: 2.683786 [I 2022-10-08 12:15:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 910/2304 [I 2022-10-08 12:15:32] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:18:05 [I 2022-10-08 12:15:32] ...trap/engines/engine.py.229: process: 0.77965 | load: 0.00300 [I 2022-10-08 12:15:32] ...trap/engines/engine.py.230: loss: 0.25035 [I 2022-10-08 12:15:32] ...optimizers/trijoint.py.151: total_norm: 2.791750 [I 2022-10-08 12:15:40] ...trap/engines/engine.py.224: train: epoch 4 | batch 920/2304 [I 2022-10-08 12:15:40] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:17:50 [I 2022-10-08 12:15:40] ...trap/engines/engine.py.229: process: 0.76194 | load: 0.00305 [I 2022-10-08 12:15:40] ...trap/engines/engine.py.230: loss: 0.15910 [I 2022-10-08 12:15:40] ...optimizers/trijoint.py.151: total_norm: 2.632636 [I 2022-10-08 12:15:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 930/2304 [I 2022-10-08 12:15:47] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:49 [I 2022-10-08 12:15:47] ...trap/engines/engine.py.229: process: 0.78439 | load: 0.00302 [I 2022-10-08 12:15:47] ...trap/engines/engine.py.230: loss: 0.21189 [I 2022-10-08 12:15:47] ...optimizers/trijoint.py.151: total_norm: 3.064750 [I 2022-10-08 12:15:55] ...trap/engines/engine.py.224: train: epoch 4 | batch 940/2304 [I 2022-10-08 12:15:55] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:44 [I 2022-10-08 12:15:55] ...trap/engines/engine.py.229: process: 0.78895 | load: 0.00303 [I 2022-10-08 12:15:55] ...trap/engines/engine.py.230: loss: 0.27268 [I 2022-10-08 12:15:55] ...optimizers/trijoint.py.151: total_norm: 3.228262 [I 2022-10-08 12:16:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 950/2304 [I 2022-10-08 12:16:03] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:17:31 [I 2022-10-08 12:16:03] ...trap/engines/engine.py.229: process: 0.78293 | load: 0.00307 [I 2022-10-08 12:16:03] ...trap/engines/engine.py.230: loss: 0.16009 [I 2022-10-08 12:16:03] ...optimizers/trijoint.py.151: total_norm: 2.188186 [I 2022-10-08 12:16:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 960/2304 [I 2022-10-08 12:16:11] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:17:18 [I 2022-10-08 12:16:11] ...trap/engines/engine.py.229: process: 0.77609 | load: 0.00309 [I 2022-10-08 12:16:11] ...trap/engines/engine.py.230: loss: 0.22423 [I 2022-10-08 12:16:11] ...optimizers/trijoint.py.151: total_norm: 2.914577 [I 2022-10-08 12:16:19] ...trap/engines/engine.py.224: train: epoch 4 | batch 970/2304 [I 2022-10-08 12:16:19] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:17:16 [I 2022-10-08 12:16:19] ...trap/engines/engine.py.229: process: 0.77859 | load: 0.00307 [I 2022-10-08 12:16:19] ...trap/engines/engine.py.230: loss: 0.16732 [I 2022-10-08 12:16:19] ...optimizers/trijoint.py.151: total_norm: 2.388334 [I 2022-10-08 12:16:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 980/2304 [I 2022-10-08 12:16:26] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:17:02 [I 2022-10-08 12:16:26] ...trap/engines/engine.py.229: process: 0.75847 | load: 0.00300 [I 2022-10-08 12:16:26] ...trap/engines/engine.py.230: loss: 0.30831 [I 2022-10-08 12:16:26] ...optimizers/trijoint.py.151: total_norm: 3.127511 [I 2022-10-08 12:16:34] ...trap/engines/engine.py.224: train: epoch 4 | batch 990/2304 [I 2022-10-08 12:16:34] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:16:57 [I 2022-10-08 12:16:34] ...trap/engines/engine.py.229: process: 0.77675 | load: 0.00304 [I 2022-10-08 12:16:34] ...trap/engines/engine.py.230: loss: 0.31321 [I 2022-10-08 12:16:34] ...optimizers/trijoint.py.151: total_norm: 3.064630 [I 2022-10-08 12:16:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 1000/2304 [I 2022-10-08 12:16:42] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:16:54 [I 2022-10-08 12:16:42] ...trap/engines/engine.py.229: process: 0.78925 | load: 0.00301 [I 2022-10-08 12:16:42] ...trap/engines/engine.py.230: loss: 0.18090 [I 2022-10-08 12:16:42] ...optimizers/trijoint.py.151: total_norm: 2.801192 [I 2022-10-08 12:16:50] ...trap/engines/engine.py.224: train: epoch 4 | batch 1010/2304 [I 2022-10-08 12:16:50] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:16:48 [I 2022-10-08 12:16:50] ...trap/engines/engine.py.229: process: 0.77297 | load: 0.00301 [I 2022-10-08 12:16:50] ...trap/engines/engine.py.230: loss: 0.24048 [I 2022-10-08 12:16:50] ...optimizers/trijoint.py.151: total_norm: 2.414784 [I 2022-10-08 12:16:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 1020/2304 [I 2022-10-08 12:16:58] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:16:37 [I 2022-10-08 12:16:58] ...trap/engines/engine.py.229: process: 0.77158 | load: 0.00301 [I 2022-10-08 12:16:58] ...trap/engines/engine.py.230: loss: 0.19948 [I 2022-10-08 12:16:58] ...optimizers/trijoint.py.151: total_norm: 2.884221 [I 2022-10-08 12:17:05] ...trap/engines/engine.py.224: train: epoch 4 | batch 1030/2304 [I 2022-10-08 12:17:05] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:16:24 [I 2022-10-08 12:17:05] ...trap/engines/engine.py.229: process: 0.76686 | load: 0.00299 [I 2022-10-08 12:17:05] ...trap/engines/engine.py.230: loss: 0.22888 [I 2022-10-08 12:17:05] ...optimizers/trijoint.py.151: total_norm: 3.950076 [I 2022-10-08 12:17:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 1040/2304 [I 2022-10-08 12:17:13] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:16:16 [I 2022-10-08 12:17:13] ...trap/engines/engine.py.229: process: 0.76843 | load: 0.00300 [I 2022-10-08 12:17:13] ...trap/engines/engine.py.230: loss: 0.27529 [I 2022-10-08 12:17:13] ...optimizers/trijoint.py.151: total_norm: 3.009954 [I 2022-10-08 12:17:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1050/2304 [I 2022-10-08 12:17:21] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:16:12 [I 2022-10-08 12:17:21] ...trap/engines/engine.py.229: process: 0.76422 | load: 0.00301 [I 2022-10-08 12:17:21] ...trap/engines/engine.py.230: loss: 0.30873 [I 2022-10-08 12:17:21] ...optimizers/trijoint.py.151: total_norm: 3.395279 [I 2022-10-08 12:17:29] ...trap/engines/engine.py.224: train: epoch 4 | batch 1060/2304 [I 2022-10-08 12:17:29] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:16:02 [I 2022-10-08 12:17:29] ...trap/engines/engine.py.229: process: 0.77148 | load: 0.00308 [I 2022-10-08 12:17:29] ...trap/engines/engine.py.230: loss: 0.29188 [I 2022-10-08 12:17:29] ...optimizers/trijoint.py.151: total_norm: 3.050996 [I 2022-10-08 12:17:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1070/2304 [I 2022-10-08 12:17:36] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:16:00 [I 2022-10-08 12:17:36] ...trap/engines/engine.py.229: process: 0.79414 | load: 0.00305 [I 2022-10-08 12:17:36] ...trap/engines/engine.py.230: loss: 0.30806 [I 2022-10-08 12:17:36] ...optimizers/trijoint.py.151: total_norm: 3.868990 [I 2022-10-08 12:17:44] ...trap/engines/engine.py.224: train: epoch 4 | batch 1080/2304 [I 2022-10-08 12:17:44] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:15:49 [I 2022-10-08 12:17:44] ...trap/engines/engine.py.229: process: 0.78007 | load: 0.00297 [I 2022-10-08 12:17:44] ...trap/engines/engine.py.230: loss: 0.22095 [I 2022-10-08 12:17:44] ...optimizers/trijoint.py.151: total_norm: 2.949965 [I 2022-10-08 12:17:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 1090/2304 [I 2022-10-08 12:17:52] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:15:42 [I 2022-10-08 12:17:52] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00301 [I 2022-10-08 12:17:52] ...trap/engines/engine.py.230: loss: 0.34707 [I 2022-10-08 12:17:52] ...optimizers/trijoint.py.151: total_norm: 3.627015 [I 2022-10-08 12:18:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 1100/2304 [I 2022-10-08 12:18:00] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:15:35 [I 2022-10-08 12:18:00] ...trap/engines/engine.py.229: process: 0.77078 | load: 0.00305 [I 2022-10-08 12:18:00] ...trap/engines/engine.py.230: loss: 0.22255 [I 2022-10-08 12:18:00] ...optimizers/trijoint.py.151: total_norm: 2.718632 [I 2022-10-08 12:18:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 1110/2304 [I 2022-10-08 12:18:07] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:15:23 [I 2022-10-08 12:18:07] ...trap/engines/engine.py.229: process: 0.77674 | load: 0.00302 [I 2022-10-08 12:18:07] ...trap/engines/engine.py.230: loss: 0.21679 [I 2022-10-08 12:18:07] ...optimizers/trijoint.py.151: total_norm: 2.083785 [I 2022-10-08 12:18:15] ...trap/engines/engine.py.224: train: epoch 4 | batch 1120/2304 [I 2022-10-08 12:18:15] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:15:18 [I 2022-10-08 12:18:15] ...trap/engines/engine.py.229: process: 0.78863 | load: 0.00301 [I 2022-10-08 12:18:15] ...trap/engines/engine.py.230: loss: 0.22176 [I 2022-10-08 12:18:15] ...optimizers/trijoint.py.151: total_norm: 2.640841 [I 2022-10-08 12:18:23] ...trap/engines/engine.py.224: train: epoch 4 | batch 1130/2304 [I 2022-10-08 12:18:23] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:15:13 [I 2022-10-08 12:18:23] ...trap/engines/engine.py.229: process: 0.77949 | load: 0.00303 [I 2022-10-08 12:18:23] ...trap/engines/engine.py.230: loss: 0.28621 [I 2022-10-08 12:18:23] ...optimizers/trijoint.py.151: total_norm: 3.070577 [I 2022-10-08 12:18:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1140/2304 [I 2022-10-08 12:18:31] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:15:07 [I 2022-10-08 12:18:31] ...trap/engines/engine.py.229: process: 0.78053 | load: 0.00303 [I 2022-10-08 12:18:31] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-08 12:18:31] ...optimizers/trijoint.py.151: total_norm: 1.689042 [I 2022-10-08 12:18:39] ...trap/engines/engine.py.224: train: epoch 4 | batch 1150/2304 [I 2022-10-08 12:18:39] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:14:58 [I 2022-10-08 12:18:39] ...trap/engines/engine.py.229: process: 0.78049 | load: 0.00299 [I 2022-10-08 12:18:39] ...trap/engines/engine.py.230: loss: 0.30787 [I 2022-10-08 12:18:39] ...optimizers/trijoint.py.151: total_norm: 4.169767 [I 2022-10-08 12:18:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1160/2304 [I 2022-10-08 12:18:46] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:14:48 [I 2022-10-08 12:18:46] ...trap/engines/engine.py.229: process: 0.78310 | load: 0.00299 [I 2022-10-08 12:18:46] ...trap/engines/engine.py.230: loss: 0.23382 [I 2022-10-08 12:18:46] ...optimizers/trijoint.py.151: total_norm: 2.968126 [I 2022-10-08 12:18:54] ...trap/engines/engine.py.224: train: epoch 4 | batch 1170/2304 [I 2022-10-08 12:18:54] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:14:41 [I 2022-10-08 12:18:54] ...trap/engines/engine.py.229: process: 0.77645 | load: 0.00303 [I 2022-10-08 12:18:54] ...trap/engines/engine.py.230: loss: 0.19168 [I 2022-10-08 12:18:54] ...optimizers/trijoint.py.151: total_norm: 2.868418 [I 2022-10-08 12:19:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 1180/2304 [I 2022-10-08 12:19:02] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:14:25 [I 2022-10-08 12:19:02] ...trap/engines/engine.py.229: process: 0.76151 | load: 0.00303 [I 2022-10-08 12:19:02] ...trap/engines/engine.py.230: loss: 0.32463 [I 2022-10-08 12:19:02] ...optimizers/trijoint.py.151: total_norm: 3.554395 [I 2022-10-08 12:19:10] ...trap/engines/engine.py.224: train: epoch 4 | batch 1190/2304 [I 2022-10-08 12:19:10] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:14:27 [I 2022-10-08 12:19:10] ...trap/engines/engine.py.229: process: 0.78461 | load: 0.00302 [I 2022-10-08 12:19:10] ...trap/engines/engine.py.230: loss: 0.21185 [I 2022-10-08 12:19:10] ...optimizers/trijoint.py.151: total_norm: 2.727197 [I 2022-10-08 12:19:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 1200/2304 [I 2022-10-08 12:19:18] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:14:20 [I 2022-10-08 12:19:18] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00302 [I 2022-10-08 12:19:18] ...trap/engines/engine.py.230: loss: 0.22900 [I 2022-10-08 12:19:18] ...optimizers/trijoint.py.151: total_norm: 2.769903 [I 2022-10-08 12:19:25] ...trap/engines/engine.py.224: train: epoch 4 | batch 1210/2304 [I 2022-10-08 12:19:25] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:14:15 [I 2022-10-08 12:19:25] ...trap/engines/engine.py.229: process: 0.77712 | load: 0.00305 [I 2022-10-08 12:19:25] ...trap/engines/engine.py.230: loss: 0.16594 [I 2022-10-08 12:19:25] ...optimizers/trijoint.py.151: total_norm: 2.335484 [I 2022-10-08 12:19:33] ...trap/engines/engine.py.224: train: epoch 4 | batch 1220/2304 [I 2022-10-08 12:19:33] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:14:03 [I 2022-10-08 12:19:33] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00305 [I 2022-10-08 12:19:33] ...trap/engines/engine.py.230: loss: 0.30719 [I 2022-10-08 12:19:33] ...optimizers/trijoint.py.151: total_norm: 4.380492 [I 2022-10-08 12:19:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1230/2304 [I 2022-10-08 12:19:41] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:13:59 [I 2022-10-08 12:19:41] ...trap/engines/engine.py.229: process: 0.78213 | load: 0.00300 [I 2022-10-08 12:19:41] ...trap/engines/engine.py.230: loss: 0.18782 [I 2022-10-08 12:19:41] ...optimizers/trijoint.py.151: total_norm: 3.063835 [I 2022-10-08 12:19:49] ...trap/engines/engine.py.224: train: epoch 4 | batch 1240/2304 [I 2022-10-08 12:19:49] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:13:43 [I 2022-10-08 12:19:49] ...trap/engines/engine.py.229: process: 0.77912 | load: 0.00304 [I 2022-10-08 12:19:49] ...trap/engines/engine.py.230: loss: 0.22542 [I 2022-10-08 12:19:49] ...optimizers/trijoint.py.151: total_norm: 2.534453 [I 2022-10-08 12:19:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 1250/2304 [I 2022-10-08 12:19:57] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:13:40 [I 2022-10-08 12:19:57] ...trap/engines/engine.py.229: process: 0.77768 | load: 0.00306 [I 2022-10-08 12:19:57] ...trap/engines/engine.py.230: loss: 0.28863 [I 2022-10-08 12:19:57] ...optimizers/trijoint.py.151: total_norm: 2.827683 [I 2022-10-08 12:20:04] ...trap/engines/engine.py.224: train: epoch 4 | batch 1260/2304 [I 2022-10-08 12:20:04] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:13:34 [I 2022-10-08 12:20:04] ...trap/engines/engine.py.229: process: 0.78460 | load: 0.00305 [I 2022-10-08 12:20:04] ...trap/engines/engine.py.230: loss: 0.17351 [I 2022-10-08 12:20:04] ...optimizers/trijoint.py.151: total_norm: 2.522380 [I 2022-10-08 12:20:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 1270/2304 [I 2022-10-08 12:20:12] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:13:22 [I 2022-10-08 12:20:12] ...trap/engines/engine.py.229: process: 0.77361 | load: 0.00302 [I 2022-10-08 12:20:12] ...trap/engines/engine.py.230: loss: 0.23587 [I 2022-10-08 12:20:12] ...optimizers/trijoint.py.151: total_norm: 2.644950 [I 2022-10-08 12:20:20] ...trap/engines/engine.py.224: train: epoch 4 | batch 1280/2304 [I 2022-10-08 12:20:20] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:13:14 [I 2022-10-08 12:20:20] ...trap/engines/engine.py.229: process: 0.78685 | load: 0.00302 [I 2022-10-08 12:20:20] ...trap/engines/engine.py.230: loss: 0.29732 [I 2022-10-08 12:20:20] ...optimizers/trijoint.py.151: total_norm: 2.860391 [I 2022-10-08 12:20:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 1290/2304 [I 2022-10-08 12:20:28] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:13:10 [I 2022-10-08 12:20:28] ...trap/engines/engine.py.229: process: 0.78299 | load: 0.00302 [I 2022-10-08 12:20:28] ...trap/engines/engine.py.230: loss: 0.19626 [I 2022-10-08 12:20:28] ...optimizers/trijoint.py.151: total_norm: 2.081345 [I 2022-10-08 12:20:35] ...trap/engines/engine.py.224: train: epoch 4 | batch 1300/2304 [I 2022-10-08 12:20:35] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:13:00 [I 2022-10-08 12:20:35] ...trap/engines/engine.py.229: process: 0.78052 | load: 0.00300 [I 2022-10-08 12:20:35] ...trap/engines/engine.py.230: loss: 0.23767 [I 2022-10-08 12:20:35] ...optimizers/trijoint.py.151: total_norm: 4.311762 [I 2022-10-08 12:20:43] ...trap/engines/engine.py.224: train: epoch 4 | batch 1310/2304 [I 2022-10-08 12:20:43] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:12:55 [I 2022-10-08 12:20:43] ...trap/engines/engine.py.229: process: 0.78973 | load: 0.00304 [I 2022-10-08 12:20:43] ...trap/engines/engine.py.230: loss: 0.27702 [I 2022-10-08 12:20:43] ...optimizers/trijoint.py.151: total_norm: 2.706469 [I 2022-10-08 12:20:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1320/2304 [I 2022-10-08 12:20:51] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:12:41 [I 2022-10-08 12:20:51] ...trap/engines/engine.py.229: process: 0.77465 | load: 0.00298 [I 2022-10-08 12:20:51] ...trap/engines/engine.py.230: loss: 0.27626 [I 2022-10-08 12:20:51] ...optimizers/trijoint.py.151: total_norm: 3.048017 [I 2022-10-08 12:20:59] ...trap/engines/engine.py.224: train: epoch 4 | batch 1330/2304 [I 2022-10-08 12:20:59] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:12:36 [I 2022-10-08 12:20:59] ...trap/engines/engine.py.229: process: 0.77221 | load: 0.00329 [I 2022-10-08 12:20:59] ...trap/engines/engine.py.230: loss: 0.22576 [I 2022-10-08 12:20:59] ...optimizers/trijoint.py.151: total_norm: 2.226373 [I 2022-10-08 12:21:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 1340/2304 [I 2022-10-08 12:21:07] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:12:25 [I 2022-10-08 12:21:07] ...trap/engines/engine.py.229: process: 0.76874 | load: 0.00301 [I 2022-10-08 12:21:07] ...trap/engines/engine.py.230: loss: 0.18384 [I 2022-10-08 12:21:07] ...optimizers/trijoint.py.151: total_norm: 2.539009 [I 2022-10-08 12:21:14] ...trap/engines/engine.py.224: train: epoch 4 | batch 1350/2304 [I 2022-10-08 12:21:14] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:12:23 [I 2022-10-08 12:21:14] ...trap/engines/engine.py.229: process: 0.79335 | load: 0.00301 [I 2022-10-08 12:21:14] ...trap/engines/engine.py.230: loss: 0.19652 [I 2022-10-08 12:21:14] ...optimizers/trijoint.py.151: total_norm: 2.362231 [I 2022-10-08 12:21:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 1360/2304 [I 2022-10-08 12:21:22] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:12:14 [I 2022-10-08 12:21:22] ...trap/engines/engine.py.229: process: 0.77351 | load: 0.00311 [I 2022-10-08 12:21:22] ...trap/engines/engine.py.230: loss: 0.18665 [I 2022-10-08 12:21:22] ...optimizers/trijoint.py.151: total_norm: 1.930568 [I 2022-10-08 12:21:30] ...trap/engines/engine.py.224: train: epoch 4 | batch 1370/2304 [I 2022-10-08 12:21:30] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:12:08 [I 2022-10-08 12:21:30] ...trap/engines/engine.py.229: process: 0.77809 | load: 0.00309 [I 2022-10-08 12:21:30] ...trap/engines/engine.py.230: loss: 0.23130 [I 2022-10-08 12:21:30] ...optimizers/trijoint.py.151: total_norm: 2.225963 [I 2022-10-08 12:21:38] ...trap/engines/engine.py.224: train: epoch 4 | batch 1380/2304 [I 2022-10-08 12:21:38] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:11:54 [I 2022-10-08 12:21:38] ...trap/engines/engine.py.229: process: 0.76429 | load: 0.00301 [I 2022-10-08 12:21:38] ...trap/engines/engine.py.230: loss: 0.24339 [I 2022-10-08 12:21:38] ...optimizers/trijoint.py.151: total_norm: 3.785526 [I 2022-10-08 12:21:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1390/2304 [I 2022-10-08 12:21:46] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:11:51 [I 2022-10-08 12:21:46] ...trap/engines/engine.py.229: process: 0.77930 | load: 0.00300 [I 2022-10-08 12:21:46] ...trap/engines/engine.py.230: loss: 0.20109 [I 2022-10-08 12:21:46] ...optimizers/trijoint.py.151: total_norm: 2.799564 [I 2022-10-08 12:21:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 1400/2304 [I 2022-10-08 12:21:53] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:11:46 [I 2022-10-08 12:21:53] ...trap/engines/engine.py.229: process: 0.77568 | load: 0.00303 [I 2022-10-08 12:21:53] ...trap/engines/engine.py.230: loss: 0.25994 [I 2022-10-08 12:21:53] ...optimizers/trijoint.py.151: total_norm: 2.631345 [I 2022-10-08 12:22:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1410/2304 [I 2022-10-08 12:22:01] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:11:33 [I 2022-10-08 12:22:01] ...trap/engines/engine.py.229: process: 0.77595 | load: 0.00307 [I 2022-10-08 12:22:01] ...trap/engines/engine.py.230: loss: 0.27976 [I 2022-10-08 12:22:01] ...optimizers/trijoint.py.151: total_norm: 3.279258 [I 2022-10-08 12:22:09] ...trap/engines/engine.py.224: train: epoch 4 | batch 1420/2304 [I 2022-10-08 12:22:09] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:11:25 [I 2022-10-08 12:22:09] ...trap/engines/engine.py.229: process: 0.76369 | load: 0.00303 [I 2022-10-08 12:22:09] ...trap/engines/engine.py.230: loss: 0.21394 [I 2022-10-08 12:22:09] ...optimizers/trijoint.py.151: total_norm: 2.499991 [I 2022-10-08 12:22:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 1430/2304 [I 2022-10-08 12:22:17] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:11:19 [I 2022-10-08 12:22:17] ...trap/engines/engine.py.229: process: 0.78782 | load: 0.00300 [I 2022-10-08 12:22:17] ...trap/engines/engine.py.230: loss: 0.22133 [I 2022-10-08 12:22:17] ...optimizers/trijoint.py.151: total_norm: 3.078995 [I 2022-10-08 12:22:24] ...trap/engines/engine.py.224: train: epoch 4 | batch 1440/2304 [I 2022-10-08 12:22:24] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:11:11 [I 2022-10-08 12:22:24] ...trap/engines/engine.py.229: process: 0.78135 | load: 0.00305 [I 2022-10-08 12:22:24] ...trap/engines/engine.py.230: loss: 0.25067 [I 2022-10-08 12:22:24] ...optimizers/trijoint.py.151: total_norm: 2.996623 [I 2022-10-08 12:22:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 1450/2304 [I 2022-10-08 12:22:32] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:11:00 [I 2022-10-08 12:22:32] ...trap/engines/engine.py.229: process: 0.77382 | load: 0.00303 [I 2022-10-08 12:22:32] ...trap/engines/engine.py.230: loss: 0.30736 [I 2022-10-08 12:22:32] ...optimizers/trijoint.py.151: total_norm: 3.774729 [I 2022-10-08 12:22:40] ...trap/engines/engine.py.224: train: epoch 4 | batch 1460/2304 [I 2022-10-08 12:22:40] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:10:55 [I 2022-10-08 12:22:40] ...trap/engines/engine.py.229: process: 0.77435 | load: 0.00301 [I 2022-10-08 12:22:40] ...trap/engines/engine.py.230: loss: 0.24657 [I 2022-10-08 12:22:40] ...optimizers/trijoint.py.151: total_norm: 2.893049 [I 2022-10-08 12:22:48] ...trap/engines/engine.py.224: train: epoch 4 | batch 1470/2304 [I 2022-10-08 12:22:48] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:10:48 [I 2022-10-08 12:22:48] ...trap/engines/engine.py.229: process: 0.77784 | load: 0.00304 [I 2022-10-08 12:22:48] ...trap/engines/engine.py.230: loss: 0.22014 [I 2022-10-08 12:22:48] ...optimizers/trijoint.py.151: total_norm: 2.237373 [I 2022-10-08 12:22:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1480/2304 [I 2022-10-08 12:22:56] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:10:40 [I 2022-10-08 12:22:56] ...trap/engines/engine.py.229: process: 0.77852 | load: 0.00302 [I 2022-10-08 12:22:56] ...trap/engines/engine.py.230: loss: 0.24828 [I 2022-10-08 12:22:56] ...optimizers/trijoint.py.151: total_norm: 3.590097 [I 2022-10-08 12:23:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 1490/2304 [I 2022-10-08 12:23:03] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:10:29 [I 2022-10-08 12:23:03] ...trap/engines/engine.py.229: process: 0.76424 | load: 0.00304 [I 2022-10-08 12:23:03] ...trap/engines/engine.py.230: loss: 0.30213 [I 2022-10-08 12:23:03] ...optimizers/trijoint.py.151: total_norm: 2.964788 [I 2022-10-08 12:23:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1500/2304 [I 2022-10-08 12:23:11] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:10:22 [I 2022-10-08 12:23:11] ...trap/engines/engine.py.229: process: 0.77629 | load: 0.00301 [I 2022-10-08 12:23:11] ...trap/engines/engine.py.230: loss: 0.28932 [I 2022-10-08 12:23:11] ...optimizers/trijoint.py.151: total_norm: 3.342537 [I 2022-10-08 12:23:19] ...trap/engines/engine.py.224: train: epoch 4 | batch 1510/2304 [I 2022-10-08 12:23:19] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:10:17 [I 2022-10-08 12:23:19] ...trap/engines/engine.py.229: process: 0.76324 | load: 0.00304 [I 2022-10-08 12:23:19] ...trap/engines/engine.py.230: loss: 0.26964 [I 2022-10-08 12:23:19] ...optimizers/trijoint.py.151: total_norm: 3.039428 [I 2022-10-08 12:23:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 1520/2304 [I 2022-10-08 12:23:27] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:10:05 [I 2022-10-08 12:23:27] ...trap/engines/engine.py.229: process: 0.77034 | load: 0.00306 [I 2022-10-08 12:23:27] ...trap/engines/engine.py.230: loss: 0.24437 [I 2022-10-08 12:23:27] ...optimizers/trijoint.py.151: total_norm: 2.479068 [I 2022-10-08 12:23:34] ...trap/engines/engine.py.224: train: epoch 4 | batch 1530/2304 [I 2022-10-08 12:23:34] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:09:59 [I 2022-10-08 12:23:34] ...trap/engines/engine.py.229: process: 0.77955 | load: 0.00301 [I 2022-10-08 12:23:34] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-08 12:23:34] ...optimizers/trijoint.py.151: total_norm: 2.363622 [I 2022-10-08 12:23:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 1540/2304 [I 2022-10-08 12:23:42] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:09:52 [I 2022-10-08 12:23:42] ...trap/engines/engine.py.229: process: 0.77992 | load: 0.00313 [I 2022-10-08 12:23:42] ...trap/engines/engine.py.230: loss: 0.16504 [I 2022-10-08 12:23:42] ...optimizers/trijoint.py.151: total_norm: 2.137232 [I 2022-10-08 12:23:50] ...trap/engines/engine.py.224: train: epoch 4 | batch 1550/2304 [I 2022-10-08 12:23:50] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:09:46 [I 2022-10-08 12:23:50] ...trap/engines/engine.py.229: process: 0.77537 | load: 0.00314 [I 2022-10-08 12:23:50] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-08 12:23:50] ...optimizers/trijoint.py.151: total_norm: 2.612545 [I 2022-10-08 12:23:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 1560/2304 [I 2022-10-08 12:23:58] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:09:45 [I 2022-10-08 12:23:58] ...trap/engines/engine.py.229: process: 0.79225 | load: 0.00309 [I 2022-10-08 12:23:58] ...trap/engines/engine.py.230: loss: 0.18860 [I 2022-10-08 12:23:58] ...optimizers/trijoint.py.151: total_norm: 2.723638 [I 2022-10-08 12:24:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1570/2304 [I 2022-10-08 12:24:06] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:09:32 [I 2022-10-08 12:24:06] ...trap/engines/engine.py.229: process: 0.77671 | load: 0.00315 [I 2022-10-08 12:24:06] ...trap/engines/engine.py.230: loss: 0.22301 [I 2022-10-08 12:24:06] ...optimizers/trijoint.py.151: total_norm: 2.441330 [I 2022-10-08 12:24:14] ...trap/engines/engine.py.224: train: epoch 4 | batch 1580/2304 [I 2022-10-08 12:24:14] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:09:24 [I 2022-10-08 12:24:14] ...trap/engines/engine.py.229: process: 0.78425 | load: 0.00301 [I 2022-10-08 12:24:14] ...trap/engines/engine.py.230: loss: 0.24676 [I 2022-10-08 12:24:14] ...optimizers/trijoint.py.151: total_norm: 2.381172 [I 2022-10-08 12:24:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1590/2304 [I 2022-10-08 12:24:21] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:09:15 [I 2022-10-08 12:24:21] ...trap/engines/engine.py.229: process: 0.76993 | load: 0.00300 [I 2022-10-08 12:24:21] ...trap/engines/engine.py.230: loss: 0.26803 [I 2022-10-08 12:24:21] ...optimizers/trijoint.py.151: total_norm: 3.113434 [I 2022-10-08 12:24:29] ...trap/engines/engine.py.224: train: epoch 4 | batch 1600/2304 [I 2022-10-08 12:24:29] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:09:06 [I 2022-10-08 12:24:29] ...trap/engines/engine.py.229: process: 0.77311 | load: 0.00302 [I 2022-10-08 12:24:29] ...trap/engines/engine.py.230: loss: 0.28581 [I 2022-10-08 12:24:29] ...optimizers/trijoint.py.151: total_norm: 2.155165 [I 2022-10-08 12:24:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 1610/2304 [I 2022-10-08 12:24:37] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:09:03 [I 2022-10-08 12:24:37] ...trap/engines/engine.py.229: process: 0.77310 | load: 0.00299 [I 2022-10-08 12:24:37] ...trap/engines/engine.py.230: loss: 0.28225 [I 2022-10-08 12:24:37] ...optimizers/trijoint.py.151: total_norm: 3.339000 [I 2022-10-08 12:24:45] ...trap/engines/engine.py.224: train: epoch 4 | batch 1620/2304 [I 2022-10-08 12:24:45] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:08:52 [I 2022-10-08 12:24:45] ...trap/engines/engine.py.229: process: 0.78141 | load: 0.00309 [I 2022-10-08 12:24:45] ...trap/engines/engine.py.230: loss: 0.25561 [I 2022-10-08 12:24:45] ...optimizers/trijoint.py.151: total_norm: 3.091973 [I 2022-10-08 12:24:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 1630/2304 [I 2022-10-08 12:24:53] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:08:43 [I 2022-10-08 12:24:53] ...trap/engines/engine.py.229: process: 0.76086 | load: 0.00314 [I 2022-10-08 12:24:53] ...trap/engines/engine.py.230: loss: 0.20793 [I 2022-10-08 12:24:53] ...optimizers/trijoint.py.151: total_norm: 2.903728 [I 2022-10-08 12:25:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 1640/2304 [I 2022-10-08 12:25:00] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:08:34 [I 2022-10-08 12:25:00] ...trap/engines/engine.py.229: process: 0.78312 | load: 0.00309 [I 2022-10-08 12:25:00] ...trap/engines/engine.py.230: loss: 0.21826 [I 2022-10-08 12:25:00] ...optimizers/trijoint.py.151: total_norm: 2.099134 [I 2022-10-08 12:25:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 1650/2304 [I 2022-10-08 12:25:08] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:08:28 [I 2022-10-08 12:25:08] ...trap/engines/engine.py.229: process: 0.77643 | load: 0.00300 [I 2022-10-08 12:25:08] ...trap/engines/engine.py.230: loss: 0.29013 [I 2022-10-08 12:25:08] ...optimizers/trijoint.py.151: total_norm: 2.613637 [I 2022-10-08 12:25:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1660/2304 [I 2022-10-08 12:25:16] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:08:21 [I 2022-10-08 12:25:16] ...trap/engines/engine.py.229: process: 0.77492 | load: 0.00301 [I 2022-10-08 12:25:16] ...trap/engines/engine.py.230: loss: 0.27210 [I 2022-10-08 12:25:16] ...optimizers/trijoint.py.151: total_norm: 2.846051 [I 2022-10-08 12:25:24] ...trap/engines/engine.py.224: train: epoch 4 | batch 1670/2304 [I 2022-10-08 12:25:24] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:08:13 [I 2022-10-08 12:25:24] ...trap/engines/engine.py.229: process: 0.77547 | load: 0.00303 [I 2022-10-08 12:25:24] ...trap/engines/engine.py.230: loss: 0.20050 [I 2022-10-08 12:25:24] ...optimizers/trijoint.py.151: total_norm: 2.115067 [I 2022-10-08 12:25:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 1680/2304 [I 2022-10-08 12:25:32] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:08:04 [I 2022-10-08 12:25:32] ...trap/engines/engine.py.229: process: 0.76872 | load: 0.00303 [I 2022-10-08 12:25:32] ...trap/engines/engine.py.230: loss: 0.32093 [I 2022-10-08 12:25:32] ...optimizers/trijoint.py.151: total_norm: 3.570777 [I 2022-10-08 12:25:39] ...trap/engines/engine.py.224: train: epoch 4 | batch 1690/2304 [I 2022-10-08 12:25:39] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:07:57 [I 2022-10-08 12:25:39] ...trap/engines/engine.py.229: process: 0.78326 | load: 0.00305 [I 2022-10-08 12:25:39] ...trap/engines/engine.py.230: loss: 0.16758 [I 2022-10-08 12:25:39] ...optimizers/trijoint.py.151: total_norm: 2.287349 [I 2022-10-08 12:25:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 1700/2304 [I 2022-10-08 12:25:47] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:07:49 [I 2022-10-08 12:25:47] ...trap/engines/engine.py.229: process: 0.77829 | load: 0.00303 [I 2022-10-08 12:25:47] ...trap/engines/engine.py.230: loss: 0.22789 [I 2022-10-08 12:25:47] ...optimizers/trijoint.py.151: total_norm: 2.219810 [I 2022-10-08 12:25:55] ...trap/engines/engine.py.224: train: epoch 4 | batch 1710/2304 [I 2022-10-08 12:25:55] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:07:43 [I 2022-10-08 12:25:55] ...trap/engines/engine.py.229: process: 0.78330 | load: 0.00301 [I 2022-10-08 12:25:55] ...trap/engines/engine.py.230: loss: 0.20169 [I 2022-10-08 12:25:55] ...optimizers/trijoint.py.151: total_norm: 2.369233 [I 2022-10-08 12:26:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 1720/2304 [I 2022-10-08 12:26:03] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:07:35 [I 2022-10-08 12:26:03] ...trap/engines/engine.py.229: process: 0.78170 | load: 0.00307 [I 2022-10-08 12:26:03] ...trap/engines/engine.py.230: loss: 0.29699 [I 2022-10-08 12:26:03] ...optimizers/trijoint.py.151: total_norm: 3.427346 [I 2022-10-08 12:26:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1730/2304 [I 2022-10-08 12:26:11] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:07:27 [I 2022-10-08 12:26:11] ...trap/engines/engine.py.229: process: 0.77284 | load: 0.00303 [I 2022-10-08 12:26:11] ...trap/engines/engine.py.230: loss: 0.20067 [I 2022-10-08 12:26:11] ...optimizers/trijoint.py.151: total_norm: 2.324819 [I 2022-10-08 12:26:19] ...trap/engines/engine.py.224: train: epoch 4 | batch 1740/2304 [I 2022-10-08 12:26:19] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:07:21 [I 2022-10-08 12:26:19] ...trap/engines/engine.py.229: process: 0.78616 | load: 0.00302 [I 2022-10-08 12:26:19] ...trap/engines/engine.py.230: loss: 0.23914 [I 2022-10-08 12:26:19] ...optimizers/trijoint.py.151: total_norm: 2.500833 [I 2022-10-08 12:26:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1750/2304 [I 2022-10-08 12:26:26] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:07:10 [I 2022-10-08 12:26:26] ...trap/engines/engine.py.229: process: 0.77929 | load: 0.00308 [I 2022-10-08 12:26:26] ...trap/engines/engine.py.230: loss: 0.18052 [I 2022-10-08 12:26:26] ...optimizers/trijoint.py.151: total_norm: 2.155153 [I 2022-10-08 12:26:34] ...trap/engines/engine.py.224: train: epoch 4 | batch 1760/2304 [I 2022-10-08 12:26:34] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:07:05 [I 2022-10-08 12:26:34] ...trap/engines/engine.py.229: process: 0.77608 | load: 0.00309 [I 2022-10-08 12:26:34] ...trap/engines/engine.py.230: loss: 0.23742 [I 2022-10-08 12:26:34] ...optimizers/trijoint.py.151: total_norm: 2.949750 [I 2022-10-08 12:26:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 1770/2304 [I 2022-10-08 12:26:42] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:06:54 [I 2022-10-08 12:26:42] ...trap/engines/engine.py.229: process: 0.76171 | load: 0.00310 [I 2022-10-08 12:26:42] ...trap/engines/engine.py.230: loss: 0.20460 [I 2022-10-08 12:26:42] ...optimizers/trijoint.py.151: total_norm: 2.571758 [I 2022-10-08 12:26:50] ...trap/engines/engine.py.224: train: epoch 4 | batch 1780/2304 [I 2022-10-08 12:26:50] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:06:48 [I 2022-10-08 12:26:50] ...trap/engines/engine.py.229: process: 0.79084 | load: 0.00312 [I 2022-10-08 12:26:50] ...trap/engines/engine.py.230: loss: 0.18533 [I 2022-10-08 12:26:50] ...optimizers/trijoint.py.151: total_norm: 1.934442 [I 2022-10-08 12:26:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 1790/2304 [I 2022-10-08 12:26:58] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:06:41 [I 2022-10-08 12:26:58] ...trap/engines/engine.py.229: process: 0.77315 | load: 0.00301 [I 2022-10-08 12:26:58] ...trap/engines/engine.py.230: loss: 0.19689 [I 2022-10-08 12:26:58] ...optimizers/trijoint.py.151: total_norm: 2.747589 [I 2022-10-08 12:27:05] ...trap/engines/engine.py.224: train: epoch 4 | batch 1800/2304 [I 2022-10-08 12:27:05] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:06:31 [I 2022-10-08 12:27:05] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00311 [I 2022-10-08 12:27:05] ...trap/engines/engine.py.230: loss: 0.14741 [I 2022-10-08 12:27:05] ...optimizers/trijoint.py.151: total_norm: 2.026379 [I 2022-10-08 12:27:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 1810/2304 [I 2022-10-08 12:27:13] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:06:23 [I 2022-10-08 12:27:13] ...trap/engines/engine.py.229: process: 0.78723 | load: 0.00302 [I 2022-10-08 12:27:13] ...trap/engines/engine.py.230: loss: 0.20396 [I 2022-10-08 12:27:13] ...optimizers/trijoint.py.151: total_norm: 2.790360 [I 2022-10-08 12:27:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1820/2304 [I 2022-10-08 12:27:21] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:06:15 [I 2022-10-08 12:27:21] ...trap/engines/engine.py.229: process: 0.77635 | load: 0.00305 [I 2022-10-08 12:27:21] ...trap/engines/engine.py.230: loss: 0.23051 [I 2022-10-08 12:27:21] ...optimizers/trijoint.py.151: total_norm: 2.651257 [I 2022-10-08 12:27:29] ...trap/engines/engine.py.224: train: epoch 4 | batch 1830/2304 [I 2022-10-08 12:27:29] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:06:08 [I 2022-10-08 12:27:29] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00313 [I 2022-10-08 12:27:29] ...trap/engines/engine.py.230: loss: 0.25793 [I 2022-10-08 12:27:29] ...optimizers/trijoint.py.151: total_norm: 2.595329 [I 2022-10-08 12:27:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 1840/2304 [I 2022-10-08 12:27:37] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:06:02 [I 2022-10-08 12:27:37] ...trap/engines/engine.py.229: process: 0.78617 | load: 0.00314 [I 2022-10-08 12:27:37] ...trap/engines/engine.py.230: loss: 0.24881 [I 2022-10-08 12:27:37] ...optimizers/trijoint.py.151: total_norm: 2.713181 [I 2022-10-08 12:27:44] ...trap/engines/engine.py.224: train: epoch 4 | batch 1850/2304 [I 2022-10-08 12:27:44] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:05:53 [I 2022-10-08 12:27:44] ...trap/engines/engine.py.229: process: 0.78167 | load: 0.00314 [I 2022-10-08 12:27:44] ...trap/engines/engine.py.230: loss: 0.21440 [I 2022-10-08 12:27:44] ...optimizers/trijoint.py.151: total_norm: 2.905974 [I 2022-10-08 12:27:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 1860/2304 [I 2022-10-08 12:27:52] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:05:45 [I 2022-10-08 12:27:52] ...trap/engines/engine.py.229: process: 0.77831 | load: 0.00310 [I 2022-10-08 12:27:52] ...trap/engines/engine.py.230: loss: 0.16417 [I 2022-10-08 12:27:52] ...optimizers/trijoint.py.151: total_norm: 2.735200 [I 2022-10-08 12:28:00] ...trap/engines/engine.py.224: train: epoch 4 | batch 1870/2304 [I 2022-10-08 12:28:00] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:05:39 [I 2022-10-08 12:28:00] ...trap/engines/engine.py.229: process: 0.78999 | load: 0.00313 [I 2022-10-08 12:28:00] ...trap/engines/engine.py.230: loss: 0.23582 [I 2022-10-08 12:28:00] ...optimizers/trijoint.py.151: total_norm: 3.038659 [I 2022-10-08 12:28:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 1880/2304 [I 2022-10-08 12:28:08] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:05:31 [I 2022-10-08 12:28:08] ...trap/engines/engine.py.229: process: 0.78357 | load: 0.00308 [I 2022-10-08 12:28:08] ...trap/engines/engine.py.230: loss: 0.23167 [I 2022-10-08 12:28:08] ...optimizers/trijoint.py.151: total_norm: 2.610502 [I 2022-10-08 12:28:15] ...trap/engines/engine.py.224: train: epoch 4 | batch 1890/2304 [I 2022-10-08 12:28:15] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:05:22 [I 2022-10-08 12:28:15] ...trap/engines/engine.py.229: process: 0.78585 | load: 0.00310 [I 2022-10-08 12:28:15] ...trap/engines/engine.py.230: loss: 0.24474 [I 2022-10-08 12:28:15] ...optimizers/trijoint.py.151: total_norm: 3.532701 [I 2022-10-08 12:28:23] ...trap/engines/engine.py.224: train: epoch 4 | batch 1900/2304 [I 2022-10-08 12:28:23] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:05:13 [I 2022-10-08 12:28:23] ...trap/engines/engine.py.229: process: 0.76561 | load: 0.00313 [I 2022-10-08 12:28:23] ...trap/engines/engine.py.230: loss: 0.29497 [I 2022-10-08 12:28:23] ...optimizers/trijoint.py.151: total_norm: 3.423089 [I 2022-10-08 12:28:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1910/2304 [I 2022-10-08 12:28:31] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:05:05 [I 2022-10-08 12:28:31] ...trap/engines/engine.py.229: process: 0.77736 | load: 0.00309 [I 2022-10-08 12:28:31] ...trap/engines/engine.py.230: loss: 0.20990 [I 2022-10-08 12:28:31] ...optimizers/trijoint.py.151: total_norm: 2.909562 [I 2022-10-08 12:28:39] ...trap/engines/engine.py.224: train: epoch 4 | batch 1920/2304 [I 2022-10-08 12:28:39] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:04:57 [I 2022-10-08 12:28:39] ...trap/engines/engine.py.229: process: 0.78415 | load: 0.00311 [I 2022-10-08 12:28:39] ...trap/engines/engine.py.230: loss: 0.25231 [I 2022-10-08 12:28:39] ...optimizers/trijoint.py.151: total_norm: 2.465094 [I 2022-10-08 12:28:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 1930/2304 [I 2022-10-08 12:28:47] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:04:52 [I 2022-10-08 12:28:47] ...trap/engines/engine.py.229: process: 0.79183 | load: 0.00311 [I 2022-10-08 12:28:47] ...trap/engines/engine.py.230: loss: 0.32708 [I 2022-10-08 12:28:47] ...optimizers/trijoint.py.151: total_norm: 3.160757 [I 2022-10-08 12:28:54] ...trap/engines/engine.py.224: train: epoch 4 | batch 1940/2304 [I 2022-10-08 12:28:54] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:04:42 [I 2022-10-08 12:28:54] ...trap/engines/engine.py.229: process: 0.77403 | load: 0.00309 [I 2022-10-08 12:28:54] ...trap/engines/engine.py.230: loss: 0.23682 [I 2022-10-08 12:28:54] ...optimizers/trijoint.py.151: total_norm: 2.996902 [I 2022-10-08 12:29:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 1950/2304 [I 2022-10-08 12:29:02] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:04:36 [I 2022-10-08 12:29:02] ...trap/engines/engine.py.229: process: 0.77838 | load: 0.00302 [I 2022-10-08 12:29:02] ...trap/engines/engine.py.230: loss: 0.11259 [I 2022-10-08 12:29:02] ...optimizers/trijoint.py.151: total_norm: 1.820314 [I 2022-10-08 12:29:10] ...trap/engines/engine.py.224: train: epoch 4 | batch 1960/2304 [I 2022-10-08 12:29:10] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:04:28 [I 2022-10-08 12:29:10] ...trap/engines/engine.py.229: process: 0.78920 | load: 0.00302 [I 2022-10-08 12:29:10] ...trap/engines/engine.py.230: loss: 0.21368 [I 2022-10-08 12:29:10] ...optimizers/trijoint.py.151: total_norm: 2.131803 [I 2022-10-08 12:29:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 1970/2304 [I 2022-10-08 12:29:18] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:04:20 [I 2022-10-08 12:29:18] ...trap/engines/engine.py.229: process: 0.78398 | load: 0.00300 [I 2022-10-08 12:29:18] ...trap/engines/engine.py.230: loss: 0.13792 [I 2022-10-08 12:29:18] ...optimizers/trijoint.py.151: total_norm: 2.099787 [I 2022-10-08 12:29:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1980/2304 [I 2022-10-08 12:29:26] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:04:10 [I 2022-10-08 12:29:26] ...trap/engines/engine.py.229: process: 0.77301 | load: 0.00302 [I 2022-10-08 12:29:26] ...trap/engines/engine.py.230: loss: 0.24221 [I 2022-10-08 12:29:26] ...optimizers/trijoint.py.151: total_norm: 2.616395 [I 2022-10-08 12:29:33] ...trap/engines/engine.py.224: train: epoch 4 | batch 1990/2304 [I 2022-10-08 12:29:33] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:04:05 [I 2022-10-08 12:29:33] ...trap/engines/engine.py.229: process: 0.77207 | load: 0.00302 [I 2022-10-08 12:29:33] ...trap/engines/engine.py.230: loss: 0.16702 [I 2022-10-08 12:29:33] ...optimizers/trijoint.py.151: total_norm: 2.637174 [I 2022-10-08 12:29:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 2000/2304 [I 2022-10-08 12:29:41] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:03:58 [I 2022-10-08 12:29:41] ...trap/engines/engine.py.229: process: 0.79069 | load: 0.00309 [I 2022-10-08 12:29:41] ...trap/engines/engine.py.230: loss: 0.35620 [I 2022-10-08 12:29:41] ...optimizers/trijoint.py.151: total_norm: 3.800341 [I 2022-10-08 12:29:49] ...trap/engines/engine.py.224: train: epoch 4 | batch 2010/2304 [I 2022-10-08 12:29:49] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:03:50 [I 2022-10-08 12:29:49] ...trap/engines/engine.py.229: process: 0.78126 | load: 0.00302 [I 2022-10-08 12:29:49] ...trap/engines/engine.py.230: loss: 0.19803 [I 2022-10-08 12:29:49] ...optimizers/trijoint.py.151: total_norm: 2.240591 [I 2022-10-08 12:29:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 2020/2304 [I 2022-10-08 12:29:57] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:03:41 [I 2022-10-08 12:29:57] ...trap/engines/engine.py.229: process: 0.77745 | load: 0.00304 [I 2022-10-08 12:29:57] ...trap/engines/engine.py.230: loss: 0.18057 [I 2022-10-08 12:29:57] ...optimizers/trijoint.py.151: total_norm: 2.309690 [I 2022-10-08 12:30:05] ...trap/engines/engine.py.224: train: epoch 4 | batch 2030/2304 [I 2022-10-08 12:30:05] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:03:32 [I 2022-10-08 12:30:05] ...trap/engines/engine.py.229: process: 0.77191 | load: 0.00303 [I 2022-10-08 12:30:05] ...trap/engines/engine.py.230: loss: 0.21865 [I 2022-10-08 12:30:05] ...optimizers/trijoint.py.151: total_norm: 2.931827 [I 2022-10-08 12:30:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 2040/2304 [I 2022-10-08 12:30:12] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:03:26 [I 2022-10-08 12:30:12] ...trap/engines/engine.py.229: process: 0.78302 | load: 0.00303 [I 2022-10-08 12:30:12] ...trap/engines/engine.py.230: loss: 0.23781 [I 2022-10-08 12:30:12] ...optimizers/trijoint.py.151: total_norm: 2.832563 [I 2022-10-08 12:30:20] ...trap/engines/engine.py.224: train: epoch 4 | batch 2050/2304 [I 2022-10-08 12:30:20] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:03:16 [I 2022-10-08 12:30:20] ...trap/engines/engine.py.229: process: 0.77586 | load: 0.00306 [I 2022-10-08 12:30:20] ...trap/engines/engine.py.230: loss: 0.23650 [I 2022-10-08 12:30:20] ...optimizers/trijoint.py.151: total_norm: 3.075394 [I 2022-10-08 12:30:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 2060/2304 [I 2022-10-08 12:30:28] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:03:09 [I 2022-10-08 12:30:28] ...trap/engines/engine.py.229: process: 0.78444 | load: 0.00301 [I 2022-10-08 12:30:28] ...trap/engines/engine.py.230: loss: 0.21215 [I 2022-10-08 12:30:28] ...optimizers/trijoint.py.151: total_norm: 2.611892 [I 2022-10-08 12:30:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 2070/2304 [I 2022-10-08 12:30:36] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:03:01 [I 2022-10-08 12:30:36] ...trap/engines/engine.py.229: process: 0.77606 | load: 0.00303 [I 2022-10-08 12:30:36] ...trap/engines/engine.py.230: loss: 0.23446 [I 2022-10-08 12:30:36] ...optimizers/trijoint.py.151: total_norm: 2.859617 [I 2022-10-08 12:30:44] ...trap/engines/engine.py.224: train: epoch 4 | batch 2080/2304 [I 2022-10-08 12:30:44] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:02:53 [I 2022-10-08 12:30:44] ...trap/engines/engine.py.229: process: 0.77073 | load: 0.00300 [I 2022-10-08 12:30:44] ...trap/engines/engine.py.230: loss: 0.25958 [I 2022-10-08 12:30:44] ...optimizers/trijoint.py.151: total_norm: 2.530616 [I 2022-10-08 12:30:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 2090/2304 [I 2022-10-08 12:30:51] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:02:46 [I 2022-10-08 12:30:51] ...trap/engines/engine.py.229: process: 0.79024 | load: 0.00303 [I 2022-10-08 12:30:51] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-08 12:30:51] ...optimizers/trijoint.py.151: total_norm: 2.160159 [I 2022-10-08 12:30:59] ...trap/engines/engine.py.224: train: epoch 4 | batch 2100/2304 [I 2022-10-08 12:30:59] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:02:38 [I 2022-10-08 12:30:59] ...trap/engines/engine.py.229: process: 0.76736 | load: 0.00304 [I 2022-10-08 12:30:59] ...trap/engines/engine.py.230: loss: 0.25744 [I 2022-10-08 12:30:59] ...optimizers/trijoint.py.151: total_norm: 2.826569 [I 2022-10-08 12:31:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 2110/2304 [I 2022-10-08 12:31:07] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:02:30 [I 2022-10-08 12:31:07] ...trap/engines/engine.py.229: process: 0.76366 | load: 0.00303 [I 2022-10-08 12:31:07] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-08 12:31:07] ...optimizers/trijoint.py.151: total_norm: 2.044119 [I 2022-10-08 12:31:15] ...trap/engines/engine.py.224: train: epoch 4 | batch 2120/2304 [I 2022-10-08 12:31:15] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:02:22 [I 2022-10-08 12:31:15] ...trap/engines/engine.py.229: process: 0.77319 | load: 0.00307 [I 2022-10-08 12:31:15] ...trap/engines/engine.py.230: loss: 0.20170 [I 2022-10-08 12:31:15] ...optimizers/trijoint.py.151: total_norm: 2.958373 [I 2022-10-08 12:31:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 2130/2304 [I 2022-10-08 12:31:22] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:02:14 [I 2022-10-08 12:31:22] ...trap/engines/engine.py.229: process: 0.75856 | load: 0.00310 [I 2022-10-08 12:31:22] ...trap/engines/engine.py.230: loss: 0.26422 [I 2022-10-08 12:31:22] ...optimizers/trijoint.py.151: total_norm: 2.568917 [I 2022-10-08 12:31:30] ...trap/engines/engine.py.224: train: epoch 4 | batch 2140/2304 [I 2022-10-08 12:31:30] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:02:07 [I 2022-10-08 12:31:30] ...trap/engines/engine.py.229: process: 0.78557 | load: 0.00301 [I 2022-10-08 12:31:30] ...trap/engines/engine.py.230: loss: 0.23510 [I 2022-10-08 12:31:30] ...optimizers/trijoint.py.151: total_norm: 3.130632 [I 2022-10-08 12:31:38] ...trap/engines/engine.py.224: train: epoch 4 | batch 2150/2304 [I 2022-10-08 12:31:38] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:01:59 [I 2022-10-08 12:31:38] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00301 [I 2022-10-08 12:31:38] ...trap/engines/engine.py.230: loss: 0.29477 [I 2022-10-08 12:31:38] ...optimizers/trijoint.py.151: total_norm: 3.586136 [I 2022-10-08 12:31:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 2160/2304 [I 2022-10-08 12:31:46] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:01:51 [I 2022-10-08 12:31:46] ...trap/engines/engine.py.229: process: 0.77568 | load: 0.00302 [I 2022-10-08 12:31:46] ...trap/engines/engine.py.230: loss: 0.30054 [I 2022-10-08 12:31:46] ...optimizers/trijoint.py.151: total_norm: 3.603213 [I 2022-10-08 12:31:54] ...trap/engines/engine.py.224: train: epoch 4 | batch 2170/2304 [I 2022-10-08 12:31:54] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:01:44 [I 2022-10-08 12:31:54] ...trap/engines/engine.py.229: process: 0.77287 | load: 0.00303 [I 2022-10-08 12:31:54] ...trap/engines/engine.py.230: loss: 0.16916 [I 2022-10-08 12:31:54] ...optimizers/trijoint.py.151: total_norm: 2.231745 [I 2022-10-08 12:32:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 2180/2304 [I 2022-10-08 12:32:01] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:01:36 [I 2022-10-08 12:32:01] ...trap/engines/engine.py.229: process: 0.78824 | load: 0.00301 [I 2022-10-08 12:32:01] ...trap/engines/engine.py.230: loss: 0.17887 [I 2022-10-08 12:32:01] ...optimizers/trijoint.py.151: total_norm: 2.572557 [I 2022-10-08 12:32:09] ...trap/engines/engine.py.224: train: epoch 4 | batch 2190/2304 [I 2022-10-08 12:32:09] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:01:28 [I 2022-10-08 12:32:09] ...trap/engines/engine.py.229: process: 0.78016 | load: 0.00300 [I 2022-10-08 12:32:09] ...trap/engines/engine.py.230: loss: 0.33530 [I 2022-10-08 12:32:09] ...optimizers/trijoint.py.151: total_norm: 3.260699 [I 2022-10-08 12:32:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 2200/2304 [I 2022-10-08 12:32:17] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:01:20 [I 2022-10-08 12:32:17] ...trap/engines/engine.py.229: process: 0.77737 | load: 0.00300 [I 2022-10-08 12:32:17] ...trap/engines/engine.py.230: loss: 0.23836 [I 2022-10-08 12:32:17] ...optimizers/trijoint.py.151: total_norm: 2.688039 [I 2022-10-08 12:32:25] ...trap/engines/engine.py.224: train: epoch 4 | batch 2210/2304 [I 2022-10-08 12:32:25] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:01:12 [I 2022-10-08 12:32:25] ...trap/engines/engine.py.229: process: 0.76392 | load: 0.00301 [I 2022-10-08 12:32:25] ...trap/engines/engine.py.230: loss: 0.22418 [I 2022-10-08 12:32:25] ...optimizers/trijoint.py.151: total_norm: 3.057750 [I 2022-10-08 12:32:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 2220/2304 [I 2022-10-08 12:32:32] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:01:04 [I 2022-10-08 12:32:32] ...trap/engines/engine.py.229: process: 0.78246 | load: 0.00306 [I 2022-10-08 12:32:32] ...trap/engines/engine.py.230: loss: 0.17094 [I 2022-10-08 12:32:32] ...optimizers/trijoint.py.151: total_norm: 2.565685 [I 2022-10-08 12:32:40] ...trap/engines/engine.py.224: train: epoch 4 | batch 2230/2304 [I 2022-10-08 12:32:40] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:00:57 [I 2022-10-08 12:32:40] ...trap/engines/engine.py.229: process: 0.78600 | load: 0.00301 [I 2022-10-08 12:32:40] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-08 12:32:40] ...optimizers/trijoint.py.151: total_norm: 2.527685 [I 2022-10-08 12:32:48] ...trap/engines/engine.py.224: train: epoch 4 | batch 2240/2304 [I 2022-10-08 12:32:48] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:00:49 [I 2022-10-08 12:32:48] ...trap/engines/engine.py.229: process: 0.78671 | load: 0.00306 [I 2022-10-08 12:32:48] ...trap/engines/engine.py.230: loss: 0.18708 [I 2022-10-08 12:32:48] ...optimizers/trijoint.py.151: total_norm: 2.375610 [I 2022-10-08 12:32:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 2250/2304 [I 2022-10-08 12:32:56] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:00:41 [I 2022-10-08 12:32:56] ...trap/engines/engine.py.229: process: 0.76642 | load: 0.00300 [I 2022-10-08 12:32:56] ...trap/engines/engine.py.230: loss: 0.23489 [I 2022-10-08 12:32:56] ...optimizers/trijoint.py.151: total_norm: 2.890544 [I 2022-10-08 12:33:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 2260/2304 [I 2022-10-08 12:33:03] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:00:34 [I 2022-10-08 12:33:03] ...trap/engines/engine.py.229: process: 0.77974 | load: 0.00301 [I 2022-10-08 12:33:03] ...trap/engines/engine.py.230: loss: 0.21653 [I 2022-10-08 12:33:03] ...optimizers/trijoint.py.151: total_norm: 2.633915 [I 2022-10-08 12:33:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 2270/2304 [I 2022-10-08 12:33:11] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:00:26 [I 2022-10-08 12:33:11] ...trap/engines/engine.py.229: process: 0.78117 | load: 0.00302 [I 2022-10-08 12:33:11] ...trap/engines/engine.py.230: loss: 0.14634 [I 2022-10-08 12:33:11] ...optimizers/trijoint.py.151: total_norm: 2.370804 [I 2022-10-08 12:33:19] ...trap/engines/engine.py.224: train: epoch 4 | batch 2280/2304 [I 2022-10-08 12:33:19] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:00:18 [I 2022-10-08 12:33:19] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00305 [I 2022-10-08 12:33:19] ...trap/engines/engine.py.230: loss: 0.21996 [I 2022-10-08 12:33:19] ...optimizers/trijoint.py.151: total_norm: 2.779216 [I 2022-10-08 12:33:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 2290/2304 [I 2022-10-08 12:33:27] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:00:10 [I 2022-10-08 12:33:27] ...trap/engines/engine.py.229: process: 0.78752 | load: 0.00301 [I 2022-10-08 12:33:27] ...trap/engines/engine.py.230: loss: 0.33596 [I 2022-10-08 12:33:27] ...optimizers/trijoint.py.151: total_norm: 3.395633 [I 2022-10-08 12:33:35] ...trap/engines/engine.py.224: train: epoch 4 | batch 2300/2304 [I 2022-10-08 12:33:35] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:00:03 [I 2022-10-08 12:33:35] ...trap/engines/engine.py.229: process: 0.74982 | load: 0.00017 [I 2022-10-08 12:33:35] ...trap/engines/engine.py.230: loss: 0.14086 [I 2022-10-08 12:33:35] ...optimizers/trijoint.py.151: total_norm: 2.611467 [I 2022-10-08 12:33:38] ...trap/engines/engine.py.224: train: epoch 4 | batch 2304/2304 [I 2022-10-08 12:33:38] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:00:00 [I 2022-10-08 12:33:38] ...trap/engines/engine.py.229: process: 0.76251 | load: 0.00017 [I 2022-10-08 12:33:38] ...trap/engines/engine.py.230: loss: 0.20566 [I 2022-10-08 12:33:38] ...optimizers/trijoint.py.151: total_norm: 2.809415 [S 2022-10-08 12:33:39] ...trap/engines/engine.py.129: train_epoch.epoch: 4 [S 2022-10-08 12:33:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.23024132051656665 [I 2022-10-08 12:33:39] ...s/criterions/triplet.py.84: margin = 0.07500000000000001 [I 2022-10-08 12:33:39] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 4 [I 2022-10-08 12:33:39] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 12:33:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 12:33:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 12:33:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 12:33:41] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 12:33:42] ...trap/engines/engine.py.321: eval: epoch 4 | batch 0/510 [I 2022-10-08 12:33:42] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:58 [I 2022-10-08 12:33:42] ...trap/engines/engine.py.326: process: 2.58595 | load: 2.05600 [I 2022-10-08 12:33:47] ...trap/engines/engine.py.321: eval: epoch 4 | batch 10/510 [I 2022-10-08 12:33:47] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:20 [I 2022-10-08 12:33:47] ...trap/engines/engine.py.326: process: 0.51047 | load: 0.00019 [I 2022-10-08 12:33:53] ...trap/engines/engine.py.321: eval: epoch 4 | batch 20/510 [I 2022-10-08 12:33:53] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:58 [I 2022-10-08 12:33:53] ...trap/engines/engine.py.326: process: 0.64881 | load: 0.00020 [I 2022-10-08 12:33:59] ...trap/engines/engine.py.321: eval: epoch 4 | batch 30/510 [I 2022-10-08 12:33:59] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:59 [I 2022-10-08 12:33:59] ...trap/engines/engine.py.326: process: 0.79211 | load: 0.00018 [I 2022-10-08 12:34:06] ...trap/engines/engine.py.321: eval: epoch 4 | batch 40/510 [I 2022-10-08 12:34:06] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:17 [I 2022-10-08 12:34:06] ...trap/engines/engine.py.326: process: 0.49207 | load: 0.00019 [I 2022-10-08 12:34:10] ...trap/engines/engine.py.321: eval: epoch 4 | batch 50/510 [I 2022-10-08 12:34:10] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:36 [I 2022-10-08 12:34:10] ...trap/engines/engine.py.326: process: 0.43734 | load: 0.00018 [I 2022-10-08 12:34:15] ...trap/engines/engine.py.321: eval: epoch 4 | batch 60/510 [I 2022-10-08 12:34:15] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:53 [I 2022-10-08 12:34:15] ...trap/engines/engine.py.326: process: 0.53773 | load: 0.00022 [I 2022-10-08 12:34:21] ...trap/engines/engine.py.321: eval: epoch 4 | batch 70/510 [I 2022-10-08 12:34:21] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:43 [I 2022-10-08 12:34:21] ...trap/engines/engine.py.326: process: 0.42580 | load: 0.00018 [I 2022-10-08 12:34:26] ...trap/engines/engine.py.321: eval: epoch 4 | batch 80/510 [I 2022-10-08 12:34:26] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:39 [I 2022-10-08 12:34:26] ...trap/engines/engine.py.326: process: 0.47274 | load: 0.00018 [I 2022-10-08 12:34:30] ...trap/engines/engine.py.321: eval: epoch 4 | batch 90/510 [I 2022-10-08 12:34:30] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:05 [I 2022-10-08 12:34:30] ...trap/engines/engine.py.326: process: 0.42456 | load: 0.00021 [I 2022-10-08 12:34:34] ...trap/engines/engine.py.321: eval: epoch 4 | batch 100/510 [I 2022-10-08 12:34:34] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:49 [I 2022-10-08 12:34:34] ...trap/engines/engine.py.326: process: 0.30659 | load: 0.00018 [I 2022-10-08 12:34:37] ...trap/engines/engine.py.321: eval: epoch 4 | batch 110/510 [I 2022-10-08 12:34:37] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:11 [I 2022-10-08 12:34:37] ...trap/engines/engine.py.326: process: 0.31853 | load: 0.00020 [I 2022-10-08 12:34:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 120/510 [I 2022-10-08 12:34:41] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-08 12:34:41] ...trap/engines/engine.py.326: process: 0.31416 | load: 0.00019 [I 2022-10-08 12:34:44] ...trap/engines/engine.py.321: eval: epoch 4 | batch 130/510 [I 2022-10-08 12:34:44] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-08 12:34:44] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00021 [I 2022-10-08 12:34:47] ...trap/engines/engine.py.321: eval: epoch 4 | batch 140/510 [I 2022-10-08 12:34:47] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:57 [I 2022-10-08 12:34:47] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00020 [I 2022-10-08 12:34:50] ...trap/engines/engine.py.321: eval: epoch 4 | batch 150/510 [I 2022-10-08 12:34:50] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:52 [I 2022-10-08 12:34:50] ...trap/engines/engine.py.326: process: 0.31249 | load: 0.00019 [I 2022-10-08 12:34:53] ...trap/engines/engine.py.321: eval: epoch 4 | batch 160/510 [I 2022-10-08 12:34:53] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:50 [I 2022-10-08 12:34:53] ...trap/engines/engine.py.326: process: 0.31735 | load: 0.00020 [I 2022-10-08 12:34:56] ...trap/engines/engine.py.321: eval: epoch 4 | batch 170/510 [I 2022-10-08 12:34:56] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-08 12:34:56] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00019 [I 2022-10-08 12:35:00] ...trap/engines/engine.py.321: eval: epoch 4 | batch 180/510 [I 2022-10-08 12:35:00] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:44 [I 2022-10-08 12:35:00] ...trap/engines/engine.py.326: process: 0.31753 | load: 0.00019 [I 2022-10-08 12:35:03] ...trap/engines/engine.py.321: eval: epoch 4 | batch 190/510 [I 2022-10-08 12:35:03] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:40 [I 2022-10-08 12:35:03] ...trap/engines/engine.py.326: process: 0.31205 | load: 0.00019 [I 2022-10-08 12:35:06] ...trap/engines/engine.py.321: eval: epoch 4 | batch 200/510 [I 2022-10-08 12:35:06] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-08 12:35:06] ...trap/engines/engine.py.326: process: 0.32097 | load: 0.00020 [I 2022-10-08 12:35:09] ...trap/engines/engine.py.321: eval: epoch 4 | batch 210/510 [I 2022-10-08 12:35:09] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-08 12:35:09] ...trap/engines/engine.py.326: process: 0.31753 | load: 0.00021 [I 2022-10-08 12:35:12] ...trap/engines/engine.py.321: eval: epoch 4 | batch 220/510 [I 2022-10-08 12:35:12] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:31 [I 2022-10-08 12:35:12] ...trap/engines/engine.py.326: process: 0.31701 | load: 0.00019 [I 2022-10-08 12:35:16] ...trap/engines/engine.py.321: eval: epoch 4 | batch 230/510 [I 2022-10-08 12:35:16] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-08 12:35:16] ...trap/engines/engine.py.326: process: 0.32436 | load: 0.00019 [I 2022-10-08 12:35:19] ...trap/engines/engine.py.321: eval: epoch 4 | batch 240/510 [I 2022-10-08 12:35:19] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-08 12:35:19] ...trap/engines/engine.py.326: process: 0.31541 | load: 0.00021 [I 2022-10-08 12:35:22] ...trap/engines/engine.py.321: eval: epoch 4 | batch 250/510 [I 2022-10-08 12:35:22] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:21 [I 2022-10-08 12:35:22] ...trap/engines/engine.py.326: process: 0.31055 | load: 0.00020 [I 2022-10-08 12:35:25] ...trap/engines/engine.py.321: eval: epoch 4 | batch 260/510 [I 2022-10-08 12:35:25] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:19 [I 2022-10-08 12:35:25] ...trap/engines/engine.py.326: process: 0.31151 | load: 0.00020 [I 2022-10-08 12:35:28] ...trap/engines/engine.py.321: eval: epoch 4 | batch 270/510 [I 2022-10-08 12:35:28] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-08 12:35:28] ...trap/engines/engine.py.326: process: 0.31573 | load: 0.00020 [I 2022-10-08 12:35:31] ...trap/engines/engine.py.321: eval: epoch 4 | batch 280/510 [I 2022-10-08 12:35:31] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-08 12:35:31] ...trap/engines/engine.py.326: process: 0.31759 | load: 0.00021 [I 2022-10-08 12:35:35] ...trap/engines/engine.py.321: eval: epoch 4 | batch 290/510 [I 2022-10-08 12:35:35] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-08 12:35:35] ...trap/engines/engine.py.326: process: 0.32109 | load: 0.00020 [I 2022-10-08 12:35:38] ...trap/engines/engine.py.321: eval: epoch 4 | batch 300/510 [I 2022-10-08 12:35:38] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:05 [I 2022-10-08 12:35:38] ...trap/engines/engine.py.326: process: 0.31019 | load: 0.00022 [I 2022-10-08 12:35:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 310/510 [I 2022-10-08 12:35:41] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:02 [I 2022-10-08 12:35:41] ...trap/engines/engine.py.326: process: 0.31583 | load: 0.00020 [I 2022-10-08 12:35:44] ...trap/engines/engine.py.321: eval: epoch 4 | batch 320/510 [I 2022-10-08 12:35:44] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-08 12:35:44] ...trap/engines/engine.py.326: process: 0.31743 | load: 0.00020 [I 2022-10-08 12:35:47] ...trap/engines/engine.py.321: eval: epoch 4 | batch 330/510 [I 2022-10-08 12:35:47] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-08 12:35:47] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00020 [I 2022-10-08 12:35:50] ...trap/engines/engine.py.321: eval: epoch 4 | batch 340/510 [I 2022-10-08 12:35:50] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:53 [I 2022-10-08 12:35:50] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00021 [I 2022-10-08 12:35:54] ...trap/engines/engine.py.321: eval: epoch 4 | batch 350/510 [I 2022-10-08 12:35:54] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:50 [I 2022-10-08 12:35:54] ...trap/engines/engine.py.326: process: 0.31382 | load: 0.00019 [I 2022-10-08 12:35:57] ...trap/engines/engine.py.321: eval: epoch 4 | batch 360/510 [I 2022-10-08 12:35:57] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-08 12:35:57] ...trap/engines/engine.py.326: process: 0.31644 | load: 0.00020 [I 2022-10-08 12:36:00] ...trap/engines/engine.py.321: eval: epoch 4 | batch 370/510 [I 2022-10-08 12:36:00] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-08 12:36:00] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00019 [I 2022-10-08 12:36:03] ...trap/engines/engine.py.321: eval: epoch 4 | batch 380/510 [I 2022-10-08 12:36:03] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-08 12:36:03] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00020 [I 2022-10-08 12:36:06] ...trap/engines/engine.py.321: eval: epoch 4 | batch 390/510 [I 2022-10-08 12:36:06] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-08 12:36:06] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00021 [I 2022-10-08 12:36:09] ...trap/engines/engine.py.321: eval: epoch 4 | batch 400/510 [I 2022-10-08 12:36:10] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-08 12:36:10] ...trap/engines/engine.py.326: process: 0.31683 | load: 0.00020 [I 2022-10-08 12:36:13] ...trap/engines/engine.py.321: eval: epoch 4 | batch 410/510 [I 2022-10-08 12:36:13] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-08 12:36:13] ...trap/engines/engine.py.326: process: 0.32130 | load: 0.00020 [I 2022-10-08 12:36:16] ...trap/engines/engine.py.321: eval: epoch 4 | batch 420/510 [I 2022-10-08 12:36:16] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-08 12:36:16] ...trap/engines/engine.py.326: process: 0.31506 | load: 0.00022 [I 2022-10-08 12:36:19] ...trap/engines/engine.py.321: eval: epoch 4 | batch 430/510 [I 2022-10-08 12:36:19] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-08 12:36:19] ...trap/engines/engine.py.326: process: 0.31462 | load: 0.00020 [I 2022-10-08 12:36:22] ...trap/engines/engine.py.321: eval: epoch 4 | batch 440/510 [I 2022-10-08 12:36:22] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-08 12:36:22] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00020 [I 2022-10-08 12:36:25] ...trap/engines/engine.py.321: eval: epoch 4 | batch 450/510 [I 2022-10-08 12:36:25] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-08 12:36:25] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00020 [I 2022-10-08 12:36:29] ...trap/engines/engine.py.321: eval: epoch 4 | batch 460/510 [I 2022-10-08 12:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-08 12:36:29] ...trap/engines/engine.py.326: process: 0.31502 | load: 0.00021 [I 2022-10-08 12:36:32] ...trap/engines/engine.py.321: eval: epoch 4 | batch 470/510 [I 2022-10-08 12:36:32] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-08 12:36:32] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-08 12:36:35] ...trap/engines/engine.py.321: eval: epoch 4 | batch 480/510 [I 2022-10-08 12:36:35] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-08 12:36:35] ...trap/engines/engine.py.326: process: 0.30737 | load: 0.00020 [I 2022-10-08 12:36:38] ...trap/engines/engine.py.321: eval: epoch 4 | batch 490/510 [I 2022-10-08 12:36:38] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-08 12:36:38] ...trap/engines/engine.py.326: process: 0.31803 | load: 0.00020 [I 2022-10-08 12:36:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 500/510 [I 2022-10-08 12:36:41] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-08 12:36:41] ...trap/engines/engine.py.326: process: 0.31577 | load: 0.00020 [I 2022-10-08 12:36:44] ...trap/engines/engine.py.321: eval: epoch 4 | batch 510/510 [I 2022-10-08 12:36:44] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-08 12:36:44] ...trap/engines/engine.py.326: process: 0.31583 | load: 0.00017 [S 2022-10-08 12:36:45] ...trap/engines/engine.py.132: eval_epoch.epoch: 4 [I 2022-10-08 12:36:45] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.542800 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.824300 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.892700 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.554400 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.832400 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.894100 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014323 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012490 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010826 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014023 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012682 [S 2022-10-08 12:37:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010212 [I 2022-10-08 12:37:17] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-08 12:37:17] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 12:37:17] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 12:37:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 12:37:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 12:37:17] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 12:37:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:37:24] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:37:24] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 12:37:24] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 12:37:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:37:32] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:37:32] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 12:37:32] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 12:37:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 12:37:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 12:37:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 5 [I 2022-10-08 12:37:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 12:37:42] ...trap/engines/engine.py.224: train: epoch 5 | batch 0/2304 [I 2022-10-08 12:37:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:03 [I 2022-10-08 12:37:42] ...trap/engines/engine.py.229: process: 2.86623 | load: 2.02743 [I 2022-10-08 12:37:42] ...trap/engines/engine.py.230: loss: 0.22530 [I 2022-10-08 12:37:42] ...optimizers/trijoint.py.151: total_norm: 2.671227 [I 2022-10-08 12:37:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 10/2304 [I 2022-10-08 12:37:50] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:20 [I 2022-10-08 12:37:50] ...trap/engines/engine.py.229: process: 0.77737 | load: 0.00677 [I 2022-10-08 12:37:50] ...trap/engines/engine.py.230: loss: 0.24135 [I 2022-10-08 12:37:50] ...optimizers/trijoint.py.151: total_norm: 4.113965 [I 2022-10-08 12:37:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 20/2304 [I 2022-10-08 12:37:57] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:39 [I 2022-10-08 12:37:57] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00451 [I 2022-10-08 12:37:57] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-08 12:37:57] ...optimizers/trijoint.py.151: total_norm: 1.957740 [I 2022-10-08 12:38:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 30/2304 [I 2022-10-08 12:38:05] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:31 [I 2022-10-08 12:38:05] ...trap/engines/engine.py.229: process: 0.76908 | load: 0.00383 [I 2022-10-08 12:38:05] ...trap/engines/engine.py.230: loss: 0.14296 [I 2022-10-08 12:38:05] ...optimizers/trijoint.py.151: total_norm: 2.366372 [I 2022-10-08 12:38:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 40/2304 [I 2022-10-08 12:38:13] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:13 [I 2022-10-08 12:38:13] ...trap/engines/engine.py.229: process: 0.77026 | load: 0.00413 [I 2022-10-08 12:38:13] ...trap/engines/engine.py.230: loss: 0.19022 [I 2022-10-08 12:38:13] ...optimizers/trijoint.py.151: total_norm: 2.835231 [I 2022-10-08 12:38:21] ...trap/engines/engine.py.224: train: epoch 5 | batch 50/2304 [I 2022-10-08 12:38:21] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:06 [I 2022-10-08 12:38:21] ...trap/engines/engine.py.229: process: 0.77879 | load: 0.00365 [I 2022-10-08 12:38:21] ...trap/engines/engine.py.230: loss: 0.25952 [I 2022-10-08 12:38:21] ...optimizers/trijoint.py.151: total_norm: 3.495554 [I 2022-10-08 12:38:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 60/2304 [I 2022-10-08 12:38:29] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:01 [I 2022-10-08 12:38:29] ...trap/engines/engine.py.229: process: 0.76223 | load: 0.00350 [I 2022-10-08 12:38:29] ...trap/engines/engine.py.230: loss: 0.16166 [I 2022-10-08 12:38:29] ...optimizers/trijoint.py.151: total_norm: 2.562221 [I 2022-10-08 12:38:36] ...trap/engines/engine.py.224: train: epoch 5 | batch 70/2304 [I 2022-10-08 12:38:36] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:06 [I 2022-10-08 12:38:36] ...trap/engines/engine.py.229: process: 0.78698 | load: 0.00416 [I 2022-10-08 12:38:36] ...trap/engines/engine.py.230: loss: 0.24978 [I 2022-10-08 12:38:36] ...optimizers/trijoint.py.151: total_norm: 3.335265 [I 2022-10-08 12:38:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 80/2304 [I 2022-10-08 12:38:44] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:48 [I 2022-10-08 12:38:44] ...trap/engines/engine.py.229: process: 0.77391 | load: 0.00375 [I 2022-10-08 12:38:44] ...trap/engines/engine.py.230: loss: 0.19750 [I 2022-10-08 12:38:44] ...optimizers/trijoint.py.151: total_norm: 2.889393 [I 2022-10-08 12:38:52] ...trap/engines/engine.py.224: train: epoch 5 | batch 90/2304 [I 2022-10-08 12:38:52] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:45 [I 2022-10-08 12:38:52] ...trap/engines/engine.py.229: process: 0.78430 | load: 0.00322 [I 2022-10-08 12:38:52] ...trap/engines/engine.py.230: loss: 0.14711 [I 2022-10-08 12:38:52] ...optimizers/trijoint.py.151: total_norm: 2.564296 [I 2022-10-08 12:39:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 100/2304 [I 2022-10-08 12:39:00] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:34 [I 2022-10-08 12:39:00] ...trap/engines/engine.py.229: process: 0.76647 | load: 0.00332 [I 2022-10-08 12:39:00] ...trap/engines/engine.py.230: loss: 0.23145 [I 2022-10-08 12:39:00] ...optimizers/trijoint.py.151: total_norm: 2.247599 [I 2022-10-08 12:39:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 110/2304 [I 2022-10-08 12:39:08] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:31 [I 2022-10-08 12:39:08] ...trap/engines/engine.py.229: process: 0.77915 | load: 0.00343 [I 2022-10-08 12:39:08] ...trap/engines/engine.py.230: loss: 0.21712 [I 2022-10-08 12:39:08] ...optimizers/trijoint.py.151: total_norm: 2.442679 [I 2022-10-08 12:39:16] ...trap/engines/engine.py.224: train: epoch 5 | batch 120/2304 [I 2022-10-08 12:39:16] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:21 [I 2022-10-08 12:39:16] ...trap/engines/engine.py.229: process: 0.77060 | load: 0.00300 [I 2022-10-08 12:39:16] ...trap/engines/engine.py.230: loss: 0.22593 [I 2022-10-08 12:39:16] ...optimizers/trijoint.py.151: total_norm: 2.368139 [I 2022-10-08 12:39:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 130/2304 [I 2022-10-08 12:39:23] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:11 [I 2022-10-08 12:39:23] ...trap/engines/engine.py.229: process: 0.77057 | load: 0.00303 [I 2022-10-08 12:39:23] ...trap/engines/engine.py.230: loss: 0.23379 [I 2022-10-08 12:39:23] ...optimizers/trijoint.py.151: total_norm: 3.499411 [I 2022-10-08 12:39:31] ...trap/engines/engine.py.224: train: epoch 5 | batch 140/2304 [I 2022-10-08 12:39:31] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:03 [I 2022-10-08 12:39:31] ...trap/engines/engine.py.229: process: 0.77250 | load: 0.00352 [I 2022-10-08 12:39:31] ...trap/engines/engine.py.230: loss: 0.30043 [I 2022-10-08 12:39:31] ...optimizers/trijoint.py.151: total_norm: 3.004389 [I 2022-10-08 12:39:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 150/2304 [I 2022-10-08 12:39:39] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:27:42 [I 2022-10-08 12:39:39] ...trap/engines/engine.py.229: process: 0.76026 | load: 0.00321 [I 2022-10-08 12:39:39] ...trap/engines/engine.py.230: loss: 0.17226 [I 2022-10-08 12:39:39] ...optimizers/trijoint.py.151: total_norm: 2.640265 [I 2022-10-08 12:39:47] ...trap/engines/engine.py.224: train: epoch 5 | batch 160/2304 [I 2022-10-08 12:39:47] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:41 [I 2022-10-08 12:39:47] ...trap/engines/engine.py.229: process: 0.78460 | load: 0.00314 [I 2022-10-08 12:39:47] ...trap/engines/engine.py.230: loss: 0.25169 [I 2022-10-08 12:39:47] ...optimizers/trijoint.py.151: total_norm: 3.665554 [I 2022-10-08 12:39:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 170/2304 [I 2022-10-08 12:39:54] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:42 [I 2022-10-08 12:39:54] ...trap/engines/engine.py.229: process: 0.78885 | load: 0.00350 [I 2022-10-08 12:39:54] ...trap/engines/engine.py.230: loss: 0.24631 [I 2022-10-08 12:39:54] ...optimizers/trijoint.py.151: total_norm: 4.014965 [I 2022-10-08 12:40:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 180/2304 [I 2022-10-08 12:40:02] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:34 [I 2022-10-08 12:40:02] ...trap/engines/engine.py.229: process: 0.78656 | load: 0.00334 [I 2022-10-08 12:40:02] ...trap/engines/engine.py.230: loss: 0.30344 [I 2022-10-08 12:40:02] ...optimizers/trijoint.py.151: total_norm: 3.478058 [I 2022-10-08 12:40:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 190/2304 [I 2022-10-08 12:40:10] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:22 [I 2022-10-08 12:40:10] ...trap/engines/engine.py.229: process: 0.77100 | load: 0.00336 [I 2022-10-08 12:40:10] ...trap/engines/engine.py.230: loss: 0.20483 [I 2022-10-08 12:40:10] ...optimizers/trijoint.py.151: total_norm: 2.711932 [I 2022-10-08 12:40:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 200/2304 [I 2022-10-08 12:40:18] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:27:25 [I 2022-10-08 12:40:18] ...trap/engines/engine.py.229: process: 0.77880 | load: 0.00314 [I 2022-10-08 12:40:18] ...trap/engines/engine.py.230: loss: 0.21294 [I 2022-10-08 12:40:18] ...optimizers/trijoint.py.151: total_norm: 1.970619 [I 2022-10-08 12:40:26] ...trap/engines/engine.py.224: train: epoch 5 | batch 210/2304 [I 2022-10-08 12:40:26] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:07 [I 2022-10-08 12:40:26] ...trap/engines/engine.py.229: process: 0.77296 | load: 0.00302 [I 2022-10-08 12:40:26] ...trap/engines/engine.py.230: loss: 0.23802 [I 2022-10-08 12:40:26] ...optimizers/trijoint.py.151: total_norm: 3.565999 [I 2022-10-08 12:40:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 220/2304 [I 2022-10-08 12:40:33] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:26:52 [I 2022-10-08 12:40:33] ...trap/engines/engine.py.229: process: 0.77146 | load: 0.00305 [I 2022-10-08 12:40:33] ...trap/engines/engine.py.230: loss: 0.22840 [I 2022-10-08 12:40:33] ...optimizers/trijoint.py.151: total_norm: 2.383914 [I 2022-10-08 12:40:41] ...trap/engines/engine.py.224: train: epoch 5 | batch 230/2304 [I 2022-10-08 12:40:41] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:27:02 [I 2022-10-08 12:40:41] ...trap/engines/engine.py.229: process: 0.79457 | load: 0.00300 [I 2022-10-08 12:40:41] ...trap/engines/engine.py.230: loss: 0.22412 [I 2022-10-08 12:40:41] ...optimizers/trijoint.py.151: total_norm: 2.276320 [I 2022-10-08 12:40:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 240/2304 [I 2022-10-08 12:40:49] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:55 [I 2022-10-08 12:40:49] ...trap/engines/engine.py.229: process: 0.78298 | load: 0.00315 [I 2022-10-08 12:40:49] ...trap/engines/engine.py.230: loss: 0.26673 [I 2022-10-08 12:40:49] ...optimizers/trijoint.py.151: total_norm: 3.264824 [I 2022-10-08 12:40:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 250/2304 [I 2022-10-08 12:40:57] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:26:58 [I 2022-10-08 12:40:57] ...trap/engines/engine.py.229: process: 0.80478 | load: 0.00307 [I 2022-10-08 12:40:57] ...trap/engines/engine.py.230: loss: 0.18716 [I 2022-10-08 12:40:57] ...optimizers/trijoint.py.151: total_norm: 1.821985 [I 2022-10-08 12:41:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 260/2304 [I 2022-10-08 12:41:05] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:35 [I 2022-10-08 12:41:05] ...trap/engines/engine.py.229: process: 0.77922 | load: 0.00344 [I 2022-10-08 12:41:05] ...trap/engines/engine.py.230: loss: 0.23356 [I 2022-10-08 12:41:05] ...optimizers/trijoint.py.151: total_norm: 2.740673 [I 2022-10-08 12:41:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 270/2304 [I 2022-10-08 12:41:13] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:32 [I 2022-10-08 12:41:13] ...trap/engines/engine.py.229: process: 0.79385 | load: 0.00301 [I 2022-10-08 12:41:13] ...trap/engines/engine.py.230: loss: 0.27270 [I 2022-10-08 12:41:13] ...optimizers/trijoint.py.151: total_norm: 2.990166 [I 2022-10-08 12:41:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 280/2304 [I 2022-10-08 12:41:20] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:21 [I 2022-10-08 12:41:20] ...trap/engines/engine.py.229: process: 0.78398 | load: 0.00305 [I 2022-10-08 12:41:20] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-08 12:41:20] ...optimizers/trijoint.py.151: total_norm: 2.678103 [I 2022-10-08 12:41:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 290/2304 [I 2022-10-08 12:41:28] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:13 [I 2022-10-08 12:41:28] ...trap/engines/engine.py.229: process: 0.78227 | load: 0.00318 [I 2022-10-08 12:41:28] ...trap/engines/engine.py.230: loss: 0.34662 [I 2022-10-08 12:41:28] ...optimizers/trijoint.py.151: total_norm: 3.288749 [I 2022-10-08 12:41:36] ...trap/engines/engine.py.224: train: epoch 5 | batch 300/2304 [I 2022-10-08 12:41:36] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:56 [I 2022-10-08 12:41:36] ...trap/engines/engine.py.229: process: 0.76743 | load: 0.00301 [I 2022-10-08 12:41:36] ...trap/engines/engine.py.230: loss: 0.21609 [I 2022-10-08 12:41:36] ...optimizers/trijoint.py.151: total_norm: 3.034010 [I 2022-10-08 12:41:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 310/2304 [I 2022-10-08 12:41:44] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:26:03 [I 2022-10-08 12:41:44] ...trap/engines/engine.py.229: process: 0.80155 | load: 0.00311 [I 2022-10-08 12:41:44] ...trap/engines/engine.py.230: loss: 0.22859 [I 2022-10-08 12:41:44] ...optimizers/trijoint.py.151: total_norm: 2.618604 [I 2022-10-08 12:41:52] ...trap/engines/engine.py.224: train: epoch 5 | batch 320/2304 [I 2022-10-08 12:41:52] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:52 [I 2022-10-08 12:41:52] ...trap/engines/engine.py.229: process: 0.78946 | load: 0.00317 [I 2022-10-08 12:41:52] ...trap/engines/engine.py.230: loss: 0.24144 [I 2022-10-08 12:41:52] ...optimizers/trijoint.py.151: total_norm: 3.173930 [I 2022-10-08 12:42:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 330/2304 [I 2022-10-08 12:42:00] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:48 [I 2022-10-08 12:42:00] ...trap/engines/engine.py.229: process: 0.78823 | load: 0.00301 [I 2022-10-08 12:42:00] ...trap/engines/engine.py.230: loss: 0.24470 [I 2022-10-08 12:42:00] ...optimizers/trijoint.py.151: total_norm: 2.997555 [I 2022-10-08 12:42:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 340/2304 [I 2022-10-08 12:42:07] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:38 [I 2022-10-08 12:42:07] ...trap/engines/engine.py.229: process: 0.77907 | load: 0.00311 [I 2022-10-08 12:42:07] ...trap/engines/engine.py.230: loss: 0.24983 [I 2022-10-08 12:42:07] ...optimizers/trijoint.py.151: total_norm: 3.284218 [I 2022-10-08 12:42:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 350/2304 [I 2022-10-08 12:42:15] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:22 [I 2022-10-08 12:42:15] ...trap/engines/engine.py.229: process: 0.76251 | load: 0.00302 [I 2022-10-08 12:42:15] ...trap/engines/engine.py.230: loss: 0.17536 [I 2022-10-08 12:42:15] ...optimizers/trijoint.py.151: total_norm: 2.359762 [I 2022-10-08 12:42:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 360/2304 [I 2022-10-08 12:42:23] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:19 [I 2022-10-08 12:42:23] ...trap/engines/engine.py.229: process: 0.78570 | load: 0.00301 [I 2022-10-08 12:42:23] ...trap/engines/engine.py.230: loss: 0.19788 [I 2022-10-08 12:42:23] ...optimizers/trijoint.py.151: total_norm: 2.555454 [I 2022-10-08 12:42:31] ...trap/engines/engine.py.224: train: epoch 5 | batch 370/2304 [I 2022-10-08 12:42:31] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:01 [I 2022-10-08 12:42:31] ...trap/engines/engine.py.229: process: 0.77516 | load: 0.00302 [I 2022-10-08 12:42:31] ...trap/engines/engine.py.230: loss: 0.23531 [I 2022-10-08 12:42:31] ...optimizers/trijoint.py.151: total_norm: 2.986695 [I 2022-10-08 12:42:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 380/2304 [I 2022-10-08 12:42:39] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:24:58 [I 2022-10-08 12:42:39] ...trap/engines/engine.py.229: process: 0.77894 | load: 0.00306 [I 2022-10-08 12:42:39] ...trap/engines/engine.py.230: loss: 0.34050 [I 2022-10-08 12:42:39] ...optimizers/trijoint.py.151: total_norm: 3.673212 [I 2022-10-08 12:42:46] ...trap/engines/engine.py.224: train: epoch 5 | batch 390/2304 [I 2022-10-08 12:42:46] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:53 [I 2022-10-08 12:42:46] ...trap/engines/engine.py.229: process: 0.78314 | load: 0.00312 [I 2022-10-08 12:42:46] ...trap/engines/engine.py.230: loss: 0.14339 [I 2022-10-08 12:42:46] ...optimizers/trijoint.py.151: total_norm: 2.636955 [I 2022-10-08 12:42:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 400/2304 [I 2022-10-08 12:42:54] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:48 [I 2022-10-08 12:42:54] ...trap/engines/engine.py.229: process: 0.77248 | load: 0.00303 [I 2022-10-08 12:42:54] ...trap/engines/engine.py.230: loss: 0.19176 [I 2022-10-08 12:42:54] ...optimizers/trijoint.py.151: total_norm: 2.121392 [I 2022-10-08 12:43:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 410/2304 [I 2022-10-08 12:43:02] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:32 [I 2022-10-08 12:43:02] ...trap/engines/engine.py.229: process: 0.76277 | load: 0.00314 [I 2022-10-08 12:43:02] ...trap/engines/engine.py.230: loss: 0.19962 [I 2022-10-08 12:43:02] ...optimizers/trijoint.py.151: total_norm: 2.111500 [I 2022-10-08 12:43:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 420/2304 [I 2022-10-08 12:43:10] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:16 [I 2022-10-08 12:43:10] ...trap/engines/engine.py.229: process: 0.76679 | load: 0.00307 [I 2022-10-08 12:43:10] ...trap/engines/engine.py.230: loss: 0.13931 [I 2022-10-08 12:43:10] ...optimizers/trijoint.py.151: total_norm: 2.064411 [I 2022-10-08 12:43:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 430/2304 [I 2022-10-08 12:43:18] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:09 [I 2022-10-08 12:43:18] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00302 [I 2022-10-08 12:43:18] ...trap/engines/engine.py.230: loss: 0.18824 [I 2022-10-08 12:43:18] ...optimizers/trijoint.py.151: total_norm: 3.115375 [I 2022-10-08 12:43:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 440/2304 [I 2022-10-08 12:43:25] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:25 [I 2022-10-08 12:43:25] ...trap/engines/engine.py.229: process: 0.78377 | load: 0.00302 [I 2022-10-08 12:43:25] ...trap/engines/engine.py.230: loss: 0.35783 [I 2022-10-08 12:43:25] ...optimizers/trijoint.py.151: total_norm: 3.472118 [I 2022-10-08 12:43:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 450/2304 [I 2022-10-08 12:43:33] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:03 [I 2022-10-08 12:43:33] ...trap/engines/engine.py.229: process: 0.76456 | load: 0.00307 [I 2022-10-08 12:43:33] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-08 12:43:33] ...optimizers/trijoint.py.151: total_norm: 3.155631 [I 2022-10-08 12:43:41] ...trap/engines/engine.py.224: train: epoch 5 | batch 460/2304 [I 2022-10-08 12:43:41] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:49 [I 2022-10-08 12:43:41] ...trap/engines/engine.py.229: process: 0.77298 | load: 0.00306 [I 2022-10-08 12:43:41] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-08 12:43:41] ...optimizers/trijoint.py.151: total_norm: 2.508540 [I 2022-10-08 12:43:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 470/2304 [I 2022-10-08 12:43:49] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:50 [I 2022-10-08 12:43:49] ...trap/engines/engine.py.229: process: 0.78597 | load: 0.00304 [I 2022-10-08 12:43:49] ...trap/engines/engine.py.230: loss: 0.23149 [I 2022-10-08 12:43:49] ...optimizers/trijoint.py.151: total_norm: 3.312228 [I 2022-10-08 12:43:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 480/2304 [I 2022-10-08 12:43:57] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:47 [I 2022-10-08 12:43:57] ...trap/engines/engine.py.229: process: 0.78825 | load: 0.00306 [I 2022-10-08 12:43:57] ...trap/engines/engine.py.230: loss: 0.19909 [I 2022-10-08 12:43:57] ...optimizers/trijoint.py.151: total_norm: 2.564840 [I 2022-10-08 12:44:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 490/2304 [I 2022-10-08 12:44:04] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:33 [I 2022-10-08 12:44:04] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00306 [I 2022-10-08 12:44:04] ...trap/engines/engine.py.230: loss: 0.31112 [I 2022-10-08 12:44:04] ...optimizers/trijoint.py.151: total_norm: 3.893385 [I 2022-10-08 12:44:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 500/2304 [I 2022-10-08 12:44:12] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:23 [I 2022-10-08 12:44:12] ...trap/engines/engine.py.229: process: 0.76149 | load: 0.00302 [I 2022-10-08 12:44:12] ...trap/engines/engine.py.230: loss: 0.20859 [I 2022-10-08 12:44:12] ...optimizers/trijoint.py.151: total_norm: 2.520528 [I 2022-10-08 12:44:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 510/2304 [I 2022-10-08 12:44:20] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:15 [I 2022-10-08 12:44:20] ...trap/engines/engine.py.229: process: 0.77480 | load: 0.00303 [I 2022-10-08 12:44:20] ...trap/engines/engine.py.230: loss: 0.19181 [I 2022-10-08 12:44:20] ...optimizers/trijoint.py.151: total_norm: 2.306001 [I 2022-10-08 12:44:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 520/2304 [I 2022-10-08 12:44:28] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:23:09 [I 2022-10-08 12:44:28] ...trap/engines/engine.py.229: process: 0.78150 | load: 0.00321 [I 2022-10-08 12:44:28] ...trap/engines/engine.py.230: loss: 0.23157 [I 2022-10-08 12:44:28] ...optimizers/trijoint.py.151: total_norm: 2.930657 [I 2022-10-08 12:44:36] ...trap/engines/engine.py.224: train: epoch 5 | batch 530/2304 [I 2022-10-08 12:44:36] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:07 [I 2022-10-08 12:44:36] ...trap/engines/engine.py.229: process: 0.78840 | load: 0.00315 [I 2022-10-08 12:44:36] ...trap/engines/engine.py.230: loss: 0.22200 [I 2022-10-08 12:44:36] ...optimizers/trijoint.py.151: total_norm: 2.717790 [I 2022-10-08 12:44:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 540/2304 [I 2022-10-08 12:44:43] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:46 [I 2022-10-08 12:44:43] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00312 [I 2022-10-08 12:44:43] ...trap/engines/engine.py.230: loss: 0.20096 [I 2022-10-08 12:44:43] ...optimizers/trijoint.py.151: total_norm: 2.362073 [I 2022-10-08 12:44:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 550/2304 [I 2022-10-08 12:44:51] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:44 [I 2022-10-08 12:44:51] ...trap/engines/engine.py.229: process: 0.78170 | load: 0.00304 [I 2022-10-08 12:44:51] ...trap/engines/engine.py.230: loss: 0.21123 [I 2022-10-08 12:44:51] ...optimizers/trijoint.py.151: total_norm: 2.832279 [I 2022-10-08 12:44:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 560/2304 [I 2022-10-08 12:44:59] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:31 [I 2022-10-08 12:44:59] ...trap/engines/engine.py.229: process: 0.77648 | load: 0.00330 [I 2022-10-08 12:44:59] ...trap/engines/engine.py.230: loss: 0.23915 [I 2022-10-08 12:44:59] ...optimizers/trijoint.py.151: total_norm: 2.953496 [I 2022-10-08 12:45:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 570/2304 [I 2022-10-08 12:45:07] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:22:26 [I 2022-10-08 12:45:07] ...trap/engines/engine.py.229: process: 0.77589 | load: 0.00300 [I 2022-10-08 12:45:07] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-08 12:45:07] ...optimizers/trijoint.py.151: total_norm: 2.827883 [I 2022-10-08 12:45:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 580/2304 [I 2022-10-08 12:45:15] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:29 [I 2022-10-08 12:45:15] ...trap/engines/engine.py.229: process: 0.78955 | load: 0.00309 [I 2022-10-08 12:45:15] ...trap/engines/engine.py.230: loss: 0.16556 [I 2022-10-08 12:45:15] ...optimizers/trijoint.py.151: total_norm: 2.257684 [I 2022-10-08 12:45:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 590/2304 [I 2022-10-08 12:45:23] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:23 [I 2022-10-08 12:45:23] ...trap/engines/engine.py.229: process: 0.78461 | load: 0.00305 [I 2022-10-08 12:45:23] ...trap/engines/engine.py.230: loss: 0.32742 [I 2022-10-08 12:45:23] ...optimizers/trijoint.py.151: total_norm: 4.124273 [I 2022-10-08 12:45:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 600/2304 [I 2022-10-08 12:45:30] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:07 [I 2022-10-08 12:45:30] ...trap/engines/engine.py.229: process: 0.78390 | load: 0.00307 [I 2022-10-08 12:45:30] ...trap/engines/engine.py.230: loss: 0.17798 [I 2022-10-08 12:45:30] ...optimizers/trijoint.py.151: total_norm: 3.454788 [I 2022-10-08 12:45:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 610/2304 [I 2022-10-08 12:45:38] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:03 [I 2022-10-08 12:45:38] ...trap/engines/engine.py.229: process: 0.78310 | load: 0.00304 [I 2022-10-08 12:45:38] ...trap/engines/engine.py.230: loss: 0.23015 [I 2022-10-08 12:45:38] ...optimizers/trijoint.py.151: total_norm: 3.137596 [I 2022-10-08 12:45:46] ...trap/engines/engine.py.224: train: epoch 5 | batch 620/2304 [I 2022-10-08 12:45:46] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:21:53 [I 2022-10-08 12:45:46] ...trap/engines/engine.py.229: process: 0.77685 | load: 0.00300 [I 2022-10-08 12:45:46] ...trap/engines/engine.py.230: loss: 0.15566 [I 2022-10-08 12:45:46] ...optimizers/trijoint.py.151: total_norm: 2.626368 [I 2022-10-08 12:45:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 630/2304 [I 2022-10-08 12:45:54] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:21:42 [I 2022-10-08 12:45:54] ...trap/engines/engine.py.229: process: 0.77112 | load: 0.00305 [I 2022-10-08 12:45:54] ...trap/engines/engine.py.230: loss: 0.20881 [I 2022-10-08 12:45:54] ...optimizers/trijoint.py.151: total_norm: 3.028324 [I 2022-10-08 12:46:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 640/2304 [I 2022-10-08 12:46:02] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:39 [I 2022-10-08 12:46:02] ...trap/engines/engine.py.229: process: 0.78385 | load: 0.00302 [I 2022-10-08 12:46:02] ...trap/engines/engine.py.230: loss: 0.19015 [I 2022-10-08 12:46:02] ...optimizers/trijoint.py.151: total_norm: 3.790973 [I 2022-10-08 12:46:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 650/2304 [I 2022-10-08 12:46:09] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:27 [I 2022-10-08 12:46:09] ...trap/engines/engine.py.229: process: 0.77272 | load: 0.00306 [I 2022-10-08 12:46:09] ...trap/engines/engine.py.230: loss: 0.19862 [I 2022-10-08 12:46:09] ...optimizers/trijoint.py.151: total_norm: 2.459107 [I 2022-10-08 12:46:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 660/2304 [I 2022-10-08 12:46:17] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:28 [I 2022-10-08 12:46:17] ...trap/engines/engine.py.229: process: 0.77890 | load: 0.00300 [I 2022-10-08 12:46:17] ...trap/engines/engine.py.230: loss: 0.22282 [I 2022-10-08 12:46:17] ...optimizers/trijoint.py.151: total_norm: 2.964847 [I 2022-10-08 12:46:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 670/2304 [I 2022-10-08 12:46:25] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:09 [I 2022-10-08 12:46:25] ...trap/engines/engine.py.229: process: 0.77705 | load: 0.00303 [I 2022-10-08 12:46:25] ...trap/engines/engine.py.230: loss: 0.14359 [I 2022-10-08 12:46:25] ...optimizers/trijoint.py.151: total_norm: 2.090106 [I 2022-10-08 12:46:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 680/2304 [I 2022-10-08 12:46:33] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:21:06 [I 2022-10-08 12:46:33] ...trap/engines/engine.py.229: process: 0.78030 | load: 0.00313 [I 2022-10-08 12:46:33] ...trap/engines/engine.py.230: loss: 0.26626 [I 2022-10-08 12:46:33] ...optimizers/trijoint.py.151: total_norm: 3.264348 [I 2022-10-08 12:46:41] ...trap/engines/engine.py.224: train: epoch 5 | batch 690/2304 [I 2022-10-08 12:46:41] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:57 [I 2022-10-08 12:46:41] ...trap/engines/engine.py.229: process: 0.78165 | load: 0.00302 [I 2022-10-08 12:46:41] ...trap/engines/engine.py.230: loss: 0.23026 [I 2022-10-08 12:46:41] ...optimizers/trijoint.py.151: total_norm: 2.343145 [I 2022-10-08 12:46:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 700/2304 [I 2022-10-08 12:46:48] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:48 [I 2022-10-08 12:46:48] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00307 [I 2022-10-08 12:46:48] ...trap/engines/engine.py.230: loss: 0.22281 [I 2022-10-08 12:46:48] ...optimizers/trijoint.py.151: total_norm: 2.836034 [I 2022-10-08 12:46:56] ...trap/engines/engine.py.224: train: epoch 5 | batch 710/2304 [I 2022-10-08 12:46:56] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:40 [I 2022-10-08 12:46:56] ...trap/engines/engine.py.229: process: 0.78358 | load: 0.00302 [I 2022-10-08 12:46:56] ...trap/engines/engine.py.230: loss: 0.29891 [I 2022-10-08 12:46:56] ...optimizers/trijoint.py.151: total_norm: 2.538180 [I 2022-10-08 12:47:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 720/2304 [I 2022-10-08 12:47:04] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:20:31 [I 2022-10-08 12:47:04] ...trap/engines/engine.py.229: process: 0.77378 | load: 0.00301 [I 2022-10-08 12:47:04] ...trap/engines/engine.py.230: loss: 0.21525 [I 2022-10-08 12:47:04] ...optimizers/trijoint.py.151: total_norm: 3.343621 [I 2022-10-08 12:47:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 730/2304 [I 2022-10-08 12:47:12] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:20:34 [I 2022-10-08 12:47:12] ...trap/engines/engine.py.229: process: 0.79268 | load: 0.00304 [I 2022-10-08 12:47:12] ...trap/engines/engine.py.230: loss: 0.23399 [I 2022-10-08 12:47:12] ...optimizers/trijoint.py.151: total_norm: 2.131521 [I 2022-10-08 12:47:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 740/2304 [I 2022-10-08 12:47:20] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:28 [I 2022-10-08 12:47:20] ...trap/engines/engine.py.229: process: 0.79859 | load: 0.00307 [I 2022-10-08 12:47:20] ...trap/engines/engine.py.230: loss: 0.20919 [I 2022-10-08 12:47:20] ...optimizers/trijoint.py.151: total_norm: 2.670405 [I 2022-10-08 12:47:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 750/2304 [I 2022-10-08 12:47:28] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:23 [I 2022-10-08 12:47:28] ...trap/engines/engine.py.229: process: 0.79677 | load: 0.00305 [I 2022-10-08 12:47:28] ...trap/engines/engine.py.230: loss: 0.22690 [I 2022-10-08 12:47:28] ...optimizers/trijoint.py.151: total_norm: 2.591585 [I 2022-10-08 12:47:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 760/2304 [I 2022-10-08 12:47:35] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:11 [I 2022-10-08 12:47:35] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00306 [I 2022-10-08 12:47:35] ...trap/engines/engine.py.230: loss: 0.29008 [I 2022-10-08 12:47:35] ...optimizers/trijoint.py.151: total_norm: 3.262422 [I 2022-10-08 12:47:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 770/2304 [I 2022-10-08 12:47:43] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:20:00 [I 2022-10-08 12:47:43] ...trap/engines/engine.py.229: process: 0.79179 | load: 0.00304 [I 2022-10-08 12:47:43] ...trap/engines/engine.py.230: loss: 0.28643 [I 2022-10-08 12:47:43] ...optimizers/trijoint.py.151: total_norm: 3.571414 [I 2022-10-08 12:48:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 780/2304 [I 2022-10-08 12:48:09] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:43:57 [I 2022-10-08 12:48:09] ...trap/engines/engine.py.229: process: 0.78974 | load: 0.00315 [I 2022-10-08 12:48:09] ...trap/engines/engine.py.230: loss: 0.22512 [I 2022-10-08 12:48:09] ...optimizers/trijoint.py.151: total_norm: 2.599899 [I 2022-10-08 12:48:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 790/2304 [I 2022-10-08 12:48:17] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:22:11 [I 2022-10-08 12:48:17] ...trap/engines/engine.py.229: process: 0.78111 | load: 0.00302 [I 2022-10-08 12:48:17] ...trap/engines/engine.py.230: loss: 0.25188 [I 2022-10-08 12:48:17] ...optimizers/trijoint.py.151: total_norm: 2.677094 [I 2022-10-08 12:48:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 800/2304 [I 2022-10-08 12:48:25] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:19:48 [I 2022-10-08 12:48:25] ...trap/engines/engine.py.229: process: 0.78255 | load: 0.00309 [I 2022-10-08 12:48:25] ...trap/engines/engine.py.230: loss: 0.18615 [I 2022-10-08 12:48:25] ...optimizers/trijoint.py.151: total_norm: 2.655995 [I 2022-10-08 12:48:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 810/2304 [I 2022-10-08 12:48:33] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:19:26 [I 2022-10-08 12:48:33] ...trap/engines/engine.py.229: process: 0.77772 | load: 0.00303 [I 2022-10-08 12:48:33] ...trap/engines/engine.py.230: loss: 0.20714 [I 2022-10-08 12:48:33] ...optimizers/trijoint.py.151: total_norm: 2.789879 [I 2022-10-08 12:48:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 820/2304 [I 2022-10-08 12:48:40] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:19:22 [I 2022-10-08 12:48:40] ...trap/engines/engine.py.229: process: 0.77889 | load: 0.00310 [I 2022-10-08 12:48:40] ...trap/engines/engine.py.230: loss: 0.23194 [I 2022-10-08 12:48:40] ...optimizers/trijoint.py.151: total_norm: 2.641763 [I 2022-10-08 12:48:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 830/2304 [I 2022-10-08 12:48:48] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:19:06 [I 2022-10-08 12:48:48] ...trap/engines/engine.py.229: process: 0.77182 | load: 0.00308 [I 2022-10-08 12:48:48] ...trap/engines/engine.py.230: loss: 0.25346 [I 2022-10-08 12:48:48] ...optimizers/trijoint.py.151: total_norm: 2.754303 [I 2022-10-08 12:48:56] ...trap/engines/engine.py.224: train: epoch 5 | batch 840/2304 [I 2022-10-08 12:48:56] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:18:58 [I 2022-10-08 12:48:56] ...trap/engines/engine.py.229: process: 0.77136 | load: 0.00308 [I 2022-10-08 12:48:56] ...trap/engines/engine.py.230: loss: 0.20817 [I 2022-10-08 12:48:56] ...optimizers/trijoint.py.151: total_norm: 3.508815 [I 2022-10-08 12:49:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 850/2304 [I 2022-10-08 12:49:04] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:18:51 [I 2022-10-08 12:49:04] ...trap/engines/engine.py.229: process: 0.77827 | load: 0.00301 [I 2022-10-08 12:49:04] ...trap/engines/engine.py.230: loss: 0.13315 [I 2022-10-08 12:49:04] ...optimizers/trijoint.py.151: total_norm: 2.212380 [I 2022-10-08 12:49:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 860/2304 [I 2022-10-08 12:49:12] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:18:35 [I 2022-10-08 12:49:12] ...trap/engines/engine.py.229: process: 0.75989 | load: 0.00304 [I 2022-10-08 12:49:12] ...trap/engines/engine.py.230: loss: 0.27435 [I 2022-10-08 12:49:12] ...optimizers/trijoint.py.151: total_norm: 3.295726 [I 2022-10-08 12:49:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 870/2304 [I 2022-10-08 12:49:19] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:18:30 [I 2022-10-08 12:49:19] ...trap/engines/engine.py.229: process: 0.76811 | load: 0.00303 [I 2022-10-08 12:49:19] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-08 12:49:19] ...optimizers/trijoint.py.151: total_norm: 2.120409 [I 2022-10-08 12:49:27] ...trap/engines/engine.py.224: train: epoch 5 | batch 880/2304 [I 2022-10-08 12:49:27] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:18:22 [I 2022-10-08 12:49:27] ...trap/engines/engine.py.229: process: 0.76350 | load: 0.00303 [I 2022-10-08 12:49:27] ...trap/engines/engine.py.230: loss: 0.15162 [I 2022-10-08 12:49:27] ...optimizers/trijoint.py.151: total_norm: 2.081342 [I 2022-10-08 12:49:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 890/2304 [I 2022-10-08 12:49:35] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:18:19 [I 2022-10-08 12:49:35] ...trap/engines/engine.py.229: process: 0.77780 | load: 0.00304 [I 2022-10-08 12:49:35] ...trap/engines/engine.py.230: loss: 0.22674 [I 2022-10-08 12:49:35] ...optimizers/trijoint.py.151: total_norm: 3.135168 [I 2022-10-08 12:49:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 900/2304 [I 2022-10-08 12:49:43] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:18:08 [I 2022-10-08 12:49:43] ...trap/engines/engine.py.229: process: 0.78236 | load: 0.00302 [I 2022-10-08 12:49:43] ...trap/engines/engine.py.230: loss: 0.26834 [I 2022-10-08 12:49:43] ...optimizers/trijoint.py.151: total_norm: 3.316657 [I 2022-10-08 12:49:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 910/2304 [I 2022-10-08 12:49:51] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:18:06 [I 2022-10-08 12:49:51] ...trap/engines/engine.py.229: process: 0.77938 | load: 0.00307 [I 2022-10-08 12:49:51] ...trap/engines/engine.py.230: loss: 0.21586 [I 2022-10-08 12:49:51] ...optimizers/trijoint.py.151: total_norm: 4.020045 [I 2022-10-08 12:49:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 920/2304 [I 2022-10-08 12:49:58] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:17:54 [I 2022-10-08 12:49:58] ...trap/engines/engine.py.229: process: 0.77793 | load: 0.00302 [I 2022-10-08 12:49:58] ...trap/engines/engine.py.230: loss: 0.36242 [I 2022-10-08 12:49:58] ...optimizers/trijoint.py.151: total_norm: 3.909040 [I 2022-10-08 12:50:06] ...trap/engines/engine.py.224: train: epoch 5 | batch 930/2304 [I 2022-10-08 12:50:06] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:17:50 [I 2022-10-08 12:50:06] ...trap/engines/engine.py.229: process: 0.78985 | load: 0.00302 [I 2022-10-08 12:50:06] ...trap/engines/engine.py.230: loss: 0.25789 [I 2022-10-08 12:50:06] ...optimizers/trijoint.py.151: total_norm: 2.998186 [I 2022-10-08 12:50:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 940/2304 [I 2022-10-08 12:50:14] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:17:39 [I 2022-10-08 12:50:14] ...trap/engines/engine.py.229: process: 0.77513 | load: 0.00301 [I 2022-10-08 12:50:14] ...trap/engines/engine.py.230: loss: 0.25237 [I 2022-10-08 12:50:14] ...optimizers/trijoint.py.151: total_norm: 2.866049 [I 2022-10-08 12:50:22] ...trap/engines/engine.py.224: train: epoch 5 | batch 950/2304 [I 2022-10-08 12:50:22] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:17:32 [I 2022-10-08 12:50:22] ...trap/engines/engine.py.229: process: 0.78086 | load: 0.00303 [I 2022-10-08 12:50:22] ...trap/engines/engine.py.230: loss: 0.21119 [I 2022-10-08 12:50:22] ...optimizers/trijoint.py.151: total_norm: 2.497594 [I 2022-10-08 12:50:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 960/2304 [I 2022-10-08 12:50:29] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:17:25 [I 2022-10-08 12:50:29] ...trap/engines/engine.py.229: process: 0.77907 | load: 0.00303 [I 2022-10-08 12:50:29] ...trap/engines/engine.py.230: loss: 0.17654 [I 2022-10-08 12:50:29] ...optimizers/trijoint.py.151: total_norm: 2.173943 [I 2022-10-08 12:50:37] ...trap/engines/engine.py.224: train: epoch 5 | batch 970/2304 [I 2022-10-08 12:50:37] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:17:16 [I 2022-10-08 12:50:37] ...trap/engines/engine.py.229: process: 0.77417 | load: 0.00306 [I 2022-10-08 12:50:37] ...trap/engines/engine.py.230: loss: 0.20896 [I 2022-10-08 12:50:37] ...optimizers/trijoint.py.151: total_norm: 2.174211 [I 2022-10-08 12:50:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 980/2304 [I 2022-10-08 12:50:45] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:17:11 [I 2022-10-08 12:50:45] ...trap/engines/engine.py.229: process: 0.78553 | load: 0.00303 [I 2022-10-08 12:50:45] ...trap/engines/engine.py.230: loss: 0.26825 [I 2022-10-08 12:50:45] ...optimizers/trijoint.py.151: total_norm: 3.673158 [I 2022-10-08 12:50:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 990/2304 [I 2022-10-08 12:50:53] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:17:03 [I 2022-10-08 12:50:53] ...trap/engines/engine.py.229: process: 0.77790 | load: 0.00305 [I 2022-10-08 12:50:53] ...trap/engines/engine.py.230: loss: 0.26194 [I 2022-10-08 12:50:53] ...optimizers/trijoint.py.151: total_norm: 3.317278 [I 2022-10-08 12:51:01] ...trap/engines/engine.py.224: train: epoch 5 | batch 1000/2304 [I 2022-10-08 12:51:01] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:16:46 [I 2022-10-08 12:51:01] ...trap/engines/engine.py.229: process: 0.74683 | load: 0.00308 [I 2022-10-08 12:51:01] ...trap/engines/engine.py.230: loss: 0.15410 [I 2022-10-08 12:51:01] ...optimizers/trijoint.py.151: total_norm: 2.402045 [I 2022-10-08 12:51:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1010/2304 [I 2022-10-08 12:51:08] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:16:51 [I 2022-10-08 12:51:08] ...trap/engines/engine.py.229: process: 0.77361 | load: 0.00309 [I 2022-10-08 12:51:08] ...trap/engines/engine.py.230: loss: 0.30869 [I 2022-10-08 12:51:08] ...optimizers/trijoint.py.151: total_norm: 3.284842 [I 2022-10-08 12:51:16] ...trap/engines/engine.py.224: train: epoch 5 | batch 1020/2304 [I 2022-10-08 12:51:16] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:16:40 [I 2022-10-08 12:51:16] ...trap/engines/engine.py.229: process: 0.77251 | load: 0.00315 [I 2022-10-08 12:51:16] ...trap/engines/engine.py.230: loss: 0.25530 [I 2022-10-08 12:51:16] ...optimizers/trijoint.py.151: total_norm: 2.536236 [I 2022-10-08 12:51:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 1030/2304 [I 2022-10-08 12:51:24] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:16:27 [I 2022-10-08 12:51:24] ...trap/engines/engine.py.229: process: 0.77824 | load: 0.00310 [I 2022-10-08 12:51:24] ...trap/engines/engine.py.230: loss: 0.16421 [I 2022-10-08 12:51:24] ...optimizers/trijoint.py.151: total_norm: 2.662330 [I 2022-10-08 12:51:32] ...trap/engines/engine.py.224: train: epoch 5 | batch 1040/2304 [I 2022-10-08 12:51:32] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:16:25 [I 2022-10-08 12:51:32] ...trap/engines/engine.py.229: process: 0.78195 | load: 0.00312 [I 2022-10-08 12:51:32] ...trap/engines/engine.py.230: loss: 0.33624 [I 2022-10-08 12:51:32] ...optimizers/trijoint.py.151: total_norm: 2.975612 [I 2022-10-08 12:51:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 1050/2304 [I 2022-10-08 12:51:40] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:16:21 [I 2022-10-08 12:51:40] ...trap/engines/engine.py.229: process: 0.78242 | load: 0.00322 [I 2022-10-08 12:51:40] ...trap/engines/engine.py.230: loss: 0.27454 [I 2022-10-08 12:51:40] ...optimizers/trijoint.py.151: total_norm: 3.300472 [I 2022-10-08 12:51:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 1060/2304 [I 2022-10-08 12:51:48] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:16:13 [I 2022-10-08 12:51:48] ...trap/engines/engine.py.229: process: 0.78542 | load: 0.00315 [I 2022-10-08 12:51:48] ...trap/engines/engine.py.230: loss: 0.18107 [I 2022-10-08 12:51:48] ...optimizers/trijoint.py.151: total_norm: 2.881251 [I 2022-10-08 12:51:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 1070/2304 [I 2022-10-08 12:51:55] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:15:59 [I 2022-10-08 12:51:55] ...trap/engines/engine.py.229: process: 0.78317 | load: 0.00304 [I 2022-10-08 12:51:55] ...trap/engines/engine.py.230: loss: 0.22381 [I 2022-10-08 12:51:55] ...optimizers/trijoint.py.151: total_norm: 3.009418 [I 2022-10-08 12:52:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 1080/2304 [I 2022-10-08 12:52:03] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:15:50 [I 2022-10-08 12:52:03] ...trap/engines/engine.py.229: process: 0.76921 | load: 0.00305 [I 2022-10-08 12:52:03] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-08 12:52:03] ...optimizers/trijoint.py.151: total_norm: 2.275046 [I 2022-10-08 12:52:11] ...trap/engines/engine.py.224: train: epoch 5 | batch 1090/2304 [I 2022-10-08 12:52:11] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:15:45 [I 2022-10-08 12:52:11] ...trap/engines/engine.py.229: process: 0.78451 | load: 0.00305 [I 2022-10-08 12:52:11] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-08 12:52:11] ...optimizers/trijoint.py.151: total_norm: 2.390248 [I 2022-10-08 12:52:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 1100/2304 [I 2022-10-08 12:52:19] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:15:33 [I 2022-10-08 12:52:19] ...trap/engines/engine.py.229: process: 0.76998 | load: 0.00304 [I 2022-10-08 12:52:19] ...trap/engines/engine.py.230: loss: 0.23016 [I 2022-10-08 12:52:19] ...optimizers/trijoint.py.151: total_norm: 3.217638 [I 2022-10-08 12:52:26] ...trap/engines/engine.py.224: train: epoch 5 | batch 1110/2304 [I 2022-10-08 12:52:26] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:15:27 [I 2022-10-08 12:52:26] ...trap/engines/engine.py.229: process: 0.78345 | load: 0.00302 [I 2022-10-08 12:52:26] ...trap/engines/engine.py.230: loss: 0.29969 [I 2022-10-08 12:52:26] ...optimizers/trijoint.py.151: total_norm: 3.066190 [I 2022-10-08 12:52:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 1120/2304 [I 2022-10-08 12:52:34] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:15:23 [I 2022-10-08 12:52:34] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00302 [I 2022-10-08 12:52:34] ...trap/engines/engine.py.230: loss: 0.21801 [I 2022-10-08 12:52:34] ...optimizers/trijoint.py.151: total_norm: 2.564335 [I 2022-10-08 12:52:42] ...trap/engines/engine.py.224: train: epoch 5 | batch 1130/2304 [I 2022-10-08 12:52:42] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:15:14 [I 2022-10-08 12:52:42] ...trap/engines/engine.py.229: process: 0.77403 | load: 0.00304 [I 2022-10-08 12:52:42] ...trap/engines/engine.py.230: loss: 0.23858 [I 2022-10-08 12:52:42] ...optimizers/trijoint.py.151: total_norm: 3.332987 [I 2022-10-08 12:52:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 1140/2304 [I 2022-10-08 12:52:50] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:15:01 [I 2022-10-08 12:52:50] ...trap/engines/engine.py.229: process: 0.76732 | load: 0.00301 [I 2022-10-08 12:52:50] ...trap/engines/engine.py.230: loss: 0.26331 [I 2022-10-08 12:52:50] ...optimizers/trijoint.py.151: total_norm: 3.987789 [I 2022-10-08 12:52:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 1150/2304 [I 2022-10-08 12:52:58] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:15:01 [I 2022-10-08 12:52:58] ...trap/engines/engine.py.229: process: 0.76807 | load: 0.00301 [I 2022-10-08 12:52:58] ...trap/engines/engine.py.230: loss: 0.22306 [I 2022-10-08 12:52:58] ...optimizers/trijoint.py.151: total_norm: 2.423374 [I 2022-10-08 12:53:06] ...trap/engines/engine.py.224: train: epoch 5 | batch 1160/2304 [I 2022-10-08 12:53:06] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:14:54 [I 2022-10-08 12:53:06] ...trap/engines/engine.py.229: process: 0.79075 | load: 0.00306 [I 2022-10-08 12:53:06] ...trap/engines/engine.py.230: loss: 0.24738 [I 2022-10-08 12:53:06] ...optimizers/trijoint.py.151: total_norm: 2.760808 [I 2022-10-08 12:53:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 1170/2304 [I 2022-10-08 12:53:13] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:14:46 [I 2022-10-08 12:53:13] ...trap/engines/engine.py.229: process: 0.77837 | load: 0.00299 [I 2022-10-08 12:53:13] ...trap/engines/engine.py.230: loss: 0.26509 [I 2022-10-08 12:53:13] ...optimizers/trijoint.py.151: total_norm: 2.945807 [I 2022-10-08 12:53:21] ...trap/engines/engine.py.224: train: epoch 5 | batch 1180/2304 [I 2022-10-08 12:53:21] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:14:37 [I 2022-10-08 12:53:21] ...trap/engines/engine.py.229: process: 0.77696 | load: 0.00301 [I 2022-10-08 12:53:21] ...trap/engines/engine.py.230: loss: 0.17946 [I 2022-10-08 12:53:21] ...optimizers/trijoint.py.151: total_norm: 2.234175 [I 2022-10-08 12:53:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 1190/2304 [I 2022-10-08 12:53:29] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:14:32 [I 2022-10-08 12:53:29] ...trap/engines/engine.py.229: process: 0.77998 | load: 0.00304 [I 2022-10-08 12:53:29] ...trap/engines/engine.py.230: loss: 0.22785 [I 2022-10-08 12:53:29] ...optimizers/trijoint.py.151: total_norm: 3.456439 [I 2022-10-08 12:53:37] ...trap/engines/engine.py.224: train: epoch 5 | batch 1200/2304 [I 2022-10-08 12:53:37] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:14:20 [I 2022-10-08 12:53:37] ...trap/engines/engine.py.229: process: 0.76702 | load: 0.00302 [I 2022-10-08 12:53:37] ...trap/engines/engine.py.230: loss: 0.25877 [I 2022-10-08 12:53:37] ...optimizers/trijoint.py.151: total_norm: 2.484329 [I 2022-10-08 12:53:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 1210/2304 [I 2022-10-08 12:53:45] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:14:18 [I 2022-10-08 12:53:45] ...trap/engines/engine.py.229: process: 0.78286 | load: 0.00304 [I 2022-10-08 12:53:45] ...trap/engines/engine.py.230: loss: 0.24119 [I 2022-10-08 12:53:45] ...optimizers/trijoint.py.151: total_norm: 3.386957 [I 2022-10-08 12:53:52] ...trap/engines/engine.py.224: train: epoch 5 | batch 1220/2304 [I 2022-10-08 12:53:52] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:13:59 [I 2022-10-08 12:53:52] ...trap/engines/engine.py.229: process: 0.76950 | load: 0.00302 [I 2022-10-08 12:53:52] ...trap/engines/engine.py.230: loss: 0.18663 [I 2022-10-08 12:53:52] ...optimizers/trijoint.py.151: total_norm: 2.277001 [I 2022-10-08 12:54:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 1230/2304 [I 2022-10-08 12:54:00] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:13:58 [I 2022-10-08 12:54:00] ...trap/engines/engine.py.229: process: 0.78129 | load: 0.00305 [I 2022-10-08 12:54:00] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-08 12:54:00] ...optimizers/trijoint.py.151: total_norm: 2.920830 [I 2022-10-08 12:54:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1240/2304 [I 2022-10-08 12:54:08] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:13:45 [I 2022-10-08 12:54:08] ...trap/engines/engine.py.229: process: 0.77095 | load: 0.00303 [I 2022-10-08 12:54:08] ...trap/engines/engine.py.230: loss: 0.18531 [I 2022-10-08 12:54:08] ...optimizers/trijoint.py.151: total_norm: 1.786166 [I 2022-10-08 12:54:16] ...trap/engines/engine.py.224: train: epoch 5 | batch 1250/2304 [I 2022-10-08 12:54:16] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:13:41 [I 2022-10-08 12:54:16] ...trap/engines/engine.py.229: process: 0.77735 | load: 0.00308 [I 2022-10-08 12:54:16] ...trap/engines/engine.py.230: loss: 0.24308 [I 2022-10-08 12:54:16] ...optimizers/trijoint.py.151: total_norm: 3.411688 [I 2022-10-08 12:54:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 1260/2304 [I 2022-10-08 12:54:24] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:13:36 [I 2022-10-08 12:54:24] ...trap/engines/engine.py.229: process: 0.78503 | load: 0.00304 [I 2022-10-08 12:54:24] ...trap/engines/engine.py.230: loss: 0.21618 [I 2022-10-08 12:54:24] ...optimizers/trijoint.py.151: total_norm: 2.334271 [I 2022-10-08 12:54:31] ...trap/engines/engine.py.224: train: epoch 5 | batch 1270/2304 [I 2022-10-08 12:54:31] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:13:22 [I 2022-10-08 12:54:31] ...trap/engines/engine.py.229: process: 0.77184 | load: 0.00305 [I 2022-10-08 12:54:31] ...trap/engines/engine.py.230: loss: 0.18849 [I 2022-10-08 12:54:31] ...optimizers/trijoint.py.151: total_norm: 2.686010 [I 2022-10-08 12:54:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 1280/2304 [I 2022-10-08 12:54:39] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:13:19 [I 2022-10-08 12:54:39] ...trap/engines/engine.py.229: process: 0.78764 | load: 0.00306 [I 2022-10-08 12:54:39] ...trap/engines/engine.py.230: loss: 0.20465 [I 2022-10-08 12:54:39] ...optimizers/trijoint.py.151: total_norm: 2.908975 [I 2022-10-08 12:54:47] ...trap/engines/engine.py.224: train: epoch 5 | batch 1290/2304 [I 2022-10-08 12:54:47] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:13:09 [I 2022-10-08 12:54:47] ...trap/engines/engine.py.229: process: 0.77779 | load: 0.00311 [I 2022-10-08 12:54:47] ...trap/engines/engine.py.230: loss: 0.24498 [I 2022-10-08 12:54:47] ...optimizers/trijoint.py.151: total_norm: 2.667900 [I 2022-10-08 12:54:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 1300/2304 [I 2022-10-08 12:54:55] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:13:05 [I 2022-10-08 12:54:55] ...trap/engines/engine.py.229: process: 0.78710 | load: 0.00314 [I 2022-10-08 12:54:55] ...trap/engines/engine.py.230: loss: 0.41978 [I 2022-10-08 12:54:55] ...optimizers/trijoint.py.151: total_norm: 3.592277 [I 2022-10-08 12:55:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 1310/2304 [I 2022-10-08 12:55:03] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:13:01 [I 2022-10-08 12:55:03] ...trap/engines/engine.py.229: process: 0.79150 | load: 0.00310 [I 2022-10-08 12:55:03] ...trap/engines/engine.py.230: loss: 0.17416 [I 2022-10-08 12:55:03] ...optimizers/trijoint.py.151: total_norm: 2.657380 [I 2022-10-08 12:55:11] ...trap/engines/engine.py.224: train: epoch 5 | batch 1320/2304 [I 2022-10-08 12:55:11] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:12:46 [I 2022-10-08 12:55:11] ...trap/engines/engine.py.229: process: 0.78244 | load: 0.00313 [I 2022-10-08 12:55:11] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-08 12:55:11] ...optimizers/trijoint.py.151: total_norm: 2.611802 [I 2022-10-08 12:55:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 1330/2304 [I 2022-10-08 12:55:18] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:12:40 [I 2022-10-08 12:55:18] ...trap/engines/engine.py.229: process: 0.79265 | load: 0.00313 [I 2022-10-08 12:55:18] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-08 12:55:18] ...optimizers/trijoint.py.151: total_norm: 1.402705 [I 2022-10-08 12:55:26] ...trap/engines/engine.py.224: train: epoch 5 | batch 1340/2304 [I 2022-10-08 12:55:26] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:12:34 [I 2022-10-08 12:55:26] ...trap/engines/engine.py.229: process: 0.78883 | load: 0.00314 [I 2022-10-08 12:55:26] ...trap/engines/engine.py.230: loss: 0.23322 [I 2022-10-08 12:55:26] ...optimizers/trijoint.py.151: total_norm: 2.906502 [I 2022-10-08 12:55:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 1350/2304 [I 2022-10-08 12:55:34] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:12:23 [I 2022-10-08 12:55:34] ...trap/engines/engine.py.229: process: 0.77691 | load: 0.00309 [I 2022-10-08 12:55:34] ...trap/engines/engine.py.230: loss: 0.21372 [I 2022-10-08 12:55:34] ...optimizers/trijoint.py.151: total_norm: 3.112011 [I 2022-10-08 12:55:42] ...trap/engines/engine.py.224: train: epoch 5 | batch 1360/2304 [I 2022-10-08 12:55:42] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:12:13 [I 2022-10-08 12:55:42] ...trap/engines/engine.py.229: process: 0.76728 | load: 0.00311 [I 2022-10-08 12:55:42] ...trap/engines/engine.py.230: loss: 0.28368 [I 2022-10-08 12:55:42] ...optimizers/trijoint.py.151: total_norm: 3.098193 [I 2022-10-08 12:55:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 1370/2304 [I 2022-10-08 12:55:50] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:12:07 [I 2022-10-08 12:55:50] ...trap/engines/engine.py.229: process: 0.77617 | load: 0.00319 [I 2022-10-08 12:55:50] ...trap/engines/engine.py.230: loss: 0.21202 [I 2022-10-08 12:55:50] ...optimizers/trijoint.py.151: total_norm: 2.763326 [I 2022-10-08 12:55:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 1380/2304 [I 2022-10-08 12:55:57] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:11:59 [I 2022-10-08 12:55:57] ...trap/engines/engine.py.229: process: 0.77771 | load: 0.00310 [I 2022-10-08 12:55:57] ...trap/engines/engine.py.230: loss: 0.20704 [I 2022-10-08 12:55:57] ...optimizers/trijoint.py.151: total_norm: 2.973601 [I 2022-10-08 12:56:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 1390/2304 [I 2022-10-08 12:56:05] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:11:52 [I 2022-10-08 12:56:05] ...trap/engines/engine.py.229: process: 0.78018 | load: 0.00311 [I 2022-10-08 12:56:05] ...trap/engines/engine.py.230: loss: 0.27221 [I 2022-10-08 12:56:05] ...optimizers/trijoint.py.151: total_norm: 2.582465 [I 2022-10-08 12:56:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 1400/2304 [I 2022-10-08 12:56:13] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:11:44 [I 2022-10-08 12:56:13] ...trap/engines/engine.py.229: process: 0.78849 | load: 0.00313 [I 2022-10-08 12:56:13] ...trap/engines/engine.py.230: loss: 0.26405 [I 2022-10-08 12:56:13] ...optimizers/trijoint.py.151: total_norm: 2.399400 [I 2022-10-08 12:56:21] ...trap/engines/engine.py.224: train: epoch 5 | batch 1410/2304 [I 2022-10-08 12:56:21] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:11:32 [I 2022-10-08 12:56:21] ...trap/engines/engine.py.229: process: 0.76360 | load: 0.00311 [I 2022-10-08 12:56:21] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-08 12:56:21] ...optimizers/trijoint.py.151: total_norm: 2.338642 [I 2022-10-08 12:56:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 1420/2304 [I 2022-10-08 12:56:29] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:11:27 [I 2022-10-08 12:56:29] ...trap/engines/engine.py.229: process: 0.78102 | load: 0.00313 [I 2022-10-08 12:56:29] ...trap/engines/engine.py.230: loss: 0.25767 [I 2022-10-08 12:56:29] ...optimizers/trijoint.py.151: total_norm: 2.859337 [I 2022-10-08 12:56:36] ...trap/engines/engine.py.224: train: epoch 5 | batch 1430/2304 [I 2022-10-08 12:56:36] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:11:19 [I 2022-10-08 12:56:36] ...trap/engines/engine.py.229: process: 0.76932 | load: 0.00310 [I 2022-10-08 12:56:36] ...trap/engines/engine.py.230: loss: 0.21166 [I 2022-10-08 12:56:36] ...optimizers/trijoint.py.151: total_norm: 2.420161 [I 2022-10-08 12:56:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 1440/2304 [I 2022-10-08 12:56:44] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:11:16 [I 2022-10-08 12:56:44] ...trap/engines/engine.py.229: process: 0.77759 | load: 0.00309 [I 2022-10-08 12:56:44] ...trap/engines/engine.py.230: loss: 0.27603 [I 2022-10-08 12:56:44] ...optimizers/trijoint.py.151: total_norm: 2.644723 [I 2022-10-08 12:56:52] ...trap/engines/engine.py.224: train: epoch 5 | batch 1450/2304 [I 2022-10-08 12:56:52] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:11:05 [I 2022-10-08 12:56:52] ...trap/engines/engine.py.229: process: 0.77476 | load: 0.00312 [I 2022-10-08 12:56:52] ...trap/engines/engine.py.230: loss: 0.18967 [I 2022-10-08 12:56:52] ...optimizers/trijoint.py.151: total_norm: 2.255806 [I 2022-10-08 12:57:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 1460/2304 [I 2022-10-08 12:57:00] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:11:00 [I 2022-10-08 12:57:00] ...trap/engines/engine.py.229: process: 0.79200 | load: 0.00301 [I 2022-10-08 12:57:00] ...trap/engines/engine.py.230: loss: 0.22525 [I 2022-10-08 12:57:00] ...optimizers/trijoint.py.151: total_norm: 2.584894 [I 2022-10-08 12:57:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1470/2304 [I 2022-10-08 12:57:08] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:10:47 [I 2022-10-08 12:57:08] ...trap/engines/engine.py.229: process: 0.76281 | load: 0.00303 [I 2022-10-08 12:57:08] ...trap/engines/engine.py.230: loss: 0.26190 [I 2022-10-08 12:57:08] ...optimizers/trijoint.py.151: total_norm: 2.525475 [I 2022-10-08 12:57:16] ...trap/engines/engine.py.224: train: epoch 5 | batch 1480/2304 [I 2022-10-08 12:57:16] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:10:40 [I 2022-10-08 12:57:16] ...trap/engines/engine.py.229: process: 0.77536 | load: 0.00299 [I 2022-10-08 12:57:16] ...trap/engines/engine.py.230: loss: 0.36899 [I 2022-10-08 12:57:16] ...optimizers/trijoint.py.151: total_norm: 3.307541 [I 2022-10-08 12:57:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 1490/2304 [I 2022-10-08 12:57:23] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:10:32 [I 2022-10-08 12:57:23] ...trap/engines/engine.py.229: process: 0.77161 | load: 0.00303 [I 2022-10-08 12:57:23] ...trap/engines/engine.py.230: loss: 0.27827 [I 2022-10-08 12:57:23] ...optimizers/trijoint.py.151: total_norm: 3.397331 [I 2022-10-08 12:57:31] ...trap/engines/engine.py.224: train: epoch 5 | batch 1500/2304 [I 2022-10-08 12:57:31] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:10:23 [I 2022-10-08 12:57:31] ...trap/engines/engine.py.229: process: 0.77669 | load: 0.00302 [I 2022-10-08 12:57:31] ...trap/engines/engine.py.230: loss: 0.22635 [I 2022-10-08 12:57:31] ...optimizers/trijoint.py.151: total_norm: 2.331148 [I 2022-10-08 12:57:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 1510/2304 [I 2022-10-08 12:57:39] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:10:16 [I 2022-10-08 12:57:39] ...trap/engines/engine.py.229: process: 0.77303 | load: 0.00301 [I 2022-10-08 12:57:39] ...trap/engines/engine.py.230: loss: 0.28493 [I 2022-10-08 12:57:39] ...optimizers/trijoint.py.151: total_norm: 3.208997 [I 2022-10-08 12:57:47] ...trap/engines/engine.py.224: train: epoch 5 | batch 1520/2304 [I 2022-10-08 12:57:47] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:10:09 [I 2022-10-08 12:57:47] ...trap/engines/engine.py.229: process: 0.77770 | load: 0.00303 [I 2022-10-08 12:57:47] ...trap/engines/engine.py.230: loss: 0.22174 [I 2022-10-08 12:57:47] ...optimizers/trijoint.py.151: total_norm: 3.117435 [I 2022-10-08 12:57:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 1530/2304 [I 2022-10-08 12:57:54] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:09:59 [I 2022-10-08 12:57:54] ...trap/engines/engine.py.229: process: 0.76228 | load: 0.00320 [I 2022-10-08 12:57:54] ...trap/engines/engine.py.230: loss: 0.26915 [I 2022-10-08 12:57:54] ...optimizers/trijoint.py.151: total_norm: 3.254187 [I 2022-10-08 12:58:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 1540/2304 [I 2022-10-08 12:58:02] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:09:49 [I 2022-10-08 12:58:02] ...trap/engines/engine.py.229: process: 0.75808 | load: 0.00302 [I 2022-10-08 12:58:02] ...trap/engines/engine.py.230: loss: 0.23804 [I 2022-10-08 12:58:02] ...optimizers/trijoint.py.151: total_norm: 2.235443 [I 2022-10-08 12:58:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 1550/2304 [I 2022-10-08 12:58:10] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:09:47 [I 2022-10-08 12:58:10] ...trap/engines/engine.py.229: process: 0.78085 | load: 0.00300 [I 2022-10-08 12:58:10] ...trap/engines/engine.py.230: loss: 0.21355 [I 2022-10-08 12:58:10] ...optimizers/trijoint.py.151: total_norm: 3.245275 [I 2022-10-08 12:58:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 1560/2304 [I 2022-10-08 12:58:18] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:09:37 [I 2022-10-08 12:58:18] ...trap/engines/engine.py.229: process: 0.77094 | load: 0.00306 [I 2022-10-08 12:58:18] ...trap/engines/engine.py.230: loss: 0.24416 [I 2022-10-08 12:58:18] ...optimizers/trijoint.py.151: total_norm: 2.826204 [I 2022-10-08 12:58:26] ...trap/engines/engine.py.224: train: epoch 5 | batch 1570/2304 [I 2022-10-08 12:58:26] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:09:27 [I 2022-10-08 12:58:26] ...trap/engines/engine.py.229: process: 0.76697 | load: 0.00304 [I 2022-10-08 12:58:26] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-08 12:58:26] ...optimizers/trijoint.py.151: total_norm: 2.578737 [I 2022-10-08 12:58:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 1580/2304 [I 2022-10-08 12:58:33] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:09:23 [I 2022-10-08 12:58:33] ...trap/engines/engine.py.229: process: 0.78314 | load: 0.00300 [I 2022-10-08 12:58:33] ...trap/engines/engine.py.230: loss: 0.18333 [I 2022-10-08 12:58:33] ...optimizers/trijoint.py.151: total_norm: 2.474278 [I 2022-10-08 12:58:41] ...trap/engines/engine.py.224: train: epoch 5 | batch 1590/2304 [I 2022-10-08 12:58:41] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:09:17 [I 2022-10-08 12:58:41] ...trap/engines/engine.py.229: process: 0.77592 | load: 0.00307 [I 2022-10-08 12:58:41] ...trap/engines/engine.py.230: loss: 0.21597 [I 2022-10-08 12:58:41] ...optimizers/trijoint.py.151: total_norm: 2.023489 [I 2022-10-08 12:58:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 1600/2304 [I 2022-10-08 12:58:49] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:09:11 [I 2022-10-08 12:58:49] ...trap/engines/engine.py.229: process: 0.78254 | load: 0.00307 [I 2022-10-08 12:58:49] ...trap/engines/engine.py.230: loss: 0.24623 [I 2022-10-08 12:58:49] ...optimizers/trijoint.py.151: total_norm: 2.846446 [I 2022-10-08 12:58:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 1610/2304 [I 2022-10-08 12:58:57] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:09:01 [I 2022-10-08 12:58:57] ...trap/engines/engine.py.229: process: 0.78471 | load: 0.00302 [I 2022-10-08 12:58:57] ...trap/engines/engine.py.230: loss: 0.24002 [I 2022-10-08 12:58:57] ...optimizers/trijoint.py.151: total_norm: 2.533097 [I 2022-10-08 12:59:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 1620/2304 [I 2022-10-08 12:59:05] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:08:53 [I 2022-10-08 12:59:05] ...trap/engines/engine.py.229: process: 0.78054 | load: 0.00303 [I 2022-10-08 12:59:05] ...trap/engines/engine.py.230: loss: 0.28223 [I 2022-10-08 12:59:05] ...optimizers/trijoint.py.151: total_norm: 3.607630 [I 2022-10-08 12:59:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 1630/2304 [I 2022-10-08 12:59:12] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:08:42 [I 2022-10-08 12:59:12] ...trap/engines/engine.py.229: process: 0.78016 | load: 0.00302 [I 2022-10-08 12:59:12] ...trap/engines/engine.py.230: loss: 0.30172 [I 2022-10-08 12:59:12] ...optimizers/trijoint.py.151: total_norm: 3.055907 [I 2022-10-08 12:59:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 1640/2304 [I 2022-10-08 12:59:20] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:08:37 [I 2022-10-08 12:59:20] ...trap/engines/engine.py.229: process: 0.77939 | load: 0.00302 [I 2022-10-08 12:59:20] ...trap/engines/engine.py.230: loss: 0.21851 [I 2022-10-08 12:59:20] ...optimizers/trijoint.py.151: total_norm: 2.358886 [I 2022-10-08 12:59:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 1650/2304 [I 2022-10-08 12:59:28] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:08:28 [I 2022-10-08 12:59:28] ...trap/engines/engine.py.229: process: 0.78055 | load: 0.00306 [I 2022-10-08 12:59:28] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-08 12:59:28] ...optimizers/trijoint.py.151: total_norm: 2.453438 [I 2022-10-08 12:59:36] ...trap/engines/engine.py.224: train: epoch 5 | batch 1660/2304 [I 2022-10-08 12:59:36] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:08:21 [I 2022-10-08 12:59:36] ...trap/engines/engine.py.229: process: 0.77976 | load: 0.00301 [I 2022-10-08 12:59:36] ...trap/engines/engine.py.230: loss: 0.18166 [I 2022-10-08 12:59:36] ...optimizers/trijoint.py.151: total_norm: 2.414717 [I 2022-10-08 12:59:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 1670/2304 [I 2022-10-08 12:59:44] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:08:13 [I 2022-10-08 12:59:44] ...trap/engines/engine.py.229: process: 0.78188 | load: 0.00301 [I 2022-10-08 12:59:44] ...trap/engines/engine.py.230: loss: 0.27692 [I 2022-10-08 12:59:44] ...optimizers/trijoint.py.151: total_norm: 3.381645 [I 2022-10-08 12:59:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 1680/2304 [I 2022-10-08 12:59:51] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:08:06 [I 2022-10-08 12:59:51] ...trap/engines/engine.py.229: process: 0.77758 | load: 0.00302 [I 2022-10-08 12:59:51] ...trap/engines/engine.py.230: loss: 0.25026 [I 2022-10-08 12:59:51] ...optimizers/trijoint.py.151: total_norm: 2.990444 [I 2022-10-08 12:59:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 1690/2304 [I 2022-10-08 12:59:59] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:07:59 [I 2022-10-08 12:59:59] ...trap/engines/engine.py.229: process: 0.78155 | load: 0.00302 [I 2022-10-08 12:59:59] ...trap/engines/engine.py.230: loss: 0.24377 [I 2022-10-08 12:59:59] ...optimizers/trijoint.py.151: total_norm: 3.301359 [I 2022-10-08 13:00:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 1700/2304 [I 2022-10-08 13:00:07] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:07:50 [I 2022-10-08 13:00:07] ...trap/engines/engine.py.229: process: 0.78590 | load: 0.00306 [I 2022-10-08 13:00:07] ...trap/engines/engine.py.230: loss: 0.18138 [I 2022-10-08 13:00:07] ...optimizers/trijoint.py.151: total_norm: 2.265054 [I 2022-10-08 13:00:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 1710/2304 [I 2022-10-08 13:00:15] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:07:43 [I 2022-10-08 13:00:15] ...trap/engines/engine.py.229: process: 0.78117 | load: 0.00303 [I 2022-10-08 13:00:15] ...trap/engines/engine.py.230: loss: 0.27080 [I 2022-10-08 13:00:15] ...optimizers/trijoint.py.151: total_norm: 3.371956 [I 2022-10-08 13:00:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 1720/2304 [I 2022-10-08 13:00:23] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:07:38 [I 2022-10-08 13:00:23] ...trap/engines/engine.py.229: process: 0.80086 | load: 0.00304 [I 2022-10-08 13:00:23] ...trap/engines/engine.py.230: loss: 0.24560 [I 2022-10-08 13:00:23] ...optimizers/trijoint.py.151: total_norm: 2.561157 [I 2022-10-08 13:00:31] ...trap/engines/engine.py.224: train: epoch 5 | batch 1730/2304 [I 2022-10-08 13:00:31] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:07:31 [I 2022-10-08 13:00:31] ...trap/engines/engine.py.229: process: 0.79893 | load: 0.00305 [I 2022-10-08 13:00:31] ...trap/engines/engine.py.230: loss: 0.14204 [I 2022-10-08 13:00:31] ...optimizers/trijoint.py.151: total_norm: 2.346929 [I 2022-10-08 13:00:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 1740/2304 [I 2022-10-08 13:00:38] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:07:15 [I 2022-10-08 13:00:38] ...trap/engines/engine.py.229: process: 0.76510 | load: 0.00305 [I 2022-10-08 13:00:38] ...trap/engines/engine.py.230: loss: 0.21446 [I 2022-10-08 13:00:38] ...optimizers/trijoint.py.151: total_norm: 2.036842 [I 2022-10-08 13:00:46] ...trap/engines/engine.py.224: train: epoch 5 | batch 1750/2304 [I 2022-10-08 13:00:46] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:07:12 [I 2022-10-08 13:00:46] ...trap/engines/engine.py.229: process: 0.78005 | load: 0.00305 [I 2022-10-08 13:00:46] ...trap/engines/engine.py.230: loss: 0.27164 [I 2022-10-08 13:00:46] ...optimizers/trijoint.py.151: total_norm: 3.744313 [I 2022-10-08 13:00:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 1760/2304 [I 2022-10-08 13:00:54] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:07:03 [I 2022-10-08 13:00:54] ...trap/engines/engine.py.229: process: 0.76849 | load: 0.00302 [I 2022-10-08 13:00:54] ...trap/engines/engine.py.230: loss: 0.17138 [I 2022-10-08 13:00:54] ...optimizers/trijoint.py.151: total_norm: 2.081472 [I 2022-10-08 13:01:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 1770/2304 [I 2022-10-08 13:01:02] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:06:54 [I 2022-10-08 13:01:02] ...trap/engines/engine.py.229: process: 0.78482 | load: 0.00302 [I 2022-10-08 13:01:02] ...trap/engines/engine.py.230: loss: 0.25164 [I 2022-10-08 13:01:02] ...optimizers/trijoint.py.151: total_norm: 3.083465 [I 2022-10-08 13:01:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 1780/2304 [I 2022-10-08 13:01:09] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:06:50 [I 2022-10-08 13:01:09] ...trap/engines/engine.py.229: process: 0.78603 | load: 0.00303 [I 2022-10-08 13:01:09] ...trap/engines/engine.py.230: loss: 0.33461 [I 2022-10-08 13:01:09] ...optimizers/trijoint.py.151: total_norm: 2.860370 [I 2022-10-08 13:01:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 1790/2304 [I 2022-10-08 13:01:17] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:06:42 [I 2022-10-08 13:01:17] ...trap/engines/engine.py.229: process: 0.77991 | load: 0.00302 [I 2022-10-08 13:01:17] ...trap/engines/engine.py.230: loss: 0.28503 [I 2022-10-08 13:01:17] ...optimizers/trijoint.py.151: total_norm: 2.722895 [I 2022-10-08 13:01:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 1800/2304 [I 2022-10-08 13:01:25] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:06:33 [I 2022-10-08 13:01:25] ...trap/engines/engine.py.229: process: 0.77424 | load: 0.00305 [I 2022-10-08 13:01:25] ...trap/engines/engine.py.230: loss: 0.24302 [I 2022-10-08 13:01:25] ...optimizers/trijoint.py.151: total_norm: 2.891338 [I 2022-10-08 13:01:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 1810/2304 [I 2022-10-08 13:01:33] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:06:25 [I 2022-10-08 13:01:33] ...trap/engines/engine.py.229: process: 0.78371 | load: 0.00302 [I 2022-10-08 13:01:33] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-08 13:01:33] ...optimizers/trijoint.py.151: total_norm: 2.353689 [I 2022-10-08 13:01:41] ...trap/engines/engine.py.224: train: epoch 5 | batch 1820/2304 [I 2022-10-08 13:01:41] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:06:18 [I 2022-10-08 13:01:41] ...trap/engines/engine.py.229: process: 0.78011 | load: 0.00301 [I 2022-10-08 13:01:41] ...trap/engines/engine.py.230: loss: 0.20387 [I 2022-10-08 13:01:41] ...optimizers/trijoint.py.151: total_norm: 2.752076 [I 2022-10-08 13:01:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 1830/2304 [I 2022-10-08 13:01:49] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:06:10 [I 2022-10-08 13:01:49] ...trap/engines/engine.py.229: process: 0.78383 | load: 0.00304 [I 2022-10-08 13:01:49] ...trap/engines/engine.py.230: loss: 0.19998 [I 2022-10-08 13:01:49] ...optimizers/trijoint.py.151: total_norm: 3.536785 [I 2022-10-08 13:01:56] ...trap/engines/engine.py.224: train: epoch 5 | batch 1840/2304 [I 2022-10-08 13:01:56] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:06:00 [I 2022-10-08 13:01:56] ...trap/engines/engine.py.229: process: 0.77931 | load: 0.00307 [I 2022-10-08 13:01:56] ...trap/engines/engine.py.230: loss: 0.20217 [I 2022-10-08 13:01:56] ...optimizers/trijoint.py.151: total_norm: 2.751694 [I 2022-10-08 13:02:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 1850/2304 [I 2022-10-08 13:02:04] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:05:51 [I 2022-10-08 13:02:04] ...trap/engines/engine.py.229: process: 0.77561 | load: 0.00301 [I 2022-10-08 13:02:04] ...trap/engines/engine.py.230: loss: 0.24725 [I 2022-10-08 13:02:04] ...optimizers/trijoint.py.151: total_norm: 2.632256 [I 2022-10-08 13:02:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 1860/2304 [I 2022-10-08 13:02:12] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:05:45 [I 2022-10-08 13:02:12] ...trap/engines/engine.py.229: process: 0.78298 | load: 0.00299 [I 2022-10-08 13:02:12] ...trap/engines/engine.py.230: loss: 0.25653 [I 2022-10-08 13:02:12] ...optimizers/trijoint.py.151: total_norm: 2.954432 [I 2022-10-08 13:02:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 1870/2304 [I 2022-10-08 13:02:20] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:05:37 [I 2022-10-08 13:02:20] ...trap/engines/engine.py.229: process: 0.77817 | load: 0.00302 [I 2022-10-08 13:02:20] ...trap/engines/engine.py.230: loss: 0.17444 [I 2022-10-08 13:02:20] ...optimizers/trijoint.py.151: total_norm: 2.133860 [I 2022-10-08 13:02:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 1880/2304 [I 2022-10-08 13:02:28] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:05:30 [I 2022-10-08 13:02:28] ...trap/engines/engine.py.229: process: 0.78830 | load: 0.00302 [I 2022-10-08 13:02:28] ...trap/engines/engine.py.230: loss: 0.23175 [I 2022-10-08 13:02:28] ...optimizers/trijoint.py.151: total_norm: 2.515613 [I 2022-10-08 13:02:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 1890/2304 [I 2022-10-08 13:02:35] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:05:22 [I 2022-10-08 13:02:35] ...trap/engines/engine.py.229: process: 0.77385 | load: 0.00304 [I 2022-10-08 13:02:35] ...trap/engines/engine.py.230: loss: 0.20179 [I 2022-10-08 13:02:35] ...optimizers/trijoint.py.151: total_norm: 2.535505 [I 2022-10-08 13:02:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 1900/2304 [I 2022-10-08 13:02:43] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:05:13 [I 2022-10-08 13:02:43] ...trap/engines/engine.py.229: process: 0.78690 | load: 0.00299 [I 2022-10-08 13:02:43] ...trap/engines/engine.py.230: loss: 0.18444 [I 2022-10-08 13:02:43] ...optimizers/trijoint.py.151: total_norm: 2.370296 [I 2022-10-08 13:02:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 1910/2304 [I 2022-10-08 13:02:51] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:05:05 [I 2022-10-08 13:02:51] ...trap/engines/engine.py.229: process: 0.78416 | load: 0.00299 [I 2022-10-08 13:02:51] ...trap/engines/engine.py.230: loss: 0.17604 [I 2022-10-08 13:02:51] ...optimizers/trijoint.py.151: total_norm: 1.964085 [I 2022-10-08 13:02:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 1920/2304 [I 2022-10-08 13:02:59] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:04:59 [I 2022-10-08 13:02:59] ...trap/engines/engine.py.229: process: 0.79363 | load: 0.00301 [I 2022-10-08 13:02:59] ...trap/engines/engine.py.230: loss: 0.24734 [I 2022-10-08 13:02:59] ...optimizers/trijoint.py.151: total_norm: 2.970953 [I 2022-10-08 13:03:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 1930/2304 [I 2022-10-08 13:03:07] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:04:51 [I 2022-10-08 13:03:07] ...trap/engines/engine.py.229: process: 0.77989 | load: 0.00304 [I 2022-10-08 13:03:07] ...trap/engines/engine.py.230: loss: 0.27028 [I 2022-10-08 13:03:07] ...optimizers/trijoint.py.151: total_norm: 2.810131 [I 2022-10-08 13:03:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 1940/2304 [I 2022-10-08 13:03:14] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:04:41 [I 2022-10-08 13:03:14] ...trap/engines/engine.py.229: process: 0.76717 | load: 0.00304 [I 2022-10-08 13:03:14] ...trap/engines/engine.py.230: loss: 0.24463 [I 2022-10-08 13:03:14] ...optimizers/trijoint.py.151: total_norm: 2.599880 [I 2022-10-08 13:03:22] ...trap/engines/engine.py.224: train: epoch 5 | batch 1950/2304 [I 2022-10-08 13:03:22] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:04:36 [I 2022-10-08 13:03:22] ...trap/engines/engine.py.229: process: 0.78113 | load: 0.00305 [I 2022-10-08 13:03:22] ...trap/engines/engine.py.230: loss: 0.28638 [I 2022-10-08 13:03:22] ...optimizers/trijoint.py.151: total_norm: 2.849435 [I 2022-10-08 13:03:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 1960/2304 [I 2022-10-08 13:03:30] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:04:28 [I 2022-10-08 13:03:30] ...trap/engines/engine.py.229: process: 0.77523 | load: 0.00301 [I 2022-10-08 13:03:30] ...trap/engines/engine.py.230: loss: 0.29023 [I 2022-10-08 13:03:30] ...optimizers/trijoint.py.151: total_norm: 3.701695 [I 2022-10-08 13:03:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 1970/2304 [I 2022-10-08 13:03:38] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:04:20 [I 2022-10-08 13:03:38] ...trap/engines/engine.py.229: process: 0.78048 | load: 0.00302 [I 2022-10-08 13:03:38] ...trap/engines/engine.py.230: loss: 0.21548 [I 2022-10-08 13:03:38] ...optimizers/trijoint.py.151: total_norm: 2.635558 [I 2022-10-08 13:03:46] ...trap/engines/engine.py.224: train: epoch 5 | batch 1980/2304 [I 2022-10-08 13:03:46] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:04:11 [I 2022-10-08 13:03:46] ...trap/engines/engine.py.229: process: 0.77663 | load: 0.00303 [I 2022-10-08 13:03:46] ...trap/engines/engine.py.230: loss: 0.12870 [I 2022-10-08 13:03:46] ...optimizers/trijoint.py.151: total_norm: 2.021350 [I 2022-10-08 13:03:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 1990/2304 [I 2022-10-08 13:03:53] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:04:04 [I 2022-10-08 13:03:53] ...trap/engines/engine.py.229: process: 0.77875 | load: 0.00300 [I 2022-10-08 13:03:53] ...trap/engines/engine.py.230: loss: 0.28926 [I 2022-10-08 13:03:53] ...optimizers/trijoint.py.151: total_norm: 3.266284 [I 2022-10-08 13:04:01] ...trap/engines/engine.py.224: train: epoch 5 | batch 2000/2304 [I 2022-10-08 13:04:01] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:03:56 [I 2022-10-08 13:04:01] ...trap/engines/engine.py.229: process: 0.78159 | load: 0.00305 [I 2022-10-08 13:04:01] ...trap/engines/engine.py.230: loss: 0.21435 [I 2022-10-08 13:04:01] ...optimizers/trijoint.py.151: total_norm: 2.251775 [I 2022-10-08 13:04:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 2010/2304 [I 2022-10-08 13:04:09] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:49 [I 2022-10-08 13:04:09] ...trap/engines/engine.py.229: process: 0.77485 | load: 0.00304 [I 2022-10-08 13:04:09] ...trap/engines/engine.py.230: loss: 0.14265 [I 2022-10-08 13:04:09] ...optimizers/trijoint.py.151: total_norm: 2.228422 [I 2022-10-08 13:04:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 2020/2304 [I 2022-10-08 13:04:17] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:03:42 [I 2022-10-08 13:04:17] ...trap/engines/engine.py.229: process: 0.78777 | load: 0.00302 [I 2022-10-08 13:04:17] ...trap/engines/engine.py.230: loss: 0.20481 [I 2022-10-08 13:04:17] ...optimizers/trijoint.py.151: total_norm: 3.268925 [I 2022-10-08 13:04:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 2030/2304 [I 2022-10-08 13:04:25] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:03:33 [I 2022-10-08 13:04:25] ...trap/engines/engine.py.229: process: 0.77601 | load: 0.00302 [I 2022-10-08 13:04:25] ...trap/engines/engine.py.230: loss: 0.26417 [I 2022-10-08 13:04:25] ...optimizers/trijoint.py.151: total_norm: 2.908087 [I 2022-10-08 13:04:32] ...trap/engines/engine.py.224: train: epoch 5 | batch 2040/2304 [I 2022-10-08 13:04:32] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:03:24 [I 2022-10-08 13:04:32] ...trap/engines/engine.py.229: process: 0.76599 | load: 0.00308 [I 2022-10-08 13:04:32] ...trap/engines/engine.py.230: loss: 0.23908 [I 2022-10-08 13:04:32] ...optimizers/trijoint.py.151: total_norm: 2.671087 [I 2022-10-08 13:04:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 2050/2304 [I 2022-10-08 13:04:40] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:03:16 [I 2022-10-08 13:04:40] ...trap/engines/engine.py.229: process: 0.77410 | load: 0.00304 [I 2022-10-08 13:04:40] ...trap/engines/engine.py.230: loss: 0.22295 [I 2022-10-08 13:04:40] ...optimizers/trijoint.py.151: total_norm: 2.977076 [I 2022-10-08 13:04:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 2060/2304 [I 2022-10-08 13:04:48] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:03:09 [I 2022-10-08 13:04:48] ...trap/engines/engine.py.229: process: 0.77960 | load: 0.00305 [I 2022-10-08 13:04:48] ...trap/engines/engine.py.230: loss: 0.23313 [I 2022-10-08 13:04:48] ...optimizers/trijoint.py.151: total_norm: 2.492806 [I 2022-10-08 13:04:56] ...trap/engines/engine.py.224: train: epoch 5 | batch 2070/2304 [I 2022-10-08 13:04:56] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:03:02 [I 2022-10-08 13:04:56] ...trap/engines/engine.py.229: process: 0.77902 | load: 0.00303 [I 2022-10-08 13:04:56] ...trap/engines/engine.py.230: loss: 0.25856 [I 2022-10-08 13:04:56] ...optimizers/trijoint.py.151: total_norm: 3.325551 [I 2022-10-08 13:05:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 2080/2304 [I 2022-10-08 13:05:03] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:02:54 [I 2022-10-08 13:05:03] ...trap/engines/engine.py.229: process: 0.78719 | load: 0.00301 [I 2022-10-08 13:05:03] ...trap/engines/engine.py.230: loss: 0.22544 [I 2022-10-08 13:05:03] ...optimizers/trijoint.py.151: total_norm: 2.641371 [I 2022-10-08 13:05:11] ...trap/engines/engine.py.224: train: epoch 5 | batch 2090/2304 [I 2022-10-08 13:05:11] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:02:47 [I 2022-10-08 13:05:11] ...trap/engines/engine.py.229: process: 0.78146 | load: 0.00301 [I 2022-10-08 13:05:11] ...trap/engines/engine.py.230: loss: 0.17619 [I 2022-10-08 13:05:11] ...optimizers/trijoint.py.151: total_norm: 1.824332 [I 2022-10-08 13:05:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 2100/2304 [I 2022-10-08 13:05:19] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:02:39 [I 2022-10-08 13:05:19] ...trap/engines/engine.py.229: process: 0.78095 | load: 0.00302 [I 2022-10-08 13:05:19] ...trap/engines/engine.py.230: loss: 0.19112 [I 2022-10-08 13:05:19] ...optimizers/trijoint.py.151: total_norm: 2.209770 [I 2022-10-08 13:05:27] ...trap/engines/engine.py.224: train: epoch 5 | batch 2110/2304 [I 2022-10-08 13:05:27] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:02:31 [I 2022-10-08 13:05:27] ...trap/engines/engine.py.229: process: 0.77911 | load: 0.00305 [I 2022-10-08 13:05:27] ...trap/engines/engine.py.230: loss: 0.21856 [I 2022-10-08 13:05:27] ...optimizers/trijoint.py.151: total_norm: 2.906363 [I 2022-10-08 13:05:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 2120/2304 [I 2022-10-08 13:05:35] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:24 [I 2022-10-08 13:05:35] ...trap/engines/engine.py.229: process: 0.79317 | load: 0.00301 [I 2022-10-08 13:05:35] ...trap/engines/engine.py.230: loss: 0.27574 [I 2022-10-08 13:05:35] ...optimizers/trijoint.py.151: total_norm: 3.363432 [I 2022-10-08 13:05:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 2130/2304 [I 2022-10-08 13:05:43] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:02:16 [I 2022-10-08 13:05:43] ...trap/engines/engine.py.229: process: 0.76801 | load: 0.00300 [I 2022-10-08 13:05:43] ...trap/engines/engine.py.230: loss: 0.24239 [I 2022-10-08 13:05:43] ...optimizers/trijoint.py.151: total_norm: 2.906637 [I 2022-10-08 13:05:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 2140/2304 [I 2022-10-08 13:05:51] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:02:07 [I 2022-10-08 13:05:51] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00300 [I 2022-10-08 13:05:51] ...trap/engines/engine.py.230: loss: 0.17827 [I 2022-10-08 13:05:51] ...optimizers/trijoint.py.151: total_norm: 2.400028 [I 2022-10-08 13:05:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 2150/2304 [I 2022-10-08 13:05:58] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:01:59 [I 2022-10-08 13:05:58] ...trap/engines/engine.py.229: process: 0.76336 | load: 0.00302 [I 2022-10-08 13:05:58] ...trap/engines/engine.py.230: loss: 0.29400 [I 2022-10-08 13:05:58] ...optimizers/trijoint.py.151: total_norm: 3.461042 [I 2022-10-08 13:06:06] ...trap/engines/engine.py.224: train: epoch 5 | batch 2160/2304 [I 2022-10-08 13:06:06] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:01:51 [I 2022-10-08 13:06:06] ...trap/engines/engine.py.229: process: 0.76398 | load: 0.00301 [I 2022-10-08 13:06:06] ...trap/engines/engine.py.230: loss: 0.27308 [I 2022-10-08 13:06:06] ...optimizers/trijoint.py.151: total_norm: 2.871727 [I 2022-10-08 13:06:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 2170/2304 [I 2022-10-08 13:06:14] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:01:44 [I 2022-10-08 13:06:14] ...trap/engines/engine.py.229: process: 0.77675 | load: 0.00300 [I 2022-10-08 13:06:14] ...trap/engines/engine.py.230: loss: 0.18088 [I 2022-10-08 13:06:14] ...optimizers/trijoint.py.151: total_norm: 2.818893 [I 2022-10-08 13:06:22] ...trap/engines/engine.py.224: train: epoch 5 | batch 2180/2304 [I 2022-10-08 13:06:22] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:01:36 [I 2022-10-08 13:06:22] ...trap/engines/engine.py.229: process: 0.76761 | load: 0.00304 [I 2022-10-08 13:06:22] ...trap/engines/engine.py.230: loss: 0.31175 [I 2022-10-08 13:06:22] ...optimizers/trijoint.py.151: total_norm: 2.831587 [I 2022-10-08 13:06:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 2190/2304 [I 2022-10-08 13:06:30] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:01:28 [I 2022-10-08 13:06:30] ...trap/engines/engine.py.229: process: 0.76398 | load: 0.00301 [I 2022-10-08 13:06:30] ...trap/engines/engine.py.230: loss: 0.22135 [I 2022-10-08 13:06:30] ...optimizers/trijoint.py.151: total_norm: 3.073087 [I 2022-10-08 13:06:37] ...trap/engines/engine.py.224: train: epoch 5 | batch 2200/2304 [I 2022-10-08 13:06:37] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:01:20 [I 2022-10-08 13:06:37] ...trap/engines/engine.py.229: process: 0.77464 | load: 0.00301 [I 2022-10-08 13:06:37] ...trap/engines/engine.py.230: loss: 0.21295 [I 2022-10-08 13:06:37] ...optimizers/trijoint.py.151: total_norm: 3.699024 [I 2022-10-08 13:06:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 2210/2304 [I 2022-10-08 13:06:45] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:01:12 [I 2022-10-08 13:06:45] ...trap/engines/engine.py.229: process: 0.78211 | load: 0.00300 [I 2022-10-08 13:06:45] ...trap/engines/engine.py.230: loss: 0.20452 [I 2022-10-08 13:06:45] ...optimizers/trijoint.py.151: total_norm: 2.607410 [I 2022-10-08 13:06:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 2220/2304 [I 2022-10-08 13:06:53] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:01:05 [I 2022-10-08 13:06:53] ...trap/engines/engine.py.229: process: 0.77247 | load: 0.00305 [I 2022-10-08 13:06:53] ...trap/engines/engine.py.230: loss: 0.28847 [I 2022-10-08 13:06:53] ...optimizers/trijoint.py.151: total_norm: 2.180362 [I 2022-10-08 13:07:01] ...trap/engines/engine.py.224: train: epoch 5 | batch 2230/2304 [I 2022-10-08 13:07:01] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:00:57 [I 2022-10-08 13:07:01] ...trap/engines/engine.py.229: process: 0.78481 | load: 0.00299 [I 2022-10-08 13:07:01] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-08 13:07:01] ...optimizers/trijoint.py.151: total_norm: 1.909136 [I 2022-10-08 13:07:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 2240/2304 [I 2022-10-08 13:07:09] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:00:49 [I 2022-10-08 13:07:09] ...trap/engines/engine.py.229: process: 0.77621 | load: 0.00301 [I 2022-10-08 13:07:09] ...trap/engines/engine.py.230: loss: 0.25657 [I 2022-10-08 13:07:09] ...optimizers/trijoint.py.151: total_norm: 3.046418 [I 2022-10-08 13:07:16] ...trap/engines/engine.py.224: train: epoch 5 | batch 2250/2304 [I 2022-10-08 13:07:16] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:00:42 [I 2022-10-08 13:07:16] ...trap/engines/engine.py.229: process: 0.79318 | load: 0.00299 [I 2022-10-08 13:07:16] ...trap/engines/engine.py.230: loss: 0.21742 [I 2022-10-08 13:07:16] ...optimizers/trijoint.py.151: total_norm: 2.454024 [I 2022-10-08 13:07:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 2260/2304 [I 2022-10-08 13:07:24] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:00:34 [I 2022-10-08 13:07:24] ...trap/engines/engine.py.229: process: 0.78520 | load: 0.00302 [I 2022-10-08 13:07:24] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-08 13:07:24] ...optimizers/trijoint.py.151: total_norm: 1.974492 [I 2022-10-08 13:07:32] ...trap/engines/engine.py.224: train: epoch 5 | batch 2270/2304 [I 2022-10-08 13:07:32] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:26 [I 2022-10-08 13:07:32] ...trap/engines/engine.py.229: process: 0.78151 | load: 0.00308 [I 2022-10-08 13:07:32] ...trap/engines/engine.py.230: loss: 0.17929 [I 2022-10-08 13:07:32] ...optimizers/trijoint.py.151: total_norm: 2.552860 [I 2022-10-08 13:07:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 2280/2304 [I 2022-10-08 13:07:40] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:00:18 [I 2022-10-08 13:07:40] ...trap/engines/engine.py.229: process: 0.77631 | load: 0.00305 [I 2022-10-08 13:07:40] ...trap/engines/engine.py.230: loss: 0.12205 [I 2022-10-08 13:07:40] ...optimizers/trijoint.py.151: total_norm: 2.098211 [I 2022-10-08 13:07:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 2290/2304 [I 2022-10-08 13:07:48] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:00:10 [I 2022-10-08 13:07:48] ...trap/engines/engine.py.229: process: 0.78001 | load: 0.00301 [I 2022-10-08 13:07:48] ...trap/engines/engine.py.230: loss: 0.23505 [I 2022-10-08 13:07:48] ...optimizers/trijoint.py.151: total_norm: 3.184522 [I 2022-10-08 13:07:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 2300/2304 [I 2022-10-08 13:07:55] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:00:03 [I 2022-10-08 13:07:55] ...trap/engines/engine.py.229: process: 0.76872 | load: 0.00017 [I 2022-10-08 13:07:55] ...trap/engines/engine.py.230: loss: 0.27487 [I 2022-10-08 13:07:55] ...optimizers/trijoint.py.151: total_norm: 2.569114 [I 2022-10-08 13:07:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 2304/2304 [I 2022-10-08 13:07:58] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:00:00 [I 2022-10-08 13:07:58] ...trap/engines/engine.py.229: process: 0.76957 | load: 0.00018 [I 2022-10-08 13:07:58] ...trap/engines/engine.py.230: loss: 0.25690 [I 2022-10-08 13:07:58] ...optimizers/trijoint.py.151: total_norm: 2.352999 [S 2022-10-08 13:08:00] ...trap/engines/engine.py.129: train_epoch.epoch: 5 [S 2022-10-08 13:08:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.22264648304197637 [I 2022-10-08 13:08:00] ...s/criterions/triplet.py.84: margin = 0.08000000000000002 [I 2022-10-08 13:08:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 5 [I 2022-10-08 13:08:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 13:08:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 13:08:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 13:08:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 13:08:02] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 13:08:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 0/510 [I 2022-10-08 13:08:02] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:56 [I 2022-10-08 13:08:02] ...trap/engines/engine.py.326: process: 2.58062 | load: 2.04118 [I 2022-10-08 13:08:07] ...trap/engines/engine.py.321: eval: epoch 5 | batch 10/510 [I 2022-10-08 13:08:07] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:10 [I 2022-10-08 13:08:07] ...trap/engines/engine.py.326: process: 0.55380 | load: 0.00018 [I 2022-10-08 13:08:14] ...trap/engines/engine.py.321: eval: epoch 5 | batch 20/510 [I 2022-10-08 13:08:14] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:05:12 [I 2022-10-08 13:08:14] ...trap/engines/engine.py.326: process: 0.70118 | load: 0.00019 [I 2022-10-08 13:08:19] ...trap/engines/engine.py.321: eval: epoch 5 | batch 30/510 [I 2022-10-08 13:08:19] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:07 [I 2022-10-08 13:08:19] ...trap/engines/engine.py.326: process: 0.57152 | load: 0.00019 [I 2022-10-08 13:08:24] ...trap/engines/engine.py.321: eval: epoch 5 | batch 40/510 [I 2022-10-08 13:08:24] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:46 [I 2022-10-08 13:08:24] ...trap/engines/engine.py.326: process: 0.43212 | load: 0.00018 [I 2022-10-08 13:08:28] ...trap/engines/engine.py.321: eval: epoch 5 | batch 50/510 [I 2022-10-08 13:08:28] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:18 [I 2022-10-08 13:08:28] ...trap/engines/engine.py.326: process: 0.44230 | load: 0.00020 [I 2022-10-08 13:08:33] ...trap/engines/engine.py.321: eval: epoch 5 | batch 60/510 [I 2022-10-08 13:08:33] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:56 [I 2022-10-08 13:08:33] ...trap/engines/engine.py.326: process: 0.59516 | load: 0.00019 [I 2022-10-08 13:08:38] ...trap/engines/engine.py.321: eval: epoch 5 | batch 70/510 [I 2022-10-08 13:08:38] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:34 [I 2022-10-08 13:08:38] ...trap/engines/engine.py.326: process: 0.49123 | load: 0.00019 [I 2022-10-08 13:08:43] ...trap/engines/engine.py.321: eval: epoch 5 | batch 80/510 [I 2022-10-08 13:08:43] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:35 [I 2022-10-08 13:08:43] ...trap/engines/engine.py.326: process: 0.45172 | load: 0.00018 [I 2022-10-08 13:08:48] ...trap/engines/engine.py.321: eval: epoch 5 | batch 90/510 [I 2022-10-08 13:08:48] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:13 [I 2022-10-08 13:08:48] ...trap/engines/engine.py.326: process: 0.47622 | load: 0.00020 [I 2022-10-08 13:08:52] ...trap/engines/engine.py.321: eval: epoch 5 | batch 100/510 [I 2022-10-08 13:08:52] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:50 [I 2022-10-08 13:08:52] ...trap/engines/engine.py.326: process: 0.31101 | load: 0.00018 [I 2022-10-08 13:08:55] ...trap/engines/engine.py.321: eval: epoch 5 | batch 110/510 [I 2022-10-08 13:08:55] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-08 13:08:55] ...trap/engines/engine.py.326: process: 0.31316 | load: 0.00019 [I 2022-10-08 13:08:58] ...trap/engines/engine.py.321: eval: epoch 5 | batch 120/510 [I 2022-10-08 13:08:58] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:03 [I 2022-10-08 13:08:58] ...trap/engines/engine.py.326: process: 0.31864 | load: 0.00019 [I 2022-10-08 13:09:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 130/510 [I 2022-10-08 13:09:02] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:59 [I 2022-10-08 13:09:02] ...trap/engines/engine.py.326: process: 0.31177 | load: 0.00020 [I 2022-10-08 13:09:05] ...trap/engines/engine.py.321: eval: epoch 5 | batch 140/510 [I 2022-10-08 13:09:05] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:56 [I 2022-10-08 13:09:05] ...trap/engines/engine.py.326: process: 0.31714 | load: 0.00019 [I 2022-10-08 13:09:08] ...trap/engines/engine.py.321: eval: epoch 5 | batch 150/510 [I 2022-10-08 13:09:08] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:53 [I 2022-10-08 13:09:08] ...trap/engines/engine.py.326: process: 0.31822 | load: 0.00020 [I 2022-10-08 13:09:11] ...trap/engines/engine.py.321: eval: epoch 5 | batch 160/510 [I 2022-10-08 13:09:11] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:50 [I 2022-10-08 13:09:11] ...trap/engines/engine.py.326: process: 0.31334 | load: 0.00019 [I 2022-10-08 13:09:14] ...trap/engines/engine.py.321: eval: epoch 5 | batch 170/510 [I 2022-10-08 13:09:14] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-08 13:09:14] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00020 [I 2022-10-08 13:09:17] ...trap/engines/engine.py.321: eval: epoch 5 | batch 180/510 [I 2022-10-08 13:09:17] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:45 [I 2022-10-08 13:09:17] ...trap/engines/engine.py.326: process: 0.32121 | load: 0.00020 [I 2022-10-08 13:09:21] ...trap/engines/engine.py.321: eval: epoch 5 | batch 190/510 [I 2022-10-08 13:09:21] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:42 [I 2022-10-08 13:09:21] ...trap/engines/engine.py.326: process: 0.32683 | load: 0.00019 [I 2022-10-08 13:09:24] ...trap/engines/engine.py.321: eval: epoch 5 | batch 200/510 [I 2022-10-08 13:09:24] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:38 [I 2022-10-08 13:09:24] ...trap/engines/engine.py.326: process: 0.31336 | load: 0.00020 [I 2022-10-08 13:09:27] ...trap/engines/engine.py.321: eval: epoch 5 | batch 210/510 [I 2022-10-08 13:09:27] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:35 [I 2022-10-08 13:09:27] ...trap/engines/engine.py.326: process: 0.31479 | load: 0.00020 [I 2022-10-08 13:09:30] ...trap/engines/engine.py.321: eval: epoch 5 | batch 220/510 [I 2022-10-08 13:09:30] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:32 [I 2022-10-08 13:09:30] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00020 [I 2022-10-08 13:09:33] ...trap/engines/engine.py.321: eval: epoch 5 | batch 230/510 [I 2022-10-08 13:09:33] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:29 [I 2022-10-08 13:09:33] ...trap/engines/engine.py.326: process: 0.31656 | load: 0.00019 [I 2022-10-08 13:09:36] ...trap/engines/engine.py.321: eval: epoch 5 | batch 240/510 [I 2022-10-08 13:09:36] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:26 [I 2022-10-08 13:09:36] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00021 [I 2022-10-08 13:09:40] ...trap/engines/engine.py.321: eval: epoch 5 | batch 250/510 [I 2022-10-08 13:09:40] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-08 13:09:40] ...trap/engines/engine.py.326: process: 0.31287 | load: 0.00020 [I 2022-10-08 13:09:43] ...trap/engines/engine.py.321: eval: epoch 5 | batch 260/510 [I 2022-10-08 13:09:43] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:19 [I 2022-10-08 13:09:43] ...trap/engines/engine.py.326: process: 0.31965 | load: 0.00020 [I 2022-10-08 13:09:46] ...trap/engines/engine.py.321: eval: epoch 5 | batch 270/510 [I 2022-10-08 13:09:46] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:16 [I 2022-10-08 13:09:46] ...trap/engines/engine.py.326: process: 0.31510 | load: 0.00020 [I 2022-10-08 13:09:49] ...trap/engines/engine.py.321: eval: epoch 5 | batch 280/510 [I 2022-10-08 13:09:49] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-08 13:09:49] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00019 [I 2022-10-08 13:09:52] ...trap/engines/engine.py.321: eval: epoch 5 | batch 290/510 [I 2022-10-08 13:09:52] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:10 [I 2022-10-08 13:09:52] ...trap/engines/engine.py.326: process: 0.32197 | load: 0.00020 [I 2022-10-08 13:09:56] ...trap/engines/engine.py.321: eval: epoch 5 | batch 300/510 [I 2022-10-08 13:09:56] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-08 13:09:56] ...trap/engines/engine.py.326: process: 0.31040 | load: 0.00020 [I 2022-10-08 13:09:59] ...trap/engines/engine.py.321: eval: epoch 5 | batch 310/510 [I 2022-10-08 13:09:59] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-08 13:09:59] ...trap/engines/engine.py.326: process: 0.32251 | load: 0.00020 [I 2022-10-08 13:10:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 320/510 [I 2022-10-08 13:10:02] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-08 13:10:02] ...trap/engines/engine.py.326: process: 0.31972 | load: 0.00020 [I 2022-10-08 13:10:05] ...trap/engines/engine.py.321: eval: epoch 5 | batch 330/510 [I 2022-10-08 13:10:05] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-08 13:10:05] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00020 [I 2022-10-08 13:10:08] ...trap/engines/engine.py.321: eval: epoch 5 | batch 340/510 [I 2022-10-08 13:10:08] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:53 [I 2022-10-08 13:10:08] ...trap/engines/engine.py.326: process: 0.31642 | load: 0.00020 [I 2022-10-08 13:10:11] ...trap/engines/engine.py.321: eval: epoch 5 | batch 350/510 [I 2022-10-08 13:10:11] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-08 13:10:11] ...trap/engines/engine.py.326: process: 0.31645 | load: 0.00019 [I 2022-10-08 13:10:15] ...trap/engines/engine.py.321: eval: epoch 5 | batch 360/510 [I 2022-10-08 13:10:15] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-08 13:10:15] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00020 [I 2022-10-08 13:10:18] ...trap/engines/engine.py.321: eval: epoch 5 | batch 370/510 [I 2022-10-08 13:10:18] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-08 13:10:18] ...trap/engines/engine.py.326: process: 0.32342 | load: 0.00019 [I 2022-10-08 13:10:21] ...trap/engines/engine.py.321: eval: epoch 5 | batch 380/510 [I 2022-10-08 13:10:21] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-08 13:10:21] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00019 [I 2022-10-08 13:10:24] ...trap/engines/engine.py.321: eval: epoch 5 | batch 390/510 [I 2022-10-08 13:10:24] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:37 [I 2022-10-08 13:10:24] ...trap/engines/engine.py.326: process: 0.31049 | load: 0.00019 [I 2022-10-08 13:10:27] ...trap/engines/engine.py.321: eval: epoch 5 | batch 400/510 [I 2022-10-08 13:10:27] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-08 13:10:27] ...trap/engines/engine.py.326: process: 0.31898 | load: 0.00020 [I 2022-10-08 13:10:31] ...trap/engines/engine.py.321: eval: epoch 5 | batch 410/510 [I 2022-10-08 13:10:31] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-08 13:10:31] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00021 [I 2022-10-08 13:10:34] ...trap/engines/engine.py.321: eval: epoch 5 | batch 420/510 [I 2022-10-08 13:10:34] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-08 13:10:34] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00020 [I 2022-10-08 13:10:37] ...trap/engines/engine.py.321: eval: epoch 5 | batch 430/510 [I 2022-10-08 13:10:37] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-08 13:10:37] ...trap/engines/engine.py.326: process: 0.31100 | load: 0.00020 [I 2022-10-08 13:10:40] ...trap/engines/engine.py.321: eval: epoch 5 | batch 440/510 [I 2022-10-08 13:10:40] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-08 13:10:40] ...trap/engines/engine.py.326: process: 0.31723 | load: 0.00019 [I 2022-10-08 13:10:43] ...trap/engines/engine.py.321: eval: epoch 5 | batch 450/510 [I 2022-10-08 13:10:43] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:18 [I 2022-10-08 13:10:43] ...trap/engines/engine.py.326: process: 0.31264 | load: 0.00021 [I 2022-10-08 13:10:46] ...trap/engines/engine.py.321: eval: epoch 5 | batch 460/510 [I 2022-10-08 13:10:46] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-08 13:10:46] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00020 [I 2022-10-08 13:10:50] ...trap/engines/engine.py.321: eval: epoch 5 | batch 470/510 [I 2022-10-08 13:10:50] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-08 13:10:50] ...trap/engines/engine.py.326: process: 0.32770 | load: 0.00020 [I 2022-10-08 13:10:53] ...trap/engines/engine.py.321: eval: epoch 5 | batch 480/510 [I 2022-10-08 13:10:53] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-08 13:10:53] ...trap/engines/engine.py.326: process: 0.31710 | load: 0.00019 [I 2022-10-08 13:10:56] ...trap/engines/engine.py.321: eval: epoch 5 | batch 490/510 [I 2022-10-08 13:10:56] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-08 13:10:56] ...trap/engines/engine.py.326: process: 0.31936 | load: 0.00020 [I 2022-10-08 13:10:59] ...trap/engines/engine.py.321: eval: epoch 5 | batch 500/510 [I 2022-10-08 13:10:59] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-08 13:10:59] ...trap/engines/engine.py.326: process: 0.31609 | load: 0.00019 [I 2022-10-08 13:11:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 510/510 [I 2022-10-08 13:11:02] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-08 13:11:02] ...trap/engines/engine.py.326: process: 0.31952 | load: 0.00016 [S 2022-10-08 13:11:03] ...trap/engines/engine.py.132: eval_epoch.epoch: 5 [I 2022-10-08 13:11:03] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.559800 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.836200 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.897600 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.579100 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.842800 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.899600 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010294 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008998 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008139 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010578 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008364 [S 2022-10-08 13:11:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009992 [I 2022-10-08 13:11:34] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 13:11:34] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 13:11:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 13:11:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 13:11:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 13:11:34] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 13:11:37] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 13:11:41] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 13:11:41] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 13:11:41] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 13:11:44] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 13:11:49] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 13:11:49] ...trap/engines/engine.py.163: Training model on trainset for epoch 6 [I 2022-10-08 13:11:49] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 13:11:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 0/2304 [I 2022-10-08 13:11:52] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:53:00 [I 2022-10-08 13:11:52] ...trap/engines/engine.py.229: process: 2.94304 | load: 2.09056 [I 2022-10-08 13:11:52] ...trap/engines/engine.py.230: loss: 0.22032 [I 2022-10-08 13:11:52] ...optimizers/trijoint.py.151: total_norm: 2.883037 [I 2022-10-08 13:11:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 10/2304 [I 2022-10-08 13:11:59] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:45 [I 2022-10-08 13:11:59] ...trap/engines/engine.py.229: process: 0.78482 | load: 0.00448 [I 2022-10-08 13:11:59] ...trap/engines/engine.py.230: loss: 0.21536 [I 2022-10-08 13:11:59] ...optimizers/trijoint.py.151: total_norm: 2.823451 [I 2022-10-08 13:12:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 20/2304 [I 2022-10-08 13:12:07] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:30 [I 2022-10-08 13:12:07] ...trap/engines/engine.py.229: process: 0.78124 | load: 0.00453 [I 2022-10-08 13:12:07] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-08 13:12:07] ...optimizers/trijoint.py.151: total_norm: 2.080115 [I 2022-10-08 13:12:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 30/2304 [I 2022-10-08 13:12:15] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:28 [I 2022-10-08 13:12:15] ...trap/engines/engine.py.229: process: 0.75710 | load: 0.00388 [I 2022-10-08 13:12:15] ...trap/engines/engine.py.230: loss: 0.35575 [I 2022-10-08 13:12:15] ...optimizers/trijoint.py.151: total_norm: 3.229606 [I 2022-10-08 13:12:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 40/2304 [I 2022-10-08 13:12:23] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:14 [I 2022-10-08 13:12:23] ...trap/engines/engine.py.229: process: 0.75943 | load: 0.00390 [I 2022-10-08 13:12:23] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-08 13:12:23] ...optimizers/trijoint.py.151: total_norm: 2.150019 [I 2022-10-08 13:12:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 50/2304 [I 2022-10-08 13:12:31] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:21 [I 2022-10-08 13:12:31] ...trap/engines/engine.py.229: process: 0.78181 | load: 0.00377 [I 2022-10-08 13:12:31] ...trap/engines/engine.py.230: loss: 0.22962 [I 2022-10-08 13:12:31] ...optimizers/trijoint.py.151: total_norm: 2.968677 [I 2022-10-08 13:12:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 60/2304 [I 2022-10-08 13:12:38] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:07 [I 2022-10-08 13:12:38] ...trap/engines/engine.py.229: process: 0.77073 | load: 0.00376 [I 2022-10-08 13:12:38] ...trap/engines/engine.py.230: loss: 0.25694 [I 2022-10-08 13:12:38] ...optimizers/trijoint.py.151: total_norm: 3.578256 [I 2022-10-08 13:12:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 70/2304 [I 2022-10-08 13:12:46] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:59 [I 2022-10-08 13:12:46] ...trap/engines/engine.py.229: process: 0.78302 | load: 0.00370 [I 2022-10-08 13:12:46] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-08 13:12:46] ...optimizers/trijoint.py.151: total_norm: 2.123395 [I 2022-10-08 13:12:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 80/2304 [I 2022-10-08 13:12:54] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:54 [I 2022-10-08 13:12:54] ...trap/engines/engine.py.229: process: 0.77018 | load: 0.00343 [I 2022-10-08 13:12:54] ...trap/engines/engine.py.230: loss: 0.22087 [I 2022-10-08 13:12:54] ...optimizers/trijoint.py.151: total_norm: 2.856495 [I 2022-10-08 13:13:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 90/2304 [I 2022-10-08 13:13:02] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:49 [I 2022-10-08 13:13:02] ...trap/engines/engine.py.229: process: 0.78197 | load: 0.00347 [I 2022-10-08 13:13:02] ...trap/engines/engine.py.230: loss: 0.22339 [I 2022-10-08 13:13:02] ...optimizers/trijoint.py.151: total_norm: 3.528958 [I 2022-10-08 13:13:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 100/2304 [I 2022-10-08 13:13:10] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:37 [I 2022-10-08 13:13:10] ...trap/engines/engine.py.229: process: 0.77971 | load: 0.00333 [I 2022-10-08 13:13:10] ...trap/engines/engine.py.230: loss: 0.27686 [I 2022-10-08 13:13:10] ...optimizers/trijoint.py.151: total_norm: 2.856220 [I 2022-10-08 13:13:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 110/2304 [I 2022-10-08 13:13:17] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:28 [I 2022-10-08 13:13:17] ...trap/engines/engine.py.229: process: 0.77733 | load: 0.00335 [I 2022-10-08 13:13:17] ...trap/engines/engine.py.230: loss: 0.17981 [I 2022-10-08 13:13:17] ...optimizers/trijoint.py.151: total_norm: 2.848937 [I 2022-10-08 13:13:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 120/2304 [I 2022-10-08 13:13:25] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:22 [I 2022-10-08 13:13:25] ...trap/engines/engine.py.229: process: 0.78501 | load: 0.00329 [I 2022-10-08 13:13:25] ...trap/engines/engine.py.230: loss: 0.20738 [I 2022-10-08 13:13:25] ...optimizers/trijoint.py.151: total_norm: 2.466196 [I 2022-10-08 13:13:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 130/2304 [I 2022-10-08 13:13:33] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:19 [I 2022-10-08 13:13:33] ...trap/engines/engine.py.229: process: 0.78862 | load: 0.00372 [I 2022-10-08 13:13:33] ...trap/engines/engine.py.230: loss: 0.29941 [I 2022-10-08 13:13:33] ...optimizers/trijoint.py.151: total_norm: 3.756682 [I 2022-10-08 13:13:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 140/2304 [I 2022-10-08 13:13:41] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:09 [I 2022-10-08 13:13:41] ...trap/engines/engine.py.229: process: 0.77194 | load: 0.00314 [I 2022-10-08 13:13:41] ...trap/engines/engine.py.230: loss: 0.22397 [I 2022-10-08 13:13:41] ...optimizers/trijoint.py.151: total_norm: 3.065948 [I 2022-10-08 13:13:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 150/2304 [I 2022-10-08 13:13:49] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:59 [I 2022-10-08 13:13:49] ...trap/engines/engine.py.229: process: 0.78296 | load: 0.00318 [I 2022-10-08 13:13:49] ...trap/engines/engine.py.230: loss: 0.19420 [I 2022-10-08 13:13:49] ...optimizers/trijoint.py.151: total_norm: 2.159066 [I 2022-10-08 13:13:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 160/2304 [I 2022-10-08 13:13:56] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:44 [I 2022-10-08 13:13:56] ...trap/engines/engine.py.229: process: 0.77339 | load: 0.00338 [I 2022-10-08 13:13:56] ...trap/engines/engine.py.230: loss: 0.21985 [I 2022-10-08 13:13:56] ...optimizers/trijoint.py.151: total_norm: 3.007426 [I 2022-10-08 13:14:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 170/2304 [I 2022-10-08 13:14:04] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:43 [I 2022-10-08 13:14:04] ...trap/engines/engine.py.229: process: 0.78450 | load: 0.00313 [I 2022-10-08 13:14:04] ...trap/engines/engine.py.230: loss: 0.25252 [I 2022-10-08 13:14:04] ...optimizers/trijoint.py.151: total_norm: 3.008440 [I 2022-10-08 13:14:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 180/2304 [I 2022-10-08 13:14:12] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:29 [I 2022-10-08 13:14:12] ...trap/engines/engine.py.229: process: 0.77284 | load: 0.00302 [I 2022-10-08 13:14:12] ...trap/engines/engine.py.230: loss: 0.23691 [I 2022-10-08 13:14:12] ...optimizers/trijoint.py.151: total_norm: 3.062398 [I 2022-10-08 13:14:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 190/2304 [I 2022-10-08 13:14:20] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:25 [I 2022-10-08 13:14:20] ...trap/engines/engine.py.229: process: 0.78275 | load: 0.00323 [I 2022-10-08 13:14:20] ...trap/engines/engine.py.230: loss: 0.28917 [I 2022-10-08 13:14:20] ...optimizers/trijoint.py.151: total_norm: 2.527205 [I 2022-10-08 13:14:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 200/2304 [I 2022-10-08 13:14:28] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:26 [I 2022-10-08 13:14:28] ...trap/engines/engine.py.229: process: 0.78421 | load: 0.00310 [I 2022-10-08 13:14:28] ...trap/engines/engine.py.230: loss: 0.19795 [I 2022-10-08 13:14:28] ...optimizers/trijoint.py.151: total_norm: 1.795958 [I 2022-10-08 13:14:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 210/2304 [I 2022-10-08 13:14:35] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:18 [I 2022-10-08 13:14:35] ...trap/engines/engine.py.229: process: 0.78948 | load: 0.00303 [I 2022-10-08 13:14:35] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-08 13:14:35] ...optimizers/trijoint.py.151: total_norm: 2.528824 [I 2022-10-08 13:14:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 220/2304 [I 2022-10-08 13:14:43] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:10 [I 2022-10-08 13:14:43] ...trap/engines/engine.py.229: process: 0.78032 | load: 0.00307 [I 2022-10-08 13:14:43] ...trap/engines/engine.py.230: loss: 0.25835 [I 2022-10-08 13:14:43] ...optimizers/trijoint.py.151: total_norm: 3.492371 [I 2022-10-08 13:14:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 230/2304 [I 2022-10-08 13:14:51] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:53 [I 2022-10-08 13:14:51] ...trap/engines/engine.py.229: process: 0.78017 | load: 0.00299 [I 2022-10-08 13:14:51] ...trap/engines/engine.py.230: loss: 0.23946 [I 2022-10-08 13:14:51] ...optimizers/trijoint.py.151: total_norm: 2.509453 [I 2022-10-08 13:14:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 240/2304 [I 2022-10-08 13:14:59] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:47 [I 2022-10-08 13:14:59] ...trap/engines/engine.py.229: process: 0.77489 | load: 0.00301 [I 2022-10-08 13:14:59] ...trap/engines/engine.py.230: loss: 0.17247 [I 2022-10-08 13:14:59] ...optimizers/trijoint.py.151: total_norm: 2.443980 [I 2022-10-08 13:15:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 250/2304 [I 2022-10-08 13:15:07] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:38 [I 2022-10-08 13:15:07] ...trap/engines/engine.py.229: process: 0.77730 | load: 0.00304 [I 2022-10-08 13:15:07] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-08 13:15:07] ...optimizers/trijoint.py.151: total_norm: 1.945125 [I 2022-10-08 13:15:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 260/2304 [I 2022-10-08 13:15:15] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:40 [I 2022-10-08 13:15:15] ...trap/engines/engine.py.229: process: 0.77991 | load: 0.00299 [I 2022-10-08 13:15:15] ...trap/engines/engine.py.230: loss: 0.14507 [I 2022-10-08 13:15:15] ...optimizers/trijoint.py.151: total_norm: 2.520946 [I 2022-10-08 13:15:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 270/2304 [I 2022-10-08 13:15:40] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 2:29:04 [I 2022-10-08 13:15:40] ...trap/engines/engine.py.229: process: 18.86716 | load: 0.00304 [I 2022-10-08 13:15:40] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-08 13:15:40] ...optimizers/trijoint.py.151: total_norm: 2.195396 [I 2022-10-08 13:15:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 280/2304 [I 2022-10-08 13:15:48] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:39:28 [I 2022-10-08 13:15:48] ...trap/engines/engine.py.229: process: 0.78051 | load: 0.00318 [I 2022-10-08 13:15:48] ...trap/engines/engine.py.230: loss: 0.20623 [I 2022-10-08 13:15:48] ...optimizers/trijoint.py.151: total_norm: 2.007741 [I 2022-10-08 13:15:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 290/2304 [I 2022-10-08 13:15:56] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:27:26 [I 2022-10-08 13:15:56] ...trap/engines/engine.py.229: process: 0.77197 | load: 0.00314 [I 2022-10-08 13:15:56] ...trap/engines/engine.py.230: loss: 0.16045 [I 2022-10-08 13:15:56] ...optimizers/trijoint.py.151: total_norm: 1.846335 [I 2022-10-08 13:16:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 300/2304 [I 2022-10-08 13:16:04] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:26:08 [I 2022-10-08 13:16:04] ...trap/engines/engine.py.229: process: 0.77881 | load: 0.00304 [I 2022-10-08 13:16:04] ...trap/engines/engine.py.230: loss: 0.19288 [I 2022-10-08 13:16:04] ...optimizers/trijoint.py.151: total_norm: 2.831637 [I 2022-10-08 13:16:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 310/2304 [I 2022-10-08 13:16:12] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:25:56 [I 2022-10-08 13:16:12] ...trap/engines/engine.py.229: process: 0.79160 | load: 0.00301 [I 2022-10-08 13:16:12] ...trap/engines/engine.py.230: loss: 0.21881 [I 2022-10-08 13:16:12] ...optimizers/trijoint.py.151: total_norm: 3.316270 [I 2022-10-08 13:16:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 320/2304 [I 2022-10-08 13:16:19] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:25:33 [I 2022-10-08 13:16:19] ...trap/engines/engine.py.229: process: 0.76668 | load: 0.00305 [I 2022-10-08 13:16:19] ...trap/engines/engine.py.230: loss: 0.20251 [I 2022-10-08 13:16:19] ...optimizers/trijoint.py.151: total_norm: 3.997156 [I 2022-10-08 13:16:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 330/2304 [I 2022-10-08 13:16:27] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:25:30 [I 2022-10-08 13:16:27] ...trap/engines/engine.py.229: process: 0.77419 | load: 0.00312 [I 2022-10-08 13:16:27] ...trap/engines/engine.py.230: loss: 0.24885 [I 2022-10-08 13:16:27] ...optimizers/trijoint.py.151: total_norm: 2.949939 [I 2022-10-08 13:16:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 340/2304 [I 2022-10-08 13:16:35] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:25:23 [I 2022-10-08 13:16:35] ...trap/engines/engine.py.229: process: 0.78213 | load: 0.00302 [I 2022-10-08 13:16:35] ...trap/engines/engine.py.230: loss: 0.20510 [I 2022-10-08 13:16:35] ...optimizers/trijoint.py.151: total_norm: 2.856423 [I 2022-10-08 13:16:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 350/2304 [I 2022-10-08 13:16:43] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:25:25 [I 2022-10-08 13:16:43] ...trap/engines/engine.py.229: process: 0.78682 | load: 0.00307 [I 2022-10-08 13:16:43] ...trap/engines/engine.py.230: loss: 0.20127 [I 2022-10-08 13:16:43] ...optimizers/trijoint.py.151: total_norm: 1.886944 [I 2022-10-08 13:16:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 360/2304 [I 2022-10-08 13:16:51] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:25:07 [I 2022-10-08 13:16:51] ...trap/engines/engine.py.229: process: 0.78270 | load: 0.00312 [I 2022-10-08 13:16:51] ...trap/engines/engine.py.230: loss: 0.25466 [I 2022-10-08 13:16:51] ...optimizers/trijoint.py.151: total_norm: 3.346184 [I 2022-10-08 13:16:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 370/2304 [I 2022-10-08 13:16:58] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:25:01 [I 2022-10-08 13:16:58] ...trap/engines/engine.py.229: process: 0.77140 | load: 0.00315 [I 2022-10-08 13:16:58] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-08 13:16:58] ...optimizers/trijoint.py.151: total_norm: 1.895434 [I 2022-10-08 13:17:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 380/2304 [I 2022-10-08 13:17:06] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:24:53 [I 2022-10-08 13:17:06] ...trap/engines/engine.py.229: process: 0.77930 | load: 0.00301 [I 2022-10-08 13:17:06] ...trap/engines/engine.py.230: loss: 0.20289 [I 2022-10-08 13:17:06] ...optimizers/trijoint.py.151: total_norm: 3.210767 [I 2022-10-08 13:17:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 390/2304 [I 2022-10-08 13:17:14] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:24:41 [I 2022-10-08 13:17:14] ...trap/engines/engine.py.229: process: 0.77562 | load: 0.00313 [I 2022-10-08 13:17:14] ...trap/engines/engine.py.230: loss: 0.23724 [I 2022-10-08 13:17:14] ...optimizers/trijoint.py.151: total_norm: 2.933695 [I 2022-10-08 13:17:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 400/2304 [I 2022-10-08 13:17:22] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:24:40 [I 2022-10-08 13:17:22] ...trap/engines/engine.py.229: process: 0.78510 | load: 0.00305 [I 2022-10-08 13:17:22] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-08 13:17:22] ...optimizers/trijoint.py.151: total_norm: 2.596802 [I 2022-10-08 13:17:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 410/2304 [I 2022-10-08 13:17:29] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:24:36 [I 2022-10-08 13:17:29] ...trap/engines/engine.py.229: process: 0.77671 | load: 0.00303 [I 2022-10-08 13:17:29] ...trap/engines/engine.py.230: loss: 0.20922 [I 2022-10-08 13:17:29] ...optimizers/trijoint.py.151: total_norm: 2.602292 [I 2022-10-08 13:17:37] ...trap/engines/engine.py.224: train: epoch 6 | batch 420/2304 [I 2022-10-08 13:17:37] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:24:31 [I 2022-10-08 13:17:37] ...trap/engines/engine.py.229: process: 0.79005 | load: 0.00300 [I 2022-10-08 13:17:37] ...trap/engines/engine.py.230: loss: 0.20431 [I 2022-10-08 13:17:37] ...optimizers/trijoint.py.151: total_norm: 2.182785 [I 2022-10-08 13:17:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 430/2304 [I 2022-10-08 13:17:45] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:24:25 [I 2022-10-08 13:17:45] ...trap/engines/engine.py.229: process: 0.78340 | load: 0.00297 [I 2022-10-08 13:17:45] ...trap/engines/engine.py.230: loss: 0.22590 [I 2022-10-08 13:17:45] ...optimizers/trijoint.py.151: total_norm: 3.245780 [I 2022-10-08 13:17:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 440/2304 [I 2022-10-08 13:17:53] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:24:13 [I 2022-10-08 13:17:53] ...trap/engines/engine.py.229: process: 0.77428 | load: 0.00298 [I 2022-10-08 13:17:53] ...trap/engines/engine.py.230: loss: 0.23407 [I 2022-10-08 13:17:53] ...optimizers/trijoint.py.151: total_norm: 2.051324 [I 2022-10-08 13:18:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 450/2304 [I 2022-10-08 13:18:01] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:24:02 [I 2022-10-08 13:18:01] ...trap/engines/engine.py.229: process: 0.77699 | load: 0.00302 [I 2022-10-08 13:18:01] ...trap/engines/engine.py.230: loss: 0.12022 [I 2022-10-08 13:18:01] ...optimizers/trijoint.py.151: total_norm: 1.881939 [I 2022-10-08 13:18:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 460/2304 [I 2022-10-08 13:18:09] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:23:58 [I 2022-10-08 13:18:09] ...trap/engines/engine.py.229: process: 0.77568 | load: 0.00304 [I 2022-10-08 13:18:09] ...trap/engines/engine.py.230: loss: 0.29370 [I 2022-10-08 13:18:09] ...optimizers/trijoint.py.151: total_norm: 3.504753 [I 2022-10-08 13:18:16] ...trap/engines/engine.py.224: train: epoch 6 | batch 470/2304 [I 2022-10-08 13:18:16] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:23:40 [I 2022-10-08 13:18:16] ...trap/engines/engine.py.229: process: 0.76936 | load: 0.00300 [I 2022-10-08 13:18:16] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-08 13:18:16] ...optimizers/trijoint.py.151: total_norm: 2.274975 [I 2022-10-08 13:18:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 480/2304 [I 2022-10-08 13:18:24] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:23:34 [I 2022-10-08 13:18:24] ...trap/engines/engine.py.229: process: 0.77672 | load: 0.00304 [I 2022-10-08 13:18:24] ...trap/engines/engine.py.230: loss: 0.22566 [I 2022-10-08 13:18:24] ...optimizers/trijoint.py.151: total_norm: 2.456131 [I 2022-10-08 13:18:32] ...trap/engines/engine.py.224: train: epoch 6 | batch 490/2304 [I 2022-10-08 13:18:32] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:23:34 [I 2022-10-08 13:18:32] ...trap/engines/engine.py.229: process: 0.79461 | load: 0.00299 [I 2022-10-08 13:18:32] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-08 13:18:32] ...optimizers/trijoint.py.151: total_norm: 2.632872 [I 2022-10-08 13:18:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 500/2304 [I 2022-10-08 13:18:40] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:23:27 [I 2022-10-08 13:18:40] ...trap/engines/engine.py.229: process: 0.78254 | load: 0.00302 [I 2022-10-08 13:18:40] ...trap/engines/engine.py.230: loss: 0.23024 [I 2022-10-08 13:18:40] ...optimizers/trijoint.py.151: total_norm: 3.303539 [I 2022-10-08 13:18:47] ...trap/engines/engine.py.224: train: epoch 6 | batch 510/2304 [I 2022-10-08 13:18:47] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:23:18 [I 2022-10-08 13:18:47] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00303 [I 2022-10-08 13:18:47] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-08 13:18:47] ...optimizers/trijoint.py.151: total_norm: 2.064988 [I 2022-10-08 13:18:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 520/2304 [I 2022-10-08 13:18:55] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:23:04 [I 2022-10-08 13:18:55] ...trap/engines/engine.py.229: process: 0.76589 | load: 0.00305 [I 2022-10-08 13:18:55] ...trap/engines/engine.py.230: loss: 0.29206 [I 2022-10-08 13:18:55] ...optimizers/trijoint.py.151: total_norm: 3.948632 [I 2022-10-08 13:19:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 530/2304 [I 2022-10-08 13:19:03] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:23:05 [I 2022-10-08 13:19:03] ...trap/engines/engine.py.229: process: 0.78583 | load: 0.00301 [I 2022-10-08 13:19:03] ...trap/engines/engine.py.230: loss: 0.23403 [I 2022-10-08 13:19:03] ...optimizers/trijoint.py.151: total_norm: 3.740291 [I 2022-10-08 13:19:11] ...trap/engines/engine.py.224: train: epoch 6 | batch 540/2304 [I 2022-10-08 13:19:11] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:22:51 [I 2022-10-08 13:19:11] ...trap/engines/engine.py.229: process: 0.78789 | load: 0.00303 [I 2022-10-08 13:19:11] ...trap/engines/engine.py.230: loss: 0.21075 [I 2022-10-08 13:19:11] ...optimizers/trijoint.py.151: total_norm: 2.495507 [I 2022-10-08 13:19:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 550/2304 [I 2022-10-08 13:19:19] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:22:46 [I 2022-10-08 13:19:19] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00302 [I 2022-10-08 13:19:19] ...trap/engines/engine.py.230: loss: 0.30255 [I 2022-10-08 13:19:19] ...optimizers/trijoint.py.151: total_norm: 3.487701 [I 2022-10-08 13:19:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 560/2304 [I 2022-10-08 13:19:27] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:22:48 [I 2022-10-08 13:19:27] ...trap/engines/engine.py.229: process: 0.78632 | load: 0.00305 [I 2022-10-08 13:19:27] ...trap/engines/engine.py.230: loss: 0.23093 [I 2022-10-08 13:19:27] ...optimizers/trijoint.py.151: total_norm: 2.591240 [I 2022-10-08 13:19:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 570/2304 [I 2022-10-08 13:19:34] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:22:31 [I 2022-10-08 13:19:34] ...trap/engines/engine.py.229: process: 0.78283 | load: 0.00305 [I 2022-10-08 13:19:34] ...trap/engines/engine.py.230: loss: 0.31041 [I 2022-10-08 13:19:34] ...optimizers/trijoint.py.151: total_norm: 3.227582 [I 2022-10-08 13:19:42] ...trap/engines/engine.py.224: train: epoch 6 | batch 580/2304 [I 2022-10-08 13:19:42] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:22:14 [I 2022-10-08 13:19:42] ...trap/engines/engine.py.229: process: 0.78137 | load: 0.00302 [I 2022-10-08 13:19:42] ...trap/engines/engine.py.230: loss: 0.22257 [I 2022-10-08 13:19:42] ...optimizers/trijoint.py.151: total_norm: 2.801355 [I 2022-10-08 13:19:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 590/2304 [I 2022-10-08 13:19:50] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:22:11 [I 2022-10-08 13:19:50] ...trap/engines/engine.py.229: process: 0.77288 | load: 0.00301 [I 2022-10-08 13:19:50] ...trap/engines/engine.py.230: loss: 0.28533 [I 2022-10-08 13:19:50] ...optimizers/trijoint.py.151: total_norm: 2.253049 [I 2022-10-08 13:19:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 600/2304 [I 2022-10-08 13:19:58] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:22:07 [I 2022-10-08 13:19:58] ...trap/engines/engine.py.229: process: 0.78497 | load: 0.00303 [I 2022-10-08 13:19:58] ...trap/engines/engine.py.230: loss: 0.22454 [I 2022-10-08 13:19:58] ...optimizers/trijoint.py.151: total_norm: 1.933900 [I 2022-10-08 13:20:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 610/2304 [I 2022-10-08 13:20:06] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:22:01 [I 2022-10-08 13:20:06] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00304 [I 2022-10-08 13:20:06] ...trap/engines/engine.py.230: loss: 0.28157 [I 2022-10-08 13:20:06] ...optimizers/trijoint.py.151: total_norm: 2.557469 [I 2022-10-08 13:20:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 620/2304 [I 2022-10-08 13:20:13] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:21:41 [I 2022-10-08 13:20:13] ...trap/engines/engine.py.229: process: 0.78129 | load: 0.00301 [I 2022-10-08 13:20:13] ...trap/engines/engine.py.230: loss: 0.20131 [I 2022-10-08 13:20:13] ...optimizers/trijoint.py.151: total_norm: 2.083314 [I 2022-10-08 13:20:21] ...trap/engines/engine.py.224: train: epoch 6 | batch 630/2304 [I 2022-10-08 13:20:21] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:21:39 [I 2022-10-08 13:20:21] ...trap/engines/engine.py.229: process: 0.78731 | load: 0.00307 [I 2022-10-08 13:20:21] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-08 13:20:21] ...optimizers/trijoint.py.151: total_norm: 2.749585 [I 2022-10-08 13:20:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 640/2304 [I 2022-10-08 13:20:29] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:21:41 [I 2022-10-08 13:20:29] ...trap/engines/engine.py.229: process: 0.77874 | load: 0.00302 [I 2022-10-08 13:20:29] ...trap/engines/engine.py.230: loss: 0.23195 [I 2022-10-08 13:20:29] ...optimizers/trijoint.py.151: total_norm: 3.548992 [I 2022-10-08 13:20:37] ...trap/engines/engine.py.224: train: epoch 6 | batch 650/2304 [I 2022-10-08 13:20:37] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:21:32 [I 2022-10-08 13:20:37] ...trap/engines/engine.py.229: process: 0.79305 | load: 0.00304 [I 2022-10-08 13:20:37] ...trap/engines/engine.py.230: loss: 0.24741 [I 2022-10-08 13:20:37] ...optimizers/trijoint.py.151: total_norm: 3.159639 [I 2022-10-08 13:20:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 660/2304 [I 2022-10-08 13:20:45] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:21:26 [I 2022-10-08 13:20:45] ...trap/engines/engine.py.229: process: 0.78816 | load: 0.00306 [I 2022-10-08 13:20:45] ...trap/engines/engine.py.230: loss: 0.25462 [I 2022-10-08 13:20:45] ...optimizers/trijoint.py.151: total_norm: 2.537833 [I 2022-10-08 13:20:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 670/2304 [I 2022-10-08 13:20:52] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:21:16 [I 2022-10-08 13:20:52] ...trap/engines/engine.py.229: process: 0.77205 | load: 0.00303 [I 2022-10-08 13:20:52] ...trap/engines/engine.py.230: loss: 0.20792 [I 2022-10-08 13:20:52] ...optimizers/trijoint.py.151: total_norm: 2.528730 [I 2022-10-08 13:21:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 680/2304 [I 2022-10-08 13:21:00] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:21:08 [I 2022-10-08 13:21:00] ...trap/engines/engine.py.229: process: 0.77991 | load: 0.00307 [I 2022-10-08 13:21:00] ...trap/engines/engine.py.230: loss: 0.21961 [I 2022-10-08 13:21:00] ...optimizers/trijoint.py.151: total_norm: 3.800058 [I 2022-10-08 13:21:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 690/2304 [I 2022-10-08 13:21:08] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:20:54 [I 2022-10-08 13:21:08] ...trap/engines/engine.py.229: process: 0.77002 | load: 0.00300 [I 2022-10-08 13:21:08] ...trap/engines/engine.py.230: loss: 0.17456 [I 2022-10-08 13:21:08] ...optimizers/trijoint.py.151: total_norm: 2.407053 [I 2022-10-08 13:21:16] ...trap/engines/engine.py.224: train: epoch 6 | batch 700/2304 [I 2022-10-08 13:21:16] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:20:46 [I 2022-10-08 13:21:16] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00301 [I 2022-10-08 13:21:16] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-08 13:21:16] ...optimizers/trijoint.py.151: total_norm: 2.056616 [I 2022-10-08 13:21:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 710/2304 [I 2022-10-08 13:21:24] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:20:35 [I 2022-10-08 13:21:24] ...trap/engines/engine.py.229: process: 0.77244 | load: 0.00307 [I 2022-10-08 13:21:24] ...trap/engines/engine.py.230: loss: 0.22140 [I 2022-10-08 13:21:24] ...optimizers/trijoint.py.151: total_norm: 3.410430 [I 2022-10-08 13:21:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 720/2304 [I 2022-10-08 13:21:31] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:20:29 [I 2022-10-08 13:21:31] ...trap/engines/engine.py.229: process: 0.78282 | load: 0.00305 [I 2022-10-08 13:21:31] ...trap/engines/engine.py.230: loss: 0.28196 [I 2022-10-08 13:21:31] ...optimizers/trijoint.py.151: total_norm: 3.260165 [I 2022-10-08 13:21:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 730/2304 [I 2022-10-08 13:21:39] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:20:27 [I 2022-10-08 13:21:39] ...trap/engines/engine.py.229: process: 0.77843 | load: 0.00303 [I 2022-10-08 13:21:39] ...trap/engines/engine.py.230: loss: 0.17354 [I 2022-10-08 13:21:39] ...optimizers/trijoint.py.151: total_norm: 2.442558 [I 2022-10-08 13:21:47] ...trap/engines/engine.py.224: train: epoch 6 | batch 740/2304 [I 2022-10-08 13:21:47] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:20:16 [I 2022-10-08 13:21:47] ...trap/engines/engine.py.229: process: 0.77571 | load: 0.00304 [I 2022-10-08 13:21:47] ...trap/engines/engine.py.230: loss: 0.19131 [I 2022-10-08 13:21:47] ...optimizers/trijoint.py.151: total_norm: 2.798373 [I 2022-10-08 13:21:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 750/2304 [I 2022-10-08 13:21:55] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:20:14 [I 2022-10-08 13:21:55] ...trap/engines/engine.py.229: process: 0.77883 | load: 0.00301 [I 2022-10-08 13:21:55] ...trap/engines/engine.py.230: loss: 0.18590 [I 2022-10-08 13:21:55] ...optimizers/trijoint.py.151: total_norm: 2.379880 [I 2022-10-08 13:22:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 760/2304 [I 2022-10-08 13:22:03] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:20:10 [I 2022-10-08 13:22:03] ...trap/engines/engine.py.229: process: 0.78841 | load: 0.00303 [I 2022-10-08 13:22:03] ...trap/engines/engine.py.230: loss: 0.26588 [I 2022-10-08 13:22:03] ...optimizers/trijoint.py.151: total_norm: 2.313066 [I 2022-10-08 13:22:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 770/2304 [I 2022-10-08 13:22:10] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:19:49 [I 2022-10-08 13:22:10] ...trap/engines/engine.py.229: process: 0.77279 | load: 0.00307 [I 2022-10-08 13:22:10] ...trap/engines/engine.py.230: loss: 0.21897 [I 2022-10-08 13:22:10] ...optimizers/trijoint.py.151: total_norm: 2.630109 [I 2022-10-08 13:22:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 780/2304 [I 2022-10-08 13:22:18] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:19:44 [I 2022-10-08 13:22:18] ...trap/engines/engine.py.229: process: 0.76602 | load: 0.00309 [I 2022-10-08 13:22:18] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-08 13:22:18] ...optimizers/trijoint.py.151: total_norm: 2.676711 [I 2022-10-08 13:22:26] ...trap/engines/engine.py.224: train: epoch 6 | batch 790/2304 [I 2022-10-08 13:22:26] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:19:43 [I 2022-10-08 13:22:26] ...trap/engines/engine.py.229: process: 0.78994 | load: 0.00310 [I 2022-10-08 13:22:26] ...trap/engines/engine.py.230: loss: 0.16535 [I 2022-10-08 13:22:26] ...optimizers/trijoint.py.151: total_norm: 2.325120 [I 2022-10-08 13:22:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 800/2304 [I 2022-10-08 13:22:34] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:19:26 [I 2022-10-08 13:22:34] ...trap/engines/engine.py.229: process: 0.77507 | load: 0.00311 [I 2022-10-08 13:22:34] ...trap/engines/engine.py.230: loss: 0.17115 [I 2022-10-08 13:22:34] ...optimizers/trijoint.py.151: total_norm: 3.029405 [I 2022-10-08 13:22:42] ...trap/engines/engine.py.224: train: epoch 6 | batch 810/2304 [I 2022-10-08 13:22:42] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:19:25 [I 2022-10-08 13:22:42] ...trap/engines/engine.py.229: process: 0.78287 | load: 0.00316 [I 2022-10-08 13:22:42] ...trap/engines/engine.py.230: loss: 0.20307 [I 2022-10-08 13:22:42] ...optimizers/trijoint.py.151: total_norm: 3.276150 [I 2022-10-08 13:22:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 820/2304 [I 2022-10-08 13:22:49] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:19:18 [I 2022-10-08 13:22:49] ...trap/engines/engine.py.229: process: 0.78360 | load: 0.00309 [I 2022-10-08 13:22:49] ...trap/engines/engine.py.230: loss: 0.21447 [I 2022-10-08 13:22:49] ...optimizers/trijoint.py.151: total_norm: 2.530220 [I 2022-10-08 13:22:57] ...trap/engines/engine.py.224: train: epoch 6 | batch 830/2304 [I 2022-10-08 13:22:57] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:19:06 [I 2022-10-08 13:22:57] ...trap/engines/engine.py.229: process: 0.76732 | load: 0.00316 [I 2022-10-08 13:22:57] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-08 13:22:57] ...optimizers/trijoint.py.151: total_norm: 2.381799 [I 2022-10-08 13:23:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 840/2304 [I 2022-10-08 13:23:05] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:18:59 [I 2022-10-08 13:23:05] ...trap/engines/engine.py.229: process: 0.77040 | load: 0.00307 [I 2022-10-08 13:23:05] ...trap/engines/engine.py.230: loss: 0.22652 [I 2022-10-08 13:23:05] ...optimizers/trijoint.py.151: total_norm: 2.916021 [I 2022-10-08 13:23:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 850/2304 [I 2022-10-08 13:23:13] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:18:48 [I 2022-10-08 13:23:13] ...trap/engines/engine.py.229: process: 0.78144 | load: 0.00316 [I 2022-10-08 13:23:13] ...trap/engines/engine.py.230: loss: 0.22952 [I 2022-10-08 13:23:13] ...optimizers/trijoint.py.151: total_norm: 2.717211 [I 2022-10-08 13:23:21] ...trap/engines/engine.py.224: train: epoch 6 | batch 860/2304 [I 2022-10-08 13:23:21] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:18:39 [I 2022-10-08 13:23:21] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00303 [I 2022-10-08 13:23:21] ...trap/engines/engine.py.230: loss: 0.27244 [I 2022-10-08 13:23:21] ...optimizers/trijoint.py.151: total_norm: 2.143636 [I 2022-10-08 13:23:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 870/2304 [I 2022-10-08 13:23:28] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:18:37 [I 2022-10-08 13:23:28] ...trap/engines/engine.py.229: process: 0.77049 | load: 0.00302 [I 2022-10-08 13:23:28] ...trap/engines/engine.py.230: loss: 0.22507 [I 2022-10-08 13:23:28] ...optimizers/trijoint.py.151: total_norm: 2.927223 [I 2022-10-08 13:23:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 880/2304 [I 2022-10-08 13:23:36] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:18:37 [I 2022-10-08 13:23:36] ...trap/engines/engine.py.229: process: 0.78903 | load: 0.00303 [I 2022-10-08 13:23:36] ...trap/engines/engine.py.230: loss: 0.12897 [I 2022-10-08 13:23:36] ...optimizers/trijoint.py.151: total_norm: 2.028201 [I 2022-10-08 13:23:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 890/2304 [I 2022-10-08 13:23:44] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:18:19 [I 2022-10-08 13:23:44] ...trap/engines/engine.py.229: process: 0.78203 | load: 0.00303 [I 2022-10-08 13:23:44] ...trap/engines/engine.py.230: loss: 0.18407 [I 2022-10-08 13:23:44] ...optimizers/trijoint.py.151: total_norm: 2.062069 [I 2022-10-08 13:23:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 900/2304 [I 2022-10-08 13:23:52] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:18:11 [I 2022-10-08 13:23:52] ...trap/engines/engine.py.229: process: 0.77321 | load: 0.00304 [I 2022-10-08 13:23:52] ...trap/engines/engine.py.230: loss: 0.21751 [I 2022-10-08 13:23:52] ...optimizers/trijoint.py.151: total_norm: 2.859020 [I 2022-10-08 13:24:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 910/2304 [I 2022-10-08 13:24:00] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:18:06 [I 2022-10-08 13:24:00] ...trap/engines/engine.py.229: process: 0.77844 | load: 0.00306 [I 2022-10-08 13:24:00] ...trap/engines/engine.py.230: loss: 0.22156 [I 2022-10-08 13:24:00] ...optimizers/trijoint.py.151: total_norm: 2.622629 [I 2022-10-08 13:24:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 920/2304 [I 2022-10-08 13:24:08] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:18:03 [I 2022-10-08 13:24:08] ...trap/engines/engine.py.229: process: 0.78983 | load: 0.00306 [I 2022-10-08 13:24:08] ...trap/engines/engine.py.230: loss: 0.21296 [I 2022-10-08 13:24:08] ...optimizers/trijoint.py.151: total_norm: 2.625955 [I 2022-10-08 13:24:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 930/2304 [I 2022-10-08 13:24:15] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:17:53 [I 2022-10-08 13:24:15] ...trap/engines/engine.py.229: process: 0.78872 | load: 0.00299 [I 2022-10-08 13:24:15] ...trap/engines/engine.py.230: loss: 0.22943 [I 2022-10-08 13:24:15] ...optimizers/trijoint.py.151: total_norm: 3.428202 [I 2022-10-08 13:24:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 940/2304 [I 2022-10-08 13:24:23] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:17:48 [I 2022-10-08 13:24:23] ...trap/engines/engine.py.229: process: 0.77489 | load: 0.00306 [I 2022-10-08 13:24:23] ...trap/engines/engine.py.230: loss: 0.22706 [I 2022-10-08 13:24:23] ...optimizers/trijoint.py.151: total_norm: 2.377298 [I 2022-10-08 13:24:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 950/2304 [I 2022-10-08 13:24:31] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:17:40 [I 2022-10-08 13:24:31] ...trap/engines/engine.py.229: process: 0.78547 | load: 0.00302 [I 2022-10-08 13:24:31] ...trap/engines/engine.py.230: loss: 0.27085 [I 2022-10-08 13:24:31] ...optimizers/trijoint.py.151: total_norm: 3.173689 [I 2022-10-08 13:24:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 960/2304 [I 2022-10-08 13:24:39] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:17:27 [I 2022-10-08 13:24:39] ...trap/engines/engine.py.229: process: 0.76992 | load: 0.00301 [I 2022-10-08 13:24:39] ...trap/engines/engine.py.230: loss: 0.14831 [I 2022-10-08 13:24:39] ...optimizers/trijoint.py.151: total_norm: 2.012957 [I 2022-10-08 13:24:47] ...trap/engines/engine.py.224: train: epoch 6 | batch 970/2304 [I 2022-10-08 13:24:47] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:17:16 [I 2022-10-08 13:24:47] ...trap/engines/engine.py.229: process: 0.77569 | load: 0.00305 [I 2022-10-08 13:24:47] ...trap/engines/engine.py.230: loss: 0.18836 [I 2022-10-08 13:24:47] ...optimizers/trijoint.py.151: total_norm: 2.471277 [I 2022-10-08 13:24:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 980/2304 [I 2022-10-08 13:24:54] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:17:10 [I 2022-10-08 13:24:54] ...trap/engines/engine.py.229: process: 0.78469 | load: 0.00304 [I 2022-10-08 13:24:54] ...trap/engines/engine.py.230: loss: 0.17897 [I 2022-10-08 13:24:55] ...optimizers/trijoint.py.151: total_norm: 2.528127 [I 2022-10-08 13:25:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 990/2304 [I 2022-10-08 13:25:02] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:16:58 [I 2022-10-08 13:25:02] ...trap/engines/engine.py.229: process: 0.76437 | load: 0.00312 [I 2022-10-08 13:25:02] ...trap/engines/engine.py.230: loss: 0.29834 [I 2022-10-08 13:25:02] ...optimizers/trijoint.py.151: total_norm: 2.913675 [I 2022-10-08 13:25:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 1000/2304 [I 2022-10-08 13:25:10] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:16:56 [I 2022-10-08 13:25:10] ...trap/engines/engine.py.229: process: 0.78526 | load: 0.00300 [I 2022-10-08 13:25:10] ...trap/engines/engine.py.230: loss: 0.20977 [I 2022-10-08 13:25:10] ...optimizers/trijoint.py.151: total_norm: 3.061871 [I 2022-10-08 13:25:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 1010/2304 [I 2022-10-08 13:25:18] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:16:45 [I 2022-10-08 13:25:18] ...trap/engines/engine.py.229: process: 0.77063 | load: 0.00303 [I 2022-10-08 13:25:18] ...trap/engines/engine.py.230: loss: 0.20177 [I 2022-10-08 13:25:18] ...optimizers/trijoint.py.151: total_norm: 3.316749 [I 2022-10-08 13:25:26] ...trap/engines/engine.py.224: train: epoch 6 | batch 1020/2304 [I 2022-10-08 13:25:26] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:16:36 [I 2022-10-08 13:25:26] ...trap/engines/engine.py.229: process: 0.77505 | load: 0.00302 [I 2022-10-08 13:25:26] ...trap/engines/engine.py.230: loss: 0.19814 [I 2022-10-08 13:25:26] ...optimizers/trijoint.py.151: total_norm: 2.698378 [I 2022-10-08 13:25:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 1030/2304 [I 2022-10-08 13:25:34] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:16:30 [I 2022-10-08 13:25:34] ...trap/engines/engine.py.229: process: 0.76898 | load: 0.00301 [I 2022-10-08 13:25:34] ...trap/engines/engine.py.230: loss: 0.28692 [I 2022-10-08 13:25:34] ...optimizers/trijoint.py.151: total_norm: 3.300818 [I 2022-10-08 13:25:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 1040/2304 [I 2022-10-08 13:25:41] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:16:29 [I 2022-10-08 13:25:41] ...trap/engines/engine.py.229: process: 0.78943 | load: 0.00303 [I 2022-10-08 13:25:41] ...trap/engines/engine.py.230: loss: 0.21551 [I 2022-10-08 13:25:41] ...optimizers/trijoint.py.151: total_norm: 2.990003 [I 2022-10-08 13:25:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 1050/2304 [I 2022-10-08 13:25:49] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:16:20 [I 2022-10-08 13:25:49] ...trap/engines/engine.py.229: process: 0.77792 | load: 0.00299 [I 2022-10-08 13:25:49] ...trap/engines/engine.py.230: loss: 0.17196 [I 2022-10-08 13:25:49] ...optimizers/trijoint.py.151: total_norm: 2.052859 [I 2022-10-08 13:25:57] ...trap/engines/engine.py.224: train: epoch 6 | batch 1060/2304 [I 2022-10-08 13:25:57] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:16:13 [I 2022-10-08 13:25:57] ...trap/engines/engine.py.229: process: 0.78214 | load: 0.00300 [I 2022-10-08 13:25:57] ...trap/engines/engine.py.230: loss: 0.21540 [I 2022-10-08 13:25:57] ...optimizers/trijoint.py.151: total_norm: 2.664618 [I 2022-10-08 13:26:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 1070/2304 [I 2022-10-08 13:26:05] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:16:02 [I 2022-10-08 13:26:05] ...trap/engines/engine.py.229: process: 0.79425 | load: 0.00304 [I 2022-10-08 13:26:05] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-08 13:26:05] ...optimizers/trijoint.py.151: total_norm: 2.207578 [I 2022-10-08 13:26:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 1080/2304 [I 2022-10-08 13:26:13] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:15:48 [I 2022-10-08 13:26:13] ...trap/engines/engine.py.229: process: 0.76561 | load: 0.00302 [I 2022-10-08 13:26:13] ...trap/engines/engine.py.230: loss: 0.23457 [I 2022-10-08 13:26:13] ...optimizers/trijoint.py.151: total_norm: 2.641285 [I 2022-10-08 13:26:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1090/2304 [I 2022-10-08 13:26:20] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:15:48 [I 2022-10-08 13:26:20] ...trap/engines/engine.py.229: process: 0.79111 | load: 0.00303 [I 2022-10-08 13:26:20] ...trap/engines/engine.py.230: loss: 0.28758 [I 2022-10-08 13:26:20] ...optimizers/trijoint.py.151: total_norm: 3.241107 [I 2022-10-08 13:26:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 1100/2304 [I 2022-10-08 13:26:28] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:15:38 [I 2022-10-08 13:26:28] ...trap/engines/engine.py.229: process: 0.78410 | load: 0.00302 [I 2022-10-08 13:26:28] ...trap/engines/engine.py.230: loss: 0.19061 [I 2022-10-08 13:26:28] ...optimizers/trijoint.py.151: total_norm: 2.205909 [I 2022-10-08 13:26:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 1110/2304 [I 2022-10-08 13:26:36] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:15:30 [I 2022-10-08 13:26:36] ...trap/engines/engine.py.229: process: 0.78077 | load: 0.00302 [I 2022-10-08 13:26:36] ...trap/engines/engine.py.230: loss: 0.23194 [I 2022-10-08 13:26:36] ...optimizers/trijoint.py.151: total_norm: 3.068761 [I 2022-10-08 13:26:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 1120/2304 [I 2022-10-08 13:26:44] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:15:22 [I 2022-10-08 13:26:44] ...trap/engines/engine.py.229: process: 0.78161 | load: 0.00308 [I 2022-10-08 13:26:44] ...trap/engines/engine.py.230: loss: 0.21666 [I 2022-10-08 13:26:44] ...optimizers/trijoint.py.151: total_norm: 2.930486 [I 2022-10-08 13:26:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 1130/2304 [I 2022-10-08 13:26:52] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:15:13 [I 2022-10-08 13:26:52] ...trap/engines/engine.py.229: process: 0.77452 | load: 0.00298 [I 2022-10-08 13:26:52] ...trap/engines/engine.py.230: loss: 0.23063 [I 2022-10-08 13:26:52] ...optimizers/trijoint.py.151: total_norm: 2.891016 [I 2022-10-08 13:26:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 1140/2304 [I 2022-10-08 13:26:59] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:15:04 [I 2022-10-08 13:26:59] ...trap/engines/engine.py.229: process: 0.78066 | load: 0.00303 [I 2022-10-08 13:26:59] ...trap/engines/engine.py.230: loss: 0.20785 [I 2022-10-08 13:26:59] ...optimizers/trijoint.py.151: total_norm: 2.238591 [I 2022-10-08 13:27:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 1150/2304 [I 2022-10-08 13:27:07] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:15:03 [I 2022-10-08 13:27:07] ...trap/engines/engine.py.229: process: 0.76949 | load: 0.00303 [I 2022-10-08 13:27:07] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-08 13:27:07] ...optimizers/trijoint.py.151: total_norm: 2.423500 [I 2022-10-08 13:27:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1160/2304 [I 2022-10-08 13:27:15] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:14:55 [I 2022-10-08 13:27:15] ...trap/engines/engine.py.229: process: 0.78853 | load: 0.00300 [I 2022-10-08 13:27:15] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-08 13:27:15] ...optimizers/trijoint.py.151: total_norm: 1.610137 [I 2022-10-08 13:27:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 1170/2304 [I 2022-10-08 13:27:23] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:14:44 [I 2022-10-08 13:27:23] ...trap/engines/engine.py.229: process: 0.77600 | load: 0.00303 [I 2022-10-08 13:27:23] ...trap/engines/engine.py.230: loss: 0.17516 [I 2022-10-08 13:27:23] ...optimizers/trijoint.py.151: total_norm: 2.243980 [I 2022-10-08 13:27:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 1180/2304 [I 2022-10-08 13:27:31] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:14:33 [I 2022-10-08 13:27:31] ...trap/engines/engine.py.229: process: 0.78316 | load: 0.00305 [I 2022-10-08 13:27:31] ...trap/engines/engine.py.230: loss: 0.22465 [I 2022-10-08 13:27:31] ...optimizers/trijoint.py.151: total_norm: 2.542444 [I 2022-10-08 13:27:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 1190/2304 [I 2022-10-08 13:27:39] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:14:26 [I 2022-10-08 13:27:39] ...trap/engines/engine.py.229: process: 0.77792 | load: 0.00305 [I 2022-10-08 13:27:39] ...trap/engines/engine.py.230: loss: 0.22831 [I 2022-10-08 13:27:39] ...optimizers/trijoint.py.151: total_norm: 3.272712 [I 2022-10-08 13:27:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 1200/2304 [I 2022-10-08 13:27:46] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:14:17 [I 2022-10-08 13:27:46] ...trap/engines/engine.py.229: process: 0.78043 | load: 0.00304 [I 2022-10-08 13:27:46] ...trap/engines/engine.py.230: loss: 0.22062 [I 2022-10-08 13:27:46] ...optimizers/trijoint.py.151: total_norm: 2.398546 [I 2022-10-08 13:27:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 1210/2304 [I 2022-10-08 13:27:54] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:14:12 [I 2022-10-08 13:27:54] ...trap/engines/engine.py.229: process: 0.78383 | load: 0.00301 [I 2022-10-08 13:27:54] ...trap/engines/engine.py.230: loss: 0.17896 [I 2022-10-08 13:27:54] ...optimizers/trijoint.py.151: total_norm: 3.278525 [I 2022-10-08 13:28:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 1220/2304 [I 2022-10-08 13:28:02] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:14:02 [I 2022-10-08 13:28:02] ...trap/engines/engine.py.229: process: 0.77307 | load: 0.00305 [I 2022-10-08 13:28:02] ...trap/engines/engine.py.230: loss: 0.24294 [I 2022-10-08 13:28:02] ...optimizers/trijoint.py.151: total_norm: 2.878672 [I 2022-10-08 13:28:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 1230/2304 [I 2022-10-08 13:28:10] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:13:57 [I 2022-10-08 13:28:10] ...trap/engines/engine.py.229: process: 0.77128 | load: 0.00303 [I 2022-10-08 13:28:10] ...trap/engines/engine.py.230: loss: 0.23976 [I 2022-10-08 13:28:10] ...optimizers/trijoint.py.151: total_norm: 2.633642 [I 2022-10-08 13:28:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 1240/2304 [I 2022-10-08 13:28:18] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:13:45 [I 2022-10-08 13:28:18] ...trap/engines/engine.py.229: process: 0.76816 | load: 0.00307 [I 2022-10-08 13:28:18] ...trap/engines/engine.py.230: loss: 0.14313 [I 2022-10-08 13:28:18] ...optimizers/trijoint.py.151: total_norm: 2.124065 [I 2022-10-08 13:28:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1250/2304 [I 2022-10-08 13:28:25] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:13:40 [I 2022-10-08 13:28:25] ...trap/engines/engine.py.229: process: 0.77587 | load: 0.00302 [I 2022-10-08 13:28:25] ...trap/engines/engine.py.230: loss: 0.23914 [I 2022-10-08 13:28:25] ...optimizers/trijoint.py.151: total_norm: 2.609607 [I 2022-10-08 13:28:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 1260/2304 [I 2022-10-08 13:28:33] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:13:39 [I 2022-10-08 13:28:33] ...trap/engines/engine.py.229: process: 0.79169 | load: 0.00300 [I 2022-10-08 13:28:33] ...trap/engines/engine.py.230: loss: 0.20752 [I 2022-10-08 13:28:33] ...optimizers/trijoint.py.151: total_norm: 2.131321 [I 2022-10-08 13:28:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 1270/2304 [I 2022-10-08 13:28:41] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:13:26 [I 2022-10-08 13:28:41] ...trap/engines/engine.py.229: process: 0.78087 | load: 0.00300 [I 2022-10-08 13:28:41] ...trap/engines/engine.py.230: loss: 0.19035 [I 2022-10-08 13:28:41] ...optimizers/trijoint.py.151: total_norm: 2.114819 [I 2022-10-08 13:28:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 1280/2304 [I 2022-10-08 13:28:49] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:13:19 [I 2022-10-08 13:28:49] ...trap/engines/engine.py.229: process: 0.78324 | load: 0.00301 [I 2022-10-08 13:28:49] ...trap/engines/engine.py.230: loss: 0.25559 [I 2022-10-08 13:28:49] ...optimizers/trijoint.py.151: total_norm: 2.648518 [I 2022-10-08 13:28:57] ...trap/engines/engine.py.224: train: epoch 6 | batch 1290/2304 [I 2022-10-08 13:28:57] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:13:07 [I 2022-10-08 13:28:57] ...trap/engines/engine.py.229: process: 0.77964 | load: 0.00301 [I 2022-10-08 13:28:57] ...trap/engines/engine.py.230: loss: 0.23091 [I 2022-10-08 13:28:57] ...optimizers/trijoint.py.151: total_norm: 2.548466 [I 2022-10-08 13:29:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1300/2304 [I 2022-10-08 13:29:04] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:13:02 [I 2022-10-08 13:29:04] ...trap/engines/engine.py.229: process: 0.78040 | load: 0.00299 [I 2022-10-08 13:29:04] ...trap/engines/engine.py.230: loss: 0.23780 [I 2022-10-08 13:29:04] ...optimizers/trijoint.py.151: total_norm: 2.985284 [I 2022-10-08 13:29:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 1310/2304 [I 2022-10-08 13:29:12] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:12:52 [I 2022-10-08 13:29:12] ...trap/engines/engine.py.229: process: 0.77701 | load: 0.00300 [I 2022-10-08 13:29:12] ...trap/engines/engine.py.230: loss: 0.14934 [I 2022-10-08 13:29:12] ...optimizers/trijoint.py.151: total_norm: 2.234469 [I 2022-10-08 13:29:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1320/2304 [I 2022-10-08 13:29:20] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:12:45 [I 2022-10-08 13:29:20] ...trap/engines/engine.py.229: process: 0.77985 | load: 0.00301 [I 2022-10-08 13:29:20] ...trap/engines/engine.py.230: loss: 0.25379 [I 2022-10-08 13:29:20] ...optimizers/trijoint.py.151: total_norm: 2.535925 [I 2022-10-08 13:29:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 1330/2304 [I 2022-10-08 13:29:28] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:12:41 [I 2022-10-08 13:29:28] ...trap/engines/engine.py.229: process: 0.76990 | load: 0.00305 [I 2022-10-08 13:29:28] ...trap/engines/engine.py.230: loss: 0.22037 [I 2022-10-08 13:29:28] ...optimizers/trijoint.py.151: total_norm: 2.644432 [I 2022-10-08 13:29:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 1340/2304 [I 2022-10-08 13:29:36] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:12:39 [I 2022-10-08 13:29:36] ...trap/engines/engine.py.229: process: 0.80701 | load: 0.00299 [I 2022-10-08 13:29:36] ...trap/engines/engine.py.230: loss: 0.22281 [I 2022-10-08 13:29:36] ...optimizers/trijoint.py.151: total_norm: 2.957378 [I 2022-10-08 13:29:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 1350/2304 [I 2022-10-08 13:29:44] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:12:30 [I 2022-10-08 13:29:44] ...trap/engines/engine.py.229: process: 0.79860 | load: 0.00305 [I 2022-10-08 13:29:44] ...trap/engines/engine.py.230: loss: 0.16412 [I 2022-10-08 13:29:44] ...optimizers/trijoint.py.151: total_norm: 2.888649 [I 2022-10-08 13:29:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 1360/2304 [I 2022-10-08 13:29:51] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:12:18 [I 2022-10-08 13:29:51] ...trap/engines/engine.py.229: process: 0.76817 | load: 0.00304 [I 2022-10-08 13:29:51] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-08 13:29:51] ...optimizers/trijoint.py.151: total_norm: 2.157663 [I 2022-10-08 13:29:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 1370/2304 [I 2022-10-08 13:29:59] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:12:09 [I 2022-10-08 13:29:59] ...trap/engines/engine.py.229: process: 0.78015 | load: 0.00303 [I 2022-10-08 13:29:59] ...trap/engines/engine.py.230: loss: 0.26938 [I 2022-10-08 13:29:59] ...optimizers/trijoint.py.151: total_norm: 3.113674 [I 2022-10-08 13:30:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 1380/2304 [I 2022-10-08 13:30:07] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:12:00 [I 2022-10-08 13:30:07] ...trap/engines/engine.py.229: process: 0.78124 | load: 0.00308 [I 2022-10-08 13:30:07] ...trap/engines/engine.py.230: loss: 0.22541 [I 2022-10-08 13:30:07] ...optimizers/trijoint.py.151: total_norm: 3.167185 [I 2022-10-08 13:30:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1390/2304 [I 2022-10-08 13:30:15] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:11:50 [I 2022-10-08 13:30:15] ...trap/engines/engine.py.229: process: 0.77478 | load: 0.00303 [I 2022-10-08 13:30:15] ...trap/engines/engine.py.230: loss: 0.16067 [I 2022-10-08 13:30:15] ...optimizers/trijoint.py.151: total_norm: 2.294300 [I 2022-10-08 13:30:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 1400/2304 [I 2022-10-08 13:30:23] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:11:44 [I 2022-10-08 13:30:23] ...trap/engines/engine.py.229: process: 0.77531 | load: 0.00302 [I 2022-10-08 13:30:23] ...trap/engines/engine.py.230: loss: 0.27196 [I 2022-10-08 13:30:23] ...optimizers/trijoint.py.151: total_norm: 2.986495 [I 2022-10-08 13:30:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1410/2304 [I 2022-10-08 13:30:30] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:11:35 [I 2022-10-08 13:30:30] ...trap/engines/engine.py.229: process: 0.78239 | load: 0.00302 [I 2022-10-08 13:30:30] ...trap/engines/engine.py.230: loss: 0.15446 [I 2022-10-08 13:30:30] ...optimizers/trijoint.py.151: total_norm: 2.620088 [I 2022-10-08 13:30:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 1420/2304 [I 2022-10-08 13:30:38] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:11:24 [I 2022-10-08 13:30:38] ...trap/engines/engine.py.229: process: 0.76289 | load: 0.00297 [I 2022-10-08 13:30:38] ...trap/engines/engine.py.230: loss: 0.18122 [I 2022-10-08 13:30:38] ...optimizers/trijoint.py.151: total_norm: 2.581755 [I 2022-10-08 13:30:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 1430/2304 [I 2022-10-08 13:30:46] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:11:20 [I 2022-10-08 13:30:46] ...trap/engines/engine.py.229: process: 0.78055 | load: 0.00306 [I 2022-10-08 13:30:46] ...trap/engines/engine.py.230: loss: 0.22743 [I 2022-10-08 13:30:46] ...optimizers/trijoint.py.151: total_norm: 3.508830 [I 2022-10-08 13:30:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 1440/2304 [I 2022-10-08 13:30:54] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:11:14 [I 2022-10-08 13:30:54] ...trap/engines/engine.py.229: process: 0.78370 | load: 0.00304 [I 2022-10-08 13:30:54] ...trap/engines/engine.py.230: loss: 0.18993 [I 2022-10-08 13:30:54] ...optimizers/trijoint.py.151: total_norm: 2.036706 [I 2022-10-08 13:31:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 1450/2304 [I 2022-10-08 13:31:02] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:11:03 [I 2022-10-08 13:31:02] ...trap/engines/engine.py.229: process: 0.77725 | load: 0.00302 [I 2022-10-08 13:31:02] ...trap/engines/engine.py.230: loss: 0.27240 [I 2022-10-08 13:31:02] ...optimizers/trijoint.py.151: total_norm: 2.410503 [I 2022-10-08 13:31:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 1460/2304 [I 2022-10-08 13:31:09] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:10:57 [I 2022-10-08 13:31:09] ...trap/engines/engine.py.229: process: 0.78275 | load: 0.00299 [I 2022-10-08 13:31:09] ...trap/engines/engine.py.230: loss: 0.21332 [I 2022-10-08 13:31:09] ...optimizers/trijoint.py.151: total_norm: 2.743187 [I 2022-10-08 13:31:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 1470/2304 [I 2022-10-08 13:31:17] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:10:53 [I 2022-10-08 13:31:17] ...trap/engines/engine.py.229: process: 0.78222 | load: 0.00305 [I 2022-10-08 13:31:17] ...trap/engines/engine.py.230: loss: 0.22857 [I 2022-10-08 13:31:17] ...optimizers/trijoint.py.151: total_norm: 2.414805 [I 2022-10-08 13:31:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1480/2304 [I 2022-10-08 13:31:25] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:10:45 [I 2022-10-08 13:31:25] ...trap/engines/engine.py.229: process: 0.77867 | load: 0.00306 [I 2022-10-08 13:31:25] ...trap/engines/engine.py.230: loss: 0.13429 [I 2022-10-08 13:31:25] ...optimizers/trijoint.py.151: total_norm: 1.815547 [I 2022-10-08 13:31:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 1490/2304 [I 2022-10-08 13:31:33] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:10:36 [I 2022-10-08 13:31:33] ...trap/engines/engine.py.229: process: 0.79241 | load: 0.00305 [I 2022-10-08 13:31:33] ...trap/engines/engine.py.230: loss: 0.18603 [I 2022-10-08 13:31:33] ...optimizers/trijoint.py.151: total_norm: 1.861022 [I 2022-10-08 13:31:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 1500/2304 [I 2022-10-08 13:31:41] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:10:25 [I 2022-10-08 13:31:41] ...trap/engines/engine.py.229: process: 0.77670 | load: 0.00303 [I 2022-10-08 13:31:41] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-08 13:31:41] ...optimizers/trijoint.py.151: total_norm: 2.042858 [I 2022-10-08 13:31:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 1510/2304 [I 2022-10-08 13:31:49] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:10:17 [I 2022-10-08 13:31:49] ...trap/engines/engine.py.229: process: 0.76717 | load: 0.00308 [I 2022-10-08 13:31:49] ...trap/engines/engine.py.230: loss: 0.17646 [I 2022-10-08 13:31:49] ...optimizers/trijoint.py.151: total_norm: 1.912777 [I 2022-10-08 13:31:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1520/2304 [I 2022-10-08 13:31:56] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:10:11 [I 2022-10-08 13:31:56] ...trap/engines/engine.py.229: process: 0.78494 | load: 0.00313 [I 2022-10-08 13:31:56] ...trap/engines/engine.py.230: loss: 0.20824 [I 2022-10-08 13:31:56] ...optimizers/trijoint.py.151: total_norm: 3.403589 [I 2022-10-08 13:32:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1530/2304 [I 2022-10-08 13:32:04] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:10:01 [I 2022-10-08 13:32:04] ...trap/engines/engine.py.229: process: 0.78447 | load: 0.00303 [I 2022-10-08 13:32:04] ...trap/engines/engine.py.230: loss: 0.34457 [I 2022-10-08 13:32:04] ...optimizers/trijoint.py.151: total_norm: 3.731453 [I 2022-10-08 13:32:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 1540/2304 [I 2022-10-08 13:32:12] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:09:57 [I 2022-10-08 13:32:12] ...trap/engines/engine.py.229: process: 0.78249 | load: 0.00311 [I 2022-10-08 13:32:12] ...trap/engines/engine.py.230: loss: 0.16555 [I 2022-10-08 13:32:12] ...optimizers/trijoint.py.151: total_norm: 2.524875 [I 2022-10-08 13:32:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1550/2304 [I 2022-10-08 13:32:20] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:09:49 [I 2022-10-08 13:32:20] ...trap/engines/engine.py.229: process: 0.78878 | load: 0.00305 [I 2022-10-08 13:32:20] ...trap/engines/engine.py.230: loss: 0.21445 [I 2022-10-08 13:32:20] ...optimizers/trijoint.py.151: total_norm: 2.618377 [I 2022-10-08 13:32:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 1560/2304 [I 2022-10-08 13:32:28] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:09:40 [I 2022-10-08 13:32:28] ...trap/engines/engine.py.229: process: 0.77306 | load: 0.00305 [I 2022-10-08 13:32:28] ...trap/engines/engine.py.230: loss: 0.20185 [I 2022-10-08 13:32:28] ...optimizers/trijoint.py.151: total_norm: 2.519683 [I 2022-10-08 13:32:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 1570/2304 [I 2022-10-08 13:32:36] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:09:33 [I 2022-10-08 13:32:36] ...trap/engines/engine.py.229: process: 0.77439 | load: 0.00301 [I 2022-10-08 13:32:36] ...trap/engines/engine.py.230: loss: 0.19130 [I 2022-10-08 13:32:36] ...optimizers/trijoint.py.151: total_norm: 2.546239 [I 2022-10-08 13:32:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 1580/2304 [I 2022-10-08 13:32:43] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:09:26 [I 2022-10-08 13:32:43] ...trap/engines/engine.py.229: process: 0.77975 | load: 0.00304 [I 2022-10-08 13:32:43] ...trap/engines/engine.py.230: loss: 0.23668 [I 2022-10-08 13:32:43] ...optimizers/trijoint.py.151: total_norm: 2.593897 [I 2022-10-08 13:32:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 1590/2304 [I 2022-10-08 13:32:51] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:09:13 [I 2022-10-08 13:32:51] ...trap/engines/engine.py.229: process: 0.76582 | load: 0.00310 [I 2022-10-08 13:32:51] ...trap/engines/engine.py.230: loss: 0.25182 [I 2022-10-08 13:32:51] ...optimizers/trijoint.py.151: total_norm: 3.073900 [I 2022-10-08 13:32:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 1600/2304 [I 2022-10-08 13:32:59] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:09:10 [I 2022-10-08 13:32:59] ...trap/engines/engine.py.229: process: 0.78886 | load: 0.00307 [I 2022-10-08 13:32:59] ...trap/engines/engine.py.230: loss: 0.22875 [I 2022-10-08 13:32:59] ...optimizers/trijoint.py.151: total_norm: 2.636002 [I 2022-10-08 13:33:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 1610/2304 [I 2022-10-08 13:33:07] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:09:00 [I 2022-10-08 13:33:07] ...trap/engines/engine.py.229: process: 0.78467 | load: 0.00305 [I 2022-10-08 13:33:07] ...trap/engines/engine.py.230: loss: 0.18132 [I 2022-10-08 13:33:07] ...optimizers/trijoint.py.151: total_norm: 2.412533 [I 2022-10-08 13:33:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1620/2304 [I 2022-10-08 13:33:15] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:08:49 [I 2022-10-08 13:33:15] ...trap/engines/engine.py.229: process: 0.77681 | load: 0.00302 [I 2022-10-08 13:33:15] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-08 13:33:15] ...optimizers/trijoint.py.151: total_norm: 2.404200 [I 2022-10-08 13:33:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 1630/2304 [I 2022-10-08 13:33:22] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:08:45 [I 2022-10-08 13:33:22] ...trap/engines/engine.py.229: process: 0.77835 | load: 0.00304 [I 2022-10-08 13:33:22] ...trap/engines/engine.py.230: loss: 0.15924 [I 2022-10-08 13:33:22] ...optimizers/trijoint.py.151: total_norm: 2.723668 [I 2022-10-08 13:33:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1640/2304 [I 2022-10-08 13:33:30] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:08:36 [I 2022-10-08 13:33:30] ...trap/engines/engine.py.229: process: 0.78196 | load: 0.00305 [I 2022-10-08 13:33:30] ...trap/engines/engine.py.230: loss: 0.26012 [I 2022-10-08 13:33:30] ...optimizers/trijoint.py.151: total_norm: 2.912480 [I 2022-10-08 13:33:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 1650/2304 [I 2022-10-08 13:33:38] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:08:29 [I 2022-10-08 13:33:38] ...trap/engines/engine.py.229: process: 0.76738 | load: 0.00301 [I 2022-10-08 13:33:38] ...trap/engines/engine.py.230: loss: 0.31261 [I 2022-10-08 13:33:38] ...optimizers/trijoint.py.151: total_norm: 3.416469 [I 2022-10-08 13:33:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 1660/2304 [I 2022-10-08 13:33:46] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:08:21 [I 2022-10-08 13:33:46] ...trap/engines/engine.py.229: process: 0.77785 | load: 0.00303 [I 2022-10-08 13:33:46] ...trap/engines/engine.py.230: loss: 0.25876 [I 2022-10-08 13:33:46] ...optimizers/trijoint.py.151: total_norm: 3.024962 [I 2022-10-08 13:33:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 1670/2304 [I 2022-10-08 13:33:54] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:08:14 [I 2022-10-08 13:33:54] ...trap/engines/engine.py.229: process: 0.78142 | load: 0.00300 [I 2022-10-08 13:33:54] ...trap/engines/engine.py.230: loss: 0.25101 [I 2022-10-08 13:33:54] ...optimizers/trijoint.py.151: total_norm: 3.201205 [I 2022-10-08 13:34:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 1680/2304 [I 2022-10-08 13:34:01] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:08:05 [I 2022-10-08 13:34:01] ...trap/engines/engine.py.229: process: 0.78100 | load: 0.00301 [I 2022-10-08 13:34:01] ...trap/engines/engine.py.230: loss: 0.23586 [I 2022-10-08 13:34:01] ...optimizers/trijoint.py.151: total_norm: 3.026428 [I 2022-10-08 13:34:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 1690/2304 [I 2022-10-08 13:34:09] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:07:58 [I 2022-10-08 13:34:09] ...trap/engines/engine.py.229: process: 0.78253 | load: 0.00301 [I 2022-10-08 13:34:09] ...trap/engines/engine.py.230: loss: 0.20611 [I 2022-10-08 13:34:09] ...optimizers/trijoint.py.151: total_norm: 2.955091 [I 2022-10-08 13:34:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 1700/2304 [I 2022-10-08 13:34:17] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:07:53 [I 2022-10-08 13:34:17] ...trap/engines/engine.py.229: process: 0.77751 | load: 0.00309 [I 2022-10-08 13:34:17] ...trap/engines/engine.py.230: loss: 0.25394 [I 2022-10-08 13:34:17] ...optimizers/trijoint.py.151: total_norm: 2.639896 [I 2022-10-08 13:34:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1710/2304 [I 2022-10-08 13:34:25] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:07:39 [I 2022-10-08 13:34:25] ...trap/engines/engine.py.229: process: 0.77054 | load: 0.00301 [I 2022-10-08 13:34:25] ...trap/engines/engine.py.230: loss: 0.24766 [I 2022-10-08 13:34:25] ...optimizers/trijoint.py.151: total_norm: 2.754387 [I 2022-10-08 13:34:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 1720/2304 [I 2022-10-08 13:34:33] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:07:34 [I 2022-10-08 13:34:33] ...trap/engines/engine.py.229: process: 0.78654 | load: 0.00305 [I 2022-10-08 13:34:33] ...trap/engines/engine.py.230: loss: 0.22845 [I 2022-10-08 13:34:33] ...optimizers/trijoint.py.151: total_norm: 2.550840 [I 2022-10-08 13:34:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 1730/2304 [I 2022-10-08 13:34:40] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:07:26 [I 2022-10-08 13:34:40] ...trap/engines/engine.py.229: process: 0.78164 | load: 0.00303 [I 2022-10-08 13:34:40] ...trap/engines/engine.py.230: loss: 0.24204 [I 2022-10-08 13:34:40] ...optimizers/trijoint.py.151: total_norm: 3.234115 [I 2022-10-08 13:34:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 1740/2304 [I 2022-10-08 13:34:48] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:07:17 [I 2022-10-08 13:34:48] ...trap/engines/engine.py.229: process: 0.77574 | load: 0.00305 [I 2022-10-08 13:34:48] ...trap/engines/engine.py.230: loss: 0.29667 [I 2022-10-08 13:34:48] ...optimizers/trijoint.py.151: total_norm: 3.221137 [I 2022-10-08 13:34:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1750/2304 [I 2022-10-08 13:34:56] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:07:13 [I 2022-10-08 13:34:56] ...trap/engines/engine.py.229: process: 0.79158 | load: 0.00306 [I 2022-10-08 13:34:56] ...trap/engines/engine.py.230: loss: 0.23128 [I 2022-10-08 13:34:56] ...optimizers/trijoint.py.151: total_norm: 3.035022 [I 2022-10-08 13:35:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1760/2304 [I 2022-10-08 13:35:04] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:07:03 [I 2022-10-08 13:35:04] ...trap/engines/engine.py.229: process: 0.77262 | load: 0.00302 [I 2022-10-08 13:35:04] ...trap/engines/engine.py.230: loss: 0.19417 [I 2022-10-08 13:35:04] ...optimizers/trijoint.py.151: total_norm: 2.542545 [I 2022-10-08 13:35:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 1770/2304 [I 2022-10-08 13:35:12] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:06:56 [I 2022-10-08 13:35:12] ...trap/engines/engine.py.229: process: 0.78507 | load: 0.00304 [I 2022-10-08 13:35:12] ...trap/engines/engine.py.230: loss: 0.21477 [I 2022-10-08 13:35:12] ...optimizers/trijoint.py.151: total_norm: 2.620228 [I 2022-10-08 13:35:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 1780/2304 [I 2022-10-08 13:35:19] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:06:48 [I 2022-10-08 13:35:19] ...trap/engines/engine.py.229: process: 0.78084 | load: 0.00303 [I 2022-10-08 13:35:19] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-08 13:35:19] ...optimizers/trijoint.py.151: total_norm: 2.095823 [I 2022-10-08 13:35:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 1790/2304 [I 2022-10-08 13:35:27] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:06:40 [I 2022-10-08 13:35:27] ...trap/engines/engine.py.229: process: 0.79146 | load: 0.00304 [I 2022-10-08 13:35:27] ...trap/engines/engine.py.230: loss: 0.25877 [I 2022-10-08 13:35:27] ...optimizers/trijoint.py.151: total_norm: 3.507187 [I 2022-10-08 13:35:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 1800/2304 [I 2022-10-08 13:35:35] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:06:32 [I 2022-10-08 13:35:35] ...trap/engines/engine.py.229: process: 0.77253 | load: 0.00305 [I 2022-10-08 13:35:35] ...trap/engines/engine.py.230: loss: 0.15106 [I 2022-10-08 13:35:35] ...optimizers/trijoint.py.151: total_norm: 2.980299 [I 2022-10-08 13:35:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 1810/2304 [I 2022-10-08 13:35:43] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:06:25 [I 2022-10-08 13:35:43] ...trap/engines/engine.py.229: process: 0.79421 | load: 0.00302 [I 2022-10-08 13:35:43] ...trap/engines/engine.py.230: loss: 0.18235 [I 2022-10-08 13:35:43] ...optimizers/trijoint.py.151: total_norm: 2.097820 [I 2022-10-08 13:35:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 1820/2304 [I 2022-10-08 13:35:51] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:06:19 [I 2022-10-08 13:35:51] ...trap/engines/engine.py.229: process: 0.77981 | load: 0.00302 [I 2022-10-08 13:35:51] ...trap/engines/engine.py.230: loss: 0.27424 [I 2022-10-08 13:35:51] ...optimizers/trijoint.py.151: total_norm: 3.134778 [I 2022-10-08 13:35:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 1830/2304 [I 2022-10-08 13:35:58] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:06:08 [I 2022-10-08 13:35:58] ...trap/engines/engine.py.229: process: 0.77664 | load: 0.00301 [I 2022-10-08 13:35:58] ...trap/engines/engine.py.230: loss: 0.17840 [I 2022-10-08 13:35:59] ...optimizers/trijoint.py.151: total_norm: 2.436424 [I 2022-10-08 13:36:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 1840/2304 [I 2022-10-08 13:36:06] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:06:01 [I 2022-10-08 13:36:06] ...trap/engines/engine.py.229: process: 0.78048 | load: 0.00301 [I 2022-10-08 13:36:06] ...trap/engines/engine.py.230: loss: 0.26274 [I 2022-10-08 13:36:06] ...optimizers/trijoint.py.151: total_norm: 2.585466 [I 2022-10-08 13:36:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 1850/2304 [I 2022-10-08 13:36:14] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:05:53 [I 2022-10-08 13:36:14] ...trap/engines/engine.py.229: process: 0.78824 | load: 0.00304 [I 2022-10-08 13:36:14] ...trap/engines/engine.py.230: loss: 0.20320 [I 2022-10-08 13:36:14] ...optimizers/trijoint.py.151: total_norm: 2.301133 [I 2022-10-08 13:36:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 1860/2304 [I 2022-10-08 13:36:22] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:05:42 [I 2022-10-08 13:36:22] ...trap/engines/engine.py.229: process: 0.76210 | load: 0.00303 [I 2022-10-08 13:36:22] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-08 13:36:22] ...optimizers/trijoint.py.151: total_norm: 2.071468 [I 2022-10-08 13:36:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1870/2304 [I 2022-10-08 13:36:30] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:05:39 [I 2022-10-08 13:36:30] ...trap/engines/engine.py.229: process: 0.78410 | load: 0.00302 [I 2022-10-08 13:36:30] ...trap/engines/engine.py.230: loss: 0.20270 [I 2022-10-08 13:36:30] ...optimizers/trijoint.py.151: total_norm: 2.383371 [I 2022-10-08 13:36:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 1880/2304 [I 2022-10-08 13:36:38] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:05:29 [I 2022-10-08 13:36:38] ...trap/engines/engine.py.229: process: 0.77233 | load: 0.00306 [I 2022-10-08 13:36:38] ...trap/engines/engine.py.230: loss: 0.23431 [I 2022-10-08 13:36:38] ...optimizers/trijoint.py.151: total_norm: 2.976490 [I 2022-10-08 13:36:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 1890/2304 [I 2022-10-08 13:36:45] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:05:21 [I 2022-10-08 13:36:45] ...trap/engines/engine.py.229: process: 0.78716 | load: 0.00310 [I 2022-10-08 13:36:45] ...trap/engines/engine.py.230: loss: 0.18508 [I 2022-10-08 13:36:45] ...optimizers/trijoint.py.151: total_norm: 2.752238 [I 2022-10-08 13:36:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 1900/2304 [I 2022-10-08 13:36:53] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:05:14 [I 2022-10-08 13:36:53] ...trap/engines/engine.py.229: process: 0.78202 | load: 0.00303 [I 2022-10-08 13:36:53] ...trap/engines/engine.py.230: loss: 0.14087 [I 2022-10-08 13:36:53] ...optimizers/trijoint.py.151: total_norm: 1.945438 [I 2022-10-08 13:37:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 1910/2304 [I 2022-10-08 13:37:01] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:05:07 [I 2022-10-08 13:37:01] ...trap/engines/engine.py.229: process: 0.78839 | load: 0.00308 [I 2022-10-08 13:37:01] ...trap/engines/engine.py.230: loss: 0.16738 [I 2022-10-08 13:37:01] ...optimizers/trijoint.py.151: total_norm: 2.431868 [I 2022-10-08 13:37:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 1920/2304 [I 2022-10-08 13:37:09] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:05:00 [I 2022-10-08 13:37:09] ...trap/engines/engine.py.229: process: 0.78030 | load: 0.00307 [I 2022-10-08 13:37:09] ...trap/engines/engine.py.230: loss: 0.19399 [I 2022-10-08 13:37:09] ...optimizers/trijoint.py.151: total_norm: 2.798407 [I 2022-10-08 13:37:16] ...trap/engines/engine.py.224: train: epoch 6 | batch 1930/2304 [I 2022-10-08 13:37:16] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:04:50 [I 2022-10-08 13:37:16] ...trap/engines/engine.py.229: process: 0.77257 | load: 0.00309 [I 2022-10-08 13:37:16] ...trap/engines/engine.py.230: loss: 0.21920 [I 2022-10-08 13:37:16] ...optimizers/trijoint.py.151: total_norm: 3.272444 [I 2022-10-08 13:37:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 1940/2304 [I 2022-10-08 13:37:24] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:04:44 [I 2022-10-08 13:37:24] ...trap/engines/engine.py.229: process: 0.79612 | load: 0.00303 [I 2022-10-08 13:37:24] ...trap/engines/engine.py.230: loss: 0.30044 [I 2022-10-08 13:37:24] ...optimizers/trijoint.py.151: total_norm: 3.660145 [I 2022-10-08 13:37:32] ...trap/engines/engine.py.224: train: epoch 6 | batch 1950/2304 [I 2022-10-08 13:37:32] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:04:35 [I 2022-10-08 13:37:32] ...trap/engines/engine.py.229: process: 0.77989 | load: 0.00303 [I 2022-10-08 13:37:32] ...trap/engines/engine.py.230: loss: 0.21392 [I 2022-10-08 13:37:32] ...optimizers/trijoint.py.151: total_norm: 1.995680 [I 2022-10-08 13:37:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 1960/2304 [I 2022-10-08 13:37:40] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:04:27 [I 2022-10-08 13:37:40] ...trap/engines/engine.py.229: process: 0.77539 | load: 0.00301 [I 2022-10-08 13:37:40] ...trap/engines/engine.py.230: loss: 0.19614 [I 2022-10-08 13:37:40] ...optimizers/trijoint.py.151: total_norm: 2.747476 [I 2022-10-08 13:37:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 1970/2304 [I 2022-10-08 13:37:48] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:04:20 [I 2022-10-08 13:37:48] ...trap/engines/engine.py.229: process: 0.77771 | load: 0.00300 [I 2022-10-08 13:37:48] ...trap/engines/engine.py.230: loss: 0.23568 [I 2022-10-08 13:37:48] ...optimizers/trijoint.py.151: total_norm: 3.774324 [I 2022-10-08 13:37:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1980/2304 [I 2022-10-08 13:37:56] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:04:11 [I 2022-10-08 13:37:56] ...trap/engines/engine.py.229: process: 0.77246 | load: 0.00305 [I 2022-10-08 13:37:56] ...trap/engines/engine.py.230: loss: 0.18939 [I 2022-10-08 13:37:56] ...optimizers/trijoint.py.151: total_norm: 2.763410 [I 2022-10-08 13:38:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1990/2304 [I 2022-10-08 13:38:04] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:04:18 [I 2022-10-08 13:38:04] ...trap/engines/engine.py.229: process: 0.79478 | load: 0.00309 [I 2022-10-08 13:38:04] ...trap/engines/engine.py.230: loss: 0.15580 [I 2022-10-08 13:38:04] ...optimizers/trijoint.py.151: total_norm: 2.090389 [I 2022-10-08 13:38:11] ...trap/engines/engine.py.224: train: epoch 6 | batch 2000/2304 [I 2022-10-08 13:38:11] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:03:57 [I 2022-10-08 13:38:11] ...trap/engines/engine.py.229: process: 0.77651 | load: 0.00309 [I 2022-10-08 13:38:11] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-08 13:38:11] ...optimizers/trijoint.py.151: total_norm: 1.894755 [I 2022-10-08 13:38:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 2010/2304 [I 2022-10-08 13:38:19] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:03:49 [I 2022-10-08 13:38:19] ...trap/engines/engine.py.229: process: 0.78612 | load: 0.00300 [I 2022-10-08 13:38:19] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-08 13:38:19] ...optimizers/trijoint.py.151: total_norm: 2.317790 [I 2022-10-08 13:38:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 2020/2304 [I 2022-10-08 13:38:27] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:03:41 [I 2022-10-08 13:38:27] ...trap/engines/engine.py.229: process: 0.76853 | load: 0.00305 [I 2022-10-08 13:38:27] ...trap/engines/engine.py.230: loss: 0.19481 [I 2022-10-08 13:38:27] ...optimizers/trijoint.py.151: total_norm: 2.448016 [I 2022-10-08 13:38:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 2030/2304 [I 2022-10-08 13:38:35] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:03:33 [I 2022-10-08 13:38:35] ...trap/engines/engine.py.229: process: 0.78149 | load: 0.00302 [I 2022-10-08 13:38:35] ...trap/engines/engine.py.230: loss: 0.25070 [I 2022-10-08 13:38:35] ...optimizers/trijoint.py.151: total_norm: 3.495587 [I 2022-10-08 13:38:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 2040/2304 [I 2022-10-08 13:38:43] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:03:23 [I 2022-10-08 13:38:43] ...trap/engines/engine.py.229: process: 0.76592 | load: 0.00300 [I 2022-10-08 13:38:43] ...trap/engines/engine.py.230: loss: 0.26330 [I 2022-10-08 13:38:43] ...optimizers/trijoint.py.151: total_norm: 2.844973 [I 2022-10-08 13:38:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 2050/2304 [I 2022-10-08 13:38:50] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:03:18 [I 2022-10-08 13:38:50] ...trap/engines/engine.py.229: process: 0.78649 | load: 0.00302 [I 2022-10-08 13:38:50] ...trap/engines/engine.py.230: loss: 0.18651 [I 2022-10-08 13:38:50] ...optimizers/trijoint.py.151: total_norm: 1.940861 [I 2022-10-08 13:38:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 2060/2304 [I 2022-10-08 13:38:58] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:03:10 [I 2022-10-08 13:38:58] ...trap/engines/engine.py.229: process: 0.77781 | load: 0.00301 [I 2022-10-08 13:38:58] ...trap/engines/engine.py.230: loss: 0.27798 [I 2022-10-08 13:38:58] ...optimizers/trijoint.py.151: total_norm: 3.503092 [I 2022-10-08 13:39:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 2070/2304 [I 2022-10-08 13:39:06] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:03:02 [I 2022-10-08 13:39:06] ...trap/engines/engine.py.229: process: 0.78185 | load: 0.00302 [I 2022-10-08 13:39:06] ...trap/engines/engine.py.230: loss: 0.20847 [I 2022-10-08 13:39:06] ...optimizers/trijoint.py.151: total_norm: 2.563673 [I 2022-10-08 13:39:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 2080/2304 [I 2022-10-08 13:39:14] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:02:53 [I 2022-10-08 13:39:14] ...trap/engines/engine.py.229: process: 0.78454 | load: 0.00303 [I 2022-10-08 13:39:14] ...trap/engines/engine.py.230: loss: 0.21429 [I 2022-10-08 13:39:14] ...optimizers/trijoint.py.151: total_norm: 2.728954 [I 2022-10-08 13:39:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 2090/2304 [I 2022-10-08 13:39:22] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:02:46 [I 2022-10-08 13:39:22] ...trap/engines/engine.py.229: process: 0.78127 | load: 0.00309 [I 2022-10-08 13:39:22] ...trap/engines/engine.py.230: loss: 0.21562 [I 2022-10-08 13:39:22] ...optimizers/trijoint.py.151: total_norm: 3.057497 [I 2022-10-08 13:39:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 2100/2304 [I 2022-10-08 13:39:30] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:02:40 [I 2022-10-08 13:39:30] ...trap/engines/engine.py.229: process: 0.78730 | load: 0.00307 [I 2022-10-08 13:39:30] ...trap/engines/engine.py.230: loss: 0.13634 [I 2022-10-08 13:39:30] ...optimizers/trijoint.py.151: total_norm: 2.231004 [I 2022-10-08 13:39:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 2110/2304 [I 2022-10-08 13:39:38] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:02:33 [I 2022-10-08 13:39:38] ...trap/engines/engine.py.229: process: 0.79723 | load: 0.00299 [I 2022-10-08 13:39:38] ...trap/engines/engine.py.230: loss: 0.19555 [I 2022-10-08 13:39:38] ...optimizers/trijoint.py.151: total_norm: 2.195325 [I 2022-10-08 13:39:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 2120/2304 [I 2022-10-08 13:39:45] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:02:24 [I 2022-10-08 13:39:45] ...trap/engines/engine.py.229: process: 0.79346 | load: 0.00301 [I 2022-10-08 13:39:45] ...trap/engines/engine.py.230: loss: 0.21968 [I 2022-10-08 13:39:45] ...optimizers/trijoint.py.151: total_norm: 2.863137 [I 2022-10-08 13:39:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 2130/2304 [I 2022-10-08 13:39:53] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:02:16 [I 2022-10-08 13:39:53] ...trap/engines/engine.py.229: process: 0.78213 | load: 0.00299 [I 2022-10-08 13:39:53] ...trap/engines/engine.py.230: loss: 0.17518 [I 2022-10-08 13:39:53] ...optimizers/trijoint.py.151: total_norm: 2.135002 [I 2022-10-08 13:40:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 2140/2304 [I 2022-10-08 13:40:01] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:02:08 [I 2022-10-08 13:40:01] ...trap/engines/engine.py.229: process: 0.77349 | load: 0.00301 [I 2022-10-08 13:40:01] ...trap/engines/engine.py.230: loss: 0.26802 [I 2022-10-08 13:40:01] ...optimizers/trijoint.py.151: total_norm: 2.828277 [I 2022-10-08 13:40:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 2150/2304 [I 2022-10-08 13:40:09] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:02:00 [I 2022-10-08 13:40:09] ...trap/engines/engine.py.229: process: 0.76989 | load: 0.00302 [I 2022-10-08 13:40:09] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-08 13:40:09] ...optimizers/trijoint.py.151: total_norm: 2.084724 [I 2022-10-08 13:40:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 2160/2304 [I 2022-10-08 13:40:17] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:01:52 [I 2022-10-08 13:40:17] ...trap/engines/engine.py.229: process: 0.78052 | load: 0.00300 [I 2022-10-08 13:40:17] ...trap/engines/engine.py.230: loss: 0.17713 [I 2022-10-08 13:40:17] ...optimizers/trijoint.py.151: total_norm: 2.624263 [I 2022-10-08 13:40:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 2170/2304 [I 2022-10-08 13:40:25] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:01:44 [I 2022-10-08 13:40:25] ...trap/engines/engine.py.229: process: 0.78432 | load: 0.00307 [I 2022-10-08 13:40:25] ...trap/engines/engine.py.230: loss: 0.18262 [I 2022-10-08 13:40:25] ...optimizers/trijoint.py.151: total_norm: 2.861479 [I 2022-10-08 13:40:32] ...trap/engines/engine.py.224: train: epoch 6 | batch 2180/2304 [I 2022-10-08 13:40:32] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:01:36 [I 2022-10-08 13:40:32] ...trap/engines/engine.py.229: process: 0.77960 | load: 0.00303 [I 2022-10-08 13:40:32] ...trap/engines/engine.py.230: loss: 0.35652 [I 2022-10-08 13:40:32] ...optimizers/trijoint.py.151: total_norm: 3.636687 [I 2022-10-08 13:40:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 2190/2304 [I 2022-10-08 13:40:40] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:01:28 [I 2022-10-08 13:40:40] ...trap/engines/engine.py.229: process: 0.78464 | load: 0.00306 [I 2022-10-08 13:40:40] ...trap/engines/engine.py.230: loss: 0.27239 [I 2022-10-08 13:40:40] ...optimizers/trijoint.py.151: total_norm: 3.127576 [I 2022-10-08 13:40:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 2200/2304 [I 2022-10-08 13:40:48] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:01:20 [I 2022-10-08 13:40:48] ...trap/engines/engine.py.229: process: 0.77204 | load: 0.00302 [I 2022-10-08 13:40:48] ...trap/engines/engine.py.230: loss: 0.23012 [I 2022-10-08 13:40:48] ...optimizers/trijoint.py.151: total_norm: 3.147699 [I 2022-10-08 13:40:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 2210/2304 [I 2022-10-08 13:40:56] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:01:13 [I 2022-10-08 13:40:56] ...trap/engines/engine.py.229: process: 0.77880 | load: 0.00307 [I 2022-10-08 13:40:56] ...trap/engines/engine.py.230: loss: 0.27278 [I 2022-10-08 13:40:56] ...optimizers/trijoint.py.151: total_norm: 2.986549 [I 2022-10-08 13:41:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 2220/2304 [I 2022-10-08 13:41:04] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:01:05 [I 2022-10-08 13:41:04] ...trap/engines/engine.py.229: process: 0.76234 | load: 0.00303 [I 2022-10-08 13:41:04] ...trap/engines/engine.py.230: loss: 0.15515 [I 2022-10-08 13:41:04] ...optimizers/trijoint.py.151: total_norm: 2.070172 [I 2022-10-08 13:41:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 2230/2304 [I 2022-10-08 13:41:28] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:02:01 [I 2022-10-08 13:41:28] ...trap/engines/engine.py.229: process: 0.80190 | load: 0.00305 [I 2022-10-08 13:41:28] ...trap/engines/engine.py.230: loss: 0.23483 [I 2022-10-08 13:41:28] ...optimizers/trijoint.py.151: total_norm: 3.171669 [I 2022-10-08 13:41:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 2240/2304 [I 2022-10-08 13:41:36] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:00:55 [I 2022-10-08 13:41:36] ...trap/engines/engine.py.229: process: 0.78319 | load: 0.00304 [I 2022-10-08 13:41:36] ...trap/engines/engine.py.230: loss: 0.23320 [I 2022-10-08 13:41:36] ...optimizers/trijoint.py.151: total_norm: 2.897080 [I 2022-10-08 13:41:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 2250/2304 [I 2022-10-08 13:41:43] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:00:42 [I 2022-10-08 13:41:43] ...trap/engines/engine.py.229: process: 0.77719 | load: 0.00301 [I 2022-10-08 13:41:43] ...trap/engines/engine.py.230: loss: 0.27787 [I 2022-10-08 13:41:43] ...optimizers/trijoint.py.151: total_norm: 2.374688 [I 2022-10-08 13:41:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 2260/2304 [I 2022-10-08 13:41:51] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:00:34 [I 2022-10-08 13:41:51] ...trap/engines/engine.py.229: process: 0.77229 | load: 0.00301 [I 2022-10-08 13:41:51] ...trap/engines/engine.py.230: loss: 0.26063 [I 2022-10-08 13:41:51] ...optimizers/trijoint.py.151: total_norm: 2.666941 [I 2022-10-08 13:41:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 2270/2304 [I 2022-10-08 13:41:59] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:00:26 [I 2022-10-08 13:41:59] ...trap/engines/engine.py.229: process: 0.78068 | load: 0.00302 [I 2022-10-08 13:41:59] ...trap/engines/engine.py.230: loss: 0.25937 [I 2022-10-08 13:41:59] ...optimizers/trijoint.py.151: total_norm: 3.086694 [I 2022-10-08 13:42:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 2280/2304 [I 2022-10-08 13:42:07] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:00:18 [I 2022-10-08 13:42:07] ...trap/engines/engine.py.229: process: 0.77866 | load: 0.00301 [I 2022-10-08 13:42:07] ...trap/engines/engine.py.230: loss: 0.22281 [I 2022-10-08 13:42:07] ...optimizers/trijoint.py.151: total_norm: 2.359650 [I 2022-10-08 13:42:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 2290/2304 [I 2022-10-08 13:42:15] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:00:10 [I 2022-10-08 13:42:15] ...trap/engines/engine.py.229: process: 0.78008 | load: 0.00304 [I 2022-10-08 13:42:15] ...trap/engines/engine.py.230: loss: 0.20786 [I 2022-10-08 13:42:15] ...optimizers/trijoint.py.151: total_norm: 2.782297 [I 2022-10-08 13:42:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 2300/2304 [I 2022-10-08 13:42:22] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:00:03 [I 2022-10-08 13:42:22] ...trap/engines/engine.py.229: process: 0.76149 | load: 0.00017 [I 2022-10-08 13:42:22] ...trap/engines/engine.py.230: loss: 0.23166 [I 2022-10-08 13:42:22] ...optimizers/trijoint.py.151: total_norm: 2.812957 [I 2022-10-08 13:42:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 2304/2304 [I 2022-10-08 13:42:25] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:00:00 [I 2022-10-08 13:42:25] ...trap/engines/engine.py.229: process: 0.76574 | load: 0.00017 [I 2022-10-08 13:42:25] ...trap/engines/engine.py.230: loss: 0.23395 [I 2022-10-08 13:42:25] ...optimizers/trijoint.py.151: total_norm: 2.965951 [S 2022-10-08 13:42:27] ...trap/engines/engine.py.129: train_epoch.epoch: 6 [S 2022-10-08 13:42:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.21634272266171242 [I 2022-10-08 13:42:27] ...s/criterions/triplet.py.84: margin = 0.08500000000000002 [I 2022-10-08 13:42:27] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 6 [I 2022-10-08 13:42:27] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 13:42:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 13:42:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 13:42:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 13:42:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 13:42:29] ...trap/engines/engine.py.321: eval: epoch 6 | batch 0/510 [I 2022-10-08 13:42:29] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:05 [I 2022-10-08 13:42:29] ...trap/engines/engine.py.326: process: 2.59991 | load: 2.08998 [I 2022-10-08 13:42:34] ...trap/engines/engine.py.321: eval: epoch 6 | batch 10/510 [I 2022-10-08 13:42:34] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:09 [I 2022-10-08 13:42:34] ...trap/engines/engine.py.326: process: 0.65405 | load: 0.00019 [I 2022-10-08 13:42:41] ...trap/engines/engine.py.321: eval: epoch 6 | batch 20/510 [I 2022-10-08 13:42:41] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:05:34 [I 2022-10-08 13:42:41] ...trap/engines/engine.py.326: process: 0.67012 | load: 0.00018 [I 2022-10-08 13:42:46] ...trap/engines/engine.py.321: eval: epoch 6 | batch 30/510 [I 2022-10-08 13:42:46] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:04 [I 2022-10-08 13:42:46] ...trap/engines/engine.py.326: process: 0.42356 | load: 0.00018 [I 2022-10-08 13:42:52] ...trap/engines/engine.py.321: eval: epoch 6 | batch 40/510 [I 2022-10-08 13:42:52] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:22 [I 2022-10-08 13:42:52] ...trap/engines/engine.py.326: process: 0.43532 | load: 0.00020 [I 2022-10-08 13:42:56] ...trap/engines/engine.py.321: eval: epoch 6 | batch 50/510 [I 2022-10-08 13:42:56] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:33 [I 2022-10-08 13:42:56] ...trap/engines/engine.py.326: process: 0.42718 | load: 0.00019 [I 2022-10-08 13:43:01] ...trap/engines/engine.py.321: eval: epoch 6 | batch 60/510 [I 2022-10-08 13:43:01] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:20 [I 2022-10-08 13:43:01] ...trap/engines/engine.py.326: process: 0.47686 | load: 0.00018 [I 2022-10-08 13:43:06] ...trap/engines/engine.py.321: eval: epoch 6 | batch 70/510 [I 2022-10-08 13:43:06] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:25 [I 2022-10-08 13:43:06] ...trap/engines/engine.py.326: process: 0.42704 | load: 0.00017 [I 2022-10-08 13:43:10] ...trap/engines/engine.py.321: eval: epoch 6 | batch 80/510 [I 2022-10-08 13:43:10] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:08 [I 2022-10-08 13:43:10] ...trap/engines/engine.py.326: process: 0.45974 | load: 0.00018 [I 2022-10-08 13:43:14] ...trap/engines/engine.py.321: eval: epoch 6 | batch 90/510 [I 2022-10-08 13:43:14] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:05 [I 2022-10-08 13:43:14] ...trap/engines/engine.py.326: process: 0.44525 | load: 0.00019 [I 2022-10-08 13:43:19] ...trap/engines/engine.py.321: eval: epoch 6 | batch 100/510 [I 2022-10-08 13:43:19] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:50 [I 2022-10-08 13:43:19] ...trap/engines/engine.py.326: process: 0.31043 | load: 0.00019 [I 2022-10-08 13:43:22] ...trap/engines/engine.py.321: eval: epoch 6 | batch 110/510 [I 2022-10-08 13:43:22] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-08 13:43:22] ...trap/engines/engine.py.326: process: 0.31912 | load: 0.00019 [I 2022-10-08 13:43:25] ...trap/engines/engine.py.321: eval: epoch 6 | batch 120/510 [I 2022-10-08 13:43:25] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:02 [I 2022-10-08 13:43:25] ...trap/engines/engine.py.326: process: 0.30956 | load: 0.00019 [I 2022-10-08 13:43:28] ...trap/engines/engine.py.321: eval: epoch 6 | batch 130/510 [I 2022-10-08 13:43:28] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:59 [I 2022-10-08 13:43:28] ...trap/engines/engine.py.326: process: 0.31212 | load: 0.00019 [I 2022-10-08 13:43:31] ...trap/engines/engine.py.321: eval: epoch 6 | batch 140/510 [I 2022-10-08 13:43:31] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:56 [I 2022-10-08 13:43:31] ...trap/engines/engine.py.326: process: 0.31684 | load: 0.00019 [I 2022-10-08 13:43:35] ...trap/engines/engine.py.321: eval: epoch 6 | batch 150/510 [I 2022-10-08 13:43:35] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:53 [I 2022-10-08 13:43:35] ...trap/engines/engine.py.326: process: 0.31264 | load: 0.00020 [I 2022-10-08 13:43:38] ...trap/engines/engine.py.321: eval: epoch 6 | batch 160/510 [I 2022-10-08 13:43:38] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:49 [I 2022-10-08 13:43:38] ...trap/engines/engine.py.326: process: 0.31207 | load: 0.00019 [I 2022-10-08 13:43:41] ...trap/engines/engine.py.321: eval: epoch 6 | batch 170/510 [I 2022-10-08 13:43:41] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-08 13:43:41] ...trap/engines/engine.py.326: process: 0.31389 | load: 0.00020 [I 2022-10-08 13:43:44] ...trap/engines/engine.py.321: eval: epoch 6 | batch 180/510 [I 2022-10-08 13:43:44] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:43 [I 2022-10-08 13:43:44] ...trap/engines/engine.py.326: process: 0.30817 | load: 0.00019 [I 2022-10-08 13:43:47] ...trap/engines/engine.py.321: eval: epoch 6 | batch 190/510 [I 2022-10-08 13:43:47] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:41 [I 2022-10-08 13:43:47] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00020 [I 2022-10-08 13:43:50] ...trap/engines/engine.py.321: eval: epoch 6 | batch 200/510 [I 2022-10-08 13:43:50] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:38 [I 2022-10-08 13:43:50] ...trap/engines/engine.py.326: process: 0.32115 | load: 0.00019 [I 2022-10-08 13:43:54] ...trap/engines/engine.py.321: eval: epoch 6 | batch 210/510 [I 2022-10-08 13:43:54] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:34 [I 2022-10-08 13:43:54] ...trap/engines/engine.py.326: process: 0.31422 | load: 0.00019 [I 2022-10-08 13:43:57] ...trap/engines/engine.py.321: eval: epoch 6 | batch 220/510 [I 2022-10-08 13:43:57] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:32 [I 2022-10-08 13:43:57] ...trap/engines/engine.py.326: process: 0.32378 | load: 0.00019 [I 2022-10-08 13:44:00] ...trap/engines/engine.py.321: eval: epoch 6 | batch 230/510 [I 2022-10-08 13:44:00] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-08 13:44:00] ...trap/engines/engine.py.326: process: 0.31882 | load: 0.00019 [I 2022-10-08 13:44:03] ...trap/engines/engine.py.321: eval: epoch 6 | batch 240/510 [I 2022-10-08 13:44:03] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:25 [I 2022-10-08 13:44:03] ...trap/engines/engine.py.326: process: 0.31478 | load: 0.00020 [I 2022-10-08 13:44:06] ...trap/engines/engine.py.321: eval: epoch 6 | batch 250/510 [I 2022-10-08 13:44:06] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-08 13:44:06] ...trap/engines/engine.py.326: process: 0.32550 | load: 0.00020 [I 2022-10-08 13:44:09] ...trap/engines/engine.py.321: eval: epoch 6 | batch 260/510 [I 2022-10-08 13:44:09] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-08 13:44:09] ...trap/engines/engine.py.326: process: 0.32012 | load: 0.00020 [I 2022-10-08 13:44:13] ...trap/engines/engine.py.321: eval: epoch 6 | batch 270/510 [I 2022-10-08 13:44:13] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:16 [I 2022-10-08 13:44:13] ...trap/engines/engine.py.326: process: 0.31994 | load: 0.00020 [I 2022-10-08 13:44:16] ...trap/engines/engine.py.321: eval: epoch 6 | batch 280/510 [I 2022-10-08 13:44:16] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:12 [I 2022-10-08 13:44:16] ...trap/engines/engine.py.326: process: 0.31120 | load: 0.00020 [I 2022-10-08 13:44:19] ...trap/engines/engine.py.321: eval: epoch 6 | batch 290/510 [I 2022-10-08 13:44:19] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:09 [I 2022-10-08 13:44:19] ...trap/engines/engine.py.326: process: 0.31735 | load: 0.00020 [I 2022-10-08 13:44:22] ...trap/engines/engine.py.321: eval: epoch 6 | batch 300/510 [I 2022-10-08 13:44:22] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-08 13:44:22] ...trap/engines/engine.py.326: process: 0.31715 | load: 0.00020 [I 2022-10-08 13:44:25] ...trap/engines/engine.py.321: eval: epoch 6 | batch 310/510 [I 2022-10-08 13:44:25] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-08 13:44:25] ...trap/engines/engine.py.326: process: 0.31357 | load: 0.00019 [I 2022-10-08 13:44:29] ...trap/engines/engine.py.321: eval: epoch 6 | batch 320/510 [I 2022-10-08 13:44:29] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-08 13:44:29] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00020 [I 2022-10-08 13:44:32] ...trap/engines/engine.py.321: eval: epoch 6 | batch 330/510 [I 2022-10-08 13:44:32] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:56 [I 2022-10-08 13:44:32] ...trap/engines/engine.py.326: process: 0.31740 | load: 0.00021 [I 2022-10-08 13:44:35] ...trap/engines/engine.py.321: eval: epoch 6 | batch 340/510 [I 2022-10-08 13:44:35] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:53 [I 2022-10-08 13:44:35] ...trap/engines/engine.py.326: process: 0.31669 | load: 0.00020 [I 2022-10-08 13:44:38] ...trap/engines/engine.py.321: eval: epoch 6 | batch 350/510 [I 2022-10-08 13:44:38] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-08 13:44:38] ...trap/engines/engine.py.326: process: 0.31650 | load: 0.00020 [I 2022-10-08 13:44:41] ...trap/engines/engine.py.321: eval: epoch 6 | batch 360/510 [I 2022-10-08 13:44:41] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-08 13:44:41] ...trap/engines/engine.py.326: process: 0.31710 | load: 0.00019 [I 2022-10-08 13:44:44] ...trap/engines/engine.py.321: eval: epoch 6 | batch 370/510 [I 2022-10-08 13:44:44] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:44 [I 2022-10-08 13:44:44] ...trap/engines/engine.py.326: process: 0.31689 | load: 0.00020 [I 2022-10-08 13:44:48] ...trap/engines/engine.py.321: eval: epoch 6 | batch 380/510 [I 2022-10-08 13:44:48] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:41 [I 2022-10-08 13:44:48] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00020 [I 2022-10-08 13:44:51] ...trap/engines/engine.py.321: eval: epoch 6 | batch 390/510 [I 2022-10-08 13:44:51] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:38 [I 2022-10-08 13:44:51] ...trap/engines/engine.py.326: process: 0.31432 | load: 0.00019 [I 2022-10-08 13:44:54] ...trap/engines/engine.py.321: eval: epoch 6 | batch 400/510 [I 2022-10-08 13:44:54] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-08 13:44:54] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00020 [I 2022-10-08 13:44:57] ...trap/engines/engine.py.321: eval: epoch 6 | batch 410/510 [I 2022-10-08 13:44:57] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-08 13:44:57] ...trap/engines/engine.py.326: process: 0.31915 | load: 0.00020 [I 2022-10-08 13:45:00] ...trap/engines/engine.py.321: eval: epoch 6 | batch 420/510 [I 2022-10-08 13:45:00] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-08 13:45:00] ...trap/engines/engine.py.326: process: 0.31550 | load: 0.00019 [I 2022-10-08 13:45:03] ...trap/engines/engine.py.321: eval: epoch 6 | batch 430/510 [I 2022-10-08 13:45:03] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:25 [I 2022-10-08 13:45:03] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00020 [I 2022-10-08 13:45:07] ...trap/engines/engine.py.321: eval: epoch 6 | batch 440/510 [I 2022-10-08 13:45:07] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:22 [I 2022-10-08 13:45:07] ...trap/engines/engine.py.326: process: 0.31898 | load: 0.00021 [I 2022-10-08 13:45:10] ...trap/engines/engine.py.321: eval: epoch 6 | batch 450/510 [I 2022-10-08 13:45:10] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-08 13:45:10] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00020 [I 2022-10-08 13:45:13] ...trap/engines/engine.py.321: eval: epoch 6 | batch 460/510 [I 2022-10-08 13:45:13] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:16 [I 2022-10-08 13:45:13] ...trap/engines/engine.py.326: process: 0.32646 | load: 0.00020 [I 2022-10-08 13:45:16] ...trap/engines/engine.py.321: eval: epoch 6 | batch 470/510 [I 2022-10-08 13:45:16] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-08 13:45:16] ...trap/engines/engine.py.326: process: 0.31677 | load: 0.00020 [I 2022-10-08 13:45:19] ...trap/engines/engine.py.321: eval: epoch 6 | batch 480/510 [I 2022-10-08 13:45:19] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-08 13:45:19] ...trap/engines/engine.py.326: process: 0.32093 | load: 0.00020 [I 2022-10-08 13:45:23] ...trap/engines/engine.py.321: eval: epoch 6 | batch 490/510 [I 2022-10-08 13:45:23] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:06 [I 2022-10-08 13:45:23] ...trap/engines/engine.py.326: process: 0.32341 | load: 0.00021 [I 2022-10-08 13:45:26] ...trap/engines/engine.py.321: eval: epoch 6 | batch 500/510 [I 2022-10-08 13:45:26] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:03 [I 2022-10-08 13:45:26] ...trap/engines/engine.py.326: process: 0.31353 | load: 0.00020 [I 2022-10-08 13:45:29] ...trap/engines/engine.py.321: eval: epoch 6 | batch 510/510 [I 2022-10-08 13:45:29] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-08 13:45:29] ...trap/engines/engine.py.326: process: 0.31251 | load: 0.00015 [S 2022-10-08 13:45:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 6 [I 2022-10-08 13:45:30] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.564400 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.837100 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.896000 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.569500 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.844000 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.902400 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014712 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010540 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009726 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018896 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009445 [S 2022-10-08 13:45:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007214 [I 2022-10-08 13:46:02] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 13:46:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 13:46:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 13:46:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 13:46:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 13:46:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 13:46:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 13:46:09] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 13:46:09] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 13:46:09] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 13:46:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 13:46:17] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 13:46:17] ...trap/engines/engine.py.163: Training model on trainset for epoch 7 [I 2022-10-08 13:46:17] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 13:46:20] ...trap/engines/engine.py.224: train: epoch 7 | batch 0/2304 [I 2022-10-08 13:46:20] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:40 [I 2022-10-08 13:46:20] ...trap/engines/engine.py.229: process: 2.98631 | load: 2.18187 [I 2022-10-08 13:46:20] ...trap/engines/engine.py.230: loss: 0.29600 [I 2022-10-08 13:46:20] ...optimizers/trijoint.py.151: total_norm: 2.871614 [I 2022-10-08 13:46:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 10/2304 [I 2022-10-08 13:46:28] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:55 [I 2022-10-08 13:46:28] ...trap/engines/engine.py.229: process: 0.77599 | load: 0.00608 [I 2022-10-08 13:46:28] ...trap/engines/engine.py.230: loss: 0.10873 [I 2022-10-08 13:46:28] ...optimizers/trijoint.py.151: total_norm: 1.533589 [I 2022-10-08 13:46:35] ...trap/engines/engine.py.224: train: epoch 7 | batch 20/2304 [I 2022-10-08 13:46:35] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:37 [I 2022-10-08 13:46:35] ...trap/engines/engine.py.229: process: 0.78846 | load: 0.00456 [I 2022-10-08 13:46:35] ...trap/engines/engine.py.230: loss: 0.16627 [I 2022-10-08 13:46:35] ...optimizers/trijoint.py.151: total_norm: 1.919494 [I 2022-10-08 13:46:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 30/2304 [I 2022-10-08 13:46:43] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:38 [I 2022-10-08 13:46:43] ...trap/engines/engine.py.229: process: 0.77302 | load: 0.00404 [I 2022-10-08 13:46:43] ...trap/engines/engine.py.230: loss: 0.17198 [I 2022-10-08 13:46:43] ...optimizers/trijoint.py.151: total_norm: 2.717643 [I 2022-10-08 13:46:51] ...trap/engines/engine.py.224: train: epoch 7 | batch 40/2304 [I 2022-10-08 13:46:51] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:20 [I 2022-10-08 13:46:51] ...trap/engines/engine.py.229: process: 0.77074 | load: 0.00422 [I 2022-10-08 13:46:51] ...trap/engines/engine.py.230: loss: 0.19141 [I 2022-10-08 13:46:51] ...optimizers/trijoint.py.151: total_norm: 2.357860 [I 2022-10-08 13:46:59] ...trap/engines/engine.py.224: train: epoch 7 | batch 50/2304 [I 2022-10-08 13:46:59] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:16 [I 2022-10-08 13:46:59] ...trap/engines/engine.py.229: process: 0.77362 | load: 0.00367 [I 2022-10-08 13:46:59] ...trap/engines/engine.py.230: loss: 0.23165 [I 2022-10-08 13:46:59] ...optimizers/trijoint.py.151: total_norm: 2.287398 [I 2022-10-08 13:47:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 60/2304 [I 2022-10-08 13:47:07] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:28:52 [I 2022-10-08 13:47:07] ...trap/engines/engine.py.229: process: 0.77028 | load: 0.00409 [I 2022-10-08 13:47:07] ...trap/engines/engine.py.230: loss: 0.22768 [I 2022-10-08 13:47:07] ...optimizers/trijoint.py.151: total_norm: 3.599725 [I 2022-10-08 13:47:14] ...trap/engines/engine.py.224: train: epoch 7 | batch 70/2304 [I 2022-10-08 13:47:14] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:13 [I 2022-10-08 13:47:14] ...trap/engines/engine.py.229: process: 0.77635 | load: 0.00338 [I 2022-10-08 13:47:14] ...trap/engines/engine.py.230: loss: 0.16896 [I 2022-10-08 13:47:14] ...optimizers/trijoint.py.151: total_norm: 2.776964 [I 2022-10-08 13:47:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 80/2304 [I 2022-10-08 13:47:22] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:52 [I 2022-10-08 13:47:22] ...trap/engines/engine.py.229: process: 0.77771 | load: 0.00335 [I 2022-10-08 13:47:22] ...trap/engines/engine.py.230: loss: 0.24472 [I 2022-10-08 13:47:22] ...optimizers/trijoint.py.151: total_norm: 3.291109 [I 2022-10-08 13:47:30] ...trap/engines/engine.py.224: train: epoch 7 | batch 90/2304 [I 2022-10-08 13:47:30] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:34 [I 2022-10-08 13:47:30] ...trap/engines/engine.py.229: process: 0.76314 | load: 0.00337 [I 2022-10-08 13:47:30] ...trap/engines/engine.py.230: loss: 0.19960 [I 2022-10-08 13:47:30] ...optimizers/trijoint.py.151: total_norm: 2.380416 [I 2022-10-08 13:47:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 100/2304 [I 2022-10-08 13:47:38] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:33 [I 2022-10-08 13:47:38] ...trap/engines/engine.py.229: process: 0.77288 | load: 0.00337 [I 2022-10-08 13:47:38] ...trap/engines/engine.py.230: loss: 0.20016 [I 2022-10-08 13:47:38] ...optimizers/trijoint.py.151: total_norm: 2.248258 [I 2022-10-08 13:47:46] ...trap/engines/engine.py.224: train: epoch 7 | batch 110/2304 [I 2022-10-08 13:47:46] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:26 [I 2022-10-08 13:47:46] ...trap/engines/engine.py.229: process: 0.77944 | load: 0.00359 [I 2022-10-08 13:47:46] ...trap/engines/engine.py.230: loss: 0.17786 [I 2022-10-08 13:47:46] ...optimizers/trijoint.py.151: total_norm: 2.061203 [I 2022-10-08 13:47:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 120/2304 [I 2022-10-08 13:47:53] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:22 [I 2022-10-08 13:47:53] ...trap/engines/engine.py.229: process: 0.78732 | load: 0.00351 [I 2022-10-08 13:47:53] ...trap/engines/engine.py.230: loss: 0.22151 [I 2022-10-08 13:47:53] ...optimizers/trijoint.py.151: total_norm: 2.193820 [I 2022-10-08 13:48:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 130/2304 [I 2022-10-08 13:48:01] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:08 [I 2022-10-08 13:48:01] ...trap/engines/engine.py.229: process: 0.78455 | load: 0.00343 [I 2022-10-08 13:48:01] ...trap/engines/engine.py.230: loss: 0.24710 [I 2022-10-08 13:48:01] ...optimizers/trijoint.py.151: total_norm: 2.193262 [I 2022-10-08 13:48:09] ...trap/engines/engine.py.224: train: epoch 7 | batch 140/2304 [I 2022-10-08 13:48:09] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:02 [I 2022-10-08 13:48:09] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00322 [I 2022-10-08 13:48:09] ...trap/engines/engine.py.230: loss: 0.15607 [I 2022-10-08 13:48:09] ...optimizers/trijoint.py.151: total_norm: 2.189189 [I 2022-10-08 13:48:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 150/2304 [I 2022-10-08 13:48:17] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:27:56 [I 2022-10-08 13:48:17] ...trap/engines/engine.py.229: process: 0.78145 | load: 0.00330 [I 2022-10-08 13:48:17] ...trap/engines/engine.py.230: loss: 0.26066 [I 2022-10-08 13:48:17] ...optimizers/trijoint.py.151: total_norm: 2.687716 [I 2022-10-08 13:48:25] ...trap/engines/engine.py.224: train: epoch 7 | batch 160/2304 [I 2022-10-08 13:48:25] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:51 [I 2022-10-08 13:48:25] ...trap/engines/engine.py.229: process: 0.77787 | load: 0.00334 [I 2022-10-08 13:48:25] ...trap/engines/engine.py.230: loss: 0.28425 [I 2022-10-08 13:48:25] ...optimizers/trijoint.py.151: total_norm: 3.620079 [I 2022-10-08 13:48:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 170/2304 [I 2022-10-08 13:48:32] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:36 [I 2022-10-08 13:48:32] ...trap/engines/engine.py.229: process: 0.76843 | load: 0.00315 [I 2022-10-08 13:48:32] ...trap/engines/engine.py.230: loss: 0.31144 [I 2022-10-08 13:48:32] ...optimizers/trijoint.py.151: total_norm: 3.459180 [I 2022-10-08 13:48:40] ...trap/engines/engine.py.224: train: epoch 7 | batch 180/2304 [I 2022-10-08 13:48:40] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:38 [I 2022-10-08 13:48:40] ...trap/engines/engine.py.229: process: 0.78222 | load: 0.00318 [I 2022-10-08 13:48:40] ...trap/engines/engine.py.230: loss: 0.22869 [I 2022-10-08 13:48:40] ...optimizers/trijoint.py.151: total_norm: 3.296620 [I 2022-10-08 13:48:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 190/2304 [I 2022-10-08 13:48:48] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:27 [I 2022-10-08 13:48:48] ...trap/engines/engine.py.229: process: 0.77845 | load: 0.00354 [I 2022-10-08 13:48:48] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-08 13:48:48] ...optimizers/trijoint.py.151: total_norm: 2.303266 [I 2022-10-08 13:48:56] ...trap/engines/engine.py.224: train: epoch 7 | batch 200/2304 [I 2022-10-08 13:48:56] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:27:11 [I 2022-10-08 13:48:56] ...trap/engines/engine.py.229: process: 0.77326 | load: 0.00299 [I 2022-10-08 13:48:56] ...trap/engines/engine.py.230: loss: 0.19370 [I 2022-10-08 13:48:56] ...optimizers/trijoint.py.151: total_norm: 2.775604 [I 2022-10-08 13:49:04] ...trap/engines/engine.py.224: train: epoch 7 | batch 210/2304 [I 2022-10-08 13:49:04] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:05 [I 2022-10-08 13:49:04] ...trap/engines/engine.py.229: process: 0.77854 | load: 0.00314 [I 2022-10-08 13:49:04] ...trap/engines/engine.py.230: loss: 0.21055 [I 2022-10-08 13:49:04] ...optimizers/trijoint.py.151: total_norm: 2.832590 [I 2022-10-08 13:49:11] ...trap/engines/engine.py.224: train: epoch 7 | batch 220/2304 [I 2022-10-08 13:49:11] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:04 [I 2022-10-08 13:49:11] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00299 [I 2022-10-08 13:49:11] ...trap/engines/engine.py.230: loss: 0.24835 [I 2022-10-08 13:49:11] ...optimizers/trijoint.py.151: total_norm: 2.869100 [I 2022-10-08 13:49:19] ...trap/engines/engine.py.224: train: epoch 7 | batch 230/2304 [I 2022-10-08 13:49:19] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:56 [I 2022-10-08 13:49:19] ...trap/engines/engine.py.229: process: 0.78559 | load: 0.00312 [I 2022-10-08 13:49:19] ...trap/engines/engine.py.230: loss: 0.23608 [I 2022-10-08 13:49:19] ...optimizers/trijoint.py.151: total_norm: 2.760395 [I 2022-10-08 13:49:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 240/2304 [I 2022-10-08 13:49:27] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:39 [I 2022-10-08 13:49:27] ...trap/engines/engine.py.229: process: 0.76803 | load: 0.00316 [I 2022-10-08 13:49:27] ...trap/engines/engine.py.230: loss: 0.16415 [I 2022-10-08 13:49:27] ...optimizers/trijoint.py.151: total_norm: 2.714854 [I 2022-10-08 13:49:35] ...trap/engines/engine.py.224: train: epoch 7 | batch 250/2304 [I 2022-10-08 13:49:35] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:26:48 [I 2022-10-08 13:49:35] ...trap/engines/engine.py.229: process: 0.78779 | load: 0.00304 [I 2022-10-08 13:49:35] ...trap/engines/engine.py.230: loss: 0.20184 [I 2022-10-08 13:49:35] ...optimizers/trijoint.py.151: total_norm: 2.050810 [I 2022-10-08 13:49:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 260/2304 [I 2022-10-08 13:49:43] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:24 [I 2022-10-08 13:49:43] ...trap/engines/engine.py.229: process: 0.76635 | load: 0.00313 [I 2022-10-08 13:49:43] ...trap/engines/engine.py.230: loss: 0.29537 [I 2022-10-08 13:49:43] ...optimizers/trijoint.py.151: total_norm: 2.914299 [I 2022-10-08 13:49:50] ...trap/engines/engine.py.224: train: epoch 7 | batch 270/2304 [I 2022-10-08 13:49:50] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:16 [I 2022-10-08 13:49:50] ...trap/engines/engine.py.229: process: 0.78285 | load: 0.00302 [I 2022-10-08 13:49:50] ...trap/engines/engine.py.230: loss: 0.16733 [I 2022-10-08 13:49:50] ...optimizers/trijoint.py.151: total_norm: 2.227560 [I 2022-10-08 13:49:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 280/2304 [I 2022-10-08 13:49:58] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:11 [I 2022-10-08 13:49:58] ...trap/engines/engine.py.229: process: 0.76293 | load: 0.00302 [I 2022-10-08 13:49:58] ...trap/engines/engine.py.230: loss: 0.17078 [I 2022-10-08 13:49:58] ...optimizers/trijoint.py.151: total_norm: 2.577587 [I 2022-10-08 13:50:06] ...trap/engines/engine.py.224: train: epoch 7 | batch 290/2304 [I 2022-10-08 13:50:06] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:00 [I 2022-10-08 13:50:06] ...trap/engines/engine.py.229: process: 0.78393 | load: 0.00303 [I 2022-10-08 13:50:06] ...trap/engines/engine.py.230: loss: 0.18574 [I 2022-10-08 13:50:06] ...optimizers/trijoint.py.151: total_norm: 2.660301 [I 2022-10-08 13:50:14] ...trap/engines/engine.py.224: train: epoch 7 | batch 300/2304 [I 2022-10-08 13:50:14] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:25:53 [I 2022-10-08 13:50:14] ...trap/engines/engine.py.229: process: 0.75570 | load: 0.00306 [I 2022-10-08 13:50:14] ...trap/engines/engine.py.230: loss: 0.18047 [I 2022-10-08 13:50:14] ...optimizers/trijoint.py.151: total_norm: 2.535161 [I 2022-10-08 13:50:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 310/2304 [I 2022-10-08 13:50:22] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:26:05 [I 2022-10-08 13:50:22] ...trap/engines/engine.py.229: process: 0.79782 | load: 0.00304 [I 2022-10-08 13:50:22] ...trap/engines/engine.py.230: loss: 0.30579 [I 2022-10-08 13:50:22] ...optimizers/trijoint.py.151: total_norm: 2.774364 [I 2022-10-08 13:50:29] ...trap/engines/engine.py.224: train: epoch 7 | batch 320/2304 [I 2022-10-08 13:50:29] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:55 [I 2022-10-08 13:50:29] ...trap/engines/engine.py.229: process: 0.78989 | load: 0.00304 [I 2022-10-08 13:50:29] ...trap/engines/engine.py.230: loss: 0.17938 [I 2022-10-08 13:50:29] ...optimizers/trijoint.py.151: total_norm: 2.347768 [I 2022-10-08 13:50:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 330/2304 [I 2022-10-08 13:50:37] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:43 [I 2022-10-08 13:50:37] ...trap/engines/engine.py.229: process: 0.79076 | load: 0.00314 [I 2022-10-08 13:50:37] ...trap/engines/engine.py.230: loss: 0.21815 [I 2022-10-08 13:50:37] ...optimizers/trijoint.py.151: total_norm: 2.220282 [I 2022-10-08 13:50:45] ...trap/engines/engine.py.224: train: epoch 7 | batch 340/2304 [I 2022-10-08 13:50:45] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:21 [I 2022-10-08 13:50:45] ...trap/engines/engine.py.229: process: 0.76821 | load: 0.00306 [I 2022-10-08 13:50:45] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-08 13:50:45] ...optimizers/trijoint.py.151: total_norm: 2.669917 [I 2022-10-08 13:50:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 350/2304 [I 2022-10-08 13:50:53] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:19 [I 2022-10-08 13:50:53] ...trap/engines/engine.py.229: process: 0.77480 | load: 0.00303 [I 2022-10-08 13:50:53] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-08 13:50:53] ...optimizers/trijoint.py.151: total_norm: 2.143081 [I 2022-10-08 13:51:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 360/2304 [I 2022-10-08 13:51:01] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:18 [I 2022-10-08 13:51:01] ...trap/engines/engine.py.229: process: 0.77592 | load: 0.00304 [I 2022-10-08 13:51:01] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-08 13:51:01] ...optimizers/trijoint.py.151: total_norm: 2.000814 [I 2022-10-08 13:51:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 370/2304 [I 2022-10-08 13:51:08] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:08 [I 2022-10-08 13:51:08] ...trap/engines/engine.py.229: process: 0.78197 | load: 0.00302 [I 2022-10-08 13:51:08] ...trap/engines/engine.py.230: loss: 0.18935 [I 2022-10-08 13:51:08] ...optimizers/trijoint.py.151: total_norm: 2.627222 [I 2022-10-08 13:51:16] ...trap/engines/engine.py.224: train: epoch 7 | batch 380/2304 [I 2022-10-08 13:51:16] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:24:58 [I 2022-10-08 13:51:16] ...trap/engines/engine.py.229: process: 0.76395 | load: 0.00306 [I 2022-10-08 13:51:16] ...trap/engines/engine.py.230: loss: 0.19014 [I 2022-10-08 13:51:16] ...optimizers/trijoint.py.151: total_norm: 3.132170 [I 2022-10-08 13:51:24] ...trap/engines/engine.py.224: train: epoch 7 | batch 390/2304 [I 2022-10-08 13:51:24] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:58 [I 2022-10-08 13:51:24] ...trap/engines/engine.py.229: process: 0.78149 | load: 0.00306 [I 2022-10-08 13:51:24] ...trap/engines/engine.py.230: loss: 0.17848 [I 2022-10-08 13:51:24] ...optimizers/trijoint.py.151: total_norm: 2.283880 [I 2022-10-08 13:51:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 400/2304 [I 2022-10-08 13:51:32] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:40 [I 2022-10-08 13:51:32] ...trap/engines/engine.py.229: process: 0.78077 | load: 0.00303 [I 2022-10-08 13:51:32] ...trap/engines/engine.py.230: loss: 0.27217 [I 2022-10-08 13:51:32] ...optimizers/trijoint.py.151: total_norm: 2.556082 [I 2022-10-08 13:51:40] ...trap/engines/engine.py.224: train: epoch 7 | batch 410/2304 [I 2022-10-08 13:51:40] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:24:38 [I 2022-10-08 13:51:40] ...trap/engines/engine.py.229: process: 0.78505 | load: 0.00302 [I 2022-10-08 13:51:40] ...trap/engines/engine.py.230: loss: 0.28309 [I 2022-10-08 13:51:40] ...optimizers/trijoint.py.151: total_norm: 3.770696 [I 2022-10-08 13:51:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 420/2304 [I 2022-10-08 13:51:48] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:34 [I 2022-10-08 13:51:48] ...trap/engines/engine.py.229: process: 0.78899 | load: 0.00300 [I 2022-10-08 13:51:48] ...trap/engines/engine.py.230: loss: 0.12014 [I 2022-10-08 13:51:48] ...optimizers/trijoint.py.151: total_norm: 2.632408 [I 2022-10-08 13:51:55] ...trap/engines/engine.py.224: train: epoch 7 | batch 430/2304 [I 2022-10-08 13:51:55] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:22 [I 2022-10-08 13:51:55] ...trap/engines/engine.py.229: process: 0.78633 | load: 0.00301 [I 2022-10-08 13:51:55] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-08 13:51:55] ...optimizers/trijoint.py.151: total_norm: 2.491955 [I 2022-10-08 13:52:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 440/2304 [I 2022-10-08 13:52:03] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:09 [I 2022-10-08 13:52:03] ...trap/engines/engine.py.229: process: 0.77905 | load: 0.00302 [I 2022-10-08 13:52:03] ...trap/engines/engine.py.230: loss: 0.19393 [I 2022-10-08 13:52:03] ...optimizers/trijoint.py.151: total_norm: 2.750265 [I 2022-10-08 13:52:11] ...trap/engines/engine.py.224: train: epoch 7 | batch 450/2304 [I 2022-10-08 13:52:11] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:02 [I 2022-10-08 13:52:11] ...trap/engines/engine.py.229: process: 0.78896 | load: 0.00303 [I 2022-10-08 13:52:11] ...trap/engines/engine.py.230: loss: 0.23092 [I 2022-10-08 13:52:11] ...optimizers/trijoint.py.151: total_norm: 2.986624 [I 2022-10-08 13:52:19] ...trap/engines/engine.py.224: train: epoch 7 | batch 460/2304 [I 2022-10-08 13:52:19] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:54 [I 2022-10-08 13:52:19] ...trap/engines/engine.py.229: process: 0.76769 | load: 0.00313 [I 2022-10-08 13:52:19] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-08 13:52:19] ...optimizers/trijoint.py.151: total_norm: 2.178296 [I 2022-10-08 13:52:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 470/2304 [I 2022-10-08 13:52:27] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:56 [I 2022-10-08 13:52:27] ...trap/engines/engine.py.229: process: 0.78583 | load: 0.00312 [I 2022-10-08 13:52:27] ...trap/engines/engine.py.230: loss: 0.24325 [I 2022-10-08 13:52:27] ...optimizers/trijoint.py.151: total_norm: 3.068616 [I 2022-10-08 13:52:35] ...trap/engines/engine.py.224: train: epoch 7 | batch 480/2304 [I 2022-10-08 13:52:35] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:46 [I 2022-10-08 13:52:35] ...trap/engines/engine.py.229: process: 0.78295 | load: 0.00305 [I 2022-10-08 13:52:35] ...trap/engines/engine.py.230: loss: 0.26209 [I 2022-10-08 13:52:35] ...optimizers/trijoint.py.151: total_norm: 3.083606 [I 2022-10-08 13:52:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 490/2304 [I 2022-10-08 13:52:42] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:35 [I 2022-10-08 13:52:42] ...trap/engines/engine.py.229: process: 0.77508 | load: 0.00300 [I 2022-10-08 13:52:42] ...trap/engines/engine.py.230: loss: 0.22726 [I 2022-10-08 13:52:42] ...optimizers/trijoint.py.151: total_norm: 2.102705 [I 2022-10-08 13:52:50] ...trap/engines/engine.py.224: train: epoch 7 | batch 500/2304 [I 2022-10-08 13:52:50] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:33 [I 2022-10-08 13:52:50] ...trap/engines/engine.py.229: process: 0.78561 | load: 0.00305 [I 2022-10-08 13:52:50] ...trap/engines/engine.py.230: loss: 0.21130 [I 2022-10-08 13:52:50] ...optimizers/trijoint.py.151: total_norm: 2.431406 [I 2022-10-08 13:52:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 510/2304 [I 2022-10-08 13:52:58] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:16 [I 2022-10-08 13:52:58] ...trap/engines/engine.py.229: process: 0.77120 | load: 0.00309 [I 2022-10-08 13:52:58] ...trap/engines/engine.py.230: loss: 0.18136 [I 2022-10-08 13:52:58] ...optimizers/trijoint.py.151: total_norm: 2.510471 [I 2022-10-08 13:53:06] ...trap/engines/engine.py.224: train: epoch 7 | batch 520/2304 [I 2022-10-08 13:53:06] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:23:03 [I 2022-10-08 13:53:06] ...trap/engines/engine.py.229: process: 0.77756 | load: 0.00306 [I 2022-10-08 13:53:06] ...trap/engines/engine.py.230: loss: 0.18511 [I 2022-10-08 13:53:06] ...optimizers/trijoint.py.151: total_norm: 2.778419 [I 2022-10-08 13:53:14] ...trap/engines/engine.py.224: train: epoch 7 | batch 530/2304 [I 2022-10-08 13:53:14] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:02 [I 2022-10-08 13:53:14] ...trap/engines/engine.py.229: process: 0.77349 | load: 0.00299 [I 2022-10-08 13:53:14] ...trap/engines/engine.py.230: loss: 0.22831 [I 2022-10-08 13:53:14] ...optimizers/trijoint.py.151: total_norm: 3.213923 [I 2022-10-08 13:53:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 540/2304 [I 2022-10-08 13:53:21] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:56 [I 2022-10-08 13:53:21] ...trap/engines/engine.py.229: process: 0.78777 | load: 0.00304 [I 2022-10-08 13:53:21] ...trap/engines/engine.py.230: loss: 0.22763 [I 2022-10-08 13:53:21] ...optimizers/trijoint.py.151: total_norm: 2.542671 [I 2022-10-08 13:53:29] ...trap/engines/engine.py.224: train: epoch 7 | batch 550/2304 [I 2022-10-08 13:53:29] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:54 [I 2022-10-08 13:53:29] ...trap/engines/engine.py.229: process: 0.79097 | load: 0.00302 [I 2022-10-08 13:53:29] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-08 13:53:29] ...optimizers/trijoint.py.151: total_norm: 2.651694 [I 2022-10-08 13:53:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 560/2304 [I 2022-10-08 13:53:37] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:49 [I 2022-10-08 13:53:37] ...trap/engines/engine.py.229: process: 0.79714 | load: 0.00303 [I 2022-10-08 13:53:37] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-08 13:53:37] ...optimizers/trijoint.py.151: total_norm: 2.837978 [I 2022-10-08 13:53:45] ...trap/engines/engine.py.224: train: epoch 7 | batch 570/2304 [I 2022-10-08 13:53:45] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:37 [I 2022-10-08 13:53:45] ...trap/engines/engine.py.229: process: 0.77873 | load: 0.00309 [I 2022-10-08 13:53:45] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-08 13:53:45] ...optimizers/trijoint.py.151: total_norm: 1.605206 [I 2022-10-08 13:53:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 580/2304 [I 2022-10-08 13:53:53] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:33 [I 2022-10-08 13:53:53] ...trap/engines/engine.py.229: process: 0.79820 | load: 0.00328 [I 2022-10-08 13:53:53] ...trap/engines/engine.py.230: loss: 0.20967 [I 2022-10-08 13:53:53] ...optimizers/trijoint.py.151: total_norm: 2.416891 [I 2022-10-08 13:54:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 590/2304 [I 2022-10-08 13:54:01] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:15 [I 2022-10-08 13:54:01] ...trap/engines/engine.py.229: process: 0.77260 | load: 0.00301 [I 2022-10-08 13:54:01] ...trap/engines/engine.py.230: loss: 0.30981 [I 2022-10-08 13:54:01] ...optimizers/trijoint.py.151: total_norm: 3.046747 [I 2022-10-08 13:54:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 600/2304 [I 2022-10-08 13:54:08] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:13 [I 2022-10-08 13:54:08] ...trap/engines/engine.py.229: process: 0.79694 | load: 0.00302 [I 2022-10-08 13:54:08] ...trap/engines/engine.py.230: loss: 0.12788 [I 2022-10-08 13:54:08] ...optimizers/trijoint.py.151: total_norm: 2.058298 [I 2022-10-08 13:54:16] ...trap/engines/engine.py.224: train: epoch 7 | batch 610/2304 [I 2022-10-08 13:54:16] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:02 [I 2022-10-08 13:54:16] ...trap/engines/engine.py.229: process: 0.78870 | load: 0.00306 [I 2022-10-08 13:54:16] ...trap/engines/engine.py.230: loss: 0.21948 [I 2022-10-08 13:54:16] ...optimizers/trijoint.py.151: total_norm: 3.438277 [I 2022-10-08 13:54:24] ...trap/engines/engine.py.224: train: epoch 7 | batch 620/2304 [I 2022-10-08 13:54:24] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:54 [I 2022-10-08 13:54:24] ...trap/engines/engine.py.229: process: 0.77880 | load: 0.00306 [I 2022-10-08 13:54:24] ...trap/engines/engine.py.230: loss: 0.23041 [I 2022-10-08 13:54:24] ...optimizers/trijoint.py.151: total_norm: 2.379912 [I 2022-10-08 13:54:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 630/2304 [I 2022-10-08 13:54:32] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:48 [I 2022-10-08 13:54:32] ...trap/engines/engine.py.229: process: 0.79142 | load: 0.00302 [I 2022-10-08 13:54:32] ...trap/engines/engine.py.230: loss: 0.26257 [I 2022-10-08 13:54:32] ...optimizers/trijoint.py.151: total_norm: 2.922704 [I 2022-10-08 13:54:40] ...trap/engines/engine.py.224: train: epoch 7 | batch 640/2304 [I 2022-10-08 13:54:40] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:34 [I 2022-10-08 13:54:40] ...trap/engines/engine.py.229: process: 0.76433 | load: 0.00311 [I 2022-10-08 13:54:40] ...trap/engines/engine.py.230: loss: 0.21500 [I 2022-10-08 13:54:40] ...optimizers/trijoint.py.151: total_norm: 3.317506 [I 2022-10-08 13:54:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 650/2304 [I 2022-10-08 13:54:47] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:26 [I 2022-10-08 13:54:47] ...trap/engines/engine.py.229: process: 0.78526 | load: 0.00310 [I 2022-10-08 13:54:47] ...trap/engines/engine.py.230: loss: 0.10745 [I 2022-10-08 13:54:47] ...optimizers/trijoint.py.151: total_norm: 2.296926 [I 2022-10-08 13:54:55] ...trap/engines/engine.py.224: train: epoch 7 | batch 660/2304 [I 2022-10-08 13:54:55] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:13 [I 2022-10-08 13:54:55] ...trap/engines/engine.py.229: process: 0.76636 | load: 0.00311 [I 2022-10-08 13:54:55] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-08 13:54:55] ...optimizers/trijoint.py.151: total_norm: 1.906646 [I 2022-10-08 13:55:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 670/2304 [I 2022-10-08 13:55:03] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:06 [I 2022-10-08 13:55:03] ...trap/engines/engine.py.229: process: 0.76481 | load: 0.00312 [I 2022-10-08 13:55:03] ...trap/engines/engine.py.230: loss: 0.23740 [I 2022-10-08 13:55:03] ...optimizers/trijoint.py.151: total_norm: 4.189309 [I 2022-10-08 13:55:11] ...trap/engines/engine.py.224: train: epoch 7 | batch 680/2304 [I 2022-10-08 13:55:11] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:04 [I 2022-10-08 13:55:11] ...trap/engines/engine.py.229: process: 0.77509 | load: 0.00313 [I 2022-10-08 13:55:11] ...trap/engines/engine.py.230: loss: 0.22612 [I 2022-10-08 13:55:11] ...optimizers/trijoint.py.151: total_norm: 2.459032 [I 2022-10-08 13:55:19] ...trap/engines/engine.py.224: train: epoch 7 | batch 690/2304 [I 2022-10-08 13:55:19] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:58 [I 2022-10-08 13:55:19] ...trap/engines/engine.py.229: process: 0.77912 | load: 0.00320 [I 2022-10-08 13:55:19] ...trap/engines/engine.py.230: loss: 0.19773 [I 2022-10-08 13:55:19] ...optimizers/trijoint.py.151: total_norm: 2.384570 [I 2022-10-08 13:55:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 700/2304 [I 2022-10-08 13:55:26] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:50 [I 2022-10-08 13:55:26] ...trap/engines/engine.py.229: process: 0.77126 | load: 0.00314 [I 2022-10-08 13:55:26] ...trap/engines/engine.py.230: loss: 0.20730 [I 2022-10-08 13:55:26] ...optimizers/trijoint.py.151: total_norm: 2.360012 [I 2022-10-08 13:55:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 710/2304 [I 2022-10-08 13:55:34] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:38 [I 2022-10-08 13:55:34] ...trap/engines/engine.py.229: process: 0.78153 | load: 0.00310 [I 2022-10-08 13:55:34] ...trap/engines/engine.py.230: loss: 0.19231 [I 2022-10-08 13:55:34] ...optimizers/trijoint.py.151: total_norm: 2.839452 [I 2022-10-08 13:55:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 720/2304 [I 2022-10-08 13:55:42] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:25 [I 2022-10-08 13:55:42] ...trap/engines/engine.py.229: process: 0.75709 | load: 0.00316 [I 2022-10-08 13:55:42] ...trap/engines/engine.py.230: loss: 0.21168 [I 2022-10-08 13:55:42] ...optimizers/trijoint.py.151: total_norm: 2.974412 [I 2022-10-08 13:55:50] ...trap/engines/engine.py.224: train: epoch 7 | batch 730/2304 [I 2022-10-08 13:55:50] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:27 [I 2022-10-08 13:55:50] ...trap/engines/engine.py.229: process: 0.77280 | load: 0.00315 [I 2022-10-08 13:55:50] ...trap/engines/engine.py.230: loss: 0.23606 [I 2022-10-08 13:55:50] ...optimizers/trijoint.py.151: total_norm: 3.186321 [I 2022-10-08 13:55:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 740/2304 [I 2022-10-08 13:55:58] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:17 [I 2022-10-08 13:55:58] ...trap/engines/engine.py.229: process: 0.77822 | load: 0.00313 [I 2022-10-08 13:55:58] ...trap/engines/engine.py.230: loss: 0.18423 [I 2022-10-08 13:55:58] ...optimizers/trijoint.py.151: total_norm: 2.496103 [I 2022-10-08 13:56:05] ...trap/engines/engine.py.224: train: epoch 7 | batch 750/2304 [I 2022-10-08 13:56:05] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:04 [I 2022-10-08 13:56:05] ...trap/engines/engine.py.229: process: 0.77815 | load: 0.00313 [I 2022-10-08 13:56:05] ...trap/engines/engine.py.230: loss: 0.23489 [I 2022-10-08 13:56:05] ...optimizers/trijoint.py.151: total_norm: 2.428643 [I 2022-10-08 13:56:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 760/2304 [I 2022-10-08 13:56:13] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:03 [I 2022-10-08 13:56:13] ...trap/engines/engine.py.229: process: 0.77757 | load: 0.00320 [I 2022-10-08 13:56:13] ...trap/engines/engine.py.230: loss: 0.21064 [I 2022-10-08 13:56:13] ...optimizers/trijoint.py.151: total_norm: 2.735932 [I 2022-10-08 13:56:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 770/2304 [I 2022-10-08 13:56:21] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:48 [I 2022-10-08 13:56:21] ...trap/engines/engine.py.229: process: 0.76100 | load: 0.00309 [I 2022-10-08 13:56:21] ...trap/engines/engine.py.230: loss: 0.28260 [I 2022-10-08 13:56:21] ...optimizers/trijoint.py.151: total_norm: 3.591595 [I 2022-10-08 13:56:29] ...trap/engines/engine.py.224: train: epoch 7 | batch 780/2304 [I 2022-10-08 13:56:29] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:50 [I 2022-10-08 13:56:29] ...trap/engines/engine.py.229: process: 0.79239 | load: 0.00301 [I 2022-10-08 13:56:29] ...trap/engines/engine.py.230: loss: 0.24496 [I 2022-10-08 13:56:29] ...optimizers/trijoint.py.151: total_norm: 2.791153 [I 2022-10-08 13:56:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 790/2304 [I 2022-10-08 13:56:37] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:41 [I 2022-10-08 13:56:37] ...trap/engines/engine.py.229: process: 0.78529 | load: 0.00306 [I 2022-10-08 13:56:37] ...trap/engines/engine.py.230: loss: 0.22596 [I 2022-10-08 13:56:37] ...optimizers/trijoint.py.151: total_norm: 2.583986 [I 2022-10-08 13:56:44] ...trap/engines/engine.py.224: train: epoch 7 | batch 800/2304 [I 2022-10-08 13:56:44] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:30 [I 2022-10-08 13:56:44] ...trap/engines/engine.py.229: process: 0.76975 | load: 0.00301 [I 2022-10-08 13:56:44] ...trap/engines/engine.py.230: loss: 0.16792 [I 2022-10-08 13:56:44] ...optimizers/trijoint.py.151: total_norm: 2.465904 [I 2022-10-08 13:56:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 810/2304 [I 2022-10-08 13:56:52] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:24 [I 2022-10-08 13:56:52] ...trap/engines/engine.py.229: process: 0.77901 | load: 0.00302 [I 2022-10-08 13:56:52] ...trap/engines/engine.py.230: loss: 0.24424 [I 2022-10-08 13:56:52] ...optimizers/trijoint.py.151: total_norm: 3.016489 [I 2022-10-08 13:57:00] ...trap/engines/engine.py.224: train: epoch 7 | batch 820/2304 [I 2022-10-08 13:57:00] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:07 [I 2022-10-08 13:57:00] ...trap/engines/engine.py.229: process: 0.76996 | load: 0.00301 [I 2022-10-08 13:57:00] ...trap/engines/engine.py.230: loss: 0.31140 [I 2022-10-08 13:57:00] ...optimizers/trijoint.py.151: total_norm: 3.251442 [I 2022-10-08 13:57:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 830/2304 [I 2022-10-08 13:57:08] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:06 [I 2022-10-08 13:57:08] ...trap/engines/engine.py.229: process: 0.78606 | load: 0.00306 [I 2022-10-08 13:57:08] ...trap/engines/engine.py.230: loss: 0.26932 [I 2022-10-08 13:57:08] ...optimizers/trijoint.py.151: total_norm: 3.705452 [I 2022-10-08 13:57:16] ...trap/engines/engine.py.224: train: epoch 7 | batch 840/2304 [I 2022-10-08 13:57:16] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:18:58 [I 2022-10-08 13:57:16] ...trap/engines/engine.py.229: process: 0.78089 | load: 0.00306 [I 2022-10-08 13:57:16] ...trap/engines/engine.py.230: loss: 0.31018 [I 2022-10-08 13:57:16] ...optimizers/trijoint.py.151: total_norm: 3.403952 [I 2022-10-08 13:57:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 850/2304 [I 2022-10-08 13:57:23] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:56 [I 2022-10-08 13:57:23] ...trap/engines/engine.py.229: process: 0.78118 | load: 0.00312 [I 2022-10-08 13:57:23] ...trap/engines/engine.py.230: loss: 0.10813 [I 2022-10-08 13:57:23] ...optimizers/trijoint.py.151: total_norm: 1.698216 [I 2022-10-08 13:57:31] ...trap/engines/engine.py.224: train: epoch 7 | batch 860/2304 [I 2022-10-08 13:57:31] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:41 [I 2022-10-08 13:57:31] ...trap/engines/engine.py.229: process: 0.76861 | load: 0.00313 [I 2022-10-08 13:57:31] ...trap/engines/engine.py.230: loss: 0.23026 [I 2022-10-08 13:57:31] ...optimizers/trijoint.py.151: total_norm: 2.718610 [I 2022-10-08 13:57:39] ...trap/engines/engine.py.224: train: epoch 7 | batch 870/2304 [I 2022-10-08 13:57:39] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:41 [I 2022-10-08 13:57:39] ...trap/engines/engine.py.229: process: 0.79098 | load: 0.00307 [I 2022-10-08 13:57:39] ...trap/engines/engine.py.230: loss: 0.32314 [I 2022-10-08 13:57:39] ...optimizers/trijoint.py.151: total_norm: 3.250646 [I 2022-10-08 13:57:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 880/2304 [I 2022-10-08 13:57:47] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:30 [I 2022-10-08 13:57:47] ...trap/engines/engine.py.229: process: 0.77967 | load: 0.00315 [I 2022-10-08 13:57:47] ...trap/engines/engine.py.230: loss: 0.15325 [I 2022-10-08 13:57:47] ...optimizers/trijoint.py.151: total_norm: 2.485134 [I 2022-10-08 13:57:55] ...trap/engines/engine.py.224: train: epoch 7 | batch 890/2304 [I 2022-10-08 13:57:55] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:26 [I 2022-10-08 13:57:55] ...trap/engines/engine.py.229: process: 0.77947 | load: 0.00316 [I 2022-10-08 13:57:55] ...trap/engines/engine.py.230: loss: 0.27206 [I 2022-10-08 13:57:55] ...optimizers/trijoint.py.151: total_norm: 3.912486 [I 2022-10-08 13:58:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 900/2304 [I 2022-10-08 13:58:02] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:15 [I 2022-10-08 13:58:02] ...trap/engines/engine.py.229: process: 0.79508 | load: 0.00311 [I 2022-10-08 13:58:02] ...trap/engines/engine.py.230: loss: 0.32384 [I 2022-10-08 13:58:02] ...optimizers/trijoint.py.151: total_norm: 2.839741 [I 2022-10-08 13:58:10] ...trap/engines/engine.py.224: train: epoch 7 | batch 910/2304 [I 2022-10-08 13:58:10] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:09 [I 2022-10-08 13:58:10] ...trap/engines/engine.py.229: process: 0.78000 | load: 0.00312 [I 2022-10-08 13:58:10] ...trap/engines/engine.py.230: loss: 0.26515 [I 2022-10-08 13:58:10] ...optimizers/trijoint.py.151: total_norm: 2.875367 [I 2022-10-08 13:58:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 920/2304 [I 2022-10-08 13:58:18] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:18:01 [I 2022-10-08 13:58:18] ...trap/engines/engine.py.229: process: 0.79146 | load: 0.00301 [I 2022-10-08 13:58:18] ...trap/engines/engine.py.230: loss: 0.21896 [I 2022-10-08 13:58:18] ...optimizers/trijoint.py.151: total_norm: 2.403936 [I 2022-10-08 13:58:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 930/2304 [I 2022-10-08 13:58:26] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:50 [I 2022-10-08 13:58:26] ...trap/engines/engine.py.229: process: 0.79278 | load: 0.00305 [I 2022-10-08 13:58:26] ...trap/engines/engine.py.230: loss: 0.20214 [I 2022-10-08 13:58:26] ...optimizers/trijoint.py.151: total_norm: 2.101069 [I 2022-10-08 13:58:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 940/2304 [I 2022-10-08 13:58:34] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:35 [I 2022-10-08 13:58:34] ...trap/engines/engine.py.229: process: 0.76322 | load: 0.00302 [I 2022-10-08 13:58:34] ...trap/engines/engine.py.230: loss: 0.15939 [I 2022-10-08 13:58:34] ...optimizers/trijoint.py.151: total_norm: 1.951277 [I 2022-10-08 13:58:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 950/2304 [I 2022-10-08 13:58:42] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:33 [I 2022-10-08 13:58:42] ...trap/engines/engine.py.229: process: 0.78358 | load: 0.00301 [I 2022-10-08 13:58:42] ...trap/engines/engine.py.230: loss: 0.29185 [I 2022-10-08 13:58:42] ...optimizers/trijoint.py.151: total_norm: 2.947343 [I 2022-10-08 13:58:49] ...trap/engines/engine.py.224: train: epoch 7 | batch 960/2304 [I 2022-10-08 13:58:49] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:17:24 [I 2022-10-08 13:58:49] ...trap/engines/engine.py.229: process: 0.77521 | load: 0.00302 [I 2022-10-08 13:58:49] ...trap/engines/engine.py.230: loss: 0.23368 [I 2022-10-08 13:58:49] ...optimizers/trijoint.py.151: total_norm: 3.018590 [I 2022-10-08 13:58:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 970/2304 [I 2022-10-08 13:58:57] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:18 [I 2022-10-08 13:58:57] ...trap/engines/engine.py.229: process: 0.76875 | load: 0.00300 [I 2022-10-08 13:58:57] ...trap/engines/engine.py.230: loss: 0.14232 [I 2022-10-08 13:58:57] ...optimizers/trijoint.py.151: total_norm: 2.300529 [I 2022-10-08 13:59:05] ...trap/engines/engine.py.224: train: epoch 7 | batch 980/2304 [I 2022-10-08 13:59:05] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:12 [I 2022-10-08 13:59:05] ...trap/engines/engine.py.229: process: 0.78308 | load: 0.00305 [I 2022-10-08 13:59:05] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-08 13:59:05] ...optimizers/trijoint.py.151: total_norm: 2.385415 [I 2022-10-08 13:59:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 990/2304 [I 2022-10-08 13:59:13] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:17:02 [I 2022-10-08 13:59:13] ...trap/engines/engine.py.229: process: 0.78983 | load: 0.00302 [I 2022-10-08 13:59:13] ...trap/engines/engine.py.230: loss: 0.15915 [I 2022-10-08 13:59:13] ...optimizers/trijoint.py.151: total_norm: 2.192473 [I 2022-10-08 13:59:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 1000/2304 [I 2022-10-08 13:59:21] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:17:00 [I 2022-10-08 13:59:21] ...trap/engines/engine.py.229: process: 0.79383 | load: 0.00302 [I 2022-10-08 13:59:21] ...trap/engines/engine.py.230: loss: 0.14269 [I 2022-10-08 13:59:21] ...optimizers/trijoint.py.151: total_norm: 2.060481 [I 2022-10-08 13:59:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1010/2304 [I 2022-10-08 13:59:28] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:16:52 [I 2022-10-08 13:59:28] ...trap/engines/engine.py.229: process: 0.79386 | load: 0.00303 [I 2022-10-08 13:59:28] ...trap/engines/engine.py.230: loss: 0.29841 [I 2022-10-08 13:59:28] ...optimizers/trijoint.py.151: total_norm: 2.952154 [I 2022-10-08 13:59:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 1020/2304 [I 2022-10-08 13:59:36] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:16:44 [I 2022-10-08 13:59:36] ...trap/engines/engine.py.229: process: 0.77130 | load: 0.00308 [I 2022-10-08 13:59:36] ...trap/engines/engine.py.230: loss: 0.31117 [I 2022-10-08 13:59:36] ...optimizers/trijoint.py.151: total_norm: 3.076350 [I 2022-10-08 13:59:44] ...trap/engines/engine.py.224: train: epoch 7 | batch 1030/2304 [I 2022-10-08 13:59:44] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:25 [I 2022-10-08 13:59:44] ...trap/engines/engine.py.229: process: 0.76025 | load: 0.00305 [I 2022-10-08 13:59:44] ...trap/engines/engine.py.230: loss: 0.28364 [I 2022-10-08 13:59:44] ...optimizers/trijoint.py.151: total_norm: 2.409991 [I 2022-10-08 13:59:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 1040/2304 [I 2022-10-08 13:59:52] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:25 [I 2022-10-08 13:59:52] ...trap/engines/engine.py.229: process: 0.78513 | load: 0.00303 [I 2022-10-08 13:59:52] ...trap/engines/engine.py.230: loss: 0.21308 [I 2022-10-08 13:59:52] ...optimizers/trijoint.py.151: total_norm: 2.641422 [I 2022-10-08 14:00:00] ...trap/engines/engine.py.224: train: epoch 7 | batch 1050/2304 [I 2022-10-08 14:00:00] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:16:12 [I 2022-10-08 14:00:00] ...trap/engines/engine.py.229: process: 0.78275 | load: 0.00301 [I 2022-10-08 14:00:00] ...trap/engines/engine.py.230: loss: 0.20042 [I 2022-10-08 14:00:00] ...optimizers/trijoint.py.151: total_norm: 2.794425 [I 2022-10-08 14:00:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 1060/2304 [I 2022-10-08 14:00:07] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:16:04 [I 2022-10-08 14:00:07] ...trap/engines/engine.py.229: process: 0.78043 | load: 0.00302 [I 2022-10-08 14:00:07] ...trap/engines/engine.py.230: loss: 0.23915 [I 2022-10-08 14:00:07] ...optimizers/trijoint.py.151: total_norm: 3.907032 [I 2022-10-08 14:00:15] ...trap/engines/engine.py.224: train: epoch 7 | batch 1070/2304 [I 2022-10-08 14:00:15] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:16:02 [I 2022-10-08 14:00:15] ...trap/engines/engine.py.229: process: 0.77001 | load: 0.00304 [I 2022-10-08 14:00:15] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-08 14:00:15] ...optimizers/trijoint.py.151: total_norm: 2.042948 [I 2022-10-08 14:00:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1080/2304 [I 2022-10-08 14:00:23] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:59 [I 2022-10-08 14:00:23] ...trap/engines/engine.py.229: process: 0.77664 | load: 0.00301 [I 2022-10-08 14:00:23] ...trap/engines/engine.py.230: loss: 0.25553 [I 2022-10-08 14:00:23] ...optimizers/trijoint.py.151: total_norm: 2.992488 [I 2022-10-08 14:00:31] ...trap/engines/engine.py.224: train: epoch 7 | batch 1090/2304 [I 2022-10-08 14:00:31] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:52 [I 2022-10-08 14:00:31] ...trap/engines/engine.py.229: process: 0.77869 | load: 0.00298 [I 2022-10-08 14:00:31] ...trap/engines/engine.py.230: loss: 0.22185 [I 2022-10-08 14:00:31] ...optimizers/trijoint.py.151: total_norm: 2.616188 [I 2022-10-08 14:00:39] ...trap/engines/engine.py.224: train: epoch 7 | batch 1100/2304 [I 2022-10-08 14:00:39] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:47 [I 2022-10-08 14:00:39] ...trap/engines/engine.py.229: process: 0.78714 | load: 0.00303 [I 2022-10-08 14:00:39] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-08 14:00:39] ...optimizers/trijoint.py.151: total_norm: 2.171953 [I 2022-10-08 14:00:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 1110/2304 [I 2022-10-08 14:00:47] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:15:26 [I 2022-10-08 14:00:47] ...trap/engines/engine.py.229: process: 0.77268 | load: 0.00302 [I 2022-10-08 14:00:47] ...trap/engines/engine.py.230: loss: 0.18429 [I 2022-10-08 14:00:47] ...optimizers/trijoint.py.151: total_norm: 3.095950 [I 2022-10-08 14:00:54] ...trap/engines/engine.py.224: train: epoch 7 | batch 1120/2304 [I 2022-10-08 14:00:54] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:15:25 [I 2022-10-08 14:00:54] ...trap/engines/engine.py.229: process: 0.78236 | load: 0.00302 [I 2022-10-08 14:00:54] ...trap/engines/engine.py.230: loss: 0.23854 [I 2022-10-08 14:00:54] ...optimizers/trijoint.py.151: total_norm: 3.853657 [I 2022-10-08 14:01:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 1130/2304 [I 2022-10-08 14:01:02] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:15:17 [I 2022-10-08 14:01:02] ...trap/engines/engine.py.229: process: 0.78225 | load: 0.00302 [I 2022-10-08 14:01:02] ...trap/engines/engine.py.230: loss: 0.27589 [I 2022-10-08 14:01:02] ...optimizers/trijoint.py.151: total_norm: 3.057236 [I 2022-10-08 14:01:10] ...trap/engines/engine.py.224: train: epoch 7 | batch 1140/2304 [I 2022-10-08 14:01:10] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:07 [I 2022-10-08 14:01:10] ...trap/engines/engine.py.229: process: 0.76952 | load: 0.00304 [I 2022-10-08 14:01:10] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-08 14:01:10] ...optimizers/trijoint.py.151: total_norm: 1.964837 [I 2022-10-08 14:01:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1150/2304 [I 2022-10-08 14:01:18] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:15:02 [I 2022-10-08 14:01:18] ...trap/engines/engine.py.229: process: 0.78406 | load: 0.00301 [I 2022-10-08 14:01:18] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-08 14:01:18] ...optimizers/trijoint.py.151: total_norm: 2.032863 [I 2022-10-08 14:01:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 1160/2304 [I 2022-10-08 14:01:26] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:14:52 [I 2022-10-08 14:01:26] ...trap/engines/engine.py.229: process: 0.78029 | load: 0.00305 [I 2022-10-08 14:01:26] ...trap/engines/engine.py.230: loss: 0.22654 [I 2022-10-08 14:01:26] ...optimizers/trijoint.py.151: total_norm: 2.070653 [I 2022-10-08 14:01:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 1170/2304 [I 2022-10-08 14:01:34] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:14:46 [I 2022-10-08 14:01:34] ...trap/engines/engine.py.229: process: 0.78798 | load: 0.00306 [I 2022-10-08 14:01:34] ...trap/engines/engine.py.230: loss: 0.18025 [I 2022-10-08 14:01:34] ...optimizers/trijoint.py.151: total_norm: 2.275340 [I 2022-10-08 14:01:41] ...trap/engines/engine.py.224: train: epoch 7 | batch 1180/2304 [I 2022-10-08 14:01:41] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:14:31 [I 2022-10-08 14:01:41] ...trap/engines/engine.py.229: process: 0.78372 | load: 0.00300 [I 2022-10-08 14:01:41] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-08 14:01:41] ...optimizers/trijoint.py.151: total_norm: 2.049648 [I 2022-10-08 14:01:49] ...trap/engines/engine.py.224: train: epoch 7 | batch 1190/2304 [I 2022-10-08 14:01:49] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:14:30 [I 2022-10-08 14:01:49] ...trap/engines/engine.py.229: process: 0.78382 | load: 0.00302 [I 2022-10-08 14:01:49] ...trap/engines/engine.py.230: loss: 0.35727 [I 2022-10-08 14:01:49] ...optimizers/trijoint.py.151: total_norm: 3.949264 [I 2022-10-08 14:01:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 1200/2304 [I 2022-10-08 14:01:57] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:23 [I 2022-10-08 14:01:57] ...trap/engines/engine.py.229: process: 0.78662 | load: 0.00301 [I 2022-10-08 14:01:57] ...trap/engines/engine.py.230: loss: 0.22146 [I 2022-10-08 14:01:57] ...optimizers/trijoint.py.151: total_norm: 2.869346 [I 2022-10-08 14:02:05] ...trap/engines/engine.py.224: train: epoch 7 | batch 1210/2304 [I 2022-10-08 14:02:05] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:14:10 [I 2022-10-08 14:02:05] ...trap/engines/engine.py.229: process: 0.76878 | load: 0.00300 [I 2022-10-08 14:02:05] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-08 14:02:05] ...optimizers/trijoint.py.151: total_norm: 1.873627 [I 2022-10-08 14:02:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1220/2304 [I 2022-10-08 14:02:13] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:14:03 [I 2022-10-08 14:02:13] ...trap/engines/engine.py.229: process: 0.78621 | load: 0.00298 [I 2022-10-08 14:02:13] ...trap/engines/engine.py.230: loss: 0.29526 [I 2022-10-08 14:02:13] ...optimizers/trijoint.py.151: total_norm: 3.336302 [I 2022-10-08 14:02:20] ...trap/engines/engine.py.224: train: epoch 7 | batch 1230/2304 [I 2022-10-08 14:02:20] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:13:54 [I 2022-10-08 14:02:20] ...trap/engines/engine.py.229: process: 0.77306 | load: 0.00298 [I 2022-10-08 14:02:20] ...trap/engines/engine.py.230: loss: 0.21788 [I 2022-10-08 14:02:20] ...optimizers/trijoint.py.151: total_norm: 3.186207 [I 2022-10-08 14:02:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1240/2304 [I 2022-10-08 14:02:28] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:13:52 [I 2022-10-08 14:02:28] ...trap/engines/engine.py.229: process: 0.77594 | load: 0.00300 [I 2022-10-08 14:02:28] ...trap/engines/engine.py.230: loss: 0.24224 [I 2022-10-08 14:02:28] ...optimizers/trijoint.py.151: total_norm: 3.073198 [I 2022-10-08 14:02:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 1250/2304 [I 2022-10-08 14:02:36] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:13:37 [I 2022-10-08 14:02:36] ...trap/engines/engine.py.229: process: 0.78320 | load: 0.00304 [I 2022-10-08 14:02:36] ...trap/engines/engine.py.230: loss: 0.17228 [I 2022-10-08 14:02:36] ...optimizers/trijoint.py.151: total_norm: 2.161793 [I 2022-10-08 14:02:44] ...trap/engines/engine.py.224: train: epoch 7 | batch 1260/2304 [I 2022-10-08 14:02:44] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:31 [I 2022-10-08 14:02:44] ...trap/engines/engine.py.229: process: 0.77099 | load: 0.00302 [I 2022-10-08 14:02:44] ...trap/engines/engine.py.230: loss: 0.20610 [I 2022-10-08 14:02:44] ...optimizers/trijoint.py.151: total_norm: 3.004943 [I 2022-10-08 14:02:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 1270/2304 [I 2022-10-08 14:02:52] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:13:23 [I 2022-10-08 14:02:52] ...trap/engines/engine.py.229: process: 0.77522 | load: 0.00304 [I 2022-10-08 14:02:52] ...trap/engines/engine.py.230: loss: 0.22022 [I 2022-10-08 14:02:52] ...optimizers/trijoint.py.151: total_norm: 2.332519 [I 2022-10-08 14:02:59] ...trap/engines/engine.py.224: train: epoch 7 | batch 1280/2304 [I 2022-10-08 14:02:59] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:13:17 [I 2022-10-08 14:02:59] ...trap/engines/engine.py.229: process: 0.77705 | load: 0.00310 [I 2022-10-08 14:02:59] ...trap/engines/engine.py.230: loss: 0.15550 [I 2022-10-08 14:02:59] ...optimizers/trijoint.py.151: total_norm: 2.043504 [I 2022-10-08 14:03:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 1290/2304 [I 2022-10-08 14:03:07] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:13:07 [I 2022-10-08 14:03:07] ...trap/engines/engine.py.229: process: 0.77185 | load: 0.00306 [I 2022-10-08 14:03:07] ...trap/engines/engine.py.230: loss: 0.21123 [I 2022-10-08 14:03:07] ...optimizers/trijoint.py.151: total_norm: 2.248356 [I 2022-10-08 14:03:15] ...trap/engines/engine.py.224: train: epoch 7 | batch 1300/2304 [I 2022-10-08 14:03:15] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:13:05 [I 2022-10-08 14:03:15] ...trap/engines/engine.py.229: process: 0.77953 | load: 0.00305 [I 2022-10-08 14:03:15] ...trap/engines/engine.py.230: loss: 0.21995 [I 2022-10-08 14:03:15] ...optimizers/trijoint.py.151: total_norm: 2.295495 [I 2022-10-08 14:03:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1310/2304 [I 2022-10-08 14:03:23] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:57 [I 2022-10-08 14:03:23] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00302 [I 2022-10-08 14:03:23] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-08 14:03:23] ...optimizers/trijoint.py.151: total_norm: 2.909652 [I 2022-10-08 14:03:31] ...trap/engines/engine.py.224: train: epoch 7 | batch 1320/2304 [I 2022-10-08 14:03:31] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:12:48 [I 2022-10-08 14:03:31] ...trap/engines/engine.py.229: process: 0.77256 | load: 0.00301 [I 2022-10-08 14:03:31] ...trap/engines/engine.py.230: loss: 0.25132 [I 2022-10-08 14:03:31] ...optimizers/trijoint.py.151: total_norm: 3.132246 [I 2022-10-08 14:03:39] ...trap/engines/engine.py.224: train: epoch 7 | batch 1330/2304 [I 2022-10-08 14:03:39] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:12:39 [I 2022-10-08 14:03:39] ...trap/engines/engine.py.229: process: 0.77278 | load: 0.00304 [I 2022-10-08 14:03:39] ...trap/engines/engine.py.230: loss: 0.22626 [I 2022-10-08 14:03:39] ...optimizers/trijoint.py.151: total_norm: 3.738226 [I 2022-10-08 14:03:46] ...trap/engines/engine.py.224: train: epoch 7 | batch 1340/2304 [I 2022-10-08 14:03:46] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:12:38 [I 2022-10-08 14:03:46] ...trap/engines/engine.py.229: process: 0.79256 | load: 0.00305 [I 2022-10-08 14:03:46] ...trap/engines/engine.py.230: loss: 0.19282 [I 2022-10-08 14:03:46] ...optimizers/trijoint.py.151: total_norm: 2.492329 [I 2022-10-08 14:03:54] ...trap/engines/engine.py.224: train: epoch 7 | batch 1350/2304 [I 2022-10-08 14:03:54] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:12:27 [I 2022-10-08 14:03:54] ...trap/engines/engine.py.229: process: 0.79290 | load: 0.00302 [I 2022-10-08 14:03:54] ...trap/engines/engine.py.230: loss: 0.18490 [I 2022-10-08 14:03:54] ...optimizers/trijoint.py.151: total_norm: 2.729136 [I 2022-10-08 14:04:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 1360/2304 [I 2022-10-08 14:04:02] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:19 [I 2022-10-08 14:04:02] ...trap/engines/engine.py.229: process: 0.78307 | load: 0.00304 [I 2022-10-08 14:04:02] ...trap/engines/engine.py.230: loss: 0.16348 [I 2022-10-08 14:04:02] ...optimizers/trijoint.py.151: total_norm: 2.509772 [I 2022-10-08 14:04:10] ...trap/engines/engine.py.224: train: epoch 7 | batch 1370/2304 [I 2022-10-08 14:04:10] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:15 [I 2022-10-08 14:04:10] ...trap/engines/engine.py.229: process: 0.78402 | load: 0.00307 [I 2022-10-08 14:04:10] ...trap/engines/engine.py.230: loss: 0.26214 [I 2022-10-08 14:04:10] ...optimizers/trijoint.py.151: total_norm: 3.104555 [I 2022-10-08 14:04:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1380/2304 [I 2022-10-08 14:04:18] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:00 [I 2022-10-08 14:04:18] ...trap/engines/engine.py.229: process: 0.78206 | load: 0.00306 [I 2022-10-08 14:04:18] ...trap/engines/engine.py.230: loss: 0.25387 [I 2022-10-08 14:04:18] ...optimizers/trijoint.py.151: total_norm: 2.961086 [I 2022-10-08 14:04:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 1390/2304 [I 2022-10-08 14:04:26] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:11:57 [I 2022-10-08 14:04:26] ...trap/engines/engine.py.229: process: 0.78798 | load: 0.00302 [I 2022-10-08 14:04:26] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-08 14:04:26] ...optimizers/trijoint.py.151: total_norm: 2.179949 [I 2022-10-08 14:04:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 1400/2304 [I 2022-10-08 14:04:34] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:11:47 [I 2022-10-08 14:04:34] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00305 [I 2022-10-08 14:04:34] ...trap/engines/engine.py.230: loss: 0.27883 [I 2022-10-08 14:04:34] ...optimizers/trijoint.py.151: total_norm: 3.839828 [I 2022-10-08 14:04:41] ...trap/engines/engine.py.224: train: epoch 7 | batch 1410/2304 [I 2022-10-08 14:04:41] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:11:35 [I 2022-10-08 14:04:41] ...trap/engines/engine.py.229: process: 0.77233 | load: 0.00310 [I 2022-10-08 14:04:41] ...trap/engines/engine.py.230: loss: 0.19084 [I 2022-10-08 14:04:41] ...optimizers/trijoint.py.151: total_norm: 2.465963 [I 2022-10-08 14:04:49] ...trap/engines/engine.py.224: train: epoch 7 | batch 1420/2304 [I 2022-10-08 14:04:49] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:30 [I 2022-10-08 14:04:49] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00305 [I 2022-10-08 14:04:49] ...trap/engines/engine.py.230: loss: 0.20149 [I 2022-10-08 14:04:49] ...optimizers/trijoint.py.151: total_norm: 2.088803 [I 2022-10-08 14:04:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 1430/2304 [I 2022-10-08 14:04:57] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:19 [I 2022-10-08 14:04:57] ...trap/engines/engine.py.229: process: 0.77731 | load: 0.00302 [I 2022-10-08 14:04:57] ...trap/engines/engine.py.230: loss: 0.16625 [I 2022-10-08 14:04:57] ...optimizers/trijoint.py.151: total_norm: 2.202654 [I 2022-10-08 14:05:05] ...trap/engines/engine.py.224: train: epoch 7 | batch 1440/2304 [I 2022-10-08 14:05:05] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:17 [I 2022-10-08 14:05:05] ...trap/engines/engine.py.229: process: 0.77990 | load: 0.00306 [I 2022-10-08 14:05:05] ...trap/engines/engine.py.230: loss: 0.18006 [I 2022-10-08 14:05:05] ...optimizers/trijoint.py.151: total_norm: 2.519943 [I 2022-10-08 14:05:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1450/2304 [I 2022-10-08 14:05:13] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:11:12 [I 2022-10-08 14:05:13] ...trap/engines/engine.py.229: process: 0.78238 | load: 0.00301 [I 2022-10-08 14:05:13] ...trap/engines/engine.py.230: loss: 0.25231 [I 2022-10-08 14:05:13] ...optimizers/trijoint.py.151: total_norm: 4.576081 [I 2022-10-08 14:05:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 1460/2304 [I 2022-10-08 14:05:21] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:10:57 [I 2022-10-08 14:05:21] ...trap/engines/engine.py.229: process: 0.79040 | load: 0.00314 [I 2022-10-08 14:05:21] ...trap/engines/engine.py.230: loss: 0.20626 [I 2022-10-08 14:05:21] ...optimizers/trijoint.py.151: total_norm: 2.800148 [I 2022-10-08 14:05:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1470/2304 [I 2022-10-08 14:05:28] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:10:48 [I 2022-10-08 14:05:28] ...trap/engines/engine.py.229: process: 0.76870 | load: 0.00304 [I 2022-10-08 14:05:28] ...trap/engines/engine.py.230: loss: 0.23729 [I 2022-10-08 14:05:28] ...optimizers/trijoint.py.151: total_norm: 2.493181 [I 2022-10-08 14:05:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 1480/2304 [I 2022-10-08 14:05:36] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:39 [I 2022-10-08 14:05:36] ...trap/engines/engine.py.229: process: 0.77941 | load: 0.00300 [I 2022-10-08 14:05:36] ...trap/engines/engine.py.230: loss: 0.32469 [I 2022-10-08 14:05:36] ...optimizers/trijoint.py.151: total_norm: 3.336128 [I 2022-10-08 14:05:44] ...trap/engines/engine.py.224: train: epoch 7 | batch 1490/2304 [I 2022-10-08 14:05:44] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:33 [I 2022-10-08 14:05:44] ...trap/engines/engine.py.229: process: 0.77889 | load: 0.00299 [I 2022-10-08 14:05:44] ...trap/engines/engine.py.230: loss: 0.24053 [I 2022-10-08 14:05:44] ...optimizers/trijoint.py.151: total_norm: 3.045542 [I 2022-10-08 14:05:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 1500/2304 [I 2022-10-08 14:05:52] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:26 [I 2022-10-08 14:05:52] ...trap/engines/engine.py.229: process: 0.78601 | load: 0.00301 [I 2022-10-08 14:05:52] ...trap/engines/engine.py.230: loss: 0.14714 [I 2022-10-08 14:05:52] ...optimizers/trijoint.py.151: total_norm: 1.821468 [I 2022-10-08 14:06:00] ...trap/engines/engine.py.224: train: epoch 7 | batch 1510/2304 [I 2022-10-08 14:06:00] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:10:21 [I 2022-10-08 14:06:00] ...trap/engines/engine.py.229: process: 0.78817 | load: 0.00307 [I 2022-10-08 14:06:00] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-08 14:06:00] ...optimizers/trijoint.py.151: total_norm: 2.887397 [I 2022-10-08 14:06:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 1520/2304 [I 2022-10-08 14:06:07] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:10:12 [I 2022-10-08 14:06:07] ...trap/engines/engine.py.229: process: 0.78015 | load: 0.00300 [I 2022-10-08 14:06:07] ...trap/engines/engine.py.230: loss: 0.13949 [I 2022-10-08 14:06:07] ...optimizers/trijoint.py.151: total_norm: 2.389473 [I 2022-10-08 14:06:15] ...trap/engines/engine.py.224: train: epoch 7 | batch 1530/2304 [I 2022-10-08 14:06:15] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:04 [I 2022-10-08 14:06:15] ...trap/engines/engine.py.229: process: 0.78271 | load: 0.00302 [I 2022-10-08 14:06:15] ...trap/engines/engine.py.230: loss: 0.28347 [I 2022-10-08 14:06:15] ...optimizers/trijoint.py.151: total_norm: 3.133444 [I 2022-10-08 14:06:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1540/2304 [I 2022-10-08 14:06:23] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:57 [I 2022-10-08 14:06:23] ...trap/engines/engine.py.229: process: 0.77789 | load: 0.00303 [I 2022-10-08 14:06:23] ...trap/engines/engine.py.230: loss: 0.09937 [I 2022-10-08 14:06:23] ...optimizers/trijoint.py.151: total_norm: 2.108182 [I 2022-10-08 14:06:31] ...trap/engines/engine.py.224: train: epoch 7 | batch 1550/2304 [I 2022-10-08 14:06:31] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:49 [I 2022-10-08 14:06:31] ...trap/engines/engine.py.229: process: 0.77926 | load: 0.00305 [I 2022-10-08 14:06:31] ...trap/engines/engine.py.230: loss: 0.24694 [I 2022-10-08 14:06:31] ...optimizers/trijoint.py.151: total_norm: 2.572061 [I 2022-10-08 14:06:39] ...trap/engines/engine.py.224: train: epoch 7 | batch 1560/2304 [I 2022-10-08 14:06:39] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:40 [I 2022-10-08 14:06:39] ...trap/engines/engine.py.229: process: 0.77054 | load: 0.00299 [I 2022-10-08 14:06:39] ...trap/engines/engine.py.230: loss: 0.28806 [I 2022-10-08 14:06:39] ...optimizers/trijoint.py.151: total_norm: 3.118770 [I 2022-10-08 14:06:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 1570/2304 [I 2022-10-08 14:06:47] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:34 [I 2022-10-08 14:06:47] ...trap/engines/engine.py.229: process: 0.78354 | load: 0.00301 [I 2022-10-08 14:06:47] ...trap/engines/engine.py.230: loss: 0.22434 [I 2022-10-08 14:06:47] ...optimizers/trijoint.py.151: total_norm: 3.251377 [I 2022-10-08 14:06:54] ...trap/engines/engine.py.224: train: epoch 7 | batch 1580/2304 [I 2022-10-08 14:06:54] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:26 [I 2022-10-08 14:06:54] ...trap/engines/engine.py.229: process: 0.78304 | load: 0.00302 [I 2022-10-08 14:06:54] ...trap/engines/engine.py.230: loss: 0.22532 [I 2022-10-08 14:06:54] ...optimizers/trijoint.py.151: total_norm: 2.510838 [I 2022-10-08 14:07:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 1590/2304 [I 2022-10-08 14:07:02] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:19 [I 2022-10-08 14:07:02] ...trap/engines/engine.py.229: process: 0.78092 | load: 0.00307 [I 2022-10-08 14:07:02] ...trap/engines/engine.py.230: loss: 0.17354 [I 2022-10-08 14:07:02] ...optimizers/trijoint.py.151: total_norm: 2.471965 [I 2022-10-08 14:07:10] ...trap/engines/engine.py.224: train: epoch 7 | batch 1600/2304 [I 2022-10-08 14:07:10] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:10 [I 2022-10-08 14:07:10] ...trap/engines/engine.py.229: process: 0.78206 | load: 0.00302 [I 2022-10-08 14:07:10] ...trap/engines/engine.py.230: loss: 0.17532 [I 2022-10-08 14:07:10] ...optimizers/trijoint.py.151: total_norm: 1.863372 [I 2022-10-08 14:07:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1610/2304 [I 2022-10-08 14:07:18] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:09:02 [I 2022-10-08 14:07:18] ...trap/engines/engine.py.229: process: 0.78835 | load: 0.00300 [I 2022-10-08 14:07:18] ...trap/engines/engine.py.230: loss: 0.16019 [I 2022-10-08 14:07:18] ...optimizers/trijoint.py.151: total_norm: 2.119588 [I 2022-10-08 14:07:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 1620/2304 [I 2022-10-08 14:07:26] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:52 [I 2022-10-08 14:07:26] ...trap/engines/engine.py.229: process: 0.77246 | load: 0.00304 [I 2022-10-08 14:07:26] ...trap/engines/engine.py.230: loss: 0.22662 [I 2022-10-08 14:07:26] ...optimizers/trijoint.py.151: total_norm: 2.361155 [I 2022-10-08 14:07:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 1630/2304 [I 2022-10-08 14:07:34] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:45 [I 2022-10-08 14:07:34] ...trap/engines/engine.py.229: process: 0.78370 | load: 0.00303 [I 2022-10-08 14:07:34] ...trap/engines/engine.py.230: loss: 0.24875 [I 2022-10-08 14:07:34] ...optimizers/trijoint.py.151: total_norm: 3.672062 [I 2022-10-08 14:07:41] ...trap/engines/engine.py.224: train: epoch 7 | batch 1640/2304 [I 2022-10-08 14:07:41] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:39 [I 2022-10-08 14:07:41] ...trap/engines/engine.py.229: process: 0.78058 | load: 0.00304 [I 2022-10-08 14:07:41] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-08 14:07:41] ...optimizers/trijoint.py.151: total_norm: 2.662113 [I 2022-10-08 14:07:49] ...trap/engines/engine.py.224: train: epoch 7 | batch 1650/2304 [I 2022-10-08 14:07:49] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:28 [I 2022-10-08 14:07:49] ...trap/engines/engine.py.229: process: 0.78220 | load: 0.00312 [I 2022-10-08 14:07:49] ...trap/engines/engine.py.230: loss: 0.17544 [I 2022-10-08 14:07:49] ...optimizers/trijoint.py.151: total_norm: 2.743911 [I 2022-10-08 14:07:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 1660/2304 [I 2022-10-08 14:07:57] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:21 [I 2022-10-08 14:07:57] ...trap/engines/engine.py.229: process: 0.78052 | load: 0.00306 [I 2022-10-08 14:07:57] ...trap/engines/engine.py.230: loss: 0.18535 [I 2022-10-08 14:07:57] ...optimizers/trijoint.py.151: total_norm: 2.192784 [I 2022-10-08 14:08:05] ...trap/engines/engine.py.224: train: epoch 7 | batch 1670/2304 [I 2022-10-08 14:08:05] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:12 [I 2022-10-08 14:08:05] ...trap/engines/engine.py.229: process: 0.77470 | load: 0.00304 [I 2022-10-08 14:08:05] ...trap/engines/engine.py.230: loss: 0.20847 [I 2022-10-08 14:08:05] ...optimizers/trijoint.py.151: total_norm: 2.742732 [I 2022-10-08 14:08:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1680/2304 [I 2022-10-08 14:08:13] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:03 [I 2022-10-08 14:08:13] ...trap/engines/engine.py.229: process: 0.77892 | load: 0.00307 [I 2022-10-08 14:08:13] ...trap/engines/engine.py.230: loss: 0.28126 [I 2022-10-08 14:08:13] ...optimizers/trijoint.py.151: total_norm: 3.677401 [I 2022-10-08 14:08:20] ...trap/engines/engine.py.224: train: epoch 7 | batch 1690/2304 [I 2022-10-08 14:08:20] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:07:57 [I 2022-10-08 14:08:20] ...trap/engines/engine.py.229: process: 0.77436 | load: 0.00304 [I 2022-10-08 14:08:20] ...trap/engines/engine.py.230: loss: 0.26865 [I 2022-10-08 14:08:20] ...optimizers/trijoint.py.151: total_norm: 3.871351 [I 2022-10-08 14:08:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1700/2304 [I 2022-10-08 14:08:28] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:48 [I 2022-10-08 14:08:28] ...trap/engines/engine.py.229: process: 0.77995 | load: 0.00304 [I 2022-10-08 14:08:28] ...trap/engines/engine.py.230: loss: 0.26639 [I 2022-10-08 14:08:28] ...optimizers/trijoint.py.151: total_norm: 3.178632 [I 2022-10-08 14:08:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 1710/2304 [I 2022-10-08 14:08:36] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:39 [I 2022-10-08 14:08:36] ...trap/engines/engine.py.229: process: 0.76543 | load: 0.00303 [I 2022-10-08 14:08:36] ...trap/engines/engine.py.230: loss: 0.19633 [I 2022-10-08 14:08:36] ...optimizers/trijoint.py.151: total_norm: 2.086381 [I 2022-10-08 14:08:44] ...trap/engines/engine.py.224: train: epoch 7 | batch 1720/2304 [I 2022-10-08 14:08:44] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:07:33 [I 2022-10-08 14:08:44] ...trap/engines/engine.py.229: process: 0.78037 | load: 0.00302 [I 2022-10-08 14:08:44] ...trap/engines/engine.py.230: loss: 0.16382 [I 2022-10-08 14:08:44] ...optimizers/trijoint.py.151: total_norm: 2.151222 [I 2022-10-08 14:08:51] ...trap/engines/engine.py.224: train: epoch 7 | batch 1730/2304 [I 2022-10-08 14:08:51] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:25 [I 2022-10-08 14:08:51] ...trap/engines/engine.py.229: process: 0.78023 | load: 0.00307 [I 2022-10-08 14:08:51] ...trap/engines/engine.py.230: loss: 0.17316 [I 2022-10-08 14:08:51] ...optimizers/trijoint.py.151: total_norm: 2.389085 [I 2022-10-08 14:08:59] ...trap/engines/engine.py.224: train: epoch 7 | batch 1740/2304 [I 2022-10-08 14:08:59] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:19 [I 2022-10-08 14:08:59] ...trap/engines/engine.py.229: process: 0.78121 | load: 0.00303 [I 2022-10-08 14:08:59] ...trap/engines/engine.py.230: loss: 0.22456 [I 2022-10-08 14:08:59] ...optimizers/trijoint.py.151: total_norm: 1.819515 [I 2022-10-08 14:09:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 1750/2304 [I 2022-10-08 14:09:07] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:11 [I 2022-10-08 14:09:07] ...trap/engines/engine.py.229: process: 0.77924 | load: 0.00309 [I 2022-10-08 14:09:07] ...trap/engines/engine.py.230: loss: 0.20189 [I 2022-10-08 14:09:07] ...optimizers/trijoint.py.151: total_norm: 2.750973 [I 2022-10-08 14:09:15] ...trap/engines/engine.py.224: train: epoch 7 | batch 1760/2304 [I 2022-10-08 14:09:15] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:01 [I 2022-10-08 14:09:15] ...trap/engines/engine.py.229: process: 0.77493 | load: 0.00300 [I 2022-10-08 14:09:15] ...trap/engines/engine.py.230: loss: 0.19289 [I 2022-10-08 14:09:15] ...optimizers/trijoint.py.151: total_norm: 3.080385 [I 2022-10-08 14:09:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1770/2304 [I 2022-10-08 14:09:23] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:52 [I 2022-10-08 14:09:23] ...trap/engines/engine.py.229: process: 0.77406 | load: 0.00304 [I 2022-10-08 14:09:23] ...trap/engines/engine.py.230: loss: 0.15824 [I 2022-10-08 14:09:23] ...optimizers/trijoint.py.151: total_norm: 2.521015 [I 2022-10-08 14:09:30] ...trap/engines/engine.py.224: train: epoch 7 | batch 1780/2304 [I 2022-10-08 14:09:30] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:46 [I 2022-10-08 14:09:30] ...trap/engines/engine.py.229: process: 0.78156 | load: 0.00302 [I 2022-10-08 14:09:30] ...trap/engines/engine.py.230: loss: 0.20518 [I 2022-10-08 14:09:30] ...optimizers/trijoint.py.151: total_norm: 2.968968 [I 2022-10-08 14:09:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1790/2304 [I 2022-10-08 14:09:38] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:38 [I 2022-10-08 14:09:38] ...trap/engines/engine.py.229: process: 0.78268 | load: 0.00302 [I 2022-10-08 14:09:38] ...trap/engines/engine.py.230: loss: 0.24149 [I 2022-10-08 14:09:38] ...optimizers/trijoint.py.151: total_norm: 2.682911 [I 2022-10-08 14:09:46] ...trap/engines/engine.py.224: train: epoch 7 | batch 1800/2304 [I 2022-10-08 14:09:46] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:33 [I 2022-10-08 14:09:46] ...trap/engines/engine.py.229: process: 0.78394 | load: 0.00312 [I 2022-10-08 14:09:46] ...trap/engines/engine.py.230: loss: 0.18872 [I 2022-10-08 14:09:46] ...optimizers/trijoint.py.151: total_norm: 2.718907 [I 2022-10-08 14:09:54] ...trap/engines/engine.py.224: train: epoch 7 | batch 1810/2304 [I 2022-10-08 14:09:54] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:23 [I 2022-10-08 14:09:54] ...trap/engines/engine.py.229: process: 0.77752 | load: 0.00304 [I 2022-10-08 14:09:54] ...trap/engines/engine.py.230: loss: 0.26755 [I 2022-10-08 14:09:54] ...optimizers/trijoint.py.151: total_norm: 3.583877 [I 2022-10-08 14:10:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 1820/2304 [I 2022-10-08 14:10:02] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:06:16 [I 2022-10-08 14:10:02] ...trap/engines/engine.py.229: process: 0.76970 | load: 0.00305 [I 2022-10-08 14:10:02] ...trap/engines/engine.py.230: loss: 0.24728 [I 2022-10-08 14:10:02] ...optimizers/trijoint.py.151: total_norm: 3.319300 [I 2022-10-08 14:10:09] ...trap/engines/engine.py.224: train: epoch 7 | batch 1830/2304 [I 2022-10-08 14:10:09] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:07 [I 2022-10-08 14:10:09] ...trap/engines/engine.py.229: process: 0.76498 | load: 0.00302 [I 2022-10-08 14:10:09] ...trap/engines/engine.py.230: loss: 0.20784 [I 2022-10-08 14:10:09] ...optimizers/trijoint.py.151: total_norm: 2.904711 [I 2022-10-08 14:10:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 1840/2304 [I 2022-10-08 14:10:17] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:01 [I 2022-10-08 14:10:17] ...trap/engines/engine.py.229: process: 0.78025 | load: 0.00304 [I 2022-10-08 14:10:17] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-08 14:10:17] ...optimizers/trijoint.py.151: total_norm: 1.755852 [I 2022-10-08 14:10:25] ...trap/engines/engine.py.224: train: epoch 7 | batch 1850/2304 [I 2022-10-08 14:10:25] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:05:52 [I 2022-10-08 14:10:25] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00303 [I 2022-10-08 14:10:25] ...trap/engines/engine.py.230: loss: 0.18757 [I 2022-10-08 14:10:25] ...optimizers/trijoint.py.151: total_norm: 2.285722 [I 2022-10-08 14:10:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1860/2304 [I 2022-10-08 14:10:33] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:05:43 [I 2022-10-08 14:10:33] ...trap/engines/engine.py.229: process: 0.76936 | load: 0.00302 [I 2022-10-08 14:10:33] ...trap/engines/engine.py.230: loss: 0.26073 [I 2022-10-08 14:10:33] ...optimizers/trijoint.py.151: total_norm: 3.303443 [I 2022-10-08 14:10:40] ...trap/engines/engine.py.224: train: epoch 7 | batch 1870/2304 [I 2022-10-08 14:10:40] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:36 [I 2022-10-08 14:10:40] ...trap/engines/engine.py.229: process: 0.76969 | load: 0.00302 [I 2022-10-08 14:10:40] ...trap/engines/engine.py.230: loss: 0.21874 [I 2022-10-08 14:10:40] ...optimizers/trijoint.py.151: total_norm: 2.061634 [I 2022-10-08 14:11:06] ...trap/engines/engine.py.224: train: epoch 7 | batch 1880/2304 [I 2022-10-08 14:11:06] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:15:54 [I 2022-10-08 14:11:06] ...trap/engines/engine.py.229: process: 0.78205 | load: 0.00305 [I 2022-10-08 14:11:06] ...trap/engines/engine.py.230: loss: 0.27521 [I 2022-10-08 14:11:06] ...optimizers/trijoint.py.151: total_norm: 3.257804 [I 2022-10-08 14:11:14] ...trap/engines/engine.py.224: train: epoch 7 | batch 1890/2304 [I 2022-10-08 14:11:14] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:06:25 [I 2022-10-08 14:11:14] ...trap/engines/engine.py.229: process: 0.77943 | load: 0.00304 [I 2022-10-08 14:11:14] ...trap/engines/engine.py.230: loss: 0.16488 [I 2022-10-08 14:11:14] ...optimizers/trijoint.py.151: total_norm: 2.337234 [I 2022-10-08 14:11:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 1900/2304 [I 2022-10-08 14:11:22] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:05:19 [I 2022-10-08 14:11:22] ...trap/engines/engine.py.229: process: 0.77024 | load: 0.00321 [I 2022-10-08 14:11:22] ...trap/engines/engine.py.230: loss: 0.21963 [I 2022-10-08 14:11:22] ...optimizers/trijoint.py.151: total_norm: 2.871084 [I 2022-10-08 14:11:30] ...trap/engines/engine.py.224: train: epoch 7 | batch 1910/2304 [I 2022-10-08 14:11:30] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:05:06 [I 2022-10-08 14:11:30] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00308 [I 2022-10-08 14:11:30] ...trap/engines/engine.py.230: loss: 0.20399 [I 2022-10-08 14:11:30] ...optimizers/trijoint.py.151: total_norm: 2.554130 [I 2022-10-08 14:11:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 1920/2304 [I 2022-10-08 14:11:37] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:05:00 [I 2022-10-08 14:11:37] ...trap/engines/engine.py.229: process: 0.77882 | load: 0.00303 [I 2022-10-08 14:11:37] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-08 14:11:38] ...optimizers/trijoint.py.151: total_norm: 1.790291 [I 2022-10-08 14:11:45] ...trap/engines/engine.py.224: train: epoch 7 | batch 1930/2304 [I 2022-10-08 14:11:45] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:04:51 [I 2022-10-08 14:11:45] ...trap/engines/engine.py.229: process: 0.77387 | load: 0.00299 [I 2022-10-08 14:11:45] ...trap/engines/engine.py.230: loss: 0.32805 [I 2022-10-08 14:11:45] ...optimizers/trijoint.py.151: total_norm: 3.060599 [I 2022-10-08 14:11:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1940/2304 [I 2022-10-08 14:11:53] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:04:43 [I 2022-10-08 14:11:53] ...trap/engines/engine.py.229: process: 0.78177 | load: 0.00307 [I 2022-10-08 14:11:53] ...trap/engines/engine.py.230: loss: 0.18325 [I 2022-10-08 14:11:53] ...optimizers/trijoint.py.151: total_norm: 2.933921 [I 2022-10-08 14:12:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 1950/2304 [I 2022-10-08 14:12:01] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:04:34 [I 2022-10-08 14:12:01] ...trap/engines/engine.py.229: process: 0.76255 | load: 0.00305 [I 2022-10-08 14:12:01] ...trap/engines/engine.py.230: loss: 0.19133 [I 2022-10-08 14:12:01] ...optimizers/trijoint.py.151: total_norm: 2.370973 [I 2022-10-08 14:12:09] ...trap/engines/engine.py.224: train: epoch 7 | batch 1960/2304 [I 2022-10-08 14:12:09] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:04:26 [I 2022-10-08 14:12:09] ...trap/engines/engine.py.229: process: 0.76644 | load: 0.00308 [I 2022-10-08 14:12:09] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-08 14:12:09] ...optimizers/trijoint.py.151: total_norm: 3.131952 [I 2022-10-08 14:12:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 1970/2304 [I 2022-10-08 14:12:17] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:04:19 [I 2022-10-08 14:12:17] ...trap/engines/engine.py.229: process: 0.78445 | load: 0.00299 [I 2022-10-08 14:12:17] ...trap/engines/engine.py.230: loss: 0.14297 [I 2022-10-08 14:12:17] ...optimizers/trijoint.py.151: total_norm: 1.900513 [I 2022-10-08 14:12:24] ...trap/engines/engine.py.224: train: epoch 7 | batch 1980/2304 [I 2022-10-08 14:12:24] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:04:11 [I 2022-10-08 14:12:24] ...trap/engines/engine.py.229: process: 0.77594 | load: 0.00301 [I 2022-10-08 14:12:24] ...trap/engines/engine.py.230: loss: 0.17373 [I 2022-10-08 14:12:24] ...optimizers/trijoint.py.151: total_norm: 2.472739 [I 2022-10-08 14:12:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 1990/2304 [I 2022-10-08 14:12:32] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:04:03 [I 2022-10-08 14:12:32] ...trap/engines/engine.py.229: process: 0.77086 | load: 0.00307 [I 2022-10-08 14:12:32] ...trap/engines/engine.py.230: loss: 0.30105 [I 2022-10-08 14:12:32] ...optimizers/trijoint.py.151: total_norm: 3.133674 [I 2022-10-08 14:12:40] ...trap/engines/engine.py.224: train: epoch 7 | batch 2000/2304 [I 2022-10-08 14:12:40] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:03:58 [I 2022-10-08 14:12:40] ...trap/engines/engine.py.229: process: 0.78864 | load: 0.00304 [I 2022-10-08 14:12:40] ...trap/engines/engine.py.230: loss: 0.19487 [I 2022-10-08 14:12:40] ...optimizers/trijoint.py.151: total_norm: 2.450974 [I 2022-10-08 14:12:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 2010/2304 [I 2022-10-08 14:12:48] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:03:48 [I 2022-10-08 14:12:48] ...trap/engines/engine.py.229: process: 0.76705 | load: 0.00303 [I 2022-10-08 14:12:48] ...trap/engines/engine.py.230: loss: 0.20985 [I 2022-10-08 14:12:48] ...optimizers/trijoint.py.151: total_norm: 2.667794 [I 2022-10-08 14:12:56] ...trap/engines/engine.py.224: train: epoch 7 | batch 2020/2304 [I 2022-10-08 14:12:56] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:03:41 [I 2022-10-08 14:12:56] ...trap/engines/engine.py.229: process: 0.78363 | load: 0.00302 [I 2022-10-08 14:12:56] ...trap/engines/engine.py.230: loss: 0.21004 [I 2022-10-08 14:12:56] ...optimizers/trijoint.py.151: total_norm: 2.616949 [I 2022-10-08 14:13:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 2030/2304 [I 2022-10-08 14:13:03] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:03:33 [I 2022-10-08 14:13:03] ...trap/engines/engine.py.229: process: 0.76417 | load: 0.00303 [I 2022-10-08 14:13:03] ...trap/engines/engine.py.230: loss: 0.21701 [I 2022-10-08 14:13:03] ...optimizers/trijoint.py.151: total_norm: 3.060106 [I 2022-10-08 14:13:11] ...trap/engines/engine.py.224: train: epoch 7 | batch 2040/2304 [I 2022-10-08 14:13:11] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:03:26 [I 2022-10-08 14:13:11] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00301 [I 2022-10-08 14:13:11] ...trap/engines/engine.py.230: loss: 0.18264 [I 2022-10-08 14:13:11] ...optimizers/trijoint.py.151: total_norm: 2.611063 [I 2022-10-08 14:13:19] ...trap/engines/engine.py.224: train: epoch 7 | batch 2050/2304 [I 2022-10-08 14:13:19] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:03:17 [I 2022-10-08 14:13:19] ...trap/engines/engine.py.229: process: 0.77458 | load: 0.00305 [I 2022-10-08 14:13:19] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-08 14:13:19] ...optimizers/trijoint.py.151: total_norm: 2.064910 [I 2022-10-08 14:13:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 2060/2304 [I 2022-10-08 14:13:27] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:03:10 [I 2022-10-08 14:13:27] ...trap/engines/engine.py.229: process: 0.77776 | load: 0.00312 [I 2022-10-08 14:13:27] ...trap/engines/engine.py.230: loss: 0.21578 [I 2022-10-08 14:13:27] ...optimizers/trijoint.py.151: total_norm: 2.514601 [I 2022-10-08 14:13:35] ...trap/engines/engine.py.224: train: epoch 7 | batch 2070/2304 [I 2022-10-08 14:13:35] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:03:02 [I 2022-10-08 14:13:35] ...trap/engines/engine.py.229: process: 0.78483 | load: 0.00304 [I 2022-10-08 14:13:35] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-08 14:13:35] ...optimizers/trijoint.py.151: total_norm: 2.397342 [I 2022-10-08 14:13:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 2080/2304 [I 2022-10-08 14:13:42] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:02:53 [I 2022-10-08 14:13:42] ...trap/engines/engine.py.229: process: 0.78296 | load: 0.00303 [I 2022-10-08 14:13:42] ...trap/engines/engine.py.230: loss: 0.25274 [I 2022-10-08 14:13:42] ...optimizers/trijoint.py.151: total_norm: 2.752363 [I 2022-10-08 14:13:50] ...trap/engines/engine.py.224: train: epoch 7 | batch 2090/2304 [I 2022-10-08 14:13:50] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:02:47 [I 2022-10-08 14:13:50] ...trap/engines/engine.py.229: process: 0.78913 | load: 0.00302 [I 2022-10-08 14:13:50] ...trap/engines/engine.py.230: loss: 0.22137 [I 2022-10-08 14:13:50] ...optimizers/trijoint.py.151: total_norm: 2.078686 [I 2022-10-08 14:13:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 2100/2304 [I 2022-10-08 14:13:58] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:02:38 [I 2022-10-08 14:13:58] ...trap/engines/engine.py.229: process: 0.78219 | load: 0.00309 [I 2022-10-08 14:13:58] ...trap/engines/engine.py.230: loss: 0.20113 [I 2022-10-08 14:13:58] ...optimizers/trijoint.py.151: total_norm: 2.516544 [I 2022-10-08 14:14:06] ...trap/engines/engine.py.224: train: epoch 7 | batch 2110/2304 [I 2022-10-08 14:14:06] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:02:30 [I 2022-10-08 14:14:06] ...trap/engines/engine.py.229: process: 0.76707 | load: 0.00305 [I 2022-10-08 14:14:06] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-08 14:14:06] ...optimizers/trijoint.py.151: total_norm: 2.447946 [I 2022-10-08 14:14:14] ...trap/engines/engine.py.224: train: epoch 7 | batch 2120/2304 [I 2022-10-08 14:14:14] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:02:23 [I 2022-10-08 14:14:14] ...trap/engines/engine.py.229: process: 0.77594 | load: 0.00301 [I 2022-10-08 14:14:14] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-08 14:14:14] ...optimizers/trijoint.py.151: total_norm: 2.537432 [I 2022-10-08 14:14:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 2130/2304 [I 2022-10-08 14:14:21] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:02:15 [I 2022-10-08 14:14:21] ...trap/engines/engine.py.229: process: 0.76864 | load: 0.00302 [I 2022-10-08 14:14:21] ...trap/engines/engine.py.230: loss: 0.17265 [I 2022-10-08 14:14:21] ...optimizers/trijoint.py.151: total_norm: 2.643016 [I 2022-10-08 14:14:29] ...trap/engines/engine.py.224: train: epoch 7 | batch 2140/2304 [I 2022-10-08 14:14:29] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:02:08 [I 2022-10-08 14:14:29] ...trap/engines/engine.py.229: process: 0.78543 | load: 0.00303 [I 2022-10-08 14:14:29] ...trap/engines/engine.py.230: loss: 0.23383 [I 2022-10-08 14:14:29] ...optimizers/trijoint.py.151: total_norm: 2.501336 [I 2022-10-08 14:14:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 2150/2304 [I 2022-10-08 14:14:37] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:01:59 [I 2022-10-08 14:14:37] ...trap/engines/engine.py.229: process: 0.78649 | load: 0.00303 [I 2022-10-08 14:14:37] ...trap/engines/engine.py.230: loss: 0.19395 [I 2022-10-08 14:14:37] ...optimizers/trijoint.py.151: total_norm: 2.399547 [I 2022-10-08 14:14:45] ...trap/engines/engine.py.224: train: epoch 7 | batch 2160/2304 [I 2022-10-08 14:14:45] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:01:51 [I 2022-10-08 14:14:45] ...trap/engines/engine.py.229: process: 0.76485 | load: 0.00305 [I 2022-10-08 14:14:45] ...trap/engines/engine.py.230: loss: 0.15266 [I 2022-10-08 14:14:45] ...optimizers/trijoint.py.151: total_norm: 2.904093 [I 2022-10-08 14:14:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 2170/2304 [I 2022-10-08 14:14:53] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:01:44 [I 2022-10-08 14:14:53] ...trap/engines/engine.py.229: process: 0.77186 | load: 0.00306 [I 2022-10-08 14:14:53] ...trap/engines/engine.py.230: loss: 0.28312 [I 2022-10-08 14:14:53] ...optimizers/trijoint.py.151: total_norm: 2.879066 [I 2022-10-08 14:15:00] ...trap/engines/engine.py.224: train: epoch 7 | batch 2180/2304 [I 2022-10-08 14:15:00] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:01:36 [I 2022-10-08 14:15:00] ...trap/engines/engine.py.229: process: 0.77314 | load: 0.00303 [I 2022-10-08 14:15:00] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-08 14:15:00] ...optimizers/trijoint.py.151: total_norm: 2.262044 [I 2022-10-08 14:15:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 2190/2304 [I 2022-10-08 14:15:08] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:01:28 [I 2022-10-08 14:15:08] ...trap/engines/engine.py.229: process: 0.77804 | load: 0.00303 [I 2022-10-08 14:15:08] ...trap/engines/engine.py.230: loss: 0.28983 [I 2022-10-08 14:15:08] ...optimizers/trijoint.py.151: total_norm: 3.082329 [I 2022-10-08 14:15:16] ...trap/engines/engine.py.224: train: epoch 7 | batch 2200/2304 [I 2022-10-08 14:15:16] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:01:21 [I 2022-10-08 14:15:16] ...trap/engines/engine.py.229: process: 0.79795 | load: 0.00309 [I 2022-10-08 14:15:16] ...trap/engines/engine.py.230: loss: 0.28709 [I 2022-10-08 14:15:16] ...optimizers/trijoint.py.151: total_norm: 3.500480 [I 2022-10-08 14:15:24] ...trap/engines/engine.py.224: train: epoch 7 | batch 2210/2304 [I 2022-10-08 14:15:24] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:01:13 [I 2022-10-08 14:15:24] ...trap/engines/engine.py.229: process: 0.77395 | load: 0.00307 [I 2022-10-08 14:15:24] ...trap/engines/engine.py.230: loss: 0.19585 [I 2022-10-08 14:15:24] ...optimizers/trijoint.py.151: total_norm: 2.355180 [I 2022-10-08 14:15:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 2220/2304 [I 2022-10-08 14:15:32] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:01:05 [I 2022-10-08 14:15:32] ...trap/engines/engine.py.229: process: 0.77396 | load: 0.00302 [I 2022-10-08 14:15:32] ...trap/engines/engine.py.230: loss: 0.23756 [I 2022-10-08 14:15:32] ...optimizers/trijoint.py.151: total_norm: 2.674636 [I 2022-10-08 14:15:39] ...trap/engines/engine.py.224: train: epoch 7 | batch 2230/2304 [I 2022-10-08 14:15:39] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:00:57 [I 2022-10-08 14:15:39] ...trap/engines/engine.py.229: process: 0.77916 | load: 0.00300 [I 2022-10-08 14:15:39] ...trap/engines/engine.py.230: loss: 0.25113 [I 2022-10-08 14:15:39] ...optimizers/trijoint.py.151: total_norm: 3.025458 [I 2022-10-08 14:15:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 2240/2304 [I 2022-10-08 14:15:47] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:00:50 [I 2022-10-08 14:15:47] ...trap/engines/engine.py.229: process: 0.79511 | load: 0.00305 [I 2022-10-08 14:15:47] ...trap/engines/engine.py.230: loss: 0.23022 [I 2022-10-08 14:15:47] ...optimizers/trijoint.py.151: total_norm: 2.709918 [I 2022-10-08 14:15:55] ...trap/engines/engine.py.224: train: epoch 7 | batch 2250/2304 [I 2022-10-08 14:15:55] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:00:42 [I 2022-10-08 14:15:55] ...trap/engines/engine.py.229: process: 0.78441 | load: 0.00304 [I 2022-10-08 14:15:55] ...trap/engines/engine.py.230: loss: 0.20393 [I 2022-10-08 14:15:55] ...optimizers/trijoint.py.151: total_norm: 2.552069 [I 2022-10-08 14:16:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 2260/2304 [I 2022-10-08 14:16:03] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:00:34 [I 2022-10-08 14:16:03] ...trap/engines/engine.py.229: process: 0.79147 | load: 0.00314 [I 2022-10-08 14:16:03] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-08 14:16:03] ...optimizers/trijoint.py.151: total_norm: 2.159436 [I 2022-10-08 14:16:11] ...trap/engines/engine.py.224: train: epoch 7 | batch 2270/2304 [I 2022-10-08 14:16:11] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:00:26 [I 2022-10-08 14:16:11] ...trap/engines/engine.py.229: process: 0.77819 | load: 0.00303 [I 2022-10-08 14:16:11] ...trap/engines/engine.py.230: loss: 0.15699 [I 2022-10-08 14:16:11] ...optimizers/trijoint.py.151: total_norm: 2.404267 [I 2022-10-08 14:16:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 2280/2304 [I 2022-10-08 14:16:18] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:00:18 [I 2022-10-08 14:16:18] ...trap/engines/engine.py.229: process: 0.76709 | load: 0.00300 [I 2022-10-08 14:16:18] ...trap/engines/engine.py.230: loss: 0.24658 [I 2022-10-08 14:16:18] ...optimizers/trijoint.py.151: total_norm: 2.595935 [I 2022-10-08 14:16:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 2290/2304 [I 2022-10-08 14:16:26] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:00:10 [I 2022-10-08 14:16:26] ...trap/engines/engine.py.229: process: 0.76632 | load: 0.00300 [I 2022-10-08 14:16:26] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-08 14:16:26] ...optimizers/trijoint.py.151: total_norm: 2.710820 [I 2022-10-08 14:16:34] ...trap/engines/engine.py.224: train: epoch 7 | batch 2300/2304 [I 2022-10-08 14:16:34] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:00:03 [I 2022-10-08 14:16:34] ...trap/engines/engine.py.229: process: 0.77040 | load: 0.00016 [I 2022-10-08 14:16:34] ...trap/engines/engine.py.230: loss: 0.40167 [I 2022-10-08 14:16:34] ...optimizers/trijoint.py.151: total_norm: 3.636851 [I 2022-10-08 14:16:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 2304/2304 [I 2022-10-08 14:16:37] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:00:00 [I 2022-10-08 14:16:37] ...trap/engines/engine.py.229: process: 0.76705 | load: 0.00016 [I 2022-10-08 14:16:37] ...trap/engines/engine.py.230: loss: 0.19332 [I 2022-10-08 14:16:37] ...optimizers/trijoint.py.151: total_norm: 2.530031 [S 2022-10-08 14:16:38] ...trap/engines/engine.py.129: train_epoch.epoch: 7 [S 2022-10-08 14:16:38] ...trap/engines/engine.py.129: train_epoch.loss: 0.21145922090704683 [I 2022-10-08 14:16:38] ...s/criterions/triplet.py.84: margin = 0.09000000000000002 [I 2022-10-08 14:16:39] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 7 [I 2022-10-08 14:16:39] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 14:16:39] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 14:16:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 14:16:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 14:16:41] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 14:16:41] ...trap/engines/engine.py.321: eval: epoch 7 | batch 0/510 [I 2022-10-08 14:16:41] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:05 [I 2022-10-08 14:16:41] ...trap/engines/engine.py.326: process: 2.59885 | load: 2.06004 [I 2022-10-08 14:16:47] ...trap/engines/engine.py.321: eval: epoch 7 | batch 10/510 [I 2022-10-08 14:16:47] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:27 [I 2022-10-08 14:16:47] ...trap/engines/engine.py.326: process: 0.53762 | load: 0.00018 [I 2022-10-08 14:16:53] ...trap/engines/engine.py.321: eval: epoch 7 | batch 20/510 [I 2022-10-08 14:16:53] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:37 [I 2022-10-08 14:16:53] ...trap/engines/engine.py.326: process: 0.75624 | load: 0.00020 [I 2022-10-08 14:16:59] ...trap/engines/engine.py.321: eval: epoch 7 | batch 30/510 [I 2022-10-08 14:16:59] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:05 [I 2022-10-08 14:16:59] ...trap/engines/engine.py.326: process: 0.43527 | load: 0.00018 [I 2022-10-08 14:17:03] ...trap/engines/engine.py.321: eval: epoch 7 | batch 40/510 [I 2022-10-08 14:17:03] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:25 [I 2022-10-08 14:17:03] ...trap/engines/engine.py.326: process: 0.43208 | load: 0.00019 [I 2022-10-08 14:17:07] ...trap/engines/engine.py.321: eval: epoch 7 | batch 50/510 [I 2022-10-08 14:17:07] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:29 [I 2022-10-08 14:17:07] ...trap/engines/engine.py.326: process: 0.48697 | load: 0.00019 [I 2022-10-08 14:17:13] ...trap/engines/engine.py.321: eval: epoch 7 | batch 60/510 [I 2022-10-08 14:17:13] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:47 [I 2022-10-08 14:17:13] ...trap/engines/engine.py.326: process: 0.43027 | load: 0.00018 [I 2022-10-08 14:17:17] ...trap/engines/engine.py.321: eval: epoch 7 | batch 70/510 [I 2022-10-08 14:17:17] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:22 [I 2022-10-08 14:17:17] ...trap/engines/engine.py.326: process: 0.42509 | load: 0.00018 [I 2022-10-08 14:17:22] ...trap/engines/engine.py.321: eval: epoch 7 | batch 80/510 [I 2022-10-08 14:17:22] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:23 [I 2022-10-08 14:17:22] ...trap/engines/engine.py.326: process: 0.47363 | load: 0.00019 [I 2022-10-08 14:17:26] ...trap/engines/engine.py.321: eval: epoch 7 | batch 90/510 [I 2022-10-08 14:17:26] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:02:57 [I 2022-10-08 14:17:26] ...trap/engines/engine.py.326: process: 0.40177 | load: 0.00019 [I 2022-10-08 14:17:30] ...trap/engines/engine.py.321: eval: epoch 7 | batch 100/510 [I 2022-10-08 14:17:30] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:02:47 [I 2022-10-08 14:17:30] ...trap/engines/engine.py.326: process: 0.30300 | load: 0.00018 [I 2022-10-08 14:17:34] ...trap/engines/engine.py.321: eval: epoch 7 | batch 110/510 [I 2022-10-08 14:17:34] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-08 14:17:34] ...trap/engines/engine.py.326: process: 0.31498 | load: 0.00020 [I 2022-10-08 14:17:37] ...trap/engines/engine.py.321: eval: epoch 7 | batch 120/510 [I 2022-10-08 14:17:37] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:03 [I 2022-10-08 14:17:37] ...trap/engines/engine.py.326: process: 0.31942 | load: 0.00020 [I 2022-10-08 14:17:40] ...trap/engines/engine.py.321: eval: epoch 7 | batch 130/510 [I 2022-10-08 14:17:40] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:00 [I 2022-10-08 14:17:40] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00019 [I 2022-10-08 14:17:43] ...trap/engines/engine.py.321: eval: epoch 7 | batch 140/510 [I 2022-10-08 14:17:43] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:56 [I 2022-10-08 14:17:43] ...trap/engines/engine.py.326: process: 0.31633 | load: 0.00019 [I 2022-10-08 14:17:46] ...trap/engines/engine.py.321: eval: epoch 7 | batch 150/510 [I 2022-10-08 14:17:46] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:53 [I 2022-10-08 14:17:46] ...trap/engines/engine.py.326: process: 0.31262 | load: 0.00020 [I 2022-10-08 14:17:49] ...trap/engines/engine.py.321: eval: epoch 7 | batch 160/510 [I 2022-10-08 14:17:49] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:50 [I 2022-10-08 14:17:49] ...trap/engines/engine.py.326: process: 0.30809 | load: 0.00020 [I 2022-10-08 14:17:53] ...trap/engines/engine.py.321: eval: epoch 7 | batch 170/510 [I 2022-10-08 14:17:53] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:47 [I 2022-10-08 14:17:53] ...trap/engines/engine.py.326: process: 0.31627 | load: 0.00019 [I 2022-10-08 14:17:56] ...trap/engines/engine.py.321: eval: epoch 7 | batch 180/510 [I 2022-10-08 14:17:56] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:44 [I 2022-10-08 14:17:56] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00019 [I 2022-10-08 14:17:59] ...trap/engines/engine.py.321: eval: epoch 7 | batch 190/510 [I 2022-10-08 14:17:59] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:42 [I 2022-10-08 14:17:59] ...trap/engines/engine.py.326: process: 0.32003 | load: 0.00018 [I 2022-10-08 14:18:02] ...trap/engines/engine.py.321: eval: epoch 7 | batch 200/510 [I 2022-10-08 14:18:02] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:38 [I 2022-10-08 14:18:02] ...trap/engines/engine.py.326: process: 0.31090 | load: 0.00019 [I 2022-10-08 14:18:05] ...trap/engines/engine.py.321: eval: epoch 7 | batch 210/510 [I 2022-10-08 14:18:05] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:34 [I 2022-10-08 14:18:05] ...trap/engines/engine.py.326: process: 0.30856 | load: 0.00019 [I 2022-10-08 14:18:08] ...trap/engines/engine.py.321: eval: epoch 7 | batch 220/510 [I 2022-10-08 14:18:08] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:32 [I 2022-10-08 14:18:08] ...trap/engines/engine.py.326: process: 0.31868 | load: 0.00019 [I 2022-10-08 14:18:12] ...trap/engines/engine.py.321: eval: epoch 7 | batch 230/510 [I 2022-10-08 14:18:12] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-08 14:18:12] ...trap/engines/engine.py.326: process: 0.31603 | load: 0.00019 [I 2022-10-08 14:18:15] ...trap/engines/engine.py.321: eval: epoch 7 | batch 240/510 [I 2022-10-08 14:18:15] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:25 [I 2022-10-08 14:18:15] ...trap/engines/engine.py.326: process: 0.31942 | load: 0.00020 [I 2022-10-08 14:18:18] ...trap/engines/engine.py.321: eval: epoch 7 | batch 250/510 [I 2022-10-08 14:18:18] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-08 14:18:18] ...trap/engines/engine.py.326: process: 0.31230 | load: 0.00020 [I 2022-10-08 14:18:21] ...trap/engines/engine.py.321: eval: epoch 7 | batch 260/510 [I 2022-10-08 14:18:21] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-08 14:18:21] ...trap/engines/engine.py.326: process: 0.31810 | load: 0.00020 [I 2022-10-08 14:18:24] ...trap/engines/engine.py.321: eval: epoch 7 | batch 270/510 [I 2022-10-08 14:18:24] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:15 [I 2022-10-08 14:18:24] ...trap/engines/engine.py.326: process: 0.31616 | load: 0.00019 [I 2022-10-08 14:18:27] ...trap/engines/engine.py.321: eval: epoch 7 | batch 280/510 [I 2022-10-08 14:18:27] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:13 [I 2022-10-08 14:18:27] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00020 [I 2022-10-08 14:18:31] ...trap/engines/engine.py.321: eval: epoch 7 | batch 290/510 [I 2022-10-08 14:18:31] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:10 [I 2022-10-08 14:18:31] ...trap/engines/engine.py.326: process: 0.32103 | load: 0.00020 [I 2022-10-08 14:18:34] ...trap/engines/engine.py.321: eval: epoch 7 | batch 300/510 [I 2022-10-08 14:18:34] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-08 14:18:34] ...trap/engines/engine.py.326: process: 0.31579 | load: 0.00020 [I 2022-10-08 14:18:37] ...trap/engines/engine.py.321: eval: epoch 7 | batch 310/510 [I 2022-10-08 14:18:37] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-08 14:18:37] ...trap/engines/engine.py.326: process: 0.31827 | load: 0.00019 [I 2022-10-08 14:18:40] ...trap/engines/engine.py.321: eval: epoch 7 | batch 320/510 [I 2022-10-08 14:18:40] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-08 14:18:40] ...trap/engines/engine.py.326: process: 0.32601 | load: 0.00020 [I 2022-10-08 14:18:43] ...trap/engines/engine.py.321: eval: epoch 7 | batch 330/510 [I 2022-10-08 14:18:43] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:57 [I 2022-10-08 14:18:43] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00021 [I 2022-10-08 14:18:47] ...trap/engines/engine.py.321: eval: epoch 7 | batch 340/510 [I 2022-10-08 14:18:47] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-08 14:18:47] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00019 [I 2022-10-08 14:18:50] ...trap/engines/engine.py.321: eval: epoch 7 | batch 350/510 [I 2022-10-08 14:18:50] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-08 14:18:50] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-08 14:18:53] ...trap/engines/engine.py.321: eval: epoch 7 | batch 360/510 [I 2022-10-08 14:18:53] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-08 14:18:53] ...trap/engines/engine.py.326: process: 0.31722 | load: 0.00019 [I 2022-10-08 14:18:56] ...trap/engines/engine.py.321: eval: epoch 7 | batch 370/510 [I 2022-10-08 14:18:56] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:44 [I 2022-10-08 14:18:56] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00020 [I 2022-10-08 14:18:59] ...trap/engines/engine.py.321: eval: epoch 7 | batch 380/510 [I 2022-10-08 14:18:59] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:41 [I 2022-10-08 14:18:59] ...trap/engines/engine.py.326: process: 0.31837 | load: 0.00020 [I 2022-10-08 14:19:02] ...trap/engines/engine.py.321: eval: epoch 7 | batch 390/510 [I 2022-10-08 14:19:02] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:37 [I 2022-10-08 14:19:02] ...trap/engines/engine.py.326: process: 0.31031 | load: 0.00019 [I 2022-10-08 14:19:06] ...trap/engines/engine.py.321: eval: epoch 7 | batch 400/510 [I 2022-10-08 14:19:06] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-08 14:19:06] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00019 [I 2022-10-08 14:19:09] ...trap/engines/engine.py.321: eval: epoch 7 | batch 410/510 [I 2022-10-08 14:19:09] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-08 14:19:09] ...trap/engines/engine.py.326: process: 0.32147 | load: 0.00021 [I 2022-10-08 14:19:12] ...trap/engines/engine.py.321: eval: epoch 7 | batch 420/510 [I 2022-10-08 14:19:12] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-08 14:19:12] ...trap/engines/engine.py.326: process: 0.31093 | load: 0.00020 [I 2022-10-08 14:19:15] ...trap/engines/engine.py.321: eval: epoch 7 | batch 430/510 [I 2022-10-08 14:19:15] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:25 [I 2022-10-08 14:19:15] ...trap/engines/engine.py.326: process: 0.31763 | load: 0.00020 [I 2022-10-08 14:19:18] ...trap/engines/engine.py.321: eval: epoch 7 | batch 440/510 [I 2022-10-08 14:19:18] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:22 [I 2022-10-08 14:19:18] ...trap/engines/engine.py.326: process: 0.32068 | load: 0.00020 [I 2022-10-08 14:19:21] ...trap/engines/engine.py.321: eval: epoch 7 | batch 450/510 [I 2022-10-08 14:19:21] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:19 [I 2022-10-08 14:19:21] ...trap/engines/engine.py.326: process: 0.32086 | load: 0.00020 [I 2022-10-08 14:19:25] ...trap/engines/engine.py.321: eval: epoch 7 | batch 460/510 [I 2022-10-08 14:19:25] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-08 14:19:25] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00019 [I 2022-10-08 14:19:28] ...trap/engines/engine.py.321: eval: epoch 7 | batch 470/510 [I 2022-10-08 14:19:28] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-08 14:19:28] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00021 [I 2022-10-08 14:19:31] ...trap/engines/engine.py.321: eval: epoch 7 | batch 480/510 [I 2022-10-08 14:19:31] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-08 14:19:31] ...trap/engines/engine.py.326: process: 0.31649 | load: 0.00019 [I 2022-10-08 14:19:34] ...trap/engines/engine.py.321: eval: epoch 7 | batch 490/510 [I 2022-10-08 14:19:34] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:06 [I 2022-10-08 14:19:34] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00019 [I 2022-10-08 14:19:37] ...trap/engines/engine.py.321: eval: epoch 7 | batch 500/510 [I 2022-10-08 14:19:37] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:03 [I 2022-10-08 14:19:37] ...trap/engines/engine.py.326: process: 0.31518 | load: 0.00020 [I 2022-10-08 14:19:41] ...trap/engines/engine.py.321: eval: epoch 7 | batch 510/510 [I 2022-10-08 14:19:41] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-08 14:19:41] ...trap/engines/engine.py.326: process: 0.30441 | load: 0.00015 [S 2022-10-08 14:19:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 7 [I 2022-10-08 14:19:41] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.571600 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.838500 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.900400 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.580300 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.843700 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.900700 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010855 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009233 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007130 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009980 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006649 [S 2022-10-08 14:20:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007497 [I 2022-10-08 14:20:13] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 14:20:13] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 14:20:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 14:20:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 14:20:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 14:20:13] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 14:20:16] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 14:20:20] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 14:20:20] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 14:20:20] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 14:20:23] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 14:20:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 14:20:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 8 [I 2022-10-08 14:20:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 14:20:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 0/2304 [I 2022-10-08 14:20:31] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:52:21 [I 2022-10-08 14:20:31] ...trap/engines/engine.py.229: process: 2.92598 | load: 2.11048 [I 2022-10-08 14:20:31] ...trap/engines/engine.py.230: loss: 0.23275 [I 2022-10-08 14:20:31] ...optimizers/trijoint.py.151: total_norm: 2.440114 [I 2022-10-08 14:20:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 10/2304 [I 2022-10-08 14:20:38] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:46 [I 2022-10-08 14:20:38] ...trap/engines/engine.py.229: process: 0.79529 | load: 0.00502 [I 2022-10-08 14:20:38] ...trap/engines/engine.py.230: loss: 0.19119 [I 2022-10-08 14:20:38] ...optimizers/trijoint.py.151: total_norm: 1.722595 [I 2022-10-08 14:20:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 20/2304 [I 2022-10-08 14:20:46] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:50 [I 2022-10-08 14:20:46] ...trap/engines/engine.py.229: process: 0.78755 | load: 0.00490 [I 2022-10-08 14:20:46] ...trap/engines/engine.py.230: loss: 0.17291 [I 2022-10-08 14:20:46] ...optimizers/trijoint.py.151: total_norm: 2.648018 [I 2022-10-08 14:20:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 30/2304 [I 2022-10-08 14:20:54] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:47 [I 2022-10-08 14:20:54] ...trap/engines/engine.py.229: process: 0.76300 | load: 0.00440 [I 2022-10-08 14:20:54] ...trap/engines/engine.py.230: loss: 0.26429 [I 2022-10-08 14:20:54] ...optimizers/trijoint.py.151: total_norm: 2.705521 [I 2022-10-08 14:21:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 40/2304 [I 2022-10-08 14:21:02] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:29 [I 2022-10-08 14:21:02] ...trap/engines/engine.py.229: process: 0.78518 | load: 0.00417 [I 2022-10-08 14:21:02] ...trap/engines/engine.py.230: loss: 0.21342 [I 2022-10-08 14:21:02] ...optimizers/trijoint.py.151: total_norm: 2.844443 [I 2022-10-08 14:21:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 50/2304 [I 2022-10-08 14:21:10] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:15 [I 2022-10-08 14:21:10] ...trap/engines/engine.py.229: process: 0.78190 | load: 0.00443 [I 2022-10-08 14:21:10] ...trap/engines/engine.py.230: loss: 0.16809 [I 2022-10-08 14:21:10] ...optimizers/trijoint.py.151: total_norm: 2.434226 [I 2022-10-08 14:21:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 60/2304 [I 2022-10-08 14:21:18] ...trap/engines/engine.py.228: elapsed: 0:00:50 | left: 0:29:11 [I 2022-10-08 14:21:18] ...trap/engines/engine.py.229: process: 0.77852 | load: 0.00380 [I 2022-10-08 14:21:18] ...trap/engines/engine.py.230: loss: 0.18607 [I 2022-10-08 14:21:18] ...optimizers/trijoint.py.151: total_norm: 2.326319 [I 2022-10-08 14:21:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 70/2304 [I 2022-10-08 14:21:26] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:03 [I 2022-10-08 14:21:26] ...trap/engines/engine.py.229: process: 0.77826 | load: 0.00398 [I 2022-10-08 14:21:26] ...trap/engines/engine.py.230: loss: 0.14717 [I 2022-10-08 14:21:26] ...optimizers/trijoint.py.151: total_norm: 2.980472 [I 2022-10-08 14:21:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 80/2304 [I 2022-10-08 14:21:33] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:55 [I 2022-10-08 14:21:33] ...trap/engines/engine.py.229: process: 0.78636 | load: 0.00353 [I 2022-10-08 14:21:33] ...trap/engines/engine.py.230: loss: 0.25993 [I 2022-10-08 14:21:33] ...optimizers/trijoint.py.151: total_norm: 3.218209 [I 2022-10-08 14:21:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 90/2304 [I 2022-10-08 14:21:41] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:47 [I 2022-10-08 14:21:41] ...trap/engines/engine.py.229: process: 0.77755 | load: 0.00351 [I 2022-10-08 14:21:41] ...trap/engines/engine.py.230: loss: 0.26018 [I 2022-10-08 14:21:41] ...optimizers/trijoint.py.151: total_norm: 2.861862 [I 2022-10-08 14:21:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 100/2304 [I 2022-10-08 14:21:49] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:42 [I 2022-10-08 14:21:49] ...trap/engines/engine.py.229: process: 0.78548 | load: 0.00370 [I 2022-10-08 14:21:49] ...trap/engines/engine.py.230: loss: 0.21857 [I 2022-10-08 14:21:49] ...optimizers/trijoint.py.151: total_norm: 3.385837 [I 2022-10-08 14:21:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 110/2304 [I 2022-10-08 14:21:57] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:29 [I 2022-10-08 14:21:57] ...trap/engines/engine.py.229: process: 0.77675 | load: 0.00343 [I 2022-10-08 14:21:57] ...trap/engines/engine.py.230: loss: 0.15856 [I 2022-10-08 14:21:57] ...optimizers/trijoint.py.151: total_norm: 2.312053 [I 2022-10-08 14:22:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 120/2304 [I 2022-10-08 14:22:05] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:12 [I 2022-10-08 14:22:05] ...trap/engines/engine.py.229: process: 0.77352 | load: 0.00324 [I 2022-10-08 14:22:05] ...trap/engines/engine.py.230: loss: 0.27939 [I 2022-10-08 14:22:05] ...optimizers/trijoint.py.151: total_norm: 3.486183 [I 2022-10-08 14:22:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 130/2304 [I 2022-10-08 14:22:12] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:10 [I 2022-10-08 14:22:12] ...trap/engines/engine.py.229: process: 0.78196 | load: 0.00328 [I 2022-10-08 14:22:12] ...trap/engines/engine.py.230: loss: 0.16958 [I 2022-10-08 14:22:12] ...optimizers/trijoint.py.151: total_norm: 2.289516 [I 2022-10-08 14:22:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 140/2304 [I 2022-10-08 14:22:20] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:03 [I 2022-10-08 14:22:20] ...trap/engines/engine.py.229: process: 0.77152 | load: 0.00311 [I 2022-10-08 14:22:20] ...trap/engines/engine.py.230: loss: 0.24150 [I 2022-10-08 14:22:20] ...optimizers/trijoint.py.151: total_norm: 2.838312 [I 2022-10-08 14:22:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 150/2304 [I 2022-10-08 14:22:28] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:58 [I 2022-10-08 14:22:28] ...trap/engines/engine.py.229: process: 0.77429 | load: 0.00312 [I 2022-10-08 14:22:28] ...trap/engines/engine.py.230: loss: 0.17005 [I 2022-10-08 14:22:28] ...optimizers/trijoint.py.151: total_norm: 3.209099 [I 2022-10-08 14:22:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 160/2304 [I 2022-10-08 14:22:36] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:28:03 [I 2022-10-08 14:22:36] ...trap/engines/engine.py.229: process: 0.79517 | load: 0.00320 [I 2022-10-08 14:22:36] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-08 14:22:36] ...optimizers/trijoint.py.151: total_norm: 2.329823 [I 2022-10-08 14:22:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 170/2304 [I 2022-10-08 14:22:44] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:27:44 [I 2022-10-08 14:22:44] ...trap/engines/engine.py.229: process: 0.78493 | load: 0.00333 [I 2022-10-08 14:22:44] ...trap/engines/engine.py.230: loss: 0.17136 [I 2022-10-08 14:22:44] ...optimizers/trijoint.py.151: total_norm: 2.574111 [I 2022-10-08 14:22:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 180/2304 [I 2022-10-08 14:22:52] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:34 [I 2022-10-08 14:22:52] ...trap/engines/engine.py.229: process: 0.77007 | load: 0.00301 [I 2022-10-08 14:22:52] ...trap/engines/engine.py.230: loss: 0.28383 [I 2022-10-08 14:22:52] ...optimizers/trijoint.py.151: total_norm: 4.037970 [I 2022-10-08 14:22:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 190/2304 [I 2022-10-08 14:22:59] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:13 [I 2022-10-08 14:22:59] ...trap/engines/engine.py.229: process: 0.77740 | load: 0.00313 [I 2022-10-08 14:22:59] ...trap/engines/engine.py.230: loss: 0.25711 [I 2022-10-08 14:22:59] ...optimizers/trijoint.py.151: total_norm: 3.363419 [I 2022-10-08 14:23:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 200/2304 [I 2022-10-08 14:23:07] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:05 [I 2022-10-08 14:23:07] ...trap/engines/engine.py.229: process: 0.77689 | load: 0.00313 [I 2022-10-08 14:23:07] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-08 14:23:07] ...optimizers/trijoint.py.151: total_norm: 2.405394 [I 2022-10-08 14:23:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 210/2304 [I 2022-10-08 14:23:15] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:04 [I 2022-10-08 14:23:15] ...trap/engines/engine.py.229: process: 0.77506 | load: 0.00300 [I 2022-10-08 14:23:15] ...trap/engines/engine.py.230: loss: 0.20817 [I 2022-10-08 14:23:15] ...optimizers/trijoint.py.151: total_norm: 2.803164 [I 2022-10-08 14:23:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 220/2304 [I 2022-10-08 14:23:23] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:26:56 [I 2022-10-08 14:23:23] ...trap/engines/engine.py.229: process: 0.77492 | load: 0.00302 [I 2022-10-08 14:23:23] ...trap/engines/engine.py.230: loss: 0.22017 [I 2022-10-08 14:23:23] ...optimizers/trijoint.py.151: total_norm: 2.657420 [I 2022-10-08 14:23:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 230/2304 [I 2022-10-08 14:23:30] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:50 [I 2022-10-08 14:23:30] ...trap/engines/engine.py.229: process: 0.76795 | load: 0.00303 [I 2022-10-08 14:23:30] ...trap/engines/engine.py.230: loss: 0.21544 [I 2022-10-08 14:23:30] ...optimizers/trijoint.py.151: total_norm: 2.680705 [I 2022-10-08 14:23:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 240/2304 [I 2022-10-08 14:23:38] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:37 [I 2022-10-08 14:23:38] ...trap/engines/engine.py.229: process: 0.75624 | load: 0.00304 [I 2022-10-08 14:23:38] ...trap/engines/engine.py.230: loss: 0.24606 [I 2022-10-08 14:23:38] ...optimizers/trijoint.py.151: total_norm: 3.086406 [I 2022-10-08 14:23:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 250/2304 [I 2022-10-08 14:23:46] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:29 [I 2022-10-08 14:23:46] ...trap/engines/engine.py.229: process: 0.76938 | load: 0.00309 [I 2022-10-08 14:23:46] ...trap/engines/engine.py.230: loss: 0.15981 [I 2022-10-08 14:23:46] ...optimizers/trijoint.py.151: total_norm: 2.842036 [I 2022-10-08 14:23:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 260/2304 [I 2022-10-08 14:23:54] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:19 [I 2022-10-08 14:23:54] ...trap/engines/engine.py.229: process: 0.76943 | load: 0.00307 [I 2022-10-08 14:23:54] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-08 14:23:54] ...optimizers/trijoint.py.151: total_norm: 2.490014 [I 2022-10-08 14:24:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 270/2304 [I 2022-10-08 14:24:01] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:17 [I 2022-10-08 14:24:01] ...trap/engines/engine.py.229: process: 0.76936 | load: 0.00321 [I 2022-10-08 14:24:01] ...trap/engines/engine.py.230: loss: 0.24195 [I 2022-10-08 14:24:01] ...optimizers/trijoint.py.151: total_norm: 3.274877 [I 2022-10-08 14:24:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 280/2304 [I 2022-10-08 14:24:09] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:07 [I 2022-10-08 14:24:09] ...trap/engines/engine.py.229: process: 0.77124 | load: 0.00318 [I 2022-10-08 14:24:09] ...trap/engines/engine.py.230: loss: 0.25632 [I 2022-10-08 14:24:09] ...optimizers/trijoint.py.151: total_norm: 2.830750 [I 2022-10-08 14:24:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 290/2304 [I 2022-10-08 14:24:17] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:08 [I 2022-10-08 14:24:17] ...trap/engines/engine.py.229: process: 0.77270 | load: 0.00305 [I 2022-10-08 14:24:17] ...trap/engines/engine.py.230: loss: 0.21207 [I 2022-10-08 14:24:17] ...optimizers/trijoint.py.151: total_norm: 2.689353 [I 2022-10-08 14:24:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 300/2304 [I 2022-10-08 14:24:25] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:56 [I 2022-10-08 14:24:25] ...trap/engines/engine.py.229: process: 0.77963 | load: 0.00303 [I 2022-10-08 14:24:25] ...trap/engines/engine.py.230: loss: 0.13791 [I 2022-10-08 14:24:25] ...optimizers/trijoint.py.151: total_norm: 2.232979 [I 2022-10-08 14:24:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 310/2304 [I 2022-10-08 14:24:33] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:25:47 [I 2022-10-08 14:24:33] ...trap/engines/engine.py.229: process: 0.77304 | load: 0.00304 [I 2022-10-08 14:24:33] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-08 14:24:33] ...optimizers/trijoint.py.151: total_norm: 2.584503 [I 2022-10-08 14:24:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 320/2304 [I 2022-10-08 14:24:40] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:41 [I 2022-10-08 14:24:40] ...trap/engines/engine.py.229: process: 0.77918 | load: 0.00303 [I 2022-10-08 14:24:40] ...trap/engines/engine.py.230: loss: 0.19832 [I 2022-10-08 14:24:40] ...optimizers/trijoint.py.151: total_norm: 2.617698 [I 2022-10-08 14:24:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 330/2304 [I 2022-10-08 14:24:48] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:36 [I 2022-10-08 14:24:48] ...trap/engines/engine.py.229: process: 0.75801 | load: 0.00307 [I 2022-10-08 14:24:48] ...trap/engines/engine.py.230: loss: 0.29584 [I 2022-10-08 14:24:48] ...optimizers/trijoint.py.151: total_norm: 3.379341 [I 2022-10-08 14:24:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 340/2304 [I 2022-10-08 14:24:56] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:14 [I 2022-10-08 14:24:56] ...trap/engines/engine.py.229: process: 0.77998 | load: 0.00301 [I 2022-10-08 14:24:56] ...trap/engines/engine.py.230: loss: 0.23430 [I 2022-10-08 14:24:56] ...optimizers/trijoint.py.151: total_norm: 2.491254 [I 2022-10-08 14:25:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 350/2304 [I 2022-10-08 14:25:04] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:13 [I 2022-10-08 14:25:04] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00301 [I 2022-10-08 14:25:04] ...trap/engines/engine.py.230: loss: 0.16079 [I 2022-10-08 14:25:04] ...optimizers/trijoint.py.151: total_norm: 3.096283 [I 2022-10-08 14:25:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 360/2304 [I 2022-10-08 14:25:11] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:05 [I 2022-10-08 14:25:11] ...trap/engines/engine.py.229: process: 0.77633 | load: 0.00304 [I 2022-10-08 14:25:11] ...trap/engines/engine.py.230: loss: 0.14235 [I 2022-10-08 14:25:11] ...optimizers/trijoint.py.151: total_norm: 2.107589 [I 2022-10-08 14:25:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 370/2304 [I 2022-10-08 14:25:19] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:03 [I 2022-10-08 14:25:19] ...trap/engines/engine.py.229: process: 0.78210 | load: 0.00307 [I 2022-10-08 14:25:19] ...trap/engines/engine.py.230: loss: 0.17471 [I 2022-10-08 14:25:19] ...optimizers/trijoint.py.151: total_norm: 2.053169 [I 2022-10-08 14:25:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 380/2304 [I 2022-10-08 14:25:27] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:00 [I 2022-10-08 14:25:27] ...trap/engines/engine.py.229: process: 0.79037 | load: 0.00305 [I 2022-10-08 14:25:27] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-08 14:25:27] ...optimizers/trijoint.py.151: total_norm: 2.390130 [I 2022-10-08 14:25:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 390/2304 [I 2022-10-08 14:25:35] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:53 [I 2022-10-08 14:25:35] ...trap/engines/engine.py.229: process: 0.78871 | load: 0.00313 [I 2022-10-08 14:25:35] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-08 14:25:35] ...optimizers/trijoint.py.151: total_norm: 2.309669 [I 2022-10-08 14:25:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 400/2304 [I 2022-10-08 14:25:43] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:49 [I 2022-10-08 14:25:43] ...trap/engines/engine.py.229: process: 0.79656 | load: 0.00305 [I 2022-10-08 14:25:43] ...trap/engines/engine.py.230: loss: 0.35029 [I 2022-10-08 14:25:43] ...optimizers/trijoint.py.151: total_norm: 3.379961 [I 2022-10-08 14:25:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 410/2304 [I 2022-10-08 14:25:51] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:24:37 [I 2022-10-08 14:25:51] ...trap/engines/engine.py.229: process: 0.76951 | load: 0.00304 [I 2022-10-08 14:25:51] ...trap/engines/engine.py.230: loss: 0.23586 [I 2022-10-08 14:25:51] ...optimizers/trijoint.py.151: total_norm: 2.794405 [I 2022-10-08 14:25:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 420/2304 [I 2022-10-08 14:25:58] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:25 [I 2022-10-08 14:25:58] ...trap/engines/engine.py.229: process: 0.76754 | load: 0.00300 [I 2022-10-08 14:25:58] ...trap/engines/engine.py.230: loss: 0.13421 [I 2022-10-08 14:25:58] ...optimizers/trijoint.py.151: total_norm: 2.916462 [I 2022-10-08 14:26:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 430/2304 [I 2022-10-08 14:26:06] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:33 [I 2022-10-08 14:26:06] ...trap/engines/engine.py.229: process: 0.79105 | load: 0.00302 [I 2022-10-08 14:26:06] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-08 14:26:06] ...optimizers/trijoint.py.151: total_norm: 2.684176 [I 2022-10-08 14:26:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 440/2304 [I 2022-10-08 14:26:14] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:19 [I 2022-10-08 14:26:14] ...trap/engines/engine.py.229: process: 0.77275 | load: 0.00301 [I 2022-10-08 14:26:14] ...trap/engines/engine.py.230: loss: 0.27543 [I 2022-10-08 14:26:14] ...optimizers/trijoint.py.151: total_norm: 3.073049 [I 2022-10-08 14:26:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 450/2304 [I 2022-10-08 14:26:22] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:15 [I 2022-10-08 14:26:22] ...trap/engines/engine.py.229: process: 0.79693 | load: 0.00300 [I 2022-10-08 14:26:22] ...trap/engines/engine.py.230: loss: 0.24170 [I 2022-10-08 14:26:22] ...optimizers/trijoint.py.151: total_norm: 3.088374 [I 2022-10-08 14:26:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 460/2304 [I 2022-10-08 14:26:30] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:59 [I 2022-10-08 14:26:30] ...trap/engines/engine.py.229: process: 0.77382 | load: 0.00306 [I 2022-10-08 14:26:30] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-08 14:26:30] ...optimizers/trijoint.py.151: total_norm: 2.619842 [I 2022-10-08 14:26:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 470/2304 [I 2022-10-08 14:26:38] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:52 [I 2022-10-08 14:26:38] ...trap/engines/engine.py.229: process: 0.78426 | load: 0.00304 [I 2022-10-08 14:26:38] ...trap/engines/engine.py.230: loss: 0.17429 [I 2022-10-08 14:26:38] ...optimizers/trijoint.py.151: total_norm: 2.186766 [I 2022-10-08 14:26:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 480/2304 [I 2022-10-08 14:26:45] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:50 [I 2022-10-08 14:26:45] ...trap/engines/engine.py.229: process: 0.78464 | load: 0.00300 [I 2022-10-08 14:26:45] ...trap/engines/engine.py.230: loss: 0.26753 [I 2022-10-08 14:26:45] ...optimizers/trijoint.py.151: total_norm: 3.501570 [I 2022-10-08 14:26:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 490/2304 [I 2022-10-08 14:26:53] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:52 [I 2022-10-08 14:26:53] ...trap/engines/engine.py.229: process: 0.79726 | load: 0.00313 [I 2022-10-08 14:26:53] ...trap/engines/engine.py.230: loss: 0.20424 [I 2022-10-08 14:26:53] ...optimizers/trijoint.py.151: total_norm: 2.800051 [I 2022-10-08 14:27:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 500/2304 [I 2022-10-08 14:27:01] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:24 [I 2022-10-08 14:27:01] ...trap/engines/engine.py.229: process: 0.77394 | load: 0.00305 [I 2022-10-08 14:27:01] ...trap/engines/engine.py.230: loss: 0.25729 [I 2022-10-08 14:27:01] ...optimizers/trijoint.py.151: total_norm: 2.649962 [I 2022-10-08 14:27:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 510/2304 [I 2022-10-08 14:27:09] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:17 [I 2022-10-08 14:27:09] ...trap/engines/engine.py.229: process: 0.77841 | load: 0.00304 [I 2022-10-08 14:27:09] ...trap/engines/engine.py.230: loss: 0.27179 [I 2022-10-08 14:27:09] ...optimizers/trijoint.py.151: total_norm: 3.082806 [I 2022-10-08 14:27:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 520/2304 [I 2022-10-08 14:27:17] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:06 [I 2022-10-08 14:27:17] ...trap/engines/engine.py.229: process: 0.77631 | load: 0.00303 [I 2022-10-08 14:27:17] ...trap/engines/engine.py.230: loss: 0.13965 [I 2022-10-08 14:27:17] ...optimizers/trijoint.py.151: total_norm: 1.682190 [I 2022-10-08 14:27:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 530/2304 [I 2022-10-08 14:27:25] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:10 [I 2022-10-08 14:27:25] ...trap/engines/engine.py.229: process: 0.79880 | load: 0.00301 [I 2022-10-08 14:27:25] ...trap/engines/engine.py.230: loss: 0.16477 [I 2022-10-08 14:27:25] ...optimizers/trijoint.py.151: total_norm: 3.197905 [I 2022-10-08 14:27:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 540/2304 [I 2022-10-08 14:27:32] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:55 [I 2022-10-08 14:27:32] ...trap/engines/engine.py.229: process: 0.77963 | load: 0.00306 [I 2022-10-08 14:27:32] ...trap/engines/engine.py.230: loss: 0.17534 [I 2022-10-08 14:27:32] ...optimizers/trijoint.py.151: total_norm: 2.339017 [I 2022-10-08 14:27:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 550/2304 [I 2022-10-08 14:27:40] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:44 [I 2022-10-08 14:27:40] ...trap/engines/engine.py.229: process: 0.77184 | load: 0.00304 [I 2022-10-08 14:27:40] ...trap/engines/engine.py.230: loss: 0.21959 [I 2022-10-08 14:27:40] ...optimizers/trijoint.py.151: total_norm: 2.839322 [I 2022-10-08 14:27:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 560/2304 [I 2022-10-08 14:27:48] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:46 [I 2022-10-08 14:27:48] ...trap/engines/engine.py.229: process: 0.78276 | load: 0.00303 [I 2022-10-08 14:27:48] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-08 14:27:48] ...optimizers/trijoint.py.151: total_norm: 1.898690 [I 2022-10-08 14:27:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 570/2304 [I 2022-10-08 14:27:56] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:33 [I 2022-10-08 14:27:56] ...trap/engines/engine.py.229: process: 0.77708 | load: 0.00302 [I 2022-10-08 14:27:56] ...trap/engines/engine.py.230: loss: 0.18677 [I 2022-10-08 14:27:56] ...optimizers/trijoint.py.151: total_norm: 2.599265 [I 2022-10-08 14:28:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 580/2304 [I 2022-10-08 14:28:04] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:21 [I 2022-10-08 14:28:04] ...trap/engines/engine.py.229: process: 0.76917 | load: 0.00301 [I 2022-10-08 14:28:04] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-08 14:28:04] ...optimizers/trijoint.py.151: total_norm: 2.242328 [I 2022-10-08 14:28:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 590/2304 [I 2022-10-08 14:28:11] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:17 [I 2022-10-08 14:28:11] ...trap/engines/engine.py.229: process: 0.78131 | load: 0.00302 [I 2022-10-08 14:28:11] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-08 14:28:11] ...optimizers/trijoint.py.151: total_norm: 2.135616 [I 2022-10-08 14:28:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 600/2304 [I 2022-10-08 14:28:19] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:06 [I 2022-10-08 14:28:19] ...trap/engines/engine.py.229: process: 0.78033 | load: 0.00306 [I 2022-10-08 14:28:19] ...trap/engines/engine.py.230: loss: 0.25868 [I 2022-10-08 14:28:19] ...optimizers/trijoint.py.151: total_norm: 3.044630 [I 2022-10-08 14:28:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 610/2304 [I 2022-10-08 14:28:27] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:05 [I 2022-10-08 14:28:27] ...trap/engines/engine.py.229: process: 0.78995 | load: 0.00307 [I 2022-10-08 14:28:27] ...trap/engines/engine.py.230: loss: 0.21837 [I 2022-10-08 14:28:27] ...optimizers/trijoint.py.151: total_norm: 3.428811 [I 2022-10-08 14:28:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 620/2304 [I 2022-10-08 14:28:35] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:53 [I 2022-10-08 14:28:35] ...trap/engines/engine.py.229: process: 0.77874 | load: 0.00304 [I 2022-10-08 14:28:35] ...trap/engines/engine.py.230: loss: 0.17837 [I 2022-10-08 14:28:35] ...optimizers/trijoint.py.151: total_norm: 2.445230 [I 2022-10-08 14:28:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 630/2304 [I 2022-10-08 14:28:43] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:44 [I 2022-10-08 14:28:43] ...trap/engines/engine.py.229: process: 0.79062 | load: 0.00300 [I 2022-10-08 14:28:43] ...trap/engines/engine.py.230: loss: 0.23676 [I 2022-10-08 14:28:43] ...optimizers/trijoint.py.151: total_norm: 2.863003 [I 2022-10-08 14:28:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 640/2304 [I 2022-10-08 14:28:51] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:38 [I 2022-10-08 14:28:51] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00301 [I 2022-10-08 14:28:51] ...trap/engines/engine.py.230: loss: 0.18273 [I 2022-10-08 14:28:51] ...optimizers/trijoint.py.151: total_norm: 2.399578 [I 2022-10-08 14:28:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 650/2304 [I 2022-10-08 14:28:58] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:34 [I 2022-10-08 14:28:58] ...trap/engines/engine.py.229: process: 0.79319 | load: 0.00305 [I 2022-10-08 14:28:58] ...trap/engines/engine.py.230: loss: 0.30402 [I 2022-10-08 14:28:58] ...optimizers/trijoint.py.151: total_norm: 3.150563 [I 2022-10-08 14:29:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 660/2304 [I 2022-10-08 14:29:06] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:17 [I 2022-10-08 14:29:06] ...trap/engines/engine.py.229: process: 0.76730 | load: 0.00310 [I 2022-10-08 14:29:06] ...trap/engines/engine.py.230: loss: 0.21494 [I 2022-10-08 14:29:06] ...optimizers/trijoint.py.151: total_norm: 3.228848 [I 2022-10-08 14:29:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 670/2304 [I 2022-10-08 14:29:14] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:07 [I 2022-10-08 14:29:14] ...trap/engines/engine.py.229: process: 0.77454 | load: 0.00306 [I 2022-10-08 14:29:14] ...trap/engines/engine.py.230: loss: 0.24701 [I 2022-10-08 14:29:14] ...optimizers/trijoint.py.151: total_norm: 2.622295 [I 2022-10-08 14:29:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 680/2304 [I 2022-10-08 14:29:22] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:06 [I 2022-10-08 14:29:22] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00302 [I 2022-10-08 14:29:22] ...trap/engines/engine.py.230: loss: 0.14845 [I 2022-10-08 14:29:22] ...optimizers/trijoint.py.151: total_norm: 1.895550 [I 2022-10-08 14:29:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 690/2304 [I 2022-10-08 14:29:30] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:21:01 [I 2022-10-08 14:29:30] ...trap/engines/engine.py.229: process: 0.79424 | load: 0.00302 [I 2022-10-08 14:29:30] ...trap/engines/engine.py.230: loss: 0.19952 [I 2022-10-08 14:29:30] ...optimizers/trijoint.py.151: total_norm: 2.669342 [I 2022-10-08 14:29:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 700/2304 [I 2022-10-08 14:29:37] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:42 [I 2022-10-08 14:29:37] ...trap/engines/engine.py.229: process: 0.78224 | load: 0.00303 [I 2022-10-08 14:29:37] ...trap/engines/engine.py.230: loss: 0.26482 [I 2022-10-08 14:29:37] ...optimizers/trijoint.py.151: total_norm: 2.826750 [I 2022-10-08 14:29:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 710/2304 [I 2022-10-08 14:29:45] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:30 [I 2022-10-08 14:29:45] ...trap/engines/engine.py.229: process: 0.76613 | load: 0.00308 [I 2022-10-08 14:29:45] ...trap/engines/engine.py.230: loss: 0.23552 [I 2022-10-08 14:29:45] ...optimizers/trijoint.py.151: total_norm: 3.339442 [I 2022-10-08 14:29:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 720/2304 [I 2022-10-08 14:29:53] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:36 [I 2022-10-08 14:29:53] ...trap/engines/engine.py.229: process: 0.77033 | load: 0.00304 [I 2022-10-08 14:29:53] ...trap/engines/engine.py.230: loss: 0.17770 [I 2022-10-08 14:29:53] ...optimizers/trijoint.py.151: total_norm: 2.533736 [I 2022-10-08 14:30:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 730/2304 [I 2022-10-08 14:30:01] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:25 [I 2022-10-08 14:30:01] ...trap/engines/engine.py.229: process: 0.78616 | load: 0.00302 [I 2022-10-08 14:30:01] ...trap/engines/engine.py.230: loss: 0.19490 [I 2022-10-08 14:30:01] ...optimizers/trijoint.py.151: total_norm: 2.288853 [I 2022-10-08 14:30:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 740/2304 [I 2022-10-08 14:30:09] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:23 [I 2022-10-08 14:30:09] ...trap/engines/engine.py.229: process: 0.78433 | load: 0.00300 [I 2022-10-08 14:30:09] ...trap/engines/engine.py.230: loss: 0.19820 [I 2022-10-08 14:30:09] ...optimizers/trijoint.py.151: total_norm: 2.332860 [I 2022-10-08 14:30:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 750/2304 [I 2022-10-08 14:30:16] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:05 [I 2022-10-08 14:30:16] ...trap/engines/engine.py.229: process: 0.78183 | load: 0.00307 [I 2022-10-08 14:30:16] ...trap/engines/engine.py.230: loss: 0.22590 [I 2022-10-08 14:30:16] ...optimizers/trijoint.py.151: total_norm: 2.571893 [I 2022-10-08 14:30:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 760/2304 [I 2022-10-08 14:30:24] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:19:59 [I 2022-10-08 14:30:24] ...trap/engines/engine.py.229: process: 0.77349 | load: 0.00305 [I 2022-10-08 14:30:24] ...trap/engines/engine.py.230: loss: 0.17176 [I 2022-10-08 14:30:24] ...optimizers/trijoint.py.151: total_norm: 2.410856 [I 2022-10-08 14:30:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 770/2304 [I 2022-10-08 14:30:32] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:49 [I 2022-10-08 14:30:32] ...trap/engines/engine.py.229: process: 0.77705 | load: 0.00299 [I 2022-10-08 14:30:32] ...trap/engines/engine.py.230: loss: 0.16478 [I 2022-10-08 14:30:32] ...optimizers/trijoint.py.151: total_norm: 2.193517 [I 2022-10-08 14:30:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 780/2304 [I 2022-10-08 14:30:40] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:47 [I 2022-10-08 14:30:40] ...trap/engines/engine.py.229: process: 0.78095 | load: 0.00306 [I 2022-10-08 14:30:40] ...trap/engines/engine.py.230: loss: 0.19017 [I 2022-10-08 14:30:40] ...optimizers/trijoint.py.151: total_norm: 2.050055 [I 2022-10-08 14:30:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 790/2304 [I 2022-10-08 14:30:48] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:51 [I 2022-10-08 14:30:48] ...trap/engines/engine.py.229: process: 0.80201 | load: 0.00309 [I 2022-10-08 14:30:48] ...trap/engines/engine.py.230: loss: 0.18723 [I 2022-10-08 14:30:48] ...optimizers/trijoint.py.151: total_norm: 1.895728 [I 2022-10-08 14:30:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 800/2304 [I 2022-10-08 14:30:55] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:29 [I 2022-10-08 14:30:55] ...trap/engines/engine.py.229: process: 0.76809 | load: 0.00303 [I 2022-10-08 14:30:55] ...trap/engines/engine.py.230: loss: 0.23277 [I 2022-10-08 14:30:55] ...optimizers/trijoint.py.151: total_norm: 2.045009 [I 2022-10-08 14:31:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 810/2304 [I 2022-10-08 14:31:03] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:18 [I 2022-10-08 14:31:03] ...trap/engines/engine.py.229: process: 0.78094 | load: 0.00307 [I 2022-10-08 14:31:03] ...trap/engines/engine.py.230: loss: 0.16661 [I 2022-10-08 14:31:03] ...optimizers/trijoint.py.151: total_norm: 1.838968 [I 2022-10-08 14:31:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 820/2304 [I 2022-10-08 14:31:11] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:11 [I 2022-10-08 14:31:11] ...trap/engines/engine.py.229: process: 0.77183 | load: 0.00302 [I 2022-10-08 14:31:11] ...trap/engines/engine.py.230: loss: 0.17661 [I 2022-10-08 14:31:11] ...optimizers/trijoint.py.151: total_norm: 2.762169 [I 2022-10-08 14:31:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 830/2304 [I 2022-10-08 14:31:19] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:03 [I 2022-10-08 14:31:19] ...trap/engines/engine.py.229: process: 0.75642 | load: 0.00302 [I 2022-10-08 14:31:19] ...trap/engines/engine.py.230: loss: 0.24707 [I 2022-10-08 14:31:19] ...optimizers/trijoint.py.151: total_norm: 3.260179 [I 2022-10-08 14:31:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 840/2304 [I 2022-10-08 14:31:26] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:18:50 [I 2022-10-08 14:31:26] ...trap/engines/engine.py.229: process: 0.77897 | load: 0.00305 [I 2022-10-08 14:31:26] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-08 14:31:26] ...optimizers/trijoint.py.151: total_norm: 2.243845 [I 2022-10-08 14:31:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 850/2304 [I 2022-10-08 14:31:34] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:46 [I 2022-10-08 14:31:34] ...trap/engines/engine.py.229: process: 0.77319 | load: 0.00303 [I 2022-10-08 14:31:34] ...trap/engines/engine.py.230: loss: 0.22488 [I 2022-10-08 14:31:34] ...optimizers/trijoint.py.151: total_norm: 2.292767 [I 2022-10-08 14:31:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 860/2304 [I 2022-10-08 14:31:42] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:47 [I 2022-10-08 14:31:42] ...trap/engines/engine.py.229: process: 0.77621 | load: 0.00302 [I 2022-10-08 14:31:42] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-08 14:31:42] ...optimizers/trijoint.py.151: total_norm: 2.557627 [I 2022-10-08 14:31:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 870/2304 [I 2022-10-08 14:31:50] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:30 [I 2022-10-08 14:31:50] ...trap/engines/engine.py.229: process: 0.76473 | load: 0.00308 [I 2022-10-08 14:31:50] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-08 14:31:50] ...optimizers/trijoint.py.151: total_norm: 2.582817 [I 2022-10-08 14:31:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 880/2304 [I 2022-10-08 14:31:58] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:18:31 [I 2022-10-08 14:31:58] ...trap/engines/engine.py.229: process: 0.78048 | load: 0.00302 [I 2022-10-08 14:31:58] ...trap/engines/engine.py.230: loss: 0.20212 [I 2022-10-08 14:31:58] ...optimizers/trijoint.py.151: total_norm: 3.259938 [I 2022-10-08 14:32:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 890/2304 [I 2022-10-08 14:32:05] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:18 [I 2022-10-08 14:32:05] ...trap/engines/engine.py.229: process: 0.78028 | load: 0.00301 [I 2022-10-08 14:32:05] ...trap/engines/engine.py.230: loss: 0.21762 [I 2022-10-08 14:32:05] ...optimizers/trijoint.py.151: total_norm: 2.546842 [I 2022-10-08 14:32:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 900/2304 [I 2022-10-08 14:32:13] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:12 [I 2022-10-08 14:32:13] ...trap/engines/engine.py.229: process: 0.78617 | load: 0.00308 [I 2022-10-08 14:32:13] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-08 14:32:13] ...optimizers/trijoint.py.151: total_norm: 3.137032 [I 2022-10-08 14:32:21] ...trap/engines/engine.py.224: train: epoch 8 | batch 910/2304 [I 2022-10-08 14:32:21] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:04 [I 2022-10-08 14:32:21] ...trap/engines/engine.py.229: process: 0.78408 | load: 0.00308 [I 2022-10-08 14:32:21] ...trap/engines/engine.py.230: loss: 0.21475 [I 2022-10-08 14:32:21] ...optimizers/trijoint.py.151: total_norm: 2.949848 [I 2022-10-08 14:32:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 920/2304 [I 2022-10-08 14:32:29] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:17:58 [I 2022-10-08 14:32:29] ...trap/engines/engine.py.229: process: 0.78011 | load: 0.00301 [I 2022-10-08 14:32:29] ...trap/engines/engine.py.230: loss: 0.31978 [I 2022-10-08 14:32:29] ...optimizers/trijoint.py.151: total_norm: 3.364277 [I 2022-10-08 14:32:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 930/2304 [I 2022-10-08 14:32:37] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:43 [I 2022-10-08 14:32:37] ...trap/engines/engine.py.229: process: 0.77898 | load: 0.00305 [I 2022-10-08 14:32:37] ...trap/engines/engine.py.230: loss: 0.26578 [I 2022-10-08 14:32:37] ...optimizers/trijoint.py.151: total_norm: 2.887804 [I 2022-10-08 14:32:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 940/2304 [I 2022-10-08 14:32:44] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:43 [I 2022-10-08 14:32:44] ...trap/engines/engine.py.229: process: 0.78905 | load: 0.00303 [I 2022-10-08 14:32:44] ...trap/engines/engine.py.230: loss: 0.22252 [I 2022-10-08 14:32:44] ...optimizers/trijoint.py.151: total_norm: 2.868617 [I 2022-10-08 14:32:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 950/2304 [I 2022-10-08 14:32:52] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:30 [I 2022-10-08 14:32:52] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00305 [I 2022-10-08 14:32:52] ...trap/engines/engine.py.230: loss: 0.21969 [I 2022-10-08 14:32:52] ...optimizers/trijoint.py.151: total_norm: 3.264946 [I 2022-10-08 14:33:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 960/2304 [I 2022-10-08 14:33:00] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:17:26 [I 2022-10-08 14:33:00] ...trap/engines/engine.py.229: process: 0.78610 | load: 0.00305 [I 2022-10-08 14:33:00] ...trap/engines/engine.py.230: loss: 0.17796 [I 2022-10-08 14:33:00] ...optimizers/trijoint.py.151: total_norm: 2.491172 [I 2022-10-08 14:33:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 970/2304 [I 2022-10-08 14:33:08] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:16 [I 2022-10-08 14:33:08] ...trap/engines/engine.py.229: process: 0.78296 | load: 0.00301 [I 2022-10-08 14:33:08] ...trap/engines/engine.py.230: loss: 0.17461 [I 2022-10-08 14:33:08] ...optimizers/trijoint.py.151: total_norm: 2.125684 [I 2022-10-08 14:33:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 980/2304 [I 2022-10-08 14:33:15] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:17:10 [I 2022-10-08 14:33:15] ...trap/engines/engine.py.229: process: 0.77597 | load: 0.00304 [I 2022-10-08 14:33:15] ...trap/engines/engine.py.230: loss: 0.21145 [I 2022-10-08 14:33:15] ...optimizers/trijoint.py.151: total_norm: 2.868737 [I 2022-10-08 14:33:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 990/2304 [I 2022-10-08 14:33:23] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:16:58 [I 2022-10-08 14:33:23] ...trap/engines/engine.py.229: process: 0.76950 | load: 0.00305 [I 2022-10-08 14:33:23] ...trap/engines/engine.py.230: loss: 0.20392 [I 2022-10-08 14:33:23] ...optimizers/trijoint.py.151: total_norm: 2.453555 [I 2022-10-08 14:33:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 1000/2304 [I 2022-10-08 14:33:31] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:16:50 [I 2022-10-08 14:33:31] ...trap/engines/engine.py.229: process: 0.77107 | load: 0.00302 [I 2022-10-08 14:33:31] ...trap/engines/engine.py.230: loss: 0.23550 [I 2022-10-08 14:33:31] ...optimizers/trijoint.py.151: total_norm: 3.060326 [I 2022-10-08 14:33:39] ...trap/engines/engine.py.224: train: epoch 8 | batch 1010/2304 [I 2022-10-08 14:33:39] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:16:52 [I 2022-10-08 14:33:39] ...trap/engines/engine.py.229: process: 0.78565 | load: 0.00304 [I 2022-10-08 14:33:39] ...trap/engines/engine.py.230: loss: 0.24066 [I 2022-10-08 14:33:39] ...optimizers/trijoint.py.151: total_norm: 2.727957 [I 2022-10-08 14:33:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 1020/2304 [I 2022-10-08 14:33:47] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:16:38 [I 2022-10-08 14:33:47] ...trap/engines/engine.py.229: process: 0.77563 | load: 0.00303 [I 2022-10-08 14:33:47] ...trap/engines/engine.py.230: loss: 0.15931 [I 2022-10-08 14:33:47] ...optimizers/trijoint.py.151: total_norm: 2.611740 [I 2022-10-08 14:33:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 1030/2304 [I 2022-10-08 14:33:54] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:16:35 [I 2022-10-08 14:33:54] ...trap/engines/engine.py.229: process: 0.77813 | load: 0.00307 [I 2022-10-08 14:33:54] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-08 14:33:54] ...optimizers/trijoint.py.151: total_norm: 2.721812 [I 2022-10-08 14:34:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1040/2304 [I 2022-10-08 14:34:02] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:16:28 [I 2022-10-08 14:34:02] ...trap/engines/engine.py.229: process: 0.76819 | load: 0.00303 [I 2022-10-08 14:34:02] ...trap/engines/engine.py.230: loss: 0.22335 [I 2022-10-08 14:34:02] ...optimizers/trijoint.py.151: total_norm: 2.306880 [I 2022-10-08 14:34:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 1050/2304 [I 2022-10-08 14:34:10] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:16:19 [I 2022-10-08 14:34:10] ...trap/engines/engine.py.229: process: 0.79043 | load: 0.00307 [I 2022-10-08 14:34:10] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-08 14:34:10] ...optimizers/trijoint.py.151: total_norm: 2.314058 [I 2022-10-08 14:34:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1060/2304 [I 2022-10-08 14:34:18] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:16:05 [I 2022-10-08 14:34:18] ...trap/engines/engine.py.229: process: 0.76926 | load: 0.00300 [I 2022-10-08 14:34:18] ...trap/engines/engine.py.230: loss: 0.13643 [I 2022-10-08 14:34:18] ...optimizers/trijoint.py.151: total_norm: 2.728189 [I 2022-10-08 14:34:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 1070/2304 [I 2022-10-08 14:34:26] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:15:59 [I 2022-10-08 14:34:26] ...trap/engines/engine.py.229: process: 0.77415 | load: 0.00306 [I 2022-10-08 14:34:26] ...trap/engines/engine.py.230: loss: 0.17181 [I 2022-10-08 14:34:26] ...optimizers/trijoint.py.151: total_norm: 2.255573 [I 2022-10-08 14:34:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 1080/2304 [I 2022-10-08 14:34:34] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:15:53 [I 2022-10-08 14:34:34] ...trap/engines/engine.py.229: process: 0.78014 | load: 0.00317 [I 2022-10-08 14:34:34] ...trap/engines/engine.py.230: loss: 0.18154 [I 2022-10-08 14:34:34] ...optimizers/trijoint.py.151: total_norm: 2.338601 [I 2022-10-08 14:34:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 1090/2304 [I 2022-10-08 14:34:41] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:15:44 [I 2022-10-08 14:34:41] ...trap/engines/engine.py.229: process: 0.77658 | load: 0.00305 [I 2022-10-08 14:34:41] ...trap/engines/engine.py.230: loss: 0.19008 [I 2022-10-08 14:34:41] ...optimizers/trijoint.py.151: total_norm: 3.387741 [I 2022-10-08 14:34:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 1100/2304 [I 2022-10-08 14:34:49] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:15:35 [I 2022-10-08 14:34:49] ...trap/engines/engine.py.229: process: 0.77672 | load: 0.00302 [I 2022-10-08 14:34:49] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-08 14:34:49] ...optimizers/trijoint.py.151: total_norm: 1.682435 [I 2022-10-08 14:34:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 1110/2304 [I 2022-10-08 14:34:57] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:15:28 [I 2022-10-08 14:34:57] ...trap/engines/engine.py.229: process: 0.78125 | load: 0.00302 [I 2022-10-08 14:34:57] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-08 14:34:57] ...optimizers/trijoint.py.151: total_norm: 2.590094 [I 2022-10-08 14:35:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 1120/2304 [I 2022-10-08 14:35:05] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:15:15 [I 2022-10-08 14:35:05] ...trap/engines/engine.py.229: process: 0.77284 | load: 0.00305 [I 2022-10-08 14:35:05] ...trap/engines/engine.py.230: loss: 0.30718 [I 2022-10-08 14:35:05] ...optimizers/trijoint.py.151: total_norm: 3.221202 [I 2022-10-08 14:35:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 1130/2304 [I 2022-10-08 14:35:12] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:15:12 [I 2022-10-08 14:35:12] ...trap/engines/engine.py.229: process: 0.76727 | load: 0.00302 [I 2022-10-08 14:35:12] ...trap/engines/engine.py.230: loss: 0.28406 [I 2022-10-08 14:35:12] ...optimizers/trijoint.py.151: total_norm: 2.818771 [I 2022-10-08 14:35:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 1140/2304 [I 2022-10-08 14:35:20] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:15:11 [I 2022-10-08 14:35:20] ...trap/engines/engine.py.229: process: 0.78525 | load: 0.00307 [I 2022-10-08 14:35:20] ...trap/engines/engine.py.230: loss: 0.18576 [I 2022-10-08 14:35:20] ...optimizers/trijoint.py.151: total_norm: 2.782012 [I 2022-10-08 14:35:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1150/2304 [I 2022-10-08 14:35:28] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:14:54 [I 2022-10-08 14:35:28] ...trap/engines/engine.py.229: process: 0.76521 | load: 0.00304 [I 2022-10-08 14:35:28] ...trap/engines/engine.py.230: loss: 0.18030 [I 2022-10-08 14:35:28] ...optimizers/trijoint.py.151: total_norm: 2.847685 [I 2022-10-08 14:35:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 1160/2304 [I 2022-10-08 14:35:36] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:14:51 [I 2022-10-08 14:35:36] ...trap/engines/engine.py.229: process: 0.78672 | load: 0.00306 [I 2022-10-08 14:35:36] ...trap/engines/engine.py.230: loss: 0.28633 [I 2022-10-08 14:35:36] ...optimizers/trijoint.py.151: total_norm: 3.412751 [I 2022-10-08 14:35:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 1170/2304 [I 2022-10-08 14:35:44] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:14:39 [I 2022-10-08 14:35:44] ...trap/engines/engine.py.229: process: 0.77993 | load: 0.00305 [I 2022-10-08 14:35:44] ...trap/engines/engine.py.230: loss: 0.26583 [I 2022-10-08 14:35:44] ...optimizers/trijoint.py.151: total_norm: 3.165447 [I 2022-10-08 14:35:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 1180/2304 [I 2022-10-08 14:35:51] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:14:35 [I 2022-10-08 14:35:51] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00310 [I 2022-10-08 14:35:51] ...trap/engines/engine.py.230: loss: 0.17173 [I 2022-10-08 14:35:51] ...optimizers/trijoint.py.151: total_norm: 2.714462 [I 2022-10-08 14:35:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 1190/2304 [I 2022-10-08 14:35:59] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:14:25 [I 2022-10-08 14:35:59] ...trap/engines/engine.py.229: process: 0.76737 | load: 0.00307 [I 2022-10-08 14:35:59] ...trap/engines/engine.py.230: loss: 0.20770 [I 2022-10-08 14:35:59] ...optimizers/trijoint.py.151: total_norm: 2.445548 [I 2022-10-08 14:36:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 1200/2304 [I 2022-10-08 14:36:07] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:14:20 [I 2022-10-08 14:36:07] ...trap/engines/engine.py.229: process: 0.77042 | load: 0.00304 [I 2022-10-08 14:36:07] ...trap/engines/engine.py.230: loss: 0.18960 [I 2022-10-08 14:36:07] ...optimizers/trijoint.py.151: total_norm: 2.977259 [I 2022-10-08 14:36:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 1210/2304 [I 2022-10-08 14:36:15] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:14:16 [I 2022-10-08 14:36:15] ...trap/engines/engine.py.229: process: 0.77841 | load: 0.00302 [I 2022-10-08 14:36:15] ...trap/engines/engine.py.230: loss: 0.18912 [I 2022-10-08 14:36:15] ...optimizers/trijoint.py.151: total_norm: 2.139418 [I 2022-10-08 14:36:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1220/2304 [I 2022-10-08 14:36:23] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:14:06 [I 2022-10-08 14:36:23] ...trap/engines/engine.py.229: process: 0.77897 | load: 0.00300 [I 2022-10-08 14:36:23] ...trap/engines/engine.py.230: loss: 0.16439 [I 2022-10-08 14:36:23] ...optimizers/trijoint.py.151: total_norm: 1.864872 [I 2022-10-08 14:36:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 1230/2304 [I 2022-10-08 14:36:30] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:13:51 [I 2022-10-08 14:36:30] ...trap/engines/engine.py.229: process: 0.76854 | load: 0.00301 [I 2022-10-08 14:36:30] ...trap/engines/engine.py.230: loss: 0.22107 [I 2022-10-08 14:36:30] ...optimizers/trijoint.py.151: total_norm: 3.483123 [I 2022-10-08 14:36:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1240/2304 [I 2022-10-08 14:36:38] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:13:52 [I 2022-10-08 14:36:38] ...trap/engines/engine.py.229: process: 0.78774 | load: 0.00301 [I 2022-10-08 14:36:38] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-08 14:36:38] ...optimizers/trijoint.py.151: total_norm: 3.135715 [I 2022-10-08 14:36:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 1250/2304 [I 2022-10-08 14:36:46] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:13:37 [I 2022-10-08 14:36:46] ...trap/engines/engine.py.229: process: 0.77764 | load: 0.00299 [I 2022-10-08 14:36:46] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-08 14:36:46] ...optimizers/trijoint.py.151: total_norm: 2.363624 [I 2022-10-08 14:36:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 1260/2304 [I 2022-10-08 14:36:54] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:13:26 [I 2022-10-08 14:36:54] ...trap/engines/engine.py.229: process: 0.76940 | load: 0.00304 [I 2022-10-08 14:36:54] ...trap/engines/engine.py.230: loss: 0.17084 [I 2022-10-08 14:36:54] ...optimizers/trijoint.py.151: total_norm: 2.086939 [I 2022-10-08 14:37:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1270/2304 [I 2022-10-08 14:37:02] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:13:22 [I 2022-10-08 14:37:02] ...trap/engines/engine.py.229: process: 0.77965 | load: 0.00306 [I 2022-10-08 14:37:02] ...trap/engines/engine.py.230: loss: 0.20903 [I 2022-10-08 14:37:02] ...optimizers/trijoint.py.151: total_norm: 2.563313 [I 2022-10-08 14:37:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 1280/2304 [I 2022-10-08 14:37:09] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:13:12 [I 2022-10-08 14:37:09] ...trap/engines/engine.py.229: process: 0.77799 | load: 0.00301 [I 2022-10-08 14:37:09] ...trap/engines/engine.py.230: loss: 0.18048 [I 2022-10-08 14:37:09] ...optimizers/trijoint.py.151: total_norm: 2.268239 [I 2022-10-08 14:37:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 1290/2304 [I 2022-10-08 14:37:17] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:13:05 [I 2022-10-08 14:37:17] ...trap/engines/engine.py.229: process: 0.77247 | load: 0.00304 [I 2022-10-08 14:37:17] ...trap/engines/engine.py.230: loss: 0.16231 [I 2022-10-08 14:37:17] ...optimizers/trijoint.py.151: total_norm: 3.268429 [I 2022-10-08 14:37:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 1300/2304 [I 2022-10-08 14:37:25] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:12:58 [I 2022-10-08 14:37:25] ...trap/engines/engine.py.229: process: 0.78277 | load: 0.00305 [I 2022-10-08 14:37:25] ...trap/engines/engine.py.230: loss: 0.32728 [I 2022-10-08 14:37:25] ...optimizers/trijoint.py.151: total_norm: 3.578188 [I 2022-10-08 14:37:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1310/2304 [I 2022-10-08 14:37:33] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:12:53 [I 2022-10-08 14:37:33] ...trap/engines/engine.py.229: process: 0.78023 | load: 0.00305 [I 2022-10-08 14:37:33] ...trap/engines/engine.py.230: loss: 0.18369 [I 2022-10-08 14:37:33] ...optimizers/trijoint.py.151: total_norm: 2.127922 [I 2022-10-08 14:37:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 1320/2304 [I 2022-10-08 14:37:40] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:12:45 [I 2022-10-08 14:37:40] ...trap/engines/engine.py.229: process: 0.77836 | load: 0.00303 [I 2022-10-08 14:37:40] ...trap/engines/engine.py.230: loss: 0.16790 [I 2022-10-08 14:37:40] ...optimizers/trijoint.py.151: total_norm: 2.801606 [I 2022-10-08 14:37:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1330/2304 [I 2022-10-08 14:37:48] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:12:32 [I 2022-10-08 14:37:48] ...trap/engines/engine.py.229: process: 0.76806 | load: 0.00301 [I 2022-10-08 14:37:48] ...trap/engines/engine.py.230: loss: 0.16184 [I 2022-10-08 14:37:48] ...optimizers/trijoint.py.151: total_norm: 2.146614 [I 2022-10-08 14:37:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 1340/2304 [I 2022-10-08 14:37:56] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:12:29 [I 2022-10-08 14:37:56] ...trap/engines/engine.py.229: process: 0.77633 | load: 0.00303 [I 2022-10-08 14:37:56] ...trap/engines/engine.py.230: loss: 0.22866 [I 2022-10-08 14:37:56] ...optimizers/trijoint.py.151: total_norm: 2.142279 [I 2022-10-08 14:38:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 1350/2304 [I 2022-10-08 14:38:04] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:12:24 [I 2022-10-08 14:38:04] ...trap/engines/engine.py.229: process: 0.78312 | load: 0.00301 [I 2022-10-08 14:38:04] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-08 14:38:04] ...optimizers/trijoint.py.151: total_norm: 1.974600 [I 2022-10-08 14:38:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 1360/2304 [I 2022-10-08 14:38:12] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:12:14 [I 2022-10-08 14:38:12] ...trap/engines/engine.py.229: process: 0.77628 | load: 0.00302 [I 2022-10-08 14:38:12] ...trap/engines/engine.py.230: loss: 0.20072 [I 2022-10-08 14:38:12] ...optimizers/trijoint.py.151: total_norm: 3.044237 [I 2022-10-08 14:38:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 1370/2304 [I 2022-10-08 14:38:19] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:12:11 [I 2022-10-08 14:38:19] ...trap/engines/engine.py.229: process: 0.77959 | load: 0.00303 [I 2022-10-08 14:38:19] ...trap/engines/engine.py.230: loss: 0.11223 [I 2022-10-08 14:38:19] ...optimizers/trijoint.py.151: total_norm: 2.292472 [I 2022-10-08 14:38:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 1380/2304 [I 2022-10-08 14:38:27] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:11:56 [I 2022-10-08 14:38:27] ...trap/engines/engine.py.229: process: 0.76783 | load: 0.00305 [I 2022-10-08 14:38:27] ...trap/engines/engine.py.230: loss: 0.18046 [I 2022-10-08 14:38:27] ...optimizers/trijoint.py.151: total_norm: 2.672495 [I 2022-10-08 14:38:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 1390/2304 [I 2022-10-08 14:38:35] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:11:54 [I 2022-10-08 14:38:35] ...trap/engines/engine.py.229: process: 0.78229 | load: 0.00308 [I 2022-10-08 14:38:35] ...trap/engines/engine.py.230: loss: 0.21617 [I 2022-10-08 14:38:35] ...optimizers/trijoint.py.151: total_norm: 2.134603 [I 2022-10-08 14:39:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 1400/2304 [I 2022-10-08 14:39:01] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:46:38 [I 2022-10-08 14:39:01] ...trap/engines/engine.py.229: process: 0.75950 | load: 0.00301 [I 2022-10-08 14:39:01] ...trap/engines/engine.py.230: loss: 0.15621 [I 2022-10-08 14:39:01] ...optimizers/trijoint.py.151: total_norm: 2.651911 [I 2022-10-08 14:39:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 1410/2304 [I 2022-10-08 14:39:09] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:15:18 [I 2022-10-08 14:39:09] ...trap/engines/engine.py.229: process: 0.78606 | load: 0.00305 [I 2022-10-08 14:39:09] ...trap/engines/engine.py.230: loss: 0.26260 [I 2022-10-08 14:39:09] ...optimizers/trijoint.py.151: total_norm: 3.049859 [I 2022-10-08 14:39:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 1420/2304 [I 2022-10-08 14:39:17] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:46 [I 2022-10-08 14:39:17] ...trap/engines/engine.py.229: process: 0.77183 | load: 0.00302 [I 2022-10-08 14:39:17] ...trap/engines/engine.py.230: loss: 0.25245 [I 2022-10-08 14:39:17] ...optimizers/trijoint.py.151: total_norm: 2.477916 [I 2022-10-08 14:39:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 1430/2304 [I 2022-10-08 14:39:24] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:17 [I 2022-10-08 14:39:24] ...trap/engines/engine.py.229: process: 0.76856 | load: 0.00306 [I 2022-10-08 14:39:24] ...trap/engines/engine.py.230: loss: 0.15802 [I 2022-10-08 14:39:24] ...optimizers/trijoint.py.151: total_norm: 2.276178 [I 2022-10-08 14:39:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 1440/2304 [I 2022-10-08 14:39:32] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:11:12 [I 2022-10-08 14:39:32] ...trap/engines/engine.py.229: process: 0.77414 | load: 0.00307 [I 2022-10-08 14:39:32] ...trap/engines/engine.py.230: loss: 0.16651 [I 2022-10-08 14:39:32] ...optimizers/trijoint.py.151: total_norm: 2.404039 [I 2022-10-08 14:39:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 1450/2304 [I 2022-10-08 14:39:40] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:11:06 [I 2022-10-08 14:39:40] ...trap/engines/engine.py.229: process: 0.78658 | load: 0.00302 [I 2022-10-08 14:39:40] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-08 14:39:40] ...optimizers/trijoint.py.151: total_norm: 2.606273 [I 2022-10-08 14:39:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1460/2304 [I 2022-10-08 14:39:48] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:10:57 [I 2022-10-08 14:39:48] ...trap/engines/engine.py.229: process: 0.78176 | load: 0.00306 [I 2022-10-08 14:39:48] ...trap/engines/engine.py.230: loss: 0.35697 [I 2022-10-08 14:39:48] ...optimizers/trijoint.py.151: total_norm: 4.637905 [I 2022-10-08 14:39:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 1470/2304 [I 2022-10-08 14:39:55] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:47 [I 2022-10-08 14:39:55] ...trap/engines/engine.py.229: process: 0.77927 | load: 0.00300 [I 2022-10-08 14:39:55] ...trap/engines/engine.py.230: loss: 0.17484 [I 2022-10-08 14:39:55] ...optimizers/trijoint.py.151: total_norm: 2.215923 [I 2022-10-08 14:40:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 1480/2304 [I 2022-10-08 14:40:03] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:40 [I 2022-10-08 14:40:03] ...trap/engines/engine.py.229: process: 0.77547 | load: 0.00304 [I 2022-10-08 14:40:03] ...trap/engines/engine.py.230: loss: 0.19148 [I 2022-10-08 14:40:03] ...optimizers/trijoint.py.151: total_norm: 2.273287 [I 2022-10-08 14:40:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 1490/2304 [I 2022-10-08 14:40:11] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:31 [I 2022-10-08 14:40:11] ...trap/engines/engine.py.229: process: 0.76073 | load: 0.00302 [I 2022-10-08 14:40:11] ...trap/engines/engine.py.230: loss: 0.26170 [I 2022-10-08 14:40:11] ...optimizers/trijoint.py.151: total_norm: 2.465332 [I 2022-10-08 14:40:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 1500/2304 [I 2022-10-08 14:40:19] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:23 [I 2022-10-08 14:40:19] ...trap/engines/engine.py.229: process: 0.77629 | load: 0.00306 [I 2022-10-08 14:40:19] ...trap/engines/engine.py.230: loss: 0.18882 [I 2022-10-08 14:40:19] ...optimizers/trijoint.py.151: total_norm: 2.387237 [I 2022-10-08 14:40:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 1510/2304 [I 2022-10-08 14:40:27] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:14 [I 2022-10-08 14:40:27] ...trap/engines/engine.py.229: process: 0.77545 | load: 0.00300 [I 2022-10-08 14:40:27] ...trap/engines/engine.py.230: loss: 0.15294 [I 2022-10-08 14:40:27] ...optimizers/trijoint.py.151: total_norm: 2.300813 [I 2022-10-08 14:40:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 1520/2304 [I 2022-10-08 14:40:34] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:10:09 [I 2022-10-08 14:40:34] ...trap/engines/engine.py.229: process: 0.77428 | load: 0.00308 [I 2022-10-08 14:40:34] ...trap/engines/engine.py.230: loss: 0.16868 [I 2022-10-08 14:40:34] ...optimizers/trijoint.py.151: total_norm: 2.186256 [I 2022-10-08 14:40:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 1530/2304 [I 2022-10-08 14:40:42] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:10:03 [I 2022-10-08 14:40:42] ...trap/engines/engine.py.229: process: 0.78240 | load: 0.00301 [I 2022-10-08 14:40:42] ...trap/engines/engine.py.230: loss: 0.14356 [I 2022-10-08 14:40:42] ...optimizers/trijoint.py.151: total_norm: 2.729690 [I 2022-10-08 14:40:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 1540/2304 [I 2022-10-08 14:40:50] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:56 [I 2022-10-08 14:40:50] ...trap/engines/engine.py.229: process: 0.79619 | load: 0.00302 [I 2022-10-08 14:40:50] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-08 14:40:50] ...optimizers/trijoint.py.151: total_norm: 1.778309 [I 2022-10-08 14:40:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 1550/2304 [I 2022-10-08 14:40:58] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:09:44 [I 2022-10-08 14:40:58] ...trap/engines/engine.py.229: process: 0.78831 | load: 0.00299 [I 2022-10-08 14:40:58] ...trap/engines/engine.py.230: loss: 0.16511 [I 2022-10-08 14:40:58] ...optimizers/trijoint.py.151: total_norm: 3.484151 [I 2022-10-08 14:41:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 1560/2304 [I 2022-10-08 14:41:05] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:37 [I 2022-10-08 14:41:05] ...trap/engines/engine.py.229: process: 0.77769 | load: 0.00304 [I 2022-10-08 14:41:05] ...trap/engines/engine.py.230: loss: 0.20723 [I 2022-10-08 14:41:05] ...optimizers/trijoint.py.151: total_norm: 2.623608 [I 2022-10-08 14:41:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1570/2304 [I 2022-10-08 14:41:13] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:28 [I 2022-10-08 14:41:13] ...trap/engines/engine.py.229: process: 0.77203 | load: 0.00305 [I 2022-10-08 14:41:13] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-08 14:41:13] ...optimizers/trijoint.py.151: total_norm: 2.742733 [I 2022-10-08 14:41:21] ...trap/engines/engine.py.224: train: epoch 8 | batch 1580/2304 [I 2022-10-08 14:41:21] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:22 [I 2022-10-08 14:41:21] ...trap/engines/engine.py.229: process: 0.77975 | load: 0.00312 [I 2022-10-08 14:41:21] ...trap/engines/engine.py.230: loss: 0.27466 [I 2022-10-08 14:41:21] ...optimizers/trijoint.py.151: total_norm: 3.287558 [I 2022-10-08 14:41:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 1590/2304 [I 2022-10-08 14:41:29] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:09:16 [I 2022-10-08 14:41:29] ...trap/engines/engine.py.229: process: 0.77988 | load: 0.00310 [I 2022-10-08 14:41:29] ...trap/engines/engine.py.230: loss: 0.20049 [I 2022-10-08 14:41:29] ...optimizers/trijoint.py.151: total_norm: 2.896181 [I 2022-10-08 14:41:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 1600/2304 [I 2022-10-08 14:41:37] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:09:07 [I 2022-10-08 14:41:37] ...trap/engines/engine.py.229: process: 0.78271 | load: 0.00312 [I 2022-10-08 14:41:37] ...trap/engines/engine.py.230: loss: 0.21894 [I 2022-10-08 14:41:37] ...optimizers/trijoint.py.151: total_norm: 2.919992 [I 2022-10-08 14:41:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 1610/2304 [I 2022-10-08 14:41:44] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:09:02 [I 2022-10-08 14:41:44] ...trap/engines/engine.py.229: process: 0.78687 | load: 0.00310 [I 2022-10-08 14:41:44] ...trap/engines/engine.py.230: loss: 0.34638 [I 2022-10-08 14:41:44] ...optimizers/trijoint.py.151: total_norm: 3.207620 [I 2022-10-08 14:41:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 1620/2304 [I 2022-10-08 14:41:52] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:54 [I 2022-10-08 14:41:52] ...trap/engines/engine.py.229: process: 0.78454 | load: 0.00312 [I 2022-10-08 14:41:52] ...trap/engines/engine.py.230: loss: 0.18434 [I 2022-10-08 14:41:52] ...optimizers/trijoint.py.151: total_norm: 2.532923 [I 2022-10-08 14:42:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 1630/2304 [I 2022-10-08 14:42:00] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:46 [I 2022-10-08 14:42:00] ...trap/engines/engine.py.229: process: 0.79336 | load: 0.00314 [I 2022-10-08 14:42:00] ...trap/engines/engine.py.230: loss: 0.22144 [I 2022-10-08 14:42:00] ...optimizers/trijoint.py.151: total_norm: 2.656020 [I 2022-10-08 14:42:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 1640/2304 [I 2022-10-08 14:42:08] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:38 [I 2022-10-08 14:42:08] ...trap/engines/engine.py.229: process: 0.78342 | load: 0.00317 [I 2022-10-08 14:42:08] ...trap/engines/engine.py.230: loss: 0.28772 [I 2022-10-08 14:42:08] ...optimizers/trijoint.py.151: total_norm: 3.952712 [I 2022-10-08 14:42:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 1650/2304 [I 2022-10-08 14:42:16] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:30 [I 2022-10-08 14:42:16] ...trap/engines/engine.py.229: process: 0.77820 | load: 0.00311 [I 2022-10-08 14:42:16] ...trap/engines/engine.py.230: loss: 0.35197 [I 2022-10-08 14:42:16] ...optimizers/trijoint.py.151: total_norm: 3.997118 [I 2022-10-08 14:42:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 1660/2304 [I 2022-10-08 14:42:24] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:20 [I 2022-10-08 14:42:24] ...trap/engines/engine.py.229: process: 0.77832 | load: 0.00314 [I 2022-10-08 14:42:24] ...trap/engines/engine.py.230: loss: 0.29175 [I 2022-10-08 14:42:24] ...optimizers/trijoint.py.151: total_norm: 3.498070 [I 2022-10-08 14:42:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 1670/2304 [I 2022-10-08 14:42:31] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:08:14 [I 2022-10-08 14:42:31] ...trap/engines/engine.py.229: process: 0.77716 | load: 0.00310 [I 2022-10-08 14:42:31] ...trap/engines/engine.py.230: loss: 0.27692 [I 2022-10-08 14:42:31] ...optimizers/trijoint.py.151: total_norm: 3.351846 [I 2022-10-08 14:42:39] ...trap/engines/engine.py.224: train: epoch 8 | batch 1680/2304 [I 2022-10-08 14:42:39] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:08:06 [I 2022-10-08 14:42:39] ...trap/engines/engine.py.229: process: 0.79355 | load: 0.00315 [I 2022-10-08 14:42:39] ...trap/engines/engine.py.230: loss: 0.26527 [I 2022-10-08 14:42:39] ...optimizers/trijoint.py.151: total_norm: 2.906740 [I 2022-10-08 14:42:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 1690/2304 [I 2022-10-08 14:42:47] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:56 [I 2022-10-08 14:42:47] ...trap/engines/engine.py.229: process: 0.77488 | load: 0.00311 [I 2022-10-08 14:42:47] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-08 14:42:47] ...optimizers/trijoint.py.151: total_norm: 1.989691 [I 2022-10-08 14:42:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 1700/2304 [I 2022-10-08 14:42:55] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:07:49 [I 2022-10-08 14:42:55] ...trap/engines/engine.py.229: process: 0.77924 | load: 0.00306 [I 2022-10-08 14:42:55] ...trap/engines/engine.py.230: loss: 0.22427 [I 2022-10-08 14:42:55] ...optimizers/trijoint.py.151: total_norm: 3.455693 [I 2022-10-08 14:43:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 1710/2304 [I 2022-10-08 14:43:03] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:42 [I 2022-10-08 14:43:03] ...trap/engines/engine.py.229: process: 0.77137 | load: 0.00322 [I 2022-10-08 14:43:03] ...trap/engines/engine.py.230: loss: 0.19256 [I 2022-10-08 14:43:03] ...optimizers/trijoint.py.151: total_norm: 2.375685 [I 2022-10-08 14:43:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 1720/2304 [I 2022-10-08 14:43:10] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:35 [I 2022-10-08 14:43:10] ...trap/engines/engine.py.229: process: 0.78236 | load: 0.00303 [I 2022-10-08 14:43:10] ...trap/engines/engine.py.230: loss: 0.15276 [I 2022-10-08 14:43:10] ...optimizers/trijoint.py.151: total_norm: 2.769012 [I 2022-10-08 14:43:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1730/2304 [I 2022-10-08 14:43:18] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:26 [I 2022-10-08 14:43:18] ...trap/engines/engine.py.229: process: 0.78578 | load: 0.00302 [I 2022-10-08 14:43:18] ...trap/engines/engine.py.230: loss: 0.23571 [I 2022-10-08 14:43:18] ...optimizers/trijoint.py.151: total_norm: 3.830382 [I 2022-10-08 14:43:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 1740/2304 [I 2022-10-08 14:43:26] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:16 [I 2022-10-08 14:43:26] ...trap/engines/engine.py.229: process: 0.76227 | load: 0.00305 [I 2022-10-08 14:43:26] ...trap/engines/engine.py.230: loss: 0.18140 [I 2022-10-08 14:43:26] ...optimizers/trijoint.py.151: total_norm: 2.275100 [I 2022-10-08 14:43:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 1750/2304 [I 2022-10-08 14:43:34] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:07:07 [I 2022-10-08 14:43:34] ...trap/engines/engine.py.229: process: 0.76914 | load: 0.00303 [I 2022-10-08 14:43:34] ...trap/engines/engine.py.230: loss: 0.26210 [I 2022-10-08 14:43:34] ...optimizers/trijoint.py.151: total_norm: 2.580245 [I 2022-10-08 14:43:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 1760/2304 [I 2022-10-08 14:43:41] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:07:02 [I 2022-10-08 14:43:41] ...trap/engines/engine.py.229: process: 0.77738 | load: 0.00305 [I 2022-10-08 14:43:41] ...trap/engines/engine.py.230: loss: 0.24439 [I 2022-10-08 14:43:41] ...optimizers/trijoint.py.151: total_norm: 2.779931 [I 2022-10-08 14:43:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 1770/2304 [I 2022-10-08 14:43:49] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:55 [I 2022-10-08 14:43:49] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00302 [I 2022-10-08 14:43:49] ...trap/engines/engine.py.230: loss: 0.21857 [I 2022-10-08 14:43:49] ...optimizers/trijoint.py.151: total_norm: 2.391741 [I 2022-10-08 14:43:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 1780/2304 [I 2022-10-08 14:43:57] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:48 [I 2022-10-08 14:43:57] ...trap/engines/engine.py.229: process: 0.78604 | load: 0.00302 [I 2022-10-08 14:43:57] ...trap/engines/engine.py.230: loss: 0.17354 [I 2022-10-08 14:43:57] ...optimizers/trijoint.py.151: total_norm: 2.213366 [I 2022-10-08 14:44:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 1790/2304 [I 2022-10-08 14:44:05] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:41 [I 2022-10-08 14:44:05] ...trap/engines/engine.py.229: process: 0.79105 | load: 0.00299 [I 2022-10-08 14:44:05] ...trap/engines/engine.py.230: loss: 0.14431 [I 2022-10-08 14:44:05] ...optimizers/trijoint.py.151: total_norm: 1.637609 [I 2022-10-08 14:44:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1800/2304 [I 2022-10-08 14:44:13] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:06:32 [I 2022-10-08 14:44:13] ...trap/engines/engine.py.229: process: 0.77915 | load: 0.00304 [I 2022-10-08 14:44:13] ...trap/engines/engine.py.230: loss: 0.21190 [I 2022-10-08 14:44:13] ...optimizers/trijoint.py.151: total_norm: 2.274396 [I 2022-10-08 14:44:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 1810/2304 [I 2022-10-08 14:44:20] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:24 [I 2022-10-08 14:44:20] ...trap/engines/engine.py.229: process: 0.77605 | load: 0.00303 [I 2022-10-08 14:44:20] ...trap/engines/engine.py.230: loss: 0.23149 [I 2022-10-08 14:44:20] ...optimizers/trijoint.py.151: total_norm: 3.638227 [I 2022-10-08 14:44:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1820/2304 [I 2022-10-08 14:44:28] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:14 [I 2022-10-08 14:44:28] ...trap/engines/engine.py.229: process: 0.76479 | load: 0.00304 [I 2022-10-08 14:44:28] ...trap/engines/engine.py.230: loss: 0.19576 [I 2022-10-08 14:44:28] ...optimizers/trijoint.py.151: total_norm: 2.908587 [I 2022-10-08 14:44:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 1830/2304 [I 2022-10-08 14:44:36] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:06:05 [I 2022-10-08 14:44:36] ...trap/engines/engine.py.229: process: 0.75665 | load: 0.00301 [I 2022-10-08 14:44:36] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-08 14:44:36] ...optimizers/trijoint.py.151: total_norm: 1.927987 [I 2022-10-08 14:44:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 1840/2304 [I 2022-10-08 14:44:44] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:59 [I 2022-10-08 14:44:44] ...trap/engines/engine.py.229: process: 0.78030 | load: 0.00303 [I 2022-10-08 14:44:44] ...trap/engines/engine.py.230: loss: 0.18141 [I 2022-10-08 14:44:44] ...optimizers/trijoint.py.151: total_norm: 2.296343 [I 2022-10-08 14:44:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 1850/2304 [I 2022-10-08 14:44:52] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:53 [I 2022-10-08 14:44:52] ...trap/engines/engine.py.229: process: 0.78365 | load: 0.00302 [I 2022-10-08 14:44:52] ...trap/engines/engine.py.230: loss: 0.22034 [I 2022-10-08 14:44:52] ...optimizers/trijoint.py.151: total_norm: 2.384408 [I 2022-10-08 14:44:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 1860/2304 [I 2022-10-08 14:44:59] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:05:46 [I 2022-10-08 14:44:59] ...trap/engines/engine.py.229: process: 0.78651 | load: 0.00307 [I 2022-10-08 14:44:59] ...trap/engines/engine.py.230: loss: 0.24986 [I 2022-10-08 14:44:59] ...optimizers/trijoint.py.151: total_norm: 2.660276 [I 2022-10-08 14:45:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 1870/2304 [I 2022-10-08 14:45:07] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:05:36 [I 2022-10-08 14:45:07] ...trap/engines/engine.py.229: process: 0.77790 | load: 0.00305 [I 2022-10-08 14:45:07] ...trap/engines/engine.py.230: loss: 0.22015 [I 2022-10-08 14:45:07] ...optimizers/trijoint.py.151: total_norm: 2.460552 [I 2022-10-08 14:45:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 1880/2304 [I 2022-10-08 14:45:15] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:29 [I 2022-10-08 14:45:15] ...trap/engines/engine.py.229: process: 0.77637 | load: 0.00301 [I 2022-10-08 14:45:15] ...trap/engines/engine.py.230: loss: 0.21833 [I 2022-10-08 14:45:15] ...optimizers/trijoint.py.151: total_norm: 2.867935 [I 2022-10-08 14:45:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1890/2304 [I 2022-10-08 14:45:23] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:21 [I 2022-10-08 14:45:23] ...trap/engines/engine.py.229: process: 0.77459 | load: 0.00302 [I 2022-10-08 14:45:23] ...trap/engines/engine.py.230: loss: 0.25588 [I 2022-10-08 14:45:23] ...optimizers/trijoint.py.151: total_norm: 3.735284 [I 2022-10-08 14:45:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 1900/2304 [I 2022-10-08 14:45:31] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:05:13 [I 2022-10-08 14:45:31] ...trap/engines/engine.py.229: process: 0.78029 | load: 0.00306 [I 2022-10-08 14:45:31] ...trap/engines/engine.py.230: loss: 0.21762 [I 2022-10-08 14:45:31] ...optimizers/trijoint.py.151: total_norm: 3.068824 [I 2022-10-08 14:45:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1910/2304 [I 2022-10-08 14:45:38] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:05:06 [I 2022-10-08 14:45:38] ...trap/engines/engine.py.229: process: 0.77196 | load: 0.00303 [I 2022-10-08 14:45:38] ...trap/engines/engine.py.230: loss: 0.21515 [I 2022-10-08 14:45:38] ...optimizers/trijoint.py.151: total_norm: 3.030020 [I 2022-10-08 14:45:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 1920/2304 [I 2022-10-08 14:45:46] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:04:59 [I 2022-10-08 14:45:46] ...trap/engines/engine.py.229: process: 0.76579 | load: 0.00305 [I 2022-10-08 14:45:46] ...trap/engines/engine.py.230: loss: 0.21218 [I 2022-10-08 14:45:46] ...optimizers/trijoint.py.151: total_norm: 2.582590 [I 2022-10-08 14:45:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 1930/2304 [I 2022-10-08 14:45:54] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:51 [I 2022-10-08 14:45:54] ...trap/engines/engine.py.229: process: 0.76488 | load: 0.00304 [I 2022-10-08 14:45:54] ...trap/engines/engine.py.230: loss: 0.18163 [I 2022-10-08 14:45:54] ...optimizers/trijoint.py.151: total_norm: 2.804015 [I 2022-10-08 14:46:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1940/2304 [I 2022-10-08 14:46:02] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:04:43 [I 2022-10-08 14:46:02] ...trap/engines/engine.py.229: process: 0.78468 | load: 0.00301 [I 2022-10-08 14:46:02] ...trap/engines/engine.py.230: loss: 0.22051 [I 2022-10-08 14:46:02] ...optimizers/trijoint.py.151: total_norm: 2.903737 [I 2022-10-08 14:46:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 1950/2304 [I 2022-10-08 14:46:10] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:04:35 [I 2022-10-08 14:46:10] ...trap/engines/engine.py.229: process: 0.78331 | load: 0.00303 [I 2022-10-08 14:46:10] ...trap/engines/engine.py.230: loss: 0.22504 [I 2022-10-08 14:46:10] ...optimizers/trijoint.py.151: total_norm: 2.890311 [I 2022-10-08 14:46:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 1960/2304 [I 2022-10-08 14:46:17] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:04:27 [I 2022-10-08 14:46:17] ...trap/engines/engine.py.229: process: 0.77062 | load: 0.00310 [I 2022-10-08 14:46:17] ...trap/engines/engine.py.230: loss: 0.19955 [I 2022-10-08 14:46:17] ...optimizers/trijoint.py.151: total_norm: 2.709871 [I 2022-10-08 14:46:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 1970/2304 [I 2022-10-08 14:46:25] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:19 [I 2022-10-08 14:46:25] ...trap/engines/engine.py.229: process: 0.77822 | load: 0.00308 [I 2022-10-08 14:46:25] ...trap/engines/engine.py.230: loss: 0.40217 [I 2022-10-08 14:46:25] ...optimizers/trijoint.py.151: total_norm: 3.168728 [I 2022-10-08 14:46:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1980/2304 [I 2022-10-08 14:46:33] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:04:10 [I 2022-10-08 14:46:33] ...trap/engines/engine.py.229: process: 0.77346 | load: 0.00304 [I 2022-10-08 14:46:33] ...trap/engines/engine.py.230: loss: 0.19525 [I 2022-10-08 14:46:33] ...optimizers/trijoint.py.151: total_norm: 1.980940 [I 2022-10-08 14:46:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 1990/2304 [I 2022-10-08 14:46:41] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:04:03 [I 2022-10-08 14:46:41] ...trap/engines/engine.py.229: process: 0.77499 | load: 0.00305 [I 2022-10-08 14:46:41] ...trap/engines/engine.py.230: loss: 0.23564 [I 2022-10-08 14:46:41] ...optimizers/trijoint.py.151: total_norm: 2.301960 [I 2022-10-08 14:46:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 2000/2304 [I 2022-10-08 14:46:49] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:03:56 [I 2022-10-08 14:46:49] ...trap/engines/engine.py.229: process: 0.77946 | load: 0.00306 [I 2022-10-08 14:46:49] ...trap/engines/engine.py.230: loss: 0.22880 [I 2022-10-08 14:46:49] ...optimizers/trijoint.py.151: total_norm: 2.973679 [I 2022-10-08 14:46:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 2010/2304 [I 2022-10-08 14:46:56] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:03:48 [I 2022-10-08 14:46:56] ...trap/engines/engine.py.229: process: 0.76773 | load: 0.00302 [I 2022-10-08 14:46:56] ...trap/engines/engine.py.230: loss: 0.15471 [I 2022-10-08 14:46:56] ...optimizers/trijoint.py.151: total_norm: 2.163867 [I 2022-10-08 14:47:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 2020/2304 [I 2022-10-08 14:47:04] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:03:39 [I 2022-10-08 14:47:04] ...trap/engines/engine.py.229: process: 0.77611 | load: 0.00302 [I 2022-10-08 14:47:04] ...trap/engines/engine.py.230: loss: 0.20077 [I 2022-10-08 14:47:04] ...optimizers/trijoint.py.151: total_norm: 1.972960 [I 2022-10-08 14:47:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 2030/2304 [I 2022-10-08 14:47:12] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:03:32 [I 2022-10-08 14:47:12] ...trap/engines/engine.py.229: process: 0.76930 | load: 0.00303 [I 2022-10-08 14:47:12] ...trap/engines/engine.py.230: loss: 0.21113 [I 2022-10-08 14:47:12] ...optimizers/trijoint.py.151: total_norm: 2.287924 [I 2022-10-08 14:47:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 2040/2304 [I 2022-10-08 14:47:20] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:03:24 [I 2022-10-08 14:47:20] ...trap/engines/engine.py.229: process: 0.78445 | load: 0.00301 [I 2022-10-08 14:47:20] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-08 14:47:20] ...optimizers/trijoint.py.151: total_norm: 2.413342 [I 2022-10-08 14:47:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 2050/2304 [I 2022-10-08 14:47:27] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:03:17 [I 2022-10-08 14:47:27] ...trap/engines/engine.py.229: process: 0.78296 | load: 0.00303 [I 2022-10-08 14:47:27] ...trap/engines/engine.py.230: loss: 0.22396 [I 2022-10-08 14:47:27] ...optimizers/trijoint.py.151: total_norm: 3.628309 [I 2022-10-08 14:47:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 2060/2304 [I 2022-10-08 14:47:35] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:03:10 [I 2022-10-08 14:47:35] ...trap/engines/engine.py.229: process: 0.80504 | load: 0.00302 [I 2022-10-08 14:47:35] ...trap/engines/engine.py.230: loss: 0.27579 [I 2022-10-08 14:47:35] ...optimizers/trijoint.py.151: total_norm: 3.095441 [I 2022-10-08 14:47:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 2070/2304 [I 2022-10-08 14:47:43] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:03:02 [I 2022-10-08 14:47:43] ...trap/engines/engine.py.229: process: 0.77354 | load: 0.00310 [I 2022-10-08 14:47:43] ...trap/engines/engine.py.230: loss: 0.16685 [I 2022-10-08 14:47:43] ...optimizers/trijoint.py.151: total_norm: 2.544627 [I 2022-10-08 14:47:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 2080/2304 [I 2022-10-08 14:47:51] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:54 [I 2022-10-08 14:47:51] ...trap/engines/engine.py.229: process: 0.78288 | load: 0.00306 [I 2022-10-08 14:47:51] ...trap/engines/engine.py.230: loss: 0.18021 [I 2022-10-08 14:47:51] ...optimizers/trijoint.py.151: total_norm: 2.566084 [I 2022-10-08 14:47:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 2090/2304 [I 2022-10-08 14:47:59] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:46 [I 2022-10-08 14:47:59] ...trap/engines/engine.py.229: process: 0.78576 | load: 0.00301 [I 2022-10-08 14:47:59] ...trap/engines/engine.py.230: loss: 0.20567 [I 2022-10-08 14:47:59] ...optimizers/trijoint.py.151: total_norm: 3.727260 [I 2022-10-08 14:48:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 2100/2304 [I 2022-10-08 14:48:06] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:02:39 [I 2022-10-08 14:48:06] ...trap/engines/engine.py.229: process: 0.79770 | load: 0.00310 [I 2022-10-08 14:48:06] ...trap/engines/engine.py.230: loss: 0.20329 [I 2022-10-08 14:48:06] ...optimizers/trijoint.py.151: total_norm: 2.544928 [I 2022-10-08 14:48:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 2110/2304 [I 2022-10-08 14:48:14] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:02:31 [I 2022-10-08 14:48:14] ...trap/engines/engine.py.229: process: 0.78243 | load: 0.00310 [I 2022-10-08 14:48:14] ...trap/engines/engine.py.230: loss: 0.18747 [I 2022-10-08 14:48:14] ...optimizers/trijoint.py.151: total_norm: 2.024701 [I 2022-10-08 14:48:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 2120/2304 [I 2022-10-08 14:48:22] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:02:23 [I 2022-10-08 14:48:22] ...trap/engines/engine.py.229: process: 0.78331 | load: 0.00303 [I 2022-10-08 14:48:22] ...trap/engines/engine.py.230: loss: 0.19325 [I 2022-10-08 14:48:22] ...optimizers/trijoint.py.151: total_norm: 2.428033 [I 2022-10-08 14:48:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 2130/2304 [I 2022-10-08 14:48:30] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:02:15 [I 2022-10-08 14:48:30] ...trap/engines/engine.py.229: process: 0.77059 | load: 0.00305 [I 2022-10-08 14:48:30] ...trap/engines/engine.py.230: loss: 0.23301 [I 2022-10-08 14:48:30] ...optimizers/trijoint.py.151: total_norm: 3.259490 [I 2022-10-08 14:48:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 2140/2304 [I 2022-10-08 14:48:38] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:02:07 [I 2022-10-08 14:48:38] ...trap/engines/engine.py.229: process: 0.78325 | load: 0.00302 [I 2022-10-08 14:48:38] ...trap/engines/engine.py.230: loss: 0.19175 [I 2022-10-08 14:48:38] ...optimizers/trijoint.py.151: total_norm: 2.568902 [I 2022-10-08 14:48:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 2150/2304 [I 2022-10-08 14:48:45] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:01:59 [I 2022-10-08 14:48:45] ...trap/engines/engine.py.229: process: 0.77133 | load: 0.00304 [I 2022-10-08 14:48:45] ...trap/engines/engine.py.230: loss: 0.18027 [I 2022-10-08 14:48:45] ...optimizers/trijoint.py.151: total_norm: 2.305982 [I 2022-10-08 14:48:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 2160/2304 [I 2022-10-08 14:48:53] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:01:52 [I 2022-10-08 14:48:53] ...trap/engines/engine.py.229: process: 0.76683 | load: 0.00310 [I 2022-10-08 14:48:53] ...trap/engines/engine.py.230: loss: 0.30054 [I 2022-10-08 14:48:53] ...optimizers/trijoint.py.151: total_norm: 3.129201 [I 2022-10-08 14:49:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 2170/2304 [I 2022-10-08 14:49:01] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:01:45 [I 2022-10-08 14:49:01] ...trap/engines/engine.py.229: process: 0.77268 | load: 0.00303 [I 2022-10-08 14:49:01] ...trap/engines/engine.py.230: loss: 0.18568 [I 2022-10-08 14:49:01] ...optimizers/trijoint.py.151: total_norm: 2.775083 [I 2022-10-08 14:49:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 2180/2304 [I 2022-10-08 14:49:09] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:01:36 [I 2022-10-08 14:49:09] ...trap/engines/engine.py.229: process: 0.78580 | load: 0.00303 [I 2022-10-08 14:49:09] ...trap/engines/engine.py.230: loss: 0.13234 [I 2022-10-08 14:49:09] ...optimizers/trijoint.py.151: total_norm: 1.940314 [I 2022-10-08 14:49:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 2190/2304 [I 2022-10-08 14:49:17] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:01:28 [I 2022-10-08 14:49:17] ...trap/engines/engine.py.229: process: 0.77684 | load: 0.00302 [I 2022-10-08 14:49:17] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-08 14:49:17] ...optimizers/trijoint.py.151: total_norm: 3.272706 [I 2022-10-08 14:49:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 2200/2304 [I 2022-10-08 14:49:24] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:01:21 [I 2022-10-08 14:49:24] ...trap/engines/engine.py.229: process: 0.77832 | load: 0.00303 [I 2022-10-08 14:49:24] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-08 14:49:24] ...optimizers/trijoint.py.151: total_norm: 2.486314 [I 2022-10-08 14:49:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 2210/2304 [I 2022-10-08 14:49:32] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:01:13 [I 2022-10-08 14:49:32] ...trap/engines/engine.py.229: process: 0.77248 | load: 0.00308 [I 2022-10-08 14:49:32] ...trap/engines/engine.py.230: loss: 0.13824 [I 2022-10-08 14:49:32] ...optimizers/trijoint.py.151: total_norm: 1.673183 [I 2022-10-08 14:49:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 2220/2304 [I 2022-10-08 14:49:40] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:01:04 [I 2022-10-08 14:49:40] ...trap/engines/engine.py.229: process: 0.77516 | load: 0.00303 [I 2022-10-08 14:49:40] ...trap/engines/engine.py.230: loss: 0.20345 [I 2022-10-08 14:49:40] ...optimizers/trijoint.py.151: total_norm: 2.952283 [I 2022-10-08 14:49:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 2230/2304 [I 2022-10-08 14:49:48] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:00:57 [I 2022-10-08 14:49:48] ...trap/engines/engine.py.229: process: 0.77830 | load: 0.00303 [I 2022-10-08 14:49:48] ...trap/engines/engine.py.230: loss: 0.26317 [I 2022-10-08 14:49:48] ...optimizers/trijoint.py.151: total_norm: 3.821895 [I 2022-10-08 14:49:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 2240/2304 [I 2022-10-08 14:49:56] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:00:49 [I 2022-10-08 14:49:56] ...trap/engines/engine.py.229: process: 0.78598 | load: 0.00305 [I 2022-10-08 14:49:56] ...trap/engines/engine.py.230: loss: 0.20115 [I 2022-10-08 14:49:56] ...optimizers/trijoint.py.151: total_norm: 2.105612 [I 2022-10-08 14:50:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 2250/2304 [I 2022-10-08 14:50:03] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:00:41 [I 2022-10-08 14:50:03] ...trap/engines/engine.py.229: process: 0.76978 | load: 0.00305 [I 2022-10-08 14:50:03] ...trap/engines/engine.py.230: loss: 0.28934 [I 2022-10-08 14:50:03] ...optimizers/trijoint.py.151: total_norm: 2.739141 [I 2022-10-08 14:50:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 2260/2304 [I 2022-10-08 14:50:11] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:00:34 [I 2022-10-08 14:50:11] ...trap/engines/engine.py.229: process: 0.77416 | load: 0.00301 [I 2022-10-08 14:50:11] ...trap/engines/engine.py.230: loss: 0.20361 [I 2022-10-08 14:50:11] ...optimizers/trijoint.py.151: total_norm: 2.774543 [I 2022-10-08 14:50:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 2270/2304 [I 2022-10-08 14:50:19] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:00:26 [I 2022-10-08 14:50:19] ...trap/engines/engine.py.229: process: 0.77697 | load: 0.00305 [I 2022-10-08 14:50:19] ...trap/engines/engine.py.230: loss: 0.21907 [I 2022-10-08 14:50:19] ...optimizers/trijoint.py.151: total_norm: 2.908068 [I 2022-10-08 14:50:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 2280/2304 [I 2022-10-08 14:50:27] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:00:18 [I 2022-10-08 14:50:27] ...trap/engines/engine.py.229: process: 0.77561 | load: 0.00308 [I 2022-10-08 14:50:27] ...trap/engines/engine.py.230: loss: 0.29901 [I 2022-10-08 14:50:27] ...optimizers/trijoint.py.151: total_norm: 3.140273 [I 2022-10-08 14:50:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 2290/2304 [I 2022-10-08 14:50:35] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:00:10 [I 2022-10-08 14:50:35] ...trap/engines/engine.py.229: process: 0.77129 | load: 0.00302 [I 2022-10-08 14:50:35] ...trap/engines/engine.py.230: loss: 0.23696 [I 2022-10-08 14:50:35] ...optimizers/trijoint.py.151: total_norm: 2.728088 [I 2022-10-08 14:50:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 2300/2304 [I 2022-10-08 14:50:42] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:00:03 [I 2022-10-08 14:50:42] ...trap/engines/engine.py.229: process: 0.76933 | load: 0.00017 [I 2022-10-08 14:50:42] ...trap/engines/engine.py.230: loss: 0.15297 [I 2022-10-08 14:50:42] ...optimizers/trijoint.py.151: total_norm: 2.123883 [I 2022-10-08 14:50:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 2304/2304 [I 2022-10-08 14:50:45] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:00:00 [I 2022-10-08 14:50:45] ...trap/engines/engine.py.229: process: 0.75997 | load: 0.00017 [I 2022-10-08 14:50:45] ...trap/engines/engine.py.230: loss: 0.18080 [I 2022-10-08 14:50:45] ...optimizers/trijoint.py.151: total_norm: 3.548529 [S 2022-10-08 14:50:47] ...trap/engines/engine.py.129: train_epoch.epoch: 8 [S 2022-10-08 14:50:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.20580098858818316 [I 2022-10-08 14:50:47] ...s/criterions/triplet.py.84: margin = 0.09500000000000003 [I 2022-10-08 14:50:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 8 [I 2022-10-08 14:50:47] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 14:50:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 14:50:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 14:50:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 14:50:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 14:50:50] ...trap/engines/engine.py.321: eval: epoch 8 | batch 0/510 [I 2022-10-08 14:50:50] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:14 [I 2022-10-08 14:50:50] ...trap/engines/engine.py.326: process: 2.73396 | load: 2.25189 [I 2022-10-08 14:50:55] ...trap/engines/engine.py.321: eval: epoch 8 | batch 10/510 [I 2022-10-08 14:50:55] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:34 [I 2022-10-08 14:50:55] ...trap/engines/engine.py.326: process: 0.48841 | load: 0.00018 [I 2022-10-08 14:51:01] ...trap/engines/engine.py.321: eval: epoch 8 | batch 20/510 [I 2022-10-08 14:51:01] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:22 [I 2022-10-08 14:51:01] ...trap/engines/engine.py.326: process: 0.68803 | load: 0.00019 [I 2022-10-08 14:51:08] ...trap/engines/engine.py.321: eval: epoch 8 | batch 30/510 [I 2022-10-08 14:51:08] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:05:27 [I 2022-10-08 14:51:08] ...trap/engines/engine.py.326: process: 0.78636 | load: 0.00019 [I 2022-10-08 14:51:13] ...trap/engines/engine.py.321: eval: epoch 8 | batch 40/510 [I 2022-10-08 14:51:13] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:22 [I 2022-10-08 14:51:13] ...trap/engines/engine.py.326: process: 0.42074 | load: 0.00018 [I 2022-10-08 14:51:18] ...trap/engines/engine.py.321: eval: epoch 8 | batch 50/510 [I 2022-10-08 14:51:18] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:29 [I 2022-10-08 14:51:18] ...trap/engines/engine.py.326: process: 0.41157 | load: 0.00017 [I 2022-10-08 14:51:22] ...trap/engines/engine.py.321: eval: epoch 8 | batch 60/510 [I 2022-10-08 14:51:22] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:25 [I 2022-10-08 14:51:22] ...trap/engines/engine.py.326: process: 0.45877 | load: 0.00018 [I 2022-10-08 14:51:27] ...trap/engines/engine.py.321: eval: epoch 8 | batch 70/510 [I 2022-10-08 14:51:27] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:09 [I 2022-10-08 14:51:27] ...trap/engines/engine.py.326: process: 0.42630 | load: 0.00019 [I 2022-10-08 14:51:32] ...trap/engines/engine.py.321: eval: epoch 8 | batch 80/510 [I 2022-10-08 14:51:32] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:50 [I 2022-10-08 14:51:32] ...trap/engines/engine.py.326: process: 0.54245 | load: 0.00020 [I 2022-10-08 14:51:37] ...trap/engines/engine.py.321: eval: epoch 8 | batch 90/510 [I 2022-10-08 14:51:37] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:10 [I 2022-10-08 14:51:37] ...trap/engines/engine.py.326: process: 0.44191 | load: 0.00018 [I 2022-10-08 14:51:41] ...trap/engines/engine.py.321: eval: epoch 8 | batch 100/510 [I 2022-10-08 14:51:41] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:51 [I 2022-10-08 14:51:41] ...trap/engines/engine.py.326: process: 0.30832 | load: 0.00018 [I 2022-10-08 14:51:44] ...trap/engines/engine.py.321: eval: epoch 8 | batch 110/510 [I 2022-10-08 14:51:44] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:11 [I 2022-10-08 14:51:44] ...trap/engines/engine.py.326: process: 0.31586 | load: 0.00020 [I 2022-10-08 14:51:47] ...trap/engines/engine.py.321: eval: epoch 8 | batch 120/510 [I 2022-10-08 14:51:47] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:04 [I 2022-10-08 14:51:47] ...trap/engines/engine.py.326: process: 0.31439 | load: 0.00020 [I 2022-10-08 14:51:51] ...trap/engines/engine.py.321: eval: epoch 8 | batch 130/510 [I 2022-10-08 14:51:51] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:59 [I 2022-10-08 14:51:51] ...trap/engines/engine.py.326: process: 0.30898 | load: 0.00020 [I 2022-10-08 14:51:54] ...trap/engines/engine.py.321: eval: epoch 8 | batch 140/510 [I 2022-10-08 14:51:54] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:57 [I 2022-10-08 14:51:54] ...trap/engines/engine.py.326: process: 0.31881 | load: 0.00019 [I 2022-10-08 14:51:57] ...trap/engines/engine.py.321: eval: epoch 8 | batch 150/510 [I 2022-10-08 14:51:57] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-08 14:51:57] ...trap/engines/engine.py.326: process: 0.31881 | load: 0.00019 [I 2022-10-08 14:52:00] ...trap/engines/engine.py.321: eval: epoch 8 | batch 160/510 [I 2022-10-08 14:52:00] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-08 14:52:00] ...trap/engines/engine.py.326: process: 0.31072 | load: 0.00019 [I 2022-10-08 14:52:03] ...trap/engines/engine.py.321: eval: epoch 8 | batch 170/510 [I 2022-10-08 14:52:03] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-08 14:52:03] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00019 [I 2022-10-08 14:52:06] ...trap/engines/engine.py.321: eval: epoch 8 | batch 180/510 [I 2022-10-08 14:52:06] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-08 14:52:06] ...trap/engines/engine.py.326: process: 0.34163 | load: 0.00020 [I 2022-10-08 14:52:10] ...trap/engines/engine.py.321: eval: epoch 8 | batch 190/510 [I 2022-10-08 14:52:10] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:42 [I 2022-10-08 14:52:10] ...trap/engines/engine.py.326: process: 0.32127 | load: 0.00020 [I 2022-10-08 14:52:13] ...trap/engines/engine.py.321: eval: epoch 8 | batch 200/510 [I 2022-10-08 14:52:13] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:38 [I 2022-10-08 14:52:13] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00021 [I 2022-10-08 14:52:16] ...trap/engines/engine.py.321: eval: epoch 8 | batch 210/510 [I 2022-10-08 14:52:16] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-08 14:52:16] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00020 [I 2022-10-08 14:52:19] ...trap/engines/engine.py.321: eval: epoch 8 | batch 220/510 [I 2022-10-08 14:52:19] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:32 [I 2022-10-08 14:52:19] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00020 [I 2022-10-08 14:52:22] ...trap/engines/engine.py.321: eval: epoch 8 | batch 230/510 [I 2022-10-08 14:52:22] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:28 [I 2022-10-08 14:52:22] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00020 [I 2022-10-08 14:52:25] ...trap/engines/engine.py.321: eval: epoch 8 | batch 240/510 [I 2022-10-08 14:52:25] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-08 14:52:25] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00021 [I 2022-10-08 14:52:29] ...trap/engines/engine.py.321: eval: epoch 8 | batch 250/510 [I 2022-10-08 14:52:29] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:22 [I 2022-10-08 14:52:29] ...trap/engines/engine.py.326: process: 0.31599 | load: 0.00020 [I 2022-10-08 14:52:32] ...trap/engines/engine.py.321: eval: epoch 8 | batch 260/510 [I 2022-10-08 14:52:32] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:19 [I 2022-10-08 14:52:32] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00020 [I 2022-10-08 14:52:35] ...trap/engines/engine.py.321: eval: epoch 8 | batch 270/510 [I 2022-10-08 14:52:35] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:15 [I 2022-10-08 14:52:35] ...trap/engines/engine.py.326: process: 0.31774 | load: 0.00021 [I 2022-10-08 14:52:38] ...trap/engines/engine.py.321: eval: epoch 8 | batch 280/510 [I 2022-10-08 14:52:38] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-08 14:52:38] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00019 [I 2022-10-08 14:52:41] ...trap/engines/engine.py.321: eval: epoch 8 | batch 290/510 [I 2022-10-08 14:52:41] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:09 [I 2022-10-08 14:52:41] ...trap/engines/engine.py.326: process: 0.31698 | load: 0.00020 [I 2022-10-08 14:52:45] ...trap/engines/engine.py.321: eval: epoch 8 | batch 300/510 [I 2022-10-08 14:52:45] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:06 [I 2022-10-08 14:52:45] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00020 [I 2022-10-08 14:52:48] ...trap/engines/engine.py.321: eval: epoch 8 | batch 310/510 [I 2022-10-08 14:52:48] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-08 14:52:48] ...trap/engines/engine.py.326: process: 0.32005 | load: 0.00021 [I 2022-10-08 14:52:51] ...trap/engines/engine.py.321: eval: epoch 8 | batch 320/510 [I 2022-10-08 14:52:51] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-08 14:52:51] ...trap/engines/engine.py.326: process: 0.32161 | load: 0.00020 [I 2022-10-08 14:52:54] ...trap/engines/engine.py.321: eval: epoch 8 | batch 330/510 [I 2022-10-08 14:52:54] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-08 14:52:54] ...trap/engines/engine.py.326: process: 0.32093 | load: 0.00021 [I 2022-10-08 14:52:57] ...trap/engines/engine.py.321: eval: epoch 8 | batch 340/510 [I 2022-10-08 14:52:57] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-08 14:52:57] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00020 [I 2022-10-08 14:53:00] ...trap/engines/engine.py.321: eval: epoch 8 | batch 350/510 [I 2022-10-08 14:53:00] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-08 14:53:00] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00021 [I 2022-10-08 14:53:04] ...trap/engines/engine.py.321: eval: epoch 8 | batch 360/510 [I 2022-10-08 14:53:04] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-08 14:53:04] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00019 [I 2022-10-08 14:53:07] ...trap/engines/engine.py.321: eval: epoch 8 | batch 370/510 [I 2022-10-08 14:53:07] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-08 14:53:07] ...trap/engines/engine.py.326: process: 0.31231 | load: 0.00020 [I 2022-10-08 14:53:10] ...trap/engines/engine.py.321: eval: epoch 8 | batch 380/510 [I 2022-10-08 14:53:10] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-08 14:53:10] ...trap/engines/engine.py.326: process: 0.31636 | load: 0.00020 [I 2022-10-08 14:53:13] ...trap/engines/engine.py.321: eval: epoch 8 | batch 390/510 [I 2022-10-08 14:53:13] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-08 14:53:13] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00020 [I 2022-10-08 14:53:16] ...trap/engines/engine.py.321: eval: epoch 8 | batch 400/510 [I 2022-10-08 14:53:16] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-08 14:53:16] ...trap/engines/engine.py.326: process: 0.31891 | load: 0.00019 [I 2022-10-08 14:53:20] ...trap/engines/engine.py.321: eval: epoch 8 | batch 410/510 [I 2022-10-08 14:53:20] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:31 [I 2022-10-08 14:53:20] ...trap/engines/engine.py.326: process: 0.32254 | load: 0.00020 [I 2022-10-08 14:53:23] ...trap/engines/engine.py.321: eval: epoch 8 | batch 420/510 [I 2022-10-08 14:53:23] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:28 [I 2022-10-08 14:53:23] ...trap/engines/engine.py.326: process: 0.31774 | load: 0.00019 [I 2022-10-08 14:53:26] ...trap/engines/engine.py.321: eval: epoch 8 | batch 430/510 [I 2022-10-08 14:53:26] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-08 14:53:26] ...trap/engines/engine.py.326: process: 0.32256 | load: 0.00021 [I 2022-10-08 14:53:29] ...trap/engines/engine.py.321: eval: epoch 8 | batch 440/510 [I 2022-10-08 14:53:29] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-08 14:53:29] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00019 [I 2022-10-08 14:53:32] ...trap/engines/engine.py.321: eval: epoch 8 | batch 450/510 [I 2022-10-08 14:53:32] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-08 14:53:32] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00020 [I 2022-10-08 14:53:35] ...trap/engines/engine.py.321: eval: epoch 8 | batch 460/510 [I 2022-10-08 14:53:35] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:15 [I 2022-10-08 14:53:35] ...trap/engines/engine.py.326: process: 0.31369 | load: 0.00019 [I 2022-10-08 14:53:39] ...trap/engines/engine.py.321: eval: epoch 8 | batch 470/510 [I 2022-10-08 14:53:39] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:12 [I 2022-10-08 14:53:39] ...trap/engines/engine.py.326: process: 0.31908 | load: 0.00021 [I 2022-10-08 14:53:42] ...trap/engines/engine.py.321: eval: epoch 8 | batch 480/510 [I 2022-10-08 14:53:42] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-08 14:53:42] ...trap/engines/engine.py.326: process: 0.31964 | load: 0.00019 [I 2022-10-08 14:53:45] ...trap/engines/engine.py.321: eval: epoch 8 | batch 490/510 [I 2022-10-08 14:53:45] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-08 14:53:45] ...trap/engines/engine.py.326: process: 0.31392 | load: 0.00021 [I 2022-10-08 14:53:48] ...trap/engines/engine.py.321: eval: epoch 8 | batch 500/510 [I 2022-10-08 14:53:48] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-08 14:53:48] ...trap/engines/engine.py.326: process: 0.31701 | load: 0.00022 [I 2022-10-08 14:53:51] ...trap/engines/engine.py.321: eval: epoch 8 | batch 510/510 [I 2022-10-08 14:53:51] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-08 14:53:51] ...trap/engines/engine.py.326: process: 0.31778 | load: 0.00016 [S 2022-10-08 14:53:52] ...trap/engines/engine.py.132: eval_epoch.epoch: 8 [I 2022-10-08 14:53:52] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.580100 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.842600 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.902300 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.582700 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.842400 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.902600 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012849 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013822 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011019 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013712 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008002 [S 2022-10-08 14:54:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007228 [I 2022-10-08 14:54:25] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 14:54:25] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 14:54:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 14:54:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 14:54:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 14:54:25] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 14:54:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 14:54:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 14:54:33] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 14:54:33] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 14:54:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 14:54:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 14:54:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 9 [I 2022-10-08 14:54:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 14:54:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 0/2304 [I 2022-10-08 14:54:43] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:48 [I 2022-10-08 14:54:43] ...trap/engines/engine.py.229: process: 2.88546 | load: 2.04686 [I 2022-10-08 14:54:43] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-08 14:54:43] ...optimizers/trijoint.py.151: total_norm: 1.799305 [I 2022-10-08 14:54:51] ...trap/engines/engine.py.224: train: epoch 9 | batch 10/2304 [I 2022-10-08 14:54:51] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:24 [I 2022-10-08 14:54:51] ...trap/engines/engine.py.229: process: 0.78217 | load: 0.00523 [I 2022-10-08 14:54:51] ...trap/engines/engine.py.230: loss: 0.24685 [I 2022-10-08 14:54:51] ...optimizers/trijoint.py.151: total_norm: 2.595105 [I 2022-10-08 14:54:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 20/2304 [I 2022-10-08 14:54:59] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:42 [I 2022-10-08 14:54:59] ...trap/engines/engine.py.229: process: 0.77478 | load: 0.00540 [I 2022-10-08 14:54:59] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-08 14:54:59] ...optimizers/trijoint.py.151: total_norm: 2.114555 [I 2022-10-08 14:55:07] ...trap/engines/engine.py.224: train: epoch 9 | batch 30/2304 [I 2022-10-08 14:55:07] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:46 [I 2022-10-08 14:55:07] ...trap/engines/engine.py.229: process: 0.77979 | load: 0.00489 [I 2022-10-08 14:55:07] ...trap/engines/engine.py.230: loss: 0.19962 [I 2022-10-08 14:55:07] ...optimizers/trijoint.py.151: total_norm: 2.829592 [I 2022-10-08 14:55:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 40/2304 [I 2022-10-08 14:55:15] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:29 [I 2022-10-08 14:55:15] ...trap/engines/engine.py.229: process: 0.77040 | load: 0.00405 [I 2022-10-08 14:55:15] ...trap/engines/engine.py.230: loss: 0.19061 [I 2022-10-08 14:55:15] ...optimizers/trijoint.py.151: total_norm: 2.150209 [I 2022-10-08 14:55:22] ...trap/engines/engine.py.224: train: epoch 9 | batch 50/2304 [I 2022-10-08 14:55:22] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:09 [I 2022-10-08 14:55:22] ...trap/engines/engine.py.229: process: 0.77836 | load: 0.00393 [I 2022-10-08 14:55:22] ...trap/engines/engine.py.230: loss: 0.24591 [I 2022-10-08 14:55:22] ...optimizers/trijoint.py.151: total_norm: 3.071720 [I 2022-10-08 14:55:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 60/2304 [I 2022-10-08 14:55:30] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:04 [I 2022-10-08 14:55:30] ...trap/engines/engine.py.229: process: 0.76908 | load: 0.00360 [I 2022-10-08 14:55:30] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-08 14:55:30] ...optimizers/trijoint.py.151: total_norm: 1.816125 [I 2022-10-08 14:55:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 70/2304 [I 2022-10-08 14:55:38] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:58 [I 2022-10-08 14:55:38] ...trap/engines/engine.py.229: process: 0.78512 | load: 0.00360 [I 2022-10-08 14:55:38] ...trap/engines/engine.py.230: loss: 0.20678 [I 2022-10-08 14:55:38] ...optimizers/trijoint.py.151: total_norm: 2.185196 [I 2022-10-08 14:55:46] ...trap/engines/engine.py.224: train: epoch 9 | batch 80/2304 [I 2022-10-08 14:55:46] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:48 [I 2022-10-08 14:55:46] ...trap/engines/engine.py.229: process: 0.77864 | load: 0.00340 [I 2022-10-08 14:55:46] ...trap/engines/engine.py.230: loss: 0.21533 [I 2022-10-08 14:55:46] ...optimizers/trijoint.py.151: total_norm: 2.230797 [I 2022-10-08 14:55:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 90/2304 [I 2022-10-08 14:55:54] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:50 [I 2022-10-08 14:55:54] ...trap/engines/engine.py.229: process: 0.77266 | load: 0.00358 [I 2022-10-08 14:55:54] ...trap/engines/engine.py.230: loss: 0.22053 [I 2022-10-08 14:55:54] ...optimizers/trijoint.py.151: total_norm: 3.235996 [I 2022-10-08 14:56:01] ...trap/engines/engine.py.224: train: epoch 9 | batch 100/2304 [I 2022-10-08 14:56:01] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:31 [I 2022-10-08 14:56:01] ...trap/engines/engine.py.229: process: 0.76844 | load: 0.00366 [I 2022-10-08 14:56:01] ...trap/engines/engine.py.230: loss: 0.23989 [I 2022-10-08 14:56:01] ...optimizers/trijoint.py.151: total_norm: 2.834489 [I 2022-10-08 14:56:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 110/2304 [I 2022-10-08 14:56:09] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:25 [I 2022-10-08 14:56:09] ...trap/engines/engine.py.229: process: 0.78647 | load: 0.00353 [I 2022-10-08 14:56:09] ...trap/engines/engine.py.230: loss: 0.22297 [I 2022-10-08 14:56:09] ...optimizers/trijoint.py.151: total_norm: 2.237194 [I 2022-10-08 14:56:17] ...trap/engines/engine.py.224: train: epoch 9 | batch 120/2304 [I 2022-10-08 14:56:17] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:23 [I 2022-10-08 14:56:17] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00339 [I 2022-10-08 14:56:17] ...trap/engines/engine.py.230: loss: 0.24092 [I 2022-10-08 14:56:17] ...optimizers/trijoint.py.151: total_norm: 2.828887 [I 2022-10-08 14:56:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 130/2304 [I 2022-10-08 14:56:25] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:11 [I 2022-10-08 14:56:25] ...trap/engines/engine.py.229: process: 0.78273 | load: 0.00323 [I 2022-10-08 14:56:25] ...trap/engines/engine.py.230: loss: 0.17916 [I 2022-10-08 14:56:25] ...optimizers/trijoint.py.151: total_norm: 3.242168 [I 2022-10-08 14:56:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 140/2304 [I 2022-10-08 14:56:33] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:12 [I 2022-10-08 14:56:33] ...trap/engines/engine.py.229: process: 0.78603 | load: 0.00350 [I 2022-10-08 14:56:33] ...trap/engines/engine.py.230: loss: 0.17034 [I 2022-10-08 14:56:33] ...optimizers/trijoint.py.151: total_norm: 2.237890 [I 2022-10-08 14:56:41] ...trap/engines/engine.py.224: train: epoch 9 | batch 150/2304 [I 2022-10-08 14:56:41] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:59 [I 2022-10-08 14:56:41] ...trap/engines/engine.py.229: process: 0.77600 | load: 0.00298 [I 2022-10-08 14:56:41] ...trap/engines/engine.py.230: loss: 0.20270 [I 2022-10-08 14:56:41] ...optimizers/trijoint.py.151: total_norm: 2.637935 [I 2022-10-08 14:56:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 160/2304 [I 2022-10-08 14:56:48] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:44 [I 2022-10-08 14:56:48] ...trap/engines/engine.py.229: process: 0.78245 | load: 0.00326 [I 2022-10-08 14:56:48] ...trap/engines/engine.py.230: loss: 0.18071 [I 2022-10-08 14:56:48] ...optimizers/trijoint.py.151: total_norm: 2.318414 [I 2022-10-08 14:56:56] ...trap/engines/engine.py.224: train: epoch 9 | batch 170/2304 [I 2022-10-08 14:56:56] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:40 [I 2022-10-08 14:56:56] ...trap/engines/engine.py.229: process: 0.77545 | load: 0.00325 [I 2022-10-08 14:56:56] ...trap/engines/engine.py.230: loss: 0.22482 [I 2022-10-08 14:56:56] ...optimizers/trijoint.py.151: total_norm: 4.175583 [I 2022-10-08 14:57:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 180/2304 [I 2022-10-08 14:57:04] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:27 [I 2022-10-08 14:57:04] ...trap/engines/engine.py.229: process: 0.77185 | load: 0.00334 [I 2022-10-08 14:57:04] ...trap/engines/engine.py.230: loss: 0.17794 [I 2022-10-08 14:57:04] ...optimizers/trijoint.py.151: total_norm: 2.132696 [I 2022-10-08 14:57:12] ...trap/engines/engine.py.224: train: epoch 9 | batch 190/2304 [I 2022-10-08 14:57:12] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:22 [I 2022-10-08 14:57:12] ...trap/engines/engine.py.229: process: 0.77535 | load: 0.00318 [I 2022-10-08 14:57:12] ...trap/engines/engine.py.230: loss: 0.19392 [I 2022-10-08 14:57:12] ...optimizers/trijoint.py.151: total_norm: 2.641204 [I 2022-10-08 14:57:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 200/2304 [I 2022-10-08 14:57:20] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:06 [I 2022-10-08 14:57:20] ...trap/engines/engine.py.229: process: 0.76637 | load: 0.00327 [I 2022-10-08 14:57:20] ...trap/engines/engine.py.230: loss: 0.22395 [I 2022-10-08 14:57:20] ...optimizers/trijoint.py.151: total_norm: 2.372476 [I 2022-10-08 14:57:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 210/2304 [I 2022-10-08 14:57:27] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:09 [I 2022-10-08 14:57:27] ...trap/engines/engine.py.229: process: 0.77326 | load: 0.00310 [I 2022-10-08 14:57:27] ...trap/engines/engine.py.230: loss: 0.18539 [I 2022-10-08 14:57:27] ...optimizers/trijoint.py.151: total_norm: 2.919405 [I 2022-10-08 14:57:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 220/2304 [I 2022-10-08 14:57:35] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:09 [I 2022-10-08 14:57:35] ...trap/engines/engine.py.229: process: 0.78527 | load: 0.00314 [I 2022-10-08 14:57:35] ...trap/engines/engine.py.230: loss: 0.21202 [I 2022-10-08 14:57:35] ...optimizers/trijoint.py.151: total_norm: 3.258720 [I 2022-10-08 14:57:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 230/2304 [I 2022-10-08 14:57:43] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:27:05 [I 2022-10-08 14:57:43] ...trap/engines/engine.py.229: process: 0.78950 | load: 0.00321 [I 2022-10-08 14:57:43] ...trap/engines/engine.py.230: loss: 0.18828 [I 2022-10-08 14:57:43] ...optimizers/trijoint.py.151: total_norm: 2.512596 [I 2022-10-08 14:57:51] ...trap/engines/engine.py.224: train: epoch 9 | batch 240/2304 [I 2022-10-08 14:57:51] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:46 [I 2022-10-08 14:57:51] ...trap/engines/engine.py.229: process: 0.75901 | load: 0.00339 [I 2022-10-08 14:57:51] ...trap/engines/engine.py.230: loss: 0.26135 [I 2022-10-08 14:57:51] ...optimizers/trijoint.py.151: total_norm: 3.318445 [I 2022-10-08 14:57:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 250/2304 [I 2022-10-08 14:57:59] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:37 [I 2022-10-08 14:57:59] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00316 [I 2022-10-08 14:57:59] ...trap/engines/engine.py.230: loss: 0.29030 [I 2022-10-08 14:57:59] ...optimizers/trijoint.py.151: total_norm: 3.815530 [I 2022-10-08 14:58:06] ...trap/engines/engine.py.224: train: epoch 9 | batch 260/2304 [I 2022-10-08 14:58:06] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:33 [I 2022-10-08 14:58:06] ...trap/engines/engine.py.229: process: 0.77902 | load: 0.00306 [I 2022-10-08 14:58:06] ...trap/engines/engine.py.230: loss: 0.24184 [I 2022-10-08 14:58:06] ...optimizers/trijoint.py.151: total_norm: 3.047747 [I 2022-10-08 14:58:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 270/2304 [I 2022-10-08 14:58:14] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:23 [I 2022-10-08 14:58:14] ...trap/engines/engine.py.229: process: 0.78552 | load: 0.00302 [I 2022-10-08 14:58:14] ...trap/engines/engine.py.230: loss: 0.19991 [I 2022-10-08 14:58:14] ...optimizers/trijoint.py.151: total_norm: 3.141946 [I 2022-10-08 14:58:22] ...trap/engines/engine.py.224: train: epoch 9 | batch 280/2304 [I 2022-10-08 14:58:22] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:12 [I 2022-10-08 14:58:22] ...trap/engines/engine.py.229: process: 0.78233 | load: 0.00311 [I 2022-10-08 14:58:22] ...trap/engines/engine.py.230: loss: 0.23426 [I 2022-10-08 14:58:22] ...optimizers/trijoint.py.151: total_norm: 2.538241 [I 2022-10-08 14:58:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 290/2304 [I 2022-10-08 14:58:30] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:07 [I 2022-10-08 14:58:30] ...trap/engines/engine.py.229: process: 0.77394 | load: 0.00303 [I 2022-10-08 14:58:30] ...trap/engines/engine.py.230: loss: 0.20978 [I 2022-10-08 14:58:30] ...optimizers/trijoint.py.151: total_norm: 2.977032 [I 2022-10-08 14:58:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 300/2304 [I 2022-10-08 14:58:38] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:54 [I 2022-10-08 14:58:38] ...trap/engines/engine.py.229: process: 0.76677 | load: 0.00313 [I 2022-10-08 14:58:38] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-08 14:58:38] ...optimizers/trijoint.py.151: total_norm: 2.942281 [I 2022-10-08 14:58:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 310/2304 [I 2022-10-08 14:58:45] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:25:55 [I 2022-10-08 14:58:45] ...trap/engines/engine.py.229: process: 0.78071 | load: 0.00301 [I 2022-10-08 14:58:45] ...trap/engines/engine.py.230: loss: 0.23009 [I 2022-10-08 14:58:45] ...optimizers/trijoint.py.151: total_norm: 3.349725 [I 2022-10-08 14:58:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 320/2304 [I 2022-10-08 14:58:53] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:57 [I 2022-10-08 14:58:53] ...trap/engines/engine.py.229: process: 0.80402 | load: 0.00301 [I 2022-10-08 14:58:53] ...trap/engines/engine.py.230: loss: 0.20793 [I 2022-10-08 14:58:53] ...optimizers/trijoint.py.151: total_norm: 3.425648 [I 2022-10-08 14:59:01] ...trap/engines/engine.py.224: train: epoch 9 | batch 330/2304 [I 2022-10-08 14:59:01] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:47 [I 2022-10-08 14:59:01] ...trap/engines/engine.py.229: process: 0.79026 | load: 0.00313 [I 2022-10-08 14:59:01] ...trap/engines/engine.py.230: loss: 0.23914 [I 2022-10-08 14:59:01] ...optimizers/trijoint.py.151: total_norm: 3.221064 [I 2022-10-08 14:59:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 340/2304 [I 2022-10-08 14:59:09] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:24 [I 2022-10-08 14:59:09] ...trap/engines/engine.py.229: process: 0.77909 | load: 0.00305 [I 2022-10-08 14:59:09] ...trap/engines/engine.py.230: loss: 0.18074 [I 2022-10-08 14:59:09] ...optimizers/trijoint.py.151: total_norm: 2.766344 [I 2022-10-08 14:59:17] ...trap/engines/engine.py.224: train: epoch 9 | batch 350/2304 [I 2022-10-08 14:59:17] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:13 [I 2022-10-08 14:59:17] ...trap/engines/engine.py.229: process: 0.77830 | load: 0.00304 [I 2022-10-08 14:59:17] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-08 14:59:17] ...optimizers/trijoint.py.151: total_norm: 2.326662 [I 2022-10-08 14:59:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 360/2304 [I 2022-10-08 14:59:24] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:14 [I 2022-10-08 14:59:24] ...trap/engines/engine.py.229: process: 0.77965 | load: 0.00304 [I 2022-10-08 14:59:24] ...trap/engines/engine.py.230: loss: 0.16829 [I 2022-10-08 14:59:24] ...optimizers/trijoint.py.151: total_norm: 2.370402 [I 2022-10-08 14:59:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 370/2304 [I 2022-10-08 14:59:32] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:01 [I 2022-10-08 14:59:32] ...trap/engines/engine.py.229: process: 0.77686 | load: 0.00301 [I 2022-10-08 14:59:32] ...trap/engines/engine.py.230: loss: 0.22513 [I 2022-10-08 14:59:32] ...optimizers/trijoint.py.151: total_norm: 2.568007 [I 2022-10-08 14:59:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 380/2304 [I 2022-10-08 14:59:40] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:05 [I 2022-10-08 14:59:40] ...trap/engines/engine.py.229: process: 0.78531 | load: 0.00302 [I 2022-10-08 14:59:40] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-08 14:59:40] ...optimizers/trijoint.py.151: total_norm: 2.925466 [I 2022-10-08 14:59:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 390/2304 [I 2022-10-08 14:59:48] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:50 [I 2022-10-08 14:59:48] ...trap/engines/engine.py.229: process: 0.78082 | load: 0.00306 [I 2022-10-08 14:59:48] ...trap/engines/engine.py.230: loss: 0.22858 [I 2022-10-08 14:59:48] ...optimizers/trijoint.py.151: total_norm: 2.817956 [I 2022-10-08 14:59:56] ...trap/engines/engine.py.224: train: epoch 9 | batch 400/2304 [I 2022-10-08 14:59:56] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:55 [I 2022-10-08 14:59:56] ...trap/engines/engine.py.229: process: 0.79666 | load: 0.00306 [I 2022-10-08 14:59:56] ...trap/engines/engine.py.230: loss: 0.15980 [I 2022-10-08 14:59:56] ...optimizers/trijoint.py.151: total_norm: 2.329096 [I 2022-10-08 15:00:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 410/2304 [I 2022-10-08 15:00:04] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:31 [I 2022-10-08 15:00:04] ...trap/engines/engine.py.229: process: 0.77137 | load: 0.00305 [I 2022-10-08 15:00:04] ...trap/engines/engine.py.230: loss: 0.21091 [I 2022-10-08 15:00:04] ...optimizers/trijoint.py.151: total_norm: 2.103204 [I 2022-10-08 15:00:11] ...trap/engines/engine.py.224: train: epoch 9 | batch 420/2304 [I 2022-10-08 15:00:11] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:32 [I 2022-10-08 15:00:11] ...trap/engines/engine.py.229: process: 0.77982 | load: 0.00302 [I 2022-10-08 15:00:11] ...trap/engines/engine.py.230: loss: 0.19698 [I 2022-10-08 15:00:11] ...optimizers/trijoint.py.151: total_norm: 3.631357 [I 2022-10-08 15:00:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 430/2304 [I 2022-10-08 15:00:19] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:29 [I 2022-10-08 15:00:19] ...trap/engines/engine.py.229: process: 0.77799 | load: 0.00302 [I 2022-10-08 15:00:19] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-08 15:00:19] ...optimizers/trijoint.py.151: total_norm: 3.032626 [I 2022-10-08 15:00:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 440/2304 [I 2022-10-08 15:00:27] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:09 [I 2022-10-08 15:00:27] ...trap/engines/engine.py.229: process: 0.78751 | load: 0.00307 [I 2022-10-08 15:00:27] ...trap/engines/engine.py.230: loss: 0.17849 [I 2022-10-08 15:00:27] ...optimizers/trijoint.py.151: total_norm: 2.137085 [I 2022-10-08 15:00:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 450/2304 [I 2022-10-08 15:00:35] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:02 [I 2022-10-08 15:00:35] ...trap/engines/engine.py.229: process: 0.77592 | load: 0.00306 [I 2022-10-08 15:00:35] ...trap/engines/engine.py.230: loss: 0.28562 [I 2022-10-08 15:00:35] ...optimizers/trijoint.py.151: total_norm: 3.821214 [I 2022-10-08 15:00:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 460/2304 [I 2022-10-08 15:00:43] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:51 [I 2022-10-08 15:00:43] ...trap/engines/engine.py.229: process: 0.77438 | load: 0.00302 [I 2022-10-08 15:00:43] ...trap/engines/engine.py.230: loss: 0.18256 [I 2022-10-08 15:00:43] ...optimizers/trijoint.py.151: total_norm: 2.909901 [I 2022-10-08 15:00:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 470/2304 [I 2022-10-08 15:00:50] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:52 [I 2022-10-08 15:00:50] ...trap/engines/engine.py.229: process: 0.78998 | load: 0.00302 [I 2022-10-08 15:00:50] ...trap/engines/engine.py.230: loss: 0.21095 [I 2022-10-08 15:00:50] ...optimizers/trijoint.py.151: total_norm: 2.567336 [I 2022-10-08 15:00:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 480/2304 [I 2022-10-08 15:00:58] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:41 [I 2022-10-08 15:00:58] ...trap/engines/engine.py.229: process: 0.77542 | load: 0.00303 [I 2022-10-08 15:00:58] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-08 15:00:58] ...optimizers/trijoint.py.151: total_norm: 2.641877 [I 2022-10-08 15:01:06] ...trap/engines/engine.py.224: train: epoch 9 | batch 490/2304 [I 2022-10-08 15:01:06] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:44 [I 2022-10-08 15:01:06] ...trap/engines/engine.py.229: process: 0.79043 | load: 0.00304 [I 2022-10-08 15:01:06] ...trap/engines/engine.py.230: loss: 0.17930 [I 2022-10-08 15:01:06] ...optimizers/trijoint.py.151: total_norm: 3.001669 [I 2022-10-08 15:01:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 500/2304 [I 2022-10-08 15:01:14] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:27 [I 2022-10-08 15:01:14] ...trap/engines/engine.py.229: process: 0.77344 | load: 0.00306 [I 2022-10-08 15:01:14] ...trap/engines/engine.py.230: loss: 0.18443 [I 2022-10-08 15:01:14] ...optimizers/trijoint.py.151: total_norm: 2.018497 [I 2022-10-08 15:01:22] ...trap/engines/engine.py.224: train: epoch 9 | batch 510/2304 [I 2022-10-08 15:01:22] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:12 [I 2022-10-08 15:01:22] ...trap/engines/engine.py.229: process: 0.76920 | load: 0.00302 [I 2022-10-08 15:01:22] ...trap/engines/engine.py.230: loss: 0.12807 [I 2022-10-08 15:01:22] ...optimizers/trijoint.py.151: total_norm: 1.895169 [I 2022-10-08 15:01:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 520/2304 [I 2022-10-08 15:01:30] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:06 [I 2022-10-08 15:01:30] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00305 [I 2022-10-08 15:01:30] ...trap/engines/engine.py.230: loss: 0.13301 [I 2022-10-08 15:01:30] ...optimizers/trijoint.py.151: total_norm: 2.693792 [I 2022-10-08 15:01:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 530/2304 [I 2022-10-08 15:01:37] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:09 [I 2022-10-08 15:01:37] ...trap/engines/engine.py.229: process: 0.78950 | load: 0.00307 [I 2022-10-08 15:01:37] ...trap/engines/engine.py.230: loss: 0.23003 [I 2022-10-08 15:01:37] ...optimizers/trijoint.py.151: total_norm: 2.889962 [I 2022-10-08 15:01:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 540/2304 [I 2022-10-08 15:01:45] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:23:02 [I 2022-10-08 15:01:45] ...trap/engines/engine.py.229: process: 0.78474 | load: 0.00305 [I 2022-10-08 15:01:45] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-08 15:01:45] ...optimizers/trijoint.py.151: total_norm: 1.981015 [I 2022-10-08 15:01:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 550/2304 [I 2022-10-08 15:01:53] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:52 [I 2022-10-08 15:01:53] ...trap/engines/engine.py.229: process: 0.78160 | load: 0.00301 [I 2022-10-08 15:01:53] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-08 15:01:53] ...optimizers/trijoint.py.151: total_norm: 2.017834 [I 2022-10-08 15:02:01] ...trap/engines/engine.py.224: train: epoch 9 | batch 560/2304 [I 2022-10-08 15:02:01] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:40 [I 2022-10-08 15:02:01] ...trap/engines/engine.py.229: process: 0.77815 | load: 0.00305 [I 2022-10-08 15:02:01] ...trap/engines/engine.py.230: loss: 0.20374 [I 2022-10-08 15:02:01] ...optimizers/trijoint.py.151: total_norm: 2.540689 [I 2022-10-08 15:02:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 570/2304 [I 2022-10-08 15:02:09] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:30 [I 2022-10-08 15:02:09] ...trap/engines/engine.py.229: process: 0.78184 | load: 0.00308 [I 2022-10-08 15:02:09] ...trap/engines/engine.py.230: loss: 0.16993 [I 2022-10-08 15:02:09] ...optimizers/trijoint.py.151: total_norm: 2.319086 [I 2022-10-08 15:02:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 580/2304 [I 2022-10-08 15:02:16] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:21 [I 2022-10-08 15:02:16] ...trap/engines/engine.py.229: process: 0.77397 | load: 0.00310 [I 2022-10-08 15:02:16] ...trap/engines/engine.py.230: loss: 0.17563 [I 2022-10-08 15:02:16] ...optimizers/trijoint.py.151: total_norm: 2.114476 [I 2022-10-08 15:02:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 590/2304 [I 2022-10-08 15:02:24] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:17 [I 2022-10-08 15:02:24] ...trap/engines/engine.py.229: process: 0.78210 | load: 0.00304 [I 2022-10-08 15:02:24] ...trap/engines/engine.py.230: loss: 0.23166 [I 2022-10-08 15:02:24] ...optimizers/trijoint.py.151: total_norm: 3.660760 [I 2022-10-08 15:02:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 600/2304 [I 2022-10-08 15:02:32] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:11 [I 2022-10-08 15:02:32] ...trap/engines/engine.py.229: process: 0.78067 | load: 0.00303 [I 2022-10-08 15:02:32] ...trap/engines/engine.py.230: loss: 0.18888 [I 2022-10-08 15:02:32] ...optimizers/trijoint.py.151: total_norm: 2.514695 [I 2022-10-08 15:02:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 610/2304 [I 2022-10-08 15:02:40] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:21:58 [I 2022-10-08 15:02:40] ...trap/engines/engine.py.229: process: 0.76439 | load: 0.00303 [I 2022-10-08 15:02:40] ...trap/engines/engine.py.230: loss: 0.24203 [I 2022-10-08 15:02:40] ...optimizers/trijoint.py.151: total_norm: 3.508941 [I 2022-10-08 15:02:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 620/2304 [I 2022-10-08 15:02:48] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:56 [I 2022-10-08 15:02:48] ...trap/engines/engine.py.229: process: 0.77992 | load: 0.00309 [I 2022-10-08 15:02:48] ...trap/engines/engine.py.230: loss: 0.29093 [I 2022-10-08 15:02:48] ...optimizers/trijoint.py.151: total_norm: 3.350726 [I 2022-10-08 15:02:56] ...trap/engines/engine.py.224: train: epoch 9 | batch 630/2304 [I 2022-10-08 15:02:56] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:40 [I 2022-10-08 15:02:56] ...trap/engines/engine.py.229: process: 0.76876 | load: 0.00303 [I 2022-10-08 15:02:56] ...trap/engines/engine.py.230: loss: 0.21733 [I 2022-10-08 15:02:56] ...optimizers/trijoint.py.151: total_norm: 2.313151 [I 2022-10-08 15:03:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 640/2304 [I 2022-10-08 15:03:03] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:33 [I 2022-10-08 15:03:03] ...trap/engines/engine.py.229: process: 0.77650 | load: 0.00304 [I 2022-10-08 15:03:03] ...trap/engines/engine.py.230: loss: 0.27748 [I 2022-10-08 15:03:03] ...optimizers/trijoint.py.151: total_norm: 2.948056 [I 2022-10-08 15:03:11] ...trap/engines/engine.py.224: train: epoch 9 | batch 650/2304 [I 2022-10-08 15:03:11] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:33 [I 2022-10-08 15:03:11] ...trap/engines/engine.py.229: process: 0.78876 | load: 0.00302 [I 2022-10-08 15:03:11] ...trap/engines/engine.py.230: loss: 0.22330 [I 2022-10-08 15:03:11] ...optimizers/trijoint.py.151: total_norm: 2.949803 [I 2022-10-08 15:03:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 660/2304 [I 2022-10-08 15:03:19] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:27 [I 2022-10-08 15:03:19] ...trap/engines/engine.py.229: process: 0.78646 | load: 0.00304 [I 2022-10-08 15:03:19] ...trap/engines/engine.py.230: loss: 0.14508 [I 2022-10-08 15:03:19] ...optimizers/trijoint.py.151: total_norm: 3.133352 [I 2022-10-08 15:03:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 670/2304 [I 2022-10-08 15:03:27] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:14 [I 2022-10-08 15:03:27] ...trap/engines/engine.py.229: process: 0.77767 | load: 0.00303 [I 2022-10-08 15:03:27] ...trap/engines/engine.py.230: loss: 0.18627 [I 2022-10-08 15:03:27] ...optimizers/trijoint.py.151: total_norm: 2.868931 [I 2022-10-08 15:03:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 680/2304 [I 2022-10-08 15:03:35] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:03 [I 2022-10-08 15:03:35] ...trap/engines/engine.py.229: process: 0.78309 | load: 0.00303 [I 2022-10-08 15:03:35] ...trap/engines/engine.py.230: loss: 0.16398 [I 2022-10-08 15:03:35] ...optimizers/trijoint.py.151: total_norm: 3.147728 [I 2022-10-08 15:03:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 690/2304 [I 2022-10-08 15:03:42] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:57 [I 2022-10-08 15:03:42] ...trap/engines/engine.py.229: process: 0.76964 | load: 0.00297 [I 2022-10-08 15:03:42] ...trap/engines/engine.py.230: loss: 0.20657 [I 2022-10-08 15:03:42] ...optimizers/trijoint.py.151: total_norm: 2.282962 [I 2022-10-08 15:03:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 700/2304 [I 2022-10-08 15:03:50] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:55 [I 2022-10-08 15:03:50] ...trap/engines/engine.py.229: process: 0.80031 | load: 0.00304 [I 2022-10-08 15:03:50] ...trap/engines/engine.py.230: loss: 0.10453 [I 2022-10-08 15:03:50] ...optimizers/trijoint.py.151: total_norm: 1.578944 [I 2022-10-08 15:03:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 710/2304 [I 2022-10-08 15:03:58] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:43 [I 2022-10-08 15:03:58] ...trap/engines/engine.py.229: process: 0.77989 | load: 0.00295 [I 2022-10-08 15:03:58] ...trap/engines/engine.py.230: loss: 0.25474 [I 2022-10-08 15:03:58] ...optimizers/trijoint.py.151: total_norm: 2.815742 [I 2022-10-08 15:04:06] ...trap/engines/engine.py.224: train: epoch 9 | batch 720/2304 [I 2022-10-08 15:04:06] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:30 [I 2022-10-08 15:04:06] ...trap/engines/engine.py.229: process: 0.78626 | load: 0.00297 [I 2022-10-08 15:04:06] ...trap/engines/engine.py.230: loss: 0.16140 [I 2022-10-08 15:04:06] ...optimizers/trijoint.py.151: total_norm: 1.648737 [I 2022-10-08 15:04:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 730/2304 [I 2022-10-08 15:04:14] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:29 [I 2022-10-08 15:04:14] ...trap/engines/engine.py.229: process: 0.78261 | load: 0.00304 [I 2022-10-08 15:04:14] ...trap/engines/engine.py.230: loss: 0.17319 [I 2022-10-08 15:04:14] ...optimizers/trijoint.py.151: total_norm: 2.457689 [I 2022-10-08 15:04:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 740/2304 [I 2022-10-08 15:04:21] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:12 [I 2022-10-08 15:04:21] ...trap/engines/engine.py.229: process: 0.77656 | load: 0.00306 [I 2022-10-08 15:04:21] ...trap/engines/engine.py.230: loss: 0.21832 [I 2022-10-08 15:04:21] ...optimizers/trijoint.py.151: total_norm: 2.715815 [I 2022-10-08 15:04:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 750/2304 [I 2022-10-08 15:04:29] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:11 [I 2022-10-08 15:04:29] ...trap/engines/engine.py.229: process: 0.77404 | load: 0.00308 [I 2022-10-08 15:04:29] ...trap/engines/engine.py.230: loss: 0.21960 [I 2022-10-08 15:04:29] ...optimizers/trijoint.py.151: total_norm: 2.271356 [I 2022-10-08 15:04:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 760/2304 [I 2022-10-08 15:04:37] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:09 [I 2022-10-08 15:04:37] ...trap/engines/engine.py.229: process: 0.78174 | load: 0.00303 [I 2022-10-08 15:04:37] ...trap/engines/engine.py.230: loss: 0.17028 [I 2022-10-08 15:04:37] ...optimizers/trijoint.py.151: total_norm: 2.288826 [I 2022-10-08 15:04:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 770/2304 [I 2022-10-08 15:04:45] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:57 [I 2022-10-08 15:04:45] ...trap/engines/engine.py.229: process: 0.77846 | load: 0.00309 [I 2022-10-08 15:04:45] ...trap/engines/engine.py.230: loss: 0.28128 [I 2022-10-08 15:04:45] ...optimizers/trijoint.py.151: total_norm: 3.037755 [I 2022-10-08 15:04:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 780/2304 [I 2022-10-08 15:04:53] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:48 [I 2022-10-08 15:04:53] ...trap/engines/engine.py.229: process: 0.77279 | load: 0.00305 [I 2022-10-08 15:04:53] ...trap/engines/engine.py.230: loss: 0.19415 [I 2022-10-08 15:04:53] ...optimizers/trijoint.py.151: total_norm: 2.842259 [I 2022-10-08 15:05:01] ...trap/engines/engine.py.224: train: epoch 9 | batch 790/2304 [I 2022-10-08 15:05:01] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:38 [I 2022-10-08 15:05:01] ...trap/engines/engine.py.229: process: 0.78285 | load: 0.00301 [I 2022-10-08 15:05:01] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-08 15:05:01] ...optimizers/trijoint.py.151: total_norm: 1.627676 [I 2022-10-08 15:05:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 800/2304 [I 2022-10-08 15:05:08] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:31 [I 2022-10-08 15:05:08] ...trap/engines/engine.py.229: process: 0.77510 | load: 0.00306 [I 2022-10-08 15:05:08] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-08 15:05:08] ...optimizers/trijoint.py.151: total_norm: 2.122301 [I 2022-10-08 15:05:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 810/2304 [I 2022-10-08 15:05:16] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:31 [I 2022-10-08 15:05:16] ...trap/engines/engine.py.229: process: 0.78796 | load: 0.00301 [I 2022-10-08 15:05:16] ...trap/engines/engine.py.230: loss: 0.24160 [I 2022-10-08 15:05:16] ...optimizers/trijoint.py.151: total_norm: 2.474008 [I 2022-10-08 15:05:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 820/2304 [I 2022-10-08 15:05:24] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:12 [I 2022-10-08 15:05:24] ...trap/engines/engine.py.229: process: 0.77479 | load: 0.00305 [I 2022-10-08 15:05:24] ...trap/engines/engine.py.230: loss: 0.26451 [I 2022-10-08 15:05:24] ...optimizers/trijoint.py.151: total_norm: 2.768636 [I 2022-10-08 15:05:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 830/2304 [I 2022-10-08 15:05:32] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:12 [I 2022-10-08 15:05:32] ...trap/engines/engine.py.229: process: 0.78637 | load: 0.00305 [I 2022-10-08 15:05:32] ...trap/engines/engine.py.230: loss: 0.17812 [I 2022-10-08 15:05:32] ...optimizers/trijoint.py.151: total_norm: 3.410479 [I 2022-10-08 15:05:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 840/2304 [I 2022-10-08 15:05:40] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:19:03 [I 2022-10-08 15:05:40] ...trap/engines/engine.py.229: process: 0.77552 | load: 0.00304 [I 2022-10-08 15:05:40] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-08 15:05:40] ...optimizers/trijoint.py.151: total_norm: 2.579951 [I 2022-10-08 15:05:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 850/2304 [I 2022-10-08 15:05:48] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:18:53 [I 2022-10-08 15:05:48] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00306 [I 2022-10-08 15:05:48] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-08 15:05:48] ...optimizers/trijoint.py.151: total_norm: 2.411679 [I 2022-10-08 15:05:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 860/2304 [I 2022-10-08 15:05:55] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:45 [I 2022-10-08 15:05:55] ...trap/engines/engine.py.229: process: 0.77387 | load: 0.00304 [I 2022-10-08 15:05:55] ...trap/engines/engine.py.230: loss: 0.19918 [I 2022-10-08 15:05:55] ...optimizers/trijoint.py.151: total_norm: 2.871683 [I 2022-10-08 15:06:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 870/2304 [I 2022-10-08 15:06:03] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:35 [I 2022-10-08 15:06:03] ...trap/engines/engine.py.229: process: 0.77471 | load: 0.00305 [I 2022-10-08 15:06:03] ...trap/engines/engine.py.230: loss: 0.16042 [I 2022-10-08 15:06:03] ...optimizers/trijoint.py.151: total_norm: 3.511994 [I 2022-10-08 15:06:11] ...trap/engines/engine.py.224: train: epoch 9 | batch 880/2304 [I 2022-10-08 15:06:11] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:24 [I 2022-10-08 15:06:11] ...trap/engines/engine.py.229: process: 0.78924 | load: 0.00304 [I 2022-10-08 15:06:11] ...trap/engines/engine.py.230: loss: 0.16125 [I 2022-10-08 15:06:11] ...optimizers/trijoint.py.151: total_norm: 2.648378 [I 2022-10-08 15:06:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 890/2304 [I 2022-10-08 15:06:19] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:25 [I 2022-10-08 15:06:19] ...trap/engines/engine.py.229: process: 0.78788 | load: 0.00303 [I 2022-10-08 15:06:19] ...trap/engines/engine.py.230: loss: 0.21657 [I 2022-10-08 15:06:19] ...optimizers/trijoint.py.151: total_norm: 2.475122 [I 2022-10-08 15:06:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 900/2304 [I 2022-10-08 15:06:27] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:16 [I 2022-10-08 15:06:27] ...trap/engines/engine.py.229: process: 0.78023 | load: 0.00305 [I 2022-10-08 15:06:27] ...trap/engines/engine.py.230: loss: 0.23180 [I 2022-10-08 15:06:27] ...optimizers/trijoint.py.151: total_norm: 2.397934 [I 2022-10-08 15:06:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 910/2304 [I 2022-10-08 15:06:34] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:18:07 [I 2022-10-08 15:06:34] ...trap/engines/engine.py.229: process: 0.78801 | load: 0.00308 [I 2022-10-08 15:06:34] ...trap/engines/engine.py.230: loss: 0.33826 [I 2022-10-08 15:06:34] ...optimizers/trijoint.py.151: total_norm: 3.722901 [I 2022-10-08 15:06:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 920/2304 [I 2022-10-08 15:06:42] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:17:55 [I 2022-10-08 15:06:42] ...trap/engines/engine.py.229: process: 0.77698 | load: 0.00304 [I 2022-10-08 15:06:42] ...trap/engines/engine.py.230: loss: 0.16640 [I 2022-10-08 15:06:42] ...optimizers/trijoint.py.151: total_norm: 2.344251 [I 2022-10-08 15:06:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 930/2304 [I 2022-10-08 15:06:50] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:54 [I 2022-10-08 15:06:50] ...trap/engines/engine.py.229: process: 0.77979 | load: 0.00302 [I 2022-10-08 15:06:50] ...trap/engines/engine.py.230: loss: 0.20058 [I 2022-10-08 15:06:50] ...optimizers/trijoint.py.151: total_norm: 2.867890 [I 2022-10-08 15:06:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 940/2304 [I 2022-10-08 15:06:58] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:43 [I 2022-10-08 15:06:58] ...trap/engines/engine.py.229: process: 0.76694 | load: 0.00305 [I 2022-10-08 15:06:58] ...trap/engines/engine.py.230: loss: 0.24658 [I 2022-10-08 15:06:58] ...optimizers/trijoint.py.151: total_norm: 3.736401 [I 2022-10-08 15:07:06] ...trap/engines/engine.py.224: train: epoch 9 | batch 950/2304 [I 2022-10-08 15:07:06] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:32 [I 2022-10-08 15:07:06] ...trap/engines/engine.py.229: process: 0.77329 | load: 0.00300 [I 2022-10-08 15:07:06] ...trap/engines/engine.py.230: loss: 0.22832 [I 2022-10-08 15:07:06] ...optimizers/trijoint.py.151: total_norm: 2.699712 [I 2022-10-08 15:07:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 960/2304 [I 2022-10-08 15:07:14] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:30 [I 2022-10-08 15:07:14] ...trap/engines/engine.py.229: process: 0.78413 | load: 0.00304 [I 2022-10-08 15:07:14] ...trap/engines/engine.py.230: loss: 0.22756 [I 2022-10-08 15:07:14] ...optimizers/trijoint.py.151: total_norm: 2.981227 [I 2022-10-08 15:07:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 970/2304 [I 2022-10-08 15:07:21] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:18 [I 2022-10-08 15:07:21] ...trap/engines/engine.py.229: process: 0.77776 | load: 0.00301 [I 2022-10-08 15:07:21] ...trap/engines/engine.py.230: loss: 0.19183 [I 2022-10-08 15:07:21] ...optimizers/trijoint.py.151: total_norm: 2.307354 [I 2022-10-08 15:07:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 980/2304 [I 2022-10-08 15:07:29] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:05 [I 2022-10-08 15:07:29] ...trap/engines/engine.py.229: process: 0.76209 | load: 0.00300 [I 2022-10-08 15:07:29] ...trap/engines/engine.py.230: loss: 0.19439 [I 2022-10-08 15:07:29] ...optimizers/trijoint.py.151: total_norm: 2.993856 [I 2022-10-08 15:07:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 990/2304 [I 2022-10-08 15:07:37] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:17:04 [I 2022-10-08 15:07:37] ...trap/engines/engine.py.229: process: 0.78548 | load: 0.00301 [I 2022-10-08 15:07:37] ...trap/engines/engine.py.230: loss: 0.27440 [I 2022-10-08 15:07:37] ...optimizers/trijoint.py.151: total_norm: 3.119265 [I 2022-10-08 15:07:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 1000/2304 [I 2022-10-08 15:07:45] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:16:53 [I 2022-10-08 15:07:45] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00305 [I 2022-10-08 15:07:45] ...trap/engines/engine.py.230: loss: 0.22920 [I 2022-10-08 15:07:45] ...optimizers/trijoint.py.151: total_norm: 2.922390 [I 2022-10-08 15:07:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 1010/2304 [I 2022-10-08 15:07:53] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:49 [I 2022-10-08 15:07:53] ...trap/engines/engine.py.229: process: 0.77760 | load: 0.00303 [I 2022-10-08 15:07:53] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-08 15:07:53] ...optimizers/trijoint.py.151: total_norm: 2.371357 [I 2022-10-08 15:08:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1020/2304 [I 2022-10-08 15:08:00] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:43 [I 2022-10-08 15:08:00] ...trap/engines/engine.py.229: process: 0.78201 | load: 0.00305 [I 2022-10-08 15:08:00] ...trap/engines/engine.py.230: loss: 0.19233 [I 2022-10-08 15:08:00] ...optimizers/trijoint.py.151: total_norm: 2.458608 [I 2022-10-08 15:08:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1030/2304 [I 2022-10-08 15:08:08] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:32 [I 2022-10-08 15:08:08] ...trap/engines/engine.py.229: process: 0.78038 | load: 0.00307 [I 2022-10-08 15:08:08] ...trap/engines/engine.py.230: loss: 0.23971 [I 2022-10-08 15:08:08] ...optimizers/trijoint.py.151: total_norm: 2.909194 [I 2022-10-08 15:08:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 1040/2304 [I 2022-10-08 15:08:16] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:22 [I 2022-10-08 15:08:16] ...trap/engines/engine.py.229: process: 0.76282 | load: 0.00306 [I 2022-10-08 15:08:16] ...trap/engines/engine.py.230: loss: 0.25589 [I 2022-10-08 15:08:16] ...optimizers/trijoint.py.151: total_norm: 2.418926 [I 2022-10-08 15:08:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1050/2304 [I 2022-10-08 15:08:24] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:16:17 [I 2022-10-08 15:08:24] ...trap/engines/engine.py.229: process: 0.78144 | load: 0.00312 [I 2022-10-08 15:08:24] ...trap/engines/engine.py.230: loss: 0.21112 [I 2022-10-08 15:08:24] ...optimizers/trijoint.py.151: total_norm: 2.838275 [I 2022-10-08 15:08:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 1060/2304 [I 2022-10-08 15:08:32] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:15 [I 2022-10-08 15:08:32] ...trap/engines/engine.py.229: process: 0.77698 | load: 0.00307 [I 2022-10-08 15:08:32] ...trap/engines/engine.py.230: loss: 0.19195 [I 2022-10-08 15:08:32] ...optimizers/trijoint.py.151: total_norm: 3.219345 [I 2022-10-08 15:08:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 1070/2304 [I 2022-10-08 15:08:40] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:15:59 [I 2022-10-08 15:08:40] ...trap/engines/engine.py.229: process: 0.76955 | load: 0.00301 [I 2022-10-08 15:08:40] ...trap/engines/engine.py.230: loss: 0.17640 [I 2022-10-08 15:08:40] ...optimizers/trijoint.py.151: total_norm: 2.895778 [I 2022-10-08 15:08:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 1080/2304 [I 2022-10-08 15:08:47] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:57 [I 2022-10-08 15:08:47] ...trap/engines/engine.py.229: process: 0.77189 | load: 0.00305 [I 2022-10-08 15:08:47] ...trap/engines/engine.py.230: loss: 0.25565 [I 2022-10-08 15:08:47] ...optimizers/trijoint.py.151: total_norm: 2.840920 [I 2022-10-08 15:08:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1090/2304 [I 2022-10-08 15:08:55] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:49 [I 2022-10-08 15:08:55] ...trap/engines/engine.py.229: process: 0.78209 | load: 0.00309 [I 2022-10-08 15:08:55] ...trap/engines/engine.py.230: loss: 0.20884 [I 2022-10-08 15:08:55] ...optimizers/trijoint.py.151: total_norm: 2.156625 [I 2022-10-08 15:09:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1100/2304 [I 2022-10-08 15:09:03] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:36 [I 2022-10-08 15:09:03] ...trap/engines/engine.py.229: process: 0.78181 | load: 0.00302 [I 2022-10-08 15:09:03] ...trap/engines/engine.py.230: loss: 0.17028 [I 2022-10-08 15:09:03] ...optimizers/trijoint.py.151: total_norm: 2.455951 [I 2022-10-08 15:09:11] ...trap/engines/engine.py.224: train: epoch 9 | batch 1110/2304 [I 2022-10-08 15:09:11] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:15:34 [I 2022-10-08 15:09:11] ...trap/engines/engine.py.229: process: 0.78522 | load: 0.00304 [I 2022-10-08 15:09:11] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-08 15:09:11] ...optimizers/trijoint.py.151: total_norm: 2.333290 [I 2022-10-08 15:09:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 1120/2304 [I 2022-10-08 15:09:19] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:17 [I 2022-10-08 15:09:19] ...trap/engines/engine.py.229: process: 0.76639 | load: 0.00302 [I 2022-10-08 15:09:19] ...trap/engines/engine.py.230: loss: 0.11710 [I 2022-10-08 15:09:19] ...optimizers/trijoint.py.151: total_norm: 1.966317 [I 2022-10-08 15:09:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 1130/2304 [I 2022-10-08 15:09:26] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:08 [I 2022-10-08 15:09:26] ...trap/engines/engine.py.229: process: 0.76948 | load: 0.00301 [I 2022-10-08 15:09:26] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-08 15:09:26] ...optimizers/trijoint.py.151: total_norm: 1.960150 [I 2022-10-08 15:09:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1140/2304 [I 2022-10-08 15:09:34] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:04 [I 2022-10-08 15:09:34] ...trap/engines/engine.py.229: process: 0.77516 | load: 0.00304 [I 2022-10-08 15:09:34] ...trap/engines/engine.py.230: loss: 0.20120 [I 2022-10-08 15:09:34] ...optimizers/trijoint.py.151: total_norm: 2.186779 [I 2022-10-08 15:09:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 1150/2304 [I 2022-10-08 15:09:42] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:14:52 [I 2022-10-08 15:09:42] ...trap/engines/engine.py.229: process: 0.77163 | load: 0.00305 [I 2022-10-08 15:09:42] ...trap/engines/engine.py.230: loss: 0.23306 [I 2022-10-08 15:09:42] ...optimizers/trijoint.py.151: total_norm: 2.970211 [I 2022-10-08 15:09:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 1160/2304 [I 2022-10-08 15:09:50] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:14:52 [I 2022-10-08 15:09:50] ...trap/engines/engine.py.229: process: 0.78040 | load: 0.00303 [I 2022-10-08 15:09:50] ...trap/engines/engine.py.230: loss: 0.22091 [I 2022-10-08 15:09:50] ...optimizers/trijoint.py.151: total_norm: 2.489997 [I 2022-10-08 15:09:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 1170/2304 [I 2022-10-08 15:09:58] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:45 [I 2022-10-08 15:09:58] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00302 [I 2022-10-08 15:09:58] ...trap/engines/engine.py.230: loss: 0.31631 [I 2022-10-08 15:09:58] ...optimizers/trijoint.py.151: total_norm: 3.739649 [I 2022-10-08 15:10:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1180/2304 [I 2022-10-08 15:10:05] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:14:34 [I 2022-10-08 15:10:05] ...trap/engines/engine.py.229: process: 0.77080 | load: 0.00306 [I 2022-10-08 15:10:05] ...trap/engines/engine.py.230: loss: 0.19043 [I 2022-10-08 15:10:05] ...optimizers/trijoint.py.151: total_norm: 2.168195 [I 2022-10-08 15:10:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1190/2304 [I 2022-10-08 15:10:13] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:14:28 [I 2022-10-08 15:10:13] ...trap/engines/engine.py.229: process: 0.78189 | load: 0.00303 [I 2022-10-08 15:10:13] ...trap/engines/engine.py.230: loss: 0.25431 [I 2022-10-08 15:10:13] ...optimizers/trijoint.py.151: total_norm: 3.308000 [I 2022-10-08 15:10:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 1200/2304 [I 2022-10-08 15:10:21] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:18 [I 2022-10-08 15:10:21] ...trap/engines/engine.py.229: process: 0.78225 | load: 0.00302 [I 2022-10-08 15:10:21] ...trap/engines/engine.py.230: loss: 0.15428 [I 2022-10-08 15:10:21] ...optimizers/trijoint.py.151: total_norm: 2.577794 [I 2022-10-08 15:10:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1210/2304 [I 2022-10-08 15:10:29] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:14:14 [I 2022-10-08 15:10:29] ...trap/engines/engine.py.229: process: 0.79149 | load: 0.00307 [I 2022-10-08 15:10:29] ...trap/engines/engine.py.230: loss: 0.13357 [I 2022-10-08 15:10:29] ...optimizers/trijoint.py.151: total_norm: 2.361036 [I 2022-10-08 15:10:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 1220/2304 [I 2022-10-08 15:10:37] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:14:00 [I 2022-10-08 15:10:37] ...trap/engines/engine.py.229: process: 0.76654 | load: 0.00301 [I 2022-10-08 15:10:37] ...trap/engines/engine.py.230: loss: 0.21365 [I 2022-10-08 15:10:37] ...optimizers/trijoint.py.151: total_norm: 3.712508 [I 2022-10-08 15:10:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1230/2304 [I 2022-10-08 15:10:44] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:13:55 [I 2022-10-08 15:10:44] ...trap/engines/engine.py.229: process: 0.77363 | load: 0.00302 [I 2022-10-08 15:10:44] ...trap/engines/engine.py.230: loss: 0.24375 [I 2022-10-08 15:10:44] ...optimizers/trijoint.py.151: total_norm: 3.557768 [I 2022-10-08 15:10:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 1240/2304 [I 2022-10-08 15:10:52] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:13:48 [I 2022-10-08 15:10:52] ...trap/engines/engine.py.229: process: 0.78554 | load: 0.00305 [I 2022-10-08 15:10:52] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-08 15:10:52] ...optimizers/trijoint.py.151: total_norm: 2.279419 [I 2022-10-08 15:11:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1250/2304 [I 2022-10-08 15:11:00] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:13:40 [I 2022-10-08 15:11:00] ...trap/engines/engine.py.229: process: 0.78005 | load: 0.00304 [I 2022-10-08 15:11:00] ...trap/engines/engine.py.230: loss: 0.18516 [I 2022-10-08 15:11:00] ...optimizers/trijoint.py.151: total_norm: 2.555158 [I 2022-10-08 15:11:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1260/2304 [I 2022-10-08 15:11:08] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:35 [I 2022-10-08 15:11:08] ...trap/engines/engine.py.229: process: 0.78350 | load: 0.00303 [I 2022-10-08 15:11:08] ...trap/engines/engine.py.230: loss: 0.19427 [I 2022-10-08 15:11:08] ...optimizers/trijoint.py.151: total_norm: 3.252049 [I 2022-10-08 15:11:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 1270/2304 [I 2022-10-08 15:11:16] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:13:26 [I 2022-10-08 15:11:16] ...trap/engines/engine.py.229: process: 0.76774 | load: 0.00304 [I 2022-10-08 15:11:16] ...trap/engines/engine.py.230: loss: 0.21019 [I 2022-10-08 15:11:16] ...optimizers/trijoint.py.151: total_norm: 3.566359 [I 2022-10-08 15:11:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 1280/2304 [I 2022-10-08 15:11:23] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:13:15 [I 2022-10-08 15:11:23] ...trap/engines/engine.py.229: process: 0.76850 | load: 0.00303 [I 2022-10-08 15:11:23] ...trap/engines/engine.py.230: loss: 0.24228 [I 2022-10-08 15:11:23] ...optimizers/trijoint.py.151: total_norm: 3.085371 [I 2022-10-08 15:11:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 1290/2304 [I 2022-10-08 15:11:31] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:13:15 [I 2022-10-08 15:11:31] ...trap/engines/engine.py.229: process: 0.78952 | load: 0.00307 [I 2022-10-08 15:11:31] ...trap/engines/engine.py.230: loss: 0.21844 [I 2022-10-08 15:11:31] ...optimizers/trijoint.py.151: total_norm: 3.470724 [I 2022-10-08 15:11:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1300/2304 [I 2022-10-08 15:11:39] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:13:03 [I 2022-10-08 15:11:39] ...trap/engines/engine.py.229: process: 0.78770 | load: 0.00303 [I 2022-10-08 15:11:39] ...trap/engines/engine.py.230: loss: 0.16316 [I 2022-10-08 15:11:39] ...optimizers/trijoint.py.151: total_norm: 2.737857 [I 2022-10-08 15:11:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 1310/2304 [I 2022-10-08 15:11:47] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:13:00 [I 2022-10-08 15:11:47] ...trap/engines/engine.py.229: process: 0.79720 | load: 0.00310 [I 2022-10-08 15:11:47] ...trap/engines/engine.py.230: loss: 0.14599 [I 2022-10-08 15:11:47] ...optimizers/trijoint.py.151: total_norm: 1.939004 [I 2022-10-08 15:11:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1320/2304 [I 2022-10-08 15:11:55] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:12:44 [I 2022-10-08 15:11:55] ...trap/engines/engine.py.229: process: 0.76630 | load: 0.00303 [I 2022-10-08 15:11:55] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-08 15:11:55] ...optimizers/trijoint.py.151: total_norm: 2.984397 [I 2022-10-08 15:12:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1330/2304 [I 2022-10-08 15:12:03] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:37 [I 2022-10-08 15:12:03] ...trap/engines/engine.py.229: process: 0.77559 | load: 0.00300 [I 2022-10-08 15:12:03] ...trap/engines/engine.py.230: loss: 0.20496 [I 2022-10-08 15:12:03] ...optimizers/trijoint.py.151: total_norm: 2.582052 [I 2022-10-08 15:12:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 1340/2304 [I 2022-10-08 15:12:10] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:12:31 [I 2022-10-08 15:12:10] ...trap/engines/engine.py.229: process: 0.77357 | load: 0.00302 [I 2022-10-08 15:12:10] ...trap/engines/engine.py.230: loss: 0.20280 [I 2022-10-08 15:12:10] ...optimizers/trijoint.py.151: total_norm: 2.790926 [I 2022-10-08 15:12:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1350/2304 [I 2022-10-08 15:12:18] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:12:27 [I 2022-10-08 15:12:18] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00304 [I 2022-10-08 15:12:18] ...trap/engines/engine.py.230: loss: 0.25666 [I 2022-10-08 15:12:18] ...optimizers/trijoint.py.151: total_norm: 3.300694 [I 2022-10-08 15:12:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 1360/2304 [I 2022-10-08 15:12:26] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:16 [I 2022-10-08 15:12:26] ...trap/engines/engine.py.229: process: 0.78163 | load: 0.00302 [I 2022-10-08 15:12:26] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-08 15:12:26] ...optimizers/trijoint.py.151: total_norm: 2.611025 [I 2022-10-08 15:12:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1370/2304 [I 2022-10-08 15:12:34] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:09 [I 2022-10-08 15:12:34] ...trap/engines/engine.py.229: process: 0.77783 | load: 0.00304 [I 2022-10-08 15:12:34] ...trap/engines/engine.py.230: loss: 0.17570 [I 2022-10-08 15:12:34] ...optimizers/trijoint.py.151: total_norm: 2.368874 [I 2022-10-08 15:12:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 1380/2304 [I 2022-10-08 15:12:42] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:04 [I 2022-10-08 15:12:42] ...trap/engines/engine.py.229: process: 0.79126 | load: 0.00303 [I 2022-10-08 15:12:42] ...trap/engines/engine.py.230: loss: 0.20274 [I 2022-10-08 15:12:42] ...optimizers/trijoint.py.151: total_norm: 2.472147 [I 2022-10-08 15:12:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 1390/2304 [I 2022-10-08 15:12:50] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:11:56 [I 2022-10-08 15:12:50] ...trap/engines/engine.py.229: process: 0.79397 | load: 0.00305 [I 2022-10-08 15:12:50] ...trap/engines/engine.py.230: loss: 0.18662 [I 2022-10-08 15:12:50] ...optimizers/trijoint.py.151: total_norm: 2.547078 [I 2022-10-08 15:12:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 1400/2304 [I 2022-10-08 15:12:57] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:11:48 [I 2022-10-08 15:12:57] ...trap/engines/engine.py.229: process: 0.78160 | load: 0.00300 [I 2022-10-08 15:12:57] ...trap/engines/engine.py.230: loss: 0.19989 [I 2022-10-08 15:12:57] ...optimizers/trijoint.py.151: total_norm: 3.932728 [I 2022-10-08 15:13:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1410/2304 [I 2022-10-08 15:13:05] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:11:39 [I 2022-10-08 15:13:05] ...trap/engines/engine.py.229: process: 0.78677 | load: 0.00301 [I 2022-10-08 15:13:05] ...trap/engines/engine.py.230: loss: 0.18642 [I 2022-10-08 15:13:05] ...optimizers/trijoint.py.151: total_norm: 2.523684 [I 2022-10-08 15:13:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1420/2304 [I 2022-10-08 15:13:13] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:29 [I 2022-10-08 15:13:13] ...trap/engines/engine.py.229: process: 0.76909 | load: 0.00303 [I 2022-10-08 15:13:13] ...trap/engines/engine.py.230: loss: 0.20961 [I 2022-10-08 15:13:13] ...optimizers/trijoint.py.151: total_norm: 2.713982 [I 2022-10-08 15:13:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 1430/2304 [I 2022-10-08 15:13:21] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:21 [I 2022-10-08 15:13:21] ...trap/engines/engine.py.229: process: 0.79402 | load: 0.00306 [I 2022-10-08 15:13:21] ...trap/engines/engine.py.230: loss: 0.20860 [I 2022-10-08 15:13:21] ...optimizers/trijoint.py.151: total_norm: 2.318863 [I 2022-10-08 15:13:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1440/2304 [I 2022-10-08 15:13:29] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:11 [I 2022-10-08 15:13:29] ...trap/engines/engine.py.229: process: 0.77735 | load: 0.00305 [I 2022-10-08 15:13:29] ...trap/engines/engine.py.230: loss: 0.22450 [I 2022-10-08 15:13:29] ...optimizers/trijoint.py.151: total_norm: 3.180968 [I 2022-10-08 15:13:36] ...trap/engines/engine.py.224: train: epoch 9 | batch 1450/2304 [I 2022-10-08 15:13:36] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:04 [I 2022-10-08 15:13:36] ...trap/engines/engine.py.229: process: 0.77238 | load: 0.00309 [I 2022-10-08 15:13:36] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-08 15:13:36] ...optimizers/trijoint.py.151: total_norm: 2.019496 [I 2022-10-08 15:13:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1460/2304 [I 2022-10-08 15:13:44] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:11:00 [I 2022-10-08 15:13:44] ...trap/engines/engine.py.229: process: 0.77186 | load: 0.00310 [I 2022-10-08 15:13:44] ...trap/engines/engine.py.230: loss: 0.15698 [I 2022-10-08 15:13:44] ...optimizers/trijoint.py.151: total_norm: 1.985200 [I 2022-10-08 15:13:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 1470/2304 [I 2022-10-08 15:13:52] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:10:51 [I 2022-10-08 15:13:52] ...trap/engines/engine.py.229: process: 0.78534 | load: 0.00302 [I 2022-10-08 15:13:52] ...trap/engines/engine.py.230: loss: 0.14415 [I 2022-10-08 15:13:52] ...optimizers/trijoint.py.151: total_norm: 1.623459 [I 2022-10-08 15:14:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1480/2304 [I 2022-10-08 15:14:00] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:47 [I 2022-10-08 15:14:00] ...trap/engines/engine.py.229: process: 0.77394 | load: 0.00306 [I 2022-10-08 15:14:00] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-08 15:14:00] ...optimizers/trijoint.py.151: total_norm: 2.008052 [I 2022-10-08 15:14:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1490/2304 [I 2022-10-08 15:14:08] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:40 [I 2022-10-08 15:14:08] ...trap/engines/engine.py.229: process: 0.78065 | load: 0.00303 [I 2022-10-08 15:14:08] ...trap/engines/engine.py.230: loss: 0.23908 [I 2022-10-08 15:14:08] ...optimizers/trijoint.py.151: total_norm: 3.696081 [I 2022-10-08 15:14:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 1500/2304 [I 2022-10-08 15:14:16] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:28 [I 2022-10-08 15:14:16] ...trap/engines/engine.py.229: process: 0.77609 | load: 0.00306 [I 2022-10-08 15:14:16] ...trap/engines/engine.py.230: loss: 0.17896 [I 2022-10-08 15:14:16] ...optimizers/trijoint.py.151: total_norm: 2.524321 [I 2022-10-08 15:14:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1510/2304 [I 2022-10-08 15:14:24] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:22 [I 2022-10-08 15:14:24] ...trap/engines/engine.py.229: process: 0.79547 | load: 0.00302 [I 2022-10-08 15:14:24] ...trap/engines/engine.py.230: loss: 0.18471 [I 2022-10-08 15:14:24] ...optimizers/trijoint.py.151: total_norm: 2.531057 [I 2022-10-08 15:14:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 1520/2304 [I 2022-10-08 15:14:31] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:14 [I 2022-10-08 15:14:31] ...trap/engines/engine.py.229: process: 0.78854 | load: 0.00307 [I 2022-10-08 15:14:31] ...trap/engines/engine.py.230: loss: 0.23940 [I 2022-10-08 15:14:31] ...optimizers/trijoint.py.151: total_norm: 2.804972 [I 2022-10-08 15:14:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1530/2304 [I 2022-10-08 15:14:39] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:03 [I 2022-10-08 15:14:39] ...trap/engines/engine.py.229: process: 0.77426 | load: 0.00305 [I 2022-10-08 15:14:39] ...trap/engines/engine.py.230: loss: 0.20028 [I 2022-10-08 15:14:39] ...optimizers/trijoint.py.151: total_norm: 2.066816 [I 2022-10-08 15:14:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 1540/2304 [I 2022-10-08 15:14:47] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:56 [I 2022-10-08 15:14:47] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00301 [I 2022-10-08 15:14:47] ...trap/engines/engine.py.230: loss: 0.21259 [I 2022-10-08 15:14:47] ...optimizers/trijoint.py.151: total_norm: 2.632126 [I 2022-10-08 15:14:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1550/2304 [I 2022-10-08 15:14:55] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:47 [I 2022-10-08 15:14:55] ...trap/engines/engine.py.229: process: 0.77914 | load: 0.00307 [I 2022-10-08 15:14:55] ...trap/engines/engine.py.230: loss: 0.15116 [I 2022-10-08 15:14:55] ...optimizers/trijoint.py.151: total_norm: 2.368581 [I 2022-10-08 15:15:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1560/2304 [I 2022-10-08 15:15:03] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:42 [I 2022-10-08 15:15:03] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00308 [I 2022-10-08 15:15:03] ...trap/engines/engine.py.230: loss: 0.20948 [I 2022-10-08 15:15:03] ...optimizers/trijoint.py.151: total_norm: 2.659268 [I 2022-10-08 15:15:11] ...trap/engines/engine.py.224: train: epoch 9 | batch 1570/2304 [I 2022-10-08 15:15:11] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:09:32 [I 2022-10-08 15:15:11] ...trap/engines/engine.py.229: process: 0.79127 | load: 0.00305 [I 2022-10-08 15:15:11] ...trap/engines/engine.py.230: loss: 0.25989 [I 2022-10-08 15:15:11] ...optimizers/trijoint.py.151: total_norm: 3.525875 [I 2022-10-08 15:15:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1580/2304 [I 2022-10-08 15:15:18] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:23 [I 2022-10-08 15:15:18] ...trap/engines/engine.py.229: process: 0.78433 | load: 0.00302 [I 2022-10-08 15:15:18] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-08 15:15:18] ...optimizers/trijoint.py.151: total_norm: 2.419270 [I 2022-10-08 15:15:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 1590/2304 [I 2022-10-08 15:15:26] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:14 [I 2022-10-08 15:15:26] ...trap/engines/engine.py.229: process: 0.78306 | load: 0.00300 [I 2022-10-08 15:15:26] ...trap/engines/engine.py.230: loss: 0.19095 [I 2022-10-08 15:15:26] ...optimizers/trijoint.py.151: total_norm: 2.705046 [I 2022-10-08 15:15:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1600/2304 [I 2022-10-08 15:15:34] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:10 [I 2022-10-08 15:15:34] ...trap/engines/engine.py.229: process: 0.77984 | load: 0.00307 [I 2022-10-08 15:15:34] ...trap/engines/engine.py.230: loss: 0.23150 [I 2022-10-08 15:15:34] ...optimizers/trijoint.py.151: total_norm: 2.274441 [I 2022-10-08 15:15:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 1610/2304 [I 2022-10-08 15:15:42] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:08:58 [I 2022-10-08 15:15:42] ...trap/engines/engine.py.229: process: 0.76377 | load: 0.00303 [I 2022-10-08 15:15:42] ...trap/engines/engine.py.230: loss: 0.16936 [I 2022-10-08 15:15:42] ...optimizers/trijoint.py.151: total_norm: 2.159997 [I 2022-10-08 15:15:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 1620/2304 [I 2022-10-08 15:15:50] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:08:53 [I 2022-10-08 15:15:50] ...trap/engines/engine.py.229: process: 0.78455 | load: 0.00302 [I 2022-10-08 15:15:50] ...trap/engines/engine.py.230: loss: 0.30182 [I 2022-10-08 15:15:50] ...optimizers/trijoint.py.151: total_norm: 3.680284 [I 2022-10-08 15:15:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 1630/2304 [I 2022-10-08 15:15:57] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:08:46 [I 2022-10-08 15:15:57] ...trap/engines/engine.py.229: process: 0.78605 | load: 0.00299 [I 2022-10-08 15:15:57] ...trap/engines/engine.py.230: loss: 0.20689 [I 2022-10-08 15:15:57] ...optimizers/trijoint.py.151: total_norm: 2.820333 [I 2022-10-08 15:16:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1640/2304 [I 2022-10-08 15:16:05] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:38 [I 2022-10-08 15:16:05] ...trap/engines/engine.py.229: process: 0.78257 | load: 0.00301 [I 2022-10-08 15:16:05] ...trap/engines/engine.py.230: loss: 0.22740 [I 2022-10-08 15:16:05] ...optimizers/trijoint.py.151: total_norm: 3.355625 [I 2022-10-08 15:16:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1650/2304 [I 2022-10-08 15:16:13] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:31 [I 2022-10-08 15:16:13] ...trap/engines/engine.py.229: process: 0.78978 | load: 0.00304 [I 2022-10-08 15:16:13] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-08 15:16:13] ...optimizers/trijoint.py.151: total_norm: 2.489342 [I 2022-10-08 15:16:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 1660/2304 [I 2022-10-08 15:16:21] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:24 [I 2022-10-08 15:16:21] ...trap/engines/engine.py.229: process: 0.78303 | load: 0.00302 [I 2022-10-08 15:16:21] ...trap/engines/engine.py.230: loss: 0.23577 [I 2022-10-08 15:16:21] ...optimizers/trijoint.py.151: total_norm: 2.154558 [I 2022-10-08 15:16:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1670/2304 [I 2022-10-08 15:16:29] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:15 [I 2022-10-08 15:16:29] ...trap/engines/engine.py.229: process: 0.77426 | load: 0.00299 [I 2022-10-08 15:16:29] ...trap/engines/engine.py.230: loss: 0.25939 [I 2022-10-08 15:16:29] ...optimizers/trijoint.py.151: total_norm: 3.255946 [I 2022-10-08 15:16:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 1680/2304 [I 2022-10-08 15:16:37] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:08:06 [I 2022-10-08 15:16:37] ...trap/engines/engine.py.229: process: 0.78463 | load: 0.00305 [I 2022-10-08 15:16:37] ...trap/engines/engine.py.230: loss: 0.23176 [I 2022-10-08 15:16:37] ...optimizers/trijoint.py.151: total_norm: 3.017022 [I 2022-10-08 15:16:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1690/2304 [I 2022-10-08 15:16:44] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:07:58 [I 2022-10-08 15:16:44] ...trap/engines/engine.py.229: process: 0.77532 | load: 0.00308 [I 2022-10-08 15:16:44] ...trap/engines/engine.py.230: loss: 0.22249 [I 2022-10-08 15:16:44] ...optimizers/trijoint.py.151: total_norm: 2.872520 [I 2022-10-08 15:16:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 1700/2304 [I 2022-10-08 15:16:52] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:49 [I 2022-10-08 15:16:52] ...trap/engines/engine.py.229: process: 0.76752 | load: 0.00301 [I 2022-10-08 15:16:52] ...trap/engines/engine.py.230: loss: 0.14157 [I 2022-10-08 15:16:52] ...optimizers/trijoint.py.151: total_norm: 1.864806 [I 2022-10-08 15:17:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1710/2304 [I 2022-10-08 15:17:00] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:43 [I 2022-10-08 15:17:00] ...trap/engines/engine.py.229: process: 0.78014 | load: 0.00303 [I 2022-10-08 15:17:00] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-08 15:17:00] ...optimizers/trijoint.py.151: total_norm: 4.101420 [I 2022-10-08 15:17:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1720/2304 [I 2022-10-08 15:17:08] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:07:36 [I 2022-10-08 15:17:08] ...trap/engines/engine.py.229: process: 0.77335 | load: 0.00305 [I 2022-10-08 15:17:08] ...trap/engines/engine.py.230: loss: 0.18945 [I 2022-10-08 15:17:08] ...optimizers/trijoint.py.151: total_norm: 3.004532 [I 2022-10-08 15:17:16] ...trap/engines/engine.py.224: train: epoch 9 | batch 1730/2304 [I 2022-10-08 15:17:16] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:07:29 [I 2022-10-08 15:17:16] ...trap/engines/engine.py.229: process: 0.78513 | load: 0.00303 [I 2022-10-08 15:17:16] ...trap/engines/engine.py.230: loss: 0.19489 [I 2022-10-08 15:17:16] ...optimizers/trijoint.py.151: total_norm: 2.887428 [I 2022-10-08 15:17:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1740/2304 [I 2022-10-08 15:17:24] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:07:19 [I 2022-10-08 15:17:24] ...trap/engines/engine.py.229: process: 0.78397 | load: 0.00299 [I 2022-10-08 15:17:24] ...trap/engines/engine.py.230: loss: 0.18377 [I 2022-10-08 15:17:24] ...optimizers/trijoint.py.151: total_norm: 2.741826 [I 2022-10-08 15:17:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 1750/2304 [I 2022-10-08 15:17:31] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:14 [I 2022-10-08 15:17:31] ...trap/engines/engine.py.229: process: 0.78786 | load: 0.00308 [I 2022-10-08 15:17:31] ...trap/engines/engine.py.230: loss: 0.30313 [I 2022-10-08 15:17:31] ...optimizers/trijoint.py.151: total_norm: 3.323256 [I 2022-10-08 15:17:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1760/2304 [I 2022-10-08 15:17:39] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:05 [I 2022-10-08 15:17:39] ...trap/engines/engine.py.229: process: 0.78401 | load: 0.00314 [I 2022-10-08 15:17:39] ...trap/engines/engine.py.230: loss: 0.19219 [I 2022-10-08 15:17:39] ...optimizers/trijoint.py.151: total_norm: 2.815406 [I 2022-10-08 15:17:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 1770/2304 [I 2022-10-08 15:17:47] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:06:54 [I 2022-10-08 15:17:47] ...trap/engines/engine.py.229: process: 0.78285 | load: 0.00307 [I 2022-10-08 15:17:47] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-08 15:17:47] ...optimizers/trijoint.py.151: total_norm: 2.174439 [I 2022-10-08 15:17:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1780/2304 [I 2022-10-08 15:17:55] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:06:47 [I 2022-10-08 15:17:55] ...trap/engines/engine.py.229: process: 0.78212 | load: 0.00305 [I 2022-10-08 15:17:55] ...trap/engines/engine.py.230: loss: 0.17535 [I 2022-10-08 15:17:55] ...optimizers/trijoint.py.151: total_norm: 2.181957 [I 2022-10-08 15:18:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1790/2304 [I 2022-10-08 15:18:03] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:06:42 [I 2022-10-08 15:18:03] ...trap/engines/engine.py.229: process: 0.78179 | load: 0.00300 [I 2022-10-08 15:18:03] ...trap/engines/engine.py.230: loss: 0.20574 [I 2022-10-08 15:18:03] ...optimizers/trijoint.py.151: total_norm: 4.069298 [I 2022-10-08 15:18:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 1800/2304 [I 2022-10-08 15:18:10] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:06:32 [I 2022-10-08 15:18:10] ...trap/engines/engine.py.229: process: 0.77678 | load: 0.00302 [I 2022-10-08 15:18:10] ...trap/engines/engine.py.230: loss: 0.20700 [I 2022-10-08 15:18:10] ...optimizers/trijoint.py.151: total_norm: 2.925607 [I 2022-10-08 15:18:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1810/2304 [I 2022-10-08 15:18:18] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:24 [I 2022-10-08 15:18:18] ...trap/engines/engine.py.229: process: 0.77988 | load: 0.00300 [I 2022-10-08 15:18:18] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-08 15:18:18] ...optimizers/trijoint.py.151: total_norm: 3.006186 [I 2022-10-08 15:18:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 1820/2304 [I 2022-10-08 15:18:26] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:06:17 [I 2022-10-08 15:18:26] ...trap/engines/engine.py.229: process: 0.77175 | load: 0.00300 [I 2022-10-08 15:18:26] ...trap/engines/engine.py.230: loss: 0.17884 [I 2022-10-08 15:18:26] ...optimizers/trijoint.py.151: total_norm: 3.503080 [I 2022-10-08 15:18:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1830/2304 [I 2022-10-08 15:18:34] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:06:10 [I 2022-10-08 15:18:34] ...trap/engines/engine.py.229: process: 0.77580 | load: 0.00303 [I 2022-10-08 15:18:34] ...trap/engines/engine.py.230: loss: 0.17895 [I 2022-10-08 15:18:34] ...optimizers/trijoint.py.151: total_norm: 2.937816 [I 2022-10-08 15:18:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 1840/2304 [I 2022-10-08 15:18:42] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:05:58 [I 2022-10-08 15:18:42] ...trap/engines/engine.py.229: process: 0.76154 | load: 0.00299 [I 2022-10-08 15:18:42] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-08 15:18:42] ...optimizers/trijoint.py.151: total_norm: 1.943589 [I 2022-10-08 15:18:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 1850/2304 [I 2022-10-08 15:18:49] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:05:50 [I 2022-10-08 15:18:49] ...trap/engines/engine.py.229: process: 0.77029 | load: 0.00310 [I 2022-10-08 15:18:49] ...trap/engines/engine.py.230: loss: 0.14549 [I 2022-10-08 15:18:49] ...optimizers/trijoint.py.151: total_norm: 2.422273 [I 2022-10-08 15:18:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 1860/2304 [I 2022-10-08 15:18:57] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:45 [I 2022-10-08 15:18:57] ...trap/engines/engine.py.229: process: 0.77982 | load: 0.00301 [I 2022-10-08 15:18:57] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-08 15:18:57] ...optimizers/trijoint.py.151: total_norm: 2.363245 [I 2022-10-08 15:19:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1870/2304 [I 2022-10-08 15:19:05] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:05:37 [I 2022-10-08 15:19:05] ...trap/engines/engine.py.229: process: 0.77371 | load: 0.00311 [I 2022-10-08 15:19:05] ...trap/engines/engine.py.230: loss: 0.22769 [I 2022-10-08 15:19:05] ...optimizers/trijoint.py.151: total_norm: 3.244063 [I 2022-10-08 15:19:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1880/2304 [I 2022-10-08 15:19:13] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:05:28 [I 2022-10-08 15:19:13] ...trap/engines/engine.py.229: process: 0.76135 | load: 0.00306 [I 2022-10-08 15:19:13] ...trap/engines/engine.py.230: loss: 0.13669 [I 2022-10-08 15:19:13] ...optimizers/trijoint.py.151: total_norm: 2.253590 [I 2022-10-08 15:19:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 1890/2304 [I 2022-10-08 15:19:21] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:05:20 [I 2022-10-08 15:19:21] ...trap/engines/engine.py.229: process: 0.78255 | load: 0.00300 [I 2022-10-08 15:19:21] ...trap/engines/engine.py.230: loss: 0.19491 [I 2022-10-08 15:19:21] ...optimizers/trijoint.py.151: total_norm: 2.086456 [I 2022-10-08 15:19:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 1900/2304 [I 2022-10-08 15:19:28] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:05:14 [I 2022-10-08 15:19:28] ...trap/engines/engine.py.229: process: 0.78238 | load: 0.00306 [I 2022-10-08 15:19:28] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-08 15:19:28] ...optimizers/trijoint.py.151: total_norm: 1.863589 [I 2022-10-08 15:19:36] ...trap/engines/engine.py.224: train: epoch 9 | batch 1910/2304 [I 2022-10-08 15:19:36] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:07 [I 2022-10-08 15:19:36] ...trap/engines/engine.py.229: process: 0.77292 | load: 0.00303 [I 2022-10-08 15:19:36] ...trap/engines/engine.py.230: loss: 0.16433 [I 2022-10-08 15:19:36] ...optimizers/trijoint.py.151: total_norm: 2.411764 [I 2022-10-08 15:19:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1920/2304 [I 2022-10-08 15:19:44] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:05:00 [I 2022-10-08 15:19:44] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00304 [I 2022-10-08 15:19:44] ...trap/engines/engine.py.230: loss: 0.19702 [I 2022-10-08 15:19:44] ...optimizers/trijoint.py.151: total_norm: 2.666352 [I 2022-10-08 15:19:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 1930/2304 [I 2022-10-08 15:19:52] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:04:52 [I 2022-10-08 15:19:52] ...trap/engines/engine.py.229: process: 0.78693 | load: 0.00310 [I 2022-10-08 15:19:52] ...trap/engines/engine.py.230: loss: 0.16393 [I 2022-10-08 15:19:52] ...optimizers/trijoint.py.151: total_norm: 2.268186 [I 2022-10-08 15:20:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1940/2304 [I 2022-10-08 15:20:00] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:04:44 [I 2022-10-08 15:20:00] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00311 [I 2022-10-08 15:20:00] ...trap/engines/engine.py.230: loss: 0.19509 [I 2022-10-08 15:20:00] ...optimizers/trijoint.py.151: total_norm: 2.966385 [I 2022-10-08 15:20:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1950/2304 [I 2022-10-08 15:20:08] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:04:37 [I 2022-10-08 15:20:08] ...trap/engines/engine.py.229: process: 0.78652 | load: 0.00309 [I 2022-10-08 15:20:08] ...trap/engines/engine.py.230: loss: 0.17367 [I 2022-10-08 15:20:08] ...optimizers/trijoint.py.151: total_norm: 2.526983 [I 2022-10-08 15:20:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 1960/2304 [I 2022-10-08 15:20:15] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:04:30 [I 2022-10-08 15:20:15] ...trap/engines/engine.py.229: process: 0.80234 | load: 0.00310 [I 2022-10-08 15:20:15] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-08 15:20:15] ...optimizers/trijoint.py.151: total_norm: 2.278986 [I 2022-10-08 15:20:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 1970/2304 [I 2022-10-08 15:20:23] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:04:19 [I 2022-10-08 15:20:23] ...trap/engines/engine.py.229: process: 0.76864 | load: 0.00310 [I 2022-10-08 15:20:23] ...trap/engines/engine.py.230: loss: 0.28371 [I 2022-10-08 15:20:23] ...optimizers/trijoint.py.151: total_norm: 2.740189 [I 2022-10-08 15:20:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 1980/2304 [I 2022-10-08 15:20:31] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:04:12 [I 2022-10-08 15:20:31] ...trap/engines/engine.py.229: process: 0.78328 | load: 0.00311 [I 2022-10-08 15:20:31] ...trap/engines/engine.py.230: loss: 0.23460 [I 2022-10-08 15:20:31] ...optimizers/trijoint.py.151: total_norm: 3.083547 [I 2022-10-08 15:20:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1990/2304 [I 2022-10-08 15:20:39] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:04:04 [I 2022-10-08 15:20:39] ...trap/engines/engine.py.229: process: 0.77295 | load: 0.00313 [I 2022-10-08 15:20:39] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-08 15:20:39] ...optimizers/trijoint.py.151: total_norm: 2.648569 [I 2022-10-08 15:20:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 2000/2304 [I 2022-10-08 15:20:47] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:03:56 [I 2022-10-08 15:20:47] ...trap/engines/engine.py.229: process: 0.77845 | load: 0.00317 [I 2022-10-08 15:20:47] ...trap/engines/engine.py.230: loss: 0.25718 [I 2022-10-08 15:20:47] ...optimizers/trijoint.py.151: total_norm: 2.890831 [I 2022-10-08 15:20:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 2010/2304 [I 2022-10-08 15:20:55] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:03:50 [I 2022-10-08 15:20:55] ...trap/engines/engine.py.229: process: 0.77907 | load: 0.00307 [I 2022-10-08 15:20:55] ...trap/engines/engine.py.230: loss: 0.18826 [I 2022-10-08 15:20:55] ...optimizers/trijoint.py.151: total_norm: 2.511713 [I 2022-10-08 15:21:02] ...trap/engines/engine.py.224: train: epoch 9 | batch 2020/2304 [I 2022-10-08 15:21:02] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:03:41 [I 2022-10-08 15:21:02] ...trap/engines/engine.py.229: process: 0.78461 | load: 0.00299 [I 2022-10-08 15:21:02] ...trap/engines/engine.py.230: loss: 0.23229 [I 2022-10-08 15:21:02] ...optimizers/trijoint.py.151: total_norm: 2.806156 [I 2022-10-08 15:21:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 2030/2304 [I 2022-10-08 15:21:10] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:34 [I 2022-10-08 15:21:10] ...trap/engines/engine.py.229: process: 0.78459 | load: 0.00305 [I 2022-10-08 15:21:10] ...trap/engines/engine.py.230: loss: 0.20389 [I 2022-10-08 15:21:10] ...optimizers/trijoint.py.151: total_norm: 2.674168 [I 2022-10-08 15:21:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 2040/2304 [I 2022-10-08 15:21:18] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:03:27 [I 2022-10-08 15:21:18] ...trap/engines/engine.py.229: process: 0.80246 | load: 0.00302 [I 2022-10-08 15:21:18] ...trap/engines/engine.py.230: loss: 0.25041 [I 2022-10-08 15:21:18] ...optimizers/trijoint.py.151: total_norm: 3.155846 [I 2022-10-08 15:21:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 2050/2304 [I 2022-10-08 15:21:26] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:03:19 [I 2022-10-08 15:21:26] ...trap/engines/engine.py.229: process: 0.78034 | load: 0.00306 [I 2022-10-08 15:21:26] ...trap/engines/engine.py.230: loss: 0.21302 [I 2022-10-08 15:21:26] ...optimizers/trijoint.py.151: total_norm: 2.974297 [I 2022-10-08 15:21:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 2060/2304 [I 2022-10-08 15:21:34] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:03:11 [I 2022-10-08 15:21:34] ...trap/engines/engine.py.229: process: 0.78979 | load: 0.00304 [I 2022-10-08 15:21:34] ...trap/engines/engine.py.230: loss: 0.21689 [I 2022-10-08 15:21:34] ...optimizers/trijoint.py.151: total_norm: 2.816855 [I 2022-10-08 15:21:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 2070/2304 [I 2022-10-08 15:21:42] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:03:02 [I 2022-10-08 15:21:42] ...trap/engines/engine.py.229: process: 0.78323 | load: 0.00306 [I 2022-10-08 15:21:42] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-08 15:21:42] ...optimizers/trijoint.py.151: total_norm: 2.272752 [I 2022-10-08 15:21:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 2080/2304 [I 2022-10-08 15:21:49] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:02:54 [I 2022-10-08 15:21:49] ...trap/engines/engine.py.229: process: 0.76540 | load: 0.00313 [I 2022-10-08 15:21:49] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-08 15:21:49] ...optimizers/trijoint.py.151: total_norm: 2.645009 [I 2022-10-08 15:21:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 2090/2304 [I 2022-10-08 15:21:57] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:02:46 [I 2022-10-08 15:21:57] ...trap/engines/engine.py.229: process: 0.78024 | load: 0.00312 [I 2022-10-08 15:21:57] ...trap/engines/engine.py.230: loss: 0.33153 [I 2022-10-08 15:21:57] ...optimizers/trijoint.py.151: total_norm: 4.943058 [I 2022-10-08 15:22:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 2100/2304 [I 2022-10-08 15:22:05] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:02:38 [I 2022-10-08 15:22:05] ...trap/engines/engine.py.229: process: 0.76328 | load: 0.00318 [I 2022-10-08 15:22:05] ...trap/engines/engine.py.230: loss: 0.16416 [I 2022-10-08 15:22:05] ...optimizers/trijoint.py.151: total_norm: 1.930877 [I 2022-10-08 15:22:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 2110/2304 [I 2022-10-08 15:22:13] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:02:30 [I 2022-10-08 15:22:13] ...trap/engines/engine.py.229: process: 0.76378 | load: 0.00316 [I 2022-10-08 15:22:13] ...trap/engines/engine.py.230: loss: 0.21288 [I 2022-10-08 15:22:13] ...optimizers/trijoint.py.151: total_norm: 3.127683 [I 2022-10-08 15:22:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 2120/2304 [I 2022-10-08 15:22:21] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:02:23 [I 2022-10-08 15:22:21] ...trap/engines/engine.py.229: process: 0.77376 | load: 0.00307 [I 2022-10-08 15:22:21] ...trap/engines/engine.py.230: loss: 0.32210 [I 2022-10-08 15:22:21] ...optimizers/trijoint.py.151: total_norm: 3.669489 [I 2022-10-08 15:22:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 2130/2304 [I 2022-10-08 15:22:28] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:02:15 [I 2022-10-08 15:22:28] ...trap/engines/engine.py.229: process: 0.77939 | load: 0.00309 [I 2022-10-08 15:22:28] ...trap/engines/engine.py.230: loss: 0.21792 [I 2022-10-08 15:22:28] ...optimizers/trijoint.py.151: total_norm: 2.986239 [I 2022-10-08 15:22:36] ...trap/engines/engine.py.224: train: epoch 9 | batch 2140/2304 [I 2022-10-08 15:22:36] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:07 [I 2022-10-08 15:22:36] ...trap/engines/engine.py.229: process: 0.78113 | load: 0.00310 [I 2022-10-08 15:22:36] ...trap/engines/engine.py.230: loss: 0.23674 [I 2022-10-08 15:22:36] ...optimizers/trijoint.py.151: total_norm: 3.323317 [I 2022-10-08 15:22:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 2150/2304 [I 2022-10-08 15:22:44] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:01:59 [I 2022-10-08 15:22:44] ...trap/engines/engine.py.229: process: 0.76672 | load: 0.00313 [I 2022-10-08 15:22:44] ...trap/engines/engine.py.230: loss: 0.22954 [I 2022-10-08 15:22:44] ...optimizers/trijoint.py.151: total_norm: 3.566565 [I 2022-10-08 15:22:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 2160/2304 [I 2022-10-08 15:22:52] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:01:52 [I 2022-10-08 15:22:52] ...trap/engines/engine.py.229: process: 0.76860 | load: 0.00316 [I 2022-10-08 15:22:52] ...trap/engines/engine.py.230: loss: 0.21983 [I 2022-10-08 15:22:52] ...optimizers/trijoint.py.151: total_norm: 3.654130 [I 2022-10-08 15:23:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 2170/2304 [I 2022-10-08 15:23:00] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:01:44 [I 2022-10-08 15:23:00] ...trap/engines/engine.py.229: process: 0.78370 | load: 0.00301 [I 2022-10-08 15:23:00] ...trap/engines/engine.py.230: loss: 0.12619 [I 2022-10-08 15:23:00] ...optimizers/trijoint.py.151: total_norm: 2.217073 [I 2022-10-08 15:23:07] ...trap/engines/engine.py.224: train: epoch 9 | batch 2180/2304 [I 2022-10-08 15:23:07] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:01:36 [I 2022-10-08 15:23:08] ...trap/engines/engine.py.229: process: 0.75982 | load: 0.00299 [I 2022-10-08 15:23:08] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-08 15:23:08] ...optimizers/trijoint.py.151: total_norm: 2.845705 [I 2022-10-08 15:23:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 2190/2304 [I 2022-10-08 15:23:15] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:01:28 [I 2022-10-08 15:23:15] ...trap/engines/engine.py.229: process: 0.78237 | load: 0.00303 [I 2022-10-08 15:23:15] ...trap/engines/engine.py.230: loss: 0.19984 [I 2022-10-08 15:23:15] ...optimizers/trijoint.py.151: total_norm: 2.869813 [I 2022-10-08 15:23:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 2200/2304 [I 2022-10-08 15:23:23] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:01:20 [I 2022-10-08 15:23:23] ...trap/engines/engine.py.229: process: 0.76680 | load: 0.00305 [I 2022-10-08 15:23:23] ...trap/engines/engine.py.230: loss: 0.25955 [I 2022-10-08 15:23:23] ...optimizers/trijoint.py.151: total_norm: 3.311910 [I 2022-10-08 15:23:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 2210/2304 [I 2022-10-08 15:23:31] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:01:12 [I 2022-10-08 15:23:31] ...trap/engines/engine.py.229: process: 0.76613 | load: 0.00302 [I 2022-10-08 15:23:31] ...trap/engines/engine.py.230: loss: 0.23472 [I 2022-10-08 15:23:31] ...optimizers/trijoint.py.151: total_norm: 3.833064 [I 2022-10-08 15:23:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 2220/2304 [I 2022-10-08 15:23:39] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:01:05 [I 2022-10-08 15:23:39] ...trap/engines/engine.py.229: process: 0.76998 | load: 0.00300 [I 2022-10-08 15:23:39] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-08 15:23:39] ...optimizers/trijoint.py.151: total_norm: 2.055482 [I 2022-10-08 15:23:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 2230/2304 [I 2022-10-08 15:23:47] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:00:57 [I 2022-10-08 15:23:47] ...trap/engines/engine.py.229: process: 0.77288 | load: 0.00301 [I 2022-10-08 15:23:47] ...trap/engines/engine.py.230: loss: 0.09451 [I 2022-10-08 15:23:47] ...optimizers/trijoint.py.151: total_norm: 1.608941 [I 2022-10-08 15:23:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 2240/2304 [I 2022-10-08 15:23:54] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:00:49 [I 2022-10-08 15:23:54] ...trap/engines/engine.py.229: process: 0.78143 | load: 0.00300 [I 2022-10-08 15:23:54] ...trap/engines/engine.py.230: loss: 0.30074 [I 2022-10-08 15:23:54] ...optimizers/trijoint.py.151: total_norm: 3.366166 [I 2022-10-08 15:24:02] ...trap/engines/engine.py.224: train: epoch 9 | batch 2250/2304 [I 2022-10-08 15:24:02] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:00:41 [I 2022-10-08 15:24:02] ...trap/engines/engine.py.229: process: 0.77912 | load: 0.00301 [I 2022-10-08 15:24:02] ...trap/engines/engine.py.230: loss: 0.22443 [I 2022-10-08 15:24:02] ...optimizers/trijoint.py.151: total_norm: 3.049824 [I 2022-10-08 15:24:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 2260/2304 [I 2022-10-08 15:24:10] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:00:34 [I 2022-10-08 15:24:10] ...trap/engines/engine.py.229: process: 0.78586 | load: 0.00305 [I 2022-10-08 15:24:10] ...trap/engines/engine.py.230: loss: 0.24677 [I 2022-10-08 15:24:10] ...optimizers/trijoint.py.151: total_norm: 3.134141 [I 2022-10-08 15:24:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 2270/2304 [I 2022-10-08 15:24:18] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:00:26 [I 2022-10-08 15:24:18] ...trap/engines/engine.py.229: process: 0.77058 | load: 0.00302 [I 2022-10-08 15:24:18] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-08 15:24:18] ...optimizers/trijoint.py.151: total_norm: 2.336902 [I 2022-10-08 15:24:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 2280/2304 [I 2022-10-08 15:24:25] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:00:18 [I 2022-10-08 15:24:25] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00303 [I 2022-10-08 15:24:25] ...trap/engines/engine.py.230: loss: 0.19116 [I 2022-10-08 15:24:25] ...optimizers/trijoint.py.151: total_norm: 2.679939 [I 2022-10-08 15:24:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 2290/2304 [I 2022-10-08 15:24:33] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:10 [I 2022-10-08 15:24:33] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00303 [I 2022-10-08 15:24:33] ...trap/engines/engine.py.230: loss: 0.18366 [I 2022-10-08 15:24:33] ...optimizers/trijoint.py.151: total_norm: 2.293761 [I 2022-10-08 15:24:41] ...trap/engines/engine.py.224: train: epoch 9 | batch 2300/2304 [I 2022-10-08 15:24:41] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:00:03 [I 2022-10-08 15:24:41] ...trap/engines/engine.py.229: process: 0.77050 | load: 0.00017 [I 2022-10-08 15:24:41] ...trap/engines/engine.py.230: loss: 0.25973 [I 2022-10-08 15:24:41] ...optimizers/trijoint.py.151: total_norm: 3.570809 [I 2022-10-08 15:24:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 2304/2304 [I 2022-10-08 15:24:44] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:00:00 [I 2022-10-08 15:24:44] ...trap/engines/engine.py.229: process: 0.76067 | load: 0.00017 [I 2022-10-08 15:24:44] ...trap/engines/engine.py.230: loss: 0.23140 [I 2022-10-08 15:24:44] ...optimizers/trijoint.py.151: total_norm: 3.478462 [S 2022-10-08 15:24:45] ...trap/engines/engine.py.129: train_epoch.epoch: 9 [S 2022-10-08 15:24:45] ...trap/engines/engine.py.129: train_epoch.loss: 0.20266700544093538 [I 2022-10-08 15:24:45] ...s/criterions/triplet.py.84: margin = 0.10000000000000003 [I 2022-10-08 15:24:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 9 [I 2022-10-08 15:24:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 15:24:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 15:24:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 15:24:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 15:24:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 15:24:48] ...trap/engines/engine.py.321: eval: epoch 9 | batch 0/510 [I 2022-10-08 15:24:48] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:38 [I 2022-10-08 15:24:48] ...trap/engines/engine.py.326: process: 2.54618 | load: 2.02308 [I 2022-10-08 15:24:52] ...trap/engines/engine.py.321: eval: epoch 9 | batch 10/510 [I 2022-10-08 15:24:52] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:28 [I 2022-10-08 15:24:52] ...trap/engines/engine.py.326: process: 0.42391 | load: 0.00018 [I 2022-10-08 15:24:59] ...trap/engines/engine.py.321: eval: epoch 9 | batch 20/510 [I 2022-10-08 15:24:59] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:05:12 [I 2022-10-08 15:24:59] ...trap/engines/engine.py.326: process: 0.65194 | load: 0.00022 [I 2022-10-08 15:25:04] ...trap/engines/engine.py.321: eval: epoch 9 | batch 30/510 [I 2022-10-08 15:25:04] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:10 [I 2022-10-08 15:25:04] ...trap/engines/engine.py.326: process: 0.42390 | load: 0.00018 [I 2022-10-08 15:25:10] ...trap/engines/engine.py.321: eval: epoch 9 | batch 40/510 [I 2022-10-08 15:25:10] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:05 [I 2022-10-08 15:25:10] ...trap/engines/engine.py.326: process: 0.45551 | load: 0.00018 [I 2022-10-08 15:25:14] ...trap/engines/engine.py.321: eval: epoch 9 | batch 50/510 [I 2022-10-08 15:25:14] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:38 [I 2022-10-08 15:25:14] ...trap/engines/engine.py.326: process: 0.45262 | load: 0.00019 [I 2022-10-08 15:25:20] ...trap/engines/engine.py.321: eval: epoch 9 | batch 60/510 [I 2022-10-08 15:25:20] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:57 [I 2022-10-08 15:25:20] ...trap/engines/engine.py.326: process: 0.55426 | load: 0.00018 [I 2022-10-08 15:25:25] ...trap/engines/engine.py.321: eval: epoch 9 | batch 70/510 [I 2022-10-08 15:25:25] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:30 [I 2022-10-08 15:25:25] ...trap/engines/engine.py.326: process: 0.43319 | load: 0.00019 [I 2022-10-08 15:25:30] ...trap/engines/engine.py.321: eval: epoch 9 | batch 80/510 [I 2022-10-08 15:25:30] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:45 [I 2022-10-08 15:25:30] ...trap/engines/engine.py.326: process: 0.52628 | load: 0.00019 [I 2022-10-08 15:25:35] ...trap/engines/engine.py.321: eval: epoch 9 | batch 90/510 [I 2022-10-08 15:25:35] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:05 [I 2022-10-08 15:25:35] ...trap/engines/engine.py.326: process: 0.39664 | load: 0.00018 [I 2022-10-08 15:25:39] ...trap/engines/engine.py.321: eval: epoch 9 | batch 100/510 [I 2022-10-08 15:25:39] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:48 [I 2022-10-08 15:25:39] ...trap/engines/engine.py.326: process: 0.30450 | load: 0.00018 [I 2022-10-08 15:25:42] ...trap/engines/engine.py.321: eval: epoch 9 | batch 110/510 [I 2022-10-08 15:25:42] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:11 [I 2022-10-08 15:25:42] ...trap/engines/engine.py.326: process: 0.32120 | load: 0.00019 [I 2022-10-08 15:25:45] ...trap/engines/engine.py.321: eval: epoch 9 | batch 120/510 [I 2022-10-08 15:25:45] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:04 [I 2022-10-08 15:25:45] ...trap/engines/engine.py.326: process: 0.31698 | load: 0.00019 [I 2022-10-08 15:25:48] ...trap/engines/engine.py.321: eval: epoch 9 | batch 130/510 [I 2022-10-08 15:25:48] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:00 [I 2022-10-08 15:25:48] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00019 [I 2022-10-08 15:25:51] ...trap/engines/engine.py.321: eval: epoch 9 | batch 140/510 [I 2022-10-08 15:25:51] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:56 [I 2022-10-08 15:25:51] ...trap/engines/engine.py.326: process: 0.31412 | load: 0.00019 [I 2022-10-08 15:25:55] ...trap/engines/engine.py.321: eval: epoch 9 | batch 150/510 [I 2022-10-08 15:25:55] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:54 [I 2022-10-08 15:25:55] ...trap/engines/engine.py.326: process: 0.32298 | load: 0.00020 [I 2022-10-08 15:25:58] ...trap/engines/engine.py.321: eval: epoch 9 | batch 160/510 [I 2022-10-08 15:25:58] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:50 [I 2022-10-08 15:25:58] ...trap/engines/engine.py.326: process: 0.31571 | load: 0.00019 [I 2022-10-08 15:26:01] ...trap/engines/engine.py.321: eval: epoch 9 | batch 170/510 [I 2022-10-08 15:26:01] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:47 [I 2022-10-08 15:26:01] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00021 [I 2022-10-08 15:26:04] ...trap/engines/engine.py.321: eval: epoch 9 | batch 180/510 [I 2022-10-08 15:26:04] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:44 [I 2022-10-08 15:26:04] ...trap/engines/engine.py.326: process: 0.32227 | load: 0.00020 [I 2022-10-08 15:26:07] ...trap/engines/engine.py.321: eval: epoch 9 | batch 190/510 [I 2022-10-08 15:26:07] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:41 [I 2022-10-08 15:26:07] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00020 [I 2022-10-08 15:26:10] ...trap/engines/engine.py.321: eval: epoch 9 | batch 200/510 [I 2022-10-08 15:26:10] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:38 [I 2022-10-08 15:26:10] ...trap/engines/engine.py.326: process: 0.31939 | load: 0.00019 [I 2022-10-08 15:26:14] ...trap/engines/engine.py.321: eval: epoch 9 | batch 210/510 [I 2022-10-08 15:26:14] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:35 [I 2022-10-08 15:26:14] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00019 [I 2022-10-08 15:26:17] ...trap/engines/engine.py.321: eval: epoch 9 | batch 220/510 [I 2022-10-08 15:26:17] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:32 [I 2022-10-08 15:26:17] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00020 [I 2022-10-08 15:26:20] ...trap/engines/engine.py.321: eval: epoch 9 | batch 230/510 [I 2022-10-08 15:26:20] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:29 [I 2022-10-08 15:26:20] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00020 [I 2022-10-08 15:26:23] ...trap/engines/engine.py.321: eval: epoch 9 | batch 240/510 [I 2022-10-08 15:26:23] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:25 [I 2022-10-08 15:26:23] ...trap/engines/engine.py.326: process: 0.31649 | load: 0.00020 [I 2022-10-08 15:26:26] ...trap/engines/engine.py.321: eval: epoch 9 | batch 250/510 [I 2022-10-08 15:26:26] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-08 15:26:26] ...trap/engines/engine.py.326: process: 0.31426 | load: 0.00019 [I 2022-10-08 15:26:30] ...trap/engines/engine.py.321: eval: epoch 9 | batch 260/510 [I 2022-10-08 15:26:30] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:18 [I 2022-10-08 15:26:30] ...trap/engines/engine.py.326: process: 0.31207 | load: 0.00020 [I 2022-10-08 15:26:33] ...trap/engines/engine.py.321: eval: epoch 9 | batch 270/510 [I 2022-10-08 15:26:33] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:15 [I 2022-10-08 15:26:33] ...trap/engines/engine.py.326: process: 0.31038 | load: 0.00021 [I 2022-10-08 15:26:36] ...trap/engines/engine.py.321: eval: epoch 9 | batch 280/510 [I 2022-10-08 15:26:36] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:13 [I 2022-10-08 15:26:36] ...trap/engines/engine.py.326: process: 0.32147 | load: 0.00020 [I 2022-10-08 15:26:39] ...trap/engines/engine.py.321: eval: epoch 9 | batch 290/510 [I 2022-10-08 15:26:39] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:10 [I 2022-10-08 15:26:39] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00020 [I 2022-10-08 15:26:42] ...trap/engines/engine.py.321: eval: epoch 9 | batch 300/510 [I 2022-10-08 15:26:42] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:06 [I 2022-10-08 15:26:42] ...trap/engines/engine.py.326: process: 0.32320 | load: 0.00021 [I 2022-10-08 15:26:45] ...trap/engines/engine.py.321: eval: epoch 9 | batch 310/510 [I 2022-10-08 15:26:45] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:04 [I 2022-10-08 15:26:45] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00021 [I 2022-10-08 15:26:49] ...trap/engines/engine.py.321: eval: epoch 9 | batch 320/510 [I 2022-10-08 15:26:49] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-08 15:26:49] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00020 [I 2022-10-08 15:26:52] ...trap/engines/engine.py.321: eval: epoch 9 | batch 330/510 [I 2022-10-08 15:26:52] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:57 [I 2022-10-08 15:26:52] ...trap/engines/engine.py.326: process: 0.31907 | load: 0.00021 [I 2022-10-08 15:26:55] ...trap/engines/engine.py.321: eval: epoch 9 | batch 340/510 [I 2022-10-08 15:26:55] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:54 [I 2022-10-08 15:26:55] ...trap/engines/engine.py.326: process: 0.31578 | load: 0.00019 [I 2022-10-08 15:26:58] ...trap/engines/engine.py.321: eval: epoch 9 | batch 350/510 [I 2022-10-08 15:26:58] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:50 [I 2022-10-08 15:26:58] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00020 [I 2022-10-08 15:27:01] ...trap/engines/engine.py.321: eval: epoch 9 | batch 360/510 [I 2022-10-08 15:27:01] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-08 15:27:01] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00020 [I 2022-10-08 15:27:05] ...trap/engines/engine.py.321: eval: epoch 9 | batch 370/510 [I 2022-10-08 15:27:05] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-08 15:27:05] ...trap/engines/engine.py.326: process: 0.31860 | load: 0.00020 [I 2022-10-08 15:27:08] ...trap/engines/engine.py.321: eval: epoch 9 | batch 380/510 [I 2022-10-08 15:27:08] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:41 [I 2022-10-08 15:27:08] ...trap/engines/engine.py.326: process: 0.31679 | load: 0.00020 [I 2022-10-08 15:27:11] ...trap/engines/engine.py.321: eval: epoch 9 | batch 390/510 [I 2022-10-08 15:27:11] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:38 [I 2022-10-08 15:27:11] ...trap/engines/engine.py.326: process: 0.31880 | load: 0.00020 [I 2022-10-08 15:27:14] ...trap/engines/engine.py.321: eval: epoch 9 | batch 400/510 [I 2022-10-08 15:27:14] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:35 [I 2022-10-08 15:27:14] ...trap/engines/engine.py.326: process: 0.31605 | load: 0.00020 [I 2022-10-08 15:27:17] ...trap/engines/engine.py.321: eval: epoch 9 | batch 410/510 [I 2022-10-08 15:27:17] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:31 [I 2022-10-08 15:27:17] ...trap/engines/engine.py.326: process: 0.32098 | load: 0.00019 [I 2022-10-08 15:27:20] ...trap/engines/engine.py.321: eval: epoch 9 | batch 420/510 [I 2022-10-08 15:27:20] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-08 15:27:20] ...trap/engines/engine.py.326: process: 0.32281 | load: 0.00020 [I 2022-10-08 15:27:24] ...trap/engines/engine.py.321: eval: epoch 9 | batch 430/510 [I 2022-10-08 15:27:24] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-08 15:27:24] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00020 [I 2022-10-08 15:27:27] ...trap/engines/engine.py.321: eval: epoch 9 | batch 440/510 [I 2022-10-08 15:27:27] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-08 15:27:27] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00020 [I 2022-10-08 15:27:30] ...trap/engines/engine.py.321: eval: epoch 9 | batch 450/510 [I 2022-10-08 15:27:30] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:19 [I 2022-10-08 15:27:30] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00022 [I 2022-10-08 15:27:33] ...trap/engines/engine.py.321: eval: epoch 9 | batch 460/510 [I 2022-10-08 15:27:33] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:15 [I 2022-10-08 15:27:33] ...trap/engines/engine.py.326: process: 0.31365 | load: 0.00019 [I 2022-10-08 15:27:36] ...trap/engines/engine.py.321: eval: epoch 9 | batch 470/510 [I 2022-10-08 15:27:36] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-08 15:27:36] ...trap/engines/engine.py.326: process: 0.32446 | load: 0.00020 [I 2022-10-08 15:27:40] ...trap/engines/engine.py.321: eval: epoch 9 | batch 480/510 [I 2022-10-08 15:27:40] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-08 15:27:40] ...trap/engines/engine.py.326: process: 0.32209 | load: 0.00021 [I 2022-10-08 15:27:43] ...trap/engines/engine.py.321: eval: epoch 9 | batch 490/510 [I 2022-10-08 15:27:43] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-08 15:27:43] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00020 [I 2022-10-08 15:27:46] ...trap/engines/engine.py.321: eval: epoch 9 | batch 500/510 [I 2022-10-08 15:27:46] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:03 [I 2022-10-08 15:27:46] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00020 [I 2022-10-08 15:27:49] ...trap/engines/engine.py.321: eval: epoch 9 | batch 510/510 [I 2022-10-08 15:27:49] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:00 [I 2022-10-08 15:27:49] ...trap/engines/engine.py.326: process: 0.31562 | load: 0.00015 [S 2022-10-08 15:27:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 9 [I 2022-10-08 15:27:50] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.578300 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.844200 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.902300 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.590500 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.848100 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.901200 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014297 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013833 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011055 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013611 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010709 [S 2022-10-08 15:28:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008483 [I 2022-10-08 15:28:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 15:28:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 15:28:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 15:28:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 15:28:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 15:28:22] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 15:28:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 15:28:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 15:28:29] ...trap/engines/engine.py.163: Training model on trainset for epoch 10 [I 2022-10-08 15:28:29] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 15:28:32] ...trap/engines/engine.py.224: train: epoch 10 | batch 0/2304 [I 2022-10-08 15:28:32] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:46 [I 2022-10-08 15:28:32] ...trap/engines/engine.py.229: process: 2.98912 | load: 2.16089 [I 2022-10-08 15:28:32] ...trap/engines/engine.py.230: loss: 0.34670 [I 2022-10-08 15:28:32] ...optimizers/trijoint.py.151: total_norm: 3.254433 [I 2022-10-08 15:28:39] ...trap/engines/engine.py.224: train: epoch 10 | batch 10/2304 [I 2022-10-08 15:28:39] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:39:01 [I 2022-10-08 15:28:39] ...trap/engines/engine.py.229: process: 0.78398 | load: 0.00624 [I 2022-10-08 15:28:39] ...trap/engines/engine.py.230: loss: 0.18101 [I 2022-10-08 15:28:39] ...optimizers/trijoint.py.151: total_norm: 2.558332 [I 2022-10-08 15:28:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 20/2304 [I 2022-10-08 15:28:47] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:47 [I 2022-10-08 15:28:47] ...trap/engines/engine.py.229: process: 0.77852 | load: 0.00588 [I 2022-10-08 15:28:47] ...trap/engines/engine.py.230: loss: 0.20014 [I 2022-10-08 15:28:47] ...optimizers/trijoint.py.151: total_norm: 2.425281 [I 2022-10-08 15:28:55] ...trap/engines/engine.py.224: train: epoch 10 | batch 30/2304 [I 2022-10-08 15:28:55] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:38 [I 2022-10-08 15:28:55] ...trap/engines/engine.py.229: process: 0.78257 | load: 0.00419 [I 2022-10-08 15:28:55] ...trap/engines/engine.py.230: loss: 0.17176 [I 2022-10-08 15:28:55] ...optimizers/trijoint.py.151: total_norm: 2.855977 [I 2022-10-08 15:29:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 40/2304 [I 2022-10-08 15:29:03] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:18 [I 2022-10-08 15:29:03] ...trap/engines/engine.py.229: process: 0.77570 | load: 0.00389 [I 2022-10-08 15:29:03] ...trap/engines/engine.py.230: loss: 0.19464 [I 2022-10-08 15:29:03] ...optimizers/trijoint.py.151: total_norm: 2.623293 [I 2022-10-08 15:29:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 50/2304 [I 2022-10-08 15:29:11] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:28 [I 2022-10-08 15:29:11] ...trap/engines/engine.py.229: process: 0.79156 | load: 0.00395 [I 2022-10-08 15:29:11] ...trap/engines/engine.py.230: loss: 0.14965 [I 2022-10-08 15:29:11] ...optimizers/trijoint.py.151: total_norm: 1.911914 [I 2022-10-08 15:29:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 60/2304 [I 2022-10-08 15:29:18] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:12 [I 2022-10-08 15:29:18] ...trap/engines/engine.py.229: process: 0.77036 | load: 0.00368 [I 2022-10-08 15:29:18] ...trap/engines/engine.py.230: loss: 0.25294 [I 2022-10-08 15:29:18] ...optimizers/trijoint.py.151: total_norm: 2.888992 [I 2022-10-08 15:29:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 70/2304 [I 2022-10-08 15:29:26] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:07 [I 2022-10-08 15:29:26] ...trap/engines/engine.py.229: process: 0.77572 | load: 0.00369 [I 2022-10-08 15:29:26] ...trap/engines/engine.py.230: loss: 0.22174 [I 2022-10-08 15:29:26] ...optimizers/trijoint.py.151: total_norm: 3.280878 [I 2022-10-08 15:29:34] ...trap/engines/engine.py.224: train: epoch 10 | batch 80/2304 [I 2022-10-08 15:29:34] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:51 [I 2022-10-08 15:29:34] ...trap/engines/engine.py.229: process: 0.77165 | load: 0.00358 [I 2022-10-08 15:29:34] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-08 15:29:34] ...optimizers/trijoint.py.151: total_norm: 2.909617 [I 2022-10-08 15:29:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 90/2304 [I 2022-10-08 15:29:42] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:44 [I 2022-10-08 15:29:42] ...trap/engines/engine.py.229: process: 0.78244 | load: 0.00334 [I 2022-10-08 15:29:42] ...trap/engines/engine.py.230: loss: 0.22655 [I 2022-10-08 15:29:42] ...optimizers/trijoint.py.151: total_norm: 2.731042 [I 2022-10-08 15:29:50] ...trap/engines/engine.py.224: train: epoch 10 | batch 100/2304 [I 2022-10-08 15:29:50] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:41 [I 2022-10-08 15:29:50] ...trap/engines/engine.py.229: process: 0.78751 | load: 0.00316 [I 2022-10-08 15:29:50] ...trap/engines/engine.py.230: loss: 0.26728 [I 2022-10-08 15:29:50] ...optimizers/trijoint.py.151: total_norm: 3.184461 [I 2022-10-08 15:29:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 110/2304 [I 2022-10-08 15:29:58] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:32 [I 2022-10-08 15:29:58] ...trap/engines/engine.py.229: process: 0.76953 | load: 0.00361 [I 2022-10-08 15:29:58] ...trap/engines/engine.py.230: loss: 0.22758 [I 2022-10-08 15:29:58] ...optimizers/trijoint.py.151: total_norm: 3.551923 [I 2022-10-08 15:30:05] ...trap/engines/engine.py.224: train: epoch 10 | batch 120/2304 [I 2022-10-08 15:30:05] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:27 [I 2022-10-08 15:30:05] ...trap/engines/engine.py.229: process: 0.78250 | load: 0.00333 [I 2022-10-08 15:30:05] ...trap/engines/engine.py.230: loss: 0.22812 [I 2022-10-08 15:30:05] ...optimizers/trijoint.py.151: total_norm: 3.010593 [I 2022-10-08 15:30:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 130/2304 [I 2022-10-08 15:30:13] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:22 [I 2022-10-08 15:30:13] ...trap/engines/engine.py.229: process: 0.78451 | load: 0.00317 [I 2022-10-08 15:30:13] ...trap/engines/engine.py.230: loss: 0.15627 [I 2022-10-08 15:30:13] ...optimizers/trijoint.py.151: total_norm: 2.802080 [I 2022-10-08 15:30:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 140/2304 [I 2022-10-08 15:30:21] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:09 [I 2022-10-08 15:30:21] ...trap/engines/engine.py.229: process: 0.78203 | load: 0.00312 [I 2022-10-08 15:30:21] ...trap/engines/engine.py.230: loss: 0.30007 [I 2022-10-08 15:30:21] ...optimizers/trijoint.py.151: total_norm: 3.454513 [I 2022-10-08 15:30:29] ...trap/engines/engine.py.224: train: epoch 10 | batch 150/2304 [I 2022-10-08 15:30:29] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:00 [I 2022-10-08 15:30:29] ...trap/engines/engine.py.229: process: 0.76116 | load: 0.00339 [I 2022-10-08 15:30:29] ...trap/engines/engine.py.230: loss: 0.27176 [I 2022-10-08 15:30:29] ...optimizers/trijoint.py.151: total_norm: 3.134142 [I 2022-10-08 15:30:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 160/2304 [I 2022-10-08 15:30:37] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:48 [I 2022-10-08 15:30:37] ...trap/engines/engine.py.229: process: 0.77363 | load: 0.00300 [I 2022-10-08 15:30:37] ...trap/engines/engine.py.230: loss: 0.24496 [I 2022-10-08 15:30:37] ...optimizers/trijoint.py.151: total_norm: 2.601927 [I 2022-10-08 15:30:45] ...trap/engines/engine.py.224: train: epoch 10 | batch 170/2304 [I 2022-10-08 15:30:45] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:55 [I 2022-10-08 15:30:45] ...trap/engines/engine.py.229: process: 0.78314 | load: 0.00324 [I 2022-10-08 15:30:45] ...trap/engines/engine.py.230: loss: 0.19148 [I 2022-10-08 15:30:45] ...optimizers/trijoint.py.151: total_norm: 2.532744 [I 2022-10-08 15:30:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 180/2304 [I 2022-10-08 15:30:52] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:36 [I 2022-10-08 15:30:52] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00330 [I 2022-10-08 15:30:52] ...trap/engines/engine.py.230: loss: 0.16884 [I 2022-10-08 15:30:52] ...optimizers/trijoint.py.151: total_norm: 2.718355 [I 2022-10-08 15:31:00] ...trap/engines/engine.py.224: train: epoch 10 | batch 190/2304 [I 2022-10-08 15:31:00] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:27 [I 2022-10-08 15:31:00] ...trap/engines/engine.py.229: process: 0.78214 | load: 0.00314 [I 2022-10-08 15:31:00] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-08 15:31:00] ...optimizers/trijoint.py.151: total_norm: 1.904301 [I 2022-10-08 15:31:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 200/2304 [I 2022-10-08 15:31:08] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:14 [I 2022-10-08 15:31:08] ...trap/engines/engine.py.229: process: 0.77867 | load: 0.00317 [I 2022-10-08 15:31:08] ...trap/engines/engine.py.230: loss: 0.15575 [I 2022-10-08 15:31:08] ...optimizers/trijoint.py.151: total_norm: 2.158790 [I 2022-10-08 15:31:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 210/2304 [I 2022-10-08 15:31:16] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:04 [I 2022-10-08 15:31:16] ...trap/engines/engine.py.229: process: 0.76602 | load: 0.00304 [I 2022-10-08 15:31:16] ...trap/engines/engine.py.230: loss: 0.19022 [I 2022-10-08 15:31:16] ...optimizers/trijoint.py.151: total_norm: 2.892948 [I 2022-10-08 15:31:24] ...trap/engines/engine.py.224: train: epoch 10 | batch 220/2304 [I 2022-10-08 15:31:24] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:27:07 [I 2022-10-08 15:31:24] ...trap/engines/engine.py.229: process: 0.77843 | load: 0.00303 [I 2022-10-08 15:31:24] ...trap/engines/engine.py.230: loss: 0.19363 [I 2022-10-08 15:31:24] ...optimizers/trijoint.py.151: total_norm: 1.997327 [I 2022-10-08 15:31:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 230/2304 [I 2022-10-08 15:31:31] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:27:02 [I 2022-10-08 15:31:31] ...trap/engines/engine.py.229: process: 0.78263 | load: 0.00315 [I 2022-10-08 15:31:31] ...trap/engines/engine.py.230: loss: 0.30598 [I 2022-10-08 15:31:31] ...optimizers/trijoint.py.151: total_norm: 3.192444 [I 2022-10-08 15:31:39] ...trap/engines/engine.py.224: train: epoch 10 | batch 240/2304 [I 2022-10-08 15:31:39] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:55 [I 2022-10-08 15:31:39] ...trap/engines/engine.py.229: process: 0.77912 | load: 0.00311 [I 2022-10-08 15:31:39] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-08 15:31:39] ...optimizers/trijoint.py.151: total_norm: 2.100921 [I 2022-10-08 15:31:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 250/2304 [I 2022-10-08 15:31:47] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:52 [I 2022-10-08 15:31:47] ...trap/engines/engine.py.229: process: 0.78474 | load: 0.00300 [I 2022-10-08 15:31:47] ...trap/engines/engine.py.230: loss: 0.24209 [I 2022-10-08 15:31:47] ...optimizers/trijoint.py.151: total_norm: 2.784479 [I 2022-10-08 15:31:55] ...trap/engines/engine.py.224: train: epoch 10 | batch 260/2304 [I 2022-10-08 15:31:55] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:45 [I 2022-10-08 15:31:55] ...trap/engines/engine.py.229: process: 0.77784 | load: 0.00308 [I 2022-10-08 15:31:55] ...trap/engines/engine.py.230: loss: 0.26835 [I 2022-10-08 15:31:55] ...optimizers/trijoint.py.151: total_norm: 3.474594 [I 2022-10-08 15:32:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 270/2304 [I 2022-10-08 15:32:03] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:28 [I 2022-10-08 15:32:03] ...trap/engines/engine.py.229: process: 0.78156 | load: 0.00325 [I 2022-10-08 15:32:03] ...trap/engines/engine.py.230: loss: 0.18224 [I 2022-10-08 15:32:03] ...optimizers/trijoint.py.151: total_norm: 2.348490 [I 2022-10-08 15:32:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 280/2304 [I 2022-10-08 15:32:11] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:26:14 [I 2022-10-08 15:32:11] ...trap/engines/engine.py.229: process: 0.77473 | load: 0.00312 [I 2022-10-08 15:32:11] ...trap/engines/engine.py.230: loss: 0.19256 [I 2022-10-08 15:32:11] ...optimizers/trijoint.py.151: total_norm: 2.149279 [I 2022-10-08 15:32:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 290/2304 [I 2022-10-08 15:32:18] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:01 [I 2022-10-08 15:32:18] ...trap/engines/engine.py.229: process: 0.77966 | load: 0.00326 [I 2022-10-08 15:32:18] ...trap/engines/engine.py.230: loss: 0.12073 [I 2022-10-08 15:32:18] ...optimizers/trijoint.py.151: total_norm: 2.128369 [I 2022-10-08 15:32:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 300/2304 [I 2022-10-08 15:32:26] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:26:04 [I 2022-10-08 15:32:26] ...trap/engines/engine.py.229: process: 0.78896 | load: 0.00310 [I 2022-10-08 15:32:26] ...trap/engines/engine.py.230: loss: 0.17141 [I 2022-10-08 15:32:26] ...optimizers/trijoint.py.151: total_norm: 2.719701 [I 2022-10-08 15:32:34] ...trap/engines/engine.py.224: train: epoch 10 | batch 310/2304 [I 2022-10-08 15:32:34] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:50 [I 2022-10-08 15:32:34] ...trap/engines/engine.py.229: process: 0.77597 | load: 0.00316 [I 2022-10-08 15:32:34] ...trap/engines/engine.py.230: loss: 0.22945 [I 2022-10-08 15:32:34] ...optimizers/trijoint.py.151: total_norm: 2.942340 [I 2022-10-08 15:32:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 320/2304 [I 2022-10-08 15:32:42] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:45 [I 2022-10-08 15:32:42] ...trap/engines/engine.py.229: process: 0.77116 | load: 0.00309 [I 2022-10-08 15:32:42] ...trap/engines/engine.py.230: loss: 0.21348 [I 2022-10-08 15:32:42] ...optimizers/trijoint.py.151: total_norm: 3.069294 [I 2022-10-08 15:32:50] ...trap/engines/engine.py.224: train: epoch 10 | batch 330/2304 [I 2022-10-08 15:32:50] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:30 [I 2022-10-08 15:32:50] ...trap/engines/engine.py.229: process: 0.76227 | load: 0.00310 [I 2022-10-08 15:32:50] ...trap/engines/engine.py.230: loss: 0.24419 [I 2022-10-08 15:32:50] ...optimizers/trijoint.py.151: total_norm: 2.030893 [I 2022-10-08 15:32:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 340/2304 [I 2022-10-08 15:32:57] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:33 [I 2022-10-08 15:32:57] ...trap/engines/engine.py.229: process: 0.78351 | load: 0.00314 [I 2022-10-08 15:32:57] ...trap/engines/engine.py.230: loss: 0.17968 [I 2022-10-08 15:32:57] ...optimizers/trijoint.py.151: total_norm: 2.688820 [I 2022-10-08 15:33:05] ...trap/engines/engine.py.224: train: epoch 10 | batch 350/2304 [I 2022-10-08 15:33:05] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:21 [I 2022-10-08 15:33:05] ...trap/engines/engine.py.229: process: 0.78516 | load: 0.00313 [I 2022-10-08 15:33:05] ...trap/engines/engine.py.230: loss: 0.31710 [I 2022-10-08 15:33:05] ...optimizers/trijoint.py.151: total_norm: 2.999461 [I 2022-10-08 15:33:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 360/2304 [I 2022-10-08 15:33:13] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:24 [I 2022-10-08 15:33:13] ...trap/engines/engine.py.229: process: 0.77451 | load: 0.00314 [I 2022-10-08 15:33:13] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-08 15:33:13] ...optimizers/trijoint.py.151: total_norm: 2.475753 [I 2022-10-08 15:33:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 370/2304 [I 2022-10-08 15:33:21] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:25:22 [I 2022-10-08 15:33:21] ...trap/engines/engine.py.229: process: 0.79250 | load: 0.00309 [I 2022-10-08 15:33:21] ...trap/engines/engine.py.230: loss: 0.20886 [I 2022-10-08 15:33:21] ...optimizers/trijoint.py.151: total_norm: 3.384449 [I 2022-10-08 15:33:29] ...trap/engines/engine.py.224: train: epoch 10 | batch 380/2304 [I 2022-10-08 15:33:29] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:53 [I 2022-10-08 15:33:29] ...trap/engines/engine.py.229: process: 0.76793 | load: 0.00317 [I 2022-10-08 15:33:29] ...trap/engines/engine.py.230: loss: 0.20212 [I 2022-10-08 15:33:29] ...optimizers/trijoint.py.151: total_norm: 2.271076 [I 2022-10-08 15:33:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 390/2304 [I 2022-10-08 15:33:37] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:24:54 [I 2022-10-08 15:33:37] ...trap/engines/engine.py.229: process: 0.77884 | load: 0.00314 [I 2022-10-08 15:33:37] ...trap/engines/engine.py.230: loss: 0.31177 [I 2022-10-08 15:33:37] ...optimizers/trijoint.py.151: total_norm: 2.946738 [I 2022-10-08 15:33:44] ...trap/engines/engine.py.224: train: epoch 10 | batch 400/2304 [I 2022-10-08 15:33:44] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:45 [I 2022-10-08 15:33:44] ...trap/engines/engine.py.229: process: 0.77691 | load: 0.00322 [I 2022-10-08 15:33:44] ...trap/engines/engine.py.230: loss: 0.16180 [I 2022-10-08 15:33:44] ...optimizers/trijoint.py.151: total_norm: 3.172304 [I 2022-10-08 15:33:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 410/2304 [I 2022-10-08 15:33:52] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:37 [I 2022-10-08 15:33:52] ...trap/engines/engine.py.229: process: 0.77794 | load: 0.00341 [I 2022-10-08 15:33:52] ...trap/engines/engine.py.230: loss: 0.20845 [I 2022-10-08 15:33:52] ...optimizers/trijoint.py.151: total_norm: 3.033754 [I 2022-10-08 15:34:00] ...trap/engines/engine.py.224: train: epoch 10 | batch 420/2304 [I 2022-10-08 15:34:00] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:26 [I 2022-10-08 15:34:00] ...trap/engines/engine.py.229: process: 0.77326 | load: 0.00328 [I 2022-10-08 15:34:00] ...trap/engines/engine.py.230: loss: 0.21834 [I 2022-10-08 15:34:00] ...optimizers/trijoint.py.151: total_norm: 2.250382 [I 2022-10-08 15:34:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 430/2304 [I 2022-10-08 15:34:08] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:18 [I 2022-10-08 15:34:08] ...trap/engines/engine.py.229: process: 0.77967 | load: 0.00314 [I 2022-10-08 15:34:08] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-08 15:34:08] ...optimizers/trijoint.py.151: total_norm: 1.946131 [I 2022-10-08 15:34:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 440/2304 [I 2022-10-08 15:34:16] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:24:08 [I 2022-10-08 15:34:16] ...trap/engines/engine.py.229: process: 0.77547 | load: 0.00317 [I 2022-10-08 15:34:16] ...trap/engines/engine.py.230: loss: 0.29891 [I 2022-10-08 15:34:16] ...optimizers/trijoint.py.151: total_norm: 4.034739 [I 2022-10-08 15:34:24] ...trap/engines/engine.py.224: train: epoch 10 | batch 450/2304 [I 2022-10-08 15:34:24] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:11 [I 2022-10-08 15:34:24] ...trap/engines/engine.py.229: process: 0.78709 | load: 0.00315 [I 2022-10-08 15:34:24] ...trap/engines/engine.py.230: loss: 0.21985 [I 2022-10-08 15:34:24] ...optimizers/trijoint.py.151: total_norm: 2.833083 [I 2022-10-08 15:34:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 460/2304 [I 2022-10-08 15:34:31] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:51 [I 2022-10-08 15:34:31] ...trap/engines/engine.py.229: process: 0.77896 | load: 0.00305 [I 2022-10-08 15:34:31] ...trap/engines/engine.py.230: loss: 0.20900 [I 2022-10-08 15:34:31] ...optimizers/trijoint.py.151: total_norm: 2.355514 [I 2022-10-08 15:34:39] ...trap/engines/engine.py.224: train: epoch 10 | batch 470/2304 [I 2022-10-08 15:34:39] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:55 [I 2022-10-08 15:34:39] ...trap/engines/engine.py.229: process: 0.78313 | load: 0.00307 [I 2022-10-08 15:34:39] ...trap/engines/engine.py.230: loss: 0.21731 [I 2022-10-08 15:34:39] ...optimizers/trijoint.py.151: total_norm: 2.989119 [I 2022-10-08 15:34:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 480/2304 [I 2022-10-08 15:34:47] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:37 [I 2022-10-08 15:34:47] ...trap/engines/engine.py.229: process: 0.77819 | load: 0.00307 [I 2022-10-08 15:34:47] ...trap/engines/engine.py.230: loss: 0.18581 [I 2022-10-08 15:34:47] ...optimizers/trijoint.py.151: total_norm: 2.033395 [I 2022-10-08 15:34:55] ...trap/engines/engine.py.224: train: epoch 10 | batch 490/2304 [I 2022-10-08 15:34:55] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:30 [I 2022-10-08 15:34:55] ...trap/engines/engine.py.229: process: 0.77295 | load: 0.00302 [I 2022-10-08 15:34:55] ...trap/engines/engine.py.230: loss: 0.16634 [I 2022-10-08 15:34:55] ...optimizers/trijoint.py.151: total_norm: 2.716773 [I 2022-10-08 15:35:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 500/2304 [I 2022-10-08 15:35:03] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:25 [I 2022-10-08 15:35:03] ...trap/engines/engine.py.229: process: 0.77800 | load: 0.00302 [I 2022-10-08 15:35:03] ...trap/engines/engine.py.230: loss: 0.20379 [I 2022-10-08 15:35:03] ...optimizers/trijoint.py.151: total_norm: 3.239190 [I 2022-10-08 15:35:10] ...trap/engines/engine.py.224: train: epoch 10 | batch 510/2304 [I 2022-10-08 15:35:10] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:19 [I 2022-10-08 15:35:10] ...trap/engines/engine.py.229: process: 0.78031 | load: 0.00302 [I 2022-10-08 15:35:10] ...trap/engines/engine.py.230: loss: 0.22602 [I 2022-10-08 15:35:10] ...optimizers/trijoint.py.151: total_norm: 3.032446 [I 2022-10-08 15:35:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 520/2304 [I 2022-10-08 15:35:18] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:08 [I 2022-10-08 15:35:18] ...trap/engines/engine.py.229: process: 0.77151 | load: 0.00304 [I 2022-10-08 15:35:18] ...trap/engines/engine.py.230: loss: 0.14844 [I 2022-10-08 15:35:18] ...optimizers/trijoint.py.151: total_norm: 3.603479 [I 2022-10-08 15:35:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 530/2304 [I 2022-10-08 15:35:26] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:22:54 [I 2022-10-08 15:35:26] ...trap/engines/engine.py.229: process: 0.77253 | load: 0.00309 [I 2022-10-08 15:35:26] ...trap/engines/engine.py.230: loss: 0.28708 [I 2022-10-08 15:35:26] ...optimizers/trijoint.py.151: total_norm: 4.320642 [I 2022-10-08 15:35:34] ...trap/engines/engine.py.224: train: epoch 10 | batch 540/2304 [I 2022-10-08 15:35:34] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:49 [I 2022-10-08 15:35:34] ...trap/engines/engine.py.229: process: 0.78247 | load: 0.00305 [I 2022-10-08 15:35:34] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-08 15:35:34] ...optimizers/trijoint.py.151: total_norm: 1.706100 [I 2022-10-08 15:35:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 550/2304 [I 2022-10-08 15:35:42] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:53 [I 2022-10-08 15:35:42] ...trap/engines/engine.py.229: process: 0.79048 | load: 0.00309 [I 2022-10-08 15:35:42] ...trap/engines/engine.py.230: loss: 0.17426 [I 2022-10-08 15:35:42] ...optimizers/trijoint.py.151: total_norm: 2.637177 [I 2022-10-08 15:35:49] ...trap/engines/engine.py.224: train: epoch 10 | batch 560/2304 [I 2022-10-08 15:35:49] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:33 [I 2022-10-08 15:35:49] ...trap/engines/engine.py.229: process: 0.76667 | load: 0.00310 [I 2022-10-08 15:35:49] ...trap/engines/engine.py.230: loss: 0.28803 [I 2022-10-08 15:35:49] ...optimizers/trijoint.py.151: total_norm: 3.002947 [I 2022-10-08 15:35:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 570/2304 [I 2022-10-08 15:35:57] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:32 [I 2022-10-08 15:35:57] ...trap/engines/engine.py.229: process: 0.77444 | load: 0.00314 [I 2022-10-08 15:35:57] ...trap/engines/engine.py.230: loss: 0.17465 [I 2022-10-08 15:35:57] ...optimizers/trijoint.py.151: total_norm: 2.460096 [I 2022-10-08 15:36:05] ...trap/engines/engine.py.224: train: epoch 10 | batch 580/2304 [I 2022-10-08 15:36:05] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:33 [I 2022-10-08 15:36:05] ...trap/engines/engine.py.229: process: 0.78418 | load: 0.00310 [I 2022-10-08 15:36:05] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-08 15:36:05] ...optimizers/trijoint.py.151: total_norm: 2.095746 [I 2022-10-08 15:36:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 590/2304 [I 2022-10-08 15:36:13] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:19 [I 2022-10-08 15:36:13] ...trap/engines/engine.py.229: process: 0.77691 | load: 0.00311 [I 2022-10-08 15:36:13] ...trap/engines/engine.py.230: loss: 0.23209 [I 2022-10-08 15:36:13] ...optimizers/trijoint.py.151: total_norm: 2.978865 [I 2022-10-08 15:36:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 600/2304 [I 2022-10-08 15:36:21] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:09 [I 2022-10-08 15:36:21] ...trap/engines/engine.py.229: process: 0.78745 | load: 0.00316 [I 2022-10-08 15:36:21] ...trap/engines/engine.py.230: loss: 0.25050 [I 2022-10-08 15:36:21] ...optimizers/trijoint.py.151: total_norm: 2.877272 [I 2022-10-08 15:36:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 610/2304 [I 2022-10-08 15:36:28] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:21:58 [I 2022-10-08 15:36:28] ...trap/engines/engine.py.229: process: 0.77112 | load: 0.00314 [I 2022-10-08 15:36:28] ...trap/engines/engine.py.230: loss: 0.19092 [I 2022-10-08 15:36:28] ...optimizers/trijoint.py.151: total_norm: 2.391084 [I 2022-10-08 15:36:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 620/2304 [I 2022-10-08 15:36:36] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:42 [I 2022-10-08 15:36:36] ...trap/engines/engine.py.229: process: 0.76653 | load: 0.00308 [I 2022-10-08 15:36:36] ...trap/engines/engine.py.230: loss: 0.22967 [I 2022-10-08 15:36:36] ...optimizers/trijoint.py.151: total_norm: 2.597444 [I 2022-10-08 15:36:44] ...trap/engines/engine.py.224: train: epoch 10 | batch 630/2304 [I 2022-10-08 15:36:44] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:52 [I 2022-10-08 15:36:44] ...trap/engines/engine.py.229: process: 0.78797 | load: 0.00312 [I 2022-10-08 15:36:44] ...trap/engines/engine.py.230: loss: 0.19034 [I 2022-10-08 15:36:44] ...optimizers/trijoint.py.151: total_norm: 2.379363 [I 2022-10-08 15:36:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 640/2304 [I 2022-10-08 15:36:52] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:37 [I 2022-10-08 15:36:52] ...trap/engines/engine.py.229: process: 0.77359 | load: 0.00317 [I 2022-10-08 15:36:52] ...trap/engines/engine.py.230: loss: 0.19446 [I 2022-10-08 15:36:52] ...optimizers/trijoint.py.151: total_norm: 2.570248 [I 2022-10-08 15:37:00] ...trap/engines/engine.py.224: train: epoch 10 | batch 650/2304 [I 2022-10-08 15:37:00] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:30 [I 2022-10-08 15:37:00] ...trap/engines/engine.py.229: process: 0.79147 | load: 0.00309 [I 2022-10-08 15:37:00] ...trap/engines/engine.py.230: loss: 0.22317 [I 2022-10-08 15:37:00] ...optimizers/trijoint.py.151: total_norm: 3.174844 [I 2022-10-08 15:37:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 660/2304 [I 2022-10-08 15:37:07] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:13 [I 2022-10-08 15:37:07] ...trap/engines/engine.py.229: process: 0.77665 | load: 0.00314 [I 2022-10-08 15:37:07] ...trap/engines/engine.py.230: loss: 0.21454 [I 2022-10-08 15:37:07] ...optimizers/trijoint.py.151: total_norm: 2.367598 [I 2022-10-08 15:37:15] ...trap/engines/engine.py.224: train: epoch 10 | batch 670/2304 [I 2022-10-08 15:37:15] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:16 [I 2022-10-08 15:37:15] ...trap/engines/engine.py.229: process: 0.78409 | load: 0.00311 [I 2022-10-08 15:37:15] ...trap/engines/engine.py.230: loss: 0.22761 [I 2022-10-08 15:37:15] ...optimizers/trijoint.py.151: total_norm: 2.743100 [I 2022-10-08 15:37:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 680/2304 [I 2022-10-08 15:37:23] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:03 [I 2022-10-08 15:37:23] ...trap/engines/engine.py.229: process: 0.78152 | load: 0.00310 [I 2022-10-08 15:37:23] ...trap/engines/engine.py.230: loss: 0.22931 [I 2022-10-08 15:37:23] ...optimizers/trijoint.py.151: total_norm: 3.038233 [I 2022-10-08 15:37:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 690/2304 [I 2022-10-08 15:37:31] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:54 [I 2022-10-08 15:37:31] ...trap/engines/engine.py.229: process: 0.78807 | load: 0.00318 [I 2022-10-08 15:37:31] ...trap/engines/engine.py.230: loss: 0.19960 [I 2022-10-08 15:37:31] ...optimizers/trijoint.py.151: total_norm: 1.641618 [I 2022-10-08 15:37:39] ...trap/engines/engine.py.224: train: epoch 10 | batch 700/2304 [I 2022-10-08 15:37:39] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:53 [I 2022-10-08 15:37:39] ...trap/engines/engine.py.229: process: 0.78081 | load: 0.00309 [I 2022-10-08 15:37:39] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-08 15:37:39] ...optimizers/trijoint.py.151: total_norm: 1.832455 [I 2022-10-08 15:37:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 710/2304 [I 2022-10-08 15:37:47] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:45 [I 2022-10-08 15:37:47] ...trap/engines/engine.py.229: process: 0.78170 | load: 0.00313 [I 2022-10-08 15:37:47] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-08 15:37:47] ...optimizers/trijoint.py.151: total_norm: 2.668332 [I 2022-10-08 15:37:54] ...trap/engines/engine.py.224: train: epoch 10 | batch 720/2304 [I 2022-10-08 15:37:54] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:36 [I 2022-10-08 15:37:54] ...trap/engines/engine.py.229: process: 0.78534 | load: 0.00313 [I 2022-10-08 15:37:54] ...trap/engines/engine.py.230: loss: 0.29580 [I 2022-10-08 15:37:54] ...optimizers/trijoint.py.151: total_norm: 2.905594 [I 2022-10-08 15:38:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 730/2304 [I 2022-10-08 15:38:02] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:19 [I 2022-10-08 15:38:02] ...trap/engines/engine.py.229: process: 0.76939 | load: 0.00314 [I 2022-10-08 15:38:02] ...trap/engines/engine.py.230: loss: 0.17589 [I 2022-10-08 15:38:02] ...optimizers/trijoint.py.151: total_norm: 2.084543 [I 2022-10-08 15:38:10] ...trap/engines/engine.py.224: train: epoch 10 | batch 740/2304 [I 2022-10-08 15:38:10] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:10 [I 2022-10-08 15:38:10] ...trap/engines/engine.py.229: process: 0.76760 | load: 0.00311 [I 2022-10-08 15:38:10] ...trap/engines/engine.py.230: loss: 0.19464 [I 2022-10-08 15:38:10] ...optimizers/trijoint.py.151: total_norm: 2.245213 [I 2022-10-08 15:38:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 750/2304 [I 2022-10-08 15:38:18] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:07 [I 2022-10-08 15:38:18] ...trap/engines/engine.py.229: process: 0.78329 | load: 0.00313 [I 2022-10-08 15:38:18] ...trap/engines/engine.py.230: loss: 0.25209 [I 2022-10-08 15:38:18] ...optimizers/trijoint.py.151: total_norm: 3.222158 [I 2022-10-08 15:38:25] ...trap/engines/engine.py.224: train: epoch 10 | batch 760/2304 [I 2022-10-08 15:38:25] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:03 [I 2022-10-08 15:38:25] ...trap/engines/engine.py.229: process: 0.77834 | load: 0.00309 [I 2022-10-08 15:38:25] ...trap/engines/engine.py.230: loss: 0.14806 [I 2022-10-08 15:38:25] ...optimizers/trijoint.py.151: total_norm: 2.671898 [I 2022-10-08 15:38:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 770/2304 [I 2022-10-08 15:38:33] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:52 [I 2022-10-08 15:38:33] ...trap/engines/engine.py.229: process: 0.76598 | load: 0.00311 [I 2022-10-08 15:38:33] ...trap/engines/engine.py.230: loss: 0.30042 [I 2022-10-08 15:38:33] ...optimizers/trijoint.py.151: total_norm: 4.084574 [I 2022-10-08 15:38:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 780/2304 [I 2022-10-08 15:38:41] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:57 [I 2022-10-08 15:38:41] ...trap/engines/engine.py.229: process: 0.78428 | load: 0.00310 [I 2022-10-08 15:38:41] ...trap/engines/engine.py.230: loss: 0.19220 [I 2022-10-08 15:38:41] ...optimizers/trijoint.py.151: total_norm: 2.137447 [I 2022-10-08 15:38:49] ...trap/engines/engine.py.224: train: epoch 10 | batch 790/2304 [I 2022-10-08 15:38:49] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:38 [I 2022-10-08 15:38:49] ...trap/engines/engine.py.229: process: 0.77617 | load: 0.00310 [I 2022-10-08 15:38:49] ...trap/engines/engine.py.230: loss: 0.21458 [I 2022-10-08 15:38:49] ...optimizers/trijoint.py.151: total_norm: 2.047486 [I 2022-10-08 15:38:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 800/2304 [I 2022-10-08 15:38:57] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:35 [I 2022-10-08 15:38:57] ...trap/engines/engine.py.229: process: 0.78755 | load: 0.00307 [I 2022-10-08 15:38:57] ...trap/engines/engine.py.230: loss: 0.16502 [I 2022-10-08 15:38:57] ...optimizers/trijoint.py.151: total_norm: 2.493225 [I 2022-10-08 15:39:05] ...trap/engines/engine.py.224: train: epoch 10 | batch 810/2304 [I 2022-10-08 15:39:05] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:32 [I 2022-10-08 15:39:05] ...trap/engines/engine.py.229: process: 0.79641 | load: 0.00305 [I 2022-10-08 15:39:05] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-08 15:39:05] ...optimizers/trijoint.py.151: total_norm: 2.567424 [I 2022-10-08 15:39:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 820/2304 [I 2022-10-08 15:39:12] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:21 [I 2022-10-08 15:39:12] ...trap/engines/engine.py.229: process: 0.78782 | load: 0.00304 [I 2022-10-08 15:39:12] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-08 15:39:12] ...optimizers/trijoint.py.151: total_norm: 2.667456 [I 2022-10-08 15:39:20] ...trap/engines/engine.py.224: train: epoch 10 | batch 830/2304 [I 2022-10-08 15:39:20] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:10 [I 2022-10-08 15:39:20] ...trap/engines/engine.py.229: process: 0.77764 | load: 0.00315 [I 2022-10-08 15:39:20] ...trap/engines/engine.py.230: loss: 0.11689 [I 2022-10-08 15:39:20] ...optimizers/trijoint.py.151: total_norm: 2.151942 [I 2022-10-08 15:39:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 840/2304 [I 2022-10-08 15:39:28] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:19:05 [I 2022-10-08 15:39:28] ...trap/engines/engine.py.229: process: 0.78264 | load: 0.00302 [I 2022-10-08 15:39:28] ...trap/engines/engine.py.230: loss: 0.18258 [I 2022-10-08 15:39:28] ...optimizers/trijoint.py.151: total_norm: 3.185526 [I 2022-10-08 15:39:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 850/2304 [I 2022-10-08 15:39:36] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:19:03 [I 2022-10-08 15:39:36] ...trap/engines/engine.py.229: process: 0.78518 | load: 0.00303 [I 2022-10-08 15:39:36] ...trap/engines/engine.py.230: loss: 0.20819 [I 2022-10-08 15:39:36] ...optimizers/trijoint.py.151: total_norm: 3.124439 [I 2022-10-08 15:39:44] ...trap/engines/engine.py.224: train: epoch 10 | batch 860/2304 [I 2022-10-08 15:39:44] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:44 [I 2022-10-08 15:39:44] ...trap/engines/engine.py.229: process: 0.77611 | load: 0.00304 [I 2022-10-08 15:39:44] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-08 15:39:44] ...optimizers/trijoint.py.151: total_norm: 2.549627 [I 2022-10-08 15:39:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 870/2304 [I 2022-10-08 15:39:52] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:33 [I 2022-10-08 15:39:52] ...trap/engines/engine.py.229: process: 0.77410 | load: 0.00300 [I 2022-10-08 15:39:52] ...trap/engines/engine.py.230: loss: 0.23008 [I 2022-10-08 15:39:52] ...optimizers/trijoint.py.151: total_norm: 2.906718 [I 2022-10-08 15:39:59] ...trap/engines/engine.py.224: train: epoch 10 | batch 880/2304 [I 2022-10-08 15:39:59] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:27 [I 2022-10-08 15:39:59] ...trap/engines/engine.py.229: process: 0.76711 | load: 0.00308 [I 2022-10-08 15:39:59] ...trap/engines/engine.py.230: loss: 0.25457 [I 2022-10-08 15:39:59] ...optimizers/trijoint.py.151: total_norm: 3.693660 [I 2022-10-08 15:40:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 890/2304 [I 2022-10-08 15:40:07] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:21 [I 2022-10-08 15:40:07] ...trap/engines/engine.py.229: process: 0.78973 | load: 0.00303 [I 2022-10-08 15:40:07] ...trap/engines/engine.py.230: loss: 0.20015 [I 2022-10-08 15:40:07] ...optimizers/trijoint.py.151: total_norm: 3.004711 [I 2022-10-08 15:40:15] ...trap/engines/engine.py.224: train: epoch 10 | batch 900/2304 [I 2022-10-08 15:40:15] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:14 [I 2022-10-08 15:40:15] ...trap/engines/engine.py.229: process: 0.78626 | load: 0.00302 [I 2022-10-08 15:40:15] ...trap/engines/engine.py.230: loss: 0.12209 [I 2022-10-08 15:40:15] ...optimizers/trijoint.py.151: total_norm: 2.064871 [I 2022-10-08 15:40:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 910/2304 [I 2022-10-08 15:40:23] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:18:09 [I 2022-10-08 15:40:23] ...trap/engines/engine.py.229: process: 0.78430 | load: 0.00302 [I 2022-10-08 15:40:23] ...trap/engines/engine.py.230: loss: 0.18995 [I 2022-10-08 15:40:23] ...optimizers/trijoint.py.151: total_norm: 2.770476 [I 2022-10-08 15:40:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 920/2304 [I 2022-10-08 15:40:31] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:17:58 [I 2022-10-08 15:40:31] ...trap/engines/engine.py.229: process: 0.78250 | load: 0.00302 [I 2022-10-08 15:40:31] ...trap/engines/engine.py.230: loss: 0.20328 [I 2022-10-08 15:40:31] ...optimizers/trijoint.py.151: total_norm: 3.210025 [I 2022-10-08 15:40:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 930/2304 [I 2022-10-08 15:40:38] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:52 [I 2022-10-08 15:40:38] ...trap/engines/engine.py.229: process: 0.79317 | load: 0.00303 [I 2022-10-08 15:40:38] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-08 15:40:38] ...optimizers/trijoint.py.151: total_norm: 1.979002 [I 2022-10-08 15:40:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 940/2304 [I 2022-10-08 15:40:46] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:43 [I 2022-10-08 15:40:46] ...trap/engines/engine.py.229: process: 0.78312 | load: 0.00304 [I 2022-10-08 15:40:46] ...trap/engines/engine.py.230: loss: 0.25989 [I 2022-10-08 15:40:46] ...optimizers/trijoint.py.151: total_norm: 3.328286 [I 2022-10-08 15:40:54] ...trap/engines/engine.py.224: train: epoch 10 | batch 950/2304 [I 2022-10-08 15:40:54] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:29 [I 2022-10-08 15:40:54] ...trap/engines/engine.py.229: process: 0.78032 | load: 0.00300 [I 2022-10-08 15:40:54] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-08 15:40:54] ...optimizers/trijoint.py.151: total_norm: 2.483240 [I 2022-10-08 15:41:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 960/2304 [I 2022-10-08 15:41:02] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:29 [I 2022-10-08 15:41:02] ...trap/engines/engine.py.229: process: 0.78339 | load: 0.00299 [I 2022-10-08 15:41:02] ...trap/engines/engine.py.230: loss: 0.36468 [I 2022-10-08 15:41:02] ...optimizers/trijoint.py.151: total_norm: 3.905441 [I 2022-10-08 15:41:10] ...trap/engines/engine.py.224: train: epoch 10 | batch 970/2304 [I 2022-10-08 15:41:10] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:20 [I 2022-10-08 15:41:10] ...trap/engines/engine.py.229: process: 0.78864 | load: 0.00304 [I 2022-10-08 15:41:10] ...trap/engines/engine.py.230: loss: 0.20760 [I 2022-10-08 15:41:10] ...optimizers/trijoint.py.151: total_norm: 2.982757 [I 2022-10-08 15:41:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 980/2304 [I 2022-10-08 15:41:17] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:05 [I 2022-10-08 15:41:17] ...trap/engines/engine.py.229: process: 0.77479 | load: 0.00305 [I 2022-10-08 15:41:17] ...trap/engines/engine.py.230: loss: 0.25205 [I 2022-10-08 15:41:17] ...optimizers/trijoint.py.151: total_norm: 3.321980 [I 2022-10-08 15:41:25] ...trap/engines/engine.py.224: train: epoch 10 | batch 990/2304 [I 2022-10-08 15:41:25] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:17:01 [I 2022-10-08 15:41:25] ...trap/engines/engine.py.229: process: 0.77419 | load: 0.00308 [I 2022-10-08 15:41:25] ...trap/engines/engine.py.230: loss: 0.20227 [I 2022-10-08 15:41:25] ...optimizers/trijoint.py.151: total_norm: 2.744593 [I 2022-10-08 15:41:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 1000/2304 [I 2022-10-08 15:41:33] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:16:52 [I 2022-10-08 15:41:33] ...trap/engines/engine.py.229: process: 0.78012 | load: 0.00301 [I 2022-10-08 15:41:33] ...trap/engines/engine.py.230: loss: 0.17806 [I 2022-10-08 15:41:33] ...optimizers/trijoint.py.151: total_norm: 2.570558 [I 2022-10-08 15:41:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1010/2304 [I 2022-10-08 15:41:41] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:55 [I 2022-10-08 15:41:41] ...trap/engines/engine.py.229: process: 0.78983 | load: 0.00302 [I 2022-10-08 15:41:41] ...trap/engines/engine.py.230: loss: 0.15018 [I 2022-10-08 15:41:41] ...optimizers/trijoint.py.151: total_norm: 2.544780 [I 2022-10-08 15:41:49] ...trap/engines/engine.py.224: train: epoch 10 | batch 1020/2304 [I 2022-10-08 15:41:49] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:43 [I 2022-10-08 15:41:49] ...trap/engines/engine.py.229: process: 0.77779 | load: 0.00302 [I 2022-10-08 15:41:49] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-08 15:41:49] ...optimizers/trijoint.py.151: total_norm: 2.373544 [I 2022-10-08 15:41:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1030/2304 [I 2022-10-08 15:41:56] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:39 [I 2022-10-08 15:41:56] ...trap/engines/engine.py.229: process: 0.77985 | load: 0.00302 [I 2022-10-08 15:41:56] ...trap/engines/engine.py.230: loss: 0.16647 [I 2022-10-08 15:41:56] ...optimizers/trijoint.py.151: total_norm: 2.238245 [I 2022-10-08 15:42:04] ...trap/engines/engine.py.224: train: epoch 10 | batch 1040/2304 [I 2022-10-08 15:42:04] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:30 [I 2022-10-08 15:42:04] ...trap/engines/engine.py.229: process: 0.77494 | load: 0.00306 [I 2022-10-08 15:42:04] ...trap/engines/engine.py.230: loss: 0.23954 [I 2022-10-08 15:42:04] ...optimizers/trijoint.py.151: total_norm: 2.685804 [I 2022-10-08 15:42:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 1050/2304 [I 2022-10-08 15:42:12] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:16:20 [I 2022-10-08 15:42:12] ...trap/engines/engine.py.229: process: 0.77080 | load: 0.00304 [I 2022-10-08 15:42:12] ...trap/engines/engine.py.230: loss: 0.11691 [I 2022-10-08 15:42:12] ...optimizers/trijoint.py.151: total_norm: 2.799168 [I 2022-10-08 15:42:20] ...trap/engines/engine.py.224: train: epoch 10 | batch 1060/2304 [I 2022-10-08 15:42:20] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:05 [I 2022-10-08 15:42:20] ...trap/engines/engine.py.229: process: 0.77262 | load: 0.00306 [I 2022-10-08 15:42:20] ...trap/engines/engine.py.230: loss: 0.21887 [I 2022-10-08 15:42:20] ...optimizers/trijoint.py.151: total_norm: 3.496781 [I 2022-10-08 15:42:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 1070/2304 [I 2022-10-08 15:42:28] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:16:03 [I 2022-10-08 15:42:28] ...trap/engines/engine.py.229: process: 0.78211 | load: 0.00303 [I 2022-10-08 15:42:28] ...trap/engines/engine.py.230: loss: 0.22001 [I 2022-10-08 15:42:28] ...optimizers/trijoint.py.151: total_norm: 2.961177 [I 2022-10-08 15:42:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 1080/2304 [I 2022-10-08 15:42:36] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:47 [I 2022-10-08 15:42:36] ...trap/engines/engine.py.229: process: 0.78116 | load: 0.00302 [I 2022-10-08 15:42:36] ...trap/engines/engine.py.230: loss: 0.19509 [I 2022-10-08 15:42:36] ...optimizers/trijoint.py.151: total_norm: 2.985294 [I 2022-10-08 15:42:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 1090/2304 [I 2022-10-08 15:42:43] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:42 [I 2022-10-08 15:42:43] ...trap/engines/engine.py.229: process: 0.77762 | load: 0.00306 [I 2022-10-08 15:42:43] ...trap/engines/engine.py.230: loss: 0.16180 [I 2022-10-08 15:42:43] ...optimizers/trijoint.py.151: total_norm: 1.773019 [I 2022-10-08 15:42:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1100/2304 [I 2022-10-08 15:42:51] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:37 [I 2022-10-08 15:42:51] ...trap/engines/engine.py.229: process: 0.78710 | load: 0.00303 [I 2022-10-08 15:42:51] ...trap/engines/engine.py.230: loss: 0.18212 [I 2022-10-08 15:42:51] ...optimizers/trijoint.py.151: total_norm: 2.715217 [I 2022-10-08 15:42:59] ...trap/engines/engine.py.224: train: epoch 10 | batch 1110/2304 [I 2022-10-08 15:42:59] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:15:35 [I 2022-10-08 15:42:59] ...trap/engines/engine.py.229: process: 0.78044 | load: 0.00305 [I 2022-10-08 15:42:59] ...trap/engines/engine.py.230: loss: 0.18825 [I 2022-10-08 15:42:59] ...optimizers/trijoint.py.151: total_norm: 2.848164 [I 2022-10-08 15:43:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 1120/2304 [I 2022-10-08 15:43:07] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:19 [I 2022-10-08 15:43:07] ...trap/engines/engine.py.229: process: 0.77871 | load: 0.00302 [I 2022-10-08 15:43:07] ...trap/engines/engine.py.230: loss: 0.20282 [I 2022-10-08 15:43:07] ...optimizers/trijoint.py.151: total_norm: 3.119943 [I 2022-10-08 15:43:15] ...trap/engines/engine.py.224: train: epoch 10 | batch 1130/2304 [I 2022-10-08 15:43:15] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:18 [I 2022-10-08 15:43:15] ...trap/engines/engine.py.229: process: 0.79379 | load: 0.00302 [I 2022-10-08 15:43:15] ...trap/engines/engine.py.230: loss: 0.18955 [I 2022-10-08 15:43:15] ...optimizers/trijoint.py.151: total_norm: 2.283784 [I 2022-10-08 15:43:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1140/2304 [I 2022-10-08 15:43:22] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:03 [I 2022-10-08 15:43:22] ...trap/engines/engine.py.229: process: 0.77861 | load: 0.00300 [I 2022-10-08 15:43:22] ...trap/engines/engine.py.230: loss: 0.21238 [I 2022-10-08 15:43:22] ...optimizers/trijoint.py.151: total_norm: 2.887356 [I 2022-10-08 15:43:30] ...trap/engines/engine.py.224: train: epoch 10 | batch 1150/2304 [I 2022-10-08 15:43:30] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:14:57 [I 2022-10-08 15:43:30] ...trap/engines/engine.py.229: process: 0.77688 | load: 0.00301 [I 2022-10-08 15:43:30] ...trap/engines/engine.py.230: loss: 0.22459 [I 2022-10-08 15:43:30] ...optimizers/trijoint.py.151: total_norm: 3.145114 [I 2022-10-08 15:43:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 1160/2304 [I 2022-10-08 15:43:38] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:14:50 [I 2022-10-08 15:43:38] ...trap/engines/engine.py.229: process: 0.76658 | load: 0.00305 [I 2022-10-08 15:43:38] ...trap/engines/engine.py.230: loss: 0.23673 [I 2022-10-08 15:43:38] ...optimizers/trijoint.py.151: total_norm: 2.747109 [I 2022-10-08 15:43:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 1170/2304 [I 2022-10-08 15:43:46] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:45 [I 2022-10-08 15:43:46] ...trap/engines/engine.py.229: process: 0.77908 | load: 0.00304 [I 2022-10-08 15:43:46] ...trap/engines/engine.py.230: loss: 0.22916 [I 2022-10-08 15:43:46] ...optimizers/trijoint.py.151: total_norm: 3.135888 [I 2022-10-08 15:43:54] ...trap/engines/engine.py.224: train: epoch 10 | batch 1180/2304 [I 2022-10-08 15:43:54] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:14:40 [I 2022-10-08 15:43:54] ...trap/engines/engine.py.229: process: 0.78795 | load: 0.00301 [I 2022-10-08 15:43:54] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-08 15:43:54] ...optimizers/trijoint.py.151: total_norm: 1.795163 [I 2022-10-08 15:44:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 1190/2304 [I 2022-10-08 15:44:02] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:14:33 [I 2022-10-08 15:44:02] ...trap/engines/engine.py.229: process: 0.78923 | load: 0.00301 [I 2022-10-08 15:44:02] ...trap/engines/engine.py.230: loss: 0.23198 [I 2022-10-08 15:44:02] ...optimizers/trijoint.py.151: total_norm: 2.946554 [I 2022-10-08 15:44:09] ...trap/engines/engine.py.224: train: epoch 10 | batch 1200/2304 [I 2022-10-08 15:44:09] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:21 [I 2022-10-08 15:44:09] ...trap/engines/engine.py.229: process: 0.77456 | load: 0.00303 [I 2022-10-08 15:44:09] ...trap/engines/engine.py.230: loss: 0.23941 [I 2022-10-08 15:44:09] ...optimizers/trijoint.py.151: total_norm: 2.776551 [I 2022-10-08 15:44:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 1210/2304 [I 2022-10-08 15:44:17] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:14:17 [I 2022-10-08 15:44:17] ...trap/engines/engine.py.229: process: 0.79245 | load: 0.00302 [I 2022-10-08 15:44:17] ...trap/engines/engine.py.230: loss: 0.22617 [I 2022-10-08 15:44:17] ...optimizers/trijoint.py.151: total_norm: 3.098981 [I 2022-10-08 15:44:25] ...trap/engines/engine.py.224: train: epoch 10 | batch 1220/2304 [I 2022-10-08 15:44:25] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:14:04 [I 2022-10-08 15:44:25] ...trap/engines/engine.py.229: process: 0.77737 | load: 0.00303 [I 2022-10-08 15:44:25] ...trap/engines/engine.py.230: loss: 0.14502 [I 2022-10-08 15:44:25] ...optimizers/trijoint.py.151: total_norm: 1.513322 [I 2022-10-08 15:44:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 1230/2304 [I 2022-10-08 15:44:33] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:13:57 [I 2022-10-08 15:44:33] ...trap/engines/engine.py.229: process: 0.78079 | load: 0.00305 [I 2022-10-08 15:44:33] ...trap/engines/engine.py.230: loss: 0.17972 [I 2022-10-08 15:44:33] ...optimizers/trijoint.py.151: total_norm: 2.059298 [I 2022-10-08 15:44:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1240/2304 [I 2022-10-08 15:44:41] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:13:51 [I 2022-10-08 15:44:41] ...trap/engines/engine.py.229: process: 0.79278 | load: 0.00306 [I 2022-10-08 15:44:41] ...trap/engines/engine.py.230: loss: 0.22980 [I 2022-10-08 15:44:41] ...optimizers/trijoint.py.151: total_norm: 3.955165 [I 2022-10-08 15:44:49] ...trap/engines/engine.py.224: train: epoch 10 | batch 1250/2304 [I 2022-10-08 15:44:49] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:13:43 [I 2022-10-08 15:44:49] ...trap/engines/engine.py.229: process: 0.77828 | load: 0.00306 [I 2022-10-08 15:44:49] ...trap/engines/engine.py.230: loss: 0.19318 [I 2022-10-08 15:44:49] ...optimizers/trijoint.py.151: total_norm: 3.329834 [I 2022-10-08 15:44:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1260/2304 [I 2022-10-08 15:44:56] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:34 [I 2022-10-08 15:44:56] ...trap/engines/engine.py.229: process: 0.77895 | load: 0.00300 [I 2022-10-08 15:44:56] ...trap/engines/engine.py.230: loss: 0.21666 [I 2022-10-08 15:44:56] ...optimizers/trijoint.py.151: total_norm: 2.843493 [I 2022-10-08 15:45:04] ...trap/engines/engine.py.224: train: epoch 10 | batch 1270/2304 [I 2022-10-08 15:45:04] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:13:21 [I 2022-10-08 15:45:04] ...trap/engines/engine.py.229: process: 0.75970 | load: 0.00303 [I 2022-10-08 15:45:04] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-08 15:45:04] ...optimizers/trijoint.py.151: total_norm: 2.193513 [I 2022-10-08 15:45:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 1280/2304 [I 2022-10-08 15:45:12] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:13:20 [I 2022-10-08 15:45:12] ...trap/engines/engine.py.229: process: 0.77919 | load: 0.00300 [I 2022-10-08 15:45:12] ...trap/engines/engine.py.230: loss: 0.25757 [I 2022-10-08 15:45:12] ...optimizers/trijoint.py.151: total_norm: 2.805006 [I 2022-10-08 15:45:20] ...trap/engines/engine.py.224: train: epoch 10 | batch 1290/2304 [I 2022-10-08 15:45:20] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:13:11 [I 2022-10-08 15:45:20] ...trap/engines/engine.py.229: process: 0.77066 | load: 0.00301 [I 2022-10-08 15:45:20] ...trap/engines/engine.py.230: loss: 0.21986 [I 2022-10-08 15:45:20] ...optimizers/trijoint.py.151: total_norm: 3.545747 [I 2022-10-08 15:45:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 1300/2304 [I 2022-10-08 15:45:28] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:13:01 [I 2022-10-08 15:45:28] ...trap/engines/engine.py.229: process: 0.76628 | load: 0.00302 [I 2022-10-08 15:45:28] ...trap/engines/engine.py.230: loss: 0.19486 [I 2022-10-08 15:45:28] ...optimizers/trijoint.py.151: total_norm: 3.003163 [I 2022-10-08 15:45:35] ...trap/engines/engine.py.224: train: epoch 10 | batch 1310/2304 [I 2022-10-08 15:45:35] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:53 [I 2022-10-08 15:45:35] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00304 [I 2022-10-08 15:45:35] ...trap/engines/engine.py.230: loss: 0.31166 [I 2022-10-08 15:45:35] ...optimizers/trijoint.py.151: total_norm: 3.824225 [I 2022-10-08 15:45:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 1320/2304 [I 2022-10-08 15:45:43] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:12:43 [I 2022-10-08 15:45:43] ...trap/engines/engine.py.229: process: 0.76724 | load: 0.00306 [I 2022-10-08 15:45:43] ...trap/engines/engine.py.230: loss: 0.18156 [I 2022-10-08 15:45:43] ...optimizers/trijoint.py.151: total_norm: 2.151181 [I 2022-10-08 15:45:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1330/2304 [I 2022-10-08 15:45:51] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:34 [I 2022-10-08 15:45:51] ...trap/engines/engine.py.229: process: 0.78303 | load: 0.00316 [I 2022-10-08 15:45:51] ...trap/engines/engine.py.230: loss: 0.25001 [I 2022-10-08 15:45:51] ...optimizers/trijoint.py.151: total_norm: 3.998990 [I 2022-10-08 15:45:59] ...trap/engines/engine.py.224: train: epoch 10 | batch 1340/2304 [I 2022-10-08 15:45:59] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:12:29 [I 2022-10-08 15:45:59] ...trap/engines/engine.py.229: process: 0.77632 | load: 0.00303 [I 2022-10-08 15:45:59] ...trap/engines/engine.py.230: loss: 0.28172 [I 2022-10-08 15:45:59] ...optimizers/trijoint.py.151: total_norm: 2.968573 [I 2022-10-08 15:46:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 1350/2304 [I 2022-10-08 15:46:07] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:12:23 [I 2022-10-08 15:46:07] ...trap/engines/engine.py.229: process: 0.77903 | load: 0.00304 [I 2022-10-08 15:46:07] ...trap/engines/engine.py.230: loss: 0.18992 [I 2022-10-08 15:46:07] ...optimizers/trijoint.py.151: total_norm: 3.381903 [I 2022-10-08 15:46:14] ...trap/engines/engine.py.224: train: epoch 10 | batch 1360/2304 [I 2022-10-08 15:46:14] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:15 [I 2022-10-08 15:46:14] ...trap/engines/engine.py.229: process: 0.77830 | load: 0.00306 [I 2022-10-08 15:46:14] ...trap/engines/engine.py.230: loss: 0.17245 [I 2022-10-08 15:46:14] ...optimizers/trijoint.py.151: total_norm: 2.152601 [I 2022-10-08 15:46:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1370/2304 [I 2022-10-08 15:46:22] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:09 [I 2022-10-08 15:46:22] ...trap/engines/engine.py.229: process: 0.77719 | load: 0.00304 [I 2022-10-08 15:46:22] ...trap/engines/engine.py.230: loss: 0.17442 [I 2022-10-08 15:46:22] ...optimizers/trijoint.py.151: total_norm: 2.151582 [I 2022-10-08 15:46:30] ...trap/engines/engine.py.224: train: epoch 10 | batch 1380/2304 [I 2022-10-08 15:46:30] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:03 [I 2022-10-08 15:46:30] ...trap/engines/engine.py.229: process: 0.78356 | load: 0.00304 [I 2022-10-08 15:46:30] ...trap/engines/engine.py.230: loss: 0.23063 [I 2022-10-08 15:46:30] ...optimizers/trijoint.py.151: total_norm: 3.284920 [I 2022-10-08 15:46:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 1390/2304 [I 2022-10-08 15:46:38] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:11:56 [I 2022-10-08 15:46:38] ...trap/engines/engine.py.229: process: 0.79206 | load: 0.00315 [I 2022-10-08 15:46:38] ...trap/engines/engine.py.230: loss: 0.14469 [I 2022-10-08 15:46:38] ...optimizers/trijoint.py.151: total_norm: 1.923974 [I 2022-10-08 15:46:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 1400/2304 [I 2022-10-08 15:46:46] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:11:47 [I 2022-10-08 15:46:46] ...trap/engines/engine.py.229: process: 0.78396 | load: 0.00307 [I 2022-10-08 15:46:46] ...trap/engines/engine.py.230: loss: 0.19467 [I 2022-10-08 15:46:46] ...optimizers/trijoint.py.151: total_norm: 3.170051 [I 2022-10-08 15:46:54] ...trap/engines/engine.py.224: train: epoch 10 | batch 1410/2304 [I 2022-10-08 15:46:54] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:11:41 [I 2022-10-08 15:46:54] ...trap/engines/engine.py.229: process: 0.79809 | load: 0.00304 [I 2022-10-08 15:46:54] ...trap/engines/engine.py.230: loss: 0.16933 [I 2022-10-08 15:46:54] ...optimizers/trijoint.py.151: total_norm: 2.179913 [I 2022-10-08 15:47:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1420/2304 [I 2022-10-08 15:47:01] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:33 [I 2022-10-08 15:47:01] ...trap/engines/engine.py.229: process: 0.79550 | load: 0.00317 [I 2022-10-08 15:47:01] ...trap/engines/engine.py.230: loss: 0.20476 [I 2022-10-08 15:47:01] ...optimizers/trijoint.py.151: total_norm: 2.647028 [I 2022-10-08 15:47:09] ...trap/engines/engine.py.224: train: epoch 10 | batch 1430/2304 [I 2022-10-08 15:47:09] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:21 [I 2022-10-08 15:47:09] ...trap/engines/engine.py.229: process: 0.79093 | load: 0.00314 [I 2022-10-08 15:47:09] ...trap/engines/engine.py.230: loss: 0.18767 [I 2022-10-08 15:47:09] ...optimizers/trijoint.py.151: total_norm: 2.518433 [I 2022-10-08 15:47:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 1440/2304 [I 2022-10-08 15:47:17] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:15 [I 2022-10-08 15:47:17] ...trap/engines/engine.py.229: process: 0.78380 | load: 0.00308 [I 2022-10-08 15:47:17] ...trap/engines/engine.py.230: loss: 0.28865 [I 2022-10-08 15:47:17] ...optimizers/trijoint.py.151: total_norm: 3.270809 [I 2022-10-08 15:47:25] ...trap/engines/engine.py.224: train: epoch 10 | batch 1450/2304 [I 2022-10-08 15:47:25] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:04 [I 2022-10-08 15:47:25] ...trap/engines/engine.py.229: process: 0.77578 | load: 0.00302 [I 2022-10-08 15:47:25] ...trap/engines/engine.py.230: loss: 0.23209 [I 2022-10-08 15:47:25] ...optimizers/trijoint.py.151: total_norm: 2.817770 [I 2022-10-08 15:47:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 1460/2304 [I 2022-10-08 15:47:33] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:11:01 [I 2022-10-08 15:47:33] ...trap/engines/engine.py.229: process: 0.78744 | load: 0.00304 [I 2022-10-08 15:47:33] ...trap/engines/engine.py.230: loss: 0.21230 [I 2022-10-08 15:47:33] ...optimizers/trijoint.py.151: total_norm: 2.637246 [I 2022-10-08 15:47:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1470/2304 [I 2022-10-08 15:47:41] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:10:50 [I 2022-10-08 15:47:41] ...trap/engines/engine.py.229: process: 0.77028 | load: 0.00302 [I 2022-10-08 15:47:41] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-08 15:47:41] ...optimizers/trijoint.py.151: total_norm: 3.361795 [I 2022-10-08 15:47:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 1480/2304 [I 2022-10-08 15:47:48] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:44 [I 2022-10-08 15:47:48] ...trap/engines/engine.py.229: process: 0.79563 | load: 0.00303 [I 2022-10-08 15:47:48] ...trap/engines/engine.py.230: loss: 0.26016 [I 2022-10-08 15:47:48] ...optimizers/trijoint.py.151: total_norm: 2.637506 [I 2022-10-08 15:47:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1490/2304 [I 2022-10-08 15:47:56] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:34 [I 2022-10-08 15:47:56] ...trap/engines/engine.py.229: process: 0.78164 | load: 0.00302 [I 2022-10-08 15:47:56] ...trap/engines/engine.py.230: loss: 0.22591 [I 2022-10-08 15:47:56] ...optimizers/trijoint.py.151: total_norm: 2.607427 [I 2022-10-08 15:48:04] ...trap/engines/engine.py.224: train: epoch 10 | batch 1500/2304 [I 2022-10-08 15:48:04] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:30 [I 2022-10-08 15:48:04] ...trap/engines/engine.py.229: process: 0.79767 | load: 0.00306 [I 2022-10-08 15:48:04] ...trap/engines/engine.py.230: loss: 0.25750 [I 2022-10-08 15:48:04] ...optimizers/trijoint.py.151: total_norm: 2.575185 [I 2022-10-08 15:48:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 1510/2304 [I 2022-10-08 15:48:12] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:17 [I 2022-10-08 15:48:12] ...trap/engines/engine.py.229: process: 0.77947 | load: 0.00306 [I 2022-10-08 15:48:12] ...trap/engines/engine.py.230: loss: 0.23142 [I 2022-10-08 15:48:12] ...optimizers/trijoint.py.151: total_norm: 3.349837 [I 2022-10-08 15:48:20] ...trap/engines/engine.py.224: train: epoch 10 | batch 1520/2304 [I 2022-10-08 15:48:20] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:11 [I 2022-10-08 15:48:20] ...trap/engines/engine.py.229: process: 0.77491 | load: 0.00301 [I 2022-10-08 15:48:20] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-08 15:48:20] ...optimizers/trijoint.py.151: total_norm: 2.386172 [I 2022-10-08 15:48:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 1530/2304 [I 2022-10-08 15:48:27] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:03 [I 2022-10-08 15:48:27] ...trap/engines/engine.py.229: process: 0.78201 | load: 0.00304 [I 2022-10-08 15:48:27] ...trap/engines/engine.py.230: loss: 0.26776 [I 2022-10-08 15:48:27] ...optimizers/trijoint.py.151: total_norm: 2.678684 [I 2022-10-08 15:48:35] ...trap/engines/engine.py.224: train: epoch 10 | batch 1540/2304 [I 2022-10-08 15:48:35] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:52 [I 2022-10-08 15:48:35] ...trap/engines/engine.py.229: process: 0.77134 | load: 0.00312 [I 2022-10-08 15:48:35] ...trap/engines/engine.py.230: loss: 0.19003 [I 2022-10-08 15:48:35] ...optimizers/trijoint.py.151: total_norm: 2.632378 [I 2022-10-08 15:48:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 1550/2304 [I 2022-10-08 15:48:43] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:47 [I 2022-10-08 15:48:43] ...trap/engines/engine.py.229: process: 0.78269 | load: 0.00311 [I 2022-10-08 15:48:43] ...trap/engines/engine.py.230: loss: 0.18643 [I 2022-10-08 15:48:43] ...optimizers/trijoint.py.151: total_norm: 2.223529 [I 2022-10-08 15:48:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1560/2304 [I 2022-10-08 15:48:51] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:37 [I 2022-10-08 15:48:51] ...trap/engines/engine.py.229: process: 0.77727 | load: 0.00300 [I 2022-10-08 15:48:51] ...trap/engines/engine.py.230: loss: 0.22159 [I 2022-10-08 15:48:51] ...optimizers/trijoint.py.151: total_norm: 2.920182 [I 2022-10-08 15:48:59] ...trap/engines/engine.py.224: train: epoch 10 | batch 1570/2304 [I 2022-10-08 15:48:59] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:29 [I 2022-10-08 15:48:59] ...trap/engines/engine.py.229: process: 0.77259 | load: 0.00299 [I 2022-10-08 15:48:59] ...trap/engines/engine.py.230: loss: 0.17191 [I 2022-10-08 15:48:59] ...optimizers/trijoint.py.151: total_norm: 2.617073 [I 2022-10-08 15:49:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1580/2304 [I 2022-10-08 15:49:06] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:22 [I 2022-10-08 15:49:06] ...trap/engines/engine.py.229: process: 0.76913 | load: 0.00307 [I 2022-10-08 15:49:06] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-08 15:49:06] ...optimizers/trijoint.py.151: total_norm: 2.359459 [I 2022-10-08 15:49:14] ...trap/engines/engine.py.224: train: epoch 10 | batch 1590/2304 [I 2022-10-08 15:49:14] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:16 [I 2022-10-08 15:49:14] ...trap/engines/engine.py.229: process: 0.78914 | load: 0.00313 [I 2022-10-08 15:49:14] ...trap/engines/engine.py.230: loss: 0.17155 [I 2022-10-08 15:49:14] ...optimizers/trijoint.py.151: total_norm: 2.769281 [I 2022-10-08 15:49:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1600/2304 [I 2022-10-08 15:49:22] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:11 [I 2022-10-08 15:49:22] ...trap/engines/engine.py.229: process: 0.78823 | load: 0.00311 [I 2022-10-08 15:49:22] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-08 15:49:22] ...optimizers/trijoint.py.151: total_norm: 2.466797 [I 2022-10-08 15:49:30] ...trap/engines/engine.py.224: train: epoch 10 | batch 1610/2304 [I 2022-10-08 15:49:30] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:09:01 [I 2022-10-08 15:49:30] ...trap/engines/engine.py.229: process: 0.78428 | load: 0.00315 [I 2022-10-08 15:49:30] ...trap/engines/engine.py.230: loss: 0.20514 [I 2022-10-08 15:49:30] ...optimizers/trijoint.py.151: total_norm: 2.368830 [I 2022-10-08 15:49:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 1620/2304 [I 2022-10-08 15:49:38] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:08:54 [I 2022-10-08 15:49:38] ...trap/engines/engine.py.229: process: 0.78102 | load: 0.00313 [I 2022-10-08 15:49:38] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-08 15:49:38] ...optimizers/trijoint.py.151: total_norm: 1.888564 [I 2022-10-08 15:49:45] ...trap/engines/engine.py.224: train: epoch 10 | batch 1630/2304 [I 2022-10-08 15:49:45] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:44 [I 2022-10-08 15:49:45] ...trap/engines/engine.py.229: process: 0.77692 | load: 0.00303 [I 2022-10-08 15:49:45] ...trap/engines/engine.py.230: loss: 0.17489 [I 2022-10-08 15:49:45] ...optimizers/trijoint.py.151: total_norm: 2.283427 [I 2022-10-08 15:49:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 1640/2304 [I 2022-10-08 15:49:53] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:37 [I 2022-10-08 15:49:53] ...trap/engines/engine.py.229: process: 0.77348 | load: 0.00301 [I 2022-10-08 15:49:53] ...trap/engines/engine.py.230: loss: 0.18564 [I 2022-10-08 15:49:53] ...optimizers/trijoint.py.151: total_norm: 2.398454 [I 2022-10-08 15:50:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1650/2304 [I 2022-10-08 15:50:01] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:30 [I 2022-10-08 15:50:01] ...trap/engines/engine.py.229: process: 0.79280 | load: 0.00303 [I 2022-10-08 15:50:01] ...trap/engines/engine.py.230: loss: 0.20708 [I 2022-10-08 15:50:01] ...optimizers/trijoint.py.151: total_norm: 2.624409 [I 2022-10-08 15:50:09] ...trap/engines/engine.py.224: train: epoch 10 | batch 1660/2304 [I 2022-10-08 15:50:09] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:18 [I 2022-10-08 15:50:09] ...trap/engines/engine.py.229: process: 0.76807 | load: 0.00302 [I 2022-10-08 15:50:09] ...trap/engines/engine.py.230: loss: 0.17546 [I 2022-10-08 15:50:09] ...optimizers/trijoint.py.151: total_norm: 2.689933 [I 2022-10-08 15:50:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 1670/2304 [I 2022-10-08 15:50:17] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:12 [I 2022-10-08 15:50:17] ...trap/engines/engine.py.229: process: 0.77814 | load: 0.00308 [I 2022-10-08 15:50:17] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-08 15:50:17] ...optimizers/trijoint.py.151: total_norm: 2.395252 [I 2022-10-08 15:50:24] ...trap/engines/engine.py.224: train: epoch 10 | batch 1680/2304 [I 2022-10-08 15:50:24] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:05 [I 2022-10-08 15:50:24] ...trap/engines/engine.py.229: process: 0.78125 | load: 0.00307 [I 2022-10-08 15:50:24] ...trap/engines/engine.py.230: loss: 0.19862 [I 2022-10-08 15:50:24] ...optimizers/trijoint.py.151: total_norm: 2.331999 [I 2022-10-08 15:50:32] ...trap/engines/engine.py.224: train: epoch 10 | batch 1690/2304 [I 2022-10-08 15:50:32] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:07:59 [I 2022-10-08 15:50:32] ...trap/engines/engine.py.229: process: 0.77489 | load: 0.00305 [I 2022-10-08 15:50:32] ...trap/engines/engine.py.230: loss: 0.17802 [I 2022-10-08 15:50:32] ...optimizers/trijoint.py.151: total_norm: 2.528954 [I 2022-10-08 15:50:40] ...trap/engines/engine.py.224: train: epoch 10 | batch 1700/2304 [I 2022-10-08 15:50:40] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:49 [I 2022-10-08 15:50:40] ...trap/engines/engine.py.229: process: 0.78008 | load: 0.00305 [I 2022-10-08 15:50:40] ...trap/engines/engine.py.230: loss: 0.13650 [I 2022-10-08 15:50:40] ...optimizers/trijoint.py.151: total_norm: 2.253280 [I 2022-10-08 15:50:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 1710/2304 [I 2022-10-08 15:50:48] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:40 [I 2022-10-08 15:50:48] ...trap/engines/engine.py.229: process: 0.76833 | load: 0.00304 [I 2022-10-08 15:50:48] ...trap/engines/engine.py.230: loss: 0.19435 [I 2022-10-08 15:50:48] ...optimizers/trijoint.py.151: total_norm: 2.464317 [I 2022-10-08 15:50:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1720/2304 [I 2022-10-08 15:50:56] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:07:35 [I 2022-10-08 15:50:56] ...trap/engines/engine.py.229: process: 0.77566 | load: 0.00307 [I 2022-10-08 15:50:56] ...trap/engines/engine.py.230: loss: 0.14831 [I 2022-10-08 15:50:56] ...optimizers/trijoint.py.151: total_norm: 2.195788 [I 2022-10-08 15:51:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 1730/2304 [I 2022-10-08 15:51:03] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:25 [I 2022-10-08 15:51:03] ...trap/engines/engine.py.229: process: 0.77728 | load: 0.00301 [I 2022-10-08 15:51:03] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-08 15:51:03] ...optimizers/trijoint.py.151: total_norm: 1.337291 [I 2022-10-08 15:51:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 1740/2304 [I 2022-10-08 15:51:11] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:19 [I 2022-10-08 15:51:11] ...trap/engines/engine.py.229: process: 0.77953 | load: 0.00307 [I 2022-10-08 15:51:11] ...trap/engines/engine.py.230: loss: 0.19447 [I 2022-10-08 15:51:11] ...optimizers/trijoint.py.151: total_norm: 2.685709 [I 2022-10-08 15:51:19] ...trap/engines/engine.py.224: train: epoch 10 | batch 1750/2304 [I 2022-10-08 15:51:19] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:10 [I 2022-10-08 15:51:19] ...trap/engines/engine.py.229: process: 0.77839 | load: 0.00303 [I 2022-10-08 15:51:19] ...trap/engines/engine.py.230: loss: 0.14308 [I 2022-10-08 15:51:19] ...optimizers/trijoint.py.151: total_norm: 2.069070 [I 2022-10-08 15:51:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 1760/2304 [I 2022-10-08 15:51:27] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:01 [I 2022-10-08 15:51:27] ...trap/engines/engine.py.229: process: 0.76450 | load: 0.00302 [I 2022-10-08 15:51:27] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-08 15:51:27] ...optimizers/trijoint.py.151: total_norm: 1.946905 [I 2022-10-08 15:51:34] ...trap/engines/engine.py.224: train: epoch 10 | batch 1770/2304 [I 2022-10-08 15:51:34] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:55 [I 2022-10-08 15:51:34] ...trap/engines/engine.py.229: process: 0.77662 | load: 0.00301 [I 2022-10-08 15:51:34] ...trap/engines/engine.py.230: loss: 0.34532 [I 2022-10-08 15:51:34] ...optimizers/trijoint.py.151: total_norm: 4.915660 [I 2022-10-08 15:51:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 1780/2304 [I 2022-10-08 15:51:42] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:44 [I 2022-10-08 15:51:42] ...trap/engines/engine.py.229: process: 0.77556 | load: 0.00304 [I 2022-10-08 15:51:42] ...trap/engines/engine.py.230: loss: 0.13271 [I 2022-10-08 15:51:42] ...optimizers/trijoint.py.151: total_norm: 1.496435 [I 2022-10-08 15:51:50] ...trap/engines/engine.py.224: train: epoch 10 | batch 1790/2304 [I 2022-10-08 15:51:50] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:37 [I 2022-10-08 15:51:50] ...trap/engines/engine.py.229: process: 0.76507 | load: 0.00311 [I 2022-10-08 15:51:50] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-08 15:51:50] ...optimizers/trijoint.py.151: total_norm: 2.692749 [I 2022-10-08 15:51:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 1800/2304 [I 2022-10-08 15:51:58] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:31 [I 2022-10-08 15:51:58] ...trap/engines/engine.py.229: process: 0.78305 | load: 0.00302 [I 2022-10-08 15:51:58] ...trap/engines/engine.py.230: loss: 0.21230 [I 2022-10-08 15:51:58] ...optimizers/trijoint.py.151: total_norm: 2.770842 [I 2022-10-08 15:52:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1810/2304 [I 2022-10-08 15:52:06] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:24 [I 2022-10-08 15:52:06] ...trap/engines/engine.py.229: process: 0.78132 | load: 0.00303 [I 2022-10-08 15:52:06] ...trap/engines/engine.py.230: loss: 0.26229 [I 2022-10-08 15:52:06] ...optimizers/trijoint.py.151: total_norm: 2.996071 [I 2022-10-08 15:52:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 1820/2304 [I 2022-10-08 15:52:13] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:06:15 [I 2022-10-08 15:52:13] ...trap/engines/engine.py.229: process: 0.78273 | load: 0.00301 [I 2022-10-08 15:52:13] ...trap/engines/engine.py.230: loss: 0.18703 [I 2022-10-08 15:52:13] ...optimizers/trijoint.py.151: total_norm: 3.251871 [I 2022-10-08 15:52:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 1830/2304 [I 2022-10-08 15:52:21] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:08 [I 2022-10-08 15:52:21] ...trap/engines/engine.py.229: process: 0.78288 | load: 0.00305 [I 2022-10-08 15:52:21] ...trap/engines/engine.py.230: loss: 0.19101 [I 2022-10-08 15:52:21] ...optimizers/trijoint.py.151: total_norm: 2.677978 [I 2022-10-08 15:52:29] ...trap/engines/engine.py.224: train: epoch 10 | batch 1840/2304 [I 2022-10-08 15:52:29] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:01 [I 2022-10-08 15:52:29] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00304 [I 2022-10-08 15:52:29] ...trap/engines/engine.py.230: loss: 0.18665 [I 2022-10-08 15:52:29] ...optimizers/trijoint.py.151: total_norm: 2.325601 [I 2022-10-08 15:52:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 1850/2304 [I 2022-10-08 15:52:37] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:05:54 [I 2022-10-08 15:52:37] ...trap/engines/engine.py.229: process: 0.78508 | load: 0.00304 [I 2022-10-08 15:52:37] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-08 15:52:37] ...optimizers/trijoint.py.151: total_norm: 2.431386 [I 2022-10-08 15:52:45] ...trap/engines/engine.py.224: train: epoch 10 | batch 1860/2304 [I 2022-10-08 15:52:45] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:44 [I 2022-10-08 15:52:45] ...trap/engines/engine.py.229: process: 0.76941 | load: 0.00302 [I 2022-10-08 15:52:45] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-08 15:52:45] ...optimizers/trijoint.py.151: total_norm: 2.026847 [I 2022-10-08 15:52:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 1870/2304 [I 2022-10-08 15:52:52] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:37 [I 2022-10-08 15:52:52] ...trap/engines/engine.py.229: process: 0.78495 | load: 0.00303 [I 2022-10-08 15:52:52] ...trap/engines/engine.py.230: loss: 0.27090 [I 2022-10-08 15:52:52] ...optimizers/trijoint.py.151: total_norm: 3.037113 [I 2022-10-08 15:53:00] ...trap/engines/engine.py.224: train: epoch 10 | batch 1880/2304 [I 2022-10-08 15:53:00] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:05:30 [I 2022-10-08 15:53:00] ...trap/engines/engine.py.229: process: 0.77993 | load: 0.00300 [I 2022-10-08 15:53:00] ...trap/engines/engine.py.230: loss: 0.15223 [I 2022-10-08 15:53:00] ...optimizers/trijoint.py.151: total_norm: 2.449542 [I 2022-10-08 15:53:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 1890/2304 [I 2022-10-08 15:53:08] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:05:22 [I 2022-10-08 15:53:08] ...trap/engines/engine.py.229: process: 0.78504 | load: 0.00310 [I 2022-10-08 15:53:08] ...trap/engines/engine.py.230: loss: 0.14215 [I 2022-10-08 15:53:08] ...optimizers/trijoint.py.151: total_norm: 1.908575 [I 2022-10-08 15:53:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 1900/2304 [I 2022-10-08 15:53:16] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:13 [I 2022-10-08 15:53:16] ...trap/engines/engine.py.229: process: 0.77427 | load: 0.00312 [I 2022-10-08 15:53:16] ...trap/engines/engine.py.230: loss: 0.18005 [I 2022-10-08 15:53:16] ...optimizers/trijoint.py.151: total_norm: 1.907146 [I 2022-10-08 15:53:24] ...trap/engines/engine.py.224: train: epoch 10 | batch 1910/2304 [I 2022-10-08 15:53:24] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:05:06 [I 2022-10-08 15:53:24] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00312 [I 2022-10-08 15:53:24] ...trap/engines/engine.py.230: loss: 0.20706 [I 2022-10-08 15:53:24] ...optimizers/trijoint.py.151: total_norm: 3.379085 [I 2022-10-08 15:53:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 1920/2304 [I 2022-10-08 15:53:31] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:04:56 [I 2022-10-08 15:53:31] ...trap/engines/engine.py.229: process: 0.76827 | load: 0.00301 [I 2022-10-08 15:53:31] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-08 15:53:31] ...optimizers/trijoint.py.151: total_norm: 1.997651 [I 2022-10-08 15:53:39] ...trap/engines/engine.py.224: train: epoch 10 | batch 1930/2304 [I 2022-10-08 15:53:39] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:04:50 [I 2022-10-08 15:53:39] ...trap/engines/engine.py.229: process: 0.77857 | load: 0.00300 [I 2022-10-08 15:53:39] ...trap/engines/engine.py.230: loss: 0.20091 [I 2022-10-08 15:53:39] ...optimizers/trijoint.py.151: total_norm: 2.357250 [I 2022-10-08 15:53:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 1940/2304 [I 2022-10-08 15:53:47] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:04:43 [I 2022-10-08 15:53:47] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00304 [I 2022-10-08 15:53:47] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-08 15:53:47] ...optimizers/trijoint.py.151: total_norm: 2.767958 [I 2022-10-08 15:53:55] ...trap/engines/engine.py.224: train: epoch 10 | batch 1950/2304 [I 2022-10-08 15:53:55] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:34 [I 2022-10-08 15:53:55] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00307 [I 2022-10-08 15:53:55] ...trap/engines/engine.py.230: loss: 0.17586 [I 2022-10-08 15:53:55] ...optimizers/trijoint.py.151: total_norm: 2.658793 [I 2022-10-08 15:54:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 1960/2304 [I 2022-10-08 15:54:02] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:04:27 [I 2022-10-08 15:54:02] ...trap/engines/engine.py.229: process: 0.78220 | load: 0.00303 [I 2022-10-08 15:54:02] ...trap/engines/engine.py.230: loss: 0.19486 [I 2022-10-08 15:54:02] ...optimizers/trijoint.py.151: total_norm: 3.266962 [I 2022-10-08 15:54:10] ...trap/engines/engine.py.224: train: epoch 10 | batch 1970/2304 [I 2022-10-08 15:54:10] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:04:18 [I 2022-10-08 15:54:10] ...trap/engines/engine.py.229: process: 0.77940 | load: 0.00301 [I 2022-10-08 15:54:10] ...trap/engines/engine.py.230: loss: 0.20618 [I 2022-10-08 15:54:10] ...optimizers/trijoint.py.151: total_norm: 2.901644 [I 2022-10-08 15:54:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 1980/2304 [I 2022-10-08 15:54:18] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:04:10 [I 2022-10-08 15:54:18] ...trap/engines/engine.py.229: process: 0.76870 | load: 0.00301 [I 2022-10-08 15:54:18] ...trap/engines/engine.py.230: loss: 0.16387 [I 2022-10-08 15:54:18] ...optimizers/trijoint.py.151: total_norm: 2.390352 [I 2022-10-08 15:54:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 1990/2304 [I 2022-10-08 15:54:26] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:04 [I 2022-10-08 15:54:26] ...trap/engines/engine.py.229: process: 0.77546 | load: 0.00301 [I 2022-10-08 15:54:26] ...trap/engines/engine.py.230: loss: 0.19958 [I 2022-10-08 15:54:26] ...optimizers/trijoint.py.151: total_norm: 2.213892 [I 2022-10-08 15:54:34] ...trap/engines/engine.py.224: train: epoch 10 | batch 2000/2304 [I 2022-10-08 15:54:34] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:03:56 [I 2022-10-08 15:54:34] ...trap/engines/engine.py.229: process: 0.77923 | load: 0.00304 [I 2022-10-08 15:54:34] ...trap/engines/engine.py.230: loss: 0.17939 [I 2022-10-08 15:54:34] ...optimizers/trijoint.py.151: total_norm: 3.345537 [I 2022-10-08 15:54:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 2010/2304 [I 2022-10-08 15:54:41] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:03:48 [I 2022-10-08 15:54:41] ...trap/engines/engine.py.229: process: 0.77706 | load: 0.00302 [I 2022-10-08 15:54:41] ...trap/engines/engine.py.230: loss: 0.22932 [I 2022-10-08 15:54:41] ...optimizers/trijoint.py.151: total_norm: 2.551657 [I 2022-10-08 15:54:49] ...trap/engines/engine.py.224: train: epoch 10 | batch 2020/2304 [I 2022-10-08 15:54:49] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:03:41 [I 2022-10-08 15:54:49] ...trap/engines/engine.py.229: process: 0.77727 | load: 0.00299 [I 2022-10-08 15:54:49] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-08 15:54:49] ...optimizers/trijoint.py.151: total_norm: 3.169660 [I 2022-10-08 15:54:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 2030/2304 [I 2022-10-08 15:54:57] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:03:33 [I 2022-10-08 15:54:57] ...trap/engines/engine.py.229: process: 0.77865 | load: 0.00309 [I 2022-10-08 15:54:57] ...trap/engines/engine.py.230: loss: 0.19256 [I 2022-10-08 15:54:57] ...optimizers/trijoint.py.151: total_norm: 2.397420 [I 2022-10-08 15:55:05] ...trap/engines/engine.py.224: train: epoch 10 | batch 2040/2304 [I 2022-10-08 15:55:05] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:03:25 [I 2022-10-08 15:55:05] ...trap/engines/engine.py.229: process: 0.77056 | load: 0.00303 [I 2022-10-08 15:55:05] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-08 15:55:05] ...optimizers/trijoint.py.151: total_norm: 2.643941 [I 2022-10-08 15:55:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 2050/2304 [I 2022-10-08 15:55:13] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:03:18 [I 2022-10-08 15:55:13] ...trap/engines/engine.py.229: process: 0.79057 | load: 0.00310 [I 2022-10-08 15:55:13] ...trap/engines/engine.py.230: loss: 0.17228 [I 2022-10-08 15:55:13] ...optimizers/trijoint.py.151: total_norm: 2.127124 [I 2022-10-08 15:55:20] ...trap/engines/engine.py.224: train: epoch 10 | batch 2060/2304 [I 2022-10-08 15:55:20] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:03:09 [I 2022-10-08 15:55:20] ...trap/engines/engine.py.229: process: 0.78063 | load: 0.00305 [I 2022-10-08 15:55:20] ...trap/engines/engine.py.230: loss: 0.24627 [I 2022-10-08 15:55:20] ...optimizers/trijoint.py.151: total_norm: 3.532272 [I 2022-10-08 15:55:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 2070/2304 [I 2022-10-08 15:55:28] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:03:02 [I 2022-10-08 15:55:28] ...trap/engines/engine.py.229: process: 0.78040 | load: 0.00302 [I 2022-10-08 15:55:28] ...trap/engines/engine.py.230: loss: 0.16733 [I 2022-10-08 15:55:28] ...optimizers/trijoint.py.151: total_norm: 2.675197 [I 2022-10-08 15:55:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 2080/2304 [I 2022-10-08 15:55:36] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:02:54 [I 2022-10-08 15:55:36] ...trap/engines/engine.py.229: process: 0.76753 | load: 0.00305 [I 2022-10-08 15:55:36] ...trap/engines/engine.py.230: loss: 0.18458 [I 2022-10-08 15:55:36] ...optimizers/trijoint.py.151: total_norm: 2.720570 [I 2022-10-08 15:55:44] ...trap/engines/engine.py.224: train: epoch 10 | batch 2090/2304 [I 2022-10-08 15:55:44] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:02:47 [I 2022-10-08 15:55:44] ...trap/engines/engine.py.229: process: 0.78248 | load: 0.00306 [I 2022-10-08 15:55:44] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-08 15:55:44] ...optimizers/trijoint.py.151: total_norm: 2.712895 [I 2022-10-08 15:55:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 2100/2304 [I 2022-10-08 15:55:52] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:38 [I 2022-10-08 15:55:52] ...trap/engines/engine.py.229: process: 0.77244 | load: 0.00301 [I 2022-10-08 15:55:52] ...trap/engines/engine.py.230: loss: 0.20448 [I 2022-10-08 15:55:52] ...optimizers/trijoint.py.151: total_norm: 2.318291 [I 2022-10-08 15:55:59] ...trap/engines/engine.py.224: train: epoch 10 | batch 2110/2304 [I 2022-10-08 15:55:59] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:02:29 [I 2022-10-08 15:55:59] ...trap/engines/engine.py.229: process: 0.77376 | load: 0.00299 [I 2022-10-08 15:55:59] ...trap/engines/engine.py.230: loss: 0.28077 [I 2022-10-08 15:55:59] ...optimizers/trijoint.py.151: total_norm: 2.895934 [I 2022-10-08 15:56:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 2120/2304 [I 2022-10-08 15:56:07] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:02:22 [I 2022-10-08 15:56:07] ...trap/engines/engine.py.229: process: 0.76210 | load: 0.00301 [I 2022-10-08 15:56:07] ...trap/engines/engine.py.230: loss: 0.21719 [I 2022-10-08 15:56:07] ...optimizers/trijoint.py.151: total_norm: 2.698737 [I 2022-10-08 15:56:15] ...trap/engines/engine.py.224: train: epoch 10 | batch 2130/2304 [I 2022-10-08 15:56:15] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:02:15 [I 2022-10-08 15:56:15] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00301 [I 2022-10-08 15:56:15] ...trap/engines/engine.py.230: loss: 0.22157 [I 2022-10-08 15:56:15] ...optimizers/trijoint.py.151: total_norm: 2.574926 [I 2022-10-08 15:56:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 2140/2304 [I 2022-10-08 15:56:23] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:02:08 [I 2022-10-08 15:56:23] ...trap/engines/engine.py.229: process: 0.78501 | load: 0.00304 [I 2022-10-08 15:56:23] ...trap/engines/engine.py.230: loss: 0.16178 [I 2022-10-08 15:56:23] ...optimizers/trijoint.py.151: total_norm: 2.423467 [I 2022-10-08 15:56:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 2150/2304 [I 2022-10-08 15:56:31] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:01:59 [I 2022-10-08 15:56:31] ...trap/engines/engine.py.229: process: 0.76633 | load: 0.00303 [I 2022-10-08 15:56:31] ...trap/engines/engine.py.230: loss: 0.22887 [I 2022-10-08 15:56:31] ...optimizers/trijoint.py.151: total_norm: 2.949986 [I 2022-10-08 15:56:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 2160/2304 [I 2022-10-08 15:56:38] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:01:51 [I 2022-10-08 15:56:38] ...trap/engines/engine.py.229: process: 0.77492 | load: 0.00304 [I 2022-10-08 15:56:38] ...trap/engines/engine.py.230: loss: 0.22048 [I 2022-10-08 15:56:38] ...optimizers/trijoint.py.151: total_norm: 2.644901 [I 2022-10-08 15:56:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 2170/2304 [I 2022-10-08 15:56:46] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:01:43 [I 2022-10-08 15:56:46] ...trap/engines/engine.py.229: process: 0.77129 | load: 0.00308 [I 2022-10-08 15:56:46] ...trap/engines/engine.py.230: loss: 0.24607 [I 2022-10-08 15:56:46] ...optimizers/trijoint.py.151: total_norm: 2.699894 [I 2022-10-08 15:56:54] ...trap/engines/engine.py.224: train: epoch 10 | batch 2180/2304 [I 2022-10-08 15:56:54] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:01:36 [I 2022-10-08 15:56:54] ...trap/engines/engine.py.229: process: 0.78118 | load: 0.00307 [I 2022-10-08 15:56:54] ...trap/engines/engine.py.230: loss: 0.18479 [I 2022-10-08 15:56:54] ...optimizers/trijoint.py.151: total_norm: 3.168724 [I 2022-10-08 15:57:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 2190/2304 [I 2022-10-08 15:57:02] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:01:29 [I 2022-10-08 15:57:02] ...trap/engines/engine.py.229: process: 0.78705 | load: 0.00307 [I 2022-10-08 15:57:02] ...trap/engines/engine.py.230: loss: 0.24085 [I 2022-10-08 15:57:02] ...optimizers/trijoint.py.151: total_norm: 3.029351 [I 2022-10-08 15:57:09] ...trap/engines/engine.py.224: train: epoch 10 | batch 2200/2304 [I 2022-10-08 15:57:09] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:01:20 [I 2022-10-08 15:57:09] ...trap/engines/engine.py.229: process: 0.77256 | load: 0.00303 [I 2022-10-08 15:57:09] ...trap/engines/engine.py.230: loss: 0.17253 [I 2022-10-08 15:57:09] ...optimizers/trijoint.py.151: total_norm: 2.658677 [I 2022-10-08 15:57:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 2210/2304 [I 2022-10-08 15:57:17] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:01:13 [I 2022-10-08 15:57:17] ...trap/engines/engine.py.229: process: 0.77839 | load: 0.00305 [I 2022-10-08 15:57:17] ...trap/engines/engine.py.230: loss: 0.14431 [I 2022-10-08 15:57:17] ...optimizers/trijoint.py.151: total_norm: 2.753946 [I 2022-10-08 15:57:25] ...trap/engines/engine.py.224: train: epoch 10 | batch 2220/2304 [I 2022-10-08 15:57:25] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:01:05 [I 2022-10-08 15:57:25] ...trap/engines/engine.py.229: process: 0.76362 | load: 0.00306 [I 2022-10-08 15:57:25] ...trap/engines/engine.py.230: loss: 0.19119 [I 2022-10-08 15:57:25] ...optimizers/trijoint.py.151: total_norm: 3.103322 [I 2022-10-08 15:57:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 2230/2304 [I 2022-10-08 15:57:33] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:00:57 [I 2022-10-08 15:57:33] ...trap/engines/engine.py.229: process: 0.77540 | load: 0.00306 [I 2022-10-08 15:57:33] ...trap/engines/engine.py.230: loss: 0.21594 [I 2022-10-08 15:57:33] ...optimizers/trijoint.py.151: total_norm: 2.560110 [I 2022-10-08 15:57:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 2240/2304 [I 2022-10-08 15:57:41] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:00:49 [I 2022-10-08 15:57:41] ...trap/engines/engine.py.229: process: 0.77177 | load: 0.00308 [I 2022-10-08 15:57:41] ...trap/engines/engine.py.230: loss: 0.14935 [I 2022-10-08 15:57:41] ...optimizers/trijoint.py.151: total_norm: 2.206148 [I 2022-10-08 15:57:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 2250/2304 [I 2022-10-08 15:57:48] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:00:41 [I 2022-10-08 15:57:48] ...trap/engines/engine.py.229: process: 0.77231 | load: 0.00302 [I 2022-10-08 15:57:48] ...trap/engines/engine.py.230: loss: 0.21250 [I 2022-10-08 15:57:48] ...optimizers/trijoint.py.151: total_norm: 2.495651 [I 2022-10-08 15:57:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 2260/2304 [I 2022-10-08 15:57:56] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:00:33 [I 2022-10-08 15:57:56] ...trap/engines/engine.py.229: process: 0.76048 | load: 0.00299 [I 2022-10-08 15:57:56] ...trap/engines/engine.py.230: loss: 0.18816 [I 2022-10-08 15:57:56] ...optimizers/trijoint.py.151: total_norm: 2.419036 [I 2022-10-08 15:58:04] ...trap/engines/engine.py.224: train: epoch 10 | batch 2270/2304 [I 2022-10-08 15:58:04] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:00:26 [I 2022-10-08 15:58:04] ...trap/engines/engine.py.229: process: 0.76692 | load: 0.00301 [I 2022-10-08 15:58:04] ...trap/engines/engine.py.230: loss: 0.21146 [I 2022-10-08 15:58:04] ...optimizers/trijoint.py.151: total_norm: 2.966098 [I 2022-10-08 15:58:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 2280/2304 [I 2022-10-08 15:58:12] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:00:18 [I 2022-10-08 15:58:12] ...trap/engines/engine.py.229: process: 0.77529 | load: 0.00306 [I 2022-10-08 15:58:12] ...trap/engines/engine.py.230: loss: 0.18219 [I 2022-10-08 15:58:12] ...optimizers/trijoint.py.151: total_norm: 2.833295 [I 2022-10-08 15:58:19] ...trap/engines/engine.py.224: train: epoch 10 | batch 2290/2304 [I 2022-10-08 15:58:19] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:00:10 [I 2022-10-08 15:58:19] ...trap/engines/engine.py.229: process: 0.78468 | load: 0.00307 [I 2022-10-08 15:58:19] ...trap/engines/engine.py.230: loss: 0.24453 [I 2022-10-08 15:58:19] ...optimizers/trijoint.py.151: total_norm: 3.527347 [I 2022-10-08 15:58:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 2300/2304 [I 2022-10-08 15:58:27] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:00:03 [I 2022-10-08 15:58:27] ...trap/engines/engine.py.229: process: 0.76842 | load: 0.00017 [I 2022-10-08 15:58:27] ...trap/engines/engine.py.230: loss: 0.18936 [I 2022-10-08 15:58:27] ...optimizers/trijoint.py.151: total_norm: 2.894307 [I 2022-10-08 15:58:30] ...trap/engines/engine.py.224: train: epoch 10 | batch 2304/2304 [I 2022-10-08 15:58:30] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:00:00 [I 2022-10-08 15:58:30] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00018 [I 2022-10-08 15:58:30] ...trap/engines/engine.py.230: loss: 0.19145 [I 2022-10-08 15:58:30] ...optimizers/trijoint.py.151: total_norm: 2.854406 [S 2022-10-08 15:58:32] ...trap/engines/engine.py.129: train_epoch.epoch: 10 [S 2022-10-08 15:58:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.199597892782564 [I 2022-10-08 15:58:32] ...s/criterions/triplet.py.84: margin = 0.10500000000000004 [I 2022-10-08 15:58:32] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 10 [I 2022-10-08 15:58:32] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 15:58:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 15:58:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 15:58:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 15:58:34] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 15:58:34] ...trap/engines/engine.py.321: eval: epoch 10 | batch 0/510 [I 2022-10-08 15:58:34] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:14 [I 2022-10-08 15:58:34] ...trap/engines/engine.py.326: process: 2.73526 | load: 2.24004 [I 2022-10-08 15:58:39] ...trap/engines/engine.py.321: eval: epoch 10 | batch 10/510 [I 2022-10-08 15:58:39] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:07 [I 2022-10-08 15:58:39] ...trap/engines/engine.py.326: process: 0.49106 | load: 0.00019 [I 2022-10-08 15:58:44] ...trap/engines/engine.py.321: eval: epoch 10 | batch 20/510 [I 2022-10-08 15:58:44] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:34 [I 2022-10-08 15:58:44] ...trap/engines/engine.py.326: process: 0.57919 | load: 0.00020 [I 2022-10-08 15:58:50] ...trap/engines/engine.py.321: eval: epoch 10 | batch 30/510 [I 2022-10-08 15:58:50] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:40 [I 2022-10-08 15:58:50] ...trap/engines/engine.py.326: process: 0.54679 | load: 0.00021 [I 2022-10-08 15:58:55] ...trap/engines/engine.py.321: eval: epoch 10 | batch 40/510 [I 2022-10-08 15:58:55] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:04:17 [I 2022-10-08 15:58:55] ...trap/engines/engine.py.326: process: 0.49542 | load: 0.00020 [I 2022-10-08 15:59:00] ...trap/engines/engine.py.321: eval: epoch 10 | batch 50/510 [I 2022-10-08 15:59:00] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:52 [I 2022-10-08 15:59:00] ...trap/engines/engine.py.326: process: 0.44976 | load: 0.00021 [I 2022-10-08 15:59:05] ...trap/engines/engine.py.321: eval: epoch 10 | batch 60/510 [I 2022-10-08 15:59:05] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:43 [I 2022-10-08 15:59:05] ...trap/engines/engine.py.326: process: 0.53615 | load: 0.00019 [I 2022-10-08 15:59:10] ...trap/engines/engine.py.321: eval: epoch 10 | batch 70/510 [I 2022-10-08 15:59:10] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:28 [I 2022-10-08 15:59:10] ...trap/engines/engine.py.326: process: 0.43390 | load: 0.00018 [I 2022-10-08 15:59:15] ...trap/engines/engine.py.321: eval: epoch 10 | batch 80/510 [I 2022-10-08 15:59:15] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:32 [I 2022-10-08 15:59:15] ...trap/engines/engine.py.326: process: 0.46885 | load: 0.00022 [I 2022-10-08 15:59:20] ...trap/engines/engine.py.321: eval: epoch 10 | batch 90/510 [I 2022-10-08 15:59:20] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:17 [I 2022-10-08 15:59:20] ...trap/engines/engine.py.326: process: 0.47608 | load: 0.00018 [I 2022-10-08 15:59:24] ...trap/engines/engine.py.321: eval: epoch 10 | batch 100/510 [I 2022-10-08 15:59:24] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:55 [I 2022-10-08 15:59:24] ...trap/engines/engine.py.326: process: 0.30983 | load: 0.00019 [I 2022-10-08 15:59:27] ...trap/engines/engine.py.321: eval: epoch 10 | batch 110/510 [I 2022-10-08 15:59:27] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:12 [I 2022-10-08 15:59:27] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00019 [I 2022-10-08 15:59:30] ...trap/engines/engine.py.321: eval: epoch 10 | batch 120/510 [I 2022-10-08 15:59:30] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:04 [I 2022-10-08 15:59:30] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00020 [I 2022-10-08 15:59:34] ...trap/engines/engine.py.321: eval: epoch 10 | batch 130/510 [I 2022-10-08 15:59:34] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:00 [I 2022-10-08 15:59:34] ...trap/engines/engine.py.326: process: 0.31295 | load: 0.00019 [I 2022-10-08 15:59:37] ...trap/engines/engine.py.321: eval: epoch 10 | batch 140/510 [I 2022-10-08 15:59:37] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:57 [I 2022-10-08 15:59:37] ...trap/engines/engine.py.326: process: 0.31685 | load: 0.00019 [I 2022-10-08 15:59:40] ...trap/engines/engine.py.321: eval: epoch 10 | batch 150/510 [I 2022-10-08 15:59:40] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:53 [I 2022-10-08 15:59:40] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00019 [I 2022-10-08 15:59:43] ...trap/engines/engine.py.321: eval: epoch 10 | batch 160/510 [I 2022-10-08 15:59:43] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:51 [I 2022-10-08 15:59:43] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00020 [I 2022-10-08 15:59:46] ...trap/engines/engine.py.321: eval: epoch 10 | batch 170/510 [I 2022-10-08 15:59:46] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-08 15:59:46] ...trap/engines/engine.py.326: process: 0.31605 | load: 0.00019 [I 2022-10-08 15:59:49] ...trap/engines/engine.py.321: eval: epoch 10 | batch 180/510 [I 2022-10-08 15:59:49] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:44 [I 2022-10-08 15:59:49] ...trap/engines/engine.py.326: process: 0.31721 | load: 0.00019 [I 2022-10-08 15:59:53] ...trap/engines/engine.py.321: eval: epoch 10 | batch 190/510 [I 2022-10-08 15:59:53] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:41 [I 2022-10-08 15:59:53] ...trap/engines/engine.py.326: process: 0.32020 | load: 0.00019 [I 2022-10-08 15:59:56] ...trap/engines/engine.py.321: eval: epoch 10 | batch 200/510 [I 2022-10-08 15:59:56] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:38 [I 2022-10-08 15:59:56] ...trap/engines/engine.py.326: process: 0.31494 | load: 0.00019 [I 2022-10-08 15:59:59] ...trap/engines/engine.py.321: eval: epoch 10 | batch 210/510 [I 2022-10-08 15:59:59] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:35 [I 2022-10-08 15:59:59] ...trap/engines/engine.py.326: process: 0.31567 | load: 0.00020 [I 2022-10-08 16:00:02] ...trap/engines/engine.py.321: eval: epoch 10 | batch 220/510 [I 2022-10-08 16:00:02] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:31 [I 2022-10-08 16:00:02] ...trap/engines/engine.py.326: process: 0.31509 | load: 0.00019 [I 2022-10-08 16:00:05] ...trap/engines/engine.py.321: eval: epoch 10 | batch 230/510 [I 2022-10-08 16:00:05] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:29 [I 2022-10-08 16:00:05] ...trap/engines/engine.py.326: process: 0.32075 | load: 0.00019 [I 2022-10-08 16:00:08] ...trap/engines/engine.py.321: eval: epoch 10 | batch 240/510 [I 2022-10-08 16:00:08] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:26 [I 2022-10-08 16:00:08] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00020 [I 2022-10-08 16:00:12] ...trap/engines/engine.py.321: eval: epoch 10 | batch 250/510 [I 2022-10-08 16:00:12] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-08 16:00:12] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00019 [I 2022-10-08 16:00:15] ...trap/engines/engine.py.321: eval: epoch 10 | batch 260/510 [I 2022-10-08 16:00:15] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-08 16:00:15] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00020 [I 2022-10-08 16:00:18] ...trap/engines/engine.py.321: eval: epoch 10 | batch 270/510 [I 2022-10-08 16:00:18] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:16 [I 2022-10-08 16:00:18] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00021 [I 2022-10-08 16:00:21] ...trap/engines/engine.py.321: eval: epoch 10 | batch 280/510 [I 2022-10-08 16:00:21] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-08 16:00:21] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00021 [I 2022-10-08 16:00:24] ...trap/engines/engine.py.321: eval: epoch 10 | batch 290/510 [I 2022-10-08 16:00:24] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:10 [I 2022-10-08 16:00:24] ...trap/engines/engine.py.326: process: 0.31734 | load: 0.00019 [I 2022-10-08 16:00:28] ...trap/engines/engine.py.321: eval: epoch 10 | batch 300/510 [I 2022-10-08 16:00:28] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:07 [I 2022-10-08 16:00:28] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00020 [I 2022-10-08 16:00:31] ...trap/engines/engine.py.321: eval: epoch 10 | batch 310/510 [I 2022-10-08 16:00:31] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-08 16:00:31] ...trap/engines/engine.py.326: process: 0.31960 | load: 0.00020 [I 2022-10-08 16:00:34] ...trap/engines/engine.py.321: eval: epoch 10 | batch 320/510 [I 2022-10-08 16:00:34] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-08 16:00:34] ...trap/engines/engine.py.326: process: 0.31641 | load: 0.00021 [I 2022-10-08 16:00:37] ...trap/engines/engine.py.321: eval: epoch 10 | batch 330/510 [I 2022-10-08 16:00:37] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-08 16:00:37] ...trap/engines/engine.py.326: process: 0.31946 | load: 0.00020 [I 2022-10-08 16:00:40] ...trap/engines/engine.py.321: eval: epoch 10 | batch 340/510 [I 2022-10-08 16:00:40] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-08 16:00:40] ...trap/engines/engine.py.326: process: 0.32103 | load: 0.00021 [I 2022-10-08 16:00:43] ...trap/engines/engine.py.321: eval: epoch 10 | batch 350/510 [I 2022-10-08 16:00:43] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-08 16:00:43] ...trap/engines/engine.py.326: process: 0.32174 | load: 0.00019 [I 2022-10-08 16:00:47] ...trap/engines/engine.py.321: eval: epoch 10 | batch 360/510 [I 2022-10-08 16:00:47] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-08 16:00:47] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00020 [I 2022-10-08 16:00:50] ...trap/engines/engine.py.321: eval: epoch 10 | batch 370/510 [I 2022-10-08 16:00:50] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-08 16:00:50] ...trap/engines/engine.py.326: process: 0.31305 | load: 0.00021 [I 2022-10-08 16:00:53] ...trap/engines/engine.py.321: eval: epoch 10 | batch 380/510 [I 2022-10-08 16:00:53] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-08 16:00:53] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00020 [I 2022-10-08 16:00:56] ...trap/engines/engine.py.321: eval: epoch 10 | batch 390/510 [I 2022-10-08 16:00:56] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:37 [I 2022-10-08 16:00:56] ...trap/engines/engine.py.326: process: 0.31194 | load: 0.00020 [I 2022-10-08 16:00:59] ...trap/engines/engine.py.321: eval: epoch 10 | batch 400/510 [I 2022-10-08 16:00:59] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-08 16:00:59] ...trap/engines/engine.py.326: process: 0.32057 | load: 0.00020 [I 2022-10-08 16:01:03] ...trap/engines/engine.py.321: eval: epoch 10 | batch 410/510 [I 2022-10-08 16:01:03] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-08 16:01:03] ...trap/engines/engine.py.326: process: 0.32130 | load: 0.00020 [I 2022-10-08 16:01:06] ...trap/engines/engine.py.321: eval: epoch 10 | batch 420/510 [I 2022-10-08 16:01:06] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-08 16:01:06] ...trap/engines/engine.py.326: process: 0.31041 | load: 0.00020 [I 2022-10-08 16:01:09] ...trap/engines/engine.py.321: eval: epoch 10 | batch 430/510 [I 2022-10-08 16:01:09] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-08 16:01:09] ...trap/engines/engine.py.326: process: 0.32114 | load: 0.00021 [I 2022-10-08 16:01:12] ...trap/engines/engine.py.321: eval: epoch 10 | batch 440/510 [I 2022-10-08 16:01:12] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-08 16:01:12] ...trap/engines/engine.py.326: process: 0.31954 | load: 0.00020 [I 2022-10-08 16:01:15] ...trap/engines/engine.py.321: eval: epoch 10 | batch 450/510 [I 2022-10-08 16:01:15] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-08 16:01:15] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00020 [I 2022-10-08 16:01:18] ...trap/engines/engine.py.321: eval: epoch 10 | batch 460/510 [I 2022-10-08 16:01:18] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-08 16:01:18] ...trap/engines/engine.py.326: process: 0.32235 | load: 0.00020 [I 2022-10-08 16:01:22] ...trap/engines/engine.py.321: eval: epoch 10 | batch 470/510 [I 2022-10-08 16:01:22] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-08 16:01:22] ...trap/engines/engine.py.326: process: 0.31861 | load: 0.00020 [I 2022-10-08 16:01:25] ...trap/engines/engine.py.321: eval: epoch 10 | batch 480/510 [I 2022-10-08 16:01:25] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-08 16:01:25] ...trap/engines/engine.py.326: process: 0.31665 | load: 0.00021 [I 2022-10-08 16:01:28] ...trap/engines/engine.py.321: eval: epoch 10 | batch 490/510 [I 2022-10-08 16:01:28] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-08 16:01:28] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00020 [I 2022-10-08 16:01:31] ...trap/engines/engine.py.321: eval: epoch 10 | batch 500/510 [I 2022-10-08 16:01:31] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-08 16:01:31] ...trap/engines/engine.py.326: process: 0.31293 | load: 0.00019 [I 2022-10-08 16:01:34] ...trap/engines/engine.py.321: eval: epoch 10 | batch 510/510 [I 2022-10-08 16:01:34] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-08 16:01:34] ...trap/engines/engine.py.326: process: 0.31609 | load: 0.00016 [S 2022-10-08 16:01:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 10 [I 2022-10-08 16:01:35] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.583100 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.849800 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.907500 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.591000 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.851100 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.907200 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011104 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009108 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010679 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015498 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010222 [S 2022-10-08 16:01:56] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007068 [I 2022-10-08 16:02:09] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 16:02:09] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 16:02:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 16:02:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 16:02:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 16:02:10] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 16:02:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 16:02:16] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 16:02:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 16:02:16] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 16:02:19] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 16:02:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 16:02:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 11 [I 2022-10-08 16:02:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 16:02:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 0/2304 [I 2022-10-08 16:02:26] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:15 [I 2022-10-08 16:02:26] ...trap/engines/engine.py.229: process: 2.97561 | load: 2.15712 [I 2022-10-08 16:02:26] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-08 16:02:26] ...optimizers/trijoint.py.151: total_norm: 2.334058 [I 2022-10-08 16:02:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 10/2304 [I 2022-10-08 16:02:34] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:44 [I 2022-10-08 16:02:34] ...trap/engines/engine.py.229: process: 0.76551 | load: 0.00563 [I 2022-10-08 16:02:34] ...trap/engines/engine.py.230: loss: 0.20125 [I 2022-10-08 16:02:34] ...optimizers/trijoint.py.151: total_norm: 3.209097 [I 2022-10-08 16:02:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 20/2304 [I 2022-10-08 16:02:42] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:33 [I 2022-10-08 16:02:42] ...trap/engines/engine.py.229: process: 0.78346 | load: 0.00416 [I 2022-10-08 16:02:42] ...trap/engines/engine.py.230: loss: 0.18349 [I 2022-10-08 16:02:42] ...optimizers/trijoint.py.151: total_norm: 2.006968 [I 2022-10-08 16:02:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 30/2304 [I 2022-10-08 16:02:50] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:48 [I 2022-10-08 16:02:50] ...trap/engines/engine.py.229: process: 0.79737 | load: 0.00446 [I 2022-10-08 16:02:50] ...trap/engines/engine.py.230: loss: 0.17833 [I 2022-10-08 16:02:50] ...optimizers/trijoint.py.151: total_norm: 2.942863 [I 2022-10-08 16:02:58] ...trap/engines/engine.py.224: train: epoch 11 | batch 40/2304 [I 2022-10-08 16:02:58] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:22 [I 2022-10-08 16:02:58] ...trap/engines/engine.py.229: process: 0.77257 | load: 0.00340 [I 2022-10-08 16:02:58] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-08 16:02:58] ...optimizers/trijoint.py.151: total_norm: 2.138046 [I 2022-10-08 16:03:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 50/2304 [I 2022-10-08 16:03:06] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:23 [I 2022-10-08 16:03:06] ...trap/engines/engine.py.229: process: 0.77949 | load: 0.00357 [I 2022-10-08 16:03:06] ...trap/engines/engine.py.230: loss: 0.23040 [I 2022-10-08 16:03:06] ...optimizers/trijoint.py.151: total_norm: 2.967089 [I 2022-10-08 16:03:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 60/2304 [I 2022-10-08 16:03:13] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:12 [I 2022-10-08 16:03:13] ...trap/engines/engine.py.229: process: 0.78541 | load: 0.00335 [I 2022-10-08 16:03:13] ...trap/engines/engine.py.230: loss: 0.23682 [I 2022-10-08 16:03:13] ...optimizers/trijoint.py.151: total_norm: 2.855507 [I 2022-10-08 16:03:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 70/2304 [I 2022-10-08 16:03:21] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:52 [I 2022-10-08 16:03:21] ...trap/engines/engine.py.229: process: 0.76160 | load: 0.00360 [I 2022-10-08 16:03:21] ...trap/engines/engine.py.230: loss: 0.21450 [I 2022-10-08 16:03:21] ...optimizers/trijoint.py.151: total_norm: 2.310910 [I 2022-10-08 16:03:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 80/2304 [I 2022-10-08 16:03:29] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:48 [I 2022-10-08 16:03:29] ...trap/engines/engine.py.229: process: 0.76844 | load: 0.00368 [I 2022-10-08 16:03:29] ...trap/engines/engine.py.230: loss: 0.27332 [I 2022-10-08 16:03:29] ...optimizers/trijoint.py.151: total_norm: 2.687951 [I 2022-10-08 16:03:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 90/2304 [I 2022-10-08 16:03:37] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:51 [I 2022-10-08 16:03:37] ...trap/engines/engine.py.229: process: 0.79552 | load: 0.00370 [I 2022-10-08 16:03:37] ...trap/engines/engine.py.230: loss: 0.19996 [I 2022-10-08 16:03:37] ...optimizers/trijoint.py.151: total_norm: 2.607201 [I 2022-10-08 16:03:45] ...trap/engines/engine.py.224: train: epoch 11 | batch 100/2304 [I 2022-10-08 16:03:45] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:44 [I 2022-10-08 16:03:45] ...trap/engines/engine.py.229: process: 0.77848 | load: 0.00343 [I 2022-10-08 16:03:45] ...trap/engines/engine.py.230: loss: 0.20812 [I 2022-10-08 16:03:45] ...optimizers/trijoint.py.151: total_norm: 2.873818 [I 2022-10-08 16:03:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 110/2304 [I 2022-10-08 16:03:52] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:35 [I 2022-10-08 16:03:52] ...trap/engines/engine.py.229: process: 0.78405 | load: 0.00359 [I 2022-10-08 16:03:52] ...trap/engines/engine.py.230: loss: 0.17335 [I 2022-10-08 16:03:52] ...optimizers/trijoint.py.151: total_norm: 2.989405 [I 2022-10-08 16:04:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 120/2304 [I 2022-10-08 16:04:00] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:24 [I 2022-10-08 16:04:00] ...trap/engines/engine.py.229: process: 0.78372 | load: 0.00299 [I 2022-10-08 16:04:00] ...trap/engines/engine.py.230: loss: 0.21172 [I 2022-10-08 16:04:00] ...optimizers/trijoint.py.151: total_norm: 3.237811 [I 2022-10-08 16:04:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 130/2304 [I 2022-10-08 16:04:08] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:12 [I 2022-10-08 16:04:08] ...trap/engines/engine.py.229: process: 0.78390 | load: 0.00318 [I 2022-10-08 16:04:08] ...trap/engines/engine.py.230: loss: 0.20036 [I 2022-10-08 16:04:08] ...optimizers/trijoint.py.151: total_norm: 2.444467 [I 2022-10-08 16:04:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 140/2304 [I 2022-10-08 16:04:16] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:10 [I 2022-10-08 16:04:16] ...trap/engines/engine.py.229: process: 0.77789 | load: 0.00317 [I 2022-10-08 16:04:16] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-08 16:04:16] ...optimizers/trijoint.py.151: total_norm: 3.719722 [I 2022-10-08 16:04:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 150/2304 [I 2022-10-08 16:04:24] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:01 [I 2022-10-08 16:04:24] ...trap/engines/engine.py.229: process: 0.78884 | load: 0.00332 [I 2022-10-08 16:04:24] ...trap/engines/engine.py.230: loss: 0.28433 [I 2022-10-08 16:04:24] ...optimizers/trijoint.py.151: total_norm: 3.365707 [I 2022-10-08 16:04:32] ...trap/engines/engine.py.224: train: epoch 11 | batch 160/2304 [I 2022-10-08 16:04:32] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:46 [I 2022-10-08 16:04:32] ...trap/engines/engine.py.229: process: 0.78155 | load: 0.00340 [I 2022-10-08 16:04:32] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-08 16:04:32] ...optimizers/trijoint.py.151: total_norm: 1.993228 [I 2022-10-08 16:04:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 170/2304 [I 2022-10-08 16:04:39] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:39 [I 2022-10-08 16:04:39] ...trap/engines/engine.py.229: process: 0.78162 | load: 0.00359 [I 2022-10-08 16:04:39] ...trap/engines/engine.py.230: loss: 0.13925 [I 2022-10-08 16:04:39] ...optimizers/trijoint.py.151: total_norm: 2.559076 [I 2022-10-08 16:04:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 180/2304 [I 2022-10-08 16:04:47] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:37 [I 2022-10-08 16:04:47] ...trap/engines/engine.py.229: process: 0.77687 | load: 0.00303 [I 2022-10-08 16:04:47] ...trap/engines/engine.py.230: loss: 0.19024 [I 2022-10-08 16:04:47] ...optimizers/trijoint.py.151: total_norm: 2.626856 [I 2022-10-08 16:04:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 190/2304 [I 2022-10-08 16:04:55] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:24 [I 2022-10-08 16:04:55] ...trap/engines/engine.py.229: process: 0.76858 | load: 0.00312 [I 2022-10-08 16:04:55] ...trap/engines/engine.py.230: loss: 0.13483 [I 2022-10-08 16:04:55] ...optimizers/trijoint.py.151: total_norm: 2.700667 [I 2022-10-08 16:05:03] ...trap/engines/engine.py.224: train: epoch 11 | batch 200/2304 [I 2022-10-08 16:05:03] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:21 [I 2022-10-08 16:05:03] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00319 [I 2022-10-08 16:05:03] ...trap/engines/engine.py.230: loss: 0.24460 [I 2022-10-08 16:05:03] ...optimizers/trijoint.py.151: total_norm: 2.855614 [I 2022-10-08 16:05:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 210/2304 [I 2022-10-08 16:05:11] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:15 [I 2022-10-08 16:05:11] ...trap/engines/engine.py.229: process: 0.78713 | load: 0.00306 [I 2022-10-08 16:05:11] ...trap/engines/engine.py.230: loss: 0.16910 [I 2022-10-08 16:05:11] ...optimizers/trijoint.py.151: total_norm: 2.864195 [I 2022-10-08 16:05:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 220/2304 [I 2022-10-08 16:05:19] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:09 [I 2022-10-08 16:05:19] ...trap/engines/engine.py.229: process: 0.77223 | load: 0.00328 [I 2022-10-08 16:05:19] ...trap/engines/engine.py.230: loss: 0.29802 [I 2022-10-08 16:05:19] ...optimizers/trijoint.py.151: total_norm: 4.282800 [I 2022-10-08 16:05:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 230/2304 [I 2022-10-08 16:05:26] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:54 [I 2022-10-08 16:05:26] ...trap/engines/engine.py.229: process: 0.78337 | load: 0.00312 [I 2022-10-08 16:05:26] ...trap/engines/engine.py.230: loss: 0.19656 [I 2022-10-08 16:05:26] ...optimizers/trijoint.py.151: total_norm: 2.371618 [I 2022-10-08 16:05:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 240/2304 [I 2022-10-08 16:05:34] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:44 [I 2022-10-08 16:05:34] ...trap/engines/engine.py.229: process: 0.77265 | load: 0.00323 [I 2022-10-08 16:05:34] ...trap/engines/engine.py.230: loss: 0.20280 [I 2022-10-08 16:05:34] ...optimizers/trijoint.py.151: total_norm: 2.812247 [I 2022-10-08 16:05:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 250/2304 [I 2022-10-08 16:05:42] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:46 [I 2022-10-08 16:05:42] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00326 [I 2022-10-08 16:05:42] ...trap/engines/engine.py.230: loss: 0.24256 [I 2022-10-08 16:05:42] ...optimizers/trijoint.py.151: total_norm: 3.118429 [I 2022-10-08 16:05:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 260/2304 [I 2022-10-08 16:05:50] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:36 [I 2022-10-08 16:05:50] ...trap/engines/engine.py.229: process: 0.79025 | load: 0.00312 [I 2022-10-08 16:05:50] ...trap/engines/engine.py.230: loss: 0.15130 [I 2022-10-08 16:05:50] ...optimizers/trijoint.py.151: total_norm: 1.641770 [I 2022-10-08 16:05:58] ...trap/engines/engine.py.224: train: epoch 11 | batch 270/2304 [I 2022-10-08 16:05:58] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:26 [I 2022-10-08 16:05:58] ...trap/engines/engine.py.229: process: 0.78374 | load: 0.00311 [I 2022-10-08 16:05:58] ...trap/engines/engine.py.230: loss: 0.23720 [I 2022-10-08 16:05:58] ...optimizers/trijoint.py.151: total_norm: 2.412872 [I 2022-10-08 16:06:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 280/2304 [I 2022-10-08 16:06:05] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:18 [I 2022-10-08 16:06:05] ...trap/engines/engine.py.229: process: 0.77945 | load: 0.00313 [I 2022-10-08 16:06:05] ...trap/engines/engine.py.230: loss: 0.20799 [I 2022-10-08 16:06:05] ...optimizers/trijoint.py.151: total_norm: 3.205579 [I 2022-10-08 16:06:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 290/2304 [I 2022-10-08 16:06:13] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:17 [I 2022-10-08 16:06:13] ...trap/engines/engine.py.229: process: 0.78410 | load: 0.00312 [I 2022-10-08 16:06:13] ...trap/engines/engine.py.230: loss: 0.28277 [I 2022-10-08 16:06:13] ...optimizers/trijoint.py.151: total_norm: 2.771991 [I 2022-10-08 16:06:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 300/2304 [I 2022-10-08 16:06:21] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:57 [I 2022-10-08 16:06:21] ...trap/engines/engine.py.229: process: 0.77022 | load: 0.00323 [I 2022-10-08 16:06:21] ...trap/engines/engine.py.230: loss: 0.21226 [I 2022-10-08 16:06:21] ...optimizers/trijoint.py.151: total_norm: 3.121858 [I 2022-10-08 16:06:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 310/2304 [I 2022-10-08 16:06:29] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:59 [I 2022-10-08 16:06:29] ...trap/engines/engine.py.229: process: 0.77407 | load: 0.00305 [I 2022-10-08 16:06:29] ...trap/engines/engine.py.230: loss: 0.17454 [I 2022-10-08 16:06:29] ...optimizers/trijoint.py.151: total_norm: 3.004992 [I 2022-10-08 16:06:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 320/2304 [I 2022-10-08 16:06:37] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:45 [I 2022-10-08 16:06:37] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00303 [I 2022-10-08 16:06:37] ...trap/engines/engine.py.230: loss: 0.18773 [I 2022-10-08 16:06:37] ...optimizers/trijoint.py.151: total_norm: 2.624761 [I 2022-10-08 16:06:45] ...trap/engines/engine.py.224: train: epoch 11 | batch 330/2304 [I 2022-10-08 16:06:45] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:43 [I 2022-10-08 16:06:45] ...trap/engines/engine.py.229: process: 0.78219 | load: 0.00306 [I 2022-10-08 16:06:45] ...trap/engines/engine.py.230: loss: 0.27951 [I 2022-10-08 16:06:45] ...optimizers/trijoint.py.151: total_norm: 2.970366 [I 2022-10-08 16:06:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 340/2304 [I 2022-10-08 16:06:52] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:26 [I 2022-10-08 16:06:52] ...trap/engines/engine.py.229: process: 0.76344 | load: 0.00304 [I 2022-10-08 16:06:52] ...trap/engines/engine.py.230: loss: 0.14596 [I 2022-10-08 16:06:52] ...optimizers/trijoint.py.151: total_norm: 2.791066 [I 2022-10-08 16:07:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 350/2304 [I 2022-10-08 16:07:00] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:23 [I 2022-10-08 16:07:00] ...trap/engines/engine.py.229: process: 0.77402 | load: 0.00300 [I 2022-10-08 16:07:00] ...trap/engines/engine.py.230: loss: 0.21852 [I 2022-10-08 16:07:00] ...optimizers/trijoint.py.151: total_norm: 2.633525 [I 2022-10-08 16:07:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 360/2304 [I 2022-10-08 16:07:08] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:12 [I 2022-10-08 16:07:08] ...trap/engines/engine.py.229: process: 0.77561 | load: 0.00302 [I 2022-10-08 16:07:08] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-08 16:07:08] ...optimizers/trijoint.py.151: total_norm: 2.414430 [I 2022-10-08 16:07:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 370/2304 [I 2022-10-08 16:07:16] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:25:04 [I 2022-10-08 16:07:16] ...trap/engines/engine.py.229: process: 0.76830 | load: 0.00304 [I 2022-10-08 16:07:16] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-08 16:07:16] ...optimizers/trijoint.py.151: total_norm: 2.445332 [I 2022-10-08 16:07:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 380/2304 [I 2022-10-08 16:07:24] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:59 [I 2022-10-08 16:07:24] ...trap/engines/engine.py.229: process: 0.78855 | load: 0.00313 [I 2022-10-08 16:07:24] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-08 16:07:24] ...optimizers/trijoint.py.151: total_norm: 2.825269 [I 2022-10-08 16:07:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 390/2304 [I 2022-10-08 16:07:31] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:48 [I 2022-10-08 16:07:31] ...trap/engines/engine.py.229: process: 0.78482 | load: 0.00303 [I 2022-10-08 16:07:31] ...trap/engines/engine.py.230: loss: 0.18293 [I 2022-10-08 16:07:31] ...optimizers/trijoint.py.151: total_norm: 3.425412 [I 2022-10-08 16:07:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 400/2304 [I 2022-10-08 16:07:39] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:47 [I 2022-10-08 16:07:39] ...trap/engines/engine.py.229: process: 0.79283 | load: 0.00301 [I 2022-10-08 16:07:39] ...trap/engines/engine.py.230: loss: 0.20772 [I 2022-10-08 16:07:39] ...optimizers/trijoint.py.151: total_norm: 2.575245 [I 2022-10-08 16:07:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 410/2304 [I 2022-10-08 16:07:47] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:42 [I 2022-10-08 16:07:47] ...trap/engines/engine.py.229: process: 0.79090 | load: 0.00306 [I 2022-10-08 16:07:47] ...trap/engines/engine.py.230: loss: 0.13654 [I 2022-10-08 16:07:47] ...optimizers/trijoint.py.151: total_norm: 2.423388 [I 2022-10-08 16:07:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 420/2304 [I 2022-10-08 16:07:55] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:27 [I 2022-10-08 16:07:55] ...trap/engines/engine.py.229: process: 0.78516 | load: 0.00303 [I 2022-10-08 16:07:55] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-08 16:07:55] ...optimizers/trijoint.py.151: total_norm: 2.317994 [I 2022-10-08 16:08:03] ...trap/engines/engine.py.224: train: epoch 11 | batch 430/2304 [I 2022-10-08 16:08:03] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:25 [I 2022-10-08 16:08:03] ...trap/engines/engine.py.229: process: 0.77663 | load: 0.00316 [I 2022-10-08 16:08:03] ...trap/engines/engine.py.230: loss: 0.19349 [I 2022-10-08 16:08:03] ...optimizers/trijoint.py.151: total_norm: 2.836210 [I 2022-10-08 16:08:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 440/2304 [I 2022-10-08 16:08:11] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:24:14 [I 2022-10-08 16:08:11] ...trap/engines/engine.py.229: process: 0.78605 | load: 0.00305 [I 2022-10-08 16:08:11] ...trap/engines/engine.py.230: loss: 0.24997 [I 2022-10-08 16:08:11] ...optimizers/trijoint.py.151: total_norm: 2.654591 [I 2022-10-08 16:08:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 450/2304 [I 2022-10-08 16:08:18] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:00 [I 2022-10-08 16:08:18] ...trap/engines/engine.py.229: process: 0.77491 | load: 0.00304 [I 2022-10-08 16:08:18] ...trap/engines/engine.py.230: loss: 0.24371 [I 2022-10-08 16:08:18] ...optimizers/trijoint.py.151: total_norm: 2.976107 [I 2022-10-08 16:08:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 460/2304 [I 2022-10-08 16:08:26] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:24:00 [I 2022-10-08 16:08:26] ...trap/engines/engine.py.229: process: 0.77365 | load: 0.00304 [I 2022-10-08 16:08:26] ...trap/engines/engine.py.230: loss: 0.15424 [I 2022-10-08 16:08:26] ...optimizers/trijoint.py.151: total_norm: 2.539467 [I 2022-10-08 16:08:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 470/2304 [I 2022-10-08 16:08:34] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:51 [I 2022-10-08 16:08:34] ...trap/engines/engine.py.229: process: 0.77591 | load: 0.00299 [I 2022-10-08 16:08:34] ...trap/engines/engine.py.230: loss: 0.20586 [I 2022-10-08 16:08:34] ...optimizers/trijoint.py.151: total_norm: 2.931841 [I 2022-10-08 16:08:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 480/2304 [I 2022-10-08 16:08:42] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:49 [I 2022-10-08 16:08:42] ...trap/engines/engine.py.229: process: 0.78424 | load: 0.00301 [I 2022-10-08 16:08:42] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-08 16:08:42] ...optimizers/trijoint.py.151: total_norm: 2.602158 [I 2022-10-08 16:08:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 490/2304 [I 2022-10-08 16:08:50] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:39 [I 2022-10-08 16:08:50] ...trap/engines/engine.py.229: process: 0.78169 | load: 0.00308 [I 2022-10-08 16:08:50] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-08 16:08:50] ...optimizers/trijoint.py.151: total_norm: 2.406000 [I 2022-10-08 16:08:58] ...trap/engines/engine.py.224: train: epoch 11 | batch 500/2304 [I 2022-10-08 16:08:58] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:23:36 [I 2022-10-08 16:08:58] ...trap/engines/engine.py.229: process: 0.79566 | load: 0.00303 [I 2022-10-08 16:08:58] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-08 16:08:58] ...optimizers/trijoint.py.151: total_norm: 2.487141 [I 2022-10-08 16:09:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 510/2304 [I 2022-10-08 16:09:05] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:20 [I 2022-10-08 16:09:05] ...trap/engines/engine.py.229: process: 0.77997 | load: 0.00298 [I 2022-10-08 16:09:05] ...trap/engines/engine.py.230: loss: 0.15127 [I 2022-10-08 16:09:05] ...optimizers/trijoint.py.151: total_norm: 3.036831 [I 2022-10-08 16:09:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 520/2304 [I 2022-10-08 16:09:13] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:09 [I 2022-10-08 16:09:13] ...trap/engines/engine.py.229: process: 0.78367 | load: 0.00312 [I 2022-10-08 16:09:13] ...trap/engines/engine.py.230: loss: 0.20691 [I 2022-10-08 16:09:13] ...optimizers/trijoint.py.151: total_norm: 3.026666 [I 2022-10-08 16:09:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 530/2304 [I 2022-10-08 16:09:21] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:22:59 [I 2022-10-08 16:09:21] ...trap/engines/engine.py.229: process: 0.78366 | load: 0.00298 [I 2022-10-08 16:09:21] ...trap/engines/engine.py.230: loss: 0.13509 [I 2022-10-08 16:09:21] ...optimizers/trijoint.py.151: total_norm: 2.477107 [I 2022-10-08 16:09:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 540/2304 [I 2022-10-08 16:09:29] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:47 [I 2022-10-08 16:09:29] ...trap/engines/engine.py.229: process: 0.77152 | load: 0.00301 [I 2022-10-08 16:09:29] ...trap/engines/engine.py.230: loss: 0.28244 [I 2022-10-08 16:09:29] ...optimizers/trijoint.py.151: total_norm: 2.919311 [I 2022-10-08 16:09:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 550/2304 [I 2022-10-08 16:09:37] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:22:34 [I 2022-10-08 16:09:37] ...trap/engines/engine.py.229: process: 0.76501 | load: 0.00300 [I 2022-10-08 16:09:37] ...trap/engines/engine.py.230: loss: 0.17588 [I 2022-10-08 16:09:37] ...optimizers/trijoint.py.151: total_norm: 2.847467 [I 2022-10-08 16:09:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 560/2304 [I 2022-10-08 16:09:44] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:40 [I 2022-10-08 16:09:44] ...trap/engines/engine.py.229: process: 0.77232 | load: 0.00301 [I 2022-10-08 16:09:44] ...trap/engines/engine.py.230: loss: 0.29238 [I 2022-10-08 16:09:44] ...optimizers/trijoint.py.151: total_norm: 3.061412 [I 2022-10-08 16:09:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 570/2304 [I 2022-10-08 16:09:52] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:36 [I 2022-10-08 16:09:52] ...trap/engines/engine.py.229: process: 0.77794 | load: 0.00301 [I 2022-10-08 16:09:52] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-08 16:09:52] ...optimizers/trijoint.py.151: total_norm: 2.964938 [I 2022-10-08 16:10:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 580/2304 [I 2022-10-08 16:10:00] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:25 [I 2022-10-08 16:10:00] ...trap/engines/engine.py.229: process: 0.78306 | load: 0.00299 [I 2022-10-08 16:10:00] ...trap/engines/engine.py.230: loss: 0.18659 [I 2022-10-08 16:10:00] ...optimizers/trijoint.py.151: total_norm: 2.642857 [I 2022-10-08 16:10:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 590/2304 [I 2022-10-08 16:10:08] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:14 [I 2022-10-08 16:10:08] ...trap/engines/engine.py.229: process: 0.78114 | load: 0.00329 [I 2022-10-08 16:10:08] ...trap/engines/engine.py.230: loss: 0.24330 [I 2022-10-08 16:10:08] ...optimizers/trijoint.py.151: total_norm: 2.716956 [I 2022-10-08 16:10:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 600/2304 [I 2022-10-08 16:10:16] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:09 [I 2022-10-08 16:10:16] ...trap/engines/engine.py.229: process: 0.78105 | load: 0.00300 [I 2022-10-08 16:10:16] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-08 16:10:16] ...optimizers/trijoint.py.151: total_norm: 3.038921 [I 2022-10-08 16:10:23] ...trap/engines/engine.py.224: train: epoch 11 | batch 610/2304 [I 2022-10-08 16:10:23] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:21:58 [I 2022-10-08 16:10:23] ...trap/engines/engine.py.229: process: 0.77275 | load: 0.00307 [I 2022-10-08 16:10:23] ...trap/engines/engine.py.230: loss: 0.22554 [I 2022-10-08 16:10:23] ...optimizers/trijoint.py.151: total_norm: 2.615705 [I 2022-10-08 16:10:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 620/2304 [I 2022-10-08 16:10:31] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:50 [I 2022-10-08 16:10:31] ...trap/engines/engine.py.229: process: 0.78005 | load: 0.00297 [I 2022-10-08 16:10:31] ...trap/engines/engine.py.230: loss: 0.34848 [I 2022-10-08 16:10:31] ...optimizers/trijoint.py.151: total_norm: 3.405390 [I 2022-10-08 16:10:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 630/2304 [I 2022-10-08 16:10:39] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:46 [I 2022-10-08 16:10:39] ...trap/engines/engine.py.229: process: 0.77595 | load: 0.00301 [I 2022-10-08 16:10:39] ...trap/engines/engine.py.230: loss: 0.24754 [I 2022-10-08 16:10:39] ...optimizers/trijoint.py.151: total_norm: 3.036028 [I 2022-10-08 16:10:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 640/2304 [I 2022-10-08 16:10:47] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:38 [I 2022-10-08 16:10:47] ...trap/engines/engine.py.229: process: 0.78423 | load: 0.00301 [I 2022-10-08 16:10:47] ...trap/engines/engine.py.230: loss: 0.12129 [I 2022-10-08 16:10:47] ...optimizers/trijoint.py.151: total_norm: 1.966495 [I 2022-10-08 16:10:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 650/2304 [I 2022-10-08 16:10:55] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:34 [I 2022-10-08 16:10:55] ...trap/engines/engine.py.229: process: 0.78755 | load: 0.00309 [I 2022-10-08 16:10:55] ...trap/engines/engine.py.230: loss: 0.24601 [I 2022-10-08 16:10:55] ...optimizers/trijoint.py.151: total_norm: 3.122907 [I 2022-10-08 16:11:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 660/2304 [I 2022-10-08 16:11:02] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:18 [I 2022-10-08 16:11:02] ...trap/engines/engine.py.229: process: 0.77710 | load: 0.00300 [I 2022-10-08 16:11:02] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-08 16:11:02] ...optimizers/trijoint.py.151: total_norm: 2.906535 [I 2022-10-08 16:11:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 670/2304 [I 2022-10-08 16:11:10] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:13 [I 2022-10-08 16:11:10] ...trap/engines/engine.py.229: process: 0.77521 | load: 0.00303 [I 2022-10-08 16:11:10] ...trap/engines/engine.py.230: loss: 0.13736 [I 2022-10-08 16:11:10] ...optimizers/trijoint.py.151: total_norm: 1.778487 [I 2022-10-08 16:11:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 680/2304 [I 2022-10-08 16:11:18] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:08 [I 2022-10-08 16:11:18] ...trap/engines/engine.py.229: process: 0.78006 | load: 0.00304 [I 2022-10-08 16:11:18] ...trap/engines/engine.py.230: loss: 0.19479 [I 2022-10-08 16:11:18] ...optimizers/trijoint.py.151: total_norm: 2.496912 [I 2022-10-08 16:11:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 690/2304 [I 2022-10-08 16:11:26] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:57 [I 2022-10-08 16:11:26] ...trap/engines/engine.py.229: process: 0.77558 | load: 0.00297 [I 2022-10-08 16:11:26] ...trap/engines/engine.py.230: loss: 0.23298 [I 2022-10-08 16:11:26] ...optimizers/trijoint.py.151: total_norm: 2.455753 [I 2022-10-08 16:11:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 700/2304 [I 2022-10-08 16:11:34] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:49 [I 2022-10-08 16:11:34] ...trap/engines/engine.py.229: process: 0.78010 | load: 0.00308 [I 2022-10-08 16:11:34] ...trap/engines/engine.py.230: loss: 0.19726 [I 2022-10-08 16:11:34] ...optimizers/trijoint.py.151: total_norm: 2.287083 [I 2022-10-08 16:11:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 710/2304 [I 2022-10-08 16:11:42] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:20:44 [I 2022-10-08 16:11:42] ...trap/engines/engine.py.229: process: 0.78332 | load: 0.00302 [I 2022-10-08 16:11:42] ...trap/engines/engine.py.230: loss: 0.17905 [I 2022-10-08 16:11:42] ...optimizers/trijoint.py.151: total_norm: 2.252667 [I 2022-10-08 16:11:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 720/2304 [I 2022-10-08 16:11:49] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:36 [I 2022-10-08 16:11:49] ...trap/engines/engine.py.229: process: 0.78114 | load: 0.00297 [I 2022-10-08 16:11:49] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-08 16:11:49] ...optimizers/trijoint.py.151: total_norm: 2.670608 [I 2022-10-08 16:11:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 730/2304 [I 2022-10-08 16:11:57] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:31 [I 2022-10-08 16:11:57] ...trap/engines/engine.py.229: process: 0.79049 | load: 0.00304 [I 2022-10-08 16:11:57] ...trap/engines/engine.py.230: loss: 0.17363 [I 2022-10-08 16:11:57] ...optimizers/trijoint.py.151: total_norm: 3.271965 [I 2022-10-08 16:12:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 740/2304 [I 2022-10-08 16:12:05] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:18 [I 2022-10-08 16:12:05] ...trap/engines/engine.py.229: process: 0.77743 | load: 0.00302 [I 2022-10-08 16:12:05] ...trap/engines/engine.py.230: loss: 0.23487 [I 2022-10-08 16:12:05] ...optimizers/trijoint.py.151: total_norm: 2.729130 [I 2022-10-08 16:12:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 750/2304 [I 2022-10-08 16:12:13] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:14 [I 2022-10-08 16:12:13] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00300 [I 2022-10-08 16:12:13] ...trap/engines/engine.py.230: loss: 0.15341 [I 2022-10-08 16:12:13] ...optimizers/trijoint.py.151: total_norm: 2.743479 [I 2022-10-08 16:12:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 760/2304 [I 2022-10-08 16:12:21] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:20:02 [I 2022-10-08 16:12:21] ...trap/engines/engine.py.229: process: 0.76781 | load: 0.00305 [I 2022-10-08 16:12:21] ...trap/engines/engine.py.230: loss: 0.21988 [I 2022-10-08 16:12:21] ...optimizers/trijoint.py.151: total_norm: 2.380818 [I 2022-10-08 16:12:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 770/2304 [I 2022-10-08 16:12:29] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:20:02 [I 2022-10-08 16:12:29] ...trap/engines/engine.py.229: process: 0.78500 | load: 0.00320 [I 2022-10-08 16:12:29] ...trap/engines/engine.py.230: loss: 0.10906 [I 2022-10-08 16:12:29] ...optimizers/trijoint.py.151: total_norm: 2.217223 [I 2022-10-08 16:12:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 780/2304 [I 2022-10-08 16:12:37] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:19:49 [I 2022-10-08 16:12:37] ...trap/engines/engine.py.229: process: 0.78499 | load: 0.00303 [I 2022-10-08 16:12:37] ...trap/engines/engine.py.230: loss: 0.23275 [I 2022-10-08 16:12:37] ...optimizers/trijoint.py.151: total_norm: 2.637172 [I 2022-10-08 16:12:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 790/2304 [I 2022-10-08 16:12:44] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:39 [I 2022-10-08 16:12:44] ...trap/engines/engine.py.229: process: 0.78347 | load: 0.00304 [I 2022-10-08 16:12:44] ...trap/engines/engine.py.230: loss: 0.17834 [I 2022-10-08 16:12:44] ...optimizers/trijoint.py.151: total_norm: 3.001746 [I 2022-10-08 16:12:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 800/2304 [I 2022-10-08 16:12:52] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:30 [I 2022-10-08 16:12:52] ...trap/engines/engine.py.229: process: 0.77695 | load: 0.00302 [I 2022-10-08 16:12:52] ...trap/engines/engine.py.230: loss: 0.16788 [I 2022-10-08 16:12:52] ...optimizers/trijoint.py.151: total_norm: 2.759496 [I 2022-10-08 16:13:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 810/2304 [I 2022-10-08 16:13:00] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:25 [I 2022-10-08 16:13:00] ...trap/engines/engine.py.229: process: 0.78312 | load: 0.00304 [I 2022-10-08 16:13:00] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-08 16:13:00] ...optimizers/trijoint.py.151: total_norm: 2.237093 [I 2022-10-08 16:13:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 820/2304 [I 2022-10-08 16:13:08] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:19:18 [I 2022-10-08 16:13:08] ...trap/engines/engine.py.229: process: 0.77636 | load: 0.00304 [I 2022-10-08 16:13:08] ...trap/engines/engine.py.230: loss: 0.18609 [I 2022-10-08 16:13:08] ...optimizers/trijoint.py.151: total_norm: 2.281513 [I 2022-10-08 16:13:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 830/2304 [I 2022-10-08 16:13:16] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:19:14 [I 2022-10-08 16:13:16] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00302 [I 2022-10-08 16:13:16] ...trap/engines/engine.py.230: loss: 0.19812 [I 2022-10-08 16:13:16] ...optimizers/trijoint.py.151: total_norm: 3.031953 [I 2022-10-08 16:13:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 840/2304 [I 2022-10-08 16:13:24] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:19:06 [I 2022-10-08 16:13:24] ...trap/engines/engine.py.229: process: 0.79407 | load: 0.00302 [I 2022-10-08 16:13:24] ...trap/engines/engine.py.230: loss: 0.22155 [I 2022-10-08 16:13:24] ...optimizers/trijoint.py.151: total_norm: 4.473151 [I 2022-10-08 16:13:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 850/2304 [I 2022-10-08 16:13:31] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:18:51 [I 2022-10-08 16:13:31] ...trap/engines/engine.py.229: process: 0.77875 | load: 0.00301 [I 2022-10-08 16:13:31] ...trap/engines/engine.py.230: loss: 0.22367 [I 2022-10-08 16:13:31] ...optimizers/trijoint.py.151: total_norm: 2.497009 [I 2022-10-08 16:13:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 860/2304 [I 2022-10-08 16:13:39] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:42 [I 2022-10-08 16:13:39] ...trap/engines/engine.py.229: process: 0.78789 | load: 0.00302 [I 2022-10-08 16:13:39] ...trap/engines/engine.py.230: loss: 0.23321 [I 2022-10-08 16:13:39] ...optimizers/trijoint.py.151: total_norm: 2.567567 [I 2022-10-08 16:13:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 870/2304 [I 2022-10-08 16:13:47] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:40 [I 2022-10-08 16:13:47] ...trap/engines/engine.py.229: process: 0.78174 | load: 0.00304 [I 2022-10-08 16:13:47] ...trap/engines/engine.py.230: loss: 0.19286 [I 2022-10-08 16:13:47] ...optimizers/trijoint.py.151: total_norm: 2.602558 [I 2022-10-08 16:13:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 880/2304 [I 2022-10-08 16:13:55] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:18:29 [I 2022-10-08 16:13:55] ...trap/engines/engine.py.229: process: 0.78532 | load: 0.00302 [I 2022-10-08 16:13:55] ...trap/engines/engine.py.230: loss: 0.24525 [I 2022-10-08 16:13:55] ...optimizers/trijoint.py.151: total_norm: 3.511637 [I 2022-10-08 16:14:03] ...trap/engines/engine.py.224: train: epoch 11 | batch 890/2304 [I 2022-10-08 16:14:03] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:18:22 [I 2022-10-08 16:14:03] ...trap/engines/engine.py.229: process: 0.78320 | load: 0.00304 [I 2022-10-08 16:14:03] ...trap/engines/engine.py.230: loss: 0.13255 [I 2022-10-08 16:14:03] ...optimizers/trijoint.py.151: total_norm: 1.939418 [I 2022-10-08 16:14:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 900/2304 [I 2022-10-08 16:14:10] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:09 [I 2022-10-08 16:14:10] ...trap/engines/engine.py.229: process: 0.77912 | load: 0.00305 [I 2022-10-08 16:14:10] ...trap/engines/engine.py.230: loss: 0.24296 [I 2022-10-08 16:14:10] ...optimizers/trijoint.py.151: total_norm: 2.470604 [I 2022-10-08 16:14:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 910/2304 [I 2022-10-08 16:14:18] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:18:06 [I 2022-10-08 16:14:18] ...trap/engines/engine.py.229: process: 0.77169 | load: 0.00303 [I 2022-10-08 16:14:18] ...trap/engines/engine.py.230: loss: 0.24933 [I 2022-10-08 16:14:18] ...optimizers/trijoint.py.151: total_norm: 2.720549 [I 2022-10-08 16:14:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 920/2304 [I 2022-10-08 16:14:26] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:18:02 [I 2022-10-08 16:14:26] ...trap/engines/engine.py.229: process: 0.78909 | load: 0.00304 [I 2022-10-08 16:14:26] ...trap/engines/engine.py.230: loss: 0.15899 [I 2022-10-08 16:14:26] ...optimizers/trijoint.py.151: total_norm: 2.494938 [I 2022-10-08 16:14:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 930/2304 [I 2022-10-08 16:14:34] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:17:53 [I 2022-10-08 16:14:34] ...trap/engines/engine.py.229: process: 0.78207 | load: 0.00305 [I 2022-10-08 16:14:34] ...trap/engines/engine.py.230: loss: 0.18992 [I 2022-10-08 16:14:34] ...optimizers/trijoint.py.151: total_norm: 1.767068 [I 2022-10-08 16:14:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 940/2304 [I 2022-10-08 16:14:42] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:17:47 [I 2022-10-08 16:14:42] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00305 [I 2022-10-08 16:14:42] ...trap/engines/engine.py.230: loss: 0.20109 [I 2022-10-08 16:14:42] ...optimizers/trijoint.py.151: total_norm: 3.509788 [I 2022-10-08 16:14:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 950/2304 [I 2022-10-08 16:14:50] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:30 [I 2022-10-08 16:14:50] ...trap/engines/engine.py.229: process: 0.76476 | load: 0.00310 [I 2022-10-08 16:14:50] ...trap/engines/engine.py.230: loss: 0.31274 [I 2022-10-08 16:14:50] ...optimizers/trijoint.py.151: total_norm: 2.896828 [I 2022-10-08 16:14:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 960/2304 [I 2022-10-08 16:14:57] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:21 [I 2022-10-08 16:14:57] ...trap/engines/engine.py.229: process: 0.77864 | load: 0.00303 [I 2022-10-08 16:14:57] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-08 16:14:57] ...optimizers/trijoint.py.151: total_norm: 2.338562 [I 2022-10-08 16:15:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 970/2304 [I 2022-10-08 16:15:05] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:15 [I 2022-10-08 16:15:05] ...trap/engines/engine.py.229: process: 0.77488 | load: 0.00303 [I 2022-10-08 16:15:05] ...trap/engines/engine.py.230: loss: 0.22032 [I 2022-10-08 16:15:05] ...optimizers/trijoint.py.151: total_norm: 3.899961 [I 2022-10-08 16:15:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 980/2304 [I 2022-10-08 16:15:13] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:17:02 [I 2022-10-08 16:15:13] ...trap/engines/engine.py.229: process: 0.78660 | load: 0.00303 [I 2022-10-08 16:15:13] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-08 16:15:13] ...optimizers/trijoint.py.151: total_norm: 2.348573 [I 2022-10-08 16:15:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 990/2304 [I 2022-10-08 16:15:21] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:17:03 [I 2022-10-08 16:15:21] ...trap/engines/engine.py.229: process: 0.77376 | load: 0.00304 [I 2022-10-08 16:15:21] ...trap/engines/engine.py.230: loss: 0.23884 [I 2022-10-08 16:15:21] ...optimizers/trijoint.py.151: total_norm: 2.591210 [I 2022-10-08 16:15:28] ...trap/engines/engine.py.224: train: epoch 11 | batch 1000/2304 [I 2022-10-08 16:15:28] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:16:54 [I 2022-10-08 16:15:28] ...trap/engines/engine.py.229: process: 0.76843 | load: 0.00302 [I 2022-10-08 16:15:28] ...trap/engines/engine.py.230: loss: 0.21197 [I 2022-10-08 16:15:28] ...optimizers/trijoint.py.151: total_norm: 3.314271 [I 2022-10-08 16:15:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 1010/2304 [I 2022-10-08 16:15:36] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:49 [I 2022-10-08 16:15:36] ...trap/engines/engine.py.229: process: 0.77735 | load: 0.00300 [I 2022-10-08 16:15:36] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-08 16:15:36] ...optimizers/trijoint.py.151: total_norm: 2.214721 [I 2022-10-08 16:15:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1020/2304 [I 2022-10-08 16:15:44] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:47 [I 2022-10-08 16:15:44] ...trap/engines/engine.py.229: process: 0.79337 | load: 0.00309 [I 2022-10-08 16:15:44] ...trap/engines/engine.py.230: loss: 0.21813 [I 2022-10-08 16:15:44] ...optimizers/trijoint.py.151: total_norm: 2.826509 [I 2022-10-08 16:15:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1030/2304 [I 2022-10-08 16:15:52] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:16:29 [I 2022-10-08 16:15:52] ...trap/engines/engine.py.229: process: 0.77142 | load: 0.00313 [I 2022-10-08 16:15:52] ...trap/engines/engine.py.230: loss: 0.25068 [I 2022-10-08 16:15:52] ...optimizers/trijoint.py.151: total_norm: 3.058824 [I 2022-10-08 16:16:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 1040/2304 [I 2022-10-08 16:16:00] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:16:24 [I 2022-10-08 16:16:00] ...trap/engines/engine.py.229: process: 0.78036 | load: 0.00317 [I 2022-10-08 16:16:00] ...trap/engines/engine.py.230: loss: 0.24734 [I 2022-10-08 16:16:00] ...optimizers/trijoint.py.151: total_norm: 3.072241 [I 2022-10-08 16:16:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 1050/2304 [I 2022-10-08 16:16:08] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:16:20 [I 2022-10-08 16:16:08] ...trap/engines/engine.py.229: process: 0.77975 | load: 0.00314 [I 2022-10-08 16:16:08] ...trap/engines/engine.py.230: loss: 0.19884 [I 2022-10-08 16:16:08] ...optimizers/trijoint.py.151: total_norm: 2.976553 [I 2022-10-08 16:16:15] ...trap/engines/engine.py.224: train: epoch 11 | batch 1060/2304 [I 2022-10-08 16:16:15] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:11 [I 2022-10-08 16:16:15] ...trap/engines/engine.py.229: process: 0.78577 | load: 0.00310 [I 2022-10-08 16:16:15] ...trap/engines/engine.py.230: loss: 0.21633 [I 2022-10-08 16:16:15] ...optimizers/trijoint.py.151: total_norm: 3.295849 [I 2022-10-08 16:16:23] ...trap/engines/engine.py.224: train: epoch 11 | batch 1070/2304 [I 2022-10-08 16:16:23] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:16:00 [I 2022-10-08 16:16:23] ...trap/engines/engine.py.229: process: 0.78233 | load: 0.00299 [I 2022-10-08 16:16:23] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-08 16:16:23] ...optimizers/trijoint.py.151: total_norm: 2.230443 [I 2022-10-08 16:16:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 1080/2304 [I 2022-10-08 16:16:31] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:15:58 [I 2022-10-08 16:16:31] ...trap/engines/engine.py.229: process: 0.78429 | load: 0.00302 [I 2022-10-08 16:16:31] ...trap/engines/engine.py.230: loss: 0.21716 [I 2022-10-08 16:16:31] ...optimizers/trijoint.py.151: total_norm: 3.251072 [I 2022-10-08 16:16:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1090/2304 [I 2022-10-08 16:16:39] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:15:56 [I 2022-10-08 16:16:39] ...trap/engines/engine.py.229: process: 0.80541 | load: 0.00300 [I 2022-10-08 16:16:39] ...trap/engines/engine.py.230: loss: 0.13927 [I 2022-10-08 16:16:39] ...optimizers/trijoint.py.151: total_norm: 2.046966 [I 2022-10-08 16:16:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 1100/2304 [I 2022-10-08 16:16:47] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:15:37 [I 2022-10-08 16:16:47] ...trap/engines/engine.py.229: process: 0.76833 | load: 0.00302 [I 2022-10-08 16:16:47] ...trap/engines/engine.py.230: loss: 0.21402 [I 2022-10-08 16:16:47] ...optimizers/trijoint.py.151: total_norm: 2.779622 [I 2022-10-08 16:16:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 1110/2304 [I 2022-10-08 16:16:55] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:15:37 [I 2022-10-08 16:16:55] ...trap/engines/engine.py.229: process: 0.81696 | load: 0.00304 [I 2022-10-08 16:16:55] ...trap/engines/engine.py.230: loss: 0.19769 [I 2022-10-08 16:16:55] ...optimizers/trijoint.py.151: total_norm: 2.475727 [I 2022-10-08 16:17:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 1120/2304 [I 2022-10-08 16:17:02] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:27 [I 2022-10-08 16:17:02] ...trap/engines/engine.py.229: process: 0.78189 | load: 0.00300 [I 2022-10-08 16:17:02] ...trap/engines/engine.py.230: loss: 0.17141 [I 2022-10-08 16:17:02] ...optimizers/trijoint.py.151: total_norm: 2.425788 [I 2022-10-08 16:17:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 1130/2304 [I 2022-10-08 16:17:10] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:23 [I 2022-10-08 16:17:10] ...trap/engines/engine.py.229: process: 0.78949 | load: 0.00301 [I 2022-10-08 16:17:10] ...trap/engines/engine.py.230: loss: 0.22392 [I 2022-10-08 16:17:10] ...optimizers/trijoint.py.151: total_norm: 2.213401 [I 2022-10-08 16:17:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 1140/2304 [I 2022-10-08 16:17:18] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:15:18 [I 2022-10-08 16:17:18] ...trap/engines/engine.py.229: process: 0.78363 | load: 0.00303 [I 2022-10-08 16:17:18] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-08 16:17:18] ...optimizers/trijoint.py.151: total_norm: 1.753602 [I 2022-10-08 16:17:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 1150/2304 [I 2022-10-08 16:17:26] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:15:03 [I 2022-10-08 16:17:26] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00302 [I 2022-10-08 16:17:26] ...trap/engines/engine.py.230: loss: 0.22298 [I 2022-10-08 16:17:26] ...optimizers/trijoint.py.151: total_norm: 3.152453 [I 2022-10-08 16:17:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1160/2304 [I 2022-10-08 16:17:34] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:14:53 [I 2022-10-08 16:17:34] ...trap/engines/engine.py.229: process: 0.77582 | load: 0.00301 [I 2022-10-08 16:17:34] ...trap/engines/engine.py.230: loss: 0.21654 [I 2022-10-08 16:17:34] ...optimizers/trijoint.py.151: total_norm: 2.790306 [I 2022-10-08 16:17:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 1170/2304 [I 2022-10-08 16:17:42] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:14:47 [I 2022-10-08 16:17:42] ...trap/engines/engine.py.229: process: 0.78256 | load: 0.00299 [I 2022-10-08 16:17:42] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-08 16:17:42] ...optimizers/trijoint.py.151: total_norm: 2.071532 [I 2022-10-08 16:17:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 1180/2304 [I 2022-10-08 16:17:50] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:14:38 [I 2022-10-08 16:17:50] ...trap/engines/engine.py.229: process: 0.77679 | load: 0.00298 [I 2022-10-08 16:17:50] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-08 16:17:50] ...optimizers/trijoint.py.151: total_norm: 2.492081 [I 2022-10-08 16:17:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1190/2304 [I 2022-10-08 16:17:57] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:14:30 [I 2022-10-08 16:17:57] ...trap/engines/engine.py.229: process: 0.78768 | load: 0.00305 [I 2022-10-08 16:17:57] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-08 16:17:57] ...optimizers/trijoint.py.151: total_norm: 2.045320 [I 2022-10-08 16:18:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 1200/2304 [I 2022-10-08 16:18:05] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:14:21 [I 2022-10-08 16:18:05] ...trap/engines/engine.py.229: process: 0.76874 | load: 0.00301 [I 2022-10-08 16:18:05] ...trap/engines/engine.py.230: loss: 0.19901 [I 2022-10-08 16:18:05] ...optimizers/trijoint.py.151: total_norm: 3.000715 [I 2022-10-08 16:18:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 1210/2304 [I 2022-10-08 16:18:13] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:14:17 [I 2022-10-08 16:18:13] ...trap/engines/engine.py.229: process: 0.78497 | load: 0.00308 [I 2022-10-08 16:18:13] ...trap/engines/engine.py.230: loss: 0.10758 [I 2022-10-08 16:18:13] ...optimizers/trijoint.py.151: total_norm: 1.607015 [I 2022-10-08 16:18:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 1220/2304 [I 2022-10-08 16:18:21] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:14:06 [I 2022-10-08 16:18:21] ...trap/engines/engine.py.229: process: 0.78078 | load: 0.00302 [I 2022-10-08 16:18:21] ...trap/engines/engine.py.230: loss: 0.18354 [I 2022-10-08 16:18:21] ...optimizers/trijoint.py.151: total_norm: 2.061842 [I 2022-10-08 16:18:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1230/2304 [I 2022-10-08 16:18:29] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:13:58 [I 2022-10-08 16:18:29] ...trap/engines/engine.py.229: process: 0.78306 | load: 0.00303 [I 2022-10-08 16:18:29] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-08 16:18:29] ...optimizers/trijoint.py.151: total_norm: 2.642709 [I 2022-10-08 16:18:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 1240/2304 [I 2022-10-08 16:18:37] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:13:46 [I 2022-10-08 16:18:37] ...trap/engines/engine.py.229: process: 0.77511 | load: 0.00302 [I 2022-10-08 16:18:37] ...trap/engines/engine.py.230: loss: 0.28179 [I 2022-10-08 16:18:37] ...optimizers/trijoint.py.151: total_norm: 3.568135 [I 2022-10-08 16:18:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1250/2304 [I 2022-10-08 16:18:44] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:13:35 [I 2022-10-08 16:18:44] ...trap/engines/engine.py.229: process: 0.76438 | load: 0.00300 [I 2022-10-08 16:18:44] ...trap/engines/engine.py.230: loss: 0.28525 [I 2022-10-08 16:18:44] ...optimizers/trijoint.py.151: total_norm: 3.137168 [I 2022-10-08 16:18:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1260/2304 [I 2022-10-08 16:18:52] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:13:30 [I 2022-10-08 16:18:52] ...trap/engines/engine.py.229: process: 0.76361 | load: 0.00305 [I 2022-10-08 16:18:52] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-08 16:18:52] ...optimizers/trijoint.py.151: total_norm: 2.775091 [I 2022-10-08 16:19:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 1270/2304 [I 2022-10-08 16:19:00] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:13:30 [I 2022-10-08 16:19:00] ...trap/engines/engine.py.229: process: 0.78349 | load: 0.00301 [I 2022-10-08 16:19:00] ...trap/engines/engine.py.230: loss: 0.17013 [I 2022-10-08 16:19:00] ...optimizers/trijoint.py.151: total_norm: 3.173626 [I 2022-10-08 16:19:08] ...trap/engines/engine.py.224: train: epoch 11 | batch 1280/2304 [I 2022-10-08 16:19:08] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:13:21 [I 2022-10-08 16:19:08] ...trap/engines/engine.py.229: process: 0.78215 | load: 0.00300 [I 2022-10-08 16:19:08] ...trap/engines/engine.py.230: loss: 0.18627 [I 2022-10-08 16:19:08] ...optimizers/trijoint.py.151: total_norm: 2.573922 [I 2022-10-08 16:19:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 1290/2304 [I 2022-10-08 16:19:16] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:13:08 [I 2022-10-08 16:19:16] ...trap/engines/engine.py.229: process: 0.77440 | load: 0.00300 [I 2022-10-08 16:19:16] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-08 16:19:16] ...optimizers/trijoint.py.151: total_norm: 1.983533 [I 2022-10-08 16:19:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1300/2304 [I 2022-10-08 16:19:24] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:13:01 [I 2022-10-08 16:19:24] ...trap/engines/engine.py.229: process: 0.77750 | load: 0.00302 [I 2022-10-08 16:19:24] ...trap/engines/engine.py.230: loss: 0.22349 [I 2022-10-08 16:19:24] ...optimizers/trijoint.py.151: total_norm: 3.068876 [I 2022-10-08 16:19:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 1310/2304 [I 2022-10-08 16:19:31] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:13:02 [I 2022-10-08 16:19:31] ...trap/engines/engine.py.229: process: 0.79137 | load: 0.00301 [I 2022-10-08 16:19:31] ...trap/engines/engine.py.230: loss: 0.20811 [I 2022-10-08 16:19:31] ...optimizers/trijoint.py.151: total_norm: 2.680863 [I 2022-10-08 16:19:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1320/2304 [I 2022-10-08 16:19:39] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:12:46 [I 2022-10-08 16:19:39] ...trap/engines/engine.py.229: process: 0.77532 | load: 0.00305 [I 2022-10-08 16:19:39] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-08 16:19:39] ...optimizers/trijoint.py.151: total_norm: 2.723629 [I 2022-10-08 16:19:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 1330/2304 [I 2022-10-08 16:19:47] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:12:38 [I 2022-10-08 16:19:47] ...trap/engines/engine.py.229: process: 0.78492 | load: 0.00305 [I 2022-10-08 16:19:47] ...trap/engines/engine.py.230: loss: 0.16986 [I 2022-10-08 16:19:47] ...optimizers/trijoint.py.151: total_norm: 2.382849 [I 2022-10-08 16:19:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 1340/2304 [I 2022-10-08 16:19:55] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:12:28 [I 2022-10-08 16:19:55] ...trap/engines/engine.py.229: process: 0.77537 | load: 0.00303 [I 2022-10-08 16:19:55] ...trap/engines/engine.py.230: loss: 0.26374 [I 2022-10-08 16:19:55] ...optimizers/trijoint.py.151: total_norm: 3.621753 [I 2022-10-08 16:20:03] ...trap/engines/engine.py.224: train: epoch 11 | batch 1350/2304 [I 2022-10-08 16:20:03] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:12:26 [I 2022-10-08 16:20:03] ...trap/engines/engine.py.229: process: 0.78865 | load: 0.00307 [I 2022-10-08 16:20:03] ...trap/engines/engine.py.230: loss: 0.14418 [I 2022-10-08 16:20:03] ...optimizers/trijoint.py.151: total_norm: 2.628979 [I 2022-10-08 16:20:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 1360/2304 [I 2022-10-08 16:20:10] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:12:15 [I 2022-10-08 16:20:10] ...trap/engines/engine.py.229: process: 0.78567 | load: 0.00302 [I 2022-10-08 16:20:10] ...trap/engines/engine.py.230: loss: 0.16417 [I 2022-10-08 16:20:10] ...optimizers/trijoint.py.151: total_norm: 2.259893 [I 2022-10-08 16:20:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 1370/2304 [I 2022-10-08 16:20:18] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:12:09 [I 2022-10-08 16:20:18] ...trap/engines/engine.py.229: process: 0.77351 | load: 0.00301 [I 2022-10-08 16:20:18] ...trap/engines/engine.py.230: loss: 0.26363 [I 2022-10-08 16:20:18] ...optimizers/trijoint.py.151: total_norm: 3.852248 [I 2022-10-08 16:20:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 1380/2304 [I 2022-10-08 16:20:26] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:12:03 [I 2022-10-08 16:20:26] ...trap/engines/engine.py.229: process: 0.78829 | load: 0.00301 [I 2022-10-08 16:20:26] ...trap/engines/engine.py.230: loss: 0.14120 [I 2022-10-08 16:20:26] ...optimizers/trijoint.py.151: total_norm: 2.433340 [I 2022-10-08 16:20:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1390/2304 [I 2022-10-08 16:20:34] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:11:52 [I 2022-10-08 16:20:34] ...trap/engines/engine.py.229: process: 0.77175 | load: 0.00304 [I 2022-10-08 16:20:34] ...trap/engines/engine.py.230: loss: 0.17913 [I 2022-10-08 16:20:34] ...optimizers/trijoint.py.151: total_norm: 2.130848 [I 2022-10-08 16:20:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 1400/2304 [I 2022-10-08 16:20:42] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:11:43 [I 2022-10-08 16:20:42] ...trap/engines/engine.py.229: process: 0.77203 | load: 0.00304 [I 2022-10-08 16:20:42] ...trap/engines/engine.py.230: loss: 0.23620 [I 2022-10-08 16:20:42] ...optimizers/trijoint.py.151: total_norm: 2.577396 [I 2022-10-08 16:20:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 1410/2304 [I 2022-10-08 16:20:50] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:11:35 [I 2022-10-08 16:20:50] ...trap/engines/engine.py.229: process: 0.78198 | load: 0.00307 [I 2022-10-08 16:20:50] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-08 16:20:50] ...optimizers/trijoint.py.151: total_norm: 2.104716 [I 2022-10-08 16:20:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1420/2304 [I 2022-10-08 16:20:57] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:11:30 [I 2022-10-08 16:20:57] ...trap/engines/engine.py.229: process: 0.79452 | load: 0.00301 [I 2022-10-08 16:20:57] ...trap/engines/engine.py.230: loss: 0.19623 [I 2022-10-08 16:20:57] ...optimizers/trijoint.py.151: total_norm: 2.637302 [I 2022-10-08 16:21:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 1430/2304 [I 2022-10-08 16:21:05] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:11:23 [I 2022-10-08 16:21:05] ...trap/engines/engine.py.229: process: 0.78662 | load: 0.00300 [I 2022-10-08 16:21:05] ...trap/engines/engine.py.230: loss: 0.17168 [I 2022-10-08 16:21:05] ...optimizers/trijoint.py.151: total_norm: 3.325935 [I 2022-10-08 16:21:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 1440/2304 [I 2022-10-08 16:21:13] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:11:12 [I 2022-10-08 16:21:13] ...trap/engines/engine.py.229: process: 0.77520 | load: 0.00301 [I 2022-10-08 16:21:13] ...trap/engines/engine.py.230: loss: 0.34316 [I 2022-10-08 16:21:13] ...optimizers/trijoint.py.151: total_norm: 3.779580 [I 2022-10-08 16:21:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 1450/2304 [I 2022-10-08 16:21:21] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:11:03 [I 2022-10-08 16:21:21] ...trap/engines/engine.py.229: process: 0.77120 | load: 0.00303 [I 2022-10-08 16:21:21] ...trap/engines/engine.py.230: loss: 0.21139 [I 2022-10-08 16:21:21] ...optimizers/trijoint.py.151: total_norm: 2.635449 [I 2022-10-08 16:21:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1460/2304 [I 2022-10-08 16:21:29] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:10:55 [I 2022-10-08 16:21:29] ...trap/engines/engine.py.229: process: 0.78117 | load: 0.00305 [I 2022-10-08 16:21:29] ...trap/engines/engine.py.230: loss: 0.17089 [I 2022-10-08 16:21:29] ...optimizers/trijoint.py.151: total_norm: 2.280254 [I 2022-10-08 16:21:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 1470/2304 [I 2022-10-08 16:21:36] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:10:49 [I 2022-10-08 16:21:36] ...trap/engines/engine.py.229: process: 0.78482 | load: 0.00300 [I 2022-10-08 16:21:36] ...trap/engines/engine.py.230: loss: 0.21442 [I 2022-10-08 16:21:36] ...optimizers/trijoint.py.151: total_norm: 3.205559 [I 2022-10-08 16:21:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1480/2304 [I 2022-10-08 16:21:44] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:10:43 [I 2022-10-08 16:21:44] ...trap/engines/engine.py.229: process: 0.77033 | load: 0.00297 [I 2022-10-08 16:21:44] ...trap/engines/engine.py.230: loss: 0.19842 [I 2022-10-08 16:21:44] ...optimizers/trijoint.py.151: total_norm: 3.209765 [I 2022-10-08 16:21:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1490/2304 [I 2022-10-08 16:21:52] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:10:36 [I 2022-10-08 16:21:52] ...trap/engines/engine.py.229: process: 0.78909 | load: 0.00299 [I 2022-10-08 16:21:52] ...trap/engines/engine.py.230: loss: 0.23748 [I 2022-10-08 16:21:52] ...optimizers/trijoint.py.151: total_norm: 3.485634 [I 2022-10-08 16:22:18] ...trap/engines/engine.py.224: train: epoch 11 | batch 1500/2304 [I 2022-10-08 16:22:18] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:49:10 [I 2022-10-08 16:22:18] ...trap/engines/engine.py.229: process: 0.78074 | load: 0.00301 [I 2022-10-08 16:22:18] ...trap/engines/engine.py.230: loss: 0.19793 [I 2022-10-08 16:22:18] ...optimizers/trijoint.py.151: total_norm: 2.509019 [I 2022-10-08 16:22:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 1510/2304 [I 2022-10-08 16:22:26] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:14:26 [I 2022-10-08 16:22:26] ...trap/engines/engine.py.229: process: 0.77595 | load: 0.00304 [I 2022-10-08 16:22:26] ...trap/engines/engine.py.230: loss: 0.20178 [I 2022-10-08 16:22:26] ...optimizers/trijoint.py.151: total_norm: 2.472230 [I 2022-10-08 16:22:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1520/2304 [I 2022-10-08 16:22:34] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:10:35 [I 2022-10-08 16:22:34] ...trap/engines/engine.py.229: process: 0.77404 | load: 0.00302 [I 2022-10-08 16:22:34] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-08 16:22:34] ...optimizers/trijoint.py.151: total_norm: 1.651373 [I 2022-10-08 16:22:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 1530/2304 [I 2022-10-08 16:22:41] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:10:05 [I 2022-10-08 16:22:41] ...trap/engines/engine.py.229: process: 0.78266 | load: 0.00302 [I 2022-10-08 16:22:41] ...trap/engines/engine.py.230: loss: 0.24258 [I 2022-10-08 16:22:41] ...optimizers/trijoint.py.151: total_norm: 3.249879 [I 2022-10-08 16:22:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1540/2304 [I 2022-10-08 16:22:49] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:09:57 [I 2022-10-08 16:22:49] ...trap/engines/engine.py.229: process: 0.78157 | load: 0.00302 [I 2022-10-08 16:22:49] ...trap/engines/engine.py.230: loss: 0.20524 [I 2022-10-08 16:22:49] ...optimizers/trijoint.py.151: total_norm: 3.390604 [I 2022-10-08 16:22:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1550/2304 [I 2022-10-08 16:22:57] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:09:48 [I 2022-10-08 16:22:57] ...trap/engines/engine.py.229: process: 0.78455 | load: 0.00303 [I 2022-10-08 16:22:57] ...trap/engines/engine.py.230: loss: 0.28370 [I 2022-10-08 16:22:57] ...optimizers/trijoint.py.151: total_norm: 3.121006 [I 2022-10-08 16:23:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 1560/2304 [I 2022-10-08 16:23:05] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:09:40 [I 2022-10-08 16:23:05] ...trap/engines/engine.py.229: process: 0.77087 | load: 0.00304 [I 2022-10-08 16:23:05] ...trap/engines/engine.py.230: loss: 0.22898 [I 2022-10-08 16:23:05] ...optimizers/trijoint.py.151: total_norm: 3.094137 [I 2022-10-08 16:23:13] ...trap/engines/engine.py.224: train: epoch 11 | batch 1570/2304 [I 2022-10-08 16:23:13] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:09:35 [I 2022-10-08 16:23:13] ...trap/engines/engine.py.229: process: 0.78291 | load: 0.00302 [I 2022-10-08 16:23:13] ...trap/engines/engine.py.230: loss: 0.19493 [I 2022-10-08 16:23:13] ...optimizers/trijoint.py.151: total_norm: 1.838788 [I 2022-10-08 16:23:20] ...trap/engines/engine.py.224: train: epoch 11 | batch 1580/2304 [I 2022-10-08 16:23:20] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:09:24 [I 2022-10-08 16:23:20] ...trap/engines/engine.py.229: process: 0.78623 | load: 0.00303 [I 2022-10-08 16:23:20] ...trap/engines/engine.py.230: loss: 0.15237 [I 2022-10-08 16:23:20] ...optimizers/trijoint.py.151: total_norm: 1.640930 [I 2022-10-08 16:23:28] ...trap/engines/engine.py.224: train: epoch 11 | batch 1590/2304 [I 2022-10-08 16:23:28] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:09:15 [I 2022-10-08 16:23:28] ...trap/engines/engine.py.229: process: 0.77657 | load: 0.00304 [I 2022-10-08 16:23:28] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-08 16:23:28] ...optimizers/trijoint.py.151: total_norm: 2.164115 [I 2022-10-08 16:23:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 1600/2304 [I 2022-10-08 16:23:36] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:09:06 [I 2022-10-08 16:23:36] ...trap/engines/engine.py.229: process: 0.78259 | load: 0.00306 [I 2022-10-08 16:23:36] ...trap/engines/engine.py.230: loss: 0.13979 [I 2022-10-08 16:23:36] ...optimizers/trijoint.py.151: total_norm: 2.347737 [I 2022-10-08 16:23:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1610/2304 [I 2022-10-08 16:23:44] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:08:58 [I 2022-10-08 16:23:44] ...trap/engines/engine.py.229: process: 0.77550 | load: 0.00306 [I 2022-10-08 16:23:44] ...trap/engines/engine.py.230: loss: 0.26245 [I 2022-10-08 16:23:44] ...optimizers/trijoint.py.151: total_norm: 3.681639 [I 2022-10-08 16:23:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1620/2304 [I 2022-10-08 16:23:52] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:08:54 [I 2022-10-08 16:23:52] ...trap/engines/engine.py.229: process: 0.79958 | load: 0.00301 [I 2022-10-08 16:23:52] ...trap/engines/engine.py.230: loss: 0.21258 [I 2022-10-08 16:23:52] ...optimizers/trijoint.py.151: total_norm: 3.124127 [I 2022-10-08 16:23:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1630/2304 [I 2022-10-08 16:23:59] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:08:43 [I 2022-10-08 16:23:59] ...trap/engines/engine.py.229: process: 0.77022 | load: 0.00309 [I 2022-10-08 16:23:59] ...trap/engines/engine.py.230: loss: 0.17630 [I 2022-10-08 16:23:59] ...optimizers/trijoint.py.151: total_norm: 2.784004 [I 2022-10-08 16:24:07] ...trap/engines/engine.py.224: train: epoch 11 | batch 1640/2304 [I 2022-10-08 16:24:07] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:08:36 [I 2022-10-08 16:24:07] ...trap/engines/engine.py.229: process: 0.76846 | load: 0.00299 [I 2022-10-08 16:24:07] ...trap/engines/engine.py.230: loss: 0.20922 [I 2022-10-08 16:24:07] ...optimizers/trijoint.py.151: total_norm: 2.943088 [I 2022-10-08 16:24:15] ...trap/engines/engine.py.224: train: epoch 11 | batch 1650/2304 [I 2022-10-08 16:24:15] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:08:29 [I 2022-10-08 16:24:15] ...trap/engines/engine.py.229: process: 0.78459 | load: 0.00300 [I 2022-10-08 16:24:15] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-08 16:24:15] ...optimizers/trijoint.py.151: total_norm: 2.240877 [I 2022-10-08 16:24:23] ...trap/engines/engine.py.224: train: epoch 11 | batch 1660/2304 [I 2022-10-08 16:24:23] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:08:20 [I 2022-10-08 16:24:23] ...trap/engines/engine.py.229: process: 0.77986 | load: 0.00303 [I 2022-10-08 16:24:23] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-08 16:24:23] ...optimizers/trijoint.py.151: total_norm: 2.079089 [I 2022-10-08 16:24:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 1670/2304 [I 2022-10-08 16:24:31] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:08:14 [I 2022-10-08 16:24:31] ...trap/engines/engine.py.229: process: 0.77900 | load: 0.00303 [I 2022-10-08 16:24:31] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-08 16:24:31] ...optimizers/trijoint.py.151: total_norm: 1.758514 [I 2022-10-08 16:24:38] ...trap/engines/engine.py.224: train: epoch 11 | batch 1680/2304 [I 2022-10-08 16:24:38] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:08:07 [I 2022-10-08 16:24:38] ...trap/engines/engine.py.229: process: 0.77215 | load: 0.00305 [I 2022-10-08 16:24:38] ...trap/engines/engine.py.230: loss: 0.18105 [I 2022-10-08 16:24:38] ...optimizers/trijoint.py.151: total_norm: 2.572090 [I 2022-10-08 16:24:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 1690/2304 [I 2022-10-08 16:24:46] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:07:57 [I 2022-10-08 16:24:46] ...trap/engines/engine.py.229: process: 0.77605 | load: 0.00302 [I 2022-10-08 16:24:46] ...trap/engines/engine.py.230: loss: 0.22717 [I 2022-10-08 16:24:46] ...optimizers/trijoint.py.151: total_norm: 3.568850 [I 2022-10-08 16:24:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1700/2304 [I 2022-10-08 16:24:54] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:07:50 [I 2022-10-08 16:24:54] ...trap/engines/engine.py.229: process: 0.76716 | load: 0.00306 [I 2022-10-08 16:24:54] ...trap/engines/engine.py.230: loss: 0.14488 [I 2022-10-08 16:24:54] ...optimizers/trijoint.py.151: total_norm: 2.599956 [I 2022-10-08 16:25:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 1710/2304 [I 2022-10-08 16:25:02] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:07:42 [I 2022-10-08 16:25:02] ...trap/engines/engine.py.229: process: 0.77683 | load: 0.00303 [I 2022-10-08 16:25:02] ...trap/engines/engine.py.230: loss: 0.22974 [I 2022-10-08 16:25:02] ...optimizers/trijoint.py.151: total_norm: 3.425028 [I 2022-10-08 16:25:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 1720/2304 [I 2022-10-08 16:25:10] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:07:34 [I 2022-10-08 16:25:10] ...trap/engines/engine.py.229: process: 0.77945 | load: 0.00302 [I 2022-10-08 16:25:10] ...trap/engines/engine.py.230: loss: 0.23294 [I 2022-10-08 16:25:10] ...optimizers/trijoint.py.151: total_norm: 3.115619 [I 2022-10-08 16:25:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 1730/2304 [I 2022-10-08 16:25:17] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:07:26 [I 2022-10-08 16:25:17] ...trap/engines/engine.py.229: process: 0.79093 | load: 0.00303 [I 2022-10-08 16:25:17] ...trap/engines/engine.py.230: loss: 0.16956 [I 2022-10-08 16:25:17] ...optimizers/trijoint.py.151: total_norm: 1.728224 [I 2022-10-08 16:25:25] ...trap/engines/engine.py.224: train: epoch 11 | batch 1740/2304 [I 2022-10-08 16:25:25] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:07:18 [I 2022-10-08 16:25:25] ...trap/engines/engine.py.229: process: 0.77945 | load: 0.00307 [I 2022-10-08 16:25:25] ...trap/engines/engine.py.230: loss: 0.16009 [I 2022-10-08 16:25:25] ...optimizers/trijoint.py.151: total_norm: 2.210815 [I 2022-10-08 16:25:33] ...trap/engines/engine.py.224: train: epoch 11 | batch 1750/2304 [I 2022-10-08 16:25:33] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:07:11 [I 2022-10-08 16:25:33] ...trap/engines/engine.py.229: process: 0.76926 | load: 0.00305 [I 2022-10-08 16:25:33] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-08 16:25:33] ...optimizers/trijoint.py.151: total_norm: 2.732504 [I 2022-10-08 16:25:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 1760/2304 [I 2022-10-08 16:25:41] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:07:03 [I 2022-10-08 16:25:41] ...trap/engines/engine.py.229: process: 0.78610 | load: 0.00304 [I 2022-10-08 16:25:41] ...trap/engines/engine.py.230: loss: 0.20755 [I 2022-10-08 16:25:41] ...optimizers/trijoint.py.151: total_norm: 3.110605 [I 2022-10-08 16:25:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1770/2304 [I 2022-10-08 16:25:49] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:06:54 [I 2022-10-08 16:25:49] ...trap/engines/engine.py.229: process: 0.77134 | load: 0.00305 [I 2022-10-08 16:25:49] ...trap/engines/engine.py.230: loss: 0.17906 [I 2022-10-08 16:25:49] ...optimizers/trijoint.py.151: total_norm: 3.032381 [I 2022-10-08 16:25:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 1780/2304 [I 2022-10-08 16:25:56] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:06:45 [I 2022-10-08 16:25:56] ...trap/engines/engine.py.229: process: 0.77587 | load: 0.00308 [I 2022-10-08 16:25:56] ...trap/engines/engine.py.230: loss: 0.21354 [I 2022-10-08 16:25:57] ...optimizers/trijoint.py.151: total_norm: 2.897512 [I 2022-10-08 16:26:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1790/2304 [I 2022-10-08 16:26:04] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:06:41 [I 2022-10-08 16:26:04] ...trap/engines/engine.py.229: process: 0.78625 | load: 0.00304 [I 2022-10-08 16:26:04] ...trap/engines/engine.py.230: loss: 0.26359 [I 2022-10-08 16:26:04] ...optimizers/trijoint.py.151: total_norm: 2.667672 [I 2022-10-08 16:26:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 1800/2304 [I 2022-10-08 16:26:12] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:06:31 [I 2022-10-08 16:26:12] ...trap/engines/engine.py.229: process: 0.76250 | load: 0.00304 [I 2022-10-08 16:26:12] ...trap/engines/engine.py.230: loss: 0.24000 [I 2022-10-08 16:26:12] ...optimizers/trijoint.py.151: total_norm: 3.272346 [I 2022-10-08 16:26:20] ...trap/engines/engine.py.224: train: epoch 11 | batch 1810/2304 [I 2022-10-08 16:26:20] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:06:26 [I 2022-10-08 16:26:20] ...trap/engines/engine.py.229: process: 0.78866 | load: 0.00307 [I 2022-10-08 16:26:20] ...trap/engines/engine.py.230: loss: 0.20602 [I 2022-10-08 16:26:20] ...optimizers/trijoint.py.151: total_norm: 2.609908 [I 2022-10-08 16:26:28] ...trap/engines/engine.py.224: train: epoch 11 | batch 1820/2304 [I 2022-10-08 16:26:28] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:06:18 [I 2022-10-08 16:26:28] ...trap/engines/engine.py.229: process: 0.77233 | load: 0.00301 [I 2022-10-08 16:26:28] ...trap/engines/engine.py.230: loss: 0.22309 [I 2022-10-08 16:26:28] ...optimizers/trijoint.py.151: total_norm: 2.878947 [I 2022-10-08 16:26:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 1830/2304 [I 2022-10-08 16:26:36] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:06:09 [I 2022-10-08 16:26:36] ...trap/engines/engine.py.229: process: 0.77564 | load: 0.00301 [I 2022-10-08 16:26:36] ...trap/engines/engine.py.230: loss: 0.26539 [I 2022-10-08 16:26:36] ...optimizers/trijoint.py.151: total_norm: 2.559080 [I 2022-10-08 16:26:43] ...trap/engines/engine.py.224: train: epoch 11 | batch 1840/2304 [I 2022-10-08 16:26:43] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:06:01 [I 2022-10-08 16:26:43] ...trap/engines/engine.py.229: process: 0.76552 | load: 0.00305 [I 2022-10-08 16:26:43] ...trap/engines/engine.py.230: loss: 0.17120 [I 2022-10-08 16:26:43] ...optimizers/trijoint.py.151: total_norm: 2.344247 [I 2022-10-08 16:26:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 1850/2304 [I 2022-10-08 16:26:51] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:05:54 [I 2022-10-08 16:26:51] ...trap/engines/engine.py.229: process: 0.78521 | load: 0.00301 [I 2022-10-08 16:26:51] ...trap/engines/engine.py.230: loss: 0.20356 [I 2022-10-08 16:26:51] ...optimizers/trijoint.py.151: total_norm: 2.455050 [I 2022-10-08 16:26:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1860/2304 [I 2022-10-08 16:26:59] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:05:47 [I 2022-10-08 16:26:59] ...trap/engines/engine.py.229: process: 0.77682 | load: 0.00300 [I 2022-10-08 16:26:59] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-08 16:26:59] ...optimizers/trijoint.py.151: total_norm: 1.638955 [I 2022-10-08 16:27:07] ...trap/engines/engine.py.224: train: epoch 11 | batch 1870/2304 [I 2022-10-08 16:27:07] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:05:38 [I 2022-10-08 16:27:07] ...trap/engines/engine.py.229: process: 0.76471 | load: 0.00309 [I 2022-10-08 16:27:07] ...trap/engines/engine.py.230: loss: 0.17347 [I 2022-10-08 16:27:07] ...optimizers/trijoint.py.151: total_norm: 2.392259 [I 2022-10-08 16:27:15] ...trap/engines/engine.py.224: train: epoch 11 | batch 1880/2304 [I 2022-10-08 16:27:15] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:05:29 [I 2022-10-08 16:27:15] ...trap/engines/engine.py.229: process: 0.76922 | load: 0.00306 [I 2022-10-08 16:27:15] ...trap/engines/engine.py.230: loss: 0.25839 [I 2022-10-08 16:27:15] ...optimizers/trijoint.py.151: total_norm: 2.934257 [I 2022-10-08 16:27:22] ...trap/engines/engine.py.224: train: epoch 11 | batch 1890/2304 [I 2022-10-08 16:27:22] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:05:20 [I 2022-10-08 16:27:22] ...trap/engines/engine.py.229: process: 0.76338 | load: 0.00306 [I 2022-10-08 16:27:22] ...trap/engines/engine.py.230: loss: 0.18961 [I 2022-10-08 16:27:22] ...optimizers/trijoint.py.151: total_norm: 2.895947 [I 2022-10-08 16:27:30] ...trap/engines/engine.py.224: train: epoch 11 | batch 1900/2304 [I 2022-10-08 16:27:30] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:05:14 [I 2022-10-08 16:27:30] ...trap/engines/engine.py.229: process: 0.77317 | load: 0.00312 [I 2022-10-08 16:27:30] ...trap/engines/engine.py.230: loss: 0.17986 [I 2022-10-08 16:27:30] ...optimizers/trijoint.py.151: total_norm: 2.377121 [I 2022-10-08 16:27:38] ...trap/engines/engine.py.224: train: epoch 11 | batch 1910/2304 [I 2022-10-08 16:27:38] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:05:06 [I 2022-10-08 16:27:38] ...trap/engines/engine.py.229: process: 0.77040 | load: 0.00306 [I 2022-10-08 16:27:38] ...trap/engines/engine.py.230: loss: 0.20027 [I 2022-10-08 16:27:38] ...optimizers/trijoint.py.151: total_norm: 2.154264 [I 2022-10-08 16:27:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 1920/2304 [I 2022-10-08 16:27:46] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:04:59 [I 2022-10-08 16:27:46] ...trap/engines/engine.py.229: process: 0.76804 | load: 0.00304 [I 2022-10-08 16:27:46] ...trap/engines/engine.py.230: loss: 0.21329 [I 2022-10-08 16:27:46] ...optimizers/trijoint.py.151: total_norm: 3.279508 [I 2022-10-08 16:27:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1930/2304 [I 2022-10-08 16:27:54] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:04:51 [I 2022-10-08 16:27:54] ...trap/engines/engine.py.229: process: 0.76892 | load: 0.00296 [I 2022-10-08 16:27:54] ...trap/engines/engine.py.230: loss: 0.19653 [I 2022-10-08 16:27:54] ...optimizers/trijoint.py.151: total_norm: 2.731762 [I 2022-10-08 16:28:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 1940/2304 [I 2022-10-08 16:28:01] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:04:41 [I 2022-10-08 16:28:01] ...trap/engines/engine.py.229: process: 0.76568 | load: 0.00304 [I 2022-10-08 16:28:01] ...trap/engines/engine.py.230: loss: 0.14965 [I 2022-10-08 16:28:01] ...optimizers/trijoint.py.151: total_norm: 2.407991 [I 2022-10-08 16:28:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1950/2304 [I 2022-10-08 16:28:09] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:04:35 [I 2022-10-08 16:28:09] ...trap/engines/engine.py.229: process: 0.79244 | load: 0.00304 [I 2022-10-08 16:28:09] ...trap/engines/engine.py.230: loss: 0.25485 [I 2022-10-08 16:28:09] ...optimizers/trijoint.py.151: total_norm: 3.543924 [I 2022-10-08 16:28:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 1960/2304 [I 2022-10-08 16:28:17] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:04:28 [I 2022-10-08 16:28:17] ...trap/engines/engine.py.229: process: 0.78533 | load: 0.00301 [I 2022-10-08 16:28:17] ...trap/engines/engine.py.230: loss: 0.23284 [I 2022-10-08 16:28:17] ...optimizers/trijoint.py.151: total_norm: 2.976161 [I 2022-10-08 16:28:25] ...trap/engines/engine.py.224: train: epoch 11 | batch 1970/2304 [I 2022-10-08 16:28:25] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:04:20 [I 2022-10-08 16:28:25] ...trap/engines/engine.py.229: process: 0.77387 | load: 0.00306 [I 2022-10-08 16:28:25] ...trap/engines/engine.py.230: loss: 0.21287 [I 2022-10-08 16:28:25] ...optimizers/trijoint.py.151: total_norm: 2.548610 [I 2022-10-08 16:28:33] ...trap/engines/engine.py.224: train: epoch 11 | batch 1980/2304 [I 2022-10-08 16:28:33] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:04:13 [I 2022-10-08 16:28:33] ...trap/engines/engine.py.229: process: 0.78836 | load: 0.00300 [I 2022-10-08 16:28:33] ...trap/engines/engine.py.230: loss: 0.28144 [I 2022-10-08 16:28:33] ...optimizers/trijoint.py.151: total_norm: 2.916436 [I 2022-10-08 16:28:40] ...trap/engines/engine.py.224: train: epoch 11 | batch 1990/2304 [I 2022-10-08 16:28:40] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:04:04 [I 2022-10-08 16:28:40] ...trap/engines/engine.py.229: process: 0.78420 | load: 0.00301 [I 2022-10-08 16:28:40] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-08 16:28:40] ...optimizers/trijoint.py.151: total_norm: 3.285859 [I 2022-10-08 16:28:48] ...trap/engines/engine.py.224: train: epoch 11 | batch 2000/2304 [I 2022-10-08 16:28:48] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:03:57 [I 2022-10-08 16:28:48] ...trap/engines/engine.py.229: process: 0.78182 | load: 0.00308 [I 2022-10-08 16:28:48] ...trap/engines/engine.py.230: loss: 0.17771 [I 2022-10-08 16:28:48] ...optimizers/trijoint.py.151: total_norm: 2.069370 [I 2022-10-08 16:28:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 2010/2304 [I 2022-10-08 16:28:56] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:03:48 [I 2022-10-08 16:28:56] ...trap/engines/engine.py.229: process: 0.78689 | load: 0.00300 [I 2022-10-08 16:28:56] ...trap/engines/engine.py.230: loss: 0.25877 [I 2022-10-08 16:28:56] ...optimizers/trijoint.py.151: total_norm: 2.905201 [I 2022-10-08 16:29:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 2020/2304 [I 2022-10-08 16:29:04] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:03:41 [I 2022-10-08 16:29:04] ...trap/engines/engine.py.229: process: 0.78773 | load: 0.00308 [I 2022-10-08 16:29:04] ...trap/engines/engine.py.230: loss: 0.25092 [I 2022-10-08 16:29:04] ...optimizers/trijoint.py.151: total_norm: 3.023106 [I 2022-10-08 16:29:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 2030/2304 [I 2022-10-08 16:29:12] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:03:32 [I 2022-10-08 16:29:12] ...trap/engines/engine.py.229: process: 0.76649 | load: 0.00303 [I 2022-10-08 16:29:12] ...trap/engines/engine.py.230: loss: 0.15744 [I 2022-10-08 16:29:12] ...optimizers/trijoint.py.151: total_norm: 2.438097 [I 2022-10-08 16:29:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 2040/2304 [I 2022-10-08 16:29:19] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:03:25 [I 2022-10-08 16:29:19] ...trap/engines/engine.py.229: process: 0.78476 | load: 0.00301 [I 2022-10-08 16:29:19] ...trap/engines/engine.py.230: loss: 0.23268 [I 2022-10-08 16:29:19] ...optimizers/trijoint.py.151: total_norm: 3.538290 [I 2022-10-08 16:29:27] ...trap/engines/engine.py.224: train: epoch 11 | batch 2050/2304 [I 2022-10-08 16:29:27] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:03:16 [I 2022-10-08 16:29:27] ...trap/engines/engine.py.229: process: 0.77273 | load: 0.00303 [I 2022-10-08 16:29:27] ...trap/engines/engine.py.230: loss: 0.24315 [I 2022-10-08 16:29:27] ...optimizers/trijoint.py.151: total_norm: 2.454358 [I 2022-10-08 16:29:35] ...trap/engines/engine.py.224: train: epoch 11 | batch 2060/2304 [I 2022-10-08 16:29:35] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:03:09 [I 2022-10-08 16:29:35] ...trap/engines/engine.py.229: process: 0.77038 | load: 0.00299 [I 2022-10-08 16:29:35] ...trap/engines/engine.py.230: loss: 0.19756 [I 2022-10-08 16:29:35] ...optimizers/trijoint.py.151: total_norm: 2.744925 [I 2022-10-08 16:29:43] ...trap/engines/engine.py.224: train: epoch 11 | batch 2070/2304 [I 2022-10-08 16:29:43] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:03:00 [I 2022-10-08 16:29:43] ...trap/engines/engine.py.229: process: 0.77764 | load: 0.00306 [I 2022-10-08 16:29:43] ...trap/engines/engine.py.230: loss: 0.32612 [I 2022-10-08 16:29:43] ...optimizers/trijoint.py.151: total_norm: 3.968884 [I 2022-10-08 16:29:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 2080/2304 [I 2022-10-08 16:29:51] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:02:54 [I 2022-10-08 16:29:51] ...trap/engines/engine.py.229: process: 0.77465 | load: 0.00302 [I 2022-10-08 16:29:51] ...trap/engines/engine.py.230: loss: 0.25547 [I 2022-10-08 16:29:51] ...optimizers/trijoint.py.151: total_norm: 2.737709 [I 2022-10-08 16:29:58] ...trap/engines/engine.py.224: train: epoch 11 | batch 2090/2304 [I 2022-10-08 16:29:58] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:02:46 [I 2022-10-08 16:29:58] ...trap/engines/engine.py.229: process: 0.77145 | load: 0.00302 [I 2022-10-08 16:29:58] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-08 16:29:58] ...optimizers/trijoint.py.151: total_norm: 1.649843 [I 2022-10-08 16:30:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 2100/2304 [I 2022-10-08 16:30:06] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:02:38 [I 2022-10-08 16:30:06] ...trap/engines/engine.py.229: process: 0.78279 | load: 0.00302 [I 2022-10-08 16:30:06] ...trap/engines/engine.py.230: loss: 0.20693 [I 2022-10-08 16:30:06] ...optimizers/trijoint.py.151: total_norm: 3.052170 [I 2022-10-08 16:30:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 2110/2304 [I 2022-10-08 16:30:14] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:02:31 [I 2022-10-08 16:30:14] ...trap/engines/engine.py.229: process: 0.78512 | load: 0.00303 [I 2022-10-08 16:30:14] ...trap/engines/engine.py.230: loss: 0.25708 [I 2022-10-08 16:30:14] ...optimizers/trijoint.py.151: total_norm: 2.544512 [I 2022-10-08 16:30:22] ...trap/engines/engine.py.224: train: epoch 11 | batch 2120/2304 [I 2022-10-08 16:30:22] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:02:23 [I 2022-10-08 16:30:22] ...trap/engines/engine.py.229: process: 0.78569 | load: 0.00302 [I 2022-10-08 16:30:22] ...trap/engines/engine.py.230: loss: 0.20461 [I 2022-10-08 16:30:22] ...optimizers/trijoint.py.151: total_norm: 2.540322 [I 2022-10-08 16:30:30] ...trap/engines/engine.py.224: train: epoch 11 | batch 2130/2304 [I 2022-10-08 16:30:30] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:02:14 [I 2022-10-08 16:30:30] ...trap/engines/engine.py.229: process: 0.77733 | load: 0.00306 [I 2022-10-08 16:30:30] ...trap/engines/engine.py.230: loss: 0.18863 [I 2022-10-08 16:30:30] ...optimizers/trijoint.py.151: total_norm: 2.675419 [I 2022-10-08 16:30:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 2140/2304 [I 2022-10-08 16:30:37] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:02:07 [I 2022-10-08 16:30:37] ...trap/engines/engine.py.229: process: 0.78087 | load: 0.00302 [I 2022-10-08 16:30:37] ...trap/engines/engine.py.230: loss: 0.29060 [I 2022-10-08 16:30:37] ...optimizers/trijoint.py.151: total_norm: 2.971004 [I 2022-10-08 16:30:45] ...trap/engines/engine.py.224: train: epoch 11 | batch 2150/2304 [I 2022-10-08 16:30:45] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:02:00 [I 2022-10-08 16:30:45] ...trap/engines/engine.py.229: process: 0.78100 | load: 0.00303 [I 2022-10-08 16:30:45] ...trap/engines/engine.py.230: loss: 0.24427 [I 2022-10-08 16:30:45] ...optimizers/trijoint.py.151: total_norm: 3.231254 [I 2022-10-08 16:30:53] ...trap/engines/engine.py.224: train: epoch 11 | batch 2160/2304 [I 2022-10-08 16:30:53] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:01:53 [I 2022-10-08 16:30:53] ...trap/engines/engine.py.229: process: 0.78877 | load: 0.00302 [I 2022-10-08 16:30:53] ...trap/engines/engine.py.230: loss: 0.21412 [I 2022-10-08 16:30:53] ...optimizers/trijoint.py.151: total_norm: 2.906890 [I 2022-10-08 16:31:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 2170/2304 [I 2022-10-08 16:31:01] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:01:45 [I 2022-10-08 16:31:01] ...trap/engines/engine.py.229: process: 0.79367 | load: 0.00300 [I 2022-10-08 16:31:01] ...trap/engines/engine.py.230: loss: 0.09396 [I 2022-10-08 16:31:01] ...optimizers/trijoint.py.151: total_norm: 1.778288 [I 2022-10-08 16:31:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 2180/2304 [I 2022-10-08 16:31:09] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:01:37 [I 2022-10-08 16:31:09] ...trap/engines/engine.py.229: process: 0.77294 | load: 0.00300 [I 2022-10-08 16:31:09] ...trap/engines/engine.py.230: loss: 0.17498 [I 2022-10-08 16:31:09] ...optimizers/trijoint.py.151: total_norm: 1.661051 [I 2022-10-08 16:31:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 2190/2304 [I 2022-10-08 16:31:17] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:01:29 [I 2022-10-08 16:31:17] ...trap/engines/engine.py.229: process: 0.77896 | load: 0.00301 [I 2022-10-08 16:31:17] ...trap/engines/engine.py.230: loss: 0.22127 [I 2022-10-08 16:31:17] ...optimizers/trijoint.py.151: total_norm: 3.048692 [I 2022-10-08 16:31:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 2200/2304 [I 2022-10-08 16:31:24] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:01:20 [I 2022-10-08 16:31:24] ...trap/engines/engine.py.229: process: 0.76089 | load: 0.00300 [I 2022-10-08 16:31:24] ...trap/engines/engine.py.230: loss: 0.15914 [I 2022-10-08 16:31:24] ...optimizers/trijoint.py.151: total_norm: 2.167562 [I 2022-10-08 16:31:32] ...trap/engines/engine.py.224: train: epoch 11 | batch 2210/2304 [I 2022-10-08 16:31:32] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:01:13 [I 2022-10-08 16:31:32] ...trap/engines/engine.py.229: process: 0.79094 | load: 0.00301 [I 2022-10-08 16:31:32] ...trap/engines/engine.py.230: loss: 0.20640 [I 2022-10-08 16:31:32] ...optimizers/trijoint.py.151: total_norm: 2.511882 [I 2022-10-08 16:31:40] ...trap/engines/engine.py.224: train: epoch 11 | batch 2220/2304 [I 2022-10-08 16:31:40] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:01:05 [I 2022-10-08 16:31:40] ...trap/engines/engine.py.229: process: 0.76110 | load: 0.00302 [I 2022-10-08 16:31:40] ...trap/engines/engine.py.230: loss: 0.18085 [I 2022-10-08 16:31:40] ...optimizers/trijoint.py.151: total_norm: 3.038505 [I 2022-10-08 16:31:48] ...trap/engines/engine.py.224: train: epoch 11 | batch 2230/2304 [I 2022-10-08 16:31:48] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:00:57 [I 2022-10-08 16:31:48] ...trap/engines/engine.py.229: process: 0.78760 | load: 0.00304 [I 2022-10-08 16:31:48] ...trap/engines/engine.py.230: loss: 0.21701 [I 2022-10-08 16:31:48] ...optimizers/trijoint.py.151: total_norm: 4.022706 [I 2022-10-08 16:31:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 2240/2304 [I 2022-10-08 16:31:56] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:00:51 [I 2022-10-08 16:31:56] ...trap/engines/engine.py.229: process: 0.78277 | load: 0.00317 [I 2022-10-08 16:31:56] ...trap/engines/engine.py.230: loss: 0.25394 [I 2022-10-08 16:31:56] ...optimizers/trijoint.py.151: total_norm: 3.190209 [I 2022-10-08 16:32:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 2250/2304 [I 2022-10-08 16:32:04] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:00:42 [I 2022-10-08 16:32:04] ...trap/engines/engine.py.229: process: 0.78939 | load: 0.00311 [I 2022-10-08 16:32:04] ...trap/engines/engine.py.230: loss: 0.24142 [I 2022-10-08 16:32:04] ...optimizers/trijoint.py.151: total_norm: 3.124475 [I 2022-10-08 16:32:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 2260/2304 [I 2022-10-08 16:32:12] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:00:34 [I 2022-10-08 16:32:12] ...trap/engines/engine.py.229: process: 0.76981 | load: 0.00311 [I 2022-10-08 16:32:12] ...trap/engines/engine.py.230: loss: 0.26401 [I 2022-10-08 16:32:12] ...optimizers/trijoint.py.151: total_norm: 2.694488 [I 2022-10-08 16:32:20] ...trap/engines/engine.py.224: train: epoch 11 | batch 2270/2304 [I 2022-10-08 16:32:20] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:00:26 [I 2022-10-08 16:32:20] ...trap/engines/engine.py.229: process: 0.77941 | load: 0.00303 [I 2022-10-08 16:32:20] ...trap/engines/engine.py.230: loss: 0.15992 [I 2022-10-08 16:32:20] ...optimizers/trijoint.py.151: total_norm: 1.788198 [I 2022-10-08 16:32:28] ...trap/engines/engine.py.224: train: epoch 11 | batch 2280/2304 [I 2022-10-08 16:32:28] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:00:18 [I 2022-10-08 16:32:28] ...trap/engines/engine.py.229: process: 0.78451 | load: 0.00303 [I 2022-10-08 16:32:28] ...trap/engines/engine.py.230: loss: 0.16968 [I 2022-10-08 16:32:28] ...optimizers/trijoint.py.151: total_norm: 1.845285 [I 2022-10-08 16:32:35] ...trap/engines/engine.py.224: train: epoch 11 | batch 2290/2304 [I 2022-10-08 16:32:35] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:00:10 [I 2022-10-08 16:32:35] ...trap/engines/engine.py.229: process: 0.77998 | load: 0.00300 [I 2022-10-08 16:32:35] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-08 16:32:35] ...optimizers/trijoint.py.151: total_norm: 2.758071 [I 2022-10-08 16:32:43] ...trap/engines/engine.py.224: train: epoch 11 | batch 2300/2304 [I 2022-10-08 16:32:43] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:00:03 [I 2022-10-08 16:32:43] ...trap/engines/engine.py.229: process: 0.75855 | load: 0.00016 [I 2022-10-08 16:32:43] ...trap/engines/engine.py.230: loss: 0.20855 [I 2022-10-08 16:32:43] ...optimizers/trijoint.py.151: total_norm: 2.525928 [I 2022-10-08 16:32:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 2304/2304 [I 2022-10-08 16:32:46] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:00:00 [I 2022-10-08 16:32:46] ...trap/engines/engine.py.229: process: 0.76418 | load: 0.00018 [I 2022-10-08 16:32:46] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-08 16:32:46] ...optimizers/trijoint.py.151: total_norm: 2.571278 [S 2022-10-08 16:32:48] ...trap/engines/engine.py.129: train_epoch.epoch: 11 [S 2022-10-08 16:32:48] ...trap/engines/engine.py.129: train_epoch.loss: 0.19685457720867724 [I 2022-10-08 16:32:48] ...s/criterions/triplet.py.84: margin = 0.11000000000000004 [I 2022-10-08 16:32:48] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 11 [I 2022-10-08 16:32:48] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 16:32:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 16:32:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 16:32:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 16:32:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 16:32:50] ...trap/engines/engine.py.321: eval: epoch 11 | batch 0/510 [I 2022-10-08 16:32:50] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:54 [I 2022-10-08 16:32:50] ...trap/engines/engine.py.326: process: 2.57670 | load: 2.07110 [I 2022-10-08 16:32:56] ...trap/engines/engine.py.321: eval: epoch 11 | batch 10/510 [I 2022-10-08 16:32:56] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:12 [I 2022-10-08 16:32:56] ...trap/engines/engine.py.326: process: 0.42156 | load: 0.00017 [I 2022-10-08 16:33:02] ...trap/engines/engine.py.321: eval: epoch 11 | batch 20/510 [I 2022-10-08 16:33:02] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:27 [I 2022-10-08 16:33:02] ...trap/engines/engine.py.326: process: 0.60937 | load: 0.00019 [I 2022-10-08 16:33:08] ...trap/engines/engine.py.321: eval: epoch 11 | batch 30/510 [I 2022-10-08 16:33:08] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:17 [I 2022-10-08 16:33:08] ...trap/engines/engine.py.326: process: 0.53534 | load: 0.00019 [I 2022-10-08 16:33:14] ...trap/engines/engine.py.321: eval: epoch 11 | batch 40/510 [I 2022-10-08 16:33:14] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:18 [I 2022-10-08 16:33:14] ...trap/engines/engine.py.326: process: 0.45652 | load: 0.00018 [I 2022-10-08 16:33:19] ...trap/engines/engine.py.321: eval: epoch 11 | batch 50/510 [I 2022-10-08 16:33:19] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:57 [I 2022-10-08 16:33:19] ...trap/engines/engine.py.326: process: 0.49985 | load: 0.00019 [I 2022-10-08 16:33:23] ...trap/engines/engine.py.321: eval: epoch 11 | batch 60/510 [I 2022-10-08 16:33:23] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:33 [I 2022-10-08 16:33:23] ...trap/engines/engine.py.326: process: 0.55598 | load: 0.00019 [I 2022-10-08 16:33:29] ...trap/engines/engine.py.321: eval: epoch 11 | batch 70/510 [I 2022-10-08 16:33:29] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:50 [I 2022-10-08 16:33:29] ...trap/engines/engine.py.326: process: 0.45098 | load: 0.00019 [I 2022-10-08 16:33:34] ...trap/engines/engine.py.321: eval: epoch 11 | batch 80/510 [I 2022-10-08 16:33:34] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:33 [I 2022-10-08 16:33:34] ...trap/engines/engine.py.326: process: 0.42101 | load: 0.00020 [I 2022-10-08 16:33:39] ...trap/engines/engine.py.321: eval: epoch 11 | batch 90/510 [I 2022-10-08 16:33:39] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:13 [I 2022-10-08 16:33:39] ...trap/engines/engine.py.326: process: 0.58270 | load: 0.00020 [I 2022-10-08 16:33:43] ...trap/engines/engine.py.321: eval: epoch 11 | batch 100/510 [I 2022-10-08 16:33:43] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:49 [I 2022-10-08 16:33:43] ...trap/engines/engine.py.326: process: 0.30941 | load: 0.00019 [I 2022-10-08 16:33:46] ...trap/engines/engine.py.321: eval: epoch 11 | batch 110/510 [I 2022-10-08 16:33:46] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:10 [I 2022-10-08 16:33:46] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00020 [I 2022-10-08 16:33:49] ...trap/engines/engine.py.321: eval: epoch 11 | batch 120/510 [I 2022-10-08 16:33:49] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-08 16:33:49] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00019 [I 2022-10-08 16:33:52] ...trap/engines/engine.py.321: eval: epoch 11 | batch 130/510 [I 2022-10-08 16:33:52] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-08 16:33:52] ...trap/engines/engine.py.326: process: 0.31866 | load: 0.00019 [I 2022-10-08 16:33:56] ...trap/engines/engine.py.321: eval: epoch 11 | batch 140/510 [I 2022-10-08 16:33:56] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-08 16:33:56] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00019 [I 2022-10-08 16:33:59] ...trap/engines/engine.py.321: eval: epoch 11 | batch 150/510 [I 2022-10-08 16:33:59] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:54 [I 2022-10-08 16:33:59] ...trap/engines/engine.py.326: process: 0.31511 | load: 0.00020 [I 2022-10-08 16:34:02] ...trap/engines/engine.py.321: eval: epoch 11 | batch 160/510 [I 2022-10-08 16:34:02] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:50 [I 2022-10-08 16:34:02] ...trap/engines/engine.py.326: process: 0.31977 | load: 0.00019 [I 2022-10-08 16:34:05] ...trap/engines/engine.py.321: eval: epoch 11 | batch 170/510 [I 2022-10-08 16:34:05] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:47 [I 2022-10-08 16:34:05] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00020 [I 2022-10-08 16:34:08] ...trap/engines/engine.py.321: eval: epoch 11 | batch 180/510 [I 2022-10-08 16:34:08] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-08 16:34:08] ...trap/engines/engine.py.326: process: 0.31861 | load: 0.00020 [I 2022-10-08 16:34:12] ...trap/engines/engine.py.321: eval: epoch 11 | batch 190/510 [I 2022-10-08 16:34:12] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-08 16:34:12] ...trap/engines/engine.py.326: process: 0.32014 | load: 0.00020 [I 2022-10-08 16:34:15] ...trap/engines/engine.py.321: eval: epoch 11 | batch 200/510 [I 2022-10-08 16:34:15] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:38 [I 2022-10-08 16:34:15] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00020 [I 2022-10-08 16:34:18] ...trap/engines/engine.py.321: eval: epoch 11 | batch 210/510 [I 2022-10-08 16:34:18] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-08 16:34:18] ...trap/engines/engine.py.326: process: 0.31445 | load: 0.00021 [I 2022-10-08 16:34:21] ...trap/engines/engine.py.321: eval: epoch 11 | batch 220/510 [I 2022-10-08 16:34:21] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-08 16:34:21] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00020 [I 2022-10-08 16:34:24] ...trap/engines/engine.py.321: eval: epoch 11 | batch 230/510 [I 2022-10-08 16:34:24] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-08 16:34:24] ...trap/engines/engine.py.326: process: 0.31606 | load: 0.00019 [I 2022-10-08 16:34:27] ...trap/engines/engine.py.321: eval: epoch 11 | batch 240/510 [I 2022-10-08 16:34:27] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-08 16:34:27] ...trap/engines/engine.py.326: process: 0.31435 | load: 0.00019 [I 2022-10-08 16:34:31] ...trap/engines/engine.py.321: eval: epoch 11 | batch 250/510 [I 2022-10-08 16:34:31] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-08 16:34:31] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00019 [I 2022-10-08 16:34:34] ...trap/engines/engine.py.321: eval: epoch 11 | batch 260/510 [I 2022-10-08 16:34:34] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:18 [I 2022-10-08 16:34:34] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00020 [I 2022-10-08 16:34:37] ...trap/engines/engine.py.321: eval: epoch 11 | batch 270/510 [I 2022-10-08 16:34:37] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-08 16:34:37] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00020 [I 2022-10-08 16:34:40] ...trap/engines/engine.py.321: eval: epoch 11 | batch 280/510 [I 2022-10-08 16:34:40] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-08 16:34:40] ...trap/engines/engine.py.326: process: 0.32306 | load: 0.00022 [I 2022-10-08 16:34:43] ...trap/engines/engine.py.321: eval: epoch 11 | batch 290/510 [I 2022-10-08 16:34:43] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-08 16:34:43] ...trap/engines/engine.py.326: process: 0.31457 | load: 0.00020 [I 2022-10-08 16:34:47] ...trap/engines/engine.py.321: eval: epoch 11 | batch 300/510 [I 2022-10-08 16:34:47] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-08 16:34:47] ...trap/engines/engine.py.326: process: 0.32212 | load: 0.00020 [I 2022-10-08 16:34:50] ...trap/engines/engine.py.321: eval: epoch 11 | batch 310/510 [I 2022-10-08 16:34:50] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:03 [I 2022-10-08 16:34:50] ...trap/engines/engine.py.326: process: 0.31529 | load: 0.00021 [I 2022-10-08 16:34:53] ...trap/engines/engine.py.321: eval: epoch 11 | batch 320/510 [I 2022-10-08 16:34:53] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-08 16:34:53] ...trap/engines/engine.py.326: process: 0.31502 | load: 0.00020 [I 2022-10-08 16:34:56] ...trap/engines/engine.py.321: eval: epoch 11 | batch 330/510 [I 2022-10-08 16:34:56] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-08 16:34:56] ...trap/engines/engine.py.326: process: 0.31621 | load: 0.00020 [I 2022-10-08 16:34:59] ...trap/engines/engine.py.321: eval: epoch 11 | batch 340/510 [I 2022-10-08 16:34:59] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-08 16:34:59] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00021 [I 2022-10-08 16:35:02] ...trap/engines/engine.py.321: eval: epoch 11 | batch 350/510 [I 2022-10-08 16:35:02] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:50 [I 2022-10-08 16:35:02] ...trap/engines/engine.py.326: process: 0.31493 | load: 0.00020 [I 2022-10-08 16:35:06] ...trap/engines/engine.py.321: eval: epoch 11 | batch 360/510 [I 2022-10-08 16:35:06] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-08 16:35:06] ...trap/engines/engine.py.326: process: 0.32499 | load: 0.00020 [I 2022-10-08 16:35:09] ...trap/engines/engine.py.321: eval: epoch 11 | batch 370/510 [I 2022-10-08 16:35:09] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-08 16:35:09] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00020 [I 2022-10-08 16:35:12] ...trap/engines/engine.py.321: eval: epoch 11 | batch 380/510 [I 2022-10-08 16:35:12] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-08 16:35:12] ...trap/engines/engine.py.326: process: 0.31654 | load: 0.00020 [I 2022-10-08 16:35:15] ...trap/engines/engine.py.321: eval: epoch 11 | batch 390/510 [I 2022-10-08 16:35:15] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-08 16:35:15] ...trap/engines/engine.py.326: process: 0.32167 | load: 0.00020 [I 2022-10-08 16:35:18] ...trap/engines/engine.py.321: eval: epoch 11 | batch 400/510 [I 2022-10-08 16:35:18] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-08 16:35:18] ...trap/engines/engine.py.326: process: 0.32186 | load: 0.00020 [I 2022-10-08 16:35:22] ...trap/engines/engine.py.321: eval: epoch 11 | batch 410/510 [I 2022-10-08 16:35:22] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-08 16:35:22] ...trap/engines/engine.py.326: process: 0.31921 | load: 0.00019 [I 2022-10-08 16:35:25] ...trap/engines/engine.py.321: eval: epoch 11 | batch 420/510 [I 2022-10-08 16:35:25] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-08 16:35:25] ...trap/engines/engine.py.326: process: 0.32105 | load: 0.00021 [I 2022-10-08 16:35:28] ...trap/engines/engine.py.321: eval: epoch 11 | batch 430/510 [I 2022-10-08 16:35:28] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-08 16:35:28] ...trap/engines/engine.py.326: process: 0.32547 | load: 0.00021 [I 2022-10-08 16:35:31] ...trap/engines/engine.py.321: eval: epoch 11 | batch 440/510 [I 2022-10-08 16:35:31] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-08 16:35:31] ...trap/engines/engine.py.326: process: 0.32047 | load: 0.00020 [I 2022-10-08 16:35:34] ...trap/engines/engine.py.321: eval: epoch 11 | batch 450/510 [I 2022-10-08 16:35:34] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-08 16:35:34] ...trap/engines/engine.py.326: process: 0.31206 | load: 0.00019 [I 2022-10-08 16:35:38] ...trap/engines/engine.py.321: eval: epoch 11 | batch 460/510 [I 2022-10-08 16:35:38] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-08 16:35:38] ...trap/engines/engine.py.326: process: 0.31532 | load: 0.00021 [I 2022-10-08 16:35:41] ...trap/engines/engine.py.321: eval: epoch 11 | batch 470/510 [I 2022-10-08 16:35:41] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-08 16:35:41] ...trap/engines/engine.py.326: process: 0.32394 | load: 0.00020 [I 2022-10-08 16:35:44] ...trap/engines/engine.py.321: eval: epoch 11 | batch 480/510 [I 2022-10-08 16:35:44] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-08 16:35:44] ...trap/engines/engine.py.326: process: 0.32229 | load: 0.00020 [I 2022-10-08 16:35:47] ...trap/engines/engine.py.321: eval: epoch 11 | batch 490/510 [I 2022-10-08 16:35:47] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-08 16:35:47] ...trap/engines/engine.py.326: process: 0.31758 | load: 0.00021 [I 2022-10-08 16:35:50] ...trap/engines/engine.py.321: eval: epoch 11 | batch 500/510 [I 2022-10-08 16:35:50] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-08 16:35:50] ...trap/engines/engine.py.326: process: 0.32327 | load: 0.00020 [I 2022-10-08 16:35:54] ...trap/engines/engine.py.321: eval: epoch 11 | batch 510/510 [I 2022-10-08 16:35:54] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-08 16:35:54] ...trap/engines/engine.py.326: process: 0.31742 | load: 0.00015 [S 2022-10-08 16:35:54] ...trap/engines/engine.py.132: eval_epoch.epoch: 11 [I 2022-10-08 16:35:54] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.593400 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.849000 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.903100 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.597700 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.850800 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.904400 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014874 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007308 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007842 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017012 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005564 [S 2022-10-08 16:36:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007017 [I 2022-10-08 16:36:27] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 16:36:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 16:36:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 16:36:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 16:36:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 16:36:28] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 16:36:30] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 16:36:35] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 16:36:35] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 16:36:35] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 16:36:38] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 16:36:43] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 16:36:43] ...trap/engines/engine.py.163: Training model on trainset for epoch 12 [I 2022-10-08 16:36:43] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 16:36:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 0/2304 [I 2022-10-08 16:36:46] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:00 [I 2022-10-08 16:36:46] ...trap/engines/engine.py.229: process: 3.02120 | load: 2.18916 [I 2022-10-08 16:36:46] ...trap/engines/engine.py.230: loss: 0.18593 [I 2022-10-08 16:36:46] ...optimizers/trijoint.py.151: total_norm: 2.414334 [I 2022-10-08 16:36:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 10/2304 [I 2022-10-08 16:36:54] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:39:10 [I 2022-10-08 16:36:54] ...trap/engines/engine.py.229: process: 0.77826 | load: 0.00567 [I 2022-10-08 16:36:54] ...trap/engines/engine.py.230: loss: 0.22499 [I 2022-10-08 16:36:54] ...optimizers/trijoint.py.151: total_norm: 2.481452 [I 2022-10-08 16:37:02] ...trap/engines/engine.py.224: train: epoch 12 | batch 20/2304 [I 2022-10-08 16:37:02] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:58 [I 2022-10-08 16:37:02] ...trap/engines/engine.py.229: process: 0.79814 | load: 0.00639 [I 2022-10-08 16:37:02] ...trap/engines/engine.py.230: loss: 0.21338 [I 2022-10-08 16:37:02] ...optimizers/trijoint.py.151: total_norm: 3.089219 [I 2022-10-08 16:37:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 30/2304 [I 2022-10-08 16:37:09] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:46 [I 2022-10-08 16:37:09] ...trap/engines/engine.py.229: process: 0.78728 | load: 0.00394 [I 2022-10-08 16:37:09] ...trap/engines/engine.py.230: loss: 0.21319 [I 2022-10-08 16:37:09] ...optimizers/trijoint.py.151: total_norm: 3.654228 [I 2022-10-08 16:37:17] ...trap/engines/engine.py.224: train: epoch 12 | batch 40/2304 [I 2022-10-08 16:37:17] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:23 [I 2022-10-08 16:37:17] ...trap/engines/engine.py.229: process: 0.76369 | load: 0.00413 [I 2022-10-08 16:37:17] ...trap/engines/engine.py.230: loss: 0.26023 [I 2022-10-08 16:37:17] ...optimizers/trijoint.py.151: total_norm: 2.917419 [I 2022-10-08 16:37:25] ...trap/engines/engine.py.224: train: epoch 12 | batch 50/2304 [I 2022-10-08 16:37:25] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:17 [I 2022-10-08 16:37:25] ...trap/engines/engine.py.229: process: 0.78084 | load: 0.00399 [I 2022-10-08 16:37:25] ...trap/engines/engine.py.230: loss: 0.21500 [I 2022-10-08 16:37:25] ...optimizers/trijoint.py.151: total_norm: 2.863877 [I 2022-10-08 16:37:33] ...trap/engines/engine.py.224: train: epoch 12 | batch 60/2304 [I 2022-10-08 16:37:33] ...trap/engines/engine.py.228: elapsed: 0:00:50 | left: 0:29:12 [I 2022-10-08 16:37:33] ...trap/engines/engine.py.229: process: 0.77975 | load: 0.00341 [I 2022-10-08 16:37:33] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-08 16:37:33] ...optimizers/trijoint.py.151: total_norm: 2.065129 [I 2022-10-08 16:37:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 70/2304 [I 2022-10-08 16:37:41] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:01 [I 2022-10-08 16:37:41] ...trap/engines/engine.py.229: process: 0.79019 | load: 0.00334 [I 2022-10-08 16:37:41] ...trap/engines/engine.py.230: loss: 0.24079 [I 2022-10-08 16:37:41] ...optimizers/trijoint.py.151: total_norm: 2.802821 [I 2022-10-08 16:37:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 80/2304 [I 2022-10-08 16:37:49] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:59 [I 2022-10-08 16:37:49] ...trap/engines/engine.py.229: process: 0.79373 | load: 0.00391 [I 2022-10-08 16:37:49] ...trap/engines/engine.py.230: loss: 0.15165 [I 2022-10-08 16:37:49] ...optimizers/trijoint.py.151: total_norm: 2.307213 [I 2022-10-08 16:37:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 90/2304 [I 2022-10-08 16:37:56] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:33 [I 2022-10-08 16:37:56] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00308 [I 2022-10-08 16:37:56] ...trap/engines/engine.py.230: loss: 0.19156 [I 2022-10-08 16:37:56] ...optimizers/trijoint.py.151: total_norm: 3.220990 [I 2022-10-08 16:38:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 100/2304 [I 2022-10-08 16:38:04] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:41 [I 2022-10-08 16:38:04] ...trap/engines/engine.py.229: process: 0.77590 | load: 0.00328 [I 2022-10-08 16:38:04] ...trap/engines/engine.py.230: loss: 0.21308 [I 2022-10-08 16:38:04] ...optimizers/trijoint.py.151: total_norm: 1.965452 [I 2022-10-08 16:38:12] ...trap/engines/engine.py.224: train: epoch 12 | batch 110/2304 [I 2022-10-08 16:38:12] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:24 [I 2022-10-08 16:38:12] ...trap/engines/engine.py.229: process: 0.78139 | load: 0.00327 [I 2022-10-08 16:38:12] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-08 16:38:12] ...optimizers/trijoint.py.151: total_norm: 1.892101 [I 2022-10-08 16:38:20] ...trap/engines/engine.py.224: train: epoch 12 | batch 120/2304 [I 2022-10-08 16:38:20] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:19 [I 2022-10-08 16:38:20] ...trap/engines/engine.py.229: process: 0.77985 | load: 0.00350 [I 2022-10-08 16:38:20] ...trap/engines/engine.py.230: loss: 0.20408 [I 2022-10-08 16:38:20] ...optimizers/trijoint.py.151: total_norm: 3.315235 [I 2022-10-08 16:38:28] ...trap/engines/engine.py.224: train: epoch 12 | batch 130/2304 [I 2022-10-08 16:38:28] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:25 [I 2022-10-08 16:38:28] ...trap/engines/engine.py.229: process: 0.79694 | load: 0.00322 [I 2022-10-08 16:38:28] ...trap/engines/engine.py.230: loss: 0.25236 [I 2022-10-08 16:38:28] ...optimizers/trijoint.py.151: total_norm: 3.440782 [I 2022-10-08 16:38:35] ...trap/engines/engine.py.224: train: epoch 12 | batch 140/2304 [I 2022-10-08 16:38:35] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:04 [I 2022-10-08 16:38:35] ...trap/engines/engine.py.229: process: 0.78123 | load: 0.00312 [I 2022-10-08 16:38:35] ...trap/engines/engine.py.230: loss: 0.24384 [I 2022-10-08 16:38:35] ...optimizers/trijoint.py.151: total_norm: 3.413767 [I 2022-10-08 16:38:43] ...trap/engines/engine.py.224: train: epoch 12 | batch 150/2304 [I 2022-10-08 16:38:43] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:56 [I 2022-10-08 16:38:43] ...trap/engines/engine.py.229: process: 0.78487 | load: 0.00361 [I 2022-10-08 16:38:43] ...trap/engines/engine.py.230: loss: 0.19472 [I 2022-10-08 16:38:43] ...optimizers/trijoint.py.151: total_norm: 3.042754 [I 2022-10-08 16:38:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 160/2304 [I 2022-10-08 16:38:51] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:54 [I 2022-10-08 16:38:51] ...trap/engines/engine.py.229: process: 0.77930 | load: 0.00338 [I 2022-10-08 16:38:51] ...trap/engines/engine.py.230: loss: 0.18313 [I 2022-10-08 16:38:51] ...optimizers/trijoint.py.151: total_norm: 1.994783 [I 2022-10-08 16:38:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 170/2304 [I 2022-10-08 16:38:59] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:48 [I 2022-10-08 16:38:59] ...trap/engines/engine.py.229: process: 0.78039 | load: 0.00312 [I 2022-10-08 16:38:59] ...trap/engines/engine.py.230: loss: 0.28627 [I 2022-10-08 16:38:59] ...optimizers/trijoint.py.151: total_norm: 3.352441 [I 2022-10-08 16:39:07] ...trap/engines/engine.py.224: train: epoch 12 | batch 180/2304 [I 2022-10-08 16:39:07] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:28 [I 2022-10-08 16:39:07] ...trap/engines/engine.py.229: process: 0.76583 | load: 0.00303 [I 2022-10-08 16:39:07] ...trap/engines/engine.py.230: loss: 0.10703 [I 2022-10-08 16:39:07] ...optimizers/trijoint.py.151: total_norm: 2.403378 [I 2022-10-08 16:39:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 190/2304 [I 2022-10-08 16:39:14] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:17 [I 2022-10-08 16:39:14] ...trap/engines/engine.py.229: process: 0.75942 | load: 0.00304 [I 2022-10-08 16:39:14] ...trap/engines/engine.py.230: loss: 0.18517 [I 2022-10-08 16:39:14] ...optimizers/trijoint.py.151: total_norm: 2.349886 [I 2022-10-08 16:39:22] ...trap/engines/engine.py.224: train: epoch 12 | batch 200/2304 [I 2022-10-08 16:39:22] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:11 [I 2022-10-08 16:39:22] ...trap/engines/engine.py.229: process: 0.78247 | load: 0.00327 [I 2022-10-08 16:39:22] ...trap/engines/engine.py.230: loss: 0.15385 [I 2022-10-08 16:39:22] ...optimizers/trijoint.py.151: total_norm: 2.887641 [I 2022-10-08 16:39:30] ...trap/engines/engine.py.224: train: epoch 12 | batch 210/2304 [I 2022-10-08 16:39:30] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:26:54 [I 2022-10-08 16:39:30] ...trap/engines/engine.py.229: process: 0.76806 | load: 0.00324 [I 2022-10-08 16:39:30] ...trap/engines/engine.py.230: loss: 0.24945 [I 2022-10-08 16:39:30] ...optimizers/trijoint.py.151: total_norm: 2.636758 [I 2022-10-08 16:39:38] ...trap/engines/engine.py.224: train: epoch 12 | batch 220/2304 [I 2022-10-08 16:39:38] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:04 [I 2022-10-08 16:39:38] ...trap/engines/engine.py.229: process: 0.78101 | load: 0.00325 [I 2022-10-08 16:39:38] ...trap/engines/engine.py.230: loss: 0.24310 [I 2022-10-08 16:39:38] ...optimizers/trijoint.py.151: total_norm: 2.055031 [I 2022-10-08 16:39:45] ...trap/engines/engine.py.224: train: epoch 12 | batch 230/2304 [I 2022-10-08 16:39:45] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:55 [I 2022-10-08 16:39:45] ...trap/engines/engine.py.229: process: 0.77886 | load: 0.00299 [I 2022-10-08 16:39:45] ...trap/engines/engine.py.230: loss: 0.19241 [I 2022-10-08 16:39:45] ...optimizers/trijoint.py.151: total_norm: 2.607070 [I 2022-10-08 16:39:53] ...trap/engines/engine.py.224: train: epoch 12 | batch 240/2304 [I 2022-10-08 16:39:53] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:56 [I 2022-10-08 16:39:53] ...trap/engines/engine.py.229: process: 0.79254 | load: 0.00299 [I 2022-10-08 16:39:53] ...trap/engines/engine.py.230: loss: 0.16281 [I 2022-10-08 16:39:53] ...optimizers/trijoint.py.151: total_norm: 1.945751 [I 2022-10-08 16:40:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 250/2304 [I 2022-10-08 16:40:01] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:33 [I 2022-10-08 16:40:01] ...trap/engines/engine.py.229: process: 0.75564 | load: 0.00310 [I 2022-10-08 16:40:01] ...trap/engines/engine.py.230: loss: 0.17119 [I 2022-10-08 16:40:01] ...optimizers/trijoint.py.151: total_norm: 2.454414 [I 2022-10-08 16:40:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 260/2304 [I 2022-10-08 16:40:09] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:28 [I 2022-10-08 16:40:09] ...trap/engines/engine.py.229: process: 0.77657 | load: 0.00301 [I 2022-10-08 16:40:09] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-08 16:40:09] ...optimizers/trijoint.py.151: total_norm: 2.187524 [I 2022-10-08 16:40:17] ...trap/engines/engine.py.224: train: epoch 12 | batch 270/2304 [I 2022-10-08 16:40:17] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:23 [I 2022-10-08 16:40:17] ...trap/engines/engine.py.229: process: 0.76537 | load: 0.00303 [I 2022-10-08 16:40:17] ...trap/engines/engine.py.230: loss: 0.23559 [I 2022-10-08 16:40:17] ...optimizers/trijoint.py.151: total_norm: 3.276871 [I 2022-10-08 16:40:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 280/2304 [I 2022-10-08 16:40:24] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:12 [I 2022-10-08 16:40:24] ...trap/engines/engine.py.229: process: 0.76913 | load: 0.00314 [I 2022-10-08 16:40:24] ...trap/engines/engine.py.230: loss: 0.23272 [I 2022-10-08 16:40:24] ...optimizers/trijoint.py.151: total_norm: 3.075077 [I 2022-10-08 16:40:32] ...trap/engines/engine.py.224: train: epoch 12 | batch 290/2304 [I 2022-10-08 16:40:32] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:03 [I 2022-10-08 16:40:32] ...trap/engines/engine.py.229: process: 0.77342 | load: 0.00304 [I 2022-10-08 16:40:32] ...trap/engines/engine.py.230: loss: 0.20120 [I 2022-10-08 16:40:32] ...optimizers/trijoint.py.151: total_norm: 2.864252 [I 2022-10-08 16:40:40] ...trap/engines/engine.py.224: train: epoch 12 | batch 300/2304 [I 2022-10-08 16:40:40] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:51 [I 2022-10-08 16:40:40] ...trap/engines/engine.py.229: process: 0.76932 | load: 0.00302 [I 2022-10-08 16:40:40] ...trap/engines/engine.py.230: loss: 0.27236 [I 2022-10-08 16:40:40] ...optimizers/trijoint.py.151: total_norm: 3.505038 [I 2022-10-08 16:40:48] ...trap/engines/engine.py.224: train: epoch 12 | batch 310/2304 [I 2022-10-08 16:40:48] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:25:58 [I 2022-10-08 16:40:48] ...trap/engines/engine.py.229: process: 0.78320 | load: 0.00305 [I 2022-10-08 16:40:48] ...trap/engines/engine.py.230: loss: 0.08993 [I 2022-10-08 16:40:48] ...optimizers/trijoint.py.151: total_norm: 1.323964 [I 2022-10-08 16:40:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 320/2304 [I 2022-10-08 16:40:56] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:41 [I 2022-10-08 16:40:56] ...trap/engines/engine.py.229: process: 0.77423 | load: 0.00300 [I 2022-10-08 16:40:56] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-08 16:40:56] ...optimizers/trijoint.py.151: total_norm: 2.774428 [I 2022-10-08 16:41:03] ...trap/engines/engine.py.224: train: epoch 12 | batch 330/2304 [I 2022-10-08 16:41:03] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:28 [I 2022-10-08 16:41:03] ...trap/engines/engine.py.229: process: 0.76817 | load: 0.00303 [I 2022-10-08 16:41:03] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-08 16:41:03] ...optimizers/trijoint.py.151: total_norm: 2.351858 [I 2022-10-08 16:41:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 340/2304 [I 2022-10-08 16:41:11] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:21 [I 2022-10-08 16:41:11] ...trap/engines/engine.py.229: process: 0.76549 | load: 0.00304 [I 2022-10-08 16:41:11] ...trap/engines/engine.py.230: loss: 0.19017 [I 2022-10-08 16:41:11] ...optimizers/trijoint.py.151: total_norm: 2.290320 [I 2022-10-08 16:41:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 350/2304 [I 2022-10-08 16:41:19] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:14 [I 2022-10-08 16:41:19] ...trap/engines/engine.py.229: process: 0.77403 | load: 0.00306 [I 2022-10-08 16:41:19] ...trap/engines/engine.py.230: loss: 0.14098 [I 2022-10-08 16:41:19] ...optimizers/trijoint.py.151: total_norm: 1.660514 [I 2022-10-08 16:41:27] ...trap/engines/engine.py.224: train: epoch 12 | batch 360/2304 [I 2022-10-08 16:41:27] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:19 [I 2022-10-08 16:41:27] ...trap/engines/engine.py.229: process: 0.80278 | load: 0.00301 [I 2022-10-08 16:41:27] ...trap/engines/engine.py.230: loss: 0.17095 [I 2022-10-08 16:41:27] ...optimizers/trijoint.py.151: total_norm: 3.333817 [I 2022-10-08 16:41:35] ...trap/engines/engine.py.224: train: epoch 12 | batch 370/2304 [I 2022-10-08 16:41:35] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:01 [I 2022-10-08 16:41:35] ...trap/engines/engine.py.229: process: 0.76276 | load: 0.00299 [I 2022-10-08 16:41:35] ...trap/engines/engine.py.230: loss: 0.17430 [I 2022-10-08 16:41:35] ...optimizers/trijoint.py.151: total_norm: 3.179574 [I 2022-10-08 16:41:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 380/2304 [I 2022-10-08 16:41:42] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:04 [I 2022-10-08 16:41:42] ...trap/engines/engine.py.229: process: 0.78163 | load: 0.00301 [I 2022-10-08 16:41:42] ...trap/engines/engine.py.230: loss: 0.18734 [I 2022-10-08 16:41:42] ...optimizers/trijoint.py.151: total_norm: 2.230104 [I 2022-10-08 16:41:50] ...trap/engines/engine.py.224: train: epoch 12 | batch 390/2304 [I 2022-10-08 16:41:50] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:50 [I 2022-10-08 16:41:50] ...trap/engines/engine.py.229: process: 0.77813 | load: 0.00307 [I 2022-10-08 16:41:50] ...trap/engines/engine.py.230: loss: 0.16242 [I 2022-10-08 16:41:50] ...optimizers/trijoint.py.151: total_norm: 2.797358 [I 2022-10-08 16:41:58] ...trap/engines/engine.py.224: train: epoch 12 | batch 400/2304 [I 2022-10-08 16:41:58] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:42 [I 2022-10-08 16:41:58] ...trap/engines/engine.py.229: process: 0.76638 | load: 0.00301 [I 2022-10-08 16:41:58] ...trap/engines/engine.py.230: loss: 0.17208 [I 2022-10-08 16:41:58] ...optimizers/trijoint.py.151: total_norm: 2.228086 [I 2022-10-08 16:42:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 410/2304 [I 2022-10-08 16:42:06] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:42 [I 2022-10-08 16:42:06] ...trap/engines/engine.py.229: process: 0.79844 | load: 0.00312 [I 2022-10-08 16:42:06] ...trap/engines/engine.py.230: loss: 0.12531 [I 2022-10-08 16:42:06] ...optimizers/trijoint.py.151: total_norm: 2.585948 [I 2022-10-08 16:42:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 420/2304 [I 2022-10-08 16:42:14] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:25 [I 2022-10-08 16:42:14] ...trap/engines/engine.py.229: process: 0.76549 | load: 0.00319 [I 2022-10-08 16:42:14] ...trap/engines/engine.py.230: loss: 0.26439 [I 2022-10-08 16:42:14] ...optimizers/trijoint.py.151: total_norm: 3.300296 [I 2022-10-08 16:42:22] ...trap/engines/engine.py.224: train: epoch 12 | batch 430/2304 [I 2022-10-08 16:42:22] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:21 [I 2022-10-08 16:42:22] ...trap/engines/engine.py.229: process: 0.77740 | load: 0.00329 [I 2022-10-08 16:42:22] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-08 16:42:22] ...optimizers/trijoint.py.151: total_norm: 2.465247 [I 2022-10-08 16:42:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 440/2304 [I 2022-10-08 16:42:29] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:14 [I 2022-10-08 16:42:29] ...trap/engines/engine.py.229: process: 0.77490 | load: 0.00311 [I 2022-10-08 16:42:29] ...trap/engines/engine.py.230: loss: 0.17679 [I 2022-10-08 16:42:29] ...optimizers/trijoint.py.151: total_norm: 3.273335 [I 2022-10-08 16:42:37] ...trap/engines/engine.py.224: train: epoch 12 | batch 450/2304 [I 2022-10-08 16:42:37] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:23:57 [I 2022-10-08 16:42:37] ...trap/engines/engine.py.229: process: 0.76802 | load: 0.00309 [I 2022-10-08 16:42:37] ...trap/engines/engine.py.230: loss: 0.19989 [I 2022-10-08 16:42:37] ...optimizers/trijoint.py.151: total_norm: 3.054168 [I 2022-10-08 16:42:45] ...trap/engines/engine.py.224: train: epoch 12 | batch 460/2304 [I 2022-10-08 16:42:45] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:58 [I 2022-10-08 16:42:45] ...trap/engines/engine.py.229: process: 0.77034 | load: 0.00316 [I 2022-10-08 16:42:45] ...trap/engines/engine.py.230: loss: 0.12211 [I 2022-10-08 16:42:45] ...optimizers/trijoint.py.151: total_norm: 2.147352 [I 2022-10-08 16:42:53] ...trap/engines/engine.py.224: train: epoch 12 | batch 470/2304 [I 2022-10-08 16:42:53] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:51 [I 2022-10-08 16:42:53] ...trap/engines/engine.py.229: process: 0.78854 | load: 0.00312 [I 2022-10-08 16:42:53] ...trap/engines/engine.py.230: loss: 0.24231 [I 2022-10-08 16:42:53] ...optimizers/trijoint.py.151: total_norm: 3.466029 [I 2022-10-08 16:43:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 480/2304 [I 2022-10-08 16:43:01] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:35 [I 2022-10-08 16:43:01] ...trap/engines/engine.py.229: process: 0.77593 | load: 0.00307 [I 2022-10-08 16:43:01] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-08 16:43:01] ...optimizers/trijoint.py.151: total_norm: 2.946142 [I 2022-10-08 16:43:08] ...trap/engines/engine.py.224: train: epoch 12 | batch 490/2304 [I 2022-10-08 16:43:08] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:31 [I 2022-10-08 16:43:08] ...trap/engines/engine.py.229: process: 0.77734 | load: 0.00313 [I 2022-10-08 16:43:08] ...trap/engines/engine.py.230: loss: 0.22842 [I 2022-10-08 16:43:08] ...optimizers/trijoint.py.151: total_norm: 3.129776 [I 2022-10-08 16:43:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 500/2304 [I 2022-10-08 16:43:16] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:28 [I 2022-10-08 16:43:16] ...trap/engines/engine.py.229: process: 0.78743 | load: 0.00317 [I 2022-10-08 16:43:16] ...trap/engines/engine.py.230: loss: 0.23567 [I 2022-10-08 16:43:16] ...optimizers/trijoint.py.151: total_norm: 3.898584 [I 2022-10-08 16:43:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 510/2304 [I 2022-10-08 16:43:24] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:18 [I 2022-10-08 16:43:24] ...trap/engines/engine.py.229: process: 0.78118 | load: 0.00307 [I 2022-10-08 16:43:24] ...trap/engines/engine.py.230: loss: 0.18642 [I 2022-10-08 16:43:24] ...optimizers/trijoint.py.151: total_norm: 2.485405 [I 2022-10-08 16:43:32] ...trap/engines/engine.py.224: train: epoch 12 | batch 520/2304 [I 2022-10-08 16:43:32] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:23:00 [I 2022-10-08 16:43:32] ...trap/engines/engine.py.229: process: 0.78040 | load: 0.00309 [I 2022-10-08 16:43:32] ...trap/engines/engine.py.230: loss: 0.27360 [I 2022-10-08 16:43:32] ...optimizers/trijoint.py.151: total_norm: 3.407516 [I 2022-10-08 16:43:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 530/2304 [I 2022-10-08 16:43:39] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:22:58 [I 2022-10-08 16:43:39] ...trap/engines/engine.py.229: process: 0.77652 | load: 0.00312 [I 2022-10-08 16:43:39] ...trap/engines/engine.py.230: loss: 0.17424 [I 2022-10-08 16:43:39] ...optimizers/trijoint.py.151: total_norm: 2.416679 [I 2022-10-08 16:43:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 540/2304 [I 2022-10-08 16:43:47] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:58 [I 2022-10-08 16:43:47] ...trap/engines/engine.py.229: process: 0.78966 | load: 0.00312 [I 2022-10-08 16:43:47] ...trap/engines/engine.py.230: loss: 0.16871 [I 2022-10-08 16:43:47] ...optimizers/trijoint.py.151: total_norm: 2.394980 [I 2022-10-08 16:43:55] ...trap/engines/engine.py.224: train: epoch 12 | batch 550/2304 [I 2022-10-08 16:43:55] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:41 [I 2022-10-08 16:43:55] ...trap/engines/engine.py.229: process: 0.77305 | load: 0.00305 [I 2022-10-08 16:43:55] ...trap/engines/engine.py.230: loss: 0.23315 [I 2022-10-08 16:43:55] ...optimizers/trijoint.py.151: total_norm: 3.692940 [I 2022-10-08 16:44:03] ...trap/engines/engine.py.224: train: epoch 12 | batch 560/2304 [I 2022-10-08 16:44:03] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:22:32 [I 2022-10-08 16:44:03] ...trap/engines/engine.py.229: process: 0.76990 | load: 0.00301 [I 2022-10-08 16:44:03] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-08 16:44:03] ...optimizers/trijoint.py.151: total_norm: 1.420133 [I 2022-10-08 16:44:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 570/2304 [I 2022-10-08 16:44:11] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:22:32 [I 2022-10-08 16:44:11] ...trap/engines/engine.py.229: process: 0.78259 | load: 0.00305 [I 2022-10-08 16:44:11] ...trap/engines/engine.py.230: loss: 0.16449 [I 2022-10-08 16:44:11] ...optimizers/trijoint.py.151: total_norm: 3.963351 [I 2022-10-08 16:44:18] ...trap/engines/engine.py.224: train: epoch 12 | batch 580/2304 [I 2022-10-08 16:44:18] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:19 [I 2022-10-08 16:44:18] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00303 [I 2022-10-08 16:44:18] ...trap/engines/engine.py.230: loss: 0.12767 [I 2022-10-08 16:44:18] ...optimizers/trijoint.py.151: total_norm: 1.924878 [I 2022-10-08 16:44:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 590/2304 [I 2022-10-08 16:44:26] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:18 [I 2022-10-08 16:44:26] ...trap/engines/engine.py.229: process: 0.78644 | load: 0.00320 [I 2022-10-08 16:44:26] ...trap/engines/engine.py.230: loss: 0.13737 [I 2022-10-08 16:44:26] ...optimizers/trijoint.py.151: total_norm: 1.907383 [I 2022-10-08 16:44:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 600/2304 [I 2022-10-08 16:44:34] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:04 [I 2022-10-08 16:44:34] ...trap/engines/engine.py.229: process: 0.76857 | load: 0.00306 [I 2022-10-08 16:44:34] ...trap/engines/engine.py.230: loss: 0.22322 [I 2022-10-08 16:44:34] ...optimizers/trijoint.py.151: total_norm: 3.908837 [I 2022-10-08 16:44:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 610/2304 [I 2022-10-08 16:44:42] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:21:56 [I 2022-10-08 16:44:42] ...trap/engines/engine.py.229: process: 0.77899 | load: 0.00305 [I 2022-10-08 16:44:42] ...trap/engines/engine.py.230: loss: 0.16677 [I 2022-10-08 16:44:42] ...optimizers/trijoint.py.151: total_norm: 2.423837 [I 2022-10-08 16:44:50] ...trap/engines/engine.py.224: train: epoch 12 | batch 620/2304 [I 2022-10-08 16:44:50] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:21:51 [I 2022-10-08 16:44:50] ...trap/engines/engine.py.229: process: 0.77593 | load: 0.00300 [I 2022-10-08 16:44:50] ...trap/engines/engine.py.230: loss: 0.15833 [I 2022-10-08 16:44:50] ...optimizers/trijoint.py.151: total_norm: 3.229458 [I 2022-10-08 16:44:57] ...trap/engines/engine.py.224: train: epoch 12 | batch 630/2304 [I 2022-10-08 16:44:57] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:21:49 [I 2022-10-08 16:44:57] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00302 [I 2022-10-08 16:44:57] ...trap/engines/engine.py.230: loss: 0.16704 [I 2022-10-08 16:44:57] ...optimizers/trijoint.py.151: total_norm: 3.079130 [I 2022-10-08 16:45:05] ...trap/engines/engine.py.224: train: epoch 12 | batch 640/2304 [I 2022-10-08 16:45:05] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:35 [I 2022-10-08 16:45:05] ...trap/engines/engine.py.229: process: 0.77113 | load: 0.00307 [I 2022-10-08 16:45:05] ...trap/engines/engine.py.230: loss: 0.30455 [I 2022-10-08 16:45:05] ...optimizers/trijoint.py.151: total_norm: 3.904682 [I 2022-10-08 16:45:13] ...trap/engines/engine.py.224: train: epoch 12 | batch 650/2304 [I 2022-10-08 16:45:13] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:23 [I 2022-10-08 16:45:13] ...trap/engines/engine.py.229: process: 0.78586 | load: 0.00303 [I 2022-10-08 16:45:13] ...trap/engines/engine.py.230: loss: 0.14366 [I 2022-10-08 16:45:13] ...optimizers/trijoint.py.151: total_norm: 1.316132 [I 2022-10-08 16:45:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 660/2304 [I 2022-10-08 16:45:21] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:21:18 [I 2022-10-08 16:45:21] ...trap/engines/engine.py.229: process: 0.78115 | load: 0.00302 [I 2022-10-08 16:45:21] ...trap/engines/engine.py.230: loss: 0.17040 [I 2022-10-08 16:45:21] ...optimizers/trijoint.py.151: total_norm: 2.535928 [I 2022-10-08 16:45:28] ...trap/engines/engine.py.224: train: epoch 12 | batch 670/2304 [I 2022-10-08 16:45:29] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:21:08 [I 2022-10-08 16:45:29] ...trap/engines/engine.py.229: process: 0.77622 | load: 0.00303 [I 2022-10-08 16:45:29] ...trap/engines/engine.py.230: loss: 0.13985 [I 2022-10-08 16:45:29] ...optimizers/trijoint.py.151: total_norm: 1.900916 [I 2022-10-08 16:45:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 680/2304 [I 2022-10-08 16:45:36] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:20:55 [I 2022-10-08 16:45:36] ...trap/engines/engine.py.229: process: 0.76124 | load: 0.00297 [I 2022-10-08 16:45:36] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-08 16:45:36] ...optimizers/trijoint.py.151: total_norm: 2.448718 [I 2022-10-08 16:45:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 690/2304 [I 2022-10-08 16:45:44] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:50 [I 2022-10-08 16:45:44] ...trap/engines/engine.py.229: process: 0.75674 | load: 0.00303 [I 2022-10-08 16:45:44] ...trap/engines/engine.py.230: loss: 0.23529 [I 2022-10-08 16:45:44] ...optimizers/trijoint.py.151: total_norm: 2.394287 [I 2022-10-08 16:45:52] ...trap/engines/engine.py.224: train: epoch 12 | batch 700/2304 [I 2022-10-08 16:45:52] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:47 [I 2022-10-08 16:45:52] ...trap/engines/engine.py.229: process: 0.78367 | load: 0.00303 [I 2022-10-08 16:45:52] ...trap/engines/engine.py.230: loss: 0.22690 [I 2022-10-08 16:45:52] ...optimizers/trijoint.py.151: total_norm: 2.666853 [I 2022-10-08 16:46:00] ...trap/engines/engine.py.224: train: epoch 12 | batch 710/2304 [I 2022-10-08 16:46:00] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:20:42 [I 2022-10-08 16:46:00] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00299 [I 2022-10-08 16:46:00] ...trap/engines/engine.py.230: loss: 0.21326 [I 2022-10-08 16:46:00] ...optimizers/trijoint.py.151: total_norm: 3.016425 [I 2022-10-08 16:46:07] ...trap/engines/engine.py.224: train: epoch 12 | batch 720/2304 [I 2022-10-08 16:46:07] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:20:34 [I 2022-10-08 16:46:07] ...trap/engines/engine.py.229: process: 0.78222 | load: 0.00305 [I 2022-10-08 16:46:07] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-08 16:46:07] ...optimizers/trijoint.py.151: total_norm: 2.057957 [I 2022-10-08 16:46:15] ...trap/engines/engine.py.224: train: epoch 12 | batch 730/2304 [I 2022-10-08 16:46:15] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:20:28 [I 2022-10-08 16:46:15] ...trap/engines/engine.py.229: process: 0.78096 | load: 0.00313 [I 2022-10-08 16:46:15] ...trap/engines/engine.py.230: loss: 0.21788 [I 2022-10-08 16:46:15] ...optimizers/trijoint.py.151: total_norm: 2.386015 [I 2022-10-08 16:46:23] ...trap/engines/engine.py.224: train: epoch 12 | batch 740/2304 [I 2022-10-08 16:46:23] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:19 [I 2022-10-08 16:46:23] ...trap/engines/engine.py.229: process: 0.78568 | load: 0.00309 [I 2022-10-08 16:46:23] ...trap/engines/engine.py.230: loss: 0.19387 [I 2022-10-08 16:46:23] ...optimizers/trijoint.py.151: total_norm: 2.233665 [I 2022-10-08 16:46:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 750/2304 [I 2022-10-08 16:46:31] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:09 [I 2022-10-08 16:46:31] ...trap/engines/engine.py.229: process: 0.77319 | load: 0.00299 [I 2022-10-08 16:46:31] ...trap/engines/engine.py.230: loss: 0.23748 [I 2022-10-08 16:46:31] ...optimizers/trijoint.py.151: total_norm: 2.828448 [I 2022-10-08 16:46:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 760/2304 [I 2022-10-08 16:46:39] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:19:55 [I 2022-10-08 16:46:39] ...trap/engines/engine.py.229: process: 0.77095 | load: 0.00299 [I 2022-10-08 16:46:39] ...trap/engines/engine.py.230: loss: 0.21762 [I 2022-10-08 16:46:39] ...optimizers/trijoint.py.151: total_norm: 2.778399 [I 2022-10-08 16:46:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 770/2304 [I 2022-10-08 16:46:47] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:19:52 [I 2022-10-08 16:46:47] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00301 [I 2022-10-08 16:46:47] ...trap/engines/engine.py.230: loss: 0.17153 [I 2022-10-08 16:46:47] ...optimizers/trijoint.py.151: total_norm: 2.599984 [I 2022-10-08 16:46:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 780/2304 [I 2022-10-08 16:46:54] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:19:41 [I 2022-10-08 16:46:54] ...trap/engines/engine.py.229: process: 0.77217 | load: 0.00302 [I 2022-10-08 16:46:54] ...trap/engines/engine.py.230: loss: 0.18807 [I 2022-10-08 16:46:54] ...optimizers/trijoint.py.151: total_norm: 3.210736 [I 2022-10-08 16:47:02] ...trap/engines/engine.py.224: train: epoch 12 | batch 790/2304 [I 2022-10-08 16:47:02] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:37 [I 2022-10-08 16:47:02] ...trap/engines/engine.py.229: process: 0.78278 | load: 0.00299 [I 2022-10-08 16:47:02] ...trap/engines/engine.py.230: loss: 0.27456 [I 2022-10-08 16:47:02] ...optimizers/trijoint.py.151: total_norm: 4.013066 [I 2022-10-08 16:47:10] ...trap/engines/engine.py.224: train: epoch 12 | batch 800/2304 [I 2022-10-08 16:47:10] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:29 [I 2022-10-08 16:47:10] ...trap/engines/engine.py.229: process: 0.77989 | load: 0.00306 [I 2022-10-08 16:47:10] ...trap/engines/engine.py.230: loss: 0.20088 [I 2022-10-08 16:47:10] ...optimizers/trijoint.py.151: total_norm: 2.920738 [I 2022-10-08 16:47:18] ...trap/engines/engine.py.224: train: epoch 12 | batch 810/2304 [I 2022-10-08 16:47:18] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:19:25 [I 2022-10-08 16:47:18] ...trap/engines/engine.py.229: process: 0.78466 | load: 0.00300 [I 2022-10-08 16:47:18] ...trap/engines/engine.py.230: loss: 0.16263 [I 2022-10-08 16:47:18] ...optimizers/trijoint.py.151: total_norm: 2.711856 [I 2022-10-08 16:47:25] ...trap/engines/engine.py.224: train: epoch 12 | batch 820/2304 [I 2022-10-08 16:47:25] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:19:11 [I 2022-10-08 16:47:25] ...trap/engines/engine.py.229: process: 0.76851 | load: 0.00305 [I 2022-10-08 16:47:25] ...trap/engines/engine.py.230: loss: 0.18501 [I 2022-10-08 16:47:25] ...optimizers/trijoint.py.151: total_norm: 3.279710 [I 2022-10-08 16:47:33] ...trap/engines/engine.py.224: train: epoch 12 | batch 830/2304 [I 2022-10-08 16:47:33] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:19:06 [I 2022-10-08 16:47:33] ...trap/engines/engine.py.229: process: 0.77987 | load: 0.00304 [I 2022-10-08 16:47:33] ...trap/engines/engine.py.230: loss: 0.16274 [I 2022-10-08 16:47:33] ...optimizers/trijoint.py.151: total_norm: 2.669776 [I 2022-10-08 16:47:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 840/2304 [I 2022-10-08 16:47:41] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:18:53 [I 2022-10-08 16:47:41] ...trap/engines/engine.py.229: process: 0.78727 | load: 0.00312 [I 2022-10-08 16:47:41] ...trap/engines/engine.py.230: loss: 0.20607 [I 2022-10-08 16:47:41] ...optimizers/trijoint.py.151: total_norm: 2.677510 [I 2022-10-08 16:47:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 850/2304 [I 2022-10-08 16:47:49] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:18:48 [I 2022-10-08 16:47:49] ...trap/engines/engine.py.229: process: 0.78009 | load: 0.00308 [I 2022-10-08 16:47:49] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-08 16:47:49] ...optimizers/trijoint.py.151: total_norm: 3.446819 [I 2022-10-08 16:47:57] ...trap/engines/engine.py.224: train: epoch 12 | batch 860/2304 [I 2022-10-08 16:47:57] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:18:39 [I 2022-10-08 16:47:57] ...trap/engines/engine.py.229: process: 0.78484 | load: 0.00300 [I 2022-10-08 16:47:57] ...trap/engines/engine.py.230: loss: 0.21950 [I 2022-10-08 16:47:57] ...optimizers/trijoint.py.151: total_norm: 3.560078 [I 2022-10-08 16:48:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 870/2304 [I 2022-10-08 16:48:04] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:18:35 [I 2022-10-08 16:48:04] ...trap/engines/engine.py.229: process: 0.78442 | load: 0.00300 [I 2022-10-08 16:48:04] ...trap/engines/engine.py.230: loss: 0.24012 [I 2022-10-08 16:48:04] ...optimizers/trijoint.py.151: total_norm: 4.091156 [I 2022-10-08 16:48:12] ...trap/engines/engine.py.224: train: epoch 12 | batch 880/2304 [I 2022-10-08 16:48:12] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:18:33 [I 2022-10-08 16:48:12] ...trap/engines/engine.py.229: process: 0.78707 | load: 0.00303 [I 2022-10-08 16:48:12] ...trap/engines/engine.py.230: loss: 0.25174 [I 2022-10-08 16:48:12] ...optimizers/trijoint.py.151: total_norm: 1.925764 [I 2022-10-08 16:48:20] ...trap/engines/engine.py.224: train: epoch 12 | batch 890/2304 [I 2022-10-08 16:48:20] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:31 [I 2022-10-08 16:48:20] ...trap/engines/engine.py.229: process: 0.78645 | load: 0.00302 [I 2022-10-08 16:48:20] ...trap/engines/engine.py.230: loss: 0.26454 [I 2022-10-08 16:48:20] ...optimizers/trijoint.py.151: total_norm: 2.962552 [I 2022-10-08 16:48:28] ...trap/engines/engine.py.224: train: epoch 12 | batch 900/2304 [I 2022-10-08 16:48:28] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:13 [I 2022-10-08 16:48:28] ...trap/engines/engine.py.229: process: 0.77996 | load: 0.00307 [I 2022-10-08 16:48:28] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-08 16:48:28] ...optimizers/trijoint.py.151: total_norm: 2.052570 [I 2022-10-08 16:48:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 910/2304 [I 2022-10-08 16:48:36] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:17:57 [I 2022-10-08 16:48:36] ...trap/engines/engine.py.229: process: 0.76897 | load: 0.00302 [I 2022-10-08 16:48:36] ...trap/engines/engine.py.230: loss: 0.17482 [I 2022-10-08 16:48:36] ...optimizers/trijoint.py.151: total_norm: 1.857195 [I 2022-10-08 16:48:43] ...trap/engines/engine.py.224: train: epoch 12 | batch 920/2304 [I 2022-10-08 16:48:43] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:17:59 [I 2022-10-08 16:48:43] ...trap/engines/engine.py.229: process: 0.79524 | load: 0.00310 [I 2022-10-08 16:48:43] ...trap/engines/engine.py.230: loss: 0.19978 [I 2022-10-08 16:48:43] ...optimizers/trijoint.py.151: total_norm: 2.100941 [I 2022-10-08 16:48:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 930/2304 [I 2022-10-08 16:48:51] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:17:40 [I 2022-10-08 16:48:51] ...trap/engines/engine.py.229: process: 0.76819 | load: 0.00305 [I 2022-10-08 16:48:51] ...trap/engines/engine.py.230: loss: 0.25197 [I 2022-10-08 16:48:51] ...optimizers/trijoint.py.151: total_norm: 4.108849 [I 2022-10-08 16:48:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 940/2304 [I 2022-10-08 16:48:59] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:17:41 [I 2022-10-08 16:48:59] ...trap/engines/engine.py.229: process: 0.77015 | load: 0.00305 [I 2022-10-08 16:48:59] ...trap/engines/engine.py.230: loss: 0.25736 [I 2022-10-08 16:48:59] ...optimizers/trijoint.py.151: total_norm: 3.226942 [I 2022-10-08 16:49:07] ...trap/engines/engine.py.224: train: epoch 12 | batch 950/2304 [I 2022-10-08 16:49:07] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:35 [I 2022-10-08 16:49:07] ...trap/engines/engine.py.229: process: 0.77611 | load: 0.00313 [I 2022-10-08 16:49:07] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-08 16:49:07] ...optimizers/trijoint.py.151: total_norm: 2.215683 [I 2022-10-08 16:49:15] ...trap/engines/engine.py.224: train: epoch 12 | batch 960/2304 [I 2022-10-08 16:49:15] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:17:23 [I 2022-10-08 16:49:15] ...trap/engines/engine.py.229: process: 0.76878 | load: 0.00306 [I 2022-10-08 16:49:15] ...trap/engines/engine.py.230: loss: 0.26902 [I 2022-10-08 16:49:15] ...optimizers/trijoint.py.151: total_norm: 2.716139 [I 2022-10-08 16:49:23] ...trap/engines/engine.py.224: train: epoch 12 | batch 970/2304 [I 2022-10-08 16:49:23] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:17:20 [I 2022-10-08 16:49:23] ...trap/engines/engine.py.229: process: 0.78171 | load: 0.00303 [I 2022-10-08 16:49:23] ...trap/engines/engine.py.230: loss: 0.24304 [I 2022-10-08 16:49:23] ...optimizers/trijoint.py.151: total_norm: 2.855252 [I 2022-10-08 16:49:30] ...trap/engines/engine.py.224: train: epoch 12 | batch 980/2304 [I 2022-10-08 16:49:30] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:17:09 [I 2022-10-08 16:49:30] ...trap/engines/engine.py.229: process: 0.78592 | load: 0.00314 [I 2022-10-08 16:49:30] ...trap/engines/engine.py.230: loss: 0.16337 [I 2022-10-08 16:49:30] ...optimizers/trijoint.py.151: total_norm: 2.609638 [I 2022-10-08 16:49:38] ...trap/engines/engine.py.224: train: epoch 12 | batch 990/2304 [I 2022-10-08 16:49:38] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:16:59 [I 2022-10-08 16:49:38] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00303 [I 2022-10-08 16:49:38] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-08 16:49:38] ...optimizers/trijoint.py.151: total_norm: 2.571518 [I 2022-10-08 16:49:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1000/2304 [I 2022-10-08 16:49:46] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:16:46 [I 2022-10-08 16:49:46] ...trap/engines/engine.py.229: process: 0.76821 | load: 0.00304 [I 2022-10-08 16:49:46] ...trap/engines/engine.py.230: loss: 0.14002 [I 2022-10-08 16:49:46] ...optimizers/trijoint.py.151: total_norm: 2.538411 [I 2022-10-08 16:49:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 1010/2304 [I 2022-10-08 16:49:54] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:16:43 [I 2022-10-08 16:49:54] ...trap/engines/engine.py.229: process: 0.76700 | load: 0.00299 [I 2022-10-08 16:49:54] ...trap/engines/engine.py.230: loss: 0.16465 [I 2022-10-08 16:49:54] ...optimizers/trijoint.py.151: total_norm: 1.962433 [I 2022-10-08 16:50:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1020/2304 [I 2022-10-08 16:50:01] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:16:31 [I 2022-10-08 16:50:01] ...trap/engines/engine.py.229: process: 0.76259 | load: 0.00302 [I 2022-10-08 16:50:01] ...trap/engines/engine.py.230: loss: 0.17704 [I 2022-10-08 16:50:01] ...optimizers/trijoint.py.151: total_norm: 2.855405 [I 2022-10-08 16:50:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 1030/2304 [I 2022-10-08 16:50:09] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:16:24 [I 2022-10-08 16:50:09] ...trap/engines/engine.py.229: process: 0.75822 | load: 0.00330 [I 2022-10-08 16:50:09] ...trap/engines/engine.py.230: loss: 0.20331 [I 2022-10-08 16:50:09] ...optimizers/trijoint.py.151: total_norm: 2.655641 [I 2022-10-08 16:50:17] ...trap/engines/engine.py.224: train: epoch 12 | batch 1040/2304 [I 2022-10-08 16:50:17] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:16:21 [I 2022-10-08 16:50:17] ...trap/engines/engine.py.229: process: 0.78982 | load: 0.00301 [I 2022-10-08 16:50:17] ...trap/engines/engine.py.230: loss: 0.29788 [I 2022-10-08 16:50:17] ...optimizers/trijoint.py.151: total_norm: 3.310912 [I 2022-10-08 16:50:25] ...trap/engines/engine.py.224: train: epoch 12 | batch 1050/2304 [I 2022-10-08 16:50:25] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:16:10 [I 2022-10-08 16:50:25] ...trap/engines/engine.py.229: process: 0.76954 | load: 0.00302 [I 2022-10-08 16:50:25] ...trap/engines/engine.py.230: loss: 0.20679 [I 2022-10-08 16:50:25] ...optimizers/trijoint.py.151: total_norm: 3.351120 [I 2022-10-08 16:50:32] ...trap/engines/engine.py.224: train: epoch 12 | batch 1060/2304 [I 2022-10-08 16:50:32] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:16:03 [I 2022-10-08 16:50:32] ...trap/engines/engine.py.229: process: 0.76623 | load: 0.00303 [I 2022-10-08 16:50:32] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-08 16:50:32] ...optimizers/trijoint.py.151: total_norm: 2.663903 [I 2022-10-08 16:50:40] ...trap/engines/engine.py.224: train: epoch 12 | batch 1070/2304 [I 2022-10-08 16:50:40] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:16:02 [I 2022-10-08 16:50:40] ...trap/engines/engine.py.229: process: 0.78162 | load: 0.00302 [I 2022-10-08 16:50:40] ...trap/engines/engine.py.230: loss: 0.15689 [I 2022-10-08 16:50:40] ...optimizers/trijoint.py.151: total_norm: 2.334342 [I 2022-10-08 16:50:48] ...trap/engines/engine.py.224: train: epoch 12 | batch 1080/2304 [I 2022-10-08 16:50:48] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:15:49 [I 2022-10-08 16:50:48] ...trap/engines/engine.py.229: process: 0.77416 | load: 0.00307 [I 2022-10-08 16:50:48] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-08 16:50:48] ...optimizers/trijoint.py.151: total_norm: 2.213841 [I 2022-10-08 16:50:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1090/2304 [I 2022-10-08 16:50:56] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:15:46 [I 2022-10-08 16:50:56] ...trap/engines/engine.py.229: process: 0.78496 | load: 0.00298 [I 2022-10-08 16:50:56] ...trap/engines/engine.py.230: loss: 0.24367 [I 2022-10-08 16:50:56] ...optimizers/trijoint.py.151: total_norm: 2.539872 [I 2022-10-08 16:51:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 1100/2304 [I 2022-10-08 16:51:04] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:15:28 [I 2022-10-08 16:51:04] ...trap/engines/engine.py.229: process: 0.76403 | load: 0.00304 [I 2022-10-08 16:51:04] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-08 16:51:04] ...optimizers/trijoint.py.151: total_norm: 3.173834 [I 2022-10-08 16:51:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1110/2304 [I 2022-10-08 16:51:11] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:15:30 [I 2022-10-08 16:51:11] ...trap/engines/engine.py.229: process: 0.78780 | load: 0.00301 [I 2022-10-08 16:51:11] ...trap/engines/engine.py.230: loss: 0.16359 [I 2022-10-08 16:51:11] ...optimizers/trijoint.py.151: total_norm: 2.066746 [I 2022-10-08 16:51:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 1120/2304 [I 2022-10-08 16:51:19] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:15:20 [I 2022-10-08 16:51:19] ...trap/engines/engine.py.229: process: 0.76550 | load: 0.00300 [I 2022-10-08 16:51:19] ...trap/engines/engine.py.230: loss: 0.29046 [I 2022-10-08 16:51:19] ...optimizers/trijoint.py.151: total_norm: 4.134723 [I 2022-10-08 16:51:27] ...trap/engines/engine.py.224: train: epoch 12 | batch 1130/2304 [I 2022-10-08 16:51:27] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:15:15 [I 2022-10-08 16:51:27] ...trap/engines/engine.py.229: process: 0.78578 | load: 0.00304 [I 2022-10-08 16:51:27] ...trap/engines/engine.py.230: loss: 0.20546 [I 2022-10-08 16:51:27] ...optimizers/trijoint.py.151: total_norm: 3.680176 [I 2022-10-08 16:51:35] ...trap/engines/engine.py.224: train: epoch 12 | batch 1140/2304 [I 2022-10-08 16:51:35] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:15:08 [I 2022-10-08 16:51:35] ...trap/engines/engine.py.229: process: 0.78519 | load: 0.00309 [I 2022-10-08 16:51:35] ...trap/engines/engine.py.230: loss: 0.11915 [I 2022-10-08 16:51:35] ...optimizers/trijoint.py.151: total_norm: 2.338300 [I 2022-10-08 16:51:43] ...trap/engines/engine.py.224: train: epoch 12 | batch 1150/2304 [I 2022-10-08 16:51:43] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:15:02 [I 2022-10-08 16:51:43] ...trap/engines/engine.py.229: process: 0.77513 | load: 0.00324 [I 2022-10-08 16:51:43] ...trap/engines/engine.py.230: loss: 0.23021 [I 2022-10-08 16:51:43] ...optimizers/trijoint.py.151: total_norm: 3.588945 [I 2022-10-08 16:51:50] ...trap/engines/engine.py.224: train: epoch 12 | batch 1160/2304 [I 2022-10-08 16:51:50] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:14:51 [I 2022-10-08 16:51:50] ...trap/engines/engine.py.229: process: 0.77589 | load: 0.00306 [I 2022-10-08 16:51:50] ...trap/engines/engine.py.230: loss: 0.20518 [I 2022-10-08 16:51:50] ...optimizers/trijoint.py.151: total_norm: 2.208978 [I 2022-10-08 16:51:58] ...trap/engines/engine.py.224: train: epoch 12 | batch 1170/2304 [I 2022-10-08 16:51:58] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:14:38 [I 2022-10-08 16:51:58] ...trap/engines/engine.py.229: process: 0.78099 | load: 0.00301 [I 2022-10-08 16:51:58] ...trap/engines/engine.py.230: loss: 0.25970 [I 2022-10-08 16:51:58] ...optimizers/trijoint.py.151: total_norm: 2.914427 [I 2022-10-08 16:52:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1180/2304 [I 2022-10-08 16:52:06] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:14:26 [I 2022-10-08 16:52:06] ...trap/engines/engine.py.229: process: 0.76604 | load: 0.00302 [I 2022-10-08 16:52:06] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-08 16:52:06] ...optimizers/trijoint.py.151: total_norm: 2.215066 [I 2022-10-08 16:52:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 1190/2304 [I 2022-10-08 16:52:14] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:14:22 [I 2022-10-08 16:52:14] ...trap/engines/engine.py.229: process: 0.77286 | load: 0.00303 [I 2022-10-08 16:52:14] ...trap/engines/engine.py.230: loss: 0.28455 [I 2022-10-08 16:52:14] ...optimizers/trijoint.py.151: total_norm: 2.780449 [I 2022-10-08 16:52:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1200/2304 [I 2022-10-08 16:52:21] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:14:11 [I 2022-10-08 16:52:21] ...trap/engines/engine.py.229: process: 0.76071 | load: 0.00305 [I 2022-10-08 16:52:21] ...trap/engines/engine.py.230: loss: 0.19860 [I 2022-10-08 16:52:21] ...optimizers/trijoint.py.151: total_norm: 3.080392 [I 2022-10-08 16:52:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 1210/2304 [I 2022-10-08 16:52:29] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:14:13 [I 2022-10-08 16:52:29] ...trap/engines/engine.py.229: process: 0.78372 | load: 0.00300 [I 2022-10-08 16:52:29] ...trap/engines/engine.py.230: loss: 0.12492 [I 2022-10-08 16:52:29] ...optimizers/trijoint.py.151: total_norm: 2.574568 [I 2022-10-08 16:52:37] ...trap/engines/engine.py.224: train: epoch 12 | batch 1220/2304 [I 2022-10-08 16:52:37] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:14:00 [I 2022-10-08 16:52:37] ...trap/engines/engine.py.229: process: 0.77587 | load: 0.00301 [I 2022-10-08 16:52:37] ...trap/engines/engine.py.230: loss: 0.16617 [I 2022-10-08 16:52:37] ...optimizers/trijoint.py.151: total_norm: 2.555187 [I 2022-10-08 16:52:45] ...trap/engines/engine.py.224: train: epoch 12 | batch 1230/2304 [I 2022-10-08 16:52:45] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:13:54 [I 2022-10-08 16:52:45] ...trap/engines/engine.py.229: process: 0.78609 | load: 0.00304 [I 2022-10-08 16:52:45] ...trap/engines/engine.py.230: loss: 0.15472 [I 2022-10-08 16:52:45] ...optimizers/trijoint.py.151: total_norm: 1.919106 [I 2022-10-08 16:52:53] ...trap/engines/engine.py.224: train: epoch 12 | batch 1240/2304 [I 2022-10-08 16:52:53] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:13:46 [I 2022-10-08 16:52:53] ...trap/engines/engine.py.229: process: 0.76676 | load: 0.00304 [I 2022-10-08 16:52:53] ...trap/engines/engine.py.230: loss: 0.19767 [I 2022-10-08 16:52:53] ...optimizers/trijoint.py.151: total_norm: 2.246024 [I 2022-10-08 16:53:00] ...trap/engines/engine.py.224: train: epoch 12 | batch 1250/2304 [I 2022-10-08 16:53:00] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:13:40 [I 2022-10-08 16:53:00] ...trap/engines/engine.py.229: process: 0.77737 | load: 0.00305 [I 2022-10-08 16:53:00] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-08 16:53:00] ...optimizers/trijoint.py.151: total_norm: 2.572028 [I 2022-10-08 16:53:08] ...trap/engines/engine.py.224: train: epoch 12 | batch 1260/2304 [I 2022-10-08 16:53:08] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:13:29 [I 2022-10-08 16:53:08] ...trap/engines/engine.py.229: process: 0.77403 | load: 0.00303 [I 2022-10-08 16:53:08] ...trap/engines/engine.py.230: loss: 0.17478 [I 2022-10-08 16:53:08] ...optimizers/trijoint.py.151: total_norm: 2.048745 [I 2022-10-08 16:53:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1270/2304 [I 2022-10-08 16:53:16] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:13:21 [I 2022-10-08 16:53:16] ...trap/engines/engine.py.229: process: 0.76645 | load: 0.00303 [I 2022-10-08 16:53:16] ...trap/engines/engine.py.230: loss: 0.14104 [I 2022-10-08 16:53:16] ...optimizers/trijoint.py.151: total_norm: 2.727334 [I 2022-10-08 16:53:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 1280/2304 [I 2022-10-08 16:53:24] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:13:13 [I 2022-10-08 16:53:24] ...trap/engines/engine.py.229: process: 0.77266 | load: 0.00306 [I 2022-10-08 16:53:24] ...trap/engines/engine.py.230: loss: 0.19617 [I 2022-10-08 16:53:24] ...optimizers/trijoint.py.151: total_norm: 3.639822 [I 2022-10-08 16:53:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1290/2304 [I 2022-10-08 16:53:31] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:13:08 [I 2022-10-08 16:53:31] ...trap/engines/engine.py.229: process: 0.78220 | load: 0.00308 [I 2022-10-08 16:53:31] ...trap/engines/engine.py.230: loss: 0.28238 [I 2022-10-08 16:53:31] ...optimizers/trijoint.py.151: total_norm: 3.750997 [I 2022-10-08 16:53:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 1300/2304 [I 2022-10-08 16:53:39] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:13:01 [I 2022-10-08 16:53:39] ...trap/engines/engine.py.229: process: 0.78547 | load: 0.00306 [I 2022-10-08 16:53:39] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-08 16:53:39] ...optimizers/trijoint.py.151: total_norm: 2.094245 [I 2022-10-08 16:53:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 1310/2304 [I 2022-10-08 16:53:47] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:12:53 [I 2022-10-08 16:53:47] ...trap/engines/engine.py.229: process: 0.78437 | load: 0.00306 [I 2022-10-08 16:53:47] ...trap/engines/engine.py.230: loss: 0.21137 [I 2022-10-08 16:53:47] ...optimizers/trijoint.py.151: total_norm: 2.388278 [I 2022-10-08 16:53:55] ...trap/engines/engine.py.224: train: epoch 12 | batch 1320/2304 [I 2022-10-08 16:53:55] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:12:46 [I 2022-10-08 16:53:55] ...trap/engines/engine.py.229: process: 0.78376 | load: 0.00303 [I 2022-10-08 16:53:55] ...trap/engines/engine.py.230: loss: 0.21519 [I 2022-10-08 16:53:55] ...optimizers/trijoint.py.151: total_norm: 3.117210 [I 2022-10-08 16:54:03] ...trap/engines/engine.py.224: train: epoch 12 | batch 1330/2304 [I 2022-10-08 16:54:03] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:12:34 [I 2022-10-08 16:54:03] ...trap/engines/engine.py.229: process: 0.76955 | load: 0.00305 [I 2022-10-08 16:54:03] ...trap/engines/engine.py.230: loss: 0.21309 [I 2022-10-08 16:54:03] ...optimizers/trijoint.py.151: total_norm: 2.486253 [I 2022-10-08 16:54:10] ...trap/engines/engine.py.224: train: epoch 12 | batch 1340/2304 [I 2022-10-08 16:54:10] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:12:25 [I 2022-10-08 16:54:10] ...trap/engines/engine.py.229: process: 0.76695 | load: 0.00299 [I 2022-10-08 16:54:10] ...trap/engines/engine.py.230: loss: 0.20464 [I 2022-10-08 16:54:10] ...optimizers/trijoint.py.151: total_norm: 2.583809 [I 2022-10-08 16:54:18] ...trap/engines/engine.py.224: train: epoch 12 | batch 1350/2304 [I 2022-10-08 16:54:18] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:12:22 [I 2022-10-08 16:54:18] ...trap/engines/engine.py.229: process: 0.78344 | load: 0.00308 [I 2022-10-08 16:54:18] ...trap/engines/engine.py.230: loss: 0.22965 [I 2022-10-08 16:54:18] ...optimizers/trijoint.py.151: total_norm: 2.589857 [I 2022-10-08 16:54:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1360/2304 [I 2022-10-08 16:54:26] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:12:15 [I 2022-10-08 16:54:26] ...trap/engines/engine.py.229: process: 0.78056 | load: 0.00304 [I 2022-10-08 16:54:26] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-08 16:54:26] ...optimizers/trijoint.py.151: total_norm: 2.393122 [I 2022-10-08 16:54:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 1370/2304 [I 2022-10-08 16:54:34] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:12:02 [I 2022-10-08 16:54:34] ...trap/engines/engine.py.229: process: 0.77264 | load: 0.00303 [I 2022-10-08 16:54:34] ...trap/engines/engine.py.230: loss: 0.25926 [I 2022-10-08 16:54:34] ...optimizers/trijoint.py.151: total_norm: 3.545944 [I 2022-10-08 16:54:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 1380/2304 [I 2022-10-08 16:54:42] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:11:51 [I 2022-10-08 16:54:42] ...trap/engines/engine.py.229: process: 0.76255 | load: 0.00309 [I 2022-10-08 16:54:42] ...trap/engines/engine.py.230: loss: 0.15642 [I 2022-10-08 16:54:42] ...optimizers/trijoint.py.151: total_norm: 2.673794 [I 2022-10-08 16:54:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 1390/2304 [I 2022-10-08 16:54:49] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:11:47 [I 2022-10-08 16:54:49] ...trap/engines/engine.py.229: process: 0.76716 | load: 0.00302 [I 2022-10-08 16:54:49] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-08 16:54:49] ...optimizers/trijoint.py.151: total_norm: 2.629910 [I 2022-10-08 16:54:57] ...trap/engines/engine.py.224: train: epoch 12 | batch 1400/2304 [I 2022-10-08 16:54:57] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:11:43 [I 2022-10-08 16:54:57] ...trap/engines/engine.py.229: process: 0.78383 | load: 0.00302 [I 2022-10-08 16:54:57] ...trap/engines/engine.py.230: loss: 0.29080 [I 2022-10-08 16:54:57] ...optimizers/trijoint.py.151: total_norm: 3.865696 [I 2022-10-08 16:55:05] ...trap/engines/engine.py.224: train: epoch 12 | batch 1410/2304 [I 2022-10-08 16:55:05] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:11:34 [I 2022-10-08 16:55:05] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00306 [I 2022-10-08 16:55:05] ...trap/engines/engine.py.230: loss: 0.25266 [I 2022-10-08 16:55:05] ...optimizers/trijoint.py.151: total_norm: 3.661560 [I 2022-10-08 16:55:13] ...trap/engines/engine.py.224: train: epoch 12 | batch 1420/2304 [I 2022-10-08 16:55:13] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:11:22 [I 2022-10-08 16:55:13] ...trap/engines/engine.py.229: process: 0.77545 | load: 0.00305 [I 2022-10-08 16:55:13] ...trap/engines/engine.py.230: loss: 0.18356 [I 2022-10-08 16:55:13] ...optimizers/trijoint.py.151: total_norm: 2.260091 [I 2022-10-08 16:55:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1430/2304 [I 2022-10-08 16:55:21] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:11:18 [I 2022-10-08 16:55:21] ...trap/engines/engine.py.229: process: 0.76430 | load: 0.00309 [I 2022-10-08 16:55:21] ...trap/engines/engine.py.230: loss: 0.19361 [I 2022-10-08 16:55:21] ...optimizers/trijoint.py.151: total_norm: 3.014566 [I 2022-10-08 16:55:28] ...trap/engines/engine.py.224: train: epoch 12 | batch 1440/2304 [I 2022-10-08 16:55:28] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:11:10 [I 2022-10-08 16:55:28] ...trap/engines/engine.py.229: process: 0.76224 | load: 0.00308 [I 2022-10-08 16:55:28] ...trap/engines/engine.py.230: loss: 0.21234 [I 2022-10-08 16:55:28] ...optimizers/trijoint.py.151: total_norm: 2.615230 [I 2022-10-08 16:55:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1450/2304 [I 2022-10-08 16:55:36] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:11:07 [I 2022-10-08 16:55:36] ...trap/engines/engine.py.229: process: 0.78193 | load: 0.00314 [I 2022-10-08 16:55:36] ...trap/engines/engine.py.230: loss: 0.18946 [I 2022-10-08 16:55:36] ...optimizers/trijoint.py.151: total_norm: 2.449680 [I 2022-10-08 16:55:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 1460/2304 [I 2022-10-08 16:55:44] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:10:55 [I 2022-10-08 16:55:44] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00311 [I 2022-10-08 16:55:44] ...trap/engines/engine.py.230: loss: 0.26169 [I 2022-10-08 16:55:44] ...optimizers/trijoint.py.151: total_norm: 3.642190 [I 2022-10-08 16:55:52] ...trap/engines/engine.py.224: train: epoch 12 | batch 1470/2304 [I 2022-10-08 16:55:52] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:10:50 [I 2022-10-08 16:55:52] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00312 [I 2022-10-08 16:55:52] ...trap/engines/engine.py.230: loss: 0.24543 [I 2022-10-08 16:55:52] ...optimizers/trijoint.py.151: total_norm: 2.581272 [I 2022-10-08 16:56:00] ...trap/engines/engine.py.224: train: epoch 12 | batch 1480/2304 [I 2022-10-08 16:56:00] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:10:43 [I 2022-10-08 16:56:00] ...trap/engines/engine.py.229: process: 0.79032 | load: 0.00299 [I 2022-10-08 16:56:00] ...trap/engines/engine.py.230: loss: 0.29255 [I 2022-10-08 16:56:00] ...optimizers/trijoint.py.151: total_norm: 4.007531 [I 2022-10-08 16:56:07] ...trap/engines/engine.py.224: train: epoch 12 | batch 1490/2304 [I 2022-10-08 16:56:07] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:10:32 [I 2022-10-08 16:56:07] ...trap/engines/engine.py.229: process: 0.77688 | load: 0.00303 [I 2022-10-08 16:56:07] ...trap/engines/engine.py.230: loss: 0.19311 [I 2022-10-08 16:56:07] ...optimizers/trijoint.py.151: total_norm: 2.246217 [I 2022-10-08 16:56:15] ...trap/engines/engine.py.224: train: epoch 12 | batch 1500/2304 [I 2022-10-08 16:56:15] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:10:24 [I 2022-10-08 16:56:15] ...trap/engines/engine.py.229: process: 0.77583 | load: 0.00304 [I 2022-10-08 16:56:15] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-08 16:56:15] ...optimizers/trijoint.py.151: total_norm: 2.039212 [I 2022-10-08 16:56:23] ...trap/engines/engine.py.224: train: epoch 12 | batch 1510/2304 [I 2022-10-08 16:56:23] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:10:17 [I 2022-10-08 16:56:23] ...trap/engines/engine.py.229: process: 0.77004 | load: 0.00303 [I 2022-10-08 16:56:23] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-08 16:56:23] ...optimizers/trijoint.py.151: total_norm: 2.725353 [I 2022-10-08 16:56:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1520/2304 [I 2022-10-08 16:56:31] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:10:13 [I 2022-10-08 16:56:31] ...trap/engines/engine.py.229: process: 0.80144 | load: 0.00300 [I 2022-10-08 16:56:31] ...trap/engines/engine.py.230: loss: 0.27069 [I 2022-10-08 16:56:31] ...optimizers/trijoint.py.151: total_norm: 3.739180 [I 2022-10-08 16:56:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 1530/2304 [I 2022-10-08 16:56:39] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:10:11 [I 2022-10-08 16:56:39] ...trap/engines/engine.py.229: process: 0.79516 | load: 0.00306 [I 2022-10-08 16:56:39] ...trap/engines/engine.py.230: loss: 0.26498 [I 2022-10-08 16:56:39] ...optimizers/trijoint.py.151: total_norm: 3.356057 [I 2022-10-08 16:56:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 1540/2304 [I 2022-10-08 16:56:47] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:09:54 [I 2022-10-08 16:56:47] ...trap/engines/engine.py.229: process: 0.78034 | load: 0.00302 [I 2022-10-08 16:56:47] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-08 16:56:47] ...optimizers/trijoint.py.151: total_norm: 2.374168 [I 2022-10-08 16:56:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 1550/2304 [I 2022-10-08 16:56:54] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:09:51 [I 2022-10-08 16:56:54] ...trap/engines/engine.py.229: process: 0.77755 | load: 0.00293 [I 2022-10-08 16:56:54] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-08 16:56:54] ...optimizers/trijoint.py.151: total_norm: 1.664489 [I 2022-10-08 16:57:02] ...trap/engines/engine.py.224: train: epoch 12 | batch 1560/2304 [I 2022-10-08 16:57:02] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:09:38 [I 2022-10-08 16:57:02] ...trap/engines/engine.py.229: process: 0.78167 | load: 0.00301 [I 2022-10-08 16:57:02] ...trap/engines/engine.py.230: loss: 0.15599 [I 2022-10-08 16:57:02] ...optimizers/trijoint.py.151: total_norm: 2.628359 [I 2022-10-08 16:57:10] ...trap/engines/engine.py.224: train: epoch 12 | batch 1570/2304 [I 2022-10-08 16:57:10] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:09:30 [I 2022-10-08 16:57:10] ...trap/engines/engine.py.229: process: 0.78078 | load: 0.00303 [I 2022-10-08 16:57:10] ...trap/engines/engine.py.230: loss: 0.21181 [I 2022-10-08 16:57:10] ...optimizers/trijoint.py.151: total_norm: 3.186816 [I 2022-10-08 16:57:18] ...trap/engines/engine.py.224: train: epoch 12 | batch 1580/2304 [I 2022-10-08 16:57:18] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:09:21 [I 2022-10-08 16:57:18] ...trap/engines/engine.py.229: process: 0.78149 | load: 0.00301 [I 2022-10-08 16:57:18] ...trap/engines/engine.py.230: loss: 0.26778 [I 2022-10-08 16:57:18] ...optimizers/trijoint.py.151: total_norm: 2.954126 [I 2022-10-08 16:57:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1590/2304 [I 2022-10-08 16:57:26] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:09:14 [I 2022-10-08 16:57:26] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00301 [I 2022-10-08 16:57:26] ...trap/engines/engine.py.230: loss: 0.25593 [I 2022-10-08 16:57:26] ...optimizers/trijoint.py.151: total_norm: 3.236904 [I 2022-10-08 16:57:33] ...trap/engines/engine.py.224: train: epoch 12 | batch 1600/2304 [I 2022-10-08 16:57:33] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:09:06 [I 2022-10-08 16:57:33] ...trap/engines/engine.py.229: process: 0.77099 | load: 0.00301 [I 2022-10-08 16:57:33] ...trap/engines/engine.py.230: loss: 0.19697 [I 2022-10-08 16:57:33] ...optimizers/trijoint.py.151: total_norm: 2.965384 [I 2022-10-08 16:57:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1610/2304 [I 2022-10-08 16:57:41] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:08:59 [I 2022-10-08 16:57:41] ...trap/engines/engine.py.229: process: 0.77357 | load: 0.00300 [I 2022-10-08 16:57:41] ...trap/engines/engine.py.230: loss: 0.21718 [I 2022-10-08 16:57:41] ...optimizers/trijoint.py.151: total_norm: 2.821040 [I 2022-10-08 16:57:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 1620/2304 [I 2022-10-08 16:57:49] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:08:48 [I 2022-10-08 16:57:49] ...trap/engines/engine.py.229: process: 0.76387 | load: 0.00302 [I 2022-10-08 16:57:49] ...trap/engines/engine.py.230: loss: 0.19343 [I 2022-10-08 16:57:49] ...optimizers/trijoint.py.151: total_norm: 2.351378 [I 2022-10-08 16:57:57] ...trap/engines/engine.py.224: train: epoch 12 | batch 1630/2304 [I 2022-10-08 16:57:57] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:08:40 [I 2022-10-08 16:57:57] ...trap/engines/engine.py.229: process: 0.75278 | load: 0.00304 [I 2022-10-08 16:57:57] ...trap/engines/engine.py.230: loss: 0.11924 [I 2022-10-08 16:57:57] ...optimizers/trijoint.py.151: total_norm: 2.119935 [I 2022-10-08 16:58:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 1640/2304 [I 2022-10-08 16:58:04] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:08:37 [I 2022-10-08 16:58:04] ...trap/engines/engine.py.229: process: 0.77904 | load: 0.00304 [I 2022-10-08 16:58:04] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-08 16:58:04] ...optimizers/trijoint.py.151: total_norm: 3.142931 [I 2022-10-08 16:58:12] ...trap/engines/engine.py.224: train: epoch 12 | batch 1650/2304 [I 2022-10-08 16:58:12] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:08:31 [I 2022-10-08 16:58:12] ...trap/engines/engine.py.229: process: 0.78305 | load: 0.00304 [I 2022-10-08 16:58:12] ...trap/engines/engine.py.230: loss: 0.17262 [I 2022-10-08 16:58:12] ...optimizers/trijoint.py.151: total_norm: 3.260424 [I 2022-10-08 16:58:20] ...trap/engines/engine.py.224: train: epoch 12 | batch 1660/2304 [I 2022-10-08 16:58:20] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:08:23 [I 2022-10-08 16:58:20] ...trap/engines/engine.py.229: process: 0.78292 | load: 0.00302 [I 2022-10-08 16:58:20] ...trap/engines/engine.py.230: loss: 0.19693 [I 2022-10-08 16:58:20] ...optimizers/trijoint.py.151: total_norm: 3.506941 [I 2022-10-08 16:58:28] ...trap/engines/engine.py.224: train: epoch 12 | batch 1670/2304 [I 2022-10-08 16:58:28] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:08:14 [I 2022-10-08 16:58:28] ...trap/engines/engine.py.229: process: 0.79580 | load: 0.00301 [I 2022-10-08 16:58:28] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-08 16:58:28] ...optimizers/trijoint.py.151: total_norm: 2.278477 [I 2022-10-08 16:58:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1680/2304 [I 2022-10-08 16:58:36] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:08:06 [I 2022-10-08 16:58:36] ...trap/engines/engine.py.229: process: 0.78596 | load: 0.00311 [I 2022-10-08 16:58:36] ...trap/engines/engine.py.230: loss: 0.18627 [I 2022-10-08 16:58:36] ...optimizers/trijoint.py.151: total_norm: 2.694585 [I 2022-10-08 16:58:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 1690/2304 [I 2022-10-08 16:58:44] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:07:57 [I 2022-10-08 16:58:44] ...trap/engines/engine.py.229: process: 0.78837 | load: 0.00309 [I 2022-10-08 16:58:44] ...trap/engines/engine.py.230: loss: 0.18618 [I 2022-10-08 16:58:44] ...optimizers/trijoint.py.151: total_norm: 2.740131 [I 2022-10-08 16:58:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1700/2304 [I 2022-10-08 16:58:51] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:07:49 [I 2022-10-08 16:58:51] ...trap/engines/engine.py.229: process: 0.77646 | load: 0.00310 [I 2022-10-08 16:58:51] ...trap/engines/engine.py.230: loss: 0.25654 [I 2022-10-08 16:58:51] ...optimizers/trijoint.py.151: total_norm: 3.894120 [I 2022-10-08 16:58:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 1710/2304 [I 2022-10-08 16:58:59] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:07:43 [I 2022-10-08 16:58:59] ...trap/engines/engine.py.229: process: 0.77871 | load: 0.00314 [I 2022-10-08 16:58:59] ...trap/engines/engine.py.230: loss: 0.13591 [I 2022-10-08 16:58:59] ...optimizers/trijoint.py.151: total_norm: 2.585225 [I 2022-10-08 16:59:07] ...trap/engines/engine.py.224: train: epoch 12 | batch 1720/2304 [I 2022-10-08 16:59:07] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:07:32 [I 2022-10-08 16:59:07] ...trap/engines/engine.py.229: process: 0.77159 | load: 0.00309 [I 2022-10-08 16:59:07] ...trap/engines/engine.py.230: loss: 0.19163 [I 2022-10-08 16:59:07] ...optimizers/trijoint.py.151: total_norm: 2.447394 [I 2022-10-08 16:59:15] ...trap/engines/engine.py.224: train: epoch 12 | batch 1730/2304 [I 2022-10-08 16:59:15] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:07:25 [I 2022-10-08 16:59:15] ...trap/engines/engine.py.229: process: 0.78168 | load: 0.00308 [I 2022-10-08 16:59:15] ...trap/engines/engine.py.230: loss: 0.17804 [I 2022-10-08 16:59:15] ...optimizers/trijoint.py.151: total_norm: 2.283244 [I 2022-10-08 16:59:22] ...trap/engines/engine.py.224: train: epoch 12 | batch 1740/2304 [I 2022-10-08 16:59:22] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:07:16 [I 2022-10-08 16:59:22] ...trap/engines/engine.py.229: process: 0.76143 | load: 0.00311 [I 2022-10-08 16:59:22] ...trap/engines/engine.py.230: loss: 0.22959 [I 2022-10-08 16:59:22] ...optimizers/trijoint.py.151: total_norm: 3.576388 [I 2022-10-08 16:59:30] ...trap/engines/engine.py.224: train: epoch 12 | batch 1750/2304 [I 2022-10-08 16:59:30] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:07:10 [I 2022-10-08 16:59:30] ...trap/engines/engine.py.229: process: 0.77792 | load: 0.00312 [I 2022-10-08 16:59:30] ...trap/engines/engine.py.230: loss: 0.24619 [I 2022-10-08 16:59:30] ...optimizers/trijoint.py.151: total_norm: 3.435896 [I 2022-10-08 16:59:38] ...trap/engines/engine.py.224: train: epoch 12 | batch 1760/2304 [I 2022-10-08 16:59:38] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:07:02 [I 2022-10-08 16:59:38] ...trap/engines/engine.py.229: process: 0.78484 | load: 0.00304 [I 2022-10-08 16:59:38] ...trap/engines/engine.py.230: loss: 0.14395 [I 2022-10-08 16:59:38] ...optimizers/trijoint.py.151: total_norm: 1.732893 [I 2022-10-08 16:59:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1770/2304 [I 2022-10-08 16:59:46] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:06:54 [I 2022-10-08 16:59:46] ...trap/engines/engine.py.229: process: 0.78310 | load: 0.00315 [I 2022-10-08 16:59:46] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-08 16:59:46] ...optimizers/trijoint.py.151: total_norm: 3.393062 [I 2022-10-08 16:59:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 1780/2304 [I 2022-10-08 16:59:54] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:06:46 [I 2022-10-08 16:59:54] ...trap/engines/engine.py.229: process: 0.77740 | load: 0.00308 [I 2022-10-08 16:59:54] ...trap/engines/engine.py.230: loss: 0.24836 [I 2022-10-08 16:59:54] ...optimizers/trijoint.py.151: total_norm: 2.805766 [I 2022-10-08 17:00:02] ...trap/engines/engine.py.224: train: epoch 12 | batch 1790/2304 [I 2022-10-08 17:00:02] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:06:37 [I 2022-10-08 17:00:02] ...trap/engines/engine.py.229: process: 0.78324 | load: 0.00314 [I 2022-10-08 17:00:02] ...trap/engines/engine.py.230: loss: 0.26758 [I 2022-10-08 17:00:02] ...optimizers/trijoint.py.151: total_norm: 3.187755 [I 2022-10-08 17:00:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 1800/2304 [I 2022-10-08 17:00:09] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:06:33 [I 2022-10-08 17:00:09] ...trap/engines/engine.py.229: process: 0.79031 | load: 0.00310 [I 2022-10-08 17:00:09] ...trap/engines/engine.py.230: loss: 0.18661 [I 2022-10-08 17:00:09] ...optimizers/trijoint.py.151: total_norm: 1.852729 [I 2022-10-08 17:00:17] ...trap/engines/engine.py.224: train: epoch 12 | batch 1810/2304 [I 2022-10-08 17:00:17] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:06:23 [I 2022-10-08 17:00:17] ...trap/engines/engine.py.229: process: 0.77565 | load: 0.00314 [I 2022-10-08 17:00:17] ...trap/engines/engine.py.230: loss: 0.20104 [I 2022-10-08 17:00:17] ...optimizers/trijoint.py.151: total_norm: 2.878172 [I 2022-10-08 17:00:25] ...trap/engines/engine.py.224: train: epoch 12 | batch 1820/2304 [I 2022-10-08 17:00:25] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:06:16 [I 2022-10-08 17:00:25] ...trap/engines/engine.py.229: process: 0.78161 | load: 0.00311 [I 2022-10-08 17:00:25] ...trap/engines/engine.py.230: loss: 0.22021 [I 2022-10-08 17:00:25] ...optimizers/trijoint.py.151: total_norm: 2.280227 [I 2022-10-08 17:00:33] ...trap/engines/engine.py.224: train: epoch 12 | batch 1830/2304 [I 2022-10-08 17:00:33] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:06:07 [I 2022-10-08 17:00:33] ...trap/engines/engine.py.229: process: 0.76969 | load: 0.00305 [I 2022-10-08 17:00:33] ...trap/engines/engine.py.230: loss: 0.16549 [I 2022-10-08 17:00:33] ...optimizers/trijoint.py.151: total_norm: 2.052967 [I 2022-10-08 17:00:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1840/2304 [I 2022-10-08 17:00:41] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:05:57 [I 2022-10-08 17:00:41] ...trap/engines/engine.py.229: process: 0.76883 | load: 0.00308 [I 2022-10-08 17:00:41] ...trap/engines/engine.py.230: loss: 0.23456 [I 2022-10-08 17:00:41] ...optimizers/trijoint.py.151: total_norm: 3.215761 [I 2022-10-08 17:00:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 1850/2304 [I 2022-10-08 17:00:49] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:05:52 [I 2022-10-08 17:00:49] ...trap/engines/engine.py.229: process: 0.77794 | load: 0.00303 [I 2022-10-08 17:00:49] ...trap/engines/engine.py.230: loss: 0.15636 [I 2022-10-08 17:00:49] ...optimizers/trijoint.py.151: total_norm: 2.664693 [I 2022-10-08 17:00:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1860/2304 [I 2022-10-08 17:00:56] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:05:45 [I 2022-10-08 17:00:56] ...trap/engines/engine.py.229: process: 0.78671 | load: 0.00302 [I 2022-10-08 17:00:56] ...trap/engines/engine.py.230: loss: 0.20145 [I 2022-10-08 17:00:56] ...optimizers/trijoint.py.151: total_norm: 2.341539 [I 2022-10-08 17:01:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 1870/2304 [I 2022-10-08 17:01:04] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:05:36 [I 2022-10-08 17:01:04] ...trap/engines/engine.py.229: process: 0.77657 | load: 0.00306 [I 2022-10-08 17:01:04] ...trap/engines/engine.py.230: loss: 0.22864 [I 2022-10-08 17:01:04] ...optimizers/trijoint.py.151: total_norm: 3.189960 [I 2022-10-08 17:01:12] ...trap/engines/engine.py.224: train: epoch 12 | batch 1880/2304 [I 2022-10-08 17:01:12] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:05:28 [I 2022-10-08 17:01:12] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00308 [I 2022-10-08 17:01:12] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-08 17:01:12] ...optimizers/trijoint.py.151: total_norm: 2.486446 [I 2022-10-08 17:01:20] ...trap/engines/engine.py.224: train: epoch 12 | batch 1890/2304 [I 2022-10-08 17:01:20] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:05:21 [I 2022-10-08 17:01:20] ...trap/engines/engine.py.229: process: 0.76602 | load: 0.00305 [I 2022-10-08 17:01:20] ...trap/engines/engine.py.230: loss: 0.22281 [I 2022-10-08 17:01:20] ...optimizers/trijoint.py.151: total_norm: 1.880235 [I 2022-10-08 17:01:27] ...trap/engines/engine.py.224: train: epoch 12 | batch 1900/2304 [I 2022-10-08 17:01:27] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:05:13 [I 2022-10-08 17:01:27] ...trap/engines/engine.py.229: process: 0.76831 | load: 0.00309 [I 2022-10-08 17:01:27] ...trap/engines/engine.py.230: loss: 0.22387 [I 2022-10-08 17:01:27] ...optimizers/trijoint.py.151: total_norm: 2.614728 [I 2022-10-08 17:01:35] ...trap/engines/engine.py.224: train: epoch 12 | batch 1910/2304 [I 2022-10-08 17:01:35] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:05:05 [I 2022-10-08 17:01:35] ...trap/engines/engine.py.229: process: 0.77295 | load: 0.00308 [I 2022-10-08 17:01:35] ...trap/engines/engine.py.230: loss: 0.21345 [I 2022-10-08 17:01:35] ...optimizers/trijoint.py.151: total_norm: 3.557914 [I 2022-10-08 17:01:43] ...trap/engines/engine.py.224: train: epoch 12 | batch 1920/2304 [I 2022-10-08 17:01:43] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:04:58 [I 2022-10-08 17:01:43] ...trap/engines/engine.py.229: process: 0.78401 | load: 0.00309 [I 2022-10-08 17:01:43] ...trap/engines/engine.py.230: loss: 0.20132 [I 2022-10-08 17:01:43] ...optimizers/trijoint.py.151: total_norm: 3.179937 [I 2022-10-08 17:01:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1930/2304 [I 2022-10-08 17:01:51] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:04:49 [I 2022-10-08 17:01:51] ...trap/engines/engine.py.229: process: 0.77186 | load: 0.00311 [I 2022-10-08 17:01:51] ...trap/engines/engine.py.230: loss: 0.22972 [I 2022-10-08 17:01:51] ...optimizers/trijoint.py.151: total_norm: 2.664535 [I 2022-10-08 17:01:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 1940/2304 [I 2022-10-08 17:01:59] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:04:43 [I 2022-10-08 17:01:59] ...trap/engines/engine.py.229: process: 0.77935 | load: 0.00312 [I 2022-10-08 17:01:59] ...trap/engines/engine.py.230: loss: 0.19558 [I 2022-10-08 17:01:59] ...optimizers/trijoint.py.151: total_norm: 3.098842 [I 2022-10-08 17:02:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1950/2304 [I 2022-10-08 17:02:06] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:04:34 [I 2022-10-08 17:02:06] ...trap/engines/engine.py.229: process: 0.78423 | load: 0.00315 [I 2022-10-08 17:02:06] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-08 17:02:06] ...optimizers/trijoint.py.151: total_norm: 2.219810 [I 2022-10-08 17:02:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 1960/2304 [I 2022-10-08 17:02:14] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:04:28 [I 2022-10-08 17:02:14] ...trap/engines/engine.py.229: process: 0.79969 | load: 0.00300 [I 2022-10-08 17:02:14] ...trap/engines/engine.py.230: loss: 0.19344 [I 2022-10-08 17:02:14] ...optimizers/trijoint.py.151: total_norm: 3.479349 [I 2022-10-08 17:02:22] ...trap/engines/engine.py.224: train: epoch 12 | batch 1970/2304 [I 2022-10-08 17:02:22] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:04:20 [I 2022-10-08 17:02:22] ...trap/engines/engine.py.229: process: 0.78588 | load: 0.00297 [I 2022-10-08 17:02:22] ...trap/engines/engine.py.230: loss: 0.21566 [I 2022-10-08 17:02:22] ...optimizers/trijoint.py.151: total_norm: 2.922895 [I 2022-10-08 17:02:30] ...trap/engines/engine.py.224: train: epoch 12 | batch 1980/2304 [I 2022-10-08 17:02:30] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:04:11 [I 2022-10-08 17:02:30] ...trap/engines/engine.py.229: process: 0.78316 | load: 0.00304 [I 2022-10-08 17:02:30] ...trap/engines/engine.py.230: loss: 0.21438 [I 2022-10-08 17:02:30] ...optimizers/trijoint.py.151: total_norm: 3.245396 [I 2022-10-08 17:02:38] ...trap/engines/engine.py.224: train: epoch 12 | batch 1990/2304 [I 2022-10-08 17:02:38] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:04:04 [I 2022-10-08 17:02:38] ...trap/engines/engine.py.229: process: 0.78056 | load: 0.00303 [I 2022-10-08 17:02:38] ...trap/engines/engine.py.230: loss: 0.21681 [I 2022-10-08 17:02:38] ...optimizers/trijoint.py.151: total_norm: 2.711412 [I 2022-10-08 17:02:45] ...trap/engines/engine.py.224: train: epoch 12 | batch 2000/2304 [I 2022-10-08 17:02:45] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:03:55 [I 2022-10-08 17:02:45] ...trap/engines/engine.py.229: process: 0.77635 | load: 0.00303 [I 2022-10-08 17:02:45] ...trap/engines/engine.py.230: loss: 0.11804 [I 2022-10-08 17:02:45] ...optimizers/trijoint.py.151: total_norm: 2.686886 [I 2022-10-08 17:02:53] ...trap/engines/engine.py.224: train: epoch 12 | batch 2010/2304 [I 2022-10-08 17:02:53] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:03:49 [I 2022-10-08 17:02:53] ...trap/engines/engine.py.229: process: 0.77787 | load: 0.00299 [I 2022-10-08 17:02:53] ...trap/engines/engine.py.230: loss: 0.21706 [I 2022-10-08 17:02:53] ...optimizers/trijoint.py.151: total_norm: 2.256079 [I 2022-10-08 17:03:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 2020/2304 [I 2022-10-08 17:03:01] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:03:41 [I 2022-10-08 17:03:01] ...trap/engines/engine.py.229: process: 0.78679 | load: 0.00302 [I 2022-10-08 17:03:01] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-08 17:03:01] ...optimizers/trijoint.py.151: total_norm: 2.553896 [I 2022-10-08 17:03:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 2030/2304 [I 2022-10-08 17:03:09] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:03:32 [I 2022-10-08 17:03:09] ...trap/engines/engine.py.229: process: 0.77977 | load: 0.00301 [I 2022-10-08 17:03:09] ...trap/engines/engine.py.230: loss: 0.17015 [I 2022-10-08 17:03:09] ...optimizers/trijoint.py.151: total_norm: 3.342167 [I 2022-10-08 17:03:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 2040/2304 [I 2022-10-08 17:03:16] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:03:25 [I 2022-10-08 17:03:16] ...trap/engines/engine.py.229: process: 0.79337 | load: 0.00301 [I 2022-10-08 17:03:16] ...trap/engines/engine.py.230: loss: 0.20764 [I 2022-10-08 17:03:16] ...optimizers/trijoint.py.151: total_norm: 1.906197 [I 2022-10-08 17:03:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 2050/2304 [I 2022-10-08 17:03:24] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:03:16 [I 2022-10-08 17:03:24] ...trap/engines/engine.py.229: process: 0.77240 | load: 0.00296 [I 2022-10-08 17:03:24] ...trap/engines/engine.py.230: loss: 0.18451 [I 2022-10-08 17:03:24] ...optimizers/trijoint.py.151: total_norm: 2.838738 [I 2022-10-08 17:03:32] ...trap/engines/engine.py.224: train: epoch 12 | batch 2060/2304 [I 2022-10-08 17:03:32] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:03:09 [I 2022-10-08 17:03:32] ...trap/engines/engine.py.229: process: 0.77548 | load: 0.00303 [I 2022-10-08 17:03:32] ...trap/engines/engine.py.230: loss: 0.20270 [I 2022-10-08 17:03:32] ...optimizers/trijoint.py.151: total_norm: 2.729031 [I 2022-10-08 17:03:40] ...trap/engines/engine.py.224: train: epoch 12 | batch 2070/2304 [I 2022-10-08 17:03:40] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:03:01 [I 2022-10-08 17:03:40] ...trap/engines/engine.py.229: process: 0.76668 | load: 0.00300 [I 2022-10-08 17:03:40] ...trap/engines/engine.py.230: loss: 0.18276 [I 2022-10-08 17:03:40] ...optimizers/trijoint.py.151: total_norm: 2.158817 [I 2022-10-08 17:03:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 2080/2304 [I 2022-10-08 17:03:47] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:02:52 [I 2022-10-08 17:03:47] ...trap/engines/engine.py.229: process: 0.76543 | load: 0.00298 [I 2022-10-08 17:03:47] ...trap/engines/engine.py.230: loss: 0.17059 [I 2022-10-08 17:03:47] ...optimizers/trijoint.py.151: total_norm: 3.254028 [I 2022-10-08 17:03:55] ...trap/engines/engine.py.224: train: epoch 12 | batch 2090/2304 [I 2022-10-08 17:03:55] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:02:45 [I 2022-10-08 17:03:55] ...trap/engines/engine.py.229: process: 0.77832 | load: 0.00309 [I 2022-10-08 17:03:55] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-08 17:03:55] ...optimizers/trijoint.py.151: total_norm: 2.262079 [I 2022-10-08 17:04:03] ...trap/engines/engine.py.224: train: epoch 12 | batch 2100/2304 [I 2022-10-08 17:04:03] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:02:38 [I 2022-10-08 17:04:03] ...trap/engines/engine.py.229: process: 0.77342 | load: 0.00305 [I 2022-10-08 17:04:03] ...trap/engines/engine.py.230: loss: 0.18498 [I 2022-10-08 17:04:03] ...optimizers/trijoint.py.151: total_norm: 3.405716 [I 2022-10-08 17:04:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 2110/2304 [I 2022-10-08 17:04:11] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:02:31 [I 2022-10-08 17:04:11] ...trap/engines/engine.py.229: process: 0.78081 | load: 0.00300 [I 2022-10-08 17:04:11] ...trap/engines/engine.py.230: loss: 0.19759 [I 2022-10-08 17:04:11] ...optimizers/trijoint.py.151: total_norm: 2.792318 [I 2022-10-08 17:04:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 2120/2304 [I 2022-10-08 17:04:19] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:02:22 [I 2022-10-08 17:04:19] ...trap/engines/engine.py.229: process: 0.77298 | load: 0.00302 [I 2022-10-08 17:04:19] ...trap/engines/engine.py.230: loss: 0.18679 [I 2022-10-08 17:04:19] ...optimizers/trijoint.py.151: total_norm: 3.040818 [I 2022-10-08 17:04:27] ...trap/engines/engine.py.224: train: epoch 12 | batch 2130/2304 [I 2022-10-08 17:04:27] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:02:15 [I 2022-10-08 17:04:27] ...trap/engines/engine.py.229: process: 0.78974 | load: 0.00299 [I 2022-10-08 17:04:27] ...trap/engines/engine.py.230: loss: 0.16705 [I 2022-10-08 17:04:27] ...optimizers/trijoint.py.151: total_norm: 2.094338 [I 2022-10-08 17:04:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 2140/2304 [I 2022-10-08 17:04:34] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:02:07 [I 2022-10-08 17:04:34] ...trap/engines/engine.py.229: process: 0.77119 | load: 0.00300 [I 2022-10-08 17:04:34] ...trap/engines/engine.py.230: loss: 0.18637 [I 2022-10-08 17:04:34] ...optimizers/trijoint.py.151: total_norm: 2.786285 [I 2022-10-08 17:04:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 2150/2304 [I 2022-10-08 17:04:42] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:01:59 [I 2022-10-08 17:04:42] ...trap/engines/engine.py.229: process: 0.77853 | load: 0.00300 [I 2022-10-08 17:04:42] ...trap/engines/engine.py.230: loss: 0.19135 [I 2022-10-08 17:04:42] ...optimizers/trijoint.py.151: total_norm: 2.736881 [I 2022-10-08 17:04:50] ...trap/engines/engine.py.224: train: epoch 12 | batch 2160/2304 [I 2022-10-08 17:04:50] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:01:52 [I 2022-10-08 17:04:50] ...trap/engines/engine.py.229: process: 0.78176 | load: 0.00308 [I 2022-10-08 17:04:50] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-08 17:04:50] ...optimizers/trijoint.py.151: total_norm: 2.129601 [I 2022-10-08 17:04:58] ...trap/engines/engine.py.224: train: epoch 12 | batch 2170/2304 [I 2022-10-08 17:04:58] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:01:44 [I 2022-10-08 17:04:58] ...trap/engines/engine.py.229: process: 0.77706 | load: 0.00304 [I 2022-10-08 17:04:58] ...trap/engines/engine.py.230: loss: 0.16106 [I 2022-10-08 17:04:58] ...optimizers/trijoint.py.151: total_norm: 2.113560 [I 2022-10-08 17:05:05] ...trap/engines/engine.py.224: train: epoch 12 | batch 2180/2304 [I 2022-10-08 17:05:05] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:01:36 [I 2022-10-08 17:05:05] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00307 [I 2022-10-08 17:05:05] ...trap/engines/engine.py.230: loss: 0.19964 [I 2022-10-08 17:05:05] ...optimizers/trijoint.py.151: total_norm: 3.050766 [I 2022-10-08 17:05:13] ...trap/engines/engine.py.224: train: epoch 12 | batch 2190/2304 [I 2022-10-08 17:05:13] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:01:28 [I 2022-10-08 17:05:13] ...trap/engines/engine.py.229: process: 0.78149 | load: 0.00312 [I 2022-10-08 17:05:13] ...trap/engines/engine.py.230: loss: 0.25200 [I 2022-10-08 17:05:13] ...optimizers/trijoint.py.151: total_norm: 3.148198 [I 2022-10-08 17:05:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 2200/2304 [I 2022-10-08 17:05:21] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:01:20 [I 2022-10-08 17:05:21] ...trap/engines/engine.py.229: process: 0.77622 | load: 0.00302 [I 2022-10-08 17:05:21] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-08 17:05:21] ...optimizers/trijoint.py.151: total_norm: 1.908538 [I 2022-10-08 17:05:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 2210/2304 [I 2022-10-08 17:05:29] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:01:13 [I 2022-10-08 17:05:29] ...trap/engines/engine.py.229: process: 0.78274 | load: 0.00307 [I 2022-10-08 17:05:29] ...trap/engines/engine.py.230: loss: 0.20466 [I 2022-10-08 17:05:29] ...optimizers/trijoint.py.151: total_norm: 2.227154 [I 2022-10-08 17:05:37] ...trap/engines/engine.py.224: train: epoch 12 | batch 2220/2304 [I 2022-10-08 17:05:37] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:01:05 [I 2022-10-08 17:05:37] ...trap/engines/engine.py.229: process: 0.77553 | load: 0.00305 [I 2022-10-08 17:05:37] ...trap/engines/engine.py.230: loss: 0.18504 [I 2022-10-08 17:05:37] ...optimizers/trijoint.py.151: total_norm: 1.972044 [I 2022-10-08 17:05:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 2230/2304 [I 2022-10-08 17:05:44] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:00:57 [I 2022-10-08 17:05:44] ...trap/engines/engine.py.229: process: 0.78387 | load: 0.00300 [I 2022-10-08 17:05:44] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-08 17:05:44] ...optimizers/trijoint.py.151: total_norm: 3.044347 [I 2022-10-08 17:05:52] ...trap/engines/engine.py.224: train: epoch 12 | batch 2240/2304 [I 2022-10-08 17:05:52] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:00:49 [I 2022-10-08 17:05:52] ...trap/engines/engine.py.229: process: 0.76556 | load: 0.00301 [I 2022-10-08 17:05:52] ...trap/engines/engine.py.230: loss: 0.11852 [I 2022-10-08 17:05:52] ...optimizers/trijoint.py.151: total_norm: 1.564151 [I 2022-10-08 17:06:00] ...trap/engines/engine.py.224: train: epoch 12 | batch 2250/2304 [I 2022-10-08 17:06:00] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:00:41 [I 2022-10-08 17:06:00] ...trap/engines/engine.py.229: process: 0.76935 | load: 0.00309 [I 2022-10-08 17:06:00] ...trap/engines/engine.py.230: loss: 0.15942 [I 2022-10-08 17:06:00] ...optimizers/trijoint.py.151: total_norm: 2.314920 [I 2022-10-08 17:06:08] ...trap/engines/engine.py.224: train: epoch 12 | batch 2260/2304 [I 2022-10-08 17:06:08] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:00:34 [I 2022-10-08 17:06:08] ...trap/engines/engine.py.229: process: 0.77497 | load: 0.00302 [I 2022-10-08 17:06:08] ...trap/engines/engine.py.230: loss: 0.19186 [I 2022-10-08 17:06:08] ...optimizers/trijoint.py.151: total_norm: 2.265733 [I 2022-10-08 17:06:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 2270/2304 [I 2022-10-08 17:06:16] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:00:26 [I 2022-10-08 17:06:16] ...trap/engines/engine.py.229: process: 0.77501 | load: 0.00307 [I 2022-10-08 17:06:16] ...trap/engines/engine.py.230: loss: 0.15869 [I 2022-10-08 17:06:16] ...optimizers/trijoint.py.151: total_norm: 2.913655 [I 2022-10-08 17:06:23] ...trap/engines/engine.py.224: train: epoch 12 | batch 2280/2304 [I 2022-10-08 17:06:23] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:00:18 [I 2022-10-08 17:06:23] ...trap/engines/engine.py.229: process: 0.78128 | load: 0.00309 [I 2022-10-08 17:06:23] ...trap/engines/engine.py.230: loss: 0.30505 [I 2022-10-08 17:06:23] ...optimizers/trijoint.py.151: total_norm: 3.735477 [I 2022-10-08 17:06:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 2290/2304 [I 2022-10-08 17:06:31] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:00:10 [I 2022-10-08 17:06:31] ...trap/engines/engine.py.229: process: 0.77528 | load: 0.00304 [I 2022-10-08 17:06:31] ...trap/engines/engine.py.230: loss: 0.16033 [I 2022-10-08 17:06:31] ...optimizers/trijoint.py.151: total_norm: 2.967185 [I 2022-10-08 17:06:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 2300/2304 [I 2022-10-08 17:06:39] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:00:03 [I 2022-10-08 17:06:39] ...trap/engines/engine.py.229: process: 0.77242 | load: 0.00017 [I 2022-10-08 17:06:39] ...trap/engines/engine.py.230: loss: 0.24728 [I 2022-10-08 17:06:39] ...optimizers/trijoint.py.151: total_norm: 3.218284 [I 2022-10-08 17:06:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 2304/2304 [I 2022-10-08 17:06:42] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:00:00 [I 2022-10-08 17:06:42] ...trap/engines/engine.py.229: process: 0.76308 | load: 0.00017 [I 2022-10-08 17:06:42] ...trap/engines/engine.py.230: loss: 0.21753 [I 2022-10-08 17:06:42] ...optimizers/trijoint.py.151: total_norm: 2.815205 [S 2022-10-08 17:06:43] ...trap/engines/engine.py.129: train_epoch.epoch: 12 [S 2022-10-08 17:06:43] ...trap/engines/engine.py.129: train_epoch.loss: 0.1948121914539058 [I 2022-10-08 17:06:43] ...s/criterions/triplet.py.84: margin = 0.11500000000000005 [I 2022-10-08 17:06:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 12 [I 2022-10-08 17:06:44] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 17:06:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 17:06:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 17:06:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 17:06:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 17:06:46] ...trap/engines/engine.py.321: eval: epoch 12 | batch 0/510 [I 2022-10-08 17:06:46] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:37 [I 2022-10-08 17:06:46] ...trap/engines/engine.py.326: process: 2.78009 | load: 2.27679 [I 2022-10-08 17:06:51] ...trap/engines/engine.py.321: eval: epoch 12 | batch 10/510 [I 2022-10-08 17:06:51] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:40 [I 2022-10-08 17:06:51] ...trap/engines/engine.py.326: process: 0.38919 | load: 0.00018 [I 2022-10-08 17:06:56] ...trap/engines/engine.py.321: eval: epoch 12 | batch 20/510 [I 2022-10-08 17:06:56] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:05:10 [I 2022-10-08 17:06:56] ...trap/engines/engine.py.326: process: 0.77369 | load: 0.00019 [I 2022-10-08 17:07:02] ...trap/engines/engine.py.321: eval: epoch 12 | batch 30/510 [I 2022-10-08 17:07:02] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:17 [I 2022-10-08 17:07:02] ...trap/engines/engine.py.326: process: 0.41346 | load: 0.00019 [I 2022-10-08 17:07:09] ...trap/engines/engine.py.321: eval: epoch 12 | batch 40/510 [I 2022-10-08 17:07:09] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:04 [I 2022-10-08 17:07:09] ...trap/engines/engine.py.326: process: 0.60995 | load: 0.00019 [I 2022-10-08 17:07:13] ...trap/engines/engine.py.321: eval: epoch 12 | batch 50/510 [I 2022-10-08 17:07:13] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:27 [I 2022-10-08 17:07:13] ...trap/engines/engine.py.326: process: 0.40565 | load: 0.00017 [I 2022-10-08 17:07:18] ...trap/engines/engine.py.321: eval: epoch 12 | batch 60/510 [I 2022-10-08 17:07:18] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:54 [I 2022-10-08 17:07:18] ...trap/engines/engine.py.326: process: 0.55327 | load: 0.00018 [I 2022-10-08 17:07:24] ...trap/engines/engine.py.321: eval: epoch 12 | batch 70/510 [I 2022-10-08 17:07:24] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:01 [I 2022-10-08 17:07:24] ...trap/engines/engine.py.326: process: 0.55194 | load: 0.00019 [I 2022-10-08 17:07:29] ...trap/engines/engine.py.321: eval: epoch 12 | batch 80/510 [I 2022-10-08 17:07:29] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:53 [I 2022-10-08 17:07:29] ...trap/engines/engine.py.326: process: 0.68167 | load: 0.00018 [I 2022-10-08 17:07:34] ...trap/engines/engine.py.321: eval: epoch 12 | batch 90/510 [I 2022-10-08 17:07:34] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:38 [I 2022-10-08 17:07:34] ...trap/engines/engine.py.326: process: 0.56033 | load: 0.00019 [I 2022-10-08 17:07:39] ...trap/engines/engine.py.321: eval: epoch 12 | batch 100/510 [I 2022-10-08 17:07:39] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:59 [I 2022-10-08 17:07:39] ...trap/engines/engine.py.326: process: 0.30981 | load: 0.00018 [I 2022-10-08 17:07:42] ...trap/engines/engine.py.321: eval: epoch 12 | batch 110/510 [I 2022-10-08 17:07:42] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:11 [I 2022-10-08 17:07:42] ...trap/engines/engine.py.326: process: 0.31643 | load: 0.00019 [I 2022-10-08 17:07:45] ...trap/engines/engine.py.321: eval: epoch 12 | batch 120/510 [I 2022-10-08 17:07:45] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-08 17:07:45] ...trap/engines/engine.py.326: process: 0.32032 | load: 0.00019 [I 2022-10-08 17:07:48] ...trap/engines/engine.py.321: eval: epoch 12 | batch 130/510 [I 2022-10-08 17:07:48] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-08 17:07:48] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00020 [I 2022-10-08 17:07:51] ...trap/engines/engine.py.321: eval: epoch 12 | batch 140/510 [I 2022-10-08 17:07:51] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:57 [I 2022-10-08 17:07:51] ...trap/engines/engine.py.326: process: 0.31828 | load: 0.00020 [I 2022-10-08 17:07:55] ...trap/engines/engine.py.321: eval: epoch 12 | batch 150/510 [I 2022-10-08 17:07:55] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:53 [I 2022-10-08 17:07:55] ...trap/engines/engine.py.326: process: 0.31568 | load: 0.00019 [I 2022-10-08 17:07:58] ...trap/engines/engine.py.321: eval: epoch 12 | batch 160/510 [I 2022-10-08 17:07:58] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:50 [I 2022-10-08 17:07:58] ...trap/engines/engine.py.326: process: 0.31643 | load: 0.00019 [I 2022-10-08 17:08:01] ...trap/engines/engine.py.321: eval: epoch 12 | batch 170/510 [I 2022-10-08 17:08:01] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:47 [I 2022-10-08 17:08:01] ...trap/engines/engine.py.326: process: 0.31286 | load: 0.00019 [I 2022-10-08 17:08:04] ...trap/engines/engine.py.321: eval: epoch 12 | batch 180/510 [I 2022-10-08 17:08:04] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:44 [I 2022-10-08 17:08:04] ...trap/engines/engine.py.326: process: 0.31320 | load: 0.00019 [I 2022-10-08 17:08:07] ...trap/engines/engine.py.321: eval: epoch 12 | batch 190/510 [I 2022-10-08 17:08:07] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-08 17:08:07] ...trap/engines/engine.py.326: process: 0.32004 | load: 0.00020 [I 2022-10-08 17:08:11] ...trap/engines/engine.py.321: eval: epoch 12 | batch 200/510 [I 2022-10-08 17:08:11] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-08 17:08:11] ...trap/engines/engine.py.326: process: 0.31732 | load: 0.00020 [I 2022-10-08 17:08:14] ...trap/engines/engine.py.321: eval: epoch 12 | batch 210/510 [I 2022-10-08 17:08:14] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-08 17:08:14] ...trap/engines/engine.py.326: process: 0.32042 | load: 0.00020 [I 2022-10-08 17:08:17] ...trap/engines/engine.py.321: eval: epoch 12 | batch 220/510 [I 2022-10-08 17:08:17] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:31 [I 2022-10-08 17:08:17] ...trap/engines/engine.py.326: process: 0.31437 | load: 0.00020 [I 2022-10-08 17:08:20] ...trap/engines/engine.py.321: eval: epoch 12 | batch 230/510 [I 2022-10-08 17:08:20] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-08 17:08:20] ...trap/engines/engine.py.326: process: 0.31798 | load: 0.00020 [I 2022-10-08 17:08:23] ...trap/engines/engine.py.321: eval: epoch 12 | batch 240/510 [I 2022-10-08 17:08:23] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-08 17:08:23] ...trap/engines/engine.py.326: process: 0.31450 | load: 0.00019 [I 2022-10-08 17:08:26] ...trap/engines/engine.py.321: eval: epoch 12 | batch 250/510 [I 2022-10-08 17:08:26] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-08 17:08:26] ...trap/engines/engine.py.326: process: 0.31773 | load: 0.00019 [I 2022-10-08 17:08:30] ...trap/engines/engine.py.321: eval: epoch 12 | batch 260/510 [I 2022-10-08 17:08:30] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:19 [I 2022-10-08 17:08:30] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00020 [I 2022-10-08 17:08:33] ...trap/engines/engine.py.321: eval: epoch 12 | batch 270/510 [I 2022-10-08 17:08:33] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-08 17:08:33] ...trap/engines/engine.py.326: process: 0.32096 | load: 0.00020 [I 2022-10-08 17:08:36] ...trap/engines/engine.py.321: eval: epoch 12 | batch 280/510 [I 2022-10-08 17:08:36] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-08 17:08:36] ...trap/engines/engine.py.326: process: 0.32783 | load: 0.00020 [I 2022-10-08 17:08:39] ...trap/engines/engine.py.321: eval: epoch 12 | batch 290/510 [I 2022-10-08 17:08:39] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-08 17:08:39] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00019 [I 2022-10-08 17:08:42] ...trap/engines/engine.py.321: eval: epoch 12 | batch 300/510 [I 2022-10-08 17:08:42] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:07 [I 2022-10-08 17:08:42] ...trap/engines/engine.py.326: process: 0.32411 | load: 0.00020 [I 2022-10-08 17:08:46] ...trap/engines/engine.py.321: eval: epoch 12 | batch 310/510 [I 2022-10-08 17:08:46] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-08 17:08:46] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00021 [I 2022-10-08 17:08:49] ...trap/engines/engine.py.321: eval: epoch 12 | batch 320/510 [I 2022-10-08 17:08:49] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-08 17:08:49] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00019 [I 2022-10-08 17:08:52] ...trap/engines/engine.py.321: eval: epoch 12 | batch 330/510 [I 2022-10-08 17:08:52] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-08 17:08:52] ...trap/engines/engine.py.326: process: 0.32142 | load: 0.00020 [I 2022-10-08 17:08:55] ...trap/engines/engine.py.321: eval: epoch 12 | batch 340/510 [I 2022-10-08 17:08:55] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-08 17:08:55] ...trap/engines/engine.py.326: process: 0.32410 | load: 0.00021 [I 2022-10-08 17:08:58] ...trap/engines/engine.py.321: eval: epoch 12 | batch 350/510 [I 2022-10-08 17:08:58] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:50 [I 2022-10-08 17:08:58] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00020 [I 2022-10-08 17:09:01] ...trap/engines/engine.py.321: eval: epoch 12 | batch 360/510 [I 2022-10-08 17:09:01] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-08 17:09:01] ...trap/engines/engine.py.326: process: 0.31555 | load: 0.00020 [I 2022-10-08 17:09:05] ...trap/engines/engine.py.321: eval: epoch 12 | batch 370/510 [I 2022-10-08 17:09:05] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-08 17:09:05] ...trap/engines/engine.py.326: process: 0.31469 | load: 0.00019 [I 2022-10-08 17:09:08] ...trap/engines/engine.py.321: eval: epoch 12 | batch 380/510 [I 2022-10-08 17:09:08] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-08 17:09:08] ...trap/engines/engine.py.326: process: 0.31488 | load: 0.00020 [I 2022-10-08 17:09:11] ...trap/engines/engine.py.321: eval: epoch 12 | batch 390/510 [I 2022-10-08 17:09:11] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-08 17:09:11] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00020 [I 2022-10-08 17:09:14] ...trap/engines/engine.py.321: eval: epoch 12 | batch 400/510 [I 2022-10-08 17:09:14] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-08 17:09:14] ...trap/engines/engine.py.326: process: 0.32609 | load: 0.00020 [I 2022-10-08 17:09:17] ...trap/engines/engine.py.321: eval: epoch 12 | batch 410/510 [I 2022-10-08 17:09:17] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-08 17:09:17] ...trap/engines/engine.py.326: process: 0.31509 | load: 0.00020 [I 2022-10-08 17:09:21] ...trap/engines/engine.py.321: eval: epoch 12 | batch 420/510 [I 2022-10-08 17:09:21] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:29 [I 2022-10-08 17:09:21] ...trap/engines/engine.py.326: process: 0.32984 | load: 0.00020 [I 2022-10-08 17:09:24] ...trap/engines/engine.py.321: eval: epoch 12 | batch 430/510 [I 2022-10-08 17:09:24] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-08 17:09:24] ...trap/engines/engine.py.326: process: 0.31855 | load: 0.00020 [I 2022-10-08 17:09:27] ...trap/engines/engine.py.321: eval: epoch 12 | batch 440/510 [I 2022-10-08 17:09:27] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-08 17:09:27] ...trap/engines/engine.py.326: process: 0.32020 | load: 0.00020 [I 2022-10-08 17:09:30] ...trap/engines/engine.py.321: eval: epoch 12 | batch 450/510 [I 2022-10-08 17:09:30] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-08 17:09:30] ...trap/engines/engine.py.326: process: 0.31773 | load: 0.00020 [I 2022-10-08 17:09:33] ...trap/engines/engine.py.321: eval: epoch 12 | batch 460/510 [I 2022-10-08 17:09:33] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-08 17:09:33] ...trap/engines/engine.py.326: process: 0.31969 | load: 0.00020 [I 2022-10-08 17:09:37] ...trap/engines/engine.py.321: eval: epoch 12 | batch 470/510 [I 2022-10-08 17:09:37] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-08 17:09:37] ...trap/engines/engine.py.326: process: 0.31812 | load: 0.00020 [I 2022-10-08 17:09:40] ...trap/engines/engine.py.321: eval: epoch 12 | batch 480/510 [I 2022-10-08 17:09:40] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-08 17:09:40] ...trap/engines/engine.py.326: process: 0.32197 | load: 0.00020 [I 2022-10-08 17:09:43] ...trap/engines/engine.py.321: eval: epoch 12 | batch 490/510 [I 2022-10-08 17:09:43] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-08 17:09:43] ...trap/engines/engine.py.326: process: 0.31932 | load: 0.00020 [I 2022-10-08 17:09:46] ...trap/engines/engine.py.321: eval: epoch 12 | batch 500/510 [I 2022-10-08 17:09:46] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-08 17:09:46] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00020 [I 2022-10-08 17:09:49] ...trap/engines/engine.py.321: eval: epoch 12 | batch 510/510 [I 2022-10-08 17:09:49] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-08 17:09:49] ...trap/engines/engine.py.326: process: 0.31326 | load: 0.00016 [S 2022-10-08 17:09:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 12 [I 2022-10-08 17:09:50] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.593900 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.849400 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.901800 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.590100 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.847600 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.902700 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012778 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012555 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010553 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017219 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012706 [S 2022-10-08 17:10:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008013 [I 2022-10-08 17:10:22] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 17:10:22] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 17:10:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 17:10:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 17:10:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 17:10:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 17:10:26] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 17:10:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 17:10:30] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 17:10:30] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 17:10:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 17:10:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 17:10:37] ...trap/engines/engine.py.163: Training model on trainset for epoch 13 [I 2022-10-08 17:10:37] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 17:10:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 0/2304 [I 2022-10-08 17:10:40] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:52:05 [I 2022-10-08 17:10:40] ...trap/engines/engine.py.229: process: 2.91892 | load: 2.06683 [I 2022-10-08 17:10:40] ...trap/engines/engine.py.230: loss: 0.21108 [I 2022-10-08 17:10:40] ...optimizers/trijoint.py.151: total_norm: 2.688796 [I 2022-10-08 17:10:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 10/2304 [I 2022-10-08 17:10:48] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:34 [I 2022-10-08 17:10:48] ...trap/engines/engine.py.229: process: 0.77737 | load: 0.00575 [I 2022-10-08 17:10:48] ...trap/engines/engine.py.230: loss: 0.19736 [I 2022-10-08 17:10:48] ...optimizers/trijoint.py.151: total_norm: 2.890240 [I 2022-10-08 17:10:56] ...trap/engines/engine.py.224: train: epoch 13 | batch 20/2304 [I 2022-10-08 17:10:56] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:34 [I 2022-10-08 17:10:56] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00579 [I 2022-10-08 17:10:56] ...trap/engines/engine.py.230: loss: 0.17340 [I 2022-10-08 17:10:56] ...optimizers/trijoint.py.151: total_norm: 2.457761 [I 2022-10-08 17:11:04] ...trap/engines/engine.py.224: train: epoch 13 | batch 30/2304 [I 2022-10-08 17:11:04] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:44 [I 2022-10-08 17:11:04] ...trap/engines/engine.py.229: process: 0.77854 | load: 0.00396 [I 2022-10-08 17:11:04] ...trap/engines/engine.py.230: loss: 0.17108 [I 2022-10-08 17:11:04] ...optimizers/trijoint.py.151: total_norm: 2.106395 [I 2022-10-08 17:11:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 40/2304 [I 2022-10-08 17:11:12] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:27 [I 2022-10-08 17:11:12] ...trap/engines/engine.py.229: process: 0.78233 | load: 0.00340 [I 2022-10-08 17:11:12] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-08 17:11:12] ...optimizers/trijoint.py.151: total_norm: 1.816908 [I 2022-10-08 17:11:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 50/2304 [I 2022-10-08 17:11:19] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:08 [I 2022-10-08 17:11:19] ...trap/engines/engine.py.229: process: 0.76329 | load: 0.00402 [I 2022-10-08 17:11:19] ...trap/engines/engine.py.230: loss: 0.16656 [I 2022-10-08 17:11:19] ...optimizers/trijoint.py.151: total_norm: 2.247670 [I 2022-10-08 17:11:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 60/2304 [I 2022-10-08 17:11:27] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:06 [I 2022-10-08 17:11:27] ...trap/engines/engine.py.229: process: 0.77828 | load: 0.00352 [I 2022-10-08 17:11:27] ...trap/engines/engine.py.230: loss: 0.23666 [I 2022-10-08 17:11:27] ...optimizers/trijoint.py.151: total_norm: 2.416561 [I 2022-10-08 17:11:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 70/2304 [I 2022-10-08 17:11:35] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:05 [I 2022-10-08 17:11:35] ...trap/engines/engine.py.229: process: 0.78205 | load: 0.00370 [I 2022-10-08 17:11:35] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-08 17:11:35] ...optimizers/trijoint.py.151: total_norm: 2.072743 [I 2022-10-08 17:11:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 80/2304 [I 2022-10-08 17:11:43] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:52 [I 2022-10-08 17:11:43] ...trap/engines/engine.py.229: process: 0.77109 | load: 0.00329 [I 2022-10-08 17:11:43] ...trap/engines/engine.py.230: loss: 0.22416 [I 2022-10-08 17:11:43] ...optimizers/trijoint.py.151: total_norm: 3.056008 [I 2022-10-08 17:11:51] ...trap/engines/engine.py.224: train: epoch 13 | batch 90/2304 [I 2022-10-08 17:11:51] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:39 [I 2022-10-08 17:11:51] ...trap/engines/engine.py.229: process: 0.77041 | load: 0.00342 [I 2022-10-08 17:11:51] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-08 17:11:51] ...optimizers/trijoint.py.151: total_norm: 2.889875 [I 2022-10-08 17:11:59] ...trap/engines/engine.py.224: train: epoch 13 | batch 100/2304 [I 2022-10-08 17:11:59] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:40 [I 2022-10-08 17:11:59] ...trap/engines/engine.py.229: process: 0.78004 | load: 0.00322 [I 2022-10-08 17:11:59] ...trap/engines/engine.py.230: loss: 0.22215 [I 2022-10-08 17:11:59] ...optimizers/trijoint.py.151: total_norm: 3.642737 [I 2022-10-08 17:12:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 110/2304 [I 2022-10-08 17:12:06] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:35 [I 2022-10-08 17:12:06] ...trap/engines/engine.py.229: process: 0.79011 | load: 0.00348 [I 2022-10-08 17:12:06] ...trap/engines/engine.py.230: loss: 0.19499 [I 2022-10-08 17:12:06] ...optimizers/trijoint.py.151: total_norm: 2.754105 [I 2022-10-08 17:12:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 120/2304 [I 2022-10-08 17:12:14] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:27 [I 2022-10-08 17:12:14] ...trap/engines/engine.py.229: process: 0.77799 | load: 0.00350 [I 2022-10-08 17:12:14] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-08 17:12:14] ...optimizers/trijoint.py.151: total_norm: 2.542973 [I 2022-10-08 17:12:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 130/2304 [I 2022-10-08 17:12:22] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:13 [I 2022-10-08 17:12:22] ...trap/engines/engine.py.229: process: 0.77010 | load: 0.00336 [I 2022-10-08 17:12:22] ...trap/engines/engine.py.230: loss: 0.19488 [I 2022-10-08 17:12:22] ...optimizers/trijoint.py.151: total_norm: 2.733097 [I 2022-10-08 17:12:30] ...trap/engines/engine.py.224: train: epoch 13 | batch 140/2304 [I 2022-10-08 17:12:30] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:27:54 [I 2022-10-08 17:12:30] ...trap/engines/engine.py.229: process: 0.77886 | load: 0.00336 [I 2022-10-08 17:12:30] ...trap/engines/engine.py.230: loss: 0.21141 [I 2022-10-08 17:12:30] ...optimizers/trijoint.py.151: total_norm: 3.128814 [I 2022-10-08 17:12:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 150/2304 [I 2022-10-08 17:12:38] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:58 [I 2022-10-08 17:12:38] ...trap/engines/engine.py.229: process: 0.77607 | load: 0.00298 [I 2022-10-08 17:12:38] ...trap/engines/engine.py.230: loss: 0.24309 [I 2022-10-08 17:12:38] ...optimizers/trijoint.py.151: total_norm: 2.510319 [I 2022-10-08 17:12:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 160/2304 [I 2022-10-08 17:12:45] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:49 [I 2022-10-08 17:12:45] ...trap/engines/engine.py.229: process: 0.78683 | load: 0.00310 [I 2022-10-08 17:12:45] ...trap/engines/engine.py.230: loss: 0.16101 [I 2022-10-08 17:12:45] ...optimizers/trijoint.py.151: total_norm: 1.996464 [I 2022-10-08 17:12:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 170/2304 [I 2022-10-08 17:12:53] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:48 [I 2022-10-08 17:12:53] ...trap/engines/engine.py.229: process: 0.77304 | load: 0.00326 [I 2022-10-08 17:12:53] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-08 17:12:53] ...optimizers/trijoint.py.151: total_norm: 2.120979 [I 2022-10-08 17:13:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 180/2304 [I 2022-10-08 17:13:01] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:49 [I 2022-10-08 17:13:01] ...trap/engines/engine.py.229: process: 0.78810 | load: 0.00319 [I 2022-10-08 17:13:01] ...trap/engines/engine.py.230: loss: 0.18095 [I 2022-10-08 17:13:01] ...optimizers/trijoint.py.151: total_norm: 2.134181 [I 2022-10-08 17:13:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 190/2304 [I 2022-10-08 17:13:09] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:30 [I 2022-10-08 17:13:09] ...trap/engines/engine.py.229: process: 0.77703 | load: 0.00313 [I 2022-10-08 17:13:09] ...trap/engines/engine.py.230: loss: 0.17192 [I 2022-10-08 17:13:09] ...optimizers/trijoint.py.151: total_norm: 3.162004 [I 2022-10-08 17:13:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 200/2304 [I 2022-10-08 17:13:17] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:20 [I 2022-10-08 17:13:17] ...trap/engines/engine.py.229: process: 0.77848 | load: 0.00312 [I 2022-10-08 17:13:17] ...trap/engines/engine.py.230: loss: 0.14123 [I 2022-10-08 17:13:17] ...optimizers/trijoint.py.151: total_norm: 2.311196 [I 2022-10-08 17:13:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 210/2304 [I 2022-10-08 17:13:24] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:13 [I 2022-10-08 17:13:25] ...trap/engines/engine.py.229: process: 0.77842 | load: 0.00321 [I 2022-10-08 17:13:25] ...trap/engines/engine.py.230: loss: 0.21426 [I 2022-10-08 17:13:25] ...optimizers/trijoint.py.151: total_norm: 2.907742 [I 2022-10-08 17:13:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 220/2304 [I 2022-10-08 17:13:32] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:08 [I 2022-10-08 17:13:32] ...trap/engines/engine.py.229: process: 0.78479 | load: 0.00331 [I 2022-10-08 17:13:32] ...trap/engines/engine.py.230: loss: 0.18030 [I 2022-10-08 17:13:32] ...optimizers/trijoint.py.151: total_norm: 1.775371 [I 2022-10-08 17:13:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 230/2304 [I 2022-10-08 17:13:40] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:53 [I 2022-10-08 17:13:40] ...trap/engines/engine.py.229: process: 0.78346 | load: 0.00319 [I 2022-10-08 17:13:40] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-08 17:13:40] ...optimizers/trijoint.py.151: total_norm: 1.979863 [I 2022-10-08 17:13:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 240/2304 [I 2022-10-08 17:13:48] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:55 [I 2022-10-08 17:13:48] ...trap/engines/engine.py.229: process: 0.78734 | load: 0.00303 [I 2022-10-08 17:13:48] ...trap/engines/engine.py.230: loss: 0.20105 [I 2022-10-08 17:13:48] ...optimizers/trijoint.py.151: total_norm: 2.937896 [I 2022-10-08 17:13:56] ...trap/engines/engine.py.224: train: epoch 13 | batch 250/2304 [I 2022-10-08 17:13:56] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:46 [I 2022-10-08 17:13:56] ...trap/engines/engine.py.229: process: 0.78535 | load: 0.00327 [I 2022-10-08 17:13:56] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-08 17:13:56] ...optimizers/trijoint.py.151: total_norm: 2.155133 [I 2022-10-08 17:14:04] ...trap/engines/engine.py.224: train: epoch 13 | batch 260/2304 [I 2022-10-08 17:14:04] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:38 [I 2022-10-08 17:14:04] ...trap/engines/engine.py.229: process: 0.78605 | load: 0.00307 [I 2022-10-08 17:14:04] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-08 17:14:04] ...optimizers/trijoint.py.151: total_norm: 1.954240 [I 2022-10-08 17:14:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 270/2304 [I 2022-10-08 17:14:11] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:25 [I 2022-10-08 17:14:11] ...trap/engines/engine.py.229: process: 0.77436 | load: 0.00303 [I 2022-10-08 17:14:11] ...trap/engines/engine.py.230: loss: 0.18177 [I 2022-10-08 17:14:11] ...optimizers/trijoint.py.151: total_norm: 2.063632 [I 2022-10-08 17:14:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 280/2304 [I 2022-10-08 17:14:19] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:18 [I 2022-10-08 17:14:19] ...trap/engines/engine.py.229: process: 0.78521 | load: 0.00305 [I 2022-10-08 17:14:19] ...trap/engines/engine.py.230: loss: 0.20513 [I 2022-10-08 17:14:19] ...optimizers/trijoint.py.151: total_norm: 2.087271 [I 2022-10-08 17:14:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 290/2304 [I 2022-10-08 17:14:27] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:14 [I 2022-10-08 17:14:27] ...trap/engines/engine.py.229: process: 0.77288 | load: 0.00317 [I 2022-10-08 17:14:27] ...trap/engines/engine.py.230: loss: 0.22110 [I 2022-10-08 17:14:27] ...optimizers/trijoint.py.151: total_norm: 2.828951 [I 2022-10-08 17:14:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 300/2304 [I 2022-10-08 17:14:35] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:58 [I 2022-10-08 17:14:35] ...trap/engines/engine.py.229: process: 0.77624 | load: 0.00323 [I 2022-10-08 17:14:35] ...trap/engines/engine.py.230: loss: 0.24092 [I 2022-10-08 17:14:35] ...optimizers/trijoint.py.151: total_norm: 3.294656 [I 2022-10-08 17:14:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 310/2304 [I 2022-10-08 17:14:43] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:50 [I 2022-10-08 17:14:43] ...trap/engines/engine.py.229: process: 0.77554 | load: 0.00304 [I 2022-10-08 17:14:43] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-08 17:14:43] ...optimizers/trijoint.py.151: total_norm: 2.367804 [I 2022-10-08 17:14:51] ...trap/engines/engine.py.224: train: epoch 13 | batch 320/2304 [I 2022-10-08 17:14:51] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:42 [I 2022-10-08 17:14:51] ...trap/engines/engine.py.229: process: 0.76975 | load: 0.00302 [I 2022-10-08 17:14:51] ...trap/engines/engine.py.230: loss: 0.20535 [I 2022-10-08 17:14:51] ...optimizers/trijoint.py.151: total_norm: 2.759670 [I 2022-10-08 17:14:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 330/2304 [I 2022-10-08 17:14:58] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:47 [I 2022-10-08 17:14:58] ...trap/engines/engine.py.229: process: 0.78787 | load: 0.00306 [I 2022-10-08 17:14:58] ...trap/engines/engine.py.230: loss: 0.11936 [I 2022-10-08 17:14:58] ...optimizers/trijoint.py.151: total_norm: 2.419215 [I 2022-10-08 17:15:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 340/2304 [I 2022-10-08 17:15:06] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:30 [I 2022-10-08 17:15:06] ...trap/engines/engine.py.229: process: 0.77638 | load: 0.00301 [I 2022-10-08 17:15:06] ...trap/engines/engine.py.230: loss: 0.21892 [I 2022-10-08 17:15:06] ...optimizers/trijoint.py.151: total_norm: 2.809813 [I 2022-10-08 17:15:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 350/2304 [I 2022-10-08 17:15:14] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:21 [I 2022-10-08 17:15:14] ...trap/engines/engine.py.229: process: 0.77846 | load: 0.00316 [I 2022-10-08 17:15:14] ...trap/engines/engine.py.230: loss: 0.16011 [I 2022-10-08 17:15:14] ...optimizers/trijoint.py.151: total_norm: 2.740233 [I 2022-10-08 17:15:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 360/2304 [I 2022-10-08 17:15:22] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:13 [I 2022-10-08 17:15:22] ...trap/engines/engine.py.229: process: 0.77168 | load: 0.00307 [I 2022-10-08 17:15:22] ...trap/engines/engine.py.230: loss: 0.25253 [I 2022-10-08 17:15:22] ...optimizers/trijoint.py.151: total_norm: 3.453257 [I 2022-10-08 17:15:30] ...trap/engines/engine.py.224: train: epoch 13 | batch 370/2304 [I 2022-10-08 17:15:30] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:25:09 [I 2022-10-08 17:15:30] ...trap/engines/engine.py.229: process: 0.79064 | load: 0.00302 [I 2022-10-08 17:15:30] ...trap/engines/engine.py.230: loss: 0.18323 [I 2022-10-08 17:15:30] ...optimizers/trijoint.py.151: total_norm: 2.688527 [I 2022-10-08 17:15:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 380/2304 [I 2022-10-08 17:15:37] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:24:55 [I 2022-10-08 17:15:37] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00309 [I 2022-10-08 17:15:37] ...trap/engines/engine.py.230: loss: 0.23266 [I 2022-10-08 17:15:37] ...optimizers/trijoint.py.151: total_norm: 2.960258 [I 2022-10-08 17:15:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 390/2304 [I 2022-10-08 17:15:45] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:52 [I 2022-10-08 17:15:45] ...trap/engines/engine.py.229: process: 0.78090 | load: 0.00313 [I 2022-10-08 17:15:45] ...trap/engines/engine.py.230: loss: 0.22457 [I 2022-10-08 17:15:45] ...optimizers/trijoint.py.151: total_norm: 3.379140 [I 2022-10-08 17:15:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 400/2304 [I 2022-10-08 17:15:53] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:44 [I 2022-10-08 17:15:53] ...trap/engines/engine.py.229: process: 0.77535 | load: 0.00305 [I 2022-10-08 17:15:53] ...trap/engines/engine.py.230: loss: 0.18296 [I 2022-10-08 17:15:53] ...optimizers/trijoint.py.151: total_norm: 2.340912 [I 2022-10-08 17:16:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 410/2304 [I 2022-10-08 17:16:01] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:47 [I 2022-10-08 17:16:01] ...trap/engines/engine.py.229: process: 0.77929 | load: 0.00300 [I 2022-10-08 17:16:01] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-08 17:16:01] ...optimizers/trijoint.py.151: total_norm: 2.009238 [I 2022-10-08 17:16:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 420/2304 [I 2022-10-08 17:16:09] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:53 [I 2022-10-08 17:16:09] ...trap/engines/engine.py.229: process: 0.80616 | load: 0.00316 [I 2022-10-08 17:16:09] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-08 17:16:09] ...optimizers/trijoint.py.151: total_norm: 2.226076 [I 2022-10-08 17:16:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 430/2304 [I 2022-10-08 17:16:17] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:26 [I 2022-10-08 17:16:17] ...trap/engines/engine.py.229: process: 0.77633 | load: 0.00299 [I 2022-10-08 17:16:17] ...trap/engines/engine.py.230: loss: 0.17663 [I 2022-10-08 17:16:17] ...optimizers/trijoint.py.151: total_norm: 2.540659 [I 2022-10-08 17:16:25] ...trap/engines/engine.py.224: train: epoch 13 | batch 440/2304 [I 2022-10-08 17:16:25] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:24:24 [I 2022-10-08 17:16:25] ...trap/engines/engine.py.229: process: 0.80139 | load: 0.00313 [I 2022-10-08 17:16:25] ...trap/engines/engine.py.230: loss: 0.14941 [I 2022-10-08 17:16:25] ...optimizers/trijoint.py.151: total_norm: 2.193394 [I 2022-10-08 17:16:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 450/2304 [I 2022-10-08 17:16:32] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:23:55 [I 2022-10-08 17:16:32] ...trap/engines/engine.py.229: process: 0.76699 | load: 0.00303 [I 2022-10-08 17:16:32] ...trap/engines/engine.py.230: loss: 0.24861 [I 2022-10-08 17:16:32] ...optimizers/trijoint.py.151: total_norm: 3.316882 [I 2022-10-08 17:16:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 460/2304 [I 2022-10-08 17:16:40] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:47 [I 2022-10-08 17:16:40] ...trap/engines/engine.py.229: process: 0.77526 | load: 0.00428 [I 2022-10-08 17:16:40] ...trap/engines/engine.py.230: loss: 0.18909 [I 2022-10-08 17:16:40] ...optimizers/trijoint.py.151: total_norm: 2.603910 [I 2022-10-08 17:16:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 470/2304 [I 2022-10-08 17:16:48] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:49 [I 2022-10-08 17:16:48] ...trap/engines/engine.py.229: process: 0.79161 | load: 0.00312 [I 2022-10-08 17:16:48] ...trap/engines/engine.py.230: loss: 0.23981 [I 2022-10-08 17:16:48] ...optimizers/trijoint.py.151: total_norm: 2.710059 [I 2022-10-08 17:16:56] ...trap/engines/engine.py.224: train: epoch 13 | batch 480/2304 [I 2022-10-08 17:16:56] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:36 [I 2022-10-08 17:16:56] ...trap/engines/engine.py.229: process: 0.76800 | load: 0.00302 [I 2022-10-08 17:16:56] ...trap/engines/engine.py.230: loss: 0.22954 [I 2022-10-08 17:16:56] ...optimizers/trijoint.py.151: total_norm: 2.792307 [I 2022-10-08 17:17:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 490/2304 [I 2022-10-08 17:17:03] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:44 [I 2022-10-08 17:17:03] ...trap/engines/engine.py.229: process: 0.78429 | load: 0.00302 [I 2022-10-08 17:17:03] ...trap/engines/engine.py.230: loss: 0.24523 [I 2022-10-08 17:17:03] ...optimizers/trijoint.py.151: total_norm: 2.705616 [I 2022-10-08 17:17:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 500/2304 [I 2022-10-08 17:17:11] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:34 [I 2022-10-08 17:17:11] ...trap/engines/engine.py.229: process: 0.78576 | load: 0.00301 [I 2022-10-08 17:17:11] ...trap/engines/engine.py.230: loss: 0.18212 [I 2022-10-08 17:17:11] ...optimizers/trijoint.py.151: total_norm: 2.582322 [I 2022-10-08 17:17:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 510/2304 [I 2022-10-08 17:17:19] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:21 [I 2022-10-08 17:17:19] ...trap/engines/engine.py.229: process: 0.77439 | load: 0.00305 [I 2022-10-08 17:17:19] ...trap/engines/engine.py.230: loss: 0.12993 [I 2022-10-08 17:17:19] ...optimizers/trijoint.py.151: total_norm: 1.993735 [I 2022-10-08 17:17:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 520/2304 [I 2022-10-08 17:17:27] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:17 [I 2022-10-08 17:17:27] ...trap/engines/engine.py.229: process: 0.78806 | load: 0.00312 [I 2022-10-08 17:17:27] ...trap/engines/engine.py.230: loss: 0.17229 [I 2022-10-08 17:17:27] ...optimizers/trijoint.py.151: total_norm: 2.493725 [I 2022-10-08 17:17:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 530/2304 [I 2022-10-08 17:17:35] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:23:03 [I 2022-10-08 17:17:35] ...trap/engines/engine.py.229: process: 0.78554 | load: 0.00302 [I 2022-10-08 17:17:35] ...trap/engines/engine.py.230: loss: 0.20354 [I 2022-10-08 17:17:35] ...optimizers/trijoint.py.151: total_norm: 2.753081 [I 2022-10-08 17:17:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 540/2304 [I 2022-10-08 17:17:43] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:51 [I 2022-10-08 17:17:43] ...trap/engines/engine.py.229: process: 0.77891 | load: 0.00313 [I 2022-10-08 17:17:43] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-08 17:17:43] ...optimizers/trijoint.py.151: total_norm: 2.248720 [I 2022-10-08 17:17:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 550/2304 [I 2022-10-08 17:17:50] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:43 [I 2022-10-08 17:17:50] ...trap/engines/engine.py.229: process: 0.76579 | load: 0.00301 [I 2022-10-08 17:17:50] ...trap/engines/engine.py.230: loss: 0.15614 [I 2022-10-08 17:17:50] ...optimizers/trijoint.py.151: total_norm: 2.453065 [I 2022-10-08 17:17:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 560/2304 [I 2022-10-08 17:17:58] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:37 [I 2022-10-08 17:17:58] ...trap/engines/engine.py.229: process: 0.77387 | load: 0.00308 [I 2022-10-08 17:17:58] ...trap/engines/engine.py.230: loss: 0.22912 [I 2022-10-08 17:17:58] ...optimizers/trijoint.py.151: total_norm: 2.704431 [I 2022-10-08 17:18:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 570/2304 [I 2022-10-08 17:18:06] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:28 [I 2022-10-08 17:18:06] ...trap/engines/engine.py.229: process: 0.78369 | load: 0.00306 [I 2022-10-08 17:18:06] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-08 17:18:06] ...optimizers/trijoint.py.151: total_norm: 2.460011 [I 2022-10-08 17:18:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 580/2304 [I 2022-10-08 17:18:14] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:20 [I 2022-10-08 17:18:14] ...trap/engines/engine.py.229: process: 0.77464 | load: 0.00309 [I 2022-10-08 17:18:14] ...trap/engines/engine.py.230: loss: 0.18980 [I 2022-10-08 17:18:14] ...optimizers/trijoint.py.151: total_norm: 2.287276 [I 2022-10-08 17:18:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 590/2304 [I 2022-10-08 17:18:22] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:15 [I 2022-10-08 17:18:22] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00304 [I 2022-10-08 17:18:22] ...trap/engines/engine.py.230: loss: 0.18216 [I 2022-10-08 17:18:22] ...optimizers/trijoint.py.151: total_norm: 2.468180 [I 2022-10-08 17:18:30] ...trap/engines/engine.py.224: train: epoch 13 | batch 600/2304 [I 2022-10-08 17:18:30] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:08 [I 2022-10-08 17:18:30] ...trap/engines/engine.py.229: process: 0.77061 | load: 0.00300 [I 2022-10-08 17:18:30] ...trap/engines/engine.py.230: loss: 0.18274 [I 2022-10-08 17:18:30] ...optimizers/trijoint.py.151: total_norm: 2.973369 [I 2022-10-08 17:18:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 610/2304 [I 2022-10-08 17:18:38] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:22:07 [I 2022-10-08 17:18:38] ...trap/engines/engine.py.229: process: 0.78019 | load: 0.00304 [I 2022-10-08 17:18:38] ...trap/engines/engine.py.230: loss: 0.15219 [I 2022-10-08 17:18:38] ...optimizers/trijoint.py.151: total_norm: 2.233177 [I 2022-10-08 17:18:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 620/2304 [I 2022-10-08 17:18:45] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:52 [I 2022-10-08 17:18:45] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00300 [I 2022-10-08 17:18:45] ...trap/engines/engine.py.230: loss: 0.20361 [I 2022-10-08 17:18:45] ...optimizers/trijoint.py.151: total_norm: 4.265401 [I 2022-10-08 17:18:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 630/2304 [I 2022-10-08 17:18:53] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:48 [I 2022-10-08 17:18:53] ...trap/engines/engine.py.229: process: 0.78981 | load: 0.00305 [I 2022-10-08 17:18:53] ...trap/engines/engine.py.230: loss: 0.19861 [I 2022-10-08 17:18:53] ...optimizers/trijoint.py.151: total_norm: 2.513186 [I 2022-10-08 17:19:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 640/2304 [I 2022-10-08 17:19:01] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:28 [I 2022-10-08 17:19:01] ...trap/engines/engine.py.229: process: 0.76633 | load: 0.00298 [I 2022-10-08 17:19:01] ...trap/engines/engine.py.230: loss: 0.15080 [I 2022-10-08 17:19:01] ...optimizers/trijoint.py.151: total_norm: 2.355509 [I 2022-10-08 17:19:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 650/2304 [I 2022-10-08 17:19:09] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:35 [I 2022-10-08 17:19:09] ...trap/engines/engine.py.229: process: 0.78676 | load: 0.00306 [I 2022-10-08 17:19:09] ...trap/engines/engine.py.230: loss: 0.15724 [I 2022-10-08 17:19:09] ...optimizers/trijoint.py.151: total_norm: 2.208826 [I 2022-10-08 17:19:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 660/2304 [I 2022-10-08 17:19:17] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:21:17 [I 2022-10-08 17:19:17] ...trap/engines/engine.py.229: process: 0.77139 | load: 0.00303 [I 2022-10-08 17:19:17] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-08 17:19:17] ...optimizers/trijoint.py.151: total_norm: 1.507138 [I 2022-10-08 17:19:25] ...trap/engines/engine.py.224: train: epoch 13 | batch 670/2304 [I 2022-10-08 17:19:25] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:21:11 [I 2022-10-08 17:19:25] ...trap/engines/engine.py.229: process: 0.77586 | load: 0.00307 [I 2022-10-08 17:19:25] ...trap/engines/engine.py.230: loss: 0.17375 [I 2022-10-08 17:19:25] ...optimizers/trijoint.py.151: total_norm: 2.309858 [I 2022-10-08 17:19:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 680/2304 [I 2022-10-08 17:19:32] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:07 [I 2022-10-08 17:19:32] ...trap/engines/engine.py.229: process: 0.78518 | load: 0.00303 [I 2022-10-08 17:19:32] ...trap/engines/engine.py.230: loss: 0.19785 [I 2022-10-08 17:19:32] ...optimizers/trijoint.py.151: total_norm: 2.137356 [I 2022-10-08 17:19:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 690/2304 [I 2022-10-08 17:19:40] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:51 [I 2022-10-08 17:19:40] ...trap/engines/engine.py.229: process: 0.78150 | load: 0.00312 [I 2022-10-08 17:19:40] ...trap/engines/engine.py.230: loss: 0.16564 [I 2022-10-08 17:19:40] ...optimizers/trijoint.py.151: total_norm: 2.740018 [I 2022-10-08 17:19:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 700/2304 [I 2022-10-08 17:19:48] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:45 [I 2022-10-08 17:19:48] ...trap/engines/engine.py.229: process: 0.77302 | load: 0.00301 [I 2022-10-08 17:19:48] ...trap/engines/engine.py.230: loss: 0.15684 [I 2022-10-08 17:19:48] ...optimizers/trijoint.py.151: total_norm: 2.699377 [I 2022-10-08 17:19:56] ...trap/engines/engine.py.224: train: epoch 13 | batch 710/2304 [I 2022-10-08 17:19:56] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:20:38 [I 2022-10-08 17:19:56] ...trap/engines/engine.py.229: process: 0.78466 | load: 0.00303 [I 2022-10-08 17:19:56] ...trap/engines/engine.py.230: loss: 0.20547 [I 2022-10-08 17:19:56] ...optimizers/trijoint.py.151: total_norm: 2.313920 [I 2022-10-08 17:20:04] ...trap/engines/engine.py.224: train: epoch 13 | batch 720/2304 [I 2022-10-08 17:20:04] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:20:30 [I 2022-10-08 17:20:04] ...trap/engines/engine.py.229: process: 0.78376 | load: 0.00301 [I 2022-10-08 17:20:04] ...trap/engines/engine.py.230: loss: 0.20481 [I 2022-10-08 17:20:04] ...optimizers/trijoint.py.151: total_norm: 3.155867 [I 2022-10-08 17:20:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 730/2304 [I 2022-10-08 17:20:11] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:25 [I 2022-10-08 17:20:11] ...trap/engines/engine.py.229: process: 0.77623 | load: 0.00307 [I 2022-10-08 17:20:11] ...trap/engines/engine.py.230: loss: 0.20644 [I 2022-10-08 17:20:11] ...optimizers/trijoint.py.151: total_norm: 3.089015 [I 2022-10-08 17:20:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 740/2304 [I 2022-10-08 17:20:19] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:15 [I 2022-10-08 17:20:19] ...trap/engines/engine.py.229: process: 0.77071 | load: 0.00298 [I 2022-10-08 17:20:19] ...trap/engines/engine.py.230: loss: 0.23727 [I 2022-10-08 17:20:19] ...optimizers/trijoint.py.151: total_norm: 2.404669 [I 2022-10-08 17:20:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 750/2304 [I 2022-10-08 17:20:27] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:10 [I 2022-10-08 17:20:27] ...trap/engines/engine.py.229: process: 0.76957 | load: 0.00316 [I 2022-10-08 17:20:27] ...trap/engines/engine.py.230: loss: 0.15250 [I 2022-10-08 17:20:27] ...optimizers/trijoint.py.151: total_norm: 2.468403 [I 2022-10-08 17:20:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 760/2304 [I 2022-10-08 17:20:35] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:20:06 [I 2022-10-08 17:20:35] ...trap/engines/engine.py.229: process: 0.79035 | load: 0.00316 [I 2022-10-08 17:20:35] ...trap/engines/engine.py.230: loss: 0.17458 [I 2022-10-08 17:20:35] ...optimizers/trijoint.py.151: total_norm: 2.803320 [I 2022-10-08 17:20:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 770/2304 [I 2022-10-08 17:20:43] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:20:03 [I 2022-10-08 17:20:43] ...trap/engines/engine.py.229: process: 0.79004 | load: 0.00312 [I 2022-10-08 17:20:43] ...trap/engines/engine.py.230: loss: 0.24280 [I 2022-10-08 17:20:43] ...optimizers/trijoint.py.151: total_norm: 2.708529 [I 2022-10-08 17:20:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 780/2304 [I 2022-10-08 17:20:50] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:43 [I 2022-10-08 17:20:50] ...trap/engines/engine.py.229: process: 0.76980 | load: 0.00318 [I 2022-10-08 17:20:50] ...trap/engines/engine.py.230: loss: 0.18333 [I 2022-10-08 17:20:50] ...optimizers/trijoint.py.151: total_norm: 2.726033 [I 2022-10-08 17:20:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 790/2304 [I 2022-10-08 17:20:58] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:45 [I 2022-10-08 17:20:58] ...trap/engines/engine.py.229: process: 0.78734 | load: 0.00312 [I 2022-10-08 17:20:58] ...trap/engines/engine.py.230: loss: 0.21062 [I 2022-10-08 17:20:58] ...optimizers/trijoint.py.151: total_norm: 3.264633 [I 2022-10-08 17:21:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 800/2304 [I 2022-10-08 17:21:06] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:38 [I 2022-10-08 17:21:06] ...trap/engines/engine.py.229: process: 0.78992 | load: 0.00312 [I 2022-10-08 17:21:06] ...trap/engines/engine.py.230: loss: 0.18064 [I 2022-10-08 17:21:06] ...optimizers/trijoint.py.151: total_norm: 2.911553 [I 2022-10-08 17:21:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 810/2304 [I 2022-10-08 17:21:14] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:26 [I 2022-10-08 17:21:14] ...trap/engines/engine.py.229: process: 0.77872 | load: 0.00314 [I 2022-10-08 17:21:14] ...trap/engines/engine.py.230: loss: 0.29713 [I 2022-10-08 17:21:14] ...optimizers/trijoint.py.151: total_norm: 3.748566 [I 2022-10-08 17:21:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 820/2304 [I 2022-10-08 17:21:22] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:19:15 [I 2022-10-08 17:21:22] ...trap/engines/engine.py.229: process: 0.77061 | load: 0.00315 [I 2022-10-08 17:21:22] ...trap/engines/engine.py.230: loss: 0.17558 [I 2022-10-08 17:21:22] ...optimizers/trijoint.py.151: total_norm: 3.029691 [I 2022-10-08 17:21:30] ...trap/engines/engine.py.224: train: epoch 13 | batch 830/2304 [I 2022-10-08 17:21:30] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:19:09 [I 2022-10-08 17:21:30] ...trap/engines/engine.py.229: process: 0.78414 | load: 0.00311 [I 2022-10-08 17:21:30] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-08 17:21:30] ...optimizers/trijoint.py.151: total_norm: 2.527056 [I 2022-10-08 17:21:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 840/2304 [I 2022-10-08 17:21:38] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:19:03 [I 2022-10-08 17:21:38] ...trap/engines/engine.py.229: process: 0.78578 | load: 0.00310 [I 2022-10-08 17:21:38] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-08 17:21:38] ...optimizers/trijoint.py.151: total_norm: 2.913190 [I 2022-10-08 17:21:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 850/2304 [I 2022-10-08 17:21:45] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:18:50 [I 2022-10-08 17:21:45] ...trap/engines/engine.py.229: process: 0.76915 | load: 0.00311 [I 2022-10-08 17:21:45] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-08 17:21:45] ...optimizers/trijoint.py.151: total_norm: 1.442244 [I 2022-10-08 17:21:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 860/2304 [I 2022-10-08 17:21:53] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:41 [I 2022-10-08 17:21:53] ...trap/engines/engine.py.229: process: 0.78243 | load: 0.00311 [I 2022-10-08 17:21:53] ...trap/engines/engine.py.230: loss: 0.19815 [I 2022-10-08 17:21:53] ...optimizers/trijoint.py.151: total_norm: 3.420136 [I 2022-10-08 17:22:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 870/2304 [I 2022-10-08 17:22:01] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:42 [I 2022-10-08 17:22:01] ...trap/engines/engine.py.229: process: 0.78209 | load: 0.00310 [I 2022-10-08 17:22:01] ...trap/engines/engine.py.230: loss: 0.18482 [I 2022-10-08 17:22:01] ...optimizers/trijoint.py.151: total_norm: 4.581913 [I 2022-10-08 17:22:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 880/2304 [I 2022-10-08 17:22:09] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:18:29 [I 2022-10-08 17:22:09] ...trap/engines/engine.py.229: process: 0.76578 | load: 0.00312 [I 2022-10-08 17:22:09] ...trap/engines/engine.py.230: loss: 0.20001 [I 2022-10-08 17:22:09] ...optimizers/trijoint.py.151: total_norm: 2.418936 [I 2022-10-08 17:22:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 890/2304 [I 2022-10-08 17:22:17] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:18:22 [I 2022-10-08 17:22:17] ...trap/engines/engine.py.229: process: 0.77543 | load: 0.00312 [I 2022-10-08 17:22:17] ...trap/engines/engine.py.230: loss: 0.21891 [I 2022-10-08 17:22:17] ...optimizers/trijoint.py.151: total_norm: 2.728459 [I 2022-10-08 17:22:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 900/2304 [I 2022-10-08 17:22:24] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:08 [I 2022-10-08 17:22:24] ...trap/engines/engine.py.229: process: 0.77510 | load: 0.00299 [I 2022-10-08 17:22:24] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-08 17:22:24] ...optimizers/trijoint.py.151: total_norm: 1.640926 [I 2022-10-08 17:22:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 910/2304 [I 2022-10-08 17:22:32] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:18:04 [I 2022-10-08 17:22:32] ...trap/engines/engine.py.229: process: 0.76337 | load: 0.00307 [I 2022-10-08 17:22:32] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-08 17:22:32] ...optimizers/trijoint.py.151: total_norm: 2.245319 [I 2022-10-08 17:22:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 920/2304 [I 2022-10-08 17:22:40] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:17:59 [I 2022-10-08 17:22:40] ...trap/engines/engine.py.229: process: 0.76322 | load: 0.00304 [I 2022-10-08 17:22:40] ...trap/engines/engine.py.230: loss: 0.17408 [I 2022-10-08 17:22:40] ...optimizers/trijoint.py.151: total_norm: 2.776922 [I 2022-10-08 17:22:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 930/2304 [I 2022-10-08 17:22:48] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:17:52 [I 2022-10-08 17:22:48] ...trap/engines/engine.py.229: process: 0.78507 | load: 0.00302 [I 2022-10-08 17:22:48] ...trap/engines/engine.py.230: loss: 0.21677 [I 2022-10-08 17:22:48] ...optimizers/trijoint.py.151: total_norm: 3.486514 [I 2022-10-08 17:23:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 940/2304 [I 2022-10-08 17:23:14] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:59:39 [I 2022-10-08 17:23:14] ...trap/engines/engine.py.229: process: 0.77644 | load: 0.00298 [I 2022-10-08 17:23:14] ...trap/engines/engine.py.230: loss: 0.22183 [I 2022-10-08 17:23:14] ...optimizers/trijoint.py.151: total_norm: 3.533111 [I 2022-10-08 17:23:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 950/2304 [I 2022-10-08 17:23:22] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:21:57 [I 2022-10-08 17:23:22] ...trap/engines/engine.py.229: process: 0.76445 | load: 0.00313 [I 2022-10-08 17:23:22] ...trap/engines/engine.py.230: loss: 0.13120 [I 2022-10-08 17:23:22] ...optimizers/trijoint.py.151: total_norm: 2.441351 [I 2022-10-08 17:23:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 960/2304 [I 2022-10-08 17:23:29] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:17:52 [I 2022-10-08 17:23:29] ...trap/engines/engine.py.229: process: 0.78707 | load: 0.00308 [I 2022-10-08 17:23:29] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-08 17:23:29] ...optimizers/trijoint.py.151: total_norm: 1.940494 [I 2022-10-08 17:23:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 970/2304 [I 2022-10-08 17:23:37] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:17:20 [I 2022-10-08 17:23:37] ...trap/engines/engine.py.229: process: 0.78127 | load: 0.00308 [I 2022-10-08 17:23:37] ...trap/engines/engine.py.230: loss: 0.19673 [I 2022-10-08 17:23:37] ...optimizers/trijoint.py.151: total_norm: 2.844930 [I 2022-10-08 17:23:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 980/2304 [I 2022-10-08 17:23:45] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:17:13 [I 2022-10-08 17:23:45] ...trap/engines/engine.py.229: process: 0.78227 | load: 0.00304 [I 2022-10-08 17:23:45] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-08 17:23:45] ...optimizers/trijoint.py.151: total_norm: 2.650036 [I 2022-10-08 17:23:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 990/2304 [I 2022-10-08 17:23:53] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:17:04 [I 2022-10-08 17:23:53] ...trap/engines/engine.py.229: process: 0.77085 | load: 0.00301 [I 2022-10-08 17:23:53] ...trap/engines/engine.py.230: loss: 0.26828 [I 2022-10-08 17:23:53] ...optimizers/trijoint.py.151: total_norm: 3.404327 [I 2022-10-08 17:24:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 1000/2304 [I 2022-10-08 17:24:01] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:16:59 [I 2022-10-08 17:24:01] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00304 [I 2022-10-08 17:24:01] ...trap/engines/engine.py.230: loss: 0.16546 [I 2022-10-08 17:24:01] ...optimizers/trijoint.py.151: total_norm: 1.970176 [I 2022-10-08 17:24:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1010/2304 [I 2022-10-08 17:24:08] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:16:53 [I 2022-10-08 17:24:08] ...trap/engines/engine.py.229: process: 0.77783 | load: 0.00301 [I 2022-10-08 17:24:08] ...trap/engines/engine.py.230: loss: 0.24140 [I 2022-10-08 17:24:08] ...optimizers/trijoint.py.151: total_norm: 2.409989 [I 2022-10-08 17:24:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 1020/2304 [I 2022-10-08 17:24:16] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:16:45 [I 2022-10-08 17:24:16] ...trap/engines/engine.py.229: process: 0.77610 | load: 0.00304 [I 2022-10-08 17:24:16] ...trap/engines/engine.py.230: loss: 0.20546 [I 2022-10-08 17:24:16] ...optimizers/trijoint.py.151: total_norm: 2.813804 [I 2022-10-08 17:24:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 1030/2304 [I 2022-10-08 17:24:24] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:16:32 [I 2022-10-08 17:24:24] ...trap/engines/engine.py.229: process: 0.77359 | load: 0.00302 [I 2022-10-08 17:24:24] ...trap/engines/engine.py.230: loss: 0.26808 [I 2022-10-08 17:24:24] ...optimizers/trijoint.py.151: total_norm: 3.734465 [I 2022-10-08 17:24:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1040/2304 [I 2022-10-08 17:24:32] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:16:27 [I 2022-10-08 17:24:32] ...trap/engines/engine.py.229: process: 0.78845 | load: 0.00305 [I 2022-10-08 17:24:32] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-08 17:24:32] ...optimizers/trijoint.py.151: total_norm: 3.104280 [I 2022-10-08 17:24:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 1050/2304 [I 2022-10-08 17:24:40] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:16:11 [I 2022-10-08 17:24:40] ...trap/engines/engine.py.229: process: 0.76579 | load: 0.00306 [I 2022-10-08 17:24:40] ...trap/engines/engine.py.230: loss: 0.19727 [I 2022-10-08 17:24:40] ...optimizers/trijoint.py.151: total_norm: 2.302252 [I 2022-10-08 17:24:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1060/2304 [I 2022-10-08 17:24:48] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:16:14 [I 2022-10-08 17:24:48] ...trap/engines/engine.py.229: process: 0.78420 | load: 0.00310 [I 2022-10-08 17:24:48] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-08 17:24:48] ...optimizers/trijoint.py.151: total_norm: 2.360291 [I 2022-10-08 17:24:55] ...trap/engines/engine.py.224: train: epoch 13 | batch 1070/2304 [I 2022-10-08 17:24:55] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:16:03 [I 2022-10-08 17:24:55] ...trap/engines/engine.py.229: process: 0.77450 | load: 0.00306 [I 2022-10-08 17:24:55] ...trap/engines/engine.py.230: loss: 0.23072 [I 2022-10-08 17:24:55] ...optimizers/trijoint.py.151: total_norm: 3.204096 [I 2022-10-08 17:25:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1080/2304 [I 2022-10-08 17:25:03] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:15:47 [I 2022-10-08 17:25:03] ...trap/engines/engine.py.229: process: 0.76814 | load: 0.00305 [I 2022-10-08 17:25:03] ...trap/engines/engine.py.230: loss: 0.21749 [I 2022-10-08 17:25:03] ...optimizers/trijoint.py.151: total_norm: 2.651402 [I 2022-10-08 17:25:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 1090/2304 [I 2022-10-08 17:25:11] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:15:40 [I 2022-10-08 17:25:11] ...trap/engines/engine.py.229: process: 0.76852 | load: 0.00303 [I 2022-10-08 17:25:11] ...trap/engines/engine.py.230: loss: 0.18639 [I 2022-10-08 17:25:11] ...optimizers/trijoint.py.151: total_norm: 3.094535 [I 2022-10-08 17:25:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 1100/2304 [I 2022-10-08 17:25:19] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:15:40 [I 2022-10-08 17:25:19] ...trap/engines/engine.py.229: process: 0.78573 | load: 0.00305 [I 2022-10-08 17:25:19] ...trap/engines/engine.py.230: loss: 0.18699 [I 2022-10-08 17:25:19] ...optimizers/trijoint.py.151: total_norm: 2.584395 [I 2022-10-08 17:25:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 1110/2304 [I 2022-10-08 17:25:27] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:15:36 [I 2022-10-08 17:25:27] ...trap/engines/engine.py.229: process: 0.79587 | load: 0.00303 [I 2022-10-08 17:25:27] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-08 17:25:27] ...optimizers/trijoint.py.151: total_norm: 2.319493 [I 2022-10-08 17:25:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 1120/2304 [I 2022-10-08 17:25:35] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:15:30 [I 2022-10-08 17:25:35] ...trap/engines/engine.py.229: process: 0.78893 | load: 0.00306 [I 2022-10-08 17:25:35] ...trap/engines/engine.py.230: loss: 0.15594 [I 2022-10-08 17:25:35] ...optimizers/trijoint.py.151: total_norm: 2.484864 [I 2022-10-08 17:25:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 1130/2304 [I 2022-10-08 17:25:42] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:15:19 [I 2022-10-08 17:25:42] ...trap/engines/engine.py.229: process: 0.79175 | load: 0.00305 [I 2022-10-08 17:25:42] ...trap/engines/engine.py.230: loss: 0.15369 [I 2022-10-08 17:25:42] ...optimizers/trijoint.py.151: total_norm: 2.458490 [I 2022-10-08 17:25:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 1140/2304 [I 2022-10-08 17:25:50] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:15:07 [I 2022-10-08 17:25:50] ...trap/engines/engine.py.229: process: 0.77818 | load: 0.00305 [I 2022-10-08 17:25:50] ...trap/engines/engine.py.230: loss: 0.24732 [I 2022-10-08 17:25:50] ...optimizers/trijoint.py.151: total_norm: 3.623684 [I 2022-10-08 17:25:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1150/2304 [I 2022-10-08 17:25:58] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:15:00 [I 2022-10-08 17:25:58] ...trap/engines/engine.py.229: process: 0.77623 | load: 0.00307 [I 2022-10-08 17:25:58] ...trap/engines/engine.py.230: loss: 0.18553 [I 2022-10-08 17:25:58] ...optimizers/trijoint.py.151: total_norm: 2.182447 [I 2022-10-08 17:26:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 1160/2304 [I 2022-10-08 17:26:06] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:14:49 [I 2022-10-08 17:26:06] ...trap/engines/engine.py.229: process: 0.77943 | load: 0.00305 [I 2022-10-08 17:26:06] ...trap/engines/engine.py.230: loss: 0.20099 [I 2022-10-08 17:26:06] ...optimizers/trijoint.py.151: total_norm: 2.496485 [I 2022-10-08 17:26:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 1170/2304 [I 2022-10-08 17:26:14] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:14:40 [I 2022-10-08 17:26:14] ...trap/engines/engine.py.229: process: 0.76918 | load: 0.00304 [I 2022-10-08 17:26:14] ...trap/engines/engine.py.230: loss: 0.20199 [I 2022-10-08 17:26:14] ...optimizers/trijoint.py.151: total_norm: 2.670511 [I 2022-10-08 17:26:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 1180/2304 [I 2022-10-08 17:26:22] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:14:37 [I 2022-10-08 17:26:22] ...trap/engines/engine.py.229: process: 0.78051 | load: 0.00305 [I 2022-10-08 17:26:22] ...trap/engines/engine.py.230: loss: 0.18732 [I 2022-10-08 17:26:22] ...optimizers/trijoint.py.151: total_norm: 2.733913 [I 2022-10-08 17:26:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 1190/2304 [I 2022-10-08 17:26:29] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:14:26 [I 2022-10-08 17:26:29] ...trap/engines/engine.py.229: process: 0.76781 | load: 0.00307 [I 2022-10-08 17:26:29] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-08 17:26:29] ...optimizers/trijoint.py.151: total_norm: 2.387827 [I 2022-10-08 17:26:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 1200/2304 [I 2022-10-08 17:26:37] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:14:18 [I 2022-10-08 17:26:37] ...trap/engines/engine.py.229: process: 0.76681 | load: 0.00303 [I 2022-10-08 17:26:37] ...trap/engines/engine.py.230: loss: 0.16099 [I 2022-10-08 17:26:37] ...optimizers/trijoint.py.151: total_norm: 3.048092 [I 2022-10-08 17:26:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 1210/2304 [I 2022-10-08 17:26:45] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:14:18 [I 2022-10-08 17:26:45] ...trap/engines/engine.py.229: process: 0.78157 | load: 0.00303 [I 2022-10-08 17:26:45] ...trap/engines/engine.py.230: loss: 0.20956 [I 2022-10-08 17:26:45] ...optimizers/trijoint.py.151: total_norm: 2.687203 [I 2022-10-08 17:26:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1220/2304 [I 2022-10-08 17:26:53] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:14:03 [I 2022-10-08 17:26:53] ...trap/engines/engine.py.229: process: 0.77477 | load: 0.00301 [I 2022-10-08 17:26:53] ...trap/engines/engine.py.230: loss: 0.18484 [I 2022-10-08 17:26:53] ...optimizers/trijoint.py.151: total_norm: 2.048237 [I 2022-10-08 17:27:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 1230/2304 [I 2022-10-08 17:27:01] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:13:55 [I 2022-10-08 17:27:01] ...trap/engines/engine.py.229: process: 0.77683 | load: 0.00300 [I 2022-10-08 17:27:01] ...trap/engines/engine.py.230: loss: 0.19696 [I 2022-10-08 17:27:01] ...optimizers/trijoint.py.151: total_norm: 2.329360 [I 2022-10-08 17:27:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1240/2304 [I 2022-10-08 17:27:08] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:13:51 [I 2022-10-08 17:27:08] ...trap/engines/engine.py.229: process: 0.78583 | load: 0.00302 [I 2022-10-08 17:27:08] ...trap/engines/engine.py.230: loss: 0.23674 [I 2022-10-08 17:27:08] ...optimizers/trijoint.py.151: total_norm: 3.051682 [I 2022-10-08 17:27:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 1250/2304 [I 2022-10-08 17:27:16] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:13:44 [I 2022-10-08 17:27:16] ...trap/engines/engine.py.229: process: 0.78678 | load: 0.00302 [I 2022-10-08 17:27:16] ...trap/engines/engine.py.230: loss: 0.19369 [I 2022-10-08 17:27:16] ...optimizers/trijoint.py.151: total_norm: 2.360787 [I 2022-10-08 17:27:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 1260/2304 [I 2022-10-08 17:27:24] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:13:38 [I 2022-10-08 17:27:24] ...trap/engines/engine.py.229: process: 0.78620 | load: 0.00304 [I 2022-10-08 17:27:24] ...trap/engines/engine.py.230: loss: 0.17310 [I 2022-10-08 17:27:24] ...optimizers/trijoint.py.151: total_norm: 2.823642 [I 2022-10-08 17:27:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1270/2304 [I 2022-10-08 17:27:32] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:13:27 [I 2022-10-08 17:27:32] ...trap/engines/engine.py.229: process: 0.78338 | load: 0.00300 [I 2022-10-08 17:27:32] ...trap/engines/engine.py.230: loss: 0.21950 [I 2022-10-08 17:27:32] ...optimizers/trijoint.py.151: total_norm: 2.153638 [I 2022-10-08 17:27:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 1280/2304 [I 2022-10-08 17:27:40] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:13:19 [I 2022-10-08 17:27:40] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00306 [I 2022-10-08 17:27:40] ...trap/engines/engine.py.230: loss: 0.23810 [I 2022-10-08 17:27:40] ...optimizers/trijoint.py.151: total_norm: 3.388827 [I 2022-10-08 17:27:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1290/2304 [I 2022-10-08 17:27:48] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:13:14 [I 2022-10-08 17:27:48] ...trap/engines/engine.py.229: process: 0.77808 | load: 0.00305 [I 2022-10-08 17:27:48] ...trap/engines/engine.py.230: loss: 0.21705 [I 2022-10-08 17:27:48] ...optimizers/trijoint.py.151: total_norm: 2.957770 [I 2022-10-08 17:27:56] ...trap/engines/engine.py.224: train: epoch 13 | batch 1300/2304 [I 2022-10-08 17:27:56] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:13:10 [I 2022-10-08 17:27:56] ...trap/engines/engine.py.229: process: 0.78947 | load: 0.00303 [I 2022-10-08 17:27:56] ...trap/engines/engine.py.230: loss: 0.19518 [I 2022-10-08 17:27:56] ...optimizers/trijoint.py.151: total_norm: 2.022813 [I 2022-10-08 17:28:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1310/2304 [I 2022-10-08 17:28:03] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:12:56 [I 2022-10-08 17:28:03] ...trap/engines/engine.py.229: process: 0.79219 | load: 0.00304 [I 2022-10-08 17:28:03] ...trap/engines/engine.py.230: loss: 0.12798 [I 2022-10-08 17:28:03] ...optimizers/trijoint.py.151: total_norm: 1.745596 [I 2022-10-08 17:28:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 1320/2304 [I 2022-10-08 17:28:11] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:12:47 [I 2022-10-08 17:28:11] ...trap/engines/engine.py.229: process: 0.77906 | load: 0.00302 [I 2022-10-08 17:28:11] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-08 17:28:11] ...optimizers/trijoint.py.151: total_norm: 2.189125 [I 2022-10-08 17:28:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 1330/2304 [I 2022-10-08 17:28:19] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:12:38 [I 2022-10-08 17:28:19] ...trap/engines/engine.py.229: process: 0.78406 | load: 0.00302 [I 2022-10-08 17:28:19] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-08 17:28:19] ...optimizers/trijoint.py.151: total_norm: 3.037908 [I 2022-10-08 17:28:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 1340/2304 [I 2022-10-08 17:28:27] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:12:29 [I 2022-10-08 17:28:27] ...trap/engines/engine.py.229: process: 0.77646 | load: 0.00296 [I 2022-10-08 17:28:27] ...trap/engines/engine.py.230: loss: 0.23405 [I 2022-10-08 17:28:27] ...optimizers/trijoint.py.151: total_norm: 2.613073 [I 2022-10-08 17:28:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 1350/2304 [I 2022-10-08 17:28:35] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:12:24 [I 2022-10-08 17:28:35] ...trap/engines/engine.py.229: process: 0.76994 | load: 0.00306 [I 2022-10-08 17:28:35] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-08 17:28:35] ...optimizers/trijoint.py.151: total_norm: 1.887951 [I 2022-10-08 17:28:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 1360/2304 [I 2022-10-08 17:28:42] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:12:15 [I 2022-10-08 17:28:42] ...trap/engines/engine.py.229: process: 0.77985 | load: 0.00304 [I 2022-10-08 17:28:42] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-08 17:28:42] ...optimizers/trijoint.py.151: total_norm: 2.587085 [I 2022-10-08 17:28:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 1370/2304 [I 2022-10-08 17:28:50] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:12:07 [I 2022-10-08 17:28:50] ...trap/engines/engine.py.229: process: 0.77585 | load: 0.00307 [I 2022-10-08 17:28:50] ...trap/engines/engine.py.230: loss: 0.19778 [I 2022-10-08 17:28:50] ...optimizers/trijoint.py.151: total_norm: 2.322954 [I 2022-10-08 17:28:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1380/2304 [I 2022-10-08 17:28:58] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:12:05 [I 2022-10-08 17:28:58] ...trap/engines/engine.py.229: process: 0.79024 | load: 0.00300 [I 2022-10-08 17:28:58] ...trap/engines/engine.py.230: loss: 0.31607 [I 2022-10-08 17:28:58] ...optimizers/trijoint.py.151: total_norm: 3.453625 [I 2022-10-08 17:29:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 1390/2304 [I 2022-10-08 17:29:06] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:11:54 [I 2022-10-08 17:29:06] ...trap/engines/engine.py.229: process: 0.78417 | load: 0.00301 [I 2022-10-08 17:29:06] ...trap/engines/engine.py.230: loss: 0.18486 [I 2022-10-08 17:29:06] ...optimizers/trijoint.py.151: total_norm: 2.859663 [I 2022-10-08 17:29:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 1400/2304 [I 2022-10-08 17:29:14] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:11:46 [I 2022-10-08 17:29:14] ...trap/engines/engine.py.229: process: 0.78676 | load: 0.00302 [I 2022-10-08 17:29:14] ...trap/engines/engine.py.230: loss: 0.18477 [I 2022-10-08 17:29:14] ...optimizers/trijoint.py.151: total_norm: 1.818002 [I 2022-10-08 17:29:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 1410/2304 [I 2022-10-08 17:29:22] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:11:37 [I 2022-10-08 17:29:22] ...trap/engines/engine.py.229: process: 0.78635 | load: 0.00306 [I 2022-10-08 17:29:22] ...trap/engines/engine.py.230: loss: 0.18617 [I 2022-10-08 17:29:22] ...optimizers/trijoint.py.151: total_norm: 2.885355 [I 2022-10-08 17:29:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 1420/2304 [I 2022-10-08 17:29:29] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:11:26 [I 2022-10-08 17:29:29] ...trap/engines/engine.py.229: process: 0.77361 | load: 0.00295 [I 2022-10-08 17:29:29] ...trap/engines/engine.py.230: loss: 0.18579 [I 2022-10-08 17:29:29] ...optimizers/trijoint.py.151: total_norm: 2.155578 [I 2022-10-08 17:29:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 1430/2304 [I 2022-10-08 17:29:37] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:11:25 [I 2022-10-08 17:29:37] ...trap/engines/engine.py.229: process: 0.78936 | load: 0.00301 [I 2022-10-08 17:29:37] ...trap/engines/engine.py.230: loss: 0.22585 [I 2022-10-08 17:29:37] ...optimizers/trijoint.py.151: total_norm: 3.072220 [I 2022-10-08 17:29:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 1440/2304 [I 2022-10-08 17:29:45] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:11:20 [I 2022-10-08 17:29:45] ...trap/engines/engine.py.229: process: 0.78852 | load: 0.00301 [I 2022-10-08 17:29:45] ...trap/engines/engine.py.230: loss: 0.20760 [I 2022-10-08 17:29:45] ...optimizers/trijoint.py.151: total_norm: 2.755450 [I 2022-10-08 17:29:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1450/2304 [I 2022-10-08 17:29:53] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:11:11 [I 2022-10-08 17:29:53] ...trap/engines/engine.py.229: process: 0.78241 | load: 0.00303 [I 2022-10-08 17:29:53] ...trap/engines/engine.py.230: loss: 0.20479 [I 2022-10-08 17:29:53] ...optimizers/trijoint.py.151: total_norm: 3.321495 [I 2022-10-08 17:30:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 1460/2304 [I 2022-10-08 17:30:01] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:10:56 [I 2022-10-08 17:30:01] ...trap/engines/engine.py.229: process: 0.77548 | load: 0.00305 [I 2022-10-08 17:30:01] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-08 17:30:01] ...optimizers/trijoint.py.151: total_norm: 2.072527 [I 2022-10-08 17:30:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 1470/2304 [I 2022-10-08 17:30:09] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:10:51 [I 2022-10-08 17:30:09] ...trap/engines/engine.py.229: process: 0.78341 | load: 0.00307 [I 2022-10-08 17:30:09] ...trap/engines/engine.py.230: loss: 0.19849 [I 2022-10-08 17:30:09] ...optimizers/trijoint.py.151: total_norm: 2.290989 [I 2022-10-08 17:30:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 1480/2304 [I 2022-10-08 17:30:16] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:10:43 [I 2022-10-08 17:30:16] ...trap/engines/engine.py.229: process: 0.78574 | load: 0.00305 [I 2022-10-08 17:30:16] ...trap/engines/engine.py.230: loss: 0.19147 [I 2022-10-08 17:30:16] ...optimizers/trijoint.py.151: total_norm: 2.907598 [I 2022-10-08 17:30:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 1490/2304 [I 2022-10-08 17:30:24] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:10:34 [I 2022-10-08 17:30:24] ...trap/engines/engine.py.229: process: 0.77417 | load: 0.00302 [I 2022-10-08 17:30:24] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-08 17:30:24] ...optimizers/trijoint.py.151: total_norm: 3.322926 [I 2022-10-08 17:30:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1500/2304 [I 2022-10-08 17:30:32] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:10:29 [I 2022-10-08 17:30:32] ...trap/engines/engine.py.229: process: 0.78149 | load: 0.00304 [I 2022-10-08 17:30:32] ...trap/engines/engine.py.230: loss: 0.22430 [I 2022-10-08 17:30:32] ...optimizers/trijoint.py.151: total_norm: 3.465353 [I 2022-10-08 17:30:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 1510/2304 [I 2022-10-08 17:30:40] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:10:19 [I 2022-10-08 17:30:40] ...trap/engines/engine.py.229: process: 0.78008 | load: 0.00304 [I 2022-10-08 17:30:40] ...trap/engines/engine.py.230: loss: 0.20065 [I 2022-10-08 17:30:40] ...optimizers/trijoint.py.151: total_norm: 2.334374 [I 2022-10-08 17:30:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1520/2304 [I 2022-10-08 17:30:48] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:10:14 [I 2022-10-08 17:30:48] ...trap/engines/engine.py.229: process: 0.77938 | load: 0.00300 [I 2022-10-08 17:30:48] ...trap/engines/engine.py.230: loss: 0.20093 [I 2022-10-08 17:30:48] ...optimizers/trijoint.py.151: total_norm: 3.047244 [I 2022-10-08 17:30:55] ...trap/engines/engine.py.224: train: epoch 13 | batch 1530/2304 [I 2022-10-08 17:30:55] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:10:04 [I 2022-10-08 17:30:55] ...trap/engines/engine.py.229: process: 0.77982 | load: 0.00301 [I 2022-10-08 17:30:55] ...trap/engines/engine.py.230: loss: 0.20392 [I 2022-10-08 17:30:55] ...optimizers/trijoint.py.151: total_norm: 3.086687 [I 2022-10-08 17:31:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1540/2304 [I 2022-10-08 17:31:03] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:09:58 [I 2022-10-08 17:31:03] ...trap/engines/engine.py.229: process: 0.77894 | load: 0.00298 [I 2022-10-08 17:31:03] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-08 17:31:03] ...optimizers/trijoint.py.151: total_norm: 1.787316 [I 2022-10-08 17:31:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 1550/2304 [I 2022-10-08 17:31:11] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:09:49 [I 2022-10-08 17:31:11] ...trap/engines/engine.py.229: process: 0.78723 | load: 0.00305 [I 2022-10-08 17:31:11] ...trap/engines/engine.py.230: loss: 0.23012 [I 2022-10-08 17:31:11] ...optimizers/trijoint.py.151: total_norm: 3.156929 [I 2022-10-08 17:31:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 1560/2304 [I 2022-10-08 17:31:19] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:09:40 [I 2022-10-08 17:31:19] ...trap/engines/engine.py.229: process: 0.78387 | load: 0.00300 [I 2022-10-08 17:31:19] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-08 17:31:19] ...optimizers/trijoint.py.151: total_norm: 2.027087 [I 2022-10-08 17:31:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 1570/2304 [I 2022-10-08 17:31:27] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:09:28 [I 2022-10-08 17:31:27] ...trap/engines/engine.py.229: process: 0.76577 | load: 0.00300 [I 2022-10-08 17:31:27] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-08 17:31:27] ...optimizers/trijoint.py.151: total_norm: 3.176515 [I 2022-10-08 17:31:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 1580/2304 [I 2022-10-08 17:31:35] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:09:24 [I 2022-10-08 17:31:35] ...trap/engines/engine.py.229: process: 0.78395 | load: 0.00313 [I 2022-10-08 17:31:35] ...trap/engines/engine.py.230: loss: 0.22669 [I 2022-10-08 17:31:35] ...optimizers/trijoint.py.151: total_norm: 3.242872 [I 2022-10-08 17:31:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 1590/2304 [I 2022-10-08 17:31:42] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:09:15 [I 2022-10-08 17:31:42] ...trap/engines/engine.py.229: process: 0.78193 | load: 0.00311 [I 2022-10-08 17:31:42] ...trap/engines/engine.py.230: loss: 0.18051 [I 2022-10-08 17:31:42] ...optimizers/trijoint.py.151: total_norm: 4.292589 [I 2022-10-08 17:31:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 1600/2304 [I 2022-10-08 17:31:50] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:09:09 [I 2022-10-08 17:31:50] ...trap/engines/engine.py.229: process: 0.76701 | load: 0.00309 [I 2022-10-08 17:31:50] ...trap/engines/engine.py.230: loss: 0.29449 [I 2022-10-08 17:31:50] ...optimizers/trijoint.py.151: total_norm: 3.288226 [I 2022-10-08 17:31:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1610/2304 [I 2022-10-08 17:31:58] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:09:02 [I 2022-10-08 17:31:58] ...trap/engines/engine.py.229: process: 0.78753 | load: 0.00299 [I 2022-10-08 17:31:58] ...trap/engines/engine.py.230: loss: 0.26622 [I 2022-10-08 17:31:58] ...optimizers/trijoint.py.151: total_norm: 3.350641 [I 2022-10-08 17:32:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 1620/2304 [I 2022-10-08 17:32:06] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:08:53 [I 2022-10-08 17:32:06] ...trap/engines/engine.py.229: process: 0.78511 | load: 0.00301 [I 2022-10-08 17:32:06] ...trap/engines/engine.py.230: loss: 0.13182 [I 2022-10-08 17:32:06] ...optimizers/trijoint.py.151: total_norm: 1.935540 [I 2022-10-08 17:32:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 1630/2304 [I 2022-10-08 17:32:14] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:08:49 [I 2022-10-08 17:32:14] ...trap/engines/engine.py.229: process: 0.78322 | load: 0.00308 [I 2022-10-08 17:32:14] ...trap/engines/engine.py.230: loss: 0.23889 [I 2022-10-08 17:32:14] ...optimizers/trijoint.py.151: total_norm: 3.164099 [I 2022-10-08 17:32:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 1640/2304 [I 2022-10-08 17:32:22] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:08:36 [I 2022-10-08 17:32:22] ...trap/engines/engine.py.229: process: 0.75883 | load: 0.00301 [I 2022-10-08 17:32:22] ...trap/engines/engine.py.230: loss: 0.19873 [I 2022-10-08 17:32:22] ...optimizers/trijoint.py.151: total_norm: 2.485833 [I 2022-10-08 17:32:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 1650/2304 [I 2022-10-08 17:32:29] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:08:31 [I 2022-10-08 17:32:29] ...trap/engines/engine.py.229: process: 0.78202 | load: 0.00307 [I 2022-10-08 17:32:29] ...trap/engines/engine.py.230: loss: 0.15786 [I 2022-10-08 17:32:29] ...optimizers/trijoint.py.151: total_norm: 2.188756 [I 2022-10-08 17:32:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 1660/2304 [I 2022-10-08 17:32:37] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:08:24 [I 2022-10-08 17:32:37] ...trap/engines/engine.py.229: process: 0.78108 | load: 0.00303 [I 2022-10-08 17:32:37] ...trap/engines/engine.py.230: loss: 0.16955 [I 2022-10-08 17:32:37] ...optimizers/trijoint.py.151: total_norm: 2.802443 [I 2022-10-08 17:32:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 1670/2304 [I 2022-10-08 17:32:45] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:08:11 [I 2022-10-08 17:32:45] ...trap/engines/engine.py.229: process: 0.75315 | load: 0.00303 [I 2022-10-08 17:32:45] ...trap/engines/engine.py.230: loss: 0.13620 [I 2022-10-08 17:32:45] ...optimizers/trijoint.py.151: total_norm: 1.587498 [I 2022-10-08 17:32:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1680/2304 [I 2022-10-08 17:32:53] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:08:04 [I 2022-10-08 17:32:53] ...trap/engines/engine.py.229: process: 0.77758 | load: 0.00295 [I 2022-10-08 17:32:53] ...trap/engines/engine.py.230: loss: 0.18058 [I 2022-10-08 17:32:53] ...optimizers/trijoint.py.151: total_norm: 2.842750 [I 2022-10-08 17:33:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 1690/2304 [I 2022-10-08 17:33:01] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:07:58 [I 2022-10-08 17:33:01] ...trap/engines/engine.py.229: process: 0.78242 | load: 0.00302 [I 2022-10-08 17:33:01] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-08 17:33:01] ...optimizers/trijoint.py.151: total_norm: 2.083984 [I 2022-10-08 17:33:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1700/2304 [I 2022-10-08 17:33:08] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:07:50 [I 2022-10-08 17:33:08] ...trap/engines/engine.py.229: process: 0.77178 | load: 0.00303 [I 2022-10-08 17:33:08] ...trap/engines/engine.py.230: loss: 0.18768 [I 2022-10-08 17:33:08] ...optimizers/trijoint.py.151: total_norm: 2.433318 [I 2022-10-08 17:33:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 1710/2304 [I 2022-10-08 17:33:16] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:07:45 [I 2022-10-08 17:33:16] ...trap/engines/engine.py.229: process: 0.78888 | load: 0.00304 [I 2022-10-08 17:33:16] ...trap/engines/engine.py.230: loss: 0.20250 [I 2022-10-08 17:33:16] ...optimizers/trijoint.py.151: total_norm: 2.843616 [I 2022-10-08 17:33:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 1720/2304 [I 2022-10-08 17:33:24] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:07:35 [I 2022-10-08 17:33:24] ...trap/engines/engine.py.229: process: 0.77543 | load: 0.00311 [I 2022-10-08 17:33:24] ...trap/engines/engine.py.230: loss: 0.17148 [I 2022-10-08 17:33:24] ...optimizers/trijoint.py.151: total_norm: 2.570846 [I 2022-10-08 17:33:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1730/2304 [I 2022-10-08 17:33:32] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:07:29 [I 2022-10-08 17:33:32] ...trap/engines/engine.py.229: process: 0.78253 | load: 0.00303 [I 2022-10-08 17:33:32] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-08 17:33:32] ...optimizers/trijoint.py.151: total_norm: 2.471887 [I 2022-10-08 17:33:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 1740/2304 [I 2022-10-08 17:33:40] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:07:22 [I 2022-10-08 17:33:40] ...trap/engines/engine.py.229: process: 0.77840 | load: 0.00300 [I 2022-10-08 17:33:40] ...trap/engines/engine.py.230: loss: 0.22161 [I 2022-10-08 17:33:40] ...optimizers/trijoint.py.151: total_norm: 3.415583 [I 2022-10-08 17:33:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1750/2304 [I 2022-10-08 17:33:48] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:07:12 [I 2022-10-08 17:33:48] ...trap/engines/engine.py.229: process: 0.77643 | load: 0.00307 [I 2022-10-08 17:33:48] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-08 17:33:48] ...optimizers/trijoint.py.151: total_norm: 2.060961 [I 2022-10-08 17:33:55] ...trap/engines/engine.py.224: train: epoch 13 | batch 1760/2304 [I 2022-10-08 17:33:55] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:07:04 [I 2022-10-08 17:33:55] ...trap/engines/engine.py.229: process: 0.78048 | load: 0.00302 [I 2022-10-08 17:33:55] ...trap/engines/engine.py.230: loss: 0.21402 [I 2022-10-08 17:33:55] ...optimizers/trijoint.py.151: total_norm: 3.182859 [I 2022-10-08 17:34:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1770/2304 [I 2022-10-08 17:34:03] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:06:59 [I 2022-10-08 17:34:03] ...trap/engines/engine.py.229: process: 0.79357 | load: 0.00302 [I 2022-10-08 17:34:03] ...trap/engines/engine.py.230: loss: 0.20488 [I 2022-10-08 17:34:03] ...optimizers/trijoint.py.151: total_norm: 2.710862 [I 2022-10-08 17:34:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 1780/2304 [I 2022-10-08 17:34:11] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:06:47 [I 2022-10-08 17:34:11] ...trap/engines/engine.py.229: process: 0.77911 | load: 0.00304 [I 2022-10-08 17:34:11] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-08 17:34:11] ...optimizers/trijoint.py.151: total_norm: 2.358237 [I 2022-10-08 17:34:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 1790/2304 [I 2022-10-08 17:34:19] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:06:41 [I 2022-10-08 17:34:19] ...trap/engines/engine.py.229: process: 0.77697 | load: 0.00303 [I 2022-10-08 17:34:19] ...trap/engines/engine.py.230: loss: 0.17759 [I 2022-10-08 17:34:19] ...optimizers/trijoint.py.151: total_norm: 3.015435 [I 2022-10-08 17:34:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 1800/2304 [I 2022-10-08 17:34:27] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:06:34 [I 2022-10-08 17:34:27] ...trap/engines/engine.py.229: process: 0.78902 | load: 0.00300 [I 2022-10-08 17:34:27] ...trap/engines/engine.py.230: loss: 0.18111 [I 2022-10-08 17:34:27] ...optimizers/trijoint.py.151: total_norm: 3.375840 [I 2022-10-08 17:34:35] ...trap/engines/engine.py.224: train: epoch 13 | batch 1810/2304 [I 2022-10-08 17:34:35] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:06:28 [I 2022-10-08 17:34:35] ...trap/engines/engine.py.229: process: 0.79416 | load: 0.00304 [I 2022-10-08 17:34:35] ...trap/engines/engine.py.230: loss: 0.19214 [I 2022-10-08 17:34:35] ...optimizers/trijoint.py.151: total_norm: 2.906861 [I 2022-10-08 17:34:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 1820/2304 [I 2022-10-08 17:34:42] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:06:16 [I 2022-10-08 17:34:42] ...trap/engines/engine.py.229: process: 0.77472 | load: 0.00300 [I 2022-10-08 17:34:42] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-08 17:34:42] ...optimizers/trijoint.py.151: total_norm: 2.452906 [I 2022-10-08 17:34:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 1830/2304 [I 2022-10-08 17:34:50] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:06:08 [I 2022-10-08 17:34:50] ...trap/engines/engine.py.229: process: 0.78551 | load: 0.00302 [I 2022-10-08 17:34:50] ...trap/engines/engine.py.230: loss: 0.20753 [I 2022-10-08 17:34:50] ...optimizers/trijoint.py.151: total_norm: 3.286900 [I 2022-10-08 17:34:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1840/2304 [I 2022-10-08 17:34:58] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:06:01 [I 2022-10-08 17:34:58] ...trap/engines/engine.py.229: process: 0.77632 | load: 0.00301 [I 2022-10-08 17:34:58] ...trap/engines/engine.py.230: loss: 0.21973 [I 2022-10-08 17:34:58] ...optimizers/trijoint.py.151: total_norm: 3.639330 [I 2022-10-08 17:35:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 1850/2304 [I 2022-10-08 17:35:06] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:05:56 [I 2022-10-08 17:35:06] ...trap/engines/engine.py.229: process: 0.79152 | load: 0.00305 [I 2022-10-08 17:35:06] ...trap/engines/engine.py.230: loss: 0.15962 [I 2022-10-08 17:35:06] ...optimizers/trijoint.py.151: total_norm: 3.322950 [I 2022-10-08 17:35:14] ...trap/engines/engine.py.224: train: epoch 13 | batch 1860/2304 [I 2022-10-08 17:35:14] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:05:46 [I 2022-10-08 17:35:14] ...trap/engines/engine.py.229: process: 0.77262 | load: 0.00310 [I 2022-10-08 17:35:14] ...trap/engines/engine.py.230: loss: 0.18611 [I 2022-10-08 17:35:14] ...optimizers/trijoint.py.151: total_norm: 2.460047 [I 2022-10-08 17:35:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 1870/2304 [I 2022-10-08 17:35:22] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:05:42 [I 2022-10-08 17:35:22] ...trap/engines/engine.py.229: process: 0.79540 | load: 0.00303 [I 2022-10-08 17:35:22] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-08 17:35:22] ...optimizers/trijoint.py.151: total_norm: 2.754581 [I 2022-10-08 17:35:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 1880/2304 [I 2022-10-08 17:35:29] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:05:32 [I 2022-10-08 17:35:29] ...trap/engines/engine.py.229: process: 0.79920 | load: 0.00308 [I 2022-10-08 17:35:29] ...trap/engines/engine.py.230: loss: 0.22545 [I 2022-10-08 17:35:29] ...optimizers/trijoint.py.151: total_norm: 2.719923 [I 2022-10-08 17:35:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 1890/2304 [I 2022-10-08 17:35:37] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:05:22 [I 2022-10-08 17:35:37] ...trap/engines/engine.py.229: process: 0.78385 | load: 0.00299 [I 2022-10-08 17:35:37] ...trap/engines/engine.py.230: loss: 0.22838 [I 2022-10-08 17:35:37] ...optimizers/trijoint.py.151: total_norm: 2.894220 [I 2022-10-08 17:35:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 1900/2304 [I 2022-10-08 17:35:45] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:05:16 [I 2022-10-08 17:35:45] ...trap/engines/engine.py.229: process: 0.78905 | load: 0.00308 [I 2022-10-08 17:35:45] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-08 17:35:45] ...optimizers/trijoint.py.151: total_norm: 2.719388 [I 2022-10-08 17:35:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1910/2304 [I 2022-10-08 17:35:53] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:05:08 [I 2022-10-08 17:35:53] ...trap/engines/engine.py.229: process: 0.78226 | load: 0.00303 [I 2022-10-08 17:35:53] ...trap/engines/engine.py.230: loss: 0.19177 [I 2022-10-08 17:35:53] ...optimizers/trijoint.py.151: total_norm: 3.446237 [I 2022-10-08 17:36:01] ...trap/engines/engine.py.224: train: epoch 13 | batch 1920/2304 [I 2022-10-08 17:36:01] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:05:00 [I 2022-10-08 17:36:01] ...trap/engines/engine.py.229: process: 0.78549 | load: 0.00304 [I 2022-10-08 17:36:01] ...trap/engines/engine.py.230: loss: 0.17592 [I 2022-10-08 17:36:01] ...optimizers/trijoint.py.151: total_norm: 2.305121 [I 2022-10-08 17:36:09] ...trap/engines/engine.py.224: train: epoch 13 | batch 1930/2304 [I 2022-10-08 17:36:09] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:04:51 [I 2022-10-08 17:36:09] ...trap/engines/engine.py.229: process: 0.77358 | load: 0.00307 [I 2022-10-08 17:36:09] ...trap/engines/engine.py.230: loss: 0.24237 [I 2022-10-08 17:36:09] ...optimizers/trijoint.py.151: total_norm: 3.948498 [I 2022-10-08 17:36:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 1940/2304 [I 2022-10-08 17:36:16] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:04:44 [I 2022-10-08 17:36:17] ...trap/engines/engine.py.229: process: 0.78313 | load: 0.00310 [I 2022-10-08 17:36:17] ...trap/engines/engine.py.230: loss: 0.26102 [I 2022-10-08 17:36:17] ...optimizers/trijoint.py.151: total_norm: 2.791369 [I 2022-10-08 17:36:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 1950/2304 [I 2022-10-08 17:36:24] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:04:34 [I 2022-10-08 17:36:24] ...trap/engines/engine.py.229: process: 0.76844 | load: 0.00303 [I 2022-10-08 17:36:24] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-08 17:36:24] ...optimizers/trijoint.py.151: total_norm: 3.370367 [I 2022-10-08 17:36:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1960/2304 [I 2022-10-08 17:36:32] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:04:26 [I 2022-10-08 17:36:32] ...trap/engines/engine.py.229: process: 0.76763 | load: 0.00305 [I 2022-10-08 17:36:32] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-08 17:36:32] ...optimizers/trijoint.py.151: total_norm: 2.291794 [I 2022-10-08 17:36:40] ...trap/engines/engine.py.224: train: epoch 13 | batch 1970/2304 [I 2022-10-08 17:36:40] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:04:20 [I 2022-10-08 17:36:40] ...trap/engines/engine.py.229: process: 0.77731 | load: 0.00305 [I 2022-10-08 17:36:40] ...trap/engines/engine.py.230: loss: 0.16956 [I 2022-10-08 17:36:40] ...optimizers/trijoint.py.151: total_norm: 2.097270 [I 2022-10-08 17:36:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1980/2304 [I 2022-10-08 17:36:48] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:04:12 [I 2022-10-08 17:36:48] ...trap/engines/engine.py.229: process: 0.77379 | load: 0.00303 [I 2022-10-08 17:36:48] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-08 17:36:48] ...optimizers/trijoint.py.151: total_norm: 1.746860 [I 2022-10-08 17:36:55] ...trap/engines/engine.py.224: train: epoch 13 | batch 1990/2304 [I 2022-10-08 17:36:55] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:04:03 [I 2022-10-08 17:36:55] ...trap/engines/engine.py.229: process: 0.78197 | load: 0.00305 [I 2022-10-08 17:36:55] ...trap/engines/engine.py.230: loss: 0.28652 [I 2022-10-08 17:36:55] ...optimizers/trijoint.py.151: total_norm: 3.956578 [I 2022-10-08 17:37:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 2000/2304 [I 2022-10-08 17:37:03] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:03:57 [I 2022-10-08 17:37:03] ...trap/engines/engine.py.229: process: 0.77571 | load: 0.00303 [I 2022-10-08 17:37:03] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-08 17:37:03] ...optimizers/trijoint.py.151: total_norm: 2.568153 [I 2022-10-08 17:37:11] ...trap/engines/engine.py.224: train: epoch 13 | batch 2010/2304 [I 2022-10-08 17:37:11] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:03:48 [I 2022-10-08 17:37:11] ...trap/engines/engine.py.229: process: 0.77716 | load: 0.00309 [I 2022-10-08 17:37:11] ...trap/engines/engine.py.230: loss: 0.29434 [I 2022-10-08 17:37:11] ...optimizers/trijoint.py.151: total_norm: 2.863745 [I 2022-10-08 17:37:19] ...trap/engines/engine.py.224: train: epoch 13 | batch 2020/2304 [I 2022-10-08 17:37:19] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:03:41 [I 2022-10-08 17:37:19] ...trap/engines/engine.py.229: process: 0.79179 | load: 0.00305 [I 2022-10-08 17:37:19] ...trap/engines/engine.py.230: loss: 0.17323 [I 2022-10-08 17:37:19] ...optimizers/trijoint.py.151: total_norm: 2.544183 [I 2022-10-08 17:37:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 2030/2304 [I 2022-10-08 17:37:27] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:03:32 [I 2022-10-08 17:37:27] ...trap/engines/engine.py.229: process: 0.77005 | load: 0.00355 [I 2022-10-08 17:37:27] ...trap/engines/engine.py.230: loss: 0.16893 [I 2022-10-08 17:37:27] ...optimizers/trijoint.py.151: total_norm: 2.066457 [I 2022-10-08 17:37:34] ...trap/engines/engine.py.224: train: epoch 13 | batch 2040/2304 [I 2022-10-08 17:37:34] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:03:27 [I 2022-10-08 17:37:34] ...trap/engines/engine.py.229: process: 0.79266 | load: 0.00303 [I 2022-10-08 17:37:34] ...trap/engines/engine.py.230: loss: 0.20909 [I 2022-10-08 17:37:34] ...optimizers/trijoint.py.151: total_norm: 2.160626 [I 2022-10-08 17:37:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 2050/2304 [I 2022-10-08 17:37:42] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:03:18 [I 2022-10-08 17:37:42] ...trap/engines/engine.py.229: process: 0.77398 | load: 0.00302 [I 2022-10-08 17:37:42] ...trap/engines/engine.py.230: loss: 0.21856 [I 2022-10-08 17:37:42] ...optimizers/trijoint.py.151: total_norm: 2.770956 [I 2022-10-08 17:37:50] ...trap/engines/engine.py.224: train: epoch 13 | batch 2060/2304 [I 2022-10-08 17:37:50] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:03:10 [I 2022-10-08 17:37:50] ...trap/engines/engine.py.229: process: 0.78090 | load: 0.00304 [I 2022-10-08 17:37:50] ...trap/engines/engine.py.230: loss: 0.18188 [I 2022-10-08 17:37:50] ...optimizers/trijoint.py.151: total_norm: 2.151259 [I 2022-10-08 17:37:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 2070/2304 [I 2022-10-08 17:37:58] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:03:01 [I 2022-10-08 17:37:58] ...trap/engines/engine.py.229: process: 0.77013 | load: 0.00307 [I 2022-10-08 17:37:58] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-08 17:37:58] ...optimizers/trijoint.py.151: total_norm: 2.243150 [I 2022-10-08 17:38:06] ...trap/engines/engine.py.224: train: epoch 13 | batch 2080/2304 [I 2022-10-08 17:38:06] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:02:54 [I 2022-10-08 17:38:06] ...trap/engines/engine.py.229: process: 0.77881 | load: 0.00300 [I 2022-10-08 17:38:06] ...trap/engines/engine.py.230: loss: 0.19394 [I 2022-10-08 17:38:06] ...optimizers/trijoint.py.151: total_norm: 2.146274 [I 2022-10-08 17:38:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 2090/2304 [I 2022-10-08 17:38:14] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:02:47 [I 2022-10-08 17:38:14] ...trap/engines/engine.py.229: process: 0.77485 | load: 0.00298 [I 2022-10-08 17:38:14] ...trap/engines/engine.py.230: loss: 0.17896 [I 2022-10-08 17:38:14] ...optimizers/trijoint.py.151: total_norm: 2.957204 [I 2022-10-08 17:38:21] ...trap/engines/engine.py.224: train: epoch 13 | batch 2100/2304 [I 2022-10-08 17:38:21] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:02:38 [I 2022-10-08 17:38:21] ...trap/engines/engine.py.229: process: 0.77187 | load: 0.00298 [I 2022-10-08 17:38:21] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-08 17:38:21] ...optimizers/trijoint.py.151: total_norm: 2.230743 [I 2022-10-08 17:38:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 2110/2304 [I 2022-10-08 17:38:29] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:02:30 [I 2022-10-08 17:38:29] ...trap/engines/engine.py.229: process: 0.78591 | load: 0.00304 [I 2022-10-08 17:38:29] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-08 17:38:29] ...optimizers/trijoint.py.151: total_norm: 1.479701 [I 2022-10-08 17:38:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 2120/2304 [I 2022-10-08 17:38:37] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:02:22 [I 2022-10-08 17:38:37] ...trap/engines/engine.py.229: process: 0.77188 | load: 0.00306 [I 2022-10-08 17:38:37] ...trap/engines/engine.py.230: loss: 0.28543 [I 2022-10-08 17:38:37] ...optimizers/trijoint.py.151: total_norm: 3.531929 [I 2022-10-08 17:38:45] ...trap/engines/engine.py.224: train: epoch 13 | batch 2130/2304 [I 2022-10-08 17:38:45] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:02:14 [I 2022-10-08 17:38:45] ...trap/engines/engine.py.229: process: 0.75727 | load: 0.00304 [I 2022-10-08 17:38:45] ...trap/engines/engine.py.230: loss: 0.24767 [I 2022-10-08 17:38:45] ...optimizers/trijoint.py.151: total_norm: 3.117896 [I 2022-10-08 17:38:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 2140/2304 [I 2022-10-08 17:38:52] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:02:07 [I 2022-10-08 17:38:52] ...trap/engines/engine.py.229: process: 0.78081 | load: 0.00304 [I 2022-10-08 17:38:52] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-08 17:38:52] ...optimizers/trijoint.py.151: total_norm: 2.796504 [I 2022-10-08 17:39:00] ...trap/engines/engine.py.224: train: epoch 13 | batch 2150/2304 [I 2022-10-08 17:39:00] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:01:59 [I 2022-10-08 17:39:00] ...trap/engines/engine.py.229: process: 0.77234 | load: 0.00303 [I 2022-10-08 17:39:00] ...trap/engines/engine.py.230: loss: 0.17615 [I 2022-10-08 17:39:00] ...optimizers/trijoint.py.151: total_norm: 2.902107 [I 2022-10-08 17:39:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 2160/2304 [I 2022-10-08 17:39:08] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:01:52 [I 2022-10-08 17:39:08] ...trap/engines/engine.py.229: process: 0.79886 | load: 0.00306 [I 2022-10-08 17:39:08] ...trap/engines/engine.py.230: loss: 0.26760 [I 2022-10-08 17:39:08] ...optimizers/trijoint.py.151: total_norm: 4.048088 [I 2022-10-08 17:39:16] ...trap/engines/engine.py.224: train: epoch 13 | batch 2170/2304 [I 2022-10-08 17:39:16] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:01:44 [I 2022-10-08 17:39:16] ...trap/engines/engine.py.229: process: 0.78340 | load: 0.00301 [I 2022-10-08 17:39:16] ...trap/engines/engine.py.230: loss: 0.18703 [I 2022-10-08 17:39:16] ...optimizers/trijoint.py.151: total_norm: 3.214353 [I 2022-10-08 17:39:24] ...trap/engines/engine.py.224: train: epoch 13 | batch 2180/2304 [I 2022-10-08 17:39:24] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:01:36 [I 2022-10-08 17:39:24] ...trap/engines/engine.py.229: process: 0.77695 | load: 0.00304 [I 2022-10-08 17:39:24] ...trap/engines/engine.py.230: loss: 0.17043 [I 2022-10-08 17:39:24] ...optimizers/trijoint.py.151: total_norm: 3.487076 [I 2022-10-08 17:39:31] ...trap/engines/engine.py.224: train: epoch 13 | batch 2190/2304 [I 2022-10-08 17:39:31] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:01:28 [I 2022-10-08 17:39:31] ...trap/engines/engine.py.229: process: 0.76795 | load: 0.00301 [I 2022-10-08 17:39:31] ...trap/engines/engine.py.230: loss: 0.20112 [I 2022-10-08 17:39:31] ...optimizers/trijoint.py.151: total_norm: 3.646171 [I 2022-10-08 17:39:39] ...trap/engines/engine.py.224: train: epoch 13 | batch 2200/2304 [I 2022-10-08 17:39:39] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:01:21 [I 2022-10-08 17:39:39] ...trap/engines/engine.py.229: process: 0.77382 | load: 0.00303 [I 2022-10-08 17:39:39] ...trap/engines/engine.py.230: loss: 0.20136 [I 2022-10-08 17:39:39] ...optimizers/trijoint.py.151: total_norm: 3.566811 [I 2022-10-08 17:39:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 2210/2304 [I 2022-10-08 17:39:47] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:01:13 [I 2022-10-08 17:39:47] ...trap/engines/engine.py.229: process: 0.78894 | load: 0.00308 [I 2022-10-08 17:39:47] ...trap/engines/engine.py.230: loss: 0.18052 [I 2022-10-08 17:39:47] ...optimizers/trijoint.py.151: total_norm: 2.637812 [I 2022-10-08 17:39:55] ...trap/engines/engine.py.224: train: epoch 13 | batch 2220/2304 [I 2022-10-08 17:39:55] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:01:05 [I 2022-10-08 17:39:55] ...trap/engines/engine.py.229: process: 0.79196 | load: 0.00304 [I 2022-10-08 17:39:55] ...trap/engines/engine.py.230: loss: 0.24984 [I 2022-10-08 17:39:55] ...optimizers/trijoint.py.151: total_norm: 3.155619 [I 2022-10-08 17:40:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 2230/2304 [I 2022-10-08 17:40:03] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:00:57 [I 2022-10-08 17:40:03] ...trap/engines/engine.py.229: process: 0.77394 | load: 0.00304 [I 2022-10-08 17:40:03] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-08 17:40:03] ...optimizers/trijoint.py.151: total_norm: 2.107039 [I 2022-10-08 17:40:10] ...trap/engines/engine.py.224: train: epoch 13 | batch 2240/2304 [I 2022-10-08 17:40:10] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:00:49 [I 2022-10-08 17:40:10] ...trap/engines/engine.py.229: process: 0.76630 | load: 0.00305 [I 2022-10-08 17:40:10] ...trap/engines/engine.py.230: loss: 0.28723 [I 2022-10-08 17:40:10] ...optimizers/trijoint.py.151: total_norm: 3.502203 [I 2022-10-08 17:40:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 2250/2304 [I 2022-10-08 17:40:18] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:00:41 [I 2022-10-08 17:40:18] ...trap/engines/engine.py.229: process: 0.77339 | load: 0.00304 [I 2022-10-08 17:40:18] ...trap/engines/engine.py.230: loss: 0.20883 [I 2022-10-08 17:40:18] ...optimizers/trijoint.py.151: total_norm: 2.314300 [I 2022-10-08 17:40:26] ...trap/engines/engine.py.224: train: epoch 13 | batch 2260/2304 [I 2022-10-08 17:40:26] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:00:34 [I 2022-10-08 17:40:26] ...trap/engines/engine.py.229: process: 0.77788 | load: 0.00303 [I 2022-10-08 17:40:26] ...trap/engines/engine.py.230: loss: 0.22499 [I 2022-10-08 17:40:26] ...optimizers/trijoint.py.151: total_norm: 2.755991 [I 2022-10-08 17:40:34] ...trap/engines/engine.py.224: train: epoch 13 | batch 2270/2304 [I 2022-10-08 17:40:34] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:00:26 [I 2022-10-08 17:40:34] ...trap/engines/engine.py.229: process: 0.77253 | load: 0.00303 [I 2022-10-08 17:40:34] ...trap/engines/engine.py.230: loss: 0.21481 [I 2022-10-08 17:40:34] ...optimizers/trijoint.py.151: total_norm: 2.540793 [I 2022-10-08 17:40:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 2280/2304 [I 2022-10-08 17:40:42] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:00:18 [I 2022-10-08 17:40:42] ...trap/engines/engine.py.229: process: 0.76330 | load: 0.00303 [I 2022-10-08 17:40:42] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-08 17:40:42] ...optimizers/trijoint.py.151: total_norm: 2.492033 [I 2022-10-08 17:40:49] ...trap/engines/engine.py.224: train: epoch 13 | batch 2290/2304 [I 2022-10-08 17:40:49] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:00:10 [I 2022-10-08 17:40:49] ...trap/engines/engine.py.229: process: 0.77481 | load: 0.00303 [I 2022-10-08 17:40:49] ...trap/engines/engine.py.230: loss: 0.20909 [I 2022-10-08 17:40:49] ...optimizers/trijoint.py.151: total_norm: 2.459247 [I 2022-10-08 17:40:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 2300/2304 [I 2022-10-08 17:40:57] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:00:03 [I 2022-10-08 17:40:57] ...trap/engines/engine.py.229: process: 0.76872 | load: 0.00017 [I 2022-10-08 17:40:57] ...trap/engines/engine.py.230: loss: 0.16882 [I 2022-10-08 17:40:57] ...optimizers/trijoint.py.151: total_norm: 2.504187 [I 2022-10-08 17:41:00] ...trap/engines/engine.py.224: train: epoch 13 | batch 2304/2304 [I 2022-10-08 17:41:00] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:00:00 [I 2022-10-08 17:41:00] ...trap/engines/engine.py.229: process: 0.76370 | load: 0.00018 [I 2022-10-08 17:41:00] ...trap/engines/engine.py.230: loss: 0.21771 [I 2022-10-08 17:41:00] ...optimizers/trijoint.py.151: total_norm: 2.576459 [S 2022-10-08 17:41:02] ...trap/engines/engine.py.129: train_epoch.epoch: 13 [S 2022-10-08 17:41:02] ...trap/engines/engine.py.129: train_epoch.loss: 0.19126973370344674 [I 2022-10-08 17:41:02] ...s/criterions/triplet.py.84: margin = 0.12000000000000005 [I 2022-10-08 17:41:02] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 13 [I 2022-10-08 17:41:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 17:41:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 17:41:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 17:41:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 17:41:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 17:41:05] ...trap/engines/engine.py.321: eval: epoch 13 | batch 0/510 [I 2022-10-08 17:41:05] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:44 [I 2022-10-08 17:41:05] ...trap/engines/engine.py.326: process: 2.67475 | load: 2.13103 [I 2022-10-08 17:41:09] ...trap/engines/engine.py.321: eval: epoch 13 | batch 10/510 [I 2022-10-08 17:41:09] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:31 [I 2022-10-08 17:41:09] ...trap/engines/engine.py.326: process: 0.41275 | load: 0.00018 [I 2022-10-08 17:41:15] ...trap/engines/engine.py.321: eval: epoch 13 | batch 20/510 [I 2022-10-08 17:41:15] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:56 [I 2022-10-08 17:41:15] ...trap/engines/engine.py.326: process: 0.41347 | load: 0.00019 [I 2022-10-08 17:41:20] ...trap/engines/engine.py.321: eval: epoch 13 | batch 30/510 [I 2022-10-08 17:41:20] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:41 [I 2022-10-08 17:41:20] ...trap/engines/engine.py.326: process: 0.80757 | load: 0.00019 [I 2022-10-08 17:41:26] ...trap/engines/engine.py.321: eval: epoch 13 | batch 40/510 [I 2022-10-08 17:41:26] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:41 [I 2022-10-08 17:41:26] ...trap/engines/engine.py.326: process: 0.62517 | load: 0.00018 [I 2022-10-08 17:41:31] ...trap/engines/engine.py.321: eval: epoch 13 | batch 50/510 [I 2022-10-08 17:41:31] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:38 [I 2022-10-08 17:41:31] ...trap/engines/engine.py.326: process: 0.50458 | load: 0.00018 [I 2022-10-08 17:41:36] ...trap/engines/engine.py.321: eval: epoch 13 | batch 60/510 [I 2022-10-08 17:41:36] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:46 [I 2022-10-08 17:41:36] ...trap/engines/engine.py.326: process: 0.47157 | load: 0.00019 [I 2022-10-08 17:41:41] ...trap/engines/engine.py.321: eval: epoch 13 | batch 70/510 [I 2022-10-08 17:41:41] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:46 [I 2022-10-08 17:41:41] ...trap/engines/engine.py.326: process: 0.44522 | load: 0.00018 [I 2022-10-08 17:41:46] ...trap/engines/engine.py.321: eval: epoch 13 | batch 80/510 [I 2022-10-08 17:41:46] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:40 [I 2022-10-08 17:41:46] ...trap/engines/engine.py.326: process: 0.50976 | load: 0.00020 [I 2022-10-08 17:41:51] ...trap/engines/engine.py.321: eval: epoch 13 | batch 90/510 [I 2022-10-08 17:41:51] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:29 [I 2022-10-08 17:41:51] ...trap/engines/engine.py.326: process: 0.58264 | load: 0.00019 [I 2022-10-08 17:41:56] ...trap/engines/engine.py.321: eval: epoch 13 | batch 100/510 [I 2022-10-08 17:41:56] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:06 [I 2022-10-08 17:41:56] ...trap/engines/engine.py.326: process: 0.31227 | load: 0.00018 [I 2022-10-08 17:41:59] ...trap/engines/engine.py.321: eval: epoch 13 | batch 110/510 [I 2022-10-08 17:41:59] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:12 [I 2022-10-08 17:41:59] ...trap/engines/engine.py.326: process: 0.30940 | load: 0.00020 [I 2022-10-08 17:42:02] ...trap/engines/engine.py.321: eval: epoch 13 | batch 120/510 [I 2022-10-08 17:42:02] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-08 17:42:02] ...trap/engines/engine.py.326: process: 0.31727 | load: 0.00020 [I 2022-10-08 17:42:05] ...trap/engines/engine.py.321: eval: epoch 13 | batch 130/510 [I 2022-10-08 17:42:05] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-08 17:42:05] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00021 [I 2022-10-08 17:42:08] ...trap/engines/engine.py.321: eval: epoch 13 | batch 140/510 [I 2022-10-08 17:42:08] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:57 [I 2022-10-08 17:42:08] ...trap/engines/engine.py.326: process: 0.31942 | load: 0.00020 [I 2022-10-08 17:42:12] ...trap/engines/engine.py.321: eval: epoch 13 | batch 150/510 [I 2022-10-08 17:42:12] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:54 [I 2022-10-08 17:42:12] ...trap/engines/engine.py.326: process: 0.31542 | load: 0.00019 [I 2022-10-08 17:42:15] ...trap/engines/engine.py.321: eval: epoch 13 | batch 160/510 [I 2022-10-08 17:42:15] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:51 [I 2022-10-08 17:42:15] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00019 [I 2022-10-08 17:42:18] ...trap/engines/engine.py.321: eval: epoch 13 | batch 170/510 [I 2022-10-08 17:42:18] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-08 17:42:18] ...trap/engines/engine.py.326: process: 0.30999 | load: 0.00019 [I 2022-10-08 17:42:21] ...trap/engines/engine.py.321: eval: epoch 13 | batch 180/510 [I 2022-10-08 17:42:21] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-08 17:42:21] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00020 [I 2022-10-08 17:42:24] ...trap/engines/engine.py.321: eval: epoch 13 | batch 190/510 [I 2022-10-08 17:42:24] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:40 [I 2022-10-08 17:42:24] ...trap/engines/engine.py.326: process: 0.31247 | load: 0.00020 [I 2022-10-08 17:42:27] ...trap/engines/engine.py.321: eval: epoch 13 | batch 200/510 [I 2022-10-08 17:42:27] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:37 [I 2022-10-08 17:42:27] ...trap/engines/engine.py.326: process: 0.31367 | load: 0.00019 [I 2022-10-08 17:42:31] ...trap/engines/engine.py.321: eval: epoch 13 | batch 210/510 [I 2022-10-08 17:42:31] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:35 [I 2022-10-08 17:42:31] ...trap/engines/engine.py.326: process: 0.31790 | load: 0.00019 [I 2022-10-08 17:42:34] ...trap/engines/engine.py.321: eval: epoch 13 | batch 220/510 [I 2022-10-08 17:42:34] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:31 [I 2022-10-08 17:42:34] ...trap/engines/engine.py.326: process: 0.31615 | load: 0.00019 [I 2022-10-08 17:42:37] ...trap/engines/engine.py.321: eval: epoch 13 | batch 230/510 [I 2022-10-08 17:42:37] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:28 [I 2022-10-08 17:42:37] ...trap/engines/engine.py.326: process: 0.32070 | load: 0.00019 [I 2022-10-08 17:42:40] ...trap/engines/engine.py.321: eval: epoch 13 | batch 240/510 [I 2022-10-08 17:42:40] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:25 [I 2022-10-08 17:42:40] ...trap/engines/engine.py.326: process: 0.31317 | load: 0.00019 [I 2022-10-08 17:42:43] ...trap/engines/engine.py.321: eval: epoch 13 | batch 250/510 [I 2022-10-08 17:42:43] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:22 [I 2022-10-08 17:42:43] ...trap/engines/engine.py.326: process: 0.31399 | load: 0.00020 [I 2022-10-08 17:42:46] ...trap/engines/engine.py.321: eval: epoch 13 | batch 260/510 [I 2022-10-08 17:42:46] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-08 17:42:46] ...trap/engines/engine.py.326: process: 0.32050 | load: 0.00021 [I 2022-10-08 17:42:50] ...trap/engines/engine.py.321: eval: epoch 13 | batch 270/510 [I 2022-10-08 17:42:50] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:16 [I 2022-10-08 17:42:50] ...trap/engines/engine.py.326: process: 0.32071 | load: 0.00020 [I 2022-10-08 17:42:53] ...trap/engines/engine.py.321: eval: epoch 13 | batch 280/510 [I 2022-10-08 17:42:53] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:13 [I 2022-10-08 17:42:53] ...trap/engines/engine.py.326: process: 0.31921 | load: 0.00020 [I 2022-10-08 17:42:56] ...trap/engines/engine.py.321: eval: epoch 13 | batch 290/510 [I 2022-10-08 17:42:56] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:09 [I 2022-10-08 17:42:56] ...trap/engines/engine.py.326: process: 0.31322 | load: 0.00021 [I 2022-10-08 17:42:59] ...trap/engines/engine.py.321: eval: epoch 13 | batch 300/510 [I 2022-10-08 17:42:59] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:06 [I 2022-10-08 17:42:59] ...trap/engines/engine.py.326: process: 0.32020 | load: 0.00021 [I 2022-10-08 17:43:02] ...trap/engines/engine.py.321: eval: epoch 13 | batch 310/510 [I 2022-10-08 17:43:02] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-08 17:43:02] ...trap/engines/engine.py.326: process: 0.31714 | load: 0.00020 [I 2022-10-08 17:43:06] ...trap/engines/engine.py.321: eval: epoch 13 | batch 320/510 [I 2022-10-08 17:43:06] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-08 17:43:06] ...trap/engines/engine.py.326: process: 0.31644 | load: 0.00020 [I 2022-10-08 17:43:09] ...trap/engines/engine.py.321: eval: epoch 13 | batch 330/510 [I 2022-10-08 17:43:09] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:57 [I 2022-10-08 17:43:09] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00020 [I 2022-10-08 17:43:12] ...trap/engines/engine.py.321: eval: epoch 13 | batch 340/510 [I 2022-10-08 17:43:12] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-08 17:43:12] ...trap/engines/engine.py.326: process: 0.31749 | load: 0.00020 [I 2022-10-08 17:43:15] ...trap/engines/engine.py.321: eval: epoch 13 | batch 350/510 [I 2022-10-08 17:43:15] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-08 17:43:15] ...trap/engines/engine.py.326: process: 0.32138 | load: 0.00020 [I 2022-10-08 17:43:18] ...trap/engines/engine.py.321: eval: epoch 13 | batch 360/510 [I 2022-10-08 17:43:18] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-08 17:43:18] ...trap/engines/engine.py.326: process: 0.31832 | load: 0.00019 [I 2022-10-08 17:43:21] ...trap/engines/engine.py.321: eval: epoch 13 | batch 370/510 [I 2022-10-08 17:43:21] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:44 [I 2022-10-08 17:43:21] ...trap/engines/engine.py.326: process: 0.31640 | load: 0.00020 [I 2022-10-08 17:43:25] ...trap/engines/engine.py.321: eval: epoch 13 | batch 380/510 [I 2022-10-08 17:43:25] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:41 [I 2022-10-08 17:43:25] ...trap/engines/engine.py.326: process: 0.32046 | load: 0.00020 [I 2022-10-08 17:43:28] ...trap/engines/engine.py.321: eval: epoch 13 | batch 390/510 [I 2022-10-08 17:43:28] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-08 17:43:28] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00021 [I 2022-10-08 17:43:31] ...trap/engines/engine.py.321: eval: epoch 13 | batch 400/510 [I 2022-10-08 17:43:31] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-08 17:43:31] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00021 [I 2022-10-08 17:43:34] ...trap/engines/engine.py.321: eval: epoch 13 | batch 410/510 [I 2022-10-08 17:43:34] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:31 [I 2022-10-08 17:43:34] ...trap/engines/engine.py.326: process: 0.31912 | load: 0.00020 [I 2022-10-08 17:43:37] ...trap/engines/engine.py.321: eval: epoch 13 | batch 420/510 [I 2022-10-08 17:43:37] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:28 [I 2022-10-08 17:43:37] ...trap/engines/engine.py.326: process: 0.32197 | load: 0.00020 [I 2022-10-08 17:43:41] ...trap/engines/engine.py.321: eval: epoch 13 | batch 430/510 [I 2022-10-08 17:43:41] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-08 17:43:41] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00021 [I 2022-10-08 17:43:44] ...trap/engines/engine.py.321: eval: epoch 13 | batch 440/510 [I 2022-10-08 17:43:44] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-08 17:43:44] ...trap/engines/engine.py.326: process: 0.31478 | load: 0.00020 [I 2022-10-08 17:43:47] ...trap/engines/engine.py.321: eval: epoch 13 | batch 450/510 [I 2022-10-08 17:43:47] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-08 17:43:47] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00020 [I 2022-10-08 17:43:50] ...trap/engines/engine.py.321: eval: epoch 13 | batch 460/510 [I 2022-10-08 17:43:50] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:15 [I 2022-10-08 17:43:50] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00020 [I 2022-10-08 17:43:53] ...trap/engines/engine.py.321: eval: epoch 13 | batch 470/510 [I 2022-10-08 17:43:53] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:12 [I 2022-10-08 17:43:53] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00020 [I 2022-10-08 17:43:57] ...trap/engines/engine.py.321: eval: epoch 13 | batch 480/510 [I 2022-10-08 17:43:57] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-08 17:43:57] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00019 [I 2022-10-08 17:44:00] ...trap/engines/engine.py.321: eval: epoch 13 | batch 490/510 [I 2022-10-08 17:44:00] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-08 17:44:00] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00020 [I 2022-10-08 17:44:03] ...trap/engines/engine.py.321: eval: epoch 13 | batch 500/510 [I 2022-10-08 17:44:03] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-08 17:44:03] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00019 [I 2022-10-08 17:44:06] ...trap/engines/engine.py.321: eval: epoch 13 | batch 510/510 [I 2022-10-08 17:44:06] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-08 17:44:06] ...trap/engines/engine.py.326: process: 0.31422 | load: 0.00016 [S 2022-10-08 17:44:07] ...trap/engines/engine.py.132: eval_epoch.epoch: 13 [I 2022-10-08 17:44:07] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.605500 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.858200 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.911600 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.609900 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.859600 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.910100 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008310 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006585 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007446 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010885 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007130 [S 2022-10-08 17:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006685 [I 2022-10-08 17:44:39] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 17:44:39] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 17:44:39] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 17:44:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 17:44:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 17:44:40] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 17:44:42] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 17:44:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 17:44:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 17:44:47] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 17:44:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 17:44:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 17:44:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 14 [I 2022-10-08 17:44:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 17:44:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 0/2304 [I 2022-10-08 17:44:57] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:51:52 [I 2022-10-08 17:44:57] ...trap/engines/engine.py.229: process: 2.91326 | load: 2.08484 [I 2022-10-08 17:44:57] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-08 17:44:57] ...optimizers/trijoint.py.151: total_norm: 2.134159 [I 2022-10-08 17:45:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 10/2304 [I 2022-10-08 17:45:05] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:35 [I 2022-10-08 17:45:05] ...trap/engines/engine.py.229: process: 0.78843 | load: 0.00538 [I 2022-10-08 17:45:05] ...trap/engines/engine.py.230: loss: 0.15060 [I 2022-10-08 17:45:05] ...optimizers/trijoint.py.151: total_norm: 2.722918 [I 2022-10-08 17:45:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 20/2304 [I 2022-10-08 17:45:13] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:39 [I 2022-10-08 17:45:13] ...trap/engines/engine.py.229: process: 0.76944 | load: 0.00492 [I 2022-10-08 17:45:13] ...trap/engines/engine.py.230: loss: 0.17523 [I 2022-10-08 17:45:13] ...optimizers/trijoint.py.151: total_norm: 2.188004 [I 2022-10-08 17:45:20] ...trap/engines/engine.py.224: train: epoch 14 | batch 30/2304 [I 2022-10-08 17:45:20] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:22 [I 2022-10-08 17:45:20] ...trap/engines/engine.py.229: process: 0.77502 | load: 0.00454 [I 2022-10-08 17:45:20] ...trap/engines/engine.py.230: loss: 0.21009 [I 2022-10-08 17:45:20] ...optimizers/trijoint.py.151: total_norm: 2.904783 [I 2022-10-08 17:45:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 40/2304 [I 2022-10-08 17:45:28] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:19 [I 2022-10-08 17:45:28] ...trap/engines/engine.py.229: process: 0.77088 | load: 0.00389 [I 2022-10-08 17:45:28] ...trap/engines/engine.py.230: loss: 0.22071 [I 2022-10-08 17:45:28] ...optimizers/trijoint.py.151: total_norm: 2.331954 [I 2022-10-08 17:45:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 50/2304 [I 2022-10-08 17:45:36] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:20 [I 2022-10-08 17:45:36] ...trap/engines/engine.py.229: process: 0.78018 | load: 0.00348 [I 2022-10-08 17:45:36] ...trap/engines/engine.py.230: loss: 0.21653 [I 2022-10-08 17:45:36] ...optimizers/trijoint.py.151: total_norm: 3.090312 [I 2022-10-08 17:45:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 60/2304 [I 2022-10-08 17:45:44] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:07 [I 2022-10-08 17:45:44] ...trap/engines/engine.py.229: process: 0.77574 | load: 0.00330 [I 2022-10-08 17:45:44] ...trap/engines/engine.py.230: loss: 0.18039 [I 2022-10-08 17:45:44] ...optimizers/trijoint.py.151: total_norm: 2.606455 [I 2022-10-08 17:45:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 70/2304 [I 2022-10-08 17:45:52] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:29:05 [I 2022-10-08 17:45:52] ...trap/engines/engine.py.229: process: 0.78809 | load: 0.00377 [I 2022-10-08 17:45:52] ...trap/engines/engine.py.230: loss: 0.23181 [I 2022-10-08 17:45:52] ...optimizers/trijoint.py.151: total_norm: 3.354551 [I 2022-10-08 17:45:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 80/2304 [I 2022-10-08 17:45:59] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:48 [I 2022-10-08 17:45:59] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00329 [I 2022-10-08 17:45:59] ...trap/engines/engine.py.230: loss: 0.22517 [I 2022-10-08 17:45:59] ...optimizers/trijoint.py.151: total_norm: 3.884927 [I 2022-10-08 17:46:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 90/2304 [I 2022-10-08 17:46:07] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:32 [I 2022-10-08 17:46:07] ...trap/engines/engine.py.229: process: 0.76695 | load: 0.00335 [I 2022-10-08 17:46:07] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-08 17:46:07] ...optimizers/trijoint.py.151: total_norm: 1.958376 [I 2022-10-08 17:46:15] ...trap/engines/engine.py.224: train: epoch 14 | batch 100/2304 [I 2022-10-08 17:46:15] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:39 [I 2022-10-08 17:46:15] ...trap/engines/engine.py.229: process: 0.77889 | load: 0.00326 [I 2022-10-08 17:46:15] ...trap/engines/engine.py.230: loss: 0.18952 [I 2022-10-08 17:46:15] ...optimizers/trijoint.py.151: total_norm: 2.802248 [I 2022-10-08 17:46:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 110/2304 [I 2022-10-08 17:46:23] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:29 [I 2022-10-08 17:46:23] ...trap/engines/engine.py.229: process: 0.77530 | load: 0.00341 [I 2022-10-08 17:46:23] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-08 17:46:23] ...optimizers/trijoint.py.151: total_norm: 2.829042 [I 2022-10-08 17:46:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 120/2304 [I 2022-10-08 17:46:31] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:13 [I 2022-10-08 17:46:31] ...trap/engines/engine.py.229: process: 0.76577 | load: 0.00341 [I 2022-10-08 17:46:31] ...trap/engines/engine.py.230: loss: 0.23549 [I 2022-10-08 17:46:31] ...optimizers/trijoint.py.151: total_norm: 3.081502 [I 2022-10-08 17:46:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 130/2304 [I 2022-10-08 17:46:38] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:22 [I 2022-10-08 17:46:38] ...trap/engines/engine.py.229: process: 0.78396 | load: 0.00338 [I 2022-10-08 17:46:38] ...trap/engines/engine.py.230: loss: 0.15678 [I 2022-10-08 17:46:38] ...optimizers/trijoint.py.151: total_norm: 1.933807 [I 2022-10-08 17:46:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 140/2304 [I 2022-10-08 17:46:46] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:05 [I 2022-10-08 17:46:46] ...trap/engines/engine.py.229: process: 0.78152 | load: 0.00303 [I 2022-10-08 17:46:46] ...trap/engines/engine.py.230: loss: 0.15860 [I 2022-10-08 17:46:46] ...optimizers/trijoint.py.151: total_norm: 1.740886 [I 2022-10-08 17:46:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 150/2304 [I 2022-10-08 17:46:54] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:04 [I 2022-10-08 17:46:54] ...trap/engines/engine.py.229: process: 0.77969 | load: 0.00316 [I 2022-10-08 17:46:54] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-08 17:46:54] ...optimizers/trijoint.py.151: total_norm: 2.430456 [I 2022-10-08 17:47:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 160/2304 [I 2022-10-08 17:47:02] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:52 [I 2022-10-08 17:47:02] ...trap/engines/engine.py.229: process: 0.78100 | load: 0.00312 [I 2022-10-08 17:47:02] ...trap/engines/engine.py.230: loss: 0.17906 [I 2022-10-08 17:47:02] ...optimizers/trijoint.py.151: total_norm: 2.284831 [I 2022-10-08 17:47:10] ...trap/engines/engine.py.224: train: epoch 14 | batch 170/2304 [I 2022-10-08 17:47:10] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:42 [I 2022-10-08 17:47:10] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00303 [I 2022-10-08 17:47:10] ...trap/engines/engine.py.230: loss: 0.21800 [I 2022-10-08 17:47:10] ...optimizers/trijoint.py.151: total_norm: 2.956587 [I 2022-10-08 17:47:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 180/2304 [I 2022-10-08 17:47:18] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:39 [I 2022-10-08 17:47:18] ...trap/engines/engine.py.229: process: 0.78381 | load: 0.00305 [I 2022-10-08 17:47:18] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-08 17:47:18] ...optimizers/trijoint.py.151: total_norm: 2.421119 [I 2022-10-08 17:47:25] ...trap/engines/engine.py.224: train: epoch 14 | batch 190/2304 [I 2022-10-08 17:47:25] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:23 [I 2022-10-08 17:47:25] ...trap/engines/engine.py.229: process: 0.77868 | load: 0.00306 [I 2022-10-08 17:47:25] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-08 17:47:25] ...optimizers/trijoint.py.151: total_norm: 2.125503 [I 2022-10-08 17:47:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 200/2304 [I 2022-10-08 17:47:33] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:19 [I 2022-10-08 17:47:33] ...trap/engines/engine.py.229: process: 0.77681 | load: 0.00322 [I 2022-10-08 17:47:33] ...trap/engines/engine.py.230: loss: 0.23098 [I 2022-10-08 17:47:33] ...optimizers/trijoint.py.151: total_norm: 2.823187 [I 2022-10-08 17:47:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 210/2304 [I 2022-10-08 17:47:41] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:15 [I 2022-10-08 17:47:41] ...trap/engines/engine.py.229: process: 0.77725 | load: 0.00296 [I 2022-10-08 17:47:41] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-08 17:47:41] ...optimizers/trijoint.py.151: total_norm: 1.867389 [I 2022-10-08 17:47:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 220/2304 [I 2022-10-08 17:47:49] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:07 [I 2022-10-08 17:47:49] ...trap/engines/engine.py.229: process: 0.78629 | load: 0.00306 [I 2022-10-08 17:47:49] ...trap/engines/engine.py.230: loss: 0.16914 [I 2022-10-08 17:47:49] ...optimizers/trijoint.py.151: total_norm: 2.542330 [I 2022-10-08 17:47:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 230/2304 [I 2022-10-08 17:47:57] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:51 [I 2022-10-08 17:47:57] ...trap/engines/engine.py.229: process: 0.76870 | load: 0.00300 [I 2022-10-08 17:47:57] ...trap/engines/engine.py.230: loss: 0.23762 [I 2022-10-08 17:47:57] ...optimizers/trijoint.py.151: total_norm: 3.464790 [I 2022-10-08 17:48:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 240/2304 [I 2022-10-08 17:48:05] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:41 [I 2022-10-08 17:48:05] ...trap/engines/engine.py.229: process: 0.77400 | load: 0.00299 [I 2022-10-08 17:48:05] ...trap/engines/engine.py.230: loss: 0.23283 [I 2022-10-08 17:48:05] ...optimizers/trijoint.py.151: total_norm: 3.327260 [I 2022-10-08 17:48:12] ...trap/engines/engine.py.224: train: epoch 14 | batch 250/2304 [I 2022-10-08 17:48:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:35 [I 2022-10-08 17:48:12] ...trap/engines/engine.py.229: process: 0.77183 | load: 0.00303 [I 2022-10-08 17:48:12] ...trap/engines/engine.py.230: loss: 0.17142 [I 2022-10-08 17:48:12] ...optimizers/trijoint.py.151: total_norm: 3.066131 [I 2022-10-08 17:48:20] ...trap/engines/engine.py.224: train: epoch 14 | batch 260/2304 [I 2022-10-08 17:48:20] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:29 [I 2022-10-08 17:48:20] ...trap/engines/engine.py.229: process: 0.77582 | load: 0.00343 [I 2022-10-08 17:48:20] ...trap/engines/engine.py.230: loss: 0.13298 [I 2022-10-08 17:48:20] ...optimizers/trijoint.py.151: total_norm: 2.575055 [I 2022-10-08 17:48:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 270/2304 [I 2022-10-08 17:48:28] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:25 [I 2022-10-08 17:48:28] ...trap/engines/engine.py.229: process: 0.77874 | load: 0.00300 [I 2022-10-08 17:48:28] ...trap/engines/engine.py.230: loss: 0.19553 [I 2022-10-08 17:48:28] ...optimizers/trijoint.py.151: total_norm: 3.074537 [I 2022-10-08 17:48:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 280/2304 [I 2022-10-08 17:48:36] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:20 [I 2022-10-08 17:48:36] ...trap/engines/engine.py.229: process: 0.77680 | load: 0.00303 [I 2022-10-08 17:48:36] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-08 17:48:36] ...optimizers/trijoint.py.151: total_norm: 1.971409 [I 2022-10-08 17:48:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 290/2304 [I 2022-10-08 17:48:44] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:19 [I 2022-10-08 17:48:44] ...trap/engines/engine.py.229: process: 0.78380 | load: 0.00303 [I 2022-10-08 17:48:44] ...trap/engines/engine.py.230: loss: 0.20274 [I 2022-10-08 17:48:44] ...optimizers/trijoint.py.151: total_norm: 2.964203 [I 2022-10-08 17:48:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 300/2304 [I 2022-10-08 17:48:51] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:59 [I 2022-10-08 17:48:51] ...trap/engines/engine.py.229: process: 0.77468 | load: 0.00302 [I 2022-10-08 17:48:51] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-08 17:48:51] ...optimizers/trijoint.py.151: total_norm: 1.435810 [I 2022-10-08 17:48:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 310/2304 [I 2022-10-08 17:48:59] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:43 [I 2022-10-08 17:48:59] ...trap/engines/engine.py.229: process: 0.75957 | load: 0.00310 [I 2022-10-08 17:48:59] ...trap/engines/engine.py.230: loss: 0.22051 [I 2022-10-08 17:48:59] ...optimizers/trijoint.py.151: total_norm: 2.923929 [I 2022-10-08 17:49:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 320/2304 [I 2022-10-08 17:49:07] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:39 [I 2022-10-08 17:49:07] ...trap/engines/engine.py.229: process: 0.77533 | load: 0.00299 [I 2022-10-08 17:49:07] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-08 17:49:07] ...optimizers/trijoint.py.151: total_norm: 3.902720 [I 2022-10-08 17:49:15] ...trap/engines/engine.py.224: train: epoch 14 | batch 330/2304 [I 2022-10-08 17:49:15] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:33 [I 2022-10-08 17:49:15] ...trap/engines/engine.py.229: process: 0.76265 | load: 0.00306 [I 2022-10-08 17:49:15] ...trap/engines/engine.py.230: loss: 0.18064 [I 2022-10-08 17:49:15] ...optimizers/trijoint.py.151: total_norm: 3.402278 [I 2022-10-08 17:49:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 340/2304 [I 2022-10-08 17:49:23] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:22 [I 2022-10-08 17:49:23] ...trap/engines/engine.py.229: process: 0.77971 | load: 0.00300 [I 2022-10-08 17:49:23] ...trap/engines/engine.py.230: loss: 0.15915 [I 2022-10-08 17:49:23] ...optimizers/trijoint.py.151: total_norm: 3.107977 [I 2022-10-08 17:49:30] ...trap/engines/engine.py.224: train: epoch 14 | batch 350/2304 [I 2022-10-08 17:49:30] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:07 [I 2022-10-08 17:49:30] ...trap/engines/engine.py.229: process: 0.76405 | load: 0.00314 [I 2022-10-08 17:49:30] ...trap/engines/engine.py.230: loss: 0.25495 [I 2022-10-08 17:49:30] ...optimizers/trijoint.py.151: total_norm: 3.894700 [I 2022-10-08 17:49:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 360/2304 [I 2022-10-08 17:49:38] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:14 [I 2022-10-08 17:49:38] ...trap/engines/engine.py.229: process: 0.76552 | load: 0.00314 [I 2022-10-08 17:49:38] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-08 17:49:38] ...optimizers/trijoint.py.151: total_norm: 2.643523 [I 2022-10-08 17:49:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 370/2304 [I 2022-10-08 17:49:46] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:11 [I 2022-10-08 17:49:46] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00300 [I 2022-10-08 17:49:46] ...trap/engines/engine.py.230: loss: 0.14401 [I 2022-10-08 17:49:46] ...optimizers/trijoint.py.151: total_norm: 2.220012 [I 2022-10-08 17:49:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 380/2304 [I 2022-10-08 17:49:54] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:00 [I 2022-10-08 17:49:54] ...trap/engines/engine.py.229: process: 0.77555 | load: 0.00303 [I 2022-10-08 17:49:54] ...trap/engines/engine.py.230: loss: 0.17947 [I 2022-10-08 17:49:54] ...optimizers/trijoint.py.151: total_norm: 2.378052 [I 2022-10-08 17:50:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 390/2304 [I 2022-10-08 17:50:02] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:50 [I 2022-10-08 17:50:02] ...trap/engines/engine.py.229: process: 0.78487 | load: 0.00302 [I 2022-10-08 17:50:02] ...trap/engines/engine.py.230: loss: 0.21777 [I 2022-10-08 17:50:02] ...optimizers/trijoint.py.151: total_norm: 2.975719 [I 2022-10-08 17:50:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 400/2304 [I 2022-10-08 17:50:09] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:41 [I 2022-10-08 17:50:09] ...trap/engines/engine.py.229: process: 0.76677 | load: 0.00302 [I 2022-10-08 17:50:09] ...trap/engines/engine.py.230: loss: 0.21577 [I 2022-10-08 17:50:09] ...optimizers/trijoint.py.151: total_norm: 3.034977 [I 2022-10-08 17:50:17] ...trap/engines/engine.py.224: train: epoch 14 | batch 410/2304 [I 2022-10-08 17:50:17] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:36 [I 2022-10-08 17:50:17] ...trap/engines/engine.py.229: process: 0.77464 | load: 0.00307 [I 2022-10-08 17:50:17] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-08 17:50:17] ...optimizers/trijoint.py.151: total_norm: 1.868549 [I 2022-10-08 17:50:25] ...trap/engines/engine.py.224: train: epoch 14 | batch 420/2304 [I 2022-10-08 17:50:25] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:30 [I 2022-10-08 17:50:25] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00302 [I 2022-10-08 17:50:25] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-08 17:50:25] ...optimizers/trijoint.py.151: total_norm: 2.247492 [I 2022-10-08 17:50:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 430/2304 [I 2022-10-08 17:50:33] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:31 [I 2022-10-08 17:50:33] ...trap/engines/engine.py.229: process: 0.78416 | load: 0.00303 [I 2022-10-08 17:50:33] ...trap/engines/engine.py.230: loss: 0.13358 [I 2022-10-08 17:50:33] ...optimizers/trijoint.py.151: total_norm: 2.371775 [I 2022-10-08 17:50:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 440/2304 [I 2022-10-08 17:50:41] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:14 [I 2022-10-08 17:50:41] ...trap/engines/engine.py.229: process: 0.77179 | load: 0.00315 [I 2022-10-08 17:50:41] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-08 17:50:41] ...optimizers/trijoint.py.151: total_norm: 1.970250 [I 2022-10-08 17:50:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 450/2304 [I 2022-10-08 17:50:49] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:11 [I 2022-10-08 17:50:49] ...trap/engines/engine.py.229: process: 0.79261 | load: 0.00302 [I 2022-10-08 17:50:49] ...trap/engines/engine.py.230: loss: 0.17067 [I 2022-10-08 17:50:49] ...optimizers/trijoint.py.151: total_norm: 2.263969 [I 2022-10-08 17:50:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 460/2304 [I 2022-10-08 17:50:56] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:57 [I 2022-10-08 17:50:56] ...trap/engines/engine.py.229: process: 0.79103 | load: 0.00302 [I 2022-10-08 17:50:56] ...trap/engines/engine.py.230: loss: 0.21001 [I 2022-10-08 17:50:56] ...optimizers/trijoint.py.151: total_norm: 3.010610 [I 2022-10-08 17:51:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 470/2304 [I 2022-10-08 17:51:04] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:42 [I 2022-10-08 17:51:04] ...trap/engines/engine.py.229: process: 0.77334 | load: 0.00302 [I 2022-10-08 17:51:04] ...trap/engines/engine.py.230: loss: 0.26194 [I 2022-10-08 17:51:04] ...optimizers/trijoint.py.151: total_norm: 2.816738 [I 2022-10-08 17:51:12] ...trap/engines/engine.py.224: train: epoch 14 | batch 480/2304 [I 2022-10-08 17:51:12] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:39 [I 2022-10-08 17:51:12] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00305 [I 2022-10-08 17:51:12] ...trap/engines/engine.py.230: loss: 0.11800 [I 2022-10-08 17:51:12] ...optimizers/trijoint.py.151: total_norm: 2.569156 [I 2022-10-08 17:51:20] ...trap/engines/engine.py.224: train: epoch 14 | batch 490/2304 [I 2022-10-08 17:51:20] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:35 [I 2022-10-08 17:51:20] ...trap/engines/engine.py.229: process: 0.77955 | load: 0.00306 [I 2022-10-08 17:51:20] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-08 17:51:20] ...optimizers/trijoint.py.151: total_norm: 2.448629 [I 2022-10-08 17:51:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 500/2304 [I 2022-10-08 17:51:28] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:17 [I 2022-10-08 17:51:28] ...trap/engines/engine.py.229: process: 0.76590 | load: 0.00302 [I 2022-10-08 17:51:28] ...trap/engines/engine.py.230: loss: 0.10458 [I 2022-10-08 17:51:28] ...optimizers/trijoint.py.151: total_norm: 1.939694 [I 2022-10-08 17:51:35] ...trap/engines/engine.py.224: train: epoch 14 | batch 510/2304 [I 2022-10-08 17:51:35] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:10 [I 2022-10-08 17:51:35] ...trap/engines/engine.py.229: process: 0.76690 | load: 0.00302 [I 2022-10-08 17:51:35] ...trap/engines/engine.py.230: loss: 0.16927 [I 2022-10-08 17:51:35] ...optimizers/trijoint.py.151: total_norm: 3.302956 [I 2022-10-08 17:51:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 520/2304 [I 2022-10-08 17:51:43] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:10 [I 2022-10-08 17:51:43] ...trap/engines/engine.py.229: process: 0.77438 | load: 0.00300 [I 2022-10-08 17:51:43] ...trap/engines/engine.py.230: loss: 0.16571 [I 2022-10-08 17:51:43] ...optimizers/trijoint.py.151: total_norm: 2.540718 [I 2022-10-08 17:51:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 530/2304 [I 2022-10-08 17:51:51] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:22:51 [I 2022-10-08 17:51:51] ...trap/engines/engine.py.229: process: 0.76489 | load: 0.00303 [I 2022-10-08 17:51:51] ...trap/engines/engine.py.230: loss: 0.20349 [I 2022-10-08 17:51:51] ...optimizers/trijoint.py.151: total_norm: 3.404765 [I 2022-10-08 17:51:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 540/2304 [I 2022-10-08 17:51:59] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:50 [I 2022-10-08 17:51:59] ...trap/engines/engine.py.229: process: 0.78460 | load: 0.00305 [I 2022-10-08 17:51:59] ...trap/engines/engine.py.230: loss: 0.24617 [I 2022-10-08 17:51:59] ...optimizers/trijoint.py.151: total_norm: 2.682435 [I 2022-10-08 17:52:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 550/2304 [I 2022-10-08 17:52:06] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:42 [I 2022-10-08 17:52:06] ...trap/engines/engine.py.229: process: 0.77586 | load: 0.00304 [I 2022-10-08 17:52:06] ...trap/engines/engine.py.230: loss: 0.19344 [I 2022-10-08 17:52:06] ...optimizers/trijoint.py.151: total_norm: 2.552923 [I 2022-10-08 17:52:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 560/2304 [I 2022-10-08 17:52:14] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:34 [I 2022-10-08 17:52:14] ...trap/engines/engine.py.229: process: 0.77176 | load: 0.00306 [I 2022-10-08 17:52:14] ...trap/engines/engine.py.230: loss: 0.28140 [I 2022-10-08 17:52:14] ...optimizers/trijoint.py.151: total_norm: 3.440020 [I 2022-10-08 17:52:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 570/2304 [I 2022-10-08 17:52:22] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:22:34 [I 2022-10-08 17:52:22] ...trap/engines/engine.py.229: process: 0.78278 | load: 0.00306 [I 2022-10-08 17:52:22] ...trap/engines/engine.py.230: loss: 0.18968 [I 2022-10-08 17:52:22] ...optimizers/trijoint.py.151: total_norm: 2.964253 [I 2022-10-08 17:52:30] ...trap/engines/engine.py.224: train: epoch 14 | batch 580/2304 [I 2022-10-08 17:52:30] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:22:25 [I 2022-10-08 17:52:30] ...trap/engines/engine.py.229: process: 0.78265 | load: 0.00303 [I 2022-10-08 17:52:30] ...trap/engines/engine.py.230: loss: 0.13496 [I 2022-10-08 17:52:30] ...optimizers/trijoint.py.151: total_norm: 2.156361 [I 2022-10-08 17:52:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 590/2304 [I 2022-10-08 17:52:38] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:13 [I 2022-10-08 17:52:38] ...trap/engines/engine.py.229: process: 0.76547 | load: 0.00304 [I 2022-10-08 17:52:38] ...trap/engines/engine.py.230: loss: 0.26618 [I 2022-10-08 17:52:38] ...optimizers/trijoint.py.151: total_norm: 4.569016 [I 2022-10-08 17:52:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 600/2304 [I 2022-10-08 17:52:45] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:07 [I 2022-10-08 17:52:45] ...trap/engines/engine.py.229: process: 0.77295 | load: 0.00306 [I 2022-10-08 17:52:45] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-08 17:52:45] ...optimizers/trijoint.py.151: total_norm: 1.923692 [I 2022-10-08 17:52:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 610/2304 [I 2022-10-08 17:52:53] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:21:57 [I 2022-10-08 17:52:53] ...trap/engines/engine.py.229: process: 0.77060 | load: 0.00300 [I 2022-10-08 17:52:53] ...trap/engines/engine.py.230: loss: 0.18082 [I 2022-10-08 17:52:53] ...optimizers/trijoint.py.151: total_norm: 2.462232 [I 2022-10-08 17:53:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 620/2304 [I 2022-10-08 17:53:01] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:21:51 [I 2022-10-08 17:53:01] ...trap/engines/engine.py.229: process: 0.78431 | load: 0.00303 [I 2022-10-08 17:53:01] ...trap/engines/engine.py.230: loss: 0.17136 [I 2022-10-08 17:53:01] ...optimizers/trijoint.py.151: total_norm: 2.623758 [I 2022-10-08 17:53:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 630/2304 [I 2022-10-08 17:53:09] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:21:39 [I 2022-10-08 17:53:09] ...trap/engines/engine.py.229: process: 0.77219 | load: 0.00304 [I 2022-10-08 17:53:09] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-08 17:53:09] ...optimizers/trijoint.py.151: total_norm: 2.718378 [I 2022-10-08 17:53:17] ...trap/engines/engine.py.224: train: epoch 14 | batch 640/2304 [I 2022-10-08 17:53:17] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:48 [I 2022-10-08 17:53:17] ...trap/engines/engine.py.229: process: 0.79140 | load: 0.00309 [I 2022-10-08 17:53:17] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-08 17:53:17] ...optimizers/trijoint.py.151: total_norm: 2.744007 [I 2022-10-08 17:53:25] ...trap/engines/engine.py.224: train: epoch 14 | batch 650/2304 [I 2022-10-08 17:53:25] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:33 [I 2022-10-08 17:53:25] ...trap/engines/engine.py.229: process: 0.77939 | load: 0.00304 [I 2022-10-08 17:53:25] ...trap/engines/engine.py.230: loss: 0.18106 [I 2022-10-08 17:53:25] ...optimizers/trijoint.py.151: total_norm: 2.435675 [I 2022-10-08 17:53:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 660/2304 [I 2022-10-08 17:53:32] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:15 [I 2022-10-08 17:53:32] ...trap/engines/engine.py.229: process: 0.77211 | load: 0.00303 [I 2022-10-08 17:53:32] ...trap/engines/engine.py.230: loss: 0.21074 [I 2022-10-08 17:53:32] ...optimizers/trijoint.py.151: total_norm: 3.119330 [I 2022-10-08 17:53:40] ...trap/engines/engine.py.224: train: epoch 14 | batch 670/2304 [I 2022-10-08 17:53:40] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:10 [I 2022-10-08 17:53:40] ...trap/engines/engine.py.229: process: 0.76728 | load: 0.00303 [I 2022-10-08 17:53:40] ...trap/engines/engine.py.230: loss: 0.18722 [I 2022-10-08 17:53:40] ...optimizers/trijoint.py.151: total_norm: 2.238237 [I 2022-10-08 17:53:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 680/2304 [I 2022-10-08 17:53:48] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:21:07 [I 2022-10-08 17:53:48] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00305 [I 2022-10-08 17:53:48] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-08 17:53:48] ...optimizers/trijoint.py.151: total_norm: 2.777586 [I 2022-10-08 17:53:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 690/2304 [I 2022-10-08 17:53:56] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:56 [I 2022-10-08 17:53:56] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00303 [I 2022-10-08 17:53:56] ...trap/engines/engine.py.230: loss: 0.13979 [I 2022-10-08 17:53:56] ...optimizers/trijoint.py.151: total_norm: 2.522699 [I 2022-10-08 17:54:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 700/2304 [I 2022-10-08 17:54:04] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:47 [I 2022-10-08 17:54:04] ...trap/engines/engine.py.229: process: 0.76596 | load: 0.00306 [I 2022-10-08 17:54:04] ...trap/engines/engine.py.230: loss: 0.15647 [I 2022-10-08 17:54:04] ...optimizers/trijoint.py.151: total_norm: 1.947629 [I 2022-10-08 17:54:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 710/2304 [I 2022-10-08 17:54:11] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:44 [I 2022-10-08 17:54:11] ...trap/engines/engine.py.229: process: 0.78704 | load: 0.00303 [I 2022-10-08 17:54:11] ...trap/engines/engine.py.230: loss: 0.14431 [I 2022-10-08 17:54:11] ...optimizers/trijoint.py.151: total_norm: 2.048826 [I 2022-10-08 17:54:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 720/2304 [I 2022-10-08 17:54:19] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:34 [I 2022-10-08 17:54:19] ...trap/engines/engine.py.229: process: 0.78083 | load: 0.00301 [I 2022-10-08 17:54:19] ...trap/engines/engine.py.230: loss: 0.14155 [I 2022-10-08 17:54:19] ...optimizers/trijoint.py.151: total_norm: 2.264105 [I 2022-10-08 17:54:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 730/2304 [I 2022-10-08 17:54:27] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:20:24 [I 2022-10-08 17:54:27] ...trap/engines/engine.py.229: process: 0.77328 | load: 0.00302 [I 2022-10-08 17:54:27] ...trap/engines/engine.py.230: loss: 0.24170 [I 2022-10-08 17:54:27] ...optimizers/trijoint.py.151: total_norm: 2.961271 [I 2022-10-08 17:54:35] ...trap/engines/engine.py.224: train: epoch 14 | batch 740/2304 [I 2022-10-08 17:54:35] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:18 [I 2022-10-08 17:54:35] ...trap/engines/engine.py.229: process: 0.78244 | load: 0.00313 [I 2022-10-08 17:54:35] ...trap/engines/engine.py.230: loss: 0.18187 [I 2022-10-08 17:54:35] ...optimizers/trijoint.py.151: total_norm: 2.506248 [I 2022-10-08 17:54:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 750/2304 [I 2022-10-08 17:54:43] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:14 [I 2022-10-08 17:54:43] ...trap/engines/engine.py.229: process: 0.78362 | load: 0.00304 [I 2022-10-08 17:54:43] ...trap/engines/engine.py.230: loss: 0.14770 [I 2022-10-08 17:54:43] ...optimizers/trijoint.py.151: total_norm: 2.209021 [I 2022-10-08 17:54:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 760/2304 [I 2022-10-08 17:54:50] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:00 [I 2022-10-08 17:54:50] ...trap/engines/engine.py.229: process: 0.77445 | load: 0.00304 [I 2022-10-08 17:54:50] ...trap/engines/engine.py.230: loss: 0.13772 [I 2022-10-08 17:54:50] ...optimizers/trijoint.py.151: total_norm: 1.961443 [I 2022-10-08 17:54:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 770/2304 [I 2022-10-08 17:54:58] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:20:00 [I 2022-10-08 17:54:58] ...trap/engines/engine.py.229: process: 0.78949 | load: 0.00302 [I 2022-10-08 17:54:58] ...trap/engines/engine.py.230: loss: 0.22102 [I 2022-10-08 17:54:58] ...optimizers/trijoint.py.151: total_norm: 2.208445 [I 2022-10-08 17:55:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 780/2304 [I 2022-10-08 17:55:06] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:57 [I 2022-10-08 17:55:06] ...trap/engines/engine.py.229: process: 0.78222 | load: 0.00300 [I 2022-10-08 17:55:06] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-08 17:55:06] ...optimizers/trijoint.py.151: total_norm: 2.274558 [I 2022-10-08 17:55:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 790/2304 [I 2022-10-08 17:55:14] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:19:46 [I 2022-10-08 17:55:14] ...trap/engines/engine.py.229: process: 0.78958 | load: 0.00297 [I 2022-10-08 17:55:14] ...trap/engines/engine.py.230: loss: 0.17442 [I 2022-10-08 17:55:14] ...optimizers/trijoint.py.151: total_norm: 3.342643 [I 2022-10-08 17:55:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 800/2304 [I 2022-10-08 17:55:22] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:32 [I 2022-10-08 17:55:22] ...trap/engines/engine.py.229: process: 0.77766 | load: 0.00304 [I 2022-10-08 17:55:22] ...trap/engines/engine.py.230: loss: 0.20706 [I 2022-10-08 17:55:22] ...optimizers/trijoint.py.151: total_norm: 3.136293 [I 2022-10-08 17:55:30] ...trap/engines/engine.py.224: train: epoch 14 | batch 810/2304 [I 2022-10-08 17:55:30] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:15 [I 2022-10-08 17:55:30] ...trap/engines/engine.py.229: process: 0.78228 | load: 0.00303 [I 2022-10-08 17:55:30] ...trap/engines/engine.py.230: loss: 0.19031 [I 2022-10-08 17:55:30] ...optimizers/trijoint.py.151: total_norm: 3.275783 [I 2022-10-08 17:55:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 820/2304 [I 2022-10-08 17:55:37] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:09 [I 2022-10-08 17:55:37] ...trap/engines/engine.py.229: process: 0.76179 | load: 0.00304 [I 2022-10-08 17:55:37] ...trap/engines/engine.py.230: loss: 0.22921 [I 2022-10-08 17:55:37] ...optimizers/trijoint.py.151: total_norm: 3.450669 [I 2022-10-08 17:55:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 830/2304 [I 2022-10-08 17:55:45] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:04 [I 2022-10-08 17:55:45] ...trap/engines/engine.py.229: process: 0.77802 | load: 0.00304 [I 2022-10-08 17:55:45] ...trap/engines/engine.py.230: loss: 0.25104 [I 2022-10-08 17:55:45] ...optimizers/trijoint.py.151: total_norm: 3.548034 [I 2022-10-08 17:55:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 840/2304 [I 2022-10-08 17:55:53] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:19:04 [I 2022-10-08 17:55:53] ...trap/engines/engine.py.229: process: 0.79114 | load: 0.00305 [I 2022-10-08 17:55:53] ...trap/engines/engine.py.230: loss: 0.19448 [I 2022-10-08 17:55:53] ...optimizers/trijoint.py.151: total_norm: 3.355237 [I 2022-10-08 17:56:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 850/2304 [I 2022-10-08 17:56:01] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:52 [I 2022-10-08 17:56:01] ...trap/engines/engine.py.229: process: 0.78583 | load: 0.00302 [I 2022-10-08 17:56:01] ...trap/engines/engine.py.230: loss: 0.16429 [I 2022-10-08 17:56:01] ...optimizers/trijoint.py.151: total_norm: 2.044523 [I 2022-10-08 17:56:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 860/2304 [I 2022-10-08 17:56:09] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:48 [I 2022-10-08 17:56:09] ...trap/engines/engine.py.229: process: 0.78974 | load: 0.00303 [I 2022-10-08 17:56:09] ...trap/engines/engine.py.230: loss: 0.17162 [I 2022-10-08 17:56:09] ...optimizers/trijoint.py.151: total_norm: 2.119121 [I 2022-10-08 17:56:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 870/2304 [I 2022-10-08 17:56:16] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:33 [I 2022-10-08 17:56:16] ...trap/engines/engine.py.229: process: 0.77837 | load: 0.00303 [I 2022-10-08 17:56:16] ...trap/engines/engine.py.230: loss: 0.13345 [I 2022-10-08 17:56:16] ...optimizers/trijoint.py.151: total_norm: 2.341415 [I 2022-10-08 17:56:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 880/2304 [I 2022-10-08 17:56:24] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:29 [I 2022-10-08 17:56:24] ...trap/engines/engine.py.229: process: 0.77383 | load: 0.00307 [I 2022-10-08 17:56:24] ...trap/engines/engine.py.230: loss: 0.20179 [I 2022-10-08 17:56:24] ...optimizers/trijoint.py.151: total_norm: 2.919024 [I 2022-10-08 17:56:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 890/2304 [I 2022-10-08 17:56:32] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:24 [I 2022-10-08 17:56:32] ...trap/engines/engine.py.229: process: 0.77909 | load: 0.00303 [I 2022-10-08 17:56:32] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-08 17:56:32] ...optimizers/trijoint.py.151: total_norm: 1.667584 [I 2022-10-08 17:56:40] ...trap/engines/engine.py.224: train: epoch 14 | batch 900/2304 [I 2022-10-08 17:56:40] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:16 [I 2022-10-08 17:56:40] ...trap/engines/engine.py.229: process: 0.78041 | load: 0.00303 [I 2022-10-08 17:56:40] ...trap/engines/engine.py.230: loss: 0.26764 [I 2022-10-08 17:56:40] ...optimizers/trijoint.py.151: total_norm: 3.590563 [I 2022-10-08 17:56:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 910/2304 [I 2022-10-08 17:56:48] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:12 [I 2022-10-08 17:56:48] ...trap/engines/engine.py.229: process: 0.78687 | load: 0.00302 [I 2022-10-08 17:56:48] ...trap/engines/engine.py.230: loss: 0.20426 [I 2022-10-08 17:56:48] ...optimizers/trijoint.py.151: total_norm: 3.046796 [I 2022-10-08 17:56:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 920/2304 [I 2022-10-08 17:56:55] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:17:59 [I 2022-10-08 17:56:55] ...trap/engines/engine.py.229: process: 0.78320 | load: 0.00300 [I 2022-10-08 17:56:55] ...trap/engines/engine.py.230: loss: 0.18623 [I 2022-10-08 17:56:55] ...optimizers/trijoint.py.151: total_norm: 3.280334 [I 2022-10-08 17:57:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 930/2304 [I 2022-10-08 17:57:03] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:51 [I 2022-10-08 17:57:03] ...trap/engines/engine.py.229: process: 0.78823 | load: 0.00301 [I 2022-10-08 17:57:03] ...trap/engines/engine.py.230: loss: 0.23150 [I 2022-10-08 17:57:03] ...optimizers/trijoint.py.151: total_norm: 3.916657 [I 2022-10-08 17:57:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 940/2304 [I 2022-10-08 17:57:11] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:49 [I 2022-10-08 17:57:11] ...trap/engines/engine.py.229: process: 0.77842 | load: 0.00303 [I 2022-10-08 17:57:11] ...trap/engines/engine.py.230: loss: 0.22492 [I 2022-10-08 17:57:11] ...optimizers/trijoint.py.151: total_norm: 3.007692 [I 2022-10-08 17:57:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 950/2304 [I 2022-10-08 17:57:19] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:45 [I 2022-10-08 17:57:19] ...trap/engines/engine.py.229: process: 0.77873 | load: 0.00300 [I 2022-10-08 17:57:19] ...trap/engines/engine.py.230: loss: 0.13778 [I 2022-10-08 17:57:19] ...optimizers/trijoint.py.151: total_norm: 1.841751 [I 2022-10-08 17:57:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 960/2304 [I 2022-10-08 17:57:27] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:17:31 [I 2022-10-08 17:57:27] ...trap/engines/engine.py.229: process: 0.77723 | load: 0.00301 [I 2022-10-08 17:57:27] ...trap/engines/engine.py.230: loss: 0.22938 [I 2022-10-08 17:57:27] ...optimizers/trijoint.py.151: total_norm: 3.223740 [I 2022-10-08 17:57:35] ...trap/engines/engine.py.224: train: epoch 14 | batch 970/2304 [I 2022-10-08 17:57:35] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:24 [I 2022-10-08 17:57:35] ...trap/engines/engine.py.229: process: 0.79134 | load: 0.00308 [I 2022-10-08 17:57:35] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-08 17:57:35] ...optimizers/trijoint.py.151: total_norm: 2.850197 [I 2022-10-08 17:57:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 980/2304 [I 2022-10-08 17:57:43] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:19 [I 2022-10-08 17:57:43] ...trap/engines/engine.py.229: process: 0.79637 | load: 0.00301 [I 2022-10-08 17:57:43] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-08 17:57:43] ...optimizers/trijoint.py.151: total_norm: 1.989709 [I 2022-10-08 17:57:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 990/2304 [I 2022-10-08 17:57:50] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:17:03 [I 2022-10-08 17:57:50] ...trap/engines/engine.py.229: process: 0.77765 | load: 0.00302 [I 2022-10-08 17:57:50] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-08 17:57:50] ...optimizers/trijoint.py.151: total_norm: 2.365125 [I 2022-10-08 17:57:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1000/2304 [I 2022-10-08 17:57:58] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:17:03 [I 2022-10-08 17:57:58] ...trap/engines/engine.py.229: process: 0.80135 | load: 0.00301 [I 2022-10-08 17:57:58] ...trap/engines/engine.py.230: loss: 0.17681 [I 2022-10-08 17:57:58] ...optimizers/trijoint.py.151: total_norm: 3.076109 [I 2022-10-08 17:58:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 1010/2304 [I 2022-10-08 17:58:06] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:16:46 [I 2022-10-08 17:58:06] ...trap/engines/engine.py.229: process: 0.77057 | load: 0.00302 [I 2022-10-08 17:58:06] ...trap/engines/engine.py.230: loss: 0.33589 [I 2022-10-08 17:58:06] ...optimizers/trijoint.py.151: total_norm: 4.246475 [I 2022-10-08 17:58:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 1020/2304 [I 2022-10-08 17:58:14] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:16:34 [I 2022-10-08 17:58:14] ...trap/engines/engine.py.229: process: 0.77199 | load: 0.00304 [I 2022-10-08 17:58:14] ...trap/engines/engine.py.230: loss: 0.14114 [I 2022-10-08 17:58:14] ...optimizers/trijoint.py.151: total_norm: 2.337464 [I 2022-10-08 17:58:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 1030/2304 [I 2022-10-08 17:58:22] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:34 [I 2022-10-08 17:58:22] ...trap/engines/engine.py.229: process: 0.78886 | load: 0.00305 [I 2022-10-08 17:58:22] ...trap/engines/engine.py.230: loss: 0.24066 [I 2022-10-08 17:58:22] ...optimizers/trijoint.py.151: total_norm: 4.755331 [I 2022-10-08 17:58:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1040/2304 [I 2022-10-08 17:58:29] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:29 [I 2022-10-08 17:58:29] ...trap/engines/engine.py.229: process: 0.77950 | load: 0.00303 [I 2022-10-08 17:58:29] ...trap/engines/engine.py.230: loss: 0.16056 [I 2022-10-08 17:58:29] ...optimizers/trijoint.py.151: total_norm: 2.283390 [I 2022-10-08 17:58:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1050/2304 [I 2022-10-08 17:58:37] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:16:16 [I 2022-10-08 17:58:37] ...trap/engines/engine.py.229: process: 0.77842 | load: 0.00302 [I 2022-10-08 17:58:37] ...trap/engines/engine.py.230: loss: 0.18847 [I 2022-10-08 17:58:37] ...optimizers/trijoint.py.151: total_norm: 2.456350 [I 2022-10-08 17:58:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 1060/2304 [I 2022-10-08 17:58:45] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:16:12 [I 2022-10-08 17:58:45] ...trap/engines/engine.py.229: process: 0.79907 | load: 0.00305 [I 2022-10-08 17:58:45] ...trap/engines/engine.py.230: loss: 0.15924 [I 2022-10-08 17:58:45] ...optimizers/trijoint.py.151: total_norm: 2.573115 [I 2022-10-08 17:58:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1070/2304 [I 2022-10-08 17:58:53] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:16:04 [I 2022-10-08 17:58:53] ...trap/engines/engine.py.229: process: 0.77157 | load: 0.00304 [I 2022-10-08 17:58:53] ...trap/engines/engine.py.230: loss: 0.15183 [I 2022-10-08 17:58:53] ...optimizers/trijoint.py.151: total_norm: 2.925817 [I 2022-10-08 17:59:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 1080/2304 [I 2022-10-08 17:59:01] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:50 [I 2022-10-08 17:59:01] ...trap/engines/engine.py.229: process: 0.76549 | load: 0.00307 [I 2022-10-08 17:59:01] ...trap/engines/engine.py.230: loss: 0.17769 [I 2022-10-08 17:59:01] ...optimizers/trijoint.py.151: total_norm: 2.267448 [I 2022-10-08 17:59:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1090/2304 [I 2022-10-08 17:59:08] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:44 [I 2022-10-08 17:59:08] ...trap/engines/engine.py.229: process: 0.78791 | load: 0.00304 [I 2022-10-08 17:59:08] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-08 17:59:08] ...optimizers/trijoint.py.151: total_norm: 2.127540 [I 2022-10-08 17:59:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 1100/2304 [I 2022-10-08 17:59:16] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:34 [I 2022-10-08 17:59:16] ...trap/engines/engine.py.229: process: 0.77798 | load: 0.00303 [I 2022-10-08 17:59:16] ...trap/engines/engine.py.230: loss: 0.24218 [I 2022-10-08 17:59:16] ...optimizers/trijoint.py.151: total_norm: 2.821774 [I 2022-10-08 17:59:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1110/2304 [I 2022-10-08 17:59:24] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:15:30 [I 2022-10-08 17:59:24] ...trap/engines/engine.py.229: process: 0.78709 | load: 0.00302 [I 2022-10-08 17:59:24] ...trap/engines/engine.py.230: loss: 0.21467 [I 2022-10-08 17:59:24] ...optimizers/trijoint.py.151: total_norm: 2.067273 [I 2022-10-08 17:59:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1120/2304 [I 2022-10-08 17:59:32] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:15:31 [I 2022-10-08 17:59:32] ...trap/engines/engine.py.229: process: 0.79289 | load: 0.00299 [I 2022-10-08 17:59:32] ...trap/engines/engine.py.230: loss: 0.18952 [I 2022-10-08 17:59:32] ...optimizers/trijoint.py.151: total_norm: 3.577332 [I 2022-10-08 17:59:40] ...trap/engines/engine.py.224: train: epoch 14 | batch 1130/2304 [I 2022-10-08 17:59:40] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:15:20 [I 2022-10-08 17:59:40] ...trap/engines/engine.py.229: process: 0.77252 | load: 0.00305 [I 2022-10-08 17:59:40] ...trap/engines/engine.py.230: loss: 0.21324 [I 2022-10-08 17:59:40] ...optimizers/trijoint.py.151: total_norm: 3.154531 [I 2022-10-08 17:59:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 1140/2304 [I 2022-10-08 17:59:48] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:11 [I 2022-10-08 17:59:48] ...trap/engines/engine.py.229: process: 0.78949 | load: 0.00301 [I 2022-10-08 17:59:48] ...trap/engines/engine.py.230: loss: 0.25362 [I 2022-10-08 17:59:48] ...optimizers/trijoint.py.151: total_norm: 3.157746 [I 2022-10-08 17:59:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 1150/2304 [I 2022-10-08 17:59:55] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:14:49 [I 2022-10-08 17:59:55] ...trap/engines/engine.py.229: process: 0.77563 | load: 0.00303 [I 2022-10-08 17:59:55] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-08 17:59:55] ...optimizers/trijoint.py.151: total_norm: 2.547748 [I 2022-10-08 18:00:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1160/2304 [I 2022-10-08 18:00:03] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:14:48 [I 2022-10-08 18:00:03] ...trap/engines/engine.py.229: process: 0.76801 | load: 0.00302 [I 2022-10-08 18:00:03] ...trap/engines/engine.py.230: loss: 0.17500 [I 2022-10-08 18:00:03] ...optimizers/trijoint.py.151: total_norm: 2.381076 [I 2022-10-08 18:00:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 1170/2304 [I 2022-10-08 18:00:11] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:14:44 [I 2022-10-08 18:00:11] ...trap/engines/engine.py.229: process: 0.78169 | load: 0.00300 [I 2022-10-08 18:00:11] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-08 18:00:11] ...optimizers/trijoint.py.151: total_norm: 1.688311 [I 2022-10-08 18:00:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 1180/2304 [I 2022-10-08 18:00:19] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:14:33 [I 2022-10-08 18:00:19] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00304 [I 2022-10-08 18:00:19] ...trap/engines/engine.py.230: loss: 0.16858 [I 2022-10-08 18:00:19] ...optimizers/trijoint.py.151: total_norm: 3.196045 [I 2022-10-08 18:00:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 1190/2304 [I 2022-10-08 18:00:27] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:14:25 [I 2022-10-08 18:00:27] ...trap/engines/engine.py.229: process: 0.77245 | load: 0.00304 [I 2022-10-08 18:00:27] ...trap/engines/engine.py.230: loss: 0.17364 [I 2022-10-08 18:00:27] ...optimizers/trijoint.py.151: total_norm: 2.447192 [I 2022-10-08 18:00:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1200/2304 [I 2022-10-08 18:00:34] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:15 [I 2022-10-08 18:00:34] ...trap/engines/engine.py.229: process: 0.77804 | load: 0.00303 [I 2022-10-08 18:00:34] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-08 18:00:34] ...optimizers/trijoint.py.151: total_norm: 2.459888 [I 2022-10-08 18:00:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1210/2304 [I 2022-10-08 18:00:42] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:14:09 [I 2022-10-08 18:00:42] ...trap/engines/engine.py.229: process: 0.76802 | load: 0.00304 [I 2022-10-08 18:00:42] ...trap/engines/engine.py.230: loss: 0.18794 [I 2022-10-08 18:00:42] ...optimizers/trijoint.py.151: total_norm: 2.309770 [I 2022-10-08 18:00:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 1220/2304 [I 2022-10-08 18:00:50] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:14:00 [I 2022-10-08 18:00:50] ...trap/engines/engine.py.229: process: 0.76784 | load: 0.00301 [I 2022-10-08 18:00:50] ...trap/engines/engine.py.230: loss: 0.29194 [I 2022-10-08 18:00:50] ...optimizers/trijoint.py.151: total_norm: 3.816870 [I 2022-10-08 18:00:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1230/2304 [I 2022-10-08 18:00:58] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:13:58 [I 2022-10-08 18:00:58] ...trap/engines/engine.py.229: process: 0.78018 | load: 0.00303 [I 2022-10-08 18:00:58] ...trap/engines/engine.py.230: loss: 0.14646 [I 2022-10-08 18:00:58] ...optimizers/trijoint.py.151: total_norm: 2.966966 [I 2022-10-08 18:01:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 1240/2304 [I 2022-10-08 18:01:06] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:13:54 [I 2022-10-08 18:01:06] ...trap/engines/engine.py.229: process: 0.78747 | load: 0.00298 [I 2022-10-08 18:01:06] ...trap/engines/engine.py.230: loss: 0.22156 [I 2022-10-08 18:01:06] ...optimizers/trijoint.py.151: total_norm: 2.777661 [I 2022-10-08 18:01:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1250/2304 [I 2022-10-08 18:01:13] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:13:47 [I 2022-10-08 18:01:13] ...trap/engines/engine.py.229: process: 0.80074 | load: 0.00304 [I 2022-10-08 18:01:13] ...trap/engines/engine.py.230: loss: 0.14323 [I 2022-10-08 18:01:13] ...optimizers/trijoint.py.151: total_norm: 2.277340 [I 2022-10-08 18:01:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 1260/2304 [I 2022-10-08 18:01:21] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:40 [I 2022-10-08 18:01:21] ...trap/engines/engine.py.229: process: 0.78985 | load: 0.00297 [I 2022-10-08 18:01:21] ...trap/engines/engine.py.230: loss: 0.15200 [I 2022-10-08 18:01:21] ...optimizers/trijoint.py.151: total_norm: 2.596889 [I 2022-10-08 18:01:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1270/2304 [I 2022-10-08 18:01:29] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:13:32 [I 2022-10-08 18:01:29] ...trap/engines/engine.py.229: process: 0.78119 | load: 0.00302 [I 2022-10-08 18:01:29] ...trap/engines/engine.py.230: loss: 0.24161 [I 2022-10-08 18:01:29] ...optimizers/trijoint.py.151: total_norm: 3.279397 [I 2022-10-08 18:01:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1280/2304 [I 2022-10-08 18:01:37] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:13:16 [I 2022-10-08 18:01:37] ...trap/engines/engine.py.229: process: 0.78256 | load: 0.00303 [I 2022-10-08 18:01:37] ...trap/engines/engine.py.230: loss: 0.20474 [I 2022-10-08 18:01:37] ...optimizers/trijoint.py.151: total_norm: 2.447368 [I 2022-10-08 18:01:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 1290/2304 [I 2022-10-08 18:01:45] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:13:10 [I 2022-10-08 18:01:45] ...trap/engines/engine.py.229: process: 0.78335 | load: 0.00308 [I 2022-10-08 18:01:45] ...trap/engines/engine.py.230: loss: 0.17572 [I 2022-10-08 18:01:45] ...optimizers/trijoint.py.151: total_norm: 2.742472 [I 2022-10-08 18:01:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1300/2304 [I 2022-10-08 18:01:53] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:12:58 [I 2022-10-08 18:01:53] ...trap/engines/engine.py.229: process: 0.77181 | load: 0.00302 [I 2022-10-08 18:01:53] ...trap/engines/engine.py.230: loss: 0.17685 [I 2022-10-08 18:01:53] ...optimizers/trijoint.py.151: total_norm: 2.069758 [I 2022-10-08 18:02:00] ...trap/engines/engine.py.224: train: epoch 14 | batch 1310/2304 [I 2022-10-08 18:02:00] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:53 [I 2022-10-08 18:02:00] ...trap/engines/engine.py.229: process: 0.77850 | load: 0.00299 [I 2022-10-08 18:02:00] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-08 18:02:00] ...optimizers/trijoint.py.151: total_norm: 2.553782 [I 2022-10-08 18:02:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1320/2304 [I 2022-10-08 18:02:08] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:12:45 [I 2022-10-08 18:02:08] ...trap/engines/engine.py.229: process: 0.78174 | load: 0.00299 [I 2022-10-08 18:02:08] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-08 18:02:08] ...optimizers/trijoint.py.151: total_norm: 1.855632 [I 2022-10-08 18:02:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 1330/2304 [I 2022-10-08 18:02:16] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:12:41 [I 2022-10-08 18:02:16] ...trap/engines/engine.py.229: process: 0.78220 | load: 0.00300 [I 2022-10-08 18:02:16] ...trap/engines/engine.py.230: loss: 0.20073 [I 2022-10-08 18:02:16] ...optimizers/trijoint.py.151: total_norm: 3.207974 [I 2022-10-08 18:02:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1340/2304 [I 2022-10-08 18:02:24] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:12:31 [I 2022-10-08 18:02:24] ...trap/engines/engine.py.229: process: 0.78253 | load: 0.00303 [I 2022-10-08 18:02:24] ...trap/engines/engine.py.230: loss: 0.15498 [I 2022-10-08 18:02:24] ...optimizers/trijoint.py.151: total_norm: 2.164055 [I 2022-10-08 18:02:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1350/2304 [I 2022-10-08 18:02:32] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:12:24 [I 2022-10-08 18:02:32] ...trap/engines/engine.py.229: process: 0.76941 | load: 0.00302 [I 2022-10-08 18:02:32] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-08 18:02:32] ...optimizers/trijoint.py.151: total_norm: 2.195783 [I 2022-10-08 18:02:40] ...trap/engines/engine.py.224: train: epoch 14 | batch 1360/2304 [I 2022-10-08 18:02:40] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:18 [I 2022-10-08 18:02:40] ...trap/engines/engine.py.229: process: 0.77931 | load: 0.00299 [I 2022-10-08 18:02:40] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-08 18:02:40] ...optimizers/trijoint.py.151: total_norm: 2.130634 [I 2022-10-08 18:02:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 1370/2304 [I 2022-10-08 18:02:47] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:07 [I 2022-10-08 18:02:47] ...trap/engines/engine.py.229: process: 0.78116 | load: 0.00326 [I 2022-10-08 18:02:47] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-08 18:02:47] ...optimizers/trijoint.py.151: total_norm: 2.756618 [I 2022-10-08 18:02:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 1380/2304 [I 2022-10-08 18:02:55] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:00 [I 2022-10-08 18:02:55] ...trap/engines/engine.py.229: process: 0.77169 | load: 0.00307 [I 2022-10-08 18:02:55] ...trap/engines/engine.py.230: loss: 0.27307 [I 2022-10-08 18:02:55] ...optimizers/trijoint.py.151: total_norm: 3.427336 [I 2022-10-08 18:03:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1390/2304 [I 2022-10-08 18:03:03] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:11:52 [I 2022-10-08 18:03:03] ...trap/engines/engine.py.229: process: 0.77997 | load: 0.00301 [I 2022-10-08 18:03:03] ...trap/engines/engine.py.230: loss: 0.24475 [I 2022-10-08 18:03:03] ...optimizers/trijoint.py.151: total_norm: 4.149771 [I 2022-10-08 18:03:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 1400/2304 [I 2022-10-08 18:03:11] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:11:44 [I 2022-10-08 18:03:11] ...trap/engines/engine.py.229: process: 0.78781 | load: 0.00305 [I 2022-10-08 18:03:11] ...trap/engines/engine.py.230: loss: 0.16746 [I 2022-10-08 18:03:11] ...optimizers/trijoint.py.151: total_norm: 2.425176 [I 2022-10-08 18:03:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 1410/2304 [I 2022-10-08 18:03:19] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:11:34 [I 2022-10-08 18:03:19] ...trap/engines/engine.py.229: process: 0.77984 | load: 0.00309 [I 2022-10-08 18:03:19] ...trap/engines/engine.py.230: loss: 0.20252 [I 2022-10-08 18:03:19] ...optimizers/trijoint.py.151: total_norm: 2.270645 [I 2022-10-08 18:03:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 1420/2304 [I 2022-10-08 18:03:26] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:31 [I 2022-10-08 18:03:26] ...trap/engines/engine.py.229: process: 0.78620 | load: 0.00303 [I 2022-10-08 18:03:26] ...trap/engines/engine.py.230: loss: 0.21137 [I 2022-10-08 18:03:26] ...optimizers/trijoint.py.151: total_norm: 2.450722 [I 2022-10-08 18:03:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1430/2304 [I 2022-10-08 18:03:34] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:21 [I 2022-10-08 18:03:34] ...trap/engines/engine.py.229: process: 0.77195 | load: 0.00304 [I 2022-10-08 18:03:34] ...trap/engines/engine.py.230: loss: 0.26050 [I 2022-10-08 18:03:34] ...optimizers/trijoint.py.151: total_norm: 3.611125 [I 2022-10-08 18:03:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1440/2304 [I 2022-10-08 18:03:42] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:13 [I 2022-10-08 18:03:42] ...trap/engines/engine.py.229: process: 0.78078 | load: 0.00301 [I 2022-10-08 18:03:42] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-08 18:03:42] ...optimizers/trijoint.py.151: total_norm: 2.742482 [I 2022-10-08 18:03:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 1450/2304 [I 2022-10-08 18:03:50] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:06 [I 2022-10-08 18:03:50] ...trap/engines/engine.py.229: process: 0.77157 | load: 0.00303 [I 2022-10-08 18:03:50] ...trap/engines/engine.py.230: loss: 0.24330 [I 2022-10-08 18:03:50] ...optimizers/trijoint.py.151: total_norm: 2.795777 [I 2022-10-08 18:03:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1460/2304 [I 2022-10-08 18:03:58] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:10:54 [I 2022-10-08 18:03:58] ...trap/engines/engine.py.229: process: 0.76848 | load: 0.00304 [I 2022-10-08 18:03:58] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-08 18:03:58] ...optimizers/trijoint.py.151: total_norm: 1.882867 [I 2022-10-08 18:04:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 1470/2304 [I 2022-10-08 18:04:06] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:10:50 [I 2022-10-08 18:04:06] ...trap/engines/engine.py.229: process: 0.78904 | load: 0.00301 [I 2022-10-08 18:04:06] ...trap/engines/engine.py.230: loss: 0.21715 [I 2022-10-08 18:04:06] ...optimizers/trijoint.py.151: total_norm: 2.839363 [I 2022-10-08 18:04:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1480/2304 [I 2022-10-08 18:04:13] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:35 [I 2022-10-08 18:04:13] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00301 [I 2022-10-08 18:04:13] ...trap/engines/engine.py.230: loss: 0.23480 [I 2022-10-08 18:04:13] ...optimizers/trijoint.py.151: total_norm: 2.705914 [I 2022-10-08 18:04:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 1490/2304 [I 2022-10-08 18:04:21] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:33 [I 2022-10-08 18:04:21] ...trap/engines/engine.py.229: process: 0.78307 | load: 0.00301 [I 2022-10-08 18:04:21] ...trap/engines/engine.py.230: loss: 0.15848 [I 2022-10-08 18:04:21] ...optimizers/trijoint.py.151: total_norm: 1.682707 [I 2022-10-08 18:04:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1500/2304 [I 2022-10-08 18:04:29] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:10:27 [I 2022-10-08 18:04:29] ...trap/engines/engine.py.229: process: 0.77477 | load: 0.00301 [I 2022-10-08 18:04:29] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-08 18:04:29] ...optimizers/trijoint.py.151: total_norm: 2.939593 [I 2022-10-08 18:04:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1510/2304 [I 2022-10-08 18:04:37] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:10:21 [I 2022-10-08 18:04:37] ...trap/engines/engine.py.229: process: 0.78994 | load: 0.00306 [I 2022-10-08 18:04:37] ...trap/engines/engine.py.230: loss: 0.16185 [I 2022-10-08 18:04:37] ...optimizers/trijoint.py.151: total_norm: 1.757663 [I 2022-10-08 18:04:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 1520/2304 [I 2022-10-08 18:04:45] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:10:10 [I 2022-10-08 18:04:45] ...trap/engines/engine.py.229: process: 0.78011 | load: 0.00303 [I 2022-10-08 18:04:45] ...trap/engines/engine.py.230: loss: 0.16672 [I 2022-10-08 18:04:45] ...optimizers/trijoint.py.151: total_norm: 2.338212 [I 2022-10-08 18:04:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1530/2304 [I 2022-10-08 18:04:53] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:04 [I 2022-10-08 18:04:53] ...trap/engines/engine.py.229: process: 0.78606 | load: 0.00297 [I 2022-10-08 18:04:53] ...trap/engines/engine.py.230: loss: 0.14985 [I 2022-10-08 18:04:53] ...optimizers/trijoint.py.151: total_norm: 2.481431 [I 2022-10-08 18:05:00] ...trap/engines/engine.py.224: train: epoch 14 | batch 1540/2304 [I 2022-10-08 18:05:00] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:55 [I 2022-10-08 18:05:00] ...trap/engines/engine.py.229: process: 0.77999 | load: 0.00302 [I 2022-10-08 18:05:00] ...trap/engines/engine.py.230: loss: 0.17323 [I 2022-10-08 18:05:00] ...optimizers/trijoint.py.151: total_norm: 2.139841 [I 2022-10-08 18:05:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1550/2304 [I 2022-10-08 18:05:08] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:46 [I 2022-10-08 18:05:08] ...trap/engines/engine.py.229: process: 0.78476 | load: 0.00300 [I 2022-10-08 18:05:08] ...trap/engines/engine.py.230: loss: 0.23794 [I 2022-10-08 18:05:08] ...optimizers/trijoint.py.151: total_norm: 2.209091 [I 2022-10-08 18:05:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 1560/2304 [I 2022-10-08 18:05:16] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:09:37 [I 2022-10-08 18:05:16] ...trap/engines/engine.py.229: process: 0.77024 | load: 0.00307 [I 2022-10-08 18:05:16] ...trap/engines/engine.py.230: loss: 0.18179 [I 2022-10-08 18:05:16] ...optimizers/trijoint.py.151: total_norm: 2.972575 [I 2022-10-08 18:05:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1570/2304 [I 2022-10-08 18:05:24] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:32 [I 2022-10-08 18:05:24] ...trap/engines/engine.py.229: process: 0.79375 | load: 0.00304 [I 2022-10-08 18:05:24] ...trap/engines/engine.py.230: loss: 0.14476 [I 2022-10-08 18:05:24] ...optimizers/trijoint.py.151: total_norm: 2.346114 [I 2022-10-08 18:05:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1580/2304 [I 2022-10-08 18:05:32] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:24 [I 2022-10-08 18:05:32] ...trap/engines/engine.py.229: process: 0.77978 | load: 0.00307 [I 2022-10-08 18:05:32] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-08 18:05:32] ...optimizers/trijoint.py.151: total_norm: 1.585029 [I 2022-10-08 18:05:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 1590/2304 [I 2022-10-08 18:05:39] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:14 [I 2022-10-08 18:05:39] ...trap/engines/engine.py.229: process: 0.76503 | load: 0.00305 [I 2022-10-08 18:05:39] ...trap/engines/engine.py.230: loss: 0.24646 [I 2022-10-08 18:05:39] ...optimizers/trijoint.py.151: total_norm: 3.162740 [I 2022-10-08 18:05:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 1600/2304 [I 2022-10-08 18:05:47] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:11 [I 2022-10-08 18:05:47] ...trap/engines/engine.py.229: process: 0.78734 | load: 0.00303 [I 2022-10-08 18:05:47] ...trap/engines/engine.py.230: loss: 0.21265 [I 2022-10-08 18:05:47] ...optimizers/trijoint.py.151: total_norm: 2.949946 [I 2022-10-08 18:05:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 1610/2304 [I 2022-10-08 18:05:55] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:09:02 [I 2022-10-08 18:05:55] ...trap/engines/engine.py.229: process: 0.77203 | load: 0.00304 [I 2022-10-08 18:05:55] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-08 18:05:55] ...optimizers/trijoint.py.151: total_norm: 1.856235 [I 2022-10-08 18:06:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1620/2304 [I 2022-10-08 18:06:03] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:55 [I 2022-10-08 18:06:03] ...trap/engines/engine.py.229: process: 0.78465 | load: 0.00301 [I 2022-10-08 18:06:03] ...trap/engines/engine.py.230: loss: 0.20225 [I 2022-10-08 18:06:03] ...optimizers/trijoint.py.151: total_norm: 2.911244 [I 2022-10-08 18:06:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 1630/2304 [I 2022-10-08 18:06:11] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:43 [I 2022-10-08 18:06:11] ...trap/engines/engine.py.229: process: 0.77065 | load: 0.00301 [I 2022-10-08 18:06:11] ...trap/engines/engine.py.230: loss: 0.15515 [I 2022-10-08 18:06:11] ...optimizers/trijoint.py.151: total_norm: 1.870332 [I 2022-10-08 18:06:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1640/2304 [I 2022-10-08 18:06:18] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:35 [I 2022-10-08 18:06:18] ...trap/engines/engine.py.229: process: 0.77031 | load: 0.00300 [I 2022-10-08 18:06:18] ...trap/engines/engine.py.230: loss: 0.24193 [I 2022-10-08 18:06:18] ...optimizers/trijoint.py.151: total_norm: 2.601706 [I 2022-10-08 18:06:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 1650/2304 [I 2022-10-08 18:06:26] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:30 [I 2022-10-08 18:06:26] ...trap/engines/engine.py.229: process: 0.79372 | load: 0.00302 [I 2022-10-08 18:06:26] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-08 18:06:26] ...optimizers/trijoint.py.151: total_norm: 2.299661 [I 2022-10-08 18:06:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1660/2304 [I 2022-10-08 18:06:34] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:08:22 [I 2022-10-08 18:06:34] ...trap/engines/engine.py.229: process: 0.78530 | load: 0.00306 [I 2022-10-08 18:06:34] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-08 18:06:34] ...optimizers/trijoint.py.151: total_norm: 2.152543 [I 2022-10-08 18:06:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1670/2304 [I 2022-10-08 18:06:42] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:08:17 [I 2022-10-08 18:06:42] ...trap/engines/engine.py.229: process: 0.79549 | load: 0.00305 [I 2022-10-08 18:06:42] ...trap/engines/engine.py.230: loss: 0.28201 [I 2022-10-08 18:06:42] ...optimizers/trijoint.py.151: total_norm: 3.825444 [I 2022-10-08 18:06:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 1680/2304 [I 2022-10-08 18:06:50] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:04 [I 2022-10-08 18:06:50] ...trap/engines/engine.py.229: process: 0.76377 | load: 0.00306 [I 2022-10-08 18:06:50] ...trap/engines/engine.py.230: loss: 0.19844 [I 2022-10-08 18:06:50] ...optimizers/trijoint.py.151: total_norm: 2.363597 [I 2022-10-08 18:06:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1690/2304 [I 2022-10-08 18:06:58] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:07:57 [I 2022-10-08 18:06:58] ...trap/engines/engine.py.229: process: 0.76798 | load: 0.00305 [I 2022-10-08 18:06:58] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-08 18:06:58] ...optimizers/trijoint.py.151: total_norm: 2.620834 [I 2022-10-08 18:07:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 1700/2304 [I 2022-10-08 18:07:05] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:50 [I 2022-10-08 18:07:05] ...trap/engines/engine.py.229: process: 0.78330 | load: 0.00308 [I 2022-10-08 18:07:05] ...trap/engines/engine.py.230: loss: 0.21957 [I 2022-10-08 18:07:05] ...optimizers/trijoint.py.151: total_norm: 3.876799 [I 2022-10-08 18:07:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1710/2304 [I 2022-10-08 18:07:13] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:45 [I 2022-10-08 18:07:13] ...trap/engines/engine.py.229: process: 0.77500 | load: 0.00307 [I 2022-10-08 18:07:13] ...trap/engines/engine.py.230: loss: 0.19844 [I 2022-10-08 18:07:13] ...optimizers/trijoint.py.151: total_norm: 2.577101 [I 2022-10-08 18:07:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 1720/2304 [I 2022-10-08 18:07:21] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:07:34 [I 2022-10-08 18:07:21] ...trap/engines/engine.py.229: process: 0.77764 | load: 0.00306 [I 2022-10-08 18:07:21] ...trap/engines/engine.py.230: loss: 0.19177 [I 2022-10-08 18:07:21] ...optimizers/trijoint.py.151: total_norm: 2.450401 [I 2022-10-08 18:07:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1730/2304 [I 2022-10-08 18:07:29] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:27 [I 2022-10-08 18:07:29] ...trap/engines/engine.py.229: process: 0.79198 | load: 0.00302 [I 2022-10-08 18:07:29] ...trap/engines/engine.py.230: loss: 0.19762 [I 2022-10-08 18:07:29] ...optimizers/trijoint.py.151: total_norm: 3.083063 [I 2022-10-08 18:07:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1740/2304 [I 2022-10-08 18:07:37] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:22 [I 2022-10-08 18:07:37] ...trap/engines/engine.py.229: process: 0.78593 | load: 0.00306 [I 2022-10-08 18:07:37] ...trap/engines/engine.py.230: loss: 0.19140 [I 2022-10-08 18:07:37] ...optimizers/trijoint.py.151: total_norm: 2.574535 [I 2022-10-08 18:07:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 1750/2304 [I 2022-10-08 18:07:44] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:11 [I 2022-10-08 18:07:44] ...trap/engines/engine.py.229: process: 0.77432 | load: 0.00306 [I 2022-10-08 18:07:44] ...trap/engines/engine.py.230: loss: 0.19063 [I 2022-10-08 18:07:44] ...optimizers/trijoint.py.151: total_norm: 2.566282 [I 2022-10-08 18:07:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 1760/2304 [I 2022-10-08 18:07:52] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:04 [I 2022-10-08 18:07:52] ...trap/engines/engine.py.229: process: 0.78689 | load: 0.00303 [I 2022-10-08 18:07:52] ...trap/engines/engine.py.230: loss: 0.19526 [I 2022-10-08 18:07:52] ...optimizers/trijoint.py.151: total_norm: 3.471032 [I 2022-10-08 18:08:00] ...trap/engines/engine.py.224: train: epoch 14 | batch 1770/2304 [I 2022-10-08 18:08:00] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:58 [I 2022-10-08 18:08:00] ...trap/engines/engine.py.229: process: 0.77764 | load: 0.00303 [I 2022-10-08 18:08:00] ...trap/engines/engine.py.230: loss: 0.22994 [I 2022-10-08 18:08:00] ...optimizers/trijoint.py.151: total_norm: 3.496091 [I 2022-10-08 18:08:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1780/2304 [I 2022-10-08 18:08:08] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:48 [I 2022-10-08 18:08:08] ...trap/engines/engine.py.229: process: 0.78398 | load: 0.00302 [I 2022-10-08 18:08:08] ...trap/engines/engine.py.230: loss: 0.22746 [I 2022-10-08 18:08:08] ...optimizers/trijoint.py.151: total_norm: 2.977923 [I 2022-10-08 18:08:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 1790/2304 [I 2022-10-08 18:08:16] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:40 [I 2022-10-08 18:08:16] ...trap/engines/engine.py.229: process: 0.77561 | load: 0.00304 [I 2022-10-08 18:08:16] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-08 18:08:16] ...optimizers/trijoint.py.151: total_norm: 2.670589 [I 2022-10-08 18:08:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 1800/2304 [I 2022-10-08 18:08:23] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:33 [I 2022-10-08 18:08:23] ...trap/engines/engine.py.229: process: 0.78452 | load: 0.00303 [I 2022-10-08 18:08:23] ...trap/engines/engine.py.230: loss: 0.21814 [I 2022-10-08 18:08:23] ...optimizers/trijoint.py.151: total_norm: 3.071403 [I 2022-10-08 18:08:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1810/2304 [I 2022-10-08 18:08:32] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:25 [I 2022-10-08 18:08:32] ...trap/engines/engine.py.229: process: 0.77857 | load: 0.00302 [I 2022-10-08 18:08:32] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-08 18:08:32] ...optimizers/trijoint.py.151: total_norm: 1.796936 [I 2022-10-08 18:08:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 1820/2304 [I 2022-10-08 18:08:39] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:06:17 [I 2022-10-08 18:08:39] ...trap/engines/engine.py.229: process: 0.79128 | load: 0.00303 [I 2022-10-08 18:08:39] ...trap/engines/engine.py.230: loss: 0.21036 [I 2022-10-08 18:08:39] ...optimizers/trijoint.py.151: total_norm: 3.047714 [I 2022-10-08 18:08:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 1830/2304 [I 2022-10-08 18:08:47] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:06:09 [I 2022-10-08 18:08:47] ...trap/engines/engine.py.229: process: 0.77919 | load: 0.00305 [I 2022-10-08 18:08:47] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-08 18:08:47] ...optimizers/trijoint.py.151: total_norm: 3.075204 [I 2022-10-08 18:08:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 1840/2304 [I 2022-10-08 18:08:55] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:02 [I 2022-10-08 18:08:55] ...trap/engines/engine.py.229: process: 0.79035 | load: 0.00303 [I 2022-10-08 18:08:55] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-08 18:08:55] ...optimizers/trijoint.py.151: total_norm: 3.092827 [I 2022-10-08 18:09:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1850/2304 [I 2022-10-08 18:09:03] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:05:54 [I 2022-10-08 18:09:03] ...trap/engines/engine.py.229: process: 0.77044 | load: 0.00307 [I 2022-10-08 18:09:03] ...trap/engines/engine.py.230: loss: 0.21897 [I 2022-10-08 18:09:03] ...optimizers/trijoint.py.151: total_norm: 3.943933 [I 2022-10-08 18:09:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 1860/2304 [I 2022-10-08 18:09:11] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:47 [I 2022-10-08 18:09:11] ...trap/engines/engine.py.229: process: 0.78049 | load: 0.00303 [I 2022-10-08 18:09:11] ...trap/engines/engine.py.230: loss: 0.18780 [I 2022-10-08 18:09:11] ...optimizers/trijoint.py.151: total_norm: 3.493685 [I 2022-10-08 18:09:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1870/2304 [I 2022-10-08 18:09:18] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:05:37 [I 2022-10-08 18:09:19] ...trap/engines/engine.py.229: process: 0.77131 | load: 0.00302 [I 2022-10-08 18:09:19] ...trap/engines/engine.py.230: loss: 0.19662 [I 2022-10-08 18:09:19] ...optimizers/trijoint.py.151: total_norm: 3.000074 [I 2022-10-08 18:09:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 1880/2304 [I 2022-10-08 18:09:26] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:05:29 [I 2022-10-08 18:09:26] ...trap/engines/engine.py.229: process: 0.77417 | load: 0.00300 [I 2022-10-08 18:09:26] ...trap/engines/engine.py.230: loss: 0.18813 [I 2022-10-08 18:09:26] ...optimizers/trijoint.py.151: total_norm: 2.514537 [I 2022-10-08 18:09:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1890/2304 [I 2022-10-08 18:09:34] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:05:22 [I 2022-10-08 18:09:34] ...trap/engines/engine.py.229: process: 0.78421 | load: 0.00302 [I 2022-10-08 18:09:34] ...trap/engines/engine.py.230: loss: 0.22428 [I 2022-10-08 18:09:34] ...optimizers/trijoint.py.151: total_norm: 3.345387 [I 2022-10-08 18:09:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1900/2304 [I 2022-10-08 18:09:42] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:14 [I 2022-10-08 18:09:42] ...trap/engines/engine.py.229: process: 0.77915 | load: 0.00303 [I 2022-10-08 18:09:42] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-08 18:09:42] ...optimizers/trijoint.py.151: total_norm: 2.116097 [I 2022-10-08 18:09:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 1910/2304 [I 2022-10-08 18:09:50] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:08 [I 2022-10-08 18:09:50] ...trap/engines/engine.py.229: process: 0.78849 | load: 0.00305 [I 2022-10-08 18:09:50] ...trap/engines/engine.py.230: loss: 0.36641 [I 2022-10-08 18:09:50] ...optimizers/trijoint.py.151: total_norm: 4.837497 [I 2022-10-08 18:09:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1920/2304 [I 2022-10-08 18:09:58] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:04:58 [I 2022-10-08 18:09:58] ...trap/engines/engine.py.229: process: 0.77915 | load: 0.00298 [I 2022-10-08 18:09:58] ...trap/engines/engine.py.230: loss: 0.24741 [I 2022-10-08 18:09:58] ...optimizers/trijoint.py.151: total_norm: 2.863718 [I 2022-10-08 18:10:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 1930/2304 [I 2022-10-08 18:10:05] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:04:51 [I 2022-10-08 18:10:05] ...trap/engines/engine.py.229: process: 0.77414 | load: 0.00296 [I 2022-10-08 18:10:05] ...trap/engines/engine.py.230: loss: 0.11976 [I 2022-10-08 18:10:05] ...optimizers/trijoint.py.151: total_norm: 1.350966 [I 2022-10-08 18:10:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1940/2304 [I 2022-10-08 18:10:13] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:04:42 [I 2022-10-08 18:10:13] ...trap/engines/engine.py.229: process: 0.76763 | load: 0.00318 [I 2022-10-08 18:10:13] ...trap/engines/engine.py.230: loss: 0.30195 [I 2022-10-08 18:10:13] ...optimizers/trijoint.py.151: total_norm: 2.358544 [I 2022-10-08 18:10:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 1950/2304 [I 2022-10-08 18:10:21] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:36 [I 2022-10-08 18:10:21] ...trap/engines/engine.py.229: process: 0.78341 | load: 0.00303 [I 2022-10-08 18:10:21] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-08 18:10:21] ...optimizers/trijoint.py.151: total_norm: 2.672463 [I 2022-10-08 18:10:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1960/2304 [I 2022-10-08 18:10:29] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:04:27 [I 2022-10-08 18:10:29] ...trap/engines/engine.py.229: process: 0.77771 | load: 0.00304 [I 2022-10-08 18:10:29] ...trap/engines/engine.py.230: loss: 0.18851 [I 2022-10-08 18:10:29] ...optimizers/trijoint.py.151: total_norm: 1.929703 [I 2022-10-08 18:10:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1970/2304 [I 2022-10-08 18:10:37] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:04:20 [I 2022-10-08 18:10:37] ...trap/engines/engine.py.229: process: 0.78909 | load: 0.00301 [I 2022-10-08 18:10:37] ...trap/engines/engine.py.230: loss: 0.12560 [I 2022-10-08 18:10:37] ...optimizers/trijoint.py.151: total_norm: 1.908660 [I 2022-10-08 18:10:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 1980/2304 [I 2022-10-08 18:10:44] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:04:11 [I 2022-10-08 18:10:44] ...trap/engines/engine.py.229: process: 0.77037 | load: 0.00308 [I 2022-10-08 18:10:44] ...trap/engines/engine.py.230: loss: 0.18926 [I 2022-10-08 18:10:44] ...optimizers/trijoint.py.151: total_norm: 2.610985 [I 2022-10-08 18:10:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 1990/2304 [I 2022-10-08 18:10:52] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:04:04 [I 2022-10-08 18:10:52] ...trap/engines/engine.py.229: process: 0.77880 | load: 0.00300 [I 2022-10-08 18:10:52] ...trap/engines/engine.py.230: loss: 0.15049 [I 2022-10-08 18:10:52] ...optimizers/trijoint.py.151: total_norm: 2.204999 [I 2022-10-08 18:11:00] ...trap/engines/engine.py.224: train: epoch 14 | batch 2000/2304 [I 2022-10-08 18:11:00] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:03:56 [I 2022-10-08 18:11:00] ...trap/engines/engine.py.229: process: 0.78324 | load: 0.00300 [I 2022-10-08 18:11:00] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-08 18:11:00] ...optimizers/trijoint.py.151: total_norm: 3.397426 [I 2022-10-08 18:11:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 2010/2304 [I 2022-10-08 18:11:08] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:03:49 [I 2022-10-08 18:11:08] ...trap/engines/engine.py.229: process: 0.78399 | load: 0.00302 [I 2022-10-08 18:11:08] ...trap/engines/engine.py.230: loss: 0.14697 [I 2022-10-08 18:11:08] ...optimizers/trijoint.py.151: total_norm: 2.991001 [I 2022-10-08 18:11:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 2020/2304 [I 2022-10-08 18:11:16] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:03:40 [I 2022-10-08 18:11:16] ...trap/engines/engine.py.229: process: 0.76773 | load: 0.00299 [I 2022-10-08 18:11:16] ...trap/engines/engine.py.230: loss: 0.13516 [I 2022-10-08 18:11:16] ...optimizers/trijoint.py.151: total_norm: 2.185178 [I 2022-10-08 18:11:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 2030/2304 [I 2022-10-08 18:11:23] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:32 [I 2022-10-08 18:11:23] ...trap/engines/engine.py.229: process: 0.78060 | load: 0.00298 [I 2022-10-08 18:11:23] ...trap/engines/engine.py.230: loss: 0.27860 [I 2022-10-08 18:11:23] ...optimizers/trijoint.py.151: total_norm: 4.014208 [I 2022-10-08 18:11:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 2040/2304 [I 2022-10-08 18:11:31] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:03:25 [I 2022-10-08 18:11:31] ...trap/engines/engine.py.229: process: 0.77757 | load: 0.00299 [I 2022-10-08 18:11:31] ...trap/engines/engine.py.230: loss: 0.17222 [I 2022-10-08 18:11:31] ...optimizers/trijoint.py.151: total_norm: 1.974014 [I 2022-10-08 18:11:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 2050/2304 [I 2022-10-08 18:11:39] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:03:18 [I 2022-10-08 18:11:39] ...trap/engines/engine.py.229: process: 0.78298 | load: 0.00304 [I 2022-10-08 18:11:39] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-08 18:11:39] ...optimizers/trijoint.py.151: total_norm: 2.139240 [I 2022-10-08 18:11:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 2060/2304 [I 2022-10-08 18:11:47] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:03:11 [I 2022-10-08 18:11:47] ...trap/engines/engine.py.229: process: 0.78417 | load: 0.00302 [I 2022-10-08 18:11:47] ...trap/engines/engine.py.230: loss: 0.20293 [I 2022-10-08 18:11:47] ...optimizers/trijoint.py.151: total_norm: 3.232614 [I 2022-10-08 18:11:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 2070/2304 [I 2022-10-08 18:11:55] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:03:02 [I 2022-10-08 18:11:55] ...trap/engines/engine.py.229: process: 0.77409 | load: 0.00296 [I 2022-10-08 18:11:55] ...trap/engines/engine.py.230: loss: 0.15790 [I 2022-10-08 18:11:55] ...optimizers/trijoint.py.151: total_norm: 2.331868 [I 2022-10-08 18:12:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 2080/2304 [I 2022-10-08 18:12:02] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:02:53 [I 2022-10-08 18:12:02] ...trap/engines/engine.py.229: process: 0.77005 | load: 0.00298 [I 2022-10-08 18:12:02] ...trap/engines/engine.py.230: loss: 0.22408 [I 2022-10-08 18:12:02] ...optimizers/trijoint.py.151: total_norm: 2.722262 [I 2022-10-08 18:12:10] ...trap/engines/engine.py.224: train: epoch 14 | batch 2090/2304 [I 2022-10-08 18:12:10] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:02:47 [I 2022-10-08 18:12:10] ...trap/engines/engine.py.229: process: 0.78334 | load: 0.00303 [I 2022-10-08 18:12:10] ...trap/engines/engine.py.230: loss: 0.17529 [I 2022-10-08 18:12:10] ...optimizers/trijoint.py.151: total_norm: 2.570799 [I 2022-10-08 18:12:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 2100/2304 [I 2022-10-08 18:12:18] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:38 [I 2022-10-08 18:12:18] ...trap/engines/engine.py.229: process: 0.76924 | load: 0.00303 [I 2022-10-08 18:12:18] ...trap/engines/engine.py.230: loss: 0.29104 [I 2022-10-08 18:12:18] ...optimizers/trijoint.py.151: total_norm: 2.996598 [I 2022-10-08 18:12:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 2110/2304 [I 2022-10-08 18:12:26] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:30 [I 2022-10-08 18:12:26] ...trap/engines/engine.py.229: process: 0.78087 | load: 0.00304 [I 2022-10-08 18:12:26] ...trap/engines/engine.py.230: loss: 0.18694 [I 2022-10-08 18:12:26] ...optimizers/trijoint.py.151: total_norm: 3.211435 [I 2022-10-08 18:12:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 2120/2304 [I 2022-10-08 18:12:34] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:02:23 [I 2022-10-08 18:12:34] ...trap/engines/engine.py.229: process: 0.77293 | load: 0.00297 [I 2022-10-08 18:12:34] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-08 18:12:34] ...optimizers/trijoint.py.151: total_norm: 2.367417 [I 2022-10-08 18:12:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 2130/2304 [I 2022-10-08 18:12:41] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:02:14 [I 2022-10-08 18:12:41] ...trap/engines/engine.py.229: process: 0.77285 | load: 0.00303 [I 2022-10-08 18:12:41] ...trap/engines/engine.py.230: loss: 0.23230 [I 2022-10-08 18:12:41] ...optimizers/trijoint.py.151: total_norm: 3.287160 [I 2022-10-08 18:12:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 2140/2304 [I 2022-10-08 18:12:49] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:06 [I 2022-10-08 18:12:49] ...trap/engines/engine.py.229: process: 0.76685 | load: 0.00307 [I 2022-10-08 18:12:49] ...trap/engines/engine.py.230: loss: 0.23009 [I 2022-10-08 18:12:49] ...optimizers/trijoint.py.151: total_norm: 2.581086 [I 2022-10-08 18:12:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 2150/2304 [I 2022-10-08 18:12:57] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:02:00 [I 2022-10-08 18:12:57] ...trap/engines/engine.py.229: process: 0.75989 | load: 0.00304 [I 2022-10-08 18:12:57] ...trap/engines/engine.py.230: loss: 0.29924 [I 2022-10-08 18:12:57] ...optimizers/trijoint.py.151: total_norm: 2.941449 [I 2022-10-08 18:13:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 2160/2304 [I 2022-10-08 18:13:05] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:01:52 [I 2022-10-08 18:13:05] ...trap/engines/engine.py.229: process: 0.77909 | load: 0.00314 [I 2022-10-08 18:13:05] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-08 18:13:05] ...optimizers/trijoint.py.151: total_norm: 3.224667 [I 2022-10-08 18:13:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 2170/2304 [I 2022-10-08 18:13:13] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:01:44 [I 2022-10-08 18:13:13] ...trap/engines/engine.py.229: process: 0.77881 | load: 0.00315 [I 2022-10-08 18:13:13] ...trap/engines/engine.py.230: loss: 0.17299 [I 2022-10-08 18:13:13] ...optimizers/trijoint.py.151: total_norm: 2.329932 [I 2022-10-08 18:13:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 2180/2304 [I 2022-10-08 18:13:21] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:01:36 [I 2022-10-08 18:13:21] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00312 [I 2022-10-08 18:13:21] ...trap/engines/engine.py.230: loss: 0.12043 [I 2022-10-08 18:13:21] ...optimizers/trijoint.py.151: total_norm: 1.794780 [I 2022-10-08 18:13:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 2190/2304 [I 2022-10-08 18:13:28] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:01:28 [I 2022-10-08 18:13:28] ...trap/engines/engine.py.229: process: 0.77629 | load: 0.00303 [I 2022-10-08 18:13:28] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-08 18:13:28] ...optimizers/trijoint.py.151: total_norm: 1.870641 [I 2022-10-08 18:13:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 2200/2304 [I 2022-10-08 18:13:36] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:01:21 [I 2022-10-08 18:13:36] ...trap/engines/engine.py.229: process: 0.78409 | load: 0.00301 [I 2022-10-08 18:13:36] ...trap/engines/engine.py.230: loss: 0.17037 [I 2022-10-08 18:13:36] ...optimizers/trijoint.py.151: total_norm: 1.888936 [I 2022-10-08 18:13:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 2210/2304 [I 2022-10-08 18:13:44] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:01:12 [I 2022-10-08 18:13:44] ...trap/engines/engine.py.229: process: 0.77275 | load: 0.00302 [I 2022-10-08 18:13:44] ...trap/engines/engine.py.230: loss: 0.14898 [I 2022-10-08 18:13:44] ...optimizers/trijoint.py.151: total_norm: 2.445433 [I 2022-10-08 18:13:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 2220/2304 [I 2022-10-08 18:13:52] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:01:05 [I 2022-10-08 18:13:52] ...trap/engines/engine.py.229: process: 0.77827 | load: 0.00313 [I 2022-10-08 18:13:52] ...trap/engines/engine.py.230: loss: 0.21272 [I 2022-10-08 18:13:52] ...optimizers/trijoint.py.151: total_norm: 3.105983 [I 2022-10-08 18:13:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 2230/2304 [I 2022-10-08 18:13:59] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:00:57 [I 2022-10-08 18:13:59] ...trap/engines/engine.py.229: process: 0.78261 | load: 0.00309 [I 2022-10-08 18:13:59] ...trap/engines/engine.py.230: loss: 0.32947 [I 2022-10-08 18:13:59] ...optimizers/trijoint.py.151: total_norm: 3.469443 [I 2022-10-08 18:14:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 2240/2304 [I 2022-10-08 18:14:07] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:00:49 [I 2022-10-08 18:14:07] ...trap/engines/engine.py.229: process: 0.77571 | load: 0.00305 [I 2022-10-08 18:14:07] ...trap/engines/engine.py.230: loss: 0.21216 [I 2022-10-08 18:14:07] ...optimizers/trijoint.py.151: total_norm: 2.550871 [I 2022-10-08 18:14:15] ...trap/engines/engine.py.224: train: epoch 14 | batch 2250/2304 [I 2022-10-08 18:14:15] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:00:42 [I 2022-10-08 18:14:15] ...trap/engines/engine.py.229: process: 0.77250 | load: 0.00306 [I 2022-10-08 18:14:15] ...trap/engines/engine.py.230: loss: 0.21544 [I 2022-10-08 18:14:15] ...optimizers/trijoint.py.151: total_norm: 3.620220 [I 2022-10-08 18:14:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 2260/2304 [I 2022-10-08 18:14:23] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:00:34 [I 2022-10-08 18:14:23] ...trap/engines/engine.py.229: process: 0.77254 | load: 0.00302 [I 2022-10-08 18:14:23] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-08 18:14:23] ...optimizers/trijoint.py.151: total_norm: 2.786812 [I 2022-10-08 18:14:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 2270/2304 [I 2022-10-08 18:14:31] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:00:26 [I 2022-10-08 18:14:31] ...trap/engines/engine.py.229: process: 0.77173 | load: 0.00300 [I 2022-10-08 18:14:31] ...trap/engines/engine.py.230: loss: 0.14513 [I 2022-10-08 18:14:31] ...optimizers/trijoint.py.151: total_norm: 2.709308 [I 2022-10-08 18:14:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 2280/2304 [I 2022-10-08 18:14:39] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:00:18 [I 2022-10-08 18:14:39] ...trap/engines/engine.py.229: process: 0.77850 | load: 0.00302 [I 2022-10-08 18:14:39] ...trap/engines/engine.py.230: loss: 0.18028 [I 2022-10-08 18:14:39] ...optimizers/trijoint.py.151: total_norm: 3.522286 [I 2022-10-08 18:14:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 2290/2304 [I 2022-10-08 18:14:47] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:11 [I 2022-10-08 18:14:47] ...trap/engines/engine.py.229: process: 0.80180 | load: 0.00300 [I 2022-10-08 18:14:47] ...trap/engines/engine.py.230: loss: 0.27956 [I 2022-10-08 18:14:47] ...optimizers/trijoint.py.151: total_norm: 3.973264 [I 2022-10-08 18:14:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 2300/2304 [I 2022-10-08 18:14:54] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:00:03 [I 2022-10-08 18:14:54] ...trap/engines/engine.py.229: process: 0.76804 | load: 0.00017 [I 2022-10-08 18:14:54] ...trap/engines/engine.py.230: loss: 0.16501 [I 2022-10-08 18:14:54] ...optimizers/trijoint.py.151: total_norm: 2.342380 [I 2022-10-08 18:14:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 2304/2304 [I 2022-10-08 18:14:57] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:00:00 [I 2022-10-08 18:14:57] ...trap/engines/engine.py.229: process: 0.76543 | load: 0.00017 [I 2022-10-08 18:14:57] ...trap/engines/engine.py.230: loss: 0.22142 [I 2022-10-08 18:14:57] ...optimizers/trijoint.py.151: total_norm: 3.029486 [S 2022-10-08 18:14:59] ...trap/engines/engine.py.129: train_epoch.epoch: 14 [S 2022-10-08 18:14:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.18850557818976504 [I 2022-10-08 18:14:59] ...s/criterions/triplet.py.84: margin = 0.12500000000000006 [I 2022-10-08 18:14:59] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 14 [I 2022-10-08 18:14:59] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 18:15:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 18:15:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 18:15:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 18:15:01] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 18:15:02] ...trap/engines/engine.py.321: eval: epoch 14 | batch 0/510 [I 2022-10-08 18:15:02] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:13 [I 2022-10-08 18:15:02] ...trap/engines/engine.py.326: process: 2.73298 | load: 2.25644 [I 2022-10-08 18:15:08] ...trap/engines/engine.py.321: eval: epoch 14 | batch 10/510 [I 2022-10-08 18:15:08] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:03 [I 2022-10-08 18:15:08] ...trap/engines/engine.py.326: process: 0.66638 | load: 0.00018 [I 2022-10-08 18:15:15] ...trap/engines/engine.py.321: eval: epoch 14 | batch 20/510 [I 2022-10-08 18:15:15] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:42 [I 2022-10-08 18:15:15] ...trap/engines/engine.py.326: process: 0.70907 | load: 0.00020 [I 2022-10-08 18:15:20] ...trap/engines/engine.py.321: eval: epoch 14 | batch 30/510 [I 2022-10-08 18:15:20] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:08 [I 2022-10-08 18:15:20] ...trap/engines/engine.py.326: process: 0.41870 | load: 0.00018 [I 2022-10-08 18:15:25] ...trap/engines/engine.py.321: eval: epoch 14 | batch 40/510 [I 2022-10-08 18:15:25] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:05 [I 2022-10-08 18:15:25] ...trap/engines/engine.py.326: process: 0.42976 | load: 0.00019 [I 2022-10-08 18:15:30] ...trap/engines/engine.py.321: eval: epoch 14 | batch 50/510 [I 2022-10-08 18:15:30] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:53 [I 2022-10-08 18:15:30] ...trap/engines/engine.py.326: process: 0.59148 | load: 0.00018 [I 2022-10-08 18:15:35] ...trap/engines/engine.py.321: eval: epoch 14 | batch 60/510 [I 2022-10-08 18:15:35] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:39 [I 2022-10-08 18:15:35] ...trap/engines/engine.py.326: process: 0.42506 | load: 0.00019 [I 2022-10-08 18:15:40] ...trap/engines/engine.py.321: eval: epoch 14 | batch 70/510 [I 2022-10-08 18:15:40] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:18 [I 2022-10-08 18:15:40] ...trap/engines/engine.py.326: process: 0.43305 | load: 0.00018 [I 2022-10-08 18:15:44] ...trap/engines/engine.py.321: eval: epoch 14 | batch 80/510 [I 2022-10-08 18:15:44] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:26 [I 2022-10-08 18:15:44] ...trap/engines/engine.py.326: process: 0.45158 | load: 0.00019 [I 2022-10-08 18:15:49] ...trap/engines/engine.py.321: eval: epoch 14 | batch 90/510 [I 2022-10-08 18:15:49] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:03 [I 2022-10-08 18:15:49] ...trap/engines/engine.py.326: process: 0.44632 | load: 0.00019 [I 2022-10-08 18:15:54] ...trap/engines/engine.py.321: eval: epoch 14 | batch 100/510 [I 2022-10-08 18:15:54] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:04 [I 2022-10-08 18:15:54] ...trap/engines/engine.py.326: process: 0.30503 | load: 0.00018 [I 2022-10-08 18:15:57] ...trap/engines/engine.py.321: eval: epoch 14 | batch 110/510 [I 2022-10-08 18:15:57] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:11 [I 2022-10-08 18:15:57] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00019 [I 2022-10-08 18:16:00] ...trap/engines/engine.py.321: eval: epoch 14 | batch 120/510 [I 2022-10-08 18:16:00] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-08 18:16:00] ...trap/engines/engine.py.326: process: 0.31604 | load: 0.00019 [I 2022-10-08 18:16:03] ...trap/engines/engine.py.321: eval: epoch 14 | batch 130/510 [I 2022-10-08 18:16:03] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-08 18:16:03] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00022 [I 2022-10-08 18:16:06] ...trap/engines/engine.py.321: eval: epoch 14 | batch 140/510 [I 2022-10-08 18:16:06] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-08 18:16:06] ...trap/engines/engine.py.326: process: 0.32000 | load: 0.00019 [I 2022-10-08 18:16:09] ...trap/engines/engine.py.321: eval: epoch 14 | batch 150/510 [I 2022-10-08 18:16:09] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-08 18:16:09] ...trap/engines/engine.py.326: process: 0.32042 | load: 0.00019 [I 2022-10-08 18:16:13] ...trap/engines/engine.py.321: eval: epoch 14 | batch 160/510 [I 2022-10-08 18:16:13] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-08 18:16:13] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00019 [I 2022-10-08 18:16:16] ...trap/engines/engine.py.321: eval: epoch 14 | batch 170/510 [I 2022-10-08 18:16:16] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:48 [I 2022-10-08 18:16:16] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00019 [I 2022-10-08 18:16:19] ...trap/engines/engine.py.321: eval: epoch 14 | batch 180/510 [I 2022-10-08 18:16:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-08 18:16:19] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00019 [I 2022-10-08 18:16:22] ...trap/engines/engine.py.321: eval: epoch 14 | batch 190/510 [I 2022-10-08 18:16:22] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-08 18:16:22] ...trap/engines/engine.py.326: process: 0.31490 | load: 0.00019 [I 2022-10-08 18:16:25] ...trap/engines/engine.py.321: eval: epoch 14 | batch 200/510 [I 2022-10-08 18:16:25] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:38 [I 2022-10-08 18:16:25] ...trap/engines/engine.py.326: process: 0.31298 | load: 0.00020 [I 2022-10-08 18:16:28] ...trap/engines/engine.py.321: eval: epoch 14 | batch 210/510 [I 2022-10-08 18:16:28] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-08 18:16:28] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00020 [I 2022-10-08 18:16:32] ...trap/engines/engine.py.321: eval: epoch 14 | batch 220/510 [I 2022-10-08 18:16:32] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:31 [I 2022-10-08 18:16:32] ...trap/engines/engine.py.326: process: 0.30925 | load: 0.00019 [I 2022-10-08 18:16:35] ...trap/engines/engine.py.321: eval: epoch 14 | batch 230/510 [I 2022-10-08 18:16:35] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:29 [I 2022-10-08 18:16:35] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00020 [I 2022-10-08 18:16:38] ...trap/engines/engine.py.321: eval: epoch 14 | batch 240/510 [I 2022-10-08 18:16:38] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-08 18:16:38] ...trap/engines/engine.py.326: process: 0.31857 | load: 0.00019 [I 2022-10-08 18:16:41] ...trap/engines/engine.py.321: eval: epoch 14 | batch 250/510 [I 2022-10-08 18:16:41] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-08 18:16:41] ...trap/engines/engine.py.326: process: 0.32518 | load: 0.00020 [I 2022-10-08 18:16:44] ...trap/engines/engine.py.321: eval: epoch 14 | batch 260/510 [I 2022-10-08 18:16:44] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:19 [I 2022-10-08 18:16:44] ...trap/engines/engine.py.326: process: 0.31855 | load: 0.00020 [I 2022-10-08 18:16:48] ...trap/engines/engine.py.321: eval: epoch 14 | batch 270/510 [I 2022-10-08 18:16:48] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-08 18:16:48] ...trap/engines/engine.py.326: process: 0.32188 | load: 0.00021 [I 2022-10-08 18:16:51] ...trap/engines/engine.py.321: eval: epoch 14 | batch 280/510 [I 2022-10-08 18:16:51] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:12 [I 2022-10-08 18:16:51] ...trap/engines/engine.py.326: process: 0.31215 | load: 0.00020 [I 2022-10-08 18:16:54] ...trap/engines/engine.py.321: eval: epoch 14 | batch 290/510 [I 2022-10-08 18:16:54] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-08 18:16:54] ...trap/engines/engine.py.326: process: 0.32074 | load: 0.00019 [I 2022-10-08 18:16:57] ...trap/engines/engine.py.321: eval: epoch 14 | batch 300/510 [I 2022-10-08 18:16:57] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-08 18:16:57] ...trap/engines/engine.py.326: process: 0.31472 | load: 0.00019 [I 2022-10-08 18:17:00] ...trap/engines/engine.py.321: eval: epoch 14 | batch 310/510 [I 2022-10-08 18:17:00] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:03 [I 2022-10-08 18:17:00] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00020 [I 2022-10-08 18:17:03] ...trap/engines/engine.py.321: eval: epoch 14 | batch 320/510 [I 2022-10-08 18:17:03] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-08 18:17:03] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00020 [I 2022-10-08 18:17:07] ...trap/engines/engine.py.321: eval: epoch 14 | batch 330/510 [I 2022-10-08 18:17:07] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-08 18:17:07] ...trap/engines/engine.py.326: process: 0.31879 | load: 0.00020 [I 2022-10-08 18:17:10] ...trap/engines/engine.py.321: eval: epoch 14 | batch 340/510 [I 2022-10-08 18:17:10] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:53 [I 2022-10-08 18:17:10] ...trap/engines/engine.py.326: process: 0.31743 | load: 0.00021 [I 2022-10-08 18:17:13] ...trap/engines/engine.py.321: eval: epoch 14 | batch 350/510 [I 2022-10-08 18:17:13] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-08 18:17:13] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00021 [I 2022-10-08 18:17:16] ...trap/engines/engine.py.321: eval: epoch 14 | batch 360/510 [I 2022-10-08 18:17:16] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-08 18:17:16] ...trap/engines/engine.py.326: process: 0.31647 | load: 0.00021 [I 2022-10-08 18:17:19] ...trap/engines/engine.py.321: eval: epoch 14 | batch 370/510 [I 2022-10-08 18:17:19] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-08 18:17:19] ...trap/engines/engine.py.326: process: 0.32069 | load: 0.00020 [I 2022-10-08 18:17:23] ...trap/engines/engine.py.321: eval: epoch 14 | batch 380/510 [I 2022-10-08 18:17:23] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-08 18:17:23] ...trap/engines/engine.py.326: process: 0.32192 | load: 0.00019 [I 2022-10-08 18:17:26] ...trap/engines/engine.py.321: eval: epoch 14 | batch 390/510 [I 2022-10-08 18:17:26] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-08 18:17:26] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00021 [I 2022-10-08 18:17:29] ...trap/engines/engine.py.321: eval: epoch 14 | batch 400/510 [I 2022-10-08 18:17:29] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:34 [I 2022-10-08 18:17:29] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00019 [I 2022-10-08 18:17:32] ...trap/engines/engine.py.321: eval: epoch 14 | batch 410/510 [I 2022-10-08 18:17:32] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-08 18:17:32] ...trap/engines/engine.py.326: process: 0.30861 | load: 0.00022 [I 2022-10-08 18:17:35] ...trap/engines/engine.py.321: eval: epoch 14 | batch 420/510 [I 2022-10-08 18:17:35] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-08 18:17:35] ...trap/engines/engine.py.326: process: 0.32244 | load: 0.00020 [I 2022-10-08 18:17:39] ...trap/engines/engine.py.321: eval: epoch 14 | batch 430/510 [I 2022-10-08 18:17:39] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-08 18:17:39] ...trap/engines/engine.py.326: process: 0.32067 | load: 0.00020 [I 2022-10-08 18:17:42] ...trap/engines/engine.py.321: eval: epoch 14 | batch 440/510 [I 2022-10-08 18:17:42] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-08 18:17:42] ...trap/engines/engine.py.326: process: 0.30938 | load: 0.00020 [I 2022-10-08 18:17:45] ...trap/engines/engine.py.321: eval: epoch 14 | batch 450/510 [I 2022-10-08 18:17:45] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-08 18:17:45] ...trap/engines/engine.py.326: process: 0.32459 | load: 0.00020 [I 2022-10-08 18:17:48] ...trap/engines/engine.py.321: eval: epoch 14 | batch 460/510 [I 2022-10-08 18:17:48] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-08 18:17:48] ...trap/engines/engine.py.326: process: 0.31746 | load: 0.00021 [I 2022-10-08 18:17:51] ...trap/engines/engine.py.321: eval: epoch 14 | batch 470/510 [I 2022-10-08 18:17:51] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-08 18:17:51] ...trap/engines/engine.py.326: process: 0.31964 | load: 0.00023 [I 2022-10-08 18:17:55] ...trap/engines/engine.py.321: eval: epoch 14 | batch 480/510 [I 2022-10-08 18:17:55] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-08 18:17:55] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00020 [I 2022-10-08 18:17:58] ...trap/engines/engine.py.321: eval: epoch 14 | batch 490/510 [I 2022-10-08 18:17:58] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-08 18:17:58] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00021 [I 2022-10-08 18:18:01] ...trap/engines/engine.py.321: eval: epoch 14 | batch 500/510 [I 2022-10-08 18:18:01] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-08 18:18:01] ...trap/engines/engine.py.326: process: 0.32399 | load: 0.00020 [I 2022-10-08 18:18:04] ...trap/engines/engine.py.321: eval: epoch 14 | batch 510/510 [I 2022-10-08 18:18:04] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-08 18:18:04] ...trap/engines/engine.py.326: process: 0.31792 | load: 0.00016 [S 2022-10-08 18:18:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 14 [I 2022-10-08 18:18:05] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.596100 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.851400 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.905700 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.603700 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.856200 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.907100 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010977 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010404 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011455 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016106 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009485 [S 2022-10-08 18:18:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009669 [I 2022-10-08 18:18:37] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 18:18:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 18:18:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 18:18:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 18:18:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 18:18:37] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 18:18:40] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 18:18:45] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 18:18:45] ...trap/engines/engine.py.163: Training model on trainset for epoch 15 [I 2022-10-08 18:18:46] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 18:18:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 0/2304 [I 2022-10-08 18:18:48] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:53:08 [I 2022-10-08 18:18:48] ...trap/engines/engine.py.229: process: 2.94630 | load: 2.12290 [I 2022-10-08 18:18:48] ...trap/engines/engine.py.230: loss: 0.20709 [I 2022-10-08 18:18:48] ...optimizers/trijoint.py.151: total_norm: 2.528886 [I 2022-10-08 18:18:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 10/2304 [I 2022-10-08 18:18:56] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:55 [I 2022-10-08 18:18:56] ...trap/engines/engine.py.229: process: 0.78739 | load: 0.00539 [I 2022-10-08 18:18:56] ...trap/engines/engine.py.230: loss: 0.17192 [I 2022-10-08 18:18:56] ...optimizers/trijoint.py.151: total_norm: 1.866203 [I 2022-10-08 18:19:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 20/2304 [I 2022-10-08 18:19:04] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:36 [I 2022-10-08 18:19:04] ...trap/engines/engine.py.229: process: 0.78380 | load: 0.00468 [I 2022-10-08 18:19:04] ...trap/engines/engine.py.230: loss: 0.20156 [I 2022-10-08 18:19:04] ...optimizers/trijoint.py.151: total_norm: 3.496194 [I 2022-10-08 18:19:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 30/2304 [I 2022-10-08 18:19:12] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:40 [I 2022-10-08 18:19:12] ...trap/engines/engine.py.229: process: 0.78464 | load: 0.00463 [I 2022-10-08 18:19:12] ...trap/engines/engine.py.230: loss: 0.21982 [I 2022-10-08 18:19:12] ...optimizers/trijoint.py.151: total_norm: 2.368475 [I 2022-10-08 18:19:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 40/2304 [I 2022-10-08 18:19:20] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:23 [I 2022-10-08 18:19:20] ...trap/engines/engine.py.229: process: 0.78240 | load: 0.00404 [I 2022-10-08 18:19:20] ...trap/engines/engine.py.230: loss: 0.12204 [I 2022-10-08 18:19:20] ...optimizers/trijoint.py.151: total_norm: 2.098119 [I 2022-10-08 18:19:28] ...trap/engines/engine.py.224: train: epoch 15 | batch 50/2304 [I 2022-10-08 18:19:28] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:24 [I 2022-10-08 18:19:28] ...trap/engines/engine.py.229: process: 0.77943 | load: 0.00434 [I 2022-10-08 18:19:28] ...trap/engines/engine.py.230: loss: 0.19331 [I 2022-10-08 18:19:28] ...optimizers/trijoint.py.151: total_norm: 2.446824 [I 2022-10-08 18:19:35] ...trap/engines/engine.py.224: train: epoch 15 | batch 60/2304 [I 2022-10-08 18:19:35] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:06 [I 2022-10-08 18:19:35] ...trap/engines/engine.py.229: process: 0.77936 | load: 0.00361 [I 2022-10-08 18:19:35] ...trap/engines/engine.py.230: loss: 0.19644 [I 2022-10-08 18:19:35] ...optimizers/trijoint.py.151: total_norm: 2.543439 [I 2022-10-08 18:19:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 70/2304 [I 2022-10-08 18:19:43] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:56 [I 2022-10-08 18:19:43] ...trap/engines/engine.py.229: process: 0.77133 | load: 0.00391 [I 2022-10-08 18:19:43] ...trap/engines/engine.py.230: loss: 0.20417 [I 2022-10-08 18:19:43] ...optimizers/trijoint.py.151: total_norm: 2.911867 [I 2022-10-08 18:19:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 80/2304 [I 2022-10-08 18:19:51] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:29:05 [I 2022-10-08 18:19:51] ...trap/engines/engine.py.229: process: 0.78338 | load: 0.00363 [I 2022-10-08 18:19:51] ...trap/engines/engine.py.230: loss: 0.15258 [I 2022-10-08 18:19:51] ...optimizers/trijoint.py.151: total_norm: 2.900140 [I 2022-10-08 18:19:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 90/2304 [I 2022-10-08 18:19:59] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:49 [I 2022-10-08 18:19:59] ...trap/engines/engine.py.229: process: 0.79208 | load: 0.00367 [I 2022-10-08 18:19:59] ...trap/engines/engine.py.230: loss: 0.21929 [I 2022-10-08 18:19:59] ...optimizers/trijoint.py.151: total_norm: 3.620977 [I 2022-10-08 18:20:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 100/2304 [I 2022-10-08 18:20:07] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:41 [I 2022-10-08 18:20:07] ...trap/engines/engine.py.229: process: 0.77509 | load: 0.00361 [I 2022-10-08 18:20:07] ...trap/engines/engine.py.230: loss: 0.26142 [I 2022-10-08 18:20:07] ...optimizers/trijoint.py.151: total_norm: 3.933200 [I 2022-10-08 18:20:15] ...trap/engines/engine.py.224: train: epoch 15 | batch 110/2304 [I 2022-10-08 18:20:15] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:41 [I 2022-10-08 18:20:15] ...trap/engines/engine.py.229: process: 0.79053 | load: 0.00365 [I 2022-10-08 18:20:15] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-08 18:20:15] ...optimizers/trijoint.py.151: total_norm: 2.356495 [I 2022-10-08 18:20:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 120/2304 [I 2022-10-08 18:20:22] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:23 [I 2022-10-08 18:20:22] ...trap/engines/engine.py.229: process: 0.78064 | load: 0.00330 [I 2022-10-08 18:20:22] ...trap/engines/engine.py.230: loss: 0.14541 [I 2022-10-08 18:20:22] ...optimizers/trijoint.py.151: total_norm: 2.738533 [I 2022-10-08 18:20:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 130/2304 [I 2022-10-08 18:20:30] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:15 [I 2022-10-08 18:20:30] ...trap/engines/engine.py.229: process: 0.77884 | load: 0.00341 [I 2022-10-08 18:20:30] ...trap/engines/engine.py.230: loss: 0.13297 [I 2022-10-08 18:20:30] ...optimizers/trijoint.py.151: total_norm: 2.197302 [I 2022-10-08 18:20:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 140/2304 [I 2022-10-08 18:20:38] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:05 [I 2022-10-08 18:20:38] ...trap/engines/engine.py.229: process: 0.78486 | load: 0.00322 [I 2022-10-08 18:20:38] ...trap/engines/engine.py.230: loss: 0.18122 [I 2022-10-08 18:20:38] ...optimizers/trijoint.py.151: total_norm: 2.638116 [I 2022-10-08 18:20:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 150/2304 [I 2022-10-08 18:20:46] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:01 [I 2022-10-08 18:20:46] ...trap/engines/engine.py.229: process: 0.78441 | load: 0.00304 [I 2022-10-08 18:20:46] ...trap/engines/engine.py.230: loss: 0.17686 [I 2022-10-08 18:20:46] ...optimizers/trijoint.py.151: total_norm: 2.868691 [I 2022-10-08 18:20:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 160/2304 [I 2022-10-08 18:20:54] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:59 [I 2022-10-08 18:20:54] ...trap/engines/engine.py.229: process: 0.79069 | load: 0.00316 [I 2022-10-08 18:20:54] ...trap/engines/engine.py.230: loss: 0.23536 [I 2022-10-08 18:20:54] ...optimizers/trijoint.py.151: total_norm: 3.028190 [I 2022-10-08 18:21:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 170/2304 [I 2022-10-08 18:21:02] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:27:54 [I 2022-10-08 18:21:02] ...trap/engines/engine.py.229: process: 0.78797 | load: 0.00306 [I 2022-10-08 18:21:02] ...trap/engines/engine.py.230: loss: 0.17284 [I 2022-10-08 18:21:02] ...optimizers/trijoint.py.151: total_norm: 2.307042 [I 2022-10-08 18:21:10] ...trap/engines/engine.py.224: train: epoch 15 | batch 180/2304 [I 2022-10-08 18:21:10] ...trap/engines/engine.py.228: elapsed: 0:02:24 | left: 0:27:35 [I 2022-10-08 18:21:10] ...trap/engines/engine.py.229: process: 0.77182 | load: 0.00299 [I 2022-10-08 18:21:10] ...trap/engines/engine.py.230: loss: 0.21706 [I 2022-10-08 18:21:10] ...optimizers/trijoint.py.151: total_norm: 2.401118 [I 2022-10-08 18:21:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 190/2304 [I 2022-10-08 18:21:17] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:31 [I 2022-10-08 18:21:17] ...trap/engines/engine.py.229: process: 0.77859 | load: 0.00328 [I 2022-10-08 18:21:17] ...trap/engines/engine.py.230: loss: 0.18405 [I 2022-10-08 18:21:17] ...optimizers/trijoint.py.151: total_norm: 3.457867 [I 2022-10-08 18:21:25] ...trap/engines/engine.py.224: train: epoch 15 | batch 200/2304 [I 2022-10-08 18:21:25] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:24 [I 2022-10-08 18:21:25] ...trap/engines/engine.py.229: process: 0.78960 | load: 0.00300 [I 2022-10-08 18:21:25] ...trap/engines/engine.py.230: loss: 0.16633 [I 2022-10-08 18:21:25] ...optimizers/trijoint.py.151: total_norm: 2.568542 [I 2022-10-08 18:21:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 210/2304 [I 2022-10-08 18:21:33] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:02 [I 2022-10-08 18:21:33] ...trap/engines/engine.py.229: process: 0.76388 | load: 0.00315 [I 2022-10-08 18:21:33] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-08 18:21:33] ...optimizers/trijoint.py.151: total_norm: 1.757111 [I 2022-10-08 18:21:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 220/2304 [I 2022-10-08 18:21:41] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:26:54 [I 2022-10-08 18:21:41] ...trap/engines/engine.py.229: process: 0.78057 | load: 0.00300 [I 2022-10-08 18:21:41] ...trap/engines/engine.py.230: loss: 0.19852 [I 2022-10-08 18:21:41] ...optimizers/trijoint.py.151: total_norm: 2.416633 [I 2022-10-08 18:21:49] ...trap/engines/engine.py.224: train: epoch 15 | batch 230/2304 [I 2022-10-08 18:21:49] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:26:53 [I 2022-10-08 18:21:49] ...trap/engines/engine.py.229: process: 0.77448 | load: 0.00300 [I 2022-10-08 18:21:49] ...trap/engines/engine.py.230: loss: 0.18887 [I 2022-10-08 18:21:49] ...optimizers/trijoint.py.151: total_norm: 2.878157 [I 2022-10-08 18:21:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 240/2304 [I 2022-10-08 18:21:56] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:56 [I 2022-10-08 18:21:56] ...trap/engines/engine.py.229: process: 0.79140 | load: 0.00302 [I 2022-10-08 18:21:56] ...trap/engines/engine.py.230: loss: 0.20574 [I 2022-10-08 18:21:56] ...optimizers/trijoint.py.151: total_norm: 3.407537 [I 2022-10-08 18:22:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 250/2304 [I 2022-10-08 18:22:04] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:49 [I 2022-10-08 18:22:04] ...trap/engines/engine.py.229: process: 0.78990 | load: 0.00302 [I 2022-10-08 18:22:04] ...trap/engines/engine.py.230: loss: 0.15298 [I 2022-10-08 18:22:04] ...optimizers/trijoint.py.151: total_norm: 2.453123 [I 2022-10-08 18:22:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 260/2304 [I 2022-10-08 18:22:12] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:39 [I 2022-10-08 18:22:12] ...trap/engines/engine.py.229: process: 0.77516 | load: 0.00304 [I 2022-10-08 18:22:12] ...trap/engines/engine.py.230: loss: 0.14040 [I 2022-10-08 18:22:12] ...optimizers/trijoint.py.151: total_norm: 3.305970 [I 2022-10-08 18:22:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 270/2304 [I 2022-10-08 18:22:20] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:15 [I 2022-10-08 18:22:20] ...trap/engines/engine.py.229: process: 0.76934 | load: 0.00302 [I 2022-10-08 18:22:20] ...trap/engines/engine.py.230: loss: 0.19604 [I 2022-10-08 18:22:20] ...optimizers/trijoint.py.151: total_norm: 3.019867 [I 2022-10-08 18:22:28] ...trap/engines/engine.py.224: train: epoch 15 | batch 280/2304 [I 2022-10-08 18:22:28] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:26:24 [I 2022-10-08 18:22:28] ...trap/engines/engine.py.229: process: 0.78326 | load: 0.00301 [I 2022-10-08 18:22:28] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-08 18:22:28] ...optimizers/trijoint.py.151: total_norm: 2.377569 [I 2022-10-08 18:22:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 290/2304 [I 2022-10-08 18:22:36] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:26:18 [I 2022-10-08 18:22:36] ...trap/engines/engine.py.229: process: 0.79686 | load: 0.00304 [I 2022-10-08 18:22:36] ...trap/engines/engine.py.230: loss: 0.15360 [I 2022-10-08 18:22:36] ...optimizers/trijoint.py.151: total_norm: 4.719151 [I 2022-10-08 18:22:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 300/2304 [I 2022-10-08 18:22:43] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:59 [I 2022-10-08 18:22:43] ...trap/engines/engine.py.229: process: 0.78248 | load: 0.00301 [I 2022-10-08 18:22:43] ...trap/engines/engine.py.230: loss: 0.14747 [I 2022-10-08 18:22:43] ...optimizers/trijoint.py.151: total_norm: 2.177843 [I 2022-10-08 18:22:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 310/2304 [I 2022-10-08 18:22:51] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:53 [I 2022-10-08 18:22:51] ...trap/engines/engine.py.229: process: 0.77187 | load: 0.00303 [I 2022-10-08 18:22:51] ...trap/engines/engine.py.230: loss: 0.18836 [I 2022-10-08 18:22:51] ...optimizers/trijoint.py.151: total_norm: 3.311206 [I 2022-10-08 18:22:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 320/2304 [I 2022-10-08 18:22:59] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:50 [I 2022-10-08 18:22:59] ...trap/engines/engine.py.229: process: 0.78381 | load: 0.00301 [I 2022-10-08 18:22:59] ...trap/engines/engine.py.230: loss: 0.22783 [I 2022-10-08 18:22:59] ...optimizers/trijoint.py.151: total_norm: 3.124049 [I 2022-10-08 18:23:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 330/2304 [I 2022-10-08 18:23:07] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:41 [I 2022-10-08 18:23:07] ...trap/engines/engine.py.229: process: 0.77585 | load: 0.00300 [I 2022-10-08 18:23:07] ...trap/engines/engine.py.230: loss: 0.22756 [I 2022-10-08 18:23:07] ...optimizers/trijoint.py.151: total_norm: 3.525274 [I 2022-10-08 18:23:15] ...trap/engines/engine.py.224: train: epoch 15 | batch 340/2304 [I 2022-10-08 18:23:15] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:25:27 [I 2022-10-08 18:23:15] ...trap/engines/engine.py.229: process: 0.77787 | load: 0.00301 [I 2022-10-08 18:23:15] ...trap/engines/engine.py.230: loss: 0.17549 [I 2022-10-08 18:23:15] ...optimizers/trijoint.py.151: total_norm: 2.537570 [I 2022-10-08 18:23:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 350/2304 [I 2022-10-08 18:23:22] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:17 [I 2022-10-08 18:23:22] ...trap/engines/engine.py.229: process: 0.77705 | load: 0.00301 [I 2022-10-08 18:23:22] ...trap/engines/engine.py.230: loss: 0.20836 [I 2022-10-08 18:23:22] ...optimizers/trijoint.py.151: total_norm: 2.858247 [I 2022-10-08 18:23:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 360/2304 [I 2022-10-08 18:23:30] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:16 [I 2022-10-08 18:23:30] ...trap/engines/engine.py.229: process: 0.78674 | load: 0.00302 [I 2022-10-08 18:23:30] ...trap/engines/engine.py.230: loss: 0.19041 [I 2022-10-08 18:23:30] ...optimizers/trijoint.py.151: total_norm: 2.588062 [I 2022-10-08 18:23:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 370/2304 [I 2022-10-08 18:23:38] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:25:07 [I 2022-10-08 18:23:38] ...trap/engines/engine.py.229: process: 0.78144 | load: 0.00307 [I 2022-10-08 18:23:38] ...trap/engines/engine.py.230: loss: 0.23077 [I 2022-10-08 18:23:38] ...optimizers/trijoint.py.151: total_norm: 2.410747 [I 2022-10-08 18:23:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 380/2304 [I 2022-10-08 18:23:46] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:57 [I 2022-10-08 18:23:46] ...trap/engines/engine.py.229: process: 0.76708 | load: 0.00313 [I 2022-10-08 18:23:46] ...trap/engines/engine.py.230: loss: 0.21444 [I 2022-10-08 18:23:46] ...optimizers/trijoint.py.151: total_norm: 2.791466 [I 2022-10-08 18:23:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 390/2304 [I 2022-10-08 18:23:54] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:24:50 [I 2022-10-08 18:23:54] ...trap/engines/engine.py.229: process: 0.78363 | load: 0.00303 [I 2022-10-08 18:23:54] ...trap/engines/engine.py.230: loss: 0.22594 [I 2022-10-08 18:23:54] ...optimizers/trijoint.py.151: total_norm: 2.802188 [I 2022-10-08 18:24:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 400/2304 [I 2022-10-08 18:24:02] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:24:44 [I 2022-10-08 18:24:02] ...trap/engines/engine.py.229: process: 0.78651 | load: 0.00304 [I 2022-10-08 18:24:02] ...trap/engines/engine.py.230: loss: 0.21520 [I 2022-10-08 18:24:02] ...optimizers/trijoint.py.151: total_norm: 3.067718 [I 2022-10-08 18:24:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 410/2304 [I 2022-10-08 18:24:09] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:39 [I 2022-10-08 18:24:09] ...trap/engines/engine.py.229: process: 0.78187 | load: 0.00314 [I 2022-10-08 18:24:09] ...trap/engines/engine.py.230: loss: 0.10876 [I 2022-10-08 18:24:09] ...optimizers/trijoint.py.151: total_norm: 1.872013 [I 2022-10-08 18:24:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 420/2304 [I 2022-10-08 18:24:17] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:30 [I 2022-10-08 18:24:17] ...trap/engines/engine.py.229: process: 0.78938 | load: 0.00307 [I 2022-10-08 18:24:17] ...trap/engines/engine.py.230: loss: 0.16375 [I 2022-10-08 18:24:17] ...optimizers/trijoint.py.151: total_norm: 2.770041 [I 2022-10-08 18:24:25] ...trap/engines/engine.py.224: train: epoch 15 | batch 430/2304 [I 2022-10-08 18:24:25] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:14 [I 2022-10-08 18:24:25] ...trap/engines/engine.py.229: process: 0.75576 | load: 0.00301 [I 2022-10-08 18:24:25] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-08 18:24:25] ...optimizers/trijoint.py.151: total_norm: 2.239586 [I 2022-10-08 18:24:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 440/2304 [I 2022-10-08 18:24:33] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:24:16 [I 2022-10-08 18:24:33] ...trap/engines/engine.py.229: process: 0.78410 | load: 0.00303 [I 2022-10-08 18:24:33] ...trap/engines/engine.py.230: loss: 0.20590 [I 2022-10-08 18:24:33] ...optimizers/trijoint.py.151: total_norm: 3.680314 [I 2022-10-08 18:24:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 450/2304 [I 2022-10-08 18:24:41] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:24:04 [I 2022-10-08 18:24:41] ...trap/engines/engine.py.229: process: 0.78264 | load: 0.00307 [I 2022-10-08 18:24:41] ...trap/engines/engine.py.230: loss: 0.18273 [I 2022-10-08 18:24:41] ...optimizers/trijoint.py.151: total_norm: 2.743311 [I 2022-10-08 18:24:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 460/2304 [I 2022-10-08 18:24:48] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:54 [I 2022-10-08 18:24:48] ...trap/engines/engine.py.229: process: 0.78153 | load: 0.00300 [I 2022-10-08 18:24:48] ...trap/engines/engine.py.230: loss: 0.18745 [I 2022-10-08 18:24:48] ...optimizers/trijoint.py.151: total_norm: 3.153376 [I 2022-10-08 18:24:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 470/2304 [I 2022-10-08 18:24:56] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:50 [I 2022-10-08 18:24:56] ...trap/engines/engine.py.229: process: 0.77240 | load: 0.00305 [I 2022-10-08 18:24:56] ...trap/engines/engine.py.230: loss: 0.14474 [I 2022-10-08 18:24:56] ...optimizers/trijoint.py.151: total_norm: 2.848226 [I 2022-10-08 18:25:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 480/2304 [I 2022-10-08 18:25:04] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:46 [I 2022-10-08 18:25:04] ...trap/engines/engine.py.229: process: 0.78134 | load: 0.00302 [I 2022-10-08 18:25:04] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-08 18:25:04] ...optimizers/trijoint.py.151: total_norm: 3.449324 [I 2022-10-08 18:25:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 490/2304 [I 2022-10-08 18:25:12] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:33 [I 2022-10-08 18:25:12] ...trap/engines/engine.py.229: process: 0.78526 | load: 0.00300 [I 2022-10-08 18:25:12] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-08 18:25:12] ...optimizers/trijoint.py.151: total_norm: 2.313673 [I 2022-10-08 18:25:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 500/2304 [I 2022-10-08 18:25:20] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:23:27 [I 2022-10-08 18:25:20] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00305 [I 2022-10-08 18:25:20] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-08 18:25:20] ...optimizers/trijoint.py.151: total_norm: 1.963996 [I 2022-10-08 18:25:28] ...trap/engines/engine.py.224: train: epoch 15 | batch 510/2304 [I 2022-10-08 18:25:28] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:23:21 [I 2022-10-08 18:25:28] ...trap/engines/engine.py.229: process: 0.78586 | load: 0.00305 [I 2022-10-08 18:25:28] ...trap/engines/engine.py.230: loss: 0.26519 [I 2022-10-08 18:25:28] ...optimizers/trijoint.py.151: total_norm: 3.300475 [I 2022-10-08 18:25:35] ...trap/engines/engine.py.224: train: epoch 15 | batch 520/2304 [I 2022-10-08 18:25:35] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:19 [I 2022-10-08 18:25:35] ...trap/engines/engine.py.229: process: 0.78694 | load: 0.00302 [I 2022-10-08 18:25:35] ...trap/engines/engine.py.230: loss: 0.21645 [I 2022-10-08 18:25:35] ...optimizers/trijoint.py.151: total_norm: 2.572663 [I 2022-10-08 18:25:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 530/2304 [I 2022-10-08 18:25:43] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:23:00 [I 2022-10-08 18:25:43] ...trap/engines/engine.py.229: process: 0.77546 | load: 0.00303 [I 2022-10-08 18:25:43] ...trap/engines/engine.py.230: loss: 0.17149 [I 2022-10-08 18:25:43] ...optimizers/trijoint.py.151: total_norm: 1.676593 [I 2022-10-08 18:25:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 540/2304 [I 2022-10-08 18:25:51] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:56 [I 2022-10-08 18:25:51] ...trap/engines/engine.py.229: process: 0.77578 | load: 0.00314 [I 2022-10-08 18:25:51] ...trap/engines/engine.py.230: loss: 0.12964 [I 2022-10-08 18:25:51] ...optimizers/trijoint.py.151: total_norm: 2.168098 [I 2022-10-08 18:25:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 550/2304 [I 2022-10-08 18:25:59] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:22:52 [I 2022-10-08 18:25:59] ...trap/engines/engine.py.229: process: 0.78522 | load: 0.00301 [I 2022-10-08 18:25:59] ...trap/engines/engine.py.230: loss: 0.20039 [I 2022-10-08 18:25:59] ...optimizers/trijoint.py.151: total_norm: 3.037847 [I 2022-10-08 18:26:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 560/2304 [I 2022-10-08 18:26:07] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:22:36 [I 2022-10-08 18:26:07] ...trap/engines/engine.py.229: process: 0.78412 | load: 0.00301 [I 2022-10-08 18:26:07] ...trap/engines/engine.py.230: loss: 0.25511 [I 2022-10-08 18:26:07] ...optimizers/trijoint.py.151: total_norm: 3.522785 [I 2022-10-08 18:26:14] ...trap/engines/engine.py.224: train: epoch 15 | batch 570/2304 [I 2022-10-08 18:26:14] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:35 [I 2022-10-08 18:26:14] ...trap/engines/engine.py.229: process: 0.78127 | load: 0.00299 [I 2022-10-08 18:26:14] ...trap/engines/engine.py.230: loss: 0.19572 [I 2022-10-08 18:26:14] ...optimizers/trijoint.py.151: total_norm: 2.890922 [I 2022-10-08 18:26:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 580/2304 [I 2022-10-08 18:26:22] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:24 [I 2022-10-08 18:26:22] ...trap/engines/engine.py.229: process: 0.77845 | load: 0.00303 [I 2022-10-08 18:26:22] ...trap/engines/engine.py.230: loss: 0.19474 [I 2022-10-08 18:26:22] ...optimizers/trijoint.py.151: total_norm: 2.419991 [I 2022-10-08 18:26:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 590/2304 [I 2022-10-08 18:26:30] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:14 [I 2022-10-08 18:26:30] ...trap/engines/engine.py.229: process: 0.76973 | load: 0.00300 [I 2022-10-08 18:26:30] ...trap/engines/engine.py.230: loss: 0.17556 [I 2022-10-08 18:26:30] ...optimizers/trijoint.py.151: total_norm: 2.233967 [I 2022-10-08 18:26:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 600/2304 [I 2022-10-08 18:26:38] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:08 [I 2022-10-08 18:26:38] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00303 [I 2022-10-08 18:26:38] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-08 18:26:38] ...optimizers/trijoint.py.151: total_norm: 2.817650 [I 2022-10-08 18:26:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 610/2304 [I 2022-10-08 18:26:46] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:21:59 [I 2022-10-08 18:26:46] ...trap/engines/engine.py.229: process: 0.78201 | load: 0.00302 [I 2022-10-08 18:26:46] ...trap/engines/engine.py.230: loss: 0.18005 [I 2022-10-08 18:26:46] ...optimizers/trijoint.py.151: total_norm: 2.539211 [I 2022-10-08 18:26:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 620/2304 [I 2022-10-08 18:26:54] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:21:56 [I 2022-10-08 18:26:54] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00300 [I 2022-10-08 18:26:54] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-08 18:26:54] ...optimizers/trijoint.py.151: total_norm: 2.883293 [I 2022-10-08 18:27:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 630/2304 [I 2022-10-08 18:27:01] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:51 [I 2022-10-08 18:27:01] ...trap/engines/engine.py.229: process: 0.78700 | load: 0.00299 [I 2022-10-08 18:27:01] ...trap/engines/engine.py.230: loss: 0.24640 [I 2022-10-08 18:27:01] ...optimizers/trijoint.py.151: total_norm: 3.482015 [I 2022-10-08 18:27:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 640/2304 [I 2022-10-08 18:27:09] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:39 [I 2022-10-08 18:27:09] ...trap/engines/engine.py.229: process: 0.77598 | load: 0.00302 [I 2022-10-08 18:27:09] ...trap/engines/engine.py.230: loss: 0.22182 [I 2022-10-08 18:27:09] ...optimizers/trijoint.py.151: total_norm: 3.134694 [I 2022-10-08 18:27:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 650/2304 [I 2022-10-08 18:27:17] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:31 [I 2022-10-08 18:27:17] ...trap/engines/engine.py.229: process: 0.77809 | load: 0.00302 [I 2022-10-08 18:27:17] ...trap/engines/engine.py.230: loss: 0.19917 [I 2022-10-08 18:27:17] ...optimizers/trijoint.py.151: total_norm: 2.903118 [I 2022-10-08 18:27:25] ...trap/engines/engine.py.224: train: epoch 15 | batch 660/2304 [I 2022-10-08 18:27:25] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:21:20 [I 2022-10-08 18:27:25] ...trap/engines/engine.py.229: process: 0.78670 | load: 0.00301 [I 2022-10-08 18:27:25] ...trap/engines/engine.py.230: loss: 0.15133 [I 2022-10-08 18:27:25] ...optimizers/trijoint.py.151: total_norm: 2.181490 [I 2022-10-08 18:27:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 670/2304 [I 2022-10-08 18:27:33] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:21:12 [I 2022-10-08 18:27:33] ...trap/engines/engine.py.229: process: 0.78520 | load: 0.00303 [I 2022-10-08 18:27:33] ...trap/engines/engine.py.230: loss: 0.27963 [I 2022-10-08 18:27:33] ...optimizers/trijoint.py.151: total_norm: 2.955865 [I 2022-10-08 18:27:40] ...trap/engines/engine.py.224: train: epoch 15 | batch 680/2304 [I 2022-10-08 18:27:40] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:01 [I 2022-10-08 18:27:40] ...trap/engines/engine.py.229: process: 0.77538 | load: 0.00305 [I 2022-10-08 18:27:40] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-08 18:27:40] ...optimizers/trijoint.py.151: total_norm: 2.576345 [I 2022-10-08 18:27:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 690/2304 [I 2022-10-08 18:27:48] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:55 [I 2022-10-08 18:27:48] ...trap/engines/engine.py.229: process: 0.78632 | load: 0.00303 [I 2022-10-08 18:27:48] ...trap/engines/engine.py.230: loss: 0.12906 [I 2022-10-08 18:27:48] ...optimizers/trijoint.py.151: total_norm: 2.478477 [I 2022-10-08 18:27:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 700/2304 [I 2022-10-08 18:27:56] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:51 [I 2022-10-08 18:27:56] ...trap/engines/engine.py.229: process: 0.78455 | load: 0.00300 [I 2022-10-08 18:27:56] ...trap/engines/engine.py.230: loss: 0.14900 [I 2022-10-08 18:27:56] ...optimizers/trijoint.py.151: total_norm: 2.540574 [I 2022-10-08 18:28:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 710/2304 [I 2022-10-08 18:28:04] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:20:41 [I 2022-10-08 18:28:04] ...trap/engines/engine.py.229: process: 0.77744 | load: 0.00302 [I 2022-10-08 18:28:04] ...trap/engines/engine.py.230: loss: 0.21526 [I 2022-10-08 18:28:04] ...optimizers/trijoint.py.151: total_norm: 3.830108 [I 2022-10-08 18:28:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 720/2304 [I 2022-10-08 18:28:12] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:20:36 [I 2022-10-08 18:28:12] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00302 [I 2022-10-08 18:28:12] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-08 18:28:12] ...optimizers/trijoint.py.151: total_norm: 2.449062 [I 2022-10-08 18:28:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 730/2304 [I 2022-10-08 18:28:20] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:20:32 [I 2022-10-08 18:28:20] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00302 [I 2022-10-08 18:28:20] ...trap/engines/engine.py.230: loss: 0.17674 [I 2022-10-08 18:28:20] ...optimizers/trijoint.py.151: total_norm: 3.080973 [I 2022-10-08 18:28:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 740/2304 [I 2022-10-08 18:28:27] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:25 [I 2022-10-08 18:28:27] ...trap/engines/engine.py.229: process: 0.78496 | load: 0.00306 [I 2022-10-08 18:28:27] ...trap/engines/engine.py.230: loss: 0.14700 [I 2022-10-08 18:28:27] ...optimizers/trijoint.py.151: total_norm: 3.538988 [I 2022-10-08 18:28:35] ...trap/engines/engine.py.224: train: epoch 15 | batch 750/2304 [I 2022-10-08 18:28:35] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:13 [I 2022-10-08 18:28:35] ...trap/engines/engine.py.229: process: 0.78868 | load: 0.00303 [I 2022-10-08 18:28:35] ...trap/engines/engine.py.230: loss: 0.20644 [I 2022-10-08 18:28:35] ...optimizers/trijoint.py.151: total_norm: 2.933285 [I 2022-10-08 18:28:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 760/2304 [I 2022-10-08 18:28:43] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:20:10 [I 2022-10-08 18:28:43] ...trap/engines/engine.py.229: process: 0.78535 | load: 0.00303 [I 2022-10-08 18:28:43] ...trap/engines/engine.py.230: loss: 0.14509 [I 2022-10-08 18:28:43] ...optimizers/trijoint.py.151: total_norm: 1.990423 [I 2022-10-08 18:28:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 770/2304 [I 2022-10-08 18:28:51] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:19:55 [I 2022-10-08 18:28:51] ...trap/engines/engine.py.229: process: 0.78288 | load: 0.00302 [I 2022-10-08 18:28:51] ...trap/engines/engine.py.230: loss: 0.29072 [I 2022-10-08 18:28:51] ...optimizers/trijoint.py.151: total_norm: 4.269667 [I 2022-10-08 18:28:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 780/2304 [I 2022-10-08 18:28:59] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:19:46 [I 2022-10-08 18:28:59] ...trap/engines/engine.py.229: process: 0.78202 | load: 0.00305 [I 2022-10-08 18:28:59] ...trap/engines/engine.py.230: loss: 0.16337 [I 2022-10-08 18:28:59] ...optimizers/trijoint.py.151: total_norm: 3.274834 [I 2022-10-08 18:29:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 790/2304 [I 2022-10-08 18:29:07] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:19:37 [I 2022-10-08 18:29:07] ...trap/engines/engine.py.229: process: 0.77315 | load: 0.00303 [I 2022-10-08 18:29:07] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-08 18:29:07] ...optimizers/trijoint.py.151: total_norm: 2.339682 [I 2022-10-08 18:29:15] ...trap/engines/engine.py.224: train: epoch 15 | batch 800/2304 [I 2022-10-08 18:29:15] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:19:41 [I 2022-10-08 18:29:15] ...trap/engines/engine.py.229: process: 0.79183 | load: 0.00302 [I 2022-10-08 18:29:15] ...trap/engines/engine.py.230: loss: 0.15293 [I 2022-10-08 18:29:15] ...optimizers/trijoint.py.151: total_norm: 2.701888 [I 2022-10-08 18:29:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 810/2304 [I 2022-10-08 18:29:22] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:23 [I 2022-10-08 18:29:22] ...trap/engines/engine.py.229: process: 0.78257 | load: 0.00301 [I 2022-10-08 18:29:22] ...trap/engines/engine.py.230: loss: 0.14776 [I 2022-10-08 18:29:22] ...optimizers/trijoint.py.151: total_norm: 2.739704 [I 2022-10-08 18:29:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 820/2304 [I 2022-10-08 18:29:30] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:19:20 [I 2022-10-08 18:29:30] ...trap/engines/engine.py.229: process: 0.78642 | load: 0.00303 [I 2022-10-08 18:29:30] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-08 18:29:30] ...optimizers/trijoint.py.151: total_norm: 2.853087 [I 2022-10-08 18:29:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 830/2304 [I 2022-10-08 18:29:38] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:19:10 [I 2022-10-08 18:29:38] ...trap/engines/engine.py.229: process: 0.77992 | load: 0.00300 [I 2022-10-08 18:29:38] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-08 18:29:38] ...optimizers/trijoint.py.151: total_norm: 2.369176 [I 2022-10-08 18:29:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 840/2304 [I 2022-10-08 18:29:46] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:19:00 [I 2022-10-08 18:29:46] ...trap/engines/engine.py.229: process: 0.77990 | load: 0.00302 [I 2022-10-08 18:29:46] ...trap/engines/engine.py.230: loss: 0.20533 [I 2022-10-08 18:29:46] ...optimizers/trijoint.py.151: total_norm: 2.935166 [I 2022-10-08 18:29:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 850/2304 [I 2022-10-08 18:29:54] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:18:55 [I 2022-10-08 18:29:54] ...trap/engines/engine.py.229: process: 0.76846 | load: 0.00301 [I 2022-10-08 18:29:54] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-08 18:29:54] ...optimizers/trijoint.py.151: total_norm: 2.603183 [I 2022-10-08 18:30:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 860/2304 [I 2022-10-08 18:30:01] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:44 [I 2022-10-08 18:30:01] ...trap/engines/engine.py.229: process: 0.78287 | load: 0.00301 [I 2022-10-08 18:30:01] ...trap/engines/engine.py.230: loss: 0.20104 [I 2022-10-08 18:30:01] ...optimizers/trijoint.py.151: total_norm: 2.789294 [I 2022-10-08 18:30:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 870/2304 [I 2022-10-08 18:30:09] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:33 [I 2022-10-08 18:30:09] ...trap/engines/engine.py.229: process: 0.76816 | load: 0.00303 [I 2022-10-08 18:30:09] ...trap/engines/engine.py.230: loss: 0.19402 [I 2022-10-08 18:30:09] ...optimizers/trijoint.py.151: total_norm: 1.726498 [I 2022-10-08 18:30:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 880/2304 [I 2022-10-08 18:30:17] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:18:32 [I 2022-10-08 18:30:17] ...trap/engines/engine.py.229: process: 0.79201 | load: 0.00300 [I 2022-10-08 18:30:17] ...trap/engines/engine.py.230: loss: 0.18269 [I 2022-10-08 18:30:17] ...optimizers/trijoint.py.151: total_norm: 2.338433 [I 2022-10-08 18:30:25] ...trap/engines/engine.py.224: train: epoch 15 | batch 890/2304 [I 2022-10-08 18:30:25] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:18:24 [I 2022-10-08 18:30:25] ...trap/engines/engine.py.229: process: 0.80345 | load: 0.00300 [I 2022-10-08 18:30:25] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-08 18:30:25] ...optimizers/trijoint.py.151: total_norm: 2.659702 [I 2022-10-08 18:30:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 900/2304 [I 2022-10-08 18:30:33] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:18:25 [I 2022-10-08 18:30:33] ...trap/engines/engine.py.229: process: 0.79235 | load: 0.00303 [I 2022-10-08 18:30:33] ...trap/engines/engine.py.230: loss: 0.20514 [I 2022-10-08 18:30:33] ...optimizers/trijoint.py.151: total_norm: 2.663636 [I 2022-10-08 18:30:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 910/2304 [I 2022-10-08 18:30:41] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:18:09 [I 2022-10-08 18:30:41] ...trap/engines/engine.py.229: process: 0.78440 | load: 0.00300 [I 2022-10-08 18:30:41] ...trap/engines/engine.py.230: loss: 0.21118 [I 2022-10-08 18:30:41] ...optimizers/trijoint.py.151: total_norm: 3.104727 [I 2022-10-08 18:30:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 920/2304 [I 2022-10-08 18:30:48] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:17:55 [I 2022-10-08 18:30:48] ...trap/engines/engine.py.229: process: 0.76792 | load: 0.00305 [I 2022-10-08 18:30:48] ...trap/engines/engine.py.230: loss: 0.18546 [I 2022-10-08 18:30:48] ...optimizers/trijoint.py.151: total_norm: 3.106217 [I 2022-10-08 18:30:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 930/2304 [I 2022-10-08 18:30:56] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:17:44 [I 2022-10-08 18:30:56] ...trap/engines/engine.py.229: process: 0.77612 | load: 0.00305 [I 2022-10-08 18:30:56] ...trap/engines/engine.py.230: loss: 0.13249 [I 2022-10-08 18:30:56] ...optimizers/trijoint.py.151: total_norm: 3.143439 [I 2022-10-08 18:31:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 940/2304 [I 2022-10-08 18:31:04] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:17:44 [I 2022-10-08 18:31:04] ...trap/engines/engine.py.229: process: 0.77936 | load: 0.00301 [I 2022-10-08 18:31:04] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-08 18:31:04] ...optimizers/trijoint.py.151: total_norm: 2.252098 [I 2022-10-08 18:31:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 950/2304 [I 2022-10-08 18:31:12] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:17:35 [I 2022-10-08 18:31:12] ...trap/engines/engine.py.229: process: 0.77638 | load: 0.00302 [I 2022-10-08 18:31:12] ...trap/engines/engine.py.230: loss: 0.19532 [I 2022-10-08 18:31:12] ...optimizers/trijoint.py.151: total_norm: 3.064993 [I 2022-10-08 18:31:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 960/2304 [I 2022-10-08 18:31:20] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:17:26 [I 2022-10-08 18:31:20] ...trap/engines/engine.py.229: process: 0.77766 | load: 0.00311 [I 2022-10-08 18:31:20] ...trap/engines/engine.py.230: loss: 0.21080 [I 2022-10-08 18:31:20] ...optimizers/trijoint.py.151: total_norm: 3.156610 [I 2022-10-08 18:31:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 970/2304 [I 2022-10-08 18:31:27] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:22 [I 2022-10-08 18:31:27] ...trap/engines/engine.py.229: process: 0.78804 | load: 0.00309 [I 2022-10-08 18:31:27] ...trap/engines/engine.py.230: loss: 0.17817 [I 2022-10-08 18:31:27] ...optimizers/trijoint.py.151: total_norm: 3.292324 [I 2022-10-08 18:31:35] ...trap/engines/engine.py.224: train: epoch 15 | batch 980/2304 [I 2022-10-08 18:31:35] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:17:08 [I 2022-10-08 18:31:35] ...trap/engines/engine.py.229: process: 0.77977 | load: 0.00303 [I 2022-10-08 18:31:35] ...trap/engines/engine.py.230: loss: 0.19239 [I 2022-10-08 18:31:35] ...optimizers/trijoint.py.151: total_norm: 2.497181 [I 2022-10-08 18:31:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 990/2304 [I 2022-10-08 18:31:43] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:17:05 [I 2022-10-08 18:31:43] ...trap/engines/engine.py.229: process: 0.78240 | load: 0.00304 [I 2022-10-08 18:31:43] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-08 18:31:43] ...optimizers/trijoint.py.151: total_norm: 2.075089 [I 2022-10-08 18:31:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 1000/2304 [I 2022-10-08 18:31:51] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:16:49 [I 2022-10-08 18:31:51] ...trap/engines/engine.py.229: process: 0.77096 | load: 0.00301 [I 2022-10-08 18:31:51] ...trap/engines/engine.py.230: loss: 0.21502 [I 2022-10-08 18:31:51] ...optimizers/trijoint.py.151: total_norm: 2.846803 [I 2022-10-08 18:31:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 1010/2304 [I 2022-10-08 18:31:59] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:16:46 [I 2022-10-08 18:31:59] ...trap/engines/engine.py.229: process: 0.78880 | load: 0.00298 [I 2022-10-08 18:31:59] ...trap/engines/engine.py.230: loss: 0.19667 [I 2022-10-08 18:31:59] ...optimizers/trijoint.py.151: total_norm: 3.362575 [I 2022-10-08 18:32:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1020/2304 [I 2022-10-08 18:32:06] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:42 [I 2022-10-08 18:32:06] ...trap/engines/engine.py.229: process: 0.79221 | load: 0.00300 [I 2022-10-08 18:32:06] ...trap/engines/engine.py.230: loss: 0.17954 [I 2022-10-08 18:32:06] ...optimizers/trijoint.py.151: total_norm: 2.878539 [I 2022-10-08 18:32:14] ...trap/engines/engine.py.224: train: epoch 15 | batch 1030/2304 [I 2022-10-08 18:32:14] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:16:30 [I 2022-10-08 18:32:14] ...trap/engines/engine.py.229: process: 0.77879 | load: 0.00305 [I 2022-10-08 18:32:14] ...trap/engines/engine.py.230: loss: 0.20094 [I 2022-10-08 18:32:14] ...optimizers/trijoint.py.151: total_norm: 3.108591 [I 2022-10-08 18:32:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1040/2304 [I 2022-10-08 18:32:22] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:16:27 [I 2022-10-08 18:32:22] ...trap/engines/engine.py.229: process: 0.78493 | load: 0.00303 [I 2022-10-08 18:32:22] ...trap/engines/engine.py.230: loss: 0.20328 [I 2022-10-08 18:32:22] ...optimizers/trijoint.py.151: total_norm: 2.643453 [I 2022-10-08 18:32:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 1050/2304 [I 2022-10-08 18:32:30] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:16:20 [I 2022-10-08 18:32:30] ...trap/engines/engine.py.229: process: 0.78671 | load: 0.00303 [I 2022-10-08 18:32:30] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-08 18:32:30] ...optimizers/trijoint.py.151: total_norm: 2.339185 [I 2022-10-08 18:32:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 1060/2304 [I 2022-10-08 18:32:38] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:16:08 [I 2022-10-08 18:32:38] ...trap/engines/engine.py.229: process: 0.76786 | load: 0.00306 [I 2022-10-08 18:32:38] ...trap/engines/engine.py.230: loss: 0.19441 [I 2022-10-08 18:32:38] ...optimizers/trijoint.py.151: total_norm: 3.076446 [I 2022-10-08 18:32:45] ...trap/engines/engine.py.224: train: epoch 15 | batch 1070/2304 [I 2022-10-08 18:32:45] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:15:59 [I 2022-10-08 18:32:45] ...trap/engines/engine.py.229: process: 0.77947 | load: 0.00301 [I 2022-10-08 18:32:45] ...trap/engines/engine.py.230: loss: 0.24979 [I 2022-10-08 18:32:45] ...optimizers/trijoint.py.151: total_norm: 4.197251 [I 2022-10-08 18:32:53] ...trap/engines/engine.py.224: train: epoch 15 | batch 1080/2304 [I 2022-10-08 18:32:53] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:16:01 [I 2022-10-08 18:32:53] ...trap/engines/engine.py.229: process: 0.78749 | load: 0.00304 [I 2022-10-08 18:32:53] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-08 18:32:53] ...optimizers/trijoint.py.151: total_norm: 2.203019 [I 2022-10-08 18:33:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 1090/2304 [I 2022-10-08 18:33:01] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:15:47 [I 2022-10-08 18:33:01] ...trap/engines/engine.py.229: process: 0.78446 | load: 0.00305 [I 2022-10-08 18:33:01] ...trap/engines/engine.py.230: loss: 0.26887 [I 2022-10-08 18:33:01] ...optimizers/trijoint.py.151: total_norm: 3.602806 [I 2022-10-08 18:33:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 1100/2304 [I 2022-10-08 18:33:09] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:15:36 [I 2022-10-08 18:33:09] ...trap/engines/engine.py.229: process: 0.78504 | load: 0.00317 [I 2022-10-08 18:33:09] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-08 18:33:09] ...optimizers/trijoint.py.151: total_norm: 2.086580 [I 2022-10-08 18:33:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1110/2304 [I 2022-10-08 18:33:17] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:15:29 [I 2022-10-08 18:33:17] ...trap/engines/engine.py.229: process: 0.77699 | load: 0.00307 [I 2022-10-08 18:33:17] ...trap/engines/engine.py.230: loss: 0.16504 [I 2022-10-08 18:33:17] ...optimizers/trijoint.py.151: total_norm: 2.476486 [I 2022-10-08 18:33:24] ...trap/engines/engine.py.224: train: epoch 15 | batch 1120/2304 [I 2022-10-08 18:33:24] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:22 [I 2022-10-08 18:33:24] ...trap/engines/engine.py.229: process: 0.77805 | load: 0.00301 [I 2022-10-08 18:33:24] ...trap/engines/engine.py.230: loss: 0.12774 [I 2022-10-08 18:33:24] ...optimizers/trijoint.py.151: total_norm: 1.805001 [I 2022-10-08 18:33:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1130/2304 [I 2022-10-08 18:33:32] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:15 [I 2022-10-08 18:33:32] ...trap/engines/engine.py.229: process: 0.77225 | load: 0.00301 [I 2022-10-08 18:33:32] ...trap/engines/engine.py.230: loss: 0.18654 [I 2022-10-08 18:33:32] ...optimizers/trijoint.py.151: total_norm: 3.260623 [I 2022-10-08 18:33:58] ...trap/engines/engine.py.224: train: epoch 15 | batch 1140/2304 [I 2022-10-08 18:33:58] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:26:50 [I 2022-10-08 18:33:58] ...trap/engines/engine.py.229: process: 0.77534 | load: 0.00299 [I 2022-10-08 18:33:58] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-08 18:33:58] ...optimizers/trijoint.py.151: total_norm: 1.860992 [I 2022-10-08 18:34:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1150/2304 [I 2022-10-08 18:34:06] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:16:11 [I 2022-10-08 18:34:06] ...trap/engines/engine.py.229: process: 0.77536 | load: 0.00303 [I 2022-10-08 18:34:06] ...trap/engines/engine.py.230: loss: 0.20419 [I 2022-10-08 18:34:06] ...optimizers/trijoint.py.151: total_norm: 2.732814 [I 2022-10-08 18:34:14] ...trap/engines/engine.py.224: train: epoch 15 | batch 1160/2304 [I 2022-10-08 18:34:14] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:14:53 [I 2022-10-08 18:34:14] ...trap/engines/engine.py.229: process: 0.77453 | load: 0.00308 [I 2022-10-08 18:34:14] ...trap/engines/engine.py.230: loss: 0.22151 [I 2022-10-08 18:34:14] ...optimizers/trijoint.py.151: total_norm: 2.489408 [I 2022-10-08 18:34:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1170/2304 [I 2022-10-08 18:34:22] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:14:38 [I 2022-10-08 18:34:22] ...trap/engines/engine.py.229: process: 0.77856 | load: 0.00301 [I 2022-10-08 18:34:22] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-08 18:34:22] ...optimizers/trijoint.py.151: total_norm: 2.940222 [I 2022-10-08 18:34:29] ...trap/engines/engine.py.224: train: epoch 15 | batch 1180/2304 [I 2022-10-08 18:34:29] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:14:35 [I 2022-10-08 18:34:29] ...trap/engines/engine.py.229: process: 0.78498 | load: 0.00303 [I 2022-10-08 18:34:29] ...trap/engines/engine.py.230: loss: 0.18187 [I 2022-10-08 18:34:29] ...optimizers/trijoint.py.151: total_norm: 1.969596 [I 2022-10-08 18:34:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1190/2304 [I 2022-10-08 18:34:37] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:14:30 [I 2022-10-08 18:34:37] ...trap/engines/engine.py.229: process: 0.78817 | load: 0.00302 [I 2022-10-08 18:34:37] ...trap/engines/engine.py.230: loss: 0.16758 [I 2022-10-08 18:34:37] ...optimizers/trijoint.py.151: total_norm: 2.538411 [I 2022-10-08 18:34:45] ...trap/engines/engine.py.224: train: epoch 15 | batch 1200/2304 [I 2022-10-08 18:34:45] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:14:17 [I 2022-10-08 18:34:45] ...trap/engines/engine.py.229: process: 0.77155 | load: 0.00305 [I 2022-10-08 18:34:45] ...trap/engines/engine.py.230: loss: 0.10171 [I 2022-10-08 18:34:45] ...optimizers/trijoint.py.151: total_norm: 2.008924 [I 2022-10-08 18:34:53] ...trap/engines/engine.py.224: train: epoch 15 | batch 1210/2304 [I 2022-10-08 18:34:53] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:14:07 [I 2022-10-08 18:34:53] ...trap/engines/engine.py.229: process: 0.76480 | load: 0.00301 [I 2022-10-08 18:34:53] ...trap/engines/engine.py.230: loss: 0.19897 [I 2022-10-08 18:34:53] ...optimizers/trijoint.py.151: total_norm: 2.274554 [I 2022-10-08 18:35:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 1220/2304 [I 2022-10-08 18:35:01] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:14:00 [I 2022-10-08 18:35:01] ...trap/engines/engine.py.229: process: 0.77676 | load: 0.00302 [I 2022-10-08 18:35:01] ...trap/engines/engine.py.230: loss: 0.20013 [I 2022-10-08 18:35:01] ...optimizers/trijoint.py.151: total_norm: 2.629073 [I 2022-10-08 18:35:08] ...trap/engines/engine.py.224: train: epoch 15 | batch 1230/2304 [I 2022-10-08 18:35:08] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:13:52 [I 2022-10-08 18:35:08] ...trap/engines/engine.py.229: process: 0.77633 | load: 0.00302 [I 2022-10-08 18:35:08] ...trap/engines/engine.py.230: loss: 0.10606 [I 2022-10-08 18:35:08] ...optimizers/trijoint.py.151: total_norm: 1.515065 [I 2022-10-08 18:35:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 1240/2304 [I 2022-10-08 18:35:16] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:13:51 [I 2022-10-08 18:35:16] ...trap/engines/engine.py.229: process: 0.76885 | load: 0.00301 [I 2022-10-08 18:35:16] ...trap/engines/engine.py.230: loss: 0.28001 [I 2022-10-08 18:35:16] ...optimizers/trijoint.py.151: total_norm: 2.390942 [I 2022-10-08 18:35:24] ...trap/engines/engine.py.224: train: epoch 15 | batch 1250/2304 [I 2022-10-08 18:35:24] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:13:43 [I 2022-10-08 18:35:24] ...trap/engines/engine.py.229: process: 0.78691 | load: 0.00304 [I 2022-10-08 18:35:24] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-08 18:35:24] ...optimizers/trijoint.py.151: total_norm: 2.606530 [I 2022-10-08 18:35:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1260/2304 [I 2022-10-08 18:35:32] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:13:32 [I 2022-10-08 18:35:32] ...trap/engines/engine.py.229: process: 0.77113 | load: 0.00305 [I 2022-10-08 18:35:32] ...trap/engines/engine.py.230: loss: 0.19615 [I 2022-10-08 18:35:32] ...optimizers/trijoint.py.151: total_norm: 2.818678 [I 2022-10-08 18:35:40] ...trap/engines/engine.py.224: train: epoch 15 | batch 1270/2304 [I 2022-10-08 18:35:40] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:13:23 [I 2022-10-08 18:35:40] ...trap/engines/engine.py.229: process: 0.78246 | load: 0.00303 [I 2022-10-08 18:35:40] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-08 18:35:40] ...optimizers/trijoint.py.151: total_norm: 2.751063 [I 2022-10-08 18:35:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1280/2304 [I 2022-10-08 18:35:47] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:13:17 [I 2022-10-08 18:35:47] ...trap/engines/engine.py.229: process: 0.78006 | load: 0.00302 [I 2022-10-08 18:35:47] ...trap/engines/engine.py.230: loss: 0.17429 [I 2022-10-08 18:35:47] ...optimizers/trijoint.py.151: total_norm: 1.817944 [I 2022-10-08 18:35:55] ...trap/engines/engine.py.224: train: epoch 15 | batch 1290/2304 [I 2022-10-08 18:35:55] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:13:09 [I 2022-10-08 18:35:55] ...trap/engines/engine.py.229: process: 0.79126 | load: 0.00309 [I 2022-10-08 18:35:55] ...trap/engines/engine.py.230: loss: 0.20323 [I 2022-10-08 18:35:55] ...optimizers/trijoint.py.151: total_norm: 3.827056 [I 2022-10-08 18:36:03] ...trap/engines/engine.py.224: train: epoch 15 | batch 1300/2304 [I 2022-10-08 18:36:03] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:12:59 [I 2022-10-08 18:36:03] ...trap/engines/engine.py.229: process: 0.77222 | load: 0.00304 [I 2022-10-08 18:36:03] ...trap/engines/engine.py.230: loss: 0.26263 [I 2022-10-08 18:36:03] ...optimizers/trijoint.py.151: total_norm: 3.067273 [I 2022-10-08 18:36:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 1310/2304 [I 2022-10-08 18:36:11] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:12:58 [I 2022-10-08 18:36:11] ...trap/engines/engine.py.229: process: 0.78576 | load: 0.00302 [I 2022-10-08 18:36:11] ...trap/engines/engine.py.230: loss: 0.10020 [I 2022-10-08 18:36:11] ...optimizers/trijoint.py.151: total_norm: 2.199965 [I 2022-10-08 18:36:19] ...trap/engines/engine.py.224: train: epoch 15 | batch 1320/2304 [I 2022-10-08 18:36:19] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:12:47 [I 2022-10-08 18:36:19] ...trap/engines/engine.py.229: process: 0.78181 | load: 0.00305 [I 2022-10-08 18:36:19] ...trap/engines/engine.py.230: loss: 0.14026 [I 2022-10-08 18:36:19] ...optimizers/trijoint.py.151: total_norm: 2.671280 [I 2022-10-08 18:36:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1330/2304 [I 2022-10-08 18:36:26] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:12:37 [I 2022-10-08 18:36:26] ...trap/engines/engine.py.229: process: 0.76714 | load: 0.00305 [I 2022-10-08 18:36:26] ...trap/engines/engine.py.230: loss: 0.18539 [I 2022-10-08 18:36:26] ...optimizers/trijoint.py.151: total_norm: 2.205792 [I 2022-10-08 18:36:34] ...trap/engines/engine.py.224: train: epoch 15 | batch 1340/2304 [I 2022-10-08 18:36:34] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:12:31 [I 2022-10-08 18:36:34] ...trap/engines/engine.py.229: process: 0.78585 | load: 0.00295 [I 2022-10-08 18:36:34] ...trap/engines/engine.py.230: loss: 0.18458 [I 2022-10-08 18:36:34] ...optimizers/trijoint.py.151: total_norm: 2.212758 [I 2022-10-08 18:36:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1350/2304 [I 2022-10-08 18:36:42] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:12:19 [I 2022-10-08 18:36:42] ...trap/engines/engine.py.229: process: 0.75441 | load: 0.00300 [I 2022-10-08 18:36:42] ...trap/engines/engine.py.230: loss: 0.14344 [I 2022-10-08 18:36:42] ...optimizers/trijoint.py.151: total_norm: 2.357825 [I 2022-10-08 18:36:50] ...trap/engines/engine.py.224: train: epoch 15 | batch 1360/2304 [I 2022-10-08 18:36:50] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:12:13 [I 2022-10-08 18:36:50] ...trap/engines/engine.py.229: process: 0.76333 | load: 0.00299 [I 2022-10-08 18:36:50] ...trap/engines/engine.py.230: loss: 0.13279 [I 2022-10-08 18:36:50] ...optimizers/trijoint.py.151: total_norm: 1.922421 [I 2022-10-08 18:36:58] ...trap/engines/engine.py.224: train: epoch 15 | batch 1370/2304 [I 2022-10-08 18:36:58] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:12:06 [I 2022-10-08 18:36:58] ...trap/engines/engine.py.229: process: 0.77212 | load: 0.00301 [I 2022-10-08 18:36:58] ...trap/engines/engine.py.230: loss: 0.19771 [I 2022-10-08 18:36:58] ...optimizers/trijoint.py.151: total_norm: 2.751837 [I 2022-10-08 18:37:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1380/2304 [I 2022-10-08 18:37:06] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:12:00 [I 2022-10-08 18:37:06] ...trap/engines/engine.py.229: process: 0.77268 | load: 0.00300 [I 2022-10-08 18:37:06] ...trap/engines/engine.py.230: loss: 0.18246 [I 2022-10-08 18:37:06] ...optimizers/trijoint.py.151: total_norm: 2.300556 [I 2022-10-08 18:37:13] ...trap/engines/engine.py.224: train: epoch 15 | batch 1390/2304 [I 2022-10-08 18:37:13] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:11:45 [I 2022-10-08 18:37:13] ...trap/engines/engine.py.229: process: 0.76875 | load: 0.00301 [I 2022-10-08 18:37:13] ...trap/engines/engine.py.230: loss: 0.20748 [I 2022-10-08 18:37:13] ...optimizers/trijoint.py.151: total_norm: 4.018991 [I 2022-10-08 18:37:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 1400/2304 [I 2022-10-08 18:37:21] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:11:40 [I 2022-10-08 18:37:21] ...trap/engines/engine.py.229: process: 0.77581 | load: 0.00300 [I 2022-10-08 18:37:21] ...trap/engines/engine.py.230: loss: 0.20932 [I 2022-10-08 18:37:21] ...optimizers/trijoint.py.151: total_norm: 3.337620 [I 2022-10-08 18:37:29] ...trap/engines/engine.py.224: train: epoch 15 | batch 1410/2304 [I 2022-10-08 18:37:29] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:11:35 [I 2022-10-08 18:37:29] ...trap/engines/engine.py.229: process: 0.77690 | load: 0.00306 [I 2022-10-08 18:37:29] ...trap/engines/engine.py.230: loss: 0.22873 [I 2022-10-08 18:37:29] ...optimizers/trijoint.py.151: total_norm: 3.408161 [I 2022-10-08 18:37:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1420/2304 [I 2022-10-08 18:37:37] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:11:24 [I 2022-10-08 18:37:37] ...trap/engines/engine.py.229: process: 0.77094 | load: 0.00302 [I 2022-10-08 18:37:37] ...trap/engines/engine.py.230: loss: 0.15586 [I 2022-10-08 18:37:37] ...optimizers/trijoint.py.151: total_norm: 1.599598 [I 2022-10-08 18:37:44] ...trap/engines/engine.py.224: train: epoch 15 | batch 1430/2304 [I 2022-10-08 18:37:44] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:11:21 [I 2022-10-08 18:37:44] ...trap/engines/engine.py.229: process: 0.79168 | load: 0.00301 [I 2022-10-08 18:37:44] ...trap/engines/engine.py.230: loss: 0.18020 [I 2022-10-08 18:37:44] ...optimizers/trijoint.py.151: total_norm: 2.626564 [I 2022-10-08 18:37:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1440/2304 [I 2022-10-08 18:37:52] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:11:12 [I 2022-10-08 18:37:52] ...trap/engines/engine.py.229: process: 0.78304 | load: 0.00309 [I 2022-10-08 18:37:52] ...trap/engines/engine.py.230: loss: 0.09466 [I 2022-10-08 18:37:52] ...optimizers/trijoint.py.151: total_norm: 1.795702 [I 2022-10-08 18:38:00] ...trap/engines/engine.py.224: train: epoch 15 | batch 1450/2304 [I 2022-10-08 18:38:00] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:11:04 [I 2022-10-08 18:38:00] ...trap/engines/engine.py.229: process: 0.78828 | load: 0.00303 [I 2022-10-08 18:38:00] ...trap/engines/engine.py.230: loss: 0.19736 [I 2022-10-08 18:38:00] ...optimizers/trijoint.py.151: total_norm: 2.293189 [I 2022-10-08 18:38:08] ...trap/engines/engine.py.224: train: epoch 15 | batch 1460/2304 [I 2022-10-08 18:38:08] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:10:56 [I 2022-10-08 18:38:08] ...trap/engines/engine.py.229: process: 0.77583 | load: 0.00298 [I 2022-10-08 18:38:08] ...trap/engines/engine.py.230: loss: 0.19935 [I 2022-10-08 18:38:08] ...optimizers/trijoint.py.151: total_norm: 3.463941 [I 2022-10-08 18:38:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 1470/2304 [I 2022-10-08 18:38:16] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:10:48 [I 2022-10-08 18:38:16] ...trap/engines/engine.py.229: process: 0.77575 | load: 0.00305 [I 2022-10-08 18:38:16] ...trap/engines/engine.py.230: loss: 0.19028 [I 2022-10-08 18:38:16] ...optimizers/trijoint.py.151: total_norm: 2.285267 [I 2022-10-08 18:38:23] ...trap/engines/engine.py.224: train: epoch 15 | batch 1480/2304 [I 2022-10-08 18:38:23] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:10:39 [I 2022-10-08 18:38:23] ...trap/engines/engine.py.229: process: 0.77530 | load: 0.00305 [I 2022-10-08 18:38:23] ...trap/engines/engine.py.230: loss: 0.19443 [I 2022-10-08 18:38:23] ...optimizers/trijoint.py.151: total_norm: 2.642904 [I 2022-10-08 18:38:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 1490/2304 [I 2022-10-08 18:38:31] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:10:36 [I 2022-10-08 18:38:31] ...trap/engines/engine.py.229: process: 0.76744 | load: 0.00303 [I 2022-10-08 18:38:31] ...trap/engines/engine.py.230: loss: 0.29196 [I 2022-10-08 18:38:31] ...optimizers/trijoint.py.151: total_norm: 3.870976 [I 2022-10-08 18:38:39] ...trap/engines/engine.py.224: train: epoch 15 | batch 1500/2304 [I 2022-10-08 18:38:39] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:10:24 [I 2022-10-08 18:38:39] ...trap/engines/engine.py.229: process: 0.77855 | load: 0.00304 [I 2022-10-08 18:38:39] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-08 18:38:39] ...optimizers/trijoint.py.151: total_norm: 2.197128 [I 2022-10-08 18:38:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1510/2304 [I 2022-10-08 18:38:47] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:10:23 [I 2022-10-08 18:38:47] ...trap/engines/engine.py.229: process: 0.78924 | load: 0.00304 [I 2022-10-08 18:38:47] ...trap/engines/engine.py.230: loss: 0.21928 [I 2022-10-08 18:38:47] ...optimizers/trijoint.py.151: total_norm: 2.832370 [I 2022-10-08 18:38:55] ...trap/engines/engine.py.224: train: epoch 15 | batch 1520/2304 [I 2022-10-08 18:38:55] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:10:08 [I 2022-10-08 18:38:55] ...trap/engines/engine.py.229: process: 0.78104 | load: 0.00302 [I 2022-10-08 18:38:55] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-08 18:38:55] ...optimizers/trijoint.py.151: total_norm: 3.078413 [I 2022-10-08 18:39:03] ...trap/engines/engine.py.224: train: epoch 15 | batch 1530/2304 [I 2022-10-08 18:39:03] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:10:04 [I 2022-10-08 18:39:03] ...trap/engines/engine.py.229: process: 0.77717 | load: 0.00301 [I 2022-10-08 18:39:03] ...trap/engines/engine.py.230: loss: 0.19631 [I 2022-10-08 18:39:03] ...optimizers/trijoint.py.151: total_norm: 1.413793 [I 2022-10-08 18:39:10] ...trap/engines/engine.py.224: train: epoch 15 | batch 1540/2304 [I 2022-10-08 18:39:10] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:09:52 [I 2022-10-08 18:39:10] ...trap/engines/engine.py.229: process: 0.76819 | load: 0.00303 [I 2022-10-08 18:39:10] ...trap/engines/engine.py.230: loss: 0.22648 [I 2022-10-08 18:39:10] ...optimizers/trijoint.py.151: total_norm: 3.077916 [I 2022-10-08 18:39:18] ...trap/engines/engine.py.224: train: epoch 15 | batch 1550/2304 [I 2022-10-08 18:39:18] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:09:45 [I 2022-10-08 18:39:18] ...trap/engines/engine.py.229: process: 0.78420 | load: 0.00299 [I 2022-10-08 18:39:18] ...trap/engines/engine.py.230: loss: 0.17357 [I 2022-10-08 18:39:18] ...optimizers/trijoint.py.151: total_norm: 2.684503 [I 2022-10-08 18:39:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1560/2304 [I 2022-10-08 18:39:26] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:09:37 [I 2022-10-08 18:39:26] ...trap/engines/engine.py.229: process: 0.77067 | load: 0.00300 [I 2022-10-08 18:39:26] ...trap/engines/engine.py.230: loss: 0.20328 [I 2022-10-08 18:39:26] ...optimizers/trijoint.py.151: total_norm: 2.134502 [I 2022-10-08 18:39:34] ...trap/engines/engine.py.224: train: epoch 15 | batch 1570/2304 [I 2022-10-08 18:39:34] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:09:33 [I 2022-10-08 18:39:34] ...trap/engines/engine.py.229: process: 0.77489 | load: 0.00305 [I 2022-10-08 18:39:34] ...trap/engines/engine.py.230: loss: 0.19367 [I 2022-10-08 18:39:34] ...optimizers/trijoint.py.151: total_norm: 2.994172 [I 2022-10-08 18:39:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1580/2304 [I 2022-10-08 18:39:42] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:09:20 [I 2022-10-08 18:39:42] ...trap/engines/engine.py.229: process: 0.76575 | load: 0.00309 [I 2022-10-08 18:39:42] ...trap/engines/engine.py.230: loss: 0.23175 [I 2022-10-08 18:39:42] ...optimizers/trijoint.py.151: total_norm: 2.239546 [I 2022-10-08 18:39:49] ...trap/engines/engine.py.224: train: epoch 15 | batch 1590/2304 [I 2022-10-08 18:39:49] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:09:18 [I 2022-10-08 18:39:49] ...trap/engines/engine.py.229: process: 0.79184 | load: 0.00310 [I 2022-10-08 18:39:49] ...trap/engines/engine.py.230: loss: 0.16687 [I 2022-10-08 18:39:49] ...optimizers/trijoint.py.151: total_norm: 2.369262 [I 2022-10-08 18:39:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1600/2304 [I 2022-10-08 18:39:57] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:09:10 [I 2022-10-08 18:39:57] ...trap/engines/engine.py.229: process: 0.77579 | load: 0.00300 [I 2022-10-08 18:39:57] ...trap/engines/engine.py.230: loss: 0.17731 [I 2022-10-08 18:39:57] ...optimizers/trijoint.py.151: total_norm: 2.340747 [I 2022-10-08 18:40:05] ...trap/engines/engine.py.224: train: epoch 15 | batch 1610/2304 [I 2022-10-08 18:40:05] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:09:02 [I 2022-10-08 18:40:05] ...trap/engines/engine.py.229: process: 0.78858 | load: 0.00301 [I 2022-10-08 18:40:05] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-08 18:40:05] ...optimizers/trijoint.py.151: total_norm: 1.770696 [I 2022-10-08 18:40:13] ...trap/engines/engine.py.224: train: epoch 15 | batch 1620/2304 [I 2022-10-08 18:40:13] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:08:50 [I 2022-10-08 18:40:13] ...trap/engines/engine.py.229: process: 0.76832 | load: 0.00304 [I 2022-10-08 18:40:13] ...trap/engines/engine.py.230: loss: 0.18605 [I 2022-10-08 18:40:13] ...optimizers/trijoint.py.151: total_norm: 2.109303 [I 2022-10-08 18:40:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 1630/2304 [I 2022-10-08 18:40:21] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:08:45 [I 2022-10-08 18:40:21] ...trap/engines/engine.py.229: process: 0.77710 | load: 0.00302 [I 2022-10-08 18:40:21] ...trap/engines/engine.py.230: loss: 0.19116 [I 2022-10-08 18:40:21] ...optimizers/trijoint.py.151: total_norm: 3.044620 [I 2022-10-08 18:40:28] ...trap/engines/engine.py.224: train: epoch 15 | batch 1640/2304 [I 2022-10-08 18:40:28] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:08:36 [I 2022-10-08 18:40:28] ...trap/engines/engine.py.229: process: 0.78078 | load: 0.00306 [I 2022-10-08 18:40:28] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-08 18:40:28] ...optimizers/trijoint.py.151: total_norm: 2.664247 [I 2022-10-08 18:40:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 1650/2304 [I 2022-10-08 18:40:36] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:08:29 [I 2022-10-08 18:40:36] ...trap/engines/engine.py.229: process: 0.77939 | load: 0.00304 [I 2022-10-08 18:40:36] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-08 18:40:36] ...optimizers/trijoint.py.151: total_norm: 1.914962 [I 2022-10-08 18:40:44] ...trap/engines/engine.py.224: train: epoch 15 | batch 1660/2304 [I 2022-10-08 18:40:44] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:08:21 [I 2022-10-08 18:40:44] ...trap/engines/engine.py.229: process: 0.77666 | load: 0.00301 [I 2022-10-08 18:40:44] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-08 18:40:44] ...optimizers/trijoint.py.151: total_norm: 2.685477 [I 2022-10-08 18:40:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1670/2304 [I 2022-10-08 18:40:52] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:08:13 [I 2022-10-08 18:40:52] ...trap/engines/engine.py.229: process: 0.77690 | load: 0.00301 [I 2022-10-08 18:40:52] ...trap/engines/engine.py.230: loss: 0.19298 [I 2022-10-08 18:40:52] ...optimizers/trijoint.py.151: total_norm: 3.539638 [I 2022-10-08 18:41:00] ...trap/engines/engine.py.224: train: epoch 15 | batch 1680/2304 [I 2022-10-08 18:41:00] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:08:04 [I 2022-10-08 18:41:00] ...trap/engines/engine.py.229: process: 0.76486 | load: 0.00304 [I 2022-10-08 18:41:00] ...trap/engines/engine.py.230: loss: 0.24084 [I 2022-10-08 18:41:00] ...optimizers/trijoint.py.151: total_norm: 3.663070 [I 2022-10-08 18:41:08] ...trap/engines/engine.py.224: train: epoch 15 | batch 1690/2304 [I 2022-10-08 18:41:08] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:08:01 [I 2022-10-08 18:41:08] ...trap/engines/engine.py.229: process: 0.78626 | load: 0.00300 [I 2022-10-08 18:41:08] ...trap/engines/engine.py.230: loss: 0.23090 [I 2022-10-08 18:41:08] ...optimizers/trijoint.py.151: total_norm: 3.124480 [I 2022-10-08 18:41:15] ...trap/engines/engine.py.224: train: epoch 15 | batch 1700/2304 [I 2022-10-08 18:41:15] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:07:48 [I 2022-10-08 18:41:15] ...trap/engines/engine.py.229: process: 0.77733 | load: 0.00306 [I 2022-10-08 18:41:15] ...trap/engines/engine.py.230: loss: 0.24725 [I 2022-10-08 18:41:15] ...optimizers/trijoint.py.151: total_norm: 3.019372 [I 2022-10-08 18:41:23] ...trap/engines/engine.py.224: train: epoch 15 | batch 1710/2304 [I 2022-10-08 18:41:23] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:07:44 [I 2022-10-08 18:41:23] ...trap/engines/engine.py.229: process: 0.77904 | load: 0.00301 [I 2022-10-08 18:41:23] ...trap/engines/engine.py.230: loss: 0.17883 [I 2022-10-08 18:41:23] ...optimizers/trijoint.py.151: total_norm: 2.734535 [I 2022-10-08 18:41:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 1720/2304 [I 2022-10-08 18:41:31] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:07:33 [I 2022-10-08 18:41:31] ...trap/engines/engine.py.229: process: 0.76748 | load: 0.00300 [I 2022-10-08 18:41:31] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-08 18:41:31] ...optimizers/trijoint.py.151: total_norm: 2.763974 [I 2022-10-08 18:41:39] ...trap/engines/engine.py.224: train: epoch 15 | batch 1730/2304 [I 2022-10-08 18:41:39] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:07:27 [I 2022-10-08 18:41:39] ...trap/engines/engine.py.229: process: 0.78107 | load: 0.00304 [I 2022-10-08 18:41:39] ...trap/engines/engine.py.230: loss: 0.19885 [I 2022-10-08 18:41:39] ...optimizers/trijoint.py.151: total_norm: 2.273631 [I 2022-10-08 18:41:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1740/2304 [I 2022-10-08 18:41:47] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:07:21 [I 2022-10-08 18:41:47] ...trap/engines/engine.py.229: process: 0.78347 | load: 0.00307 [I 2022-10-08 18:41:47] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-08 18:41:47] ...optimizers/trijoint.py.151: total_norm: 2.402357 [I 2022-10-08 18:41:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 1750/2304 [I 2022-10-08 18:41:54] ...trap/engines/engine.py.228: elapsed: 0:23:08 | left: 0:07:12 [I 2022-10-08 18:41:54] ...trap/engines/engine.py.229: process: 0.77485 | load: 0.00302 [I 2022-10-08 18:41:54] ...trap/engines/engine.py.230: loss: 0.20940 [I 2022-10-08 18:41:54] ...optimizers/trijoint.py.151: total_norm: 2.736512 [I 2022-10-08 18:42:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1760/2304 [I 2022-10-08 18:42:02] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:07:04 [I 2022-10-08 18:42:02] ...trap/engines/engine.py.229: process: 0.79153 | load: 0.00301 [I 2022-10-08 18:42:02] ...trap/engines/engine.py.230: loss: 0.23180 [I 2022-10-08 18:42:02] ...optimizers/trijoint.py.151: total_norm: 3.222929 [I 2022-10-08 18:42:10] ...trap/engines/engine.py.224: train: epoch 15 | batch 1770/2304 [I 2022-10-08 18:42:10] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:06:58 [I 2022-10-08 18:42:10] ...trap/engines/engine.py.229: process: 0.79227 | load: 0.00305 [I 2022-10-08 18:42:10] ...trap/engines/engine.py.230: loss: 0.15763 [I 2022-10-08 18:42:10] ...optimizers/trijoint.py.151: total_norm: 2.632480 [I 2022-10-08 18:42:18] ...trap/engines/engine.py.224: train: epoch 15 | batch 1780/2304 [I 2022-10-08 18:42:18] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:06:48 [I 2022-10-08 18:42:18] ...trap/engines/engine.py.229: process: 0.78195 | load: 0.00302 [I 2022-10-08 18:42:18] ...trap/engines/engine.py.230: loss: 0.23112 [I 2022-10-08 18:42:18] ...optimizers/trijoint.py.151: total_norm: 2.871228 [I 2022-10-08 18:42:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1790/2304 [I 2022-10-08 18:42:26] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:06:38 [I 2022-10-08 18:42:26] ...trap/engines/engine.py.229: process: 0.77410 | load: 0.00303 [I 2022-10-08 18:42:26] ...trap/engines/engine.py.230: loss: 0.14321 [I 2022-10-08 18:42:26] ...optimizers/trijoint.py.151: total_norm: 2.207449 [I 2022-10-08 18:42:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 1800/2304 [I 2022-10-08 18:42:33] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:06:31 [I 2022-10-08 18:42:33] ...trap/engines/engine.py.229: process: 0.77369 | load: 0.00307 [I 2022-10-08 18:42:33] ...trap/engines/engine.py.230: loss: 0.17608 [I 2022-10-08 18:42:33] ...optimizers/trijoint.py.151: total_norm: 2.484381 [I 2022-10-08 18:42:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 1810/2304 [I 2022-10-08 18:42:41] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:06:27 [I 2022-10-08 18:42:41] ...trap/engines/engine.py.229: process: 0.78639 | load: 0.00302 [I 2022-10-08 18:42:41] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-08 18:42:41] ...optimizers/trijoint.py.151: total_norm: 2.036392 [I 2022-10-08 18:42:49] ...trap/engines/engine.py.224: train: epoch 15 | batch 1820/2304 [I 2022-10-08 18:42:49] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:06:15 [I 2022-10-08 18:42:49] ...trap/engines/engine.py.229: process: 0.78281 | load: 0.00303 [I 2022-10-08 18:42:49] ...trap/engines/engine.py.230: loss: 0.15237 [I 2022-10-08 18:42:49] ...optimizers/trijoint.py.151: total_norm: 1.772698 [I 2022-10-08 18:42:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1830/2304 [I 2022-10-08 18:42:57] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:06:08 [I 2022-10-08 18:42:57] ...trap/engines/engine.py.229: process: 0.78172 | load: 0.00303 [I 2022-10-08 18:42:57] ...trap/engines/engine.py.230: loss: 0.16550 [I 2022-10-08 18:42:57] ...optimizers/trijoint.py.151: total_norm: 2.388129 [I 2022-10-08 18:43:05] ...trap/engines/engine.py.224: train: epoch 15 | batch 1840/2304 [I 2022-10-08 18:43:05] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:06:01 [I 2022-10-08 18:43:05] ...trap/engines/engine.py.229: process: 0.77840 | load: 0.00301 [I 2022-10-08 18:43:05] ...trap/engines/engine.py.230: loss: 0.18394 [I 2022-10-08 18:43:05] ...optimizers/trijoint.py.151: total_norm: 2.662862 [I 2022-10-08 18:43:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1850/2304 [I 2022-10-08 18:43:12] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:05:52 [I 2022-10-08 18:43:12] ...trap/engines/engine.py.229: process: 0.78283 | load: 0.00308 [I 2022-10-08 18:43:12] ...trap/engines/engine.py.230: loss: 0.14787 [I 2022-10-08 18:43:12] ...optimizers/trijoint.py.151: total_norm: 2.766991 [I 2022-10-08 18:43:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 1860/2304 [I 2022-10-08 18:43:20] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:05:47 [I 2022-10-08 18:43:20] ...trap/engines/engine.py.229: process: 0.79300 | load: 0.00308 [I 2022-10-08 18:43:20] ...trap/engines/engine.py.230: loss: 0.18775 [I 2022-10-08 18:43:20] ...optimizers/trijoint.py.151: total_norm: 2.702940 [I 2022-10-08 18:43:28] ...trap/engines/engine.py.224: train: epoch 15 | batch 1870/2304 [I 2022-10-08 18:43:28] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:05:38 [I 2022-10-08 18:43:28] ...trap/engines/engine.py.229: process: 0.78237 | load: 0.00302 [I 2022-10-08 18:43:28] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-08 18:43:28] ...optimizers/trijoint.py.151: total_norm: 2.786798 [I 2022-10-08 18:43:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 1880/2304 [I 2022-10-08 18:43:36] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:05:31 [I 2022-10-08 18:43:36] ...trap/engines/engine.py.229: process: 0.77151 | load: 0.00302 [I 2022-10-08 18:43:36] ...trap/engines/engine.py.230: loss: 0.12318 [I 2022-10-08 18:43:36] ...optimizers/trijoint.py.151: total_norm: 2.246897 [I 2022-10-08 18:43:44] ...trap/engines/engine.py.224: train: epoch 15 | batch 1890/2304 [I 2022-10-08 18:43:44] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:05:22 [I 2022-10-08 18:43:44] ...trap/engines/engine.py.229: process: 0.78013 | load: 0.00305 [I 2022-10-08 18:43:44] ...trap/engines/engine.py.230: loss: 0.21584 [I 2022-10-08 18:43:44] ...optimizers/trijoint.py.151: total_norm: 2.674014 [I 2022-10-08 18:43:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 1900/2304 [I 2022-10-08 18:43:51] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:05:15 [I 2022-10-08 18:43:51] ...trap/engines/engine.py.229: process: 0.77537 | load: 0.00303 [I 2022-10-08 18:43:51] ...trap/engines/engine.py.230: loss: 0.16767 [I 2022-10-08 18:43:51] ...optimizers/trijoint.py.151: total_norm: 2.631575 [I 2022-10-08 18:43:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 1910/2304 [I 2022-10-08 18:43:59] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:05:07 [I 2022-10-08 18:43:59] ...trap/engines/engine.py.229: process: 0.78245 | load: 0.00308 [I 2022-10-08 18:43:59] ...trap/engines/engine.py.230: loss: 0.25571 [I 2022-10-08 18:43:59] ...optimizers/trijoint.py.151: total_norm: 3.431764 [I 2022-10-08 18:44:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1920/2304 [I 2022-10-08 18:44:07] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:04:59 [I 2022-10-08 18:44:07] ...trap/engines/engine.py.229: process: 0.78135 | load: 0.00308 [I 2022-10-08 18:44:07] ...trap/engines/engine.py.230: loss: 0.23224 [I 2022-10-08 18:44:07] ...optimizers/trijoint.py.151: total_norm: 2.377623 [I 2022-10-08 18:44:15] ...trap/engines/engine.py.224: train: epoch 15 | batch 1930/2304 [I 2022-10-08 18:44:15] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:04:51 [I 2022-10-08 18:44:15] ...trap/engines/engine.py.229: process: 0.77569 | load: 0.00304 [I 2022-10-08 18:44:15] ...trap/engines/engine.py.230: loss: 0.20656 [I 2022-10-08 18:44:15] ...optimizers/trijoint.py.151: total_norm: 2.973213 [I 2022-10-08 18:44:23] ...trap/engines/engine.py.224: train: epoch 15 | batch 1940/2304 [I 2022-10-08 18:44:23] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:04:42 [I 2022-10-08 18:44:23] ...trap/engines/engine.py.229: process: 0.76643 | load: 0.00304 [I 2022-10-08 18:44:23] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-08 18:44:23] ...optimizers/trijoint.py.151: total_norm: 2.201156 [I 2022-10-08 18:44:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 1950/2304 [I 2022-10-08 18:44:31] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:04:35 [I 2022-10-08 18:44:31] ...trap/engines/engine.py.229: process: 0.78577 | load: 0.00304 [I 2022-10-08 18:44:31] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-08 18:44:31] ...optimizers/trijoint.py.151: total_norm: 2.074512 [I 2022-10-08 18:44:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 1960/2304 [I 2022-10-08 18:44:38] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:04:26 [I 2022-10-08 18:44:38] ...trap/engines/engine.py.229: process: 0.77362 | load: 0.00310 [I 2022-10-08 18:44:38] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-08 18:44:38] ...optimizers/trijoint.py.151: total_norm: 2.459996 [I 2022-10-08 18:44:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 1970/2304 [I 2022-10-08 18:44:46] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:04:21 [I 2022-10-08 18:44:46] ...trap/engines/engine.py.229: process: 0.79508 | load: 0.00304 [I 2022-10-08 18:44:46] ...trap/engines/engine.py.230: loss: 0.12671 [I 2022-10-08 18:44:46] ...optimizers/trijoint.py.151: total_norm: 2.157839 [I 2022-10-08 18:44:54] ...trap/engines/engine.py.224: train: epoch 15 | batch 1980/2304 [I 2022-10-08 18:44:54] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:04:10 [I 2022-10-08 18:44:54] ...trap/engines/engine.py.229: process: 0.76129 | load: 0.00303 [I 2022-10-08 18:44:54] ...trap/engines/engine.py.230: loss: 0.19172 [I 2022-10-08 18:44:54] ...optimizers/trijoint.py.151: total_norm: 3.020203 [I 2022-10-08 18:45:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1990/2304 [I 2022-10-08 18:45:02] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:04:04 [I 2022-10-08 18:45:02] ...trap/engines/engine.py.229: process: 0.77749 | load: 0.00309 [I 2022-10-08 18:45:02] ...trap/engines/engine.py.230: loss: 0.15999 [I 2022-10-08 18:45:02] ...optimizers/trijoint.py.151: total_norm: 2.179576 [I 2022-10-08 18:45:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 2000/2304 [I 2022-10-08 18:45:09] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:03:55 [I 2022-10-08 18:45:09] ...trap/engines/engine.py.229: process: 0.76722 | load: 0.00295 [I 2022-10-08 18:45:09] ...trap/engines/engine.py.230: loss: 0.17726 [I 2022-10-08 18:45:09] ...optimizers/trijoint.py.151: total_norm: 2.609808 [I 2022-10-08 18:45:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 2010/2304 [I 2022-10-08 18:45:17] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:03:47 [I 2022-10-08 18:45:17] ...trap/engines/engine.py.229: process: 0.76798 | load: 0.00300 [I 2022-10-08 18:45:17] ...trap/engines/engine.py.230: loss: 0.15823 [I 2022-10-08 18:45:17] ...optimizers/trijoint.py.151: total_norm: 2.098552 [I 2022-10-08 18:45:25] ...trap/engines/engine.py.224: train: epoch 15 | batch 2020/2304 [I 2022-10-08 18:45:25] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:03:41 [I 2022-10-08 18:45:25] ...trap/engines/engine.py.229: process: 0.77883 | load: 0.00302 [I 2022-10-08 18:45:25] ...trap/engines/engine.py.230: loss: 0.12849 [I 2022-10-08 18:45:25] ...optimizers/trijoint.py.151: total_norm: 2.292464 [I 2022-10-08 18:45:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 2030/2304 [I 2022-10-08 18:45:33] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:03:34 [I 2022-10-08 18:45:33] ...trap/engines/engine.py.229: process: 0.77622 | load: 0.00302 [I 2022-10-08 18:45:33] ...trap/engines/engine.py.230: loss: 0.23127 [I 2022-10-08 18:45:33] ...optimizers/trijoint.py.151: total_norm: 3.322415 [I 2022-10-08 18:45:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 2040/2304 [I 2022-10-08 18:45:41] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:03:25 [I 2022-10-08 18:45:41] ...trap/engines/engine.py.229: process: 0.78852 | load: 0.00301 [I 2022-10-08 18:45:41] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-08 18:45:41] ...optimizers/trijoint.py.151: total_norm: 1.814525 [I 2022-10-08 18:45:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 2050/2304 [I 2022-10-08 18:45:48] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:03:16 [I 2022-10-08 18:45:48] ...trap/engines/engine.py.229: process: 0.77164 | load: 0.00299 [I 2022-10-08 18:45:48] ...trap/engines/engine.py.230: loss: 0.21132 [I 2022-10-08 18:45:48] ...optimizers/trijoint.py.151: total_norm: 2.876745 [I 2022-10-08 18:45:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 2060/2304 [I 2022-10-08 18:45:56] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:03:10 [I 2022-10-08 18:45:56] ...trap/engines/engine.py.229: process: 0.78440 | load: 0.00301 [I 2022-10-08 18:45:56] ...trap/engines/engine.py.230: loss: 0.23315 [I 2022-10-08 18:45:56] ...optimizers/trijoint.py.151: total_norm: 2.993675 [I 2022-10-08 18:46:04] ...trap/engines/engine.py.224: train: epoch 15 | batch 2070/2304 [I 2022-10-08 18:46:04] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:03:02 [I 2022-10-08 18:46:04] ...trap/engines/engine.py.229: process: 0.77446 | load: 0.00300 [I 2022-10-08 18:46:04] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-08 18:46:04] ...optimizers/trijoint.py.151: total_norm: 2.629008 [I 2022-10-08 18:46:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 2080/2304 [I 2022-10-08 18:46:12] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:02:54 [I 2022-10-08 18:46:12] ...trap/engines/engine.py.229: process: 0.78362 | load: 0.00303 [I 2022-10-08 18:46:12] ...trap/engines/engine.py.230: loss: 0.21313 [I 2022-10-08 18:46:12] ...optimizers/trijoint.py.151: total_norm: 2.397218 [I 2022-10-08 18:46:20] ...trap/engines/engine.py.224: train: epoch 15 | batch 2090/2304 [I 2022-10-08 18:46:20] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:02:46 [I 2022-10-08 18:46:20] ...trap/engines/engine.py.229: process: 0.77526 | load: 0.00303 [I 2022-10-08 18:46:20] ...trap/engines/engine.py.230: loss: 0.18773 [I 2022-10-08 18:46:20] ...optimizers/trijoint.py.151: total_norm: 2.190289 [I 2022-10-08 18:46:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 2100/2304 [I 2022-10-08 18:46:27] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:02:38 [I 2022-10-08 18:46:27] ...trap/engines/engine.py.229: process: 0.78555 | load: 0.00308 [I 2022-10-08 18:46:27] ...trap/engines/engine.py.230: loss: 0.13964 [I 2022-10-08 18:46:27] ...optimizers/trijoint.py.151: total_norm: 2.110753 [I 2022-10-08 18:46:35] ...trap/engines/engine.py.224: train: epoch 15 | batch 2110/2304 [I 2022-10-08 18:46:35] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:02:30 [I 2022-10-08 18:46:35] ...trap/engines/engine.py.229: process: 0.76972 | load: 0.00298 [I 2022-10-08 18:46:35] ...trap/engines/engine.py.230: loss: 0.17218 [I 2022-10-08 18:46:35] ...optimizers/trijoint.py.151: total_norm: 2.756300 [I 2022-10-08 18:46:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 2120/2304 [I 2022-10-08 18:46:43] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:02:23 [I 2022-10-08 18:46:43] ...trap/engines/engine.py.229: process: 0.77331 | load: 0.00308 [I 2022-10-08 18:46:43] ...trap/engines/engine.py.230: loss: 0.18933 [I 2022-10-08 18:46:43] ...optimizers/trijoint.py.151: total_norm: 2.460217 [I 2022-10-08 18:46:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 2130/2304 [I 2022-10-08 18:46:51] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:02:16 [I 2022-10-08 18:46:51] ...trap/engines/engine.py.229: process: 0.79685 | load: 0.00299 [I 2022-10-08 18:46:51] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-08 18:46:51] ...optimizers/trijoint.py.151: total_norm: 3.269701 [I 2022-10-08 18:46:59] ...trap/engines/engine.py.224: train: epoch 15 | batch 2140/2304 [I 2022-10-08 18:46:59] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:02:07 [I 2022-10-08 18:46:59] ...trap/engines/engine.py.229: process: 0.77413 | load: 0.00299 [I 2022-10-08 18:46:59] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-08 18:46:59] ...optimizers/trijoint.py.151: total_norm: 2.378661 [I 2022-10-08 18:47:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 2150/2304 [I 2022-10-08 18:47:06] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:01:59 [I 2022-10-08 18:47:06] ...trap/engines/engine.py.229: process: 0.77999 | load: 0.00300 [I 2022-10-08 18:47:06] ...trap/engines/engine.py.230: loss: 0.17848 [I 2022-10-08 18:47:06] ...optimizers/trijoint.py.151: total_norm: 2.605697 [I 2022-10-08 18:47:14] ...trap/engines/engine.py.224: train: epoch 15 | batch 2160/2304 [I 2022-10-08 18:47:14] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:01:51 [I 2022-10-08 18:47:14] ...trap/engines/engine.py.229: process: 0.77181 | load: 0.00303 [I 2022-10-08 18:47:14] ...trap/engines/engine.py.230: loss: 0.27146 [I 2022-10-08 18:47:14] ...optimizers/trijoint.py.151: total_norm: 4.970037 [I 2022-10-08 18:47:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 2170/2304 [I 2022-10-08 18:47:22] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:01:44 [I 2022-10-08 18:47:22] ...trap/engines/engine.py.229: process: 0.76625 | load: 0.00312 [I 2022-10-08 18:47:22] ...trap/engines/engine.py.230: loss: 0.18511 [I 2022-10-08 18:47:22] ...optimizers/trijoint.py.151: total_norm: 2.874118 [I 2022-10-08 18:47:30] ...trap/engines/engine.py.224: train: epoch 15 | batch 2180/2304 [I 2022-10-08 18:47:30] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:01:36 [I 2022-10-08 18:47:30] ...trap/engines/engine.py.229: process: 0.77400 | load: 0.00297 [I 2022-10-08 18:47:30] ...trap/engines/engine.py.230: loss: 0.17931 [I 2022-10-08 18:47:30] ...optimizers/trijoint.py.151: total_norm: 2.757446 [I 2022-10-08 18:47:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 2190/2304 [I 2022-10-08 18:47:38] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:01:28 [I 2022-10-08 18:47:38] ...trap/engines/engine.py.229: process: 0.76001 | load: 0.00306 [I 2022-10-08 18:47:38] ...trap/engines/engine.py.230: loss: 0.28535 [I 2022-10-08 18:47:38] ...optimizers/trijoint.py.151: total_norm: 2.774102 [I 2022-10-08 18:47:45] ...trap/engines/engine.py.224: train: epoch 15 | batch 2200/2304 [I 2022-10-08 18:47:45] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:01:20 [I 2022-10-08 18:47:45] ...trap/engines/engine.py.229: process: 0.76683 | load: 0.00302 [I 2022-10-08 18:47:45] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-08 18:47:45] ...optimizers/trijoint.py.151: total_norm: 2.592266 [I 2022-10-08 18:47:53] ...trap/engines/engine.py.224: train: epoch 15 | batch 2210/2304 [I 2022-10-08 18:47:53] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:01:13 [I 2022-10-08 18:47:53] ...trap/engines/engine.py.229: process: 0.78121 | load: 0.00300 [I 2022-10-08 18:47:53] ...trap/engines/engine.py.230: loss: 0.19408 [I 2022-10-08 18:47:53] ...optimizers/trijoint.py.151: total_norm: 2.989573 [I 2022-10-08 18:48:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 2220/2304 [I 2022-10-08 18:48:01] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:01:05 [I 2022-10-08 18:48:01] ...trap/engines/engine.py.229: process: 0.76197 | load: 0.00300 [I 2022-10-08 18:48:01] ...trap/engines/engine.py.230: loss: 0.21210 [I 2022-10-08 18:48:01] ...optimizers/trijoint.py.151: total_norm: 3.041911 [I 2022-10-08 18:48:09] ...trap/engines/engine.py.224: train: epoch 15 | batch 2230/2304 [I 2022-10-08 18:48:09] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:00:57 [I 2022-10-08 18:48:09] ...trap/engines/engine.py.229: process: 0.78183 | load: 0.00303 [I 2022-10-08 18:48:09] ...trap/engines/engine.py.230: loss: 0.19001 [I 2022-10-08 18:48:09] ...optimizers/trijoint.py.151: total_norm: 2.760302 [I 2022-10-08 18:48:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 2240/2304 [I 2022-10-08 18:48:17] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:00:50 [I 2022-10-08 18:48:17] ...trap/engines/engine.py.229: process: 0.77795 | load: 0.00304 [I 2022-10-08 18:48:17] ...trap/engines/engine.py.230: loss: 0.21913 [I 2022-10-08 18:48:17] ...optimizers/trijoint.py.151: total_norm: 1.969847 [I 2022-10-08 18:48:24] ...trap/engines/engine.py.224: train: epoch 15 | batch 2250/2304 [I 2022-10-08 18:48:24] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:00:41 [I 2022-10-08 18:48:24] ...trap/engines/engine.py.229: process: 0.77927 | load: 0.00305 [I 2022-10-08 18:48:24] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-08 18:48:24] ...optimizers/trijoint.py.151: total_norm: 2.777680 [I 2022-10-08 18:48:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 2260/2304 [I 2022-10-08 18:48:32] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:00:34 [I 2022-10-08 18:48:32] ...trap/engines/engine.py.229: process: 0.76302 | load: 0.00304 [I 2022-10-08 18:48:32] ...trap/engines/engine.py.230: loss: 0.17927 [I 2022-10-08 18:48:32] ...optimizers/trijoint.py.151: total_norm: 2.844721 [I 2022-10-08 18:48:40] ...trap/engines/engine.py.224: train: epoch 15 | batch 2270/2304 [I 2022-10-08 18:48:40] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:00:26 [I 2022-10-08 18:48:40] ...trap/engines/engine.py.229: process: 0.77637 | load: 0.00304 [I 2022-10-08 18:48:40] ...trap/engines/engine.py.230: loss: 0.29656 [I 2022-10-08 18:48:40] ...optimizers/trijoint.py.151: total_norm: 3.063476 [I 2022-10-08 18:48:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 2280/2304 [I 2022-10-08 18:48:48] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:00:18 [I 2022-10-08 18:48:48] ...trap/engines/engine.py.229: process: 0.78238 | load: 0.00309 [I 2022-10-08 18:48:48] ...trap/engines/engine.py.230: loss: 0.18399 [I 2022-10-08 18:48:48] ...optimizers/trijoint.py.151: total_norm: 2.718245 [I 2022-10-08 18:48:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 2290/2304 [I 2022-10-08 18:48:56] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:00:10 [I 2022-10-08 18:48:56] ...trap/engines/engine.py.229: process: 0.78288 | load: 0.00302 [I 2022-10-08 18:48:56] ...trap/engines/engine.py.230: loss: 0.19234 [I 2022-10-08 18:48:56] ...optimizers/trijoint.py.151: total_norm: 2.270439 [I 2022-10-08 18:49:03] ...trap/engines/engine.py.224: train: epoch 15 | batch 2300/2304 [I 2022-10-08 18:49:03] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:00:03 [I 2022-10-08 18:49:03] ...trap/engines/engine.py.229: process: 0.77260 | load: 0.00017 [I 2022-10-08 18:49:03] ...trap/engines/engine.py.230: loss: 0.15926 [I 2022-10-08 18:49:03] ...optimizers/trijoint.py.151: total_norm: 1.954748 [I 2022-10-08 18:49:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 2304/2304 [I 2022-10-08 18:49:06] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:00:00 [I 2022-10-08 18:49:06] ...trap/engines/engine.py.229: process: 0.75254 | load: 0.00017 [I 2022-10-08 18:49:06] ...trap/engines/engine.py.230: loss: 0.19369 [I 2022-10-08 18:49:06] ...optimizers/trijoint.py.151: total_norm: 3.070308 [S 2022-10-08 18:49:08] ...trap/engines/engine.py.129: train_epoch.epoch: 15 [S 2022-10-08 18:49:08] ...trap/engines/engine.py.129: train_epoch.loss: 0.18869521076793008 [I 2022-10-08 18:49:08] ...s/criterions/triplet.py.84: margin = 0.13000000000000006 [I 2022-10-08 18:49:08] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 15 [I 2022-10-08 18:49:08] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 18:49:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 18:49:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 18:49:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 18:49:10] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 18:49:11] ...trap/engines/engine.py.321: eval: epoch 15 | batch 0/510 [I 2022-10-08 18:49:11] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:24:50 [I 2022-10-08 18:49:11] ...trap/engines/engine.py.326: process: 2.92205 | load: 2.41284 [I 2022-10-08 18:49:16] ...trap/engines/engine.py.321: eval: epoch 15 | batch 10/510 [I 2022-10-08 18:49:16] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:19 [I 2022-10-08 18:49:16] ...trap/engines/engine.py.326: process: 0.60733 | load: 0.00018 [I 2022-10-08 18:49:21] ...trap/engines/engine.py.321: eval: epoch 15 | batch 20/510 [I 2022-10-08 18:49:21] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:26 [I 2022-10-08 18:49:21] ...trap/engines/engine.py.326: process: 0.52260 | load: 0.00018 [I 2022-10-08 18:49:27] ...trap/engines/engine.py.321: eval: epoch 15 | batch 30/510 [I 2022-10-08 18:49:27] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:32 [I 2022-10-08 18:49:27] ...trap/engines/engine.py.326: process: 0.77402 | load: 0.00019 [I 2022-10-08 18:49:32] ...trap/engines/engine.py.321: eval: epoch 15 | batch 40/510 [I 2022-10-08 18:49:32] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:03:41 [I 2022-10-08 18:49:32] ...trap/engines/engine.py.326: process: 0.45873 | load: 0.00019 [I 2022-10-08 18:49:37] ...trap/engines/engine.py.321: eval: epoch 15 | batch 50/510 [I 2022-10-08 18:49:37] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:00 [I 2022-10-08 18:49:37] ...trap/engines/engine.py.326: process: 0.50444 | load: 0.00019 [I 2022-10-08 18:49:42] ...trap/engines/engine.py.321: eval: epoch 15 | batch 60/510 [I 2022-10-08 18:49:42] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:41 [I 2022-10-08 18:49:42] ...trap/engines/engine.py.326: process: 0.50578 | load: 0.00019 [I 2022-10-08 18:49:47] ...trap/engines/engine.py.321: eval: epoch 15 | batch 70/510 [I 2022-10-08 18:49:47] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:04:06 [I 2022-10-08 18:49:47] ...trap/engines/engine.py.326: process: 0.58568 | load: 0.00019 [I 2022-10-08 18:49:52] ...trap/engines/engine.py.321: eval: epoch 15 | batch 80/510 [I 2022-10-08 18:49:52] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:41 [I 2022-10-08 18:49:52] ...trap/engines/engine.py.326: process: 0.40310 | load: 0.00020 [I 2022-10-08 18:49:57] ...trap/engines/engine.py.321: eval: epoch 15 | batch 90/510 [I 2022-10-08 18:49:57] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:06 [I 2022-10-08 18:49:57] ...trap/engines/engine.py.326: process: 0.44433 | load: 0.00019 [I 2022-10-08 18:50:01] ...trap/engines/engine.py.321: eval: epoch 15 | batch 100/510 [I 2022-10-08 18:50:01] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:50 [I 2022-10-08 18:50:01] ...trap/engines/engine.py.326: process: 0.30919 | load: 0.00020 [I 2022-10-08 18:50:04] ...trap/engines/engine.py.321: eval: epoch 15 | batch 110/510 [I 2022-10-08 18:50:04] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:10 [I 2022-10-08 18:50:04] ...trap/engines/engine.py.326: process: 0.31908 | load: 0.00020 [I 2022-10-08 18:50:08] ...trap/engines/engine.py.321: eval: epoch 15 | batch 120/510 [I 2022-10-08 18:50:08] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:04 [I 2022-10-08 18:50:08] ...trap/engines/engine.py.326: process: 0.31764 | load: 0.00020 [I 2022-10-08 18:50:11] ...trap/engines/engine.py.321: eval: epoch 15 | batch 130/510 [I 2022-10-08 18:50:11] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:00 [I 2022-10-08 18:50:11] ...trap/engines/engine.py.326: process: 0.32064 | load: 0.00020 [I 2022-10-08 18:50:14] ...trap/engines/engine.py.321: eval: epoch 15 | batch 140/510 [I 2022-10-08 18:50:14] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:56 [I 2022-10-08 18:50:14] ...trap/engines/engine.py.326: process: 0.30675 | load: 0.00019 [I 2022-10-08 18:50:17] ...trap/engines/engine.py.321: eval: epoch 15 | batch 150/510 [I 2022-10-08 18:50:17] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:54 [I 2022-10-08 18:50:17] ...trap/engines/engine.py.326: process: 0.31904 | load: 0.00019 [I 2022-10-08 18:50:20] ...trap/engines/engine.py.321: eval: epoch 15 | batch 160/510 [I 2022-10-08 18:50:20] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:50 [I 2022-10-08 18:50:20] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00020 [I 2022-10-08 18:50:24] ...trap/engines/engine.py.321: eval: epoch 15 | batch 170/510 [I 2022-10-08 18:50:24] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:47 [I 2022-10-08 18:50:24] ...trap/engines/engine.py.326: process: 0.31578 | load: 0.00019 [I 2022-10-08 18:50:27] ...trap/engines/engine.py.321: eval: epoch 15 | batch 180/510 [I 2022-10-08 18:50:27] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:43 [I 2022-10-08 18:50:27] ...trap/engines/engine.py.326: process: 0.30898 | load: 0.00019 [I 2022-10-08 18:50:30] ...trap/engines/engine.py.321: eval: epoch 15 | batch 190/510 [I 2022-10-08 18:50:30] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:40 [I 2022-10-08 18:50:30] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00019 [I 2022-10-08 18:50:33] ...trap/engines/engine.py.321: eval: epoch 15 | batch 200/510 [I 2022-10-08 18:50:33] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:38 [I 2022-10-08 18:50:33] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00020 [I 2022-10-08 18:50:36] ...trap/engines/engine.py.321: eval: epoch 15 | batch 210/510 [I 2022-10-08 18:50:36] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:35 [I 2022-10-08 18:50:36] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00020 [I 2022-10-08 18:50:39] ...trap/engines/engine.py.321: eval: epoch 15 | batch 220/510 [I 2022-10-08 18:50:39] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:32 [I 2022-10-08 18:50:39] ...trap/engines/engine.py.326: process: 0.31866 | load: 0.00021 [I 2022-10-08 18:50:43] ...trap/engines/engine.py.321: eval: epoch 15 | batch 230/510 [I 2022-10-08 18:50:43] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:28 [I 2022-10-08 18:50:43] ...trap/engines/engine.py.326: process: 0.31749 | load: 0.00020 [I 2022-10-08 18:50:46] ...trap/engines/engine.py.321: eval: epoch 15 | batch 240/510 [I 2022-10-08 18:50:46] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:25 [I 2022-10-08 18:50:46] ...trap/engines/engine.py.326: process: 0.31085 | load: 0.00019 [I 2022-10-08 18:50:49] ...trap/engines/engine.py.321: eval: epoch 15 | batch 250/510 [I 2022-10-08 18:50:49] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-08 18:50:49] ...trap/engines/engine.py.326: process: 0.31189 | load: 0.00021 [I 2022-10-08 18:50:52] ...trap/engines/engine.py.321: eval: epoch 15 | batch 260/510 [I 2022-10-08 18:50:52] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-08 18:50:52] ...trap/engines/engine.py.326: process: 0.31791 | load: 0.00020 [I 2022-10-08 18:50:55] ...trap/engines/engine.py.321: eval: epoch 15 | batch 270/510 [I 2022-10-08 18:50:55] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:16 [I 2022-10-08 18:50:55] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00021 [I 2022-10-08 18:50:58] ...trap/engines/engine.py.321: eval: epoch 15 | batch 280/510 [I 2022-10-08 18:50:58] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:12 [I 2022-10-08 18:50:58] ...trap/engines/engine.py.326: process: 0.30953 | load: 0.00020 [I 2022-10-08 18:51:02] ...trap/engines/engine.py.321: eval: epoch 15 | batch 290/510 [I 2022-10-08 18:51:02] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:10 [I 2022-10-08 18:51:02] ...trap/engines/engine.py.326: process: 0.32114 | load: 0.00020 [I 2022-10-08 18:51:05] ...trap/engines/engine.py.321: eval: epoch 15 | batch 300/510 [I 2022-10-08 18:51:05] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:07 [I 2022-10-08 18:51:05] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00020 [I 2022-10-08 18:51:08] ...trap/engines/engine.py.321: eval: epoch 15 | batch 310/510 [I 2022-10-08 18:51:08] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:03 [I 2022-10-08 18:51:08] ...trap/engines/engine.py.326: process: 0.31356 | load: 0.00020 [I 2022-10-08 18:51:11] ...trap/engines/engine.py.321: eval: epoch 15 | batch 320/510 [I 2022-10-08 18:51:11] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-08 18:51:11] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00019 [I 2022-10-08 18:51:14] ...trap/engines/engine.py.321: eval: epoch 15 | batch 330/510 [I 2022-10-08 18:51:14] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:57 [I 2022-10-08 18:51:14] ...trap/engines/engine.py.326: process: 0.32217 | load: 0.00021 [I 2022-10-08 18:51:17] ...trap/engines/engine.py.321: eval: epoch 15 | batch 340/510 [I 2022-10-08 18:51:17] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:53 [I 2022-10-08 18:51:17] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00020 [I 2022-10-08 18:51:21] ...trap/engines/engine.py.321: eval: epoch 15 | batch 350/510 [I 2022-10-08 18:51:21] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:50 [I 2022-10-08 18:51:21] ...trap/engines/engine.py.326: process: 0.31506 | load: 0.00021 [I 2022-10-08 18:51:24] ...trap/engines/engine.py.321: eval: epoch 15 | batch 360/510 [I 2022-10-08 18:51:24] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-08 18:51:24] ...trap/engines/engine.py.326: process: 0.31696 | load: 0.00020 [I 2022-10-08 18:51:27] ...trap/engines/engine.py.321: eval: epoch 15 | batch 370/510 [I 2022-10-08 18:51:27] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:44 [I 2022-10-08 18:51:27] ...trap/engines/engine.py.326: process: 0.32230 | load: 0.00020 [I 2022-10-08 18:51:30] ...trap/engines/engine.py.321: eval: epoch 15 | batch 380/510 [I 2022-10-08 18:51:30] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:40 [I 2022-10-08 18:51:30] ...trap/engines/engine.py.326: process: 0.31810 | load: 0.00019 [I 2022-10-08 18:51:33] ...trap/engines/engine.py.321: eval: epoch 15 | batch 390/510 [I 2022-10-08 18:51:33] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:38 [I 2022-10-08 18:51:33] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00019 [I 2022-10-08 18:51:37] ...trap/engines/engine.py.321: eval: epoch 15 | batch 400/510 [I 2022-10-08 18:51:37] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:35 [I 2022-10-08 18:51:37] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00020 [I 2022-10-08 18:51:40] ...trap/engines/engine.py.321: eval: epoch 15 | batch 410/510 [I 2022-10-08 18:51:40] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:31 [I 2022-10-08 18:51:40] ...trap/engines/engine.py.326: process: 0.31692 | load: 0.00020 [I 2022-10-08 18:51:43] ...trap/engines/engine.py.321: eval: epoch 15 | batch 420/510 [I 2022-10-08 18:51:43] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-08 18:51:43] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00021 [I 2022-10-08 18:51:46] ...trap/engines/engine.py.321: eval: epoch 15 | batch 430/510 [I 2022-10-08 18:51:46] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-08 18:51:46] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00021 [I 2022-10-08 18:51:49] ...trap/engines/engine.py.321: eval: epoch 15 | batch 440/510 [I 2022-10-08 18:51:49] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-08 18:51:49] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00020 [I 2022-10-08 18:51:53] ...trap/engines/engine.py.321: eval: epoch 15 | batch 450/510 [I 2022-10-08 18:51:53] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:19 [I 2022-10-08 18:51:53] ...trap/engines/engine.py.326: process: 0.32068 | load: 0.00020 [I 2022-10-08 18:51:56] ...trap/engines/engine.py.321: eval: epoch 15 | batch 460/510 [I 2022-10-08 18:51:56] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:16 [I 2022-10-08 18:51:56] ...trap/engines/engine.py.326: process: 0.32388 | load: 0.00020 [I 2022-10-08 18:51:59] ...trap/engines/engine.py.321: eval: epoch 15 | batch 470/510 [I 2022-10-08 18:51:59] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-08 18:51:59] ...trap/engines/engine.py.326: process: 0.30771 | load: 0.00020 [I 2022-10-08 18:52:02] ...trap/engines/engine.py.321: eval: epoch 15 | batch 480/510 [I 2022-10-08 18:52:02] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-08 18:52:02] ...trap/engines/engine.py.326: process: 0.32287 | load: 0.00020 [I 2022-10-08 18:52:05] ...trap/engines/engine.py.321: eval: epoch 15 | batch 490/510 [I 2022-10-08 18:52:05] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-08 18:52:05] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00020 [I 2022-10-08 18:52:08] ...trap/engines/engine.py.321: eval: epoch 15 | batch 500/510 [I 2022-10-08 18:52:08] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:03 [I 2022-10-08 18:52:08] ...trap/engines/engine.py.326: process: 0.31617 | load: 0.00020 [I 2022-10-08 18:52:12] ...trap/engines/engine.py.321: eval: epoch 15 | batch 510/510 [I 2022-10-08 18:52:12] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:00 [I 2022-10-08 18:52:12] ...trap/engines/engine.py.326: process: 0.31418 | load: 0.00016 [S 2022-10-08 18:52:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 15 [I 2022-10-08 18:52:12] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.590500 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.847800 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.907400 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.596100 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.851800 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.908700 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.019096 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006400 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009666 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013867 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010675 [S 2022-10-08 18:52:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007444 [W 2022-10-08 18:52:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [I 2022-10-08 18:52:45] ...trap/engines/engine.py.141: Saving last checkpoint [W 2022-10-08 18:52:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 18:52:45] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 18:52:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 18:52:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 18:52:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 18:52:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 18:52:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 16 [I 2022-10-08 18:52:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 18:52:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 0/2304 [I 2022-10-08 18:52:56] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:52:15 [I 2022-10-08 18:52:56] ...trap/engines/engine.py.229: process: 2.92338 | load: 2.06035 [I 2022-10-08 18:52:56] ...trap/engines/engine.py.230: loss: 0.23550 [I 2022-10-08 18:52:56] ...optimizers/trijoint.py.151: total_norm: 3.254798 [I 2022-10-08 18:53:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 10/2304 [I 2022-10-08 18:53:04] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:26 [I 2022-10-08 18:53:04] ...trap/engines/engine.py.229: process: 0.76289 | load: 0.00617 [I 2022-10-08 18:53:04] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-08 18:53:04] ...optimizers/trijoint.py.151: total_norm: 2.764688 [I 2022-10-08 18:53:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 20/2304 [I 2022-10-08 18:53:11] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:40 [I 2022-10-08 18:53:11] ...trap/engines/engine.py.229: process: 0.78499 | load: 0.00454 [I 2022-10-08 18:53:11] ...trap/engines/engine.py.230: loss: 0.22007 [I 2022-10-08 18:53:11] ...optimizers/trijoint.py.151: total_norm: 4.596614 [I 2022-10-08 18:53:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 30/2304 [I 2022-10-08 18:53:19] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:43 [I 2022-10-08 18:53:19] ...trap/engines/engine.py.229: process: 0.77354 | load: 0.00351 [I 2022-10-08 18:53:19] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-08 18:53:19] ...optimizers/trijoint.py.151: total_norm: 2.223051 [I 2022-10-08 18:53:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 40/2304 [I 2022-10-08 18:53:27] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:15 [I 2022-10-08 18:53:27] ...trap/engines/engine.py.229: process: 0.76777 | load: 0.00413 [I 2022-10-08 18:53:27] ...trap/engines/engine.py.230: loss: 0.17417 [I 2022-10-08 18:53:27] ...optimizers/trijoint.py.151: total_norm: 2.555397 [I 2022-10-08 18:53:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 50/2304 [I 2022-10-08 18:53:35] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:07 [I 2022-10-08 18:53:35] ...trap/engines/engine.py.229: process: 0.77149 | load: 0.00394 [I 2022-10-08 18:53:35] ...trap/engines/engine.py.230: loss: 0.13732 [I 2022-10-08 18:53:35] ...optimizers/trijoint.py.151: total_norm: 2.752621 [I 2022-10-08 18:53:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 60/2304 [I 2022-10-08 18:53:43] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:05 [I 2022-10-08 18:53:43] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00338 [I 2022-10-08 18:53:43] ...trap/engines/engine.py.230: loss: 0.16157 [I 2022-10-08 18:53:43] ...optimizers/trijoint.py.151: total_norm: 2.909801 [I 2022-10-08 18:53:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 70/2304 [I 2022-10-08 18:53:50] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:56 [I 2022-10-08 18:53:50] ...trap/engines/engine.py.229: process: 0.77745 | load: 0.00374 [I 2022-10-08 18:53:50] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-08 18:53:50] ...optimizers/trijoint.py.151: total_norm: 1.457114 [I 2022-10-08 18:53:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 80/2304 [I 2022-10-08 18:53:58] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:57 [I 2022-10-08 18:53:58] ...trap/engines/engine.py.229: process: 0.77959 | load: 0.00363 [I 2022-10-08 18:53:58] ...trap/engines/engine.py.230: loss: 0.12700 [I 2022-10-08 18:53:58] ...optimizers/trijoint.py.151: total_norm: 2.169767 [I 2022-10-08 18:54:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 90/2304 [I 2022-10-08 18:54:06] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:29:00 [I 2022-10-08 18:54:06] ...trap/engines/engine.py.229: process: 0.78538 | load: 0.00376 [I 2022-10-08 18:54:06] ...trap/engines/engine.py.230: loss: 0.23201 [I 2022-10-08 18:54:06] ...optimizers/trijoint.py.151: total_norm: 3.393382 [I 2022-10-08 18:54:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 100/2304 [I 2022-10-08 18:54:14] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:40 [I 2022-10-08 18:54:14] ...trap/engines/engine.py.229: process: 0.78031 | load: 0.00336 [I 2022-10-08 18:54:14] ...trap/engines/engine.py.230: loss: 0.17652 [I 2022-10-08 18:54:14] ...optimizers/trijoint.py.151: total_norm: 2.516603 [I 2022-10-08 18:54:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 110/2304 [I 2022-10-08 18:54:22] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:37 [I 2022-10-08 18:54:22] ...trap/engines/engine.py.229: process: 0.78446 | load: 0.00332 [I 2022-10-08 18:54:22] ...trap/engines/engine.py.230: loss: 0.25460 [I 2022-10-08 18:54:22] ...optimizers/trijoint.py.151: total_norm: 3.195517 [I 2022-10-08 18:54:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 120/2304 [I 2022-10-08 18:54:30] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:23 [I 2022-10-08 18:54:30] ...trap/engines/engine.py.229: process: 0.79214 | load: 0.00300 [I 2022-10-08 18:54:30] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-08 18:54:30] ...optimizers/trijoint.py.151: total_norm: 2.971601 [I 2022-10-08 18:54:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 130/2304 [I 2022-10-08 18:54:37] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:14 [I 2022-10-08 18:54:37] ...trap/engines/engine.py.229: process: 0.78064 | load: 0.00392 [I 2022-10-08 18:54:37] ...trap/engines/engine.py.230: loss: 0.26644 [I 2022-10-08 18:54:37] ...optimizers/trijoint.py.151: total_norm: 2.992666 [I 2022-10-08 18:54:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 140/2304 [I 2022-10-08 18:54:45] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:10 [I 2022-10-08 18:54:45] ...trap/engines/engine.py.229: process: 0.77494 | load: 0.00358 [I 2022-10-08 18:54:45] ...trap/engines/engine.py.230: loss: 0.22921 [I 2022-10-08 18:54:45] ...optimizers/trijoint.py.151: total_norm: 3.866979 [I 2022-10-08 18:54:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 150/2304 [I 2022-10-08 18:54:53] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:28:10 [I 2022-10-08 18:54:53] ...trap/engines/engine.py.229: process: 0.78425 | load: 0.00341 [I 2022-10-08 18:54:53] ...trap/engines/engine.py.230: loss: 0.22065 [I 2022-10-08 18:54:53] ...optimizers/trijoint.py.151: total_norm: 3.309733 [I 2022-10-08 18:55:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 160/2304 [I 2022-10-08 18:55:01] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:58 [I 2022-10-08 18:55:01] ...trap/engines/engine.py.229: process: 0.77072 | load: 0.00329 [I 2022-10-08 18:55:01] ...trap/engines/engine.py.230: loss: 0.22870 [I 2022-10-08 18:55:01] ...optimizers/trijoint.py.151: total_norm: 3.510611 [I 2022-10-08 18:55:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 170/2304 [I 2022-10-08 18:55:09] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:27:43 [I 2022-10-08 18:55:09] ...trap/engines/engine.py.229: process: 0.77053 | load: 0.00316 [I 2022-10-08 18:55:09] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-08 18:55:09] ...optimizers/trijoint.py.151: total_norm: 3.174432 [I 2022-10-08 18:55:17] ...trap/engines/engine.py.224: train: epoch 16 | batch 180/2304 [I 2022-10-08 18:55:17] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:33 [I 2022-10-08 18:55:17] ...trap/engines/engine.py.229: process: 0.78274 | load: 0.00302 [I 2022-10-08 18:55:17] ...trap/engines/engine.py.230: loss: 0.20935 [I 2022-10-08 18:55:17] ...optimizers/trijoint.py.151: total_norm: 4.325352 [I 2022-10-08 18:55:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 190/2304 [I 2022-10-08 18:55:25] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:33 [I 2022-10-08 18:55:25] ...trap/engines/engine.py.229: process: 0.78522 | load: 0.00321 [I 2022-10-08 18:55:25] ...trap/engines/engine.py.230: loss: 0.22275 [I 2022-10-08 18:55:25] ...optimizers/trijoint.py.151: total_norm: 3.958302 [I 2022-10-08 18:55:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 200/2304 [I 2022-10-08 18:55:32] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:12 [I 2022-10-08 18:55:32] ...trap/engines/engine.py.229: process: 0.77193 | load: 0.00313 [I 2022-10-08 18:55:32] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-08 18:55:32] ...optimizers/trijoint.py.151: total_norm: 2.784032 [I 2022-10-08 18:55:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 210/2304 [I 2022-10-08 18:55:40] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:15 [I 2022-10-08 18:55:40] ...trap/engines/engine.py.229: process: 0.77583 | load: 0.00323 [I 2022-10-08 18:55:40] ...trap/engines/engine.py.230: loss: 0.13033 [I 2022-10-08 18:55:40] ...optimizers/trijoint.py.151: total_norm: 1.903065 [I 2022-10-08 18:55:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 220/2304 [I 2022-10-08 18:55:48] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:27:07 [I 2022-10-08 18:55:48] ...trap/engines/engine.py.229: process: 0.78216 | load: 0.00312 [I 2022-10-08 18:55:48] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-08 18:55:48] ...optimizers/trijoint.py.151: total_norm: 1.887016 [I 2022-10-08 18:55:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 230/2304 [I 2022-10-08 18:55:56] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:26:59 [I 2022-10-08 18:55:56] ...trap/engines/engine.py.229: process: 0.78305 | load: 0.00316 [I 2022-10-08 18:55:56] ...trap/engines/engine.py.230: loss: 0.13052 [I 2022-10-08 18:55:56] ...optimizers/trijoint.py.151: total_norm: 2.301716 [I 2022-10-08 18:56:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 240/2304 [I 2022-10-08 18:56:04] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:50 [I 2022-10-08 18:56:04] ...trap/engines/engine.py.229: process: 0.79391 | load: 0.00312 [I 2022-10-08 18:56:04] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-08 18:56:04] ...optimizers/trijoint.py.151: total_norm: 3.000234 [I 2022-10-08 18:56:12] ...trap/engines/engine.py.224: train: epoch 16 | batch 250/2304 [I 2022-10-08 18:56:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:43 [I 2022-10-08 18:56:12] ...trap/engines/engine.py.229: process: 0.77043 | load: 0.00299 [I 2022-10-08 18:56:12] ...trap/engines/engine.py.230: loss: 0.09236 [I 2022-10-08 18:56:12] ...optimizers/trijoint.py.151: total_norm: 1.592837 [I 2022-10-08 18:56:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 260/2304 [I 2022-10-08 18:56:19] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:32 [I 2022-10-08 18:56:19] ...trap/engines/engine.py.229: process: 0.77968 | load: 0.00312 [I 2022-10-08 18:56:19] ...trap/engines/engine.py.230: loss: 0.14370 [I 2022-10-08 18:56:19] ...optimizers/trijoint.py.151: total_norm: 3.401520 [I 2022-10-08 18:56:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 270/2304 [I 2022-10-08 18:56:27] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:18 [I 2022-10-08 18:56:27] ...trap/engines/engine.py.229: process: 0.75935 | load: 0.00303 [I 2022-10-08 18:56:27] ...trap/engines/engine.py.230: loss: 0.26635 [I 2022-10-08 18:56:27] ...optimizers/trijoint.py.151: total_norm: 3.528690 [I 2022-10-08 18:56:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 280/2304 [I 2022-10-08 18:56:35] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:26:11 [I 2022-10-08 18:56:35] ...trap/engines/engine.py.229: process: 0.76766 | load: 0.00303 [I 2022-10-08 18:56:35] ...trap/engines/engine.py.230: loss: 0.15357 [I 2022-10-08 18:56:35] ...optimizers/trijoint.py.151: total_norm: 2.393217 [I 2022-10-08 18:56:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 290/2304 [I 2022-10-08 18:56:43] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:14 [I 2022-10-08 18:56:43] ...trap/engines/engine.py.229: process: 0.78444 | load: 0.00316 [I 2022-10-08 18:56:43] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-08 18:56:43] ...optimizers/trijoint.py.151: total_norm: 2.617515 [I 2022-10-08 18:56:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 300/2304 [I 2022-10-08 18:56:51] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:26:00 [I 2022-10-08 18:56:51] ...trap/engines/engine.py.229: process: 0.78587 | load: 0.00312 [I 2022-10-08 18:56:51] ...trap/engines/engine.py.230: loss: 0.26950 [I 2022-10-08 18:56:51] ...optimizers/trijoint.py.151: total_norm: 3.405988 [I 2022-10-08 18:56:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 310/2304 [I 2022-10-08 18:56:58] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:26:00 [I 2022-10-08 18:56:58] ...trap/engines/engine.py.229: process: 0.78456 | load: 0.00303 [I 2022-10-08 18:56:58] ...trap/engines/engine.py.230: loss: 0.15953 [I 2022-10-08 18:56:58] ...optimizers/trijoint.py.151: total_norm: 2.761644 [I 2022-10-08 18:57:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 320/2304 [I 2022-10-08 18:57:06] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:43 [I 2022-10-08 18:57:06] ...trap/engines/engine.py.229: process: 0.77751 | load: 0.00303 [I 2022-10-08 18:57:06] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-08 18:57:06] ...optimizers/trijoint.py.151: total_norm: 2.948882 [I 2022-10-08 18:57:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 330/2304 [I 2022-10-08 18:57:14] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:31 [I 2022-10-08 18:57:14] ...trap/engines/engine.py.229: process: 0.77272 | load: 0.00300 [I 2022-10-08 18:57:14] ...trap/engines/engine.py.230: loss: 0.20372 [I 2022-10-08 18:57:14] ...optimizers/trijoint.py.151: total_norm: 2.720340 [I 2022-10-08 18:57:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 340/2304 [I 2022-10-08 18:57:22] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:18 [I 2022-10-08 18:57:22] ...trap/engines/engine.py.229: process: 0.75996 | load: 0.00303 [I 2022-10-08 18:57:22] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-08 18:57:22] ...optimizers/trijoint.py.151: total_norm: 2.685891 [I 2022-10-08 18:57:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 350/2304 [I 2022-10-08 18:57:30] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:20 [I 2022-10-08 18:57:30] ...trap/engines/engine.py.229: process: 0.78490 | load: 0.00309 [I 2022-10-08 18:57:30] ...trap/engines/engine.py.230: loss: 0.18054 [I 2022-10-08 18:57:30] ...optimizers/trijoint.py.151: total_norm: 3.335314 [I 2022-10-08 18:57:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 360/2304 [I 2022-10-08 18:57:37] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:13 [I 2022-10-08 18:57:37] ...trap/engines/engine.py.229: process: 0.77992 | load: 0.00305 [I 2022-10-08 18:57:37] ...trap/engines/engine.py.230: loss: 0.18127 [I 2022-10-08 18:57:37] ...optimizers/trijoint.py.151: total_norm: 2.616192 [I 2022-10-08 18:57:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 370/2304 [I 2022-10-08 18:57:45] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:24:56 [I 2022-10-08 18:57:45] ...trap/engines/engine.py.229: process: 0.77858 | load: 0.00308 [I 2022-10-08 18:57:45] ...trap/engines/engine.py.230: loss: 0.18519 [I 2022-10-08 18:57:45] ...optimizers/trijoint.py.151: total_norm: 3.061375 [I 2022-10-08 18:57:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 380/2304 [I 2022-10-08 18:57:53] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:49 [I 2022-10-08 18:57:53] ...trap/engines/engine.py.229: process: 0.77523 | load: 0.00304 [I 2022-10-08 18:57:53] ...trap/engines/engine.py.230: loss: 0.25479 [I 2022-10-08 18:57:53] ...optimizers/trijoint.py.151: total_norm: 3.858159 [I 2022-10-08 18:58:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 390/2304 [I 2022-10-08 18:58:01] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:47 [I 2022-10-08 18:58:01] ...trap/engines/engine.py.229: process: 0.77631 | load: 0.00299 [I 2022-10-08 18:58:01] ...trap/engines/engine.py.230: loss: 0.19175 [I 2022-10-08 18:58:01] ...optimizers/trijoint.py.151: total_norm: 2.961743 [I 2022-10-08 18:58:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 400/2304 [I 2022-10-08 18:58:08] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:41 [I 2022-10-08 18:58:08] ...trap/engines/engine.py.229: process: 0.78710 | load: 0.00303 [I 2022-10-08 18:58:08] ...trap/engines/engine.py.230: loss: 0.17480 [I 2022-10-08 18:58:08] ...optimizers/trijoint.py.151: total_norm: 3.107385 [I 2022-10-08 18:58:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 410/2304 [I 2022-10-08 18:58:16] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:36 [I 2022-10-08 18:58:16] ...trap/engines/engine.py.229: process: 0.78002 | load: 0.00307 [I 2022-10-08 18:58:16] ...trap/engines/engine.py.230: loss: 0.16374 [I 2022-10-08 18:58:16] ...optimizers/trijoint.py.151: total_norm: 2.257812 [I 2022-10-08 18:58:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 420/2304 [I 2022-10-08 18:58:24] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:27 [I 2022-10-08 18:58:24] ...trap/engines/engine.py.229: process: 0.77073 | load: 0.00305 [I 2022-10-08 18:58:24] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-08 18:58:24] ...optimizers/trijoint.py.151: total_norm: 2.190846 [I 2022-10-08 18:58:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 430/2304 [I 2022-10-08 18:58:32] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:21 [I 2022-10-08 18:58:32] ...trap/engines/engine.py.229: process: 0.77742 | load: 0.00314 [I 2022-10-08 18:58:32] ...trap/engines/engine.py.230: loss: 0.15870 [I 2022-10-08 18:58:32] ...optimizers/trijoint.py.151: total_norm: 2.742292 [I 2022-10-08 18:58:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 440/2304 [I 2022-10-08 18:58:40] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:00 [I 2022-10-08 18:58:40] ...trap/engines/engine.py.229: process: 0.75951 | load: 0.00304 [I 2022-10-08 18:58:40] ...trap/engines/engine.py.230: loss: 0.20554 [I 2022-10-08 18:58:40] ...optimizers/trijoint.py.151: total_norm: 2.876790 [I 2022-10-08 18:58:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 450/2304 [I 2022-10-08 18:58:48] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:23:59 [I 2022-10-08 18:58:48] ...trap/engines/engine.py.229: process: 0.77122 | load: 0.00305 [I 2022-10-08 18:58:48] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-08 18:58:48] ...optimizers/trijoint.py.151: total_norm: 2.448974 [I 2022-10-08 18:58:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 460/2304 [I 2022-10-08 18:58:55] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:55 [I 2022-10-08 18:58:55] ...trap/engines/engine.py.229: process: 0.78561 | load: 0.00321 [I 2022-10-08 18:58:55] ...trap/engines/engine.py.230: loss: 0.19377 [I 2022-10-08 18:58:55] ...optimizers/trijoint.py.151: total_norm: 2.051494 [I 2022-10-08 18:59:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 470/2304 [I 2022-10-08 18:59:03] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:42 [I 2022-10-08 18:59:03] ...trap/engines/engine.py.229: process: 0.75763 | load: 0.00299 [I 2022-10-08 18:59:03] ...trap/engines/engine.py.230: loss: 0.14094 [I 2022-10-08 18:59:03] ...optimizers/trijoint.py.151: total_norm: 1.950192 [I 2022-10-08 18:59:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 480/2304 [I 2022-10-08 18:59:11] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:36 [I 2022-10-08 18:59:11] ...trap/engines/engine.py.229: process: 0.77509 | load: 0.00303 [I 2022-10-08 18:59:11] ...trap/engines/engine.py.230: loss: 0.23109 [I 2022-10-08 18:59:11] ...optimizers/trijoint.py.151: total_norm: 2.658019 [I 2022-10-08 18:59:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 490/2304 [I 2022-10-08 18:59:19] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:37 [I 2022-10-08 18:59:19] ...trap/engines/engine.py.229: process: 0.77574 | load: 0.00304 [I 2022-10-08 18:59:19] ...trap/engines/engine.py.230: loss: 0.21255 [I 2022-10-08 18:59:19] ...optimizers/trijoint.py.151: total_norm: 3.880666 [I 2022-10-08 18:59:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 500/2304 [I 2022-10-08 18:59:27] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:23:24 [I 2022-10-08 18:59:27] ...trap/engines/engine.py.229: process: 0.77865 | load: 0.00306 [I 2022-10-08 18:59:27] ...trap/engines/engine.py.230: loss: 0.18466 [I 2022-10-08 18:59:27] ...optimizers/trijoint.py.151: total_norm: 2.704883 [I 2022-10-08 18:59:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 510/2304 [I 2022-10-08 18:59:35] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:17 [I 2022-10-08 18:59:35] ...trap/engines/engine.py.229: process: 0.78613 | load: 0.00308 [I 2022-10-08 18:59:35] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-08 18:59:35] ...optimizers/trijoint.py.151: total_norm: 2.029253 [I 2022-10-08 18:59:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 520/2304 [I 2022-10-08 18:59:42] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:00 [I 2022-10-08 18:59:42] ...trap/engines/engine.py.229: process: 0.76963 | load: 0.00302 [I 2022-10-08 18:59:42] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-08 18:59:42] ...optimizers/trijoint.py.151: total_norm: 2.432487 [I 2022-10-08 18:59:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 530/2304 [I 2022-10-08 18:59:50] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:23:00 [I 2022-10-08 18:59:50] ...trap/engines/engine.py.229: process: 0.76787 | load: 0.00305 [I 2022-10-08 18:59:50] ...trap/engines/engine.py.230: loss: 0.26067 [I 2022-10-08 18:59:50] ...optimizers/trijoint.py.151: total_norm: 3.584462 [I 2022-10-08 18:59:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 540/2304 [I 2022-10-08 18:59:58] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:53 [I 2022-10-08 18:59:58] ...trap/engines/engine.py.229: process: 0.78061 | load: 0.00305 [I 2022-10-08 18:59:58] ...trap/engines/engine.py.230: loss: 0.11232 [I 2022-10-08 18:59:58] ...optimizers/trijoint.py.151: total_norm: 1.879223 [I 2022-10-08 19:00:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 550/2304 [I 2022-10-08 19:00:06] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:22:48 [I 2022-10-08 19:00:06] ...trap/engines/engine.py.229: process: 0.78104 | load: 0.00298 [I 2022-10-08 19:00:06] ...trap/engines/engine.py.230: loss: 0.17651 [I 2022-10-08 19:00:06] ...optimizers/trijoint.py.151: total_norm: 2.425764 [I 2022-10-08 19:00:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 560/2304 [I 2022-10-08 19:00:14] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:38 [I 2022-10-08 19:00:14] ...trap/engines/engine.py.229: process: 0.78722 | load: 0.00301 [I 2022-10-08 19:00:14] ...trap/engines/engine.py.230: loss: 0.18681 [I 2022-10-08 19:00:14] ...optimizers/trijoint.py.151: total_norm: 2.466261 [I 2022-10-08 19:00:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 570/2304 [I 2022-10-08 19:00:22] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:31 [I 2022-10-08 19:00:22] ...trap/engines/engine.py.229: process: 0.77450 | load: 0.00307 [I 2022-10-08 19:00:22] ...trap/engines/engine.py.230: loss: 0.17970 [I 2022-10-08 19:00:22] ...optimizers/trijoint.py.151: total_norm: 2.728098 [I 2022-10-08 19:00:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 580/2304 [I 2022-10-08 19:00:29] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:20 [I 2022-10-08 19:00:29] ...trap/engines/engine.py.229: process: 0.78766 | load: 0.00303 [I 2022-10-08 19:00:29] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-08 19:00:29] ...optimizers/trijoint.py.151: total_norm: 2.365773 [I 2022-10-08 19:00:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 590/2304 [I 2022-10-08 19:00:37] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:17 [I 2022-10-08 19:00:37] ...trap/engines/engine.py.229: process: 0.78390 | load: 0.00308 [I 2022-10-08 19:00:37] ...trap/engines/engine.py.230: loss: 0.17949 [I 2022-10-08 19:00:37] ...optimizers/trijoint.py.151: total_norm: 2.585703 [I 2022-10-08 19:00:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 600/2304 [I 2022-10-08 19:00:45] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:00 [I 2022-10-08 19:00:45] ...trap/engines/engine.py.229: process: 0.76336 | load: 0.00302 [I 2022-10-08 19:00:45] ...trap/engines/engine.py.230: loss: 0.33437 [I 2022-10-08 19:00:45] ...optimizers/trijoint.py.151: total_norm: 4.549393 [I 2022-10-08 19:00:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 610/2304 [I 2022-10-08 19:00:53] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:00 [I 2022-10-08 19:00:53] ...trap/engines/engine.py.229: process: 0.77794 | load: 0.00303 [I 2022-10-08 19:00:53] ...trap/engines/engine.py.230: loss: 0.20025 [I 2022-10-08 19:00:53] ...optimizers/trijoint.py.151: total_norm: 2.464182 [I 2022-10-08 19:01:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 620/2304 [I 2022-10-08 19:01:01] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:54 [I 2022-10-08 19:01:01] ...trap/engines/engine.py.229: process: 0.79540 | load: 0.00304 [I 2022-10-08 19:01:01] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-08 19:01:01] ...optimizers/trijoint.py.151: total_norm: 2.585247 [I 2022-10-08 19:01:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 630/2304 [I 2022-10-08 19:01:08] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:43 [I 2022-10-08 19:01:08] ...trap/engines/engine.py.229: process: 0.77718 | load: 0.00300 [I 2022-10-08 19:01:08] ...trap/engines/engine.py.230: loss: 0.20863 [I 2022-10-08 19:01:08] ...optimizers/trijoint.py.151: total_norm: 2.671095 [I 2022-10-08 19:01:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 640/2304 [I 2022-10-08 19:01:16] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:34 [I 2022-10-08 19:01:16] ...trap/engines/engine.py.229: process: 0.77528 | load: 0.00304 [I 2022-10-08 19:01:16] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-08 19:01:16] ...optimizers/trijoint.py.151: total_norm: 2.883332 [I 2022-10-08 19:01:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 650/2304 [I 2022-10-08 19:01:24] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:27 [I 2022-10-08 19:01:24] ...trap/engines/engine.py.229: process: 0.78421 | load: 0.00300 [I 2022-10-08 19:01:24] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-08 19:01:24] ...optimizers/trijoint.py.151: total_norm: 3.514750 [I 2022-10-08 19:01:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 660/2304 [I 2022-10-08 19:01:32] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:22 [I 2022-10-08 19:01:32] ...trap/engines/engine.py.229: process: 0.77816 | load: 0.00313 [I 2022-10-08 19:01:32] ...trap/engines/engine.py.230: loss: 0.23161 [I 2022-10-08 19:01:32] ...optimizers/trijoint.py.151: total_norm: 2.335007 [I 2022-10-08 19:01:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 670/2304 [I 2022-10-08 19:01:40] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:15 [I 2022-10-08 19:01:40] ...trap/engines/engine.py.229: process: 0.77954 | load: 0.00304 [I 2022-10-08 19:01:40] ...trap/engines/engine.py.230: loss: 0.18033 [I 2022-10-08 19:01:40] ...optimizers/trijoint.py.151: total_norm: 2.409989 [I 2022-10-08 19:01:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 680/2304 [I 2022-10-08 19:01:47] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:06 [I 2022-10-08 19:01:47] ...trap/engines/engine.py.229: process: 0.77910 | load: 0.00300 [I 2022-10-08 19:01:47] ...trap/engines/engine.py.230: loss: 0.16787 [I 2022-10-08 19:01:47] ...optimizers/trijoint.py.151: total_norm: 2.580781 [I 2022-10-08 19:01:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 690/2304 [I 2022-10-08 19:01:55] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:59 [I 2022-10-08 19:01:55] ...trap/engines/engine.py.229: process: 0.77913 | load: 0.00303 [I 2022-10-08 19:01:55] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-08 19:01:55] ...optimizers/trijoint.py.151: total_norm: 3.222934 [I 2022-10-08 19:02:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 700/2304 [I 2022-10-08 19:02:03] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:48 [I 2022-10-08 19:02:03] ...trap/engines/engine.py.229: process: 0.78189 | load: 0.00311 [I 2022-10-08 19:02:03] ...trap/engines/engine.py.230: loss: 0.18701 [I 2022-10-08 19:02:03] ...optimizers/trijoint.py.151: total_norm: 2.455766 [I 2022-10-08 19:02:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 710/2304 [I 2022-10-08 19:02:11] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:36 [I 2022-10-08 19:02:11] ...trap/engines/engine.py.229: process: 0.77389 | load: 0.00304 [I 2022-10-08 19:02:11] ...trap/engines/engine.py.230: loss: 0.15546 [I 2022-10-08 19:02:11] ...optimizers/trijoint.py.151: total_norm: 2.155500 [I 2022-10-08 19:02:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 720/2304 [I 2022-10-08 19:02:19] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:37 [I 2022-10-08 19:02:19] ...trap/engines/engine.py.229: process: 0.78383 | load: 0.00300 [I 2022-10-08 19:02:19] ...trap/engines/engine.py.230: loss: 0.23634 [I 2022-10-08 19:02:19] ...optimizers/trijoint.py.151: total_norm: 4.330946 [I 2022-10-08 19:02:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 730/2304 [I 2022-10-08 19:02:26] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:31 [I 2022-10-08 19:02:26] ...trap/engines/engine.py.229: process: 0.78875 | load: 0.00303 [I 2022-10-08 19:02:26] ...trap/engines/engine.py.230: loss: 0.14984 [I 2022-10-08 19:02:26] ...optimizers/trijoint.py.151: total_norm: 2.251627 [I 2022-10-08 19:02:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 740/2304 [I 2022-10-08 19:02:34] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:21 [I 2022-10-08 19:02:34] ...trap/engines/engine.py.229: process: 0.78028 | load: 0.00304 [I 2022-10-08 19:02:34] ...trap/engines/engine.py.230: loss: 0.21159 [I 2022-10-08 19:02:34] ...optimizers/trijoint.py.151: total_norm: 3.068021 [I 2022-10-08 19:02:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 750/2304 [I 2022-10-08 19:02:42] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:00 [I 2022-10-08 19:02:42] ...trap/engines/engine.py.229: process: 0.76223 | load: 0.00297 [I 2022-10-08 19:02:42] ...trap/engines/engine.py.230: loss: 0.26253 [I 2022-10-08 19:02:42] ...optimizers/trijoint.py.151: total_norm: 2.910745 [I 2022-10-08 19:02:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 760/2304 [I 2022-10-08 19:02:50] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:19:47 [I 2022-10-08 19:02:50] ...trap/engines/engine.py.229: process: 0.76614 | load: 0.00306 [I 2022-10-08 19:02:50] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-08 19:02:50] ...optimizers/trijoint.py.151: total_norm: 2.905815 [I 2022-10-08 19:02:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 770/2304 [I 2022-10-08 19:02:57] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:44 [I 2022-10-08 19:02:57] ...trap/engines/engine.py.229: process: 0.77629 | load: 0.00307 [I 2022-10-08 19:02:57] ...trap/engines/engine.py.230: loss: 0.20527 [I 2022-10-08 19:02:57] ...optimizers/trijoint.py.151: total_norm: 3.021873 [I 2022-10-08 19:03:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 780/2304 [I 2022-10-08 19:03:05] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:47 [I 2022-10-08 19:03:05] ...trap/engines/engine.py.229: process: 0.78393 | load: 0.00304 [I 2022-10-08 19:03:05] ...trap/engines/engine.py.230: loss: 0.17454 [I 2022-10-08 19:03:05] ...optimizers/trijoint.py.151: total_norm: 2.257748 [I 2022-10-08 19:03:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 790/2304 [I 2022-10-08 19:03:13] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:41 [I 2022-10-08 19:03:13] ...trap/engines/engine.py.229: process: 0.78976 | load: 0.00307 [I 2022-10-08 19:03:13] ...trap/engines/engine.py.230: loss: 0.14938 [I 2022-10-08 19:03:13] ...optimizers/trijoint.py.151: total_norm: 2.231535 [I 2022-10-08 19:03:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 800/2304 [I 2022-10-08 19:03:21] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:28 [I 2022-10-08 19:03:21] ...trap/engines/engine.py.229: process: 0.78257 | load: 0.00307 [I 2022-10-08 19:03:21] ...trap/engines/engine.py.230: loss: 0.23883 [I 2022-10-08 19:03:21] ...optimizers/trijoint.py.151: total_norm: 3.551385 [I 2022-10-08 19:03:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 810/2304 [I 2022-10-08 19:03:29] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:21 [I 2022-10-08 19:03:29] ...trap/engines/engine.py.229: process: 0.77829 | load: 0.00304 [I 2022-10-08 19:03:29] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-08 19:03:29] ...optimizers/trijoint.py.151: total_norm: 3.028503 [I 2022-10-08 19:03:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 820/2304 [I 2022-10-08 19:03:36] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:08 [I 2022-10-08 19:03:36] ...trap/engines/engine.py.229: process: 0.77632 | load: 0.00302 [I 2022-10-08 19:03:36] ...trap/engines/engine.py.230: loss: 0.20989 [I 2022-10-08 19:03:36] ...optimizers/trijoint.py.151: total_norm: 2.476160 [I 2022-10-08 19:03:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 830/2304 [I 2022-10-08 19:03:44] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:06 [I 2022-10-08 19:03:44] ...trap/engines/engine.py.229: process: 0.76725 | load: 0.00310 [I 2022-10-08 19:03:44] ...trap/engines/engine.py.230: loss: 0.14655 [I 2022-10-08 19:03:44] ...optimizers/trijoint.py.151: total_norm: 1.924836 [I 2022-10-08 19:03:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 840/2304 [I 2022-10-08 19:03:52] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:18:52 [I 2022-10-08 19:03:52] ...trap/engines/engine.py.229: process: 0.78196 | load: 0.00296 [I 2022-10-08 19:03:52] ...trap/engines/engine.py.230: loss: 0.24159 [I 2022-10-08 19:03:52] ...optimizers/trijoint.py.151: total_norm: 3.507398 [I 2022-10-08 19:04:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 850/2304 [I 2022-10-08 19:04:00] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:49 [I 2022-10-08 19:04:00] ...trap/engines/engine.py.229: process: 0.77239 | load: 0.00306 [I 2022-10-08 19:04:00] ...trap/engines/engine.py.230: loss: 0.24590 [I 2022-10-08 19:04:00] ...optimizers/trijoint.py.151: total_norm: 3.579283 [I 2022-10-08 19:04:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 860/2304 [I 2022-10-08 19:04:08] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:46 [I 2022-10-08 19:04:08] ...trap/engines/engine.py.229: process: 0.77119 | load: 0.00308 [I 2022-10-08 19:04:08] ...trap/engines/engine.py.230: loss: 0.23765 [I 2022-10-08 19:04:08] ...optimizers/trijoint.py.151: total_norm: 3.723701 [I 2022-10-08 19:04:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 870/2304 [I 2022-10-08 19:04:15] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:35 [I 2022-10-08 19:04:15] ...trap/engines/engine.py.229: process: 0.77551 | load: 0.00303 [I 2022-10-08 19:04:15] ...trap/engines/engine.py.230: loss: 0.25785 [I 2022-10-08 19:04:15] ...optimizers/trijoint.py.151: total_norm: 3.570703 [I 2022-10-08 19:04:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 880/2304 [I 2022-10-08 19:04:23] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:32 [I 2022-10-08 19:04:23] ...trap/engines/engine.py.229: process: 0.78620 | load: 0.00319 [I 2022-10-08 19:04:23] ...trap/engines/engine.py.230: loss: 0.17696 [I 2022-10-08 19:04:23] ...optimizers/trijoint.py.151: total_norm: 2.812203 [I 2022-10-08 19:04:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 890/2304 [I 2022-10-08 19:04:31] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:21 [I 2022-10-08 19:04:31] ...trap/engines/engine.py.229: process: 0.77583 | load: 0.00301 [I 2022-10-08 19:04:31] ...trap/engines/engine.py.230: loss: 0.27206 [I 2022-10-08 19:04:31] ...optimizers/trijoint.py.151: total_norm: 2.577047 [I 2022-10-08 19:04:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 900/2304 [I 2022-10-08 19:04:39] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:14 [I 2022-10-08 19:04:39] ...trap/engines/engine.py.229: process: 0.77446 | load: 0.00309 [I 2022-10-08 19:04:39] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-08 19:04:39] ...optimizers/trijoint.py.151: total_norm: 2.179024 [I 2022-10-08 19:04:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 910/2304 [I 2022-10-08 19:04:47] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:09 [I 2022-10-08 19:04:47] ...trap/engines/engine.py.229: process: 0.78028 | load: 0.00302 [I 2022-10-08 19:04:47] ...trap/engines/engine.py.230: loss: 0.24748 [I 2022-10-08 19:04:47] ...optimizers/trijoint.py.151: total_norm: 4.193675 [I 2022-10-08 19:04:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 920/2304 [I 2022-10-08 19:04:55] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:18:03 [I 2022-10-08 19:04:55] ...trap/engines/engine.py.229: process: 0.79209 | load: 0.00306 [I 2022-10-08 19:04:55] ...trap/engines/engine.py.230: loss: 0.21151 [I 2022-10-08 19:04:55] ...optimizers/trijoint.py.151: total_norm: 3.117151 [I 2022-10-08 19:05:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 930/2304 [I 2022-10-08 19:05:03] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:50 [I 2022-10-08 19:05:03] ...trap/engines/engine.py.229: process: 0.77013 | load: 0.00305 [I 2022-10-08 19:05:03] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-08 19:05:03] ...optimizers/trijoint.py.151: total_norm: 2.002620 [I 2022-10-08 19:05:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 940/2304 [I 2022-10-08 19:05:10] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:47 [I 2022-10-08 19:05:10] ...trap/engines/engine.py.229: process: 0.77954 | load: 0.00317 [I 2022-10-08 19:05:10] ...trap/engines/engine.py.230: loss: 0.23249 [I 2022-10-08 19:05:10] ...optimizers/trijoint.py.151: total_norm: 3.636918 [I 2022-10-08 19:05:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 950/2304 [I 2022-10-08 19:05:18] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:32 [I 2022-10-08 19:05:18] ...trap/engines/engine.py.229: process: 0.76667 | load: 0.00301 [I 2022-10-08 19:05:18] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-08 19:05:18] ...optimizers/trijoint.py.151: total_norm: 2.332859 [I 2022-10-08 19:05:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 960/2304 [I 2022-10-08 19:05:26] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:27 [I 2022-10-08 19:05:26] ...trap/engines/engine.py.229: process: 0.78216 | load: 0.00305 [I 2022-10-08 19:05:26] ...trap/engines/engine.py.230: loss: 0.17414 [I 2022-10-08 19:05:26] ...optimizers/trijoint.py.151: total_norm: 2.502386 [I 2022-10-08 19:05:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 970/2304 [I 2022-10-08 19:05:34] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:19 [I 2022-10-08 19:05:34] ...trap/engines/engine.py.229: process: 0.77678 | load: 0.00301 [I 2022-10-08 19:05:34] ...trap/engines/engine.py.230: loss: 0.17081 [I 2022-10-08 19:05:34] ...optimizers/trijoint.py.151: total_norm: 2.257439 [I 2022-10-08 19:05:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 980/2304 [I 2022-10-08 19:05:42] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:10 [I 2022-10-08 19:05:42] ...trap/engines/engine.py.229: process: 0.77962 | load: 0.00302 [I 2022-10-08 19:05:42] ...trap/engines/engine.py.230: loss: 0.13793 [I 2022-10-08 19:05:42] ...optimizers/trijoint.py.151: total_norm: 2.102277 [I 2022-10-08 19:05:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 990/2304 [I 2022-10-08 19:05:49] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:17:07 [I 2022-10-08 19:05:49] ...trap/engines/engine.py.229: process: 0.79772 | load: 0.00303 [I 2022-10-08 19:05:49] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-08 19:05:49] ...optimizers/trijoint.py.151: total_norm: 1.783900 [I 2022-10-08 19:05:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 1000/2304 [I 2022-10-08 19:05:57] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:16:54 [I 2022-10-08 19:05:57] ...trap/engines/engine.py.229: process: 0.75797 | load: 0.00302 [I 2022-10-08 19:05:57] ...trap/engines/engine.py.230: loss: 0.26194 [I 2022-10-08 19:05:57] ...optimizers/trijoint.py.151: total_norm: 2.695955 [I 2022-10-08 19:06:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1010/2304 [I 2022-10-08 19:06:05] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:54 [I 2022-10-08 19:06:05] ...trap/engines/engine.py.229: process: 0.77998 | load: 0.00303 [I 2022-10-08 19:06:05] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-08 19:06:05] ...optimizers/trijoint.py.151: total_norm: 1.619706 [I 2022-10-08 19:06:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 1020/2304 [I 2022-10-08 19:06:13] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:40 [I 2022-10-08 19:06:13] ...trap/engines/engine.py.229: process: 0.78972 | load: 0.00305 [I 2022-10-08 19:06:13] ...trap/engines/engine.py.230: loss: 0.19345 [I 2022-10-08 19:06:13] ...optimizers/trijoint.py.151: total_norm: 2.453506 [I 2022-10-08 19:06:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 1030/2304 [I 2022-10-08 19:06:21] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:37 [I 2022-10-08 19:06:21] ...trap/engines/engine.py.229: process: 0.78247 | load: 0.00305 [I 2022-10-08 19:06:21] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-08 19:06:21] ...optimizers/trijoint.py.151: total_norm: 2.185114 [I 2022-10-08 19:06:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 1040/2304 [I 2022-10-08 19:06:28] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:25 [I 2022-10-08 19:06:28] ...trap/engines/engine.py.229: process: 0.78242 | load: 0.00303 [I 2022-10-08 19:06:28] ...trap/engines/engine.py.230: loss: 0.20256 [I 2022-10-08 19:06:28] ...optimizers/trijoint.py.151: total_norm: 2.927153 [I 2022-10-08 19:06:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 1050/2304 [I 2022-10-08 19:06:36] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:16:23 [I 2022-10-08 19:06:36] ...trap/engines/engine.py.229: process: 0.79478 | load: 0.00307 [I 2022-10-08 19:06:36] ...trap/engines/engine.py.230: loss: 0.22164 [I 2022-10-08 19:06:36] ...optimizers/trijoint.py.151: total_norm: 2.744629 [I 2022-10-08 19:06:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1060/2304 [I 2022-10-08 19:06:44] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:15 [I 2022-10-08 19:06:44] ...trap/engines/engine.py.229: process: 0.78895 | load: 0.00302 [I 2022-10-08 19:06:44] ...trap/engines/engine.py.230: loss: 0.29161 [I 2022-10-08 19:06:44] ...optimizers/trijoint.py.151: total_norm: 4.049320 [I 2022-10-08 19:06:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 1070/2304 [I 2022-10-08 19:06:52] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:15:57 [I 2022-10-08 19:06:52] ...trap/engines/engine.py.229: process: 0.77887 | load: 0.00308 [I 2022-10-08 19:06:52] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-08 19:06:52] ...optimizers/trijoint.py.151: total_norm: 2.479713 [I 2022-10-08 19:07:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1080/2304 [I 2022-10-08 19:07:00] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:53 [I 2022-10-08 19:07:00] ...trap/engines/engine.py.229: process: 0.78216 | load: 0.00308 [I 2022-10-08 19:07:00] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-08 19:07:00] ...optimizers/trijoint.py.151: total_norm: 2.848728 [I 2022-10-08 19:07:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 1090/2304 [I 2022-10-08 19:07:08] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:43 [I 2022-10-08 19:07:08] ...trap/engines/engine.py.229: process: 0.78037 | load: 0.00302 [I 2022-10-08 19:07:08] ...trap/engines/engine.py.230: loss: 0.23502 [I 2022-10-08 19:07:08] ...optimizers/trijoint.py.151: total_norm: 2.561092 [I 2022-10-08 19:07:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1100/2304 [I 2022-10-08 19:07:15] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:40 [I 2022-10-08 19:07:15] ...trap/engines/engine.py.229: process: 0.77066 | load: 0.00304 [I 2022-10-08 19:07:15] ...trap/engines/engine.py.230: loss: 0.30607 [I 2022-10-08 19:07:15] ...optimizers/trijoint.py.151: total_norm: 3.934785 [I 2022-10-08 19:07:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 1110/2304 [I 2022-10-08 19:07:23] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:15:31 [I 2022-10-08 19:07:23] ...trap/engines/engine.py.229: process: 0.78221 | load: 0.00301 [I 2022-10-08 19:07:23] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-08 19:07:23] ...optimizers/trijoint.py.151: total_norm: 2.513229 [I 2022-10-08 19:07:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 1120/2304 [I 2022-10-08 19:07:31] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:24 [I 2022-10-08 19:07:31] ...trap/engines/engine.py.229: process: 0.78672 | load: 0.00303 [I 2022-10-08 19:07:31] ...trap/engines/engine.py.230: loss: 0.16069 [I 2022-10-08 19:07:31] ...optimizers/trijoint.py.151: total_norm: 1.987217 [I 2022-10-08 19:07:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1130/2304 [I 2022-10-08 19:07:39] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:15:10 [I 2022-10-08 19:07:39] ...trap/engines/engine.py.229: process: 0.76194 | load: 0.00301 [I 2022-10-08 19:07:39] ...trap/engines/engine.py.230: loss: 0.18822 [I 2022-10-08 19:07:39] ...optimizers/trijoint.py.151: total_norm: 2.571982 [I 2022-10-08 19:07:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 1140/2304 [I 2022-10-08 19:07:47] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:05 [I 2022-10-08 19:07:47] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00302 [I 2022-10-08 19:07:47] ...trap/engines/engine.py.230: loss: 0.23801 [I 2022-10-08 19:07:47] ...optimizers/trijoint.py.151: total_norm: 3.732968 [I 2022-10-08 19:07:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1150/2304 [I 2022-10-08 19:07:54] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:14:58 [I 2022-10-08 19:07:54] ...trap/engines/engine.py.229: process: 0.77280 | load: 0.00301 [I 2022-10-08 19:07:54] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-08 19:07:54] ...optimizers/trijoint.py.151: total_norm: 2.213024 [I 2022-10-08 19:08:02] ...trap/engines/engine.py.224: train: epoch 16 | batch 1160/2304 [I 2022-10-08 19:08:02] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:14:48 [I 2022-10-08 19:08:02] ...trap/engines/engine.py.229: process: 0.78415 | load: 0.00299 [I 2022-10-08 19:08:02] ...trap/engines/engine.py.230: loss: 0.13886 [I 2022-10-08 19:08:02] ...optimizers/trijoint.py.151: total_norm: 1.812090 [I 2022-10-08 19:08:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 1170/2304 [I 2022-10-08 19:08:10] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:50 [I 2022-10-08 19:08:10] ...trap/engines/engine.py.229: process: 0.79507 | load: 0.00309 [I 2022-10-08 19:08:10] ...trap/engines/engine.py.230: loss: 0.21748 [I 2022-10-08 19:08:10] ...optimizers/trijoint.py.151: total_norm: 3.058188 [I 2022-10-08 19:08:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 1180/2304 [I 2022-10-08 19:08:18] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:14:34 [I 2022-10-08 19:08:18] ...trap/engines/engine.py.229: process: 0.76811 | load: 0.00304 [I 2022-10-08 19:08:18] ...trap/engines/engine.py.230: loss: 0.22292 [I 2022-10-08 19:08:18] ...optimizers/trijoint.py.151: total_norm: 2.561565 [I 2022-10-08 19:08:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 1190/2304 [I 2022-10-08 19:08:26] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:14:25 [I 2022-10-08 19:08:26] ...trap/engines/engine.py.229: process: 0.77859 | load: 0.00306 [I 2022-10-08 19:08:26] ...trap/engines/engine.py.230: loss: 0.17936 [I 2022-10-08 19:08:26] ...optimizers/trijoint.py.151: total_norm: 3.565923 [I 2022-10-08 19:08:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 1200/2304 [I 2022-10-08 19:08:33] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:18 [I 2022-10-08 19:08:33] ...trap/engines/engine.py.229: process: 0.77183 | load: 0.00306 [I 2022-10-08 19:08:33] ...trap/engines/engine.py.230: loss: 0.19840 [I 2022-10-08 19:08:33] ...optimizers/trijoint.py.151: total_norm: 3.437163 [I 2022-10-08 19:08:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 1210/2304 [I 2022-10-08 19:08:41] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:14:14 [I 2022-10-08 19:08:41] ...trap/engines/engine.py.229: process: 0.78006 | load: 0.00304 [I 2022-10-08 19:08:41] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-08 19:08:41] ...optimizers/trijoint.py.151: total_norm: 2.569366 [I 2022-10-08 19:08:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1220/2304 [I 2022-10-08 19:08:49] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:14:05 [I 2022-10-08 19:08:49] ...trap/engines/engine.py.229: process: 0.77746 | load: 0.00299 [I 2022-10-08 19:08:49] ...trap/engines/engine.py.230: loss: 0.12694 [I 2022-10-08 19:08:49] ...optimizers/trijoint.py.151: total_norm: 1.303134 [I 2022-10-08 19:08:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 1230/2304 [I 2022-10-08 19:08:57] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:13:56 [I 2022-10-08 19:08:57] ...trap/engines/engine.py.229: process: 0.77255 | load: 0.00297 [I 2022-10-08 19:08:57] ...trap/engines/engine.py.230: loss: 0.19760 [I 2022-10-08 19:08:57] ...optimizers/trijoint.py.151: total_norm: 2.972568 [I 2022-10-08 19:09:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1240/2304 [I 2022-10-08 19:09:05] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:13:43 [I 2022-10-08 19:09:05] ...trap/engines/engine.py.229: process: 0.77311 | load: 0.00302 [I 2022-10-08 19:09:05] ...trap/engines/engine.py.230: loss: 0.19305 [I 2022-10-08 19:09:05] ...optimizers/trijoint.py.151: total_norm: 3.423457 [I 2022-10-08 19:09:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 1250/2304 [I 2022-10-08 19:09:13] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:13:32 [I 2022-10-08 19:09:13] ...trap/engines/engine.py.229: process: 0.75918 | load: 0.00304 [I 2022-10-08 19:09:13] ...trap/engines/engine.py.230: loss: 0.14436 [I 2022-10-08 19:09:13] ...optimizers/trijoint.py.151: total_norm: 1.892975 [I 2022-10-08 19:09:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 1260/2304 [I 2022-10-08 19:09:20] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:30 [I 2022-10-08 19:09:20] ...trap/engines/engine.py.229: process: 0.78480 | load: 0.00332 [I 2022-10-08 19:09:20] ...trap/engines/engine.py.230: loss: 0.19246 [I 2022-10-08 19:09:20] ...optimizers/trijoint.py.151: total_norm: 3.727543 [I 2022-10-08 19:09:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 1270/2304 [I 2022-10-08 19:09:28] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:13:19 [I 2022-10-08 19:09:28] ...trap/engines/engine.py.229: process: 0.76343 | load: 0.00307 [I 2022-10-08 19:09:28] ...trap/engines/engine.py.230: loss: 0.25849 [I 2022-10-08 19:09:28] ...optimizers/trijoint.py.151: total_norm: 3.806137 [I 2022-10-08 19:09:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 1280/2304 [I 2022-10-08 19:09:36] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:13:15 [I 2022-10-08 19:09:36] ...trap/engines/engine.py.229: process: 0.77645 | load: 0.00299 [I 2022-10-08 19:09:36] ...trap/engines/engine.py.230: loss: 0.13452 [I 2022-10-08 19:09:36] ...optimizers/trijoint.py.151: total_norm: 2.435304 [I 2022-10-08 19:09:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1290/2304 [I 2022-10-08 19:09:44] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:13:11 [I 2022-10-08 19:09:44] ...trap/engines/engine.py.229: process: 0.78341 | load: 0.00304 [I 2022-10-08 19:09:44] ...trap/engines/engine.py.230: loss: 0.26255 [I 2022-10-08 19:09:44] ...optimizers/trijoint.py.151: total_norm: 2.763949 [I 2022-10-08 19:09:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 1300/2304 [I 2022-10-08 19:09:52] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:13:03 [I 2022-10-08 19:09:52] ...trap/engines/engine.py.229: process: 0.78638 | load: 0.00301 [I 2022-10-08 19:09:52] ...trap/engines/engine.py.230: loss: 0.20459 [I 2022-10-08 19:09:52] ...optimizers/trijoint.py.151: total_norm: 2.680315 [I 2022-10-08 19:09:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1310/2304 [I 2022-10-08 19:09:59] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:54 [I 2022-10-08 19:09:59] ...trap/engines/engine.py.229: process: 0.78104 | load: 0.00303 [I 2022-10-08 19:09:59] ...trap/engines/engine.py.230: loss: 0.19356 [I 2022-10-08 19:09:59] ...optimizers/trijoint.py.151: total_norm: 3.361387 [I 2022-10-08 19:10:07] ...trap/engines/engine.py.224: train: epoch 16 | batch 1320/2304 [I 2022-10-08 19:10:07] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:12:44 [I 2022-10-08 19:10:07] ...trap/engines/engine.py.229: process: 0.78423 | load: 0.00302 [I 2022-10-08 19:10:07] ...trap/engines/engine.py.230: loss: 0.14531 [I 2022-10-08 19:10:07] ...optimizers/trijoint.py.151: total_norm: 2.139329 [I 2022-10-08 19:10:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1330/2304 [I 2022-10-08 19:10:15] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:38 [I 2022-10-08 19:10:15] ...trap/engines/engine.py.229: process: 0.79330 | load: 0.00301 [I 2022-10-08 19:10:15] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-08 19:10:15] ...optimizers/trijoint.py.151: total_norm: 2.219029 [I 2022-10-08 19:10:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 1340/2304 [I 2022-10-08 19:10:23] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:12:29 [I 2022-10-08 19:10:23] ...trap/engines/engine.py.229: process: 0.76729 | load: 0.00300 [I 2022-10-08 19:10:23] ...trap/engines/engine.py.230: loss: 0.17569 [I 2022-10-08 19:10:23] ...optimizers/trijoint.py.151: total_norm: 3.379949 [I 2022-10-08 19:10:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 1350/2304 [I 2022-10-08 19:10:31] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:12:18 [I 2022-10-08 19:10:31] ...trap/engines/engine.py.229: process: 0.77084 | load: 0.00303 [I 2022-10-08 19:10:31] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-08 19:10:31] ...optimizers/trijoint.py.151: total_norm: 2.060332 [I 2022-10-08 19:10:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 1360/2304 [I 2022-10-08 19:10:38] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:07 [I 2022-10-08 19:10:38] ...trap/engines/engine.py.229: process: 0.76130 | load: 0.00304 [I 2022-10-08 19:10:38] ...trap/engines/engine.py.230: loss: 0.20102 [I 2022-10-08 19:10:38] ...optimizers/trijoint.py.151: total_norm: 2.685181 [I 2022-10-08 19:10:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 1370/2304 [I 2022-10-08 19:10:46] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:00 [I 2022-10-08 19:10:46] ...trap/engines/engine.py.229: process: 0.77642 | load: 0.00303 [I 2022-10-08 19:10:46] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-08 19:10:46] ...optimizers/trijoint.py.151: total_norm: 2.206945 [I 2022-10-08 19:10:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1380/2304 [I 2022-10-08 19:10:54] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:11:58 [I 2022-10-08 19:10:54] ...trap/engines/engine.py.229: process: 0.77381 | load: 0.00300 [I 2022-10-08 19:10:54] ...trap/engines/engine.py.230: loss: 0.15857 [I 2022-10-08 19:10:54] ...optimizers/trijoint.py.151: total_norm: 2.702633 [I 2022-10-08 19:11:02] ...trap/engines/engine.py.224: train: epoch 16 | batch 1390/2304 [I 2022-10-08 19:11:02] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:11:50 [I 2022-10-08 19:11:02] ...trap/engines/engine.py.229: process: 0.78164 | load: 0.00301 [I 2022-10-08 19:11:02] ...trap/engines/engine.py.230: loss: 0.19521 [I 2022-10-08 19:11:02] ...optimizers/trijoint.py.151: total_norm: 2.859611 [I 2022-10-08 19:11:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1400/2304 [I 2022-10-08 19:11:09] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:11:47 [I 2022-10-08 19:11:09] ...trap/engines/engine.py.229: process: 0.79945 | load: 0.00304 [I 2022-10-08 19:11:09] ...trap/engines/engine.py.230: loss: 0.12525 [I 2022-10-08 19:11:09] ...optimizers/trijoint.py.151: total_norm: 1.888224 [I 2022-10-08 19:11:17] ...trap/engines/engine.py.224: train: epoch 16 | batch 1410/2304 [I 2022-10-08 19:11:17] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:11:37 [I 2022-10-08 19:11:17] ...trap/engines/engine.py.229: process: 0.77973 | load: 0.00301 [I 2022-10-08 19:11:17] ...trap/engines/engine.py.230: loss: 0.19399 [I 2022-10-08 19:11:17] ...optimizers/trijoint.py.151: total_norm: 3.044400 [I 2022-10-08 19:11:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 1420/2304 [I 2022-10-08 19:11:25] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:23 [I 2022-10-08 19:11:25] ...trap/engines/engine.py.229: process: 0.78519 | load: 0.00306 [I 2022-10-08 19:11:25] ...trap/engines/engine.py.230: loss: 0.24345 [I 2022-10-08 19:11:25] ...optimizers/trijoint.py.151: total_norm: 3.072234 [I 2022-10-08 19:11:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 1430/2304 [I 2022-10-08 19:11:33] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:11:18 [I 2022-10-08 19:11:33] ...trap/engines/engine.py.229: process: 0.78178 | load: 0.00302 [I 2022-10-08 19:11:33] ...trap/engines/engine.py.230: loss: 0.14959 [I 2022-10-08 19:11:33] ...optimizers/trijoint.py.151: total_norm: 2.598251 [I 2022-10-08 19:11:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 1440/2304 [I 2022-10-08 19:11:40] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:11:07 [I 2022-10-08 19:11:40] ...trap/engines/engine.py.229: process: 0.76805 | load: 0.00310 [I 2022-10-08 19:11:40] ...trap/engines/engine.py.230: loss: 0.24248 [I 2022-10-08 19:11:40] ...optimizers/trijoint.py.151: total_norm: 3.167025 [I 2022-10-08 19:11:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 1450/2304 [I 2022-10-08 19:11:48] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:11:05 [I 2022-10-08 19:11:48] ...trap/engines/engine.py.229: process: 0.76219 | load: 0.00307 [I 2022-10-08 19:11:48] ...trap/engines/engine.py.230: loss: 0.15300 [I 2022-10-08 19:11:48] ...optimizers/trijoint.py.151: total_norm: 3.520855 [I 2022-10-08 19:11:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 1460/2304 [I 2022-10-08 19:11:56] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:11:00 [I 2022-10-08 19:11:56] ...trap/engines/engine.py.229: process: 0.77702 | load: 0.00311 [I 2022-10-08 19:11:56] ...trap/engines/engine.py.230: loss: 0.17800 [I 2022-10-08 19:11:56] ...optimizers/trijoint.py.151: total_norm: 2.505336 [I 2022-10-08 19:12:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1470/2304 [I 2022-10-08 19:12:04] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:10:51 [I 2022-10-08 19:12:04] ...trap/engines/engine.py.229: process: 0.78069 | load: 0.00300 [I 2022-10-08 19:12:04] ...trap/engines/engine.py.230: loss: 0.18891 [I 2022-10-08 19:12:04] ...optimizers/trijoint.py.151: total_norm: 2.210243 [I 2022-10-08 19:12:12] ...trap/engines/engine.py.224: train: epoch 16 | batch 1480/2304 [I 2022-10-08 19:12:12] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:10:42 [I 2022-10-08 19:12:12] ...trap/engines/engine.py.229: process: 0.77903 | load: 0.00305 [I 2022-10-08 19:12:12] ...trap/engines/engine.py.230: loss: 0.22312 [I 2022-10-08 19:12:12] ...optimizers/trijoint.py.151: total_norm: 2.867052 [I 2022-10-08 19:12:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 1490/2304 [I 2022-10-08 19:12:20] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:10:31 [I 2022-10-08 19:12:20] ...trap/engines/engine.py.229: process: 0.78104 | load: 0.00303 [I 2022-10-08 19:12:20] ...trap/engines/engine.py.230: loss: 0.19402 [I 2022-10-08 19:12:20] ...optimizers/trijoint.py.151: total_norm: 2.216083 [I 2022-10-08 19:12:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 1500/2304 [I 2022-10-08 19:12:27] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:10:23 [I 2022-10-08 19:12:27] ...trap/engines/engine.py.229: process: 0.77580 | load: 0.00296 [I 2022-10-08 19:12:27] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-08 19:12:27] ...optimizers/trijoint.py.151: total_norm: 2.178395 [I 2022-10-08 19:12:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 1510/2304 [I 2022-10-08 19:12:35] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:10:17 [I 2022-10-08 19:12:35] ...trap/engines/engine.py.229: process: 0.78404 | load: 0.00302 [I 2022-10-08 19:12:35] ...trap/engines/engine.py.230: loss: 0.26295 [I 2022-10-08 19:12:35] ...optimizers/trijoint.py.151: total_norm: 2.911087 [I 2022-10-08 19:12:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 1520/2304 [I 2022-10-08 19:12:43] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:10:11 [I 2022-10-08 19:12:43] ...trap/engines/engine.py.229: process: 0.77940 | load: 0.00303 [I 2022-10-08 19:12:43] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-08 19:12:43] ...optimizers/trijoint.py.151: total_norm: 3.484046 [I 2022-10-08 19:12:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 1530/2304 [I 2022-10-08 19:12:51] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:09:59 [I 2022-10-08 19:12:51] ...trap/engines/engine.py.229: process: 0.75230 | load: 0.00303 [I 2022-10-08 19:12:51] ...trap/engines/engine.py.230: loss: 0.23586 [I 2022-10-08 19:12:51] ...optimizers/trijoint.py.151: total_norm: 2.788679 [I 2022-10-08 19:13:17] ...trap/engines/engine.py.224: train: epoch 16 | batch 1540/2304 [I 2022-10-08 19:13:17] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:19:34 [I 2022-10-08 19:13:17] ...trap/engines/engine.py.229: process: 0.77306 | load: 0.00305 [I 2022-10-08 19:13:17] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-08 19:13:17] ...optimizers/trijoint.py.151: total_norm: 2.651356 [I 2022-10-08 19:13:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1550/2304 [I 2022-10-08 19:13:24] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:10:47 [I 2022-10-08 19:13:24] ...trap/engines/engine.py.229: process: 0.77852 | load: 0.00303 [I 2022-10-08 19:13:24] ...trap/engines/engine.py.230: loss: 0.20352 [I 2022-10-08 19:13:24] ...optimizers/trijoint.py.151: total_norm: 3.111779 [I 2022-10-08 19:13:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 1560/2304 [I 2022-10-08 19:13:32] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:09:46 [I 2022-10-08 19:13:32] ...trap/engines/engine.py.229: process: 0.78871 | load: 0.00311 [I 2022-10-08 19:13:32] ...trap/engines/engine.py.230: loss: 0.18574 [I 2022-10-08 19:13:32] ...optimizers/trijoint.py.151: total_norm: 2.231960 [I 2022-10-08 19:13:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 1570/2304 [I 2022-10-08 19:13:40] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:09:31 [I 2022-10-08 19:13:40] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00303 [I 2022-10-08 19:13:40] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-08 19:13:40] ...optimizers/trijoint.py.151: total_norm: 2.675385 [I 2022-10-08 19:13:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 1580/2304 [I 2022-10-08 19:13:48] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:09:26 [I 2022-10-08 19:13:48] ...trap/engines/engine.py.229: process: 0.78555 | load: 0.00307 [I 2022-10-08 19:13:48] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-08 19:13:48] ...optimizers/trijoint.py.151: total_norm: 2.281270 [I 2022-10-08 19:13:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 1590/2304 [I 2022-10-08 19:13:56] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:09:14 [I 2022-10-08 19:13:56] ...trap/engines/engine.py.229: process: 0.77700 | load: 0.00304 [I 2022-10-08 19:13:56] ...trap/engines/engine.py.230: loss: 0.22899 [I 2022-10-08 19:13:56] ...optimizers/trijoint.py.151: total_norm: 2.104680 [I 2022-10-08 19:14:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 1600/2304 [I 2022-10-08 19:14:03] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:09:08 [I 2022-10-08 19:14:03] ...trap/engines/engine.py.229: process: 0.77885 | load: 0.00304 [I 2022-10-08 19:14:03] ...trap/engines/engine.py.230: loss: 0.14037 [I 2022-10-08 19:14:03] ...optimizers/trijoint.py.151: total_norm: 2.595087 [I 2022-10-08 19:14:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 1610/2304 [I 2022-10-08 19:14:11] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:08:59 [I 2022-10-08 19:14:11] ...trap/engines/engine.py.229: process: 0.77969 | load: 0.00306 [I 2022-10-08 19:14:11] ...trap/engines/engine.py.230: loss: 0.18207 [I 2022-10-08 19:14:11] ...optimizers/trijoint.py.151: total_norm: 2.259744 [I 2022-10-08 19:14:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1620/2304 [I 2022-10-08 19:14:19] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:08:53 [I 2022-10-08 19:14:19] ...trap/engines/engine.py.229: process: 0.78939 | load: 0.00301 [I 2022-10-08 19:14:19] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-08 19:14:19] ...optimizers/trijoint.py.151: total_norm: 2.163697 [I 2022-10-08 19:14:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 1630/2304 [I 2022-10-08 19:14:27] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:08:45 [I 2022-10-08 19:14:27] ...trap/engines/engine.py.229: process: 0.79267 | load: 0.00304 [I 2022-10-08 19:14:27] ...trap/engines/engine.py.230: loss: 0.22638 [I 2022-10-08 19:14:27] ...optimizers/trijoint.py.151: total_norm: 3.007689 [I 2022-10-08 19:14:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 1640/2304 [I 2022-10-08 19:14:35] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:08:36 [I 2022-10-08 19:14:35] ...trap/engines/engine.py.229: process: 0.77706 | load: 0.00308 [I 2022-10-08 19:14:35] ...trap/engines/engine.py.230: loss: 0.20730 [I 2022-10-08 19:14:35] ...optimizers/trijoint.py.151: total_norm: 2.456203 [I 2022-10-08 19:14:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 1650/2304 [I 2022-10-08 19:14:42] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:08:30 [I 2022-10-08 19:14:42] ...trap/engines/engine.py.229: process: 0.78519 | load: 0.00303 [I 2022-10-08 19:14:42] ...trap/engines/engine.py.230: loss: 0.19796 [I 2022-10-08 19:14:42] ...optimizers/trijoint.py.151: total_norm: 2.400079 [I 2022-10-08 19:14:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 1660/2304 [I 2022-10-08 19:14:50] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:08:24 [I 2022-10-08 19:14:50] ...trap/engines/engine.py.229: process: 0.77581 | load: 0.00301 [I 2022-10-08 19:14:50] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-08 19:14:50] ...optimizers/trijoint.py.151: total_norm: 1.348468 [I 2022-10-08 19:14:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 1670/2304 [I 2022-10-08 19:14:58] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:08:14 [I 2022-10-08 19:14:58] ...trap/engines/engine.py.229: process: 0.77881 | load: 0.00306 [I 2022-10-08 19:14:58] ...trap/engines/engine.py.230: loss: 0.17004 [I 2022-10-08 19:14:58] ...optimizers/trijoint.py.151: total_norm: 2.641185 [I 2022-10-08 19:15:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 1680/2304 [I 2022-10-08 19:15:06] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:08:04 [I 2022-10-08 19:15:06] ...trap/engines/engine.py.229: process: 0.78073 | load: 0.00308 [I 2022-10-08 19:15:06] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-08 19:15:06] ...optimizers/trijoint.py.151: total_norm: 3.097584 [I 2022-10-08 19:15:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1690/2304 [I 2022-10-08 19:15:14] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:07:57 [I 2022-10-08 19:15:14] ...trap/engines/engine.py.229: process: 0.78846 | load: 0.00307 [I 2022-10-08 19:15:14] ...trap/engines/engine.py.230: loss: 0.18225 [I 2022-10-08 19:15:14] ...optimizers/trijoint.py.151: total_norm: 2.710217 [I 2022-10-08 19:15:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 1700/2304 [I 2022-10-08 19:15:21] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:07:53 [I 2022-10-08 19:15:21] ...trap/engines/engine.py.229: process: 0.77886 | load: 0.00304 [I 2022-10-08 19:15:21] ...trap/engines/engine.py.230: loss: 0.20989 [I 2022-10-08 19:15:21] ...optimizers/trijoint.py.151: total_norm: 2.496194 [I 2022-10-08 19:15:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1710/2304 [I 2022-10-08 19:15:29] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:07:42 [I 2022-10-08 19:15:29] ...trap/engines/engine.py.229: process: 0.77543 | load: 0.00302 [I 2022-10-08 19:15:29] ...trap/engines/engine.py.230: loss: 0.17963 [I 2022-10-08 19:15:29] ...optimizers/trijoint.py.151: total_norm: 2.884654 [I 2022-10-08 19:15:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 1720/2304 [I 2022-10-08 19:15:37] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:07:34 [I 2022-10-08 19:15:37] ...trap/engines/engine.py.229: process: 0.78435 | load: 0.00306 [I 2022-10-08 19:15:37] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-08 19:15:37] ...optimizers/trijoint.py.151: total_norm: 3.277926 [I 2022-10-08 19:15:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 1730/2304 [I 2022-10-08 19:15:45] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:07:28 [I 2022-10-08 19:15:45] ...trap/engines/engine.py.229: process: 0.78742 | load: 0.00300 [I 2022-10-08 19:15:45] ...trap/engines/engine.py.230: loss: 0.14985 [I 2022-10-08 19:15:45] ...optimizers/trijoint.py.151: total_norm: 2.155025 [I 2022-10-08 19:15:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 1740/2304 [I 2022-10-08 19:15:53] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:07:20 [I 2022-10-08 19:15:53] ...trap/engines/engine.py.229: process: 0.78861 | load: 0.00303 [I 2022-10-08 19:15:53] ...trap/engines/engine.py.230: loss: 0.13080 [I 2022-10-08 19:15:53] ...optimizers/trijoint.py.151: total_norm: 1.873791 [I 2022-10-08 19:16:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1750/2304 [I 2022-10-08 19:16:00] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:07:10 [I 2022-10-08 19:16:00] ...trap/engines/engine.py.229: process: 0.78546 | load: 0.00303 [I 2022-10-08 19:16:00] ...trap/engines/engine.py.230: loss: 0.20448 [I 2022-10-08 19:16:00] ...optimizers/trijoint.py.151: total_norm: 3.308438 [I 2022-10-08 19:16:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 1760/2304 [I 2022-10-08 19:16:08] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:07:01 [I 2022-10-08 19:16:08] ...trap/engines/engine.py.229: process: 0.77626 | load: 0.00305 [I 2022-10-08 19:16:08] ...trap/engines/engine.py.230: loss: 0.16786 [I 2022-10-08 19:16:08] ...optimizers/trijoint.py.151: total_norm: 2.841541 [I 2022-10-08 19:16:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 1770/2304 [I 2022-10-08 19:16:16] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:06:56 [I 2022-10-08 19:16:16] ...trap/engines/engine.py.229: process: 0.78720 | load: 0.00304 [I 2022-10-08 19:16:16] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-08 19:16:16] ...optimizers/trijoint.py.151: total_norm: 2.944475 [I 2022-10-08 19:16:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1780/2304 [I 2022-10-08 19:16:24] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:06:49 [I 2022-10-08 19:16:24] ...trap/engines/engine.py.229: process: 0.78436 | load: 0.00313 [I 2022-10-08 19:16:24] ...trap/engines/engine.py.230: loss: 0.22311 [I 2022-10-08 19:16:24] ...optimizers/trijoint.py.151: total_norm: 3.252499 [I 2022-10-08 19:16:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 1790/2304 [I 2022-10-08 19:16:32] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:06:41 [I 2022-10-08 19:16:32] ...trap/engines/engine.py.229: process: 0.79123 | load: 0.00310 [I 2022-10-08 19:16:32] ...trap/engines/engine.py.230: loss: 0.18108 [I 2022-10-08 19:16:32] ...optimizers/trijoint.py.151: total_norm: 2.667834 [I 2022-10-08 19:16:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1800/2304 [I 2022-10-08 19:16:39] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:06:31 [I 2022-10-08 19:16:39] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00311 [I 2022-10-08 19:16:39] ...trap/engines/engine.py.230: loss: 0.23152 [I 2022-10-08 19:16:39] ...optimizers/trijoint.py.151: total_norm: 2.812009 [I 2022-10-08 19:16:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 1810/2304 [I 2022-10-08 19:16:47] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:06:22 [I 2022-10-08 19:16:47] ...trap/engines/engine.py.229: process: 0.76935 | load: 0.00313 [I 2022-10-08 19:16:47] ...trap/engines/engine.py.230: loss: 0.18184 [I 2022-10-08 19:16:47] ...optimizers/trijoint.py.151: total_norm: 2.902850 [I 2022-10-08 19:16:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 1820/2304 [I 2022-10-08 19:16:55] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:06:17 [I 2022-10-08 19:16:55] ...trap/engines/engine.py.229: process: 0.78459 | load: 0.00304 [I 2022-10-08 19:16:55] ...trap/engines/engine.py.230: loss: 0.26150 [I 2022-10-08 19:16:55] ...optimizers/trijoint.py.151: total_norm: 3.104956 [I 2022-10-08 19:17:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 1830/2304 [I 2022-10-08 19:17:03] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:06:09 [I 2022-10-08 19:17:03] ...trap/engines/engine.py.229: process: 0.77819 | load: 0.00309 [I 2022-10-08 19:17:03] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-08 19:17:03] ...optimizers/trijoint.py.151: total_norm: 1.779469 [I 2022-10-08 19:17:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 1840/2304 [I 2022-10-08 19:17:11] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:06:05 [I 2022-10-08 19:17:11] ...trap/engines/engine.py.229: process: 0.79354 | load: 0.00302 [I 2022-10-08 19:17:11] ...trap/engines/engine.py.230: loss: 0.20015 [I 2022-10-08 19:17:11] ...optimizers/trijoint.py.151: total_norm: 2.784597 [I 2022-10-08 19:17:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1850/2304 [I 2022-10-08 19:17:19] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:05:54 [I 2022-10-08 19:17:19] ...trap/engines/engine.py.229: process: 0.78365 | load: 0.00303 [I 2022-10-08 19:17:19] ...trap/engines/engine.py.230: loss: 0.17640 [I 2022-10-08 19:17:19] ...optimizers/trijoint.py.151: total_norm: 3.063112 [I 2022-10-08 19:17:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 1860/2304 [I 2022-10-08 19:17:26] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:05:46 [I 2022-10-08 19:17:26] ...trap/engines/engine.py.229: process: 0.78086 | load: 0.00302 [I 2022-10-08 19:17:26] ...trap/engines/engine.py.230: loss: 0.19982 [I 2022-10-08 19:17:26] ...optimizers/trijoint.py.151: total_norm: 4.062569 [I 2022-10-08 19:17:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1870/2304 [I 2022-10-08 19:17:34] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:05:35 [I 2022-10-08 19:17:34] ...trap/engines/engine.py.229: process: 0.76789 | load: 0.00309 [I 2022-10-08 19:17:34] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-08 19:17:34] ...optimizers/trijoint.py.151: total_norm: 2.703655 [I 2022-10-08 19:17:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 1880/2304 [I 2022-10-08 19:17:42] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:05:29 [I 2022-10-08 19:17:42] ...trap/engines/engine.py.229: process: 0.78418 | load: 0.00303 [I 2022-10-08 19:17:42] ...trap/engines/engine.py.230: loss: 0.16296 [I 2022-10-08 19:17:42] ...optimizers/trijoint.py.151: total_norm: 1.726874 [I 2022-10-08 19:17:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 1890/2304 [I 2022-10-08 19:17:50] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:05:21 [I 2022-10-08 19:17:50] ...trap/engines/engine.py.229: process: 0.77172 | load: 0.00303 [I 2022-10-08 19:17:50] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-08 19:17:50] ...optimizers/trijoint.py.151: total_norm: 2.562696 [I 2022-10-08 19:17:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 1900/2304 [I 2022-10-08 19:17:57] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:05:12 [I 2022-10-08 19:17:57] ...trap/engines/engine.py.229: process: 0.77126 | load: 0.00305 [I 2022-10-08 19:17:57] ...trap/engines/engine.py.230: loss: 0.21844 [I 2022-10-08 19:17:57] ...optimizers/trijoint.py.151: total_norm: 3.400134 [I 2022-10-08 19:18:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1910/2304 [I 2022-10-08 19:18:05] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:05:05 [I 2022-10-08 19:18:05] ...trap/engines/engine.py.229: process: 0.77268 | load: 0.00303 [I 2022-10-08 19:18:05] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-08 19:18:05] ...optimizers/trijoint.py.151: total_norm: 2.312096 [I 2022-10-08 19:18:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 1920/2304 [I 2022-10-08 19:18:13] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:04:58 [I 2022-10-08 19:18:13] ...trap/engines/engine.py.229: process: 0.77694 | load: 0.00305 [I 2022-10-08 19:18:13] ...trap/engines/engine.py.230: loss: 0.17603 [I 2022-10-08 19:18:13] ...optimizers/trijoint.py.151: total_norm: 2.161646 [I 2022-10-08 19:18:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 1930/2304 [I 2022-10-08 19:18:21] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:04:50 [I 2022-10-08 19:18:21] ...trap/engines/engine.py.229: process: 0.76440 | load: 0.00307 [I 2022-10-08 19:18:21] ...trap/engines/engine.py.230: loss: 0.33263 [I 2022-10-08 19:18:21] ...optimizers/trijoint.py.151: total_norm: 4.195707 [I 2022-10-08 19:18:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1940/2304 [I 2022-10-08 19:18:29] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:04:43 [I 2022-10-08 19:18:29] ...trap/engines/engine.py.229: process: 0.78613 | load: 0.00302 [I 2022-10-08 19:18:29] ...trap/engines/engine.py.230: loss: 0.16470 [I 2022-10-08 19:18:29] ...optimizers/trijoint.py.151: total_norm: 3.402651 [I 2022-10-08 19:18:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 1950/2304 [I 2022-10-08 19:18:36] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:04:36 [I 2022-10-08 19:18:36] ...trap/engines/engine.py.229: process: 0.78484 | load: 0.00304 [I 2022-10-08 19:18:36] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-08 19:18:36] ...optimizers/trijoint.py.151: total_norm: 2.183789 [I 2022-10-08 19:18:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1960/2304 [I 2022-10-08 19:18:44] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:04:28 [I 2022-10-08 19:18:44] ...trap/engines/engine.py.229: process: 0.79073 | load: 0.00303 [I 2022-10-08 19:18:44] ...trap/engines/engine.py.230: loss: 0.24083 [I 2022-10-08 19:18:44] ...optimizers/trijoint.py.151: total_norm: 3.463060 [I 2022-10-08 19:18:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 1970/2304 [I 2022-10-08 19:18:52] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:04:18 [I 2022-10-08 19:18:52] ...trap/engines/engine.py.229: process: 0.76258 | load: 0.00296 [I 2022-10-08 19:18:52] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-08 19:18:52] ...optimizers/trijoint.py.151: total_norm: 2.043336 [I 2022-10-08 19:19:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1980/2304 [I 2022-10-08 19:19:00] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:04:10 [I 2022-10-08 19:19:00] ...trap/engines/engine.py.229: process: 0.75588 | load: 0.00305 [I 2022-10-08 19:19:00] ...trap/engines/engine.py.230: loss: 0.25277 [I 2022-10-08 19:19:00] ...optimizers/trijoint.py.151: total_norm: 2.696268 [I 2022-10-08 19:19:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 1990/2304 [I 2022-10-08 19:19:08] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:04:05 [I 2022-10-08 19:19:08] ...trap/engines/engine.py.229: process: 0.79066 | load: 0.00307 [I 2022-10-08 19:19:08] ...trap/engines/engine.py.230: loss: 0.16686 [I 2022-10-08 19:19:08] ...optimizers/trijoint.py.151: total_norm: 2.666638 [I 2022-10-08 19:19:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 2000/2304 [I 2022-10-08 19:19:15] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:03:55 [I 2022-10-08 19:19:15] ...trap/engines/engine.py.229: process: 0.76969 | load: 0.00306 [I 2022-10-08 19:19:15] ...trap/engines/engine.py.230: loss: 0.22668 [I 2022-10-08 19:19:15] ...optimizers/trijoint.py.151: total_norm: 3.542043 [I 2022-10-08 19:19:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 2010/2304 [I 2022-10-08 19:19:23] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:03:47 [I 2022-10-08 19:19:23] ...trap/engines/engine.py.229: process: 0.77444 | load: 0.00294 [I 2022-10-08 19:19:23] ...trap/engines/engine.py.230: loss: 0.18035 [I 2022-10-08 19:19:23] ...optimizers/trijoint.py.151: total_norm: 3.085539 [I 2022-10-08 19:19:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 2020/2304 [I 2022-10-08 19:19:31] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:03:40 [I 2022-10-08 19:19:31] ...trap/engines/engine.py.229: process: 0.78168 | load: 0.00308 [I 2022-10-08 19:19:31] ...trap/engines/engine.py.230: loss: 0.18646 [I 2022-10-08 19:19:31] ...optimizers/trijoint.py.151: total_norm: 2.176947 [I 2022-10-08 19:19:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 2030/2304 [I 2022-10-08 19:19:39] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:03:33 [I 2022-10-08 19:19:39] ...trap/engines/engine.py.229: process: 0.77720 | load: 0.00304 [I 2022-10-08 19:19:39] ...trap/engines/engine.py.230: loss: 0.21116 [I 2022-10-08 19:19:39] ...optimizers/trijoint.py.151: total_norm: 2.880122 [I 2022-10-08 19:19:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 2040/2304 [I 2022-10-08 19:19:47] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:03:26 [I 2022-10-08 19:19:47] ...trap/engines/engine.py.229: process: 0.79697 | load: 0.00299 [I 2022-10-08 19:19:47] ...trap/engines/engine.py.230: loss: 0.27654 [I 2022-10-08 19:19:47] ...optimizers/trijoint.py.151: total_norm: 2.969434 [I 2022-10-08 19:19:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 2050/2304 [I 2022-10-08 19:19:54] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:03:18 [I 2022-10-08 19:19:54] ...trap/engines/engine.py.229: process: 0.78660 | load: 0.00308 [I 2022-10-08 19:19:54] ...trap/engines/engine.py.230: loss: 0.19991 [I 2022-10-08 19:19:54] ...optimizers/trijoint.py.151: total_norm: 2.454669 [I 2022-10-08 19:20:02] ...trap/engines/engine.py.224: train: epoch 16 | batch 2060/2304 [I 2022-10-08 19:20:02] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:03:08 [I 2022-10-08 19:20:02] ...trap/engines/engine.py.229: process: 0.75874 | load: 0.00306 [I 2022-10-08 19:20:02] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-08 19:20:02] ...optimizers/trijoint.py.151: total_norm: 2.298664 [I 2022-10-08 19:20:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 2070/2304 [I 2022-10-08 19:20:10] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:03:01 [I 2022-10-08 19:20:10] ...trap/engines/engine.py.229: process: 0.78161 | load: 0.00302 [I 2022-10-08 19:20:10] ...trap/engines/engine.py.230: loss: 0.18236 [I 2022-10-08 19:20:10] ...optimizers/trijoint.py.151: total_norm: 2.531672 [I 2022-10-08 19:20:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 2080/2304 [I 2022-10-08 19:20:18] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:02:53 [I 2022-10-08 19:20:18] ...trap/engines/engine.py.229: process: 0.79972 | load: 0.00305 [I 2022-10-08 19:20:18] ...trap/engines/engine.py.230: loss: 0.25221 [I 2022-10-08 19:20:18] ...optimizers/trijoint.py.151: total_norm: 2.436297 [I 2022-10-08 19:20:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 2090/2304 [I 2022-10-08 19:20:25] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:02:46 [I 2022-10-08 19:20:25] ...trap/engines/engine.py.229: process: 0.78366 | load: 0.00300 [I 2022-10-08 19:20:25] ...trap/engines/engine.py.230: loss: 0.27281 [I 2022-10-08 19:20:25] ...optimizers/trijoint.py.151: total_norm: 2.879333 [I 2022-10-08 19:20:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 2100/2304 [I 2022-10-08 19:20:33] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:02:38 [I 2022-10-08 19:20:33] ...trap/engines/engine.py.229: process: 0.77792 | load: 0.00304 [I 2022-10-08 19:20:33] ...trap/engines/engine.py.230: loss: 0.13768 [I 2022-10-08 19:20:33] ...optimizers/trijoint.py.151: total_norm: 1.925953 [I 2022-10-08 19:20:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 2110/2304 [I 2022-10-08 19:20:41] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:02:30 [I 2022-10-08 19:20:41] ...trap/engines/engine.py.229: process: 0.76729 | load: 0.00304 [I 2022-10-08 19:20:41] ...trap/engines/engine.py.230: loss: 0.18118 [I 2022-10-08 19:20:41] ...optimizers/trijoint.py.151: total_norm: 2.365604 [I 2022-10-08 19:20:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 2120/2304 [I 2022-10-08 19:20:49] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:23 [I 2022-10-08 19:20:49] ...trap/engines/engine.py.229: process: 0.77944 | load: 0.00305 [I 2022-10-08 19:20:49] ...trap/engines/engine.py.230: loss: 0.18082 [I 2022-10-08 19:20:49] ...optimizers/trijoint.py.151: total_norm: 2.222722 [I 2022-10-08 19:20:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 2130/2304 [I 2022-10-08 19:20:57] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:02:15 [I 2022-10-08 19:20:57] ...trap/engines/engine.py.229: process: 0.76993 | load: 0.00297 [I 2022-10-08 19:20:57] ...trap/engines/engine.py.230: loss: 0.22077 [I 2022-10-08 19:20:57] ...optimizers/trijoint.py.151: total_norm: 2.588029 [I 2022-10-08 19:21:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 2140/2304 [I 2022-10-08 19:21:04] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:02:07 [I 2022-10-08 19:21:04] ...trap/engines/engine.py.229: process: 0.78536 | load: 0.00306 [I 2022-10-08 19:21:04] ...trap/engines/engine.py.230: loss: 0.20244 [I 2022-10-08 19:21:04] ...optimizers/trijoint.py.151: total_norm: 2.413244 [I 2022-10-08 19:21:12] ...trap/engines/engine.py.224: train: epoch 16 | batch 2150/2304 [I 2022-10-08 19:21:12] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:01:59 [I 2022-10-08 19:21:12] ...trap/engines/engine.py.229: process: 0.76504 | load: 0.00302 [I 2022-10-08 19:21:12] ...trap/engines/engine.py.230: loss: 0.19355 [I 2022-10-08 19:21:12] ...optimizers/trijoint.py.151: total_norm: 3.826413 [I 2022-10-08 19:21:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 2160/2304 [I 2022-10-08 19:21:20] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:01:52 [I 2022-10-08 19:21:20] ...trap/engines/engine.py.229: process: 0.77813 | load: 0.00307 [I 2022-10-08 19:21:20] ...trap/engines/engine.py.230: loss: 0.25814 [I 2022-10-08 19:21:20] ...optimizers/trijoint.py.151: total_norm: 3.327211 [I 2022-10-08 19:21:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 2170/2304 [I 2022-10-08 19:21:28] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:01:44 [I 2022-10-08 19:21:28] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00303 [I 2022-10-08 19:21:28] ...trap/engines/engine.py.230: loss: 0.17662 [I 2022-10-08 19:21:28] ...optimizers/trijoint.py.151: total_norm: 2.520362 [I 2022-10-08 19:21:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 2180/2304 [I 2022-10-08 19:21:35] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:01:36 [I 2022-10-08 19:21:35] ...trap/engines/engine.py.229: process: 0.77663 | load: 0.00300 [I 2022-10-08 19:21:35] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-08 19:21:35] ...optimizers/trijoint.py.151: total_norm: 2.311815 [I 2022-10-08 19:21:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 2190/2304 [I 2022-10-08 19:21:43] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:01:28 [I 2022-10-08 19:21:43] ...trap/engines/engine.py.229: process: 0.77099 | load: 0.00309 [I 2022-10-08 19:21:43] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-08 19:21:43] ...optimizers/trijoint.py.151: total_norm: 3.216843 [I 2022-10-08 19:21:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 2200/2304 [I 2022-10-08 19:21:51] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:01:20 [I 2022-10-08 19:21:51] ...trap/engines/engine.py.229: process: 0.77883 | load: 0.00305 [I 2022-10-08 19:21:51] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-08 19:21:51] ...optimizers/trijoint.py.151: total_norm: 2.314675 [I 2022-10-08 19:21:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 2210/2304 [I 2022-10-08 19:21:59] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:01:12 [I 2022-10-08 19:21:59] ...trap/engines/engine.py.229: process: 0.78757 | load: 0.00295 [I 2022-10-08 19:21:59] ...trap/engines/engine.py.230: loss: 0.23309 [I 2022-10-08 19:21:59] ...optimizers/trijoint.py.151: total_norm: 3.067932 [I 2022-10-08 19:22:07] ...trap/engines/engine.py.224: train: epoch 16 | batch 2220/2304 [I 2022-10-08 19:22:07] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:01:04 [I 2022-10-08 19:22:07] ...trap/engines/engine.py.229: process: 0.76434 | load: 0.00307 [I 2022-10-08 19:22:07] ...trap/engines/engine.py.230: loss: 0.15209 [I 2022-10-08 19:22:07] ...optimizers/trijoint.py.151: total_norm: 1.916146 [I 2022-10-08 19:22:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 2230/2304 [I 2022-10-08 19:22:14] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:00:57 [I 2022-10-08 19:22:14] ...trap/engines/engine.py.229: process: 0.78940 | load: 0.00304 [I 2022-10-08 19:22:14] ...trap/engines/engine.py.230: loss: 0.15824 [I 2022-10-08 19:22:14] ...optimizers/trijoint.py.151: total_norm: 2.008481 [I 2022-10-08 19:22:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 2240/2304 [I 2022-10-08 19:22:22] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:00:49 [I 2022-10-08 19:22:22] ...trap/engines/engine.py.229: process: 0.76336 | load: 0.00308 [I 2022-10-08 19:22:22] ...trap/engines/engine.py.230: loss: 0.21455 [I 2022-10-08 19:22:22] ...optimizers/trijoint.py.151: total_norm: 2.844586 [I 2022-10-08 19:22:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 2250/2304 [I 2022-10-08 19:22:30] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:00:42 [I 2022-10-08 19:22:30] ...trap/engines/engine.py.229: process: 0.78472 | load: 0.00307 [I 2022-10-08 19:22:30] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-08 19:22:30] ...optimizers/trijoint.py.151: total_norm: 1.840615 [I 2022-10-08 19:22:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 2260/2304 [I 2022-10-08 19:22:38] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:00:34 [I 2022-10-08 19:22:38] ...trap/engines/engine.py.229: process: 0.78253 | load: 0.00300 [I 2022-10-08 19:22:38] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-08 19:22:38] ...optimizers/trijoint.py.151: total_norm: 2.144038 [I 2022-10-08 19:22:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 2270/2304 [I 2022-10-08 19:22:46] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:26 [I 2022-10-08 19:22:46] ...trap/engines/engine.py.229: process: 0.77756 | load: 0.00315 [I 2022-10-08 19:22:46] ...trap/engines/engine.py.230: loss: 0.22396 [I 2022-10-08 19:22:46] ...optimizers/trijoint.py.151: total_norm: 3.341319 [I 2022-10-08 19:22:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 2280/2304 [I 2022-10-08 19:22:53] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:00:18 [I 2022-10-08 19:22:53] ...trap/engines/engine.py.229: process: 0.79234 | load: 0.00313 [I 2022-10-08 19:22:53] ...trap/engines/engine.py.230: loss: 0.21568 [I 2022-10-08 19:22:53] ...optimizers/trijoint.py.151: total_norm: 2.921033 [I 2022-10-08 19:23:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 2290/2304 [I 2022-10-08 19:23:01] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:00:10 [I 2022-10-08 19:23:01] ...trap/engines/engine.py.229: process: 0.77333 | load: 0.00310 [I 2022-10-08 19:23:01] ...trap/engines/engine.py.230: loss: 0.15007 [I 2022-10-08 19:23:01] ...optimizers/trijoint.py.151: total_norm: 2.127484 [I 2022-10-08 19:23:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 2300/2304 [I 2022-10-08 19:23:09] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:00:03 [I 2022-10-08 19:23:09] ...trap/engines/engine.py.229: process: 0.76966 | load: 0.00017 [I 2022-10-08 19:23:09] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-08 19:23:09] ...optimizers/trijoint.py.151: total_norm: 2.392067 [I 2022-10-08 19:23:12] ...trap/engines/engine.py.224: train: epoch 16 | batch 2304/2304 [I 2022-10-08 19:23:12] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:00:00 [I 2022-10-08 19:23:12] ...trap/engines/engine.py.229: process: 0.77012 | load: 0.00017 [I 2022-10-08 19:23:12] ...trap/engines/engine.py.230: loss: 0.19026 [I 2022-10-08 19:23:12] ...optimizers/trijoint.py.151: total_norm: 3.071830 [S 2022-10-08 19:23:13] ...trap/engines/engine.py.129: train_epoch.epoch: 16 [S 2022-10-08 19:23:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.18462623923111893 [I 2022-10-08 19:23:13] ...s/criterions/triplet.py.84: margin = 0.13500000000000006 [I 2022-10-08 19:23:14] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 16 [I 2022-10-08 19:23:14] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 19:23:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 19:23:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 19:23:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 19:23:16] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 19:23:16] ...trap/engines/engine.py.321: eval: epoch 16 | batch 0/510 [I 2022-10-08 19:23:16] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:39 [I 2022-10-08 19:23:16] ...trap/engines/engine.py.326: process: 2.78413 | load: 2.28684 [I 2022-10-08 19:23:22] ...trap/engines/engine.py.321: eval: epoch 16 | batch 10/510 [I 2022-10-08 19:23:22] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:57 [I 2022-10-08 19:23:22] ...trap/engines/engine.py.326: process: 0.94133 | load: 0.00019 [I 2022-10-08 19:23:28] ...trap/engines/engine.py.321: eval: epoch 16 | batch 20/510 [I 2022-10-08 19:23:28] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:55 [I 2022-10-08 19:23:28] ...trap/engines/engine.py.326: process: 0.83163 | load: 0.00019 [I 2022-10-08 19:23:35] ...trap/engines/engine.py.321: eval: epoch 16 | batch 30/510 [I 2022-10-08 19:23:35] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:05:01 [I 2022-10-08 19:23:35] ...trap/engines/engine.py.326: process: 0.53141 | load: 0.00019 [I 2022-10-08 19:23:39] ...trap/engines/engine.py.321: eval: epoch 16 | batch 40/510 [I 2022-10-08 19:23:39] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:47 [I 2022-10-08 19:23:39] ...trap/engines/engine.py.326: process: 0.43842 | load: 0.00019 [I 2022-10-08 19:23:45] ...trap/engines/engine.py.321: eval: epoch 16 | batch 50/510 [I 2022-10-08 19:23:45] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:41 [I 2022-10-08 19:23:45] ...trap/engines/engine.py.326: process: 0.41864 | load: 0.00019 [I 2022-10-08 19:23:49] ...trap/engines/engine.py.321: eval: epoch 16 | batch 60/510 [I 2022-10-08 19:23:49] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:15 [I 2022-10-08 19:23:49] ...trap/engines/engine.py.326: process: 0.41924 | load: 0.00018 [I 2022-10-08 19:23:54] ...trap/engines/engine.py.321: eval: epoch 16 | batch 70/510 [I 2022-10-08 19:23:54] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:42 [I 2022-10-08 19:23:54] ...trap/engines/engine.py.326: process: 0.55696 | load: 0.00019 [I 2022-10-08 19:24:00] ...trap/engines/engine.py.321: eval: epoch 16 | batch 80/510 [I 2022-10-08 19:24:00] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:59 [I 2022-10-08 19:24:00] ...trap/engines/engine.py.326: process: 0.49668 | load: 0.00019 [I 2022-10-08 19:24:04] ...trap/engines/engine.py.321: eval: epoch 16 | batch 90/510 [I 2022-10-08 19:24:04] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:24 [I 2022-10-08 19:24:04] ...trap/engines/engine.py.326: process: 0.46737 | load: 0.00019 [I 2022-10-08 19:24:09] ...trap/engines/engine.py.321: eval: epoch 16 | batch 100/510 [I 2022-10-08 19:24:09] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:59 [I 2022-10-08 19:24:09] ...trap/engines/engine.py.326: process: 0.30915 | load: 0.00018 [I 2022-10-08 19:24:12] ...trap/engines/engine.py.321: eval: epoch 16 | batch 110/510 [I 2022-10-08 19:24:12] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-08 19:24:12] ...trap/engines/engine.py.326: process: 0.30959 | load: 0.00020 [I 2022-10-08 19:24:15] ...trap/engines/engine.py.321: eval: epoch 16 | batch 120/510 [I 2022-10-08 19:24:15] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-08 19:24:15] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00019 [I 2022-10-08 19:24:18] ...trap/engines/engine.py.321: eval: epoch 16 | batch 130/510 [I 2022-10-08 19:24:18] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-08 19:24:18] ...trap/engines/engine.py.326: process: 0.31873 | load: 0.00019 [I 2022-10-08 19:24:22] ...trap/engines/engine.py.321: eval: epoch 16 | batch 140/510 [I 2022-10-08 19:24:22] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:56 [I 2022-10-08 19:24:22] ...trap/engines/engine.py.326: process: 0.30833 | load: 0.00019 [I 2022-10-08 19:24:25] ...trap/engines/engine.py.321: eval: epoch 16 | batch 150/510 [I 2022-10-08 19:24:25] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:53 [I 2022-10-08 19:24:25] ...trap/engines/engine.py.326: process: 0.30888 | load: 0.00019 [I 2022-10-08 19:24:28] ...trap/engines/engine.py.321: eval: epoch 16 | batch 160/510 [I 2022-10-08 19:24:28] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-08 19:24:28] ...trap/engines/engine.py.326: process: 0.32060 | load: 0.00020 [I 2022-10-08 19:24:31] ...trap/engines/engine.py.321: eval: epoch 16 | batch 170/510 [I 2022-10-08 19:24:31] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-08 19:24:31] ...trap/engines/engine.py.326: process: 0.32286 | load: 0.00019 [I 2022-10-08 19:24:34] ...trap/engines/engine.py.321: eval: epoch 16 | batch 180/510 [I 2022-10-08 19:24:34] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-08 19:24:34] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00019 [I 2022-10-08 19:24:37] ...trap/engines/engine.py.321: eval: epoch 16 | batch 190/510 [I 2022-10-08 19:24:37] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-08 19:24:37] ...trap/engines/engine.py.326: process: 0.31806 | load: 0.00020 [I 2022-10-08 19:24:41] ...trap/engines/engine.py.321: eval: epoch 16 | batch 200/510 [I 2022-10-08 19:24:41] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-08 19:24:41] ...trap/engines/engine.py.326: process: 0.31028 | load: 0.00019 [I 2022-10-08 19:24:44] ...trap/engines/engine.py.321: eval: epoch 16 | batch 210/510 [I 2022-10-08 19:24:44] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-08 19:24:44] ...trap/engines/engine.py.326: process: 0.31942 | load: 0.00019 [I 2022-10-08 19:24:47] ...trap/engines/engine.py.321: eval: epoch 16 | batch 220/510 [I 2022-10-08 19:24:47] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:31 [I 2022-10-08 19:24:47] ...trap/engines/engine.py.326: process: 0.31147 | load: 0.00019 [I 2022-10-08 19:24:50] ...trap/engines/engine.py.321: eval: epoch 16 | batch 230/510 [I 2022-10-08 19:24:50] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-08 19:24:50] ...trap/engines/engine.py.326: process: 0.31845 | load: 0.00021 [I 2022-10-08 19:24:53] ...trap/engines/engine.py.321: eval: epoch 16 | batch 240/510 [I 2022-10-08 19:24:53] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-08 19:24:53] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00019 [I 2022-10-08 19:24:57] ...trap/engines/engine.py.321: eval: epoch 16 | batch 250/510 [I 2022-10-08 19:24:57] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-08 19:24:57] ...trap/engines/engine.py.326: process: 0.31850 | load: 0.00020 [I 2022-10-08 19:25:00] ...trap/engines/engine.py.321: eval: epoch 16 | batch 260/510 [I 2022-10-08 19:25:00] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-08 19:25:00] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00019 [I 2022-10-08 19:25:03] ...trap/engines/engine.py.321: eval: epoch 16 | batch 270/510 [I 2022-10-08 19:25:03] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-08 19:25:03] ...trap/engines/engine.py.326: process: 0.32264 | load: 0.00021 [I 2022-10-08 19:25:06] ...trap/engines/engine.py.321: eval: epoch 16 | batch 280/510 [I 2022-10-08 19:25:06] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:12 [I 2022-10-08 19:25:06] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-08 19:25:09] ...trap/engines/engine.py.321: eval: epoch 16 | batch 290/510 [I 2022-10-08 19:25:09] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-08 19:25:09] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00021 [I 2022-10-08 19:25:12] ...trap/engines/engine.py.321: eval: epoch 16 | batch 300/510 [I 2022-10-08 19:25:12] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-08 19:25:12] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00019 [I 2022-10-08 19:25:16] ...trap/engines/engine.py.321: eval: epoch 16 | batch 310/510 [I 2022-10-08 19:25:16] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-08 19:25:16] ...trap/engines/engine.py.326: process: 0.32641 | load: 0.00020 [I 2022-10-08 19:25:19] ...trap/engines/engine.py.321: eval: epoch 16 | batch 320/510 [I 2022-10-08 19:25:19] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-08 19:25:19] ...trap/engines/engine.py.326: process: 0.31803 | load: 0.00019 [I 2022-10-08 19:25:22] ...trap/engines/engine.py.321: eval: epoch 16 | batch 330/510 [I 2022-10-08 19:25:22] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-08 19:25:22] ...trap/engines/engine.py.326: process: 0.32139 | load: 0.00020 [I 2022-10-08 19:25:25] ...trap/engines/engine.py.321: eval: epoch 16 | batch 340/510 [I 2022-10-08 19:25:25] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-08 19:25:25] ...trap/engines/engine.py.326: process: 0.31799 | load: 0.00020 [I 2022-10-08 19:25:28] ...trap/engines/engine.py.321: eval: epoch 16 | batch 350/510 [I 2022-10-08 19:25:28] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-08 19:25:28] ...trap/engines/engine.py.326: process: 0.32389 | load: 0.00020 [I 2022-10-08 19:25:32] ...trap/engines/engine.py.321: eval: epoch 16 | batch 360/510 [I 2022-10-08 19:25:32] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-08 19:25:32] ...trap/engines/engine.py.326: process: 0.31943 | load: 0.00019 [I 2022-10-08 19:25:35] ...trap/engines/engine.py.321: eval: epoch 16 | batch 370/510 [I 2022-10-08 19:25:35] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-08 19:25:35] ...trap/engines/engine.py.326: process: 0.31048 | load: 0.00021 [I 2022-10-08 19:25:38] ...trap/engines/engine.py.321: eval: epoch 16 | batch 380/510 [I 2022-10-08 19:25:38] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-08 19:25:38] ...trap/engines/engine.py.326: process: 0.32059 | load: 0.00019 [I 2022-10-08 19:25:41] ...trap/engines/engine.py.321: eval: epoch 16 | batch 390/510 [I 2022-10-08 19:25:41] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-08 19:25:41] ...trap/engines/engine.py.326: process: 0.32186 | load: 0.00019 [I 2022-10-08 19:25:44] ...trap/engines/engine.py.321: eval: epoch 16 | batch 400/510 [I 2022-10-08 19:25:44] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-08 19:25:44] ...trap/engines/engine.py.326: process: 0.31652 | load: 0.00020 [I 2022-10-08 19:25:48] ...trap/engines/engine.py.321: eval: epoch 16 | batch 410/510 [I 2022-10-08 19:25:48] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-08 19:25:48] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00020 [I 2022-10-08 19:25:51] ...trap/engines/engine.py.321: eval: epoch 16 | batch 420/510 [I 2022-10-08 19:25:51] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-08 19:25:51] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00019 [I 2022-10-08 19:25:54] ...trap/engines/engine.py.321: eval: epoch 16 | batch 430/510 [I 2022-10-08 19:25:54] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-08 19:25:54] ...trap/engines/engine.py.326: process: 0.31700 | load: 0.00020 [I 2022-10-08 19:25:57] ...trap/engines/engine.py.321: eval: epoch 16 | batch 440/510 [I 2022-10-08 19:25:57] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-08 19:25:57] ...trap/engines/engine.py.326: process: 0.32013 | load: 0.00020 [I 2022-10-08 19:26:00] ...trap/engines/engine.py.321: eval: epoch 16 | batch 450/510 [I 2022-10-08 19:26:00] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-08 19:26:00] ...trap/engines/engine.py.326: process: 0.32063 | load: 0.00020 [I 2022-10-08 19:26:03] ...trap/engines/engine.py.321: eval: epoch 16 | batch 460/510 [I 2022-10-08 19:26:03] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-08 19:26:03] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00020 [I 2022-10-08 19:26:07] ...trap/engines/engine.py.321: eval: epoch 16 | batch 470/510 [I 2022-10-08 19:26:07] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-08 19:26:07] ...trap/engines/engine.py.326: process: 0.31531 | load: 0.00019 [I 2022-10-08 19:26:10] ...trap/engines/engine.py.321: eval: epoch 16 | batch 480/510 [I 2022-10-08 19:26:10] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-08 19:26:10] ...trap/engines/engine.py.326: process: 0.31583 | load: 0.00019 [I 2022-10-08 19:26:13] ...trap/engines/engine.py.321: eval: epoch 16 | batch 490/510 [I 2022-10-08 19:26:13] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-08 19:26:13] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00020 [I 2022-10-08 19:26:16] ...trap/engines/engine.py.321: eval: epoch 16 | batch 500/510 [I 2022-10-08 19:26:16] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-08 19:26:16] ...trap/engines/engine.py.326: process: 0.31588 | load: 0.00019 [I 2022-10-08 19:26:19] ...trap/engines/engine.py.321: eval: epoch 16 | batch 510/510 [I 2022-10-08 19:26:19] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-08 19:26:19] ...trap/engines/engine.py.326: process: 0.30975 | load: 0.00016 [S 2022-10-08 19:26:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 16 [I 2022-10-08 19:26:20] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.608900 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.855600 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.909000 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.608100 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.860600 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.911200 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016610 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007269 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009176 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013881 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010594 [S 2022-10-08 19:26:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007587 [I 2022-10-08 19:26:51] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 19:26:51] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 19:26:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 19:26:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 19:26:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 19:26:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 19:26:55] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 19:26:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 19:27:00] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 19:27:00] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 19:27:02] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 19:27:07] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 19:27:07] ...trap/engines/engine.py.163: Training model on trainset for epoch 17 [I 2022-10-08 19:27:07] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 19:27:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 0/2304 [I 2022-10-08 19:27:10] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:43 [I 2022-10-08 19:27:10] ...trap/engines/engine.py.229: process: 2.88355 | load: 2.04084 [I 2022-10-08 19:27:10] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-08 19:27:10] ...optimizers/trijoint.py.151: total_norm: 2.997281 [I 2022-10-08 19:27:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 10/2304 [I 2022-10-08 19:27:18] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:20 [I 2022-10-08 19:27:18] ...trap/engines/engine.py.229: process: 0.77027 | load: 0.00651 [I 2022-10-08 19:27:18] ...trap/engines/engine.py.230: loss: 0.16859 [I 2022-10-08 19:27:18] ...optimizers/trijoint.py.151: total_norm: 2.733415 [I 2022-10-08 19:27:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 20/2304 [I 2022-10-08 19:27:25] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:26 [I 2022-10-08 19:27:25] ...trap/engines/engine.py.229: process: 0.77572 | load: 0.00474 [I 2022-10-08 19:27:25] ...trap/engines/engine.py.230: loss: 0.13162 [I 2022-10-08 19:27:25] ...optimizers/trijoint.py.151: total_norm: 2.345531 [I 2022-10-08 19:27:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 30/2304 [I 2022-10-08 19:27:33] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:26 [I 2022-10-08 19:27:33] ...trap/engines/engine.py.229: process: 0.77536 | load: 0.00443 [I 2022-10-08 19:27:33] ...trap/engines/engine.py.230: loss: 0.25222 [I 2022-10-08 19:27:33] ...optimizers/trijoint.py.151: total_norm: 2.928090 [I 2022-10-08 19:27:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 40/2304 [I 2022-10-08 19:27:41] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:30 [I 2022-10-08 19:27:41] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00388 [I 2022-10-08 19:27:41] ...trap/engines/engine.py.230: loss: 0.21672 [I 2022-10-08 19:27:41] ...optimizers/trijoint.py.151: total_norm: 2.981375 [I 2022-10-08 19:27:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 50/2304 [I 2022-10-08 19:27:49] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:29:14 [I 2022-10-08 19:27:49] ...trap/engines/engine.py.229: process: 0.77476 | load: 0.00367 [I 2022-10-08 19:27:49] ...trap/engines/engine.py.230: loss: 0.16915 [I 2022-10-08 19:27:49] ...optimizers/trijoint.py.151: total_norm: 2.265044 [I 2022-10-08 19:27:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 60/2304 [I 2022-10-08 19:27:57] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:29:05 [I 2022-10-08 19:27:57] ...trap/engines/engine.py.229: process: 0.78329 | load: 0.00326 [I 2022-10-08 19:27:57] ...trap/engines/engine.py.230: loss: 0.19752 [I 2022-10-08 19:27:57] ...optimizers/trijoint.py.151: total_norm: 2.895559 [I 2022-10-08 19:28:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 70/2304 [I 2022-10-08 19:28:04] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:59 [I 2022-10-08 19:28:04] ...trap/engines/engine.py.229: process: 0.77504 | load: 0.00315 [I 2022-10-08 19:28:04] ...trap/engines/engine.py.230: loss: 0.21524 [I 2022-10-08 19:28:04] ...optimizers/trijoint.py.151: total_norm: 3.662396 [I 2022-10-08 19:28:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 80/2304 [I 2022-10-08 19:28:12] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:49 [I 2022-10-08 19:28:12] ...trap/engines/engine.py.229: process: 0.77545 | load: 0.00334 [I 2022-10-08 19:28:12] ...trap/engines/engine.py.230: loss: 0.16984 [I 2022-10-08 19:28:12] ...optimizers/trijoint.py.151: total_norm: 2.774509 [I 2022-10-08 19:28:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 90/2304 [I 2022-10-08 19:28:20] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:42 [I 2022-10-08 19:28:20] ...trap/engines/engine.py.229: process: 0.78163 | load: 0.00328 [I 2022-10-08 19:28:20] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-08 19:28:20] ...optimizers/trijoint.py.151: total_norm: 1.871407 [I 2022-10-08 19:28:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 100/2304 [I 2022-10-08 19:28:28] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:39 [I 2022-10-08 19:28:28] ...trap/engines/engine.py.229: process: 0.79075 | load: 0.00322 [I 2022-10-08 19:28:28] ...trap/engines/engine.py.230: loss: 0.22892 [I 2022-10-08 19:28:28] ...optimizers/trijoint.py.151: total_norm: 3.092946 [I 2022-10-08 19:28:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 110/2304 [I 2022-10-08 19:28:36] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:29 [I 2022-10-08 19:28:36] ...trap/engines/engine.py.229: process: 0.78022 | load: 0.00318 [I 2022-10-08 19:28:36] ...trap/engines/engine.py.230: loss: 0.17768 [I 2022-10-08 19:28:36] ...optimizers/trijoint.py.151: total_norm: 3.896132 [I 2022-10-08 19:28:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 120/2304 [I 2022-10-08 19:28:43] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:28 [I 2022-10-08 19:28:43] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00355 [I 2022-10-08 19:28:43] ...trap/engines/engine.py.230: loss: 0.16150 [I 2022-10-08 19:28:43] ...optimizers/trijoint.py.151: total_norm: 1.648743 [I 2022-10-08 19:28:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 130/2304 [I 2022-10-08 19:28:51] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:20 [I 2022-10-08 19:28:51] ...trap/engines/engine.py.229: process: 0.78379 | load: 0.00315 [I 2022-10-08 19:28:51] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-08 19:28:51] ...optimizers/trijoint.py.151: total_norm: 2.441198 [I 2022-10-08 19:28:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 140/2304 [I 2022-10-08 19:28:59] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:10 [I 2022-10-08 19:28:59] ...trap/engines/engine.py.229: process: 0.79125 | load: 0.00329 [I 2022-10-08 19:28:59] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-08 19:28:59] ...optimizers/trijoint.py.151: total_norm: 2.460912 [I 2022-10-08 19:29:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 150/2304 [I 2022-10-08 19:29:07] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:27:58 [I 2022-10-08 19:29:07] ...trap/engines/engine.py.229: process: 0.78384 | load: 0.00318 [I 2022-10-08 19:29:07] ...trap/engines/engine.py.230: loss: 0.16424 [I 2022-10-08 19:29:07] ...optimizers/trijoint.py.151: total_norm: 2.883282 [I 2022-10-08 19:29:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 160/2304 [I 2022-10-08 19:29:15] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:57 [I 2022-10-08 19:29:15] ...trap/engines/engine.py.229: process: 0.80061 | load: 0.00332 [I 2022-10-08 19:29:15] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-08 19:29:15] ...optimizers/trijoint.py.151: total_norm: 1.763970 [I 2022-10-08 19:29:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 170/2304 [I 2022-10-08 19:29:22] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:56 [I 2022-10-08 19:29:22] ...trap/engines/engine.py.229: process: 0.78695 | load: 0.00318 [I 2022-10-08 19:29:22] ...trap/engines/engine.py.230: loss: 0.17263 [I 2022-10-08 19:29:22] ...optimizers/trijoint.py.151: total_norm: 2.252250 [I 2022-10-08 19:29:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 180/2304 [I 2022-10-08 19:29:30] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:42 [I 2022-10-08 19:29:30] ...trap/engines/engine.py.229: process: 0.79064 | load: 0.00330 [I 2022-10-08 19:29:30] ...trap/engines/engine.py.230: loss: 0.11611 [I 2022-10-08 19:29:30] ...optimizers/trijoint.py.151: total_norm: 1.847546 [I 2022-10-08 19:29:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 190/2304 [I 2022-10-08 19:29:38] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:38 [I 2022-10-08 19:29:38] ...trap/engines/engine.py.229: process: 0.77563 | load: 0.00321 [I 2022-10-08 19:29:38] ...trap/engines/engine.py.230: loss: 0.18231 [I 2022-10-08 19:29:38] ...optimizers/trijoint.py.151: total_norm: 2.481514 [I 2022-10-08 19:29:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 200/2304 [I 2022-10-08 19:29:46] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:15 [I 2022-10-08 19:29:46] ...trap/engines/engine.py.229: process: 0.77256 | load: 0.00305 [I 2022-10-08 19:29:46] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-08 19:29:46] ...optimizers/trijoint.py.151: total_norm: 2.122333 [I 2022-10-08 19:29:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 210/2304 [I 2022-10-08 19:29:54] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:16 [I 2022-10-08 19:29:54] ...trap/engines/engine.py.229: process: 0.77731 | load: 0.00303 [I 2022-10-08 19:29:54] ...trap/engines/engine.py.230: loss: 0.18184 [I 2022-10-08 19:29:54] ...optimizers/trijoint.py.151: total_norm: 2.781149 [I 2022-10-08 19:30:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 220/2304 [I 2022-10-08 19:30:02] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:26:59 [I 2022-10-08 19:30:02] ...trap/engines/engine.py.229: process: 0.77442 | load: 0.00314 [I 2022-10-08 19:30:02] ...trap/engines/engine.py.230: loss: 0.16971 [I 2022-10-08 19:30:02] ...optimizers/trijoint.py.151: total_norm: 2.134662 [I 2022-10-08 19:30:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 230/2304 [I 2022-10-08 19:30:09] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:49 [I 2022-10-08 19:30:09] ...trap/engines/engine.py.229: process: 0.77954 | load: 0.00313 [I 2022-10-08 19:30:09] ...trap/engines/engine.py.230: loss: 0.19114 [I 2022-10-08 19:30:09] ...optimizers/trijoint.py.151: total_norm: 2.374628 [I 2022-10-08 19:30:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 240/2304 [I 2022-10-08 19:30:17] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:43 [I 2022-10-08 19:30:17] ...trap/engines/engine.py.229: process: 0.77690 | load: 0.00305 [I 2022-10-08 19:30:17] ...trap/engines/engine.py.230: loss: 0.26743 [I 2022-10-08 19:30:17] ...optimizers/trijoint.py.151: total_norm: 2.401855 [I 2022-10-08 19:30:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 250/2304 [I 2022-10-08 19:30:25] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:31 [I 2022-10-08 19:30:25] ...trap/engines/engine.py.229: process: 0.77757 | load: 0.00302 [I 2022-10-08 19:30:25] ...trap/engines/engine.py.230: loss: 0.12892 [I 2022-10-08 19:30:25] ...optimizers/trijoint.py.151: total_norm: 2.418257 [I 2022-10-08 19:30:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 260/2304 [I 2022-10-08 19:30:33] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:37 [I 2022-10-08 19:30:33] ...trap/engines/engine.py.229: process: 0.79591 | load: 0.00299 [I 2022-10-08 19:30:33] ...trap/engines/engine.py.230: loss: 0.24722 [I 2022-10-08 19:30:33] ...optimizers/trijoint.py.151: total_norm: 2.789945 [I 2022-10-08 19:30:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 270/2304 [I 2022-10-08 19:30:41] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:29 [I 2022-10-08 19:30:41] ...trap/engines/engine.py.229: process: 0.79122 | load: 0.00326 [I 2022-10-08 19:30:41] ...trap/engines/engine.py.230: loss: 0.18769 [I 2022-10-08 19:30:41] ...optimizers/trijoint.py.151: total_norm: 2.179031 [I 2022-10-08 19:30:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 280/2304 [I 2022-10-08 19:30:48] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:08 [I 2022-10-08 19:30:48] ...trap/engines/engine.py.229: process: 0.76921 | load: 0.00303 [I 2022-10-08 19:30:48] ...trap/engines/engine.py.230: loss: 0.15352 [I 2022-10-08 19:30:48] ...optimizers/trijoint.py.151: total_norm: 2.754589 [I 2022-10-08 19:30:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 290/2304 [I 2022-10-08 19:30:56] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:00 [I 2022-10-08 19:30:56] ...trap/engines/engine.py.229: process: 0.76598 | load: 0.00330 [I 2022-10-08 19:30:56] ...trap/engines/engine.py.230: loss: 0.16535 [I 2022-10-08 19:30:56] ...optimizers/trijoint.py.151: total_norm: 2.102022 [I 2022-10-08 19:31:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 300/2304 [I 2022-10-08 19:31:04] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:59 [I 2022-10-08 19:31:04] ...trap/engines/engine.py.229: process: 0.77869 | load: 0.00308 [I 2022-10-08 19:31:04] ...trap/engines/engine.py.230: loss: 0.15473 [I 2022-10-08 19:31:04] ...optimizers/trijoint.py.151: total_norm: 2.908590 [I 2022-10-08 19:31:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 310/2304 [I 2022-10-08 19:31:12] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:25:49 [I 2022-10-08 19:31:12] ...trap/engines/engine.py.229: process: 0.77816 | load: 0.00299 [I 2022-10-08 19:31:12] ...trap/engines/engine.py.230: loss: 0.23822 [I 2022-10-08 19:31:12] ...optimizers/trijoint.py.151: total_norm: 3.396319 [I 2022-10-08 19:31:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 320/2304 [I 2022-10-08 19:31:19] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:47 [I 2022-10-08 19:31:19] ...trap/engines/engine.py.229: process: 0.77997 | load: 0.00302 [I 2022-10-08 19:31:19] ...trap/engines/engine.py.230: loss: 0.17350 [I 2022-10-08 19:31:19] ...optimizers/trijoint.py.151: total_norm: 2.565050 [I 2022-10-08 19:31:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 330/2304 [I 2022-10-08 19:31:27] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:29 [I 2022-10-08 19:31:27] ...trap/engines/engine.py.229: process: 0.76940 | load: 0.00300 [I 2022-10-08 19:31:27] ...trap/engines/engine.py.230: loss: 0.22961 [I 2022-10-08 19:31:27] ...optimizers/trijoint.py.151: total_norm: 2.977667 [I 2022-10-08 19:31:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 340/2304 [I 2022-10-08 19:31:35] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:27 [I 2022-10-08 19:31:35] ...trap/engines/engine.py.229: process: 0.77605 | load: 0.00301 [I 2022-10-08 19:31:35] ...trap/engines/engine.py.230: loss: 0.16751 [I 2022-10-08 19:31:35] ...optimizers/trijoint.py.151: total_norm: 2.344428 [I 2022-10-08 19:31:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 350/2304 [I 2022-10-08 19:31:43] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:21 [I 2022-10-08 19:31:43] ...trap/engines/engine.py.229: process: 0.77400 | load: 0.00300 [I 2022-10-08 19:31:43] ...trap/engines/engine.py.230: loss: 0.12774 [I 2022-10-08 19:31:43] ...optimizers/trijoint.py.151: total_norm: 2.971970 [I 2022-10-08 19:31:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 360/2304 [I 2022-10-08 19:31:51] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:12 [I 2022-10-08 19:31:51] ...trap/engines/engine.py.229: process: 0.78549 | load: 0.00303 [I 2022-10-08 19:31:51] ...trap/engines/engine.py.230: loss: 0.23241 [I 2022-10-08 19:31:51] ...optimizers/trijoint.py.151: total_norm: 3.075367 [I 2022-10-08 19:31:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 370/2304 [I 2022-10-08 19:31:59] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:15 [I 2022-10-08 19:31:59] ...trap/engines/engine.py.229: process: 0.79305 | load: 0.00303 [I 2022-10-08 19:31:59] ...trap/engines/engine.py.230: loss: 0.15833 [I 2022-10-08 19:31:59] ...optimizers/trijoint.py.151: total_norm: 2.469945 [I 2022-10-08 19:32:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 380/2304 [I 2022-10-08 19:32:06] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:02 [I 2022-10-08 19:32:06] ...trap/engines/engine.py.229: process: 0.77521 | load: 0.00304 [I 2022-10-08 19:32:06] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-08 19:32:06] ...optimizers/trijoint.py.151: total_norm: 2.863528 [I 2022-10-08 19:32:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 390/2304 [I 2022-10-08 19:32:14] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:24:45 [I 2022-10-08 19:32:14] ...trap/engines/engine.py.229: process: 0.77333 | load: 0.00300 [I 2022-10-08 19:32:14] ...trap/engines/engine.py.230: loss: 0.17576 [I 2022-10-08 19:32:14] ...optimizers/trijoint.py.151: total_norm: 3.433568 [I 2022-10-08 19:32:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 400/2304 [I 2022-10-08 19:32:22] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:44 [I 2022-10-08 19:32:22] ...trap/engines/engine.py.229: process: 0.76986 | load: 0.00300 [I 2022-10-08 19:32:22] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-08 19:32:22] ...optimizers/trijoint.py.151: total_norm: 2.308624 [I 2022-10-08 19:32:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 410/2304 [I 2022-10-08 19:32:30] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:24:41 [I 2022-10-08 19:32:30] ...trap/engines/engine.py.229: process: 0.79165 | load: 0.00303 [I 2022-10-08 19:32:30] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-08 19:32:30] ...optimizers/trijoint.py.151: total_norm: 2.429640 [I 2022-10-08 19:32:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 420/2304 [I 2022-10-08 19:32:38] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:24:30 [I 2022-10-08 19:32:38] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00306 [I 2022-10-08 19:32:38] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-08 19:32:38] ...optimizers/trijoint.py.151: total_norm: 2.944960 [I 2022-10-08 19:32:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 430/2304 [I 2022-10-08 19:32:45] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:13 [I 2022-10-08 19:32:45] ...trap/engines/engine.py.229: process: 0.76802 | load: 0.00302 [I 2022-10-08 19:32:45] ...trap/engines/engine.py.230: loss: 0.18847 [I 2022-10-08 19:32:45] ...optimizers/trijoint.py.151: total_norm: 4.636451 [I 2022-10-08 19:32:53] ...trap/engines/engine.py.224: train: epoch 17 | batch 440/2304 [I 2022-10-08 19:32:53] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:10 [I 2022-10-08 19:32:53] ...trap/engines/engine.py.229: process: 0.76793 | load: 0.00304 [I 2022-10-08 19:32:53] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-08 19:32:53] ...optimizers/trijoint.py.151: total_norm: 1.982248 [I 2022-10-08 19:33:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 450/2304 [I 2022-10-08 19:33:01] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:11 [I 2022-10-08 19:33:01] ...trap/engines/engine.py.229: process: 0.80327 | load: 0.00303 [I 2022-10-08 19:33:01] ...trap/engines/engine.py.230: loss: 0.17391 [I 2022-10-08 19:33:01] ...optimizers/trijoint.py.151: total_norm: 2.670885 [I 2022-10-08 19:33:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 460/2304 [I 2022-10-08 19:33:09] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:57 [I 2022-10-08 19:33:09] ...trap/engines/engine.py.229: process: 0.77956 | load: 0.00303 [I 2022-10-08 19:33:09] ...trap/engines/engine.py.230: loss: 0.13766 [I 2022-10-08 19:33:09] ...optimizers/trijoint.py.151: total_norm: 3.355302 [I 2022-10-08 19:33:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 470/2304 [I 2022-10-08 19:33:17] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:23:49 [I 2022-10-08 19:33:17] ...trap/engines/engine.py.229: process: 0.78438 | load: 0.00298 [I 2022-10-08 19:33:17] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-08 19:33:17] ...optimizers/trijoint.py.151: total_norm: 2.141988 [I 2022-10-08 19:33:24] ...trap/engines/engine.py.224: train: epoch 17 | batch 480/2304 [I 2022-10-08 19:33:24] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:23:46 [I 2022-10-08 19:33:24] ...trap/engines/engine.py.229: process: 0.79117 | load: 0.00302 [I 2022-10-08 19:33:24] ...trap/engines/engine.py.230: loss: 0.18100 [I 2022-10-08 19:33:24] ...optimizers/trijoint.py.151: total_norm: 2.979720 [I 2022-10-08 19:33:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 490/2304 [I 2022-10-08 19:33:32] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:37 [I 2022-10-08 19:33:32] ...trap/engines/engine.py.229: process: 0.78793 | load: 0.00305 [I 2022-10-08 19:33:32] ...trap/engines/engine.py.230: loss: 0.16938 [I 2022-10-08 19:33:32] ...optimizers/trijoint.py.151: total_norm: 2.442777 [I 2022-10-08 19:33:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 500/2304 [I 2022-10-08 19:33:40] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:28 [I 2022-10-08 19:33:40] ...trap/engines/engine.py.229: process: 0.76912 | load: 0.00301 [I 2022-10-08 19:33:40] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-08 19:33:40] ...optimizers/trijoint.py.151: total_norm: 2.074740 [I 2022-10-08 19:33:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 510/2304 [I 2022-10-08 19:33:48] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:22 [I 2022-10-08 19:33:48] ...trap/engines/engine.py.229: process: 0.78682 | load: 0.00305 [I 2022-10-08 19:33:48] ...trap/engines/engine.py.230: loss: 0.23613 [I 2022-10-08 19:33:48] ...optimizers/trijoint.py.151: total_norm: 3.101434 [I 2022-10-08 19:33:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 520/2304 [I 2022-10-08 19:33:56] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:23:04 [I 2022-10-08 19:33:56] ...trap/engines/engine.py.229: process: 0.76367 | load: 0.00344 [I 2022-10-08 19:33:56] ...trap/engines/engine.py.230: loss: 0.16637 [I 2022-10-08 19:33:56] ...optimizers/trijoint.py.151: total_norm: 3.081531 [I 2022-10-08 19:34:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 530/2304 [I 2022-10-08 19:34:04] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:23:03 [I 2022-10-08 19:34:04] ...trap/engines/engine.py.229: process: 0.78468 | load: 0.00308 [I 2022-10-08 19:34:04] ...trap/engines/engine.py.230: loss: 0.23139 [I 2022-10-08 19:34:04] ...optimizers/trijoint.py.151: total_norm: 4.200491 [I 2022-10-08 19:34:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 540/2304 [I 2022-10-08 19:34:12] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:22:57 [I 2022-10-08 19:34:12] ...trap/engines/engine.py.229: process: 0.77286 | load: 0.00300 [I 2022-10-08 19:34:12] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-08 19:34:12] ...optimizers/trijoint.py.151: total_norm: 3.358159 [I 2022-10-08 19:34:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 550/2304 [I 2022-10-08 19:34:19] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:53 [I 2022-10-08 19:34:19] ...trap/engines/engine.py.229: process: 0.77230 | load: 0.00298 [I 2022-10-08 19:34:19] ...trap/engines/engine.py.230: loss: 0.18166 [I 2022-10-08 19:34:19] ...optimizers/trijoint.py.151: total_norm: 2.346300 [I 2022-10-08 19:34:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 560/2304 [I 2022-10-08 19:34:27] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:39 [I 2022-10-08 19:34:27] ...trap/engines/engine.py.229: process: 0.78248 | load: 0.00306 [I 2022-10-08 19:34:27] ...trap/engines/engine.py.230: loss: 0.20474 [I 2022-10-08 19:34:27] ...optimizers/trijoint.py.151: total_norm: 2.908590 [I 2022-10-08 19:34:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 570/2304 [I 2022-10-08 19:34:35] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:28 [I 2022-10-08 19:34:35] ...trap/engines/engine.py.229: process: 0.77327 | load: 0.00300 [I 2022-10-08 19:34:35] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-08 19:34:35] ...optimizers/trijoint.py.151: total_norm: 1.470893 [I 2022-10-08 19:34:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 580/2304 [I 2022-10-08 19:34:43] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:19 [I 2022-10-08 19:34:43] ...trap/engines/engine.py.229: process: 0.77497 | load: 0.00294 [I 2022-10-08 19:34:43] ...trap/engines/engine.py.230: loss: 0.27707 [I 2022-10-08 19:34:43] ...optimizers/trijoint.py.151: total_norm: 3.652586 [I 2022-10-08 19:34:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 590/2304 [I 2022-10-08 19:34:51] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:22:04 [I 2022-10-08 19:34:51] ...trap/engines/engine.py.229: process: 0.76342 | load: 0.00293 [I 2022-10-08 19:34:51] ...trap/engines/engine.py.230: loss: 0.17335 [I 2022-10-08 19:34:51] ...optimizers/trijoint.py.151: total_norm: 2.100763 [I 2022-10-08 19:34:58] ...trap/engines/engine.py.224: train: epoch 17 | batch 600/2304 [I 2022-10-08 19:34:58] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:21:58 [I 2022-10-08 19:34:58] ...trap/engines/engine.py.229: process: 0.78064 | load: 0.00296 [I 2022-10-08 19:34:58] ...trap/engines/engine.py.230: loss: 0.23745 [I 2022-10-08 19:34:58] ...optimizers/trijoint.py.151: total_norm: 3.686885 [I 2022-10-08 19:35:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 610/2304 [I 2022-10-08 19:35:06] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:21:52 [I 2022-10-08 19:35:06] ...trap/engines/engine.py.229: process: 0.77898 | load: 0.00320 [I 2022-10-08 19:35:06] ...trap/engines/engine.py.230: loss: 0.16236 [I 2022-10-08 19:35:06] ...optimizers/trijoint.py.151: total_norm: 2.097773 [I 2022-10-08 19:35:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 620/2304 [I 2022-10-08 19:35:14] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:44 [I 2022-10-08 19:35:14] ...trap/engines/engine.py.229: process: 0.77620 | load: 0.00297 [I 2022-10-08 19:35:14] ...trap/engines/engine.py.230: loss: 0.21559 [I 2022-10-08 19:35:14] ...optimizers/trijoint.py.151: total_norm: 3.572807 [I 2022-10-08 19:35:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 630/2304 [I 2022-10-08 19:35:22] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:21:49 [I 2022-10-08 19:35:22] ...trap/engines/engine.py.229: process: 0.78033 | load: 0.00294 [I 2022-10-08 19:35:22] ...trap/engines/engine.py.230: loss: 0.20660 [I 2022-10-08 19:35:22] ...optimizers/trijoint.py.151: total_norm: 2.030041 [I 2022-10-08 19:35:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 640/2304 [I 2022-10-08 19:35:30] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:21:34 [I 2022-10-08 19:35:30] ...trap/engines/engine.py.229: process: 0.77833 | load: 0.00304 [I 2022-10-08 19:35:30] ...trap/engines/engine.py.230: loss: 0.19634 [I 2022-10-08 19:35:30] ...optimizers/trijoint.py.151: total_norm: 2.397079 [I 2022-10-08 19:35:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 650/2304 [I 2022-10-08 19:35:37] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:21:22 [I 2022-10-08 19:35:37] ...trap/engines/engine.py.229: process: 0.76749 | load: 0.00305 [I 2022-10-08 19:35:37] ...trap/engines/engine.py.230: loss: 0.19970 [I 2022-10-08 19:35:37] ...optimizers/trijoint.py.151: total_norm: 3.739192 [I 2022-10-08 19:35:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 660/2304 [I 2022-10-08 19:35:45] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:18 [I 2022-10-08 19:35:45] ...trap/engines/engine.py.229: process: 0.78219 | load: 0.00312 [I 2022-10-08 19:35:45] ...trap/engines/engine.py.230: loss: 0.15351 [I 2022-10-08 19:35:45] ...optimizers/trijoint.py.151: total_norm: 2.111516 [I 2022-10-08 19:35:53] ...trap/engines/engine.py.224: train: epoch 17 | batch 670/2304 [I 2022-10-08 19:35:53] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:15 [I 2022-10-08 19:35:53] ...trap/engines/engine.py.229: process: 0.78142 | load: 0.00305 [I 2022-10-08 19:35:53] ...trap/engines/engine.py.230: loss: 0.18554 [I 2022-10-08 19:35:53] ...optimizers/trijoint.py.151: total_norm: 3.234780 [I 2022-10-08 19:36:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 680/2304 [I 2022-10-08 19:36:01] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:07 [I 2022-10-08 19:36:01] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00309 [I 2022-10-08 19:36:01] ...trap/engines/engine.py.230: loss: 0.19157 [I 2022-10-08 19:36:01] ...optimizers/trijoint.py.151: total_norm: 2.926941 [I 2022-10-08 19:36:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 690/2304 [I 2022-10-08 19:36:09] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:20:56 [I 2022-10-08 19:36:09] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00316 [I 2022-10-08 19:36:09] ...trap/engines/engine.py.230: loss: 0.28305 [I 2022-10-08 19:36:09] ...optimizers/trijoint.py.151: total_norm: 2.915627 [I 2022-10-08 19:36:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 700/2304 [I 2022-10-08 19:36:16] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:20:54 [I 2022-10-08 19:36:16] ...trap/engines/engine.py.229: process: 0.78278 | load: 0.00304 [I 2022-10-08 19:36:16] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-08 19:36:16] ...optimizers/trijoint.py.151: total_norm: 2.435840 [I 2022-10-08 19:36:24] ...trap/engines/engine.py.224: train: epoch 17 | batch 710/2304 [I 2022-10-08 19:36:24] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:36 [I 2022-10-08 19:36:24] ...trap/engines/engine.py.229: process: 0.76696 | load: 0.00305 [I 2022-10-08 19:36:24] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-08 19:36:24] ...optimizers/trijoint.py.151: total_norm: 3.781533 [I 2022-10-08 19:36:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 720/2304 [I 2022-10-08 19:36:32] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:34 [I 2022-10-08 19:36:32] ...trap/engines/engine.py.229: process: 0.77928 | load: 0.00302 [I 2022-10-08 19:36:32] ...trap/engines/engine.py.230: loss: 0.23816 [I 2022-10-08 19:36:32] ...optimizers/trijoint.py.151: total_norm: 3.397279 [I 2022-10-08 19:36:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 730/2304 [I 2022-10-08 19:36:40] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:18 [I 2022-10-08 19:36:40] ...trap/engines/engine.py.229: process: 0.78095 | load: 0.00304 [I 2022-10-08 19:36:40] ...trap/engines/engine.py.230: loss: 0.17496 [I 2022-10-08 19:36:40] ...optimizers/trijoint.py.151: total_norm: 3.544851 [I 2022-10-08 19:36:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 740/2304 [I 2022-10-08 19:36:48] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:20:19 [I 2022-10-08 19:36:48] ...trap/engines/engine.py.229: process: 0.78711 | load: 0.00298 [I 2022-10-08 19:36:48] ...trap/engines/engine.py.230: loss: 0.23075 [I 2022-10-08 19:36:48] ...optimizers/trijoint.py.151: total_norm: 2.605248 [I 2022-10-08 19:36:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 750/2304 [I 2022-10-08 19:36:56] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:20:15 [I 2022-10-08 19:36:56] ...trap/engines/engine.py.229: process: 0.78656 | load: 0.00304 [I 2022-10-08 19:36:56] ...trap/engines/engine.py.230: loss: 0.17313 [I 2022-10-08 19:36:56] ...optimizers/trijoint.py.151: total_norm: 1.839820 [I 2022-10-08 19:37:03] ...trap/engines/engine.py.224: train: epoch 17 | batch 760/2304 [I 2022-10-08 19:37:03] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:20:09 [I 2022-10-08 19:37:03] ...trap/engines/engine.py.229: process: 0.77959 | load: 0.00305 [I 2022-10-08 19:37:03] ...trap/engines/engine.py.230: loss: 0.22535 [I 2022-10-08 19:37:03] ...optimizers/trijoint.py.151: total_norm: 2.534319 [I 2022-10-08 19:37:11] ...trap/engines/engine.py.224: train: epoch 17 | batch 770/2304 [I 2022-10-08 19:37:11] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:19:59 [I 2022-10-08 19:37:11] ...trap/engines/engine.py.229: process: 0.78304 | load: 0.00307 [I 2022-10-08 19:37:11] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-08 19:37:11] ...optimizers/trijoint.py.151: total_norm: 2.502932 [I 2022-10-08 19:37:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 780/2304 [I 2022-10-08 19:37:19] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:48 [I 2022-10-08 19:37:19] ...trap/engines/engine.py.229: process: 0.77520 | load: 0.00300 [I 2022-10-08 19:37:19] ...trap/engines/engine.py.230: loss: 0.17363 [I 2022-10-08 19:37:19] ...optimizers/trijoint.py.151: total_norm: 2.457414 [I 2022-10-08 19:37:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 790/2304 [I 2022-10-08 19:37:27] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:41 [I 2022-10-08 19:37:27] ...trap/engines/engine.py.229: process: 0.78413 | load: 0.00306 [I 2022-10-08 19:37:27] ...trap/engines/engine.py.230: loss: 0.13501 [I 2022-10-08 19:37:27] ...optimizers/trijoint.py.151: total_norm: 2.279123 [I 2022-10-08 19:37:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 800/2304 [I 2022-10-08 19:37:35] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:19:32 [I 2022-10-08 19:37:35] ...trap/engines/engine.py.229: process: 0.78075 | load: 0.00307 [I 2022-10-08 19:37:35] ...trap/engines/engine.py.230: loss: 0.27771 [I 2022-10-08 19:37:35] ...optimizers/trijoint.py.151: total_norm: 3.439217 [I 2022-10-08 19:37:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 810/2304 [I 2022-10-08 19:37:42] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:19:24 [I 2022-10-08 19:37:42] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00302 [I 2022-10-08 19:37:42] ...trap/engines/engine.py.230: loss: 0.16276 [I 2022-10-08 19:37:42] ...optimizers/trijoint.py.151: total_norm: 1.878230 [I 2022-10-08 19:37:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 820/2304 [I 2022-10-08 19:37:50] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:10 [I 2022-10-08 19:37:50] ...trap/engines/engine.py.229: process: 0.77792 | load: 0.00302 [I 2022-10-08 19:37:50] ...trap/engines/engine.py.230: loss: 0.24269 [I 2022-10-08 19:37:50] ...optimizers/trijoint.py.151: total_norm: 3.625156 [I 2022-10-08 19:37:58] ...trap/engines/engine.py.224: train: epoch 17 | batch 830/2304 [I 2022-10-08 19:37:58] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:10 [I 2022-10-08 19:37:58] ...trap/engines/engine.py.229: process: 0.78117 | load: 0.00302 [I 2022-10-08 19:37:58] ...trap/engines/engine.py.230: loss: 0.17205 [I 2022-10-08 19:37:58] ...optimizers/trijoint.py.151: total_norm: 2.341375 [I 2022-10-08 19:38:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 840/2304 [I 2022-10-08 19:38:06] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:18:59 [I 2022-10-08 19:38:06] ...trap/engines/engine.py.229: process: 0.79081 | load: 0.00302 [I 2022-10-08 19:38:06] ...trap/engines/engine.py.230: loss: 0.24812 [I 2022-10-08 19:38:06] ...optimizers/trijoint.py.151: total_norm: 3.303782 [I 2022-10-08 19:38:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 850/2304 [I 2022-10-08 19:38:14] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:18:53 [I 2022-10-08 19:38:14] ...trap/engines/engine.py.229: process: 0.78804 | load: 0.00309 [I 2022-10-08 19:38:14] ...trap/engines/engine.py.230: loss: 0.18338 [I 2022-10-08 19:38:14] ...optimizers/trijoint.py.151: total_norm: 2.425762 [I 2022-10-08 19:38:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 860/2304 [I 2022-10-08 19:38:21] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:18:34 [I 2022-10-08 19:38:21] ...trap/engines/engine.py.229: process: 0.76483 | load: 0.00301 [I 2022-10-08 19:38:21] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-08 19:38:21] ...optimizers/trijoint.py.151: total_norm: 2.376358 [I 2022-10-08 19:38:29] ...trap/engines/engine.py.224: train: epoch 17 | batch 870/2304 [I 2022-10-08 19:38:29] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:18:37 [I 2022-10-08 19:38:29] ...trap/engines/engine.py.229: process: 0.77500 | load: 0.00303 [I 2022-10-08 19:38:29] ...trap/engines/engine.py.230: loss: 0.16270 [I 2022-10-08 19:38:29] ...optimizers/trijoint.py.151: total_norm: 2.410234 [I 2022-10-08 19:38:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 880/2304 [I 2022-10-08 19:38:37] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:28 [I 2022-10-08 19:38:37] ...trap/engines/engine.py.229: process: 0.77030 | load: 0.00312 [I 2022-10-08 19:38:37] ...trap/engines/engine.py.230: loss: 0.18087 [I 2022-10-08 19:38:37] ...optimizers/trijoint.py.151: total_norm: 2.984991 [I 2022-10-08 19:38:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 890/2304 [I 2022-10-08 19:38:45] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:18:21 [I 2022-10-08 19:38:45] ...trap/engines/engine.py.229: process: 0.77103 | load: 0.00310 [I 2022-10-08 19:38:45] ...trap/engines/engine.py.230: loss: 0.19707 [I 2022-10-08 19:38:45] ...optimizers/trijoint.py.151: total_norm: 2.657019 [I 2022-10-08 19:38:53] ...trap/engines/engine.py.224: train: epoch 17 | batch 900/2304 [I 2022-10-08 19:38:53] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:18:16 [I 2022-10-08 19:38:53] ...trap/engines/engine.py.229: process: 0.78189 | load: 0.00307 [I 2022-10-08 19:38:53] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-08 19:38:53] ...optimizers/trijoint.py.151: total_norm: 2.268167 [I 2022-10-08 19:39:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 910/2304 [I 2022-10-08 19:39:00] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:18:06 [I 2022-10-08 19:39:00] ...trap/engines/engine.py.229: process: 0.77243 | load: 0.00311 [I 2022-10-08 19:39:00] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-08 19:39:00] ...optimizers/trijoint.py.151: total_norm: 1.715041 [I 2022-10-08 19:39:08] ...trap/engines/engine.py.224: train: epoch 17 | batch 920/2304 [I 2022-10-08 19:39:08] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:18:01 [I 2022-10-08 19:39:08] ...trap/engines/engine.py.229: process: 0.78817 | load: 0.00313 [I 2022-10-08 19:39:08] ...trap/engines/engine.py.230: loss: 0.22108 [I 2022-10-08 19:39:08] ...optimizers/trijoint.py.151: total_norm: 3.337282 [I 2022-10-08 19:39:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 930/2304 [I 2022-10-08 19:39:16] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:17:46 [I 2022-10-08 19:39:16] ...trap/engines/engine.py.229: process: 0.78100 | load: 0.00309 [I 2022-10-08 19:39:16] ...trap/engines/engine.py.230: loss: 0.24718 [I 2022-10-08 19:39:16] ...optimizers/trijoint.py.151: total_norm: 3.116819 [I 2022-10-08 19:39:24] ...trap/engines/engine.py.224: train: epoch 17 | batch 940/2304 [I 2022-10-08 19:39:24] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:47 [I 2022-10-08 19:39:24] ...trap/engines/engine.py.229: process: 0.78547 | load: 0.00311 [I 2022-10-08 19:39:24] ...trap/engines/engine.py.230: loss: 0.21026 [I 2022-10-08 19:39:24] ...optimizers/trijoint.py.151: total_norm: 2.401765 [I 2022-10-08 19:39:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 950/2304 [I 2022-10-08 19:39:32] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:17:38 [I 2022-10-08 19:39:32] ...trap/engines/engine.py.229: process: 0.77215 | load: 0.00307 [I 2022-10-08 19:39:32] ...trap/engines/engine.py.230: loss: 0.19774 [I 2022-10-08 19:39:32] ...optimizers/trijoint.py.151: total_norm: 2.528684 [I 2022-10-08 19:39:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 960/2304 [I 2022-10-08 19:39:40] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:17:23 [I 2022-10-08 19:39:40] ...trap/engines/engine.py.229: process: 0.76337 | load: 0.00303 [I 2022-10-08 19:39:40] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-08 19:39:40] ...optimizers/trijoint.py.151: total_norm: 1.521122 [I 2022-10-08 19:39:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 970/2304 [I 2022-10-08 19:39:47] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:17:19 [I 2022-10-08 19:39:47] ...trap/engines/engine.py.229: process: 0.78168 | load: 0.00302 [I 2022-10-08 19:39:47] ...trap/engines/engine.py.230: loss: 0.15326 [I 2022-10-08 19:39:47] ...optimizers/trijoint.py.151: total_norm: 2.750493 [I 2022-10-08 19:39:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 980/2304 [I 2022-10-08 19:39:55] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:17:07 [I 2022-10-08 19:39:55] ...trap/engines/engine.py.229: process: 0.76853 | load: 0.00307 [I 2022-10-08 19:39:55] ...trap/engines/engine.py.230: loss: 0.19968 [I 2022-10-08 19:39:55] ...optimizers/trijoint.py.151: total_norm: 3.239187 [I 2022-10-08 19:40:03] ...trap/engines/engine.py.224: train: epoch 17 | batch 990/2304 [I 2022-10-08 19:40:03] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:17:02 [I 2022-10-08 19:40:03] ...trap/engines/engine.py.229: process: 0.77482 | load: 0.00301 [I 2022-10-08 19:40:03] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-08 19:40:03] ...optimizers/trijoint.py.151: total_norm: 3.191721 [I 2022-10-08 19:40:11] ...trap/engines/engine.py.224: train: epoch 17 | batch 1000/2304 [I 2022-10-08 19:40:11] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:17:00 [I 2022-10-08 19:40:11] ...trap/engines/engine.py.229: process: 0.77434 | load: 0.00304 [I 2022-10-08 19:40:11] ...trap/engines/engine.py.230: loss: 0.17382 [I 2022-10-08 19:40:11] ...optimizers/trijoint.py.151: total_norm: 2.865529 [I 2022-10-08 19:40:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 1010/2304 [I 2022-10-08 19:40:19] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:16:45 [I 2022-10-08 19:40:19] ...trap/engines/engine.py.229: process: 0.76585 | load: 0.00305 [I 2022-10-08 19:40:19] ...trap/engines/engine.py.230: loss: 0.16863 [I 2022-10-08 19:40:19] ...optimizers/trijoint.py.151: total_norm: 2.694649 [I 2022-10-08 19:40:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 1020/2304 [I 2022-10-08 19:40:26] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:16:40 [I 2022-10-08 19:40:26] ...trap/engines/engine.py.229: process: 0.77358 | load: 0.00309 [I 2022-10-08 19:40:26] ...trap/engines/engine.py.230: loss: 0.18181 [I 2022-10-08 19:40:26] ...optimizers/trijoint.py.151: total_norm: 2.230387 [I 2022-10-08 19:40:34] ...trap/engines/engine.py.224: train: epoch 17 | batch 1030/2304 [I 2022-10-08 19:40:34] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:16:33 [I 2022-10-08 19:40:34] ...trap/engines/engine.py.229: process: 0.78632 | load: 0.00306 [I 2022-10-08 19:40:34] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-08 19:40:34] ...optimizers/trijoint.py.151: total_norm: 3.165367 [I 2022-10-08 19:40:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 1040/2304 [I 2022-10-08 19:40:42] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:17 [I 2022-10-08 19:40:42] ...trap/engines/engine.py.229: process: 0.76274 | load: 0.00300 [I 2022-10-08 19:40:42] ...trap/engines/engine.py.230: loss: 0.18331 [I 2022-10-08 19:40:42] ...optimizers/trijoint.py.151: total_norm: 2.578070 [I 2022-10-08 19:40:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 1050/2304 [I 2022-10-08 19:40:50] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:16:15 [I 2022-10-08 19:40:50] ...trap/engines/engine.py.229: process: 0.77787 | load: 0.00304 [I 2022-10-08 19:40:50] ...trap/engines/engine.py.230: loss: 0.35843 [I 2022-10-08 19:40:50] ...optimizers/trijoint.py.151: total_norm: 4.269409 [I 2022-10-08 19:40:58] ...trap/engines/engine.py.224: train: epoch 17 | batch 1060/2304 [I 2022-10-08 19:40:58] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:16:06 [I 2022-10-08 19:40:58] ...trap/engines/engine.py.229: process: 0.77334 | load: 0.00304 [I 2022-10-08 19:40:58] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-08 19:40:58] ...optimizers/trijoint.py.151: total_norm: 2.906117 [I 2022-10-08 19:41:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 1070/2304 [I 2022-10-08 19:41:05] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:16:04 [I 2022-10-08 19:41:05] ...trap/engines/engine.py.229: process: 0.78912 | load: 0.00303 [I 2022-10-08 19:41:05] ...trap/engines/engine.py.230: loss: 0.24507 [I 2022-10-08 19:41:05] ...optimizers/trijoint.py.151: total_norm: 2.989315 [I 2022-10-08 19:41:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1080/2304 [I 2022-10-08 19:41:13] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:15:55 [I 2022-10-08 19:41:13] ...trap/engines/engine.py.229: process: 0.78577 | load: 0.00302 [I 2022-10-08 19:41:13] ...trap/engines/engine.py.230: loss: 0.17095 [I 2022-10-08 19:41:13] ...optimizers/trijoint.py.151: total_norm: 3.019613 [I 2022-10-08 19:41:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 1090/2304 [I 2022-10-08 19:41:21] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:43 [I 2022-10-08 19:41:21] ...trap/engines/engine.py.229: process: 0.78636 | load: 0.00305 [I 2022-10-08 19:41:21] ...trap/engines/engine.py.230: loss: 0.18937 [I 2022-10-08 19:41:21] ...optimizers/trijoint.py.151: total_norm: 3.250716 [I 2022-10-08 19:41:29] ...trap/engines/engine.py.224: train: epoch 17 | batch 1100/2304 [I 2022-10-08 19:41:29] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:15:33 [I 2022-10-08 19:41:29] ...trap/engines/engine.py.229: process: 0.76085 | load: 0.00302 [I 2022-10-08 19:41:29] ...trap/engines/engine.py.230: loss: 0.20918 [I 2022-10-08 19:41:29] ...optimizers/trijoint.py.151: total_norm: 2.397275 [I 2022-10-08 19:41:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 1110/2304 [I 2022-10-08 19:41:37] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:15:32 [I 2022-10-08 19:41:37] ...trap/engines/engine.py.229: process: 0.78642 | load: 0.00302 [I 2022-10-08 19:41:37] ...trap/engines/engine.py.230: loss: 0.15452 [I 2022-10-08 19:41:37] ...optimizers/trijoint.py.151: total_norm: 3.157053 [I 2022-10-08 19:41:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1120/2304 [I 2022-10-08 19:41:44] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:15:20 [I 2022-10-08 19:41:44] ...trap/engines/engine.py.229: process: 0.78506 | load: 0.00306 [I 2022-10-08 19:41:44] ...trap/engines/engine.py.230: loss: 0.18812 [I 2022-10-08 19:41:44] ...optimizers/trijoint.py.151: total_norm: 3.141861 [I 2022-10-08 19:41:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1130/2304 [I 2022-10-08 19:41:52] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:15:13 [I 2022-10-08 19:41:52] ...trap/engines/engine.py.229: process: 0.77940 | load: 0.00303 [I 2022-10-08 19:41:52] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-08 19:41:52] ...optimizers/trijoint.py.151: total_norm: 2.300401 [I 2022-10-08 19:42:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 1140/2304 [I 2022-10-08 19:42:00] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:15:03 [I 2022-10-08 19:42:00] ...trap/engines/engine.py.229: process: 0.77290 | load: 0.00303 [I 2022-10-08 19:42:00] ...trap/engines/engine.py.230: loss: 0.13270 [I 2022-10-08 19:42:00] ...optimizers/trijoint.py.151: total_norm: 2.897500 [I 2022-10-08 19:42:08] ...trap/engines/engine.py.224: train: epoch 17 | batch 1150/2304 [I 2022-10-08 19:42:08] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:14:59 [I 2022-10-08 19:42:08] ...trap/engines/engine.py.229: process: 0.77475 | load: 0.00306 [I 2022-10-08 19:42:08] ...trap/engines/engine.py.230: loss: 0.26151 [I 2022-10-08 19:42:08] ...optimizers/trijoint.py.151: total_norm: 3.402840 [I 2022-10-08 19:42:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 1160/2304 [I 2022-10-08 19:42:16] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:14:50 [I 2022-10-08 19:42:16] ...trap/engines/engine.py.229: process: 0.77992 | load: 0.00299 [I 2022-10-08 19:42:16] ...trap/engines/engine.py.230: loss: 0.24805 [I 2022-10-08 19:42:16] ...optimizers/trijoint.py.151: total_norm: 3.972931 [I 2022-10-08 19:42:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1170/2304 [I 2022-10-08 19:42:23] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:14:43 [I 2022-10-08 19:42:23] ...trap/engines/engine.py.229: process: 0.78630 | load: 0.00302 [I 2022-10-08 19:42:23] ...trap/engines/engine.py.230: loss: 0.22865 [I 2022-10-08 19:42:23] ...optimizers/trijoint.py.151: total_norm: 3.006438 [I 2022-10-08 19:42:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 1180/2304 [I 2022-10-08 19:42:31] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:14:36 [I 2022-10-08 19:42:31] ...trap/engines/engine.py.229: process: 0.78347 | load: 0.00305 [I 2022-10-08 19:42:31] ...trap/engines/engine.py.230: loss: 0.18348 [I 2022-10-08 19:42:31] ...optimizers/trijoint.py.151: total_norm: 2.394301 [I 2022-10-08 19:42:39] ...trap/engines/engine.py.224: train: epoch 17 | batch 1190/2304 [I 2022-10-08 19:42:39] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:14:30 [I 2022-10-08 19:42:39] ...trap/engines/engine.py.229: process: 0.78376 | load: 0.00303 [I 2022-10-08 19:42:39] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-08 19:42:39] ...optimizers/trijoint.py.151: total_norm: 2.713002 [I 2022-10-08 19:42:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 1200/2304 [I 2022-10-08 19:42:47] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:18 [I 2022-10-08 19:42:47] ...trap/engines/engine.py.229: process: 0.76715 | load: 0.00303 [I 2022-10-08 19:42:47] ...trap/engines/engine.py.230: loss: 0.21809 [I 2022-10-08 19:42:47] ...optimizers/trijoint.py.151: total_norm: 2.902684 [I 2022-10-08 19:42:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 1210/2304 [I 2022-10-08 19:42:55] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:14:12 [I 2022-10-08 19:42:55] ...trap/engines/engine.py.229: process: 0.76711 | load: 0.00301 [I 2022-10-08 19:42:55] ...trap/engines/engine.py.230: loss: 0.16858 [I 2022-10-08 19:42:55] ...optimizers/trijoint.py.151: total_norm: 2.691606 [I 2022-10-08 19:43:03] ...trap/engines/engine.py.224: train: epoch 17 | batch 1220/2304 [I 2022-10-08 19:43:03] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:14:04 [I 2022-10-08 19:43:03] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00302 [I 2022-10-08 19:43:03] ...trap/engines/engine.py.230: loss: 0.22173 [I 2022-10-08 19:43:03] ...optimizers/trijoint.py.151: total_norm: 2.759644 [I 2022-10-08 19:43:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 1230/2304 [I 2022-10-08 19:43:10] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:13:57 [I 2022-10-08 19:43:10] ...trap/engines/engine.py.229: process: 0.77881 | load: 0.00305 [I 2022-10-08 19:43:10] ...trap/engines/engine.py.230: loss: 0.16620 [I 2022-10-08 19:43:10] ...optimizers/trijoint.py.151: total_norm: 2.884906 [I 2022-10-08 19:43:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1240/2304 [I 2022-10-08 19:43:18] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:13:43 [I 2022-10-08 19:43:18] ...trap/engines/engine.py.229: process: 0.77762 | load: 0.00306 [I 2022-10-08 19:43:18] ...trap/engines/engine.py.230: loss: 0.17314 [I 2022-10-08 19:43:18] ...optimizers/trijoint.py.151: total_norm: 2.978510 [I 2022-10-08 19:43:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 1250/2304 [I 2022-10-08 19:43:26] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:13:41 [I 2022-10-08 19:43:26] ...trap/engines/engine.py.229: process: 0.78288 | load: 0.00304 [I 2022-10-08 19:43:26] ...trap/engines/engine.py.230: loss: 0.15078 [I 2022-10-08 19:43:26] ...optimizers/trijoint.py.151: total_norm: 2.382491 [I 2022-10-08 19:43:34] ...trap/engines/engine.py.224: train: epoch 17 | batch 1260/2304 [I 2022-10-08 19:43:34] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:13:34 [I 2022-10-08 19:43:34] ...trap/engines/engine.py.229: process: 0.78386 | load: 0.00304 [I 2022-10-08 19:43:34] ...trap/engines/engine.py.230: loss: 0.24178 [I 2022-10-08 19:43:34] ...optimizers/trijoint.py.151: total_norm: 3.333048 [I 2022-10-08 19:43:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 1270/2304 [I 2022-10-08 19:43:42] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:13:33 [I 2022-10-08 19:43:42] ...trap/engines/engine.py.229: process: 0.80198 | load: 0.00304 [I 2022-10-08 19:43:42] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-08 19:43:42] ...optimizers/trijoint.py.151: total_norm: 3.068614 [I 2022-10-08 19:43:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1280/2304 [I 2022-10-08 19:43:49] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:13:21 [I 2022-10-08 19:43:49] ...trap/engines/engine.py.229: process: 0.78278 | load: 0.00305 [I 2022-10-08 19:43:49] ...trap/engines/engine.py.230: loss: 0.10693 [I 2022-10-08 19:43:49] ...optimizers/trijoint.py.151: total_norm: 1.812034 [I 2022-10-08 19:43:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1290/2304 [I 2022-10-08 19:43:57] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:13:05 [I 2022-10-08 19:43:57] ...trap/engines/engine.py.229: process: 0.77478 | load: 0.00304 [I 2022-10-08 19:43:57] ...trap/engines/engine.py.230: loss: 0.13787 [I 2022-10-08 19:43:57] ...optimizers/trijoint.py.151: total_norm: 2.188018 [I 2022-10-08 19:44:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 1300/2304 [I 2022-10-08 19:44:05] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:13:09 [I 2022-10-08 19:44:05] ...trap/engines/engine.py.229: process: 0.77794 | load: 0.00304 [I 2022-10-08 19:44:05] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-08 19:44:05] ...optimizers/trijoint.py.151: total_norm: 2.687696 [I 2022-10-08 19:44:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1310/2304 [I 2022-10-08 19:44:13] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:54 [I 2022-10-08 19:44:13] ...trap/engines/engine.py.229: process: 0.77778 | load: 0.00303 [I 2022-10-08 19:44:13] ...trap/engines/engine.py.230: loss: 0.25464 [I 2022-10-08 19:44:13] ...optimizers/trijoint.py.151: total_norm: 3.463532 [I 2022-10-08 19:44:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 1320/2304 [I 2022-10-08 19:44:21] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:12:43 [I 2022-10-08 19:44:21] ...trap/engines/engine.py.229: process: 0.77392 | load: 0.00306 [I 2022-10-08 19:44:21] ...trap/engines/engine.py.230: loss: 0.18080 [I 2022-10-08 19:44:21] ...optimizers/trijoint.py.151: total_norm: 2.847158 [I 2022-10-08 19:44:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1330/2304 [I 2022-10-08 19:44:28] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:12:34 [I 2022-10-08 19:44:28] ...trap/engines/engine.py.229: process: 0.76792 | load: 0.00299 [I 2022-10-08 19:44:28] ...trap/engines/engine.py.230: loss: 0.18607 [I 2022-10-08 19:44:28] ...optimizers/trijoint.py.151: total_norm: 2.851628 [I 2022-10-08 19:44:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 1340/2304 [I 2022-10-08 19:44:36] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:12:31 [I 2022-10-08 19:44:36] ...trap/engines/engine.py.229: process: 0.78000 | load: 0.00299 [I 2022-10-08 19:44:36] ...trap/engines/engine.py.230: loss: 0.27694 [I 2022-10-08 19:44:36] ...optimizers/trijoint.py.151: total_norm: 3.449245 [I 2022-10-08 19:44:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1350/2304 [I 2022-10-08 19:44:44] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:12:20 [I 2022-10-08 19:44:44] ...trap/engines/engine.py.229: process: 0.77710 | load: 0.00302 [I 2022-10-08 19:44:44] ...trap/engines/engine.py.230: loss: 0.19434 [I 2022-10-08 19:44:44] ...optimizers/trijoint.py.151: total_norm: 2.414277 [I 2022-10-08 19:44:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1360/2304 [I 2022-10-08 19:44:52] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:12:14 [I 2022-10-08 19:44:52] ...trap/engines/engine.py.229: process: 0.77530 | load: 0.00304 [I 2022-10-08 19:44:52] ...trap/engines/engine.py.230: loss: 0.19199 [I 2022-10-08 19:44:52] ...optimizers/trijoint.py.151: total_norm: 2.766465 [I 2022-10-08 19:45:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 1370/2304 [I 2022-10-08 19:45:00] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:12:07 [I 2022-10-08 19:45:00] ...trap/engines/engine.py.229: process: 0.77307 | load: 0.00301 [I 2022-10-08 19:45:00] ...trap/engines/engine.py.230: loss: 0.15602 [I 2022-10-08 19:45:00] ...optimizers/trijoint.py.151: total_norm: 2.399705 [I 2022-10-08 19:45:08] ...trap/engines/engine.py.224: train: epoch 17 | batch 1380/2304 [I 2022-10-08 19:45:08] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:12:02 [I 2022-10-08 19:45:08] ...trap/engines/engine.py.229: process: 0.77386 | load: 0.00301 [I 2022-10-08 19:45:08] ...trap/engines/engine.py.230: loss: 0.16739 [I 2022-10-08 19:45:08] ...optimizers/trijoint.py.151: total_norm: 2.112634 [I 2022-10-08 19:45:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 1390/2304 [I 2022-10-08 19:45:15] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:11:49 [I 2022-10-08 19:45:15] ...trap/engines/engine.py.229: process: 0.78007 | load: 0.00303 [I 2022-10-08 19:45:15] ...trap/engines/engine.py.230: loss: 0.21141 [I 2022-10-08 19:45:15] ...optimizers/trijoint.py.151: total_norm: 3.761486 [I 2022-10-08 19:45:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1400/2304 [I 2022-10-08 19:45:23] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:11:44 [I 2022-10-08 19:45:23] ...trap/engines/engine.py.229: process: 0.77109 | load: 0.00304 [I 2022-10-08 19:45:23] ...trap/engines/engine.py.230: loss: 0.22055 [I 2022-10-08 19:45:23] ...optimizers/trijoint.py.151: total_norm: 3.366803 [I 2022-10-08 19:45:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 1410/2304 [I 2022-10-08 19:45:31] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:11:37 [I 2022-10-08 19:45:31] ...trap/engines/engine.py.229: process: 0.77838 | load: 0.00302 [I 2022-10-08 19:45:31] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-08 19:45:31] ...optimizers/trijoint.py.151: total_norm: 1.978655 [I 2022-10-08 19:45:39] ...trap/engines/engine.py.224: train: epoch 17 | batch 1420/2304 [I 2022-10-08 19:45:39] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:11:26 [I 2022-10-08 19:45:39] ...trap/engines/engine.py.229: process: 0.76462 | load: 0.00303 [I 2022-10-08 19:45:39] ...trap/engines/engine.py.230: loss: 0.21014 [I 2022-10-08 19:45:39] ...optimizers/trijoint.py.151: total_norm: 2.582578 [I 2022-10-08 19:45:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 1430/2304 [I 2022-10-08 19:45:47] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:11:21 [I 2022-10-08 19:45:47] ...trap/engines/engine.py.229: process: 0.78424 | load: 0.00301 [I 2022-10-08 19:45:47] ...trap/engines/engine.py.230: loss: 0.18809 [I 2022-10-08 19:45:47] ...optimizers/trijoint.py.151: total_norm: 2.559432 [I 2022-10-08 19:45:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1440/2304 [I 2022-10-08 19:45:54] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:11:15 [I 2022-10-08 19:45:54] ...trap/engines/engine.py.229: process: 0.79061 | load: 0.00310 [I 2022-10-08 19:45:54] ...trap/engines/engine.py.230: loss: 0.21934 [I 2022-10-08 19:45:54] ...optimizers/trijoint.py.151: total_norm: 3.216585 [I 2022-10-08 19:46:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1450/2304 [I 2022-10-08 19:46:02] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:11:07 [I 2022-10-08 19:46:02] ...trap/engines/engine.py.229: process: 0.79027 | load: 0.00311 [I 2022-10-08 19:46:02] ...trap/engines/engine.py.230: loss: 0.23532 [I 2022-10-08 19:46:02] ...optimizers/trijoint.py.151: total_norm: 3.755275 [I 2022-10-08 19:46:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 1460/2304 [I 2022-10-08 19:46:10] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:11:00 [I 2022-10-08 19:46:10] ...trap/engines/engine.py.229: process: 0.78292 | load: 0.00311 [I 2022-10-08 19:46:10] ...trap/engines/engine.py.230: loss: 0.17356 [I 2022-10-08 19:46:10] ...optimizers/trijoint.py.151: total_norm: 2.121336 [I 2022-10-08 19:46:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1470/2304 [I 2022-10-08 19:46:18] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:10:49 [I 2022-10-08 19:46:18] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00310 [I 2022-10-08 19:46:18] ...trap/engines/engine.py.230: loss: 0.16986 [I 2022-10-08 19:46:18] ...optimizers/trijoint.py.151: total_norm: 2.802876 [I 2022-10-08 19:46:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 1480/2304 [I 2022-10-08 19:46:26] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:10:41 [I 2022-10-08 19:46:26] ...trap/engines/engine.py.229: process: 0.77425 | load: 0.00301 [I 2022-10-08 19:46:26] ...trap/engines/engine.py.230: loss: 0.23410 [I 2022-10-08 19:46:26] ...optimizers/trijoint.py.151: total_norm: 2.796968 [I 2022-10-08 19:46:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1490/2304 [I 2022-10-08 19:46:33] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:10:36 [I 2022-10-08 19:46:33] ...trap/engines/engine.py.229: process: 0.78810 | load: 0.00304 [I 2022-10-08 19:46:33] ...trap/engines/engine.py.230: loss: 0.21650 [I 2022-10-08 19:46:33] ...optimizers/trijoint.py.151: total_norm: 2.892011 [I 2022-10-08 19:46:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 1500/2304 [I 2022-10-08 19:46:41] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:10:26 [I 2022-10-08 19:46:41] ...trap/engines/engine.py.229: process: 0.77542 | load: 0.00301 [I 2022-10-08 19:46:41] ...trap/engines/engine.py.230: loss: 0.14743 [I 2022-10-08 19:46:41] ...optimizers/trijoint.py.151: total_norm: 2.049623 [I 2022-10-08 19:46:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1510/2304 [I 2022-10-08 19:46:49] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:10:18 [I 2022-10-08 19:46:49] ...trap/engines/engine.py.229: process: 0.77292 | load: 0.00303 [I 2022-10-08 19:46:49] ...trap/engines/engine.py.230: loss: 0.20945 [I 2022-10-08 19:46:49] ...optimizers/trijoint.py.151: total_norm: 1.652212 [I 2022-10-08 19:46:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1520/2304 [I 2022-10-08 19:46:57] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:10:14 [I 2022-10-08 19:46:57] ...trap/engines/engine.py.229: process: 0.78878 | load: 0.00302 [I 2022-10-08 19:46:57] ...trap/engines/engine.py.230: loss: 0.14231 [I 2022-10-08 19:46:57] ...optimizers/trijoint.py.151: total_norm: 2.543551 [I 2022-10-08 19:47:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 1530/2304 [I 2022-10-08 19:47:05] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:10:04 [I 2022-10-08 19:47:05] ...trap/engines/engine.py.229: process: 0.77822 | load: 0.00301 [I 2022-10-08 19:47:05] ...trap/engines/engine.py.230: loss: 0.25886 [I 2022-10-08 19:47:05] ...optimizers/trijoint.py.151: total_norm: 3.274285 [I 2022-10-08 19:47:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1540/2304 [I 2022-10-08 19:47:13] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:09:55 [I 2022-10-08 19:47:13] ...trap/engines/engine.py.229: process: 0.76802 | load: 0.00304 [I 2022-10-08 19:47:13] ...trap/engines/engine.py.230: loss: 0.23908 [I 2022-10-08 19:47:13] ...optimizers/trijoint.py.151: total_norm: 2.498003 [I 2022-10-08 19:47:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 1550/2304 [I 2022-10-08 19:47:20] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:09:47 [I 2022-10-08 19:47:20] ...trap/engines/engine.py.229: process: 0.77229 | load: 0.00300 [I 2022-10-08 19:47:20] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-08 19:47:20] ...optimizers/trijoint.py.151: total_norm: 2.970367 [I 2022-10-08 19:47:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1560/2304 [I 2022-10-08 19:47:28] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:09:40 [I 2022-10-08 19:47:28] ...trap/engines/engine.py.229: process: 0.78346 | load: 0.00301 [I 2022-10-08 19:47:28] ...trap/engines/engine.py.230: loss: 0.14387 [I 2022-10-08 19:47:28] ...optimizers/trijoint.py.151: total_norm: 2.296683 [I 2022-10-08 19:47:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 1570/2304 [I 2022-10-08 19:47:36] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:30 [I 2022-10-08 19:47:36] ...trap/engines/engine.py.229: process: 0.77468 | load: 0.00306 [I 2022-10-08 19:47:36] ...trap/engines/engine.py.230: loss: 0.15905 [I 2022-10-08 19:47:36] ...optimizers/trijoint.py.151: total_norm: 2.780373 [I 2022-10-08 19:47:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1580/2304 [I 2022-10-08 19:47:44] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:22 [I 2022-10-08 19:47:44] ...trap/engines/engine.py.229: process: 0.77516 | load: 0.00299 [I 2022-10-08 19:47:44] ...trap/engines/engine.py.230: loss: 0.16562 [I 2022-10-08 19:47:44] ...optimizers/trijoint.py.151: total_norm: 2.977257 [I 2022-10-08 19:47:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1590/2304 [I 2022-10-08 19:47:52] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:09:16 [I 2022-10-08 19:47:52] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00303 [I 2022-10-08 19:47:52] ...trap/engines/engine.py.230: loss: 0.15260 [I 2022-10-08 19:47:52] ...optimizers/trijoint.py.151: total_norm: 2.635663 [I 2022-10-08 19:47:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 1600/2304 [I 2022-10-08 19:47:59] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:09:07 [I 2022-10-08 19:47:59] ...trap/engines/engine.py.229: process: 0.77827 | load: 0.00302 [I 2022-10-08 19:47:59] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-08 19:47:59] ...optimizers/trijoint.py.151: total_norm: 2.853174 [I 2022-10-08 19:48:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 1610/2304 [I 2022-10-08 19:48:07] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:08:58 [I 2022-10-08 19:48:07] ...trap/engines/engine.py.229: process: 0.77141 | load: 0.00302 [I 2022-10-08 19:48:07] ...trap/engines/engine.py.230: loss: 0.20717 [I 2022-10-08 19:48:07] ...optimizers/trijoint.py.151: total_norm: 3.129530 [I 2022-10-08 19:48:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 1620/2304 [I 2022-10-08 19:48:15] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:54 [I 2022-10-08 19:48:15] ...trap/engines/engine.py.229: process: 0.79264 | load: 0.00298 [I 2022-10-08 19:48:15] ...trap/engines/engine.py.230: loss: 0.17253 [I 2022-10-08 19:48:15] ...optimizers/trijoint.py.151: total_norm: 1.379600 [I 2022-10-08 19:48:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1630/2304 [I 2022-10-08 19:48:23] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:08:44 [I 2022-10-08 19:48:23] ...trap/engines/engine.py.229: process: 0.77115 | load: 0.00302 [I 2022-10-08 19:48:23] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-08 19:48:23] ...optimizers/trijoint.py.151: total_norm: 2.491589 [I 2022-10-08 19:48:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 1640/2304 [I 2022-10-08 19:48:31] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:08:39 [I 2022-10-08 19:48:31] ...trap/engines/engine.py.229: process: 0.78634 | load: 0.00304 [I 2022-10-08 19:48:31] ...trap/engines/engine.py.230: loss: 0.22505 [I 2022-10-08 19:48:31] ...optimizers/trijoint.py.151: total_norm: 2.936344 [I 2022-10-08 19:48:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 1650/2304 [I 2022-10-08 19:48:38] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:08:28 [I 2022-10-08 19:48:38] ...trap/engines/engine.py.229: process: 0.78305 | load: 0.00302 [I 2022-10-08 19:48:38] ...trap/engines/engine.py.230: loss: 0.19991 [I 2022-10-08 19:48:38] ...optimizers/trijoint.py.151: total_norm: 2.892844 [I 2022-10-08 19:48:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 1660/2304 [I 2022-10-08 19:48:46] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:08:23 [I 2022-10-08 19:48:46] ...trap/engines/engine.py.229: process: 0.79116 | load: 0.00304 [I 2022-10-08 19:48:46] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-08 19:48:46] ...optimizers/trijoint.py.151: total_norm: 2.492656 [I 2022-10-08 19:48:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1670/2304 [I 2022-10-08 19:48:54] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:08:15 [I 2022-10-08 19:48:54] ...trap/engines/engine.py.229: process: 0.78625 | load: 0.00299 [I 2022-10-08 19:48:54] ...trap/engines/engine.py.230: loss: 0.28960 [I 2022-10-08 19:48:54] ...optimizers/trijoint.py.151: total_norm: 2.693670 [I 2022-10-08 19:49:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1680/2304 [I 2022-10-08 19:49:02] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:07 [I 2022-10-08 19:49:02] ...trap/engines/engine.py.229: process: 0.78864 | load: 0.00304 [I 2022-10-08 19:49:02] ...trap/engines/engine.py.230: loss: 0.22397 [I 2022-10-08 19:49:02] ...optimizers/trijoint.py.151: total_norm: 2.852552 [I 2022-10-08 19:49:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 1690/2304 [I 2022-10-08 19:49:10] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:07:59 [I 2022-10-08 19:49:10] ...trap/engines/engine.py.229: process: 0.77176 | load: 0.00300 [I 2022-10-08 19:49:10] ...trap/engines/engine.py.230: loss: 0.19605 [I 2022-10-08 19:49:10] ...optimizers/trijoint.py.151: total_norm: 2.980587 [I 2022-10-08 19:49:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1700/2304 [I 2022-10-08 19:49:18] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:07:51 [I 2022-10-08 19:49:18] ...trap/engines/engine.py.229: process: 0.78304 | load: 0.00304 [I 2022-10-08 19:49:18] ...trap/engines/engine.py.230: loss: 0.22655 [I 2022-10-08 19:49:18] ...optimizers/trijoint.py.151: total_norm: 3.190791 [I 2022-10-08 19:49:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 1710/2304 [I 2022-10-08 19:49:25] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:07:46 [I 2022-10-08 19:49:25] ...trap/engines/engine.py.229: process: 0.78400 | load: 0.00296 [I 2022-10-08 19:49:25] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-08 19:49:25] ...optimizers/trijoint.py.151: total_norm: 2.499179 [I 2022-10-08 19:49:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1720/2304 [I 2022-10-08 19:49:33] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:07:34 [I 2022-10-08 19:49:33] ...trap/engines/engine.py.229: process: 0.77313 | load: 0.00307 [I 2022-10-08 19:49:33] ...trap/engines/engine.py.230: loss: 0.21061 [I 2022-10-08 19:49:33] ...optimizers/trijoint.py.151: total_norm: 3.142160 [I 2022-10-08 19:49:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 1730/2304 [I 2022-10-08 19:49:41] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:29 [I 2022-10-08 19:49:41] ...trap/engines/engine.py.229: process: 0.78093 | load: 0.00300 [I 2022-10-08 19:49:41] ...trap/engines/engine.py.230: loss: 0.22728 [I 2022-10-08 19:49:41] ...optimizers/trijoint.py.151: total_norm: 3.473337 [I 2022-10-08 19:49:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1740/2304 [I 2022-10-08 19:49:49] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:17 [I 2022-10-08 19:49:49] ...trap/engines/engine.py.229: process: 0.77597 | load: 0.00299 [I 2022-10-08 19:49:49] ...trap/engines/engine.py.230: loss: 0.19071 [I 2022-10-08 19:49:49] ...optimizers/trijoint.py.151: total_norm: 2.618084 [I 2022-10-08 19:49:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1750/2304 [I 2022-10-08 19:49:57] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:07:12 [I 2022-10-08 19:49:57] ...trap/engines/engine.py.229: process: 0.78990 | load: 0.00311 [I 2022-10-08 19:49:57] ...trap/engines/engine.py.230: loss: 0.23361 [I 2022-10-08 19:49:57] ...optimizers/trijoint.py.151: total_norm: 3.657170 [I 2022-10-08 19:50:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 1760/2304 [I 2022-10-08 19:50:05] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:07:04 [I 2022-10-08 19:50:05] ...trap/engines/engine.py.229: process: 0.78941 | load: 0.00303 [I 2022-10-08 19:50:05] ...trap/engines/engine.py.230: loss: 0.24215 [I 2022-10-08 19:50:05] ...optimizers/trijoint.py.151: total_norm: 2.812462 [I 2022-10-08 19:50:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 1770/2304 [I 2022-10-08 19:50:12] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:56 [I 2022-10-08 19:50:12] ...trap/engines/engine.py.229: process: 0.77917 | load: 0.00303 [I 2022-10-08 19:50:12] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-08 19:50:12] ...optimizers/trijoint.py.151: total_norm: 1.471002 [I 2022-10-08 19:50:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 1780/2304 [I 2022-10-08 19:50:20] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:48 [I 2022-10-08 19:50:20] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00304 [I 2022-10-08 19:50:20] ...trap/engines/engine.py.230: loss: 0.14774 [I 2022-10-08 19:50:20] ...optimizers/trijoint.py.151: total_norm: 3.207740 [I 2022-10-08 19:50:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1790/2304 [I 2022-10-08 19:50:28] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:39 [I 2022-10-08 19:50:28] ...trap/engines/engine.py.229: process: 0.78089 | load: 0.00299 [I 2022-10-08 19:50:28] ...trap/engines/engine.py.230: loss: 0.21601 [I 2022-10-08 19:50:28] ...optimizers/trijoint.py.151: total_norm: 3.214567 [I 2022-10-08 19:50:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 1800/2304 [I 2022-10-08 19:50:36] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:06:31 [I 2022-10-08 19:50:36] ...trap/engines/engine.py.229: process: 0.78718 | load: 0.00308 [I 2022-10-08 19:50:36] ...trap/engines/engine.py.230: loss: 0.24321 [I 2022-10-08 19:50:36] ...optimizers/trijoint.py.151: total_norm: 3.925615 [I 2022-10-08 19:50:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1810/2304 [I 2022-10-08 19:50:44] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:06:26 [I 2022-10-08 19:50:44] ...trap/engines/engine.py.229: process: 0.78700 | load: 0.00303 [I 2022-10-08 19:50:44] ...trap/engines/engine.py.230: loss: 0.30469 [I 2022-10-08 19:50:44] ...optimizers/trijoint.py.151: total_norm: 3.841887 [I 2022-10-08 19:50:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1820/2304 [I 2022-10-08 19:50:52] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:06:18 [I 2022-10-08 19:50:52] ...trap/engines/engine.py.229: process: 0.77656 | load: 0.00307 [I 2022-10-08 19:50:52] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-08 19:50:52] ...optimizers/trijoint.py.151: total_norm: 3.020827 [I 2022-10-08 19:50:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 1830/2304 [I 2022-10-08 19:50:59] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:08 [I 2022-10-08 19:50:59] ...trap/engines/engine.py.229: process: 0.77686 | load: 0.00306 [I 2022-10-08 19:50:59] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-08 19:50:59] ...optimizers/trijoint.py.151: total_norm: 2.484600 [I 2022-10-08 19:51:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 1840/2304 [I 2022-10-08 19:51:07] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:03 [I 2022-10-08 19:51:07] ...trap/engines/engine.py.229: process: 0.78754 | load: 0.00304 [I 2022-10-08 19:51:07] ...trap/engines/engine.py.230: loss: 0.18548 [I 2022-10-08 19:51:07] ...optimizers/trijoint.py.151: total_norm: 3.870105 [I 2022-10-08 19:51:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 1850/2304 [I 2022-10-08 19:51:15] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:05:51 [I 2022-10-08 19:51:15] ...trap/engines/engine.py.229: process: 0.77350 | load: 0.00302 [I 2022-10-08 19:51:15] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-08 19:51:15] ...optimizers/trijoint.py.151: total_norm: 1.555430 [I 2022-10-08 19:51:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1860/2304 [I 2022-10-08 19:51:23] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:05:45 [I 2022-10-08 19:51:23] ...trap/engines/engine.py.229: process: 0.77743 | load: 0.00302 [I 2022-10-08 19:51:23] ...trap/engines/engine.py.230: loss: 0.24161 [I 2022-10-08 19:51:23] ...optimizers/trijoint.py.151: total_norm: 3.405019 [I 2022-10-08 19:51:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 1870/2304 [I 2022-10-08 19:51:31] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:38 [I 2022-10-08 19:51:31] ...trap/engines/engine.py.229: process: 0.78074 | load: 0.00301 [I 2022-10-08 19:51:31] ...trap/engines/engine.py.230: loss: 0.13194 [I 2022-10-08 19:51:31] ...optimizers/trijoint.py.151: total_norm: 2.147209 [I 2022-10-08 19:51:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 1880/2304 [I 2022-10-08 19:51:38] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:05:29 [I 2022-10-08 19:51:38] ...trap/engines/engine.py.229: process: 0.78041 | load: 0.00306 [I 2022-10-08 19:51:38] ...trap/engines/engine.py.230: loss: 0.14496 [I 2022-10-08 19:51:38] ...optimizers/trijoint.py.151: total_norm: 2.680037 [I 2022-10-08 19:51:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 1890/2304 [I 2022-10-08 19:51:46] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:05:21 [I 2022-10-08 19:51:46] ...trap/engines/engine.py.229: process: 0.78576 | load: 0.00304 [I 2022-10-08 19:51:46] ...trap/engines/engine.py.230: loss: 0.16456 [I 2022-10-08 19:51:46] ...optimizers/trijoint.py.151: total_norm: 2.559686 [I 2022-10-08 19:51:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1900/2304 [I 2022-10-08 19:51:54] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:14 [I 2022-10-08 19:51:54] ...trap/engines/engine.py.229: process: 0.78115 | load: 0.00303 [I 2022-10-08 19:51:54] ...trap/engines/engine.py.230: loss: 0.22090 [I 2022-10-08 19:51:54] ...optimizers/trijoint.py.151: total_norm: 3.243735 [I 2022-10-08 19:52:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1910/2304 [I 2022-10-08 19:52:02] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:05:08 [I 2022-10-08 19:52:02] ...trap/engines/engine.py.229: process: 0.78687 | load: 0.00302 [I 2022-10-08 19:52:02] ...trap/engines/engine.py.230: loss: 0.20794 [I 2022-10-08 19:52:02] ...optimizers/trijoint.py.151: total_norm: 2.110545 [I 2022-10-08 19:52:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 1920/2304 [I 2022-10-08 19:52:10] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:04:57 [I 2022-10-08 19:52:10] ...trap/engines/engine.py.229: process: 0.77413 | load: 0.00305 [I 2022-10-08 19:52:10] ...trap/engines/engine.py.230: loss: 0.26693 [I 2022-10-08 19:52:10] ...optimizers/trijoint.py.151: total_norm: 3.483999 [I 2022-10-08 19:52:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 1930/2304 [I 2022-10-08 19:52:17] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:04:52 [I 2022-10-08 19:52:17] ...trap/engines/engine.py.229: process: 0.77862 | load: 0.00299 [I 2022-10-08 19:52:17] ...trap/engines/engine.py.230: loss: 0.26330 [I 2022-10-08 19:52:17] ...optimizers/trijoint.py.151: total_norm: 2.905367 [I 2022-10-08 19:52:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 1940/2304 [I 2022-10-08 19:52:25] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:04:44 [I 2022-10-08 19:52:25] ...trap/engines/engine.py.229: process: 0.77006 | load: 0.00306 [I 2022-10-08 19:52:25] ...trap/engines/engine.py.230: loss: 0.19471 [I 2022-10-08 19:52:25] ...optimizers/trijoint.py.151: total_norm: 2.821007 [I 2022-10-08 19:52:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1950/2304 [I 2022-10-08 19:52:33] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:34 [I 2022-10-08 19:52:33] ...trap/engines/engine.py.229: process: 0.76381 | load: 0.00302 [I 2022-10-08 19:52:33] ...trap/engines/engine.py.230: loss: 0.17543 [I 2022-10-08 19:52:33] ...optimizers/trijoint.py.151: total_norm: 2.687749 [I 2022-10-08 19:52:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 1960/2304 [I 2022-10-08 19:52:41] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:04:28 [I 2022-10-08 19:52:41] ...trap/engines/engine.py.229: process: 0.78429 | load: 0.00306 [I 2022-10-08 19:52:41] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-08 19:52:41] ...optimizers/trijoint.py.151: total_norm: 2.973218 [I 2022-10-08 19:52:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1970/2304 [I 2022-10-08 19:52:49] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:04:23 [I 2022-10-08 19:52:49] ...trap/engines/engine.py.229: process: 0.77923 | load: 0.00306 [I 2022-10-08 19:52:49] ...trap/engines/engine.py.230: loss: 0.20062 [I 2022-10-08 19:52:49] ...optimizers/trijoint.py.151: total_norm: 2.561996 [I 2022-10-08 19:52:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1980/2304 [I 2022-10-08 19:52:57] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:04:14 [I 2022-10-08 19:52:57] ...trap/engines/engine.py.229: process: 0.78318 | load: 0.00304 [I 2022-10-08 19:52:57] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-08 19:52:57] ...optimizers/trijoint.py.151: total_norm: 2.156257 [I 2022-10-08 19:53:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 1990/2304 [I 2022-10-08 19:53:04] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:07 [I 2022-10-08 19:53:04] ...trap/engines/engine.py.229: process: 0.78494 | load: 0.00303 [I 2022-10-08 19:53:04] ...trap/engines/engine.py.230: loss: 0.25456 [I 2022-10-08 19:53:04] ...optimizers/trijoint.py.151: total_norm: 2.709180 [I 2022-10-08 19:53:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 2000/2304 [I 2022-10-08 19:53:12] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:03:57 [I 2022-10-08 19:53:12] ...trap/engines/engine.py.229: process: 0.77451 | load: 0.00304 [I 2022-10-08 19:53:12] ...trap/engines/engine.py.230: loss: 0.21138 [I 2022-10-08 19:53:12] ...optimizers/trijoint.py.151: total_norm: 2.919120 [I 2022-10-08 19:53:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 2010/2304 [I 2022-10-08 19:53:20] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:03:50 [I 2022-10-08 19:53:20] ...trap/engines/engine.py.229: process: 0.78720 | load: 0.00305 [I 2022-10-08 19:53:20] ...trap/engines/engine.py.230: loss: 0.25651 [I 2022-10-08 19:53:20] ...optimizers/trijoint.py.151: total_norm: 3.576397 [I 2022-10-08 19:53:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 2020/2304 [I 2022-10-08 19:53:28] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:03:41 [I 2022-10-08 19:53:28] ...trap/engines/engine.py.229: process: 0.78199 | load: 0.00303 [I 2022-10-08 19:53:28] ...trap/engines/engine.py.230: loss: 0.19762 [I 2022-10-08 19:53:28] ...optimizers/trijoint.py.151: total_norm: 2.011042 [I 2022-10-08 19:53:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 2030/2304 [I 2022-10-08 19:53:36] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:32 [I 2022-10-08 19:53:36] ...trap/engines/engine.py.229: process: 0.76556 | load: 0.00301 [I 2022-10-08 19:53:36] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-08 19:53:36] ...optimizers/trijoint.py.151: total_norm: 2.452518 [I 2022-10-08 19:53:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 2040/2304 [I 2022-10-08 19:53:44] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:03:25 [I 2022-10-08 19:53:44] ...trap/engines/engine.py.229: process: 0.76441 | load: 0.00303 [I 2022-10-08 19:53:44] ...trap/engines/engine.py.230: loss: 0.23091 [I 2022-10-08 19:53:44] ...optimizers/trijoint.py.151: total_norm: 3.017908 [I 2022-10-08 19:53:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 2050/2304 [I 2022-10-08 19:53:52] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:03:17 [I 2022-10-08 19:53:52] ...trap/engines/engine.py.229: process: 0.78066 | load: 0.00302 [I 2022-10-08 19:53:52] ...trap/engines/engine.py.230: loss: 0.16620 [I 2022-10-08 19:53:52] ...optimizers/trijoint.py.151: total_norm: 3.176567 [I 2022-10-08 19:53:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 2060/2304 [I 2022-10-08 19:53:59] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:03:10 [I 2022-10-08 19:53:59] ...trap/engines/engine.py.229: process: 0.78664 | load: 0.00302 [I 2022-10-08 19:53:59] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-08 19:53:59] ...optimizers/trijoint.py.151: total_norm: 1.941943 [I 2022-10-08 19:54:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 2070/2304 [I 2022-10-08 19:54:07] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:03:03 [I 2022-10-08 19:54:07] ...trap/engines/engine.py.229: process: 0.78595 | load: 0.00307 [I 2022-10-08 19:54:07] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-08 19:54:07] ...optimizers/trijoint.py.151: total_norm: 2.159378 [I 2022-10-08 19:54:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 2080/2304 [I 2022-10-08 19:54:15] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:02:54 [I 2022-10-08 19:54:15] ...trap/engines/engine.py.229: process: 0.78412 | load: 0.00303 [I 2022-10-08 19:54:15] ...trap/engines/engine.py.230: loss: 0.17273 [I 2022-10-08 19:54:15] ...optimizers/trijoint.py.151: total_norm: 2.864328 [I 2022-10-08 19:54:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 2090/2304 [I 2022-10-08 19:54:23] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:02:46 [I 2022-10-08 19:54:23] ...trap/engines/engine.py.229: process: 0.78598 | load: 0.00305 [I 2022-10-08 19:54:23] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-08 19:54:23] ...optimizers/trijoint.py.151: total_norm: 2.938253 [I 2022-10-08 19:54:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 2100/2304 [I 2022-10-08 19:54:31] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:39 [I 2022-10-08 19:54:31] ...trap/engines/engine.py.229: process: 0.77929 | load: 0.00303 [I 2022-10-08 19:54:31] ...trap/engines/engine.py.230: loss: 0.16837 [I 2022-10-08 19:54:31] ...optimizers/trijoint.py.151: total_norm: 3.022618 [I 2022-10-08 19:54:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 2110/2304 [I 2022-10-08 19:54:38] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:30 [I 2022-10-08 19:54:38] ...trap/engines/engine.py.229: process: 0.78421 | load: 0.00303 [I 2022-10-08 19:54:38] ...trap/engines/engine.py.230: loss: 0.17754 [I 2022-10-08 19:54:38] ...optimizers/trijoint.py.151: total_norm: 2.679663 [I 2022-10-08 19:54:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 2120/2304 [I 2022-10-08 19:54:46] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:02:23 [I 2022-10-08 19:54:46] ...trap/engines/engine.py.229: process: 0.77966 | load: 0.00302 [I 2022-10-08 19:54:46] ...trap/engines/engine.py.230: loss: 0.18923 [I 2022-10-08 19:54:46] ...optimizers/trijoint.py.151: total_norm: 2.484414 [I 2022-10-08 19:54:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 2130/2304 [I 2022-10-08 19:54:54] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:02:16 [I 2022-10-08 19:54:54] ...trap/engines/engine.py.229: process: 0.78958 | load: 0.00303 [I 2022-10-08 19:54:54] ...trap/engines/engine.py.230: loss: 0.17844 [I 2022-10-08 19:54:54] ...optimizers/trijoint.py.151: total_norm: 2.234387 [I 2022-10-08 19:55:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 2140/2304 [I 2022-10-08 19:55:02] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:08 [I 2022-10-08 19:55:02] ...trap/engines/engine.py.229: process: 0.79741 | load: 0.00301 [I 2022-10-08 19:55:02] ...trap/engines/engine.py.230: loss: 0.18648 [I 2022-10-08 19:55:02] ...optimizers/trijoint.py.151: total_norm: 2.986486 [I 2022-10-08 19:55:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 2150/2304 [I 2022-10-08 19:55:10] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:01:59 [I 2022-10-08 19:55:10] ...trap/engines/engine.py.229: process: 0.77568 | load: 0.00303 [I 2022-10-08 19:55:10] ...trap/engines/engine.py.230: loss: 0.24018 [I 2022-10-08 19:55:10] ...optimizers/trijoint.py.151: total_norm: 3.438249 [I 2022-10-08 19:55:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 2160/2304 [I 2022-10-08 19:55:17] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:01:50 [I 2022-10-08 19:55:17] ...trap/engines/engine.py.229: process: 0.76143 | load: 0.00300 [I 2022-10-08 19:55:17] ...trap/engines/engine.py.230: loss: 0.27934 [I 2022-10-08 19:55:17] ...optimizers/trijoint.py.151: total_norm: 3.547169 [I 2022-10-08 19:55:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 2170/2304 [I 2022-10-08 19:55:25] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:01:43 [I 2022-10-08 19:55:25] ...trap/engines/engine.py.229: process: 0.77676 | load: 0.00299 [I 2022-10-08 19:55:25] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-08 19:55:25] ...optimizers/trijoint.py.151: total_norm: 2.072760 [I 2022-10-08 19:55:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 2180/2304 [I 2022-10-08 19:55:33] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:01:36 [I 2022-10-08 19:55:33] ...trap/engines/engine.py.229: process: 0.77434 | load: 0.00303 [I 2022-10-08 19:55:33] ...trap/engines/engine.py.230: loss: 0.21624 [I 2022-10-08 19:55:33] ...optimizers/trijoint.py.151: total_norm: 3.029765 [I 2022-10-08 19:55:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 2190/2304 [I 2022-10-08 19:55:41] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:01:28 [I 2022-10-08 19:55:41] ...trap/engines/engine.py.229: process: 0.78631 | load: 0.00300 [I 2022-10-08 19:55:41] ...trap/engines/engine.py.230: loss: 0.21288 [I 2022-10-08 19:55:41] ...optimizers/trijoint.py.151: total_norm: 2.447302 [I 2022-10-08 19:55:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 2200/2304 [I 2022-10-08 19:55:49] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:01:20 [I 2022-10-08 19:55:49] ...trap/engines/engine.py.229: process: 0.77235 | load: 0.00303 [I 2022-10-08 19:55:49] ...trap/engines/engine.py.230: loss: 0.18813 [I 2022-10-08 19:55:49] ...optimizers/trijoint.py.151: total_norm: 3.201687 [I 2022-10-08 19:55:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 2210/2304 [I 2022-10-08 19:55:56] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:01:13 [I 2022-10-08 19:55:56] ...trap/engines/engine.py.229: process: 0.78330 | load: 0.00300 [I 2022-10-08 19:55:56] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-08 19:55:56] ...optimizers/trijoint.py.151: total_norm: 2.552447 [I 2022-10-08 19:56:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 2220/2304 [I 2022-10-08 19:56:04] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:01:05 [I 2022-10-08 19:56:04] ...trap/engines/engine.py.229: process: 0.78242 | load: 0.00304 [I 2022-10-08 19:56:04] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-08 19:56:04] ...optimizers/trijoint.py.151: total_norm: 2.313617 [I 2022-10-08 19:56:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 2230/2304 [I 2022-10-08 19:56:12] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:00:57 [I 2022-10-08 19:56:12] ...trap/engines/engine.py.229: process: 0.76923 | load: 0.00302 [I 2022-10-08 19:56:12] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-08 19:56:12] ...optimizers/trijoint.py.151: total_norm: 1.821651 [I 2022-10-08 19:56:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 2240/2304 [I 2022-10-08 19:56:20] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:00:50 [I 2022-10-08 19:56:20] ...trap/engines/engine.py.229: process: 0.78488 | load: 0.00297 [I 2022-10-08 19:56:20] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-08 19:56:20] ...optimizers/trijoint.py.151: total_norm: 1.691927 [I 2022-10-08 19:56:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 2250/2304 [I 2022-10-08 19:56:28] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:00:42 [I 2022-10-08 19:56:28] ...trap/engines/engine.py.229: process: 0.77146 | load: 0.00309 [I 2022-10-08 19:56:28] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-08 19:56:28] ...optimizers/trijoint.py.151: total_norm: 2.928696 [I 2022-10-08 19:56:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 2260/2304 [I 2022-10-08 19:56:36] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:00:34 [I 2022-10-08 19:56:36] ...trap/engines/engine.py.229: process: 0.77636 | load: 0.00304 [I 2022-10-08 19:56:36] ...trap/engines/engine.py.230: loss: 0.17220 [I 2022-10-08 19:56:36] ...optimizers/trijoint.py.151: total_norm: 3.204607 [I 2022-10-08 19:56:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 2270/2304 [I 2022-10-08 19:56:43] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:00:26 [I 2022-10-08 19:56:43] ...trap/engines/engine.py.229: process: 0.78117 | load: 0.00305 [I 2022-10-08 19:56:43] ...trap/engines/engine.py.230: loss: 0.18782 [I 2022-10-08 19:56:43] ...optimizers/trijoint.py.151: total_norm: 2.360628 [I 2022-10-08 19:56:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 2280/2304 [I 2022-10-08 19:56:51] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:00:18 [I 2022-10-08 19:56:51] ...trap/engines/engine.py.229: process: 0.78594 | load: 0.00302 [I 2022-10-08 19:56:51] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-08 19:56:51] ...optimizers/trijoint.py.151: total_norm: 2.144730 [I 2022-10-08 19:56:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 2290/2304 [I 2022-10-08 19:56:59] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:10 [I 2022-10-08 19:56:59] ...trap/engines/engine.py.229: process: 0.78693 | load: 0.00302 [I 2022-10-08 19:56:59] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-08 19:56:59] ...optimizers/trijoint.py.151: total_norm: 3.082123 [I 2022-10-08 19:57:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 2300/2304 [I 2022-10-08 19:57:07] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:00:03 [I 2022-10-08 19:57:07] ...trap/engines/engine.py.229: process: 0.77048 | load: 0.00016 [I 2022-10-08 19:57:07] ...trap/engines/engine.py.230: loss: 0.20206 [I 2022-10-08 19:57:07] ...optimizers/trijoint.py.151: total_norm: 3.038780 [I 2022-10-08 19:57:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 2304/2304 [I 2022-10-08 19:57:10] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:00:00 [I 2022-10-08 19:57:10] ...trap/engines/engine.py.229: process: 0.77934 | load: 0.00017 [I 2022-10-08 19:57:10] ...trap/engines/engine.py.230: loss: 0.20722 [I 2022-10-08 19:57:10] ...optimizers/trijoint.py.151: total_norm: 2.350955 [S 2022-10-08 19:57:11] ...trap/engines/engine.py.129: train_epoch.epoch: 17 [S 2022-10-08 19:57:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.1837449040271972 [I 2022-10-08 19:57:11] ...s/criterions/triplet.py.84: margin = 0.14000000000000007 [I 2022-10-08 19:57:11] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 17 [I 2022-10-08 19:57:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 19:57:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 19:57:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 19:57:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 19:57:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 19:57:14] ...trap/engines/engine.py.321: eval: epoch 17 | batch 0/510 [I 2022-10-08 19:57:14] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:34 [I 2022-10-08 19:57:14] ...trap/engines/engine.py.326: process: 2.77356 | load: 2.23686 [I 2022-10-08 19:57:21] ...trap/engines/engine.py.321: eval: epoch 17 | batch 10/510 [I 2022-10-08 19:57:21] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:55 [I 2022-10-08 19:57:21] ...trap/engines/engine.py.326: process: 0.50507 | load: 0.00018 [I 2022-10-08 19:57:26] ...trap/engines/engine.py.321: eval: epoch 17 | batch 20/510 [I 2022-10-08 19:57:26] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:04:40 [I 2022-10-08 19:57:26] ...trap/engines/engine.py.326: process: 0.44069 | load: 0.00019 [I 2022-10-08 19:57:32] ...trap/engines/engine.py.321: eval: epoch 17 | batch 30/510 [I 2022-10-08 19:57:32] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:26 [I 2022-10-08 19:57:32] ...trap/engines/engine.py.326: process: 0.45256 | load: 0.00019 [I 2022-10-08 19:57:38] ...trap/engines/engine.py.321: eval: epoch 17 | batch 40/510 [I 2022-10-08 19:57:38] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:18 [I 2022-10-08 19:57:38] ...trap/engines/engine.py.326: process: 0.55340 | load: 0.00019 [I 2022-10-08 19:57:42] ...trap/engines/engine.py.321: eval: epoch 17 | batch 50/510 [I 2022-10-08 19:57:42] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:36 [I 2022-10-08 19:57:42] ...trap/engines/engine.py.326: process: 0.42389 | load: 0.00018 [I 2022-10-08 19:57:48] ...trap/engines/engine.py.321: eval: epoch 17 | batch 60/510 [I 2022-10-08 19:57:48] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:24 [I 2022-10-08 19:57:48] ...trap/engines/engine.py.326: process: 0.71662 | load: 0.00017 [I 2022-10-08 19:57:54] ...trap/engines/engine.py.321: eval: epoch 17 | batch 70/510 [I 2022-10-08 19:57:54] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:11 [I 2022-10-08 19:57:54] ...trap/engines/engine.py.326: process: 0.69087 | load: 0.00019 [I 2022-10-08 19:58:00] ...trap/engines/engine.py.321: eval: epoch 17 | batch 80/510 [I 2022-10-08 19:58:00] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:48 [I 2022-10-08 19:58:00] ...trap/engines/engine.py.326: process: 0.52699 | load: 0.00018 [I 2022-10-08 19:58:04] ...trap/engines/engine.py.321: eval: epoch 17 | batch 90/510 [I 2022-10-08 19:58:04] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:24 [I 2022-10-08 19:58:04] ...trap/engines/engine.py.326: process: 0.49390 | load: 0.00018 [I 2022-10-08 19:58:09] ...trap/engines/engine.py.321: eval: epoch 17 | batch 100/510 [I 2022-10-08 19:58:09] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:57 [I 2022-10-08 19:58:09] ...trap/engines/engine.py.326: process: 0.31183 | load: 0.00020 [I 2022-10-08 19:58:12] ...trap/engines/engine.py.321: eval: epoch 17 | batch 110/510 [I 2022-10-08 19:58:12] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:11 [I 2022-10-08 19:58:12] ...trap/engines/engine.py.326: process: 0.32094 | load: 0.00019 [I 2022-10-08 19:58:15] ...trap/engines/engine.py.321: eval: epoch 17 | batch 120/510 [I 2022-10-08 19:58:15] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-08 19:58:15] ...trap/engines/engine.py.326: process: 0.31898 | load: 0.00019 [I 2022-10-08 19:58:18] ...trap/engines/engine.py.321: eval: epoch 17 | batch 130/510 [I 2022-10-08 19:58:18] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:00 [I 2022-10-08 19:58:18] ...trap/engines/engine.py.326: process: 0.31796 | load: 0.00020 [I 2022-10-08 19:58:22] ...trap/engines/engine.py.321: eval: epoch 17 | batch 140/510 [I 2022-10-08 19:58:22] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:56 [I 2022-10-08 19:58:22] ...trap/engines/engine.py.326: process: 0.30932 | load: 0.00019 [I 2022-10-08 19:58:25] ...trap/engines/engine.py.321: eval: epoch 17 | batch 150/510 [I 2022-10-08 19:58:25] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:54 [I 2022-10-08 19:58:25] ...trap/engines/engine.py.326: process: 0.31507 | load: 0.00020 [I 2022-10-08 19:58:28] ...trap/engines/engine.py.321: eval: epoch 17 | batch 160/510 [I 2022-10-08 19:58:28] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:50 [I 2022-10-08 19:58:28] ...trap/engines/engine.py.326: process: 0.31655 | load: 0.00019 [I 2022-10-08 19:58:31] ...trap/engines/engine.py.321: eval: epoch 17 | batch 170/510 [I 2022-10-08 19:58:31] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:47 [I 2022-10-08 19:58:31] ...trap/engines/engine.py.326: process: 0.31088 | load: 0.00020 [I 2022-10-08 19:58:34] ...trap/engines/engine.py.321: eval: epoch 17 | batch 180/510 [I 2022-10-08 19:58:34] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:43 [I 2022-10-08 19:58:34] ...trap/engines/engine.py.326: process: 0.31031 | load: 0.00020 [I 2022-10-08 19:58:38] ...trap/engines/engine.py.321: eval: epoch 17 | batch 190/510 [I 2022-10-08 19:58:38] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:42 [I 2022-10-08 19:58:38] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00019 [I 2022-10-08 19:58:41] ...trap/engines/engine.py.321: eval: epoch 17 | batch 200/510 [I 2022-10-08 19:58:41] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:38 [I 2022-10-08 19:58:41] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00020 [I 2022-10-08 19:58:44] ...trap/engines/engine.py.321: eval: epoch 17 | batch 210/510 [I 2022-10-08 19:58:44] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:35 [I 2022-10-08 19:58:44] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00020 [I 2022-10-08 19:58:47] ...trap/engines/engine.py.321: eval: epoch 17 | batch 220/510 [I 2022-10-08 19:58:47] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:32 [I 2022-10-08 19:58:47] ...trap/engines/engine.py.326: process: 0.32525 | load: 0.00020 [I 2022-10-08 19:58:50] ...trap/engines/engine.py.321: eval: epoch 17 | batch 230/510 [I 2022-10-08 19:58:50] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-08 19:58:50] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00020 [I 2022-10-08 19:58:53] ...trap/engines/engine.py.321: eval: epoch 17 | batch 240/510 [I 2022-10-08 19:58:53] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:25 [I 2022-10-08 19:58:53] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00020 [I 2022-10-08 19:58:57] ...trap/engines/engine.py.321: eval: epoch 17 | batch 250/510 [I 2022-10-08 19:58:57] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:22 [I 2022-10-08 19:58:57] ...trap/engines/engine.py.326: process: 0.31570 | load: 0.00019 [I 2022-10-08 19:59:00] ...trap/engines/engine.py.321: eval: epoch 17 | batch 260/510 [I 2022-10-08 19:59:00] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:19 [I 2022-10-08 19:59:00] ...trap/engines/engine.py.326: process: 0.31451 | load: 0.00020 [I 2022-10-08 19:59:03] ...trap/engines/engine.py.321: eval: epoch 17 | batch 270/510 [I 2022-10-08 19:59:03] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-08 19:59:03] ...trap/engines/engine.py.326: process: 0.32154 | load: 0.00020 [I 2022-10-08 19:59:06] ...trap/engines/engine.py.321: eval: epoch 17 | batch 280/510 [I 2022-10-08 19:59:06] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:12 [I 2022-10-08 19:59:06] ...trap/engines/engine.py.326: process: 0.31268 | load: 0.00019 [I 2022-10-08 19:59:09] ...trap/engines/engine.py.321: eval: epoch 17 | batch 290/510 [I 2022-10-08 19:59:09] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-08 19:59:09] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-08 19:59:13] ...trap/engines/engine.py.321: eval: epoch 17 | batch 300/510 [I 2022-10-08 19:59:13] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:06 [I 2022-10-08 19:59:13] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00020 [I 2022-10-08 19:59:16] ...trap/engines/engine.py.321: eval: epoch 17 | batch 310/510 [I 2022-10-08 19:59:16] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-08 19:59:16] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00020 [I 2022-10-08 19:59:19] ...trap/engines/engine.py.321: eval: epoch 17 | batch 320/510 [I 2022-10-08 19:59:19] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-08 19:59:19] ...trap/engines/engine.py.326: process: 0.31077 | load: 0.00019 [I 2022-10-08 19:59:22] ...trap/engines/engine.py.321: eval: epoch 17 | batch 330/510 [I 2022-10-08 19:59:22] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-08 19:59:22] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00020 [I 2022-10-08 19:59:25] ...trap/engines/engine.py.321: eval: epoch 17 | batch 340/510 [I 2022-10-08 19:59:25] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-08 19:59:25] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00021 [I 2022-10-08 19:59:29] ...trap/engines/engine.py.321: eval: epoch 17 | batch 350/510 [I 2022-10-08 19:59:29] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-08 19:59:29] ...trap/engines/engine.py.326: process: 0.32059 | load: 0.00020 [I 2022-10-08 19:59:32] ...trap/engines/engine.py.321: eval: epoch 17 | batch 360/510 [I 2022-10-08 19:59:32] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:47 [I 2022-10-08 19:59:32] ...trap/engines/engine.py.326: process: 0.31778 | load: 0.00020 [I 2022-10-08 19:59:35] ...trap/engines/engine.py.321: eval: epoch 17 | batch 370/510 [I 2022-10-08 19:59:35] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-08 19:59:35] ...trap/engines/engine.py.326: process: 0.31532 | load: 0.00021 [I 2022-10-08 19:59:38] ...trap/engines/engine.py.321: eval: epoch 17 | batch 380/510 [I 2022-10-08 19:59:38] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-08 19:59:38] ...trap/engines/engine.py.326: process: 0.32217 | load: 0.00020 [I 2022-10-08 19:59:41] ...trap/engines/engine.py.321: eval: epoch 17 | batch 390/510 [I 2022-10-08 19:59:41] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-08 19:59:41] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-08 19:59:44] ...trap/engines/engine.py.321: eval: epoch 17 | batch 400/510 [I 2022-10-08 19:59:44] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-08 19:59:44] ...trap/engines/engine.py.326: process: 0.31904 | load: 0.00020 [I 2022-10-08 19:59:48] ...trap/engines/engine.py.321: eval: epoch 17 | batch 410/510 [I 2022-10-08 19:59:48] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:32 [I 2022-10-08 19:59:48] ...trap/engines/engine.py.326: process: 0.32251 | load: 0.00020 [I 2022-10-08 19:59:51] ...trap/engines/engine.py.321: eval: epoch 17 | batch 420/510 [I 2022-10-08 19:59:51] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:28 [I 2022-10-08 19:59:51] ...trap/engines/engine.py.326: process: 0.31668 | load: 0.00019 [I 2022-10-08 19:59:54] ...trap/engines/engine.py.321: eval: epoch 17 | batch 430/510 [I 2022-10-08 19:59:54] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-08 19:59:54] ...trap/engines/engine.py.326: process: 0.32444 | load: 0.00020 [I 2022-10-08 19:59:57] ...trap/engines/engine.py.321: eval: epoch 17 | batch 440/510 [I 2022-10-08 19:59:57] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-08 19:59:57] ...trap/engines/engine.py.326: process: 0.31613 | load: 0.00020 [I 2022-10-08 20:00:00] ...trap/engines/engine.py.321: eval: epoch 17 | batch 450/510 [I 2022-10-08 20:00:00] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-08 20:00:00] ...trap/engines/engine.py.326: process: 0.32893 | load: 0.00020 [I 2022-10-08 20:00:04] ...trap/engines/engine.py.321: eval: epoch 17 | batch 460/510 [I 2022-10-08 20:00:04] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:15 [I 2022-10-08 20:00:04] ...trap/engines/engine.py.326: process: 0.31457 | load: 0.00020 [I 2022-10-08 20:00:07] ...trap/engines/engine.py.321: eval: epoch 17 | batch 470/510 [I 2022-10-08 20:00:07] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-08 20:00:07] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00019 [I 2022-10-08 20:00:10] ...trap/engines/engine.py.321: eval: epoch 17 | batch 480/510 [I 2022-10-08 20:00:10] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-08 20:00:10] ...trap/engines/engine.py.326: process: 0.32009 | load: 0.00020 [I 2022-10-08 20:00:13] ...trap/engines/engine.py.321: eval: epoch 17 | batch 490/510 [I 2022-10-08 20:00:13] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-08 20:00:13] ...trap/engines/engine.py.326: process: 0.32751 | load: 0.00019 [I 2022-10-08 20:00:16] ...trap/engines/engine.py.321: eval: epoch 17 | batch 500/510 [I 2022-10-08 20:00:16] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-08 20:00:16] ...trap/engines/engine.py.326: process: 0.31665 | load: 0.00021 [I 2022-10-08 20:00:20] ...trap/engines/engine.py.321: eval: epoch 17 | batch 510/510 [I 2022-10-08 20:00:20] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-08 20:00:20] ...trap/engines/engine.py.326: process: 0.31525 | load: 0.00015 [S 2022-10-08 20:00:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 17 [I 2022-10-08 20:00:20] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.606200 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.859700 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.910600 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.606800 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.862300 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.911200 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017954 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011950 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010716 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012360 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007550 [S 2022-10-08 20:00:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008256 [I 2022-10-08 20:00:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 20:00:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 20:00:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 20:00:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 20:00:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 20:00:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 20:00:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 20:01:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 20:01:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 18 [I 2022-10-08 20:01:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 20:01:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 0/2304 [I 2022-10-08 20:01:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:55 [I 2022-10-08 20:01:05] ...trap/engines/engine.py.229: process: 2.88847 | load: 2.03492 [I 2022-10-08 20:01:05] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-08 20:01:05] ...optimizers/trijoint.py.151: total_norm: 1.652874 [I 2022-10-08 20:01:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 10/2304 [I 2022-10-08 20:01:13] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:29 [I 2022-10-08 20:01:13] ...trap/engines/engine.py.229: process: 0.77855 | load: 0.00517 [I 2022-10-08 20:01:13] ...trap/engines/engine.py.230: loss: 0.20261 [I 2022-10-08 20:01:13] ...optimizers/trijoint.py.151: total_norm: 3.216607 [I 2022-10-08 20:01:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 20/2304 [I 2022-10-08 20:01:21] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:51 [I 2022-10-08 20:01:21] ...trap/engines/engine.py.229: process: 0.79146 | load: 0.00482 [I 2022-10-08 20:01:21] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-08 20:01:21] ...optimizers/trijoint.py.151: total_norm: 2.751873 [I 2022-10-08 20:01:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 30/2304 [I 2022-10-08 20:01:29] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:41 [I 2022-10-08 20:01:29] ...trap/engines/engine.py.229: process: 0.77981 | load: 0.00420 [I 2022-10-08 20:01:29] ...trap/engines/engine.py.230: loss: 0.14728 [I 2022-10-08 20:01:29] ...optimizers/trijoint.py.151: total_norm: 2.408678 [I 2022-10-08 20:01:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 40/2304 [I 2022-10-08 20:01:36] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:21 [I 2022-10-08 20:01:36] ...trap/engines/engine.py.229: process: 0.77179 | load: 0.00423 [I 2022-10-08 20:01:36] ...trap/engines/engine.py.230: loss: 0.21839 [I 2022-10-08 20:01:36] ...optimizers/trijoint.py.151: total_norm: 3.503449 [I 2022-10-08 20:01:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 50/2304 [I 2022-10-08 20:01:44] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:10 [I 2022-10-08 20:01:44] ...trap/engines/engine.py.229: process: 0.76820 | load: 0.00406 [I 2022-10-08 20:01:44] ...trap/engines/engine.py.230: loss: 0.21364 [I 2022-10-08 20:01:44] ...optimizers/trijoint.py.151: total_norm: 3.383429 [I 2022-10-08 20:01:52] ...trap/engines/engine.py.224: train: epoch 18 | batch 60/2304 [I 2022-10-08 20:01:52] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:28:57 [I 2022-10-08 20:01:52] ...trap/engines/engine.py.229: process: 0.76742 | load: 0.00387 [I 2022-10-08 20:01:52] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-08 20:01:52] ...optimizers/trijoint.py.151: total_norm: 2.607638 [I 2022-10-08 20:02:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 70/2304 [I 2022-10-08 20:02:00] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:58 [I 2022-10-08 20:02:00] ...trap/engines/engine.py.229: process: 0.76712 | load: 0.00339 [I 2022-10-08 20:02:00] ...trap/engines/engine.py.230: loss: 0.24670 [I 2022-10-08 20:02:00] ...optimizers/trijoint.py.151: total_norm: 2.895971 [I 2022-10-08 20:02:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 80/2304 [I 2022-10-08 20:02:08] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:50 [I 2022-10-08 20:02:08] ...trap/engines/engine.py.229: process: 0.78468 | load: 0.00379 [I 2022-10-08 20:02:08] ...trap/engines/engine.py.230: loss: 0.20028 [I 2022-10-08 20:02:08] ...optimizers/trijoint.py.151: total_norm: 3.063385 [I 2022-10-08 20:02:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 90/2304 [I 2022-10-08 20:02:15] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:35 [I 2022-10-08 20:02:15] ...trap/engines/engine.py.229: process: 0.77107 | load: 0.00337 [I 2022-10-08 20:02:15] ...trap/engines/engine.py.230: loss: 0.22502 [I 2022-10-08 20:02:15] ...optimizers/trijoint.py.151: total_norm: 2.820958 [I 2022-10-08 20:02:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 100/2304 [I 2022-10-08 20:02:23] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:28:34 [I 2022-10-08 20:02:23] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00321 [I 2022-10-08 20:02:23] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-08 20:02:23] ...optimizers/trijoint.py.151: total_norm: 3.033537 [I 2022-10-08 20:02:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 110/2304 [I 2022-10-08 20:02:31] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:28:35 [I 2022-10-08 20:02:31] ...trap/engines/engine.py.229: process: 0.78903 | load: 0.00339 [I 2022-10-08 20:02:31] ...trap/engines/engine.py.230: loss: 0.16286 [I 2022-10-08 20:02:31] ...optimizers/trijoint.py.151: total_norm: 3.575879 [I 2022-10-08 20:02:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 120/2304 [I 2022-10-08 20:02:39] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:12 [I 2022-10-08 20:02:39] ...trap/engines/engine.py.229: process: 0.76975 | load: 0.00344 [I 2022-10-08 20:02:39] ...trap/engines/engine.py.230: loss: 0.14293 [I 2022-10-08 20:02:39] ...optimizers/trijoint.py.151: total_norm: 2.225717 [I 2022-10-08 20:02:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 130/2304 [I 2022-10-08 20:02:46] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:05 [I 2022-10-08 20:02:46] ...trap/engines/engine.py.229: process: 0.77842 | load: 0.00300 [I 2022-10-08 20:02:46] ...trap/engines/engine.py.230: loss: 0.19928 [I 2022-10-08 20:02:46] ...optimizers/trijoint.py.151: total_norm: 4.254899 [I 2022-10-08 20:02:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 140/2304 [I 2022-10-08 20:02:54] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:10 [I 2022-10-08 20:02:54] ...trap/engines/engine.py.229: process: 0.78725 | load: 0.00336 [I 2022-10-08 20:02:54] ...trap/engines/engine.py.230: loss: 0.21564 [I 2022-10-08 20:02:54] ...optimizers/trijoint.py.151: total_norm: 2.884033 [I 2022-10-08 20:03:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 150/2304 [I 2022-10-08 20:03:02] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:28:05 [I 2022-10-08 20:03:02] ...trap/engines/engine.py.229: process: 0.79164 | load: 0.00344 [I 2022-10-08 20:03:02] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-08 20:03:02] ...optimizers/trijoint.py.151: total_norm: 2.811388 [I 2022-10-08 20:03:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 160/2304 [I 2022-10-08 20:03:10] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:27:52 [I 2022-10-08 20:03:10] ...trap/engines/engine.py.229: process: 0.77920 | load: 0.00335 [I 2022-10-08 20:03:10] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-08 20:03:10] ...optimizers/trijoint.py.151: total_norm: 2.667827 [I 2022-10-08 20:03:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 170/2304 [I 2022-10-08 20:03:18] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:27:34 [I 2022-10-08 20:03:18] ...trap/engines/engine.py.229: process: 0.77390 | load: 0.00330 [I 2022-10-08 20:03:18] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-08 20:03:18] ...optimizers/trijoint.py.151: total_norm: 1.880099 [I 2022-10-08 20:03:26] ...trap/engines/engine.py.224: train: epoch 18 | batch 180/2304 [I 2022-10-08 20:03:26] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:38 [I 2022-10-08 20:03:26] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00316 [I 2022-10-08 20:03:26] ...trap/engines/engine.py.230: loss: 0.12892 [I 2022-10-08 20:03:26] ...optimizers/trijoint.py.151: total_norm: 2.324319 [I 2022-10-08 20:03:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 190/2304 [I 2022-10-08 20:03:33] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:33 [I 2022-10-08 20:03:33] ...trap/engines/engine.py.229: process: 0.79013 | load: 0.00324 [I 2022-10-08 20:03:33] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-08 20:03:33] ...optimizers/trijoint.py.151: total_norm: 1.867972 [I 2022-10-08 20:03:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 200/2304 [I 2022-10-08 20:03:41] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:21 [I 2022-10-08 20:03:41] ...trap/engines/engine.py.229: process: 0.76982 | load: 0.00313 [I 2022-10-08 20:03:41] ...trap/engines/engine.py.230: loss: 0.28155 [I 2022-10-08 20:03:41] ...optimizers/trijoint.py.151: total_norm: 4.049232 [I 2022-10-08 20:03:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 210/2304 [I 2022-10-08 20:03:49] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:27:07 [I 2022-10-08 20:03:49] ...trap/engines/engine.py.229: process: 0.78213 | load: 0.00333 [I 2022-10-08 20:03:49] ...trap/engines/engine.py.230: loss: 0.22512 [I 2022-10-08 20:03:49] ...optimizers/trijoint.py.151: total_norm: 3.523043 [I 2022-10-08 20:03:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 220/2304 [I 2022-10-08 20:03:57] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:27:02 [I 2022-10-08 20:03:57] ...trap/engines/engine.py.229: process: 0.77085 | load: 0.00314 [I 2022-10-08 20:03:57] ...trap/engines/engine.py.230: loss: 0.22635 [I 2022-10-08 20:03:57] ...optimizers/trijoint.py.151: total_norm: 2.127236 [I 2022-10-08 20:04:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 230/2304 [I 2022-10-08 20:04:05] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:58 [I 2022-10-08 20:04:05] ...trap/engines/engine.py.229: process: 0.77847 | load: 0.00340 [I 2022-10-08 20:04:05] ...trap/engines/engine.py.230: loss: 0.21524 [I 2022-10-08 20:04:05] ...optimizers/trijoint.py.151: total_norm: 3.307722 [I 2022-10-08 20:04:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 240/2304 [I 2022-10-08 20:04:12] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:48 [I 2022-10-08 20:04:12] ...trap/engines/engine.py.229: process: 0.78057 | load: 0.00302 [I 2022-10-08 20:04:12] ...trap/engines/engine.py.230: loss: 0.22329 [I 2022-10-08 20:04:12] ...optimizers/trijoint.py.151: total_norm: 3.496127 [I 2022-10-08 20:04:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 250/2304 [I 2022-10-08 20:04:20] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:41 [I 2022-10-08 20:04:20] ...trap/engines/engine.py.229: process: 0.78729 | load: 0.00301 [I 2022-10-08 20:04:20] ...trap/engines/engine.py.230: loss: 0.18287 [I 2022-10-08 20:04:20] ...optimizers/trijoint.py.151: total_norm: 2.440676 [I 2022-10-08 20:04:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 260/2304 [I 2022-10-08 20:04:28] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:26:34 [I 2022-10-08 20:04:28] ...trap/engines/engine.py.229: process: 0.77933 | load: 0.00301 [I 2022-10-08 20:04:28] ...trap/engines/engine.py.230: loss: 0.21112 [I 2022-10-08 20:04:28] ...optimizers/trijoint.py.151: total_norm: 2.054622 [I 2022-10-08 20:04:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 270/2304 [I 2022-10-08 20:04:36] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:26:29 [I 2022-10-08 20:04:36] ...trap/engines/engine.py.229: process: 0.77905 | load: 0.00309 [I 2022-10-08 20:04:36] ...trap/engines/engine.py.230: loss: 0.16220 [I 2022-10-08 20:04:36] ...optimizers/trijoint.py.151: total_norm: 2.638688 [I 2022-10-08 20:04:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 280/2304 [I 2022-10-08 20:04:44] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:26:20 [I 2022-10-08 20:04:44] ...trap/engines/engine.py.229: process: 0.78050 | load: 0.00315 [I 2022-10-08 20:04:44] ...trap/engines/engine.py.230: loss: 0.20597 [I 2022-10-08 20:04:44] ...optimizers/trijoint.py.151: total_norm: 2.493797 [I 2022-10-08 20:04:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 290/2304 [I 2022-10-08 20:04:51] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:26:08 [I 2022-10-08 20:04:51] ...trap/engines/engine.py.229: process: 0.78098 | load: 0.00301 [I 2022-10-08 20:04:51] ...trap/engines/engine.py.230: loss: 0.15024 [I 2022-10-08 20:04:51] ...optimizers/trijoint.py.151: total_norm: 2.544784 [I 2022-10-08 20:04:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 300/2304 [I 2022-10-08 20:04:59] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:25:46 [I 2022-10-08 20:04:59] ...trap/engines/engine.py.229: process: 0.77125 | load: 0.00302 [I 2022-10-08 20:04:59] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-08 20:04:59] ...optimizers/trijoint.py.151: total_norm: 1.664498 [I 2022-10-08 20:05:07] ...trap/engines/engine.py.224: train: epoch 18 | batch 310/2304 [I 2022-10-08 20:05:07] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:26:02 [I 2022-10-08 20:05:07] ...trap/engines/engine.py.229: process: 0.79140 | load: 0.00304 [I 2022-10-08 20:05:07] ...trap/engines/engine.py.230: loss: 0.16067 [I 2022-10-08 20:05:07] ...optimizers/trijoint.py.151: total_norm: 2.605465 [I 2022-10-08 20:05:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 320/2304 [I 2022-10-08 20:05:15] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:25:36 [I 2022-10-08 20:05:15] ...trap/engines/engine.py.229: process: 0.76613 | load: 0.00300 [I 2022-10-08 20:05:15] ...trap/engines/engine.py.230: loss: 0.22051 [I 2022-10-08 20:05:15] ...optimizers/trijoint.py.151: total_norm: 2.776865 [I 2022-10-08 20:05:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 330/2304 [I 2022-10-08 20:05:23] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:25:29 [I 2022-10-08 20:05:23] ...trap/engines/engine.py.229: process: 0.76785 | load: 0.00301 [I 2022-10-08 20:05:23] ...trap/engines/engine.py.230: loss: 0.13394 [I 2022-10-08 20:05:23] ...optimizers/trijoint.py.151: total_norm: 2.430538 [I 2022-10-08 20:05:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 340/2304 [I 2022-10-08 20:05:30] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:25:32 [I 2022-10-08 20:05:30] ...trap/engines/engine.py.229: process: 0.77405 | load: 0.00315 [I 2022-10-08 20:05:30] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-08 20:05:30] ...optimizers/trijoint.py.151: total_norm: 3.074973 [I 2022-10-08 20:05:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 350/2304 [I 2022-10-08 20:05:38] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:22 [I 2022-10-08 20:05:38] ...trap/engines/engine.py.229: process: 0.78089 | load: 0.00313 [I 2022-10-08 20:05:38] ...trap/engines/engine.py.230: loss: 0.13796 [I 2022-10-08 20:05:38] ...optimizers/trijoint.py.151: total_norm: 2.405691 [I 2022-10-08 20:05:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 360/2304 [I 2022-10-08 20:05:46] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:25:18 [I 2022-10-08 20:05:46] ...trap/engines/engine.py.229: process: 0.77967 | load: 0.00311 [I 2022-10-08 20:05:46] ...trap/engines/engine.py.230: loss: 0.17104 [I 2022-10-08 20:05:46] ...optimizers/trijoint.py.151: total_norm: 2.551897 [I 2022-10-08 20:05:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 370/2304 [I 2022-10-08 20:05:54] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:25:11 [I 2022-10-08 20:05:54] ...trap/engines/engine.py.229: process: 0.78828 | load: 0.00304 [I 2022-10-08 20:05:54] ...trap/engines/engine.py.230: loss: 0.26149 [I 2022-10-08 20:05:54] ...optimizers/trijoint.py.151: total_norm: 3.147175 [I 2022-10-08 20:06:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 380/2304 [I 2022-10-08 20:06:02] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:25:02 [I 2022-10-08 20:06:02] ...trap/engines/engine.py.229: process: 0.78354 | load: 0.00312 [I 2022-10-08 20:06:02] ...trap/engines/engine.py.230: loss: 0.20721 [I 2022-10-08 20:06:02] ...optimizers/trijoint.py.151: total_norm: 3.486523 [I 2022-10-08 20:06:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 390/2304 [I 2022-10-08 20:06:10] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:25:04 [I 2022-10-08 20:06:10] ...trap/engines/engine.py.229: process: 0.79018 | load: 0.00312 [I 2022-10-08 20:06:10] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-08 20:06:10] ...optimizers/trijoint.py.151: total_norm: 1.938407 [I 2022-10-08 20:06:17] ...trap/engines/engine.py.224: train: epoch 18 | batch 400/2304 [I 2022-10-08 20:06:17] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:44 [I 2022-10-08 20:06:17] ...trap/engines/engine.py.229: process: 0.77164 | load: 0.00300 [I 2022-10-08 20:06:17] ...trap/engines/engine.py.230: loss: 0.15484 [I 2022-10-08 20:06:17] ...optimizers/trijoint.py.151: total_norm: 3.278880 [I 2022-10-08 20:06:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 410/2304 [I 2022-10-08 20:06:25] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:54 [I 2022-10-08 20:06:25] ...trap/engines/engine.py.229: process: 0.79093 | load: 0.00311 [I 2022-10-08 20:06:25] ...trap/engines/engine.py.230: loss: 0.12519 [I 2022-10-08 20:06:25] ...optimizers/trijoint.py.151: total_norm: 2.212664 [I 2022-10-08 20:06:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 420/2304 [I 2022-10-08 20:06:33] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:34 [I 2022-10-08 20:06:33] ...trap/engines/engine.py.229: process: 0.76996 | load: 0.00306 [I 2022-10-08 20:06:33] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-08 20:06:33] ...optimizers/trijoint.py.151: total_norm: 2.886658 [I 2022-10-08 20:06:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 430/2304 [I 2022-10-08 20:06:41] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:24:22 [I 2022-10-08 20:06:41] ...trap/engines/engine.py.229: process: 0.79110 | load: 0.00303 [I 2022-10-08 20:06:41] ...trap/engines/engine.py.230: loss: 0.19401 [I 2022-10-08 20:06:41] ...optimizers/trijoint.py.151: total_norm: 2.139226 [I 2022-10-08 20:06:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 440/2304 [I 2022-10-08 20:06:49] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:24:15 [I 2022-10-08 20:06:49] ...trap/engines/engine.py.229: process: 0.78733 | load: 0.00315 [I 2022-10-08 20:06:49] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-08 20:06:49] ...optimizers/trijoint.py.151: total_norm: 2.057039 [I 2022-10-08 20:06:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 450/2304 [I 2022-10-08 20:06:57] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:24:09 [I 2022-10-08 20:06:57] ...trap/engines/engine.py.229: process: 0.77892 | load: 0.00298 [I 2022-10-08 20:06:57] ...trap/engines/engine.py.230: loss: 0.31358 [I 2022-10-08 20:06:57] ...optimizers/trijoint.py.151: total_norm: 4.436098 [I 2022-10-08 20:07:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 460/2304 [I 2022-10-08 20:07:04] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:24:04 [I 2022-10-08 20:07:04] ...trap/engines/engine.py.229: process: 0.77930 | load: 0.00314 [I 2022-10-08 20:07:04] ...trap/engines/engine.py.230: loss: 0.17664 [I 2022-10-08 20:07:04] ...optimizers/trijoint.py.151: total_norm: 2.436565 [I 2022-10-08 20:07:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 470/2304 [I 2022-10-08 20:07:12] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:49 [I 2022-10-08 20:07:12] ...trap/engines/engine.py.229: process: 0.77411 | load: 0.00301 [I 2022-10-08 20:07:12] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-08 20:07:12] ...optimizers/trijoint.py.151: total_norm: 2.161628 [I 2022-10-08 20:07:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 480/2304 [I 2022-10-08 20:07:20] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:47 [I 2022-10-08 20:07:20] ...trap/engines/engine.py.229: process: 0.79084 | load: 0.00309 [I 2022-10-08 20:07:20] ...trap/engines/engine.py.230: loss: 0.20376 [I 2022-10-08 20:07:20] ...optimizers/trijoint.py.151: total_norm: 2.507510 [I 2022-10-08 20:07:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 490/2304 [I 2022-10-08 20:07:28] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:23:40 [I 2022-10-08 20:07:28] ...trap/engines/engine.py.229: process: 0.79273 | load: 0.00311 [I 2022-10-08 20:07:28] ...trap/engines/engine.py.230: loss: 0.12924 [I 2022-10-08 20:07:28] ...optimizers/trijoint.py.151: total_norm: 2.226191 [I 2022-10-08 20:07:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 500/2304 [I 2022-10-08 20:07:36] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:23:42 [I 2022-10-08 20:07:36] ...trap/engines/engine.py.229: process: 0.78465 | load: 0.00310 [I 2022-10-08 20:07:36] ...trap/engines/engine.py.230: loss: 0.23166 [I 2022-10-08 20:07:36] ...optimizers/trijoint.py.151: total_norm: 2.884313 [I 2022-10-08 20:07:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 510/2304 [I 2022-10-08 20:07:44] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:24 [I 2022-10-08 20:07:44] ...trap/engines/engine.py.229: process: 0.77873 | load: 0.00316 [I 2022-10-08 20:07:44] ...trap/engines/engine.py.230: loss: 0.14700 [I 2022-10-08 20:07:44] ...optimizers/trijoint.py.151: total_norm: 1.939251 [I 2022-10-08 20:07:52] ...trap/engines/engine.py.224: train: epoch 18 | batch 520/2304 [I 2022-10-08 20:07:52] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:17 [I 2022-10-08 20:07:52] ...trap/engines/engine.py.229: process: 0.78066 | load: 0.00311 [I 2022-10-08 20:07:52] ...trap/engines/engine.py.230: loss: 0.16029 [I 2022-10-08 20:07:52] ...optimizers/trijoint.py.151: total_norm: 2.145191 [I 2022-10-08 20:07:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 530/2304 [I 2022-10-08 20:07:59] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:23:01 [I 2022-10-08 20:07:59] ...trap/engines/engine.py.229: process: 0.77515 | load: 0.00304 [I 2022-10-08 20:07:59] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-08 20:07:59] ...optimizers/trijoint.py.151: total_norm: 2.853133 [I 2022-10-08 20:08:07] ...trap/engines/engine.py.224: train: epoch 18 | batch 540/2304 [I 2022-10-08 20:08:07] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:22:54 [I 2022-10-08 20:08:07] ...trap/engines/engine.py.229: process: 0.77714 | load: 0.00309 [I 2022-10-08 20:08:07] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-08 20:08:07] ...optimizers/trijoint.py.151: total_norm: 3.296005 [I 2022-10-08 20:08:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 550/2304 [I 2022-10-08 20:08:15] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:22:44 [I 2022-10-08 20:08:15] ...trap/engines/engine.py.229: process: 0.79207 | load: 0.00305 [I 2022-10-08 20:08:15] ...trap/engines/engine.py.230: loss: 0.16912 [I 2022-10-08 20:08:15] ...optimizers/trijoint.py.151: total_norm: 2.873798 [I 2022-10-08 20:08:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 560/2304 [I 2022-10-08 20:08:23] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:22:47 [I 2022-10-08 20:08:23] ...trap/engines/engine.py.229: process: 0.79012 | load: 0.00302 [I 2022-10-08 20:08:23] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-08 20:08:23] ...optimizers/trijoint.py.151: total_norm: 3.161398 [I 2022-10-08 20:08:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 570/2304 [I 2022-10-08 20:08:31] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:28 [I 2022-10-08 20:08:31] ...trap/engines/engine.py.229: process: 0.77465 | load: 0.00301 [I 2022-10-08 20:08:31] ...trap/engines/engine.py.230: loss: 0.15355 [I 2022-10-08 20:08:31] ...optimizers/trijoint.py.151: total_norm: 3.649706 [I 2022-10-08 20:08:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 580/2304 [I 2022-10-08 20:08:38] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:23 [I 2022-10-08 20:08:38] ...trap/engines/engine.py.229: process: 0.77481 | load: 0.00302 [I 2022-10-08 20:08:38] ...trap/engines/engine.py.230: loss: 0.18502 [I 2022-10-08 20:08:38] ...optimizers/trijoint.py.151: total_norm: 2.147606 [I 2022-10-08 20:08:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 590/2304 [I 2022-10-08 20:08:46] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:12 [I 2022-10-08 20:08:46] ...trap/engines/engine.py.229: process: 0.76603 | load: 0.00304 [I 2022-10-08 20:08:46] ...trap/engines/engine.py.230: loss: 0.13095 [I 2022-10-08 20:08:46] ...optimizers/trijoint.py.151: total_norm: 2.187673 [I 2022-10-08 20:08:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 600/2304 [I 2022-10-08 20:08:54] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:22:03 [I 2022-10-08 20:08:54] ...trap/engines/engine.py.229: process: 0.77033 | load: 0.00309 [I 2022-10-08 20:08:54] ...trap/engines/engine.py.230: loss: 0.14605 [I 2022-10-08 20:08:54] ...optimizers/trijoint.py.151: total_norm: 2.590455 [I 2022-10-08 20:09:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 610/2304 [I 2022-10-08 20:09:02] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:22:08 [I 2022-10-08 20:09:02] ...trap/engines/engine.py.229: process: 0.78725 | load: 0.00301 [I 2022-10-08 20:09:02] ...trap/engines/engine.py.230: loss: 0.18282 [I 2022-10-08 20:09:02] ...optimizers/trijoint.py.151: total_norm: 2.883554 [I 2022-10-08 20:09:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 620/2304 [I 2022-10-08 20:09:10] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:50 [I 2022-10-08 20:09:10] ...trap/engines/engine.py.229: process: 0.77352 | load: 0.00301 [I 2022-10-08 20:09:10] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-08 20:09:10] ...optimizers/trijoint.py.151: total_norm: 2.061862 [I 2022-10-08 20:09:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 630/2304 [I 2022-10-08 20:09:18] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:38 [I 2022-10-08 20:09:18] ...trap/engines/engine.py.229: process: 0.76530 | load: 0.00308 [I 2022-10-08 20:09:18] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-08 20:09:18] ...optimizers/trijoint.py.151: total_norm: 2.420476 [I 2022-10-08 20:09:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 640/2304 [I 2022-10-08 20:09:25] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:40 [I 2022-10-08 20:09:25] ...trap/engines/engine.py.229: process: 0.79076 | load: 0.00300 [I 2022-10-08 20:09:25] ...trap/engines/engine.py.230: loss: 0.20175 [I 2022-10-08 20:09:25] ...optimizers/trijoint.py.151: total_norm: 2.549083 [I 2022-10-08 20:09:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 650/2304 [I 2022-10-08 20:09:33] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:31 [I 2022-10-08 20:09:33] ...trap/engines/engine.py.229: process: 0.78074 | load: 0.00300 [I 2022-10-08 20:09:33] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-08 20:09:33] ...optimizers/trijoint.py.151: total_norm: 2.218985 [I 2022-10-08 20:09:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 660/2304 [I 2022-10-08 20:09:41] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:21:22 [I 2022-10-08 20:09:41] ...trap/engines/engine.py.229: process: 0.77562 | load: 0.00305 [I 2022-10-08 20:09:41] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-08 20:09:41] ...optimizers/trijoint.py.151: total_norm: 2.467859 [I 2022-10-08 20:09:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 670/2304 [I 2022-10-08 20:09:49] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:14 [I 2022-10-08 20:09:49] ...trap/engines/engine.py.229: process: 0.78143 | load: 0.00301 [I 2022-10-08 20:09:49] ...trap/engines/engine.py.230: loss: 0.11194 [I 2022-10-08 20:09:49] ...optimizers/trijoint.py.151: total_norm: 1.838133 [I 2022-10-08 20:09:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 680/2304 [I 2022-10-08 20:09:57] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:07 [I 2022-10-08 20:09:57] ...trap/engines/engine.py.229: process: 0.77718 | load: 0.00306 [I 2022-10-08 20:09:57] ...trap/engines/engine.py.230: loss: 0.20063 [I 2022-10-08 20:09:57] ...optimizers/trijoint.py.151: total_norm: 2.768874 [I 2022-10-08 20:10:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 690/2304 [I 2022-10-08 20:10:04] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:51 [I 2022-10-08 20:10:04] ...trap/engines/engine.py.229: process: 0.78368 | load: 0.00305 [I 2022-10-08 20:10:04] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-08 20:10:04] ...optimizers/trijoint.py.151: total_norm: 3.019898 [I 2022-10-08 20:10:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 700/2304 [I 2022-10-08 20:10:12] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:52 [I 2022-10-08 20:10:12] ...trap/engines/engine.py.229: process: 0.78962 | load: 0.00300 [I 2022-10-08 20:10:12] ...trap/engines/engine.py.230: loss: 0.16851 [I 2022-10-08 20:10:12] ...optimizers/trijoint.py.151: total_norm: 2.454638 [I 2022-10-08 20:10:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 710/2304 [I 2022-10-08 20:10:20] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:20:48 [I 2022-10-08 20:10:20] ...trap/engines/engine.py.229: process: 0.78188 | load: 0.00304 [I 2022-10-08 20:10:20] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-08 20:10:20] ...optimizers/trijoint.py.151: total_norm: 2.374145 [I 2022-10-08 20:10:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 720/2304 [I 2022-10-08 20:10:28] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:30 [I 2022-10-08 20:10:28] ...trap/engines/engine.py.229: process: 0.76727 | load: 0.00301 [I 2022-10-08 20:10:28] ...trap/engines/engine.py.230: loss: 0.17276 [I 2022-10-08 20:10:28] ...optimizers/trijoint.py.151: total_norm: 1.990993 [I 2022-10-08 20:10:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 730/2304 [I 2022-10-08 20:10:36] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:26 [I 2022-10-08 20:10:36] ...trap/engines/engine.py.229: process: 0.78849 | load: 0.00303 [I 2022-10-08 20:10:36] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-08 20:10:36] ...optimizers/trijoint.py.151: total_norm: 2.038370 [I 2022-10-08 20:10:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 740/2304 [I 2022-10-08 20:10:43] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:10 [I 2022-10-08 20:10:43] ...trap/engines/engine.py.229: process: 0.76986 | load: 0.00302 [I 2022-10-08 20:10:43] ...trap/engines/engine.py.230: loss: 0.31502 [I 2022-10-08 20:10:43] ...optimizers/trijoint.py.151: total_norm: 3.083158 [I 2022-10-08 20:10:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 750/2304 [I 2022-10-08 20:10:51] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:19 [I 2022-10-08 20:10:51] ...trap/engines/engine.py.229: process: 0.78754 | load: 0.00299 [I 2022-10-08 20:10:51] ...trap/engines/engine.py.230: loss: 0.13393 [I 2022-10-08 20:10:51] ...optimizers/trijoint.py.151: total_norm: 2.274053 [I 2022-10-08 20:10:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 760/2304 [I 2022-10-08 20:10:59] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:20:11 [I 2022-10-08 20:10:59] ...trap/engines/engine.py.229: process: 0.78378 | load: 0.00301 [I 2022-10-08 20:10:59] ...trap/engines/engine.py.230: loss: 0.15547 [I 2022-10-08 20:10:59] ...optimizers/trijoint.py.151: total_norm: 1.489794 [I 2022-10-08 20:11:07] ...trap/engines/engine.py.224: train: epoch 18 | batch 770/2304 [I 2022-10-08 20:11:07] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:20:01 [I 2022-10-08 20:11:07] ...trap/engines/engine.py.229: process: 0.78629 | load: 0.00298 [I 2022-10-08 20:11:07] ...trap/engines/engine.py.230: loss: 0.12666 [I 2022-10-08 20:11:07] ...optimizers/trijoint.py.151: total_norm: 1.959914 [I 2022-10-08 20:11:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 780/2304 [I 2022-10-08 20:11:15] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:48 [I 2022-10-08 20:11:15] ...trap/engines/engine.py.229: process: 0.79079 | load: 0.00299 [I 2022-10-08 20:11:15] ...trap/engines/engine.py.230: loss: 0.19646 [I 2022-10-08 20:11:15] ...optimizers/trijoint.py.151: total_norm: 3.740423 [I 2022-10-08 20:11:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 790/2304 [I 2022-10-08 20:11:23] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:38 [I 2022-10-08 20:11:23] ...trap/engines/engine.py.229: process: 0.78527 | load: 0.00302 [I 2022-10-08 20:11:23] ...trap/engines/engine.py.230: loss: 0.26058 [I 2022-10-08 20:11:23] ...optimizers/trijoint.py.151: total_norm: 4.172911 [I 2022-10-08 20:11:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 800/2304 [I 2022-10-08 20:11:30] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:39 [I 2022-10-08 20:11:30] ...trap/engines/engine.py.229: process: 0.78705 | load: 0.00301 [I 2022-10-08 20:11:30] ...trap/engines/engine.py.230: loss: 0.21199 [I 2022-10-08 20:11:30] ...optimizers/trijoint.py.151: total_norm: 3.148537 [I 2022-10-08 20:11:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 810/2304 [I 2022-10-08 20:11:38] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:26 [I 2022-10-08 20:11:38] ...trap/engines/engine.py.229: process: 0.77283 | load: 0.00306 [I 2022-10-08 20:11:38] ...trap/engines/engine.py.230: loss: 0.11800 [I 2022-10-08 20:11:38] ...optimizers/trijoint.py.151: total_norm: 1.743689 [I 2022-10-08 20:11:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 820/2304 [I 2022-10-08 20:11:46] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:19:17 [I 2022-10-08 20:11:46] ...trap/engines/engine.py.229: process: 0.78405 | load: 0.00302 [I 2022-10-08 20:11:46] ...trap/engines/engine.py.230: loss: 0.13812 [I 2022-10-08 20:11:46] ...optimizers/trijoint.py.151: total_norm: 2.497003 [I 2022-10-08 20:11:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 830/2304 [I 2022-10-08 20:11:54] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:09 [I 2022-10-08 20:11:54] ...trap/engines/engine.py.229: process: 0.78501 | load: 0.00310 [I 2022-10-08 20:11:54] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-08 20:11:54] ...optimizers/trijoint.py.151: total_norm: 2.352715 [I 2022-10-08 20:12:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 840/2304 [I 2022-10-08 20:12:02] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:19:01 [I 2022-10-08 20:12:02] ...trap/engines/engine.py.229: process: 0.77947 | load: 0.00305 [I 2022-10-08 20:12:02] ...trap/engines/engine.py.230: loss: 0.21432 [I 2022-10-08 20:12:02] ...optimizers/trijoint.py.151: total_norm: 2.506794 [I 2022-10-08 20:12:09] ...trap/engines/engine.py.224: train: epoch 18 | batch 850/2304 [I 2022-10-08 20:12:09] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:19:00 [I 2022-10-08 20:12:09] ...trap/engines/engine.py.229: process: 0.78730 | load: 0.00300 [I 2022-10-08 20:12:09] ...trap/engines/engine.py.230: loss: 0.18044 [I 2022-10-08 20:12:09] ...optimizers/trijoint.py.151: total_norm: 2.507644 [I 2022-10-08 20:12:17] ...trap/engines/engine.py.224: train: epoch 18 | batch 860/2304 [I 2022-10-08 20:12:17] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:46 [I 2022-10-08 20:12:17] ...trap/engines/engine.py.229: process: 0.77270 | load: 0.00305 [I 2022-10-08 20:12:17] ...trap/engines/engine.py.230: loss: 0.10029 [I 2022-10-08 20:12:17] ...optimizers/trijoint.py.151: total_norm: 1.736068 [I 2022-10-08 20:12:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 870/2304 [I 2022-10-08 20:12:25] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:49 [I 2022-10-08 20:12:25] ...trap/engines/engine.py.229: process: 0.79482 | load: 0.00303 [I 2022-10-08 20:12:25] ...trap/engines/engine.py.230: loss: 0.15438 [I 2022-10-08 20:12:25] ...optimizers/trijoint.py.151: total_norm: 2.512147 [I 2022-10-08 20:12:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 880/2304 [I 2022-10-08 20:12:33] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:18:28 [I 2022-10-08 20:12:33] ...trap/engines/engine.py.229: process: 0.76950 | load: 0.00303 [I 2022-10-08 20:12:33] ...trap/engines/engine.py.230: loss: 0.12075 [I 2022-10-08 20:12:33] ...optimizers/trijoint.py.151: total_norm: 2.583385 [I 2022-10-08 20:12:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 890/2304 [I 2022-10-08 20:12:41] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:33 [I 2022-10-08 20:12:41] ...trap/engines/engine.py.229: process: 0.78583 | load: 0.00302 [I 2022-10-08 20:12:41] ...trap/engines/engine.py.230: loss: 0.21962 [I 2022-10-08 20:12:41] ...optimizers/trijoint.py.151: total_norm: 3.426753 [I 2022-10-08 20:12:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 900/2304 [I 2022-10-08 20:12:49] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:17 [I 2022-10-08 20:12:49] ...trap/engines/engine.py.229: process: 0.78228 | load: 0.00305 [I 2022-10-08 20:12:49] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-08 20:12:49] ...optimizers/trijoint.py.151: total_norm: 3.058115 [I 2022-10-08 20:12:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 910/2304 [I 2022-10-08 20:12:57] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:18:12 [I 2022-10-08 20:12:57] ...trap/engines/engine.py.229: process: 0.78317 | load: 0.00299 [I 2022-10-08 20:12:57] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-08 20:12:57] ...optimizers/trijoint.py.151: total_norm: 2.050503 [I 2022-10-08 20:13:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 920/2304 [I 2022-10-08 20:13:04] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:18:03 [I 2022-10-08 20:13:04] ...trap/engines/engine.py.229: process: 0.78421 | load: 0.00302 [I 2022-10-08 20:13:04] ...trap/engines/engine.py.230: loss: 0.20364 [I 2022-10-08 20:13:04] ...optimizers/trijoint.py.151: total_norm: 2.742923 [I 2022-10-08 20:13:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 930/2304 [I 2022-10-08 20:13:12] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:18:00 [I 2022-10-08 20:13:12] ...trap/engines/engine.py.229: process: 0.79045 | load: 0.00300 [I 2022-10-08 20:13:12] ...trap/engines/engine.py.230: loss: 0.14352 [I 2022-10-08 20:13:12] ...optimizers/trijoint.py.151: total_norm: 2.496513 [I 2022-10-08 20:13:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 940/2304 [I 2022-10-08 20:13:20] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:49 [I 2022-10-08 20:13:20] ...trap/engines/engine.py.229: process: 0.78720 | load: 0.00304 [I 2022-10-08 20:13:20] ...trap/engines/engine.py.230: loss: 0.17059 [I 2022-10-08 20:13:20] ...optimizers/trijoint.py.151: total_norm: 1.841085 [I 2022-10-08 20:13:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 950/2304 [I 2022-10-08 20:13:28] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:45 [I 2022-10-08 20:13:28] ...trap/engines/engine.py.229: process: 0.79961 | load: 0.00299 [I 2022-10-08 20:13:28] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-08 20:13:28] ...optimizers/trijoint.py.151: total_norm: 1.895880 [I 2022-10-08 20:13:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 960/2304 [I 2022-10-08 20:13:36] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:31 [I 2022-10-08 20:13:36] ...trap/engines/engine.py.229: process: 0.78762 | load: 0.00310 [I 2022-10-08 20:13:36] ...trap/engines/engine.py.230: loss: 0.19531 [I 2022-10-08 20:13:36] ...optimizers/trijoint.py.151: total_norm: 3.118693 [I 2022-10-08 20:13:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 970/2304 [I 2022-10-08 20:13:44] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:25 [I 2022-10-08 20:13:44] ...trap/engines/engine.py.229: process: 0.78223 | load: 0.00305 [I 2022-10-08 20:13:44] ...trap/engines/engine.py.230: loss: 0.14820 [I 2022-10-08 20:13:44] ...optimizers/trijoint.py.151: total_norm: 2.319205 [I 2022-10-08 20:13:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 980/2304 [I 2022-10-08 20:13:51] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:17:14 [I 2022-10-08 20:13:51] ...trap/engines/engine.py.229: process: 0.77488 | load: 0.00299 [I 2022-10-08 20:13:51] ...trap/engines/engine.py.230: loss: 0.19776 [I 2022-10-08 20:13:51] ...optimizers/trijoint.py.151: total_norm: 3.274599 [I 2022-10-08 20:13:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 990/2304 [I 2022-10-08 20:13:59] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:17:09 [I 2022-10-08 20:13:59] ...trap/engines/engine.py.229: process: 0.77484 | load: 0.00302 [I 2022-10-08 20:13:59] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-08 20:13:59] ...optimizers/trijoint.py.151: total_norm: 2.058186 [I 2022-10-08 20:14:07] ...trap/engines/engine.py.224: train: epoch 18 | batch 1000/2304 [I 2022-10-08 20:14:07] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:17:04 [I 2022-10-08 20:14:07] ...trap/engines/engine.py.229: process: 0.78449 | load: 0.00302 [I 2022-10-08 20:14:07] ...trap/engines/engine.py.230: loss: 0.10203 [I 2022-10-08 20:14:07] ...optimizers/trijoint.py.151: total_norm: 1.929682 [I 2022-10-08 20:14:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1010/2304 [I 2022-10-08 20:14:15] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:50 [I 2022-10-08 20:14:15] ...trap/engines/engine.py.229: process: 0.77104 | load: 0.00301 [I 2022-10-08 20:14:15] ...trap/engines/engine.py.230: loss: 0.17585 [I 2022-10-08 20:14:15] ...optimizers/trijoint.py.151: total_norm: 3.156494 [I 2022-10-08 20:14:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 1020/2304 [I 2022-10-08 20:14:23] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:46 [I 2022-10-08 20:14:23] ...trap/engines/engine.py.229: process: 0.77804 | load: 0.00307 [I 2022-10-08 20:14:23] ...trap/engines/engine.py.230: loss: 0.18713 [I 2022-10-08 20:14:23] ...optimizers/trijoint.py.151: total_norm: 2.846087 [I 2022-10-08 20:14:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 1030/2304 [I 2022-10-08 20:14:31] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:16:34 [I 2022-10-08 20:14:31] ...trap/engines/engine.py.229: process: 0.76902 | load: 0.00306 [I 2022-10-08 20:14:31] ...trap/engines/engine.py.230: loss: 0.19428 [I 2022-10-08 20:14:31] ...optimizers/trijoint.py.151: total_norm: 3.456765 [I 2022-10-08 20:14:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 1040/2304 [I 2022-10-08 20:14:39] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:16:24 [I 2022-10-08 20:14:39] ...trap/engines/engine.py.229: process: 0.76741 | load: 0.00307 [I 2022-10-08 20:14:39] ...trap/engines/engine.py.230: loss: 0.17622 [I 2022-10-08 20:14:39] ...optimizers/trijoint.py.151: total_norm: 2.745491 [I 2022-10-08 20:14:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 1050/2304 [I 2022-10-08 20:14:46] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:16:19 [I 2022-10-08 20:14:46] ...trap/engines/engine.py.229: process: 0.78646 | load: 0.00306 [I 2022-10-08 20:14:46] ...trap/engines/engine.py.230: loss: 0.17026 [I 2022-10-08 20:14:46] ...optimizers/trijoint.py.151: total_norm: 2.796866 [I 2022-10-08 20:14:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 1060/2304 [I 2022-10-08 20:14:54] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:16:16 [I 2022-10-08 20:14:54] ...trap/engines/engine.py.229: process: 0.77745 | load: 0.00303 [I 2022-10-08 20:14:54] ...trap/engines/engine.py.230: loss: 0.07965 [I 2022-10-08 20:14:54] ...optimizers/trijoint.py.151: total_norm: 1.308173 [I 2022-10-08 20:15:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 1070/2304 [I 2022-10-08 20:15:02] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:16:04 [I 2022-10-08 20:15:02] ...trap/engines/engine.py.229: process: 0.79823 | load: 0.00304 [I 2022-10-08 20:15:02] ...trap/engines/engine.py.230: loss: 0.23907 [I 2022-10-08 20:15:02] ...optimizers/trijoint.py.151: total_norm: 3.743870 [I 2022-10-08 20:15:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1080/2304 [I 2022-10-08 20:15:10] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:15:57 [I 2022-10-08 20:15:10] ...trap/engines/engine.py.229: process: 0.77790 | load: 0.00307 [I 2022-10-08 20:15:10] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-08 20:15:10] ...optimizers/trijoint.py.151: total_norm: 2.447697 [I 2022-10-08 20:15:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 1090/2304 [I 2022-10-08 20:15:18] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:15:45 [I 2022-10-08 20:15:18] ...trap/engines/engine.py.229: process: 0.77753 | load: 0.00301 [I 2022-10-08 20:15:18] ...trap/engines/engine.py.230: loss: 0.18948 [I 2022-10-08 20:15:18] ...optimizers/trijoint.py.151: total_norm: 2.948000 [I 2022-10-08 20:15:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1100/2304 [I 2022-10-08 20:15:25] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:15:37 [I 2022-10-08 20:15:25] ...trap/engines/engine.py.229: process: 0.78502 | load: 0.00304 [I 2022-10-08 20:15:25] ...trap/engines/engine.py.230: loss: 0.10910 [I 2022-10-08 20:15:25] ...optimizers/trijoint.py.151: total_norm: 2.120304 [I 2022-10-08 20:15:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 1110/2304 [I 2022-10-08 20:15:33] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:15:29 [I 2022-10-08 20:15:33] ...trap/engines/engine.py.229: process: 0.78570 | load: 0.00301 [I 2022-10-08 20:15:33] ...trap/engines/engine.py.230: loss: 0.15668 [I 2022-10-08 20:15:33] ...optimizers/trijoint.py.151: total_norm: 3.403365 [I 2022-10-08 20:15:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 1120/2304 [I 2022-10-08 20:15:41] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:20 [I 2022-10-08 20:15:41] ...trap/engines/engine.py.229: process: 0.77790 | load: 0.00307 [I 2022-10-08 20:15:41] ...trap/engines/engine.py.230: loss: 0.17330 [I 2022-10-08 20:15:41] ...optimizers/trijoint.py.151: total_norm: 2.807354 [I 2022-10-08 20:15:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 1130/2304 [I 2022-10-08 20:15:49] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:16 [I 2022-10-08 20:15:49] ...trap/engines/engine.py.229: process: 0.78435 | load: 0.00300 [I 2022-10-08 20:15:49] ...trap/engines/engine.py.230: loss: 0.18648 [I 2022-10-08 20:15:49] ...optimizers/trijoint.py.151: total_norm: 3.648394 [I 2022-10-08 20:15:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 1140/2304 [I 2022-10-08 20:15:57] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:15:05 [I 2022-10-08 20:15:57] ...trap/engines/engine.py.229: process: 0.77108 | load: 0.00300 [I 2022-10-08 20:15:57] ...trap/engines/engine.py.230: loss: 0.16123 [I 2022-10-08 20:15:57] ...optimizers/trijoint.py.151: total_norm: 2.526261 [I 2022-10-08 20:16:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 1150/2304 [I 2022-10-08 20:16:04] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:14:59 [I 2022-10-08 20:16:04] ...trap/engines/engine.py.229: process: 0.79270 | load: 0.00301 [I 2022-10-08 20:16:04] ...trap/engines/engine.py.230: loss: 0.24882 [I 2022-10-08 20:16:04] ...optimizers/trijoint.py.151: total_norm: 3.686998 [I 2022-10-08 20:16:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 1160/2304 [I 2022-10-08 20:16:12] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:14:44 [I 2022-10-08 20:16:12] ...trap/engines/engine.py.229: process: 0.76313 | load: 0.00300 [I 2022-10-08 20:16:12] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-08 20:16:12] ...optimizers/trijoint.py.151: total_norm: 3.352295 [I 2022-10-08 20:16:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1170/2304 [I 2022-10-08 20:16:20] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:44 [I 2022-10-08 20:16:20] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00304 [I 2022-10-08 20:16:20] ...trap/engines/engine.py.230: loss: 0.20832 [I 2022-10-08 20:16:20] ...optimizers/trijoint.py.151: total_norm: 3.208071 [I 2022-10-08 20:16:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 1180/2304 [I 2022-10-08 20:16:28] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:14:35 [I 2022-10-08 20:16:28] ...trap/engines/engine.py.229: process: 0.78156 | load: 0.00299 [I 2022-10-08 20:16:28] ...trap/engines/engine.py.230: loss: 0.14775 [I 2022-10-08 20:16:28] ...optimizers/trijoint.py.151: total_norm: 1.636905 [I 2022-10-08 20:16:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 1190/2304 [I 2022-10-08 20:16:36] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:14:26 [I 2022-10-08 20:16:36] ...trap/engines/engine.py.229: process: 0.77410 | load: 0.00301 [I 2022-10-08 20:16:36] ...trap/engines/engine.py.230: loss: 0.20901 [I 2022-10-08 20:16:36] ...optimizers/trijoint.py.151: total_norm: 3.181245 [I 2022-10-08 20:16:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 1200/2304 [I 2022-10-08 20:16:44] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:14:23 [I 2022-10-08 20:16:44] ...trap/engines/engine.py.229: process: 0.77414 | load: 0.00304 [I 2022-10-08 20:16:44] ...trap/engines/engine.py.230: loss: 0.26245 [I 2022-10-08 20:16:44] ...optimizers/trijoint.py.151: total_norm: 3.342094 [I 2022-10-08 20:16:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 1210/2304 [I 2022-10-08 20:16:51] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:14:13 [I 2022-10-08 20:16:51] ...trap/engines/engine.py.229: process: 0.78850 | load: 0.00298 [I 2022-10-08 20:16:51] ...trap/engines/engine.py.230: loss: 0.19657 [I 2022-10-08 20:16:51] ...optimizers/trijoint.py.151: total_norm: 3.049366 [I 2022-10-08 20:16:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 1220/2304 [I 2022-10-08 20:16:59] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:14:06 [I 2022-10-08 20:16:59] ...trap/engines/engine.py.229: process: 0.77845 | load: 0.00305 [I 2022-10-08 20:16:59] ...trap/engines/engine.py.230: loss: 0.18626 [I 2022-10-08 20:16:59] ...optimizers/trijoint.py.151: total_norm: 2.312302 [I 2022-10-08 20:17:07] ...trap/engines/engine.py.224: train: epoch 18 | batch 1230/2304 [I 2022-10-08 20:17:07] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:14:01 [I 2022-10-08 20:17:07] ...trap/engines/engine.py.229: process: 0.79463 | load: 0.00299 [I 2022-10-08 20:17:07] ...trap/engines/engine.py.230: loss: 0.19916 [I 2022-10-08 20:17:07] ...optimizers/trijoint.py.151: total_norm: 3.297920 [I 2022-10-08 20:17:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1240/2304 [I 2022-10-08 20:17:15] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:13:50 [I 2022-10-08 20:17:15] ...trap/engines/engine.py.229: process: 0.78392 | load: 0.00303 [I 2022-10-08 20:17:15] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-08 20:17:15] ...optimizers/trijoint.py.151: total_norm: 1.411454 [I 2022-10-08 20:17:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 1250/2304 [I 2022-10-08 20:17:23] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:13:43 [I 2022-10-08 20:17:23] ...trap/engines/engine.py.229: process: 0.77944 | load: 0.00298 [I 2022-10-08 20:17:23] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-08 20:17:23] ...optimizers/trijoint.py.151: total_norm: 1.618463 [I 2022-10-08 20:17:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 1260/2304 [I 2022-10-08 20:17:31] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:13:35 [I 2022-10-08 20:17:31] ...trap/engines/engine.py.229: process: 0.78109 | load: 0.00305 [I 2022-10-08 20:17:31] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-08 20:17:31] ...optimizers/trijoint.py.151: total_norm: 2.774230 [I 2022-10-08 20:17:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 1270/2304 [I 2022-10-08 20:17:38] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:13:26 [I 2022-10-08 20:17:38] ...trap/engines/engine.py.229: process: 0.77299 | load: 0.00303 [I 2022-10-08 20:17:38] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-08 20:17:38] ...optimizers/trijoint.py.151: total_norm: 1.669060 [I 2022-10-08 20:17:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 1280/2304 [I 2022-10-08 20:17:46] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:13:16 [I 2022-10-08 20:17:46] ...trap/engines/engine.py.229: process: 0.77935 | load: 0.00302 [I 2022-10-08 20:17:46] ...trap/engines/engine.py.230: loss: 0.18004 [I 2022-10-08 20:17:46] ...optimizers/trijoint.py.151: total_norm: 3.089742 [I 2022-10-08 20:17:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 1290/2304 [I 2022-10-08 20:17:54] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:13:11 [I 2022-10-08 20:17:54] ...trap/engines/engine.py.229: process: 0.77743 | load: 0.00299 [I 2022-10-08 20:17:54] ...trap/engines/engine.py.230: loss: 0.27161 [I 2022-10-08 20:17:54] ...optimizers/trijoint.py.151: total_norm: 4.257353 [I 2022-10-08 20:18:02] ...trap/engines/engine.py.224: train: epoch 18 | batch 1300/2304 [I 2022-10-08 20:18:02] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:13:00 [I 2022-10-08 20:18:02] ...trap/engines/engine.py.229: process: 0.78097 | load: 0.00303 [I 2022-10-08 20:18:02] ...trap/engines/engine.py.230: loss: 0.19557 [I 2022-10-08 20:18:02] ...optimizers/trijoint.py.151: total_norm: 3.497995 [I 2022-10-08 20:18:09] ...trap/engines/engine.py.224: train: epoch 18 | batch 1310/2304 [I 2022-10-08 20:18:09] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:12:50 [I 2022-10-08 20:18:09] ...trap/engines/engine.py.229: process: 0.77455 | load: 0.00302 [I 2022-10-08 20:18:09] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-08 20:18:09] ...optimizers/trijoint.py.151: total_norm: 1.795444 [I 2022-10-08 20:18:17] ...trap/engines/engine.py.224: train: epoch 18 | batch 1320/2304 [I 2022-10-08 20:18:17] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:12:49 [I 2022-10-08 20:18:17] ...trap/engines/engine.py.229: process: 0.77471 | load: 0.00306 [I 2022-10-08 20:18:17] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-08 20:18:17] ...optimizers/trijoint.py.151: total_norm: 1.383167 [I 2022-10-08 20:18:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1330/2304 [I 2022-10-08 20:18:25] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:38 [I 2022-10-08 20:18:25] ...trap/engines/engine.py.229: process: 0.78633 | load: 0.00302 [I 2022-10-08 20:18:25] ...trap/engines/engine.py.230: loss: 0.22652 [I 2022-10-08 20:18:25] ...optimizers/trijoint.py.151: total_norm: 3.001433 [I 2022-10-08 20:18:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 1340/2304 [I 2022-10-08 20:18:33] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:12:30 [I 2022-10-08 20:18:33] ...trap/engines/engine.py.229: process: 0.78430 | load: 0.00297 [I 2022-10-08 20:18:33] ...trap/engines/engine.py.230: loss: 0.20826 [I 2022-10-08 20:18:33] ...optimizers/trijoint.py.151: total_norm: 2.840169 [I 2022-10-08 20:18:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 1350/2304 [I 2022-10-08 20:18:41] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:12:19 [I 2022-10-08 20:18:41] ...trap/engines/engine.py.229: process: 0.77482 | load: 0.00301 [I 2022-10-08 20:18:41] ...trap/engines/engine.py.230: loss: 0.15652 [I 2022-10-08 20:18:41] ...optimizers/trijoint.py.151: total_norm: 2.937891 [I 2022-10-08 20:18:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 1360/2304 [I 2022-10-08 20:18:48] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:12:11 [I 2022-10-08 20:18:48] ...trap/engines/engine.py.229: process: 0.77161 | load: 0.00309 [I 2022-10-08 20:18:48] ...trap/engines/engine.py.230: loss: 0.12269 [I 2022-10-08 20:18:48] ...optimizers/trijoint.py.151: total_norm: 2.010192 [I 2022-10-08 20:18:56] ...trap/engines/engine.py.224: train: epoch 18 | batch 1370/2304 [I 2022-10-08 20:18:56] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:12:07 [I 2022-10-08 20:18:56] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00305 [I 2022-10-08 20:18:56] ...trap/engines/engine.py.230: loss: 0.24105 [I 2022-10-08 20:18:56] ...optimizers/trijoint.py.151: total_norm: 2.492711 [I 2022-10-08 20:19:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 1380/2304 [I 2022-10-08 20:19:04] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:11:57 [I 2022-10-08 20:19:04] ...trap/engines/engine.py.229: process: 0.77310 | load: 0.00306 [I 2022-10-08 20:19:04] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-08 20:19:04] ...optimizers/trijoint.py.151: total_norm: 2.265075 [I 2022-10-08 20:19:12] ...trap/engines/engine.py.224: train: epoch 18 | batch 1390/2304 [I 2022-10-08 20:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:11:48 [I 2022-10-08 20:19:12] ...trap/engines/engine.py.229: process: 0.77908 | load: 0.00304 [I 2022-10-08 20:19:12] ...trap/engines/engine.py.230: loss: 0.18835 [I 2022-10-08 20:19:12] ...optimizers/trijoint.py.151: total_norm: 2.847037 [I 2022-10-08 20:19:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1400/2304 [I 2022-10-08 20:19:20] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:11:44 [I 2022-10-08 20:19:20] ...trap/engines/engine.py.229: process: 0.78029 | load: 0.00301 [I 2022-10-08 20:19:20] ...trap/engines/engine.py.230: loss: 0.16256 [I 2022-10-08 20:19:20] ...optimizers/trijoint.py.151: total_norm: 2.978233 [I 2022-10-08 20:19:27] ...trap/engines/engine.py.224: train: epoch 18 | batch 1410/2304 [I 2022-10-08 20:19:27] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:11:35 [I 2022-10-08 20:19:27] ...trap/engines/engine.py.229: process: 0.77216 | load: 0.00319 [I 2022-10-08 20:19:27] ...trap/engines/engine.py.230: loss: 0.16523 [I 2022-10-08 20:19:27] ...optimizers/trijoint.py.151: total_norm: 3.127258 [I 2022-10-08 20:19:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1420/2304 [I 2022-10-08 20:19:35] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:11:30 [I 2022-10-08 20:19:35] ...trap/engines/engine.py.229: process: 0.78528 | load: 0.00302 [I 2022-10-08 20:19:35] ...trap/engines/engine.py.230: loss: 0.12978 [I 2022-10-08 20:19:35] ...optimizers/trijoint.py.151: total_norm: 2.234940 [I 2022-10-08 20:19:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 1430/2304 [I 2022-10-08 20:19:43] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:16 [I 2022-10-08 20:19:43] ...trap/engines/engine.py.229: process: 0.76346 | load: 0.00302 [I 2022-10-08 20:19:43] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-08 20:19:43] ...optimizers/trijoint.py.151: total_norm: 2.897560 [I 2022-10-08 20:19:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 1440/2304 [I 2022-10-08 20:19:51] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:11 [I 2022-10-08 20:19:51] ...trap/engines/engine.py.229: process: 0.76974 | load: 0.00307 [I 2022-10-08 20:19:51] ...trap/engines/engine.py.230: loss: 0.18036 [I 2022-10-08 20:19:51] ...optimizers/trijoint.py.151: total_norm: 1.919979 [I 2022-10-08 20:19:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 1450/2304 [I 2022-10-08 20:19:59] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:10:59 [I 2022-10-08 20:19:59] ...trap/engines/engine.py.229: process: 0.76995 | load: 0.00303 [I 2022-10-08 20:19:59] ...trap/engines/engine.py.230: loss: 0.31302 [I 2022-10-08 20:19:59] ...optimizers/trijoint.py.151: total_norm: 4.001301 [I 2022-10-08 20:20:06] ...trap/engines/engine.py.224: train: epoch 18 | batch 1460/2304 [I 2022-10-08 20:20:06] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:10:59 [I 2022-10-08 20:20:06] ...trap/engines/engine.py.229: process: 0.77960 | load: 0.00306 [I 2022-10-08 20:20:06] ...trap/engines/engine.py.230: loss: 0.24030 [I 2022-10-08 20:20:06] ...optimizers/trijoint.py.151: total_norm: 3.808610 [I 2022-10-08 20:20:14] ...trap/engines/engine.py.224: train: epoch 18 | batch 1470/2304 [I 2022-10-08 20:20:14] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:10:49 [I 2022-10-08 20:20:14] ...trap/engines/engine.py.229: process: 0.76309 | load: 0.00308 [I 2022-10-08 20:20:14] ...trap/engines/engine.py.230: loss: 0.20689 [I 2022-10-08 20:20:14] ...optimizers/trijoint.py.151: total_norm: 3.131958 [I 2022-10-08 20:20:22] ...trap/engines/engine.py.224: train: epoch 18 | batch 1480/2304 [I 2022-10-08 20:20:22] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:41 [I 2022-10-08 20:20:22] ...trap/engines/engine.py.229: process: 0.77928 | load: 0.00305 [I 2022-10-08 20:20:22] ...trap/engines/engine.py.230: loss: 0.18371 [I 2022-10-08 20:20:22] ...optimizers/trijoint.py.151: total_norm: 3.722646 [I 2022-10-08 20:20:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1490/2304 [I 2022-10-08 20:20:30] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:33 [I 2022-10-08 20:20:30] ...trap/engines/engine.py.229: process: 0.78091 | load: 0.00304 [I 2022-10-08 20:20:30] ...trap/engines/engine.py.230: loss: 0.16281 [I 2022-10-08 20:20:30] ...optimizers/trijoint.py.151: total_norm: 2.844219 [I 2022-10-08 20:20:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 1500/2304 [I 2022-10-08 20:20:38] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:24 [I 2022-10-08 20:20:38] ...trap/engines/engine.py.229: process: 0.78017 | load: 0.00303 [I 2022-10-08 20:20:38] ...trap/engines/engine.py.230: loss: 0.13140 [I 2022-10-08 20:20:38] ...optimizers/trijoint.py.151: total_norm: 2.275369 [I 2022-10-08 20:20:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1510/2304 [I 2022-10-08 20:20:45] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:17 [I 2022-10-08 20:20:45] ...trap/engines/engine.py.229: process: 0.78348 | load: 0.00301 [I 2022-10-08 20:20:45] ...trap/engines/engine.py.230: loss: 0.16310 [I 2022-10-08 20:20:45] ...optimizers/trijoint.py.151: total_norm: 2.411224 [I 2022-10-08 20:20:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 1520/2304 [I 2022-10-08 20:20:53] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:10:12 [I 2022-10-08 20:20:53] ...trap/engines/engine.py.229: process: 0.78506 | load: 0.00305 [I 2022-10-08 20:20:53] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-08 20:20:53] ...optimizers/trijoint.py.151: total_norm: 2.286814 [I 2022-10-08 20:21:01] ...trap/engines/engine.py.224: train: epoch 18 | batch 1530/2304 [I 2022-10-08 20:21:01] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:02 [I 2022-10-08 20:21:01] ...trap/engines/engine.py.229: process: 0.78656 | load: 0.00298 [I 2022-10-08 20:21:01] ...trap/engines/engine.py.230: loss: 0.14107 [I 2022-10-08 20:21:01] ...optimizers/trijoint.py.151: total_norm: 2.851548 [I 2022-10-08 20:21:09] ...trap/engines/engine.py.224: train: epoch 18 | batch 1540/2304 [I 2022-10-08 20:21:09] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:55 [I 2022-10-08 20:21:09] ...trap/engines/engine.py.229: process: 0.78642 | load: 0.00302 [I 2022-10-08 20:21:09] ...trap/engines/engine.py.230: loss: 0.25605 [I 2022-10-08 20:21:09] ...optimizers/trijoint.py.151: total_norm: 4.000124 [I 2022-10-08 20:21:17] ...trap/engines/engine.py.224: train: epoch 18 | batch 1550/2304 [I 2022-10-08 20:21:17] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:47 [I 2022-10-08 20:21:17] ...trap/engines/engine.py.229: process: 0.77978 | load: 0.00301 [I 2022-10-08 20:21:17] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-08 20:21:17] ...optimizers/trijoint.py.151: total_norm: 2.562827 [I 2022-10-08 20:21:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 1560/2304 [I 2022-10-08 20:21:24] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:37 [I 2022-10-08 20:21:24] ...trap/engines/engine.py.229: process: 0.76609 | load: 0.00304 [I 2022-10-08 20:21:24] ...trap/engines/engine.py.230: loss: 0.19693 [I 2022-10-08 20:21:24] ...optimizers/trijoint.py.151: total_norm: 2.760010 [I 2022-10-08 20:21:32] ...trap/engines/engine.py.224: train: epoch 18 | batch 1570/2304 [I 2022-10-08 20:21:32] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:09:29 [I 2022-10-08 20:21:32] ...trap/engines/engine.py.229: process: 0.76314 | load: 0.00299 [I 2022-10-08 20:21:32] ...trap/engines/engine.py.230: loss: 0.23781 [I 2022-10-08 20:21:32] ...optimizers/trijoint.py.151: total_norm: 3.604322 [I 2022-10-08 20:21:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1580/2304 [I 2022-10-08 20:21:40] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:24 [I 2022-10-08 20:21:40] ...trap/engines/engine.py.229: process: 0.79069 | load: 0.00299 [I 2022-10-08 20:21:40] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-08 20:21:40] ...optimizers/trijoint.py.151: total_norm: 2.471685 [I 2022-10-08 20:21:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 1590/2304 [I 2022-10-08 20:21:48] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:16 [I 2022-10-08 20:21:48] ...trap/engines/engine.py.229: process: 0.78131 | load: 0.00302 [I 2022-10-08 20:21:48] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-08 20:21:48] ...optimizers/trijoint.py.151: total_norm: 2.394877 [I 2022-10-08 20:21:56] ...trap/engines/engine.py.224: train: epoch 18 | batch 1600/2304 [I 2022-10-08 20:21:56] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:05 [I 2022-10-08 20:21:56] ...trap/engines/engine.py.229: process: 0.77245 | load: 0.00299 [I 2022-10-08 20:21:56] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-08 20:21:56] ...optimizers/trijoint.py.151: total_norm: 2.889256 [I 2022-10-08 20:22:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 1610/2304 [I 2022-10-08 20:22:03] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:08:59 [I 2022-10-08 20:22:03] ...trap/engines/engine.py.229: process: 0.77641 | load: 0.00299 [I 2022-10-08 20:22:03] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-08 20:22:03] ...optimizers/trijoint.py.151: total_norm: 2.962456 [I 2022-10-08 20:22:11] ...trap/engines/engine.py.224: train: epoch 18 | batch 1620/2304 [I 2022-10-08 20:22:11] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:50 [I 2022-10-08 20:22:11] ...trap/engines/engine.py.229: process: 0.77711 | load: 0.00300 [I 2022-10-08 20:22:11] ...trap/engines/engine.py.230: loss: 0.14788 [I 2022-10-08 20:22:11] ...optimizers/trijoint.py.151: total_norm: 2.544051 [I 2022-10-08 20:22:19] ...trap/engines/engine.py.224: train: epoch 18 | batch 1630/2304 [I 2022-10-08 20:22:19] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:43 [I 2022-10-08 20:22:19] ...trap/engines/engine.py.229: process: 0.77867 | load: 0.00303 [I 2022-10-08 20:22:19] ...trap/engines/engine.py.230: loss: 0.14620 [I 2022-10-08 20:22:19] ...optimizers/trijoint.py.151: total_norm: 2.421099 [I 2022-10-08 20:22:27] ...trap/engines/engine.py.224: train: epoch 18 | batch 1640/2304 [I 2022-10-08 20:22:27] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:34 [I 2022-10-08 20:22:27] ...trap/engines/engine.py.229: process: 0.76427 | load: 0.00300 [I 2022-10-08 20:22:27] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-08 20:22:27] ...optimizers/trijoint.py.151: total_norm: 2.113040 [I 2022-10-08 20:22:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 1650/2304 [I 2022-10-08 20:22:34] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:30 [I 2022-10-08 20:22:34] ...trap/engines/engine.py.229: process: 0.77613 | load: 0.00302 [I 2022-10-08 20:22:34] ...trap/engines/engine.py.230: loss: 0.18107 [I 2022-10-08 20:22:34] ...optimizers/trijoint.py.151: total_norm: 2.772862 [I 2022-10-08 20:22:42] ...trap/engines/engine.py.224: train: epoch 18 | batch 1660/2304 [I 2022-10-08 20:22:42] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:22 [I 2022-10-08 20:22:42] ...trap/engines/engine.py.229: process: 0.77976 | load: 0.00302 [I 2022-10-08 20:22:42] ...trap/engines/engine.py.230: loss: 0.17584 [I 2022-10-08 20:22:42] ...optimizers/trijoint.py.151: total_norm: 2.771881 [I 2022-10-08 20:22:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1670/2304 [I 2022-10-08 20:22:50] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:08:14 [I 2022-10-08 20:22:50] ...trap/engines/engine.py.229: process: 0.78700 | load: 0.00301 [I 2022-10-08 20:22:50] ...trap/engines/engine.py.230: loss: 0.23111 [I 2022-10-08 20:22:50] ...optimizers/trijoint.py.151: total_norm: 2.827072 [I 2022-10-08 20:22:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 1680/2304 [I 2022-10-08 20:22:58] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:06 [I 2022-10-08 20:22:58] ...trap/engines/engine.py.229: process: 0.78384 | load: 0.00302 [I 2022-10-08 20:22:58] ...trap/engines/engine.py.230: loss: 0.18827 [I 2022-10-08 20:22:58] ...optimizers/trijoint.py.151: total_norm: 2.767171 [I 2022-10-08 20:23:06] ...trap/engines/engine.py.224: train: epoch 18 | batch 1690/2304 [I 2022-10-08 20:23:06] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:07:57 [I 2022-10-08 20:23:06] ...trap/engines/engine.py.229: process: 0.77358 | load: 0.00311 [I 2022-10-08 20:23:06] ...trap/engines/engine.py.230: loss: 0.16231 [I 2022-10-08 20:23:06] ...optimizers/trijoint.py.151: total_norm: 1.989967 [I 2022-10-08 20:23:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 1700/2304 [I 2022-10-08 20:23:13] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:49 [I 2022-10-08 20:23:13] ...trap/engines/engine.py.229: process: 0.77906 | load: 0.00301 [I 2022-10-08 20:23:13] ...trap/engines/engine.py.230: loss: 0.20702 [I 2022-10-08 20:23:13] ...optimizers/trijoint.py.151: total_norm: 3.028770 [I 2022-10-08 20:23:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 1710/2304 [I 2022-10-08 20:23:21] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:40 [I 2022-10-08 20:23:21] ...trap/engines/engine.py.229: process: 0.76187 | load: 0.00300 [I 2022-10-08 20:23:21] ...trap/engines/engine.py.230: loss: 0.19132 [I 2022-10-08 20:23:21] ...optimizers/trijoint.py.151: total_norm: 2.585615 [I 2022-10-08 20:23:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 1720/2304 [I 2022-10-08 20:23:29] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:07:35 [I 2022-10-08 20:23:29] ...trap/engines/engine.py.229: process: 0.77085 | load: 0.00304 [I 2022-10-08 20:23:29] ...trap/engines/engine.py.230: loss: 0.20660 [I 2022-10-08 20:23:29] ...optimizers/trijoint.py.151: total_norm: 3.540402 [I 2022-10-08 20:23:37] ...trap/engines/engine.py.224: train: epoch 18 | batch 1730/2304 [I 2022-10-08 20:23:37] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:07:26 [I 2022-10-08 20:23:37] ...trap/engines/engine.py.229: process: 0.78037 | load: 0.00301 [I 2022-10-08 20:23:37] ...trap/engines/engine.py.230: loss: 0.12855 [I 2022-10-08 20:23:37] ...optimizers/trijoint.py.151: total_norm: 2.635179 [I 2022-10-08 20:23:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1740/2304 [I 2022-10-08 20:23:45] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:19 [I 2022-10-08 20:23:45] ...trap/engines/engine.py.229: process: 0.77262 | load: 0.00300 [I 2022-10-08 20:23:45] ...trap/engines/engine.py.230: loss: 0.18170 [I 2022-10-08 20:23:45] ...optimizers/trijoint.py.151: total_norm: 2.036826 [I 2022-10-08 20:23:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 1750/2304 [I 2022-10-08 20:23:53] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:12 [I 2022-10-08 20:23:53] ...trap/engines/engine.py.229: process: 0.77928 | load: 0.00299 [I 2022-10-08 20:23:53] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-08 20:23:53] ...optimizers/trijoint.py.151: total_norm: 1.743005 [I 2022-10-08 20:24:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1760/2304 [I 2022-10-08 20:24:00] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:02 [I 2022-10-08 20:24:00] ...trap/engines/engine.py.229: process: 0.77765 | load: 0.00302 [I 2022-10-08 20:24:00] ...trap/engines/engine.py.230: loss: 0.22388 [I 2022-10-08 20:24:00] ...optimizers/trijoint.py.151: total_norm: 2.809959 [I 2022-10-08 20:24:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 1770/2304 [I 2022-10-08 20:24:08] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:06:55 [I 2022-10-08 20:24:08] ...trap/engines/engine.py.229: process: 0.78145 | load: 0.00300 [I 2022-10-08 20:24:08] ...trap/engines/engine.py.230: loss: 0.15250 [I 2022-10-08 20:24:08] ...optimizers/trijoint.py.151: total_norm: 2.535444 [I 2022-10-08 20:24:16] ...trap/engines/engine.py.224: train: epoch 18 | batch 1780/2304 [I 2022-10-08 20:24:16] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:06:46 [I 2022-10-08 20:24:16] ...trap/engines/engine.py.229: process: 0.77760 | load: 0.00304 [I 2022-10-08 20:24:16] ...trap/engines/engine.py.230: loss: 0.27853 [I 2022-10-08 20:24:16] ...optimizers/trijoint.py.151: total_norm: 4.820727 [I 2022-10-08 20:24:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 1790/2304 [I 2022-10-08 20:24:24] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:06:39 [I 2022-10-08 20:24:24] ...trap/engines/engine.py.229: process: 0.78195 | load: 0.00300 [I 2022-10-08 20:24:24] ...trap/engines/engine.py.230: loss: 0.17936 [I 2022-10-08 20:24:24] ...optimizers/trijoint.py.151: total_norm: 2.992019 [I 2022-10-08 20:24:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 1800/2304 [I 2022-10-08 20:24:31] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:31 [I 2022-10-08 20:24:31] ...trap/engines/engine.py.229: process: 0.77115 | load: 0.00300 [I 2022-10-08 20:24:31] ...trap/engines/engine.py.230: loss: 0.26376 [I 2022-10-08 20:24:31] ...optimizers/trijoint.py.151: total_norm: 3.415131 [I 2022-10-08 20:24:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 1810/2304 [I 2022-10-08 20:24:39] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:23 [I 2022-10-08 20:24:39] ...trap/engines/engine.py.229: process: 0.76568 | load: 0.00300 [I 2022-10-08 20:24:39] ...trap/engines/engine.py.230: loss: 0.18671 [I 2022-10-08 20:24:39] ...optimizers/trijoint.py.151: total_norm: 2.353319 [I 2022-10-08 20:24:47] ...trap/engines/engine.py.224: train: epoch 18 | batch 1820/2304 [I 2022-10-08 20:24:47] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:06:17 [I 2022-10-08 20:24:47] ...trap/engines/engine.py.229: process: 0.78404 | load: 0.00304 [I 2022-10-08 20:24:47] ...trap/engines/engine.py.230: loss: 0.22385 [I 2022-10-08 20:24:47] ...optimizers/trijoint.py.151: total_norm: 2.896538 [I 2022-10-08 20:24:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1830/2304 [I 2022-10-08 20:24:55] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:06:10 [I 2022-10-08 20:24:55] ...trap/engines/engine.py.229: process: 0.78195 | load: 0.00301 [I 2022-10-08 20:24:55] ...trap/engines/engine.py.230: loss: 0.16525 [I 2022-10-08 20:24:55] ...optimizers/trijoint.py.151: total_norm: 2.545407 [I 2022-10-08 20:25:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 1840/2304 [I 2022-10-08 20:25:03] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:06:00 [I 2022-10-08 20:25:03] ...trap/engines/engine.py.229: process: 0.77112 | load: 0.00303 [I 2022-10-08 20:25:03] ...trap/engines/engine.py.230: loss: 0.16345 [I 2022-10-08 20:25:03] ...optimizers/trijoint.py.151: total_norm: 2.171560 [I 2022-10-08 20:25:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1850/2304 [I 2022-10-08 20:25:10] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:05:51 [I 2022-10-08 20:25:10] ...trap/engines/engine.py.229: process: 0.77550 | load: 0.00305 [I 2022-10-08 20:25:10] ...trap/engines/engine.py.230: loss: 0.16464 [I 2022-10-08 20:25:10] ...optimizers/trijoint.py.151: total_norm: 2.341899 [I 2022-10-08 20:25:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 1860/2304 [I 2022-10-08 20:25:18] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:44 [I 2022-10-08 20:25:18] ...trap/engines/engine.py.229: process: 0.77939 | load: 0.00303 [I 2022-10-08 20:25:18] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-08 20:25:18] ...optimizers/trijoint.py.151: total_norm: 2.541651 [I 2022-10-08 20:25:26] ...trap/engines/engine.py.224: train: epoch 18 | batch 1870/2304 [I 2022-10-08 20:25:26] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:05:36 [I 2022-10-08 20:25:26] ...trap/engines/engine.py.229: process: 0.77401 | load: 0.00302 [I 2022-10-08 20:25:26] ...trap/engines/engine.py.230: loss: 0.20659 [I 2022-10-08 20:25:26] ...optimizers/trijoint.py.151: total_norm: 3.234879 [I 2022-10-08 20:25:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 1880/2304 [I 2022-10-08 20:25:34] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:05:28 [I 2022-10-08 20:25:34] ...trap/engines/engine.py.229: process: 0.77386 | load: 0.00305 [I 2022-10-08 20:25:34] ...trap/engines/engine.py.230: loss: 0.21220 [I 2022-10-08 20:25:34] ...optimizers/trijoint.py.151: total_norm: 3.113049 [I 2022-10-08 20:25:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 1890/2304 [I 2022-10-08 20:25:42] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:05:23 [I 2022-10-08 20:25:42] ...trap/engines/engine.py.229: process: 0.77971 | load: 0.00308 [I 2022-10-08 20:25:42] ...trap/engines/engine.py.230: loss: 0.17078 [I 2022-10-08 20:25:42] ...optimizers/trijoint.py.151: total_norm: 1.787056 [I 2022-10-08 20:25:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1900/2304 [I 2022-10-08 20:25:50] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:15 [I 2022-10-08 20:25:50] ...trap/engines/engine.py.229: process: 0.77376 | load: 0.00316 [I 2022-10-08 20:25:50] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-08 20:25:50] ...optimizers/trijoint.py.151: total_norm: 3.404603 [I 2022-10-08 20:25:57] ...trap/engines/engine.py.224: train: epoch 18 | batch 1910/2304 [I 2022-10-08 20:25:57] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:09 [I 2022-10-08 20:25:57] ...trap/engines/engine.py.229: process: 0.78164 | load: 0.00302 [I 2022-10-08 20:25:57] ...trap/engines/engine.py.230: loss: 0.15762 [I 2022-10-08 20:25:57] ...optimizers/trijoint.py.151: total_norm: 1.427075 [I 2022-10-08 20:26:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1920/2304 [I 2022-10-08 20:26:05] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:05:00 [I 2022-10-08 20:26:05] ...trap/engines/engine.py.229: process: 0.77196 | load: 0.00316 [I 2022-10-08 20:26:05] ...trap/engines/engine.py.230: loss: 0.20515 [I 2022-10-08 20:26:05] ...optimizers/trijoint.py.151: total_norm: 2.618979 [I 2022-10-08 20:26:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 1930/2304 [I 2022-10-08 20:26:13] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:04:52 [I 2022-10-08 20:26:13] ...trap/engines/engine.py.229: process: 0.78476 | load: 0.00301 [I 2022-10-08 20:26:13] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-08 20:26:13] ...optimizers/trijoint.py.151: total_norm: 1.653195 [I 2022-10-08 20:26:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 1940/2304 [I 2022-10-08 20:26:21] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:04:45 [I 2022-10-08 20:26:21] ...trap/engines/engine.py.229: process: 0.79197 | load: 0.00302 [I 2022-10-08 20:26:21] ...trap/engines/engine.py.230: loss: 0.24555 [I 2022-10-08 20:26:21] ...optimizers/trijoint.py.151: total_norm: 4.172562 [I 2022-10-08 20:26:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 1950/2304 [I 2022-10-08 20:26:29] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:36 [I 2022-10-08 20:26:29] ...trap/engines/engine.py.229: process: 0.77120 | load: 0.00301 [I 2022-10-08 20:26:29] ...trap/engines/engine.py.230: loss: 0.17106 [I 2022-10-08 20:26:29] ...optimizers/trijoint.py.151: total_norm: 2.174545 [I 2022-10-08 20:26:37] ...trap/engines/engine.py.224: train: epoch 18 | batch 1960/2304 [I 2022-10-08 20:26:37] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:04:30 [I 2022-10-08 20:26:37] ...trap/engines/engine.py.229: process: 0.77966 | load: 0.00301 [I 2022-10-08 20:26:37] ...trap/engines/engine.py.230: loss: 0.21868 [I 2022-10-08 20:26:37] ...optimizers/trijoint.py.151: total_norm: 3.106311 [I 2022-10-08 20:26:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 1970/2304 [I 2022-10-08 20:26:44] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:04:20 [I 2022-10-08 20:26:44] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00301 [I 2022-10-08 20:26:44] ...trap/engines/engine.py.230: loss: 0.20149 [I 2022-10-08 20:26:44] ...optimizers/trijoint.py.151: total_norm: 3.612853 [I 2022-10-08 20:26:52] ...trap/engines/engine.py.224: train: epoch 18 | batch 1980/2304 [I 2022-10-08 20:26:52] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:04:12 [I 2022-10-08 20:26:52] ...trap/engines/engine.py.229: process: 0.77535 | load: 0.00298 [I 2022-10-08 20:26:52] ...trap/engines/engine.py.230: loss: 0.20376 [I 2022-10-08 20:26:52] ...optimizers/trijoint.py.151: total_norm: 3.071831 [I 2022-10-08 20:27:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1990/2304 [I 2022-10-08 20:27:00] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:04:05 [I 2022-10-08 20:27:00] ...trap/engines/engine.py.229: process: 0.77489 | load: 0.00302 [I 2022-10-08 20:27:00] ...trap/engines/engine.py.230: loss: 0.18884 [I 2022-10-08 20:27:00] ...optimizers/trijoint.py.151: total_norm: 2.424394 [I 2022-10-08 20:27:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 2000/2304 [I 2022-10-08 20:27:08] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:03:56 [I 2022-10-08 20:27:08] ...trap/engines/engine.py.229: process: 0.76471 | load: 0.00306 [I 2022-10-08 20:27:08] ...trap/engines/engine.py.230: loss: 0.18577 [I 2022-10-08 20:27:08] ...optimizers/trijoint.py.151: total_norm: 3.109880 [I 2022-10-08 20:27:16] ...trap/engines/engine.py.224: train: epoch 18 | batch 2010/2304 [I 2022-10-08 20:27:16] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:03:49 [I 2022-10-08 20:27:16] ...trap/engines/engine.py.229: process: 0.77963 | load: 0.00300 [I 2022-10-08 20:27:16] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-08 20:27:16] ...optimizers/trijoint.py.151: total_norm: 2.376159 [I 2022-10-08 20:27:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 2020/2304 [I 2022-10-08 20:27:24] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:03:41 [I 2022-10-08 20:27:24] ...trap/engines/engine.py.229: process: 0.76998 | load: 0.00299 [I 2022-10-08 20:27:24] ...trap/engines/engine.py.230: loss: 0.27021 [I 2022-10-08 20:27:24] ...optimizers/trijoint.py.151: total_norm: 3.599098 [I 2022-10-08 20:27:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 2030/2304 [I 2022-10-08 20:27:31] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:35 [I 2022-10-08 20:27:31] ...trap/engines/engine.py.229: process: 0.79047 | load: 0.00301 [I 2022-10-08 20:27:31] ...trap/engines/engine.py.230: loss: 0.16191 [I 2022-10-08 20:27:31] ...optimizers/trijoint.py.151: total_norm: 2.798043 [I 2022-10-08 20:27:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 2040/2304 [I 2022-10-08 20:27:39] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:03:26 [I 2022-10-08 20:27:39] ...trap/engines/engine.py.229: process: 0.78435 | load: 0.00302 [I 2022-10-08 20:27:39] ...trap/engines/engine.py.230: loss: 0.20390 [I 2022-10-08 20:27:39] ...optimizers/trijoint.py.151: total_norm: 3.308392 [I 2022-10-08 20:27:47] ...trap/engines/engine.py.224: train: epoch 18 | batch 2050/2304 [I 2022-10-08 20:27:47] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:03:18 [I 2022-10-08 20:27:47] ...trap/engines/engine.py.229: process: 0.77838 | load: 0.00304 [I 2022-10-08 20:27:47] ...trap/engines/engine.py.230: loss: 0.16630 [I 2022-10-08 20:27:47] ...optimizers/trijoint.py.151: total_norm: 2.121498 [I 2022-10-08 20:27:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 2060/2304 [I 2022-10-08 20:27:55] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:03:08 [I 2022-10-08 20:27:55] ...trap/engines/engine.py.229: process: 0.76845 | load: 0.00304 [I 2022-10-08 20:27:55] ...trap/engines/engine.py.230: loss: 0.22789 [I 2022-10-08 20:27:55] ...optimizers/trijoint.py.151: total_norm: 3.485615 [I 2022-10-08 20:28:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 2070/2304 [I 2022-10-08 20:28:03] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:03:02 [I 2022-10-08 20:28:03] ...trap/engines/engine.py.229: process: 0.78817 | load: 0.00301 [I 2022-10-08 20:28:03] ...trap/engines/engine.py.230: loss: 0.18126 [I 2022-10-08 20:28:03] ...optimizers/trijoint.py.151: total_norm: 2.504191 [I 2022-10-08 20:28:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 2080/2304 [I 2022-10-08 20:28:10] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:02:54 [I 2022-10-08 20:28:10] ...trap/engines/engine.py.229: process: 0.77548 | load: 0.00298 [I 2022-10-08 20:28:10] ...trap/engines/engine.py.230: loss: 0.17679 [I 2022-10-08 20:28:10] ...optimizers/trijoint.py.151: total_norm: 3.094795 [I 2022-10-08 20:28:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 2090/2304 [I 2022-10-08 20:28:18] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:02:47 [I 2022-10-08 20:28:18] ...trap/engines/engine.py.229: process: 0.77567 | load: 0.00302 [I 2022-10-08 20:28:18] ...trap/engines/engine.py.230: loss: 0.16677 [I 2022-10-08 20:28:18] ...optimizers/trijoint.py.151: total_norm: 3.345234 [I 2022-10-08 20:28:26] ...trap/engines/engine.py.224: train: epoch 18 | batch 2100/2304 [I 2022-10-08 20:28:26] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:02:38 [I 2022-10-08 20:28:26] ...trap/engines/engine.py.229: process: 0.77834 | load: 0.00304 [I 2022-10-08 20:28:26] ...trap/engines/engine.py.230: loss: 0.17305 [I 2022-10-08 20:28:26] ...optimizers/trijoint.py.151: total_norm: 2.290022 [I 2022-10-08 20:28:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 2110/2304 [I 2022-10-08 20:28:34] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:30 [I 2022-10-08 20:28:34] ...trap/engines/engine.py.229: process: 0.78403 | load: 0.00306 [I 2022-10-08 20:28:34] ...trap/engines/engine.py.230: loss: 0.14441 [I 2022-10-08 20:28:34] ...optimizers/trijoint.py.151: total_norm: 2.661871 [I 2022-10-08 20:28:42] ...trap/engines/engine.py.224: train: epoch 18 | batch 2120/2304 [I 2022-10-08 20:28:42] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:02:22 [I 2022-10-08 20:28:42] ...trap/engines/engine.py.229: process: 0.76761 | load: 0.00300 [I 2022-10-08 20:28:42] ...trap/engines/engine.py.230: loss: 0.18299 [I 2022-10-08 20:28:42] ...optimizers/trijoint.py.151: total_norm: 2.730846 [I 2022-10-08 20:28:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 2130/2304 [I 2022-10-08 20:28:50] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:02:15 [I 2022-10-08 20:28:50] ...trap/engines/engine.py.229: process: 0.77628 | load: 0.00301 [I 2022-10-08 20:28:50] ...trap/engines/engine.py.230: loss: 0.21141 [I 2022-10-08 20:28:50] ...optimizers/trijoint.py.151: total_norm: 2.798711 [I 2022-10-08 20:29:16] ...trap/engines/engine.py.224: train: epoch 18 | batch 2140/2304 [I 2022-10-08 20:29:16] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:12:24 [I 2022-10-08 20:29:16] ...trap/engines/engine.py.229: process: 19.56818 | load: 0.00307 [I 2022-10-08 20:29:16] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-08 20:29:16] ...optimizers/trijoint.py.151: total_norm: 2.111371 [I 2022-10-08 20:29:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 2150/2304 [I 2022-10-08 20:29:24] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:03:02 [I 2022-10-08 20:29:24] ...trap/engines/engine.py.229: process: 0.77822 | load: 0.00300 [I 2022-10-08 20:29:24] ...trap/engines/engine.py.230: loss: 0.17975 [I 2022-10-08 20:29:24] ...optimizers/trijoint.py.151: total_norm: 1.908277 [I 2022-10-08 20:29:32] ...trap/engines/engine.py.224: train: epoch 18 | batch 2160/2304 [I 2022-10-08 20:29:32] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:01:58 [I 2022-10-08 20:29:32] ...trap/engines/engine.py.229: process: 0.77371 | load: 0.00301 [I 2022-10-08 20:29:32] ...trap/engines/engine.py.230: loss: 0.13813 [I 2022-10-08 20:29:32] ...optimizers/trijoint.py.151: total_norm: 2.255119 [I 2022-10-08 20:29:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 2170/2304 [I 2022-10-08 20:29:40] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:01:45 [I 2022-10-08 20:29:40] ...trap/engines/engine.py.229: process: 0.78253 | load: 0.00301 [I 2022-10-08 20:29:40] ...trap/engines/engine.py.230: loss: 0.12957 [I 2022-10-08 20:29:40] ...optimizers/trijoint.py.151: total_norm: 2.018783 [I 2022-10-08 20:29:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 2180/2304 [I 2022-10-08 20:29:48] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:01:36 [I 2022-10-08 20:29:48] ...trap/engines/engine.py.229: process: 0.78665 | load: 0.00305 [I 2022-10-08 20:29:48] ...trap/engines/engine.py.230: loss: 0.23003 [I 2022-10-08 20:29:48] ...optimizers/trijoint.py.151: total_norm: 4.069124 [I 2022-10-08 20:29:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 2190/2304 [I 2022-10-08 20:29:55] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:01:28 [I 2022-10-08 20:29:55] ...trap/engines/engine.py.229: process: 0.79518 | load: 0.00301 [I 2022-10-08 20:29:55] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-08 20:29:55] ...optimizers/trijoint.py.151: total_norm: 3.078063 [I 2022-10-08 20:30:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 2200/2304 [I 2022-10-08 20:30:03] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:01:20 [I 2022-10-08 20:30:03] ...trap/engines/engine.py.229: process: 0.77569 | load: 0.00299 [I 2022-10-08 20:30:03] ...trap/engines/engine.py.230: loss: 0.25484 [I 2022-10-08 20:30:03] ...optimizers/trijoint.py.151: total_norm: 4.105217 [I 2022-10-08 20:30:11] ...trap/engines/engine.py.224: train: epoch 18 | batch 2210/2304 [I 2022-10-08 20:30:11] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:01:13 [I 2022-10-08 20:30:11] ...trap/engines/engine.py.229: process: 0.77832 | load: 0.00301 [I 2022-10-08 20:30:11] ...trap/engines/engine.py.230: loss: 0.21833 [I 2022-10-08 20:30:11] ...optimizers/trijoint.py.151: total_norm: 3.800016 [I 2022-10-08 20:30:19] ...trap/engines/engine.py.224: train: epoch 18 | batch 2220/2304 [I 2022-10-08 20:30:19] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:01:05 [I 2022-10-08 20:30:19] ...trap/engines/engine.py.229: process: 0.78762 | load: 0.00300 [I 2022-10-08 20:30:19] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-08 20:30:19] ...optimizers/trijoint.py.151: total_norm: 2.081563 [I 2022-10-08 20:30:27] ...trap/engines/engine.py.224: train: epoch 18 | batch 2230/2304 [I 2022-10-08 20:30:27] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:00:57 [I 2022-10-08 20:30:27] ...trap/engines/engine.py.229: process: 0.77900 | load: 0.00303 [I 2022-10-08 20:30:27] ...trap/engines/engine.py.230: loss: 0.17014 [I 2022-10-08 20:30:27] ...optimizers/trijoint.py.151: total_norm: 2.971959 [I 2022-10-08 20:30:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 2240/2304 [I 2022-10-08 20:30:34] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:00:49 [I 2022-10-08 20:30:34] ...trap/engines/engine.py.229: process: 0.76494 | load: 0.00301 [I 2022-10-08 20:30:34] ...trap/engines/engine.py.230: loss: 0.23027 [I 2022-10-08 20:30:34] ...optimizers/trijoint.py.151: total_norm: 2.923444 [I 2022-10-08 20:30:42] ...trap/engines/engine.py.224: train: epoch 18 | batch 2250/2304 [I 2022-10-08 20:30:42] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:00:41 [I 2022-10-08 20:30:42] ...trap/engines/engine.py.229: process: 0.79063 | load: 0.00302 [I 2022-10-08 20:30:42] ...trap/engines/engine.py.230: loss: 0.24035 [I 2022-10-08 20:30:42] ...optimizers/trijoint.py.151: total_norm: 3.310367 [I 2022-10-08 20:30:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 2260/2304 [I 2022-10-08 20:30:50] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:00:34 [I 2022-10-08 20:30:50] ...trap/engines/engine.py.229: process: 0.77480 | load: 0.00300 [I 2022-10-08 20:30:50] ...trap/engines/engine.py.230: loss: 0.16809 [I 2022-10-08 20:30:50] ...optimizers/trijoint.py.151: total_norm: 3.481193 [I 2022-10-08 20:30:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 2270/2304 [I 2022-10-08 20:30:58] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:00:26 [I 2022-10-08 20:30:58] ...trap/engines/engine.py.229: process: 0.77235 | load: 0.00301 [I 2022-10-08 20:30:58] ...trap/engines/engine.py.230: loss: 0.21548 [I 2022-10-08 20:30:58] ...optimizers/trijoint.py.151: total_norm: 3.202716 [I 2022-10-08 20:31:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 2280/2304 [I 2022-10-08 20:31:05] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:00:18 [I 2022-10-08 20:31:05] ...trap/engines/engine.py.229: process: 0.77369 | load: 0.00301 [I 2022-10-08 20:31:05] ...trap/engines/engine.py.230: loss: 0.13293 [I 2022-10-08 20:31:05] ...optimizers/trijoint.py.151: total_norm: 2.426857 [I 2022-10-08 20:31:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 2290/2304 [I 2022-10-08 20:31:13] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:00:10 [I 2022-10-08 20:31:13] ...trap/engines/engine.py.229: process: 0.78413 | load: 0.00303 [I 2022-10-08 20:31:13] ...trap/engines/engine.py.230: loss: 0.26247 [I 2022-10-08 20:31:13] ...optimizers/trijoint.py.151: total_norm: 3.472915 [I 2022-10-08 20:31:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 2300/2304 [I 2022-10-08 20:31:21] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:00:03 [I 2022-10-08 20:31:21] ...trap/engines/engine.py.229: process: 0.76428 | load: 0.00016 [I 2022-10-08 20:31:21] ...trap/engines/engine.py.230: loss: 0.18422 [I 2022-10-08 20:31:21] ...optimizers/trijoint.py.151: total_norm: 2.724732 [I 2022-10-08 20:31:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 2304/2304 [I 2022-10-08 20:31:24] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:00:00 [I 2022-10-08 20:31:24] ...trap/engines/engine.py.229: process: 0.75849 | load: 0.00017 [I 2022-10-08 20:31:24] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-08 20:31:24] ...optimizers/trijoint.py.151: total_norm: 2.457503 [S 2022-10-08 20:31:25] ...trap/engines/engine.py.129: train_epoch.epoch: 18 [S 2022-10-08 20:31:25] ...trap/engines/engine.py.129: train_epoch.loss: 0.1810537662946997 [I 2022-10-08 20:31:25] ...s/criterions/triplet.py.84: margin = 0.14500000000000007 [I 2022-10-08 20:31:26] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 18 [I 2022-10-08 20:31:26] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 20:31:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 20:31:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 20:31:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 20:31:28] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 20:31:29] ...trap/engines/engine.py.321: eval: epoch 18 | batch 0/510 [I 2022-10-08 20:31:29] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:31 [I 2022-10-08 20:31:29] ...trap/engines/engine.py.326: process: 2.76747 | load: 2.30024 [I 2022-10-08 20:31:35] ...trap/engines/engine.py.321: eval: epoch 18 | batch 10/510 [I 2022-10-08 20:31:35] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:51 [I 2022-10-08 20:31:35] ...trap/engines/engine.py.326: process: 0.54821 | load: 0.00019 [I 2022-10-08 20:31:41] ...trap/engines/engine.py.321: eval: epoch 18 | batch 20/510 [I 2022-10-08 20:31:41] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:02 [I 2022-10-08 20:31:41] ...trap/engines/engine.py.326: process: 0.54372 | load: 0.00019 [I 2022-10-08 20:31:47] ...trap/engines/engine.py.321: eval: epoch 18 | batch 30/510 [I 2022-10-08 20:31:47] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:41 [I 2022-10-08 20:31:47] ...trap/engines/engine.py.326: process: 0.58326 | load: 0.00019 [I 2022-10-08 20:31:52] ...trap/engines/engine.py.321: eval: epoch 18 | batch 40/510 [I 2022-10-08 20:31:52] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:16 [I 2022-10-08 20:31:52] ...trap/engines/engine.py.326: process: 0.48022 | load: 0.00019 [I 2022-10-08 20:31:57] ...trap/engines/engine.py.321: eval: epoch 18 | batch 50/510 [I 2022-10-08 20:31:57] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:55 [I 2022-10-08 20:31:57] ...trap/engines/engine.py.326: process: 0.51255 | load: 0.00018 [I 2022-10-08 20:32:03] ...trap/engines/engine.py.321: eval: epoch 18 | batch 60/510 [I 2022-10-08 20:32:03] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:49 [I 2022-10-08 20:32:03] ...trap/engines/engine.py.326: process: 0.42433 | load: 0.00019 [I 2022-10-08 20:32:07] ...trap/engines/engine.py.321: eval: epoch 18 | batch 70/510 [I 2022-10-08 20:32:07] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:33 [I 2022-10-08 20:32:07] ...trap/engines/engine.py.326: process: 0.57149 | load: 0.00018 [I 2022-10-08 20:32:12] ...trap/engines/engine.py.321: eval: epoch 18 | batch 80/510 [I 2022-10-08 20:32:12] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:49 [I 2022-10-08 20:32:12] ...trap/engines/engine.py.326: process: 0.55430 | load: 0.00019 [I 2022-10-08 20:32:17] ...trap/engines/engine.py.321: eval: epoch 18 | batch 90/510 [I 2022-10-08 20:32:17] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:18 [I 2022-10-08 20:32:17] ...trap/engines/engine.py.326: process: 0.49608 | load: 0.00019 [I 2022-10-08 20:32:22] ...trap/engines/engine.py.321: eval: epoch 18 | batch 100/510 [I 2022-10-08 20:32:22] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:50 [I 2022-10-08 20:32:22] ...trap/engines/engine.py.326: process: 0.31018 | load: 0.00018 [I 2022-10-08 20:32:25] ...trap/engines/engine.py.321: eval: epoch 18 | batch 110/510 [I 2022-10-08 20:32:25] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:11 [I 2022-10-08 20:32:25] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00019 [I 2022-10-08 20:32:28] ...trap/engines/engine.py.321: eval: epoch 18 | batch 120/510 [I 2022-10-08 20:32:28] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:04 [I 2022-10-08 20:32:28] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00020 [I 2022-10-08 20:32:31] ...trap/engines/engine.py.321: eval: epoch 18 | batch 130/510 [I 2022-10-08 20:32:31] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:01 [I 2022-10-08 20:32:31] ...trap/engines/engine.py.326: process: 0.32678 | load: 0.00020 [I 2022-10-08 20:32:34] ...trap/engines/engine.py.321: eval: epoch 18 | batch 140/510 [I 2022-10-08 20:32:34] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:57 [I 2022-10-08 20:32:34] ...trap/engines/engine.py.326: process: 0.31830 | load: 0.00019 [I 2022-10-08 20:32:37] ...trap/engines/engine.py.321: eval: epoch 18 | batch 150/510 [I 2022-10-08 20:32:37] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:54 [I 2022-10-08 20:32:37] ...trap/engines/engine.py.326: process: 0.32450 | load: 0.00021 [I 2022-10-08 20:32:41] ...trap/engines/engine.py.321: eval: epoch 18 | batch 160/510 [I 2022-10-08 20:32:41] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-08 20:32:41] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00021 [I 2022-10-08 20:32:44] ...trap/engines/engine.py.321: eval: epoch 18 | batch 170/510 [I 2022-10-08 20:32:44] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:48 [I 2022-10-08 20:32:44] ...trap/engines/engine.py.326: process: 0.31784 | load: 0.00019 [I 2022-10-08 20:32:47] ...trap/engines/engine.py.321: eval: epoch 18 | batch 180/510 [I 2022-10-08 20:32:47] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:45 [I 2022-10-08 20:32:47] ...trap/engines/engine.py.326: process: 0.32320 | load: 0.00020 [I 2022-10-08 20:32:50] ...trap/engines/engine.py.321: eval: epoch 18 | batch 190/510 [I 2022-10-08 20:32:50] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:41 [I 2022-10-08 20:32:50] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00019 [I 2022-10-08 20:32:53] ...trap/engines/engine.py.321: eval: epoch 18 | batch 200/510 [I 2022-10-08 20:32:53] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-08 20:32:53] ...trap/engines/engine.py.326: process: 0.31358 | load: 0.00020 [I 2022-10-08 20:32:56] ...trap/engines/engine.py.321: eval: epoch 18 | batch 210/510 [I 2022-10-08 20:32:56] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-08 20:32:56] ...trap/engines/engine.py.326: process: 0.31426 | load: 0.00019 [I 2022-10-08 20:33:00] ...trap/engines/engine.py.321: eval: epoch 18 | batch 220/510 [I 2022-10-08 20:33:00] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-08 20:33:00] ...trap/engines/engine.py.326: process: 0.32032 | load: 0.00020 [I 2022-10-08 20:33:03] ...trap/engines/engine.py.321: eval: epoch 18 | batch 230/510 [I 2022-10-08 20:33:03] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:30 [I 2022-10-08 20:33:03] ...trap/engines/engine.py.326: process: 0.32668 | load: 0.00020 [I 2022-10-08 20:33:06] ...trap/engines/engine.py.321: eval: epoch 18 | batch 240/510 [I 2022-10-08 20:33:06] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-08 20:33:06] ...trap/engines/engine.py.326: process: 0.32166 | load: 0.00019 [I 2022-10-08 20:33:09] ...trap/engines/engine.py.321: eval: epoch 18 | batch 250/510 [I 2022-10-08 20:33:09] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:24 [I 2022-10-08 20:33:09] ...trap/engines/engine.py.326: process: 0.32980 | load: 0.00020 [I 2022-10-08 20:33:13] ...trap/engines/engine.py.321: eval: epoch 18 | batch 260/510 [I 2022-10-08 20:33:13] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-08 20:33:13] ...trap/engines/engine.py.326: process: 0.30965 | load: 0.00021 [I 2022-10-08 20:33:16] ...trap/engines/engine.py.321: eval: epoch 18 | batch 270/510 [I 2022-10-08 20:33:16] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:16 [I 2022-10-08 20:33:16] ...trap/engines/engine.py.326: process: 0.31512 | load: 0.00019 [I 2022-10-08 20:33:19] ...trap/engines/engine.py.321: eval: epoch 18 | batch 280/510 [I 2022-10-08 20:33:19] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-08 20:33:19] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00020 [I 2022-10-08 20:33:22] ...trap/engines/engine.py.321: eval: epoch 18 | batch 290/510 [I 2022-10-08 20:33:22] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:10 [I 2022-10-08 20:33:22] ...trap/engines/engine.py.326: process: 0.32063 | load: 0.00020 [I 2022-10-08 20:33:25] ...trap/engines/engine.py.321: eval: epoch 18 | batch 300/510 [I 2022-10-08 20:33:25] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:06 [I 2022-10-08 20:33:25] ...trap/engines/engine.py.326: process: 0.31295 | load: 0.00020 [I 2022-10-08 20:33:29] ...trap/engines/engine.py.321: eval: epoch 18 | batch 310/510 [I 2022-10-08 20:33:29] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-08 20:33:29] ...trap/engines/engine.py.326: process: 0.31483 | load: 0.00020 [I 2022-10-08 20:33:32] ...trap/engines/engine.py.321: eval: epoch 18 | batch 320/510 [I 2022-10-08 20:33:32] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-08 20:33:32] ...trap/engines/engine.py.326: process: 0.31151 | load: 0.00019 [I 2022-10-08 20:33:35] ...trap/engines/engine.py.321: eval: epoch 18 | batch 330/510 [I 2022-10-08 20:33:35] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-08 20:33:35] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00019 [I 2022-10-08 20:33:38] ...trap/engines/engine.py.321: eval: epoch 18 | batch 340/510 [I 2022-10-08 20:33:38] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-08 20:33:38] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00021 [I 2022-10-08 20:33:41] ...trap/engines/engine.py.321: eval: epoch 18 | batch 350/510 [I 2022-10-08 20:33:41] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-08 20:33:41] ...trap/engines/engine.py.326: process: 0.32455 | load: 0.00022 [I 2022-10-08 20:33:45] ...trap/engines/engine.py.321: eval: epoch 18 | batch 360/510 [I 2022-10-08 20:33:45] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-08 20:33:45] ...trap/engines/engine.py.326: process: 0.32187 | load: 0.00020 [I 2022-10-08 20:33:48] ...trap/engines/engine.py.321: eval: epoch 18 | batch 370/510 [I 2022-10-08 20:33:48] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:45 [I 2022-10-08 20:33:48] ...trap/engines/engine.py.326: process: 0.33164 | load: 0.00020 [I 2022-10-08 20:33:51] ...trap/engines/engine.py.321: eval: epoch 18 | batch 380/510 [I 2022-10-08 20:33:51] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-08 20:33:51] ...trap/engines/engine.py.326: process: 0.32159 | load: 0.00020 [I 2022-10-08 20:33:54] ...trap/engines/engine.py.321: eval: epoch 18 | batch 390/510 [I 2022-10-08 20:33:54] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-08 20:33:54] ...trap/engines/engine.py.326: process: 0.32916 | load: 0.00020 [I 2022-10-08 20:33:58] ...trap/engines/engine.py.321: eval: epoch 18 | batch 400/510 [I 2022-10-08 20:33:58] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-08 20:33:58] ...trap/engines/engine.py.326: process: 0.32037 | load: 0.00024 [I 2022-10-08 20:34:01] ...trap/engines/engine.py.321: eval: epoch 18 | batch 410/510 [I 2022-10-08 20:34:01] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-08 20:34:01] ...trap/engines/engine.py.326: process: 0.32518 | load: 0.00020 [I 2022-10-08 20:34:04] ...trap/engines/engine.py.321: eval: epoch 18 | batch 420/510 [I 2022-10-08 20:34:04] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-08 20:34:04] ...trap/engines/engine.py.326: process: 0.31390 | load: 0.00021 [I 2022-10-08 20:34:07] ...trap/engines/engine.py.321: eval: epoch 18 | batch 430/510 [I 2022-10-08 20:34:07] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-08 20:34:07] ...trap/engines/engine.py.326: process: 0.32230 | load: 0.00020 [I 2022-10-08 20:34:10] ...trap/engines/engine.py.321: eval: epoch 18 | batch 440/510 [I 2022-10-08 20:34:10] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-08 20:34:10] ...trap/engines/engine.py.326: process: 0.32061 | load: 0.00019 [I 2022-10-08 20:34:13] ...trap/engines/engine.py.321: eval: epoch 18 | batch 450/510 [I 2022-10-08 20:34:13] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:18 [I 2022-10-08 20:34:13] ...trap/engines/engine.py.326: process: 0.31178 | load: 0.00020 [I 2022-10-08 20:34:17] ...trap/engines/engine.py.321: eval: epoch 18 | batch 460/510 [I 2022-10-08 20:34:17] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-08 20:34:17] ...trap/engines/engine.py.326: process: 0.32027 | load: 0.00021 [I 2022-10-08 20:34:20] ...trap/engines/engine.py.321: eval: epoch 18 | batch 470/510 [I 2022-10-08 20:34:20] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-08 20:34:20] ...trap/engines/engine.py.326: process: 0.31405 | load: 0.00020 [I 2022-10-08 20:34:23] ...trap/engines/engine.py.321: eval: epoch 18 | batch 480/510 [I 2022-10-08 20:34:23] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-08 20:34:23] ...trap/engines/engine.py.326: process: 0.32009 | load: 0.00020 [I 2022-10-08 20:34:26] ...trap/engines/engine.py.321: eval: epoch 18 | batch 490/510 [I 2022-10-08 20:34:26] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-08 20:34:26] ...trap/engines/engine.py.326: process: 0.31491 | load: 0.00021 [I 2022-10-08 20:34:29] ...trap/engines/engine.py.321: eval: epoch 18 | batch 500/510 [I 2022-10-08 20:34:29] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-08 20:34:29] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00021 [I 2022-10-08 20:34:33] ...trap/engines/engine.py.321: eval: epoch 18 | batch 510/510 [I 2022-10-08 20:34:33] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-08 20:34:33] ...trap/engines/engine.py.326: process: 0.30294 | load: 0.00016 [S 2022-10-08 20:34:33] ...trap/engines/engine.py.132: eval_epoch.epoch: 18 [I 2022-10-08 20:34:33] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.616000 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.859600 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.908800 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.611800 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.858100 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.908400 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012791 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006681 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009652 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015191 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006580 [S 2022-10-08 20:34:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006785 [I 2022-10-08 20:35:13] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 20:35:13] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 20:35:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 20:35:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 20:35:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 20:35:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 20:35:16] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 20:35:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 20:35:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 20:35:21] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 20:35:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 20:35:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 20:35:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 19 [I 2022-10-08 20:35:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 20:35:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 0/2304 [I 2022-10-08 20:35:31] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:40 [I 2022-10-08 20:35:31] ...trap/engines/engine.py.229: process: 3.01251 | load: 2.20564 [I 2022-10-08 20:35:31] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-08 20:35:31] ...optimizers/trijoint.py.151: total_norm: 2.361593 [I 2022-10-08 20:35:39] ...trap/engines/engine.py.224: train: epoch 19 | batch 10/2304 [I 2022-10-08 20:35:39] ...trap/engines/engine.py.228: elapsed: 0:00:10 | left: 0:38:56 [I 2022-10-08 20:35:39] ...trap/engines/engine.py.229: process: 0.77286 | load: 0.00596 [I 2022-10-08 20:35:39] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-08 20:35:39] ...optimizers/trijoint.py.151: total_norm: 2.672578 [I 2022-10-08 20:35:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 20/2304 [I 2022-10-08 20:35:47] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:30:39 [I 2022-10-08 20:35:47] ...trap/engines/engine.py.229: process: 0.77872 | load: 0.00499 [I 2022-10-08 20:35:47] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-08 20:35:47] ...optimizers/trijoint.py.151: total_norm: 2.043079 [I 2022-10-08 20:35:55] ...trap/engines/engine.py.224: train: epoch 19 | batch 30/2304 [I 2022-10-08 20:35:55] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 0:29:43 [I 2022-10-08 20:35:55] ...trap/engines/engine.py.229: process: 0.79042 | load: 0.00363 [I 2022-10-08 20:35:55] ...trap/engines/engine.py.230: loss: 0.19699 [I 2022-10-08 20:35:55] ...optimizers/trijoint.py.151: total_norm: 3.183669 [I 2022-10-08 20:36:03] ...trap/engines/engine.py.224: train: epoch 19 | batch 40/2304 [I 2022-10-08 20:36:03] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:29:30 [I 2022-10-08 20:36:03] ...trap/engines/engine.py.229: process: 0.78494 | load: 0.00378 [I 2022-10-08 20:36:03] ...trap/engines/engine.py.230: loss: 0.26972 [I 2022-10-08 20:36:03] ...optimizers/trijoint.py.151: total_norm: 4.034727 [I 2022-10-08 20:36:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 50/2304 [I 2022-10-08 20:36:10] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:29:22 [I 2022-10-08 20:36:10] ...trap/engines/engine.py.229: process: 0.78138 | load: 0.00361 [I 2022-10-08 20:36:10] ...trap/engines/engine.py.230: loss: 0.18873 [I 2022-10-08 20:36:10] ...optimizers/trijoint.py.151: total_norm: 2.601409 [I 2022-10-08 20:36:18] ...trap/engines/engine.py.224: train: epoch 19 | batch 60/2304 [I 2022-10-08 20:36:18] ...trap/engines/engine.py.228: elapsed: 0:00:50 | left: 0:29:11 [I 2022-10-08 20:36:18] ...trap/engines/engine.py.229: process: 0.78632 | load: 0.00357 [I 2022-10-08 20:36:18] ...trap/engines/engine.py.230: loss: 0.14352 [I 2022-10-08 20:36:18] ...optimizers/trijoint.py.151: total_norm: 2.745032 [I 2022-10-08 20:36:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 70/2304 [I 2022-10-08 20:36:26] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:28:57 [I 2022-10-08 20:36:26] ...trap/engines/engine.py.229: process: 0.77676 | load: 0.00340 [I 2022-10-08 20:36:26] ...trap/engines/engine.py.230: loss: 0.11719 [I 2022-10-08 20:36:26] ...optimizers/trijoint.py.151: total_norm: 2.381033 [I 2022-10-08 20:36:34] ...trap/engines/engine.py.224: train: epoch 19 | batch 80/2304 [I 2022-10-08 20:36:34] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:28:45 [I 2022-10-08 20:36:34] ...trap/engines/engine.py.229: process: 0.78005 | load: 0.00383 [I 2022-10-08 20:36:34] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-08 20:36:34] ...optimizers/trijoint.py.151: total_norm: 2.394070 [I 2022-10-08 20:36:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 90/2304 [I 2022-10-08 20:36:42] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:28:28 [I 2022-10-08 20:36:42] ...trap/engines/engine.py.229: process: 0.75520 | load: 0.00359 [I 2022-10-08 20:36:42] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-08 20:36:42] ...optimizers/trijoint.py.151: total_norm: 1.962037 [I 2022-10-08 20:36:50] ...trap/engines/engine.py.224: train: epoch 19 | batch 100/2304 [I 2022-10-08 20:36:50] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:28:35 [I 2022-10-08 20:36:50] ...trap/engines/engine.py.229: process: 0.77292 | load: 0.00367 [I 2022-10-08 20:36:50] ...trap/engines/engine.py.230: loss: 0.20028 [I 2022-10-08 20:36:50] ...optimizers/trijoint.py.151: total_norm: 3.357955 [I 2022-10-08 20:36:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 110/2304 [I 2022-10-08 20:36:57] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:28:28 [I 2022-10-08 20:36:57] ...trap/engines/engine.py.229: process: 0.77405 | load: 0.00298 [I 2022-10-08 20:36:57] ...trap/engines/engine.py.230: loss: 0.26248 [I 2022-10-08 20:36:57] ...optimizers/trijoint.py.151: total_norm: 4.140087 [I 2022-10-08 20:37:05] ...trap/engines/engine.py.224: train: epoch 19 | batch 120/2304 [I 2022-10-08 20:37:05] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:28:19 [I 2022-10-08 20:37:05] ...trap/engines/engine.py.229: process: 0.77002 | load: 0.00340 [I 2022-10-08 20:37:05] ...trap/engines/engine.py.230: loss: 0.20555 [I 2022-10-08 20:37:05] ...optimizers/trijoint.py.151: total_norm: 2.846087 [I 2022-10-08 20:37:13] ...trap/engines/engine.py.224: train: epoch 19 | batch 130/2304 [I 2022-10-08 20:37:13] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:28:08 [I 2022-10-08 20:37:13] ...trap/engines/engine.py.229: process: 0.77014 | load: 0.00321 [I 2022-10-08 20:37:13] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-08 20:37:13] ...optimizers/trijoint.py.151: total_norm: 2.943402 [I 2022-10-08 20:37:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 140/2304 [I 2022-10-08 20:37:21] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:28:08 [I 2022-10-08 20:37:21] ...trap/engines/engine.py.229: process: 0.77822 | load: 0.00302 [I 2022-10-08 20:37:21] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-08 20:37:21] ...optimizers/trijoint.py.151: total_norm: 3.192108 [I 2022-10-08 20:37:29] ...trap/engines/engine.py.224: train: epoch 19 | batch 150/2304 [I 2022-10-08 20:37:29] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:27:49 [I 2022-10-08 20:37:29] ...trap/engines/engine.py.229: process: 0.76409 | load: 0.00302 [I 2022-10-08 20:37:29] ...trap/engines/engine.py.230: loss: 0.13490 [I 2022-10-08 20:37:29] ...optimizers/trijoint.py.151: total_norm: 2.097613 [I 2022-10-08 20:37:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 160/2304 [I 2022-10-08 20:37:36] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:27:55 [I 2022-10-08 20:37:36] ...trap/engines/engine.py.229: process: 0.78264 | load: 0.00322 [I 2022-10-08 20:37:36] ...trap/engines/engine.py.230: loss: 0.21793 [I 2022-10-08 20:37:36] ...optimizers/trijoint.py.151: total_norm: 2.572237 [I 2022-10-08 20:37:44] ...trap/engines/engine.py.224: train: epoch 19 | batch 170/2304 [I 2022-10-08 20:37:44] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:27:44 [I 2022-10-08 20:37:44] ...trap/engines/engine.py.229: process: 0.79445 | load: 0.00300 [I 2022-10-08 20:37:44] ...trap/engines/engine.py.230: loss: 0.14505 [I 2022-10-08 20:37:44] ...optimizers/trijoint.py.151: total_norm: 2.260702 [I 2022-10-08 20:37:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 180/2304 [I 2022-10-08 20:37:52] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:27:31 [I 2022-10-08 20:37:52] ...trap/engines/engine.py.229: process: 0.77687 | load: 0.00337 [I 2022-10-08 20:37:52] ...trap/engines/engine.py.230: loss: 0.18447 [I 2022-10-08 20:37:52] ...optimizers/trijoint.py.151: total_norm: 2.399476 [I 2022-10-08 20:38:00] ...trap/engines/engine.py.224: train: epoch 19 | batch 190/2304 [I 2022-10-08 20:38:00] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:27:31 [I 2022-10-08 20:38:00] ...trap/engines/engine.py.229: process: 0.77933 | load: 0.00309 [I 2022-10-08 20:38:00] ...trap/engines/engine.py.230: loss: 0.18579 [I 2022-10-08 20:38:00] ...optimizers/trijoint.py.151: total_norm: 2.046416 [I 2022-10-08 20:38:08] ...trap/engines/engine.py.224: train: epoch 19 | batch 200/2304 [I 2022-10-08 20:38:08] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:27:24 [I 2022-10-08 20:38:08] ...trap/engines/engine.py.229: process: 0.78217 | load: 0.00309 [I 2022-10-08 20:38:08] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-08 20:38:08] ...optimizers/trijoint.py.151: total_norm: 2.274797 [I 2022-10-08 20:38:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 210/2304 [I 2022-10-08 20:38:16] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:27:13 [I 2022-10-08 20:38:16] ...trap/engines/engine.py.229: process: 0.78394 | load: 0.00333 [I 2022-10-08 20:38:16] ...trap/engines/engine.py.230: loss: 0.12454 [I 2022-10-08 20:38:16] ...optimizers/trijoint.py.151: total_norm: 2.298241 [I 2022-10-08 20:38:23] ...trap/engines/engine.py.224: train: epoch 19 | batch 220/2304 [I 2022-10-08 20:38:23] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:27:07 [I 2022-10-08 20:38:23] ...trap/engines/engine.py.229: process: 0.77498 | load: 0.00315 [I 2022-10-08 20:38:23] ...trap/engines/engine.py.230: loss: 0.26201 [I 2022-10-08 20:38:23] ...optimizers/trijoint.py.151: total_norm: 3.625315 [I 2022-10-08 20:38:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 230/2304 [I 2022-10-08 20:38:31] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:26:54 [I 2022-10-08 20:38:31] ...trap/engines/engine.py.229: process: 0.77901 | load: 0.00308 [I 2022-10-08 20:38:31] ...trap/engines/engine.py.230: loss: 0.13347 [I 2022-10-08 20:38:31] ...optimizers/trijoint.py.151: total_norm: 2.291768 [I 2022-10-08 20:38:39] ...trap/engines/engine.py.224: train: epoch 19 | batch 240/2304 [I 2022-10-08 20:38:39] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:26:44 [I 2022-10-08 20:38:39] ...trap/engines/engine.py.229: process: 0.77232 | load: 0.00331 [I 2022-10-08 20:38:39] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-08 20:38:39] ...optimizers/trijoint.py.151: total_norm: 2.822764 [I 2022-10-08 20:38:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 250/2304 [I 2022-10-08 20:38:47] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:26:46 [I 2022-10-08 20:38:47] ...trap/engines/engine.py.229: process: 0.78109 | load: 0.00304 [I 2022-10-08 20:38:47] ...trap/engines/engine.py.230: loss: 0.20519 [I 2022-10-08 20:38:47] ...optimizers/trijoint.py.151: total_norm: 3.495214 [I 2022-10-08 20:38:55] ...trap/engines/engine.py.224: train: epoch 19 | batch 260/2304 [I 2022-10-08 20:38:55] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:26:29 [I 2022-10-08 20:38:55] ...trap/engines/engine.py.229: process: 0.76518 | load: 0.00306 [I 2022-10-08 20:38:55] ...trap/engines/engine.py.230: loss: 0.18140 [I 2022-10-08 20:38:55] ...optimizers/trijoint.py.151: total_norm: 3.126743 [I 2022-10-08 20:39:03] ...trap/engines/engine.py.224: train: epoch 19 | batch 270/2304 [I 2022-10-08 20:39:03] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:26:24 [I 2022-10-08 20:39:03] ...trap/engines/engine.py.229: process: 0.78251 | load: 0.00300 [I 2022-10-08 20:39:03] ...trap/engines/engine.py.230: loss: 0.14299 [I 2022-10-08 20:39:03] ...optimizers/trijoint.py.151: total_norm: 1.565029 [I 2022-10-08 20:39:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 280/2304 [I 2022-10-08 20:39:10] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:26:22 [I 2022-10-08 20:39:10] ...trap/engines/engine.py.229: process: 0.78730 | load: 0.00300 [I 2022-10-08 20:39:10] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-08 20:39:10] ...optimizers/trijoint.py.151: total_norm: 1.940367 [I 2022-10-08 20:39:18] ...trap/engines/engine.py.224: train: epoch 19 | batch 290/2304 [I 2022-10-08 20:39:18] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:26:13 [I 2022-10-08 20:39:18] ...trap/engines/engine.py.229: process: 0.78814 | load: 0.00327 [I 2022-10-08 20:39:18] ...trap/engines/engine.py.230: loss: 0.13155 [I 2022-10-08 20:39:18] ...optimizers/trijoint.py.151: total_norm: 1.590063 [I 2022-10-08 20:39:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 300/2304 [I 2022-10-08 20:39:26] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:26:02 [I 2022-10-08 20:39:26] ...trap/engines/engine.py.229: process: 0.78035 | load: 0.00302 [I 2022-10-08 20:39:26] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-08 20:39:26] ...optimizers/trijoint.py.151: total_norm: 2.669647 [I 2022-10-08 20:39:34] ...trap/engines/engine.py.224: train: epoch 19 | batch 310/2304 [I 2022-10-08 20:39:34] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:25:53 [I 2022-10-08 20:39:34] ...trap/engines/engine.py.229: process: 0.77546 | load: 0.00304 [I 2022-10-08 20:39:34] ...trap/engines/engine.py.230: loss: 0.23384 [I 2022-10-08 20:39:34] ...optimizers/trijoint.py.151: total_norm: 3.098482 [I 2022-10-08 20:39:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 320/2304 [I 2022-10-08 20:39:42] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:25:43 [I 2022-10-08 20:39:42] ...trap/engines/engine.py.229: process: 0.77966 | load: 0.00304 [I 2022-10-08 20:39:42] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-08 20:39:42] ...optimizers/trijoint.py.151: total_norm: 3.147320 [I 2022-10-08 20:39:50] ...trap/engines/engine.py.224: train: epoch 19 | batch 330/2304 [I 2022-10-08 20:39:50] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:25:45 [I 2022-10-08 20:39:50] ...trap/engines/engine.py.229: process: 0.78066 | load: 0.00336 [I 2022-10-08 20:39:50] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-08 20:39:50] ...optimizers/trijoint.py.151: total_norm: 2.040638 [I 2022-10-08 20:39:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 340/2304 [I 2022-10-08 20:39:57] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:25:30 [I 2022-10-08 20:39:57] ...trap/engines/engine.py.229: process: 0.75904 | load: 0.00298 [I 2022-10-08 20:39:57] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-08 20:39:57] ...optimizers/trijoint.py.151: total_norm: 3.221594 [I 2022-10-08 20:40:05] ...trap/engines/engine.py.224: train: epoch 19 | batch 350/2304 [I 2022-10-08 20:40:05] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:25:21 [I 2022-10-08 20:40:05] ...trap/engines/engine.py.229: process: 0.78047 | load: 0.00302 [I 2022-10-08 20:40:05] ...trap/engines/engine.py.230: loss: 0.18307 [I 2022-10-08 20:40:05] ...optimizers/trijoint.py.151: total_norm: 2.685440 [I 2022-10-08 20:40:13] ...trap/engines/engine.py.224: train: epoch 19 | batch 360/2304 [I 2022-10-08 20:40:13] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:25:14 [I 2022-10-08 20:40:13] ...trap/engines/engine.py.229: process: 0.78256 | load: 0.00307 [I 2022-10-08 20:40:13] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-08 20:40:13] ...optimizers/trijoint.py.151: total_norm: 2.015713 [I 2022-10-08 20:40:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 370/2304 [I 2022-10-08 20:40:21] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:24:58 [I 2022-10-08 20:40:21] ...trap/engines/engine.py.229: process: 0.78133 | load: 0.00301 [I 2022-10-08 20:40:21] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-08 20:40:21] ...optimizers/trijoint.py.151: total_norm: 3.369671 [I 2022-10-08 20:40:28] ...trap/engines/engine.py.224: train: epoch 19 | batch 380/2304 [I 2022-10-08 20:40:28] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:24:55 [I 2022-10-08 20:40:28] ...trap/engines/engine.py.229: process: 0.78868 | load: 0.00312 [I 2022-10-08 20:40:28] ...trap/engines/engine.py.230: loss: 0.23855 [I 2022-10-08 20:40:28] ...optimizers/trijoint.py.151: total_norm: 3.245523 [I 2022-10-08 20:40:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 390/2304 [I 2022-10-08 20:40:36] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:24:50 [I 2022-10-08 20:40:36] ...trap/engines/engine.py.229: process: 0.77621 | load: 0.00302 [I 2022-10-08 20:40:36] ...trap/engines/engine.py.230: loss: 0.17523 [I 2022-10-08 20:40:36] ...optimizers/trijoint.py.151: total_norm: 2.611830 [I 2022-10-08 20:40:44] ...trap/engines/engine.py.224: train: epoch 19 | batch 400/2304 [I 2022-10-08 20:40:44] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:24:48 [I 2022-10-08 20:40:44] ...trap/engines/engine.py.229: process: 0.78226 | load: 0.00313 [I 2022-10-08 20:40:44] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-08 20:40:44] ...optimizers/trijoint.py.151: total_norm: 2.379784 [I 2022-10-08 20:40:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 410/2304 [I 2022-10-08 20:40:52] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:24:28 [I 2022-10-08 20:40:52] ...trap/engines/engine.py.229: process: 0.76951 | load: 0.00302 [I 2022-10-08 20:40:52] ...trap/engines/engine.py.230: loss: 0.21560 [I 2022-10-08 20:40:52] ...optimizers/trijoint.py.151: total_norm: 3.174581 [I 2022-10-08 20:41:00] ...trap/engines/engine.py.224: train: epoch 19 | batch 420/2304 [I 2022-10-08 20:41:00] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:24:36 [I 2022-10-08 20:41:00] ...trap/engines/engine.py.229: process: 0.77921 | load: 0.00302 [I 2022-10-08 20:41:00] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-08 20:41:00] ...optimizers/trijoint.py.151: total_norm: 2.561832 [I 2022-10-08 20:41:08] ...trap/engines/engine.py.224: train: epoch 19 | batch 430/2304 [I 2022-10-08 20:41:08] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:24:20 [I 2022-10-08 20:41:08] ...trap/engines/engine.py.229: process: 0.77107 | load: 0.00309 [I 2022-10-08 20:41:08] ...trap/engines/engine.py.230: loss: 0.15246 [I 2022-10-08 20:41:08] ...optimizers/trijoint.py.151: total_norm: 2.684749 [I 2022-10-08 20:41:15] ...trap/engines/engine.py.224: train: epoch 19 | batch 440/2304 [I 2022-10-08 20:41:15] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:24:12 [I 2022-10-08 20:41:15] ...trap/engines/engine.py.229: process: 0.77466 | load: 0.00308 [I 2022-10-08 20:41:15] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-08 20:41:15] ...optimizers/trijoint.py.151: total_norm: 2.030676 [I 2022-10-08 20:41:23] ...trap/engines/engine.py.224: train: epoch 19 | batch 450/2304 [I 2022-10-08 20:41:23] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:24:00 [I 2022-10-08 20:41:23] ...trap/engines/engine.py.229: process: 0.76081 | load: 0.00311 [I 2022-10-08 20:41:23] ...trap/engines/engine.py.230: loss: 0.19917 [I 2022-10-08 20:41:23] ...optimizers/trijoint.py.151: total_norm: 2.626531 [I 2022-10-08 20:41:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 460/2304 [I 2022-10-08 20:41:31] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:23:52 [I 2022-10-08 20:41:31] ...trap/engines/engine.py.229: process: 0.77785 | load: 0.00309 [I 2022-10-08 20:41:31] ...trap/engines/engine.py.230: loss: 0.15248 [I 2022-10-08 20:41:31] ...optimizers/trijoint.py.151: total_norm: 2.246801 [I 2022-10-08 20:41:39] ...trap/engines/engine.py.224: train: epoch 19 | batch 470/2304 [I 2022-10-08 20:41:39] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:23:48 [I 2022-10-08 20:41:39] ...trap/engines/engine.py.229: process: 0.79097 | load: 0.00310 [I 2022-10-08 20:41:39] ...trap/engines/engine.py.230: loss: 0.22762 [I 2022-10-08 20:41:39] ...optimizers/trijoint.py.151: total_norm: 3.676105 [I 2022-10-08 20:41:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 480/2304 [I 2022-10-08 20:41:47] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:23:44 [I 2022-10-08 20:41:47] ...trap/engines/engine.py.229: process: 0.77394 | load: 0.00310 [I 2022-10-08 20:41:47] ...trap/engines/engine.py.230: loss: 0.22746 [I 2022-10-08 20:41:47] ...optimizers/trijoint.py.151: total_norm: 3.564836 [I 2022-10-08 20:41:55] ...trap/engines/engine.py.224: train: epoch 19 | batch 490/2304 [I 2022-10-08 20:41:55] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:23:37 [I 2022-10-08 20:41:55] ...trap/engines/engine.py.229: process: 0.77752 | load: 0.00313 [I 2022-10-08 20:41:55] ...trap/engines/engine.py.230: loss: 0.22187 [I 2022-10-08 20:41:55] ...optimizers/trijoint.py.151: total_norm: 3.039089 [I 2022-10-08 20:42:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 500/2304 [I 2022-10-08 20:42:02] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:23:21 [I 2022-10-08 20:42:02] ...trap/engines/engine.py.229: process: 0.77812 | load: 0.00306 [I 2022-10-08 20:42:02] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-08 20:42:02] ...optimizers/trijoint.py.151: total_norm: 2.840123 [I 2022-10-08 20:42:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 510/2304 [I 2022-10-08 20:42:10] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:23:18 [I 2022-10-08 20:42:10] ...trap/engines/engine.py.229: process: 0.78520 | load: 0.00301 [I 2022-10-08 20:42:10] ...trap/engines/engine.py.230: loss: 0.19894 [I 2022-10-08 20:42:10] ...optimizers/trijoint.py.151: total_norm: 2.955319 [I 2022-10-08 20:42:18] ...trap/engines/engine.py.224: train: epoch 19 | batch 520/2304 [I 2022-10-08 20:42:18] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:23:12 [I 2022-10-08 20:42:18] ...trap/engines/engine.py.229: process: 0.80050 | load: 0.00301 [I 2022-10-08 20:42:18] ...trap/engines/engine.py.230: loss: 0.17080 [I 2022-10-08 20:42:18] ...optimizers/trijoint.py.151: total_norm: 2.310340 [I 2022-10-08 20:42:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 530/2304 [I 2022-10-08 20:42:26] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:23:09 [I 2022-10-08 20:42:26] ...trap/engines/engine.py.229: process: 0.79168 | load: 0.00314 [I 2022-10-08 20:42:26] ...trap/engines/engine.py.230: loss: 0.24299 [I 2022-10-08 20:42:26] ...optimizers/trijoint.py.151: total_norm: 2.973194 [I 2022-10-08 20:42:34] ...trap/engines/engine.py.224: train: epoch 19 | batch 540/2304 [I 2022-10-08 20:42:34] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:23:11 [I 2022-10-08 20:42:34] ...trap/engines/engine.py.229: process: 0.79307 | load: 0.00302 [I 2022-10-08 20:42:34] ...trap/engines/engine.py.230: loss: 0.22433 [I 2022-10-08 20:42:34] ...optimizers/trijoint.py.151: total_norm: 2.962139 [I 2022-10-08 20:42:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 550/2304 [I 2022-10-08 20:42:42] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:22:43 [I 2022-10-08 20:42:42] ...trap/engines/engine.py.229: process: 0.77029 | load: 0.00303 [I 2022-10-08 20:42:42] ...trap/engines/engine.py.230: loss: 0.17209 [I 2022-10-08 20:42:42] ...optimizers/trijoint.py.151: total_norm: 3.899684 [I 2022-10-08 20:42:49] ...trap/engines/engine.py.224: train: epoch 19 | batch 560/2304 [I 2022-10-08 20:42:49] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:22:40 [I 2022-10-08 20:42:49] ...trap/engines/engine.py.229: process: 0.78264 | load: 0.00299 [I 2022-10-08 20:42:49] ...trap/engines/engine.py.230: loss: 0.16611 [I 2022-10-08 20:42:49] ...optimizers/trijoint.py.151: total_norm: 2.609055 [I 2022-10-08 20:42:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 570/2304 [I 2022-10-08 20:42:57] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:22:34 [I 2022-10-08 20:42:57] ...trap/engines/engine.py.229: process: 0.77388 | load: 0.00302 [I 2022-10-08 20:42:57] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-08 20:42:57] ...optimizers/trijoint.py.151: total_norm: 2.564399 [I 2022-10-08 20:43:05] ...trap/engines/engine.py.224: train: epoch 19 | batch 580/2304 [I 2022-10-08 20:43:05] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:22:21 [I 2022-10-08 20:43:05] ...trap/engines/engine.py.229: process: 0.78691 | load: 0.00309 [I 2022-10-08 20:43:05] ...trap/engines/engine.py.230: loss: 0.11902 [I 2022-10-08 20:43:05] ...optimizers/trijoint.py.151: total_norm: 2.023296 [I 2022-10-08 20:43:13] ...trap/engines/engine.py.224: train: epoch 19 | batch 590/2304 [I 2022-10-08 20:43:13] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:22:10 [I 2022-10-08 20:43:13] ...trap/engines/engine.py.229: process: 0.78792 | load: 0.00307 [I 2022-10-08 20:43:13] ...trap/engines/engine.py.230: loss: 0.12085 [I 2022-10-08 20:43:13] ...optimizers/trijoint.py.151: total_norm: 2.023899 [I 2022-10-08 20:43:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 600/2304 [I 2022-10-08 20:43:21] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:22:14 [I 2022-10-08 20:43:21] ...trap/engines/engine.py.229: process: 0.78316 | load: 0.00309 [I 2022-10-08 20:43:21] ...trap/engines/engine.py.230: loss: 0.21233 [I 2022-10-08 20:43:21] ...optimizers/trijoint.py.151: total_norm: 4.370496 [I 2022-10-08 20:43:28] ...trap/engines/engine.py.224: train: epoch 19 | batch 610/2304 [I 2022-10-08 20:43:28] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:22:04 [I 2022-10-08 20:43:28] ...trap/engines/engine.py.229: process: 0.77829 | load: 0.00300 [I 2022-10-08 20:43:28] ...trap/engines/engine.py.230: loss: 0.15193 [I 2022-10-08 20:43:28] ...optimizers/trijoint.py.151: total_norm: 3.148677 [I 2022-10-08 20:43:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 620/2304 [I 2022-10-08 20:43:36] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:21:52 [I 2022-10-08 20:43:36] ...trap/engines/engine.py.229: process: 0.77996 | load: 0.00305 [I 2022-10-08 20:43:36] ...trap/engines/engine.py.230: loss: 0.16969 [I 2022-10-08 20:43:36] ...optimizers/trijoint.py.151: total_norm: 2.685206 [I 2022-10-08 20:43:44] ...trap/engines/engine.py.224: train: epoch 19 | batch 630/2304 [I 2022-10-08 20:43:44] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:21:41 [I 2022-10-08 20:43:44] ...trap/engines/engine.py.229: process: 0.78064 | load: 0.00303 [I 2022-10-08 20:43:44] ...trap/engines/engine.py.230: loss: 0.22872 [I 2022-10-08 20:43:44] ...optimizers/trijoint.py.151: total_norm: 3.403762 [I 2022-10-08 20:43:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 640/2304 [I 2022-10-08 20:43:52] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:21:37 [I 2022-10-08 20:43:52] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00304 [I 2022-10-08 20:43:52] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-08 20:43:52] ...optimizers/trijoint.py.151: total_norm: 3.525135 [I 2022-10-08 20:44:00] ...trap/engines/engine.py.224: train: epoch 19 | batch 650/2304 [I 2022-10-08 20:44:00] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:21:22 [I 2022-10-08 20:44:00] ...trap/engines/engine.py.229: process: 0.77242 | load: 0.00304 [I 2022-10-08 20:44:00] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-08 20:44:00] ...optimizers/trijoint.py.151: total_norm: 2.482785 [I 2022-10-08 20:44:07] ...trap/engines/engine.py.224: train: epoch 19 | batch 660/2304 [I 2022-10-08 20:44:07] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:21:20 [I 2022-10-08 20:44:07] ...trap/engines/engine.py.229: process: 0.79141 | load: 0.00298 [I 2022-10-08 20:44:07] ...trap/engines/engine.py.230: loss: 0.18286 [I 2022-10-08 20:44:07] ...optimizers/trijoint.py.151: total_norm: 2.614975 [I 2022-10-08 20:44:15] ...trap/engines/engine.py.224: train: epoch 19 | batch 670/2304 [I 2022-10-08 20:44:15] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:21:21 [I 2022-10-08 20:44:15] ...trap/engines/engine.py.229: process: 0.78264 | load: 0.00305 [I 2022-10-08 20:44:15] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-08 20:44:15] ...optimizers/trijoint.py.151: total_norm: 1.993686 [I 2022-10-08 20:44:23] ...trap/engines/engine.py.224: train: epoch 19 | batch 680/2304 [I 2022-10-08 20:44:23] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:21:11 [I 2022-10-08 20:44:23] ...trap/engines/engine.py.229: process: 0.77841 | load: 0.00308 [I 2022-10-08 20:44:23] ...trap/engines/engine.py.230: loss: 0.15601 [I 2022-10-08 20:44:23] ...optimizers/trijoint.py.151: total_norm: 3.089009 [I 2022-10-08 20:44:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 690/2304 [I 2022-10-08 20:44:31] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:20:59 [I 2022-10-08 20:44:31] ...trap/engines/engine.py.229: process: 0.77767 | load: 0.00308 [I 2022-10-08 20:44:31] ...trap/engines/engine.py.230: loss: 0.20906 [I 2022-10-08 20:44:31] ...optimizers/trijoint.py.151: total_norm: 2.753177 [I 2022-10-08 20:44:39] ...trap/engines/engine.py.224: train: epoch 19 | batch 700/2304 [I 2022-10-08 20:44:39] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:20:46 [I 2022-10-08 20:44:39] ...trap/engines/engine.py.229: process: 0.78496 | load: 0.00299 [I 2022-10-08 20:44:39] ...trap/engines/engine.py.230: loss: 0.16963 [I 2022-10-08 20:44:39] ...optimizers/trijoint.py.151: total_norm: 2.518995 [I 2022-10-08 20:44:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 710/2304 [I 2022-10-08 20:44:46] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:20:42 [I 2022-10-08 20:44:46] ...trap/engines/engine.py.229: process: 0.78225 | load: 0.00303 [I 2022-10-08 20:44:46] ...trap/engines/engine.py.230: loss: 0.19406 [I 2022-10-08 20:44:46] ...optimizers/trijoint.py.151: total_norm: 2.812441 [I 2022-10-08 20:44:54] ...trap/engines/engine.py.224: train: epoch 19 | batch 720/2304 [I 2022-10-08 20:44:54] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:20:36 [I 2022-10-08 20:44:54] ...trap/engines/engine.py.229: process: 0.78388 | load: 0.00301 [I 2022-10-08 20:44:54] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-08 20:44:54] ...optimizers/trijoint.py.151: total_norm: 2.291746 [I 2022-10-08 20:45:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 730/2304 [I 2022-10-08 20:45:02] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:20:28 [I 2022-10-08 20:45:02] ...trap/engines/engine.py.229: process: 0.79133 | load: 0.00301 [I 2022-10-08 20:45:02] ...trap/engines/engine.py.230: loss: 0.18212 [I 2022-10-08 20:45:02] ...optimizers/trijoint.py.151: total_norm: 2.532436 [I 2022-10-08 20:45:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 740/2304 [I 2022-10-08 20:45:10] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:20:17 [I 2022-10-08 20:45:10] ...trap/engines/engine.py.229: process: 0.78104 | load: 0.00301 [I 2022-10-08 20:45:10] ...trap/engines/engine.py.230: loss: 0.25139 [I 2022-10-08 20:45:10] ...optimizers/trijoint.py.151: total_norm: 3.237062 [I 2022-10-08 20:45:18] ...trap/engines/engine.py.224: train: epoch 19 | batch 750/2304 [I 2022-10-08 20:45:18] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:20:12 [I 2022-10-08 20:45:18] ...trap/engines/engine.py.229: process: 0.77482 | load: 0.00305 [I 2022-10-08 20:45:18] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-08 20:45:18] ...optimizers/trijoint.py.151: total_norm: 2.126353 [I 2022-10-08 20:45:25] ...trap/engines/engine.py.224: train: epoch 19 | batch 760/2304 [I 2022-10-08 20:45:25] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:20:10 [I 2022-10-08 20:45:25] ...trap/engines/engine.py.229: process: 0.79573 | load: 0.00310 [I 2022-10-08 20:45:25] ...trap/engines/engine.py.230: loss: 0.22755 [I 2022-10-08 20:45:25] ...optimizers/trijoint.py.151: total_norm: 2.548770 [I 2022-10-08 20:45:33] ...trap/engines/engine.py.224: train: epoch 19 | batch 770/2304 [I 2022-10-08 20:45:33] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:19:52 [I 2022-10-08 20:45:33] ...trap/engines/engine.py.229: process: 0.77041 | load: 0.00301 [I 2022-10-08 20:45:33] ...trap/engines/engine.py.230: loss: 0.15288 [I 2022-10-08 20:45:33] ...optimizers/trijoint.py.151: total_norm: 2.396020 [I 2022-10-08 20:45:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 780/2304 [I 2022-10-08 20:45:41] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:19:52 [I 2022-10-08 20:45:41] ...trap/engines/engine.py.229: process: 0.79194 | load: 0.00301 [I 2022-10-08 20:45:41] ...trap/engines/engine.py.230: loss: 0.18423 [I 2022-10-08 20:45:41] ...optimizers/trijoint.py.151: total_norm: 2.560582 [I 2022-10-08 20:45:49] ...trap/engines/engine.py.224: train: epoch 19 | batch 790/2304 [I 2022-10-08 20:45:49] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:19:50 [I 2022-10-08 20:45:49] ...trap/engines/engine.py.229: process: 0.79257 | load: 0.00303 [I 2022-10-08 20:45:49] ...trap/engines/engine.py.230: loss: 0.19257 [I 2022-10-08 20:45:49] ...optimizers/trijoint.py.151: total_norm: 2.133455 [I 2022-10-08 20:45:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 800/2304 [I 2022-10-08 20:45:57] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:19:35 [I 2022-10-08 20:45:57] ...trap/engines/engine.py.229: process: 0.78233 | load: 0.00302 [I 2022-10-08 20:45:57] ...trap/engines/engine.py.230: loss: 0.36402 [I 2022-10-08 20:45:57] ...optimizers/trijoint.py.151: total_norm: 4.209397 [I 2022-10-08 20:46:05] ...trap/engines/engine.py.224: train: epoch 19 | batch 810/2304 [I 2022-10-08 20:46:05] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:19:22 [I 2022-10-08 20:46:05] ...trap/engines/engine.py.229: process: 0.76328 | load: 0.00303 [I 2022-10-08 20:46:05] ...trap/engines/engine.py.230: loss: 0.17099 [I 2022-10-08 20:46:05] ...optimizers/trijoint.py.151: total_norm: 1.835162 [I 2022-10-08 20:46:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 820/2304 [I 2022-10-08 20:46:12] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:19:09 [I 2022-10-08 20:46:12] ...trap/engines/engine.py.229: process: 0.78025 | load: 0.00303 [I 2022-10-08 20:46:12] ...trap/engines/engine.py.230: loss: 0.18106 [I 2022-10-08 20:46:12] ...optimizers/trijoint.py.151: total_norm: 2.251120 [I 2022-10-08 20:46:20] ...trap/engines/engine.py.224: train: epoch 19 | batch 830/2304 [I 2022-10-08 20:46:20] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:19:04 [I 2022-10-08 20:46:20] ...trap/engines/engine.py.229: process: 0.76389 | load: 0.00306 [I 2022-10-08 20:46:20] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-08 20:46:20] ...optimizers/trijoint.py.151: total_norm: 2.538444 [I 2022-10-08 20:46:28] ...trap/engines/engine.py.224: train: epoch 19 | batch 840/2304 [I 2022-10-08 20:46:28] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:19:06 [I 2022-10-08 20:46:28] ...trap/engines/engine.py.229: process: 0.78112 | load: 0.00310 [I 2022-10-08 20:46:28] ...trap/engines/engine.py.230: loss: 0.22282 [I 2022-10-08 20:46:28] ...optimizers/trijoint.py.151: total_norm: 3.079777 [I 2022-10-08 20:46:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 850/2304 [I 2022-10-08 20:46:36] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:18:56 [I 2022-10-08 20:46:36] ...trap/engines/engine.py.229: process: 0.78267 | load: 0.00309 [I 2022-10-08 20:46:36] ...trap/engines/engine.py.230: loss: 0.18156 [I 2022-10-08 20:46:36] ...optimizers/trijoint.py.151: total_norm: 2.713127 [I 2022-10-08 20:46:44] ...trap/engines/engine.py.224: train: epoch 19 | batch 860/2304 [I 2022-10-08 20:46:44] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:18:51 [I 2022-10-08 20:46:44] ...trap/engines/engine.py.229: process: 0.77596 | load: 0.00315 [I 2022-10-08 20:46:44] ...trap/engines/engine.py.230: loss: 0.24760 [I 2022-10-08 20:46:44] ...optimizers/trijoint.py.151: total_norm: 3.433995 [I 2022-10-08 20:46:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 870/2304 [I 2022-10-08 20:46:52] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:18:42 [I 2022-10-08 20:46:52] ...trap/engines/engine.py.229: process: 0.80044 | load: 0.00309 [I 2022-10-08 20:46:52] ...trap/engines/engine.py.230: loss: 0.14569 [I 2022-10-08 20:46:52] ...optimizers/trijoint.py.151: total_norm: 2.666098 [I 2022-10-08 20:46:59] ...trap/engines/engine.py.224: train: epoch 19 | batch 880/2304 [I 2022-10-08 20:46:59] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:18:29 [I 2022-10-08 20:46:59] ...trap/engines/engine.py.229: process: 0.77762 | load: 0.00302 [I 2022-10-08 20:46:59] ...trap/engines/engine.py.230: loss: 0.15462 [I 2022-10-08 20:46:59] ...optimizers/trijoint.py.151: total_norm: 2.047792 [I 2022-10-08 20:47:07] ...trap/engines/engine.py.224: train: epoch 19 | batch 890/2304 [I 2022-10-08 20:47:07] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:18:21 [I 2022-10-08 20:47:07] ...trap/engines/engine.py.229: process: 0.77553 | load: 0.00302 [I 2022-10-08 20:47:07] ...trap/engines/engine.py.230: loss: 0.19747 [I 2022-10-08 20:47:07] ...optimizers/trijoint.py.151: total_norm: 3.164669 [I 2022-10-08 20:47:15] ...trap/engines/engine.py.224: train: epoch 19 | batch 900/2304 [I 2022-10-08 20:47:15] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:18:11 [I 2022-10-08 20:47:15] ...trap/engines/engine.py.229: process: 0.77834 | load: 0.00300 [I 2022-10-08 20:47:15] ...trap/engines/engine.py.230: loss: 0.20100 [I 2022-10-08 20:47:15] ...optimizers/trijoint.py.151: total_norm: 3.117793 [I 2022-10-08 20:47:23] ...trap/engines/engine.py.224: train: epoch 19 | batch 910/2304 [I 2022-10-08 20:47:23] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:17:59 [I 2022-10-08 20:47:23] ...trap/engines/engine.py.229: process: 0.77246 | load: 0.00301 [I 2022-10-08 20:47:23] ...trap/engines/engine.py.230: loss: 0.24525 [I 2022-10-08 20:47:23] ...optimizers/trijoint.py.151: total_norm: 3.203855 [I 2022-10-08 20:47:30] ...trap/engines/engine.py.224: train: epoch 19 | batch 920/2304 [I 2022-10-08 20:47:30] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:17:59 [I 2022-10-08 20:47:30] ...trap/engines/engine.py.229: process: 0.78409 | load: 0.00302 [I 2022-10-08 20:47:30] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-08 20:47:30] ...optimizers/trijoint.py.151: total_norm: 2.444826 [I 2022-10-08 20:47:38] ...trap/engines/engine.py.224: train: epoch 19 | batch 930/2304 [I 2022-10-08 20:47:38] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:17:43 [I 2022-10-08 20:47:38] ...trap/engines/engine.py.229: process: 0.77568 | load: 0.00301 [I 2022-10-08 20:47:38] ...trap/engines/engine.py.230: loss: 0.18014 [I 2022-10-08 20:47:38] ...optimizers/trijoint.py.151: total_norm: 3.321898 [I 2022-10-08 20:47:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 940/2304 [I 2022-10-08 20:47:46] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:17:44 [I 2022-10-08 20:47:46] ...trap/engines/engine.py.229: process: 0.78805 | load: 0.00308 [I 2022-10-08 20:47:46] ...trap/engines/engine.py.230: loss: 0.25674 [I 2022-10-08 20:47:46] ...optimizers/trijoint.py.151: total_norm: 3.291391 [I 2022-10-08 20:47:54] ...trap/engines/engine.py.224: train: epoch 19 | batch 950/2304 [I 2022-10-08 20:47:54] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:17:29 [I 2022-10-08 20:47:54] ...trap/engines/engine.py.229: process: 0.77115 | load: 0.00307 [I 2022-10-08 20:47:54] ...trap/engines/engine.py.230: loss: 0.15414 [I 2022-10-08 20:47:54] ...optimizers/trijoint.py.151: total_norm: 2.712750 [I 2022-10-08 20:48:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 960/2304 [I 2022-10-08 20:48:02] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:17:30 [I 2022-10-08 20:48:02] ...trap/engines/engine.py.229: process: 0.78336 | load: 0.00305 [I 2022-10-08 20:48:02] ...trap/engines/engine.py.230: loss: 0.17031 [I 2022-10-08 20:48:02] ...optimizers/trijoint.py.151: total_norm: 3.955745 [I 2022-10-08 20:48:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 970/2304 [I 2022-10-08 20:48:10] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:17:23 [I 2022-10-08 20:48:10] ...trap/engines/engine.py.229: process: 0.77503 | load: 0.00299 [I 2022-10-08 20:48:10] ...trap/engines/engine.py.230: loss: 0.15551 [I 2022-10-08 20:48:10] ...optimizers/trijoint.py.151: total_norm: 2.072135 [I 2022-10-08 20:48:17] ...trap/engines/engine.py.224: train: epoch 19 | batch 980/2304 [I 2022-10-08 20:48:17] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:17:06 [I 2022-10-08 20:48:17] ...trap/engines/engine.py.229: process: 0.76407 | load: 0.00303 [I 2022-10-08 20:48:17] ...trap/engines/engine.py.230: loss: 0.16336 [I 2022-10-08 20:48:17] ...optimizers/trijoint.py.151: total_norm: 2.551090 [I 2022-10-08 20:48:25] ...trap/engines/engine.py.224: train: epoch 19 | batch 990/2304 [I 2022-10-08 20:48:25] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:16:58 [I 2022-10-08 20:48:25] ...trap/engines/engine.py.229: process: 0.77634 | load: 0.00302 [I 2022-10-08 20:48:25] ...trap/engines/engine.py.230: loss: 0.20446 [I 2022-10-08 20:48:25] ...optimizers/trijoint.py.151: total_norm: 3.173092 [I 2022-10-08 20:48:33] ...trap/engines/engine.py.224: train: epoch 19 | batch 1000/2304 [I 2022-10-08 20:48:33] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:16:50 [I 2022-10-08 20:48:33] ...trap/engines/engine.py.229: process: 0.77223 | load: 0.00302 [I 2022-10-08 20:48:33] ...trap/engines/engine.py.230: loss: 0.17916 [I 2022-10-08 20:48:33] ...optimizers/trijoint.py.151: total_norm: 3.418942 [I 2022-10-08 20:48:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1010/2304 [I 2022-10-08 20:48:41] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:16:44 [I 2022-10-08 20:48:41] ...trap/engines/engine.py.229: process: 0.77460 | load: 0.00297 [I 2022-10-08 20:48:41] ...trap/engines/engine.py.230: loss: 0.17611 [I 2022-10-08 20:48:41] ...optimizers/trijoint.py.151: total_norm: 2.559081 [I 2022-10-08 20:48:48] ...trap/engines/engine.py.224: train: epoch 19 | batch 1020/2304 [I 2022-10-08 20:48:48] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:16:38 [I 2022-10-08 20:48:48] ...trap/engines/engine.py.229: process: 0.76684 | load: 0.00297 [I 2022-10-08 20:48:48] ...trap/engines/engine.py.230: loss: 0.18161 [I 2022-10-08 20:48:48] ...optimizers/trijoint.py.151: total_norm: 2.605513 [I 2022-10-08 20:48:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1030/2304 [I 2022-10-08 20:48:56] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:16:25 [I 2022-10-08 20:48:56] ...trap/engines/engine.py.229: process: 0.77705 | load: 0.00299 [I 2022-10-08 20:48:56] ...trap/engines/engine.py.230: loss: 0.20619 [I 2022-10-08 20:48:56] ...optimizers/trijoint.py.151: total_norm: 3.033761 [I 2022-10-08 20:49:04] ...trap/engines/engine.py.224: train: epoch 19 | batch 1040/2304 [I 2022-10-08 20:49:04] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:16:20 [I 2022-10-08 20:49:04] ...trap/engines/engine.py.229: process: 0.76458 | load: 0.00304 [I 2022-10-08 20:49:04] ...trap/engines/engine.py.230: loss: 0.22290 [I 2022-10-08 20:49:04] ...optimizers/trijoint.py.151: total_norm: 3.060950 [I 2022-10-08 20:49:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 1050/2304 [I 2022-10-08 20:49:12] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:16:18 [I 2022-10-08 20:49:12] ...trap/engines/engine.py.229: process: 0.79031 | load: 0.00300 [I 2022-10-08 20:49:12] ...trap/engines/engine.py.230: loss: 0.23099 [I 2022-10-08 20:49:12] ...optimizers/trijoint.py.151: total_norm: 3.455970 [I 2022-10-08 20:49:20] ...trap/engines/engine.py.224: train: epoch 19 | batch 1060/2304 [I 2022-10-08 20:49:20] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:16:06 [I 2022-10-08 20:49:20] ...trap/engines/engine.py.229: process: 0.77976 | load: 0.00299 [I 2022-10-08 20:49:20] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-08 20:49:20] ...optimizers/trijoint.py.151: total_norm: 2.101089 [I 2022-10-08 20:49:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 1070/2304 [I 2022-10-08 20:49:27] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:15:56 [I 2022-10-08 20:49:27] ...trap/engines/engine.py.229: process: 0.76621 | load: 0.00301 [I 2022-10-08 20:49:27] ...trap/engines/engine.py.230: loss: 0.16359 [I 2022-10-08 20:49:27] ...optimizers/trijoint.py.151: total_norm: 2.413156 [I 2022-10-08 20:49:35] ...trap/engines/engine.py.224: train: epoch 19 | batch 1080/2304 [I 2022-10-08 20:49:35] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:15:58 [I 2022-10-08 20:49:35] ...trap/engines/engine.py.229: process: 0.78701 | load: 0.00300 [I 2022-10-08 20:49:35] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-08 20:49:35] ...optimizers/trijoint.py.151: total_norm: 2.390730 [I 2022-10-08 20:49:43] ...trap/engines/engine.py.224: train: epoch 19 | batch 1090/2304 [I 2022-10-08 20:49:43] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:15:49 [I 2022-10-08 20:49:43] ...trap/engines/engine.py.229: process: 0.78981 | load: 0.00302 [I 2022-10-08 20:49:43] ...trap/engines/engine.py.230: loss: 0.18551 [I 2022-10-08 20:49:43] ...optimizers/trijoint.py.151: total_norm: 2.591019 [I 2022-10-08 20:49:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1100/2304 [I 2022-10-08 20:49:51] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:15:35 [I 2022-10-08 20:49:51] ...trap/engines/engine.py.229: process: 0.76508 | load: 0.00306 [I 2022-10-08 20:49:51] ...trap/engines/engine.py.230: loss: 0.16275 [I 2022-10-08 20:49:51] ...optimizers/trijoint.py.151: total_norm: 2.498183 [I 2022-10-08 20:49:59] ...trap/engines/engine.py.224: train: epoch 19 | batch 1110/2304 [I 2022-10-08 20:49:59] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:15:40 [I 2022-10-08 20:49:59] ...trap/engines/engine.py.229: process: 0.78739 | load: 0.00306 [I 2022-10-08 20:49:59] ...trap/engines/engine.py.230: loss: 0.14403 [I 2022-10-08 20:49:59] ...optimizers/trijoint.py.151: total_norm: 2.716023 [I 2022-10-08 20:50:07] ...trap/engines/engine.py.224: train: epoch 19 | batch 1120/2304 [I 2022-10-08 20:50:07] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:15:29 [I 2022-10-08 20:50:07] ...trap/engines/engine.py.229: process: 0.78784 | load: 0.00305 [I 2022-10-08 20:50:07] ...trap/engines/engine.py.230: loss: 0.18723 [I 2022-10-08 20:50:07] ...optimizers/trijoint.py.151: total_norm: 2.806033 [I 2022-10-08 20:50:15] ...trap/engines/engine.py.224: train: epoch 19 | batch 1130/2304 [I 2022-10-08 20:50:15] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:15:26 [I 2022-10-08 20:50:15] ...trap/engines/engine.py.229: process: 0.78367 | load: 0.00304 [I 2022-10-08 20:50:15] ...trap/engines/engine.py.230: loss: 0.22769 [I 2022-10-08 20:50:15] ...optimizers/trijoint.py.151: total_norm: 3.988062 [I 2022-10-08 20:50:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 1140/2304 [I 2022-10-08 20:50:22] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:15:12 [I 2022-10-08 20:50:22] ...trap/engines/engine.py.229: process: 0.78101 | load: 0.00305 [I 2022-10-08 20:50:22] ...trap/engines/engine.py.230: loss: 0.16129 [I 2022-10-08 20:50:22] ...optimizers/trijoint.py.151: total_norm: 3.300022 [I 2022-10-08 20:50:30] ...trap/engines/engine.py.224: train: epoch 19 | batch 1150/2304 [I 2022-10-08 20:50:30] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:14:56 [I 2022-10-08 20:50:30] ...trap/engines/engine.py.229: process: 0.76053 | load: 0.00303 [I 2022-10-08 20:50:30] ...trap/engines/engine.py.230: loss: 0.18983 [I 2022-10-08 20:50:30] ...optimizers/trijoint.py.151: total_norm: 3.089117 [I 2022-10-08 20:50:38] ...trap/engines/engine.py.224: train: epoch 19 | batch 1160/2304 [I 2022-10-08 20:50:38] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:14:45 [I 2022-10-08 20:50:38] ...trap/engines/engine.py.229: process: 0.76499 | load: 0.00304 [I 2022-10-08 20:50:38] ...trap/engines/engine.py.230: loss: 0.21742 [I 2022-10-08 20:50:38] ...optimizers/trijoint.py.151: total_norm: 3.269034 [I 2022-10-08 20:50:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1170/2304 [I 2022-10-08 20:50:46] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:14:41 [I 2022-10-08 20:50:46] ...trap/engines/engine.py.229: process: 0.77752 | load: 0.00302 [I 2022-10-08 20:50:46] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-08 20:50:46] ...optimizers/trijoint.py.151: total_norm: 2.038488 [I 2022-10-08 20:50:54] ...trap/engines/engine.py.224: train: epoch 19 | batch 1180/2304 [I 2022-10-08 20:50:54] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:14:35 [I 2022-10-08 20:50:54] ...trap/engines/engine.py.229: process: 0.78324 | load: 0.00303 [I 2022-10-08 20:50:54] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-08 20:50:54] ...optimizers/trijoint.py.151: total_norm: 2.423995 [I 2022-10-08 20:51:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1190/2304 [I 2022-10-08 20:51:01] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:14:24 [I 2022-10-08 20:51:01] ...trap/engines/engine.py.229: process: 0.77681 | load: 0.00306 [I 2022-10-08 20:51:01] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-08 20:51:01] ...optimizers/trijoint.py.151: total_norm: 1.979214 [I 2022-10-08 20:51:09] ...trap/engines/engine.py.224: train: epoch 19 | batch 1200/2304 [I 2022-10-08 20:51:09] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:14:21 [I 2022-10-08 20:51:09] ...trap/engines/engine.py.229: process: 0.78735 | load: 0.00305 [I 2022-10-08 20:51:09] ...trap/engines/engine.py.230: loss: 0.10461 [I 2022-10-08 20:51:09] ...optimizers/trijoint.py.151: total_norm: 2.503336 [I 2022-10-08 20:51:17] ...trap/engines/engine.py.224: train: epoch 19 | batch 1210/2304 [I 2022-10-08 20:51:17] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:14:10 [I 2022-10-08 20:51:17] ...trap/engines/engine.py.229: process: 0.78143 | load: 0.00304 [I 2022-10-08 20:51:17] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-08 20:51:17] ...optimizers/trijoint.py.151: total_norm: 2.068788 [I 2022-10-08 20:51:25] ...trap/engines/engine.py.224: train: epoch 19 | batch 1220/2304 [I 2022-10-08 20:51:25] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:14:08 [I 2022-10-08 20:51:25] ...trap/engines/engine.py.229: process: 0.78763 | load: 0.00302 [I 2022-10-08 20:51:25] ...trap/engines/engine.py.230: loss: 0.18327 [I 2022-10-08 20:51:25] ...optimizers/trijoint.py.151: total_norm: 2.163725 [I 2022-10-08 20:51:33] ...trap/engines/engine.py.224: train: epoch 19 | batch 1230/2304 [I 2022-10-08 20:51:33] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:13:54 [I 2022-10-08 20:51:33] ...trap/engines/engine.py.229: process: 0.77179 | load: 0.00302 [I 2022-10-08 20:51:33] ...trap/engines/engine.py.230: loss: 0.16340 [I 2022-10-08 20:51:33] ...optimizers/trijoint.py.151: total_norm: 2.302325 [I 2022-10-08 20:51:40] ...trap/engines/engine.py.224: train: epoch 19 | batch 1240/2304 [I 2022-10-08 20:51:40] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:13:48 [I 2022-10-08 20:51:40] ...trap/engines/engine.py.229: process: 0.77693 | load: 0.00299 [I 2022-10-08 20:51:40] ...trap/engines/engine.py.230: loss: 0.21764 [I 2022-10-08 20:51:40] ...optimizers/trijoint.py.151: total_norm: 3.723037 [I 2022-10-08 20:51:48] ...trap/engines/engine.py.224: train: epoch 19 | batch 1250/2304 [I 2022-10-08 20:51:48] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:13:41 [I 2022-10-08 20:51:48] ...trap/engines/engine.py.229: process: 0.78138 | load: 0.00302 [I 2022-10-08 20:51:48] ...trap/engines/engine.py.230: loss: 0.30272 [I 2022-10-08 20:51:48] ...optimizers/trijoint.py.151: total_norm: 3.708704 [I 2022-10-08 20:51:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1260/2304 [I 2022-10-08 20:51:56] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:13:32 [I 2022-10-08 20:51:56] ...trap/engines/engine.py.229: process: 0.78228 | load: 0.00306 [I 2022-10-08 20:51:56] ...trap/engines/engine.py.230: loss: 0.15016 [I 2022-10-08 20:51:56] ...optimizers/trijoint.py.151: total_norm: 2.092860 [I 2022-10-08 20:52:04] ...trap/engines/engine.py.224: train: epoch 19 | batch 1270/2304 [I 2022-10-08 20:52:04] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:13:29 [I 2022-10-08 20:52:04] ...trap/engines/engine.py.229: process: 0.79310 | load: 0.00301 [I 2022-10-08 20:52:04] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-08 20:52:04] ...optimizers/trijoint.py.151: total_norm: 2.151634 [I 2022-10-08 20:52:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 1280/2304 [I 2022-10-08 20:52:12] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:13:19 [I 2022-10-08 20:52:12] ...trap/engines/engine.py.229: process: 0.77762 | load: 0.00305 [I 2022-10-08 20:52:12] ...trap/engines/engine.py.230: loss: 0.21548 [I 2022-10-08 20:52:12] ...optimizers/trijoint.py.151: total_norm: 2.301953 [I 2022-10-08 20:52:20] ...trap/engines/engine.py.224: train: epoch 19 | batch 1290/2304 [I 2022-10-08 20:52:20] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:13:12 [I 2022-10-08 20:52:20] ...trap/engines/engine.py.229: process: 0.78239 | load: 0.00303 [I 2022-10-08 20:52:20] ...trap/engines/engine.py.230: loss: 0.11339 [I 2022-10-08 20:52:20] ...optimizers/trijoint.py.151: total_norm: 2.223511 [I 2022-10-08 20:52:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 1300/2304 [I 2022-10-08 20:52:27] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:13:03 [I 2022-10-08 20:52:27] ...trap/engines/engine.py.229: process: 0.78254 | load: 0.00303 [I 2022-10-08 20:52:27] ...trap/engines/engine.py.230: loss: 0.19357 [I 2022-10-08 20:52:27] ...optimizers/trijoint.py.151: total_norm: 3.342378 [I 2022-10-08 20:52:35] ...trap/engines/engine.py.224: train: epoch 19 | batch 1310/2304 [I 2022-10-08 20:52:35] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:12:49 [I 2022-10-08 20:52:35] ...trap/engines/engine.py.229: process: 0.77977 | load: 0.00304 [I 2022-10-08 20:52:35] ...trap/engines/engine.py.230: loss: 0.23127 [I 2022-10-08 20:52:35] ...optimizers/trijoint.py.151: total_norm: 2.450736 [I 2022-10-08 20:52:43] ...trap/engines/engine.py.224: train: epoch 19 | batch 1320/2304 [I 2022-10-08 20:52:43] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:12:44 [I 2022-10-08 20:52:43] ...trap/engines/engine.py.229: process: 0.78826 | load: 0.00304 [I 2022-10-08 20:52:43] ...trap/engines/engine.py.230: loss: 0.15390 [I 2022-10-08 20:52:43] ...optimizers/trijoint.py.151: total_norm: 2.971170 [I 2022-10-08 20:52:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1330/2304 [I 2022-10-08 20:52:51] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:12:34 [I 2022-10-08 20:52:51] ...trap/engines/engine.py.229: process: 0.77097 | load: 0.00305 [I 2022-10-08 20:52:51] ...trap/engines/engine.py.230: loss: 0.14667 [I 2022-10-08 20:52:51] ...optimizers/trijoint.py.151: total_norm: 2.336880 [I 2022-10-08 20:52:59] ...trap/engines/engine.py.224: train: epoch 19 | batch 1340/2304 [I 2022-10-08 20:52:59] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:12:29 [I 2022-10-08 20:52:59] ...trap/engines/engine.py.229: process: 0.77916 | load: 0.00302 [I 2022-10-08 20:52:59] ...trap/engines/engine.py.230: loss: 0.19007 [I 2022-10-08 20:52:59] ...optimizers/trijoint.py.151: total_norm: 3.738081 [I 2022-10-08 20:53:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1350/2304 [I 2022-10-08 20:53:06] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:12:26 [I 2022-10-08 20:53:06] ...trap/engines/engine.py.229: process: 0.78366 | load: 0.00301 [I 2022-10-08 20:53:06] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-08 20:53:06] ...optimizers/trijoint.py.151: total_norm: 3.254259 [I 2022-10-08 20:53:14] ...trap/engines/engine.py.224: train: epoch 19 | batch 1360/2304 [I 2022-10-08 20:53:14] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:12:19 [I 2022-10-08 20:53:14] ...trap/engines/engine.py.229: process: 0.77979 | load: 0.00300 [I 2022-10-08 20:53:14] ...trap/engines/engine.py.230: loss: 0.18942 [I 2022-10-08 20:53:14] ...optimizers/trijoint.py.151: total_norm: 2.615227 [I 2022-10-08 20:53:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 1370/2304 [I 2022-10-08 20:53:22] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:12:11 [I 2022-10-08 20:53:22] ...trap/engines/engine.py.229: process: 0.77408 | load: 0.00304 [I 2022-10-08 20:53:22] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-08 20:53:22] ...optimizers/trijoint.py.151: total_norm: 2.190664 [I 2022-10-08 20:53:30] ...trap/engines/engine.py.224: train: epoch 19 | batch 1380/2304 [I 2022-10-08 20:53:30] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:12:02 [I 2022-10-08 20:53:30] ...trap/engines/engine.py.229: process: 0.78209 | load: 0.00303 [I 2022-10-08 20:53:30] ...trap/engines/engine.py.230: loss: 0.15138 [I 2022-10-08 20:53:30] ...optimizers/trijoint.py.151: total_norm: 2.054883 [I 2022-10-08 20:53:38] ...trap/engines/engine.py.224: train: epoch 19 | batch 1390/2304 [I 2022-10-08 20:53:38] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:11:51 [I 2022-10-08 20:53:38] ...trap/engines/engine.py.229: process: 0.77683 | load: 0.00301 [I 2022-10-08 20:53:38] ...trap/engines/engine.py.230: loss: 0.15579 [I 2022-10-08 20:53:38] ...optimizers/trijoint.py.151: total_norm: 1.674538 [I 2022-10-08 20:53:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1400/2304 [I 2022-10-08 20:53:46] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:11:43 [I 2022-10-08 20:53:46] ...trap/engines/engine.py.229: process: 0.76875 | load: 0.00299 [I 2022-10-08 20:53:46] ...trap/engines/engine.py.230: loss: 0.24473 [I 2022-10-08 20:53:46] ...optimizers/trijoint.py.151: total_norm: 2.799203 [I 2022-10-08 20:53:53] ...trap/engines/engine.py.224: train: epoch 19 | batch 1410/2304 [I 2022-10-08 20:53:53] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:11:32 [I 2022-10-08 20:53:53] ...trap/engines/engine.py.229: process: 0.78070 | load: 0.00300 [I 2022-10-08 20:53:53] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-08 20:53:53] ...optimizers/trijoint.py.151: total_norm: 2.135838 [I 2022-10-08 20:54:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1420/2304 [I 2022-10-08 20:54:01] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:11:23 [I 2022-10-08 20:54:01] ...trap/engines/engine.py.229: process: 0.76505 | load: 0.00300 [I 2022-10-08 20:54:01] ...trap/engines/engine.py.230: loss: 0.18186 [I 2022-10-08 20:54:01] ...optimizers/trijoint.py.151: total_norm: 2.812667 [I 2022-10-08 20:54:09] ...trap/engines/engine.py.224: train: epoch 19 | batch 1430/2304 [I 2022-10-08 20:54:09] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:11:27 [I 2022-10-08 20:54:09] ...trap/engines/engine.py.229: process: 0.80819 | load: 0.00300 [I 2022-10-08 20:54:09] ...trap/engines/engine.py.230: loss: 0.10325 [I 2022-10-08 20:54:09] ...optimizers/trijoint.py.151: total_norm: 2.306710 [I 2022-10-08 20:54:17] ...trap/engines/engine.py.224: train: epoch 19 | batch 1440/2304 [I 2022-10-08 20:54:17] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:11:15 [I 2022-10-08 20:54:17] ...trap/engines/engine.py.229: process: 0.77666 | load: 0.00302 [I 2022-10-08 20:54:17] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-08 20:54:17] ...optimizers/trijoint.py.151: total_norm: 1.770527 [I 2022-10-08 20:54:25] ...trap/engines/engine.py.224: train: epoch 19 | batch 1450/2304 [I 2022-10-08 20:54:25] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:11:04 [I 2022-10-08 20:54:25] ...trap/engines/engine.py.229: process: 0.76509 | load: 0.00296 [I 2022-10-08 20:54:25] ...trap/engines/engine.py.230: loss: 0.15395 [I 2022-10-08 20:54:25] ...optimizers/trijoint.py.151: total_norm: 1.997550 [I 2022-10-08 20:54:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 1460/2304 [I 2022-10-08 20:54:32] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:10:57 [I 2022-10-08 20:54:32] ...trap/engines/engine.py.229: process: 0.78458 | load: 0.00301 [I 2022-10-08 20:54:32] ...trap/engines/engine.py.230: loss: 0.16887 [I 2022-10-08 20:54:32] ...optimizers/trijoint.py.151: total_norm: 2.371834 [I 2022-10-08 20:54:40] ...trap/engines/engine.py.224: train: epoch 19 | batch 1470/2304 [I 2022-10-08 20:54:40] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:10:47 [I 2022-10-08 20:54:40] ...trap/engines/engine.py.229: process: 0.76702 | load: 0.00304 [I 2022-10-08 20:54:40] ...trap/engines/engine.py.230: loss: 0.18050 [I 2022-10-08 20:54:40] ...optimizers/trijoint.py.151: total_norm: 2.731150 [I 2022-10-08 20:54:48] ...trap/engines/engine.py.224: train: epoch 19 | batch 1480/2304 [I 2022-10-08 20:54:48] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:10:44 [I 2022-10-08 20:54:48] ...trap/engines/engine.py.229: process: 0.78286 | load: 0.00302 [I 2022-10-08 20:54:48] ...trap/engines/engine.py.230: loss: 0.15625 [I 2022-10-08 20:54:48] ...optimizers/trijoint.py.151: total_norm: 2.782576 [I 2022-10-08 20:54:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1490/2304 [I 2022-10-08 20:54:56] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:10:31 [I 2022-10-08 20:54:56] ...trap/engines/engine.py.229: process: 0.77771 | load: 0.00302 [I 2022-10-08 20:54:56] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-08 20:54:56] ...optimizers/trijoint.py.151: total_norm: 2.956620 [I 2022-10-08 20:55:04] ...trap/engines/engine.py.224: train: epoch 19 | batch 1500/2304 [I 2022-10-08 20:55:04] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:10:24 [I 2022-10-08 20:55:04] ...trap/engines/engine.py.229: process: 0.76662 | load: 0.00307 [I 2022-10-08 20:55:04] ...trap/engines/engine.py.230: loss: 0.22429 [I 2022-10-08 20:55:04] ...optimizers/trijoint.py.151: total_norm: 3.404005 [I 2022-10-08 20:55:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1510/2304 [I 2022-10-08 20:55:11] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:10:19 [I 2022-10-08 20:55:11] ...trap/engines/engine.py.229: process: 0.76737 | load: 0.00301 [I 2022-10-08 20:55:11] ...trap/engines/engine.py.230: loss: 0.17557 [I 2022-10-08 20:55:11] ...optimizers/trijoint.py.151: total_norm: 2.664806 [I 2022-10-08 20:55:19] ...trap/engines/engine.py.224: train: epoch 19 | batch 1520/2304 [I 2022-10-08 20:55:19] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:10:12 [I 2022-10-08 20:55:19] ...trap/engines/engine.py.229: process: 0.78374 | load: 0.00302 [I 2022-10-08 20:55:19] ...trap/engines/engine.py.230: loss: 0.22178 [I 2022-10-08 20:55:19] ...optimizers/trijoint.py.151: total_norm: 2.701249 [I 2022-10-08 20:55:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 1530/2304 [I 2022-10-08 20:55:27] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:10:02 [I 2022-10-08 20:55:27] ...trap/engines/engine.py.229: process: 0.77193 | load: 0.00339 [I 2022-10-08 20:55:27] ...trap/engines/engine.py.230: loss: 0.18749 [I 2022-10-08 20:55:27] ...optimizers/trijoint.py.151: total_norm: 3.052958 [I 2022-10-08 20:55:35] ...trap/engines/engine.py.224: train: epoch 19 | batch 1540/2304 [I 2022-10-08 20:55:35] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:09:52 [I 2022-10-08 20:55:35] ...trap/engines/engine.py.229: process: 0.76804 | load: 0.00303 [I 2022-10-08 20:55:35] ...trap/engines/engine.py.230: loss: 0.28015 [I 2022-10-08 20:55:35] ...optimizers/trijoint.py.151: total_norm: 3.846383 [I 2022-10-08 20:55:43] ...trap/engines/engine.py.224: train: epoch 19 | batch 1550/2304 [I 2022-10-08 20:55:43] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:09:46 [I 2022-10-08 20:55:43] ...trap/engines/engine.py.229: process: 0.78180 | load: 0.00307 [I 2022-10-08 20:55:43] ...trap/engines/engine.py.230: loss: 0.11581 [I 2022-10-08 20:55:43] ...optimizers/trijoint.py.151: total_norm: 1.685998 [I 2022-10-08 20:55:50] ...trap/engines/engine.py.224: train: epoch 19 | batch 1560/2304 [I 2022-10-08 20:55:50] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:09:36 [I 2022-10-08 20:55:50] ...trap/engines/engine.py.229: process: 0.77980 | load: 0.00303 [I 2022-10-08 20:55:50] ...trap/engines/engine.py.230: loss: 0.09857 [I 2022-10-08 20:55:50] ...optimizers/trijoint.py.151: total_norm: 1.656191 [I 2022-10-08 20:55:58] ...trap/engines/engine.py.224: train: epoch 19 | batch 1570/2304 [I 2022-10-08 20:55:58] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:09:33 [I 2022-10-08 20:55:58] ...trap/engines/engine.py.229: process: 0.77608 | load: 0.00302 [I 2022-10-08 20:55:58] ...trap/engines/engine.py.230: loss: 0.20385 [I 2022-10-08 20:55:58] ...optimizers/trijoint.py.151: total_norm: 3.514068 [I 2022-10-08 20:56:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1580/2304 [I 2022-10-08 20:56:06] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:09:23 [I 2022-10-08 20:56:06] ...trap/engines/engine.py.229: process: 0.77613 | load: 0.00304 [I 2022-10-08 20:56:06] ...trap/engines/engine.py.230: loss: 0.23683 [I 2022-10-08 20:56:06] ...optimizers/trijoint.py.151: total_norm: 3.450557 [I 2022-10-08 20:56:14] ...trap/engines/engine.py.224: train: epoch 19 | batch 1590/2304 [I 2022-10-08 20:56:14] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:09:15 [I 2022-10-08 20:56:14] ...trap/engines/engine.py.229: process: 0.79246 | load: 0.00301 [I 2022-10-08 20:56:14] ...trap/engines/engine.py.230: loss: 0.17405 [I 2022-10-08 20:56:14] ...optimizers/trijoint.py.151: total_norm: 2.101931 [I 2022-10-08 20:56:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 1600/2304 [I 2022-10-08 20:56:22] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:09:08 [I 2022-10-08 20:56:22] ...trap/engines/engine.py.229: process: 0.78287 | load: 0.00302 [I 2022-10-08 20:56:22] ...trap/engines/engine.py.230: loss: 0.19722 [I 2022-10-08 20:56:22] ...optimizers/trijoint.py.151: total_norm: 3.369069 [I 2022-10-08 20:56:29] ...trap/engines/engine.py.224: train: epoch 19 | batch 1610/2304 [I 2022-10-08 20:56:29] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:08:59 [I 2022-10-08 20:56:29] ...trap/engines/engine.py.229: process: 0.77988 | load: 0.00310 [I 2022-10-08 20:56:29] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-08 20:56:29] ...optimizers/trijoint.py.151: total_norm: 1.781341 [I 2022-10-08 20:56:37] ...trap/engines/engine.py.224: train: epoch 19 | batch 1620/2304 [I 2022-10-08 20:56:37] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:08:53 [I 2022-10-08 20:56:37] ...trap/engines/engine.py.229: process: 0.78581 | load: 0.00303 [I 2022-10-08 20:56:37] ...trap/engines/engine.py.230: loss: 0.22449 [I 2022-10-08 20:56:37] ...optimizers/trijoint.py.151: total_norm: 3.180617 [I 2022-10-08 20:56:45] ...trap/engines/engine.py.224: train: epoch 19 | batch 1630/2304 [I 2022-10-08 20:56:45] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:08:43 [I 2022-10-08 20:56:45] ...trap/engines/engine.py.229: process: 0.78137 | load: 0.00299 [I 2022-10-08 20:56:45] ...trap/engines/engine.py.230: loss: 0.26664 [I 2022-10-08 20:56:45] ...optimizers/trijoint.py.151: total_norm: 3.737021 [I 2022-10-08 20:56:53] ...trap/engines/engine.py.224: train: epoch 19 | batch 1640/2304 [I 2022-10-08 20:56:53] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:08:39 [I 2022-10-08 20:56:53] ...trap/engines/engine.py.229: process: 0.78592 | load: 0.00299 [I 2022-10-08 20:56:53] ...trap/engines/engine.py.230: loss: 0.19920 [I 2022-10-08 20:56:53] ...optimizers/trijoint.py.151: total_norm: 2.794340 [I 2022-10-08 20:57:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1650/2304 [I 2022-10-08 20:57:01] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:08:30 [I 2022-10-08 20:57:01] ...trap/engines/engine.py.229: process: 0.77920 | load: 0.00303 [I 2022-10-08 20:57:01] ...trap/engines/engine.py.230: loss: 0.13368 [I 2022-10-08 20:57:01] ...optimizers/trijoint.py.151: total_norm: 1.964806 [I 2022-10-08 20:57:08] ...trap/engines/engine.py.224: train: epoch 19 | batch 1660/2304 [I 2022-10-08 20:57:08] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:08:21 [I 2022-10-08 20:57:08] ...trap/engines/engine.py.229: process: 0.78451 | load: 0.00301 [I 2022-10-08 20:57:08] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-08 20:57:08] ...optimizers/trijoint.py.151: total_norm: 2.678695 [I 2022-10-08 20:57:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1670/2304 [I 2022-10-08 20:57:16] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:08:11 [I 2022-10-08 20:57:16] ...trap/engines/engine.py.229: process: 0.77739 | load: 0.00301 [I 2022-10-08 20:57:16] ...trap/engines/engine.py.230: loss: 0.18153 [I 2022-10-08 20:57:16] ...optimizers/trijoint.py.151: total_norm: 3.981539 [I 2022-10-08 20:57:24] ...trap/engines/engine.py.224: train: epoch 19 | batch 1680/2304 [I 2022-10-08 20:57:24] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:08:05 [I 2022-10-08 20:57:24] ...trap/engines/engine.py.229: process: 0.77345 | load: 0.00303 [I 2022-10-08 20:57:24] ...trap/engines/engine.py.230: loss: 0.19455 [I 2022-10-08 20:57:24] ...optimizers/trijoint.py.151: total_norm: 2.441249 [I 2022-10-08 20:57:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 1690/2304 [I 2022-10-08 20:57:32] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:07:55 [I 2022-10-08 20:57:32] ...trap/engines/engine.py.229: process: 0.77415 | load: 0.00304 [I 2022-10-08 20:57:32] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-08 20:57:32] ...optimizers/trijoint.py.151: total_norm: 2.871173 [I 2022-10-08 20:57:40] ...trap/engines/engine.py.224: train: epoch 19 | batch 1700/2304 [I 2022-10-08 20:57:40] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:07:49 [I 2022-10-08 20:57:40] ...trap/engines/engine.py.229: process: 0.78042 | load: 0.00300 [I 2022-10-08 20:57:40] ...trap/engines/engine.py.230: loss: 0.14074 [I 2022-10-08 20:57:40] ...optimizers/trijoint.py.151: total_norm: 1.710093 [I 2022-10-08 20:57:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 1710/2304 [I 2022-10-08 20:57:47] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:07:46 [I 2022-10-08 20:57:47] ...trap/engines/engine.py.229: process: 0.79500 | load: 0.00302 [I 2022-10-08 20:57:47] ...trap/engines/engine.py.230: loss: 0.19921 [I 2022-10-08 20:57:47] ...optimizers/trijoint.py.151: total_norm: 3.015417 [I 2022-10-08 20:57:55] ...trap/engines/engine.py.224: train: epoch 19 | batch 1720/2304 [I 2022-10-08 20:57:55] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:07:37 [I 2022-10-08 20:57:55] ...trap/engines/engine.py.229: process: 0.78334 | load: 0.00305 [I 2022-10-08 20:57:55] ...trap/engines/engine.py.230: loss: 0.19201 [I 2022-10-08 20:57:55] ...optimizers/trijoint.py.151: total_norm: 2.895462 [I 2022-10-08 20:58:03] ...trap/engines/engine.py.224: train: epoch 19 | batch 1730/2304 [I 2022-10-08 20:58:03] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:07:32 [I 2022-10-08 20:58:03] ...trap/engines/engine.py.229: process: 0.78989 | load: 0.00303 [I 2022-10-08 20:58:03] ...trap/engines/engine.py.230: loss: 0.17860 [I 2022-10-08 20:58:03] ...optimizers/trijoint.py.151: total_norm: 2.892028 [I 2022-10-08 20:58:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1740/2304 [I 2022-10-08 20:58:11] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:07:22 [I 2022-10-08 20:58:11] ...trap/engines/engine.py.229: process: 0.75717 | load: 0.00305 [I 2022-10-08 20:58:11] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-08 20:58:11] ...optimizers/trijoint.py.151: total_norm: 1.584368 [I 2022-10-08 20:58:19] ...trap/engines/engine.py.224: train: epoch 19 | batch 1750/2304 [I 2022-10-08 20:58:19] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:07:11 [I 2022-10-08 20:58:19] ...trap/engines/engine.py.229: process: 0.77309 | load: 0.00303 [I 2022-10-08 20:58:19] ...trap/engines/engine.py.230: loss: 0.18110 [I 2022-10-08 20:58:19] ...optimizers/trijoint.py.151: total_norm: 2.753728 [I 2022-10-08 20:58:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 1760/2304 [I 2022-10-08 20:58:27] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:07:03 [I 2022-10-08 20:58:27] ...trap/engines/engine.py.229: process: 0.77730 | load: 0.00301 [I 2022-10-08 20:58:27] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-08 20:58:27] ...optimizers/trijoint.py.151: total_norm: 2.948413 [I 2022-10-08 20:58:35] ...trap/engines/engine.py.224: train: epoch 19 | batch 1770/2304 [I 2022-10-08 20:58:35] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:06:56 [I 2022-10-08 20:58:35] ...trap/engines/engine.py.229: process: 0.78366 | load: 0.00309 [I 2022-10-08 20:58:35] ...trap/engines/engine.py.230: loss: 0.16717 [I 2022-10-08 20:58:35] ...optimizers/trijoint.py.151: total_norm: 2.074881 [I 2022-10-08 20:58:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 1780/2304 [I 2022-10-08 20:58:42] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:06:47 [I 2022-10-08 20:58:42] ...trap/engines/engine.py.229: process: 0.76105 | load: 0.00302 [I 2022-10-08 20:58:42] ...trap/engines/engine.py.230: loss: 0.16374 [I 2022-10-08 20:58:42] ...optimizers/trijoint.py.151: total_norm: 2.860191 [I 2022-10-08 20:58:50] ...trap/engines/engine.py.224: train: epoch 19 | batch 1790/2304 [I 2022-10-08 20:58:50] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:06:40 [I 2022-10-08 20:58:50] ...trap/engines/engine.py.229: process: 0.76885 | load: 0.00300 [I 2022-10-08 20:58:50] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-08 20:58:50] ...optimizers/trijoint.py.151: total_norm: 1.793437 [I 2022-10-08 20:58:58] ...trap/engines/engine.py.224: train: epoch 19 | batch 1800/2304 [I 2022-10-08 20:58:58] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:06:30 [I 2022-10-08 20:58:58] ...trap/engines/engine.py.229: process: 0.76009 | load: 0.00299 [I 2022-10-08 20:58:58] ...trap/engines/engine.py.230: loss: 0.17697 [I 2022-10-08 20:58:58] ...optimizers/trijoint.py.151: total_norm: 2.807614 [I 2022-10-08 20:59:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1810/2304 [I 2022-10-08 20:59:06] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:06:23 [I 2022-10-08 20:59:06] ...trap/engines/engine.py.229: process: 0.77076 | load: 0.00299 [I 2022-10-08 20:59:06] ...trap/engines/engine.py.230: loss: 0.20874 [I 2022-10-08 20:59:06] ...optimizers/trijoint.py.151: total_norm: 2.609554 [I 2022-10-08 20:59:14] ...trap/engines/engine.py.224: train: epoch 19 | batch 1820/2304 [I 2022-10-08 20:59:14] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:06:17 [I 2022-10-08 20:59:14] ...trap/engines/engine.py.229: process: 0.78148 | load: 0.00299 [I 2022-10-08 20:59:14] ...trap/engines/engine.py.230: loss: 0.20642 [I 2022-10-08 20:59:14] ...optimizers/trijoint.py.151: total_norm: 3.212863 [I 2022-10-08 20:59:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 1830/2304 [I 2022-10-08 20:59:22] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:06:09 [I 2022-10-08 20:59:22] ...trap/engines/engine.py.229: process: 0.78393 | load: 0.00302 [I 2022-10-08 20:59:22] ...trap/engines/engine.py.230: loss: 0.24286 [I 2022-10-08 20:59:22] ...optimizers/trijoint.py.151: total_norm: 2.764781 [I 2022-10-08 20:59:30] ...trap/engines/engine.py.224: train: epoch 19 | batch 1840/2304 [I 2022-10-08 20:59:30] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:06:02 [I 2022-10-08 20:59:30] ...trap/engines/engine.py.229: process: 0.78391 | load: 0.00302 [I 2022-10-08 20:59:30] ...trap/engines/engine.py.230: loss: 0.20122 [I 2022-10-08 20:59:30] ...optimizers/trijoint.py.151: total_norm: 3.211815 [I 2022-10-08 20:59:37] ...trap/engines/engine.py.224: train: epoch 19 | batch 1850/2304 [I 2022-10-08 20:59:37] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:05:52 [I 2022-10-08 20:59:37] ...trap/engines/engine.py.229: process: 0.77763 | load: 0.00303 [I 2022-10-08 20:59:37] ...trap/engines/engine.py.230: loss: 0.22973 [I 2022-10-08 20:59:37] ...optimizers/trijoint.py.151: total_norm: 3.160621 [I 2022-10-08 20:59:45] ...trap/engines/engine.py.224: train: epoch 19 | batch 1860/2304 [I 2022-10-08 20:59:45] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:05:44 [I 2022-10-08 20:59:45] ...trap/engines/engine.py.229: process: 0.78321 | load: 0.00302 [I 2022-10-08 20:59:45] ...trap/engines/engine.py.230: loss: 0.15548 [I 2022-10-08 20:59:45] ...optimizers/trijoint.py.151: total_norm: 2.902462 [I 2022-10-08 20:59:53] ...trap/engines/engine.py.224: train: epoch 19 | batch 1870/2304 [I 2022-10-08 20:59:53] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:05:37 [I 2022-10-08 20:59:53] ...trap/engines/engine.py.229: process: 0.77955 | load: 0.00304 [I 2022-10-08 20:59:53] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-08 20:59:53] ...optimizers/trijoint.py.151: total_norm: 3.302045 [I 2022-10-08 21:00:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1880/2304 [I 2022-10-08 21:00:01] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:05:28 [I 2022-10-08 21:00:01] ...trap/engines/engine.py.229: process: 0.76687 | load: 0.00305 [I 2022-10-08 21:00:01] ...trap/engines/engine.py.230: loss: 0.18306 [I 2022-10-08 21:00:01] ...optimizers/trijoint.py.151: total_norm: 2.912009 [I 2022-10-08 21:00:08] ...trap/engines/engine.py.224: train: epoch 19 | batch 1890/2304 [I 2022-10-08 21:00:08] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:05:22 [I 2022-10-08 21:00:08] ...trap/engines/engine.py.229: process: 0.79264 | load: 0.00303 [I 2022-10-08 21:00:08] ...trap/engines/engine.py.230: loss: 0.17755 [I 2022-10-08 21:00:08] ...optimizers/trijoint.py.151: total_norm: 2.457307 [I 2022-10-08 21:00:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1900/2304 [I 2022-10-08 21:00:16] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:05:13 [I 2022-10-08 21:00:16] ...trap/engines/engine.py.229: process: 0.78133 | load: 0.00303 [I 2022-10-08 21:00:16] ...trap/engines/engine.py.230: loss: 0.18194 [I 2022-10-08 21:00:16] ...optimizers/trijoint.py.151: total_norm: 4.031386 [I 2022-10-08 21:00:24] ...trap/engines/engine.py.224: train: epoch 19 | batch 1910/2304 [I 2022-10-08 21:00:24] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:05:06 [I 2022-10-08 21:00:24] ...trap/engines/engine.py.229: process: 0.77121 | load: 0.00303 [I 2022-10-08 21:00:24] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-08 21:00:24] ...optimizers/trijoint.py.151: total_norm: 1.737054 [I 2022-10-08 21:00:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 1920/2304 [I 2022-10-08 21:00:32] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:04:58 [I 2022-10-08 21:00:32] ...trap/engines/engine.py.229: process: 0.78239 | load: 0.00311 [I 2022-10-08 21:00:32] ...trap/engines/engine.py.230: loss: 0.20886 [I 2022-10-08 21:00:32] ...optimizers/trijoint.py.151: total_norm: 3.112992 [I 2022-10-08 21:00:40] ...trap/engines/engine.py.224: train: epoch 19 | batch 1930/2304 [I 2022-10-08 21:00:40] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:04:50 [I 2022-10-08 21:00:40] ...trap/engines/engine.py.229: process: 0.77895 | load: 0.00302 [I 2022-10-08 21:00:40] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-08 21:00:40] ...optimizers/trijoint.py.151: total_norm: 3.996073 [I 2022-10-08 21:00:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 1940/2304 [I 2022-10-08 21:00:47] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:04:42 [I 2022-10-08 21:00:47] ...trap/engines/engine.py.229: process: 0.77651 | load: 0.00300 [I 2022-10-08 21:00:47] ...trap/engines/engine.py.230: loss: 0.23267 [I 2022-10-08 21:00:47] ...optimizers/trijoint.py.151: total_norm: 3.462700 [I 2022-10-08 21:00:55] ...trap/engines/engine.py.224: train: epoch 19 | batch 1950/2304 [I 2022-10-08 21:00:55] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:04:36 [I 2022-10-08 21:00:55] ...trap/engines/engine.py.229: process: 0.77679 | load: 0.00303 [I 2022-10-08 21:00:55] ...trap/engines/engine.py.230: loss: 0.19026 [I 2022-10-08 21:00:55] ...optimizers/trijoint.py.151: total_norm: 2.849209 [I 2022-10-08 21:01:03] ...trap/engines/engine.py.224: train: epoch 19 | batch 1960/2304 [I 2022-10-08 21:01:03] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:04:28 [I 2022-10-08 21:01:03] ...trap/engines/engine.py.229: process: 0.79540 | load: 0.00302 [I 2022-10-08 21:01:03] ...trap/engines/engine.py.230: loss: 0.10851 [I 2022-10-08 21:01:03] ...optimizers/trijoint.py.151: total_norm: 1.682899 [I 2022-10-08 21:01:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1970/2304 [I 2022-10-08 21:01:11] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:04:21 [I 2022-10-08 21:01:11] ...trap/engines/engine.py.229: process: 0.78061 | load: 0.00305 [I 2022-10-08 21:01:11] ...trap/engines/engine.py.230: loss: 0.16849 [I 2022-10-08 21:01:11] ...optimizers/trijoint.py.151: total_norm: 2.223742 [I 2022-10-08 21:01:19] ...trap/engines/engine.py.224: train: epoch 19 | batch 1980/2304 [I 2022-10-08 21:01:19] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:04:12 [I 2022-10-08 21:01:19] ...trap/engines/engine.py.229: process: 0.78915 | load: 0.00296 [I 2022-10-08 21:01:19] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-08 21:01:19] ...optimizers/trijoint.py.151: total_norm: 2.651975 [I 2022-10-08 21:01:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1990/2304 [I 2022-10-08 21:01:26] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:04:05 [I 2022-10-08 21:01:26] ...trap/engines/engine.py.229: process: 0.78204 | load: 0.00304 [I 2022-10-08 21:01:26] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-08 21:01:26] ...optimizers/trijoint.py.151: total_norm: 2.426137 [I 2022-10-08 21:01:34] ...trap/engines/engine.py.224: train: epoch 19 | batch 2000/2304 [I 2022-10-08 21:01:34] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:03:57 [I 2022-10-08 21:01:34] ...trap/engines/engine.py.229: process: 0.77678 | load: 0.00305 [I 2022-10-08 21:01:34] ...trap/engines/engine.py.230: loss: 0.32928 [I 2022-10-08 21:01:34] ...optimizers/trijoint.py.151: total_norm: 4.540245 [I 2022-10-08 21:01:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 2010/2304 [I 2022-10-08 21:01:42] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:03:52 [I 2022-10-08 21:01:42] ...trap/engines/engine.py.229: process: 0.79787 | load: 0.00304 [I 2022-10-08 21:01:42] ...trap/engines/engine.py.230: loss: 0.19828 [I 2022-10-08 21:01:42] ...optimizers/trijoint.py.151: total_norm: 4.133178 [I 2022-10-08 21:01:50] ...trap/engines/engine.py.224: train: epoch 19 | batch 2020/2304 [I 2022-10-08 21:01:50] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:03:41 [I 2022-10-08 21:01:50] ...trap/engines/engine.py.229: process: 0.78063 | load: 0.00304 [I 2022-10-08 21:01:50] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-08 21:01:50] ...optimizers/trijoint.py.151: total_norm: 2.083323 [I 2022-10-08 21:01:58] ...trap/engines/engine.py.224: train: epoch 19 | batch 2030/2304 [I 2022-10-08 21:01:58] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:03:33 [I 2022-10-08 21:01:58] ...trap/engines/engine.py.229: process: 0.78068 | load: 0.00307 [I 2022-10-08 21:01:58] ...trap/engines/engine.py.230: loss: 0.15937 [I 2022-10-08 21:01:58] ...optimizers/trijoint.py.151: total_norm: 1.859163 [I 2022-10-08 21:02:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 2040/2304 [I 2022-10-08 21:02:06] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:03:25 [I 2022-10-08 21:02:06] ...trap/engines/engine.py.229: process: 0.77995 | load: 0.00302 [I 2022-10-08 21:02:06] ...trap/engines/engine.py.230: loss: 0.19696 [I 2022-10-08 21:02:06] ...optimizers/trijoint.py.151: total_norm: 2.849372 [I 2022-10-08 21:02:13] ...trap/engines/engine.py.224: train: epoch 19 | batch 2050/2304 [I 2022-10-08 21:02:13] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:03:17 [I 2022-10-08 21:02:13] ...trap/engines/engine.py.229: process: 0.77653 | load: 0.00303 [I 2022-10-08 21:02:13] ...trap/engines/engine.py.230: loss: 0.16773 [I 2022-10-08 21:02:13] ...optimizers/trijoint.py.151: total_norm: 3.045012 [I 2022-10-08 21:02:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 2060/2304 [I 2022-10-08 21:02:21] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:03:09 [I 2022-10-08 21:02:21] ...trap/engines/engine.py.229: process: 0.76312 | load: 0.00302 [I 2022-10-08 21:02:21] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-08 21:02:21] ...optimizers/trijoint.py.151: total_norm: 2.076684 [I 2022-10-08 21:02:29] ...trap/engines/engine.py.224: train: epoch 19 | batch 2070/2304 [I 2022-10-08 21:02:29] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:03:02 [I 2022-10-08 21:02:29] ...trap/engines/engine.py.229: process: 0.78849 | load: 0.00301 [I 2022-10-08 21:02:29] ...trap/engines/engine.py.230: loss: 0.19698 [I 2022-10-08 21:02:29] ...optimizers/trijoint.py.151: total_norm: 2.643507 [I 2022-10-08 21:02:37] ...trap/engines/engine.py.224: train: epoch 19 | batch 2080/2304 [I 2022-10-08 21:02:37] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:02:54 [I 2022-10-08 21:02:37] ...trap/engines/engine.py.229: process: 0.76771 | load: 0.00308 [I 2022-10-08 21:02:37] ...trap/engines/engine.py.230: loss: 0.21290 [I 2022-10-08 21:02:37] ...optimizers/trijoint.py.151: total_norm: 2.368097 [I 2022-10-08 21:02:45] ...trap/engines/engine.py.224: train: epoch 19 | batch 2090/2304 [I 2022-10-08 21:02:45] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:02:46 [I 2022-10-08 21:02:45] ...trap/engines/engine.py.229: process: 0.79243 | load: 0.00315 [I 2022-10-08 21:02:45] ...trap/engines/engine.py.230: loss: 0.15276 [I 2022-10-08 21:02:45] ...optimizers/trijoint.py.151: total_norm: 2.271838 [I 2022-10-08 21:02:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 2100/2304 [I 2022-10-08 21:02:52] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:02:38 [I 2022-10-08 21:02:52] ...trap/engines/engine.py.229: process: 0.76926 | load: 0.00295 [I 2022-10-08 21:02:52] ...trap/engines/engine.py.230: loss: 0.20456 [I 2022-10-08 21:02:52] ...optimizers/trijoint.py.151: total_norm: 3.705307 [I 2022-10-08 21:03:00] ...trap/engines/engine.py.224: train: epoch 19 | batch 2110/2304 [I 2022-10-08 21:03:00] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:02:30 [I 2022-10-08 21:03:00] ...trap/engines/engine.py.229: process: 0.77250 | load: 0.00300 [I 2022-10-08 21:03:00] ...trap/engines/engine.py.230: loss: 0.21311 [I 2022-10-08 21:03:00] ...optimizers/trijoint.py.151: total_norm: 3.338696 [I 2022-10-08 21:03:08] ...trap/engines/engine.py.224: train: epoch 19 | batch 2120/2304 [I 2022-10-08 21:03:08] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:02:22 [I 2022-10-08 21:03:08] ...trap/engines/engine.py.229: process: 0.77561 | load: 0.00300 [I 2022-10-08 21:03:08] ...trap/engines/engine.py.230: loss: 0.14186 [I 2022-10-08 21:03:08] ...optimizers/trijoint.py.151: total_norm: 2.265383 [I 2022-10-08 21:03:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 2130/2304 [I 2022-10-08 21:03:16] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:02:15 [I 2022-10-08 21:03:16] ...trap/engines/engine.py.229: process: 0.78016 | load: 0.00299 [I 2022-10-08 21:03:16] ...trap/engines/engine.py.230: loss: 0.11280 [I 2022-10-08 21:03:16] ...optimizers/trijoint.py.151: total_norm: 1.645851 [I 2022-10-08 21:03:23] ...trap/engines/engine.py.224: train: epoch 19 | batch 2140/2304 [I 2022-10-08 21:03:23] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:02:06 [I 2022-10-08 21:03:23] ...trap/engines/engine.py.229: process: 0.77230 | load: 0.00292 [I 2022-10-08 21:03:23] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-08 21:03:23] ...optimizers/trijoint.py.151: total_norm: 1.800782 [I 2022-10-08 21:03:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 2150/2304 [I 2022-10-08 21:03:31] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:01:59 [I 2022-10-08 21:03:31] ...trap/engines/engine.py.229: process: 0.76920 | load: 0.00301 [I 2022-10-08 21:03:31] ...trap/engines/engine.py.230: loss: 0.22679 [I 2022-10-08 21:03:31] ...optimizers/trijoint.py.151: total_norm: 2.916327 [I 2022-10-08 21:03:39] ...trap/engines/engine.py.224: train: epoch 19 | batch 2160/2304 [I 2022-10-08 21:03:39] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:01:51 [I 2022-10-08 21:03:39] ...trap/engines/engine.py.229: process: 0.80466 | load: 0.00299 [I 2022-10-08 21:03:39] ...trap/engines/engine.py.230: loss: 0.16728 [I 2022-10-08 21:03:39] ...optimizers/trijoint.py.151: total_norm: 3.318448 [I 2022-10-08 21:03:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 2170/2304 [I 2022-10-08 21:03:47] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:01:43 [I 2022-10-08 21:03:47] ...trap/engines/engine.py.229: process: 0.78239 | load: 0.00298 [I 2022-10-08 21:03:47] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-08 21:03:47] ...optimizers/trijoint.py.151: total_norm: 2.729361 [I 2022-10-08 21:03:54] ...trap/engines/engine.py.224: train: epoch 19 | batch 2180/2304 [I 2022-10-08 21:03:54] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:01:35 [I 2022-10-08 21:03:54] ...trap/engines/engine.py.229: process: 0.76340 | load: 0.00297 [I 2022-10-08 21:03:54] ...trap/engines/engine.py.230: loss: 0.23038 [I 2022-10-08 21:03:54] ...optimizers/trijoint.py.151: total_norm: 3.075965 [I 2022-10-08 21:04:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 2190/2304 [I 2022-10-08 21:04:02] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:01:28 [I 2022-10-08 21:04:02] ...trap/engines/engine.py.229: process: 0.77496 | load: 0.00300 [I 2022-10-08 21:04:02] ...trap/engines/engine.py.230: loss: 0.16872 [I 2022-10-08 21:04:02] ...optimizers/trijoint.py.151: total_norm: 3.125670 [I 2022-10-08 21:04:10] ...trap/engines/engine.py.224: train: epoch 19 | batch 2200/2304 [I 2022-10-08 21:04:10] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:01:20 [I 2022-10-08 21:04:10] ...trap/engines/engine.py.229: process: 0.77544 | load: 0.00304 [I 2022-10-08 21:04:10] ...trap/engines/engine.py.230: loss: 0.19075 [I 2022-10-08 21:04:10] ...optimizers/trijoint.py.151: total_norm: 2.963478 [I 2022-10-08 21:04:18] ...trap/engines/engine.py.224: train: epoch 19 | batch 2210/2304 [I 2022-10-08 21:04:18] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:01:12 [I 2022-10-08 21:04:18] ...trap/engines/engine.py.229: process: 0.77417 | load: 0.00301 [I 2022-10-08 21:04:18] ...trap/engines/engine.py.230: loss: 0.17822 [I 2022-10-08 21:04:18] ...optimizers/trijoint.py.151: total_norm: 2.120304 [I 2022-10-08 21:04:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 2220/2304 [I 2022-10-08 21:04:26] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:01:05 [I 2022-10-08 21:04:26] ...trap/engines/engine.py.229: process: 0.77833 | load: 0.00300 [I 2022-10-08 21:04:26] ...trap/engines/engine.py.230: loss: 0.16187 [I 2022-10-08 21:04:26] ...optimizers/trijoint.py.151: total_norm: 3.223784 [I 2022-10-08 21:04:34] ...trap/engines/engine.py.224: train: epoch 19 | batch 2230/2304 [I 2022-10-08 21:04:34] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:00:57 [I 2022-10-08 21:04:34] ...trap/engines/engine.py.229: process: 0.76472 | load: 0.00299 [I 2022-10-08 21:04:34] ...trap/engines/engine.py.230: loss: 0.28471 [I 2022-10-08 21:04:34] ...optimizers/trijoint.py.151: total_norm: 3.461670 [I 2022-10-08 21:04:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 2240/2304 [I 2022-10-08 21:04:41] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:00:49 [I 2022-10-08 21:04:42] ...trap/engines/engine.py.229: process: 0.78086 | load: 0.00300 [I 2022-10-08 21:04:42] ...trap/engines/engine.py.230: loss: 0.19326 [I 2022-10-08 21:04:42] ...optimizers/trijoint.py.151: total_norm: 3.122835 [I 2022-10-08 21:04:49] ...trap/engines/engine.py.224: train: epoch 19 | batch 2250/2304 [I 2022-10-08 21:04:49] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:00:41 [I 2022-10-08 21:04:49] ...trap/engines/engine.py.229: process: 0.77095 | load: 0.00298 [I 2022-10-08 21:04:49] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-08 21:04:49] ...optimizers/trijoint.py.151: total_norm: 2.210369 [I 2022-10-08 21:04:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 2260/2304 [I 2022-10-08 21:04:57] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:00:34 [I 2022-10-08 21:04:57] ...trap/engines/engine.py.229: process: 0.77482 | load: 0.00303 [I 2022-10-08 21:04:57] ...trap/engines/engine.py.230: loss: 0.24706 [I 2022-10-08 21:04:57] ...optimizers/trijoint.py.151: total_norm: 3.231970 [I 2022-10-08 21:05:05] ...trap/engines/engine.py.224: train: epoch 19 | batch 2270/2304 [I 2022-10-08 21:05:05] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:00:26 [I 2022-10-08 21:05:05] ...trap/engines/engine.py.229: process: 0.76808 | load: 0.00299 [I 2022-10-08 21:05:05] ...trap/engines/engine.py.230: loss: 0.16850 [I 2022-10-08 21:05:05] ...optimizers/trijoint.py.151: total_norm: 2.943396 [I 2022-10-08 21:05:13] ...trap/engines/engine.py.224: train: epoch 19 | batch 2280/2304 [I 2022-10-08 21:05:13] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:00:18 [I 2022-10-08 21:05:13] ...trap/engines/engine.py.229: process: 0.77272 | load: 0.00305 [I 2022-10-08 21:05:13] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-08 21:05:13] ...optimizers/trijoint.py.151: total_norm: 1.620438 [I 2022-10-08 21:05:20] ...trap/engines/engine.py.224: train: epoch 19 | batch 2290/2304 [I 2022-10-08 21:05:20] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:00:10 [I 2022-10-08 21:05:20] ...trap/engines/engine.py.229: process: 0.77359 | load: 0.00299 [I 2022-10-08 21:05:20] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-08 21:05:20] ...optimizers/trijoint.py.151: total_norm: 2.501103 [I 2022-10-08 21:05:28] ...trap/engines/engine.py.224: train: epoch 19 | batch 2300/2304 [I 2022-10-08 21:05:28] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:00:03 [I 2022-10-08 21:05:28] ...trap/engines/engine.py.229: process: 0.76937 | load: 0.00016 [I 2022-10-08 21:05:28] ...trap/engines/engine.py.230: loss: 0.23916 [I 2022-10-08 21:05:28] ...optimizers/trijoint.py.151: total_norm: 3.647505 [I 2022-10-08 21:05:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 2304/2304 [I 2022-10-08 21:05:31] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:00:00 [I 2022-10-08 21:05:31] ...trap/engines/engine.py.229: process: 0.75842 | load: 0.00016 [I 2022-10-08 21:05:31] ...trap/engines/engine.py.230: loss: 0.14440 [I 2022-10-08 21:05:31] ...optimizers/trijoint.py.151: total_norm: 2.396056 [S 2022-10-08 21:05:32] ...trap/engines/engine.py.129: train_epoch.epoch: 19 [S 2022-10-08 21:05:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.17994417916344976 [I 2022-10-08 21:05:32] ...s/criterions/triplet.py.84: margin = 0.15000000000000008 [I 2022-10-08 21:05:33] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 19 [I 2022-10-08 21:05:33] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 21:05:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 21:05:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 21:05:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 21:05:35] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 21:05:36] ...trap/engines/engine.py.321: eval: epoch 19 | batch 0/510 [I 2022-10-08 21:05:36] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:25:30 [I 2022-10-08 21:05:36] ...trap/engines/engine.py.326: process: 3.00144 | load: 2.47309 [I 2022-10-08 21:05:40] ...trap/engines/engine.py.321: eval: epoch 19 | batch 10/510 [I 2022-10-08 21:05:40] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:15 [I 2022-10-08 21:05:40] ...trap/engines/engine.py.326: process: 0.41838 | load: 0.00020 [I 2022-10-08 21:05:46] ...trap/engines/engine.py.321: eval: epoch 19 | batch 20/510 [I 2022-10-08 21:05:46] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:59 [I 2022-10-08 21:05:46] ...trap/engines/engine.py.326: process: 0.60770 | load: 0.00020 [I 2022-10-08 21:05:52] ...trap/engines/engine.py.321: eval: epoch 19 | batch 30/510 [I 2022-10-08 21:05:52] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:00 [I 2022-10-08 21:05:52] ...trap/engines/engine.py.326: process: 0.70853 | load: 0.00019 [I 2022-10-08 21:05:59] ...trap/engines/engine.py.321: eval: epoch 19 | batch 40/510 [I 2022-10-08 21:05:59] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:36 [I 2022-10-08 21:05:59] ...trap/engines/engine.py.326: process: 0.63566 | load: 0.00019 [I 2022-10-08 21:06:06] ...trap/engines/engine.py.321: eval: epoch 19 | batch 50/510 [I 2022-10-08 21:06:06] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:05:26 [I 2022-10-08 21:06:06] ...trap/engines/engine.py.326: process: 0.62744 | load: 0.00019 [I 2022-10-08 21:06:12] ...trap/engines/engine.py.321: eval: epoch 19 | batch 60/510 [I 2022-10-08 21:06:12] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:26 [I 2022-10-08 21:06:12] ...trap/engines/engine.py.326: process: 0.58601 | load: 0.00019 [I 2022-10-08 21:06:17] ...trap/engines/engine.py.321: eval: epoch 19 | batch 70/510 [I 2022-10-08 21:06:17] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:03 [I 2022-10-08 21:06:17] ...trap/engines/engine.py.326: process: 0.52948 | load: 0.00019 [I 2022-10-08 21:06:23] ...trap/engines/engine.py.321: eval: epoch 19 | batch 80/510 [I 2022-10-08 21:06:23] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:43 [I 2022-10-08 21:06:23] ...trap/engines/engine.py.326: process: 0.47691 | load: 0.00019 [I 2022-10-08 21:06:27] ...trap/engines/engine.py.321: eval: epoch 19 | batch 90/510 [I 2022-10-08 21:06:27] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:20 [I 2022-10-08 21:06:27] ...trap/engines/engine.py.326: process: 0.49604 | load: 0.00019 [I 2022-10-08 21:06:32] ...trap/engines/engine.py.321: eval: epoch 19 | batch 100/510 [I 2022-10-08 21:06:32] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:56 [I 2022-10-08 21:06:32] ...trap/engines/engine.py.326: process: 0.31084 | load: 0.00019 [I 2022-10-08 21:06:35] ...trap/engines/engine.py.321: eval: epoch 19 | batch 110/510 [I 2022-10-08 21:06:35] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:11 [I 2022-10-08 21:06:35] ...trap/engines/engine.py.326: process: 0.32142 | load: 0.00020 [I 2022-10-08 21:06:38] ...trap/engines/engine.py.321: eval: epoch 19 | batch 120/510 [I 2022-10-08 21:06:38] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:03 [I 2022-10-08 21:06:38] ...trap/engines/engine.py.326: process: 0.31559 | load: 0.00019 [I 2022-10-08 21:06:41] ...trap/engines/engine.py.321: eval: epoch 19 | batch 130/510 [I 2022-10-08 21:06:41] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:00 [I 2022-10-08 21:06:41] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00021 [I 2022-10-08 21:06:44] ...trap/engines/engine.py.321: eval: epoch 19 | batch 140/510 [I 2022-10-08 21:06:44] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:56 [I 2022-10-08 21:06:44] ...trap/engines/engine.py.326: process: 0.31158 | load: 0.00019 [I 2022-10-08 21:06:48] ...trap/engines/engine.py.321: eval: epoch 19 | batch 150/510 [I 2022-10-08 21:06:48] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:53 [I 2022-10-08 21:06:48] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00020 [I 2022-10-08 21:06:51] ...trap/engines/engine.py.321: eval: epoch 19 | batch 160/510 [I 2022-10-08 21:06:51] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:50 [I 2022-10-08 21:06:51] ...trap/engines/engine.py.326: process: 0.31793 | load: 0.00019 [I 2022-10-08 21:06:54] ...trap/engines/engine.py.321: eval: epoch 19 | batch 170/510 [I 2022-10-08 21:06:54] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:47 [I 2022-10-08 21:06:54] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00019 [I 2022-10-08 21:06:57] ...trap/engines/engine.py.321: eval: epoch 19 | batch 180/510 [I 2022-10-08 21:06:57] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:44 [I 2022-10-08 21:06:57] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00019 [I 2022-10-08 21:07:00] ...trap/engines/engine.py.321: eval: epoch 19 | batch 190/510 [I 2022-10-08 21:07:00] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:41 [I 2022-10-08 21:07:00] ...trap/engines/engine.py.326: process: 0.31649 | load: 0.00020 [I 2022-10-08 21:07:03] ...trap/engines/engine.py.321: eval: epoch 19 | batch 200/510 [I 2022-10-08 21:07:03] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:37 [I 2022-10-08 21:07:03] ...trap/engines/engine.py.326: process: 0.31259 | load: 0.00019 [I 2022-10-08 21:07:07] ...trap/engines/engine.py.321: eval: epoch 19 | batch 210/510 [I 2022-10-08 21:07:07] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:35 [I 2022-10-08 21:07:07] ...trap/engines/engine.py.326: process: 0.31438 | load: 0.00020 [I 2022-10-08 21:07:10] ...trap/engines/engine.py.321: eval: epoch 19 | batch 220/510 [I 2022-10-08 21:07:10] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:32 [I 2022-10-08 21:07:10] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00021 [I 2022-10-08 21:07:13] ...trap/engines/engine.py.321: eval: epoch 19 | batch 230/510 [I 2022-10-08 21:07:13] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:29 [I 2022-10-08 21:07:13] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00020 [I 2022-10-08 21:07:16] ...trap/engines/engine.py.321: eval: epoch 19 | batch 240/510 [I 2022-10-08 21:07:16] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:25 [I 2022-10-08 21:07:16] ...trap/engines/engine.py.326: process: 0.31775 | load: 0.00019 [I 2022-10-08 21:07:19] ...trap/engines/engine.py.321: eval: epoch 19 | batch 250/510 [I 2022-10-08 21:07:19] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:22 [I 2022-10-08 21:07:19] ...trap/engines/engine.py.326: process: 0.31409 | load: 0.00019 [I 2022-10-08 21:07:23] ...trap/engines/engine.py.321: eval: epoch 19 | batch 260/510 [I 2022-10-08 21:07:23] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:19 [I 2022-10-08 21:07:23] ...trap/engines/engine.py.326: process: 0.31536 | load: 0.00020 [I 2022-10-08 21:07:26] ...trap/engines/engine.py.321: eval: epoch 19 | batch 270/510 [I 2022-10-08 21:07:26] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:16 [I 2022-10-08 21:07:26] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00020 [I 2022-10-08 21:07:29] ...trap/engines/engine.py.321: eval: epoch 19 | batch 280/510 [I 2022-10-08 21:07:29] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:12 [I 2022-10-08 21:07:29] ...trap/engines/engine.py.326: process: 0.31301 | load: 0.00020 [I 2022-10-08 21:07:32] ...trap/engines/engine.py.321: eval: epoch 19 | batch 290/510 [I 2022-10-08 21:07:32] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:09 [I 2022-10-08 21:07:32] ...trap/engines/engine.py.326: process: 0.31699 | load: 0.00020 [I 2022-10-08 21:07:35] ...trap/engines/engine.py.321: eval: epoch 19 | batch 300/510 [I 2022-10-08 21:07:35] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:06 [I 2022-10-08 21:07:35] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00020 [I 2022-10-08 21:07:38] ...trap/engines/engine.py.321: eval: epoch 19 | batch 310/510 [I 2022-10-08 21:07:38] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:03 [I 2022-10-08 21:07:38] ...trap/engines/engine.py.326: process: 0.31555 | load: 0.00021 [I 2022-10-08 21:07:42] ...trap/engines/engine.py.321: eval: epoch 19 | batch 320/510 [I 2022-10-08 21:07:42] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:00 [I 2022-10-08 21:07:42] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00020 [I 2022-10-08 21:07:45] ...trap/engines/engine.py.321: eval: epoch 19 | batch 330/510 [I 2022-10-08 21:07:45] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:57 [I 2022-10-08 21:07:45] ...trap/engines/engine.py.326: process: 0.31902 | load: 0.00021 [I 2022-10-08 21:07:48] ...trap/engines/engine.py.321: eval: epoch 19 | batch 340/510 [I 2022-10-08 21:07:48] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:54 [I 2022-10-08 21:07:48] ...trap/engines/engine.py.326: process: 0.31832 | load: 0.00020 [I 2022-10-08 21:07:51] ...trap/engines/engine.py.321: eval: epoch 19 | batch 350/510 [I 2022-10-08 21:07:51] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:51 [I 2022-10-08 21:07:51] ...trap/engines/engine.py.326: process: 0.32305 | load: 0.00022 [I 2022-10-08 21:07:54] ...trap/engines/engine.py.321: eval: epoch 19 | batch 360/510 [I 2022-10-08 21:07:54] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:48 [I 2022-10-08 21:07:54] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00019 [I 2022-10-08 21:07:58] ...trap/engines/engine.py.321: eval: epoch 19 | batch 370/510 [I 2022-10-08 21:07:58] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:44 [I 2022-10-08 21:07:58] ...trap/engines/engine.py.326: process: 0.32606 | load: 0.00021 [I 2022-10-08 21:08:01] ...trap/engines/engine.py.321: eval: epoch 19 | batch 380/510 [I 2022-10-08 21:08:01] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-08 21:08:01] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-08 21:08:04] ...trap/engines/engine.py.321: eval: epoch 19 | batch 390/510 [I 2022-10-08 21:08:04] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:38 [I 2022-10-08 21:08:04] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00022 [I 2022-10-08 21:08:07] ...trap/engines/engine.py.321: eval: epoch 19 | batch 400/510 [I 2022-10-08 21:08:07] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:35 [I 2022-10-08 21:08:07] ...trap/engines/engine.py.326: process: 0.32145 | load: 0.00022 [I 2022-10-08 21:08:10] ...trap/engines/engine.py.321: eval: epoch 19 | batch 410/510 [I 2022-10-08 21:08:10] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:32 [I 2022-10-08 21:08:10] ...trap/engines/engine.py.326: process: 0.32057 | load: 0.00021 [I 2022-10-08 21:08:14] ...trap/engines/engine.py.321: eval: epoch 19 | batch 420/510 [I 2022-10-08 21:08:14] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:28 [I 2022-10-08 21:08:14] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00021 [I 2022-10-08 21:08:17] ...trap/engines/engine.py.321: eval: epoch 19 | batch 430/510 [I 2022-10-08 21:08:17] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-08 21:08:17] ...trap/engines/engine.py.326: process: 0.32367 | load: 0.00021 [I 2022-10-08 21:08:20] ...trap/engines/engine.py.321: eval: epoch 19 | batch 440/510 [I 2022-10-08 21:08:20] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:22 [I 2022-10-08 21:08:20] ...trap/engines/engine.py.326: process: 0.32477 | load: 0.00020 [I 2022-10-08 21:08:23] ...trap/engines/engine.py.321: eval: epoch 19 | batch 450/510 [I 2022-10-08 21:08:23] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:19 [I 2022-10-08 21:08:23] ...trap/engines/engine.py.326: process: 0.32319 | load: 0.00023 [I 2022-10-08 21:08:27] ...trap/engines/engine.py.321: eval: epoch 19 | batch 460/510 [I 2022-10-08 21:08:27] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:16 [I 2022-10-08 21:08:27] ...trap/engines/engine.py.326: process: 0.32577 | load: 0.00020 [I 2022-10-08 21:08:30] ...trap/engines/engine.py.321: eval: epoch 19 | batch 470/510 [I 2022-10-08 21:08:30] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-08 21:08:30] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00020 [I 2022-10-08 21:08:33] ...trap/engines/engine.py.321: eval: epoch 19 | batch 480/510 [I 2022-10-08 21:08:33] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-08 21:08:33] ...trap/engines/engine.py.326: process: 0.31593 | load: 0.00020 [I 2022-10-08 21:08:36] ...trap/engines/engine.py.321: eval: epoch 19 | batch 490/510 [I 2022-10-08 21:08:36] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-08 21:08:36] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00022 [I 2022-10-08 21:08:39] ...trap/engines/engine.py.321: eval: epoch 19 | batch 500/510 [I 2022-10-08 21:08:39] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:03 [I 2022-10-08 21:08:39] ...trap/engines/engine.py.326: process: 0.32651 | load: 0.00020 [I 2022-10-08 21:08:43] ...trap/engines/engine.py.321: eval: epoch 19 | batch 510/510 [I 2022-10-08 21:08:43] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:00 [I 2022-10-08 21:08:43] ...trap/engines/engine.py.326: process: 0.30943 | load: 0.00016 [S 2022-10-08 21:08:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 19 [I 2022-10-08 21:08:43] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.618500 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.859900 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.909700 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.607900 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.859800 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.908500 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010122 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.014734 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.015054 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014501 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013106 [S 2022-10-08 21:09:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010717 [I 2022-10-08 21:09:24] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 21:09:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 21:09:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 21:09:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 21:09:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 21:09:25] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 21:09:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 21:09:32] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 21:09:32] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 21:09:32] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 21:09:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 21:09:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 21:09:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 20 [I 2022-10-08 21:09:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 21:09:40] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-08 21:09:40] ...optimizers/trijoint.py.189: Switched to optimizer recipe&image [I 2022-10-08 21:09:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 0/2304 [I 2022-10-08 21:09:44] ...trap/engines/engine.py.228: elapsed: 0:00:04 | left: 2:46:42 [I 2022-10-08 21:09:44] ...trap/engines/engine.py.229: process: 4.34123 | load: 2.12863 [I 2022-10-08 21:09:44] ...trap/engines/engine.py.230: loss: 0.24228 [I 2022-10-08 21:09:44] ...optimizers/trijoint.py.151: total_norm: 24.884672 [I 2022-10-08 21:09:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 10/2304 [I 2022-10-08 21:09:56] ...trap/engines/engine.py.228: elapsed: 0:00:16 | left: 0:53:21 [I 2022-10-08 21:09:56] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00461 [I 2022-10-08 21:09:56] ...trap/engines/engine.py.230: loss: 0.18766 [I 2022-10-08 21:09:56] ...optimizers/trijoint.py.151: total_norm: 16.900215 [I 2022-10-08 21:10:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 20/2304 [I 2022-10-08 21:10:05] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:35:38 [I 2022-10-08 21:10:05] ...trap/engines/engine.py.229: process: 0.81483 | load: 0.00514 [I 2022-10-08 21:10:05] ...trap/engines/engine.py.230: loss: 0.16347 [I 2022-10-08 21:10:05] ...optimizers/trijoint.py.151: total_norm: 33.265316 [I 2022-10-08 21:10:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 30/2304 [I 2022-10-08 21:10:14] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:32:12 [I 2022-10-08 21:10:14] ...trap/engines/engine.py.229: process: 0.84097 | load: 0.00370 [I 2022-10-08 21:10:14] ...trap/engines/engine.py.230: loss: 0.23351 [I 2022-10-08 21:10:14] ...optimizers/trijoint.py.151: total_norm: 20.767780 [I 2022-10-08 21:10:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 40/2304 [I 2022-10-08 21:10:22] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:31:56 [I 2022-10-08 21:10:22] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00414 [I 2022-10-08 21:10:22] ...trap/engines/engine.py.230: loss: 0.21283 [I 2022-10-08 21:10:22] ...optimizers/trijoint.py.151: total_norm: 26.035589 [I 2022-10-08 21:10:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 50/2304 [I 2022-10-08 21:10:31] ...trap/engines/engine.py.228: elapsed: 0:00:51 | left: 0:32:21 [I 2022-10-08 21:10:31] ...trap/engines/engine.py.229: process: 0.83854 | load: 0.00470 [I 2022-10-08 21:10:31] ...trap/engines/engine.py.230: loss: 0.23463 [I 2022-10-08 21:10:31] ...optimizers/trijoint.py.151: total_norm: 35.158028 [I 2022-10-08 21:10:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 60/2304 [I 2022-10-08 21:10:40] ...trap/engines/engine.py.228: elapsed: 0:01:00 | left: 0:31:29 [I 2022-10-08 21:10:40] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00350 [I 2022-10-08 21:10:40] ...trap/engines/engine.py.230: loss: 0.10504 [I 2022-10-08 21:10:40] ...optimizers/trijoint.py.151: total_norm: 15.437305 [I 2022-10-08 21:10:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 70/2304 [I 2022-10-08 21:10:48] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:31:13 [I 2022-10-08 21:10:48] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00361 [I 2022-10-08 21:10:48] ...trap/engines/engine.py.230: loss: 0.18620 [I 2022-10-08 21:10:48] ...optimizers/trijoint.py.151: total_norm: 23.753735 [I 2022-10-08 21:10:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 80/2304 [I 2022-10-08 21:10:57] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:31:53 [I 2022-10-08 21:10:57] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00321 [I 2022-10-08 21:10:57] ...trap/engines/engine.py.230: loss: 0.19689 [I 2022-10-08 21:10:57] ...optimizers/trijoint.py.151: total_norm: 25.244692 [I 2022-10-08 21:11:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 90/2304 [I 2022-10-08 21:11:06] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:31:02 [I 2022-10-08 21:11:06] ...trap/engines/engine.py.229: process: 0.83573 | load: 0.00358 [I 2022-10-08 21:11:06] ...trap/engines/engine.py.230: loss: 0.14797 [I 2022-10-08 21:11:06] ...optimizers/trijoint.py.151: total_norm: 27.142303 [I 2022-10-08 21:11:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 100/2304 [I 2022-10-08 21:11:14] ...trap/engines/engine.py.228: elapsed: 0:01:34 | left: 0:30:44 [I 2022-10-08 21:11:14] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00340 [I 2022-10-08 21:11:14] ...trap/engines/engine.py.230: loss: 0.15798 [I 2022-10-08 21:11:14] ...optimizers/trijoint.py.151: total_norm: 28.247841 [I 2022-10-08 21:11:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 110/2304 [I 2022-10-08 21:11:22] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:30:39 [I 2022-10-08 21:11:22] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00334 [I 2022-10-08 21:11:22] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-08 21:11:22] ...optimizers/trijoint.py.151: total_norm: 31.129318 [I 2022-10-08 21:11:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 120/2304 [I 2022-10-08 21:11:31] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:30:35 [I 2022-10-08 21:11:31] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00334 [I 2022-10-08 21:11:31] ...trap/engines/engine.py.230: loss: 0.25321 [I 2022-10-08 21:11:31] ...optimizers/trijoint.py.151: total_norm: 20.800344 [I 2022-10-08 21:11:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 130/2304 [I 2022-10-08 21:11:39] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:30:33 [I 2022-10-08 21:11:39] ...trap/engines/engine.py.229: process: 0.83636 | load: 0.00324 [I 2022-10-08 21:11:39] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-08 21:11:39] ...optimizers/trijoint.py.151: total_norm: 19.007635 [I 2022-10-08 21:11:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 140/2304 [I 2022-10-08 21:11:48] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:30:15 [I 2022-10-08 21:11:48] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00319 [I 2022-10-08 21:11:48] ...trap/engines/engine.py.230: loss: 0.15271 [I 2022-10-08 21:11:48] ...optimizers/trijoint.py.151: total_norm: 15.387039 [I 2022-10-08 21:11:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 150/2304 [I 2022-10-08 21:11:56] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:30:03 [I 2022-10-08 21:11:56] ...trap/engines/engine.py.229: process: 0.82723 | load: 0.00314 [I 2022-10-08 21:11:56] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-08 21:11:56] ...optimizers/trijoint.py.151: total_norm: 22.209665 [I 2022-10-08 21:12:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 160/2304 [I 2022-10-08 21:12:05] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:29:58 [I 2022-10-08 21:12:05] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00349 [I 2022-10-08 21:12:05] ...trap/engines/engine.py.230: loss: 0.21102 [I 2022-10-08 21:12:05] ...optimizers/trijoint.py.151: total_norm: 20.155352 [I 2022-10-08 21:12:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 170/2304 [I 2022-10-08 21:12:13] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:29:46 [I 2022-10-08 21:12:13] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00314 [I 2022-10-08 21:12:13] ...trap/engines/engine.py.230: loss: 0.24599 [I 2022-10-08 21:12:13] ...optimizers/trijoint.py.151: total_norm: 32.700935 [I 2022-10-08 21:12:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 180/2304 [I 2022-10-08 21:12:21] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:29:40 [I 2022-10-08 21:12:21] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00335 [I 2022-10-08 21:12:21] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-08 21:12:21] ...optimizers/trijoint.py.151: total_norm: 16.187067 [I 2022-10-08 21:12:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 190/2304 [I 2022-10-08 21:12:30] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:29:37 [I 2022-10-08 21:12:30] ...trap/engines/engine.py.229: process: 0.84467 | load: 0.00323 [I 2022-10-08 21:12:30] ...trap/engines/engine.py.230: loss: 0.13277 [I 2022-10-08 21:12:30] ...optimizers/trijoint.py.151: total_norm: 16.401682 [I 2022-10-08 21:12:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 200/2304 [I 2022-10-08 21:12:38] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:29:23 [I 2022-10-08 21:12:38] ...trap/engines/engine.py.229: process: 0.82995 | load: 0.00319 [I 2022-10-08 21:12:38] ...trap/engines/engine.py.230: loss: 0.11501 [I 2022-10-08 21:12:38] ...optimizers/trijoint.py.151: total_norm: 14.947124 [I 2022-10-08 21:12:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 210/2304 [I 2022-10-08 21:12:47] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:29:14 [I 2022-10-08 21:12:47] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00300 [I 2022-10-08 21:12:47] ...trap/engines/engine.py.230: loss: 0.21077 [I 2022-10-08 21:12:47] ...optimizers/trijoint.py.151: total_norm: 22.581614 [I 2022-10-08 21:12:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 220/2304 [I 2022-10-08 21:12:55] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:29:22 [I 2022-10-08 21:12:55] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00311 [I 2022-10-08 21:12:55] ...trap/engines/engine.py.230: loss: 0.22514 [I 2022-10-08 21:12:55] ...optimizers/trijoint.py.151: total_norm: 26.591204 [I 2022-10-08 21:13:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 230/2304 [I 2022-10-08 21:13:04] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:29:03 [I 2022-10-08 21:13:04] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00317 [I 2022-10-08 21:13:04] ...trap/engines/engine.py.230: loss: 0.15434 [I 2022-10-08 21:13:04] ...optimizers/trijoint.py.151: total_norm: 14.787482 [I 2022-10-08 21:13:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 240/2304 [I 2022-10-08 21:13:12] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:28:55 [I 2022-10-08 21:13:12] ...trap/engines/engine.py.229: process: 0.83626 | load: 0.00298 [I 2022-10-08 21:13:12] ...trap/engines/engine.py.230: loss: 0.20097 [I 2022-10-08 21:13:12] ...optimizers/trijoint.py.151: total_norm: 15.125101 [I 2022-10-08 21:13:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 250/2304 [I 2022-10-08 21:13:21] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:52 [I 2022-10-08 21:13:21] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00309 [I 2022-10-08 21:13:21] ...trap/engines/engine.py.230: loss: 0.20825 [I 2022-10-08 21:13:21] ...optimizers/trijoint.py.151: total_norm: 14.771720 [I 2022-10-08 21:13:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 260/2304 [I 2022-10-08 21:13:29] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:28:39 [I 2022-10-08 21:13:29] ...trap/engines/engine.py.229: process: 0.84251 | load: 0.00316 [I 2022-10-08 21:13:29] ...trap/engines/engine.py.230: loss: 0.20260 [I 2022-10-08 21:13:29] ...optimizers/trijoint.py.151: total_norm: 26.822771 [I 2022-10-08 21:13:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 270/2304 [I 2022-10-08 21:13:38] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:25 [I 2022-10-08 21:13:38] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00302 [I 2022-10-08 21:13:38] ...trap/engines/engine.py.230: loss: 0.14801 [I 2022-10-08 21:13:38] ...optimizers/trijoint.py.151: total_norm: 13.246304 [I 2022-10-08 21:13:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 280/2304 [I 2022-10-08 21:13:46] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:28:29 [I 2022-10-08 21:13:46] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00322 [I 2022-10-08 21:13:46] ...trap/engines/engine.py.230: loss: 0.23138 [I 2022-10-08 21:13:46] ...optimizers/trijoint.py.151: total_norm: 19.362213 [I 2022-10-08 21:13:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 290/2304 [I 2022-10-08 21:13:55] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:09 [I 2022-10-08 21:13:55] ...trap/engines/engine.py.229: process: 0.83329 | load: 0.00310 [I 2022-10-08 21:13:55] ...trap/engines/engine.py.230: loss: 0.14840 [I 2022-10-08 21:13:55] ...optimizers/trijoint.py.151: total_norm: 19.568413 [I 2022-10-08 21:14:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 300/2304 [I 2022-10-08 21:14:03] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:27:58 [I 2022-10-08 21:14:03] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00298 [I 2022-10-08 21:14:03] ...trap/engines/engine.py.230: loss: 0.22471 [I 2022-10-08 21:14:03] ...optimizers/trijoint.py.151: total_norm: 42.178181 [I 2022-10-08 21:14:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 310/2304 [I 2022-10-08 21:14:11] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:27:52 [I 2022-10-08 21:14:11] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00310 [I 2022-10-08 21:14:11] ...trap/engines/engine.py.230: loss: 0.12884 [I 2022-10-08 21:14:11] ...optimizers/trijoint.py.151: total_norm: 20.517712 [I 2022-10-08 21:14:20] ...trap/engines/engine.py.224: train: epoch 20 | batch 320/2304 [I 2022-10-08 21:14:20] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:47 [I 2022-10-08 21:14:20] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00303 [I 2022-10-08 21:14:20] ...trap/engines/engine.py.230: loss: 0.21786 [I 2022-10-08 21:14:20] ...optimizers/trijoint.py.151: total_norm: 28.092983 [I 2022-10-08 21:14:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 330/2304 [I 2022-10-08 21:14:28] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:27:37 [I 2022-10-08 21:14:28] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00310 [I 2022-10-08 21:14:28] ...trap/engines/engine.py.230: loss: 0.23800 [I 2022-10-08 21:14:28] ...optimizers/trijoint.py.151: total_norm: 125.509224 [I 2022-10-08 21:14:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 340/2304 [I 2022-10-08 21:14:37] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:29 [I 2022-10-08 21:14:37] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00301 [I 2022-10-08 21:14:37] ...trap/engines/engine.py.230: loss: 0.17245 [I 2022-10-08 21:14:37] ...optimizers/trijoint.py.151: total_norm: 25.944439 [I 2022-10-08 21:14:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 350/2304 [I 2022-10-08 21:14:45] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:20 [I 2022-10-08 21:14:45] ...trap/engines/engine.py.229: process: 0.84292 | load: 0.00308 [I 2022-10-08 21:14:45] ...trap/engines/engine.py.230: loss: 0.16004 [I 2022-10-08 21:14:45] ...optimizers/trijoint.py.151: total_norm: 20.222647 [I 2022-10-08 21:14:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 360/2304 [I 2022-10-08 21:14:53] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:26:57 [I 2022-10-08 21:14:53] ...trap/engines/engine.py.229: process: 0.82719 | load: 0.00302 [I 2022-10-08 21:14:53] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-08 21:14:53] ...optimizers/trijoint.py.151: total_norm: 25.044327 [I 2022-10-08 21:15:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 370/2304 [I 2022-10-08 21:15:02] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:06 [I 2022-10-08 21:15:02] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00323 [I 2022-10-08 21:15:02] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-08 21:15:02] ...optimizers/trijoint.py.151: total_norm: 11.613663 [I 2022-10-08 21:15:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 380/2304 [I 2022-10-08 21:15:11] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:32:02 [I 2022-10-08 21:15:11] ...trap/engines/engine.py.229: process: 1.65658 | load: 0.00307 [I 2022-10-08 21:15:11] ...trap/engines/engine.py.230: loss: 0.17265 [I 2022-10-08 21:15:11] ...optimizers/trijoint.py.151: total_norm: 19.106253 [I 2022-10-08 21:15:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 390/2304 [I 2022-10-08 21:15:19] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:18 [I 2022-10-08 21:15:19] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00298 [I 2022-10-08 21:15:19] ...trap/engines/engine.py.230: loss: 0.25258 [I 2022-10-08 21:15:19] ...optimizers/trijoint.py.151: total_norm: 29.858530 [I 2022-10-08 21:15:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 400/2304 [I 2022-10-08 21:15:28] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:45 [I 2022-10-08 21:15:28] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00314 [I 2022-10-08 21:15:28] ...trap/engines/engine.py.230: loss: 0.10886 [I 2022-10-08 21:15:28] ...optimizers/trijoint.py.151: total_norm: 13.325647 [I 2022-10-08 21:15:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 410/2304 [I 2022-10-08 21:15:36] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:34 [I 2022-10-08 21:15:36] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00301 [I 2022-10-08 21:15:36] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-08 21:15:36] ...optimizers/trijoint.py.151: total_norm: 14.785109 [I 2022-10-08 21:15:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 420/2304 [I 2022-10-08 21:15:45] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:21 [I 2022-10-08 21:15:45] ...trap/engines/engine.py.229: process: 0.83457 | load: 0.00301 [I 2022-10-08 21:15:45] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-08 21:15:45] ...optimizers/trijoint.py.151: total_norm: 14.619747 [I 2022-10-08 21:15:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 430/2304 [I 2022-10-08 21:15:53] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:13 [I 2022-10-08 21:15:53] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00300 [I 2022-10-08 21:15:53] ...trap/engines/engine.py.230: loss: 0.14548 [I 2022-10-08 21:15:53] ...optimizers/trijoint.py.151: total_norm: 12.713489 [I 2022-10-08 21:16:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 440/2304 [I 2022-10-08 21:16:01] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:03 [I 2022-10-08 21:16:01] ...trap/engines/engine.py.229: process: 0.83472 | load: 0.00299 [I 2022-10-08 21:16:01] ...trap/engines/engine.py.230: loss: 0.24011 [I 2022-10-08 21:16:01] ...optimizers/trijoint.py.151: total_norm: 25.319996 [I 2022-10-08 21:16:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 450/2304 [I 2022-10-08 21:16:10] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:52 [I 2022-10-08 21:16:10] ...trap/engines/engine.py.229: process: 0.82975 | load: 0.00298 [I 2022-10-08 21:16:10] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-08 21:16:10] ...optimizers/trijoint.py.151: total_norm: 14.087075 [I 2022-10-08 21:16:18] ...trap/engines/engine.py.224: train: epoch 20 | batch 460/2304 [I 2022-10-08 21:16:18] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:47 [I 2022-10-08 21:16:18] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00300 [I 2022-10-08 21:16:18] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-08 21:16:18] ...optimizers/trijoint.py.151: total_norm: 26.501934 [I 2022-10-08 21:16:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 470/2304 [I 2022-10-08 21:16:27] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:44 [I 2022-10-08 21:16:27] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00298 [I 2022-10-08 21:16:27] ...trap/engines/engine.py.230: loss: 0.19777 [I 2022-10-08 21:16:27] ...optimizers/trijoint.py.151: total_norm: 20.529257 [I 2022-10-08 21:16:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 480/2304 [I 2022-10-08 21:16:35] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:26 [I 2022-10-08 21:16:35] ...trap/engines/engine.py.229: process: 0.82827 | load: 0.00298 [I 2022-10-08 21:16:35] ...trap/engines/engine.py.230: loss: 0.14355 [I 2022-10-08 21:16:35] ...optimizers/trijoint.py.151: total_norm: 17.351994 [I 2022-10-08 21:16:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 490/2304 [I 2022-10-08 21:16:44] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:25 [I 2022-10-08 21:16:44] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00298 [I 2022-10-08 21:16:44] ...trap/engines/engine.py.230: loss: 0.19832 [I 2022-10-08 21:16:44] ...optimizers/trijoint.py.151: total_norm: 32.445377 [I 2022-10-08 21:16:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 500/2304 [I 2022-10-08 21:16:52] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:18 [I 2022-10-08 21:16:52] ...trap/engines/engine.py.229: process: 0.85604 | load: 0.00299 [I 2022-10-08 21:16:52] ...trap/engines/engine.py.230: loss: 0.27099 [I 2022-10-08 21:16:52] ...optimizers/trijoint.py.151: total_norm: 24.306564 [I 2022-10-08 21:17:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 510/2304 [I 2022-10-08 21:17:00] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:25:12 [I 2022-10-08 21:17:00] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00312 [I 2022-10-08 21:17:00] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-08 21:17:00] ...optimizers/trijoint.py.151: total_norm: 20.435661 [I 2022-10-08 21:17:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 520/2304 [I 2022-10-08 21:17:09] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:58 [I 2022-10-08 21:17:09] ...trap/engines/engine.py.229: process: 0.83564 | load: 0.00297 [I 2022-10-08 21:17:09] ...trap/engines/engine.py.230: loss: 0.16990 [I 2022-10-08 21:17:09] ...optimizers/trijoint.py.151: total_norm: 16.818758 [I 2022-10-08 21:17:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 530/2304 [I 2022-10-08 21:17:17] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:50 [I 2022-10-08 21:17:17] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00311 [I 2022-10-08 21:17:17] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-08 21:17:17] ...optimizers/trijoint.py.151: total_norm: 15.541315 [I 2022-10-08 21:17:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 540/2304 [I 2022-10-08 21:17:26] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:35 [I 2022-10-08 21:17:26] ...trap/engines/engine.py.229: process: 0.82700 | load: 0.00300 [I 2022-10-08 21:17:26] ...trap/engines/engine.py.230: loss: 0.22645 [I 2022-10-08 21:17:26] ...optimizers/trijoint.py.151: total_norm: 24.977644 [I 2022-10-08 21:17:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 550/2304 [I 2022-10-08 21:17:35] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:25 [I 2022-10-08 21:17:35] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00303 [I 2022-10-08 21:17:35] ...trap/engines/engine.py.230: loss: 0.10675 [I 2022-10-08 21:17:35] ...optimizers/trijoint.py.151: total_norm: 15.805649 [I 2022-10-08 21:17:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 560/2304 [I 2022-10-08 21:17:43] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:31 [I 2022-10-08 21:17:43] ...trap/engines/engine.py.229: process: 0.83504 | load: 0.00300 [I 2022-10-08 21:17:43] ...trap/engines/engine.py.230: loss: 0.18729 [I 2022-10-08 21:17:43] ...optimizers/trijoint.py.151: total_norm: 39.312115 [I 2022-10-08 21:17:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 570/2304 [I 2022-10-08 21:17:51] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:20 [I 2022-10-08 21:17:51] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00307 [I 2022-10-08 21:17:51] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-08 21:17:51] ...optimizers/trijoint.py.151: total_norm: 26.561983 [I 2022-10-08 21:18:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 580/2304 [I 2022-10-08 21:18:00] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:13 [I 2022-10-08 21:18:00] ...trap/engines/engine.py.229: process: 0.84047 | load: 0.00299 [I 2022-10-08 21:18:00] ...trap/engines/engine.py.230: loss: 0.18753 [I 2022-10-08 21:18:00] ...optimizers/trijoint.py.151: total_norm: 20.088251 [I 2022-10-08 21:18:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 590/2304 [I 2022-10-08 21:18:08] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:24:02 [I 2022-10-08 21:18:08] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00303 [I 2022-10-08 21:18:08] ...trap/engines/engine.py.230: loss: 0.17431 [I 2022-10-08 21:18:08] ...optimizers/trijoint.py.151: total_norm: 16.842222 [I 2022-10-08 21:18:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 600/2304 [I 2022-10-08 21:18:17] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:24:01 [I 2022-10-08 21:18:17] ...trap/engines/engine.py.229: process: 0.84425 | load: 0.00302 [I 2022-10-08 21:18:17] ...trap/engines/engine.py.230: loss: 0.21148 [I 2022-10-08 21:18:17] ...optimizers/trijoint.py.151: total_norm: 18.917980 [I 2022-10-08 21:18:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 610/2304 [I 2022-10-08 21:18:25] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:52 [I 2022-10-08 21:18:25] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00300 [I 2022-10-08 21:18:25] ...trap/engines/engine.py.230: loss: 0.15334 [I 2022-10-08 21:18:25] ...optimizers/trijoint.py.151: total_norm: 18.271887 [I 2022-10-08 21:18:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 620/2304 [I 2022-10-08 21:18:34] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:45 [I 2022-10-08 21:18:34] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00298 [I 2022-10-08 21:18:34] ...trap/engines/engine.py.230: loss: 0.20168 [I 2022-10-08 21:18:34] ...optimizers/trijoint.py.151: total_norm: 13.941394 [I 2022-10-08 21:18:42] ...trap/engines/engine.py.224: train: epoch 20 | batch 630/2304 [I 2022-10-08 21:18:42] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:26 [I 2022-10-08 21:18:42] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00300 [I 2022-10-08 21:18:42] ...trap/engines/engine.py.230: loss: 0.13461 [I 2022-10-08 21:18:42] ...optimizers/trijoint.py.151: total_norm: 14.688823 [I 2022-10-08 21:18:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 640/2304 [I 2022-10-08 21:18:51] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:15 [I 2022-10-08 21:18:51] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00299 [I 2022-10-08 21:18:51] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-08 21:18:51] ...optimizers/trijoint.py.151: total_norm: 15.806690 [I 2022-10-08 21:18:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 650/2304 [I 2022-10-08 21:18:59] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:11 [I 2022-10-08 21:18:59] ...trap/engines/engine.py.229: process: 0.84577 | load: 0.00303 [I 2022-10-08 21:18:59] ...trap/engines/engine.py.230: loss: 0.16979 [I 2022-10-08 21:18:59] ...optimizers/trijoint.py.151: total_norm: 28.424215 [I 2022-10-08 21:19:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 660/2304 [I 2022-10-08 21:19:07] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:57 [I 2022-10-08 21:19:07] ...trap/engines/engine.py.229: process: 0.83681 | load: 0.00299 [I 2022-10-08 21:19:07] ...trap/engines/engine.py.230: loss: 0.24590 [I 2022-10-08 21:19:07] ...optimizers/trijoint.py.151: total_norm: 20.556929 [I 2022-10-08 21:19:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 670/2304 [I 2022-10-08 21:19:16] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:52 [I 2022-10-08 21:19:16] ...trap/engines/engine.py.229: process: 0.84889 | load: 0.00299 [I 2022-10-08 21:19:16] ...trap/engines/engine.py.230: loss: 0.13326 [I 2022-10-08 21:19:16] ...optimizers/trijoint.py.151: total_norm: 21.371410 [I 2022-10-08 21:19:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 680/2304 [I 2022-10-08 21:19:24] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:43 [I 2022-10-08 21:19:24] ...trap/engines/engine.py.229: process: 0.83126 | load: 0.00304 [I 2022-10-08 21:19:24] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-08 21:19:24] ...optimizers/trijoint.py.151: total_norm: 19.794796 [I 2022-10-08 21:19:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 690/2304 [I 2022-10-08 21:19:32] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:26 [I 2022-10-08 21:19:32] ...trap/engines/engine.py.229: process: 0.82931 | load: 0.00300 [I 2022-10-08 21:19:32] ...trap/engines/engine.py.230: loss: 0.10868 [I 2022-10-08 21:19:32] ...optimizers/trijoint.py.151: total_norm: 12.075189 [I 2022-10-08 21:19:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 700/2304 [I 2022-10-08 21:19:41] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:22 [I 2022-10-08 21:19:41] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00300 [I 2022-10-08 21:19:41] ...trap/engines/engine.py.230: loss: 0.11901 [I 2022-10-08 21:19:41] ...optimizers/trijoint.py.151: total_norm: 17.302034 [I 2022-10-08 21:19:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 710/2304 [I 2022-10-08 21:19:49] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:14 [I 2022-10-08 21:19:49] ...trap/engines/engine.py.229: process: 0.83049 | load: 0.00304 [I 2022-10-08 21:19:49] ...trap/engines/engine.py.230: loss: 0.19117 [I 2022-10-08 21:19:49] ...optimizers/trijoint.py.151: total_norm: 38.409206 [I 2022-10-08 21:19:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 720/2304 [I 2022-10-08 21:19:58] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:12 [I 2022-10-08 21:19:58] ...trap/engines/engine.py.229: process: 0.84976 | load: 0.00297 [I 2022-10-08 21:19:58] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-08 21:19:58] ...optimizers/trijoint.py.151: total_norm: 22.862537 [I 2022-10-08 21:20:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 730/2304 [I 2022-10-08 21:20:06] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:07 [I 2022-10-08 21:20:06] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00299 [I 2022-10-08 21:20:06] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-08 21:20:06] ...optimizers/trijoint.py.151: total_norm: 12.916982 [I 2022-10-08 21:20:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 740/2304 [I 2022-10-08 21:20:15] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:51 [I 2022-10-08 21:20:15] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00299 [I 2022-10-08 21:20:15] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-08 21:20:15] ...optimizers/trijoint.py.151: total_norm: 14.729161 [I 2022-10-08 21:20:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 750/2304 [I 2022-10-08 21:20:23] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:39 [I 2022-10-08 21:20:23] ...trap/engines/engine.py.229: process: 0.83478 | load: 0.00303 [I 2022-10-08 21:20:23] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-08 21:20:23] ...optimizers/trijoint.py.151: total_norm: 13.449704 [I 2022-10-08 21:20:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 760/2304 [I 2022-10-08 21:20:31] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:38 [I 2022-10-08 21:20:31] ...trap/engines/engine.py.229: process: 0.83130 | load: 0.00299 [I 2022-10-08 21:20:31] ...trap/engines/engine.py.230: loss: 0.28754 [I 2022-10-08 21:20:31] ...optimizers/trijoint.py.151: total_norm: 24.096346 [I 2022-10-08 21:20:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 770/2304 [I 2022-10-08 21:20:40] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:30 [I 2022-10-08 21:20:40] ...trap/engines/engine.py.229: process: 0.85383 | load: 0.00298 [I 2022-10-08 21:20:40] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-08 21:20:40] ...optimizers/trijoint.py.151: total_norm: 18.804762 [I 2022-10-08 21:20:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 780/2304 [I 2022-10-08 21:20:48] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:18 [I 2022-10-08 21:20:48] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00301 [I 2022-10-08 21:20:48] ...trap/engines/engine.py.230: loss: 0.19434 [I 2022-10-08 21:20:48] ...optimizers/trijoint.py.151: total_norm: 14.834563 [I 2022-10-08 21:20:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 790/2304 [I 2022-10-08 21:20:57] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:12 [I 2022-10-08 21:20:57] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00299 [I 2022-10-08 21:20:57] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-08 21:20:57] ...optimizers/trijoint.py.151: total_norm: 26.894314 [I 2022-10-08 21:21:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 800/2304 [I 2022-10-08 21:21:05] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:03 [I 2022-10-08 21:21:05] ...trap/engines/engine.py.229: process: 0.84176 | load: 0.00303 [I 2022-10-08 21:21:05] ...trap/engines/engine.py.230: loss: 0.15402 [I 2022-10-08 21:21:05] ...optimizers/trijoint.py.151: total_norm: 16.151220 [I 2022-10-08 21:21:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 810/2304 [I 2022-10-08 21:21:14] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:56 [I 2022-10-08 21:21:14] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00311 [I 2022-10-08 21:21:14] ...trap/engines/engine.py.230: loss: 0.21333 [I 2022-10-08 21:21:14] ...optimizers/trijoint.py.151: total_norm: 28.203539 [I 2022-10-08 21:21:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 820/2304 [I 2022-10-08 21:21:22] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:54 [I 2022-10-08 21:21:22] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00301 [I 2022-10-08 21:21:22] ...trap/engines/engine.py.230: loss: 0.28215 [I 2022-10-08 21:21:22] ...optimizers/trijoint.py.151: total_norm: 33.243256 [I 2022-10-08 21:21:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 830/2304 [I 2022-10-08 21:21:30] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:42 [I 2022-10-08 21:21:30] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00306 [I 2022-10-08 21:21:30] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-08 21:21:30] ...optimizers/trijoint.py.151: total_norm: 22.411398 [I 2022-10-08 21:21:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 840/2304 [I 2022-10-08 21:21:39] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:39 [I 2022-10-08 21:21:39] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00308 [I 2022-10-08 21:21:39] ...trap/engines/engine.py.230: loss: 0.16966 [I 2022-10-08 21:21:39] ...optimizers/trijoint.py.151: total_norm: 21.252510 [I 2022-10-08 21:21:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 850/2304 [I 2022-10-08 21:21:47] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:26 [I 2022-10-08 21:21:47] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00307 [I 2022-10-08 21:21:47] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-08 21:21:47] ...optimizers/trijoint.py.151: total_norm: 12.396329 [I 2022-10-08 21:21:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 860/2304 [I 2022-10-08 21:21:56] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:17 [I 2022-10-08 21:21:56] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00309 [I 2022-10-08 21:21:56] ...trap/engines/engine.py.230: loss: 0.16316 [I 2022-10-08 21:21:56] ...optimizers/trijoint.py.151: total_norm: 30.642179 [I 2022-10-08 21:22:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 870/2304 [I 2022-10-08 21:22:04] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:07 [I 2022-10-08 21:22:04] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00304 [I 2022-10-08 21:22:04] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-08 21:22:04] ...optimizers/trijoint.py.151: total_norm: 23.099884 [I 2022-10-08 21:22:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 880/2304 [I 2022-10-08 21:22:13] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:00 [I 2022-10-08 21:22:13] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00300 [I 2022-10-08 21:22:13] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-08 21:22:13] ...optimizers/trijoint.py.151: total_norm: 24.628462 [I 2022-10-08 21:22:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 890/2304 [I 2022-10-08 21:22:21] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:48 [I 2022-10-08 21:22:21] ...trap/engines/engine.py.229: process: 0.83487 | load: 0.00301 [I 2022-10-08 21:22:21] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-08 21:22:21] ...optimizers/trijoint.py.151: total_norm: 28.584799 [I 2022-10-08 21:22:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 900/2304 [I 2022-10-08 21:22:29] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:36 [I 2022-10-08 21:22:29] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00301 [I 2022-10-08 21:22:29] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-08 21:22:29] ...optimizers/trijoint.py.151: total_norm: 14.304545 [I 2022-10-08 21:22:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 910/2304 [I 2022-10-08 21:22:38] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:32 [I 2022-10-08 21:22:38] ...trap/engines/engine.py.229: process: 0.83899 | load: 0.00304 [I 2022-10-08 21:22:38] ...trap/engines/engine.py.230: loss: 0.22494 [I 2022-10-08 21:22:38] ...optimizers/trijoint.py.151: total_norm: 48.850544 [I 2022-10-08 21:22:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 920/2304 [I 2022-10-08 21:22:46] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:24 [I 2022-10-08 21:22:46] ...trap/engines/engine.py.229: process: 0.82971 | load: 0.00302 [I 2022-10-08 21:22:46] ...trap/engines/engine.py.230: loss: 0.09023 [I 2022-10-08 21:22:46] ...optimizers/trijoint.py.151: total_norm: 15.559005 [I 2022-10-08 21:22:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 930/2304 [I 2022-10-08 21:22:55] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:18 [I 2022-10-08 21:22:55] ...trap/engines/engine.py.229: process: 0.83979 | load: 0.00299 [I 2022-10-08 21:22:55] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-08 21:22:55] ...optimizers/trijoint.py.151: total_norm: 14.323581 [I 2022-10-08 21:23:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 940/2304 [I 2022-10-08 21:23:03] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:07 [I 2022-10-08 21:23:03] ...trap/engines/engine.py.229: process: 0.83477 | load: 0.00302 [I 2022-10-08 21:23:03] ...trap/engines/engine.py.230: loss: 0.16589 [I 2022-10-08 21:23:03] ...optimizers/trijoint.py.151: total_norm: 28.066092 [I 2022-10-08 21:23:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 950/2304 [I 2022-10-08 21:23:12] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:00 [I 2022-10-08 21:23:12] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00300 [I 2022-10-08 21:23:12] ...trap/engines/engine.py.230: loss: 0.18776 [I 2022-10-08 21:23:12] ...optimizers/trijoint.py.151: total_norm: 32.426899 [I 2022-10-08 21:23:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 960/2304 [I 2022-10-08 21:23:22] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:24:23 [I 2022-10-08 21:23:22] ...trap/engines/engine.py.229: process: 1.26493 | load: 0.00299 [I 2022-10-08 21:23:22] ...trap/engines/engine.py.230: loss: 0.15988 [I 2022-10-08 21:23:22] ...optimizers/trijoint.py.151: total_norm: 27.493973 [I 2022-10-08 21:23:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 970/2304 [I 2022-10-08 21:23:32] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:21:39 [I 2022-10-08 21:23:32] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00302 [I 2022-10-08 21:23:32] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-08 21:23:32] ...optimizers/trijoint.py.151: total_norm: 11.927498 [I 2022-10-08 21:23:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 980/2304 [I 2022-10-08 21:23:40] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:50 [I 2022-10-08 21:23:40] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00302 [I 2022-10-08 21:23:40] ...trap/engines/engine.py.230: loss: 0.10208 [I 2022-10-08 21:23:40] ...optimizers/trijoint.py.151: total_norm: 13.776029 [I 2022-10-08 21:23:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 990/2304 [I 2022-10-08 21:23:49] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:30 [I 2022-10-08 21:23:49] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00318 [I 2022-10-08 21:23:49] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-08 21:23:49] ...optimizers/trijoint.py.151: total_norm: 22.467684 [I 2022-10-08 21:23:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 1000/2304 [I 2022-10-08 21:23:57] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:17 [I 2022-10-08 21:23:57] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00301 [I 2022-10-08 21:23:57] ...trap/engines/engine.py.230: loss: 0.16624 [I 2022-10-08 21:23:57] ...optimizers/trijoint.py.151: total_norm: 30.421480 [I 2022-10-08 21:24:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 1010/2304 [I 2022-10-08 21:24:05] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:12 [I 2022-10-08 21:24:05] ...trap/engines/engine.py.229: process: 0.83018 | load: 0.00311 [I 2022-10-08 21:24:05] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-08 21:24:05] ...optimizers/trijoint.py.151: total_norm: 15.583975 [I 2022-10-08 21:24:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 1020/2304 [I 2022-10-08 21:24:14] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:02 [I 2022-10-08 21:24:14] ...trap/engines/engine.py.229: process: 0.84625 | load: 0.00309 [I 2022-10-08 21:24:14] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-08 21:24:14] ...optimizers/trijoint.py.151: total_norm: 22.207693 [I 2022-10-08 21:24:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 1030/2304 [I 2022-10-08 21:24:22] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:17:52 [I 2022-10-08 21:24:22] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00311 [I 2022-10-08 21:24:22] ...trap/engines/engine.py.230: loss: 0.21104 [I 2022-10-08 21:24:22] ...optimizers/trijoint.py.151: total_norm: 21.030458 [I 2022-10-08 21:24:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 1040/2304 [I 2022-10-08 21:24:31] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:39 [I 2022-10-08 21:24:31] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00308 [I 2022-10-08 21:24:31] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-08 21:24:31] ...optimizers/trijoint.py.151: total_norm: 19.977856 [I 2022-10-08 21:24:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 1050/2304 [I 2022-10-08 21:24:39] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:34 [I 2022-10-08 21:24:39] ...trap/engines/engine.py.229: process: 0.83425 | load: 0.00308 [I 2022-10-08 21:24:39] ...trap/engines/engine.py.230: loss: 0.12869 [I 2022-10-08 21:24:39] ...optimizers/trijoint.py.151: total_norm: 17.699982 [I 2022-10-08 21:24:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 1060/2304 [I 2022-10-08 21:24:48] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:26 [I 2022-10-08 21:24:48] ...trap/engines/engine.py.229: process: 0.83272 | load: 0.00307 [I 2022-10-08 21:24:48] ...trap/engines/engine.py.230: loss: 0.18132 [I 2022-10-08 21:24:48] ...optimizers/trijoint.py.151: total_norm: 134.502579 [I 2022-10-08 21:24:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 1070/2304 [I 2022-10-08 21:24:56] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:21 [I 2022-10-08 21:24:56] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00298 [I 2022-10-08 21:24:56] ...trap/engines/engine.py.230: loss: 0.23340 [I 2022-10-08 21:24:56] ...optimizers/trijoint.py.151: total_norm: 41.834629 [I 2022-10-08 21:25:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 1080/2304 [I 2022-10-08 21:25:05] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:08 [I 2022-10-08 21:25:05] ...trap/engines/engine.py.229: process: 0.84580 | load: 0.00302 [I 2022-10-08 21:25:05] ...trap/engines/engine.py.230: loss: 0.18476 [I 2022-10-08 21:25:05] ...optimizers/trijoint.py.151: total_norm: 52.800602 [I 2022-10-08 21:25:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 1090/2304 [I 2022-10-08 21:25:13] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:16:59 [I 2022-10-08 21:25:13] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00305 [I 2022-10-08 21:25:13] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-08 21:25:13] ...optimizers/trijoint.py.151: total_norm: 17.288280 [I 2022-10-08 21:25:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 1100/2304 [I 2022-10-08 21:25:22] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:54 [I 2022-10-08 21:25:22] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00298 [I 2022-10-08 21:25:22] ...trap/engines/engine.py.230: loss: 0.15322 [I 2022-10-08 21:25:22] ...optimizers/trijoint.py.151: total_norm: 25.258461 [I 2022-10-08 21:25:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 1110/2304 [I 2022-10-08 21:25:30] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:45 [I 2022-10-08 21:25:30] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00300 [I 2022-10-08 21:25:30] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-08 21:25:30] ...optimizers/trijoint.py.151: total_norm: 17.654863 [I 2022-10-08 21:25:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 1120/2304 [I 2022-10-08 21:25:38] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:34 [I 2022-10-08 21:25:38] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00297 [I 2022-10-08 21:25:38] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-08 21:25:38] ...optimizers/trijoint.py.151: total_norm: 21.712656 [I 2022-10-08 21:25:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 1130/2304 [I 2022-10-08 21:25:47] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:27 [I 2022-10-08 21:25:47] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00301 [I 2022-10-08 21:25:47] ...trap/engines/engine.py.230: loss: 0.25552 [I 2022-10-08 21:25:47] ...optimizers/trijoint.py.151: total_norm: 26.003317 [I 2022-10-08 21:25:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 1140/2304 [I 2022-10-08 21:25:55] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:21 [I 2022-10-08 21:25:55] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00298 [I 2022-10-08 21:25:55] ...trap/engines/engine.py.230: loss: 0.18677 [I 2022-10-08 21:25:55] ...optimizers/trijoint.py.151: total_norm: 21.457233 [I 2022-10-08 21:26:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 1150/2304 [I 2022-10-08 21:26:04] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:14 [I 2022-10-08 21:26:04] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00301 [I 2022-10-08 21:26:04] ...trap/engines/engine.py.230: loss: 0.14900 [I 2022-10-08 21:26:04] ...optimizers/trijoint.py.151: total_norm: 25.476223 [I 2022-10-08 21:26:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 1160/2304 [I 2022-10-08 21:26:12] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:02 [I 2022-10-08 21:26:12] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00303 [I 2022-10-08 21:26:12] ...trap/engines/engine.py.230: loss: 0.19516 [I 2022-10-08 21:26:12] ...optimizers/trijoint.py.151: total_norm: 30.441179 [I 2022-10-08 21:26:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 1170/2304 [I 2022-10-08 21:26:21] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:51 [I 2022-10-08 21:26:21] ...trap/engines/engine.py.229: process: 0.83568 | load: 0.00301 [I 2022-10-08 21:26:21] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-08 21:26:21] ...optimizers/trijoint.py.151: total_norm: 15.693820 [I 2022-10-08 21:26:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 1180/2304 [I 2022-10-08 21:26:29] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:41 [I 2022-10-08 21:26:29] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00297 [I 2022-10-08 21:26:29] ...trap/engines/engine.py.230: loss: 0.21085 [I 2022-10-08 21:26:29] ...optimizers/trijoint.py.151: total_norm: 20.750854 [I 2022-10-08 21:26:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 1190/2304 [I 2022-10-08 21:26:37] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:35 [I 2022-10-08 21:26:37] ...trap/engines/engine.py.229: process: 0.83433 | load: 0.00299 [I 2022-10-08 21:26:37] ...trap/engines/engine.py.230: loss: 0.20369 [I 2022-10-08 21:26:37] ...optimizers/trijoint.py.151: total_norm: 34.162090 [I 2022-10-08 21:26:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 1200/2304 [I 2022-10-08 21:26:46] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:30 [I 2022-10-08 21:26:46] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00299 [I 2022-10-08 21:26:46] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-08 21:26:46] ...optimizers/trijoint.py.151: total_norm: 19.792160 [I 2022-10-08 21:26:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 1210/2304 [I 2022-10-08 21:26:54] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:21 [I 2022-10-08 21:26:54] ...trap/engines/engine.py.229: process: 0.84994 | load: 0.00302 [I 2022-10-08 21:26:54] ...trap/engines/engine.py.230: loss: 0.22624 [I 2022-10-08 21:26:54] ...optimizers/trijoint.py.151: total_norm: 21.361109 [I 2022-10-08 21:27:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 1220/2304 [I 2022-10-08 21:27:03] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:14 [I 2022-10-08 21:27:03] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00302 [I 2022-10-08 21:27:03] ...trap/engines/engine.py.230: loss: 0.18285 [I 2022-10-08 21:27:03] ...optimizers/trijoint.py.151: total_norm: 21.573311 [I 2022-10-08 21:27:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 1230/2304 [I 2022-10-08 21:27:11] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:06 [I 2022-10-08 21:27:11] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00304 [I 2022-10-08 21:27:11] ...trap/engines/engine.py.230: loss: 0.24064 [I 2022-10-08 21:27:11] ...optimizers/trijoint.py.151: total_norm: 21.389233 [I 2022-10-08 21:27:20] ...trap/engines/engine.py.224: train: epoch 20 | batch 1240/2304 [I 2022-10-08 21:27:20] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:29 [I 2022-10-08 21:27:20] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00300 [I 2022-10-08 21:27:20] ...trap/engines/engine.py.230: loss: 0.16384 [I 2022-10-08 21:27:20] ...optimizers/trijoint.py.151: total_norm: 22.657078 [I 2022-10-08 21:27:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 1250/2304 [I 2022-10-08 21:27:28] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:49 [I 2022-10-08 21:27:28] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00300 [I 2022-10-08 21:27:28] ...trap/engines/engine.py.230: loss: 0.24390 [I 2022-10-08 21:27:28] ...optimizers/trijoint.py.151: total_norm: 21.115026 [I 2022-10-08 21:27:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 1260/2304 [I 2022-10-08 21:27:37] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:32 [I 2022-10-08 21:27:37] ...trap/engines/engine.py.229: process: 0.84144 | load: 0.00299 [I 2022-10-08 21:27:37] ...trap/engines/engine.py.230: loss: 0.09651 [I 2022-10-08 21:27:37] ...optimizers/trijoint.py.151: total_norm: 14.670957 [I 2022-10-08 21:27:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 1270/2304 [I 2022-10-08 21:27:45] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:29 [I 2022-10-08 21:27:45] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00296 [I 2022-10-08 21:27:45] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-08 21:27:45] ...optimizers/trijoint.py.151: total_norm: 23.747532 [I 2022-10-08 21:27:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 1280/2304 [I 2022-10-08 21:27:54] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:24 [I 2022-10-08 21:27:54] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00303 [I 2022-10-08 21:27:54] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-08 21:27:54] ...optimizers/trijoint.py.151: total_norm: 20.980249 [I 2022-10-08 21:28:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 1290/2304 [I 2022-10-08 21:28:02] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:12 [I 2022-10-08 21:28:02] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00302 [I 2022-10-08 21:28:02] ...trap/engines/engine.py.230: loss: 0.19227 [I 2022-10-08 21:28:02] ...optimizers/trijoint.py.151: total_norm: 20.177567 [I 2022-10-08 21:28:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 1300/2304 [I 2022-10-08 21:28:11] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:08 [I 2022-10-08 21:28:11] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00299 [I 2022-10-08 21:28:11] ...trap/engines/engine.py.230: loss: 0.21035 [I 2022-10-08 21:28:11] ...optimizers/trijoint.py.151: total_norm: 22.509119 [I 2022-10-08 21:28:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 1310/2304 [I 2022-10-08 21:28:19] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:50 [I 2022-10-08 21:28:19] ...trap/engines/engine.py.229: process: 0.82475 | load: 0.00297 [I 2022-10-08 21:28:19] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-08 21:28:19] ...optimizers/trijoint.py.151: total_norm: 27.333298 [I 2022-10-08 21:28:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 1320/2304 [I 2022-10-08 21:28:28] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:52 [I 2022-10-08 21:28:28] ...trap/engines/engine.py.229: process: 0.84475 | load: 0.00301 [I 2022-10-08 21:28:28] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-08 21:28:28] ...optimizers/trijoint.py.151: total_norm: 15.095694 [I 2022-10-08 21:28:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 1330/2304 [I 2022-10-08 21:28:36] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:39 [I 2022-10-08 21:28:36] ...trap/engines/engine.py.229: process: 0.83440 | load: 0.00299 [I 2022-10-08 21:28:36] ...trap/engines/engine.py.230: loss: 0.15193 [I 2022-10-08 21:28:36] ...optimizers/trijoint.py.151: total_norm: 11.541289 [I 2022-10-08 21:28:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 1340/2304 [I 2022-10-08 21:28:45] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:31 [I 2022-10-08 21:28:45] ...trap/engines/engine.py.229: process: 0.84112 | load: 0.00299 [I 2022-10-08 21:28:45] ...trap/engines/engine.py.230: loss: 0.15527 [I 2022-10-08 21:28:45] ...optimizers/trijoint.py.151: total_norm: 14.750025 [I 2022-10-08 21:28:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 1350/2304 [I 2022-10-08 21:28:53] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:23 [I 2022-10-08 21:28:53] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00304 [I 2022-10-08 21:28:53] ...trap/engines/engine.py.230: loss: 0.29387 [I 2022-10-08 21:28:53] ...optimizers/trijoint.py.151: total_norm: 26.935572 [I 2022-10-08 21:29:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 1360/2304 [I 2022-10-08 21:29:01] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:07 [I 2022-10-08 21:29:01] ...trap/engines/engine.py.229: process: 0.82283 | load: 0.00303 [I 2022-10-08 21:29:01] ...trap/engines/engine.py.230: loss: 0.14178 [I 2022-10-08 21:29:01] ...optimizers/trijoint.py.151: total_norm: 16.220509 [I 2022-10-08 21:29:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 1370/2304 [I 2022-10-08 21:29:10] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:03 [I 2022-10-08 21:29:10] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00301 [I 2022-10-08 21:29:10] ...trap/engines/engine.py.230: loss: 0.18565 [I 2022-10-08 21:29:10] ...optimizers/trijoint.py.151: total_norm: 10.157702 [I 2022-10-08 21:29:18] ...trap/engines/engine.py.224: train: epoch 20 | batch 1380/2304 [I 2022-10-08 21:29:18] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:56 [I 2022-10-08 21:29:18] ...trap/engines/engine.py.229: process: 0.83841 | load: 0.00299 [I 2022-10-08 21:29:18] ...trap/engines/engine.py.230: loss: 0.20751 [I 2022-10-08 21:29:18] ...optimizers/trijoint.py.151: total_norm: 21.919106 [I 2022-10-08 21:29:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 1390/2304 [I 2022-10-08 21:29:27] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:49 [I 2022-10-08 21:29:27] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00300 [I 2022-10-08 21:29:27] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-08 21:29:27] ...optimizers/trijoint.py.151: total_norm: 16.230259 [I 2022-10-08 21:29:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 1400/2304 [I 2022-10-08 21:29:35] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:39 [I 2022-10-08 21:29:35] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00303 [I 2022-10-08 21:29:35] ...trap/engines/engine.py.230: loss: 0.26078 [I 2022-10-08 21:29:35] ...optimizers/trijoint.py.151: total_norm: 30.623583 [I 2022-10-08 21:29:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 1410/2304 [I 2022-10-08 21:29:44] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:27 [I 2022-10-08 21:29:44] ...trap/engines/engine.py.229: process: 0.83268 | load: 0.00307 [I 2022-10-08 21:29:44] ...trap/engines/engine.py.230: loss: 0.15832 [I 2022-10-08 21:29:44] ...optimizers/trijoint.py.151: total_norm: 19.560822 [I 2022-10-08 21:29:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 1420/2304 [I 2022-10-08 21:29:52] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:19 [I 2022-10-08 21:29:52] ...trap/engines/engine.py.229: process: 0.83390 | load: 0.00299 [I 2022-10-08 21:29:52] ...trap/engines/engine.py.230: loss: 0.22067 [I 2022-10-08 21:29:52] ...optimizers/trijoint.py.151: total_norm: 19.041241 [I 2022-10-08 21:30:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 1430/2304 [I 2022-10-08 21:30:00] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:11 [I 2022-10-08 21:30:00] ...trap/engines/engine.py.229: process: 0.83392 | load: 0.00302 [I 2022-10-08 21:30:00] ...trap/engines/engine.py.230: loss: 0.22192 [I 2022-10-08 21:30:00] ...optimizers/trijoint.py.151: total_norm: 15.595311 [I 2022-10-08 21:30:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 1440/2304 [I 2022-10-08 21:30:09] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:02 [I 2022-10-08 21:30:09] ...trap/engines/engine.py.229: process: 0.82587 | load: 0.00299 [I 2022-10-08 21:30:09] ...trap/engines/engine.py.230: loss: 0.17206 [I 2022-10-08 21:30:09] ...optimizers/trijoint.py.151: total_norm: 15.160771 [I 2022-10-08 21:30:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 1450/2304 [I 2022-10-08 21:30:17] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:11:54 [I 2022-10-08 21:30:17] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00300 [I 2022-10-08 21:30:17] ...trap/engines/engine.py.230: loss: 0.18821 [I 2022-10-08 21:30:17] ...optimizers/trijoint.py.151: total_norm: 26.249687 [I 2022-10-08 21:30:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 1460/2304 [I 2022-10-08 21:30:25] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:11:50 [I 2022-10-08 21:30:25] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00301 [I 2022-10-08 21:30:25] ...trap/engines/engine.py.230: loss: 0.18239 [I 2022-10-08 21:30:25] ...optimizers/trijoint.py.151: total_norm: 14.548448 [I 2022-10-08 21:30:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 1470/2304 [I 2022-10-08 21:30:34] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:41 [I 2022-10-08 21:30:34] ...trap/engines/engine.py.229: process: 0.83838 | load: 0.00300 [I 2022-10-08 21:30:34] ...trap/engines/engine.py.230: loss: 0.14300 [I 2022-10-08 21:30:34] ...optimizers/trijoint.py.151: total_norm: 14.666757 [I 2022-10-08 21:30:42] ...trap/engines/engine.py.224: train: epoch 20 | batch 1480/2304 [I 2022-10-08 21:30:42] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:30 [I 2022-10-08 21:30:42] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00301 [I 2022-10-08 21:30:42] ...trap/engines/engine.py.230: loss: 0.14716 [I 2022-10-08 21:30:42] ...optimizers/trijoint.py.151: total_norm: 17.451942 [I 2022-10-08 21:30:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 1490/2304 [I 2022-10-08 21:30:51] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:27 [I 2022-10-08 21:30:51] ...trap/engines/engine.py.229: process: 0.83341 | load: 0.00303 [I 2022-10-08 21:30:51] ...trap/engines/engine.py.230: loss: 0.19731 [I 2022-10-08 21:30:51] ...optimizers/trijoint.py.151: total_norm: 21.330645 [I 2022-10-08 21:30:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 1500/2304 [I 2022-10-08 21:30:59] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:17 [I 2022-10-08 21:30:59] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00297 [I 2022-10-08 21:30:59] ...trap/engines/engine.py.230: loss: 0.22566 [I 2022-10-08 21:30:59] ...optimizers/trijoint.py.151: total_norm: 18.016333 [I 2022-10-08 21:31:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 1510/2304 [I 2022-10-08 21:31:08] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:07 [I 2022-10-08 21:31:08] ...trap/engines/engine.py.229: process: 0.83045 | load: 0.00295 [I 2022-10-08 21:31:08] ...trap/engines/engine.py.230: loss: 0.20640 [I 2022-10-08 21:31:08] ...optimizers/trijoint.py.151: total_norm: 20.174339 [I 2022-10-08 21:31:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 1520/2304 [I 2022-10-08 21:31:16] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:11:00 [I 2022-10-08 21:31:16] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00297 [I 2022-10-08 21:31:16] ...trap/engines/engine.py.230: loss: 0.17979 [I 2022-10-08 21:31:16] ...optimizers/trijoint.py.151: total_norm: 20.920933 [I 2022-10-08 21:31:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 1530/2304 [I 2022-10-08 21:31:25] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:10:55 [I 2022-10-08 21:31:25] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00301 [I 2022-10-08 21:31:25] ...trap/engines/engine.py.230: loss: 0.17096 [I 2022-10-08 21:31:25] ...optimizers/trijoint.py.151: total_norm: 18.999916 [I 2022-10-08 21:31:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 1540/2304 [I 2022-10-08 21:31:33] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:10:45 [I 2022-10-08 21:31:33] ...trap/engines/engine.py.229: process: 0.82839 | load: 0.00300 [I 2022-10-08 21:31:33] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-08 21:31:33] ...optimizers/trijoint.py.151: total_norm: 26.501812 [I 2022-10-08 21:31:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 1550/2304 [I 2022-10-08 21:31:41] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:35 [I 2022-10-08 21:31:41] ...trap/engines/engine.py.229: process: 0.82990 | load: 0.00302 [I 2022-10-08 21:31:41] ...trap/engines/engine.py.230: loss: 0.15203 [I 2022-10-08 21:31:41] ...optimizers/trijoint.py.151: total_norm: 17.091021 [I 2022-10-08 21:31:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 1560/2304 [I 2022-10-08 21:31:50] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:10:27 [I 2022-10-08 21:31:50] ...trap/engines/engine.py.229: process: 0.83516 | load: 0.00308 [I 2022-10-08 21:31:50] ...trap/engines/engine.py.230: loss: 0.21674 [I 2022-10-08 21:31:50] ...optimizers/trijoint.py.151: total_norm: 23.435795 [I 2022-10-08 21:31:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 1570/2304 [I 2022-10-08 21:31:58] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:17 [I 2022-10-08 21:31:58] ...trap/engines/engine.py.229: process: 0.83610 | load: 0.00303 [I 2022-10-08 21:31:58] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-08 21:31:58] ...optimizers/trijoint.py.151: total_norm: 24.030981 [I 2022-10-08 21:32:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 1580/2304 [I 2022-10-08 21:32:07] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:08 [I 2022-10-08 21:32:07] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00297 [I 2022-10-08 21:32:07] ...trap/engines/engine.py.230: loss: 0.23122 [I 2022-10-08 21:32:07] ...optimizers/trijoint.py.151: total_norm: 20.922649 [I 2022-10-08 21:32:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 1590/2304 [I 2022-10-08 21:32:15] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:09:57 [I 2022-10-08 21:32:15] ...trap/engines/engine.py.229: process: 0.82916 | load: 0.00305 [I 2022-10-08 21:32:15] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-08 21:32:15] ...optimizers/trijoint.py.151: total_norm: 20.060377 [I 2022-10-08 21:32:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 1600/2304 [I 2022-10-08 21:32:24] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:09:51 [I 2022-10-08 21:32:24] ...trap/engines/engine.py.229: process: 0.83482 | load: 0.00300 [I 2022-10-08 21:32:24] ...trap/engines/engine.py.230: loss: 0.22459 [I 2022-10-08 21:32:24] ...optimizers/trijoint.py.151: total_norm: 29.892420 [I 2022-10-08 21:32:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 1610/2304 [I 2022-10-08 21:32:32] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:09:42 [I 2022-10-08 21:32:32] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00299 [I 2022-10-08 21:32:32] ...trap/engines/engine.py.230: loss: 0.17469 [I 2022-10-08 21:32:32] ...optimizers/trijoint.py.151: total_norm: 13.252135 [I 2022-10-08 21:32:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 1620/2304 [I 2022-10-08 21:32:40] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:09:34 [I 2022-10-08 21:32:40] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00301 [I 2022-10-08 21:32:40] ...trap/engines/engine.py.230: loss: 0.14666 [I 2022-10-08 21:32:40] ...optimizers/trijoint.py.151: total_norm: 12.186183 [I 2022-10-08 21:32:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 1630/2304 [I 2022-10-08 21:32:49] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:28 [I 2022-10-08 21:32:49] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00300 [I 2022-10-08 21:32:49] ...trap/engines/engine.py.230: loss: 0.20399 [I 2022-10-08 21:32:49] ...optimizers/trijoint.py.151: total_norm: 20.345604 [I 2022-10-08 21:32:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 1640/2304 [I 2022-10-08 21:32:57] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:19 [I 2022-10-08 21:32:57] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00306 [I 2022-10-08 21:32:57] ...trap/engines/engine.py.230: loss: 0.20645 [I 2022-10-08 21:32:57] ...optimizers/trijoint.py.151: total_norm: 20.306995 [I 2022-10-08 21:33:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 1650/2304 [I 2022-10-08 21:33:06] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:11 [I 2022-10-08 21:33:06] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00301 [I 2022-10-08 21:33:06] ...trap/engines/engine.py.230: loss: 0.17768 [I 2022-10-08 21:33:06] ...optimizers/trijoint.py.151: total_norm: 35.380024 [I 2022-10-08 21:33:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 1660/2304 [I 2022-10-08 21:33:14] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:12 [I 2022-10-08 21:33:14] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00303 [I 2022-10-08 21:33:14] ...trap/engines/engine.py.230: loss: 0.11898 [I 2022-10-08 21:33:14] ...optimizers/trijoint.py.151: total_norm: 13.272214 [I 2022-10-08 21:33:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 1670/2304 [I 2022-10-08 21:33:23] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:08:55 [I 2022-10-08 21:33:23] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00302 [I 2022-10-08 21:33:23] ...trap/engines/engine.py.230: loss: 0.18990 [I 2022-10-08 21:33:23] ...optimizers/trijoint.py.151: total_norm: 17.331057 [I 2022-10-08 21:33:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 1680/2304 [I 2022-10-08 21:33:31] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:08:44 [I 2022-10-08 21:33:31] ...trap/engines/engine.py.229: process: 0.82943 | load: 0.00299 [I 2022-10-08 21:33:31] ...trap/engines/engine.py.230: loss: 0.22931 [I 2022-10-08 21:33:31] ...optimizers/trijoint.py.151: total_norm: 28.444136 [I 2022-10-08 21:33:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 1690/2304 [I 2022-10-08 21:33:40] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:08:32 [I 2022-10-08 21:33:40] ...trap/engines/engine.py.229: process: 0.81750 | load: 0.00303 [I 2022-10-08 21:33:40] ...trap/engines/engine.py.230: loss: 0.20112 [I 2022-10-08 21:33:40] ...optimizers/trijoint.py.151: total_norm: 15.067540 [I 2022-10-08 21:33:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 1700/2304 [I 2022-10-08 21:33:48] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:08:28 [I 2022-10-08 21:33:48] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00297 [I 2022-10-08 21:33:48] ...trap/engines/engine.py.230: loss: 0.15599 [I 2022-10-08 21:33:48] ...optimizers/trijoint.py.151: total_norm: 13.386580 [I 2022-10-08 21:33:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 1710/2304 [I 2022-10-08 21:33:57] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:19 [I 2022-10-08 21:33:57] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00302 [I 2022-10-08 21:33:57] ...trap/engines/engine.py.230: loss: 0.13226 [I 2022-10-08 21:33:57] ...optimizers/trijoint.py.151: total_norm: 19.308893 [I 2022-10-08 21:34:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 1720/2304 [I 2022-10-08 21:34:05] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:08:12 [I 2022-10-08 21:34:05] ...trap/engines/engine.py.229: process: 0.83701 | load: 0.00302 [I 2022-10-08 21:34:05] ...trap/engines/engine.py.230: loss: 0.25764 [I 2022-10-08 21:34:05] ...optimizers/trijoint.py.151: total_norm: 22.382109 [I 2022-10-08 21:34:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 1730/2304 [I 2022-10-08 21:34:13] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:02 [I 2022-10-08 21:34:13] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00301 [I 2022-10-08 21:34:13] ...trap/engines/engine.py.230: loss: 0.13975 [I 2022-10-08 21:34:13] ...optimizers/trijoint.py.151: total_norm: 21.845268 [I 2022-10-08 21:34:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 1740/2304 [I 2022-10-08 21:34:22] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:07:51 [I 2022-10-08 21:34:22] ...trap/engines/engine.py.229: process: 0.82807 | load: 0.00300 [I 2022-10-08 21:34:22] ...trap/engines/engine.py.230: loss: 0.16017 [I 2022-10-08 21:34:22] ...optimizers/trijoint.py.151: total_norm: 16.857056 [I 2022-10-08 21:34:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 1750/2304 [I 2022-10-08 21:34:30] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:07:47 [I 2022-10-08 21:34:30] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00308 [I 2022-10-08 21:34:30] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-08 21:34:30] ...optimizers/trijoint.py.151: total_norm: 20.484699 [I 2022-10-08 21:34:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 1760/2304 [I 2022-10-08 21:34:39] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:07:36 [I 2022-10-08 21:34:39] ...trap/engines/engine.py.229: process: 0.83927 | load: 0.00298 [I 2022-10-08 21:34:39] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-08 21:34:39] ...optimizers/trijoint.py.151: total_norm: 8.987395 [I 2022-10-08 21:34:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 1770/2304 [I 2022-10-08 21:34:47] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:07:29 [I 2022-10-08 21:34:47] ...trap/engines/engine.py.229: process: 0.85214 | load: 0.00299 [I 2022-10-08 21:34:47] ...trap/engines/engine.py.230: loss: 0.17668 [I 2022-10-08 21:34:47] ...optimizers/trijoint.py.151: total_norm: 15.848626 [I 2022-10-08 21:34:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 1780/2304 [I 2022-10-08 21:34:55] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:19 [I 2022-10-08 21:34:55] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00297 [I 2022-10-08 21:34:55] ...trap/engines/engine.py.230: loss: 0.16269 [I 2022-10-08 21:34:55] ...optimizers/trijoint.py.151: total_norm: 17.884287 [I 2022-10-08 21:35:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 1790/2304 [I 2022-10-08 21:35:04] ...trap/engines/engine.py.228: elapsed: 0:25:24 | left: 0:07:07 [I 2022-10-08 21:35:04] ...trap/engines/engine.py.229: process: 0.82659 | load: 0.00301 [I 2022-10-08 21:35:04] ...trap/engines/engine.py.230: loss: 0.17092 [I 2022-10-08 21:35:04] ...optimizers/trijoint.py.151: total_norm: 19.597441 [I 2022-10-08 21:35:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 1800/2304 [I 2022-10-08 21:35:12] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:02 [I 2022-10-08 21:35:12] ...trap/engines/engine.py.229: process: 0.84120 | load: 0.00299 [I 2022-10-08 21:35:12] ...trap/engines/engine.py.230: loss: 0.17555 [I 2022-10-08 21:35:12] ...optimizers/trijoint.py.151: total_norm: 32.502357 [I 2022-10-08 21:35:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 1810/2304 [I 2022-10-08 21:35:21] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:06:54 [I 2022-10-08 21:35:21] ...trap/engines/engine.py.229: process: 0.83575 | load: 0.00300 [I 2022-10-08 21:35:21] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-08 21:35:21] ...optimizers/trijoint.py.151: total_norm: 40.417171 [I 2022-10-08 21:35:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 1820/2304 [I 2022-10-08 21:35:29] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:06:49 [I 2022-10-08 21:35:29] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00300 [I 2022-10-08 21:35:29] ...trap/engines/engine.py.230: loss: 0.21112 [I 2022-10-08 21:35:29] ...optimizers/trijoint.py.151: total_norm: 22.589100 [I 2022-10-08 21:35:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 1830/2304 [I 2022-10-08 21:35:38] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:06:39 [I 2022-10-08 21:35:38] ...trap/engines/engine.py.229: process: 0.84023 | load: 0.00300 [I 2022-10-08 21:35:38] ...trap/engines/engine.py.230: loss: 0.14142 [I 2022-10-08 21:35:38] ...optimizers/trijoint.py.151: total_norm: 9.260135 [I 2022-10-08 21:35:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 1840/2304 [I 2022-10-08 21:35:46] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:30 [I 2022-10-08 21:35:46] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00301 [I 2022-10-08 21:35:46] ...trap/engines/engine.py.230: loss: 0.13002 [I 2022-10-08 21:35:46] ...optimizers/trijoint.py.151: total_norm: 16.169491 [I 2022-10-08 21:35:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 1850/2304 [I 2022-10-08 21:35:54] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:19 [I 2022-10-08 21:35:54] ...trap/engines/engine.py.229: process: 0.83386 | load: 0.00305 [I 2022-10-08 21:35:54] ...trap/engines/engine.py.230: loss: 0.13550 [I 2022-10-08 21:35:54] ...optimizers/trijoint.py.151: total_norm: 15.174891 [I 2022-10-08 21:36:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 1860/2304 [I 2022-10-08 21:36:03] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:13 [I 2022-10-08 21:36:03] ...trap/engines/engine.py.229: process: 0.83540 | load: 0.00298 [I 2022-10-08 21:36:03] ...trap/engines/engine.py.230: loss: 0.20833 [I 2022-10-08 21:36:03] ...optimizers/trijoint.py.151: total_norm: 19.507605 [I 2022-10-08 21:36:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 1870/2304 [I 2022-10-08 21:36:11] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:04 [I 2022-10-08 21:36:11] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00300 [I 2022-10-08 21:36:11] ...trap/engines/engine.py.230: loss: 0.17504 [I 2022-10-08 21:36:11] ...optimizers/trijoint.py.151: total_norm: 22.432596 [I 2022-10-08 21:36:20] ...trap/engines/engine.py.224: train: epoch 20 | batch 1880/2304 [I 2022-10-08 21:36:20] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:05:57 [I 2022-10-08 21:36:20] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00297 [I 2022-10-08 21:36:20] ...trap/engines/engine.py.230: loss: 0.21837 [I 2022-10-08 21:36:20] ...optimizers/trijoint.py.151: total_norm: 23.608555 [I 2022-10-08 21:36:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 1890/2304 [I 2022-10-08 21:36:28] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:05:46 [I 2022-10-08 21:36:28] ...trap/engines/engine.py.229: process: 0.83638 | load: 0.00298 [I 2022-10-08 21:36:28] ...trap/engines/engine.py.230: loss: 0.20783 [I 2022-10-08 21:36:28] ...optimizers/trijoint.py.151: total_norm: 16.373541 [I 2022-10-08 21:36:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 1900/2304 [I 2022-10-08 21:36:36] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:39 [I 2022-10-08 21:36:36] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00299 [I 2022-10-08 21:36:36] ...trap/engines/engine.py.230: loss: 0.19239 [I 2022-10-08 21:36:36] ...optimizers/trijoint.py.151: total_norm: 29.034533 [I 2022-10-08 21:36:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 1910/2304 [I 2022-10-08 21:36:45] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:29 [I 2022-10-08 21:36:45] ...trap/engines/engine.py.229: process: 0.83589 | load: 0.00299 [I 2022-10-08 21:36:45] ...trap/engines/engine.py.230: loss: 0.15438 [I 2022-10-08 21:36:45] ...optimizers/trijoint.py.151: total_norm: 19.404043 [I 2022-10-08 21:36:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 1920/2304 [I 2022-10-08 21:36:53] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:22 [I 2022-10-08 21:36:53] ...trap/engines/engine.py.229: process: 0.83105 | load: 0.00303 [I 2022-10-08 21:36:53] ...trap/engines/engine.py.230: loss: 0.26178 [I 2022-10-08 21:36:53] ...optimizers/trijoint.py.151: total_norm: 55.335266 [I 2022-10-08 21:37:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 1930/2304 [I 2022-10-08 21:37:02] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:14 [I 2022-10-08 21:37:02] ...trap/engines/engine.py.229: process: 0.83829 | load: 0.00305 [I 2022-10-08 21:37:02] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-08 21:37:02] ...optimizers/trijoint.py.151: total_norm: 21.910992 [I 2022-10-08 21:37:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 1940/2304 [I 2022-10-08 21:37:10] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:06 [I 2022-10-08 21:37:10] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00299 [I 2022-10-08 21:37:10] ...trap/engines/engine.py.230: loss: 0.18071 [I 2022-10-08 21:37:10] ...optimizers/trijoint.py.151: total_norm: 29.165071 [I 2022-10-08 21:37:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 1950/2304 [I 2022-10-08 21:37:19] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:04:58 [I 2022-10-08 21:37:19] ...trap/engines/engine.py.229: process: 0.83195 | load: 0.00298 [I 2022-10-08 21:37:19] ...trap/engines/engine.py.230: loss: 0.17444 [I 2022-10-08 21:37:19] ...optimizers/trijoint.py.151: total_norm: 20.049864 [I 2022-10-08 21:37:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 1960/2304 [I 2022-10-08 21:37:27] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:04:49 [I 2022-10-08 21:37:27] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00304 [I 2022-10-08 21:37:27] ...trap/engines/engine.py.230: loss: 0.21215 [I 2022-10-08 21:37:27] ...optimizers/trijoint.py.151: total_norm: 22.773687 [I 2022-10-08 21:37:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 1970/2304 [I 2022-10-08 21:37:35] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:42 [I 2022-10-08 21:37:35] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00299 [I 2022-10-08 21:37:35] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-08 21:37:35] ...optimizers/trijoint.py.151: total_norm: 15.946328 [I 2022-10-08 21:37:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 1980/2304 [I 2022-10-08 21:37:44] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:33 [I 2022-10-08 21:37:44] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00300 [I 2022-10-08 21:37:44] ...trap/engines/engine.py.230: loss: 0.16293 [I 2022-10-08 21:37:44] ...optimizers/trijoint.py.151: total_norm: 18.458239 [I 2022-10-08 21:37:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 1990/2304 [I 2022-10-08 21:37:52] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:23 [I 2022-10-08 21:37:52] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00301 [I 2022-10-08 21:37:52] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-08 21:37:52] ...optimizers/trijoint.py.151: total_norm: 15.092402 [I 2022-10-08 21:38:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 2000/2304 [I 2022-10-08 21:38:01] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:14 [I 2022-10-08 21:38:01] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00308 [I 2022-10-08 21:38:01] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-08 21:38:01] ...optimizers/trijoint.py.151: total_norm: 23.365555 [I 2022-10-08 21:38:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 2010/2304 [I 2022-10-08 21:38:09] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:06 [I 2022-10-08 21:38:09] ...trap/engines/engine.py.229: process: 0.82867 | load: 0.00303 [I 2022-10-08 21:38:09] ...trap/engines/engine.py.230: loss: 0.19001 [I 2022-10-08 21:38:09] ...optimizers/trijoint.py.151: total_norm: 14.626734 [I 2022-10-08 21:38:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 2020/2304 [I 2022-10-08 21:38:17] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:03:58 [I 2022-10-08 21:38:17] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00298 [I 2022-10-08 21:38:17] ...trap/engines/engine.py.230: loss: 0.17111 [I 2022-10-08 21:38:17] ...optimizers/trijoint.py.151: total_norm: 35.450424 [I 2022-10-08 21:38:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 2030/2304 [I 2022-10-08 21:38:26] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:03:49 [I 2022-10-08 21:38:26] ...trap/engines/engine.py.229: process: 0.82208 | load: 0.00299 [I 2022-10-08 21:38:26] ...trap/engines/engine.py.230: loss: 0.18974 [I 2022-10-08 21:38:26] ...optimizers/trijoint.py.151: total_norm: 17.302111 [I 2022-10-08 21:38:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 2040/2304 [I 2022-10-08 21:38:34] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:03:41 [I 2022-10-08 21:38:34] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00297 [I 2022-10-08 21:38:34] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-08 21:38:34] ...optimizers/trijoint.py.151: total_norm: 16.034203 [I 2022-10-08 21:38:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 2050/2304 [I 2022-10-08 21:38:43] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:34 [I 2022-10-08 21:38:43] ...trap/engines/engine.py.229: process: 0.85778 | load: 0.00300 [I 2022-10-08 21:38:43] ...trap/engines/engine.py.230: loss: 0.17585 [I 2022-10-08 21:38:43] ...optimizers/trijoint.py.151: total_norm: 15.525680 [I 2022-10-08 21:38:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 2060/2304 [I 2022-10-08 21:38:51] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:24 [I 2022-10-08 21:38:51] ...trap/engines/engine.py.229: process: 0.82473 | load: 0.00299 [I 2022-10-08 21:38:51] ...trap/engines/engine.py.230: loss: 0.19553 [I 2022-10-08 21:38:51] ...optimizers/trijoint.py.151: total_norm: 24.680876 [I 2022-10-08 21:38:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 2070/2304 [I 2022-10-08 21:38:59] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:14 [I 2022-10-08 21:38:59] ...trap/engines/engine.py.229: process: 0.83139 | load: 0.00297 [I 2022-10-08 21:38:59] ...trap/engines/engine.py.230: loss: 0.20781 [I 2022-10-08 21:38:59] ...optimizers/trijoint.py.151: total_norm: 21.318985 [I 2022-10-08 21:39:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 2080/2304 [I 2022-10-08 21:39:08] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:08 [I 2022-10-08 21:39:08] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00298 [I 2022-10-08 21:39:08] ...trap/engines/engine.py.230: loss: 0.24420 [I 2022-10-08 21:39:08] ...optimizers/trijoint.py.151: total_norm: 29.055006 [I 2022-10-08 21:39:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 2090/2304 [I 2022-10-08 21:39:16] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:02:58 [I 2022-10-08 21:39:16] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00299 [I 2022-10-08 21:39:16] ...trap/engines/engine.py.230: loss: 0.20113 [I 2022-10-08 21:39:16] ...optimizers/trijoint.py.151: total_norm: 17.931858 [I 2022-10-08 21:39:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 2100/2304 [I 2022-10-08 21:39:25] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:02:50 [I 2022-10-08 21:39:25] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00297 [I 2022-10-08 21:39:25] ...trap/engines/engine.py.230: loss: 0.25800 [I 2022-10-08 21:39:25] ...optimizers/trijoint.py.151: total_norm: 20.800732 [I 2022-10-08 21:39:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 2110/2304 [I 2022-10-08 21:39:33] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:02:42 [I 2022-10-08 21:39:33] ...trap/engines/engine.py.229: process: 0.83997 | load: 0.00299 [I 2022-10-08 21:39:33] ...trap/engines/engine.py.230: loss: 0.16589 [I 2022-10-08 21:39:33] ...optimizers/trijoint.py.151: total_norm: 19.898100 [I 2022-10-08 21:39:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 2120/2304 [I 2022-10-08 21:39:41] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:33 [I 2022-10-08 21:39:41] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00300 [I 2022-10-08 21:39:41] ...trap/engines/engine.py.230: loss: 0.20618 [I 2022-10-08 21:39:41] ...optimizers/trijoint.py.151: total_norm: 26.634777 [I 2022-10-08 21:39:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 2130/2304 [I 2022-10-08 21:39:50] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:25 [I 2022-10-08 21:39:50] ...trap/engines/engine.py.229: process: 0.82863 | load: 0.00300 [I 2022-10-08 21:39:50] ...trap/engines/engine.py.230: loss: 0.17368 [I 2022-10-08 21:39:50] ...optimizers/trijoint.py.151: total_norm: 22.408957 [I 2022-10-08 21:39:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 2140/2304 [I 2022-10-08 21:39:58] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:17 [I 2022-10-08 21:39:58] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00301 [I 2022-10-08 21:39:58] ...trap/engines/engine.py.230: loss: 0.18601 [I 2022-10-08 21:39:58] ...optimizers/trijoint.py.151: total_norm: 22.128729 [I 2022-10-08 21:40:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 2150/2304 [I 2022-10-08 21:40:07] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:09 [I 2022-10-08 21:40:07] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00299 [I 2022-10-08 21:40:07] ...trap/engines/engine.py.230: loss: 0.26646 [I 2022-10-08 21:40:07] ...optimizers/trijoint.py.151: total_norm: 18.491877 [I 2022-10-08 21:40:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 2160/2304 [I 2022-10-08 21:40:15] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:00 [I 2022-10-08 21:40:15] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00307 [I 2022-10-08 21:40:15] ...trap/engines/engine.py.230: loss: 0.12837 [I 2022-10-08 21:40:15] ...optimizers/trijoint.py.151: total_norm: 20.027538 [I 2022-10-08 21:40:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 2170/2304 [I 2022-10-08 21:40:24] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:01:53 [I 2022-10-08 21:40:24] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00304 [I 2022-10-08 21:40:24] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-08 21:40:24] ...optimizers/trijoint.py.151: total_norm: 19.099794 [I 2022-10-08 21:40:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 2180/2304 [I 2022-10-08 21:40:32] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:44 [I 2022-10-08 21:40:32] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00310 [I 2022-10-08 21:40:32] ...trap/engines/engine.py.230: loss: 0.21981 [I 2022-10-08 21:40:32] ...optimizers/trijoint.py.151: total_norm: 22.513281 [I 2022-10-08 21:40:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 2190/2304 [I 2022-10-08 21:40:40] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:01:35 [I 2022-10-08 21:40:40] ...trap/engines/engine.py.229: process: 0.84712 | load: 0.00305 [I 2022-10-08 21:40:40] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-08 21:40:40] ...optimizers/trijoint.py.151: total_norm: 15.823851 [I 2022-10-08 21:40:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 2200/2304 [I 2022-10-08 21:40:49] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:27 [I 2022-10-08 21:40:49] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00309 [I 2022-10-08 21:40:49] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-08 21:40:49] ...optimizers/trijoint.py.151: total_norm: 15.550804 [I 2022-10-08 21:40:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 2210/2304 [I 2022-10-08 21:40:57] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:19 [I 2022-10-08 21:40:57] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00314 [I 2022-10-08 21:40:57] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-08 21:40:57] ...optimizers/trijoint.py.151: total_norm: 15.722878 [I 2022-10-08 21:41:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 2220/2304 [I 2022-10-08 21:41:06] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:10 [I 2022-10-08 21:41:06] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00312 [I 2022-10-08 21:41:06] ...trap/engines/engine.py.230: loss: 0.21135 [I 2022-10-08 21:41:06] ...optimizers/trijoint.py.151: total_norm: 40.620110 [I 2022-10-08 21:41:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 2230/2304 [I 2022-10-08 21:41:14] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:02 [I 2022-10-08 21:41:14] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00306 [I 2022-10-08 21:41:14] ...trap/engines/engine.py.230: loss: 0.20604 [I 2022-10-08 21:41:14] ...optimizers/trijoint.py.151: total_norm: 21.410158 [I 2022-10-08 21:41:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 2240/2304 [I 2022-10-08 21:41:22] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:00:53 [I 2022-10-08 21:41:22] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00303 [I 2022-10-08 21:41:22] ...trap/engines/engine.py.230: loss: 0.20188 [I 2022-10-08 21:41:22] ...optimizers/trijoint.py.151: total_norm: 18.292416 [I 2022-10-08 21:41:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 2250/2304 [I 2022-10-08 21:41:31] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:00:45 [I 2022-10-08 21:41:31] ...trap/engines/engine.py.229: process: 0.83367 | load: 0.00299 [I 2022-10-08 21:41:31] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-08 21:41:31] ...optimizers/trijoint.py.151: total_norm: 18.338242 [I 2022-10-08 21:41:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 2260/2304 [I 2022-10-08 21:41:39] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:00:36 [I 2022-10-08 21:41:39] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00306 [I 2022-10-08 21:41:39] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-08 21:41:39] ...optimizers/trijoint.py.151: total_norm: 18.369198 [I 2022-10-08 21:41:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 2270/2304 [I 2022-10-08 21:41:48] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:00:28 [I 2022-10-08 21:41:48] ...trap/engines/engine.py.229: process: 0.82969 | load: 0.00305 [I 2022-10-08 21:41:48] ...trap/engines/engine.py.230: loss: 0.29421 [I 2022-10-08 21:41:48] ...optimizers/trijoint.py.151: total_norm: 23.261482 [I 2022-10-08 21:41:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 2280/2304 [I 2022-10-08 21:41:56] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:20 [I 2022-10-08 21:41:56] ...trap/engines/engine.py.229: process: 0.82995 | load: 0.00299 [I 2022-10-08 21:41:56] ...trap/engines/engine.py.230: loss: 0.16286 [I 2022-10-08 21:41:56] ...optimizers/trijoint.py.151: total_norm: 19.573523 [I 2022-10-08 21:42:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 2290/2304 [I 2022-10-08 21:42:05] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:11 [I 2022-10-08 21:42:05] ...trap/engines/engine.py.229: process: 0.83851 | load: 0.00300 [I 2022-10-08 21:42:05] ...trap/engines/engine.py.230: loss: 0.12919 [I 2022-10-08 21:42:05] ...optimizers/trijoint.py.151: total_norm: 11.287372 [I 2022-10-08 21:42:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 2300/2304 [I 2022-10-08 21:42:13] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:03 [I 2022-10-08 21:42:13] ...trap/engines/engine.py.229: process: 0.82088 | load: 0.00016 [I 2022-10-08 21:42:13] ...trap/engines/engine.py.230: loss: 0.16631 [I 2022-10-08 21:42:13] ...optimizers/trijoint.py.151: total_norm: 10.601262 [I 2022-10-08 21:42:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 2304/2304 [I 2022-10-08 21:42:16] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:00 [I 2022-10-08 21:42:16] ...trap/engines/engine.py.229: process: 0.83360 | load: 0.00017 [I 2022-10-08 21:42:16] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-08 21:42:16] ...optimizers/trijoint.py.151: total_norm: 30.085052 [S 2022-10-08 21:42:17] ...trap/engines/engine.py.129: train_epoch.epoch: 20 [S 2022-10-08 21:42:17] ...trap/engines/engine.py.129: train_epoch.loss: 0.176779823845612 [I 2022-10-08 21:42:17] ...s/criterions/triplet.py.84: margin = 0.15500000000000008 [I 2022-10-08 21:42:18] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 20 [I 2022-10-08 21:42:18] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 21:42:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 21:42:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 21:42:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 21:42:20] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 21:42:22] ...trap/engines/engine.py.321: eval: epoch 20 | batch 0/510 [I 2022-10-08 21:42:22] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:34:28 [I 2022-10-08 21:42:22] ...trap/engines/engine.py.326: process: 4.05657 | load: 2.39452 [I 2022-10-08 21:42:27] ...trap/engines/engine.py.321: eval: epoch 20 | batch 10/510 [I 2022-10-08 21:42:27] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:59 [I 2022-10-08 21:42:27] ...trap/engines/engine.py.326: process: 0.40265 | load: 0.00020 [I 2022-10-08 21:42:33] ...trap/engines/engine.py.321: eval: epoch 20 | batch 20/510 [I 2022-10-08 21:42:33] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:04 [I 2022-10-08 21:42:33] ...trap/engines/engine.py.326: process: 0.61058 | load: 0.00020 [I 2022-10-08 21:42:40] ...trap/engines/engine.py.321: eval: epoch 20 | batch 30/510 [I 2022-10-08 21:42:40] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:57 [I 2022-10-08 21:42:40] ...trap/engines/engine.py.326: process: 0.63075 | load: 0.00019 [I 2022-10-08 21:42:45] ...trap/engines/engine.py.321: eval: epoch 20 | batch 40/510 [I 2022-10-08 21:42:45] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:01 [I 2022-10-08 21:42:45] ...trap/engines/engine.py.326: process: 0.51353 | load: 0.00019 [I 2022-10-08 21:42:49] ...trap/engines/engine.py.321: eval: epoch 20 | batch 50/510 [I 2022-10-08 21:42:49] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:37 [I 2022-10-08 21:42:49] ...trap/engines/engine.py.326: process: 0.50122 | load: 0.00018 [I 2022-10-08 21:42:54] ...trap/engines/engine.py.321: eval: epoch 20 | batch 60/510 [I 2022-10-08 21:42:54] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:43 [I 2022-10-08 21:42:54] ...trap/engines/engine.py.326: process: 0.43107 | load: 0.00019 [I 2022-10-08 21:43:00] ...trap/engines/engine.py.321: eval: epoch 20 | batch 70/510 [I 2022-10-08 21:43:00] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:02 [I 2022-10-08 21:43:00] ...trap/engines/engine.py.326: process: 0.66888 | load: 0.00021 [I 2022-10-08 21:43:05] ...trap/engines/engine.py.321: eval: epoch 20 | batch 80/510 [I 2022-10-08 21:43:05] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:50 [I 2022-10-08 21:43:05] ...trap/engines/engine.py.326: process: 0.66095 | load: 0.00021 [I 2022-10-08 21:43:10] ...trap/engines/engine.py.321: eval: epoch 20 | batch 90/510 [I 2022-10-08 21:43:10] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:50 [I 2022-10-08 21:43:10] ...trap/engines/engine.py.326: process: 0.42351 | load: 0.00019 [I 2022-10-08 21:43:14] ...trap/engines/engine.py.321: eval: epoch 20 | batch 100/510 [I 2022-10-08 21:43:14] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:59 [I 2022-10-08 21:43:14] ...trap/engines/engine.py.326: process: 0.31106 | load: 0.00019 [I 2022-10-08 21:43:18] ...trap/engines/engine.py.321: eval: epoch 20 | batch 110/510 [I 2022-10-08 21:43:18] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:11 [I 2022-10-08 21:43:18] ...trap/engines/engine.py.326: process: 0.31319 | load: 0.00020 [I 2022-10-08 21:43:21] ...trap/engines/engine.py.321: eval: epoch 20 | batch 120/510 [I 2022-10-08 21:43:21] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:04 [I 2022-10-08 21:43:21] ...trap/engines/engine.py.326: process: 0.31996 | load: 0.00019 [I 2022-10-08 21:43:24] ...trap/engines/engine.py.321: eval: epoch 20 | batch 130/510 [I 2022-10-08 21:43:24] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:01 [I 2022-10-08 21:43:24] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00020 [I 2022-10-08 21:43:27] ...trap/engines/engine.py.321: eval: epoch 20 | batch 140/510 [I 2022-10-08 21:43:27] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:57 [I 2022-10-08 21:43:27] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00019 [I 2022-10-08 21:43:30] ...trap/engines/engine.py.321: eval: epoch 20 | batch 150/510 [I 2022-10-08 21:43:30] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-08 21:43:30] ...trap/engines/engine.py.326: process: 0.31387 | load: 0.00020 [I 2022-10-08 21:43:33] ...trap/engines/engine.py.321: eval: epoch 20 | batch 160/510 [I 2022-10-08 21:43:33] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:51 [I 2022-10-08 21:43:33] ...trap/engines/engine.py.326: process: 0.31724 | load: 0.00019 [I 2022-10-08 21:43:37] ...trap/engines/engine.py.321: eval: epoch 20 | batch 170/510 [I 2022-10-08 21:43:37] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:48 [I 2022-10-08 21:43:37] ...trap/engines/engine.py.326: process: 0.32083 | load: 0.00019 [I 2022-10-08 21:43:40] ...trap/engines/engine.py.321: eval: epoch 20 | batch 180/510 [I 2022-10-08 21:43:40] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:44 [I 2022-10-08 21:43:40] ...trap/engines/engine.py.326: process: 0.31346 | load: 0.00020 [I 2022-10-08 21:43:43] ...trap/engines/engine.py.321: eval: epoch 20 | batch 190/510 [I 2022-10-08 21:43:43] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:41 [I 2022-10-08 21:43:43] ...trap/engines/engine.py.326: process: 0.30922 | load: 0.00020 [I 2022-10-08 21:43:46] ...trap/engines/engine.py.321: eval: epoch 20 | batch 200/510 [I 2022-10-08 21:43:46] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:38 [I 2022-10-08 21:43:46] ...trap/engines/engine.py.326: process: 0.32015 | load: 0.00020 [I 2022-10-08 21:43:49] ...trap/engines/engine.py.321: eval: epoch 20 | batch 210/510 [I 2022-10-08 21:43:49] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-08 21:43:49] ...trap/engines/engine.py.326: process: 0.31991 | load: 0.00020 [I 2022-10-08 21:43:53] ...trap/engines/engine.py.321: eval: epoch 20 | batch 220/510 [I 2022-10-08 21:43:53] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:32 [I 2022-10-08 21:43:53] ...trap/engines/engine.py.326: process: 0.31614 | load: 0.00020 [I 2022-10-08 21:43:56] ...trap/engines/engine.py.321: eval: epoch 20 | batch 230/510 [I 2022-10-08 21:43:56] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:28 [I 2022-10-08 21:43:56] ...trap/engines/engine.py.326: process: 0.31212 | load: 0.00021 [I 2022-10-08 21:43:59] ...trap/engines/engine.py.321: eval: epoch 20 | batch 240/510 [I 2022-10-08 21:43:59] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-08 21:43:59] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00020 [I 2022-10-08 21:44:02] ...trap/engines/engine.py.321: eval: epoch 20 | batch 250/510 [I 2022-10-08 21:44:02] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:23 [I 2022-10-08 21:44:02] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00020 [I 2022-10-08 21:44:05] ...trap/engines/engine.py.321: eval: epoch 20 | batch 260/510 [I 2022-10-08 21:44:05] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:18 [I 2022-10-08 21:44:05] ...trap/engines/engine.py.326: process: 0.30839 | load: 0.00021 [I 2022-10-08 21:44:08] ...trap/engines/engine.py.321: eval: epoch 20 | batch 270/510 [I 2022-10-08 21:44:08] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:16 [I 2022-10-08 21:44:08] ...trap/engines/engine.py.326: process: 0.31822 | load: 0.00021 [I 2022-10-08 21:44:12] ...trap/engines/engine.py.321: eval: epoch 20 | batch 280/510 [I 2022-10-08 21:44:12] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-08 21:44:12] ...trap/engines/engine.py.326: process: 0.32206 | load: 0.00020 [I 2022-10-08 21:44:15] ...trap/engines/engine.py.321: eval: epoch 20 | batch 290/510 [I 2022-10-08 21:44:15] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-08 21:44:15] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00020 [I 2022-10-08 21:44:18] ...trap/engines/engine.py.321: eval: epoch 20 | batch 300/510 [I 2022-10-08 21:44:18] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-08 21:44:18] ...trap/engines/engine.py.326: process: 0.32344 | load: 0.00021 [I 2022-10-08 21:44:21] ...trap/engines/engine.py.321: eval: epoch 20 | batch 310/510 [I 2022-10-08 21:44:21] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:04 [I 2022-10-08 21:44:21] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00020 [I 2022-10-08 21:44:24] ...trap/engines/engine.py.321: eval: epoch 20 | batch 320/510 [I 2022-10-08 21:44:24] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-08 21:44:24] ...trap/engines/engine.py.326: process: 0.32074 | load: 0.00020 [I 2022-10-08 21:44:28] ...trap/engines/engine.py.321: eval: epoch 20 | batch 330/510 [I 2022-10-08 21:44:28] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-08 21:44:28] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00021 [I 2022-10-08 21:44:31] ...trap/engines/engine.py.321: eval: epoch 20 | batch 340/510 [I 2022-10-08 21:44:31] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:53 [I 2022-10-08 21:44:31] ...trap/engines/engine.py.326: process: 0.32091 | load: 0.00021 [I 2022-10-08 21:44:34] ...trap/engines/engine.py.321: eval: epoch 20 | batch 350/510 [I 2022-10-08 21:44:34] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:50 [I 2022-10-08 21:44:34] ...trap/engines/engine.py.326: process: 0.32018 | load: 0.00020 [I 2022-10-08 21:44:37] ...trap/engines/engine.py.321: eval: epoch 20 | batch 360/510 [I 2022-10-08 21:44:37] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-08 21:44:37] ...trap/engines/engine.py.326: process: 0.31635 | load: 0.00020 [I 2022-10-08 21:44:40] ...trap/engines/engine.py.321: eval: epoch 20 | batch 370/510 [I 2022-10-08 21:44:40] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-08 21:44:40] ...trap/engines/engine.py.326: process: 0.31502 | load: 0.00020 [I 2022-10-08 21:44:44] ...trap/engines/engine.py.321: eval: epoch 20 | batch 380/510 [I 2022-10-08 21:44:44] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-08 21:44:44] ...trap/engines/engine.py.326: process: 0.31963 | load: 0.00020 [I 2022-10-08 21:44:47] ...trap/engines/engine.py.321: eval: epoch 20 | batch 390/510 [I 2022-10-08 21:44:47] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:37 [I 2022-10-08 21:44:47] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00021 [I 2022-10-08 21:44:50] ...trap/engines/engine.py.321: eval: epoch 20 | batch 400/510 [I 2022-10-08 21:44:50] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-08 21:44:50] ...trap/engines/engine.py.326: process: 0.31838 | load: 0.00021 [I 2022-10-08 21:44:53] ...trap/engines/engine.py.321: eval: epoch 20 | batch 410/510 [I 2022-10-08 21:44:53] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-08 21:44:53] ...trap/engines/engine.py.326: process: 0.31814 | load: 0.00020 [I 2022-10-08 21:44:56] ...trap/engines/engine.py.321: eval: epoch 20 | batch 420/510 [I 2022-10-08 21:44:56] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-08 21:44:56] ...trap/engines/engine.py.326: process: 0.31525 | load: 0.00020 [I 2022-10-08 21:44:59] ...trap/engines/engine.py.321: eval: epoch 20 | batch 430/510 [I 2022-10-08 21:44:59] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-08 21:44:59] ...trap/engines/engine.py.326: process: 0.31399 | load: 0.00020 [I 2022-10-08 21:45:03] ...trap/engines/engine.py.321: eval: epoch 20 | batch 440/510 [I 2022-10-08 21:45:03] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-08 21:45:03] ...trap/engines/engine.py.326: process: 0.32359 | load: 0.00020 [I 2022-10-08 21:45:06] ...trap/engines/engine.py.321: eval: epoch 20 | batch 450/510 [I 2022-10-08 21:45:06] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-08 21:45:06] ...trap/engines/engine.py.326: process: 0.31522 | load: 0.00021 [I 2022-10-08 21:45:09] ...trap/engines/engine.py.321: eval: epoch 20 | batch 460/510 [I 2022-10-08 21:45:09] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-08 21:45:09] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00020 [I 2022-10-08 21:45:12] ...trap/engines/engine.py.321: eval: epoch 20 | batch 470/510 [I 2022-10-08 21:45:12] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-08 21:45:12] ...trap/engines/engine.py.326: process: 0.31219 | load: 0.00019 [I 2022-10-08 21:45:15] ...trap/engines/engine.py.321: eval: epoch 20 | batch 480/510 [I 2022-10-08 21:45:15] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-08 21:45:15] ...trap/engines/engine.py.326: process: 0.31756 | load: 0.00020 [I 2022-10-08 21:45:18] ...trap/engines/engine.py.321: eval: epoch 20 | batch 490/510 [I 2022-10-08 21:45:18] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-08 21:45:18] ...trap/engines/engine.py.326: process: 0.31656 | load: 0.00020 [I 2022-10-08 21:45:22] ...trap/engines/engine.py.321: eval: epoch 20 | batch 500/510 [I 2022-10-08 21:45:22] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-08 21:45:22] ...trap/engines/engine.py.326: process: 0.31216 | load: 0.00020 [I 2022-10-08 21:45:25] ...trap/engines/engine.py.321: eval: epoch 20 | batch 510/510 [I 2022-10-08 21:45:25] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-08 21:45:25] ...trap/engines/engine.py.326: process: 0.31417 | load: 0.00016 [S 2022-10-08 21:45:26] ...trap/engines/engine.py.132: eval_epoch.epoch: 20 [I 2022-10-08 21:45:26] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.641500 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.871700 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.918700 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.631800 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.871700 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.915000 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014861 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007682 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007950 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014246 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010080 [S 2022-10-08 21:45:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009747 [I 2022-10-08 21:46:06] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 21:46:06] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 21:46:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 21:46:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 21:46:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 21:46:07] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 21:46:09] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 21:46:16] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 21:46:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 21:46:16] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 21:46:19] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 21:46:24] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 21:46:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 21 [I 2022-10-08 21:46:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 21:46:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 0/2304 [I 2022-10-08 21:46:29] ...trap/engines/engine.py.228: elapsed: 0:00:04 | left: 3:02:57 [I 2022-10-08 21:46:29] ...trap/engines/engine.py.229: process: 4.76460 | load: 2.11089 [I 2022-10-08 21:46:29] ...trap/engines/engine.py.230: loss: 0.18599 [I 2022-10-08 21:46:29] ...optimizers/trijoint.py.151: total_norm: 20.322714 [I 2022-10-08 21:46:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 10/2304 [I 2022-10-08 21:46:40] ...trap/engines/engine.py.228: elapsed: 0:00:15 | left: 0:51:56 [I 2022-10-08 21:46:40] ...trap/engines/engine.py.229: process: 0.83357 | load: 0.00577 [I 2022-10-08 21:46:40] ...trap/engines/engine.py.230: loss: 0.15200 [I 2022-10-08 21:46:40] ...optimizers/trijoint.py.151: total_norm: 14.421526 [I 2022-10-08 21:46:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 20/2304 [I 2022-10-08 21:46:48] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:34:20 [I 2022-10-08 21:46:48] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00536 [I 2022-10-08 21:46:48] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-08 21:46:48] ...optimizers/trijoint.py.151: total_norm: 14.433366 [I 2022-10-08 21:46:57] ...trap/engines/engine.py.224: train: epoch 21 | batch 30/2304 [I 2022-10-08 21:46:57] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:31:59 [I 2022-10-08 21:46:57] ...trap/engines/engine.py.229: process: 0.83152 | load: 0.00460 [I 2022-10-08 21:46:57] ...trap/engines/engine.py.230: loss: 0.11467 [I 2022-10-08 21:46:57] ...optimizers/trijoint.py.151: total_norm: 17.222563 [I 2022-10-08 21:47:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 40/2304 [I 2022-10-08 21:47:05] ...trap/engines/engine.py.228: elapsed: 0:00:40 | left: 0:31:59 [I 2022-10-08 21:47:05] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00315 [I 2022-10-08 21:47:05] ...trap/engines/engine.py.230: loss: 0.21681 [I 2022-10-08 21:47:05] ...optimizers/trijoint.py.151: total_norm: 20.867001 [I 2022-10-08 21:47:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 50/2304 [I 2022-10-08 21:47:14] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:31:41 [I 2022-10-08 21:47:14] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00350 [I 2022-10-08 21:47:14] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-08 21:47:14] ...optimizers/trijoint.py.151: total_norm: 25.161556 [I 2022-10-08 21:47:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 60/2304 [I 2022-10-08 21:47:22] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:31:28 [I 2022-10-08 21:47:22] ...trap/engines/engine.py.229: process: 0.84038 | load: 0.00351 [I 2022-10-08 21:47:22] ...trap/engines/engine.py.230: loss: 0.14594 [I 2022-10-08 21:47:22] ...optimizers/trijoint.py.151: total_norm: 21.326090 [I 2022-10-08 21:47:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 70/2304 [I 2022-10-08 21:47:31] ...trap/engines/engine.py.228: elapsed: 0:01:06 | left: 0:31:10 [I 2022-10-08 21:47:31] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00386 [I 2022-10-08 21:47:31] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-08 21:47:31] ...optimizers/trijoint.py.151: total_norm: 24.462461 [I 2022-10-08 21:47:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 80/2304 [I 2022-10-08 21:47:39] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:31:10 [I 2022-10-08 21:47:39] ...trap/engines/engine.py.229: process: 0.83486 | load: 0.00329 [I 2022-10-08 21:47:39] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-08 21:47:39] ...optimizers/trijoint.py.151: total_norm: 25.014351 [I 2022-10-08 21:47:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 90/2304 [I 2022-10-08 21:47:47] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:31:07 [I 2022-10-08 21:47:47] ...trap/engines/engine.py.229: process: 0.83921 | load: 0.00331 [I 2022-10-08 21:47:47] ...trap/engines/engine.py.230: loss: 0.20359 [I 2022-10-08 21:47:47] ...optimizers/trijoint.py.151: total_norm: 13.518027 [I 2022-10-08 21:47:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 100/2304 [I 2022-10-08 21:47:56] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:30:52 [I 2022-10-08 21:47:56] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00324 [I 2022-10-08 21:47:56] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-08 21:47:56] ...optimizers/trijoint.py.151: total_norm: 16.475611 [I 2022-10-08 21:48:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 110/2304 [I 2022-10-08 21:48:05] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:30:52 [I 2022-10-08 21:48:05] ...trap/engines/engine.py.229: process: 0.83716 | load: 0.00349 [I 2022-10-08 21:48:05] ...trap/engines/engine.py.230: loss: 0.12756 [I 2022-10-08 21:48:05] ...optimizers/trijoint.py.151: total_norm: 16.432108 [I 2022-10-08 21:48:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 120/2304 [I 2022-10-08 21:48:13] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:30:32 [I 2022-10-08 21:48:13] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00336 [I 2022-10-08 21:48:13] ...trap/engines/engine.py.230: loss: 0.15371 [I 2022-10-08 21:48:13] ...optimizers/trijoint.py.151: total_norm: 12.067206 [I 2022-10-08 21:48:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 130/2304 [I 2022-10-08 21:48:21] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:30:33 [I 2022-10-08 21:48:21] ...trap/engines/engine.py.229: process: 0.83648 | load: 0.00318 [I 2022-10-08 21:48:21] ...trap/engines/engine.py.230: loss: 0.15915 [I 2022-10-08 21:48:21] ...optimizers/trijoint.py.151: total_norm: 20.533407 [I 2022-10-08 21:48:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 140/2304 [I 2022-10-08 21:48:30] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:30:26 [I 2022-10-08 21:48:30] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00342 [I 2022-10-08 21:48:30] ...trap/engines/engine.py.230: loss: 0.22121 [I 2022-10-08 21:48:30] ...optimizers/trijoint.py.151: total_norm: 16.975565 [I 2022-10-08 21:48:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 150/2304 [I 2022-10-08 21:48:38] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:01 [I 2022-10-08 21:48:38] ...trap/engines/engine.py.229: process: 0.82389 | load: 0.00319 [I 2022-10-08 21:48:38] ...trap/engines/engine.py.230: loss: 0.08394 [I 2022-10-08 21:48:38] ...optimizers/trijoint.py.151: total_norm: 10.100545 [I 2022-10-08 21:48:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 160/2304 [I 2022-10-08 21:48:47] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:29:52 [I 2022-10-08 21:48:47] ...trap/engines/engine.py.229: process: 0.81505 | load: 0.00298 [I 2022-10-08 21:48:47] ...trap/engines/engine.py.230: loss: 0.28663 [I 2022-10-08 21:48:47] ...optimizers/trijoint.py.151: total_norm: 18.399227 [I 2022-10-08 21:48:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 170/2304 [I 2022-10-08 21:48:55] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:29:48 [I 2022-10-08 21:48:55] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00321 [I 2022-10-08 21:48:55] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-08 21:48:55] ...optimizers/trijoint.py.151: total_norm: 19.308626 [I 2022-10-08 21:49:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 180/2304 [I 2022-10-08 21:49:03] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:29:47 [I 2022-10-08 21:49:03] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00327 [I 2022-10-08 21:49:03] ...trap/engines/engine.py.230: loss: 0.22055 [I 2022-10-08 21:49:03] ...optimizers/trijoint.py.151: total_norm: 21.727274 [I 2022-10-08 21:49:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 190/2304 [I 2022-10-08 21:49:12] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:29:39 [I 2022-10-08 21:49:12] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00321 [I 2022-10-08 21:49:12] ...trap/engines/engine.py.230: loss: 0.27301 [I 2022-10-08 21:49:12] ...optimizers/trijoint.py.151: total_norm: 34.164528 [I 2022-10-08 21:49:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 200/2304 [I 2022-10-08 21:49:20] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:24 [I 2022-10-08 21:49:20] ...trap/engines/engine.py.229: process: 0.83557 | load: 0.00304 [I 2022-10-08 21:49:20] ...trap/engines/engine.py.230: loss: 0.14130 [I 2022-10-08 21:49:20] ...optimizers/trijoint.py.151: total_norm: 33.230316 [I 2022-10-08 21:49:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 210/2304 [I 2022-10-08 21:49:29] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:15 [I 2022-10-08 21:49:29] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00320 [I 2022-10-08 21:49:29] ...trap/engines/engine.py.230: loss: 0.15572 [I 2022-10-08 21:49:29] ...optimizers/trijoint.py.151: total_norm: 16.298031 [I 2022-10-08 21:49:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 220/2304 [I 2022-10-08 21:49:37] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:15 [I 2022-10-08 21:49:37] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00300 [I 2022-10-08 21:49:37] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-08 21:49:37] ...optimizers/trijoint.py.151: total_norm: 22.898134 [I 2022-10-08 21:49:46] ...trap/engines/engine.py.224: train: epoch 21 | batch 230/2304 [I 2022-10-08 21:49:46] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:01 [I 2022-10-08 21:49:46] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00312 [I 2022-10-08 21:49:46] ...trap/engines/engine.py.230: loss: 0.16541 [I 2022-10-08 21:49:46] ...optimizers/trijoint.py.151: total_norm: 21.451426 [I 2022-10-08 21:49:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 240/2304 [I 2022-10-08 21:49:54] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:28:54 [I 2022-10-08 21:49:54] ...trap/engines/engine.py.229: process: 0.83319 | load: 0.00314 [I 2022-10-08 21:49:54] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-08 21:49:54] ...optimizers/trijoint.py.151: total_norm: 15.388783 [I 2022-10-08 21:50:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 250/2304 [I 2022-10-08 21:50:02] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:28:47 [I 2022-10-08 21:50:02] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00306 [I 2022-10-08 21:50:02] ...trap/engines/engine.py.230: loss: 0.17196 [I 2022-10-08 21:50:02] ...optimizers/trijoint.py.151: total_norm: 15.010983 [I 2022-10-08 21:50:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 260/2304 [I 2022-10-08 21:50:11] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:28:37 [I 2022-10-08 21:50:11] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00301 [I 2022-10-08 21:50:11] ...trap/engines/engine.py.230: loss: 0.18212 [I 2022-10-08 21:50:11] ...optimizers/trijoint.py.151: total_norm: 19.981926 [I 2022-10-08 21:50:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 270/2304 [I 2022-10-08 21:50:19] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:40 [I 2022-10-08 21:50:19] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00314 [I 2022-10-08 21:50:19] ...trap/engines/engine.py.230: loss: 0.16715 [I 2022-10-08 21:50:19] ...optimizers/trijoint.py.151: total_norm: 17.243544 [I 2022-10-08 21:50:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 280/2304 [I 2022-10-08 21:50:28] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:29:06 [I 2022-10-08 21:50:28] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00300 [I 2022-10-08 21:50:28] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-08 21:50:28] ...optimizers/trijoint.py.151: total_norm: 11.628616 [I 2022-10-08 21:50:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 290/2304 [I 2022-10-08 21:50:37] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:26 [I 2022-10-08 21:50:37] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00306 [I 2022-10-08 21:50:37] ...trap/engines/engine.py.230: loss: 0.18794 [I 2022-10-08 21:50:37] ...optimizers/trijoint.py.151: total_norm: 18.056280 [I 2022-10-08 21:50:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 300/2304 [I 2022-10-08 21:50:45] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:05 [I 2022-10-08 21:50:45] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00297 [I 2022-10-08 21:50:45] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-08 21:50:45] ...optimizers/trijoint.py.151: total_norm: 16.004559 [I 2022-10-08 21:50:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 310/2304 [I 2022-10-08 21:50:54] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:27:56 [I 2022-10-08 21:50:54] ...trap/engines/engine.py.229: process: 0.84463 | load: 0.00311 [I 2022-10-08 21:50:54] ...trap/engines/engine.py.230: loss: 0.14089 [I 2022-10-08 21:50:54] ...optimizers/trijoint.py.151: total_norm: 16.752178 [I 2022-10-08 21:51:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 320/2304 [I 2022-10-08 21:51:02] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:27:44 [I 2022-10-08 21:51:02] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00310 [I 2022-10-08 21:51:02] ...trap/engines/engine.py.230: loss: 0.12665 [I 2022-10-08 21:51:02] ...optimizers/trijoint.py.151: total_norm: 18.100952 [I 2022-10-08 21:51:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 330/2304 [I 2022-10-08 21:51:10] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:27:34 [I 2022-10-08 21:51:10] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00301 [I 2022-10-08 21:51:10] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-08 21:51:10] ...optimizers/trijoint.py.151: total_norm: 19.455069 [I 2022-10-08 21:51:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 340/2304 [I 2022-10-08 21:51:19] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:27:31 [I 2022-10-08 21:51:19] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00299 [I 2022-10-08 21:51:19] ...trap/engines/engine.py.230: loss: 0.20464 [I 2022-10-08 21:51:19] ...optimizers/trijoint.py.151: total_norm: 17.978422 [I 2022-10-08 21:51:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 350/2304 [I 2022-10-08 21:51:27] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:19 [I 2022-10-08 21:51:27] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00298 [I 2022-10-08 21:51:27] ...trap/engines/engine.py.230: loss: 0.23182 [I 2022-10-08 21:51:27] ...optimizers/trijoint.py.151: total_norm: 20.054831 [I 2022-10-08 21:51:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 360/2304 [I 2022-10-08 21:51:36] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:13 [I 2022-10-08 21:51:36] ...trap/engines/engine.py.229: process: 0.83688 | load: 0.00318 [I 2022-10-08 21:51:36] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-08 21:51:36] ...optimizers/trijoint.py.151: total_norm: 16.743118 [I 2022-10-08 21:51:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 370/2304 [I 2022-10-08 21:51:44] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:06 [I 2022-10-08 21:51:44] ...trap/engines/engine.py.229: process: 0.83050 | load: 0.00303 [I 2022-10-08 21:51:44] ...trap/engines/engine.py.230: loss: 0.16899 [I 2022-10-08 21:51:44] ...optimizers/trijoint.py.151: total_norm: 13.491577 [I 2022-10-08 21:51:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 380/2304 [I 2022-10-08 21:51:53] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:01 [I 2022-10-08 21:51:53] ...trap/engines/engine.py.229: process: 0.84241 | load: 0.00306 [I 2022-10-08 21:51:53] ...trap/engines/engine.py.230: loss: 0.17317 [I 2022-10-08 21:51:53] ...optimizers/trijoint.py.151: total_norm: 19.579880 [I 2022-10-08 21:52:01] ...trap/engines/engine.py.224: train: epoch 21 | batch 390/2304 [I 2022-10-08 21:52:01] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:26:47 [I 2022-10-08 21:52:01] ...trap/engines/engine.py.229: process: 0.83634 | load: 0.00311 [I 2022-10-08 21:52:01] ...trap/engines/engine.py.230: loss: 0.16607 [I 2022-10-08 21:52:01] ...optimizers/trijoint.py.151: total_norm: 33.815269 [I 2022-10-08 21:52:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 400/2304 [I 2022-10-08 21:52:09] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:26:46 [I 2022-10-08 21:52:09] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00296 [I 2022-10-08 21:52:09] ...trap/engines/engine.py.230: loss: 0.13825 [I 2022-10-08 21:52:09] ...optimizers/trijoint.py.151: total_norm: 13.620958 [I 2022-10-08 21:52:18] ...trap/engines/engine.py.224: train: epoch 21 | batch 410/2304 [I 2022-10-08 21:52:18] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:32 [I 2022-10-08 21:52:18] ...trap/engines/engine.py.229: process: 0.83395 | load: 0.00299 [I 2022-10-08 21:52:18] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-08 21:52:18] ...optimizers/trijoint.py.151: total_norm: 21.721542 [I 2022-10-08 21:52:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 420/2304 [I 2022-10-08 21:52:26] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:33 [I 2022-10-08 21:52:26] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00300 [I 2022-10-08 21:52:26] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-08 21:52:26] ...optimizers/trijoint.py.151: total_norm: 29.353289 [I 2022-10-08 21:52:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 430/2304 [I 2022-10-08 21:52:35] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:29 [I 2022-10-08 21:52:35] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00302 [I 2022-10-08 21:52:35] ...trap/engines/engine.py.230: loss: 0.16223 [I 2022-10-08 21:52:35] ...optimizers/trijoint.py.151: total_norm: 17.711466 [I 2022-10-08 21:52:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 440/2304 [I 2022-10-08 21:52:43] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:15 [I 2022-10-08 21:52:43] ...trap/engines/engine.py.229: process: 0.86491 | load: 0.00308 [I 2022-10-08 21:52:43] ...trap/engines/engine.py.230: loss: 0.24324 [I 2022-10-08 21:52:43] ...optimizers/trijoint.py.151: total_norm: 28.356159 [I 2022-10-08 21:52:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 450/2304 [I 2022-10-08 21:52:52] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:06 [I 2022-10-08 21:52:52] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00301 [I 2022-10-08 21:52:52] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-08 21:52:52] ...optimizers/trijoint.py.151: total_norm: 15.342834 [I 2022-10-08 21:53:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 460/2304 [I 2022-10-08 21:53:00] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:03 [I 2022-10-08 21:53:00] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00317 [I 2022-10-08 21:53:00] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-08 21:53:00] ...optimizers/trijoint.py.151: total_norm: 10.578042 [I 2022-10-08 21:53:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 470/2304 [I 2022-10-08 21:53:09] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:46 [I 2022-10-08 21:53:09] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00299 [I 2022-10-08 21:53:09] ...trap/engines/engine.py.230: loss: 0.17317 [I 2022-10-08 21:53:09] ...optimizers/trijoint.py.151: total_norm: 25.205759 [I 2022-10-08 21:53:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 480/2304 [I 2022-10-08 21:53:17] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:30 [I 2022-10-08 21:53:17] ...trap/engines/engine.py.229: process: 0.83416 | load: 0.00304 [I 2022-10-08 21:53:17] ...trap/engines/engine.py.230: loss: 0.17048 [I 2022-10-08 21:53:17] ...optimizers/trijoint.py.151: total_norm: 28.641418 [I 2022-10-08 21:53:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 490/2304 [I 2022-10-08 21:53:25] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:23 [I 2022-10-08 21:53:25] ...trap/engines/engine.py.229: process: 0.83989 | load: 0.00304 [I 2022-10-08 21:53:25] ...trap/engines/engine.py.230: loss: 0.19810 [I 2022-10-08 21:53:25] ...optimizers/trijoint.py.151: total_norm: 41.671017 [I 2022-10-08 21:53:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 500/2304 [I 2022-10-08 21:53:34] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:17 [I 2022-10-08 21:53:34] ...trap/engines/engine.py.229: process: 0.83669 | load: 0.00304 [I 2022-10-08 21:53:34] ...trap/engines/engine.py.230: loss: 0.11905 [I 2022-10-08 21:53:34] ...optimizers/trijoint.py.151: total_norm: 16.131657 [I 2022-10-08 21:53:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 510/2304 [I 2022-10-08 21:53:42] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:05 [I 2022-10-08 21:53:42] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00304 [I 2022-10-08 21:53:42] ...trap/engines/engine.py.230: loss: 0.22059 [I 2022-10-08 21:53:42] ...optimizers/trijoint.py.151: total_norm: 22.654715 [I 2022-10-08 21:53:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 520/2304 [I 2022-10-08 21:53:51] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:00 [I 2022-10-08 21:53:51] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00298 [I 2022-10-08 21:53:51] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-08 21:53:51] ...optimizers/trijoint.py.151: total_norm: 12.958502 [I 2022-10-08 21:53:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 530/2304 [I 2022-10-08 21:53:59] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:50 [I 2022-10-08 21:53:59] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00301 [I 2022-10-08 21:53:59] ...trap/engines/engine.py.230: loss: 0.17632 [I 2022-10-08 21:53:59] ...optimizers/trijoint.py.151: total_norm: 14.739450 [I 2022-10-08 21:54:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 540/2304 [I 2022-10-08 21:54:08] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:42 [I 2022-10-08 21:54:08] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00298 [I 2022-10-08 21:54:08] ...trap/engines/engine.py.230: loss: 0.13665 [I 2022-10-08 21:54:08] ...optimizers/trijoint.py.151: total_norm: 18.288124 [I 2022-10-08 21:54:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 550/2304 [I 2022-10-08 21:54:16] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:37 [I 2022-10-08 21:54:16] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00300 [I 2022-10-08 21:54:16] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-08 21:54:16] ...optimizers/trijoint.py.151: total_norm: 24.452454 [I 2022-10-08 21:54:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 560/2304 [I 2022-10-08 21:54:24] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:26 [I 2022-10-08 21:54:24] ...trap/engines/engine.py.229: process: 0.83598 | load: 0.00300 [I 2022-10-08 21:54:24] ...trap/engines/engine.py.230: loss: 0.17059 [I 2022-10-08 21:54:24] ...optimizers/trijoint.py.151: total_norm: 23.095253 [I 2022-10-08 21:54:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 570/2304 [I 2022-10-08 21:54:33] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:16 [I 2022-10-08 21:54:33] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00299 [I 2022-10-08 21:54:33] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-08 21:54:33] ...optimizers/trijoint.py.151: total_norm: 18.542633 [I 2022-10-08 21:54:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 580/2304 [I 2022-10-08 21:54:41] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:07 [I 2022-10-08 21:54:41] ...trap/engines/engine.py.229: process: 0.84341 | load: 0.00303 [I 2022-10-08 21:54:41] ...trap/engines/engine.py.230: loss: 0.23126 [I 2022-10-08 21:54:41] ...optimizers/trijoint.py.151: total_norm: 22.748867 [I 2022-10-08 21:55:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 590/2304 [I 2022-10-08 21:55:08] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 1:47:21 [I 2022-10-08 21:55:08] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00386 [I 2022-10-08 21:55:08] ...trap/engines/engine.py.230: loss: 0.23620 [I 2022-10-08 21:55:08] ...optimizers/trijoint.py.151: total_norm: 18.897554 [I 2022-10-08 21:55:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 600/2304 [I 2022-10-08 21:55:16] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:32:43 [I 2022-10-08 21:55:16] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00300 [I 2022-10-08 21:55:16] ...trap/engines/engine.py.230: loss: 0.20981 [I 2022-10-08 21:55:16] ...optimizers/trijoint.py.151: total_norm: 22.327051 [I 2022-10-08 21:55:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 610/2304 [I 2022-10-08 21:55:25] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:24:47 [I 2022-10-08 21:55:25] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00302 [I 2022-10-08 21:55:25] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-08 21:55:25] ...optimizers/trijoint.py.151: total_norm: 12.953052 [I 2022-10-08 21:55:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 620/2304 [I 2022-10-08 21:55:33] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:31 [I 2022-10-08 21:55:33] ...trap/engines/engine.py.229: process: 0.82673 | load: 0.00303 [I 2022-10-08 21:55:33] ...trap/engines/engine.py.230: loss: 0.22931 [I 2022-10-08 21:55:33] ...optimizers/trijoint.py.151: total_norm: 20.334255 [I 2022-10-08 21:55:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 630/2304 [I 2022-10-08 21:55:41] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:15 [I 2022-10-08 21:55:41] ...trap/engines/engine.py.229: process: 0.84227 | load: 0.00305 [I 2022-10-08 21:55:41] ...trap/engines/engine.py.230: loss: 0.21668 [I 2022-10-08 21:55:42] ...optimizers/trijoint.py.151: total_norm: 23.775112 [I 2022-10-08 21:55:50] ...trap/engines/engine.py.224: train: epoch 21 | batch 640/2304 [I 2022-10-08 21:55:50] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:13 [I 2022-10-08 21:55:50] ...trap/engines/engine.py.229: process: 0.84144 | load: 0.00305 [I 2022-10-08 21:55:50] ...trap/engines/engine.py.230: loss: 0.27227 [I 2022-10-08 21:55:50] ...optimizers/trijoint.py.151: total_norm: 15.801521 [I 2022-10-08 21:55:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 650/2304 [I 2022-10-08 21:55:58] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:14 [I 2022-10-08 21:55:58] ...trap/engines/engine.py.229: process: 0.84158 | load: 0.00302 [I 2022-10-08 21:55:58] ...trap/engines/engine.py.230: loss: 0.17672 [I 2022-10-08 21:55:58] ...optimizers/trijoint.py.151: total_norm: 16.317699 [I 2022-10-08 21:56:07] ...trap/engines/engine.py.224: train: epoch 21 | batch 660/2304 [I 2022-10-08 21:56:07] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:56 [I 2022-10-08 21:56:07] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00299 [I 2022-10-08 21:56:07] ...trap/engines/engine.py.230: loss: 0.18997 [I 2022-10-08 21:56:07] ...optimizers/trijoint.py.151: total_norm: 18.903803 [I 2022-10-08 21:56:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 670/2304 [I 2022-10-08 21:56:15] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:55 [I 2022-10-08 21:56:15] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00299 [I 2022-10-08 21:56:15] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-08 21:56:15] ...optimizers/trijoint.py.151: total_norm: 23.342571 [I 2022-10-08 21:56:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 680/2304 [I 2022-10-08 21:56:24] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:45 [I 2022-10-08 21:56:24] ...trap/engines/engine.py.229: process: 0.85072 | load: 0.00300 [I 2022-10-08 21:56:24] ...trap/engines/engine.py.230: loss: 0.16080 [I 2022-10-08 21:56:24] ...optimizers/trijoint.py.151: total_norm: 21.783478 [I 2022-10-08 21:56:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 690/2304 [I 2022-10-08 21:56:32] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:37 [I 2022-10-08 21:56:32] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00304 [I 2022-10-08 21:56:32] ...trap/engines/engine.py.230: loss: 0.13795 [I 2022-10-08 21:56:32] ...optimizers/trijoint.py.151: total_norm: 18.307726 [I 2022-10-08 21:56:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 700/2304 [I 2022-10-08 21:56:40] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:25 [I 2022-10-08 21:56:40] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00302 [I 2022-10-08 21:56:40] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-08 21:56:40] ...optimizers/trijoint.py.151: total_norm: 10.291972 [I 2022-10-08 21:56:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 710/2304 [I 2022-10-08 21:56:49] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:15 [I 2022-10-08 21:56:49] ...trap/engines/engine.py.229: process: 0.83743 | load: 0.00300 [I 2022-10-08 21:56:49] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-08 21:56:49] ...optimizers/trijoint.py.151: total_norm: 14.562401 [I 2022-10-08 21:56:57] ...trap/engines/engine.py.224: train: epoch 21 | batch 720/2304 [I 2022-10-08 21:56:57] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:14 [I 2022-10-08 21:56:57] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00302 [I 2022-10-08 21:56:57] ...trap/engines/engine.py.230: loss: 0.12303 [I 2022-10-08 21:56:57] ...optimizers/trijoint.py.151: total_norm: 13.464358 [I 2022-10-08 21:57:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 730/2304 [I 2022-10-08 21:57:06] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:59 [I 2022-10-08 21:57:06] ...trap/engines/engine.py.229: process: 0.84471 | load: 0.00301 [I 2022-10-08 21:57:06] ...trap/engines/engine.py.230: loss: 0.20306 [I 2022-10-08 21:57:06] ...optimizers/trijoint.py.151: total_norm: 34.282722 [I 2022-10-08 21:57:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 740/2304 [I 2022-10-08 21:57:14] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:42 [I 2022-10-08 21:57:14] ...trap/engines/engine.py.229: process: 0.83066 | load: 0.00301 [I 2022-10-08 21:57:14] ...trap/engines/engine.py.230: loss: 0.15277 [I 2022-10-08 21:57:14] ...optimizers/trijoint.py.151: total_norm: 14.109537 [I 2022-10-08 21:57:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 750/2304 [I 2022-10-08 21:57:22] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:46 [I 2022-10-08 21:57:22] ...trap/engines/engine.py.229: process: 0.83998 | load: 0.00300 [I 2022-10-08 21:57:22] ...trap/engines/engine.py.230: loss: 0.20696 [I 2022-10-08 21:57:22] ...optimizers/trijoint.py.151: total_norm: 22.257111 [I 2022-10-08 21:57:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 760/2304 [I 2022-10-08 21:57:31] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:25 [I 2022-10-08 21:57:31] ...trap/engines/engine.py.229: process: 0.83550 | load: 0.00299 [I 2022-10-08 21:57:31] ...trap/engines/engine.py.230: loss: 0.17073 [I 2022-10-08 21:57:31] ...optimizers/trijoint.py.151: total_norm: 20.171982 [I 2022-10-08 21:57:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 770/2304 [I 2022-10-08 21:57:39] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:34 [I 2022-10-08 21:57:39] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00300 [I 2022-10-08 21:57:39] ...trap/engines/engine.py.230: loss: 0.21720 [I 2022-10-08 21:57:39] ...optimizers/trijoint.py.151: total_norm: 48.693813 [I 2022-10-08 21:57:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 780/2304 [I 2022-10-08 21:57:48] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:20 [I 2022-10-08 21:57:48] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00300 [I 2022-10-08 21:57:48] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-08 21:57:48] ...optimizers/trijoint.py.151: total_norm: 12.812863 [I 2022-10-08 21:57:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 790/2304 [I 2022-10-08 21:57:56] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:17 [I 2022-10-08 21:57:56] ...trap/engines/engine.py.229: process: 0.84263 | load: 0.00301 [I 2022-10-08 21:57:56] ...trap/engines/engine.py.230: loss: 0.26967 [I 2022-10-08 21:57:56] ...optimizers/trijoint.py.151: total_norm: 14.353772 [I 2022-10-08 21:58:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 800/2304 [I 2022-10-08 21:58:05] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:08 [I 2022-10-08 21:58:05] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00302 [I 2022-10-08 21:58:05] ...trap/engines/engine.py.230: loss: 0.11808 [I 2022-10-08 21:58:05] ...optimizers/trijoint.py.151: total_norm: 23.914110 [I 2022-10-08 21:58:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 810/2304 [I 2022-10-08 21:58:13] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:20:58 [I 2022-10-08 21:58:13] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00299 [I 2022-10-08 21:58:13] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-08 21:58:13] ...optimizers/trijoint.py.151: total_norm: 27.878971 [I 2022-10-08 21:58:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 820/2304 [I 2022-10-08 21:58:22] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:20:50 [I 2022-10-08 21:58:22] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00301 [I 2022-10-08 21:58:22] ...trap/engines/engine.py.230: loss: 0.10581 [I 2022-10-08 21:58:22] ...optimizers/trijoint.py.151: total_norm: 11.953900 [I 2022-10-08 21:58:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 830/2304 [I 2022-10-08 21:58:30] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:20:36 [I 2022-10-08 21:58:30] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00297 [I 2022-10-08 21:58:30] ...trap/engines/engine.py.230: loss: 0.17241 [I 2022-10-08 21:58:30] ...optimizers/trijoint.py.151: total_norm: 21.888689 [I 2022-10-08 21:58:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 840/2304 [I 2022-10-08 21:58:38] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:27 [I 2022-10-08 21:58:38] ...trap/engines/engine.py.229: process: 0.82884 | load: 0.00308 [I 2022-10-08 21:58:38] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-08 21:58:38] ...optimizers/trijoint.py.151: total_norm: 17.156839 [I 2022-10-08 21:58:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 850/2304 [I 2022-10-08 21:58:47] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:20:19 [I 2022-10-08 21:58:47] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00312 [I 2022-10-08 21:58:47] ...trap/engines/engine.py.230: loss: 0.23180 [I 2022-10-08 21:58:47] ...optimizers/trijoint.py.151: total_norm: 27.587563 [I 2022-10-08 21:58:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 860/2304 [I 2022-10-08 21:58:55] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:15 [I 2022-10-08 21:58:55] ...trap/engines/engine.py.229: process: 0.83439 | load: 0.00311 [I 2022-10-08 21:58:55] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-08 21:58:55] ...optimizers/trijoint.py.151: total_norm: 17.576202 [I 2022-10-08 21:59:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 870/2304 [I 2022-10-08 21:59:04] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:20:11 [I 2022-10-08 21:59:04] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00309 [I 2022-10-08 21:59:04] ...trap/engines/engine.py.230: loss: 0.15892 [I 2022-10-08 21:59:04] ...optimizers/trijoint.py.151: total_norm: 15.554646 [I 2022-10-08 21:59:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 880/2304 [I 2022-10-08 21:59:12] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:19:59 [I 2022-10-08 21:59:12] ...trap/engines/engine.py.229: process: 0.84040 | load: 0.00312 [I 2022-10-08 21:59:12] ...trap/engines/engine.py.230: loss: 0.22305 [I 2022-10-08 21:59:12] ...optimizers/trijoint.py.151: total_norm: 17.979654 [I 2022-10-08 21:59:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 890/2304 [I 2022-10-08 21:59:21] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:19:46 [I 2022-10-08 21:59:21] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00309 [I 2022-10-08 21:59:21] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-08 21:59:21] ...optimizers/trijoint.py.151: total_norm: 10.070150 [I 2022-10-08 21:59:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 900/2304 [I 2022-10-08 21:59:29] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:41 [I 2022-10-08 21:59:29] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00311 [I 2022-10-08 21:59:29] ...trap/engines/engine.py.230: loss: 0.20897 [I 2022-10-08 21:59:29] ...optimizers/trijoint.py.151: total_norm: 56.756477 [I 2022-10-08 21:59:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 910/2304 [I 2022-10-08 21:59:37] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:19:33 [I 2022-10-08 21:59:37] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00315 [I 2022-10-08 21:59:37] ...trap/engines/engine.py.230: loss: 0.17973 [I 2022-10-08 21:59:37] ...optimizers/trijoint.py.151: total_norm: 13.646071 [I 2022-10-08 21:59:46] ...trap/engines/engine.py.224: train: epoch 21 | batch 920/2304 [I 2022-10-08 21:59:46] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:18 [I 2022-10-08 21:59:46] ...trap/engines/engine.py.229: process: 0.83425 | load: 0.00311 [I 2022-10-08 21:59:46] ...trap/engines/engine.py.230: loss: 0.17839 [I 2022-10-08 21:59:46] ...optimizers/trijoint.py.151: total_norm: 19.962828 [I 2022-10-08 21:59:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 930/2304 [I 2022-10-08 21:59:54] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:19:13 [I 2022-10-08 21:59:54] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00307 [I 2022-10-08 21:59:54] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-08 21:59:54] ...optimizers/trijoint.py.151: total_norm: 25.204906 [I 2022-10-08 22:00:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 940/2304 [I 2022-10-08 22:00:03] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:19:07 [I 2022-10-08 22:00:03] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00309 [I 2022-10-08 22:00:03] ...trap/engines/engine.py.230: loss: 0.14987 [I 2022-10-08 22:00:03] ...optimizers/trijoint.py.151: total_norm: 32.809204 [I 2022-10-08 22:00:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 950/2304 [I 2022-10-08 22:00:11] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:18:57 [I 2022-10-08 22:00:11] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00309 [I 2022-10-08 22:00:11] ...trap/engines/engine.py.230: loss: 0.24330 [I 2022-10-08 22:00:11] ...optimizers/trijoint.py.151: total_norm: 25.878092 [I 2022-10-08 22:00:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 960/2304 [I 2022-10-08 22:00:19] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:18:54 [I 2022-10-08 22:00:19] ...trap/engines/engine.py.229: process: 0.84955 | load: 0.00312 [I 2022-10-08 22:00:19] ...trap/engines/engine.py.230: loss: 0.16209 [I 2022-10-08 22:00:19] ...optimizers/trijoint.py.151: total_norm: 22.066275 [I 2022-10-08 22:00:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 970/2304 [I 2022-10-08 22:00:28] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:18:46 [I 2022-10-08 22:00:28] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00306 [I 2022-10-08 22:00:28] ...trap/engines/engine.py.230: loss: 0.19072 [I 2022-10-08 22:00:28] ...optimizers/trijoint.py.151: total_norm: 22.057432 [I 2022-10-08 22:00:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 980/2304 [I 2022-10-08 22:00:36] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:33 [I 2022-10-08 22:00:36] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00309 [I 2022-10-08 22:00:36] ...trap/engines/engine.py.230: loss: 0.18381 [I 2022-10-08 22:00:36] ...optimizers/trijoint.py.151: total_norm: 18.062191 [I 2022-10-08 22:00:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 990/2304 [I 2022-10-08 22:00:45] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:18:28 [I 2022-10-08 22:00:45] ...trap/engines/engine.py.229: process: 0.84277 | load: 0.00309 [I 2022-10-08 22:00:45] ...trap/engines/engine.py.230: loss: 0.15710 [I 2022-10-08 22:00:45] ...optimizers/trijoint.py.151: total_norm: 13.476643 [I 2022-10-08 22:00:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 1000/2304 [I 2022-10-08 22:00:53] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:18 [I 2022-10-08 22:00:53] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00307 [I 2022-10-08 22:00:53] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-08 22:00:53] ...optimizers/trijoint.py.151: total_norm: 23.322357 [I 2022-10-08 22:01:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 1010/2304 [I 2022-10-08 22:01:02] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:18:02 [I 2022-10-08 22:01:02] ...trap/engines/engine.py.229: process: 0.83224 | load: 0.00308 [I 2022-10-08 22:01:02] ...trap/engines/engine.py.230: loss: 0.25883 [I 2022-10-08 22:01:02] ...optimizers/trijoint.py.151: total_norm: 25.547894 [I 2022-10-08 22:01:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 1020/2304 [I 2022-10-08 22:01:10] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:18:00 [I 2022-10-08 22:01:10] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00312 [I 2022-10-08 22:01:10] ...trap/engines/engine.py.230: loss: 0.16106 [I 2022-10-08 22:01:10] ...optimizers/trijoint.py.151: total_norm: 21.217733 [I 2022-10-08 22:01:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 1030/2304 [I 2022-10-08 22:01:19] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:17:52 [I 2022-10-08 22:01:19] ...trap/engines/engine.py.229: process: 0.84120 | load: 0.00308 [I 2022-10-08 22:01:19] ...trap/engines/engine.py.230: loss: 0.20195 [I 2022-10-08 22:01:19] ...optimizers/trijoint.py.151: total_norm: 47.416428 [I 2022-10-08 22:01:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 1040/2304 [I 2022-10-08 22:01:27] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:17:36 [I 2022-10-08 22:01:27] ...trap/engines/engine.py.229: process: 0.82324 | load: 0.00317 [I 2022-10-08 22:01:27] ...trap/engines/engine.py.230: loss: 0.28453 [I 2022-10-08 22:01:27] ...optimizers/trijoint.py.151: total_norm: 35.275494 [I 2022-10-08 22:01:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 1050/2304 [I 2022-10-08 22:01:35] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:17:31 [I 2022-10-08 22:01:35] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00306 [I 2022-10-08 22:01:35] ...trap/engines/engine.py.230: loss: 0.23251 [I 2022-10-08 22:01:35] ...optimizers/trijoint.py.151: total_norm: 20.700708 [I 2022-10-08 22:01:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 1060/2304 [I 2022-10-08 22:01:44] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:21 [I 2022-10-08 22:01:44] ...trap/engines/engine.py.229: process: 0.82336 | load: 0.00307 [I 2022-10-08 22:01:44] ...trap/engines/engine.py.230: loss: 0.17314 [I 2022-10-08 22:01:44] ...optimizers/trijoint.py.151: total_norm: 28.163643 [I 2022-10-08 22:01:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 1070/2304 [I 2022-10-08 22:01:52] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:13 [I 2022-10-08 22:01:52] ...trap/engines/engine.py.229: process: 0.83973 | load: 0.00311 [I 2022-10-08 22:01:52] ...trap/engines/engine.py.230: loss: 0.17148 [I 2022-10-08 22:01:52] ...optimizers/trijoint.py.151: total_norm: 17.178541 [I 2022-10-08 22:02:01] ...trap/engines/engine.py.224: train: epoch 21 | batch 1080/2304 [I 2022-10-08 22:02:01] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:06 [I 2022-10-08 22:02:01] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00308 [I 2022-10-08 22:02:01] ...trap/engines/engine.py.230: loss: 0.21991 [I 2022-10-08 22:02:01] ...optimizers/trijoint.py.151: total_norm: 24.412136 [I 2022-10-08 22:02:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 1090/2304 [I 2022-10-08 22:02:09] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:16:59 [I 2022-10-08 22:02:09] ...trap/engines/engine.py.229: process: 0.82776 | load: 0.00306 [I 2022-10-08 22:02:09] ...trap/engines/engine.py.230: loss: 0.28911 [I 2022-10-08 22:02:09] ...optimizers/trijoint.py.151: total_norm: 22.999825 [I 2022-10-08 22:02:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1100/2304 [I 2022-10-08 22:02:17] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:16:53 [I 2022-10-08 22:02:17] ...trap/engines/engine.py.229: process: 0.85713 | load: 0.00308 [I 2022-10-08 22:02:17] ...trap/engines/engine.py.230: loss: 0.17317 [I 2022-10-08 22:02:17] ...optimizers/trijoint.py.151: total_norm: 23.857618 [I 2022-10-08 22:02:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 1110/2304 [I 2022-10-08 22:02:26] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:47 [I 2022-10-08 22:02:26] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00308 [I 2022-10-08 22:02:26] ...trap/engines/engine.py.230: loss: 0.16986 [I 2022-10-08 22:02:26] ...optimizers/trijoint.py.151: total_norm: 21.636497 [I 2022-10-08 22:02:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1120/2304 [I 2022-10-08 22:02:34] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:35 [I 2022-10-08 22:02:34] ...trap/engines/engine.py.229: process: 0.83651 | load: 0.00309 [I 2022-10-08 22:02:34] ...trap/engines/engine.py.230: loss: 0.22555 [I 2022-10-08 22:02:34] ...optimizers/trijoint.py.151: total_norm: 18.005186 [I 2022-10-08 22:02:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 1130/2304 [I 2022-10-08 22:02:43] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:28 [I 2022-10-08 22:02:43] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00309 [I 2022-10-08 22:02:43] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-08 22:02:43] ...optimizers/trijoint.py.151: total_norm: 13.141810 [I 2022-10-08 22:02:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 1140/2304 [I 2022-10-08 22:02:51] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:18 [I 2022-10-08 22:02:51] ...trap/engines/engine.py.229: process: 0.83400 | load: 0.00319 [I 2022-10-08 22:02:51] ...trap/engines/engine.py.230: loss: 0.21473 [I 2022-10-08 22:02:51] ...optimizers/trijoint.py.151: total_norm: 18.057426 [I 2022-10-08 22:03:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 1150/2304 [I 2022-10-08 22:03:00] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:06 [I 2022-10-08 22:03:00] ...trap/engines/engine.py.229: process: 0.83154 | load: 0.00309 [I 2022-10-08 22:03:00] ...trap/engines/engine.py.230: loss: 0.18024 [I 2022-10-08 22:03:00] ...optimizers/trijoint.py.151: total_norm: 9.016166 [I 2022-10-08 22:03:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 1160/2304 [I 2022-10-08 22:03:08] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:15:57 [I 2022-10-08 22:03:08] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00308 [I 2022-10-08 22:03:08] ...trap/engines/engine.py.230: loss: 0.17317 [I 2022-10-08 22:03:08] ...optimizers/trijoint.py.151: total_norm: 27.322260 [I 2022-10-08 22:03:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 1170/2304 [I 2022-10-08 22:03:16] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:15:52 [I 2022-10-08 22:03:16] ...trap/engines/engine.py.229: process: 0.83250 | load: 0.00306 [I 2022-10-08 22:03:16] ...trap/engines/engine.py.230: loss: 0.21616 [I 2022-10-08 22:03:16] ...optimizers/trijoint.py.151: total_norm: 28.151346 [I 2022-10-08 22:03:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 1180/2304 [I 2022-10-08 22:03:25] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:15:45 [I 2022-10-08 22:03:25] ...trap/engines/engine.py.229: process: 0.83537 | load: 0.00314 [I 2022-10-08 22:03:25] ...trap/engines/engine.py.230: loss: 0.23085 [I 2022-10-08 22:03:25] ...optimizers/trijoint.py.151: total_norm: 23.790545 [I 2022-10-08 22:03:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 1190/2304 [I 2022-10-08 22:03:33] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:38 [I 2022-10-08 22:03:33] ...trap/engines/engine.py.229: process: 0.84097 | load: 0.00307 [I 2022-10-08 22:03:33] ...trap/engines/engine.py.230: loss: 0.19984 [I 2022-10-08 22:03:33] ...optimizers/trijoint.py.151: total_norm: 19.675625 [I 2022-10-08 22:03:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 1200/2304 [I 2022-10-08 22:03:42] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:30 [I 2022-10-08 22:03:42] ...trap/engines/engine.py.229: process: 0.83766 | load: 0.00309 [I 2022-10-08 22:03:42] ...trap/engines/engine.py.230: loss: 0.14074 [I 2022-10-08 22:03:42] ...optimizers/trijoint.py.151: total_norm: 21.483509 [I 2022-10-08 22:03:50] ...trap/engines/engine.py.224: train: epoch 21 | batch 1210/2304 [I 2022-10-08 22:03:50] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:17 [I 2022-10-08 22:03:50] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00310 [I 2022-10-08 22:03:50] ...trap/engines/engine.py.230: loss: 0.17715 [I 2022-10-08 22:03:50] ...optimizers/trijoint.py.151: total_norm: 19.937309 [I 2022-10-08 22:03:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 1220/2304 [I 2022-10-08 22:03:59] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:14 [I 2022-10-08 22:03:59] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00304 [I 2022-10-08 22:03:59] ...trap/engines/engine.py.230: loss: 0.16057 [I 2022-10-08 22:03:59] ...optimizers/trijoint.py.151: total_norm: 26.452482 [I 2022-10-08 22:04:07] ...trap/engines/engine.py.224: train: epoch 21 | batch 1230/2304 [I 2022-10-08 22:04:07] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:14:59 [I 2022-10-08 22:04:07] ...trap/engines/engine.py.229: process: 0.83515 | load: 0.00310 [I 2022-10-08 22:04:07] ...trap/engines/engine.py.230: loss: 0.20314 [I 2022-10-08 22:04:07] ...optimizers/trijoint.py.151: total_norm: 19.455824 [I 2022-10-08 22:04:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 1240/2304 [I 2022-10-08 22:04:15] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:14:54 [I 2022-10-08 22:04:15] ...trap/engines/engine.py.229: process: 0.84379 | load: 0.00307 [I 2022-10-08 22:04:15] ...trap/engines/engine.py.230: loss: 0.26731 [I 2022-10-08 22:04:15] ...optimizers/trijoint.py.151: total_norm: 24.515928 [I 2022-10-08 22:04:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 1250/2304 [I 2022-10-08 22:04:24] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:49 [I 2022-10-08 22:04:24] ...trap/engines/engine.py.229: process: 0.84358 | load: 0.00311 [I 2022-10-08 22:04:24] ...trap/engines/engine.py.230: loss: 0.09892 [I 2022-10-08 22:04:24] ...optimizers/trijoint.py.151: total_norm: 10.717455 [I 2022-10-08 22:04:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 1260/2304 [I 2022-10-08 22:04:32] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:42 [I 2022-10-08 22:04:32] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00311 [I 2022-10-08 22:04:32] ...trap/engines/engine.py.230: loss: 0.17946 [I 2022-10-08 22:04:32] ...optimizers/trijoint.py.151: total_norm: 24.379684 [I 2022-10-08 22:04:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 1270/2304 [I 2022-10-08 22:04:41] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:27 [I 2022-10-08 22:04:41] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00314 [I 2022-10-08 22:04:41] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-08 22:04:41] ...optimizers/trijoint.py.151: total_norm: 16.062914 [I 2022-10-08 22:04:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 1280/2304 [I 2022-10-08 22:04:49] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:24 [I 2022-10-08 22:04:49] ...trap/engines/engine.py.229: process: 0.84818 | load: 0.00309 [I 2022-10-08 22:04:49] ...trap/engines/engine.py.230: loss: 0.19749 [I 2022-10-08 22:04:49] ...optimizers/trijoint.py.151: total_norm: 23.143631 [I 2022-10-08 22:04:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 1290/2304 [I 2022-10-08 22:04:58] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:11 [I 2022-10-08 22:04:58] ...trap/engines/engine.py.229: process: 0.84045 | load: 0.00309 [I 2022-10-08 22:04:58] ...trap/engines/engine.py.230: loss: 0.18534 [I 2022-10-08 22:04:58] ...optimizers/trijoint.py.151: total_norm: 14.414419 [I 2022-10-08 22:05:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 1300/2304 [I 2022-10-08 22:05:06] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:02 [I 2022-10-08 22:05:06] ...trap/engines/engine.py.229: process: 0.83913 | load: 0.00308 [I 2022-10-08 22:05:06] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-08 22:05:06] ...optimizers/trijoint.py.151: total_norm: 13.208005 [I 2022-10-08 22:05:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 1310/2304 [I 2022-10-08 22:05:14] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:13:57 [I 2022-10-08 22:05:14] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00300 [I 2022-10-08 22:05:14] ...trap/engines/engine.py.230: loss: 0.15297 [I 2022-10-08 22:05:14] ...optimizers/trijoint.py.151: total_norm: 15.322097 [I 2022-10-08 22:05:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1320/2304 [I 2022-10-08 22:05:23] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:13:45 [I 2022-10-08 22:05:23] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00300 [I 2022-10-08 22:05:23] ...trap/engines/engine.py.230: loss: 0.20784 [I 2022-10-08 22:05:23] ...optimizers/trijoint.py.151: total_norm: 25.233906 [I 2022-10-08 22:05:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 1330/2304 [I 2022-10-08 22:05:31] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:30 [I 2022-10-08 22:05:31] ...trap/engines/engine.py.229: process: 0.83034 | load: 0.00299 [I 2022-10-08 22:05:31] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-08 22:05:31] ...optimizers/trijoint.py.151: total_norm: 18.688307 [I 2022-10-08 22:05:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 1340/2304 [I 2022-10-08 22:05:40] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:27 [I 2022-10-08 22:05:40] ...trap/engines/engine.py.229: process: 0.83083 | load: 0.00304 [I 2022-10-08 22:05:40] ...trap/engines/engine.py.230: loss: 0.17929 [I 2022-10-08 22:05:40] ...optimizers/trijoint.py.151: total_norm: 19.096846 [I 2022-10-08 22:05:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 1350/2304 [I 2022-10-08 22:05:48] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:20 [I 2022-10-08 22:05:48] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00299 [I 2022-10-08 22:05:48] ...trap/engines/engine.py.230: loss: 0.13107 [I 2022-10-08 22:05:48] ...optimizers/trijoint.py.151: total_norm: 30.086052 [I 2022-10-08 22:05:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 1360/2304 [I 2022-10-08 22:05:56] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:17 [I 2022-10-08 22:05:56] ...trap/engines/engine.py.229: process: 0.86894 | load: 0.00299 [I 2022-10-08 22:05:56] ...trap/engines/engine.py.230: loss: 0.16193 [I 2022-10-08 22:05:56] ...optimizers/trijoint.py.151: total_norm: 25.318735 [I 2022-10-08 22:06:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 1370/2304 [I 2022-10-08 22:06:05] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:02 [I 2022-10-08 22:06:05] ...trap/engines/engine.py.229: process: 0.83172 | load: 0.00300 [I 2022-10-08 22:06:05] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-08 22:06:05] ...optimizers/trijoint.py.151: total_norm: 19.741533 [I 2022-10-08 22:06:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 1380/2304 [I 2022-10-08 22:06:13] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:04 [I 2022-10-08 22:06:13] ...trap/engines/engine.py.229: process: 0.87053 | load: 0.00304 [I 2022-10-08 22:06:13] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-08 22:06:13] ...optimizers/trijoint.py.151: total_norm: 30.291323 [I 2022-10-08 22:06:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 1390/2304 [I 2022-10-08 22:06:22] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:48 [I 2022-10-08 22:06:22] ...trap/engines/engine.py.229: process: 0.83254 | load: 0.00301 [I 2022-10-08 22:06:22] ...trap/engines/engine.py.230: loss: 0.15700 [I 2022-10-08 22:06:22] ...optimizers/trijoint.py.151: total_norm: 24.590673 [I 2022-10-08 22:06:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 1400/2304 [I 2022-10-08 22:06:30] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:12:44 [I 2022-10-08 22:06:30] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00299 [I 2022-10-08 22:06:30] ...trap/engines/engine.py.230: loss: 0.16008 [I 2022-10-08 22:06:30] ...optimizers/trijoint.py.151: total_norm: 27.790314 [I 2022-10-08 22:06:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 1410/2304 [I 2022-10-08 22:06:39] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:31 [I 2022-10-08 22:06:39] ...trap/engines/engine.py.229: process: 0.84217 | load: 0.00300 [I 2022-10-08 22:06:39] ...trap/engines/engine.py.230: loss: 0.18336 [I 2022-10-08 22:06:39] ...optimizers/trijoint.py.151: total_norm: 30.233397 [I 2022-10-08 22:06:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 1420/2304 [I 2022-10-08 22:06:47] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:25 [I 2022-10-08 22:06:47] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00301 [I 2022-10-08 22:06:47] ...trap/engines/engine.py.230: loss: 0.24985 [I 2022-10-08 22:06:47] ...optimizers/trijoint.py.151: total_norm: 24.446573 [I 2022-10-08 22:06:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 1430/2304 [I 2022-10-08 22:06:56] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:15 [I 2022-10-08 22:06:56] ...trap/engines/engine.py.229: process: 0.82991 | load: 0.00310 [I 2022-10-08 22:06:56] ...trap/engines/engine.py.230: loss: 0.19468 [I 2022-10-08 22:06:56] ...optimizers/trijoint.py.151: total_norm: 38.617176 [I 2022-10-08 22:07:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 1440/2304 [I 2022-10-08 22:07:04] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:05 [I 2022-10-08 22:07:04] ...trap/engines/engine.py.229: process: 0.83176 | load: 0.00305 [I 2022-10-08 22:07:04] ...trap/engines/engine.py.230: loss: 0.16062 [I 2022-10-08 22:07:04] ...optimizers/trijoint.py.151: total_norm: 10.458217 [I 2022-10-08 22:07:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 1450/2304 [I 2022-10-08 22:07:12] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:11:57 [I 2022-10-08 22:07:12] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00300 [I 2022-10-08 22:07:12] ...trap/engines/engine.py.230: loss: 0.13961 [I 2022-10-08 22:07:12] ...optimizers/trijoint.py.151: total_norm: 15.560844 [I 2022-10-08 22:07:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 1460/2304 [I 2022-10-08 22:07:21] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:50 [I 2022-10-08 22:07:21] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00300 [I 2022-10-08 22:07:21] ...trap/engines/engine.py.230: loss: 0.20361 [I 2022-10-08 22:07:21] ...optimizers/trijoint.py.151: total_norm: 26.147821 [I 2022-10-08 22:07:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 1470/2304 [I 2022-10-08 22:07:29] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:38 [I 2022-10-08 22:07:29] ...trap/engines/engine.py.229: process: 0.82215 | load: 0.00303 [I 2022-10-08 22:07:29] ...trap/engines/engine.py.230: loss: 0.16649 [I 2022-10-08 22:07:29] ...optimizers/trijoint.py.151: total_norm: 11.663450 [I 2022-10-08 22:07:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 1480/2304 [I 2022-10-08 22:07:38] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:28 [I 2022-10-08 22:07:38] ...trap/engines/engine.py.229: process: 0.83280 | load: 0.00301 [I 2022-10-08 22:07:38] ...trap/engines/engine.py.230: loss: 0.18286 [I 2022-10-08 22:07:38] ...optimizers/trijoint.py.151: total_norm: 21.782637 [I 2022-10-08 22:07:46] ...trap/engines/engine.py.224: train: epoch 21 | batch 1490/2304 [I 2022-10-08 22:07:46] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:24 [I 2022-10-08 22:07:46] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00304 [I 2022-10-08 22:07:46] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-08 22:07:46] ...optimizers/trijoint.py.151: total_norm: 48.861755 [I 2022-10-08 22:07:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 1500/2304 [I 2022-10-08 22:07:54] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:11 [I 2022-10-08 22:07:54] ...trap/engines/engine.py.229: process: 0.82949 | load: 0.00298 [I 2022-10-08 22:07:54] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-08 22:07:54] ...optimizers/trijoint.py.151: total_norm: 21.064749 [I 2022-10-08 22:08:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 1510/2304 [I 2022-10-08 22:08:03] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:06 [I 2022-10-08 22:08:03] ...trap/engines/engine.py.229: process: 0.82978 | load: 0.00302 [I 2022-10-08 22:08:03] ...trap/engines/engine.py.230: loss: 0.19550 [I 2022-10-08 22:08:03] ...optimizers/trijoint.py.151: total_norm: 26.610062 [I 2022-10-08 22:08:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 1520/2304 [I 2022-10-08 22:08:11] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:57 [I 2022-10-08 22:08:11] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00301 [I 2022-10-08 22:08:11] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-08 22:08:11] ...optimizers/trijoint.py.151: total_norm: 23.466164 [I 2022-10-08 22:08:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 1530/2304 [I 2022-10-08 22:08:20] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:50 [I 2022-10-08 22:08:20] ...trap/engines/engine.py.229: process: 0.83876 | load: 0.00303 [I 2022-10-08 22:08:20] ...trap/engines/engine.py.230: loss: 0.16242 [I 2022-10-08 22:08:20] ...optimizers/trijoint.py.151: total_norm: 18.883924 [I 2022-10-08 22:08:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 1540/2304 [I 2022-10-08 22:08:28] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:43 [I 2022-10-08 22:08:28] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00302 [I 2022-10-08 22:08:28] ...trap/engines/engine.py.230: loss: 0.14483 [I 2022-10-08 22:08:28] ...optimizers/trijoint.py.151: total_norm: 13.274589 [I 2022-10-08 22:08:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 1550/2304 [I 2022-10-08 22:08:37] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:29 [I 2022-10-08 22:08:37] ...trap/engines/engine.py.229: process: 0.82781 | load: 0.00303 [I 2022-10-08 22:08:37] ...trap/engines/engine.py.230: loss: 0.14509 [I 2022-10-08 22:08:37] ...optimizers/trijoint.py.151: total_norm: 18.691383 [I 2022-10-08 22:08:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 1560/2304 [I 2022-10-08 22:08:45] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:25 [I 2022-10-08 22:08:45] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00301 [I 2022-10-08 22:08:45] ...trap/engines/engine.py.230: loss: 0.14869 [I 2022-10-08 22:08:45] ...optimizers/trijoint.py.151: total_norm: 16.621782 [I 2022-10-08 22:08:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 1570/2304 [I 2022-10-08 22:08:53] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:17 [I 2022-10-08 22:08:53] ...trap/engines/engine.py.229: process: 0.83365 | load: 0.00301 [I 2022-10-08 22:08:53] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-08 22:08:53] ...optimizers/trijoint.py.151: total_norm: 33.861954 [I 2022-10-08 22:09:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 1580/2304 [I 2022-10-08 22:09:02] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:08 [I 2022-10-08 22:09:02] ...trap/engines/engine.py.229: process: 0.82488 | load: 0.00304 [I 2022-10-08 22:09:02] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-08 22:09:02] ...optimizers/trijoint.py.151: total_norm: 21.870949 [I 2022-10-08 22:09:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 1590/2304 [I 2022-10-08 22:09:10] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:09:59 [I 2022-10-08 22:09:10] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00306 [I 2022-10-08 22:09:10] ...trap/engines/engine.py.230: loss: 0.14446 [I 2022-10-08 22:09:10] ...optimizers/trijoint.py.151: total_norm: 19.814444 [I 2022-10-08 22:09:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 1600/2304 [I 2022-10-08 22:09:19] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:49 [I 2022-10-08 22:09:19] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00302 [I 2022-10-08 22:09:19] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-08 22:09:19] ...optimizers/trijoint.py.151: total_norm: 23.117626 [I 2022-10-08 22:09:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 1610/2304 [I 2022-10-08 22:09:27] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:43 [I 2022-10-08 22:09:27] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00301 [I 2022-10-08 22:09:27] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-08 22:09:27] ...optimizers/trijoint.py.151: total_norm: 16.239422 [I 2022-10-08 22:09:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 1620/2304 [I 2022-10-08 22:09:35] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:34 [I 2022-10-08 22:09:35] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00305 [I 2022-10-08 22:09:35] ...trap/engines/engine.py.230: loss: 0.18557 [I 2022-10-08 22:09:35] ...optimizers/trijoint.py.151: total_norm: 12.762243 [I 2022-10-08 22:09:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 1630/2304 [I 2022-10-08 22:09:44] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:27 [I 2022-10-08 22:09:44] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00301 [I 2022-10-08 22:09:44] ...trap/engines/engine.py.230: loss: 0.15541 [I 2022-10-08 22:09:44] ...optimizers/trijoint.py.151: total_norm: 11.221841 [I 2022-10-08 22:09:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 1640/2304 [I 2022-10-08 22:09:52] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:18 [I 2022-10-08 22:09:52] ...trap/engines/engine.py.229: process: 0.83590 | load: 0.00309 [I 2022-10-08 22:09:52] ...trap/engines/engine.py.230: loss: 0.22002 [I 2022-10-08 22:09:52] ...optimizers/trijoint.py.151: total_norm: 16.236038 [I 2022-10-08 22:10:01] ...trap/engines/engine.py.224: train: epoch 21 | batch 1650/2304 [I 2022-10-08 22:10:01] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:06 [I 2022-10-08 22:10:01] ...trap/engines/engine.py.229: process: 0.82571 | load: 0.00301 [I 2022-10-08 22:10:01] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-08 22:10:01] ...optimizers/trijoint.py.151: total_norm: 14.083454 [I 2022-10-08 22:10:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 1660/2304 [I 2022-10-08 22:10:09] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:00 [I 2022-10-08 22:10:09] ...trap/engines/engine.py.229: process: 0.85705 | load: 0.00300 [I 2022-10-08 22:10:09] ...trap/engines/engine.py.230: loss: 0.15789 [I 2022-10-08 22:10:09] ...optimizers/trijoint.py.151: total_norm: 23.357492 [I 2022-10-08 22:10:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1670/2304 [I 2022-10-08 22:10:17] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:08:53 [I 2022-10-08 22:10:17] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00304 [I 2022-10-08 22:10:17] ...trap/engines/engine.py.230: loss: 0.18937 [I 2022-10-08 22:10:17] ...optimizers/trijoint.py.151: total_norm: 24.718609 [I 2022-10-08 22:10:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 1680/2304 [I 2022-10-08 22:10:26] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:43 [I 2022-10-08 22:10:26] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00302 [I 2022-10-08 22:10:26] ...trap/engines/engine.py.230: loss: 0.18037 [I 2022-10-08 22:10:26] ...optimizers/trijoint.py.151: total_norm: 20.894159 [I 2022-10-08 22:10:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1690/2304 [I 2022-10-08 22:10:34] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:34 [I 2022-10-08 22:10:34] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00301 [I 2022-10-08 22:10:34] ...trap/engines/engine.py.230: loss: 0.14785 [I 2022-10-08 22:10:34] ...optimizers/trijoint.py.151: total_norm: 22.713463 [I 2022-10-08 22:10:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 1700/2304 [I 2022-10-08 22:10:43] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:26 [I 2022-10-08 22:10:43] ...trap/engines/engine.py.229: process: 0.83379 | load: 0.00300 [I 2022-10-08 22:10:43] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-08 22:10:43] ...optimizers/trijoint.py.151: total_norm: 10.337750 [I 2022-10-08 22:10:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 1710/2304 [I 2022-10-08 22:10:51] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:18 [I 2022-10-08 22:10:51] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00299 [I 2022-10-08 22:10:51] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-08 22:10:51] ...optimizers/trijoint.py.151: total_norm: 14.132849 [I 2022-10-08 22:10:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 1720/2304 [I 2022-10-08 22:10:59] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:08 [I 2022-10-08 22:10:59] ...trap/engines/engine.py.229: process: 0.82678 | load: 0.00303 [I 2022-10-08 22:10:59] ...trap/engines/engine.py.230: loss: 0.42467 [I 2022-10-08 22:10:59] ...optimizers/trijoint.py.151: total_norm: 21.435965 [I 2022-10-08 22:11:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 1730/2304 [I 2022-10-08 22:11:08] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:02 [I 2022-10-08 22:11:08] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00300 [I 2022-10-08 22:11:08] ...trap/engines/engine.py.230: loss: 0.15688 [I 2022-10-08 22:11:08] ...optimizers/trijoint.py.151: total_norm: 16.527918 [I 2022-10-08 22:11:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 1740/2304 [I 2022-10-08 22:11:16] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:07:54 [I 2022-10-08 22:11:16] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00304 [I 2022-10-08 22:11:16] ...trap/engines/engine.py.230: loss: 0.18456 [I 2022-10-08 22:11:16] ...optimizers/trijoint.py.151: total_norm: 21.921938 [I 2022-10-08 22:11:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 1750/2304 [I 2022-10-08 22:11:25] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:43 [I 2022-10-08 22:11:25] ...trap/engines/engine.py.229: process: 0.83118 | load: 0.00298 [I 2022-10-08 22:11:25] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-08 22:11:25] ...optimizers/trijoint.py.151: total_norm: 15.995891 [I 2022-10-08 22:11:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 1760/2304 [I 2022-10-08 22:11:33] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:07:35 [I 2022-10-08 22:11:33] ...trap/engines/engine.py.229: process: 0.83062 | load: 0.00298 [I 2022-10-08 22:11:33] ...trap/engines/engine.py.230: loss: 0.22814 [I 2022-10-08 22:11:33] ...optimizers/trijoint.py.151: total_norm: 40.573006 [I 2022-10-08 22:11:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 1770/2304 [I 2022-10-08 22:11:42] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:27 [I 2022-10-08 22:11:42] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00302 [I 2022-10-08 22:11:42] ...trap/engines/engine.py.230: loss: 0.22261 [I 2022-10-08 22:11:42] ...optimizers/trijoint.py.151: total_norm: 30.903204 [I 2022-10-08 22:11:50] ...trap/engines/engine.py.224: train: epoch 21 | batch 1780/2304 [I 2022-10-08 22:11:50] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:17 [I 2022-10-08 22:11:50] ...trap/engines/engine.py.229: process: 0.83239 | load: 0.00299 [I 2022-10-08 22:11:50] ...trap/engines/engine.py.230: loss: 0.14114 [I 2022-10-08 22:11:50] ...optimizers/trijoint.py.151: total_norm: 15.512486 [I 2022-10-08 22:11:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 1790/2304 [I 2022-10-08 22:11:58] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:09 [I 2022-10-08 22:11:58] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00301 [I 2022-10-08 22:11:58] ...trap/engines/engine.py.230: loss: 0.26756 [I 2022-10-08 22:11:58] ...optimizers/trijoint.py.151: total_norm: 14.411522 [I 2022-10-08 22:12:07] ...trap/engines/engine.py.224: train: epoch 21 | batch 1800/2304 [I 2022-10-08 22:12:07] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:02 [I 2022-10-08 22:12:07] ...trap/engines/engine.py.229: process: 0.83998 | load: 0.00299 [I 2022-10-08 22:12:07] ...trap/engines/engine.py.230: loss: 0.15278 [I 2022-10-08 22:12:07] ...optimizers/trijoint.py.151: total_norm: 28.448820 [I 2022-10-08 22:12:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 1810/2304 [I 2022-10-08 22:12:15] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:06:55 [I 2022-10-08 22:12:15] ...trap/engines/engine.py.229: process: 0.84016 | load: 0.00298 [I 2022-10-08 22:12:15] ...trap/engines/engine.py.230: loss: 0.18104 [I 2022-10-08 22:12:15] ...optimizers/trijoint.py.151: total_norm: 18.822958 [I 2022-10-08 22:12:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1820/2304 [I 2022-10-08 22:12:23] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:06:44 [I 2022-10-08 22:12:23] ...trap/engines/engine.py.229: process: 0.83209 | load: 0.00299 [I 2022-10-08 22:12:23] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-08 22:12:23] ...optimizers/trijoint.py.151: total_norm: 33.198345 [I 2022-10-08 22:12:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 1830/2304 [I 2022-10-08 22:12:32] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:06:36 [I 2022-10-08 22:12:32] ...trap/engines/engine.py.229: process: 0.83583 | load: 0.00298 [I 2022-10-08 22:12:32] ...trap/engines/engine.py.230: loss: 0.21389 [I 2022-10-08 22:12:32] ...optimizers/trijoint.py.151: total_norm: 24.553566 [I 2022-10-08 22:12:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 1840/2304 [I 2022-10-08 22:12:40] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:28 [I 2022-10-08 22:12:40] ...trap/engines/engine.py.229: process: 0.83422 | load: 0.00298 [I 2022-10-08 22:12:40] ...trap/engines/engine.py.230: loss: 0.21211 [I 2022-10-08 22:12:40] ...optimizers/trijoint.py.151: total_norm: 16.355043 [I 2022-10-08 22:12:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 1850/2304 [I 2022-10-08 22:12:49] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:06:19 [I 2022-10-08 22:12:49] ...trap/engines/engine.py.229: process: 0.83862 | load: 0.00300 [I 2022-10-08 22:12:49] ...trap/engines/engine.py.230: loss: 0.28575 [I 2022-10-08 22:12:49] ...optimizers/trijoint.py.151: total_norm: 15.745380 [I 2022-10-08 22:12:57] ...trap/engines/engine.py.224: train: epoch 21 | batch 1860/2304 [I 2022-10-08 22:12:57] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:11 [I 2022-10-08 22:12:57] ...trap/engines/engine.py.229: process: 0.83023 | load: 0.00299 [I 2022-10-08 22:12:57] ...trap/engines/engine.py.230: loss: 0.16322 [I 2022-10-08 22:12:57] ...optimizers/trijoint.py.151: total_norm: 14.469856 [I 2022-10-08 22:13:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 1870/2304 [I 2022-10-08 22:13:05] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:05 [I 2022-10-08 22:13:05] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00304 [I 2022-10-08 22:13:05] ...trap/engines/engine.py.230: loss: 0.21985 [I 2022-10-08 22:13:05] ...optimizers/trijoint.py.151: total_norm: 22.659420 [I 2022-10-08 22:13:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 1880/2304 [I 2022-10-08 22:13:14] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:05:53 [I 2022-10-08 22:13:14] ...trap/engines/engine.py.229: process: 0.82554 | load: 0.00302 [I 2022-10-08 22:13:14] ...trap/engines/engine.py.230: loss: 0.12436 [I 2022-10-08 22:13:14] ...optimizers/trijoint.py.151: total_norm: 14.467415 [I 2022-10-08 22:13:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 1890/2304 [I 2022-10-08 22:13:22] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:05:46 [I 2022-10-08 22:13:22] ...trap/engines/engine.py.229: process: 0.81842 | load: 0.00299 [I 2022-10-08 22:13:22] ...trap/engines/engine.py.230: loss: 0.24276 [I 2022-10-08 22:13:22] ...optimizers/trijoint.py.151: total_norm: 29.085606 [I 2022-10-08 22:13:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 1900/2304 [I 2022-10-08 22:13:31] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:05:39 [I 2022-10-08 22:13:31] ...trap/engines/engine.py.229: process: 0.83497 | load: 0.00300 [I 2022-10-08 22:13:31] ...trap/engines/engine.py.230: loss: 0.12465 [I 2022-10-08 22:13:31] ...optimizers/trijoint.py.151: total_norm: 12.603619 [I 2022-10-08 22:13:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 1910/2304 [I 2022-10-08 22:13:39] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:05:30 [I 2022-10-08 22:13:39] ...trap/engines/engine.py.229: process: 0.83485 | load: 0.00300 [I 2022-10-08 22:13:39] ...trap/engines/engine.py.230: loss: 0.23728 [I 2022-10-08 22:13:39] ...optimizers/trijoint.py.151: total_norm: 28.823792 [I 2022-10-08 22:13:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 1920/2304 [I 2022-10-08 22:13:47] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:22 [I 2022-10-08 22:13:47] ...trap/engines/engine.py.229: process: 0.83329 | load: 0.00300 [I 2022-10-08 22:13:47] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-08 22:13:47] ...optimizers/trijoint.py.151: total_norm: 21.558533 [I 2022-10-08 22:13:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 1930/2304 [I 2022-10-08 22:13:56] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:11 [I 2022-10-08 22:13:56] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00303 [I 2022-10-08 22:13:56] ...trap/engines/engine.py.230: loss: 0.21424 [I 2022-10-08 22:13:56] ...optimizers/trijoint.py.151: total_norm: 41.596867 [I 2022-10-08 22:14:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 1940/2304 [I 2022-10-08 22:14:04] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:06 [I 2022-10-08 22:14:04] ...trap/engines/engine.py.229: process: 0.85622 | load: 0.00301 [I 2022-10-08 22:14:04] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-08 22:14:04] ...optimizers/trijoint.py.151: total_norm: 17.539202 [I 2022-10-08 22:14:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 1950/2304 [I 2022-10-08 22:14:13] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:04:56 [I 2022-10-08 22:14:13] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00300 [I 2022-10-08 22:14:13] ...trap/engines/engine.py.230: loss: 0.22917 [I 2022-10-08 22:14:13] ...optimizers/trijoint.py.151: total_norm: 26.157623 [I 2022-10-08 22:14:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 1960/2304 [I 2022-10-08 22:14:21] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:04:48 [I 2022-10-08 22:14:21] ...trap/engines/engine.py.229: process: 0.83241 | load: 0.00304 [I 2022-10-08 22:14:21] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-08 22:14:21] ...optimizers/trijoint.py.151: total_norm: 21.058466 [I 2022-10-08 22:14:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 1970/2304 [I 2022-10-08 22:14:29] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:39 [I 2022-10-08 22:14:29] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00302 [I 2022-10-08 22:14:29] ...trap/engines/engine.py.230: loss: 0.13009 [I 2022-10-08 22:14:29] ...optimizers/trijoint.py.151: total_norm: 16.344492 [I 2022-10-08 22:14:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 1980/2304 [I 2022-10-08 22:14:38] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:30 [I 2022-10-08 22:14:38] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00300 [I 2022-10-08 22:14:38] ...trap/engines/engine.py.230: loss: 0.13729 [I 2022-10-08 22:14:38] ...optimizers/trijoint.py.151: total_norm: 14.033529 [I 2022-10-08 22:14:46] ...trap/engines/engine.py.224: train: epoch 21 | batch 1990/2304 [I 2022-10-08 22:14:46] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:23 [I 2022-10-08 22:14:46] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00299 [I 2022-10-08 22:14:46] ...trap/engines/engine.py.230: loss: 0.24261 [I 2022-10-08 22:14:46] ...optimizers/trijoint.py.151: total_norm: 20.417686 [I 2022-10-08 22:14:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 2000/2304 [I 2022-10-08 22:14:55] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:14 [I 2022-10-08 22:14:55] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00298 [I 2022-10-08 22:14:55] ...trap/engines/engine.py.230: loss: 0.15175 [I 2022-10-08 22:14:55] ...optimizers/trijoint.py.151: total_norm: 29.028133 [I 2022-10-08 22:15:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 2010/2304 [I 2022-10-08 22:15:03] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:06 [I 2022-10-08 22:15:03] ...trap/engines/engine.py.229: process: 0.83441 | load: 0.00300 [I 2022-10-08 22:15:03] ...trap/engines/engine.py.230: loss: 0.23889 [I 2022-10-08 22:15:03] ...optimizers/trijoint.py.151: total_norm: 22.991188 [I 2022-10-08 22:15:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 2020/2304 [I 2022-10-08 22:15:11] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:03:58 [I 2022-10-08 22:15:11] ...trap/engines/engine.py.229: process: 0.83637 | load: 0.00304 [I 2022-10-08 22:15:11] ...trap/engines/engine.py.230: loss: 0.20268 [I 2022-10-08 22:15:11] ...optimizers/trijoint.py.151: total_norm: 22.061373 [I 2022-10-08 22:15:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 2030/2304 [I 2022-10-08 22:15:20] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:03:49 [I 2022-10-08 22:15:20] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00300 [I 2022-10-08 22:15:20] ...trap/engines/engine.py.230: loss: 0.15522 [I 2022-10-08 22:15:20] ...optimizers/trijoint.py.151: total_norm: 14.372794 [I 2022-10-08 22:15:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 2040/2304 [I 2022-10-08 22:15:28] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:41 [I 2022-10-08 22:15:28] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00305 [I 2022-10-08 22:15:28] ...trap/engines/engine.py.230: loss: 0.18096 [I 2022-10-08 22:15:28] ...optimizers/trijoint.py.151: total_norm: 24.879522 [I 2022-10-08 22:15:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 2050/2304 [I 2022-10-08 22:15:36] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:31 [I 2022-10-08 22:15:36] ...trap/engines/engine.py.229: process: 0.83198 | load: 0.00300 [I 2022-10-08 22:15:36] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-08 22:15:36] ...optimizers/trijoint.py.151: total_norm: 25.609892 [I 2022-10-08 22:15:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 2060/2304 [I 2022-10-08 22:15:45] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:23 [I 2022-10-08 22:15:45] ...trap/engines/engine.py.229: process: 0.82793 | load: 0.00298 [I 2022-10-08 22:15:45] ...trap/engines/engine.py.230: loss: 0.23923 [I 2022-10-08 22:15:45] ...optimizers/trijoint.py.151: total_norm: 18.498957 [I 2022-10-08 22:15:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 2070/2304 [I 2022-10-08 22:15:53] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:15 [I 2022-10-08 22:15:53] ...trap/engines/engine.py.229: process: 0.83725 | load: 0.00297 [I 2022-10-08 22:15:53] ...trap/engines/engine.py.230: loss: 0.14020 [I 2022-10-08 22:15:53] ...optimizers/trijoint.py.151: total_norm: 18.341080 [I 2022-10-08 22:16:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 2080/2304 [I 2022-10-08 22:16:02] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:08 [I 2022-10-08 22:16:02] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00298 [I 2022-10-08 22:16:02] ...trap/engines/engine.py.230: loss: 0.27767 [I 2022-10-08 22:16:02] ...optimizers/trijoint.py.151: total_norm: 32.092651 [I 2022-10-08 22:16:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 2090/2304 [I 2022-10-08 22:16:10] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:02:59 [I 2022-10-08 22:16:10] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00301 [I 2022-10-08 22:16:10] ...trap/engines/engine.py.230: loss: 0.16304 [I 2022-10-08 22:16:10] ...optimizers/trijoint.py.151: total_norm: 11.919920 [I 2022-10-08 22:16:18] ...trap/engines/engine.py.224: train: epoch 21 | batch 2100/2304 [I 2022-10-08 22:16:18] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:02:51 [I 2022-10-08 22:16:18] ...trap/engines/engine.py.229: process: 0.83914 | load: 0.00300 [I 2022-10-08 22:16:18] ...trap/engines/engine.py.230: loss: 0.23058 [I 2022-10-08 22:16:18] ...optimizers/trijoint.py.151: total_norm: 30.205315 [I 2022-10-08 22:16:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 2110/2304 [I 2022-10-08 22:16:27] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:41 [I 2022-10-08 22:16:27] ...trap/engines/engine.py.229: process: 0.83087 | load: 0.00299 [I 2022-10-08 22:16:27] ...trap/engines/engine.py.230: loss: 0.19394 [I 2022-10-08 22:16:27] ...optimizers/trijoint.py.151: total_norm: 29.410736 [I 2022-10-08 22:16:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 2120/2304 [I 2022-10-08 22:16:35] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:33 [I 2022-10-08 22:16:35] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00295 [I 2022-10-08 22:16:35] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-08 22:16:35] ...optimizers/trijoint.py.151: total_norm: 9.761244 [I 2022-10-08 22:16:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 2130/2304 [I 2022-10-08 22:16:44] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:25 [I 2022-10-08 22:16:44] ...trap/engines/engine.py.229: process: 0.83788 | load: 0.00301 [I 2022-10-08 22:16:44] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-08 22:16:44] ...optimizers/trijoint.py.151: total_norm: 12.238261 [I 2022-10-08 22:16:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 2140/2304 [I 2022-10-08 22:16:52] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:17 [I 2022-10-08 22:16:52] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00298 [I 2022-10-08 22:16:52] ...trap/engines/engine.py.230: loss: 0.21766 [I 2022-10-08 22:16:52] ...optimizers/trijoint.py.151: total_norm: 20.830904 [I 2022-10-08 22:17:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 2150/2304 [I 2022-10-08 22:17:00] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:09 [I 2022-10-08 22:17:00] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00298 [I 2022-10-08 22:17:00] ...trap/engines/engine.py.230: loss: 0.15530 [I 2022-10-08 22:17:00] ...optimizers/trijoint.py.151: total_norm: 14.187347 [I 2022-10-08 22:17:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 2160/2304 [I 2022-10-08 22:17:09] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:00 [I 2022-10-08 22:17:09] ...trap/engines/engine.py.229: process: 0.83030 | load: 0.00299 [I 2022-10-08 22:17:09] ...trap/engines/engine.py.230: loss: 0.23139 [I 2022-10-08 22:17:09] ...optimizers/trijoint.py.151: total_norm: 22.930790 [I 2022-10-08 22:17:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 2170/2304 [I 2022-10-08 22:17:17] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:52 [I 2022-10-08 22:17:17] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00299 [I 2022-10-08 22:17:17] ...trap/engines/engine.py.230: loss: 0.14199 [I 2022-10-08 22:17:17] ...optimizers/trijoint.py.151: total_norm: 14.980414 [I 2022-10-08 22:17:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 2180/2304 [I 2022-10-08 22:17:25] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:43 [I 2022-10-08 22:17:25] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00299 [I 2022-10-08 22:17:25] ...trap/engines/engine.py.230: loss: 0.19799 [I 2022-10-08 22:17:25] ...optimizers/trijoint.py.151: total_norm: 26.436911 [I 2022-10-08 22:17:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 2190/2304 [I 2022-10-08 22:17:34] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:35 [I 2022-10-08 22:17:34] ...trap/engines/engine.py.229: process: 0.82962 | load: 0.00298 [I 2022-10-08 22:17:34] ...trap/engines/engine.py.230: loss: 0.26936 [I 2022-10-08 22:17:34] ...optimizers/trijoint.py.151: total_norm: 25.157492 [I 2022-10-08 22:17:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 2200/2304 [I 2022-10-08 22:17:42] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:27 [I 2022-10-08 22:17:42] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00301 [I 2022-10-08 22:17:42] ...trap/engines/engine.py.230: loss: 0.19134 [I 2022-10-08 22:17:42] ...optimizers/trijoint.py.151: total_norm: 12.308026 [I 2022-10-08 22:17:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 2210/2304 [I 2022-10-08 22:17:51] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:18 [I 2022-10-08 22:17:51] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00300 [I 2022-10-08 22:17:51] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-08 22:17:51] ...optimizers/trijoint.py.151: total_norm: 19.100769 [I 2022-10-08 22:17:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 2220/2304 [I 2022-10-08 22:17:59] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:10 [I 2022-10-08 22:17:59] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00299 [I 2022-10-08 22:17:59] ...trap/engines/engine.py.230: loss: 0.22569 [I 2022-10-08 22:17:59] ...optimizers/trijoint.py.151: total_norm: 16.505331 [I 2022-10-08 22:18:07] ...trap/engines/engine.py.224: train: epoch 21 | batch 2230/2304 [I 2022-10-08 22:18:07] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:02 [I 2022-10-08 22:18:07] ...trap/engines/engine.py.229: process: 0.83020 | load: 0.00307 [I 2022-10-08 22:18:07] ...trap/engines/engine.py.230: loss: 0.22086 [I 2022-10-08 22:18:07] ...optimizers/trijoint.py.151: total_norm: 26.237972 [I 2022-10-08 22:18:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 2240/2304 [I 2022-10-08 22:18:16] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:00:53 [I 2022-10-08 22:18:16] ...trap/engines/engine.py.229: process: 0.83677 | load: 0.00310 [I 2022-10-08 22:18:16] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-08 22:18:16] ...optimizers/trijoint.py.151: total_norm: 12.773576 [I 2022-10-08 22:18:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 2250/2304 [I 2022-10-08 22:18:24] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:00:45 [I 2022-10-08 22:18:24] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00308 [I 2022-10-08 22:18:24] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-08 22:18:24] ...optimizers/trijoint.py.151: total_norm: 15.334680 [I 2022-10-08 22:18:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 2260/2304 [I 2022-10-08 22:18:33] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:00:36 [I 2022-10-08 22:18:33] ...trap/engines/engine.py.229: process: 0.83850 | load: 0.00299 [I 2022-10-08 22:18:33] ...trap/engines/engine.py.230: loss: 0.20979 [I 2022-10-08 22:18:33] ...optimizers/trijoint.py.151: total_norm: 17.486120 [I 2022-10-08 22:18:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 2270/2304 [I 2022-10-08 22:18:41] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:28 [I 2022-10-08 22:18:41] ...trap/engines/engine.py.229: process: 0.82577 | load: 0.00300 [I 2022-10-08 22:18:41] ...trap/engines/engine.py.230: loss: 0.19567 [I 2022-10-08 22:18:41] ...optimizers/trijoint.py.151: total_norm: 12.783619 [I 2022-10-08 22:18:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 2280/2304 [I 2022-10-08 22:18:49] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:19 [I 2022-10-08 22:18:49] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00299 [I 2022-10-08 22:18:49] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-08 22:18:49] ...optimizers/trijoint.py.151: total_norm: 20.058403 [I 2022-10-08 22:18:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 2290/2304 [I 2022-10-08 22:18:58] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:11 [I 2022-10-08 22:18:58] ...trap/engines/engine.py.229: process: 0.85162 | load: 0.00299 [I 2022-10-08 22:18:58] ...trap/engines/engine.py.230: loss: 0.16635 [I 2022-10-08 22:18:58] ...optimizers/trijoint.py.151: total_norm: 17.501099 [I 2022-10-08 22:19:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 2300/2304 [I 2022-10-08 22:19:06] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:03 [I 2022-10-08 22:19:06] ...trap/engines/engine.py.229: process: 0.83334 | load: 0.00017 [I 2022-10-08 22:19:06] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-08 22:19:06] ...optimizers/trijoint.py.151: total_norm: 23.568497 [I 2022-10-08 22:19:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 2304/2304 [I 2022-10-08 22:19:10] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:00 [I 2022-10-08 22:19:10] ...trap/engines/engine.py.229: process: 0.81614 | load: 0.00017 [I 2022-10-08 22:19:10] ...trap/engines/engine.py.230: loss: 0.15053 [I 2022-10-08 22:19:10] ...optimizers/trijoint.py.151: total_norm: 15.838002 [S 2022-10-08 22:19:11] ...trap/engines/engine.py.129: train_epoch.epoch: 21 [S 2022-10-08 22:19:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.17873036591517952 [I 2022-10-08 22:19:11] ...s/criterions/triplet.py.84: margin = 0.1600000000000001 [I 2022-10-08 22:19:11] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 21 [I 2022-10-08 22:19:11] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 22:19:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 22:19:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 22:19:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 22:19:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 22:19:14] ...trap/engines/engine.py.321: eval: epoch 21 | batch 0/510 [I 2022-10-08 22:19:14] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:25:31 [I 2022-10-08 22:19:14] ...trap/engines/engine.py.326: process: 3.00270 | load: 2.46673 [I 2022-10-08 22:19:19] ...trap/engines/engine.py.321: eval: epoch 21 | batch 10/510 [I 2022-10-08 22:19:19] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:37 [I 2022-10-08 22:19:19] ...trap/engines/engine.py.326: process: 0.55705 | load: 0.00018 [I 2022-10-08 22:19:26] ...trap/engines/engine.py.321: eval: epoch 21 | batch 20/510 [I 2022-10-08 22:19:26] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:24 [I 2022-10-08 22:19:26] ...trap/engines/engine.py.326: process: 0.64972 | load: 0.00019 [I 2022-10-08 22:19:33] ...trap/engines/engine.py.321: eval: epoch 21 | batch 30/510 [I 2022-10-08 22:19:33] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:47 [I 2022-10-08 22:19:33] ...trap/engines/engine.py.326: process: 0.81356 | load: 0.00019 [I 2022-10-08 22:19:40] ...trap/engines/engine.py.321: eval: epoch 21 | batch 40/510 [I 2022-10-08 22:19:40] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:05:42 [I 2022-10-08 22:19:40] ...trap/engines/engine.py.326: process: 0.58683 | load: 0.00018 [I 2022-10-08 22:19:45] ...trap/engines/engine.py.321: eval: epoch 21 | batch 50/510 [I 2022-10-08 22:19:45] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:04:12 [I 2022-10-08 22:19:45] ...trap/engines/engine.py.326: process: 0.50681 | load: 0.00019 [I 2022-10-08 22:19:50] ...trap/engines/engine.py.321: eval: epoch 21 | batch 60/510 [I 2022-10-08 22:19:50] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:39 [I 2022-10-08 22:19:50] ...trap/engines/engine.py.326: process: 0.42517 | load: 0.00018 [I 2022-10-08 22:19:55] ...trap/engines/engine.py.321: eval: epoch 21 | batch 70/510 [I 2022-10-08 22:19:55] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:20 [I 2022-10-08 22:19:55] ...trap/engines/engine.py.326: process: 0.43732 | load: 0.00018 [I 2022-10-08 22:20:01] ...trap/engines/engine.py.321: eval: epoch 21 | batch 80/510 [I 2022-10-08 22:20:01] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:04:22 [I 2022-10-08 22:20:01] ...trap/engines/engine.py.326: process: 0.51778 | load: 0.00018 [I 2022-10-08 22:20:06] ...trap/engines/engine.py.321: eval: epoch 21 | batch 90/510 [I 2022-10-08 22:20:06] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:16 [I 2022-10-08 22:20:06] ...trap/engines/engine.py.326: process: 0.45995 | load: 0.00019 [I 2022-10-08 22:20:10] ...trap/engines/engine.py.321: eval: epoch 21 | batch 100/510 [I 2022-10-08 22:20:10] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:57 [I 2022-10-08 22:20:10] ...trap/engines/engine.py.326: process: 0.30967 | load: 0.00019 [I 2022-10-08 22:20:13] ...trap/engines/engine.py.321: eval: epoch 21 | batch 110/510 [I 2022-10-08 22:20:13] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:11 [I 2022-10-08 22:20:13] ...trap/engines/engine.py.326: process: 0.31482 | load: 0.00020 [I 2022-10-08 22:20:16] ...trap/engines/engine.py.321: eval: epoch 21 | batch 120/510 [I 2022-10-08 22:20:16] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:04 [I 2022-10-08 22:20:16] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00019 [I 2022-10-08 22:20:20] ...trap/engines/engine.py.321: eval: epoch 21 | batch 130/510 [I 2022-10-08 22:20:20] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:00 [I 2022-10-08 22:20:20] ...trap/engines/engine.py.326: process: 0.31895 | load: 0.00020 [I 2022-10-08 22:20:23] ...trap/engines/engine.py.321: eval: epoch 21 | batch 140/510 [I 2022-10-08 22:20:23] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:57 [I 2022-10-08 22:20:23] ...trap/engines/engine.py.326: process: 0.31758 | load: 0.00019 [I 2022-10-08 22:20:26] ...trap/engines/engine.py.321: eval: epoch 21 | batch 150/510 [I 2022-10-08 22:20:26] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:54 [I 2022-10-08 22:20:26] ...trap/engines/engine.py.326: process: 0.32114 | load: 0.00019 [I 2022-10-08 22:20:29] ...trap/engines/engine.py.321: eval: epoch 21 | batch 160/510 [I 2022-10-08 22:20:29] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:51 [I 2022-10-08 22:20:29] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00020 [I 2022-10-08 22:20:32] ...trap/engines/engine.py.321: eval: epoch 21 | batch 170/510 [I 2022-10-08 22:20:32] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:48 [I 2022-10-08 22:20:32] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00020 [I 2022-10-08 22:20:36] ...trap/engines/engine.py.321: eval: epoch 21 | batch 180/510 [I 2022-10-08 22:20:36] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:44 [I 2022-10-08 22:20:36] ...trap/engines/engine.py.326: process: 0.31776 | load: 0.00019 [I 2022-10-08 22:20:39] ...trap/engines/engine.py.321: eval: epoch 21 | batch 190/510 [I 2022-10-08 22:20:39] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:42 [I 2022-10-08 22:20:39] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00019 [I 2022-10-08 22:20:42] ...trap/engines/engine.py.321: eval: epoch 21 | batch 200/510 [I 2022-10-08 22:20:42] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:38 [I 2022-10-08 22:20:42] ...trap/engines/engine.py.326: process: 0.31754 | load: 0.00020 [I 2022-10-08 22:20:45] ...trap/engines/engine.py.321: eval: epoch 21 | batch 210/510 [I 2022-10-08 22:20:45] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:35 [I 2022-10-08 22:20:45] ...trap/engines/engine.py.326: process: 0.31931 | load: 0.00020 [I 2022-10-08 22:20:48] ...trap/engines/engine.py.321: eval: epoch 21 | batch 220/510 [I 2022-10-08 22:20:48] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:32 [I 2022-10-08 22:20:48] ...trap/engines/engine.py.326: process: 0.32099 | load: 0.00020 [I 2022-10-08 22:20:51] ...trap/engines/engine.py.321: eval: epoch 21 | batch 230/510 [I 2022-10-08 22:20:51] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:28 [I 2022-10-08 22:20:51] ...trap/engines/engine.py.326: process: 0.31838 | load: 0.00019 [I 2022-10-08 22:20:55] ...trap/engines/engine.py.321: eval: epoch 21 | batch 240/510 [I 2022-10-08 22:20:55] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:26 [I 2022-10-08 22:20:55] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00019 [I 2022-10-08 22:20:58] ...trap/engines/engine.py.321: eval: epoch 21 | batch 250/510 [I 2022-10-08 22:20:58] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:23 [I 2022-10-08 22:20:58] ...trap/engines/engine.py.326: process: 0.31157 | load: 0.00020 [I 2022-10-08 22:21:01] ...trap/engines/engine.py.321: eval: epoch 21 | batch 260/510 [I 2022-10-08 22:21:01] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:19 [I 2022-10-08 22:21:01] ...trap/engines/engine.py.326: process: 0.31832 | load: 0.00020 [I 2022-10-08 22:21:04] ...trap/engines/engine.py.321: eval: epoch 21 | batch 270/510 [I 2022-10-08 22:21:04] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:16 [I 2022-10-08 22:21:04] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-08 22:21:07] ...trap/engines/engine.py.321: eval: epoch 21 | batch 280/510 [I 2022-10-08 22:21:07] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:13 [I 2022-10-08 22:21:07] ...trap/engines/engine.py.326: process: 0.32408 | load: 0.00020 [I 2022-10-08 22:21:11] ...trap/engines/engine.py.321: eval: epoch 21 | batch 290/510 [I 2022-10-08 22:21:11] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:10 [I 2022-10-08 22:21:11] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00020 [I 2022-10-08 22:21:14] ...trap/engines/engine.py.321: eval: epoch 21 | batch 300/510 [I 2022-10-08 22:21:14] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:06 [I 2022-10-08 22:21:14] ...trap/engines/engine.py.326: process: 0.31284 | load: 0.00020 [I 2022-10-08 22:21:17] ...trap/engines/engine.py.321: eval: epoch 21 | batch 310/510 [I 2022-10-08 22:21:17] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:04 [I 2022-10-08 22:21:17] ...trap/engines/engine.py.326: process: 0.32372 | load: 0.00020 [I 2022-10-08 22:21:20] ...trap/engines/engine.py.321: eval: epoch 21 | batch 320/510 [I 2022-10-08 22:21:20] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:00 [I 2022-10-08 22:21:20] ...trap/engines/engine.py.326: process: 0.30942 | load: 0.00020 [I 2022-10-08 22:21:23] ...trap/engines/engine.py.321: eval: epoch 21 | batch 330/510 [I 2022-10-08 22:21:23] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:57 [I 2022-10-08 22:21:23] ...trap/engines/engine.py.326: process: 0.31833 | load: 0.00021 [I 2022-10-08 22:21:27] ...trap/engines/engine.py.321: eval: epoch 21 | batch 340/510 [I 2022-10-08 22:21:27] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:54 [I 2022-10-08 22:21:27] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00021 [I 2022-10-08 22:21:30] ...trap/engines/engine.py.321: eval: epoch 21 | batch 350/510 [I 2022-10-08 22:21:30] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:50 [I 2022-10-08 22:21:30] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00022 [I 2022-10-08 22:21:33] ...trap/engines/engine.py.321: eval: epoch 21 | batch 360/510 [I 2022-10-08 22:21:33] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:47 [I 2022-10-08 22:21:33] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00020 [I 2022-10-08 22:21:36] ...trap/engines/engine.py.321: eval: epoch 21 | batch 370/510 [I 2022-10-08 22:21:36] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-08 22:21:36] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00021 [I 2022-10-08 22:21:39] ...trap/engines/engine.py.321: eval: epoch 21 | batch 380/510 [I 2022-10-08 22:21:39] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-08 22:21:39] ...trap/engines/engine.py.326: process: 0.32126 | load: 0.00020 [I 2022-10-08 22:21:43] ...trap/engines/engine.py.321: eval: epoch 21 | batch 390/510 [I 2022-10-08 22:21:43] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:38 [I 2022-10-08 22:21:43] ...trap/engines/engine.py.326: process: 0.32212 | load: 0.00020 [I 2022-10-08 22:21:46] ...trap/engines/engine.py.321: eval: epoch 21 | batch 400/510 [I 2022-10-08 22:21:46] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:35 [I 2022-10-08 22:21:46] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00022 [I 2022-10-08 22:21:49] ...trap/engines/engine.py.321: eval: epoch 21 | batch 410/510 [I 2022-10-08 22:21:49] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:31 [I 2022-10-08 22:21:49] ...trap/engines/engine.py.326: process: 0.31962 | load: 0.00021 [I 2022-10-08 22:21:52] ...trap/engines/engine.py.321: eval: epoch 21 | batch 420/510 [I 2022-10-08 22:21:52] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-08 22:21:52] ...trap/engines/engine.py.326: process: 0.31585 | load: 0.00019 [I 2022-10-08 22:21:55] ...trap/engines/engine.py.321: eval: epoch 21 | batch 430/510 [I 2022-10-08 22:21:55] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-08 22:21:55] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00020 [I 2022-10-08 22:21:59] ...trap/engines/engine.py.321: eval: epoch 21 | batch 440/510 [I 2022-10-08 22:21:59] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:22 [I 2022-10-08 22:21:59] ...trap/engines/engine.py.326: process: 0.32027 | load: 0.00020 [I 2022-10-08 22:22:02] ...trap/engines/engine.py.321: eval: epoch 21 | batch 450/510 [I 2022-10-08 22:22:02] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:19 [I 2022-10-08 22:22:02] ...trap/engines/engine.py.326: process: 0.31880 | load: 0.00021 [I 2022-10-08 22:22:05] ...trap/engines/engine.py.321: eval: epoch 21 | batch 460/510 [I 2022-10-08 22:22:05] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:15 [I 2022-10-08 22:22:05] ...trap/engines/engine.py.326: process: 0.31518 | load: 0.00022 [I 2022-10-08 22:22:08] ...trap/engines/engine.py.321: eval: epoch 21 | batch 470/510 [I 2022-10-08 22:22:08] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-08 22:22:08] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00020 [I 2022-10-08 22:22:11] ...trap/engines/engine.py.321: eval: epoch 21 | batch 480/510 [I 2022-10-08 22:22:11] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-08 22:22:11] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00020 [I 2022-10-08 22:22:15] ...trap/engines/engine.py.321: eval: epoch 21 | batch 490/510 [I 2022-10-08 22:22:15] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-08 22:22:15] ...trap/engines/engine.py.326: process: 0.32459 | load: 0.00020 [I 2022-10-08 22:22:18] ...trap/engines/engine.py.321: eval: epoch 21 | batch 500/510 [I 2022-10-08 22:22:18] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:03 [I 2022-10-08 22:22:18] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00020 [I 2022-10-08 22:22:21] ...trap/engines/engine.py.321: eval: epoch 21 | batch 510/510 [I 2022-10-08 22:22:21] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:00 [I 2022-10-08 22:22:21] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00016 [S 2022-10-08 22:22:22] ...trap/engines/engine.py.132: eval_epoch.epoch: 21 [I 2022-10-08 22:22:22] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.657800 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.880600 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.924900 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.647500 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.878100 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.919700 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013430 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008405 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007422 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013720 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008006 [S 2022-10-08 22:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009089 [I 2022-10-08 22:23:02] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 22:23:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 22:23:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 22:23:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 22:23:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 22:23:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 22:23:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 22:23:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 22:23:12] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 22:23:12] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 22:23:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 22:23:20] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 22:23:20] ...trap/engines/engine.py.163: Training model on trainset for epoch 22 [I 2022-10-08 22:23:20] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 22:23:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 0/2304 [I 2022-10-08 22:23:24] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:02:51 [I 2022-10-08 22:23:24] ...trap/engines/engine.py.229: process: 3.19965 | load: 2.32847 [I 2022-10-08 22:23:24] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-08 22:23:24] ...optimizers/trijoint.py.151: total_norm: 32.480331 [I 2022-10-08 22:23:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 10/2304 [I 2022-10-08 22:23:32] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:45 [I 2022-10-08 22:23:32] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00613 [I 2022-10-08 22:23:32] ...trap/engines/engine.py.230: loss: 0.24043 [I 2022-10-08 22:23:32] ...optimizers/trijoint.py.151: total_norm: 12.572858 [I 2022-10-08 22:23:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 20/2304 [I 2022-10-08 22:23:40] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:32:54 [I 2022-10-08 22:23:40] ...trap/engines/engine.py.229: process: 0.82892 | load: 0.00557 [I 2022-10-08 22:23:40] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-08 22:23:40] ...optimizers/trijoint.py.151: total_norm: 13.139519 [I 2022-10-08 22:23:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 30/2304 [I 2022-10-08 22:23:49] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:47 [I 2022-10-08 22:23:49] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00537 [I 2022-10-08 22:23:49] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-08 22:23:49] ...optimizers/trijoint.py.151: total_norm: 33.265579 [I 2022-10-08 22:23:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 40/2304 [I 2022-10-08 22:23:57] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:31 [I 2022-10-08 22:23:57] ...trap/engines/engine.py.229: process: 0.83040 | load: 0.00403 [I 2022-10-08 22:23:57] ...trap/engines/engine.py.230: loss: 0.16655 [I 2022-10-08 22:23:57] ...optimizers/trijoint.py.151: total_norm: 20.676882 [I 2022-10-08 22:24:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 50/2304 [I 2022-10-08 22:24:06] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:49 [I 2022-10-08 22:24:06] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00329 [I 2022-10-08 22:24:06] ...trap/engines/engine.py.230: loss: 0.18578 [I 2022-10-08 22:24:06] ...optimizers/trijoint.py.151: total_norm: 25.543493 [I 2022-10-08 22:24:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 60/2304 [I 2022-10-08 22:24:14] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:25 [I 2022-10-08 22:24:14] ...trap/engines/engine.py.229: process: 0.83418 | load: 0.00340 [I 2022-10-08 22:24:14] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-08 22:24:14] ...optimizers/trijoint.py.151: total_norm: 13.149182 [I 2022-10-08 22:24:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 70/2304 [I 2022-10-08 22:24:23] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:19 [I 2022-10-08 22:24:23] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00372 [I 2022-10-08 22:24:23] ...trap/engines/engine.py.230: loss: 0.15560 [I 2022-10-08 22:24:23] ...optimizers/trijoint.py.151: total_norm: 17.083248 [I 2022-10-08 22:24:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 80/2304 [I 2022-10-08 22:24:31] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:30:54 [I 2022-10-08 22:24:31] ...trap/engines/engine.py.229: process: 0.82414 | load: 0.00320 [I 2022-10-08 22:24:31] ...trap/engines/engine.py.230: loss: 0.20879 [I 2022-10-08 22:24:31] ...optimizers/trijoint.py.151: total_norm: 25.506474 [I 2022-10-08 22:24:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 90/2304 [I 2022-10-08 22:24:39] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:52 [I 2022-10-08 22:24:39] ...trap/engines/engine.py.229: process: 0.83410 | load: 0.00351 [I 2022-10-08 22:24:39] ...trap/engines/engine.py.230: loss: 0.14029 [I 2022-10-08 22:24:39] ...optimizers/trijoint.py.151: total_norm: 13.902927 [I 2022-10-08 22:24:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 100/2304 [I 2022-10-08 22:24:48] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:46 [I 2022-10-08 22:24:48] ...trap/engines/engine.py.229: process: 0.83759 | load: 0.00353 [I 2022-10-08 22:24:48] ...trap/engines/engine.py.230: loss: 0.15438 [I 2022-10-08 22:24:48] ...optimizers/trijoint.py.151: total_norm: 15.323034 [I 2022-10-08 22:24:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 110/2304 [I 2022-10-08 22:24:56] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:42 [I 2022-10-08 22:24:56] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00317 [I 2022-10-08 22:24:56] ...trap/engines/engine.py.230: loss: 0.20993 [I 2022-10-08 22:24:56] ...optimizers/trijoint.py.151: total_norm: 17.869209 [I 2022-10-08 22:25:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 120/2304 [I 2022-10-08 22:25:05] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:39 [I 2022-10-08 22:25:05] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00336 [I 2022-10-08 22:25:05] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-08 22:25:05] ...optimizers/trijoint.py.151: total_norm: 15.435237 [I 2022-10-08 22:25:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 130/2304 [I 2022-10-08 22:25:13] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:30 [I 2022-10-08 22:25:13] ...trap/engines/engine.py.229: process: 0.84892 | load: 0.00314 [I 2022-10-08 22:25:13] ...trap/engines/engine.py.230: loss: 0.15869 [I 2022-10-08 22:25:13] ...optimizers/trijoint.py.151: total_norm: 17.441696 [I 2022-10-08 22:25:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 140/2304 [I 2022-10-08 22:25:21] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:16 [I 2022-10-08 22:25:21] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00342 [I 2022-10-08 22:25:21] ...trap/engines/engine.py.230: loss: 0.16421 [I 2022-10-08 22:25:21] ...optimizers/trijoint.py.151: total_norm: 13.753821 [I 2022-10-08 22:25:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 150/2304 [I 2022-10-08 22:25:30] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:03 [I 2022-10-08 22:25:30] ...trap/engines/engine.py.229: process: 0.83057 | load: 0.00334 [I 2022-10-08 22:25:30] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-08 22:25:30] ...optimizers/trijoint.py.151: total_norm: 14.730333 [I 2022-10-08 22:25:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 160/2304 [I 2022-10-08 22:25:38] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:26 [I 2022-10-08 22:25:38] ...trap/engines/engine.py.229: process: 0.83536 | load: 0.00342 [I 2022-10-08 22:25:38] ...trap/engines/engine.py.230: loss: 0.16095 [I 2022-10-08 22:25:38] ...optimizers/trijoint.py.151: total_norm: 13.434415 [I 2022-10-08 22:25:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 170/2304 [I 2022-10-08 22:25:47] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:53 [I 2022-10-08 22:25:47] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00305 [I 2022-10-08 22:25:47] ...trap/engines/engine.py.230: loss: 0.13246 [I 2022-10-08 22:25:47] ...optimizers/trijoint.py.151: total_norm: 9.415118 [I 2022-10-08 22:25:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 180/2304 [I 2022-10-08 22:25:55] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:42 [I 2022-10-08 22:25:55] ...trap/engines/engine.py.229: process: 0.83661 | load: 0.00310 [I 2022-10-08 22:25:55] ...trap/engines/engine.py.230: loss: 0.20894 [I 2022-10-08 22:25:55] ...optimizers/trijoint.py.151: total_norm: 12.412713 [I 2022-10-08 22:26:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 190/2304 [I 2022-10-08 22:26:04] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:35 [I 2022-10-08 22:26:04] ...trap/engines/engine.py.229: process: 0.84099 | load: 0.00317 [I 2022-10-08 22:26:04] ...trap/engines/engine.py.230: loss: 0.19324 [I 2022-10-08 22:26:04] ...optimizers/trijoint.py.151: total_norm: 31.121788 [I 2022-10-08 22:26:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 200/2304 [I 2022-10-08 22:26:12] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:25 [I 2022-10-08 22:26:12] ...trap/engines/engine.py.229: process: 0.82659 | load: 0.00309 [I 2022-10-08 22:26:12] ...trap/engines/engine.py.230: loss: 0.15078 [I 2022-10-08 22:26:12] ...optimizers/trijoint.py.151: total_norm: 12.442747 [I 2022-10-08 22:26:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 210/2304 [I 2022-10-08 22:26:20] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:25 [I 2022-10-08 22:26:20] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00324 [I 2022-10-08 22:26:20] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-08 22:26:20] ...optimizers/trijoint.py.151: total_norm: 36.542282 [I 2022-10-08 22:26:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 220/2304 [I 2022-10-08 22:26:29] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:09 [I 2022-10-08 22:26:29] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00313 [I 2022-10-08 22:26:29] ...trap/engines/engine.py.230: loss: 0.19724 [I 2022-10-08 22:26:29] ...optimizers/trijoint.py.151: total_norm: 24.158203 [I 2022-10-08 22:26:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 230/2304 [I 2022-10-08 22:26:37] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:28:57 [I 2022-10-08 22:26:37] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00300 [I 2022-10-08 22:26:37] ...trap/engines/engine.py.230: loss: 0.19670 [I 2022-10-08 22:26:37] ...optimizers/trijoint.py.151: total_norm: 18.732561 [I 2022-10-08 22:26:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 240/2304 [I 2022-10-08 22:26:46] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:45 [I 2022-10-08 22:26:46] ...trap/engines/engine.py.229: process: 0.83050 | load: 0.00302 [I 2022-10-08 22:26:46] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-08 22:26:46] ...optimizers/trijoint.py.151: total_norm: 18.571768 [I 2022-10-08 22:26:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 250/2304 [I 2022-10-08 22:26:54] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:42 [I 2022-10-08 22:26:54] ...trap/engines/engine.py.229: process: 0.83748 | load: 0.00297 [I 2022-10-08 22:26:54] ...trap/engines/engine.py.230: loss: 0.17434 [I 2022-10-08 22:26:54] ...optimizers/trijoint.py.151: total_norm: 15.561666 [I 2022-10-08 22:27:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 260/2304 [I 2022-10-08 22:27:03] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:35 [I 2022-10-08 22:27:03] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00298 [I 2022-10-08 22:27:03] ...trap/engines/engine.py.230: loss: 0.19336 [I 2022-10-08 22:27:03] ...optimizers/trijoint.py.151: total_norm: 28.510715 [I 2022-10-08 22:27:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 270/2304 [I 2022-10-08 22:27:11] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:12 [I 2022-10-08 22:27:11] ...trap/engines/engine.py.229: process: 0.83572 | load: 0.00299 [I 2022-10-08 22:27:11] ...trap/engines/engine.py.230: loss: 0.25730 [I 2022-10-08 22:27:11] ...optimizers/trijoint.py.151: total_norm: 34.872368 [I 2022-10-08 22:27:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 280/2304 [I 2022-10-08 22:27:19] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:24 [I 2022-10-08 22:27:19] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00300 [I 2022-10-08 22:27:19] ...trap/engines/engine.py.230: loss: 0.23147 [I 2022-10-08 22:27:19] ...optimizers/trijoint.py.151: total_norm: 19.290848 [I 2022-10-08 22:27:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 290/2304 [I 2022-10-08 22:27:28] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:08 [I 2022-10-08 22:27:28] ...trap/engines/engine.py.229: process: 0.82538 | load: 0.00323 [I 2022-10-08 22:27:28] ...trap/engines/engine.py.230: loss: 0.16519 [I 2022-10-08 22:27:28] ...optimizers/trijoint.py.151: total_norm: 15.796774 [I 2022-10-08 22:27:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 300/2304 [I 2022-10-08 22:27:54] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:44:02 [I 2022-10-08 22:27:54] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00303 [I 2022-10-08 22:27:54] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-08 22:27:54] ...optimizers/trijoint.py.151: total_norm: 18.035849 [I 2022-10-08 22:28:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 310/2304 [I 2022-10-08 22:28:03] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:29:27 [I 2022-10-08 22:28:03] ...trap/engines/engine.py.229: process: 0.82790 | load: 0.00307 [I 2022-10-08 22:28:03] ...trap/engines/engine.py.230: loss: 0.19615 [I 2022-10-08 22:28:03] ...optimizers/trijoint.py.151: total_norm: 48.928993 [I 2022-10-08 22:28:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 320/2304 [I 2022-10-08 22:28:11] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:51 [I 2022-10-08 22:28:11] ...trap/engines/engine.py.229: process: 0.83884 | load: 0.00301 [I 2022-10-08 22:28:11] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-08 22:28:11] ...optimizers/trijoint.py.151: total_norm: 18.439198 [I 2022-10-08 22:28:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 330/2304 [I 2022-10-08 22:28:19] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:27 [I 2022-10-08 22:28:19] ...trap/engines/engine.py.229: process: 0.82528 | load: 0.00300 [I 2022-10-08 22:28:19] ...trap/engines/engine.py.230: loss: 0.21737 [I 2022-10-08 22:28:19] ...optimizers/trijoint.py.151: total_norm: 19.279509 [I 2022-10-08 22:28:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 340/2304 [I 2022-10-08 22:28:28] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:40 [I 2022-10-08 22:28:28] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00323 [I 2022-10-08 22:28:28] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-08 22:28:28] ...optimizers/trijoint.py.151: total_norm: 20.145279 [I 2022-10-08 22:28:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 350/2304 [I 2022-10-08 22:28:36] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:17 [I 2022-10-08 22:28:36] ...trap/engines/engine.py.229: process: 0.84013 | load: 0.00299 [I 2022-10-08 22:28:36] ...trap/engines/engine.py.230: loss: 0.16976 [I 2022-10-08 22:28:36] ...optimizers/trijoint.py.151: total_norm: 21.119921 [I 2022-10-08 22:28:45] ...trap/engines/engine.py.224: train: epoch 22 | batch 360/2304 [I 2022-10-08 22:28:45] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:17 [I 2022-10-08 22:28:45] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00312 [I 2022-10-08 22:28:45] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-08 22:28:45] ...optimizers/trijoint.py.151: total_norm: 19.386911 [I 2022-10-08 22:28:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 370/2304 [I 2022-10-08 22:28:53] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:05 [I 2022-10-08 22:28:53] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00300 [I 2022-10-08 22:28:53] ...trap/engines/engine.py.230: loss: 0.28335 [I 2022-10-08 22:28:53] ...optimizers/trijoint.py.151: total_norm: 29.582800 [I 2022-10-08 22:29:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 380/2304 [I 2022-10-08 22:29:01] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:52 [I 2022-10-08 22:29:01] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00306 [I 2022-10-08 22:29:01] ...trap/engines/engine.py.230: loss: 0.21969 [I 2022-10-08 22:29:01] ...optimizers/trijoint.py.151: total_norm: 20.901680 [I 2022-10-08 22:29:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 390/2304 [I 2022-10-08 22:29:10] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:49 [I 2022-10-08 22:29:10] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00306 [I 2022-10-08 22:29:10] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-08 22:29:10] ...optimizers/trijoint.py.151: total_norm: 17.950314 [I 2022-10-08 22:29:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 400/2304 [I 2022-10-08 22:29:18] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:36 [I 2022-10-08 22:29:18] ...trap/engines/engine.py.229: process: 0.82587 | load: 0.00308 [I 2022-10-08 22:29:18] ...trap/engines/engine.py.230: loss: 0.21189 [I 2022-10-08 22:29:18] ...optimizers/trijoint.py.151: total_norm: 21.279503 [I 2022-10-08 22:29:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 410/2304 [I 2022-10-08 22:29:27] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:30 [I 2022-10-08 22:29:27] ...trap/engines/engine.py.229: process: 0.83924 | load: 0.00300 [I 2022-10-08 22:29:27] ...trap/engines/engine.py.230: loss: 0.19431 [I 2022-10-08 22:29:27] ...optimizers/trijoint.py.151: total_norm: 17.137310 [I 2022-10-08 22:29:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 420/2304 [I 2022-10-08 22:29:35] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:25 [I 2022-10-08 22:29:35] ...trap/engines/engine.py.229: process: 0.84573 | load: 0.00299 [I 2022-10-08 22:29:35] ...trap/engines/engine.py.230: loss: 0.15028 [I 2022-10-08 22:29:35] ...optimizers/trijoint.py.151: total_norm: 16.965971 [I 2022-10-08 22:29:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 430/2304 [I 2022-10-08 22:29:44] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:17 [I 2022-10-08 22:29:44] ...trap/engines/engine.py.229: process: 0.85468 | load: 0.00314 [I 2022-10-08 22:29:44] ...trap/engines/engine.py.230: loss: 0.20807 [I 2022-10-08 22:29:44] ...optimizers/trijoint.py.151: total_norm: 24.398960 [I 2022-10-08 22:29:52] ...trap/engines/engine.py.224: train: epoch 22 | batch 440/2304 [I 2022-10-08 22:29:52] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:25:56 [I 2022-10-08 22:29:52] ...trap/engines/engine.py.229: process: 0.83338 | load: 0.00303 [I 2022-10-08 22:29:52] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-08 22:29:52] ...optimizers/trijoint.py.151: total_norm: 19.752672 [I 2022-10-08 22:30:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 450/2304 [I 2022-10-08 22:30:00] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:51 [I 2022-10-08 22:30:00] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00307 [I 2022-10-08 22:30:00] ...trap/engines/engine.py.230: loss: 0.16734 [I 2022-10-08 22:30:00] ...optimizers/trijoint.py.151: total_norm: 28.100208 [I 2022-10-08 22:30:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 460/2304 [I 2022-10-08 22:30:09] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:45 [I 2022-10-08 22:30:09] ...trap/engines/engine.py.229: process: 0.84583 | load: 0.00300 [I 2022-10-08 22:30:09] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-08 22:30:09] ...optimizers/trijoint.py.151: total_norm: 26.115389 [I 2022-10-08 22:30:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 470/2304 [I 2022-10-08 22:30:17] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:40 [I 2022-10-08 22:30:17] ...trap/engines/engine.py.229: process: 0.83445 | load: 0.00298 [I 2022-10-08 22:30:17] ...trap/engines/engine.py.230: loss: 0.26922 [I 2022-10-08 22:30:17] ...optimizers/trijoint.py.151: total_norm: 19.507238 [I 2022-10-08 22:30:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 480/2304 [I 2022-10-08 22:30:26] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:24 [I 2022-10-08 22:30:26] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00299 [I 2022-10-08 22:30:26] ...trap/engines/engine.py.230: loss: 0.19306 [I 2022-10-08 22:30:26] ...optimizers/trijoint.py.151: total_norm: 18.830524 [I 2022-10-08 22:30:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 490/2304 [I 2022-10-08 22:30:34] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:17 [I 2022-10-08 22:30:34] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00305 [I 2022-10-08 22:30:34] ...trap/engines/engine.py.230: loss: 0.17465 [I 2022-10-08 22:30:34] ...optimizers/trijoint.py.151: total_norm: 35.283543 [I 2022-10-08 22:30:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 500/2304 [I 2022-10-08 22:30:42] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:13 [I 2022-10-08 22:30:42] ...trap/engines/engine.py.229: process: 0.83414 | load: 0.00299 [I 2022-10-08 22:30:42] ...trap/engines/engine.py.230: loss: 0.17747 [I 2022-10-08 22:30:42] ...optimizers/trijoint.py.151: total_norm: 18.769079 [I 2022-10-08 22:30:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 510/2304 [I 2022-10-08 22:30:51] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:00 [I 2022-10-08 22:30:51] ...trap/engines/engine.py.229: process: 0.83588 | load: 0.00301 [I 2022-10-08 22:30:51] ...trap/engines/engine.py.230: loss: 0.20115 [I 2022-10-08 22:30:51] ...optimizers/trijoint.py.151: total_norm: 16.360416 [I 2022-10-08 22:30:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 520/2304 [I 2022-10-08 22:30:59] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:50 [I 2022-10-08 22:30:59] ...trap/engines/engine.py.229: process: 0.83136 | load: 0.00301 [I 2022-10-08 22:30:59] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-08 22:30:59] ...optimizers/trijoint.py.151: total_norm: 17.727341 [I 2022-10-08 22:31:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 530/2304 [I 2022-10-08 22:31:08] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:44 [I 2022-10-08 22:31:08] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00301 [I 2022-10-08 22:31:08] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-08 22:31:08] ...optimizers/trijoint.py.151: total_norm: 14.574510 [I 2022-10-08 22:31:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 540/2304 [I 2022-10-08 22:31:16] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:33 [I 2022-10-08 22:31:16] ...trap/engines/engine.py.229: process: 0.83690 | load: 0.00298 [I 2022-10-08 22:31:16] ...trap/engines/engine.py.230: loss: 0.20924 [I 2022-10-08 22:31:16] ...optimizers/trijoint.py.151: total_norm: 21.765440 [I 2022-10-08 22:31:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 550/2304 [I 2022-10-08 22:31:24] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:26 [I 2022-10-08 22:31:24] ...trap/engines/engine.py.229: process: 0.83249 | load: 0.00297 [I 2022-10-08 22:31:24] ...trap/engines/engine.py.230: loss: 0.16796 [I 2022-10-08 22:31:24] ...optimizers/trijoint.py.151: total_norm: 25.468752 [I 2022-10-08 22:31:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 560/2304 [I 2022-10-08 22:31:33] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:17 [I 2022-10-08 22:31:33] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00312 [I 2022-10-08 22:31:33] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-08 22:31:33] ...optimizers/trijoint.py.151: total_norm: 18.937645 [I 2022-10-08 22:31:41] ...trap/engines/engine.py.224: train: epoch 22 | batch 570/2304 [I 2022-10-08 22:31:41] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:13 [I 2022-10-08 22:31:41] ...trap/engines/engine.py.229: process: 0.83971 | load: 0.00297 [I 2022-10-08 22:31:41] ...trap/engines/engine.py.230: loss: 0.19400 [I 2022-10-08 22:31:41] ...optimizers/trijoint.py.151: total_norm: 29.898235 [I 2022-10-08 22:31:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 580/2304 [I 2022-10-08 22:31:49] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:24:03 [I 2022-10-08 22:31:49] ...trap/engines/engine.py.229: process: 0.83728 | load: 0.00299 [I 2022-10-08 22:31:49] ...trap/engines/engine.py.230: loss: 0.10837 [I 2022-10-08 22:31:49] ...optimizers/trijoint.py.151: total_norm: 14.610547 [I 2022-10-08 22:31:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 590/2304 [I 2022-10-08 22:31:58] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:24:00 [I 2022-10-08 22:31:58] ...trap/engines/engine.py.229: process: 0.84732 | load: 0.00302 [I 2022-10-08 22:31:58] ...trap/engines/engine.py.230: loss: 0.22455 [I 2022-10-08 22:31:58] ...optimizers/trijoint.py.151: total_norm: 25.639809 [I 2022-10-08 22:32:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 600/2304 [I 2022-10-08 22:32:06] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:48 [I 2022-10-08 22:32:06] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00301 [I 2022-10-08 22:32:06] ...trap/engines/engine.py.230: loss: 0.10540 [I 2022-10-08 22:32:06] ...optimizers/trijoint.py.151: total_norm: 22.887924 [I 2022-10-08 22:32:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 610/2304 [I 2022-10-08 22:32:15] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:44 [I 2022-10-08 22:32:15] ...trap/engines/engine.py.229: process: 0.82955 | load: 0.00305 [I 2022-10-08 22:32:15] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-08 22:32:15] ...optimizers/trijoint.py.151: total_norm: 8.891230 [I 2022-10-08 22:32:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 620/2304 [I 2022-10-08 22:32:23] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:43 [I 2022-10-08 22:32:23] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00316 [I 2022-10-08 22:32:23] ...trap/engines/engine.py.230: loss: 0.24230 [I 2022-10-08 22:32:23] ...optimizers/trijoint.py.151: total_norm: 19.334764 [I 2022-10-08 22:32:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 630/2304 [I 2022-10-08 22:32:32] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:27 [I 2022-10-08 22:32:32] ...trap/engines/engine.py.229: process: 0.83112 | load: 0.00301 [I 2022-10-08 22:32:32] ...trap/engines/engine.py.230: loss: 0.19005 [I 2022-10-08 22:32:32] ...optimizers/trijoint.py.151: total_norm: 21.197853 [I 2022-10-08 22:32:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 640/2304 [I 2022-10-08 22:32:40] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:19 [I 2022-10-08 22:32:40] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00302 [I 2022-10-08 22:32:40] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-08 22:32:40] ...optimizers/trijoint.py.151: total_norm: 20.208822 [I 2022-10-08 22:32:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 650/2304 [I 2022-10-08 22:32:49] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:23:10 [I 2022-10-08 22:32:49] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00299 [I 2022-10-08 22:32:49] ...trap/engines/engine.py.230: loss: 0.18968 [I 2022-10-08 22:32:49] ...optimizers/trijoint.py.151: total_norm: 20.102694 [I 2022-10-08 22:32:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 660/2304 [I 2022-10-08 22:32:57] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:56 [I 2022-10-08 22:32:57] ...trap/engines/engine.py.229: process: 0.82682 | load: 0.00301 [I 2022-10-08 22:32:57] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-08 22:32:57] ...optimizers/trijoint.py.151: total_norm: 27.697678 [I 2022-10-08 22:33:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 670/2304 [I 2022-10-08 22:33:05] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:23:01 [I 2022-10-08 22:33:05] ...trap/engines/engine.py.229: process: 0.86435 | load: 0.00300 [I 2022-10-08 22:33:05] ...trap/engines/engine.py.230: loss: 0.17624 [I 2022-10-08 22:33:05] ...optimizers/trijoint.py.151: total_norm: 18.558664 [I 2022-10-08 22:33:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 680/2304 [I 2022-10-08 22:33:14] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:48 [I 2022-10-08 22:33:14] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00298 [I 2022-10-08 22:33:14] ...trap/engines/engine.py.230: loss: 0.19261 [I 2022-10-08 22:33:14] ...optimizers/trijoint.py.151: total_norm: 17.570984 [I 2022-10-08 22:33:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 690/2304 [I 2022-10-08 22:33:22] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:46 [I 2022-10-08 22:33:22] ...trap/engines/engine.py.229: process: 0.86092 | load: 0.00300 [I 2022-10-08 22:33:22] ...trap/engines/engine.py.230: loss: 0.17940 [I 2022-10-08 22:33:22] ...optimizers/trijoint.py.151: total_norm: 31.251158 [I 2022-10-08 22:33:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 700/2304 [I 2022-10-08 22:33:31] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:31 [I 2022-10-08 22:33:31] ...trap/engines/engine.py.229: process: 0.84275 | load: 0.00297 [I 2022-10-08 22:33:31] ...trap/engines/engine.py.230: loss: 0.19458 [I 2022-10-08 22:33:31] ...optimizers/trijoint.py.151: total_norm: 19.321016 [I 2022-10-08 22:33:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 710/2304 [I 2022-10-08 22:33:39] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:24 [I 2022-10-08 22:33:39] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00298 [I 2022-10-08 22:33:39] ...trap/engines/engine.py.230: loss: 0.12114 [I 2022-10-08 22:33:39] ...optimizers/trijoint.py.151: total_norm: 14.707499 [I 2022-10-08 22:33:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 720/2304 [I 2022-10-08 22:33:48] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:02 [I 2022-10-08 22:33:48] ...trap/engines/engine.py.229: process: 0.81973 | load: 0.00300 [I 2022-10-08 22:33:48] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-08 22:33:48] ...optimizers/trijoint.py.151: total_norm: 21.387487 [I 2022-10-08 22:33:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 730/2304 [I 2022-10-08 22:33:56] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:02 [I 2022-10-08 22:33:56] ...trap/engines/engine.py.229: process: 0.83723 | load: 0.00303 [I 2022-10-08 22:33:56] ...trap/engines/engine.py.230: loss: 0.14231 [I 2022-10-08 22:33:56] ...optimizers/trijoint.py.151: total_norm: 22.537136 [I 2022-10-08 22:34:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 740/2304 [I 2022-10-08 22:34:05] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:56 [I 2022-10-08 22:34:05] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00301 [I 2022-10-08 22:34:05] ...trap/engines/engine.py.230: loss: 0.16861 [I 2022-10-08 22:34:05] ...optimizers/trijoint.py.151: total_norm: 12.284801 [I 2022-10-08 22:34:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 750/2304 [I 2022-10-08 22:34:13] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:42 [I 2022-10-08 22:34:13] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00303 [I 2022-10-08 22:34:13] ...trap/engines/engine.py.230: loss: 0.18637 [I 2022-10-08 22:34:13] ...optimizers/trijoint.py.151: total_norm: 19.596573 [I 2022-10-08 22:34:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 760/2304 [I 2022-10-08 22:34:21] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:33 [I 2022-10-08 22:34:21] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00303 [I 2022-10-08 22:34:21] ...trap/engines/engine.py.230: loss: 0.18812 [I 2022-10-08 22:34:21] ...optimizers/trijoint.py.151: total_norm: 22.665514 [I 2022-10-08 22:34:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 770/2304 [I 2022-10-08 22:34:30] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:25 [I 2022-10-08 22:34:30] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00302 [I 2022-10-08 22:34:30] ...trap/engines/engine.py.230: loss: 0.18971 [I 2022-10-08 22:34:30] ...optimizers/trijoint.py.151: total_norm: 13.509566 [I 2022-10-08 22:34:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 780/2304 [I 2022-10-08 22:34:38] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:15 [I 2022-10-08 22:34:38] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00301 [I 2022-10-08 22:34:38] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-08 22:34:38] ...optimizers/trijoint.py.151: total_norm: 12.897301 [I 2022-10-08 22:34:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 790/2304 [I 2022-10-08 22:34:47] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:19 [I 2022-10-08 22:34:47] ...trap/engines/engine.py.229: process: 0.87421 | load: 0.00303 [I 2022-10-08 22:34:47] ...trap/engines/engine.py.230: loss: 0.10732 [I 2022-10-08 22:34:47] ...optimizers/trijoint.py.151: total_norm: 11.574025 [I 2022-10-08 22:34:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 800/2304 [I 2022-10-08 22:34:55] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:12 [I 2022-10-08 22:34:55] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00299 [I 2022-10-08 22:34:55] ...trap/engines/engine.py.230: loss: 0.15527 [I 2022-10-08 22:34:55] ...optimizers/trijoint.py.151: total_norm: 14.577548 [I 2022-10-08 22:35:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 810/2304 [I 2022-10-08 22:35:04] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:56 [I 2022-10-08 22:35:04] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00299 [I 2022-10-08 22:35:04] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-08 22:35:04] ...optimizers/trijoint.py.151: total_norm: 16.332701 [I 2022-10-08 22:35:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 820/2304 [I 2022-10-08 22:35:12] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:45 [I 2022-10-08 22:35:12] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00300 [I 2022-10-08 22:35:12] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-08 22:35:12] ...optimizers/trijoint.py.151: total_norm: 20.724833 [I 2022-10-08 22:35:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 830/2304 [I 2022-10-08 22:35:20] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:44 [I 2022-10-08 22:35:20] ...trap/engines/engine.py.229: process: 0.86382 | load: 0.00299 [I 2022-10-08 22:35:20] ...trap/engines/engine.py.230: loss: 0.11621 [I 2022-10-08 22:35:20] ...optimizers/trijoint.py.151: total_norm: 12.298098 [I 2022-10-08 22:35:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 840/2304 [I 2022-10-08 22:35:29] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:24 [I 2022-10-08 22:35:29] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00299 [I 2022-10-08 22:35:29] ...trap/engines/engine.py.230: loss: 0.13114 [I 2022-10-08 22:35:29] ...optimizers/trijoint.py.151: total_norm: 20.186693 [I 2022-10-08 22:35:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 850/2304 [I 2022-10-08 22:35:37] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:17 [I 2022-10-08 22:35:37] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00306 [I 2022-10-08 22:35:37] ...trap/engines/engine.py.230: loss: 0.21823 [I 2022-10-08 22:35:37] ...optimizers/trijoint.py.151: total_norm: 25.772203 [I 2022-10-08 22:35:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 860/2304 [I 2022-10-08 22:35:46] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:10 [I 2022-10-08 22:35:46] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00307 [I 2022-10-08 22:35:46] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-08 22:35:46] ...optimizers/trijoint.py.151: total_norm: 15.944822 [I 2022-10-08 22:35:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 870/2304 [I 2022-10-08 22:35:54] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:01 [I 2022-10-08 22:35:54] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00302 [I 2022-10-08 22:35:54] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-08 22:35:54] ...optimizers/trijoint.py.151: total_norm: 17.244329 [I 2022-10-08 22:36:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 880/2304 [I 2022-10-08 22:36:02] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:43 [I 2022-10-08 22:36:02] ...trap/engines/engine.py.229: process: 0.82707 | load: 0.00300 [I 2022-10-08 22:36:02] ...trap/engines/engine.py.230: loss: 0.17312 [I 2022-10-08 22:36:02] ...optimizers/trijoint.py.151: total_norm: 25.605986 [I 2022-10-08 22:36:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 890/2304 [I 2022-10-08 22:36:11] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:45 [I 2022-10-08 22:36:11] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00299 [I 2022-10-08 22:36:11] ...trap/engines/engine.py.230: loss: 0.21469 [I 2022-10-08 22:36:11] ...optimizers/trijoint.py.151: total_norm: 25.645908 [I 2022-10-08 22:36:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 900/2304 [I 2022-10-08 22:36:19] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:35 [I 2022-10-08 22:36:19] ...trap/engines/engine.py.229: process: 0.83685 | load: 0.00301 [I 2022-10-08 22:36:19] ...trap/engines/engine.py.230: loss: 0.23937 [I 2022-10-08 22:36:19] ...optimizers/trijoint.py.151: total_norm: 42.296043 [I 2022-10-08 22:36:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 910/2304 [I 2022-10-08 22:36:28] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:29 [I 2022-10-08 22:36:28] ...trap/engines/engine.py.229: process: 0.82963 | load: 0.00309 [I 2022-10-08 22:36:28] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-08 22:36:28] ...optimizers/trijoint.py.151: total_norm: 24.778227 [I 2022-10-08 22:36:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 920/2304 [I 2022-10-08 22:36:36] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:26 [I 2022-10-08 22:36:36] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00300 [I 2022-10-08 22:36:36] ...trap/engines/engine.py.230: loss: 0.14085 [I 2022-10-08 22:36:36] ...optimizers/trijoint.py.151: total_norm: 14.987312 [I 2022-10-08 22:36:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 930/2304 [I 2022-10-08 22:36:44] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:09 [I 2022-10-08 22:36:44] ...trap/engines/engine.py.229: process: 0.82850 | load: 0.00301 [I 2022-10-08 22:36:44] ...trap/engines/engine.py.230: loss: 0.20200 [I 2022-10-08 22:36:44] ...optimizers/trijoint.py.151: total_norm: 17.987688 [I 2022-10-08 22:36:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 940/2304 [I 2022-10-08 22:36:53] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:04 [I 2022-10-08 22:36:53] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00300 [I 2022-10-08 22:36:53] ...trap/engines/engine.py.230: loss: 0.17427 [I 2022-10-08 22:36:53] ...optimizers/trijoint.py.151: total_norm: 20.500883 [I 2022-10-08 22:37:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 950/2304 [I 2022-10-08 22:37:01] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:55 [I 2022-10-08 22:37:01] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00304 [I 2022-10-08 22:37:01] ...trap/engines/engine.py.230: loss: 0.21007 [I 2022-10-08 22:37:01] ...optimizers/trijoint.py.151: total_norm: 20.945545 [I 2022-10-08 22:37:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 960/2304 [I 2022-10-08 22:37:10] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:53 [I 2022-10-08 22:37:10] ...trap/engines/engine.py.229: process: 0.84712 | load: 0.00304 [I 2022-10-08 22:37:10] ...trap/engines/engine.py.230: loss: 0.18589 [I 2022-10-08 22:37:10] ...optimizers/trijoint.py.151: total_norm: 13.039027 [I 2022-10-08 22:37:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 970/2304 [I 2022-10-08 22:37:18] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:38 [I 2022-10-08 22:37:18] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00310 [I 2022-10-08 22:37:18] ...trap/engines/engine.py.230: loss: 0.13916 [I 2022-10-08 22:37:18] ...optimizers/trijoint.py.151: total_norm: 16.810865 [I 2022-10-08 22:37:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 980/2304 [I 2022-10-08 22:37:26] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:30 [I 2022-10-08 22:37:26] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00313 [I 2022-10-08 22:37:26] ...trap/engines/engine.py.230: loss: 0.18638 [I 2022-10-08 22:37:26] ...optimizers/trijoint.py.151: total_norm: 29.708950 [I 2022-10-08 22:37:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 990/2304 [I 2022-10-08 22:37:35] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:21 [I 2022-10-08 22:37:35] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00312 [I 2022-10-08 22:37:35] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-08 22:37:35] ...optimizers/trijoint.py.151: total_norm: 20.672247 [I 2022-10-08 22:37:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 1000/2304 [I 2022-10-08 22:37:43] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:21 [I 2022-10-08 22:37:43] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00308 [I 2022-10-08 22:37:43] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-08 22:37:43] ...optimizers/trijoint.py.151: total_norm: 20.817719 [I 2022-10-08 22:37:52] ...trap/engines/engine.py.224: train: epoch 22 | batch 1010/2304 [I 2022-10-08 22:37:52] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:08 [I 2022-10-08 22:37:52] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00298 [I 2022-10-08 22:37:52] ...trap/engines/engine.py.230: loss: 0.12317 [I 2022-10-08 22:37:52] ...optimizers/trijoint.py.151: total_norm: 21.138336 [I 2022-10-08 22:38:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 1020/2304 [I 2022-10-08 22:38:00] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:54 [I 2022-10-08 22:38:00] ...trap/engines/engine.py.229: process: 0.83167 | load: 0.00299 [I 2022-10-08 22:38:00] ...trap/engines/engine.py.230: loss: 0.18856 [I 2022-10-08 22:38:00] ...optimizers/trijoint.py.151: total_norm: 15.487818 [I 2022-10-08 22:38:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 1030/2304 [I 2022-10-08 22:38:08] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:41 [I 2022-10-08 22:38:08] ...trap/engines/engine.py.229: process: 0.83313 | load: 0.00298 [I 2022-10-08 22:38:08] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-08 22:38:08] ...optimizers/trijoint.py.151: total_norm: 18.595766 [I 2022-10-08 22:38:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 1040/2304 [I 2022-10-08 22:38:17] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:39 [I 2022-10-08 22:38:17] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00298 [I 2022-10-08 22:38:17] ...trap/engines/engine.py.230: loss: 0.19422 [I 2022-10-08 22:38:17] ...optimizers/trijoint.py.151: total_norm: 25.862577 [I 2022-10-08 22:38:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 1050/2304 [I 2022-10-08 22:38:25] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:38 [I 2022-10-08 22:38:25] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00300 [I 2022-10-08 22:38:25] ...trap/engines/engine.py.230: loss: 0.20832 [I 2022-10-08 22:38:25] ...optimizers/trijoint.py.151: total_norm: 15.116079 [I 2022-10-08 22:38:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 1060/2304 [I 2022-10-08 22:38:34] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:19 [I 2022-10-08 22:38:34] ...trap/engines/engine.py.229: process: 0.83175 | load: 0.00301 [I 2022-10-08 22:38:34] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-08 22:38:34] ...optimizers/trijoint.py.151: total_norm: 14.861532 [I 2022-10-08 22:38:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 1070/2304 [I 2022-10-08 22:38:42] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:17:14 [I 2022-10-08 22:38:42] ...trap/engines/engine.py.229: process: 0.83399 | load: 0.00301 [I 2022-10-08 22:38:42] ...trap/engines/engine.py.230: loss: 0.15044 [I 2022-10-08 22:38:42] ...optimizers/trijoint.py.151: total_norm: 15.685512 [I 2022-10-08 22:38:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 1080/2304 [I 2022-10-08 22:38:51] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:09 [I 2022-10-08 22:38:51] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00299 [I 2022-10-08 22:38:51] ...trap/engines/engine.py.230: loss: 0.20630 [I 2022-10-08 22:38:51] ...optimizers/trijoint.py.151: total_norm: 23.245489 [I 2022-10-08 22:38:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 1090/2304 [I 2022-10-08 22:38:59] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:17:00 [I 2022-10-08 22:38:59] ...trap/engines/engine.py.229: process: 0.84747 | load: 0.00301 [I 2022-10-08 22:38:59] ...trap/engines/engine.py.230: loss: 0.20623 [I 2022-10-08 22:38:59] ...optimizers/trijoint.py.151: total_norm: 18.833824 [I 2022-10-08 22:39:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 1100/2304 [I 2022-10-08 22:39:07] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:16:51 [I 2022-10-08 22:39:07] ...trap/engines/engine.py.229: process: 0.83728 | load: 0.00301 [I 2022-10-08 22:39:07] ...trap/engines/engine.py.230: loss: 0.17881 [I 2022-10-08 22:39:07] ...optimizers/trijoint.py.151: total_norm: 37.223785 [I 2022-10-08 22:39:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 1110/2304 [I 2022-10-08 22:39:16] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:32 [I 2022-10-08 22:39:16] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00298 [I 2022-10-08 22:39:16] ...trap/engines/engine.py.230: loss: 0.15697 [I 2022-10-08 22:39:16] ...optimizers/trijoint.py.151: total_norm: 18.831291 [I 2022-10-08 22:39:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 1120/2304 [I 2022-10-08 22:39:24] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:16:35 [I 2022-10-08 22:39:24] ...trap/engines/engine.py.229: process: 0.84163 | load: 0.00303 [I 2022-10-08 22:39:24] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-08 22:39:24] ...optimizers/trijoint.py.151: total_norm: 25.978127 [I 2022-10-08 22:39:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 1130/2304 [I 2022-10-08 22:39:33] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:27 [I 2022-10-08 22:39:33] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00299 [I 2022-10-08 22:39:33] ...trap/engines/engine.py.230: loss: 0.19278 [I 2022-10-08 22:39:33] ...optimizers/trijoint.py.151: total_norm: 12.206226 [I 2022-10-08 22:39:41] ...trap/engines/engine.py.224: train: epoch 22 | batch 1140/2304 [I 2022-10-08 22:39:41] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:14 [I 2022-10-08 22:39:41] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00298 [I 2022-10-08 22:39:41] ...trap/engines/engine.py.230: loss: 0.17303 [I 2022-10-08 22:39:41] ...optimizers/trijoint.py.151: total_norm: 24.884155 [I 2022-10-08 22:39:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 1150/2304 [I 2022-10-08 22:39:49] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:04 [I 2022-10-08 22:39:49] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00304 [I 2022-10-08 22:39:49] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-08 22:39:49] ...optimizers/trijoint.py.151: total_norm: 13.066212 [I 2022-10-08 22:39:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 1160/2304 [I 2022-10-08 22:39:58] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:15:58 [I 2022-10-08 22:39:58] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00301 [I 2022-10-08 22:39:58] ...trap/engines/engine.py.230: loss: 0.13388 [I 2022-10-08 22:39:58] ...optimizers/trijoint.py.151: total_norm: 17.081408 [I 2022-10-08 22:40:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 1170/2304 [I 2022-10-08 22:40:06] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:15:50 [I 2022-10-08 22:40:06] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00303 [I 2022-10-08 22:40:06] ...trap/engines/engine.py.230: loss: 0.24345 [I 2022-10-08 22:40:06] ...optimizers/trijoint.py.151: total_norm: 23.644651 [I 2022-10-08 22:40:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 1180/2304 [I 2022-10-08 22:40:15] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:15:41 [I 2022-10-08 22:40:15] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00304 [I 2022-10-08 22:40:15] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-08 22:40:15] ...optimizers/trijoint.py.151: total_norm: 21.417723 [I 2022-10-08 22:40:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 1190/2304 [I 2022-10-08 22:40:23] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:33 [I 2022-10-08 22:40:23] ...trap/engines/engine.py.229: process: 0.82859 | load: 0.00298 [I 2022-10-08 22:40:23] ...trap/engines/engine.py.230: loss: 0.12849 [I 2022-10-08 22:40:23] ...optimizers/trijoint.py.151: total_norm: 15.491798 [I 2022-10-08 22:40:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 1200/2304 [I 2022-10-08 22:40:32] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:24 [I 2022-10-08 22:40:32] ...trap/engines/engine.py.229: process: 0.83353 | load: 0.00305 [I 2022-10-08 22:40:32] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-08 22:40:32] ...optimizers/trijoint.py.151: total_norm: 21.034853 [I 2022-10-08 22:40:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 1210/2304 [I 2022-10-08 22:40:40] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:16 [I 2022-10-08 22:40:40] ...trap/engines/engine.py.229: process: 0.84228 | load: 0.00300 [I 2022-10-08 22:40:40] ...trap/engines/engine.py.230: loss: 0.13861 [I 2022-10-08 22:40:40] ...optimizers/trijoint.py.151: total_norm: 13.881036 [I 2022-10-08 22:40:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1220/2304 [I 2022-10-08 22:40:48] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:08 [I 2022-10-08 22:40:48] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00302 [I 2022-10-08 22:40:48] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-08 22:40:48] ...optimizers/trijoint.py.151: total_norm: 22.054697 [I 2022-10-08 22:40:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 1230/2304 [I 2022-10-08 22:40:57] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:02 [I 2022-10-08 22:40:57] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00300 [I 2022-10-08 22:40:57] ...trap/engines/engine.py.230: loss: 0.24045 [I 2022-10-08 22:40:57] ...optimizers/trijoint.py.151: total_norm: 13.869372 [I 2022-10-08 22:41:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1240/2304 [I 2022-10-08 22:41:05] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:14:55 [I 2022-10-08 22:41:05] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00304 [I 2022-10-08 22:41:05] ...trap/engines/engine.py.230: loss: 0.16673 [I 2022-10-08 22:41:05] ...optimizers/trijoint.py.151: total_norm: 22.026743 [I 2022-10-08 22:41:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 1250/2304 [I 2022-10-08 22:41:14] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:51 [I 2022-10-08 22:41:14] ...trap/engines/engine.py.229: process: 0.86043 | load: 0.00304 [I 2022-10-08 22:41:14] ...trap/engines/engine.py.230: loss: 0.23607 [I 2022-10-08 22:41:14] ...optimizers/trijoint.py.151: total_norm: 20.440834 [I 2022-10-08 22:41:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 1260/2304 [I 2022-10-08 22:41:22] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:40 [I 2022-10-08 22:41:22] ...trap/engines/engine.py.229: process: 0.84045 | load: 0.00301 [I 2022-10-08 22:41:22] ...trap/engines/engine.py.230: loss: 0.18258 [I 2022-10-08 22:41:22] ...optimizers/trijoint.py.151: total_norm: 20.059820 [I 2022-10-08 22:41:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 1270/2304 [I 2022-10-08 22:41:31] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:33 [I 2022-10-08 22:41:31] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00301 [I 2022-10-08 22:41:31] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-08 22:41:31] ...optimizers/trijoint.py.151: total_norm: 22.613392 [I 2022-10-08 22:41:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 1280/2304 [I 2022-10-08 22:41:39] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:27 [I 2022-10-08 22:41:39] ...trap/engines/engine.py.229: process: 0.85740 | load: 0.00298 [I 2022-10-08 22:41:39] ...trap/engines/engine.py.230: loss: 0.17029 [I 2022-10-08 22:41:39] ...optimizers/trijoint.py.151: total_norm: 42.156960 [I 2022-10-08 22:41:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1290/2304 [I 2022-10-08 22:41:48] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:18 [I 2022-10-08 22:41:48] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00301 [I 2022-10-08 22:41:48] ...trap/engines/engine.py.230: loss: 0.21764 [I 2022-10-08 22:41:48] ...optimizers/trijoint.py.151: total_norm: 33.158108 [I 2022-10-08 22:41:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 1300/2304 [I 2022-10-08 22:41:56] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:10 [I 2022-10-08 22:41:56] ...trap/engines/engine.py.229: process: 0.85879 | load: 0.00302 [I 2022-10-08 22:41:56] ...trap/engines/engine.py.230: loss: 0.23098 [I 2022-10-08 22:41:56] ...optimizers/trijoint.py.151: total_norm: 18.236710 [I 2022-10-08 22:42:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1310/2304 [I 2022-10-08 22:42:05] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:02 [I 2022-10-08 22:42:05] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00301 [I 2022-10-08 22:42:05] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-08 22:42:05] ...optimizers/trijoint.py.151: total_norm: 13.585553 [I 2022-10-08 22:42:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 1320/2304 [I 2022-10-08 22:42:13] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:48 [I 2022-10-08 22:42:13] ...trap/engines/engine.py.229: process: 0.83997 | load: 0.00298 [I 2022-10-08 22:42:13] ...trap/engines/engine.py.230: loss: 0.15004 [I 2022-10-08 22:42:13] ...optimizers/trijoint.py.151: total_norm: 20.024929 [I 2022-10-08 22:42:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 1330/2304 [I 2022-10-08 22:42:21] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:13:30 [I 2022-10-08 22:42:21] ...trap/engines/engine.py.229: process: 0.83467 | load: 0.00297 [I 2022-10-08 22:42:21] ...trap/engines/engine.py.230: loss: 0.18364 [I 2022-10-08 22:42:21] ...optimizers/trijoint.py.151: total_norm: 18.322227 [I 2022-10-08 22:42:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 1340/2304 [I 2022-10-08 22:42:30] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:24 [I 2022-10-08 22:42:30] ...trap/engines/engine.py.229: process: 0.83028 | load: 0.00303 [I 2022-10-08 22:42:30] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-08 22:42:30] ...optimizers/trijoint.py.151: total_norm: 19.291399 [I 2022-10-08 22:42:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 1350/2304 [I 2022-10-08 22:42:38] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:22 [I 2022-10-08 22:42:38] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00302 [I 2022-10-08 22:42:38] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-08 22:42:38] ...optimizers/trijoint.py.151: total_norm: 14.878034 [I 2022-10-08 22:42:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 1360/2304 [I 2022-10-08 22:42:46] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:12 [I 2022-10-08 22:42:46] ...trap/engines/engine.py.229: process: 0.83531 | load: 0.00298 [I 2022-10-08 22:42:46] ...trap/engines/engine.py.230: loss: 0.20589 [I 2022-10-08 22:42:46] ...optimizers/trijoint.py.151: total_norm: 22.785866 [I 2022-10-08 22:42:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 1370/2304 [I 2022-10-08 22:42:55] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:01 [I 2022-10-08 22:42:55] ...trap/engines/engine.py.229: process: 0.82334 | load: 0.00296 [I 2022-10-08 22:42:55] ...trap/engines/engine.py.230: loss: 0.20615 [I 2022-10-08 22:42:55] ...optimizers/trijoint.py.151: total_norm: 25.710623 [I 2022-10-08 22:43:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 1380/2304 [I 2022-10-08 22:43:03] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:12:55 [I 2022-10-08 22:43:03] ...trap/engines/engine.py.229: process: 0.83879 | load: 0.00300 [I 2022-10-08 22:43:03] ...trap/engines/engine.py.230: loss: 0.24997 [I 2022-10-08 22:43:03] ...optimizers/trijoint.py.151: total_norm: 22.377712 [I 2022-10-08 22:43:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 1390/2304 [I 2022-10-08 22:43:12] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:12:49 [I 2022-10-08 22:43:12] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00299 [I 2022-10-08 22:43:12] ...trap/engines/engine.py.230: loss: 0.18646 [I 2022-10-08 22:43:12] ...optimizers/trijoint.py.151: total_norm: 25.212648 [I 2022-10-08 22:43:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 1400/2304 [I 2022-10-08 22:43:20] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:38 [I 2022-10-08 22:43:20] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00303 [I 2022-10-08 22:43:20] ...trap/engines/engine.py.230: loss: 0.13899 [I 2022-10-08 22:43:20] ...optimizers/trijoint.py.151: total_norm: 10.663086 [I 2022-10-08 22:43:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 1410/2304 [I 2022-10-08 22:43:29] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:37 [I 2022-10-08 22:43:29] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00309 [I 2022-10-08 22:43:29] ...trap/engines/engine.py.230: loss: 0.20841 [I 2022-10-08 22:43:29] ...optimizers/trijoint.py.151: total_norm: 22.404833 [I 2022-10-08 22:43:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 1420/2304 [I 2022-10-08 22:43:37] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:24 [I 2022-10-08 22:43:37] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00332 [I 2022-10-08 22:43:37] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-08 22:43:37] ...optimizers/trijoint.py.151: total_norm: 10.190498 [I 2022-10-08 22:43:45] ...trap/engines/engine.py.224: train: epoch 22 | batch 1430/2304 [I 2022-10-08 22:43:45] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:12 [I 2022-10-08 22:43:45] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00300 [I 2022-10-08 22:43:45] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-08 22:43:45] ...optimizers/trijoint.py.151: total_norm: 29.905491 [I 2022-10-08 22:43:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 1440/2304 [I 2022-10-08 22:43:54] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:02 [I 2022-10-08 22:43:54] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00303 [I 2022-10-08 22:43:54] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-08 22:43:54] ...optimizers/trijoint.py.151: total_norm: 14.239955 [I 2022-10-08 22:44:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 1450/2304 [I 2022-10-08 22:44:02] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:11:55 [I 2022-10-08 22:44:02] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00298 [I 2022-10-08 22:44:02] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-08 22:44:02] ...optimizers/trijoint.py.151: total_norm: 19.806585 [I 2022-10-08 22:44:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 1460/2304 [I 2022-10-08 22:44:11] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:46 [I 2022-10-08 22:44:11] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00299 [I 2022-10-08 22:44:11] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-08 22:44:11] ...optimizers/trijoint.py.151: total_norm: 13.362624 [I 2022-10-08 22:44:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 1470/2304 [I 2022-10-08 22:44:19] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:39 [I 2022-10-08 22:44:19] ...trap/engines/engine.py.229: process: 0.83690 | load: 0.00298 [I 2022-10-08 22:44:19] ...trap/engines/engine.py.230: loss: 0.21164 [I 2022-10-08 22:44:19] ...optimizers/trijoint.py.151: total_norm: 20.026829 [I 2022-10-08 22:44:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 1480/2304 [I 2022-10-08 22:44:27] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:29 [I 2022-10-08 22:44:27] ...trap/engines/engine.py.229: process: 0.84483 | load: 0.00300 [I 2022-10-08 22:44:27] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-08 22:44:27] ...optimizers/trijoint.py.151: total_norm: 20.723110 [I 2022-10-08 22:44:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 1490/2304 [I 2022-10-08 22:44:36] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:20 [I 2022-10-08 22:44:36] ...trap/engines/engine.py.229: process: 0.82489 | load: 0.00305 [I 2022-10-08 22:44:36] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-08 22:44:36] ...optimizers/trijoint.py.151: total_norm: 13.443764 [I 2022-10-08 22:44:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 1500/2304 [I 2022-10-08 22:44:44] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:14 [I 2022-10-08 22:44:44] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00302 [I 2022-10-08 22:44:44] ...trap/engines/engine.py.230: loss: 0.19002 [I 2022-10-08 22:44:44] ...optimizers/trijoint.py.151: total_norm: 80.931030 [I 2022-10-08 22:44:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 1510/2304 [I 2022-10-08 22:44:53] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:05 [I 2022-10-08 22:44:53] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00302 [I 2022-10-08 22:44:53] ...trap/engines/engine.py.230: loss: 0.12442 [I 2022-10-08 22:44:53] ...optimizers/trijoint.py.151: total_norm: 12.602507 [I 2022-10-08 22:45:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 1520/2304 [I 2022-10-08 22:45:01] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:10:55 [I 2022-10-08 22:45:01] ...trap/engines/engine.py.229: process: 0.82750 | load: 0.00299 [I 2022-10-08 22:45:01] ...trap/engines/engine.py.230: loss: 0.23495 [I 2022-10-08 22:45:01] ...optimizers/trijoint.py.151: total_norm: 20.597168 [I 2022-10-08 22:45:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 1530/2304 [I 2022-10-08 22:45:09] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:10:46 [I 2022-10-08 22:45:09] ...trap/engines/engine.py.229: process: 0.82736 | load: 0.00300 [I 2022-10-08 22:45:09] ...trap/engines/engine.py.230: loss: 0.13634 [I 2022-10-08 22:45:09] ...optimizers/trijoint.py.151: total_norm: 17.152943 [I 2022-10-08 22:45:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 1540/2304 [I 2022-10-08 22:45:18] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:37 [I 2022-10-08 22:45:18] ...trap/engines/engine.py.229: process: 0.81322 | load: 0.00307 [I 2022-10-08 22:45:18] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-08 22:45:18] ...optimizers/trijoint.py.151: total_norm: 13.004393 [I 2022-10-08 22:45:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 1550/2304 [I 2022-10-08 22:45:26] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:29 [I 2022-10-08 22:45:26] ...trap/engines/engine.py.229: process: 0.82402 | load: 0.00301 [I 2022-10-08 22:45:26] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-08 22:45:26] ...optimizers/trijoint.py.151: total_norm: 21.562403 [I 2022-10-08 22:45:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 1560/2304 [I 2022-10-08 22:45:34] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:22 [I 2022-10-08 22:45:34] ...trap/engines/engine.py.229: process: 0.83964 | load: 0.00300 [I 2022-10-08 22:45:34] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-08 22:45:34] ...optimizers/trijoint.py.151: total_norm: 13.181610 [I 2022-10-08 22:45:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 1570/2304 [I 2022-10-08 22:45:43] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:16 [I 2022-10-08 22:45:43] ...trap/engines/engine.py.229: process: 0.85160 | load: 0.00299 [I 2022-10-08 22:45:43] ...trap/engines/engine.py.230: loss: 0.20090 [I 2022-10-08 22:45:43] ...optimizers/trijoint.py.151: total_norm: 23.683378 [I 2022-10-08 22:45:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 1580/2304 [I 2022-10-08 22:45:51] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:10 [I 2022-10-08 22:45:51] ...trap/engines/engine.py.229: process: 0.83397 | load: 0.00301 [I 2022-10-08 22:45:51] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-08 22:45:51] ...optimizers/trijoint.py.151: total_norm: 24.913204 [I 2022-10-08 22:46:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 1590/2304 [I 2022-10-08 22:46:00] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:09:58 [I 2022-10-08 22:46:00] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00302 [I 2022-10-08 22:46:00] ...trap/engines/engine.py.230: loss: 0.23789 [I 2022-10-08 22:46:00] ...optimizers/trijoint.py.151: total_norm: 32.636517 [I 2022-10-08 22:46:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 1600/2304 [I 2022-10-08 22:46:08] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:49 [I 2022-10-08 22:46:08] ...trap/engines/engine.py.229: process: 0.83398 | load: 0.00301 [I 2022-10-08 22:46:08] ...trap/engines/engine.py.230: loss: 0.29196 [I 2022-10-08 22:46:08] ...optimizers/trijoint.py.151: total_norm: 33.886356 [I 2022-10-08 22:46:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 1610/2304 [I 2022-10-08 22:46:16] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:39 [I 2022-10-08 22:46:16] ...trap/engines/engine.py.229: process: 0.83596 | load: 0.00306 [I 2022-10-08 22:46:16] ...trap/engines/engine.py.230: loss: 0.20208 [I 2022-10-08 22:46:16] ...optimizers/trijoint.py.151: total_norm: 17.884857 [I 2022-10-08 22:46:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 1620/2304 [I 2022-10-08 22:46:25] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:32 [I 2022-10-08 22:46:25] ...trap/engines/engine.py.229: process: 0.80729 | load: 0.00299 [I 2022-10-08 22:46:25] ...trap/engines/engine.py.230: loss: 0.21907 [I 2022-10-08 22:46:25] ...optimizers/trijoint.py.151: total_norm: 18.954897 [I 2022-10-08 22:46:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 1630/2304 [I 2022-10-08 22:46:33] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:24 [I 2022-10-08 22:46:33] ...trap/engines/engine.py.229: process: 0.82654 | load: 0.00297 [I 2022-10-08 22:46:33] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-08 22:46:33] ...optimizers/trijoint.py.151: total_norm: 9.917595 [I 2022-10-08 22:46:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 1640/2304 [I 2022-10-08 22:46:42] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:15 [I 2022-10-08 22:46:42] ...trap/engines/engine.py.229: process: 0.83352 | load: 0.00302 [I 2022-10-08 22:46:42] ...trap/engines/engine.py.230: loss: 0.17920 [I 2022-10-08 22:46:42] ...optimizers/trijoint.py.151: total_norm: 21.012348 [I 2022-10-08 22:46:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 1650/2304 [I 2022-10-08 22:46:50] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:05 [I 2022-10-08 22:46:50] ...trap/engines/engine.py.229: process: 0.84038 | load: 0.00300 [I 2022-10-08 22:46:50] ...trap/engines/engine.py.230: loss: 0.22037 [I 2022-10-08 22:46:50] ...optimizers/trijoint.py.151: total_norm: 21.910664 [I 2022-10-08 22:46:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 1660/2304 [I 2022-10-08 22:46:58] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:08:58 [I 2022-10-08 22:46:58] ...trap/engines/engine.py.229: process: 0.84007 | load: 0.00306 [I 2022-10-08 22:46:58] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-08 22:46:58] ...optimizers/trijoint.py.151: total_norm: 19.522068 [I 2022-10-08 22:47:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 1670/2304 [I 2022-10-08 22:47:07] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:53 [I 2022-10-08 22:47:07] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00298 [I 2022-10-08 22:47:07] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-08 22:47:07] ...optimizers/trijoint.py.151: total_norm: 20.336174 [I 2022-10-08 22:47:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 1680/2304 [I 2022-10-08 22:47:15] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:46 [I 2022-10-08 22:47:15] ...trap/engines/engine.py.229: process: 0.85569 | load: 0.00301 [I 2022-10-08 22:47:15] ...trap/engines/engine.py.230: loss: 0.27191 [I 2022-10-08 22:47:15] ...optimizers/trijoint.py.151: total_norm: 49.107510 [I 2022-10-08 22:47:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 1690/2304 [I 2022-10-08 22:47:24] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:35 [I 2022-10-08 22:47:24] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00305 [I 2022-10-08 22:47:24] ...trap/engines/engine.py.230: loss: 0.19408 [I 2022-10-08 22:47:24] ...optimizers/trijoint.py.151: total_norm: 15.304971 [I 2022-10-08 22:47:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 1700/2304 [I 2022-10-08 22:47:32] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:26 [I 2022-10-08 22:47:32] ...trap/engines/engine.py.229: process: 0.82863 | load: 0.00300 [I 2022-10-08 22:47:32] ...trap/engines/engine.py.230: loss: 0.18878 [I 2022-10-08 22:47:32] ...optimizers/trijoint.py.151: total_norm: 11.251135 [I 2022-10-08 22:47:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 1710/2304 [I 2022-10-08 22:47:40] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:18 [I 2022-10-08 22:47:40] ...trap/engines/engine.py.229: process: 0.82757 | load: 0.00299 [I 2022-10-08 22:47:40] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-08 22:47:40] ...optimizers/trijoint.py.151: total_norm: 25.271778 [I 2022-10-08 22:47:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 1720/2304 [I 2022-10-08 22:47:49] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:12 [I 2022-10-08 22:47:49] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00301 [I 2022-10-08 22:47:49] ...trap/engines/engine.py.230: loss: 0.25514 [I 2022-10-08 22:47:49] ...optimizers/trijoint.py.151: total_norm: 21.918722 [I 2022-10-08 22:47:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 1730/2304 [I 2022-10-08 22:47:57] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:03 [I 2022-10-08 22:47:57] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00300 [I 2022-10-08 22:47:57] ...trap/engines/engine.py.230: loss: 0.15186 [I 2022-10-08 22:47:57] ...optimizers/trijoint.py.151: total_norm: 14.348903 [I 2022-10-08 22:48:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 1740/2304 [I 2022-10-08 22:48:06] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:53 [I 2022-10-08 22:48:06] ...trap/engines/engine.py.229: process: 0.83089 | load: 0.00296 [I 2022-10-08 22:48:06] ...trap/engines/engine.py.230: loss: 0.17135 [I 2022-10-08 22:48:06] ...optimizers/trijoint.py.151: total_norm: 17.373592 [I 2022-10-08 22:48:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 1750/2304 [I 2022-10-08 22:48:14] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:45 [I 2022-10-08 22:48:14] ...trap/engines/engine.py.229: process: 0.83052 | load: 0.00300 [I 2022-10-08 22:48:14] ...trap/engines/engine.py.230: loss: 0.22071 [I 2022-10-08 22:48:14] ...optimizers/trijoint.py.151: total_norm: 31.616236 [I 2022-10-08 22:48:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 1760/2304 [I 2022-10-08 22:48:23] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:38 [I 2022-10-08 22:48:23] ...trap/engines/engine.py.229: process: 0.83857 | load: 0.00302 [I 2022-10-08 22:48:23] ...trap/engines/engine.py.230: loss: 0.15019 [I 2022-10-08 22:48:23] ...optimizers/trijoint.py.151: total_norm: 16.554768 [I 2022-10-08 22:48:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 1770/2304 [I 2022-10-08 22:48:31] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:28 [I 2022-10-08 22:48:31] ...trap/engines/engine.py.229: process: 0.83256 | load: 0.00304 [I 2022-10-08 22:48:31] ...trap/engines/engine.py.230: loss: 0.34317 [I 2022-10-08 22:48:31] ...optimizers/trijoint.py.151: total_norm: 36.692421 [I 2022-10-08 22:48:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 1780/2304 [I 2022-10-08 22:48:39] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:21 [I 2022-10-08 22:48:39] ...trap/engines/engine.py.229: process: 0.84091 | load: 0.00302 [I 2022-10-08 22:48:39] ...trap/engines/engine.py.230: loss: 0.18138 [I 2022-10-08 22:48:39] ...optimizers/trijoint.py.151: total_norm: 25.659182 [I 2022-10-08 22:48:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1790/2304 [I 2022-10-08 22:48:48] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:13 [I 2022-10-08 22:48:48] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00299 [I 2022-10-08 22:48:48] ...trap/engines/engine.py.230: loss: 0.22911 [I 2022-10-08 22:48:48] ...optimizers/trijoint.py.151: total_norm: 27.782175 [I 2022-10-08 22:48:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 1800/2304 [I 2022-10-08 22:48:56] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:05 [I 2022-10-08 22:48:56] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00301 [I 2022-10-08 22:48:56] ...trap/engines/engine.py.230: loss: 0.16809 [I 2022-10-08 22:48:56] ...optimizers/trijoint.py.151: total_norm: 14.792300 [I 2022-10-08 22:49:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1810/2304 [I 2022-10-08 22:49:05] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:54 [I 2022-10-08 22:49:05] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00302 [I 2022-10-08 22:49:05] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-08 22:49:05] ...optimizers/trijoint.py.151: total_norm: 14.840383 [I 2022-10-08 22:49:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 1820/2304 [I 2022-10-08 22:49:13] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:45 [I 2022-10-08 22:49:13] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00300 [I 2022-10-08 22:49:13] ...trap/engines/engine.py.230: loss: 0.19237 [I 2022-10-08 22:49:13] ...optimizers/trijoint.py.151: total_norm: 21.695959 [I 2022-10-08 22:49:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 1830/2304 [I 2022-10-08 22:49:21] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:35 [I 2022-10-08 22:49:21] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00303 [I 2022-10-08 22:49:21] ...trap/engines/engine.py.230: loss: 0.18163 [I 2022-10-08 22:49:21] ...optimizers/trijoint.py.151: total_norm: 13.174815 [I 2022-10-08 22:49:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 1840/2304 [I 2022-10-08 22:49:30] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:30 [I 2022-10-08 22:49:30] ...trap/engines/engine.py.229: process: 0.84711 | load: 0.00305 [I 2022-10-08 22:49:30] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-08 22:49:30] ...optimizers/trijoint.py.151: total_norm: 11.784066 [I 2022-10-08 22:49:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 1850/2304 [I 2022-10-08 22:49:38] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:21 [I 2022-10-08 22:49:38] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00299 [I 2022-10-08 22:49:38] ...trap/engines/engine.py.230: loss: 0.18394 [I 2022-10-08 22:49:38] ...optimizers/trijoint.py.151: total_norm: 16.543432 [I 2022-10-08 22:49:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 1860/2304 [I 2022-10-08 22:49:47] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:11 [I 2022-10-08 22:49:47] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00302 [I 2022-10-08 22:49:47] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-08 22:49:47] ...optimizers/trijoint.py.151: total_norm: 19.921978 [I 2022-10-08 22:49:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 1870/2304 [I 2022-10-08 22:49:55] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:05 [I 2022-10-08 22:49:55] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00299 [I 2022-10-08 22:49:55] ...trap/engines/engine.py.230: loss: 0.19146 [I 2022-10-08 22:49:55] ...optimizers/trijoint.py.151: total_norm: 17.769672 [I 2022-10-08 22:50:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 1880/2304 [I 2022-10-08 22:50:04] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:57 [I 2022-10-08 22:50:04] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00301 [I 2022-10-08 22:50:04] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-08 22:50:04] ...optimizers/trijoint.py.151: total_norm: 16.303036 [I 2022-10-08 22:50:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 1890/2304 [I 2022-10-08 22:50:12] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:47 [I 2022-10-08 22:50:12] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00304 [I 2022-10-08 22:50:12] ...trap/engines/engine.py.230: loss: 0.22852 [I 2022-10-08 22:50:12] ...optimizers/trijoint.py.151: total_norm: 27.615351 [I 2022-10-08 22:50:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 1900/2304 [I 2022-10-08 22:50:20] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:36 [I 2022-10-08 22:50:20] ...trap/engines/engine.py.229: process: 0.83424 | load: 0.00300 [I 2022-10-08 22:50:20] ...trap/engines/engine.py.230: loss: 0.14896 [I 2022-10-08 22:50:20] ...optimizers/trijoint.py.151: total_norm: 21.965267 [I 2022-10-08 22:50:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 1910/2304 [I 2022-10-08 22:50:29] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:30 [I 2022-10-08 22:50:29] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00299 [I 2022-10-08 22:50:29] ...trap/engines/engine.py.230: loss: 0.13500 [I 2022-10-08 22:50:29] ...optimizers/trijoint.py.151: total_norm: 11.685375 [I 2022-10-08 22:50:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 1920/2304 [I 2022-10-08 22:50:37] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:23 [I 2022-10-08 22:50:37] ...trap/engines/engine.py.229: process: 0.83657 | load: 0.00304 [I 2022-10-08 22:50:37] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-08 22:50:37] ...optimizers/trijoint.py.151: total_norm: 22.193308 [I 2022-10-08 22:50:45] ...trap/engines/engine.py.224: train: epoch 22 | batch 1930/2304 [I 2022-10-08 22:50:45] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:12 [I 2022-10-08 22:50:45] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00300 [I 2022-10-08 22:50:45] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-08 22:50:46] ...optimizers/trijoint.py.151: total_norm: 10.961528 [I 2022-10-08 22:50:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 1940/2304 [I 2022-10-08 22:50:54] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:03 [I 2022-10-08 22:50:54] ...trap/engines/engine.py.229: process: 0.81734 | load: 0.00300 [I 2022-10-08 22:50:54] ...trap/engines/engine.py.230: loss: 0.17609 [I 2022-10-08 22:50:54] ...optimizers/trijoint.py.151: total_norm: 24.401670 [I 2022-10-08 22:51:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 1950/2304 [I 2022-10-08 22:51:02] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:04:56 [I 2022-10-08 22:51:02] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00305 [I 2022-10-08 22:51:02] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-08 22:51:02] ...optimizers/trijoint.py.151: total_norm: 15.878592 [I 2022-10-08 22:51:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 1960/2304 [I 2022-10-08 22:51:11] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:47 [I 2022-10-08 22:51:11] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00301 [I 2022-10-08 22:51:11] ...trap/engines/engine.py.230: loss: 0.19617 [I 2022-10-08 22:51:11] ...optimizers/trijoint.py.151: total_norm: 14.112172 [I 2022-10-08 22:51:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 1970/2304 [I 2022-10-08 22:51:19] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:04:40 [I 2022-10-08 22:51:19] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00304 [I 2022-10-08 22:51:19] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-08 22:51:19] ...optimizers/trijoint.py.151: total_norm: 17.202288 [I 2022-10-08 22:51:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 1980/2304 [I 2022-10-08 22:51:27] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:31 [I 2022-10-08 22:51:27] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00304 [I 2022-10-08 22:51:27] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-08 22:51:27] ...optimizers/trijoint.py.151: total_norm: 20.752161 [I 2022-10-08 22:51:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 1990/2304 [I 2022-10-08 22:51:36] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:22 [I 2022-10-08 22:51:36] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00302 [I 2022-10-08 22:51:36] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-08 22:51:36] ...optimizers/trijoint.py.151: total_norm: 15.144189 [I 2022-10-08 22:51:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 2000/2304 [I 2022-10-08 22:51:44] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:13 [I 2022-10-08 22:51:44] ...trap/engines/engine.py.229: process: 0.83746 | load: 0.00301 [I 2022-10-08 22:51:44] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-08 22:51:44] ...optimizers/trijoint.py.151: total_norm: 17.648094 [I 2022-10-08 22:51:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 2010/2304 [I 2022-10-08 22:51:53] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:05 [I 2022-10-08 22:51:53] ...trap/engines/engine.py.229: process: 0.84031 | load: 0.00302 [I 2022-10-08 22:51:53] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-08 22:51:53] ...optimizers/trijoint.py.151: total_norm: 33.272984 [I 2022-10-08 22:52:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 2020/2304 [I 2022-10-08 22:52:01] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:03:57 [I 2022-10-08 22:52:01] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00299 [I 2022-10-08 22:52:01] ...trap/engines/engine.py.230: loss: 0.31631 [I 2022-10-08 22:52:01] ...optimizers/trijoint.py.151: total_norm: 30.693792 [I 2022-10-08 22:52:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 2030/2304 [I 2022-10-08 22:52:09] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:03:49 [I 2022-10-08 22:52:09] ...trap/engines/engine.py.229: process: 0.84854 | load: 0.00309 [I 2022-10-08 22:52:09] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-08 22:52:09] ...optimizers/trijoint.py.151: total_norm: 20.221371 [I 2022-10-08 22:52:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 2040/2304 [I 2022-10-08 22:52:18] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:03:40 [I 2022-10-08 22:52:18] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00310 [I 2022-10-08 22:52:18] ...trap/engines/engine.py.230: loss: 0.22344 [I 2022-10-08 22:52:18] ...optimizers/trijoint.py.151: total_norm: 27.849962 [I 2022-10-08 22:52:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 2050/2304 [I 2022-10-08 22:52:26] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:03:31 [I 2022-10-08 22:52:26] ...trap/engines/engine.py.229: process: 0.81560 | load: 0.00302 [I 2022-10-08 22:52:26] ...trap/engines/engine.py.230: loss: 0.10804 [I 2022-10-08 22:52:26] ...optimizers/trijoint.py.151: total_norm: 17.363413 [I 2022-10-08 22:52:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 2060/2304 [I 2022-10-08 22:52:34] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:23 [I 2022-10-08 22:52:34] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00302 [I 2022-10-08 22:52:34] ...trap/engines/engine.py.230: loss: 0.16441 [I 2022-10-08 22:52:34] ...optimizers/trijoint.py.151: total_norm: 13.865384 [I 2022-10-08 22:52:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 2070/2304 [I 2022-10-08 22:52:43] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:15 [I 2022-10-08 22:52:43] ...trap/engines/engine.py.229: process: 0.83607 | load: 0.00301 [I 2022-10-08 22:52:43] ...trap/engines/engine.py.230: loss: 0.20180 [I 2022-10-08 22:52:43] ...optimizers/trijoint.py.151: total_norm: 18.814133 [I 2022-10-08 22:52:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 2080/2304 [I 2022-10-08 22:52:51] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:07 [I 2022-10-08 22:52:51] ...trap/engines/engine.py.229: process: 0.82680 | load: 0.00312 [I 2022-10-08 22:52:51] ...trap/engines/engine.py.230: loss: 0.25350 [I 2022-10-08 22:52:51] ...optimizers/trijoint.py.151: total_norm: 26.899361 [I 2022-10-08 22:52:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 2090/2304 [I 2022-10-08 22:52:59] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:02:57 [I 2022-10-08 22:52:59] ...trap/engines/engine.py.229: process: 0.83113 | load: 0.00300 [I 2022-10-08 22:52:59] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-08 22:52:59] ...optimizers/trijoint.py.151: total_norm: 19.297382 [I 2022-10-08 22:53:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 2100/2304 [I 2022-10-08 22:53:08] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:02:50 [I 2022-10-08 22:53:08] ...trap/engines/engine.py.229: process: 0.84731 | load: 0.00300 [I 2022-10-08 22:53:08] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-08 22:53:08] ...optimizers/trijoint.py.151: total_norm: 9.987860 [I 2022-10-08 22:53:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 2110/2304 [I 2022-10-08 22:53:16] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:02:40 [I 2022-10-08 22:53:16] ...trap/engines/engine.py.229: process: 0.82104 | load: 0.00300 [I 2022-10-08 22:53:16] ...trap/engines/engine.py.230: loss: 0.21547 [I 2022-10-08 22:53:16] ...optimizers/trijoint.py.151: total_norm: 61.368973 [I 2022-10-08 22:53:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 2120/2304 [I 2022-10-08 22:53:24] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:02:33 [I 2022-10-08 22:53:24] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00298 [I 2022-10-08 22:53:24] ...trap/engines/engine.py.230: loss: 0.16975 [I 2022-10-08 22:53:24] ...optimizers/trijoint.py.151: total_norm: 16.894377 [I 2022-10-08 22:53:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 2130/2304 [I 2022-10-08 22:53:33] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:25 [I 2022-10-08 22:53:33] ...trap/engines/engine.py.229: process: 0.83527 | load: 0.00301 [I 2022-10-08 22:53:33] ...trap/engines/engine.py.230: loss: 0.17425 [I 2022-10-08 22:53:33] ...optimizers/trijoint.py.151: total_norm: 23.571754 [I 2022-10-08 22:53:41] ...trap/engines/engine.py.224: train: epoch 22 | batch 2140/2304 [I 2022-10-08 22:53:41] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:17 [I 2022-10-08 22:53:41] ...trap/engines/engine.py.229: process: 0.84060 | load: 0.00302 [I 2022-10-08 22:53:41] ...trap/engines/engine.py.230: loss: 0.19848 [I 2022-10-08 22:53:41] ...optimizers/trijoint.py.151: total_norm: 17.595320 [I 2022-10-08 22:53:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 2150/2304 [I 2022-10-08 22:53:50] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:08 [I 2022-10-08 22:53:50] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00299 [I 2022-10-08 22:53:50] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-08 22:53:50] ...optimizers/trijoint.py.151: total_norm: 24.279610 [I 2022-10-08 22:53:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 2160/2304 [I 2022-10-08 22:53:58] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:00 [I 2022-10-08 22:53:58] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00300 [I 2022-10-08 22:53:58] ...trap/engines/engine.py.230: loss: 0.15522 [I 2022-10-08 22:53:58] ...optimizers/trijoint.py.151: total_norm: 18.134676 [I 2022-10-08 22:54:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 2170/2304 [I 2022-10-08 22:54:06] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:01:51 [I 2022-10-08 22:54:06] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00299 [I 2022-10-08 22:54:06] ...trap/engines/engine.py.230: loss: 0.21355 [I 2022-10-08 22:54:06] ...optimizers/trijoint.py.151: total_norm: 18.176748 [I 2022-10-08 22:54:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 2180/2304 [I 2022-10-08 22:54:15] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:01:43 [I 2022-10-08 22:54:15] ...trap/engines/engine.py.229: process: 0.83207 | load: 0.00297 [I 2022-10-08 22:54:15] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-08 22:54:15] ...optimizers/trijoint.py.151: total_norm: 19.762774 [I 2022-10-08 22:54:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 2190/2304 [I 2022-10-08 22:54:23] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:01:35 [I 2022-10-08 22:54:23] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00310 [I 2022-10-08 22:54:23] ...trap/engines/engine.py.230: loss: 0.27238 [I 2022-10-08 22:54:23] ...optimizers/trijoint.py.151: total_norm: 24.385849 [I 2022-10-08 22:54:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 2200/2304 [I 2022-10-08 22:54:31] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:01:26 [I 2022-10-08 22:54:31] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00298 [I 2022-10-08 22:54:31] ...trap/engines/engine.py.230: loss: 0.17941 [I 2022-10-08 22:54:31] ...optimizers/trijoint.py.151: total_norm: 41.667427 [I 2022-10-08 22:54:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 2210/2304 [I 2022-10-08 22:54:40] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:18 [I 2022-10-08 22:54:40] ...trap/engines/engine.py.229: process: 0.82600 | load: 0.00301 [I 2022-10-08 22:54:40] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-08 22:54:40] ...optimizers/trijoint.py.151: total_norm: 20.720821 [I 2022-10-08 22:54:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 2220/2304 [I 2022-10-08 22:54:48] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:10 [I 2022-10-08 22:54:48] ...trap/engines/engine.py.229: process: 0.83444 | load: 0.00298 [I 2022-10-08 22:54:48] ...trap/engines/engine.py.230: loss: 0.12449 [I 2022-10-08 22:54:48] ...optimizers/trijoint.py.151: total_norm: 18.093622 [I 2022-10-08 22:54:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 2230/2304 [I 2022-10-08 22:54:56] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:01 [I 2022-10-08 22:54:56] ...trap/engines/engine.py.229: process: 0.82240 | load: 0.00301 [I 2022-10-08 22:54:56] ...trap/engines/engine.py.230: loss: 0.20051 [I 2022-10-08 22:54:56] ...optimizers/trijoint.py.151: total_norm: 18.672855 [I 2022-10-08 22:55:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 2240/2304 [I 2022-10-08 22:55:05] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:00:53 [I 2022-10-08 22:55:05] ...trap/engines/engine.py.229: process: 0.82702 | load: 0.00309 [I 2022-10-08 22:55:05] ...trap/engines/engine.py.230: loss: 0.19220 [I 2022-10-08 22:55:05] ...optimizers/trijoint.py.151: total_norm: 33.858025 [I 2022-10-08 22:55:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 2250/2304 [I 2022-10-08 22:55:13] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:45 [I 2022-10-08 22:55:13] ...trap/engines/engine.py.229: process: 0.84030 | load: 0.00304 [I 2022-10-08 22:55:13] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-08 22:55:13] ...optimizers/trijoint.py.151: total_norm: 8.889459 [I 2022-10-08 22:55:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 2260/2304 [I 2022-10-08 22:55:22] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:00:36 [I 2022-10-08 22:55:22] ...trap/engines/engine.py.229: process: 0.81584 | load: 0.00304 [I 2022-10-08 22:55:22] ...trap/engines/engine.py.230: loss: 0.18004 [I 2022-10-08 22:55:22] ...optimizers/trijoint.py.151: total_norm: 16.903196 [I 2022-10-08 22:55:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 2270/2304 [I 2022-10-08 22:55:30] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:28 [I 2022-10-08 22:55:30] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00301 [I 2022-10-08 22:55:30] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-08 22:55:30] ...optimizers/trijoint.py.151: total_norm: 11.020417 [I 2022-10-08 22:55:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 2280/2304 [I 2022-10-08 22:55:38] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:19 [I 2022-10-08 22:55:38] ...trap/engines/engine.py.229: process: 0.83775 | load: 0.00300 [I 2022-10-08 22:55:38] ...trap/engines/engine.py.230: loss: 0.24929 [I 2022-10-08 22:55:38] ...optimizers/trijoint.py.151: total_norm: 15.741677 [I 2022-10-08 22:55:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 2290/2304 [I 2022-10-08 22:55:47] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:11 [I 2022-10-08 22:55:47] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00302 [I 2022-10-08 22:55:47] ...trap/engines/engine.py.230: loss: 0.20219 [I 2022-10-08 22:55:47] ...optimizers/trijoint.py.151: total_norm: 23.079599 [I 2022-10-08 22:55:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 2300/2304 [I 2022-10-08 22:55:55] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:03 [I 2022-10-08 22:55:55] ...trap/engines/engine.py.229: process: 0.83349 | load: 0.00018 [I 2022-10-08 22:55:55] ...trap/engines/engine.py.230: loss: 0.21818 [I 2022-10-08 22:55:55] ...optimizers/trijoint.py.151: total_norm: 28.669983 [I 2022-10-08 22:55:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 2304/2304 [I 2022-10-08 22:55:58] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:00 [I 2022-10-08 22:55:58] ...trap/engines/engine.py.229: process: 0.82630 | load: 0.00017 [I 2022-10-08 22:55:58] ...trap/engines/engine.py.230: loss: 0.15252 [I 2022-10-08 22:55:58] ...optimizers/trijoint.py.151: total_norm: 14.124230 [S 2022-10-08 22:56:00] ...trap/engines/engine.py.129: train_epoch.epoch: 22 [S 2022-10-08 22:56:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.17824585357935466 [I 2022-10-08 22:56:00] ...s/criterions/triplet.py.84: margin = 0.1650000000000001 [I 2022-10-08 22:56:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 22 [I 2022-10-08 22:56:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 22:56:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 22:56:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 22:56:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 22:56:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 22:56:03] ...trap/engines/engine.py.321: eval: epoch 22 | batch 0/510 [I 2022-10-08 22:56:03] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:26:22 [I 2022-10-08 22:56:03] ...trap/engines/engine.py.326: process: 3.10328 | load: 2.56507 [I 2022-10-08 22:56:09] ...trap/engines/engine.py.321: eval: epoch 22 | batch 10/510 [I 2022-10-08 22:56:09] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:43 [I 2022-10-08 22:56:09] ...trap/engines/engine.py.326: process: 0.62532 | load: 0.00018 [I 2022-10-08 22:56:14] ...trap/engines/engine.py.321: eval: epoch 22 | batch 20/510 [I 2022-10-08 22:56:14] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:03:57 [I 2022-10-08 22:56:14] ...trap/engines/engine.py.326: process: 0.42452 | load: 0.00018 [I 2022-10-08 22:56:19] ...trap/engines/engine.py.321: eval: epoch 22 | batch 30/510 [I 2022-10-08 22:56:19] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:15 [I 2022-10-08 22:56:19] ...trap/engines/engine.py.326: process: 0.54867 | load: 0.00019 [I 2022-10-08 22:56:25] ...trap/engines/engine.py.321: eval: epoch 22 | batch 40/510 [I 2022-10-08 22:56:25] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:44 [I 2022-10-08 22:56:25] ...trap/engines/engine.py.326: process: 0.44898 | load: 0.00019 [I 2022-10-08 22:56:31] ...trap/engines/engine.py.321: eval: epoch 22 | batch 50/510 [I 2022-10-08 22:56:31] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:37 [I 2022-10-08 22:56:31] ...trap/engines/engine.py.326: process: 0.71639 | load: 0.00019 [I 2022-10-08 22:56:36] ...trap/engines/engine.py.321: eval: epoch 22 | batch 60/510 [I 2022-10-08 22:56:36] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:59 [I 2022-10-08 22:56:36] ...trap/engines/engine.py.326: process: 0.49099 | load: 0.00019 [I 2022-10-08 22:56:42] ...trap/engines/engine.py.321: eval: epoch 22 | batch 70/510 [I 2022-10-08 22:56:42] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:54 [I 2022-10-08 22:56:42] ...trap/engines/engine.py.326: process: 0.55466 | load: 0.00019 [I 2022-10-08 22:56:47] ...trap/engines/engine.py.321: eval: epoch 22 | batch 80/510 [I 2022-10-08 22:56:47] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:51 [I 2022-10-08 22:56:47] ...trap/engines/engine.py.326: process: 0.54922 | load: 0.00019 [I 2022-10-08 22:56:52] ...trap/engines/engine.py.321: eval: epoch 22 | batch 90/510 [I 2022-10-08 22:56:52] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:29 [I 2022-10-08 22:56:52] ...trap/engines/engine.py.326: process: 0.47004 | load: 0.00020 [I 2022-10-08 22:56:57] ...trap/engines/engine.py.321: eval: epoch 22 | batch 100/510 [I 2022-10-08 22:56:57] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:14 [I 2022-10-08 22:56:57] ...trap/engines/engine.py.326: process: 0.31225 | load: 0.00019 [I 2022-10-08 22:57:00] ...trap/engines/engine.py.321: eval: epoch 22 | batch 110/510 [I 2022-10-08 22:57:00] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:13 [I 2022-10-08 22:57:00] ...trap/engines/engine.py.326: process: 0.31837 | load: 0.00019 [I 2022-10-08 22:57:03] ...trap/engines/engine.py.321: eval: epoch 22 | batch 120/510 [I 2022-10-08 22:57:03] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-08 22:57:03] ...trap/engines/engine.py.326: process: 0.31696 | load: 0.00020 [I 2022-10-08 22:57:06] ...trap/engines/engine.py.321: eval: epoch 22 | batch 130/510 [I 2022-10-08 22:57:06] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:00 [I 2022-10-08 22:57:06] ...trap/engines/engine.py.326: process: 0.31763 | load: 0.00020 [I 2022-10-08 22:57:10] ...trap/engines/engine.py.321: eval: epoch 22 | batch 140/510 [I 2022-10-08 22:57:10] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:57 [I 2022-10-08 22:57:10] ...trap/engines/engine.py.326: process: 0.31561 | load: 0.00021 [I 2022-10-08 22:57:13] ...trap/engines/engine.py.321: eval: epoch 22 | batch 150/510 [I 2022-10-08 22:57:13] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-08 22:57:13] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00020 [I 2022-10-08 22:57:16] ...trap/engines/engine.py.321: eval: epoch 22 | batch 160/510 [I 2022-10-08 22:57:16] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:51 [I 2022-10-08 22:57:16] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00020 [I 2022-10-08 22:57:19] ...trap/engines/engine.py.321: eval: epoch 22 | batch 170/510 [I 2022-10-08 22:57:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-08 22:57:19] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00019 [I 2022-10-08 22:57:22] ...trap/engines/engine.py.321: eval: epoch 22 | batch 180/510 [I 2022-10-08 22:57:22] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:45 [I 2022-10-08 22:57:22] ...trap/engines/engine.py.326: process: 0.32026 | load: 0.00021 [I 2022-10-08 22:57:26] ...trap/engines/engine.py.321: eval: epoch 22 | batch 190/510 [I 2022-10-08 22:57:26] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:42 [I 2022-10-08 22:57:26] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00021 [I 2022-10-08 22:57:29] ...trap/engines/engine.py.321: eval: epoch 22 | batch 200/510 [I 2022-10-08 22:57:29] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-08 22:57:29] ...trap/engines/engine.py.326: process: 0.31713 | load: 0.00021 [I 2022-10-08 22:57:32] ...trap/engines/engine.py.321: eval: epoch 22 | batch 210/510 [I 2022-10-08 22:57:32] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-08 22:57:32] ...trap/engines/engine.py.326: process: 0.31504 | load: 0.00020 [I 2022-10-08 22:57:35] ...trap/engines/engine.py.321: eval: epoch 22 | batch 220/510 [I 2022-10-08 22:57:35] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:32 [I 2022-10-08 22:57:35] ...trap/engines/engine.py.326: process: 0.31841 | load: 0.00020 [I 2022-10-08 22:57:38] ...trap/engines/engine.py.321: eval: epoch 22 | batch 230/510 [I 2022-10-08 22:57:38] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:28 [I 2022-10-08 22:57:38] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00019 [I 2022-10-08 22:57:41] ...trap/engines/engine.py.321: eval: epoch 22 | batch 240/510 [I 2022-10-08 22:57:41] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:25 [I 2022-10-08 22:57:41] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00020 [I 2022-10-08 22:57:45] ...trap/engines/engine.py.321: eval: epoch 22 | batch 250/510 [I 2022-10-08 22:57:45] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:22 [I 2022-10-08 22:57:45] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00020 [I 2022-10-08 22:57:48] ...trap/engines/engine.py.321: eval: epoch 22 | batch 260/510 [I 2022-10-08 22:57:48] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:19 [I 2022-10-08 22:57:48] ...trap/engines/engine.py.326: process: 0.31611 | load: 0.00021 [I 2022-10-08 22:57:51] ...trap/engines/engine.py.321: eval: epoch 22 | batch 270/510 [I 2022-10-08 22:57:51] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-08 22:57:51] ...trap/engines/engine.py.326: process: 0.32156 | load: 0.00020 [I 2022-10-08 22:57:54] ...trap/engines/engine.py.321: eval: epoch 22 | batch 280/510 [I 2022-10-08 22:57:54] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-08 22:57:54] ...trap/engines/engine.py.326: process: 0.31476 | load: 0.00021 [I 2022-10-08 22:57:57] ...trap/engines/engine.py.321: eval: epoch 22 | batch 290/510 [I 2022-10-08 22:57:57] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-08 22:57:57] ...trap/engines/engine.py.326: process: 0.32662 | load: 0.00020 [I 2022-10-08 22:58:01] ...trap/engines/engine.py.321: eval: epoch 22 | batch 300/510 [I 2022-10-08 22:58:01] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-08 22:58:01] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00019 [I 2022-10-08 22:58:04] ...trap/engines/engine.py.321: eval: epoch 22 | batch 310/510 [I 2022-10-08 22:58:04] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:03 [I 2022-10-08 22:58:04] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00020 [I 2022-10-08 22:58:07] ...trap/engines/engine.py.321: eval: epoch 22 | batch 320/510 [I 2022-10-08 22:58:07] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-08 22:58:07] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00020 [I 2022-10-08 22:58:10] ...trap/engines/engine.py.321: eval: epoch 22 | batch 330/510 [I 2022-10-08 22:58:10] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:56 [I 2022-10-08 22:58:10] ...trap/engines/engine.py.326: process: 0.31388 | load: 0.00020 [I 2022-10-08 22:58:13] ...trap/engines/engine.py.321: eval: epoch 22 | batch 340/510 [I 2022-10-08 22:58:13] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-08 22:58:13] ...trap/engines/engine.py.326: process: 0.31946 | load: 0.00020 [I 2022-10-08 22:58:17] ...trap/engines/engine.py.321: eval: epoch 22 | batch 350/510 [I 2022-10-08 22:58:17] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-08 22:58:17] ...trap/engines/engine.py.326: process: 0.31933 | load: 0.00020 [I 2022-10-08 22:58:20] ...trap/engines/engine.py.321: eval: epoch 22 | batch 360/510 [I 2022-10-08 22:58:20] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:48 [I 2022-10-08 22:58:20] ...trap/engines/engine.py.326: process: 0.32030 | load: 0.00021 [I 2022-10-08 22:58:23] ...trap/engines/engine.py.321: eval: epoch 22 | batch 370/510 [I 2022-10-08 22:58:23] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-08 22:58:23] ...trap/engines/engine.py.326: process: 0.31670 | load: 0.00020 [I 2022-10-08 22:58:26] ...trap/engines/engine.py.321: eval: epoch 22 | batch 380/510 [I 2022-10-08 22:58:26] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-08 22:58:26] ...trap/engines/engine.py.326: process: 0.32095 | load: 0.00020 [I 2022-10-08 22:58:29] ...trap/engines/engine.py.321: eval: epoch 22 | batch 390/510 [I 2022-10-08 22:58:29] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-08 22:58:29] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00020 [I 2022-10-08 22:58:32] ...trap/engines/engine.py.321: eval: epoch 22 | batch 400/510 [I 2022-10-08 22:58:32] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:34 [I 2022-10-08 22:58:32] ...trap/engines/engine.py.326: process: 0.31360 | load: 0.00020 [I 2022-10-08 22:58:36] ...trap/engines/engine.py.321: eval: epoch 22 | batch 410/510 [I 2022-10-08 22:58:36] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-08 22:58:36] ...trap/engines/engine.py.326: process: 0.31433 | load: 0.00020 [I 2022-10-08 22:58:39] ...trap/engines/engine.py.321: eval: epoch 22 | batch 420/510 [I 2022-10-08 22:58:39] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-08 22:58:39] ...trap/engines/engine.py.326: process: 0.31402 | load: 0.00020 [I 2022-10-08 22:58:42] ...trap/engines/engine.py.321: eval: epoch 22 | batch 430/510 [I 2022-10-08 22:58:42] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-08 22:58:42] ...trap/engines/engine.py.326: process: 0.31750 | load: 0.00020 [I 2022-10-08 22:58:45] ...trap/engines/engine.py.321: eval: epoch 22 | batch 440/510 [I 2022-10-08 22:58:45] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-08 22:58:45] ...trap/engines/engine.py.326: process: 0.32277 | load: 0.00021 [I 2022-10-08 22:58:48] ...trap/engines/engine.py.321: eval: epoch 22 | batch 450/510 [I 2022-10-08 22:58:48] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-08 22:58:48] ...trap/engines/engine.py.326: process: 0.32584 | load: 0.00019 [I 2022-10-08 22:58:52] ...trap/engines/engine.py.321: eval: epoch 22 | batch 460/510 [I 2022-10-08 22:58:52] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:16 [I 2022-10-08 22:58:52] ...trap/engines/engine.py.326: process: 0.32231 | load: 0.00020 [I 2022-10-08 22:58:55] ...trap/engines/engine.py.321: eval: epoch 22 | batch 470/510 [I 2022-10-08 22:58:55] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-08 22:58:55] ...trap/engines/engine.py.326: process: 0.32203 | load: 0.00020 [I 2022-10-08 22:58:58] ...trap/engines/engine.py.321: eval: epoch 22 | batch 480/510 [I 2022-10-08 22:58:58] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-08 22:58:58] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00021 [I 2022-10-08 22:59:01] ...trap/engines/engine.py.321: eval: epoch 22 | batch 490/510 [I 2022-10-08 22:59:01] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-08 22:59:01] ...trap/engines/engine.py.326: process: 0.31896 | load: 0.00020 [I 2022-10-08 22:59:04] ...trap/engines/engine.py.321: eval: epoch 22 | batch 500/510 [I 2022-10-08 22:59:04] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-08 22:59:04] ...trap/engines/engine.py.326: process: 0.31792 | load: 0.00021 [I 2022-10-08 22:59:08] ...trap/engines/engine.py.321: eval: epoch 22 | batch 510/510 [I 2022-10-08 22:59:08] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-08 22:59:08] ...trap/engines/engine.py.326: process: 0.31300 | load: 0.00016 [S 2022-10-08 22:59:08] ...trap/engines/engine.py.132: eval_epoch.epoch: 22 [I 2022-10-08 22:59:08] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.656400 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.878800 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.924700 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.648700 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.879000 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.920900 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.006515 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009948 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007457 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009487 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006943 [S 2022-10-08 22:59:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007245 [I 2022-10-08 22:59:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 22:59:48] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 22:59:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 22:59:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 22:59:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 22:59:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 22:59:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 22:59:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 22:59:58] ...trap/engines/engine.py.163: Training model on trainset for epoch 23 [I 2022-10-08 22:59:58] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 23:00:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 0/2304 [I 2022-10-08 23:00:01] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:56 [I 2022-10-08 23:00:01] ...trap/engines/engine.py.229: process: 3.09753 | load: 2.23468 [I 2022-10-08 23:00:01] ...trap/engines/engine.py.230: loss: 0.15855 [I 2022-10-08 23:00:01] ...optimizers/trijoint.py.151: total_norm: 22.153576 [I 2022-10-08 23:00:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 10/2304 [I 2022-10-08 23:00:09] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:20 [I 2022-10-08 23:00:09] ...trap/engines/engine.py.229: process: 0.85898 | load: 0.00517 [I 2022-10-08 23:00:09] ...trap/engines/engine.py.230: loss: 0.19654 [I 2022-10-08 23:00:09] ...optimizers/trijoint.py.151: total_norm: 26.158457 [I 2022-10-08 23:00:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 20/2304 [I 2022-10-08 23:00:18] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:45 [I 2022-10-08 23:00:18] ...trap/engines/engine.py.229: process: 0.83225 | load: 0.00469 [I 2022-10-08 23:00:18] ...trap/engines/engine.py.230: loss: 0.12309 [I 2022-10-08 23:00:18] ...optimizers/trijoint.py.151: total_norm: 14.538248 [I 2022-10-08 23:00:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 30/2304 [I 2022-10-08 23:00:26] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:42 [I 2022-10-08 23:00:26] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00448 [I 2022-10-08 23:00:26] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-08 23:00:26] ...optimizers/trijoint.py.151: total_norm: 16.246012 [I 2022-10-08 23:00:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 40/2304 [I 2022-10-08 23:00:34] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:34 [I 2022-10-08 23:00:34] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00392 [I 2022-10-08 23:00:34] ...trap/engines/engine.py.230: loss: 0.21031 [I 2022-10-08 23:00:35] ...optimizers/trijoint.py.151: total_norm: 10.998877 [I 2022-10-08 23:00:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 50/2304 [I 2022-10-08 23:00:43] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:32 [I 2022-10-08 23:00:43] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00353 [I 2022-10-08 23:00:43] ...trap/engines/engine.py.230: loss: 0.12837 [I 2022-10-08 23:00:43] ...optimizers/trijoint.py.151: total_norm: 18.377125 [I 2022-10-08 23:00:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 60/2304 [I 2022-10-08 23:00:51] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:25 [I 2022-10-08 23:00:51] ...trap/engines/engine.py.229: process: 0.84057 | load: 0.00389 [I 2022-10-08 23:00:51] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-08 23:00:51] ...optimizers/trijoint.py.151: total_norm: 13.544932 [I 2022-10-08 23:01:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 70/2304 [I 2022-10-08 23:01:00] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:15 [I 2022-10-08 23:01:00] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00341 [I 2022-10-08 23:01:00] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-08 23:01:00] ...optimizers/trijoint.py.151: total_norm: 16.821106 [I 2022-10-08 23:01:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 80/2304 [I 2022-10-08 23:01:08] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:08 [I 2022-10-08 23:01:08] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00387 [I 2022-10-08 23:01:08] ...trap/engines/engine.py.230: loss: 0.19964 [I 2022-10-08 23:01:08] ...optimizers/trijoint.py.151: total_norm: 22.241728 [I 2022-10-08 23:01:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 90/2304 [I 2022-10-08 23:01:17] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:08 [I 2022-10-08 23:01:17] ...trap/engines/engine.py.229: process: 0.83924 | load: 0.00394 [I 2022-10-08 23:01:17] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-08 23:01:17] ...optimizers/trijoint.py.151: total_norm: 13.774585 [I 2022-10-08 23:01:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 100/2304 [I 2022-10-08 23:01:25] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:02 [I 2022-10-08 23:01:25] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00325 [I 2022-10-08 23:01:25] ...trap/engines/engine.py.230: loss: 0.18792 [I 2022-10-08 23:01:25] ...optimizers/trijoint.py.151: total_norm: 17.778439 [I 2022-10-08 23:01:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 110/2304 [I 2022-10-08 23:01:34] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:42 [I 2022-10-08 23:01:34] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00332 [I 2022-10-08 23:01:34] ...trap/engines/engine.py.230: loss: 0.17493 [I 2022-10-08 23:01:34] ...optimizers/trijoint.py.151: total_norm: 26.712204 [I 2022-10-08 23:01:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 120/2304 [I 2022-10-08 23:01:42] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:30 [I 2022-10-08 23:01:42] ...trap/engines/engine.py.229: process: 0.82836 | load: 0.00328 [I 2022-10-08 23:01:42] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-08 23:01:42] ...optimizers/trijoint.py.151: total_norm: 14.631172 [I 2022-10-08 23:01:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 130/2304 [I 2022-10-08 23:01:50] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:04 [I 2022-10-08 23:01:50] ...trap/engines/engine.py.229: process: 0.80389 | load: 0.00326 [I 2022-10-08 23:01:50] ...trap/engines/engine.py.230: loss: 0.16734 [I 2022-10-08 23:01:50] ...optimizers/trijoint.py.151: total_norm: 20.204283 [I 2022-10-08 23:01:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 140/2304 [I 2022-10-08 23:01:59] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:06 [I 2022-10-08 23:01:59] ...trap/engines/engine.py.229: process: 0.83141 | load: 0.00326 [I 2022-10-08 23:01:59] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-08 23:01:59] ...optimizers/trijoint.py.151: total_norm: 9.666909 [I 2022-10-08 23:02:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 150/2304 [I 2022-10-08 23:02:07] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:03 [I 2022-10-08 23:02:07] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00320 [I 2022-10-08 23:02:07] ...trap/engines/engine.py.230: loss: 0.17971 [I 2022-10-08 23:02:07] ...optimizers/trijoint.py.151: total_norm: 16.801750 [I 2022-10-08 23:02:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 160/2304 [I 2022-10-08 23:02:16] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:29:46 [I 2022-10-08 23:02:16] ...trap/engines/engine.py.229: process: 0.83044 | load: 0.00358 [I 2022-10-08 23:02:16] ...trap/engines/engine.py.230: loss: 0.14622 [I 2022-10-08 23:02:16] ...optimizers/trijoint.py.151: total_norm: 18.783180 [I 2022-10-08 23:02:24] ...trap/engines/engine.py.224: train: epoch 23 | batch 170/2304 [I 2022-10-08 23:02:24] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:52 [I 2022-10-08 23:02:24] ...trap/engines/engine.py.229: process: 0.83030 | load: 0.00323 [I 2022-10-08 23:02:24] ...trap/engines/engine.py.230: loss: 0.25476 [I 2022-10-08 23:02:24] ...optimizers/trijoint.py.151: total_norm: 21.226007 [I 2022-10-08 23:02:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 180/2304 [I 2022-10-08 23:02:32] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:44 [I 2022-10-08 23:02:32] ...trap/engines/engine.py.229: process: 0.83619 | load: 0.00302 [I 2022-10-08 23:02:32] ...trap/engines/engine.py.230: loss: 0.19262 [I 2022-10-08 23:02:32] ...optimizers/trijoint.py.151: total_norm: 17.234982 [I 2022-10-08 23:02:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 190/2304 [I 2022-10-08 23:02:41] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:38 [I 2022-10-08 23:02:41] ...trap/engines/engine.py.229: process: 0.84623 | load: 0.00306 [I 2022-10-08 23:02:41] ...trap/engines/engine.py.230: loss: 0.21296 [I 2022-10-08 23:02:41] ...optimizers/trijoint.py.151: total_norm: 24.549681 [I 2022-10-08 23:02:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 200/2304 [I 2022-10-08 23:02:49] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:27 [I 2022-10-08 23:02:49] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00304 [I 2022-10-08 23:02:49] ...trap/engines/engine.py.230: loss: 0.20460 [I 2022-10-08 23:02:49] ...optimizers/trijoint.py.151: total_norm: 26.025076 [I 2022-10-08 23:02:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 210/2304 [I 2022-10-08 23:02:58] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:19 [I 2022-10-08 23:02:58] ...trap/engines/engine.py.229: process: 0.83982 | load: 0.00324 [I 2022-10-08 23:02:58] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-08 23:02:58] ...optimizers/trijoint.py.151: total_norm: 20.279921 [I 2022-10-08 23:03:06] ...trap/engines/engine.py.224: train: epoch 23 | batch 220/2304 [I 2022-10-08 23:03:06] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:28:56 [I 2022-10-08 23:03:06] ...trap/engines/engine.py.229: process: 0.82802 | load: 0.00308 [I 2022-10-08 23:03:06] ...trap/engines/engine.py.230: loss: 0.17577 [I 2022-10-08 23:03:06] ...optimizers/trijoint.py.151: total_norm: 30.046274 [I 2022-10-08 23:03:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 230/2304 [I 2022-10-08 23:03:14] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:28:58 [I 2022-10-08 23:03:14] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00310 [I 2022-10-08 23:03:14] ...trap/engines/engine.py.230: loss: 0.16804 [I 2022-10-08 23:03:14] ...optimizers/trijoint.py.151: total_norm: 17.847214 [I 2022-10-08 23:03:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 240/2304 [I 2022-10-08 23:03:23] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:51 [I 2022-10-08 23:03:23] ...trap/engines/engine.py.229: process: 0.83296 | load: 0.00321 [I 2022-10-08 23:03:23] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-08 23:03:23] ...optimizers/trijoint.py.151: total_norm: 16.574120 [I 2022-10-08 23:03:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 250/2304 [I 2022-10-08 23:03:31] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:41 [I 2022-10-08 23:03:31] ...trap/engines/engine.py.229: process: 0.83473 | load: 0.00299 [I 2022-10-08 23:03:31] ...trap/engines/engine.py.230: loss: 0.33665 [I 2022-10-08 23:03:31] ...optimizers/trijoint.py.151: total_norm: 23.999678 [I 2022-10-08 23:03:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 260/2304 [I 2022-10-08 23:03:40] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:57 [I 2022-10-08 23:03:40] ...trap/engines/engine.py.229: process: 0.86610 | load: 0.00300 [I 2022-10-08 23:03:40] ...trap/engines/engine.py.230: loss: 0.17205 [I 2022-10-08 23:03:40] ...optimizers/trijoint.py.151: total_norm: 27.614256 [I 2022-10-08 23:03:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 270/2304 [I 2022-10-08 23:03:48] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:27 [I 2022-10-08 23:03:48] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00306 [I 2022-10-08 23:03:48] ...trap/engines/engine.py.230: loss: 0.18360 [I 2022-10-08 23:03:48] ...optimizers/trijoint.py.151: total_norm: 21.182989 [I 2022-10-08 23:03:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 280/2304 [I 2022-10-08 23:03:57] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:31 [I 2022-10-08 23:03:57] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00313 [I 2022-10-08 23:03:57] ...trap/engines/engine.py.230: loss: 0.12835 [I 2022-10-08 23:03:57] ...optimizers/trijoint.py.151: total_norm: 16.141771 [I 2022-10-08 23:04:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 290/2304 [I 2022-10-08 23:04:05] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:08 [I 2022-10-08 23:04:05] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00308 [I 2022-10-08 23:04:05] ...trap/engines/engine.py.230: loss: 0.22195 [I 2022-10-08 23:04:05] ...optimizers/trijoint.py.151: total_norm: 23.300125 [I 2022-10-08 23:04:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 300/2304 [I 2022-10-08 23:04:13] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:58 [I 2022-10-08 23:04:13] ...trap/engines/engine.py.229: process: 0.82443 | load: 0.00313 [I 2022-10-08 23:04:13] ...trap/engines/engine.py.230: loss: 0.22414 [I 2022-10-08 23:04:13] ...optimizers/trijoint.py.151: total_norm: 51.562794 [I 2022-10-08 23:04:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 310/2304 [I 2022-10-08 23:04:22] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:04 [I 2022-10-08 23:04:22] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00308 [I 2022-10-08 23:04:22] ...trap/engines/engine.py.230: loss: 0.19702 [I 2022-10-08 23:04:22] ...optimizers/trijoint.py.151: total_norm: 19.231991 [I 2022-10-08 23:04:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 320/2304 [I 2022-10-08 23:04:30] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:49 [I 2022-10-08 23:04:30] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00325 [I 2022-10-08 23:04:30] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-08 23:04:30] ...optimizers/trijoint.py.151: total_norm: 20.425581 [I 2022-10-08 23:04:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 330/2304 [I 2022-10-08 23:04:39] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:34 [I 2022-10-08 23:04:39] ...trap/engines/engine.py.229: process: 0.83426 | load: 0.00311 [I 2022-10-08 23:04:39] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-08 23:04:39] ...optimizers/trijoint.py.151: total_norm: 16.542568 [I 2022-10-08 23:04:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 340/2304 [I 2022-10-08 23:04:47] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:31 [I 2022-10-08 23:04:47] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00311 [I 2022-10-08 23:04:47] ...trap/engines/engine.py.230: loss: 0.15358 [I 2022-10-08 23:04:47] ...optimizers/trijoint.py.151: total_norm: 28.811865 [I 2022-10-08 23:04:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 350/2304 [I 2022-10-08 23:04:56] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:15 [I 2022-10-08 23:04:56] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00303 [I 2022-10-08 23:04:56] ...trap/engines/engine.py.230: loss: 0.21345 [I 2022-10-08 23:04:56] ...optimizers/trijoint.py.151: total_norm: 20.295763 [I 2022-10-08 23:05:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 360/2304 [I 2022-10-08 23:05:04] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:03 [I 2022-10-08 23:05:04] ...trap/engines/engine.py.229: process: 0.82753 | load: 0.00299 [I 2022-10-08 23:05:04] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-08 23:05:04] ...optimizers/trijoint.py.151: total_norm: 16.687809 [I 2022-10-08 23:05:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 370/2304 [I 2022-10-08 23:05:12] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:26:53 [I 2022-10-08 23:05:12] ...trap/engines/engine.py.229: process: 0.82512 | load: 0.00304 [I 2022-10-08 23:05:12] ...trap/engines/engine.py.230: loss: 0.26198 [I 2022-10-08 23:05:12] ...optimizers/trijoint.py.151: total_norm: 33.366463 [I 2022-10-08 23:05:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 380/2304 [I 2022-10-08 23:05:21] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:47 [I 2022-10-08 23:05:21] ...trap/engines/engine.py.229: process: 0.83666 | load: 0.00298 [I 2022-10-08 23:05:21] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-08 23:05:21] ...optimizers/trijoint.py.151: total_norm: 15.449746 [I 2022-10-08 23:05:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 390/2304 [I 2022-10-08 23:05:29] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:41 [I 2022-10-08 23:05:29] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00312 [I 2022-10-08 23:05:29] ...trap/engines/engine.py.230: loss: 0.20694 [I 2022-10-08 23:05:29] ...optimizers/trijoint.py.151: total_norm: 24.718563 [I 2022-10-08 23:05:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 400/2304 [I 2022-10-08 23:05:37] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:38 [I 2022-10-08 23:05:37] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00298 [I 2022-10-08 23:05:37] ...trap/engines/engine.py.230: loss: 0.17806 [I 2022-10-08 23:05:37] ...optimizers/trijoint.py.151: total_norm: 11.901918 [I 2022-10-08 23:05:46] ...trap/engines/engine.py.224: train: epoch 23 | batch 410/2304 [I 2022-10-08 23:05:46] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:26:24 [I 2022-10-08 23:05:46] ...trap/engines/engine.py.229: process: 0.84031 | load: 0.00317 [I 2022-10-08 23:05:46] ...trap/engines/engine.py.230: loss: 0.16882 [I 2022-10-08 23:05:46] ...optimizers/trijoint.py.151: total_norm: 18.884449 [I 2022-10-08 23:05:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 420/2304 [I 2022-10-08 23:05:54] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:29 [I 2022-10-08 23:05:54] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00309 [I 2022-10-08 23:05:54] ...trap/engines/engine.py.230: loss: 0.18116 [I 2022-10-08 23:05:54] ...optimizers/trijoint.py.151: total_norm: 17.287754 [I 2022-10-08 23:06:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 430/2304 [I 2022-10-08 23:06:03] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:14 [I 2022-10-08 23:06:03] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00297 [I 2022-10-08 23:06:03] ...trap/engines/engine.py.230: loss: 0.14436 [I 2022-10-08 23:06:03] ...optimizers/trijoint.py.151: total_norm: 17.027203 [I 2022-10-08 23:06:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 440/2304 [I 2022-10-08 23:06:11] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:10 [I 2022-10-08 23:06:11] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00298 [I 2022-10-08 23:06:11] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-08 23:06:11] ...optimizers/trijoint.py.151: total_norm: 19.955088 [I 2022-10-08 23:06:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 450/2304 [I 2022-10-08 23:06:20] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:25:50 [I 2022-10-08 23:06:20] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00298 [I 2022-10-08 23:06:20] ...trap/engines/engine.py.230: loss: 0.17324 [I 2022-10-08 23:06:20] ...optimizers/trijoint.py.151: total_norm: 29.622051 [I 2022-10-08 23:06:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 460/2304 [I 2022-10-08 23:06:28] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:44 [I 2022-10-08 23:06:28] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00300 [I 2022-10-08 23:06:28] ...trap/engines/engine.py.230: loss: 0.13883 [I 2022-10-08 23:06:28] ...optimizers/trijoint.py.151: total_norm: 24.090584 [I 2022-10-08 23:06:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 470/2304 [I 2022-10-08 23:06:36] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:39 [I 2022-10-08 23:06:36] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00301 [I 2022-10-08 23:06:36] ...trap/engines/engine.py.230: loss: 0.18578 [I 2022-10-08 23:06:36] ...optimizers/trijoint.py.151: total_norm: 19.988482 [I 2022-10-08 23:06:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 480/2304 [I 2022-10-08 23:06:45] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:25:28 [I 2022-10-08 23:06:45] ...trap/engines/engine.py.229: process: 0.83123 | load: 0.00311 [I 2022-10-08 23:06:45] ...trap/engines/engine.py.230: loss: 0.16416 [I 2022-10-08 23:06:45] ...optimizers/trijoint.py.151: total_norm: 23.201000 [I 2022-10-08 23:06:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 490/2304 [I 2022-10-08 23:06:53] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:16 [I 2022-10-08 23:06:53] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00310 [I 2022-10-08 23:06:53] ...trap/engines/engine.py.230: loss: 0.26623 [I 2022-10-08 23:06:53] ...optimizers/trijoint.py.151: total_norm: 28.983171 [I 2022-10-08 23:07:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 500/2304 [I 2022-10-08 23:07:02] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:25:10 [I 2022-10-08 23:07:02] ...trap/engines/engine.py.229: process: 0.83732 | load: 0.00307 [I 2022-10-08 23:07:02] ...trap/engines/engine.py.230: loss: 0.20723 [I 2022-10-08 23:07:02] ...optimizers/trijoint.py.151: total_norm: 32.309437 [I 2022-10-08 23:07:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 510/2304 [I 2022-10-08 23:07:10] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:05 [I 2022-10-08 23:07:10] ...trap/engines/engine.py.229: process: 0.83558 | load: 0.00305 [I 2022-10-08 23:07:10] ...trap/engines/engine.py.230: loss: 0.18538 [I 2022-10-08 23:07:10] ...optimizers/trijoint.py.151: total_norm: 17.847439 [I 2022-10-08 23:07:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 520/2304 [I 2022-10-08 23:07:18] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:24:56 [I 2022-10-08 23:07:18] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00301 [I 2022-10-08 23:07:18] ...trap/engines/engine.py.230: loss: 0.21896 [I 2022-10-08 23:07:18] ...optimizers/trijoint.py.151: total_norm: 24.582335 [I 2022-10-08 23:07:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 530/2304 [I 2022-10-08 23:07:27] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:24:41 [I 2022-10-08 23:07:27] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00302 [I 2022-10-08 23:07:27] ...trap/engines/engine.py.230: loss: 0.19604 [I 2022-10-08 23:07:27] ...optimizers/trijoint.py.151: total_norm: 20.310291 [I 2022-10-08 23:07:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 540/2304 [I 2022-10-08 23:07:35] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:44 [I 2022-10-08 23:07:35] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00300 [I 2022-10-08 23:07:35] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-08 23:07:35] ...optimizers/trijoint.py.151: total_norm: 15.404315 [I 2022-10-08 23:07:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 550/2304 [I 2022-10-08 23:07:44] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:24:30 [I 2022-10-08 23:07:44] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00299 [I 2022-10-08 23:07:44] ...trap/engines/engine.py.230: loss: 0.19814 [I 2022-10-08 23:07:44] ...optimizers/trijoint.py.151: total_norm: 28.390888 [I 2022-10-08 23:07:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 560/2304 [I 2022-10-08 23:07:52] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:24 [I 2022-10-08 23:07:52] ...trap/engines/engine.py.229: process: 0.84292 | load: 0.00300 [I 2022-10-08 23:07:52] ...trap/engines/engine.py.230: loss: 0.18813 [I 2022-10-08 23:07:52] ...optimizers/trijoint.py.151: total_norm: 22.022511 [I 2022-10-08 23:08:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 570/2304 [I 2022-10-08 23:08:00] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:11 [I 2022-10-08 23:08:00] ...trap/engines/engine.py.229: process: 0.82794 | load: 0.00303 [I 2022-10-08 23:08:00] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-08 23:08:00] ...optimizers/trijoint.py.151: total_norm: 25.546183 [I 2022-10-08 23:08:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 580/2304 [I 2022-10-08 23:08:09] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:23:59 [I 2022-10-08 23:08:09] ...trap/engines/engine.py.229: process: 0.81617 | load: 0.00299 [I 2022-10-08 23:08:09] ...trap/engines/engine.py.230: loss: 0.11853 [I 2022-10-08 23:08:09] ...optimizers/trijoint.py.151: total_norm: 13.661930 [I 2022-10-08 23:08:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 590/2304 [I 2022-10-08 23:08:17] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:23:50 [I 2022-10-08 23:08:17] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00300 [I 2022-10-08 23:08:17] ...trap/engines/engine.py.230: loss: 0.15163 [I 2022-10-08 23:08:17] ...optimizers/trijoint.py.151: total_norm: 18.269663 [I 2022-10-08 23:08:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 600/2304 [I 2022-10-08 23:08:25] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:23:45 [I 2022-10-08 23:08:25] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00301 [I 2022-10-08 23:08:25] ...trap/engines/engine.py.230: loss: 0.13485 [I 2022-10-08 23:08:25] ...optimizers/trijoint.py.151: total_norm: 19.919403 [I 2022-10-08 23:08:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 610/2304 [I 2022-10-08 23:08:34] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:23:36 [I 2022-10-08 23:08:34] ...trap/engines/engine.py.229: process: 0.82766 | load: 0.00302 [I 2022-10-08 23:08:34] ...trap/engines/engine.py.230: loss: 0.20847 [I 2022-10-08 23:08:34] ...optimizers/trijoint.py.151: total_norm: 27.631603 [I 2022-10-08 23:08:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 620/2304 [I 2022-10-08 23:08:42] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:31 [I 2022-10-08 23:08:42] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00302 [I 2022-10-08 23:08:42] ...trap/engines/engine.py.230: loss: 0.24273 [I 2022-10-08 23:08:42] ...optimizers/trijoint.py.151: total_norm: 29.042240 [I 2022-10-08 23:08:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 630/2304 [I 2022-10-08 23:08:51] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:27 [I 2022-10-08 23:08:51] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00314 [I 2022-10-08 23:08:51] ...trap/engines/engine.py.230: loss: 0.23095 [I 2022-10-08 23:08:51] ...optimizers/trijoint.py.151: total_norm: 30.463573 [I 2022-10-08 23:08:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 640/2304 [I 2022-10-08 23:08:59] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:15 [I 2022-10-08 23:08:59] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00299 [I 2022-10-08 23:08:59] ...trap/engines/engine.py.230: loss: 0.17576 [I 2022-10-08 23:08:59] ...optimizers/trijoint.py.151: total_norm: 10.896715 [I 2022-10-08 23:09:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 650/2304 [I 2022-10-08 23:09:08] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:15 [I 2022-10-08 23:09:08] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00300 [I 2022-10-08 23:09:08] ...trap/engines/engine.py.230: loss: 0.23154 [I 2022-10-08 23:09:08] ...optimizers/trijoint.py.151: total_norm: 30.212538 [I 2022-10-08 23:09:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 660/2304 [I 2022-10-08 23:09:16] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:01 [I 2022-10-08 23:09:16] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00301 [I 2022-10-08 23:09:16] ...trap/engines/engine.py.230: loss: 0.20661 [I 2022-10-08 23:09:16] ...optimizers/trijoint.py.151: total_norm: 25.534519 [I 2022-10-08 23:09:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 670/2304 [I 2022-10-08 23:09:25] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:22:54 [I 2022-10-08 23:09:25] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00298 [I 2022-10-08 23:09:25] ...trap/engines/engine.py.230: loss: 0.19734 [I 2022-10-08 23:09:25] ...optimizers/trijoint.py.151: total_norm: 21.402006 [I 2022-10-08 23:09:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 680/2304 [I 2022-10-08 23:09:33] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:39 [I 2022-10-08 23:09:33] ...trap/engines/engine.py.229: process: 0.82471 | load: 0.00297 [I 2022-10-08 23:09:33] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-08 23:09:33] ...optimizers/trijoint.py.151: total_norm: 13.853992 [I 2022-10-08 23:09:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 690/2304 [I 2022-10-08 23:09:41] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:22:29 [I 2022-10-08 23:09:41] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00301 [I 2022-10-08 23:09:41] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-08 23:09:41] ...optimizers/trijoint.py.151: total_norm: 18.608053 [I 2022-10-08 23:09:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 700/2304 [I 2022-10-08 23:09:50] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:31 [I 2022-10-08 23:09:50] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00299 [I 2022-10-08 23:09:50] ...trap/engines/engine.py.230: loss: 0.25890 [I 2022-10-08 23:09:50] ...optimizers/trijoint.py.151: total_norm: 27.346544 [I 2022-10-08 23:09:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 710/2304 [I 2022-10-08 23:09:58] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:20 [I 2022-10-08 23:09:58] ...trap/engines/engine.py.229: process: 0.84074 | load: 0.00298 [I 2022-10-08 23:09:58] ...trap/engines/engine.py.230: loss: 0.22631 [I 2022-10-08 23:09:58] ...optimizers/trijoint.py.151: total_norm: 27.675850 [I 2022-10-08 23:10:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 720/2304 [I 2022-10-08 23:10:07] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:08 [I 2022-10-08 23:10:07] ...trap/engines/engine.py.229: process: 0.83602 | load: 0.00307 [I 2022-10-08 23:10:07] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-08 23:10:07] ...optimizers/trijoint.py.151: total_norm: 20.654264 [I 2022-10-08 23:10:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 730/2304 [I 2022-10-08 23:10:15] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:21:55 [I 2022-10-08 23:10:15] ...trap/engines/engine.py.229: process: 0.82586 | load: 0.00310 [I 2022-10-08 23:10:15] ...trap/engines/engine.py.230: loss: 0.16761 [I 2022-10-08 23:10:15] ...optimizers/trijoint.py.151: total_norm: 13.498457 [I 2022-10-08 23:10:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 740/2304 [I 2022-10-08 23:10:23] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:21:55 [I 2022-10-08 23:10:23] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00299 [I 2022-10-08 23:10:23] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-08 23:10:23] ...optimizers/trijoint.py.151: total_norm: 25.587011 [I 2022-10-08 23:10:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 750/2304 [I 2022-10-08 23:10:32] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:21:26 [I 2022-10-08 23:10:32] ...trap/engines/engine.py.229: process: 0.81493 | load: 0.00302 [I 2022-10-08 23:10:32] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-08 23:10:32] ...optimizers/trijoint.py.151: total_norm: 14.793739 [I 2022-10-08 23:10:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 760/2304 [I 2022-10-08 23:10:40] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:35 [I 2022-10-08 23:10:40] ...trap/engines/engine.py.229: process: 0.83945 | load: 0.00300 [I 2022-10-08 23:10:40] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-08 23:10:40] ...optimizers/trijoint.py.151: total_norm: 29.582600 [I 2022-10-08 23:10:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 770/2304 [I 2022-10-08 23:10:48] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:27 [I 2022-10-08 23:10:48] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00302 [I 2022-10-08 23:10:48] ...trap/engines/engine.py.230: loss: 0.21678 [I 2022-10-08 23:10:48] ...optimizers/trijoint.py.151: total_norm: 20.992907 [I 2022-10-08 23:10:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 780/2304 [I 2022-10-08 23:10:57] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:13 [I 2022-10-08 23:10:57] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00301 [I 2022-10-08 23:10:57] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-08 23:10:57] ...optimizers/trijoint.py.151: total_norm: 15.471345 [I 2022-10-08 23:11:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 790/2304 [I 2022-10-08 23:11:05] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:08 [I 2022-10-08 23:11:05] ...trap/engines/engine.py.229: process: 0.82758 | load: 0.00303 [I 2022-10-08 23:11:05] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-08 23:11:05] ...optimizers/trijoint.py.151: total_norm: 16.055996 [I 2022-10-08 23:11:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 800/2304 [I 2022-10-08 23:11:14] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:20:53 [I 2022-10-08 23:11:14] ...trap/engines/engine.py.229: process: 0.83369 | load: 0.00301 [I 2022-10-08 23:11:14] ...trap/engines/engine.py.230: loss: 0.25201 [I 2022-10-08 23:11:14] ...optimizers/trijoint.py.151: total_norm: 22.382484 [I 2022-10-08 23:11:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 810/2304 [I 2022-10-08 23:11:22] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:20:45 [I 2022-10-08 23:11:22] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00301 [I 2022-10-08 23:11:22] ...trap/engines/engine.py.230: loss: 0.23446 [I 2022-10-08 23:11:22] ...optimizers/trijoint.py.151: total_norm: 21.543306 [I 2022-10-08 23:11:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 820/2304 [I 2022-10-08 23:11:30] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:20:43 [I 2022-10-08 23:11:30] ...trap/engines/engine.py.229: process: 0.83909 | load: 0.00305 [I 2022-10-08 23:11:30] ...trap/engines/engine.py.230: loss: 0.13697 [I 2022-10-08 23:11:30] ...optimizers/trijoint.py.151: total_norm: 22.498627 [I 2022-10-08 23:11:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 830/2304 [I 2022-10-08 23:11:39] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:20:32 [I 2022-10-08 23:11:39] ...trap/engines/engine.py.229: process: 0.83125 | load: 0.00303 [I 2022-10-08 23:11:39] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-08 23:11:39] ...optimizers/trijoint.py.151: total_norm: 22.213839 [I 2022-10-08 23:11:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 840/2304 [I 2022-10-08 23:11:47] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:38 [I 2022-10-08 23:11:47] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00298 [I 2022-10-08 23:11:47] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-08 23:11:47] ...optimizers/trijoint.py.151: total_norm: 20.673504 [I 2022-10-08 23:12:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 850/2304 [I 2022-10-08 23:12:14] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:49:21 [I 2022-10-08 23:12:14] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00300 [I 2022-10-08 23:12:14] ...trap/engines/engine.py.230: loss: 0.20918 [I 2022-10-08 23:12:14] ...optimizers/trijoint.py.151: total_norm: 28.194437 [I 2022-10-08 23:12:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 860/2304 [I 2022-10-08 23:12:22] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:23:19 [I 2022-10-08 23:12:22] ...trap/engines/engine.py.229: process: 0.83158 | load: 0.00300 [I 2022-10-08 23:12:22] ...trap/engines/engine.py.230: loss: 0.23521 [I 2022-10-08 23:12:22] ...optimizers/trijoint.py.151: total_norm: 23.558016 [I 2022-10-08 23:12:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 870/2304 [I 2022-10-08 23:12:31] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:19 [I 2022-10-08 23:12:31] ...trap/engines/engine.py.229: process: 0.83061 | load: 0.00300 [I 2022-10-08 23:12:31] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-08 23:12:31] ...optimizers/trijoint.py.151: total_norm: 19.968122 [I 2022-10-08 23:12:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 880/2304 [I 2022-10-08 23:12:39] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:57 [I 2022-10-08 23:12:39] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00299 [I 2022-10-08 23:12:39] ...trap/engines/engine.py.230: loss: 0.20259 [I 2022-10-08 23:12:39] ...optimizers/trijoint.py.151: total_norm: 16.338085 [I 2022-10-08 23:12:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 890/2304 [I 2022-10-08 23:12:47] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:45 [I 2022-10-08 23:12:47] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00300 [I 2022-10-08 23:12:47] ...trap/engines/engine.py.230: loss: 0.25173 [I 2022-10-08 23:12:47] ...optimizers/trijoint.py.151: total_norm: 19.496906 [I 2022-10-08 23:12:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 900/2304 [I 2022-10-08 23:12:56] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:41 [I 2022-10-08 23:12:56] ...trap/engines/engine.py.229: process: 0.83068 | load: 0.00301 [I 2022-10-08 23:12:56] ...trap/engines/engine.py.230: loss: 0.25887 [I 2022-10-08 23:12:56] ...optimizers/trijoint.py.151: total_norm: 53.523617 [I 2022-10-08 23:13:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 910/2304 [I 2022-10-08 23:13:04] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:28 [I 2022-10-08 23:13:04] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00299 [I 2022-10-08 23:13:04] ...trap/engines/engine.py.230: loss: 0.26965 [I 2022-10-08 23:13:04] ...optimizers/trijoint.py.151: total_norm: 25.676319 [I 2022-10-08 23:13:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 920/2304 [I 2022-10-08 23:13:13] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:14 [I 2022-10-08 23:13:13] ...trap/engines/engine.py.229: process: 0.82917 | load: 0.00299 [I 2022-10-08 23:13:13] ...trap/engines/engine.py.230: loss: 0.14530 [I 2022-10-08 23:13:13] ...optimizers/trijoint.py.151: total_norm: 10.319312 [I 2022-10-08 23:13:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 930/2304 [I 2022-10-08 23:13:21] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:15 [I 2022-10-08 23:13:21] ...trap/engines/engine.py.229: process: 0.83703 | load: 0.00301 [I 2022-10-08 23:13:21] ...trap/engines/engine.py.230: loss: 0.13370 [I 2022-10-08 23:13:21] ...optimizers/trijoint.py.151: total_norm: 25.622774 [I 2022-10-08 23:13:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 940/2304 [I 2022-10-08 23:13:29] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:19:04 [I 2022-10-08 23:13:29] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00301 [I 2022-10-08 23:13:29] ...trap/engines/engine.py.230: loss: 0.11852 [I 2022-10-08 23:13:29] ...optimizers/trijoint.py.151: total_norm: 25.635427 [I 2022-10-08 23:13:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 950/2304 [I 2022-10-08 23:13:38] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:52 [I 2022-10-08 23:13:38] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00302 [I 2022-10-08 23:13:38] ...trap/engines/engine.py.230: loss: 0.19079 [I 2022-10-08 23:13:38] ...optimizers/trijoint.py.151: total_norm: 17.027443 [I 2022-10-08 23:13:46] ...trap/engines/engine.py.224: train: epoch 23 | batch 960/2304 [I 2022-10-08 23:13:46] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:18:46 [I 2022-10-08 23:13:46] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00302 [I 2022-10-08 23:13:46] ...trap/engines/engine.py.230: loss: 0.17208 [I 2022-10-08 23:13:46] ...optimizers/trijoint.py.151: total_norm: 20.947237 [I 2022-10-08 23:13:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 970/2304 [I 2022-10-08 23:13:55] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:32 [I 2022-10-08 23:13:55] ...trap/engines/engine.py.229: process: 0.83244 | load: 0.00301 [I 2022-10-08 23:13:55] ...trap/engines/engine.py.230: loss: 0.17725 [I 2022-10-08 23:13:55] ...optimizers/trijoint.py.151: total_norm: 32.997032 [I 2022-10-08 23:14:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 980/2304 [I 2022-10-08 23:14:03] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:24 [I 2022-10-08 23:14:03] ...trap/engines/engine.py.229: process: 0.82427 | load: 0.00304 [I 2022-10-08 23:14:03] ...trap/engines/engine.py.230: loss: 0.15316 [I 2022-10-08 23:14:03] ...optimizers/trijoint.py.151: total_norm: 14.347213 [I 2022-10-08 23:14:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 990/2304 [I 2022-10-08 23:14:11] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:19 [I 2022-10-08 23:14:11] ...trap/engines/engine.py.229: process: 0.82547 | load: 0.00301 [I 2022-10-08 23:14:11] ...trap/engines/engine.py.230: loss: 0.23111 [I 2022-10-08 23:14:11] ...optimizers/trijoint.py.151: total_norm: 39.088615 [I 2022-10-08 23:14:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 1000/2304 [I 2022-10-08 23:14:20] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:18:11 [I 2022-10-08 23:14:20] ...trap/engines/engine.py.229: process: 0.82675 | load: 0.00297 [I 2022-10-08 23:14:20] ...trap/engines/engine.py.230: loss: 0.22146 [I 2022-10-08 23:14:20] ...optimizers/trijoint.py.151: total_norm: 25.084826 [I 2022-10-08 23:14:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 1010/2304 [I 2022-10-08 23:14:28] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:17:58 [I 2022-10-08 23:14:28] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00302 [I 2022-10-08 23:14:28] ...trap/engines/engine.py.230: loss: 0.14406 [I 2022-10-08 23:14:28] ...optimizers/trijoint.py.151: total_norm: 22.315609 [I 2022-10-08 23:14:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 1020/2304 [I 2022-10-08 23:14:37] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:17:57 [I 2022-10-08 23:14:37] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00302 [I 2022-10-08 23:14:37] ...trap/engines/engine.py.230: loss: 0.22497 [I 2022-10-08 23:14:37] ...optimizers/trijoint.py.151: total_norm: 20.886848 [I 2022-10-08 23:14:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 1030/2304 [I 2022-10-08 23:14:45] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:17:48 [I 2022-10-08 23:14:45] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00302 [I 2022-10-08 23:14:45] ...trap/engines/engine.py.230: loss: 0.18002 [I 2022-10-08 23:14:45] ...optimizers/trijoint.py.151: total_norm: 19.146496 [I 2022-10-08 23:14:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 1040/2304 [I 2022-10-08 23:14:53] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:17:42 [I 2022-10-08 23:14:53] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00303 [I 2022-10-08 23:14:53] ...trap/engines/engine.py.230: loss: 0.22473 [I 2022-10-08 23:14:53] ...optimizers/trijoint.py.151: total_norm: 18.330803 [I 2022-10-08 23:15:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 1050/2304 [I 2022-10-08 23:15:02] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:34 [I 2022-10-08 23:15:02] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00301 [I 2022-10-08 23:15:02] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-08 23:15:02] ...optimizers/trijoint.py.151: total_norm: 21.404179 [I 2022-10-08 23:15:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 1060/2304 [I 2022-10-08 23:15:10] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:23 [I 2022-10-08 23:15:10] ...trap/engines/engine.py.229: process: 0.82800 | load: 0.00307 [I 2022-10-08 23:15:10] ...trap/engines/engine.py.230: loss: 0.20404 [I 2022-10-08 23:15:10] ...optimizers/trijoint.py.151: total_norm: 13.554979 [I 2022-10-08 23:15:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 1070/2304 [I 2022-10-08 23:15:19] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:12 [I 2022-10-08 23:15:19] ...trap/engines/engine.py.229: process: 0.84136 | load: 0.00298 [I 2022-10-08 23:15:19] ...trap/engines/engine.py.230: loss: 0.15770 [I 2022-10-08 23:15:19] ...optimizers/trijoint.py.151: total_norm: 15.648055 [I 2022-10-08 23:15:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 1080/2304 [I 2022-10-08 23:15:27] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:17:00 [I 2022-10-08 23:15:27] ...trap/engines/engine.py.229: process: 0.83734 | load: 0.00302 [I 2022-10-08 23:15:27] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-08 23:15:27] ...optimizers/trijoint.py.151: total_norm: 16.801075 [I 2022-10-08 23:15:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 1090/2304 [I 2022-10-08 23:15:35] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:16:56 [I 2022-10-08 23:15:35] ...trap/engines/engine.py.229: process: 0.84257 | load: 0.00303 [I 2022-10-08 23:15:35] ...trap/engines/engine.py.230: loss: 0.18280 [I 2022-10-08 23:15:35] ...optimizers/trijoint.py.151: total_norm: 25.488092 [I 2022-10-08 23:15:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 1100/2304 [I 2022-10-08 23:15:44] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:16:44 [I 2022-10-08 23:15:44] ...trap/engines/engine.py.229: process: 0.83221 | load: 0.00314 [I 2022-10-08 23:15:44] ...trap/engines/engine.py.230: loss: 0.18852 [I 2022-10-08 23:15:44] ...optimizers/trijoint.py.151: total_norm: 17.212208 [I 2022-10-08 23:15:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 1110/2304 [I 2022-10-08 23:15:52] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:16:45 [I 2022-10-08 23:15:52] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00316 [I 2022-10-08 23:15:52] ...trap/engines/engine.py.230: loss: 0.21821 [I 2022-10-08 23:15:52] ...optimizers/trijoint.py.151: total_norm: 16.793133 [I 2022-10-08 23:16:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 1120/2304 [I 2022-10-08 23:16:00] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:37 [I 2022-10-08 23:16:00] ...trap/engines/engine.py.229: process: 0.84532 | load: 0.00301 [I 2022-10-08 23:16:00] ...trap/engines/engine.py.230: loss: 0.15022 [I 2022-10-08 23:16:00] ...optimizers/trijoint.py.151: total_norm: 14.061488 [I 2022-10-08 23:16:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 1130/2304 [I 2022-10-08 23:16:09] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:21 [I 2022-10-08 23:16:09] ...trap/engines/engine.py.229: process: 0.82123 | load: 0.00299 [I 2022-10-08 23:16:09] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-08 23:16:09] ...optimizers/trijoint.py.151: total_norm: 19.272705 [I 2022-10-08 23:16:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 1140/2304 [I 2022-10-08 23:16:17] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:17 [I 2022-10-08 23:16:17] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00300 [I 2022-10-08 23:16:17] ...trap/engines/engine.py.230: loss: 0.18468 [I 2022-10-08 23:16:17] ...optimizers/trijoint.py.151: total_norm: 21.445082 [I 2022-10-08 23:16:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 1150/2304 [I 2022-10-08 23:16:26] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:06 [I 2022-10-08 23:16:26] ...trap/engines/engine.py.229: process: 0.83110 | load: 0.00298 [I 2022-10-08 23:16:26] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-08 23:16:26] ...optimizers/trijoint.py.151: total_norm: 14.136827 [I 2022-10-08 23:16:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 1160/2304 [I 2022-10-08 23:16:34] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:15:59 [I 2022-10-08 23:16:34] ...trap/engines/engine.py.229: process: 0.85287 | load: 0.00300 [I 2022-10-08 23:16:34] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-08 23:16:34] ...optimizers/trijoint.py.151: total_norm: 14.832223 [I 2022-10-08 23:16:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 1170/2304 [I 2022-10-08 23:16:42] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:15:47 [I 2022-10-08 23:16:42] ...trap/engines/engine.py.229: process: 0.82485 | load: 0.00296 [I 2022-10-08 23:16:42] ...trap/engines/engine.py.230: loss: 0.26342 [I 2022-10-08 23:16:42] ...optimizers/trijoint.py.151: total_norm: 16.737492 [I 2022-10-08 23:16:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 1180/2304 [I 2022-10-08 23:16:51] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:15:37 [I 2022-10-08 23:16:51] ...trap/engines/engine.py.229: process: 0.83479 | load: 0.00302 [I 2022-10-08 23:16:51] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-08 23:16:51] ...optimizers/trijoint.py.151: total_norm: 17.022392 [I 2022-10-08 23:16:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 1190/2304 [I 2022-10-08 23:16:59] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:15:28 [I 2022-10-08 23:16:59] ...trap/engines/engine.py.229: process: 0.84081 | load: 0.00302 [I 2022-10-08 23:16:59] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-08 23:16:59] ...optimizers/trijoint.py.151: total_norm: 19.932583 [I 2022-10-08 23:17:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 1200/2304 [I 2022-10-08 23:17:08] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:26 [I 2022-10-08 23:17:08] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00305 [I 2022-10-08 23:17:08] ...trap/engines/engine.py.230: loss: 0.12057 [I 2022-10-08 23:17:08] ...optimizers/trijoint.py.151: total_norm: 12.450500 [I 2022-10-08 23:17:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 1210/2304 [I 2022-10-08 23:17:16] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:15 [I 2022-10-08 23:17:16] ...trap/engines/engine.py.229: process: 0.82622 | load: 0.00299 [I 2022-10-08 23:17:16] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-08 23:17:16] ...optimizers/trijoint.py.151: total_norm: 19.400059 [I 2022-10-08 23:17:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1220/2304 [I 2022-10-08 23:17:25] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:10 [I 2022-10-08 23:17:25] ...trap/engines/engine.py.229: process: 0.83885 | load: 0.00304 [I 2022-10-08 23:17:25] ...trap/engines/engine.py.230: loss: 0.27271 [I 2022-10-08 23:17:25] ...optimizers/trijoint.py.151: total_norm: 16.492979 [I 2022-10-08 23:17:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 1230/2304 [I 2022-10-08 23:17:33] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:01 [I 2022-10-08 23:17:33] ...trap/engines/engine.py.229: process: 0.82764 | load: 0.00304 [I 2022-10-08 23:17:33] ...trap/engines/engine.py.230: loss: 0.15596 [I 2022-10-08 23:17:33] ...optimizers/trijoint.py.151: total_norm: 17.390331 [I 2022-10-08 23:17:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 1240/2304 [I 2022-10-08 23:17:41] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:14:56 [I 2022-10-08 23:17:41] ...trap/engines/engine.py.229: process: 0.85928 | load: 0.00306 [I 2022-10-08 23:17:41] ...trap/engines/engine.py.230: loss: 0.17492 [I 2022-10-08 23:17:41] ...optimizers/trijoint.py.151: total_norm: 22.596893 [I 2022-10-08 23:17:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 1250/2304 [I 2022-10-08 23:17:50] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:14:48 [I 2022-10-08 23:17:50] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00311 [I 2022-10-08 23:17:50] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-08 23:17:50] ...optimizers/trijoint.py.151: total_norm: 18.177505 [I 2022-10-08 23:17:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 1260/2304 [I 2022-10-08 23:17:58] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:14:41 [I 2022-10-08 23:17:58] ...trap/engines/engine.py.229: process: 0.84197 | load: 0.00308 [I 2022-10-08 23:17:58] ...trap/engines/engine.py.230: loss: 0.19279 [I 2022-10-08 23:17:58] ...optimizers/trijoint.py.151: total_norm: 23.056486 [I 2022-10-08 23:18:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 1270/2304 [I 2022-10-08 23:18:07] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:28 [I 2022-10-08 23:18:07] ...trap/engines/engine.py.229: process: 0.83367 | load: 0.00307 [I 2022-10-08 23:18:07] ...trap/engines/engine.py.230: loss: 0.25319 [I 2022-10-08 23:18:07] ...optimizers/trijoint.py.151: total_norm: 22.027960 [I 2022-10-08 23:18:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 1280/2304 [I 2022-10-08 23:18:15] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:19 [I 2022-10-08 23:18:15] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00314 [I 2022-10-08 23:18:15] ...trap/engines/engine.py.230: loss: 0.12975 [I 2022-10-08 23:18:15] ...optimizers/trijoint.py.151: total_norm: 18.800501 [I 2022-10-08 23:18:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 1290/2304 [I 2022-10-08 23:18:23] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:14 [I 2022-10-08 23:18:23] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00308 [I 2022-10-08 23:18:23] ...trap/engines/engine.py.230: loss: 0.11807 [I 2022-10-08 23:18:24] ...optimizers/trijoint.py.151: total_norm: 11.790339 [I 2022-10-08 23:18:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 1300/2304 [I 2022-10-08 23:18:32] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:07 [I 2022-10-08 23:18:32] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00311 [I 2022-10-08 23:18:32] ...trap/engines/engine.py.230: loss: 0.24029 [I 2022-10-08 23:18:32] ...optimizers/trijoint.py.151: total_norm: 22.860418 [I 2022-10-08 23:18:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 1310/2304 [I 2022-10-08 23:18:40] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:13:55 [I 2022-10-08 23:18:40] ...trap/engines/engine.py.229: process: 0.83748 | load: 0.00302 [I 2022-10-08 23:18:40] ...trap/engines/engine.py.230: loss: 0.19425 [I 2022-10-08 23:18:40] ...optimizers/trijoint.py.151: total_norm: 18.888977 [I 2022-10-08 23:18:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 1320/2304 [I 2022-10-08 23:18:49] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:13:48 [I 2022-10-08 23:18:49] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00300 [I 2022-10-08 23:18:49] ...trap/engines/engine.py.230: loss: 0.14240 [I 2022-10-08 23:18:49] ...optimizers/trijoint.py.151: total_norm: 26.510551 [I 2022-10-08 23:18:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 1330/2304 [I 2022-10-08 23:18:57] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:35 [I 2022-10-08 23:18:57] ...trap/engines/engine.py.229: process: 0.82647 | load: 0.00301 [I 2022-10-08 23:18:57] ...trap/engines/engine.py.230: loss: 0.29004 [I 2022-10-08 23:18:57] ...optimizers/trijoint.py.151: total_norm: 21.119835 [I 2022-10-08 23:19:06] ...trap/engines/engine.py.224: train: epoch 23 | batch 1340/2304 [I 2022-10-08 23:19:06] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:30 [I 2022-10-08 23:19:06] ...trap/engines/engine.py.229: process: 0.83363 | load: 0.00300 [I 2022-10-08 23:19:06] ...trap/engines/engine.py.230: loss: 0.17448 [I 2022-10-08 23:19:06] ...optimizers/trijoint.py.151: total_norm: 13.909226 [I 2022-10-08 23:19:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 1350/2304 [I 2022-10-08 23:19:14] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:18 [I 2022-10-08 23:19:14] ...trap/engines/engine.py.229: process: 0.82189 | load: 0.00299 [I 2022-10-08 23:19:14] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-08 23:19:14] ...optimizers/trijoint.py.151: total_norm: 11.607484 [I 2022-10-08 23:19:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 1360/2304 [I 2022-10-08 23:19:22] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:10 [I 2022-10-08 23:19:22] ...trap/engines/engine.py.229: process: 0.82382 | load: 0.00306 [I 2022-10-08 23:19:22] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-08 23:19:22] ...optimizers/trijoint.py.151: total_norm: 25.207380 [I 2022-10-08 23:19:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 1370/2304 [I 2022-10-08 23:19:31] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:03 [I 2022-10-08 23:19:31] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00300 [I 2022-10-08 23:19:31] ...trap/engines/engine.py.230: loss: 0.32830 [I 2022-10-08 23:19:31] ...optimizers/trijoint.py.151: total_norm: 28.793512 [I 2022-10-08 23:19:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 1380/2304 [I 2022-10-08 23:19:39] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:12:52 [I 2022-10-08 23:19:39] ...trap/engines/engine.py.229: process: 0.83578 | load: 0.00298 [I 2022-10-08 23:19:39] ...trap/engines/engine.py.230: loss: 0.20267 [I 2022-10-08 23:19:39] ...optimizers/trijoint.py.151: total_norm: 17.835821 [I 2022-10-08 23:19:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 1390/2304 [I 2022-10-08 23:19:48] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:12:41 [I 2022-10-08 23:19:48] ...trap/engines/engine.py.229: process: 0.83145 | load: 0.00306 [I 2022-10-08 23:19:48] ...trap/engines/engine.py.230: loss: 0.18839 [I 2022-10-08 23:19:48] ...optimizers/trijoint.py.151: total_norm: 24.194809 [I 2022-10-08 23:19:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 1400/2304 [I 2022-10-08 23:19:56] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:12:37 [I 2022-10-08 23:19:56] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00301 [I 2022-10-08 23:19:56] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-08 23:19:56] ...optimizers/trijoint.py.151: total_norm: 17.784981 [I 2022-10-08 23:20:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 1410/2304 [I 2022-10-08 23:20:04] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:30 [I 2022-10-08 23:20:04] ...trap/engines/engine.py.229: process: 0.82289 | load: 0.00303 [I 2022-10-08 23:20:04] ...trap/engines/engine.py.230: loss: 0.17630 [I 2022-10-08 23:20:04] ...optimizers/trijoint.py.151: total_norm: 24.867323 [I 2022-10-08 23:20:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 1420/2304 [I 2022-10-08 23:20:13] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:23 [I 2022-10-08 23:20:13] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00301 [I 2022-10-08 23:20:13] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-08 23:20:13] ...optimizers/trijoint.py.151: total_norm: 10.070954 [I 2022-10-08 23:20:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 1430/2304 [I 2022-10-08 23:20:21] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:12 [I 2022-10-08 23:20:21] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00303 [I 2022-10-08 23:20:21] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-08 23:20:21] ...optimizers/trijoint.py.151: total_norm: 19.394539 [I 2022-10-08 23:20:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 1440/2304 [I 2022-10-08 23:20:30] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:08 [I 2022-10-08 23:20:30] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00303 [I 2022-10-08 23:20:30] ...trap/engines/engine.py.230: loss: 0.24474 [I 2022-10-08 23:20:30] ...optimizers/trijoint.py.151: total_norm: 40.883408 [I 2022-10-08 23:20:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 1450/2304 [I 2022-10-08 23:20:38] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:11:57 [I 2022-10-08 23:20:38] ...trap/engines/engine.py.229: process: 0.83742 | load: 0.00299 [I 2022-10-08 23:20:38] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-08 23:20:38] ...optimizers/trijoint.py.151: total_norm: 12.346290 [I 2022-10-08 23:20:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 1460/2304 [I 2022-10-08 23:20:47] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:11:46 [I 2022-10-08 23:20:47] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00306 [I 2022-10-08 23:20:47] ...trap/engines/engine.py.230: loss: 0.21366 [I 2022-10-08 23:20:47] ...optimizers/trijoint.py.151: total_norm: 18.275438 [I 2022-10-08 23:20:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 1470/2304 [I 2022-10-08 23:20:55] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:11:39 [I 2022-10-08 23:20:55] ...trap/engines/engine.py.229: process: 0.82933 | load: 0.00301 [I 2022-10-08 23:20:55] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-08 23:20:55] ...optimizers/trijoint.py.151: total_norm: 23.922041 [I 2022-10-08 23:21:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 1480/2304 [I 2022-10-08 23:21:03] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:30 [I 2022-10-08 23:21:03] ...trap/engines/engine.py.229: process: 0.82760 | load: 0.00299 [I 2022-10-08 23:21:03] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-08 23:21:03] ...optimizers/trijoint.py.151: total_norm: 10.669024 [I 2022-10-08 23:21:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 1490/2304 [I 2022-10-08 23:21:12] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:24 [I 2022-10-08 23:21:12] ...trap/engines/engine.py.229: process: 0.85548 | load: 0.00299 [I 2022-10-08 23:21:12] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-08 23:21:12] ...optimizers/trijoint.py.151: total_norm: 12.953010 [I 2022-10-08 23:21:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 1500/2304 [I 2022-10-08 23:21:20] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:13 [I 2022-10-08 23:21:20] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00299 [I 2022-10-08 23:21:20] ...trap/engines/engine.py.230: loss: 0.26073 [I 2022-10-08 23:21:20] ...optimizers/trijoint.py.151: total_norm: 20.124472 [I 2022-10-08 23:21:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 1510/2304 [I 2022-10-08 23:21:29] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:04 [I 2022-10-08 23:21:29] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00301 [I 2022-10-08 23:21:29] ...trap/engines/engine.py.230: loss: 0.19543 [I 2022-10-08 23:21:29] ...optimizers/trijoint.py.151: total_norm: 23.774698 [I 2022-10-08 23:21:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 1520/2304 [I 2022-10-08 23:21:38] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:12:03 [I 2022-10-08 23:21:38] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00299 [I 2022-10-08 23:21:38] ...trap/engines/engine.py.230: loss: 0.21601 [I 2022-10-08 23:21:38] ...optimizers/trijoint.py.151: total_norm: 23.570822 [I 2022-10-08 23:21:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 1530/2304 [I 2022-10-08 23:21:47] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:10:57 [I 2022-10-08 23:21:47] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00299 [I 2022-10-08 23:21:47] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-08 23:21:47] ...optimizers/trijoint.py.151: total_norm: 14.164999 [I 2022-10-08 23:21:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 1540/2304 [I 2022-10-08 23:21:55] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:40 [I 2022-10-08 23:21:55] ...trap/engines/engine.py.229: process: 0.83453 | load: 0.00301 [I 2022-10-08 23:21:55] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-08 23:21:55] ...optimizers/trijoint.py.151: total_norm: 9.037141 [I 2022-10-08 23:22:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 1550/2304 [I 2022-10-08 23:22:03] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:32 [I 2022-10-08 23:22:03] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00300 [I 2022-10-08 23:22:03] ...trap/engines/engine.py.230: loss: 0.22014 [I 2022-10-08 23:22:03] ...optimizers/trijoint.py.151: total_norm: 22.259874 [I 2022-10-08 23:22:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 1560/2304 [I 2022-10-08 23:22:12] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:25 [I 2022-10-08 23:22:12] ...trap/engines/engine.py.229: process: 0.84556 | load: 0.00302 [I 2022-10-08 23:22:12] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-08 23:22:12] ...optimizers/trijoint.py.151: total_norm: 22.783342 [I 2022-10-08 23:22:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 1570/2304 [I 2022-10-08 23:22:20] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:14 [I 2022-10-08 23:22:20] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00310 [I 2022-10-08 23:22:20] ...trap/engines/engine.py.230: loss: 0.14395 [I 2022-10-08 23:22:20] ...optimizers/trijoint.py.151: total_norm: 18.095285 [I 2022-10-08 23:22:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 1580/2304 [I 2022-10-08 23:22:29] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:07 [I 2022-10-08 23:22:29] ...trap/engines/engine.py.229: process: 0.83429 | load: 0.00299 [I 2022-10-08 23:22:29] ...trap/engines/engine.py.230: loss: 0.17652 [I 2022-10-08 23:22:29] ...optimizers/trijoint.py.151: total_norm: 19.800520 [I 2022-10-08 23:22:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 1590/2304 [I 2022-10-08 23:22:37] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:09:54 [I 2022-10-08 23:22:37] ...trap/engines/engine.py.229: process: 0.83133 | load: 0.00304 [I 2022-10-08 23:22:37] ...trap/engines/engine.py.230: loss: 0.18516 [I 2022-10-08 23:22:37] ...optimizers/trijoint.py.151: total_norm: 19.595343 [I 2022-10-08 23:22:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 1600/2304 [I 2022-10-08 23:22:45] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:49 [I 2022-10-08 23:22:45] ...trap/engines/engine.py.229: process: 0.82896 | load: 0.00301 [I 2022-10-08 23:22:45] ...trap/engines/engine.py.230: loss: 0.13823 [I 2022-10-08 23:22:45] ...optimizers/trijoint.py.151: total_norm: 20.675678 [I 2022-10-08 23:22:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 1610/2304 [I 2022-10-08 23:22:54] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:44 [I 2022-10-08 23:22:54] ...trap/engines/engine.py.229: process: 0.83862 | load: 0.00300 [I 2022-10-08 23:22:54] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-08 23:22:54] ...optimizers/trijoint.py.151: total_norm: 18.732365 [I 2022-10-08 23:23:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 1620/2304 [I 2022-10-08 23:23:02] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:33 [I 2022-10-08 23:23:02] ...trap/engines/engine.py.229: process: 0.83338 | load: 0.00299 [I 2022-10-08 23:23:02] ...trap/engines/engine.py.230: loss: 0.18771 [I 2022-10-08 23:23:02] ...optimizers/trijoint.py.151: total_norm: 18.038437 [I 2022-10-08 23:23:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 1630/2304 [I 2022-10-08 23:23:11] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:24 [I 2022-10-08 23:23:11] ...trap/engines/engine.py.229: process: 0.83510 | load: 0.00298 [I 2022-10-08 23:23:11] ...trap/engines/engine.py.230: loss: 0.17482 [I 2022-10-08 23:23:11] ...optimizers/trijoint.py.151: total_norm: 21.921757 [I 2022-10-08 23:23:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 1640/2304 [I 2022-10-08 23:23:19] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:16 [I 2022-10-08 23:23:19] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00299 [I 2022-10-08 23:23:19] ...trap/engines/engine.py.230: loss: 0.25638 [I 2022-10-08 23:23:19] ...optimizers/trijoint.py.151: total_norm: 21.363167 [I 2022-10-08 23:23:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 1650/2304 [I 2022-10-08 23:23:28] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:09 [I 2022-10-08 23:23:28] ...trap/engines/engine.py.229: process: 0.83927 | load: 0.00301 [I 2022-10-08 23:23:28] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-08 23:23:28] ...optimizers/trijoint.py.151: total_norm: 12.933684 [I 2022-10-08 23:23:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 1660/2304 [I 2022-10-08 23:23:36] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:08:58 [I 2022-10-08 23:23:36] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00306 [I 2022-10-08 23:23:36] ...trap/engines/engine.py.230: loss: 0.16751 [I 2022-10-08 23:23:36] ...optimizers/trijoint.py.151: total_norm: 20.352507 [I 2022-10-08 23:23:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 1670/2304 [I 2022-10-08 23:23:44] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:53 [I 2022-10-08 23:23:44] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00301 [I 2022-10-08 23:23:44] ...trap/engines/engine.py.230: loss: 0.19885 [I 2022-10-08 23:23:44] ...optimizers/trijoint.py.151: total_norm: 18.505253 [I 2022-10-08 23:23:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 1680/2304 [I 2022-10-08 23:23:53] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:41 [I 2022-10-08 23:23:53] ...trap/engines/engine.py.229: process: 0.83494 | load: 0.00301 [I 2022-10-08 23:23:53] ...trap/engines/engine.py.230: loss: 0.19240 [I 2022-10-08 23:23:53] ...optimizers/trijoint.py.151: total_norm: 30.161245 [I 2022-10-08 23:24:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 1690/2304 [I 2022-10-08 23:24:01] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:33 [I 2022-10-08 23:24:01] ...trap/engines/engine.py.229: process: 0.82958 | load: 0.00300 [I 2022-10-08 23:24:01] ...trap/engines/engine.py.230: loss: 0.18526 [I 2022-10-08 23:24:01] ...optimizers/trijoint.py.151: total_norm: 22.213369 [I 2022-10-08 23:24:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 1700/2304 [I 2022-10-08 23:24:10] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:26 [I 2022-10-08 23:24:10] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00301 [I 2022-10-08 23:24:10] ...trap/engines/engine.py.230: loss: 0.12942 [I 2022-10-08 23:24:10] ...optimizers/trijoint.py.151: total_norm: 22.285669 [I 2022-10-08 23:24:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 1710/2304 [I 2022-10-08 23:24:18] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:16 [I 2022-10-08 23:24:18] ...trap/engines/engine.py.229: process: 0.82644 | load: 0.00311 [I 2022-10-08 23:24:18] ...trap/engines/engine.py.230: loss: 0.21519 [I 2022-10-08 23:24:18] ...optimizers/trijoint.py.151: total_norm: 20.861055 [I 2022-10-08 23:24:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 1720/2304 [I 2022-10-08 23:24:26] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:08 [I 2022-10-08 23:24:26] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00308 [I 2022-10-08 23:24:26] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-08 23:24:26] ...optimizers/trijoint.py.151: total_norm: 15.729950 [I 2022-10-08 23:24:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 1730/2304 [I 2022-10-08 23:24:35] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:01 [I 2022-10-08 23:24:35] ...trap/engines/engine.py.229: process: 0.83792 | load: 0.00308 [I 2022-10-08 23:24:35] ...trap/engines/engine.py.230: loss: 0.15032 [I 2022-10-08 23:24:35] ...optimizers/trijoint.py.151: total_norm: 17.530224 [I 2022-10-08 23:24:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 1740/2304 [I 2022-10-08 23:24:43] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:53 [I 2022-10-08 23:24:43] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00315 [I 2022-10-08 23:24:43] ...trap/engines/engine.py.230: loss: 0.16937 [I 2022-10-08 23:24:43] ...optimizers/trijoint.py.151: total_norm: 20.675591 [I 2022-10-08 23:24:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 1750/2304 [I 2022-10-08 23:24:52] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:44 [I 2022-10-08 23:24:52] ...trap/engines/engine.py.229: process: 0.83729 | load: 0.00306 [I 2022-10-08 23:24:52] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-08 23:24:52] ...optimizers/trijoint.py.151: total_norm: 9.951385 [I 2022-10-08 23:25:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 1760/2304 [I 2022-10-08 23:25:00] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:34 [I 2022-10-08 23:25:00] ...trap/engines/engine.py.229: process: 0.82828 | load: 0.00309 [I 2022-10-08 23:25:00] ...trap/engines/engine.py.230: loss: 0.12418 [I 2022-10-08 23:25:00] ...optimizers/trijoint.py.151: total_norm: 17.436819 [I 2022-10-08 23:25:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 1770/2304 [I 2022-10-08 23:25:08] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:29 [I 2022-10-08 23:25:08] ...trap/engines/engine.py.229: process: 0.83724 | load: 0.00309 [I 2022-10-08 23:25:08] ...trap/engines/engine.py.230: loss: 0.16796 [I 2022-10-08 23:25:08] ...optimizers/trijoint.py.151: total_norm: 36.415203 [I 2022-10-08 23:25:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 1780/2304 [I 2022-10-08 23:25:17] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:16 [I 2022-10-08 23:25:17] ...trap/engines/engine.py.229: process: 0.83260 | load: 0.00310 [I 2022-10-08 23:25:17] ...trap/engines/engine.py.230: loss: 0.17682 [I 2022-10-08 23:25:17] ...optimizers/trijoint.py.151: total_norm: 11.272109 [I 2022-10-08 23:25:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1790/2304 [I 2022-10-08 23:25:25] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:13 [I 2022-10-08 23:25:25] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00308 [I 2022-10-08 23:25:25] ...trap/engines/engine.py.230: loss: 0.12626 [I 2022-10-08 23:25:25] ...optimizers/trijoint.py.151: total_norm: 15.858603 [I 2022-10-08 23:25:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 1800/2304 [I 2022-10-08 23:25:34] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:00 [I 2022-10-08 23:25:34] ...trap/engines/engine.py.229: process: 0.82457 | load: 0.00302 [I 2022-10-08 23:25:34] ...trap/engines/engine.py.230: loss: 0.17668 [I 2022-10-08 23:25:34] ...optimizers/trijoint.py.151: total_norm: 16.284323 [I 2022-10-08 23:25:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 1810/2304 [I 2022-10-08 23:25:42] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:53 [I 2022-10-08 23:25:42] ...trap/engines/engine.py.229: process: 0.82579 | load: 0.00303 [I 2022-10-08 23:25:42] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-08 23:25:42] ...optimizers/trijoint.py.151: total_norm: 16.536757 [I 2022-10-08 23:25:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 1820/2304 [I 2022-10-08 23:25:50] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:45 [I 2022-10-08 23:25:50] ...trap/engines/engine.py.229: process: 0.83536 | load: 0.00307 [I 2022-10-08 23:25:50] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-08 23:25:50] ...optimizers/trijoint.py.151: total_norm: 24.660608 [I 2022-10-08 23:25:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 1830/2304 [I 2022-10-08 23:25:59] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:39 [I 2022-10-08 23:25:59] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00303 [I 2022-10-08 23:25:59] ...trap/engines/engine.py.230: loss: 0.23837 [I 2022-10-08 23:25:59] ...optimizers/trijoint.py.151: total_norm: 16.682716 [I 2022-10-08 23:26:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 1840/2304 [I 2022-10-08 23:26:07] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:29 [I 2022-10-08 23:26:07] ...trap/engines/engine.py.229: process: 0.82854 | load: 0.00300 [I 2022-10-08 23:26:07] ...trap/engines/engine.py.230: loss: 0.23458 [I 2022-10-08 23:26:07] ...optimizers/trijoint.py.151: total_norm: 19.664932 [I 2022-10-08 23:26:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 1850/2304 [I 2022-10-08 23:26:16] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:29 [I 2022-10-08 23:26:16] ...trap/engines/engine.py.229: process: 0.83125 | load: 0.00300 [I 2022-10-08 23:26:16] ...trap/engines/engine.py.230: loss: 0.18993 [I 2022-10-08 23:26:16] ...optimizers/trijoint.py.151: total_norm: 13.866624 [I 2022-10-08 23:26:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1860/2304 [I 2022-10-08 23:26:25] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:12 [I 2022-10-08 23:26:25] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00299 [I 2022-10-08 23:26:25] ...trap/engines/engine.py.230: loss: 0.21097 [I 2022-10-08 23:26:25] ...optimizers/trijoint.py.151: total_norm: 16.633993 [I 2022-10-08 23:26:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 1870/2304 [I 2022-10-08 23:26:33] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:03 [I 2022-10-08 23:26:33] ...trap/engines/engine.py.229: process: 0.83120 | load: 0.00301 [I 2022-10-08 23:26:33] ...trap/engines/engine.py.230: loss: 0.18877 [I 2022-10-08 23:26:33] ...optimizers/trijoint.py.151: total_norm: 22.531378 [I 2022-10-08 23:26:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 1880/2304 [I 2022-10-08 23:26:41] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:55 [I 2022-10-08 23:26:41] ...trap/engines/engine.py.229: process: 0.83418 | load: 0.00296 [I 2022-10-08 23:26:41] ...trap/engines/engine.py.230: loss: 0.27804 [I 2022-10-08 23:26:41] ...optimizers/trijoint.py.151: total_norm: 16.890713 [I 2022-10-08 23:26:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 1890/2304 [I 2022-10-08 23:26:50] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:47 [I 2022-10-08 23:26:50] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00303 [I 2022-10-08 23:26:50] ...trap/engines/engine.py.230: loss: 0.17862 [I 2022-10-08 23:26:50] ...optimizers/trijoint.py.151: total_norm: 16.345993 [I 2022-10-08 23:26:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 1900/2304 [I 2022-10-08 23:26:58] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:38 [I 2022-10-08 23:26:58] ...trap/engines/engine.py.229: process: 0.84163 | load: 0.00299 [I 2022-10-08 23:26:58] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-08 23:26:58] ...optimizers/trijoint.py.151: total_norm: 22.003963 [I 2022-10-08 23:27:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 1910/2304 [I 2022-10-08 23:27:07] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:31 [I 2022-10-08 23:27:07] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00298 [I 2022-10-08 23:27:07] ...trap/engines/engine.py.230: loss: 0.22118 [I 2022-10-08 23:27:07] ...optimizers/trijoint.py.151: total_norm: 16.849422 [I 2022-10-08 23:27:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 1920/2304 [I 2022-10-08 23:27:15] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:23 [I 2022-10-08 23:27:15] ...trap/engines/engine.py.229: process: 0.83954 | load: 0.00299 [I 2022-10-08 23:27:15] ...trap/engines/engine.py.230: loss: 0.16119 [I 2022-10-08 23:27:15] ...optimizers/trijoint.py.151: total_norm: 14.506915 [I 2022-10-08 23:27:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 1930/2304 [I 2022-10-08 23:27:23] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:13 [I 2022-10-08 23:27:23] ...trap/engines/engine.py.229: process: 0.83705 | load: 0.00300 [I 2022-10-08 23:27:23] ...trap/engines/engine.py.230: loss: 0.21011 [I 2022-10-08 23:27:23] ...optimizers/trijoint.py.151: total_norm: 15.235271 [I 2022-10-08 23:27:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 1940/2304 [I 2022-10-08 23:27:32] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:05 [I 2022-10-08 23:27:32] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00302 [I 2022-10-08 23:27:32] ...trap/engines/engine.py.230: loss: 0.14907 [I 2022-10-08 23:27:32] ...optimizers/trijoint.py.151: total_norm: 22.218704 [I 2022-10-08 23:27:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 1950/2304 [I 2022-10-08 23:27:40] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:56 [I 2022-10-08 23:27:40] ...trap/engines/engine.py.229: process: 0.82528 | load: 0.00296 [I 2022-10-08 23:27:40] ...trap/engines/engine.py.230: loss: 0.10268 [I 2022-10-08 23:27:40] ...optimizers/trijoint.py.151: total_norm: 12.750109 [I 2022-10-08 23:27:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 1960/2304 [I 2022-10-08 23:27:49] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:48 [I 2022-10-08 23:27:49] ...trap/engines/engine.py.229: process: 0.83613 | load: 0.00300 [I 2022-10-08 23:27:49] ...trap/engines/engine.py.230: loss: 0.31910 [I 2022-10-08 23:27:49] ...optimizers/trijoint.py.151: total_norm: 23.248116 [I 2022-10-08 23:27:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 1970/2304 [I 2022-10-08 23:27:57] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:41 [I 2022-10-08 23:27:57] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00298 [I 2022-10-08 23:27:57] ...trap/engines/engine.py.230: loss: 0.32024 [I 2022-10-08 23:27:57] ...optimizers/trijoint.py.151: total_norm: 18.903660 [I 2022-10-08 23:28:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 1980/2304 [I 2022-10-08 23:28:05] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:31 [I 2022-10-08 23:28:05] ...trap/engines/engine.py.229: process: 0.84867 | load: 0.00302 [I 2022-10-08 23:28:05] ...trap/engines/engine.py.230: loss: 0.18072 [I 2022-10-08 23:28:05] ...optimizers/trijoint.py.151: total_norm: 20.259987 [I 2022-10-08 23:28:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 1990/2304 [I 2022-10-08 23:28:14] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:22 [I 2022-10-08 23:28:14] ...trap/engines/engine.py.229: process: 0.83557 | load: 0.00304 [I 2022-10-08 23:28:14] ...trap/engines/engine.py.230: loss: 0.18302 [I 2022-10-08 23:28:14] ...optimizers/trijoint.py.151: total_norm: 14.848498 [I 2022-10-08 23:28:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 2000/2304 [I 2022-10-08 23:28:22] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:15 [I 2022-10-08 23:28:22] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00300 [I 2022-10-08 23:28:22] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-08 23:28:22] ...optimizers/trijoint.py.151: total_norm: 16.384613 [I 2022-10-08 23:28:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 2010/2304 [I 2022-10-08 23:28:31] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:06 [I 2022-10-08 23:28:31] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00301 [I 2022-10-08 23:28:31] ...trap/engines/engine.py.230: loss: 0.18197 [I 2022-10-08 23:28:31] ...optimizers/trijoint.py.151: total_norm: 13.917206 [I 2022-10-08 23:28:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 2020/2304 [I 2022-10-08 23:28:39] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:56 [I 2022-10-08 23:28:39] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00304 [I 2022-10-08 23:28:39] ...trap/engines/engine.py.230: loss: 0.14525 [I 2022-10-08 23:28:39] ...optimizers/trijoint.py.151: total_norm: 18.420382 [I 2022-10-08 23:28:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 2030/2304 [I 2022-10-08 23:28:48] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:04 [I 2022-10-08 23:28:48] ...trap/engines/engine.py.229: process: 1.12929 | load: 0.00303 [I 2022-10-08 23:28:48] ...trap/engines/engine.py.230: loss: 0.25745 [I 2022-10-08 23:28:48] ...optimizers/trijoint.py.151: total_norm: 45.338196 [I 2022-10-08 23:28:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 2040/2304 [I 2022-10-08 23:28:56] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:42 [I 2022-10-08 23:28:56] ...trap/engines/engine.py.229: process: 0.83875 | load: 0.00301 [I 2022-10-08 23:28:56] ...trap/engines/engine.py.230: loss: 0.14738 [I 2022-10-08 23:28:56] ...optimizers/trijoint.py.151: total_norm: 23.634502 [I 2022-10-08 23:29:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 2050/2304 [I 2022-10-08 23:29:04] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:31 [I 2022-10-08 23:29:04] ...trap/engines/engine.py.229: process: 0.82759 | load: 0.00303 [I 2022-10-08 23:29:04] ...trap/engines/engine.py.230: loss: 0.18472 [I 2022-10-08 23:29:04] ...optimizers/trijoint.py.151: total_norm: 34.749084 [I 2022-10-08 23:29:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 2060/2304 [I 2022-10-08 23:29:13] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:27 [I 2022-10-08 23:29:13] ...trap/engines/engine.py.229: process: 0.83581 | load: 0.00304 [I 2022-10-08 23:29:13] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-08 23:29:13] ...optimizers/trijoint.py.151: total_norm: 20.440933 [I 2022-10-08 23:29:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 2070/2304 [I 2022-10-08 23:29:21] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:16 [I 2022-10-08 23:29:21] ...trap/engines/engine.py.229: process: 0.82816 | load: 0.00306 [I 2022-10-08 23:29:21] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-08 23:29:21] ...optimizers/trijoint.py.151: total_norm: 17.381948 [I 2022-10-08 23:29:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 2080/2304 [I 2022-10-08 23:29:30] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:07 [I 2022-10-08 23:29:30] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00303 [I 2022-10-08 23:29:30] ...trap/engines/engine.py.230: loss: 0.13764 [I 2022-10-08 23:29:30] ...optimizers/trijoint.py.151: total_norm: 15.299004 [I 2022-10-08 23:29:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 2090/2304 [I 2022-10-08 23:29:38] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:58 [I 2022-10-08 23:29:38] ...trap/engines/engine.py.229: process: 0.81962 | load: 0.00303 [I 2022-10-08 23:29:38] ...trap/engines/engine.py.230: loss: 0.20625 [I 2022-10-08 23:29:38] ...optimizers/trijoint.py.151: total_norm: 12.092840 [I 2022-10-08 23:29:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 2100/2304 [I 2022-10-08 23:29:47] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:50 [I 2022-10-08 23:29:47] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00301 [I 2022-10-08 23:29:47] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-08 23:29:47] ...optimizers/trijoint.py.151: total_norm: 24.313616 [I 2022-10-08 23:29:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 2110/2304 [I 2022-10-08 23:29:55] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:41 [I 2022-10-08 23:29:55] ...trap/engines/engine.py.229: process: 0.83301 | load: 0.00305 [I 2022-10-08 23:29:55] ...trap/engines/engine.py.230: loss: 0.21918 [I 2022-10-08 23:29:55] ...optimizers/trijoint.py.151: total_norm: 27.362379 [I 2022-10-08 23:30:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 2120/2304 [I 2022-10-08 23:30:03] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:33 [I 2022-10-08 23:30:03] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00300 [I 2022-10-08 23:30:03] ...trap/engines/engine.py.230: loss: 0.24078 [I 2022-10-08 23:30:03] ...optimizers/trijoint.py.151: total_norm: 23.346889 [I 2022-10-08 23:30:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 2130/2304 [I 2022-10-08 23:30:12] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:25 [I 2022-10-08 23:30:12] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00309 [I 2022-10-08 23:30:12] ...trap/engines/engine.py.230: loss: 0.25852 [I 2022-10-08 23:30:12] ...optimizers/trijoint.py.151: total_norm: 21.687117 [I 2022-10-08 23:30:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 2140/2304 [I 2022-10-08 23:30:20] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:16 [I 2022-10-08 23:30:20] ...trap/engines/engine.py.229: process: 0.83220 | load: 0.00311 [I 2022-10-08 23:30:20] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-08 23:30:20] ...optimizers/trijoint.py.151: total_norm: 16.199490 [I 2022-10-08 23:30:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 2150/2304 [I 2022-10-08 23:30:29] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:08 [I 2022-10-08 23:30:29] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00312 [I 2022-10-08 23:30:29] ...trap/engines/engine.py.230: loss: 0.16972 [I 2022-10-08 23:30:29] ...optimizers/trijoint.py.151: total_norm: 25.606731 [I 2022-10-08 23:30:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 2160/2304 [I 2022-10-08 23:30:37] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:00 [I 2022-10-08 23:30:37] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00312 [I 2022-10-08 23:30:37] ...trap/engines/engine.py.230: loss: 0.17500 [I 2022-10-08 23:30:37] ...optimizers/trijoint.py.151: total_norm: 13.309731 [I 2022-10-08 23:30:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 2170/2304 [I 2022-10-08 23:30:45] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:52 [I 2022-10-08 23:30:45] ...trap/engines/engine.py.229: process: 0.83371 | load: 0.00302 [I 2022-10-08 23:30:45] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-08 23:30:45] ...optimizers/trijoint.py.151: total_norm: 29.566305 [I 2022-10-08 23:30:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 2180/2304 [I 2022-10-08 23:30:54] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:01:43 [I 2022-10-08 23:30:54] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00300 [I 2022-10-08 23:30:54] ...trap/engines/engine.py.230: loss: 0.21575 [I 2022-10-08 23:30:54] ...optimizers/trijoint.py.151: total_norm: 20.469011 [I 2022-10-08 23:31:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 2190/2304 [I 2022-10-08 23:31:02] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:35 [I 2022-10-08 23:31:02] ...trap/engines/engine.py.229: process: 0.83273 | load: 0.00306 [I 2022-10-08 23:31:02] ...trap/engines/engine.py.230: loss: 0.16800 [I 2022-10-08 23:31:02] ...optimizers/trijoint.py.151: total_norm: 16.203291 [I 2022-10-08 23:31:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 2200/2304 [I 2022-10-08 23:31:10] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:26 [I 2022-10-08 23:31:10] ...trap/engines/engine.py.229: process: 0.83869 | load: 0.00305 [I 2022-10-08 23:31:10] ...trap/engines/engine.py.230: loss: 0.13158 [I 2022-10-08 23:31:10] ...optimizers/trijoint.py.151: total_norm: 12.351363 [I 2022-10-08 23:31:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 2210/2304 [I 2022-10-08 23:31:19] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:18 [I 2022-10-08 23:31:19] ...trap/engines/engine.py.229: process: 0.83300 | load: 0.00299 [I 2022-10-08 23:31:19] ...trap/engines/engine.py.230: loss: 0.29116 [I 2022-10-08 23:31:19] ...optimizers/trijoint.py.151: total_norm: 20.177448 [I 2022-10-08 23:31:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 2220/2304 [I 2022-10-08 23:31:27] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:10 [I 2022-10-08 23:31:27] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00300 [I 2022-10-08 23:31:27] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-08 23:31:27] ...optimizers/trijoint.py.151: total_norm: 18.798780 [I 2022-10-08 23:31:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 2230/2304 [I 2022-10-08 23:31:36] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:01 [I 2022-10-08 23:31:36] ...trap/engines/engine.py.229: process: 0.83444 | load: 0.00301 [I 2022-10-08 23:31:36] ...trap/engines/engine.py.230: loss: 0.13038 [I 2022-10-08 23:31:36] ...optimizers/trijoint.py.151: total_norm: 16.764149 [I 2022-10-08 23:31:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 2240/2304 [I 2022-10-08 23:31:44] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:53 [I 2022-10-08 23:31:44] ...trap/engines/engine.py.229: process: 0.84357 | load: 0.00307 [I 2022-10-08 23:31:44] ...trap/engines/engine.py.230: loss: 0.23382 [I 2022-10-08 23:31:44] ...optimizers/trijoint.py.151: total_norm: 23.038177 [I 2022-10-08 23:31:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 2250/2304 [I 2022-10-08 23:31:52] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:00:45 [I 2022-10-08 23:31:52] ...trap/engines/engine.py.229: process: 0.83186 | load: 0.00299 [I 2022-10-08 23:31:52] ...trap/engines/engine.py.230: loss: 0.16135 [I 2022-10-08 23:31:52] ...optimizers/trijoint.py.151: total_norm: 16.998302 [I 2022-10-08 23:32:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 2260/2304 [I 2022-10-08 23:32:01] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:36 [I 2022-10-08 23:32:01] ...trap/engines/engine.py.229: process: 0.82677 | load: 0.00306 [I 2022-10-08 23:32:01] ...trap/engines/engine.py.230: loss: 0.10550 [I 2022-10-08 23:32:01] ...optimizers/trijoint.py.151: total_norm: 9.881083 [I 2022-10-08 23:32:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 2270/2304 [I 2022-10-08 23:32:09] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:28 [I 2022-10-08 23:32:09] ...trap/engines/engine.py.229: process: 0.82597 | load: 0.00306 [I 2022-10-08 23:32:09] ...trap/engines/engine.py.230: loss: 0.11571 [I 2022-10-08 23:32:09] ...optimizers/trijoint.py.151: total_norm: 10.483812 [I 2022-10-08 23:32:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 2280/2304 [I 2022-10-08 23:32:17] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:20 [I 2022-10-08 23:32:17] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00304 [I 2022-10-08 23:32:17] ...trap/engines/engine.py.230: loss: 0.21971 [I 2022-10-08 23:32:17] ...optimizers/trijoint.py.151: total_norm: 23.190466 [I 2022-10-08 23:32:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 2290/2304 [I 2022-10-08 23:32:26] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:11 [I 2022-10-08 23:32:26] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00303 [I 2022-10-08 23:32:26] ...trap/engines/engine.py.230: loss: 0.19626 [I 2022-10-08 23:32:26] ...optimizers/trijoint.py.151: total_norm: 17.966759 [I 2022-10-08 23:32:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 2300/2304 [I 2022-10-08 23:32:34] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:03 [I 2022-10-08 23:32:34] ...trap/engines/engine.py.229: process: 0.82889 | load: 0.00018 [I 2022-10-08 23:32:34] ...trap/engines/engine.py.230: loss: 0.13564 [I 2022-10-08 23:32:34] ...optimizers/trijoint.py.151: total_norm: 17.931828 [I 2022-10-08 23:32:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 2304/2304 [I 2022-10-08 23:32:37] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:00 [I 2022-10-08 23:32:37] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00017 [I 2022-10-08 23:32:37] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-08 23:32:37] ...optimizers/trijoint.py.151: total_norm: 18.481201 [S 2022-10-08 23:32:39] ...trap/engines/engine.py.129: train_epoch.epoch: 23 [S 2022-10-08 23:32:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.17611092794161298 [I 2022-10-08 23:32:39] ...s/criterions/triplet.py.84: margin = 0.1700000000000001 [I 2022-10-08 23:32:39] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 23 [I 2022-10-08 23:32:39] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-08 23:32:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 23:32:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 23:32:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 23:32:42] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 23:32:42] ...trap/engines/engine.py.321: eval: epoch 23 | batch 0/510 [I 2022-10-08 23:32:42] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:26:59 [I 2022-10-08 23:32:42] ...trap/engines/engine.py.326: process: 3.17576 | load: 2.65138 [I 2022-10-08 23:32:48] ...trap/engines/engine.py.321: eval: epoch 23 | batch 10/510 [I 2022-10-08 23:32:48] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:51 [I 2022-10-08 23:32:48] ...trap/engines/engine.py.326: process: 0.64663 | load: 0.00019 [I 2022-10-08 23:32:54] ...trap/engines/engine.py.321: eval: epoch 23 | batch 20/510 [I 2022-10-08 23:32:54] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:38 [I 2022-10-08 23:32:54] ...trap/engines/engine.py.326: process: 0.57085 | load: 0.00019 [I 2022-10-08 23:33:01] ...trap/engines/engine.py.321: eval: epoch 23 | batch 30/510 [I 2022-10-08 23:33:01] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:18 [I 2022-10-08 23:33:01] ...trap/engines/engine.py.326: process: 0.63114 | load: 0.00020 [I 2022-10-08 23:33:06] ...trap/engines/engine.py.321: eval: epoch 23 | batch 40/510 [I 2022-10-08 23:33:06] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:09 [I 2022-10-08 23:33:06] ...trap/engines/engine.py.326: process: 0.43151 | load: 0.00019 [I 2022-10-08 23:33:11] ...trap/engines/engine.py.321: eval: epoch 23 | batch 50/510 [I 2022-10-08 23:33:11] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:55 [I 2022-10-08 23:33:11] ...trap/engines/engine.py.326: process: 0.53525 | load: 0.00019 [I 2022-10-08 23:33:16] ...trap/engines/engine.py.321: eval: epoch 23 | batch 60/510 [I 2022-10-08 23:33:16] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:26 [I 2022-10-08 23:33:16] ...trap/engines/engine.py.326: process: 0.42934 | load: 0.00020 [I 2022-10-08 23:33:21] ...trap/engines/engine.py.321: eval: epoch 23 | batch 70/510 [I 2022-10-08 23:33:21] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:45 [I 2022-10-08 23:33:21] ...trap/engines/engine.py.326: process: 0.49614 | load: 0.00019 [I 2022-10-08 23:33:27] ...trap/engines/engine.py.321: eval: epoch 23 | batch 80/510 [I 2022-10-08 23:33:27] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:57 [I 2022-10-08 23:33:27] ...trap/engines/engine.py.326: process: 0.43200 | load: 0.00019 [I 2022-10-08 23:33:31] ...trap/engines/engine.py.321: eval: epoch 23 | batch 90/510 [I 2022-10-08 23:33:31] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:04 [I 2022-10-08 23:33:31] ...trap/engines/engine.py.326: process: 0.43397 | load: 0.00019 [I 2022-10-08 23:33:36] ...trap/engines/engine.py.321: eval: epoch 23 | batch 100/510 [I 2022-10-08 23:33:36] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:59 [I 2022-10-08 23:33:36] ...trap/engines/engine.py.326: process: 0.30675 | load: 0.00020 [I 2022-10-08 23:33:39] ...trap/engines/engine.py.321: eval: epoch 23 | batch 110/510 [I 2022-10-08 23:33:39] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:12 [I 2022-10-08 23:33:39] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00019 [I 2022-10-08 23:33:42] ...trap/engines/engine.py.321: eval: epoch 23 | batch 120/510 [I 2022-10-08 23:33:42] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-08 23:33:42] ...trap/engines/engine.py.326: process: 0.32216 | load: 0.00022 [I 2022-10-08 23:33:45] ...trap/engines/engine.py.321: eval: epoch 23 | batch 130/510 [I 2022-10-08 23:33:45] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:00 [I 2022-10-08 23:33:45] ...trap/engines/engine.py.326: process: 0.31019 | load: 0.00019 [I 2022-10-08 23:33:49] ...trap/engines/engine.py.321: eval: epoch 23 | batch 140/510 [I 2022-10-08 23:33:49] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:57 [I 2022-10-08 23:33:49] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00020 [I 2022-10-08 23:33:52] ...trap/engines/engine.py.321: eval: epoch 23 | batch 150/510 [I 2022-10-08 23:33:52] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-08 23:33:52] ...trap/engines/engine.py.326: process: 0.32068 | load: 0.00020 [I 2022-10-08 23:33:55] ...trap/engines/engine.py.321: eval: epoch 23 | batch 160/510 [I 2022-10-08 23:33:55] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:51 [I 2022-10-08 23:33:55] ...trap/engines/engine.py.326: process: 0.32069 | load: 0.00020 [I 2022-10-08 23:33:58] ...trap/engines/engine.py.321: eval: epoch 23 | batch 170/510 [I 2022-10-08 23:33:58] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-08 23:33:58] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00020 [I 2022-10-08 23:34:01] ...trap/engines/engine.py.321: eval: epoch 23 | batch 180/510 [I 2022-10-08 23:34:01] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:45 [I 2022-10-08 23:34:01] ...trap/engines/engine.py.326: process: 0.31903 | load: 0.00020 [I 2022-10-08 23:34:04] ...trap/engines/engine.py.321: eval: epoch 23 | batch 190/510 [I 2022-10-08 23:34:04] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:41 [I 2022-10-08 23:34:04] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00019 [I 2022-10-08 23:34:08] ...trap/engines/engine.py.321: eval: epoch 23 | batch 200/510 [I 2022-10-08 23:34:08] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:38 [I 2022-10-08 23:34:08] ...trap/engines/engine.py.326: process: 0.31503 | load: 0.00020 [I 2022-10-08 23:34:11] ...trap/engines/engine.py.321: eval: epoch 23 | batch 210/510 [I 2022-10-08 23:34:11] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-08 23:34:11] ...trap/engines/engine.py.326: process: 0.31815 | load: 0.00020 [I 2022-10-08 23:34:14] ...trap/engines/engine.py.321: eval: epoch 23 | batch 220/510 [I 2022-10-08 23:34:14] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:31 [I 2022-10-08 23:34:14] ...trap/engines/engine.py.326: process: 0.31489 | load: 0.00020 [I 2022-10-08 23:34:17] ...trap/engines/engine.py.321: eval: epoch 23 | batch 230/510 [I 2022-10-08 23:34:17] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:28 [I 2022-10-08 23:34:17] ...trap/engines/engine.py.326: process: 0.31145 | load: 0.00020 [I 2022-10-08 23:34:20] ...trap/engines/engine.py.321: eval: epoch 23 | batch 240/510 [I 2022-10-08 23:34:20] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-08 23:34:20] ...trap/engines/engine.py.326: process: 0.32121 | load: 0.00020 [I 2022-10-08 23:34:24] ...trap/engines/engine.py.321: eval: epoch 23 | batch 250/510 [I 2022-10-08 23:34:24] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:22 [I 2022-10-08 23:34:24] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00021 [I 2022-10-08 23:34:27] ...trap/engines/engine.py.321: eval: epoch 23 | batch 260/510 [I 2022-10-08 23:34:27] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:20 [I 2022-10-08 23:34:27] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00021 [I 2022-10-08 23:34:30] ...trap/engines/engine.py.321: eval: epoch 23 | batch 270/510 [I 2022-10-08 23:34:30] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:16 [I 2022-10-08 23:34:30] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00019 [I 2022-10-08 23:34:33] ...trap/engines/engine.py.321: eval: epoch 23 | batch 280/510 [I 2022-10-08 23:34:33] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-08 23:34:33] ...trap/engines/engine.py.326: process: 0.31682 | load: 0.00021 [I 2022-10-08 23:34:36] ...trap/engines/engine.py.321: eval: epoch 23 | batch 290/510 [I 2022-10-08 23:34:36] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:09 [I 2022-10-08 23:34:36] ...trap/engines/engine.py.326: process: 0.31685 | load: 0.00021 [I 2022-10-08 23:34:40] ...trap/engines/engine.py.321: eval: epoch 23 | batch 300/510 [I 2022-10-08 23:34:40] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-08 23:34:40] ...trap/engines/engine.py.326: process: 0.31922 | load: 0.00020 [I 2022-10-08 23:34:43] ...trap/engines/engine.py.321: eval: epoch 23 | batch 310/510 [I 2022-10-08 23:34:43] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:03 [I 2022-10-08 23:34:43] ...trap/engines/engine.py.326: process: 0.31738 | load: 0.00020 [I 2022-10-08 23:34:46] ...trap/engines/engine.py.321: eval: epoch 23 | batch 320/510 [I 2022-10-08 23:34:46] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-08 23:34:46] ...trap/engines/engine.py.326: process: 0.31857 | load: 0.00021 [I 2022-10-08 23:34:49] ...trap/engines/engine.py.321: eval: epoch 23 | batch 330/510 [I 2022-10-08 23:34:49] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-08 23:34:49] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00020 [I 2022-10-08 23:34:52] ...trap/engines/engine.py.321: eval: epoch 23 | batch 340/510 [I 2022-10-08 23:34:52] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-08 23:34:52] ...trap/engines/engine.py.326: process: 0.31988 | load: 0.00022 [I 2022-10-08 23:34:56] ...trap/engines/engine.py.321: eval: epoch 23 | batch 350/510 [I 2022-10-08 23:34:56] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-08 23:34:56] ...trap/engines/engine.py.326: process: 0.31965 | load: 0.00021 [I 2022-10-08 23:34:59] ...trap/engines/engine.py.321: eval: epoch 23 | batch 360/510 [I 2022-10-08 23:34:59] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-08 23:34:59] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00020 [I 2022-10-08 23:35:02] ...trap/engines/engine.py.321: eval: epoch 23 | batch 370/510 [I 2022-10-08 23:35:02] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-08 23:35:02] ...trap/engines/engine.py.326: process: 0.31271 | load: 0.00020 [I 2022-10-08 23:35:05] ...trap/engines/engine.py.321: eval: epoch 23 | batch 380/510 [I 2022-10-08 23:35:05] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-08 23:35:05] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00021 [I 2022-10-08 23:35:08] ...trap/engines/engine.py.321: eval: epoch 23 | batch 390/510 [I 2022-10-08 23:35:08] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-08 23:35:08] ...trap/engines/engine.py.326: process: 0.31389 | load: 0.00019 [I 2022-10-08 23:35:11] ...trap/engines/engine.py.321: eval: epoch 23 | batch 400/510 [I 2022-10-08 23:35:11] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-08 23:35:11] ...trap/engines/engine.py.326: process: 0.31415 | load: 0.00020 [I 2022-10-08 23:35:15] ...trap/engines/engine.py.321: eval: epoch 23 | batch 410/510 [I 2022-10-08 23:35:15] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-08 23:35:15] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00020 [I 2022-10-08 23:35:18] ...trap/engines/engine.py.321: eval: epoch 23 | batch 420/510 [I 2022-10-08 23:35:18] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-08 23:35:18] ...trap/engines/engine.py.326: process: 0.32217 | load: 0.00020 [I 2022-10-08 23:35:21] ...trap/engines/engine.py.321: eval: epoch 23 | batch 430/510 [I 2022-10-08 23:35:21] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-08 23:35:21] ...trap/engines/engine.py.326: process: 0.31647 | load: 0.00020 [I 2022-10-08 23:35:24] ...trap/engines/engine.py.321: eval: epoch 23 | batch 440/510 [I 2022-10-08 23:35:24] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-08 23:35:24] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00020 [I 2022-10-08 23:35:27] ...trap/engines/engine.py.321: eval: epoch 23 | batch 450/510 [I 2022-10-08 23:35:27] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-08 23:35:27] ...trap/engines/engine.py.326: process: 0.31664 | load: 0.00020 [I 2022-10-08 23:35:31] ...trap/engines/engine.py.321: eval: epoch 23 | batch 460/510 [I 2022-10-08 23:35:31] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-08 23:35:31] ...trap/engines/engine.py.326: process: 0.32188 | load: 0.00020 [I 2022-10-08 23:35:34] ...trap/engines/engine.py.321: eval: epoch 23 | batch 470/510 [I 2022-10-08 23:35:34] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-08 23:35:34] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00020 [I 2022-10-08 23:35:37] ...trap/engines/engine.py.321: eval: epoch 23 | batch 480/510 [I 2022-10-08 23:35:37] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-08 23:35:37] ...trap/engines/engine.py.326: process: 0.31734 | load: 0.00020 [I 2022-10-08 23:35:40] ...trap/engines/engine.py.321: eval: epoch 23 | batch 490/510 [I 2022-10-08 23:35:40] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-08 23:35:40] ...trap/engines/engine.py.326: process: 0.32156 | load: 0.00021 [I 2022-10-08 23:35:43] ...trap/engines/engine.py.321: eval: epoch 23 | batch 500/510 [I 2022-10-08 23:35:43] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-08 23:35:43] ...trap/engines/engine.py.326: process: 0.32269 | load: 0.00020 [I 2022-10-08 23:35:47] ...trap/engines/engine.py.321: eval: epoch 23 | batch 510/510 [I 2022-10-08 23:35:47] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-08 23:35:47] ...trap/engines/engine.py.326: process: 0.30750 | load: 0.00015 [S 2022-10-08 23:35:47] ...trap/engines/engine.py.132: eval_epoch.epoch: 23 [I 2022-10-08 23:35:47] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.667400 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.883400 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.926000 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.662900 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.886000 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.923700 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013574 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006391 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007510 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008420 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008978 [S 2022-10-08 23:36:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010593 [I 2022-10-08 23:36:28] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-08 23:36:28] ...trap/engines/engine.py.470: Saving model... [W 2022-10-08 23:36:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-08 23:36:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-08 23:36:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-08 23:36:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-08 23:36:31] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 23:36:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 23:36:37] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-08 23:36:37] ...trap/engines/engine.py.470: Saving model... [I 2022-10-08 23:36:40] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-08 23:36:46] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-08 23:36:46] ...trap/engines/engine.py.163: Training model on trainset for epoch 24 [I 2022-10-08 23:36:46] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-08 23:36:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 0/2304 [I 2022-10-08 23:36:49] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:21 [I 2022-10-08 23:36:49] ...trap/engines/engine.py.229: process: 2.97825 | load: 2.07639 [I 2022-10-08 23:36:49] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-08 23:36:49] ...optimizers/trijoint.py.151: total_norm: 10.591510 [I 2022-10-08 23:36:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 10/2304 [I 2022-10-08 23:36:57] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:43 [I 2022-10-08 23:36:57] ...trap/engines/engine.py.229: process: 0.83436 | load: 0.00466 [I 2022-10-08 23:36:57] ...trap/engines/engine.py.230: loss: 0.26347 [I 2022-10-08 23:36:57] ...optimizers/trijoint.py.151: total_norm: 21.688787 [I 2022-10-08 23:37:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 20/2304 [I 2022-10-08 23:37:06] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:51 [I 2022-10-08 23:37:06] ...trap/engines/engine.py.229: process: 0.83617 | load: 0.00404 [I 2022-10-08 23:37:06] ...trap/engines/engine.py.230: loss: 0.24479 [I 2022-10-08 23:37:06] ...optimizers/trijoint.py.151: total_norm: 17.728607 [I 2022-10-08 23:37:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 30/2304 [I 2022-10-08 23:37:14] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:54 [I 2022-10-08 23:37:14] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00470 [I 2022-10-08 23:37:14] ...trap/engines/engine.py.230: loss: 0.16802 [I 2022-10-08 23:37:14] ...optimizers/trijoint.py.151: total_norm: 40.834122 [I 2022-10-08 23:37:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 40/2304 [I 2022-10-08 23:37:23] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:40 [I 2022-10-08 23:37:23] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00373 [I 2022-10-08 23:37:23] ...trap/engines/engine.py.230: loss: 0.20622 [I 2022-10-08 23:37:23] ...optimizers/trijoint.py.151: total_norm: 19.856926 [I 2022-10-08 23:37:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 50/2304 [I 2022-10-08 23:37:31] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:25 [I 2022-10-08 23:37:31] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00355 [I 2022-10-08 23:37:31] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-08 23:37:31] ...optimizers/trijoint.py.151: total_norm: 21.441963 [I 2022-10-08 23:37:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 60/2304 [I 2022-10-08 23:37:39] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:30 [I 2022-10-08 23:37:39] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00328 [I 2022-10-08 23:37:39] ...trap/engines/engine.py.230: loss: 0.15945 [I 2022-10-08 23:37:39] ...optimizers/trijoint.py.151: total_norm: 21.322571 [I 2022-10-08 23:37:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 70/2304 [I 2022-10-08 23:37:48] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:22 [I 2022-10-08 23:37:48] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00387 [I 2022-10-08 23:37:48] ...trap/engines/engine.py.230: loss: 0.17889 [I 2022-10-08 23:37:48] ...optimizers/trijoint.py.151: total_norm: 16.944611 [I 2022-10-08 23:37:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 80/2304 [I 2022-10-08 23:37:56] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:15 [I 2022-10-08 23:37:56] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00356 [I 2022-10-08 23:37:56] ...trap/engines/engine.py.230: loss: 0.14730 [I 2022-10-08 23:37:56] ...optimizers/trijoint.py.151: total_norm: 22.475391 [I 2022-10-08 23:38:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 90/2304 [I 2022-10-08 23:38:05] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:55 [I 2022-10-08 23:38:05] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00330 [I 2022-10-08 23:38:05] ...trap/engines/engine.py.230: loss: 0.16101 [I 2022-10-08 23:38:05] ...optimizers/trijoint.py.151: total_norm: 9.980757 [I 2022-10-08 23:38:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 100/2304 [I 2022-10-08 23:38:13] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:55 [I 2022-10-08 23:38:13] ...trap/engines/engine.py.229: process: 0.83751 | load: 0.00335 [I 2022-10-08 23:38:13] ...trap/engines/engine.py.230: loss: 0.16627 [I 2022-10-08 23:38:13] ...optimizers/trijoint.py.151: total_norm: 19.864079 [I 2022-10-08 23:38:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 110/2304 [I 2022-10-08 23:38:22] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:44 [I 2022-10-08 23:38:22] ...trap/engines/engine.py.229: process: 0.83752 | load: 0.00319 [I 2022-10-08 23:38:22] ...trap/engines/engine.py.230: loss: 0.21982 [I 2022-10-08 23:38:22] ...optimizers/trijoint.py.151: total_norm: 24.322031 [I 2022-10-08 23:38:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 120/2304 [I 2022-10-08 23:38:30] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:40 [I 2022-10-08 23:38:30] ...trap/engines/engine.py.229: process: 0.83606 | load: 0.00352 [I 2022-10-08 23:38:30] ...trap/engines/engine.py.230: loss: 0.21139 [I 2022-10-08 23:38:30] ...optimizers/trijoint.py.151: total_norm: 44.617085 [I 2022-10-08 23:38:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 130/2304 [I 2022-10-08 23:38:38] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:27 [I 2022-10-08 23:38:38] ...trap/engines/engine.py.229: process: 0.84823 | load: 0.00353 [I 2022-10-08 23:38:38] ...trap/engines/engine.py.230: loss: 0.13715 [I 2022-10-08 23:38:38] ...optimizers/trijoint.py.151: total_norm: 15.914007 [I 2022-10-08 23:38:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 140/2304 [I 2022-10-08 23:38:47] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:16 [I 2022-10-08 23:38:47] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00338 [I 2022-10-08 23:38:47] ...trap/engines/engine.py.230: loss: 0.25322 [I 2022-10-08 23:38:47] ...optimizers/trijoint.py.151: total_norm: 27.893229 [I 2022-10-08 23:38:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 150/2304 [I 2022-10-08 23:38:55] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:09 [I 2022-10-08 23:38:55] ...trap/engines/engine.py.229: process: 0.84781 | load: 0.00337 [I 2022-10-08 23:38:55] ...trap/engines/engine.py.230: loss: 0.20979 [I 2022-10-08 23:38:55] ...optimizers/trijoint.py.151: total_norm: 11.478121 [I 2022-10-08 23:39:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 160/2304 [I 2022-10-08 23:39:04] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:29:55 [I 2022-10-08 23:39:04] ...trap/engines/engine.py.229: process: 0.82726 | load: 0.00323 [I 2022-10-08 23:39:04] ...trap/engines/engine.py.230: loss: 0.15848 [I 2022-10-08 23:39:04] ...optimizers/trijoint.py.151: total_norm: 14.331433 [I 2022-10-08 23:39:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 170/2304 [I 2022-10-08 23:39:12] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:29:51 [I 2022-10-08 23:39:12] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00310 [I 2022-10-08 23:39:12] ...trap/engines/engine.py.230: loss: 0.09871 [I 2022-10-08 23:39:12] ...optimizers/trijoint.py.151: total_norm: 6.109280 [I 2022-10-08 23:39:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 180/2304 [I 2022-10-08 23:39:20] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:42 [I 2022-10-08 23:39:20] ...trap/engines/engine.py.229: process: 0.84217 | load: 0.00337 [I 2022-10-08 23:39:20] ...trap/engines/engine.py.230: loss: 0.16235 [I 2022-10-08 23:39:20] ...optimizers/trijoint.py.151: total_norm: 19.526968 [I 2022-10-08 23:39:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 190/2304 [I 2022-10-08 23:39:29] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:27 [I 2022-10-08 23:39:29] ...trap/engines/engine.py.229: process: 0.83595 | load: 0.00299 [I 2022-10-08 23:39:29] ...trap/engines/engine.py.230: loss: 0.22688 [I 2022-10-08 23:39:29] ...optimizers/trijoint.py.151: total_norm: 38.315079 [I 2022-10-08 23:39:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 200/2304 [I 2022-10-08 23:39:37] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:26 [I 2022-10-08 23:39:37] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00301 [I 2022-10-08 23:39:37] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-08 23:39:37] ...optimizers/trijoint.py.151: total_norm: 17.126907 [I 2022-10-08 23:39:46] ...trap/engines/engine.py.224: train: epoch 24 | batch 210/2304 [I 2022-10-08 23:39:46] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:22 [I 2022-10-08 23:39:46] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00301 [I 2022-10-08 23:39:46] ...trap/engines/engine.py.230: loss: 0.18031 [I 2022-10-08 23:39:46] ...optimizers/trijoint.py.151: total_norm: 44.827736 [I 2022-10-08 23:39:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 220/2304 [I 2022-10-08 23:39:54] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:09 [I 2022-10-08 23:39:54] ...trap/engines/engine.py.229: process: 0.83982 | load: 0.00319 [I 2022-10-08 23:39:54] ...trap/engines/engine.py.230: loss: 0.13678 [I 2022-10-08 23:39:54] ...optimizers/trijoint.py.151: total_norm: 14.313407 [I 2022-10-08 23:40:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 230/2304 [I 2022-10-08 23:40:03] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:28:56 [I 2022-10-08 23:40:03] ...trap/engines/engine.py.229: process: 0.82646 | load: 0.00300 [I 2022-10-08 23:40:03] ...trap/engines/engine.py.230: loss: 0.12990 [I 2022-10-08 23:40:03] ...optimizers/trijoint.py.151: total_norm: 10.682590 [I 2022-10-08 23:40:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 240/2304 [I 2022-10-08 23:40:11] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:29:00 [I 2022-10-08 23:40:11] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00312 [I 2022-10-08 23:40:11] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-08 23:40:11] ...optimizers/trijoint.py.151: total_norm: 24.220081 [I 2022-10-08 23:40:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 250/2304 [I 2022-10-08 23:40:19] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:43 [I 2022-10-08 23:40:19] ...trap/engines/engine.py.229: process: 0.83778 | load: 0.00299 [I 2022-10-08 23:40:19] ...trap/engines/engine.py.230: loss: 0.18511 [I 2022-10-08 23:40:19] ...optimizers/trijoint.py.151: total_norm: 36.347408 [I 2022-10-08 23:40:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 260/2304 [I 2022-10-08 23:40:28] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:31 [I 2022-10-08 23:40:28] ...trap/engines/engine.py.229: process: 0.83182 | load: 0.00308 [I 2022-10-08 23:40:28] ...trap/engines/engine.py.230: loss: 0.17258 [I 2022-10-08 23:40:28] ...optimizers/trijoint.py.151: total_norm: 17.945791 [I 2022-10-08 23:40:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 270/2304 [I 2022-10-08 23:40:36] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:27 [I 2022-10-08 23:40:36] ...trap/engines/engine.py.229: process: 0.83163 | load: 0.00304 [I 2022-10-08 23:40:36] ...trap/engines/engine.py.230: loss: 0.27263 [I 2022-10-08 23:40:36] ...optimizers/trijoint.py.151: total_norm: 22.665253 [I 2022-10-08 23:40:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 280/2304 [I 2022-10-08 23:40:45] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:29 [I 2022-10-08 23:40:45] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00305 [I 2022-10-08 23:40:45] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-08 23:40:45] ...optimizers/trijoint.py.151: total_norm: 34.040184 [I 2022-10-08 23:40:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 290/2304 [I 2022-10-08 23:40:53] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:13 [I 2022-10-08 23:40:53] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00313 [I 2022-10-08 23:40:53] ...trap/engines/engine.py.230: loss: 0.17556 [I 2022-10-08 23:40:53] ...optimizers/trijoint.py.151: total_norm: 17.848797 [I 2022-10-08 23:41:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 300/2304 [I 2022-10-08 23:41:02] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:03 [I 2022-10-08 23:41:02] ...trap/engines/engine.py.229: process: 0.82456 | load: 0.00300 [I 2022-10-08 23:41:02] ...trap/engines/engine.py.230: loss: 0.17048 [I 2022-10-08 23:41:02] ...optimizers/trijoint.py.151: total_norm: 17.034307 [I 2022-10-08 23:41:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 310/2304 [I 2022-10-08 23:41:10] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:28:04 [I 2022-10-08 23:41:10] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00299 [I 2022-10-08 23:41:10] ...trap/engines/engine.py.230: loss: 0.17211 [I 2022-10-08 23:41:10] ...optimizers/trijoint.py.151: total_norm: 19.104012 [I 2022-10-08 23:41:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 320/2304 [I 2022-10-08 23:41:18] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:34 [I 2022-10-08 23:41:18] ...trap/engines/engine.py.229: process: 0.82705 | load: 0.00301 [I 2022-10-08 23:41:18] ...trap/engines/engine.py.230: loss: 0.16238 [I 2022-10-08 23:41:18] ...optimizers/trijoint.py.151: total_norm: 28.630272 [I 2022-10-08 23:41:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 330/2304 [I 2022-10-08 23:41:27] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:40 [I 2022-10-08 23:41:27] ...trap/engines/engine.py.229: process: 0.84804 | load: 0.00315 [I 2022-10-08 23:41:27] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-08 23:41:27] ...optimizers/trijoint.py.151: total_norm: 25.529985 [I 2022-10-08 23:41:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 340/2304 [I 2022-10-08 23:41:35] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:35 [I 2022-10-08 23:41:35] ...trap/engines/engine.py.229: process: 0.82982 | load: 0.00308 [I 2022-10-08 23:41:35] ...trap/engines/engine.py.230: loss: 0.11129 [I 2022-10-08 23:41:35] ...optimizers/trijoint.py.151: total_norm: 30.059006 [I 2022-10-08 23:41:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 350/2304 [I 2022-10-08 23:41:44] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:21 [I 2022-10-08 23:41:44] ...trap/engines/engine.py.229: process: 0.83918 | load: 0.00301 [I 2022-10-08 23:41:44] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-08 23:41:44] ...optimizers/trijoint.py.151: total_norm: 14.920223 [I 2022-10-08 23:41:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 360/2304 [I 2022-10-08 23:41:52] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:03 [I 2022-10-08 23:41:52] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00300 [I 2022-10-08 23:41:52] ...trap/engines/engine.py.230: loss: 0.18119 [I 2022-10-08 23:41:52] ...optimizers/trijoint.py.151: total_norm: 19.850632 [I 2022-10-08 23:42:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 370/2304 [I 2022-10-08 23:42:01] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:26:59 [I 2022-10-08 23:42:01] ...trap/engines/engine.py.229: process: 0.82348 | load: 0.00298 [I 2022-10-08 23:42:01] ...trap/engines/engine.py.230: loss: 0.17552 [I 2022-10-08 23:42:01] ...optimizers/trijoint.py.151: total_norm: 33.055237 [I 2022-10-08 23:42:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 380/2304 [I 2022-10-08 23:42:09] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:52 [I 2022-10-08 23:42:09] ...trap/engines/engine.py.229: process: 0.84661 | load: 0.00304 [I 2022-10-08 23:42:09] ...trap/engines/engine.py.230: loss: 0.15756 [I 2022-10-08 23:42:09] ...optimizers/trijoint.py.151: total_norm: 16.657061 [I 2022-10-08 23:42:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 390/2304 [I 2022-10-08 23:42:18] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:35 [I 2022-10-08 23:42:18] ...trap/engines/engine.py.229: process: 0.82580 | load: 0.00311 [I 2022-10-08 23:42:18] ...trap/engines/engine.py.230: loss: 0.17504 [I 2022-10-08 23:42:18] ...optimizers/trijoint.py.151: total_norm: 15.169415 [I 2022-10-08 23:42:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 400/2304 [I 2022-10-08 23:42:26] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:35 [I 2022-10-08 23:42:26] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00299 [I 2022-10-08 23:42:26] ...trap/engines/engine.py.230: loss: 0.17943 [I 2022-10-08 23:42:26] ...optimizers/trijoint.py.151: total_norm: 20.577894 [I 2022-10-08 23:42:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 410/2304 [I 2022-10-08 23:42:34] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:20 [I 2022-10-08 23:42:34] ...trap/engines/engine.py.229: process: 0.84135 | load: 0.00302 [I 2022-10-08 23:42:34] ...trap/engines/engine.py.230: loss: 0.13239 [I 2022-10-08 23:42:34] ...optimizers/trijoint.py.151: total_norm: 17.927443 [I 2022-10-08 23:42:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 420/2304 [I 2022-10-08 23:42:43] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:07 [I 2022-10-08 23:42:43] ...trap/engines/engine.py.229: process: 0.83912 | load: 0.00298 [I 2022-10-08 23:42:43] ...trap/engines/engine.py.230: loss: 0.17517 [I 2022-10-08 23:42:43] ...optimizers/trijoint.py.151: total_norm: 21.474426 [I 2022-10-08 23:42:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 430/2304 [I 2022-10-08 23:42:51] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:02 [I 2022-10-08 23:42:51] ...trap/engines/engine.py.229: process: 0.83182 | load: 0.00308 [I 2022-10-08 23:42:51] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-08 23:42:51] ...optimizers/trijoint.py.151: total_norm: 13.002321 [I 2022-10-08 23:42:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 440/2304 [I 2022-10-08 23:42:59] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:25:54 [I 2022-10-08 23:42:59] ...trap/engines/engine.py.229: process: 0.84206 | load: 0.00304 [I 2022-10-08 23:42:59] ...trap/engines/engine.py.230: loss: 0.10154 [I 2022-10-08 23:42:59] ...optimizers/trijoint.py.151: total_norm: 9.907529 [I 2022-10-08 23:43:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 450/2304 [I 2022-10-08 23:43:08] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:25:38 [I 2022-10-08 23:43:08] ...trap/engines/engine.py.229: process: 0.83012 | load: 0.00302 [I 2022-10-08 23:43:08] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-08 23:43:08] ...optimizers/trijoint.py.151: total_norm: 19.360312 [I 2022-10-08 23:43:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 460/2304 [I 2022-10-08 23:43:16] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:25:46 [I 2022-10-08 23:43:16] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00298 [I 2022-10-08 23:43:16] ...trap/engines/engine.py.230: loss: 0.15178 [I 2022-10-08 23:43:16] ...optimizers/trijoint.py.151: total_norm: 17.094801 [I 2022-10-08 23:43:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 470/2304 [I 2022-10-08 23:43:24] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:31 [I 2022-10-08 23:43:24] ...trap/engines/engine.py.229: process: 0.83274 | load: 0.00302 [I 2022-10-08 23:43:24] ...trap/engines/engine.py.230: loss: 0.20708 [I 2022-10-08 23:43:24] ...optimizers/trijoint.py.151: total_norm: 21.548267 [I 2022-10-08 23:43:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 480/2304 [I 2022-10-08 23:43:33] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:25:27 [I 2022-10-08 23:43:33] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00303 [I 2022-10-08 23:43:33] ...trap/engines/engine.py.230: loss: 0.20030 [I 2022-10-08 23:43:33] ...optimizers/trijoint.py.151: total_norm: 19.526701 [I 2022-10-08 23:43:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 490/2304 [I 2022-10-08 23:43:41] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:10 [I 2022-10-08 23:43:41] ...trap/engines/engine.py.229: process: 0.82089 | load: 0.00308 [I 2022-10-08 23:43:41] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-08 23:43:41] ...optimizers/trijoint.py.151: total_norm: 14.453035 [I 2022-10-08 23:43:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 500/2304 [I 2022-10-08 23:43:50] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:25:09 [I 2022-10-08 23:43:50] ...trap/engines/engine.py.229: process: 0.83353 | load: 0.00299 [I 2022-10-08 23:43:50] ...trap/engines/engine.py.230: loss: 0.19434 [I 2022-10-08 23:43:50] ...optimizers/trijoint.py.151: total_norm: 17.857275 [I 2022-10-08 23:43:58] ...trap/engines/engine.py.224: train: epoch 24 | batch 510/2304 [I 2022-10-08 23:43:58] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:25:05 [I 2022-10-08 23:43:58] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00301 [I 2022-10-08 23:43:58] ...trap/engines/engine.py.230: loss: 0.19080 [I 2022-10-08 23:43:58] ...optimizers/trijoint.py.151: total_norm: 20.415976 [I 2022-10-08 23:44:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 520/2304 [I 2022-10-08 23:44:06] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:24:58 [I 2022-10-08 23:44:06] ...trap/engines/engine.py.229: process: 0.84916 | load: 0.00301 [I 2022-10-08 23:44:06] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-08 23:44:06] ...optimizers/trijoint.py.151: total_norm: 21.457710 [I 2022-10-08 23:44:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 530/2304 [I 2022-10-08 23:44:15] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:24:44 [I 2022-10-08 23:44:15] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00303 [I 2022-10-08 23:44:15] ...trap/engines/engine.py.230: loss: 0.26411 [I 2022-10-08 23:44:15] ...optimizers/trijoint.py.151: total_norm: 33.441254 [I 2022-10-08 23:44:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 540/2304 [I 2022-10-08 23:44:23] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:24:40 [I 2022-10-08 23:44:23] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00309 [I 2022-10-08 23:44:23] ...trap/engines/engine.py.230: loss: 0.17784 [I 2022-10-08 23:44:23] ...optimizers/trijoint.py.151: total_norm: 24.055120 [I 2022-10-08 23:44:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 550/2304 [I 2022-10-08 23:44:31] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:24:19 [I 2022-10-08 23:44:31] ...trap/engines/engine.py.229: process: 0.82552 | load: 0.00300 [I 2022-10-08 23:44:31] ...trap/engines/engine.py.230: loss: 0.18742 [I 2022-10-08 23:44:31] ...optimizers/trijoint.py.151: total_norm: 21.033569 [I 2022-10-08 23:44:40] ...trap/engines/engine.py.224: train: epoch 24 | batch 560/2304 [I 2022-10-08 23:44:40] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:24:15 [I 2022-10-08 23:44:40] ...trap/engines/engine.py.229: process: 0.82861 | load: 0.00300 [I 2022-10-08 23:44:40] ...trap/engines/engine.py.230: loss: 0.15235 [I 2022-10-08 23:44:40] ...optimizers/trijoint.py.151: total_norm: 17.837317 [I 2022-10-08 23:44:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 570/2304 [I 2022-10-08 23:44:48] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:11 [I 2022-10-08 23:44:48] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00299 [I 2022-10-08 23:44:48] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-08 23:44:48] ...optimizers/trijoint.py.151: total_norm: 15.260497 [I 2022-10-08 23:44:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 580/2304 [I 2022-10-08 23:44:56] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:24:02 [I 2022-10-08 23:44:56] ...trap/engines/engine.py.229: process: 0.82518 | load: 0.00297 [I 2022-10-08 23:44:56] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-08 23:44:56] ...optimizers/trijoint.py.151: total_norm: 18.736546 [I 2022-10-08 23:45:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 590/2304 [I 2022-10-08 23:45:05] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:00 [I 2022-10-08 23:45:05] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00303 [I 2022-10-08 23:45:05] ...trap/engines/engine.py.230: loss: 0.16066 [I 2022-10-08 23:45:05] ...optimizers/trijoint.py.151: total_norm: 24.971878 [I 2022-10-08 23:45:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 600/2304 [I 2022-10-08 23:45:13] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:23:50 [I 2022-10-08 23:45:13] ...trap/engines/engine.py.229: process: 0.83624 | load: 0.00295 [I 2022-10-08 23:45:13] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-08 23:45:13] ...optimizers/trijoint.py.151: total_norm: 17.467649 [I 2022-10-08 23:45:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 610/2304 [I 2022-10-08 23:45:22] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:23:39 [I 2022-10-08 23:45:22] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00323 [I 2022-10-08 23:45:22] ...trap/engines/engine.py.230: loss: 0.17516 [I 2022-10-08 23:45:22] ...optimizers/trijoint.py.151: total_norm: 21.603237 [I 2022-10-08 23:45:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 620/2304 [I 2022-10-08 23:45:30] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:33 [I 2022-10-08 23:45:30] ...trap/engines/engine.py.229: process: 0.83998 | load: 0.00298 [I 2022-10-08 23:45:30] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-08 23:45:30] ...optimizers/trijoint.py.151: total_norm: 9.523274 [I 2022-10-08 23:45:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 630/2304 [I 2022-10-08 23:45:39] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:27 [I 2022-10-08 23:45:39] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00296 [I 2022-10-08 23:45:39] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-08 23:45:39] ...optimizers/trijoint.py.151: total_norm: 22.783602 [I 2022-10-08 23:45:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 640/2304 [I 2022-10-08 23:45:47] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:04 [I 2022-10-08 23:45:47] ...trap/engines/engine.py.229: process: 0.82203 | load: 0.00314 [I 2022-10-08 23:45:47] ...trap/engines/engine.py.230: loss: 0.17183 [I 2022-10-08 23:45:47] ...optimizers/trijoint.py.151: total_norm: 14.570679 [I 2022-10-08 23:45:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 650/2304 [I 2022-10-08 23:45:55] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:13 [I 2022-10-08 23:45:55] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00298 [I 2022-10-08 23:45:55] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-08 23:45:55] ...optimizers/trijoint.py.151: total_norm: 20.462503 [I 2022-10-08 23:46:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 660/2304 [I 2022-10-08 23:46:04] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:22:51 [I 2022-10-08 23:46:04] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00297 [I 2022-10-08 23:46:04] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-08 23:46:04] ...optimizers/trijoint.py.151: total_norm: 10.962080 [I 2022-10-08 23:46:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 670/2304 [I 2022-10-08 23:46:12] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:22:48 [I 2022-10-08 23:46:12] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00297 [I 2022-10-08 23:46:12] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-08 23:46:12] ...optimizers/trijoint.py.151: total_norm: 11.791118 [I 2022-10-08 23:46:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 680/2304 [I 2022-10-08 23:46:21] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:22:35 [I 2022-10-08 23:46:21] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00306 [I 2022-10-08 23:46:21] ...trap/engines/engine.py.230: loss: 0.17790 [I 2022-10-08 23:46:21] ...optimizers/trijoint.py.151: total_norm: 68.678848 [I 2022-10-08 23:46:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 690/2304 [I 2022-10-08 23:46:29] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:24 [I 2022-10-08 23:46:29] ...trap/engines/engine.py.229: process: 0.82976 | load: 0.00297 [I 2022-10-08 23:46:29] ...trap/engines/engine.py.230: loss: 0.17565 [I 2022-10-08 23:46:29] ...optimizers/trijoint.py.151: total_norm: 20.488096 [I 2022-10-08 23:46:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 700/2304 [I 2022-10-08 23:46:37] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:19 [I 2022-10-08 23:46:37] ...trap/engines/engine.py.229: process: 0.82966 | load: 0.00300 [I 2022-10-08 23:46:37] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-08 23:46:37] ...optimizers/trijoint.py.151: total_norm: 29.923899 [I 2022-10-08 23:46:46] ...trap/engines/engine.py.224: train: epoch 24 | batch 710/2304 [I 2022-10-08 23:46:46] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:19 [I 2022-10-08 23:46:46] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00299 [I 2022-10-08 23:46:46] ...trap/engines/engine.py.230: loss: 0.23216 [I 2022-10-08 23:46:46] ...optimizers/trijoint.py.151: total_norm: 27.588499 [I 2022-10-08 23:46:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 720/2304 [I 2022-10-08 23:46:54] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:04 [I 2022-10-08 23:46:54] ...trap/engines/engine.py.229: process: 0.84048 | load: 0.00295 [I 2022-10-08 23:46:54] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-08 23:46:54] ...optimizers/trijoint.py.151: total_norm: 28.437521 [I 2022-10-08 23:47:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 730/2304 [I 2022-10-08 23:47:02] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:21:51 [I 2022-10-08 23:47:02] ...trap/engines/engine.py.229: process: 0.83650 | load: 0.00298 [I 2022-10-08 23:47:02] ...trap/engines/engine.py.230: loss: 0.19554 [I 2022-10-08 23:47:02] ...optimizers/trijoint.py.151: total_norm: 14.923085 [I 2022-10-08 23:47:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 740/2304 [I 2022-10-08 23:47:11] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:21:46 [I 2022-10-08 23:47:11] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00302 [I 2022-10-08 23:47:11] ...trap/engines/engine.py.230: loss: 0.19869 [I 2022-10-08 23:47:11] ...optimizers/trijoint.py.151: total_norm: 18.334150 [I 2022-10-08 23:47:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 750/2304 [I 2022-10-08 23:47:19] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:21:39 [I 2022-10-08 23:47:19] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00302 [I 2022-10-08 23:47:19] ...trap/engines/engine.py.230: loss: 0.16294 [I 2022-10-08 23:47:19] ...optimizers/trijoint.py.151: total_norm: 18.282114 [I 2022-10-08 23:47:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 760/2304 [I 2022-10-08 23:47:28] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:39 [I 2022-10-08 23:47:28] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00298 [I 2022-10-08 23:47:28] ...trap/engines/engine.py.230: loss: 0.23370 [I 2022-10-08 23:47:28] ...optimizers/trijoint.py.151: total_norm: 20.440889 [I 2022-10-08 23:47:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 770/2304 [I 2022-10-08 23:47:36] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:24 [I 2022-10-08 23:47:36] ...trap/engines/engine.py.229: process: 0.83400 | load: 0.00299 [I 2022-10-08 23:47:36] ...trap/engines/engine.py.230: loss: 0.19465 [I 2022-10-08 23:47:36] ...optimizers/trijoint.py.151: total_norm: 16.400467 [I 2022-10-08 23:47:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 780/2304 [I 2022-10-08 23:47:44] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:16 [I 2022-10-08 23:47:44] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00302 [I 2022-10-08 23:47:44] ...trap/engines/engine.py.230: loss: 0.17545 [I 2022-10-08 23:47:44] ...optimizers/trijoint.py.151: total_norm: 15.479177 [I 2022-10-08 23:47:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 790/2304 [I 2022-10-08 23:47:53] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:07 [I 2022-10-08 23:47:53] ...trap/engines/engine.py.229: process: 0.83296 | load: 0.00302 [I 2022-10-08 23:47:53] ...trap/engines/engine.py.230: loss: 0.17404 [I 2022-10-08 23:47:53] ...optimizers/trijoint.py.151: total_norm: 35.921673 [I 2022-10-08 23:48:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 800/2304 [I 2022-10-08 23:48:01] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:00 [I 2022-10-08 23:48:01] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00300 [I 2022-10-08 23:48:01] ...trap/engines/engine.py.230: loss: 0.14718 [I 2022-10-08 23:48:01] ...optimizers/trijoint.py.151: total_norm: 27.155964 [I 2022-10-08 23:48:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 810/2304 [I 2022-10-08 23:48:10] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:20:49 [I 2022-10-08 23:48:10] ...trap/engines/engine.py.229: process: 0.82741 | load: 0.00303 [I 2022-10-08 23:48:10] ...trap/engines/engine.py.230: loss: 0.12690 [I 2022-10-08 23:48:10] ...optimizers/trijoint.py.151: total_norm: 12.016705 [I 2022-10-08 23:48:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 820/2304 [I 2022-10-08 23:48:18] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:20:37 [I 2022-10-08 23:48:18] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00299 [I 2022-10-08 23:48:18] ...trap/engines/engine.py.230: loss: 0.25739 [I 2022-10-08 23:48:18] ...optimizers/trijoint.py.151: total_norm: 21.125240 [I 2022-10-08 23:48:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 830/2304 [I 2022-10-08 23:48:26] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:20:34 [I 2022-10-08 23:48:26] ...trap/engines/engine.py.229: process: 0.84007 | load: 0.00305 [I 2022-10-08 23:48:26] ...trap/engines/engine.py.230: loss: 0.21491 [I 2022-10-08 23:48:26] ...optimizers/trijoint.py.151: total_norm: 19.509739 [I 2022-10-08 23:48:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 840/2304 [I 2022-10-08 23:48:35] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:20:29 [I 2022-10-08 23:48:35] ...trap/engines/engine.py.229: process: 0.82196 | load: 0.00302 [I 2022-10-08 23:48:35] ...trap/engines/engine.py.230: loss: 0.17416 [I 2022-10-08 23:48:35] ...optimizers/trijoint.py.151: total_norm: 16.071177 [I 2022-10-08 23:48:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 850/2304 [I 2022-10-08 23:48:43] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:19 [I 2022-10-08 23:48:43] ...trap/engines/engine.py.229: process: 0.83348 | load: 0.00299 [I 2022-10-08 23:48:43] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-08 23:48:43] ...optimizers/trijoint.py.151: total_norm: 15.686049 [I 2022-10-08 23:48:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 860/2304 [I 2022-10-08 23:48:51] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:20:09 [I 2022-10-08 23:48:51] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00300 [I 2022-10-08 23:48:51] ...trap/engines/engine.py.230: loss: 0.15179 [I 2022-10-08 23:48:51] ...optimizers/trijoint.py.151: total_norm: 16.717184 [I 2022-10-08 23:49:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 870/2304 [I 2022-10-08 23:49:00] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:19:59 [I 2022-10-08 23:49:00] ...trap/engines/engine.py.229: process: 0.83315 | load: 0.00303 [I 2022-10-08 23:49:00] ...trap/engines/engine.py.230: loss: 0.16906 [I 2022-10-08 23:49:00] ...optimizers/trijoint.py.151: total_norm: 17.028610 [I 2022-10-08 23:49:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 880/2304 [I 2022-10-08 23:49:08] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:19:57 [I 2022-10-08 23:49:08] ...trap/engines/engine.py.229: process: 0.85449 | load: 0.00304 [I 2022-10-08 23:49:08] ...trap/engines/engine.py.230: loss: 0.19774 [I 2022-10-08 23:49:08] ...optimizers/trijoint.py.151: total_norm: 36.823704 [I 2022-10-08 23:49:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 890/2304 [I 2022-10-08 23:49:17] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:19:45 [I 2022-10-08 23:49:17] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00300 [I 2022-10-08 23:49:17] ...trap/engines/engine.py.230: loss: 0.13029 [I 2022-10-08 23:49:17] ...optimizers/trijoint.py.151: total_norm: 17.832235 [I 2022-10-08 23:49:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 900/2304 [I 2022-10-08 23:49:25] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:19:34 [I 2022-10-08 23:49:25] ...trap/engines/engine.py.229: process: 0.83656 | load: 0.00299 [I 2022-10-08 23:49:25] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-08 23:49:25] ...optimizers/trijoint.py.151: total_norm: 17.417179 [I 2022-10-08 23:49:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 910/2304 [I 2022-10-08 23:49:33] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:19:21 [I 2022-10-08 23:49:33] ...trap/engines/engine.py.229: process: 0.82273 | load: 0.00300 [I 2022-10-08 23:49:33] ...trap/engines/engine.py.230: loss: 0.17075 [I 2022-10-08 23:49:33] ...optimizers/trijoint.py.151: total_norm: 25.454111 [I 2022-10-08 23:49:42] ...trap/engines/engine.py.224: train: epoch 24 | batch 920/2304 [I 2022-10-08 23:49:42] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:19:22 [I 2022-10-08 23:49:42] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00300 [I 2022-10-08 23:49:42] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-08 23:49:42] ...optimizers/trijoint.py.151: total_norm: 14.980336 [I 2022-10-08 23:49:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 930/2304 [I 2022-10-08 23:49:50] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:12 [I 2022-10-08 23:49:50] ...trap/engines/engine.py.229: process: 0.83075 | load: 0.00302 [I 2022-10-08 23:49:50] ...trap/engines/engine.py.230: loss: 0.18222 [I 2022-10-08 23:49:50] ...optimizers/trijoint.py.151: total_norm: 50.146225 [I 2022-10-08 23:49:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 940/2304 [I 2022-10-08 23:49:59] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:19:07 [I 2022-10-08 23:49:59] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00301 [I 2022-10-08 23:49:59] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-08 23:49:59] ...optimizers/trijoint.py.151: total_norm: 29.126955 [I 2022-10-08 23:50:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 950/2304 [I 2022-10-08 23:50:07] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:18:51 [I 2022-10-08 23:50:07] ...trap/engines/engine.py.229: process: 0.81627 | load: 0.00302 [I 2022-10-08 23:50:07] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-08 23:50:07] ...optimizers/trijoint.py.151: total_norm: 25.850012 [I 2022-10-08 23:50:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 960/2304 [I 2022-10-08 23:50:15] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:18:48 [I 2022-10-08 23:50:15] ...trap/engines/engine.py.229: process: 0.84004 | load: 0.00301 [I 2022-10-08 23:50:15] ...trap/engines/engine.py.230: loss: 0.21292 [I 2022-10-08 23:50:15] ...optimizers/trijoint.py.151: total_norm: 15.274254 [I 2022-10-08 23:50:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 970/2304 [I 2022-10-08 23:50:24] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:18:26 [I 2022-10-08 23:50:24] ...trap/engines/engine.py.229: process: 0.82512 | load: 0.00298 [I 2022-10-08 23:50:24] ...trap/engines/engine.py.230: loss: 0.16430 [I 2022-10-08 23:50:24] ...optimizers/trijoint.py.151: total_norm: 26.163307 [I 2022-10-08 23:50:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 980/2304 [I 2022-10-08 23:50:32] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:18:28 [I 2022-10-08 23:50:32] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00298 [I 2022-10-08 23:50:32] ...trap/engines/engine.py.230: loss: 0.22123 [I 2022-10-08 23:50:32] ...optimizers/trijoint.py.151: total_norm: 21.847837 [I 2022-10-08 23:50:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 990/2304 [I 2022-10-08 23:50:41] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:20 [I 2022-10-08 23:50:41] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00300 [I 2022-10-08 23:50:41] ...trap/engines/engine.py.230: loss: 0.18377 [I 2022-10-08 23:50:41] ...optimizers/trijoint.py.151: total_norm: 13.854856 [I 2022-10-08 23:50:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 1000/2304 [I 2022-10-08 23:50:49] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:16 [I 2022-10-08 23:50:49] ...trap/engines/engine.py.229: process: 0.83715 | load: 0.00306 [I 2022-10-08 23:50:49] ...trap/engines/engine.py.230: loss: 0.15259 [I 2022-10-08 23:50:49] ...optimizers/trijoint.py.151: total_norm: 16.867649 [I 2022-10-08 23:50:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 1010/2304 [I 2022-10-08 23:50:57] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:10 [I 2022-10-08 23:50:57] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00301 [I 2022-10-08 23:50:57] ...trap/engines/engine.py.230: loss: 0.25059 [I 2022-10-08 23:50:57] ...optimizers/trijoint.py.151: total_norm: 16.445320 [I 2022-10-08 23:51:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 1020/2304 [I 2022-10-08 23:51:06] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:00 [I 2022-10-08 23:51:06] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00301 [I 2022-10-08 23:51:06] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-08 23:51:06] ...optimizers/trijoint.py.151: total_norm: 23.260086 [I 2022-10-08 23:51:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 1030/2304 [I 2022-10-08 23:51:14] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:17:52 [I 2022-10-08 23:51:14] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00299 [I 2022-10-08 23:51:14] ...trap/engines/engine.py.230: loss: 0.18699 [I 2022-10-08 23:51:14] ...optimizers/trijoint.py.151: total_norm: 28.618919 [I 2022-10-08 23:51:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 1040/2304 [I 2022-10-08 23:51:23] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:17:40 [I 2022-10-08 23:51:23] ...trap/engines/engine.py.229: process: 0.84264 | load: 0.00299 [I 2022-10-08 23:51:23] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-08 23:51:23] ...optimizers/trijoint.py.151: total_norm: 14.758821 [I 2022-10-08 23:51:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 1050/2304 [I 2022-10-08 23:51:31] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:17:32 [I 2022-10-08 23:51:31] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00301 [I 2022-10-08 23:51:31] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-08 23:51:31] ...optimizers/trijoint.py.151: total_norm: 19.649872 [I 2022-10-08 23:51:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 1060/2304 [I 2022-10-08 23:51:39] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:17:13 [I 2022-10-08 23:51:39] ...trap/engines/engine.py.229: process: 0.81887 | load: 0.00300 [I 2022-10-08 23:51:39] ...trap/engines/engine.py.230: loss: 0.29287 [I 2022-10-08 23:51:39] ...optimizers/trijoint.py.151: total_norm: 26.260851 [I 2022-10-08 23:51:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 1070/2304 [I 2022-10-08 23:51:48] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:16 [I 2022-10-08 23:51:48] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00301 [I 2022-10-08 23:51:48] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-08 23:51:48] ...optimizers/trijoint.py.151: total_norm: 22.192501 [I 2022-10-08 23:51:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 1080/2304 [I 2022-10-08 23:51:56] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:17:11 [I 2022-10-08 23:51:56] ...trap/engines/engine.py.229: process: 0.85355 | load: 0.00309 [I 2022-10-08 23:51:56] ...trap/engines/engine.py.230: loss: 0.18985 [I 2022-10-08 23:51:56] ...optimizers/trijoint.py.151: total_norm: 17.479477 [I 2022-10-08 23:52:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 1090/2304 [I 2022-10-08 23:52:05] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:16:56 [I 2022-10-08 23:52:05] ...trap/engines/engine.py.229: process: 0.83469 | load: 0.00309 [I 2022-10-08 23:52:05] ...trap/engines/engine.py.230: loss: 0.14467 [I 2022-10-08 23:52:05] ...optimizers/trijoint.py.151: total_norm: 16.653833 [I 2022-10-08 23:52:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 1100/2304 [I 2022-10-08 23:52:13] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:16:51 [I 2022-10-08 23:52:13] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00309 [I 2022-10-08 23:52:13] ...trap/engines/engine.py.230: loss: 0.10673 [I 2022-10-08 23:52:13] ...optimizers/trijoint.py.151: total_norm: 13.323019 [I 2022-10-08 23:52:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 1110/2304 [I 2022-10-08 23:52:21] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:16:40 [I 2022-10-08 23:52:21] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00300 [I 2022-10-08 23:52:21] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-08 23:52:21] ...optimizers/trijoint.py.151: total_norm: 21.942974 [I 2022-10-08 23:52:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 1120/2304 [I 2022-10-08 23:52:30] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:16:35 [I 2022-10-08 23:52:30] ...trap/engines/engine.py.229: process: 0.83624 | load: 0.00299 [I 2022-10-08 23:52:30] ...trap/engines/engine.py.230: loss: 0.13002 [I 2022-10-08 23:52:30] ...optimizers/trijoint.py.151: total_norm: 16.022333 [I 2022-10-08 23:52:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 1130/2304 [I 2022-10-08 23:52:38] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:16:18 [I 2022-10-08 23:52:38] ...trap/engines/engine.py.229: process: 0.82585 | load: 0.00299 [I 2022-10-08 23:52:38] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-08 23:52:38] ...optimizers/trijoint.py.151: total_norm: 24.893183 [I 2022-10-08 23:52:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 1140/2304 [I 2022-10-08 23:52:47] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:10 [I 2022-10-08 23:52:47] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00297 [I 2022-10-08 23:52:47] ...trap/engines/engine.py.230: loss: 0.24558 [I 2022-10-08 23:52:47] ...optimizers/trijoint.py.151: total_norm: 24.610704 [I 2022-10-08 23:52:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 1150/2304 [I 2022-10-08 23:52:55] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:03 [I 2022-10-08 23:52:55] ...trap/engines/engine.py.229: process: 0.82823 | load: 0.00298 [I 2022-10-08 23:52:55] ...trap/engines/engine.py.230: loss: 0.19074 [I 2022-10-08 23:52:55] ...optimizers/trijoint.py.151: total_norm: 20.082916 [I 2022-10-08 23:53:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 1160/2304 [I 2022-10-08 23:53:03] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:00 [I 2022-10-08 23:53:03] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00303 [I 2022-10-08 23:53:03] ...trap/engines/engine.py.230: loss: 0.13326 [I 2022-10-08 23:53:03] ...optimizers/trijoint.py.151: total_norm: 13.131843 [I 2022-10-08 23:53:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 1170/2304 [I 2022-10-08 23:53:12] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:15:44 [I 2022-10-08 23:53:12] ...trap/engines/engine.py.229: process: 0.82594 | load: 0.00300 [I 2022-10-08 23:53:12] ...trap/engines/engine.py.230: loss: 0.22058 [I 2022-10-08 23:53:12] ...optimizers/trijoint.py.151: total_norm: 23.916159 [I 2022-10-08 23:53:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 1180/2304 [I 2022-10-08 23:53:20] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:15:35 [I 2022-10-08 23:53:20] ...trap/engines/engine.py.229: process: 0.83644 | load: 0.00299 [I 2022-10-08 23:53:20] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-08 23:53:20] ...optimizers/trijoint.py.151: total_norm: 18.382982 [I 2022-10-08 23:53:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 1190/2304 [I 2022-10-08 23:53:29] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:15:30 [I 2022-10-08 23:53:29] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00299 [I 2022-10-08 23:53:29] ...trap/engines/engine.py.230: loss: 0.12819 [I 2022-10-08 23:53:29] ...optimizers/trijoint.py.151: total_norm: 16.631477 [I 2022-10-08 23:53:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 1200/2304 [I 2022-10-08 23:53:37] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:20 [I 2022-10-08 23:53:37] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00299 [I 2022-10-08 23:53:37] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-08 23:53:37] ...optimizers/trijoint.py.151: total_norm: 21.064260 [I 2022-10-08 23:53:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 1210/2304 [I 2022-10-08 23:53:45] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:07 [I 2022-10-08 23:53:45] ...trap/engines/engine.py.229: process: 0.83014 | load: 0.00297 [I 2022-10-08 23:53:45] ...trap/engines/engine.py.230: loss: 0.14338 [I 2022-10-08 23:53:45] ...optimizers/trijoint.py.151: total_norm: 23.728605 [I 2022-10-08 23:53:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 1220/2304 [I 2022-10-08 23:53:54] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:07 [I 2022-10-08 23:53:54] ...trap/engines/engine.py.229: process: 0.83673 | load: 0.00301 [I 2022-10-08 23:53:54] ...trap/engines/engine.py.230: loss: 0.26869 [I 2022-10-08 23:53:54] ...optimizers/trijoint.py.151: total_norm: 12.117770 [I 2022-10-08 23:54:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 1230/2304 [I 2022-10-08 23:54:02] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:06 [I 2022-10-08 23:54:02] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00300 [I 2022-10-08 23:54:02] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-08 23:54:02] ...optimizers/trijoint.py.151: total_norm: 23.950371 [I 2022-10-08 23:54:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 1240/2304 [I 2022-10-08 23:54:11] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:14:53 [I 2022-10-08 23:54:11] ...trap/engines/engine.py.229: process: 0.83588 | load: 0.00301 [I 2022-10-08 23:54:11] ...trap/engines/engine.py.230: loss: 0.12190 [I 2022-10-08 23:54:11] ...optimizers/trijoint.py.151: total_norm: 10.939051 [I 2022-10-08 23:54:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 1250/2304 [I 2022-10-08 23:54:19] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:40 [I 2022-10-08 23:54:19] ...trap/engines/engine.py.229: process: 0.83633 | load: 0.00297 [I 2022-10-08 23:54:19] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-08 23:54:19] ...optimizers/trijoint.py.151: total_norm: 23.009556 [I 2022-10-08 23:54:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 1260/2304 [I 2022-10-08 23:54:27] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:14:40 [I 2022-10-08 23:54:27] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00302 [I 2022-10-08 23:54:27] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-08 23:54:27] ...optimizers/trijoint.py.151: total_norm: 10.848624 [I 2022-10-08 23:54:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 1270/2304 [I 2022-10-08 23:54:36] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:14:26 [I 2022-10-08 23:54:36] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00302 [I 2022-10-08 23:54:36] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-08 23:54:36] ...optimizers/trijoint.py.151: total_norm: 21.656021 [I 2022-10-08 23:54:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 1280/2304 [I 2022-10-08 23:54:44] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:16 [I 2022-10-08 23:54:44] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00302 [I 2022-10-08 23:54:44] ...trap/engines/engine.py.230: loss: 0.23652 [I 2022-10-08 23:54:44] ...optimizers/trijoint.py.151: total_norm: 22.888985 [I 2022-10-08 23:54:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 1290/2304 [I 2022-10-08 23:54:52] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:05 [I 2022-10-08 23:54:52] ...trap/engines/engine.py.229: process: 0.83367 | load: 0.00303 [I 2022-10-08 23:54:52] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-08 23:54:52] ...optimizers/trijoint.py.151: total_norm: 10.501256 [I 2022-10-08 23:55:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 1300/2304 [I 2022-10-08 23:55:01] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:13:58 [I 2022-10-08 23:55:01] ...trap/engines/engine.py.229: process: 0.83377 | load: 0.00305 [I 2022-10-08 23:55:01] ...trap/engines/engine.py.230: loss: 0.24289 [I 2022-10-08 23:55:01] ...optimizers/trijoint.py.151: total_norm: 20.568455 [I 2022-10-08 23:55:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 1310/2304 [I 2022-10-08 23:55:09] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:13:52 [I 2022-10-08 23:55:09] ...trap/engines/engine.py.229: process: 0.83922 | load: 0.00310 [I 2022-10-08 23:55:09] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-08 23:55:09] ...optimizers/trijoint.py.151: total_norm: 25.866249 [I 2022-10-08 23:55:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 1320/2304 [I 2022-10-08 23:55:18] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:13:45 [I 2022-10-08 23:55:18] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00303 [I 2022-10-08 23:55:18] ...trap/engines/engine.py.230: loss: 0.17753 [I 2022-10-08 23:55:18] ...optimizers/trijoint.py.151: total_norm: 24.231892 [I 2022-10-08 23:55:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 1330/2304 [I 2022-10-08 23:55:26] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:34 [I 2022-10-08 23:55:26] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00302 [I 2022-10-08 23:55:26] ...trap/engines/engine.py.230: loss: 0.20115 [I 2022-10-08 23:55:26] ...optimizers/trijoint.py.151: total_norm: 21.925735 [I 2022-10-08 23:55:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 1340/2304 [I 2022-10-08 23:55:34] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:22 [I 2022-10-08 23:55:34] ...trap/engines/engine.py.229: process: 0.82884 | load: 0.00307 [I 2022-10-08 23:55:34] ...trap/engines/engine.py.230: loss: 0.12169 [I 2022-10-08 23:55:34] ...optimizers/trijoint.py.151: total_norm: 16.410992 [I 2022-10-08 23:55:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 1350/2304 [I 2022-10-08 23:55:43] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:18 [I 2022-10-08 23:55:43] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00305 [I 2022-10-08 23:55:43] ...trap/engines/engine.py.230: loss: 0.18603 [I 2022-10-08 23:55:43] ...optimizers/trijoint.py.151: total_norm: 18.393370 [I 2022-10-08 23:55:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 1360/2304 [I 2022-10-08 23:55:51] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:06 [I 2022-10-08 23:55:51] ...trap/engines/engine.py.229: process: 0.82907 | load: 0.00302 [I 2022-10-08 23:55:51] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-08 23:55:51] ...optimizers/trijoint.py.151: total_norm: 23.184111 [I 2022-10-08 23:55:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 1370/2304 [I 2022-10-08 23:55:59] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:02 [I 2022-10-08 23:55:59] ...trap/engines/engine.py.229: process: 0.83847 | load: 0.00304 [I 2022-10-08 23:55:59] ...trap/engines/engine.py.230: loss: 0.18728 [I 2022-10-08 23:55:59] ...optimizers/trijoint.py.151: total_norm: 31.199203 [I 2022-10-08 23:56:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 1380/2304 [I 2022-10-08 23:56:08] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:12:53 [I 2022-10-08 23:56:08] ...trap/engines/engine.py.229: process: 0.83278 | load: 0.00302 [I 2022-10-08 23:56:08] ...trap/engines/engine.py.230: loss: 0.20522 [I 2022-10-08 23:56:08] ...optimizers/trijoint.py.151: total_norm: 21.329931 [I 2022-10-08 23:56:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 1390/2304 [I 2022-10-08 23:56:16] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:12:45 [I 2022-10-08 23:56:16] ...trap/engines/engine.py.229: process: 0.83549 | load: 0.00302 [I 2022-10-08 23:56:16] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-08 23:56:16] ...optimizers/trijoint.py.151: total_norm: 21.515619 [I 2022-10-08 23:56:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 1400/2304 [I 2022-10-08 23:56:25] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:35 [I 2022-10-08 23:56:25] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00297 [I 2022-10-08 23:56:25] ...trap/engines/engine.py.230: loss: 0.15471 [I 2022-10-08 23:56:25] ...optimizers/trijoint.py.151: total_norm: 21.110735 [I 2022-10-08 23:56:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 1410/2304 [I 2022-10-08 23:56:33] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:24 [I 2022-10-08 23:56:33] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00302 [I 2022-10-08 23:56:33] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-08 23:56:33] ...optimizers/trijoint.py.151: total_norm: 16.415270 [I 2022-10-08 23:56:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 1420/2304 [I 2022-10-08 23:56:41] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:19 [I 2022-10-08 23:56:41] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00300 [I 2022-10-08 23:56:41] ...trap/engines/engine.py.230: loss: 0.13611 [I 2022-10-08 23:56:41] ...optimizers/trijoint.py.151: total_norm: 22.662464 [I 2022-10-08 23:56:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 1430/2304 [I 2022-10-08 23:56:50] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:07 [I 2022-10-08 23:56:50] ...trap/engines/engine.py.229: process: 0.82682 | load: 0.00298 [I 2022-10-08 23:56:50] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-08 23:56:50] ...optimizers/trijoint.py.151: total_norm: 17.754395 [I 2022-10-08 23:56:58] ...trap/engines/engine.py.224: train: epoch 24 | batch 1440/2304 [I 2022-10-08 23:56:58] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:00 [I 2022-10-08 23:56:58] ...trap/engines/engine.py.229: process: 0.83084 | load: 0.00301 [I 2022-10-08 23:56:58] ...trap/engines/engine.py.230: loss: 0.20499 [I 2022-10-08 23:56:58] ...optimizers/trijoint.py.151: total_norm: 32.308128 [I 2022-10-08 23:57:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 1450/2304 [I 2022-10-08 23:57:06] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:11:50 [I 2022-10-08 23:57:06] ...trap/engines/engine.py.229: process: 0.83385 | load: 0.00300 [I 2022-10-08 23:57:06] ...trap/engines/engine.py.230: loss: 0.19919 [I 2022-10-08 23:57:06] ...optimizers/trijoint.py.151: total_norm: 18.424381 [I 2022-10-08 23:57:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 1460/2304 [I 2022-10-08 23:57:15] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:11:47 [I 2022-10-08 23:57:15] ...trap/engines/engine.py.229: process: 0.83973 | load: 0.00301 [I 2022-10-08 23:57:15] ...trap/engines/engine.py.230: loss: 0.20124 [I 2022-10-08 23:57:15] ...optimizers/trijoint.py.151: total_norm: 30.378305 [I 2022-10-08 23:57:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 1470/2304 [I 2022-10-08 23:57:23] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:11:43 [I 2022-10-08 23:57:23] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00303 [I 2022-10-08 23:57:23] ...trap/engines/engine.py.230: loss: 0.17105 [I 2022-10-08 23:57:23] ...optimizers/trijoint.py.151: total_norm: 15.774328 [I 2022-10-08 23:57:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 1480/2304 [I 2022-10-08 23:57:32] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:11:32 [I 2022-10-08 23:57:32] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00299 [I 2022-10-08 23:57:32] ...trap/engines/engine.py.230: loss: 0.19566 [I 2022-10-08 23:57:32] ...optimizers/trijoint.py.151: total_norm: 20.218786 [I 2022-10-08 23:57:40] ...trap/engines/engine.py.224: train: epoch 24 | batch 1490/2304 [I 2022-10-08 23:57:40] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:11:24 [I 2022-10-08 23:57:40] ...trap/engines/engine.py.229: process: 0.83317 | load: 0.00298 [I 2022-10-08 23:57:40] ...trap/engines/engine.py.230: loss: 0.22727 [I 2022-10-08 23:57:40] ...optimizers/trijoint.py.151: total_norm: 20.391945 [I 2022-10-08 23:57:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 1500/2304 [I 2022-10-08 23:57:48] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:14 [I 2022-10-08 23:57:48] ...trap/engines/engine.py.229: process: 0.83560 | load: 0.00303 [I 2022-10-08 23:57:48] ...trap/engines/engine.py.230: loss: 0.16697 [I 2022-10-08 23:57:48] ...optimizers/trijoint.py.151: total_norm: 19.283010 [I 2022-10-08 23:57:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 1510/2304 [I 2022-10-08 23:57:57] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:06 [I 2022-10-08 23:57:57] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00300 [I 2022-10-08 23:57:57] ...trap/engines/engine.py.230: loss: 0.19002 [I 2022-10-08 23:57:57] ...optimizers/trijoint.py.151: total_norm: 15.134825 [I 2022-10-08 23:58:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 1520/2304 [I 2022-10-08 23:58:05] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:10:55 [I 2022-10-08 23:58:05] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00316 [I 2022-10-08 23:58:05] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-08 23:58:05] ...optimizers/trijoint.py.151: total_norm: 27.578047 [I 2022-10-08 23:58:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 1530/2304 [I 2022-10-08 23:58:14] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:10:47 [I 2022-10-08 23:58:14] ...trap/engines/engine.py.229: process: 0.83549 | load: 0.00298 [I 2022-10-08 23:58:14] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-08 23:58:14] ...optimizers/trijoint.py.151: total_norm: 16.995459 [I 2022-10-08 23:58:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 1540/2304 [I 2022-10-08 23:58:22] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:10:41 [I 2022-10-08 23:58:22] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00300 [I 2022-10-08 23:58:22] ...trap/engines/engine.py.230: loss: 0.15414 [I 2022-10-08 23:58:22] ...optimizers/trijoint.py.151: total_norm: 21.109161 [I 2022-10-08 23:58:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 1550/2304 [I 2022-10-08 23:58:30] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:10:30 [I 2022-10-08 23:58:30] ...trap/engines/engine.py.229: process: 0.83137 | load: 0.00306 [I 2022-10-08 23:58:30] ...trap/engines/engine.py.230: loss: 0.20833 [I 2022-10-08 23:58:30] ...optimizers/trijoint.py.151: total_norm: 18.494488 [I 2022-10-08 23:58:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 1560/2304 [I 2022-10-08 23:58:39] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:10:18 [I 2022-10-08 23:58:39] ...trap/engines/engine.py.229: process: 0.82651 | load: 0.00298 [I 2022-10-08 23:58:39] ...trap/engines/engine.py.230: loss: 0.13144 [I 2022-10-08 23:58:39] ...optimizers/trijoint.py.151: total_norm: 14.343710 [I 2022-10-08 23:58:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 1570/2304 [I 2022-10-08 23:58:47] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:10:13 [I 2022-10-08 23:58:47] ...trap/engines/engine.py.229: process: 0.83480 | load: 0.00298 [I 2022-10-08 23:58:47] ...trap/engines/engine.py.230: loss: 0.18894 [I 2022-10-08 23:58:47] ...optimizers/trijoint.py.151: total_norm: 36.755051 [I 2022-10-08 23:58:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 1580/2304 [I 2022-10-08 23:58:55] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:02 [I 2022-10-08 23:58:55] ...trap/engines/engine.py.229: process: 0.82887 | load: 0.00298 [I 2022-10-08 23:58:55] ...trap/engines/engine.py.230: loss: 0.19980 [I 2022-10-08 23:58:55] ...optimizers/trijoint.py.151: total_norm: 19.306448 [I 2022-10-08 23:59:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 1590/2304 [I 2022-10-08 23:59:04] ...trap/engines/engine.py.228: elapsed: 0:22:17 | left: 0:09:56 [I 2022-10-08 23:59:04] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00301 [I 2022-10-08 23:59:04] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-08 23:59:04] ...optimizers/trijoint.py.151: total_norm: 12.376247 [I 2022-10-08 23:59:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 1600/2304 [I 2022-10-08 23:59:12] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:09:50 [I 2022-10-08 23:59:12] ...trap/engines/engine.py.229: process: 0.84001 | load: 0.00298 [I 2022-10-08 23:59:12] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-08 23:59:12] ...optimizers/trijoint.py.151: total_norm: 14.861652 [I 2022-10-08 23:59:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 1610/2304 [I 2022-10-08 23:59:21] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:09:41 [I 2022-10-08 23:59:21] ...trap/engines/engine.py.229: process: 0.83959 | load: 0.00295 [I 2022-10-08 23:59:21] ...trap/engines/engine.py.230: loss: 0.25115 [I 2022-10-08 23:59:21] ...optimizers/trijoint.py.151: total_norm: 24.091890 [I 2022-10-08 23:59:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 1620/2304 [I 2022-10-08 23:59:47] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:20:15 [I 2022-10-08 23:59:47] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00301 [I 2022-10-08 23:59:47] ...trap/engines/engine.py.230: loss: 0.10554 [I 2022-10-08 23:59:47] ...optimizers/trijoint.py.151: total_norm: 12.703132 [I 2022-10-08 23:59:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 1630/2304 [I 2022-10-08 23:59:55] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:10:31 [I 2022-10-08 23:59:55] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00303 [I 2022-10-08 23:59:55] ...trap/engines/engine.py.230: loss: 0.23260 [I 2022-10-08 23:59:55] ...optimizers/trijoint.py.151: total_norm: 23.177376 [I 2022-10-09 00:00:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 1640/2304 [I 2022-10-09 00:00:04] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:22 [I 2022-10-09 00:00:04] ...trap/engines/engine.py.229: process: 0.83648 | load: 0.00300 [I 2022-10-09 00:00:04] ...trap/engines/engine.py.230: loss: 0.18979 [I 2022-10-09 00:00:04] ...optimizers/trijoint.py.151: total_norm: 19.465502 [I 2022-10-09 00:00:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 1650/2304 [I 2022-10-09 00:00:12] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:09:02 [I 2022-10-09 00:00:12] ...trap/engines/engine.py.229: process: 0.82370 | load: 0.00300 [I 2022-10-09 00:00:12] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-09 00:00:12] ...optimizers/trijoint.py.151: total_norm: 12.624412 [I 2022-10-09 00:00:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 1660/2304 [I 2022-10-09 00:00:20] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:00 [I 2022-10-09 00:00:20] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00301 [I 2022-10-09 00:00:20] ...trap/engines/engine.py.230: loss: 0.23471 [I 2022-10-09 00:00:20] ...optimizers/trijoint.py.151: total_norm: 17.910305 [I 2022-10-09 00:00:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 1670/2304 [I 2022-10-09 00:00:29] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:08:49 [I 2022-10-09 00:00:29] ...trap/engines/engine.py.229: process: 0.84256 | load: 0.00299 [I 2022-10-09 00:00:29] ...trap/engines/engine.py.230: loss: 0.21486 [I 2022-10-09 00:00:29] ...optimizers/trijoint.py.151: total_norm: 12.949530 [I 2022-10-09 00:00:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 1680/2304 [I 2022-10-09 00:00:37] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:08:42 [I 2022-10-09 00:00:37] ...trap/engines/engine.py.229: process: 0.82900 | load: 0.00298 [I 2022-10-09 00:00:37] ...trap/engines/engine.py.230: loss: 0.15322 [I 2022-10-09 00:00:37] ...optimizers/trijoint.py.151: total_norm: 17.106958 [I 2022-10-09 00:00:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 1690/2304 [I 2022-10-09 00:00:45] ...trap/engines/engine.py.228: elapsed: 0:23:59 | left: 0:08:36 [I 2022-10-09 00:00:45] ...trap/engines/engine.py.229: process: 0.84052 | load: 0.00302 [I 2022-10-09 00:00:45] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-09 00:00:45] ...optimizers/trijoint.py.151: total_norm: 20.372988 [I 2022-10-09 00:00:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 1700/2304 [I 2022-10-09 00:00:54] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:25 [I 2022-10-09 00:00:54] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00300 [I 2022-10-09 00:00:54] ...trap/engines/engine.py.230: loss: 0.21125 [I 2022-10-09 00:00:54] ...optimizers/trijoint.py.151: total_norm: 21.477083 [I 2022-10-09 00:01:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 1710/2304 [I 2022-10-09 00:01:02] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:08:16 [I 2022-10-09 00:01:02] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00307 [I 2022-10-09 00:01:02] ...trap/engines/engine.py.230: loss: 0.19960 [I 2022-10-09 00:01:02] ...optimizers/trijoint.py.151: total_norm: 30.943312 [I 2022-10-09 00:01:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 1720/2304 [I 2022-10-09 00:01:11] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:10 [I 2022-10-09 00:01:11] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00309 [I 2022-10-09 00:01:11] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-09 00:01:11] ...optimizers/trijoint.py.151: total_norm: 15.154401 [I 2022-10-09 00:01:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 1730/2304 [I 2022-10-09 00:01:19] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:07:57 [I 2022-10-09 00:01:19] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00306 [I 2022-10-09 00:01:19] ...trap/engines/engine.py.230: loss: 0.14126 [I 2022-10-09 00:01:19] ...optimizers/trijoint.py.151: total_norm: 13.787262 [I 2022-10-09 00:01:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 1740/2304 [I 2022-10-09 00:01:27] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:07:51 [I 2022-10-09 00:01:27] ...trap/engines/engine.py.229: process: 0.83606 | load: 0.00298 [I 2022-10-09 00:01:27] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-09 00:01:27] ...optimizers/trijoint.py.151: total_norm: 26.958513 [I 2022-10-09 00:01:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 1750/2304 [I 2022-10-09 00:01:36] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:07:44 [I 2022-10-09 00:01:36] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00300 [I 2022-10-09 00:01:36] ...trap/engines/engine.py.230: loss: 0.14494 [I 2022-10-09 00:01:36] ...optimizers/trijoint.py.151: total_norm: 11.795186 [I 2022-10-09 00:01:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 1760/2304 [I 2022-10-09 00:01:44] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:07:35 [I 2022-10-09 00:01:44] ...trap/engines/engine.py.229: process: 0.82644 | load: 0.00300 [I 2022-10-09 00:01:44] ...trap/engines/engine.py.230: loss: 0.13750 [I 2022-10-09 00:01:44] ...optimizers/trijoint.py.151: total_norm: 12.756824 [I 2022-10-09 00:01:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 1770/2304 [I 2022-10-09 00:01:53] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:28 [I 2022-10-09 00:01:53] ...trap/engines/engine.py.229: process: 0.83505 | load: 0.00302 [I 2022-10-09 00:01:53] ...trap/engines/engine.py.230: loss: 0.20679 [I 2022-10-09 00:01:53] ...optimizers/trijoint.py.151: total_norm: 15.936986 [I 2022-10-09 00:02:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 1780/2304 [I 2022-10-09 00:02:01] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:19 [I 2022-10-09 00:02:01] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00300 [I 2022-10-09 00:02:01] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-09 00:02:01] ...optimizers/trijoint.py.151: total_norm: 17.827366 [I 2022-10-09 00:02:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 1790/2304 [I 2022-10-09 00:02:10] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:12 [I 2022-10-09 00:02:10] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00300 [I 2022-10-09 00:02:10] ...trap/engines/engine.py.230: loss: 0.19657 [I 2022-10-09 00:02:10] ...optimizers/trijoint.py.151: total_norm: 19.073135 [I 2022-10-09 00:02:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 1800/2304 [I 2022-10-09 00:02:18] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:01 [I 2022-10-09 00:02:18] ...trap/engines/engine.py.229: process: 0.83063 | load: 0.00300 [I 2022-10-09 00:02:18] ...trap/engines/engine.py.230: loss: 0.18267 [I 2022-10-09 00:02:18] ...optimizers/trijoint.py.151: total_norm: 18.612694 [I 2022-10-09 00:02:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 1810/2304 [I 2022-10-09 00:02:26] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:06:52 [I 2022-10-09 00:02:26] ...trap/engines/engine.py.229: process: 0.83194 | load: 0.00301 [I 2022-10-09 00:02:26] ...trap/engines/engine.py.230: loss: 0.19760 [I 2022-10-09 00:02:26] ...optimizers/trijoint.py.151: total_norm: 27.556599 [I 2022-10-09 00:02:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 1820/2304 [I 2022-10-09 00:02:35] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:06:45 [I 2022-10-09 00:02:35] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00300 [I 2022-10-09 00:02:35] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-09 00:02:35] ...optimizers/trijoint.py.151: total_norm: 15.299043 [I 2022-10-09 00:02:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 1830/2304 [I 2022-10-09 00:02:43] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:06:39 [I 2022-10-09 00:02:43] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00300 [I 2022-10-09 00:02:43] ...trap/engines/engine.py.230: loss: 0.28877 [I 2022-10-09 00:02:43] ...optimizers/trijoint.py.151: total_norm: 35.677532 [I 2022-10-09 00:02:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 1840/2304 [I 2022-10-09 00:02:52] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:28 [I 2022-10-09 00:02:52] ...trap/engines/engine.py.229: process: 0.84023 | load: 0.00301 [I 2022-10-09 00:02:52] ...trap/engines/engine.py.230: loss: 0.26597 [I 2022-10-09 00:02:52] ...optimizers/trijoint.py.151: total_norm: 20.933111 [I 2022-10-09 00:03:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 1850/2304 [I 2022-10-09 00:03:00] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:21 [I 2022-10-09 00:03:00] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00299 [I 2022-10-09 00:03:00] ...trap/engines/engine.py.230: loss: 0.17168 [I 2022-10-09 00:03:00] ...optimizers/trijoint.py.151: total_norm: 30.864422 [I 2022-10-09 00:03:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 1860/2304 [I 2022-10-09 00:03:08] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:11 [I 2022-10-09 00:03:08] ...trap/engines/engine.py.229: process: 0.83339 | load: 0.00301 [I 2022-10-09 00:03:08] ...trap/engines/engine.py.230: loss: 0.22598 [I 2022-10-09 00:03:08] ...optimizers/trijoint.py.151: total_norm: 27.379620 [I 2022-10-09 00:03:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 1870/2304 [I 2022-10-09 00:03:17] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:01 [I 2022-10-09 00:03:17] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00300 [I 2022-10-09 00:03:17] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-09 00:03:17] ...optimizers/trijoint.py.151: total_norm: 11.149533 [I 2022-10-09 00:03:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 1880/2304 [I 2022-10-09 00:03:25] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:05:55 [I 2022-10-09 00:03:25] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00301 [I 2022-10-09 00:03:25] ...trap/engines/engine.py.230: loss: 0.16561 [I 2022-10-09 00:03:25] ...optimizers/trijoint.py.151: total_norm: 18.977198 [I 2022-10-09 00:03:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 1890/2304 [I 2022-10-09 00:03:33] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:05:47 [I 2022-10-09 00:03:33] ...trap/engines/engine.py.229: process: 0.83775 | load: 0.00303 [I 2022-10-09 00:03:33] ...trap/engines/engine.py.230: loss: 0.15079 [I 2022-10-09 00:03:33] ...optimizers/trijoint.py.151: total_norm: 20.027472 [I 2022-10-09 00:03:42] ...trap/engines/engine.py.224: train: epoch 24 | batch 1900/2304 [I 2022-10-09 00:03:42] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:05:39 [I 2022-10-09 00:03:42] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00300 [I 2022-10-09 00:03:42] ...trap/engines/engine.py.230: loss: 0.16652 [I 2022-10-09 00:03:42] ...optimizers/trijoint.py.151: total_norm: 17.006311 [I 2022-10-09 00:03:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 1910/2304 [I 2022-10-09 00:03:50] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:30 [I 2022-10-09 00:03:50] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00299 [I 2022-10-09 00:03:50] ...trap/engines/engine.py.230: loss: 0.15396 [I 2022-10-09 00:03:50] ...optimizers/trijoint.py.151: total_norm: 67.704109 [I 2022-10-09 00:03:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 1920/2304 [I 2022-10-09 00:03:59] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:21 [I 2022-10-09 00:03:59] ...trap/engines/engine.py.229: process: 0.82739 | load: 0.00306 [I 2022-10-09 00:03:59] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-09 00:03:59] ...optimizers/trijoint.py.151: total_norm: 25.473000 [I 2022-10-09 00:04:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 1930/2304 [I 2022-10-09 00:04:07] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:12 [I 2022-10-09 00:04:07] ...trap/engines/engine.py.229: process: 0.82402 | load: 0.00300 [I 2022-10-09 00:04:07] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-09 00:04:07] ...optimizers/trijoint.py.151: total_norm: 12.661748 [I 2022-10-09 00:04:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 1940/2304 [I 2022-10-09 00:04:16] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:05 [I 2022-10-09 00:04:16] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00312 [I 2022-10-09 00:04:16] ...trap/engines/engine.py.230: loss: 0.22142 [I 2022-10-09 00:04:16] ...optimizers/trijoint.py.151: total_norm: 27.893888 [I 2022-10-09 00:04:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 1950/2304 [I 2022-10-09 00:04:24] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:04:55 [I 2022-10-09 00:04:24] ...trap/engines/engine.py.229: process: 0.83777 | load: 0.00300 [I 2022-10-09 00:04:24] ...trap/engines/engine.py.230: loss: 0.19774 [I 2022-10-09 00:04:24] ...optimizers/trijoint.py.151: total_norm: 22.363602 [I 2022-10-09 00:04:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 1960/2304 [I 2022-10-09 00:04:32] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:04:49 [I 2022-10-09 00:04:32] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00300 [I 2022-10-09 00:04:32] ...trap/engines/engine.py.230: loss: 0.14822 [I 2022-10-09 00:04:32] ...optimizers/trijoint.py.151: total_norm: 17.143808 [I 2022-10-09 00:04:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 1970/2304 [I 2022-10-09 00:04:41] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:04:37 [I 2022-10-09 00:04:41] ...trap/engines/engine.py.229: process: 0.82217 | load: 0.00303 [I 2022-10-09 00:04:41] ...trap/engines/engine.py.230: loss: 0.17250 [I 2022-10-09 00:04:41] ...optimizers/trijoint.py.151: total_norm: 14.659498 [I 2022-10-09 00:04:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 1980/2304 [I 2022-10-09 00:04:49] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:31 [I 2022-10-09 00:04:49] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00296 [I 2022-10-09 00:04:49] ...trap/engines/engine.py.230: loss: 0.20733 [I 2022-10-09 00:04:49] ...optimizers/trijoint.py.151: total_norm: 14.906658 [I 2022-10-09 00:04:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 1990/2304 [I 2022-10-09 00:04:57] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:21 [I 2022-10-09 00:04:57] ...trap/engines/engine.py.229: process: 0.83131 | load: 0.00305 [I 2022-10-09 00:04:57] ...trap/engines/engine.py.230: loss: 0.14978 [I 2022-10-09 00:04:57] ...optimizers/trijoint.py.151: total_norm: 16.892836 [I 2022-10-09 00:05:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 2000/2304 [I 2022-10-09 00:05:06] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:14 [I 2022-10-09 00:05:06] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00310 [I 2022-10-09 00:05:06] ...trap/engines/engine.py.230: loss: 0.14131 [I 2022-10-09 00:05:06] ...optimizers/trijoint.py.151: total_norm: 13.634336 [I 2022-10-09 00:05:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 2010/2304 [I 2022-10-09 00:05:14] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:05 [I 2022-10-09 00:05:14] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00301 [I 2022-10-09 00:05:14] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-09 00:05:14] ...optimizers/trijoint.py.151: total_norm: 24.289179 [I 2022-10-09 00:05:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 2020/2304 [I 2022-10-09 00:05:23] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:03:57 [I 2022-10-09 00:05:23] ...trap/engines/engine.py.229: process: 0.83210 | load: 0.00298 [I 2022-10-09 00:05:23] ...trap/engines/engine.py.230: loss: 0.18525 [I 2022-10-09 00:05:23] ...optimizers/trijoint.py.151: total_norm: 21.158113 [I 2022-10-09 00:05:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 2030/2304 [I 2022-10-09 00:05:31] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:03:50 [I 2022-10-09 00:05:31] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00301 [I 2022-10-09 00:05:31] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-09 00:05:31] ...optimizers/trijoint.py.151: total_norm: 20.742748 [I 2022-10-09 00:05:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 2040/2304 [I 2022-10-09 00:05:39] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:03:41 [I 2022-10-09 00:05:39] ...trap/engines/engine.py.229: process: 0.82475 | load: 0.00302 [I 2022-10-09 00:05:39] ...trap/engines/engine.py.230: loss: 0.13871 [I 2022-10-09 00:05:39] ...optimizers/trijoint.py.151: total_norm: 19.048584 [I 2022-10-09 00:05:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 2050/2304 [I 2022-10-09 00:05:48] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:32 [I 2022-10-09 00:05:48] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00301 [I 2022-10-09 00:05:48] ...trap/engines/engine.py.230: loss: 0.15542 [I 2022-10-09 00:05:48] ...optimizers/trijoint.py.151: total_norm: 40.240784 [I 2022-10-09 00:05:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 2060/2304 [I 2022-10-09 00:05:56] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:23 [I 2022-10-09 00:05:56] ...trap/engines/engine.py.229: process: 0.83490 | load: 0.00302 [I 2022-10-09 00:05:56] ...trap/engines/engine.py.230: loss: 0.15335 [I 2022-10-09 00:05:56] ...optimizers/trijoint.py.151: total_norm: 12.722409 [I 2022-10-09 00:06:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 2070/2304 [I 2022-10-09 00:06:04] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:15 [I 2022-10-09 00:06:04] ...trap/engines/engine.py.229: process: 0.84580 | load: 0.00302 [I 2022-10-09 00:06:04] ...trap/engines/engine.py.230: loss: 0.21299 [I 2022-10-09 00:06:04] ...optimizers/trijoint.py.151: total_norm: 22.244291 [I 2022-10-09 00:06:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 2080/2304 [I 2022-10-09 00:06:13] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:07 [I 2022-10-09 00:06:13] ...trap/engines/engine.py.229: process: 0.82832 | load: 0.00300 [I 2022-10-09 00:06:13] ...trap/engines/engine.py.230: loss: 0.14656 [I 2022-10-09 00:06:13] ...optimizers/trijoint.py.151: total_norm: 13.053720 [I 2022-10-09 00:06:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 2090/2304 [I 2022-10-09 00:06:21] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:02:57 [I 2022-10-09 00:06:21] ...trap/engines/engine.py.229: process: 0.82740 | load: 0.00301 [I 2022-10-09 00:06:21] ...trap/engines/engine.py.230: loss: 0.21881 [I 2022-10-09 00:06:21] ...optimizers/trijoint.py.151: total_norm: 27.153887 [I 2022-10-09 00:06:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 2100/2304 [I 2022-10-09 00:06:29] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:02:49 [I 2022-10-09 00:06:29] ...trap/engines/engine.py.229: process: 0.82146 | load: 0.00300 [I 2022-10-09 00:06:29] ...trap/engines/engine.py.230: loss: 0.17762 [I 2022-10-09 00:06:29] ...optimizers/trijoint.py.151: total_norm: 27.970829 [I 2022-10-09 00:06:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 2110/2304 [I 2022-10-09 00:06:38] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:02:42 [I 2022-10-09 00:06:38] ...trap/engines/engine.py.229: process: 0.82737 | load: 0.00299 [I 2022-10-09 00:06:38] ...trap/engines/engine.py.230: loss: 0.20635 [I 2022-10-09 00:06:38] ...optimizers/trijoint.py.151: total_norm: 27.115477 [I 2022-10-09 00:06:46] ...trap/engines/engine.py.224: train: epoch 24 | batch 2120/2304 [I 2022-10-09 00:06:46] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:02:34 [I 2022-10-09 00:06:46] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00306 [I 2022-10-09 00:06:46] ...trap/engines/engine.py.230: loss: 0.19798 [I 2022-10-09 00:06:46] ...optimizers/trijoint.py.151: total_norm: 23.628780 [I 2022-10-09 00:06:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 2130/2304 [I 2022-10-09 00:06:55] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:26 [I 2022-10-09 00:06:55] ...trap/engines/engine.py.229: process: 0.83457 | load: 0.00302 [I 2022-10-09 00:06:55] ...trap/engines/engine.py.230: loss: 0.18821 [I 2022-10-09 00:06:55] ...optimizers/trijoint.py.151: total_norm: 20.823175 [I 2022-10-09 00:07:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 2140/2304 [I 2022-10-09 00:07:03] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:16 [I 2022-10-09 00:07:03] ...trap/engines/engine.py.229: process: 0.82385 | load: 0.00301 [I 2022-10-09 00:07:03] ...trap/engines/engine.py.230: loss: 0.13331 [I 2022-10-09 00:07:03] ...optimizers/trijoint.py.151: total_norm: 14.016695 [I 2022-10-09 00:07:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 2150/2304 [I 2022-10-09 00:07:11] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:08 [I 2022-10-09 00:07:11] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00300 [I 2022-10-09 00:07:11] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-09 00:07:11] ...optimizers/trijoint.py.151: total_norm: 20.236683 [I 2022-10-09 00:07:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 2160/2304 [I 2022-10-09 00:07:20] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:00 [I 2022-10-09 00:07:20] ...trap/engines/engine.py.229: process: 0.84145 | load: 0.00299 [I 2022-10-09 00:07:20] ...trap/engines/engine.py.230: loss: 0.18272 [I 2022-10-09 00:07:20] ...optimizers/trijoint.py.151: total_norm: 15.920227 [I 2022-10-09 00:07:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 2170/2304 [I 2022-10-09 00:07:28] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:01:51 [I 2022-10-09 00:07:28] ...trap/engines/engine.py.229: process: 0.83477 | load: 0.00305 [I 2022-10-09 00:07:28] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-09 00:07:28] ...optimizers/trijoint.py.151: total_norm: 14.967139 [I 2022-10-09 00:07:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 2180/2304 [I 2022-10-09 00:07:36] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:01:42 [I 2022-10-09 00:07:36] ...trap/engines/engine.py.229: process: 0.82022 | load: 0.00300 [I 2022-10-09 00:07:36] ...trap/engines/engine.py.230: loss: 0.14887 [I 2022-10-09 00:07:36] ...optimizers/trijoint.py.151: total_norm: 19.279745 [I 2022-10-09 00:07:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 2190/2304 [I 2022-10-09 00:07:45] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:01:34 [I 2022-10-09 00:07:45] ...trap/engines/engine.py.229: process: 0.82995 | load: 0.00302 [I 2022-10-09 00:07:45] ...trap/engines/engine.py.230: loss: 0.24032 [I 2022-10-09 00:07:45] ...optimizers/trijoint.py.151: total_norm: 41.873116 [I 2022-10-09 00:07:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 2200/2304 [I 2022-10-09 00:07:53] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:26 [I 2022-10-09 00:07:53] ...trap/engines/engine.py.229: process: 0.83164 | load: 0.00302 [I 2022-10-09 00:07:53] ...trap/engines/engine.py.230: loss: 0.15523 [I 2022-10-09 00:07:53] ...optimizers/trijoint.py.151: total_norm: 16.232021 [I 2022-10-09 00:08:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 2210/2304 [I 2022-10-09 00:08:02] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:18 [I 2022-10-09 00:08:02] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00302 [I 2022-10-09 00:08:02] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-09 00:08:02] ...optimizers/trijoint.py.151: total_norm: 22.403500 [I 2022-10-09 00:08:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 2220/2304 [I 2022-10-09 00:08:10] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:09 [I 2022-10-09 00:08:10] ...trap/engines/engine.py.229: process: 0.83091 | load: 0.00307 [I 2022-10-09 00:08:10] ...trap/engines/engine.py.230: loss: 0.19564 [I 2022-10-09 00:08:10] ...optimizers/trijoint.py.151: total_norm: 21.875311 [I 2022-10-09 00:08:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 2230/2304 [I 2022-10-09 00:08:18] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:01 [I 2022-10-09 00:08:18] ...trap/engines/engine.py.229: process: 0.84047 | load: 0.00299 [I 2022-10-09 00:08:18] ...trap/engines/engine.py.230: loss: 0.15691 [I 2022-10-09 00:08:18] ...optimizers/trijoint.py.151: total_norm: 17.041754 [I 2022-10-09 00:08:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 2240/2304 [I 2022-10-09 00:08:27] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:00:53 [I 2022-10-09 00:08:27] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00300 [I 2022-10-09 00:08:27] ...trap/engines/engine.py.230: loss: 0.21247 [I 2022-10-09 00:08:27] ...optimizers/trijoint.py.151: total_norm: 16.310326 [I 2022-10-09 00:08:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 2250/2304 [I 2022-10-09 00:08:35] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:44 [I 2022-10-09 00:08:35] ...trap/engines/engine.py.229: process: 0.82263 | load: 0.00299 [I 2022-10-09 00:08:35] ...trap/engines/engine.py.230: loss: 0.17758 [I 2022-10-09 00:08:35] ...optimizers/trijoint.py.151: total_norm: 19.735605 [I 2022-10-09 00:08:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 2260/2304 [I 2022-10-09 00:08:43] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:36 [I 2022-10-09 00:08:43] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00300 [I 2022-10-09 00:08:43] ...trap/engines/engine.py.230: loss: 0.20901 [I 2022-10-09 00:08:43] ...optimizers/trijoint.py.151: total_norm: 23.734207 [I 2022-10-09 00:08:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 2270/2304 [I 2022-10-09 00:08:52] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:28 [I 2022-10-09 00:08:52] ...trap/engines/engine.py.229: process: 0.81125 | load: 0.00299 [I 2022-10-09 00:08:52] ...trap/engines/engine.py.230: loss: 0.15792 [I 2022-10-09 00:08:52] ...optimizers/trijoint.py.151: total_norm: 22.123335 [I 2022-10-09 00:09:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 2280/2304 [I 2022-10-09 00:09:00] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:19 [I 2022-10-09 00:09:00] ...trap/engines/engine.py.229: process: 0.83404 | load: 0.00301 [I 2022-10-09 00:09:00] ...trap/engines/engine.py.230: loss: 0.16913 [I 2022-10-09 00:09:00] ...optimizers/trijoint.py.151: total_norm: 15.812175 [I 2022-10-09 00:09:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 2290/2304 [I 2022-10-09 00:09:08] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:11 [I 2022-10-09 00:09:08] ...trap/engines/engine.py.229: process: 0.84032 | load: 0.00304 [I 2022-10-09 00:09:08] ...trap/engines/engine.py.230: loss: 0.19148 [I 2022-10-09 00:09:08] ...optimizers/trijoint.py.151: total_norm: 21.372522 [I 2022-10-09 00:09:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 2300/2304 [I 2022-10-09 00:09:17] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:03 [I 2022-10-09 00:09:17] ...trap/engines/engine.py.229: process: 0.81804 | load: 0.00018 [I 2022-10-09 00:09:17] ...trap/engines/engine.py.230: loss: 0.26985 [I 2022-10-09 00:09:17] ...optimizers/trijoint.py.151: total_norm: 24.570168 [I 2022-10-09 00:09:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 2304/2304 [I 2022-10-09 00:09:20] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:00 [I 2022-10-09 00:09:20] ...trap/engines/engine.py.229: process: 0.82751 | load: 0.00018 [I 2022-10-09 00:09:20] ...trap/engines/engine.py.230: loss: 0.18885 [I 2022-10-09 00:09:20] ...optimizers/trijoint.py.151: total_norm: 16.598883 [S 2022-10-09 00:09:21] ...trap/engines/engine.py.129: train_epoch.epoch: 24 [S 2022-10-09 00:09:21] ...trap/engines/engine.py.129: train_epoch.loss: 0.17357853570884324 [I 2022-10-09 00:09:21] ...s/criterions/triplet.py.84: margin = 0.1750000000000001 [I 2022-10-09 00:09:22] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 24 [I 2022-10-09 00:09:22] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 00:09:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 00:09:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 00:09:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 00:09:24] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 00:09:25] ...trap/engines/engine.py.321: eval: epoch 24 | batch 0/510 [I 2022-10-09 00:09:25] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:26:08 [I 2022-10-09 00:09:25] ...trap/engines/engine.py.326: process: 3.07564 | load: 2.54837 [I 2022-10-09 00:09:29] ...trap/engines/engine.py.321: eval: epoch 24 | batch 10/510 [I 2022-10-09 00:09:29] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:54 [I 2022-10-09 00:09:29] ...trap/engines/engine.py.326: process: 0.42113 | load: 0.00018 [I 2022-10-09 00:09:34] ...trap/engines/engine.py.321: eval: epoch 24 | batch 20/510 [I 2022-10-09 00:09:34] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:04:03 [I 2022-10-09 00:09:34] ...trap/engines/engine.py.326: process: 0.55540 | load: 0.00018 [I 2022-10-09 00:09:40] ...trap/engines/engine.py.321: eval: epoch 24 | batch 30/510 [I 2022-10-09 00:09:40] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:57 [I 2022-10-09 00:09:40] ...trap/engines/engine.py.326: process: 0.53583 | load: 0.00019 [I 2022-10-09 00:09:47] ...trap/engines/engine.py.321: eval: epoch 24 | batch 40/510 [I 2022-10-09 00:09:47] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:21 [I 2022-10-09 00:09:47] ...trap/engines/engine.py.326: process: 0.64387 | load: 0.00019 [I 2022-10-09 00:09:52] ...trap/engines/engine.py.321: eval: epoch 24 | batch 50/510 [I 2022-10-09 00:09:52] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:01 [I 2022-10-09 00:09:52] ...trap/engines/engine.py.326: process: 0.53501 | load: 0.00018 [I 2022-10-09 00:09:57] ...trap/engines/engine.py.321: eval: epoch 24 | batch 60/510 [I 2022-10-09 00:09:57] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:48 [I 2022-10-09 00:09:57] ...trap/engines/engine.py.326: process: 0.58343 | load: 0.00018 [I 2022-10-09 00:10:02] ...trap/engines/engine.py.321: eval: epoch 24 | batch 70/510 [I 2022-10-09 00:10:02] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:05 [I 2022-10-09 00:10:02] ...trap/engines/engine.py.326: process: 0.60586 | load: 0.00020 [I 2022-10-09 00:10:08] ...trap/engines/engine.py.321: eval: epoch 24 | batch 80/510 [I 2022-10-09 00:10:08] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:36 [I 2022-10-09 00:10:08] ...trap/engines/engine.py.326: process: 0.45856 | load: 0.00018 [I 2022-10-09 00:10:12] ...trap/engines/engine.py.321: eval: epoch 24 | batch 90/510 [I 2022-10-09 00:10:12] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:13 [I 2022-10-09 00:10:12] ...trap/engines/engine.py.326: process: 0.46088 | load: 0.00018 [I 2022-10-09 00:10:16] ...trap/engines/engine.py.321: eval: epoch 24 | batch 100/510 [I 2022-10-09 00:10:16] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:50 [I 2022-10-09 00:10:16] ...trap/engines/engine.py.326: process: 0.31196 | load: 0.00018 [I 2022-10-09 00:10:19] ...trap/engines/engine.py.321: eval: epoch 24 | batch 110/510 [I 2022-10-09 00:10:19] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:11 [I 2022-10-09 00:10:19] ...trap/engines/engine.py.326: process: 0.31878 | load: 0.00020 [I 2022-10-09 00:10:23] ...trap/engines/engine.py.321: eval: epoch 24 | batch 120/510 [I 2022-10-09 00:10:23] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-09 00:10:23] ...trap/engines/engine.py.326: process: 0.30958 | load: 0.00019 [I 2022-10-09 00:10:26] ...trap/engines/engine.py.321: eval: epoch 24 | batch 130/510 [I 2022-10-09 00:10:26] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-09 00:10:26] ...trap/engines/engine.py.326: process: 0.31663 | load: 0.00019 [I 2022-10-09 00:10:29] ...trap/engines/engine.py.321: eval: epoch 24 | batch 140/510 [I 2022-10-09 00:10:29] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:56 [I 2022-10-09 00:10:29] ...trap/engines/engine.py.326: process: 0.31526 | load: 0.00020 [I 2022-10-09 00:10:32] ...trap/engines/engine.py.321: eval: epoch 24 | batch 150/510 [I 2022-10-09 00:10:32] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-09 00:10:32] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00020 [I 2022-10-09 00:10:35] ...trap/engines/engine.py.321: eval: epoch 24 | batch 160/510 [I 2022-10-09 00:10:35] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-09 00:10:35] ...trap/engines/engine.py.326: process: 0.31133 | load: 0.00019 [I 2022-10-09 00:10:38] ...trap/engines/engine.py.321: eval: epoch 24 | batch 170/510 [I 2022-10-09 00:10:38] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-09 00:10:38] ...trap/engines/engine.py.326: process: 0.31561 | load: 0.00019 [I 2022-10-09 00:10:42] ...trap/engines/engine.py.321: eval: epoch 24 | batch 180/510 [I 2022-10-09 00:10:42] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-09 00:10:42] ...trap/engines/engine.py.326: process: 0.32337 | load: 0.00020 [I 2022-10-09 00:10:45] ...trap/engines/engine.py.321: eval: epoch 24 | batch 190/510 [I 2022-10-09 00:10:45] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:42 [I 2022-10-09 00:10:45] ...trap/engines/engine.py.326: process: 0.31447 | load: 0.00019 [I 2022-10-09 00:10:48] ...trap/engines/engine.py.321: eval: epoch 24 | batch 200/510 [I 2022-10-09 00:10:48] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-09 00:10:48] ...trap/engines/engine.py.326: process: 0.32253 | load: 0.00021 [I 2022-10-09 00:10:51] ...trap/engines/engine.py.321: eval: epoch 24 | batch 210/510 [I 2022-10-09 00:10:51] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:36 [I 2022-10-09 00:10:51] ...trap/engines/engine.py.326: process: 0.31869 | load: 0.00020 [I 2022-10-09 00:10:55] ...trap/engines/engine.py.321: eval: epoch 24 | batch 220/510 [I 2022-10-09 00:10:55] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:33 [I 2022-10-09 00:10:55] ...trap/engines/engine.py.326: process: 0.32166 | load: 0.00020 [I 2022-10-09 00:10:58] ...trap/engines/engine.py.321: eval: epoch 24 | batch 230/510 [I 2022-10-09 00:10:58] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-09 00:10:58] ...trap/engines/engine.py.326: process: 0.32279 | load: 0.00021 [I 2022-10-09 00:11:01] ...trap/engines/engine.py.321: eval: epoch 24 | batch 240/510 [I 2022-10-09 00:11:01] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-09 00:11:01] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00028 [I 2022-10-09 00:11:04] ...trap/engines/engine.py.321: eval: epoch 24 | batch 250/510 [I 2022-10-09 00:11:04] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-09 00:11:04] ...trap/engines/engine.py.326: process: 0.32309 | load: 0.00021 [I 2022-10-09 00:11:07] ...trap/engines/engine.py.321: eval: epoch 24 | batch 260/510 [I 2022-10-09 00:11:07] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:20 [I 2022-10-09 00:11:07] ...trap/engines/engine.py.326: process: 0.32200 | load: 0.00019 [I 2022-10-09 00:11:11] ...trap/engines/engine.py.321: eval: epoch 24 | batch 270/510 [I 2022-10-09 00:11:11] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-09 00:11:11] ...trap/engines/engine.py.326: process: 0.31977 | load: 0.00020 [I 2022-10-09 00:11:14] ...trap/engines/engine.py.321: eval: epoch 24 | batch 280/510 [I 2022-10-09 00:11:14] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 00:11:14] ...trap/engines/engine.py.326: process: 0.31726 | load: 0.00021 [I 2022-10-09 00:11:17] ...trap/engines/engine.py.321: eval: epoch 24 | batch 290/510 [I 2022-10-09 00:11:17] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-09 00:11:17] ...trap/engines/engine.py.326: process: 0.32138 | load: 0.00020 [I 2022-10-09 00:11:20] ...trap/engines/engine.py.321: eval: epoch 24 | batch 300/510 [I 2022-10-09 00:11:20] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-09 00:11:20] ...trap/engines/engine.py.326: process: 0.32275 | load: 0.00021 [I 2022-10-09 00:11:23] ...trap/engines/engine.py.321: eval: epoch 24 | batch 310/510 [I 2022-10-09 00:11:23] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:04 [I 2022-10-09 00:11:23] ...trap/engines/engine.py.326: process: 0.32018 | load: 0.00021 [I 2022-10-09 00:11:27] ...trap/engines/engine.py.321: eval: epoch 24 | batch 320/510 [I 2022-10-09 00:11:27] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 00:11:27] ...trap/engines/engine.py.326: process: 0.31516 | load: 0.00021 [I 2022-10-09 00:11:30] ...trap/engines/engine.py.321: eval: epoch 24 | batch 330/510 [I 2022-10-09 00:11:30] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 00:11:30] ...trap/engines/engine.py.326: process: 0.32100 | load: 0.00021 [I 2022-10-09 00:11:33] ...trap/engines/engine.py.321: eval: epoch 24 | batch 340/510 [I 2022-10-09 00:11:33] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 00:11:33] ...trap/engines/engine.py.326: process: 0.32365 | load: 0.00021 [I 2022-10-09 00:11:36] ...trap/engines/engine.py.321: eval: epoch 24 | batch 350/510 [I 2022-10-09 00:11:36] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-09 00:11:36] ...trap/engines/engine.py.326: process: 0.32071 | load: 0.00020 [I 2022-10-09 00:11:39] ...trap/engines/engine.py.321: eval: epoch 24 | batch 360/510 [I 2022-10-09 00:11:39] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:48 [I 2022-10-09 00:11:39] ...trap/engines/engine.py.326: process: 0.32137 | load: 0.00020 [I 2022-10-09 00:11:43] ...trap/engines/engine.py.321: eval: epoch 24 | batch 370/510 [I 2022-10-09 00:11:43] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-09 00:11:43] ...trap/engines/engine.py.326: process: 0.31491 | load: 0.00019 [I 2022-10-09 00:11:46] ...trap/engines/engine.py.321: eval: epoch 24 | batch 380/510 [I 2022-10-09 00:11:46] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 00:11:46] ...trap/engines/engine.py.326: process: 0.32333 | load: 0.00021 [I 2022-10-09 00:11:49] ...trap/engines/engine.py.321: eval: epoch 24 | batch 390/510 [I 2022-10-09 00:11:49] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-09 00:11:49] ...trap/engines/engine.py.326: process: 0.31365 | load: 0.00020 [I 2022-10-09 00:11:52] ...trap/engines/engine.py.321: eval: epoch 24 | batch 400/510 [I 2022-10-09 00:11:52] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-09 00:11:52] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00020 [I 2022-10-09 00:11:55] ...trap/engines/engine.py.321: eval: epoch 24 | batch 410/510 [I 2022-10-09 00:11:55] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-09 00:11:55] ...trap/engines/engine.py.326: process: 0.31742 | load: 0.00021 [I 2022-10-09 00:11:59] ...trap/engines/engine.py.321: eval: epoch 24 | batch 420/510 [I 2022-10-09 00:11:59] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 00:11:59] ...trap/engines/engine.py.326: process: 0.31424 | load: 0.00021 [I 2022-10-09 00:12:02] ...trap/engines/engine.py.321: eval: epoch 24 | batch 430/510 [I 2022-10-09 00:12:02] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 00:12:02] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00021 [I 2022-10-09 00:12:05] ...trap/engines/engine.py.321: eval: epoch 24 | batch 440/510 [I 2022-10-09 00:12:05] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 00:12:05] ...trap/engines/engine.py.326: process: 0.32163 | load: 0.00020 [I 2022-10-09 00:12:08] ...trap/engines/engine.py.321: eval: epoch 24 | batch 450/510 [I 2022-10-09 00:12:08] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 00:12:08] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00020 [I 2022-10-09 00:12:11] ...trap/engines/engine.py.321: eval: epoch 24 | batch 460/510 [I 2022-10-09 00:12:11] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-09 00:12:11] ...trap/engines/engine.py.326: process: 0.31544 | load: 0.00021 [I 2022-10-09 00:12:15] ...trap/engines/engine.py.321: eval: epoch 24 | batch 470/510 [I 2022-10-09 00:12:15] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 00:12:15] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00020 [I 2022-10-09 00:12:18] ...trap/engines/engine.py.321: eval: epoch 24 | batch 480/510 [I 2022-10-09 00:12:18] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 00:12:18] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00020 [I 2022-10-09 00:12:21] ...trap/engines/engine.py.321: eval: epoch 24 | batch 490/510 [I 2022-10-09 00:12:21] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 00:12:21] ...trap/engines/engine.py.326: process: 0.32506 | load: 0.00021 [I 2022-10-09 00:12:24] ...trap/engines/engine.py.321: eval: epoch 24 | batch 500/510 [I 2022-10-09 00:12:24] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 00:12:24] ...trap/engines/engine.py.326: process: 0.31766 | load: 0.00022 [I 2022-10-09 00:12:27] ...trap/engines/engine.py.321: eval: epoch 24 | batch 510/510 [I 2022-10-09 00:12:27] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-09 00:12:27] ...trap/engines/engine.py.326: process: 0.31647 | load: 0.00016 [S 2022-10-09 00:12:28] ...trap/engines/engine.py.132: eval_epoch.epoch: 24 [I 2022-10-09 00:12:28] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.672400 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.886800 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929600 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.667200 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.884500 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.926200 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009982 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009867 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006960 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014702 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011192 [S 2022-10-09 00:12:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007054 [I 2022-10-09 00:13:10] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 00:13:10] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 00:13:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 00:13:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 00:13:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 00:13:11] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 00:13:14] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 00:13:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 00:13:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 00:13:19] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 00:13:22] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 00:13:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 00:13:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 25 [I 2022-10-09 00:13:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 00:13:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 0/2304 [I 2022-10-09 00:13:31] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:17 [I 2022-10-09 00:13:31] ...trap/engines/engine.py.229: process: 3.00260 | load: 2.09859 [I 2022-10-09 00:13:31] ...trap/engines/engine.py.230: loss: 0.17672 [I 2022-10-09 00:13:31] ...optimizers/trijoint.py.151: total_norm: 17.178833 [I 2022-10-09 00:13:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 10/2304 [I 2022-10-09 00:13:40] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:06 [I 2022-10-09 00:13:40] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00628 [I 2022-10-09 00:13:40] ...trap/engines/engine.py.230: loss: 0.13079 [I 2022-10-09 00:13:40] ...optimizers/trijoint.py.151: total_norm: 12.727807 [I 2022-10-09 00:13:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 20/2304 [I 2022-10-09 00:13:48] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:52 [I 2022-10-09 00:13:48] ...trap/engines/engine.py.229: process: 0.83071 | load: 0.00410 [I 2022-10-09 00:13:48] ...trap/engines/engine.py.230: loss: 0.12210 [I 2022-10-09 00:13:48] ...optimizers/trijoint.py.151: total_norm: 24.336273 [I 2022-10-09 00:13:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 30/2304 [I 2022-10-09 00:13:57] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:05 [I 2022-10-09 00:13:57] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00422 [I 2022-10-09 00:13:57] ...trap/engines/engine.py.230: loss: 0.18592 [I 2022-10-09 00:13:57] ...optimizers/trijoint.py.151: total_norm: 20.810352 [I 2022-10-09 00:14:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 40/2304 [I 2022-10-09 00:14:05] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:47 [I 2022-10-09 00:14:05] ...trap/engines/engine.py.229: process: 0.83677 | load: 0.00423 [I 2022-10-09 00:14:05] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-09 00:14:05] ...optimizers/trijoint.py.151: total_norm: 10.856900 [I 2022-10-09 00:14:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 50/2304 [I 2022-10-09 00:14:14] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:35 [I 2022-10-09 00:14:14] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00367 [I 2022-10-09 00:14:14] ...trap/engines/engine.py.230: loss: 0.19424 [I 2022-10-09 00:14:14] ...optimizers/trijoint.py.151: total_norm: 28.261732 [I 2022-10-09 00:14:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 60/2304 [I 2022-10-09 00:14:22] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:26 [I 2022-10-09 00:14:22] ...trap/engines/engine.py.229: process: 0.82372 | load: 0.00382 [I 2022-10-09 00:14:22] ...trap/engines/engine.py.230: loss: 0.17332 [I 2022-10-09 00:14:22] ...optimizers/trijoint.py.151: total_norm: 21.984812 [I 2022-10-09 00:14:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 70/2304 [I 2022-10-09 00:14:31] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:27 [I 2022-10-09 00:14:31] ...trap/engines/engine.py.229: process: 0.85858 | load: 0.00355 [I 2022-10-09 00:14:31] ...trap/engines/engine.py.230: loss: 0.17102 [I 2022-10-09 00:14:31] ...optimizers/trijoint.py.151: total_norm: 27.306324 [I 2022-10-09 00:14:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 80/2304 [I 2022-10-09 00:14:39] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:03 [I 2022-10-09 00:14:39] ...trap/engines/engine.py.229: process: 0.83784 | load: 0.00348 [I 2022-10-09 00:14:39] ...trap/engines/engine.py.230: loss: 0.11364 [I 2022-10-09 00:14:39] ...optimizers/trijoint.py.151: total_norm: 20.470692 [I 2022-10-09 00:14:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 90/2304 [I 2022-10-09 00:14:47] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:59 [I 2022-10-09 00:14:47] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00358 [I 2022-10-09 00:14:47] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-09 00:14:47] ...optimizers/trijoint.py.151: total_norm: 9.976677 [I 2022-10-09 00:14:56] ...trap/engines/engine.py.224: train: epoch 25 | batch 100/2304 [I 2022-10-09 00:14:56] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:51 [I 2022-10-09 00:14:56] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00337 [I 2022-10-09 00:14:56] ...trap/engines/engine.py.230: loss: 0.16770 [I 2022-10-09 00:14:56] ...optimizers/trijoint.py.151: total_norm: 12.270229 [I 2022-10-09 00:15:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 110/2304 [I 2022-10-09 00:15:04] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:33 [I 2022-10-09 00:15:04] ...trap/engines/engine.py.229: process: 0.83261 | load: 0.00315 [I 2022-10-09 00:15:04] ...trap/engines/engine.py.230: loss: 0.16562 [I 2022-10-09 00:15:04] ...optimizers/trijoint.py.151: total_norm: 10.524944 [I 2022-10-09 00:15:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 120/2304 [I 2022-10-09 00:15:13] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:32 [I 2022-10-09 00:15:13] ...trap/engines/engine.py.229: process: 0.82712 | load: 0.00325 [I 2022-10-09 00:15:13] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-09 00:15:13] ...optimizers/trijoint.py.151: total_norm: 13.310024 [I 2022-10-09 00:15:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 130/2304 [I 2022-10-09 00:15:21] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:29 [I 2022-10-09 00:15:21] ...trap/engines/engine.py.229: process: 0.83406 | load: 0.00317 [I 2022-10-09 00:15:21] ...trap/engines/engine.py.230: loss: 0.15062 [I 2022-10-09 00:15:21] ...optimizers/trijoint.py.151: total_norm: 14.937824 [I 2022-10-09 00:15:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 140/2304 [I 2022-10-09 00:15:29] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:10 [I 2022-10-09 00:15:29] ...trap/engines/engine.py.229: process: 0.82171 | load: 0.00322 [I 2022-10-09 00:15:29] ...trap/engines/engine.py.230: loss: 0.08814 [I 2022-10-09 00:15:29] ...optimizers/trijoint.py.151: total_norm: 7.696023 [I 2022-10-09 00:15:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 150/2304 [I 2022-10-09 00:15:38] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:00 [I 2022-10-09 00:15:38] ...trap/engines/engine.py.229: process: 0.83784 | load: 0.00320 [I 2022-10-09 00:15:38] ...trap/engines/engine.py.230: loss: 0.16601 [I 2022-10-09 00:15:38] ...optimizers/trijoint.py.151: total_norm: 10.218605 [I 2022-10-09 00:15:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 160/2304 [I 2022-10-09 00:15:46] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:29:51 [I 2022-10-09 00:15:46] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00328 [I 2022-10-09 00:15:46] ...trap/engines/engine.py.230: loss: 0.22353 [I 2022-10-09 00:15:46] ...optimizers/trijoint.py.151: total_norm: 22.294420 [I 2022-10-09 00:15:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 170/2304 [I 2022-10-09 00:15:54] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:44 [I 2022-10-09 00:15:54] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00313 [I 2022-10-09 00:15:54] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-09 00:15:54] ...optimizers/trijoint.py.151: total_norm: 25.252794 [I 2022-10-09 00:16:03] ...trap/engines/engine.py.224: train: epoch 25 | batch 180/2304 [I 2022-10-09 00:16:03] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:35 [I 2022-10-09 00:16:03] ...trap/engines/engine.py.229: process: 0.81811 | load: 0.00298 [I 2022-10-09 00:16:03] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-09 00:16:03] ...optimizers/trijoint.py.151: total_norm: 12.992970 [I 2022-10-09 00:16:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 190/2304 [I 2022-10-09 00:16:11] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:33 [I 2022-10-09 00:16:11] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00315 [I 2022-10-09 00:16:11] ...trap/engines/engine.py.230: loss: 0.19857 [I 2022-10-09 00:16:11] ...optimizers/trijoint.py.151: total_norm: 16.305515 [I 2022-10-09 00:16:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 200/2304 [I 2022-10-09 00:16:20] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:24 [I 2022-10-09 00:16:20] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00304 [I 2022-10-09 00:16:20] ...trap/engines/engine.py.230: loss: 0.16584 [I 2022-10-09 00:16:20] ...optimizers/trijoint.py.151: total_norm: 12.911343 [I 2022-10-09 00:16:28] ...trap/engines/engine.py.224: train: epoch 25 | batch 210/2304 [I 2022-10-09 00:16:28] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:20 [I 2022-10-09 00:16:28] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00307 [I 2022-10-09 00:16:28] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-09 00:16:28] ...optimizers/trijoint.py.151: total_norm: 9.515321 [I 2022-10-09 00:16:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 220/2304 [I 2022-10-09 00:16:36] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:09 [I 2022-10-09 00:16:36] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00326 [I 2022-10-09 00:16:36] ...trap/engines/engine.py.230: loss: 0.13516 [I 2022-10-09 00:16:36] ...optimizers/trijoint.py.151: total_norm: 18.119434 [I 2022-10-09 00:16:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 230/2304 [I 2022-10-09 00:16:45] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:03 [I 2022-10-09 00:16:45] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00304 [I 2022-10-09 00:16:45] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-09 00:16:45] ...optimizers/trijoint.py.151: total_norm: 22.376320 [I 2022-10-09 00:16:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 240/2304 [I 2022-10-09 00:16:53] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:28:49 [I 2022-10-09 00:16:53] ...trap/engines/engine.py.229: process: 0.82851 | load: 0.00319 [I 2022-10-09 00:16:53] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-09 00:16:53] ...optimizers/trijoint.py.151: total_norm: 11.104432 [I 2022-10-09 00:17:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 250/2304 [I 2022-10-09 00:17:02] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:38 [I 2022-10-09 00:17:02] ...trap/engines/engine.py.229: process: 0.83672 | load: 0.00315 [I 2022-10-09 00:17:02] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-09 00:17:02] ...optimizers/trijoint.py.151: total_norm: 15.772622 [I 2022-10-09 00:17:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 260/2304 [I 2022-10-09 00:17:10] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:43 [I 2022-10-09 00:17:10] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00315 [I 2022-10-09 00:17:10] ...trap/engines/engine.py.230: loss: 0.24074 [I 2022-10-09 00:17:10] ...optimizers/trijoint.py.151: total_norm: 28.841057 [I 2022-10-09 00:17:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 270/2304 [I 2022-10-09 00:17:19] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:30 [I 2022-10-09 00:17:19] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00311 [I 2022-10-09 00:17:19] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-09 00:17:19] ...optimizers/trijoint.py.151: total_norm: 15.571340 [I 2022-10-09 00:17:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 280/2304 [I 2022-10-09 00:17:27] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:23 [I 2022-10-09 00:17:27] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00310 [I 2022-10-09 00:17:27] ...trap/engines/engine.py.230: loss: 0.12290 [I 2022-10-09 00:17:27] ...optimizers/trijoint.py.151: total_norm: 12.844205 [I 2022-10-09 00:17:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 290/2304 [I 2022-10-09 00:17:35] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:28:13 [I 2022-10-09 00:17:35] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00316 [I 2022-10-09 00:17:35] ...trap/engines/engine.py.230: loss: 0.14869 [I 2022-10-09 00:17:35] ...optimizers/trijoint.py.151: total_norm: 25.190609 [I 2022-10-09 00:17:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 300/2304 [I 2022-10-09 00:17:44] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:03 [I 2022-10-09 00:17:44] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00302 [I 2022-10-09 00:17:44] ...trap/engines/engine.py.230: loss: 0.15106 [I 2022-10-09 00:17:44] ...optimizers/trijoint.py.151: total_norm: 17.307177 [I 2022-10-09 00:17:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 310/2304 [I 2022-10-09 00:17:52] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:27:51 [I 2022-10-09 00:17:52] ...trap/engines/engine.py.229: process: 0.82606 | load: 0.00300 [I 2022-10-09 00:17:52] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-09 00:17:52] ...optimizers/trijoint.py.151: total_norm: 13.488460 [I 2022-10-09 00:18:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 320/2304 [I 2022-10-09 00:18:01] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:43 [I 2022-10-09 00:18:01] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00315 [I 2022-10-09 00:18:01] ...trap/engines/engine.py.230: loss: 0.14754 [I 2022-10-09 00:18:01] ...optimizers/trijoint.py.151: total_norm: 19.115927 [I 2022-10-09 00:18:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 330/2304 [I 2022-10-09 00:18:09] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:25 [I 2022-10-09 00:18:09] ...trap/engines/engine.py.229: process: 0.82844 | load: 0.00307 [I 2022-10-09 00:18:09] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-09 00:18:09] ...optimizers/trijoint.py.151: total_norm: 26.490688 [I 2022-10-09 00:18:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 340/2304 [I 2022-10-09 00:18:17] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:41 [I 2022-10-09 00:18:17] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00298 [I 2022-10-09 00:18:17] ...trap/engines/engine.py.230: loss: 0.13244 [I 2022-10-09 00:18:17] ...optimizers/trijoint.py.151: total_norm: 16.647869 [I 2022-10-09 00:18:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 350/2304 [I 2022-10-09 00:18:26] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:17 [I 2022-10-09 00:18:26] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00308 [I 2022-10-09 00:18:26] ...trap/engines/engine.py.230: loss: 0.13627 [I 2022-10-09 00:18:26] ...optimizers/trijoint.py.151: total_norm: 12.983290 [I 2022-10-09 00:18:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 360/2304 [I 2022-10-09 00:18:34] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:11 [I 2022-10-09 00:18:34] ...trap/engines/engine.py.229: process: 0.83756 | load: 0.00300 [I 2022-10-09 00:18:34] ...trap/engines/engine.py.230: loss: 0.21933 [I 2022-10-09 00:18:34] ...optimizers/trijoint.py.151: total_norm: 27.287371 [I 2022-10-09 00:18:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 370/2304 [I 2022-10-09 00:18:43] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:07 [I 2022-10-09 00:18:43] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00307 [I 2022-10-09 00:18:43] ...trap/engines/engine.py.230: loss: 0.16645 [I 2022-10-09 00:18:43] ...optimizers/trijoint.py.151: total_norm: 25.314819 [I 2022-10-09 00:18:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 380/2304 [I 2022-10-09 00:18:51] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:49 [I 2022-10-09 00:18:51] ...trap/engines/engine.py.229: process: 0.82636 | load: 0.00302 [I 2022-10-09 00:18:51] ...trap/engines/engine.py.230: loss: 0.18935 [I 2022-10-09 00:18:51] ...optimizers/trijoint.py.151: total_norm: 24.622780 [I 2022-10-09 00:18:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 390/2304 [I 2022-10-09 00:18:59] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:49 [I 2022-10-09 00:18:59] ...trap/engines/engine.py.229: process: 0.84976 | load: 0.00311 [I 2022-10-09 00:18:59] ...trap/engines/engine.py.230: loss: 0.22600 [I 2022-10-09 00:18:59] ...optimizers/trijoint.py.151: total_norm: 16.432467 [I 2022-10-09 00:19:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 400/2304 [I 2022-10-09 00:19:08] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:38 [I 2022-10-09 00:19:08] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00301 [I 2022-10-09 00:19:08] ...trap/engines/engine.py.230: loss: 0.22306 [I 2022-10-09 00:19:08] ...optimizers/trijoint.py.151: total_norm: 17.111031 [I 2022-10-09 00:19:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 410/2304 [I 2022-10-09 00:19:16] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:26:32 [I 2022-10-09 00:19:16] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00301 [I 2022-10-09 00:19:16] ...trap/engines/engine.py.230: loss: 0.21385 [I 2022-10-09 00:19:16] ...optimizers/trijoint.py.151: total_norm: 25.838348 [I 2022-10-09 00:19:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 420/2304 [I 2022-10-09 00:19:25] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:25 [I 2022-10-09 00:19:25] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00311 [I 2022-10-09 00:19:25] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-09 00:19:25] ...optimizers/trijoint.py.151: total_norm: 31.782558 [I 2022-10-09 00:19:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 430/2304 [I 2022-10-09 00:19:33] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:19 [I 2022-10-09 00:19:33] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00300 [I 2022-10-09 00:19:33] ...trap/engines/engine.py.230: loss: 0.17277 [I 2022-10-09 00:19:33] ...optimizers/trijoint.py.151: total_norm: 16.955822 [I 2022-10-09 00:19:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 440/2304 [I 2022-10-09 00:19:41] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:00 [I 2022-10-09 00:19:41] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00302 [I 2022-10-09 00:19:41] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-09 00:19:41] ...optimizers/trijoint.py.151: total_norm: 17.322580 [I 2022-10-09 00:19:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 450/2304 [I 2022-10-09 00:19:50] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:00 [I 2022-10-09 00:19:50] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00301 [I 2022-10-09 00:19:50] ...trap/engines/engine.py.230: loss: 0.13253 [I 2022-10-09 00:19:50] ...optimizers/trijoint.py.151: total_norm: 11.630595 [I 2022-10-09 00:19:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 460/2304 [I 2022-10-09 00:19:58] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:25:41 [I 2022-10-09 00:19:58] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00299 [I 2022-10-09 00:19:58] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-09 00:19:58] ...optimizers/trijoint.py.151: total_norm: 17.349228 [I 2022-10-09 00:20:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 470/2304 [I 2022-10-09 00:20:07] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:43 [I 2022-10-09 00:20:07] ...trap/engines/engine.py.229: process: 0.83303 | load: 0.00305 [I 2022-10-09 00:20:07] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-09 00:20:07] ...optimizers/trijoint.py.151: total_norm: 23.430983 [I 2022-10-09 00:20:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 480/2304 [I 2022-10-09 00:20:15] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:25:34 [I 2022-10-09 00:20:15] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00300 [I 2022-10-09 00:20:15] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-09 00:20:15] ...optimizers/trijoint.py.151: total_norm: 10.321154 [I 2022-10-09 00:20:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 490/2304 [I 2022-10-09 00:20:24] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:17 [I 2022-10-09 00:20:24] ...trap/engines/engine.py.229: process: 0.83040 | load: 0.00302 [I 2022-10-09 00:20:24] ...trap/engines/engine.py.230: loss: 0.19088 [I 2022-10-09 00:20:24] ...optimizers/trijoint.py.151: total_norm: 20.908812 [I 2022-10-09 00:20:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 500/2304 [I 2022-10-09 00:20:32] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:25:14 [I 2022-10-09 00:20:32] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00302 [I 2022-10-09 00:20:32] ...trap/engines/engine.py.230: loss: 0.13804 [I 2022-10-09 00:20:32] ...optimizers/trijoint.py.151: total_norm: 11.923278 [I 2022-10-09 00:20:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 510/2304 [I 2022-10-09 00:20:40] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:25:07 [I 2022-10-09 00:20:40] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00305 [I 2022-10-09 00:20:40] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-09 00:20:40] ...optimizers/trijoint.py.151: total_norm: 19.216564 [I 2022-10-09 00:20:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 520/2304 [I 2022-10-09 00:20:49] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:25:01 [I 2022-10-09 00:20:49] ...trap/engines/engine.py.229: process: 0.83797 | load: 0.00301 [I 2022-10-09 00:20:49] ...trap/engines/engine.py.230: loss: 0.31333 [I 2022-10-09 00:20:49] ...optimizers/trijoint.py.151: total_norm: 33.084148 [I 2022-10-09 00:20:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 530/2304 [I 2022-10-09 00:20:57] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:24:54 [I 2022-10-09 00:20:57] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00311 [I 2022-10-09 00:20:57] ...trap/engines/engine.py.230: loss: 0.12567 [I 2022-10-09 00:20:57] ...optimizers/trijoint.py.151: total_norm: 16.107145 [I 2022-10-09 00:21:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 540/2304 [I 2022-10-09 00:21:06] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:43 [I 2022-10-09 00:21:06] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00308 [I 2022-10-09 00:21:06] ...trap/engines/engine.py.230: loss: 0.20314 [I 2022-10-09 00:21:06] ...optimizers/trijoint.py.151: total_norm: 16.800423 [I 2022-10-09 00:21:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 550/2304 [I 2022-10-09 00:21:14] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:24:35 [I 2022-10-09 00:21:14] ...trap/engines/engine.py.229: process: 0.83610 | load: 0.00313 [I 2022-10-09 00:21:14] ...trap/engines/engine.py.230: loss: 0.17058 [I 2022-10-09 00:21:14] ...optimizers/trijoint.py.151: total_norm: 14.829309 [I 2022-10-09 00:21:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 560/2304 [I 2022-10-09 00:21:22] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:21 [I 2022-10-09 00:21:22] ...trap/engines/engine.py.229: process: 0.82597 | load: 0.00299 [I 2022-10-09 00:21:22] ...trap/engines/engine.py.230: loss: 0.15217 [I 2022-10-09 00:21:22] ...optimizers/trijoint.py.151: total_norm: 20.709101 [I 2022-10-09 00:21:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 570/2304 [I 2022-10-09 00:21:31] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:14 [I 2022-10-09 00:21:31] ...trap/engines/engine.py.229: process: 0.83009 | load: 0.00302 [I 2022-10-09 00:21:31] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-09 00:21:31] ...optimizers/trijoint.py.151: total_norm: 14.639364 [I 2022-10-09 00:21:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 580/2304 [I 2022-10-09 00:21:39] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:24:09 [I 2022-10-09 00:21:39] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00299 [I 2022-10-09 00:21:39] ...trap/engines/engine.py.230: loss: 0.16749 [I 2022-10-09 00:21:39] ...optimizers/trijoint.py.151: total_norm: 38.835457 [I 2022-10-09 00:21:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 590/2304 [I 2022-10-09 00:21:48] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:24:01 [I 2022-10-09 00:21:48] ...trap/engines/engine.py.229: process: 0.83546 | load: 0.00300 [I 2022-10-09 00:21:48] ...trap/engines/engine.py.230: loss: 0.20835 [I 2022-10-09 00:21:48] ...optimizers/trijoint.py.151: total_norm: 23.255566 [I 2022-10-09 00:21:56] ...trap/engines/engine.py.224: train: epoch 25 | batch 600/2304 [I 2022-10-09 00:21:56] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:23:52 [I 2022-10-09 00:21:56] ...trap/engines/engine.py.229: process: 0.83044 | load: 0.00300 [I 2022-10-09 00:21:56] ...trap/engines/engine.py.230: loss: 0.19392 [I 2022-10-09 00:21:56] ...optimizers/trijoint.py.151: total_norm: 23.309036 [I 2022-10-09 00:22:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 610/2304 [I 2022-10-09 00:22:05] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:23:39 [I 2022-10-09 00:22:05] ...trap/engines/engine.py.229: process: 0.82667 | load: 0.00300 [I 2022-10-09 00:22:05] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-09 00:22:05] ...optimizers/trijoint.py.151: total_norm: 11.742682 [I 2022-10-09 00:22:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 620/2304 [I 2022-10-09 00:22:13] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:38 [I 2022-10-09 00:22:13] ...trap/engines/engine.py.229: process: 0.83613 | load: 0.00304 [I 2022-10-09 00:22:13] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-09 00:22:13] ...optimizers/trijoint.py.151: total_norm: 14.811241 [I 2022-10-09 00:22:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 630/2304 [I 2022-10-09 00:22:22] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:35 [I 2022-10-09 00:22:22] ...trap/engines/engine.py.229: process: 0.84323 | load: 0.00303 [I 2022-10-09 00:22:22] ...trap/engines/engine.py.230: loss: 0.19075 [I 2022-10-09 00:22:22] ...optimizers/trijoint.py.151: total_norm: 17.071436 [I 2022-10-09 00:22:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 640/2304 [I 2022-10-09 00:22:30] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:28 [I 2022-10-09 00:22:30] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00300 [I 2022-10-09 00:22:30] ...trap/engines/engine.py.230: loss: 0.20576 [I 2022-10-09 00:22:30] ...optimizers/trijoint.py.151: total_norm: 17.830521 [I 2022-10-09 00:22:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 650/2304 [I 2022-10-09 00:22:38] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:09 [I 2022-10-09 00:22:38] ...trap/engines/engine.py.229: process: 0.83401 | load: 0.00303 [I 2022-10-09 00:22:38] ...trap/engines/engine.py.230: loss: 0.19805 [I 2022-10-09 00:22:38] ...optimizers/trijoint.py.151: total_norm: 34.091106 [I 2022-10-09 00:22:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 660/2304 [I 2022-10-09 00:22:47] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:14 [I 2022-10-09 00:22:47] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00299 [I 2022-10-09 00:22:47] ...trap/engines/engine.py.230: loss: 0.15580 [I 2022-10-09 00:22:47] ...optimizers/trijoint.py.151: total_norm: 20.417995 [I 2022-10-09 00:22:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 670/2304 [I 2022-10-09 00:22:55] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:03 [I 2022-10-09 00:22:55] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00299 [I 2022-10-09 00:22:55] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-09 00:22:55] ...optimizers/trijoint.py.151: total_norm: 23.103519 [I 2022-10-09 00:23:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 680/2304 [I 2022-10-09 00:23:04] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:47 [I 2022-10-09 00:23:04] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00298 [I 2022-10-09 00:23:04] ...trap/engines/engine.py.230: loss: 0.13609 [I 2022-10-09 00:23:04] ...optimizers/trijoint.py.151: total_norm: 18.618837 [I 2022-10-09 00:23:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 690/2304 [I 2022-10-09 00:23:12] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:22:36 [I 2022-10-09 00:23:12] ...trap/engines/engine.py.229: process: 0.83392 | load: 0.00302 [I 2022-10-09 00:23:12] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-09 00:23:12] ...optimizers/trijoint.py.151: total_norm: 21.056942 [I 2022-10-09 00:23:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 700/2304 [I 2022-10-09 00:23:21] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:30 [I 2022-10-09 00:23:21] ...trap/engines/engine.py.229: process: 0.85333 | load: 0.00301 [I 2022-10-09 00:23:21] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-09 00:23:21] ...optimizers/trijoint.py.151: total_norm: 14.607289 [I 2022-10-09 00:23:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 710/2304 [I 2022-10-09 00:23:29] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:21 [I 2022-10-09 00:23:29] ...trap/engines/engine.py.229: process: 0.84251 | load: 0.00302 [I 2022-10-09 00:23:29] ...trap/engines/engine.py.230: loss: 0.24701 [I 2022-10-09 00:23:29] ...optimizers/trijoint.py.151: total_norm: 38.364162 [I 2022-10-09 00:23:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 720/2304 [I 2022-10-09 00:23:37] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:13 [I 2022-10-09 00:23:37] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00302 [I 2022-10-09 00:23:37] ...trap/engines/engine.py.230: loss: 0.16937 [I 2022-10-09 00:23:37] ...optimizers/trijoint.py.151: total_norm: 31.045481 [I 2022-10-09 00:23:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 730/2304 [I 2022-10-09 00:23:46] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:04 [I 2022-10-09 00:23:46] ...trap/engines/engine.py.229: process: 0.85822 | load: 0.00298 [I 2022-10-09 00:23:46] ...trap/engines/engine.py.230: loss: 0.17972 [I 2022-10-09 00:23:46] ...optimizers/trijoint.py.151: total_norm: 16.054815 [I 2022-10-09 00:23:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 740/2304 [I 2022-10-09 00:23:54] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:21:52 [I 2022-10-09 00:23:54] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00301 [I 2022-10-09 00:23:54] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-09 00:23:54] ...optimizers/trijoint.py.151: total_norm: 14.458250 [I 2022-10-09 00:24:03] ...trap/engines/engine.py.224: train: epoch 25 | batch 750/2304 [I 2022-10-09 00:24:03] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:51 [I 2022-10-09 00:24:03] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00300 [I 2022-10-09 00:24:03] ...trap/engines/engine.py.230: loss: 0.15415 [I 2022-10-09 00:24:03] ...optimizers/trijoint.py.151: total_norm: 16.106060 [I 2022-10-09 00:24:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 760/2304 [I 2022-10-09 00:24:11] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:34 [I 2022-10-09 00:24:11] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00300 [I 2022-10-09 00:24:11] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-09 00:24:11] ...optimizers/trijoint.py.151: total_norm: 11.844654 [I 2022-10-09 00:24:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 770/2304 [I 2022-10-09 00:24:20] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:32 [I 2022-10-09 00:24:20] ...trap/engines/engine.py.229: process: 0.83101 | load: 0.00313 [I 2022-10-09 00:24:20] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-09 00:24:20] ...optimizers/trijoint.py.151: total_norm: 24.177971 [I 2022-10-09 00:24:28] ...trap/engines/engine.py.224: train: epoch 25 | batch 780/2304 [I 2022-10-09 00:24:28] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:26 [I 2022-10-09 00:24:28] ...trap/engines/engine.py.229: process: 0.85824 | load: 0.00299 [I 2022-10-09 00:24:28] ...trap/engines/engine.py.230: loss: 0.17917 [I 2022-10-09 00:24:28] ...optimizers/trijoint.py.151: total_norm: 16.671812 [I 2022-10-09 00:24:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 790/2304 [I 2022-10-09 00:24:36] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:13 [I 2022-10-09 00:24:36] ...trap/engines/engine.py.229: process: 0.83213 | load: 0.00300 [I 2022-10-09 00:24:36] ...trap/engines/engine.py.230: loss: 0.26252 [I 2022-10-09 00:24:36] ...optimizers/trijoint.py.151: total_norm: 16.145630 [I 2022-10-09 00:24:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 800/2304 [I 2022-10-09 00:24:45] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:20:59 [I 2022-10-09 00:24:45] ...trap/engines/engine.py.229: process: 0.83241 | load: 0.00299 [I 2022-10-09 00:24:45] ...trap/engines/engine.py.230: loss: 0.16584 [I 2022-10-09 00:24:45] ...optimizers/trijoint.py.151: total_norm: 16.459864 [I 2022-10-09 00:24:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 810/2304 [I 2022-10-09 00:24:53] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:20:52 [I 2022-10-09 00:24:53] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00300 [I 2022-10-09 00:24:53] ...trap/engines/engine.py.230: loss: 0.25899 [I 2022-10-09 00:24:53] ...optimizers/trijoint.py.151: total_norm: 25.087391 [I 2022-10-09 00:25:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 820/2304 [I 2022-10-09 00:25:02] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:20:51 [I 2022-10-09 00:25:02] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00300 [I 2022-10-09 00:25:02] ...trap/engines/engine.py.230: loss: 0.12935 [I 2022-10-09 00:25:02] ...optimizers/trijoint.py.151: total_norm: 27.071594 [I 2022-10-09 00:25:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 830/2304 [I 2022-10-09 00:25:10] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:36 [I 2022-10-09 00:25:10] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00302 [I 2022-10-09 00:25:10] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-09 00:25:10] ...optimizers/trijoint.py.151: total_norm: 9.901066 [I 2022-10-09 00:25:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 840/2304 [I 2022-10-09 00:25:18] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:27 [I 2022-10-09 00:25:18] ...trap/engines/engine.py.229: process: 0.82449 | load: 0.00307 [I 2022-10-09 00:25:18] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-09 00:25:18] ...optimizers/trijoint.py.151: total_norm: 26.882938 [I 2022-10-09 00:25:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 850/2304 [I 2022-10-09 00:25:27] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:24 [I 2022-10-09 00:25:27] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00307 [I 2022-10-09 00:25:27] ...trap/engines/engine.py.230: loss: 0.23931 [I 2022-10-09 00:25:27] ...optimizers/trijoint.py.151: total_norm: 23.063622 [I 2022-10-09 00:25:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 860/2304 [I 2022-10-09 00:25:35] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:08 [I 2022-10-09 00:25:35] ...trap/engines/engine.py.229: process: 0.83900 | load: 0.00309 [I 2022-10-09 00:25:35] ...trap/engines/engine.py.230: loss: 0.24855 [I 2022-10-09 00:25:35] ...optimizers/trijoint.py.151: total_norm: 28.194761 [I 2022-10-09 00:25:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 870/2304 [I 2022-10-09 00:25:44] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:02 [I 2022-10-09 00:25:44] ...trap/engines/engine.py.229: process: 0.83523 | load: 0.00310 [I 2022-10-09 00:25:44] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-09 00:25:44] ...optimizers/trijoint.py.151: total_norm: 23.373577 [I 2022-10-09 00:25:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 880/2304 [I 2022-10-09 00:25:52] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:19:50 [I 2022-10-09 00:25:52] ...trap/engines/engine.py.229: process: 0.83691 | load: 0.00308 [I 2022-10-09 00:25:52] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-09 00:25:52] ...optimizers/trijoint.py.151: total_norm: 20.328209 [I 2022-10-09 00:26:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 890/2304 [I 2022-10-09 00:26:00] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:19:41 [I 2022-10-09 00:26:00] ...trap/engines/engine.py.229: process: 0.82221 | load: 0.00302 [I 2022-10-09 00:26:00] ...trap/engines/engine.py.230: loss: 0.20083 [I 2022-10-09 00:26:00] ...optimizers/trijoint.py.151: total_norm: 19.414520 [I 2022-10-09 00:26:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 900/2304 [I 2022-10-09 00:26:09] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:19:37 [I 2022-10-09 00:26:09] ...trap/engines/engine.py.229: process: 0.82951 | load: 0.00300 [I 2022-10-09 00:26:09] ...trap/engines/engine.py.230: loss: 0.18657 [I 2022-10-09 00:26:09] ...optimizers/trijoint.py.151: total_norm: 14.752712 [I 2022-10-09 00:26:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 910/2304 [I 2022-10-09 00:26:17] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:19:30 [I 2022-10-09 00:26:17] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00302 [I 2022-10-09 00:26:17] ...trap/engines/engine.py.230: loss: 0.23525 [I 2022-10-09 00:26:17] ...optimizers/trijoint.py.151: total_norm: 23.956907 [I 2022-10-09 00:26:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 920/2304 [I 2022-10-09 00:26:26] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:18 [I 2022-10-09 00:26:26] ...trap/engines/engine.py.229: process: 0.84277 | load: 0.00301 [I 2022-10-09 00:26:26] ...trap/engines/engine.py.230: loss: 0.17763 [I 2022-10-09 00:26:26] ...optimizers/trijoint.py.151: total_norm: 35.755161 [I 2022-10-09 00:26:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 930/2304 [I 2022-10-09 00:26:34] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:19:15 [I 2022-10-09 00:26:34] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00297 [I 2022-10-09 00:26:34] ...trap/engines/engine.py.230: loss: 0.09842 [I 2022-10-09 00:26:34] ...optimizers/trijoint.py.151: total_norm: 17.991919 [I 2022-10-09 00:26:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 940/2304 [I 2022-10-09 00:26:42] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:02 [I 2022-10-09 00:26:42] ...trap/engines/engine.py.229: process: 0.84223 | load: 0.00305 [I 2022-10-09 00:26:42] ...trap/engines/engine.py.230: loss: 0.22130 [I 2022-10-09 00:26:42] ...optimizers/trijoint.py.151: total_norm: 20.594402 [I 2022-10-09 00:26:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 950/2304 [I 2022-10-09 00:26:51] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:19:01 [I 2022-10-09 00:26:51] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00300 [I 2022-10-09 00:26:51] ...trap/engines/engine.py.230: loss: 0.13132 [I 2022-10-09 00:26:51] ...optimizers/trijoint.py.151: total_norm: 10.546447 [I 2022-10-09 00:26:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 960/2304 [I 2022-10-09 00:26:59] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:18:46 [I 2022-10-09 00:26:59] ...trap/engines/engine.py.229: process: 0.83737 | load: 0.00302 [I 2022-10-09 00:26:59] ...trap/engines/engine.py.230: loss: 0.14655 [I 2022-10-09 00:26:59] ...optimizers/trijoint.py.151: total_norm: 15.350883 [I 2022-10-09 00:27:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 970/2304 [I 2022-10-09 00:27:08] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:18:38 [I 2022-10-09 00:27:08] ...trap/engines/engine.py.229: process: 0.81868 | load: 0.00299 [I 2022-10-09 00:27:08] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-09 00:27:08] ...optimizers/trijoint.py.151: total_norm: 19.094948 [I 2022-10-09 00:27:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 980/2304 [I 2022-10-09 00:27:16] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:18:35 [I 2022-10-09 00:27:16] ...trap/engines/engine.py.229: process: 0.82945 | load: 0.00303 [I 2022-10-09 00:27:16] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-09 00:27:16] ...optimizers/trijoint.py.151: total_norm: 17.839209 [I 2022-10-09 00:27:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 990/2304 [I 2022-10-09 00:27:25] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:28 [I 2022-10-09 00:27:25] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00301 [I 2022-10-09 00:27:25] ...trap/engines/engine.py.230: loss: 0.20070 [I 2022-10-09 00:27:25] ...optimizers/trijoint.py.151: total_norm: 19.027163 [I 2022-10-09 00:27:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 1000/2304 [I 2022-10-09 00:27:33] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:17 [I 2022-10-09 00:27:33] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00301 [I 2022-10-09 00:27:33] ...trap/engines/engine.py.230: loss: 0.20031 [I 2022-10-09 00:27:33] ...optimizers/trijoint.py.151: total_norm: 19.794073 [I 2022-10-09 00:27:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 1010/2304 [I 2022-10-09 00:27:41] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:05 [I 2022-10-09 00:27:41] ...trap/engines/engine.py.229: process: 0.84158 | load: 0.00302 [I 2022-10-09 00:27:41] ...trap/engines/engine.py.230: loss: 0.22887 [I 2022-10-09 00:27:41] ...optimizers/trijoint.py.151: total_norm: 28.317955 [I 2022-10-09 00:27:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 1020/2304 [I 2022-10-09 00:27:50] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:18:01 [I 2022-10-09 00:27:50] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00304 [I 2022-10-09 00:27:50] ...trap/engines/engine.py.230: loss: 0.14323 [I 2022-10-09 00:27:50] ...optimizers/trijoint.py.151: total_norm: 14.812502 [I 2022-10-09 00:27:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 1030/2304 [I 2022-10-09 00:27:58] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:17:44 [I 2022-10-09 00:27:58] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00301 [I 2022-10-09 00:27:58] ...trap/engines/engine.py.230: loss: 0.13912 [I 2022-10-09 00:27:58] ...optimizers/trijoint.py.151: total_norm: 48.204826 [I 2022-10-09 00:28:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 1040/2304 [I 2022-10-09 00:28:07] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:17:41 [I 2022-10-09 00:28:07] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00301 [I 2022-10-09 00:28:07] ...trap/engines/engine.py.230: loss: 0.17104 [I 2022-10-09 00:28:07] ...optimizers/trijoint.py.151: total_norm: 16.297533 [I 2022-10-09 00:28:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 1050/2304 [I 2022-10-09 00:28:15] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:17:37 [I 2022-10-09 00:28:15] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00301 [I 2022-10-09 00:28:15] ...trap/engines/engine.py.230: loss: 0.15174 [I 2022-10-09 00:28:15] ...optimizers/trijoint.py.151: total_norm: 23.370760 [I 2022-10-09 00:28:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 1060/2304 [I 2022-10-09 00:28:24] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:17:27 [I 2022-10-09 00:28:24] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00302 [I 2022-10-09 00:28:24] ...trap/engines/engine.py.230: loss: 0.18045 [I 2022-10-09 00:28:24] ...optimizers/trijoint.py.151: total_norm: 17.832350 [I 2022-10-09 00:28:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 1070/2304 [I 2022-10-09 00:28:32] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:20 [I 2022-10-09 00:28:32] ...trap/engines/engine.py.229: process: 0.84702 | load: 0.00303 [I 2022-10-09 00:28:32] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-09 00:28:32] ...optimizers/trijoint.py.151: total_norm: 8.273125 [I 2022-10-09 00:28:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 1080/2304 [I 2022-10-09 00:28:40] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:06 [I 2022-10-09 00:28:40] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00311 [I 2022-10-09 00:28:40] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-09 00:28:40] ...optimizers/trijoint.py.151: total_norm: 31.353210 [I 2022-10-09 00:28:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 1090/2304 [I 2022-10-09 00:28:49] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:16:55 [I 2022-10-09 00:28:49] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00303 [I 2022-10-09 00:28:49] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-09 00:28:49] ...optimizers/trijoint.py.151: total_norm: 35.571182 [I 2022-10-09 00:28:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 1100/2304 [I 2022-10-09 00:28:57] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:16:52 [I 2022-10-09 00:28:57] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00304 [I 2022-10-09 00:28:57] ...trap/engines/engine.py.230: loss: 0.18338 [I 2022-10-09 00:28:57] ...optimizers/trijoint.py.151: total_norm: 13.967937 [I 2022-10-09 00:29:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 1110/2304 [I 2022-10-09 00:29:06] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:16:44 [I 2022-10-09 00:29:06] ...trap/engines/engine.py.229: process: 0.84732 | load: 0.00301 [I 2022-10-09 00:29:06] ...trap/engines/engine.py.230: loss: 0.18131 [I 2022-10-09 00:29:06] ...optimizers/trijoint.py.151: total_norm: 15.981606 [I 2022-10-09 00:29:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 1120/2304 [I 2022-10-09 00:29:14] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:16:34 [I 2022-10-09 00:29:14] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00303 [I 2022-10-09 00:29:14] ...trap/engines/engine.py.230: loss: 0.13038 [I 2022-10-09 00:29:14] ...optimizers/trijoint.py.151: total_norm: 15.851852 [I 2022-10-09 00:29:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 1130/2304 [I 2022-10-09 00:29:22] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:16:28 [I 2022-10-09 00:29:22] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00300 [I 2022-10-09 00:29:22] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-09 00:29:22] ...optimizers/trijoint.py.151: total_norm: 5.998930 [I 2022-10-09 00:29:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 1140/2304 [I 2022-10-09 00:29:31] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:14 [I 2022-10-09 00:29:31] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00303 [I 2022-10-09 00:29:31] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-09 00:29:31] ...optimizers/trijoint.py.151: total_norm: 12.464279 [I 2022-10-09 00:29:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 1150/2304 [I 2022-10-09 00:29:39] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:16:10 [I 2022-10-09 00:29:39] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00301 [I 2022-10-09 00:29:39] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-09 00:29:39] ...optimizers/trijoint.py.151: total_norm: 24.011875 [I 2022-10-09 00:29:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 1160/2304 [I 2022-10-09 00:29:48] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:15:59 [I 2022-10-09 00:29:48] ...trap/engines/engine.py.229: process: 0.84433 | load: 0.00300 [I 2022-10-09 00:29:48] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-09 00:29:48] ...optimizers/trijoint.py.151: total_norm: 14.080940 [I 2022-10-09 00:29:56] ...trap/engines/engine.py.224: train: epoch 25 | batch 1170/2304 [I 2022-10-09 00:29:56] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:15:49 [I 2022-10-09 00:29:56] ...trap/engines/engine.py.229: process: 0.83581 | load: 0.00300 [I 2022-10-09 00:29:56] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-09 00:29:56] ...optimizers/trijoint.py.151: total_norm: 16.988287 [I 2022-10-09 00:30:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 1180/2304 [I 2022-10-09 00:30:05] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:15:44 [I 2022-10-09 00:30:05] ...trap/engines/engine.py.229: process: 0.83263 | load: 0.00299 [I 2022-10-09 00:30:05] ...trap/engines/engine.py.230: loss: 0.16634 [I 2022-10-09 00:30:05] ...optimizers/trijoint.py.151: total_norm: 12.694072 [I 2022-10-09 00:30:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 1190/2304 [I 2022-10-09 00:30:13] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:15:31 [I 2022-10-09 00:30:13] ...trap/engines/engine.py.229: process: 0.83061 | load: 0.00310 [I 2022-10-09 00:30:13] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-09 00:30:13] ...optimizers/trijoint.py.151: total_norm: 21.017412 [I 2022-10-09 00:30:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 1200/2304 [I 2022-10-09 00:30:21] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:15:33 [I 2022-10-09 00:30:21] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00309 [I 2022-10-09 00:30:21] ...trap/engines/engine.py.230: loss: 0.31069 [I 2022-10-09 00:30:21] ...optimizers/trijoint.py.151: total_norm: 12.892093 [I 2022-10-09 00:30:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 1210/2304 [I 2022-10-09 00:30:30] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:15:18 [I 2022-10-09 00:30:30] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00309 [I 2022-10-09 00:30:30] ...trap/engines/engine.py.230: loss: 0.15646 [I 2022-10-09 00:30:30] ...optimizers/trijoint.py.151: total_norm: 17.703375 [I 2022-10-09 00:30:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 1220/2304 [I 2022-10-09 00:30:38] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:13 [I 2022-10-09 00:30:38] ...trap/engines/engine.py.229: process: 0.84804 | load: 0.00311 [I 2022-10-09 00:30:38] ...trap/engines/engine.py.230: loss: 0.17089 [I 2022-10-09 00:30:38] ...optimizers/trijoint.py.151: total_norm: 14.151887 [I 2022-10-09 00:30:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 1230/2304 [I 2022-10-09 00:30:47] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:01 [I 2022-10-09 00:30:47] ...trap/engines/engine.py.229: process: 0.83063 | load: 0.00300 [I 2022-10-09 00:30:47] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-09 00:30:47] ...optimizers/trijoint.py.151: total_norm: 17.180269 [I 2022-10-09 00:30:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 1240/2304 [I 2022-10-09 00:30:55] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:14:57 [I 2022-10-09 00:30:55] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00300 [I 2022-10-09 00:30:55] ...trap/engines/engine.py.230: loss: 0.18160 [I 2022-10-09 00:30:55] ...optimizers/trijoint.py.151: total_norm: 14.434398 [I 2022-10-09 00:31:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 1250/2304 [I 2022-10-09 00:31:04] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:14:44 [I 2022-10-09 00:31:04] ...trap/engines/engine.py.229: process: 0.82527 | load: 0.00297 [I 2022-10-09 00:31:04] ...trap/engines/engine.py.230: loss: 0.22389 [I 2022-10-09 00:31:04] ...optimizers/trijoint.py.151: total_norm: 21.145050 [I 2022-10-09 00:31:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 1260/2304 [I 2022-10-09 00:31:12] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:14:39 [I 2022-10-09 00:31:12] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00302 [I 2022-10-09 00:31:12] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-09 00:31:12] ...optimizers/trijoint.py.151: total_norm: 12.934758 [I 2022-10-09 00:31:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 1270/2304 [I 2022-10-09 00:31:20] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:14:25 [I 2022-10-09 00:31:20] ...trap/engines/engine.py.229: process: 0.82531 | load: 0.00299 [I 2022-10-09 00:31:20] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-09 00:31:20] ...optimizers/trijoint.py.151: total_norm: 16.762247 [I 2022-10-09 00:31:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 1280/2304 [I 2022-10-09 00:31:29] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:14:21 [I 2022-10-09 00:31:29] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00298 [I 2022-10-09 00:31:29] ...trap/engines/engine.py.230: loss: 0.27679 [I 2022-10-09 00:31:29] ...optimizers/trijoint.py.151: total_norm: 18.245914 [I 2022-10-09 00:31:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 1290/2304 [I 2022-10-09 00:31:37] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:10 [I 2022-10-09 00:31:37] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00301 [I 2022-10-09 00:31:37] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-09 00:31:37] ...optimizers/trijoint.py.151: total_norm: 23.990993 [I 2022-10-09 00:31:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 1300/2304 [I 2022-10-09 00:31:46] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:13:59 [I 2022-10-09 00:31:46] ...trap/engines/engine.py.229: process: 0.83133 | load: 0.00298 [I 2022-10-09 00:31:46] ...trap/engines/engine.py.230: loss: 0.23277 [I 2022-10-09 00:31:46] ...optimizers/trijoint.py.151: total_norm: 27.405931 [I 2022-10-09 00:31:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 1310/2304 [I 2022-10-09 00:31:54] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:13:53 [I 2022-10-09 00:31:54] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00301 [I 2022-10-09 00:31:54] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-09 00:31:54] ...optimizers/trijoint.py.151: total_norm: 14.869415 [I 2022-10-09 00:32:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 1320/2304 [I 2022-10-09 00:32:02] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:13:48 [I 2022-10-09 00:32:02] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00307 [I 2022-10-09 00:32:02] ...trap/engines/engine.py.230: loss: 0.17326 [I 2022-10-09 00:32:02] ...optimizers/trijoint.py.151: total_norm: 40.890312 [I 2022-10-09 00:32:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 1330/2304 [I 2022-10-09 00:32:11] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:13:43 [I 2022-10-09 00:32:11] ...trap/engines/engine.py.229: process: 0.85206 | load: 0.00312 [I 2022-10-09 00:32:11] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-09 00:32:11] ...optimizers/trijoint.py.151: total_norm: 14.446129 [I 2022-10-09 00:32:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 1340/2304 [I 2022-10-09 00:32:19] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:13:29 [I 2022-10-09 00:32:19] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00305 [I 2022-10-09 00:32:19] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-09 00:32:19] ...optimizers/trijoint.py.151: total_norm: 14.239998 [I 2022-10-09 00:32:28] ...trap/engines/engine.py.224: train: epoch 25 | batch 1350/2304 [I 2022-10-09 00:32:28] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:27 [I 2022-10-09 00:32:28] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00308 [I 2022-10-09 00:32:28] ...trap/engines/engine.py.230: loss: 0.12049 [I 2022-10-09 00:32:28] ...optimizers/trijoint.py.151: total_norm: 208.920471 [I 2022-10-09 00:32:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 1360/2304 [I 2022-10-09 00:32:36] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:13 [I 2022-10-09 00:32:36] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00300 [I 2022-10-09 00:32:36] ...trap/engines/engine.py.230: loss: 0.20622 [I 2022-10-09 00:32:36] ...optimizers/trijoint.py.151: total_norm: 13.515679 [I 2022-10-09 00:32:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 1370/2304 [I 2022-10-09 00:32:45] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:01 [I 2022-10-09 00:32:45] ...trap/engines/engine.py.229: process: 0.83284 | load: 0.00301 [I 2022-10-09 00:32:45] ...trap/engines/engine.py.230: loss: 0.19776 [I 2022-10-09 00:32:45] ...optimizers/trijoint.py.151: total_norm: 26.347191 [I 2022-10-09 00:32:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 1380/2304 [I 2022-10-09 00:32:53] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:12:58 [I 2022-10-09 00:32:53] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00305 [I 2022-10-09 00:32:53] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-09 00:32:53] ...optimizers/trijoint.py.151: total_norm: 30.869947 [I 2022-10-09 00:33:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 1390/2304 [I 2022-10-09 00:33:01] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:12:44 [I 2022-10-09 00:33:01] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00300 [I 2022-10-09 00:33:01] ...trap/engines/engine.py.230: loss: 0.09639 [I 2022-10-09 00:33:01] ...optimizers/trijoint.py.151: total_norm: 9.051708 [I 2022-10-09 00:33:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 1400/2304 [I 2022-10-09 00:33:10] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:12:37 [I 2022-10-09 00:33:10] ...trap/engines/engine.py.229: process: 0.82952 | load: 0.00305 [I 2022-10-09 00:33:10] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-09 00:33:10] ...optimizers/trijoint.py.151: total_norm: 18.904846 [I 2022-10-09 00:33:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 1410/2304 [I 2022-10-09 00:33:18] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:12:29 [I 2022-10-09 00:33:18] ...trap/engines/engine.py.229: process: 0.83291 | load: 0.00298 [I 2022-10-09 00:33:18] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-09 00:33:18] ...optimizers/trijoint.py.151: total_norm: 9.313259 [I 2022-10-09 00:33:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 1420/2304 [I 2022-10-09 00:33:27] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:12:27 [I 2022-10-09 00:33:27] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00310 [I 2022-10-09 00:33:27] ...trap/engines/engine.py.230: loss: 0.12580 [I 2022-10-09 00:33:27] ...optimizers/trijoint.py.151: total_norm: 11.271399 [I 2022-10-09 00:33:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 1430/2304 [I 2022-10-09 00:33:35] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:19 [I 2022-10-09 00:33:35] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00308 [I 2022-10-09 00:33:35] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-09 00:33:35] ...optimizers/trijoint.py.151: total_norm: 18.129896 [I 2022-10-09 00:33:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 1440/2304 [I 2022-10-09 00:33:44] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:12 [I 2022-10-09 00:33:44] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00313 [I 2022-10-09 00:33:44] ...trap/engines/engine.py.230: loss: 0.16131 [I 2022-10-09 00:33:44] ...optimizers/trijoint.py.151: total_norm: 14.528030 [I 2022-10-09 00:33:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 1450/2304 [I 2022-10-09 00:33:52] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:07 [I 2022-10-09 00:33:52] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00306 [I 2022-10-09 00:33:52] ...trap/engines/engine.py.230: loss: 0.17928 [I 2022-10-09 00:33:52] ...optimizers/trijoint.py.151: total_norm: 21.696547 [I 2022-10-09 00:34:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 1460/2304 [I 2022-10-09 00:34:01] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:11:57 [I 2022-10-09 00:34:01] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00309 [I 2022-10-09 00:34:01] ...trap/engines/engine.py.230: loss: 0.15064 [I 2022-10-09 00:34:01] ...optimizers/trijoint.py.151: total_norm: 18.827412 [I 2022-10-09 00:34:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 1470/2304 [I 2022-10-09 00:34:09] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:11:47 [I 2022-10-09 00:34:09] ...trap/engines/engine.py.229: process: 0.83603 | load: 0.00310 [I 2022-10-09 00:34:09] ...trap/engines/engine.py.230: loss: 0.26327 [I 2022-10-09 00:34:09] ...optimizers/trijoint.py.151: total_norm: 9.614713 [I 2022-10-09 00:34:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 1480/2304 [I 2022-10-09 00:34:18] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:11:42 [I 2022-10-09 00:34:18] ...trap/engines/engine.py.229: process: 0.85265 | load: 0.00314 [I 2022-10-09 00:34:18] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-09 00:34:18] ...optimizers/trijoint.py.151: total_norm: 23.673325 [I 2022-10-09 00:34:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 1490/2304 [I 2022-10-09 00:34:26] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:11:22 [I 2022-10-09 00:34:26] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00301 [I 2022-10-09 00:34:26] ...trap/engines/engine.py.230: loss: 0.20294 [I 2022-10-09 00:34:26] ...optimizers/trijoint.py.151: total_norm: 16.516159 [I 2022-10-09 00:34:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 1500/2304 [I 2022-10-09 00:34:34] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:12 [I 2022-10-09 00:34:34] ...trap/engines/engine.py.229: process: 0.82797 | load: 0.00301 [I 2022-10-09 00:34:34] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-09 00:34:34] ...optimizers/trijoint.py.151: total_norm: 15.089849 [I 2022-10-09 00:34:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 1510/2304 [I 2022-10-09 00:34:43] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:03 [I 2022-10-09 00:34:43] ...trap/engines/engine.py.229: process: 0.82403 | load: 0.00302 [I 2022-10-09 00:34:43] ...trap/engines/engine.py.230: loss: 0.21249 [I 2022-10-09 00:34:43] ...optimizers/trijoint.py.151: total_norm: 25.853054 [I 2022-10-09 00:34:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 1520/2304 [I 2022-10-09 00:34:51] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:10:55 [I 2022-10-09 00:34:51] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00307 [I 2022-10-09 00:34:51] ...trap/engines/engine.py.230: loss: 0.15775 [I 2022-10-09 00:34:51] ...optimizers/trijoint.py.151: total_norm: 19.798594 [I 2022-10-09 00:35:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 1530/2304 [I 2022-10-09 00:35:00] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:10:50 [I 2022-10-09 00:35:00] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00304 [I 2022-10-09 00:35:00] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-09 00:35:00] ...optimizers/trijoint.py.151: total_norm: 11.177708 [I 2022-10-09 00:35:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 1540/2304 [I 2022-10-09 00:35:08] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:10:41 [I 2022-10-09 00:35:08] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00302 [I 2022-10-09 00:35:08] ...trap/engines/engine.py.230: loss: 0.21867 [I 2022-10-09 00:35:08] ...optimizers/trijoint.py.151: total_norm: 15.451522 [I 2022-10-09 00:35:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 1550/2304 [I 2022-10-09 00:35:16] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:10:33 [I 2022-10-09 00:35:16] ...trap/engines/engine.py.229: process: 0.83702 | load: 0.00299 [I 2022-10-09 00:35:16] ...trap/engines/engine.py.230: loss: 0.27252 [I 2022-10-09 00:35:16] ...optimizers/trijoint.py.151: total_norm: 14.759565 [I 2022-10-09 00:35:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 1560/2304 [I 2022-10-09 00:35:25] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:24 [I 2022-10-09 00:35:25] ...trap/engines/engine.py.229: process: 0.83695 | load: 0.00301 [I 2022-10-09 00:35:25] ...trap/engines/engine.py.230: loss: 0.14552 [I 2022-10-09 00:35:25] ...optimizers/trijoint.py.151: total_norm: 37.796654 [I 2022-10-09 00:35:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 1570/2304 [I 2022-10-09 00:35:33] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:15 [I 2022-10-09 00:35:33] ...trap/engines/engine.py.229: process: 0.82986 | load: 0.00301 [I 2022-10-09 00:35:33] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-09 00:35:33] ...optimizers/trijoint.py.151: total_norm: 20.740801 [I 2022-10-09 00:35:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 1580/2304 [I 2022-10-09 00:35:42] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:11 [I 2022-10-09 00:35:42] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00308 [I 2022-10-09 00:35:42] ...trap/engines/engine.py.230: loss: 0.15685 [I 2022-10-09 00:35:42] ...optimizers/trijoint.py.151: total_norm: 18.790649 [I 2022-10-09 00:35:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 1590/2304 [I 2022-10-09 00:35:50] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:00 [I 2022-10-09 00:35:50] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00300 [I 2022-10-09 00:35:50] ...trap/engines/engine.py.230: loss: 0.18173 [I 2022-10-09 00:35:50] ...optimizers/trijoint.py.151: total_norm: 28.748766 [I 2022-10-09 00:35:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 1600/2304 [I 2022-10-09 00:35:59] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:09:53 [I 2022-10-09 00:35:59] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00302 [I 2022-10-09 00:35:59] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-09 00:35:59] ...optimizers/trijoint.py.151: total_norm: 19.361168 [I 2022-10-09 00:36:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 1610/2304 [I 2022-10-09 00:36:07] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:09:42 [I 2022-10-09 00:36:07] ...trap/engines/engine.py.229: process: 0.83256 | load: 0.00302 [I 2022-10-09 00:36:07] ...trap/engines/engine.py.230: loss: 0.14575 [I 2022-10-09 00:36:07] ...optimizers/trijoint.py.151: total_norm: 18.792715 [I 2022-10-09 00:36:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 1620/2304 [I 2022-10-09 00:36:15] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:34 [I 2022-10-09 00:36:15] ...trap/engines/engine.py.229: process: 0.83822 | load: 0.00301 [I 2022-10-09 00:36:15] ...trap/engines/engine.py.230: loss: 0.18187 [I 2022-10-09 00:36:15] ...optimizers/trijoint.py.151: total_norm: 30.557699 [I 2022-10-09 00:36:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 1630/2304 [I 2022-10-09 00:36:24] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:29 [I 2022-10-09 00:36:24] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00302 [I 2022-10-09 00:36:24] ...trap/engines/engine.py.230: loss: 0.13921 [I 2022-10-09 00:36:24] ...optimizers/trijoint.py.151: total_norm: 15.275860 [I 2022-10-09 00:36:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 1640/2304 [I 2022-10-09 00:36:32] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:16 [I 2022-10-09 00:36:32] ...trap/engines/engine.py.229: process: 0.83967 | load: 0.00308 [I 2022-10-09 00:36:32] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-09 00:36:32] ...optimizers/trijoint.py.151: total_norm: 15.242023 [I 2022-10-09 00:36:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 1650/2304 [I 2022-10-09 00:36:41] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:13 [I 2022-10-09 00:36:41] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00303 [I 2022-10-09 00:36:41] ...trap/engines/engine.py.230: loss: 0.23086 [I 2022-10-09 00:36:41] ...optimizers/trijoint.py.151: total_norm: 14.517336 [I 2022-10-09 00:36:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 1660/2304 [I 2022-10-09 00:36:49] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:04 [I 2022-10-09 00:36:49] ...trap/engines/engine.py.229: process: 0.84747 | load: 0.00301 [I 2022-10-09 00:36:49] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-09 00:36:49] ...optimizers/trijoint.py.151: total_norm: 17.274433 [I 2022-10-09 00:37:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 1670/2304 [I 2022-10-09 00:37:16] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:46:56 [I 2022-10-09 00:37:16] ...trap/engines/engine.py.229: process: 18.85054 | load: 0.00300 [I 2022-10-09 00:37:16] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-09 00:37:16] ...optimizers/trijoint.py.151: total_norm: 14.118328 [I 2022-10-09 00:37:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 1680/2304 [I 2022-10-09 00:37:24] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:12:42 [I 2022-10-09 00:37:24] ...trap/engines/engine.py.229: process: 0.82976 | load: 0.00303 [I 2022-10-09 00:37:24] ...trap/engines/engine.py.230: loss: 0.15135 [I 2022-10-09 00:37:24] ...optimizers/trijoint.py.151: total_norm: 11.861249 [I 2022-10-09 00:37:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 1690/2304 [I 2022-10-09 00:37:32] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:59 [I 2022-10-09 00:37:32] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00299 [I 2022-10-09 00:37:32] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-09 00:37:32] ...optimizers/trijoint.py.151: total_norm: 12.995674 [I 2022-10-09 00:37:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 1700/2304 [I 2022-10-09 00:37:41] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:26 [I 2022-10-09 00:37:41] ...trap/engines/engine.py.229: process: 0.82635 | load: 0.00300 [I 2022-10-09 00:37:41] ...trap/engines/engine.py.230: loss: 0.16102 [I 2022-10-09 00:37:41] ...optimizers/trijoint.py.151: total_norm: 13.312783 [I 2022-10-09 00:37:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 1710/2304 [I 2022-10-09 00:37:49] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:16 [I 2022-10-09 00:37:49] ...trap/engines/engine.py.229: process: 0.84278 | load: 0.00302 [I 2022-10-09 00:37:49] ...trap/engines/engine.py.230: loss: 0.18315 [I 2022-10-09 00:37:49] ...optimizers/trijoint.py.151: total_norm: 26.662037 [I 2022-10-09 00:37:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 1720/2304 [I 2022-10-09 00:37:58] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:06 [I 2022-10-09 00:37:58] ...trap/engines/engine.py.229: process: 0.82534 | load: 0.00302 [I 2022-10-09 00:37:58] ...trap/engines/engine.py.230: loss: 0.17788 [I 2022-10-09 00:37:58] ...optimizers/trijoint.py.151: total_norm: 20.231554 [I 2022-10-09 00:38:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 1730/2304 [I 2022-10-09 00:38:06] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:00 [I 2022-10-09 00:38:06] ...trap/engines/engine.py.229: process: 0.82909 | load: 0.00299 [I 2022-10-09 00:38:06] ...trap/engines/engine.py.230: loss: 0.26443 [I 2022-10-09 00:38:06] ...optimizers/trijoint.py.151: total_norm: 20.628872 [I 2022-10-09 00:38:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 1740/2304 [I 2022-10-09 00:38:14] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:52 [I 2022-10-09 00:38:14] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00302 [I 2022-10-09 00:38:14] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-09 00:38:14] ...optimizers/trijoint.py.151: total_norm: 10.958407 [I 2022-10-09 00:38:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 1750/2304 [I 2022-10-09 00:38:23] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:44 [I 2022-10-09 00:38:23] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00305 [I 2022-10-09 00:38:23] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-09 00:38:23] ...optimizers/trijoint.py.151: total_norm: 10.161547 [I 2022-10-09 00:38:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 1760/2304 [I 2022-10-09 00:38:31] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:34 [I 2022-10-09 00:38:31] ...trap/engines/engine.py.229: process: 0.82390 | load: 0.00300 [I 2022-10-09 00:38:31] ...trap/engines/engine.py.230: loss: 0.13654 [I 2022-10-09 00:38:31] ...optimizers/trijoint.py.151: total_norm: 21.626148 [I 2022-10-09 00:38:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 1770/2304 [I 2022-10-09 00:38:39] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:29 [I 2022-10-09 00:38:39] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00304 [I 2022-10-09 00:38:39] ...trap/engines/engine.py.230: loss: 0.19785 [I 2022-10-09 00:38:39] ...optimizers/trijoint.py.151: total_norm: 26.105125 [I 2022-10-09 00:38:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 1780/2304 [I 2022-10-09 00:38:48] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:16 [I 2022-10-09 00:38:48] ...trap/engines/engine.py.229: process: 0.82334 | load: 0.00302 [I 2022-10-09 00:38:48] ...trap/engines/engine.py.230: loss: 0.15466 [I 2022-10-09 00:38:48] ...optimizers/trijoint.py.151: total_norm: 18.087513 [I 2022-10-09 00:38:56] ...trap/engines/engine.py.224: train: epoch 25 | batch 1790/2304 [I 2022-10-09 00:38:56] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:12 [I 2022-10-09 00:38:56] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00300 [I 2022-10-09 00:38:56] ...trap/engines/engine.py.230: loss: 0.12061 [I 2022-10-09 00:38:56] ...optimizers/trijoint.py.151: total_norm: 21.759972 [I 2022-10-09 00:39:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 1800/2304 [I 2022-10-09 00:39:05] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:01 [I 2022-10-09 00:39:05] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00301 [I 2022-10-09 00:39:05] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-09 00:39:05] ...optimizers/trijoint.py.151: total_norm: 20.555065 [I 2022-10-09 00:39:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 1810/2304 [I 2022-10-09 00:39:13] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:54 [I 2022-10-09 00:39:13] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00299 [I 2022-10-09 00:39:13] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-09 00:39:13] ...optimizers/trijoint.py.151: total_norm: 40.256218 [I 2022-10-09 00:39:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 1820/2304 [I 2022-10-09 00:39:21] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:47 [I 2022-10-09 00:39:21] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00305 [I 2022-10-09 00:39:21] ...trap/engines/engine.py.230: loss: 0.20033 [I 2022-10-09 00:39:21] ...optimizers/trijoint.py.151: total_norm: 12.378195 [I 2022-10-09 00:39:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 1830/2304 [I 2022-10-09 00:39:30] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:36 [I 2022-10-09 00:39:30] ...trap/engines/engine.py.229: process: 0.81999 | load: 0.00299 [I 2022-10-09 00:39:30] ...trap/engines/engine.py.230: loss: 0.13481 [I 2022-10-09 00:39:30] ...optimizers/trijoint.py.151: total_norm: 12.035586 [I 2022-10-09 00:39:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 1840/2304 [I 2022-10-09 00:39:38] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:28 [I 2022-10-09 00:39:38] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00302 [I 2022-10-09 00:39:38] ...trap/engines/engine.py.230: loss: 0.10495 [I 2022-10-09 00:39:38] ...optimizers/trijoint.py.151: total_norm: 18.255152 [I 2022-10-09 00:39:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 1850/2304 [I 2022-10-09 00:39:47] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:19 [I 2022-10-09 00:39:47] ...trap/engines/engine.py.229: process: 0.83336 | load: 0.00300 [I 2022-10-09 00:39:47] ...trap/engines/engine.py.230: loss: 0.23838 [I 2022-10-09 00:39:47] ...optimizers/trijoint.py.151: total_norm: 30.461964 [I 2022-10-09 00:39:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 1860/2304 [I 2022-10-09 00:39:55] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:13 [I 2022-10-09 00:39:55] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00300 [I 2022-10-09 00:39:55] ...trap/engines/engine.py.230: loss: 0.25293 [I 2022-10-09 00:39:55] ...optimizers/trijoint.py.151: total_norm: 27.418859 [I 2022-10-09 00:40:03] ...trap/engines/engine.py.224: train: epoch 25 | batch 1870/2304 [I 2022-10-09 00:40:03] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:03 [I 2022-10-09 00:40:03] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00302 [I 2022-10-09 00:40:03] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-09 00:40:03] ...optimizers/trijoint.py.151: total_norm: 12.669169 [I 2022-10-09 00:40:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 1880/2304 [I 2022-10-09 00:40:12] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:57 [I 2022-10-09 00:40:12] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00300 [I 2022-10-09 00:40:12] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-09 00:40:12] ...optimizers/trijoint.py.151: total_norm: 16.805618 [I 2022-10-09 00:40:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 1890/2304 [I 2022-10-09 00:40:20] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:45 [I 2022-10-09 00:40:20] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00302 [I 2022-10-09 00:40:20] ...trap/engines/engine.py.230: loss: 0.18385 [I 2022-10-09 00:40:20] ...optimizers/trijoint.py.151: total_norm: 27.202911 [I 2022-10-09 00:40:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 1900/2304 [I 2022-10-09 00:40:29] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:39 [I 2022-10-09 00:40:29] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00301 [I 2022-10-09 00:40:29] ...trap/engines/engine.py.230: loss: 0.19885 [I 2022-10-09 00:40:29] ...optimizers/trijoint.py.151: total_norm: 22.748037 [I 2022-10-09 00:40:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 1910/2304 [I 2022-10-09 00:40:37] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:29 [I 2022-10-09 00:40:37] ...trap/engines/engine.py.229: process: 0.83610 | load: 0.00301 [I 2022-10-09 00:40:37] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-09 00:40:37] ...optimizers/trijoint.py.151: total_norm: 10.210380 [I 2022-10-09 00:40:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 1920/2304 [I 2022-10-09 00:40:45] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:21 [I 2022-10-09 00:40:45] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00301 [I 2022-10-09 00:40:45] ...trap/engines/engine.py.230: loss: 0.20133 [I 2022-10-09 00:40:45] ...optimizers/trijoint.py.151: total_norm: 20.015257 [I 2022-10-09 00:40:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 1930/2304 [I 2022-10-09 00:40:54] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:14 [I 2022-10-09 00:40:54] ...trap/engines/engine.py.229: process: 0.83955 | load: 0.00299 [I 2022-10-09 00:40:54] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-09 00:40:54] ...optimizers/trijoint.py.151: total_norm: 19.790426 [I 2022-10-09 00:41:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 1940/2304 [I 2022-10-09 00:41:02] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:08 [I 2022-10-09 00:41:02] ...trap/engines/engine.py.229: process: 0.86481 | load: 0.00298 [I 2022-10-09 00:41:02] ...trap/engines/engine.py.230: loss: 0.11525 [I 2022-10-09 00:41:02] ...optimizers/trijoint.py.151: total_norm: 40.114468 [I 2022-10-09 00:41:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 1950/2304 [I 2022-10-09 00:41:11] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:57 [I 2022-10-09 00:41:11] ...trap/engines/engine.py.229: process: 0.83863 | load: 0.00301 [I 2022-10-09 00:41:11] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-09 00:41:11] ...optimizers/trijoint.py.151: total_norm: 20.286678 [I 2022-10-09 00:41:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 1960/2304 [I 2022-10-09 00:41:19] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:49 [I 2022-10-09 00:41:19] ...trap/engines/engine.py.229: process: 0.85537 | load: 0.00301 [I 2022-10-09 00:41:19] ...trap/engines/engine.py.230: loss: 0.30737 [I 2022-10-09 00:41:19] ...optimizers/trijoint.py.151: total_norm: 23.541361 [I 2022-10-09 00:41:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 1970/2304 [I 2022-10-09 00:41:27] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:40 [I 2022-10-09 00:41:27] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00300 [I 2022-10-09 00:41:27] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-09 00:41:27] ...optimizers/trijoint.py.151: total_norm: 9.228601 [I 2022-10-09 00:41:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 1980/2304 [I 2022-10-09 00:41:36] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:34 [I 2022-10-09 00:41:36] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00303 [I 2022-10-09 00:41:36] ...trap/engines/engine.py.230: loss: 0.13616 [I 2022-10-09 00:41:36] ...optimizers/trijoint.py.151: total_norm: 14.344519 [I 2022-10-09 00:41:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 1990/2304 [I 2022-10-09 00:41:44] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:23 [I 2022-10-09 00:41:44] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00303 [I 2022-10-09 00:41:44] ...trap/engines/engine.py.230: loss: 0.12738 [I 2022-10-09 00:41:44] ...optimizers/trijoint.py.151: total_norm: 23.336475 [I 2022-10-09 00:41:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 2000/2304 [I 2022-10-09 00:41:53] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:16 [I 2022-10-09 00:41:53] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00302 [I 2022-10-09 00:41:53] ...trap/engines/engine.py.230: loss: 0.17651 [I 2022-10-09 00:41:53] ...optimizers/trijoint.py.151: total_norm: 16.086069 [I 2022-10-09 00:42:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 2010/2304 [I 2022-10-09 00:42:01] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:07 [I 2022-10-09 00:42:01] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00304 [I 2022-10-09 00:42:01] ...trap/engines/engine.py.230: loss: 0.10182 [I 2022-10-09 00:42:01] ...optimizers/trijoint.py.151: total_norm: 13.893457 [I 2022-10-09 00:42:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 2020/2304 [I 2022-10-09 00:42:10] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:01 [I 2022-10-09 00:42:10] ...trap/engines/engine.py.229: process: 0.86535 | load: 0.00310 [I 2022-10-09 00:42:10] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-09 00:42:10] ...optimizers/trijoint.py.151: total_norm: 22.551725 [I 2022-10-09 00:42:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 2030/2304 [I 2022-10-09 00:42:18] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:53 [I 2022-10-09 00:42:18] ...trap/engines/engine.py.229: process: 0.86366 | load: 0.00308 [I 2022-10-09 00:42:18] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-09 00:42:18] ...optimizers/trijoint.py.151: total_norm: 8.315849 [I 2022-10-09 00:42:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 2040/2304 [I 2022-10-09 00:42:27] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:43 [I 2022-10-09 00:42:27] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00309 [I 2022-10-09 00:42:27] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-09 00:42:27] ...optimizers/trijoint.py.151: total_norm: 16.722490 [I 2022-10-09 00:42:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 2050/2304 [I 2022-10-09 00:42:35] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:34 [I 2022-10-09 00:42:35] ...trap/engines/engine.py.229: process: 0.86044 | load: 0.00308 [I 2022-10-09 00:42:35] ...trap/engines/engine.py.230: loss: 0.21825 [I 2022-10-09 00:42:35] ...optimizers/trijoint.py.151: total_norm: 21.834038 [I 2022-10-09 00:42:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 2060/2304 [I 2022-10-09 00:42:44] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:24 [I 2022-10-09 00:42:44] ...trap/engines/engine.py.229: process: 0.82901 | load: 0.00309 [I 2022-10-09 00:42:44] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-09 00:42:44] ...optimizers/trijoint.py.151: total_norm: 11.325081 [I 2022-10-09 00:42:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 2070/2304 [I 2022-10-09 00:42:52] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:18 [I 2022-10-09 00:42:52] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00311 [I 2022-10-09 00:42:52] ...trap/engines/engine.py.230: loss: 0.10574 [I 2022-10-09 00:42:52] ...optimizers/trijoint.py.151: total_norm: 16.601585 [I 2022-10-09 00:43:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 2080/2304 [I 2022-10-09 00:43:01] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:08 [I 2022-10-09 00:43:01] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00313 [I 2022-10-09 00:43:01] ...trap/engines/engine.py.230: loss: 0.13795 [I 2022-10-09 00:43:01] ...optimizers/trijoint.py.151: total_norm: 11.881561 [I 2022-10-09 00:43:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 2090/2304 [I 2022-10-09 00:43:09] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:59 [I 2022-10-09 00:43:09] ...trap/engines/engine.py.229: process: 0.84705 | load: 0.00310 [I 2022-10-09 00:43:09] ...trap/engines/engine.py.230: loss: 0.14766 [I 2022-10-09 00:43:09] ...optimizers/trijoint.py.151: total_norm: 12.581234 [I 2022-10-09 00:43:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 2100/2304 [I 2022-10-09 00:43:17] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:02:51 [I 2022-10-09 00:43:17] ...trap/engines/engine.py.229: process: 0.83651 | load: 0.00301 [I 2022-10-09 00:43:17] ...trap/engines/engine.py.230: loss: 0.19128 [I 2022-10-09 00:43:17] ...optimizers/trijoint.py.151: total_norm: 18.902943 [I 2022-10-09 00:43:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 2110/2304 [I 2022-10-09 00:43:26] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:43 [I 2022-10-09 00:43:26] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00301 [I 2022-10-09 00:43:26] ...trap/engines/engine.py.230: loss: 0.13939 [I 2022-10-09 00:43:26] ...optimizers/trijoint.py.151: total_norm: 19.359665 [I 2022-10-09 00:43:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 2120/2304 [I 2022-10-09 00:43:34] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:35 [I 2022-10-09 00:43:34] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00302 [I 2022-10-09 00:43:34] ...trap/engines/engine.py.230: loss: 0.18324 [I 2022-10-09 00:43:34] ...optimizers/trijoint.py.151: total_norm: 26.509970 [I 2022-10-09 00:43:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 2130/2304 [I 2022-10-09 00:43:43] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:27 [I 2022-10-09 00:43:43] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00300 [I 2022-10-09 00:43:43] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-09 00:43:43] ...optimizers/trijoint.py.151: total_norm: 17.800282 [I 2022-10-09 00:43:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 2140/2304 [I 2022-10-09 00:43:51] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:17 [I 2022-10-09 00:43:51] ...trap/engines/engine.py.229: process: 0.83269 | load: 0.00300 [I 2022-10-09 00:43:51] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-09 00:43:51] ...optimizers/trijoint.py.151: total_norm: 13.527169 [I 2022-10-09 00:44:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 2150/2304 [I 2022-10-09 00:44:00] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:09 [I 2022-10-09 00:44:00] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00301 [I 2022-10-09 00:44:00] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-09 00:44:00] ...optimizers/trijoint.py.151: total_norm: 10.497315 [I 2022-10-09 00:44:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 2160/2304 [I 2022-10-09 00:44:08] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:00 [I 2022-10-09 00:44:08] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00300 [I 2022-10-09 00:44:08] ...trap/engines/engine.py.230: loss: 0.19916 [I 2022-10-09 00:44:08] ...optimizers/trijoint.py.151: total_norm: 18.096052 [I 2022-10-09 00:44:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 2170/2304 [I 2022-10-09 00:44:16] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:01:52 [I 2022-10-09 00:44:16] ...trap/engines/engine.py.229: process: 0.83399 | load: 0.00302 [I 2022-10-09 00:44:16] ...trap/engines/engine.py.230: loss: 0.15789 [I 2022-10-09 00:44:16] ...optimizers/trijoint.py.151: total_norm: 12.202953 [I 2022-10-09 00:44:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 2180/2304 [I 2022-10-09 00:44:25] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:43 [I 2022-10-09 00:44:25] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00302 [I 2022-10-09 00:44:25] ...trap/engines/engine.py.230: loss: 0.15300 [I 2022-10-09 00:44:25] ...optimizers/trijoint.py.151: total_norm: 25.629681 [I 2022-10-09 00:44:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 2190/2304 [I 2022-10-09 00:44:33] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:35 [I 2022-10-09 00:44:33] ...trap/engines/engine.py.229: process: 0.82293 | load: 0.00309 [I 2022-10-09 00:44:33] ...trap/engines/engine.py.230: loss: 0.15541 [I 2022-10-09 00:44:33] ...optimizers/trijoint.py.151: total_norm: 20.478228 [I 2022-10-09 00:44:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 2200/2304 [I 2022-10-09 00:44:42] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:27 [I 2022-10-09 00:44:42] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00301 [I 2022-10-09 00:44:42] ...trap/engines/engine.py.230: loss: 0.15122 [I 2022-10-09 00:44:42] ...optimizers/trijoint.py.151: total_norm: 18.199986 [I 2022-10-09 00:44:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 2210/2304 [I 2022-10-09 00:44:50] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:18 [I 2022-10-09 00:44:50] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00304 [I 2022-10-09 00:44:50] ...trap/engines/engine.py.230: loss: 0.29143 [I 2022-10-09 00:44:50] ...optimizers/trijoint.py.151: total_norm: 21.598795 [I 2022-10-09 00:44:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 2220/2304 [I 2022-10-09 00:44:58] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:10 [I 2022-10-09 00:44:58] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00302 [I 2022-10-09 00:44:58] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-09 00:44:58] ...optimizers/trijoint.py.151: total_norm: 12.794584 [I 2022-10-09 00:45:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 2230/2304 [I 2022-10-09 00:45:07] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:01 [I 2022-10-09 00:45:07] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00302 [I 2022-10-09 00:45:07] ...trap/engines/engine.py.230: loss: 0.13540 [I 2022-10-09 00:45:07] ...optimizers/trijoint.py.151: total_norm: 14.764497 [I 2022-10-09 00:45:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 2240/2304 [I 2022-10-09 00:45:15] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:53 [I 2022-10-09 00:45:15] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00302 [I 2022-10-09 00:45:15] ...trap/engines/engine.py.230: loss: 0.19319 [I 2022-10-09 00:45:15] ...optimizers/trijoint.py.151: total_norm: 20.297258 [I 2022-10-09 00:45:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 2250/2304 [I 2022-10-09 00:45:24] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:45 [I 2022-10-09 00:45:24] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00301 [I 2022-10-09 00:45:24] ...trap/engines/engine.py.230: loss: 0.22889 [I 2022-10-09 00:45:24] ...optimizers/trijoint.py.151: total_norm: 31.552778 [I 2022-10-09 00:45:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 2260/2304 [I 2022-10-09 00:45:32] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:36 [I 2022-10-09 00:45:32] ...trap/engines/engine.py.229: process: 0.83144 | load: 0.00298 [I 2022-10-09 00:45:32] ...trap/engines/engine.py.230: loss: 0.25672 [I 2022-10-09 00:45:32] ...optimizers/trijoint.py.151: total_norm: 33.124130 [I 2022-10-09 00:45:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 2270/2304 [I 2022-10-09 00:45:41] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:28 [I 2022-10-09 00:45:41] ...trap/engines/engine.py.229: process: 0.84059 | load: 0.00299 [I 2022-10-09 00:45:41] ...trap/engines/engine.py.230: loss: 0.18485 [I 2022-10-09 00:45:41] ...optimizers/trijoint.py.151: total_norm: 29.894995 [I 2022-10-09 00:45:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 2280/2304 [I 2022-10-09 00:45:49] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:20 [I 2022-10-09 00:45:49] ...trap/engines/engine.py.229: process: 0.83443 | load: 0.00301 [I 2022-10-09 00:45:49] ...trap/engines/engine.py.230: loss: 0.18531 [I 2022-10-09 00:45:49] ...optimizers/trijoint.py.151: total_norm: 10.594109 [I 2022-10-09 00:45:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 2290/2304 [I 2022-10-09 00:45:57] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:11 [I 2022-10-09 00:45:57] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00299 [I 2022-10-09 00:45:57] ...trap/engines/engine.py.230: loss: 0.15111 [I 2022-10-09 00:45:57] ...optimizers/trijoint.py.151: total_norm: 19.187899 [I 2022-10-09 00:46:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 2300/2304 [I 2022-10-09 00:46:06] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:03 [I 2022-10-09 00:46:06] ...trap/engines/engine.py.229: process: 0.81937 | load: 0.00017 [I 2022-10-09 00:46:06] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-09 00:46:06] ...optimizers/trijoint.py.151: total_norm: 23.148285 [I 2022-10-09 00:46:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 2304/2304 [I 2022-10-09 00:46:09] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:00 [I 2022-10-09 00:46:09] ...trap/engines/engine.py.229: process: 0.81540 | load: 0.00016 [I 2022-10-09 00:46:09] ...trap/engines/engine.py.230: loss: 0.18203 [I 2022-10-09 00:46:09] ...optimizers/trijoint.py.151: total_norm: 27.666348 [S 2022-10-09 00:46:10] ...trap/engines/engine.py.129: train_epoch.epoch: 25 [S 2022-10-09 00:46:10] ...trap/engines/engine.py.129: train_epoch.loss: 0.17232878639679927 [I 2022-10-09 00:46:10] ...s/criterions/triplet.py.84: margin = 0.1800000000000001 [I 2022-10-09 00:46:11] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 25 [I 2022-10-09 00:46:11] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 00:46:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 00:46:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 00:46:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 00:46:13] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 00:46:14] ...trap/engines/engine.py.321: eval: epoch 25 | batch 0/510 [I 2022-10-09 00:46:14] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:02 [I 2022-10-09 00:46:14] ...trap/engines/engine.py.326: process: 3.18199 | load: 2.70070 [I 2022-10-09 00:46:19] ...trap/engines/engine.py.321: eval: epoch 25 | batch 10/510 [I 2022-10-09 00:46:19] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:38 [I 2022-10-09 00:46:19] ...trap/engines/engine.py.326: process: 0.53751 | load: 0.00019 [I 2022-10-09 00:46:24] ...trap/engines/engine.py.321: eval: epoch 25 | batch 20/510 [I 2022-10-09 00:46:24] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:39 [I 2022-10-09 00:46:24] ...trap/engines/engine.py.326: process: 0.61835 | load: 0.00018 [I 2022-10-09 00:46:29] ...trap/engines/engine.py.321: eval: epoch 25 | batch 30/510 [I 2022-10-09 00:46:29] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:01 [I 2022-10-09 00:46:29] ...trap/engines/engine.py.326: process: 0.53815 | load: 0.00019 [I 2022-10-09 00:46:34] ...trap/engines/engine.py.321: eval: epoch 25 | batch 40/510 [I 2022-10-09 00:46:34] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:03:49 [I 2022-10-09 00:46:34] ...trap/engines/engine.py.326: process: 0.44532 | load: 0.00019 [I 2022-10-09 00:46:39] ...trap/engines/engine.py.321: eval: epoch 25 | batch 50/510 [I 2022-10-09 00:46:39] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:42 [I 2022-10-09 00:46:39] ...trap/engines/engine.py.326: process: 0.42544 | load: 0.00018 [I 2022-10-09 00:46:44] ...trap/engines/engine.py.321: eval: epoch 25 | batch 60/510 [I 2022-10-09 00:46:44] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:45 [I 2022-10-09 00:46:44] ...trap/engines/engine.py.326: process: 0.53772 | load: 0.00018 [I 2022-10-09 00:46:49] ...trap/engines/engine.py.321: eval: epoch 25 | batch 70/510 [I 2022-10-09 00:46:49] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:40 [I 2022-10-09 00:46:49] ...trap/engines/engine.py.326: process: 0.46817 | load: 0.00020 [I 2022-10-09 00:46:54] ...trap/engines/engine.py.321: eval: epoch 25 | batch 80/510 [I 2022-10-09 00:46:54] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:35 [I 2022-10-09 00:46:54] ...trap/engines/engine.py.326: process: 0.46244 | load: 0.00019 [I 2022-10-09 00:46:59] ...trap/engines/engine.py.321: eval: epoch 25 | batch 90/510 [I 2022-10-09 00:46:59] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:10 [I 2022-10-09 00:46:59] ...trap/engines/engine.py.326: process: 0.46501 | load: 0.00020 [I 2022-10-09 00:47:03] ...trap/engines/engine.py.321: eval: epoch 25 | batch 100/510 [I 2022-10-09 00:47:03] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:54 [I 2022-10-09 00:47:03] ...trap/engines/engine.py.326: process: 0.30678 | load: 0.00018 [I 2022-10-09 00:47:06] ...trap/engines/engine.py.321: eval: epoch 25 | batch 110/510 [I 2022-10-09 00:47:06] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:11 [I 2022-10-09 00:47:06] ...trap/engines/engine.py.326: process: 0.32001 | load: 0.00020 [I 2022-10-09 00:47:10] ...trap/engines/engine.py.321: eval: epoch 25 | batch 120/510 [I 2022-10-09 00:47:10] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:06 [I 2022-10-09 00:47:10] ...trap/engines/engine.py.326: process: 0.33143 | load: 0.00020 [I 2022-10-09 00:47:13] ...trap/engines/engine.py.321: eval: epoch 25 | batch 130/510 [I 2022-10-09 00:47:13] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:01 [I 2022-10-09 00:47:13] ...trap/engines/engine.py.326: process: 0.32338 | load: 0.00023 [I 2022-10-09 00:47:16] ...trap/engines/engine.py.321: eval: epoch 25 | batch 140/510 [I 2022-10-09 00:47:16] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:59 [I 2022-10-09 00:47:16] ...trap/engines/engine.py.326: process: 0.32116 | load: 0.00019 [I 2022-10-09 00:47:19] ...trap/engines/engine.py.321: eval: epoch 25 | batch 150/510 [I 2022-10-09 00:47:19] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:54 [I 2022-10-09 00:47:19] ...trap/engines/engine.py.326: process: 0.31192 | load: 0.00019 [I 2022-10-09 00:47:22] ...trap/engines/engine.py.321: eval: epoch 25 | batch 160/510 [I 2022-10-09 00:47:22] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:51 [I 2022-10-09 00:47:22] ...trap/engines/engine.py.326: process: 0.31784 | load: 0.00019 [I 2022-10-09 00:47:26] ...trap/engines/engine.py.321: eval: epoch 25 | batch 170/510 [I 2022-10-09 00:47:26] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:48 [I 2022-10-09 00:47:26] ...trap/engines/engine.py.326: process: 0.31706 | load: 0.00020 [I 2022-10-09 00:47:29] ...trap/engines/engine.py.321: eval: epoch 25 | batch 180/510 [I 2022-10-09 00:47:29] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:45 [I 2022-10-09 00:47:29] ...trap/engines/engine.py.326: process: 0.32309 | load: 0.00020 [I 2022-10-09 00:47:32] ...trap/engines/engine.py.321: eval: epoch 25 | batch 190/510 [I 2022-10-09 00:47:32] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:42 [I 2022-10-09 00:47:32] ...trap/engines/engine.py.326: process: 0.31737 | load: 0.00019 [I 2022-10-09 00:47:35] ...trap/engines/engine.py.321: eval: epoch 25 | batch 200/510 [I 2022-10-09 00:47:35] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:38 [I 2022-10-09 00:47:35] ...trap/engines/engine.py.326: process: 0.31214 | load: 0.00020 [I 2022-10-09 00:47:38] ...trap/engines/engine.py.321: eval: epoch 25 | batch 210/510 [I 2022-10-09 00:47:38] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:35 [I 2022-10-09 00:47:38] ...trap/engines/engine.py.326: process: 0.31024 | load: 0.00020 [I 2022-10-09 00:47:42] ...trap/engines/engine.py.321: eval: epoch 25 | batch 220/510 [I 2022-10-09 00:47:42] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:32 [I 2022-10-09 00:47:42] ...trap/engines/engine.py.326: process: 0.31652 | load: 0.00020 [I 2022-10-09 00:47:45] ...trap/engines/engine.py.321: eval: epoch 25 | batch 230/510 [I 2022-10-09 00:47:45] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:29 [I 2022-10-09 00:47:45] ...trap/engines/engine.py.326: process: 0.31523 | load: 0.00019 [I 2022-10-09 00:47:48] ...trap/engines/engine.py.321: eval: epoch 25 | batch 240/510 [I 2022-10-09 00:47:48] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:26 [I 2022-10-09 00:47:48] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00020 [I 2022-10-09 00:47:51] ...trap/engines/engine.py.321: eval: epoch 25 | batch 250/510 [I 2022-10-09 00:47:51] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-09 00:47:51] ...trap/engines/engine.py.326: process: 0.31296 | load: 0.00022 [I 2022-10-09 00:47:54] ...trap/engines/engine.py.321: eval: epoch 25 | batch 260/510 [I 2022-10-09 00:47:54] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:19 [I 2022-10-09 00:47:54] ...trap/engines/engine.py.326: process: 0.31554 | load: 0.00020 [I 2022-10-09 00:47:58] ...trap/engines/engine.py.321: eval: epoch 25 | batch 270/510 [I 2022-10-09 00:47:58] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:16 [I 2022-10-09 00:47:58] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00020 [I 2022-10-09 00:48:01] ...trap/engines/engine.py.321: eval: epoch 25 | batch 280/510 [I 2022-10-09 00:48:01] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-09 00:48:01] ...trap/engines/engine.py.326: process: 0.32377 | load: 0.00020 [I 2022-10-09 00:48:04] ...trap/engines/engine.py.321: eval: epoch 25 | batch 290/510 [I 2022-10-09 00:48:04] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:10 [I 2022-10-09 00:48:04] ...trap/engines/engine.py.326: process: 0.32010 | load: 0.00021 [I 2022-10-09 00:48:07] ...trap/engines/engine.py.321: eval: epoch 25 | batch 300/510 [I 2022-10-09 00:48:07] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:06 [I 2022-10-09 00:48:07] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00019 [I 2022-10-09 00:48:10] ...trap/engines/engine.py.321: eval: epoch 25 | batch 310/510 [I 2022-10-09 00:48:10] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:04 [I 2022-10-09 00:48:10] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00020 [I 2022-10-09 00:48:14] ...trap/engines/engine.py.321: eval: epoch 25 | batch 320/510 [I 2022-10-09 00:48:14] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-09 00:48:14] ...trap/engines/engine.py.326: process: 0.32186 | load: 0.00020 [I 2022-10-09 00:48:17] ...trap/engines/engine.py.321: eval: epoch 25 | batch 330/510 [I 2022-10-09 00:48:17] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-09 00:48:17] ...trap/engines/engine.py.326: process: 0.31257 | load: 0.00020 [I 2022-10-09 00:48:20] ...trap/engines/engine.py.321: eval: epoch 25 | batch 340/510 [I 2022-10-09 00:48:20] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:55 [I 2022-10-09 00:48:20] ...trap/engines/engine.py.326: process: 0.33270 | load: 0.00021 [I 2022-10-09 00:48:23] ...trap/engines/engine.py.321: eval: epoch 25 | batch 350/510 [I 2022-10-09 00:48:23] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:51 [I 2022-10-09 00:48:23] ...trap/engines/engine.py.326: process: 0.31987 | load: 0.00020 [I 2022-10-09 00:48:26] ...trap/engines/engine.py.321: eval: epoch 25 | batch 360/510 [I 2022-10-09 00:48:26] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:48 [I 2022-10-09 00:48:26] ...trap/engines/engine.py.326: process: 0.32165 | load: 0.00020 [I 2022-10-09 00:48:30] ...trap/engines/engine.py.321: eval: epoch 25 | batch 370/510 [I 2022-10-09 00:48:30] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-09 00:48:30] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00020 [I 2022-10-09 00:48:33] ...trap/engines/engine.py.321: eval: epoch 25 | batch 380/510 [I 2022-10-09 00:48:33] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:41 [I 2022-10-09 00:48:33] ...trap/engines/engine.py.326: process: 0.32140 | load: 0.00021 [I 2022-10-09 00:48:36] ...trap/engines/engine.py.321: eval: epoch 25 | batch 390/510 [I 2022-10-09 00:48:36] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:38 [I 2022-10-09 00:48:36] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00020 [I 2022-10-09 00:48:39] ...trap/engines/engine.py.321: eval: epoch 25 | batch 400/510 [I 2022-10-09 00:48:39] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:35 [I 2022-10-09 00:48:39] ...trap/engines/engine.py.326: process: 0.32570 | load: 0.00020 [I 2022-10-09 00:48:42] ...trap/engines/engine.py.321: eval: epoch 25 | batch 410/510 [I 2022-10-09 00:48:42] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:32 [I 2022-10-09 00:48:42] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00020 [I 2022-10-09 00:48:46] ...trap/engines/engine.py.321: eval: epoch 25 | batch 420/510 [I 2022-10-09 00:48:46] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:29 [I 2022-10-09 00:48:46] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00019 [I 2022-10-09 00:48:49] ...trap/engines/engine.py.321: eval: epoch 25 | batch 430/510 [I 2022-10-09 00:48:49] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-09 00:48:49] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00021 [I 2022-10-09 00:48:52] ...trap/engines/engine.py.321: eval: epoch 25 | batch 440/510 [I 2022-10-09 00:48:52] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-09 00:48:52] ...trap/engines/engine.py.326: process: 0.31887 | load: 0.00019 [I 2022-10-09 00:48:55] ...trap/engines/engine.py.321: eval: epoch 25 | batch 450/510 [I 2022-10-09 00:48:55] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:19 [I 2022-10-09 00:48:55] ...trap/engines/engine.py.326: process: 0.32137 | load: 0.00019 [I 2022-10-09 00:48:58] ...trap/engines/engine.py.321: eval: epoch 25 | batch 460/510 [I 2022-10-09 00:48:58] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:16 [I 2022-10-09 00:48:58] ...trap/engines/engine.py.326: process: 0.32345 | load: 0.00017 [I 2022-10-09 00:49:02] ...trap/engines/engine.py.321: eval: epoch 25 | batch 470/510 [I 2022-10-09 00:49:02] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-09 00:49:02] ...trap/engines/engine.py.326: process: 0.32222 | load: 0.00021 [I 2022-10-09 00:49:05] ...trap/engines/engine.py.321: eval: epoch 25 | batch 480/510 [I 2022-10-09 00:49:05] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-09 00:49:05] ...trap/engines/engine.py.326: process: 0.32245 | load: 0.00019 [I 2022-10-09 00:49:08] ...trap/engines/engine.py.321: eval: epoch 25 | batch 490/510 [I 2022-10-09 00:49:08] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-09 00:49:08] ...trap/engines/engine.py.326: process: 0.32233 | load: 0.00020 [I 2022-10-09 00:49:11] ...trap/engines/engine.py.321: eval: epoch 25 | batch 500/510 [I 2022-10-09 00:49:11] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:03 [I 2022-10-09 00:49:11] ...trap/engines/engine.py.326: process: 0.31259 | load: 0.00020 [I 2022-10-09 00:49:14] ...trap/engines/engine.py.321: eval: epoch 25 | batch 510/510 [I 2022-10-09 00:49:14] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:00 [I 2022-10-09 00:49:14] ...trap/engines/engine.py.326: process: 0.30691 | load: 0.00016 [S 2022-10-09 00:49:15] ...trap/engines/engine.py.132: eval_epoch.epoch: 25 [I 2022-10-09 00:49:15] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.672700 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.882200 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.927000 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.659100 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.881900 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.924100 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015265 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010294 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008695 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010044 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010995 [S 2022-10-09 00:49:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007674 [I 2022-10-09 00:49:55] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 00:49:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 00:49:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 00:49:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 00:49:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 00:49:56] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 00:49:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 00:50:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 00:50:04] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 00:50:04] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 00:50:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 00:50:13] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 00:50:13] ...trap/engines/engine.py.163: Training model on trainset for epoch 26 [I 2022-10-09 00:50:13] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 00:50:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 0/2304 [I 2022-10-09 00:50:16] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:57 [I 2022-10-09 00:50:16] ...trap/engines/engine.py.229: process: 2.99385 | load: 2.09723 [I 2022-10-09 00:50:16] ...trap/engines/engine.py.230: loss: 0.18063 [I 2022-10-09 00:50:16] ...optimizers/trijoint.py.151: total_norm: 29.464535 [I 2022-10-09 00:50:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 10/2304 [I 2022-10-09 00:50:24] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:00 [I 2022-10-09 00:50:24] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00549 [I 2022-10-09 00:50:24] ...trap/engines/engine.py.230: loss: 0.19748 [I 2022-10-09 00:50:24] ...optimizers/trijoint.py.151: total_norm: 12.224912 [I 2022-10-09 00:50:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 20/2304 [I 2022-10-09 00:50:33] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:05 [I 2022-10-09 00:50:33] ...trap/engines/engine.py.229: process: 0.85134 | load: 0.00478 [I 2022-10-09 00:50:33] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-09 00:50:33] ...optimizers/trijoint.py.151: total_norm: 12.842569 [I 2022-10-09 00:50:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 30/2304 [I 2022-10-09 00:50:41] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:07 [I 2022-10-09 00:50:41] ...trap/engines/engine.py.229: process: 0.85090 | load: 0.00388 [I 2022-10-09 00:50:41] ...trap/engines/engine.py.230: loss: 0.13887 [I 2022-10-09 00:50:41] ...optimizers/trijoint.py.151: total_norm: 22.579338 [I 2022-10-09 00:50:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 40/2304 [I 2022-10-09 00:50:49] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:51 [I 2022-10-09 00:50:49] ...trap/engines/engine.py.229: process: 0.84230 | load: 0.00409 [I 2022-10-09 00:50:49] ...trap/engines/engine.py.230: loss: 0.15520 [I 2022-10-09 00:50:49] ...optimizers/trijoint.py.151: total_norm: 16.093990 [I 2022-10-09 00:50:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 50/2304 [I 2022-10-09 00:50:58] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:37 [I 2022-10-09 00:50:58] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00328 [I 2022-10-09 00:50:58] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-09 00:50:58] ...optimizers/trijoint.py.151: total_norm: 30.949013 [I 2022-10-09 00:51:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 60/2304 [I 2022-10-09 00:51:06] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:39 [I 2022-10-09 00:51:06] ...trap/engines/engine.py.229: process: 0.83177 | load: 0.00394 [I 2022-10-09 00:51:06] ...trap/engines/engine.py.230: loss: 0.18489 [I 2022-10-09 00:51:06] ...optimizers/trijoint.py.151: total_norm: 18.314745 [I 2022-10-09 00:51:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 70/2304 [I 2022-10-09 00:51:15] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:32 [I 2022-10-09 00:51:15] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00371 [I 2022-10-09 00:51:15] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-09 00:51:15] ...optimizers/trijoint.py.151: total_norm: 38.318718 [I 2022-10-09 00:51:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 80/2304 [I 2022-10-09 00:51:23] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:19 [I 2022-10-09 00:51:23] ...trap/engines/engine.py.229: process: 0.84277 | load: 0.00363 [I 2022-10-09 00:51:23] ...trap/engines/engine.py.230: loss: 0.13097 [I 2022-10-09 00:51:23] ...optimizers/trijoint.py.151: total_norm: 15.653769 [I 2022-10-09 00:51:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 90/2304 [I 2022-10-09 00:51:32] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:04 [I 2022-10-09 00:51:32] ...trap/engines/engine.py.229: process: 0.82960 | load: 0.00318 [I 2022-10-09 00:51:32] ...trap/engines/engine.py.230: loss: 0.23484 [I 2022-10-09 00:51:32] ...optimizers/trijoint.py.151: total_norm: 25.298220 [I 2022-10-09 00:51:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 100/2304 [I 2022-10-09 00:51:40] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:05 [I 2022-10-09 00:51:40] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00304 [I 2022-10-09 00:51:40] ...trap/engines/engine.py.230: loss: 0.17326 [I 2022-10-09 00:51:40] ...optimizers/trijoint.py.151: total_norm: 16.287169 [I 2022-10-09 00:51:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 110/2304 [I 2022-10-09 00:51:49] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:54 [I 2022-10-09 00:51:49] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00332 [I 2022-10-09 00:51:49] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-09 00:51:49] ...optimizers/trijoint.py.151: total_norm: 16.129574 [I 2022-10-09 00:51:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 120/2304 [I 2022-10-09 00:51:57] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:46 [I 2022-10-09 00:51:57] ...trap/engines/engine.py.229: process: 0.83426 | load: 0.00325 [I 2022-10-09 00:51:57] ...trap/engines/engine.py.230: loss: 0.15927 [I 2022-10-09 00:51:57] ...optimizers/trijoint.py.151: total_norm: 13.609282 [I 2022-10-09 00:52:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 130/2304 [I 2022-10-09 00:52:06] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:25 [I 2022-10-09 00:52:06] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00320 [I 2022-10-09 00:52:06] ...trap/engines/engine.py.230: loss: 0.22770 [I 2022-10-09 00:52:06] ...optimizers/trijoint.py.151: total_norm: 14.935122 [I 2022-10-09 00:52:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 140/2304 [I 2022-10-09 00:52:14] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:25 [I 2022-10-09 00:52:14] ...trap/engines/engine.py.229: process: 0.82329 | load: 0.00299 [I 2022-10-09 00:52:14] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-09 00:52:14] ...optimizers/trijoint.py.151: total_norm: 28.074287 [I 2022-10-09 00:52:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 150/2304 [I 2022-10-09 00:52:22] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:08 [I 2022-10-09 00:52:22] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00330 [I 2022-10-09 00:52:22] ...trap/engines/engine.py.230: loss: 0.20302 [I 2022-10-09 00:52:22] ...optimizers/trijoint.py.151: total_norm: 16.791965 [I 2022-10-09 00:52:31] ...trap/engines/engine.py.224: train: epoch 26 | batch 160/2304 [I 2022-10-09 00:52:31] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:07 [I 2022-10-09 00:52:31] ...trap/engines/engine.py.229: process: 0.83099 | load: 0.00314 [I 2022-10-09 00:52:31] ...trap/engines/engine.py.230: loss: 0.13691 [I 2022-10-09 00:52:31] ...optimizers/trijoint.py.151: total_norm: 13.294632 [I 2022-10-09 00:52:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 170/2304 [I 2022-10-09 00:52:39] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:11 [I 2022-10-09 00:52:39] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00301 [I 2022-10-09 00:52:39] ...trap/engines/engine.py.230: loss: 0.17076 [I 2022-10-09 00:52:39] ...optimizers/trijoint.py.151: total_norm: 20.451168 [I 2022-10-09 00:52:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 180/2304 [I 2022-10-09 00:52:48] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:53 [I 2022-10-09 00:52:48] ...trap/engines/engine.py.229: process: 0.84355 | load: 0.00313 [I 2022-10-09 00:52:48] ...trap/engines/engine.py.230: loss: 0.17255 [I 2022-10-09 00:52:48] ...optimizers/trijoint.py.151: total_norm: 25.742210 [I 2022-10-09 00:52:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 190/2304 [I 2022-10-09 00:52:56] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:39 [I 2022-10-09 00:52:56] ...trap/engines/engine.py.229: process: 0.83090 | load: 0.00305 [I 2022-10-09 00:52:56] ...trap/engines/engine.py.230: loss: 0.14169 [I 2022-10-09 00:52:56] ...optimizers/trijoint.py.151: total_norm: 24.516302 [I 2022-10-09 00:53:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 200/2304 [I 2022-10-09 00:53:05] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:35 [I 2022-10-09 00:53:05] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00327 [I 2022-10-09 00:53:05] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-09 00:53:05] ...optimizers/trijoint.py.151: total_norm: 15.758993 [I 2022-10-09 00:53:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 210/2304 [I 2022-10-09 00:53:13] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:31 [I 2022-10-09 00:53:13] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00321 [I 2022-10-09 00:53:13] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-09 00:53:13] ...optimizers/trijoint.py.151: total_norm: 18.385002 [I 2022-10-09 00:53:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 220/2304 [I 2022-10-09 00:53:22] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:27 [I 2022-10-09 00:53:22] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00301 [I 2022-10-09 00:53:22] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-09 00:53:22] ...optimizers/trijoint.py.151: total_norm: 15.321218 [I 2022-10-09 00:53:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 230/2304 [I 2022-10-09 00:53:30] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:05 [I 2022-10-09 00:53:30] ...trap/engines/engine.py.229: process: 0.83440 | load: 0.00310 [I 2022-10-09 00:53:30] ...trap/engines/engine.py.230: loss: 0.19908 [I 2022-10-09 00:53:30] ...optimizers/trijoint.py.151: total_norm: 12.325523 [I 2022-10-09 00:53:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 240/2304 [I 2022-10-09 00:53:39] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:28:57 [I 2022-10-09 00:53:39] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00312 [I 2022-10-09 00:53:39] ...trap/engines/engine.py.230: loss: 0.24921 [I 2022-10-09 00:53:39] ...optimizers/trijoint.py.151: total_norm: 13.750995 [I 2022-10-09 00:53:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 250/2304 [I 2022-10-09 00:53:47] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:57 [I 2022-10-09 00:53:47] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00300 [I 2022-10-09 00:53:47] ...trap/engines/engine.py.230: loss: 0.27162 [I 2022-10-09 00:53:47] ...optimizers/trijoint.py.151: total_norm: 18.253233 [I 2022-10-09 00:53:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 260/2304 [I 2022-10-09 00:53:56] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:46 [I 2022-10-09 00:53:56] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00339 [I 2022-10-09 00:53:56] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-09 00:53:56] ...optimizers/trijoint.py.151: total_norm: 17.442179 [I 2022-10-09 00:54:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 270/2304 [I 2022-10-09 00:54:04] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:40 [I 2022-10-09 00:54:04] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00302 [I 2022-10-09 00:54:04] ...trap/engines/engine.py.230: loss: 0.16250 [I 2022-10-09 00:54:04] ...optimizers/trijoint.py.151: total_norm: 27.487520 [I 2022-10-09 00:54:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 280/2304 [I 2022-10-09 00:54:13] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:20 [I 2022-10-09 00:54:13] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00312 [I 2022-10-09 00:54:13] ...trap/engines/engine.py.230: loss: 0.15431 [I 2022-10-09 00:54:13] ...optimizers/trijoint.py.151: total_norm: 10.516838 [I 2022-10-09 00:54:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 290/2304 [I 2022-10-09 00:54:21] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:06 [I 2022-10-09 00:54:21] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00324 [I 2022-10-09 00:54:21] ...trap/engines/engine.py.230: loss: 0.17049 [I 2022-10-09 00:54:21] ...optimizers/trijoint.py.151: total_norm: 19.974447 [I 2022-10-09 00:54:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 300/2304 [I 2022-10-09 00:54:29] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:01 [I 2022-10-09 00:54:29] ...trap/engines/engine.py.229: process: 0.82982 | load: 0.00315 [I 2022-10-09 00:54:29] ...trap/engines/engine.py.230: loss: 0.16495 [I 2022-10-09 00:54:29] ...optimizers/trijoint.py.151: total_norm: 16.281027 [I 2022-10-09 00:54:38] ...trap/engines/engine.py.224: train: epoch 26 | batch 310/2304 [I 2022-10-09 00:54:38] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:27:57 [I 2022-10-09 00:54:38] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00303 [I 2022-10-09 00:54:38] ...trap/engines/engine.py.230: loss: 0.14169 [I 2022-10-09 00:54:38] ...optimizers/trijoint.py.151: total_norm: 24.689293 [I 2022-10-09 00:54:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 320/2304 [I 2022-10-09 00:54:46] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:43 [I 2022-10-09 00:54:46] ...trap/engines/engine.py.229: process: 0.82909 | load: 0.00299 [I 2022-10-09 00:54:46] ...trap/engines/engine.py.230: loss: 0.29564 [I 2022-10-09 00:54:46] ...optimizers/trijoint.py.151: total_norm: 30.686203 [I 2022-10-09 00:54:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 330/2304 [I 2022-10-09 00:54:55] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:42 [I 2022-10-09 00:54:55] ...trap/engines/engine.py.229: process: 0.83572 | load: 0.00301 [I 2022-10-09 00:54:55] ...trap/engines/engine.py.230: loss: 0.15233 [I 2022-10-09 00:54:55] ...optimizers/trijoint.py.151: total_norm: 14.828486 [I 2022-10-09 00:55:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 340/2304 [I 2022-10-09 00:55:03] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:35 [I 2022-10-09 00:55:03] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00310 [I 2022-10-09 00:55:03] ...trap/engines/engine.py.230: loss: 0.20885 [I 2022-10-09 00:55:03] ...optimizers/trijoint.py.151: total_norm: 24.316525 [I 2022-10-09 00:55:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 350/2304 [I 2022-10-09 00:55:12] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:27 [I 2022-10-09 00:55:12] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00304 [I 2022-10-09 00:55:12] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-09 00:55:12] ...optimizers/trijoint.py.151: total_norm: 13.596187 [I 2022-10-09 00:55:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 360/2304 [I 2022-10-09 00:55:20] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:13 [I 2022-10-09 00:55:20] ...trap/engines/engine.py.229: process: 0.83151 | load: 0.00298 [I 2022-10-09 00:55:20] ...trap/engines/engine.py.230: loss: 0.21850 [I 2022-10-09 00:55:20] ...optimizers/trijoint.py.151: total_norm: 19.668379 [I 2022-10-09 00:55:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 370/2304 [I 2022-10-09 00:55:28] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:07 [I 2022-10-09 00:55:28] ...trap/engines/engine.py.229: process: 0.84373 | load: 0.00309 [I 2022-10-09 00:55:28] ...trap/engines/engine.py.230: loss: 0.17529 [I 2022-10-09 00:55:28] ...optimizers/trijoint.py.151: total_norm: 23.846378 [I 2022-10-09 00:55:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 380/2304 [I 2022-10-09 00:55:37] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:04 [I 2022-10-09 00:55:37] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00299 [I 2022-10-09 00:55:37] ...trap/engines/engine.py.230: loss: 0.18552 [I 2022-10-09 00:55:37] ...optimizers/trijoint.py.151: total_norm: 22.088457 [I 2022-10-09 00:55:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 390/2304 [I 2022-10-09 00:55:45] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:45 [I 2022-10-09 00:55:45] ...trap/engines/engine.py.229: process: 0.83414 | load: 0.00301 [I 2022-10-09 00:55:45] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-09 00:55:45] ...optimizers/trijoint.py.151: total_norm: 16.943451 [I 2022-10-09 00:55:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 400/2304 [I 2022-10-09 00:55:54] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:39 [I 2022-10-09 00:55:54] ...trap/engines/engine.py.229: process: 0.83558 | load: 0.00309 [I 2022-10-09 00:55:54] ...trap/engines/engine.py.230: loss: 0.20105 [I 2022-10-09 00:55:54] ...optimizers/trijoint.py.151: total_norm: 54.085693 [I 2022-10-09 00:56:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 410/2304 [I 2022-10-09 00:56:02] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:23 [I 2022-10-09 00:56:02] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00301 [I 2022-10-09 00:56:02] ...trap/engines/engine.py.230: loss: 0.19248 [I 2022-10-09 00:56:02] ...optimizers/trijoint.py.151: total_norm: 12.311464 [I 2022-10-09 00:56:10] ...trap/engines/engine.py.224: train: epoch 26 | batch 420/2304 [I 2022-10-09 00:56:10] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:22 [I 2022-10-09 00:56:10] ...trap/engines/engine.py.229: process: 0.83204 | load: 0.00309 [I 2022-10-09 00:56:10] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-09 00:56:10] ...optimizers/trijoint.py.151: total_norm: 21.321537 [I 2022-10-09 00:56:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 430/2304 [I 2022-10-09 00:56:19] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:10 [I 2022-10-09 00:56:19] ...trap/engines/engine.py.229: process: 0.83164 | load: 0.00298 [I 2022-10-09 00:56:19] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-09 00:56:19] ...optimizers/trijoint.py.151: total_norm: 16.048870 [I 2022-10-09 00:56:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 440/2304 [I 2022-10-09 00:56:27] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:25:59 [I 2022-10-09 00:56:27] ...trap/engines/engine.py.229: process: 0.83039 | load: 0.00303 [I 2022-10-09 00:56:27] ...trap/engines/engine.py.230: loss: 0.22713 [I 2022-10-09 00:56:27] ...optimizers/trijoint.py.151: total_norm: 15.239029 [I 2022-10-09 00:56:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 450/2304 [I 2022-10-09 00:56:36] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:52 [I 2022-10-09 00:56:36] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00303 [I 2022-10-09 00:56:36] ...trap/engines/engine.py.230: loss: 0.14519 [I 2022-10-09 00:56:36] ...optimizers/trijoint.py.151: total_norm: 16.627531 [I 2022-10-09 00:56:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 460/2304 [I 2022-10-09 00:56:44] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:25:53 [I 2022-10-09 00:56:44] ...trap/engines/engine.py.229: process: 0.82877 | load: 0.00306 [I 2022-10-09 00:56:44] ...trap/engines/engine.py.230: loss: 0.17569 [I 2022-10-09 00:56:44] ...optimizers/trijoint.py.151: total_norm: 23.808012 [I 2022-10-09 00:56:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 470/2304 [I 2022-10-09 00:56:52] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:43 [I 2022-10-09 00:56:52] ...trap/engines/engine.py.229: process: 0.84625 | load: 0.00300 [I 2022-10-09 00:56:52] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-09 00:56:52] ...optimizers/trijoint.py.151: total_norm: 16.048862 [I 2022-10-09 00:57:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 480/2304 [I 2022-10-09 00:57:01] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:31 [I 2022-10-09 00:57:01] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00299 [I 2022-10-09 00:57:01] ...trap/engines/engine.py.230: loss: 0.13423 [I 2022-10-09 00:57:01] ...optimizers/trijoint.py.151: total_norm: 10.652616 [I 2022-10-09 00:57:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 490/2304 [I 2022-10-09 00:57:09] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:20 [I 2022-10-09 00:57:09] ...trap/engines/engine.py.229: process: 0.83777 | load: 0.00297 [I 2022-10-09 00:57:09] ...trap/engines/engine.py.230: loss: 0.21614 [I 2022-10-09 00:57:09] ...optimizers/trijoint.py.151: total_norm: 17.352867 [I 2022-10-09 00:57:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 500/2304 [I 2022-10-09 00:57:18] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:15 [I 2022-10-09 00:57:18] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00313 [I 2022-10-09 00:57:18] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-09 00:57:18] ...optimizers/trijoint.py.151: total_norm: 19.220346 [I 2022-10-09 00:57:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 510/2304 [I 2022-10-09 00:57:26] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:16 [I 2022-10-09 00:57:26] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00299 [I 2022-10-09 00:57:26] ...trap/engines/engine.py.230: loss: 0.14816 [I 2022-10-09 00:57:26] ...optimizers/trijoint.py.151: total_norm: 14.599012 [I 2022-10-09 00:57:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 520/2304 [I 2022-10-09 00:57:35] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:07 [I 2022-10-09 00:57:35] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00303 [I 2022-10-09 00:57:35] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-09 00:57:35] ...optimizers/trijoint.py.151: total_norm: 35.138695 [I 2022-10-09 00:57:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 530/2304 [I 2022-10-09 00:57:43] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:24:51 [I 2022-10-09 00:57:43] ...trap/engines/engine.py.229: process: 0.83986 | load: 0.00299 [I 2022-10-09 00:57:43] ...trap/engines/engine.py.230: loss: 0.16024 [I 2022-10-09 00:57:43] ...optimizers/trijoint.py.151: total_norm: 14.481878 [I 2022-10-09 00:57:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 540/2304 [I 2022-10-09 00:57:52] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:48 [I 2022-10-09 00:57:52] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00300 [I 2022-10-09 00:57:52] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-09 00:57:52] ...optimizers/trijoint.py.151: total_norm: 11.580829 [I 2022-10-09 00:58:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 550/2304 [I 2022-10-09 00:58:00] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:44 [I 2022-10-09 00:58:00] ...trap/engines/engine.py.229: process: 0.84832 | load: 0.00299 [I 2022-10-09 00:58:00] ...trap/engines/engine.py.230: loss: 0.18357 [I 2022-10-09 00:58:00] ...optimizers/trijoint.py.151: total_norm: 19.355383 [I 2022-10-09 00:58:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 560/2304 [I 2022-10-09 00:58:08] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:35 [I 2022-10-09 00:58:08] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00300 [I 2022-10-09 00:58:08] ...trap/engines/engine.py.230: loss: 0.17732 [I 2022-10-09 00:58:08] ...optimizers/trijoint.py.151: total_norm: 14.321607 [I 2022-10-09 00:58:17] ...trap/engines/engine.py.224: train: epoch 26 | batch 570/2304 [I 2022-10-09 00:58:17] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:17 [I 2022-10-09 00:58:17] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00299 [I 2022-10-09 00:58:17] ...trap/engines/engine.py.230: loss: 0.20055 [I 2022-10-09 00:58:17] ...optimizers/trijoint.py.151: total_norm: 56.273540 [I 2022-10-09 00:58:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 580/2304 [I 2022-10-09 00:58:25] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:09 [I 2022-10-09 00:58:25] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00298 [I 2022-10-09 00:58:25] ...trap/engines/engine.py.230: loss: 0.23399 [I 2022-10-09 00:58:25] ...optimizers/trijoint.py.151: total_norm: 22.748598 [I 2022-10-09 00:58:34] ...trap/engines/engine.py.224: train: epoch 26 | batch 590/2304 [I 2022-10-09 00:58:34] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:02 [I 2022-10-09 00:58:34] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00314 [I 2022-10-09 00:58:34] ...trap/engines/engine.py.230: loss: 0.15974 [I 2022-10-09 00:58:34] ...optimizers/trijoint.py.151: total_norm: 21.373579 [I 2022-10-09 00:58:42] ...trap/engines/engine.py.224: train: epoch 26 | batch 600/2304 [I 2022-10-09 00:58:42] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:23:52 [I 2022-10-09 00:58:42] ...trap/engines/engine.py.229: process: 0.82434 | load: 0.00301 [I 2022-10-09 00:58:42] ...trap/engines/engine.py.230: loss: 0.11559 [I 2022-10-09 00:58:42] ...optimizers/trijoint.py.151: total_norm: 7.127264 [I 2022-10-09 00:58:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 610/2304 [I 2022-10-09 00:58:51] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:44 [I 2022-10-09 00:58:51] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00302 [I 2022-10-09 00:58:51] ...trap/engines/engine.py.230: loss: 0.22411 [I 2022-10-09 00:58:51] ...optimizers/trijoint.py.151: total_norm: 20.695713 [I 2022-10-09 00:58:59] ...trap/engines/engine.py.224: train: epoch 26 | batch 620/2304 [I 2022-10-09 00:58:59] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:41 [I 2022-10-09 00:58:59] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00301 [I 2022-10-09 00:58:59] ...trap/engines/engine.py.230: loss: 0.19670 [I 2022-10-09 00:58:59] ...optimizers/trijoint.py.151: total_norm: 24.488256 [I 2022-10-09 00:59:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 630/2304 [I 2022-10-09 00:59:07] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:27 [I 2022-10-09 00:59:07] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00305 [I 2022-10-09 00:59:07] ...trap/engines/engine.py.230: loss: 0.24097 [I 2022-10-09 00:59:07] ...optimizers/trijoint.py.151: total_norm: 21.474518 [I 2022-10-09 00:59:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 640/2304 [I 2022-10-09 00:59:16] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:24 [I 2022-10-09 00:59:16] ...trap/engines/engine.py.229: process: 0.85085 | load: 0.00306 [I 2022-10-09 00:59:16] ...trap/engines/engine.py.230: loss: 0.28902 [I 2022-10-09 00:59:16] ...optimizers/trijoint.py.151: total_norm: 18.806566 [I 2022-10-09 00:59:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 650/2304 [I 2022-10-09 00:59:24] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:12 [I 2022-10-09 00:59:24] ...trap/engines/engine.py.229: process: 0.83314 | load: 0.00299 [I 2022-10-09 00:59:24] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-09 00:59:24] ...optimizers/trijoint.py.151: total_norm: 11.855148 [I 2022-10-09 00:59:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 660/2304 [I 2022-10-09 00:59:33] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:01 [I 2022-10-09 00:59:33] ...trap/engines/engine.py.229: process: 0.83859 | load: 0.00298 [I 2022-10-09 00:59:33] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-09 00:59:33] ...optimizers/trijoint.py.151: total_norm: 26.029406 [I 2022-10-09 00:59:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 670/2304 [I 2022-10-09 00:59:41] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:22:55 [I 2022-10-09 00:59:41] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00299 [I 2022-10-09 00:59:41] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-09 00:59:41] ...optimizers/trijoint.py.151: total_norm: 15.736074 [I 2022-10-09 00:59:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 680/2304 [I 2022-10-09 00:59:50] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:45 [I 2022-10-09 00:59:50] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00303 [I 2022-10-09 00:59:50] ...trap/engines/engine.py.230: loss: 0.22413 [I 2022-10-09 00:59:50] ...optimizers/trijoint.py.151: total_norm: 21.975327 [I 2022-10-09 00:59:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 690/2304 [I 2022-10-09 00:59:58] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:22:40 [I 2022-10-09 00:59:58] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00315 [I 2022-10-09 00:59:58] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-09 00:59:58] ...optimizers/trijoint.py.151: total_norm: 14.397720 [I 2022-10-09 01:00:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 700/2304 [I 2022-10-09 01:00:06] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:29 [I 2022-10-09 01:00:06] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00300 [I 2022-10-09 01:00:06] ...trap/engines/engine.py.230: loss: 0.18761 [I 2022-10-09 01:00:06] ...optimizers/trijoint.py.151: total_norm: 23.321577 [I 2022-10-09 01:00:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 710/2304 [I 2022-10-09 01:00:15] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:24 [I 2022-10-09 01:00:15] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00306 [I 2022-10-09 01:00:15] ...trap/engines/engine.py.230: loss: 0.15824 [I 2022-10-09 01:00:15] ...optimizers/trijoint.py.151: total_norm: 18.946497 [I 2022-10-09 01:00:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 720/2304 [I 2022-10-09 01:00:23] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:14 [I 2022-10-09 01:00:23] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00300 [I 2022-10-09 01:00:23] ...trap/engines/engine.py.230: loss: 0.20099 [I 2022-10-09 01:00:23] ...optimizers/trijoint.py.151: total_norm: 16.388254 [I 2022-10-09 01:00:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 730/2304 [I 2022-10-09 01:00:32] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:09 [I 2022-10-09 01:00:32] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00302 [I 2022-10-09 01:00:32] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-09 01:00:32] ...optimizers/trijoint.py.151: total_norm: 19.325516 [I 2022-10-09 01:00:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 740/2304 [I 2022-10-09 01:00:40] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:21:58 [I 2022-10-09 01:00:40] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00302 [I 2022-10-09 01:00:40] ...trap/engines/engine.py.230: loss: 0.14281 [I 2022-10-09 01:00:40] ...optimizers/trijoint.py.151: total_norm: 23.247345 [I 2022-10-09 01:00:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 750/2304 [I 2022-10-09 01:00:49] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:47 [I 2022-10-09 01:00:49] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00298 [I 2022-10-09 01:00:49] ...trap/engines/engine.py.230: loss: 0.22719 [I 2022-10-09 01:00:49] ...optimizers/trijoint.py.151: total_norm: 22.903786 [I 2022-10-09 01:00:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 760/2304 [I 2022-10-09 01:00:57] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:42 [I 2022-10-09 01:00:57] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00300 [I 2022-10-09 01:00:57] ...trap/engines/engine.py.230: loss: 0.32042 [I 2022-10-09 01:00:57] ...optimizers/trijoint.py.151: total_norm: 43.404747 [I 2022-10-09 01:01:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 770/2304 [I 2022-10-09 01:01:06] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:32 [I 2022-10-09 01:01:06] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00299 [I 2022-10-09 01:01:06] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-09 01:01:06] ...optimizers/trijoint.py.151: total_norm: 16.739729 [I 2022-10-09 01:01:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 780/2304 [I 2022-10-09 01:01:14] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:25 [I 2022-10-09 01:01:14] ...trap/engines/engine.py.229: process: 0.84710 | load: 0.00302 [I 2022-10-09 01:01:14] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-09 01:01:14] ...optimizers/trijoint.py.151: total_norm: 22.099262 [I 2022-10-09 01:01:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 790/2304 [I 2022-10-09 01:01:22] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:12 [I 2022-10-09 01:01:22] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00301 [I 2022-10-09 01:01:22] ...trap/engines/engine.py.230: loss: 0.12611 [I 2022-10-09 01:01:22] ...optimizers/trijoint.py.151: total_norm: 17.243141 [I 2022-10-09 01:01:31] ...trap/engines/engine.py.224: train: epoch 26 | batch 800/2304 [I 2022-10-09 01:01:31] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:20:58 [I 2022-10-09 01:01:31] ...trap/engines/engine.py.229: process: 0.83318 | load: 0.00297 [I 2022-10-09 01:01:31] ...trap/engines/engine.py.230: loss: 0.16934 [I 2022-10-09 01:01:31] ...optimizers/trijoint.py.151: total_norm: 29.251190 [I 2022-10-09 01:01:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 810/2304 [I 2022-10-09 01:01:39] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:20:52 [I 2022-10-09 01:01:39] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00305 [I 2022-10-09 01:01:39] ...trap/engines/engine.py.230: loss: 0.16200 [I 2022-10-09 01:01:39] ...optimizers/trijoint.py.151: total_norm: 15.122154 [I 2022-10-09 01:01:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 820/2304 [I 2022-10-09 01:01:48] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:47 [I 2022-10-09 01:01:48] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00300 [I 2022-10-09 01:01:48] ...trap/engines/engine.py.230: loss: 0.12189 [I 2022-10-09 01:01:48] ...optimizers/trijoint.py.151: total_norm: 11.383091 [I 2022-10-09 01:01:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 830/2304 [I 2022-10-09 01:01:56] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:41 [I 2022-10-09 01:01:56] ...trap/engines/engine.py.229: process: 0.84310 | load: 0.00300 [I 2022-10-09 01:01:56] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-09 01:01:56] ...optimizers/trijoint.py.151: total_norm: 15.406389 [I 2022-10-09 01:02:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 840/2304 [I 2022-10-09 01:02:04] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:27 [I 2022-10-09 01:02:04] ...trap/engines/engine.py.229: process: 0.83860 | load: 0.00301 [I 2022-10-09 01:02:04] ...trap/engines/engine.py.230: loss: 0.24189 [I 2022-10-09 01:02:04] ...optimizers/trijoint.py.151: total_norm: 12.927323 [I 2022-10-09 01:02:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 850/2304 [I 2022-10-09 01:02:13] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:08 [I 2022-10-09 01:02:13] ...trap/engines/engine.py.229: process: 0.81869 | load: 0.00300 [I 2022-10-09 01:02:13] ...trap/engines/engine.py.230: loss: 0.11670 [I 2022-10-09 01:02:13] ...optimizers/trijoint.py.151: total_norm: 12.827541 [I 2022-10-09 01:02:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 860/2304 [I 2022-10-09 01:02:21] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:15 [I 2022-10-09 01:02:21] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00299 [I 2022-10-09 01:02:21] ...trap/engines/engine.py.230: loss: 0.18579 [I 2022-10-09 01:02:21] ...optimizers/trijoint.py.151: total_norm: 17.373077 [I 2022-10-09 01:02:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 870/2304 [I 2022-10-09 01:02:30] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:06 [I 2022-10-09 01:02:30] ...trap/engines/engine.py.229: process: 0.82849 | load: 0.00303 [I 2022-10-09 01:02:30] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-09 01:02:30] ...optimizers/trijoint.py.151: total_norm: 23.788177 [I 2022-10-09 01:02:38] ...trap/engines/engine.py.224: train: epoch 26 | batch 880/2304 [I 2022-10-09 01:02:38] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:19:51 [I 2022-10-09 01:02:38] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00302 [I 2022-10-09 01:02:38] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-09 01:02:38] ...optimizers/trijoint.py.151: total_norm: 11.054334 [I 2022-10-09 01:02:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 890/2304 [I 2022-10-09 01:02:46] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:44 [I 2022-10-09 01:02:46] ...trap/engines/engine.py.229: process: 0.83123 | load: 0.00302 [I 2022-10-09 01:02:46] ...trap/engines/engine.py.230: loss: 0.21254 [I 2022-10-09 01:02:46] ...optimizers/trijoint.py.151: total_norm: 16.904091 [I 2022-10-09 01:02:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 900/2304 [I 2022-10-09 01:02:55] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:45 [I 2022-10-09 01:02:55] ...trap/engines/engine.py.229: process: 0.86121 | load: 0.00302 [I 2022-10-09 01:02:55] ...trap/engines/engine.py.230: loss: 0.17013 [I 2022-10-09 01:02:55] ...optimizers/trijoint.py.151: total_norm: 15.857440 [I 2022-10-09 01:03:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 910/2304 [I 2022-10-09 01:03:03] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:42 [I 2022-10-09 01:03:03] ...trap/engines/engine.py.229: process: 0.85705 | load: 0.00299 [I 2022-10-09 01:03:03] ...trap/engines/engine.py.230: loss: 0.19917 [I 2022-10-09 01:03:03] ...optimizers/trijoint.py.151: total_norm: 17.319286 [I 2022-10-09 01:03:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 920/2304 [I 2022-10-09 01:03:12] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:36 [I 2022-10-09 01:03:12] ...trap/engines/engine.py.229: process: 0.86632 | load: 0.00301 [I 2022-10-09 01:03:12] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-09 01:03:12] ...optimizers/trijoint.py.151: total_norm: 19.602034 [I 2022-10-09 01:03:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 930/2304 [I 2022-10-09 01:03:20] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:15 [I 2022-10-09 01:03:20] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00297 [I 2022-10-09 01:03:20] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-09 01:03:20] ...optimizers/trijoint.py.151: total_norm: 14.694396 [I 2022-10-09 01:03:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 940/2304 [I 2022-10-09 01:03:29] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:06 [I 2022-10-09 01:03:29] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00304 [I 2022-10-09 01:03:29] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-09 01:03:29] ...optimizers/trijoint.py.151: total_norm: 24.112509 [I 2022-10-09 01:03:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 950/2304 [I 2022-10-09 01:03:37] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:18:59 [I 2022-10-09 01:03:37] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00299 [I 2022-10-09 01:03:37] ...trap/engines/engine.py.230: loss: 0.19568 [I 2022-10-09 01:03:37] ...optimizers/trijoint.py.151: total_norm: 20.429628 [I 2022-10-09 01:03:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 960/2304 [I 2022-10-09 01:03:46] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:18:46 [I 2022-10-09 01:03:46] ...trap/engines/engine.py.229: process: 0.83348 | load: 0.00298 [I 2022-10-09 01:03:46] ...trap/engines/engine.py.230: loss: 0.15868 [I 2022-10-09 01:03:46] ...optimizers/trijoint.py.151: total_norm: 15.816051 [I 2022-10-09 01:03:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 970/2304 [I 2022-10-09 01:03:54] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:43 [I 2022-10-09 01:03:54] ...trap/engines/engine.py.229: process: 0.83655 | load: 0.00304 [I 2022-10-09 01:03:54] ...trap/engines/engine.py.230: loss: 0.17495 [I 2022-10-09 01:03:54] ...optimizers/trijoint.py.151: total_norm: 19.978390 [I 2022-10-09 01:04:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 980/2304 [I 2022-10-09 01:04:02] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:34 [I 2022-10-09 01:04:02] ...trap/engines/engine.py.229: process: 0.83057 | load: 0.00305 [I 2022-10-09 01:04:02] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-09 01:04:02] ...optimizers/trijoint.py.151: total_norm: 14.805840 [I 2022-10-09 01:04:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 990/2304 [I 2022-10-09 01:04:11] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:23 [I 2022-10-09 01:04:11] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00300 [I 2022-10-09 01:04:11] ...trap/engines/engine.py.230: loss: 0.12234 [I 2022-10-09 01:04:11] ...optimizers/trijoint.py.151: total_norm: 11.541544 [I 2022-10-09 01:04:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 1000/2304 [I 2022-10-09 01:04:19] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:16 [I 2022-10-09 01:04:19] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00301 [I 2022-10-09 01:04:19] ...trap/engines/engine.py.230: loss: 0.14896 [I 2022-10-09 01:04:19] ...optimizers/trijoint.py.151: total_norm: 23.296827 [I 2022-10-09 01:04:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 1010/2304 [I 2022-10-09 01:04:28] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:09 [I 2022-10-09 01:04:28] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00300 [I 2022-10-09 01:04:28] ...trap/engines/engine.py.230: loss: 0.21388 [I 2022-10-09 01:04:28] ...optimizers/trijoint.py.151: total_norm: 20.517717 [I 2022-10-09 01:04:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 1020/2304 [I 2022-10-09 01:04:36] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:04 [I 2022-10-09 01:04:36] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00301 [I 2022-10-09 01:04:36] ...trap/engines/engine.py.230: loss: 0.20009 [I 2022-10-09 01:04:36] ...optimizers/trijoint.py.151: total_norm: 18.773928 [I 2022-10-09 01:04:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 1030/2304 [I 2022-10-09 01:04:45] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:17:50 [I 2022-10-09 01:04:45] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00303 [I 2022-10-09 01:04:45] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-09 01:04:45] ...optimizers/trijoint.py.151: total_norm: 16.313423 [I 2022-10-09 01:04:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 1040/2304 [I 2022-10-09 01:04:53] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:40 [I 2022-10-09 01:04:53] ...trap/engines/engine.py.229: process: 0.83421 | load: 0.00301 [I 2022-10-09 01:04:53] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-09 01:04:53] ...optimizers/trijoint.py.151: total_norm: 12.491848 [I 2022-10-09 01:05:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 1050/2304 [I 2022-10-09 01:05:01] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:32 [I 2022-10-09 01:05:01] ...trap/engines/engine.py.229: process: 0.83661 | load: 0.00306 [I 2022-10-09 01:05:01] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-09 01:05:01] ...optimizers/trijoint.py.151: total_norm: 17.222561 [I 2022-10-09 01:05:10] ...trap/engines/engine.py.224: train: epoch 26 | batch 1060/2304 [I 2022-10-09 01:05:10] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:24 [I 2022-10-09 01:05:10] ...trap/engines/engine.py.229: process: 0.83317 | load: 0.00301 [I 2022-10-09 01:05:10] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-09 01:05:10] ...optimizers/trijoint.py.151: total_norm: 15.751854 [I 2022-10-09 01:05:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 1070/2304 [I 2022-10-09 01:05:18] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:23 [I 2022-10-09 01:05:18] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00302 [I 2022-10-09 01:05:18] ...trap/engines/engine.py.230: loss: 0.13951 [I 2022-10-09 01:05:18] ...optimizers/trijoint.py.151: total_norm: 22.044455 [I 2022-10-09 01:05:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 1080/2304 [I 2022-10-09 01:05:27] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:10 [I 2022-10-09 01:05:27] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00302 [I 2022-10-09 01:05:27] ...trap/engines/engine.py.230: loss: 0.16450 [I 2022-10-09 01:05:27] ...optimizers/trijoint.py.151: total_norm: 13.625156 [I 2022-10-09 01:05:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 1090/2304 [I 2022-10-09 01:05:35] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:16:59 [I 2022-10-09 01:05:35] ...trap/engines/engine.py.229: process: 0.83547 | load: 0.00301 [I 2022-10-09 01:05:35] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-09 01:05:35] ...optimizers/trijoint.py.151: total_norm: 32.220631 [I 2022-10-09 01:05:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 1100/2304 [I 2022-10-09 01:05:44] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:16:53 [I 2022-10-09 01:05:44] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00309 [I 2022-10-09 01:05:44] ...trap/engines/engine.py.230: loss: 0.17584 [I 2022-10-09 01:05:44] ...optimizers/trijoint.py.151: total_norm: 14.855535 [I 2022-10-09 01:05:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 1110/2304 [I 2022-10-09 01:05:52] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:44 [I 2022-10-09 01:05:52] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00301 [I 2022-10-09 01:05:52] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-09 01:05:52] ...optimizers/trijoint.py.151: total_norm: 18.442612 [I 2022-10-09 01:06:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 1120/2304 [I 2022-10-09 01:06:01] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:39 [I 2022-10-09 01:06:01] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00301 [I 2022-10-09 01:06:01] ...trap/engines/engine.py.230: loss: 0.17771 [I 2022-10-09 01:06:01] ...optimizers/trijoint.py.151: total_norm: 16.709267 [I 2022-10-09 01:06:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 1130/2304 [I 2022-10-09 01:06:09] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:28 [I 2022-10-09 01:06:09] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00299 [I 2022-10-09 01:06:09] ...trap/engines/engine.py.230: loss: 0.15380 [I 2022-10-09 01:06:09] ...optimizers/trijoint.py.151: total_norm: 18.115520 [I 2022-10-09 01:06:17] ...trap/engines/engine.py.224: train: epoch 26 | batch 1140/2304 [I 2022-10-09 01:06:17] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:21 [I 2022-10-09 01:06:17] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00299 [I 2022-10-09 01:06:17] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-09 01:06:17] ...optimizers/trijoint.py.151: total_norm: 18.815748 [I 2022-10-09 01:06:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 1150/2304 [I 2022-10-09 01:06:26] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:11 [I 2022-10-09 01:06:26] ...trap/engines/engine.py.229: process: 0.82542 | load: 0.00304 [I 2022-10-09 01:06:26] ...trap/engines/engine.py.230: loss: 0.15557 [I 2022-10-09 01:06:26] ...optimizers/trijoint.py.151: total_norm: 14.657403 [I 2022-10-09 01:06:34] ...trap/engines/engine.py.224: train: epoch 26 | batch 1160/2304 [I 2022-10-09 01:06:34] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:03 [I 2022-10-09 01:06:34] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00306 [I 2022-10-09 01:06:34] ...trap/engines/engine.py.230: loss: 0.13136 [I 2022-10-09 01:06:34] ...optimizers/trijoint.py.151: total_norm: 15.783366 [I 2022-10-09 01:06:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 1170/2304 [I 2022-10-09 01:06:43] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:15:52 [I 2022-10-09 01:06:43] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00298 [I 2022-10-09 01:06:43] ...trap/engines/engine.py.230: loss: 0.22462 [I 2022-10-09 01:06:43] ...optimizers/trijoint.py.151: total_norm: 21.606501 [I 2022-10-09 01:06:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 1180/2304 [I 2022-10-09 01:06:51] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:15:42 [I 2022-10-09 01:06:51] ...trap/engines/engine.py.229: process: 0.82127 | load: 0.00301 [I 2022-10-09 01:06:51] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-09 01:06:51] ...optimizers/trijoint.py.151: total_norm: 16.193693 [I 2022-10-09 01:07:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 1190/2304 [I 2022-10-09 01:07:00] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:36 [I 2022-10-09 01:07:00] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00301 [I 2022-10-09 01:07:00] ...trap/engines/engine.py.230: loss: 0.21265 [I 2022-10-09 01:07:00] ...optimizers/trijoint.py.151: total_norm: 27.688889 [I 2022-10-09 01:07:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 1200/2304 [I 2022-10-09 01:07:08] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:29 [I 2022-10-09 01:07:08] ...trap/engines/engine.py.229: process: 0.85392 | load: 0.00304 [I 2022-10-09 01:07:08] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-09 01:07:08] ...optimizers/trijoint.py.151: total_norm: 23.293266 [I 2022-10-09 01:07:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 1210/2304 [I 2022-10-09 01:07:16] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:20 [I 2022-10-09 01:07:16] ...trap/engines/engine.py.229: process: 0.85512 | load: 0.00301 [I 2022-10-09 01:07:16] ...trap/engines/engine.py.230: loss: 0.17572 [I 2022-10-09 01:07:16] ...optimizers/trijoint.py.151: total_norm: 11.224500 [I 2022-10-09 01:07:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 1220/2304 [I 2022-10-09 01:07:25] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:10 [I 2022-10-09 01:07:25] ...trap/engines/engine.py.229: process: 0.84379 | load: 0.00298 [I 2022-10-09 01:07:25] ...trap/engines/engine.py.230: loss: 0.17661 [I 2022-10-09 01:07:25] ...optimizers/trijoint.py.151: total_norm: 18.778692 [I 2022-10-09 01:07:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 1230/2304 [I 2022-10-09 01:07:33] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:04 [I 2022-10-09 01:07:33] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00298 [I 2022-10-09 01:07:33] ...trap/engines/engine.py.230: loss: 0.22773 [I 2022-10-09 01:07:33] ...optimizers/trijoint.py.151: total_norm: 17.506395 [I 2022-10-09 01:07:42] ...trap/engines/engine.py.224: train: epoch 26 | batch 1240/2304 [I 2022-10-09 01:07:42] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:14:54 [I 2022-10-09 01:07:42] ...trap/engines/engine.py.229: process: 0.83556 | load: 0.00298 [I 2022-10-09 01:07:42] ...trap/engines/engine.py.230: loss: 0.17025 [I 2022-10-09 01:07:42] ...optimizers/trijoint.py.151: total_norm: 34.373634 [I 2022-10-09 01:07:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 1250/2304 [I 2022-10-09 01:07:50] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:14:46 [I 2022-10-09 01:07:50] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00304 [I 2022-10-09 01:07:50] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-09 01:07:50] ...optimizers/trijoint.py.151: total_norm: 11.635180 [I 2022-10-09 01:07:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 1260/2304 [I 2022-10-09 01:07:58] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:35 [I 2022-10-09 01:07:58] ...trap/engines/engine.py.229: process: 0.82994 | load: 0.00300 [I 2022-10-09 01:07:58] ...trap/engines/engine.py.230: loss: 0.17360 [I 2022-10-09 01:07:58] ...optimizers/trijoint.py.151: total_norm: 20.865490 [I 2022-10-09 01:08:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 1270/2304 [I 2022-10-09 01:08:07] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:29 [I 2022-10-09 01:08:07] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00306 [I 2022-10-09 01:08:07] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-09 01:08:07] ...optimizers/trijoint.py.151: total_norm: 13.602116 [I 2022-10-09 01:08:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 1280/2304 [I 2022-10-09 01:08:15] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:21 [I 2022-10-09 01:08:15] ...trap/engines/engine.py.229: process: 0.84611 | load: 0.00304 [I 2022-10-09 01:08:15] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-09 01:08:15] ...optimizers/trijoint.py.151: total_norm: 27.760622 [I 2022-10-09 01:08:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 1290/2304 [I 2022-10-09 01:08:24] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:13 [I 2022-10-09 01:08:24] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00299 [I 2022-10-09 01:08:24] ...trap/engines/engine.py.230: loss: 0.25676 [I 2022-10-09 01:08:24] ...optimizers/trijoint.py.151: total_norm: 21.416868 [I 2022-10-09 01:08:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 1300/2304 [I 2022-10-09 01:08:32] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:02 [I 2022-10-09 01:08:32] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00300 [I 2022-10-09 01:08:32] ...trap/engines/engine.py.230: loss: 0.10576 [I 2022-10-09 01:08:32] ...optimizers/trijoint.py.151: total_norm: 11.742218 [I 2022-10-09 01:08:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 1310/2304 [I 2022-10-09 01:08:41] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:13:54 [I 2022-10-09 01:08:41] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00303 [I 2022-10-09 01:08:41] ...trap/engines/engine.py.230: loss: 0.19618 [I 2022-10-09 01:08:41] ...optimizers/trijoint.py.151: total_norm: 22.265398 [I 2022-10-09 01:08:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 1320/2304 [I 2022-10-09 01:08:49] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:13:52 [I 2022-10-09 01:08:49] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00300 [I 2022-10-09 01:08:49] ...trap/engines/engine.py.230: loss: 0.19911 [I 2022-10-09 01:08:49] ...optimizers/trijoint.py.151: total_norm: 18.226347 [I 2022-10-09 01:08:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 1330/2304 [I 2022-10-09 01:08:58] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:48 [I 2022-10-09 01:08:58] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00300 [I 2022-10-09 01:08:58] ...trap/engines/engine.py.230: loss: 0.28570 [I 2022-10-09 01:08:58] ...optimizers/trijoint.py.151: total_norm: 17.559214 [I 2022-10-09 01:09:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 1340/2304 [I 2022-10-09 01:09:06] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:32 [I 2022-10-09 01:09:06] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00302 [I 2022-10-09 01:09:06] ...trap/engines/engine.py.230: loss: 0.14667 [I 2022-10-09 01:09:06] ...optimizers/trijoint.py.151: total_norm: 18.407812 [I 2022-10-09 01:09:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 1350/2304 [I 2022-10-09 01:09:14] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:25 [I 2022-10-09 01:09:14] ...trap/engines/engine.py.229: process: 0.84948 | load: 0.00301 [I 2022-10-09 01:09:14] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-09 01:09:14] ...optimizers/trijoint.py.151: total_norm: 18.233082 [I 2022-10-09 01:09:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 1360/2304 [I 2022-10-09 01:09:23] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:17 [I 2022-10-09 01:09:23] ...trap/engines/engine.py.229: process: 0.83460 | load: 0.00301 [I 2022-10-09 01:09:23] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-09 01:09:23] ...optimizers/trijoint.py.151: total_norm: 15.588243 [I 2022-10-09 01:09:31] ...trap/engines/engine.py.224: train: epoch 26 | batch 1370/2304 [I 2022-10-09 01:09:31] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:03 [I 2022-10-09 01:09:31] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00299 [I 2022-10-09 01:09:31] ...trap/engines/engine.py.230: loss: 0.16336 [I 2022-10-09 01:09:31] ...optimizers/trijoint.py.151: total_norm: 19.462955 [I 2022-10-09 01:09:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 1380/2304 [I 2022-10-09 01:09:40] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:01 [I 2022-10-09 01:09:40] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00301 [I 2022-10-09 01:09:40] ...trap/engines/engine.py.230: loss: 0.29491 [I 2022-10-09 01:09:40] ...optimizers/trijoint.py.151: total_norm: 26.596041 [I 2022-10-09 01:09:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 1390/2304 [I 2022-10-09 01:09:48] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:12:50 [I 2022-10-09 01:09:48] ...trap/engines/engine.py.229: process: 0.84413 | load: 0.00299 [I 2022-10-09 01:09:48] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-09 01:09:48] ...optimizers/trijoint.py.151: total_norm: 10.659002 [I 2022-10-09 01:09:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 1400/2304 [I 2022-10-09 01:09:57] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:12:43 [I 2022-10-09 01:09:57] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00309 [I 2022-10-09 01:09:57] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-09 01:09:57] ...optimizers/trijoint.py.151: total_norm: 14.792942 [I 2022-10-09 01:10:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 1410/2304 [I 2022-10-09 01:10:05] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:12:32 [I 2022-10-09 01:10:05] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00300 [I 2022-10-09 01:10:05] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-09 01:10:05] ...optimizers/trijoint.py.151: total_norm: 20.036285 [I 2022-10-09 01:10:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 1420/2304 [I 2022-10-09 01:10:14] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:25 [I 2022-10-09 01:10:14] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00303 [I 2022-10-09 01:10:14] ...trap/engines/engine.py.230: loss: 0.19408 [I 2022-10-09 01:10:14] ...optimizers/trijoint.py.151: total_norm: 14.841659 [I 2022-10-09 01:10:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 1430/2304 [I 2022-10-09 01:10:22] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:16 [I 2022-10-09 01:10:22] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00301 [I 2022-10-09 01:10:22] ...trap/engines/engine.py.230: loss: 0.19411 [I 2022-10-09 01:10:22] ...optimizers/trijoint.py.151: total_norm: 17.495483 [I 2022-10-09 01:10:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 1440/2304 [I 2022-10-09 01:10:30] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:03 [I 2022-10-09 01:10:30] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00300 [I 2022-10-09 01:10:30] ...trap/engines/engine.py.230: loss: 0.13270 [I 2022-10-09 01:10:30] ...optimizers/trijoint.py.151: total_norm: 8.175972 [I 2022-10-09 01:10:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 1450/2304 [I 2022-10-09 01:10:39] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:04 [I 2022-10-09 01:10:39] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00304 [I 2022-10-09 01:10:39] ...trap/engines/engine.py.230: loss: 0.18701 [I 2022-10-09 01:10:39] ...optimizers/trijoint.py.151: total_norm: 26.323458 [I 2022-10-09 01:10:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 1460/2304 [I 2022-10-09 01:10:47] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:11:51 [I 2022-10-09 01:10:47] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00299 [I 2022-10-09 01:10:47] ...trap/engines/engine.py.230: loss: 0.15644 [I 2022-10-09 01:10:47] ...optimizers/trijoint.py.151: total_norm: 22.737661 [I 2022-10-09 01:10:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 1470/2304 [I 2022-10-09 01:10:56] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:11:39 [I 2022-10-09 01:10:56] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00306 [I 2022-10-09 01:10:56] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-09 01:10:56] ...optimizers/trijoint.py.151: total_norm: 23.500509 [I 2022-10-09 01:11:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 1480/2304 [I 2022-10-09 01:11:04] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:33 [I 2022-10-09 01:11:04] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00301 [I 2022-10-09 01:11:04] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-09 01:11:04] ...optimizers/trijoint.py.151: total_norm: 19.414221 [I 2022-10-09 01:11:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 1490/2304 [I 2022-10-09 01:11:13] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:11:24 [I 2022-10-09 01:11:13] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00309 [I 2022-10-09 01:11:13] ...trap/engines/engine.py.230: loss: 0.21346 [I 2022-10-09 01:11:13] ...optimizers/trijoint.py.151: total_norm: 36.656578 [I 2022-10-09 01:11:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 1500/2304 [I 2022-10-09 01:11:21] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:16 [I 2022-10-09 01:11:21] ...trap/engines/engine.py.229: process: 0.84664 | load: 0.00307 [I 2022-10-09 01:11:21] ...trap/engines/engine.py.230: loss: 0.16905 [I 2022-10-09 01:11:21] ...optimizers/trijoint.py.151: total_norm: 28.509974 [I 2022-10-09 01:11:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 1510/2304 [I 2022-10-09 01:11:30] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:06 [I 2022-10-09 01:11:30] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00305 [I 2022-10-09 01:11:30] ...trap/engines/engine.py.230: loss: 0.15836 [I 2022-10-09 01:11:30] ...optimizers/trijoint.py.151: total_norm: 15.961102 [I 2022-10-09 01:11:38] ...trap/engines/engine.py.224: train: epoch 26 | batch 1520/2304 [I 2022-10-09 01:11:38] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:00 [I 2022-10-09 01:11:38] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00304 [I 2022-10-09 01:11:38] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-09 01:11:38] ...optimizers/trijoint.py.151: total_norm: 28.920341 [I 2022-10-09 01:11:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 1530/2304 [I 2022-10-09 01:11:46] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:10:49 [I 2022-10-09 01:11:46] ...trap/engines/engine.py.229: process: 0.81832 | load: 0.00301 [I 2022-10-09 01:11:46] ...trap/engines/engine.py.230: loss: 0.15328 [I 2022-10-09 01:11:46] ...optimizers/trijoint.py.151: total_norm: 14.890792 [I 2022-10-09 01:11:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 1540/2304 [I 2022-10-09 01:11:55] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:10:43 [I 2022-10-09 01:11:55] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00301 [I 2022-10-09 01:11:55] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-09 01:11:55] ...optimizers/trijoint.py.151: total_norm: 17.621960 [I 2022-10-09 01:12:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 1550/2304 [I 2022-10-09 01:12:03] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:32 [I 2022-10-09 01:12:03] ...trap/engines/engine.py.229: process: 0.82932 | load: 0.00297 [I 2022-10-09 01:12:03] ...trap/engines/engine.py.230: loss: 0.17906 [I 2022-10-09 01:12:03] ...optimizers/trijoint.py.151: total_norm: 24.866945 [I 2022-10-09 01:12:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 1560/2304 [I 2022-10-09 01:12:12] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:20 [I 2022-10-09 01:12:12] ...trap/engines/engine.py.229: process: 0.82181 | load: 0.00299 [I 2022-10-09 01:12:12] ...trap/engines/engine.py.230: loss: 0.23231 [I 2022-10-09 01:12:12] ...optimizers/trijoint.py.151: total_norm: 19.754467 [I 2022-10-09 01:12:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 1570/2304 [I 2022-10-09 01:12:20] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:13 [I 2022-10-09 01:12:20] ...trap/engines/engine.py.229: process: 0.82441 | load: 0.00297 [I 2022-10-09 01:12:20] ...trap/engines/engine.py.230: loss: 0.23387 [I 2022-10-09 01:12:20] ...optimizers/trijoint.py.151: total_norm: 18.939480 [I 2022-10-09 01:12:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 1580/2304 [I 2022-10-09 01:12:28] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:09 [I 2022-10-09 01:12:28] ...trap/engines/engine.py.229: process: 0.85248 | load: 0.00300 [I 2022-10-09 01:12:28] ...trap/engines/engine.py.230: loss: 0.19459 [I 2022-10-09 01:12:28] ...optimizers/trijoint.py.151: total_norm: 29.327316 [I 2022-10-09 01:12:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 1590/2304 [I 2022-10-09 01:12:37] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:09:58 [I 2022-10-09 01:12:37] ...trap/engines/engine.py.229: process: 0.82471 | load: 0.00301 [I 2022-10-09 01:12:37] ...trap/engines/engine.py.230: loss: 0.18424 [I 2022-10-09 01:12:37] ...optimizers/trijoint.py.151: total_norm: 16.063450 [I 2022-10-09 01:12:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 1600/2304 [I 2022-10-09 01:12:45] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:09:52 [I 2022-10-09 01:12:45] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00303 [I 2022-10-09 01:12:45] ...trap/engines/engine.py.230: loss: 0.20855 [I 2022-10-09 01:12:45] ...optimizers/trijoint.py.151: total_norm: 19.976267 [I 2022-10-09 01:12:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 1610/2304 [I 2022-10-09 01:12:54] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:09:42 [I 2022-10-09 01:12:54] ...trap/engines/engine.py.229: process: 0.82551 | load: 0.00299 [I 2022-10-09 01:12:54] ...trap/engines/engine.py.230: loss: 0.17043 [I 2022-10-09 01:12:54] ...optimizers/trijoint.py.151: total_norm: 10.385694 [I 2022-10-09 01:13:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 1620/2304 [I 2022-10-09 01:13:02] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:36 [I 2022-10-09 01:13:02] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00300 [I 2022-10-09 01:13:02] ...trap/engines/engine.py.230: loss: 0.18768 [I 2022-10-09 01:13:02] ...optimizers/trijoint.py.151: total_norm: 24.168530 [I 2022-10-09 01:13:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 1630/2304 [I 2022-10-09 01:13:11] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:27 [I 2022-10-09 01:13:11] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00300 [I 2022-10-09 01:13:11] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-09 01:13:11] ...optimizers/trijoint.py.151: total_norm: 16.005529 [I 2022-10-09 01:13:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 1640/2304 [I 2022-10-09 01:13:19] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:16 [I 2022-10-09 01:13:19] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00300 [I 2022-10-09 01:13:19] ...trap/engines/engine.py.230: loss: 0.25465 [I 2022-10-09 01:13:19] ...optimizers/trijoint.py.151: total_norm: 45.257671 [I 2022-10-09 01:13:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 1650/2304 [I 2022-10-09 01:13:45] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:15:43 [I 2022-10-09 01:13:45] ...trap/engines/engine.py.229: process: 0.83365 | load: 0.00301 [I 2022-10-09 01:13:45] ...trap/engines/engine.py.230: loss: 0.23081 [I 2022-10-09 01:13:45] ...optimizers/trijoint.py.151: total_norm: 15.900414 [I 2022-10-09 01:13:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 1660/2304 [I 2022-10-09 01:13:54] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:42 [I 2022-10-09 01:13:54] ...trap/engines/engine.py.229: process: 0.83243 | load: 0.00302 [I 2022-10-09 01:13:54] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-09 01:13:54] ...optimizers/trijoint.py.151: total_norm: 26.324465 [I 2022-10-09 01:14:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 1670/2304 [I 2022-10-09 01:14:02] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:08:54 [I 2022-10-09 01:14:02] ...trap/engines/engine.py.229: process: 0.82422 | load: 0.00298 [I 2022-10-09 01:14:02] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-09 01:14:02] ...optimizers/trijoint.py.151: total_norm: 16.764793 [I 2022-10-09 01:14:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 1680/2304 [I 2022-10-09 01:14:11] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:42 [I 2022-10-09 01:14:11] ...trap/engines/engine.py.229: process: 0.83380 | load: 0.00299 [I 2022-10-09 01:14:11] ...trap/engines/engine.py.230: loss: 0.16212 [I 2022-10-09 01:14:11] ...optimizers/trijoint.py.151: total_norm: 14.261913 [I 2022-10-09 01:14:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 1690/2304 [I 2022-10-09 01:14:19] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:37 [I 2022-10-09 01:14:19] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00302 [I 2022-10-09 01:14:19] ...trap/engines/engine.py.230: loss: 0.15930 [I 2022-10-09 01:14:19] ...optimizers/trijoint.py.151: total_norm: 18.413412 [I 2022-10-09 01:14:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 1700/2304 [I 2022-10-09 01:14:27] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:25 [I 2022-10-09 01:14:27] ...trap/engines/engine.py.229: process: 0.84964 | load: 0.00301 [I 2022-10-09 01:14:27] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-09 01:14:27] ...optimizers/trijoint.py.151: total_norm: 14.241594 [I 2022-10-09 01:14:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 1710/2304 [I 2022-10-09 01:14:36] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:20 [I 2022-10-09 01:14:36] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00302 [I 2022-10-09 01:14:36] ...trap/engines/engine.py.230: loss: 0.18174 [I 2022-10-09 01:14:36] ...optimizers/trijoint.py.151: total_norm: 21.319254 [I 2022-10-09 01:14:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 1720/2304 [I 2022-10-09 01:14:44] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:09 [I 2022-10-09 01:14:44] ...trap/engines/engine.py.229: process: 0.83680 | load: 0.00298 [I 2022-10-09 01:14:44] ...trap/engines/engine.py.230: loss: 0.18667 [I 2022-10-09 01:14:44] ...optimizers/trijoint.py.151: total_norm: 34.985508 [I 2022-10-09 01:14:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 1730/2304 [I 2022-10-09 01:14:53] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:00 [I 2022-10-09 01:14:53] ...trap/engines/engine.py.229: process: 0.83037 | load: 0.00299 [I 2022-10-09 01:14:53] ...trap/engines/engine.py.230: loss: 0.19496 [I 2022-10-09 01:14:53] ...optimizers/trijoint.py.151: total_norm: 18.826637 [I 2022-10-09 01:15:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 1740/2304 [I 2022-10-09 01:15:01] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:07:54 [I 2022-10-09 01:15:01] ...trap/engines/engine.py.229: process: 0.84399 | load: 0.00302 [I 2022-10-09 01:15:01] ...trap/engines/engine.py.230: loss: 0.19954 [I 2022-10-09 01:15:01] ...optimizers/trijoint.py.151: total_norm: 17.699776 [I 2022-10-09 01:15:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 1750/2304 [I 2022-10-09 01:15:09] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:07:47 [I 2022-10-09 01:15:09] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00299 [I 2022-10-09 01:15:09] ...trap/engines/engine.py.230: loss: 0.20234 [I 2022-10-09 01:15:09] ...optimizers/trijoint.py.151: total_norm: 14.070021 [I 2022-10-09 01:15:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 1760/2304 [I 2022-10-09 01:15:18] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:36 [I 2022-10-09 01:15:18] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00299 [I 2022-10-09 01:15:18] ...trap/engines/engine.py.230: loss: 0.13622 [I 2022-10-09 01:15:18] ...optimizers/trijoint.py.151: total_norm: 19.425877 [I 2022-10-09 01:15:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 1770/2304 [I 2022-10-09 01:15:26] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:27 [I 2022-10-09 01:15:26] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00302 [I 2022-10-09 01:15:26] ...trap/engines/engine.py.230: loss: 0.18434 [I 2022-10-09 01:15:26] ...optimizers/trijoint.py.151: total_norm: 44.407307 [I 2022-10-09 01:15:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 1780/2304 [I 2022-10-09 01:15:35] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:22 [I 2022-10-09 01:15:35] ...trap/engines/engine.py.229: process: 0.82721 | load: 0.00305 [I 2022-10-09 01:15:35] ...trap/engines/engine.py.230: loss: 0.13853 [I 2022-10-09 01:15:35] ...optimizers/trijoint.py.151: total_norm: 29.032410 [I 2022-10-09 01:15:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 1790/2304 [I 2022-10-09 01:15:43] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:17 [I 2022-10-09 01:15:43] ...trap/engines/engine.py.229: process: 0.86456 | load: 0.00316 [I 2022-10-09 01:15:43] ...trap/engines/engine.py.230: loss: 0.17929 [I 2022-10-09 01:15:43] ...optimizers/trijoint.py.151: total_norm: 17.399788 [I 2022-10-09 01:15:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 1800/2304 [I 2022-10-09 01:15:52] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:05 [I 2022-10-09 01:15:52] ...trap/engines/engine.py.229: process: 0.83817 | load: 0.00302 [I 2022-10-09 01:15:52] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-09 01:15:52] ...optimizers/trijoint.py.151: total_norm: 19.831514 [I 2022-10-09 01:16:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 1810/2304 [I 2022-10-09 01:16:00] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:06:54 [I 2022-10-09 01:16:00] ...trap/engines/engine.py.229: process: 0.83424 | load: 0.00298 [I 2022-10-09 01:16:00] ...trap/engines/engine.py.230: loss: 0.23935 [I 2022-10-09 01:16:00] ...optimizers/trijoint.py.151: total_norm: 19.347965 [I 2022-10-09 01:16:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 1820/2304 [I 2022-10-09 01:16:09] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:06:46 [I 2022-10-09 01:16:09] ...trap/engines/engine.py.229: process: 0.83420 | load: 0.00300 [I 2022-10-09 01:16:09] ...trap/engines/engine.py.230: loss: 0.17609 [I 2022-10-09 01:16:09] ...optimizers/trijoint.py.151: total_norm: 14.577139 [I 2022-10-09 01:16:17] ...trap/engines/engine.py.224: train: epoch 26 | batch 1830/2304 [I 2022-10-09 01:16:17] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:37 [I 2022-10-09 01:16:17] ...trap/engines/engine.py.229: process: 0.83108 | load: 0.00302 [I 2022-10-09 01:16:17] ...trap/engines/engine.py.230: loss: 0.21009 [I 2022-10-09 01:16:17] ...optimizers/trijoint.py.151: total_norm: 15.873487 [I 2022-10-09 01:16:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 1840/2304 [I 2022-10-09 01:16:25] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:28 [I 2022-10-09 01:16:25] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00301 [I 2022-10-09 01:16:25] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-09 01:16:25] ...optimizers/trijoint.py.151: total_norm: 11.226791 [I 2022-10-09 01:16:34] ...trap/engines/engine.py.224: train: epoch 26 | batch 1850/2304 [I 2022-10-09 01:16:34] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:21 [I 2022-10-09 01:16:34] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00301 [I 2022-10-09 01:16:34] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-09 01:16:34] ...optimizers/trijoint.py.151: total_norm: 17.056913 [I 2022-10-09 01:16:42] ...trap/engines/engine.py.224: train: epoch 26 | batch 1860/2304 [I 2022-10-09 01:16:42] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:13 [I 2022-10-09 01:16:42] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00297 [I 2022-10-09 01:16:42] ...trap/engines/engine.py.230: loss: 0.23118 [I 2022-10-09 01:16:42] ...optimizers/trijoint.py.151: total_norm: 33.826462 [I 2022-10-09 01:16:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 1870/2304 [I 2022-10-09 01:16:51] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:03 [I 2022-10-09 01:16:51] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00303 [I 2022-10-09 01:16:51] ...trap/engines/engine.py.230: loss: 0.23306 [I 2022-10-09 01:16:51] ...optimizers/trijoint.py.151: total_norm: 15.723621 [I 2022-10-09 01:16:59] ...trap/engines/engine.py.224: train: epoch 26 | batch 1880/2304 [I 2022-10-09 01:16:59] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:05:55 [I 2022-10-09 01:16:59] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00302 [I 2022-10-09 01:16:59] ...trap/engines/engine.py.230: loss: 0.14388 [I 2022-10-09 01:16:59] ...optimizers/trijoint.py.151: total_norm: 18.192974 [I 2022-10-09 01:17:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 1890/2304 [I 2022-10-09 01:17:07] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:05:49 [I 2022-10-09 01:17:07] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00300 [I 2022-10-09 01:17:07] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-09 01:17:07] ...optimizers/trijoint.py.151: total_norm: 19.553650 [I 2022-10-09 01:17:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 1900/2304 [I 2022-10-09 01:17:16] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:05:39 [I 2022-10-09 01:17:16] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00299 [I 2022-10-09 01:17:16] ...trap/engines/engine.py.230: loss: 0.20719 [I 2022-10-09 01:17:16] ...optimizers/trijoint.py.151: total_norm: 29.346617 [I 2022-10-09 01:17:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 1910/2304 [I 2022-10-09 01:17:24] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:29 [I 2022-10-09 01:17:24] ...trap/engines/engine.py.229: process: 0.82689 | load: 0.00301 [I 2022-10-09 01:17:24] ...trap/engines/engine.py.230: loss: 0.16187 [I 2022-10-09 01:17:24] ...optimizers/trijoint.py.151: total_norm: 18.643230 [I 2022-10-09 01:17:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 1920/2304 [I 2022-10-09 01:17:33] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:23 [I 2022-10-09 01:17:33] ...trap/engines/engine.py.229: process: 0.82811 | load: 0.00302 [I 2022-10-09 01:17:33] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-09 01:17:33] ...optimizers/trijoint.py.151: total_norm: 25.476229 [I 2022-10-09 01:17:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 1930/2304 [I 2022-10-09 01:17:41] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:13 [I 2022-10-09 01:17:41] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00300 [I 2022-10-09 01:17:41] ...trap/engines/engine.py.230: loss: 0.20243 [I 2022-10-09 01:17:41] ...optimizers/trijoint.py.151: total_norm: 16.291677 [I 2022-10-09 01:17:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 1940/2304 [I 2022-10-09 01:17:50] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:06 [I 2022-10-09 01:17:50] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00301 [I 2022-10-09 01:17:50] ...trap/engines/engine.py.230: loss: 0.15570 [I 2022-10-09 01:17:50] ...optimizers/trijoint.py.151: total_norm: 30.062990 [I 2022-10-09 01:17:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 1950/2304 [I 2022-10-09 01:17:58] ...trap/engines/engine.py.228: elapsed: 0:27:45 | left: 0:04:58 [I 2022-10-09 01:17:58] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00301 [I 2022-10-09 01:17:58] ...trap/engines/engine.py.230: loss: 0.26633 [I 2022-10-09 01:17:58] ...optimizers/trijoint.py.151: total_norm: 30.824272 [I 2022-10-09 01:18:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 1960/2304 [I 2022-10-09 01:18:06] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:04:49 [I 2022-10-09 01:18:06] ...trap/engines/engine.py.229: process: 0.83713 | load: 0.00301 [I 2022-10-09 01:18:06] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-09 01:18:06] ...optimizers/trijoint.py.151: total_norm: 19.732151 [I 2022-10-09 01:18:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 1970/2304 [I 2022-10-09 01:18:15] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:40 [I 2022-10-09 01:18:15] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00304 [I 2022-10-09 01:18:15] ...trap/engines/engine.py.230: loss: 0.17313 [I 2022-10-09 01:18:15] ...optimizers/trijoint.py.151: total_norm: 23.226248 [I 2022-10-09 01:18:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 1980/2304 [I 2022-10-09 01:18:23] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:32 [I 2022-10-09 01:18:23] ...trap/engines/engine.py.229: process: 0.81798 | load: 0.00306 [I 2022-10-09 01:18:23] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-09 01:18:23] ...optimizers/trijoint.py.151: total_norm: 17.764107 [I 2022-10-09 01:18:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 1990/2304 [I 2022-10-09 01:18:32] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:23 [I 2022-10-09 01:18:32] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00307 [I 2022-10-09 01:18:32] ...trap/engines/engine.py.230: loss: 0.12518 [I 2022-10-09 01:18:32] ...optimizers/trijoint.py.151: total_norm: 20.939554 [I 2022-10-09 01:18:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 2000/2304 [I 2022-10-09 01:18:40] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:15 [I 2022-10-09 01:18:40] ...trap/engines/engine.py.229: process: 0.84004 | load: 0.00302 [I 2022-10-09 01:18:40] ...trap/engines/engine.py.230: loss: 0.15051 [I 2022-10-09 01:18:40] ...optimizers/trijoint.py.151: total_norm: 15.046251 [I 2022-10-09 01:18:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 2010/2304 [I 2022-10-09 01:18:48] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:07 [I 2022-10-09 01:18:48] ...trap/engines/engine.py.229: process: 0.83444 | load: 0.00301 [I 2022-10-09 01:18:48] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-09 01:18:48] ...optimizers/trijoint.py.151: total_norm: 22.748686 [I 2022-10-09 01:18:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 2020/2304 [I 2022-10-09 01:18:57] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:03:57 [I 2022-10-09 01:18:57] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00308 [I 2022-10-09 01:18:57] ...trap/engines/engine.py.230: loss: 0.15944 [I 2022-10-09 01:18:57] ...optimizers/trijoint.py.151: total_norm: 31.670462 [I 2022-10-09 01:19:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 2030/2304 [I 2022-10-09 01:19:05] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:48 [I 2022-10-09 01:19:05] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00304 [I 2022-10-09 01:19:05] ...trap/engines/engine.py.230: loss: 0.15032 [I 2022-10-09 01:19:05] ...optimizers/trijoint.py.151: total_norm: 17.154224 [I 2022-10-09 01:19:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 2040/2304 [I 2022-10-09 01:19:14] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:43 [I 2022-10-09 01:19:14] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00304 [I 2022-10-09 01:19:14] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-09 01:19:14] ...optimizers/trijoint.py.151: total_norm: 21.526945 [I 2022-10-09 01:19:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 2050/2304 [I 2022-10-09 01:19:22] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:32 [I 2022-10-09 01:19:22] ...trap/engines/engine.py.229: process: 0.81697 | load: 0.00296 [I 2022-10-09 01:19:22] ...trap/engines/engine.py.230: loss: 0.13971 [I 2022-10-09 01:19:22] ...optimizers/trijoint.py.151: total_norm: 19.565348 [I 2022-10-09 01:19:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 2060/2304 [I 2022-10-09 01:19:30] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:24 [I 2022-10-09 01:19:30] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00301 [I 2022-10-09 01:19:30] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-09 01:19:30] ...optimizers/trijoint.py.151: total_norm: 15.557653 [I 2022-10-09 01:19:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 2070/2304 [I 2022-10-09 01:19:39] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:17 [I 2022-10-09 01:19:39] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00301 [I 2022-10-09 01:19:39] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-09 01:19:39] ...optimizers/trijoint.py.151: total_norm: 19.656527 [I 2022-10-09 01:19:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 2080/2304 [I 2022-10-09 01:19:47] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:08 [I 2022-10-09 01:19:47] ...trap/engines/engine.py.229: process: 0.83001 | load: 0.00301 [I 2022-10-09 01:19:47] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-09 01:19:47] ...optimizers/trijoint.py.151: total_norm: 29.965618 [I 2022-10-09 01:19:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 2090/2304 [I 2022-10-09 01:19:56] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:02:58 [I 2022-10-09 01:19:56] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00299 [I 2022-10-09 01:19:56] ...trap/engines/engine.py.230: loss: 0.10717 [I 2022-10-09 01:19:56] ...optimizers/trijoint.py.151: total_norm: 17.279221 [I 2022-10-09 01:20:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 2100/2304 [I 2022-10-09 01:20:04] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:02:50 [I 2022-10-09 01:20:04] ...trap/engines/engine.py.229: process: 0.81611 | load: 0.00299 [I 2022-10-09 01:20:04] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-09 01:20:04] ...optimizers/trijoint.py.151: total_norm: 16.724365 [I 2022-10-09 01:20:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 2110/2304 [I 2022-10-09 01:20:12] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:42 [I 2022-10-09 01:20:12] ...trap/engines/engine.py.229: process: 0.83025 | load: 0.00306 [I 2022-10-09 01:20:12] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-09 01:20:12] ...optimizers/trijoint.py.151: total_norm: 13.430269 [I 2022-10-09 01:20:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 2120/2304 [I 2022-10-09 01:20:21] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:34 [I 2022-10-09 01:20:21] ...trap/engines/engine.py.229: process: 0.83648 | load: 0.00298 [I 2022-10-09 01:20:21] ...trap/engines/engine.py.230: loss: 0.12999 [I 2022-10-09 01:20:21] ...optimizers/trijoint.py.151: total_norm: 21.700006 [I 2022-10-09 01:20:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 2130/2304 [I 2022-10-09 01:20:29] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:26 [I 2022-10-09 01:20:29] ...trap/engines/engine.py.229: process: 0.85111 | load: 0.00298 [I 2022-10-09 01:20:29] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-09 01:20:29] ...optimizers/trijoint.py.151: total_norm: 17.148623 [I 2022-10-09 01:20:38] ...trap/engines/engine.py.224: train: epoch 26 | batch 2140/2304 [I 2022-10-09 01:20:38] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:17 [I 2022-10-09 01:20:38] ...trap/engines/engine.py.229: process: 0.83669 | load: 0.00300 [I 2022-10-09 01:20:38] ...trap/engines/engine.py.230: loss: 0.20267 [I 2022-10-09 01:20:38] ...optimizers/trijoint.py.151: total_norm: 15.702968 [I 2022-10-09 01:20:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 2150/2304 [I 2022-10-09 01:20:46] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:08 [I 2022-10-09 01:20:46] ...trap/engines/engine.py.229: process: 0.83760 | load: 0.00300 [I 2022-10-09 01:20:46] ...trap/engines/engine.py.230: loss: 0.19841 [I 2022-10-09 01:20:46] ...optimizers/trijoint.py.151: total_norm: 16.497416 [I 2022-10-09 01:20:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 2160/2304 [I 2022-10-09 01:20:55] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:00 [I 2022-10-09 01:20:55] ...trap/engines/engine.py.229: process: 0.83565 | load: 0.00300 [I 2022-10-09 01:20:55] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-09 01:20:55] ...optimizers/trijoint.py.151: total_norm: 29.210154 [I 2022-10-09 01:21:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 2170/2304 [I 2022-10-09 01:21:03] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:01:52 [I 2022-10-09 01:21:03] ...trap/engines/engine.py.229: process: 0.83971 | load: 0.00298 [I 2022-10-09 01:21:03] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-09 01:21:03] ...optimizers/trijoint.py.151: total_norm: 21.053270 [I 2022-10-09 01:21:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 2180/2304 [I 2022-10-09 01:21:11] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:01:44 [I 2022-10-09 01:21:11] ...trap/engines/engine.py.229: process: 0.83959 | load: 0.00298 [I 2022-10-09 01:21:11] ...trap/engines/engine.py.230: loss: 0.21248 [I 2022-10-09 01:21:11] ...optimizers/trijoint.py.151: total_norm: 18.734440 [I 2022-10-09 01:21:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 2190/2304 [I 2022-10-09 01:21:20] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:35 [I 2022-10-09 01:21:20] ...trap/engines/engine.py.229: process: 0.83146 | load: 0.00298 [I 2022-10-09 01:21:20] ...trap/engines/engine.py.230: loss: 0.16301 [I 2022-10-09 01:21:20] ...optimizers/trijoint.py.151: total_norm: 13.658525 [I 2022-10-09 01:21:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 2200/2304 [I 2022-10-09 01:21:28] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:26 [I 2022-10-09 01:21:28] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00300 [I 2022-10-09 01:21:28] ...trap/engines/engine.py.230: loss: 0.17129 [I 2022-10-09 01:21:28] ...optimizers/trijoint.py.151: total_norm: 13.203975 [I 2022-10-09 01:21:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 2210/2304 [I 2022-10-09 01:21:37] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:19 [I 2022-10-09 01:21:37] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00302 [I 2022-10-09 01:21:37] ...trap/engines/engine.py.230: loss: 0.16604 [I 2022-10-09 01:21:37] ...optimizers/trijoint.py.151: total_norm: 19.493559 [I 2022-10-09 01:21:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 2220/2304 [I 2022-10-09 01:21:45] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:11 [I 2022-10-09 01:21:45] ...trap/engines/engine.py.229: process: 0.87274 | load: 0.00302 [I 2022-10-09 01:21:45] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-09 01:21:45] ...optimizers/trijoint.py.151: total_norm: 18.667334 [I 2022-10-09 01:21:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 2230/2304 [I 2022-10-09 01:21:53] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:02 [I 2022-10-09 01:21:53] ...trap/engines/engine.py.229: process: 0.83250 | load: 0.00303 [I 2022-10-09 01:21:53] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-09 01:21:53] ...optimizers/trijoint.py.151: total_norm: 20.940502 [I 2022-10-09 01:22:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 2240/2304 [I 2022-10-09 01:22:02] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:00:53 [I 2022-10-09 01:22:02] ...trap/engines/engine.py.229: process: 0.83075 | load: 0.00307 [I 2022-10-09 01:22:02] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-09 01:22:02] ...optimizers/trijoint.py.151: total_norm: 21.085951 [I 2022-10-09 01:22:10] ...trap/engines/engine.py.224: train: epoch 26 | batch 2250/2304 [I 2022-10-09 01:22:10] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:45 [I 2022-10-09 01:22:10] ...trap/engines/engine.py.229: process: 0.83325 | load: 0.00308 [I 2022-10-09 01:22:10] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-09 01:22:10] ...optimizers/trijoint.py.151: total_norm: 16.700779 [I 2022-10-09 01:22:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 2260/2304 [I 2022-10-09 01:22:19] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:00:37 [I 2022-10-09 01:22:19] ...trap/engines/engine.py.229: process: 0.84426 | load: 0.00309 [I 2022-10-09 01:22:19] ...trap/engines/engine.py.230: loss: 0.14533 [I 2022-10-09 01:22:19] ...optimizers/trijoint.py.151: total_norm: 27.711077 [I 2022-10-09 01:22:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 2270/2304 [I 2022-10-09 01:22:27] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:28 [I 2022-10-09 01:22:27] ...trap/engines/engine.py.229: process: 0.84247 | load: 0.00309 [I 2022-10-09 01:22:27] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-09 01:22:27] ...optimizers/trijoint.py.151: total_norm: 22.918325 [I 2022-10-09 01:22:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 2280/2304 [I 2022-10-09 01:22:36] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:20 [I 2022-10-09 01:22:36] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00309 [I 2022-10-09 01:22:36] ...trap/engines/engine.py.230: loss: 0.10398 [I 2022-10-09 01:22:36] ...optimizers/trijoint.py.151: total_norm: 9.771420 [I 2022-10-09 01:22:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 2290/2304 [I 2022-10-09 01:22:44] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:11 [I 2022-10-09 01:22:44] ...trap/engines/engine.py.229: process: 0.83322 | load: 0.00307 [I 2022-10-09 01:22:44] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-09 01:22:44] ...optimizers/trijoint.py.151: total_norm: 12.251909 [I 2022-10-09 01:22:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 2300/2304 [I 2022-10-09 01:22:52] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:03 [I 2022-10-09 01:22:52] ...trap/engines/engine.py.229: process: 0.81808 | load: 0.00017 [I 2022-10-09 01:22:52] ...trap/engines/engine.py.230: loss: 0.21307 [I 2022-10-09 01:22:52] ...optimizers/trijoint.py.151: total_norm: 16.319984 [I 2022-10-09 01:22:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 2304/2304 [I 2022-10-09 01:22:56] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:00 [I 2022-10-09 01:22:56] ...trap/engines/engine.py.229: process: 0.83206 | load: 0.00018 [I 2022-10-09 01:22:56] ...trap/engines/engine.py.230: loss: 0.16434 [I 2022-10-09 01:22:56] ...optimizers/trijoint.py.151: total_norm: 32.020779 [S 2022-10-09 01:22:57] ...trap/engines/engine.py.129: train_epoch.epoch: 26 [S 2022-10-09 01:22:57] ...trap/engines/engine.py.129: train_epoch.loss: 0.17038551024656234 [I 2022-10-09 01:22:57] ...s/criterions/triplet.py.84: margin = 0.1850000000000001 [I 2022-10-09 01:22:57] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 26 [I 2022-10-09 01:22:58] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 01:22:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 01:22:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 01:22:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 01:23:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 01:23:01] ...trap/engines/engine.py.321: eval: epoch 26 | batch 0/510 [I 2022-10-09 01:23:01] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:48 [I 2022-10-09 01:23:01] ...trap/engines/engine.py.326: process: 3.27062 | load: 2.75648 [I 2022-10-09 01:23:06] ...trap/engines/engine.py.321: eval: epoch 26 | batch 10/510 [I 2022-10-09 01:23:06] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:05 [I 2022-10-09 01:23:06] ...trap/engines/engine.py.326: process: 0.56790 | load: 0.00020 [I 2022-10-09 01:23:12] ...trap/engines/engine.py.321: eval: epoch 26 | batch 20/510 [I 2022-10-09 01:23:12] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:04:50 [I 2022-10-09 01:23:12] ...trap/engines/engine.py.326: process: 0.45994 | load: 0.00019 [I 2022-10-09 01:23:17] ...trap/engines/engine.py.321: eval: epoch 26 | batch 30/510 [I 2022-10-09 01:23:17] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:13 [I 2022-10-09 01:23:17] ...trap/engines/engine.py.326: process: 0.66347 | load: 0.00020 [I 2022-10-09 01:23:23] ...trap/engines/engine.py.321: eval: epoch 26 | batch 40/510 [I 2022-10-09 01:23:23] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:44 [I 2022-10-09 01:23:23] ...trap/engines/engine.py.326: process: 0.49075 | load: 0.00018 [I 2022-10-09 01:23:29] ...trap/engines/engine.py.321: eval: epoch 26 | batch 50/510 [I 2022-10-09 01:23:29] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:57 [I 2022-10-09 01:23:29] ...trap/engines/engine.py.326: process: 0.45818 | load: 0.00018 [I 2022-10-09 01:23:34] ...trap/engines/engine.py.321: eval: epoch 26 | batch 60/510 [I 2022-10-09 01:23:34] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:45 [I 2022-10-09 01:23:34] ...trap/engines/engine.py.326: process: 0.55185 | load: 0.00019 [I 2022-10-09 01:23:38] ...trap/engines/engine.py.321: eval: epoch 26 | batch 70/510 [I 2022-10-09 01:23:38] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:36 [I 2022-10-09 01:23:38] ...trap/engines/engine.py.326: process: 0.46904 | load: 0.00018 [I 2022-10-09 01:23:43] ...trap/engines/engine.py.321: eval: epoch 26 | batch 80/510 [I 2022-10-09 01:23:43] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:22 [I 2022-10-09 01:23:43] ...trap/engines/engine.py.326: process: 0.41723 | load: 0.00020 [I 2022-10-09 01:23:48] ...trap/engines/engine.py.321: eval: epoch 26 | batch 90/510 [I 2022-10-09 01:23:48] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:14 [I 2022-10-09 01:23:48] ...trap/engines/engine.py.326: process: 0.51814 | load: 0.00018 [I 2022-10-09 01:23:52] ...trap/engines/engine.py.321: eval: epoch 26 | batch 100/510 [I 2022-10-09 01:23:52] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:57 [I 2022-10-09 01:23:52] ...trap/engines/engine.py.326: process: 0.30903 | load: 0.00019 [I 2022-10-09 01:23:55] ...trap/engines/engine.py.321: eval: epoch 26 | batch 110/510 [I 2022-10-09 01:23:55] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:12 [I 2022-10-09 01:23:55] ...trap/engines/engine.py.326: process: 0.32953 | load: 0.00020 [I 2022-10-09 01:23:59] ...trap/engines/engine.py.321: eval: epoch 26 | batch 120/510 [I 2022-10-09 01:23:59] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-09 01:23:59] ...trap/engines/engine.py.326: process: 0.32290 | load: 0.00019 [I 2022-10-09 01:24:02] ...trap/engines/engine.py.321: eval: epoch 26 | batch 130/510 [I 2022-10-09 01:24:02] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-09 01:24:02] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00019 [I 2022-10-09 01:24:05] ...trap/engines/engine.py.321: eval: epoch 26 | batch 140/510 [I 2022-10-09 01:24:05] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-09 01:24:05] ...trap/engines/engine.py.326: process: 0.31741 | load: 0.00019 [I 2022-10-09 01:24:08] ...trap/engines/engine.py.321: eval: epoch 26 | batch 150/510 [I 2022-10-09 01:24:08] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-09 01:24:08] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00020 [I 2022-10-09 01:24:11] ...trap/engines/engine.py.321: eval: epoch 26 | batch 160/510 [I 2022-10-09 01:24:11] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:51 [I 2022-10-09 01:24:11] ...trap/engines/engine.py.326: process: 0.32109 | load: 0.00019 [I 2022-10-09 01:24:14] ...trap/engines/engine.py.321: eval: epoch 26 | batch 170/510 [I 2022-10-09 01:24:14] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-09 01:24:14] ...trap/engines/engine.py.326: process: 0.32379 | load: 0.00020 [I 2022-10-09 01:24:18] ...trap/engines/engine.py.321: eval: epoch 26 | batch 180/510 [I 2022-10-09 01:24:18] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-09 01:24:18] ...trap/engines/engine.py.326: process: 0.31601 | load: 0.00020 [I 2022-10-09 01:24:21] ...trap/engines/engine.py.321: eval: epoch 26 | batch 190/510 [I 2022-10-09 01:24:21] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:40 [I 2022-10-09 01:24:21] ...trap/engines/engine.py.326: process: 0.31405 | load: 0.00020 [I 2022-10-09 01:24:24] ...trap/engines/engine.py.321: eval: epoch 26 | batch 200/510 [I 2022-10-09 01:24:24] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-09 01:24:24] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00019 [I 2022-10-09 01:24:27] ...trap/engines/engine.py.321: eval: epoch 26 | batch 210/510 [I 2022-10-09 01:24:27] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-09 01:24:27] ...trap/engines/engine.py.326: process: 0.31237 | load: 0.00019 [I 2022-10-09 01:24:30] ...trap/engines/engine.py.321: eval: epoch 26 | batch 220/510 [I 2022-10-09 01:24:30] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-09 01:24:30] ...trap/engines/engine.py.326: process: 0.31842 | load: 0.00020 [I 2022-10-09 01:24:34] ...trap/engines/engine.py.321: eval: epoch 26 | batch 230/510 [I 2022-10-09 01:24:34] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-09 01:24:34] ...trap/engines/engine.py.326: process: 0.31429 | load: 0.00020 [I 2022-10-09 01:24:37] ...trap/engines/engine.py.321: eval: epoch 26 | batch 240/510 [I 2022-10-09 01:24:37] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-09 01:24:37] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00019 [I 2022-10-09 01:24:40] ...trap/engines/engine.py.321: eval: epoch 26 | batch 250/510 [I 2022-10-09 01:24:40] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-09 01:24:40] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00019 [I 2022-10-09 01:24:43] ...trap/engines/engine.py.321: eval: epoch 26 | batch 260/510 [I 2022-10-09 01:24:43] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:20 [I 2022-10-09 01:24:43] ...trap/engines/engine.py.326: process: 0.31809 | load: 0.00020 [I 2022-10-09 01:24:47] ...trap/engines/engine.py.321: eval: epoch 26 | batch 270/510 [I 2022-10-09 01:24:47] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 01:24:47] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00021 [I 2022-10-09 01:24:50] ...trap/engines/engine.py.321: eval: epoch 26 | batch 280/510 [I 2022-10-09 01:24:50] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 01:24:50] ...trap/engines/engine.py.326: process: 0.31442 | load: 0.00019 [I 2022-10-09 01:24:53] ...trap/engines/engine.py.321: eval: epoch 26 | batch 290/510 [I 2022-10-09 01:24:53] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-09 01:24:53] ...trap/engines/engine.py.326: process: 0.33194 | load: 0.00020 [I 2022-10-09 01:24:56] ...trap/engines/engine.py.321: eval: epoch 26 | batch 300/510 [I 2022-10-09 01:24:56] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-09 01:24:56] ...trap/engines/engine.py.326: process: 0.31841 | load: 0.00020 [I 2022-10-09 01:24:59] ...trap/engines/engine.py.321: eval: epoch 26 | batch 310/510 [I 2022-10-09 01:24:59] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:03 [I 2022-10-09 01:24:59] ...trap/engines/engine.py.326: process: 0.32507 | load: 0.00020 [I 2022-10-09 01:25:03] ...trap/engines/engine.py.321: eval: epoch 26 | batch 320/510 [I 2022-10-09 01:25:03] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-09 01:25:03] ...trap/engines/engine.py.326: process: 0.31593 | load: 0.00019 [I 2022-10-09 01:25:06] ...trap/engines/engine.py.321: eval: epoch 26 | batch 330/510 [I 2022-10-09 01:25:06] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 01:25:06] ...trap/engines/engine.py.326: process: 0.31827 | load: 0.00019 [I 2022-10-09 01:25:09] ...trap/engines/engine.py.321: eval: epoch 26 | batch 340/510 [I 2022-10-09 01:25:09] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 01:25:09] ...trap/engines/engine.py.326: process: 0.31883 | load: 0.00020 [I 2022-10-09 01:25:12] ...trap/engines/engine.py.321: eval: epoch 26 | batch 350/510 [I 2022-10-09 01:25:12] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:50 [I 2022-10-09 01:25:12] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00020 [I 2022-10-09 01:25:15] ...trap/engines/engine.py.321: eval: epoch 26 | batch 360/510 [I 2022-10-09 01:25:15] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:48 [I 2022-10-09 01:25:15] ...trap/engines/engine.py.326: process: 0.32139 | load: 0.00020 [I 2022-10-09 01:25:19] ...trap/engines/engine.py.321: eval: epoch 26 | batch 370/510 [I 2022-10-09 01:25:19] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 01:25:19] ...trap/engines/engine.py.326: process: 0.32301 | load: 0.00020 [I 2022-10-09 01:25:22] ...trap/engines/engine.py.321: eval: epoch 26 | batch 380/510 [I 2022-10-09 01:25:22] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 01:25:22] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00020 [I 2022-10-09 01:25:25] ...trap/engines/engine.py.321: eval: epoch 26 | batch 390/510 [I 2022-10-09 01:25:25] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-09 01:25:25] ...trap/engines/engine.py.326: process: 0.32223 | load: 0.00020 [I 2022-10-09 01:25:28] ...trap/engines/engine.py.321: eval: epoch 26 | batch 400/510 [I 2022-10-09 01:25:28] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-09 01:25:28] ...trap/engines/engine.py.326: process: 0.32654 | load: 0.00020 [I 2022-10-09 01:25:31] ...trap/engines/engine.py.321: eval: epoch 26 | batch 410/510 [I 2022-10-09 01:25:31] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:32 [I 2022-10-09 01:25:31] ...trap/engines/engine.py.326: process: 0.32460 | load: 0.00019 [I 2022-10-09 01:25:35] ...trap/engines/engine.py.321: eval: epoch 26 | batch 420/510 [I 2022-10-09 01:25:35] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 01:25:35] ...trap/engines/engine.py.326: process: 0.32279 | load: 0.00020 [I 2022-10-09 01:25:38] ...trap/engines/engine.py.321: eval: epoch 26 | batch 430/510 [I 2022-10-09 01:25:38] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 01:25:38] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00019 [I 2022-10-09 01:25:41] ...trap/engines/engine.py.321: eval: epoch 26 | batch 440/510 [I 2022-10-09 01:25:41] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 01:25:41] ...trap/engines/engine.py.326: process: 0.31933 | load: 0.00021 [I 2022-10-09 01:25:44] ...trap/engines/engine.py.321: eval: epoch 26 | batch 450/510 [I 2022-10-09 01:25:44] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 01:25:44] ...trap/engines/engine.py.326: process: 0.32206 | load: 0.00020 [I 2022-10-09 01:25:47] ...trap/engines/engine.py.321: eval: epoch 26 | batch 460/510 [I 2022-10-09 01:25:47] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:16 [I 2022-10-09 01:25:47] ...trap/engines/engine.py.326: process: 0.32432 | load: 0.00020 [I 2022-10-09 01:25:51] ...trap/engines/engine.py.321: eval: epoch 26 | batch 470/510 [I 2022-10-09 01:25:51] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 01:25:51] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00019 [I 2022-10-09 01:25:54] ...trap/engines/engine.py.321: eval: epoch 26 | batch 480/510 [I 2022-10-09 01:25:54] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 01:25:54] ...trap/engines/engine.py.326: process: 0.32553 | load: 0.00020 [I 2022-10-09 01:25:57] ...trap/engines/engine.py.321: eval: epoch 26 | batch 490/510 [I 2022-10-09 01:25:57] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 01:25:57] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00020 [I 2022-10-09 01:26:00] ...trap/engines/engine.py.321: eval: epoch 26 | batch 500/510 [I 2022-10-09 01:26:00] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 01:26:00] ...trap/engines/engine.py.326: process: 0.31510 | load: 0.00021 [I 2022-10-09 01:26:03] ...trap/engines/engine.py.321: eval: epoch 26 | batch 510/510 [I 2022-10-09 01:26:03] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 01:26:03] ...trap/engines/engine.py.326: process: 0.31349 | load: 0.00016 [S 2022-10-09 01:26:04] ...trap/engines/engine.py.132: eval_epoch.epoch: 26 [I 2022-10-09 01:26:04] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.686300 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.894500 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935600 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.673400 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.891500 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931500 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018911 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007710 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004821 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014820 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006815 [S 2022-10-09 01:26:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004522 [I 2022-10-09 01:26:44] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 01:26:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 01:26:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 01:26:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 01:26:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 01:26:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 01:26:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 01:26:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 01:26:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 01:26:54] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 01:26:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 01:27:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 01:27:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 27 [I 2022-10-09 01:27:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 01:27:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 0/2304 [I 2022-10-09 01:27:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:53:28 [I 2022-10-09 01:27:05] ...trap/engines/engine.py.229: process: 2.95493 | load: 2.05410 [I 2022-10-09 01:27:05] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-09 01:27:05] ...optimizers/trijoint.py.151: total_norm: 12.227750 [I 2022-10-09 01:27:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 10/2304 [I 2022-10-09 01:27:14] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:48 [I 2022-10-09 01:27:14] ...trap/engines/engine.py.229: process: 0.83565 | load: 0.00581 [I 2022-10-09 01:27:14] ...trap/engines/engine.py.230: loss: 0.25399 [I 2022-10-09 01:27:14] ...optimizers/trijoint.py.151: total_norm: 19.476259 [I 2022-10-09 01:27:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 20/2304 [I 2022-10-09 01:27:22] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:06 [I 2022-10-09 01:27:22] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00534 [I 2022-10-09 01:27:22] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-09 01:27:22] ...optimizers/trijoint.py.151: total_norm: 12.445519 [I 2022-10-09 01:27:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 30/2304 [I 2022-10-09 01:27:31] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:48 [I 2022-10-09 01:27:31] ...trap/engines/engine.py.229: process: 0.81849 | load: 0.00380 [I 2022-10-09 01:27:31] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-09 01:27:31] ...optimizers/trijoint.py.151: total_norm: 16.179983 [I 2022-10-09 01:27:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 40/2304 [I 2022-10-09 01:27:39] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:56 [I 2022-10-09 01:27:39] ...trap/engines/engine.py.229: process: 0.84325 | load: 0.00442 [I 2022-10-09 01:27:39] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-09 01:27:39] ...optimizers/trijoint.py.151: total_norm: 16.655832 [I 2022-10-09 01:27:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 50/2304 [I 2022-10-09 01:27:48] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:45 [I 2022-10-09 01:27:48] ...trap/engines/engine.py.229: process: 0.83529 | load: 0.00411 [I 2022-10-09 01:27:48] ...trap/engines/engine.py.230: loss: 0.18156 [I 2022-10-09 01:27:48] ...optimizers/trijoint.py.151: total_norm: 19.363523 [I 2022-10-09 01:27:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 60/2304 [I 2022-10-09 01:27:56] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:29 [I 2022-10-09 01:27:56] ...trap/engines/engine.py.229: process: 0.83238 | load: 0.00392 [I 2022-10-09 01:27:56] ...trap/engines/engine.py.230: loss: 0.16471 [I 2022-10-09 01:27:56] ...optimizers/trijoint.py.151: total_norm: 25.609692 [I 2022-10-09 01:28:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 70/2304 [I 2022-10-09 01:28:04] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:18 [I 2022-10-09 01:28:04] ...trap/engines/engine.py.229: process: 0.82922 | load: 0.00405 [I 2022-10-09 01:28:04] ...trap/engines/engine.py.230: loss: 0.10736 [I 2022-10-09 01:28:04] ...optimizers/trijoint.py.151: total_norm: 10.484044 [I 2022-10-09 01:28:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 80/2304 [I 2022-10-09 01:28:13] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:18 [I 2022-10-09 01:28:13] ...trap/engines/engine.py.229: process: 0.83556 | load: 0.00391 [I 2022-10-09 01:28:13] ...trap/engines/engine.py.230: loss: 0.20458 [I 2022-10-09 01:28:13] ...optimizers/trijoint.py.151: total_norm: 22.822920 [I 2022-10-09 01:28:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 90/2304 [I 2022-10-09 01:28:21] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:11 [I 2022-10-09 01:28:21] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00319 [I 2022-10-09 01:28:21] ...trap/engines/engine.py.230: loss: 0.18945 [I 2022-10-09 01:28:21] ...optimizers/trijoint.py.151: total_norm: 12.740309 [I 2022-10-09 01:28:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 100/2304 [I 2022-10-09 01:28:30] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:05 [I 2022-10-09 01:28:30] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00385 [I 2022-10-09 01:28:30] ...trap/engines/engine.py.230: loss: 0.17114 [I 2022-10-09 01:28:30] ...optimizers/trijoint.py.151: total_norm: 22.802273 [I 2022-10-09 01:28:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 110/2304 [I 2022-10-09 01:28:38] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:51 [I 2022-10-09 01:28:38] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00337 [I 2022-10-09 01:28:38] ...trap/engines/engine.py.230: loss: 0.17692 [I 2022-10-09 01:28:38] ...optimizers/trijoint.py.151: total_norm: 31.499535 [I 2022-10-09 01:28:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 120/2304 [I 2022-10-09 01:28:47] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:47 [I 2022-10-09 01:28:47] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00311 [I 2022-10-09 01:28:47] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-09 01:28:47] ...optimizers/trijoint.py.151: total_norm: 14.163693 [I 2022-10-09 01:28:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 130/2304 [I 2022-10-09 01:28:55] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:38 [I 2022-10-09 01:28:55] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00329 [I 2022-10-09 01:28:55] ...trap/engines/engine.py.230: loss: 0.21799 [I 2022-10-09 01:28:55] ...optimizers/trijoint.py.151: total_norm: 23.218891 [I 2022-10-09 01:29:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 140/2304 [I 2022-10-09 01:29:04] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:34 [I 2022-10-09 01:29:04] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00317 [I 2022-10-09 01:29:04] ...trap/engines/engine.py.230: loss: 0.12280 [I 2022-10-09 01:29:04] ...optimizers/trijoint.py.151: total_norm: 12.875026 [I 2022-10-09 01:29:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 150/2304 [I 2022-10-09 01:29:12] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:22 [I 2022-10-09 01:29:12] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00320 [I 2022-10-09 01:29:12] ...trap/engines/engine.py.230: loss: 0.20899 [I 2022-10-09 01:29:12] ...optimizers/trijoint.py.151: total_norm: 18.343599 [I 2022-10-09 01:29:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 160/2304 [I 2022-10-09 01:29:21] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:08 [I 2022-10-09 01:29:21] ...trap/engines/engine.py.229: process: 0.83348 | load: 0.00309 [I 2022-10-09 01:29:21] ...trap/engines/engine.py.230: loss: 0.16822 [I 2022-10-09 01:29:21] ...optimizers/trijoint.py.151: total_norm: 12.006900 [I 2022-10-09 01:29:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 170/2304 [I 2022-10-09 01:29:29] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:59 [I 2022-10-09 01:29:29] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00314 [I 2022-10-09 01:29:29] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-09 01:29:29] ...optimizers/trijoint.py.151: total_norm: 17.884594 [I 2022-10-09 01:29:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 180/2304 [I 2022-10-09 01:29:38] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:50 [I 2022-10-09 01:29:38] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00320 [I 2022-10-09 01:29:38] ...trap/engines/engine.py.230: loss: 0.21578 [I 2022-10-09 01:29:38] ...optimizers/trijoint.py.151: total_norm: 17.973930 [I 2022-10-09 01:29:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 190/2304 [I 2022-10-09 01:29:46] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:45 [I 2022-10-09 01:29:46] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00297 [I 2022-10-09 01:29:46] ...trap/engines/engine.py.230: loss: 0.19892 [I 2022-10-09 01:29:46] ...optimizers/trijoint.py.151: total_norm: 18.741257 [I 2022-10-09 01:29:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 200/2304 [I 2022-10-09 01:29:54] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:27 [I 2022-10-09 01:29:54] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00332 [I 2022-10-09 01:29:54] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-09 01:29:54] ...optimizers/trijoint.py.151: total_norm: 12.414382 [I 2022-10-09 01:30:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 210/2304 [I 2022-10-09 01:30:03] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:17 [I 2022-10-09 01:30:03] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00300 [I 2022-10-09 01:30:03] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-09 01:30:03] ...optimizers/trijoint.py.151: total_norm: 10.278367 [I 2022-10-09 01:30:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 220/2304 [I 2022-10-09 01:30:11] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:15 [I 2022-10-09 01:30:11] ...trap/engines/engine.py.229: process: 0.83981 | load: 0.00299 [I 2022-10-09 01:30:11] ...trap/engines/engine.py.230: loss: 0.16043 [I 2022-10-09 01:30:11] ...optimizers/trijoint.py.151: total_norm: 26.099316 [I 2022-10-09 01:30:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 230/2304 [I 2022-10-09 01:30:20] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:10 [I 2022-10-09 01:30:20] ...trap/engines/engine.py.229: process: 0.85333 | load: 0.00302 [I 2022-10-09 01:30:20] ...trap/engines/engine.py.230: loss: 0.16182 [I 2022-10-09 01:30:20] ...optimizers/trijoint.py.151: total_norm: 19.471815 [I 2022-10-09 01:30:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 240/2304 [I 2022-10-09 01:30:28] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:05 [I 2022-10-09 01:30:28] ...trap/engines/engine.py.229: process: 0.85970 | load: 0.00316 [I 2022-10-09 01:30:28] ...trap/engines/engine.py.230: loss: 0.24116 [I 2022-10-09 01:30:28] ...optimizers/trijoint.py.151: total_norm: 18.916161 [I 2022-10-09 01:30:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 250/2304 [I 2022-10-09 01:30:37] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:41 [I 2022-10-09 01:30:37] ...trap/engines/engine.py.229: process: 0.82938 | load: 0.00300 [I 2022-10-09 01:30:37] ...trap/engines/engine.py.230: loss: 0.18714 [I 2022-10-09 01:30:37] ...optimizers/trijoint.py.151: total_norm: 19.993156 [I 2022-10-09 01:30:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 260/2304 [I 2022-10-09 01:30:45] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:30 [I 2022-10-09 01:30:45] ...trap/engines/engine.py.229: process: 0.83122 | load: 0.00301 [I 2022-10-09 01:30:45] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-09 01:30:45] ...optimizers/trijoint.py.151: total_norm: 19.873713 [I 2022-10-09 01:30:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 270/2304 [I 2022-10-09 01:30:53] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:24 [I 2022-10-09 01:30:53] ...trap/engines/engine.py.229: process: 0.83339 | load: 0.00301 [I 2022-10-09 01:30:53] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-09 01:30:53] ...optimizers/trijoint.py.151: total_norm: 18.862988 [I 2022-10-09 01:31:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 280/2304 [I 2022-10-09 01:31:02] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:27 [I 2022-10-09 01:31:02] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00299 [I 2022-10-09 01:31:02] ...trap/engines/engine.py.230: loss: 0.16069 [I 2022-10-09 01:31:02] ...optimizers/trijoint.py.151: total_norm: 18.520765 [I 2022-10-09 01:31:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 290/2304 [I 2022-10-09 01:31:10] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:18 [I 2022-10-09 01:31:10] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00310 [I 2022-10-09 01:31:10] ...trap/engines/engine.py.230: loss: 0.19487 [I 2022-10-09 01:31:10] ...optimizers/trijoint.py.151: total_norm: 17.722717 [I 2022-10-09 01:31:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 300/2304 [I 2022-10-09 01:31:19] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:03 [I 2022-10-09 01:31:19] ...trap/engines/engine.py.229: process: 0.83471 | load: 0.00310 [I 2022-10-09 01:31:19] ...trap/engines/engine.py.230: loss: 0.15836 [I 2022-10-09 01:31:19] ...optimizers/trijoint.py.151: total_norm: 28.116930 [I 2022-10-09 01:31:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 310/2304 [I 2022-10-09 01:31:27] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:59 [I 2022-10-09 01:31:27] ...trap/engines/engine.py.229: process: 0.83649 | load: 0.00310 [I 2022-10-09 01:31:27] ...trap/engines/engine.py.230: loss: 0.19236 [I 2022-10-09 01:31:27] ...optimizers/trijoint.py.151: total_norm: 22.276754 [I 2022-10-09 01:31:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 320/2304 [I 2022-10-09 01:31:36] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:44 [I 2022-10-09 01:31:36] ...trap/engines/engine.py.229: process: 0.83168 | load: 0.00318 [I 2022-10-09 01:31:36] ...trap/engines/engine.py.230: loss: 0.21312 [I 2022-10-09 01:31:36] ...optimizers/trijoint.py.151: total_norm: 23.911865 [I 2022-10-09 01:31:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 330/2304 [I 2022-10-09 01:31:44] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:26 [I 2022-10-09 01:31:44] ...trap/engines/engine.py.229: process: 0.82569 | load: 0.00302 [I 2022-10-09 01:31:44] ...trap/engines/engine.py.230: loss: 0.20938 [I 2022-10-09 01:31:44] ...optimizers/trijoint.py.151: total_norm: 18.782785 [I 2022-10-09 01:31:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 340/2304 [I 2022-10-09 01:31:52] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:39 [I 2022-10-09 01:31:52] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00301 [I 2022-10-09 01:31:52] ...trap/engines/engine.py.230: loss: 0.17772 [I 2022-10-09 01:31:52] ...optimizers/trijoint.py.151: total_norm: 17.721981 [I 2022-10-09 01:32:01] ...trap/engines/engine.py.224: train: epoch 27 | batch 350/2304 [I 2022-10-09 01:32:01] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:23 [I 2022-10-09 01:32:01] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00301 [I 2022-10-09 01:32:01] ...trap/engines/engine.py.230: loss: 0.19789 [I 2022-10-09 01:32:01] ...optimizers/trijoint.py.151: total_norm: 22.406813 [I 2022-10-09 01:32:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 360/2304 [I 2022-10-09 01:32:09] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:12 [I 2022-10-09 01:32:09] ...trap/engines/engine.py.229: process: 0.83321 | load: 0.00316 [I 2022-10-09 01:32:09] ...trap/engines/engine.py.230: loss: 0.18765 [I 2022-10-09 01:32:09] ...optimizers/trijoint.py.151: total_norm: 15.587775 [I 2022-10-09 01:32:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 370/2304 [I 2022-10-09 01:32:18] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:07 [I 2022-10-09 01:32:18] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00300 [I 2022-10-09 01:32:18] ...trap/engines/engine.py.230: loss: 0.17678 [I 2022-10-09 01:32:18] ...optimizers/trijoint.py.151: total_norm: 19.054386 [I 2022-10-09 01:32:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 380/2304 [I 2022-10-09 01:32:26] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:53 [I 2022-10-09 01:32:26] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00303 [I 2022-10-09 01:32:26] ...trap/engines/engine.py.230: loss: 0.16260 [I 2022-10-09 01:32:26] ...optimizers/trijoint.py.151: total_norm: 20.319088 [I 2022-10-09 01:32:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 390/2304 [I 2022-10-09 01:32:35] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:43 [I 2022-10-09 01:32:35] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00324 [I 2022-10-09 01:32:35] ...trap/engines/engine.py.230: loss: 0.20476 [I 2022-10-09 01:32:35] ...optimizers/trijoint.py.151: total_norm: 29.216846 [I 2022-10-09 01:32:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 400/2304 [I 2022-10-09 01:32:43] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:39 [I 2022-10-09 01:32:43] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00301 [I 2022-10-09 01:32:43] ...trap/engines/engine.py.230: loss: 0.20130 [I 2022-10-09 01:32:43] ...optimizers/trijoint.py.151: total_norm: 16.254255 [I 2022-10-09 01:32:51] ...trap/engines/engine.py.224: train: epoch 27 | batch 410/2304 [I 2022-10-09 01:32:51] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:32 [I 2022-10-09 01:32:51] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00299 [I 2022-10-09 01:32:51] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-09 01:32:51] ...optimizers/trijoint.py.151: total_norm: 23.838469 [I 2022-10-09 01:33:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 420/2304 [I 2022-10-09 01:33:00] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:28 [I 2022-10-09 01:33:00] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00305 [I 2022-10-09 01:33:00] ...trap/engines/engine.py.230: loss: 0.19051 [I 2022-10-09 01:33:00] ...optimizers/trijoint.py.151: total_norm: 15.651705 [I 2022-10-09 01:33:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 430/2304 [I 2022-10-09 01:33:08] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:05 [I 2022-10-09 01:33:08] ...trap/engines/engine.py.229: process: 0.83025 | load: 0.00298 [I 2022-10-09 01:33:08] ...trap/engines/engine.py.230: loss: 0.18086 [I 2022-10-09 01:33:08] ...optimizers/trijoint.py.151: total_norm: 18.496641 [I 2022-10-09 01:33:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 440/2304 [I 2022-10-09 01:33:17] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:07 [I 2022-10-09 01:33:17] ...trap/engines/engine.py.229: process: 0.84701 | load: 0.00310 [I 2022-10-09 01:33:17] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-09 01:33:17] ...optimizers/trijoint.py.151: total_norm: 12.981422 [I 2022-10-09 01:33:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 450/2304 [I 2022-10-09 01:33:25] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:54 [I 2022-10-09 01:33:25] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00302 [I 2022-10-09 01:33:25] ...trap/engines/engine.py.230: loss: 0.15245 [I 2022-10-09 01:33:25] ...optimizers/trijoint.py.151: total_norm: 12.039911 [I 2022-10-09 01:33:33] ...trap/engines/engine.py.224: train: epoch 27 | batch 460/2304 [I 2022-10-09 01:33:33] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:25:52 [I 2022-10-09 01:33:33] ...trap/engines/engine.py.229: process: 0.83872 | load: 0.00302 [I 2022-10-09 01:33:33] ...trap/engines/engine.py.230: loss: 0.21857 [I 2022-10-09 01:33:33] ...optimizers/trijoint.py.151: total_norm: 18.359734 [I 2022-10-09 01:33:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 470/2304 [I 2022-10-09 01:33:42] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:41 [I 2022-10-09 01:33:42] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00297 [I 2022-10-09 01:33:42] ...trap/engines/engine.py.230: loss: 0.15845 [I 2022-10-09 01:33:42] ...optimizers/trijoint.py.151: total_norm: 19.642637 [I 2022-10-09 01:33:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 480/2304 [I 2022-10-09 01:33:50] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:27 [I 2022-10-09 01:33:50] ...trap/engines/engine.py.229: process: 0.82779 | load: 0.00310 [I 2022-10-09 01:33:50] ...trap/engines/engine.py.230: loss: 0.18285 [I 2022-10-09 01:33:50] ...optimizers/trijoint.py.151: total_norm: 11.625235 [I 2022-10-09 01:33:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 490/2304 [I 2022-10-09 01:33:59] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:26 [I 2022-10-09 01:33:59] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00310 [I 2022-10-09 01:33:59] ...trap/engines/engine.py.230: loss: 0.24934 [I 2022-10-09 01:33:59] ...optimizers/trijoint.py.151: total_norm: 9.460862 [I 2022-10-09 01:34:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 500/2304 [I 2022-10-09 01:34:07] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:04 [I 2022-10-09 01:34:07] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00303 [I 2022-10-09 01:34:07] ...trap/engines/engine.py.230: loss: 0.17075 [I 2022-10-09 01:34:07] ...optimizers/trijoint.py.151: total_norm: 17.928543 [I 2022-10-09 01:34:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 510/2304 [I 2022-10-09 01:34:16] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:06 [I 2022-10-09 01:34:16] ...trap/engines/engine.py.229: process: 0.83955 | load: 0.00304 [I 2022-10-09 01:34:16] ...trap/engines/engine.py.230: loss: 0.14934 [I 2022-10-09 01:34:16] ...optimizers/trijoint.py.151: total_norm: 15.151526 [I 2022-10-09 01:34:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 520/2304 [I 2022-10-09 01:34:24] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:24:58 [I 2022-10-09 01:34:24] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00298 [I 2022-10-09 01:34:24] ...trap/engines/engine.py.230: loss: 0.23224 [I 2022-10-09 01:34:24] ...optimizers/trijoint.py.151: total_norm: 12.943305 [I 2022-10-09 01:34:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 530/2304 [I 2022-10-09 01:34:32] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:44 [I 2022-10-09 01:34:32] ...trap/engines/engine.py.229: process: 0.82708 | load: 0.00303 [I 2022-10-09 01:34:32] ...trap/engines/engine.py.230: loss: 0.11962 [I 2022-10-09 01:34:32] ...optimizers/trijoint.py.151: total_norm: 17.651747 [I 2022-10-09 01:34:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 540/2304 [I 2022-10-09 01:34:41] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:43 [I 2022-10-09 01:34:41] ...trap/engines/engine.py.229: process: 0.84370 | load: 0.00299 [I 2022-10-09 01:34:41] ...trap/engines/engine.py.230: loss: 0.12479 [I 2022-10-09 01:34:41] ...optimizers/trijoint.py.151: total_norm: 16.536860 [I 2022-10-09 01:34:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 550/2304 [I 2022-10-09 01:34:49] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:28 [I 2022-10-09 01:34:49] ...trap/engines/engine.py.229: process: 0.83233 | load: 0.00298 [I 2022-10-09 01:34:49] ...trap/engines/engine.py.230: loss: 0.18432 [I 2022-10-09 01:34:49] ...optimizers/trijoint.py.151: total_norm: 28.011389 [I 2022-10-09 01:34:58] ...trap/engines/engine.py.224: train: epoch 27 | batch 560/2304 [I 2022-10-09 01:34:58] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:23 [I 2022-10-09 01:34:58] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00300 [I 2022-10-09 01:34:58] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-09 01:34:58] ...optimizers/trijoint.py.151: total_norm: 42.002186 [I 2022-10-09 01:35:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 570/2304 [I 2022-10-09 01:35:06] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:25 [I 2022-10-09 01:35:06] ...trap/engines/engine.py.229: process: 0.83334 | load: 0.00299 [I 2022-10-09 01:35:06] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-09 01:35:06] ...optimizers/trijoint.py.151: total_norm: 13.563700 [I 2022-10-09 01:35:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 580/2304 [I 2022-10-09 01:35:14] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:08 [I 2022-10-09 01:35:14] ...trap/engines/engine.py.229: process: 0.83525 | load: 0.00298 [I 2022-10-09 01:35:14] ...trap/engines/engine.py.230: loss: 0.17082 [I 2022-10-09 01:35:14] ...optimizers/trijoint.py.151: total_norm: 23.115410 [I 2022-10-09 01:35:23] ...trap/engines/engine.py.224: train: epoch 27 | batch 590/2304 [I 2022-10-09 01:35:23] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:03 [I 2022-10-09 01:35:23] ...trap/engines/engine.py.229: process: 0.84241 | load: 0.00299 [I 2022-10-09 01:35:23] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-09 01:35:23] ...optimizers/trijoint.py.151: total_norm: 21.756426 [I 2022-10-09 01:35:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 600/2304 [I 2022-10-09 01:35:31] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:57 [I 2022-10-09 01:35:31] ...trap/engines/engine.py.229: process: 0.83961 | load: 0.00298 [I 2022-10-09 01:35:31] ...trap/engines/engine.py.230: loss: 0.16431 [I 2022-10-09 01:35:31] ...optimizers/trijoint.py.151: total_norm: 21.452410 [I 2022-10-09 01:35:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 610/2304 [I 2022-10-09 01:35:40] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:45 [I 2022-10-09 01:35:40] ...trap/engines/engine.py.229: process: 0.84168 | load: 0.00303 [I 2022-10-09 01:35:40] ...trap/engines/engine.py.230: loss: 0.19890 [I 2022-10-09 01:35:40] ...optimizers/trijoint.py.151: total_norm: 29.735586 [I 2022-10-09 01:35:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 620/2304 [I 2022-10-09 01:35:48] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:39 [I 2022-10-09 01:35:48] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00311 [I 2022-10-09 01:35:48] ...trap/engines/engine.py.230: loss: 0.28156 [I 2022-10-09 01:35:48] ...optimizers/trijoint.py.151: total_norm: 29.700083 [I 2022-10-09 01:35:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 630/2304 [I 2022-10-09 01:35:57] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:27 [I 2022-10-09 01:35:57] ...trap/engines/engine.py.229: process: 0.84323 | load: 0.00300 [I 2022-10-09 01:35:57] ...trap/engines/engine.py.230: loss: 0.10771 [I 2022-10-09 01:35:57] ...optimizers/trijoint.py.151: total_norm: 12.831021 [I 2022-10-09 01:36:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 640/2304 [I 2022-10-09 01:36:05] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:19 [I 2022-10-09 01:36:05] ...trap/engines/engine.py.229: process: 0.84581 | load: 0.00305 [I 2022-10-09 01:36:05] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-09 01:36:05] ...optimizers/trijoint.py.151: total_norm: 18.610794 [I 2022-10-09 01:36:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 650/2304 [I 2022-10-09 01:36:14] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:22 [I 2022-10-09 01:36:14] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00301 [I 2022-10-09 01:36:14] ...trap/engines/engine.py.230: loss: 0.10281 [I 2022-10-09 01:36:14] ...optimizers/trijoint.py.151: total_norm: 13.879214 [I 2022-10-09 01:36:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 660/2304 [I 2022-10-09 01:36:22] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:04 [I 2022-10-09 01:36:22] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00298 [I 2022-10-09 01:36:22] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-09 01:36:22] ...optimizers/trijoint.py.151: total_norm: 14.016940 [I 2022-10-09 01:36:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 670/2304 [I 2022-10-09 01:36:30] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:23:04 [I 2022-10-09 01:36:30] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00300 [I 2022-10-09 01:36:30] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-09 01:36:30] ...optimizers/trijoint.py.151: total_norm: 11.954357 [I 2022-10-09 01:36:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 680/2304 [I 2022-10-09 01:36:39] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:38 [I 2022-10-09 01:36:39] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00310 [I 2022-10-09 01:36:39] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-09 01:36:39] ...optimizers/trijoint.py.151: total_norm: 11.958302 [I 2022-10-09 01:36:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 690/2304 [I 2022-10-09 01:36:47] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:40 [I 2022-10-09 01:36:47] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00299 [I 2022-10-09 01:36:47] ...trap/engines/engine.py.230: loss: 0.21180 [I 2022-10-09 01:36:47] ...optimizers/trijoint.py.151: total_norm: 24.589397 [I 2022-10-09 01:36:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 700/2304 [I 2022-10-09 01:36:56] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:35 [I 2022-10-09 01:36:56] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00301 [I 2022-10-09 01:36:56] ...trap/engines/engine.py.230: loss: 0.21913 [I 2022-10-09 01:36:56] ...optimizers/trijoint.py.151: total_norm: 25.153824 [I 2022-10-09 01:37:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 710/2304 [I 2022-10-09 01:37:04] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:22 [I 2022-10-09 01:37:04] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00299 [I 2022-10-09 01:37:04] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-09 01:37:04] ...optimizers/trijoint.py.151: total_norm: 19.297218 [I 2022-10-09 01:37:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 720/2304 [I 2022-10-09 01:37:13] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:09 [I 2022-10-09 01:37:13] ...trap/engines/engine.py.229: process: 0.82619 | load: 0.00301 [I 2022-10-09 01:37:13] ...trap/engines/engine.py.230: loss: 0.15105 [I 2022-10-09 01:37:13] ...optimizers/trijoint.py.151: total_norm: 12.806754 [I 2022-10-09 01:37:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 730/2304 [I 2022-10-09 01:37:21] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:02 [I 2022-10-09 01:37:21] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00300 [I 2022-10-09 01:37:21] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-09 01:37:21] ...optimizers/trijoint.py.151: total_norm: 11.366595 [I 2022-10-09 01:37:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 740/2304 [I 2022-10-09 01:37:29] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:00 [I 2022-10-09 01:37:29] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00300 [I 2022-10-09 01:37:29] ...trap/engines/engine.py.230: loss: 0.16060 [I 2022-10-09 01:37:29] ...optimizers/trijoint.py.151: total_norm: 10.939479 [I 2022-10-09 01:37:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 750/2304 [I 2022-10-09 01:37:38] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:44 [I 2022-10-09 01:37:38] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00302 [I 2022-10-09 01:37:38] ...trap/engines/engine.py.230: loss: 0.15812 [I 2022-10-09 01:37:38] ...optimizers/trijoint.py.151: total_norm: 11.153119 [I 2022-10-09 01:37:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 760/2304 [I 2022-10-09 01:37:46] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:41 [I 2022-10-09 01:37:46] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00298 [I 2022-10-09 01:37:46] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-09 01:37:46] ...optimizers/trijoint.py.151: total_norm: 17.155319 [I 2022-10-09 01:37:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 770/2304 [I 2022-10-09 01:37:55] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:27 [I 2022-10-09 01:37:55] ...trap/engines/engine.py.229: process: 0.82055 | load: 0.00296 [I 2022-10-09 01:37:55] ...trap/engines/engine.py.230: loss: 0.19932 [I 2022-10-09 01:37:55] ...optimizers/trijoint.py.151: total_norm: 26.053083 [I 2022-10-09 01:38:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 780/2304 [I 2022-10-09 01:38:03] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:19 [I 2022-10-09 01:38:03] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00300 [I 2022-10-09 01:38:03] ...trap/engines/engine.py.230: loss: 0.17833 [I 2022-10-09 01:38:03] ...optimizers/trijoint.py.151: total_norm: 39.446217 [I 2022-10-09 01:38:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 790/2304 [I 2022-10-09 01:38:12] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:10 [I 2022-10-09 01:38:12] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00297 [I 2022-10-09 01:38:12] ...trap/engines/engine.py.230: loss: 0.15848 [I 2022-10-09 01:38:12] ...optimizers/trijoint.py.151: total_norm: 24.504019 [I 2022-10-09 01:38:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 800/2304 [I 2022-10-09 01:38:20] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:02 [I 2022-10-09 01:38:20] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00297 [I 2022-10-09 01:38:20] ...trap/engines/engine.py.230: loss: 0.18772 [I 2022-10-09 01:38:20] ...optimizers/trijoint.py.151: total_norm: 32.493008 [I 2022-10-09 01:38:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 810/2304 [I 2022-10-09 01:38:28] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:20:54 [I 2022-10-09 01:38:28] ...trap/engines/engine.py.229: process: 0.83406 | load: 0.00306 [I 2022-10-09 01:38:28] ...trap/engines/engine.py.230: loss: 0.20630 [I 2022-10-09 01:38:28] ...optimizers/trijoint.py.151: total_norm: 21.380308 [I 2022-10-09 01:38:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 820/2304 [I 2022-10-09 01:38:37] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:49 [I 2022-10-09 01:38:37] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00307 [I 2022-10-09 01:38:37] ...trap/engines/engine.py.230: loss: 0.23962 [I 2022-10-09 01:38:37] ...optimizers/trijoint.py.151: total_norm: 32.587582 [I 2022-10-09 01:38:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 830/2304 [I 2022-10-09 01:38:45] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:49 [I 2022-10-09 01:38:45] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00308 [I 2022-10-09 01:38:45] ...trap/engines/engine.py.230: loss: 0.23965 [I 2022-10-09 01:38:45] ...optimizers/trijoint.py.151: total_norm: 23.353180 [I 2022-10-09 01:38:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 840/2304 [I 2022-10-09 01:38:54] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:38 [I 2022-10-09 01:38:54] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00306 [I 2022-10-09 01:38:54] ...trap/engines/engine.py.230: loss: 0.21127 [I 2022-10-09 01:38:54] ...optimizers/trijoint.py.151: total_norm: 16.254547 [I 2022-10-09 01:39:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 850/2304 [I 2022-10-09 01:39:02] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:21 [I 2022-10-09 01:39:02] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00305 [I 2022-10-09 01:39:02] ...trap/engines/engine.py.230: loss: 0.17234 [I 2022-10-09 01:39:02] ...optimizers/trijoint.py.151: total_norm: 20.691895 [I 2022-10-09 01:39:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 860/2304 [I 2022-10-09 01:39:11] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:12 [I 2022-10-09 01:39:11] ...trap/engines/engine.py.229: process: 0.84759 | load: 0.00306 [I 2022-10-09 01:39:11] ...trap/engines/engine.py.230: loss: 0.18651 [I 2022-10-09 01:39:11] ...optimizers/trijoint.py.151: total_norm: 20.473940 [I 2022-10-09 01:39:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 870/2304 [I 2022-10-09 01:39:19] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:05 [I 2022-10-09 01:39:19] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00307 [I 2022-10-09 01:39:19] ...trap/engines/engine.py.230: loss: 0.18687 [I 2022-10-09 01:39:19] ...optimizers/trijoint.py.151: total_norm: 17.972660 [I 2022-10-09 01:39:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 880/2304 [I 2022-10-09 01:39:27] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:19:51 [I 2022-10-09 01:39:27] ...trap/engines/engine.py.229: process: 0.82969 | load: 0.00310 [I 2022-10-09 01:39:27] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-09 01:39:27] ...optimizers/trijoint.py.151: total_norm: 14.207630 [I 2022-10-09 01:39:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 890/2304 [I 2022-10-09 01:39:36] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:53 [I 2022-10-09 01:39:36] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00306 [I 2022-10-09 01:39:36] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-09 01:39:36] ...optimizers/trijoint.py.151: total_norm: 7.764513 [I 2022-10-09 01:39:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 900/2304 [I 2022-10-09 01:39:44] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:44 [I 2022-10-09 01:39:44] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00306 [I 2022-10-09 01:39:44] ...trap/engines/engine.py.230: loss: 0.23642 [I 2022-10-09 01:39:44] ...optimizers/trijoint.py.151: total_norm: 14.727515 [I 2022-10-09 01:39:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 910/2304 [I 2022-10-09 01:39:53] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:37 [I 2022-10-09 01:39:53] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00308 [I 2022-10-09 01:39:53] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-09 01:39:53] ...optimizers/trijoint.py.151: total_norm: 19.908443 [I 2022-10-09 01:40:01] ...trap/engines/engine.py.224: train: epoch 27 | batch 920/2304 [I 2022-10-09 01:40:01] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:19 [I 2022-10-09 01:40:01] ...trap/engines/engine.py.229: process: 0.83406 | load: 0.00312 [I 2022-10-09 01:40:01] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-09 01:40:01] ...optimizers/trijoint.py.151: total_norm: 12.075072 [I 2022-10-09 01:40:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 930/2304 [I 2022-10-09 01:40:10] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:18 [I 2022-10-09 01:40:10] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00310 [I 2022-10-09 01:40:10] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-09 01:40:10] ...optimizers/trijoint.py.151: total_norm: 11.685679 [I 2022-10-09 01:40:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 940/2304 [I 2022-10-09 01:40:18] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:05 [I 2022-10-09 01:40:18] ...trap/engines/engine.py.229: process: 0.83969 | load: 0.00306 [I 2022-10-09 01:40:18] ...trap/engines/engine.py.230: loss: 0.17678 [I 2022-10-09 01:40:18] ...optimizers/trijoint.py.151: total_norm: 14.336838 [I 2022-10-09 01:40:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 950/2304 [I 2022-10-09 01:40:27] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:01 [I 2022-10-09 01:40:27] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00308 [I 2022-10-09 01:40:27] ...trap/engines/engine.py.230: loss: 0.11494 [I 2022-10-09 01:40:27] ...optimizers/trijoint.py.151: total_norm: 18.086180 [I 2022-10-09 01:40:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 960/2304 [I 2022-10-09 01:40:35] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:18:46 [I 2022-10-09 01:40:35] ...trap/engines/engine.py.229: process: 0.83340 | load: 0.00305 [I 2022-10-09 01:40:35] ...trap/engines/engine.py.230: loss: 0.16387 [I 2022-10-09 01:40:35] ...optimizers/trijoint.py.151: total_norm: 19.706121 [I 2022-10-09 01:40:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 970/2304 [I 2022-10-09 01:40:43] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:41 [I 2022-10-09 01:40:43] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00304 [I 2022-10-09 01:40:43] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-09 01:40:43] ...optimizers/trijoint.py.151: total_norm: 13.716223 [I 2022-10-09 01:40:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 980/2304 [I 2022-10-09 01:40:52] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:34 [I 2022-10-09 01:40:52] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00306 [I 2022-10-09 01:40:52] ...trap/engines/engine.py.230: loss: 0.14591 [I 2022-10-09 01:40:52] ...optimizers/trijoint.py.151: total_norm: 12.462582 [I 2022-10-09 01:41:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 990/2304 [I 2022-10-09 01:41:00] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:28 [I 2022-10-09 01:41:00] ...trap/engines/engine.py.229: process: 0.83792 | load: 0.00309 [I 2022-10-09 01:41:00] ...trap/engines/engine.py.230: loss: 0.18862 [I 2022-10-09 01:41:00] ...optimizers/trijoint.py.151: total_norm: 29.670950 [I 2022-10-09 01:41:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 1000/2304 [I 2022-10-09 01:41:09] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:19 [I 2022-10-09 01:41:09] ...trap/engines/engine.py.229: process: 0.83715 | load: 0.00313 [I 2022-10-09 01:41:09] ...trap/engines/engine.py.230: loss: 0.15292 [I 2022-10-09 01:41:09] ...optimizers/trijoint.py.151: total_norm: 22.560089 [I 2022-10-09 01:41:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 1010/2304 [I 2022-10-09 01:41:17] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:13 [I 2022-10-09 01:41:17] ...trap/engines/engine.py.229: process: 0.85287 | load: 0.00299 [I 2022-10-09 01:41:17] ...trap/engines/engine.py.230: loss: 0.18860 [I 2022-10-09 01:41:17] ...optimizers/trijoint.py.151: total_norm: 19.140604 [I 2022-10-09 01:41:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 1020/2304 [I 2022-10-09 01:41:26] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:03 [I 2022-10-09 01:41:26] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00300 [I 2022-10-09 01:41:26] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-09 01:41:26] ...optimizers/trijoint.py.151: total_norm: 19.262964 [I 2022-10-09 01:41:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 1030/2304 [I 2022-10-09 01:41:34] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:17:53 [I 2022-10-09 01:41:34] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00302 [I 2022-10-09 01:41:34] ...trap/engines/engine.py.230: loss: 0.20731 [I 2022-10-09 01:41:34] ...optimizers/trijoint.py.151: total_norm: 12.930670 [I 2022-10-09 01:41:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 1040/2304 [I 2022-10-09 01:41:43] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:44 [I 2022-10-09 01:41:43] ...trap/engines/engine.py.229: process: 0.84488 | load: 0.00301 [I 2022-10-09 01:41:43] ...trap/engines/engine.py.230: loss: 0.15605 [I 2022-10-09 01:41:43] ...optimizers/trijoint.py.151: total_norm: 14.096328 [I 2022-10-09 01:41:51] ...trap/engines/engine.py.224: train: epoch 27 | batch 1050/2304 [I 2022-10-09 01:41:51] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:34 [I 2022-10-09 01:41:51] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00309 [I 2022-10-09 01:41:51] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-09 01:41:51] ...optimizers/trijoint.py.151: total_norm: 11.874932 [I 2022-10-09 01:41:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 1060/2304 [I 2022-10-09 01:41:59] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:27 [I 2022-10-09 01:41:59] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00302 [I 2022-10-09 01:41:59] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-09 01:41:59] ...optimizers/trijoint.py.151: total_norm: 16.557899 [I 2022-10-09 01:42:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 1070/2304 [I 2022-10-09 01:42:08] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:22 [I 2022-10-09 01:42:08] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00300 [I 2022-10-09 01:42:08] ...trap/engines/engine.py.230: loss: 0.13715 [I 2022-10-09 01:42:08] ...optimizers/trijoint.py.151: total_norm: 16.224852 [I 2022-10-09 01:42:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 1080/2304 [I 2022-10-09 01:42:16] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:07 [I 2022-10-09 01:42:16] ...trap/engines/engine.py.229: process: 0.83560 | load: 0.00306 [I 2022-10-09 01:42:16] ...trap/engines/engine.py.230: loss: 0.16252 [I 2022-10-09 01:42:16] ...optimizers/trijoint.py.151: total_norm: 24.433073 [I 2022-10-09 01:42:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 1090/2304 [I 2022-10-09 01:42:25] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:00 [I 2022-10-09 01:42:25] ...trap/engines/engine.py.229: process: 0.83581 | load: 0.00299 [I 2022-10-09 01:42:25] ...trap/engines/engine.py.230: loss: 0.21650 [I 2022-10-09 01:42:25] ...optimizers/trijoint.py.151: total_norm: 18.871735 [I 2022-10-09 01:42:33] ...trap/engines/engine.py.224: train: epoch 27 | batch 1100/2304 [I 2022-10-09 01:42:33] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:16:47 [I 2022-10-09 01:42:33] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00304 [I 2022-10-09 01:42:33] ...trap/engines/engine.py.230: loss: 0.14814 [I 2022-10-09 01:42:33] ...optimizers/trijoint.py.151: total_norm: 22.678640 [I 2022-10-09 01:42:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 1110/2304 [I 2022-10-09 01:42:41] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:38 [I 2022-10-09 01:42:41] ...trap/engines/engine.py.229: process: 0.83739 | load: 0.00298 [I 2022-10-09 01:42:41] ...trap/engines/engine.py.230: loss: 0.10936 [I 2022-10-09 01:42:41] ...optimizers/trijoint.py.151: total_norm: 12.677070 [I 2022-10-09 01:42:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 1120/2304 [I 2022-10-09 01:42:50] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:33 [I 2022-10-09 01:42:50] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00301 [I 2022-10-09 01:42:50] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-09 01:42:50] ...optimizers/trijoint.py.151: total_norm: 14.496470 [I 2022-10-09 01:42:58] ...trap/engines/engine.py.224: train: epoch 27 | batch 1130/2304 [I 2022-10-09 01:42:58] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:26 [I 2022-10-09 01:42:58] ...trap/engines/engine.py.229: process: 0.83661 | load: 0.00305 [I 2022-10-09 01:42:58] ...trap/engines/engine.py.230: loss: 0.17479 [I 2022-10-09 01:42:58] ...optimizers/trijoint.py.151: total_norm: 12.388610 [I 2022-10-09 01:43:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 1140/2304 [I 2022-10-09 01:43:07] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:13 [I 2022-10-09 01:43:07] ...trap/engines/engine.py.229: process: 0.82339 | load: 0.00300 [I 2022-10-09 01:43:07] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-09 01:43:07] ...optimizers/trijoint.py.151: total_norm: 20.518305 [I 2022-10-09 01:43:15] ...trap/engines/engine.py.224: train: epoch 27 | batch 1150/2304 [I 2022-10-09 01:43:15] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:02 [I 2022-10-09 01:43:15] ...trap/engines/engine.py.229: process: 0.83401 | load: 0.00300 [I 2022-10-09 01:43:15] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-09 01:43:15] ...optimizers/trijoint.py.151: total_norm: 13.632325 [I 2022-10-09 01:43:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 1160/2304 [I 2022-10-09 01:43:24] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:01 [I 2022-10-09 01:43:24] ...trap/engines/engine.py.229: process: 0.83576 | load: 0.00303 [I 2022-10-09 01:43:24] ...trap/engines/engine.py.230: loss: 0.14678 [I 2022-10-09 01:43:24] ...optimizers/trijoint.py.151: total_norm: 11.931086 [I 2022-10-09 01:43:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 1170/2304 [I 2022-10-09 01:43:32] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:15:55 [I 2022-10-09 01:43:32] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00299 [I 2022-10-09 01:43:32] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-09 01:43:32] ...optimizers/trijoint.py.151: total_norm: 29.269552 [I 2022-10-09 01:43:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 1180/2304 [I 2022-10-09 01:43:40] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:15:41 [I 2022-10-09 01:43:40] ...trap/engines/engine.py.229: process: 0.82935 | load: 0.00297 [I 2022-10-09 01:43:40] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-09 01:43:40] ...optimizers/trijoint.py.151: total_norm: 12.058922 [I 2022-10-09 01:43:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 1190/2304 [I 2022-10-09 01:43:49] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:37 [I 2022-10-09 01:43:49] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00297 [I 2022-10-09 01:43:49] ...trap/engines/engine.py.230: loss: 0.25503 [I 2022-10-09 01:43:49] ...optimizers/trijoint.py.151: total_norm: 14.549168 [I 2022-10-09 01:43:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 1200/2304 [I 2022-10-09 01:43:57] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:15:22 [I 2022-10-09 01:43:57] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00300 [I 2022-10-09 01:43:57] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-09 01:43:57] ...optimizers/trijoint.py.151: total_norm: 12.126143 [I 2022-10-09 01:44:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 1210/2304 [I 2022-10-09 01:44:06] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:18 [I 2022-10-09 01:44:06] ...trap/engines/engine.py.229: process: 0.82816 | load: 0.00301 [I 2022-10-09 01:44:06] ...trap/engines/engine.py.230: loss: 0.12541 [I 2022-10-09 01:44:06] ...optimizers/trijoint.py.151: total_norm: 13.952631 [I 2022-10-09 01:44:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 1220/2304 [I 2022-10-09 01:44:14] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:11 [I 2022-10-09 01:44:14] ...trap/engines/engine.py.229: process: 0.83459 | load: 0.00298 [I 2022-10-09 01:44:14] ...trap/engines/engine.py.230: loss: 0.20433 [I 2022-10-09 01:44:14] ...optimizers/trijoint.py.151: total_norm: 21.931690 [I 2022-10-09 01:44:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 1230/2304 [I 2022-10-09 01:44:22] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:04 [I 2022-10-09 01:44:22] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00301 [I 2022-10-09 01:44:22] ...trap/engines/engine.py.230: loss: 0.15057 [I 2022-10-09 01:44:22] ...optimizers/trijoint.py.151: total_norm: 10.125684 [I 2022-10-09 01:44:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 1240/2304 [I 2022-10-09 01:44:31] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:14:55 [I 2022-10-09 01:44:31] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00296 [I 2022-10-09 01:44:31] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-09 01:44:31] ...optimizers/trijoint.py.151: total_norm: 12.682420 [I 2022-10-09 01:44:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 1250/2304 [I 2022-10-09 01:44:39] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:14:44 [I 2022-10-09 01:44:39] ...trap/engines/engine.py.229: process: 0.83373 | load: 0.00303 [I 2022-10-09 01:44:39] ...trap/engines/engine.py.230: loss: 0.20643 [I 2022-10-09 01:44:39] ...optimizers/trijoint.py.151: total_norm: 27.620396 [I 2022-10-09 01:44:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 1260/2304 [I 2022-10-09 01:44:48] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:38 [I 2022-10-09 01:44:48] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00301 [I 2022-10-09 01:44:48] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-09 01:44:48] ...optimizers/trijoint.py.151: total_norm: 29.916561 [I 2022-10-09 01:44:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 1270/2304 [I 2022-10-09 01:44:56] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:28 [I 2022-10-09 01:44:56] ...trap/engines/engine.py.229: process: 0.81720 | load: 0.00305 [I 2022-10-09 01:44:56] ...trap/engines/engine.py.230: loss: 0.23981 [I 2022-10-09 01:44:56] ...optimizers/trijoint.py.151: total_norm: 41.607754 [I 2022-10-09 01:45:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 1280/2304 [I 2022-10-09 01:45:05] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:29 [I 2022-10-09 01:45:05] ...trap/engines/engine.py.229: process: 0.86807 | load: 0.00302 [I 2022-10-09 01:45:05] ...trap/engines/engine.py.230: loss: 0.10979 [I 2022-10-09 01:45:05] ...optimizers/trijoint.py.151: total_norm: 9.344872 [I 2022-10-09 01:45:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 1290/2304 [I 2022-10-09 01:45:13] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:11 [I 2022-10-09 01:45:13] ...trap/engines/engine.py.229: process: 0.83177 | load: 0.00301 [I 2022-10-09 01:45:13] ...trap/engines/engine.py.230: loss: 0.17746 [I 2022-10-09 01:45:13] ...optimizers/trijoint.py.151: total_norm: 13.763504 [I 2022-10-09 01:45:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 1300/2304 [I 2022-10-09 01:45:22] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:06 [I 2022-10-09 01:45:22] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00303 [I 2022-10-09 01:45:22] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-09 01:45:22] ...optimizers/trijoint.py.151: total_norm: 17.487959 [I 2022-10-09 01:45:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 1310/2304 [I 2022-10-09 01:45:30] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:13:58 [I 2022-10-09 01:45:30] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00299 [I 2022-10-09 01:45:30] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-09 01:45:30] ...optimizers/trijoint.py.151: total_norm: 18.487268 [I 2022-10-09 01:45:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 1320/2304 [I 2022-10-09 01:45:38] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:13:49 [I 2022-10-09 01:45:38] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00300 [I 2022-10-09 01:45:38] ...trap/engines/engine.py.230: loss: 0.22523 [I 2022-10-09 01:45:38] ...optimizers/trijoint.py.151: total_norm: 23.249842 [I 2022-10-09 01:45:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 1330/2304 [I 2022-10-09 01:45:47] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:41 [I 2022-10-09 01:45:47] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00302 [I 2022-10-09 01:45:47] ...trap/engines/engine.py.230: loss: 0.28781 [I 2022-10-09 01:45:47] ...optimizers/trijoint.py.151: total_norm: 24.378481 [I 2022-10-09 01:45:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 1340/2304 [I 2022-10-09 01:45:55] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:25 [I 2022-10-09 01:45:55] ...trap/engines/engine.py.229: process: 0.83173 | load: 0.00301 [I 2022-10-09 01:45:55] ...trap/engines/engine.py.230: loss: 0.19871 [I 2022-10-09 01:45:55] ...optimizers/trijoint.py.151: total_norm: 23.983061 [I 2022-10-09 01:46:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 1350/2304 [I 2022-10-09 01:46:04] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:21 [I 2022-10-09 01:46:04] ...trap/engines/engine.py.229: process: 0.82995 | load: 0.00297 [I 2022-10-09 01:46:04] ...trap/engines/engine.py.230: loss: 0.14014 [I 2022-10-09 01:46:04] ...optimizers/trijoint.py.151: total_norm: 12.887117 [I 2022-10-09 01:46:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 1360/2304 [I 2022-10-09 01:46:12] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:14 [I 2022-10-09 01:46:12] ...trap/engines/engine.py.229: process: 0.83731 | load: 0.00299 [I 2022-10-09 01:46:12] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-09 01:46:12] ...optimizers/trijoint.py.151: total_norm: 14.895697 [I 2022-10-09 01:46:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 1370/2304 [I 2022-10-09 01:46:20] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:02 [I 2022-10-09 01:46:20] ...trap/engines/engine.py.229: process: 0.82733 | load: 0.00301 [I 2022-10-09 01:46:20] ...trap/engines/engine.py.230: loss: 0.13045 [I 2022-10-09 01:46:20] ...optimizers/trijoint.py.151: total_norm: 21.141586 [I 2022-10-09 01:46:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 1380/2304 [I 2022-10-09 01:46:29] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:12:54 [I 2022-10-09 01:46:29] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00298 [I 2022-10-09 01:46:29] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-09 01:46:29] ...optimizers/trijoint.py.151: total_norm: 21.297585 [I 2022-10-09 01:46:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 1390/2304 [I 2022-10-09 01:46:37] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:12:47 [I 2022-10-09 01:46:37] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00299 [I 2022-10-09 01:46:37] ...trap/engines/engine.py.230: loss: 0.12648 [I 2022-10-09 01:46:37] ...optimizers/trijoint.py.151: total_norm: 18.138620 [I 2022-10-09 01:46:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 1400/2304 [I 2022-10-09 01:46:46] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:12:41 [I 2022-10-09 01:46:46] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00299 [I 2022-10-09 01:46:46] ...trap/engines/engine.py.230: loss: 0.15352 [I 2022-10-09 01:46:46] ...optimizers/trijoint.py.151: total_norm: 11.258474 [I 2022-10-09 01:46:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 1410/2304 [I 2022-10-09 01:46:54] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:12:33 [I 2022-10-09 01:46:54] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00300 [I 2022-10-09 01:46:54] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-09 01:46:54] ...optimizers/trijoint.py.151: total_norm: 16.271290 [I 2022-10-09 01:47:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 1420/2304 [I 2022-10-09 01:47:03] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:21 [I 2022-10-09 01:47:03] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00299 [I 2022-10-09 01:47:03] ...trap/engines/engine.py.230: loss: 0.17135 [I 2022-10-09 01:47:03] ...optimizers/trijoint.py.151: total_norm: 18.229748 [I 2022-10-09 01:47:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 1430/2304 [I 2022-10-09 01:47:11] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:12 [I 2022-10-09 01:47:11] ...trap/engines/engine.py.229: process: 0.83491 | load: 0.00301 [I 2022-10-09 01:47:11] ...trap/engines/engine.py.230: loss: 0.17550 [I 2022-10-09 01:47:11] ...optimizers/trijoint.py.151: total_norm: 9.087998 [I 2022-10-09 01:47:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 1440/2304 [I 2022-10-09 01:47:19] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:06 [I 2022-10-09 01:47:19] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00299 [I 2022-10-09 01:47:19] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-09 01:47:19] ...optimizers/trijoint.py.151: total_norm: 15.477288 [I 2022-10-09 01:47:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 1450/2304 [I 2022-10-09 01:47:28] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:11:57 [I 2022-10-09 01:47:28] ...trap/engines/engine.py.229: process: 0.83391 | load: 0.00301 [I 2022-10-09 01:47:28] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-09 01:47:28] ...optimizers/trijoint.py.151: total_norm: 14.096128 [I 2022-10-09 01:47:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 1460/2304 [I 2022-10-09 01:47:36] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:11:47 [I 2022-10-09 01:47:36] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00299 [I 2022-10-09 01:47:36] ...trap/engines/engine.py.230: loss: 0.25632 [I 2022-10-09 01:47:36] ...optimizers/trijoint.py.151: total_norm: 21.292702 [I 2022-10-09 01:47:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 1470/2304 [I 2022-10-09 01:47:45] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:11:43 [I 2022-10-09 01:47:45] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00303 [I 2022-10-09 01:47:45] ...trap/engines/engine.py.230: loss: 0.21810 [I 2022-10-09 01:47:45] ...optimizers/trijoint.py.151: total_norm: 29.426790 [I 2022-10-09 01:47:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 1480/2304 [I 2022-10-09 01:47:53] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:33 [I 2022-10-09 01:47:53] ...trap/engines/engine.py.229: process: 0.83755 | load: 0.00301 [I 2022-10-09 01:47:53] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-09 01:47:53] ...optimizers/trijoint.py.151: total_norm: 15.752013 [I 2022-10-09 01:48:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 1490/2304 [I 2022-10-09 01:48:02] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:11:25 [I 2022-10-09 01:48:02] ...trap/engines/engine.py.229: process: 0.85059 | load: 0.00301 [I 2022-10-09 01:48:02] ...trap/engines/engine.py.230: loss: 0.18334 [I 2022-10-09 01:48:02] ...optimizers/trijoint.py.151: total_norm: 28.649319 [I 2022-10-09 01:48:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 1500/2304 [I 2022-10-09 01:48:10] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:11:16 [I 2022-10-09 01:48:10] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00304 [I 2022-10-09 01:48:10] ...trap/engines/engine.py.230: loss: 0.18103 [I 2022-10-09 01:48:10] ...optimizers/trijoint.py.151: total_norm: 43.285110 [I 2022-10-09 01:48:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 1510/2304 [I 2022-10-09 01:48:18] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:04 [I 2022-10-09 01:48:18] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00301 [I 2022-10-09 01:48:18] ...trap/engines/engine.py.230: loss: 0.09767 [I 2022-10-09 01:48:18] ...optimizers/trijoint.py.151: total_norm: 12.309018 [I 2022-10-09 01:48:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 1520/2304 [I 2022-10-09 01:48:27] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:10:56 [I 2022-10-09 01:48:27] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00306 [I 2022-10-09 01:48:27] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-09 01:48:27] ...optimizers/trijoint.py.151: total_norm: 23.437273 [I 2022-10-09 01:48:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 1530/2304 [I 2022-10-09 01:48:35] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:10:47 [I 2022-10-09 01:48:35] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00299 [I 2022-10-09 01:48:35] ...trap/engines/engine.py.230: loss: 0.17425 [I 2022-10-09 01:48:35] ...optimizers/trijoint.py.151: total_norm: 14.223217 [I 2022-10-09 01:48:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 1540/2304 [I 2022-10-09 01:48:44] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:10:42 [I 2022-10-09 01:48:44] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00300 [I 2022-10-09 01:48:44] ...trap/engines/engine.py.230: loss: 0.16525 [I 2022-10-09 01:48:44] ...optimizers/trijoint.py.151: total_norm: 12.418089 [I 2022-10-09 01:48:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 1550/2304 [I 2022-10-09 01:48:52] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:10:40 [I 2022-10-09 01:48:52] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00301 [I 2022-10-09 01:48:52] ...trap/engines/engine.py.230: loss: 0.24856 [I 2022-10-09 01:48:52] ...optimizers/trijoint.py.151: total_norm: 44.066700 [I 2022-10-09 01:49:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 1560/2304 [I 2022-10-09 01:49:00] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:25 [I 2022-10-09 01:49:00] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00302 [I 2022-10-09 01:49:00] ...trap/engines/engine.py.230: loss: 0.23499 [I 2022-10-09 01:49:00] ...optimizers/trijoint.py.151: total_norm: 27.711197 [I 2022-10-09 01:49:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 1570/2304 [I 2022-10-09 01:49:09] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:10:23 [I 2022-10-09 01:49:09] ...trap/engines/engine.py.229: process: 0.86867 | load: 0.00301 [I 2022-10-09 01:49:09] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-09 01:49:09] ...optimizers/trijoint.py.151: total_norm: 14.579197 [I 2022-10-09 01:49:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 1580/2304 [I 2022-10-09 01:49:17] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:08 [I 2022-10-09 01:49:17] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00300 [I 2022-10-09 01:49:17] ...trap/engines/engine.py.230: loss: 0.11049 [I 2022-10-09 01:49:17] ...optimizers/trijoint.py.151: total_norm: 15.448283 [I 2022-10-09 01:49:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 1590/2304 [I 2022-10-09 01:49:26] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:09:59 [I 2022-10-09 01:49:26] ...trap/engines/engine.py.229: process: 0.83231 | load: 0.00300 [I 2022-10-09 01:49:26] ...trap/engines/engine.py.230: loss: 0.19795 [I 2022-10-09 01:49:26] ...optimizers/trijoint.py.151: total_norm: 24.387413 [I 2022-10-09 01:49:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 1600/2304 [I 2022-10-09 01:49:34] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:09:53 [I 2022-10-09 01:49:34] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00301 [I 2022-10-09 01:49:34] ...trap/engines/engine.py.230: loss: 0.21277 [I 2022-10-09 01:49:34] ...optimizers/trijoint.py.151: total_norm: 14.910935 [I 2022-10-09 01:49:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 1610/2304 [I 2022-10-09 01:49:43] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:09:44 [I 2022-10-09 01:49:43] ...trap/engines/engine.py.229: process: 0.83836 | load: 0.00300 [I 2022-10-09 01:49:43] ...trap/engines/engine.py.230: loss: 0.18619 [I 2022-10-09 01:49:43] ...optimizers/trijoint.py.151: total_norm: 24.295837 [I 2022-10-09 01:49:51] ...trap/engines/engine.py.224: train: epoch 27 | batch 1620/2304 [I 2022-10-09 01:49:51] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:09:36 [I 2022-10-09 01:49:51] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00302 [I 2022-10-09 01:49:51] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-09 01:49:51] ...optimizers/trijoint.py.151: total_norm: 10.519026 [I 2022-10-09 01:49:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 1630/2304 [I 2022-10-09 01:49:59] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:27 [I 2022-10-09 01:49:59] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00301 [I 2022-10-09 01:49:59] ...trap/engines/engine.py.230: loss: 0.21013 [I 2022-10-09 01:50:00] ...optimizers/trijoint.py.151: total_norm: 16.608803 [I 2022-10-09 01:50:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 1640/2304 [I 2022-10-09 01:50:08] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:09:17 [I 2022-10-09 01:50:08] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00298 [I 2022-10-09 01:50:08] ...trap/engines/engine.py.230: loss: 0.17851 [I 2022-10-09 01:50:08] ...optimizers/trijoint.py.151: total_norm: 23.895704 [I 2022-10-09 01:50:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 1650/2304 [I 2022-10-09 01:50:16] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:10 [I 2022-10-09 01:50:16] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00301 [I 2022-10-09 01:50:16] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-09 01:50:16] ...optimizers/trijoint.py.151: total_norm: 9.759040 [I 2022-10-09 01:50:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 1660/2304 [I 2022-10-09 01:50:25] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:08:58 [I 2022-10-09 01:50:25] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00296 [I 2022-10-09 01:50:25] ...trap/engines/engine.py.230: loss: 0.18261 [I 2022-10-09 01:50:25] ...optimizers/trijoint.py.151: total_norm: 22.766094 [I 2022-10-09 01:50:33] ...trap/engines/engine.py.224: train: epoch 27 | batch 1670/2304 [I 2022-10-09 01:50:33] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:08:52 [I 2022-10-09 01:50:33] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00300 [I 2022-10-09 01:50:33] ...trap/engines/engine.py.230: loss: 0.15009 [I 2022-10-09 01:50:33] ...optimizers/trijoint.py.151: total_norm: 19.927132 [I 2022-10-09 01:50:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 1680/2304 [I 2022-10-09 01:50:42] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:08:46 [I 2022-10-09 01:50:42] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00298 [I 2022-10-09 01:50:42] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-09 01:50:42] ...optimizers/trijoint.py.151: total_norm: 18.062759 [I 2022-10-09 01:50:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 1690/2304 [I 2022-10-09 01:50:50] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:08:33 [I 2022-10-09 01:50:50] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00332 [I 2022-10-09 01:50:50] ...trap/engines/engine.py.230: loss: 0.17794 [I 2022-10-09 01:50:50] ...optimizers/trijoint.py.151: total_norm: 32.930492 [I 2022-10-09 01:50:58] ...trap/engines/engine.py.224: train: epoch 27 | batch 1700/2304 [I 2022-10-09 01:50:58] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:30 [I 2022-10-09 01:50:58] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00301 [I 2022-10-09 01:50:58] ...trap/engines/engine.py.230: loss: 0.19264 [I 2022-10-09 01:50:58] ...optimizers/trijoint.py.151: total_norm: 17.277090 [I 2022-10-09 01:51:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 1710/2304 [I 2022-10-09 01:51:07] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:20 [I 2022-10-09 01:51:07] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00303 [I 2022-10-09 01:51:07] ...trap/engines/engine.py.230: loss: 0.21476 [I 2022-10-09 01:51:07] ...optimizers/trijoint.py.151: total_norm: 16.169773 [I 2022-10-09 01:51:15] ...trap/engines/engine.py.224: train: epoch 27 | batch 1720/2304 [I 2022-10-09 01:51:15] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:10 [I 2022-10-09 01:51:15] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00303 [I 2022-10-09 01:51:15] ...trap/engines/engine.py.230: loss: 0.22723 [I 2022-10-09 01:51:15] ...optimizers/trijoint.py.151: total_norm: 19.232428 [I 2022-10-09 01:51:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 1730/2304 [I 2022-10-09 01:51:24] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:07:59 [I 2022-10-09 01:51:24] ...trap/engines/engine.py.229: process: 0.82499 | load: 0.00303 [I 2022-10-09 01:51:24] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-09 01:51:24] ...optimizers/trijoint.py.151: total_norm: 12.039496 [I 2022-10-09 01:51:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 1740/2304 [I 2022-10-09 01:51:32] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:07:55 [I 2022-10-09 01:51:32] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00315 [I 2022-10-09 01:51:32] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-09 01:51:32] ...optimizers/trijoint.py.151: total_norm: 13.000724 [I 2022-10-09 01:51:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 1750/2304 [I 2022-10-09 01:51:41] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:07:46 [I 2022-10-09 01:51:41] ...trap/engines/engine.py.229: process: 0.82637 | load: 0.00307 [I 2022-10-09 01:51:41] ...trap/engines/engine.py.230: loss: 0.18466 [I 2022-10-09 01:51:41] ...optimizers/trijoint.py.151: total_norm: 12.223608 [I 2022-10-09 01:51:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 1760/2304 [I 2022-10-09 01:51:49] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:07:35 [I 2022-10-09 01:51:49] ...trap/engines/engine.py.229: process: 0.82379 | load: 0.00305 [I 2022-10-09 01:51:49] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-09 01:51:49] ...optimizers/trijoint.py.151: total_norm: 19.890078 [I 2022-10-09 01:51:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 1770/2304 [I 2022-10-09 01:51:57] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:29 [I 2022-10-09 01:51:57] ...trap/engines/engine.py.229: process: 0.83726 | load: 0.00308 [I 2022-10-09 01:51:57] ...trap/engines/engine.py.230: loss: 0.20837 [I 2022-10-09 01:51:57] ...optimizers/trijoint.py.151: total_norm: 19.444397 [I 2022-10-09 01:52:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 1780/2304 [I 2022-10-09 01:52:06] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:21 [I 2022-10-09 01:52:06] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00309 [I 2022-10-09 01:52:06] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-09 01:52:06] ...optimizers/trijoint.py.151: total_norm: 15.636509 [I 2022-10-09 01:52:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 1790/2304 [I 2022-10-09 01:52:14] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:15 [I 2022-10-09 01:52:14] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00306 [I 2022-10-09 01:52:14] ...trap/engines/engine.py.230: loss: 0.16690 [I 2022-10-09 01:52:14] ...optimizers/trijoint.py.151: total_norm: 16.253981 [I 2022-10-09 01:52:23] ...trap/engines/engine.py.224: train: epoch 27 | batch 1800/2304 [I 2022-10-09 01:52:23] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:04 [I 2022-10-09 01:52:23] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00308 [I 2022-10-09 01:52:23] ...trap/engines/engine.py.230: loss: 0.12875 [I 2022-10-09 01:52:23] ...optimizers/trijoint.py.151: total_norm: 15.265718 [I 2022-10-09 01:52:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 1810/2304 [I 2022-10-09 01:52:31] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:06:53 [I 2022-10-09 01:52:31] ...trap/engines/engine.py.229: process: 0.83608 | load: 0.00297 [I 2022-10-09 01:52:31] ...trap/engines/engine.py.230: loss: 0.13764 [I 2022-10-09 01:52:31] ...optimizers/trijoint.py.151: total_norm: 10.510419 [I 2022-10-09 01:52:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 1820/2304 [I 2022-10-09 01:52:40] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:06:47 [I 2022-10-09 01:52:40] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00309 [I 2022-10-09 01:52:40] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-09 01:52:40] ...optimizers/trijoint.py.151: total_norm: 13.818963 [I 2022-10-09 01:52:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 1830/2304 [I 2022-10-09 01:52:48] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:06:38 [I 2022-10-09 01:52:48] ...trap/engines/engine.py.229: process: 0.84665 | load: 0.00308 [I 2022-10-09 01:52:48] ...trap/engines/engine.py.230: loss: 0.16370 [I 2022-10-09 01:52:48] ...optimizers/trijoint.py.151: total_norm: 27.422970 [I 2022-10-09 01:52:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 1840/2304 [I 2022-10-09 01:52:56] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:27 [I 2022-10-09 01:52:56] ...trap/engines/engine.py.229: process: 0.83172 | load: 0.00308 [I 2022-10-09 01:52:56] ...trap/engines/engine.py.230: loss: 0.12979 [I 2022-10-09 01:52:56] ...optimizers/trijoint.py.151: total_norm: 12.565158 [I 2022-10-09 01:53:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 1850/2304 [I 2022-10-09 01:53:05] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:22 [I 2022-10-09 01:53:05] ...trap/engines/engine.py.229: process: 0.84003 | load: 0.00301 [I 2022-10-09 01:53:05] ...trap/engines/engine.py.230: loss: 0.17009 [I 2022-10-09 01:53:05] ...optimizers/trijoint.py.151: total_norm: 19.860893 [I 2022-10-09 01:53:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 1860/2304 [I 2022-10-09 01:53:13] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:14 [I 2022-10-09 01:53:13] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00303 [I 2022-10-09 01:53:13] ...trap/engines/engine.py.230: loss: 0.16778 [I 2022-10-09 01:53:13] ...optimizers/trijoint.py.151: total_norm: 23.003363 [I 2022-10-09 01:53:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 1870/2304 [I 2022-10-09 01:53:22] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:08 [I 2022-10-09 01:53:22] ...trap/engines/engine.py.229: process: 0.86184 | load: 0.00302 [I 2022-10-09 01:53:22] ...trap/engines/engine.py.230: loss: 0.17831 [I 2022-10-09 01:53:22] ...optimizers/trijoint.py.151: total_norm: 24.460688 [I 2022-10-09 01:53:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 1880/2304 [I 2022-10-09 01:53:30] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:05:57 [I 2022-10-09 01:53:30] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00301 [I 2022-10-09 01:53:30] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-09 01:53:30] ...optimizers/trijoint.py.151: total_norm: 12.281574 [I 2022-10-09 01:53:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 1890/2304 [I 2022-10-09 01:53:39] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:05:46 [I 2022-10-09 01:53:39] ...trap/engines/engine.py.229: process: 0.82568 | load: 0.00298 [I 2022-10-09 01:53:39] ...trap/engines/engine.py.230: loss: 0.12773 [I 2022-10-09 01:53:39] ...optimizers/trijoint.py.151: total_norm: 14.107932 [I 2022-10-09 01:53:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 1900/2304 [I 2022-10-09 01:53:47] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:05:39 [I 2022-10-09 01:53:47] ...trap/engines/engine.py.229: process: 0.84229 | load: 0.00308 [I 2022-10-09 01:53:47] ...trap/engines/engine.py.230: loss: 0.18361 [I 2022-10-09 01:53:47] ...optimizers/trijoint.py.151: total_norm: 20.326677 [I 2022-10-09 01:53:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 1910/2304 [I 2022-10-09 01:53:55] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:32 [I 2022-10-09 01:53:55] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00296 [I 2022-10-09 01:53:55] ...trap/engines/engine.py.230: loss: 0.12706 [I 2022-10-09 01:53:55] ...optimizers/trijoint.py.151: total_norm: 18.923929 [I 2022-10-09 01:54:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 1920/2304 [I 2022-10-09 01:54:04] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:23 [I 2022-10-09 01:54:04] ...trap/engines/engine.py.229: process: 0.83547 | load: 0.00302 [I 2022-10-09 01:54:04] ...trap/engines/engine.py.230: loss: 0.13568 [I 2022-10-09 01:54:04] ...optimizers/trijoint.py.151: total_norm: 17.519493 [I 2022-10-09 01:54:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 1930/2304 [I 2022-10-09 01:54:12] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:16 [I 2022-10-09 01:54:12] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00302 [I 2022-10-09 01:54:12] ...trap/engines/engine.py.230: loss: 0.21658 [I 2022-10-09 01:54:12] ...optimizers/trijoint.py.151: total_norm: 18.555458 [I 2022-10-09 01:54:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 1940/2304 [I 2022-10-09 01:54:21] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:06 [I 2022-10-09 01:54:21] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00302 [I 2022-10-09 01:54:21] ...trap/engines/engine.py.230: loss: 0.13443 [I 2022-10-09 01:54:21] ...optimizers/trijoint.py.151: total_norm: 27.809277 [I 2022-10-09 01:54:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 1950/2304 [I 2022-10-09 01:54:29] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:04:57 [I 2022-10-09 01:54:29] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00299 [I 2022-10-09 01:54:29] ...trap/engines/engine.py.230: loss: 0.33094 [I 2022-10-09 01:54:29] ...optimizers/trijoint.py.151: total_norm: 19.814779 [I 2022-10-09 01:54:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 1960/2304 [I 2022-10-09 01:54:38] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:04:49 [I 2022-10-09 01:54:38] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00299 [I 2022-10-09 01:54:38] ...trap/engines/engine.py.230: loss: 0.19505 [I 2022-10-09 01:54:38] ...optimizers/trijoint.py.151: total_norm: 20.405849 [I 2022-10-09 01:54:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 1970/2304 [I 2022-10-09 01:54:46] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:04:41 [I 2022-10-09 01:54:46] ...trap/engines/engine.py.229: process: 0.84604 | load: 0.00297 [I 2022-10-09 01:54:46] ...trap/engines/engine.py.230: loss: 0.16295 [I 2022-10-09 01:54:46] ...optimizers/trijoint.py.151: total_norm: 28.676128 [I 2022-10-09 01:54:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 1980/2304 [I 2022-10-09 01:54:54] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:33 [I 2022-10-09 01:54:54] ...trap/engines/engine.py.229: process: 0.83425 | load: 0.00298 [I 2022-10-09 01:54:54] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-09 01:54:54] ...optimizers/trijoint.py.151: total_norm: 9.189960 [I 2022-10-09 01:55:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 1990/2304 [I 2022-10-09 01:55:03] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:23 [I 2022-10-09 01:55:03] ...trap/engines/engine.py.229: process: 0.83550 | load: 0.00301 [I 2022-10-09 01:55:03] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-09 01:55:03] ...optimizers/trijoint.py.151: total_norm: 7.278916 [I 2022-10-09 01:55:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 2000/2304 [I 2022-10-09 01:55:11] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:16 [I 2022-10-09 01:55:11] ...trap/engines/engine.py.229: process: 0.84995 | load: 0.00300 [I 2022-10-09 01:55:11] ...trap/engines/engine.py.230: loss: 0.15730 [I 2022-10-09 01:55:11] ...optimizers/trijoint.py.151: total_norm: 21.435261 [I 2022-10-09 01:55:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 2010/2304 [I 2022-10-09 01:55:20] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:06 [I 2022-10-09 01:55:20] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00307 [I 2022-10-09 01:55:20] ...trap/engines/engine.py.230: loss: 0.17230 [I 2022-10-09 01:55:20] ...optimizers/trijoint.py.151: total_norm: 32.050175 [I 2022-10-09 01:55:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 2020/2304 [I 2022-10-09 01:55:28] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:03:58 [I 2022-10-09 01:55:28] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00300 [I 2022-10-09 01:55:28] ...trap/engines/engine.py.230: loss: 0.16173 [I 2022-10-09 01:55:28] ...optimizers/trijoint.py.151: total_norm: 18.256151 [I 2022-10-09 01:55:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 2030/2304 [I 2022-10-09 01:55:37] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:03:50 [I 2022-10-09 01:55:37] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00305 [I 2022-10-09 01:55:37] ...trap/engines/engine.py.230: loss: 0.12308 [I 2022-10-09 01:55:37] ...optimizers/trijoint.py.151: total_norm: 17.751490 [I 2022-10-09 01:55:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 2040/2304 [I 2022-10-09 01:55:45] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:41 [I 2022-10-09 01:55:45] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00299 [I 2022-10-09 01:55:45] ...trap/engines/engine.py.230: loss: 0.14359 [I 2022-10-09 01:55:45] ...optimizers/trijoint.py.151: total_norm: 11.744366 [I 2022-10-09 01:55:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 2050/2304 [I 2022-10-09 01:55:53] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:32 [I 2022-10-09 01:55:53] ...trap/engines/engine.py.229: process: 0.82351 | load: 0.00302 [I 2022-10-09 01:55:53] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-09 01:55:53] ...optimizers/trijoint.py.151: total_norm: 28.787386 [I 2022-10-09 01:56:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 2060/2304 [I 2022-10-09 01:56:02] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:24 [I 2022-10-09 01:56:02] ...trap/engines/engine.py.229: process: 0.82733 | load: 0.00297 [I 2022-10-09 01:56:02] ...trap/engines/engine.py.230: loss: 0.20648 [I 2022-10-09 01:56:02] ...optimizers/trijoint.py.151: total_norm: 11.600636 [I 2022-10-09 01:56:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 2070/2304 [I 2022-10-09 01:56:10] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:17 [I 2022-10-09 01:56:10] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00301 [I 2022-10-09 01:56:10] ...trap/engines/engine.py.230: loss: 0.26697 [I 2022-10-09 01:56:10] ...optimizers/trijoint.py.151: total_norm: 35.059460 [I 2022-10-09 01:56:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 2080/2304 [I 2022-10-09 01:56:19] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:08 [I 2022-10-09 01:56:19] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00302 [I 2022-10-09 01:56:19] ...trap/engines/engine.py.230: loss: 0.18103 [I 2022-10-09 01:56:19] ...optimizers/trijoint.py.151: total_norm: 14.489358 [I 2022-10-09 01:56:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 2090/2304 [I 2022-10-09 01:56:27] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:02:59 [I 2022-10-09 01:56:27] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00303 [I 2022-10-09 01:56:27] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-09 01:56:27] ...optimizers/trijoint.py.151: total_norm: 11.599917 [I 2022-10-09 01:56:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 2100/2304 [I 2022-10-09 01:56:36] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:02:51 [I 2022-10-09 01:56:36] ...trap/engines/engine.py.229: process: 0.83779 | load: 0.00300 [I 2022-10-09 01:56:36] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-09 01:56:36] ...optimizers/trijoint.py.151: total_norm: 14.710271 [I 2022-10-09 01:56:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 2110/2304 [I 2022-10-09 01:56:44] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:02:43 [I 2022-10-09 01:56:44] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00297 [I 2022-10-09 01:56:44] ...trap/engines/engine.py.230: loss: 0.14237 [I 2022-10-09 01:56:44] ...optimizers/trijoint.py.151: total_norm: 21.612703 [I 2022-10-09 01:56:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 2120/2304 [I 2022-10-09 01:56:52] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:34 [I 2022-10-09 01:56:52] ...trap/engines/engine.py.229: process: 0.84010 | load: 0.00299 [I 2022-10-09 01:56:52] ...trap/engines/engine.py.230: loss: 0.13222 [I 2022-10-09 01:56:52] ...optimizers/trijoint.py.151: total_norm: 23.853054 [I 2022-10-09 01:57:01] ...trap/engines/engine.py.224: train: epoch 27 | batch 2130/2304 [I 2022-10-09 01:57:01] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:26 [I 2022-10-09 01:57:01] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00301 [I 2022-10-09 01:57:01] ...trap/engines/engine.py.230: loss: 0.23607 [I 2022-10-09 01:57:01] ...optimizers/trijoint.py.151: total_norm: 26.028875 [I 2022-10-09 01:57:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 2140/2304 [I 2022-10-09 01:57:09] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:17 [I 2022-10-09 01:57:09] ...trap/engines/engine.py.229: process: 0.83358 | load: 0.00300 [I 2022-10-09 01:57:09] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-09 01:57:09] ...optimizers/trijoint.py.151: total_norm: 21.404104 [I 2022-10-09 01:57:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 2150/2304 [I 2022-10-09 01:57:18] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:09 [I 2022-10-09 01:57:18] ...trap/engines/engine.py.229: process: 0.84710 | load: 0.00300 [I 2022-10-09 01:57:18] ...trap/engines/engine.py.230: loss: 0.16799 [I 2022-10-09 01:57:18] ...optimizers/trijoint.py.151: total_norm: 19.284620 [I 2022-10-09 01:57:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 2160/2304 [I 2022-10-09 01:57:26] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:02 [I 2022-10-09 01:57:26] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00299 [I 2022-10-09 01:57:26] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-09 01:57:26] ...optimizers/trijoint.py.151: total_norm: 18.272348 [I 2022-10-09 01:57:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 2170/2304 [I 2022-10-09 01:57:35] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:01:53 [I 2022-10-09 01:57:35] ...trap/engines/engine.py.229: process: 0.84023 | load: 0.00304 [I 2022-10-09 01:57:35] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-09 01:57:35] ...optimizers/trijoint.py.151: total_norm: 33.734806 [I 2022-10-09 01:57:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 2180/2304 [I 2022-10-09 01:57:43] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:01:44 [I 2022-10-09 01:57:43] ...trap/engines/engine.py.229: process: 0.83770 | load: 0.00303 [I 2022-10-09 01:57:43] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-09 01:57:43] ...optimizers/trijoint.py.151: total_norm: 22.038176 [I 2022-10-09 01:57:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 2190/2304 [I 2022-10-09 01:57:52] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:36 [I 2022-10-09 01:57:52] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00302 [I 2022-10-09 01:57:52] ...trap/engines/engine.py.230: loss: 0.21408 [I 2022-10-09 01:57:52] ...optimizers/trijoint.py.151: total_norm: 20.516516 [I 2022-10-09 01:58:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 2200/2304 [I 2022-10-09 01:58:00] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:27 [I 2022-10-09 01:58:00] ...trap/engines/engine.py.229: process: 0.83062 | load: 0.00299 [I 2022-10-09 01:58:00] ...trap/engines/engine.py.230: loss: 0.13132 [I 2022-10-09 01:58:00] ...optimizers/trijoint.py.151: total_norm: 16.136417 [I 2022-10-09 01:58:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 2210/2304 [I 2022-10-09 01:58:08] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:01:18 [I 2022-10-09 01:58:08] ...trap/engines/engine.py.229: process: 0.82839 | load: 0.00301 [I 2022-10-09 01:58:08] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-09 01:58:08] ...optimizers/trijoint.py.151: total_norm: 22.864851 [I 2022-10-09 01:58:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 2220/2304 [I 2022-10-09 01:58:17] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:10 [I 2022-10-09 01:58:17] ...trap/engines/engine.py.229: process: 0.82803 | load: 0.00300 [I 2022-10-09 01:58:17] ...trap/engines/engine.py.230: loss: 0.15731 [I 2022-10-09 01:58:17] ...optimizers/trijoint.py.151: total_norm: 25.950020 [I 2022-10-09 01:58:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 2230/2304 [I 2022-10-09 01:58:25] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:02 [I 2022-10-09 01:58:25] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00307 [I 2022-10-09 01:58:25] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-09 01:58:25] ...optimizers/trijoint.py.151: total_norm: 13.382316 [I 2022-10-09 01:58:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 2240/2304 [I 2022-10-09 01:58:34] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:00:54 [I 2022-10-09 01:58:34] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00303 [I 2022-10-09 01:58:34] ...trap/engines/engine.py.230: loss: 0.22506 [I 2022-10-09 01:58:34] ...optimizers/trijoint.py.151: total_norm: 16.735525 [I 2022-10-09 01:58:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 2250/2304 [I 2022-10-09 01:58:42] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:00:45 [I 2022-10-09 01:58:42] ...trap/engines/engine.py.229: process: 0.83342 | load: 0.00299 [I 2022-10-09 01:58:42] ...trap/engines/engine.py.230: loss: 0.18574 [I 2022-10-09 01:58:42] ...optimizers/trijoint.py.151: total_norm: 20.110384 [I 2022-10-09 01:58:51] ...trap/engines/engine.py.224: train: epoch 27 | batch 2260/2304 [I 2022-10-09 01:58:51] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:36 [I 2022-10-09 01:58:51] ...trap/engines/engine.py.229: process: 0.82214 | load: 0.00301 [I 2022-10-09 01:58:51] ...trap/engines/engine.py.230: loss: 0.16665 [I 2022-10-09 01:58:51] ...optimizers/trijoint.py.151: total_norm: 18.615004 [I 2022-10-09 01:58:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 2270/2304 [I 2022-10-09 01:58:59] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:00:28 [I 2022-10-09 01:58:59] ...trap/engines/engine.py.229: process: 0.82846 | load: 0.00297 [I 2022-10-09 01:58:59] ...trap/engines/engine.py.230: loss: 0.17132 [I 2022-10-09 01:58:59] ...optimizers/trijoint.py.151: total_norm: 26.308617 [I 2022-10-09 01:59:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 2280/2304 [I 2022-10-09 01:59:07] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:20 [I 2022-10-09 01:59:07] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00300 [I 2022-10-09 01:59:07] ...trap/engines/engine.py.230: loss: 0.14935 [I 2022-10-09 01:59:07] ...optimizers/trijoint.py.151: total_norm: 27.420277 [I 2022-10-09 01:59:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 2290/2304 [I 2022-10-09 01:59:16] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:11 [I 2022-10-09 01:59:16] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00306 [I 2022-10-09 01:59:16] ...trap/engines/engine.py.230: loss: 0.18966 [I 2022-10-09 01:59:16] ...optimizers/trijoint.py.151: total_norm: 8.639368 [I 2022-10-09 01:59:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 2300/2304 [I 2022-10-09 01:59:24] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:03 [I 2022-10-09 01:59:24] ...trap/engines/engine.py.229: process: 0.83235 | load: 0.00017 [I 2022-10-09 01:59:24] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-09 01:59:24] ...optimizers/trijoint.py.151: total_norm: 23.948847 [I 2022-10-09 01:59:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 2304/2304 [I 2022-10-09 01:59:28] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:00 [I 2022-10-09 01:59:28] ...trap/engines/engine.py.229: process: 0.81485 | load: 0.00017 [I 2022-10-09 01:59:28] ...trap/engines/engine.py.230: loss: 0.19783 [I 2022-10-09 01:59:28] ...optimizers/trijoint.py.151: total_norm: 20.712376 [S 2022-10-09 01:59:29] ...trap/engines/engine.py.129: train_epoch.epoch: 27 [S 2022-10-09 01:59:29] ...trap/engines/engine.py.129: train_epoch.loss: 0.17026535350498045 [I 2022-10-09 01:59:29] ...s/criterions/triplet.py.84: margin = 0.1900000000000001 [I 2022-10-09 01:59:29] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 27 [I 2022-10-09 01:59:29] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 01:59:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 01:59:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 01:59:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 01:59:32] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 01:59:33] ...trap/engines/engine.py.321: eval: epoch 27 | batch 0/510 [I 2022-10-09 01:59:33] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:48 [I 2022-10-09 01:59:33] ...trap/engines/engine.py.326: process: 3.27217 | load: 2.76339 [I 2022-10-09 01:59:37] ...trap/engines/engine.py.321: eval: epoch 27 | batch 10/510 [I 2022-10-09 01:59:37] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:28 [I 2022-10-09 01:59:37] ...trap/engines/engine.py.326: process: 0.48211 | load: 0.00020 [I 2022-10-09 01:59:43] ...trap/engines/engine.py.321: eval: epoch 27 | batch 20/510 [I 2022-10-09 01:59:43] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:45 [I 2022-10-09 01:59:43] ...trap/engines/engine.py.326: process: 0.62883 | load: 0.00018 [I 2022-10-09 01:59:48] ...trap/engines/engine.py.321: eval: epoch 27 | batch 30/510 [I 2022-10-09 01:59:48] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:03:57 [I 2022-10-09 01:59:48] ...trap/engines/engine.py.326: process: 0.42243 | load: 0.00019 [I 2022-10-09 01:59:54] ...trap/engines/engine.py.321: eval: epoch 27 | batch 40/510 [I 2022-10-09 01:59:54] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:01 [I 2022-10-09 01:59:54] ...trap/engines/engine.py.326: process: 0.55862 | load: 0.00020 [I 2022-10-09 01:59:59] ...trap/engines/engine.py.321: eval: epoch 27 | batch 50/510 [I 2022-10-09 01:59:59] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:00 [I 2022-10-09 01:59:59] ...trap/engines/engine.py.326: process: 0.53042 | load: 0.00020 [I 2022-10-09 02:00:04] ...trap/engines/engine.py.321: eval: epoch 27 | batch 60/510 [I 2022-10-09 02:00:04] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:36 [I 2022-10-09 02:00:04] ...trap/engines/engine.py.326: process: 0.42493 | load: 0.00019 [I 2022-10-09 02:00:09] ...trap/engines/engine.py.321: eval: epoch 27 | batch 70/510 [I 2022-10-09 02:00:09] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:32 [I 2022-10-09 02:00:09] ...trap/engines/engine.py.326: process: 0.48010 | load: 0.00018 [I 2022-10-09 02:00:14] ...trap/engines/engine.py.321: eval: epoch 27 | batch 80/510 [I 2022-10-09 02:00:14] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:46 [I 2022-10-09 02:00:14] ...trap/engines/engine.py.326: process: 0.53667 | load: 0.00019 [I 2022-10-09 02:00:19] ...trap/engines/engine.py.321: eval: epoch 27 | batch 90/510 [I 2022-10-09 02:00:19] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:20 [I 2022-10-09 02:00:19] ...trap/engines/engine.py.326: process: 0.48497 | load: 0.00019 [I 2022-10-09 02:00:23] ...trap/engines/engine.py.321: eval: epoch 27 | batch 100/510 [I 2022-10-09 02:00:23] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:56 [I 2022-10-09 02:00:23] ...trap/engines/engine.py.326: process: 0.31610 | load: 0.00020 [I 2022-10-09 02:00:27] ...trap/engines/engine.py.321: eval: epoch 27 | batch 110/510 [I 2022-10-09 02:00:27] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:12 [I 2022-10-09 02:00:27] ...trap/engines/engine.py.326: process: 0.32170 | load: 0.00019 [I 2022-10-09 02:00:30] ...trap/engines/engine.py.321: eval: epoch 27 | batch 120/510 [I 2022-10-09 02:00:30] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:04 [I 2022-10-09 02:00:30] ...trap/engines/engine.py.326: process: 0.31845 | load: 0.00020 [I 2022-10-09 02:00:33] ...trap/engines/engine.py.321: eval: epoch 27 | batch 130/510 [I 2022-10-09 02:00:33] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:01 [I 2022-10-09 02:00:33] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00021 [I 2022-10-09 02:00:36] ...trap/engines/engine.py.321: eval: epoch 27 | batch 140/510 [I 2022-10-09 02:00:36] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:58 [I 2022-10-09 02:00:36] ...trap/engines/engine.py.326: process: 0.32615 | load: 0.00020 [I 2022-10-09 02:00:39] ...trap/engines/engine.py.321: eval: epoch 27 | batch 150/510 [I 2022-10-09 02:00:39] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-09 02:00:39] ...trap/engines/engine.py.326: process: 0.31553 | load: 0.00020 [I 2022-10-09 02:00:43] ...trap/engines/engine.py.321: eval: epoch 27 | batch 160/510 [I 2022-10-09 02:00:43] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:51 [I 2022-10-09 02:00:43] ...trap/engines/engine.py.326: process: 0.32222 | load: 0.00020 [I 2022-10-09 02:00:46] ...trap/engines/engine.py.321: eval: epoch 27 | batch 170/510 [I 2022-10-09 02:00:46] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:49 [I 2022-10-09 02:00:46] ...trap/engines/engine.py.326: process: 0.32189 | load: 0.00020 [I 2022-10-09 02:00:49] ...trap/engines/engine.py.321: eval: epoch 27 | batch 180/510 [I 2022-10-09 02:00:49] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-09 02:00:49] ...trap/engines/engine.py.326: process: 0.32524 | load: 0.00021 [I 2022-10-09 02:00:52] ...trap/engines/engine.py.321: eval: epoch 27 | batch 190/510 [I 2022-10-09 02:00:52] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:42 [I 2022-10-09 02:00:52] ...trap/engines/engine.py.326: process: 0.32012 | load: 0.00020 [I 2022-10-09 02:00:55] ...trap/engines/engine.py.321: eval: epoch 27 | batch 200/510 [I 2022-10-09 02:00:55] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-09 02:00:55] ...trap/engines/engine.py.326: process: 0.32255 | load: 0.00020 [I 2022-10-09 02:00:59] ...trap/engines/engine.py.321: eval: epoch 27 | batch 210/510 [I 2022-10-09 02:00:59] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-09 02:00:59] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00020 [I 2022-10-09 02:01:02] ...trap/engines/engine.py.321: eval: epoch 27 | batch 220/510 [I 2022-10-09 02:01:02] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:32 [I 2022-10-09 02:01:02] ...trap/engines/engine.py.326: process: 0.31722 | load: 0.00020 [I 2022-10-09 02:01:05] ...trap/engines/engine.py.321: eval: epoch 27 | batch 230/510 [I 2022-10-09 02:01:05] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:29 [I 2022-10-09 02:01:05] ...trap/engines/engine.py.326: process: 0.32196 | load: 0.00019 [I 2022-10-09 02:01:08] ...trap/engines/engine.py.321: eval: epoch 27 | batch 240/510 [I 2022-10-09 02:01:08] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-09 02:01:08] ...trap/engines/engine.py.326: process: 0.31565 | load: 0.00020 [I 2022-10-09 02:01:11] ...trap/engines/engine.py.321: eval: epoch 27 | batch 250/510 [I 2022-10-09 02:01:11] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-09 02:01:11] ...trap/engines/engine.py.326: process: 0.32186 | load: 0.00019 [I 2022-10-09 02:01:14] ...trap/engines/engine.py.321: eval: epoch 27 | batch 260/510 [I 2022-10-09 02:01:14] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:19 [I 2022-10-09 02:01:14] ...trap/engines/engine.py.326: process: 0.32039 | load: 0.00019 [I 2022-10-09 02:01:18] ...trap/engines/engine.py.321: eval: epoch 27 | batch 270/510 [I 2022-10-09 02:01:18] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-09 02:01:18] ...trap/engines/engine.py.326: process: 0.31404 | load: 0.00020 [I 2022-10-09 02:01:21] ...trap/engines/engine.py.321: eval: epoch 27 | batch 280/510 [I 2022-10-09 02:01:21] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-09 02:01:21] ...trap/engines/engine.py.326: process: 0.31392 | load: 0.00020 [I 2022-10-09 02:01:24] ...trap/engines/engine.py.321: eval: epoch 27 | batch 290/510 [I 2022-10-09 02:01:24] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-09 02:01:24] ...trap/engines/engine.py.326: process: 0.32368 | load: 0.00020 [I 2022-10-09 02:01:27] ...trap/engines/engine.py.321: eval: epoch 27 | batch 300/510 [I 2022-10-09 02:01:27] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-09 02:01:27] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00021 [I 2022-10-09 02:01:30] ...trap/engines/engine.py.321: eval: epoch 27 | batch 310/510 [I 2022-10-09 02:01:30] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:04 [I 2022-10-09 02:01:30] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00019 [I 2022-10-09 02:01:34] ...trap/engines/engine.py.321: eval: epoch 27 | batch 320/510 [I 2022-10-09 02:01:34] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 02:01:34] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00020 [I 2022-10-09 02:01:37] ...trap/engines/engine.py.321: eval: epoch 27 | batch 330/510 [I 2022-10-09 02:01:37] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-09 02:01:37] ...trap/engines/engine.py.326: process: 0.32154 | load: 0.00020 [I 2022-10-09 02:01:40] ...trap/engines/engine.py.321: eval: epoch 27 | batch 340/510 [I 2022-10-09 02:01:40] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-09 02:01:40] ...trap/engines/engine.py.326: process: 0.32168 | load: 0.00021 [I 2022-10-09 02:01:43] ...trap/engines/engine.py.321: eval: epoch 27 | batch 350/510 [I 2022-10-09 02:01:43] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-09 02:01:43] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00021 [I 2022-10-09 02:01:46] ...trap/engines/engine.py.321: eval: epoch 27 | batch 360/510 [I 2022-10-09 02:01:46] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:48 [I 2022-10-09 02:01:46] ...trap/engines/engine.py.326: process: 0.32302 | load: 0.00022 [I 2022-10-09 02:01:50] ...trap/engines/engine.py.321: eval: epoch 27 | batch 370/510 [I 2022-10-09 02:01:50] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:45 [I 2022-10-09 02:01:50] ...trap/engines/engine.py.326: process: 0.31512 | load: 0.00021 [I 2022-10-09 02:01:53] ...trap/engines/engine.py.321: eval: epoch 27 | batch 380/510 [I 2022-10-09 02:01:53] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-09 02:01:53] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00022 [I 2022-10-09 02:01:56] ...trap/engines/engine.py.321: eval: epoch 27 | batch 390/510 [I 2022-10-09 02:01:56] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-09 02:01:56] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00020 [I 2022-10-09 02:01:59] ...trap/engines/engine.py.321: eval: epoch 27 | batch 400/510 [I 2022-10-09 02:01:59] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-09 02:01:59] ...trap/engines/engine.py.326: process: 0.32015 | load: 0.00022 [I 2022-10-09 02:02:03] ...trap/engines/engine.py.321: eval: epoch 27 | batch 410/510 [I 2022-10-09 02:02:03] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:32 [I 2022-10-09 02:02:03] ...trap/engines/engine.py.326: process: 0.32575 | load: 0.00021 [I 2022-10-09 02:02:06] ...trap/engines/engine.py.321: eval: epoch 27 | batch 420/510 [I 2022-10-09 02:02:06] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 02:02:06] ...trap/engines/engine.py.326: process: 0.31304 | load: 0.00022 [I 2022-10-09 02:02:09] ...trap/engines/engine.py.321: eval: epoch 27 | batch 430/510 [I 2022-10-09 02:02:09] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-09 02:02:09] ...trap/engines/engine.py.326: process: 0.31751 | load: 0.00020 [I 2022-10-09 02:02:12] ...trap/engines/engine.py.321: eval: epoch 27 | batch 440/510 [I 2022-10-09 02:02:12] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-09 02:02:12] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00020 [I 2022-10-09 02:02:15] ...trap/engines/engine.py.321: eval: epoch 27 | batch 450/510 [I 2022-10-09 02:02:15] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 02:02:15] ...trap/engines/engine.py.326: process: 0.32440 | load: 0.00020 [I 2022-10-09 02:02:19] ...trap/engines/engine.py.321: eval: epoch 27 | batch 460/510 [I 2022-10-09 02:02:19] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:16 [I 2022-10-09 02:02:19] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00022 [I 2022-10-09 02:02:22] ...trap/engines/engine.py.321: eval: epoch 27 | batch 470/510 [I 2022-10-09 02:02:22] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 02:02:22] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00021 [I 2022-10-09 02:02:25] ...trap/engines/engine.py.321: eval: epoch 27 | batch 480/510 [I 2022-10-09 02:02:25] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-09 02:02:25] ...trap/engines/engine.py.326: process: 0.32269 | load: 0.00021 [I 2022-10-09 02:02:28] ...trap/engines/engine.py.321: eval: epoch 27 | batch 490/510 [I 2022-10-09 02:02:28] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-09 02:02:28] ...trap/engines/engine.py.326: process: 0.32153 | load: 0.00020 [I 2022-10-09 02:02:31] ...trap/engines/engine.py.321: eval: epoch 27 | batch 500/510 [I 2022-10-09 02:02:31] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 02:02:31] ...trap/engines/engine.py.326: process: 0.31622 | load: 0.00021 [I 2022-10-09 02:02:35] ...trap/engines/engine.py.321: eval: epoch 27 | batch 510/510 [I 2022-10-09 02:02:35] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-09 02:02:35] ...trap/engines/engine.py.326: process: 0.31929 | load: 0.00016 [S 2022-10-09 02:02:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 27 [I 2022-10-09 02:02:35] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.679100 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.885000 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.925900 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.671300 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.886000 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.923500 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011519 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011171 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008276 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008050 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009808 [S 2022-10-09 02:02:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007580 [I 2022-10-09 02:03:13] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 02:03:13] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 02:03:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 02:03:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 02:03:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 02:03:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 02:03:17] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 02:03:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 02:03:23] ...trap/engines/engine.py.163: Training model on trainset for epoch 28 [I 2022-10-09 02:03:23] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 02:03:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 0/2304 [I 2022-10-09 02:03:26] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:53 [I 2022-10-09 02:03:26] ...trap/engines/engine.py.229: process: 3.09602 | load: 2.23340 [I 2022-10-09 02:03:26] ...trap/engines/engine.py.230: loss: 0.27092 [I 2022-10-09 02:03:26] ...optimizers/trijoint.py.151: total_norm: 32.307251 [I 2022-10-09 02:03:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 10/2304 [I 2022-10-09 02:03:34] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:20 [I 2022-10-09 02:03:34] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00653 [I 2022-10-09 02:03:34] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-09 02:03:34] ...optimizers/trijoint.py.151: total_norm: 23.883505 [I 2022-10-09 02:03:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 20/2304 [I 2022-10-09 02:03:43] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:02 [I 2022-10-09 02:03:43] ...trap/engines/engine.py.229: process: 0.83720 | load: 0.00459 [I 2022-10-09 02:03:43] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-09 02:03:43] ...optimizers/trijoint.py.151: total_norm: 20.688715 [I 2022-10-09 02:03:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 30/2304 [I 2022-10-09 02:03:51] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:56 [I 2022-10-09 02:03:51] ...trap/engines/engine.py.229: process: 0.83419 | load: 0.00391 [I 2022-10-09 02:03:51] ...trap/engines/engine.py.230: loss: 0.12107 [I 2022-10-09 02:03:51] ...optimizers/trijoint.py.151: total_norm: 15.804477 [I 2022-10-09 02:03:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 40/2304 [I 2022-10-09 02:03:59] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:48 [I 2022-10-09 02:03:59] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00396 [I 2022-10-09 02:03:59] ...trap/engines/engine.py.230: loss: 0.15060 [I 2022-10-09 02:03:59] ...optimizers/trijoint.py.151: total_norm: 14.702430 [I 2022-10-09 02:04:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 50/2304 [I 2022-10-09 02:04:08] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:40 [I 2022-10-09 02:04:08] ...trap/engines/engine.py.229: process: 0.84207 | load: 0.00336 [I 2022-10-09 02:04:08] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-09 02:04:08] ...optimizers/trijoint.py.151: total_norm: 17.229158 [I 2022-10-09 02:04:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 60/2304 [I 2022-10-09 02:04:16] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:19 [I 2022-10-09 02:04:16] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00428 [I 2022-10-09 02:04:16] ...trap/engines/engine.py.230: loss: 0.17260 [I 2022-10-09 02:04:16] ...optimizers/trijoint.py.151: total_norm: 15.215133 [I 2022-10-09 02:04:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 70/2304 [I 2022-10-09 02:04:25] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:16 [I 2022-10-09 02:04:25] ...trap/engines/engine.py.229: process: 0.85335 | load: 0.00354 [I 2022-10-09 02:04:25] ...trap/engines/engine.py.230: loss: 0.22598 [I 2022-10-09 02:04:25] ...optimizers/trijoint.py.151: total_norm: 15.379873 [I 2022-10-09 02:04:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 80/2304 [I 2022-10-09 02:04:33] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:12 [I 2022-10-09 02:04:33] ...trap/engines/engine.py.229: process: 0.84536 | load: 0.00340 [I 2022-10-09 02:04:33] ...trap/engines/engine.py.230: loss: 0.17390 [I 2022-10-09 02:04:33] ...optimizers/trijoint.py.151: total_norm: 21.292717 [I 2022-10-09 02:04:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 90/2304 [I 2022-10-09 02:04:42] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:01 [I 2022-10-09 02:04:42] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00313 [I 2022-10-09 02:04:42] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-09 02:04:42] ...optimizers/trijoint.py.151: total_norm: 8.134897 [I 2022-10-09 02:04:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 100/2304 [I 2022-10-09 02:04:50] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:53 [I 2022-10-09 02:04:50] ...trap/engines/engine.py.229: process: 0.84374 | load: 0.00313 [I 2022-10-09 02:04:50] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-09 02:04:50] ...optimizers/trijoint.py.151: total_norm: 30.184284 [I 2022-10-09 02:04:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 110/2304 [I 2022-10-09 02:04:58] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:39 [I 2022-10-09 02:04:58] ...trap/engines/engine.py.229: process: 0.83095 | load: 0.00301 [I 2022-10-09 02:04:58] ...trap/engines/engine.py.230: loss: 0.16617 [I 2022-10-09 02:04:58] ...optimizers/trijoint.py.151: total_norm: 18.173702 [I 2022-10-09 02:05:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 120/2304 [I 2022-10-09 02:05:07] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:34 [I 2022-10-09 02:05:07] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00301 [I 2022-10-09 02:05:07] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-09 02:05:07] ...optimizers/trijoint.py.151: total_norm: 16.476593 [I 2022-10-09 02:05:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 130/2304 [I 2022-10-09 02:05:15] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:25 [I 2022-10-09 02:05:15] ...trap/engines/engine.py.229: process: 0.82564 | load: 0.00322 [I 2022-10-09 02:05:15] ...trap/engines/engine.py.230: loss: 0.21406 [I 2022-10-09 02:05:15] ...optimizers/trijoint.py.151: total_norm: 29.607237 [I 2022-10-09 02:05:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 140/2304 [I 2022-10-09 02:05:24] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:13 [I 2022-10-09 02:05:24] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00307 [I 2022-10-09 02:05:24] ...trap/engines/engine.py.230: loss: 0.15302 [I 2022-10-09 02:05:24] ...optimizers/trijoint.py.151: total_norm: 19.286192 [I 2022-10-09 02:05:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 150/2304 [I 2022-10-09 02:05:32] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:15 [I 2022-10-09 02:05:32] ...trap/engines/engine.py.229: process: 0.84536 | load: 0.00325 [I 2022-10-09 02:05:32] ...trap/engines/engine.py.230: loss: 0.17114 [I 2022-10-09 02:05:32] ...optimizers/trijoint.py.151: total_norm: 23.749252 [I 2022-10-09 02:05:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 160/2304 [I 2022-10-09 02:05:40] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:05 [I 2022-10-09 02:05:40] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00316 [I 2022-10-09 02:05:40] ...trap/engines/engine.py.230: loss: 0.15587 [I 2022-10-09 02:05:40] ...optimizers/trijoint.py.151: total_norm: 18.682196 [I 2022-10-09 02:05:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 170/2304 [I 2022-10-09 02:05:49] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:47 [I 2022-10-09 02:05:49] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00297 [I 2022-10-09 02:05:49] ...trap/engines/engine.py.230: loss: 0.08897 [I 2022-10-09 02:05:49] ...optimizers/trijoint.py.151: total_norm: 12.329007 [I 2022-10-09 02:05:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 180/2304 [I 2022-10-09 02:05:57] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:38 [I 2022-10-09 02:05:57] ...trap/engines/engine.py.229: process: 0.83651 | load: 0.00302 [I 2022-10-09 02:05:57] ...trap/engines/engine.py.230: loss: 0.14640 [I 2022-10-09 02:05:57] ...optimizers/trijoint.py.151: total_norm: 18.725016 [I 2022-10-09 02:06:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 190/2304 [I 2022-10-09 02:06:06] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:39 [I 2022-10-09 02:06:06] ...trap/engines/engine.py.229: process: 0.85090 | load: 0.00318 [I 2022-10-09 02:06:06] ...trap/engines/engine.py.230: loss: 0.19603 [I 2022-10-09 02:06:06] ...optimizers/trijoint.py.151: total_norm: 24.745293 [I 2022-10-09 02:06:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 200/2304 [I 2022-10-09 02:06:14] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:30 [I 2022-10-09 02:06:14] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00331 [I 2022-10-09 02:06:14] ...trap/engines/engine.py.230: loss: 0.17958 [I 2022-10-09 02:06:14] ...optimizers/trijoint.py.151: total_norm: 22.424974 [I 2022-10-09 02:06:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 210/2304 [I 2022-10-09 02:06:22] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:15 [I 2022-10-09 02:06:22] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00300 [I 2022-10-09 02:06:23] ...trap/engines/engine.py.230: loss: 0.12476 [I 2022-10-09 02:06:23] ...optimizers/trijoint.py.151: total_norm: 34.769791 [I 2022-10-09 02:06:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 220/2304 [I 2022-10-09 02:06:31] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:07 [I 2022-10-09 02:06:31] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00325 [I 2022-10-09 02:06:31] ...trap/engines/engine.py.230: loss: 0.16907 [I 2022-10-09 02:06:31] ...optimizers/trijoint.py.151: total_norm: 17.546879 [I 2022-10-09 02:06:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 230/2304 [I 2022-10-09 02:06:39] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:04 [I 2022-10-09 02:06:39] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00295 [I 2022-10-09 02:06:39] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-09 02:06:39] ...optimizers/trijoint.py.151: total_norm: 15.843407 [I 2022-10-09 02:06:48] ...trap/engines/engine.py.224: train: epoch 28 | batch 240/2304 [I 2022-10-09 02:06:48] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:51 [I 2022-10-09 02:06:48] ...trap/engines/engine.py.229: process: 0.84496 | load: 0.00323 [I 2022-10-09 02:06:48] ...trap/engines/engine.py.230: loss: 0.23908 [I 2022-10-09 02:06:48] ...optimizers/trijoint.py.151: total_norm: 16.513716 [I 2022-10-09 02:06:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 250/2304 [I 2022-10-09 02:06:56] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:45 [I 2022-10-09 02:06:56] ...trap/engines/engine.py.229: process: 0.84244 | load: 0.00330 [I 2022-10-09 02:06:56] ...trap/engines/engine.py.230: loss: 0.13542 [I 2022-10-09 02:06:56] ...optimizers/trijoint.py.151: total_norm: 25.903728 [I 2022-10-09 02:07:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 260/2304 [I 2022-10-09 02:07:05] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:41 [I 2022-10-09 02:07:05] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00312 [I 2022-10-09 02:07:05] ...trap/engines/engine.py.230: loss: 0.10137 [I 2022-10-09 02:07:05] ...optimizers/trijoint.py.151: total_norm: 16.350218 [I 2022-10-09 02:07:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 270/2304 [I 2022-10-09 02:07:13] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:30 [I 2022-10-09 02:07:13] ...trap/engines/engine.py.229: process: 0.83691 | load: 0.00300 [I 2022-10-09 02:07:13] ...trap/engines/engine.py.230: loss: 0.12325 [I 2022-10-09 02:07:13] ...optimizers/trijoint.py.151: total_norm: 12.996311 [I 2022-10-09 02:07:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 280/2304 [I 2022-10-09 02:07:21] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:29 [I 2022-10-09 02:07:21] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00309 [I 2022-10-09 02:07:21] ...trap/engines/engine.py.230: loss: 0.23091 [I 2022-10-09 02:07:21] ...optimizers/trijoint.py.151: total_norm: 30.558737 [I 2022-10-09 02:07:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 290/2304 [I 2022-10-09 02:07:30] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:17 [I 2022-10-09 02:07:30] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00310 [I 2022-10-09 02:07:30] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-09 02:07:30] ...optimizers/trijoint.py.151: total_norm: 26.825018 [I 2022-10-09 02:07:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 300/2304 [I 2022-10-09 02:07:38] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:10 [I 2022-10-09 02:07:38] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00309 [I 2022-10-09 02:07:38] ...trap/engines/engine.py.230: loss: 0.14406 [I 2022-10-09 02:07:38] ...optimizers/trijoint.py.151: total_norm: 17.207563 [I 2022-10-09 02:07:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 310/2304 [I 2022-10-09 02:07:47] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:53 [I 2022-10-09 02:07:47] ...trap/engines/engine.py.229: process: 0.83492 | load: 0.00319 [I 2022-10-09 02:07:47] ...trap/engines/engine.py.230: loss: 0.16442 [I 2022-10-09 02:07:47] ...optimizers/trijoint.py.151: total_norm: 11.879008 [I 2022-10-09 02:07:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 320/2304 [I 2022-10-09 02:07:55] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:45 [I 2022-10-09 02:07:55] ...trap/engines/engine.py.229: process: 0.83163 | load: 0.00301 [I 2022-10-09 02:07:55] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-09 02:07:55] ...optimizers/trijoint.py.151: total_norm: 15.345234 [I 2022-10-09 02:08:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 330/2304 [I 2022-10-09 02:08:04] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:53 [I 2022-10-09 02:08:04] ...trap/engines/engine.py.229: process: 0.85918 | load: 0.00301 [I 2022-10-09 02:08:04] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-09 02:08:04] ...optimizers/trijoint.py.151: total_norm: 20.656410 [I 2022-10-09 02:08:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 340/2304 [I 2022-10-09 02:08:12] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:37 [I 2022-10-09 02:08:12] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00315 [I 2022-10-09 02:08:12] ...trap/engines/engine.py.230: loss: 0.16231 [I 2022-10-09 02:08:12] ...optimizers/trijoint.py.151: total_norm: 12.669783 [I 2022-10-09 02:08:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 350/2304 [I 2022-10-09 02:08:21] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:28 [I 2022-10-09 02:08:21] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00301 [I 2022-10-09 02:08:21] ...trap/engines/engine.py.230: loss: 0.18213 [I 2022-10-09 02:08:21] ...optimizers/trijoint.py.151: total_norm: 17.956045 [I 2022-10-09 02:08:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 360/2304 [I 2022-10-09 02:08:29] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:22 [I 2022-10-09 02:08:29] ...trap/engines/engine.py.229: process: 0.83762 | load: 0.00310 [I 2022-10-09 02:08:29] ...trap/engines/engine.py.230: loss: 0.18075 [I 2022-10-09 02:08:29] ...optimizers/trijoint.py.151: total_norm: 13.492200 [I 2022-10-09 02:08:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 370/2304 [I 2022-10-09 02:08:37] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:08 [I 2022-10-09 02:08:37] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00300 [I 2022-10-09 02:08:37] ...trap/engines/engine.py.230: loss: 0.22649 [I 2022-10-09 02:08:37] ...optimizers/trijoint.py.151: total_norm: 19.381727 [I 2022-10-09 02:08:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 380/2304 [I 2022-10-09 02:08:46] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:57 [I 2022-10-09 02:08:46] ...trap/engines/engine.py.229: process: 0.83482 | load: 0.00303 [I 2022-10-09 02:08:46] ...trap/engines/engine.py.230: loss: 0.10296 [I 2022-10-09 02:08:46] ...optimizers/trijoint.py.151: total_norm: 13.249452 [I 2022-10-09 02:08:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 390/2304 [I 2022-10-09 02:08:54] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:56 [I 2022-10-09 02:08:54] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00314 [I 2022-10-09 02:08:54] ...trap/engines/engine.py.230: loss: 0.14993 [I 2022-10-09 02:08:54] ...optimizers/trijoint.py.151: total_norm: 22.881157 [I 2022-10-09 02:09:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 400/2304 [I 2022-10-09 02:09:03] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:48 [I 2022-10-09 02:09:03] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00305 [I 2022-10-09 02:09:03] ...trap/engines/engine.py.230: loss: 0.21340 [I 2022-10-09 02:09:03] ...optimizers/trijoint.py.151: total_norm: 19.836729 [I 2022-10-09 02:09:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 410/2304 [I 2022-10-09 02:09:11] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:34 [I 2022-10-09 02:09:11] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00301 [I 2022-10-09 02:09:11] ...trap/engines/engine.py.230: loss: 0.23299 [I 2022-10-09 02:09:11] ...optimizers/trijoint.py.151: total_norm: 11.160913 [I 2022-10-09 02:09:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 420/2304 [I 2022-10-09 02:09:20] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:27 [I 2022-10-09 02:09:20] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00304 [I 2022-10-09 02:09:20] ...trap/engines/engine.py.230: loss: 0.13871 [I 2022-10-09 02:09:20] ...optimizers/trijoint.py.151: total_norm: 17.252934 [I 2022-10-09 02:09:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 430/2304 [I 2022-10-09 02:09:28] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:17 [I 2022-10-09 02:09:28] ...trap/engines/engine.py.229: process: 0.83783 | load: 0.00307 [I 2022-10-09 02:09:28] ...trap/engines/engine.py.230: loss: 0.18973 [I 2022-10-09 02:09:28] ...optimizers/trijoint.py.151: total_norm: 12.116197 [I 2022-10-09 02:09:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 440/2304 [I 2022-10-09 02:09:37] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:05 [I 2022-10-09 02:09:37] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00303 [I 2022-10-09 02:09:37] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-09 02:09:37] ...optimizers/trijoint.py.151: total_norm: 22.044458 [I 2022-10-09 02:09:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 450/2304 [I 2022-10-09 02:09:45] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:59 [I 2022-10-09 02:09:45] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00300 [I 2022-10-09 02:09:45] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-09 02:09:45] ...optimizers/trijoint.py.151: total_norm: 14.396397 [I 2022-10-09 02:09:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 460/2304 [I 2022-10-09 02:09:53] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:52 [I 2022-10-09 02:09:53] ...trap/engines/engine.py.229: process: 0.85186 | load: 0.00306 [I 2022-10-09 02:09:53] ...trap/engines/engine.py.230: loss: 0.24145 [I 2022-10-09 02:09:53] ...optimizers/trijoint.py.151: total_norm: 21.195093 [I 2022-10-09 02:10:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 470/2304 [I 2022-10-09 02:10:02] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:47 [I 2022-10-09 02:10:02] ...trap/engines/engine.py.229: process: 0.83243 | load: 0.00304 [I 2022-10-09 02:10:02] ...trap/engines/engine.py.230: loss: 0.11227 [I 2022-10-09 02:10:02] ...optimizers/trijoint.py.151: total_norm: 18.612827 [I 2022-10-09 02:10:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 480/2304 [I 2022-10-09 02:10:10] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:25 [I 2022-10-09 02:10:10] ...trap/engines/engine.py.229: process: 0.82937 | load: 0.00302 [I 2022-10-09 02:10:10] ...trap/engines/engine.py.230: loss: 0.13769 [I 2022-10-09 02:10:10] ...optimizers/trijoint.py.151: total_norm: 11.479744 [I 2022-10-09 02:10:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 490/2304 [I 2022-10-09 02:10:19] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:24 [I 2022-10-09 02:10:19] ...trap/engines/engine.py.229: process: 0.83479 | load: 0.00306 [I 2022-10-09 02:10:19] ...trap/engines/engine.py.230: loss: 0.21647 [I 2022-10-09 02:10:19] ...optimizers/trijoint.py.151: total_norm: 19.698755 [I 2022-10-09 02:10:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 500/2304 [I 2022-10-09 02:10:27] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:16 [I 2022-10-09 02:10:27] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00300 [I 2022-10-09 02:10:27] ...trap/engines/engine.py.230: loss: 0.18647 [I 2022-10-09 02:10:27] ...optimizers/trijoint.py.151: total_norm: 20.470308 [I 2022-10-09 02:10:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 510/2304 [I 2022-10-09 02:10:35] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:07 [I 2022-10-09 02:10:35] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00300 [I 2022-10-09 02:10:35] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-09 02:10:35] ...optimizers/trijoint.py.151: total_norm: 22.602140 [I 2022-10-09 02:10:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 520/2304 [I 2022-10-09 02:10:44] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:24:49 [I 2022-10-09 02:10:44] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00299 [I 2022-10-09 02:10:44] ...trap/engines/engine.py.230: loss: 0.24015 [I 2022-10-09 02:10:44] ...optimizers/trijoint.py.151: total_norm: 18.863140 [I 2022-10-09 02:10:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 530/2304 [I 2022-10-09 02:10:52] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:35 [I 2022-10-09 02:10:52] ...trap/engines/engine.py.229: process: 0.82244 | load: 0.00299 [I 2022-10-09 02:10:52] ...trap/engines/engine.py.230: loss: 0.16604 [I 2022-10-09 02:10:52] ...optimizers/trijoint.py.151: total_norm: 20.281981 [I 2022-10-09 02:11:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 540/2304 [I 2022-10-09 02:11:01] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:42 [I 2022-10-09 02:11:01] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00298 [I 2022-10-09 02:11:01] ...trap/engines/engine.py.230: loss: 0.20707 [I 2022-10-09 02:11:01] ...optimizers/trijoint.py.151: total_norm: 33.711063 [I 2022-10-09 02:11:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 550/2304 [I 2022-10-09 02:11:09] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:36 [I 2022-10-09 02:11:09] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00300 [I 2022-10-09 02:11:09] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-09 02:11:09] ...optimizers/trijoint.py.151: total_norm: 20.231962 [I 2022-10-09 02:11:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 560/2304 [I 2022-10-09 02:11:18] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:27 [I 2022-10-09 02:11:18] ...trap/engines/engine.py.229: process: 0.83665 | load: 0.00296 [I 2022-10-09 02:11:18] ...trap/engines/engine.py.230: loss: 0.17180 [I 2022-10-09 02:11:18] ...optimizers/trijoint.py.151: total_norm: 14.828359 [I 2022-10-09 02:11:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 570/2304 [I 2022-10-09 02:11:26] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:09 [I 2022-10-09 02:11:26] ...trap/engines/engine.py.229: process: 0.82504 | load: 0.00299 [I 2022-10-09 02:11:26] ...trap/engines/engine.py.230: loss: 0.19029 [I 2022-10-09 02:11:26] ...optimizers/trijoint.py.151: total_norm: 17.098576 [I 2022-10-09 02:11:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 580/2304 [I 2022-10-09 02:11:34] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:04 [I 2022-10-09 02:11:34] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00299 [I 2022-10-09 02:11:34] ...trap/engines/engine.py.230: loss: 0.20293 [I 2022-10-09 02:11:34] ...optimizers/trijoint.py.151: total_norm: 19.098747 [I 2022-10-09 02:11:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 590/2304 [I 2022-10-09 02:11:43] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:01 [I 2022-10-09 02:11:43] ...trap/engines/engine.py.229: process: 0.84889 | load: 0.00300 [I 2022-10-09 02:11:43] ...trap/engines/engine.py.230: loss: 0.16970 [I 2022-10-09 02:11:43] ...optimizers/trijoint.py.151: total_norm: 18.567617 [I 2022-10-09 02:11:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 600/2304 [I 2022-10-09 02:11:51] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:47 [I 2022-10-09 02:11:51] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00298 [I 2022-10-09 02:11:51] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-09 02:11:51] ...optimizers/trijoint.py.151: total_norm: 11.824644 [I 2022-10-09 02:12:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 610/2304 [I 2022-10-09 02:12:00] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:23:47 [I 2022-10-09 02:12:00] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00299 [I 2022-10-09 02:12:00] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-09 02:12:00] ...optimizers/trijoint.py.151: total_norm: 13.844651 [I 2022-10-09 02:12:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 620/2304 [I 2022-10-09 02:12:08] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:38 [I 2022-10-09 02:12:08] ...trap/engines/engine.py.229: process: 0.83561 | load: 0.00309 [I 2022-10-09 02:12:08] ...trap/engines/engine.py.230: loss: 0.17771 [I 2022-10-09 02:12:08] ...optimizers/trijoint.py.151: total_norm: 31.331112 [I 2022-10-09 02:12:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 630/2304 [I 2022-10-09 02:12:16] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:34 [I 2022-10-09 02:12:16] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00302 [I 2022-10-09 02:12:16] ...trap/engines/engine.py.230: loss: 0.12011 [I 2022-10-09 02:12:16] ...optimizers/trijoint.py.151: total_norm: 16.684074 [I 2022-10-09 02:12:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 640/2304 [I 2022-10-09 02:12:25] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:18 [I 2022-10-09 02:12:25] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00301 [I 2022-10-09 02:12:25] ...trap/engines/engine.py.230: loss: 0.15055 [I 2022-10-09 02:12:25] ...optimizers/trijoint.py.151: total_norm: 12.808062 [I 2022-10-09 02:12:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 650/2304 [I 2022-10-09 02:12:33] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:09 [I 2022-10-09 02:12:33] ...trap/engines/engine.py.229: process: 0.84705 | load: 0.00301 [I 2022-10-09 02:12:33] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-09 02:12:33] ...optimizers/trijoint.py.151: total_norm: 12.106978 [I 2022-10-09 02:12:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 660/2304 [I 2022-10-09 02:12:42] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:08 [I 2022-10-09 02:12:42] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00300 [I 2022-10-09 02:12:42] ...trap/engines/engine.py.230: loss: 0.15431 [I 2022-10-09 02:12:42] ...optimizers/trijoint.py.151: total_norm: 15.564964 [I 2022-10-09 02:12:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 670/2304 [I 2022-10-09 02:12:50] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:03 [I 2022-10-09 02:12:50] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00301 [I 2022-10-09 02:12:50] ...trap/engines/engine.py.230: loss: 0.12388 [I 2022-10-09 02:12:50] ...optimizers/trijoint.py.151: total_norm: 20.971127 [I 2022-10-09 02:12:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 680/2304 [I 2022-10-09 02:12:59] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:58 [I 2022-10-09 02:12:59] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00299 [I 2022-10-09 02:12:59] ...trap/engines/engine.py.230: loss: 0.23834 [I 2022-10-09 02:12:59] ...optimizers/trijoint.py.151: total_norm: 38.212711 [I 2022-10-09 02:13:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 690/2304 [I 2022-10-09 02:13:07] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:43 [I 2022-10-09 02:13:07] ...trap/engines/engine.py.229: process: 0.84960 | load: 0.00301 [I 2022-10-09 02:13:07] ...trap/engines/engine.py.230: loss: 0.09104 [I 2022-10-09 02:13:07] ...optimizers/trijoint.py.151: total_norm: 15.456078 [I 2022-10-09 02:13:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 700/2304 [I 2022-10-09 02:13:16] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:38 [I 2022-10-09 02:13:16] ...trap/engines/engine.py.229: process: 0.83597 | load: 0.00299 [I 2022-10-09 02:13:16] ...trap/engines/engine.py.230: loss: 0.18726 [I 2022-10-09 02:13:16] ...optimizers/trijoint.py.151: total_norm: 18.561571 [I 2022-10-09 02:13:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 710/2304 [I 2022-10-09 02:13:24] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:21 [I 2022-10-09 02:13:24] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00301 [I 2022-10-09 02:13:24] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-09 02:13:24] ...optimizers/trijoint.py.151: total_norm: 15.545881 [I 2022-10-09 02:13:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 720/2304 [I 2022-10-09 02:13:33] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:22 [I 2022-10-09 02:13:33] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00299 [I 2022-10-09 02:13:33] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-09 02:13:33] ...optimizers/trijoint.py.151: total_norm: 8.764882 [I 2022-10-09 02:13:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 730/2304 [I 2022-10-09 02:13:41] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:13 [I 2022-10-09 02:13:41] ...trap/engines/engine.py.229: process: 0.84336 | load: 0.00309 [I 2022-10-09 02:13:41] ...trap/engines/engine.py.230: loss: 0.15516 [I 2022-10-09 02:13:41] ...optimizers/trijoint.py.151: total_norm: 18.239641 [I 2022-10-09 02:13:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 740/2304 [I 2022-10-09 02:13:49] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:45 [I 2022-10-09 02:13:49] ...trap/engines/engine.py.229: process: 0.82694 | load: 0.00300 [I 2022-10-09 02:13:49] ...trap/engines/engine.py.230: loss: 0.17800 [I 2022-10-09 02:13:49] ...optimizers/trijoint.py.151: total_norm: 16.974989 [I 2022-10-09 02:13:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 750/2304 [I 2022-10-09 02:13:58] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:47 [I 2022-10-09 02:13:58] ...trap/engines/engine.py.229: process: 0.83731 | load: 0.00301 [I 2022-10-09 02:13:58] ...trap/engines/engine.py.230: loss: 0.14363 [I 2022-10-09 02:13:58] ...optimizers/trijoint.py.151: total_norm: 13.992211 [I 2022-10-09 02:14:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 760/2304 [I 2022-10-09 02:14:06] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:40 [I 2022-10-09 02:14:06] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00303 [I 2022-10-09 02:14:06] ...trap/engines/engine.py.230: loss: 0.17346 [I 2022-10-09 02:14:06] ...optimizers/trijoint.py.151: total_norm: 13.886912 [I 2022-10-09 02:14:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 770/2304 [I 2022-10-09 02:14:15] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:41 [I 2022-10-09 02:14:15] ...trap/engines/engine.py.229: process: 0.85581 | load: 0.00313 [I 2022-10-09 02:14:15] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-09 02:14:15] ...optimizers/trijoint.py.151: total_norm: 19.854488 [I 2022-10-09 02:14:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 780/2304 [I 2022-10-09 02:14:23] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:27 [I 2022-10-09 02:14:23] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00301 [I 2022-10-09 02:14:23] ...trap/engines/engine.py.230: loss: 0.14673 [I 2022-10-09 02:14:23] ...optimizers/trijoint.py.151: total_norm: 13.781634 [I 2022-10-09 02:14:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 790/2304 [I 2022-10-09 02:14:32] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:20 [I 2022-10-09 02:14:32] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00300 [I 2022-10-09 02:14:32] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-09 02:14:32] ...optimizers/trijoint.py.151: total_norm: 22.130127 [I 2022-10-09 02:14:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 800/2304 [I 2022-10-09 02:14:40] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:12 [I 2022-10-09 02:14:40] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00300 [I 2022-10-09 02:14:40] ...trap/engines/engine.py.230: loss: 0.25870 [I 2022-10-09 02:14:40] ...optimizers/trijoint.py.151: total_norm: 14.216719 [I 2022-10-09 02:14:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 810/2304 [I 2022-10-09 02:14:49] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:20:57 [I 2022-10-09 02:14:49] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00302 [I 2022-10-09 02:14:49] ...trap/engines/engine.py.230: loss: 0.17700 [I 2022-10-09 02:14:49] ...optimizers/trijoint.py.151: total_norm: 15.989793 [I 2022-10-09 02:14:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 820/2304 [I 2022-10-09 02:14:57] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:08 [I 2022-10-09 02:14:57] ...trap/engines/engine.py.229: process: 0.83121 | load: 0.00299 [I 2022-10-09 02:14:57] ...trap/engines/engine.py.230: loss: 0.12887 [I 2022-10-09 02:14:57] ...optimizers/trijoint.py.151: total_norm: 15.273584 [I 2022-10-09 02:15:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 830/2304 [I 2022-10-09 02:15:06] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:48 [I 2022-10-09 02:15:06] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00297 [I 2022-10-09 02:15:06] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-09 02:15:06] ...optimizers/trijoint.py.151: total_norm: 29.292202 [I 2022-10-09 02:15:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 840/2304 [I 2022-10-09 02:15:14] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:36 [I 2022-10-09 02:15:14] ...trap/engines/engine.py.229: process: 0.82950 | load: 0.00300 [I 2022-10-09 02:15:14] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-09 02:15:14] ...optimizers/trijoint.py.151: total_norm: 8.629145 [I 2022-10-09 02:15:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 850/2304 [I 2022-10-09 02:15:23] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:26 [I 2022-10-09 02:15:23] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00299 [I 2022-10-09 02:15:23] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-09 02:15:23] ...optimizers/trijoint.py.151: total_norm: 32.634403 [I 2022-10-09 02:15:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 860/2304 [I 2022-10-09 02:15:31] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:11 [I 2022-10-09 02:15:31] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00299 [I 2022-10-09 02:15:31] ...trap/engines/engine.py.230: loss: 0.18532 [I 2022-10-09 02:15:31] ...optimizers/trijoint.py.151: total_norm: 15.275071 [I 2022-10-09 02:15:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 870/2304 [I 2022-10-09 02:15:39] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:07 [I 2022-10-09 02:15:39] ...trap/engines/engine.py.229: process: 0.83629 | load: 0.00305 [I 2022-10-09 02:15:39] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-09 02:15:39] ...optimizers/trijoint.py.151: total_norm: 12.829266 [I 2022-10-09 02:15:48] ...trap/engines/engine.py.224: train: epoch 28 | batch 880/2304 [I 2022-10-09 02:15:48] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:19:55 [I 2022-10-09 02:15:48] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00301 [I 2022-10-09 02:15:48] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-09 02:15:48] ...optimizers/trijoint.py.151: total_norm: 12.993450 [I 2022-10-09 02:15:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 890/2304 [I 2022-10-09 02:15:56] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:47 [I 2022-10-09 02:15:56] ...trap/engines/engine.py.229: process: 0.82834 | load: 0.00300 [I 2022-10-09 02:15:56] ...trap/engines/engine.py.230: loss: 0.23167 [I 2022-10-09 02:15:56] ...optimizers/trijoint.py.151: total_norm: 17.714514 [I 2022-10-09 02:16:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 900/2304 [I 2022-10-09 02:16:05] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:41 [I 2022-10-09 02:16:05] ...trap/engines/engine.py.229: process: 0.82839 | load: 0.00296 [I 2022-10-09 02:16:05] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-09 02:16:05] ...optimizers/trijoint.py.151: total_norm: 18.278252 [I 2022-10-09 02:16:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 910/2304 [I 2022-10-09 02:16:13] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:36 [I 2022-10-09 02:16:13] ...trap/engines/engine.py.229: process: 0.84625 | load: 0.00299 [I 2022-10-09 02:16:13] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-09 02:16:13] ...optimizers/trijoint.py.151: total_norm: 11.886562 [I 2022-10-09 02:16:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 920/2304 [I 2022-10-09 02:16:22] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:25 [I 2022-10-09 02:16:22] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00300 [I 2022-10-09 02:16:22] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-09 02:16:22] ...optimizers/trijoint.py.151: total_norm: 16.959942 [I 2022-10-09 02:16:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 930/2304 [I 2022-10-09 02:16:30] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:14 [I 2022-10-09 02:16:30] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00301 [I 2022-10-09 02:16:30] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-09 02:16:30] ...optimizers/trijoint.py.151: total_norm: 14.733382 [I 2022-10-09 02:16:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 940/2304 [I 2022-10-09 02:16:38] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:04 [I 2022-10-09 02:16:38] ...trap/engines/engine.py.229: process: 0.82940 | load: 0.00300 [I 2022-10-09 02:16:38] ...trap/engines/engine.py.230: loss: 0.22429 [I 2022-10-09 02:16:38] ...optimizers/trijoint.py.151: total_norm: 24.326340 [I 2022-10-09 02:16:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 950/2304 [I 2022-10-09 02:16:47] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:18:59 [I 2022-10-09 02:16:47] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00304 [I 2022-10-09 02:16:47] ...trap/engines/engine.py.230: loss: 0.18960 [I 2022-10-09 02:16:47] ...optimizers/trijoint.py.151: total_norm: 9.873592 [I 2022-10-09 02:16:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 960/2304 [I 2022-10-09 02:16:55] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:18:47 [I 2022-10-09 02:16:55] ...trap/engines/engine.py.229: process: 0.83132 | load: 0.00299 [I 2022-10-09 02:16:55] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-09 02:16:55] ...optimizers/trijoint.py.151: total_norm: 16.082451 [I 2022-10-09 02:17:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 970/2304 [I 2022-10-09 02:17:04] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:42 [I 2022-10-09 02:17:04] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00299 [I 2022-10-09 02:17:04] ...trap/engines/engine.py.230: loss: 0.22535 [I 2022-10-09 02:17:04] ...optimizers/trijoint.py.151: total_norm: 19.670204 [I 2022-10-09 02:17:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 980/2304 [I 2022-10-09 02:17:12] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:30 [I 2022-10-09 02:17:12] ...trap/engines/engine.py.229: process: 0.82915 | load: 0.00298 [I 2022-10-09 02:17:12] ...trap/engines/engine.py.230: loss: 0.21666 [I 2022-10-09 02:17:12] ...optimizers/trijoint.py.151: total_norm: 23.010704 [I 2022-10-09 02:17:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 990/2304 [I 2022-10-09 02:17:20] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:25 [I 2022-10-09 02:17:20] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00298 [I 2022-10-09 02:17:20] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-09 02:17:20] ...optimizers/trijoint.py.151: total_norm: 12.645485 [I 2022-10-09 02:17:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 1000/2304 [I 2022-10-09 02:17:29] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:19 [I 2022-10-09 02:17:29] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00300 [I 2022-10-09 02:17:29] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-09 02:17:29] ...optimizers/trijoint.py.151: total_norm: 12.625890 [I 2022-10-09 02:17:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 1010/2304 [I 2022-10-09 02:17:37] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:03 [I 2022-10-09 02:17:37] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00296 [I 2022-10-09 02:17:37] ...trap/engines/engine.py.230: loss: 0.17501 [I 2022-10-09 02:17:37] ...optimizers/trijoint.py.151: total_norm: 15.487234 [I 2022-10-09 02:17:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 1020/2304 [I 2022-10-09 02:17:46] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:03 [I 2022-10-09 02:17:46] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00300 [I 2022-10-09 02:17:46] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-09 02:17:46] ...optimizers/trijoint.py.151: total_norm: 14.984607 [I 2022-10-09 02:17:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 1030/2304 [I 2022-10-09 02:17:54] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:17:47 [I 2022-10-09 02:17:54] ...trap/engines/engine.py.229: process: 0.83496 | load: 0.00298 [I 2022-10-09 02:17:54] ...trap/engines/engine.py.230: loss: 0.22811 [I 2022-10-09 02:17:54] ...optimizers/trijoint.py.151: total_norm: 23.409313 [I 2022-10-09 02:18:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 1040/2304 [I 2022-10-09 02:18:03] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:42 [I 2022-10-09 02:18:03] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00298 [I 2022-10-09 02:18:03] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-09 02:18:03] ...optimizers/trijoint.py.151: total_norm: 12.448044 [I 2022-10-09 02:18:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 1050/2304 [I 2022-10-09 02:18:11] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:27 [I 2022-10-09 02:18:11] ...trap/engines/engine.py.229: process: 0.83305 | load: 0.00306 [I 2022-10-09 02:18:11] ...trap/engines/engine.py.230: loss: 0.17198 [I 2022-10-09 02:18:11] ...optimizers/trijoint.py.151: total_norm: 19.436045 [I 2022-10-09 02:18:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 1060/2304 [I 2022-10-09 02:18:19] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:25 [I 2022-10-09 02:18:19] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00299 [I 2022-10-09 02:18:19] ...trap/engines/engine.py.230: loss: 0.11259 [I 2022-10-09 02:18:19] ...optimizers/trijoint.py.151: total_norm: 9.555134 [I 2022-10-09 02:18:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 1070/2304 [I 2022-10-09 02:18:28] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:12 [I 2022-10-09 02:18:28] ...trap/engines/engine.py.229: process: 0.82429 | load: 0.00303 [I 2022-10-09 02:18:28] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-09 02:18:28] ...optimizers/trijoint.py.151: total_norm: 12.626996 [I 2022-10-09 02:18:36] ...trap/engines/engine.py.224: train: epoch 28 | batch 1080/2304 [I 2022-10-09 02:18:36] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:09 [I 2022-10-09 02:18:36] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00298 [I 2022-10-09 02:18:36] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-09 02:18:36] ...optimizers/trijoint.py.151: total_norm: 16.987057 [I 2022-10-09 02:18:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 1090/2304 [I 2022-10-09 02:18:45] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:16:57 [I 2022-10-09 02:18:45] ...trap/engines/engine.py.229: process: 0.82901 | load: 0.00298 [I 2022-10-09 02:18:45] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-09 02:18:45] ...optimizers/trijoint.py.151: total_norm: 13.283866 [I 2022-10-09 02:18:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 1100/2304 [I 2022-10-09 02:18:53] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:16:50 [I 2022-10-09 02:18:53] ...trap/engines/engine.py.229: process: 0.83463 | load: 0.00301 [I 2022-10-09 02:18:53] ...trap/engines/engine.py.230: loss: 0.21891 [I 2022-10-09 02:18:53] ...optimizers/trijoint.py.151: total_norm: 20.218193 [I 2022-10-09 02:19:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 1110/2304 [I 2022-10-09 02:19:01] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:16:44 [I 2022-10-09 02:19:01] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00298 [I 2022-10-09 02:19:01] ...trap/engines/engine.py.230: loss: 0.17207 [I 2022-10-09 02:19:01] ...optimizers/trijoint.py.151: total_norm: 17.814932 [I 2022-10-09 02:19:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 1120/2304 [I 2022-10-09 02:19:10] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:31 [I 2022-10-09 02:19:10] ...trap/engines/engine.py.229: process: 0.83944 | load: 0.00299 [I 2022-10-09 02:19:10] ...trap/engines/engine.py.230: loss: 0.18903 [I 2022-10-09 02:19:10] ...optimizers/trijoint.py.151: total_norm: 15.223987 [I 2022-10-09 02:19:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 1130/2304 [I 2022-10-09 02:19:18] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:25 [I 2022-10-09 02:19:18] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00297 [I 2022-10-09 02:19:18] ...trap/engines/engine.py.230: loss: 0.17216 [I 2022-10-09 02:19:18] ...optimizers/trijoint.py.151: total_norm: 23.533697 [I 2022-10-09 02:19:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 1140/2304 [I 2022-10-09 02:19:27] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:19 [I 2022-10-09 02:19:27] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00301 [I 2022-10-09 02:19:27] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-09 02:19:27] ...optimizers/trijoint.py.151: total_norm: 11.518931 [I 2022-10-09 02:19:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 1150/2304 [I 2022-10-09 02:19:35] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:12 [I 2022-10-09 02:19:35] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00301 [I 2022-10-09 02:19:35] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-09 02:19:35] ...optimizers/trijoint.py.151: total_norm: 11.190422 [I 2022-10-09 02:19:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 1160/2304 [I 2022-10-09 02:19:44] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:06 [I 2022-10-09 02:19:44] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00300 [I 2022-10-09 02:19:44] ...trap/engines/engine.py.230: loss: 0.12011 [I 2022-10-09 02:19:44] ...optimizers/trijoint.py.151: total_norm: 19.073357 [I 2022-10-09 02:19:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 1170/2304 [I 2022-10-09 02:19:52] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:15:54 [I 2022-10-09 02:19:52] ...trap/engines/engine.py.229: process: 0.83811 | load: 0.00298 [I 2022-10-09 02:19:52] ...trap/engines/engine.py.230: loss: 0.27897 [I 2022-10-09 02:19:52] ...optimizers/trijoint.py.151: total_norm: 34.489407 [I 2022-10-09 02:20:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 1180/2304 [I 2022-10-09 02:20:00] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:15:44 [I 2022-10-09 02:20:00] ...trap/engines/engine.py.229: process: 0.83797 | load: 0.00298 [I 2022-10-09 02:20:00] ...trap/engines/engine.py.230: loss: 0.18922 [I 2022-10-09 02:20:00] ...optimizers/trijoint.py.151: total_norm: 17.002323 [I 2022-10-09 02:20:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 1190/2304 [I 2022-10-09 02:20:09] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:37 [I 2022-10-09 02:20:09] ...trap/engines/engine.py.229: process: 0.83770 | load: 0.00300 [I 2022-10-09 02:20:09] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-09 02:20:09] ...optimizers/trijoint.py.151: total_norm: 23.277422 [I 2022-10-09 02:20:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 1200/2304 [I 2022-10-09 02:20:17] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:15:27 [I 2022-10-09 02:20:17] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00298 [I 2022-10-09 02:20:17] ...trap/engines/engine.py.230: loss: 0.13524 [I 2022-10-09 02:20:17] ...optimizers/trijoint.py.151: total_norm: 12.100163 [I 2022-10-09 02:20:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 1210/2304 [I 2022-10-09 02:20:26] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:18 [I 2022-10-09 02:20:26] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00297 [I 2022-10-09 02:20:26] ...trap/engines/engine.py.230: loss: 0.17453 [I 2022-10-09 02:20:26] ...optimizers/trijoint.py.151: total_norm: 27.221960 [I 2022-10-09 02:20:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 1220/2304 [I 2022-10-09 02:20:34] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:10 [I 2022-10-09 02:20:34] ...trap/engines/engine.py.229: process: 0.82547 | load: 0.00300 [I 2022-10-09 02:20:34] ...trap/engines/engine.py.230: loss: 0.12855 [I 2022-10-09 02:20:34] ...optimizers/trijoint.py.151: total_norm: 22.526876 [I 2022-10-09 02:20:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 1230/2304 [I 2022-10-09 02:20:42] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:14:56 [I 2022-10-09 02:20:42] ...trap/engines/engine.py.229: process: 0.82639 | load: 0.00301 [I 2022-10-09 02:20:42] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-09 02:20:42] ...optimizers/trijoint.py.151: total_norm: 14.515127 [I 2022-10-09 02:20:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 1240/2304 [I 2022-10-09 02:20:51] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:00 [I 2022-10-09 02:20:51] ...trap/engines/engine.py.229: process: 0.85536 | load: 0.00295 [I 2022-10-09 02:20:51] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-09 02:20:51] ...optimizers/trijoint.py.151: total_norm: 17.313505 [I 2022-10-09 02:20:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 1250/2304 [I 2022-10-09 02:20:59] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:14:49 [I 2022-10-09 02:20:59] ...trap/engines/engine.py.229: process: 0.84744 | load: 0.00298 [I 2022-10-09 02:20:59] ...trap/engines/engine.py.230: loss: 0.19755 [I 2022-10-09 02:20:59] ...optimizers/trijoint.py.151: total_norm: 16.767925 [I 2022-10-09 02:21:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 1260/2304 [I 2022-10-09 02:21:08] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:39 [I 2022-10-09 02:21:08] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00300 [I 2022-10-09 02:21:08] ...trap/engines/engine.py.230: loss: 0.23387 [I 2022-10-09 02:21:08] ...optimizers/trijoint.py.151: total_norm: 17.257267 [I 2022-10-09 02:21:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 1270/2304 [I 2022-10-09 02:21:16] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:30 [I 2022-10-09 02:21:16] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00298 [I 2022-10-09 02:21:16] ...trap/engines/engine.py.230: loss: 0.12395 [I 2022-10-09 02:21:16] ...optimizers/trijoint.py.151: total_norm: 11.125384 [I 2022-10-09 02:21:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 1280/2304 [I 2022-10-09 02:21:25] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:22 [I 2022-10-09 02:21:25] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00299 [I 2022-10-09 02:21:25] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-09 02:21:25] ...optimizers/trijoint.py.151: total_norm: 14.455201 [I 2022-10-09 02:21:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 1290/2304 [I 2022-10-09 02:21:33] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:13 [I 2022-10-09 02:21:33] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00298 [I 2022-10-09 02:21:33] ...trap/engines/engine.py.230: loss: 0.21274 [I 2022-10-09 02:21:33] ...optimizers/trijoint.py.151: total_norm: 15.667786 [I 2022-10-09 02:21:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 1300/2304 [I 2022-10-09 02:21:41] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:03 [I 2022-10-09 02:21:41] ...trap/engines/engine.py.229: process: 0.83793 | load: 0.00308 [I 2022-10-09 02:21:41] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-09 02:21:41] ...optimizers/trijoint.py.151: total_norm: 19.994713 [I 2022-10-09 02:21:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 1310/2304 [I 2022-10-09 02:21:50] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:13:50 [I 2022-10-09 02:21:50] ...trap/engines/engine.py.229: process: 0.82058 | load: 0.00302 [I 2022-10-09 02:21:50] ...trap/engines/engine.py.230: loss: 0.13964 [I 2022-10-09 02:21:50] ...optimizers/trijoint.py.151: total_norm: 16.912928 [I 2022-10-09 02:21:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 1320/2304 [I 2022-10-09 02:21:58] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:13:45 [I 2022-10-09 02:21:58] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00299 [I 2022-10-09 02:21:58] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-09 02:21:58] ...optimizers/trijoint.py.151: total_norm: 8.494616 [I 2022-10-09 02:22:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 1330/2304 [I 2022-10-09 02:22:07] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:40 [I 2022-10-09 02:22:07] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00303 [I 2022-10-09 02:22:07] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-09 02:22:07] ...optimizers/trijoint.py.151: total_norm: 15.541864 [I 2022-10-09 02:22:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 1340/2304 [I 2022-10-09 02:22:15] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:30 [I 2022-10-09 02:22:15] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00304 [I 2022-10-09 02:22:15] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-09 02:22:15] ...optimizers/trijoint.py.151: total_norm: 10.393227 [I 2022-10-09 02:22:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 1350/2304 [I 2022-10-09 02:22:23] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:13:18 [I 2022-10-09 02:22:23] ...trap/engines/engine.py.229: process: 0.82750 | load: 0.00299 [I 2022-10-09 02:22:23] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-09 02:22:23] ...optimizers/trijoint.py.151: total_norm: 22.300867 [I 2022-10-09 02:22:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 1360/2304 [I 2022-10-09 02:22:32] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:12 [I 2022-10-09 02:22:32] ...trap/engines/engine.py.229: process: 0.83597 | load: 0.00298 [I 2022-10-09 02:22:32] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-09 02:22:32] ...optimizers/trijoint.py.151: total_norm: 15.247277 [I 2022-10-09 02:22:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 1370/2304 [I 2022-10-09 02:22:40] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:04 [I 2022-10-09 02:22:40] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00303 [I 2022-10-09 02:22:40] ...trap/engines/engine.py.230: loss: 0.22465 [I 2022-10-09 02:22:40] ...optimizers/trijoint.py.151: total_norm: 14.650635 [I 2022-10-09 02:22:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 1380/2304 [I 2022-10-09 02:22:49] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:12:56 [I 2022-10-09 02:22:49] ...trap/engines/engine.py.229: process: 0.84040 | load: 0.00303 [I 2022-10-09 02:22:49] ...trap/engines/engine.py.230: loss: 0.19922 [I 2022-10-09 02:22:49] ...optimizers/trijoint.py.151: total_norm: 21.311853 [I 2022-10-09 02:22:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 1390/2304 [I 2022-10-09 02:22:57] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:12:49 [I 2022-10-09 02:22:57] ...trap/engines/engine.py.229: process: 0.82858 | load: 0.00300 [I 2022-10-09 02:22:57] ...trap/engines/engine.py.230: loss: 0.22820 [I 2022-10-09 02:22:57] ...optimizers/trijoint.py.151: total_norm: 36.077061 [I 2022-10-09 02:23:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 1400/2304 [I 2022-10-09 02:23:06] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:12:36 [I 2022-10-09 02:23:06] ...trap/engines/engine.py.229: process: 0.83399 | load: 0.00301 [I 2022-10-09 02:23:06] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-09 02:23:06] ...optimizers/trijoint.py.151: total_norm: 14.869748 [I 2022-10-09 02:23:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 1410/2304 [I 2022-10-09 02:23:14] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:12:30 [I 2022-10-09 02:23:14] ...trap/engines/engine.py.229: process: 0.83393 | load: 0.00300 [I 2022-10-09 02:23:14] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-09 02:23:14] ...optimizers/trijoint.py.151: total_norm: 19.310432 [I 2022-10-09 02:23:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 1420/2304 [I 2022-10-09 02:23:22] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:22 [I 2022-10-09 02:23:22] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00297 [I 2022-10-09 02:23:22] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-09 02:23:22] ...optimizers/trijoint.py.151: total_norm: 21.395302 [I 2022-10-09 02:23:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 1430/2304 [I 2022-10-09 02:23:31] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:15 [I 2022-10-09 02:23:31] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00298 [I 2022-10-09 02:23:31] ...trap/engines/engine.py.230: loss: 0.17057 [I 2022-10-09 02:23:31] ...optimizers/trijoint.py.151: total_norm: 13.660958 [I 2022-10-09 02:23:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 1440/2304 [I 2022-10-09 02:23:39] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:08 [I 2022-10-09 02:23:39] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00300 [I 2022-10-09 02:23:39] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-09 02:23:39] ...optimizers/trijoint.py.151: total_norm: 22.965334 [I 2022-10-09 02:23:48] ...trap/engines/engine.py.224: train: epoch 28 | batch 1450/2304 [I 2022-10-09 02:23:48] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:11:58 [I 2022-10-09 02:23:48] ...trap/engines/engine.py.229: process: 0.83490 | load: 0.00299 [I 2022-10-09 02:23:48] ...trap/engines/engine.py.230: loss: 0.08655 [I 2022-10-09 02:23:48] ...optimizers/trijoint.py.151: total_norm: 9.094559 [I 2022-10-09 02:23:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 1460/2304 [I 2022-10-09 02:23:56] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:11:50 [I 2022-10-09 02:23:56] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00302 [I 2022-10-09 02:23:56] ...trap/engines/engine.py.230: loss: 0.27099 [I 2022-10-09 02:23:56] ...optimizers/trijoint.py.151: total_norm: 15.168706 [I 2022-10-09 02:24:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 1470/2304 [I 2022-10-09 02:24:05] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:11:40 [I 2022-10-09 02:24:05] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00301 [I 2022-10-09 02:24:05] ...trap/engines/engine.py.230: loss: 0.21693 [I 2022-10-09 02:24:05] ...optimizers/trijoint.py.151: total_norm: 20.642179 [I 2022-10-09 02:24:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 1480/2304 [I 2022-10-09 02:24:13] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:32 [I 2022-10-09 02:24:13] ...trap/engines/engine.py.229: process: 0.84138 | load: 0.00299 [I 2022-10-09 02:24:13] ...trap/engines/engine.py.230: loss: 0.18936 [I 2022-10-09 02:24:13] ...optimizers/trijoint.py.151: total_norm: 16.173952 [I 2022-10-09 02:24:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 1490/2304 [I 2022-10-09 02:24:21] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:23 [I 2022-10-09 02:24:21] ...trap/engines/engine.py.229: process: 0.84140 | load: 0.00301 [I 2022-10-09 02:24:21] ...trap/engines/engine.py.230: loss: 0.19472 [I 2022-10-09 02:24:21] ...optimizers/trijoint.py.151: total_norm: 28.102886 [I 2022-10-09 02:24:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 1500/2304 [I 2022-10-09 02:24:30] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:11:14 [I 2022-10-09 02:24:30] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00300 [I 2022-10-09 02:24:30] ...trap/engines/engine.py.230: loss: 0.35238 [I 2022-10-09 02:24:30] ...optimizers/trijoint.py.151: total_norm: 18.978458 [I 2022-10-09 02:24:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 1510/2304 [I 2022-10-09 02:24:38] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:01 [I 2022-10-09 02:24:38] ...trap/engines/engine.py.229: process: 0.83057 | load: 0.00299 [I 2022-10-09 02:24:38] ...trap/engines/engine.py.230: loss: 0.14976 [I 2022-10-09 02:24:38] ...optimizers/trijoint.py.151: total_norm: 15.689009 [I 2022-10-09 02:24:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 1520/2304 [I 2022-10-09 02:24:47] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:10:56 [I 2022-10-09 02:24:47] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00303 [I 2022-10-09 02:24:47] ...trap/engines/engine.py.230: loss: 0.20754 [I 2022-10-09 02:24:47] ...optimizers/trijoint.py.151: total_norm: 13.268830 [I 2022-10-09 02:24:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 1530/2304 [I 2022-10-09 02:24:55] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:10:46 [I 2022-10-09 02:24:55] ...trap/engines/engine.py.229: process: 0.83514 | load: 0.00302 [I 2022-10-09 02:24:55] ...trap/engines/engine.py.230: loss: 0.15514 [I 2022-10-09 02:24:55] ...optimizers/trijoint.py.151: total_norm: 11.712145 [I 2022-10-09 02:25:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 1540/2304 [I 2022-10-09 02:25:03] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:10:37 [I 2022-10-09 02:25:03] ...trap/engines/engine.py.229: process: 0.82342 | load: 0.00310 [I 2022-10-09 02:25:03] ...trap/engines/engine.py.230: loss: 0.17474 [I 2022-10-09 02:25:03] ...optimizers/trijoint.py.151: total_norm: 30.723104 [I 2022-10-09 02:25:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 1550/2304 [I 2022-10-09 02:25:12] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:10:32 [I 2022-10-09 02:25:12] ...trap/engines/engine.py.229: process: 0.84999 | load: 0.00299 [I 2022-10-09 02:25:12] ...trap/engines/engine.py.230: loss: 0.10791 [I 2022-10-09 02:25:12] ...optimizers/trijoint.py.151: total_norm: 32.525780 [I 2022-10-09 02:25:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 1560/2304 [I 2022-10-09 02:25:20] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:24 [I 2022-10-09 02:25:20] ...trap/engines/engine.py.229: process: 0.83048 | load: 0.00298 [I 2022-10-09 02:25:20] ...trap/engines/engine.py.230: loss: 0.16682 [I 2022-10-09 02:25:20] ...optimizers/trijoint.py.151: total_norm: 20.071577 [I 2022-10-09 02:25:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 1570/2304 [I 2022-10-09 02:25:29] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:15 [I 2022-10-09 02:25:29] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00299 [I 2022-10-09 02:25:29] ...trap/engines/engine.py.230: loss: 0.17394 [I 2022-10-09 02:25:29] ...optimizers/trijoint.py.151: total_norm: 18.359892 [I 2022-10-09 02:25:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 1580/2304 [I 2022-10-09 02:25:37] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:04 [I 2022-10-09 02:25:37] ...trap/engines/engine.py.229: process: 0.83305 | load: 0.00299 [I 2022-10-09 02:25:37] ...trap/engines/engine.py.230: loss: 0.16832 [I 2022-10-09 02:25:37] ...optimizers/trijoint.py.151: total_norm: 22.769503 [I 2022-10-09 02:25:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 1590/2304 [I 2022-10-09 02:25:45] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:00 [I 2022-10-09 02:25:45] ...trap/engines/engine.py.229: process: 0.84171 | load: 0.00300 [I 2022-10-09 02:25:45] ...trap/engines/engine.py.230: loss: 0.13756 [I 2022-10-09 02:25:45] ...optimizers/trijoint.py.151: total_norm: 22.110764 [I 2022-10-09 02:25:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 1600/2304 [I 2022-10-09 02:25:54] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:09:51 [I 2022-10-09 02:25:54] ...trap/engines/engine.py.229: process: 0.84231 | load: 0.00298 [I 2022-10-09 02:25:54] ...trap/engines/engine.py.230: loss: 0.17164 [I 2022-10-09 02:25:54] ...optimizers/trijoint.py.151: total_norm: 12.928676 [I 2022-10-09 02:26:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 1610/2304 [I 2022-10-09 02:26:02] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:09:43 [I 2022-10-09 02:26:02] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00302 [I 2022-10-09 02:26:02] ...trap/engines/engine.py.230: loss: 0.17326 [I 2022-10-09 02:26:02] ...optimizers/trijoint.py.151: total_norm: 27.298817 [I 2022-10-09 02:26:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 1620/2304 [I 2022-10-09 02:26:11] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:34 [I 2022-10-09 02:26:11] ...trap/engines/engine.py.229: process: 0.84778 | load: 0.00304 [I 2022-10-09 02:26:11] ...trap/engines/engine.py.230: loss: 0.22423 [I 2022-10-09 02:26:11] ...optimizers/trijoint.py.151: total_norm: 26.354937 [I 2022-10-09 02:26:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 1630/2304 [I 2022-10-09 02:26:19] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:27 [I 2022-10-09 02:26:19] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00300 [I 2022-10-09 02:26:19] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-09 02:26:19] ...optimizers/trijoint.py.151: total_norm: 15.459108 [I 2022-10-09 02:26:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 1640/2304 [I 2022-10-09 02:26:27] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:19 [I 2022-10-09 02:26:27] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00300 [I 2022-10-09 02:26:27] ...trap/engines/engine.py.230: loss: 0.16226 [I 2022-10-09 02:26:27] ...optimizers/trijoint.py.151: total_norm: 14.584869 [I 2022-10-09 02:26:36] ...trap/engines/engine.py.224: train: epoch 28 | batch 1650/2304 [I 2022-10-09 02:26:36] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:14 [I 2022-10-09 02:26:36] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00299 [I 2022-10-09 02:26:36] ...trap/engines/engine.py.230: loss: 0.19956 [I 2022-10-09 02:26:36] ...optimizers/trijoint.py.151: total_norm: 20.891769 [I 2022-10-09 02:26:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 1660/2304 [I 2022-10-09 02:26:44] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:03 [I 2022-10-09 02:26:44] ...trap/engines/engine.py.229: process: 0.83186 | load: 0.00303 [I 2022-10-09 02:26:44] ...trap/engines/engine.py.230: loss: 0.23348 [I 2022-10-09 02:26:44] ...optimizers/trijoint.py.151: total_norm: 15.981936 [I 2022-10-09 02:26:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 1670/2304 [I 2022-10-09 02:26:53] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:08:51 [I 2022-10-09 02:26:53] ...trap/engines/engine.py.229: process: 0.83479 | load: 0.00299 [I 2022-10-09 02:26:53] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-09 02:26:53] ...optimizers/trijoint.py.151: total_norm: 24.454628 [I 2022-10-09 02:27:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 1680/2304 [I 2022-10-09 02:27:01] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:08:44 [I 2022-10-09 02:27:01] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00300 [I 2022-10-09 02:27:01] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-09 02:27:01] ...optimizers/trijoint.py.151: total_norm: 15.173430 [I 2022-10-09 02:27:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 1690/2304 [I 2022-10-09 02:27:10] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:34 [I 2022-10-09 02:27:10] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00298 [I 2022-10-09 02:27:10] ...trap/engines/engine.py.230: loss: 0.14664 [I 2022-10-09 02:27:10] ...optimizers/trijoint.py.151: total_norm: 22.171972 [I 2022-10-09 02:27:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 1700/2304 [I 2022-10-09 02:27:18] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:28 [I 2022-10-09 02:27:18] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00313 [I 2022-10-09 02:27:18] ...trap/engines/engine.py.230: loss: 0.16882 [I 2022-10-09 02:27:18] ...optimizers/trijoint.py.151: total_norm: 23.169220 [I 2022-10-09 02:27:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 1710/2304 [I 2022-10-09 02:27:26] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:17 [I 2022-10-09 02:27:26] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00310 [I 2022-10-09 02:27:26] ...trap/engines/engine.py.230: loss: 0.15479 [I 2022-10-09 02:27:26] ...optimizers/trijoint.py.151: total_norm: 13.394093 [I 2022-10-09 02:27:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 1720/2304 [I 2022-10-09 02:27:35] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:13 [I 2022-10-09 02:27:35] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00305 [I 2022-10-09 02:27:35] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-09 02:27:35] ...optimizers/trijoint.py.151: total_norm: 13.708511 [I 2022-10-09 02:27:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 1730/2304 [I 2022-10-09 02:27:43] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:01 [I 2022-10-09 02:27:43] ...trap/engines/engine.py.229: process: 0.83052 | load: 0.00305 [I 2022-10-09 02:27:43] ...trap/engines/engine.py.230: loss: 0.14247 [I 2022-10-09 02:27:43] ...optimizers/trijoint.py.151: total_norm: 11.416756 [I 2022-10-09 02:27:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 1740/2304 [I 2022-10-09 02:27:52] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:07:53 [I 2022-10-09 02:27:52] ...trap/engines/engine.py.229: process: 0.83879 | load: 0.00300 [I 2022-10-09 02:27:52] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-09 02:27:52] ...optimizers/trijoint.py.151: total_norm: 10.582373 [I 2022-10-09 02:28:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 1750/2304 [I 2022-10-09 02:28:00] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:07:47 [I 2022-10-09 02:28:00] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00299 [I 2022-10-09 02:28:00] ...trap/engines/engine.py.230: loss: 0.18976 [I 2022-10-09 02:28:00] ...optimizers/trijoint.py.151: total_norm: 12.037148 [I 2022-10-09 02:28:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 1760/2304 [I 2022-10-09 02:28:08] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:37 [I 2022-10-09 02:28:08] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00300 [I 2022-10-09 02:28:08] ...trap/engines/engine.py.230: loss: 0.18251 [I 2022-10-09 02:28:08] ...optimizers/trijoint.py.151: total_norm: 12.388852 [I 2022-10-09 02:28:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 1770/2304 [I 2022-10-09 02:28:17] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:30 [I 2022-10-09 02:28:17] ...trap/engines/engine.py.229: process: 0.84718 | load: 0.00305 [I 2022-10-09 02:28:17] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-09 02:28:17] ...optimizers/trijoint.py.151: total_norm: 17.000729 [I 2022-10-09 02:28:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 1780/2304 [I 2022-10-09 02:28:25] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:22 [I 2022-10-09 02:28:25] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00303 [I 2022-10-09 02:28:25] ...trap/engines/engine.py.230: loss: 0.20122 [I 2022-10-09 02:28:25] ...optimizers/trijoint.py.151: total_norm: 22.490440 [I 2022-10-09 02:28:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 1790/2304 [I 2022-10-09 02:28:34] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:14 [I 2022-10-09 02:28:34] ...trap/engines/engine.py.229: process: 0.84988 | load: 0.00302 [I 2022-10-09 02:28:34] ...trap/engines/engine.py.230: loss: 0.18317 [I 2022-10-09 02:28:34] ...optimizers/trijoint.py.151: total_norm: 19.457977 [I 2022-10-09 02:28:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 1800/2304 [I 2022-10-09 02:28:42] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:03 [I 2022-10-09 02:28:42] ...trap/engines/engine.py.229: process: 0.83596 | load: 0.00304 [I 2022-10-09 02:28:42] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-09 02:28:42] ...optimizers/trijoint.py.151: total_norm: 9.472095 [I 2022-10-09 02:28:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 1810/2304 [I 2022-10-09 02:28:51] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:06:54 [I 2022-10-09 02:28:51] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00301 [I 2022-10-09 02:28:51] ...trap/engines/engine.py.230: loss: 0.20027 [I 2022-10-09 02:28:51] ...optimizers/trijoint.py.151: total_norm: 12.685303 [I 2022-10-09 02:28:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 1820/2304 [I 2022-10-09 02:28:59] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:06:45 [I 2022-10-09 02:28:59] ...trap/engines/engine.py.229: process: 0.82642 | load: 0.00299 [I 2022-10-09 02:28:59] ...trap/engines/engine.py.230: loss: 0.19234 [I 2022-10-09 02:28:59] ...optimizers/trijoint.py.151: total_norm: 20.133318 [I 2022-10-09 02:29:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 1830/2304 [I 2022-10-09 02:29:07] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:38 [I 2022-10-09 02:29:07] ...trap/engines/engine.py.229: process: 0.84101 | load: 0.00301 [I 2022-10-09 02:29:07] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-09 02:29:07] ...optimizers/trijoint.py.151: total_norm: 22.003319 [I 2022-10-09 02:29:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 1840/2304 [I 2022-10-09 02:29:16] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:28 [I 2022-10-09 02:29:16] ...trap/engines/engine.py.229: process: 0.81942 | load: 0.00301 [I 2022-10-09 02:29:16] ...trap/engines/engine.py.230: loss: 0.17637 [I 2022-10-09 02:29:16] ...optimizers/trijoint.py.151: total_norm: 15.339335 [I 2022-10-09 02:29:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 1850/2304 [I 2022-10-09 02:29:24] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:21 [I 2022-10-09 02:29:24] ...trap/engines/engine.py.229: process: 0.83185 | load: 0.00298 [I 2022-10-09 02:29:24] ...trap/engines/engine.py.230: loss: 0.22014 [I 2022-10-09 02:29:24] ...optimizers/trijoint.py.151: total_norm: 77.683823 [I 2022-10-09 02:29:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 1860/2304 [I 2022-10-09 02:29:33] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:09 [I 2022-10-09 02:29:33] ...trap/engines/engine.py.229: process: 0.83142 | load: 0.00298 [I 2022-10-09 02:29:33] ...trap/engines/engine.py.230: loss: 0.20004 [I 2022-10-09 02:29:33] ...optimizers/trijoint.py.151: total_norm: 21.066196 [I 2022-10-09 02:29:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 1870/2304 [I 2022-10-09 02:29:41] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:07 [I 2022-10-09 02:29:41] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00299 [I 2022-10-09 02:29:41] ...trap/engines/engine.py.230: loss: 0.15576 [I 2022-10-09 02:29:41] ...optimizers/trijoint.py.151: total_norm: 13.429916 [I 2022-10-09 02:29:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 1880/2304 [I 2022-10-09 02:29:50] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:05:56 [I 2022-10-09 02:29:50] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00302 [I 2022-10-09 02:29:50] ...trap/engines/engine.py.230: loss: 0.25523 [I 2022-10-09 02:29:50] ...optimizers/trijoint.py.151: total_norm: 26.953955 [I 2022-10-09 02:29:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 1890/2304 [I 2022-10-09 02:29:58] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:05:48 [I 2022-10-09 02:29:58] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00298 [I 2022-10-09 02:29:58] ...trap/engines/engine.py.230: loss: 0.17143 [I 2022-10-09 02:29:58] ...optimizers/trijoint.py.151: total_norm: 12.525680 [I 2022-10-09 02:30:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 1900/2304 [I 2022-10-09 02:30:06] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:40 [I 2022-10-09 02:30:06] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00304 [I 2022-10-09 02:30:06] ...trap/engines/engine.py.230: loss: 0.11997 [I 2022-10-09 02:30:06] ...optimizers/trijoint.py.151: total_norm: 15.735636 [I 2022-10-09 02:30:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 1910/2304 [I 2022-10-09 02:30:15] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:32 [I 2022-10-09 02:30:15] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00302 [I 2022-10-09 02:30:15] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-09 02:30:15] ...optimizers/trijoint.py.151: total_norm: 16.052530 [I 2022-10-09 02:30:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 1920/2304 [I 2022-10-09 02:30:23] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:22 [I 2022-10-09 02:30:23] ...trap/engines/engine.py.229: process: 0.83160 | load: 0.00300 [I 2022-10-09 02:30:23] ...trap/engines/engine.py.230: loss: 0.13449 [I 2022-10-09 02:30:23] ...optimizers/trijoint.py.151: total_norm: 14.015172 [I 2022-10-09 02:30:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 1930/2304 [I 2022-10-09 02:30:32] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:15 [I 2022-10-09 02:30:32] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00303 [I 2022-10-09 02:30:32] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-09 02:30:32] ...optimizers/trijoint.py.151: total_norm: 12.777479 [I 2022-10-09 02:30:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 1940/2304 [I 2022-10-09 02:30:40] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:05 [I 2022-10-09 02:30:40] ...trap/engines/engine.py.229: process: 0.83058 | load: 0.00298 [I 2022-10-09 02:30:40] ...trap/engines/engine.py.230: loss: 0.24508 [I 2022-10-09 02:30:40] ...optimizers/trijoint.py.151: total_norm: 21.008526 [I 2022-10-09 02:30:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 1950/2304 [I 2022-10-09 02:30:49] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:04:56 [I 2022-10-09 02:30:49] ...trap/engines/engine.py.229: process: 0.83330 | load: 0.00300 [I 2022-10-09 02:30:49] ...trap/engines/engine.py.230: loss: 0.18242 [I 2022-10-09 02:30:49] ...optimizers/trijoint.py.151: total_norm: 15.540519 [I 2022-10-09 02:30:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 1960/2304 [I 2022-10-09 02:30:57] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:04:47 [I 2022-10-09 02:30:57] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00299 [I 2022-10-09 02:30:57] ...trap/engines/engine.py.230: loss: 0.19679 [I 2022-10-09 02:30:57] ...optimizers/trijoint.py.151: total_norm: 44.445576 [I 2022-10-09 02:31:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 1970/2304 [I 2022-10-09 02:31:05] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:40 [I 2022-10-09 02:31:05] ...trap/engines/engine.py.229: process: 0.83413 | load: 0.00311 [I 2022-10-09 02:31:05] ...trap/engines/engine.py.230: loss: 0.18221 [I 2022-10-09 02:31:05] ...optimizers/trijoint.py.151: total_norm: 30.328205 [I 2022-10-09 02:31:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 1980/2304 [I 2022-10-09 02:31:14] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:32 [I 2022-10-09 02:31:14] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00303 [I 2022-10-09 02:31:14] ...trap/engines/engine.py.230: loss: 0.18870 [I 2022-10-09 02:31:14] ...optimizers/trijoint.py.151: total_norm: 20.214512 [I 2022-10-09 02:31:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 1990/2304 [I 2022-10-09 02:31:22] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:23 [I 2022-10-09 02:31:22] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00301 [I 2022-10-09 02:31:22] ...trap/engines/engine.py.230: loss: 0.19911 [I 2022-10-09 02:31:22] ...optimizers/trijoint.py.151: total_norm: 28.598980 [I 2022-10-09 02:31:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 2000/2304 [I 2022-10-09 02:31:31] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:14 [I 2022-10-09 02:31:31] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00310 [I 2022-10-09 02:31:31] ...trap/engines/engine.py.230: loss: 0.16307 [I 2022-10-09 02:31:31] ...optimizers/trijoint.py.151: total_norm: 18.869902 [I 2022-10-09 02:31:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 2010/2304 [I 2022-10-09 02:31:39] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:06 [I 2022-10-09 02:31:39] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00302 [I 2022-10-09 02:31:39] ...trap/engines/engine.py.230: loss: 0.17227 [I 2022-10-09 02:31:39] ...optimizers/trijoint.py.151: total_norm: 18.644243 [I 2022-10-09 02:31:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 2020/2304 [I 2022-10-09 02:31:47] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:03:56 [I 2022-10-09 02:31:47] ...trap/engines/engine.py.229: process: 0.82374 | load: 0.00299 [I 2022-10-09 02:31:47] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-09 02:31:47] ...optimizers/trijoint.py.151: total_norm: 30.924765 [I 2022-10-09 02:31:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 2030/2304 [I 2022-10-09 02:31:56] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:03:49 [I 2022-10-09 02:31:56] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00301 [I 2022-10-09 02:31:56] ...trap/engines/engine.py.230: loss: 0.18093 [I 2022-10-09 02:31:56] ...optimizers/trijoint.py.151: total_norm: 15.216412 [I 2022-10-09 02:32:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 2040/2304 [I 2022-10-09 02:32:04] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:40 [I 2022-10-09 02:32:04] ...trap/engines/engine.py.229: process: 0.83291 | load: 0.00299 [I 2022-10-09 02:32:04] ...trap/engines/engine.py.230: loss: 0.24258 [I 2022-10-09 02:32:04] ...optimizers/trijoint.py.151: total_norm: 22.661005 [I 2022-10-09 02:32:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 2050/2304 [I 2022-10-09 02:32:12] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:33 [I 2022-10-09 02:32:12] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00302 [I 2022-10-09 02:32:12] ...trap/engines/engine.py.230: loss: 0.14674 [I 2022-10-09 02:32:12] ...optimizers/trijoint.py.151: total_norm: 19.437115 [I 2022-10-09 02:32:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 2060/2304 [I 2022-10-09 02:32:21] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:25 [I 2022-10-09 02:32:21] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00302 [I 2022-10-09 02:32:21] ...trap/engines/engine.py.230: loss: 0.17754 [I 2022-10-09 02:32:21] ...optimizers/trijoint.py.151: total_norm: 21.552021 [I 2022-10-09 02:32:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 2070/2304 [I 2022-10-09 02:32:29] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:16 [I 2022-10-09 02:32:29] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00301 [I 2022-10-09 02:32:29] ...trap/engines/engine.py.230: loss: 0.20517 [I 2022-10-09 02:32:29] ...optimizers/trijoint.py.151: total_norm: 23.228224 [I 2022-10-09 02:32:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 2080/2304 [I 2022-10-09 02:32:38] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:08 [I 2022-10-09 02:32:38] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00300 [I 2022-10-09 02:32:38] ...trap/engines/engine.py.230: loss: 0.25778 [I 2022-10-09 02:32:38] ...optimizers/trijoint.py.151: total_norm: 18.223236 [I 2022-10-09 02:32:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 2090/2304 [I 2022-10-09 02:32:46] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:02:59 [I 2022-10-09 02:32:46] ...trap/engines/engine.py.229: process: 0.83288 | load: 0.00300 [I 2022-10-09 02:32:46] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-09 02:32:46] ...optimizers/trijoint.py.151: total_norm: 13.988040 [I 2022-10-09 02:32:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 2100/2304 [I 2022-10-09 02:32:55] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:02:50 [I 2022-10-09 02:32:55] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00300 [I 2022-10-09 02:32:55] ...trap/engines/engine.py.230: loss: 0.14335 [I 2022-10-09 02:32:55] ...optimizers/trijoint.py.151: total_norm: 13.349223 [I 2022-10-09 02:33:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 2110/2304 [I 2022-10-09 02:33:03] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:43 [I 2022-10-09 02:33:03] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00299 [I 2022-10-09 02:33:03] ...trap/engines/engine.py.230: loss: 0.17690 [I 2022-10-09 02:33:03] ...optimizers/trijoint.py.151: total_norm: 20.024866 [I 2022-10-09 02:33:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 2120/2304 [I 2022-10-09 02:33:11] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:35 [I 2022-10-09 02:33:11] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00300 [I 2022-10-09 02:33:11] ...trap/engines/engine.py.230: loss: 0.11822 [I 2022-10-09 02:33:11] ...optimizers/trijoint.py.151: total_norm: 14.400327 [I 2022-10-09 02:33:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 2130/2304 [I 2022-10-09 02:33:20] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:25 [I 2022-10-09 02:33:20] ...trap/engines/engine.py.229: process: 0.83064 | load: 0.00298 [I 2022-10-09 02:33:20] ...trap/engines/engine.py.230: loss: 0.17402 [I 2022-10-09 02:33:20] ...optimizers/trijoint.py.151: total_norm: 10.982011 [I 2022-10-09 02:33:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 2140/2304 [I 2022-10-09 02:33:28] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:16 [I 2022-10-09 02:33:28] ...trap/engines/engine.py.229: process: 0.82240 | load: 0.00303 [I 2022-10-09 02:33:28] ...trap/engines/engine.py.230: loss: 0.14257 [I 2022-10-09 02:33:28] ...optimizers/trijoint.py.151: total_norm: 14.162103 [I 2022-10-09 02:33:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 2150/2304 [I 2022-10-09 02:33:37] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:09 [I 2022-10-09 02:33:37] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00300 [I 2022-10-09 02:33:37] ...trap/engines/engine.py.230: loss: 0.14590 [I 2022-10-09 02:33:37] ...optimizers/trijoint.py.151: total_norm: 18.648939 [I 2022-10-09 02:33:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 2160/2304 [I 2022-10-09 02:33:45] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:00 [I 2022-10-09 02:33:45] ...trap/engines/engine.py.229: process: 0.83732 | load: 0.00300 [I 2022-10-09 02:33:45] ...trap/engines/engine.py.230: loss: 0.23196 [I 2022-10-09 02:33:45] ...optimizers/trijoint.py.151: total_norm: 20.461416 [I 2022-10-09 02:33:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 2170/2304 [I 2022-10-09 02:33:53] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:01:52 [I 2022-10-09 02:33:53] ...trap/engines/engine.py.229: process: 0.82308 | load: 0.00299 [I 2022-10-09 02:33:53] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-09 02:33:53] ...optimizers/trijoint.py.151: total_norm: 14.884559 [I 2022-10-09 02:34:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 2180/2304 [I 2022-10-09 02:34:02] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:01:43 [I 2022-10-09 02:34:02] ...trap/engines/engine.py.229: process: 0.83385 | load: 0.00302 [I 2022-10-09 02:34:02] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-09 02:34:02] ...optimizers/trijoint.py.151: total_norm: 19.827850 [I 2022-10-09 02:34:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 2190/2304 [I 2022-10-09 02:34:10] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:35 [I 2022-10-09 02:34:10] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00306 [I 2022-10-09 02:34:10] ...trap/engines/engine.py.230: loss: 0.09026 [I 2022-10-09 02:34:10] ...optimizers/trijoint.py.151: total_norm: 11.364806 [I 2022-10-09 02:34:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 2200/2304 [I 2022-10-09 02:34:19] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:27 [I 2022-10-09 02:34:19] ...trap/engines/engine.py.229: process: 0.84715 | load: 0.00299 [I 2022-10-09 02:34:19] ...trap/engines/engine.py.230: loss: 0.19454 [I 2022-10-09 02:34:19] ...optimizers/trijoint.py.151: total_norm: 24.070127 [I 2022-10-09 02:34:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 2210/2304 [I 2022-10-09 02:34:27] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:19 [I 2022-10-09 02:34:27] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00302 [I 2022-10-09 02:34:27] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-09 02:34:27] ...optimizers/trijoint.py.151: total_norm: 15.732267 [I 2022-10-09 02:34:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 2220/2304 [I 2022-10-09 02:34:35] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:10 [I 2022-10-09 02:34:35] ...trap/engines/engine.py.229: process: 0.82060 | load: 0.00298 [I 2022-10-09 02:34:35] ...trap/engines/engine.py.230: loss: 0.20270 [I 2022-10-09 02:34:35] ...optimizers/trijoint.py.151: total_norm: 19.085001 [I 2022-10-09 02:34:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 2230/2304 [I 2022-10-09 02:34:44] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:02 [I 2022-10-09 02:34:44] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00299 [I 2022-10-09 02:34:44] ...trap/engines/engine.py.230: loss: 0.11894 [I 2022-10-09 02:34:44] ...optimizers/trijoint.py.151: total_norm: 11.241631 [I 2022-10-09 02:34:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 2240/2304 [I 2022-10-09 02:34:52] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:00:53 [I 2022-10-09 02:34:52] ...trap/engines/engine.py.229: process: 0.82880 | load: 0.00298 [I 2022-10-09 02:34:52] ...trap/engines/engine.py.230: loss: 0.15343 [I 2022-10-09 02:34:52] ...optimizers/trijoint.py.151: total_norm: 21.918041 [I 2022-10-09 02:35:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 2250/2304 [I 2022-10-09 02:35:01] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:00:45 [I 2022-10-09 02:35:01] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00308 [I 2022-10-09 02:35:01] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-09 02:35:01] ...optimizers/trijoint.py.151: total_norm: 27.965353 [I 2022-10-09 02:35:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 2260/2304 [I 2022-10-09 02:35:09] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:37 [I 2022-10-09 02:35:09] ...trap/engines/engine.py.229: process: 0.83991 | load: 0.00306 [I 2022-10-09 02:35:09] ...trap/engines/engine.py.230: loss: 0.11955 [I 2022-10-09 02:35:09] ...optimizers/trijoint.py.151: total_norm: 15.266961 [I 2022-10-09 02:35:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 2270/2304 [I 2022-10-09 02:35:18] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:28 [I 2022-10-09 02:35:18] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00301 [I 2022-10-09 02:35:18] ...trap/engines/engine.py.230: loss: 0.10581 [I 2022-10-09 02:35:18] ...optimizers/trijoint.py.151: total_norm: 12.981637 [I 2022-10-09 02:35:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 2280/2304 [I 2022-10-09 02:35:26] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:20 [I 2022-10-09 02:35:26] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00297 [I 2022-10-09 02:35:26] ...trap/engines/engine.py.230: loss: 0.16832 [I 2022-10-09 02:35:26] ...optimizers/trijoint.py.151: total_norm: 26.122473 [I 2022-10-09 02:35:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 2290/2304 [I 2022-10-09 02:35:34] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:11 [I 2022-10-09 02:35:34] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00299 [I 2022-10-09 02:35:34] ...trap/engines/engine.py.230: loss: 0.12230 [I 2022-10-09 02:35:34] ...optimizers/trijoint.py.151: total_norm: 12.416527 [I 2022-10-09 02:35:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 2300/2304 [I 2022-10-09 02:35:43] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:03 [I 2022-10-09 02:35:43] ...trap/engines/engine.py.229: process: 0.83088 | load: 0.00017 [I 2022-10-09 02:35:43] ...trap/engines/engine.py.230: loss: 0.15694 [I 2022-10-09 02:35:43] ...optimizers/trijoint.py.151: total_norm: 17.861092 [I 2022-10-09 02:35:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 2304/2304 [I 2022-10-09 02:35:46] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:00 [I 2022-10-09 02:35:46] ...trap/engines/engine.py.229: process: 0.82541 | load: 0.00017 [I 2022-10-09 02:35:46] ...trap/engines/engine.py.230: loss: 0.19043 [I 2022-10-09 02:35:46] ...optimizers/trijoint.py.151: total_norm: 18.407267 [S 2022-10-09 02:35:47] ...trap/engines/engine.py.129: train_epoch.epoch: 28 [S 2022-10-09 02:35:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.16729746721442507 [I 2022-10-09 02:35:47] ...s/criterions/triplet.py.84: margin = 0.19500000000000012 [I 2022-10-09 02:35:48] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 28 [I 2022-10-09 02:35:48] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 02:35:50] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 02:35:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 02:35:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 02:35:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 02:35:51] ...trap/engines/engine.py.321: eval: epoch 28 | batch 0/510 [I 2022-10-09 02:35:51] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:59 [I 2022-10-09 02:35:51] ...trap/engines/engine.py.326: process: 3.29281 | load: 2.77701 [I 2022-10-09 02:35:58] ...trap/engines/engine.py.321: eval: epoch 28 | batch 10/510 [I 2022-10-09 02:35:58] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:46 [I 2022-10-09 02:35:58] ...trap/engines/engine.py.326: process: 0.64427 | load: 0.00018 [I 2022-10-09 02:36:04] ...trap/engines/engine.py.321: eval: epoch 28 | batch 20/510 [I 2022-10-09 02:36:04] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:28 [I 2022-10-09 02:36:04] ...trap/engines/engine.py.326: process: 0.68146 | load: 0.00019 [I 2022-10-09 02:36:11] ...trap/engines/engine.py.321: eval: epoch 28 | batch 30/510 [I 2022-10-09 02:36:11] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:15 [I 2022-10-09 02:36:11] ...trap/engines/engine.py.326: process: 0.67801 | load: 0.00019 [I 2022-10-09 02:36:18] ...trap/engines/engine.py.321: eval: epoch 28 | batch 40/510 [I 2022-10-09 02:36:18] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:05:23 [I 2022-10-09 02:36:18] ...trap/engines/engine.py.326: process: 0.62808 | load: 0.00019 [I 2022-10-09 02:36:23] ...trap/engines/engine.py.321: eval: epoch 28 | batch 50/510 [I 2022-10-09 02:36:23] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:04:12 [I 2022-10-09 02:36:23] ...trap/engines/engine.py.326: process: 0.53987 | load: 0.00020 [I 2022-10-09 02:36:28] ...trap/engines/engine.py.321: eval: epoch 28 | batch 60/510 [I 2022-10-09 02:36:28] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:07 [I 2022-10-09 02:36:28] ...trap/engines/engine.py.326: process: 0.53526 | load: 0.00019 [I 2022-10-09 02:36:34] ...trap/engines/engine.py.321: eval: epoch 28 | batch 70/510 [I 2022-10-09 02:36:34] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:08 [I 2022-10-09 02:36:34] ...trap/engines/engine.py.326: process: 0.60082 | load: 0.00019 [I 2022-10-09 02:36:40] ...trap/engines/engine.py.321: eval: epoch 28 | batch 80/510 [I 2022-10-09 02:36:40] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:04:02 [I 2022-10-09 02:36:40] ...trap/engines/engine.py.326: process: 0.55536 | load: 0.00019 [I 2022-10-09 02:36:45] ...trap/engines/engine.py.321: eval: epoch 28 | batch 90/510 [I 2022-10-09 02:36:45] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:24 [I 2022-10-09 02:36:45] ...trap/engines/engine.py.326: process: 0.48970 | load: 0.00018 [I 2022-10-09 02:36:49] ...trap/engines/engine.py.321: eval: epoch 28 | batch 100/510 [I 2022-10-09 02:36:49] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:58 [I 2022-10-09 02:36:49] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00019 [I 2022-10-09 02:36:52] ...trap/engines/engine.py.321: eval: epoch 28 | batch 110/510 [I 2022-10-09 02:36:52] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:11 [I 2022-10-09 02:36:52] ...trap/engines/engine.py.326: process: 0.31272 | load: 0.00019 [I 2022-10-09 02:36:55] ...trap/engines/engine.py.321: eval: epoch 28 | batch 120/510 [I 2022-10-09 02:36:55] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:05 [I 2022-10-09 02:36:55] ...trap/engines/engine.py.326: process: 0.32516 | load: 0.00021 [I 2022-10-09 02:36:59] ...trap/engines/engine.py.321: eval: epoch 28 | batch 130/510 [I 2022-10-09 02:36:59] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:00 [I 2022-10-09 02:36:59] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00020 [I 2022-10-09 02:37:02] ...trap/engines/engine.py.321: eval: epoch 28 | batch 140/510 [I 2022-10-09 02:37:02] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:57 [I 2022-10-09 02:37:02] ...trap/engines/engine.py.326: process: 0.32478 | load: 0.00020 [I 2022-10-09 02:37:05] ...trap/engines/engine.py.321: eval: epoch 28 | batch 150/510 [I 2022-10-09 02:37:05] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:55 [I 2022-10-09 02:37:05] ...trap/engines/engine.py.326: process: 0.31757 | load: 0.00018 [I 2022-10-09 02:37:08] ...trap/engines/engine.py.321: eval: epoch 28 | batch 160/510 [I 2022-10-09 02:37:08] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:52 [I 2022-10-09 02:37:08] ...trap/engines/engine.py.326: process: 0.32189 | load: 0.00020 [I 2022-10-09 02:37:11] ...trap/engines/engine.py.321: eval: epoch 28 | batch 170/510 [I 2022-10-09 02:37:11] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:48 [I 2022-10-09 02:37:11] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00019 [I 2022-10-09 02:37:15] ...trap/engines/engine.py.321: eval: epoch 28 | batch 180/510 [I 2022-10-09 02:37:15] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:45 [I 2022-10-09 02:37:15] ...trap/engines/engine.py.326: process: 0.32226 | load: 0.00019 [I 2022-10-09 02:37:18] ...trap/engines/engine.py.321: eval: epoch 28 | batch 190/510 [I 2022-10-09 02:37:18] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:41 [I 2022-10-09 02:37:18] ...trap/engines/engine.py.326: process: 0.31975 | load: 0.00020 [I 2022-10-09 02:37:21] ...trap/engines/engine.py.321: eval: epoch 28 | batch 200/510 [I 2022-10-09 02:37:21] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:39 [I 2022-10-09 02:37:21] ...trap/engines/engine.py.326: process: 0.32169 | load: 0.00020 [I 2022-10-09 02:37:24] ...trap/engines/engine.py.321: eval: epoch 28 | batch 210/510 [I 2022-10-09 02:37:24] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:35 [I 2022-10-09 02:37:24] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00020 [I 2022-10-09 02:37:27] ...trap/engines/engine.py.321: eval: epoch 28 | batch 220/510 [I 2022-10-09 02:37:27] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:32 [I 2022-10-09 02:37:27] ...trap/engines/engine.py.326: process: 0.32100 | load: 0.00019 [I 2022-10-09 02:37:31] ...trap/engines/engine.py.321: eval: epoch 28 | batch 230/510 [I 2022-10-09 02:37:31] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:29 [I 2022-10-09 02:37:31] ...trap/engines/engine.py.326: process: 0.31723 | load: 0.00020 [I 2022-10-09 02:37:34] ...trap/engines/engine.py.321: eval: epoch 28 | batch 240/510 [I 2022-10-09 02:37:34] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:25 [I 2022-10-09 02:37:34] ...trap/engines/engine.py.326: process: 0.31918 | load: 0.00020 [I 2022-10-09 02:37:37] ...trap/engines/engine.py.321: eval: epoch 28 | batch 250/510 [I 2022-10-09 02:37:37] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:22 [I 2022-10-09 02:37:37] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00021 [I 2022-10-09 02:37:40] ...trap/engines/engine.py.321: eval: epoch 28 | batch 260/510 [I 2022-10-09 02:37:40] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:19 [I 2022-10-09 02:37:40] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00019 [I 2022-10-09 02:37:43] ...trap/engines/engine.py.321: eval: epoch 28 | batch 270/510 [I 2022-10-09 02:37:43] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:17 [I 2022-10-09 02:37:43] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00020 [I 2022-10-09 02:37:47] ...trap/engines/engine.py.321: eval: epoch 28 | batch 280/510 [I 2022-10-09 02:37:47] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:13 [I 2022-10-09 02:37:47] ...trap/engines/engine.py.326: process: 0.32704 | load: 0.00019 [I 2022-10-09 02:37:50] ...trap/engines/engine.py.321: eval: epoch 28 | batch 290/510 [I 2022-10-09 02:37:50] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:10 [I 2022-10-09 02:37:50] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00020 [I 2022-10-09 02:37:53] ...trap/engines/engine.py.321: eval: epoch 28 | batch 300/510 [I 2022-10-09 02:37:53] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:06 [I 2022-10-09 02:37:53] ...trap/engines/engine.py.326: process: 0.31187 | load: 0.00020 [I 2022-10-09 02:37:56] ...trap/engines/engine.py.321: eval: epoch 28 | batch 310/510 [I 2022-10-09 02:37:56] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:03 [I 2022-10-09 02:37:56] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00021 [I 2022-10-09 02:37:59] ...trap/engines/engine.py.321: eval: epoch 28 | batch 320/510 [I 2022-10-09 02:37:59] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:00 [I 2022-10-09 02:37:59] ...trap/engines/engine.py.326: process: 0.31900 | load: 0.00020 [I 2022-10-09 02:38:02] ...trap/engines/engine.py.321: eval: epoch 28 | batch 330/510 [I 2022-10-09 02:38:02] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:57 [I 2022-10-09 02:38:02] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00020 [I 2022-10-09 02:38:06] ...trap/engines/engine.py.321: eval: epoch 28 | batch 340/510 [I 2022-10-09 02:38:06] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:54 [I 2022-10-09 02:38:06] ...trap/engines/engine.py.326: process: 0.31563 | load: 0.00021 [I 2022-10-09 02:38:09] ...trap/engines/engine.py.321: eval: epoch 28 | batch 350/510 [I 2022-10-09 02:38:09] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:51 [I 2022-10-09 02:38:09] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00020 [I 2022-10-09 02:38:12] ...trap/engines/engine.py.321: eval: epoch 28 | batch 360/510 [I 2022-10-09 02:38:12] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:47 [I 2022-10-09 02:38:12] ...trap/engines/engine.py.326: process: 0.31285 | load: 0.00020 [I 2022-10-09 02:38:15] ...trap/engines/engine.py.321: eval: epoch 28 | batch 370/510 [I 2022-10-09 02:38:15] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:44 [I 2022-10-09 02:38:15] ...trap/engines/engine.py.326: process: 0.31484 | load: 0.00021 [I 2022-10-09 02:38:18] ...trap/engines/engine.py.321: eval: epoch 28 | batch 380/510 [I 2022-10-09 02:38:18] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:41 [I 2022-10-09 02:38:18] ...trap/engines/engine.py.326: process: 0.31184 | load: 0.00020 [I 2022-10-09 02:38:22] ...trap/engines/engine.py.321: eval: epoch 28 | batch 390/510 [I 2022-10-09 02:38:22] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:38 [I 2022-10-09 02:38:22] ...trap/engines/engine.py.326: process: 0.32464 | load: 0.00019 [I 2022-10-09 02:38:25] ...trap/engines/engine.py.321: eval: epoch 28 | batch 400/510 [I 2022-10-09 02:38:25] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:35 [I 2022-10-09 02:38:25] ...trap/engines/engine.py.326: process: 0.31871 | load: 0.00020 [I 2022-10-09 02:38:28] ...trap/engines/engine.py.321: eval: epoch 28 | batch 410/510 [I 2022-10-09 02:38:28] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:31 [I 2022-10-09 02:38:28] ...trap/engines/engine.py.326: process: 0.31649 | load: 0.00019 [I 2022-10-09 02:38:31] ...trap/engines/engine.py.321: eval: epoch 28 | batch 420/510 [I 2022-10-09 02:38:31] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:28 [I 2022-10-09 02:38:31] ...trap/engines/engine.py.326: process: 0.31456 | load: 0.00020 [I 2022-10-09 02:38:34] ...trap/engines/engine.py.321: eval: epoch 28 | batch 430/510 [I 2022-10-09 02:38:34] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:25 [I 2022-10-09 02:38:34] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00020 [I 2022-10-09 02:38:38] ...trap/engines/engine.py.321: eval: epoch 28 | batch 440/510 [I 2022-10-09 02:38:38] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:22 [I 2022-10-09 02:38:38] ...trap/engines/engine.py.326: process: 0.30585 | load: 0.00020 [I 2022-10-09 02:38:41] ...trap/engines/engine.py.321: eval: epoch 28 | batch 450/510 [I 2022-10-09 02:38:41] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:19 [I 2022-10-09 02:38:41] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00021 [I 2022-10-09 02:38:44] ...trap/engines/engine.py.321: eval: epoch 28 | batch 460/510 [I 2022-10-09 02:38:44] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:15 [I 2022-10-09 02:38:44] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00021 [I 2022-10-09 02:38:47] ...trap/engines/engine.py.321: eval: epoch 28 | batch 470/510 [I 2022-10-09 02:38:47] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:12 [I 2022-10-09 02:38:47] ...trap/engines/engine.py.326: process: 0.30888 | load: 0.00020 [I 2022-10-09 02:38:50] ...trap/engines/engine.py.321: eval: epoch 28 | batch 480/510 [I 2022-10-09 02:38:50] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:09 [I 2022-10-09 02:38:50] ...trap/engines/engine.py.326: process: 0.31534 | load: 0.00020 [I 2022-10-09 02:38:54] ...trap/engines/engine.py.321: eval: epoch 28 | batch 490/510 [I 2022-10-09 02:38:54] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:06 [I 2022-10-09 02:38:54] ...trap/engines/engine.py.326: process: 0.31616 | load: 0.00021 [I 2022-10-09 02:38:57] ...trap/engines/engine.py.321: eval: epoch 28 | batch 500/510 [I 2022-10-09 02:38:57] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:03 [I 2022-10-09 02:38:57] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00021 [I 2022-10-09 02:39:00] ...trap/engines/engine.py.321: eval: epoch 28 | batch 510/510 [I 2022-10-09 02:39:00] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:00 [I 2022-10-09 02:39:00] ...trap/engines/engine.py.326: process: 0.31352 | load: 0.00016 [S 2022-10-09 02:39:01] ...trap/engines/engine.py.132: eval_epoch.epoch: 28 [I 2022-10-09 02:39:01] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.688700 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.896300 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931900 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.683400 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.892500 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.927400 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013417 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008075 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008479 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014657 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007297 [S 2022-10-09 02:39:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006800 [I 2022-10-09 02:39:41] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 02:39:41] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 02:39:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 02:39:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 02:39:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 02:39:43] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 02:39:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 02:39:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 02:39:51] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 02:39:51] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 02:39:54] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 02:40:00] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 02:40:00] ...trap/engines/engine.py.163: Training model on trainset for epoch 29 [I 2022-10-09 02:40:00] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 02:40:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 0/2304 [I 2022-10-09 02:40:03] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:28 [I 2022-10-09 02:40:03] ...trap/engines/engine.py.229: process: 3.08513 | load: 2.16353 [I 2022-10-09 02:40:03] ...trap/engines/engine.py.230: loss: 0.18133 [I 2022-10-09 02:40:03] ...optimizers/trijoint.py.151: total_norm: 14.092325 [I 2022-10-09 02:40:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 10/2304 [I 2022-10-09 02:40:11] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:21 [I 2022-10-09 02:40:11] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00642 [I 2022-10-09 02:40:11] ...trap/engines/engine.py.230: loss: 0.15016 [I 2022-10-09 02:40:11] ...optimizers/trijoint.py.151: total_norm: 16.761578 [I 2022-10-09 02:40:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 20/2304 [I 2022-10-09 02:40:19] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:58 [I 2022-10-09 02:40:19] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00577 [I 2022-10-09 02:40:19] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-09 02:40:19] ...optimizers/trijoint.py.151: total_norm: 17.556513 [I 2022-10-09 02:40:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 30/2304 [I 2022-10-09 02:40:28] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:58 [I 2022-10-09 02:40:28] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00447 [I 2022-10-09 02:40:28] ...trap/engines/engine.py.230: loss: 0.24938 [I 2022-10-09 02:40:28] ...optimizers/trijoint.py.151: total_norm: 18.521929 [I 2022-10-09 02:40:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 40/2304 [I 2022-10-09 02:40:36] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:44 [I 2022-10-09 02:40:36] ...trap/engines/engine.py.229: process: 0.83954 | load: 0.00404 [I 2022-10-09 02:40:36] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-09 02:40:36] ...optimizers/trijoint.py.151: total_norm: 22.175741 [I 2022-10-09 02:40:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 50/2304 [I 2022-10-09 02:40:45] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:31 [I 2022-10-09 02:40:45] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00368 [I 2022-10-09 02:40:45] ...trap/engines/engine.py.230: loss: 0.19110 [I 2022-10-09 02:40:45] ...optimizers/trijoint.py.151: total_norm: 20.158157 [I 2022-10-09 02:40:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 60/2304 [I 2022-10-09 02:40:53] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:33 [I 2022-10-09 02:40:53] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00345 [I 2022-10-09 02:40:53] ...trap/engines/engine.py.230: loss: 0.09988 [I 2022-10-09 02:40:53] ...optimizers/trijoint.py.151: total_norm: 13.351778 [I 2022-10-09 02:41:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 70/2304 [I 2022-10-09 02:41:02] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:27 [I 2022-10-09 02:41:02] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00373 [I 2022-10-09 02:41:02] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-09 02:41:02] ...optimizers/trijoint.py.151: total_norm: 8.515102 [I 2022-10-09 02:41:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 80/2304 [I 2022-10-09 02:41:28] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:59:34 [I 2022-10-09 02:41:28] ...trap/engines/engine.py.229: process: 0.82874 | load: 0.00301 [I 2022-10-09 02:41:28] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-09 02:41:28] ...optimizers/trijoint.py.151: total_norm: 12.489241 [I 2022-10-09 02:41:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 90/2304 [I 2022-10-09 02:41:37] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:34:06 [I 2022-10-09 02:41:37] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00325 [I 2022-10-09 02:41:37] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-09 02:41:37] ...optimizers/trijoint.py.151: total_norm: 44.461239 [I 2022-10-09 02:41:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 100/2304 [I 2022-10-09 02:41:45] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:15 [I 2022-10-09 02:41:45] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00362 [I 2022-10-09 02:41:45] ...trap/engines/engine.py.230: loss: 0.19634 [I 2022-10-09 02:41:45] ...optimizers/trijoint.py.151: total_norm: 16.791201 [I 2022-10-09 02:41:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 110/2304 [I 2022-10-09 02:41:54] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:49 [I 2022-10-09 02:41:54] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00391 [I 2022-10-09 02:41:54] ...trap/engines/engine.py.230: loss: 0.16262 [I 2022-10-09 02:41:54] ...optimizers/trijoint.py.151: total_norm: 21.364887 [I 2022-10-09 02:42:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 120/2304 [I 2022-10-09 02:42:02] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:36 [I 2022-10-09 02:42:02] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00321 [I 2022-10-09 02:42:02] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-09 02:42:02] ...optimizers/trijoint.py.151: total_norm: 12.383953 [I 2022-10-09 02:42:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 130/2304 [I 2022-10-09 02:42:10] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:25 [I 2022-10-09 02:42:10] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00323 [I 2022-10-09 02:42:10] ...trap/engines/engine.py.230: loss: 0.20146 [I 2022-10-09 02:42:10] ...optimizers/trijoint.py.151: total_norm: 18.757654 [I 2022-10-09 02:42:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 140/2304 [I 2022-10-09 02:42:19] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:12 [I 2022-10-09 02:42:19] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00329 [I 2022-10-09 02:42:19] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-09 02:42:19] ...optimizers/trijoint.py.151: total_norm: 24.209145 [I 2022-10-09 02:42:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 150/2304 [I 2022-10-09 02:42:27] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:09 [I 2022-10-09 02:42:27] ...trap/engines/engine.py.229: process: 0.83343 | load: 0.00330 [I 2022-10-09 02:42:27] ...trap/engines/engine.py.230: loss: 0.17179 [I 2022-10-09 02:42:27] ...optimizers/trijoint.py.151: total_norm: 27.562609 [I 2022-10-09 02:42:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 160/2304 [I 2022-10-09 02:42:36] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:13 [I 2022-10-09 02:42:36] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00332 [I 2022-10-09 02:42:36] ...trap/engines/engine.py.230: loss: 0.18950 [I 2022-10-09 02:42:36] ...optimizers/trijoint.py.151: total_norm: 25.848503 [I 2022-10-09 02:42:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 170/2304 [I 2022-10-09 02:42:44] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:02 [I 2022-10-09 02:42:44] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00315 [I 2022-10-09 02:42:44] ...trap/engines/engine.py.230: loss: 0.19246 [I 2022-10-09 02:42:44] ...optimizers/trijoint.py.151: total_norm: 17.702726 [I 2022-10-09 02:42:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 180/2304 [I 2022-10-09 02:42:53] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:40 [I 2022-10-09 02:42:53] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00298 [I 2022-10-09 02:42:53] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-09 02:42:53] ...optimizers/trijoint.py.151: total_norm: 12.754121 [I 2022-10-09 02:43:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 190/2304 [I 2022-10-09 02:43:01] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:38 [I 2022-10-09 02:43:01] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00308 [I 2022-10-09 02:43:01] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-09 02:43:01] ...optimizers/trijoint.py.151: total_norm: 17.326315 [I 2022-10-09 02:43:09] ...trap/engines/engine.py.224: train: epoch 29 | batch 200/2304 [I 2022-10-09 02:43:09] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:20 [I 2022-10-09 02:43:09] ...trap/engines/engine.py.229: process: 0.82696 | load: 0.00340 [I 2022-10-09 02:43:09] ...trap/engines/engine.py.230: loss: 0.20852 [I 2022-10-09 02:43:09] ...optimizers/trijoint.py.151: total_norm: 16.982187 [I 2022-10-09 02:43:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 210/2304 [I 2022-10-09 02:43:18] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:22 [I 2022-10-09 02:43:18] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00316 [I 2022-10-09 02:43:18] ...trap/engines/engine.py.230: loss: 0.18982 [I 2022-10-09 02:43:18] ...optimizers/trijoint.py.151: total_norm: 15.099502 [I 2022-10-09 02:43:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 220/2304 [I 2022-10-09 02:43:26] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:20 [I 2022-10-09 02:43:26] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00316 [I 2022-10-09 02:43:26] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-09 02:43:26] ...optimizers/trijoint.py.151: total_norm: 25.857643 [I 2022-10-09 02:43:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 230/2304 [I 2022-10-09 02:43:35] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:08 [I 2022-10-09 02:43:35] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00316 [I 2022-10-09 02:43:35] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-09 02:43:35] ...optimizers/trijoint.py.151: total_norm: 19.954000 [I 2022-10-09 02:43:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 240/2304 [I 2022-10-09 02:43:43] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:55 [I 2022-10-09 02:43:43] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00310 [I 2022-10-09 02:43:43] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-09 02:43:43] ...optimizers/trijoint.py.151: total_norm: 11.960213 [I 2022-10-09 02:43:52] ...trap/engines/engine.py.224: train: epoch 29 | batch 250/2304 [I 2022-10-09 02:43:52] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:43 [I 2022-10-09 02:43:52] ...trap/engines/engine.py.229: process: 0.84012 | load: 0.00303 [I 2022-10-09 02:43:52] ...trap/engines/engine.py.230: loss: 0.13521 [I 2022-10-09 02:43:52] ...optimizers/trijoint.py.151: total_norm: 13.910796 [I 2022-10-09 02:44:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 260/2304 [I 2022-10-09 02:44:00] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:38 [I 2022-10-09 02:44:00] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00302 [I 2022-10-09 02:44:00] ...trap/engines/engine.py.230: loss: 0.11353 [I 2022-10-09 02:44:00] ...optimizers/trijoint.py.151: total_norm: 12.720722 [I 2022-10-09 02:44:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 270/2304 [I 2022-10-09 02:44:08] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:25 [I 2022-10-09 02:44:08] ...trap/engines/engine.py.229: process: 0.83274 | load: 0.00300 [I 2022-10-09 02:44:08] ...trap/engines/engine.py.230: loss: 0.27584 [I 2022-10-09 02:44:08] ...optimizers/trijoint.py.151: total_norm: 21.725122 [I 2022-10-09 02:44:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 280/2304 [I 2022-10-09 02:44:17] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:25 [I 2022-10-09 02:44:17] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00302 [I 2022-10-09 02:44:17] ...trap/engines/engine.py.230: loss: 0.19763 [I 2022-10-09 02:44:17] ...optimizers/trijoint.py.151: total_norm: 15.992029 [I 2022-10-09 02:44:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 290/2304 [I 2022-10-09 02:44:25] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:20 [I 2022-10-09 02:44:25] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00320 [I 2022-10-09 02:44:25] ...trap/engines/engine.py.230: loss: 0.16517 [I 2022-10-09 02:44:25] ...optimizers/trijoint.py.151: total_norm: 22.069256 [I 2022-10-09 02:44:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 300/2304 [I 2022-10-09 02:44:34] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:28:05 [I 2022-10-09 02:44:34] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00300 [I 2022-10-09 02:44:34] ...trap/engines/engine.py.230: loss: 0.16714 [I 2022-10-09 02:44:34] ...optimizers/trijoint.py.151: total_norm: 15.621483 [I 2022-10-09 02:44:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 310/2304 [I 2022-10-09 02:44:42] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:28:01 [I 2022-10-09 02:44:42] ...trap/engines/engine.py.229: process: 0.85405 | load: 0.00303 [I 2022-10-09 02:44:42] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-09 02:44:42] ...optimizers/trijoint.py.151: total_norm: 17.192350 [I 2022-10-09 02:44:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 320/2304 [I 2022-10-09 02:44:51] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:27:52 [I 2022-10-09 02:44:51] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00314 [I 2022-10-09 02:44:51] ...trap/engines/engine.py.230: loss: 0.16693 [I 2022-10-09 02:44:51] ...optimizers/trijoint.py.151: total_norm: 12.217438 [I 2022-10-09 02:44:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 330/2304 [I 2022-10-09 02:44:59] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:50 [I 2022-10-09 02:44:59] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00302 [I 2022-10-09 02:44:59] ...trap/engines/engine.py.230: loss: 0.16184 [I 2022-10-09 02:44:59] ...optimizers/trijoint.py.151: total_norm: 13.252238 [I 2022-10-09 02:45:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 340/2304 [I 2022-10-09 02:45:08] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:38 [I 2022-10-09 02:45:08] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00301 [I 2022-10-09 02:45:08] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-09 02:45:08] ...optimizers/trijoint.py.151: total_norm: 16.087946 [I 2022-10-09 02:45:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 350/2304 [I 2022-10-09 02:45:16] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:27:24 [I 2022-10-09 02:45:16] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00301 [I 2022-10-09 02:45:16] ...trap/engines/engine.py.230: loss: 0.22269 [I 2022-10-09 02:45:16] ...optimizers/trijoint.py.151: total_norm: 12.683307 [I 2022-10-09 02:45:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 360/2304 [I 2022-10-09 02:45:24] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:18 [I 2022-10-09 02:45:24] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00302 [I 2022-10-09 02:45:24] ...trap/engines/engine.py.230: loss: 0.17199 [I 2022-10-09 02:45:24] ...optimizers/trijoint.py.151: total_norm: 13.508817 [I 2022-10-09 02:45:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 370/2304 [I 2022-10-09 02:45:33] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:14 [I 2022-10-09 02:45:33] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00298 [I 2022-10-09 02:45:33] ...trap/engines/engine.py.230: loss: 0.11684 [I 2022-10-09 02:45:33] ...optimizers/trijoint.py.151: total_norm: 12.517748 [I 2022-10-09 02:45:41] ...trap/engines/engine.py.224: train: epoch 29 | batch 380/2304 [I 2022-10-09 02:45:41] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:53 [I 2022-10-09 02:45:41] ...trap/engines/engine.py.229: process: 0.83579 | load: 0.00300 [I 2022-10-09 02:45:41] ...trap/engines/engine.py.230: loss: 0.19339 [I 2022-10-09 02:45:41] ...optimizers/trijoint.py.151: total_norm: 11.593699 [I 2022-10-09 02:45:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 390/2304 [I 2022-10-09 02:45:50] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:26:54 [I 2022-10-09 02:45:50] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00302 [I 2022-10-09 02:45:50] ...trap/engines/engine.py.230: loss: 0.16437 [I 2022-10-09 02:45:50] ...optimizers/trijoint.py.151: total_norm: 20.784977 [I 2022-10-09 02:45:58] ...trap/engines/engine.py.224: train: epoch 29 | batch 400/2304 [I 2022-10-09 02:45:58] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:42 [I 2022-10-09 02:45:58] ...trap/engines/engine.py.229: process: 0.83884 | load: 0.00301 [I 2022-10-09 02:45:58] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-09 02:45:58] ...optimizers/trijoint.py.151: total_norm: 16.895802 [I 2022-10-09 02:46:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 410/2304 [I 2022-10-09 02:46:07] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:34 [I 2022-10-09 02:46:07] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00303 [I 2022-10-09 02:46:07] ...trap/engines/engine.py.230: loss: 0.11975 [I 2022-10-09 02:46:07] ...optimizers/trijoint.py.151: total_norm: 22.477451 [I 2022-10-09 02:46:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 420/2304 [I 2022-10-09 02:46:15] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:25 [I 2022-10-09 02:46:15] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00300 [I 2022-10-09 02:46:15] ...trap/engines/engine.py.230: loss: 0.14721 [I 2022-10-09 02:46:15] ...optimizers/trijoint.py.151: total_norm: 19.980970 [I 2022-10-09 02:46:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 430/2304 [I 2022-10-09 02:46:23] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:18 [I 2022-10-09 02:46:23] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00313 [I 2022-10-09 02:46:23] ...trap/engines/engine.py.230: loss: 0.18922 [I 2022-10-09 02:46:23] ...optimizers/trijoint.py.151: total_norm: 42.987080 [I 2022-10-09 02:46:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 440/2304 [I 2022-10-09 02:46:32] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:04 [I 2022-10-09 02:46:32] ...trap/engines/engine.py.229: process: 0.82919 | load: 0.00300 [I 2022-10-09 02:46:32] ...trap/engines/engine.py.230: loss: 0.11345 [I 2022-10-09 02:46:32] ...optimizers/trijoint.py.151: total_norm: 12.286239 [I 2022-10-09 02:46:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 450/2304 [I 2022-10-09 02:46:40] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:51 [I 2022-10-09 02:46:40] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00300 [I 2022-10-09 02:46:40] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-09 02:46:40] ...optimizers/trijoint.py.151: total_norm: 22.157553 [I 2022-10-09 02:46:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 460/2304 [I 2022-10-09 02:46:49] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:25:48 [I 2022-10-09 02:46:49] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00301 [I 2022-10-09 02:46:49] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-09 02:46:49] ...optimizers/trijoint.py.151: total_norm: 21.513611 [I 2022-10-09 02:46:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 470/2304 [I 2022-10-09 02:46:57] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:36 [I 2022-10-09 02:46:57] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00325 [I 2022-10-09 02:46:57] ...trap/engines/engine.py.230: loss: 0.11898 [I 2022-10-09 02:46:57] ...optimizers/trijoint.py.151: total_norm: 10.516044 [I 2022-10-09 02:47:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 480/2304 [I 2022-10-09 02:47:05] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:38 [I 2022-10-09 02:47:05] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00303 [I 2022-10-09 02:47:05] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-09 02:47:05] ...optimizers/trijoint.py.151: total_norm: 15.078070 [I 2022-10-09 02:47:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 490/2304 [I 2022-10-09 02:47:14] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:20 [I 2022-10-09 02:47:14] ...trap/engines/engine.py.229: process: 0.83859 | load: 0.00301 [I 2022-10-09 02:47:14] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-09 02:47:14] ...optimizers/trijoint.py.151: total_norm: 11.230711 [I 2022-10-09 02:47:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 500/2304 [I 2022-10-09 02:47:22] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:16 [I 2022-10-09 02:47:22] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00306 [I 2022-10-09 02:47:22] ...trap/engines/engine.py.230: loss: 0.19160 [I 2022-10-09 02:47:22] ...optimizers/trijoint.py.151: total_norm: 14.098051 [I 2022-10-09 02:47:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 510/2304 [I 2022-10-09 02:47:31] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:06 [I 2022-10-09 02:47:31] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00300 [I 2022-10-09 02:47:31] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-09 02:47:31] ...optimizers/trijoint.py.151: total_norm: 19.734253 [I 2022-10-09 02:47:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 520/2304 [I 2022-10-09 02:47:39] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:57 [I 2022-10-09 02:47:39] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00300 [I 2022-10-09 02:47:39] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-09 02:47:39] ...optimizers/trijoint.py.151: total_norm: 27.754366 [I 2022-10-09 02:47:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 530/2304 [I 2022-10-09 02:47:47] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:51 [I 2022-10-09 02:47:47] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00315 [I 2022-10-09 02:47:47] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-09 02:47:47] ...optimizers/trijoint.py.151: total_norm: 8.932177 [I 2022-10-09 02:47:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 540/2304 [I 2022-10-09 02:47:56] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:40 [I 2022-10-09 02:47:56] ...trap/engines/engine.py.229: process: 0.84017 | load: 0.00301 [I 2022-10-09 02:47:56] ...trap/engines/engine.py.230: loss: 0.14503 [I 2022-10-09 02:47:56] ...optimizers/trijoint.py.151: total_norm: 19.833183 [I 2022-10-09 02:48:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 550/2304 [I 2022-10-09 02:48:04] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:32 [I 2022-10-09 02:48:04] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00301 [I 2022-10-09 02:48:04] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-09 02:48:04] ...optimizers/trijoint.py.151: total_norm: 8.932284 [I 2022-10-09 02:48:13] ...trap/engines/engine.py.224: train: epoch 29 | batch 560/2304 [I 2022-10-09 02:48:13] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:28 [I 2022-10-09 02:48:13] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00303 [I 2022-10-09 02:48:13] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-09 02:48:13] ...optimizers/trijoint.py.151: total_norm: 13.754921 [I 2022-10-09 02:48:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 570/2304 [I 2022-10-09 02:48:21] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:16 [I 2022-10-09 02:48:21] ...trap/engines/engine.py.229: process: 0.82968 | load: 0.00303 [I 2022-10-09 02:48:21] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-09 02:48:21] ...optimizers/trijoint.py.151: total_norm: 23.791754 [I 2022-10-09 02:48:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 580/2304 [I 2022-10-09 02:48:30] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:12 [I 2022-10-09 02:48:30] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00303 [I 2022-10-09 02:48:30] ...trap/engines/engine.py.230: loss: 0.16827 [I 2022-10-09 02:48:30] ...optimizers/trijoint.py.151: total_norm: 14.837424 [I 2022-10-09 02:48:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 590/2304 [I 2022-10-09 02:48:38] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:24:08 [I 2022-10-09 02:48:38] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00303 [I 2022-10-09 02:48:38] ...trap/engines/engine.py.230: loss: 0.19922 [I 2022-10-09 02:48:38] ...optimizers/trijoint.py.151: total_norm: 28.349762 [I 2022-10-09 02:48:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 600/2304 [I 2022-10-09 02:48:46] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:48 [I 2022-10-09 02:48:46] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00304 [I 2022-10-09 02:48:46] ...trap/engines/engine.py.230: loss: 0.21054 [I 2022-10-09 02:48:46] ...optimizers/trijoint.py.151: total_norm: 27.862225 [I 2022-10-09 02:48:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 610/2304 [I 2022-10-09 02:48:55] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:48 [I 2022-10-09 02:48:55] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00313 [I 2022-10-09 02:48:55] ...trap/engines/engine.py.230: loss: 0.23553 [I 2022-10-09 02:48:55] ...optimizers/trijoint.py.151: total_norm: 26.521784 [I 2022-10-09 02:49:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 620/2304 [I 2022-10-09 02:49:03] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:32 [I 2022-10-09 02:49:03] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00301 [I 2022-10-09 02:49:03] ...trap/engines/engine.py.230: loss: 0.18015 [I 2022-10-09 02:49:03] ...optimizers/trijoint.py.151: total_norm: 13.458299 [I 2022-10-09 02:49:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 630/2304 [I 2022-10-09 02:49:12] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:29 [I 2022-10-09 02:49:12] ...trap/engines/engine.py.229: process: 0.83525 | load: 0.00304 [I 2022-10-09 02:49:12] ...trap/engines/engine.py.230: loss: 0.14288 [I 2022-10-09 02:49:12] ...optimizers/trijoint.py.151: total_norm: 12.942455 [I 2022-10-09 02:49:20] ...trap/engines/engine.py.224: train: epoch 29 | batch 640/2304 [I 2022-10-09 02:49:20] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:19 [I 2022-10-09 02:49:20] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00301 [I 2022-10-09 02:49:20] ...trap/engines/engine.py.230: loss: 0.21221 [I 2022-10-09 02:49:20] ...optimizers/trijoint.py.151: total_norm: 18.353218 [I 2022-10-09 02:49:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 650/2304 [I 2022-10-09 02:49:29] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:19 [I 2022-10-09 02:49:29] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00300 [I 2022-10-09 02:49:29] ...trap/engines/engine.py.230: loss: 0.18161 [I 2022-10-09 02:49:29] ...optimizers/trijoint.py.151: total_norm: 17.141794 [I 2022-10-09 02:49:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 660/2304 [I 2022-10-09 02:49:37] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:23:09 [I 2022-10-09 02:49:37] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00304 [I 2022-10-09 02:49:37] ...trap/engines/engine.py.230: loss: 0.17729 [I 2022-10-09 02:49:37] ...optimizers/trijoint.py.151: total_norm: 11.613588 [I 2022-10-09 02:49:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 670/2304 [I 2022-10-09 02:49:46] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:22:55 [I 2022-10-09 02:49:46] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00300 [I 2022-10-09 02:49:46] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-09 02:49:46] ...optimizers/trijoint.py.151: total_norm: 11.748461 [I 2022-10-09 02:49:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 680/2304 [I 2022-10-09 02:49:54] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:43 [I 2022-10-09 02:49:54] ...trap/engines/engine.py.229: process: 0.84475 | load: 0.00310 [I 2022-10-09 02:49:54] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-09 02:49:54] ...optimizers/trijoint.py.151: total_norm: 19.516972 [I 2022-10-09 02:50:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 690/2304 [I 2022-10-09 02:50:02] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:37 [I 2022-10-09 02:50:02] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00301 [I 2022-10-09 02:50:02] ...trap/engines/engine.py.230: loss: 0.20143 [I 2022-10-09 02:50:02] ...optimizers/trijoint.py.151: total_norm: 14.251575 [I 2022-10-09 02:50:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 700/2304 [I 2022-10-09 02:50:11] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:28 [I 2022-10-09 02:50:11] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00301 [I 2022-10-09 02:50:11] ...trap/engines/engine.py.230: loss: 0.23366 [I 2022-10-09 02:50:11] ...optimizers/trijoint.py.151: total_norm: 26.069353 [I 2022-10-09 02:50:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 710/2304 [I 2022-10-09 02:50:19] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:23 [I 2022-10-09 02:50:19] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00310 [I 2022-10-09 02:50:19] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-09 02:50:19] ...optimizers/trijoint.py.151: total_norm: 28.570826 [I 2022-10-09 02:50:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 720/2304 [I 2022-10-09 02:50:28] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:22:13 [I 2022-10-09 02:50:28] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00310 [I 2022-10-09 02:50:28] ...trap/engines/engine.py.230: loss: 0.20570 [I 2022-10-09 02:50:28] ...optimizers/trijoint.py.151: total_norm: 16.191896 [I 2022-10-09 02:50:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 730/2304 [I 2022-10-09 02:50:36] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:22:08 [I 2022-10-09 02:50:36] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00314 [I 2022-10-09 02:50:36] ...trap/engines/engine.py.230: loss: 0.19615 [I 2022-10-09 02:50:36] ...optimizers/trijoint.py.151: total_norm: 19.549582 [I 2022-10-09 02:50:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 740/2304 [I 2022-10-09 02:50:45] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:57 [I 2022-10-09 02:50:45] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00305 [I 2022-10-09 02:50:45] ...trap/engines/engine.py.230: loss: 0.15278 [I 2022-10-09 02:50:45] ...optimizers/trijoint.py.151: total_norm: 10.756477 [I 2022-10-09 02:50:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 750/2304 [I 2022-10-09 02:50:53] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:21:51 [I 2022-10-09 02:50:53] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00306 [I 2022-10-09 02:50:53] ...trap/engines/engine.py.230: loss: 0.18167 [I 2022-10-09 02:50:53] ...optimizers/trijoint.py.151: total_norm: 24.115482 [I 2022-10-09 02:51:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 760/2304 [I 2022-10-09 02:51:01] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:38 [I 2022-10-09 02:51:01] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00311 [I 2022-10-09 02:51:01] ...trap/engines/engine.py.230: loss: 0.22333 [I 2022-10-09 02:51:01] ...optimizers/trijoint.py.151: total_norm: 16.495667 [I 2022-10-09 02:51:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 770/2304 [I 2022-10-09 02:51:10] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:27 [I 2022-10-09 02:51:10] ...trap/engines/engine.py.229: process: 0.83814 | load: 0.00308 [I 2022-10-09 02:51:10] ...trap/engines/engine.py.230: loss: 0.13954 [I 2022-10-09 02:51:10] ...optimizers/trijoint.py.151: total_norm: 25.912102 [I 2022-10-09 02:51:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 780/2304 [I 2022-10-09 02:51:18] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:21 [I 2022-10-09 02:51:18] ...trap/engines/engine.py.229: process: 0.82941 | load: 0.00306 [I 2022-10-09 02:51:18] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-09 02:51:18] ...optimizers/trijoint.py.151: total_norm: 22.898468 [I 2022-10-09 02:51:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 790/2304 [I 2022-10-09 02:51:27] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:21:14 [I 2022-10-09 02:51:27] ...trap/engines/engine.py.229: process: 0.84229 | load: 0.00310 [I 2022-10-09 02:51:27] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-09 02:51:27] ...optimizers/trijoint.py.151: total_norm: 16.500032 [I 2022-10-09 02:51:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 800/2304 [I 2022-10-09 02:51:35] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:13 [I 2022-10-09 02:51:35] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00308 [I 2022-10-09 02:51:35] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-09 02:51:35] ...optimizers/trijoint.py.151: total_norm: 20.632601 [I 2022-10-09 02:51:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 810/2304 [I 2022-10-09 02:51:44] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:20:58 [I 2022-10-09 02:51:44] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00309 [I 2022-10-09 02:51:44] ...trap/engines/engine.py.230: loss: 0.17648 [I 2022-10-09 02:51:44] ...optimizers/trijoint.py.151: total_norm: 22.694214 [I 2022-10-09 02:51:52] ...trap/engines/engine.py.224: train: epoch 29 | batch 820/2304 [I 2022-10-09 02:51:52] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:20:40 [I 2022-10-09 02:51:52] ...trap/engines/engine.py.229: process: 0.84016 | load: 0.00311 [I 2022-10-09 02:51:52] ...trap/engines/engine.py.230: loss: 0.18305 [I 2022-10-09 02:51:52] ...optimizers/trijoint.py.151: total_norm: 18.955837 [I 2022-10-09 02:52:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 830/2304 [I 2022-10-09 02:52:00] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:39 [I 2022-10-09 02:52:00] ...trap/engines/engine.py.229: process: 0.84444 | load: 0.00318 [I 2022-10-09 02:52:00] ...trap/engines/engine.py.230: loss: 0.10652 [I 2022-10-09 02:52:00] ...optimizers/trijoint.py.151: total_norm: 11.951381 [I 2022-10-09 02:52:09] ...trap/engines/engine.py.224: train: epoch 29 | batch 840/2304 [I 2022-10-09 02:52:09] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:30 [I 2022-10-09 02:52:09] ...trap/engines/engine.py.229: process: 0.83766 | load: 0.00307 [I 2022-10-09 02:52:09] ...trap/engines/engine.py.230: loss: 0.13944 [I 2022-10-09 02:52:09] ...optimizers/trijoint.py.151: total_norm: 13.654295 [I 2022-10-09 02:52:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 850/2304 [I 2022-10-09 02:52:17] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:21 [I 2022-10-09 02:52:17] ...trap/engines/engine.py.229: process: 0.83879 | load: 0.00309 [I 2022-10-09 02:52:17] ...trap/engines/engine.py.230: loss: 0.15292 [I 2022-10-09 02:52:17] ...optimizers/trijoint.py.151: total_norm: 13.792741 [I 2022-10-09 02:52:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 860/2304 [I 2022-10-09 02:52:26] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:11 [I 2022-10-09 02:52:26] ...trap/engines/engine.py.229: process: 0.83987 | load: 0.00308 [I 2022-10-09 02:52:26] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-09 02:52:26] ...optimizers/trijoint.py.151: total_norm: 32.485138 [I 2022-10-09 02:52:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 870/2304 [I 2022-10-09 02:52:34] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:06 [I 2022-10-09 02:52:34] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00309 [I 2022-10-09 02:52:34] ...trap/engines/engine.py.230: loss: 0.21274 [I 2022-10-09 02:52:34] ...optimizers/trijoint.py.151: total_norm: 15.607676 [I 2022-10-09 02:52:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 880/2304 [I 2022-10-09 02:52:43] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:01 [I 2022-10-09 02:52:43] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00302 [I 2022-10-09 02:52:43] ...trap/engines/engine.py.230: loss: 0.18193 [I 2022-10-09 02:52:43] ...optimizers/trijoint.py.151: total_norm: 71.435165 [I 2022-10-09 02:52:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 890/2304 [I 2022-10-09 02:52:51] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:50 [I 2022-10-09 02:52:51] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00300 [I 2022-10-09 02:52:51] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-09 02:52:51] ...optimizers/trijoint.py.151: total_norm: 16.225397 [I 2022-10-09 02:52:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 900/2304 [I 2022-10-09 02:52:59] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:32 [I 2022-10-09 02:52:59] ...trap/engines/engine.py.229: process: 0.83691 | load: 0.00300 [I 2022-10-09 02:52:59] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-09 02:52:59] ...optimizers/trijoint.py.151: total_norm: 45.802990 [I 2022-10-09 02:53:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 910/2304 [I 2022-10-09 02:53:08] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:30 [I 2022-10-09 02:53:08] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00301 [I 2022-10-09 02:53:08] ...trap/engines/engine.py.230: loss: 0.16447 [I 2022-10-09 02:53:08] ...optimizers/trijoint.py.151: total_norm: 17.684053 [I 2022-10-09 02:53:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 920/2304 [I 2022-10-09 02:53:16] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:26 [I 2022-10-09 02:53:16] ...trap/engines/engine.py.229: process: 0.83506 | load: 0.00299 [I 2022-10-09 02:53:16] ...trap/engines/engine.py.230: loss: 0.11152 [I 2022-10-09 02:53:16] ...optimizers/trijoint.py.151: total_norm: 12.070103 [I 2022-10-09 02:53:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 930/2304 [I 2022-10-09 02:53:25] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:08 [I 2022-10-09 02:53:25] ...trap/engines/engine.py.229: process: 0.82503 | load: 0.00304 [I 2022-10-09 02:53:25] ...trap/engines/engine.py.230: loss: 0.17238 [I 2022-10-09 02:53:25] ...optimizers/trijoint.py.151: total_norm: 36.325974 [I 2022-10-09 02:53:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 940/2304 [I 2022-10-09 02:53:33] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:11 [I 2022-10-09 02:53:33] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00318 [I 2022-10-09 02:53:33] ...trap/engines/engine.py.230: loss: 0.25770 [I 2022-10-09 02:53:33] ...optimizers/trijoint.py.151: total_norm: 28.850554 [I 2022-10-09 02:53:41] ...trap/engines/engine.py.224: train: epoch 29 | batch 950/2304 [I 2022-10-09 02:53:41] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:00 [I 2022-10-09 02:53:41] ...trap/engines/engine.py.229: process: 0.84800 | load: 0.00301 [I 2022-10-09 02:53:41] ...trap/engines/engine.py.230: loss: 0.15860 [I 2022-10-09 02:53:41] ...optimizers/trijoint.py.151: total_norm: 20.896824 [I 2022-10-09 02:53:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 960/2304 [I 2022-10-09 02:53:50] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:49 [I 2022-10-09 02:53:50] ...trap/engines/engine.py.229: process: 0.83280 | load: 0.00301 [I 2022-10-09 02:53:50] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-09 02:53:50] ...optimizers/trijoint.py.151: total_norm: 22.665695 [I 2022-10-09 02:53:58] ...trap/engines/engine.py.224: train: epoch 29 | batch 970/2304 [I 2022-10-09 02:53:58] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:40 [I 2022-10-09 02:53:58] ...trap/engines/engine.py.229: process: 0.83486 | load: 0.00300 [I 2022-10-09 02:53:58] ...trap/engines/engine.py.230: loss: 0.20356 [I 2022-10-09 02:53:58] ...optimizers/trijoint.py.151: total_norm: 16.520399 [I 2022-10-09 02:54:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 980/2304 [I 2022-10-09 02:54:07] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:31 [I 2022-10-09 02:54:07] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00303 [I 2022-10-09 02:54:07] ...trap/engines/engine.py.230: loss: 0.18586 [I 2022-10-09 02:54:07] ...optimizers/trijoint.py.151: total_norm: 22.278162 [I 2022-10-09 02:54:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 990/2304 [I 2022-10-09 02:54:15] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:22 [I 2022-10-09 02:54:15] ...trap/engines/engine.py.229: process: 0.83594 | load: 0.00301 [I 2022-10-09 02:54:15] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-09 02:54:15] ...optimizers/trijoint.py.151: total_norm: 13.662451 [I 2022-10-09 02:54:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 1000/2304 [I 2022-10-09 02:54:24] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:13 [I 2022-10-09 02:54:24] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00301 [I 2022-10-09 02:54:24] ...trap/engines/engine.py.230: loss: 0.18351 [I 2022-10-09 02:54:24] ...optimizers/trijoint.py.151: total_norm: 15.244368 [I 2022-10-09 02:54:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 1010/2304 [I 2022-10-09 02:54:32] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:04 [I 2022-10-09 02:54:32] ...trap/engines/engine.py.229: process: 0.83076 | load: 0.00299 [I 2022-10-09 02:54:32] ...trap/engines/engine.py.230: loss: 0.20011 [I 2022-10-09 02:54:32] ...optimizers/trijoint.py.151: total_norm: 30.915728 [I 2022-10-09 02:54:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 1020/2304 [I 2022-10-09 02:54:40] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:59 [I 2022-10-09 02:54:40] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00331 [I 2022-10-09 02:54:40] ...trap/engines/engine.py.230: loss: 0.19372 [I 2022-10-09 02:54:40] ...optimizers/trijoint.py.151: total_norm: 18.430174 [I 2022-10-09 02:54:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 1030/2304 [I 2022-10-09 02:54:49] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:57 [I 2022-10-09 02:54:49] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00304 [I 2022-10-09 02:54:49] ...trap/engines/engine.py.230: loss: 0.24315 [I 2022-10-09 02:54:49] ...optimizers/trijoint.py.151: total_norm: 24.507339 [I 2022-10-09 02:54:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 1040/2304 [I 2022-10-09 02:54:57] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:43 [I 2022-10-09 02:54:57] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00302 [I 2022-10-09 02:54:57] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-09 02:54:57] ...optimizers/trijoint.py.151: total_norm: 18.806448 [I 2022-10-09 02:55:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 1050/2304 [I 2022-10-09 02:55:06] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:33 [I 2022-10-09 02:55:06] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00304 [I 2022-10-09 02:55:06] ...trap/engines/engine.py.230: loss: 0.17356 [I 2022-10-09 02:55:06] ...optimizers/trijoint.py.151: total_norm: 28.330849 [I 2022-10-09 02:55:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 1060/2304 [I 2022-10-09 02:55:14] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:22 [I 2022-10-09 02:55:14] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00307 [I 2022-10-09 02:55:14] ...trap/engines/engine.py.230: loss: 0.16242 [I 2022-10-09 02:55:14] ...optimizers/trijoint.py.151: total_norm: 14.630587 [I 2022-10-09 02:55:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 1070/2304 [I 2022-10-09 02:55:23] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:16 [I 2022-10-09 02:55:23] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00301 [I 2022-10-09 02:55:23] ...trap/engines/engine.py.230: loss: 0.18836 [I 2022-10-09 02:55:23] ...optimizers/trijoint.py.151: total_norm: 31.510906 [I 2022-10-09 02:55:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 1080/2304 [I 2022-10-09 02:55:31] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:11 [I 2022-10-09 02:55:31] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00301 [I 2022-10-09 02:55:31] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-09 02:55:31] ...optimizers/trijoint.py.151: total_norm: 34.085148 [I 2022-10-09 02:55:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 1090/2304 [I 2022-10-09 02:55:39] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:03 [I 2022-10-09 02:55:39] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00303 [I 2022-10-09 02:55:39] ...trap/engines/engine.py.230: loss: 0.17757 [I 2022-10-09 02:55:39] ...optimizers/trijoint.py.151: total_norm: 14.794780 [I 2022-10-09 02:55:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 1100/2304 [I 2022-10-09 02:55:48] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:53 [I 2022-10-09 02:55:48] ...trap/engines/engine.py.229: process: 0.83521 | load: 0.00300 [I 2022-10-09 02:55:48] ...trap/engines/engine.py.230: loss: 0.22298 [I 2022-10-09 02:55:48] ...optimizers/trijoint.py.151: total_norm: 30.490625 [I 2022-10-09 02:55:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 1110/2304 [I 2022-10-09 02:55:56] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:40 [I 2022-10-09 02:55:56] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00301 [I 2022-10-09 02:55:56] ...trap/engines/engine.py.230: loss: 0.20977 [I 2022-10-09 02:55:56] ...optimizers/trijoint.py.151: total_norm: 15.887280 [I 2022-10-09 02:56:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 1120/2304 [I 2022-10-09 02:56:05] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:35 [I 2022-10-09 02:56:05] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00302 [I 2022-10-09 02:56:05] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-09 02:56:05] ...optimizers/trijoint.py.151: total_norm: 21.891109 [I 2022-10-09 02:56:13] ...trap/engines/engine.py.224: train: epoch 29 | batch 1130/2304 [I 2022-10-09 02:56:13] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:22 [I 2022-10-09 02:56:13] ...trap/engines/engine.py.229: process: 0.83640 | load: 0.00300 [I 2022-10-09 02:56:13] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-09 02:56:13] ...optimizers/trijoint.py.151: total_norm: 11.121998 [I 2022-10-09 02:56:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 1140/2304 [I 2022-10-09 02:56:21] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:15 [I 2022-10-09 02:56:21] ...trap/engines/engine.py.229: process: 0.83327 | load: 0.00301 [I 2022-10-09 02:56:21] ...trap/engines/engine.py.230: loss: 0.24950 [I 2022-10-09 02:56:21] ...optimizers/trijoint.py.151: total_norm: 23.776363 [I 2022-10-09 02:56:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 1150/2304 [I 2022-10-09 02:56:30] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:08 [I 2022-10-09 02:56:30] ...trap/engines/engine.py.229: process: 0.83985 | load: 0.00300 [I 2022-10-09 02:56:30] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-09 02:56:30] ...optimizers/trijoint.py.151: total_norm: 9.909132 [I 2022-10-09 02:56:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 1160/2304 [I 2022-10-09 02:56:38] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:16:03 [I 2022-10-09 02:56:38] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00303 [I 2022-10-09 02:56:38] ...trap/engines/engine.py.230: loss: 0.17590 [I 2022-10-09 02:56:38] ...optimizers/trijoint.py.151: total_norm: 12.179625 [I 2022-10-09 02:56:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 1170/2304 [I 2022-10-09 02:56:47] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:55 [I 2022-10-09 02:56:47] ...trap/engines/engine.py.229: process: 0.83106 | load: 0.00299 [I 2022-10-09 02:56:47] ...trap/engines/engine.py.230: loss: 0.16545 [I 2022-10-09 02:56:47] ...optimizers/trijoint.py.151: total_norm: 14.647943 [I 2022-10-09 02:56:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 1180/2304 [I 2022-10-09 02:56:55] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:49 [I 2022-10-09 02:56:55] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00300 [I 2022-10-09 02:56:55] ...trap/engines/engine.py.230: loss: 0.13956 [I 2022-10-09 02:56:55] ...optimizers/trijoint.py.151: total_norm: 12.747055 [I 2022-10-09 02:57:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 1190/2304 [I 2022-10-09 02:57:04] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:40 [I 2022-10-09 02:57:04] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00301 [I 2022-10-09 02:57:04] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-09 02:57:04] ...optimizers/trijoint.py.151: total_norm: 15.180280 [I 2022-10-09 02:57:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 1200/2304 [I 2022-10-09 02:57:12] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:25 [I 2022-10-09 02:57:12] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00304 [I 2022-10-09 02:57:12] ...trap/engines/engine.py.230: loss: 0.13024 [I 2022-10-09 02:57:12] ...optimizers/trijoint.py.151: total_norm: 22.189423 [I 2022-10-09 02:57:20] ...trap/engines/engine.py.224: train: epoch 29 | batch 1210/2304 [I 2022-10-09 02:57:20] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:17 [I 2022-10-09 02:57:20] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00301 [I 2022-10-09 02:57:20] ...trap/engines/engine.py.230: loss: 0.14344 [I 2022-10-09 02:57:20] ...optimizers/trijoint.py.151: total_norm: 18.091135 [I 2022-10-09 02:57:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 1220/2304 [I 2022-10-09 02:57:29] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:10 [I 2022-10-09 02:57:29] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00300 [I 2022-10-09 02:57:29] ...trap/engines/engine.py.230: loss: 0.24330 [I 2022-10-09 02:57:29] ...optimizers/trijoint.py.151: total_norm: 13.704561 [I 2022-10-09 02:57:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 1230/2304 [I 2022-10-09 02:57:37] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:15:00 [I 2022-10-09 02:57:37] ...trap/engines/engine.py.229: process: 0.84302 | load: 0.00302 [I 2022-10-09 02:57:37] ...trap/engines/engine.py.230: loss: 0.24619 [I 2022-10-09 02:57:37] ...optimizers/trijoint.py.151: total_norm: 13.866188 [I 2022-10-09 02:57:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 1240/2304 [I 2022-10-09 02:57:46] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:14:55 [I 2022-10-09 02:57:46] ...trap/engines/engine.py.229: process: 0.83333 | load: 0.00307 [I 2022-10-09 02:57:46] ...trap/engines/engine.py.230: loss: 0.14570 [I 2022-10-09 02:57:46] ...optimizers/trijoint.py.151: total_norm: 12.117911 [I 2022-10-09 02:57:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 1250/2304 [I 2022-10-09 02:57:54] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:51 [I 2022-10-09 02:57:54] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00327 [I 2022-10-09 02:57:54] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-09 02:57:54] ...optimizers/trijoint.py.151: total_norm: 36.326595 [I 2022-10-09 02:58:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 1260/2304 [I 2022-10-09 02:58:02] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:31 [I 2022-10-09 02:58:02] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00301 [I 2022-10-09 02:58:02] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-09 02:58:02] ...optimizers/trijoint.py.151: total_norm: 20.032099 [I 2022-10-09 02:58:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 1270/2304 [I 2022-10-09 02:58:11] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:28 [I 2022-10-09 02:58:11] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00303 [I 2022-10-09 02:58:11] ...trap/engines/engine.py.230: loss: 0.20405 [I 2022-10-09 02:58:11] ...optimizers/trijoint.py.151: total_norm: 12.299228 [I 2022-10-09 02:58:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 1280/2304 [I 2022-10-09 02:58:19] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:22 [I 2022-10-09 02:58:19] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00305 [I 2022-10-09 02:58:19] ...trap/engines/engine.py.230: loss: 0.21883 [I 2022-10-09 02:58:19] ...optimizers/trijoint.py.151: total_norm: 31.859045 [I 2022-10-09 02:58:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 1290/2304 [I 2022-10-09 02:58:28] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:13 [I 2022-10-09 02:58:28] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00304 [I 2022-10-09 02:58:28] ...trap/engines/engine.py.230: loss: 0.15974 [I 2022-10-09 02:58:28] ...optimizers/trijoint.py.151: total_norm: 27.227514 [I 2022-10-09 02:58:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 1300/2304 [I 2022-10-09 02:58:36] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:02 [I 2022-10-09 02:58:36] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00302 [I 2022-10-09 02:58:36] ...trap/engines/engine.py.230: loss: 0.16960 [I 2022-10-09 02:58:36] ...optimizers/trijoint.py.151: total_norm: 20.400253 [I 2022-10-09 02:58:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 1310/2304 [I 2022-10-09 02:58:45] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:13:57 [I 2022-10-09 02:58:45] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00305 [I 2022-10-09 02:58:45] ...trap/engines/engine.py.230: loss: 0.16147 [I 2022-10-09 02:58:45] ...optimizers/trijoint.py.151: total_norm: 15.738146 [I 2022-10-09 02:58:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 1320/2304 [I 2022-10-09 02:58:53] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:46 [I 2022-10-09 02:58:53] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00300 [I 2022-10-09 02:58:53] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-09 02:58:53] ...optimizers/trijoint.py.151: total_norm: 9.907802 [I 2022-10-09 02:59:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 1330/2304 [I 2022-10-09 02:59:01] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:39 [I 2022-10-09 02:59:01] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00300 [I 2022-10-09 02:59:01] ...trap/engines/engine.py.230: loss: 0.10809 [I 2022-10-09 02:59:01] ...optimizers/trijoint.py.151: total_norm: 8.588322 [I 2022-10-09 02:59:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 1340/2304 [I 2022-10-09 02:59:10] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:33 [I 2022-10-09 02:59:10] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00302 [I 2022-10-09 02:59:10] ...trap/engines/engine.py.230: loss: 0.20075 [I 2022-10-09 02:59:10] ...optimizers/trijoint.py.151: total_norm: 10.763302 [I 2022-10-09 02:59:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 1350/2304 [I 2022-10-09 02:59:18] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:23 [I 2022-10-09 02:59:18] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00302 [I 2022-10-09 02:59:18] ...trap/engines/engine.py.230: loss: 0.16242 [I 2022-10-09 02:59:18] ...optimizers/trijoint.py.151: total_norm: 16.939877 [I 2022-10-09 02:59:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 1360/2304 [I 2022-10-09 02:59:27] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:14 [I 2022-10-09 02:59:27] ...trap/engines/engine.py.229: process: 0.84184 | load: 0.00300 [I 2022-10-09 02:59:27] ...trap/engines/engine.py.230: loss: 0.22814 [I 2022-10-09 02:59:27] ...optimizers/trijoint.py.151: total_norm: 13.643944 [I 2022-10-09 02:59:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 1370/2304 [I 2022-10-09 02:59:35] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:04 [I 2022-10-09 02:59:35] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00305 [I 2022-10-09 02:59:35] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-09 02:59:35] ...optimizers/trijoint.py.151: total_norm: 10.606091 [I 2022-10-09 02:59:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 1380/2304 [I 2022-10-09 02:59:44] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:12:57 [I 2022-10-09 02:59:44] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00305 [I 2022-10-09 02:59:44] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-09 02:59:44] ...optimizers/trijoint.py.151: total_norm: 13.253962 [I 2022-10-09 02:59:52] ...trap/engines/engine.py.224: train: epoch 29 | batch 1390/2304 [I 2022-10-09 02:59:52] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:12:47 [I 2022-10-09 02:59:52] ...trap/engines/engine.py.229: process: 0.83583 | load: 0.00300 [I 2022-10-09 02:59:52] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-09 02:59:52] ...optimizers/trijoint.py.151: total_norm: 19.599501 [I 2022-10-09 03:00:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 1400/2304 [I 2022-10-09 03:00:00] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:37 [I 2022-10-09 03:00:00] ...trap/engines/engine.py.229: process: 0.83266 | load: 0.00300 [I 2022-10-09 03:00:00] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-09 03:00:00] ...optimizers/trijoint.py.151: total_norm: 19.192972 [I 2022-10-09 03:00:09] ...trap/engines/engine.py.224: train: epoch 29 | batch 1410/2304 [I 2022-10-09 03:00:09] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:29 [I 2022-10-09 03:00:09] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00299 [I 2022-10-09 03:00:09] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-09 03:00:09] ...optimizers/trijoint.py.151: total_norm: 23.479347 [I 2022-10-09 03:00:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 1420/2304 [I 2022-10-09 03:00:17] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:25 [I 2022-10-09 03:00:17] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00300 [I 2022-10-09 03:00:17] ...trap/engines/engine.py.230: loss: 0.22946 [I 2022-10-09 03:00:17] ...optimizers/trijoint.py.151: total_norm: 32.056217 [I 2022-10-09 03:00:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 1430/2304 [I 2022-10-09 03:00:26] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:16 [I 2022-10-09 03:00:26] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00298 [I 2022-10-09 03:00:26] ...trap/engines/engine.py.230: loss: 0.27636 [I 2022-10-09 03:00:26] ...optimizers/trijoint.py.151: total_norm: 25.686188 [I 2022-10-09 03:00:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 1440/2304 [I 2022-10-09 03:00:34] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:07 [I 2022-10-09 03:00:34] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00301 [I 2022-10-09 03:00:34] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-09 03:00:34] ...optimizers/trijoint.py.151: total_norm: 19.661440 [I 2022-10-09 03:00:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 1450/2304 [I 2022-10-09 03:00:42] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:11:56 [I 2022-10-09 03:00:42] ...trap/engines/engine.py.229: process: 0.83889 | load: 0.00304 [I 2022-10-09 03:00:42] ...trap/engines/engine.py.230: loss: 0.20628 [I 2022-10-09 03:00:42] ...optimizers/trijoint.py.151: total_norm: 28.639580 [I 2022-10-09 03:00:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 1460/2304 [I 2022-10-09 03:00:51] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:48 [I 2022-10-09 03:00:51] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00301 [I 2022-10-09 03:00:51] ...trap/engines/engine.py.230: loss: 0.18012 [I 2022-10-09 03:00:51] ...optimizers/trijoint.py.151: total_norm: 44.192879 [I 2022-10-09 03:00:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 1470/2304 [I 2022-10-09 03:00:59] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:11:40 [I 2022-10-09 03:00:59] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00300 [I 2022-10-09 03:00:59] ...trap/engines/engine.py.230: loss: 0.21650 [I 2022-10-09 03:00:59] ...optimizers/trijoint.py.151: total_norm: 21.863367 [I 2022-10-09 03:01:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 1480/2304 [I 2022-10-09 03:01:08] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:32 [I 2022-10-09 03:01:08] ...trap/engines/engine.py.229: process: 0.84097 | load: 0.00302 [I 2022-10-09 03:01:08] ...trap/engines/engine.py.230: loss: 0.09531 [I 2022-10-09 03:01:08] ...optimizers/trijoint.py.151: total_norm: 10.216257 [I 2022-10-09 03:01:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 1490/2304 [I 2022-10-09 03:01:16] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:26 [I 2022-10-09 03:01:16] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00300 [I 2022-10-09 03:01:16] ...trap/engines/engine.py.230: loss: 0.20120 [I 2022-10-09 03:01:16] ...optimizers/trijoint.py.151: total_norm: 16.231190 [I 2022-10-09 03:01:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 1500/2304 [I 2022-10-09 03:01:25] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:17 [I 2022-10-09 03:01:25] ...trap/engines/engine.py.229: process: 0.83784 | load: 0.00299 [I 2022-10-09 03:01:25] ...trap/engines/engine.py.230: loss: 0.15614 [I 2022-10-09 03:01:25] ...optimizers/trijoint.py.151: total_norm: 10.654953 [I 2022-10-09 03:01:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 1510/2304 [I 2022-10-09 03:01:33] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:09 [I 2022-10-09 03:01:33] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00300 [I 2022-10-09 03:01:33] ...trap/engines/engine.py.230: loss: 0.16920 [I 2022-10-09 03:01:33] ...optimizers/trijoint.py.151: total_norm: 16.528999 [I 2022-10-09 03:01:41] ...trap/engines/engine.py.224: train: epoch 29 | batch 1520/2304 [I 2022-10-09 03:01:41] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:00 [I 2022-10-09 03:01:41] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00308 [I 2022-10-09 03:01:41] ...trap/engines/engine.py.230: loss: 0.16062 [I 2022-10-09 03:01:41] ...optimizers/trijoint.py.151: total_norm: 16.418673 [I 2022-10-09 03:01:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 1530/2304 [I 2022-10-09 03:01:50] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:50 [I 2022-10-09 03:01:50] ...trap/engines/engine.py.229: process: 0.83632 | load: 0.00301 [I 2022-10-09 03:01:50] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-09 03:01:50] ...optimizers/trijoint.py.151: total_norm: 15.646392 [I 2022-10-09 03:01:58] ...trap/engines/engine.py.224: train: epoch 29 | batch 1540/2304 [I 2022-10-09 03:01:58] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:42 [I 2022-10-09 03:01:58] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00301 [I 2022-10-09 03:01:58] ...trap/engines/engine.py.230: loss: 0.16126 [I 2022-10-09 03:01:58] ...optimizers/trijoint.py.151: total_norm: 33.898731 [I 2022-10-09 03:02:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 1550/2304 [I 2022-10-09 03:02:07] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:36 [I 2022-10-09 03:02:07] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00302 [I 2022-10-09 03:02:07] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-09 03:02:07] ...optimizers/trijoint.py.151: total_norm: 18.008081 [I 2022-10-09 03:02:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 1560/2304 [I 2022-10-09 03:02:15] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:23 [I 2022-10-09 03:02:15] ...trap/engines/engine.py.229: process: 0.82982 | load: 0.00305 [I 2022-10-09 03:02:15] ...trap/engines/engine.py.230: loss: 0.14966 [I 2022-10-09 03:02:15] ...optimizers/trijoint.py.151: total_norm: 12.708518 [I 2022-10-09 03:02:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 1570/2304 [I 2022-10-09 03:02:24] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:19 [I 2022-10-09 03:02:24] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00299 [I 2022-10-09 03:02:24] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-09 03:02:24] ...optimizers/trijoint.py.151: total_norm: 19.231441 [I 2022-10-09 03:02:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 1580/2304 [I 2022-10-09 03:02:32] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:07 [I 2022-10-09 03:02:32] ...trap/engines/engine.py.229: process: 0.83392 | load: 0.00299 [I 2022-10-09 03:02:32] ...trap/engines/engine.py.230: loss: 0.20228 [I 2022-10-09 03:02:32] ...optimizers/trijoint.py.151: total_norm: 26.308378 [I 2022-10-09 03:02:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 1590/2304 [I 2022-10-09 03:02:40] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:09:58 [I 2022-10-09 03:02:40] ...trap/engines/engine.py.229: process: 0.83016 | load: 0.00300 [I 2022-10-09 03:02:40] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-09 03:02:40] ...optimizers/trijoint.py.151: total_norm: 10.462749 [I 2022-10-09 03:02:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 1600/2304 [I 2022-10-09 03:02:49] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:51 [I 2022-10-09 03:02:49] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00300 [I 2022-10-09 03:02:49] ...trap/engines/engine.py.230: loss: 0.09078 [I 2022-10-09 03:02:49] ...optimizers/trijoint.py.151: total_norm: 9.703726 [I 2022-10-09 03:02:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 1610/2304 [I 2022-10-09 03:02:57] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:41 [I 2022-10-09 03:02:57] ...trap/engines/engine.py.229: process: 0.83229 | load: 0.00300 [I 2022-10-09 03:02:57] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-09 03:02:57] ...optimizers/trijoint.py.151: total_norm: 12.275567 [I 2022-10-09 03:03:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 1620/2304 [I 2022-10-09 03:03:06] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:33 [I 2022-10-09 03:03:06] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00302 [I 2022-10-09 03:03:06] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-09 03:03:06] ...optimizers/trijoint.py.151: total_norm: 11.323065 [I 2022-10-09 03:03:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 1630/2304 [I 2022-10-09 03:03:14] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:25 [I 2022-10-09 03:03:14] ...trap/engines/engine.py.229: process: 0.84807 | load: 0.00301 [I 2022-10-09 03:03:14] ...trap/engines/engine.py.230: loss: 0.19839 [I 2022-10-09 03:03:14] ...optimizers/trijoint.py.151: total_norm: 25.203737 [I 2022-10-09 03:03:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 1640/2304 [I 2022-10-09 03:03:22] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:17 [I 2022-10-09 03:03:22] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00304 [I 2022-10-09 03:03:22] ...trap/engines/engine.py.230: loss: 0.18607 [I 2022-10-09 03:03:22] ...optimizers/trijoint.py.151: total_norm: 25.844995 [I 2022-10-09 03:03:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 1650/2304 [I 2022-10-09 03:03:31] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:10 [I 2022-10-09 03:03:31] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00300 [I 2022-10-09 03:03:31] ...trap/engines/engine.py.230: loss: 0.18380 [I 2022-10-09 03:03:31] ...optimizers/trijoint.py.151: total_norm: 14.306440 [I 2022-10-09 03:03:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 1660/2304 [I 2022-10-09 03:03:39] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:01 [I 2022-10-09 03:03:39] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00302 [I 2022-10-09 03:03:39] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-09 03:03:39] ...optimizers/trijoint.py.151: total_norm: 9.378320 [I 2022-10-09 03:03:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 1670/2304 [I 2022-10-09 03:03:48] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:53 [I 2022-10-09 03:03:48] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00302 [I 2022-10-09 03:03:48] ...trap/engines/engine.py.230: loss: 0.17881 [I 2022-10-09 03:03:48] ...optimizers/trijoint.py.151: total_norm: 17.685673 [I 2022-10-09 03:03:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 1680/2304 [I 2022-10-09 03:03:56] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:46 [I 2022-10-09 03:03:56] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00305 [I 2022-10-09 03:03:56] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-09 03:03:56] ...optimizers/trijoint.py.151: total_norm: 12.314198 [I 2022-10-09 03:04:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 1690/2304 [I 2022-10-09 03:04:05] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:36 [I 2022-10-09 03:04:05] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00301 [I 2022-10-09 03:04:05] ...trap/engines/engine.py.230: loss: 0.20665 [I 2022-10-09 03:04:05] ...optimizers/trijoint.py.151: total_norm: 75.293365 [I 2022-10-09 03:04:13] ...trap/engines/engine.py.224: train: epoch 29 | batch 1700/2304 [I 2022-10-09 03:04:13] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:27 [I 2022-10-09 03:04:13] ...trap/engines/engine.py.229: process: 0.83023 | load: 0.00302 [I 2022-10-09 03:04:13] ...trap/engines/engine.py.230: loss: 0.19239 [I 2022-10-09 03:04:13] ...optimizers/trijoint.py.151: total_norm: 17.160896 [I 2022-10-09 03:04:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 1710/2304 [I 2022-10-09 03:04:21] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:20 [I 2022-10-09 03:04:21] ...trap/engines/engine.py.229: process: 0.84844 | load: 0.00299 [I 2022-10-09 03:04:21] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-09 03:04:21] ...optimizers/trijoint.py.151: total_norm: 6.792315 [I 2022-10-09 03:04:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 1720/2304 [I 2022-10-09 03:04:30] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:12 [I 2022-10-09 03:04:30] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00304 [I 2022-10-09 03:04:30] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-09 03:04:30] ...optimizers/trijoint.py.151: total_norm: 22.059244 [I 2022-10-09 03:04:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 1730/2304 [I 2022-10-09 03:04:38] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:03 [I 2022-10-09 03:04:38] ...trap/engines/engine.py.229: process: 0.84511 | load: 0.00300 [I 2022-10-09 03:04:38] ...trap/engines/engine.py.230: loss: 0.20877 [I 2022-10-09 03:04:38] ...optimizers/trijoint.py.151: total_norm: 31.221853 [I 2022-10-09 03:04:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 1740/2304 [I 2022-10-09 03:04:47] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:07:56 [I 2022-10-09 03:04:47] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00305 [I 2022-10-09 03:04:47] ...trap/engines/engine.py.230: loss: 0.11828 [I 2022-10-09 03:04:47] ...optimizers/trijoint.py.151: total_norm: 13.540450 [I 2022-10-09 03:04:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 1750/2304 [I 2022-10-09 03:04:55] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:46 [I 2022-10-09 03:04:55] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00301 [I 2022-10-09 03:04:55] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-09 03:04:55] ...optimizers/trijoint.py.151: total_norm: 16.170443 [I 2022-10-09 03:05:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 1760/2304 [I 2022-10-09 03:05:04] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:07:37 [I 2022-10-09 03:05:04] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00306 [I 2022-10-09 03:05:04] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-09 03:05:04] ...optimizers/trijoint.py.151: total_norm: 14.531668 [I 2022-10-09 03:05:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 1770/2304 [I 2022-10-09 03:05:12] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:31 [I 2022-10-09 03:05:12] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00308 [I 2022-10-09 03:05:12] ...trap/engines/engine.py.230: loss: 0.17346 [I 2022-10-09 03:05:12] ...optimizers/trijoint.py.151: total_norm: 32.565746 [I 2022-10-09 03:05:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 1780/2304 [I 2022-10-09 03:05:21] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:07:20 [I 2022-10-09 03:05:21] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00310 [I 2022-10-09 03:05:21] ...trap/engines/engine.py.230: loss: 0.24792 [I 2022-10-09 03:05:21] ...optimizers/trijoint.py.151: total_norm: 22.561331 [I 2022-10-09 03:05:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 1790/2304 [I 2022-10-09 03:05:29] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:13 [I 2022-10-09 03:05:29] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00303 [I 2022-10-09 03:05:29] ...trap/engines/engine.py.230: loss: 0.23021 [I 2022-10-09 03:05:29] ...optimizers/trijoint.py.151: total_norm: 13.035790 [I 2022-10-09 03:05:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 1800/2304 [I 2022-10-09 03:05:37] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:02 [I 2022-10-09 03:05:37] ...trap/engines/engine.py.229: process: 0.84357 | load: 0.00302 [I 2022-10-09 03:05:37] ...trap/engines/engine.py.230: loss: 0.20723 [I 2022-10-09 03:05:37] ...optimizers/trijoint.py.151: total_norm: 15.035983 [I 2022-10-09 03:05:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 1810/2304 [I 2022-10-09 03:05:46] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:06:55 [I 2022-10-09 03:05:46] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00302 [I 2022-10-09 03:05:46] ...trap/engines/engine.py.230: loss: 0.21922 [I 2022-10-09 03:05:46] ...optimizers/trijoint.py.151: total_norm: 20.548414 [I 2022-10-09 03:05:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 1820/2304 [I 2022-10-09 03:05:54] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:46 [I 2022-10-09 03:05:54] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00300 [I 2022-10-09 03:05:54] ...trap/engines/engine.py.230: loss: 0.24118 [I 2022-10-09 03:05:54] ...optimizers/trijoint.py.151: total_norm: 14.135504 [I 2022-10-09 03:06:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 1830/2304 [I 2022-10-09 03:06:03] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:06:37 [I 2022-10-09 03:06:03] ...trap/engines/engine.py.229: process: 0.83521 | load: 0.00301 [I 2022-10-09 03:06:03] ...trap/engines/engine.py.230: loss: 0.12117 [I 2022-10-09 03:06:03] ...optimizers/trijoint.py.151: total_norm: 17.594801 [I 2022-10-09 03:06:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 1840/2304 [I 2022-10-09 03:06:11] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:29 [I 2022-10-09 03:06:11] ...trap/engines/engine.py.229: process: 0.83076 | load: 0.00303 [I 2022-10-09 03:06:11] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-09 03:06:11] ...optimizers/trijoint.py.151: total_norm: 7.701903 [I 2022-10-09 03:06:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 1850/2304 [I 2022-10-09 03:06:19] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:20 [I 2022-10-09 03:06:19] ...trap/engines/engine.py.229: process: 0.83198 | load: 0.00301 [I 2022-10-09 03:06:19] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-09 03:06:19] ...optimizers/trijoint.py.151: total_norm: 25.191715 [I 2022-10-09 03:06:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 1860/2304 [I 2022-10-09 03:06:28] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:15 [I 2022-10-09 03:06:28] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00302 [I 2022-10-09 03:06:28] ...trap/engines/engine.py.230: loss: 0.09186 [I 2022-10-09 03:06:28] ...optimizers/trijoint.py.151: total_norm: 11.095167 [I 2022-10-09 03:06:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 1870/2304 [I 2022-10-09 03:06:36] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:05 [I 2022-10-09 03:06:36] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00304 [I 2022-10-09 03:06:36] ...trap/engines/engine.py.230: loss: 0.15332 [I 2022-10-09 03:06:36] ...optimizers/trijoint.py.151: total_norm: 14.540059 [I 2022-10-09 03:06:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 1880/2304 [I 2022-10-09 03:06:45] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:05:57 [I 2022-10-09 03:06:45] ...trap/engines/engine.py.229: process: 0.83280 | load: 0.00302 [I 2022-10-09 03:06:45] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-09 03:06:45] ...optimizers/trijoint.py.151: total_norm: 12.451950 [I 2022-10-09 03:06:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 1890/2304 [I 2022-10-09 03:06:53] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:48 [I 2022-10-09 03:06:53] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00302 [I 2022-10-09 03:06:53] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-09 03:06:53] ...optimizers/trijoint.py.151: total_norm: 14.099421 [I 2022-10-09 03:07:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 1900/2304 [I 2022-10-09 03:07:02] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:39 [I 2022-10-09 03:07:02] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00303 [I 2022-10-09 03:07:02] ...trap/engines/engine.py.230: loss: 0.20898 [I 2022-10-09 03:07:02] ...optimizers/trijoint.py.151: total_norm: 27.158239 [I 2022-10-09 03:07:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 1910/2304 [I 2022-10-09 03:07:10] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:31 [I 2022-10-09 03:07:10] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00307 [I 2022-10-09 03:07:10] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-09 03:07:10] ...optimizers/trijoint.py.151: total_norm: 9.867652 [I 2022-10-09 03:07:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 1920/2304 [I 2022-10-09 03:07:18] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:21 [I 2022-10-09 03:07:18] ...trap/engines/engine.py.229: process: 0.85046 | load: 0.00299 [I 2022-10-09 03:07:18] ...trap/engines/engine.py.230: loss: 0.16239 [I 2022-10-09 03:07:18] ...optimizers/trijoint.py.151: total_norm: 23.046206 [I 2022-10-09 03:07:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 1930/2304 [I 2022-10-09 03:07:27] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:13 [I 2022-10-09 03:07:27] ...trap/engines/engine.py.229: process: 0.82949 | load: 0.00302 [I 2022-10-09 03:07:27] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-09 03:07:27] ...optimizers/trijoint.py.151: total_norm: 25.570351 [I 2022-10-09 03:07:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 1940/2304 [I 2022-10-09 03:07:35] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:05 [I 2022-10-09 03:07:35] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00305 [I 2022-10-09 03:07:35] ...trap/engines/engine.py.230: loss: 0.14320 [I 2022-10-09 03:07:35] ...optimizers/trijoint.py.151: total_norm: 10.079926 [I 2022-10-09 03:07:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 1950/2304 [I 2022-10-09 03:07:44] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:04:56 [I 2022-10-09 03:07:44] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00300 [I 2022-10-09 03:07:44] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-09 03:07:44] ...optimizers/trijoint.py.151: total_norm: 9.705129 [I 2022-10-09 03:07:52] ...trap/engines/engine.py.224: train: epoch 29 | batch 1960/2304 [I 2022-10-09 03:07:52] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:48 [I 2022-10-09 03:07:52] ...trap/engines/engine.py.229: process: 0.83126 | load: 0.00300 [I 2022-10-09 03:07:52] ...trap/engines/engine.py.230: loss: 0.21045 [I 2022-10-09 03:07:52] ...optimizers/trijoint.py.151: total_norm: 24.007036 [I 2022-10-09 03:08:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 1970/2304 [I 2022-10-09 03:08:00] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:39 [I 2022-10-09 03:08:00] ...trap/engines/engine.py.229: process: 0.83455 | load: 0.00302 [I 2022-10-09 03:08:00] ...trap/engines/engine.py.230: loss: 0.20917 [I 2022-10-09 03:08:00] ...optimizers/trijoint.py.151: total_norm: 13.947658 [I 2022-10-09 03:08:09] ...trap/engines/engine.py.224: train: epoch 29 | batch 1980/2304 [I 2022-10-09 03:08:09] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:32 [I 2022-10-09 03:08:09] ...trap/engines/engine.py.229: process: 0.84444 | load: 0.00299 [I 2022-10-09 03:08:09] ...trap/engines/engine.py.230: loss: 0.14048 [I 2022-10-09 03:08:09] ...optimizers/trijoint.py.151: total_norm: 14.815460 [I 2022-10-09 03:08:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 1990/2304 [I 2022-10-09 03:08:17] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:22 [I 2022-10-09 03:08:17] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00297 [I 2022-10-09 03:08:17] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-09 03:08:17] ...optimizers/trijoint.py.151: total_norm: 18.874500 [I 2022-10-09 03:08:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 2000/2304 [I 2022-10-09 03:08:26] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:14 [I 2022-10-09 03:08:26] ...trap/engines/engine.py.229: process: 0.83434 | load: 0.00302 [I 2022-10-09 03:08:26] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-09 03:08:26] ...optimizers/trijoint.py.151: total_norm: 20.854240 [I 2022-10-09 03:08:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 2010/2304 [I 2022-10-09 03:08:34] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:07 [I 2022-10-09 03:08:34] ...trap/engines/engine.py.229: process: 0.83235 | load: 0.00304 [I 2022-10-09 03:08:34] ...trap/engines/engine.py.230: loss: 0.22441 [I 2022-10-09 03:08:34] ...optimizers/trijoint.py.151: total_norm: 16.238308 [I 2022-10-09 03:08:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 2020/2304 [I 2022-10-09 03:08:42] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:59 [I 2022-10-09 03:08:42] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00299 [I 2022-10-09 03:08:42] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-09 03:08:42] ...optimizers/trijoint.py.151: total_norm: 13.247973 [I 2022-10-09 03:08:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 2030/2304 [I 2022-10-09 03:08:51] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:48 [I 2022-10-09 03:08:51] ...trap/engines/engine.py.229: process: 0.82854 | load: 0.00306 [I 2022-10-09 03:08:51] ...trap/engines/engine.py.230: loss: 0.12317 [I 2022-10-09 03:08:51] ...optimizers/trijoint.py.151: total_norm: 13.498890 [I 2022-10-09 03:08:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 2040/2304 [I 2022-10-09 03:08:59] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:43 [I 2022-10-09 03:08:59] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00304 [I 2022-10-09 03:08:59] ...trap/engines/engine.py.230: loss: 0.22027 [I 2022-10-09 03:08:59] ...optimizers/trijoint.py.151: total_norm: 12.060953 [I 2022-10-09 03:09:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 2050/2304 [I 2022-10-09 03:09:08] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:32 [I 2022-10-09 03:09:08] ...trap/engines/engine.py.229: process: 0.84598 | load: 0.00309 [I 2022-10-09 03:09:08] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-09 03:09:08] ...optimizers/trijoint.py.151: total_norm: 13.833984 [I 2022-10-09 03:09:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 2060/2304 [I 2022-10-09 03:09:16] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:23 [I 2022-10-09 03:09:16] ...trap/engines/engine.py.229: process: 0.83395 | load: 0.00306 [I 2022-10-09 03:09:16] ...trap/engines/engine.py.230: loss: 0.18169 [I 2022-10-09 03:09:16] ...optimizers/trijoint.py.151: total_norm: 23.981180 [I 2022-10-09 03:09:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 2070/2304 [I 2022-10-09 03:09:25] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:17 [I 2022-10-09 03:09:25] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00303 [I 2022-10-09 03:09:25] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-09 03:09:25] ...optimizers/trijoint.py.151: total_norm: 22.918499 [I 2022-10-09 03:09:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 2080/2304 [I 2022-10-09 03:09:33] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:09 [I 2022-10-09 03:09:33] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00300 [I 2022-10-09 03:09:33] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-09 03:09:33] ...optimizers/trijoint.py.151: total_norm: 14.693501 [I 2022-10-09 03:09:41] ...trap/engines/engine.py.224: train: epoch 29 | batch 2090/2304 [I 2022-10-09 03:09:41] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:02:59 [I 2022-10-09 03:09:41] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00301 [I 2022-10-09 03:09:41] ...trap/engines/engine.py.230: loss: 0.21702 [I 2022-10-09 03:09:41] ...optimizers/trijoint.py.151: total_norm: 18.870792 [I 2022-10-09 03:09:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 2100/2304 [I 2022-10-09 03:09:50] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:51 [I 2022-10-09 03:09:50] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00305 [I 2022-10-09 03:09:50] ...trap/engines/engine.py.230: loss: 0.23047 [I 2022-10-09 03:09:50] ...optimizers/trijoint.py.151: total_norm: 18.048197 [I 2022-10-09 03:09:58] ...trap/engines/engine.py.224: train: epoch 29 | batch 2110/2304 [I 2022-10-09 03:09:58] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:43 [I 2022-10-09 03:09:58] ...trap/engines/engine.py.229: process: 0.83466 | load: 0.00302 [I 2022-10-09 03:09:58] ...trap/engines/engine.py.230: loss: 0.12013 [I 2022-10-09 03:09:58] ...optimizers/trijoint.py.151: total_norm: 11.086475 [I 2022-10-09 03:10:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 2120/2304 [I 2022-10-09 03:10:07] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:34 [I 2022-10-09 03:10:07] ...trap/engines/engine.py.229: process: 0.84074 | load: 0.00300 [I 2022-10-09 03:10:07] ...trap/engines/engine.py.230: loss: 0.12499 [I 2022-10-09 03:10:07] ...optimizers/trijoint.py.151: total_norm: 10.871661 [I 2022-10-09 03:10:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 2130/2304 [I 2022-10-09 03:10:15] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:26 [I 2022-10-09 03:10:15] ...trap/engines/engine.py.229: process: 0.82457 | load: 0.00302 [I 2022-10-09 03:10:15] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-09 03:10:15] ...optimizers/trijoint.py.151: total_norm: 20.532232 [I 2022-10-09 03:10:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 2140/2304 [I 2022-10-09 03:10:23] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:17 [I 2022-10-09 03:10:23] ...trap/engines/engine.py.229: process: 0.84601 | load: 0.00301 [I 2022-10-09 03:10:23] ...trap/engines/engine.py.230: loss: 0.21511 [I 2022-10-09 03:10:23] ...optimizers/trijoint.py.151: total_norm: 16.303377 [I 2022-10-09 03:10:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 2150/2304 [I 2022-10-09 03:10:32] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:08 [I 2022-10-09 03:10:32] ...trap/engines/engine.py.229: process: 0.83636 | load: 0.00301 [I 2022-10-09 03:10:32] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-09 03:10:32] ...optimizers/trijoint.py.151: total_norm: 12.604830 [I 2022-10-09 03:10:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 2160/2304 [I 2022-10-09 03:10:40] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:01 [I 2022-10-09 03:10:40] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00302 [I 2022-10-09 03:10:40] ...trap/engines/engine.py.230: loss: 0.15524 [I 2022-10-09 03:10:40] ...optimizers/trijoint.py.151: total_norm: 15.195634 [I 2022-10-09 03:10:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 2170/2304 [I 2022-10-09 03:10:49] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:53 [I 2022-10-09 03:10:49] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00302 [I 2022-10-09 03:10:49] ...trap/engines/engine.py.230: loss: 0.17005 [I 2022-10-09 03:10:49] ...optimizers/trijoint.py.151: total_norm: 18.795494 [I 2022-10-09 03:10:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 2180/2304 [I 2022-10-09 03:10:57] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:44 [I 2022-10-09 03:10:57] ...trap/engines/engine.py.229: process: 0.83579 | load: 0.00303 [I 2022-10-09 03:10:57] ...trap/engines/engine.py.230: loss: 0.17346 [I 2022-10-09 03:10:57] ...optimizers/trijoint.py.151: total_norm: 10.152102 [I 2022-10-09 03:11:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 2190/2304 [I 2022-10-09 03:11:06] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:36 [I 2022-10-09 03:11:06] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00309 [I 2022-10-09 03:11:06] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-09 03:11:06] ...optimizers/trijoint.py.151: total_norm: 26.160894 [I 2022-10-09 03:11:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 2200/2304 [I 2022-10-09 03:11:14] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:27 [I 2022-10-09 03:11:14] ...trap/engines/engine.py.229: process: 0.83912 | load: 0.00311 [I 2022-10-09 03:11:14] ...trap/engines/engine.py.230: loss: 0.21859 [I 2022-10-09 03:11:14] ...optimizers/trijoint.py.151: total_norm: 26.901846 [I 2022-10-09 03:11:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 2210/2304 [I 2022-10-09 03:11:22] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:19 [I 2022-10-09 03:11:22] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00316 [I 2022-10-09 03:11:22] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-09 03:11:22] ...optimizers/trijoint.py.151: total_norm: 23.513731 [I 2022-10-09 03:11:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 2220/2304 [I 2022-10-09 03:11:31] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:10 [I 2022-10-09 03:11:31] ...trap/engines/engine.py.229: process: 0.83854 | load: 0.00310 [I 2022-10-09 03:11:31] ...trap/engines/engine.py.230: loss: 0.16156 [I 2022-10-09 03:11:31] ...optimizers/trijoint.py.151: total_norm: 10.418463 [I 2022-10-09 03:11:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 2230/2304 [I 2022-10-09 03:11:39] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:02 [I 2022-10-09 03:11:39] ...trap/engines/engine.py.229: process: 0.83649 | load: 0.00309 [I 2022-10-09 03:11:39] ...trap/engines/engine.py.230: loss: 0.17713 [I 2022-10-09 03:11:39] ...optimizers/trijoint.py.151: total_norm: 22.995514 [I 2022-10-09 03:11:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 2240/2304 [I 2022-10-09 03:11:48] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:53 [I 2022-10-09 03:11:48] ...trap/engines/engine.py.229: process: 0.84021 | load: 0.00313 [I 2022-10-09 03:11:48] ...trap/engines/engine.py.230: loss: 0.18334 [I 2022-10-09 03:11:48] ...optimizers/trijoint.py.151: total_norm: 22.472330 [I 2022-10-09 03:11:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 2250/2304 [I 2022-10-09 03:11:56] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:00:45 [I 2022-10-09 03:11:56] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00311 [I 2022-10-09 03:11:56] ...trap/engines/engine.py.230: loss: 0.23718 [I 2022-10-09 03:11:56] ...optimizers/trijoint.py.151: total_norm: 16.431620 [I 2022-10-09 03:12:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 2260/2304 [I 2022-10-09 03:12:05] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:37 [I 2022-10-09 03:12:05] ...trap/engines/engine.py.229: process: 0.84310 | load: 0.00308 [I 2022-10-09 03:12:05] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-09 03:12:05] ...optimizers/trijoint.py.151: total_norm: 14.391628 [I 2022-10-09 03:12:13] ...trap/engines/engine.py.224: train: epoch 29 | batch 2270/2304 [I 2022-10-09 03:12:13] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:28 [I 2022-10-09 03:12:13] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00310 [I 2022-10-09 03:12:13] ...trap/engines/engine.py.230: loss: 0.17471 [I 2022-10-09 03:12:13] ...optimizers/trijoint.py.151: total_norm: 17.784710 [I 2022-10-09 03:12:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 2280/2304 [I 2022-10-09 03:12:21] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:20 [I 2022-10-09 03:12:21] ...trap/engines/engine.py.229: process: 0.83209 | load: 0.00312 [I 2022-10-09 03:12:21] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-09 03:12:21] ...optimizers/trijoint.py.151: total_norm: 13.037345 [I 2022-10-09 03:12:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 2290/2304 [I 2022-10-09 03:12:30] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:11 [I 2022-10-09 03:12:30] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00310 [I 2022-10-09 03:12:30] ...trap/engines/engine.py.230: loss: 0.23975 [I 2022-10-09 03:12:30] ...optimizers/trijoint.py.151: total_norm: 22.695288 [I 2022-10-09 03:12:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 2300/2304 [I 2022-10-09 03:12:38] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:03 [I 2022-10-09 03:12:38] ...trap/engines/engine.py.229: process: 0.83497 | load: 0.00017 [I 2022-10-09 03:12:38] ...trap/engines/engine.py.230: loss: 0.11522 [I 2022-10-09 03:12:38] ...optimizers/trijoint.py.151: total_norm: 12.307801 [I 2022-10-09 03:12:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 2304/2304 [I 2022-10-09 03:12:42] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:00 [I 2022-10-09 03:12:42] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00017 [I 2022-10-09 03:12:42] ...trap/engines/engine.py.230: loss: 0.26925 [I 2022-10-09 03:12:42] ...optimizers/trijoint.py.151: total_norm: 13.404657 [S 2022-10-09 03:12:43] ...trap/engines/engine.py.129: train_epoch.epoch: 29 [S 2022-10-09 03:12:43] ...trap/engines/engine.py.129: train_epoch.loss: 0.16717414078575413 [I 2022-10-09 03:12:43] ...s/criterions/triplet.py.84: margin = 0.20000000000000012 [I 2022-10-09 03:12:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 29 [I 2022-10-09 03:12:43] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 03:12:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 03:12:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 03:12:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 03:12:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 03:12:47] ...trap/engines/engine.py.321: eval: epoch 29 | batch 0/510 [I 2022-10-09 03:12:47] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:51 [I 2022-10-09 03:12:47] ...trap/engines/engine.py.326: process: 3.39570 | load: 2.84754 [I 2022-10-09 03:12:52] ...trap/engines/engine.py.321: eval: epoch 29 | batch 10/510 [I 2022-10-09 03:12:52] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:27 [I 2022-10-09 03:12:52] ...trap/engines/engine.py.326: process: 0.60202 | load: 0.00019 [I 2022-10-09 03:12:58] ...trap/engines/engine.py.321: eval: epoch 29 | batch 20/510 [I 2022-10-09 03:12:58] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:04 [I 2022-10-09 03:12:58] ...trap/engines/engine.py.326: process: 0.65235 | load: 0.00019 [I 2022-10-09 03:13:04] ...trap/engines/engine.py.321: eval: epoch 29 | batch 30/510 [I 2022-10-09 03:13:04] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:48 [I 2022-10-09 03:13:04] ...trap/engines/engine.py.326: process: 0.71825 | load: 0.00019 [I 2022-10-09 03:13:11] ...trap/engines/engine.py.321: eval: epoch 29 | batch 40/510 [I 2022-10-09 03:13:11] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:05:35 [I 2022-10-09 03:13:11] ...trap/engines/engine.py.326: process: 0.68784 | load: 0.00020 [I 2022-10-09 03:13:16] ...trap/engines/engine.py.321: eval: epoch 29 | batch 50/510 [I 2022-10-09 03:13:16] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:53 [I 2022-10-09 03:13:16] ...trap/engines/engine.py.326: process: 0.45852 | load: 0.00019 [I 2022-10-09 03:13:20] ...trap/engines/engine.py.321: eval: epoch 29 | batch 60/510 [I 2022-10-09 03:13:20] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:15 [I 2022-10-09 03:13:20] ...trap/engines/engine.py.326: process: 0.43143 | load: 0.00020 [I 2022-10-09 03:13:24] ...trap/engines/engine.py.321: eval: epoch 29 | batch 70/510 [I 2022-10-09 03:13:24] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:06 [I 2022-10-09 03:13:24] ...trap/engines/engine.py.326: process: 0.42597 | load: 0.00020 [I 2022-10-09 03:13:29] ...trap/engines/engine.py.321: eval: epoch 29 | batch 80/510 [I 2022-10-09 03:13:29] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:30 [I 2022-10-09 03:13:29] ...trap/engines/engine.py.326: process: 0.58354 | load: 0.00025 [I 2022-10-09 03:13:34] ...trap/engines/engine.py.321: eval: epoch 29 | batch 90/510 [I 2022-10-09 03:13:34] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:22 [I 2022-10-09 03:13:34] ...trap/engines/engine.py.326: process: 0.43429 | load: 0.00019 [I 2022-10-09 03:13:38] ...trap/engines/engine.py.321: eval: epoch 29 | batch 100/510 [I 2022-10-09 03:13:38] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:55 [I 2022-10-09 03:13:38] ...trap/engines/engine.py.326: process: 0.31045 | load: 0.00018 [I 2022-10-09 03:13:42] ...trap/engines/engine.py.321: eval: epoch 29 | batch 110/510 [I 2022-10-09 03:13:42] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:11 [I 2022-10-09 03:13:42] ...trap/engines/engine.py.326: process: 0.31395 | load: 0.00021 [I 2022-10-09 03:13:45] ...trap/engines/engine.py.321: eval: epoch 29 | batch 120/510 [I 2022-10-09 03:13:45] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-09 03:13:45] ...trap/engines/engine.py.326: process: 0.31760 | load: 0.00020 [I 2022-10-09 03:13:48] ...trap/engines/engine.py.321: eval: epoch 29 | batch 130/510 [I 2022-10-09 03:13:48] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-09 03:13:48] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00021 [I 2022-10-09 03:13:51] ...trap/engines/engine.py.321: eval: epoch 29 | batch 140/510 [I 2022-10-09 03:13:51] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:58 [I 2022-10-09 03:13:51] ...trap/engines/engine.py.326: process: 0.32379 | load: 0.00019 [I 2022-10-09 03:13:54] ...trap/engines/engine.py.321: eval: epoch 29 | batch 150/510 [I 2022-10-09 03:13:54] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:54 [I 2022-10-09 03:13:54] ...trap/engines/engine.py.326: process: 0.32141 | load: 0.00020 [I 2022-10-09 03:13:58] ...trap/engines/engine.py.321: eval: epoch 29 | batch 160/510 [I 2022-10-09 03:13:58] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-09 03:13:58] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00021 [I 2022-10-09 03:14:01] ...trap/engines/engine.py.321: eval: epoch 29 | batch 170/510 [I 2022-10-09 03:14:01] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-09 03:14:01] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00021 [I 2022-10-09 03:14:04] ...trap/engines/engine.py.321: eval: epoch 29 | batch 180/510 [I 2022-10-09 03:14:04] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-09 03:14:04] ...trap/engines/engine.py.326: process: 0.31874 | load: 0.00019 [I 2022-10-09 03:14:07] ...trap/engines/engine.py.321: eval: epoch 29 | batch 190/510 [I 2022-10-09 03:14:07] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-09 03:14:07] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00019 [I 2022-10-09 03:14:10] ...trap/engines/engine.py.321: eval: epoch 29 | batch 200/510 [I 2022-10-09 03:14:10] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:38 [I 2022-10-09 03:14:10] ...trap/engines/engine.py.326: process: 0.31480 | load: 0.00021 [I 2022-10-09 03:14:13] ...trap/engines/engine.py.321: eval: epoch 29 | batch 210/510 [I 2022-10-09 03:14:13] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-09 03:14:13] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00020 [I 2022-10-09 03:14:17] ...trap/engines/engine.py.321: eval: epoch 29 | batch 220/510 [I 2022-10-09 03:14:17] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:31 [I 2022-10-09 03:14:17] ...trap/engines/engine.py.326: process: 0.31561 | load: 0.00021 [I 2022-10-09 03:14:20] ...trap/engines/engine.py.321: eval: epoch 29 | batch 230/510 [I 2022-10-09 03:14:20] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-09 03:14:20] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00020 [I 2022-10-09 03:14:23] ...trap/engines/engine.py.321: eval: epoch 29 | batch 240/510 [I 2022-10-09 03:14:23] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-09 03:14:23] ...trap/engines/engine.py.326: process: 0.32285 | load: 0.00020 [I 2022-10-09 03:14:26] ...trap/engines/engine.py.321: eval: epoch 29 | batch 250/510 [I 2022-10-09 03:14:26] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-09 03:14:26] ...trap/engines/engine.py.326: process: 0.31012 | load: 0.00020 [I 2022-10-09 03:14:29] ...trap/engines/engine.py.321: eval: epoch 29 | batch 260/510 [I 2022-10-09 03:14:29] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:19 [I 2022-10-09 03:14:29] ...trap/engines/engine.py.326: process: 0.32068 | load: 0.00021 [I 2022-10-09 03:14:33] ...trap/engines/engine.py.321: eval: epoch 29 | batch 270/510 [I 2022-10-09 03:14:33] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 03:14:33] ...trap/engines/engine.py.326: process: 0.32504 | load: 0.00021 [I 2022-10-09 03:14:36] ...trap/engines/engine.py.321: eval: epoch 29 | batch 280/510 [I 2022-10-09 03:14:36] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 03:14:36] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00020 [I 2022-10-09 03:14:39] ...trap/engines/engine.py.321: eval: epoch 29 | batch 290/510 [I 2022-10-09 03:14:39] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-09 03:14:39] ...trap/engines/engine.py.326: process: 0.31663 | load: 0.00021 [I 2022-10-09 03:14:42] ...trap/engines/engine.py.321: eval: epoch 29 | batch 300/510 [I 2022-10-09 03:14:42] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-09 03:14:42] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00021 [I 2022-10-09 03:14:45] ...trap/engines/engine.py.321: eval: epoch 29 | batch 310/510 [I 2022-10-09 03:14:45] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-09 03:14:45] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00020 [I 2022-10-09 03:14:49] ...trap/engines/engine.py.321: eval: epoch 29 | batch 320/510 [I 2022-10-09 03:14:49] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-09 03:14:49] ...trap/engines/engine.py.326: process: 0.32046 | load: 0.00020 [I 2022-10-09 03:14:52] ...trap/engines/engine.py.321: eval: epoch 29 | batch 330/510 [I 2022-10-09 03:14:52] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 03:14:52] ...trap/engines/engine.py.326: process: 0.31790 | load: 0.00020 [I 2022-10-09 03:14:55] ...trap/engines/engine.py.321: eval: epoch 29 | batch 340/510 [I 2022-10-09 03:14:55] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 03:14:55] ...trap/engines/engine.py.326: process: 0.32367 | load: 0.00021 [I 2022-10-09 03:14:58] ...trap/engines/engine.py.321: eval: epoch 29 | batch 350/510 [I 2022-10-09 03:14:58] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-09 03:14:58] ...trap/engines/engine.py.326: process: 0.32215 | load: 0.00020 [I 2022-10-09 03:15:01] ...trap/engines/engine.py.321: eval: epoch 29 | batch 360/510 [I 2022-10-09 03:15:01] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-09 03:15:01] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00020 [I 2022-10-09 03:15:05] ...trap/engines/engine.py.321: eval: epoch 29 | batch 370/510 [I 2022-10-09 03:15:05] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 03:15:05] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00020 [I 2022-10-09 03:15:08] ...trap/engines/engine.py.321: eval: epoch 29 | batch 380/510 [I 2022-10-09 03:15:08] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 03:15:08] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00020 [I 2022-10-09 03:15:11] ...trap/engines/engine.py.321: eval: epoch 29 | batch 390/510 [I 2022-10-09 03:15:11] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-09 03:15:11] ...trap/engines/engine.py.326: process: 0.31227 | load: 0.00021 [I 2022-10-09 03:15:14] ...trap/engines/engine.py.321: eval: epoch 29 | batch 400/510 [I 2022-10-09 03:15:14] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-09 03:15:14] ...trap/engines/engine.py.326: process: 0.31562 | load: 0.00018 [I 2022-10-09 03:15:17] ...trap/engines/engine.py.321: eval: epoch 29 | batch 410/510 [I 2022-10-09 03:15:17] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-09 03:15:17] ...trap/engines/engine.py.326: process: 0.31611 | load: 0.00021 [I 2022-10-09 03:15:21] ...trap/engines/engine.py.321: eval: epoch 29 | batch 420/510 [I 2022-10-09 03:15:21] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 03:15:21] ...trap/engines/engine.py.326: process: 0.31675 | load: 0.00018 [I 2022-10-09 03:15:24] ...trap/engines/engine.py.321: eval: epoch 29 | batch 430/510 [I 2022-10-09 03:15:24] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 03:15:24] ...trap/engines/engine.py.326: process: 0.31868 | load: 0.00020 [I 2022-10-09 03:15:27] ...trap/engines/engine.py.321: eval: epoch 29 | batch 440/510 [I 2022-10-09 03:15:27] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 03:15:27] ...trap/engines/engine.py.326: process: 0.31531 | load: 0.00020 [I 2022-10-09 03:15:30] ...trap/engines/engine.py.321: eval: epoch 29 | batch 450/510 [I 2022-10-09 03:15:30] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 03:15:30] ...trap/engines/engine.py.326: process: 0.32688 | load: 0.00021 [I 2022-10-09 03:15:33] ...trap/engines/engine.py.321: eval: epoch 29 | batch 460/510 [I 2022-10-09 03:15:33] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-09 03:15:33] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00020 [I 2022-10-09 03:15:37] ...trap/engines/engine.py.321: eval: epoch 29 | batch 470/510 [I 2022-10-09 03:15:37] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 03:15:37] ...trap/engines/engine.py.326: process: 0.32394 | load: 0.00020 [I 2022-10-09 03:15:40] ...trap/engines/engine.py.321: eval: epoch 29 | batch 480/510 [I 2022-10-09 03:15:40] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 03:15:40] ...trap/engines/engine.py.326: process: 0.30902 | load: 0.00019 [I 2022-10-09 03:15:43] ...trap/engines/engine.py.321: eval: epoch 29 | batch 490/510 [I 2022-10-09 03:15:43] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 03:15:43] ...trap/engines/engine.py.326: process: 0.31972 | load: 0.00021 [I 2022-10-09 03:15:46] ...trap/engines/engine.py.321: eval: epoch 29 | batch 500/510 [I 2022-10-09 03:15:46] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 03:15:46] ...trap/engines/engine.py.326: process: 0.31500 | load: 0.00022 [I 2022-10-09 03:15:49] ...trap/engines/engine.py.321: eval: epoch 29 | batch 510/510 [I 2022-10-09 03:15:49] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-09 03:15:49] ...trap/engines/engine.py.326: process: 0.31341 | load: 0.00016 [S 2022-10-09 03:15:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 29 [I 2022-10-09 03:15:50] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.690000 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.892300 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929500 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.676600 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.891500 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.927900 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009187 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006246 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006652 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012159 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009821 [S 2022-10-09 03:16:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007436 [I 2022-10-09 03:16:30] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 03:16:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 03:16:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 03:16:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 03:16:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 03:16:31] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 03:16:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 03:16:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 03:16:40] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 03:16:40] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 03:16:43] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 03:16:49] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 03:16:49] ...trap/engines/engine.py.163: Training model on trainset for epoch 30 [I 2022-10-09 03:16:49] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 03:16:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 0/2304 [I 2022-10-09 03:16:52] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:57 [I 2022-10-09 03:16:52] ...trap/engines/engine.py.229: process: 3.01985 | load: 2.11605 [I 2022-10-09 03:16:52] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-09 03:16:52] ...optimizers/trijoint.py.151: total_norm: 18.103415 [I 2022-10-09 03:17:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 10/2304 [I 2022-10-09 03:17:00] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:03 [I 2022-10-09 03:17:00] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00568 [I 2022-10-09 03:17:00] ...trap/engines/engine.py.230: loss: 0.21581 [I 2022-10-09 03:17:00] ...optimizers/trijoint.py.151: total_norm: 30.632179 [I 2022-10-09 03:17:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 20/2304 [I 2022-10-09 03:17:08] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:51 [I 2022-10-09 03:17:08] ...trap/engines/engine.py.229: process: 0.83914 | load: 0.00477 [I 2022-10-09 03:17:08] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-09 03:17:08] ...optimizers/trijoint.py.151: total_norm: 19.893835 [I 2022-10-09 03:17:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 30/2304 [I 2022-10-09 03:17:17] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:02 [I 2022-10-09 03:17:17] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00581 [I 2022-10-09 03:17:17] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-09 03:17:17] ...optimizers/trijoint.py.151: total_norm: 8.857090 [I 2022-10-09 03:17:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 40/2304 [I 2022-10-09 03:17:25] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:45 [I 2022-10-09 03:17:25] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00365 [I 2022-10-09 03:17:25] ...trap/engines/engine.py.230: loss: 0.18852 [I 2022-10-09 03:17:25] ...optimizers/trijoint.py.151: total_norm: 20.396652 [I 2022-10-09 03:17:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 50/2304 [I 2022-10-09 03:17:34] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:34 [I 2022-10-09 03:17:34] ...trap/engines/engine.py.229: process: 0.83668 | load: 0.00356 [I 2022-10-09 03:17:34] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-09 03:17:34] ...optimizers/trijoint.py.151: total_norm: 22.449100 [I 2022-10-09 03:17:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 60/2304 [I 2022-10-09 03:17:42] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:32 [I 2022-10-09 03:17:42] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00331 [I 2022-10-09 03:17:42] ...trap/engines/engine.py.230: loss: 0.16595 [I 2022-10-09 03:17:42] ...optimizers/trijoint.py.151: total_norm: 16.998783 [I 2022-10-09 03:17:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 70/2304 [I 2022-10-09 03:17:51] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:22 [I 2022-10-09 03:17:51] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00353 [I 2022-10-09 03:17:51] ...trap/engines/engine.py.230: loss: 0.18625 [I 2022-10-09 03:17:51] ...optimizers/trijoint.py.151: total_norm: 39.557758 [I 2022-10-09 03:17:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 80/2304 [I 2022-10-09 03:17:59] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:07 [I 2022-10-09 03:17:59] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00343 [I 2022-10-09 03:17:59] ...trap/engines/engine.py.230: loss: 0.19779 [I 2022-10-09 03:17:59] ...optimizers/trijoint.py.151: total_norm: 18.964750 [I 2022-10-09 03:18:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 90/2304 [I 2022-10-09 03:18:07] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:55 [I 2022-10-09 03:18:07] ...trap/engines/engine.py.229: process: 0.83636 | load: 0.00328 [I 2022-10-09 03:18:07] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-09 03:18:07] ...optimizers/trijoint.py.151: total_norm: 17.952864 [I 2022-10-09 03:18:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 100/2304 [I 2022-10-09 03:18:16] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:54 [I 2022-10-09 03:18:16] ...trap/engines/engine.py.229: process: 0.83350 | load: 0.00307 [I 2022-10-09 03:18:16] ...trap/engines/engine.py.230: loss: 0.15939 [I 2022-10-09 03:18:16] ...optimizers/trijoint.py.151: total_norm: 13.366300 [I 2022-10-09 03:18:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 110/2304 [I 2022-10-09 03:18:24] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:37 [I 2022-10-09 03:18:24] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00333 [I 2022-10-09 03:18:24] ...trap/engines/engine.py.230: loss: 0.22877 [I 2022-10-09 03:18:24] ...optimizers/trijoint.py.151: total_norm: 15.535554 [I 2022-10-09 03:18:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 120/2304 [I 2022-10-09 03:18:33] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:27 [I 2022-10-09 03:18:33] ...trap/engines/engine.py.229: process: 0.83143 | load: 0.00316 [I 2022-10-09 03:18:33] ...trap/engines/engine.py.230: loss: 0.15752 [I 2022-10-09 03:18:33] ...optimizers/trijoint.py.151: total_norm: 17.940962 [I 2022-10-09 03:18:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 130/2304 [I 2022-10-09 03:18:41] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:24 [I 2022-10-09 03:18:41] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00354 [I 2022-10-09 03:18:41] ...trap/engines/engine.py.230: loss: 0.16655 [I 2022-10-09 03:18:41] ...optimizers/trijoint.py.151: total_norm: 20.290560 [I 2022-10-09 03:18:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 140/2304 [I 2022-10-09 03:18:49] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:18 [I 2022-10-09 03:18:49] ...trap/engines/engine.py.229: process: 0.82298 | load: 0.00326 [I 2022-10-09 03:18:49] ...trap/engines/engine.py.230: loss: 0.24153 [I 2022-10-09 03:18:49] ...optimizers/trijoint.py.151: total_norm: 14.985534 [I 2022-10-09 03:18:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 150/2304 [I 2022-10-09 03:18:58] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:14 [I 2022-10-09 03:18:58] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00309 [I 2022-10-09 03:18:58] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-09 03:18:58] ...optimizers/trijoint.py.151: total_norm: 8.484219 [I 2022-10-09 03:19:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 160/2304 [I 2022-10-09 03:19:06] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:13 [I 2022-10-09 03:19:06] ...trap/engines/engine.py.229: process: 0.84811 | load: 0.00327 [I 2022-10-09 03:19:06] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-09 03:19:06] ...optimizers/trijoint.py.151: total_norm: 11.510142 [I 2022-10-09 03:19:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 170/2304 [I 2022-10-09 03:19:15] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:04 [I 2022-10-09 03:19:15] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00306 [I 2022-10-09 03:19:15] ...trap/engines/engine.py.230: loss: 0.18694 [I 2022-10-09 03:19:15] ...optimizers/trijoint.py.151: total_norm: 24.268381 [I 2022-10-09 03:19:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 180/2304 [I 2022-10-09 03:19:23] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:54 [I 2022-10-09 03:19:23] ...trap/engines/engine.py.229: process: 0.83971 | load: 0.00302 [I 2022-10-09 03:19:23] ...trap/engines/engine.py.230: loss: 0.20042 [I 2022-10-09 03:19:23] ...optimizers/trijoint.py.151: total_norm: 17.814966 [I 2022-10-09 03:19:32] ...trap/engines/engine.py.224: train: epoch 30 | batch 190/2304 [I 2022-10-09 03:19:32] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:41 [I 2022-10-09 03:19:32] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00334 [I 2022-10-09 03:19:32] ...trap/engines/engine.py.230: loss: 0.24943 [I 2022-10-09 03:19:32] ...optimizers/trijoint.py.151: total_norm: 38.487606 [I 2022-10-09 03:19:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 200/2304 [I 2022-10-09 03:19:40] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:29 [I 2022-10-09 03:19:40] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00310 [I 2022-10-09 03:19:40] ...trap/engines/engine.py.230: loss: 0.18894 [I 2022-10-09 03:19:40] ...optimizers/trijoint.py.151: total_norm: 20.759380 [I 2022-10-09 03:19:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 210/2304 [I 2022-10-09 03:19:49] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:23 [I 2022-10-09 03:19:49] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00322 [I 2022-10-09 03:19:49] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-09 03:19:49] ...optimizers/trijoint.py.151: total_norm: 10.164754 [I 2022-10-09 03:19:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 220/2304 [I 2022-10-09 03:19:57] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:06 [I 2022-10-09 03:19:57] ...trap/engines/engine.py.229: process: 0.84018 | load: 0.00324 [I 2022-10-09 03:19:57] ...trap/engines/engine.py.230: loss: 0.19780 [I 2022-10-09 03:19:57] ...optimizers/trijoint.py.151: total_norm: 16.487755 [I 2022-10-09 03:20:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 230/2304 [I 2022-10-09 03:20:05] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:00 [I 2022-10-09 03:20:05] ...trap/engines/engine.py.229: process: 0.83297 | load: 0.00300 [I 2022-10-09 03:20:05] ...trap/engines/engine.py.230: loss: 0.15214 [I 2022-10-09 03:20:05] ...optimizers/trijoint.py.151: total_norm: 10.841931 [I 2022-10-09 03:20:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 240/2304 [I 2022-10-09 03:20:14] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:53 [I 2022-10-09 03:20:14] ...trap/engines/engine.py.229: process: 0.84609 | load: 0.00320 [I 2022-10-09 03:20:14] ...trap/engines/engine.py.230: loss: 0.17110 [I 2022-10-09 03:20:14] ...optimizers/trijoint.py.151: total_norm: 17.702538 [I 2022-10-09 03:20:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 250/2304 [I 2022-10-09 03:20:22] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:43 [I 2022-10-09 03:20:22] ...trap/engines/engine.py.229: process: 0.82978 | load: 0.00301 [I 2022-10-09 03:20:22] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-09 03:20:22] ...optimizers/trijoint.py.151: total_norm: 12.970056 [I 2022-10-09 03:20:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 260/2304 [I 2022-10-09 03:20:31] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:39 [I 2022-10-09 03:20:31] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00323 [I 2022-10-09 03:20:31] ...trap/engines/engine.py.230: loss: 0.26393 [I 2022-10-09 03:20:31] ...optimizers/trijoint.py.151: total_norm: 25.196463 [I 2022-10-09 03:20:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 270/2304 [I 2022-10-09 03:20:39] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:26 [I 2022-10-09 03:20:39] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00302 [I 2022-10-09 03:20:39] ...trap/engines/engine.py.230: loss: 0.16689 [I 2022-10-09 03:20:39] ...optimizers/trijoint.py.151: total_norm: 15.706930 [I 2022-10-09 03:20:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 280/2304 [I 2022-10-09 03:20:47] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:16 [I 2022-10-09 03:20:47] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00309 [I 2022-10-09 03:20:47] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-09 03:20:47] ...optimizers/trijoint.py.151: total_norm: 18.635380 [I 2022-10-09 03:20:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 290/2304 [I 2022-10-09 03:20:56] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:19 [I 2022-10-09 03:20:56] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00310 [I 2022-10-09 03:20:56] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-09 03:20:56] ...optimizers/trijoint.py.151: total_norm: 18.271664 [I 2022-10-09 03:21:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 300/2304 [I 2022-10-09 03:21:04] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:58 [I 2022-10-09 03:21:04] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00300 [I 2022-10-09 03:21:04] ...trap/engines/engine.py.230: loss: 0.17456 [I 2022-10-09 03:21:04] ...optimizers/trijoint.py.151: total_norm: 25.788225 [I 2022-10-09 03:21:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 310/2304 [I 2022-10-09 03:21:13] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:54 [I 2022-10-09 03:21:13] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00300 [I 2022-10-09 03:21:13] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-09 03:21:13] ...optimizers/trijoint.py.151: total_norm: 13.649219 [I 2022-10-09 03:21:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 320/2304 [I 2022-10-09 03:21:21] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:50 [I 2022-10-09 03:21:21] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00298 [I 2022-10-09 03:21:21] ...trap/engines/engine.py.230: loss: 0.21546 [I 2022-10-09 03:21:21] ...optimizers/trijoint.py.151: total_norm: 23.848824 [I 2022-10-09 03:21:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 330/2304 [I 2022-10-09 03:21:30] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:40 [I 2022-10-09 03:21:30] ...trap/engines/engine.py.229: process: 0.82990 | load: 0.00312 [I 2022-10-09 03:21:30] ...trap/engines/engine.py.230: loss: 0.13737 [I 2022-10-09 03:21:30] ...optimizers/trijoint.py.151: total_norm: 12.200131 [I 2022-10-09 03:21:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 340/2304 [I 2022-10-09 03:21:38] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:26 [I 2022-10-09 03:21:38] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00298 [I 2022-10-09 03:21:38] ...trap/engines/engine.py.230: loss: 0.16665 [I 2022-10-09 03:21:38] ...optimizers/trijoint.py.151: total_norm: 16.904436 [I 2022-10-09 03:21:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 350/2304 [I 2022-10-09 03:21:46] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:27 [I 2022-10-09 03:21:46] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00299 [I 2022-10-09 03:21:46] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-09 03:21:46] ...optimizers/trijoint.py.151: total_norm: 12.730336 [I 2022-10-09 03:21:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 360/2304 [I 2022-10-09 03:21:55] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:12 [I 2022-10-09 03:21:55] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00298 [I 2022-10-09 03:21:55] ...trap/engines/engine.py.230: loss: 0.19351 [I 2022-10-09 03:21:55] ...optimizers/trijoint.py.151: total_norm: 18.336271 [I 2022-10-09 03:22:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 370/2304 [I 2022-10-09 03:22:03] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:26:54 [I 2022-10-09 03:22:03] ...trap/engines/engine.py.229: process: 0.82427 | load: 0.00297 [I 2022-10-09 03:22:03] ...trap/engines/engine.py.230: loss: 0.10652 [I 2022-10-09 03:22:03] ...optimizers/trijoint.py.151: total_norm: 14.104289 [I 2022-10-09 03:22:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 380/2304 [I 2022-10-09 03:22:12] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:49 [I 2022-10-09 03:22:12] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00299 [I 2022-10-09 03:22:12] ...trap/engines/engine.py.230: loss: 0.16664 [I 2022-10-09 03:22:12] ...optimizers/trijoint.py.151: total_norm: 79.378883 [I 2022-10-09 03:22:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 390/2304 [I 2022-10-09 03:22:20] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:50 [I 2022-10-09 03:22:20] ...trap/engines/engine.py.229: process: 0.84953 | load: 0.00301 [I 2022-10-09 03:22:20] ...trap/engines/engine.py.230: loss: 0.15816 [I 2022-10-09 03:22:20] ...optimizers/trijoint.py.151: total_norm: 16.346254 [I 2022-10-09 03:22:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 400/2304 [I 2022-10-09 03:22:28] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:41 [I 2022-10-09 03:22:28] ...trap/engines/engine.py.229: process: 0.83399 | load: 0.00296 [I 2022-10-09 03:22:28] ...trap/engines/engine.py.230: loss: 0.16813 [I 2022-10-09 03:22:28] ...optimizers/trijoint.py.151: total_norm: 21.297884 [I 2022-10-09 03:22:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 410/2304 [I 2022-10-09 03:22:37] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:46 [I 2022-10-09 03:22:37] ...trap/engines/engine.py.229: process: 0.85625 | load: 0.00302 [I 2022-10-09 03:22:37] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-09 03:22:37] ...optimizers/trijoint.py.151: total_norm: 16.227903 [I 2022-10-09 03:22:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 420/2304 [I 2022-10-09 03:22:45] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:38 [I 2022-10-09 03:22:45] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00299 [I 2022-10-09 03:22:45] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-09 03:22:45] ...optimizers/trijoint.py.151: total_norm: 31.538652 [I 2022-10-09 03:22:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 430/2304 [I 2022-10-09 03:22:54] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:32 [I 2022-10-09 03:22:54] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00302 [I 2022-10-09 03:22:54] ...trap/engines/engine.py.230: loss: 0.20539 [I 2022-10-09 03:22:54] ...optimizers/trijoint.py.151: total_norm: 22.813066 [I 2022-10-09 03:23:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 440/2304 [I 2022-10-09 03:23:02] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:16 [I 2022-10-09 03:23:02] ...trap/engines/engine.py.229: process: 0.83823 | load: 0.00310 [I 2022-10-09 03:23:02] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-09 03:23:02] ...optimizers/trijoint.py.151: total_norm: 12.841167 [I 2022-10-09 03:23:11] ...trap/engines/engine.py.224: train: epoch 30 | batch 450/2304 [I 2022-10-09 03:23:11] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:14 [I 2022-10-09 03:23:11] ...trap/engines/engine.py.229: process: 0.87518 | load: 0.00306 [I 2022-10-09 03:23:11] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-09 03:23:11] ...optimizers/trijoint.py.151: total_norm: 12.834552 [I 2022-10-09 03:23:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 460/2304 [I 2022-10-09 03:23:19] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:08 [I 2022-10-09 03:23:19] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00301 [I 2022-10-09 03:23:19] ...trap/engines/engine.py.230: loss: 0.14714 [I 2022-10-09 03:23:19] ...optimizers/trijoint.py.151: total_norm: 15.022015 [I 2022-10-09 03:23:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 470/2304 [I 2022-10-09 03:23:28] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:53 [I 2022-10-09 03:23:28] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00298 [I 2022-10-09 03:23:28] ...trap/engines/engine.py.230: loss: 0.11951 [I 2022-10-09 03:23:28] ...optimizers/trijoint.py.151: total_norm: 26.155727 [I 2022-10-09 03:23:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 480/2304 [I 2022-10-09 03:23:36] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:23 [I 2022-10-09 03:23:36] ...trap/engines/engine.py.229: process: 0.83406 | load: 0.00300 [I 2022-10-09 03:23:36] ...trap/engines/engine.py.230: loss: 0.16781 [I 2022-10-09 03:23:36] ...optimizers/trijoint.py.151: total_norm: 9.656816 [I 2022-10-09 03:23:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 490/2304 [I 2022-10-09 03:23:45] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:18 [I 2022-10-09 03:23:45] ...trap/engines/engine.py.229: process: 0.83664 | load: 0.00303 [I 2022-10-09 03:23:45] ...trap/engines/engine.py.230: loss: 0.11948 [I 2022-10-09 03:23:45] ...optimizers/trijoint.py.151: total_norm: 20.547895 [I 2022-10-09 03:23:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 500/2304 [I 2022-10-09 03:23:53] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:15 [I 2022-10-09 03:23:53] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00300 [I 2022-10-09 03:23:53] ...trap/engines/engine.py.230: loss: 0.16148 [I 2022-10-09 03:23:53] ...optimizers/trijoint.py.151: total_norm: 19.361526 [I 2022-10-09 03:24:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 510/2304 [I 2022-10-09 03:24:02] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:03 [I 2022-10-09 03:24:02] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00299 [I 2022-10-09 03:24:02] ...trap/engines/engine.py.230: loss: 0.16334 [I 2022-10-09 03:24:02] ...optimizers/trijoint.py.151: total_norm: 21.833149 [I 2022-10-09 03:24:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 520/2304 [I 2022-10-09 03:24:10] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:24:55 [I 2022-10-09 03:24:10] ...trap/engines/engine.py.229: process: 0.82897 | load: 0.00303 [I 2022-10-09 03:24:10] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-09 03:24:10] ...optimizers/trijoint.py.151: total_norm: 26.311096 [I 2022-10-09 03:24:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 530/2304 [I 2022-10-09 03:24:18] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:47 [I 2022-10-09 03:24:18] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00308 [I 2022-10-09 03:24:18] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-09 03:24:18] ...optimizers/trijoint.py.151: total_norm: 19.621809 [I 2022-10-09 03:24:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 540/2304 [I 2022-10-09 03:24:27] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:47 [I 2022-10-09 03:24:27] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00303 [I 2022-10-09 03:24:27] ...trap/engines/engine.py.230: loss: 0.14124 [I 2022-10-09 03:24:27] ...optimizers/trijoint.py.151: total_norm: 30.505646 [I 2022-10-09 03:24:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 550/2304 [I 2022-10-09 03:24:35] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:26 [I 2022-10-09 03:24:35] ...trap/engines/engine.py.229: process: 0.83470 | load: 0.00297 [I 2022-10-09 03:24:35] ...trap/engines/engine.py.230: loss: 0.16816 [I 2022-10-09 03:24:35] ...optimizers/trijoint.py.151: total_norm: 20.323084 [I 2022-10-09 03:24:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 560/2304 [I 2022-10-09 03:24:44] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:29 [I 2022-10-09 03:24:44] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00311 [I 2022-10-09 03:24:44] ...trap/engines/engine.py.230: loss: 0.18253 [I 2022-10-09 03:24:44] ...optimizers/trijoint.py.151: total_norm: 19.407679 [I 2022-10-09 03:24:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 570/2304 [I 2022-10-09 03:24:52] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:20 [I 2022-10-09 03:24:52] ...trap/engines/engine.py.229: process: 0.84563 | load: 0.00304 [I 2022-10-09 03:24:52] ...trap/engines/engine.py.230: loss: 0.16384 [I 2022-10-09 03:24:52] ...optimizers/trijoint.py.151: total_norm: 16.262140 [I 2022-10-09 03:25:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 580/2304 [I 2022-10-09 03:25:00] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:08 [I 2022-10-09 03:25:00] ...trap/engines/engine.py.229: process: 0.83890 | load: 0.00302 [I 2022-10-09 03:25:00] ...trap/engines/engine.py.230: loss: 0.17109 [I 2022-10-09 03:25:00] ...optimizers/trijoint.py.151: total_norm: 15.579234 [I 2022-10-09 03:25:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 590/2304 [I 2022-10-09 03:25:09] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:08 [I 2022-10-09 03:25:09] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00314 [I 2022-10-09 03:25:09] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-09 03:25:09] ...optimizers/trijoint.py.151: total_norm: 17.184025 [I 2022-10-09 03:25:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 600/2304 [I 2022-10-09 03:25:17] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:56 [I 2022-10-09 03:25:17] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00301 [I 2022-10-09 03:25:17] ...trap/engines/engine.py.230: loss: 0.21265 [I 2022-10-09 03:25:17] ...optimizers/trijoint.py.151: total_norm: 20.312988 [I 2022-10-09 03:25:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 610/2304 [I 2022-10-09 03:25:26] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:48 [I 2022-10-09 03:25:26] ...trap/engines/engine.py.229: process: 0.83201 | load: 0.00300 [I 2022-10-09 03:25:26] ...trap/engines/engine.py.230: loss: 0.17030 [I 2022-10-09 03:25:26] ...optimizers/trijoint.py.151: total_norm: 16.954187 [I 2022-10-09 03:25:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 620/2304 [I 2022-10-09 03:25:34] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:35 [I 2022-10-09 03:25:34] ...trap/engines/engine.py.229: process: 0.85482 | load: 0.00302 [I 2022-10-09 03:25:34] ...trap/engines/engine.py.230: loss: 0.12783 [I 2022-10-09 03:25:34] ...optimizers/trijoint.py.151: total_norm: 14.275494 [I 2022-10-09 03:25:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 630/2304 [I 2022-10-09 03:25:43] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:24 [I 2022-10-09 03:25:43] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00303 [I 2022-10-09 03:25:43] ...trap/engines/engine.py.230: loss: 0.17912 [I 2022-10-09 03:25:43] ...optimizers/trijoint.py.151: total_norm: 23.446581 [I 2022-10-09 03:25:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 640/2304 [I 2022-10-09 03:25:51] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:13 [I 2022-10-09 03:25:51] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00303 [I 2022-10-09 03:25:51] ...trap/engines/engine.py.230: loss: 0.20625 [I 2022-10-09 03:25:51] ...optimizers/trijoint.py.151: total_norm: 20.742163 [I 2022-10-09 03:25:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 650/2304 [I 2022-10-09 03:25:59] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:09 [I 2022-10-09 03:25:59] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00301 [I 2022-10-09 03:25:59] ...trap/engines/engine.py.230: loss: 0.18916 [I 2022-10-09 03:25:59] ...optimizers/trijoint.py.151: total_norm: 43.023560 [I 2022-10-09 03:26:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 660/2304 [I 2022-10-09 03:26:08] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:00 [I 2022-10-09 03:26:08] ...trap/engines/engine.py.229: process: 0.83525 | load: 0.00298 [I 2022-10-09 03:26:08] ...trap/engines/engine.py.230: loss: 0.22971 [I 2022-10-09 03:26:08] ...optimizers/trijoint.py.151: total_norm: 10.698618 [I 2022-10-09 03:26:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 670/2304 [I 2022-10-09 03:26:16] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:52 [I 2022-10-09 03:26:16] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00303 [I 2022-10-09 03:26:16] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-09 03:26:16] ...optimizers/trijoint.py.151: total_norm: 14.023890 [I 2022-10-09 03:26:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 680/2304 [I 2022-10-09 03:26:25] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:45 [I 2022-10-09 03:26:25] ...trap/engines/engine.py.229: process: 0.85327 | load: 0.00300 [I 2022-10-09 03:26:25] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-09 03:26:25] ...optimizers/trijoint.py.151: total_norm: 21.925623 [I 2022-10-09 03:26:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 690/2304 [I 2022-10-09 03:26:33] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:31 [I 2022-10-09 03:26:33] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00300 [I 2022-10-09 03:26:33] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-09 03:26:33] ...optimizers/trijoint.py.151: total_norm: 10.497602 [I 2022-10-09 03:26:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 700/2304 [I 2022-10-09 03:26:41] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:23 [I 2022-10-09 03:26:41] ...trap/engines/engine.py.229: process: 0.83140 | load: 0.00298 [I 2022-10-09 03:26:41] ...trap/engines/engine.py.230: loss: 0.11041 [I 2022-10-09 03:26:41] ...optimizers/trijoint.py.151: total_norm: 11.147632 [I 2022-10-09 03:26:50] ...trap/engines/engine.py.224: train: epoch 30 | batch 710/2304 [I 2022-10-09 03:26:50] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:19 [I 2022-10-09 03:26:50] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00299 [I 2022-10-09 03:26:50] ...trap/engines/engine.py.230: loss: 0.17632 [I 2022-10-09 03:26:50] ...optimizers/trijoint.py.151: total_norm: 19.252563 [I 2022-10-09 03:26:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 720/2304 [I 2022-10-09 03:26:58] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:08 [I 2022-10-09 03:26:58] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00299 [I 2022-10-09 03:26:58] ...trap/engines/engine.py.230: loss: 0.19607 [I 2022-10-09 03:26:58] ...optimizers/trijoint.py.151: total_norm: 25.164738 [I 2022-10-09 03:27:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 730/2304 [I 2022-10-09 03:27:07] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:21:57 [I 2022-10-09 03:27:07] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00299 [I 2022-10-09 03:27:07] ...trap/engines/engine.py.230: loss: 0.22226 [I 2022-10-09 03:27:07] ...optimizers/trijoint.py.151: total_norm: 21.239374 [I 2022-10-09 03:27:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 740/2304 [I 2022-10-09 03:27:15] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:55 [I 2022-10-09 03:27:15] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00301 [I 2022-10-09 03:27:15] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-09 03:27:15] ...optimizers/trijoint.py.151: total_norm: 20.299320 [I 2022-10-09 03:27:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 750/2304 [I 2022-10-09 03:27:23] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:36 [I 2022-10-09 03:27:23] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00298 [I 2022-10-09 03:27:23] ...trap/engines/engine.py.230: loss: 0.11395 [I 2022-10-09 03:27:23] ...optimizers/trijoint.py.151: total_norm: 12.030396 [I 2022-10-09 03:27:32] ...trap/engines/engine.py.224: train: epoch 30 | batch 760/2304 [I 2022-10-09 03:27:32] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:33 [I 2022-10-09 03:27:32] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00303 [I 2022-10-09 03:27:32] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-09 03:27:32] ...optimizers/trijoint.py.151: total_norm: 15.411261 [I 2022-10-09 03:27:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 770/2304 [I 2022-10-09 03:27:40] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:25 [I 2022-10-09 03:27:40] ...trap/engines/engine.py.229: process: 0.83569 | load: 0.00302 [I 2022-10-09 03:27:40] ...trap/engines/engine.py.230: loss: 0.09840 [I 2022-10-09 03:27:40] ...optimizers/trijoint.py.151: total_norm: 9.084569 [I 2022-10-09 03:27:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 780/2304 [I 2022-10-09 03:27:49] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:21 [I 2022-10-09 03:27:49] ...trap/engines/engine.py.229: process: 0.83201 | load: 0.00298 [I 2022-10-09 03:27:49] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-09 03:27:49] ...optimizers/trijoint.py.151: total_norm: 8.321914 [I 2022-10-09 03:27:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 790/2304 [I 2022-10-09 03:27:57] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:09 [I 2022-10-09 03:27:57] ...trap/engines/engine.py.229: process: 0.83933 | load: 0.00304 [I 2022-10-09 03:27:57] ...trap/engines/engine.py.230: loss: 0.18347 [I 2022-10-09 03:27:57] ...optimizers/trijoint.py.151: total_norm: 26.847485 [I 2022-10-09 03:28:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 800/2304 [I 2022-10-09 03:28:06] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:04 [I 2022-10-09 03:28:06] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00300 [I 2022-10-09 03:28:06] ...trap/engines/engine.py.230: loss: 0.12393 [I 2022-10-09 03:28:06] ...optimizers/trijoint.py.151: total_norm: 12.870863 [I 2022-10-09 03:28:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 810/2304 [I 2022-10-09 03:28:14] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:51 [I 2022-10-09 03:28:14] ...trap/engines/engine.py.229: process: 0.82985 | load: 0.00301 [I 2022-10-09 03:28:14] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-09 03:28:14] ...optimizers/trijoint.py.151: total_norm: 30.098213 [I 2022-10-09 03:28:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 820/2304 [I 2022-10-09 03:28:22] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:20:44 [I 2022-10-09 03:28:22] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00299 [I 2022-10-09 03:28:22] ...trap/engines/engine.py.230: loss: 0.14240 [I 2022-10-09 03:28:22] ...optimizers/trijoint.py.151: total_norm: 22.601011 [I 2022-10-09 03:28:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 830/2304 [I 2022-10-09 03:28:31] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:37 [I 2022-10-09 03:28:31] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00301 [I 2022-10-09 03:28:31] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-09 03:28:31] ...optimizers/trijoint.py.151: total_norm: 17.609314 [I 2022-10-09 03:28:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 840/2304 [I 2022-10-09 03:28:39] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:29 [I 2022-10-09 03:28:39] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00297 [I 2022-10-09 03:28:39] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-09 03:28:39] ...optimizers/trijoint.py.151: total_norm: 9.545485 [I 2022-10-09 03:28:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 850/2304 [I 2022-10-09 03:28:48] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:15 [I 2022-10-09 03:28:48] ...trap/engines/engine.py.229: process: 0.82924 | load: 0.00298 [I 2022-10-09 03:28:48] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-09 03:28:48] ...optimizers/trijoint.py.151: total_norm: 26.636791 [I 2022-10-09 03:28:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 860/2304 [I 2022-10-09 03:28:56] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:10 [I 2022-10-09 03:28:56] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00299 [I 2022-10-09 03:28:56] ...trap/engines/engine.py.230: loss: 0.20562 [I 2022-10-09 03:28:56] ...optimizers/trijoint.py.151: total_norm: 18.659483 [I 2022-10-09 03:29:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 870/2304 [I 2022-10-09 03:29:04] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:19:59 [I 2022-10-09 03:29:04] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00301 [I 2022-10-09 03:29:04] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-09 03:29:04] ...optimizers/trijoint.py.151: total_norm: 15.780633 [I 2022-10-09 03:29:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 880/2304 [I 2022-10-09 03:29:13] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:54 [I 2022-10-09 03:29:13] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00298 [I 2022-10-09 03:29:13] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-09 03:29:13] ...optimizers/trijoint.py.151: total_norm: 12.519696 [I 2022-10-09 03:29:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 890/2304 [I 2022-10-09 03:29:21] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:19:40 [I 2022-10-09 03:29:21] ...trap/engines/engine.py.229: process: 0.82811 | load: 0.00299 [I 2022-10-09 03:29:21] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-09 03:29:21] ...optimizers/trijoint.py.151: total_norm: 17.411707 [I 2022-10-09 03:29:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 900/2304 [I 2022-10-09 03:29:30] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:19:36 [I 2022-10-09 03:29:30] ...trap/engines/engine.py.229: process: 0.83487 | load: 0.00300 [I 2022-10-09 03:29:30] ...trap/engines/engine.py.230: loss: 0.18640 [I 2022-10-09 03:29:30] ...optimizers/trijoint.py.151: total_norm: 17.259830 [I 2022-10-09 03:29:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 910/2304 [I 2022-10-09 03:29:38] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:27 [I 2022-10-09 03:29:38] ...trap/engines/engine.py.229: process: 0.83241 | load: 0.00298 [I 2022-10-09 03:29:38] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-09 03:29:38] ...optimizers/trijoint.py.151: total_norm: 12.877304 [I 2022-10-09 03:29:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 920/2304 [I 2022-10-09 03:29:46] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:17 [I 2022-10-09 03:29:46] ...trap/engines/engine.py.229: process: 0.83235 | load: 0.00301 [I 2022-10-09 03:29:46] ...trap/engines/engine.py.230: loss: 0.26048 [I 2022-10-09 03:29:46] ...optimizers/trijoint.py.151: total_norm: 21.122665 [I 2022-10-09 03:29:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 930/2304 [I 2022-10-09 03:29:55] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:14 [I 2022-10-09 03:29:55] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00300 [I 2022-10-09 03:29:55] ...trap/engines/engine.py.230: loss: 0.15304 [I 2022-10-09 03:29:55] ...optimizers/trijoint.py.151: total_norm: 12.885138 [I 2022-10-09 03:30:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 940/2304 [I 2022-10-09 03:30:03] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:05 [I 2022-10-09 03:30:03] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00299 [I 2022-10-09 03:30:03] ...trap/engines/engine.py.230: loss: 0.14459 [I 2022-10-09 03:30:03] ...optimizers/trijoint.py.151: total_norm: 11.114773 [I 2022-10-09 03:30:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 950/2304 [I 2022-10-09 03:30:12] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:18:52 [I 2022-10-09 03:30:12] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00298 [I 2022-10-09 03:30:12] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-09 03:30:12] ...optimizers/trijoint.py.151: total_norm: 11.715298 [I 2022-10-09 03:30:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 960/2304 [I 2022-10-09 03:30:20] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:18:55 [I 2022-10-09 03:30:20] ...trap/engines/engine.py.229: process: 0.85857 | load: 0.00297 [I 2022-10-09 03:30:20] ...trap/engines/engine.py.230: loss: 0.18615 [I 2022-10-09 03:30:20] ...optimizers/trijoint.py.151: total_norm: 18.815784 [I 2022-10-09 03:30:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 970/2304 [I 2022-10-09 03:30:28] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:18:47 [I 2022-10-09 03:30:28] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00300 [I 2022-10-09 03:30:28] ...trap/engines/engine.py.230: loss: 0.11748 [I 2022-10-09 03:30:28] ...optimizers/trijoint.py.151: total_norm: 15.763323 [I 2022-10-09 03:30:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 980/2304 [I 2022-10-09 03:30:37] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:18:33 [I 2022-10-09 03:30:37] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00301 [I 2022-10-09 03:30:37] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-09 03:30:37] ...optimizers/trijoint.py.151: total_norm: 18.720263 [I 2022-10-09 03:30:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 990/2304 [I 2022-10-09 03:30:45] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:24 [I 2022-10-09 03:30:45] ...trap/engines/engine.py.229: process: 0.83684 | load: 0.00303 [I 2022-10-09 03:30:45] ...trap/engines/engine.py.230: loss: 0.21924 [I 2022-10-09 03:30:45] ...optimizers/trijoint.py.151: total_norm: 26.306995 [I 2022-10-09 03:30:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 1000/2304 [I 2022-10-09 03:30:54] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:13 [I 2022-10-09 03:30:54] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00303 [I 2022-10-09 03:30:54] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-09 03:30:54] ...optimizers/trijoint.py.151: total_norm: 12.685882 [I 2022-10-09 03:31:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 1010/2304 [I 2022-10-09 03:31:02] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:11 [I 2022-10-09 03:31:02] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00299 [I 2022-10-09 03:31:02] ...trap/engines/engine.py.230: loss: 0.19873 [I 2022-10-09 03:31:02] ...optimizers/trijoint.py.151: total_norm: 23.552786 [I 2022-10-09 03:31:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 1020/2304 [I 2022-10-09 03:31:10] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:17:59 [I 2022-10-09 03:31:10] ...trap/engines/engine.py.229: process: 0.83665 | load: 0.00299 [I 2022-10-09 03:31:10] ...trap/engines/engine.py.230: loss: 0.15436 [I 2022-10-09 03:31:10] ...optimizers/trijoint.py.151: total_norm: 12.741249 [I 2022-10-09 03:31:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 1030/2304 [I 2022-10-09 03:31:19] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:17:52 [I 2022-10-09 03:31:19] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00300 [I 2022-10-09 03:31:19] ...trap/engines/engine.py.230: loss: 0.23423 [I 2022-10-09 03:31:19] ...optimizers/trijoint.py.151: total_norm: 20.136429 [I 2022-10-09 03:31:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 1040/2304 [I 2022-10-09 03:31:27] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:17:38 [I 2022-10-09 03:31:27] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00300 [I 2022-10-09 03:31:27] ...trap/engines/engine.py.230: loss: 0.22980 [I 2022-10-09 03:31:27] ...optimizers/trijoint.py.151: total_norm: 14.516155 [I 2022-10-09 03:31:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 1050/2304 [I 2022-10-09 03:31:36] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:17:33 [I 2022-10-09 03:31:36] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00302 [I 2022-10-09 03:31:36] ...trap/engines/engine.py.230: loss: 0.10857 [I 2022-10-09 03:31:36] ...optimizers/trijoint.py.151: total_norm: 10.158101 [I 2022-10-09 03:31:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 1060/2304 [I 2022-10-09 03:31:44] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:17:25 [I 2022-10-09 03:31:44] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00297 [I 2022-10-09 03:31:44] ...trap/engines/engine.py.230: loss: 0.15654 [I 2022-10-09 03:31:44] ...optimizers/trijoint.py.151: total_norm: 18.435007 [I 2022-10-09 03:31:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 1070/2304 [I 2022-10-09 03:31:52] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:13 [I 2022-10-09 03:31:52] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00300 [I 2022-10-09 03:31:52] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-09 03:31:52] ...optimizers/trijoint.py.151: total_norm: 18.742313 [I 2022-10-09 03:32:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 1080/2304 [I 2022-10-09 03:32:01] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:06 [I 2022-10-09 03:32:01] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00298 [I 2022-10-09 03:32:01] ...trap/engines/engine.py.230: loss: 0.14318 [I 2022-10-09 03:32:01] ...optimizers/trijoint.py.151: total_norm: 12.212378 [I 2022-10-09 03:32:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 1090/2304 [I 2022-10-09 03:32:09] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:16:59 [I 2022-10-09 03:32:09] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00300 [I 2022-10-09 03:32:09] ...trap/engines/engine.py.230: loss: 0.15714 [I 2022-10-09 03:32:09] ...optimizers/trijoint.py.151: total_norm: 15.328588 [I 2022-10-09 03:32:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 1100/2304 [I 2022-10-09 03:32:18] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:16:48 [I 2022-10-09 03:32:18] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00299 [I 2022-10-09 03:32:18] ...trap/engines/engine.py.230: loss: 0.13176 [I 2022-10-09 03:32:18] ...optimizers/trijoint.py.151: total_norm: 16.070724 [I 2022-10-09 03:32:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 1110/2304 [I 2022-10-09 03:32:26] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:16:45 [I 2022-10-09 03:32:26] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00300 [I 2022-10-09 03:32:26] ...trap/engines/engine.py.230: loss: 0.24125 [I 2022-10-09 03:32:26] ...optimizers/trijoint.py.151: total_norm: 16.090185 [I 2022-10-09 03:32:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 1120/2304 [I 2022-10-09 03:32:35] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:16:39 [I 2022-10-09 03:32:35] ...trap/engines/engine.py.229: process: 0.85588 | load: 0.00300 [I 2022-10-09 03:32:35] ...trap/engines/engine.py.230: loss: 0.25837 [I 2022-10-09 03:32:35] ...optimizers/trijoint.py.151: total_norm: 31.999765 [I 2022-10-09 03:32:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 1130/2304 [I 2022-10-09 03:32:43] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:16:28 [I 2022-10-09 03:32:43] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00299 [I 2022-10-09 03:32:43] ...trap/engines/engine.py.230: loss: 0.16997 [I 2022-10-09 03:32:43] ...optimizers/trijoint.py.151: total_norm: 24.408428 [I 2022-10-09 03:32:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 1140/2304 [I 2022-10-09 03:32:51] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:13 [I 2022-10-09 03:32:51] ...trap/engines/engine.py.229: process: 0.83235 | load: 0.00298 [I 2022-10-09 03:32:51] ...trap/engines/engine.py.230: loss: 0.17093 [I 2022-10-09 03:32:51] ...optimizers/trijoint.py.151: total_norm: 14.546384 [I 2022-10-09 03:33:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 1150/2304 [I 2022-10-09 03:33:00] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:04 [I 2022-10-09 03:33:00] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00299 [I 2022-10-09 03:33:00] ...trap/engines/engine.py.230: loss: 0.10917 [I 2022-10-09 03:33:00] ...optimizers/trijoint.py.151: total_norm: 17.893009 [I 2022-10-09 03:33:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 1160/2304 [I 2022-10-09 03:33:08] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:00 [I 2022-10-09 03:33:08] ...trap/engines/engine.py.229: process: 0.83160 | load: 0.00300 [I 2022-10-09 03:33:08] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-09 03:33:08] ...optimizers/trijoint.py.151: total_norm: 10.634033 [I 2022-10-09 03:33:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 1170/2304 [I 2022-10-09 03:33:17] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:15:56 [I 2022-10-09 03:33:17] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00299 [I 2022-10-09 03:33:17] ...trap/engines/engine.py.230: loss: 0.16484 [I 2022-10-09 03:33:17] ...optimizers/trijoint.py.151: total_norm: 15.210901 [I 2022-10-09 03:33:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 1180/2304 [I 2022-10-09 03:33:25] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:15:42 [I 2022-10-09 03:33:25] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00305 [I 2022-10-09 03:33:25] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-09 03:33:25] ...optimizers/trijoint.py.151: total_norm: 28.552397 [I 2022-10-09 03:33:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 1190/2304 [I 2022-10-09 03:33:33] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:15:37 [I 2022-10-09 03:33:33] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00300 [I 2022-10-09 03:33:33] ...trap/engines/engine.py.230: loss: 0.12667 [I 2022-10-09 03:33:33] ...optimizers/trijoint.py.151: total_norm: 17.118074 [I 2022-10-09 03:33:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 1200/2304 [I 2022-10-09 03:33:42] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:15:33 [I 2022-10-09 03:33:42] ...trap/engines/engine.py.229: process: 0.85593 | load: 0.00311 [I 2022-10-09 03:33:42] ...trap/engines/engine.py.230: loss: 0.20982 [I 2022-10-09 03:33:42] ...optimizers/trijoint.py.151: total_norm: 21.318651 [I 2022-10-09 03:33:50] ...trap/engines/engine.py.224: train: epoch 30 | batch 1210/2304 [I 2022-10-09 03:33:50] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:15:17 [I 2022-10-09 03:33:50] ...trap/engines/engine.py.229: process: 0.80833 | load: 0.00307 [I 2022-10-09 03:33:50] ...trap/engines/engine.py.230: loss: 0.10604 [I 2022-10-09 03:33:50] ...optimizers/trijoint.py.151: total_norm: 7.838655 [I 2022-10-09 03:33:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 1220/2304 [I 2022-10-09 03:33:59] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:13 [I 2022-10-09 03:33:59] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00310 [I 2022-10-09 03:33:59] ...trap/engines/engine.py.230: loss: 0.19121 [I 2022-10-09 03:33:59] ...optimizers/trijoint.py.151: total_norm: 25.545740 [I 2022-10-09 03:34:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 1230/2304 [I 2022-10-09 03:34:07] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:03 [I 2022-10-09 03:34:07] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00308 [I 2022-10-09 03:34:07] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-09 03:34:07] ...optimizers/trijoint.py.151: total_norm: 19.712540 [I 2022-10-09 03:34:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 1240/2304 [I 2022-10-09 03:34:16] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:14:54 [I 2022-10-09 03:34:16] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00309 [I 2022-10-09 03:34:16] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-09 03:34:16] ...optimizers/trijoint.py.151: total_norm: 25.927853 [I 2022-10-09 03:34:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 1250/2304 [I 2022-10-09 03:34:24] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:14:42 [I 2022-10-09 03:34:24] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00308 [I 2022-10-09 03:34:24] ...trap/engines/engine.py.230: loss: 0.16580 [I 2022-10-09 03:34:24] ...optimizers/trijoint.py.151: total_norm: 16.460722 [I 2022-10-09 03:34:32] ...trap/engines/engine.py.224: train: epoch 30 | batch 1260/2304 [I 2022-10-09 03:34:32] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:14:36 [I 2022-10-09 03:34:32] ...trap/engines/engine.py.229: process: 0.82964 | load: 0.00310 [I 2022-10-09 03:34:32] ...trap/engines/engine.py.230: loss: 0.14160 [I 2022-10-09 03:34:32] ...optimizers/trijoint.py.151: total_norm: 14.854846 [I 2022-10-09 03:34:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 1270/2304 [I 2022-10-09 03:34:41] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:14:29 [I 2022-10-09 03:34:41] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00299 [I 2022-10-09 03:34:41] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-09 03:34:41] ...optimizers/trijoint.py.151: total_norm: 21.460608 [I 2022-10-09 03:34:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 1280/2304 [I 2022-10-09 03:34:49] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:14:18 [I 2022-10-09 03:34:49] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00302 [I 2022-10-09 03:34:49] ...trap/engines/engine.py.230: loss: 0.20122 [I 2022-10-09 03:34:49] ...optimizers/trijoint.py.151: total_norm: 20.024645 [I 2022-10-09 03:34:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 1290/2304 [I 2022-10-09 03:34:58] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:14:11 [I 2022-10-09 03:34:58] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00300 [I 2022-10-09 03:34:58] ...trap/engines/engine.py.230: loss: 0.26889 [I 2022-10-09 03:34:58] ...optimizers/trijoint.py.151: total_norm: 22.991758 [I 2022-10-09 03:35:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 1300/2304 [I 2022-10-09 03:35:06] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:04 [I 2022-10-09 03:35:06] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00297 [I 2022-10-09 03:35:06] ...trap/engines/engine.py.230: loss: 0.20456 [I 2022-10-09 03:35:06] ...optimizers/trijoint.py.151: total_norm: 36.525986 [I 2022-10-09 03:35:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 1310/2304 [I 2022-10-09 03:35:15] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:13:50 [I 2022-10-09 03:35:15] ...trap/engines/engine.py.229: process: 0.81806 | load: 0.00300 [I 2022-10-09 03:35:15] ...trap/engines/engine.py.230: loss: 0.12621 [I 2022-10-09 03:35:15] ...optimizers/trijoint.py.151: total_norm: 17.501940 [I 2022-10-09 03:35:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 1320/2304 [I 2022-10-09 03:35:23] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:13:41 [I 2022-10-09 03:35:23] ...trap/engines/engine.py.229: process: 0.83051 | load: 0.00303 [I 2022-10-09 03:35:23] ...trap/engines/engine.py.230: loss: 0.24776 [I 2022-10-09 03:35:23] ...optimizers/trijoint.py.151: total_norm: 25.007652 [I 2022-10-09 03:35:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 1330/2304 [I 2022-10-09 03:35:31] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:13:39 [I 2022-10-09 03:35:31] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00301 [I 2022-10-09 03:35:31] ...trap/engines/engine.py.230: loss: 0.13592 [I 2022-10-09 03:35:31] ...optimizers/trijoint.py.151: total_norm: 9.584148 [I 2022-10-09 03:35:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 1340/2304 [I 2022-10-09 03:35:40] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:13:26 [I 2022-10-09 03:35:40] ...trap/engines/engine.py.229: process: 0.83896 | load: 0.00298 [I 2022-10-09 03:35:40] ...trap/engines/engine.py.230: loss: 0.22951 [I 2022-10-09 03:35:40] ...optimizers/trijoint.py.151: total_norm: 36.811871 [I 2022-10-09 03:35:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 1350/2304 [I 2022-10-09 03:35:48] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:23 [I 2022-10-09 03:35:48] ...trap/engines/engine.py.229: process: 0.84102 | load: 0.00299 [I 2022-10-09 03:35:48] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-09 03:35:48] ...optimizers/trijoint.py.151: total_norm: 12.249214 [I 2022-10-09 03:35:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 1360/2304 [I 2022-10-09 03:35:57] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:13:14 [I 2022-10-09 03:35:57] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00301 [I 2022-10-09 03:35:57] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-09 03:35:57] ...optimizers/trijoint.py.151: total_norm: 13.733060 [I 2022-10-09 03:36:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 1370/2304 [I 2022-10-09 03:36:05] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:04 [I 2022-10-09 03:36:05] ...trap/engines/engine.py.229: process: 0.82687 | load: 0.00302 [I 2022-10-09 03:36:05] ...trap/engines/engine.py.230: loss: 0.14771 [I 2022-10-09 03:36:05] ...optimizers/trijoint.py.151: total_norm: 24.842417 [I 2022-10-09 03:36:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 1380/2304 [I 2022-10-09 03:36:13] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:12:57 [I 2022-10-09 03:36:13] ...trap/engines/engine.py.229: process: 0.83428 | load: 0.00298 [I 2022-10-09 03:36:13] ...trap/engines/engine.py.230: loss: 0.14509 [I 2022-10-09 03:36:13] ...optimizers/trijoint.py.151: total_norm: 17.034975 [I 2022-10-09 03:36:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 1390/2304 [I 2022-10-09 03:36:22] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:12:45 [I 2022-10-09 03:36:22] ...trap/engines/engine.py.229: process: 0.84047 | load: 0.00305 [I 2022-10-09 03:36:22] ...trap/engines/engine.py.230: loss: 0.12334 [I 2022-10-09 03:36:22] ...optimizers/trijoint.py.151: total_norm: 12.972395 [I 2022-10-09 03:36:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 1400/2304 [I 2022-10-09 03:36:30] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:12:38 [I 2022-10-09 03:36:30] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00300 [I 2022-10-09 03:36:30] ...trap/engines/engine.py.230: loss: 0.22490 [I 2022-10-09 03:36:30] ...optimizers/trijoint.py.151: total_norm: 22.012083 [I 2022-10-09 03:36:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 1410/2304 [I 2022-10-09 03:36:39] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:12:31 [I 2022-10-09 03:36:39] ...trap/engines/engine.py.229: process: 0.83152 | load: 0.00300 [I 2022-10-09 03:36:39] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-09 03:36:39] ...optimizers/trijoint.py.151: total_norm: 46.243626 [I 2022-10-09 03:36:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 1420/2304 [I 2022-10-09 03:36:47] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:12:21 [I 2022-10-09 03:36:47] ...trap/engines/engine.py.229: process: 0.83455 | load: 0.00301 [I 2022-10-09 03:36:47] ...trap/engines/engine.py.230: loss: 0.16795 [I 2022-10-09 03:36:47] ...optimizers/trijoint.py.151: total_norm: 21.015705 [I 2022-10-09 03:36:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 1430/2304 [I 2022-10-09 03:36:56] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:18 [I 2022-10-09 03:36:56] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00305 [I 2022-10-09 03:36:56] ...trap/engines/engine.py.230: loss: 0.18316 [I 2022-10-09 03:36:56] ...optimizers/trijoint.py.151: total_norm: 21.470697 [I 2022-10-09 03:37:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 1440/2304 [I 2022-10-09 03:37:04] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:00 [I 2022-10-09 03:37:04] ...trap/engines/engine.py.229: process: 0.82869 | load: 0.00300 [I 2022-10-09 03:37:04] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-09 03:37:04] ...optimizers/trijoint.py.151: total_norm: 18.856325 [I 2022-10-09 03:37:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 1450/2304 [I 2022-10-09 03:37:12] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:01 [I 2022-10-09 03:37:12] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00298 [I 2022-10-09 03:37:12] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-09 03:37:12] ...optimizers/trijoint.py.151: total_norm: 18.838488 [I 2022-10-09 03:37:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 1460/2304 [I 2022-10-09 03:37:21] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:11:56 [I 2022-10-09 03:37:21] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00298 [I 2022-10-09 03:37:21] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-09 03:37:21] ...optimizers/trijoint.py.151: total_norm: 24.723383 [I 2022-10-09 03:37:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 1470/2304 [I 2022-10-09 03:37:29] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:11:44 [I 2022-10-09 03:37:29] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00302 [I 2022-10-09 03:37:29] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-09 03:37:29] ...optimizers/trijoint.py.151: total_norm: 20.725449 [I 2022-10-09 03:37:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 1480/2304 [I 2022-10-09 03:37:38] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:11:37 [I 2022-10-09 03:37:38] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00298 [I 2022-10-09 03:37:38] ...trap/engines/engine.py.230: loss: 0.24978 [I 2022-10-09 03:37:38] ...optimizers/trijoint.py.151: total_norm: 27.267948 [I 2022-10-09 03:37:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 1490/2304 [I 2022-10-09 03:37:46] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:11:27 [I 2022-10-09 03:37:46] ...trap/engines/engine.py.229: process: 0.84197 | load: 0.00303 [I 2022-10-09 03:37:46] ...trap/engines/engine.py.230: loss: 0.18594 [I 2022-10-09 03:37:46] ...optimizers/trijoint.py.151: total_norm: 22.703848 [I 2022-10-09 03:37:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 1500/2304 [I 2022-10-09 03:37:55] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:16 [I 2022-10-09 03:37:55] ...trap/engines/engine.py.229: process: 0.85259 | load: 0.00297 [I 2022-10-09 03:37:55] ...trap/engines/engine.py.230: loss: 0.16453 [I 2022-10-09 03:37:55] ...optimizers/trijoint.py.151: total_norm: 19.700350 [I 2022-10-09 03:38:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 1510/2304 [I 2022-10-09 03:38:03] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:08 [I 2022-10-09 03:38:03] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00299 [I 2022-10-09 03:38:03] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-09 03:38:03] ...optimizers/trijoint.py.151: total_norm: 16.462992 [I 2022-10-09 03:38:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 1520/2304 [I 2022-10-09 03:38:12] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:00 [I 2022-10-09 03:38:12] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00298 [I 2022-10-09 03:38:12] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-09 03:38:12] ...optimizers/trijoint.py.151: total_norm: 15.920177 [I 2022-10-09 03:38:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 1530/2304 [I 2022-10-09 03:38:20] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:10:51 [I 2022-10-09 03:38:20] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00302 [I 2022-10-09 03:38:20] ...trap/engines/engine.py.230: loss: 0.15516 [I 2022-10-09 03:38:20] ...optimizers/trijoint.py.151: total_norm: 14.051625 [I 2022-10-09 03:38:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 1540/2304 [I 2022-10-09 03:38:28] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:10:48 [I 2022-10-09 03:38:28] ...trap/engines/engine.py.229: process: 0.85960 | load: 0.00300 [I 2022-10-09 03:38:28] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-09 03:38:28] ...optimizers/trijoint.py.151: total_norm: 25.911478 [I 2022-10-09 03:38:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 1550/2304 [I 2022-10-09 03:38:37] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:10:33 [I 2022-10-09 03:38:37] ...trap/engines/engine.py.229: process: 0.83786 | load: 0.00298 [I 2022-10-09 03:38:37] ...trap/engines/engine.py.230: loss: 0.22651 [I 2022-10-09 03:38:37] ...optimizers/trijoint.py.151: total_norm: 20.564083 [I 2022-10-09 03:38:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 1560/2304 [I 2022-10-09 03:38:45] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:25 [I 2022-10-09 03:38:45] ...trap/engines/engine.py.229: process: 0.83466 | load: 0.00300 [I 2022-10-09 03:38:45] ...trap/engines/engine.py.230: loss: 0.15289 [I 2022-10-09 03:38:45] ...optimizers/trijoint.py.151: total_norm: 17.301214 [I 2022-10-09 03:38:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 1570/2304 [I 2022-10-09 03:38:54] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:12 [I 2022-10-09 03:38:54] ...trap/engines/engine.py.229: process: 0.83101 | load: 0.00303 [I 2022-10-09 03:38:54] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-09 03:38:54] ...optimizers/trijoint.py.151: total_norm: 18.080074 [I 2022-10-09 03:39:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 1580/2304 [I 2022-10-09 03:39:02] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:06 [I 2022-10-09 03:39:02] ...trap/engines/engine.py.229: process: 0.82654 | load: 0.00299 [I 2022-10-09 03:39:02] ...trap/engines/engine.py.230: loss: 0.15976 [I 2022-10-09 03:39:02] ...optimizers/trijoint.py.151: total_norm: 14.947003 [I 2022-10-09 03:39:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 1590/2304 [I 2022-10-09 03:39:10] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:09:55 [I 2022-10-09 03:39:10] ...trap/engines/engine.py.229: process: 0.83380 | load: 0.00304 [I 2022-10-09 03:39:10] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-09 03:39:10] ...optimizers/trijoint.py.151: total_norm: 14.632607 [I 2022-10-09 03:39:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 1600/2304 [I 2022-10-09 03:39:19] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:09:50 [I 2022-10-09 03:39:19] ...trap/engines/engine.py.229: process: 0.82679 | load: 0.00299 [I 2022-10-09 03:39:19] ...trap/engines/engine.py.230: loss: 0.19466 [I 2022-10-09 03:39:19] ...optimizers/trijoint.py.151: total_norm: 28.949526 [I 2022-10-09 03:39:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 1610/2304 [I 2022-10-09 03:39:27] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:09:43 [I 2022-10-09 03:39:27] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00321 [I 2022-10-09 03:39:27] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-09 03:39:27] ...optimizers/trijoint.py.151: total_norm: 14.085054 [I 2022-10-09 03:39:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 1620/2304 [I 2022-10-09 03:39:36] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:38 [I 2022-10-09 03:39:36] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00305 [I 2022-10-09 03:39:36] ...trap/engines/engine.py.230: loss: 0.10678 [I 2022-10-09 03:39:36] ...optimizers/trijoint.py.151: total_norm: 29.801336 [I 2022-10-09 03:39:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 1630/2304 [I 2022-10-09 03:39:44] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:25 [I 2022-10-09 03:39:44] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00304 [I 2022-10-09 03:39:44] ...trap/engines/engine.py.230: loss: 0.21789 [I 2022-10-09 03:39:44] ...optimizers/trijoint.py.151: total_norm: 31.394688 [I 2022-10-09 03:39:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 1640/2304 [I 2022-10-09 03:39:53] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:17 [I 2022-10-09 03:39:53] ...trap/engines/engine.py.229: process: 0.82385 | load: 0.00301 [I 2022-10-09 03:39:53] ...trap/engines/engine.py.230: loss: 0.17609 [I 2022-10-09 03:39:53] ...optimizers/trijoint.py.151: total_norm: 15.866565 [I 2022-10-09 03:40:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 1650/2304 [I 2022-10-09 03:40:01] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:09 [I 2022-10-09 03:40:01] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00297 [I 2022-10-09 03:40:01] ...trap/engines/engine.py.230: loss: 0.19736 [I 2022-10-09 03:40:01] ...optimizers/trijoint.py.151: total_norm: 15.814692 [I 2022-10-09 03:40:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 1660/2304 [I 2022-10-09 03:40:09] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:02 [I 2022-10-09 03:40:09] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00304 [I 2022-10-09 03:40:09] ...trap/engines/engine.py.230: loss: 0.15498 [I 2022-10-09 03:40:09] ...optimizers/trijoint.py.151: total_norm: 20.747416 [I 2022-10-09 03:40:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 1670/2304 [I 2022-10-09 03:40:18] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:08:52 [I 2022-10-09 03:40:18] ...trap/engines/engine.py.229: process: 0.83400 | load: 0.00306 [I 2022-10-09 03:40:18] ...trap/engines/engine.py.230: loss: 0.17324 [I 2022-10-09 03:40:18] ...optimizers/trijoint.py.151: total_norm: 15.790137 [I 2022-10-09 03:40:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 1680/2304 [I 2022-10-09 03:40:26] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:08:42 [I 2022-10-09 03:40:26] ...trap/engines/engine.py.229: process: 0.83816 | load: 0.00298 [I 2022-10-09 03:40:26] ...trap/engines/engine.py.230: loss: 0.24779 [I 2022-10-09 03:40:26] ...optimizers/trijoint.py.151: total_norm: 13.842221 [I 2022-10-09 03:40:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 1690/2304 [I 2022-10-09 03:40:35] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:37 [I 2022-10-09 03:40:35] ...trap/engines/engine.py.229: process: 0.83439 | load: 0.00298 [I 2022-10-09 03:40:35] ...trap/engines/engine.py.230: loss: 0.18486 [I 2022-10-09 03:40:35] ...optimizers/trijoint.py.151: total_norm: 16.298325 [I 2022-10-09 03:40:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 1700/2304 [I 2022-10-09 03:40:43] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:30 [I 2022-10-09 03:40:43] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00301 [I 2022-10-09 03:40:43] ...trap/engines/engine.py.230: loss: 0.17469 [I 2022-10-09 03:40:43] ...optimizers/trijoint.py.151: total_norm: 24.801060 [I 2022-10-09 03:40:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 1710/2304 [I 2022-10-09 03:40:52] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:18 [I 2022-10-09 03:40:52] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00310 [I 2022-10-09 03:40:52] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-09 03:40:52] ...optimizers/trijoint.py.151: total_norm: 18.681700 [I 2022-10-09 03:41:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 1720/2304 [I 2022-10-09 03:41:00] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:08 [I 2022-10-09 03:41:00] ...trap/engines/engine.py.229: process: 0.82396 | load: 0.00301 [I 2022-10-09 03:41:00] ...trap/engines/engine.py.230: loss: 0.12665 [I 2022-10-09 03:41:00] ...optimizers/trijoint.py.151: total_norm: 23.301062 [I 2022-10-09 03:41:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 1730/2304 [I 2022-10-09 03:41:08] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:01 [I 2022-10-09 03:41:08] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00305 [I 2022-10-09 03:41:08] ...trap/engines/engine.py.230: loss: 0.21267 [I 2022-10-09 03:41:08] ...optimizers/trijoint.py.151: total_norm: 23.365026 [I 2022-10-09 03:41:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 1740/2304 [I 2022-10-09 03:41:17] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:07:52 [I 2022-10-09 03:41:17] ...trap/engines/engine.py.229: process: 0.83341 | load: 0.00300 [I 2022-10-09 03:41:17] ...trap/engines/engine.py.230: loss: 0.22072 [I 2022-10-09 03:41:17] ...optimizers/trijoint.py.151: total_norm: 18.304047 [I 2022-10-09 03:41:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 1750/2304 [I 2022-10-09 03:41:25] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:07:46 [I 2022-10-09 03:41:25] ...trap/engines/engine.py.229: process: 0.85973 | load: 0.00301 [I 2022-10-09 03:41:25] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-09 03:41:25] ...optimizers/trijoint.py.151: total_norm: 22.523830 [I 2022-10-09 03:41:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 1760/2304 [I 2022-10-09 03:41:34] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:38 [I 2022-10-09 03:41:34] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00300 [I 2022-10-09 03:41:34] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-09 03:41:34] ...optimizers/trijoint.py.151: total_norm: 30.411873 [I 2022-10-09 03:41:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 1770/2304 [I 2022-10-09 03:41:42] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:24 [I 2022-10-09 03:41:42] ...trap/engines/engine.py.229: process: 0.81513 | load: 0.00301 [I 2022-10-09 03:41:42] ...trap/engines/engine.py.230: loss: 0.15119 [I 2022-10-09 03:41:42] ...optimizers/trijoint.py.151: total_norm: 18.198635 [I 2022-10-09 03:41:50] ...trap/engines/engine.py.224: train: epoch 30 | batch 1780/2304 [I 2022-10-09 03:41:50] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:22 [I 2022-10-09 03:41:50] ...trap/engines/engine.py.229: process: 0.84805 | load: 0.00302 [I 2022-10-09 03:41:50] ...trap/engines/engine.py.230: loss: 0.21212 [I 2022-10-09 03:41:50] ...optimizers/trijoint.py.151: total_norm: 13.180420 [I 2022-10-09 03:41:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 1790/2304 [I 2022-10-09 03:41:59] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:10 [I 2022-10-09 03:41:59] ...trap/engines/engine.py.229: process: 0.82972 | load: 0.00299 [I 2022-10-09 03:41:59] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-09 03:41:59] ...optimizers/trijoint.py.151: total_norm: 15.273314 [I 2022-10-09 03:42:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 1800/2304 [I 2022-10-09 03:42:07] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:04 [I 2022-10-09 03:42:07] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00303 [I 2022-10-09 03:42:07] ...trap/engines/engine.py.230: loss: 0.16723 [I 2022-10-09 03:42:07] ...optimizers/trijoint.py.151: total_norm: 20.906311 [I 2022-10-09 03:42:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 1810/2304 [I 2022-10-09 03:42:16] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:06:54 [I 2022-10-09 03:42:16] ...trap/engines/engine.py.229: process: 0.83037 | load: 0.00302 [I 2022-10-09 03:42:16] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-09 03:42:16] ...optimizers/trijoint.py.151: total_norm: 22.182186 [I 2022-10-09 03:42:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 1820/2304 [I 2022-10-09 03:42:24] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:06:47 [I 2022-10-09 03:42:24] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00300 [I 2022-10-09 03:42:24] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-09 03:42:24] ...optimizers/trijoint.py.151: total_norm: 12.085361 [I 2022-10-09 03:42:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 1830/2304 [I 2022-10-09 03:42:33] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:06:39 [I 2022-10-09 03:42:33] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00302 [I 2022-10-09 03:42:33] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-09 03:42:33] ...optimizers/trijoint.py.151: total_norm: 15.276307 [I 2022-10-09 03:42:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 1840/2304 [I 2022-10-09 03:42:41] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:30 [I 2022-10-09 03:42:41] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00298 [I 2022-10-09 03:42:41] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-09 03:42:41] ...optimizers/trijoint.py.151: total_norm: 11.912651 [I 2022-10-09 03:42:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 1850/2304 [I 2022-10-09 03:42:49] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:19 [I 2022-10-09 03:42:49] ...trap/engines/engine.py.229: process: 0.83149 | load: 0.00299 [I 2022-10-09 03:42:49] ...trap/engines/engine.py.230: loss: 0.15430 [I 2022-10-09 03:42:49] ...optimizers/trijoint.py.151: total_norm: 11.131056 [I 2022-10-09 03:42:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 1860/2304 [I 2022-10-09 03:42:58] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:13 [I 2022-10-09 03:42:58] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00300 [I 2022-10-09 03:42:58] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-09 03:42:58] ...optimizers/trijoint.py.151: total_norm: 13.765666 [I 2022-10-09 03:43:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 1870/2304 [I 2022-10-09 03:43:06] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:03 [I 2022-10-09 03:43:06] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00305 [I 2022-10-09 03:43:06] ...trap/engines/engine.py.230: loss: 0.18753 [I 2022-10-09 03:43:06] ...optimizers/trijoint.py.151: total_norm: 12.749006 [I 2022-10-09 03:43:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 1880/2304 [I 2022-10-09 03:43:15] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:05:56 [I 2022-10-09 03:43:15] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00298 [I 2022-10-09 03:43:15] ...trap/engines/engine.py.230: loss: 0.12710 [I 2022-10-09 03:43:15] ...optimizers/trijoint.py.151: total_norm: 13.560698 [I 2022-10-09 03:43:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 1890/2304 [I 2022-10-09 03:43:23] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:05:45 [I 2022-10-09 03:43:23] ...trap/engines/engine.py.229: process: 0.84511 | load: 0.00298 [I 2022-10-09 03:43:23] ...trap/engines/engine.py.230: loss: 0.11168 [I 2022-10-09 03:43:23] ...optimizers/trijoint.py.151: total_norm: 9.914666 [I 2022-10-09 03:43:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 1900/2304 [I 2022-10-09 03:43:31] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:05:39 [I 2022-10-09 03:43:31] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00300 [I 2022-10-09 03:43:31] ...trap/engines/engine.py.230: loss: 0.15799 [I 2022-10-09 03:43:31] ...optimizers/trijoint.py.151: total_norm: 14.094894 [I 2022-10-09 03:43:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 1910/2304 [I 2022-10-09 03:43:40] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:31 [I 2022-10-09 03:43:40] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00301 [I 2022-10-09 03:43:40] ...trap/engines/engine.py.230: loss: 0.18942 [I 2022-10-09 03:43:40] ...optimizers/trijoint.py.151: total_norm: 24.751257 [I 2022-10-09 03:43:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 1920/2304 [I 2022-10-09 03:43:48] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:21 [I 2022-10-09 03:43:48] ...trap/engines/engine.py.229: process: 0.83594 | load: 0.00301 [I 2022-10-09 03:43:48] ...trap/engines/engine.py.230: loss: 0.17546 [I 2022-10-09 03:43:48] ...optimizers/trijoint.py.151: total_norm: 47.372864 [I 2022-10-09 03:43:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 1930/2304 [I 2022-10-09 03:43:57] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:14 [I 2022-10-09 03:43:57] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00300 [I 2022-10-09 03:43:57] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-09 03:43:57] ...optimizers/trijoint.py.151: total_norm: 13.962724 [I 2022-10-09 03:44:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 1940/2304 [I 2022-10-09 03:44:05] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:04 [I 2022-10-09 03:44:05] ...trap/engines/engine.py.229: process: 0.82343 | load: 0.00304 [I 2022-10-09 03:44:05] ...trap/engines/engine.py.230: loss: 0.09940 [I 2022-10-09 03:44:05] ...optimizers/trijoint.py.151: total_norm: 12.097075 [I 2022-10-09 03:44:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 1950/2304 [I 2022-10-09 03:44:13] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:04:58 [I 2022-10-09 03:44:13] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00318 [I 2022-10-09 03:44:13] ...trap/engines/engine.py.230: loss: 0.19692 [I 2022-10-09 03:44:13] ...optimizers/trijoint.py.151: total_norm: 33.350716 [I 2022-10-09 03:44:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 1960/2304 [I 2022-10-09 03:44:22] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:04:49 [I 2022-10-09 03:44:22] ...trap/engines/engine.py.229: process: 0.83609 | load: 0.00309 [I 2022-10-09 03:44:22] ...trap/engines/engine.py.230: loss: 0.17477 [I 2022-10-09 03:44:22] ...optimizers/trijoint.py.151: total_norm: 14.686720 [I 2022-10-09 03:44:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 1970/2304 [I 2022-10-09 03:44:30] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:04:40 [I 2022-10-09 03:44:30] ...trap/engines/engine.py.229: process: 0.82672 | load: 0.00303 [I 2022-10-09 03:44:30] ...trap/engines/engine.py.230: loss: 0.19039 [I 2022-10-09 03:44:30] ...optimizers/trijoint.py.151: total_norm: 23.154760 [I 2022-10-09 03:44:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 1980/2304 [I 2022-10-09 03:44:39] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:32 [I 2022-10-09 03:44:39] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00300 [I 2022-10-09 03:44:39] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-09 03:44:39] ...optimizers/trijoint.py.151: total_norm: 20.316528 [I 2022-10-09 03:44:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 1990/2304 [I 2022-10-09 03:44:47] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:04:25 [I 2022-10-09 03:44:47] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00305 [I 2022-10-09 03:44:47] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-09 03:44:47] ...optimizers/trijoint.py.151: total_norm: 20.409725 [I 2022-10-09 03:44:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 2000/2304 [I 2022-10-09 03:44:56] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:16 [I 2022-10-09 03:44:56] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00308 [I 2022-10-09 03:44:56] ...trap/engines/engine.py.230: loss: 0.10277 [I 2022-10-09 03:44:56] ...optimizers/trijoint.py.151: total_norm: 10.300097 [I 2022-10-09 03:45:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 2010/2304 [I 2022-10-09 03:45:04] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:07 [I 2022-10-09 03:45:04] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00303 [I 2022-10-09 03:45:04] ...trap/engines/engine.py.230: loss: 0.18067 [I 2022-10-09 03:45:04] ...optimizers/trijoint.py.151: total_norm: 23.724400 [I 2022-10-09 03:45:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 2020/2304 [I 2022-10-09 03:45:12] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:03:58 [I 2022-10-09 03:45:12] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00303 [I 2022-10-09 03:45:12] ...trap/engines/engine.py.230: loss: 0.22051 [I 2022-10-09 03:45:12] ...optimizers/trijoint.py.151: total_norm: 23.523140 [I 2022-10-09 03:45:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 2030/2304 [I 2022-10-09 03:45:21] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:03:49 [I 2022-10-09 03:45:21] ...trap/engines/engine.py.229: process: 0.82988 | load: 0.00302 [I 2022-10-09 03:45:21] ...trap/engines/engine.py.230: loss: 0.17257 [I 2022-10-09 03:45:21] ...optimizers/trijoint.py.151: total_norm: 23.180344 [I 2022-10-09 03:45:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 2040/2304 [I 2022-10-09 03:45:29] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:03:43 [I 2022-10-09 03:45:29] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00355 [I 2022-10-09 03:45:29] ...trap/engines/engine.py.230: loss: 0.14657 [I 2022-10-09 03:45:29] ...optimizers/trijoint.py.151: total_norm: 15.540564 [I 2022-10-09 03:45:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 2050/2304 [I 2022-10-09 03:45:38] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:33 [I 2022-10-09 03:45:38] ...trap/engines/engine.py.229: process: 0.83546 | load: 0.00302 [I 2022-10-09 03:45:38] ...trap/engines/engine.py.230: loss: 0.18400 [I 2022-10-09 03:45:38] ...optimizers/trijoint.py.151: total_norm: 20.910740 [I 2022-10-09 03:45:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 2060/2304 [I 2022-10-09 03:45:46] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:03:26 [I 2022-10-09 03:45:46] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00297 [I 2022-10-09 03:45:46] ...trap/engines/engine.py.230: loss: 0.12670 [I 2022-10-09 03:45:46] ...optimizers/trijoint.py.151: total_norm: 14.431529 [I 2022-10-09 03:45:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 2070/2304 [I 2022-10-09 03:45:55] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:17 [I 2022-10-09 03:45:55] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00298 [I 2022-10-09 03:45:55] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-09 03:45:55] ...optimizers/trijoint.py.151: total_norm: 12.781319 [I 2022-10-09 03:46:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 2080/2304 [I 2022-10-09 03:46:03] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:08 [I 2022-10-09 03:46:03] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00297 [I 2022-10-09 03:46:03] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-09 03:46:03] ...optimizers/trijoint.py.151: total_norm: 28.280722 [I 2022-10-09 03:46:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 2090/2304 [I 2022-10-09 03:46:12] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:02:59 [I 2022-10-09 03:46:12] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00297 [I 2022-10-09 03:46:12] ...trap/engines/engine.py.230: loss: 0.12277 [I 2022-10-09 03:46:12] ...optimizers/trijoint.py.151: total_norm: 11.856105 [I 2022-10-09 03:46:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 2100/2304 [I 2022-10-09 03:46:20] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:02:52 [I 2022-10-09 03:46:20] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00301 [I 2022-10-09 03:46:20] ...trap/engines/engine.py.230: loss: 0.21774 [I 2022-10-09 03:46:20] ...optimizers/trijoint.py.151: total_norm: 15.434563 [I 2022-10-09 03:46:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 2110/2304 [I 2022-10-09 03:46:29] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:02:44 [I 2022-10-09 03:46:29] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00303 [I 2022-10-09 03:46:29] ...trap/engines/engine.py.230: loss: 0.21477 [I 2022-10-09 03:46:29] ...optimizers/trijoint.py.151: total_norm: 57.845566 [I 2022-10-09 03:46:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 2120/2304 [I 2022-10-09 03:46:37] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:35 [I 2022-10-09 03:46:37] ...trap/engines/engine.py.229: process: 0.82423 | load: 0.00302 [I 2022-10-09 03:46:37] ...trap/engines/engine.py.230: loss: 0.20532 [I 2022-10-09 03:46:37] ...optimizers/trijoint.py.151: total_norm: 97.733810 [I 2022-10-09 03:46:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 2130/2304 [I 2022-10-09 03:46:45] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:02:26 [I 2022-10-09 03:46:45] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00299 [I 2022-10-09 03:46:45] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-09 03:46:45] ...optimizers/trijoint.py.151: total_norm: 25.259916 [I 2022-10-09 03:46:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 2140/2304 [I 2022-10-09 03:46:54] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:17 [I 2022-10-09 03:46:54] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00299 [I 2022-10-09 03:46:54] ...trap/engines/engine.py.230: loss: 0.18252 [I 2022-10-09 03:46:54] ...optimizers/trijoint.py.151: total_norm: 21.373453 [I 2022-10-09 03:47:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 2150/2304 [I 2022-10-09 03:47:02] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:10 [I 2022-10-09 03:47:02] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00300 [I 2022-10-09 03:47:02] ...trap/engines/engine.py.230: loss: 0.17046 [I 2022-10-09 03:47:02] ...optimizers/trijoint.py.151: total_norm: 46.510101 [I 2022-10-09 03:47:11] ...trap/engines/engine.py.224: train: epoch 30 | batch 2160/2304 [I 2022-10-09 03:47:11] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:00 [I 2022-10-09 03:47:11] ...trap/engines/engine.py.229: process: 0.83822 | load: 0.00302 [I 2022-10-09 03:47:11] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-09 03:47:11] ...optimizers/trijoint.py.151: total_norm: 18.097242 [I 2022-10-09 03:47:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 2170/2304 [I 2022-10-09 03:47:19] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:01:52 [I 2022-10-09 03:47:19] ...trap/engines/engine.py.229: process: 0.83232 | load: 0.00298 [I 2022-10-09 03:47:19] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-09 03:47:19] ...optimizers/trijoint.py.151: total_norm: 16.029648 [I 2022-10-09 03:47:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 2180/2304 [I 2022-10-09 03:47:28] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:01:44 [I 2022-10-09 03:47:28] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00300 [I 2022-10-09 03:47:28] ...trap/engines/engine.py.230: loss: 0.17766 [I 2022-10-09 03:47:28] ...optimizers/trijoint.py.151: total_norm: 12.313263 [I 2022-10-09 03:47:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 2190/2304 [I 2022-10-09 03:47:36] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:36 [I 2022-10-09 03:47:36] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00299 [I 2022-10-09 03:47:36] ...trap/engines/engine.py.230: loss: 0.15501 [I 2022-10-09 03:47:36] ...optimizers/trijoint.py.151: total_norm: 13.551508 [I 2022-10-09 03:47:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 2200/2304 [I 2022-10-09 03:47:44] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:01:27 [I 2022-10-09 03:47:44] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00304 [I 2022-10-09 03:47:44] ...trap/engines/engine.py.230: loss: 0.18301 [I 2022-10-09 03:47:44] ...optimizers/trijoint.py.151: total_norm: 16.446829 [I 2022-10-09 03:47:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 2210/2304 [I 2022-10-09 03:47:53] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:18 [I 2022-10-09 03:47:53] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00299 [I 2022-10-09 03:47:53] ...trap/engines/engine.py.230: loss: 0.15944 [I 2022-10-09 03:47:53] ...optimizers/trijoint.py.151: total_norm: 14.602787 [I 2022-10-09 03:48:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 2220/2304 [I 2022-10-09 03:48:01] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:11 [I 2022-10-09 03:48:01] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00300 [I 2022-10-09 03:48:01] ...trap/engines/engine.py.230: loss: 0.20638 [I 2022-10-09 03:48:01] ...optimizers/trijoint.py.151: total_norm: 17.492083 [I 2022-10-09 03:48:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 2230/2304 [I 2022-10-09 03:48:10] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:02 [I 2022-10-09 03:48:10] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00303 [I 2022-10-09 03:48:10] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-09 03:48:10] ...optimizers/trijoint.py.151: total_norm: 16.261192 [I 2022-10-09 03:48:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 2240/2304 [I 2022-10-09 03:48:18] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:00:53 [I 2022-10-09 03:48:18] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00300 [I 2022-10-09 03:48:18] ...trap/engines/engine.py.230: loss: 0.14519 [I 2022-10-09 03:48:18] ...optimizers/trijoint.py.151: total_norm: 9.096135 [I 2022-10-09 03:48:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 2250/2304 [I 2022-10-09 03:48:26] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:00:44 [I 2022-10-09 03:48:26] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00300 [I 2022-10-09 03:48:26] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-09 03:48:26] ...optimizers/trijoint.py.151: total_norm: 14.538384 [I 2022-10-09 03:48:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 2260/2304 [I 2022-10-09 03:48:35] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:37 [I 2022-10-09 03:48:35] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00300 [I 2022-10-09 03:48:35] ...trap/engines/engine.py.230: loss: 0.16709 [I 2022-10-09 03:48:35] ...optimizers/trijoint.py.151: total_norm: 30.453793 [I 2022-10-09 03:48:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 2270/2304 [I 2022-10-09 03:48:43] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:00:28 [I 2022-10-09 03:48:43] ...trap/engines/engine.py.229: process: 0.85649 | load: 0.00299 [I 2022-10-09 03:48:43] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-09 03:48:43] ...optimizers/trijoint.py.151: total_norm: 11.326885 [I 2022-10-09 03:48:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 2280/2304 [I 2022-10-09 03:48:51] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:20 [I 2022-10-09 03:48:51] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00300 [I 2022-10-09 03:48:51] ...trap/engines/engine.py.230: loss: 0.18397 [I 2022-10-09 03:48:51] ...optimizers/trijoint.py.151: total_norm: 14.009711 [I 2022-10-09 03:49:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 2290/2304 [I 2022-10-09 03:49:00] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:11 [I 2022-10-09 03:49:00] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00300 [I 2022-10-09 03:49:00] ...trap/engines/engine.py.230: loss: 0.13514 [I 2022-10-09 03:49:00] ...optimizers/trijoint.py.151: total_norm: 16.612778 [I 2022-10-09 03:49:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 2300/2304 [I 2022-10-09 03:49:08] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:03 [I 2022-10-09 03:49:08] ...trap/engines/engine.py.229: process: 0.81227 | load: 0.00017 [I 2022-10-09 03:49:08] ...trap/engines/engine.py.230: loss: 0.13136 [I 2022-10-09 03:49:08] ...optimizers/trijoint.py.151: total_norm: 10.949788 [I 2022-10-09 03:49:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 2304/2304 [I 2022-10-09 03:49:12] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:00 [I 2022-10-09 03:49:12] ...trap/engines/engine.py.229: process: 0.81800 | load: 0.00016 [I 2022-10-09 03:49:12] ...trap/engines/engine.py.230: loss: 0.17550 [I 2022-10-09 03:49:12] ...optimizers/trijoint.py.151: total_norm: 43.246826 [S 2022-10-09 03:49:13] ...trap/engines/engine.py.129: train_epoch.epoch: 30 [S 2022-10-09 03:49:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.16514129613136264 [I 2022-10-09 03:49:13] ...s/criterions/triplet.py.84: margin = 0.20500000000000013 [I 2022-10-09 03:49:13] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 30 [I 2022-10-09 03:49:13] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 03:49:15] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 03:49:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 03:49:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 03:49:16] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 03:49:17] ...trap/engines/engine.py.321: eval: epoch 30 | batch 0/510 [I 2022-10-09 03:49:17] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:27 [I 2022-10-09 03:49:17] ...trap/engines/engine.py.326: process: 3.34760 | load: 2.82696 [I 2022-10-09 03:49:21] ...trap/engines/engine.py.321: eval: epoch 30 | batch 10/510 [I 2022-10-09 03:49:21] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:10 [I 2022-10-09 03:49:21] ...trap/engines/engine.py.326: process: 0.43177 | load: 0.00019 [I 2022-10-09 03:49:26] ...trap/engines/engine.py.321: eval: epoch 30 | batch 20/510 [I 2022-10-09 03:49:26] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:21 [I 2022-10-09 03:49:26] ...trap/engines/engine.py.326: process: 0.57436 | load: 0.00019 [I 2022-10-09 03:49:30] ...trap/engines/engine.py.321: eval: epoch 30 | batch 30/510 [I 2022-10-09 03:49:30] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:03:31 [I 2022-10-09 03:49:30] ...trap/engines/engine.py.326: process: 0.42422 | load: 0.00018 [I 2022-10-09 03:49:34] ...trap/engines/engine.py.321: eval: epoch 30 | batch 40/510 [I 2022-10-09 03:49:34] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:03:20 [I 2022-10-09 03:49:34] ...trap/engines/engine.py.326: process: 0.42232 | load: 0.00019 [I 2022-10-09 03:49:39] ...trap/engines/engine.py.321: eval: epoch 30 | batch 50/510 [I 2022-10-09 03:49:39] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:24 [I 2022-10-09 03:49:39] ...trap/engines/engine.py.326: process: 0.48734 | load: 0.00019 [I 2022-10-09 03:49:43] ...trap/engines/engine.py.321: eval: epoch 30 | batch 60/510 [I 2022-10-09 03:49:43] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:14 [I 2022-10-09 03:49:43] ...trap/engines/engine.py.326: process: 0.42233 | load: 0.00018 [I 2022-10-09 03:49:48] ...trap/engines/engine.py.321: eval: epoch 30 | batch 70/510 [I 2022-10-09 03:49:48] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:15 [I 2022-10-09 03:49:48] ...trap/engines/engine.py.326: process: 0.44802 | load: 0.00018 [I 2022-10-09 03:49:52] ...trap/engines/engine.py.321: eval: epoch 30 | batch 80/510 [I 2022-10-09 03:49:52] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:06 [I 2022-10-09 03:49:52] ...trap/engines/engine.py.326: process: 0.43980 | load: 0.00018 [I 2022-10-09 03:49:56] ...trap/engines/engine.py.321: eval: epoch 30 | batch 90/510 [I 2022-10-09 03:49:56] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:02:59 [I 2022-10-09 03:49:56] ...trap/engines/engine.py.326: process: 0.41792 | load: 0.00018 [I 2022-10-09 03:50:01] ...trap/engines/engine.py.321: eval: epoch 30 | batch 100/510 [I 2022-10-09 03:50:01] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:02:43 [I 2022-10-09 03:50:01] ...trap/engines/engine.py.326: process: 0.31058 | load: 0.00019 [I 2022-10-09 03:50:04] ...trap/engines/engine.py.321: eval: epoch 30 | batch 110/510 [I 2022-10-09 03:50:04] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:02:10 [I 2022-10-09 03:50:04] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00021 [I 2022-10-09 03:50:07] ...trap/engines/engine.py.321: eval: epoch 30 | batch 120/510 [I 2022-10-09 03:50:07] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:04 [I 2022-10-09 03:50:07] ...trap/engines/engine.py.326: process: 0.31827 | load: 0.00020 [I 2022-10-09 03:50:10] ...trap/engines/engine.py.321: eval: epoch 30 | batch 130/510 [I 2022-10-09 03:50:10] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:00 [I 2022-10-09 03:50:10] ...trap/engines/engine.py.326: process: 0.31553 | load: 0.00020 [I 2022-10-09 03:50:13] ...trap/engines/engine.py.321: eval: epoch 30 | batch 140/510 [I 2022-10-09 03:50:13] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:57 [I 2022-10-09 03:50:13] ...trap/engines/engine.py.326: process: 0.31963 | load: 0.00020 [I 2022-10-09 03:50:16] ...trap/engines/engine.py.321: eval: epoch 30 | batch 150/510 [I 2022-10-09 03:50:16] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:55 [I 2022-10-09 03:50:16] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00019 [I 2022-10-09 03:50:20] ...trap/engines/engine.py.321: eval: epoch 30 | batch 160/510 [I 2022-10-09 03:50:20] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:52 [I 2022-10-09 03:50:20] ...trap/engines/engine.py.326: process: 0.32224 | load: 0.00020 [I 2022-10-09 03:50:23] ...trap/engines/engine.py.321: eval: epoch 30 | batch 170/510 [I 2022-10-09 03:50:23] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:48 [I 2022-10-09 03:50:23] ...trap/engines/engine.py.326: process: 0.31897 | load: 0.00020 [I 2022-10-09 03:50:26] ...trap/engines/engine.py.321: eval: epoch 30 | batch 180/510 [I 2022-10-09 03:50:26] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:45 [I 2022-10-09 03:50:26] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00019 [I 2022-10-09 03:50:29] ...trap/engines/engine.py.321: eval: epoch 30 | batch 190/510 [I 2022-10-09 03:50:29] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:42 [I 2022-10-09 03:50:29] ...trap/engines/engine.py.326: process: 0.31466 | load: 0.00020 [I 2022-10-09 03:50:32] ...trap/engines/engine.py.321: eval: epoch 30 | batch 200/510 [I 2022-10-09 03:50:32] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:38 [I 2022-10-09 03:50:32] ...trap/engines/engine.py.326: process: 0.31737 | load: 0.00020 [I 2022-10-09 03:50:36] ...trap/engines/engine.py.321: eval: epoch 30 | batch 210/510 [I 2022-10-09 03:50:36] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:35 [I 2022-10-09 03:50:36] ...trap/engines/engine.py.326: process: 0.31678 | load: 0.00021 [I 2022-10-09 03:50:39] ...trap/engines/engine.py.321: eval: epoch 30 | batch 220/510 [I 2022-10-09 03:50:39] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:32 [I 2022-10-09 03:50:39] ...trap/engines/engine.py.326: process: 0.31743 | load: 0.00019 [I 2022-10-09 03:50:42] ...trap/engines/engine.py.321: eval: epoch 30 | batch 230/510 [I 2022-10-09 03:50:42] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:29 [I 2022-10-09 03:50:42] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00019 [I 2022-10-09 03:50:45] ...trap/engines/engine.py.321: eval: epoch 30 | batch 240/510 [I 2022-10-09 03:50:45] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:25 [I 2022-10-09 03:50:45] ...trap/engines/engine.py.326: process: 0.32091 | load: 0.00019 [I 2022-10-09 03:50:48] ...trap/engines/engine.py.321: eval: epoch 30 | batch 250/510 [I 2022-10-09 03:50:48] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:22 [I 2022-10-09 03:50:48] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00020 [I 2022-10-09 03:50:52] ...trap/engines/engine.py.321: eval: epoch 30 | batch 260/510 [I 2022-10-09 03:50:52] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:19 [I 2022-10-09 03:50:52] ...trap/engines/engine.py.326: process: 0.31954 | load: 0.00021 [I 2022-10-09 03:50:55] ...trap/engines/engine.py.321: eval: epoch 30 | batch 270/510 [I 2022-10-09 03:50:55] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:15 [I 2022-10-09 03:50:55] ...trap/engines/engine.py.326: process: 0.31204 | load: 0.00020 [I 2022-10-09 03:50:58] ...trap/engines/engine.py.321: eval: epoch 30 | batch 280/510 [I 2022-10-09 03:50:58] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:13 [I 2022-10-09 03:50:58] ...trap/engines/engine.py.326: process: 0.32294 | load: 0.00020 [I 2022-10-09 03:51:01] ...trap/engines/engine.py.321: eval: epoch 30 | batch 290/510 [I 2022-10-09 03:51:01] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:10 [I 2022-10-09 03:51:01] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00019 [I 2022-10-09 03:51:04] ...trap/engines/engine.py.321: eval: epoch 30 | batch 300/510 [I 2022-10-09 03:51:04] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:06 [I 2022-10-09 03:51:04] ...trap/engines/engine.py.326: process: 0.32096 | load: 0.00020 [I 2022-10-09 03:51:07] ...trap/engines/engine.py.321: eval: epoch 30 | batch 310/510 [I 2022-10-09 03:51:07] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:03 [I 2022-10-09 03:51:07] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00020 [I 2022-10-09 03:51:11] ...trap/engines/engine.py.321: eval: epoch 30 | batch 320/510 [I 2022-10-09 03:51:11] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:00 [I 2022-10-09 03:51:11] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00019 [I 2022-10-09 03:51:14] ...trap/engines/engine.py.321: eval: epoch 30 | batch 330/510 [I 2022-10-09 03:51:14] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:57 [I 2022-10-09 03:51:14] ...trap/engines/engine.py.326: process: 0.32216 | load: 0.00020 [I 2022-10-09 03:51:17] ...trap/engines/engine.py.321: eval: epoch 30 | batch 340/510 [I 2022-10-09 03:51:17] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:54 [I 2022-10-09 03:51:17] ...trap/engines/engine.py.326: process: 0.31741 | load: 0.00020 [I 2022-10-09 03:51:20] ...trap/engines/engine.py.321: eval: epoch 30 | batch 350/510 [I 2022-10-09 03:51:20] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:51 [I 2022-10-09 03:51:20] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00020 [I 2022-10-09 03:51:23] ...trap/engines/engine.py.321: eval: epoch 30 | batch 360/510 [I 2022-10-09 03:51:23] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:47 [I 2022-10-09 03:51:23] ...trap/engines/engine.py.326: process: 0.31366 | load: 0.00019 [I 2022-10-09 03:51:27] ...trap/engines/engine.py.321: eval: epoch 30 | batch 370/510 [I 2022-10-09 03:51:27] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:44 [I 2022-10-09 03:51:27] ...trap/engines/engine.py.326: process: 0.32247 | load: 0.00019 [I 2022-10-09 03:51:30] ...trap/engines/engine.py.321: eval: epoch 30 | batch 380/510 [I 2022-10-09 03:51:30] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:41 [I 2022-10-09 03:51:30] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00020 [I 2022-10-09 03:51:33] ...trap/engines/engine.py.321: eval: epoch 30 | batch 390/510 [I 2022-10-09 03:51:33] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:37 [I 2022-10-09 03:51:33] ...trap/engines/engine.py.326: process: 0.32030 | load: 0.00020 [I 2022-10-09 03:51:36] ...trap/engines/engine.py.321: eval: epoch 30 | batch 400/510 [I 2022-10-09 03:51:36] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:35 [I 2022-10-09 03:51:36] ...trap/engines/engine.py.326: process: 0.32183 | load: 0.00020 [I 2022-10-09 03:51:39] ...trap/engines/engine.py.321: eval: epoch 30 | batch 410/510 [I 2022-10-09 03:51:39] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:31 [I 2022-10-09 03:51:39] ...trap/engines/engine.py.326: process: 0.31505 | load: 0.00020 [I 2022-10-09 03:51:43] ...trap/engines/engine.py.321: eval: epoch 30 | batch 420/510 [I 2022-10-09 03:51:43] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:28 [I 2022-10-09 03:51:43] ...trap/engines/engine.py.326: process: 0.32290 | load: 0.00020 [I 2022-10-09 03:51:46] ...trap/engines/engine.py.321: eval: epoch 30 | batch 430/510 [I 2022-10-09 03:51:46] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:25 [I 2022-10-09 03:51:46] ...trap/engines/engine.py.326: process: 0.32069 | load: 0.00020 [I 2022-10-09 03:51:49] ...trap/engines/engine.py.321: eval: epoch 30 | batch 440/510 [I 2022-10-09 03:51:49] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:22 [I 2022-10-09 03:51:49] ...trap/engines/engine.py.326: process: 0.32294 | load: 0.00021 [I 2022-10-09 03:51:52] ...trap/engines/engine.py.321: eval: epoch 30 | batch 450/510 [I 2022-10-09 03:51:52] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:19 [I 2022-10-09 03:51:52] ...trap/engines/engine.py.326: process: 0.32305 | load: 0.00021 [I 2022-10-09 03:51:55] ...trap/engines/engine.py.321: eval: epoch 30 | batch 460/510 [I 2022-10-09 03:51:55] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:15 [I 2022-10-09 03:51:55] ...trap/engines/engine.py.326: process: 0.31461 | load: 0.00020 [I 2022-10-09 03:51:59] ...trap/engines/engine.py.321: eval: epoch 30 | batch 470/510 [I 2022-10-09 03:51:59] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:12 [I 2022-10-09 03:51:59] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00021 [I 2022-10-09 03:52:02] ...trap/engines/engine.py.321: eval: epoch 30 | batch 480/510 [I 2022-10-09 03:52:02] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:09 [I 2022-10-09 03:52:02] ...trap/engines/engine.py.326: process: 0.31366 | load: 0.00020 [I 2022-10-09 03:52:05] ...trap/engines/engine.py.321: eval: epoch 30 | batch 490/510 [I 2022-10-09 03:52:05] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:06 [I 2022-10-09 03:52:05] ...trap/engines/engine.py.326: process: 0.32027 | load: 0.00020 [I 2022-10-09 03:52:08] ...trap/engines/engine.py.321: eval: epoch 30 | batch 500/510 [I 2022-10-09 03:52:08] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:03 [I 2022-10-09 03:52:08] ...trap/engines/engine.py.326: process: 0.31212 | load: 0.00021 [I 2022-10-09 03:52:11] ...trap/engines/engine.py.321: eval: epoch 30 | batch 510/510 [I 2022-10-09 03:52:11] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:00 [I 2022-10-09 03:52:11] ...trap/engines/engine.py.326: process: 0.31636 | load: 0.00016 [S 2022-10-09 03:52:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 30 [I 2022-10-09 03:52:12] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.691100 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.895200 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931700 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.687000 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.892700 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.929200 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012259 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008577 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008568 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010354 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009166 [S 2022-10-09 03:52:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009064 [I 2022-10-09 03:52:47] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 03:52:47] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 03:52:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 03:52:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 03:52:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 03:52:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 03:52:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 03:52:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 03:52:57] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 03:52:57] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 03:53:00] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 03:53:06] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 03:53:06] ...trap/engines/engine.py.163: Training model on trainset for epoch 31 [I 2022-10-09 03:53:06] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 03:53:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 0/2304 [I 2022-10-09 03:53:09] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:59:50 [I 2022-10-09 03:53:09] ...trap/engines/engine.py.229: process: 3.12099 | load: 2.26223 [I 2022-10-09 03:53:09] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-09 03:53:09] ...optimizers/trijoint.py.151: total_norm: 11.028864 [I 2022-10-09 03:53:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 10/2304 [I 2022-10-09 03:53:17] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:23 [I 2022-10-09 03:53:17] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00549 [I 2022-10-09 03:53:17] ...trap/engines/engine.py.230: loss: 0.19672 [I 2022-10-09 03:53:17] ...optimizers/trijoint.py.151: total_norm: 16.723061 [I 2022-10-09 03:53:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 20/2304 [I 2022-10-09 03:53:26] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:55 [I 2022-10-09 03:53:26] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00447 [I 2022-10-09 03:53:26] ...trap/engines/engine.py.230: loss: 0.13277 [I 2022-10-09 03:53:26] ...optimizers/trijoint.py.151: total_norm: 19.426535 [I 2022-10-09 03:53:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 30/2304 [I 2022-10-09 03:53:34] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:58 [I 2022-10-09 03:53:34] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00413 [I 2022-10-09 03:53:34] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-09 03:53:34] ...optimizers/trijoint.py.151: total_norm: 9.952560 [I 2022-10-09 03:53:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 40/2304 [I 2022-10-09 03:53:42] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:42 [I 2022-10-09 03:53:42] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00403 [I 2022-10-09 03:53:42] ...trap/engines/engine.py.230: loss: 0.24386 [I 2022-10-09 03:53:42] ...optimizers/trijoint.py.151: total_norm: 19.957544 [I 2022-10-09 03:53:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 50/2304 [I 2022-10-09 03:53:51] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:39 [I 2022-10-09 03:53:51] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00372 [I 2022-10-09 03:53:51] ...trap/engines/engine.py.230: loss: 0.16667 [I 2022-10-09 03:53:51] ...optimizers/trijoint.py.151: total_norm: 15.465524 [I 2022-10-09 03:53:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 60/2304 [I 2022-10-09 03:53:59] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:35 [I 2022-10-09 03:53:59] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00348 [I 2022-10-09 03:53:59] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-09 03:53:59] ...optimizers/trijoint.py.151: total_norm: 19.132111 [I 2022-10-09 03:54:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 70/2304 [I 2022-10-09 03:54:08] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:18 [I 2022-10-09 03:54:08] ...trap/engines/engine.py.229: process: 0.83711 | load: 0.00332 [I 2022-10-09 03:54:08] ...trap/engines/engine.py.230: loss: 0.17817 [I 2022-10-09 03:54:08] ...optimizers/trijoint.py.151: total_norm: 23.032393 [I 2022-10-09 03:54:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 80/2304 [I 2022-10-09 03:54:16] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:13 [I 2022-10-09 03:54:16] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00374 [I 2022-10-09 03:54:16] ...trap/engines/engine.py.230: loss: 0.14460 [I 2022-10-09 03:54:16] ...optimizers/trijoint.py.151: total_norm: 11.277013 [I 2022-10-09 03:54:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 90/2304 [I 2022-10-09 03:54:25] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:04 [I 2022-10-09 03:54:25] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00352 [I 2022-10-09 03:54:25] ...trap/engines/engine.py.230: loss: 0.12666 [I 2022-10-09 03:54:25] ...optimizers/trijoint.py.151: total_norm: 46.964947 [I 2022-10-09 03:54:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 100/2304 [I 2022-10-09 03:54:33] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:49 [I 2022-10-09 03:54:33] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00336 [I 2022-10-09 03:54:33] ...trap/engines/engine.py.230: loss: 0.22198 [I 2022-10-09 03:54:33] ...optimizers/trijoint.py.151: total_norm: 24.532295 [I 2022-10-09 03:54:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 110/2304 [I 2022-10-09 03:54:42] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:31:03 [I 2022-10-09 03:54:42] ...trap/engines/engine.py.229: process: 0.84892 | load: 0.00311 [I 2022-10-09 03:54:42] ...trap/engines/engine.py.230: loss: 0.16835 [I 2022-10-09 03:54:42] ...optimizers/trijoint.py.151: total_norm: 20.138226 [I 2022-10-09 03:54:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 120/2304 [I 2022-10-09 03:54:50] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:46 [I 2022-10-09 03:54:50] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00329 [I 2022-10-09 03:54:50] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-09 03:54:50] ...optimizers/trijoint.py.151: total_norm: 13.538002 [I 2022-10-09 03:54:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 130/2304 [I 2022-10-09 03:54:59] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:26 [I 2022-10-09 03:54:59] ...trap/engines/engine.py.229: process: 0.83496 | load: 0.00318 [I 2022-10-09 03:54:59] ...trap/engines/engine.py.230: loss: 0.18380 [I 2022-10-09 03:54:59] ...optimizers/trijoint.py.151: total_norm: 20.650560 [I 2022-10-09 03:55:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 140/2304 [I 2022-10-09 03:55:07] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:18 [I 2022-10-09 03:55:07] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00298 [I 2022-10-09 03:55:07] ...trap/engines/engine.py.230: loss: 0.12398 [I 2022-10-09 03:55:07] ...optimizers/trijoint.py.151: total_norm: 14.466271 [I 2022-10-09 03:55:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 150/2304 [I 2022-10-09 03:55:15] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:13 [I 2022-10-09 03:55:15] ...trap/engines/engine.py.229: process: 0.83511 | load: 0.00308 [I 2022-10-09 03:55:15] ...trap/engines/engine.py.230: loss: 0.10969 [I 2022-10-09 03:55:15] ...optimizers/trijoint.py.151: total_norm: 11.052580 [I 2022-10-09 03:55:24] ...trap/engines/engine.py.224: train: epoch 31 | batch 160/2304 [I 2022-10-09 03:55:24] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:08 [I 2022-10-09 03:55:24] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00315 [I 2022-10-09 03:55:24] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-09 03:55:24] ...optimizers/trijoint.py.151: total_norm: 23.216959 [I 2022-10-09 03:55:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 170/2304 [I 2022-10-09 03:55:32] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:55 [I 2022-10-09 03:55:32] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00312 [I 2022-10-09 03:55:32] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-09 03:55:32] ...optimizers/trijoint.py.151: total_norm: 13.666976 [I 2022-10-09 03:55:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 180/2304 [I 2022-10-09 03:55:41] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:42 [I 2022-10-09 03:55:41] ...trap/engines/engine.py.229: process: 0.83003 | load: 0.00298 [I 2022-10-09 03:55:41] ...trap/engines/engine.py.230: loss: 0.25559 [I 2022-10-09 03:55:41] ...optimizers/trijoint.py.151: total_norm: 19.527952 [I 2022-10-09 03:55:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 190/2304 [I 2022-10-09 03:55:49] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:43 [I 2022-10-09 03:55:49] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00309 [I 2022-10-09 03:55:49] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-09 03:55:49] ...optimizers/trijoint.py.151: total_norm: 12.853251 [I 2022-10-09 03:55:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 200/2304 [I 2022-10-09 03:55:58] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:26 [I 2022-10-09 03:55:58] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00312 [I 2022-10-09 03:55:58] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-09 03:55:58] ...optimizers/trijoint.py.151: total_norm: 17.906748 [I 2022-10-09 03:56:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 210/2304 [I 2022-10-09 03:56:06] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:18 [I 2022-10-09 03:56:06] ...trap/engines/engine.py.229: process: 0.82865 | load: 0.00297 [I 2022-10-09 03:56:06] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-09 03:56:06] ...optimizers/trijoint.py.151: total_norm: 12.661519 [I 2022-10-09 03:56:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 220/2304 [I 2022-10-09 03:56:14] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:28:58 [I 2022-10-09 03:56:14] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00298 [I 2022-10-09 03:56:14] ...trap/engines/engine.py.230: loss: 0.25718 [I 2022-10-09 03:56:14] ...optimizers/trijoint.py.151: total_norm: 22.015390 [I 2022-10-09 03:56:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 230/2304 [I 2022-10-09 03:56:23] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:00 [I 2022-10-09 03:56:23] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00313 [I 2022-10-09 03:56:23] ...trap/engines/engine.py.230: loss: 0.27342 [I 2022-10-09 03:56:23] ...optimizers/trijoint.py.151: total_norm: 32.507233 [I 2022-10-09 03:56:31] ...trap/engines/engine.py.224: train: epoch 31 | batch 240/2304 [I 2022-10-09 03:56:31] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:45 [I 2022-10-09 03:56:31] ...trap/engines/engine.py.229: process: 0.83425 | load: 0.00304 [I 2022-10-09 03:56:31] ...trap/engines/engine.py.230: loss: 0.15991 [I 2022-10-09 03:56:31] ...optimizers/trijoint.py.151: total_norm: 15.148011 [I 2022-10-09 03:56:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 250/2304 [I 2022-10-09 03:56:40] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:48 [I 2022-10-09 03:56:40] ...trap/engines/engine.py.229: process: 0.83724 | load: 0.00320 [I 2022-10-09 03:56:40] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-09 03:56:40] ...optimizers/trijoint.py.151: total_norm: 16.245592 [I 2022-10-09 03:56:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 260/2304 [I 2022-10-09 03:56:48] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:37 [I 2022-10-09 03:56:48] ...trap/engines/engine.py.229: process: 0.85090 | load: 0.00298 [I 2022-10-09 03:56:48] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-09 03:56:48] ...optimizers/trijoint.py.151: total_norm: 22.349279 [I 2022-10-09 03:56:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 270/2304 [I 2022-10-09 03:56:56] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:30 [I 2022-10-09 03:56:56] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00302 [I 2022-10-09 03:56:56] ...trap/engines/engine.py.230: loss: 0.23065 [I 2022-10-09 03:56:56] ...optimizers/trijoint.py.151: total_norm: 16.367579 [I 2022-10-09 03:57:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 280/2304 [I 2022-10-09 03:57:05] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:14 [I 2022-10-09 03:57:05] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00309 [I 2022-10-09 03:57:05] ...trap/engines/engine.py.230: loss: 0.15917 [I 2022-10-09 03:57:05] ...optimizers/trijoint.py.151: total_norm: 19.920483 [I 2022-10-09 03:57:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 290/2304 [I 2022-10-09 03:57:13] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:15 [I 2022-10-09 03:57:13] ...trap/engines/engine.py.229: process: 0.83968 | load: 0.00301 [I 2022-10-09 03:57:13] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-09 03:57:13] ...optimizers/trijoint.py.151: total_norm: 15.718728 [I 2022-10-09 03:57:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 300/2304 [I 2022-10-09 03:57:22] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:08 [I 2022-10-09 03:57:22] ...trap/engines/engine.py.229: process: 0.85961 | load: 0.00321 [I 2022-10-09 03:57:22] ...trap/engines/engine.py.230: loss: 0.21450 [I 2022-10-09 03:57:22] ...optimizers/trijoint.py.151: total_norm: 14.291237 [I 2022-10-09 03:57:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 310/2304 [I 2022-10-09 03:57:30] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:04 [I 2022-10-09 03:57:30] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00305 [I 2022-10-09 03:57:30] ...trap/engines/engine.py.230: loss: 0.15161 [I 2022-10-09 03:57:30] ...optimizers/trijoint.py.151: total_norm: 19.754635 [I 2022-10-09 03:57:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 320/2304 [I 2022-10-09 03:57:38] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:47 [I 2022-10-09 03:57:38] ...trap/engines/engine.py.229: process: 0.83470 | load: 0.00303 [I 2022-10-09 03:57:38] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-09 03:57:38] ...optimizers/trijoint.py.151: total_norm: 10.198795 [I 2022-10-09 03:57:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 330/2304 [I 2022-10-09 03:57:47] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:32 [I 2022-10-09 03:57:47] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00299 [I 2022-10-09 03:57:47] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-09 03:57:47] ...optimizers/trijoint.py.151: total_norm: 11.832161 [I 2022-10-09 03:57:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 340/2304 [I 2022-10-09 03:57:55] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:18 [I 2022-10-09 03:57:55] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00311 [I 2022-10-09 03:57:55] ...trap/engines/engine.py.230: loss: 0.19049 [I 2022-10-09 03:57:55] ...optimizers/trijoint.py.151: total_norm: 22.709244 [I 2022-10-09 03:58:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 350/2304 [I 2022-10-09 03:58:04] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:13 [I 2022-10-09 03:58:04] ...trap/engines/engine.py.229: process: 0.84060 | load: 0.00331 [I 2022-10-09 03:58:04] ...trap/engines/engine.py.230: loss: 0.17615 [I 2022-10-09 03:58:04] ...optimizers/trijoint.py.151: total_norm: 12.536626 [I 2022-10-09 03:58:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 360/2304 [I 2022-10-09 03:58:12] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:03 [I 2022-10-09 03:58:12] ...trap/engines/engine.py.229: process: 0.83431 | load: 0.00301 [I 2022-10-09 03:58:12] ...trap/engines/engine.py.230: loss: 0.13712 [I 2022-10-09 03:58:12] ...optimizers/trijoint.py.151: total_norm: 15.411533 [I 2022-10-09 03:58:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 370/2304 [I 2022-10-09 03:58:20] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:26:56 [I 2022-10-09 03:58:20] ...trap/engines/engine.py.229: process: 0.82475 | load: 0.00318 [I 2022-10-09 03:58:20] ...trap/engines/engine.py.230: loss: 0.17092 [I 2022-10-09 03:58:20] ...optimizers/trijoint.py.151: total_norm: 11.990682 [I 2022-10-09 03:58:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 380/2304 [I 2022-10-09 03:58:29] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:49 [I 2022-10-09 03:58:29] ...trap/engines/engine.py.229: process: 0.83673 | load: 0.00302 [I 2022-10-09 03:58:29] ...trap/engines/engine.py.230: loss: 0.17476 [I 2022-10-09 03:58:29] ...optimizers/trijoint.py.151: total_norm: 11.468403 [I 2022-10-09 03:58:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 390/2304 [I 2022-10-09 03:58:37] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:42 [I 2022-10-09 03:58:37] ...trap/engines/engine.py.229: process: 0.82262 | load: 0.00300 [I 2022-10-09 03:58:37] ...trap/engines/engine.py.230: loss: 0.15799 [I 2022-10-09 03:58:37] ...optimizers/trijoint.py.151: total_norm: 25.176876 [I 2022-10-09 03:58:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 400/2304 [I 2022-10-09 03:58:46] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:33 [I 2022-10-09 03:58:46] ...trap/engines/engine.py.229: process: 0.83481 | load: 0.00300 [I 2022-10-09 03:58:46] ...trap/engines/engine.py.230: loss: 0.12073 [I 2022-10-09 03:58:46] ...optimizers/trijoint.py.151: total_norm: 11.639946 [I 2022-10-09 03:58:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 410/2304 [I 2022-10-09 03:58:54] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:20 [I 2022-10-09 03:58:54] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00300 [I 2022-10-09 03:58:54] ...trap/engines/engine.py.230: loss: 0.21628 [I 2022-10-09 03:58:54] ...optimizers/trijoint.py.151: total_norm: 36.030674 [I 2022-10-09 03:59:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 420/2304 [I 2022-10-09 03:59:02] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:17 [I 2022-10-09 03:59:02] ...trap/engines/engine.py.229: process: 0.83642 | load: 0.00300 [I 2022-10-09 03:59:02] ...trap/engines/engine.py.230: loss: 0.20930 [I 2022-10-09 03:59:02] ...optimizers/trijoint.py.151: total_norm: 31.090849 [I 2022-10-09 03:59:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 430/2304 [I 2022-10-09 03:59:11] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:00 [I 2022-10-09 03:59:11] ...trap/engines/engine.py.229: process: 0.83130 | load: 0.00303 [I 2022-10-09 03:59:11] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-09 03:59:11] ...optimizers/trijoint.py.151: total_norm: 17.227314 [I 2022-10-09 03:59:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 440/2304 [I 2022-10-09 03:59:19] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:04 [I 2022-10-09 03:59:19] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00300 [I 2022-10-09 03:59:19] ...trap/engines/engine.py.230: loss: 0.21569 [I 2022-10-09 03:59:19] ...optimizers/trijoint.py.151: total_norm: 20.695154 [I 2022-10-09 03:59:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 450/2304 [I 2022-10-09 03:59:27] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:25:57 [I 2022-10-09 03:59:27] ...trap/engines/engine.py.229: process: 0.83474 | load: 0.00301 [I 2022-10-09 03:59:27] ...trap/engines/engine.py.230: loss: 0.08365 [I 2022-10-09 03:59:27] ...optimizers/trijoint.py.151: total_norm: 7.576144 [I 2022-10-09 03:59:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 460/2304 [I 2022-10-09 03:59:36] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:54 [I 2022-10-09 03:59:36] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00300 [I 2022-10-09 03:59:36] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-09 03:59:36] ...optimizers/trijoint.py.151: total_norm: 17.209663 [I 2022-10-09 03:59:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 470/2304 [I 2022-10-09 03:59:44] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:38 [I 2022-10-09 03:59:44] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00302 [I 2022-10-09 03:59:44] ...trap/engines/engine.py.230: loss: 0.22205 [I 2022-10-09 03:59:44] ...optimizers/trijoint.py.151: total_norm: 17.764038 [I 2022-10-09 03:59:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 480/2304 [I 2022-10-09 03:59:53] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:25:34 [I 2022-10-09 03:59:53] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00311 [I 2022-10-09 03:59:53] ...trap/engines/engine.py.230: loss: 0.17599 [I 2022-10-09 03:59:53] ...optimizers/trijoint.py.151: total_norm: 20.267054 [I 2022-10-09 04:00:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 490/2304 [I 2022-10-09 04:00:01] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:26 [I 2022-10-09 04:00:01] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00303 [I 2022-10-09 04:00:01] ...trap/engines/engine.py.230: loss: 0.14957 [I 2022-10-09 04:00:01] ...optimizers/trijoint.py.151: total_norm: 22.885077 [I 2022-10-09 04:00:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 500/2304 [I 2022-10-09 04:00:10] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:25:06 [I 2022-10-09 04:00:10] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00298 [I 2022-10-09 04:00:10] ...trap/engines/engine.py.230: loss: 0.29728 [I 2022-10-09 04:00:10] ...optimizers/trijoint.py.151: total_norm: 24.228912 [I 2022-10-09 04:00:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 510/2304 [I 2022-10-09 04:00:18] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:06 [I 2022-10-09 04:00:18] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00300 [I 2022-10-09 04:00:18] ...trap/engines/engine.py.230: loss: 0.15513 [I 2022-10-09 04:00:18] ...optimizers/trijoint.py.151: total_norm: 14.343903 [I 2022-10-09 04:00:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 520/2304 [I 2022-10-09 04:00:26] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:24:51 [I 2022-10-09 04:00:26] ...trap/engines/engine.py.229: process: 0.83772 | load: 0.00297 [I 2022-10-09 04:00:26] ...trap/engines/engine.py.230: loss: 0.19580 [I 2022-10-09 04:00:26] ...optimizers/trijoint.py.151: total_norm: 23.503023 [I 2022-10-09 04:00:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 530/2304 [I 2022-10-09 04:00:35] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:24:56 [I 2022-10-09 04:00:35] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00299 [I 2022-10-09 04:00:35] ...trap/engines/engine.py.230: loss: 0.16105 [I 2022-10-09 04:00:35] ...optimizers/trijoint.py.151: total_norm: 14.734831 [I 2022-10-09 04:00:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 540/2304 [I 2022-10-09 04:00:43] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:40 [I 2022-10-09 04:00:43] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00299 [I 2022-10-09 04:00:43] ...trap/engines/engine.py.230: loss: 0.18006 [I 2022-10-09 04:00:43] ...optimizers/trijoint.py.151: total_norm: 20.085262 [I 2022-10-09 04:00:52] ...trap/engines/engine.py.224: train: epoch 31 | batch 550/2304 [I 2022-10-09 04:00:52] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:24:30 [I 2022-10-09 04:00:52] ...trap/engines/engine.py.229: process: 0.83202 | load: 0.00300 [I 2022-10-09 04:00:52] ...trap/engines/engine.py.230: loss: 0.14443 [I 2022-10-09 04:00:52] ...optimizers/trijoint.py.151: total_norm: 13.377589 [I 2022-10-09 04:01:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 560/2304 [I 2022-10-09 04:01:00] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:25 [I 2022-10-09 04:01:00] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00297 [I 2022-10-09 04:01:00] ...trap/engines/engine.py.230: loss: 0.22383 [I 2022-10-09 04:01:00] ...optimizers/trijoint.py.151: total_norm: 17.980101 [I 2022-10-09 04:01:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 570/2304 [I 2022-10-09 04:01:08] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:13 [I 2022-10-09 04:01:08] ...trap/engines/engine.py.229: process: 0.84804 | load: 0.00301 [I 2022-10-09 04:01:08] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-09 04:01:08] ...optimizers/trijoint.py.151: total_norm: 14.744481 [I 2022-10-09 04:01:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 580/2304 [I 2022-10-09 04:01:17] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:12 [I 2022-10-09 04:01:17] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00301 [I 2022-10-09 04:01:17] ...trap/engines/engine.py.230: loss: 0.14765 [I 2022-10-09 04:01:17] ...optimizers/trijoint.py.151: total_norm: 14.493888 [I 2022-10-09 04:01:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 590/2304 [I 2022-10-09 04:01:25] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:23:58 [I 2022-10-09 04:01:25] ...trap/engines/engine.py.229: process: 0.83330 | load: 0.00299 [I 2022-10-09 04:01:25] ...trap/engines/engine.py.230: loss: 0.19884 [I 2022-10-09 04:01:25] ...optimizers/trijoint.py.151: total_norm: 10.660329 [I 2022-10-09 04:01:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 600/2304 [I 2022-10-09 04:01:34] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:23:50 [I 2022-10-09 04:01:34] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00301 [I 2022-10-09 04:01:34] ...trap/engines/engine.py.230: loss: 0.17168 [I 2022-10-09 04:01:34] ...optimizers/trijoint.py.151: total_norm: 20.804962 [I 2022-10-09 04:01:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 610/2304 [I 2022-10-09 04:01:42] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:23:46 [I 2022-10-09 04:01:42] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00307 [I 2022-10-09 04:01:42] ...trap/engines/engine.py.230: loss: 0.23914 [I 2022-10-09 04:01:42] ...optimizers/trijoint.py.151: total_norm: 24.560495 [I 2022-10-09 04:01:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 620/2304 [I 2022-10-09 04:01:50] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:35 [I 2022-10-09 04:01:50] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00303 [I 2022-10-09 04:01:50] ...trap/engines/engine.py.230: loss: 0.13162 [I 2022-10-09 04:01:50] ...optimizers/trijoint.py.151: total_norm: 15.556292 [I 2022-10-09 04:01:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 630/2304 [I 2022-10-09 04:01:59] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:25 [I 2022-10-09 04:01:59] ...trap/engines/engine.py.229: process: 0.84045 | load: 0.00300 [I 2022-10-09 04:01:59] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-09 04:01:59] ...optimizers/trijoint.py.151: total_norm: 22.108988 [I 2022-10-09 04:02:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 640/2304 [I 2022-10-09 04:02:07] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:16 [I 2022-10-09 04:02:07] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00297 [I 2022-10-09 04:02:07] ...trap/engines/engine.py.230: loss: 0.10712 [I 2022-10-09 04:02:07] ...optimizers/trijoint.py.151: total_norm: 18.815733 [I 2022-10-09 04:02:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 650/2304 [I 2022-10-09 04:02:16] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:07 [I 2022-10-09 04:02:16] ...trap/engines/engine.py.229: process: 0.83366 | load: 0.00298 [I 2022-10-09 04:02:16] ...trap/engines/engine.py.230: loss: 0.11049 [I 2022-10-09 04:02:16] ...optimizers/trijoint.py.151: total_norm: 15.602262 [I 2022-10-09 04:02:24] ...trap/engines/engine.py.224: train: epoch 31 | batch 660/2304 [I 2022-10-09 04:02:24] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:03 [I 2022-10-09 04:02:24] ...trap/engines/engine.py.229: process: 0.83828 | load: 0.00298 [I 2022-10-09 04:02:24] ...trap/engines/engine.py.230: loss: 0.13467 [I 2022-10-09 04:02:24] ...optimizers/trijoint.py.151: total_norm: 14.978997 [I 2022-10-09 04:02:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 670/2304 [I 2022-10-09 04:02:33] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:22:50 [I 2022-10-09 04:02:33] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00300 [I 2022-10-09 04:02:33] ...trap/engines/engine.py.230: loss: 0.19688 [I 2022-10-09 04:02:33] ...optimizers/trijoint.py.151: total_norm: 20.250811 [I 2022-10-09 04:02:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 680/2304 [I 2022-10-09 04:02:41] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:42 [I 2022-10-09 04:02:41] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00302 [I 2022-10-09 04:02:41] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-09 04:02:41] ...optimizers/trijoint.py.151: total_norm: 17.779581 [I 2022-10-09 04:02:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 690/2304 [I 2022-10-09 04:02:49] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:22:33 [I 2022-10-09 04:02:49] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00300 [I 2022-10-09 04:02:49] ...trap/engines/engine.py.230: loss: 0.21743 [I 2022-10-09 04:02:49] ...optimizers/trijoint.py.151: total_norm: 24.715254 [I 2022-10-09 04:02:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 700/2304 [I 2022-10-09 04:02:58] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:16 [I 2022-10-09 04:02:58] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00300 [I 2022-10-09 04:02:58] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-09 04:02:58] ...optimizers/trijoint.py.151: total_norm: 12.648946 [I 2022-10-09 04:03:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 710/2304 [I 2022-10-09 04:03:06] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:15 [I 2022-10-09 04:03:06] ...trap/engines/engine.py.229: process: 0.83183 | load: 0.00304 [I 2022-10-09 04:03:06] ...trap/engines/engine.py.230: loss: 0.14754 [I 2022-10-09 04:03:06] ...optimizers/trijoint.py.151: total_norm: 10.091505 [I 2022-10-09 04:03:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 720/2304 [I 2022-10-09 04:03:14] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:04 [I 2022-10-09 04:03:14] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00301 [I 2022-10-09 04:03:14] ...trap/engines/engine.py.230: loss: 0.21340 [I 2022-10-09 04:03:14] ...optimizers/trijoint.py.151: total_norm: 24.500389 [I 2022-10-09 04:03:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 730/2304 [I 2022-10-09 04:03:23] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:07 [I 2022-10-09 04:03:23] ...trap/engines/engine.py.229: process: 0.85433 | load: 0.00310 [I 2022-10-09 04:03:23] ...trap/engines/engine.py.230: loss: 0.16861 [I 2022-10-09 04:03:23] ...optimizers/trijoint.py.151: total_norm: 18.346630 [I 2022-10-09 04:03:31] ...trap/engines/engine.py.224: train: epoch 31 | batch 740/2304 [I 2022-10-09 04:03:31] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:21:59 [I 2022-10-09 04:03:31] ...trap/engines/engine.py.229: process: 0.84889 | load: 0.00304 [I 2022-10-09 04:03:31] ...trap/engines/engine.py.230: loss: 0.19453 [I 2022-10-09 04:03:31] ...optimizers/trijoint.py.151: total_norm: 14.964762 [I 2022-10-09 04:03:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 750/2304 [I 2022-10-09 04:03:40] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:51 [I 2022-10-09 04:03:40] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00308 [I 2022-10-09 04:03:40] ...trap/engines/engine.py.230: loss: 0.17814 [I 2022-10-09 04:03:40] ...optimizers/trijoint.py.151: total_norm: 16.087646 [I 2022-10-09 04:03:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 760/2304 [I 2022-10-09 04:03:48] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:44 [I 2022-10-09 04:03:48] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00299 [I 2022-10-09 04:03:48] ...trap/engines/engine.py.230: loss: 0.16620 [I 2022-10-09 04:03:48] ...optimizers/trijoint.py.151: total_norm: 11.848855 [I 2022-10-09 04:03:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 770/2304 [I 2022-10-09 04:03:57] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:31 [I 2022-10-09 04:03:57] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00306 [I 2022-10-09 04:03:57] ...trap/engines/engine.py.230: loss: 0.26249 [I 2022-10-09 04:03:57] ...optimizers/trijoint.py.151: total_norm: 20.588863 [I 2022-10-09 04:04:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 780/2304 [I 2022-10-09 04:04:05] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:13 [I 2022-10-09 04:04:05] ...trap/engines/engine.py.229: process: 0.84468 | load: 0.00299 [I 2022-10-09 04:04:05] ...trap/engines/engine.py.230: loss: 0.26502 [I 2022-10-09 04:04:05] ...optimizers/trijoint.py.151: total_norm: 13.286652 [I 2022-10-09 04:04:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 790/2304 [I 2022-10-09 04:04:13] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:09 [I 2022-10-09 04:04:13] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00300 [I 2022-10-09 04:04:13] ...trap/engines/engine.py.230: loss: 0.17064 [I 2022-10-09 04:04:13] ...optimizers/trijoint.py.151: total_norm: 13.288509 [I 2022-10-09 04:04:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 800/2304 [I 2022-10-09 04:04:22] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:00 [I 2022-10-09 04:04:22] ...trap/engines/engine.py.229: process: 0.83200 | load: 0.00301 [I 2022-10-09 04:04:22] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-09 04:04:22] ...optimizers/trijoint.py.151: total_norm: 20.927742 [I 2022-10-09 04:04:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 810/2304 [I 2022-10-09 04:04:30] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:20:53 [I 2022-10-09 04:04:30] ...trap/engines/engine.py.229: process: 0.83387 | load: 0.00300 [I 2022-10-09 04:04:30] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-09 04:04:30] ...optimizers/trijoint.py.151: total_norm: 8.472761 [I 2022-10-09 04:04:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 820/2304 [I 2022-10-09 04:04:39] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:20:42 [I 2022-10-09 04:04:39] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00301 [I 2022-10-09 04:04:39] ...trap/engines/engine.py.230: loss: 0.15575 [I 2022-10-09 04:04:39] ...optimizers/trijoint.py.151: total_norm: 17.300203 [I 2022-10-09 04:04:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 830/2304 [I 2022-10-09 04:04:47] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:39 [I 2022-10-09 04:04:47] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00301 [I 2022-10-09 04:04:47] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-09 04:04:47] ...optimizers/trijoint.py.151: total_norm: 43.599232 [I 2022-10-09 04:04:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 840/2304 [I 2022-10-09 04:04:56] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:30 [I 2022-10-09 04:04:56] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00302 [I 2022-10-09 04:04:56] ...trap/engines/engine.py.230: loss: 0.19609 [I 2022-10-09 04:04:56] ...optimizers/trijoint.py.151: total_norm: 25.806093 [I 2022-10-09 04:05:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 850/2304 [I 2022-10-09 04:05:04] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:21 [I 2022-10-09 04:05:04] ...trap/engines/engine.py.229: process: 0.83578 | load: 0.00303 [I 2022-10-09 04:05:04] ...trap/engines/engine.py.230: loss: 0.14186 [I 2022-10-09 04:05:04] ...optimizers/trijoint.py.151: total_norm: 19.209869 [I 2022-10-09 04:05:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 860/2304 [I 2022-10-09 04:05:12] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:10 [I 2022-10-09 04:05:12] ...trap/engines/engine.py.229: process: 0.83240 | load: 0.00299 [I 2022-10-09 04:05:12] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-09 04:05:12] ...optimizers/trijoint.py.151: total_norm: 19.726269 [I 2022-10-09 04:05:21] ...trap/engines/engine.py.224: train: epoch 31 | batch 870/2304 [I 2022-10-09 04:05:21] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:20:01 [I 2022-10-09 04:05:21] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00299 [I 2022-10-09 04:05:21] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-09 04:05:21] ...optimizers/trijoint.py.151: total_norm: 14.139235 [I 2022-10-09 04:05:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 880/2304 [I 2022-10-09 04:05:29] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:19:55 [I 2022-10-09 04:05:29] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00299 [I 2022-10-09 04:05:29] ...trap/engines/engine.py.230: loss: 0.13241 [I 2022-10-09 04:05:29] ...optimizers/trijoint.py.151: total_norm: 15.786368 [I 2022-10-09 04:05:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 890/2304 [I 2022-10-09 04:05:37] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:19:44 [I 2022-10-09 04:05:37] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00297 [I 2022-10-09 04:05:37] ...trap/engines/engine.py.230: loss: 0.21442 [I 2022-10-09 04:05:37] ...optimizers/trijoint.py.151: total_norm: 24.138149 [I 2022-10-09 04:05:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 900/2304 [I 2022-10-09 04:05:46] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:19:38 [I 2022-10-09 04:05:46] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00300 [I 2022-10-09 04:05:46] ...trap/engines/engine.py.230: loss: 0.10940 [I 2022-10-09 04:05:46] ...optimizers/trijoint.py.151: total_norm: 9.345187 [I 2022-10-09 04:05:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 910/2304 [I 2022-10-09 04:05:54] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:19:31 [I 2022-10-09 04:05:54] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00300 [I 2022-10-09 04:05:54] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-09 04:05:54] ...optimizers/trijoint.py.151: total_norm: 9.945907 [I 2022-10-09 04:06:03] ...trap/engines/engine.py.224: train: epoch 31 | batch 920/2304 [I 2022-10-09 04:06:03] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:19:13 [I 2022-10-09 04:06:03] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00302 [I 2022-10-09 04:06:03] ...trap/engines/engine.py.230: loss: 0.09915 [I 2022-10-09 04:06:03] ...optimizers/trijoint.py.151: total_norm: 11.082231 [I 2022-10-09 04:06:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 930/2304 [I 2022-10-09 04:06:11] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:19:13 [I 2022-10-09 04:06:11] ...trap/engines/engine.py.229: process: 0.84702 | load: 0.00297 [I 2022-10-09 04:06:11] ...trap/engines/engine.py.230: loss: 0.13747 [I 2022-10-09 04:06:11] ...optimizers/trijoint.py.151: total_norm: 20.565947 [I 2022-10-09 04:06:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 940/2304 [I 2022-10-09 04:06:19] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:19:03 [I 2022-10-09 04:06:19] ...trap/engines/engine.py.229: process: 0.83557 | load: 0.00299 [I 2022-10-09 04:06:19] ...trap/engines/engine.py.230: loss: 0.15528 [I 2022-10-09 04:06:19] ...optimizers/trijoint.py.151: total_norm: 19.062017 [I 2022-10-09 04:06:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 950/2304 [I 2022-10-09 04:06:28] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:18:51 [I 2022-10-09 04:06:28] ...trap/engines/engine.py.229: process: 0.82896 | load: 0.00299 [I 2022-10-09 04:06:28] ...trap/engines/engine.py.230: loss: 0.17344 [I 2022-10-09 04:06:28] ...optimizers/trijoint.py.151: total_norm: 19.275782 [I 2022-10-09 04:06:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 960/2304 [I 2022-10-09 04:06:36] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:18:46 [I 2022-10-09 04:06:36] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00301 [I 2022-10-09 04:06:36] ...trap/engines/engine.py.230: loss: 0.15994 [I 2022-10-09 04:06:36] ...optimizers/trijoint.py.151: total_norm: 11.707928 [I 2022-10-09 04:06:45] ...trap/engines/engine.py.224: train: epoch 31 | batch 970/2304 [I 2022-10-09 04:06:45] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:18:36 [I 2022-10-09 04:06:45] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00302 [I 2022-10-09 04:06:45] ...trap/engines/engine.py.230: loss: 0.13699 [I 2022-10-09 04:06:45] ...optimizers/trijoint.py.151: total_norm: 28.807154 [I 2022-10-09 04:06:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 980/2304 [I 2022-10-09 04:06:53] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:18:31 [I 2022-10-09 04:06:53] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00302 [I 2022-10-09 04:06:53] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-09 04:06:53] ...optimizers/trijoint.py.151: total_norm: 10.995428 [I 2022-10-09 04:07:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 990/2304 [I 2022-10-09 04:07:01] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:18:18 [I 2022-10-09 04:07:01] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00300 [I 2022-10-09 04:07:01] ...trap/engines/engine.py.230: loss: 0.22259 [I 2022-10-09 04:07:01] ...optimizers/trijoint.py.151: total_norm: 12.572983 [I 2022-10-09 04:07:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 1000/2304 [I 2022-10-09 04:07:10] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:16 [I 2022-10-09 04:07:10] ...trap/engines/engine.py.229: process: 0.84054 | load: 0.00300 [I 2022-10-09 04:07:10] ...trap/engines/engine.py.230: loss: 0.25039 [I 2022-10-09 04:07:10] ...optimizers/trijoint.py.151: total_norm: 37.403442 [I 2022-10-09 04:07:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1010/2304 [I 2022-10-09 04:07:18] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:18:08 [I 2022-10-09 04:07:18] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00299 [I 2022-10-09 04:07:18] ...trap/engines/engine.py.230: loss: 0.15331 [I 2022-10-09 04:07:18] ...optimizers/trijoint.py.151: total_norm: 13.848104 [I 2022-10-09 04:07:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 1020/2304 [I 2022-10-09 04:07:27] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:17:59 [I 2022-10-09 04:07:27] ...trap/engines/engine.py.229: process: 0.82433 | load: 0.00300 [I 2022-10-09 04:07:27] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-09 04:07:27] ...optimizers/trijoint.py.151: total_norm: 14.798924 [I 2022-10-09 04:07:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 1030/2304 [I 2022-10-09 04:07:53] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:57:23 [I 2022-10-09 04:07:53] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00302 [I 2022-10-09 04:07:53] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-09 04:07:53] ...optimizers/trijoint.py.151: total_norm: 15.512347 [I 2022-10-09 04:08:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 1040/2304 [I 2022-10-09 04:08:02] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:21:52 [I 2022-10-09 04:08:02] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00302 [I 2022-10-09 04:08:02] ...trap/engines/engine.py.230: loss: 0.17623 [I 2022-10-09 04:08:02] ...optimizers/trijoint.py.151: total_norm: 18.500490 [I 2022-10-09 04:08:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 1050/2304 [I 2022-10-09 04:08:10] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:55 [I 2022-10-09 04:08:10] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00299 [I 2022-10-09 04:08:10] ...trap/engines/engine.py.230: loss: 0.14165 [I 2022-10-09 04:08:10] ...optimizers/trijoint.py.151: total_norm: 14.659683 [I 2022-10-09 04:08:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1060/2304 [I 2022-10-09 04:08:18] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:23 [I 2022-10-09 04:08:18] ...trap/engines/engine.py.229: process: 0.83514 | load: 0.00299 [I 2022-10-09 04:08:18] ...trap/engines/engine.py.230: loss: 0.20265 [I 2022-10-09 04:08:18] ...optimizers/trijoint.py.151: total_norm: 13.764457 [I 2022-10-09 04:08:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 1070/2304 [I 2022-10-09 04:08:27] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:13 [I 2022-10-09 04:08:27] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00300 [I 2022-10-09 04:08:27] ...trap/engines/engine.py.230: loss: 0.15425 [I 2022-10-09 04:08:27] ...optimizers/trijoint.py.151: total_norm: 15.653835 [I 2022-10-09 04:08:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 1080/2304 [I 2022-10-09 04:08:35] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:17:03 [I 2022-10-09 04:08:35] ...trap/engines/engine.py.229: process: 0.82796 | load: 0.00331 [I 2022-10-09 04:08:35] ...trap/engines/engine.py.230: loss: 0.18780 [I 2022-10-09 04:08:35] ...optimizers/trijoint.py.151: total_norm: 15.096072 [I 2022-10-09 04:08:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 1090/2304 [I 2022-10-09 04:08:44] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:16:54 [I 2022-10-09 04:08:44] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00305 [I 2022-10-09 04:08:44] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-09 04:08:44] ...optimizers/trijoint.py.151: total_norm: 14.092667 [I 2022-10-09 04:08:52] ...trap/engines/engine.py.224: train: epoch 31 | batch 1100/2304 [I 2022-10-09 04:08:52] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:16:46 [I 2022-10-09 04:08:52] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00304 [I 2022-10-09 04:08:52] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-09 04:08:52] ...optimizers/trijoint.py.151: total_norm: 19.784939 [I 2022-10-09 04:09:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 1110/2304 [I 2022-10-09 04:09:00] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:16:43 [I 2022-10-09 04:09:00] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00305 [I 2022-10-09 04:09:00] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-09 04:09:00] ...optimizers/trijoint.py.151: total_norm: 22.115253 [I 2022-10-09 04:09:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 1120/2304 [I 2022-10-09 04:09:09] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:28 [I 2022-10-09 04:09:09] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00300 [I 2022-10-09 04:09:09] ...trap/engines/engine.py.230: loss: 0.15112 [I 2022-10-09 04:09:09] ...optimizers/trijoint.py.151: total_norm: 12.323025 [I 2022-10-09 04:09:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 1130/2304 [I 2022-10-09 04:09:17] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:24 [I 2022-10-09 04:09:17] ...trap/engines/engine.py.229: process: 0.83744 | load: 0.00303 [I 2022-10-09 04:09:17] ...trap/engines/engine.py.230: loss: 0.16881 [I 2022-10-09 04:09:17] ...optimizers/trijoint.py.151: total_norm: 12.622866 [I 2022-10-09 04:09:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 1140/2304 [I 2022-10-09 04:09:25] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:13 [I 2022-10-09 04:09:25] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00302 [I 2022-10-09 04:09:25] ...trap/engines/engine.py.230: loss: 0.12588 [I 2022-10-09 04:09:25] ...optimizers/trijoint.py.151: total_norm: 21.011198 [I 2022-10-09 04:09:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 1150/2304 [I 2022-10-09 04:09:34] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:06 [I 2022-10-09 04:09:34] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00302 [I 2022-10-09 04:09:34] ...trap/engines/engine.py.230: loss: 0.15812 [I 2022-10-09 04:09:34] ...optimizers/trijoint.py.151: total_norm: 16.681927 [I 2022-10-09 04:09:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 1160/2304 [I 2022-10-09 04:09:42] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:02 [I 2022-10-09 04:09:42] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00309 [I 2022-10-09 04:09:42] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-09 04:09:42] ...optimizers/trijoint.py.151: total_norm: 10.020123 [I 2022-10-09 04:09:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 1170/2304 [I 2022-10-09 04:09:51] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:15:51 [I 2022-10-09 04:09:51] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00309 [I 2022-10-09 04:09:51] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-09 04:09:51] ...optimizers/trijoint.py.151: total_norm: 15.444407 [I 2022-10-09 04:09:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 1180/2304 [I 2022-10-09 04:09:59] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:15:40 [I 2022-10-09 04:09:59] ...trap/engines/engine.py.229: process: 0.83448 | load: 0.00298 [I 2022-10-09 04:09:59] ...trap/engines/engine.py.230: loss: 0.19640 [I 2022-10-09 04:09:59] ...optimizers/trijoint.py.151: total_norm: 19.816381 [I 2022-10-09 04:10:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 1190/2304 [I 2022-10-09 04:10:07] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:15:30 [I 2022-10-09 04:10:07] ...trap/engines/engine.py.229: process: 0.84743 | load: 0.00300 [I 2022-10-09 04:10:07] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-09 04:10:07] ...optimizers/trijoint.py.151: total_norm: 16.392776 [I 2022-10-09 04:10:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 1200/2304 [I 2022-10-09 04:10:16] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:28 [I 2022-10-09 04:10:16] ...trap/engines/engine.py.229: process: 0.84164 | load: 0.00300 [I 2022-10-09 04:10:16] ...trap/engines/engine.py.230: loss: 0.17283 [I 2022-10-09 04:10:16] ...optimizers/trijoint.py.151: total_norm: 15.486042 [I 2022-10-09 04:10:24] ...trap/engines/engine.py.224: train: epoch 31 | batch 1210/2304 [I 2022-10-09 04:10:24] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:15 [I 2022-10-09 04:10:24] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00299 [I 2022-10-09 04:10:24] ...trap/engines/engine.py.230: loss: 0.27229 [I 2022-10-09 04:10:24] ...optimizers/trijoint.py.151: total_norm: 16.616945 [I 2022-10-09 04:10:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 1220/2304 [I 2022-10-09 04:10:33] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:03 [I 2022-10-09 04:10:33] ...trap/engines/engine.py.229: process: 0.83293 | load: 0.00305 [I 2022-10-09 04:10:33] ...trap/engines/engine.py.230: loss: 0.16215 [I 2022-10-09 04:10:33] ...optimizers/trijoint.py.151: total_norm: 17.460318 [I 2022-10-09 04:10:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 1230/2304 [I 2022-10-09 04:10:41] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:14:56 [I 2022-10-09 04:10:41] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00299 [I 2022-10-09 04:10:41] ...trap/engines/engine.py.230: loss: 0.17215 [I 2022-10-09 04:10:41] ...optimizers/trijoint.py.151: total_norm: 30.927023 [I 2022-10-09 04:10:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 1240/2304 [I 2022-10-09 04:10:49] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:14:53 [I 2022-10-09 04:10:49] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00300 [I 2022-10-09 04:10:49] ...trap/engines/engine.py.230: loss: 0.15903 [I 2022-10-09 04:10:49] ...optimizers/trijoint.py.151: total_norm: 17.051411 [I 2022-10-09 04:10:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 1250/2304 [I 2022-10-09 04:10:58] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:14:45 [I 2022-10-09 04:10:58] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00301 [I 2022-10-09 04:10:58] ...trap/engines/engine.py.230: loss: 0.19179 [I 2022-10-09 04:10:58] ...optimizers/trijoint.py.151: total_norm: 19.520641 [I 2022-10-09 04:11:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 1260/2304 [I 2022-10-09 04:11:06] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:14:35 [I 2022-10-09 04:11:06] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00302 [I 2022-10-09 04:11:06] ...trap/engines/engine.py.230: loss: 0.09727 [I 2022-10-09 04:11:06] ...optimizers/trijoint.py.151: total_norm: 11.676212 [I 2022-10-09 04:11:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 1270/2304 [I 2022-10-09 04:11:15] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:28 [I 2022-10-09 04:11:15] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00303 [I 2022-10-09 04:11:15] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-09 04:11:15] ...optimizers/trijoint.py.151: total_norm: 24.511126 [I 2022-10-09 04:11:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 1280/2304 [I 2022-10-09 04:11:23] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:16 [I 2022-10-09 04:11:23] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00300 [I 2022-10-09 04:11:23] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-09 04:11:23] ...optimizers/trijoint.py.151: total_norm: 15.697113 [I 2022-10-09 04:11:31] ...trap/engines/engine.py.224: train: epoch 31 | batch 1290/2304 [I 2022-10-09 04:11:31] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:11 [I 2022-10-09 04:11:31] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00304 [I 2022-10-09 04:11:31] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-09 04:11:31] ...optimizers/trijoint.py.151: total_norm: 16.847702 [I 2022-10-09 04:11:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 1300/2304 [I 2022-10-09 04:11:40] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:00 [I 2022-10-09 04:11:40] ...trap/engines/engine.py.229: process: 0.83509 | load: 0.00298 [I 2022-10-09 04:11:40] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-09 04:11:40] ...optimizers/trijoint.py.151: total_norm: 12.511614 [I 2022-10-09 04:11:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 1310/2304 [I 2022-10-09 04:11:48] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:13:48 [I 2022-10-09 04:11:48] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00299 [I 2022-10-09 04:11:48] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-09 04:11:48] ...optimizers/trijoint.py.151: total_norm: 11.209944 [I 2022-10-09 04:11:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 1320/2304 [I 2022-10-09 04:11:56] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:13:43 [I 2022-10-09 04:11:56] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00301 [I 2022-10-09 04:11:56] ...trap/engines/engine.py.230: loss: 0.18920 [I 2022-10-09 04:11:56] ...optimizers/trijoint.py.151: total_norm: 15.959275 [I 2022-10-09 04:12:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 1330/2304 [I 2022-10-09 04:12:05] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:35 [I 2022-10-09 04:12:05] ...trap/engines/engine.py.229: process: 0.83112 | load: 0.00298 [I 2022-10-09 04:12:05] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-09 04:12:05] ...optimizers/trijoint.py.151: total_norm: 23.481838 [I 2022-10-09 04:12:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 1340/2304 [I 2022-10-09 04:12:13] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:25 [I 2022-10-09 04:12:13] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00297 [I 2022-10-09 04:12:13] ...trap/engines/engine.py.230: loss: 0.27786 [I 2022-10-09 04:12:13] ...optimizers/trijoint.py.151: total_norm: 21.548977 [I 2022-10-09 04:12:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 1350/2304 [I 2022-10-09 04:12:22] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:15 [I 2022-10-09 04:12:22] ...trap/engines/engine.py.229: process: 0.82669 | load: 0.00304 [I 2022-10-09 04:12:22] ...trap/engines/engine.py.230: loss: 0.18401 [I 2022-10-09 04:12:22] ...optimizers/trijoint.py.151: total_norm: 19.454723 [I 2022-10-09 04:12:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 1360/2304 [I 2022-10-09 04:12:30] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:06 [I 2022-10-09 04:12:30] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00301 [I 2022-10-09 04:12:30] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-09 04:12:30] ...optimizers/trijoint.py.151: total_norm: 11.629100 [I 2022-10-09 04:12:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 1370/2304 [I 2022-10-09 04:12:38] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:12:57 [I 2022-10-09 04:12:38] ...trap/engines/engine.py.229: process: 0.83235 | load: 0.00298 [I 2022-10-09 04:12:38] ...trap/engines/engine.py.230: loss: 0.18754 [I 2022-10-09 04:12:38] ...optimizers/trijoint.py.151: total_norm: 32.590347 [I 2022-10-09 04:12:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 1380/2304 [I 2022-10-09 04:12:47] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:12:56 [I 2022-10-09 04:12:47] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00302 [I 2022-10-09 04:12:47] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-09 04:12:47] ...optimizers/trijoint.py.151: total_norm: 20.793291 [I 2022-10-09 04:12:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 1390/2304 [I 2022-10-09 04:12:55] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:12:48 [I 2022-10-09 04:12:55] ...trap/engines/engine.py.229: process: 0.84080 | load: 0.00302 [I 2022-10-09 04:12:55] ...trap/engines/engine.py.230: loss: 0.22927 [I 2022-10-09 04:12:55] ...optimizers/trijoint.py.151: total_norm: 17.583670 [I 2022-10-09 04:13:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 1400/2304 [I 2022-10-09 04:13:04] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:38 [I 2022-10-09 04:13:04] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00304 [I 2022-10-09 04:13:04] ...trap/engines/engine.py.230: loss: 0.24636 [I 2022-10-09 04:13:04] ...optimizers/trijoint.py.151: total_norm: 24.295315 [I 2022-10-09 04:13:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 1410/2304 [I 2022-10-09 04:13:12] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:31 [I 2022-10-09 04:13:12] ...trap/engines/engine.py.229: process: 0.84173 | load: 0.00299 [I 2022-10-09 04:13:12] ...trap/engines/engine.py.230: loss: 0.17363 [I 2022-10-09 04:13:12] ...optimizers/trijoint.py.151: total_norm: 19.682055 [I 2022-10-09 04:13:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 1420/2304 [I 2022-10-09 04:13:20] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:19 [I 2022-10-09 04:13:20] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00301 [I 2022-10-09 04:13:20] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-09 04:13:20] ...optimizers/trijoint.py.151: total_norm: 9.856874 [I 2022-10-09 04:13:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 1430/2304 [I 2022-10-09 04:13:29] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:13 [I 2022-10-09 04:13:29] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00301 [I 2022-10-09 04:13:29] ...trap/engines/engine.py.230: loss: 0.19709 [I 2022-10-09 04:13:29] ...optimizers/trijoint.py.151: total_norm: 19.706757 [I 2022-10-09 04:13:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 1440/2304 [I 2022-10-09 04:13:37] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:06 [I 2022-10-09 04:13:37] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00300 [I 2022-10-09 04:13:37] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-09 04:13:37] ...optimizers/trijoint.py.151: total_norm: 11.952685 [I 2022-10-09 04:13:45] ...trap/engines/engine.py.224: train: epoch 31 | batch 1450/2304 [I 2022-10-09 04:13:45] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:11:54 [I 2022-10-09 04:13:45] ...trap/engines/engine.py.229: process: 0.84546 | load: 0.00302 [I 2022-10-09 04:13:45] ...trap/engines/engine.py.230: loss: 0.18118 [I 2022-10-09 04:13:45] ...optimizers/trijoint.py.151: total_norm: 25.176489 [I 2022-10-09 04:13:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 1460/2304 [I 2022-10-09 04:13:54] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:11:56 [I 2022-10-09 04:13:54] ...trap/engines/engine.py.229: process: 0.85337 | load: 0.00302 [I 2022-10-09 04:13:54] ...trap/engines/engine.py.230: loss: 0.09921 [I 2022-10-09 04:13:54] ...optimizers/trijoint.py.151: total_norm: 8.264791 [I 2022-10-09 04:14:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 1470/2304 [I 2022-10-09 04:14:02] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:46 [I 2022-10-09 04:14:02] ...trap/engines/engine.py.229: process: 0.83409 | load: 0.00300 [I 2022-10-09 04:14:02] ...trap/engines/engine.py.230: loss: 0.19643 [I 2022-10-09 04:14:02] ...optimizers/trijoint.py.151: total_norm: 21.465822 [I 2022-10-09 04:14:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 1480/2304 [I 2022-10-09 04:14:11] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:29 [I 2022-10-09 04:14:11] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00301 [I 2022-10-09 04:14:11] ...trap/engines/engine.py.230: loss: 0.15425 [I 2022-10-09 04:14:11] ...optimizers/trijoint.py.151: total_norm: 14.630136 [I 2022-10-09 04:14:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 1490/2304 [I 2022-10-09 04:14:19] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:18 [I 2022-10-09 04:14:19] ...trap/engines/engine.py.229: process: 0.83318 | load: 0.00306 [I 2022-10-09 04:14:19] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-09 04:14:19] ...optimizers/trijoint.py.151: total_norm: 17.400661 [I 2022-10-09 04:14:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 1500/2304 [I 2022-10-09 04:14:28] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:16 [I 2022-10-09 04:14:28] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00296 [I 2022-10-09 04:14:28] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-09 04:14:28] ...optimizers/trijoint.py.151: total_norm: 13.426671 [I 2022-10-09 04:14:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 1510/2304 [I 2022-10-09 04:14:36] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:07 [I 2022-10-09 04:14:36] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00306 [I 2022-10-09 04:14:36] ...trap/engines/engine.py.230: loss: 0.23934 [I 2022-10-09 04:14:36] ...optimizers/trijoint.py.151: total_norm: 16.681728 [I 2022-10-09 04:14:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 1520/2304 [I 2022-10-09 04:14:44] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:10:57 [I 2022-10-09 04:14:44] ...trap/engines/engine.py.229: process: 0.84334 | load: 0.00302 [I 2022-10-09 04:14:44] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-09 04:14:44] ...optimizers/trijoint.py.151: total_norm: 29.665131 [I 2022-10-09 04:14:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 1530/2304 [I 2022-10-09 04:14:53] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:10:49 [I 2022-10-09 04:14:53] ...trap/engines/engine.py.229: process: 0.82961 | load: 0.00303 [I 2022-10-09 04:14:53] ...trap/engines/engine.py.230: loss: 0.15202 [I 2022-10-09 04:14:53] ...optimizers/trijoint.py.151: total_norm: 28.318068 [I 2022-10-09 04:15:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 1540/2304 [I 2022-10-09 04:15:01] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:42 [I 2022-10-09 04:15:01] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00299 [I 2022-10-09 04:15:01] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-09 04:15:01] ...optimizers/trijoint.py.151: total_norm: 11.229114 [I 2022-10-09 04:15:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 1550/2304 [I 2022-10-09 04:15:10] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:34 [I 2022-10-09 04:15:10] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00303 [I 2022-10-09 04:15:10] ...trap/engines/engine.py.230: loss: 0.16422 [I 2022-10-09 04:15:10] ...optimizers/trijoint.py.151: total_norm: 17.232718 [I 2022-10-09 04:15:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1560/2304 [I 2022-10-09 04:15:18] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:28 [I 2022-10-09 04:15:18] ...trap/engines/engine.py.229: process: 0.86044 | load: 0.00304 [I 2022-10-09 04:15:18] ...trap/engines/engine.py.230: loss: 0.15599 [I 2022-10-09 04:15:18] ...optimizers/trijoint.py.151: total_norm: 33.880722 [I 2022-10-09 04:15:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 1570/2304 [I 2022-10-09 04:15:27] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:18 [I 2022-10-09 04:15:27] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00299 [I 2022-10-09 04:15:27] ...trap/engines/engine.py.230: loss: 0.20391 [I 2022-10-09 04:15:27] ...optimizers/trijoint.py.151: total_norm: 21.565344 [I 2022-10-09 04:15:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 1580/2304 [I 2022-10-09 04:15:35] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:08 [I 2022-10-09 04:15:35] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00300 [I 2022-10-09 04:15:35] ...trap/engines/engine.py.230: loss: 0.19880 [I 2022-10-09 04:15:35] ...optimizers/trijoint.py.151: total_norm: 32.014423 [I 2022-10-09 04:15:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 1590/2304 [I 2022-10-09 04:15:43] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:09:57 [I 2022-10-09 04:15:43] ...trap/engines/engine.py.229: process: 0.82288 | load: 0.00297 [I 2022-10-09 04:15:43] ...trap/engines/engine.py.230: loss: 0.15722 [I 2022-10-09 04:15:43] ...optimizers/trijoint.py.151: total_norm: 19.266039 [I 2022-10-09 04:15:52] ...trap/engines/engine.py.224: train: epoch 31 | batch 1600/2304 [I 2022-10-09 04:15:52] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:09:48 [I 2022-10-09 04:15:52] ...trap/engines/engine.py.229: process: 0.82186 | load: 0.00300 [I 2022-10-09 04:15:52] ...trap/engines/engine.py.230: loss: 0.17791 [I 2022-10-09 04:15:52] ...optimizers/trijoint.py.151: total_norm: 41.255165 [I 2022-10-09 04:16:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 1610/2304 [I 2022-10-09 04:16:00] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:42 [I 2022-10-09 04:16:00] ...trap/engines/engine.py.229: process: 0.83829 | load: 0.00301 [I 2022-10-09 04:16:00] ...trap/engines/engine.py.230: loss: 0.25343 [I 2022-10-09 04:16:00] ...optimizers/trijoint.py.151: total_norm: 19.259178 [I 2022-10-09 04:16:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 1620/2304 [I 2022-10-09 04:16:09] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:34 [I 2022-10-09 04:16:09] ...trap/engines/engine.py.229: process: 0.84080 | load: 0.00302 [I 2022-10-09 04:16:09] ...trap/engines/engine.py.230: loss: 0.09078 [I 2022-10-09 04:16:09] ...optimizers/trijoint.py.151: total_norm: 9.025078 [I 2022-10-09 04:16:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 1630/2304 [I 2022-10-09 04:16:17] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:26 [I 2022-10-09 04:16:17] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00306 [I 2022-10-09 04:16:17] ...trap/engines/engine.py.230: loss: 0.17741 [I 2022-10-09 04:16:17] ...optimizers/trijoint.py.151: total_norm: 20.074450 [I 2022-10-09 04:16:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 1640/2304 [I 2022-10-09 04:16:26] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:19 [I 2022-10-09 04:16:26] ...trap/engines/engine.py.229: process: 0.85134 | load: 0.00301 [I 2022-10-09 04:16:26] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-09 04:16:26] ...optimizers/trijoint.py.151: total_norm: 10.985029 [I 2022-10-09 04:16:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 1650/2304 [I 2022-10-09 04:16:34] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:09 [I 2022-10-09 04:16:34] ...trap/engines/engine.py.229: process: 0.82750 | load: 0.00301 [I 2022-10-09 04:16:34] ...trap/engines/engine.py.230: loss: 0.10187 [I 2022-10-09 04:16:34] ...optimizers/trijoint.py.151: total_norm: 7.412332 [I 2022-10-09 04:16:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 1660/2304 [I 2022-10-09 04:16:42] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:03 [I 2022-10-09 04:16:42] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00300 [I 2022-10-09 04:16:42] ...trap/engines/engine.py.230: loss: 0.13101 [I 2022-10-09 04:16:42] ...optimizers/trijoint.py.151: total_norm: 12.066699 [I 2022-10-09 04:16:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 1670/2304 [I 2022-10-09 04:16:51] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:08:54 [I 2022-10-09 04:16:51] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00298 [I 2022-10-09 04:16:51] ...trap/engines/engine.py.230: loss: 0.22494 [I 2022-10-09 04:16:51] ...optimizers/trijoint.py.151: total_norm: 36.703335 [I 2022-10-09 04:16:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 1680/2304 [I 2022-10-09 04:16:59] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:08:43 [I 2022-10-09 04:16:59] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00300 [I 2022-10-09 04:16:59] ...trap/engines/engine.py.230: loss: 0.18097 [I 2022-10-09 04:16:59] ...optimizers/trijoint.py.151: total_norm: 16.656853 [I 2022-10-09 04:17:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 1690/2304 [I 2022-10-09 04:17:08] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:37 [I 2022-10-09 04:17:08] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00305 [I 2022-10-09 04:17:08] ...trap/engines/engine.py.230: loss: 0.26781 [I 2022-10-09 04:17:08] ...optimizers/trijoint.py.151: total_norm: 23.540770 [I 2022-10-09 04:17:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 1700/2304 [I 2022-10-09 04:17:16] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:27 [I 2022-10-09 04:17:16] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00302 [I 2022-10-09 04:17:16] ...trap/engines/engine.py.230: loss: 0.15902 [I 2022-10-09 04:17:16] ...optimizers/trijoint.py.151: total_norm: 17.795458 [I 2022-10-09 04:17:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 1710/2304 [I 2022-10-09 04:17:25] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:21 [I 2022-10-09 04:17:25] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00302 [I 2022-10-09 04:17:25] ...trap/engines/engine.py.230: loss: 0.15351 [I 2022-10-09 04:17:25] ...optimizers/trijoint.py.151: total_norm: 44.750282 [I 2022-10-09 04:17:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 1720/2304 [I 2022-10-09 04:17:33] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:12 [I 2022-10-09 04:17:33] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00300 [I 2022-10-09 04:17:33] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-09 04:17:33] ...optimizers/trijoint.py.151: total_norm: 10.566998 [I 2022-10-09 04:17:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 1730/2304 [I 2022-10-09 04:17:41] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:00 [I 2022-10-09 04:17:41] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00302 [I 2022-10-09 04:17:41] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-09 04:17:41] ...optimizers/trijoint.py.151: total_norm: 24.361406 [I 2022-10-09 04:17:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 1740/2304 [I 2022-10-09 04:17:50] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:07:54 [I 2022-10-09 04:17:50] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00309 [I 2022-10-09 04:17:50] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-09 04:17:50] ...optimizers/trijoint.py.151: total_norm: 11.738853 [I 2022-10-09 04:17:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 1750/2304 [I 2022-10-09 04:17:58] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:07:45 [I 2022-10-09 04:17:58] ...trap/engines/engine.py.229: process: 0.83823 | load: 0.00304 [I 2022-10-09 04:17:58] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-09 04:17:58] ...optimizers/trijoint.py.151: total_norm: 15.585632 [I 2022-10-09 04:18:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 1760/2304 [I 2022-10-09 04:18:07] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:36 [I 2022-10-09 04:18:07] ...trap/engines/engine.py.229: process: 0.83061 | load: 0.00299 [I 2022-10-09 04:18:07] ...trap/engines/engine.py.230: loss: 0.18069 [I 2022-10-09 04:18:07] ...optimizers/trijoint.py.151: total_norm: 13.189258 [I 2022-10-09 04:18:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 1770/2304 [I 2022-10-09 04:18:15] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:27 [I 2022-10-09 04:18:15] ...trap/engines/engine.py.229: process: 0.82695 | load: 0.00302 [I 2022-10-09 04:18:15] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-09 04:18:15] ...optimizers/trijoint.py.151: total_norm: 18.299963 [I 2022-10-09 04:18:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 1780/2304 [I 2022-10-09 04:18:23] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:18 [I 2022-10-09 04:18:23] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00300 [I 2022-10-09 04:18:23] ...trap/engines/engine.py.230: loss: 0.17484 [I 2022-10-09 04:18:23] ...optimizers/trijoint.py.151: total_norm: 42.353592 [I 2022-10-09 04:18:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 1790/2304 [I 2022-10-09 04:18:32] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:12 [I 2022-10-09 04:18:32] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00298 [I 2022-10-09 04:18:32] ...trap/engines/engine.py.230: loss: 0.12242 [I 2022-10-09 04:18:32] ...optimizers/trijoint.py.151: total_norm: 11.015128 [I 2022-10-09 04:18:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 1800/2304 [I 2022-10-09 04:18:40] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:06 [I 2022-10-09 04:18:40] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00301 [I 2022-10-09 04:18:40] ...trap/engines/engine.py.230: loss: 0.18324 [I 2022-10-09 04:18:40] ...optimizers/trijoint.py.151: total_norm: 13.357100 [I 2022-10-09 04:18:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 1810/2304 [I 2022-10-09 04:18:49] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:06:56 [I 2022-10-09 04:18:49] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00305 [I 2022-10-09 04:18:49] ...trap/engines/engine.py.230: loss: 0.15122 [I 2022-10-09 04:18:49] ...optimizers/trijoint.py.151: total_norm: 20.314997 [I 2022-10-09 04:18:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 1820/2304 [I 2022-10-09 04:18:57] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:06:47 [I 2022-10-09 04:18:57] ...trap/engines/engine.py.229: process: 0.83920 | load: 0.00298 [I 2022-10-09 04:18:57] ...trap/engines/engine.py.230: loss: 0.14583 [I 2022-10-09 04:18:57] ...optimizers/trijoint.py.151: total_norm: 20.501446 [I 2022-10-09 04:19:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 1830/2304 [I 2022-10-09 04:19:06] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:06:37 [I 2022-10-09 04:19:06] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00295 [I 2022-10-09 04:19:06] ...trap/engines/engine.py.230: loss: 0.15608 [I 2022-10-09 04:19:06] ...optimizers/trijoint.py.151: total_norm: 29.107208 [I 2022-10-09 04:19:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 1840/2304 [I 2022-10-09 04:19:14] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:06:30 [I 2022-10-09 04:19:14] ...trap/engines/engine.py.229: process: 0.83330 | load: 0.00303 [I 2022-10-09 04:19:14] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-09 04:19:14] ...optimizers/trijoint.py.151: total_norm: 14.566867 [I 2022-10-09 04:19:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 1850/2304 [I 2022-10-09 04:19:22] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:20 [I 2022-10-09 04:19:22] ...trap/engines/engine.py.229: process: 0.83269 | load: 0.00301 [I 2022-10-09 04:19:22] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-09 04:19:22] ...optimizers/trijoint.py.151: total_norm: 23.086279 [I 2022-10-09 04:19:31] ...trap/engines/engine.py.224: train: epoch 31 | batch 1860/2304 [I 2022-10-09 04:19:31] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:06:10 [I 2022-10-09 04:19:31] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00303 [I 2022-10-09 04:19:31] ...trap/engines/engine.py.230: loss: 0.15378 [I 2022-10-09 04:19:31] ...optimizers/trijoint.py.151: total_norm: 17.923847 [I 2022-10-09 04:19:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 1870/2304 [I 2022-10-09 04:19:39] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:04 [I 2022-10-09 04:19:39] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00302 [I 2022-10-09 04:19:39] ...trap/engines/engine.py.230: loss: 0.10405 [I 2022-10-09 04:19:39] ...optimizers/trijoint.py.151: total_norm: 13.205556 [I 2022-10-09 04:19:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 1880/2304 [I 2022-10-09 04:19:48] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:05:56 [I 2022-10-09 04:19:48] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00301 [I 2022-10-09 04:19:48] ...trap/engines/engine.py.230: loss: 0.18866 [I 2022-10-09 04:19:48] ...optimizers/trijoint.py.151: total_norm: 16.763712 [I 2022-10-09 04:19:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 1890/2304 [I 2022-10-09 04:19:56] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:05:47 [I 2022-10-09 04:19:56] ...trap/engines/engine.py.229: process: 0.83604 | load: 0.00298 [I 2022-10-09 04:19:56] ...trap/engines/engine.py.230: loss: 0.20906 [I 2022-10-09 04:19:56] ...optimizers/trijoint.py.151: total_norm: 14.441712 [I 2022-10-09 04:20:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 1900/2304 [I 2022-10-09 04:20:04] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:05:40 [I 2022-10-09 04:20:04] ...trap/engines/engine.py.229: process: 0.83772 | load: 0.00303 [I 2022-10-09 04:20:04] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-09 04:20:04] ...optimizers/trijoint.py.151: total_norm: 10.586185 [I 2022-10-09 04:20:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 1910/2304 [I 2022-10-09 04:20:13] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:05:30 [I 2022-10-09 04:20:13] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00301 [I 2022-10-09 04:20:13] ...trap/engines/engine.py.230: loss: 0.13743 [I 2022-10-09 04:20:13] ...optimizers/trijoint.py.151: total_norm: 20.171831 [I 2022-10-09 04:20:21] ...trap/engines/engine.py.224: train: epoch 31 | batch 1920/2304 [I 2022-10-09 04:20:21] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:22 [I 2022-10-09 04:20:21] ...trap/engines/engine.py.229: process: 0.83247 | load: 0.00300 [I 2022-10-09 04:20:21] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-09 04:20:21] ...optimizers/trijoint.py.151: total_norm: 19.538555 [I 2022-10-09 04:20:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 1930/2304 [I 2022-10-09 04:20:30] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:13 [I 2022-10-09 04:20:30] ...trap/engines/engine.py.229: process: 0.82865 | load: 0.00301 [I 2022-10-09 04:20:30] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-09 04:20:30] ...optimizers/trijoint.py.151: total_norm: 7.939539 [I 2022-10-09 04:20:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 1940/2304 [I 2022-10-09 04:20:38] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:07 [I 2022-10-09 04:20:38] ...trap/engines/engine.py.229: process: 0.85459 | load: 0.00308 [I 2022-10-09 04:20:38] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-09 04:20:38] ...optimizers/trijoint.py.151: total_norm: 12.123114 [I 2022-10-09 04:20:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 1950/2304 [I 2022-10-09 04:20:47] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:04:59 [I 2022-10-09 04:20:47] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00303 [I 2022-10-09 04:20:47] ...trap/engines/engine.py.230: loss: 0.23106 [I 2022-10-09 04:20:47] ...optimizers/trijoint.py.151: total_norm: 18.944817 [I 2022-10-09 04:20:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 1960/2304 [I 2022-10-09 04:20:55] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:04:47 [I 2022-10-09 04:20:55] ...trap/engines/engine.py.229: process: 0.82942 | load: 0.00307 [I 2022-10-09 04:20:55] ...trap/engines/engine.py.230: loss: 0.14523 [I 2022-10-09 04:20:55] ...optimizers/trijoint.py.151: total_norm: 36.405693 [I 2022-10-09 04:21:03] ...trap/engines/engine.py.224: train: epoch 31 | batch 1970/2304 [I 2022-10-09 04:21:03] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:04:39 [I 2022-10-09 04:21:03] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00301 [I 2022-10-09 04:21:03] ...trap/engines/engine.py.230: loss: 0.20686 [I 2022-10-09 04:21:03] ...optimizers/trijoint.py.151: total_norm: 20.394789 [I 2022-10-09 04:21:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 1980/2304 [I 2022-10-09 04:21:12] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:31 [I 2022-10-09 04:21:12] ...trap/engines/engine.py.229: process: 0.81915 | load: 0.00299 [I 2022-10-09 04:21:12] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-09 04:21:12] ...optimizers/trijoint.py.151: total_norm: 7.460958 [I 2022-10-09 04:21:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 1990/2304 [I 2022-10-09 04:21:20] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:04:21 [I 2022-10-09 04:21:20] ...trap/engines/engine.py.229: process: 0.81958 | load: 0.00301 [I 2022-10-09 04:21:20] ...trap/engines/engine.py.230: loss: 0.14591 [I 2022-10-09 04:21:20] ...optimizers/trijoint.py.151: total_norm: 13.957168 [I 2022-10-09 04:21:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 2000/2304 [I 2022-10-09 04:21:29] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:16 [I 2022-10-09 04:21:29] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00304 [I 2022-10-09 04:21:29] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-09 04:21:29] ...optimizers/trijoint.py.151: total_norm: 15.901242 [I 2022-10-09 04:21:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 2010/2304 [I 2022-10-09 04:21:37] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:06 [I 2022-10-09 04:21:37] ...trap/engines/engine.py.229: process: 0.82466 | load: 0.00304 [I 2022-10-09 04:21:37] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-09 04:21:37] ...optimizers/trijoint.py.151: total_norm: 18.062006 [I 2022-10-09 04:21:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 2020/2304 [I 2022-10-09 04:21:46] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:03:58 [I 2022-10-09 04:21:46] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00301 [I 2022-10-09 04:21:46] ...trap/engines/engine.py.230: loss: 0.17494 [I 2022-10-09 04:21:46] ...optimizers/trijoint.py.151: total_norm: 15.785125 [I 2022-10-09 04:21:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 2030/2304 [I 2022-10-09 04:21:54] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:03:50 [I 2022-10-09 04:21:54] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00301 [I 2022-10-09 04:21:54] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-09 04:21:54] ...optimizers/trijoint.py.151: total_norm: 11.123912 [I 2022-10-09 04:22:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 2040/2304 [I 2022-10-09 04:22:02] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:03:42 [I 2022-10-09 04:22:02] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00300 [I 2022-10-09 04:22:02] ...trap/engines/engine.py.230: loss: 0.22581 [I 2022-10-09 04:22:02] ...optimizers/trijoint.py.151: total_norm: 26.454697 [I 2022-10-09 04:22:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 2050/2304 [I 2022-10-09 04:22:11] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:03:34 [I 2022-10-09 04:22:11] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00301 [I 2022-10-09 04:22:11] ...trap/engines/engine.py.230: loss: 0.16511 [I 2022-10-09 04:22:11] ...optimizers/trijoint.py.151: total_norm: 20.477110 [I 2022-10-09 04:22:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 2060/2304 [I 2022-10-09 04:22:19] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:25 [I 2022-10-09 04:22:19] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00308 [I 2022-10-09 04:22:19] ...trap/engines/engine.py.230: loss: 0.19840 [I 2022-10-09 04:22:19] ...optimizers/trijoint.py.151: total_norm: 21.412354 [I 2022-10-09 04:22:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 2070/2304 [I 2022-10-09 04:22:28] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:17 [I 2022-10-09 04:22:28] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00302 [I 2022-10-09 04:22:28] ...trap/engines/engine.py.230: loss: 0.20385 [I 2022-10-09 04:22:28] ...optimizers/trijoint.py.151: total_norm: 37.057381 [I 2022-10-09 04:22:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 2080/2304 [I 2022-10-09 04:22:36] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:08 [I 2022-10-09 04:22:36] ...trap/engines/engine.py.229: process: 0.84759 | load: 0.00299 [I 2022-10-09 04:22:36] ...trap/engines/engine.py.230: loss: 0.19586 [I 2022-10-09 04:22:36] ...optimizers/trijoint.py.151: total_norm: 24.598303 [I 2022-10-09 04:22:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 2090/2304 [I 2022-10-09 04:22:44] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:02:59 [I 2022-10-09 04:22:44] ...trap/engines/engine.py.229: process: 0.84448 | load: 0.00299 [I 2022-10-09 04:22:44] ...trap/engines/engine.py.230: loss: 0.13700 [I 2022-10-09 04:22:44] ...optimizers/trijoint.py.151: total_norm: 14.148058 [I 2022-10-09 04:22:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 2100/2304 [I 2022-10-09 04:22:53] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:02:52 [I 2022-10-09 04:22:53] ...trap/engines/engine.py.229: process: 0.85015 | load: 0.00298 [I 2022-10-09 04:22:53] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-09 04:22:53] ...optimizers/trijoint.py.151: total_norm: 16.216440 [I 2022-10-09 04:23:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 2110/2304 [I 2022-10-09 04:23:01] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:02:43 [I 2022-10-09 04:23:01] ...trap/engines/engine.py.229: process: 0.83566 | load: 0.00302 [I 2022-10-09 04:23:01] ...trap/engines/engine.py.230: loss: 0.19249 [I 2022-10-09 04:23:01] ...optimizers/trijoint.py.151: total_norm: 21.241537 [I 2022-10-09 04:23:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 2120/2304 [I 2022-10-09 04:23:10] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:02:33 [I 2022-10-09 04:23:10] ...trap/engines/engine.py.229: process: 0.83021 | load: 0.00301 [I 2022-10-09 04:23:10] ...trap/engines/engine.py.230: loss: 0.17950 [I 2022-10-09 04:23:10] ...optimizers/trijoint.py.151: total_norm: 21.168915 [I 2022-10-09 04:23:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 2130/2304 [I 2022-10-09 04:23:18] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:26 [I 2022-10-09 04:23:18] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00304 [I 2022-10-09 04:23:18] ...trap/engines/engine.py.230: loss: 0.22134 [I 2022-10-09 04:23:18] ...optimizers/trijoint.py.151: total_norm: 25.416672 [I 2022-10-09 04:23:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 2140/2304 [I 2022-10-09 04:23:27] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:17 [I 2022-10-09 04:23:27] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00302 [I 2022-10-09 04:23:27] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-09 04:23:27] ...optimizers/trijoint.py.151: total_norm: 19.891394 [I 2022-10-09 04:23:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 2150/2304 [I 2022-10-09 04:23:35] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:09 [I 2022-10-09 04:23:35] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00301 [I 2022-10-09 04:23:35] ...trap/engines/engine.py.230: loss: 0.18366 [I 2022-10-09 04:23:35] ...optimizers/trijoint.py.151: total_norm: 17.991964 [I 2022-10-09 04:23:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 2160/2304 [I 2022-10-09 04:23:43] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:00 [I 2022-10-09 04:23:43] ...trap/engines/engine.py.229: process: 0.84091 | load: 0.00299 [I 2022-10-09 04:23:43] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-09 04:23:43] ...optimizers/trijoint.py.151: total_norm: 11.431555 [I 2022-10-09 04:23:52] ...trap/engines/engine.py.224: train: epoch 31 | batch 2170/2304 [I 2022-10-09 04:23:52] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:01:52 [I 2022-10-09 04:23:52] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00303 [I 2022-10-09 04:23:52] ...trap/engines/engine.py.230: loss: 0.15301 [I 2022-10-09 04:23:52] ...optimizers/trijoint.py.151: total_norm: 16.888851 [I 2022-10-09 04:24:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 2180/2304 [I 2022-10-09 04:24:00] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:01:44 [I 2022-10-09 04:24:00] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00297 [I 2022-10-09 04:24:00] ...trap/engines/engine.py.230: loss: 0.19987 [I 2022-10-09 04:24:00] ...optimizers/trijoint.py.151: total_norm: 13.236145 [I 2022-10-09 04:24:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 2190/2304 [I 2022-10-09 04:24:09] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:01:35 [I 2022-10-09 04:24:09] ...trap/engines/engine.py.229: process: 0.82186 | load: 0.00303 [I 2022-10-09 04:24:09] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-09 04:24:09] ...optimizers/trijoint.py.151: total_norm: 12.717159 [I 2022-10-09 04:24:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 2200/2304 [I 2022-10-09 04:24:17] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:27 [I 2022-10-09 04:24:17] ...trap/engines/engine.py.229: process: 0.82958 | load: 0.00301 [I 2022-10-09 04:24:17] ...trap/engines/engine.py.230: loss: 0.16145 [I 2022-10-09 04:24:17] ...optimizers/trijoint.py.151: total_norm: 16.925999 [I 2022-10-09 04:24:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 2210/2304 [I 2022-10-09 04:24:25] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:18 [I 2022-10-09 04:24:25] ...trap/engines/engine.py.229: process: 0.83306 | load: 0.00300 [I 2022-10-09 04:24:25] ...trap/engines/engine.py.230: loss: 0.19571 [I 2022-10-09 04:24:25] ...optimizers/trijoint.py.151: total_norm: 29.635141 [I 2022-10-09 04:24:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 2220/2304 [I 2022-10-09 04:24:34] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:10 [I 2022-10-09 04:24:34] ...trap/engines/engine.py.229: process: 0.83742 | load: 0.00301 [I 2022-10-09 04:24:34] ...trap/engines/engine.py.230: loss: 0.19913 [I 2022-10-09 04:24:34] ...optimizers/trijoint.py.151: total_norm: 17.597076 [I 2022-10-09 04:24:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 2230/2304 [I 2022-10-09 04:24:42] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:01 [I 2022-10-09 04:24:42] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00300 [I 2022-10-09 04:24:42] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-09 04:24:42] ...optimizers/trijoint.py.151: total_norm: 15.516672 [I 2022-10-09 04:24:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 2240/2304 [I 2022-10-09 04:24:51] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:00:53 [I 2022-10-09 04:24:51] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00302 [I 2022-10-09 04:24:51] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-09 04:24:51] ...optimizers/trijoint.py.151: total_norm: 10.618619 [I 2022-10-09 04:24:59] ...trap/engines/engine.py.224: train: epoch 31 | batch 2250/2304 [I 2022-10-09 04:24:59] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:00:44 [I 2022-10-09 04:24:59] ...trap/engines/engine.py.229: process: 0.81852 | load: 0.00300 [I 2022-10-09 04:24:59] ...trap/engines/engine.py.230: loss: 0.14630 [I 2022-10-09 04:24:59] ...optimizers/trijoint.py.151: total_norm: 14.201231 [I 2022-10-09 04:25:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 2260/2304 [I 2022-10-09 04:25:07] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:00:36 [I 2022-10-09 04:25:07] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00309 [I 2022-10-09 04:25:07] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-09 04:25:07] ...optimizers/trijoint.py.151: total_norm: 20.598549 [I 2022-10-09 04:25:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 2270/2304 [I 2022-10-09 04:25:16] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:28 [I 2022-10-09 04:25:16] ...trap/engines/engine.py.229: process: 0.83230 | load: 0.00302 [I 2022-10-09 04:25:16] ...trap/engines/engine.py.230: loss: 0.21462 [I 2022-10-09 04:25:16] ...optimizers/trijoint.py.151: total_norm: 9.777916 [I 2022-10-09 04:25:24] ...trap/engines/engine.py.224: train: epoch 31 | batch 2280/2304 [I 2022-10-09 04:25:24] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:20 [I 2022-10-09 04:25:24] ...trap/engines/engine.py.229: process: 0.85899 | load: 0.00301 [I 2022-10-09 04:25:24] ...trap/engines/engine.py.230: loss: 0.23221 [I 2022-10-09 04:25:24] ...optimizers/trijoint.py.151: total_norm: 11.519814 [I 2022-10-09 04:25:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 2290/2304 [I 2022-10-09 04:25:33] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:11 [I 2022-10-09 04:25:33] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00300 [I 2022-10-09 04:25:33] ...trap/engines/engine.py.230: loss: 0.14344 [I 2022-10-09 04:25:33] ...optimizers/trijoint.py.151: total_norm: 15.550448 [I 2022-10-09 04:25:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 2300/2304 [I 2022-10-09 04:25:41] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:03 [I 2022-10-09 04:25:41] ...trap/engines/engine.py.229: process: 0.81852 | load: 0.00016 [I 2022-10-09 04:25:41] ...trap/engines/engine.py.230: loss: 0.13806 [I 2022-10-09 04:25:41] ...optimizers/trijoint.py.151: total_norm: 28.844233 [I 2022-10-09 04:25:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 2304/2304 [I 2022-10-09 04:25:44] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:00 [I 2022-10-09 04:25:44] ...trap/engines/engine.py.229: process: 0.81727 | load: 0.00017 [I 2022-10-09 04:25:44] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-09 04:25:44] ...optimizers/trijoint.py.151: total_norm: 19.549574 [S 2022-10-09 04:25:46] ...trap/engines/engine.py.129: train_epoch.epoch: 31 [S 2022-10-09 04:25:46] ...trap/engines/engine.py.129: train_epoch.loss: 0.16438196942759697 [I 2022-10-09 04:25:46] ...s/criterions/triplet.py.84: margin = 0.21000000000000013 [I 2022-10-09 04:25:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 31 [I 2022-10-09 04:25:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 04:25:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 04:25:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 04:25:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 04:25:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 04:25:50] ...trap/engines/engine.py.321: eval: epoch 31 | batch 0/510 [I 2022-10-09 04:25:50] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:21 [I 2022-10-09 04:25:50] ...trap/engines/engine.py.326: process: 3.33584 | load: 2.86115 [I 2022-10-09 04:25:55] ...trap/engines/engine.py.321: eval: epoch 31 | batch 10/510 [I 2022-10-09 04:25:55] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:06 [I 2022-10-09 04:25:55] ...trap/engines/engine.py.326: process: 0.61879 | load: 0.00019 [I 2022-10-09 04:26:00] ...trap/engines/engine.py.321: eval: epoch 31 | batch 20/510 [I 2022-10-09 04:26:00] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:04:53 [I 2022-10-09 04:26:00] ...trap/engines/engine.py.326: process: 0.54954 | load: 0.00019 [I 2022-10-09 04:26:05] ...trap/engines/engine.py.321: eval: epoch 31 | batch 30/510 [I 2022-10-09 04:26:05] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:03:36 [I 2022-10-09 04:26:05] ...trap/engines/engine.py.326: process: 0.43008 | load: 0.00019 [I 2022-10-09 04:26:11] ...trap/engines/engine.py.321: eval: epoch 31 | batch 40/510 [I 2022-10-09 04:26:11] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:11 [I 2022-10-09 04:26:11] ...trap/engines/engine.py.326: process: 0.60106 | load: 0.00021 [I 2022-10-09 04:26:16] ...trap/engines/engine.py.321: eval: epoch 31 | batch 50/510 [I 2022-10-09 04:26:16] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:44 [I 2022-10-09 04:26:16] ...trap/engines/engine.py.326: process: 0.40940 | load: 0.00019 [I 2022-10-09 04:26:21] ...trap/engines/engine.py.321: eval: epoch 31 | batch 60/510 [I 2022-10-09 04:26:21] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:53 [I 2022-10-09 04:26:21] ...trap/engines/engine.py.326: process: 0.46919 | load: 0.00019 [I 2022-10-09 04:26:26] ...trap/engines/engine.py.321: eval: epoch 31 | batch 70/510 [I 2022-10-09 04:26:26] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:44 [I 2022-10-09 04:26:26] ...trap/engines/engine.py.326: process: 0.55386 | load: 0.00018 [I 2022-10-09 04:26:31] ...trap/engines/engine.py.321: eval: epoch 31 | batch 80/510 [I 2022-10-09 04:26:31] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:41 [I 2022-10-09 04:26:31] ...trap/engines/engine.py.326: process: 0.53201 | load: 0.00019 [I 2022-10-09 04:26:36] ...trap/engines/engine.py.321: eval: epoch 31 | batch 90/510 [I 2022-10-09 04:26:36] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:04 [I 2022-10-09 04:26:36] ...trap/engines/engine.py.326: process: 0.41352 | load: 0.00018 [I 2022-10-09 04:26:40] ...trap/engines/engine.py.321: eval: epoch 31 | batch 100/510 [I 2022-10-09 04:26:40] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:54 [I 2022-10-09 04:26:40] ...trap/engines/engine.py.326: process: 0.30818 | load: 0.00017 [I 2022-10-09 04:26:43] ...trap/engines/engine.py.321: eval: epoch 31 | batch 110/510 [I 2022-10-09 04:26:43] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:11 [I 2022-10-09 04:26:43] ...trap/engines/engine.py.326: process: 0.31056 | load: 0.00019 [I 2022-10-09 04:26:46] ...trap/engines/engine.py.321: eval: epoch 31 | batch 120/510 [I 2022-10-09 04:26:46] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-09 04:26:46] ...trap/engines/engine.py.326: process: 0.31746 | load: 0.00020 [I 2022-10-09 04:26:50] ...trap/engines/engine.py.321: eval: epoch 31 | batch 130/510 [I 2022-10-09 04:26:50] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-09 04:26:50] ...trap/engines/engine.py.326: process: 0.31987 | load: 0.00020 [I 2022-10-09 04:26:53] ...trap/engines/engine.py.321: eval: epoch 31 | batch 140/510 [I 2022-10-09 04:26:53] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:56 [I 2022-10-09 04:26:53] ...trap/engines/engine.py.326: process: 0.31369 | load: 0.00019 [I 2022-10-09 04:26:56] ...trap/engines/engine.py.321: eval: epoch 31 | batch 150/510 [I 2022-10-09 04:26:56] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:54 [I 2022-10-09 04:26:56] ...trap/engines/engine.py.326: process: 0.32083 | load: 0.00020 [I 2022-10-09 04:26:59] ...trap/engines/engine.py.321: eval: epoch 31 | batch 160/510 [I 2022-10-09 04:26:59] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:51 [I 2022-10-09 04:26:59] ...trap/engines/engine.py.326: process: 0.31585 | load: 0.00019 [I 2022-10-09 04:27:02] ...trap/engines/engine.py.321: eval: epoch 31 | batch 170/510 [I 2022-10-09 04:27:02] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-09 04:27:02] ...trap/engines/engine.py.326: process: 0.31438 | load: 0.00019 [I 2022-10-09 04:27:05] ...trap/engines/engine.py.321: eval: epoch 31 | batch 180/510 [I 2022-10-09 04:27:05] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-09 04:27:05] ...trap/engines/engine.py.326: process: 0.31320 | load: 0.00019 [I 2022-10-09 04:27:09] ...trap/engines/engine.py.321: eval: epoch 31 | batch 190/510 [I 2022-10-09 04:27:09] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:41 [I 2022-10-09 04:27:09] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00020 [I 2022-10-09 04:27:12] ...trap/engines/engine.py.321: eval: epoch 31 | batch 200/510 [I 2022-10-09 04:27:12] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:38 [I 2022-10-09 04:27:12] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00019 [I 2022-10-09 04:27:15] ...trap/engines/engine.py.321: eval: epoch 31 | batch 210/510 [I 2022-10-09 04:27:15] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:35 [I 2022-10-09 04:27:15] ...trap/engines/engine.py.326: process: 0.31471 | load: 0.00020 [I 2022-10-09 04:27:18] ...trap/engines/engine.py.321: eval: epoch 31 | batch 220/510 [I 2022-10-09 04:27:18] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:31 [I 2022-10-09 04:27:18] ...trap/engines/engine.py.326: process: 0.31231 | load: 0.00021 [I 2022-10-09 04:27:21] ...trap/engines/engine.py.321: eval: epoch 31 | batch 230/510 [I 2022-10-09 04:27:21] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:29 [I 2022-10-09 04:27:21] ...trap/engines/engine.py.326: process: 0.31963 | load: 0.00020 [I 2022-10-09 04:27:25] ...trap/engines/engine.py.321: eval: epoch 31 | batch 240/510 [I 2022-10-09 04:27:25] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-09 04:27:25] ...trap/engines/engine.py.326: process: 0.31833 | load: 0.00021 [I 2022-10-09 04:27:28] ...trap/engines/engine.py.321: eval: epoch 31 | batch 250/510 [I 2022-10-09 04:27:28] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:23 [I 2022-10-09 04:27:28] ...trap/engines/engine.py.326: process: 0.31357 | load: 0.00020 [I 2022-10-09 04:27:31] ...trap/engines/engine.py.321: eval: epoch 31 | batch 260/510 [I 2022-10-09 04:27:31] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-09 04:27:31] ...trap/engines/engine.py.326: process: 0.32370 | load: 0.00019 [I 2022-10-09 04:27:34] ...trap/engines/engine.py.321: eval: epoch 31 | batch 270/510 [I 2022-10-09 04:27:34] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-09 04:27:34] ...trap/engines/engine.py.326: process: 0.31613 | load: 0.00019 [I 2022-10-09 04:27:37] ...trap/engines/engine.py.321: eval: epoch 31 | batch 280/510 [I 2022-10-09 04:27:37] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-09 04:27:37] ...trap/engines/engine.py.326: process: 0.31997 | load: 0.00020 [I 2022-10-09 04:27:41] ...trap/engines/engine.py.321: eval: epoch 31 | batch 290/510 [I 2022-10-09 04:27:41] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-09 04:27:41] ...trap/engines/engine.py.326: process: 0.31520 | load: 0.00020 [I 2022-10-09 04:27:44] ...trap/engines/engine.py.321: eval: epoch 31 | batch 300/510 [I 2022-10-09 04:27:44] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:07 [I 2022-10-09 04:27:44] ...trap/engines/engine.py.326: process: 0.31138 | load: 0.00019 [I 2022-10-09 04:27:47] ...trap/engines/engine.py.321: eval: epoch 31 | batch 310/510 [I 2022-10-09 04:27:47] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-09 04:27:47] ...trap/engines/engine.py.326: process: 0.31963 | load: 0.00020 [I 2022-10-09 04:27:50] ...trap/engines/engine.py.321: eval: epoch 31 | batch 320/510 [I 2022-10-09 04:27:50] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 04:27:50] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00020 [I 2022-10-09 04:27:53] ...trap/engines/engine.py.321: eval: epoch 31 | batch 330/510 [I 2022-10-09 04:27:53] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-09 04:27:53] ...trap/engines/engine.py.326: process: 0.32233 | load: 0.00020 [I 2022-10-09 04:27:57] ...trap/engines/engine.py.321: eval: epoch 31 | batch 340/510 [I 2022-10-09 04:27:57] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-09 04:27:57] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00021 [I 2022-10-09 04:28:00] ...trap/engines/engine.py.321: eval: epoch 31 | batch 350/510 [I 2022-10-09 04:28:00] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-09 04:28:00] ...trap/engines/engine.py.326: process: 0.31838 | load: 0.00021 [I 2022-10-09 04:28:03] ...trap/engines/engine.py.321: eval: epoch 31 | batch 360/510 [I 2022-10-09 04:28:03] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-09 04:28:03] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00020 [I 2022-10-09 04:28:06] ...trap/engines/engine.py.321: eval: epoch 31 | batch 370/510 [I 2022-10-09 04:28:06] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-09 04:28:06] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00020 [I 2022-10-09 04:28:09] ...trap/engines/engine.py.321: eval: epoch 31 | batch 380/510 [I 2022-10-09 04:28:09] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-09 04:28:09] ...trap/engines/engine.py.326: process: 0.32055 | load: 0.00021 [I 2022-10-09 04:28:13] ...trap/engines/engine.py.321: eval: epoch 31 | batch 390/510 [I 2022-10-09 04:28:13] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-09 04:28:13] ...trap/engines/engine.py.326: process: 0.32384 | load: 0.00021 [I 2022-10-09 04:28:16] ...trap/engines/engine.py.321: eval: epoch 31 | batch 400/510 [I 2022-10-09 04:28:16] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-09 04:28:16] ...trap/engines/engine.py.326: process: 0.31533 | load: 0.00021 [I 2022-10-09 04:28:19] ...trap/engines/engine.py.321: eval: epoch 31 | batch 410/510 [I 2022-10-09 04:28:19] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:32 [I 2022-10-09 04:28:19] ...trap/engines/engine.py.326: process: 0.32564 | load: 0.00020 [I 2022-10-09 04:28:22] ...trap/engines/engine.py.321: eval: epoch 31 | batch 420/510 [I 2022-10-09 04:28:22] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 04:28:22] ...trap/engines/engine.py.326: process: 0.32434 | load: 0.00020 [I 2022-10-09 04:28:25] ...trap/engines/engine.py.321: eval: epoch 31 | batch 430/510 [I 2022-10-09 04:28:25] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-09 04:28:25] ...trap/engines/engine.py.326: process: 0.32288 | load: 0.00020 [I 2022-10-09 04:28:29] ...trap/engines/engine.py.321: eval: epoch 31 | batch 440/510 [I 2022-10-09 04:28:29] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-09 04:28:29] ...trap/engines/engine.py.326: process: 0.32054 | load: 0.00020 [I 2022-10-09 04:28:32] ...trap/engines/engine.py.321: eval: epoch 31 | batch 450/510 [I 2022-10-09 04:28:32] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-09 04:28:32] ...trap/engines/engine.py.326: process: 0.33045 | load: 0.00021 [I 2022-10-09 04:28:35] ...trap/engines/engine.py.321: eval: epoch 31 | batch 460/510 [I 2022-10-09 04:28:35] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:16 [I 2022-10-09 04:28:35] ...trap/engines/engine.py.326: process: 0.32177 | load: 0.00020 [I 2022-10-09 04:28:38] ...trap/engines/engine.py.321: eval: epoch 31 | batch 470/510 [I 2022-10-09 04:28:38] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 04:28:38] ...trap/engines/engine.py.326: process: 0.32102 | load: 0.00020 [I 2022-10-09 04:28:41] ...trap/engines/engine.py.321: eval: epoch 31 | batch 480/510 [I 2022-10-09 04:28:41] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-09 04:28:41] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00020 [I 2022-10-09 04:28:45] ...trap/engines/engine.py.321: eval: epoch 31 | batch 490/510 [I 2022-10-09 04:28:45] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-09 04:28:45] ...trap/engines/engine.py.326: process: 0.32088 | load: 0.00020 [I 2022-10-09 04:28:48] ...trap/engines/engine.py.321: eval: epoch 31 | batch 500/510 [I 2022-10-09 04:28:48] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-09 04:28:48] ...trap/engines/engine.py.326: process: 0.32223 | load: 0.00020 [I 2022-10-09 04:28:51] ...trap/engines/engine.py.321: eval: epoch 31 | batch 510/510 [I 2022-10-09 04:28:51] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-09 04:28:51] ...trap/engines/engine.py.326: process: 0.31202 | load: 0.00016 [S 2022-10-09 04:28:52] ...trap/engines/engine.py.132: eval_epoch.epoch: 31 [I 2022-10-09 04:28:52] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.693700 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.896600 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934900 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.685300 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.894700 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.929400 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009706 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007338 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008264 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013639 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007086 [S 2022-10-09 04:29:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007513 [I 2022-10-09 04:29:32] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 04:29:32] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 04:29:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 04:29:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 04:29:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 04:29:34] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 04:29:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 04:29:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 04:29:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 04:29:42] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 04:29:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 04:29:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 04:29:51] ...trap/engines/engine.py.163: Training model on trainset for epoch 32 [I 2022-10-09 04:29:51] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 04:29:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 0/2304 [I 2022-10-09 04:29:54] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:59:17 [I 2022-10-09 04:29:54] ...trap/engines/engine.py.229: process: 3.10671 | load: 2.22476 [I 2022-10-09 04:29:54] ...trap/engines/engine.py.230: loss: 0.20101 [I 2022-10-09 04:29:54] ...optimizers/trijoint.py.151: total_norm: 26.871645 [I 2022-10-09 04:30:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 10/2304 [I 2022-10-09 04:30:02] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:32 [I 2022-10-09 04:30:02] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00708 [I 2022-10-09 04:30:02] ...trap/engines/engine.py.230: loss: 0.17839 [I 2022-10-09 04:30:02] ...optimizers/trijoint.py.151: total_norm: 13.457551 [I 2022-10-09 04:30:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 20/2304 [I 2022-10-09 04:30:11] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-09 04:30:11] ...trap/engines/engine.py.229: process: 0.83444 | load: 0.00474 [I 2022-10-09 04:30:11] ...trap/engines/engine.py.230: loss: 0.15168 [I 2022-10-09 04:30:11] ...optimizers/trijoint.py.151: total_norm: 10.944176 [I 2022-10-09 04:30:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 30/2304 [I 2022-10-09 04:30:19] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:57 [I 2022-10-09 04:30:19] ...trap/engines/engine.py.229: process: 0.83208 | load: 0.00474 [I 2022-10-09 04:30:19] ...trap/engines/engine.py.230: loss: 0.17691 [I 2022-10-09 04:30:19] ...optimizers/trijoint.py.151: total_norm: 12.578737 [I 2022-10-09 04:30:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 40/2304 [I 2022-10-09 04:30:27] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:40 [I 2022-10-09 04:30:27] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00399 [I 2022-10-09 04:30:27] ...trap/engines/engine.py.230: loss: 0.17310 [I 2022-10-09 04:30:27] ...optimizers/trijoint.py.151: total_norm: 14.696314 [I 2022-10-09 04:30:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 50/2304 [I 2022-10-09 04:30:36] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:39 [I 2022-10-09 04:30:36] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00417 [I 2022-10-09 04:30:36] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-09 04:30:36] ...optimizers/trijoint.py.151: total_norm: 8.710496 [I 2022-10-09 04:30:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 60/2304 [I 2022-10-09 04:30:44] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:10 [I 2022-10-09 04:30:44] ...trap/engines/engine.py.229: process: 0.82768 | load: 0.00366 [I 2022-10-09 04:30:44] ...trap/engines/engine.py.230: loss: 0.18085 [I 2022-10-09 04:30:44] ...optimizers/trijoint.py.151: total_norm: 13.739819 [I 2022-10-09 04:30:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 70/2304 [I 2022-10-09 04:30:53] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:18 [I 2022-10-09 04:30:53] ...trap/engines/engine.py.229: process: 0.83195 | load: 0.00348 [I 2022-10-09 04:30:53] ...trap/engines/engine.py.230: loss: 0.21067 [I 2022-10-09 04:30:53] ...optimizers/trijoint.py.151: total_norm: 18.907806 [I 2022-10-09 04:31:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 80/2304 [I 2022-10-09 04:31:01] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:05 [I 2022-10-09 04:31:01] ...trap/engines/engine.py.229: process: 0.83242 | load: 0.00356 [I 2022-10-09 04:31:01] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-09 04:31:01] ...optimizers/trijoint.py.151: total_norm: 15.200141 [I 2022-10-09 04:31:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 90/2304 [I 2022-10-09 04:31:09] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:49 [I 2022-10-09 04:31:09] ...trap/engines/engine.py.229: process: 0.82904 | load: 0.00318 [I 2022-10-09 04:31:09] ...trap/engines/engine.py.230: loss: 0.23134 [I 2022-10-09 04:31:09] ...optimizers/trijoint.py.151: total_norm: 15.573068 [I 2022-10-09 04:31:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 100/2304 [I 2022-10-09 04:31:18] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:45 [I 2022-10-09 04:31:18] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00330 [I 2022-10-09 04:31:18] ...trap/engines/engine.py.230: loss: 0.13791 [I 2022-10-09 04:31:18] ...optimizers/trijoint.py.151: total_norm: 11.205821 [I 2022-10-09 04:31:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 110/2304 [I 2022-10-09 04:31:26] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:55 [I 2022-10-09 04:31:26] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00341 [I 2022-10-09 04:31:26] ...trap/engines/engine.py.230: loss: 0.10177 [I 2022-10-09 04:31:26] ...optimizers/trijoint.py.151: total_norm: 12.998504 [I 2022-10-09 04:31:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 120/2304 [I 2022-10-09 04:31:35] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:40 [I 2022-10-09 04:31:35] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00341 [I 2022-10-09 04:31:35] ...trap/engines/engine.py.230: loss: 0.19279 [I 2022-10-09 04:31:35] ...optimizers/trijoint.py.151: total_norm: 14.536385 [I 2022-10-09 04:31:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 130/2304 [I 2022-10-09 04:31:43] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:27 [I 2022-10-09 04:31:43] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00329 [I 2022-10-09 04:31:43] ...trap/engines/engine.py.230: loss: 0.16490 [I 2022-10-09 04:31:43] ...optimizers/trijoint.py.151: total_norm: 23.620197 [I 2022-10-09 04:31:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 140/2304 [I 2022-10-09 04:31:52] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:19 [I 2022-10-09 04:31:52] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00297 [I 2022-10-09 04:31:52] ...trap/engines/engine.py.230: loss: 0.19754 [I 2022-10-09 04:31:52] ...optimizers/trijoint.py.151: total_norm: 15.881395 [I 2022-10-09 04:32:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 150/2304 [I 2022-10-09 04:32:00] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:23 [I 2022-10-09 04:32:00] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00316 [I 2022-10-09 04:32:00] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-09 04:32:00] ...optimizers/trijoint.py.151: total_norm: 20.007542 [I 2022-10-09 04:32:08] ...trap/engines/engine.py.224: train: epoch 32 | batch 160/2304 [I 2022-10-09 04:32:08] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:01 [I 2022-10-09 04:32:08] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00312 [I 2022-10-09 04:32:08] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-09 04:32:08] ...optimizers/trijoint.py.151: total_norm: 7.017996 [I 2022-10-09 04:32:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 170/2304 [I 2022-10-09 04:32:17] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:52 [I 2022-10-09 04:32:17] ...trap/engines/engine.py.229: process: 0.83372 | load: 0.00332 [I 2022-10-09 04:32:17] ...trap/engines/engine.py.230: loss: 0.13294 [I 2022-10-09 04:32:17] ...optimizers/trijoint.py.151: total_norm: 15.994435 [I 2022-10-09 04:32:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 180/2304 [I 2022-10-09 04:32:25] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:40 [I 2022-10-09 04:32:25] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00311 [I 2022-10-09 04:32:25] ...trap/engines/engine.py.230: loss: 0.18611 [I 2022-10-09 04:32:25] ...optimizers/trijoint.py.151: total_norm: 22.197212 [I 2022-10-09 04:32:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 190/2304 [I 2022-10-09 04:32:34] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:29 [I 2022-10-09 04:32:34] ...trap/engines/engine.py.229: process: 0.83226 | load: 0.00300 [I 2022-10-09 04:32:34] ...trap/engines/engine.py.230: loss: 0.17536 [I 2022-10-09 04:32:34] ...optimizers/trijoint.py.151: total_norm: 15.241037 [I 2022-10-09 04:32:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 200/2304 [I 2022-10-09 04:32:42] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:36 [I 2022-10-09 04:32:42] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00302 [I 2022-10-09 04:32:42] ...trap/engines/engine.py.230: loss: 0.18322 [I 2022-10-09 04:32:42] ...optimizers/trijoint.py.151: total_norm: 33.888836 [I 2022-10-09 04:32:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 210/2304 [I 2022-10-09 04:32:51] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:14 [I 2022-10-09 04:32:51] ...trap/engines/engine.py.229: process: 0.82414 | load: 0.00334 [I 2022-10-09 04:32:51] ...trap/engines/engine.py.230: loss: 0.19966 [I 2022-10-09 04:32:51] ...optimizers/trijoint.py.151: total_norm: 18.042425 [I 2022-10-09 04:32:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 220/2304 [I 2022-10-09 04:32:59] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:05 [I 2022-10-09 04:32:59] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00314 [I 2022-10-09 04:32:59] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-09 04:32:59] ...optimizers/trijoint.py.151: total_norm: 12.383771 [I 2022-10-09 04:33:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 230/2304 [I 2022-10-09 04:33:07] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:08 [I 2022-10-09 04:33:07] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00319 [I 2022-10-09 04:33:07] ...trap/engines/engine.py.230: loss: 0.14894 [I 2022-10-09 04:33:07] ...optimizers/trijoint.py.151: total_norm: 25.706579 [I 2022-10-09 04:33:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 240/2304 [I 2022-10-09 04:33:16] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:53 [I 2022-10-09 04:33:16] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00298 [I 2022-10-09 04:33:16] ...trap/engines/engine.py.230: loss: 0.17004 [I 2022-10-09 04:33:16] ...optimizers/trijoint.py.151: total_norm: 18.357956 [I 2022-10-09 04:33:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 250/2304 [I 2022-10-09 04:33:24] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:48 [I 2022-10-09 04:33:24] ...trap/engines/engine.py.229: process: 0.85369 | load: 0.00300 [I 2022-10-09 04:33:24] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-09 04:33:24] ...optimizers/trijoint.py.151: total_norm: 13.042796 [I 2022-10-09 04:33:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 260/2304 [I 2022-10-09 04:33:33] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:42 [I 2022-10-09 04:33:33] ...trap/engines/engine.py.229: process: 0.84413 | load: 0.00299 [I 2022-10-09 04:33:33] ...trap/engines/engine.py.230: loss: 0.24789 [I 2022-10-09 04:33:33] ...optimizers/trijoint.py.151: total_norm: 32.108070 [I 2022-10-09 04:33:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 270/2304 [I 2022-10-09 04:33:41] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:27 [I 2022-10-09 04:33:41] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00299 [I 2022-10-09 04:33:41] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-09 04:33:41] ...optimizers/trijoint.py.151: total_norm: 17.697788 [I 2022-10-09 04:33:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 280/2304 [I 2022-10-09 04:33:50] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:17 [I 2022-10-09 04:33:50] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00314 [I 2022-10-09 04:33:50] ...trap/engines/engine.py.230: loss: 0.16615 [I 2022-10-09 04:33:50] ...optimizers/trijoint.py.151: total_norm: 18.167206 [I 2022-10-09 04:33:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 290/2304 [I 2022-10-09 04:33:58] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:21 [I 2022-10-09 04:33:58] ...trap/engines/engine.py.229: process: 0.84191 | load: 0.00315 [I 2022-10-09 04:33:58] ...trap/engines/engine.py.230: loss: 0.17719 [I 2022-10-09 04:33:58] ...optimizers/trijoint.py.151: total_norm: 17.047369 [I 2022-10-09 04:34:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 300/2304 [I 2022-10-09 04:34:06] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:13 [I 2022-10-09 04:34:06] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00300 [I 2022-10-09 04:34:06] ...trap/engines/engine.py.230: loss: 0.15087 [I 2022-10-09 04:34:06] ...optimizers/trijoint.py.151: total_norm: 11.459832 [I 2022-10-09 04:34:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 310/2304 [I 2022-10-09 04:34:15] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:49 [I 2022-10-09 04:34:15] ...trap/engines/engine.py.229: process: 0.83296 | load: 0.00300 [I 2022-10-09 04:34:15] ...trap/engines/engine.py.230: loss: 0.14369 [I 2022-10-09 04:34:15] ...optimizers/trijoint.py.151: total_norm: 15.046603 [I 2022-10-09 04:34:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 320/2304 [I 2022-10-09 04:34:23] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:42 [I 2022-10-09 04:34:23] ...trap/engines/engine.py.229: process: 0.83208 | load: 0.00303 [I 2022-10-09 04:34:23] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-09 04:34:23] ...optimizers/trijoint.py.151: total_norm: 14.607347 [I 2022-10-09 04:34:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 330/2304 [I 2022-10-09 04:34:32] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:44 [I 2022-10-09 04:34:32] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00303 [I 2022-10-09 04:34:32] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-09 04:34:32] ...optimizers/trijoint.py.151: total_norm: 13.030872 [I 2022-10-09 04:34:40] ...trap/engines/engine.py.224: train: epoch 32 | batch 340/2304 [I 2022-10-09 04:34:40] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:32 [I 2022-10-09 04:34:40] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00302 [I 2022-10-09 04:34:40] ...trap/engines/engine.py.230: loss: 0.27133 [I 2022-10-09 04:34:40] ...optimizers/trijoint.py.151: total_norm: 31.592764 [I 2022-10-09 04:34:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 350/2304 [I 2022-10-09 04:34:48] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:19 [I 2022-10-09 04:34:48] ...trap/engines/engine.py.229: process: 0.82699 | load: 0.00303 [I 2022-10-09 04:34:48] ...trap/engines/engine.py.230: loss: 0.19583 [I 2022-10-09 04:34:48] ...optimizers/trijoint.py.151: total_norm: 14.125990 [I 2022-10-09 04:34:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 360/2304 [I 2022-10-09 04:34:57] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:18 [I 2022-10-09 04:34:57] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00299 [I 2022-10-09 04:34:57] ...trap/engines/engine.py.230: loss: 0.13538 [I 2022-10-09 04:34:57] ...optimizers/trijoint.py.151: total_norm: 9.193571 [I 2022-10-09 04:35:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 370/2304 [I 2022-10-09 04:35:05] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:26:59 [I 2022-10-09 04:35:05] ...trap/engines/engine.py.229: process: 0.83561 | load: 0.00303 [I 2022-10-09 04:35:05] ...trap/engines/engine.py.230: loss: 0.15842 [I 2022-10-09 04:35:05] ...optimizers/trijoint.py.151: total_norm: 12.968761 [I 2022-10-09 04:35:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 380/2304 [I 2022-10-09 04:35:14] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:56 [I 2022-10-09 04:35:14] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00298 [I 2022-10-09 04:35:14] ...trap/engines/engine.py.230: loss: 0.13644 [I 2022-10-09 04:35:14] ...optimizers/trijoint.py.151: total_norm: 14.902481 [I 2022-10-09 04:35:22] ...trap/engines/engine.py.224: train: epoch 32 | batch 390/2304 [I 2022-10-09 04:35:22] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:51 [I 2022-10-09 04:35:22] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00301 [I 2022-10-09 04:35:22] ...trap/engines/engine.py.230: loss: 0.27927 [I 2022-10-09 04:35:22] ...optimizers/trijoint.py.151: total_norm: 18.586241 [I 2022-10-09 04:35:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 400/2304 [I 2022-10-09 04:35:31] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:39 [I 2022-10-09 04:35:31] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00299 [I 2022-10-09 04:35:31] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-09 04:35:31] ...optimizers/trijoint.py.151: total_norm: 12.007090 [I 2022-10-09 04:35:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 410/2304 [I 2022-10-09 04:35:39] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:38 [I 2022-10-09 04:35:39] ...trap/engines/engine.py.229: process: 0.84443 | load: 0.00313 [I 2022-10-09 04:35:39] ...trap/engines/engine.py.230: loss: 0.19331 [I 2022-10-09 04:35:39] ...optimizers/trijoint.py.151: total_norm: 24.983946 [I 2022-10-09 04:35:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 420/2304 [I 2022-10-09 04:35:47] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:28 [I 2022-10-09 04:35:47] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00297 [I 2022-10-09 04:35:47] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-09 04:35:47] ...optimizers/trijoint.py.151: total_norm: 12.386975 [I 2022-10-09 04:35:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 430/2304 [I 2022-10-09 04:35:56] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:10 [I 2022-10-09 04:35:56] ...trap/engines/engine.py.229: process: 0.83535 | load: 0.00303 [I 2022-10-09 04:35:56] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-09 04:35:56] ...optimizers/trijoint.py.151: total_norm: 14.639687 [I 2022-10-09 04:36:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 440/2304 [I 2022-10-09 04:36:04] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:25:55 [I 2022-10-09 04:36:04] ...trap/engines/engine.py.229: process: 0.83088 | load: 0.00298 [I 2022-10-09 04:36:04] ...trap/engines/engine.py.230: loss: 0.25369 [I 2022-10-09 04:36:04] ...optimizers/trijoint.py.151: total_norm: 25.427473 [I 2022-10-09 04:36:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 450/2304 [I 2022-10-09 04:36:13] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:05 [I 2022-10-09 04:36:13] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00300 [I 2022-10-09 04:36:13] ...trap/engines/engine.py.230: loss: 0.16123 [I 2022-10-09 04:36:13] ...optimizers/trijoint.py.151: total_norm: 29.309776 [I 2022-10-09 04:36:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 460/2304 [I 2022-10-09 04:36:21] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:48 [I 2022-10-09 04:36:21] ...trap/engines/engine.py.229: process: 0.82743 | load: 0.00298 [I 2022-10-09 04:36:21] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-09 04:36:21] ...optimizers/trijoint.py.151: total_norm: 10.493435 [I 2022-10-09 04:36:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 470/2304 [I 2022-10-09 04:36:30] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:35 [I 2022-10-09 04:36:30] ...trap/engines/engine.py.229: process: 0.83895 | load: 0.00299 [I 2022-10-09 04:36:30] ...trap/engines/engine.py.230: loss: 0.11473 [I 2022-10-09 04:36:30] ...optimizers/trijoint.py.151: total_norm: 14.883795 [I 2022-10-09 04:36:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 480/2304 [I 2022-10-09 04:36:38] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:34 [I 2022-10-09 04:36:38] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00299 [I 2022-10-09 04:36:38] ...trap/engines/engine.py.230: loss: 0.20009 [I 2022-10-09 04:36:38] ...optimizers/trijoint.py.151: total_norm: 27.329689 [I 2022-10-09 04:36:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 490/2304 [I 2022-10-09 04:36:46] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:21 [I 2022-10-09 04:36:46] ...trap/engines/engine.py.229: process: 0.83492 | load: 0.00297 [I 2022-10-09 04:36:46] ...trap/engines/engine.py.230: loss: 0.10082 [I 2022-10-09 04:36:46] ...optimizers/trijoint.py.151: total_norm: 10.563850 [I 2022-10-09 04:36:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 500/2304 [I 2022-10-09 04:36:55] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:14 [I 2022-10-09 04:36:55] ...trap/engines/engine.py.229: process: 0.83922 | load: 0.00297 [I 2022-10-09 04:36:55] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-09 04:36:55] ...optimizers/trijoint.py.151: total_norm: 13.883194 [I 2022-10-09 04:37:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 510/2304 [I 2022-10-09 04:37:03] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:10 [I 2022-10-09 04:37:03] ...trap/engines/engine.py.229: process: 0.84530 | load: 0.00302 [I 2022-10-09 04:37:03] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-09 04:37:03] ...optimizers/trijoint.py.151: total_norm: 13.627218 [I 2022-10-09 04:37:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 520/2304 [I 2022-10-09 04:37:12] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:02 [I 2022-10-09 04:37:12] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00298 [I 2022-10-09 04:37:12] ...trap/engines/engine.py.230: loss: 0.20264 [I 2022-10-09 04:37:12] ...optimizers/trijoint.py.151: total_norm: 18.372156 [I 2022-10-09 04:37:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 530/2304 [I 2022-10-09 04:37:20] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:57 [I 2022-10-09 04:37:20] ...trap/engines/engine.py.229: process: 0.84448 | load: 0.00304 [I 2022-10-09 04:37:20] ...trap/engines/engine.py.230: loss: 0.14698 [I 2022-10-09 04:37:20] ...optimizers/trijoint.py.151: total_norm: 12.787724 [I 2022-10-09 04:37:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 540/2304 [I 2022-10-09 04:37:29] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:42 [I 2022-10-09 04:37:29] ...trap/engines/engine.py.229: process: 0.83942 | load: 0.00299 [I 2022-10-09 04:37:29] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-09 04:37:29] ...optimizers/trijoint.py.151: total_norm: 11.953916 [I 2022-10-09 04:37:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 550/2304 [I 2022-10-09 04:37:37] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:25 [I 2022-10-09 04:37:37] ...trap/engines/engine.py.229: process: 0.83268 | load: 0.00313 [I 2022-10-09 04:37:37] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-09 04:37:37] ...optimizers/trijoint.py.151: total_norm: 26.338614 [I 2022-10-09 04:37:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 560/2304 [I 2022-10-09 04:37:45] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:25 [I 2022-10-09 04:37:45] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00305 [I 2022-10-09 04:37:45] ...trap/engines/engine.py.230: loss: 0.16913 [I 2022-10-09 04:37:45] ...optimizers/trijoint.py.151: total_norm: 13.162533 [I 2022-10-09 04:37:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 570/2304 [I 2022-10-09 04:37:54] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:06 [I 2022-10-09 04:37:54] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00298 [I 2022-10-09 04:37:54] ...trap/engines/engine.py.230: loss: 0.09212 [I 2022-10-09 04:37:54] ...optimizers/trijoint.py.151: total_norm: 13.571147 [I 2022-10-09 04:38:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 580/2304 [I 2022-10-09 04:38:02] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:15 [I 2022-10-09 04:38:02] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00302 [I 2022-10-09 04:38:02] ...trap/engines/engine.py.230: loss: 0.11999 [I 2022-10-09 04:38:02] ...optimizers/trijoint.py.151: total_norm: 12.258655 [I 2022-10-09 04:38:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 590/2304 [I 2022-10-09 04:38:11] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:23:56 [I 2022-10-09 04:38:11] ...trap/engines/engine.py.229: process: 0.82727 | load: 0.00303 [I 2022-10-09 04:38:11] ...trap/engines/engine.py.230: loss: 0.11456 [I 2022-10-09 04:38:11] ...optimizers/trijoint.py.151: total_norm: 11.585257 [I 2022-10-09 04:38:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 600/2304 [I 2022-10-09 04:38:19] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:40 [I 2022-10-09 04:38:19] ...trap/engines/engine.py.229: process: 0.82916 | load: 0.00300 [I 2022-10-09 04:38:19] ...trap/engines/engine.py.230: loss: 0.16280 [I 2022-10-09 04:38:19] ...optimizers/trijoint.py.151: total_norm: 17.183229 [I 2022-10-09 04:38:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 610/2304 [I 2022-10-09 04:38:27] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:23:39 [I 2022-10-09 04:38:27] ...trap/engines/engine.py.229: process: 0.82817 | load: 0.00303 [I 2022-10-09 04:38:27] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-09 04:38:27] ...optimizers/trijoint.py.151: total_norm: 11.810128 [I 2022-10-09 04:38:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 620/2304 [I 2022-10-09 04:38:36] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:37 [I 2022-10-09 04:38:36] ...trap/engines/engine.py.229: process: 0.85237 | load: 0.00303 [I 2022-10-09 04:38:36] ...trap/engines/engine.py.230: loss: 0.17348 [I 2022-10-09 04:38:36] ...optimizers/trijoint.py.151: total_norm: 20.983818 [I 2022-10-09 04:38:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 630/2304 [I 2022-10-09 04:38:44] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:32 [I 2022-10-09 04:38:44] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00300 [I 2022-10-09 04:38:44] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-09 04:38:44] ...optimizers/trijoint.py.151: total_norm: 49.973965 [I 2022-10-09 04:38:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 640/2304 [I 2022-10-09 04:38:53] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:31 [I 2022-10-09 04:38:53] ...trap/engines/engine.py.229: process: 0.83891 | load: 0.00301 [I 2022-10-09 04:38:53] ...trap/engines/engine.py.230: loss: 0.14912 [I 2022-10-09 04:38:53] ...optimizers/trijoint.py.151: total_norm: 43.737865 [I 2022-10-09 04:39:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 650/2304 [I 2022-10-09 04:39:01] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:16 [I 2022-10-09 04:39:01] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00300 [I 2022-10-09 04:39:01] ...trap/engines/engine.py.230: loss: 0.12609 [I 2022-10-09 04:39:01] ...optimizers/trijoint.py.151: total_norm: 20.283777 [I 2022-10-09 04:39:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 660/2304 [I 2022-10-09 04:39:09] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:08 [I 2022-10-09 04:39:09] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00303 [I 2022-10-09 04:39:09] ...trap/engines/engine.py.230: loss: 0.12101 [I 2022-10-09 04:39:09] ...optimizers/trijoint.py.151: total_norm: 12.028399 [I 2022-10-09 04:39:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 670/2304 [I 2022-10-09 04:39:18] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:07 [I 2022-10-09 04:39:18] ...trap/engines/engine.py.229: process: 0.86296 | load: 0.00309 [I 2022-10-09 04:39:18] ...trap/engines/engine.py.230: loss: 0.13454 [I 2022-10-09 04:39:18] ...optimizers/trijoint.py.151: total_norm: 12.864436 [I 2022-10-09 04:39:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 680/2304 [I 2022-10-09 04:39:26] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:49 [I 2022-10-09 04:39:26] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00302 [I 2022-10-09 04:39:26] ...trap/engines/engine.py.230: loss: 0.20588 [I 2022-10-09 04:39:26] ...optimizers/trijoint.py.151: total_norm: 14.729767 [I 2022-10-09 04:39:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 690/2304 [I 2022-10-09 04:39:35] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:47 [I 2022-10-09 04:39:35] ...trap/engines/engine.py.229: process: 0.86659 | load: 0.00300 [I 2022-10-09 04:39:35] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-09 04:39:35] ...optimizers/trijoint.py.151: total_norm: 19.148632 [I 2022-10-09 04:39:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 700/2304 [I 2022-10-09 04:39:43] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:34 [I 2022-10-09 04:39:43] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00302 [I 2022-10-09 04:39:43] ...trap/engines/engine.py.230: loss: 0.27554 [I 2022-10-09 04:39:43] ...optimizers/trijoint.py.151: total_norm: 27.013906 [I 2022-10-09 04:39:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 710/2304 [I 2022-10-09 04:39:52] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:14 [I 2022-10-09 04:39:52] ...trap/engines/engine.py.229: process: 0.82681 | load: 0.00298 [I 2022-10-09 04:39:52] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-09 04:39:52] ...optimizers/trijoint.py.151: total_norm: 16.531759 [I 2022-10-09 04:40:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 720/2304 [I 2022-10-09 04:40:00] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:12 [I 2022-10-09 04:40:00] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00301 [I 2022-10-09 04:40:00] ...trap/engines/engine.py.230: loss: 0.23335 [I 2022-10-09 04:40:00] ...optimizers/trijoint.py.151: total_norm: 14.940853 [I 2022-10-09 04:40:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 730/2304 [I 2022-10-09 04:40:09] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:05 [I 2022-10-09 04:40:09] ...trap/engines/engine.py.229: process: 0.83337 | load: 0.00301 [I 2022-10-09 04:40:09] ...trap/engines/engine.py.230: loss: 0.12935 [I 2022-10-09 04:40:09] ...optimizers/trijoint.py.151: total_norm: 9.663870 [I 2022-10-09 04:40:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 740/2304 [I 2022-10-09 04:40:17] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:56 [I 2022-10-09 04:40:17] ...trap/engines/engine.py.229: process: 0.84743 | load: 0.00297 [I 2022-10-09 04:40:17] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-09 04:40:17] ...optimizers/trijoint.py.151: total_norm: 9.469055 [I 2022-10-09 04:40:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 750/2304 [I 2022-10-09 04:40:25] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:41 [I 2022-10-09 04:40:25] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00299 [I 2022-10-09 04:40:25] ...trap/engines/engine.py.230: loss: 0.24114 [I 2022-10-09 04:40:25] ...optimizers/trijoint.py.151: total_norm: 24.858814 [I 2022-10-09 04:40:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 760/2304 [I 2022-10-09 04:40:34] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:35 [I 2022-10-09 04:40:34] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00299 [I 2022-10-09 04:40:34] ...trap/engines/engine.py.230: loss: 0.19149 [I 2022-10-09 04:40:34] ...optimizers/trijoint.py.151: total_norm: 22.819197 [I 2022-10-09 04:40:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 770/2304 [I 2022-10-09 04:40:42] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:23 [I 2022-10-09 04:40:42] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00299 [I 2022-10-09 04:40:42] ...trap/engines/engine.py.230: loss: 0.22124 [I 2022-10-09 04:40:42] ...optimizers/trijoint.py.151: total_norm: 13.532339 [I 2022-10-09 04:40:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 780/2304 [I 2022-10-09 04:40:51] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:19 [I 2022-10-09 04:40:51] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00301 [I 2022-10-09 04:40:51] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-09 04:40:51] ...optimizers/trijoint.py.151: total_norm: 10.478580 [I 2022-10-09 04:40:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 790/2304 [I 2022-10-09 04:40:59] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:19 [I 2022-10-09 04:40:59] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00301 [I 2022-10-09 04:40:59] ...trap/engines/engine.py.230: loss: 0.19834 [I 2022-10-09 04:40:59] ...optimizers/trijoint.py.151: total_norm: 13.785627 [I 2022-10-09 04:41:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 800/2304 [I 2022-10-09 04:41:07] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:20:56 [I 2022-10-09 04:41:07] ...trap/engines/engine.py.229: process: 0.82705 | load: 0.00300 [I 2022-10-09 04:41:07] ...trap/engines/engine.py.230: loss: 0.15931 [I 2022-10-09 04:41:07] ...optimizers/trijoint.py.151: total_norm: 12.681673 [I 2022-10-09 04:41:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 810/2304 [I 2022-10-09 04:41:16] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:56 [I 2022-10-09 04:41:16] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00301 [I 2022-10-09 04:41:16] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-09 04:41:16] ...optimizers/trijoint.py.151: total_norm: 14.064596 [I 2022-10-09 04:41:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 820/2304 [I 2022-10-09 04:41:24] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:20:42 [I 2022-10-09 04:41:24] ...trap/engines/engine.py.229: process: 0.82770 | load: 0.00301 [I 2022-10-09 04:41:24] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-09 04:41:24] ...optimizers/trijoint.py.151: total_norm: 23.738232 [I 2022-10-09 04:41:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 830/2304 [I 2022-10-09 04:41:33] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:40 [I 2022-10-09 04:41:33] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00303 [I 2022-10-09 04:41:33] ...trap/engines/engine.py.230: loss: 0.23703 [I 2022-10-09 04:41:33] ...optimizers/trijoint.py.151: total_norm: 33.069870 [I 2022-10-09 04:41:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 840/2304 [I 2022-10-09 04:41:41] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:33 [I 2022-10-09 04:41:41] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00301 [I 2022-10-09 04:41:41] ...trap/engines/engine.py.230: loss: 0.17545 [I 2022-10-09 04:41:41] ...optimizers/trijoint.py.151: total_norm: 26.653891 [I 2022-10-09 04:41:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 850/2304 [I 2022-10-09 04:41:50] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:19 [I 2022-10-09 04:41:50] ...trap/engines/engine.py.229: process: 0.83330 | load: 0.00307 [I 2022-10-09 04:41:50] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-09 04:41:50] ...optimizers/trijoint.py.151: total_norm: 13.731226 [I 2022-10-09 04:41:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 860/2304 [I 2022-10-09 04:41:58] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:15 [I 2022-10-09 04:41:58] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00303 [I 2022-10-09 04:41:58] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-09 04:41:58] ...optimizers/trijoint.py.151: total_norm: 18.826744 [I 2022-10-09 04:42:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 870/2304 [I 2022-10-09 04:42:06] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:03 [I 2022-10-09 04:42:06] ...trap/engines/engine.py.229: process: 0.83237 | load: 0.00304 [I 2022-10-09 04:42:06] ...trap/engines/engine.py.230: loss: 0.16109 [I 2022-10-09 04:42:06] ...optimizers/trijoint.py.151: total_norm: 9.136214 [I 2022-10-09 04:42:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 880/2304 [I 2022-10-09 04:42:15] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:52 [I 2022-10-09 04:42:15] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00299 [I 2022-10-09 04:42:15] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-09 04:42:15] ...optimizers/trijoint.py.151: total_norm: 10.993641 [I 2022-10-09 04:42:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 890/2304 [I 2022-10-09 04:42:23] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:19:45 [I 2022-10-09 04:42:23] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00301 [I 2022-10-09 04:42:23] ...trap/engines/engine.py.230: loss: 0.19397 [I 2022-10-09 04:42:23] ...optimizers/trijoint.py.151: total_norm: 17.300217 [I 2022-10-09 04:42:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 900/2304 [I 2022-10-09 04:42:32] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:19:36 [I 2022-10-09 04:42:32] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00308 [I 2022-10-09 04:42:32] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-09 04:42:32] ...optimizers/trijoint.py.151: total_norm: 22.385429 [I 2022-10-09 04:42:40] ...trap/engines/engine.py.224: train: epoch 32 | batch 910/2304 [I 2022-10-09 04:42:40] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:32 [I 2022-10-09 04:42:40] ...trap/engines/engine.py.229: process: 0.85068 | load: 0.00303 [I 2022-10-09 04:42:40] ...trap/engines/engine.py.230: loss: 0.13052 [I 2022-10-09 04:42:40] ...optimizers/trijoint.py.151: total_norm: 13.050543 [I 2022-10-09 04:42:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 920/2304 [I 2022-10-09 04:42:48] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:23 [I 2022-10-09 04:42:48] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00302 [I 2022-10-09 04:42:48] ...trap/engines/engine.py.230: loss: 0.18642 [I 2022-10-09 04:42:48] ...optimizers/trijoint.py.151: total_norm: 15.019684 [I 2022-10-09 04:42:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 930/2304 [I 2022-10-09 04:42:57] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:10 [I 2022-10-09 04:42:57] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00300 [I 2022-10-09 04:42:57] ...trap/engines/engine.py.230: loss: 0.16564 [I 2022-10-09 04:42:57] ...optimizers/trijoint.py.151: total_norm: 37.232998 [I 2022-10-09 04:43:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 940/2304 [I 2022-10-09 04:43:05] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:08 [I 2022-10-09 04:43:05] ...trap/engines/engine.py.229: process: 0.84434 | load: 0.00301 [I 2022-10-09 04:43:05] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-09 04:43:05] ...optimizers/trijoint.py.151: total_norm: 9.562943 [I 2022-10-09 04:43:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 950/2304 [I 2022-10-09 04:43:14] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:18:59 [I 2022-10-09 04:43:14] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00300 [I 2022-10-09 04:43:14] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-09 04:43:14] ...optimizers/trijoint.py.151: total_norm: 18.215639 [I 2022-10-09 04:43:22] ...trap/engines/engine.py.224: train: epoch 32 | batch 960/2304 [I 2022-10-09 04:43:22] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:18:47 [I 2022-10-09 04:43:22] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00298 [I 2022-10-09 04:43:22] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-09 04:43:22] ...optimizers/trijoint.py.151: total_norm: 10.496434 [I 2022-10-09 04:43:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 970/2304 [I 2022-10-09 04:43:30] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:18:44 [I 2022-10-09 04:43:30] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00305 [I 2022-10-09 04:43:30] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-09 04:43:30] ...optimizers/trijoint.py.151: total_norm: 51.021294 [I 2022-10-09 04:43:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 980/2304 [I 2022-10-09 04:43:39] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:18:32 [I 2022-10-09 04:43:39] ...trap/engines/engine.py.229: process: 0.84323 | load: 0.00302 [I 2022-10-09 04:43:39] ...trap/engines/engine.py.230: loss: 0.20839 [I 2022-10-09 04:43:39] ...optimizers/trijoint.py.151: total_norm: 19.305700 [I 2022-10-09 04:43:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 990/2304 [I 2022-10-09 04:43:47] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:26 [I 2022-10-09 04:43:47] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00300 [I 2022-10-09 04:43:47] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-09 04:43:47] ...optimizers/trijoint.py.151: total_norm: 19.351591 [I 2022-10-09 04:43:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 1000/2304 [I 2022-10-09 04:43:56] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:10 [I 2022-10-09 04:43:56] ...trap/engines/engine.py.229: process: 0.82938 | load: 0.00300 [I 2022-10-09 04:43:56] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-09 04:43:56] ...optimizers/trijoint.py.151: total_norm: 28.514803 [I 2022-10-09 04:44:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 1010/2304 [I 2022-10-09 04:44:04] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:00 [I 2022-10-09 04:44:04] ...trap/engines/engine.py.229: process: 0.83569 | load: 0.00298 [I 2022-10-09 04:44:04] ...trap/engines/engine.py.230: loss: 0.10894 [I 2022-10-09 04:44:04] ...optimizers/trijoint.py.151: total_norm: 11.381812 [I 2022-10-09 04:44:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 1020/2304 [I 2022-10-09 04:44:13] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:04 [I 2022-10-09 04:44:13] ...trap/engines/engine.py.229: process: 0.83563 | load: 0.00303 [I 2022-10-09 04:44:13] ...trap/engines/engine.py.230: loss: 0.18499 [I 2022-10-09 04:44:13] ...optimizers/trijoint.py.151: total_norm: 29.300615 [I 2022-10-09 04:44:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 1030/2304 [I 2022-10-09 04:44:21] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:17:56 [I 2022-10-09 04:44:21] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00301 [I 2022-10-09 04:44:21] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-09 04:44:21] ...optimizers/trijoint.py.151: total_norm: 18.590759 [I 2022-10-09 04:44:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 1040/2304 [I 2022-10-09 04:44:29] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:17:38 [I 2022-10-09 04:44:29] ...trap/engines/engine.py.229: process: 0.82240 | load: 0.00300 [I 2022-10-09 04:44:29] ...trap/engines/engine.py.230: loss: 0.16382 [I 2022-10-09 04:44:29] ...optimizers/trijoint.py.151: total_norm: 15.452298 [I 2022-10-09 04:44:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 1050/2304 [I 2022-10-09 04:44:38] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:17:44 [I 2022-10-09 04:44:38] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00300 [I 2022-10-09 04:44:38] ...trap/engines/engine.py.230: loss: 0.13912 [I 2022-10-09 04:44:38] ...optimizers/trijoint.py.151: total_norm: 12.785750 [I 2022-10-09 04:44:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 1060/2304 [I 2022-10-09 04:44:46] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:17:31 [I 2022-10-09 04:44:46] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00308 [I 2022-10-09 04:44:46] ...trap/engines/engine.py.230: loss: 0.16515 [I 2022-10-09 04:44:46] ...optimizers/trijoint.py.151: total_norm: 22.815323 [I 2022-10-09 04:44:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 1070/2304 [I 2022-10-09 04:44:55] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:19 [I 2022-10-09 04:44:55] ...trap/engines/engine.py.229: process: 0.83871 | load: 0.00303 [I 2022-10-09 04:44:55] ...trap/engines/engine.py.230: loss: 0.17544 [I 2022-10-09 04:44:55] ...optimizers/trijoint.py.151: total_norm: 17.370573 [I 2022-10-09 04:45:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 1080/2304 [I 2022-10-09 04:45:03] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:10 [I 2022-10-09 04:45:03] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00307 [I 2022-10-09 04:45:03] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-09 04:45:03] ...optimizers/trijoint.py.151: total_norm: 18.656548 [I 2022-10-09 04:45:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 1090/2304 [I 2022-10-09 04:45:12] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:16:55 [I 2022-10-09 04:45:12] ...trap/engines/engine.py.229: process: 0.82474 | load: 0.00304 [I 2022-10-09 04:45:12] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-09 04:45:12] ...optimizers/trijoint.py.151: total_norm: 55.070866 [I 2022-10-09 04:45:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 1100/2304 [I 2022-10-09 04:45:20] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:16:51 [I 2022-10-09 04:45:20] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00310 [I 2022-10-09 04:45:20] ...trap/engines/engine.py.230: loss: 0.19051 [I 2022-10-09 04:45:20] ...optimizers/trijoint.py.151: total_norm: 14.791439 [I 2022-10-09 04:45:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 1110/2304 [I 2022-10-09 04:45:29] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:16:39 [I 2022-10-09 04:45:29] ...trap/engines/engine.py.229: process: 0.83110 | load: 0.00312 [I 2022-10-09 04:45:29] ...trap/engines/engine.py.230: loss: 0.22266 [I 2022-10-09 04:45:29] ...optimizers/trijoint.py.151: total_norm: 21.998587 [I 2022-10-09 04:45:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 1120/2304 [I 2022-10-09 04:45:37] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:16:42 [I 2022-10-09 04:45:37] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00304 [I 2022-10-09 04:45:37] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-09 04:45:37] ...optimizers/trijoint.py.151: total_norm: 15.208067 [I 2022-10-09 04:45:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 1130/2304 [I 2022-10-09 04:45:46] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:16:32 [I 2022-10-09 04:45:46] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00298 [I 2022-10-09 04:45:46] ...trap/engines/engine.py.230: loss: 0.18127 [I 2022-10-09 04:45:46] ...optimizers/trijoint.py.151: total_norm: 16.422674 [I 2022-10-09 04:45:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 1140/2304 [I 2022-10-09 04:45:54] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:16:23 [I 2022-10-09 04:45:54] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00297 [I 2022-10-09 04:45:54] ...trap/engines/engine.py.230: loss: 0.14537 [I 2022-10-09 04:45:54] ...optimizers/trijoint.py.151: total_norm: 15.661608 [I 2022-10-09 04:46:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 1150/2304 [I 2022-10-09 04:46:02] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:13 [I 2022-10-09 04:46:02] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00300 [I 2022-10-09 04:46:02] ...trap/engines/engine.py.230: loss: 0.16680 [I 2022-10-09 04:46:02] ...optimizers/trijoint.py.151: total_norm: 17.763393 [I 2022-10-09 04:46:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 1160/2304 [I 2022-10-09 04:46:11] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:01 [I 2022-10-09 04:46:11] ...trap/engines/engine.py.229: process: 0.83599 | load: 0.00305 [I 2022-10-09 04:46:11] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-09 04:46:11] ...optimizers/trijoint.py.151: total_norm: 27.482771 [I 2022-10-09 04:46:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 1170/2304 [I 2022-10-09 04:46:19] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:15:57 [I 2022-10-09 04:46:19] ...trap/engines/engine.py.229: process: 0.85455 | load: 0.00300 [I 2022-10-09 04:46:19] ...trap/engines/engine.py.230: loss: 0.18894 [I 2022-10-09 04:46:19] ...optimizers/trijoint.py.151: total_norm: 23.481739 [I 2022-10-09 04:46:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 1180/2304 [I 2022-10-09 04:46:28] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:15:41 [I 2022-10-09 04:46:28] ...trap/engines/engine.py.229: process: 0.83043 | load: 0.00302 [I 2022-10-09 04:46:28] ...trap/engines/engine.py.230: loss: 0.24397 [I 2022-10-09 04:46:28] ...optimizers/trijoint.py.151: total_norm: 21.515673 [I 2022-10-09 04:46:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 1190/2304 [I 2022-10-09 04:46:36] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:15:40 [I 2022-10-09 04:46:36] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00303 [I 2022-10-09 04:46:36] ...trap/engines/engine.py.230: loss: 0.18916 [I 2022-10-09 04:46:36] ...optimizers/trijoint.py.151: total_norm: 19.364082 [I 2022-10-09 04:46:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 1200/2304 [I 2022-10-09 04:46:45] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:15:29 [I 2022-10-09 04:46:45] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00300 [I 2022-10-09 04:46:45] ...trap/engines/engine.py.230: loss: 0.16909 [I 2022-10-09 04:46:45] ...optimizers/trijoint.py.151: total_norm: 26.560726 [I 2022-10-09 04:46:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 1210/2304 [I 2022-10-09 04:46:53] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:21 [I 2022-10-09 04:46:53] ...trap/engines/engine.py.229: process: 0.83730 | load: 0.00302 [I 2022-10-09 04:46:53] ...trap/engines/engine.py.230: loss: 0.11076 [I 2022-10-09 04:46:53] ...optimizers/trijoint.py.151: total_norm: 17.100780 [I 2022-10-09 04:47:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 1220/2304 [I 2022-10-09 04:47:01] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:09 [I 2022-10-09 04:47:01] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00300 [I 2022-10-09 04:47:01] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-09 04:47:01] ...optimizers/trijoint.py.151: total_norm: 30.075890 [I 2022-10-09 04:47:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 1230/2304 [I 2022-10-09 04:47:10] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:05 [I 2022-10-09 04:47:10] ...trap/engines/engine.py.229: process: 0.83652 | load: 0.00301 [I 2022-10-09 04:47:10] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-09 04:47:10] ...optimizers/trijoint.py.151: total_norm: 24.657698 [I 2022-10-09 04:47:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 1240/2304 [I 2022-10-09 04:47:18] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:14:56 [I 2022-10-09 04:47:18] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00302 [I 2022-10-09 04:47:18] ...trap/engines/engine.py.230: loss: 0.13382 [I 2022-10-09 04:47:18] ...optimizers/trijoint.py.151: total_norm: 42.439571 [I 2022-10-09 04:47:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 1250/2304 [I 2022-10-09 04:47:27] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:14:43 [I 2022-10-09 04:47:27] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00300 [I 2022-10-09 04:47:27] ...trap/engines/engine.py.230: loss: 0.23335 [I 2022-10-09 04:47:27] ...optimizers/trijoint.py.151: total_norm: 27.425087 [I 2022-10-09 04:47:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 1260/2304 [I 2022-10-09 04:47:35] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:14:41 [I 2022-10-09 04:47:35] ...trap/engines/engine.py.229: process: 0.84598 | load: 0.00311 [I 2022-10-09 04:47:35] ...trap/engines/engine.py.230: loss: 0.11464 [I 2022-10-09 04:47:35] ...optimizers/trijoint.py.151: total_norm: 16.390608 [I 2022-10-09 04:47:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 1270/2304 [I 2022-10-09 04:47:44] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:14:25 [I 2022-10-09 04:47:44] ...trap/engines/engine.py.229: process: 0.83619 | load: 0.00303 [I 2022-10-09 04:47:44] ...trap/engines/engine.py.230: loss: 0.23084 [I 2022-10-09 04:47:44] ...optimizers/trijoint.py.151: total_norm: 24.234758 [I 2022-10-09 04:47:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 1280/2304 [I 2022-10-09 04:47:52] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:20 [I 2022-10-09 04:47:52] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00298 [I 2022-10-09 04:47:52] ...trap/engines/engine.py.230: loss: 0.19997 [I 2022-10-09 04:47:52] ...optimizers/trijoint.py.151: total_norm: 18.111610 [I 2022-10-09 04:48:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 1290/2304 [I 2022-10-09 04:48:00] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:14:12 [I 2022-10-09 04:48:00] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00308 [I 2022-10-09 04:48:00] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-09 04:48:00] ...optimizers/trijoint.py.151: total_norm: 17.053324 [I 2022-10-09 04:48:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 1300/2304 [I 2022-10-09 04:48:09] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:03 [I 2022-10-09 04:48:09] ...trap/engines/engine.py.229: process: 0.82512 | load: 0.00299 [I 2022-10-09 04:48:09] ...trap/engines/engine.py.230: loss: 0.19859 [I 2022-10-09 04:48:09] ...optimizers/trijoint.py.151: total_norm: 24.776163 [I 2022-10-09 04:48:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 1310/2304 [I 2022-10-09 04:48:17] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:13:57 [I 2022-10-09 04:48:17] ...trap/engines/engine.py.229: process: 0.83105 | load: 0.00303 [I 2022-10-09 04:48:17] ...trap/engines/engine.py.230: loss: 0.12154 [I 2022-10-09 04:48:17] ...optimizers/trijoint.py.151: total_norm: 12.870936 [I 2022-10-09 04:48:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 1320/2304 [I 2022-10-09 04:48:26] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:13:48 [I 2022-10-09 04:48:26] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00305 [I 2022-10-09 04:48:26] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-09 04:48:26] ...optimizers/trijoint.py.151: total_norm: 19.728558 [I 2022-10-09 04:48:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 1330/2304 [I 2022-10-09 04:48:34] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:13:42 [I 2022-10-09 04:48:34] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00301 [I 2022-10-09 04:48:34] ...trap/engines/engine.py.230: loss: 0.21870 [I 2022-10-09 04:48:34] ...optimizers/trijoint.py.151: total_norm: 22.044651 [I 2022-10-09 04:48:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 1340/2304 [I 2022-10-09 04:48:43] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:26 [I 2022-10-09 04:48:43] ...trap/engines/engine.py.229: process: 0.82914 | load: 0.00301 [I 2022-10-09 04:48:43] ...trap/engines/engine.py.230: loss: 0.23389 [I 2022-10-09 04:48:43] ...optimizers/trijoint.py.151: total_norm: 16.159470 [I 2022-10-09 04:48:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 1350/2304 [I 2022-10-09 04:48:51] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:13:16 [I 2022-10-09 04:48:51] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00310 [I 2022-10-09 04:48:51] ...trap/engines/engine.py.230: loss: 0.16299 [I 2022-10-09 04:48:51] ...optimizers/trijoint.py.151: total_norm: 17.228830 [I 2022-10-09 04:48:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 1360/2304 [I 2022-10-09 04:48:59] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:13:10 [I 2022-10-09 04:48:59] ...trap/engines/engine.py.229: process: 0.83429 | load: 0.00301 [I 2022-10-09 04:48:59] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-09 04:48:59] ...optimizers/trijoint.py.151: total_norm: 15.091411 [I 2022-10-09 04:49:08] ...trap/engines/engine.py.224: train: epoch 32 | batch 1370/2304 [I 2022-10-09 04:49:08] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:05 [I 2022-10-09 04:49:08] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00305 [I 2022-10-09 04:49:08] ...trap/engines/engine.py.230: loss: 0.11949 [I 2022-10-09 04:49:08] ...optimizers/trijoint.py.151: total_norm: 7.584321 [I 2022-10-09 04:49:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 1380/2304 [I 2022-10-09 04:49:16] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:12:55 [I 2022-10-09 04:49:16] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00303 [I 2022-10-09 04:49:16] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-09 04:49:16] ...optimizers/trijoint.py.151: total_norm: 19.436720 [I 2022-10-09 04:49:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 1390/2304 [I 2022-10-09 04:49:25] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:12:48 [I 2022-10-09 04:49:25] ...trap/engines/engine.py.229: process: 0.82853 | load: 0.00301 [I 2022-10-09 04:49:25] ...trap/engines/engine.py.230: loss: 0.17915 [I 2022-10-09 04:49:25] ...optimizers/trijoint.py.151: total_norm: 15.405466 [I 2022-10-09 04:49:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 1400/2304 [I 2022-10-09 04:49:33] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:12:41 [I 2022-10-09 04:49:33] ...trap/engines/engine.py.229: process: 0.84356 | load: 0.00300 [I 2022-10-09 04:49:33] ...trap/engines/engine.py.230: loss: 0.11027 [I 2022-10-09 04:49:33] ...optimizers/trijoint.py.151: total_norm: 19.868780 [I 2022-10-09 04:49:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 1410/2304 [I 2022-10-09 04:49:41] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:12:32 [I 2022-10-09 04:49:41] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00301 [I 2022-10-09 04:49:41] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-09 04:49:41] ...optimizers/trijoint.py.151: total_norm: 13.079398 [I 2022-10-09 04:49:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 1420/2304 [I 2022-10-09 04:49:50] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:21 [I 2022-10-09 04:49:50] ...trap/engines/engine.py.229: process: 0.82616 | load: 0.00302 [I 2022-10-09 04:49:50] ...trap/engines/engine.py.230: loss: 0.15264 [I 2022-10-09 04:49:50] ...optimizers/trijoint.py.151: total_norm: 8.225348 [I 2022-10-09 04:49:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 1430/2304 [I 2022-10-09 04:49:58] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:12:18 [I 2022-10-09 04:49:58] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00300 [I 2022-10-09 04:49:58] ...trap/engines/engine.py.230: loss: 0.11479 [I 2022-10-09 04:49:58] ...optimizers/trijoint.py.151: total_norm: 9.935989 [I 2022-10-09 04:50:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 1440/2304 [I 2022-10-09 04:50:07] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:04 [I 2022-10-09 04:50:07] ...trap/engines/engine.py.229: process: 0.83376 | load: 0.00303 [I 2022-10-09 04:50:07] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-09 04:50:07] ...optimizers/trijoint.py.151: total_norm: 11.361876 [I 2022-10-09 04:50:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 1450/2304 [I 2022-10-09 04:50:15] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:00 [I 2022-10-09 04:50:15] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00301 [I 2022-10-09 04:50:15] ...trap/engines/engine.py.230: loss: 0.14433 [I 2022-10-09 04:50:15] ...optimizers/trijoint.py.151: total_norm: 11.641770 [I 2022-10-09 04:50:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 1460/2304 [I 2022-10-09 04:50:24] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:11:50 [I 2022-10-09 04:50:24] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00300 [I 2022-10-09 04:50:24] ...trap/engines/engine.py.230: loss: 0.14227 [I 2022-10-09 04:50:24] ...optimizers/trijoint.py.151: total_norm: 16.395544 [I 2022-10-09 04:50:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 1470/2304 [I 2022-10-09 04:50:32] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:11:41 [I 2022-10-09 04:50:32] ...trap/engines/engine.py.229: process: 0.83997 | load: 0.00298 [I 2022-10-09 04:50:32] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-09 04:50:32] ...optimizers/trijoint.py.151: total_norm: 16.354750 [I 2022-10-09 04:50:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 1480/2304 [I 2022-10-09 04:50:41] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:11:32 [I 2022-10-09 04:50:41] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00300 [I 2022-10-09 04:50:41] ...trap/engines/engine.py.230: loss: 0.20380 [I 2022-10-09 04:50:41] ...optimizers/trijoint.py.151: total_norm: 19.445944 [I 2022-10-09 04:50:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 1490/2304 [I 2022-10-09 04:50:49] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:21 [I 2022-10-09 04:50:49] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00304 [I 2022-10-09 04:50:49] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-09 04:50:49] ...optimizers/trijoint.py.151: total_norm: 13.156332 [I 2022-10-09 04:50:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 1500/2304 [I 2022-10-09 04:50:57] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:15 [I 2022-10-09 04:50:57] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00303 [I 2022-10-09 04:50:57] ...trap/engines/engine.py.230: loss: 0.22224 [I 2022-10-09 04:50:57] ...optimizers/trijoint.py.151: total_norm: 26.031012 [I 2022-10-09 04:51:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 1510/2304 [I 2022-10-09 04:51:06] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:06 [I 2022-10-09 04:51:06] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00299 [I 2022-10-09 04:51:06] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-09 04:51:06] ...optimizers/trijoint.py.151: total_norm: 12.453077 [I 2022-10-09 04:51:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 1520/2304 [I 2022-10-09 04:51:14] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:10:57 [I 2022-10-09 04:51:14] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00300 [I 2022-10-09 04:51:14] ...trap/engines/engine.py.230: loss: 0.23383 [I 2022-10-09 04:51:14] ...optimizers/trijoint.py.151: total_norm: 27.564301 [I 2022-10-09 04:51:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 1530/2304 [I 2022-10-09 04:51:23] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:10:54 [I 2022-10-09 04:51:23] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00300 [I 2022-10-09 04:51:23] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-09 04:51:23] ...optimizers/trijoint.py.151: total_norm: 11.386984 [I 2022-10-09 04:51:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 1540/2304 [I 2022-10-09 04:51:31] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:10:44 [I 2022-10-09 04:51:31] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00305 [I 2022-10-09 04:51:31] ...trap/engines/engine.py.230: loss: 0.17824 [I 2022-10-09 04:51:31] ...optimizers/trijoint.py.151: total_norm: 24.957773 [I 2022-10-09 04:51:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 1550/2304 [I 2022-10-09 04:51:39] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:10:34 [I 2022-10-09 04:51:39] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00313 [I 2022-10-09 04:51:39] ...trap/engines/engine.py.230: loss: 0.15089 [I 2022-10-09 04:51:39] ...optimizers/trijoint.py.151: total_norm: 13.942502 [I 2022-10-09 04:51:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 1560/2304 [I 2022-10-09 04:51:48] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:25 [I 2022-10-09 04:51:48] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00309 [I 2022-10-09 04:51:48] ...trap/engines/engine.py.230: loss: 0.19915 [I 2022-10-09 04:51:48] ...optimizers/trijoint.py.151: total_norm: 16.422531 [I 2022-10-09 04:51:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 1570/2304 [I 2022-10-09 04:51:56] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:16 [I 2022-10-09 04:51:56] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00301 [I 2022-10-09 04:51:56] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-09 04:51:56] ...optimizers/trijoint.py.151: total_norm: 11.905762 [I 2022-10-09 04:52:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 1580/2304 [I 2022-10-09 04:52:05] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:06 [I 2022-10-09 04:52:05] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00310 [I 2022-10-09 04:52:05] ...trap/engines/engine.py.230: loss: 0.13121 [I 2022-10-09 04:52:05] ...optimizers/trijoint.py.151: total_norm: 16.427479 [I 2022-10-09 04:52:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 1590/2304 [I 2022-10-09 04:52:13] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:09:59 [I 2022-10-09 04:52:13] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00308 [I 2022-10-09 04:52:13] ...trap/engines/engine.py.230: loss: 0.17278 [I 2022-10-09 04:52:13] ...optimizers/trijoint.py.151: total_norm: 15.232062 [I 2022-10-09 04:52:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 1600/2304 [I 2022-10-09 04:52:21] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:09:51 [I 2022-10-09 04:52:21] ...trap/engines/engine.py.229: process: 0.83602 | load: 0.00307 [I 2022-10-09 04:52:21] ...trap/engines/engine.py.230: loss: 0.16221 [I 2022-10-09 04:52:21] ...optimizers/trijoint.py.151: total_norm: 18.542459 [I 2022-10-09 04:52:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 1610/2304 [I 2022-10-09 04:52:30] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:09:43 [I 2022-10-09 04:52:30] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00308 [I 2022-10-09 04:52:30] ...trap/engines/engine.py.230: loss: 0.15686 [I 2022-10-09 04:52:30] ...optimizers/trijoint.py.151: total_norm: 11.908879 [I 2022-10-09 04:52:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 1620/2304 [I 2022-10-09 04:52:38] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:35 [I 2022-10-09 04:52:38] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00312 [I 2022-10-09 04:52:38] ...trap/engines/engine.py.230: loss: 0.18568 [I 2022-10-09 04:52:38] ...optimizers/trijoint.py.151: total_norm: 21.261517 [I 2022-10-09 04:52:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 1630/2304 [I 2022-10-09 04:52:47] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:27 [I 2022-10-09 04:52:47] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00302 [I 2022-10-09 04:52:47] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-09 04:52:47] ...optimizers/trijoint.py.151: total_norm: 13.302128 [I 2022-10-09 04:52:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 1640/2304 [I 2022-10-09 04:52:55] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:18 [I 2022-10-09 04:52:55] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00299 [I 2022-10-09 04:52:55] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-09 04:52:55] ...optimizers/trijoint.py.151: total_norm: 29.922974 [I 2022-10-09 04:53:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 1650/2304 [I 2022-10-09 04:53:04] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:09 [I 2022-10-09 04:53:04] ...trap/engines/engine.py.229: process: 0.83360 | load: 0.00300 [I 2022-10-09 04:53:04] ...trap/engines/engine.py.230: loss: 0.26466 [I 2022-10-09 04:53:04] ...optimizers/trijoint.py.151: total_norm: 18.486284 [I 2022-10-09 04:53:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 1660/2304 [I 2022-10-09 04:53:12] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:08:59 [I 2022-10-09 04:53:12] ...trap/engines/engine.py.229: process: 0.83153 | load: 0.00300 [I 2022-10-09 04:53:12] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-09 04:53:12] ...optimizers/trijoint.py.151: total_norm: 11.290597 [I 2022-10-09 04:53:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 1670/2304 [I 2022-10-09 04:53:20] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:08:54 [I 2022-10-09 04:53:20] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00301 [I 2022-10-09 04:53:20] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-09 04:53:20] ...optimizers/trijoint.py.151: total_norm: 14.923264 [I 2022-10-09 04:53:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 1680/2304 [I 2022-10-09 04:53:29] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:08:45 [I 2022-10-09 04:53:29] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00298 [I 2022-10-09 04:53:29] ...trap/engines/engine.py.230: loss: 0.22183 [I 2022-10-09 04:53:29] ...optimizers/trijoint.py.151: total_norm: 12.491261 [I 2022-10-09 04:53:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 1690/2304 [I 2022-10-09 04:53:37] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:39 [I 2022-10-09 04:53:37] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00305 [I 2022-10-09 04:53:37] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-09 04:53:37] ...optimizers/trijoint.py.151: total_norm: 8.501048 [I 2022-10-09 04:53:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 1700/2304 [I 2022-10-09 04:53:46] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:27 [I 2022-10-09 04:53:46] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00301 [I 2022-10-09 04:53:46] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-09 04:53:46] ...optimizers/trijoint.py.151: total_norm: 25.924568 [I 2022-10-09 04:53:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 1710/2304 [I 2022-10-09 04:53:54] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:19 [I 2022-10-09 04:53:54] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00298 [I 2022-10-09 04:53:54] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-09 04:53:54] ...optimizers/trijoint.py.151: total_norm: 20.514944 [I 2022-10-09 04:54:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 1720/2304 [I 2022-10-09 04:54:03] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:13 [I 2022-10-09 04:54:03] ...trap/engines/engine.py.229: process: 0.83145 | load: 0.00306 [I 2022-10-09 04:54:03] ...trap/engines/engine.py.230: loss: 0.22122 [I 2022-10-09 04:54:03] ...optimizers/trijoint.py.151: total_norm: 19.461458 [I 2022-10-09 04:54:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 1730/2304 [I 2022-10-09 04:54:11] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:05 [I 2022-10-09 04:54:11] ...trap/engines/engine.py.229: process: 0.82864 | load: 0.00303 [I 2022-10-09 04:54:11] ...trap/engines/engine.py.230: loss: 0.23251 [I 2022-10-09 04:54:11] ...optimizers/trijoint.py.151: total_norm: 20.898436 [I 2022-10-09 04:54:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 1740/2304 [I 2022-10-09 04:54:20] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:07:56 [I 2022-10-09 04:54:20] ...trap/engines/engine.py.229: process: 0.83334 | load: 0.00302 [I 2022-10-09 04:54:20] ...trap/engines/engine.py.230: loss: 0.15897 [I 2022-10-09 04:54:20] ...optimizers/trijoint.py.151: total_norm: 19.707745 [I 2022-10-09 04:54:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 1750/2304 [I 2022-10-09 04:54:28] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:07:46 [I 2022-10-09 04:54:28] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00300 [I 2022-10-09 04:54:28] ...trap/engines/engine.py.230: loss: 0.17611 [I 2022-10-09 04:54:28] ...optimizers/trijoint.py.151: total_norm: 30.694998 [I 2022-10-09 04:54:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 1760/2304 [I 2022-10-09 04:54:36] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:34 [I 2022-10-09 04:54:36] ...trap/engines/engine.py.229: process: 0.83419 | load: 0.00303 [I 2022-10-09 04:54:36] ...trap/engines/engine.py.230: loss: 0.14536 [I 2022-10-09 04:54:36] ...optimizers/trijoint.py.151: total_norm: 17.950954 [I 2022-10-09 04:54:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 1770/2304 [I 2022-10-09 04:54:45] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:30 [I 2022-10-09 04:54:45] ...trap/engines/engine.py.229: process: 0.83144 | load: 0.00298 [I 2022-10-09 04:54:45] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-09 04:54:45] ...optimizers/trijoint.py.151: total_norm: 12.511563 [I 2022-10-09 04:54:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 1780/2304 [I 2022-10-09 04:54:53] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:20 [I 2022-10-09 04:54:53] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00299 [I 2022-10-09 04:54:53] ...trap/engines/engine.py.230: loss: 0.15265 [I 2022-10-09 04:54:53] ...optimizers/trijoint.py.151: total_norm: 17.860376 [I 2022-10-09 04:55:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 1790/2304 [I 2022-10-09 04:55:02] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:11 [I 2022-10-09 04:55:02] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00300 [I 2022-10-09 04:55:02] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-09 04:55:02] ...optimizers/trijoint.py.151: total_norm: 12.948286 [I 2022-10-09 04:55:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 1800/2304 [I 2022-10-09 04:55:10] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:06 [I 2022-10-09 04:55:10] ...trap/engines/engine.py.229: process: 0.84853 | load: 0.00299 [I 2022-10-09 04:55:10] ...trap/engines/engine.py.230: loss: 0.21384 [I 2022-10-09 04:55:10] ...optimizers/trijoint.py.151: total_norm: 20.241594 [I 2022-10-09 04:55:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 1810/2304 [I 2022-10-09 04:55:19] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:06:53 [I 2022-10-09 04:55:19] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00302 [I 2022-10-09 04:55:19] ...trap/engines/engine.py.230: loss: 0.12757 [I 2022-10-09 04:55:19] ...optimizers/trijoint.py.151: total_norm: 12.829909 [I 2022-10-09 04:55:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 1820/2304 [I 2022-10-09 04:55:27] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:06:47 [I 2022-10-09 04:55:27] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00302 [I 2022-10-09 04:55:27] ...trap/engines/engine.py.230: loss: 0.14880 [I 2022-10-09 04:55:27] ...optimizers/trijoint.py.151: total_norm: 29.179085 [I 2022-10-09 04:55:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 1830/2304 [I 2022-10-09 04:55:35] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:38 [I 2022-10-09 04:55:35] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00300 [I 2022-10-09 04:55:35] ...trap/engines/engine.py.230: loss: 0.17421 [I 2022-10-09 04:55:35] ...optimizers/trijoint.py.151: total_norm: 12.655740 [I 2022-10-09 04:55:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 1840/2304 [I 2022-10-09 04:55:44] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:29 [I 2022-10-09 04:55:44] ...trap/engines/engine.py.229: process: 0.83762 | load: 0.00297 [I 2022-10-09 04:55:44] ...trap/engines/engine.py.230: loss: 0.23606 [I 2022-10-09 04:55:44] ...optimizers/trijoint.py.151: total_norm: 21.678997 [I 2022-10-09 04:55:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 1850/2304 [I 2022-10-09 04:55:52] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:22 [I 2022-10-09 04:55:52] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00301 [I 2022-10-09 04:55:52] ...trap/engines/engine.py.230: loss: 0.16380 [I 2022-10-09 04:55:52] ...optimizers/trijoint.py.151: total_norm: 23.833803 [I 2022-10-09 04:56:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 1860/2304 [I 2022-10-09 04:56:01] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:13 [I 2022-10-09 04:56:01] ...trap/engines/engine.py.229: process: 0.83181 | load: 0.00300 [I 2022-10-09 04:56:01] ...trap/engines/engine.py.230: loss: 0.14943 [I 2022-10-09 04:56:01] ...optimizers/trijoint.py.151: total_norm: 11.408838 [I 2022-10-09 04:56:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 1870/2304 [I 2022-10-09 04:56:09] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:04 [I 2022-10-09 04:56:09] ...trap/engines/engine.py.229: process: 0.82535 | load: 0.00300 [I 2022-10-09 04:56:09] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-09 04:56:09] ...optimizers/trijoint.py.151: total_norm: 11.677148 [I 2022-10-09 04:56:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 1880/2304 [I 2022-10-09 04:56:18] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:05:56 [I 2022-10-09 04:56:18] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00306 [I 2022-10-09 04:56:18] ...trap/engines/engine.py.230: loss: 0.14803 [I 2022-10-09 04:56:18] ...optimizers/trijoint.py.151: total_norm: 10.881234 [I 2022-10-09 04:56:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 1890/2304 [I 2022-10-09 04:56:26] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:05:46 [I 2022-10-09 04:56:26] ...trap/engines/engine.py.229: process: 0.83533 | load: 0.00302 [I 2022-10-09 04:56:26] ...trap/engines/engine.py.230: loss: 0.21431 [I 2022-10-09 04:56:26] ...optimizers/trijoint.py.151: total_norm: 17.388187 [I 2022-10-09 04:56:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 1900/2304 [I 2022-10-09 04:56:34] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:36 [I 2022-10-09 04:56:34] ...trap/engines/engine.py.229: process: 0.83399 | load: 0.00300 [I 2022-10-09 04:56:34] ...trap/engines/engine.py.230: loss: 0.15787 [I 2022-10-09 04:56:34] ...optimizers/trijoint.py.151: total_norm: 14.277847 [I 2022-10-09 04:56:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 1910/2304 [I 2022-10-09 04:56:43] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:30 [I 2022-10-09 04:56:43] ...trap/engines/engine.py.229: process: 0.83521 | load: 0.00301 [I 2022-10-09 04:56:43] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-09 04:56:43] ...optimizers/trijoint.py.151: total_norm: 8.627433 [I 2022-10-09 04:56:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 1920/2304 [I 2022-10-09 04:56:51] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:23 [I 2022-10-09 04:56:51] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00299 [I 2022-10-09 04:56:51] ...trap/engines/engine.py.230: loss: 0.14845 [I 2022-10-09 04:56:51] ...optimizers/trijoint.py.151: total_norm: 16.669773 [I 2022-10-09 04:57:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 1930/2304 [I 2022-10-09 04:57:00] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:14 [I 2022-10-09 04:57:00] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00300 [I 2022-10-09 04:57:00] ...trap/engines/engine.py.230: loss: 0.13592 [I 2022-10-09 04:57:00] ...optimizers/trijoint.py.151: total_norm: 13.009148 [I 2022-10-09 04:57:08] ...trap/engines/engine.py.224: train: epoch 32 | batch 1940/2304 [I 2022-10-09 04:57:08] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:07 [I 2022-10-09 04:57:08] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00306 [I 2022-10-09 04:57:08] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-09 04:57:08] ...optimizers/trijoint.py.151: total_norm: 12.868480 [I 2022-10-09 04:57:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 1950/2304 [I 2022-10-09 04:57:16] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:04:57 [I 2022-10-09 04:57:16] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00308 [I 2022-10-09 04:57:16] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-09 04:57:16] ...optimizers/trijoint.py.151: total_norm: 18.971769 [I 2022-10-09 04:57:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 1960/2304 [I 2022-10-09 04:57:25] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:04:48 [I 2022-10-09 04:57:25] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00312 [I 2022-10-09 04:57:25] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-09 04:57:25] ...optimizers/trijoint.py.151: total_norm: 20.742033 [I 2022-10-09 04:57:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 1970/2304 [I 2022-10-09 04:57:33] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:41 [I 2022-10-09 04:57:33] ...trap/engines/engine.py.229: process: 0.84059 | load: 0.00312 [I 2022-10-09 04:57:33] ...trap/engines/engine.py.230: loss: 0.14658 [I 2022-10-09 04:57:33] ...optimizers/trijoint.py.151: total_norm: 16.953268 [I 2022-10-09 04:57:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 1980/2304 [I 2022-10-09 04:57:42] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:32 [I 2022-10-09 04:57:42] ...trap/engines/engine.py.229: process: 0.83580 | load: 0.00303 [I 2022-10-09 04:57:42] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-09 04:57:42] ...optimizers/trijoint.py.151: total_norm: 68.883591 [I 2022-10-09 04:57:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 1990/2304 [I 2022-10-09 04:57:50] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:22 [I 2022-10-09 04:57:50] ...trap/engines/engine.py.229: process: 0.83218 | load: 0.00300 [I 2022-10-09 04:57:50] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-09 04:57:50] ...optimizers/trijoint.py.151: total_norm: 13.070372 [I 2022-10-09 04:57:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 2000/2304 [I 2022-10-09 04:57:59] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:15 [I 2022-10-09 04:57:59] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00298 [I 2022-10-09 04:57:59] ...trap/engines/engine.py.230: loss: 0.15090 [I 2022-10-09 04:57:59] ...optimizers/trijoint.py.151: total_norm: 11.081017 [I 2022-10-09 04:58:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 2010/2304 [I 2022-10-09 04:58:07] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:05 [I 2022-10-09 04:58:07] ...trap/engines/engine.py.229: process: 0.83063 | load: 0.00301 [I 2022-10-09 04:58:07] ...trap/engines/engine.py.230: loss: 0.11949 [I 2022-10-09 04:58:07] ...optimizers/trijoint.py.151: total_norm: 16.782974 [I 2022-10-09 04:58:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 2020/2304 [I 2022-10-09 04:58:15] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:03:56 [I 2022-10-09 04:58:15] ...trap/engines/engine.py.229: process: 0.81827 | load: 0.00298 [I 2022-10-09 04:58:15] ...trap/engines/engine.py.230: loss: 0.15596 [I 2022-10-09 04:58:15] ...optimizers/trijoint.py.151: total_norm: 14.068440 [I 2022-10-09 04:58:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 2030/2304 [I 2022-10-09 04:58:24] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:03:48 [I 2022-10-09 04:58:24] ...trap/engines/engine.py.229: process: 0.82859 | load: 0.00300 [I 2022-10-09 04:58:24] ...trap/engines/engine.py.230: loss: 0.16509 [I 2022-10-09 04:58:24] ...optimizers/trijoint.py.151: total_norm: 31.822300 [I 2022-10-09 04:58:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 2040/2304 [I 2022-10-09 04:58:32] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:41 [I 2022-10-09 04:58:32] ...trap/engines/engine.py.229: process: 0.84135 | load: 0.00302 [I 2022-10-09 04:58:32] ...trap/engines/engine.py.230: loss: 0.12450 [I 2022-10-09 04:58:32] ...optimizers/trijoint.py.151: total_norm: 14.018285 [I 2022-10-09 04:58:40] ...trap/engines/engine.py.224: train: epoch 32 | batch 2050/2304 [I 2022-10-09 04:58:40] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:35 [I 2022-10-09 04:58:40] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00298 [I 2022-10-09 04:58:40] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-09 04:58:40] ...optimizers/trijoint.py.151: total_norm: 17.173241 [I 2022-10-09 04:58:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 2060/2304 [I 2022-10-09 04:58:49] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:25 [I 2022-10-09 04:58:49] ...trap/engines/engine.py.229: process: 0.84463 | load: 0.00307 [I 2022-10-09 04:58:49] ...trap/engines/engine.py.230: loss: 0.20890 [I 2022-10-09 04:58:49] ...optimizers/trijoint.py.151: total_norm: 15.465526 [I 2022-10-09 04:58:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 2070/2304 [I 2022-10-09 04:58:57] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:16 [I 2022-10-09 04:58:57] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00314 [I 2022-10-09 04:58:57] ...trap/engines/engine.py.230: loss: 0.15611 [I 2022-10-09 04:58:57] ...optimizers/trijoint.py.151: total_norm: 15.184118 [I 2022-10-09 04:59:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 2080/2304 [I 2022-10-09 04:59:06] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:08 [I 2022-10-09 04:59:06] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00312 [I 2022-10-09 04:59:06] ...trap/engines/engine.py.230: loss: 0.20361 [I 2022-10-09 04:59:06] ...optimizers/trijoint.py.151: total_norm: 19.594622 [I 2022-10-09 04:59:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 2090/2304 [I 2022-10-09 04:59:14] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:00 [I 2022-10-09 04:59:14] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00308 [I 2022-10-09 04:59:14] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-09 04:59:14] ...optimizers/trijoint.py.151: total_norm: 11.390344 [I 2022-10-09 04:59:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 2100/2304 [I 2022-10-09 04:59:23] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:02:51 [I 2022-10-09 04:59:23] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00308 [I 2022-10-09 04:59:23] ...trap/engines/engine.py.230: loss: 0.17528 [I 2022-10-09 04:59:23] ...optimizers/trijoint.py.151: total_norm: 18.679510 [I 2022-10-09 04:59:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 2110/2304 [I 2022-10-09 04:59:31] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:42 [I 2022-10-09 04:59:31] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00300 [I 2022-10-09 04:59:31] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-09 04:59:31] ...optimizers/trijoint.py.151: total_norm: 18.950712 [I 2022-10-09 04:59:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 2120/2304 [I 2022-10-09 04:59:39] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:32 [I 2022-10-09 04:59:39] ...trap/engines/engine.py.229: process: 0.81691 | load: 0.00299 [I 2022-10-09 04:59:39] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-09 04:59:39] ...optimizers/trijoint.py.151: total_norm: 14.483419 [I 2022-10-09 04:59:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 2130/2304 [I 2022-10-09 04:59:48] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:25 [I 2022-10-09 04:59:48] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00302 [I 2022-10-09 04:59:48] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-09 04:59:48] ...optimizers/trijoint.py.151: total_norm: 19.929228 [I 2022-10-09 04:59:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 2140/2304 [I 2022-10-09 04:59:56] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:18 [I 2022-10-09 04:59:56] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00303 [I 2022-10-09 04:59:56] ...trap/engines/engine.py.230: loss: 0.10112 [I 2022-10-09 04:59:56] ...optimizers/trijoint.py.151: total_norm: 27.018185 [I 2022-10-09 05:00:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 2150/2304 [I 2022-10-09 05:00:05] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:09 [I 2022-10-09 05:00:05] ...trap/engines/engine.py.229: process: 0.83178 | load: 0.00299 [I 2022-10-09 05:00:05] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-09 05:00:05] ...optimizers/trijoint.py.151: total_norm: 14.061366 [I 2022-10-09 05:00:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 2160/2304 [I 2022-10-09 05:00:13] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:01 [I 2022-10-09 05:00:13] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00298 [I 2022-10-09 05:00:13] ...trap/engines/engine.py.230: loss: 0.21922 [I 2022-10-09 05:00:13] ...optimizers/trijoint.py.151: total_norm: 17.575590 [I 2022-10-09 05:00:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 2170/2304 [I 2022-10-09 05:00:21] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:01:52 [I 2022-10-09 05:00:21] ...trap/engines/engine.py.229: process: 0.83376 | load: 0.00299 [I 2022-10-09 05:00:21] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-09 05:00:21] ...optimizers/trijoint.py.151: total_norm: 18.921181 [I 2022-10-09 05:00:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 2180/2304 [I 2022-10-09 05:00:30] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:01:44 [I 2022-10-09 05:00:30] ...trap/engines/engine.py.229: process: 0.84164 | load: 0.00298 [I 2022-10-09 05:00:30] ...trap/engines/engine.py.230: loss: 0.20417 [I 2022-10-09 05:00:30] ...optimizers/trijoint.py.151: total_norm: 15.916727 [I 2022-10-09 05:00:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 2190/2304 [I 2022-10-09 05:00:38] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:36 [I 2022-10-09 05:00:38] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00302 [I 2022-10-09 05:00:38] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-09 05:00:38] ...optimizers/trijoint.py.151: total_norm: 23.398130 [I 2022-10-09 05:00:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 2200/2304 [I 2022-10-09 05:00:47] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:27 [I 2022-10-09 05:00:47] ...trap/engines/engine.py.229: process: 0.83666 | load: 0.00301 [I 2022-10-09 05:00:47] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-09 05:00:47] ...optimizers/trijoint.py.151: total_norm: 18.747536 [I 2022-10-09 05:00:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 2210/2304 [I 2022-10-09 05:00:55] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:18 [I 2022-10-09 05:00:55] ...trap/engines/engine.py.229: process: 0.82167 | load: 0.00302 [I 2022-10-09 05:00:55] ...trap/engines/engine.py.230: loss: 0.16140 [I 2022-10-09 05:00:55] ...optimizers/trijoint.py.151: total_norm: 14.627378 [I 2022-10-09 05:01:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 2220/2304 [I 2022-10-09 05:01:04] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:10 [I 2022-10-09 05:01:04] ...trap/engines/engine.py.229: process: 0.84425 | load: 0.00301 [I 2022-10-09 05:01:04] ...trap/engines/engine.py.230: loss: 0.13691 [I 2022-10-09 05:01:04] ...optimizers/trijoint.py.151: total_norm: 15.707863 [I 2022-10-09 05:01:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 2230/2304 [I 2022-10-09 05:01:12] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:02 [I 2022-10-09 05:01:12] ...trap/engines/engine.py.229: process: 0.82913 | load: 0.00299 [I 2022-10-09 05:01:12] ...trap/engines/engine.py.230: loss: 0.10690 [I 2022-10-09 05:01:12] ...optimizers/trijoint.py.151: total_norm: 10.305995 [I 2022-10-09 05:01:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 2240/2304 [I 2022-10-09 05:01:20] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:00:53 [I 2022-10-09 05:01:20] ...trap/engines/engine.py.229: process: 0.82646 | load: 0.00299 [I 2022-10-09 05:01:20] ...trap/engines/engine.py.230: loss: 0.17833 [I 2022-10-09 05:01:20] ...optimizers/trijoint.py.151: total_norm: 16.722160 [I 2022-10-09 05:01:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 2250/2304 [I 2022-10-09 05:01:29] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:00:45 [I 2022-10-09 05:01:29] ...trap/engines/engine.py.229: process: 0.83810 | load: 0.00299 [I 2022-10-09 05:01:29] ...trap/engines/engine.py.230: loss: 0.17292 [I 2022-10-09 05:01:29] ...optimizers/trijoint.py.151: total_norm: 16.526316 [I 2022-10-09 05:01:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 2260/2304 [I 2022-10-09 05:01:37] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:36 [I 2022-10-09 05:01:37] ...trap/engines/engine.py.229: process: 0.83497 | load: 0.00302 [I 2022-10-09 05:01:37] ...trap/engines/engine.py.230: loss: 0.21574 [I 2022-10-09 05:01:37] ...optimizers/trijoint.py.151: total_norm: 15.958066 [I 2022-10-09 05:01:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 2270/2304 [I 2022-10-09 05:01:46] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:28 [I 2022-10-09 05:01:46] ...trap/engines/engine.py.229: process: 0.83010 | load: 0.00301 [I 2022-10-09 05:01:46] ...trap/engines/engine.py.230: loss: 0.12614 [I 2022-10-09 05:01:46] ...optimizers/trijoint.py.151: total_norm: 15.074065 [I 2022-10-09 05:01:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 2280/2304 [I 2022-10-09 05:01:54] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:20 [I 2022-10-09 05:01:54] ...trap/engines/engine.py.229: process: 0.85469 | load: 0.00298 [I 2022-10-09 05:01:54] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-09 05:01:54] ...optimizers/trijoint.py.151: total_norm: 10.478631 [I 2022-10-09 05:02:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 2290/2304 [I 2022-10-09 05:02:03] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:11 [I 2022-10-09 05:02:03] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00301 [I 2022-10-09 05:02:03] ...trap/engines/engine.py.230: loss: 0.20598 [I 2022-10-09 05:02:03] ...optimizers/trijoint.py.151: total_norm: 16.740471 [I 2022-10-09 05:02:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 2300/2304 [I 2022-10-09 05:02:11] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:03 [I 2022-10-09 05:02:11] ...trap/engines/engine.py.229: process: 0.83121 | load: 0.00016 [I 2022-10-09 05:02:11] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-09 05:02:11] ...optimizers/trijoint.py.151: total_norm: 23.059446 [I 2022-10-09 05:02:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 2304/2304 [I 2022-10-09 05:02:14] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:00 [I 2022-10-09 05:02:14] ...trap/engines/engine.py.229: process: 0.81528 | load: 0.00016 [I 2022-10-09 05:02:14] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-09 05:02:14] ...optimizers/trijoint.py.151: total_norm: 16.506945 [S 2022-10-09 05:02:16] ...trap/engines/engine.py.129: train_epoch.epoch: 32 [S 2022-10-09 05:02:16] ...trap/engines/engine.py.129: train_epoch.loss: 0.16476494365426309 [I 2022-10-09 05:02:16] ...s/criterions/triplet.py.84: margin = 0.21500000000000014 [I 2022-10-09 05:02:16] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 32 [I 2022-10-09 05:02:16] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 05:02:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 05:02:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 05:02:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 05:02:19] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 05:02:20] ...trap/engines/engine.py.321: eval: epoch 32 | batch 0/510 [I 2022-10-09 05:02:20] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:16 [I 2022-10-09 05:02:20] ...trap/engines/engine.py.326: process: 3.44373 | load: 2.90232 [I 2022-10-09 05:02:25] ...trap/engines/engine.py.321: eval: epoch 32 | batch 10/510 [I 2022-10-09 05:02:25] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:56 [I 2022-10-09 05:02:25] ...trap/engines/engine.py.326: process: 0.57989 | load: 0.00019 [I 2022-10-09 05:02:30] ...trap/engines/engine.py.321: eval: epoch 32 | batch 20/510 [I 2022-10-09 05:02:30] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:32 [I 2022-10-09 05:02:30] ...trap/engines/engine.py.326: process: 0.43436 | load: 0.00019 [I 2022-10-09 05:02:36] ...trap/engines/engine.py.321: eval: epoch 32 | batch 30/510 [I 2022-10-09 05:02:36] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:47 [I 2022-10-09 05:02:36] ...trap/engines/engine.py.326: process: 0.74199 | load: 0.00020 [I 2022-10-09 05:02:42] ...trap/engines/engine.py.321: eval: epoch 32 | batch 40/510 [I 2022-10-09 05:02:42] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:37 [I 2022-10-09 05:02:42] ...trap/engines/engine.py.326: process: 0.55163 | load: 0.00018 [I 2022-10-09 05:02:47] ...trap/engines/engine.py.321: eval: epoch 32 | batch 50/510 [I 2022-10-09 05:02:47] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:04 [I 2022-10-09 05:02:47] ...trap/engines/engine.py.326: process: 0.46310 | load: 0.00019 [I 2022-10-09 05:02:52] ...trap/engines/engine.py.321: eval: epoch 32 | batch 60/510 [I 2022-10-09 05:02:52] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:31 [I 2022-10-09 05:02:52] ...trap/engines/engine.py.326: process: 0.42380 | load: 0.00019 [I 2022-10-09 05:02:57] ...trap/engines/engine.py.321: eval: epoch 32 | batch 70/510 [I 2022-10-09 05:02:57] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:57 [I 2022-10-09 05:02:57] ...trap/engines/engine.py.326: process: 0.59428 | load: 0.00019 [I 2022-10-09 05:03:03] ...trap/engines/engine.py.321: eval: epoch 32 | batch 80/510 [I 2022-10-09 05:03:03] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:48 [I 2022-10-09 05:03:03] ...trap/engines/engine.py.326: process: 0.46845 | load: 0.00019 [I 2022-10-09 05:03:07] ...trap/engines/engine.py.321: eval: epoch 32 | batch 90/510 [I 2022-10-09 05:03:07] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:08 [I 2022-10-09 05:03:07] ...trap/engines/engine.py.326: process: 0.45655 | load: 0.00018 [I 2022-10-09 05:03:11] ...trap/engines/engine.py.321: eval: epoch 32 | batch 100/510 [I 2022-10-09 05:03:11] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:53 [I 2022-10-09 05:03:11] ...trap/engines/engine.py.326: process: 0.31126 | load: 0.00019 [I 2022-10-09 05:03:15] ...trap/engines/engine.py.321: eval: epoch 32 | batch 110/510 [I 2022-10-09 05:03:15] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-09 05:03:15] ...trap/engines/engine.py.326: process: 0.31869 | load: 0.00019 [I 2022-10-09 05:03:18] ...trap/engines/engine.py.321: eval: epoch 32 | batch 120/510 [I 2022-10-09 05:03:18] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-09 05:03:18] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00019 [I 2022-10-09 05:03:21] ...trap/engines/engine.py.321: eval: epoch 32 | batch 130/510 [I 2022-10-09 05:03:21] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-09 05:03:21] ...trap/engines/engine.py.326: process: 0.32334 | load: 0.00021 [I 2022-10-09 05:03:24] ...trap/engines/engine.py.321: eval: epoch 32 | batch 140/510 [I 2022-10-09 05:03:24] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:58 [I 2022-10-09 05:03:24] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00020 [I 2022-10-09 05:03:27] ...trap/engines/engine.py.321: eval: epoch 32 | batch 150/510 [I 2022-10-09 05:03:27] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:54 [I 2022-10-09 05:03:27] ...trap/engines/engine.py.326: process: 0.31870 | load: 0.00019 [I 2022-10-09 05:03:31] ...trap/engines/engine.py.321: eval: epoch 32 | batch 160/510 [I 2022-10-09 05:03:31] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:52 [I 2022-10-09 05:03:31] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00020 [I 2022-10-09 05:03:34] ...trap/engines/engine.py.321: eval: epoch 32 | batch 170/510 [I 2022-10-09 05:03:34] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-09 05:03:34] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00020 [I 2022-10-09 05:03:37] ...trap/engines/engine.py.321: eval: epoch 32 | batch 180/510 [I 2022-10-09 05:03:37] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-09 05:03:37] ...trap/engines/engine.py.326: process: 0.32966 | load: 0.00019 [I 2022-10-09 05:03:40] ...trap/engines/engine.py.321: eval: epoch 32 | batch 190/510 [I 2022-10-09 05:03:40] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-09 05:03:40] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00019 [I 2022-10-09 05:03:43] ...trap/engines/engine.py.321: eval: epoch 32 | batch 200/510 [I 2022-10-09 05:03:43] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:39 [I 2022-10-09 05:03:43] ...trap/engines/engine.py.326: process: 0.32046 | load: 0.00020 [I 2022-10-09 05:03:46] ...trap/engines/engine.py.321: eval: epoch 32 | batch 210/510 [I 2022-10-09 05:03:46] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:36 [I 2022-10-09 05:03:46] ...trap/engines/engine.py.326: process: 0.32177 | load: 0.00021 [I 2022-10-09 05:03:50] ...trap/engines/engine.py.321: eval: epoch 32 | batch 220/510 [I 2022-10-09 05:03:50] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-09 05:03:50] ...trap/engines/engine.py.326: process: 0.31915 | load: 0.00019 [I 2022-10-09 05:03:53] ...trap/engines/engine.py.321: eval: epoch 32 | batch 230/510 [I 2022-10-09 05:03:53] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-09 05:03:53] ...trap/engines/engine.py.326: process: 0.31500 | load: 0.00020 [I 2022-10-09 05:03:56] ...trap/engines/engine.py.321: eval: epoch 32 | batch 240/510 [I 2022-10-09 05:03:56] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-09 05:03:56] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00020 [I 2022-10-09 05:03:59] ...trap/engines/engine.py.321: eval: epoch 32 | batch 250/510 [I 2022-10-09 05:03:59] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-09 05:03:59] ...trap/engines/engine.py.326: process: 0.32235 | load: 0.00021 [I 2022-10-09 05:04:02] ...trap/engines/engine.py.321: eval: epoch 32 | batch 260/510 [I 2022-10-09 05:04:02] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-09 05:04:02] ...trap/engines/engine.py.326: process: 0.32882 | load: 0.00021 [I 2022-10-09 05:04:06] ...trap/engines/engine.py.321: eval: epoch 32 | batch 270/510 [I 2022-10-09 05:04:06] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 05:04:06] ...trap/engines/engine.py.326: process: 0.32207 | load: 0.00020 [I 2022-10-09 05:04:09] ...trap/engines/engine.py.321: eval: epoch 32 | batch 280/510 [I 2022-10-09 05:04:09] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 05:04:09] ...trap/engines/engine.py.326: process: 0.32301 | load: 0.00021 [I 2022-10-09 05:04:12] ...trap/engines/engine.py.321: eval: epoch 32 | batch 290/510 [I 2022-10-09 05:04:12] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-09 05:04:12] ...trap/engines/engine.py.326: process: 0.32012 | load: 0.00021 [I 2022-10-09 05:04:15] ...trap/engines/engine.py.321: eval: epoch 32 | batch 300/510 [I 2022-10-09 05:04:15] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:06 [I 2022-10-09 05:04:15] ...trap/engines/engine.py.326: process: 0.31722 | load: 0.00020 [I 2022-10-09 05:04:18] ...trap/engines/engine.py.321: eval: epoch 32 | batch 310/510 [I 2022-10-09 05:04:18] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-09 05:04:18] ...trap/engines/engine.py.326: process: 0.31828 | load: 0.00019 [I 2022-10-09 05:04:22] ...trap/engines/engine.py.321: eval: epoch 32 | batch 320/510 [I 2022-10-09 05:04:22] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:01 [I 2022-10-09 05:04:22] ...trap/engines/engine.py.326: process: 0.31387 | load: 0.00021 [I 2022-10-09 05:04:25] ...trap/engines/engine.py.321: eval: epoch 32 | batch 330/510 [I 2022-10-09 05:04:25] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 05:04:25] ...trap/engines/engine.py.326: process: 0.31952 | load: 0.00021 [I 2022-10-09 05:04:28] ...trap/engines/engine.py.321: eval: epoch 32 | batch 340/510 [I 2022-10-09 05:04:28] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 05:04:28] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00020 [I 2022-10-09 05:04:31] ...trap/engines/engine.py.321: eval: epoch 32 | batch 350/510 [I 2022-10-09 05:04:31] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-09 05:04:31] ...trap/engines/engine.py.326: process: 0.31424 | load: 0.00021 [I 2022-10-09 05:04:34] ...trap/engines/engine.py.321: eval: epoch 32 | batch 360/510 [I 2022-10-09 05:04:34] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-09 05:04:34] ...trap/engines/engine.py.326: process: 0.31799 | load: 0.00020 [I 2022-10-09 05:04:38] ...trap/engines/engine.py.321: eval: epoch 32 | batch 370/510 [I 2022-10-09 05:04:38] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 05:04:38] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00020 [I 2022-10-09 05:04:41] ...trap/engines/engine.py.321: eval: epoch 32 | batch 380/510 [I 2022-10-09 05:04:41] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 05:04:41] ...trap/engines/engine.py.326: process: 0.31346 | load: 0.00021 [I 2022-10-09 05:04:44] ...trap/engines/engine.py.321: eval: epoch 32 | batch 390/510 [I 2022-10-09 05:04:44] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-09 05:04:44] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00021 [I 2022-10-09 05:04:47] ...trap/engines/engine.py.321: eval: epoch 32 | batch 400/510 [I 2022-10-09 05:04:47] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-09 05:04:47] ...trap/engines/engine.py.326: process: 0.31806 | load: 0.00020 [I 2022-10-09 05:04:51] ...trap/engines/engine.py.321: eval: epoch 32 | batch 410/510 [I 2022-10-09 05:04:51] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-09 05:04:51] ...trap/engines/engine.py.326: process: 0.32330 | load: 0.00020 [I 2022-10-09 05:04:54] ...trap/engines/engine.py.321: eval: epoch 32 | batch 420/510 [I 2022-10-09 05:04:54] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 05:04:54] ...trap/engines/engine.py.326: process: 0.31099 | load: 0.00020 [I 2022-10-09 05:04:57] ...trap/engines/engine.py.321: eval: epoch 32 | batch 430/510 [I 2022-10-09 05:04:57] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 05:04:57] ...trap/engines/engine.py.326: process: 0.32013 | load: 0.00020 [I 2022-10-09 05:05:00] ...trap/engines/engine.py.321: eval: epoch 32 | batch 440/510 [I 2022-10-09 05:05:00] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 05:05:00] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00020 [I 2022-10-09 05:05:03] ...trap/engines/engine.py.321: eval: epoch 32 | batch 450/510 [I 2022-10-09 05:05:03] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 05:05:03] ...trap/engines/engine.py.326: process: 0.32020 | load: 0.00021 [I 2022-10-09 05:05:07] ...trap/engines/engine.py.321: eval: epoch 32 | batch 460/510 [I 2022-10-09 05:05:07] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-09 05:05:07] ...trap/engines/engine.py.326: process: 0.31651 | load: 0.00019 [I 2022-10-09 05:05:10] ...trap/engines/engine.py.321: eval: epoch 32 | batch 470/510 [I 2022-10-09 05:05:10] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 05:05:10] ...trap/engines/engine.py.326: process: 0.31900 | load: 0.00020 [I 2022-10-09 05:05:13] ...trap/engines/engine.py.321: eval: epoch 32 | batch 480/510 [I 2022-10-09 05:05:13] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 05:05:13] ...trap/engines/engine.py.326: process: 0.32386 | load: 0.00020 [I 2022-10-09 05:05:16] ...trap/engines/engine.py.321: eval: epoch 32 | batch 490/510 [I 2022-10-09 05:05:16] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 05:05:16] ...trap/engines/engine.py.326: process: 0.31836 | load: 0.00020 [I 2022-10-09 05:05:19] ...trap/engines/engine.py.321: eval: epoch 32 | batch 500/510 [I 2022-10-09 05:05:19] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 05:05:19] ...trap/engines/engine.py.326: process: 0.32344 | load: 0.00020 [I 2022-10-09 05:05:22] ...trap/engines/engine.py.321: eval: epoch 32 | batch 510/510 [I 2022-10-09 05:05:22] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 05:05:22] ...trap/engines/engine.py.326: process: 0.31448 | load: 0.00016 [S 2022-10-09 05:05:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 32 [I 2022-10-09 05:05:23] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.697100 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903400 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938500 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.681900 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897200 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933400 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010540 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008261 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006932 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012755 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007639 [S 2022-10-09 05:05:49] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008212 [I 2022-10-09 05:06:04] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 05:06:04] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 05:06:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 05:06:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 05:06:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 05:06:06] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 05:06:08] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 05:06:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 05:06:14] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 05:06:14] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 05:06:17] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 05:06:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 05:06:23] ...trap/engines/engine.py.163: Training model on trainset for epoch 33 [I 2022-10-09 05:06:23] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 05:06:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 0/2304 [I 2022-10-09 05:06:26] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:00:17 [I 2022-10-09 05:06:26] ...trap/engines/engine.py.229: process: 3.13249 | load: 2.27122 [I 2022-10-09 05:06:26] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-09 05:06:26] ...optimizers/trijoint.py.151: total_norm: 12.525360 [I 2022-10-09 05:06:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 10/2304 [I 2022-10-09 05:06:34] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:23 [I 2022-10-09 05:06:34] ...trap/engines/engine.py.229: process: 0.83843 | load: 0.00553 [I 2022-10-09 05:06:34] ...trap/engines/engine.py.230: loss: 0.16258 [I 2022-10-09 05:06:34] ...optimizers/trijoint.py.151: total_norm: 11.693123 [I 2022-10-09 05:06:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 20/2304 [I 2022-10-09 05:06:43] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:49 [I 2022-10-09 05:06:43] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00461 [I 2022-10-09 05:06:43] ...trap/engines/engine.py.230: loss: 0.16691 [I 2022-10-09 05:06:43] ...optimizers/trijoint.py.151: total_norm: 23.607883 [I 2022-10-09 05:06:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 30/2304 [I 2022-10-09 05:06:51] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:05 [I 2022-10-09 05:06:51] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00394 [I 2022-10-09 05:06:51] ...trap/engines/engine.py.230: loss: 0.18835 [I 2022-10-09 05:06:51] ...optimizers/trijoint.py.151: total_norm: 14.035875 [I 2022-10-09 05:07:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 40/2304 [I 2022-10-09 05:07:00] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:38 [I 2022-10-09 05:07:00] ...trap/engines/engine.py.229: process: 0.83011 | load: 0.00385 [I 2022-10-09 05:07:00] ...trap/engines/engine.py.230: loss: 0.21008 [I 2022-10-09 05:07:00] ...optimizers/trijoint.py.151: total_norm: 49.902592 [I 2022-10-09 05:07:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 50/2304 [I 2022-10-09 05:07:08] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:30 [I 2022-10-09 05:07:08] ...trap/engines/engine.py.229: process: 0.83302 | load: 0.00372 [I 2022-10-09 05:07:08] ...trap/engines/engine.py.230: loss: 0.19872 [I 2022-10-09 05:07:08] ...optimizers/trijoint.py.151: total_norm: 14.462680 [I 2022-10-09 05:07:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 60/2304 [I 2022-10-09 05:07:16] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:18 [I 2022-10-09 05:07:16] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00364 [I 2022-10-09 05:07:16] ...trap/engines/engine.py.230: loss: 0.20561 [I 2022-10-09 05:07:16] ...optimizers/trijoint.py.151: total_norm: 16.834309 [I 2022-10-09 05:07:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 70/2304 [I 2022-10-09 05:07:25] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:07 [I 2022-10-09 05:07:25] ...trap/engines/engine.py.229: process: 0.83342 | load: 0.00350 [I 2022-10-09 05:07:25] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-09 05:07:25] ...optimizers/trijoint.py.151: total_norm: 13.107873 [I 2022-10-09 05:07:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 80/2304 [I 2022-10-09 05:07:33] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:12 [I 2022-10-09 05:07:33] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00309 [I 2022-10-09 05:07:33] ...trap/engines/engine.py.230: loss: 0.15936 [I 2022-10-09 05:07:33] ...optimizers/trijoint.py.151: total_norm: 18.931124 [I 2022-10-09 05:07:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 90/2304 [I 2022-10-09 05:07:42] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:53 [I 2022-10-09 05:07:42] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00346 [I 2022-10-09 05:07:42] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-09 05:07:42] ...optimizers/trijoint.py.151: total_norm: 15.632239 [I 2022-10-09 05:07:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 100/2304 [I 2022-10-09 05:07:50] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:49 [I 2022-10-09 05:07:50] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00379 [I 2022-10-09 05:07:50] ...trap/engines/engine.py.230: loss: 0.18570 [I 2022-10-09 05:07:50] ...optimizers/trijoint.py.151: total_norm: 22.227480 [I 2022-10-09 05:07:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 110/2304 [I 2022-10-09 05:07:59] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:44 [I 2022-10-09 05:07:59] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00320 [I 2022-10-09 05:07:59] ...trap/engines/engine.py.230: loss: 0.16080 [I 2022-10-09 05:07:59] ...optimizers/trijoint.py.151: total_norm: 13.936267 [I 2022-10-09 05:08:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 120/2304 [I 2022-10-09 05:08:07] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:35 [I 2022-10-09 05:08:07] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00319 [I 2022-10-09 05:08:07] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-09 05:08:07] ...optimizers/trijoint.py.151: total_norm: 24.096527 [I 2022-10-09 05:08:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 130/2304 [I 2022-10-09 05:08:15] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:30 [I 2022-10-09 05:08:15] ...trap/engines/engine.py.229: process: 0.83728 | load: 0.00328 [I 2022-10-09 05:08:15] ...trap/engines/engine.py.230: loss: 0.14677 [I 2022-10-09 05:08:15] ...optimizers/trijoint.py.151: total_norm: 19.318029 [I 2022-10-09 05:08:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 140/2304 [I 2022-10-09 05:08:24] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:27 [I 2022-10-09 05:08:24] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00309 [I 2022-10-09 05:08:24] ...trap/engines/engine.py.230: loss: 0.13483 [I 2022-10-09 05:08:24] ...optimizers/trijoint.py.151: total_norm: 36.180634 [I 2022-10-09 05:08:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 150/2304 [I 2022-10-09 05:08:32] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:13 [I 2022-10-09 05:08:32] ...trap/engines/engine.py.229: process: 0.83798 | load: 0.00310 [I 2022-10-09 05:08:32] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-09 05:08:32] ...optimizers/trijoint.py.151: total_norm: 8.136117 [I 2022-10-09 05:08:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 160/2304 [I 2022-10-09 05:08:41] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:02 [I 2022-10-09 05:08:41] ...trap/engines/engine.py.229: process: 0.83099 | load: 0.00336 [I 2022-10-09 05:08:41] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-09 05:08:41] ...optimizers/trijoint.py.151: total_norm: 15.252047 [I 2022-10-09 05:08:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 170/2304 [I 2022-10-09 05:08:49] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:54 [I 2022-10-09 05:08:49] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00299 [I 2022-10-09 05:08:49] ...trap/engines/engine.py.230: loss: 0.19163 [I 2022-10-09 05:08:49] ...optimizers/trijoint.py.151: total_norm: 49.428539 [I 2022-10-09 05:08:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 180/2304 [I 2022-10-09 05:08:58] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:49 [I 2022-10-09 05:08:58] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00299 [I 2022-10-09 05:08:58] ...trap/engines/engine.py.230: loss: 0.15432 [I 2022-10-09 05:08:58] ...optimizers/trijoint.py.151: total_norm: 20.722609 [I 2022-10-09 05:09:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 190/2304 [I 2022-10-09 05:09:06] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:42 [I 2022-10-09 05:09:06] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00329 [I 2022-10-09 05:09:06] ...trap/engines/engine.py.230: loss: 0.19030 [I 2022-10-09 05:09:06] ...optimizers/trijoint.py.151: total_norm: 19.949858 [I 2022-10-09 05:09:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 200/2304 [I 2022-10-09 05:09:14] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:40 [I 2022-10-09 05:09:14] ...trap/engines/engine.py.229: process: 0.86100 | load: 0.00303 [I 2022-10-09 05:09:14] ...trap/engines/engine.py.230: loss: 0.15958 [I 2022-10-09 05:09:14] ...optimizers/trijoint.py.151: total_norm: 27.319332 [I 2022-10-09 05:09:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 210/2304 [I 2022-10-09 05:09:23] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:20 [I 2022-10-09 05:09:23] ...trap/engines/engine.py.229: process: 0.83747 | load: 0.00300 [I 2022-10-09 05:09:23] ...trap/engines/engine.py.230: loss: 0.11046 [I 2022-10-09 05:09:23] ...optimizers/trijoint.py.151: total_norm: 7.355628 [I 2022-10-09 05:09:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 220/2304 [I 2022-10-09 05:09:31] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:20 [I 2022-10-09 05:09:31] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00300 [I 2022-10-09 05:09:31] ...trap/engines/engine.py.230: loss: 0.13372 [I 2022-10-09 05:09:31] ...optimizers/trijoint.py.151: total_norm: 12.074533 [I 2022-10-09 05:09:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 230/2304 [I 2022-10-09 05:09:40] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:28:59 [I 2022-10-09 05:09:40] ...trap/engines/engine.py.229: process: 0.82827 | load: 0.00299 [I 2022-10-09 05:09:40] ...trap/engines/engine.py.230: loss: 0.24383 [I 2022-10-09 05:09:40] ...optimizers/trijoint.py.151: total_norm: 11.587130 [I 2022-10-09 05:09:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 240/2304 [I 2022-10-09 05:09:48] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:56 [I 2022-10-09 05:09:48] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00296 [I 2022-10-09 05:09:48] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-09 05:09:48] ...optimizers/trijoint.py.151: total_norm: 12.555367 [I 2022-10-09 05:09:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 250/2304 [I 2022-10-09 05:09:57] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:44 [I 2022-10-09 05:09:57] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00311 [I 2022-10-09 05:09:57] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-09 05:09:57] ...optimizers/trijoint.py.151: total_norm: 15.693605 [I 2022-10-09 05:10:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 260/2304 [I 2022-10-09 05:10:05] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:37 [I 2022-10-09 05:10:05] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00315 [I 2022-10-09 05:10:05] ...trap/engines/engine.py.230: loss: 0.13954 [I 2022-10-09 05:10:05] ...optimizers/trijoint.py.151: total_norm: 18.565777 [I 2022-10-09 05:10:13] ...trap/engines/engine.py.224: train: epoch 33 | batch 270/2304 [I 2022-10-09 05:10:13] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:29 [I 2022-10-09 05:10:13] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00298 [I 2022-10-09 05:10:13] ...trap/engines/engine.py.230: loss: 0.15074 [I 2022-10-09 05:10:13] ...optimizers/trijoint.py.151: total_norm: 10.055553 [I 2022-10-09 05:10:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 280/2304 [I 2022-10-09 05:10:22] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:26 [I 2022-10-09 05:10:22] ...trap/engines/engine.py.229: process: 0.83859 | load: 0.00309 [I 2022-10-09 05:10:22] ...trap/engines/engine.py.230: loss: 0.21584 [I 2022-10-09 05:10:22] ...optimizers/trijoint.py.151: total_norm: 31.056801 [I 2022-10-09 05:10:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 290/2304 [I 2022-10-09 05:10:30] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:11 [I 2022-10-09 05:10:30] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00299 [I 2022-10-09 05:10:30] ...trap/engines/engine.py.230: loss: 0.11923 [I 2022-10-09 05:10:30] ...optimizers/trijoint.py.151: total_norm: 12.385841 [I 2022-10-09 05:10:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 300/2304 [I 2022-10-09 05:10:39] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:08 [I 2022-10-09 05:10:39] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00314 [I 2022-10-09 05:10:39] ...trap/engines/engine.py.230: loss: 0.26568 [I 2022-10-09 05:10:39] ...optimizers/trijoint.py.151: total_norm: 27.588037 [I 2022-10-09 05:10:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 310/2304 [I 2022-10-09 05:10:47] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:57 [I 2022-10-09 05:10:47] ...trap/engines/engine.py.229: process: 0.83499 | load: 0.00302 [I 2022-10-09 05:10:47] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-09 05:10:47] ...optimizers/trijoint.py.151: total_norm: 17.102720 [I 2022-10-09 05:10:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 320/2304 [I 2022-10-09 05:10:56] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:45 [I 2022-10-09 05:10:56] ...trap/engines/engine.py.229: process: 0.82827 | load: 0.00305 [I 2022-10-09 05:10:56] ...trap/engines/engine.py.230: loss: 0.16379 [I 2022-10-09 05:10:56] ...optimizers/trijoint.py.151: total_norm: 13.671696 [I 2022-10-09 05:11:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 330/2304 [I 2022-10-09 05:11:04] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:39 [I 2022-10-09 05:11:04] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00301 [I 2022-10-09 05:11:04] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-09 05:11:04] ...optimizers/trijoint.py.151: total_norm: 16.772858 [I 2022-10-09 05:11:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 340/2304 [I 2022-10-09 05:11:12] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:34 [I 2022-10-09 05:11:12] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00300 [I 2022-10-09 05:11:12] ...trap/engines/engine.py.230: loss: 0.14166 [I 2022-10-09 05:11:12] ...optimizers/trijoint.py.151: total_norm: 11.186154 [I 2022-10-09 05:11:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 350/2304 [I 2022-10-09 05:11:21] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:08 [I 2022-10-09 05:11:21] ...trap/engines/engine.py.229: process: 0.82230 | load: 0.00299 [I 2022-10-09 05:11:21] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-09 05:11:21] ...optimizers/trijoint.py.151: total_norm: 25.470148 [I 2022-10-09 05:11:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 360/2304 [I 2022-10-09 05:11:29] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:15 [I 2022-10-09 05:11:29] ...trap/engines/engine.py.229: process: 0.83085 | load: 0.00299 [I 2022-10-09 05:11:29] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-09 05:11:29] ...optimizers/trijoint.py.151: total_norm: 11.491170 [I 2022-10-09 05:11:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 370/2304 [I 2022-10-09 05:11:38] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:01 [I 2022-10-09 05:11:38] ...trap/engines/engine.py.229: process: 0.83138 | load: 0.00299 [I 2022-10-09 05:11:38] ...trap/engines/engine.py.230: loss: 0.12731 [I 2022-10-09 05:11:38] ...optimizers/trijoint.py.151: total_norm: 14.663671 [I 2022-10-09 05:11:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 380/2304 [I 2022-10-09 05:11:46] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:55 [I 2022-10-09 05:11:46] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00301 [I 2022-10-09 05:11:46] ...trap/engines/engine.py.230: loss: 0.20061 [I 2022-10-09 05:11:46] ...optimizers/trijoint.py.151: total_norm: 31.068182 [I 2022-10-09 05:11:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 390/2304 [I 2022-10-09 05:11:55] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:43 [I 2022-10-09 05:11:55] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00300 [I 2022-10-09 05:11:55] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-09 05:11:55] ...optimizers/trijoint.py.151: total_norm: 12.726656 [I 2022-10-09 05:12:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 400/2304 [I 2022-10-09 05:12:03] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:45 [I 2022-10-09 05:12:03] ...trap/engines/engine.py.229: process: 0.84760 | load: 0.00301 [I 2022-10-09 05:12:03] ...trap/engines/engine.py.230: loss: 0.12337 [I 2022-10-09 05:12:03] ...optimizers/trijoint.py.151: total_norm: 12.047868 [I 2022-10-09 05:12:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 410/2304 [I 2022-10-09 05:12:11] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:25 [I 2022-10-09 05:12:11] ...trap/engines/engine.py.229: process: 0.81863 | load: 0.00301 [I 2022-10-09 05:12:11] ...trap/engines/engine.py.230: loss: 0.16723 [I 2022-10-09 05:12:11] ...optimizers/trijoint.py.151: total_norm: 15.701605 [I 2022-10-09 05:12:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 420/2304 [I 2022-10-09 05:12:20] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:20 [I 2022-10-09 05:12:20] ...trap/engines/engine.py.229: process: 0.82191 | load: 0.00306 [I 2022-10-09 05:12:20] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-09 05:12:20] ...optimizers/trijoint.py.151: total_norm: 16.066187 [I 2022-10-09 05:12:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 430/2304 [I 2022-10-09 05:12:28] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:18 [I 2022-10-09 05:12:28] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00299 [I 2022-10-09 05:12:28] ...trap/engines/engine.py.230: loss: 0.15634 [I 2022-10-09 05:12:28] ...optimizers/trijoint.py.151: total_norm: 23.208878 [I 2022-10-09 05:12:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 440/2304 [I 2022-10-09 05:12:37] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:03 [I 2022-10-09 05:12:37] ...trap/engines/engine.py.229: process: 0.83783 | load: 0.00302 [I 2022-10-09 05:12:37] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-09 05:12:37] ...optimizers/trijoint.py.151: total_norm: 19.479256 [I 2022-10-09 05:12:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 450/2304 [I 2022-10-09 05:12:45] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:48 [I 2022-10-09 05:12:45] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00307 [I 2022-10-09 05:12:45] ...trap/engines/engine.py.230: loss: 0.15600 [I 2022-10-09 05:12:45] ...optimizers/trijoint.py.151: total_norm: 16.662270 [I 2022-10-09 05:12:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 460/2304 [I 2022-10-09 05:12:53] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:52 [I 2022-10-09 05:12:53] ...trap/engines/engine.py.229: process: 0.83644 | load: 0.00300 [I 2022-10-09 05:12:53] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-09 05:12:53] ...optimizers/trijoint.py.151: total_norm: 20.364902 [I 2022-10-09 05:13:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 470/2304 [I 2022-10-09 05:13:02] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:44 [I 2022-10-09 05:13:02] ...trap/engines/engine.py.229: process: 0.86576 | load: 0.00301 [I 2022-10-09 05:13:02] ...trap/engines/engine.py.230: loss: 0.19344 [I 2022-10-09 05:13:02] ...optimizers/trijoint.py.151: total_norm: 16.222469 [I 2022-10-09 05:13:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 480/2304 [I 2022-10-09 05:13:10] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:35 [I 2022-10-09 05:13:10] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00302 [I 2022-10-09 05:13:10] ...trap/engines/engine.py.230: loss: 0.16290 [I 2022-10-09 05:13:10] ...optimizers/trijoint.py.151: total_norm: 14.349413 [I 2022-10-09 05:13:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 490/2304 [I 2022-10-09 05:13:19] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:25 [I 2022-10-09 05:13:19] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00301 [I 2022-10-09 05:13:19] ...trap/engines/engine.py.230: loss: 0.17952 [I 2022-10-09 05:13:19] ...optimizers/trijoint.py.151: total_norm: 11.930690 [I 2022-10-09 05:13:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 500/2304 [I 2022-10-09 05:13:27] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:08 [I 2022-10-09 05:13:27] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00301 [I 2022-10-09 05:13:27] ...trap/engines/engine.py.230: loss: 0.17265 [I 2022-10-09 05:13:27] ...optimizers/trijoint.py.151: total_norm: 13.065985 [I 2022-10-09 05:13:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 510/2304 [I 2022-10-09 05:13:35] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:07 [I 2022-10-09 05:13:35] ...trap/engines/engine.py.229: process: 0.83744 | load: 0.00305 [I 2022-10-09 05:13:35] ...trap/engines/engine.py.230: loss: 0.20614 [I 2022-10-09 05:13:35] ...optimizers/trijoint.py.151: total_norm: 27.342081 [I 2022-10-09 05:13:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 520/2304 [I 2022-10-09 05:13:44] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:25:01 [I 2022-10-09 05:13:44] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00309 [I 2022-10-09 05:13:44] ...trap/engines/engine.py.230: loss: 0.13562 [I 2022-10-09 05:13:44] ...optimizers/trijoint.py.151: total_norm: 17.026608 [I 2022-10-09 05:13:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 530/2304 [I 2022-10-09 05:13:52] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:53 [I 2022-10-09 05:13:52] ...trap/engines/engine.py.229: process: 0.84601 | load: 0.00297 [I 2022-10-09 05:13:52] ...trap/engines/engine.py.230: loss: 0.20239 [I 2022-10-09 05:13:52] ...optimizers/trijoint.py.151: total_norm: 16.764597 [I 2022-10-09 05:14:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 540/2304 [I 2022-10-09 05:14:01] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:46 [I 2022-10-09 05:14:01] ...trap/engines/engine.py.229: process: 0.84807 | load: 0.00298 [I 2022-10-09 05:14:01] ...trap/engines/engine.py.230: loss: 0.20092 [I 2022-10-09 05:14:01] ...optimizers/trijoint.py.151: total_norm: 20.063826 [I 2022-10-09 05:14:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 550/2304 [I 2022-10-09 05:14:09] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:33 [I 2022-10-09 05:14:09] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00299 [I 2022-10-09 05:14:09] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-09 05:14:09] ...optimizers/trijoint.py.151: total_norm: 20.188019 [I 2022-10-09 05:14:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 560/2304 [I 2022-10-09 05:14:18] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:22 [I 2022-10-09 05:14:18] ...trap/engines/engine.py.229: process: 0.83202 | load: 0.00299 [I 2022-10-09 05:14:18] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-09 05:14:18] ...optimizers/trijoint.py.151: total_norm: 19.723257 [I 2022-10-09 05:14:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 570/2304 [I 2022-10-09 05:14:26] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:15 [I 2022-10-09 05:14:26] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00298 [I 2022-10-09 05:14:26] ...trap/engines/engine.py.230: loss: 0.17965 [I 2022-10-09 05:14:26] ...optimizers/trijoint.py.151: total_norm: 13.487980 [I 2022-10-09 05:14:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 580/2304 [I 2022-10-09 05:14:34] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:09 [I 2022-10-09 05:14:34] ...trap/engines/engine.py.229: process: 0.83683 | load: 0.00300 [I 2022-10-09 05:14:34] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-09 05:14:34] ...optimizers/trijoint.py.151: total_norm: 8.581167 [I 2022-10-09 05:14:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 590/2304 [I 2022-10-09 05:14:43] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:24:05 [I 2022-10-09 05:14:43] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00303 [I 2022-10-09 05:14:43] ...trap/engines/engine.py.230: loss: 0.16845 [I 2022-10-09 05:14:43] ...optimizers/trijoint.py.151: total_norm: 22.517178 [I 2022-10-09 05:14:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 600/2304 [I 2022-10-09 05:14:51] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:54 [I 2022-10-09 05:14:51] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00300 [I 2022-10-09 05:14:51] ...trap/engines/engine.py.230: loss: 0.22587 [I 2022-10-09 05:14:51] ...optimizers/trijoint.py.151: total_norm: 19.761250 [I 2022-10-09 05:15:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 610/2304 [I 2022-10-09 05:15:00] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:23:36 [I 2022-10-09 05:15:00] ...trap/engines/engine.py.229: process: 0.82822 | load: 0.00306 [I 2022-10-09 05:15:00] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-09 05:15:00] ...optimizers/trijoint.py.151: total_norm: 11.575281 [I 2022-10-09 05:15:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 620/2304 [I 2022-10-09 05:15:08] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:38 [I 2022-10-09 05:15:08] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00298 [I 2022-10-09 05:15:08] ...trap/engines/engine.py.230: loss: 0.21871 [I 2022-10-09 05:15:08] ...optimizers/trijoint.py.151: total_norm: 12.477246 [I 2022-10-09 05:15:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 630/2304 [I 2022-10-09 05:15:17] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:30 [I 2022-10-09 05:15:17] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00299 [I 2022-10-09 05:15:17] ...trap/engines/engine.py.230: loss: 0.20851 [I 2022-10-09 05:15:17] ...optimizers/trijoint.py.151: total_norm: 29.920549 [I 2022-10-09 05:15:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 640/2304 [I 2022-10-09 05:15:25] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:19 [I 2022-10-09 05:15:25] ...trap/engines/engine.py.229: process: 0.85213 | load: 0.00302 [I 2022-10-09 05:15:25] ...trap/engines/engine.py.230: loss: 0.17442 [I 2022-10-09 05:15:25] ...optimizers/trijoint.py.151: total_norm: 17.442604 [I 2022-10-09 05:15:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 650/2304 [I 2022-10-09 05:15:33] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:11 [I 2022-10-09 05:15:33] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00298 [I 2022-10-09 05:15:33] ...trap/engines/engine.py.230: loss: 0.21433 [I 2022-10-09 05:15:33] ...optimizers/trijoint.py.151: total_norm: 22.559225 [I 2022-10-09 05:15:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 660/2304 [I 2022-10-09 05:15:42] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:22:59 [I 2022-10-09 05:15:42] ...trap/engines/engine.py.229: process: 0.82870 | load: 0.00299 [I 2022-10-09 05:15:42] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-09 05:15:42] ...optimizers/trijoint.py.151: total_norm: 12.824121 [I 2022-10-09 05:15:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 670/2304 [I 2022-10-09 05:15:50] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:55 [I 2022-10-09 05:15:50] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00298 [I 2022-10-09 05:15:50] ...trap/engines/engine.py.230: loss: 0.12648 [I 2022-10-09 05:15:50] ...optimizers/trijoint.py.151: total_norm: 14.430408 [I 2022-10-09 05:15:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 680/2304 [I 2022-10-09 05:15:59] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:53 [I 2022-10-09 05:15:59] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00300 [I 2022-10-09 05:15:59] ...trap/engines/engine.py.230: loss: 0.14909 [I 2022-10-09 05:15:59] ...optimizers/trijoint.py.151: total_norm: 13.509716 [I 2022-10-09 05:16:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 690/2304 [I 2022-10-09 05:16:07] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:36 [I 2022-10-09 05:16:07] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00301 [I 2022-10-09 05:16:07] ...trap/engines/engine.py.230: loss: 0.15948 [I 2022-10-09 05:16:07] ...optimizers/trijoint.py.151: total_norm: 14.025300 [I 2022-10-09 05:16:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 700/2304 [I 2022-10-09 05:16:16] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:31 [I 2022-10-09 05:16:16] ...trap/engines/engine.py.229: process: 0.82943 | load: 0.00299 [I 2022-10-09 05:16:16] ...trap/engines/engine.py.230: loss: 0.14370 [I 2022-10-09 05:16:16] ...optimizers/trijoint.py.151: total_norm: 13.265057 [I 2022-10-09 05:16:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 710/2304 [I 2022-10-09 05:16:24] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:17 [I 2022-10-09 05:16:24] ...trap/engines/engine.py.229: process: 0.83447 | load: 0.00297 [I 2022-10-09 05:16:24] ...trap/engines/engine.py.230: loss: 0.20220 [I 2022-10-09 05:16:24] ...optimizers/trijoint.py.151: total_norm: 17.265030 [I 2022-10-09 05:16:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 720/2304 [I 2022-10-09 05:16:32] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:08 [I 2022-10-09 05:16:32] ...trap/engines/engine.py.229: process: 0.84305 | load: 0.00306 [I 2022-10-09 05:16:32] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-09 05:16:32] ...optimizers/trijoint.py.151: total_norm: 12.390330 [I 2022-10-09 05:16:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 730/2304 [I 2022-10-09 05:16:41] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:02 [I 2022-10-09 05:16:41] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00301 [I 2022-10-09 05:16:41] ...trap/engines/engine.py.230: loss: 0.21006 [I 2022-10-09 05:16:41] ...optimizers/trijoint.py.151: total_norm: 24.376009 [I 2022-10-09 05:16:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 740/2304 [I 2022-10-09 05:16:49] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:53 [I 2022-10-09 05:16:49] ...trap/engines/engine.py.229: process: 0.84853 | load: 0.00297 [I 2022-10-09 05:16:49] ...trap/engines/engine.py.230: loss: 0.12469 [I 2022-10-09 05:16:49] ...optimizers/trijoint.py.151: total_norm: 18.712635 [I 2022-10-09 05:16:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 750/2304 [I 2022-10-09 05:16:58] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:44 [I 2022-10-09 05:16:58] ...trap/engines/engine.py.229: process: 0.83569 | load: 0.00299 [I 2022-10-09 05:16:58] ...trap/engines/engine.py.230: loss: 0.10222 [I 2022-10-09 05:16:58] ...optimizers/trijoint.py.151: total_norm: 15.989166 [I 2022-10-09 05:17:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 760/2304 [I 2022-10-09 05:17:06] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:32 [I 2022-10-09 05:17:06] ...trap/engines/engine.py.229: process: 0.82270 | load: 0.00301 [I 2022-10-09 05:17:06] ...trap/engines/engine.py.230: loss: 0.14660 [I 2022-10-09 05:17:06] ...optimizers/trijoint.py.151: total_norm: 8.428135 [I 2022-10-09 05:17:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 770/2304 [I 2022-10-09 05:17:14] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:23 [I 2022-10-09 05:17:14] ...trap/engines/engine.py.229: process: 0.82543 | load: 0.00297 [I 2022-10-09 05:17:14] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-09 05:17:14] ...optimizers/trijoint.py.151: total_norm: 18.120159 [I 2022-10-09 05:17:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 780/2304 [I 2022-10-09 05:17:23] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:23 [I 2022-10-09 05:17:23] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00303 [I 2022-10-09 05:17:23] ...trap/engines/engine.py.230: loss: 0.14549 [I 2022-10-09 05:17:23] ...optimizers/trijoint.py.151: total_norm: 28.000294 [I 2022-10-09 05:17:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 790/2304 [I 2022-10-09 05:17:31] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:17 [I 2022-10-09 05:17:31] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00301 [I 2022-10-09 05:17:31] ...trap/engines/engine.py.230: loss: 0.16348 [I 2022-10-09 05:17:31] ...optimizers/trijoint.py.151: total_norm: 14.358500 [I 2022-10-09 05:17:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 800/2304 [I 2022-10-09 05:17:40] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:10 [I 2022-10-09 05:17:40] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00296 [I 2022-10-09 05:17:40] ...trap/engines/engine.py.230: loss: 0.19293 [I 2022-10-09 05:17:40] ...optimizers/trijoint.py.151: total_norm: 21.737213 [I 2022-10-09 05:17:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 810/2304 [I 2022-10-09 05:17:48] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:52 [I 2022-10-09 05:17:48] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00304 [I 2022-10-09 05:17:48] ...trap/engines/engine.py.230: loss: 0.12552 [I 2022-10-09 05:17:48] ...optimizers/trijoint.py.151: total_norm: 16.386213 [I 2022-10-09 05:18:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 820/2304 [I 2022-10-09 05:18:15] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:39:34 [I 2022-10-09 05:18:15] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00297 [I 2022-10-09 05:18:15] ...trap/engines/engine.py.230: loss: 0.17634 [I 2022-10-09 05:18:15] ...optimizers/trijoint.py.151: total_norm: 20.307371 [I 2022-10-09 05:18:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 830/2304 [I 2022-10-09 05:18:23] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:22:33 [I 2022-10-09 05:18:23] ...trap/engines/engine.py.229: process: 0.82803 | load: 0.00299 [I 2022-10-09 05:18:23] ...trap/engines/engine.py.230: loss: 0.17259 [I 2022-10-09 05:18:23] ...optimizers/trijoint.py.151: total_norm: 16.574915 [I 2022-10-09 05:18:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 840/2304 [I 2022-10-09 05:18:32] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:41 [I 2022-10-09 05:18:32] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00301 [I 2022-10-09 05:18:32] ...trap/engines/engine.py.230: loss: 0.16826 [I 2022-10-09 05:18:32] ...optimizers/trijoint.py.151: total_norm: 14.124316 [I 2022-10-09 05:18:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 850/2304 [I 2022-10-09 05:18:40] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:15 [I 2022-10-09 05:18:40] ...trap/engines/engine.py.229: process: 0.82652 | load: 0.00301 [I 2022-10-09 05:18:40] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-09 05:18:40] ...optimizers/trijoint.py.151: total_norm: 15.746602 [I 2022-10-09 05:18:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 860/2304 [I 2022-10-09 05:18:48] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:04 [I 2022-10-09 05:18:48] ...trap/engines/engine.py.229: process: 0.82867 | load: 0.00299 [I 2022-10-09 05:18:48] ...trap/engines/engine.py.230: loss: 0.16607 [I 2022-10-09 05:18:48] ...optimizers/trijoint.py.151: total_norm: 18.309187 [I 2022-10-09 05:18:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 870/2304 [I 2022-10-09 05:18:57] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:02 [I 2022-10-09 05:18:57] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00302 [I 2022-10-09 05:18:57] ...trap/engines/engine.py.230: loss: 0.17569 [I 2022-10-09 05:18:57] ...optimizers/trijoint.py.151: total_norm: 13.640137 [I 2022-10-09 05:19:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 880/2304 [I 2022-10-09 05:19:05] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:54 [I 2022-10-09 05:19:05] ...trap/engines/engine.py.229: process: 0.84747 | load: 0.00302 [I 2022-10-09 05:19:05] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-09 05:19:05] ...optimizers/trijoint.py.151: total_norm: 13.447384 [I 2022-10-09 05:19:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 890/2304 [I 2022-10-09 05:19:14] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:50 [I 2022-10-09 05:19:14] ...trap/engines/engine.py.229: process: 0.83494 | load: 0.00296 [I 2022-10-09 05:19:14] ...trap/engines/engine.py.230: loss: 0.18306 [I 2022-10-09 05:19:14] ...optimizers/trijoint.py.151: total_norm: 13.760845 [I 2022-10-09 05:19:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 900/2304 [I 2022-10-09 05:19:22] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:38 [I 2022-10-09 05:19:22] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00307 [I 2022-10-09 05:19:22] ...trap/engines/engine.py.230: loss: 0.15597 [I 2022-10-09 05:19:22] ...optimizers/trijoint.py.151: total_norm: 14.582161 [I 2022-10-09 05:19:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 910/2304 [I 2022-10-09 05:19:30] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:24 [I 2022-10-09 05:19:30] ...trap/engines/engine.py.229: process: 0.82123 | load: 0.00301 [I 2022-10-09 05:19:30] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-09 05:19:30] ...optimizers/trijoint.py.151: total_norm: 10.967250 [I 2022-10-09 05:19:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 920/2304 [I 2022-10-09 05:19:39] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:21 [I 2022-10-09 05:19:39] ...trap/engines/engine.py.229: process: 0.83748 | load: 0.00299 [I 2022-10-09 05:19:39] ...trap/engines/engine.py.230: loss: 0.14705 [I 2022-10-09 05:19:39] ...optimizers/trijoint.py.151: total_norm: 17.973482 [I 2022-10-09 05:19:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 930/2304 [I 2022-10-09 05:19:47] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:15 [I 2022-10-09 05:19:47] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00300 [I 2022-10-09 05:19:47] ...trap/engines/engine.py.230: loss: 0.17904 [I 2022-10-09 05:19:47] ...optimizers/trijoint.py.151: total_norm: 19.560194 [I 2022-10-09 05:19:55] ...trap/engines/engine.py.224: train: epoch 33 | batch 940/2304 [I 2022-10-09 05:19:55] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:05 [I 2022-10-09 05:19:55] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00302 [I 2022-10-09 05:19:55] ...trap/engines/engine.py.230: loss: 0.16111 [I 2022-10-09 05:19:55] ...optimizers/trijoint.py.151: total_norm: 13.713671 [I 2022-10-09 05:20:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 950/2304 [I 2022-10-09 05:20:04] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:01 [I 2022-10-09 05:20:04] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00299 [I 2022-10-09 05:20:04] ...trap/engines/engine.py.230: loss: 0.18825 [I 2022-10-09 05:20:04] ...optimizers/trijoint.py.151: total_norm: 15.184121 [I 2022-10-09 05:20:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 960/2304 [I 2022-10-09 05:20:12] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:49 [I 2022-10-09 05:20:12] ...trap/engines/engine.py.229: process: 0.82943 | load: 0.00302 [I 2022-10-09 05:20:12] ...trap/engines/engine.py.230: loss: 0.20663 [I 2022-10-09 05:20:12] ...optimizers/trijoint.py.151: total_norm: 15.067786 [I 2022-10-09 05:20:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 970/2304 [I 2022-10-09 05:20:21] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:35 [I 2022-10-09 05:20:21] ...trap/engines/engine.py.229: process: 0.82238 | load: 0.00298 [I 2022-10-09 05:20:21] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-09 05:20:21] ...optimizers/trijoint.py.151: total_norm: 16.360630 [I 2022-10-09 05:20:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 980/2304 [I 2022-10-09 05:20:29] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:34 [I 2022-10-09 05:20:29] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00299 [I 2022-10-09 05:20:29] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-09 05:20:29] ...optimizers/trijoint.py.151: total_norm: 13.607780 [I 2022-10-09 05:20:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 990/2304 [I 2022-10-09 05:20:38] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:20 [I 2022-10-09 05:20:38] ...trap/engines/engine.py.229: process: 0.82428 | load: 0.00301 [I 2022-10-09 05:20:38] ...trap/engines/engine.py.230: loss: 0.13297 [I 2022-10-09 05:20:38] ...optimizers/trijoint.py.151: total_norm: 13.925578 [I 2022-10-09 05:20:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 1000/2304 [I 2022-10-09 05:20:46] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:16 [I 2022-10-09 05:20:46] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00302 [I 2022-10-09 05:20:46] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-09 05:20:46] ...optimizers/trijoint.py.151: total_norm: 13.751080 [I 2022-10-09 05:20:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 1010/2304 [I 2022-10-09 05:20:54] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:04 [I 2022-10-09 05:20:54] ...trap/engines/engine.py.229: process: 0.82791 | load: 0.00299 [I 2022-10-09 05:20:54] ...trap/engines/engine.py.230: loss: 0.18839 [I 2022-10-09 05:20:54] ...optimizers/trijoint.py.151: total_norm: 34.530628 [I 2022-10-09 05:21:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 1020/2304 [I 2022-10-09 05:21:03] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:00 [I 2022-10-09 05:21:03] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00301 [I 2022-10-09 05:21:03] ...trap/engines/engine.py.230: loss: 0.23603 [I 2022-10-09 05:21:03] ...optimizers/trijoint.py.151: total_norm: 19.059364 [I 2022-10-09 05:21:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 1030/2304 [I 2022-10-09 05:21:11] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:49 [I 2022-10-09 05:21:11] ...trap/engines/engine.py.229: process: 0.83174 | load: 0.00300 [I 2022-10-09 05:21:11] ...trap/engines/engine.py.230: loss: 0.12960 [I 2022-10-09 05:21:11] ...optimizers/trijoint.py.151: total_norm: 22.061941 [I 2022-10-09 05:21:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 1040/2304 [I 2022-10-09 05:21:20] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:45 [I 2022-10-09 05:21:20] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00298 [I 2022-10-09 05:21:20] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-09 05:21:20] ...optimizers/trijoint.py.151: total_norm: 20.071756 [I 2022-10-09 05:21:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 1050/2304 [I 2022-10-09 05:21:28] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:35 [I 2022-10-09 05:21:28] ...trap/engines/engine.py.229: process: 0.83170 | load: 0.00298 [I 2022-10-09 05:21:28] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-09 05:21:28] ...optimizers/trijoint.py.151: total_norm: 10.947876 [I 2022-10-09 05:21:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 1060/2304 [I 2022-10-09 05:21:36] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:19 [I 2022-10-09 05:21:36] ...trap/engines/engine.py.229: process: 0.82827 | load: 0.00300 [I 2022-10-09 05:21:36] ...trap/engines/engine.py.230: loss: 0.22924 [I 2022-10-09 05:21:36] ...optimizers/trijoint.py.151: total_norm: 25.431984 [I 2022-10-09 05:21:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 1070/2304 [I 2022-10-09 05:21:45] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:17:16 [I 2022-10-09 05:21:45] ...trap/engines/engine.py.229: process: 0.83204 | load: 0.00301 [I 2022-10-09 05:21:45] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-09 05:21:45] ...optimizers/trijoint.py.151: total_norm: 12.950367 [I 2022-10-09 05:21:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 1080/2304 [I 2022-10-09 05:21:53] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:07 [I 2022-10-09 05:21:53] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00299 [I 2022-10-09 05:21:53] ...trap/engines/engine.py.230: loss: 0.16756 [I 2022-10-09 05:21:53] ...optimizers/trijoint.py.151: total_norm: 44.563583 [I 2022-10-09 05:22:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 1090/2304 [I 2022-10-09 05:22:02] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:16:59 [I 2022-10-09 05:22:02] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00299 [I 2022-10-09 05:22:02] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-09 05:22:02] ...optimizers/trijoint.py.151: total_norm: 12.892429 [I 2022-10-09 05:22:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 1100/2304 [I 2022-10-09 05:22:10] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:53 [I 2022-10-09 05:22:10] ...trap/engines/engine.py.229: process: 0.84967 | load: 0.00297 [I 2022-10-09 05:22:10] ...trap/engines/engine.py.230: loss: 0.17023 [I 2022-10-09 05:22:10] ...optimizers/trijoint.py.151: total_norm: 25.576035 [I 2022-10-09 05:22:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 1110/2304 [I 2022-10-09 05:22:19] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:39 [I 2022-10-09 05:22:19] ...trap/engines/engine.py.229: process: 0.83869 | load: 0.00307 [I 2022-10-09 05:22:19] ...trap/engines/engine.py.230: loss: 0.15074 [I 2022-10-09 05:22:19] ...optimizers/trijoint.py.151: total_norm: 17.244661 [I 2022-10-09 05:22:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 1120/2304 [I 2022-10-09 05:22:27] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:16:28 [I 2022-10-09 05:22:27] ...trap/engines/engine.py.229: process: 0.83333 | load: 0.00305 [I 2022-10-09 05:22:27] ...trap/engines/engine.py.230: loss: 0.21520 [I 2022-10-09 05:22:27] ...optimizers/trijoint.py.151: total_norm: 23.723894 [I 2022-10-09 05:22:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 1130/2304 [I 2022-10-09 05:22:35] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:27 [I 2022-10-09 05:22:35] ...trap/engines/engine.py.229: process: 0.83502 | load: 0.00301 [I 2022-10-09 05:22:35] ...trap/engines/engine.py.230: loss: 0.23231 [I 2022-10-09 05:22:35] ...optimizers/trijoint.py.151: total_norm: 19.273592 [I 2022-10-09 05:22:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 1140/2304 [I 2022-10-09 05:22:44] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:16 [I 2022-10-09 05:22:44] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00303 [I 2022-10-09 05:22:44] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-09 05:22:44] ...optimizers/trijoint.py.151: total_norm: 14.889487 [I 2022-10-09 05:22:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 1150/2304 [I 2022-10-09 05:22:52] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:16:09 [I 2022-10-09 05:22:52] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00301 [I 2022-10-09 05:22:52] ...trap/engines/engine.py.230: loss: 0.15120 [I 2022-10-09 05:22:52] ...optimizers/trijoint.py.151: total_norm: 19.012405 [I 2022-10-09 05:23:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 1160/2304 [I 2022-10-09 05:23:01] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:05 [I 2022-10-09 05:23:01] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00302 [I 2022-10-09 05:23:01] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-09 05:23:01] ...optimizers/trijoint.py.151: total_norm: 10.665154 [I 2022-10-09 05:23:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 1170/2304 [I 2022-10-09 05:23:09] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:55 [I 2022-10-09 05:23:09] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00301 [I 2022-10-09 05:23:09] ...trap/engines/engine.py.230: loss: 0.10268 [I 2022-10-09 05:23:09] ...optimizers/trijoint.py.151: total_norm: 12.765289 [I 2022-10-09 05:23:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 1180/2304 [I 2022-10-09 05:23:17] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:15:47 [I 2022-10-09 05:23:17] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00299 [I 2022-10-09 05:23:17] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-09 05:23:17] ...optimizers/trijoint.py.151: total_norm: 9.083976 [I 2022-10-09 05:23:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 1190/2304 [I 2022-10-09 05:23:26] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:39 [I 2022-10-09 05:23:26] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00299 [I 2022-10-09 05:23:26] ...trap/engines/engine.py.230: loss: 0.14807 [I 2022-10-09 05:23:26] ...optimizers/trijoint.py.151: total_norm: 11.867142 [I 2022-10-09 05:23:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 1200/2304 [I 2022-10-09 05:23:34] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:28 [I 2022-10-09 05:23:34] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00299 [I 2022-10-09 05:23:34] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-09 05:23:34] ...optimizers/trijoint.py.151: total_norm: 23.892403 [I 2022-10-09 05:23:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 1210/2304 [I 2022-10-09 05:23:43] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:17 [I 2022-10-09 05:23:43] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00299 [I 2022-10-09 05:23:43] ...trap/engines/engine.py.230: loss: 0.17095 [I 2022-10-09 05:23:43] ...optimizers/trijoint.py.151: total_norm: 22.602427 [I 2022-10-09 05:23:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 1220/2304 [I 2022-10-09 05:23:51] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:07 [I 2022-10-09 05:23:51] ...trap/engines/engine.py.229: process: 0.82105 | load: 0.00299 [I 2022-10-09 05:23:51] ...trap/engines/engine.py.230: loss: 0.15206 [I 2022-10-09 05:23:51] ...optimizers/trijoint.py.151: total_norm: 13.898177 [I 2022-10-09 05:23:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 1230/2304 [I 2022-10-09 05:23:59] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:14:57 [I 2022-10-09 05:23:59] ...trap/engines/engine.py.229: process: 0.82227 | load: 0.00297 [I 2022-10-09 05:23:59] ...trap/engines/engine.py.230: loss: 0.10215 [I 2022-10-09 05:23:59] ...optimizers/trijoint.py.151: total_norm: 21.797319 [I 2022-10-09 05:24:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 1240/2304 [I 2022-10-09 05:24:08] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:14:55 [I 2022-10-09 05:24:08] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00298 [I 2022-10-09 05:24:08] ...trap/engines/engine.py.230: loss: 0.22558 [I 2022-10-09 05:24:08] ...optimizers/trijoint.py.151: total_norm: 43.281887 [I 2022-10-09 05:24:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 1250/2304 [I 2022-10-09 05:24:16] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:45 [I 2022-10-09 05:24:16] ...trap/engines/engine.py.229: process: 0.84148 | load: 0.00298 [I 2022-10-09 05:24:16] ...trap/engines/engine.py.230: loss: 0.17903 [I 2022-10-09 05:24:16] ...optimizers/trijoint.py.151: total_norm: 19.561842 [I 2022-10-09 05:24:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 1260/2304 [I 2022-10-09 05:24:25] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:38 [I 2022-10-09 05:24:25] ...trap/engines/engine.py.229: process: 0.83489 | load: 0.00302 [I 2022-10-09 05:24:25] ...trap/engines/engine.py.230: loss: 0.21965 [I 2022-10-09 05:24:25] ...optimizers/trijoint.py.151: total_norm: 20.448612 [I 2022-10-09 05:24:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 1270/2304 [I 2022-10-09 05:24:33] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:27 [I 2022-10-09 05:24:33] ...trap/engines/engine.py.229: process: 0.82941 | load: 0.00298 [I 2022-10-09 05:24:33] ...trap/engines/engine.py.230: loss: 0.22782 [I 2022-10-09 05:24:33] ...optimizers/trijoint.py.151: total_norm: 19.072111 [I 2022-10-09 05:24:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 1280/2304 [I 2022-10-09 05:24:42] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:19 [I 2022-10-09 05:24:42] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00303 [I 2022-10-09 05:24:42] ...trap/engines/engine.py.230: loss: 0.20546 [I 2022-10-09 05:24:42] ...optimizers/trijoint.py.151: total_norm: 14.761773 [I 2022-10-09 05:24:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 1290/2304 [I 2022-10-09 05:24:50] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:09 [I 2022-10-09 05:24:50] ...trap/engines/engine.py.229: process: 0.82592 | load: 0.00302 [I 2022-10-09 05:24:50] ...trap/engines/engine.py.230: loss: 0.18213 [I 2022-10-09 05:24:50] ...optimizers/trijoint.py.151: total_norm: 18.706497 [I 2022-10-09 05:24:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 1300/2304 [I 2022-10-09 05:24:58] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:03 [I 2022-10-09 05:24:58] ...trap/engines/engine.py.229: process: 0.82614 | load: 0.00303 [I 2022-10-09 05:24:58] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-09 05:24:58] ...optimizers/trijoint.py.151: total_norm: 18.592146 [I 2022-10-09 05:25:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 1310/2304 [I 2022-10-09 05:25:07] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:13:51 [I 2022-10-09 05:25:07] ...trap/engines/engine.py.229: process: 0.83762 | load: 0.00301 [I 2022-10-09 05:25:07] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-09 05:25:07] ...optimizers/trijoint.py.151: total_norm: 10.749695 [I 2022-10-09 05:25:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 1320/2304 [I 2022-10-09 05:25:15] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:48 [I 2022-10-09 05:25:15] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00297 [I 2022-10-09 05:25:15] ...trap/engines/engine.py.230: loss: 0.18808 [I 2022-10-09 05:25:15] ...optimizers/trijoint.py.151: total_norm: 29.900581 [I 2022-10-09 05:25:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 1330/2304 [I 2022-10-09 05:25:24] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:13:40 [I 2022-10-09 05:25:24] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00299 [I 2022-10-09 05:25:24] ...trap/engines/engine.py.230: loss: 0.14058 [I 2022-10-09 05:25:24] ...optimizers/trijoint.py.151: total_norm: 10.020180 [I 2022-10-09 05:25:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 1340/2304 [I 2022-10-09 05:25:32] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:26 [I 2022-10-09 05:25:32] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00301 [I 2022-10-09 05:25:32] ...trap/engines/engine.py.230: loss: 0.15948 [I 2022-10-09 05:25:32] ...optimizers/trijoint.py.151: total_norm: 15.454604 [I 2022-10-09 05:25:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 1350/2304 [I 2022-10-09 05:25:41] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:25 [I 2022-10-09 05:25:41] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00298 [I 2022-10-09 05:25:41] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-09 05:25:41] ...optimizers/trijoint.py.151: total_norm: 8.387331 [I 2022-10-09 05:25:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 1360/2304 [I 2022-10-09 05:25:49] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:15 [I 2022-10-09 05:25:49] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00301 [I 2022-10-09 05:25:49] ...trap/engines/engine.py.230: loss: 0.19415 [I 2022-10-09 05:25:49] ...optimizers/trijoint.py.151: total_norm: 10.231859 [I 2022-10-09 05:25:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 1370/2304 [I 2022-10-09 05:25:57] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:04 [I 2022-10-09 05:25:57] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00299 [I 2022-10-09 05:25:57] ...trap/engines/engine.py.230: loss: 0.17879 [I 2022-10-09 05:25:57] ...optimizers/trijoint.py.151: total_norm: 22.568758 [I 2022-10-09 05:26:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 1380/2304 [I 2022-10-09 05:26:06] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:12:53 [I 2022-10-09 05:26:06] ...trap/engines/engine.py.229: process: 0.83277 | load: 0.00299 [I 2022-10-09 05:26:06] ...trap/engines/engine.py.230: loss: 0.11368 [I 2022-10-09 05:26:06] ...optimizers/trijoint.py.151: total_norm: 7.553347 [I 2022-10-09 05:26:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 1390/2304 [I 2022-10-09 05:26:14] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:12:45 [I 2022-10-09 05:26:14] ...trap/engines/engine.py.229: process: 0.83127 | load: 0.00300 [I 2022-10-09 05:26:14] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-09 05:26:14] ...optimizers/trijoint.py.151: total_norm: 28.039049 [I 2022-10-09 05:26:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 1400/2304 [I 2022-10-09 05:26:23] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:36 [I 2022-10-09 05:26:23] ...trap/engines/engine.py.229: process: 0.83274 | load: 0.00302 [I 2022-10-09 05:26:23] ...trap/engines/engine.py.230: loss: 0.10758 [I 2022-10-09 05:26:23] ...optimizers/trijoint.py.151: total_norm: 9.783026 [I 2022-10-09 05:26:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 1410/2304 [I 2022-10-09 05:26:31] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:30 [I 2022-10-09 05:26:31] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00299 [I 2022-10-09 05:26:31] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-09 05:26:31] ...optimizers/trijoint.py.151: total_norm: 12.625232 [I 2022-10-09 05:26:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 1420/2304 [I 2022-10-09 05:26:39] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:23 [I 2022-10-09 05:26:39] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00301 [I 2022-10-09 05:26:39] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-09 05:26:39] ...optimizers/trijoint.py.151: total_norm: 15.659552 [I 2022-10-09 05:26:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 1430/2304 [I 2022-10-09 05:26:48] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:15 [I 2022-10-09 05:26:48] ...trap/engines/engine.py.229: process: 0.83890 | load: 0.00300 [I 2022-10-09 05:26:48] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-09 05:26:48] ...optimizers/trijoint.py.151: total_norm: 15.209239 [I 2022-10-09 05:26:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 1440/2304 [I 2022-10-09 05:26:56] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:04 [I 2022-10-09 05:26:56] ...trap/engines/engine.py.229: process: 0.83822 | load: 0.00301 [I 2022-10-09 05:26:56] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-09 05:26:56] ...optimizers/trijoint.py.151: total_norm: 12.786628 [I 2022-10-09 05:27:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 1450/2304 [I 2022-10-09 05:27:05] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:11:58 [I 2022-10-09 05:27:05] ...trap/engines/engine.py.229: process: 0.84016 | load: 0.00301 [I 2022-10-09 05:27:05] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-09 05:27:05] ...optimizers/trijoint.py.151: total_norm: 12.533902 [I 2022-10-09 05:27:13] ...trap/engines/engine.py.224: train: epoch 33 | batch 1460/2304 [I 2022-10-09 05:27:13] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:47 [I 2022-10-09 05:27:13] ...trap/engines/engine.py.229: process: 0.83602 | load: 0.00370 [I 2022-10-09 05:27:13] ...trap/engines/engine.py.230: loss: 0.17909 [I 2022-10-09 05:27:13] ...optimizers/trijoint.py.151: total_norm: 10.468788 [I 2022-10-09 05:27:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 1470/2304 [I 2022-10-09 05:27:21] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:41 [I 2022-10-09 05:27:21] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00301 [I 2022-10-09 05:27:21] ...trap/engines/engine.py.230: loss: 0.18956 [I 2022-10-09 05:27:21] ...optimizers/trijoint.py.151: total_norm: 19.042957 [I 2022-10-09 05:27:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 1480/2304 [I 2022-10-09 05:27:30] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:34 [I 2022-10-09 05:27:30] ...trap/engines/engine.py.229: process: 0.83851 | load: 0.00300 [I 2022-10-09 05:27:30] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-09 05:27:30] ...optimizers/trijoint.py.151: total_norm: 9.637327 [I 2022-10-09 05:27:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 1490/2304 [I 2022-10-09 05:27:38] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:18 [I 2022-10-09 05:27:38] ...trap/engines/engine.py.229: process: 0.83154 | load: 0.00301 [I 2022-10-09 05:27:38] ...trap/engines/engine.py.230: loss: 0.14723 [I 2022-10-09 05:27:38] ...optimizers/trijoint.py.151: total_norm: 14.453210 [I 2022-10-09 05:27:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 1500/2304 [I 2022-10-09 05:27:47] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:15 [I 2022-10-09 05:27:47] ...trap/engines/engine.py.229: process: 0.83603 | load: 0.00299 [I 2022-10-09 05:27:47] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-09 05:27:47] ...optimizers/trijoint.py.151: total_norm: 19.046566 [I 2022-10-09 05:27:55] ...trap/engines/engine.py.224: train: epoch 33 | batch 1510/2304 [I 2022-10-09 05:27:55] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:07 [I 2022-10-09 05:27:55] ...trap/engines/engine.py.229: process: 0.83127 | load: 0.00301 [I 2022-10-09 05:27:55] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-09 05:27:55] ...optimizers/trijoint.py.151: total_norm: 19.559340 [I 2022-10-09 05:28:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 1520/2304 [I 2022-10-09 05:28:04] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:10:59 [I 2022-10-09 05:28:04] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00302 [I 2022-10-09 05:28:04] ...trap/engines/engine.py.230: loss: 0.18317 [I 2022-10-09 05:28:04] ...optimizers/trijoint.py.151: total_norm: 28.190207 [I 2022-10-09 05:28:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 1530/2304 [I 2022-10-09 05:28:12] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:10:52 [I 2022-10-09 05:28:12] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00297 [I 2022-10-09 05:28:12] ...trap/engines/engine.py.230: loss: 0.13550 [I 2022-10-09 05:28:12] ...optimizers/trijoint.py.151: total_norm: 12.578032 [I 2022-10-09 05:28:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 1540/2304 [I 2022-10-09 05:28:20] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:37 [I 2022-10-09 05:28:20] ...trap/engines/engine.py.229: process: 0.82576 | load: 0.00298 [I 2022-10-09 05:28:20] ...trap/engines/engine.py.230: loss: 0.21377 [I 2022-10-09 05:28:20] ...optimizers/trijoint.py.151: total_norm: 16.569593 [I 2022-10-09 05:28:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 1550/2304 [I 2022-10-09 05:28:29] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:35 [I 2022-10-09 05:28:29] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00301 [I 2022-10-09 05:28:29] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-09 05:28:29] ...optimizers/trijoint.py.151: total_norm: 11.108881 [I 2022-10-09 05:28:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 1560/2304 [I 2022-10-09 05:28:37] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:25 [I 2022-10-09 05:28:37] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00300 [I 2022-10-09 05:28:37] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-09 05:28:37] ...optimizers/trijoint.py.151: total_norm: 15.601672 [I 2022-10-09 05:28:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 1570/2304 [I 2022-10-09 05:28:46] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:19 [I 2022-10-09 05:28:46] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00297 [I 2022-10-09 05:28:46] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-09 05:28:46] ...optimizers/trijoint.py.151: total_norm: 13.495673 [I 2022-10-09 05:28:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 1580/2304 [I 2022-10-09 05:28:54] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:05 [I 2022-10-09 05:28:54] ...trap/engines/engine.py.229: process: 0.82910 | load: 0.00299 [I 2022-10-09 05:28:54] ...trap/engines/engine.py.230: loss: 0.15978 [I 2022-10-09 05:28:54] ...optimizers/trijoint.py.151: total_norm: 10.915762 [I 2022-10-09 05:29:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 1590/2304 [I 2022-10-09 05:29:02] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:00 [I 2022-10-09 05:29:02] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00300 [I 2022-10-09 05:29:02] ...trap/engines/engine.py.230: loss: 0.13430 [I 2022-10-09 05:29:02] ...optimizers/trijoint.py.151: total_norm: 9.746505 [I 2022-10-09 05:29:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 1600/2304 [I 2022-10-09 05:29:11] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:50 [I 2022-10-09 05:29:11] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00299 [I 2022-10-09 05:29:11] ...trap/engines/engine.py.230: loss: 0.21404 [I 2022-10-09 05:29:11] ...optimizers/trijoint.py.151: total_norm: 16.840248 [I 2022-10-09 05:29:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 1610/2304 [I 2022-10-09 05:29:19] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:41 [I 2022-10-09 05:29:19] ...trap/engines/engine.py.229: process: 0.83191 | load: 0.00300 [I 2022-10-09 05:29:19] ...trap/engines/engine.py.230: loss: 0.18153 [I 2022-10-09 05:29:19] ...optimizers/trijoint.py.151: total_norm: 32.621819 [I 2022-10-09 05:29:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 1620/2304 [I 2022-10-09 05:29:28] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:36 [I 2022-10-09 05:29:28] ...trap/engines/engine.py.229: process: 0.84094 | load: 0.00306 [I 2022-10-09 05:29:28] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-09 05:29:28] ...optimizers/trijoint.py.151: total_norm: 10.040636 [I 2022-10-09 05:29:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 1630/2304 [I 2022-10-09 05:29:36] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:27 [I 2022-10-09 05:29:36] ...trap/engines/engine.py.229: process: 0.83989 | load: 0.00300 [I 2022-10-09 05:29:36] ...trap/engines/engine.py.230: loss: 0.15585 [I 2022-10-09 05:29:36] ...optimizers/trijoint.py.151: total_norm: 24.760321 [I 2022-10-09 05:29:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 1640/2304 [I 2022-10-09 05:29:45] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:16 [I 2022-10-09 05:29:45] ...trap/engines/engine.py.229: process: 0.82944 | load: 0.00298 [I 2022-10-09 05:29:45] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-09 05:29:45] ...optimizers/trijoint.py.151: total_norm: 17.105503 [I 2022-10-09 05:29:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 1650/2304 [I 2022-10-09 05:29:53] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:09 [I 2022-10-09 05:29:53] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00303 [I 2022-10-09 05:29:53] ...trap/engines/engine.py.230: loss: 0.16249 [I 2022-10-09 05:29:53] ...optimizers/trijoint.py.151: total_norm: 16.603451 [I 2022-10-09 05:30:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 1660/2304 [I 2022-10-09 05:30:01] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:08:59 [I 2022-10-09 05:30:01] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00302 [I 2022-10-09 05:30:01] ...trap/engines/engine.py.230: loss: 0.16319 [I 2022-10-09 05:30:01] ...optimizers/trijoint.py.151: total_norm: 11.529868 [I 2022-10-09 05:30:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 1670/2304 [I 2022-10-09 05:30:10] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:08:51 [I 2022-10-09 05:30:10] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00301 [I 2022-10-09 05:30:10] ...trap/engines/engine.py.230: loss: 0.16774 [I 2022-10-09 05:30:10] ...optimizers/trijoint.py.151: total_norm: 21.260769 [I 2022-10-09 05:30:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 1680/2304 [I 2022-10-09 05:30:18] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:44 [I 2022-10-09 05:30:18] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00303 [I 2022-10-09 05:30:18] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-09 05:30:18] ...optimizers/trijoint.py.151: total_norm: 14.892446 [I 2022-10-09 05:30:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 1690/2304 [I 2022-10-09 05:30:27] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:39 [I 2022-10-09 05:30:27] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00306 [I 2022-10-09 05:30:27] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-09 05:30:27] ...optimizers/trijoint.py.151: total_norm: 14.169618 [I 2022-10-09 05:30:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 1700/2304 [I 2022-10-09 05:30:35] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:25 [I 2022-10-09 05:30:35] ...trap/engines/engine.py.229: process: 0.82363 | load: 0.00303 [I 2022-10-09 05:30:35] ...trap/engines/engine.py.230: loss: 0.24300 [I 2022-10-09 05:30:35] ...optimizers/trijoint.py.151: total_norm: 20.370634 [I 2022-10-09 05:30:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 1710/2304 [I 2022-10-09 05:30:43] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:17 [I 2022-10-09 05:30:43] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00302 [I 2022-10-09 05:30:43] ...trap/engines/engine.py.230: loss: 0.14861 [I 2022-10-09 05:30:43] ...optimizers/trijoint.py.151: total_norm: 11.893880 [I 2022-10-09 05:30:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 1720/2304 [I 2022-10-09 05:30:52] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:08 [I 2022-10-09 05:30:52] ...trap/engines/engine.py.229: process: 0.82385 | load: 0.00301 [I 2022-10-09 05:30:52] ...trap/engines/engine.py.230: loss: 0.21021 [I 2022-10-09 05:30:52] ...optimizers/trijoint.py.151: total_norm: 24.448046 [I 2022-10-09 05:31:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 1730/2304 [I 2022-10-09 05:31:00] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:07:58 [I 2022-10-09 05:31:00] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00297 [I 2022-10-09 05:31:00] ...trap/engines/engine.py.230: loss: 0.19617 [I 2022-10-09 05:31:00] ...optimizers/trijoint.py.151: total_norm: 20.626940 [I 2022-10-09 05:31:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 1740/2304 [I 2022-10-09 05:31:09] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:55 [I 2022-10-09 05:31:09] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00302 [I 2022-10-09 05:31:09] ...trap/engines/engine.py.230: loss: 0.16453 [I 2022-10-09 05:31:09] ...optimizers/trijoint.py.151: total_norm: 18.285624 [I 2022-10-09 05:31:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 1750/2304 [I 2022-10-09 05:31:17] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:45 [I 2022-10-09 05:31:17] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00304 [I 2022-10-09 05:31:17] ...trap/engines/engine.py.230: loss: 0.14723 [I 2022-10-09 05:31:17] ...optimizers/trijoint.py.151: total_norm: 22.541451 [I 2022-10-09 05:31:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 1760/2304 [I 2022-10-09 05:31:25] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:37 [I 2022-10-09 05:31:25] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00302 [I 2022-10-09 05:31:25] ...trap/engines/engine.py.230: loss: 0.21942 [I 2022-10-09 05:31:25] ...optimizers/trijoint.py.151: total_norm: 16.577570 [I 2022-10-09 05:31:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 1770/2304 [I 2022-10-09 05:31:34] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:29 [I 2022-10-09 05:31:34] ...trap/engines/engine.py.229: process: 0.84280 | load: 0.00299 [I 2022-10-09 05:31:34] ...trap/engines/engine.py.230: loss: 0.14783 [I 2022-10-09 05:31:34] ...optimizers/trijoint.py.151: total_norm: 9.834743 [I 2022-10-09 05:31:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 1780/2304 [I 2022-10-09 05:31:42] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:22 [I 2022-10-09 05:31:42] ...trap/engines/engine.py.229: process: 0.83001 | load: 0.00298 [I 2022-10-09 05:31:42] ...trap/engines/engine.py.230: loss: 0.22445 [I 2022-10-09 05:31:42] ...optimizers/trijoint.py.151: total_norm: 27.537474 [I 2022-10-09 05:31:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 1790/2304 [I 2022-10-09 05:31:51] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:13 [I 2022-10-09 05:31:51] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00305 [I 2022-10-09 05:31:51] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-09 05:31:51] ...optimizers/trijoint.py.151: total_norm: 17.349266 [I 2022-10-09 05:31:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 1800/2304 [I 2022-10-09 05:31:59] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:04 [I 2022-10-09 05:31:59] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00298 [I 2022-10-09 05:31:59] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-09 05:31:59] ...optimizers/trijoint.py.151: total_norm: 22.455656 [I 2022-10-09 05:32:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 1810/2304 [I 2022-10-09 05:32:08] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:55 [I 2022-10-09 05:32:08] ...trap/engines/engine.py.229: process: 0.84154 | load: 0.00299 [I 2022-10-09 05:32:08] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-09 05:32:08] ...optimizers/trijoint.py.151: total_norm: 10.426132 [I 2022-10-09 05:32:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 1820/2304 [I 2022-10-09 05:32:16] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:48 [I 2022-10-09 05:32:16] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00301 [I 2022-10-09 05:32:16] ...trap/engines/engine.py.230: loss: 0.23711 [I 2022-10-09 05:32:16] ...optimizers/trijoint.py.151: total_norm: 21.631308 [I 2022-10-09 05:32:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 1830/2304 [I 2022-10-09 05:32:24] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:37 [I 2022-10-09 05:32:24] ...trap/engines/engine.py.229: process: 0.84430 | load: 0.00299 [I 2022-10-09 05:32:24] ...trap/engines/engine.py.230: loss: 0.15797 [I 2022-10-09 05:32:24] ...optimizers/trijoint.py.151: total_norm: 29.237108 [I 2022-10-09 05:32:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 1840/2304 [I 2022-10-09 05:32:33] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:27 [I 2022-10-09 05:32:33] ...trap/engines/engine.py.229: process: 0.81458 | load: 0.00304 [I 2022-10-09 05:32:33] ...trap/engines/engine.py.230: loss: 0.08189 [I 2022-10-09 05:32:33] ...optimizers/trijoint.py.151: total_norm: 7.415383 [I 2022-10-09 05:32:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 1850/2304 [I 2022-10-09 05:32:41] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:21 [I 2022-10-09 05:32:41] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00301 [I 2022-10-09 05:32:41] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-09 05:32:41] ...optimizers/trijoint.py.151: total_norm: 16.677753 [I 2022-10-09 05:32:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 1860/2304 [I 2022-10-09 05:32:50] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:12 [I 2022-10-09 05:32:50] ...trap/engines/engine.py.229: process: 0.83029 | load: 0.00304 [I 2022-10-09 05:32:50] ...trap/engines/engine.py.230: loss: 0.20285 [I 2022-10-09 05:32:50] ...optimizers/trijoint.py.151: total_norm: 18.229645 [I 2022-10-09 05:32:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 1870/2304 [I 2022-10-09 05:32:58] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:04 [I 2022-10-09 05:32:58] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00301 [I 2022-10-09 05:32:58] ...trap/engines/engine.py.230: loss: 0.14912 [I 2022-10-09 05:32:58] ...optimizers/trijoint.py.151: total_norm: 12.835787 [I 2022-10-09 05:33:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 1880/2304 [I 2022-10-09 05:33:06] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:55 [I 2022-10-09 05:33:06] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00307 [I 2022-10-09 05:33:06] ...trap/engines/engine.py.230: loss: 0.13669 [I 2022-10-09 05:33:06] ...optimizers/trijoint.py.151: total_norm: 16.602678 [I 2022-10-09 05:33:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 1890/2304 [I 2022-10-09 05:33:15] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:48 [I 2022-10-09 05:33:15] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00300 [I 2022-10-09 05:33:15] ...trap/engines/engine.py.230: loss: 0.12914 [I 2022-10-09 05:33:15] ...optimizers/trijoint.py.151: total_norm: 19.372025 [I 2022-10-09 05:33:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 1900/2304 [I 2022-10-09 05:33:23] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:39 [I 2022-10-09 05:33:23] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00302 [I 2022-10-09 05:33:23] ...trap/engines/engine.py.230: loss: 0.16746 [I 2022-10-09 05:33:23] ...optimizers/trijoint.py.151: total_norm: 30.295761 [I 2022-10-09 05:33:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 1910/2304 [I 2022-10-09 05:33:32] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:29 [I 2022-10-09 05:33:32] ...trap/engines/engine.py.229: process: 0.82170 | load: 0.00299 [I 2022-10-09 05:33:32] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-09 05:33:32] ...optimizers/trijoint.py.151: total_norm: 10.802999 [I 2022-10-09 05:33:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 1920/2304 [I 2022-10-09 05:33:40] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:23 [I 2022-10-09 05:33:40] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00303 [I 2022-10-09 05:33:40] ...trap/engines/engine.py.230: loss: 0.16803 [I 2022-10-09 05:33:40] ...optimizers/trijoint.py.151: total_norm: 13.531598 [I 2022-10-09 05:33:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 1930/2304 [I 2022-10-09 05:33:49] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:14 [I 2022-10-09 05:33:49] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00302 [I 2022-10-09 05:33:49] ...trap/engines/engine.py.230: loss: 0.18176 [I 2022-10-09 05:33:49] ...optimizers/trijoint.py.151: total_norm: 17.023706 [I 2022-10-09 05:33:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 1940/2304 [I 2022-10-09 05:33:57] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:06 [I 2022-10-09 05:33:57] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00304 [I 2022-10-09 05:33:57] ...trap/engines/engine.py.230: loss: 0.19487 [I 2022-10-09 05:33:57] ...optimizers/trijoint.py.151: total_norm: 10.480829 [I 2022-10-09 05:34:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 1950/2304 [I 2022-10-09 05:34:05] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:56 [I 2022-10-09 05:34:05] ...trap/engines/engine.py.229: process: 0.81834 | load: 0.00297 [I 2022-10-09 05:34:05] ...trap/engines/engine.py.230: loss: 0.15403 [I 2022-10-09 05:34:05] ...optimizers/trijoint.py.151: total_norm: 14.767381 [I 2022-10-09 05:34:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 1960/2304 [I 2022-10-09 05:34:14] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:48 [I 2022-10-09 05:34:14] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00298 [I 2022-10-09 05:34:14] ...trap/engines/engine.py.230: loss: 0.16857 [I 2022-10-09 05:34:14] ...optimizers/trijoint.py.151: total_norm: 15.914373 [I 2022-10-09 05:34:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 1970/2304 [I 2022-10-09 05:34:22] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:41 [I 2022-10-09 05:34:22] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00300 [I 2022-10-09 05:34:22] ...trap/engines/engine.py.230: loss: 0.21694 [I 2022-10-09 05:34:22] ...optimizers/trijoint.py.151: total_norm: 21.271059 [I 2022-10-09 05:34:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 1980/2304 [I 2022-10-09 05:34:31] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:33 [I 2022-10-09 05:34:31] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00300 [I 2022-10-09 05:34:31] ...trap/engines/engine.py.230: loss: 0.25596 [I 2022-10-09 05:34:31] ...optimizers/trijoint.py.151: total_norm: 12.321659 [I 2022-10-09 05:34:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 1990/2304 [I 2022-10-09 05:34:39] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:22 [I 2022-10-09 05:34:39] ...trap/engines/engine.py.229: process: 0.83077 | load: 0.00306 [I 2022-10-09 05:34:39] ...trap/engines/engine.py.230: loss: 0.18367 [I 2022-10-09 05:34:39] ...optimizers/trijoint.py.151: total_norm: 18.400560 [I 2022-10-09 05:34:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 2000/2304 [I 2022-10-09 05:34:48] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:15 [I 2022-10-09 05:34:48] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00297 [I 2022-10-09 05:34:48] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-09 05:34:48] ...optimizers/trijoint.py.151: total_norm: 10.043967 [I 2022-10-09 05:34:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 2010/2304 [I 2022-10-09 05:34:56] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:06 [I 2022-10-09 05:34:56] ...trap/engines/engine.py.229: process: 0.83787 | load: 0.00299 [I 2022-10-09 05:34:56] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-09 05:34:56] ...optimizers/trijoint.py.151: total_norm: 30.784237 [I 2022-10-09 05:35:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 2020/2304 [I 2022-10-09 05:35:04] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:57 [I 2022-10-09 05:35:04] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00301 [I 2022-10-09 05:35:04] ...trap/engines/engine.py.230: loss: 0.20304 [I 2022-10-09 05:35:04] ...optimizers/trijoint.py.151: total_norm: 20.294214 [I 2022-10-09 05:35:13] ...trap/engines/engine.py.224: train: epoch 33 | batch 2030/2304 [I 2022-10-09 05:35:13] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:49 [I 2022-10-09 05:35:13] ...trap/engines/engine.py.229: process: 0.83129 | load: 0.00299 [I 2022-10-09 05:35:13] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-09 05:35:13] ...optimizers/trijoint.py.151: total_norm: 12.883976 [I 2022-10-09 05:35:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 2040/2304 [I 2022-10-09 05:35:21] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:40 [I 2022-10-09 05:35:21] ...trap/engines/engine.py.229: process: 0.84263 | load: 0.00297 [I 2022-10-09 05:35:21] ...trap/engines/engine.py.230: loss: 0.13865 [I 2022-10-09 05:35:21] ...optimizers/trijoint.py.151: total_norm: 15.079542 [I 2022-10-09 05:35:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 2050/2304 [I 2022-10-09 05:35:30] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:33 [I 2022-10-09 05:35:30] ...trap/engines/engine.py.229: process: 0.84947 | load: 0.00300 [I 2022-10-09 05:35:30] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-09 05:35:30] ...optimizers/trijoint.py.151: total_norm: 20.212267 [I 2022-10-09 05:35:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 2060/2304 [I 2022-10-09 05:35:38] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:25 [I 2022-10-09 05:35:38] ...trap/engines/engine.py.229: process: 0.83339 | load: 0.00299 [I 2022-10-09 05:35:38] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-09 05:35:38] ...optimizers/trijoint.py.151: total_norm: 15.318663 [I 2022-10-09 05:35:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 2070/2304 [I 2022-10-09 05:35:46] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:17 [I 2022-10-09 05:35:46] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00301 [I 2022-10-09 05:35:46] ...trap/engines/engine.py.230: loss: 0.17666 [I 2022-10-09 05:35:46] ...optimizers/trijoint.py.151: total_norm: 12.033881 [I 2022-10-09 05:35:55] ...trap/engines/engine.py.224: train: epoch 33 | batch 2080/2304 [I 2022-10-09 05:35:55] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:08 [I 2022-10-09 05:35:55] ...trap/engines/engine.py.229: process: 0.84263 | load: 0.00302 [I 2022-10-09 05:35:55] ...trap/engines/engine.py.230: loss: 0.18174 [I 2022-10-09 05:35:55] ...optimizers/trijoint.py.151: total_norm: 12.585421 [I 2022-10-09 05:36:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 2090/2304 [I 2022-10-09 05:36:03] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:59 [I 2022-10-09 05:36:03] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00302 [I 2022-10-09 05:36:03] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-09 05:36:03] ...optimizers/trijoint.py.151: total_norm: 25.766935 [I 2022-10-09 05:36:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 2100/2304 [I 2022-10-09 05:36:12] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:50 [I 2022-10-09 05:36:12] ...trap/engines/engine.py.229: process: 0.83676 | load: 0.00303 [I 2022-10-09 05:36:12] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-09 05:36:12] ...optimizers/trijoint.py.151: total_norm: 28.408842 [I 2022-10-09 05:36:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 2110/2304 [I 2022-10-09 05:36:20] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:43 [I 2022-10-09 05:36:20] ...trap/engines/engine.py.229: process: 0.85115 | load: 0.00299 [I 2022-10-09 05:36:20] ...trap/engines/engine.py.230: loss: 0.20757 [I 2022-10-09 05:36:20] ...optimizers/trijoint.py.151: total_norm: 16.736553 [I 2022-10-09 05:36:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 2120/2304 [I 2022-10-09 05:36:28] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:34 [I 2022-10-09 05:36:28] ...trap/engines/engine.py.229: process: 0.83019 | load: 0.00300 [I 2022-10-09 05:36:28] ...trap/engines/engine.py.230: loss: 0.13373 [I 2022-10-09 05:36:28] ...optimizers/trijoint.py.151: total_norm: 15.490137 [I 2022-10-09 05:36:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 2130/2304 [I 2022-10-09 05:36:37] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:25 [I 2022-10-09 05:36:37] ...trap/engines/engine.py.229: process: 0.83118 | load: 0.00313 [I 2022-10-09 05:36:37] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-09 05:36:37] ...optimizers/trijoint.py.151: total_norm: 13.528736 [I 2022-10-09 05:36:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 2140/2304 [I 2022-10-09 05:36:45] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:17 [I 2022-10-09 05:36:45] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00301 [I 2022-10-09 05:36:45] ...trap/engines/engine.py.230: loss: 0.13407 [I 2022-10-09 05:36:45] ...optimizers/trijoint.py.151: total_norm: 22.581749 [I 2022-10-09 05:36:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 2150/2304 [I 2022-10-09 05:36:54] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:10 [I 2022-10-09 05:36:54] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00316 [I 2022-10-09 05:36:54] ...trap/engines/engine.py.230: loss: 0.20051 [I 2022-10-09 05:36:54] ...optimizers/trijoint.py.151: total_norm: 18.779703 [I 2022-10-09 05:37:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 2160/2304 [I 2022-10-09 05:37:02] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:01 [I 2022-10-09 05:37:02] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00300 [I 2022-10-09 05:37:02] ...trap/engines/engine.py.230: loss: 0.10878 [I 2022-10-09 05:37:02] ...optimizers/trijoint.py.151: total_norm: 8.524913 [I 2022-10-09 05:37:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 2170/2304 [I 2022-10-09 05:37:11] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:53 [I 2022-10-09 05:37:11] ...trap/engines/engine.py.229: process: 0.83409 | load: 0.00312 [I 2022-10-09 05:37:11] ...trap/engines/engine.py.230: loss: 0.16650 [I 2022-10-09 05:37:11] ...optimizers/trijoint.py.151: total_norm: 26.014526 [I 2022-10-09 05:37:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 2180/2304 [I 2022-10-09 05:37:19] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:45 [I 2022-10-09 05:37:19] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00309 [I 2022-10-09 05:37:19] ...trap/engines/engine.py.230: loss: 0.18295 [I 2022-10-09 05:37:19] ...optimizers/trijoint.py.151: total_norm: 15.267000 [I 2022-10-09 05:37:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 2190/2304 [I 2022-10-09 05:37:28] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:35 [I 2022-10-09 05:37:28] ...trap/engines/engine.py.229: process: 0.82793 | load: 0.00309 [I 2022-10-09 05:37:28] ...trap/engines/engine.py.230: loss: 0.20021 [I 2022-10-09 05:37:28] ...optimizers/trijoint.py.151: total_norm: 32.282665 [I 2022-10-09 05:37:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 2200/2304 [I 2022-10-09 05:37:36] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:27 [I 2022-10-09 05:37:36] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00306 [I 2022-10-09 05:37:36] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-09 05:37:36] ...optimizers/trijoint.py.151: total_norm: 16.443661 [I 2022-10-09 05:37:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 2210/2304 [I 2022-10-09 05:37:44] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:18 [I 2022-10-09 05:37:44] ...trap/engines/engine.py.229: process: 0.82857 | load: 0.00303 [I 2022-10-09 05:37:44] ...trap/engines/engine.py.230: loss: 0.17884 [I 2022-10-09 05:37:44] ...optimizers/trijoint.py.151: total_norm: 24.056211 [I 2022-10-09 05:37:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 2220/2304 [I 2022-10-09 05:37:53] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:10 [I 2022-10-09 05:37:53] ...trap/engines/engine.py.229: process: 0.85391 | load: 0.00302 [I 2022-10-09 05:37:53] ...trap/engines/engine.py.230: loss: 0.16924 [I 2022-10-09 05:37:53] ...optimizers/trijoint.py.151: total_norm: 14.686213 [I 2022-10-09 05:38:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 2230/2304 [I 2022-10-09 05:38:01] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:01 [I 2022-10-09 05:38:01] ...trap/engines/engine.py.229: process: 0.83282 | load: 0.00300 [I 2022-10-09 05:38:01] ...trap/engines/engine.py.230: loss: 0.18885 [I 2022-10-09 05:38:01] ...optimizers/trijoint.py.151: total_norm: 17.611824 [I 2022-10-09 05:38:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 2240/2304 [I 2022-10-09 05:38:10] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:53 [I 2022-10-09 05:38:10] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00305 [I 2022-10-09 05:38:10] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-09 05:38:10] ...optimizers/trijoint.py.151: total_norm: 13.685633 [I 2022-10-09 05:38:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 2250/2304 [I 2022-10-09 05:38:18] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:45 [I 2022-10-09 05:38:18] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00304 [I 2022-10-09 05:38:18] ...trap/engines/engine.py.230: loss: 0.18762 [I 2022-10-09 05:38:18] ...optimizers/trijoint.py.151: total_norm: 24.934477 [I 2022-10-09 05:38:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 2260/2304 [I 2022-10-09 05:38:27] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:36 [I 2022-10-09 05:38:27] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00304 [I 2022-10-09 05:38:27] ...trap/engines/engine.py.230: loss: 0.14074 [I 2022-10-09 05:38:27] ...optimizers/trijoint.py.151: total_norm: 17.274803 [I 2022-10-09 05:38:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 2270/2304 [I 2022-10-09 05:38:35] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:28 [I 2022-10-09 05:38:35] ...trap/engines/engine.py.229: process: 0.85027 | load: 0.00305 [I 2022-10-09 05:38:35] ...trap/engines/engine.py.230: loss: 0.15650 [I 2022-10-09 05:38:35] ...optimizers/trijoint.py.151: total_norm: 20.647043 [I 2022-10-09 05:38:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 2280/2304 [I 2022-10-09 05:38:43] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:20 [I 2022-10-09 05:38:43] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00306 [I 2022-10-09 05:38:43] ...trap/engines/engine.py.230: loss: 0.14327 [I 2022-10-09 05:38:43] ...optimizers/trijoint.py.151: total_norm: 8.741467 [I 2022-10-09 05:38:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 2290/2304 [I 2022-10-09 05:38:52] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:11 [I 2022-10-09 05:38:52] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00304 [I 2022-10-09 05:38:52] ...trap/engines/engine.py.230: loss: 0.23659 [I 2022-10-09 05:38:52] ...optimizers/trijoint.py.151: total_norm: 21.711897 [I 2022-10-09 05:39:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 2300/2304 [I 2022-10-09 05:39:00] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:03 [I 2022-10-09 05:39:00] ...trap/engines/engine.py.229: process: 0.82935 | load: 0.00017 [I 2022-10-09 05:39:00] ...trap/engines/engine.py.230: loss: 0.12737 [I 2022-10-09 05:39:00] ...optimizers/trijoint.py.151: total_norm: 19.644077 [I 2022-10-09 05:39:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 2304/2304 [I 2022-10-09 05:39:04] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:00 [I 2022-10-09 05:39:04] ...trap/engines/engine.py.229: process: 0.83030 | load: 0.00017 [I 2022-10-09 05:39:04] ...trap/engines/engine.py.230: loss: 0.12489 [I 2022-10-09 05:39:04] ...optimizers/trijoint.py.151: total_norm: 34.732815 [S 2022-10-09 05:39:05] ...trap/engines/engine.py.129: train_epoch.epoch: 33 [S 2022-10-09 05:39:05] ...trap/engines/engine.py.129: train_epoch.loss: 0.1645112181110651 [I 2022-10-09 05:39:05] ...s/criterions/triplet.py.84: margin = 0.22000000000000014 [I 2022-10-09 05:39:05] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 33 [I 2022-10-09 05:39:06] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 05:39:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 05:39:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 05:39:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 05:39:09] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 05:39:09] ...trap/engines/engine.py.321: eval: epoch 33 | batch 0/510 [I 2022-10-09 05:39:09] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:30:43 [I 2022-10-09 05:39:09] ...trap/engines/engine.py.326: process: 3.61511 | load: 3.10960 [I 2022-10-09 05:39:15] ...trap/engines/engine.py.321: eval: epoch 33 | batch 10/510 [I 2022-10-09 05:39:15] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:36 [I 2022-10-09 05:39:15] ...trap/engines/engine.py.326: process: 0.59338 | load: 0.00019 [I 2022-10-09 05:39:21] ...trap/engines/engine.py.321: eval: epoch 33 | batch 20/510 [I 2022-10-09 05:39:21] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:23 [I 2022-10-09 05:39:21] ...trap/engines/engine.py.326: process: 0.56571 | load: 0.00019 [I 2022-10-09 05:39:26] ...trap/engines/engine.py.321: eval: epoch 33 | batch 30/510 [I 2022-10-09 05:39:26] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:56 [I 2022-10-09 05:39:26] ...trap/engines/engine.py.326: process: 0.41428 | load: 0.00019 [I 2022-10-09 05:39:30] ...trap/engines/engine.py.321: eval: epoch 33 | batch 40/510 [I 2022-10-09 05:39:30] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:46 [I 2022-10-09 05:39:30] ...trap/engines/engine.py.326: process: 0.55365 | load: 0.00018 [I 2022-10-09 05:39:35] ...trap/engines/engine.py.321: eval: epoch 33 | batch 50/510 [I 2022-10-09 05:39:35] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:52 [I 2022-10-09 05:39:35] ...trap/engines/engine.py.326: process: 0.41597 | load: 0.00019 [I 2022-10-09 05:39:41] ...trap/engines/engine.py.321: eval: epoch 33 | batch 60/510 [I 2022-10-09 05:39:41] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:18 [I 2022-10-09 05:39:41] ...trap/engines/engine.py.326: process: 0.59724 | load: 0.00018 [I 2022-10-09 05:39:46] ...trap/engines/engine.py.321: eval: epoch 33 | batch 70/510 [I 2022-10-09 05:39:46] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:27 [I 2022-10-09 05:39:46] ...trap/engines/engine.py.326: process: 0.40495 | load: 0.00019 [I 2022-10-09 05:39:51] ...trap/engines/engine.py.321: eval: epoch 33 | batch 80/510 [I 2022-10-09 05:39:51] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:34 [I 2022-10-09 05:39:51] ...trap/engines/engine.py.326: process: 0.47664 | load: 0.00020 [I 2022-10-09 05:39:56] ...trap/engines/engine.py.321: eval: epoch 33 | batch 90/510 [I 2022-10-09 05:39:56] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:29 [I 2022-10-09 05:39:56] ...trap/engines/engine.py.326: process: 0.48431 | load: 0.00019 [I 2022-10-09 05:40:00] ...trap/engines/engine.py.321: eval: epoch 33 | batch 100/510 [I 2022-10-09 05:40:00] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:01 [I 2022-10-09 05:40:00] ...trap/engines/engine.py.326: process: 0.30744 | load: 0.00018 [I 2022-10-09 05:40:04] ...trap/engines/engine.py.321: eval: epoch 33 | batch 110/510 [I 2022-10-09 05:40:04] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-09 05:40:04] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00021 [I 2022-10-09 05:40:07] ...trap/engines/engine.py.321: eval: epoch 33 | batch 120/510 [I 2022-10-09 05:40:07] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-09 05:40:07] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00019 [I 2022-10-09 05:40:10] ...trap/engines/engine.py.321: eval: epoch 33 | batch 130/510 [I 2022-10-09 05:40:10] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-09 05:40:10] ...trap/engines/engine.py.326: process: 0.31780 | load: 0.00020 [I 2022-10-09 05:40:13] ...trap/engines/engine.py.321: eval: epoch 33 | batch 140/510 [I 2022-10-09 05:40:13] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-09 05:40:13] ...trap/engines/engine.py.326: process: 0.31891 | load: 0.00020 [I 2022-10-09 05:40:16] ...trap/engines/engine.py.321: eval: epoch 33 | batch 150/510 [I 2022-10-09 05:40:16] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:55 [I 2022-10-09 05:40:16] ...trap/engines/engine.py.326: process: 0.32039 | load: 0.00020 [I 2022-10-09 05:40:19] ...trap/engines/engine.py.321: eval: epoch 33 | batch 160/510 [I 2022-10-09 05:40:19] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:51 [I 2022-10-09 05:40:19] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00019 [I 2022-10-09 05:40:23] ...trap/engines/engine.py.321: eval: epoch 33 | batch 170/510 [I 2022-10-09 05:40:23] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-09 05:40:23] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00020 [I 2022-10-09 05:40:26] ...trap/engines/engine.py.321: eval: epoch 33 | batch 180/510 [I 2022-10-09 05:40:26] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-09 05:40:26] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00020 [I 2022-10-09 05:40:29] ...trap/engines/engine.py.321: eval: epoch 33 | batch 190/510 [I 2022-10-09 05:40:29] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:41 [I 2022-10-09 05:40:29] ...trap/engines/engine.py.326: process: 0.31936 | load: 0.00020 [I 2022-10-09 05:40:32] ...trap/engines/engine.py.321: eval: epoch 33 | batch 200/510 [I 2022-10-09 05:40:32] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-09 05:40:32] ...trap/engines/engine.py.326: process: 0.32284 | load: 0.00020 [I 2022-10-09 05:40:35] ...trap/engines/engine.py.321: eval: epoch 33 | batch 210/510 [I 2022-10-09 05:40:35] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-09 05:40:35] ...trap/engines/engine.py.326: process: 0.30540 | load: 0.00020 [I 2022-10-09 05:40:39] ...trap/engines/engine.py.321: eval: epoch 33 | batch 220/510 [I 2022-10-09 05:40:39] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-09 05:40:39] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00019 [I 2022-10-09 05:40:42] ...trap/engines/engine.py.321: eval: epoch 33 | batch 230/510 [I 2022-10-09 05:40:42] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-09 05:40:42] ...trap/engines/engine.py.326: process: 0.31739 | load: 0.00021 [I 2022-10-09 05:40:45] ...trap/engines/engine.py.321: eval: epoch 33 | batch 240/510 [I 2022-10-09 05:40:45] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-09 05:40:45] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00019 [I 2022-10-09 05:40:48] ...trap/engines/engine.py.321: eval: epoch 33 | batch 250/510 [I 2022-10-09 05:40:48] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-09 05:40:48] ...trap/engines/engine.py.326: process: 0.31969 | load: 0.00023 [I 2022-10-09 05:40:51] ...trap/engines/engine.py.321: eval: epoch 33 | batch 260/510 [I 2022-10-09 05:40:51] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:19 [I 2022-10-09 05:40:51] ...trap/engines/engine.py.326: process: 0.32152 | load: 0.00021 [I 2022-10-09 05:40:54] ...trap/engines/engine.py.321: eval: epoch 33 | batch 270/510 [I 2022-10-09 05:40:54] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 05:40:54] ...trap/engines/engine.py.326: process: 0.32290 | load: 0.00022 [I 2022-10-09 05:40:58] ...trap/engines/engine.py.321: eval: epoch 33 | batch 280/510 [I 2022-10-09 05:40:58] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 05:40:58] ...trap/engines/engine.py.326: process: 0.33155 | load: 0.00020 [I 2022-10-09 05:41:01] ...trap/engines/engine.py.321: eval: epoch 33 | batch 290/510 [I 2022-10-09 05:41:01] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-09 05:41:01] ...trap/engines/engine.py.326: process: 0.31882 | load: 0.00021 [I 2022-10-09 05:41:04] ...trap/engines/engine.py.321: eval: epoch 33 | batch 300/510 [I 2022-10-09 05:41:04] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-09 05:41:04] ...trap/engines/engine.py.326: process: 0.31350 | load: 0.00020 [I 2022-10-09 05:41:07] ...trap/engines/engine.py.321: eval: epoch 33 | batch 310/510 [I 2022-10-09 05:41:07] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:03 [I 2022-10-09 05:41:07] ...trap/engines/engine.py.326: process: 0.31376 | load: 0.00020 [I 2022-10-09 05:41:10] ...trap/engines/engine.py.321: eval: epoch 33 | batch 320/510 [I 2022-10-09 05:41:10] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 05:41:10] ...trap/engines/engine.py.326: process: 0.32194 | load: 0.00020 [I 2022-10-09 05:41:14] ...trap/engines/engine.py.321: eval: epoch 33 | batch 330/510 [I 2022-10-09 05:41:14] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 05:41:14] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00020 [I 2022-10-09 05:41:17] ...trap/engines/engine.py.321: eval: epoch 33 | batch 340/510 [I 2022-10-09 05:41:17] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 05:41:17] ...trap/engines/engine.py.326: process: 0.31586 | load: 0.00022 [I 2022-10-09 05:41:20] ...trap/engines/engine.py.321: eval: epoch 33 | batch 350/510 [I 2022-10-09 05:41:20] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:50 [I 2022-10-09 05:41:20] ...trap/engines/engine.py.326: process: 0.31758 | load: 0.00020 [I 2022-10-09 05:41:23] ...trap/engines/engine.py.321: eval: epoch 33 | batch 360/510 [I 2022-10-09 05:41:23] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-09 05:41:23] ...trap/engines/engine.py.326: process: 0.31802 | load: 0.00020 [I 2022-10-09 05:41:26] ...trap/engines/engine.py.321: eval: epoch 33 | batch 370/510 [I 2022-10-09 05:41:26] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-09 05:41:26] ...trap/engines/engine.py.326: process: 0.32030 | load: 0.00020 [I 2022-10-09 05:41:30] ...trap/engines/engine.py.321: eval: epoch 33 | batch 380/510 [I 2022-10-09 05:41:30] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 05:41:30] ...trap/engines/engine.py.326: process: 0.31547 | load: 0.00021 [I 2022-10-09 05:41:33] ...trap/engines/engine.py.321: eval: epoch 33 | batch 390/510 [I 2022-10-09 05:41:33] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:37 [I 2022-10-09 05:41:33] ...trap/engines/engine.py.326: process: 0.31007 | load: 0.00021 [I 2022-10-09 05:41:36] ...trap/engines/engine.py.321: eval: epoch 33 | batch 400/510 [I 2022-10-09 05:41:36] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:34 [I 2022-10-09 05:41:36] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00020 [I 2022-10-09 05:41:39] ...trap/engines/engine.py.321: eval: epoch 33 | batch 410/510 [I 2022-10-09 05:41:39] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-09 05:41:39] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00020 [I 2022-10-09 05:41:42] ...trap/engines/engine.py.321: eval: epoch 33 | batch 420/510 [I 2022-10-09 05:41:42] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 05:41:42] ...trap/engines/engine.py.326: process: 0.31819 | load: 0.00020 [I 2022-10-09 05:41:46] ...trap/engines/engine.py.321: eval: epoch 33 | batch 430/510 [I 2022-10-09 05:41:46] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 05:41:46] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00021 [I 2022-10-09 05:41:49] ...trap/engines/engine.py.321: eval: epoch 33 | batch 440/510 [I 2022-10-09 05:41:49] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 05:41:49] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00021 [I 2022-10-09 05:41:52] ...trap/engines/engine.py.321: eval: epoch 33 | batch 450/510 [I 2022-10-09 05:41:52] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 05:41:52] ...trap/engines/engine.py.326: process: 0.32056 | load: 0.00019 [I 2022-10-09 05:41:55] ...trap/engines/engine.py.321: eval: epoch 33 | batch 460/510 [I 2022-10-09 05:41:55] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-09 05:41:55] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00020 [I 2022-10-09 05:41:58] ...trap/engines/engine.py.321: eval: epoch 33 | batch 470/510 [I 2022-10-09 05:41:58] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 05:41:58] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00020 [I 2022-10-09 05:42:01] ...trap/engines/engine.py.321: eval: epoch 33 | batch 480/510 [I 2022-10-09 05:42:01] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 05:42:01] ...trap/engines/engine.py.326: process: 0.31986 | load: 0.00020 [I 2022-10-09 05:42:05] ...trap/engines/engine.py.321: eval: epoch 33 | batch 490/510 [I 2022-10-09 05:42:05] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 05:42:05] ...trap/engines/engine.py.326: process: 0.32077 | load: 0.00020 [I 2022-10-09 05:42:08] ...trap/engines/engine.py.321: eval: epoch 33 | batch 500/510 [I 2022-10-09 05:42:08] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 05:42:08] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00021 [I 2022-10-09 05:42:11] ...trap/engines/engine.py.321: eval: epoch 33 | batch 510/510 [I 2022-10-09 05:42:11] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-09 05:42:11] ...trap/engines/engine.py.326: process: 0.31195 | load: 0.00016 [S 2022-10-09 05:42:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 33 [I 2022-10-09 05:42:12] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.704600 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.899700 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936700 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.696800 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897400 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930500 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013192 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005967 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005934 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012616 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006681 [S 2022-10-09 05:42:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004610 [I 2022-10-09 05:42:52] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 05:42:52] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 05:42:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 05:42:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 05:42:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 05:42:54] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 05:42:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 05:43:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 05:43:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 05:43:02] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 05:43:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 05:43:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 05:43:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 34 [I 2022-10-09 05:43:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 05:43:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 0/2304 [I 2022-10-09 05:43:14] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:07 [I 2022-10-09 05:43:14] ...trap/engines/engine.py.229: process: 3.07616 | load: 2.16167 [I 2022-10-09 05:43:14] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-09 05:43:14] ...optimizers/trijoint.py.151: total_norm: 13.398553 [I 2022-10-09 05:43:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 10/2304 [I 2022-10-09 05:43:23] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:19 [I 2022-10-09 05:43:23] ...trap/engines/engine.py.229: process: 0.82658 | load: 0.00565 [I 2022-10-09 05:43:23] ...trap/engines/engine.py.230: loss: 0.18090 [I 2022-10-09 05:43:23] ...optimizers/trijoint.py.151: total_norm: 29.509672 [I 2022-10-09 05:43:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 20/2304 [I 2022-10-09 05:43:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:44 [I 2022-10-09 05:43:31] ...trap/engines/engine.py.229: process: 0.82843 | load: 0.00384 [I 2022-10-09 05:43:31] ...trap/engines/engine.py.230: loss: 0.14874 [I 2022-10-09 05:43:31] ...optimizers/trijoint.py.151: total_norm: 7.503420 [I 2022-10-09 05:43:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 30/2304 [I 2022-10-09 05:43:39] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:51 [I 2022-10-09 05:43:39] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00361 [I 2022-10-09 05:43:39] ...trap/engines/engine.py.230: loss: 0.12464 [I 2022-10-09 05:43:39] ...optimizers/trijoint.py.151: total_norm: 9.117013 [I 2022-10-09 05:43:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 40/2304 [I 2022-10-09 05:43:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:39 [I 2022-10-09 05:43:48] ...trap/engines/engine.py.229: process: 0.82876 | load: 0.00438 [I 2022-10-09 05:43:48] ...trap/engines/engine.py.230: loss: 0.09757 [I 2022-10-09 05:43:48] ...optimizers/trijoint.py.151: total_norm: 9.949001 [I 2022-10-09 05:43:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 50/2304 [I 2022-10-09 05:43:56] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:26 [I 2022-10-09 05:43:56] ...trap/engines/engine.py.229: process: 0.84539 | load: 0.00399 [I 2022-10-09 05:43:56] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-09 05:43:56] ...optimizers/trijoint.py.151: total_norm: 10.606832 [I 2022-10-09 05:44:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 60/2304 [I 2022-10-09 05:44:05] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:29 [I 2022-10-09 05:44:05] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00354 [I 2022-10-09 05:44:05] ...trap/engines/engine.py.230: loss: 0.14385 [I 2022-10-09 05:44:05] ...optimizers/trijoint.py.151: total_norm: 10.172304 [I 2022-10-09 05:44:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 70/2304 [I 2022-10-09 05:44:13] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:16 [I 2022-10-09 05:44:13] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00370 [I 2022-10-09 05:44:13] ...trap/engines/engine.py.230: loss: 0.21642 [I 2022-10-09 05:44:13] ...optimizers/trijoint.py.151: total_norm: 19.812437 [I 2022-10-09 05:44:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 80/2304 [I 2022-10-09 05:44:21] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:05 [I 2022-10-09 05:44:21] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00326 [I 2022-10-09 05:44:21] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-09 05:44:21] ...optimizers/trijoint.py.151: total_norm: 15.079345 [I 2022-10-09 05:44:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 90/2304 [I 2022-10-09 05:44:30] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:57 [I 2022-10-09 05:44:30] ...trap/engines/engine.py.229: process: 0.82876 | load: 0.00337 [I 2022-10-09 05:44:30] ...trap/engines/engine.py.230: loss: 0.15091 [I 2022-10-09 05:44:30] ...optimizers/trijoint.py.151: total_norm: 25.884298 [I 2022-10-09 05:44:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 100/2304 [I 2022-10-09 05:44:38] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:45 [I 2022-10-09 05:44:38] ...trap/engines/engine.py.229: process: 0.84206 | load: 0.00319 [I 2022-10-09 05:44:38] ...trap/engines/engine.py.230: loss: 0.17925 [I 2022-10-09 05:44:38] ...optimizers/trijoint.py.151: total_norm: 13.573624 [I 2022-10-09 05:44:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 110/2304 [I 2022-10-09 05:44:47] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:37 [I 2022-10-09 05:44:47] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00321 [I 2022-10-09 05:44:47] ...trap/engines/engine.py.230: loss: 0.13333 [I 2022-10-09 05:44:47] ...optimizers/trijoint.py.151: total_norm: 9.880822 [I 2022-10-09 05:44:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 120/2304 [I 2022-10-09 05:44:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:45 [I 2022-10-09 05:44:55] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00316 [I 2022-10-09 05:44:55] ...trap/engines/engine.py.230: loss: 0.10554 [I 2022-10-09 05:44:55] ...optimizers/trijoint.py.151: total_norm: 10.715363 [I 2022-10-09 05:45:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 130/2304 [I 2022-10-09 05:45:03] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:22 [I 2022-10-09 05:45:03] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00311 [I 2022-10-09 05:45:03] ...trap/engines/engine.py.230: loss: 0.14983 [I 2022-10-09 05:45:03] ...optimizers/trijoint.py.151: total_norm: 16.989353 [I 2022-10-09 05:45:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 140/2304 [I 2022-10-09 05:45:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:30 [I 2022-10-09 05:45:12] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00336 [I 2022-10-09 05:45:12] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-09 05:45:12] ...optimizers/trijoint.py.151: total_norm: 10.517563 [I 2022-10-09 05:45:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 150/2304 [I 2022-10-09 05:45:20] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:07 [I 2022-10-09 05:45:20] ...trap/engines/engine.py.229: process: 0.84016 | load: 0.00336 [I 2022-10-09 05:45:20] ...trap/engines/engine.py.230: loss: 0.20643 [I 2022-10-09 05:45:20] ...optimizers/trijoint.py.151: total_norm: 16.989878 [I 2022-10-09 08:24:09] ...ytorch/bootstrap/run.py.73: Saving environment info [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: options: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dataset: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: aux_kw_path: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: batch_sampler: triplet_classif [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: batch_size: 100 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: database: lmdb [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: eval_split: val [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: freq_mismatch: 0.0 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: image_from: database [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: import: recipe1m.datasets.factory [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: kw_path: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: recipe1m [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_classes: 1048 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_threads: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: randkw_p: 0.3 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: randkw_p_aux: 0.5 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: tokenized_raw_text: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: train_split: train [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: use_vcs: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: engine: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: logger [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_epochs: 120 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: print_freq: 10 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max'] [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: exp: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: resume: last [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: misc: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cuda: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: data_parrallel: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: logs_name: None [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: overrite: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: seed: 1338 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: model: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: criterion: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: tri_cross_encoder: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: itm_loss_weight: 1 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: keep_background: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: retrieval_strategy: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: aggregation: valid [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: margin: 0.05 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: margin_params: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: increment: 0.005 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: increment_margin: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: max_margin: 0.3 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: triplet [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_samples: 9999 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: sampling: max_negative [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: substrategy: ['IRR', 'RII', 'SIRR', 'SRII'] [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: substrategy_weights: [1.0, 1.0, 0.1, 0.1] [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: import: recipe1m.models.factory [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: metric: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: k_test: 10 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_bags: 10 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: nb_matchs_per_bag: 1000 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: trijoint: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: network: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: activations: ['tanh', 'normalize'] [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: aux_kwords: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: aux_kwords_encoder: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: avg_concat: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cat_pos: 0 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: class_attention_fus: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cls_token_fus: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_decoder_image: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_decoder_recipe: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_attention: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_decoder: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: double_decoder: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: get_tokens: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: load_backbones: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_layers: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: query_img: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: cross_transformer: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dim_emb: 1024 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dim_image_out: 2048 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dim_ingr_out: 300 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dim_instr_in: 1024 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: dim_instr_out: 1024 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: embedding_names: ['proj_image'] [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: freeze_im: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: freeze_rec: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: get_tokens_cross_decoder_recipe: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: hidden_size: 512 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: image_backbone_name: clip [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: image_res: 224 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: kwords_same_level: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: last_cat_aux: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_heads_cross: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_heads_cross_image: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_heads_single: 4 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_layers: 2 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_layers_cross: 2 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_layers_cross_image: 1 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: n_layers_single: 2 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: num_hidden_layers_kw: 2 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: old: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: recipe_encoder: tfoodvicha-transformer [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: text_encoder: bert-base-uncased [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: vision_width: 768 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: vit_all_tokens: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: with_ingrs: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: with_instrs: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: with_titles: True [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: with_classif: False [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: optimizer: [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: batch_size_factor: None [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: clip_grad: 8.0 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: import: recipe1m.optimizers.factory [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: lr: 1e-05 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: lr_img: 1e-06 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: name: trijoint_fixed_fine_tune [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: switch_epoch: 20 [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml [S 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.75: view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm'] [I 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-a103', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64') [I 2022-10-09 08:24:16] ...ytorch/bootstrap/run.py.80: Available GPUs: [0, 1] [I 2022-10-09 08:24:16] ...strap/engines/factory.py.9: Creating engine... [I 2022-10-09 08:24:16] ...trap/datasets/factory.py.8: Creating dataset... [I 2022-10-09 08:24:30] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-09 08:24:34] ...m/datasets/recipe1m.py.165: randkw_p... 0.3 [I 2022-10-09 08:25:41] ...m/datasets/recipe1m.py.370: Calculate indices by class... [I 2022-10-09 08:25:44] ...m/datasets/recipe1m.py.375: Done! [I 2022-10-09 08:25:45] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-09 08:25:46] ...m/datasets/recipe1m.py.165: randkw_p... None [I 2022-10-09 08:26:59] ...rap/datasets/factory.py.18: Training will take place on trainset (238399 items) [I 2022-10-09 08:26:59] ...rap/datasets/factory.py.21: Evaluation will take place on valset (51119 items) [I 2022-10-09 08:26:59] ...strap/models/factory.py.21: Creating model... [I 2022-10-09 08:27:03] ...e_networks/networks.py.203: Loading CLIP Visual model... [I 2022-10-09 08:27:07] ...e_networks/networks.py.145: cuda [I 2022-10-09 08:27:08] ...networks/tfood_vicha.py.49: Load VCs encoder... [I 2022-10-09 08:27:10] .../criterions/trijoint.py.30: ignore_index=0 [I 2022-10-09 08:27:10] ...strap/models/factory.py.44: Enabling CUDA mode... [I 2022-10-09 08:27:12] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-09 08:27:12] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-09 08:27:12] ...otstrap/views/factory.py.8: Creating views... [I 2022-10-09 08:27:17] ...strap/engines/engine.py.98: Loading last checkpoint [I 2022-10-09 08:27:17] ...trap/engines/engine.py.440: Loading model... [I 2022-10-09 08:27:22] ...trap/engines/engine.py.446: Loading optimizer... [I 2022-10-09 08:27:34] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-09 08:27:34] ...trap/engines/engine.py.453: Loading engine... [I 2022-10-09 08:27:34] ...trap/engines/engine.py.125: Launching training procedures [I 2022-10-09 08:27:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 34 [I 2022-10-09 08:27:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 08:27:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 0/2304 [I 2022-10-09 08:27:41] ...trap/engines/engine.py.228: elapsed: 0:00:06 | left: 4:09:35 [I 2022-10-09 08:27:41] ...trap/engines/engine.py.229: process: 6.49997 | load: 1.92285 [I 2022-10-09 08:27:41] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-09 08:27:41] ...optimizers/trijoint.py.151: total_norm: 13.639100 [I 2022-10-09 08:27:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 10/2304 [I 2022-10-09 08:27:50] ...trap/engines/engine.py.228: elapsed: 0:00:15 | left: 0:56:02 [I 2022-10-09 08:27:50] ...trap/engines/engine.py.229: process: 0.82814 | load: 0.00540 [I 2022-10-09 08:27:50] ...trap/engines/engine.py.230: loss: 0.17996 [I 2022-10-09 08:27:50] ...optimizers/trijoint.py.151: total_norm: 29.813879 [I 2022-10-09 08:27:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 20/2304 [I 2022-10-09 08:27:58] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:34:20 [I 2022-10-09 08:27:58] ...trap/engines/engine.py.229: process: 0.84008 | load: 0.00363 [I 2022-10-09 08:27:58] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-09 08:27:58] ...optimizers/trijoint.py.151: total_norm: 7.574018 [I 2022-10-09 08:28:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 30/2304 [I 2022-10-09 08:28:06] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:31:56 [I 2022-10-09 08:28:06] ...trap/engines/engine.py.229: process: 0.83734 | load: 0.00351 [I 2022-10-09 08:28:06] ...trap/engines/engine.py.230: loss: 0.12164 [I 2022-10-09 08:28:06] ...optimizers/trijoint.py.151: total_norm: 9.015267 [I 2022-10-09 08:28:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 40/2304 [I 2022-10-09 08:28:15] ...trap/engines/engine.py.228: elapsed: 0:00:40 | left: 0:31:40 [I 2022-10-09 08:28:15] ...trap/engines/engine.py.229: process: 0.83073 | load: 0.00412 [I 2022-10-09 08:28:15] ...trap/engines/engine.py.230: loss: 0.09246 [I 2022-10-09 08:28:15] ...optimizers/trijoint.py.151: total_norm: 9.724434 [I 2022-10-09 08:28:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 50/2304 [I 2022-10-09 08:28:23] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:31:40 [I 2022-10-09 08:28:23] ...trap/engines/engine.py.229: process: 0.87515 | load: 0.00378 [I 2022-10-09 08:28:23] ...trap/engines/engine.py.230: loss: 0.11277 [I 2022-10-09 08:28:23] ...optimizers/trijoint.py.151: total_norm: 10.820257 [I 2022-10-09 08:28:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 60/2304 [I 2022-10-09 08:28:32] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:31:22 [I 2022-10-09 08:28:32] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00346 [I 2022-10-09 08:28:32] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-09 08:28:32] ...optimizers/trijoint.py.151: total_norm: 10.691145 [I 2022-10-09 08:28:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 70/2304 [I 2022-10-09 08:28:40] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:31:15 [I 2022-10-09 08:28:40] ...trap/engines/engine.py.229: process: 0.83839 | load: 0.00356 [I 2022-10-09 08:28:40] ...trap/engines/engine.py.230: loss: 0.21327 [I 2022-10-09 08:28:40] ...optimizers/trijoint.py.151: total_norm: 19.539196 [I 2022-10-09 08:28:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 80/2304 [I 2022-10-09 08:28:48] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:31:02 [I 2022-10-09 08:28:48] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00312 [I 2022-10-09 08:28:48] ...trap/engines/engine.py.230: loss: 0.16985 [I 2022-10-09 08:28:48] ...optimizers/trijoint.py.151: total_norm: 15.351596 [I 2022-10-09 08:28:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 90/2304 [I 2022-10-09 08:28:57] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:30:53 [I 2022-10-09 08:28:57] ...trap/engines/engine.py.229: process: 0.82964 | load: 0.00325 [I 2022-10-09 08:28:57] ...trap/engines/engine.py.230: loss: 0.15065 [I 2022-10-09 08:28:57] ...optimizers/trijoint.py.151: total_norm: 25.121574 [I 2022-10-09 08:29:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 100/2304 [I 2022-10-09 08:29:05] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:30:49 [I 2022-10-09 08:29:05] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00317 [I 2022-10-09 08:29:05] ...trap/engines/engine.py.230: loss: 0.17998 [I 2022-10-09 08:29:05] ...optimizers/trijoint.py.151: total_norm: 13.130210 [I 2022-10-09 08:29:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 110/2304 [I 2022-10-09 08:29:14] ...trap/engines/engine.py.228: elapsed: 0:01:39 | left: 0:30:32 [I 2022-10-09 08:29:14] ...trap/engines/engine.py.229: process: 0.83069 | load: 0.00306 [I 2022-10-09 08:29:14] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-09 08:29:14] ...optimizers/trijoint.py.151: total_norm: 9.579850 [I 2022-10-09 08:29:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 120/2304 [I 2022-10-09 08:29:22] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:30:45 [I 2022-10-09 08:29:22] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00303 [I 2022-10-09 08:29:22] ...trap/engines/engine.py.230: loss: 0.10657 [I 2022-10-09 08:29:22] ...optimizers/trijoint.py.151: total_norm: 9.632717 [I 2022-10-09 08:29:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 130/2304 [I 2022-10-09 08:29:30] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:30:22 [I 2022-10-09 08:29:30] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00308 [I 2022-10-09 08:29:30] ...trap/engines/engine.py.230: loss: 0.14933 [I 2022-10-09 08:29:30] ...optimizers/trijoint.py.151: total_norm: 16.446480 [I 2022-10-09 08:29:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 140/2304 [I 2022-10-09 08:29:39] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:30:32 [I 2022-10-09 08:29:39] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00324 [I 2022-10-09 08:29:39] ...trap/engines/engine.py.230: loss: 0.11931 [I 2022-10-09 08:29:39] ...optimizers/trijoint.py.151: total_norm: 10.418256 [I 2022-10-09 08:29:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 150/2304 [I 2022-10-09 08:29:47] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:04 [I 2022-10-09 08:29:47] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00319 [I 2022-10-09 08:29:47] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-09 08:29:47] ...optimizers/trijoint.py.151: total_norm: 17.548317 [I 2022-10-09 08:29:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 160/2304 [I 2022-10-09 08:29:56] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:02 [I 2022-10-09 08:29:56] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00310 [I 2022-10-09 08:29:56] ...trap/engines/engine.py.230: loss: 0.19201 [I 2022-10-09 08:29:56] ...optimizers/trijoint.py.151: total_norm: 16.190393 [I 2022-10-09 08:30:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 170/2304 [I 2022-10-09 08:30:04] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:29:49 [I 2022-10-09 08:30:04] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00327 [I 2022-10-09 08:30:04] ...trap/engines/engine.py.230: loss: 0.16878 [I 2022-10-09 08:30:04] ...optimizers/trijoint.py.151: total_norm: 17.896797 [I 2022-10-09 08:30:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 180/2304 [I 2022-10-09 08:30:13] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:29:51 [I 2022-10-09 08:30:13] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00310 [I 2022-10-09 08:30:13] ...trap/engines/engine.py.230: loss: 0.12204 [I 2022-10-09 08:30:13] ...optimizers/trijoint.py.151: total_norm: 27.670355 [I 2022-10-09 08:30:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 190/2304 [I 2022-10-09 08:30:21] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:29:29 [I 2022-10-09 08:30:21] ...trap/engines/engine.py.229: process: 0.83669 | load: 0.00293 [I 2022-10-09 08:30:21] ...trap/engines/engine.py.230: loss: 0.09493 [I 2022-10-09 08:30:21] ...optimizers/trijoint.py.151: total_norm: 11.824787 [I 2022-10-09 08:30:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 200/2304 [I 2022-10-09 08:30:29] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:23 [I 2022-10-09 08:30:29] ...trap/engines/engine.py.229: process: 0.83688 | load: 0.00314 [I 2022-10-09 08:30:29] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-09 08:30:29] ...optimizers/trijoint.py.151: total_norm: 8.966455 [I 2022-10-09 08:30:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 210/2304 [I 2022-10-09 08:30:38] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:12 [I 2022-10-09 08:30:38] ...trap/engines/engine.py.229: process: 0.82629 | load: 0.00307 [I 2022-10-09 08:30:38] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-09 08:30:38] ...optimizers/trijoint.py.151: total_norm: 12.144964 [I 2022-10-09 08:30:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 220/2304 [I 2022-10-09 08:30:46] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:16 [I 2022-10-09 08:30:46] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00335 [I 2022-10-09 08:30:46] ...trap/engines/engine.py.230: loss: 0.17200 [I 2022-10-09 08:30:46] ...optimizers/trijoint.py.151: total_norm: 29.747231 [I 2022-10-09 08:30:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 230/2304 [I 2022-10-09 08:30:55] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:28:59 [I 2022-10-09 08:30:55] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00300 [I 2022-10-09 08:30:55] ...trap/engines/engine.py.230: loss: 0.17733 [I 2022-10-09 08:30:55] ...optimizers/trijoint.py.151: total_norm: 14.749371 [I 2022-10-09 08:31:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 240/2304 [I 2022-10-09 08:31:03] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:28:46 [I 2022-10-09 08:31:03] ...trap/engines/engine.py.229: process: 0.82378 | load: 0.00301 [I 2022-10-09 08:31:03] ...trap/engines/engine.py.230: loss: 0.14401 [I 2022-10-09 08:31:03] ...optimizers/trijoint.py.151: total_norm: 15.704446 [I 2022-10-09 08:31:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 250/2304 [I 2022-10-09 08:31:11] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:28:50 [I 2022-10-09 08:31:11] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00289 [I 2022-10-09 08:31:11] ...trap/engines/engine.py.230: loss: 0.16814 [I 2022-10-09 08:31:11] ...optimizers/trijoint.py.151: total_norm: 24.189857 [I 2022-10-09 08:31:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 260/2304 [I 2022-10-09 08:31:20] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:39 [I 2022-10-09 08:31:20] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00291 [I 2022-10-09 08:31:20] ...trap/engines/engine.py.230: loss: 0.15436 [I 2022-10-09 08:31:20] ...optimizers/trijoint.py.151: total_norm: 27.050819 [I 2022-10-09 08:31:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 270/2304 [I 2022-10-09 08:31:28] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:24 [I 2022-10-09 08:31:28] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00290 [I 2022-10-09 08:31:28] ...trap/engines/engine.py.230: loss: 0.15528 [I 2022-10-09 08:31:28] ...optimizers/trijoint.py.151: total_norm: 20.953161 [I 2022-10-09 08:31:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 280/2304 [I 2022-10-09 08:31:37] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:14 [I 2022-10-09 08:31:37] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00311 [I 2022-10-09 08:31:37] ...trap/engines/engine.py.230: loss: 0.14958 [I 2022-10-09 08:31:37] ...optimizers/trijoint.py.151: total_norm: 18.947350 [I 2022-10-09 08:31:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 290/2304 [I 2022-10-09 08:31:45] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:06 [I 2022-10-09 08:31:45] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00306 [I 2022-10-09 08:31:45] ...trap/engines/engine.py.230: loss: 0.19558 [I 2022-10-09 08:31:45] ...optimizers/trijoint.py.151: total_norm: 14.076521 [I 2022-10-09 08:31:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 300/2304 [I 2022-10-09 08:31:53] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:05 [I 2022-10-09 08:31:53] ...trap/engines/engine.py.229: process: 0.85186 | load: 0.00299 [I 2022-10-09 08:31:53] ...trap/engines/engine.py.230: loss: 0.16863 [I 2022-10-09 08:31:53] ...optimizers/trijoint.py.151: total_norm: 30.383446 [I 2022-10-09 08:32:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 310/2304 [I 2022-10-09 08:32:02] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:03 [I 2022-10-09 08:32:02] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00303 [I 2022-10-09 08:32:02] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-09 08:32:02] ...optimizers/trijoint.py.151: total_norm: 27.549215 [I 2022-10-09 08:32:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 320/2304 [I 2022-10-09 08:32:10] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:27:46 [I 2022-10-09 08:32:10] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00288 [I 2022-10-09 08:32:10] ...trap/engines/engine.py.230: loss: 0.09150 [I 2022-10-09 08:32:10] ...optimizers/trijoint.py.151: total_norm: 13.351180 [I 2022-10-09 08:32:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 330/2304 [I 2022-10-09 08:32:19] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:37 [I 2022-10-09 08:32:19] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00304 [I 2022-10-09 08:32:19] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-09 08:32:19] ...optimizers/trijoint.py.151: total_norm: 12.098017 [I 2022-10-09 08:32:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 340/2304 [I 2022-10-09 08:32:27] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:26 [I 2022-10-09 08:32:27] ...trap/engines/engine.py.229: process: 0.83200 | load: 0.00292 [I 2022-10-09 08:32:27] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-09 08:32:27] ...optimizers/trijoint.py.151: total_norm: 20.387949 [I 2022-10-09 08:32:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 350/2304 [I 2022-10-09 08:32:35] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:16 [I 2022-10-09 08:32:35] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00288 [I 2022-10-09 08:32:35] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-09 08:32:35] ...optimizers/trijoint.py.151: total_norm: 19.237459 [I 2022-10-09 08:32:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 360/2304 [I 2022-10-09 08:32:44] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:12 [I 2022-10-09 08:32:44] ...trap/engines/engine.py.229: process: 0.84373 | load: 0.00289 [I 2022-10-09 08:32:44] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-09 08:32:44] ...optimizers/trijoint.py.151: total_norm: 11.142520 [I 2022-10-09 08:32:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 370/2304 [I 2022-10-09 08:32:52] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:09 [I 2022-10-09 08:32:52] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00289 [I 2022-10-09 08:32:52] ...trap/engines/engine.py.230: loss: 0.16622 [I 2022-10-09 08:32:52] ...optimizers/trijoint.py.151: total_norm: 14.274837 [I 2022-10-09 08:33:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 380/2304 [I 2022-10-09 08:33:01] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:26:49 [I 2022-10-09 08:33:01] ...trap/engines/engine.py.229: process: 0.83310 | load: 0.00293 [I 2022-10-09 08:33:01] ...trap/engines/engine.py.230: loss: 0.21901 [I 2022-10-09 08:33:01] ...optimizers/trijoint.py.151: total_norm: 22.955816 [I 2022-10-09 08:33:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 390/2304 [I 2022-10-09 08:33:09] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:26:58 [I 2022-10-09 08:33:09] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00293 [I 2022-10-09 08:33:09] ...trap/engines/engine.py.230: loss: 0.19356 [I 2022-10-09 08:33:09] ...optimizers/trijoint.py.151: total_norm: 18.030573 [I 2022-10-09 08:33:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 400/2304 [I 2022-10-09 08:33:17] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:34 [I 2022-10-09 08:33:17] ...trap/engines/engine.py.229: process: 0.83511 | load: 0.00299 [I 2022-10-09 08:33:17] ...trap/engines/engine.py.230: loss: 0.15670 [I 2022-10-09 08:33:17] ...optimizers/trijoint.py.151: total_norm: 21.310770 [I 2022-10-09 08:33:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 410/2304 [I 2022-10-09 08:33:26] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:33 [I 2022-10-09 08:33:26] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00293 [I 2022-10-09 08:33:26] ...trap/engines/engine.py.230: loss: 0.18424 [I 2022-10-09 08:33:26] ...optimizers/trijoint.py.151: total_norm: 19.235395 [I 2022-10-09 08:33:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 420/2304 [I 2022-10-09 08:33:34] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:23 [I 2022-10-09 08:33:34] ...trap/engines/engine.py.229: process: 0.84982 | load: 0.00290 [I 2022-10-09 08:33:34] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-09 08:33:34] ...optimizers/trijoint.py.151: total_norm: 9.067766 [I 2022-10-09 08:33:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 430/2304 [I 2022-10-09 08:33:43] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:18 [I 2022-10-09 08:33:43] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00287 [I 2022-10-09 08:33:43] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-09 08:33:43] ...optimizers/trijoint.py.151: total_norm: 23.311827 [I 2022-10-09 08:33:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 440/2304 [I 2022-10-09 08:33:51] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:09 [I 2022-10-09 08:33:51] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00290 [I 2022-10-09 08:33:51] ...trap/engines/engine.py.230: loss: 0.19817 [I 2022-10-09 08:33:51] ...optimizers/trijoint.py.151: total_norm: 18.803610 [I 2022-10-09 08:34:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 450/2304 [I 2022-10-09 08:34:00] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:01 [I 2022-10-09 08:34:00] ...trap/engines/engine.py.229: process: 0.84379 | load: 0.00293 [I 2022-10-09 08:34:00] ...trap/engines/engine.py.230: loss: 0.22632 [I 2022-10-09 08:34:00] ...optimizers/trijoint.py.151: total_norm: 12.924047 [I 2022-10-09 08:34:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 460/2304 [I 2022-10-09 08:34:08] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:25:51 [I 2022-10-09 08:34:08] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00295 [I 2022-10-09 08:34:08] ...trap/engines/engine.py.230: loss: 0.13648 [I 2022-10-09 08:34:08] ...optimizers/trijoint.py.151: total_norm: 15.887495 [I 2022-10-09 08:34:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 470/2304 [I 2022-10-09 08:34:16] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:25:35 [I 2022-10-09 08:34:16] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00289 [I 2022-10-09 08:34:16] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-09 08:34:16] ...optimizers/trijoint.py.151: total_norm: 14.831203 [I 2022-10-09 08:34:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 480/2304 [I 2022-10-09 08:34:25] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:34 [I 2022-10-09 08:34:25] ...trap/engines/engine.py.229: process: 0.82696 | load: 0.00291 [I 2022-10-09 08:34:25] ...trap/engines/engine.py.230: loss: 0.21510 [I 2022-10-09 08:34:25] ...optimizers/trijoint.py.151: total_norm: 10.855337 [I 2022-10-09 08:34:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 490/2304 [I 2022-10-09 08:34:33] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:25:41 [I 2022-10-09 08:34:33] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00292 [I 2022-10-09 08:34:33] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-09 08:34:33] ...optimizers/trijoint.py.151: total_norm: 9.622422 [I 2022-10-09 08:34:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 500/2304 [I 2022-10-09 08:34:42] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:27 [I 2022-10-09 08:34:42] ...trap/engines/engine.py.229: process: 0.82996 | load: 0.00290 [I 2022-10-09 08:34:42] ...trap/engines/engine.py.230: loss: 0.15858 [I 2022-10-09 08:34:42] ...optimizers/trijoint.py.151: total_norm: 11.449605 [I 2022-10-09 08:34:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 510/2304 [I 2022-10-09 08:34:50] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:25:21 [I 2022-10-09 08:34:50] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00290 [I 2022-10-09 08:34:50] ...trap/engines/engine.py.230: loss: 0.15675 [I 2022-10-09 08:34:50] ...optimizers/trijoint.py.151: total_norm: 13.720354 [I 2022-10-09 08:34:59] ...trap/engines/engine.py.224: train: epoch 34 | batch 520/2304 [I 2022-10-09 08:34:59] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:05 [I 2022-10-09 08:34:59] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00298 [I 2022-10-09 08:34:59] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-09 08:34:59] ...optimizers/trijoint.py.151: total_norm: 12.713747 [I 2022-10-09 08:35:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 530/2304 [I 2022-10-09 08:35:07] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:05 [I 2022-10-09 08:35:07] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00295 [I 2022-10-09 08:35:07] ...trap/engines/engine.py.230: loss: 0.17681 [I 2022-10-09 08:35:07] ...optimizers/trijoint.py.151: total_norm: 23.784855 [I 2022-10-09 08:35:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 540/2304 [I 2022-10-09 08:35:16] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:24:49 [I 2022-10-09 08:35:16] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00295 [I 2022-10-09 08:35:16] ...trap/engines/engine.py.230: loss: 0.10195 [I 2022-10-09 08:35:16] ...optimizers/trijoint.py.151: total_norm: 9.759804 [I 2022-10-09 08:35:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 550/2304 [I 2022-10-09 08:35:24] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:24:50 [I 2022-10-09 08:35:24] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00292 [I 2022-10-09 08:35:24] ...trap/engines/engine.py.230: loss: 0.18835 [I 2022-10-09 08:35:24] ...optimizers/trijoint.py.151: total_norm: 16.241110 [I 2022-10-09 08:35:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 560/2304 [I 2022-10-09 08:35:33] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:30 [I 2022-10-09 08:35:33] ...trap/engines/engine.py.229: process: 0.83501 | load: 0.00291 [I 2022-10-09 08:35:33] ...trap/engines/engine.py.230: loss: 0.19400 [I 2022-10-09 08:35:33] ...optimizers/trijoint.py.151: total_norm: 20.169233 [I 2022-10-09 08:35:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 570/2304 [I 2022-10-09 08:35:41] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:28 [I 2022-10-09 08:35:41] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00296 [I 2022-10-09 08:35:41] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-09 08:35:41] ...optimizers/trijoint.py.151: total_norm: 18.194191 [I 2022-10-09 08:35:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 580/2304 [I 2022-10-09 08:35:50] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:16 [I 2022-10-09 08:35:50] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00297 [I 2022-10-09 08:35:50] ...trap/engines/engine.py.230: loss: 0.18074 [I 2022-10-09 08:35:50] ...optimizers/trijoint.py.151: total_norm: 22.204855 [I 2022-10-09 08:35:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 590/2304 [I 2022-10-09 08:35:58] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:08 [I 2022-10-09 08:35:58] ...trap/engines/engine.py.229: process: 0.83919 | load: 0.00293 [I 2022-10-09 08:35:58] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-09 08:35:58] ...optimizers/trijoint.py.151: total_norm: 23.167267 [I 2022-10-09 08:36:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 600/2304 [I 2022-10-09 08:36:06] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:23:56 [I 2022-10-09 08:36:06] ...trap/engines/engine.py.229: process: 0.83368 | load: 0.00291 [I 2022-10-09 08:36:06] ...trap/engines/engine.py.230: loss: 0.12606 [I 2022-10-09 08:36:06] ...optimizers/trijoint.py.151: total_norm: 22.527674 [I 2022-10-09 08:36:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 610/2304 [I 2022-10-09 08:36:15] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:23:44 [I 2022-10-09 08:36:15] ...trap/engines/engine.py.229: process: 0.83521 | load: 0.00293 [I 2022-10-09 08:36:15] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-09 08:36:15] ...optimizers/trijoint.py.151: total_norm: 10.044206 [I 2022-10-09 08:36:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 620/2304 [I 2022-10-09 08:36:23] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:23:39 [I 2022-10-09 08:36:23] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00304 [I 2022-10-09 08:36:23] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-09 08:36:23] ...optimizers/trijoint.py.151: total_norm: 26.588865 [I 2022-10-09 08:36:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 630/2304 [I 2022-10-09 08:36:32] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:30 [I 2022-10-09 08:36:32] ...trap/engines/engine.py.229: process: 0.82757 | load: 0.00294 [I 2022-10-09 08:36:32] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-09 08:36:32] ...optimizers/trijoint.py.151: total_norm: 16.041210 [I 2022-10-09 08:36:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 640/2304 [I 2022-10-09 08:36:40] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:28 [I 2022-10-09 08:36:40] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00289 [I 2022-10-09 08:36:40] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-09 08:36:40] ...optimizers/trijoint.py.151: total_norm: 21.431339 [I 2022-10-09 08:36:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 650/2304 [I 2022-10-09 08:36:49] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:27 [I 2022-10-09 08:36:49] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00288 [I 2022-10-09 08:36:49] ...trap/engines/engine.py.230: loss: 0.12476 [I 2022-10-09 08:36:49] ...optimizers/trijoint.py.151: total_norm: 12.279945 [I 2022-10-09 08:36:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 660/2304 [I 2022-10-09 08:36:57] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:14 [I 2022-10-09 08:36:57] ...trap/engines/engine.py.229: process: 0.84887 | load: 0.00291 [I 2022-10-09 08:36:57] ...trap/engines/engine.py.230: loss: 0.19510 [I 2022-10-09 08:36:57] ...optimizers/trijoint.py.151: total_norm: 25.429434 [I 2022-10-09 08:37:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 670/2304 [I 2022-10-09 08:37:06] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:06 [I 2022-10-09 08:37:06] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00291 [I 2022-10-09 08:37:06] ...trap/engines/engine.py.230: loss: 0.18006 [I 2022-10-09 08:37:06] ...optimizers/trijoint.py.151: total_norm: 12.553821 [I 2022-10-09 08:37:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 680/2304 [I 2022-10-09 08:37:14] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:22:55 [I 2022-10-09 08:37:14] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00289 [I 2022-10-09 08:37:14] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-09 08:37:14] ...optimizers/trijoint.py.151: total_norm: 15.737152 [I 2022-10-09 08:37:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 690/2304 [I 2022-10-09 08:37:23] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:22:38 [I 2022-10-09 08:37:23] ...trap/engines/engine.py.229: process: 0.83132 | load: 0.00291 [I 2022-10-09 08:37:23] ...trap/engines/engine.py.230: loss: 0.17178 [I 2022-10-09 08:37:23] ...optimizers/trijoint.py.151: total_norm: 33.453976 [I 2022-10-09 08:37:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 700/2304 [I 2022-10-09 08:37:31] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:22:35 [I 2022-10-09 08:37:31] ...trap/engines/engine.py.229: process: 0.82583 | load: 0.00290 [I 2022-10-09 08:37:31] ...trap/engines/engine.py.230: loss: 0.18177 [I 2022-10-09 08:37:31] ...optimizers/trijoint.py.151: total_norm: 21.230770 [I 2022-10-09 08:37:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 710/2304 [I 2022-10-09 08:37:40] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:22 [I 2022-10-09 08:37:40] ...trap/engines/engine.py.229: process: 0.83416 | load: 0.00292 [I 2022-10-09 08:37:40] ...trap/engines/engine.py.230: loss: 0.20860 [I 2022-10-09 08:37:40] ...optimizers/trijoint.py.151: total_norm: 27.724508 [I 2022-10-09 08:37:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 720/2304 [I 2022-10-09 08:37:48] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:22:17 [I 2022-10-09 08:37:48] ...trap/engines/engine.py.229: process: 0.85657 | load: 0.00294 [I 2022-10-09 08:37:48] ...trap/engines/engine.py.230: loss: 0.20277 [I 2022-10-09 08:37:48] ...optimizers/trijoint.py.151: total_norm: 19.280792 [I 2022-10-09 08:37:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 730/2304 [I 2022-10-09 08:37:57] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:05 [I 2022-10-09 08:37:57] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00294 [I 2022-10-09 08:37:57] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-09 08:37:57] ...optimizers/trijoint.py.151: total_norm: 12.391809 [I 2022-10-09 08:38:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 740/2304 [I 2022-10-09 08:38:05] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:21:56 [I 2022-10-09 08:38:05] ...trap/engines/engine.py.229: process: 0.83412 | load: 0.00288 [I 2022-10-09 08:38:05] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-09 08:38:05] ...optimizers/trijoint.py.151: total_norm: 30.574226 [I 2022-10-09 08:38:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 750/2304 [I 2022-10-09 08:38:13] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:21:51 [I 2022-10-09 08:38:13] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00292 [I 2022-10-09 08:38:13] ...trap/engines/engine.py.230: loss: 0.22596 [I 2022-10-09 08:38:13] ...optimizers/trijoint.py.151: total_norm: 17.466270 [I 2022-10-09 08:38:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 760/2304 [I 2022-10-09 08:38:22] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:21:29 [I 2022-10-09 08:38:22] ...trap/engines/engine.py.229: process: 0.83171 | load: 0.00291 [I 2022-10-09 08:38:22] ...trap/engines/engine.py.230: loss: 0.18532 [I 2022-10-09 08:38:22] ...optimizers/trijoint.py.151: total_norm: 23.719007 [I 2022-10-09 08:38:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 770/2304 [I 2022-10-09 08:38:30] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:21:30 [I 2022-10-09 08:38:30] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00295 [I 2022-10-09 08:38:30] ...trap/engines/engine.py.230: loss: 0.19801 [I 2022-10-09 08:38:30] ...optimizers/trijoint.py.151: total_norm: 31.350439 [I 2022-10-09 08:38:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 780/2304 [I 2022-10-09 08:38:39] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:21:29 [I 2022-10-09 08:38:39] ...trap/engines/engine.py.229: process: 0.85771 | load: 0.00301 [I 2022-10-09 08:38:39] ...trap/engines/engine.py.230: loss: 0.19640 [I 2022-10-09 08:38:39] ...optimizers/trijoint.py.151: total_norm: 26.644695 [I 2022-10-09 08:38:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 790/2304 [I 2022-10-09 08:38:47] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:21:09 [I 2022-10-09 08:38:47] ...trap/engines/engine.py.229: process: 0.82801 | load: 0.00289 [I 2022-10-09 08:38:47] ...trap/engines/engine.py.230: loss: 0.18043 [I 2022-10-09 08:38:47] ...optimizers/trijoint.py.151: total_norm: 12.383980 [I 2022-10-09 08:38:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 800/2304 [I 2022-10-09 08:38:56] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:06 [I 2022-10-09 08:38:56] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00292 [I 2022-10-09 08:38:56] ...trap/engines/engine.py.230: loss: 0.16766 [I 2022-10-09 08:38:56] ...optimizers/trijoint.py.151: total_norm: 28.723627 [I 2022-10-09 08:39:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 810/2304 [I 2022-10-09 08:39:04] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:20:55 [I 2022-10-09 08:39:04] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00290 [I 2022-10-09 08:39:04] ...trap/engines/engine.py.230: loss: 0.11056 [I 2022-10-09 08:39:04] ...optimizers/trijoint.py.151: total_norm: 11.605463 [I 2022-10-09 08:39:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 820/2304 [I 2022-10-09 08:39:12] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:20:50 [I 2022-10-09 08:39:12] ...trap/engines/engine.py.229: process: 0.83672 | load: 0.00292 [I 2022-10-09 08:39:12] ...trap/engines/engine.py.230: loss: 0.17354 [I 2022-10-09 08:39:12] ...optimizers/trijoint.py.151: total_norm: 15.640951 [I 2022-10-09 08:39:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 830/2304 [I 2022-10-09 08:39:21] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:20:42 [I 2022-10-09 08:39:21] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00296 [I 2022-10-09 08:39:21] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-09 08:39:21] ...optimizers/trijoint.py.151: total_norm: 17.812550 [I 2022-10-09 08:39:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 840/2304 [I 2022-10-09 08:39:29] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:20:35 [I 2022-10-09 08:39:29] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00296 [I 2022-10-09 08:39:29] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-09 08:39:29] ...optimizers/trijoint.py.151: total_norm: 13.497103 [I 2022-10-09 08:39:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 850/2304 [I 2022-10-09 08:39:38] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:20:25 [I 2022-10-09 08:39:38] ...trap/engines/engine.py.229: process: 0.82726 | load: 0.00287 [I 2022-10-09 08:39:38] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-09 08:39:38] ...optimizers/trijoint.py.151: total_norm: 19.795023 [I 2022-10-09 08:39:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 860/2304 [I 2022-10-09 08:39:46] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:20:12 [I 2022-10-09 08:39:46] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00288 [I 2022-10-09 08:39:46] ...trap/engines/engine.py.230: loss: 0.17805 [I 2022-10-09 08:39:46] ...optimizers/trijoint.py.151: total_norm: 19.554567 [I 2022-10-09 08:39:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 870/2304 [I 2022-10-09 08:39:55] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:20:09 [I 2022-10-09 08:39:55] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00291 [I 2022-10-09 08:39:55] ...trap/engines/engine.py.230: loss: 0.20080 [I 2022-10-09 08:39:55] ...optimizers/trijoint.py.151: total_norm: 25.794426 [I 2022-10-09 08:40:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 880/2304 [I 2022-10-09 08:40:03] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:20:01 [I 2022-10-09 08:40:03] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00293 [I 2022-10-09 08:40:03] ...trap/engines/engine.py.230: loss: 0.21432 [I 2022-10-09 08:40:03] ...optimizers/trijoint.py.151: total_norm: 38.366226 [I 2022-10-09 08:40:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 890/2304 [I 2022-10-09 08:40:11] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:19:44 [I 2022-10-09 08:40:11] ...trap/engines/engine.py.229: process: 0.83886 | load: 0.00289 [I 2022-10-09 08:40:11] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-09 08:40:11] ...optimizers/trijoint.py.151: total_norm: 13.271081 [I 2022-10-09 08:40:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 900/2304 [I 2022-10-09 08:40:20] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:19:33 [I 2022-10-09 08:40:20] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00290 [I 2022-10-09 08:40:20] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-09 08:40:20] ...optimizers/trijoint.py.151: total_norm: 21.655540 [I 2022-10-09 08:40:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 910/2304 [I 2022-10-09 08:40:28] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:27 [I 2022-10-09 08:40:28] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00292 [I 2022-10-09 08:40:28] ...trap/engines/engine.py.230: loss: 0.10530 [I 2022-10-09 08:40:28] ...optimizers/trijoint.py.151: total_norm: 31.138371 [I 2022-10-09 08:40:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 920/2304 [I 2022-10-09 08:40:37] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:19:21 [I 2022-10-09 08:40:37] ...trap/engines/engine.py.229: process: 0.83102 | load: 0.00294 [I 2022-10-09 08:40:37] ...trap/engines/engine.py.230: loss: 0.16015 [I 2022-10-09 08:40:37] ...optimizers/trijoint.py.151: total_norm: 15.791974 [I 2022-10-09 08:40:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 930/2304 [I 2022-10-09 08:40:45] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:12 [I 2022-10-09 08:40:45] ...trap/engines/engine.py.229: process: 0.82980 | load: 0.00288 [I 2022-10-09 08:40:45] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-09 08:40:45] ...optimizers/trijoint.py.151: total_norm: 12.030123 [I 2022-10-09 08:40:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 940/2304 [I 2022-10-09 08:40:53] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:19:05 [I 2022-10-09 08:40:53] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00290 [I 2022-10-09 08:40:53] ...trap/engines/engine.py.230: loss: 0.18144 [I 2022-10-09 08:40:53] ...optimizers/trijoint.py.151: total_norm: 16.241034 [I 2022-10-09 08:41:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 950/2304 [I 2022-10-09 08:41:02] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:18:59 [I 2022-10-09 08:41:02] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00296 [I 2022-10-09 08:41:02] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-09 08:41:02] ...optimizers/trijoint.py.151: total_norm: 18.607731 [I 2022-10-09 08:41:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 960/2304 [I 2022-10-09 08:41:10] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:18:51 [I 2022-10-09 08:41:10] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00287 [I 2022-10-09 08:41:10] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-09 08:41:10] ...optimizers/trijoint.py.151: total_norm: 14.700233 [I 2022-10-09 08:41:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 970/2304 [I 2022-10-09 08:41:19] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:18:41 [I 2022-10-09 08:41:19] ...trap/engines/engine.py.229: process: 0.83689 | load: 0.00290 [I 2022-10-09 08:41:19] ...trap/engines/engine.py.230: loss: 0.24295 [I 2022-10-09 08:41:19] ...optimizers/trijoint.py.151: total_norm: 28.203636 [I 2022-10-09 08:41:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 980/2304 [I 2022-10-09 08:41:27] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:29 [I 2022-10-09 08:41:27] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00292 [I 2022-10-09 08:41:27] ...trap/engines/engine.py.230: loss: 0.21358 [I 2022-10-09 08:41:27] ...optimizers/trijoint.py.151: total_norm: 32.200829 [I 2022-10-09 08:41:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 990/2304 [I 2022-10-09 08:41:35] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:18:22 [I 2022-10-09 08:41:35] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00294 [I 2022-10-09 08:41:35] ...trap/engines/engine.py.230: loss: 0.14313 [I 2022-10-09 08:41:35] ...optimizers/trijoint.py.151: total_norm: 10.066851 [I 2022-10-09 08:41:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 1000/2304 [I 2022-10-09 08:41:44] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:17 [I 2022-10-09 08:41:44] ...trap/engines/engine.py.229: process: 0.84457 | load: 0.00289 [I 2022-10-09 08:41:44] ...trap/engines/engine.py.230: loss: 0.31554 [I 2022-10-09 08:41:44] ...optimizers/trijoint.py.151: total_norm: 42.665974 [I 2022-10-09 08:41:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 1010/2304 [I 2022-10-09 08:41:52] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:13 [I 2022-10-09 08:41:52] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00291 [I 2022-10-09 08:41:52] ...trap/engines/engine.py.230: loss: 0.14795 [I 2022-10-09 08:41:52] ...optimizers/trijoint.py.151: total_norm: 12.583539 [I 2022-10-09 08:42:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 1020/2304 [I 2022-10-09 08:42:01] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:17:56 [I 2022-10-09 08:42:01] ...trap/engines/engine.py.229: process: 0.83615 | load: 0.00291 [I 2022-10-09 08:42:01] ...trap/engines/engine.py.230: loss: 0.14513 [I 2022-10-09 08:42:01] ...optimizers/trijoint.py.151: total_norm: 18.250387 [I 2022-10-09 08:42:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 1030/2304 [I 2022-10-09 08:42:09] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:17:51 [I 2022-10-09 08:42:09] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00294 [I 2022-10-09 08:42:09] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-09 08:42:09] ...optimizers/trijoint.py.151: total_norm: 12.458175 [I 2022-10-09 08:42:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 1040/2304 [I 2022-10-09 08:42:18] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:44 [I 2022-10-09 08:42:18] ...trap/engines/engine.py.229: process: 0.83669 | load: 0.00289 [I 2022-10-09 08:42:18] ...trap/engines/engine.py.230: loss: 0.18556 [I 2022-10-09 08:42:18] ...optimizers/trijoint.py.151: total_norm: 15.747637 [I 2022-10-09 08:42:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 1050/2304 [I 2022-10-09 08:42:26] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:40 [I 2022-10-09 08:42:26] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00290 [I 2022-10-09 08:42:26] ...trap/engines/engine.py.230: loss: 0.19708 [I 2022-10-09 08:42:26] ...optimizers/trijoint.py.151: total_norm: 13.780542 [I 2022-10-09 08:42:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 1060/2304 [I 2022-10-09 08:42:35] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:28 [I 2022-10-09 08:42:35] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00295 [I 2022-10-09 08:42:35] ...trap/engines/engine.py.230: loss: 0.14478 [I 2022-10-09 08:42:35] ...optimizers/trijoint.py.151: total_norm: 23.451326 [I 2022-10-09 08:42:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 1070/2304 [I 2022-10-09 08:42:43] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:17 [I 2022-10-09 08:42:43] ...trap/engines/engine.py.229: process: 0.84001 | load: 0.00293 [I 2022-10-09 08:42:43] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-09 08:42:43] ...optimizers/trijoint.py.151: total_norm: 12.172618 [I 2022-10-09 08:42:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 1080/2304 [I 2022-10-09 08:42:52] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:18:29 [I 2022-10-09 08:42:52] ...trap/engines/engine.py.229: process: 1.18209 | load: 0.00289 [I 2022-10-09 08:42:52] ...trap/engines/engine.py.230: loss: 0.19261 [I 2022-10-09 08:42:52] ...optimizers/trijoint.py.151: total_norm: 11.836759 [I 2022-10-09 08:43:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 1090/2304 [I 2022-10-09 08:43:00] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:09 [I 2022-10-09 08:43:00] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00289 [I 2022-10-09 08:43:00] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-09 08:43:00] ...optimizers/trijoint.py.151: total_norm: 24.704372 [I 2022-10-09 08:43:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 1100/2304 [I 2022-10-09 08:43:09] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:16:55 [I 2022-10-09 08:43:09] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00289 [I 2022-10-09 08:43:09] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-09 08:43:09] ...optimizers/trijoint.py.151: total_norm: 19.095869 [I 2022-10-09 08:43:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 1110/2304 [I 2022-10-09 08:43:17] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:44 [I 2022-10-09 08:43:17] ...trap/engines/engine.py.229: process: 0.83173 | load: 0.00290 [I 2022-10-09 08:43:17] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-09 08:43:17] ...optimizers/trijoint.py.151: total_norm: 17.271448 [I 2022-10-09 08:43:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 1120/2304 [I 2022-10-09 08:43:25] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:36 [I 2022-10-09 08:43:25] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00291 [I 2022-10-09 08:43:25] ...trap/engines/engine.py.230: loss: 0.18246 [I 2022-10-09 08:43:25] ...optimizers/trijoint.py.151: total_norm: 23.759155 [I 2022-10-09 08:43:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 1130/2304 [I 2022-10-09 08:43:34] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:28 [I 2022-10-09 08:43:34] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00298 [I 2022-10-09 08:43:34] ...trap/engines/engine.py.230: loss: 0.16085 [I 2022-10-09 08:43:34] ...optimizers/trijoint.py.151: total_norm: 19.975792 [I 2022-10-09 08:43:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 1140/2304 [I 2022-10-09 08:43:42] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:19 [I 2022-10-09 08:43:42] ...trap/engines/engine.py.229: process: 0.83802 | load: 0.00291 [I 2022-10-09 08:43:42] ...trap/engines/engine.py.230: loss: 0.13234 [I 2022-10-09 08:43:42] ...optimizers/trijoint.py.151: total_norm: 22.204767 [I 2022-10-09 08:43:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 1150/2304 [I 2022-10-09 08:43:51] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:11 [I 2022-10-09 08:43:51] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00289 [I 2022-10-09 08:43:51] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-09 08:43:51] ...optimizers/trijoint.py.151: total_norm: 35.625572 [I 2022-10-09 08:43:59] ...trap/engines/engine.py.224: train: epoch 34 | batch 1160/2304 [I 2022-10-09 08:43:59] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:02 [I 2022-10-09 08:43:59] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00290 [I 2022-10-09 08:43:59] ...trap/engines/engine.py.230: loss: 0.22270 [I 2022-10-09 08:43:59] ...optimizers/trijoint.py.151: total_norm: 21.029442 [I 2022-10-09 08:44:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 1170/2304 [I 2022-10-09 08:44:08] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:15:57 [I 2022-10-09 08:44:08] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00289 [I 2022-10-09 08:44:08] ...trap/engines/engine.py.230: loss: 0.17695 [I 2022-10-09 08:44:08] ...optimizers/trijoint.py.151: total_norm: 19.839365 [I 2022-10-09 08:44:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 1180/2304 [I 2022-10-09 08:44:16] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:40 [I 2022-10-09 08:44:16] ...trap/engines/engine.py.229: process: 0.82869 | load: 0.00291 [I 2022-10-09 08:44:16] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-09 08:44:16] ...optimizers/trijoint.py.151: total_norm: 15.817905 [I 2022-10-09 08:44:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 1190/2304 [I 2022-10-09 08:44:24] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:39 [I 2022-10-09 08:44:24] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00296 [I 2022-10-09 08:44:24] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-09 08:44:24] ...optimizers/trijoint.py.151: total_norm: 9.877934 [I 2022-10-09 08:44:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 1200/2304 [I 2022-10-09 08:44:33] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:28 [I 2022-10-09 08:44:33] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00292 [I 2022-10-09 08:44:33] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-09 08:44:33] ...optimizers/trijoint.py.151: total_norm: 14.939014 [I 2022-10-09 08:44:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 1210/2304 [I 2022-10-09 08:44:41] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:21 [I 2022-10-09 08:44:41] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00289 [I 2022-10-09 08:44:41] ...trap/engines/engine.py.230: loss: 0.19678 [I 2022-10-09 08:44:41] ...optimizers/trijoint.py.151: total_norm: 19.713013 [I 2022-10-09 08:44:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 1220/2304 [I 2022-10-09 08:44:50] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:11 [I 2022-10-09 08:44:50] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00290 [I 2022-10-09 08:44:50] ...trap/engines/engine.py.230: loss: 0.18960 [I 2022-10-09 08:44:50] ...optimizers/trijoint.py.151: total_norm: 25.382027 [I 2022-10-09 08:44:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 1230/2304 [I 2022-10-09 08:44:58] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:14:57 [I 2022-10-09 08:44:58] ...trap/engines/engine.py.229: process: 0.83836 | load: 0.00291 [I 2022-10-09 08:44:58] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-09 08:44:58] ...optimizers/trijoint.py.151: total_norm: 22.856703 [I 2022-10-09 08:45:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 1240/2304 [I 2022-10-09 08:45:06] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:53 [I 2022-10-09 08:45:06] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00302 [I 2022-10-09 08:45:06] ...trap/engines/engine.py.230: loss: 0.15819 [I 2022-10-09 08:45:06] ...optimizers/trijoint.py.151: total_norm: 25.055199 [I 2022-10-09 08:45:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 1250/2304 [I 2022-10-09 08:45:15] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:40 [I 2022-10-09 08:45:15] ...trap/engines/engine.py.229: process: 0.82315 | load: 0.00290 [I 2022-10-09 08:45:15] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-09 08:45:15] ...optimizers/trijoint.py.151: total_norm: 12.428243 [I 2022-10-09 08:45:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 1260/2304 [I 2022-10-09 08:45:23] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:14:36 [I 2022-10-09 08:45:23] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00289 [I 2022-10-09 08:45:23] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-09 08:45:23] ...optimizers/trijoint.py.151: total_norm: 35.525040 [I 2022-10-09 08:45:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 1270/2304 [I 2022-10-09 08:45:32] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:27 [I 2022-10-09 08:45:32] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00289 [I 2022-10-09 08:45:32] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-09 08:45:32] ...optimizers/trijoint.py.151: total_norm: 17.001154 [I 2022-10-09 08:45:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 1280/2304 [I 2022-10-09 08:45:40] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:23 [I 2022-10-09 08:45:40] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00291 [I 2022-10-09 08:45:40] ...trap/engines/engine.py.230: loss: 0.15319 [I 2022-10-09 08:45:40] ...optimizers/trijoint.py.151: total_norm: 16.809591 [I 2022-10-09 08:45:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 1290/2304 [I 2022-10-09 08:45:49] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:09 [I 2022-10-09 08:45:49] ...trap/engines/engine.py.229: process: 0.83790 | load: 0.00294 [I 2022-10-09 08:45:49] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-09 08:45:49] ...optimizers/trijoint.py.151: total_norm: 17.390633 [I 2022-10-09 08:45:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 1300/2304 [I 2022-10-09 08:45:57] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:04 [I 2022-10-09 08:45:57] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00293 [I 2022-10-09 08:45:57] ...trap/engines/engine.py.230: loss: 0.20565 [I 2022-10-09 08:45:57] ...optimizers/trijoint.py.151: total_norm: 17.604563 [I 2022-10-09 08:46:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 1310/2304 [I 2022-10-09 08:46:05] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:13:57 [I 2022-10-09 08:46:05] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00290 [I 2022-10-09 08:46:05] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-09 08:46:05] ...optimizers/trijoint.py.151: total_norm: 13.381837 [I 2022-10-09 08:46:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 1320/2304 [I 2022-10-09 08:46:14] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:51 [I 2022-10-09 08:46:14] ...trap/engines/engine.py.229: process: 0.85005 | load: 0.00290 [I 2022-10-09 08:46:14] ...trap/engines/engine.py.230: loss: 0.09258 [I 2022-10-09 08:46:14] ...optimizers/trijoint.py.151: total_norm: 18.996872 [I 2022-10-09 08:46:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 1330/2304 [I 2022-10-09 08:46:22] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:40 [I 2022-10-09 08:46:22] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00290 [I 2022-10-09 08:46:22] ...trap/engines/engine.py.230: loss: 0.14387 [I 2022-10-09 08:46:22] ...optimizers/trijoint.py.151: total_norm: 20.543819 [I 2022-10-09 08:46:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 1340/2304 [I 2022-10-09 08:46:31] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:33 [I 2022-10-09 08:46:31] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00290 [I 2022-10-09 08:46:31] ...trap/engines/engine.py.230: loss: 0.18419 [I 2022-10-09 08:46:31] ...optimizers/trijoint.py.151: total_norm: 19.020676 [I 2022-10-09 08:46:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 1350/2304 [I 2022-10-09 08:46:39] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:25 [I 2022-10-09 08:46:39] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00297 [I 2022-10-09 08:46:39] ...trap/engines/engine.py.230: loss: 0.15062 [I 2022-10-09 08:46:39] ...optimizers/trijoint.py.151: total_norm: 22.490688 [I 2022-10-09 08:46:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 1360/2304 [I 2022-10-09 08:46:48] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:17 [I 2022-10-09 08:46:48] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00293 [I 2022-10-09 08:46:48] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-09 08:46:48] ...optimizers/trijoint.py.151: total_norm: 9.322556 [I 2022-10-09 08:46:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 1370/2304 [I 2022-10-09 08:46:56] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:05 [I 2022-10-09 08:46:56] ...trap/engines/engine.py.229: process: 0.82753 | load: 0.00286 [I 2022-10-09 08:46:56] ...trap/engines/engine.py.230: loss: 0.22335 [I 2022-10-09 08:46:56] ...optimizers/trijoint.py.151: total_norm: 18.546505 [I 2022-10-09 08:47:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 1380/2304 [I 2022-10-09 08:47:04] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:12:57 [I 2022-10-09 08:47:04] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00292 [I 2022-10-09 08:47:04] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-09 08:47:04] ...optimizers/trijoint.py.151: total_norm: 11.012950 [I 2022-10-09 08:47:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 1390/2304 [I 2022-10-09 08:47:13] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:48 [I 2022-10-09 08:47:13] ...trap/engines/engine.py.229: process: 0.82838 | load: 0.00291 [I 2022-10-09 08:47:13] ...trap/engines/engine.py.230: loss: 0.17435 [I 2022-10-09 08:47:13] ...optimizers/trijoint.py.151: total_norm: 19.357382 [I 2022-10-09 08:47:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 1400/2304 [I 2022-10-09 08:47:21] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:41 [I 2022-10-09 08:47:21] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00290 [I 2022-10-09 08:47:21] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-09 08:47:21] ...optimizers/trijoint.py.151: total_norm: 10.909736 [I 2022-10-09 08:47:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 1410/2304 [I 2022-10-09 08:47:30] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:30 [I 2022-10-09 08:47:30] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00297 [I 2022-10-09 08:47:30] ...trap/engines/engine.py.230: loss: 0.18870 [I 2022-10-09 08:47:30] ...optimizers/trijoint.py.151: total_norm: 22.090467 [I 2022-10-09 08:47:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 1420/2304 [I 2022-10-09 08:47:38] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:23 [I 2022-10-09 08:47:38] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00294 [I 2022-10-09 08:47:38] ...trap/engines/engine.py.230: loss: 0.12308 [I 2022-10-09 08:47:38] ...optimizers/trijoint.py.151: total_norm: 14.943557 [I 2022-10-09 08:47:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 1430/2304 [I 2022-10-09 08:47:47] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:15 [I 2022-10-09 08:47:47] ...trap/engines/engine.py.229: process: 0.83922 | load: 0.00286 [I 2022-10-09 08:47:47] ...trap/engines/engine.py.230: loss: 0.18053 [I 2022-10-09 08:47:47] ...optimizers/trijoint.py.151: total_norm: 24.560316 [I 2022-10-09 08:47:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 1440/2304 [I 2022-10-09 08:47:55] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:04 [I 2022-10-09 08:47:55] ...trap/engines/engine.py.229: process: 0.84604 | load: 0.00290 [I 2022-10-09 08:47:55] ...trap/engines/engine.py.230: loss: 0.15768 [I 2022-10-09 08:47:55] ...optimizers/trijoint.py.151: total_norm: 12.680588 [I 2022-10-09 08:48:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 1450/2304 [I 2022-10-09 08:48:03] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:11:57 [I 2022-10-09 08:48:03] ...trap/engines/engine.py.229: process: 0.84264 | load: 0.00291 [I 2022-10-09 08:48:03] ...trap/engines/engine.py.230: loss: 0.10656 [I 2022-10-09 08:48:03] ...optimizers/trijoint.py.151: total_norm: 16.463591 [I 2022-10-09 08:48:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 1460/2304 [I 2022-10-09 08:48:12] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:11:50 [I 2022-10-09 08:48:12] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00292 [I 2022-10-09 08:48:12] ...trap/engines/engine.py.230: loss: 0.18446 [I 2022-10-09 08:48:12] ...optimizers/trijoint.py.151: total_norm: 27.468058 [I 2022-10-09 08:48:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 1470/2304 [I 2022-10-09 08:48:20] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:43 [I 2022-10-09 08:48:20] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00298 [I 2022-10-09 08:48:20] ...trap/engines/engine.py.230: loss: 0.21912 [I 2022-10-09 08:48:20] ...optimizers/trijoint.py.151: total_norm: 19.035576 [I 2022-10-09 08:48:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 1480/2304 [I 2022-10-09 08:48:29] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:28 [I 2022-10-09 08:48:29] ...trap/engines/engine.py.229: process: 0.83104 | load: 0.00288 [I 2022-10-09 08:48:29] ...trap/engines/engine.py.230: loss: 0.18661 [I 2022-10-09 08:48:29] ...optimizers/trijoint.py.151: total_norm: 24.854481 [I 2022-10-09 08:48:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 1490/2304 [I 2022-10-09 08:48:37] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:26 [I 2022-10-09 08:48:37] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00290 [I 2022-10-09 08:48:37] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-09 08:48:37] ...optimizers/trijoint.py.151: total_norm: 14.280684 [I 2022-10-09 08:48:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 1500/2304 [I 2022-10-09 08:48:46] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:14 [I 2022-10-09 08:48:46] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00293 [I 2022-10-09 08:48:46] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-09 08:48:46] ...optimizers/trijoint.py.151: total_norm: 19.184090 [I 2022-10-09 08:48:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 1510/2304 [I 2022-10-09 08:48:54] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:12 [I 2022-10-09 08:48:54] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00297 [I 2022-10-09 08:48:54] ...trap/engines/engine.py.230: loss: 0.16683 [I 2022-10-09 08:48:54] ...optimizers/trijoint.py.151: total_norm: 9.954329 [I 2022-10-09 08:49:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 1520/2304 [I 2022-10-09 08:49:02] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:00 [I 2022-10-09 08:49:02] ...trap/engines/engine.py.229: process: 0.83194 | load: 0.00288 [I 2022-10-09 08:49:02] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-09 08:49:02] ...optimizers/trijoint.py.151: total_norm: 14.852242 [I 2022-10-09 08:49:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 1530/2304 [I 2022-10-09 08:49:11] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:10:52 [I 2022-10-09 08:49:11] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00290 [I 2022-10-09 08:49:11] ...trap/engines/engine.py.230: loss: 0.24205 [I 2022-10-09 08:49:11] ...optimizers/trijoint.py.151: total_norm: 20.183466 [I 2022-10-09 08:49:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 1540/2304 [I 2022-10-09 08:49:19] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:10:44 [I 2022-10-09 08:49:19] ...trap/engines/engine.py.229: process: 0.85080 | load: 0.00293 [I 2022-10-09 08:49:19] ...trap/engines/engine.py.230: loss: 0.20858 [I 2022-10-09 08:49:19] ...optimizers/trijoint.py.151: total_norm: 13.921998 [I 2022-10-09 08:49:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 1550/2304 [I 2022-10-09 08:49:28] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:10:37 [I 2022-10-09 08:49:28] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00294 [I 2022-10-09 08:49:28] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-09 08:49:28] ...optimizers/trijoint.py.151: total_norm: 13.775404 [I 2022-10-09 08:49:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 1560/2304 [I 2022-10-09 08:49:36] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:30 [I 2022-10-09 08:49:36] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00292 [I 2022-10-09 08:49:36] ...trap/engines/engine.py.230: loss: 0.15001 [I 2022-10-09 08:49:36] ...optimizers/trijoint.py.151: total_norm: 14.126188 [I 2022-10-09 08:49:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 1570/2304 [I 2022-10-09 08:49:45] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:10:21 [I 2022-10-09 08:49:45] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00289 [I 2022-10-09 08:49:45] ...trap/engines/engine.py.230: loss: 0.15494 [I 2022-10-09 08:49:45] ...optimizers/trijoint.py.151: total_norm: 21.975739 [I 2022-10-09 08:49:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 1580/2304 [I 2022-10-09 08:49:53] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:14 [I 2022-10-09 08:49:53] ...trap/engines/engine.py.229: process: 0.85481 | load: 0.00302 [I 2022-10-09 08:49:53] ...trap/engines/engine.py.230: loss: 0.12979 [I 2022-10-09 08:49:53] ...optimizers/trijoint.py.151: total_norm: 11.078786 [I 2022-10-09 08:50:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 1590/2304 [I 2022-10-09 08:50:02] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:05 [I 2022-10-09 08:50:02] ...trap/engines/engine.py.229: process: 0.83900 | load: 0.00289 [I 2022-10-09 08:50:02] ...trap/engines/engine.py.230: loss: 0.20712 [I 2022-10-09 08:50:02] ...optimizers/trijoint.py.151: total_norm: 19.845406 [I 2022-10-09 08:50:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 1600/2304 [I 2022-10-09 08:50:10] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:09:54 [I 2022-10-09 08:50:10] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00296 [I 2022-10-09 08:50:10] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-09 08:50:10] ...optimizers/trijoint.py.151: total_norm: 13.709781 [I 2022-10-09 08:50:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 1610/2304 [I 2022-10-09 08:50:19] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:09:51 [I 2022-10-09 08:50:19] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00291 [I 2022-10-09 08:50:19] ...trap/engines/engine.py.230: loss: 0.13562 [I 2022-10-09 08:50:19] ...optimizers/trijoint.py.151: total_norm: 10.452441 [I 2022-10-09 08:50:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 1620/2304 [I 2022-10-09 08:50:27] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:09:38 [I 2022-10-09 08:50:27] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00298 [I 2022-10-09 08:50:27] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-09 08:50:27] ...optimizers/trijoint.py.151: total_norm: 9.572951 [I 2022-10-09 08:50:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 1630/2304 [I 2022-10-09 08:50:36] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:09:34 [I 2022-10-09 08:50:36] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00292 [I 2022-10-09 08:50:36] ...trap/engines/engine.py.230: loss: 0.15873 [I 2022-10-09 08:50:36] ...optimizers/trijoint.py.151: total_norm: 43.668282 [I 2022-10-09 08:50:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 1640/2304 [I 2022-10-09 08:50:44] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:23 [I 2022-10-09 08:50:44] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00293 [I 2022-10-09 08:50:44] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-09 08:50:44] ...optimizers/trijoint.py.151: total_norm: 17.820307 [I 2022-10-09 08:50:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 1650/2304 [I 2022-10-09 08:50:53] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:14 [I 2022-10-09 08:50:53] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00287 [I 2022-10-09 08:50:53] ...trap/engines/engine.py.230: loss: 0.12145 [I 2022-10-09 08:50:53] ...optimizers/trijoint.py.151: total_norm: 20.853893 [I 2022-10-09 08:51:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 1660/2304 [I 2022-10-09 08:51:01] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:04 [I 2022-10-09 08:51:01] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00293 [I 2022-10-09 08:51:01] ...trap/engines/engine.py.230: loss: 0.18095 [I 2022-10-09 08:51:01] ...optimizers/trijoint.py.151: total_norm: 12.520039 [I 2022-10-09 08:51:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 1670/2304 [I 2022-10-09 08:51:10] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:08:58 [I 2022-10-09 08:51:10] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00293 [I 2022-10-09 08:51:10] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-09 08:51:10] ...optimizers/trijoint.py.151: total_norm: 14.117919 [I 2022-10-09 08:51:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 1680/2304 [I 2022-10-09 08:51:18] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:08:47 [I 2022-10-09 08:51:18] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00291 [I 2022-10-09 08:51:18] ...trap/engines/engine.py.230: loss: 0.17151 [I 2022-10-09 08:51:18] ...optimizers/trijoint.py.151: total_norm: 17.978937 [I 2022-10-09 08:51:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 1690/2304 [I 2022-10-09 08:51:27] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:08:40 [I 2022-10-09 08:51:27] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00295 [I 2022-10-09 08:51:27] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-09 08:51:27] ...optimizers/trijoint.py.151: total_norm: 10.520163 [I 2022-10-09 08:51:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 1700/2304 [I 2022-10-09 08:51:35] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:08:30 [I 2022-10-09 08:51:35] ...trap/engines/engine.py.229: process: 0.83575 | load: 0.00290 [I 2022-10-09 08:51:35] ...trap/engines/engine.py.230: loss: 0.14753 [I 2022-10-09 08:51:35] ...optimizers/trijoint.py.151: total_norm: 10.286036 [I 2022-10-09 08:51:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 1710/2304 [I 2022-10-09 08:51:44] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:23 [I 2022-10-09 08:51:44] ...trap/engines/engine.py.229: process: 0.84807 | load: 0.00292 [I 2022-10-09 08:51:44] ...trap/engines/engine.py.230: loss: 0.13249 [I 2022-10-09 08:51:44] ...optimizers/trijoint.py.151: total_norm: 8.019035 [I 2022-10-09 08:51:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 1720/2304 [I 2022-10-09 08:51:52] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:14 [I 2022-10-09 08:51:52] ...trap/engines/engine.py.229: process: 0.84341 | load: 0.00294 [I 2022-10-09 08:51:52] ...trap/engines/engine.py.230: loss: 0.14062 [I 2022-10-09 08:51:52] ...optimizers/trijoint.py.151: total_norm: 12.024365 [I 2022-10-09 08:52:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 1730/2304 [I 2022-10-09 08:52:01] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:09 [I 2022-10-09 08:52:01] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00291 [I 2022-10-09 08:52:01] ...trap/engines/engine.py.230: loss: 0.12735 [I 2022-10-09 08:52:01] ...optimizers/trijoint.py.151: total_norm: 48.625397 [I 2022-10-09 08:52:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 1740/2304 [I 2022-10-09 08:52:09] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:07:57 [I 2022-10-09 08:52:09] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00291 [I 2022-10-09 08:52:09] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-09 08:52:09] ...optimizers/trijoint.py.151: total_norm: 51.778934 [I 2022-10-09 08:52:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 1750/2304 [I 2022-10-09 08:52:18] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:07:46 [I 2022-10-09 08:52:18] ...trap/engines/engine.py.229: process: 0.83081 | load: 0.00292 [I 2022-10-09 08:52:18] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-09 08:52:18] ...optimizers/trijoint.py.151: total_norm: 24.791103 [I 2022-10-09 08:52:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 1760/2304 [I 2022-10-09 08:52:26] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:07:39 [I 2022-10-09 08:52:26] ...trap/engines/engine.py.229: process: 0.84102 | load: 0.00295 [I 2022-10-09 08:52:26] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-09 08:52:26] ...optimizers/trijoint.py.151: total_norm: 65.271202 [I 2022-10-09 08:52:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 1770/2304 [I 2022-10-09 08:52:34] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:29 [I 2022-10-09 08:52:34] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00295 [I 2022-10-09 08:52:34] ...trap/engines/engine.py.230: loss: 0.11710 [I 2022-10-09 08:52:34] ...optimizers/trijoint.py.151: total_norm: 19.334965 [I 2022-10-09 08:52:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 1780/2304 [I 2022-10-09 08:52:43] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:07:23 [I 2022-10-09 08:52:43] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00291 [I 2022-10-09 08:52:43] ...trap/engines/engine.py.230: loss: 0.15966 [I 2022-10-09 08:52:43] ...optimizers/trijoint.py.151: total_norm: 11.930716 [I 2022-10-09 08:52:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 1790/2304 [I 2022-10-09 08:52:51] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:13 [I 2022-10-09 08:52:51] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00292 [I 2022-10-09 08:52:51] ...trap/engines/engine.py.230: loss: 0.09626 [I 2022-10-09 08:52:51] ...optimizers/trijoint.py.151: total_norm: 13.174959 [I 2022-10-09 08:53:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 1800/2304 [I 2022-10-09 08:53:00] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:04 [I 2022-10-09 08:53:00] ...trap/engines/engine.py.229: process: 0.84007 | load: 0.00292 [I 2022-10-09 08:53:00] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-09 08:53:00] ...optimizers/trijoint.py.151: total_norm: 18.611801 [I 2022-10-09 08:53:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 1810/2304 [I 2022-10-09 08:53:08] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:06:56 [I 2022-10-09 08:53:08] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00294 [I 2022-10-09 08:53:08] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-09 08:53:08] ...optimizers/trijoint.py.151: total_norm: 13.885584 [I 2022-10-09 08:53:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 1820/2304 [I 2022-10-09 08:53:17] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:06:49 [I 2022-10-09 08:53:17] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00301 [I 2022-10-09 08:53:17] ...trap/engines/engine.py.230: loss: 0.19123 [I 2022-10-09 08:53:17] ...optimizers/trijoint.py.151: total_norm: 25.878986 [I 2022-10-09 08:53:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 1830/2304 [I 2022-10-09 08:53:25] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:06:39 [I 2022-10-09 08:53:25] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00290 [I 2022-10-09 08:53:25] ...trap/engines/engine.py.230: loss: 0.20310 [I 2022-10-09 08:53:25] ...optimizers/trijoint.py.151: total_norm: 25.613194 [I 2022-10-09 08:53:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 1840/2304 [I 2022-10-09 08:53:34] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:06:31 [I 2022-10-09 08:53:34] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00290 [I 2022-10-09 08:53:34] ...trap/engines/engine.py.230: loss: 0.10199 [I 2022-10-09 08:53:34] ...optimizers/trijoint.py.151: total_norm: 11.619287 [I 2022-10-09 08:53:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 1850/2304 [I 2022-10-09 08:53:42] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:06:23 [I 2022-10-09 08:53:42] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00292 [I 2022-10-09 08:53:42] ...trap/engines/engine.py.230: loss: 0.18028 [I 2022-10-09 08:53:42] ...optimizers/trijoint.py.151: total_norm: 25.952499 [I 2022-10-09 08:53:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 1860/2304 [I 2022-10-09 08:53:50] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:12 [I 2022-10-09 08:53:50] ...trap/engines/engine.py.229: process: 0.83259 | load: 0.00291 [I 2022-10-09 08:53:50] ...trap/engines/engine.py.230: loss: 0.23156 [I 2022-10-09 08:53:50] ...optimizers/trijoint.py.151: total_norm: 20.945278 [I 2022-10-09 08:53:59] ...trap/engines/engine.py.224: train: epoch 34 | batch 1870/2304 [I 2022-10-09 08:53:59] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:06:06 [I 2022-10-09 08:53:59] ...trap/engines/engine.py.229: process: 0.84669 | load: 0.00292 [I 2022-10-09 08:53:59] ...trap/engines/engine.py.230: loss: 0.25135 [I 2022-10-09 08:53:59] ...optimizers/trijoint.py.151: total_norm: 30.295784 [I 2022-10-09 08:54:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 1880/2304 [I 2022-10-09 08:54:07] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:05:57 [I 2022-10-09 08:54:07] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00292 [I 2022-10-09 08:54:07] ...trap/engines/engine.py.230: loss: 0.13378 [I 2022-10-09 08:54:07] ...optimizers/trijoint.py.151: total_norm: 12.349570 [I 2022-10-09 08:54:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 1890/2304 [I 2022-10-09 08:54:16] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:05:50 [I 2022-10-09 08:54:16] ...trap/engines/engine.py.229: process: 0.83304 | load: 0.00296 [I 2022-10-09 08:54:16] ...trap/engines/engine.py.230: loss: 0.21558 [I 2022-10-09 08:54:16] ...optimizers/trijoint.py.151: total_norm: 31.439888 [I 2022-10-09 08:54:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 1900/2304 [I 2022-10-09 08:54:24] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:05:40 [I 2022-10-09 08:54:24] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00290 [I 2022-10-09 08:54:24] ...trap/engines/engine.py.230: loss: 0.16951 [I 2022-10-09 08:54:24] ...optimizers/trijoint.py.151: total_norm: 17.119282 [I 2022-10-09 08:54:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 1910/2304 [I 2022-10-09 08:54:33] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:05:33 [I 2022-10-09 08:54:33] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00298 [I 2022-10-09 08:54:33] ...trap/engines/engine.py.230: loss: 0.15865 [I 2022-10-09 08:54:33] ...optimizers/trijoint.py.151: total_norm: 19.236282 [I 2022-10-09 08:54:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 1920/2304 [I 2022-10-09 08:54:41] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:05:24 [I 2022-10-09 08:54:41] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00291 [I 2022-10-09 08:54:41] ...trap/engines/engine.py.230: loss: 0.20632 [I 2022-10-09 08:54:41] ...optimizers/trijoint.py.151: total_norm: 28.435286 [I 2022-10-09 08:54:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 1930/2304 [I 2022-10-09 08:54:50] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:15 [I 2022-10-09 08:54:50] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00293 [I 2022-10-09 08:54:50] ...trap/engines/engine.py.230: loss: 0.20930 [I 2022-10-09 08:54:50] ...optimizers/trijoint.py.151: total_norm: 17.632635 [I 2022-10-09 08:54:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 1940/2304 [I 2022-10-09 08:54:58] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:06 [I 2022-10-09 08:54:58] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00296 [I 2022-10-09 08:54:58] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-09 08:54:58] ...optimizers/trijoint.py.151: total_norm: 11.739509 [I 2022-10-09 08:55:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 1950/2304 [I 2022-10-09 08:55:07] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:04:59 [I 2022-10-09 08:55:07] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00292 [I 2022-10-09 08:55:07] ...trap/engines/engine.py.230: loss: 0.12773 [I 2022-10-09 08:55:07] ...optimizers/trijoint.py.151: total_norm: 27.288095 [I 2022-10-09 08:55:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 1960/2304 [I 2022-10-09 08:55:15] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:04:49 [I 2022-10-09 08:55:15] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00293 [I 2022-10-09 08:55:15] ...trap/engines/engine.py.230: loss: 0.19460 [I 2022-10-09 08:55:15] ...optimizers/trijoint.py.151: total_norm: 15.401534 [I 2022-10-09 08:55:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 1970/2304 [I 2022-10-09 08:55:23] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:04:42 [I 2022-10-09 08:55:23] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00293 [I 2022-10-09 08:55:23] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-09 08:55:23] ...optimizers/trijoint.py.151: total_norm: 17.460115 [I 2022-10-09 08:55:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 1980/2304 [I 2022-10-09 08:55:32] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:04:34 [I 2022-10-09 08:55:32] ...trap/engines/engine.py.229: process: 0.85657 | load: 0.00293 [I 2022-10-09 08:55:32] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-09 08:55:32] ...optimizers/trijoint.py.151: total_norm: 10.125971 [I 2022-10-09 08:55:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 1990/2304 [I 2022-10-09 08:55:40] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:25 [I 2022-10-09 08:55:40] ...trap/engines/engine.py.229: process: 0.83372 | load: 0.00293 [I 2022-10-09 08:55:40] ...trap/engines/engine.py.230: loss: 0.18292 [I 2022-10-09 08:55:40] ...optimizers/trijoint.py.151: total_norm: 17.393839 [I 2022-10-09 08:55:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 2000/2304 [I 2022-10-09 08:55:49] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:04:17 [I 2022-10-09 08:55:49] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00294 [I 2022-10-09 08:55:49] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-09 08:55:49] ...optimizers/trijoint.py.151: total_norm: 15.472943 [I 2022-10-09 08:55:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 2010/2304 [I 2022-10-09 08:55:57] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:07 [I 2022-10-09 08:55:57] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00287 [I 2022-10-09 08:55:57] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-09 08:55:57] ...optimizers/trijoint.py.151: total_norm: 14.242527 [I 2022-10-09 08:56:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 2020/2304 [I 2022-10-09 08:56:06] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:00 [I 2022-10-09 08:56:06] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00291 [I 2022-10-09 08:56:06] ...trap/engines/engine.py.230: loss: 0.10889 [I 2022-10-09 08:56:06] ...optimizers/trijoint.py.151: total_norm: 13.244041 [I 2022-10-09 08:56:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 2030/2304 [I 2022-10-09 08:56:14] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:03:51 [I 2022-10-09 08:56:14] ...trap/engines/engine.py.229: process: 0.84471 | load: 0.00288 [I 2022-10-09 08:56:14] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-09 08:56:14] ...optimizers/trijoint.py.151: total_norm: 16.186476 [I 2022-10-09 08:56:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 2040/2304 [I 2022-10-09 08:56:22] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:03:42 [I 2022-10-09 08:56:22] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00286 [I 2022-10-09 08:56:22] ...trap/engines/engine.py.230: loss: 0.15081 [I 2022-10-09 08:56:22] ...optimizers/trijoint.py.151: total_norm: 18.771603 [I 2022-10-09 08:56:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 2050/2304 [I 2022-10-09 08:56:31] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:03:34 [I 2022-10-09 08:56:31] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00288 [I 2022-10-09 08:56:31] ...trap/engines/engine.py.230: loss: 0.13971 [I 2022-10-09 08:56:31] ...optimizers/trijoint.py.151: total_norm: 18.272228 [I 2022-10-09 08:56:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 2060/2304 [I 2022-10-09 08:56:39] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:03:26 [I 2022-10-09 08:56:39] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00298 [I 2022-10-09 08:56:39] ...trap/engines/engine.py.230: loss: 0.19500 [I 2022-10-09 08:56:39] ...optimizers/trijoint.py.151: total_norm: 18.462351 [I 2022-10-09 08:56:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 2070/2304 [I 2022-10-09 08:56:48] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:17 [I 2022-10-09 08:56:48] ...trap/engines/engine.py.229: process: 0.83788 | load: 0.00303 [I 2022-10-09 08:56:48] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-09 08:56:48] ...optimizers/trijoint.py.151: total_norm: 17.459311 [I 2022-10-09 08:56:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 2080/2304 [I 2022-10-09 08:56:56] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:08 [I 2022-10-09 08:56:56] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00291 [I 2022-10-09 08:56:56] ...trap/engines/engine.py.230: loss: 0.19169 [I 2022-10-09 08:56:56] ...optimizers/trijoint.py.151: total_norm: 17.238071 [I 2022-10-09 08:57:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 2090/2304 [I 2022-10-09 08:57:05] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:00 [I 2022-10-09 08:57:05] ...trap/engines/engine.py.229: process: 0.83151 | load: 0.00292 [I 2022-10-09 08:57:05] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-09 08:57:05] ...optimizers/trijoint.py.151: total_norm: 19.602972 [I 2022-10-09 08:57:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 2100/2304 [I 2022-10-09 08:57:13] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:02:51 [I 2022-10-09 08:57:13] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00297 [I 2022-10-09 08:57:13] ...trap/engines/engine.py.230: loss: 0.14107 [I 2022-10-09 08:57:13] ...optimizers/trijoint.py.151: total_norm: 17.885986 [I 2022-10-09 08:57:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 2110/2304 [I 2022-10-09 08:57:22] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:02:42 [I 2022-10-09 08:57:22] ...trap/engines/engine.py.229: process: 0.83909 | load: 0.00287 [I 2022-10-09 08:57:22] ...trap/engines/engine.py.230: loss: 0.16781 [I 2022-10-09 08:57:22] ...optimizers/trijoint.py.151: total_norm: 12.729630 [I 2022-10-09 08:57:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 2120/2304 [I 2022-10-09 08:57:30] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:02:35 [I 2022-10-09 08:57:30] ...trap/engines/engine.py.229: process: 0.83816 | load: 0.00290 [I 2022-10-09 08:57:30] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-09 08:57:30] ...optimizers/trijoint.py.151: total_norm: 10.878825 [I 2022-10-09 08:57:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 2130/2304 [I 2022-10-09 08:57:38] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:02:25 [I 2022-10-09 08:57:38] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00298 [I 2022-10-09 08:57:38] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-09 08:57:38] ...optimizers/trijoint.py.151: total_norm: 15.698859 [I 2022-10-09 08:57:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 2140/2304 [I 2022-10-09 08:57:47] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:18 [I 2022-10-09 08:57:47] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00291 [I 2022-10-09 08:57:47] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-09 08:57:47] ...optimizers/trijoint.py.151: total_norm: 23.625078 [I 2022-10-09 08:57:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 2150/2304 [I 2022-10-09 08:57:55] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:02:10 [I 2022-10-09 08:57:55] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00291 [I 2022-10-09 08:57:55] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-09 08:57:55] ...optimizers/trijoint.py.151: total_norm: 13.538120 [I 2022-10-09 08:58:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 2160/2304 [I 2022-10-09 08:58:04] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:01 [I 2022-10-09 08:58:04] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00291 [I 2022-10-09 08:58:04] ...trap/engines/engine.py.230: loss: 0.20249 [I 2022-10-09 08:58:04] ...optimizers/trijoint.py.151: total_norm: 18.124485 [I 2022-10-09 08:58:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 2170/2304 [I 2022-10-09 08:58:12] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:01:53 [I 2022-10-09 08:58:12] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00291 [I 2022-10-09 08:58:12] ...trap/engines/engine.py.230: loss: 0.20006 [I 2022-10-09 08:58:12] ...optimizers/trijoint.py.151: total_norm: 32.262115 [I 2022-10-09 08:58:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 2180/2304 [I 2022-10-09 08:58:21] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:01:44 [I 2022-10-09 08:58:21] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00304 [I 2022-10-09 08:58:21] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-09 08:58:21] ...optimizers/trijoint.py.151: total_norm: 13.351512 [I 2022-10-09 08:58:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 2190/2304 [I 2022-10-09 08:58:29] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:01:36 [I 2022-10-09 08:58:29] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00290 [I 2022-10-09 08:58:29] ...trap/engines/engine.py.230: loss: 0.11281 [I 2022-10-09 08:58:29] ...optimizers/trijoint.py.151: total_norm: 12.402514 [I 2022-10-09 08:58:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 2200/2304 [I 2022-10-09 08:58:38] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:01:27 [I 2022-10-09 08:58:38] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00290 [I 2022-10-09 08:58:38] ...trap/engines/engine.py.230: loss: 0.16298 [I 2022-10-09 08:58:38] ...optimizers/trijoint.py.151: total_norm: 33.008453 [I 2022-10-09 08:58:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 2210/2304 [I 2022-10-09 08:58:46] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:19 [I 2022-10-09 08:58:46] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00292 [I 2022-10-09 08:58:46] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-09 08:58:46] ...optimizers/trijoint.py.151: total_norm: 24.476532 [I 2022-10-09 08:58:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 2220/2304 [I 2022-10-09 08:58:54] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:10 [I 2022-10-09 08:58:54] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00298 [I 2022-10-09 08:58:54] ...trap/engines/engine.py.230: loss: 0.22591 [I 2022-10-09 08:58:54] ...optimizers/trijoint.py.151: total_norm: 16.456985 [I 2022-10-09 08:59:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 2230/2304 [I 2022-10-09 08:59:03] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:02 [I 2022-10-09 08:59:03] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00294 [I 2022-10-09 08:59:03] ...trap/engines/engine.py.230: loss: 0.22889 [I 2022-10-09 08:59:03] ...optimizers/trijoint.py.151: total_norm: 19.389502 [I 2022-10-09 08:59:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 2240/2304 [I 2022-10-09 08:59:11] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:00:53 [I 2022-10-09 08:59:11] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00291 [I 2022-10-09 08:59:11] ...trap/engines/engine.py.230: loss: 0.16035 [I 2022-10-09 08:59:11] ...optimizers/trijoint.py.151: total_norm: 29.247623 [I 2022-10-09 08:59:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 2250/2304 [I 2022-10-09 08:59:20] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:00:45 [I 2022-10-09 08:59:20] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00293 [I 2022-10-09 08:59:20] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-09 08:59:20] ...optimizers/trijoint.py.151: total_norm: 11.181895 [I 2022-10-09 08:59:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 2260/2304 [I 2022-10-09 08:59:28] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:00:36 [I 2022-10-09 08:59:28] ...trap/engines/engine.py.229: process: 0.84278 | load: 0.00293 [I 2022-10-09 08:59:28] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-09 08:59:28] ...optimizers/trijoint.py.151: total_norm: 15.681060 [I 2022-10-09 08:59:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 2270/2304 [I 2022-10-09 08:59:37] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:28 [I 2022-10-09 08:59:37] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00297 [I 2022-10-09 08:59:37] ...trap/engines/engine.py.230: loss: 0.15925 [I 2022-10-09 08:59:37] ...optimizers/trijoint.py.151: total_norm: 22.444187 [I 2022-10-09 08:59:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 2280/2304 [I 2022-10-09 08:59:45] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:00:20 [I 2022-10-09 08:59:45] ...trap/engines/engine.py.229: process: 0.84925 | load: 0.00291 [I 2022-10-09 08:59:45] ...trap/engines/engine.py.230: loss: 0.17238 [I 2022-10-09 08:59:45] ...optimizers/trijoint.py.151: total_norm: 17.201220 [I 2022-10-09 08:59:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 2290/2304 [I 2022-10-09 08:59:54] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:11 [I 2022-10-09 08:59:54] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00293 [I 2022-10-09 08:59:54] ...trap/engines/engine.py.230: loss: 0.16956 [I 2022-10-09 08:59:54] ...optimizers/trijoint.py.151: total_norm: 16.110506 [I 2022-10-09 09:00:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 2300/2304 [I 2022-10-09 09:00:02] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:03 [I 2022-10-09 09:00:02] ...trap/engines/engine.py.229: process: 0.83018 | load: 0.00015 [I 2022-10-09 09:00:02] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-09 09:00:02] ...optimizers/trijoint.py.151: total_norm: 16.413179 [I 2022-10-09 09:00:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 2304/2304 [I 2022-10-09 09:00:05] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:00 [I 2022-10-09 09:00:05] ...trap/engines/engine.py.229: process: 0.82916 | load: 0.00015 [I 2022-10-09 09:00:05] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-09 09:00:05] ...optimizers/trijoint.py.151: total_norm: 21.550611 [S 2022-10-09 09:00:06] ...trap/engines/engine.py.129: train_epoch.epoch: 34 [S 2022-10-09 09:00:06] ...trap/engines/engine.py.129: train_epoch.loss: 0.16286754620114013 [I 2022-10-09 09:00:06] ...s/criterions/triplet.py.84: margin = 0.22000000000000003 [I 2022-10-09 09:00:07] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 34 [I 2022-10-09 09:00:07] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [I 2022-10-09 09:00:10] ...trap/engines/engine.py.321: eval: epoch 34 | batch 0/510 [I 2022-10-09 09:00:10] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:20 [I 2022-10-09 09:00:10] ...trap/engines/engine.py.326: process: 2.74612 | load: 1.78533 [W 2022-10-09 09:00:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 09:00:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 09:00:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 09:00:16] ...trap/engines/engine.py.321: eval: epoch 34 | batch 10/510 [I 2022-10-09 09:00:16] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:06:55 [I 2022-10-09 09:00:16] ...trap/engines/engine.py.326: process: 0.58167 | load: 0.00016 [I 2022-10-09 09:00:16] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 09:00:22] ...trap/engines/engine.py.321: eval: epoch 34 | batch 20/510 [I 2022-10-09 09:00:22] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:09 [I 2022-10-09 09:00:22] ...trap/engines/engine.py.326: process: 0.58976 | load: 0.00019 [I 2022-10-09 09:00:29] ...trap/engines/engine.py.321: eval: epoch 34 | batch 30/510 [I 2022-10-09 09:00:29] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:09 [I 2022-10-09 09:00:29] ...trap/engines/engine.py.326: process: 0.70852 | load: 0.00017 [I 2022-10-09 09:00:36] ...trap/engines/engine.py.321: eval: epoch 34 | batch 40/510 [I 2022-10-09 09:00:36] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:05:25 [I 2022-10-09 09:00:36] ...trap/engines/engine.py.326: process: 0.40444 | load: 0.00017 [I 2022-10-09 09:00:40] ...trap/engines/engine.py.321: eval: epoch 34 | batch 50/510 [I 2022-10-09 09:00:40] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:45 [I 2022-10-09 09:00:40] ...trap/engines/engine.py.326: process: 0.48862 | load: 0.00017 [I 2022-10-09 09:00:45] ...trap/engines/engine.py.321: eval: epoch 34 | batch 60/510 [I 2022-10-09 09:00:45] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:54 [I 2022-10-09 09:00:45] ...trap/engines/engine.py.326: process: 0.55108 | load: 0.00017 [I 2022-10-09 09:00:50] ...trap/engines/engine.py.321: eval: epoch 34 | batch 70/510 [I 2022-10-09 09:00:50] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:45 [I 2022-10-09 09:00:50] ...trap/engines/engine.py.326: process: 0.54594 | load: 0.00017 [I 2022-10-09 09:00:56] ...trap/engines/engine.py.321: eval: epoch 34 | batch 80/510 [I 2022-10-09 09:00:56] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:44 [I 2022-10-09 09:00:56] ...trap/engines/engine.py.326: process: 0.60924 | load: 0.00017 [I 2022-10-09 09:01:00] ...trap/engines/engine.py.321: eval: epoch 34 | batch 90/510 [I 2022-10-09 09:01:00] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:11 [I 2022-10-09 09:01:00] ...trap/engines/engine.py.326: process: 0.41087 | load: 0.00017 [I 2022-10-09 09:01:05] ...trap/engines/engine.py.321: eval: epoch 34 | batch 100/510 [I 2022-10-09 09:01:05] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:57 [I 2022-10-09 09:01:05] ...trap/engines/engine.py.326: process: 0.30776 | load: 0.00017 [I 2022-10-09 09:01:08] ...trap/engines/engine.py.321: eval: epoch 34 | batch 110/510 [I 2022-10-09 09:01:08] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:10 [I 2022-10-09 09:01:08] ...trap/engines/engine.py.326: process: 0.31585 | load: 0.00018 [I 2022-10-09 09:01:11] ...trap/engines/engine.py.321: eval: epoch 34 | batch 120/510 [I 2022-10-09 09:01:11] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:03 [I 2022-10-09 09:01:11] ...trap/engines/engine.py.326: process: 0.31354 | load: 0.00017 [I 2022-10-09 09:01:14] ...trap/engines/engine.py.321: eval: epoch 34 | batch 130/510 [I 2022-10-09 09:01:14] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:59 [I 2022-10-09 09:01:14] ...trap/engines/engine.py.326: process: 0.31466 | load: 0.00018 [I 2022-10-09 09:01:17] ...trap/engines/engine.py.321: eval: epoch 34 | batch 140/510 [I 2022-10-09 09:01:17] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:56 [I 2022-10-09 09:01:17] ...trap/engines/engine.py.326: process: 0.31117 | load: 0.00018 [I 2022-10-09 09:01:20] ...trap/engines/engine.py.321: eval: epoch 34 | batch 150/510 [I 2022-10-09 09:01:20] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:53 [I 2022-10-09 09:01:20] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00018 [I 2022-10-09 09:01:24] ...trap/engines/engine.py.321: eval: epoch 34 | batch 160/510 [I 2022-10-09 09:01:24] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:49 [I 2022-10-09 09:01:24] ...trap/engines/engine.py.326: process: 0.30869 | load: 0.00018 [I 2022-10-09 09:01:27] ...trap/engines/engine.py.321: eval: epoch 34 | batch 170/510 [I 2022-10-09 09:01:27] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-09 09:01:27] ...trap/engines/engine.py.326: process: 0.30458 | load: 0.00018 [I 2022-10-09 09:01:30] ...trap/engines/engine.py.321: eval: epoch 34 | batch 180/510 [I 2022-10-09 09:01:30] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:43 [I 2022-10-09 09:01:30] ...trap/engines/engine.py.326: process: 0.31596 | load: 0.00018 [I 2022-10-09 09:01:33] ...trap/engines/engine.py.321: eval: epoch 34 | batch 190/510 [I 2022-10-09 09:01:33] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:40 [I 2022-10-09 09:01:33] ...trap/engines/engine.py.326: process: 0.31217 | load: 0.00019 [I 2022-10-09 09:01:36] ...trap/engines/engine.py.321: eval: epoch 34 | batch 200/510 [I 2022-10-09 09:01:36] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:37 [I 2022-10-09 09:01:36] ...trap/engines/engine.py.326: process: 0.31700 | load: 0.00018 [I 2022-10-09 09:01:39] ...trap/engines/engine.py.321: eval: epoch 34 | batch 210/510 [I 2022-10-09 09:01:39] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:34 [I 2022-10-09 09:01:39] ...trap/engines/engine.py.326: process: 0.31878 | load: 0.00019 [I 2022-10-09 09:01:42] ...trap/engines/engine.py.321: eval: epoch 34 | batch 220/510 [I 2022-10-09 09:01:42] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:31 [I 2022-10-09 09:01:42] ...trap/engines/engine.py.326: process: 0.31272 | load: 0.00018 [I 2022-10-09 09:01:46] ...trap/engines/engine.py.321: eval: epoch 34 | batch 230/510 [I 2022-10-09 09:01:46] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:28 [I 2022-10-09 09:01:46] ...trap/engines/engine.py.326: process: 0.32170 | load: 0.00018 [I 2022-10-09 09:01:49] ...trap/engines/engine.py.321: eval: epoch 34 | batch 240/510 [I 2022-10-09 09:01:49] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:24 [I 2022-10-09 09:01:49] ...trap/engines/engine.py.326: process: 0.31021 | load: 0.00018 [I 2022-10-09 09:01:52] ...trap/engines/engine.py.321: eval: epoch 34 | batch 250/510 [I 2022-10-09 09:01:52] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:20 [I 2022-10-09 09:01:52] ...trap/engines/engine.py.326: process: 0.31029 | load: 0.00018 [I 2022-10-09 09:01:55] ...trap/engines/engine.py.321: eval: epoch 34 | batch 260/510 [I 2022-10-09 09:01:55] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:18 [I 2022-10-09 09:01:55] ...trap/engines/engine.py.326: process: 0.31645 | load: 0.00019 [I 2022-10-09 09:01:58] ...trap/engines/engine.py.321: eval: epoch 34 | batch 270/510 [I 2022-10-09 09:01:58] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:15 [I 2022-10-09 09:01:58] ...trap/engines/engine.py.326: process: 0.31834 | load: 0.00018 [I 2022-10-09 09:02:01] ...trap/engines/engine.py.321: eval: epoch 34 | batch 280/510 [I 2022-10-09 09:02:01] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:12 [I 2022-10-09 09:02:01] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00018 [I 2022-10-09 09:02:04] ...trap/engines/engine.py.321: eval: epoch 34 | batch 290/510 [I 2022-10-09 09:02:04] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:09 [I 2022-10-09 09:02:04] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00019 [I 2022-10-09 09:02:08] ...trap/engines/engine.py.321: eval: epoch 34 | batch 300/510 [I 2022-10-09 09:02:08] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-09 09:02:08] ...trap/engines/engine.py.326: process: 0.31087 | load: 0.00019 [I 2022-10-09 09:02:11] ...trap/engines/engine.py.321: eval: epoch 34 | batch 310/510 [I 2022-10-09 09:02:11] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:02 [I 2022-10-09 09:02:11] ...trap/engines/engine.py.326: process: 0.31617 | load: 0.00018 [I 2022-10-09 09:02:14] ...trap/engines/engine.py.321: eval: epoch 34 | batch 320/510 [I 2022-10-09 09:02:14] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:59 [I 2022-10-09 09:02:14] ...trap/engines/engine.py.326: process: 0.31466 | load: 0.00018 [I 2022-10-09 09:02:17] ...trap/engines/engine.py.321: eval: epoch 34 | batch 330/510 [I 2022-10-09 09:02:17] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-09 09:02:17] ...trap/engines/engine.py.326: process: 0.31936 | load: 0.00019 [I 2022-10-09 09:02:20] ...trap/engines/engine.py.321: eval: epoch 34 | batch 340/510 [I 2022-10-09 09:02:20] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:53 [I 2022-10-09 09:02:20] ...trap/engines/engine.py.326: process: 0.30838 | load: 0.00018 [I 2022-10-09 09:02:23] ...trap/engines/engine.py.321: eval: epoch 34 | batch 350/510 [I 2022-10-09 09:02:23] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:50 [I 2022-10-09 09:02:23] ...trap/engines/engine.py.326: process: 0.31397 | load: 0.00018 [I 2022-10-09 09:02:27] ...trap/engines/engine.py.321: eval: epoch 34 | batch 360/510 [I 2022-10-09 09:02:27] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-09 09:02:27] ...trap/engines/engine.py.326: process: 0.31137 | load: 0.00018 [I 2022-10-09 09:02:30] ...trap/engines/engine.py.321: eval: epoch 34 | batch 370/510 [I 2022-10-09 09:02:30] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-09 09:02:30] ...trap/engines/engine.py.326: process: 0.32445 | load: 0.00019 [I 2022-10-09 09:02:33] ...trap/engines/engine.py.321: eval: epoch 34 | batch 380/510 [I 2022-10-09 09:02:33] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:40 [I 2022-10-09 09:02:33] ...trap/engines/engine.py.326: process: 0.31186 | load: 0.00021 [I 2022-10-09 09:02:36] ...trap/engines/engine.py.321: eval: epoch 34 | batch 390/510 [I 2022-10-09 09:02:36] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-09 09:02:36] ...trap/engines/engine.py.326: process: 0.31730 | load: 0.00020 [I 2022-10-09 09:02:39] ...trap/engines/engine.py.321: eval: epoch 34 | batch 400/510 [I 2022-10-09 09:02:39] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:34 [I 2022-10-09 09:02:39] ...trap/engines/engine.py.326: process: 0.31499 | load: 0.00018 [I 2022-10-09 09:02:42] ...trap/engines/engine.py.321: eval: epoch 34 | batch 410/510 [I 2022-10-09 09:02:42] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-09 09:02:42] ...trap/engines/engine.py.326: process: 0.31513 | load: 0.00019 [I 2022-10-09 09:02:46] ...trap/engines/engine.py.321: eval: epoch 34 | batch 420/510 [I 2022-10-09 09:02:46] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-09 09:02:46] ...trap/engines/engine.py.326: process: 0.31149 | load: 0.00019 [I 2022-10-09 09:02:49] ...trap/engines/engine.py.321: eval: epoch 34 | batch 430/510 [I 2022-10-09 09:02:49] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-09 09:02:49] ...trap/engines/engine.py.326: process: 0.31964 | load: 0.00019 [I 2022-10-09 09:02:52] ...trap/engines/engine.py.321: eval: epoch 34 | batch 440/510 [I 2022-10-09 09:02:52] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-09 09:02:52] ...trap/engines/engine.py.326: process: 0.31680 | load: 0.00018 [I 2022-10-09 09:02:55] ...trap/engines/engine.py.321: eval: epoch 34 | batch 450/510 [I 2022-10-09 09:02:55] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:18 [I 2022-10-09 09:02:55] ...trap/engines/engine.py.326: process: 0.31511 | load: 0.00019 [I 2022-10-09 09:02:58] ...trap/engines/engine.py.321: eval: epoch 34 | batch 460/510 [I 2022-10-09 09:02:58] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-09 09:02:58] ...trap/engines/engine.py.326: process: 0.32037 | load: 0.00019 [I 2022-10-09 09:03:01] ...trap/engines/engine.py.321: eval: epoch 34 | batch 470/510 [I 2022-10-09 09:03:01] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-09 09:03:01] ...trap/engines/engine.py.326: process: 0.31831 | load: 0.00019 [I 2022-10-09 09:03:05] ...trap/engines/engine.py.321: eval: epoch 34 | batch 480/510 [I 2022-10-09 09:03:05] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-09 09:03:05] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00019 [I 2022-10-09 09:03:08] ...trap/engines/engine.py.321: eval: epoch 34 | batch 490/510 [I 2022-10-09 09:03:08] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 09:03:08] ...trap/engines/engine.py.326: process: 0.31720 | load: 0.00021 [I 2022-10-09 09:03:11] ...trap/engines/engine.py.321: eval: epoch 34 | batch 500/510 [I 2022-10-09 09:03:11] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-09 09:03:11] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00020 [I 2022-10-09 09:03:14] ...trap/engines/engine.py.321: eval: epoch 34 | batch 510/510 [I 2022-10-09 09:03:14] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-09 09:03:14] ...trap/engines/engine.py.326: process: 0.31550 | load: 0.00012 [S 2022-10-09 09:03:15] ...trap/engines/engine.py.132: eval_epoch.epoch: 34 [I 2022-10-09 09:03:15] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.705300 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.897400 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933700 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.690700 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.894600 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.927200 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013039 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011164 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007212 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013335 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009769 [S 2022-10-09 09:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009347 [I 2022-10-09 09:03:53] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 09:03:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 09:03:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 09:03:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 09:03:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 09:03:56] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 09:03:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 09:04:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 09:04:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 09:04:02] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 09:04:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 09:04:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 09:04:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 35 [I 2022-10-09 09:04:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 09:04:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 0/2304 [I 2022-10-09 09:04:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:47 [I 2022-10-09 09:04:14] ...trap/engines/engine.py.229: process: 2.88513 | load: 1.70506 [I 2022-10-09 09:04:14] ...trap/engines/engine.py.230: loss: 0.19264 [I 2022-10-09 09:04:14] ...optimizers/trijoint.py.151: total_norm: 8.128606 [I 2022-10-09 09:04:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 10/2304 [I 2022-10-09 09:04:22] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:43 [I 2022-10-09 09:04:22] ...trap/engines/engine.py.229: process: 0.83439 | load: 0.00410 [I 2022-10-09 09:04:22] ...trap/engines/engine.py.230: loss: 0.19669 [I 2022-10-09 09:04:22] ...optimizers/trijoint.py.151: total_norm: 17.151171 [I 2022-10-09 09:04:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 20/2304 [I 2022-10-09 09:04:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:54 [I 2022-10-09 09:04:31] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00442 [I 2022-10-09 09:04:31] ...trap/engines/engine.py.230: loss: 0.21183 [I 2022-10-09 09:04:31] ...optimizers/trijoint.py.151: total_norm: 18.325525 [I 2022-10-09 09:04:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 30/2304 [I 2022-10-09 09:04:39] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:56 [I 2022-10-09 09:04:39] ...trap/engines/engine.py.229: process: 0.83248 | load: 0.00364 [I 2022-10-09 09:04:39] ...trap/engines/engine.py.230: loss: 0.11200 [I 2022-10-09 09:04:39] ...optimizers/trijoint.py.151: total_norm: 13.084903 [I 2022-10-09 09:04:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 40/2304 [I 2022-10-09 09:04:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:47 [I 2022-10-09 09:04:48] ...trap/engines/engine.py.229: process: 0.84500 | load: 0.00344 [I 2022-10-09 09:04:48] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-09 09:04:48] ...optimizers/trijoint.py.151: total_norm: 23.264488 [I 2022-10-09 09:04:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 50/2304 [I 2022-10-09 09:04:56] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:33 [I 2022-10-09 09:04:56] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00336 [I 2022-10-09 09:04:56] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-09 09:04:56] ...optimizers/trijoint.py.151: total_norm: 14.690860 [I 2022-10-09 09:05:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 60/2304 [I 2022-10-09 09:05:04] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:30 [I 2022-10-09 09:05:04] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00439 [I 2022-10-09 09:05:04] ...trap/engines/engine.py.230: loss: 0.23499 [I 2022-10-09 09:05:04] ...optimizers/trijoint.py.151: total_norm: 17.865995 [I 2022-10-09 09:05:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 70/2304 [I 2022-10-09 09:05:13] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:16 [I 2022-10-09 09:05:13] ...trap/engines/engine.py.229: process: 0.83800 | load: 0.00320 [I 2022-10-09 09:05:13] ...trap/engines/engine.py.230: loss: 0.15961 [I 2022-10-09 09:05:13] ...optimizers/trijoint.py.151: total_norm: 12.238045 [I 2022-10-09 09:05:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 80/2304 [I 2022-10-09 09:05:21] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:03 [I 2022-10-09 09:05:21] ...trap/engines/engine.py.229: process: 0.84068 | load: 0.00366 [I 2022-10-09 09:05:21] ...trap/engines/engine.py.230: loss: 0.18593 [I 2022-10-09 09:05:21] ...optimizers/trijoint.py.151: total_norm: 11.059437 [I 2022-10-09 09:05:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 90/2304 [I 2022-10-09 09:05:30] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:02 [I 2022-10-09 09:05:30] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00352 [I 2022-10-09 09:05:30] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-09 09:05:30] ...optimizers/trijoint.py.151: total_norm: 18.474337 [I 2022-10-09 09:05:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 100/2304 [I 2022-10-09 09:05:38] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:42 [I 2022-10-09 09:05:38] ...trap/engines/engine.py.229: process: 0.83472 | load: 0.00289 [I 2022-10-09 09:05:38] ...trap/engines/engine.py.230: loss: 0.15413 [I 2022-10-09 09:05:38] ...optimizers/trijoint.py.151: total_norm: 22.048916 [I 2022-10-09 09:05:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 110/2304 [I 2022-10-09 09:05:46] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:56 [I 2022-10-09 09:05:46] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00318 [I 2022-10-09 09:05:46] ...trap/engines/engine.py.230: loss: 0.14157 [I 2022-10-09 09:05:46] ...optimizers/trijoint.py.151: total_norm: 46.432819 [I 2022-10-09 09:05:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 120/2304 [I 2022-10-09 09:05:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:32 [I 2022-10-09 09:05:55] ...trap/engines/engine.py.229: process: 0.83420 | load: 0.00296 [I 2022-10-09 09:05:55] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-09 09:05:55] ...optimizers/trijoint.py.151: total_norm: 12.291171 [I 2022-10-09 09:06:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 130/2304 [I 2022-10-09 09:06:03] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:34 [I 2022-10-09 09:06:03] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00325 [I 2022-10-09 09:06:03] ...trap/engines/engine.py.230: loss: 0.23014 [I 2022-10-09 09:06:03] ...optimizers/trijoint.py.151: total_norm: 40.628269 [I 2022-10-09 09:06:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 140/2304 [I 2022-10-09 09:06:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:31 [I 2022-10-09 09:06:12] ...trap/engines/engine.py.229: process: 0.83860 | load: 0.00306 [I 2022-10-09 09:06:12] ...trap/engines/engine.py.230: loss: 0.16069 [I 2022-10-09 09:06:12] ...optimizers/trijoint.py.151: total_norm: 17.266729 [I 2022-10-09 09:06:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 150/2304 [I 2022-10-09 09:06:20] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:18 [I 2022-10-09 09:06:20] ...trap/engines/engine.py.229: process: 0.83293 | load: 0.00300 [I 2022-10-09 09:06:20] ...trap/engines/engine.py.230: loss: 0.17216 [I 2022-10-09 09:06:20] ...optimizers/trijoint.py.151: total_norm: 34.847450 [I 2022-10-09 09:06:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 160/2304 [I 2022-10-09 09:06:29] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:06 [I 2022-10-09 09:06:29] ...trap/engines/engine.py.229: process: 0.85465 | load: 0.00300 [I 2022-10-09 09:06:29] ...trap/engines/engine.py.230: loss: 0.16493 [I 2022-10-09 09:06:29] ...optimizers/trijoint.py.151: total_norm: 18.747736 [I 2022-10-09 09:06:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 170/2304 [I 2022-10-09 09:06:37] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:45 [I 2022-10-09 09:06:37] ...trap/engines/engine.py.229: process: 0.82272 | load: 0.00323 [I 2022-10-09 09:06:37] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-09 09:06:37] ...optimizers/trijoint.py.151: total_norm: 7.697462 [I 2022-10-09 09:06:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 180/2304 [I 2022-10-09 09:06:45] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:44 [I 2022-10-09 09:06:45] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00288 [I 2022-10-09 09:06:45] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-09 09:06:45] ...optimizers/trijoint.py.151: total_norm: 14.291058 [I 2022-10-09 09:06:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 190/2304 [I 2022-10-09 09:06:54] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:32 [I 2022-10-09 09:06:54] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00300 [I 2022-10-09 09:06:54] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-09 09:06:54] ...optimizers/trijoint.py.151: total_norm: 13.232066 [I 2022-10-09 09:07:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 200/2304 [I 2022-10-09 09:07:02] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:21 [I 2022-10-09 09:07:02] ...trap/engines/engine.py.229: process: 0.83308 | load: 0.00311 [I 2022-10-09 09:07:02] ...trap/engines/engine.py.230: loss: 0.16754 [I 2022-10-09 09:07:02] ...optimizers/trijoint.py.151: total_norm: 10.539082 [I 2022-10-09 09:07:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 210/2304 [I 2022-10-09 09:07:29] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:55:38 [I 2022-10-09 09:07:29] ...trap/engines/engine.py.229: process: 0.83144 | load: 0.00309 [I 2022-10-09 09:07:29] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-09 09:07:29] ...optimizers/trijoint.py.151: total_norm: 19.761059 [I 2022-10-09 09:07:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 220/2304 [I 2022-10-09 09:07:37] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:32:02 [I 2022-10-09 09:07:37] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00301 [I 2022-10-09 09:07:37] ...trap/engines/engine.py.230: loss: 0.15202 [I 2022-10-09 09:07:37] ...optimizers/trijoint.py.151: total_norm: 22.164818 [I 2022-10-09 09:07:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 230/2304 [I 2022-10-09 09:07:46] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:25 [I 2022-10-09 09:07:46] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00308 [I 2022-10-09 09:07:46] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-09 09:07:46] ...optimizers/trijoint.py.151: total_norm: 16.268631 [I 2022-10-09 09:07:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 240/2304 [I 2022-10-09 09:07:54] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:56 [I 2022-10-09 09:07:54] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00289 [I 2022-10-09 09:07:54] ...trap/engines/engine.py.230: loss: 0.11459 [I 2022-10-09 09:07:54] ...optimizers/trijoint.py.151: total_norm: 17.772026 [I 2022-10-09 09:08:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 250/2304 [I 2022-10-09 09:08:02] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:39 [I 2022-10-09 09:08:02] ...trap/engines/engine.py.229: process: 0.83277 | load: 0.00291 [I 2022-10-09 09:08:02] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-09 09:08:02] ...optimizers/trijoint.py.151: total_norm: 8.500454 [I 2022-10-09 09:08:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 260/2304 [I 2022-10-09 09:08:11] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:37 [I 2022-10-09 09:08:11] ...trap/engines/engine.py.229: process: 0.84032 | load: 0.00291 [I 2022-10-09 09:08:11] ...trap/engines/engine.py.230: loss: 0.14199 [I 2022-10-09 09:08:11] ...optimizers/trijoint.py.151: total_norm: 18.975311 [I 2022-10-09 09:08:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 270/2304 [I 2022-10-09 09:08:19] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:31 [I 2022-10-09 09:08:19] ...trap/engines/engine.py.229: process: 0.85586 | load: 0.00302 [I 2022-10-09 09:08:19] ...trap/engines/engine.py.230: loss: 0.17383 [I 2022-10-09 09:08:19] ...optimizers/trijoint.py.151: total_norm: 37.405304 [I 2022-10-09 09:08:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 280/2304 [I 2022-10-09 09:08:28] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:21 [I 2022-10-09 09:08:28] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00313 [I 2022-10-09 09:08:28] ...trap/engines/engine.py.230: loss: 0.21468 [I 2022-10-09 09:08:28] ...optimizers/trijoint.py.151: total_norm: 24.937359 [I 2022-10-09 09:08:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 290/2304 [I 2022-10-09 09:08:36] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:04 [I 2022-10-09 09:08:36] ...trap/engines/engine.py.229: process: 0.83135 | load: 0.00294 [I 2022-10-09 09:08:36] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-09 09:08:36] ...optimizers/trijoint.py.151: total_norm: 13.261321 [I 2022-10-09 09:08:44] ...trap/engines/engine.py.224: train: epoch 35 | batch 300/2304 [I 2022-10-09 09:08:44] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:02 [I 2022-10-09 09:08:44] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00290 [I 2022-10-09 09:08:44] ...trap/engines/engine.py.230: loss: 0.15776 [I 2022-10-09 09:08:44] ...optimizers/trijoint.py.151: total_norm: 19.855743 [I 2022-10-09 09:08:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 310/2304 [I 2022-10-09 09:08:53] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:55 [I 2022-10-09 09:08:53] ...trap/engines/engine.py.229: process: 0.82781 | load: 0.00301 [I 2022-10-09 09:08:53] ...trap/engines/engine.py.230: loss: 0.15569 [I 2022-10-09 09:08:53] ...optimizers/trijoint.py.151: total_norm: 19.983116 [I 2022-10-09 09:09:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 320/2304 [I 2022-10-09 09:09:01] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:47 [I 2022-10-09 09:09:01] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00293 [I 2022-10-09 09:09:01] ...trap/engines/engine.py.230: loss: 0.14176 [I 2022-10-09 09:09:01] ...optimizers/trijoint.py.151: total_norm: 11.273329 [I 2022-10-09 09:09:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 330/2304 [I 2022-10-09 09:09:10] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:41 [I 2022-10-09 09:09:10] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00319 [I 2022-10-09 09:09:10] ...trap/engines/engine.py.230: loss: 0.16563 [I 2022-10-09 09:09:10] ...optimizers/trijoint.py.151: total_norm: 11.759170 [I 2022-10-09 09:09:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 340/2304 [I 2022-10-09 09:09:18] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:44 [I 2022-10-09 09:09:18] ...trap/engines/engine.py.229: process: 0.86222 | load: 0.00301 [I 2022-10-09 09:09:18] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-09 09:09:18] ...optimizers/trijoint.py.151: total_norm: 17.538719 [I 2022-10-09 09:09:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 350/2304 [I 2022-10-09 09:09:26] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:19 [I 2022-10-09 09:09:26] ...trap/engines/engine.py.229: process: 0.84328 | load: 0.00291 [I 2022-10-09 09:09:26] ...trap/engines/engine.py.230: loss: 0.10403 [I 2022-10-09 09:09:26] ...optimizers/trijoint.py.151: total_norm: 20.822311 [I 2022-10-09 09:09:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 360/2304 [I 2022-10-09 09:09:35] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:17 [I 2022-10-09 09:09:35] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00290 [I 2022-10-09 09:09:35] ...trap/engines/engine.py.230: loss: 0.11919 [I 2022-10-09 09:09:35] ...optimizers/trijoint.py.151: total_norm: 11.443019 [I 2022-10-09 09:09:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 370/2304 [I 2022-10-09 09:09:43] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:02 [I 2022-10-09 09:09:43] ...trap/engines/engine.py.229: process: 0.83451 | load: 0.00293 [I 2022-10-09 09:09:43] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-09 09:09:43] ...optimizers/trijoint.py.151: total_norm: 14.368948 [I 2022-10-09 09:09:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 380/2304 [I 2022-10-09 09:09:52] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:00 [I 2022-10-09 09:09:52] ...trap/engines/engine.py.229: process: 0.83987 | load: 0.00292 [I 2022-10-09 09:09:52] ...trap/engines/engine.py.230: loss: 0.13865 [I 2022-10-09 09:09:52] ...optimizers/trijoint.py.151: total_norm: 16.265881 [I 2022-10-09 09:10:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 390/2304 [I 2022-10-09 09:10:00] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:46 [I 2022-10-09 09:10:00] ...trap/engines/engine.py.229: process: 0.83552 | load: 0.00292 [I 2022-10-09 09:10:00] ...trap/engines/engine.py.230: loss: 0.12625 [I 2022-10-09 09:10:00] ...optimizers/trijoint.py.151: total_norm: 18.790340 [I 2022-10-09 09:10:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 400/2304 [I 2022-10-09 09:10:09] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:39 [I 2022-10-09 09:10:09] ...trap/engines/engine.py.229: process: 0.83730 | load: 0.00296 [I 2022-10-09 09:10:09] ...trap/engines/engine.py.230: loss: 0.22967 [I 2022-10-09 09:10:09] ...optimizers/trijoint.py.151: total_norm: 14.969959 [I 2022-10-09 09:10:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 410/2304 [I 2022-10-09 09:10:17] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:31 [I 2022-10-09 09:10:17] ...trap/engines/engine.py.229: process: 0.84278 | load: 0.00294 [I 2022-10-09 09:10:17] ...trap/engines/engine.py.230: loss: 0.17628 [I 2022-10-09 09:10:17] ...optimizers/trijoint.py.151: total_norm: 20.303095 [I 2022-10-09 09:10:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 420/2304 [I 2022-10-09 09:10:25] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:24 [I 2022-10-09 09:10:25] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00290 [I 2022-10-09 09:10:25] ...trap/engines/engine.py.230: loss: 0.12763 [I 2022-10-09 09:10:25] ...optimizers/trijoint.py.151: total_norm: 12.287624 [I 2022-10-09 09:10:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 430/2304 [I 2022-10-09 09:10:34] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:13 [I 2022-10-09 09:10:34] ...trap/engines/engine.py.229: process: 0.82819 | load: 0.00292 [I 2022-10-09 09:10:34] ...trap/engines/engine.py.230: loss: 0.14575 [I 2022-10-09 09:10:34] ...optimizers/trijoint.py.151: total_norm: 11.788806 [I 2022-10-09 09:10:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 440/2304 [I 2022-10-09 09:10:42] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:07 [I 2022-10-09 09:10:42] ...trap/engines/engine.py.229: process: 0.83393 | load: 0.00290 [I 2022-10-09 09:10:42] ...trap/engines/engine.py.230: loss: 0.08110 [I 2022-10-09 09:10:42] ...optimizers/trijoint.py.151: total_norm: 15.344731 [I 2022-10-09 09:10:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 450/2304 [I 2022-10-09 09:10:51] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:05 [I 2022-10-09 09:10:51] ...trap/engines/engine.py.229: process: 0.84849 | load: 0.00295 [I 2022-10-09 09:10:51] ...trap/engines/engine.py.230: loss: 0.15806 [I 2022-10-09 09:10:51] ...optimizers/trijoint.py.151: total_norm: 17.149961 [I 2022-10-09 09:10:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 460/2304 [I 2022-10-09 09:10:59] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:58 [I 2022-10-09 09:10:59] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00293 [I 2022-10-09 09:10:59] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-09 09:10:59] ...optimizers/trijoint.py.151: total_norm: 15.736310 [I 2022-10-09 09:11:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 470/2304 [I 2022-10-09 09:11:08] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:47 [I 2022-10-09 09:11:08] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00291 [I 2022-10-09 09:11:08] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-09 09:11:08] ...optimizers/trijoint.py.151: total_norm: 10.845097 [I 2022-10-09 09:11:16] ...trap/engines/engine.py.224: train: epoch 35 | batch 480/2304 [I 2022-10-09 09:11:16] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:36 [I 2022-10-09 09:11:16] ...trap/engines/engine.py.229: process: 0.83611 | load: 0.00303 [I 2022-10-09 09:11:16] ...trap/engines/engine.py.230: loss: 0.13153 [I 2022-10-09 09:11:16] ...optimizers/trijoint.py.151: total_norm: 19.300581 [I 2022-10-09 09:11:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 490/2304 [I 2022-10-09 09:11:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:30 [I 2022-10-09 09:11:25] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00295 [I 2022-10-09 09:11:25] ...trap/engines/engine.py.230: loss: 0.15183 [I 2022-10-09 09:11:25] ...optimizers/trijoint.py.151: total_norm: 17.959682 [I 2022-10-09 09:11:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 500/2304 [I 2022-10-09 09:11:33] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:16 [I 2022-10-09 09:11:33] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00296 [I 2022-10-09 09:11:33] ...trap/engines/engine.py.230: loss: 0.11127 [I 2022-10-09 09:11:33] ...optimizers/trijoint.py.151: total_norm: 24.622360 [I 2022-10-09 09:11:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 510/2304 [I 2022-10-09 09:11:41] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:08 [I 2022-10-09 09:11:41] ...trap/engines/engine.py.229: process: 0.83673 | load: 0.00296 [I 2022-10-09 09:11:41] ...trap/engines/engine.py.230: loss: 0.15447 [I 2022-10-09 09:11:41] ...optimizers/trijoint.py.151: total_norm: 23.302696 [I 2022-10-09 09:11:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 520/2304 [I 2022-10-09 09:11:50] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:59 [I 2022-10-09 09:11:50] ...trap/engines/engine.py.229: process: 0.83749 | load: 0.00292 [I 2022-10-09 09:11:50] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-09 09:11:50] ...optimizers/trijoint.py.151: total_norm: 29.143473 [I 2022-10-09 09:11:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 530/2304 [I 2022-10-09 09:11:58] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:51 [I 2022-10-09 09:11:58] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00291 [I 2022-10-09 09:11:58] ...trap/engines/engine.py.230: loss: 0.12176 [I 2022-10-09 09:11:58] ...optimizers/trijoint.py.151: total_norm: 20.671730 [I 2022-10-09 09:12:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 540/2304 [I 2022-10-09 09:12:07] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:44 [I 2022-10-09 09:12:07] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00293 [I 2022-10-09 09:12:07] ...trap/engines/engine.py.230: loss: 0.21626 [I 2022-10-09 09:12:07] ...optimizers/trijoint.py.151: total_norm: 23.540087 [I 2022-10-09 09:12:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 550/2304 [I 2022-10-09 09:12:15] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:27 [I 2022-10-09 09:12:15] ...trap/engines/engine.py.229: process: 0.82781 | load: 0.00294 [I 2022-10-09 09:12:15] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-09 09:12:15] ...optimizers/trijoint.py.151: total_norm: 12.821572 [I 2022-10-09 09:12:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 560/2304 [I 2022-10-09 09:12:23] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:27 [I 2022-10-09 09:12:23] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00292 [I 2022-10-09 09:12:23] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-09 09:12:23] ...optimizers/trijoint.py.151: total_norm: 13.828080 [I 2022-10-09 09:12:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 570/2304 [I 2022-10-09 09:12:32] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:10 [I 2022-10-09 09:12:32] ...trap/engines/engine.py.229: process: 0.82736 | load: 0.00293 [I 2022-10-09 09:12:32] ...trap/engines/engine.py.230: loss: 0.21165 [I 2022-10-09 09:12:32] ...optimizers/trijoint.py.151: total_norm: 16.671335 [I 2022-10-09 09:12:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 580/2304 [I 2022-10-09 09:12:40] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:24:11 [I 2022-10-09 09:12:40] ...trap/engines/engine.py.229: process: 0.84715 | load: 0.00300 [I 2022-10-09 09:12:40] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-09 09:12:40] ...optimizers/trijoint.py.151: total_norm: 40.771133 [I 2022-10-09 09:12:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 590/2304 [I 2022-10-09 09:12:49] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:56 [I 2022-10-09 09:12:49] ...trap/engines/engine.py.229: process: 0.83760 | load: 0.00291 [I 2022-10-09 09:12:49] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-09 09:12:49] ...optimizers/trijoint.py.151: total_norm: 17.336767 [I 2022-10-09 09:12:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 600/2304 [I 2022-10-09 09:12:57] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:53 [I 2022-10-09 09:12:57] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00293 [I 2022-10-09 09:12:57] ...trap/engines/engine.py.230: loss: 0.14784 [I 2022-10-09 09:12:57] ...optimizers/trijoint.py.151: total_norm: 8.800414 [I 2022-10-09 09:13:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 610/2304 [I 2022-10-09 09:13:06] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:49 [I 2022-10-09 09:13:06] ...trap/engines/engine.py.229: process: 0.85941 | load: 0.00295 [I 2022-10-09 09:13:06] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-09 09:13:06] ...optimizers/trijoint.py.151: total_norm: 15.097480 [I 2022-10-09 09:13:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 620/2304 [I 2022-10-09 09:13:14] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:44 [I 2022-10-09 09:13:14] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00300 [I 2022-10-09 09:13:14] ...trap/engines/engine.py.230: loss: 0.22075 [I 2022-10-09 09:13:14] ...optimizers/trijoint.py.151: total_norm: 21.656052 [I 2022-10-09 09:13:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 630/2304 [I 2022-10-09 09:13:22] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:40 [I 2022-10-09 09:13:22] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00301 [I 2022-10-09 09:13:22] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-09 09:13:22] ...optimizers/trijoint.py.151: total_norm: 21.391438 [I 2022-10-09 09:13:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 640/2304 [I 2022-10-09 09:13:31] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:18 [I 2022-10-09 09:13:31] ...trap/engines/engine.py.229: process: 0.81973 | load: 0.00295 [I 2022-10-09 09:13:31] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-09 09:13:31] ...optimizers/trijoint.py.151: total_norm: 7.509329 [I 2022-10-09 09:13:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 650/2304 [I 2022-10-09 09:13:39] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:23:20 [I 2022-10-09 09:13:39] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00290 [I 2022-10-09 09:13:39] ...trap/engines/engine.py.230: loss: 0.16138 [I 2022-10-09 09:13:39] ...optimizers/trijoint.py.151: total_norm: 13.657410 [I 2022-10-09 09:13:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 660/2304 [I 2022-10-09 09:13:48] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:23:08 [I 2022-10-09 09:13:48] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00291 [I 2022-10-09 09:13:48] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-09 09:13:48] ...optimizers/trijoint.py.151: total_norm: 16.156708 [I 2022-10-09 09:13:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 670/2304 [I 2022-10-09 09:13:56] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:23:01 [I 2022-10-09 09:13:56] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00293 [I 2022-10-09 09:13:56] ...trap/engines/engine.py.230: loss: 0.30160 [I 2022-10-09 09:13:56] ...optimizers/trijoint.py.151: total_norm: 15.384703 [I 2022-10-09 09:14:05] ...trap/engines/engine.py.224: train: epoch 35 | batch 680/2304 [I 2022-10-09 09:14:05] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:49 [I 2022-10-09 09:14:05] ...trap/engines/engine.py.229: process: 0.82449 | load: 0.00293 [I 2022-10-09 09:14:05] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-09 09:14:05] ...optimizers/trijoint.py.151: total_norm: 17.762001 [I 2022-10-09 09:14:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 690/2304 [I 2022-10-09 09:14:13] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:39 [I 2022-10-09 09:14:13] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00294 [I 2022-10-09 09:14:13] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-09 09:14:13] ...optimizers/trijoint.py.151: total_norm: 11.664214 [I 2022-10-09 09:14:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 700/2304 [I 2022-10-09 09:14:22] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:36 [I 2022-10-09 09:14:22] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00295 [I 2022-10-09 09:14:22] ...trap/engines/engine.py.230: loss: 0.10717 [I 2022-10-09 09:14:22] ...optimizers/trijoint.py.151: total_norm: 9.974578 [I 2022-10-09 09:14:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 710/2304 [I 2022-10-09 09:14:30] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:31 [I 2022-10-09 09:14:30] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00302 [I 2022-10-09 09:14:30] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-09 09:14:30] ...optimizers/trijoint.py.151: total_norm: 15.671546 [I 2022-10-09 09:14:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 720/2304 [I 2022-10-09 09:14:39] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:18 [I 2022-10-09 09:14:39] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00292 [I 2022-10-09 09:14:39] ...trap/engines/engine.py.230: loss: 0.14570 [I 2022-10-09 09:14:39] ...optimizers/trijoint.py.151: total_norm: 17.680822 [I 2022-10-09 09:14:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 730/2304 [I 2022-10-09 09:14:47] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:22:10 [I 2022-10-09 09:14:47] ...trap/engines/engine.py.229: process: 0.84532 | load: 0.00292 [I 2022-10-09 09:14:47] ...trap/engines/engine.py.230: loss: 0.20470 [I 2022-10-09 09:14:47] ...optimizers/trijoint.py.151: total_norm: 13.376598 [I 2022-10-09 09:14:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 740/2304 [I 2022-10-09 09:14:55] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:54 [I 2022-10-09 09:14:55] ...trap/engines/engine.py.229: process: 0.84463 | load: 0.00292 [I 2022-10-09 09:14:55] ...trap/engines/engine.py.230: loss: 0.19303 [I 2022-10-09 09:14:55] ...optimizers/trijoint.py.151: total_norm: 15.330042 [I 2022-10-09 09:15:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 750/2304 [I 2022-10-09 09:15:04] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:21:51 [I 2022-10-09 09:15:04] ...trap/engines/engine.py.229: process: 0.83859 | load: 0.00292 [I 2022-10-09 09:15:04] ...trap/engines/engine.py.230: loss: 0.16459 [I 2022-10-09 09:15:04] ...optimizers/trijoint.py.151: total_norm: 14.831701 [I 2022-10-09 09:15:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 760/2304 [I 2022-10-09 09:15:12] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:39 [I 2022-10-09 09:15:12] ...trap/engines/engine.py.229: process: 0.83464 | load: 0.00297 [I 2022-10-09 09:15:12] ...trap/engines/engine.py.230: loss: 0.14565 [I 2022-10-09 09:15:12] ...optimizers/trijoint.py.151: total_norm: 23.076283 [I 2022-10-09 09:15:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 770/2304 [I 2022-10-09 09:15:21] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:31 [I 2022-10-09 09:15:21] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00307 [I 2022-10-09 09:15:21] ...trap/engines/engine.py.230: loss: 0.14621 [I 2022-10-09 09:15:21] ...optimizers/trijoint.py.151: total_norm: 16.163504 [I 2022-10-09 09:15:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 780/2304 [I 2022-10-09 09:15:29] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:26 [I 2022-10-09 09:15:29] ...trap/engines/engine.py.229: process: 0.83746 | load: 0.00292 [I 2022-10-09 09:15:29] ...trap/engines/engine.py.230: loss: 0.28144 [I 2022-10-09 09:15:29] ...optimizers/trijoint.py.151: total_norm: 36.145695 [I 2022-10-09 09:15:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 790/2304 [I 2022-10-09 09:15:38] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:19 [I 2022-10-09 09:15:38] ...trap/engines/engine.py.229: process: 0.83876 | load: 0.00293 [I 2022-10-09 09:15:38] ...trap/engines/engine.py.230: loss: 0.20958 [I 2022-10-09 09:15:38] ...optimizers/trijoint.py.151: total_norm: 14.539207 [I 2022-10-09 09:15:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 800/2304 [I 2022-10-09 09:15:46] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:07 [I 2022-10-09 09:15:46] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00293 [I 2022-10-09 09:15:46] ...trap/engines/engine.py.230: loss: 0.17155 [I 2022-10-09 09:15:46] ...optimizers/trijoint.py.151: total_norm: 21.034067 [I 2022-10-09 09:15:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 810/2304 [I 2022-10-09 09:15:55] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:55 [I 2022-10-09 09:15:55] ...trap/engines/engine.py.229: process: 0.83359 | load: 0.00295 [I 2022-10-09 09:15:55] ...trap/engines/engine.py.230: loss: 0.22025 [I 2022-10-09 09:15:55] ...optimizers/trijoint.py.151: total_norm: 19.723972 [I 2022-10-09 09:16:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 820/2304 [I 2022-10-09 09:16:03] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:20:50 [I 2022-10-09 09:16:03] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00305 [I 2022-10-09 09:16:03] ...trap/engines/engine.py.230: loss: 0.14060 [I 2022-10-09 09:16:03] ...optimizers/trijoint.py.151: total_norm: 17.772051 [I 2022-10-09 09:16:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 830/2304 [I 2022-10-09 09:16:12] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:41 [I 2022-10-09 09:16:12] ...trap/engines/engine.py.229: process: 0.83479 | load: 0.00290 [I 2022-10-09 09:16:12] ...trap/engines/engine.py.230: loss: 0.20588 [I 2022-10-09 09:16:12] ...optimizers/trijoint.py.151: total_norm: 16.332075 [I 2022-10-09 09:16:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 840/2304 [I 2022-10-09 09:16:20] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:26 [I 2022-10-09 09:16:20] ...trap/engines/engine.py.229: process: 0.83768 | load: 0.00293 [I 2022-10-09 09:16:20] ...trap/engines/engine.py.230: loss: 0.17302 [I 2022-10-09 09:16:20] ...optimizers/trijoint.py.151: total_norm: 11.596770 [I 2022-10-09 09:16:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 850/2304 [I 2022-10-09 09:16:28] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:24 [I 2022-10-09 09:16:28] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00294 [I 2022-10-09 09:16:28] ...trap/engines/engine.py.230: loss: 0.13531 [I 2022-10-09 09:16:28] ...optimizers/trijoint.py.151: total_norm: 29.449514 [I 2022-10-09 09:16:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 860/2304 [I 2022-10-09 09:16:37] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:16 [I 2022-10-09 09:16:37] ...trap/engines/engine.py.229: process: 0.83464 | load: 0.00296 [I 2022-10-09 09:16:37] ...trap/engines/engine.py.230: loss: 0.15789 [I 2022-10-09 09:16:37] ...optimizers/trijoint.py.151: total_norm: 45.043240 [I 2022-10-09 09:16:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 870/2304 [I 2022-10-09 09:16:45] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:04 [I 2022-10-09 09:16:45] ...trap/engines/engine.py.229: process: 0.84465 | load: 0.00291 [I 2022-10-09 09:16:45] ...trap/engines/engine.py.230: loss: 0.11631 [I 2022-10-09 09:16:45] ...optimizers/trijoint.py.151: total_norm: 12.988959 [I 2022-10-09 09:16:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 880/2304 [I 2022-10-09 09:16:54] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:53 [I 2022-10-09 09:16:54] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00294 [I 2022-10-09 09:16:54] ...trap/engines/engine.py.230: loss: 0.19923 [I 2022-10-09 09:16:54] ...optimizers/trijoint.py.151: total_norm: 36.442654 [I 2022-10-09 09:17:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 890/2304 [I 2022-10-09 09:17:02] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:01 [I 2022-10-09 09:17:02] ...trap/engines/engine.py.229: process: 0.85219 | load: 0.00300 [I 2022-10-09 09:17:02] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-09 09:17:02] ...optimizers/trijoint.py.151: total_norm: 18.539255 [I 2022-10-09 09:17:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 900/2304 [I 2022-10-09 09:17:11] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:47 [I 2022-10-09 09:17:11] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00292 [I 2022-10-09 09:17:11] ...trap/engines/engine.py.230: loss: 0.20096 [I 2022-10-09 09:17:11] ...optimizers/trijoint.py.151: total_norm: 17.208643 [I 2022-10-09 09:17:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 910/2304 [I 2022-10-09 09:17:19] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:33 [I 2022-10-09 09:17:19] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00291 [I 2022-10-09 09:17:19] ...trap/engines/engine.py.230: loss: 0.11151 [I 2022-10-09 09:17:19] ...optimizers/trijoint.py.151: total_norm: 12.507116 [I 2022-10-09 09:17:27] ...trap/engines/engine.py.224: train: epoch 35 | batch 920/2304 [I 2022-10-09 09:17:27] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:26 [I 2022-10-09 09:17:27] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00295 [I 2022-10-09 09:17:27] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-09 09:17:27] ...optimizers/trijoint.py.151: total_norm: 17.546539 [I 2022-10-09 09:17:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 930/2304 [I 2022-10-09 09:17:36] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:14 [I 2022-10-09 09:17:36] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00290 [I 2022-10-09 09:17:36] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-09 09:17:36] ...optimizers/trijoint.py.151: total_norm: 22.728243 [I 2022-10-09 09:17:44] ...trap/engines/engine.py.224: train: epoch 35 | batch 940/2304 [I 2022-10-09 09:17:44] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:10 [I 2022-10-09 09:17:44] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00291 [I 2022-10-09 09:17:44] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-09 09:17:44] ...optimizers/trijoint.py.151: total_norm: 18.353592 [I 2022-10-09 09:17:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 950/2304 [I 2022-10-09 09:17:53] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:50 [I 2022-10-09 09:17:53] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00298 [I 2022-10-09 09:17:53] ...trap/engines/engine.py.230: loss: 0.26954 [I 2022-10-09 09:17:53] ...optimizers/trijoint.py.151: total_norm: 18.659645 [I 2022-10-09 09:18:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 960/2304 [I 2022-10-09 09:18:01] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:51 [I 2022-10-09 09:18:01] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00292 [I 2022-10-09 09:18:01] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-09 09:18:01] ...optimizers/trijoint.py.151: total_norm: 13.942552 [I 2022-10-09 09:18:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 970/2304 [I 2022-10-09 09:18:10] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:44 [I 2022-10-09 09:18:10] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00293 [I 2022-10-09 09:18:10] ...trap/engines/engine.py.230: loss: 0.10330 [I 2022-10-09 09:18:10] ...optimizers/trijoint.py.151: total_norm: 8.906560 [I 2022-10-09 09:18:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 980/2304 [I 2022-10-09 09:18:18] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:41 [I 2022-10-09 09:18:18] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00294 [I 2022-10-09 09:18:18] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-09 09:18:18] ...optimizers/trijoint.py.151: total_norm: 15.754828 [I 2022-10-09 09:18:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 990/2304 [I 2022-10-09 09:18:26] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:26 [I 2022-10-09 09:18:26] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00304 [I 2022-10-09 09:18:26] ...trap/engines/engine.py.230: loss: 0.19106 [I 2022-10-09 09:18:26] ...optimizers/trijoint.py.151: total_norm: 16.682602 [I 2022-10-09 09:18:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 1000/2304 [I 2022-10-09 09:18:35] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:17 [I 2022-10-09 09:18:35] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00292 [I 2022-10-09 09:18:35] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-09 09:18:35] ...optimizers/trijoint.py.151: total_norm: 26.106457 [I 2022-10-09 09:18:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 1010/2304 [I 2022-10-09 09:18:43] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:07 [I 2022-10-09 09:18:43] ...trap/engines/engine.py.229: process: 0.83542 | load: 0.00294 [I 2022-10-09 09:18:43] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-09 09:18:43] ...optimizers/trijoint.py.151: total_norm: 16.912802 [I 2022-10-09 09:18:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 1020/2304 [I 2022-10-09 09:18:52] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:59 [I 2022-10-09 09:18:52] ...trap/engines/engine.py.229: process: 0.83225 | load: 0.00293 [I 2022-10-09 09:18:52] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-09 09:18:52] ...optimizers/trijoint.py.151: total_norm: 17.316132 [I 2022-10-09 09:19:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 1030/2304 [I 2022-10-09 09:19:00] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:53 [I 2022-10-09 09:19:00] ...trap/engines/engine.py.229: process: 0.84867 | load: 0.00303 [I 2022-10-09 09:19:00] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-09 09:19:00] ...optimizers/trijoint.py.151: total_norm: 10.705041 [I 2022-10-09 09:19:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1040/2304 [I 2022-10-09 09:19:09] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:47 [I 2022-10-09 09:19:09] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00293 [I 2022-10-09 09:19:09] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-09 09:19:09] ...optimizers/trijoint.py.151: total_norm: 15.991672 [I 2022-10-09 09:19:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 1050/2304 [I 2022-10-09 09:19:17] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:40 [I 2022-10-09 09:19:17] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00292 [I 2022-10-09 09:19:17] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-09 09:19:17] ...optimizers/trijoint.py.151: total_norm: 13.536812 [I 2022-10-09 09:19:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 1060/2304 [I 2022-10-09 09:19:25] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:31 [I 2022-10-09 09:19:25] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00293 [I 2022-10-09 09:19:25] ...trap/engines/engine.py.230: loss: 0.22064 [I 2022-10-09 09:19:25] ...optimizers/trijoint.py.151: total_norm: 22.125025 [I 2022-10-09 09:19:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 1070/2304 [I 2022-10-09 09:19:34] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:22 [I 2022-10-09 09:19:34] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00294 [I 2022-10-09 09:19:34] ...trap/engines/engine.py.230: loss: 0.20863 [I 2022-10-09 09:19:34] ...optimizers/trijoint.py.151: total_norm: 22.277554 [I 2022-10-09 09:19:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 1080/2304 [I 2022-10-09 09:19:42] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:12 [I 2022-10-09 09:19:42] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00294 [I 2022-10-09 09:19:42] ...trap/engines/engine.py.230: loss: 0.16519 [I 2022-10-09 09:19:42] ...optimizers/trijoint.py.151: total_norm: 17.447823 [I 2022-10-09 09:19:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 1090/2304 [I 2022-10-09 09:19:51] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:00 [I 2022-10-09 09:19:51] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00300 [I 2022-10-09 09:19:51] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-09 09:19:51] ...optimizers/trijoint.py.151: total_norm: 16.215693 [I 2022-10-09 09:19:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 1100/2304 [I 2022-10-09 09:19:59] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:59 [I 2022-10-09 09:19:59] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00294 [I 2022-10-09 09:19:59] ...trap/engines/engine.py.230: loss: 0.17808 [I 2022-10-09 09:19:59] ...optimizers/trijoint.py.151: total_norm: 15.782008 [I 2022-10-09 09:20:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 1110/2304 [I 2022-10-09 09:20:08] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:52 [I 2022-10-09 09:20:08] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00291 [I 2022-10-09 09:20:08] ...trap/engines/engine.py.230: loss: 0.16024 [I 2022-10-09 09:20:08] ...optimizers/trijoint.py.151: total_norm: 13.791338 [I 2022-10-09 09:20:16] ...trap/engines/engine.py.224: train: epoch 35 | batch 1120/2304 [I 2022-10-09 09:20:16] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:46 [I 2022-10-09 09:20:16] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00291 [I 2022-10-09 09:20:16] ...trap/engines/engine.py.230: loss: 0.17563 [I 2022-10-09 09:20:16] ...optimizers/trijoint.py.151: total_norm: 27.821386 [I 2022-10-09 09:20:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 1130/2304 [I 2022-10-09 09:20:25] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:33 [I 2022-10-09 09:20:25] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00293 [I 2022-10-09 09:20:25] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-09 09:20:25] ...optimizers/trijoint.py.151: total_norm: 34.441879 [I 2022-10-09 09:20:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 1140/2304 [I 2022-10-09 09:20:33] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:32 [I 2022-10-09 09:20:33] ...trap/engines/engine.py.229: process: 0.86631 | load: 0.00291 [I 2022-10-09 09:20:33] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-09 09:20:33] ...optimizers/trijoint.py.151: total_norm: 12.703747 [I 2022-10-09 09:20:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 1150/2304 [I 2022-10-09 09:20:42] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:17 [I 2022-10-09 09:20:42] ...trap/engines/engine.py.229: process: 0.83049 | load: 0.00292 [I 2022-10-09 09:20:42] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-09 09:20:42] ...optimizers/trijoint.py.151: total_norm: 18.186111 [I 2022-10-09 09:20:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 1160/2304 [I 2022-10-09 09:20:50] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:08 [I 2022-10-09 09:20:50] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00293 [I 2022-10-09 09:20:50] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-09 09:20:50] ...optimizers/trijoint.py.151: total_norm: 17.299437 [I 2022-10-09 09:20:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 1170/2304 [I 2022-10-09 09:20:59] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:00 [I 2022-10-09 09:20:59] ...trap/engines/engine.py.229: process: 0.84012 | load: 0.00292 [I 2022-10-09 09:20:59] ...trap/engines/engine.py.230: loss: 0.13007 [I 2022-10-09 09:20:59] ...optimizers/trijoint.py.151: total_norm: 15.859850 [I 2022-10-09 09:21:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 1180/2304 [I 2022-10-09 09:21:07] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:50 [I 2022-10-09 09:21:07] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00296 [I 2022-10-09 09:21:07] ...trap/engines/engine.py.230: loss: 0.15564 [I 2022-10-09 09:21:07] ...optimizers/trijoint.py.151: total_norm: 9.309693 [I 2022-10-09 09:21:16] ...trap/engines/engine.py.224: train: epoch 35 | batch 1190/2304 [I 2022-10-09 09:21:16] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:45 [I 2022-10-09 09:21:16] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00294 [I 2022-10-09 09:21:16] ...trap/engines/engine.py.230: loss: 0.19133 [I 2022-10-09 09:21:16] ...optimizers/trijoint.py.151: total_norm: 38.559456 [I 2022-10-09 09:21:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 1200/2304 [I 2022-10-09 09:21:24] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:37 [I 2022-10-09 09:21:24] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00293 [I 2022-10-09 09:21:24] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-09 09:21:24] ...optimizers/trijoint.py.151: total_norm: 18.655920 [I 2022-10-09 09:21:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 1210/2304 [I 2022-10-09 09:21:33] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:30 [I 2022-10-09 09:21:33] ...trap/engines/engine.py.229: process: 0.84798 | load: 0.00298 [I 2022-10-09 09:21:33] ...trap/engines/engine.py.230: loss: 0.11007 [I 2022-10-09 09:21:33] ...optimizers/trijoint.py.151: total_norm: 10.897421 [I 2022-10-09 09:21:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 1220/2304 [I 2022-10-09 09:21:41] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:18 [I 2022-10-09 09:21:41] ...trap/engines/engine.py.229: process: 0.86664 | load: 0.00291 [I 2022-10-09 09:21:41] ...trap/engines/engine.py.230: loss: 0.19415 [I 2022-10-09 09:21:41] ...optimizers/trijoint.py.151: total_norm: 14.408376 [I 2022-10-09 09:21:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 1230/2304 [I 2022-10-09 09:21:50] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:07 [I 2022-10-09 09:21:50] ...trap/engines/engine.py.229: process: 0.83324 | load: 0.00296 [I 2022-10-09 09:21:50] ...trap/engines/engine.py.230: loss: 0.12472 [I 2022-10-09 09:21:50] ...optimizers/trijoint.py.151: total_norm: 16.545481 [I 2022-10-09 09:21:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 1240/2304 [I 2022-10-09 09:21:58] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:59 [I 2022-10-09 09:21:58] ...trap/engines/engine.py.229: process: 0.87531 | load: 0.00295 [I 2022-10-09 09:21:58] ...trap/engines/engine.py.230: loss: 0.18639 [I 2022-10-09 09:21:58] ...optimizers/trijoint.py.151: total_norm: 23.865110 [I 2022-10-09 09:22:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 1250/2304 [I 2022-10-09 09:22:06] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:52 [I 2022-10-09 09:22:06] ...trap/engines/engine.py.229: process: 0.85042 | load: 0.00292 [I 2022-10-09 09:22:06] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-09 09:22:06] ...optimizers/trijoint.py.151: total_norm: 16.739985 [I 2022-10-09 09:22:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 1260/2304 [I 2022-10-09 09:22:15] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:47 [I 2022-10-09 09:22:15] ...trap/engines/engine.py.229: process: 0.87353 | load: 0.00296 [I 2022-10-09 09:22:15] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-09 09:22:15] ...optimizers/trijoint.py.151: total_norm: 16.051950 [I 2022-10-09 09:22:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 1270/2304 [I 2022-10-09 09:22:23] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:34 [I 2022-10-09 09:22:23] ...trap/engines/engine.py.229: process: 0.83516 | load: 0.00296 [I 2022-10-09 09:22:23] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-09 09:22:23] ...optimizers/trijoint.py.151: total_norm: 23.620745 [I 2022-10-09 09:22:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 1280/2304 [I 2022-10-09 09:22:32] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:30 [I 2022-10-09 09:22:32] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00291 [I 2022-10-09 09:22:32] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-09 09:22:32] ...optimizers/trijoint.py.151: total_norm: 24.884876 [I 2022-10-09 09:22:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 1290/2304 [I 2022-10-09 09:22:40] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:16 [I 2022-10-09 09:22:40] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00292 [I 2022-10-09 09:22:40] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-09 09:22:40] ...optimizers/trijoint.py.151: total_norm: 16.681166 [I 2022-10-09 09:22:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 1300/2304 [I 2022-10-09 09:22:49] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:14:14 [I 2022-10-09 09:22:49] ...trap/engines/engine.py.229: process: 0.87629 | load: 0.00295 [I 2022-10-09 09:22:49] ...trap/engines/engine.py.230: loss: 0.15313 [I 2022-10-09 09:22:49] ...optimizers/trijoint.py.151: total_norm: 11.979219 [I 2022-10-09 09:22:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 1310/2304 [I 2022-10-09 09:22:57] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:06 [I 2022-10-09 09:22:57] ...trap/engines/engine.py.229: process: 0.85288 | load: 0.00294 [I 2022-10-09 09:22:57] ...trap/engines/engine.py.230: loss: 0.16169 [I 2022-10-09 09:22:57] ...optimizers/trijoint.py.151: total_norm: 19.930069 [I 2022-10-09 09:23:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 1320/2304 [I 2022-10-09 09:23:06] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:13:55 [I 2022-10-09 09:23:06] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00296 [I 2022-10-09 09:23:06] ...trap/engines/engine.py.230: loss: 0.17237 [I 2022-10-09 09:23:06] ...optimizers/trijoint.py.151: total_norm: 13.721202 [I 2022-10-09 09:23:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 1330/2304 [I 2022-10-09 09:23:14] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:44 [I 2022-10-09 09:23:14] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00301 [I 2022-10-09 09:23:14] ...trap/engines/engine.py.230: loss: 0.18390 [I 2022-10-09 09:23:14] ...optimizers/trijoint.py.151: total_norm: 22.584743 [I 2022-10-09 09:23:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 1340/2304 [I 2022-10-09 09:23:23] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:44 [I 2022-10-09 09:23:23] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00305 [I 2022-10-09 09:23:23] ...trap/engines/engine.py.230: loss: 0.10471 [I 2022-10-09 09:23:23] ...optimizers/trijoint.py.151: total_norm: 15.130621 [I 2022-10-09 09:23:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 1350/2304 [I 2022-10-09 09:23:31] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:27 [I 2022-10-09 09:23:31] ...trap/engines/engine.py.229: process: 0.83258 | load: 0.00292 [I 2022-10-09 09:23:31] ...trap/engines/engine.py.230: loss: 0.19430 [I 2022-10-09 09:23:31] ...optimizers/trijoint.py.151: total_norm: 13.509294 [I 2022-10-09 09:23:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 1360/2304 [I 2022-10-09 09:23:40] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:23 [I 2022-10-09 09:23:40] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00292 [I 2022-10-09 09:23:40] ...trap/engines/engine.py.230: loss: 0.21220 [I 2022-10-09 09:23:40] ...optimizers/trijoint.py.151: total_norm: 27.777637 [I 2022-10-09 09:23:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 1370/2304 [I 2022-10-09 09:23:48] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:08 [I 2022-10-09 09:23:48] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00300 [I 2022-10-09 09:23:48] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-09 09:23:48] ...optimizers/trijoint.py.151: total_norm: 18.939278 [I 2022-10-09 09:23:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 1380/2304 [I 2022-10-09 09:23:57] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:12:56 [I 2022-10-09 09:23:57] ...trap/engines/engine.py.229: process: 0.83043 | load: 0.00300 [I 2022-10-09 09:23:57] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-09 09:23:57] ...optimizers/trijoint.py.151: total_norm: 25.213455 [I 2022-10-09 09:24:05] ...trap/engines/engine.py.224: train: epoch 35 | batch 1390/2304 [I 2022-10-09 09:24:05] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:12:47 [I 2022-10-09 09:24:05] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00293 [I 2022-10-09 09:24:05] ...trap/engines/engine.py.230: loss: 0.15855 [I 2022-10-09 09:24:05] ...optimizers/trijoint.py.151: total_norm: 11.121897 [I 2022-10-09 09:24:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 1400/2304 [I 2022-10-09 09:24:14] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:12:42 [I 2022-10-09 09:24:14] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00292 [I 2022-10-09 09:24:14] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-09 09:24:14] ...optimizers/trijoint.py.151: total_norm: 15.296765 [I 2022-10-09 09:24:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 1410/2304 [I 2022-10-09 09:24:22] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:31 [I 2022-10-09 09:24:22] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00296 [I 2022-10-09 09:24:22] ...trap/engines/engine.py.230: loss: 0.19123 [I 2022-10-09 09:24:22] ...optimizers/trijoint.py.151: total_norm: 26.513088 [I 2022-10-09 09:24:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 1420/2304 [I 2022-10-09 09:24:31] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:12:24 [I 2022-10-09 09:24:31] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00297 [I 2022-10-09 09:24:31] ...trap/engines/engine.py.230: loss: 0.24240 [I 2022-10-09 09:24:31] ...optimizers/trijoint.py.151: total_norm: 14.485372 [I 2022-10-09 09:24:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 1430/2304 [I 2022-10-09 09:24:39] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:19 [I 2022-10-09 09:24:39] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00303 [I 2022-10-09 09:24:39] ...trap/engines/engine.py.230: loss: 0.11782 [I 2022-10-09 09:24:39] ...optimizers/trijoint.py.151: total_norm: 9.999382 [I 2022-10-09 09:24:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 1440/2304 [I 2022-10-09 09:24:47] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:02 [I 2022-10-09 09:24:47] ...trap/engines/engine.py.229: process: 0.82605 | load: 0.00303 [I 2022-10-09 09:24:47] ...trap/engines/engine.py.230: loss: 0.26679 [I 2022-10-09 09:24:47] ...optimizers/trijoint.py.151: total_norm: 32.506348 [I 2022-10-09 09:24:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 1450/2304 [I 2022-10-09 09:24:56] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:02 [I 2022-10-09 09:24:56] ...trap/engines/engine.py.229: process: 0.86153 | load: 0.00301 [I 2022-10-09 09:24:56] ...trap/engines/engine.py.230: loss: 0.14901 [I 2022-10-09 09:24:56] ...optimizers/trijoint.py.151: total_norm: 12.928305 [I 2022-10-09 09:25:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 1460/2304 [I 2022-10-09 09:25:04] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:11:52 [I 2022-10-09 09:25:04] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00294 [I 2022-10-09 09:25:04] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-09 09:25:04] ...optimizers/trijoint.py.151: total_norm: 18.990246 [I 2022-10-09 09:25:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 1470/2304 [I 2022-10-09 09:25:13] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:11:48 [I 2022-10-09 09:25:13] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00298 [I 2022-10-09 09:25:13] ...trap/engines/engine.py.230: loss: 0.19207 [I 2022-10-09 09:25:13] ...optimizers/trijoint.py.151: total_norm: 18.141493 [I 2022-10-09 09:25:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 1480/2304 [I 2022-10-09 09:25:21] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:35 [I 2022-10-09 09:25:21] ...trap/engines/engine.py.229: process: 0.84772 | load: 0.00296 [I 2022-10-09 09:25:21] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-09 09:25:21] ...optimizers/trijoint.py.151: total_norm: 12.398006 [I 2022-10-09 09:25:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 1490/2304 [I 2022-10-09 09:25:30] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:30 [I 2022-10-09 09:25:30] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00293 [I 2022-10-09 09:25:30] ...trap/engines/engine.py.230: loss: 0.17811 [I 2022-10-09 09:25:30] ...optimizers/trijoint.py.151: total_norm: 25.767532 [I 2022-10-09 09:25:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 1500/2304 [I 2022-10-09 09:25:38] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:21 [I 2022-10-09 09:25:38] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00294 [I 2022-10-09 09:25:38] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-09 09:25:38] ...optimizers/trijoint.py.151: total_norm: 23.273663 [I 2022-10-09 09:25:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 1510/2304 [I 2022-10-09 09:25:47] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:14 [I 2022-10-09 09:25:47] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00296 [I 2022-10-09 09:25:47] ...trap/engines/engine.py.230: loss: 0.18887 [I 2022-10-09 09:25:47] ...optimizers/trijoint.py.151: total_norm: 16.204084 [I 2022-10-09 09:25:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 1520/2304 [I 2022-10-09 09:25:55] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:04 [I 2022-10-09 09:25:55] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00296 [I 2022-10-09 09:25:55] ...trap/engines/engine.py.230: loss: 0.15768 [I 2022-10-09 09:25:55] ...optimizers/trijoint.py.151: total_norm: 15.003740 [I 2022-10-09 09:26:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 1530/2304 [I 2022-10-09 09:26:04] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:10:50 [I 2022-10-09 09:26:04] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00296 [I 2022-10-09 09:26:04] ...trap/engines/engine.py.230: loss: 0.17429 [I 2022-10-09 09:26:04] ...optimizers/trijoint.py.151: total_norm: 14.327386 [I 2022-10-09 09:26:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 1540/2304 [I 2022-10-09 09:26:12] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:42 [I 2022-10-09 09:26:12] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00293 [I 2022-10-09 09:26:12] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-09 09:26:12] ...optimizers/trijoint.py.151: total_norm: 8.798520 [I 2022-10-09 09:26:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 1550/2304 [I 2022-10-09 09:26:21] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:35 [I 2022-10-09 09:26:21] ...trap/engines/engine.py.229: process: 0.83057 | load: 0.00295 [I 2022-10-09 09:26:21] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-09 09:26:21] ...optimizers/trijoint.py.151: total_norm: 17.703335 [I 2022-10-09 09:26:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 1560/2304 [I 2022-10-09 09:26:29] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:31 [I 2022-10-09 09:26:29] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00296 [I 2022-10-09 09:26:29] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-09 09:26:29] ...optimizers/trijoint.py.151: total_norm: 28.648405 [I 2022-10-09 09:26:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 1570/2304 [I 2022-10-09 09:26:37] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:18 [I 2022-10-09 09:26:37] ...trap/engines/engine.py.229: process: 0.83145 | load: 0.00296 [I 2022-10-09 09:26:37] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-09 09:26:37] ...optimizers/trijoint.py.151: total_norm: 11.972428 [I 2022-10-09 09:26:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 1580/2304 [I 2022-10-09 09:26:46] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:12 [I 2022-10-09 09:26:46] ...trap/engines/engine.py.229: process: 0.85195 | load: 0.00295 [I 2022-10-09 09:26:46] ...trap/engines/engine.py.230: loss: 0.19298 [I 2022-10-09 09:26:46] ...optimizers/trijoint.py.151: total_norm: 37.931831 [I 2022-10-09 09:26:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 1590/2304 [I 2022-10-09 09:26:54] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:10:01 [I 2022-10-09 09:26:54] ...trap/engines/engine.py.229: process: 0.85383 | load: 0.00295 [I 2022-10-09 09:26:54] ...trap/engines/engine.py.230: loss: 0.20113 [I 2022-10-09 09:26:54] ...optimizers/trijoint.py.151: total_norm: 27.454393 [I 2022-10-09 09:27:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 1600/2304 [I 2022-10-09 09:27:03] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:09:53 [I 2022-10-09 09:27:03] ...trap/engines/engine.py.229: process: 0.84104 | load: 0.00297 [I 2022-10-09 09:27:03] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-09 09:27:03] ...optimizers/trijoint.py.151: total_norm: 13.476303 [I 2022-10-09 09:27:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 1610/2304 [I 2022-10-09 09:27:11] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:09:41 [I 2022-10-09 09:27:11] ...trap/engines/engine.py.229: process: 0.82410 | load: 0.00299 [I 2022-10-09 09:27:11] ...trap/engines/engine.py.230: loss: 0.10893 [I 2022-10-09 09:27:11] ...optimizers/trijoint.py.151: total_norm: 6.713346 [I 2022-10-09 09:27:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 1620/2304 [I 2022-10-09 09:27:20] ...trap/engines/engine.py.228: elapsed: 0:23:08 | left: 0:09:37 [I 2022-10-09 09:27:20] ...trap/engines/engine.py.229: process: 0.84774 | load: 0.00295 [I 2022-10-09 09:27:20] ...trap/engines/engine.py.230: loss: 0.15204 [I 2022-10-09 09:27:20] ...optimizers/trijoint.py.151: total_norm: 37.155670 [I 2022-10-09 09:27:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 1630/2304 [I 2022-10-09 09:27:28] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:29 [I 2022-10-09 09:27:28] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00290 [I 2022-10-09 09:27:28] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-09 09:27:28] ...optimizers/trijoint.py.151: total_norm: 9.939553 [I 2022-10-09 09:27:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 1640/2304 [I 2022-10-09 09:27:37] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:09:17 [I 2022-10-09 09:27:37] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00289 [I 2022-10-09 09:27:37] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-09 09:27:37] ...optimizers/trijoint.py.151: total_norm: 9.455894 [I 2022-10-09 09:27:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 1650/2304 [I 2022-10-09 09:27:45] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:10 [I 2022-10-09 09:27:45] ...trap/engines/engine.py.229: process: 0.85345 | load: 0.00289 [I 2022-10-09 09:27:45] ...trap/engines/engine.py.230: loss: 0.11962 [I 2022-10-09 09:27:45] ...optimizers/trijoint.py.151: total_norm: 12.837257 [I 2022-10-09 09:27:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 1660/2304 [I 2022-10-09 09:27:53] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:09:03 [I 2022-10-09 09:27:53] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00289 [I 2022-10-09 09:27:53] ...trap/engines/engine.py.230: loss: 0.18951 [I 2022-10-09 09:27:53] ...optimizers/trijoint.py.151: total_norm: 11.044724 [I 2022-10-09 09:28:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 1670/2304 [I 2022-10-09 09:28:02] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:08:56 [I 2022-10-09 09:28:02] ...trap/engines/engine.py.229: process: 0.83671 | load: 0.00298 [I 2022-10-09 09:28:02] ...trap/engines/engine.py.230: loss: 0.22506 [I 2022-10-09 09:28:02] ...optimizers/trijoint.py.151: total_norm: 20.064039 [I 2022-10-09 09:28:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 1680/2304 [I 2022-10-09 09:28:10] ...trap/engines/engine.py.228: elapsed: 0:23:59 | left: 0:08:46 [I 2022-10-09 09:28:10] ...trap/engines/engine.py.229: process: 0.82847 | load: 0.00304 [I 2022-10-09 09:28:10] ...trap/engines/engine.py.230: loss: 0.16307 [I 2022-10-09 09:28:10] ...optimizers/trijoint.py.151: total_norm: 12.934732 [I 2022-10-09 09:28:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 1690/2304 [I 2022-10-09 09:28:19] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:37 [I 2022-10-09 09:28:19] ...trap/engines/engine.py.229: process: 0.84908 | load: 0.00292 [I 2022-10-09 09:28:19] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-09 09:28:19] ...optimizers/trijoint.py.151: total_norm: 15.910722 [I 2022-10-09 09:28:27] ...trap/engines/engine.py.224: train: epoch 35 | batch 1700/2304 [I 2022-10-09 09:28:27] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:08:30 [I 2022-10-09 09:28:27] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00292 [I 2022-10-09 09:28:27] ...trap/engines/engine.py.230: loss: 0.21614 [I 2022-10-09 09:28:27] ...optimizers/trijoint.py.151: total_norm: 18.315054 [I 2022-10-09 09:28:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 1710/2304 [I 2022-10-09 09:28:36] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:22 [I 2022-10-09 09:28:36] ...trap/engines/engine.py.229: process: 0.85472 | load: 0.00291 [I 2022-10-09 09:28:36] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-09 09:28:36] ...optimizers/trijoint.py.151: total_norm: 24.700191 [I 2022-10-09 09:28:44] ...trap/engines/engine.py.224: train: epoch 35 | batch 1720/2304 [I 2022-10-09 09:28:44] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:13 [I 2022-10-09 09:28:44] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00293 [I 2022-10-09 09:28:44] ...trap/engines/engine.py.230: loss: 0.18460 [I 2022-10-09 09:28:44] ...optimizers/trijoint.py.151: total_norm: 11.717858 [I 2022-10-09 09:28:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 1730/2304 [I 2022-10-09 09:28:53] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:03 [I 2022-10-09 09:28:53] ...trap/engines/engine.py.229: process: 0.84015 | load: 0.00296 [I 2022-10-09 09:28:53] ...trap/engines/engine.py.230: loss: 0.21460 [I 2022-10-09 09:28:53] ...optimizers/trijoint.py.151: total_norm: 28.318609 [I 2022-10-09 09:29:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 1740/2304 [I 2022-10-09 09:29:01] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:07:53 [I 2022-10-09 09:29:01] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00296 [I 2022-10-09 09:29:01] ...trap/engines/engine.py.230: loss: 0.13694 [I 2022-10-09 09:29:01] ...optimizers/trijoint.py.151: total_norm: 14.621784 [I 2022-10-09 09:29:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1750/2304 [I 2022-10-09 09:29:09] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:07:46 [I 2022-10-09 09:29:09] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00291 [I 2022-10-09 09:29:09] ...trap/engines/engine.py.230: loss: 0.16342 [I 2022-10-09 09:29:09] ...optimizers/trijoint.py.151: total_norm: 13.971922 [I 2022-10-09 09:29:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 1760/2304 [I 2022-10-09 09:29:18] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:38 [I 2022-10-09 09:29:18] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00292 [I 2022-10-09 09:29:18] ...trap/engines/engine.py.230: loss: 0.12966 [I 2022-10-09 09:29:18] ...optimizers/trijoint.py.151: total_norm: 16.513611 [I 2022-10-09 09:29:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 1770/2304 [I 2022-10-09 09:29:26] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:28 [I 2022-10-09 09:29:26] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00296 [I 2022-10-09 09:29:26] ...trap/engines/engine.py.230: loss: 0.18555 [I 2022-10-09 09:29:26] ...optimizers/trijoint.py.151: total_norm: 21.541794 [I 2022-10-09 09:29:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 1780/2304 [I 2022-10-09 09:29:35] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:22 [I 2022-10-09 09:29:35] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00305 [I 2022-10-09 09:29:35] ...trap/engines/engine.py.230: loss: 0.13631 [I 2022-10-09 09:29:35] ...optimizers/trijoint.py.151: total_norm: 17.291117 [I 2022-10-09 09:29:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 1790/2304 [I 2022-10-09 09:29:43] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:10 [I 2022-10-09 09:29:43] ...trap/engines/engine.py.229: process: 0.83287 | load: 0.00293 [I 2022-10-09 09:29:43] ...trap/engines/engine.py.230: loss: 0.21722 [I 2022-10-09 09:29:43] ...optimizers/trijoint.py.151: total_norm: 20.183544 [I 2022-10-09 09:29:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 1800/2304 [I 2022-10-09 09:29:52] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:03 [I 2022-10-09 09:29:52] ...trap/engines/engine.py.229: process: 0.82359 | load: 0.00293 [I 2022-10-09 09:29:52] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-09 09:29:52] ...optimizers/trijoint.py.151: total_norm: 10.469893 [I 2022-10-09 09:30:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 1810/2304 [I 2022-10-09 09:30:00] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:06:57 [I 2022-10-09 09:30:00] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00294 [I 2022-10-09 09:30:00] ...trap/engines/engine.py.230: loss: 0.13014 [I 2022-10-09 09:30:00] ...optimizers/trijoint.py.151: total_norm: 16.184551 [I 2022-10-09 09:30:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1820/2304 [I 2022-10-09 09:30:09] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:06:50 [I 2022-10-09 09:30:09] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00296 [I 2022-10-09 09:30:09] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-09 09:30:09] ...optimizers/trijoint.py.151: total_norm: 23.786116 [I 2022-10-09 09:30:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 1830/2304 [I 2022-10-09 09:30:17] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:40 [I 2022-10-09 09:30:17] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00303 [I 2022-10-09 09:30:17] ...trap/engines/engine.py.230: loss: 0.14907 [I 2022-10-09 09:30:17] ...optimizers/trijoint.py.151: total_norm: 17.737076 [I 2022-10-09 09:30:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 1840/2304 [I 2022-10-09 09:30:25] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:31 [I 2022-10-09 09:30:25] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00292 [I 2022-10-09 09:30:25] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-09 09:30:25] ...optimizers/trijoint.py.151: total_norm: 12.689032 [I 2022-10-09 09:30:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 1850/2304 [I 2022-10-09 09:30:34] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:20 [I 2022-10-09 09:30:34] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00292 [I 2022-10-09 09:30:34] ...trap/engines/engine.py.230: loss: 0.12885 [I 2022-10-09 09:30:34] ...optimizers/trijoint.py.151: total_norm: 8.147178 [I 2022-10-09 09:30:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 1860/2304 [I 2022-10-09 09:30:42] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:14 [I 2022-10-09 09:30:42] ...trap/engines/engine.py.229: process: 0.84074 | load: 0.00292 [I 2022-10-09 09:30:42] ...trap/engines/engine.py.230: loss: 0.08808 [I 2022-10-09 09:30:42] ...optimizers/trijoint.py.151: total_norm: 20.120365 [I 2022-10-09 09:30:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 1870/2304 [I 2022-10-09 09:30:51] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:06 [I 2022-10-09 09:30:51] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00295 [I 2022-10-09 09:30:51] ...trap/engines/engine.py.230: loss: 0.17945 [I 2022-10-09 09:30:51] ...optimizers/trijoint.py.151: total_norm: 22.477362 [I 2022-10-09 09:30:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 1880/2304 [I 2022-10-09 09:30:59] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:05:56 [I 2022-10-09 09:30:59] ...trap/engines/engine.py.229: process: 0.83272 | load: 0.00296 [I 2022-10-09 09:30:59] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-09 09:30:59] ...optimizers/trijoint.py.151: total_norm: 10.923208 [I 2022-10-09 09:31:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 1890/2304 [I 2022-10-09 09:31:08] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:48 [I 2022-10-09 09:31:08] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00289 [I 2022-10-09 09:31:08] ...trap/engines/engine.py.230: loss: 0.24933 [I 2022-10-09 09:31:08] ...optimizers/trijoint.py.151: total_norm: 14.750809 [I 2022-10-09 09:31:16] ...trap/engines/engine.py.224: train: epoch 35 | batch 1900/2304 [I 2022-10-09 09:31:16] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:40 [I 2022-10-09 09:31:16] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00292 [I 2022-10-09 09:31:16] ...trap/engines/engine.py.230: loss: 0.32601 [I 2022-10-09 09:31:16] ...optimizers/trijoint.py.151: total_norm: 24.798933 [I 2022-10-09 09:31:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 1910/2304 [I 2022-10-09 09:31:24] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:32 [I 2022-10-09 09:31:24] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00292 [I 2022-10-09 09:31:24] ...trap/engines/engine.py.230: loss: 0.23171 [I 2022-10-09 09:31:24] ...optimizers/trijoint.py.151: total_norm: 20.642265 [I 2022-10-09 09:31:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 1920/2304 [I 2022-10-09 09:31:33] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:23 [I 2022-10-09 09:31:33] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00293 [I 2022-10-09 09:31:33] ...trap/engines/engine.py.230: loss: 0.18161 [I 2022-10-09 09:31:33] ...optimizers/trijoint.py.151: total_norm: 11.724997 [I 2022-10-09 09:31:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 1930/2304 [I 2022-10-09 09:31:41] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:16 [I 2022-10-09 09:31:41] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00293 [I 2022-10-09 09:31:41] ...trap/engines/engine.py.230: loss: 0.11615 [I 2022-10-09 09:31:41] ...optimizers/trijoint.py.151: total_norm: 15.101789 [I 2022-10-09 09:31:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 1940/2304 [I 2022-10-09 09:31:50] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:06 [I 2022-10-09 09:31:50] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00299 [I 2022-10-09 09:31:50] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-09 09:31:50] ...optimizers/trijoint.py.151: total_norm: 22.970097 [I 2022-10-09 09:31:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 1950/2304 [I 2022-10-09 09:31:58] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:04:58 [I 2022-10-09 09:31:58] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00296 [I 2022-10-09 09:31:58] ...trap/engines/engine.py.230: loss: 0.24373 [I 2022-10-09 09:31:58] ...optimizers/trijoint.py.151: total_norm: 17.167904 [I 2022-10-09 09:32:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 1960/2304 [I 2022-10-09 09:32:07] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:51 [I 2022-10-09 09:32:07] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00289 [I 2022-10-09 09:32:07] ...trap/engines/engine.py.230: loss: 0.14751 [I 2022-10-09 09:32:07] ...optimizers/trijoint.py.151: total_norm: 21.848127 [I 2022-10-09 09:32:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 1970/2304 [I 2022-10-09 09:32:15] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:42 [I 2022-10-09 09:32:15] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00291 [I 2022-10-09 09:32:15] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-09 09:32:15] ...optimizers/trijoint.py.151: total_norm: 17.276834 [I 2022-10-09 09:32:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 1980/2304 [I 2022-10-09 09:32:24] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:35 [I 2022-10-09 09:32:24] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00290 [I 2022-10-09 09:32:24] ...trap/engines/engine.py.230: loss: 0.16029 [I 2022-10-09 09:32:24] ...optimizers/trijoint.py.151: total_norm: 29.595470 [I 2022-10-09 09:32:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 1990/2304 [I 2022-10-09 09:32:32] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:25 [I 2022-10-09 09:32:32] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00293 [I 2022-10-09 09:32:32] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-09 09:32:32] ...optimizers/trijoint.py.151: total_norm: 23.203842 [I 2022-10-09 09:32:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 2000/2304 [I 2022-10-09 09:32:41] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:15 [I 2022-10-09 09:32:41] ...trap/engines/engine.py.229: process: 0.83219 | load: 0.00294 [I 2022-10-09 09:32:41] ...trap/engines/engine.py.230: loss: 0.15196 [I 2022-10-09 09:32:41] ...optimizers/trijoint.py.151: total_norm: 17.030285 [I 2022-10-09 09:32:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 2010/2304 [I 2022-10-09 09:32:49] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:07 [I 2022-10-09 09:32:49] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00299 [I 2022-10-09 09:32:49] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-09 09:32:49] ...optimizers/trijoint.py.151: total_norm: 8.738297 [I 2022-10-09 09:32:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 2020/2304 [I 2022-10-09 09:32:57] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:00 [I 2022-10-09 09:32:57] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00305 [I 2022-10-09 09:32:57] ...trap/engines/engine.py.230: loss: 0.15502 [I 2022-10-09 09:32:57] ...optimizers/trijoint.py.151: total_norm: 9.354177 [I 2022-10-09 09:33:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 2030/2304 [I 2022-10-09 09:33:06] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:03:49 [I 2022-10-09 09:33:06] ...trap/engines/engine.py.229: process: 0.83499 | load: 0.00293 [I 2022-10-09 09:33:06] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-09 09:33:06] ...optimizers/trijoint.py.151: total_norm: 47.303402 [I 2022-10-09 09:33:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 2040/2304 [I 2022-10-09 09:33:14] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:41 [I 2022-10-09 09:33:14] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00293 [I 2022-10-09 09:33:14] ...trap/engines/engine.py.230: loss: 0.15994 [I 2022-10-09 09:33:14] ...optimizers/trijoint.py.151: total_norm: 9.390578 [I 2022-10-09 09:33:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 2050/2304 [I 2022-10-09 09:33:23] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:34 [I 2022-10-09 09:33:23] ...trap/engines/engine.py.229: process: 0.83774 | load: 0.00292 [I 2022-10-09 09:33:23] ...trap/engines/engine.py.230: loss: 0.20354 [I 2022-10-09 09:33:23] ...optimizers/trijoint.py.151: total_norm: 29.135763 [I 2022-10-09 09:33:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 2060/2304 [I 2022-10-09 09:33:31] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:25 [I 2022-10-09 09:33:31] ...trap/engines/engine.py.229: process: 0.84229 | load: 0.00293 [I 2022-10-09 09:33:31] ...trap/engines/engine.py.230: loss: 0.19546 [I 2022-10-09 09:33:31] ...optimizers/trijoint.py.151: total_norm: 16.839361 [I 2022-10-09 09:33:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 2070/2304 [I 2022-10-09 09:33:40] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:17 [I 2022-10-09 09:33:40] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00293 [I 2022-10-09 09:33:40] ...trap/engines/engine.py.230: loss: 0.15775 [I 2022-10-09 09:33:40] ...optimizers/trijoint.py.151: total_norm: 11.030446 [I 2022-10-09 09:33:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 2080/2304 [I 2022-10-09 09:33:48] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:09 [I 2022-10-09 09:33:48] ...trap/engines/engine.py.229: process: 0.85092 | load: 0.00292 [I 2022-10-09 09:33:48] ...trap/engines/engine.py.230: loss: 0.15241 [I 2022-10-09 09:33:48] ...optimizers/trijoint.py.151: total_norm: 31.686161 [I 2022-10-09 09:33:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 2090/2304 [I 2022-10-09 09:33:57] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:00 [I 2022-10-09 09:33:57] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00293 [I 2022-10-09 09:33:57] ...trap/engines/engine.py.230: loss: 0.19183 [I 2022-10-09 09:33:57] ...optimizers/trijoint.py.151: total_norm: 16.939156 [I 2022-10-09 09:34:05] ...trap/engines/engine.py.224: train: epoch 35 | batch 2100/2304 [I 2022-10-09 09:34:05] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:02:51 [I 2022-10-09 09:34:05] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00302 [I 2022-10-09 09:34:05] ...trap/engines/engine.py.230: loss: 0.09510 [I 2022-10-09 09:34:05] ...optimizers/trijoint.py.151: total_norm: 8.103810 [I 2022-10-09 09:34:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 2110/2304 [I 2022-10-09 09:34:13] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:43 [I 2022-10-09 09:34:13] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00293 [I 2022-10-09 09:34:13] ...trap/engines/engine.py.230: loss: 0.19381 [I 2022-10-09 09:34:13] ...optimizers/trijoint.py.151: total_norm: 13.473518 [I 2022-10-09 09:34:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 2120/2304 [I 2022-10-09 09:34:22] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:33 [I 2022-10-09 09:34:22] ...trap/engines/engine.py.229: process: 0.82788 | load: 0.00293 [I 2022-10-09 09:34:22] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-09 09:34:22] ...optimizers/trijoint.py.151: total_norm: 12.970811 [I 2022-10-09 09:34:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 2130/2304 [I 2022-10-09 09:34:30] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:26 [I 2022-10-09 09:34:30] ...trap/engines/engine.py.229: process: 0.83649 | load: 0.00293 [I 2022-10-09 09:34:30] ...trap/engines/engine.py.230: loss: 0.15430 [I 2022-10-09 09:34:30] ...optimizers/trijoint.py.151: total_norm: 14.841401 [I 2022-10-09 09:34:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 2140/2304 [I 2022-10-09 09:34:39] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:18 [I 2022-10-09 09:34:39] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00292 [I 2022-10-09 09:34:39] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-09 09:34:39] ...optimizers/trijoint.py.151: total_norm: 13.779930 [I 2022-10-09 09:34:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 2150/2304 [I 2022-10-09 09:34:47] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:09 [I 2022-10-09 09:34:47] ...trap/engines/engine.py.229: process: 0.83508 | load: 0.00292 [I 2022-10-09 09:34:47] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-09 09:34:47] ...optimizers/trijoint.py.151: total_norm: 9.494146 [I 2022-10-09 09:34:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 2160/2304 [I 2022-10-09 09:34:56] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:01 [I 2022-10-09 09:34:56] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00294 [I 2022-10-09 09:34:56] ...trap/engines/engine.py.230: loss: 0.14430 [I 2022-10-09 09:34:56] ...optimizers/trijoint.py.151: total_norm: 13.040491 [I 2022-10-09 09:35:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 2170/2304 [I 2022-10-09 09:35:04] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:01:52 [I 2022-10-09 09:35:04] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00304 [I 2022-10-09 09:35:04] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-09 09:35:04] ...optimizers/trijoint.py.151: total_norm: 11.389936 [I 2022-10-09 09:35:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 2180/2304 [I 2022-10-09 09:35:12] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:44 [I 2022-10-09 09:35:12] ...trap/engines/engine.py.229: process: 0.83338 | load: 0.00289 [I 2022-10-09 09:35:12] ...trap/engines/engine.py.230: loss: 0.22834 [I 2022-10-09 09:35:12] ...optimizers/trijoint.py.151: total_norm: 39.925514 [I 2022-10-09 09:35:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 2190/2304 [I 2022-10-09 09:35:21] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:35 [I 2022-10-09 09:35:21] ...trap/engines/engine.py.229: process: 0.83517 | load: 0.00293 [I 2022-10-09 09:35:21] ...trap/engines/engine.py.230: loss: 0.22897 [I 2022-10-09 09:35:21] ...optimizers/trijoint.py.151: total_norm: 14.835143 [I 2022-10-09 09:35:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 2200/2304 [I 2022-10-09 09:35:29] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:27 [I 2022-10-09 09:35:29] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00292 [I 2022-10-09 09:35:29] ...trap/engines/engine.py.230: loss: 0.10810 [I 2022-10-09 09:35:29] ...optimizers/trijoint.py.151: total_norm: 11.293572 [I 2022-10-09 09:35:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 2210/2304 [I 2022-10-09 09:35:38] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:19 [I 2022-10-09 09:35:38] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00294 [I 2022-10-09 09:35:38] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-09 09:35:38] ...optimizers/trijoint.py.151: total_norm: 14.044668 [I 2022-10-09 09:35:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 2220/2304 [I 2022-10-09 09:35:46] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:10 [I 2022-10-09 09:35:46] ...trap/engines/engine.py.229: process: 0.83719 | load: 0.00293 [I 2022-10-09 09:35:46] ...trap/engines/engine.py.230: loss: 0.12760 [I 2022-10-09 09:35:46] ...optimizers/trijoint.py.151: total_norm: 16.332880 [I 2022-10-09 09:35:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 2230/2304 [I 2022-10-09 09:35:55] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:02 [I 2022-10-09 09:35:55] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00299 [I 2022-10-09 09:35:55] ...trap/engines/engine.py.230: loss: 0.14533 [I 2022-10-09 09:35:55] ...optimizers/trijoint.py.151: total_norm: 17.351500 [I 2022-10-09 09:36:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 2240/2304 [I 2022-10-09 09:36:03] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:53 [I 2022-10-09 09:36:03] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00290 [I 2022-10-09 09:36:03] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-09 09:36:03] ...optimizers/trijoint.py.151: total_norm: 22.281364 [I 2022-10-09 09:36:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 2250/2304 [I 2022-10-09 09:36:11] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:45 [I 2022-10-09 09:36:11] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00292 [I 2022-10-09 09:36:11] ...trap/engines/engine.py.230: loss: 0.12483 [I 2022-10-09 09:36:11] ...optimizers/trijoint.py.151: total_norm: 14.624188 [I 2022-10-09 09:36:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 2260/2304 [I 2022-10-09 09:36:20] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:00:37 [I 2022-10-09 09:36:20] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00290 [I 2022-10-09 09:36:20] ...trap/engines/engine.py.230: loss: 0.13153 [I 2022-10-09 09:36:20] ...optimizers/trijoint.py.151: total_norm: 23.420412 [I 2022-10-09 09:36:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 2270/2304 [I 2022-10-09 09:36:28] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:28 [I 2022-10-09 09:36:28] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00293 [I 2022-10-09 09:36:28] ...trap/engines/engine.py.230: loss: 0.14991 [I 2022-10-09 09:36:28] ...optimizers/trijoint.py.151: total_norm: 16.072184 [I 2022-10-09 09:36:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 2280/2304 [I 2022-10-09 09:36:37] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:20 [I 2022-10-09 09:36:37] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00292 [I 2022-10-09 09:36:37] ...trap/engines/engine.py.230: loss: 0.16558 [I 2022-10-09 09:36:37] ...optimizers/trijoint.py.151: total_norm: 20.359533 [I 2022-10-09 09:36:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 2290/2304 [I 2022-10-09 09:36:45] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:11 [I 2022-10-09 09:36:45] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00299 [I 2022-10-09 09:36:45] ...trap/engines/engine.py.230: loss: 0.22101 [I 2022-10-09 09:36:45] ...optimizers/trijoint.py.151: total_norm: 16.174324 [I 2022-10-09 09:36:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 2300/2304 [I 2022-10-09 09:36:54] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:03 [I 2022-10-09 09:36:54] ...trap/engines/engine.py.229: process: 0.82955 | load: 0.00017 [I 2022-10-09 09:36:54] ...trap/engines/engine.py.230: loss: 0.14986 [I 2022-10-09 09:36:54] ...optimizers/trijoint.py.151: total_norm: 15.048824 [I 2022-10-09 09:36:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 2304/2304 [I 2022-10-09 09:36:57] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:00 [I 2022-10-09 09:36:57] ...trap/engines/engine.py.229: process: 0.83921 | load: 0.00015 [I 2022-10-09 09:36:57] ...trap/engines/engine.py.230: loss: 0.19368 [I 2022-10-09 09:36:57] ...optimizers/trijoint.py.151: total_norm: 38.882042 [S 2022-10-09 09:36:58] ...trap/engines/engine.py.129: train_epoch.epoch: 35 [S 2022-10-09 09:36:58] ...trap/engines/engine.py.129: train_epoch.loss: 0.161342644316514 [I 2022-10-09 09:36:58] ...s/criterions/triplet.py.84: margin = 0.22500000000000003 [I 2022-10-09 09:36:59] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 35 [I 2022-10-09 09:36:59] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 09:36:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 09:37:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 09:37:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 09:37:01] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 09:37:02] ...trap/engines/engine.py.321: eval: epoch 35 | batch 0/510 [I 2022-10-09 09:37:02] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:26:07 [I 2022-10-09 09:37:02] ...trap/engines/engine.py.326: process: 3.07413 | load: 2.58628 [I 2022-10-09 09:37:06] ...trap/engines/engine.py.321: eval: epoch 35 | batch 10/510 [I 2022-10-09 09:37:06] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:23 [I 2022-10-09 09:37:06] ...trap/engines/engine.py.326: process: 0.50267 | load: 0.00019 [I 2022-10-09 09:37:11] ...trap/engines/engine.py.321: eval: epoch 35 | batch 20/510 [I 2022-10-09 09:37:11] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:08 [I 2022-10-09 09:37:11] ...trap/engines/engine.py.326: process: 0.46858 | load: 0.00018 [I 2022-10-09 09:37:16] ...trap/engines/engine.py.321: eval: epoch 35 | batch 30/510 [I 2022-10-09 09:37:16] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:03:32 [I 2022-10-09 09:37:16] ...trap/engines/engine.py.326: process: 0.39731 | load: 0.00019 [I 2022-10-09 09:37:20] ...trap/engines/engine.py.321: eval: epoch 35 | batch 40/510 [I 2022-10-09 09:37:20] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:03:11 [I 2022-10-09 09:37:20] ...trap/engines/engine.py.326: process: 0.41701 | load: 0.00018 [I 2022-10-09 09:37:24] ...trap/engines/engine.py.321: eval: epoch 35 | batch 50/510 [I 2022-10-09 09:37:24] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:04 [I 2022-10-09 09:37:24] ...trap/engines/engine.py.326: process: 0.40528 | load: 0.00018 [I 2022-10-09 09:37:28] ...trap/engines/engine.py.321: eval: epoch 35 | batch 60/510 [I 2022-10-09 09:37:28] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:21 [I 2022-10-09 09:37:28] ...trap/engines/engine.py.326: process: 0.40263 | load: 0.00017 [I 2022-10-09 09:37:33] ...trap/engines/engine.py.321: eval: epoch 35 | batch 70/510 [I 2022-10-09 09:37:33] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:25 [I 2022-10-09 09:37:33] ...trap/engines/engine.py.326: process: 0.51474 | load: 0.00017 [I 2022-10-09 09:37:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 80/510 [I 2022-10-09 09:37:37] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:10 [I 2022-10-09 09:37:37] ...trap/engines/engine.py.326: process: 0.43697 | load: 0.00018 [I 2022-10-09 09:37:41] ...trap/engines/engine.py.321: eval: epoch 35 | batch 90/510 [I 2022-10-09 09:37:41] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:01 [I 2022-10-09 09:37:41] ...trap/engines/engine.py.326: process: 0.42706 | load: 0.00018 [I 2022-10-09 09:37:46] ...trap/engines/engine.py.321: eval: epoch 35 | batch 100/510 [I 2022-10-09 09:37:46] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:02:53 [I 2022-10-09 09:37:46] ...trap/engines/engine.py.326: process: 0.30894 | load: 0.00017 [I 2022-10-09 09:37:49] ...trap/engines/engine.py.321: eval: epoch 35 | batch 110/510 [I 2022-10-09 09:37:49] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:02:11 [I 2022-10-09 09:37:49] ...trap/engines/engine.py.326: process: 0.31803 | load: 0.00019 [I 2022-10-09 09:37:52] ...trap/engines/engine.py.321: eval: epoch 35 | batch 120/510 [I 2022-10-09 09:37:52] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:03 [I 2022-10-09 09:37:52] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00018 [I 2022-10-09 09:37:55] ...trap/engines/engine.py.321: eval: epoch 35 | batch 130/510 [I 2022-10-09 09:37:55] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:00 [I 2022-10-09 09:37:55] ...trap/engines/engine.py.326: process: 0.31776 | load: 0.00019 [I 2022-10-09 09:37:58] ...trap/engines/engine.py.321: eval: epoch 35 | batch 140/510 [I 2022-10-09 09:37:58] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:57 [I 2022-10-09 09:37:58] ...trap/engines/engine.py.326: process: 0.31362 | load: 0.00019 [I 2022-10-09 09:38:02] ...trap/engines/engine.py.321: eval: epoch 35 | batch 150/510 [I 2022-10-09 09:38:02] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:54 [I 2022-10-09 09:38:02] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-09 09:38:05] ...trap/engines/engine.py.321: eval: epoch 35 | batch 160/510 [I 2022-10-09 09:38:05] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:50 [I 2022-10-09 09:38:05] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00020 [I 2022-10-09 09:38:08] ...trap/engines/engine.py.321: eval: epoch 35 | batch 170/510 [I 2022-10-09 09:38:08] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:47 [I 2022-10-09 09:38:08] ...trap/engines/engine.py.326: process: 0.31652 | load: 0.00019 [I 2022-10-09 09:38:11] ...trap/engines/engine.py.321: eval: epoch 35 | batch 180/510 [I 2022-10-09 09:38:11] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:44 [I 2022-10-09 09:38:11] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00018 [I 2022-10-09 09:38:14] ...trap/engines/engine.py.321: eval: epoch 35 | batch 190/510 [I 2022-10-09 09:38:14] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:41 [I 2022-10-09 09:38:14] ...trap/engines/engine.py.326: process: 0.30864 | load: 0.00020 [I 2022-10-09 09:38:17] ...trap/engines/engine.py.321: eval: epoch 35 | batch 200/510 [I 2022-10-09 09:38:17] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:38 [I 2022-10-09 09:38:17] ...trap/engines/engine.py.326: process: 0.31659 | load: 0.00019 [I 2022-10-09 09:38:21] ...trap/engines/engine.py.321: eval: epoch 35 | batch 210/510 [I 2022-10-09 09:38:21] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:35 [I 2022-10-09 09:38:21] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00020 [I 2022-10-09 09:38:24] ...trap/engines/engine.py.321: eval: epoch 35 | batch 220/510 [I 2022-10-09 09:38:24] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:32 [I 2022-10-09 09:38:24] ...trap/engines/engine.py.326: process: 0.32059 | load: 0.00018 [I 2022-10-09 09:38:27] ...trap/engines/engine.py.321: eval: epoch 35 | batch 230/510 [I 2022-10-09 09:38:27] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:28 [I 2022-10-09 09:38:27] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00019 [I 2022-10-09 09:38:30] ...trap/engines/engine.py.321: eval: epoch 35 | batch 240/510 [I 2022-10-09 09:38:30] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:26 [I 2022-10-09 09:38:30] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00020 [I 2022-10-09 09:38:33] ...trap/engines/engine.py.321: eval: epoch 35 | batch 250/510 [I 2022-10-09 09:38:33] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:22 [I 2022-10-09 09:38:33] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00020 [I 2022-10-09 09:38:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 260/510 [I 2022-10-09 09:38:37] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:19 [I 2022-10-09 09:38:37] ...trap/engines/engine.py.326: process: 0.31090 | load: 0.00019 [I 2022-10-09 09:38:40] ...trap/engines/engine.py.321: eval: epoch 35 | batch 270/510 [I 2022-10-09 09:38:40] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:15 [I 2022-10-09 09:38:40] ...trap/engines/engine.py.326: process: 0.31276 | load: 0.00020 [I 2022-10-09 09:38:43] ...trap/engines/engine.py.321: eval: epoch 35 | batch 280/510 [I 2022-10-09 09:38:43] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:13 [I 2022-10-09 09:38:43] ...trap/engines/engine.py.326: process: 0.31856 | load: 0.00019 [I 2022-10-09 09:38:46] ...trap/engines/engine.py.321: eval: epoch 35 | batch 290/510 [I 2022-10-09 09:38:46] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:09 [I 2022-10-09 09:38:46] ...trap/engines/engine.py.326: process: 0.31824 | load: 0.00019 [I 2022-10-09 09:38:49] ...trap/engines/engine.py.321: eval: epoch 35 | batch 300/510 [I 2022-10-09 09:38:49] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:06 [I 2022-10-09 09:38:49] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00019 [I 2022-10-09 09:38:52] ...trap/engines/engine.py.321: eval: epoch 35 | batch 310/510 [I 2022-10-09 09:38:52] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:03 [I 2022-10-09 09:38:52] ...trap/engines/engine.py.326: process: 0.31997 | load: 0.00019 [I 2022-10-09 09:38:56] ...trap/engines/engine.py.321: eval: epoch 35 | batch 320/510 [I 2022-10-09 09:38:56] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:00 [I 2022-10-09 09:38:56] ...trap/engines/engine.py.326: process: 0.31482 | load: 0.00019 [I 2022-10-09 09:38:59] ...trap/engines/engine.py.321: eval: epoch 35 | batch 330/510 [I 2022-10-09 09:38:59] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:57 [I 2022-10-09 09:38:59] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00020 [I 2022-10-09 09:39:02] ...trap/engines/engine.py.321: eval: epoch 35 | batch 340/510 [I 2022-10-09 09:39:02] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:53 [I 2022-10-09 09:39:02] ...trap/engines/engine.py.326: process: 0.31462 | load: 0.00018 [I 2022-10-09 09:39:05] ...trap/engines/engine.py.321: eval: epoch 35 | batch 350/510 [I 2022-10-09 09:39:05] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:50 [I 2022-10-09 09:39:05] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00019 [I 2022-10-09 09:39:08] ...trap/engines/engine.py.321: eval: epoch 35 | batch 360/510 [I 2022-10-09 09:39:08] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:47 [I 2022-10-09 09:39:08] ...trap/engines/engine.py.326: process: 0.31224 | load: 0.00020 [I 2022-10-09 09:39:12] ...trap/engines/engine.py.321: eval: epoch 35 | batch 370/510 [I 2022-10-09 09:39:12] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:44 [I 2022-10-09 09:39:12] ...trap/engines/engine.py.326: process: 0.31804 | load: 0.00020 [I 2022-10-09 09:39:15] ...trap/engines/engine.py.321: eval: epoch 35 | batch 380/510 [I 2022-10-09 09:39:15] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:41 [I 2022-10-09 09:39:15] ...trap/engines/engine.py.326: process: 0.31372 | load: 0.00018 [I 2022-10-09 09:39:18] ...trap/engines/engine.py.321: eval: epoch 35 | batch 390/510 [I 2022-10-09 09:39:18] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:38 [I 2022-10-09 09:39:18] ...trap/engines/engine.py.326: process: 0.31234 | load: 0.00020 [I 2022-10-09 09:39:21] ...trap/engines/engine.py.321: eval: epoch 35 | batch 400/510 [I 2022-10-09 09:39:21] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:34 [I 2022-10-09 09:39:21] ...trap/engines/engine.py.326: process: 0.31566 | load: 0.00019 [I 2022-10-09 09:39:24] ...trap/engines/engine.py.321: eval: epoch 35 | batch 410/510 [I 2022-10-09 09:39:24] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:31 [I 2022-10-09 09:39:24] ...trap/engines/engine.py.326: process: 0.32177 | load: 0.00020 [I 2022-10-09 09:39:27] ...trap/engines/engine.py.321: eval: epoch 35 | batch 420/510 [I 2022-10-09 09:39:27] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:28 [I 2022-10-09 09:39:27] ...trap/engines/engine.py.326: process: 0.31756 | load: 0.00019 [I 2022-10-09 09:39:31] ...trap/engines/engine.py.321: eval: epoch 35 | batch 430/510 [I 2022-10-09 09:39:31] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:25 [I 2022-10-09 09:39:31] ...trap/engines/engine.py.326: process: 0.31587 | load: 0.00020 [I 2022-10-09 09:39:34] ...trap/engines/engine.py.321: eval: epoch 35 | batch 440/510 [I 2022-10-09 09:39:34] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:22 [I 2022-10-09 09:39:34] ...trap/engines/engine.py.326: process: 0.31601 | load: 0.00018 [I 2022-10-09 09:39:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 450/510 [I 2022-10-09 09:39:37] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:19 [I 2022-10-09 09:39:37] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00019 [I 2022-10-09 09:39:40] ...trap/engines/engine.py.321: eval: epoch 35 | batch 460/510 [I 2022-10-09 09:39:40] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:15 [I 2022-10-09 09:39:40] ...trap/engines/engine.py.326: process: 0.31787 | load: 0.00019 [I 2022-10-09 09:39:43] ...trap/engines/engine.py.321: eval: epoch 35 | batch 470/510 [I 2022-10-09 09:39:43] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:12 [I 2022-10-09 09:39:43] ...trap/engines/engine.py.326: process: 0.32120 | load: 0.00019 [I 2022-10-09 09:39:47] ...trap/engines/engine.py.321: eval: epoch 35 | batch 480/510 [I 2022-10-09 09:39:47] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:09 [I 2022-10-09 09:39:47] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00018 [I 2022-10-09 09:39:50] ...trap/engines/engine.py.321: eval: epoch 35 | batch 490/510 [I 2022-10-09 09:39:50] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:06 [I 2022-10-09 09:39:50] ...trap/engines/engine.py.326: process: 0.31711 | load: 0.00019 [I 2022-10-09 09:39:53] ...trap/engines/engine.py.321: eval: epoch 35 | batch 500/510 [I 2022-10-09 09:39:53] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:03 [I 2022-10-09 09:39:53] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00019 [I 2022-10-09 09:39:56] ...trap/engines/engine.py.321: eval: epoch 35 | batch 510/510 [I 2022-10-09 09:39:56] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:00 [I 2022-10-09 09:39:56] ...trap/engines/engine.py.326: process: 0.31565 | load: 0.00014 [S 2022-10-09 09:39:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 35 [I 2022-10-09 09:39:57] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.705900 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901800 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936900 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.698600 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897100 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931300 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015254 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009282 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010232 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013742 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009731 [S 2022-10-09 09:40:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009349 [I 2022-10-09 09:40:35] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 09:40:35] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 09:40:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 09:40:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 09:40:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 09:40:38] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 09:40:39] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 09:40:44] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 09:40:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 09:40:44] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 09:40:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 09:40:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 09:40:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 36 [I 2022-10-09 09:40:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 09:40:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 0/2304 [I 2022-10-09 09:40:56] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:27 [I 2022-10-09 09:40:56] ...trap/engines/engine.py.229: process: 2.66833 | load: 1.76747 [I 2022-10-09 09:40:56] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-09 09:40:56] ...optimizers/trijoint.py.151: total_norm: 21.572548 [I 2022-10-09 09:41:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 10/2304 [I 2022-10-09 09:41:04] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:47 [I 2022-10-09 09:41:04] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00464 [I 2022-10-09 09:41:04] ...trap/engines/engine.py.230: loss: 0.19808 [I 2022-10-09 09:41:04] ...optimizers/trijoint.py.151: total_norm: 16.544401 [I 2022-10-09 09:41:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 20/2304 [I 2022-10-09 09:41:12] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:53 [I 2022-10-09 09:41:12] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00460 [I 2022-10-09 09:41:12] ...trap/engines/engine.py.230: loss: 0.14059 [I 2022-10-09 09:41:12] ...optimizers/trijoint.py.151: total_norm: 13.025741 [I 2022-10-09 09:41:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 30/2304 [I 2022-10-09 09:41:21] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:31:57 [I 2022-10-09 09:41:21] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00442 [I 2022-10-09 09:41:21] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-09 09:41:21] ...optimizers/trijoint.py.151: total_norm: 20.594816 [I 2022-10-09 09:41:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 40/2304 [I 2022-10-09 09:41:29] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:58 [I 2022-10-09 09:41:29] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00374 [I 2022-10-09 09:41:29] ...trap/engines/engine.py.230: loss: 0.15867 [I 2022-10-09 09:41:29] ...optimizers/trijoint.py.151: total_norm: 15.580538 [I 2022-10-09 09:41:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 50/2304 [I 2022-10-09 09:41:38] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:44 [I 2022-10-09 09:41:38] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00449 [I 2022-10-09 09:41:38] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-09 09:41:38] ...optimizers/trijoint.py.151: total_norm: 9.708825 [I 2022-10-09 09:41:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 60/2304 [I 2022-10-09 09:41:46] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:42 [I 2022-10-09 09:41:46] ...trap/engines/engine.py.229: process: 0.85024 | load: 0.00323 [I 2022-10-09 09:41:46] ...trap/engines/engine.py.230: loss: 0.17884 [I 2022-10-09 09:41:46] ...optimizers/trijoint.py.151: total_norm: 19.205692 [I 2022-10-09 09:41:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 70/2304 [I 2022-10-09 09:41:55] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:23 [I 2022-10-09 09:41:55] ...trap/engines/engine.py.229: process: 0.84864 | load: 0.00359 [I 2022-10-09 09:41:55] ...trap/engines/engine.py.230: loss: 0.14751 [I 2022-10-09 09:41:55] ...optimizers/trijoint.py.151: total_norm: 23.769705 [I 2022-10-09 09:42:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 80/2304 [I 2022-10-09 09:42:03] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:08 [I 2022-10-09 09:42:03] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00329 [I 2022-10-09 09:42:03] ...trap/engines/engine.py.230: loss: 0.11827 [I 2022-10-09 09:42:03] ...optimizers/trijoint.py.151: total_norm: 15.395729 [I 2022-10-09 09:42:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 90/2304 [I 2022-10-09 09:42:11] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:59 [I 2022-10-09 09:42:11] ...trap/engines/engine.py.229: process: 0.83775 | load: 0.00299 [I 2022-10-09 09:42:11] ...trap/engines/engine.py.230: loss: 0.09120 [I 2022-10-09 09:42:11] ...optimizers/trijoint.py.151: total_norm: 7.784611 [I 2022-10-09 09:42:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 100/2304 [I 2022-10-09 09:42:20] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:50 [I 2022-10-09 09:42:20] ...trap/engines/engine.py.229: process: 0.83707 | load: 0.00316 [I 2022-10-09 09:42:20] ...trap/engines/engine.py.230: loss: 0.10716 [I 2022-10-09 09:42:20] ...optimizers/trijoint.py.151: total_norm: 11.107491 [I 2022-10-09 09:42:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 110/2304 [I 2022-10-09 09:42:28] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:50 [I 2022-10-09 09:42:28] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00306 [I 2022-10-09 09:42:28] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-09 09:42:28] ...optimizers/trijoint.py.151: total_norm: 12.910205 [I 2022-10-09 09:42:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 120/2304 [I 2022-10-09 09:42:37] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:36 [I 2022-10-09 09:42:37] ...trap/engines/engine.py.229: process: 0.84081 | load: 0.00307 [I 2022-10-09 09:42:37] ...trap/engines/engine.py.230: loss: 0.18512 [I 2022-10-09 09:42:37] ...optimizers/trijoint.py.151: total_norm: 10.734685 [I 2022-10-09 09:42:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 130/2304 [I 2022-10-09 09:42:45] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:30 [I 2022-10-09 09:42:45] ...trap/engines/engine.py.229: process: 0.83362 | load: 0.00288 [I 2022-10-09 09:42:45] ...trap/engines/engine.py.230: loss: 0.17050 [I 2022-10-09 09:42:45] ...optimizers/trijoint.py.151: total_norm: 13.089698 [I 2022-10-09 09:42:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 140/2304 [I 2022-10-09 09:42:54] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:15 [I 2022-10-09 09:42:54] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00311 [I 2022-10-09 09:42:54] ...trap/engines/engine.py.230: loss: 0.17988 [I 2022-10-09 09:42:54] ...optimizers/trijoint.py.151: total_norm: 14.593556 [I 2022-10-09 09:43:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 150/2304 [I 2022-10-09 09:43:02] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:10 [I 2022-10-09 09:43:02] ...trap/engines/engine.py.229: process: 0.83003 | load: 0.00291 [I 2022-10-09 09:43:02] ...trap/engines/engine.py.230: loss: 0.15396 [I 2022-10-09 09:43:02] ...optimizers/trijoint.py.151: total_norm: 17.531157 [I 2022-10-09 09:43:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 160/2304 [I 2022-10-09 09:43:10] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:07 [I 2022-10-09 09:43:10] ...trap/engines/engine.py.229: process: 0.83513 | load: 0.00316 [I 2022-10-09 09:43:10] ...trap/engines/engine.py.230: loss: 0.20672 [I 2022-10-09 09:43:10] ...optimizers/trijoint.py.151: total_norm: 24.865177 [I 2022-10-09 09:43:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 170/2304 [I 2022-10-09 09:43:19] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:08 [I 2022-10-09 09:43:19] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00317 [I 2022-10-09 09:43:19] ...trap/engines/engine.py.230: loss: 0.10767 [I 2022-10-09 09:43:19] ...optimizers/trijoint.py.151: total_norm: 22.975677 [I 2022-10-09 09:43:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 180/2304 [I 2022-10-09 09:43:27] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:30:05 [I 2022-10-09 09:43:27] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00303 [I 2022-10-09 09:43:27] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-09 09:43:27] ...optimizers/trijoint.py.151: total_norm: 18.428312 [I 2022-10-09 09:43:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 190/2304 [I 2022-10-09 09:43:36] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:51 [I 2022-10-09 09:43:36] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00290 [I 2022-10-09 09:43:36] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-09 09:43:36] ...optimizers/trijoint.py.151: total_norm: 13.404747 [I 2022-10-09 09:43:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 200/2304 [I 2022-10-09 09:43:44] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:43 [I 2022-10-09 09:43:44] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00299 [I 2022-10-09 09:43:44] ...trap/engines/engine.py.230: loss: 0.12679 [I 2022-10-09 09:43:44] ...optimizers/trijoint.py.151: total_norm: 15.467635 [I 2022-10-09 09:43:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 210/2304 [I 2022-10-09 09:43:53] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:29 [I 2022-10-09 09:43:53] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00299 [I 2022-10-09 09:43:53] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-09 09:43:53] ...optimizers/trijoint.py.151: total_norm: 13.723022 [I 2022-10-09 09:44:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 220/2304 [I 2022-10-09 09:44:01] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:26 [I 2022-10-09 09:44:01] ...trap/engines/engine.py.229: process: 0.86430 | load: 0.00302 [I 2022-10-09 09:44:01] ...trap/engines/engine.py.230: loss: 0.18523 [I 2022-10-09 09:44:01] ...optimizers/trijoint.py.151: total_norm: 16.767494 [I 2022-10-09 09:44:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 230/2304 [I 2022-10-09 09:44:10] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:15 [I 2022-10-09 09:44:10] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00299 [I 2022-10-09 09:44:10] ...trap/engines/engine.py.230: loss: 0.19182 [I 2022-10-09 09:44:10] ...optimizers/trijoint.py.151: total_norm: 17.409435 [I 2022-10-09 09:44:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 240/2304 [I 2022-10-09 09:44:18] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:12 [I 2022-10-09 09:44:18] ...trap/engines/engine.py.229: process: 0.85750 | load: 0.00306 [I 2022-10-09 09:44:18] ...trap/engines/engine.py.230: loss: 0.18692 [I 2022-10-09 09:44:18] ...optimizers/trijoint.py.151: total_norm: 24.346701 [I 2022-10-09 09:44:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 250/2304 [I 2022-10-09 09:44:27] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:54 [I 2022-10-09 09:44:27] ...trap/engines/engine.py.229: process: 0.84014 | load: 0.00290 [I 2022-10-09 09:44:27] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-09 09:44:27] ...optimizers/trijoint.py.151: total_norm: 13.075530 [I 2022-10-09 09:44:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 260/2304 [I 2022-10-09 09:44:35] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:42 [I 2022-10-09 09:44:35] ...trap/engines/engine.py.229: process: 0.83007 | load: 0.00294 [I 2022-10-09 09:44:35] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-09 09:44:35] ...optimizers/trijoint.py.151: total_norm: 14.602409 [I 2022-10-09 09:44:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 270/2304 [I 2022-10-09 09:44:44] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:35 [I 2022-10-09 09:44:44] ...trap/engines/engine.py.229: process: 0.84309 | load: 0.00308 [I 2022-10-09 09:44:44] ...trap/engines/engine.py.230: loss: 0.16601 [I 2022-10-09 09:44:44] ...optimizers/trijoint.py.151: total_norm: 23.941471 [I 2022-10-09 09:44:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 280/2304 [I 2022-10-09 09:44:52] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:21 [I 2022-10-09 09:44:52] ...trap/engines/engine.py.229: process: 0.84067 | load: 0.00293 [I 2022-10-09 09:44:52] ...trap/engines/engine.py.230: loss: 0.12436 [I 2022-10-09 09:44:52] ...optimizers/trijoint.py.151: total_norm: 14.142397 [I 2022-10-09 09:45:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 290/2304 [I 2022-10-09 09:45:01] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:10 [I 2022-10-09 09:45:01] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00292 [I 2022-10-09 09:45:01] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-09 09:45:01] ...optimizers/trijoint.py.151: total_norm: 15.490949 [I 2022-10-09 09:45:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 300/2304 [I 2022-10-09 09:45:09] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:10 [I 2022-10-09 09:45:09] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00293 [I 2022-10-09 09:45:09] ...trap/engines/engine.py.230: loss: 0.15815 [I 2022-10-09 09:45:09] ...optimizers/trijoint.py.151: total_norm: 15.037807 [I 2022-10-09 09:45:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 310/2304 [I 2022-10-09 09:45:17] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:01 [I 2022-10-09 09:45:17] ...trap/engines/engine.py.229: process: 0.83241 | load: 0.00308 [I 2022-10-09 09:45:17] ...trap/engines/engine.py.230: loss: 0.14752 [I 2022-10-09 09:45:17] ...optimizers/trijoint.py.151: total_norm: 15.131127 [I 2022-10-09 09:45:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 320/2304 [I 2022-10-09 09:45:26] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:45 [I 2022-10-09 09:45:26] ...trap/engines/engine.py.229: process: 0.82569 | load: 0.00302 [I 2022-10-09 09:45:26] ...trap/engines/engine.py.230: loss: 0.18289 [I 2022-10-09 09:45:26] ...optimizers/trijoint.py.151: total_norm: 27.326273 [I 2022-10-09 09:45:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 330/2304 [I 2022-10-09 09:45:34] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:44 [I 2022-10-09 09:45:34] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00294 [I 2022-10-09 09:45:34] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-09 09:45:34] ...optimizers/trijoint.py.151: total_norm: 11.648221 [I 2022-10-09 09:45:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 340/2304 [I 2022-10-09 09:45:43] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:25 [I 2022-10-09 09:45:43] ...trap/engines/engine.py.229: process: 0.83092 | load: 0.00309 [I 2022-10-09 09:45:43] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-09 09:45:43] ...optimizers/trijoint.py.151: total_norm: 21.494751 [I 2022-10-09 09:45:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 350/2304 [I 2022-10-09 09:45:51] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:34 [I 2022-10-09 09:45:51] ...trap/engines/engine.py.229: process: 0.85743 | load: 0.00291 [I 2022-10-09 09:45:51] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-09 09:45:51] ...optimizers/trijoint.py.151: total_norm: 15.875475 [I 2022-10-09 09:46:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 360/2304 [I 2022-10-09 09:46:00] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:17 [I 2022-10-09 09:46:00] ...trap/engines/engine.py.229: process: 0.84988 | load: 0.00292 [I 2022-10-09 09:46:00] ...trap/engines/engine.py.230: loss: 0.18039 [I 2022-10-09 09:46:00] ...optimizers/trijoint.py.151: total_norm: 20.487942 [I 2022-10-09 09:46:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 370/2304 [I 2022-10-09 09:46:08] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:12 [I 2022-10-09 09:46:08] ...trap/engines/engine.py.229: process: 0.83391 | load: 0.00292 [I 2022-10-09 09:46:08] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-09 09:46:08] ...optimizers/trijoint.py.151: total_norm: 13.497053 [I 2022-10-09 09:46:16] ...trap/engines/engine.py.224: train: epoch 36 | batch 380/2304 [I 2022-10-09 09:46:16] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:01 [I 2022-10-09 09:46:16] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00296 [I 2022-10-09 09:46:16] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-09 09:46:16] ...optimizers/trijoint.py.151: total_norm: 11.547121 [I 2022-10-09 09:46:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 390/2304 [I 2022-10-09 09:46:25] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:51 [I 2022-10-09 09:46:25] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00296 [I 2022-10-09 09:46:25] ...trap/engines/engine.py.230: loss: 0.12291 [I 2022-10-09 09:46:25] ...optimizers/trijoint.py.151: total_norm: 19.684038 [I 2022-10-09 09:46:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 400/2304 [I 2022-10-09 09:46:33] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:38 [I 2022-10-09 09:46:33] ...trap/engines/engine.py.229: process: 0.83022 | load: 0.00289 [I 2022-10-09 09:46:33] ...trap/engines/engine.py.230: loss: 0.20324 [I 2022-10-09 09:46:33] ...optimizers/trijoint.py.151: total_norm: 23.900135 [I 2022-10-09 09:46:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 410/2304 [I 2022-10-09 09:46:42] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:29 [I 2022-10-09 09:46:42] ...trap/engines/engine.py.229: process: 0.83491 | load: 0.00290 [I 2022-10-09 09:46:42] ...trap/engines/engine.py.230: loss: 0.14008 [I 2022-10-09 09:46:42] ...optimizers/trijoint.py.151: total_norm: 11.029356 [I 2022-10-09 09:46:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 420/2304 [I 2022-10-09 09:46:50] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:20 [I 2022-10-09 09:46:50] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00291 [I 2022-10-09 09:46:50] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-09 09:46:50] ...optimizers/trijoint.py.151: total_norm: 14.654812 [I 2022-10-09 09:46:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 430/2304 [I 2022-10-09 09:46:58] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:11 [I 2022-10-09 09:46:58] ...trap/engines/engine.py.229: process: 0.83321 | load: 0.00293 [I 2022-10-09 09:46:58] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-09 09:46:58] ...optimizers/trijoint.py.151: total_norm: 23.950241 [I 2022-10-09 09:47:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 440/2304 [I 2022-10-09 09:47:07] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:09 [I 2022-10-09 09:47:07] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00298 [I 2022-10-09 09:47:07] ...trap/engines/engine.py.230: loss: 0.21403 [I 2022-10-09 09:47:07] ...optimizers/trijoint.py.151: total_norm: 19.669094 [I 2022-10-09 09:47:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 450/2304 [I 2022-10-09 09:47:15] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:10 [I 2022-10-09 09:47:15] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00295 [I 2022-10-09 09:47:15] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-09 09:47:15] ...optimizers/trijoint.py.151: total_norm: 9.064718 [I 2022-10-09 09:47:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 460/2304 [I 2022-10-09 09:47:24] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:46 [I 2022-10-09 09:47:24] ...trap/engines/engine.py.229: process: 0.83578 | load: 0.00292 [I 2022-10-09 09:47:24] ...trap/engines/engine.py.230: loss: 0.14534 [I 2022-10-09 09:47:24] ...optimizers/trijoint.py.151: total_norm: 16.994070 [I 2022-10-09 09:47:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 470/2304 [I 2022-10-09 09:47:32] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:36 [I 2022-10-09 09:47:32] ...trap/engines/engine.py.229: process: 0.83325 | load: 0.00292 [I 2022-10-09 09:47:32] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-09 09:47:32] ...optimizers/trijoint.py.151: total_norm: 21.798998 [I 2022-10-09 09:47:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 480/2304 [I 2022-10-09 09:47:41] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:35 [I 2022-10-09 09:47:41] ...trap/engines/engine.py.229: process: 0.84832 | load: 0.00294 [I 2022-10-09 09:47:41] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-09 09:47:41] ...optimizers/trijoint.py.151: total_norm: 10.412562 [I 2022-10-09 09:47:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 490/2304 [I 2022-10-09 09:47:49] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:21 [I 2022-10-09 09:47:49] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00296 [I 2022-10-09 09:47:49] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-09 09:47:49] ...optimizers/trijoint.py.151: total_norm: 13.370483 [I 2022-10-09 09:47:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 500/2304 [I 2022-10-09 09:47:57] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:13 [I 2022-10-09 09:47:57] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00295 [I 2022-10-09 09:47:57] ...trap/engines/engine.py.230: loss: 0.22530 [I 2022-10-09 09:47:57] ...optimizers/trijoint.py.151: total_norm: 12.612022 [I 2022-10-09 09:48:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 510/2304 [I 2022-10-09 09:48:06] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:14 [I 2022-10-09 09:48:06] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00291 [I 2022-10-09 09:48:06] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-09 09:48:06] ...optimizers/trijoint.py.151: total_norm: 23.948332 [I 2022-10-09 09:48:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 520/2304 [I 2022-10-09 09:48:14] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:03 [I 2022-10-09 09:48:14] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00291 [I 2022-10-09 09:48:14] ...trap/engines/engine.py.230: loss: 0.19981 [I 2022-10-09 09:48:14] ...optimizers/trijoint.py.151: total_norm: 20.074455 [I 2022-10-09 09:48:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 530/2304 [I 2022-10-09 09:48:23] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:54 [I 2022-10-09 09:48:23] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00292 [I 2022-10-09 09:48:23] ...trap/engines/engine.py.230: loss: 0.16085 [I 2022-10-09 09:48:23] ...optimizers/trijoint.py.151: total_norm: 9.094612 [I 2022-10-09 09:48:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 540/2304 [I 2022-10-09 09:48:31] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:44 [I 2022-10-09 09:48:31] ...trap/engines/engine.py.229: process: 0.83787 | load: 0.00293 [I 2022-10-09 09:48:31] ...trap/engines/engine.py.230: loss: 0.11004 [I 2022-10-09 09:48:31] ...optimizers/trijoint.py.151: total_norm: 10.831213 [I 2022-10-09 09:48:40] ...trap/engines/engine.py.224: train: epoch 36 | batch 550/2304 [I 2022-10-09 09:48:40] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:37 [I 2022-10-09 09:48:40] ...trap/engines/engine.py.229: process: 0.84242 | load: 0.00292 [I 2022-10-09 09:48:40] ...trap/engines/engine.py.230: loss: 0.19151 [I 2022-10-09 09:48:40] ...optimizers/trijoint.py.151: total_norm: 15.420721 [I 2022-10-09 09:48:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 560/2304 [I 2022-10-09 09:48:48] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:22 [I 2022-10-09 09:48:48] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00301 [I 2022-10-09 09:48:48] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-09 09:48:48] ...optimizers/trijoint.py.151: total_norm: 28.996618 [I 2022-10-09 09:48:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 570/2304 [I 2022-10-09 09:48:56] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:18 [I 2022-10-09 09:48:56] ...trap/engines/engine.py.229: process: 0.83541 | load: 0.00296 [I 2022-10-09 09:48:56] ...trap/engines/engine.py.230: loss: 0.14588 [I 2022-10-09 09:48:56] ...optimizers/trijoint.py.151: total_norm: 17.404802 [I 2022-10-09 09:49:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 580/2304 [I 2022-10-09 09:49:05] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:06 [I 2022-10-09 09:49:05] ...trap/engines/engine.py.229: process: 0.83281 | load: 0.00292 [I 2022-10-09 09:49:05] ...trap/engines/engine.py.230: loss: 0.16692 [I 2022-10-09 09:49:05] ...optimizers/trijoint.py.151: total_norm: 16.400700 [I 2022-10-09 09:49:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 590/2304 [I 2022-10-09 09:49:13] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:06 [I 2022-10-09 09:49:13] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00291 [I 2022-10-09 09:49:13] ...trap/engines/engine.py.230: loss: 0.09818 [I 2022-10-09 09:49:13] ...optimizers/trijoint.py.151: total_norm: 8.567159 [I 2022-10-09 09:49:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 600/2304 [I 2022-10-09 09:49:22] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:49 [I 2022-10-09 09:49:22] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00303 [I 2022-10-09 09:49:22] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-09 09:49:22] ...optimizers/trijoint.py.151: total_norm: 14.156736 [I 2022-10-09 09:49:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 610/2304 [I 2022-10-09 09:49:30] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:46 [I 2022-10-09 09:49:30] ...trap/engines/engine.py.229: process: 0.83765 | load: 0.00294 [I 2022-10-09 09:49:30] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-09 09:49:30] ...optimizers/trijoint.py.151: total_norm: 19.986217 [I 2022-10-09 09:49:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 620/2304 [I 2022-10-09 09:49:38] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:32 [I 2022-10-09 09:49:38] ...trap/engines/engine.py.229: process: 0.82891 | load: 0.00299 [I 2022-10-09 09:49:38] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-09 09:49:38] ...optimizers/trijoint.py.151: total_norm: 14.240922 [I 2022-10-09 09:49:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 630/2304 [I 2022-10-09 09:49:47] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:28 [I 2022-10-09 09:49:47] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00296 [I 2022-10-09 09:49:47] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-09 09:49:47] ...optimizers/trijoint.py.151: total_norm: 14.185116 [I 2022-10-09 09:49:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 640/2304 [I 2022-10-09 09:49:55] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:17 [I 2022-10-09 09:49:55] ...trap/engines/engine.py.229: process: 0.82930 | load: 0.00292 [I 2022-10-09 09:49:55] ...trap/engines/engine.py.230: loss: 0.15343 [I 2022-10-09 09:49:55] ...optimizers/trijoint.py.151: total_norm: 23.368692 [I 2022-10-09 09:50:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 650/2304 [I 2022-10-09 09:50:04] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:25 [I 2022-10-09 09:50:04] ...trap/engines/engine.py.229: process: 0.87422 | load: 0.00292 [I 2022-10-09 09:50:04] ...trap/engines/engine.py.230: loss: 0.17747 [I 2022-10-09 09:50:04] ...optimizers/trijoint.py.151: total_norm: 18.809958 [I 2022-10-09 09:50:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 660/2304 [I 2022-10-09 09:50:12] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:02 [I 2022-10-09 09:50:12] ...trap/engines/engine.py.229: process: 0.83853 | load: 0.00294 [I 2022-10-09 09:50:12] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-09 09:50:12] ...optimizers/trijoint.py.151: total_norm: 20.109163 [I 2022-10-09 09:50:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 670/2304 [I 2022-10-09 09:50:21] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:59 [I 2022-10-09 09:50:21] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00295 [I 2022-10-09 09:50:21] ...trap/engines/engine.py.230: loss: 0.16349 [I 2022-10-09 09:50:21] ...optimizers/trijoint.py.151: total_norm: 22.794966 [I 2022-10-09 09:50:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 680/2304 [I 2022-10-09 09:50:29] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:50 [I 2022-10-09 09:50:29] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00292 [I 2022-10-09 09:50:29] ...trap/engines/engine.py.230: loss: 0.25720 [I 2022-10-09 09:50:29] ...optimizers/trijoint.py.151: total_norm: 20.305555 [I 2022-10-09 09:50:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 690/2304 [I 2022-10-09 09:50:38] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:38 [I 2022-10-09 09:50:38] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00292 [I 2022-10-09 09:50:38] ...trap/engines/engine.py.230: loss: 0.15666 [I 2022-10-09 09:50:38] ...optimizers/trijoint.py.151: total_norm: 13.891843 [I 2022-10-09 09:50:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 700/2304 [I 2022-10-09 09:50:46] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:26 [I 2022-10-09 09:50:46] ...trap/engines/engine.py.229: process: 0.83027 | load: 0.00298 [I 2022-10-09 09:50:46] ...trap/engines/engine.py.230: loss: 0.18581 [I 2022-10-09 09:50:46] ...optimizers/trijoint.py.151: total_norm: 26.536352 [I 2022-10-09 09:50:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 710/2304 [I 2022-10-09 09:50:54] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:17 [I 2022-10-09 09:50:54] ...trap/engines/engine.py.229: process: 0.83702 | load: 0.00290 [I 2022-10-09 09:50:54] ...trap/engines/engine.py.230: loss: 0.10302 [I 2022-10-09 09:50:54] ...optimizers/trijoint.py.151: total_norm: 23.674536 [I 2022-10-09 09:51:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 720/2304 [I 2022-10-09 09:51:03] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:13 [I 2022-10-09 09:51:03] ...trap/engines/engine.py.229: process: 0.83183 | load: 0.00292 [I 2022-10-09 09:51:03] ...trap/engines/engine.py.230: loss: 0.25587 [I 2022-10-09 09:51:03] ...optimizers/trijoint.py.151: total_norm: 35.727806 [I 2022-10-09 09:51:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 730/2304 [I 2022-10-09 09:51:11] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:13 [I 2022-10-09 09:51:11] ...trap/engines/engine.py.229: process: 0.84017 | load: 0.00291 [I 2022-10-09 09:51:11] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-09 09:51:11] ...optimizers/trijoint.py.151: total_norm: 22.416910 [I 2022-10-09 09:51:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 740/2304 [I 2022-10-09 09:51:20] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:47 [I 2022-10-09 09:51:20] ...trap/engines/engine.py.229: process: 0.82666 | load: 0.00297 [I 2022-10-09 09:51:20] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-09 09:51:20] ...optimizers/trijoint.py.151: total_norm: 10.013061 [I 2022-10-09 09:51:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 750/2304 [I 2022-10-09 09:51:28] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:50 [I 2022-10-09 09:51:28] ...trap/engines/engine.py.229: process: 0.84774 | load: 0.00291 [I 2022-10-09 09:51:28] ...trap/engines/engine.py.230: loss: 0.15935 [I 2022-10-09 09:51:28] ...optimizers/trijoint.py.151: total_norm: 18.709162 [I 2022-10-09 09:51:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 760/2304 [I 2022-10-09 09:51:37] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:41 [I 2022-10-09 09:51:37] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00291 [I 2022-10-09 09:51:37] ...trap/engines/engine.py.230: loss: 0.17572 [I 2022-10-09 09:51:37] ...optimizers/trijoint.py.151: total_norm: 19.080173 [I 2022-10-09 09:51:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 770/2304 [I 2022-10-09 09:51:45] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:38 [I 2022-10-09 09:51:45] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00292 [I 2022-10-09 09:51:45] ...trap/engines/engine.py.230: loss: 0.14294 [I 2022-10-09 09:51:45] ...optimizers/trijoint.py.151: total_norm: 20.042080 [I 2022-10-09 09:51:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 780/2304 [I 2022-10-09 09:51:54] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:29 [I 2022-10-09 09:51:54] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00292 [I 2022-10-09 09:51:54] ...trap/engines/engine.py.230: loss: 0.21814 [I 2022-10-09 09:51:54] ...optimizers/trijoint.py.151: total_norm: 26.743587 [I 2022-10-09 09:52:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 790/2304 [I 2022-10-09 09:52:02] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:13 [I 2022-10-09 09:52:02] ...trap/engines/engine.py.229: process: 0.83860 | load: 0.00293 [I 2022-10-09 09:52:02] ...trap/engines/engine.py.230: loss: 0.10817 [I 2022-10-09 09:52:02] ...optimizers/trijoint.py.151: total_norm: 11.201423 [I 2022-10-09 09:52:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 800/2304 [I 2022-10-09 09:52:10] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:03 [I 2022-10-09 09:52:10] ...trap/engines/engine.py.229: process: 0.83637 | load: 0.00290 [I 2022-10-09 09:52:10] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-09 09:52:10] ...optimizers/trijoint.py.151: total_norm: 53.780041 [I 2022-10-09 09:52:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 810/2304 [I 2022-10-09 09:52:19] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:03 [I 2022-10-09 09:52:19] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00291 [I 2022-10-09 09:52:19] ...trap/engines/engine.py.230: loss: 0.21212 [I 2022-10-09 09:52:19] ...optimizers/trijoint.py.151: total_norm: 12.007126 [I 2022-10-09 09:52:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 820/2304 [I 2022-10-09 09:52:27] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:41 [I 2022-10-09 09:52:27] ...trap/engines/engine.py.229: process: 0.83190 | load: 0.00293 [I 2022-10-09 09:52:27] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-09 09:52:27] ...optimizers/trijoint.py.151: total_norm: 10.753003 [I 2022-10-09 09:52:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 830/2304 [I 2022-10-09 09:52:36] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:30 [I 2022-10-09 09:52:36] ...trap/engines/engine.py.229: process: 0.83095 | load: 0.00298 [I 2022-10-09 09:52:36] ...trap/engines/engine.py.230: loss: 0.18288 [I 2022-10-09 09:52:36] ...optimizers/trijoint.py.151: total_norm: 25.904285 [I 2022-10-09 09:52:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 840/2304 [I 2022-10-09 09:52:44] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:29 [I 2022-10-09 09:52:44] ...trap/engines/engine.py.229: process: 0.85772 | load: 0.00288 [I 2022-10-09 09:52:44] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-09 09:52:44] ...optimizers/trijoint.py.151: total_norm: 11.255796 [I 2022-10-09 09:52:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 850/2304 [I 2022-10-09 09:52:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:22 [I 2022-10-09 09:52:52] ...trap/engines/engine.py.229: process: 0.84919 | load: 0.00292 [I 2022-10-09 09:52:52] ...trap/engines/engine.py.230: loss: 0.25473 [I 2022-10-09 09:52:52] ...optimizers/trijoint.py.151: total_norm: 15.917939 [I 2022-10-09 09:53:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 860/2304 [I 2022-10-09 09:53:01] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:10 [I 2022-10-09 09:53:01] ...trap/engines/engine.py.229: process: 0.83692 | load: 0.00292 [I 2022-10-09 09:53:01] ...trap/engines/engine.py.230: loss: 0.21108 [I 2022-10-09 09:53:01] ...optimizers/trijoint.py.151: total_norm: 31.217798 [I 2022-10-09 09:53:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 870/2304 [I 2022-10-09 09:53:09] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:09 [I 2022-10-09 09:53:09] ...trap/engines/engine.py.229: process: 0.83600 | load: 0.00299 [I 2022-10-09 09:53:09] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-09 09:53:09] ...optimizers/trijoint.py.151: total_norm: 12.604219 [I 2022-10-09 09:53:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 880/2304 [I 2022-10-09 09:53:18] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:59 [I 2022-10-09 09:53:18] ...trap/engines/engine.py.229: process: 0.84189 | load: 0.00295 [I 2022-10-09 09:53:18] ...trap/engines/engine.py.230: loss: 0.24756 [I 2022-10-09 09:53:18] ...optimizers/trijoint.py.151: total_norm: 15.977633 [I 2022-10-09 09:53:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 890/2304 [I 2022-10-09 09:53:26] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:57 [I 2022-10-09 09:53:26] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00292 [I 2022-10-09 09:53:26] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-09 09:53:26] ...optimizers/trijoint.py.151: total_norm: 11.641552 [I 2022-10-09 09:53:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 900/2304 [I 2022-10-09 09:53:35] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:37 [I 2022-10-09 09:53:35] ...trap/engines/engine.py.229: process: 0.83015 | load: 0.00292 [I 2022-10-09 09:53:35] ...trap/engines/engine.py.230: loss: 0.10740 [I 2022-10-09 09:53:35] ...optimizers/trijoint.py.151: total_norm: 10.717551 [I 2022-10-09 09:53:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 910/2304 [I 2022-10-09 09:53:43] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:34 [I 2022-10-09 09:53:43] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00294 [I 2022-10-09 09:53:43] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-09 09:53:43] ...optimizers/trijoint.py.151: total_norm: 15.588355 [I 2022-10-09 09:53:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 920/2304 [I 2022-10-09 09:53:52] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:26 [I 2022-10-09 09:53:52] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00294 [I 2022-10-09 09:53:52] ...trap/engines/engine.py.230: loss: 0.13446 [I 2022-10-09 09:53:52] ...optimizers/trijoint.py.151: total_norm: 16.667347 [I 2022-10-09 09:54:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 930/2304 [I 2022-10-09 09:54:00] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:13 [I 2022-10-09 09:54:00] ...trap/engines/engine.py.229: process: 0.83411 | load: 0.00303 [I 2022-10-09 09:54:00] ...trap/engines/engine.py.230: loss: 0.17176 [I 2022-10-09 09:54:00] ...optimizers/trijoint.py.151: total_norm: 35.316826 [I 2022-10-09 09:54:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 940/2304 [I 2022-10-09 09:54:08] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:06 [I 2022-10-09 09:54:08] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00289 [I 2022-10-09 09:54:08] ...trap/engines/engine.py.230: loss: 0.21108 [I 2022-10-09 09:54:08] ...optimizers/trijoint.py.151: total_norm: 14.146799 [I 2022-10-09 09:54:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 950/2304 [I 2022-10-09 09:54:17] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:02 [I 2022-10-09 09:54:17] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00291 [I 2022-10-09 09:54:17] ...trap/engines/engine.py.230: loss: 0.18467 [I 2022-10-09 09:54:17] ...optimizers/trijoint.py.151: total_norm: 19.324486 [I 2022-10-09 09:54:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 960/2304 [I 2022-10-09 09:54:43] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 1:23:30 [I 2022-10-09 09:54:43] ...trap/engines/engine.py.229: process: 0.84014 | load: 0.00294 [I 2022-10-09 09:54:43] ...trap/engines/engine.py.230: loss: 0.15138 [I 2022-10-09 09:54:43] ...optimizers/trijoint.py.151: total_norm: 13.630836 [I 2022-10-09 09:54:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 970/2304 [I 2022-10-09 09:54:52] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:25:36 [I 2022-10-09 09:54:52] ...trap/engines/engine.py.229: process: 0.84060 | load: 0.00290 [I 2022-10-09 09:54:52] ...trap/engines/engine.py.230: loss: 0.10358 [I 2022-10-09 09:54:52] ...optimizers/trijoint.py.151: total_norm: 12.939699 [I 2022-10-09 09:55:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 980/2304 [I 2022-10-09 09:55:00] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:19:17 [I 2022-10-09 09:55:00] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00301 [I 2022-10-09 09:55:00] ...trap/engines/engine.py.230: loss: 0.16734 [I 2022-10-09 09:55:00] ...optimizers/trijoint.py.151: total_norm: 8.986035 [I 2022-10-09 09:55:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 990/2304 [I 2022-10-09 09:55:08] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:31 [I 2022-10-09 09:55:08] ...trap/engines/engine.py.229: process: 0.84825 | load: 0.00292 [I 2022-10-09 09:55:08] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-09 09:55:08] ...optimizers/trijoint.py.151: total_norm: 8.536860 [I 2022-10-09 09:55:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 1000/2304 [I 2022-10-09 09:55:17] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:18 [I 2022-10-09 09:55:17] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00292 [I 2022-10-09 09:55:17] ...trap/engines/engine.py.230: loss: 0.21203 [I 2022-10-09 09:55:17] ...optimizers/trijoint.py.151: total_norm: 56.081635 [I 2022-10-09 09:55:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 1010/2304 [I 2022-10-09 09:55:25] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:08 [I 2022-10-09 09:55:25] ...trap/engines/engine.py.229: process: 0.83832 | load: 0.00295 [I 2022-10-09 09:55:25] ...trap/engines/engine.py.230: loss: 0.14187 [I 2022-10-09 09:55:25] ...optimizers/trijoint.py.151: total_norm: 65.492592 [I 2022-10-09 09:55:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 1020/2304 [I 2022-10-09 09:55:34] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:53 [I 2022-10-09 09:55:34] ...trap/engines/engine.py.229: process: 0.83025 | load: 0.00290 [I 2022-10-09 09:55:34] ...trap/engines/engine.py.230: loss: 0.19396 [I 2022-10-09 09:55:34] ...optimizers/trijoint.py.151: total_norm: 9.785336 [I 2022-10-09 09:55:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 1030/2304 [I 2022-10-09 09:55:42] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:52 [I 2022-10-09 09:55:42] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00291 [I 2022-10-09 09:55:42] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-09 09:55:42] ...optimizers/trijoint.py.151: total_norm: 18.529261 [I 2022-10-09 09:55:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 1040/2304 [I 2022-10-09 09:55:51] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:46 [I 2022-10-09 09:55:51] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00295 [I 2022-10-09 09:55:51] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-09 09:55:51] ...optimizers/trijoint.py.151: total_norm: 11.567668 [I 2022-10-09 09:55:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 1050/2304 [I 2022-10-09 09:55:59] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:30 [I 2022-10-09 09:55:59] ...trap/engines/engine.py.229: process: 0.83634 | load: 0.00292 [I 2022-10-09 09:55:59] ...trap/engines/engine.py.230: loss: 0.10912 [I 2022-10-09 09:55:59] ...optimizers/trijoint.py.151: total_norm: 11.745873 [I 2022-10-09 09:56:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 1060/2304 [I 2022-10-09 09:56:07] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:32 [I 2022-10-09 09:56:07] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00296 [I 2022-10-09 09:56:07] ...trap/engines/engine.py.230: loss: 0.20617 [I 2022-10-09 09:56:07] ...optimizers/trijoint.py.151: total_norm: 16.746826 [I 2022-10-09 09:56:16] ...trap/engines/engine.py.224: train: epoch 36 | batch 1070/2304 [I 2022-10-09 09:56:16] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:28 [I 2022-10-09 09:56:16] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00371 [I 2022-10-09 09:56:16] ...trap/engines/engine.py.230: loss: 0.15993 [I 2022-10-09 09:56:16] ...optimizers/trijoint.py.151: total_norm: 13.595350 [I 2022-10-09 09:56:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 1080/2304 [I 2022-10-09 09:56:24] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:11 [I 2022-10-09 09:56:24] ...trap/engines/engine.py.229: process: 0.82628 | load: 0.00306 [I 2022-10-09 09:56:24] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-09 09:56:24] ...optimizers/trijoint.py.151: total_norm: 14.526328 [I 2022-10-09 09:56:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 1090/2304 [I 2022-10-09 09:56:33] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:05 [I 2022-10-09 09:56:33] ...trap/engines/engine.py.229: process: 0.82891 | load: 0.00300 [I 2022-10-09 09:56:33] ...trap/engines/engine.py.230: loss: 0.15372 [I 2022-10-09 09:56:33] ...optimizers/trijoint.py.151: total_norm: 14.841077 [I 2022-10-09 09:56:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 1100/2304 [I 2022-10-09 09:56:41] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:58 [I 2022-10-09 09:56:41] ...trap/engines/engine.py.229: process: 0.83487 | load: 0.00293 [I 2022-10-09 09:56:41] ...trap/engines/engine.py.230: loss: 0.20237 [I 2022-10-09 09:56:41] ...optimizers/trijoint.py.151: total_norm: 18.214676 [I 2022-10-09 09:56:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 1110/2304 [I 2022-10-09 09:56:50] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:49 [I 2022-10-09 09:56:50] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00299 [I 2022-10-09 09:56:50] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-09 09:56:50] ...optimizers/trijoint.py.151: total_norm: 20.735809 [I 2022-10-09 09:56:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 1120/2304 [I 2022-10-09 09:56:58] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:39 [I 2022-10-09 09:56:58] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00297 [I 2022-10-09 09:56:58] ...trap/engines/engine.py.230: loss: 0.24326 [I 2022-10-09 09:56:58] ...optimizers/trijoint.py.151: total_norm: 24.290705 [I 2022-10-09 09:57:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 1130/2304 [I 2022-10-09 09:57:07] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:36 [I 2022-10-09 09:57:07] ...trap/engines/engine.py.229: process: 0.85961 | load: 0.00296 [I 2022-10-09 09:57:07] ...trap/engines/engine.py.230: loss: 0.11248 [I 2022-10-09 09:57:07] ...optimizers/trijoint.py.151: total_norm: 23.612415 [I 2022-10-09 09:57:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 1140/2304 [I 2022-10-09 09:57:15] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:16 [I 2022-10-09 09:57:15] ...trap/engines/engine.py.229: process: 0.82379 | load: 0.00294 [I 2022-10-09 09:57:15] ...trap/engines/engine.py.230: loss: 0.17589 [I 2022-10-09 09:57:15] ...optimizers/trijoint.py.151: total_norm: 21.615717 [I 2022-10-09 09:57:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 1150/2304 [I 2022-10-09 09:57:24] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:14 [I 2022-10-09 09:57:24] ...trap/engines/engine.py.229: process: 0.85004 | load: 0.00299 [I 2022-10-09 09:57:24] ...trap/engines/engine.py.230: loss: 0.15689 [I 2022-10-09 09:57:24] ...optimizers/trijoint.py.151: total_norm: 22.560375 [I 2022-10-09 09:57:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 1160/2304 [I 2022-10-09 09:57:32] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:06 [I 2022-10-09 09:57:32] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00293 [I 2022-10-09 09:57:32] ...trap/engines/engine.py.230: loss: 0.09733 [I 2022-10-09 09:57:32] ...optimizers/trijoint.py.151: total_norm: 11.809229 [I 2022-10-09 09:57:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 1170/2304 [I 2022-10-09 09:57:41] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:59 [I 2022-10-09 09:57:41] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00294 [I 2022-10-09 09:57:41] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-09 09:57:41] ...optimizers/trijoint.py.151: total_norm: 19.172680 [I 2022-10-09 09:57:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 1180/2304 [I 2022-10-09 09:57:49] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:51 [I 2022-10-09 09:57:49] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00298 [I 2022-10-09 09:57:49] ...trap/engines/engine.py.230: loss: 0.14262 [I 2022-10-09 09:57:49] ...optimizers/trijoint.py.151: total_norm: 40.438175 [I 2022-10-09 09:57:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 1190/2304 [I 2022-10-09 09:57:57] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:37 [I 2022-10-09 09:57:57] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00294 [I 2022-10-09 09:57:57] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-09 09:57:57] ...optimizers/trijoint.py.151: total_norm: 12.557669 [I 2022-10-09 09:58:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 1200/2304 [I 2022-10-09 09:58:06] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:25 [I 2022-10-09 09:58:06] ...trap/engines/engine.py.229: process: 0.82446 | load: 0.00293 [I 2022-10-09 09:58:06] ...trap/engines/engine.py.230: loss: 0.11393 [I 2022-10-09 09:58:06] ...optimizers/trijoint.py.151: total_norm: 7.450643 [I 2022-10-09 09:58:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 1210/2304 [I 2022-10-09 09:58:14] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:23 [I 2022-10-09 09:58:14] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00297 [I 2022-10-09 09:58:14] ...trap/engines/engine.py.230: loss: 0.17361 [I 2022-10-09 09:58:14] ...optimizers/trijoint.py.151: total_norm: 26.938047 [I 2022-10-09 09:58:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 1220/2304 [I 2022-10-09 09:58:23] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:12 [I 2022-10-09 09:58:23] ...trap/engines/engine.py.229: process: 0.83123 | load: 0.00300 [I 2022-10-09 09:58:23] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-09 09:58:23] ...optimizers/trijoint.py.151: total_norm: 11.695262 [I 2022-10-09 09:58:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1230/2304 [I 2022-10-09 09:58:31] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:05 [I 2022-10-09 09:58:31] ...trap/engines/engine.py.229: process: 0.83636 | load: 0.00289 [I 2022-10-09 09:58:31] ...trap/engines/engine.py.230: loss: 0.19345 [I 2022-10-09 09:58:31] ...optimizers/trijoint.py.151: total_norm: 23.755896 [I 2022-10-09 09:58:40] ...trap/engines/engine.py.224: train: epoch 36 | batch 1240/2304 [I 2022-10-09 09:58:40] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:14:54 [I 2022-10-09 09:58:40] ...trap/engines/engine.py.229: process: 0.83410 | load: 0.00290 [I 2022-10-09 09:58:40] ...trap/engines/engine.py.230: loss: 0.20313 [I 2022-10-09 09:58:40] ...optimizers/trijoint.py.151: total_norm: 38.237835 [I 2022-10-09 09:58:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 1250/2304 [I 2022-10-09 09:58:48] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:46 [I 2022-10-09 09:58:48] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00289 [I 2022-10-09 09:58:48] ...trap/engines/engine.py.230: loss: 0.29242 [I 2022-10-09 09:58:48] ...optimizers/trijoint.py.151: total_norm: 15.308633 [I 2022-10-09 09:58:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 1260/2304 [I 2022-10-09 09:58:57] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:14:39 [I 2022-10-09 09:58:57] ...trap/engines/engine.py.229: process: 0.82633 | load: 0.00290 [I 2022-10-09 09:58:57] ...trap/engines/engine.py.230: loss: 0.15321 [I 2022-10-09 09:58:57] ...optimizers/trijoint.py.151: total_norm: 14.744326 [I 2022-10-09 09:59:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 1270/2304 [I 2022-10-09 09:59:05] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:32 [I 2022-10-09 09:59:05] ...trap/engines/engine.py.229: process: 0.83895 | load: 0.00292 [I 2022-10-09 09:59:05] ...trap/engines/engine.py.230: loss: 0.18921 [I 2022-10-09 09:59:05] ...optimizers/trijoint.py.151: total_norm: 14.371040 [I 2022-10-09 09:59:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 1280/2304 [I 2022-10-09 09:59:13] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:22 [I 2022-10-09 09:59:13] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00296 [I 2022-10-09 09:59:13] ...trap/engines/engine.py.230: loss: 0.18535 [I 2022-10-09 09:59:13] ...optimizers/trijoint.py.151: total_norm: 17.550575 [I 2022-10-09 09:59:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 1290/2304 [I 2022-10-09 09:59:22] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:35 [I 2022-10-09 09:59:22] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00293 [I 2022-10-09 09:59:22] ...trap/engines/engine.py.230: loss: 0.12258 [I 2022-10-09 09:59:22] ...optimizers/trijoint.py.151: total_norm: 12.535210 [I 2022-10-09 09:59:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1300/2304 [I 2022-10-09 09:59:31] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:14:08 [I 2022-10-09 09:59:31] ...trap/engines/engine.py.229: process: 0.83957 | load: 0.00291 [I 2022-10-09 09:59:31] ...trap/engines/engine.py.230: loss: 0.20334 [I 2022-10-09 09:59:31] ...optimizers/trijoint.py.151: total_norm: 16.327808 [I 2022-10-09 09:59:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 1310/2304 [I 2022-10-09 09:59:39] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:13:56 [I 2022-10-09 09:59:39] ...trap/engines/engine.py.229: process: 0.83871 | load: 0.00293 [I 2022-10-09 09:59:39] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-09 09:59:39] ...optimizers/trijoint.py.151: total_norm: 15.054602 [I 2022-10-09 09:59:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 1320/2304 [I 2022-10-09 09:59:47] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:13:45 [I 2022-10-09 09:59:47] ...trap/engines/engine.py.229: process: 0.81974 | load: 0.00294 [I 2022-10-09 09:59:47] ...trap/engines/engine.py.230: loss: 0.18888 [I 2022-10-09 09:59:47] ...optimizers/trijoint.py.151: total_norm: 11.522264 [I 2022-10-09 09:59:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 1330/2304 [I 2022-10-09 09:59:56] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:13:39 [I 2022-10-09 09:59:56] ...trap/engines/engine.py.229: process: 0.83874 | load: 0.00305 [I 2022-10-09 09:59:56] ...trap/engines/engine.py.230: loss: 0.14734 [I 2022-10-09 09:59:56] ...optimizers/trijoint.py.151: total_norm: 17.909336 [I 2022-10-09 10:00:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 1340/2304 [I 2022-10-09 10:00:04] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:13:34 [I 2022-10-09 10:00:04] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00296 [I 2022-10-09 10:00:04] ...trap/engines/engine.py.230: loss: 0.16280 [I 2022-10-09 10:00:04] ...optimizers/trijoint.py.151: total_norm: 19.062225 [I 2022-10-09 10:00:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 1350/2304 [I 2022-10-09 10:00:13] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:24 [I 2022-10-09 10:00:13] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00288 [I 2022-10-09 10:00:13] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-09 10:00:13] ...optimizers/trijoint.py.151: total_norm: 18.601721 [I 2022-10-09 10:00:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 1360/2304 [I 2022-10-09 10:00:21] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:14 [I 2022-10-09 10:00:21] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00292 [I 2022-10-09 10:00:21] ...trap/engines/engine.py.230: loss: 0.22587 [I 2022-10-09 10:00:21] ...optimizers/trijoint.py.151: total_norm: 17.863148 [I 2022-10-09 10:00:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 1370/2304 [I 2022-10-09 10:00:30] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:04 [I 2022-10-09 10:00:30] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00293 [I 2022-10-09 10:00:30] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-09 10:00:30] ...optimizers/trijoint.py.151: total_norm: 14.001512 [I 2022-10-09 10:00:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 1380/2304 [I 2022-10-09 10:00:38] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:12:59 [I 2022-10-09 10:00:38] ...trap/engines/engine.py.229: process: 0.84623 | load: 0.00293 [I 2022-10-09 10:00:38] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-09 10:00:38] ...optimizers/trijoint.py.151: total_norm: 12.190500 [I 2022-10-09 10:00:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 1390/2304 [I 2022-10-09 10:00:46] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:12:49 [I 2022-10-09 10:00:46] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00291 [I 2022-10-09 10:00:46] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-09 10:00:46] ...optimizers/trijoint.py.151: total_norm: 7.141039 [I 2022-10-09 10:00:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 1400/2304 [I 2022-10-09 10:00:55] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:12:42 [I 2022-10-09 10:00:55] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00293 [I 2022-10-09 10:00:55] ...trap/engines/engine.py.230: loss: 0.11597 [I 2022-10-09 10:00:55] ...optimizers/trijoint.py.151: total_norm: 13.771094 [I 2022-10-09 10:01:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 1410/2304 [I 2022-10-09 10:01:03] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:33 [I 2022-10-09 10:01:03] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00303 [I 2022-10-09 10:01:03] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-09 10:01:03] ...optimizers/trijoint.py.151: total_norm: 9.477407 [I 2022-10-09 10:01:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 1420/2304 [I 2022-10-09 10:01:12] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:25 [I 2022-10-09 10:01:12] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00297 [I 2022-10-09 10:01:12] ...trap/engines/engine.py.230: loss: 0.15160 [I 2022-10-09 10:01:12] ...optimizers/trijoint.py.151: total_norm: 8.467381 [I 2022-10-09 10:01:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 1430/2304 [I 2022-10-09 10:01:20] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:11 [I 2022-10-09 10:01:20] ...trap/engines/engine.py.229: process: 0.82951 | load: 0.00292 [I 2022-10-09 10:01:20] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-09 10:01:20] ...optimizers/trijoint.py.151: total_norm: 12.412164 [I 2022-10-09 10:01:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 1440/2304 [I 2022-10-09 10:01:29] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:09 [I 2022-10-09 10:01:29] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00293 [I 2022-10-09 10:01:29] ...trap/engines/engine.py.230: loss: 0.17384 [I 2022-10-09 10:01:29] ...optimizers/trijoint.py.151: total_norm: 28.996916 [I 2022-10-09 10:01:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 1450/2304 [I 2022-10-09 10:01:37] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:03 [I 2022-10-09 10:01:37] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00293 [I 2022-10-09 10:01:37] ...trap/engines/engine.py.230: loss: 0.13120 [I 2022-10-09 10:01:37] ...optimizers/trijoint.py.151: total_norm: 11.040249 [I 2022-10-09 10:01:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 1460/2304 [I 2022-10-09 10:01:46] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:11:49 [I 2022-10-09 10:01:46] ...trap/engines/engine.py.229: process: 0.83833 | load: 0.00304 [I 2022-10-09 10:01:46] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-09 10:01:46] ...optimizers/trijoint.py.151: total_norm: 7.382176 [I 2022-10-09 10:01:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 1470/2304 [I 2022-10-09 10:01:54] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:11:43 [I 2022-10-09 10:01:54] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00290 [I 2022-10-09 10:01:54] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-09 10:01:54] ...optimizers/trijoint.py.151: total_norm: 18.751436 [I 2022-10-09 10:02:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 1480/2304 [I 2022-10-09 10:02:02] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:11:33 [I 2022-10-09 10:02:02] ...trap/engines/engine.py.229: process: 0.83724 | load: 0.00293 [I 2022-10-09 10:02:02] ...trap/engines/engine.py.230: loss: 0.16785 [I 2022-10-09 10:02:02] ...optimizers/trijoint.py.151: total_norm: 12.015282 [I 2022-10-09 10:02:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 1490/2304 [I 2022-10-09 10:02:11] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:23 [I 2022-10-09 10:02:11] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00292 [I 2022-10-09 10:02:11] ...trap/engines/engine.py.230: loss: 0.22164 [I 2022-10-09 10:02:11] ...optimizers/trijoint.py.151: total_norm: 27.682861 [I 2022-10-09 10:02:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 1500/2304 [I 2022-10-09 10:02:19] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:19 [I 2022-10-09 10:02:19] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00294 [I 2022-10-09 10:02:19] ...trap/engines/engine.py.230: loss: 0.16085 [I 2022-10-09 10:02:19] ...optimizers/trijoint.py.151: total_norm: 11.591453 [I 2022-10-09 10:02:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 1510/2304 [I 2022-10-09 10:02:28] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:11:09 [I 2022-10-09 10:02:28] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00303 [I 2022-10-09 10:02:28] ...trap/engines/engine.py.230: loss: 0.13100 [I 2022-10-09 10:02:28] ...optimizers/trijoint.py.151: total_norm: 18.069592 [I 2022-10-09 10:02:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 1520/2304 [I 2022-10-09 10:02:36] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:11:04 [I 2022-10-09 10:02:36] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00291 [I 2022-10-09 10:02:36] ...trap/engines/engine.py.230: loss: 0.14556 [I 2022-10-09 10:02:36] ...optimizers/trijoint.py.151: total_norm: 10.779420 [I 2022-10-09 10:02:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 1530/2304 [I 2022-10-09 10:02:45] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:10:52 [I 2022-10-09 10:02:45] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00293 [I 2022-10-09 10:02:45] ...trap/engines/engine.py.230: loss: 0.09223 [I 2022-10-09 10:02:45] ...optimizers/trijoint.py.151: total_norm: 11.448285 [I 2022-10-09 10:02:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 1540/2304 [I 2022-10-09 10:02:53] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:10:44 [I 2022-10-09 10:02:53] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00293 [I 2022-10-09 10:02:53] ...trap/engines/engine.py.230: loss: 0.18728 [I 2022-10-09 10:02:53] ...optimizers/trijoint.py.151: total_norm: 20.441050 [I 2022-10-09 10:03:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 1550/2304 [I 2022-10-09 10:03:01] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:10:33 [I 2022-10-09 10:03:01] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00295 [I 2022-10-09 10:03:01] ...trap/engines/engine.py.230: loss: 0.19554 [I 2022-10-09 10:03:01] ...optimizers/trijoint.py.151: total_norm: 28.150019 [I 2022-10-09 10:03:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 1560/2304 [I 2022-10-09 10:03:10] ...trap/engines/engine.py.228: elapsed: 0:22:17 | left: 0:10:24 [I 2022-10-09 10:03:10] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00295 [I 2022-10-09 10:03:10] ...trap/engines/engine.py.230: loss: 0.14484 [I 2022-10-09 10:03:10] ...optimizers/trijoint.py.151: total_norm: 11.508591 [I 2022-10-09 10:03:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 1570/2304 [I 2022-10-09 10:03:18] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:10:17 [I 2022-10-09 10:03:18] ...trap/engines/engine.py.229: process: 0.82303 | load: 0.00290 [I 2022-10-09 10:03:18] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-09 10:03:18] ...optimizers/trijoint.py.151: total_norm: 11.663739 [I 2022-10-09 10:03:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 1580/2304 [I 2022-10-09 10:03:27] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:05 [I 2022-10-09 10:03:27] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00293 [I 2022-10-09 10:03:27] ...trap/engines/engine.py.230: loss: 0.20265 [I 2022-10-09 10:03:27] ...optimizers/trijoint.py.151: total_norm: 21.843151 [I 2022-10-09 10:03:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 1590/2304 [I 2022-10-09 10:03:35] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:09:59 [I 2022-10-09 10:03:35] ...trap/engines/engine.py.229: process: 0.83130 | load: 0.00290 [I 2022-10-09 10:03:35] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-09 10:03:35] ...optimizers/trijoint.py.151: total_norm: 20.071434 [I 2022-10-09 10:03:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 1600/2304 [I 2022-10-09 10:03:44] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:09:53 [I 2022-10-09 10:03:44] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00291 [I 2022-10-09 10:03:44] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-09 10:03:44] ...optimizers/trijoint.py.151: total_norm: 12.677692 [I 2022-10-09 10:03:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 1610/2304 [I 2022-10-09 10:03:52] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:09:43 [I 2022-10-09 10:03:52] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00299 [I 2022-10-09 10:03:52] ...trap/engines/engine.py.230: loss: 0.20440 [I 2022-10-09 10:03:52] ...optimizers/trijoint.py.151: total_norm: 28.355440 [I 2022-10-09 10:04:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 1620/2304 [I 2022-10-09 10:04:00] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:09:36 [I 2022-10-09 10:04:00] ...trap/engines/engine.py.229: process: 0.84895 | load: 0.00291 [I 2022-10-09 10:04:00] ...trap/engines/engine.py.230: loss: 0.17497 [I 2022-10-09 10:04:00] ...optimizers/trijoint.py.151: total_norm: 18.939486 [I 2022-10-09 10:04:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 1630/2304 [I 2022-10-09 10:04:09] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:28 [I 2022-10-09 10:04:09] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00291 [I 2022-10-09 10:04:09] ...trap/engines/engine.py.230: loss: 0.17783 [I 2022-10-09 10:04:09] ...optimizers/trijoint.py.151: total_norm: 27.647749 [I 2022-10-09 10:04:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 1640/2304 [I 2022-10-09 10:04:17] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:23 [I 2022-10-09 10:04:17] ...trap/engines/engine.py.229: process: 0.86192 | load: 0.00297 [I 2022-10-09 10:04:17] ...trap/engines/engine.py.230: loss: 0.16101 [I 2022-10-09 10:04:17] ...optimizers/trijoint.py.151: total_norm: 32.522911 [I 2022-10-09 10:04:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 1650/2304 [I 2022-10-09 10:04:26] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:12 [I 2022-10-09 10:04:26] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00289 [I 2022-10-09 10:04:26] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-09 10:04:26] ...optimizers/trijoint.py.151: total_norm: 16.407661 [I 2022-10-09 10:04:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 1660/2304 [I 2022-10-09 10:04:34] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:03 [I 2022-10-09 10:04:34] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00291 [I 2022-10-09 10:04:34] ...trap/engines/engine.py.230: loss: 0.24023 [I 2022-10-09 10:04:34] ...optimizers/trijoint.py.151: total_norm: 28.840515 [I 2022-10-09 10:04:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 1670/2304 [I 2022-10-09 10:04:43] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:08:53 [I 2022-10-09 10:04:43] ...trap/engines/engine.py.229: process: 0.83721 | load: 0.00294 [I 2022-10-09 10:04:43] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-09 10:04:43] ...optimizers/trijoint.py.151: total_norm: 12.160476 [I 2022-10-09 10:04:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 1680/2304 [I 2022-10-09 10:04:51] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:08:46 [I 2022-10-09 10:04:51] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00297 [I 2022-10-09 10:04:51] ...trap/engines/engine.py.230: loss: 0.16417 [I 2022-10-09 10:04:51] ...optimizers/trijoint.py.151: total_norm: 10.046370 [I 2022-10-09 10:05:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 1690/2304 [I 2022-10-09 10:05:00] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:36 [I 2022-10-09 10:05:00] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00294 [I 2022-10-09 10:05:00] ...trap/engines/engine.py.230: loss: 0.11982 [I 2022-10-09 10:05:00] ...optimizers/trijoint.py.151: total_norm: 10.880929 [I 2022-10-09 10:05:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 1700/2304 [I 2022-10-09 10:05:08] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:30 [I 2022-10-09 10:05:08] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00289 [I 2022-10-09 10:05:08] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-09 10:05:08] ...optimizers/trijoint.py.151: total_norm: 10.714828 [I 2022-10-09 10:05:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 1710/2304 [I 2022-10-09 10:05:17] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:22 [I 2022-10-09 10:05:17] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00292 [I 2022-10-09 10:05:17] ...trap/engines/engine.py.230: loss: 0.24923 [I 2022-10-09 10:05:17] ...optimizers/trijoint.py.151: total_norm: 19.066818 [I 2022-10-09 10:05:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 1720/2304 [I 2022-10-09 10:05:25] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:11 [I 2022-10-09 10:05:25] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00290 [I 2022-10-09 10:05:25] ...trap/engines/engine.py.230: loss: 0.11648 [I 2022-10-09 10:05:25] ...optimizers/trijoint.py.151: total_norm: 20.454149 [I 2022-10-09 10:05:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 1730/2304 [I 2022-10-09 10:05:33] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:02 [I 2022-10-09 10:05:33] ...trap/engines/engine.py.229: process: 0.83075 | load: 0.00293 [I 2022-10-09 10:05:33] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-09 10:05:33] ...optimizers/trijoint.py.151: total_norm: 37.966370 [I 2022-10-09 10:05:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 1740/2304 [I 2022-10-09 10:05:42] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:07:56 [I 2022-10-09 10:05:42] ...trap/engines/engine.py.229: process: 0.84344 | load: 0.00292 [I 2022-10-09 10:05:42] ...trap/engines/engine.py.230: loss: 0.12642 [I 2022-10-09 10:05:42] ...optimizers/trijoint.py.151: total_norm: 10.836834 [I 2022-10-09 10:05:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 1750/2304 [I 2022-10-09 10:05:50] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:07:46 [I 2022-10-09 10:05:50] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00294 [I 2022-10-09 10:05:50] ...trap/engines/engine.py.230: loss: 0.10160 [I 2022-10-09 10:05:50] ...optimizers/trijoint.py.151: total_norm: 11.707380 [I 2022-10-09 10:05:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 1760/2304 [I 2022-10-09 10:05:59] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:35 [I 2022-10-09 10:05:59] ...trap/engines/engine.py.229: process: 0.82360 | load: 0.00300 [I 2022-10-09 10:05:59] ...trap/engines/engine.py.230: loss: 0.14073 [I 2022-10-09 10:05:59] ...optimizers/trijoint.py.151: total_norm: 22.973532 [I 2022-10-09 10:06:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 1770/2304 [I 2022-10-09 10:06:07] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:30 [I 2022-10-09 10:06:07] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00294 [I 2022-10-09 10:06:07] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-09 10:06:07] ...optimizers/trijoint.py.151: total_norm: 22.154747 [I 2022-10-09 10:06:16] ...trap/engines/engine.py.224: train: epoch 36 | batch 1780/2304 [I 2022-10-09 10:06:16] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:21 [I 2022-10-09 10:06:16] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00292 [I 2022-10-09 10:06:16] ...trap/engines/engine.py.230: loss: 0.12812 [I 2022-10-09 10:06:16] ...optimizers/trijoint.py.151: total_norm: 16.130373 [I 2022-10-09 10:06:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 1790/2304 [I 2022-10-09 10:06:24] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:13 [I 2022-10-09 10:06:24] ...trap/engines/engine.py.229: process: 0.83634 | load: 0.00292 [I 2022-10-09 10:06:24] ...trap/engines/engine.py.230: loss: 0.14667 [I 2022-10-09 10:06:24] ...optimizers/trijoint.py.151: total_norm: 13.423849 [I 2022-10-09 10:06:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 1800/2304 [I 2022-10-09 10:06:33] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:05 [I 2022-10-09 10:06:33] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00291 [I 2022-10-09 10:06:33] ...trap/engines/engine.py.230: loss: 0.14754 [I 2022-10-09 10:06:33] ...optimizers/trijoint.py.151: total_norm: 14.776934 [I 2022-10-09 10:06:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 1810/2304 [I 2022-10-09 10:06:41] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:06:57 [I 2022-10-09 10:06:41] ...trap/engines/engine.py.229: process: 0.83687 | load: 0.00292 [I 2022-10-09 10:06:41] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-09 10:06:41] ...optimizers/trijoint.py.151: total_norm: 20.501356 [I 2022-10-09 10:06:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 1820/2304 [I 2022-10-09 10:06:49] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:45 [I 2022-10-09 10:06:49] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00294 [I 2022-10-09 10:06:49] ...trap/engines/engine.py.230: loss: 0.19806 [I 2022-10-09 10:06:49] ...optimizers/trijoint.py.151: total_norm: 24.303682 [I 2022-10-09 10:06:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 1830/2304 [I 2022-10-09 10:06:58] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:38 [I 2022-10-09 10:06:58] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00297 [I 2022-10-09 10:06:58] ...trap/engines/engine.py.230: loss: 0.20058 [I 2022-10-09 10:06:58] ...optimizers/trijoint.py.151: total_norm: 24.521336 [I 2022-10-09 10:07:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 1840/2304 [I 2022-10-09 10:07:06] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:29 [I 2022-10-09 10:07:06] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00291 [I 2022-10-09 10:07:06] ...trap/engines/engine.py.230: loss: 0.33037 [I 2022-10-09 10:07:06] ...optimizers/trijoint.py.151: total_norm: 32.908417 [I 2022-10-09 10:07:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 1850/2304 [I 2022-10-09 10:07:15] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:24 [I 2022-10-09 10:07:15] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00291 [I 2022-10-09 10:07:15] ...trap/engines/engine.py.230: loss: 0.14080 [I 2022-10-09 10:07:15] ...optimizers/trijoint.py.151: total_norm: 20.717445 [I 2022-10-09 10:07:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 1860/2304 [I 2022-10-09 10:07:23] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:15 [I 2022-10-09 10:07:23] ...trap/engines/engine.py.229: process: 0.84240 | load: 0.00294 [I 2022-10-09 10:07:23] ...trap/engines/engine.py.230: loss: 0.16267 [I 2022-10-09 10:07:23] ...optimizers/trijoint.py.151: total_norm: 33.184299 [I 2022-10-09 10:07:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 1870/2304 [I 2022-10-09 10:07:32] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:06 [I 2022-10-09 10:07:32] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00299 [I 2022-10-09 10:07:32] ...trap/engines/engine.py.230: loss: 0.18980 [I 2022-10-09 10:07:32] ...optimizers/trijoint.py.151: total_norm: 18.380623 [I 2022-10-09 10:07:40] ...trap/engines/engine.py.224: train: epoch 36 | batch 1880/2304 [I 2022-10-09 10:07:40] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:05:56 [I 2022-10-09 10:07:40] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00296 [I 2022-10-09 10:07:40] ...trap/engines/engine.py.230: loss: 0.13689 [I 2022-10-09 10:07:40] ...optimizers/trijoint.py.151: total_norm: 23.917488 [I 2022-10-09 10:07:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 1890/2304 [I 2022-10-09 10:07:48] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:05:47 [I 2022-10-09 10:07:48] ...trap/engines/engine.py.229: process: 0.83441 | load: 0.00293 [I 2022-10-09 10:07:48] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-09 10:07:48] ...optimizers/trijoint.py.151: total_norm: 9.727940 [I 2022-10-09 10:07:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 1900/2304 [I 2022-10-09 10:07:57] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:41 [I 2022-10-09 10:07:57] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00291 [I 2022-10-09 10:07:57] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-09 10:07:57] ...optimizers/trijoint.py.151: total_norm: 12.291889 [I 2022-10-09 10:08:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 1910/2304 [I 2022-10-09 10:08:05] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:32 [I 2022-10-09 10:08:05] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00293 [I 2022-10-09 10:08:05] ...trap/engines/engine.py.230: loss: 0.10530 [I 2022-10-09 10:08:05] ...optimizers/trijoint.py.151: total_norm: 14.104241 [I 2022-10-09 10:08:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 1920/2304 [I 2022-10-09 10:08:14] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:22 [I 2022-10-09 10:08:14] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00294 [I 2022-10-09 10:08:14] ...trap/engines/engine.py.230: loss: 0.11325 [I 2022-10-09 10:08:14] ...optimizers/trijoint.py.151: total_norm: 15.799339 [I 2022-10-09 10:08:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 1930/2304 [I 2022-10-09 10:08:22] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:13 [I 2022-10-09 10:08:22] ...trap/engines/engine.py.229: process: 0.82771 | load: 0.00291 [I 2022-10-09 10:08:22] ...trap/engines/engine.py.230: loss: 0.17856 [I 2022-10-09 10:08:22] ...optimizers/trijoint.py.151: total_norm: 21.306883 [I 2022-10-09 10:08:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1940/2304 [I 2022-10-09 10:08:31] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:05 [I 2022-10-09 10:08:31] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00291 [I 2022-10-09 10:08:31] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-09 10:08:31] ...optimizers/trijoint.py.151: total_norm: 11.754531 [I 2022-10-09 10:08:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 1950/2304 [I 2022-10-09 10:08:39] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:04:56 [I 2022-10-09 10:08:39] ...trap/engines/engine.py.229: process: 0.82760 | load: 0.00295 [I 2022-10-09 10:08:39] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-09 10:08:39] ...optimizers/trijoint.py.151: total_norm: 12.380793 [I 2022-10-09 10:08:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 1960/2304 [I 2022-10-09 10:08:47] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:04:48 [I 2022-10-09 10:08:47] ...trap/engines/engine.py.229: process: 0.84230 | load: 0.00289 [I 2022-10-09 10:08:47] ...trap/engines/engine.py.230: loss: 0.21573 [I 2022-10-09 10:08:47] ...optimizers/trijoint.py.151: total_norm: 23.757885 [I 2022-10-09 10:08:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 1970/2304 [I 2022-10-09 10:08:56] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:40 [I 2022-10-09 10:08:56] ...trap/engines/engine.py.229: process: 0.83676 | load: 0.00291 [I 2022-10-09 10:08:56] ...trap/engines/engine.py.230: loss: 0.10045 [I 2022-10-09 10:08:56] ...optimizers/trijoint.py.151: total_norm: 7.127626 [I 2022-10-09 10:09:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 1980/2304 [I 2022-10-09 10:09:04] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:31 [I 2022-10-09 10:09:04] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00300 [I 2022-10-09 10:09:04] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-09 10:09:04] ...optimizers/trijoint.py.151: total_norm: 16.088263 [I 2022-10-09 10:09:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 1990/2304 [I 2022-10-09 10:09:13] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:25 [I 2022-10-09 10:09:13] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00290 [I 2022-10-09 10:09:13] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-09 10:09:13] ...optimizers/trijoint.py.151: total_norm: 15.221880 [I 2022-10-09 10:09:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 2000/2304 [I 2022-10-09 10:09:21] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:16 [I 2022-10-09 10:09:21] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00299 [I 2022-10-09 10:09:21] ...trap/engines/engine.py.230: loss: 0.23743 [I 2022-10-09 10:09:21] ...optimizers/trijoint.py.151: total_norm: 21.968472 [I 2022-10-09 10:09:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 2010/2304 [I 2022-10-09 10:09:30] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:08 [I 2022-10-09 10:09:30] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00292 [I 2022-10-09 10:09:30] ...trap/engines/engine.py.230: loss: 0.17460 [I 2022-10-09 10:09:30] ...optimizers/trijoint.py.151: total_norm: 22.972553 [I 2022-10-09 10:09:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 2020/2304 [I 2022-10-09 10:09:38] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:03:58 [I 2022-10-09 10:09:38] ...trap/engines/engine.py.229: process: 0.82822 | load: 0.00298 [I 2022-10-09 10:09:38] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-09 10:09:38] ...optimizers/trijoint.py.151: total_norm: 19.143087 [I 2022-10-09 10:09:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 2030/2304 [I 2022-10-09 10:09:46] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:03:49 [I 2022-10-09 10:09:46] ...trap/engines/engine.py.229: process: 0.82009 | load: 0.00290 [I 2022-10-09 10:09:46] ...trap/engines/engine.py.230: loss: 0.18555 [I 2022-10-09 10:09:46] ...optimizers/trijoint.py.151: total_norm: 13.593554 [I 2022-10-09 10:09:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 2040/2304 [I 2022-10-09 10:09:55] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:42 [I 2022-10-09 10:09:55] ...trap/engines/engine.py.229: process: 0.83180 | load: 0.00295 [I 2022-10-09 10:09:55] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-09 10:09:55] ...optimizers/trijoint.py.151: total_norm: 14.712330 [I 2022-10-09 10:10:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 2050/2304 [I 2022-10-09 10:10:03] ...trap/engines/engine.py.228: elapsed: 0:29:10 | left: 0:03:34 [I 2022-10-09 10:10:03] ...trap/engines/engine.py.229: process: 0.83547 | load: 0.00289 [I 2022-10-09 10:10:03] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-09 10:10:03] ...optimizers/trijoint.py.151: total_norm: 18.047777 [I 2022-10-09 10:10:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 2060/2304 [I 2022-10-09 10:10:12] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:25 [I 2022-10-09 10:10:12] ...trap/engines/engine.py.229: process: 0.83609 | load: 0.00295 [I 2022-10-09 10:10:12] ...trap/engines/engine.py.230: loss: 0.16268 [I 2022-10-09 10:10:12] ...optimizers/trijoint.py.151: total_norm: 10.574784 [I 2022-10-09 10:10:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 2070/2304 [I 2022-10-09 10:10:20] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:16 [I 2022-10-09 10:10:20] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00292 [I 2022-10-09 10:10:20] ...trap/engines/engine.py.230: loss: 0.23617 [I 2022-10-09 10:10:20] ...optimizers/trijoint.py.151: total_norm: 19.021130 [I 2022-10-09 10:10:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 2080/2304 [I 2022-10-09 10:10:29] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:09 [I 2022-10-09 10:10:29] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00295 [I 2022-10-09 10:10:29] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-09 10:10:29] ...optimizers/trijoint.py.151: total_norm: 16.907724 [I 2022-10-09 10:10:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 2090/2304 [I 2022-10-09 10:10:37] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:00 [I 2022-10-09 10:10:37] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00292 [I 2022-10-09 10:10:37] ...trap/engines/engine.py.230: loss: 0.13942 [I 2022-10-09 10:10:37] ...optimizers/trijoint.py.151: total_norm: 17.503931 [I 2022-10-09 10:10:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 2100/2304 [I 2022-10-09 10:10:45] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:02:51 [I 2022-10-09 10:10:45] ...trap/engines/engine.py.229: process: 0.82717 | load: 0.00301 [I 2022-10-09 10:10:45] ...trap/engines/engine.py.230: loss: 0.18478 [I 2022-10-09 10:10:45] ...optimizers/trijoint.py.151: total_norm: 17.372190 [I 2022-10-09 10:10:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 2110/2304 [I 2022-10-09 10:10:54] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:42 [I 2022-10-09 10:10:54] ...trap/engines/engine.py.229: process: 0.83453 | load: 0.00291 [I 2022-10-09 10:10:54] ...trap/engines/engine.py.230: loss: 0.18377 [I 2022-10-09 10:10:54] ...optimizers/trijoint.py.151: total_norm: 17.292122 [I 2022-10-09 10:11:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 2120/2304 [I 2022-10-09 10:11:02] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:35 [I 2022-10-09 10:11:02] ...trap/engines/engine.py.229: process: 0.83898 | load: 0.00300 [I 2022-10-09 10:11:02] ...trap/engines/engine.py.230: loss: 0.10607 [I 2022-10-09 10:11:02] ...optimizers/trijoint.py.151: total_norm: 9.492555 [I 2022-10-09 10:11:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 2130/2304 [I 2022-10-09 10:11:11] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:27 [I 2022-10-09 10:11:11] ...trap/engines/engine.py.229: process: 0.84805 | load: 0.00291 [I 2022-10-09 10:11:11] ...trap/engines/engine.py.230: loss: 0.20977 [I 2022-10-09 10:11:11] ...optimizers/trijoint.py.151: total_norm: 33.774483 [I 2022-10-09 10:11:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 2140/2304 [I 2022-10-09 10:11:19] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:18 [I 2022-10-09 10:11:19] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00299 [I 2022-10-09 10:11:19] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-09 10:11:19] ...optimizers/trijoint.py.151: total_norm: 18.910904 [I 2022-10-09 10:11:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 2150/2304 [I 2022-10-09 10:11:28] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:09 [I 2022-10-09 10:11:28] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00293 [I 2022-10-09 10:11:28] ...trap/engines/engine.py.230: loss: 0.18866 [I 2022-10-09 10:11:28] ...optimizers/trijoint.py.151: total_norm: 15.688510 [I 2022-10-09 10:11:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 2160/2304 [I 2022-10-09 10:11:36] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:01 [I 2022-10-09 10:11:36] ...trap/engines/engine.py.229: process: 0.84475 | load: 0.00297 [I 2022-10-09 10:11:36] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-09 10:11:36] ...optimizers/trijoint.py.151: total_norm: 12.307413 [I 2022-10-09 10:11:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 2170/2304 [I 2022-10-09 10:11:45] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:01:53 [I 2022-10-09 10:11:45] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00292 [I 2022-10-09 10:11:45] ...trap/engines/engine.py.230: loss: 0.14098 [I 2022-10-09 10:11:45] ...optimizers/trijoint.py.151: total_norm: 18.860506 [I 2022-10-09 10:11:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 2180/2304 [I 2022-10-09 10:11:53] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:01:44 [I 2022-10-09 10:11:53] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00298 [I 2022-10-09 10:11:53] ...trap/engines/engine.py.230: loss: 0.17677 [I 2022-10-09 10:11:53] ...optimizers/trijoint.py.151: total_norm: 20.270870 [I 2022-10-09 10:12:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 2190/2304 [I 2022-10-09 10:12:01] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:35 [I 2022-10-09 10:12:01] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00292 [I 2022-10-09 10:12:01] ...trap/engines/engine.py.230: loss: 0.15537 [I 2022-10-09 10:12:01] ...optimizers/trijoint.py.151: total_norm: 17.536470 [I 2022-10-09 10:12:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 2200/2304 [I 2022-10-09 10:12:10] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:27 [I 2022-10-09 10:12:10] ...trap/engines/engine.py.229: process: 0.83639 | load: 0.00299 [I 2022-10-09 10:12:10] ...trap/engines/engine.py.230: loss: 0.26544 [I 2022-10-09 10:12:10] ...optimizers/trijoint.py.151: total_norm: 27.825802 [I 2022-10-09 10:12:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 2210/2304 [I 2022-10-09 10:12:18] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:19 [I 2022-10-09 10:12:18] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00290 [I 2022-10-09 10:12:18] ...trap/engines/engine.py.230: loss: 0.22903 [I 2022-10-09 10:12:18] ...optimizers/trijoint.py.151: total_norm: 13.950929 [I 2022-10-09 10:12:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 2220/2304 [I 2022-10-09 10:12:27] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:10 [I 2022-10-09 10:12:27] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00292 [I 2022-10-09 10:12:27] ...trap/engines/engine.py.230: loss: 0.20410 [I 2022-10-09 10:12:27] ...optimizers/trijoint.py.151: total_norm: 20.128206 [I 2022-10-09 10:12:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 2230/2304 [I 2022-10-09 10:12:35] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:02 [I 2022-10-09 10:12:35] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00289 [I 2022-10-09 10:12:35] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-09 10:12:35] ...optimizers/trijoint.py.151: total_norm: 22.331835 [I 2022-10-09 10:12:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 2240/2304 [I 2022-10-09 10:12:44] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:00:54 [I 2022-10-09 10:12:44] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00292 [I 2022-10-09 10:12:44] ...trap/engines/engine.py.230: loss: 0.11335 [I 2022-10-09 10:12:44] ...optimizers/trijoint.py.151: total_norm: 8.427126 [I 2022-10-09 10:12:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 2250/2304 [I 2022-10-09 10:12:52] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:00:45 [I 2022-10-09 10:12:52] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00291 [I 2022-10-09 10:12:52] ...trap/engines/engine.py.230: loss: 0.22917 [I 2022-10-09 10:12:52] ...optimizers/trijoint.py.151: total_norm: 15.492374 [I 2022-10-09 10:13:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 2260/2304 [I 2022-10-09 10:13:01] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:37 [I 2022-10-09 10:13:01] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00291 [I 2022-10-09 10:13:01] ...trap/engines/engine.py.230: loss: 0.17299 [I 2022-10-09 10:13:01] ...optimizers/trijoint.py.151: total_norm: 22.592470 [I 2022-10-09 10:13:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 2270/2304 [I 2022-10-09 10:13:09] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:28 [I 2022-10-09 10:13:09] ...trap/engines/engine.py.229: process: 0.84008 | load: 0.00289 [I 2022-10-09 10:13:09] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-09 10:13:09] ...optimizers/trijoint.py.151: total_norm: 10.281064 [I 2022-10-09 10:13:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 2280/2304 [I 2022-10-09 10:13:18] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:20 [I 2022-10-09 10:13:18] ...trap/engines/engine.py.229: process: 0.83151 | load: 0.00292 [I 2022-10-09 10:13:18] ...trap/engines/engine.py.230: loss: 0.19772 [I 2022-10-09 10:13:18] ...optimizers/trijoint.py.151: total_norm: 28.945936 [I 2022-10-09 10:13:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 2290/2304 [I 2022-10-09 10:13:26] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:11 [I 2022-10-09 10:13:26] ...trap/engines/engine.py.229: process: 0.85394 | load: 0.00289 [I 2022-10-09 10:13:26] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-09 10:13:26] ...optimizers/trijoint.py.151: total_norm: 18.960583 [I 2022-10-09 10:13:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 2300/2304 [I 2022-10-09 10:13:34] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:03 [I 2022-10-09 10:13:34] ...trap/engines/engine.py.229: process: 0.82462 | load: 0.00016 [I 2022-10-09 10:13:34] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-09 10:13:34] ...optimizers/trijoint.py.151: total_norm: 20.340929 [I 2022-10-09 10:13:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 2304/2304 [I 2022-10-09 10:13:38] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:00 [I 2022-10-09 10:13:38] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00018 [I 2022-10-09 10:13:38] ...trap/engines/engine.py.230: loss: 0.17503 [I 2022-10-09 10:13:38] ...optimizers/trijoint.py.151: total_norm: 13.109295 [S 2022-10-09 10:13:39] ...trap/engines/engine.py.129: train_epoch.epoch: 36 [S 2022-10-09 10:13:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.16163287330216802 [I 2022-10-09 10:13:39] ...s/criterions/triplet.py.84: margin = 0.23000000000000004 [I 2022-10-09 10:13:40] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 36 [I 2022-10-09 10:13:40] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 10:13:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 10:13:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 10:13:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 10:13:42] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 10:13:43] ...trap/engines/engine.py.321: eval: epoch 36 | batch 0/510 [I 2022-10-09 10:13:43] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:57 [I 2022-10-09 10:13:43] ...trap/engines/engine.py.326: process: 3.28845 | load: 2.79294 [I 2022-10-09 10:13:48] ...trap/engines/engine.py.321: eval: epoch 36 | batch 10/510 [I 2022-10-09 10:13:48] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:54 [I 2022-10-09 10:13:48] ...trap/engines/engine.py.326: process: 0.63374 | load: 0.00017 [I 2022-10-09 10:13:53] ...trap/engines/engine.py.321: eval: epoch 36 | batch 20/510 [I 2022-10-09 10:13:53] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:32 [I 2022-10-09 10:13:53] ...trap/engines/engine.py.326: process: 0.64682 | load: 0.00019 [I 2022-10-09 10:13:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 30/510 [I 2022-10-09 10:13:58] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:37 [I 2022-10-09 10:13:58] ...trap/engines/engine.py.326: process: 0.57613 | load: 0.00018 [I 2022-10-09 10:14:04] ...trap/engines/engine.py.321: eval: epoch 36 | batch 40/510 [I 2022-10-09 10:14:04] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:33 [I 2022-10-09 10:14:04] ...trap/engines/engine.py.326: process: 0.54211 | load: 0.00019 [I 2022-10-09 10:14:09] ...trap/engines/engine.py.321: eval: epoch 36 | batch 50/510 [I 2022-10-09 10:14:09] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:54 [I 2022-10-09 10:14:09] ...trap/engines/engine.py.326: process: 0.57105 | load: 0.00019 [I 2022-10-09 10:14:14] ...trap/engines/engine.py.321: eval: epoch 36 | batch 60/510 [I 2022-10-09 10:14:14] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:36 [I 2022-10-09 10:14:14] ...trap/engines/engine.py.326: process: 0.45434 | load: 0.00019 [I 2022-10-09 10:14:19] ...trap/engines/engine.py.321: eval: epoch 36 | batch 70/510 [I 2022-10-09 10:14:19] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:34 [I 2022-10-09 10:14:19] ...trap/engines/engine.py.326: process: 0.45778 | load: 0.00018 [I 2022-10-09 10:14:24] ...trap/engines/engine.py.321: eval: epoch 36 | batch 80/510 [I 2022-10-09 10:14:24] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:11 [I 2022-10-09 10:14:24] ...trap/engines/engine.py.326: process: 0.42358 | load: 0.00019 [I 2022-10-09 10:14:28] ...trap/engines/engine.py.321: eval: epoch 36 | batch 90/510 [I 2022-10-09 10:14:28] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:01 [I 2022-10-09 10:14:28] ...trap/engines/engine.py.326: process: 0.42018 | load: 0.00018 [I 2022-10-09 10:14:33] ...trap/engines/engine.py.321: eval: epoch 36 | batch 100/510 [I 2022-10-09 10:14:33] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:56 [I 2022-10-09 10:14:33] ...trap/engines/engine.py.326: process: 0.30529 | load: 0.00018 [I 2022-10-09 10:14:36] ...trap/engines/engine.py.321: eval: epoch 36 | batch 110/510 [I 2022-10-09 10:14:36] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:10 [I 2022-10-09 10:14:36] ...trap/engines/engine.py.326: process: 0.30386 | load: 0.00020 [I 2022-10-09 10:14:39] ...trap/engines/engine.py.321: eval: epoch 36 | batch 120/510 [I 2022-10-09 10:14:39] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:02 [I 2022-10-09 10:14:39] ...trap/engines/engine.py.326: process: 0.31634 | load: 0.00019 [I 2022-10-09 10:14:42] ...trap/engines/engine.py.321: eval: epoch 36 | batch 130/510 [I 2022-10-09 10:14:42] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:59 [I 2022-10-09 10:14:42] ...trap/engines/engine.py.326: process: 0.31181 | load: 0.00020 [I 2022-10-09 10:14:45] ...trap/engines/engine.py.321: eval: epoch 36 | batch 140/510 [I 2022-10-09 10:14:45] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:55 [I 2022-10-09 10:14:45] ...trap/engines/engine.py.326: process: 0.31013 | load: 0.00018 [I 2022-10-09 10:14:48] ...trap/engines/engine.py.321: eval: epoch 36 | batch 150/510 [I 2022-10-09 10:14:48] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:53 [I 2022-10-09 10:14:48] ...trap/engines/engine.py.326: process: 0.31196 | load: 0.00020 [I 2022-10-09 10:14:52] ...trap/engines/engine.py.321: eval: epoch 36 | batch 160/510 [I 2022-10-09 10:14:52] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:50 [I 2022-10-09 10:14:52] ...trap/engines/engine.py.326: process: 0.31080 | load: 0.00019 [I 2022-10-09 10:14:55] ...trap/engines/engine.py.321: eval: epoch 36 | batch 170/510 [I 2022-10-09 10:14:55] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:46 [I 2022-10-09 10:14:55] ...trap/engines/engine.py.326: process: 0.30747 | load: 0.00019 [I 2022-10-09 10:14:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 180/510 [I 2022-10-09 10:14:58] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:43 [I 2022-10-09 10:14:58] ...trap/engines/engine.py.326: process: 0.30651 | load: 0.00021 [I 2022-10-09 10:15:01] ...trap/engines/engine.py.321: eval: epoch 36 | batch 190/510 [I 2022-10-09 10:15:01] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:40 [I 2022-10-09 10:15:01] ...trap/engines/engine.py.326: process: 0.31837 | load: 0.00018 [I 2022-10-09 10:15:04] ...trap/engines/engine.py.321: eval: epoch 36 | batch 200/510 [I 2022-10-09 10:15:04] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:38 [I 2022-10-09 10:15:04] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00019 [I 2022-10-09 10:15:07] ...trap/engines/engine.py.321: eval: epoch 36 | batch 210/510 [I 2022-10-09 10:15:07] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:34 [I 2022-10-09 10:15:07] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00020 [I 2022-10-09 10:15:10] ...trap/engines/engine.py.321: eval: epoch 36 | batch 220/510 [I 2022-10-09 10:15:10] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:31 [I 2022-10-09 10:15:10] ...trap/engines/engine.py.326: process: 0.31583 | load: 0.00019 [I 2022-10-09 10:15:14] ...trap/engines/engine.py.321: eval: epoch 36 | batch 230/510 [I 2022-10-09 10:15:14] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:28 [I 2022-10-09 10:15:14] ...trap/engines/engine.py.326: process: 0.32141 | load: 0.00018 [I 2022-10-09 10:15:17] ...trap/engines/engine.py.321: eval: epoch 36 | batch 240/510 [I 2022-10-09 10:15:17] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:25 [I 2022-10-09 10:15:17] ...trap/engines/engine.py.326: process: 0.31525 | load: 0.00020 [I 2022-10-09 10:15:20] ...trap/engines/engine.py.321: eval: epoch 36 | batch 250/510 [I 2022-10-09 10:15:20] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-09 10:15:20] ...trap/engines/engine.py.326: process: 0.31723 | load: 0.00020 [I 2022-10-09 10:15:23] ...trap/engines/engine.py.321: eval: epoch 36 | batch 260/510 [I 2022-10-09 10:15:23] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:19 [I 2022-10-09 10:15:23] ...trap/engines/engine.py.326: process: 0.31630 | load: 0.00020 [I 2022-10-09 10:15:26] ...trap/engines/engine.py.321: eval: epoch 36 | batch 270/510 [I 2022-10-09 10:15:26] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:15 [I 2022-10-09 10:15:26] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00020 [I 2022-10-09 10:15:29] ...trap/engines/engine.py.321: eval: epoch 36 | batch 280/510 [I 2022-10-09 10:15:29] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-09 10:15:29] ...trap/engines/engine.py.326: process: 0.31365 | load: 0.00019 [I 2022-10-09 10:15:33] ...trap/engines/engine.py.321: eval: epoch 36 | batch 290/510 [I 2022-10-09 10:15:33] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:09 [I 2022-10-09 10:15:33] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00019 [I 2022-10-09 10:15:36] ...trap/engines/engine.py.321: eval: epoch 36 | batch 300/510 [I 2022-10-09 10:15:36] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:06 [I 2022-10-09 10:15:36] ...trap/engines/engine.py.326: process: 0.31693 | load: 0.00019 [I 2022-10-09 10:15:39] ...trap/engines/engine.py.321: eval: epoch 36 | batch 310/510 [I 2022-10-09 10:15:39] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:03 [I 2022-10-09 10:15:39] ...trap/engines/engine.py.326: process: 0.32065 | load: 0.00020 [I 2022-10-09 10:15:42] ...trap/engines/engine.py.321: eval: epoch 36 | batch 320/510 [I 2022-10-09 10:15:42] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-09 10:15:42] ...trap/engines/engine.py.326: process: 0.31564 | load: 0.00019 [I 2022-10-09 10:15:45] ...trap/engines/engine.py.321: eval: epoch 36 | batch 330/510 [I 2022-10-09 10:15:45] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-09 10:15:45] ...trap/engines/engine.py.326: process: 0.31224 | load: 0.00020 [I 2022-10-09 10:15:49] ...trap/engines/engine.py.321: eval: epoch 36 | batch 340/510 [I 2022-10-09 10:15:49] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-09 10:15:49] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00019 [I 2022-10-09 10:15:52] ...trap/engines/engine.py.321: eval: epoch 36 | batch 350/510 [I 2022-10-09 10:15:52] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:50 [I 2022-10-09 10:15:52] ...trap/engines/engine.py.326: process: 0.31881 | load: 0.00019 [I 2022-10-09 10:15:55] ...trap/engines/engine.py.321: eval: epoch 36 | batch 360/510 [I 2022-10-09 10:15:55] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-09 10:15:55] ...trap/engines/engine.py.326: process: 0.31629 | load: 0.00020 [I 2022-10-09 10:15:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 370/510 [I 2022-10-09 10:15:58] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-09 10:15:58] ...trap/engines/engine.py.326: process: 0.31500 | load: 0.00021 [I 2022-10-09 10:16:01] ...trap/engines/engine.py.321: eval: epoch 36 | batch 380/510 [I 2022-10-09 10:16:01] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:40 [I 2022-10-09 10:16:01] ...trap/engines/engine.py.326: process: 0.31103 | load: 0.00020 [I 2022-10-09 10:16:04] ...trap/engines/engine.py.321: eval: epoch 36 | batch 390/510 [I 2022-10-09 10:16:04] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-09 10:16:04] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00019 [I 2022-10-09 10:16:07] ...trap/engines/engine.py.321: eval: epoch 36 | batch 400/510 [I 2022-10-09 10:16:07] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-09 10:16:07] ...trap/engines/engine.py.326: process: 0.31106 | load: 0.00020 [I 2022-10-09 10:16:11] ...trap/engines/engine.py.321: eval: epoch 36 | batch 410/510 [I 2022-10-09 10:16:11] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:31 [I 2022-10-09 10:16:11] ...trap/engines/engine.py.326: process: 0.30814 | load: 0.00019 [I 2022-10-09 10:16:14] ...trap/engines/engine.py.321: eval: epoch 36 | batch 420/510 [I 2022-10-09 10:16:14] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-09 10:16:14] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00019 [I 2022-10-09 10:16:17] ...trap/engines/engine.py.321: eval: epoch 36 | batch 430/510 [I 2022-10-09 10:16:17] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-09 10:16:17] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00020 [I 2022-10-09 10:16:20] ...trap/engines/engine.py.321: eval: epoch 36 | batch 440/510 [I 2022-10-09 10:16:20] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-09 10:16:20] ...trap/engines/engine.py.326: process: 0.31897 | load: 0.00021 [I 2022-10-09 10:16:23] ...trap/engines/engine.py.321: eval: epoch 36 | batch 450/510 [I 2022-10-09 10:16:23] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-09 10:16:23] ...trap/engines/engine.py.326: process: 0.31353 | load: 0.00019 [I 2022-10-09 10:16:27] ...trap/engines/engine.py.321: eval: epoch 36 | batch 460/510 [I 2022-10-09 10:16:27] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-09 10:16:27] ...trap/engines/engine.py.326: process: 0.31158 | load: 0.00021 [I 2022-10-09 10:16:30] ...trap/engines/engine.py.321: eval: epoch 36 | batch 470/510 [I 2022-10-09 10:16:30] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-09 10:16:30] ...trap/engines/engine.py.326: process: 0.31679 | load: 0.00019 [I 2022-10-09 10:16:33] ...trap/engines/engine.py.321: eval: epoch 36 | batch 480/510 [I 2022-10-09 10:16:33] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-09 10:16:33] ...trap/engines/engine.py.326: process: 0.31549 | load: 0.00019 [I 2022-10-09 10:16:36] ...trap/engines/engine.py.321: eval: epoch 36 | batch 490/510 [I 2022-10-09 10:16:36] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-09 10:16:36] ...trap/engines/engine.py.326: process: 0.31483 | load: 0.00020 [I 2022-10-09 10:16:39] ...trap/engines/engine.py.321: eval: epoch 36 | batch 500/510 [I 2022-10-09 10:16:39] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-09 10:16:39] ...trap/engines/engine.py.326: process: 0.31932 | load: 0.00019 [I 2022-10-09 10:16:42] ...trap/engines/engine.py.321: eval: epoch 36 | batch 510/510 [I 2022-10-09 10:16:42] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-09 10:16:42] ...trap/engines/engine.py.326: process: 0.31370 | load: 0.00014 [S 2022-10-09 10:16:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 36 [I 2022-10-09 10:16:43] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.704900 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.898500 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934800 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.693400 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.892500 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.928900 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010737 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008523 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008244 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013071 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005518 [S 2022-10-09 10:17:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007341 [I 2022-10-09 10:17:24] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 10:17:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 10:17:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 10:17:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 10:17:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 10:17:27] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 10:17:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 10:17:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 10:17:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 37 [I 2022-10-09 10:17:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 10:17:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 0/2304 [I 2022-10-09 10:17:36] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:13 [I 2022-10-09 10:17:36] ...trap/engines/engine.py.229: process: 2.66198 | load: 1.75029 [I 2022-10-09 10:17:36] ...trap/engines/engine.py.230: loss: 0.11501 [I 2022-10-09 10:17:36] ...optimizers/trijoint.py.151: total_norm: 11.173938 [I 2022-10-09 10:17:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 10/2304 [I 2022-10-09 10:17:45] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:56 [I 2022-10-09 10:17:45] ...trap/engines/engine.py.229: process: 0.86624 | load: 0.00467 [I 2022-10-09 10:17:45] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-09 10:17:45] ...optimizers/trijoint.py.151: total_norm: 15.284651 [I 2022-10-09 10:17:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 20/2304 [I 2022-10-09 10:17:53] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:02 [I 2022-10-09 10:17:53] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00608 [I 2022-10-09 10:17:53] ...trap/engines/engine.py.230: loss: 0.12272 [I 2022-10-09 10:17:53] ...optimizers/trijoint.py.151: total_norm: 18.742605 [I 2022-10-09 10:18:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 30/2304 [I 2022-10-09 10:18:02] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:10 [I 2022-10-09 10:18:02] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00419 [I 2022-10-09 10:18:02] ...trap/engines/engine.py.230: loss: 0.12815 [I 2022-10-09 10:18:02] ...optimizers/trijoint.py.151: total_norm: 9.732515 [I 2022-10-09 10:18:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 40/2304 [I 2022-10-09 10:18:10] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:48 [I 2022-10-09 10:18:10] ...trap/engines/engine.py.229: process: 0.83647 | load: 0.00367 [I 2022-10-09 10:18:10] ...trap/engines/engine.py.230: loss: 0.19821 [I 2022-10-09 10:18:10] ...optimizers/trijoint.py.151: total_norm: 13.163827 [I 2022-10-09 10:18:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 50/2304 [I 2022-10-09 10:18:19] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:28 [I 2022-10-09 10:18:19] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00368 [I 2022-10-09 10:18:19] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-09 10:18:19] ...optimizers/trijoint.py.151: total_norm: 16.778471 [I 2022-10-09 10:18:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 60/2304 [I 2022-10-09 10:18:27] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:22 [I 2022-10-09 10:18:27] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00385 [I 2022-10-09 10:18:27] ...trap/engines/engine.py.230: loss: 0.15379 [I 2022-10-09 10:18:27] ...optimizers/trijoint.py.151: total_norm: 13.828513 [I 2022-10-09 10:18:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 70/2304 [I 2022-10-09 10:18:36] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:19 [I 2022-10-09 10:18:36] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00358 [I 2022-10-09 10:18:36] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-09 10:18:36] ...optimizers/trijoint.py.151: total_norm: 12.141099 [I 2022-10-09 10:18:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 80/2304 [I 2022-10-09 10:18:44] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:08 [I 2022-10-09 10:18:44] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00318 [I 2022-10-09 10:18:44] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-09 10:18:44] ...optimizers/trijoint.py.151: total_norm: 23.338724 [I 2022-10-09 10:18:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 90/2304 [I 2022-10-09 10:18:52] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:08 [I 2022-10-09 10:18:52] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00323 [I 2022-10-09 10:18:52] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-09 10:18:52] ...optimizers/trijoint.py.151: total_norm: 11.665129 [I 2022-10-09 10:19:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 100/2304 [I 2022-10-09 10:19:01] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:00 [I 2022-10-09 10:19:01] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00326 [I 2022-10-09 10:19:01] ...trap/engines/engine.py.230: loss: 0.19051 [I 2022-10-09 10:19:01] ...optimizers/trijoint.py.151: total_norm: 17.983177 [I 2022-10-09 10:19:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 110/2304 [I 2022-10-09 10:19:09] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:35 [I 2022-10-09 10:19:09] ...trap/engines/engine.py.229: process: 0.82392 | load: 0.00320 [I 2022-10-09 10:19:09] ...trap/engines/engine.py.230: loss: 0.10686 [I 2022-10-09 10:19:09] ...optimizers/trijoint.py.151: total_norm: 6.651477 [I 2022-10-09 10:19:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 120/2304 [I 2022-10-09 10:19:18] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:38 [I 2022-10-09 10:19:18] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00344 [I 2022-10-09 10:19:18] ...trap/engines/engine.py.230: loss: 0.24206 [I 2022-10-09 10:19:18] ...optimizers/trijoint.py.151: total_norm: 12.771249 [I 2022-10-09 10:19:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 130/2304 [I 2022-10-09 10:19:26] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:18 [I 2022-10-09 10:19:26] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00303 [I 2022-10-09 10:19:26] ...trap/engines/engine.py.230: loss: 0.19265 [I 2022-10-09 10:19:26] ...optimizers/trijoint.py.151: total_norm: 19.943243 [I 2022-10-09 10:19:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 140/2304 [I 2022-10-09 10:19:34] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:18 [I 2022-10-09 10:19:34] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00321 [I 2022-10-09 10:19:34] ...trap/engines/engine.py.230: loss: 0.16854 [I 2022-10-09 10:19:34] ...optimizers/trijoint.py.151: total_norm: 25.295052 [I 2022-10-09 10:19:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 150/2304 [I 2022-10-09 10:19:43] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:07 [I 2022-10-09 10:19:43] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00305 [I 2022-10-09 10:19:43] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-09 10:19:43] ...optimizers/trijoint.py.151: total_norm: 15.020721 [I 2022-10-09 10:19:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 160/2304 [I 2022-10-09 10:19:51] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:04 [I 2022-10-09 10:19:51] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00308 [I 2022-10-09 10:19:51] ...trap/engines/engine.py.230: loss: 0.19215 [I 2022-10-09 10:19:51] ...optimizers/trijoint.py.151: total_norm: 16.570766 [I 2022-10-09 10:20:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 170/2304 [I 2022-10-09 10:20:00] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:30:01 [I 2022-10-09 10:20:00] ...trap/engines/engine.py.229: process: 0.84558 | load: 0.00303 [I 2022-10-09 10:20:00] ...trap/engines/engine.py.230: loss: 0.14864 [I 2022-10-09 10:20:00] ...optimizers/trijoint.py.151: total_norm: 16.168884 [I 2022-10-09 10:20:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 180/2304 [I 2022-10-09 10:20:08] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:53 [I 2022-10-09 10:20:08] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00292 [I 2022-10-09 10:20:08] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-09 10:20:08] ...optimizers/trijoint.py.151: total_norm: 14.335925 [I 2022-10-09 10:20:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 190/2304 [I 2022-10-09 10:20:17] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:37 [I 2022-10-09 10:20:17] ...trap/engines/engine.py.229: process: 0.83173 | load: 0.00293 [I 2022-10-09 10:20:17] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-09 10:20:17] ...optimizers/trijoint.py.151: total_norm: 16.310638 [I 2022-10-09 10:20:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 200/2304 [I 2022-10-09 10:20:25] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:34 [I 2022-10-09 10:20:25] ...trap/engines/engine.py.229: process: 0.84661 | load: 0.00295 [I 2022-10-09 10:20:25] ...trap/engines/engine.py.230: loss: 0.15864 [I 2022-10-09 10:20:25] ...optimizers/trijoint.py.151: total_norm: 41.908508 [I 2022-10-09 10:20:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 210/2304 [I 2022-10-09 10:20:34] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:25 [I 2022-10-09 10:20:34] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00307 [I 2022-10-09 10:20:34] ...trap/engines/engine.py.230: loss: 0.19143 [I 2022-10-09 10:20:34] ...optimizers/trijoint.py.151: total_norm: 31.891718 [I 2022-10-09 10:20:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 220/2304 [I 2022-10-09 10:20:42] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:08 [I 2022-10-09 10:20:42] ...trap/engines/engine.py.229: process: 0.84225 | load: 0.00304 [I 2022-10-09 10:20:42] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-09 10:20:42] ...optimizers/trijoint.py.151: total_norm: 20.291183 [I 2022-10-09 10:20:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 230/2304 [I 2022-10-09 10:20:50] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:02 [I 2022-10-09 10:20:50] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00313 [I 2022-10-09 10:20:50] ...trap/engines/engine.py.230: loss: 0.12167 [I 2022-10-09 10:20:50] ...optimizers/trijoint.py.151: total_norm: 21.456079 [I 2022-10-09 10:20:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 240/2304 [I 2022-10-09 10:20:59] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:29:02 [I 2022-10-09 10:20:59] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00297 [I 2022-10-09 10:20:59] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-09 10:20:59] ...optimizers/trijoint.py.151: total_norm: 9.156453 [I 2022-10-09 10:21:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 250/2304 [I 2022-10-09 10:21:07] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:46 [I 2022-10-09 10:21:07] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00293 [I 2022-10-09 10:21:07] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-09 10:21:07] ...optimizers/trijoint.py.151: total_norm: 30.043039 [I 2022-10-09 10:21:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 260/2304 [I 2022-10-09 10:21:16] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:34 [I 2022-10-09 10:21:16] ...trap/engines/engine.py.229: process: 0.83258 | load: 0.00306 [I 2022-10-09 10:21:16] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-09 10:21:16] ...optimizers/trijoint.py.151: total_norm: 16.586447 [I 2022-10-09 10:21:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 270/2304 [I 2022-10-09 10:21:24] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:35 [I 2022-10-09 10:21:24] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00292 [I 2022-10-09 10:21:24] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-09 10:21:24] ...optimizers/trijoint.py.151: total_norm: 25.176523 [I 2022-10-09 10:21:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 280/2304 [I 2022-10-09 10:21:32] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:27 [I 2022-10-09 10:21:32] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00305 [I 2022-10-09 10:21:32] ...trap/engines/engine.py.230: loss: 0.17963 [I 2022-10-09 10:21:32] ...optimizers/trijoint.py.151: total_norm: 21.434103 [I 2022-10-09 10:21:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 290/2304 [I 2022-10-09 10:21:41] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:10 [I 2022-10-09 10:21:41] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00292 [I 2022-10-09 10:21:41] ...trap/engines/engine.py.230: loss: 0.15931 [I 2022-10-09 10:21:41] ...optimizers/trijoint.py.151: total_norm: 15.678401 [I 2022-10-09 10:21:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 300/2304 [I 2022-10-09 10:21:49] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:01 [I 2022-10-09 10:21:49] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00306 [I 2022-10-09 10:21:49] ...trap/engines/engine.py.230: loss: 0.10975 [I 2022-10-09 10:21:49] ...optimizers/trijoint.py.151: total_norm: 11.167722 [I 2022-10-09 10:21:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 310/2304 [I 2022-10-09 10:21:58] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:27:59 [I 2022-10-09 10:21:58] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00293 [I 2022-10-09 10:21:58] ...trap/engines/engine.py.230: loss: 0.09933 [I 2022-10-09 10:21:58] ...optimizers/trijoint.py.151: total_norm: 9.074244 [I 2022-10-09 10:22:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 320/2304 [I 2022-10-09 10:22:06] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:44 [I 2022-10-09 10:22:06] ...trap/engines/engine.py.229: process: 0.83513 | load: 0.00296 [I 2022-10-09 10:22:06] ...trap/engines/engine.py.230: loss: 0.09491 [I 2022-10-09 10:22:06] ...optimizers/trijoint.py.151: total_norm: 8.331261 [I 2022-10-09 10:22:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 330/2304 [I 2022-10-09 10:22:15] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:39 [I 2022-10-09 10:22:15] ...trap/engines/engine.py.229: process: 0.84664 | load: 0.00293 [I 2022-10-09 10:22:15] ...trap/engines/engine.py.230: loss: 0.12407 [I 2022-10-09 10:22:15] ...optimizers/trijoint.py.151: total_norm: 11.919772 [I 2022-10-09 10:22:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 340/2304 [I 2022-10-09 10:22:23] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:28 [I 2022-10-09 10:22:23] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00290 [I 2022-10-09 10:22:23] ...trap/engines/engine.py.230: loss: 0.17804 [I 2022-10-09 10:22:23] ...optimizers/trijoint.py.151: total_norm: 21.335838 [I 2022-10-09 10:22:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 350/2304 [I 2022-10-09 10:22:31] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:27 [I 2022-10-09 10:22:31] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00305 [I 2022-10-09 10:22:31] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-09 10:22:31] ...optimizers/trijoint.py.151: total_norm: 26.236269 [I 2022-10-09 10:22:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 360/2304 [I 2022-10-09 10:22:40] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:16 [I 2022-10-09 10:22:40] ...trap/engines/engine.py.229: process: 0.83829 | load: 0.00289 [I 2022-10-09 10:22:40] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-09 10:22:40] ...optimizers/trijoint.py.151: total_norm: 24.675688 [I 2022-10-09 10:22:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 370/2304 [I 2022-10-09 10:22:48] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:09 [I 2022-10-09 10:22:48] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00293 [I 2022-10-09 10:22:48] ...trap/engines/engine.py.230: loss: 0.19639 [I 2022-10-09 10:22:48] ...optimizers/trijoint.py.151: total_norm: 19.153025 [I 2022-10-09 10:22:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 380/2304 [I 2022-10-09 10:22:57] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:26:50 [I 2022-10-09 10:22:57] ...trap/engines/engine.py.229: process: 0.83779 | load: 0.00311 [I 2022-10-09 10:22:57] ...trap/engines/engine.py.230: loss: 0.22288 [I 2022-10-09 10:22:57] ...optimizers/trijoint.py.151: total_norm: 36.292290 [I 2022-10-09 10:23:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 390/2304 [I 2022-10-09 10:23:05] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:52 [I 2022-10-09 10:23:05] ...trap/engines/engine.py.229: process: 0.83656 | load: 0.00292 [I 2022-10-09 10:23:05] ...trap/engines/engine.py.230: loss: 0.19997 [I 2022-10-09 10:23:05] ...optimizers/trijoint.py.151: total_norm: 18.919151 [I 2022-10-09 10:23:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 400/2304 [I 2022-10-09 10:23:14] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:41 [I 2022-10-09 10:23:14] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00290 [I 2022-10-09 10:23:14] ...trap/engines/engine.py.230: loss: 0.15470 [I 2022-10-09 10:23:14] ...optimizers/trijoint.py.151: total_norm: 32.203121 [I 2022-10-09 10:23:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 410/2304 [I 2022-10-09 10:23:22] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:45 [I 2022-10-09 10:23:22] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00294 [I 2022-10-09 10:23:22] ...trap/engines/engine.py.230: loss: 0.17550 [I 2022-10-09 10:23:22] ...optimizers/trijoint.py.151: total_norm: 54.234352 [I 2022-10-09 10:23:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 420/2304 [I 2022-10-09 10:23:30] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:29 [I 2022-10-09 10:23:30] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00295 [I 2022-10-09 10:23:30] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-09 10:23:30] ...optimizers/trijoint.py.151: total_norm: 17.639740 [I 2022-10-09 10:23:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 430/2304 [I 2022-10-09 10:23:39] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:21 [I 2022-10-09 10:23:39] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00310 [I 2022-10-09 10:23:39] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-09 10:23:39] ...optimizers/trijoint.py.151: total_norm: 10.473860 [I 2022-10-09 10:23:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 440/2304 [I 2022-10-09 10:23:47] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:17 [I 2022-10-09 10:23:47] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00308 [I 2022-10-09 10:23:47] ...trap/engines/engine.py.230: loss: 0.18392 [I 2022-10-09 10:23:47] ...optimizers/trijoint.py.151: total_norm: 14.371227 [I 2022-10-09 10:23:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 450/2304 [I 2022-10-09 10:23:56] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:04 [I 2022-10-09 10:23:56] ...trap/engines/engine.py.229: process: 0.83683 | load: 0.00293 [I 2022-10-09 10:23:56] ...trap/engines/engine.py.230: loss: 0.18749 [I 2022-10-09 10:23:56] ...optimizers/trijoint.py.151: total_norm: 17.215940 [I 2022-10-09 10:24:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 460/2304 [I 2022-10-09 10:24:04] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:53 [I 2022-10-09 10:24:04] ...trap/engines/engine.py.229: process: 0.83945 | load: 0.00302 [I 2022-10-09 10:24:04] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-09 10:24:04] ...optimizers/trijoint.py.151: total_norm: 34.198643 [I 2022-10-09 10:24:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 470/2304 [I 2022-10-09 10:24:13] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:50 [I 2022-10-09 10:24:13] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00291 [I 2022-10-09 10:24:13] ...trap/engines/engine.py.230: loss: 0.24796 [I 2022-10-09 10:24:13] ...optimizers/trijoint.py.151: total_norm: 19.640877 [I 2022-10-09 10:24:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 480/2304 [I 2022-10-09 10:24:21] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:42 [I 2022-10-09 10:24:21] ...trap/engines/engine.py.229: process: 0.84230 | load: 0.00303 [I 2022-10-09 10:24:21] ...trap/engines/engine.py.230: loss: 0.27046 [I 2022-10-09 10:24:21] ...optimizers/trijoint.py.151: total_norm: 13.596154 [I 2022-10-09 10:24:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 490/2304 [I 2022-10-09 10:24:30] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:44 [I 2022-10-09 10:24:30] ...trap/engines/engine.py.229: process: 0.84948 | load: 0.00300 [I 2022-10-09 10:24:30] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-09 10:24:30] ...optimizers/trijoint.py.151: total_norm: 13.301351 [I 2022-10-09 10:24:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 500/2304 [I 2022-10-09 10:24:38] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:25 [I 2022-10-09 10:24:38] ...trap/engines/engine.py.229: process: 0.82951 | load: 0.00293 [I 2022-10-09 10:24:38] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-09 10:24:38] ...optimizers/trijoint.py.151: total_norm: 11.329166 [I 2022-10-09 10:24:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 510/2304 [I 2022-10-09 10:24:47] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:21 [I 2022-10-09 10:24:47] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00292 [I 2022-10-09 10:24:47] ...trap/engines/engine.py.230: loss: 0.10964 [I 2022-10-09 10:24:47] ...optimizers/trijoint.py.151: total_norm: 35.273266 [I 2022-10-09 10:24:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 520/2304 [I 2022-10-09 10:24:55] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:12 [I 2022-10-09 10:24:55] ...trap/engines/engine.py.229: process: 0.84609 | load: 0.00293 [I 2022-10-09 10:24:55] ...trap/engines/engine.py.230: loss: 0.18031 [I 2022-10-09 10:24:55] ...optimizers/trijoint.py.151: total_norm: 24.774540 [I 2022-10-09 10:25:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 530/2304 [I 2022-10-09 10:25:04] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:25:00 [I 2022-10-09 10:25:04] ...trap/engines/engine.py.229: process: 0.83196 | load: 0.00308 [I 2022-10-09 10:25:04] ...trap/engines/engine.py.230: loss: 0.17539 [I 2022-10-09 10:25:04] ...optimizers/trijoint.py.151: total_norm: 27.288582 [I 2022-10-09 10:25:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 540/2304 [I 2022-10-09 10:25:12] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:42 [I 2022-10-09 10:25:12] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00293 [I 2022-10-09 10:25:12] ...trap/engines/engine.py.230: loss: 0.09312 [I 2022-10-09 10:25:12] ...optimizers/trijoint.py.151: total_norm: 9.865411 [I 2022-10-09 10:25:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 550/2304 [I 2022-10-09 10:25:21] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:30 [I 2022-10-09 10:25:21] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00292 [I 2022-10-09 10:25:21] ...trap/engines/engine.py.230: loss: 0.15174 [I 2022-10-09 10:25:21] ...optimizers/trijoint.py.151: total_norm: 20.194283 [I 2022-10-09 10:25:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 560/2304 [I 2022-10-09 10:25:29] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:35 [I 2022-10-09 10:25:29] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00294 [I 2022-10-09 10:25:29] ...trap/engines/engine.py.230: loss: 0.18263 [I 2022-10-09 10:25:29] ...optimizers/trijoint.py.151: total_norm: 19.060923 [I 2022-10-09 10:25:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 570/2304 [I 2022-10-09 10:25:37] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:22 [I 2022-10-09 10:25:37] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00304 [I 2022-10-09 10:25:37] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-09 10:25:37] ...optimizers/trijoint.py.151: total_norm: 25.408810 [I 2022-10-09 10:25:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 580/2304 [I 2022-10-09 10:25:46] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:05 [I 2022-10-09 10:25:46] ...trap/engines/engine.py.229: process: 0.83224 | load: 0.00291 [I 2022-10-09 10:25:46] ...trap/engines/engine.py.230: loss: 0.18091 [I 2022-10-09 10:25:46] ...optimizers/trijoint.py.151: total_norm: 19.931803 [I 2022-10-09 10:25:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 590/2304 [I 2022-10-09 10:25:54] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:04 [I 2022-10-09 10:25:54] ...trap/engines/engine.py.229: process: 0.84919 | load: 0.00291 [I 2022-10-09 10:25:54] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-09 10:25:54] ...optimizers/trijoint.py.151: total_norm: 17.838978 [I 2022-10-09 10:26:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 600/2304 [I 2022-10-09 10:26:03] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:53 [I 2022-10-09 10:26:03] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00293 [I 2022-10-09 10:26:03] ...trap/engines/engine.py.230: loss: 0.15617 [I 2022-10-09 10:26:03] ...optimizers/trijoint.py.151: total_norm: 11.276110 [I 2022-10-09 10:26:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 610/2304 [I 2022-10-09 10:26:11] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:50 [I 2022-10-09 10:26:11] ...trap/engines/engine.py.229: process: 0.86268 | load: 0.00291 [I 2022-10-09 10:26:11] ...trap/engines/engine.py.230: loss: 0.17730 [I 2022-10-09 10:26:11] ...optimizers/trijoint.py.151: total_norm: 48.358231 [I 2022-10-09 10:26:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 620/2304 [I 2022-10-09 10:26:20] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:36 [I 2022-10-09 10:26:20] ...trap/engines/engine.py.229: process: 0.82373 | load: 0.00294 [I 2022-10-09 10:26:20] ...trap/engines/engine.py.230: loss: 0.13712 [I 2022-10-09 10:26:20] ...optimizers/trijoint.py.151: total_norm: 20.237267 [I 2022-10-09 10:26:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 630/2304 [I 2022-10-09 10:26:28] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:29 [I 2022-10-09 10:26:28] ...trap/engines/engine.py.229: process: 0.82947 | load: 0.00295 [I 2022-10-09 10:26:28] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-09 10:26:28] ...optimizers/trijoint.py.151: total_norm: 10.135319 [I 2022-10-09 10:26:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 640/2304 [I 2022-10-09 10:26:36] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:22 [I 2022-10-09 10:26:36] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00294 [I 2022-10-09 10:26:36] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-09 10:26:36] ...optimizers/trijoint.py.151: total_norm: 10.003397 [I 2022-10-09 10:26:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 650/2304 [I 2022-10-09 10:26:45] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:16 [I 2022-10-09 10:26:45] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00290 [I 2022-10-09 10:26:45] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-09 10:26:45] ...optimizers/trijoint.py.151: total_norm: 10.541106 [I 2022-10-09 10:26:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 660/2304 [I 2022-10-09 10:26:53] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:04 [I 2022-10-09 10:26:53] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00291 [I 2022-10-09 10:26:53] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-09 10:26:53] ...optimizers/trijoint.py.151: total_norm: 11.600803 [I 2022-10-09 10:27:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 670/2304 [I 2022-10-09 10:27:02] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:58 [I 2022-10-09 10:27:02] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00291 [I 2022-10-09 10:27:02] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-09 10:27:02] ...optimizers/trijoint.py.151: total_norm: 35.758854 [I 2022-10-09 10:27:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 680/2304 [I 2022-10-09 10:27:10] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:46 [I 2022-10-09 10:27:10] ...trap/engines/engine.py.229: process: 0.84001 | load: 0.00291 [I 2022-10-09 10:27:10] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-09 10:27:10] ...optimizers/trijoint.py.151: total_norm: 8.678437 [I 2022-10-09 10:27:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 690/2304 [I 2022-10-09 10:27:19] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:39 [I 2022-10-09 10:27:19] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00293 [I 2022-10-09 10:27:19] ...trap/engines/engine.py.230: loss: 0.15202 [I 2022-10-09 10:27:19] ...optimizers/trijoint.py.151: total_norm: 15.259050 [I 2022-10-09 10:27:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 700/2304 [I 2022-10-09 10:27:27] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:37 [I 2022-10-09 10:27:27] ...trap/engines/engine.py.229: process: 0.85782 | load: 0.00293 [I 2022-10-09 10:27:27] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-09 10:27:27] ...optimizers/trijoint.py.151: total_norm: 11.364469 [I 2022-10-09 10:27:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 710/2304 [I 2022-10-09 10:27:36] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:25 [I 2022-10-09 10:27:36] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00292 [I 2022-10-09 10:27:36] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-09 10:27:36] ...optimizers/trijoint.py.151: total_norm: 21.673815 [I 2022-10-09 10:27:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 720/2304 [I 2022-10-09 10:27:44] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:19 [I 2022-10-09 10:27:44] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00293 [I 2022-10-09 10:27:44] ...trap/engines/engine.py.230: loss: 0.18825 [I 2022-10-09 10:27:44] ...optimizers/trijoint.py.151: total_norm: 14.966928 [I 2022-10-09 10:27:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 730/2304 [I 2022-10-09 10:27:52] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:21:59 [I 2022-10-09 10:27:52] ...trap/engines/engine.py.229: process: 0.82473 | load: 0.00295 [I 2022-10-09 10:27:52] ...trap/engines/engine.py.230: loss: 0.19415 [I 2022-10-09 10:27:52] ...optimizers/trijoint.py.151: total_norm: 30.974136 [I 2022-10-09 10:28:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 740/2304 [I 2022-10-09 10:28:01] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:03 [I 2022-10-09 10:28:01] ...trap/engines/engine.py.229: process: 0.85666 | load: 0.00294 [I 2022-10-09 10:28:01] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-09 10:28:01] ...optimizers/trijoint.py.151: total_norm: 10.997629 [I 2022-10-09 10:28:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 750/2304 [I 2022-10-09 10:28:09] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:57 [I 2022-10-09 10:28:09] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00294 [I 2022-10-09 10:28:09] ...trap/engines/engine.py.230: loss: 0.16178 [I 2022-10-09 10:28:09] ...optimizers/trijoint.py.151: total_norm: 18.986267 [I 2022-10-09 10:28:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 760/2304 [I 2022-10-09 10:28:18] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:41 [I 2022-10-09 10:28:18] ...trap/engines/engine.py.229: process: 0.83198 | load: 0.00294 [I 2022-10-09 10:28:18] ...trap/engines/engine.py.230: loss: 0.15261 [I 2022-10-09 10:28:18] ...optimizers/trijoint.py.151: total_norm: 19.872618 [I 2022-10-09 10:28:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 770/2304 [I 2022-10-09 10:28:26] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:41 [I 2022-10-09 10:28:26] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00302 [I 2022-10-09 10:28:26] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-09 10:28:26] ...optimizers/trijoint.py.151: total_norm: 16.180580 [I 2022-10-09 10:28:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 780/2304 [I 2022-10-09 10:28:35] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:30 [I 2022-10-09 10:28:35] ...trap/engines/engine.py.229: process: 0.84197 | load: 0.00295 [I 2022-10-09 10:28:35] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-09 10:28:35] ...optimizers/trijoint.py.151: total_norm: 30.871191 [I 2022-10-09 10:28:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 790/2304 [I 2022-10-09 10:28:43] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:13 [I 2022-10-09 10:28:43] ...trap/engines/engine.py.229: process: 0.83535 | load: 0.00296 [I 2022-10-09 10:28:43] ...trap/engines/engine.py.230: loss: 0.20449 [I 2022-10-09 10:28:43] ...optimizers/trijoint.py.151: total_norm: 16.797939 [I 2022-10-09 10:28:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 800/2304 [I 2022-10-09 10:28:52] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:05 [I 2022-10-09 10:28:52] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00294 [I 2022-10-09 10:28:52] ...trap/engines/engine.py.230: loss: 0.14387 [I 2022-10-09 10:28:52] ...optimizers/trijoint.py.151: total_norm: 16.631136 [I 2022-10-09 10:29:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 810/2304 [I 2022-10-09 10:29:00] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:02 [I 2022-10-09 10:29:00] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00307 [I 2022-10-09 10:29:00] ...trap/engines/engine.py.230: loss: 0.16821 [I 2022-10-09 10:29:00] ...optimizers/trijoint.py.151: total_norm: 25.537266 [I 2022-10-09 10:29:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 820/2304 [I 2022-10-09 10:29:09] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:54 [I 2022-10-09 10:29:09] ...trap/engines/engine.py.229: process: 0.83208 | load: 0.00304 [I 2022-10-09 10:29:09] ...trap/engines/engine.py.230: loss: 0.07684 [I 2022-10-09 10:29:09] ...optimizers/trijoint.py.151: total_norm: 11.920415 [I 2022-10-09 10:29:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 830/2304 [I 2022-10-09 10:29:17] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:48 [I 2022-10-09 10:29:17] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00294 [I 2022-10-09 10:29:17] ...trap/engines/engine.py.230: loss: 0.20285 [I 2022-10-09 10:29:17] ...optimizers/trijoint.py.151: total_norm: 21.887905 [I 2022-10-09 10:29:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 840/2304 [I 2022-10-09 10:29:26] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:38 [I 2022-10-09 10:29:26] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00295 [I 2022-10-09 10:29:26] ...trap/engines/engine.py.230: loss: 0.22172 [I 2022-10-09 10:29:26] ...optimizers/trijoint.py.151: total_norm: 41.359241 [I 2022-10-09 10:29:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 850/2304 [I 2022-10-09 10:29:34] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:24 [I 2022-10-09 10:29:34] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00298 [I 2022-10-09 10:29:34] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-09 10:29:34] ...optimizers/trijoint.py.151: total_norm: 13.231893 [I 2022-10-09 10:29:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 860/2304 [I 2022-10-09 10:29:42] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:24 [I 2022-10-09 10:29:42] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00298 [I 2022-10-09 10:29:42] ...trap/engines/engine.py.230: loss: 0.20221 [I 2022-10-09 10:29:42] ...optimizers/trijoint.py.151: total_norm: 15.813470 [I 2022-10-09 10:29:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 870/2304 [I 2022-10-09 10:29:51] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:06 [I 2022-10-09 10:29:51] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00296 [I 2022-10-09 10:29:51] ...trap/engines/engine.py.230: loss: 0.11512 [I 2022-10-09 10:29:51] ...optimizers/trijoint.py.151: total_norm: 12.225338 [I 2022-10-09 10:29:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 880/2304 [I 2022-10-09 10:29:59] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:01 [I 2022-10-09 10:29:59] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00301 [I 2022-10-09 10:29:59] ...trap/engines/engine.py.230: loss: 0.13595 [I 2022-10-09 10:29:59] ...optimizers/trijoint.py.151: total_norm: 19.294409 [I 2022-10-09 10:30:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 890/2304 [I 2022-10-09 10:30:08] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:51 [I 2022-10-09 10:30:08] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00288 [I 2022-10-09 10:30:08] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-09 10:30:08] ...optimizers/trijoint.py.151: total_norm: 21.773869 [I 2022-10-09 10:30:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 900/2304 [I 2022-10-09 10:30:16] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:41 [I 2022-10-09 10:30:16] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00291 [I 2022-10-09 10:30:16] ...trap/engines/engine.py.230: loss: 0.21289 [I 2022-10-09 10:30:16] ...optimizers/trijoint.py.151: total_norm: 15.525882 [I 2022-10-09 10:30:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 910/2304 [I 2022-10-09 10:30:25] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:31 [I 2022-10-09 10:30:25] ...trap/engines/engine.py.229: process: 0.82641 | load: 0.00292 [I 2022-10-09 10:30:25] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-09 10:30:25] ...optimizers/trijoint.py.151: total_norm: 14.434022 [I 2022-10-09 10:30:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 920/2304 [I 2022-10-09 10:30:33] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:21 [I 2022-10-09 10:30:33] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00289 [I 2022-10-09 10:30:33] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-09 10:30:33] ...optimizers/trijoint.py.151: total_norm: 12.050698 [I 2022-10-09 10:30:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 930/2304 [I 2022-10-09 10:30:41] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:21 [I 2022-10-09 10:30:41] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00299 [I 2022-10-09 10:30:41] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-09 10:30:41] ...optimizers/trijoint.py.151: total_norm: 13.986069 [I 2022-10-09 10:30:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 940/2304 [I 2022-10-09 10:30:50] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:18 [I 2022-10-09 10:30:50] ...trap/engines/engine.py.229: process: 0.85602 | load: 0.00299 [I 2022-10-09 10:30:50] ...trap/engines/engine.py.230: loss: 0.15502 [I 2022-10-09 10:30:50] ...optimizers/trijoint.py.151: total_norm: 20.385529 [I 2022-10-09 10:30:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 950/2304 [I 2022-10-09 10:30:58] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:05 [I 2022-10-09 10:30:58] ...trap/engines/engine.py.229: process: 0.82824 | load: 0.00291 [I 2022-10-09 10:30:58] ...trap/engines/engine.py.230: loss: 0.20043 [I 2022-10-09 10:30:58] ...optimizers/trijoint.py.151: total_norm: 20.720360 [I 2022-10-09 10:31:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 960/2304 [I 2022-10-09 10:31:07] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:18:52 [I 2022-10-09 10:31:07] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00292 [I 2022-10-09 10:31:07] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-09 10:31:07] ...optimizers/trijoint.py.151: total_norm: 12.598802 [I 2022-10-09 10:31:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 970/2304 [I 2022-10-09 10:31:15] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:47 [I 2022-10-09 10:31:15] ...trap/engines/engine.py.229: process: 0.85270 | load: 0.00292 [I 2022-10-09 10:31:15] ...trap/engines/engine.py.230: loss: 0.19311 [I 2022-10-09 10:31:15] ...optimizers/trijoint.py.151: total_norm: 17.864830 [I 2022-10-09 10:31:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 980/2304 [I 2022-10-09 10:31:24] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:32 [I 2022-10-09 10:31:24] ...trap/engines/engine.py.229: process: 0.83671 | load: 0.00296 [I 2022-10-09 10:31:24] ...trap/engines/engine.py.230: loss: 0.18091 [I 2022-10-09 10:31:24] ...optimizers/trijoint.py.151: total_norm: 14.211039 [I 2022-10-09 10:31:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 990/2304 [I 2022-10-09 10:31:32] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:28 [I 2022-10-09 10:31:32] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00303 [I 2022-10-09 10:31:32] ...trap/engines/engine.py.230: loss: 0.19138 [I 2022-10-09 10:31:32] ...optimizers/trijoint.py.151: total_norm: 18.647614 [I 2022-10-09 10:31:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1000/2304 [I 2022-10-09 10:31:41] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:10 [I 2022-10-09 10:31:41] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00301 [I 2022-10-09 10:31:41] ...trap/engines/engine.py.230: loss: 0.13757 [I 2022-10-09 10:31:41] ...optimizers/trijoint.py.151: total_norm: 56.581528 [I 2022-10-09 10:31:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 1010/2304 [I 2022-10-09 10:31:49] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:13 [I 2022-10-09 10:31:49] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00294 [I 2022-10-09 10:31:49] ...trap/engines/engine.py.230: loss: 0.23560 [I 2022-10-09 10:31:49] ...optimizers/trijoint.py.151: total_norm: 16.516520 [I 2022-10-09 10:32:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 1020/2304 [I 2022-10-09 10:32:16] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 1:07:32 [I 2022-10-09 10:32:16] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00292 [I 2022-10-09 10:32:16] ...trap/engines/engine.py.230: loss: 0.21174 [I 2022-10-09 10:32:16] ...optimizers/trijoint.py.151: total_norm: 11.500780 [I 2022-10-09 10:32:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 1030/2304 [I 2022-10-09 10:32:24] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:23:09 [I 2022-10-09 10:32:24] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00293 [I 2022-10-09 10:32:24] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-09 10:32:24] ...optimizers/trijoint.py.151: total_norm: 13.798919 [I 2022-10-09 10:32:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 1040/2304 [I 2022-10-09 10:32:32] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:14 [I 2022-10-09 10:32:32] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00290 [I 2022-10-09 10:32:32] ...trap/engines/engine.py.230: loss: 0.20744 [I 2022-10-09 10:32:32] ...optimizers/trijoint.py.151: total_norm: 11.667081 [I 2022-10-09 10:32:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1050/2304 [I 2022-10-09 10:32:41] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:28 [I 2022-10-09 10:32:41] ...trap/engines/engine.py.229: process: 0.82238 | load: 0.00296 [I 2022-10-09 10:32:41] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-09 10:32:41] ...optimizers/trijoint.py.151: total_norm: 12.851972 [I 2022-10-09 10:32:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 1060/2304 [I 2022-10-09 10:32:49] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:27 [I 2022-10-09 10:32:49] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00291 [I 2022-10-09 10:32:49] ...trap/engines/engine.py.230: loss: 0.10384 [I 2022-10-09 10:32:49] ...optimizers/trijoint.py.151: total_norm: 10.120589 [I 2022-10-09 10:32:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 1070/2304 [I 2022-10-09 10:32:58] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:20 [I 2022-10-09 10:32:58] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00298 [I 2022-10-09 10:32:58] ...trap/engines/engine.py.230: loss: 0.15854 [I 2022-10-09 10:32:58] ...optimizers/trijoint.py.151: total_norm: 13.247416 [I 2022-10-09 10:33:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 1080/2304 [I 2022-10-09 10:33:06] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:11 [I 2022-10-09 10:33:06] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00293 [I 2022-10-09 10:33:06] ...trap/engines/engine.py.230: loss: 0.10417 [I 2022-10-09 10:33:06] ...optimizers/trijoint.py.151: total_norm: 17.176060 [I 2022-10-09 10:33:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 1090/2304 [I 2022-10-09 10:33:15] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:02 [I 2022-10-09 10:33:15] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00301 [I 2022-10-09 10:33:15] ...trap/engines/engine.py.230: loss: 0.22188 [I 2022-10-09 10:33:15] ...optimizers/trijoint.py.151: total_norm: 28.953472 [I 2022-10-09 10:33:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 1100/2304 [I 2022-10-09 10:33:23] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:50 [I 2022-10-09 10:33:23] ...trap/engines/engine.py.229: process: 0.83335 | load: 0.00294 [I 2022-10-09 10:33:23] ...trap/engines/engine.py.230: loss: 0.19136 [I 2022-10-09 10:33:23] ...optimizers/trijoint.py.151: total_norm: 17.398212 [I 2022-10-09 10:33:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 1110/2304 [I 2022-10-09 10:33:31] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:43 [I 2022-10-09 10:33:31] ...trap/engines/engine.py.229: process: 0.84711 | load: 0.00296 [I 2022-10-09 10:33:31] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-09 10:33:31] ...optimizers/trijoint.py.151: total_norm: 15.307011 [I 2022-10-09 10:33:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 1120/2304 [I 2022-10-09 10:33:40] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:35 [I 2022-10-09 10:33:40] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00292 [I 2022-10-09 10:33:40] ...trap/engines/engine.py.230: loss: 0.12829 [I 2022-10-09 10:33:40] ...optimizers/trijoint.py.151: total_norm: 8.323139 [I 2022-10-09 10:33:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 1130/2304 [I 2022-10-09 10:33:48] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:26 [I 2022-10-09 10:33:48] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00292 [I 2022-10-09 10:33:48] ...trap/engines/engine.py.230: loss: 0.18705 [I 2022-10-09 10:33:48] ...optimizers/trijoint.py.151: total_norm: 18.411684 [I 2022-10-09 10:33:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 1140/2304 [I 2022-10-09 10:33:57] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:21 [I 2022-10-09 10:33:57] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00293 [I 2022-10-09 10:33:57] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-09 10:33:57] ...optimizers/trijoint.py.151: total_norm: 11.876240 [I 2022-10-09 10:34:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 1150/2304 [I 2022-10-09 10:34:05] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:12 [I 2022-10-09 10:34:05] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00295 [I 2022-10-09 10:34:05] ...trap/engines/engine.py.230: loss: 0.22823 [I 2022-10-09 10:34:05] ...optimizers/trijoint.py.151: total_norm: 20.913462 [I 2022-10-09 10:34:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1160/2304 [I 2022-10-09 10:34:13] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:01 [I 2022-10-09 10:34:13] ...trap/engines/engine.py.229: process: 0.82624 | load: 0.00294 [I 2022-10-09 10:34:13] ...trap/engines/engine.py.230: loss: 0.21311 [I 2022-10-09 10:34:13] ...optimizers/trijoint.py.151: total_norm: 22.980968 [I 2022-10-09 10:34:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 1170/2304 [I 2022-10-09 10:34:22] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:52 [I 2022-10-09 10:34:22] ...trap/engines/engine.py.229: process: 0.83352 | load: 0.00297 [I 2022-10-09 10:34:22] ...trap/engines/engine.py.230: loss: 0.17026 [I 2022-10-09 10:34:22] ...optimizers/trijoint.py.151: total_norm: 17.757977 [I 2022-10-09 10:34:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 1180/2304 [I 2022-10-09 10:34:30] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:47 [I 2022-10-09 10:34:30] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00292 [I 2022-10-09 10:34:30] ...trap/engines/engine.py.230: loss: 0.19753 [I 2022-10-09 10:34:30] ...optimizers/trijoint.py.151: total_norm: 41.264046 [I 2022-10-09 10:34:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 1190/2304 [I 2022-10-09 10:34:39] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:35 [I 2022-10-09 10:34:39] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00293 [I 2022-10-09 10:34:39] ...trap/engines/engine.py.230: loss: 0.10917 [I 2022-10-09 10:34:39] ...optimizers/trijoint.py.151: total_norm: 13.532076 [I 2022-10-09 10:34:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 1200/2304 [I 2022-10-09 10:34:47] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:29 [I 2022-10-09 10:34:47] ...trap/engines/engine.py.229: process: 0.83216 | load: 0.00293 [I 2022-10-09 10:34:47] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-09 10:34:47] ...optimizers/trijoint.py.151: total_norm: 19.189520 [I 2022-10-09 10:34:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 1210/2304 [I 2022-10-09 10:34:56] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:27 [I 2022-10-09 10:34:56] ...trap/engines/engine.py.229: process: 0.85921 | load: 0.00291 [I 2022-10-09 10:34:56] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-09 10:34:56] ...optimizers/trijoint.py.151: total_norm: 13.906098 [I 2022-10-09 10:35:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 1220/2304 [I 2022-10-09 10:35:04] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:12 [I 2022-10-09 10:35:04] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00294 [I 2022-10-09 10:35:04] ...trap/engines/engine.py.230: loss: 0.17888 [I 2022-10-09 10:35:04] ...optimizers/trijoint.py.151: total_norm: 14.527088 [I 2022-10-09 10:35:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1230/2304 [I 2022-10-09 10:35:13] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:07 [I 2022-10-09 10:35:13] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00298 [I 2022-10-09 10:35:13] ...trap/engines/engine.py.230: loss: 0.20742 [I 2022-10-09 10:35:13] ...optimizers/trijoint.py.151: total_norm: 35.675159 [I 2022-10-09 10:35:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 1240/2304 [I 2022-10-09 10:35:21] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:54 [I 2022-10-09 10:35:21] ...trap/engines/engine.py.229: process: 0.83353 | load: 0.00291 [I 2022-10-09 10:35:21] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-09 10:35:21] ...optimizers/trijoint.py.151: total_norm: 19.637016 [I 2022-10-09 10:35:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 1250/2304 [I 2022-10-09 10:35:29] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:49 [I 2022-10-09 10:35:29] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00292 [I 2022-10-09 10:35:29] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-09 10:35:29] ...optimizers/trijoint.py.151: total_norm: 12.743730 [I 2022-10-09 10:35:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 1260/2304 [I 2022-10-09 10:35:38] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:14:34 [I 2022-10-09 10:35:38] ...trap/engines/engine.py.229: process: 0.83744 | load: 0.00296 [I 2022-10-09 10:35:38] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-09 10:35:38] ...optimizers/trijoint.py.151: total_norm: 19.166147 [I 2022-10-09 10:35:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 1270/2304 [I 2022-10-09 10:35:46] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:30 [I 2022-10-09 10:35:46] ...trap/engines/engine.py.229: process: 0.82696 | load: 0.00297 [I 2022-10-09 10:35:46] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-09 10:35:46] ...optimizers/trijoint.py.151: total_norm: 20.454748 [I 2022-10-09 10:35:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 1280/2304 [I 2022-10-09 10:35:55] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:20 [I 2022-10-09 10:35:55] ...trap/engines/engine.py.229: process: 0.84719 | load: 0.00292 [I 2022-10-09 10:35:55] ...trap/engines/engine.py.230: loss: 0.15768 [I 2022-10-09 10:35:55] ...optimizers/trijoint.py.151: total_norm: 17.777060 [I 2022-10-09 10:36:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 1290/2304 [I 2022-10-09 10:36:03] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:17 [I 2022-10-09 10:36:03] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00293 [I 2022-10-09 10:36:03] ...trap/engines/engine.py.230: loss: 0.14508 [I 2022-10-09 10:36:03] ...optimizers/trijoint.py.151: total_norm: 14.542780 [I 2022-10-09 10:36:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 1300/2304 [I 2022-10-09 10:36:12] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:14:05 [I 2022-10-09 10:36:12] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00293 [I 2022-10-09 10:36:12] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-09 10:36:12] ...optimizers/trijoint.py.151: total_norm: 14.585726 [I 2022-10-09 10:36:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 1310/2304 [I 2022-10-09 10:36:20] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:13:55 [I 2022-10-09 10:36:20] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00296 [I 2022-10-09 10:36:20] ...trap/engines/engine.py.230: loss: 0.16408 [I 2022-10-09 10:36:20] ...optimizers/trijoint.py.151: total_norm: 19.294958 [I 2022-10-09 10:36:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 1320/2304 [I 2022-10-09 10:36:28] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:13:52 [I 2022-10-09 10:36:28] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00292 [I 2022-10-09 10:36:28] ...trap/engines/engine.py.230: loss: 0.23543 [I 2022-10-09 10:36:28] ...optimizers/trijoint.py.151: total_norm: 20.023594 [I 2022-10-09 10:36:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 1330/2304 [I 2022-10-09 10:36:37] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:41 [I 2022-10-09 10:36:37] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00292 [I 2022-10-09 10:36:37] ...trap/engines/engine.py.230: loss: 0.15690 [I 2022-10-09 10:36:37] ...optimizers/trijoint.py.151: total_norm: 24.539885 [I 2022-10-09 10:36:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 1340/2304 [I 2022-10-09 10:36:45] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:13:29 [I 2022-10-09 10:36:45] ...trap/engines/engine.py.229: process: 0.84189 | load: 0.00292 [I 2022-10-09 10:36:45] ...trap/engines/engine.py.230: loss: 0.10412 [I 2022-10-09 10:36:45] ...optimizers/trijoint.py.151: total_norm: 7.264543 [I 2022-10-09 10:36:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 1350/2304 [I 2022-10-09 10:36:54] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:18 [I 2022-10-09 10:36:54] ...trap/engines/engine.py.229: process: 0.83130 | load: 0.00293 [I 2022-10-09 10:36:54] ...trap/engines/engine.py.230: loss: 0.19869 [I 2022-10-09 10:36:54] ...optimizers/trijoint.py.151: total_norm: 30.210197 [I 2022-10-09 10:37:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 1360/2304 [I 2022-10-09 10:37:02] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:19 [I 2022-10-09 10:37:02] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00297 [I 2022-10-09 10:37:02] ...trap/engines/engine.py.230: loss: 0.21264 [I 2022-10-09 10:37:02] ...optimizers/trijoint.py.151: total_norm: 22.154943 [I 2022-10-09 10:37:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 1370/2304 [I 2022-10-09 10:37:11] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:06 [I 2022-10-09 10:37:11] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00298 [I 2022-10-09 10:37:11] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-09 10:37:11] ...optimizers/trijoint.py.151: total_norm: 25.408077 [I 2022-10-09 10:37:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 1380/2304 [I 2022-10-09 10:37:19] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:12:55 [I 2022-10-09 10:37:19] ...trap/engines/engine.py.229: process: 0.83286 | load: 0.00296 [I 2022-10-09 10:37:19] ...trap/engines/engine.py.230: loss: 0.14446 [I 2022-10-09 10:37:19] ...optimizers/trijoint.py.151: total_norm: 9.701716 [I 2022-10-09 10:37:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 1390/2304 [I 2022-10-09 10:37:27] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:12:49 [I 2022-10-09 10:37:27] ...trap/engines/engine.py.229: process: 0.84436 | load: 0.00290 [I 2022-10-09 10:37:27] ...trap/engines/engine.py.230: loss: 0.18839 [I 2022-10-09 10:37:27] ...optimizers/trijoint.py.151: total_norm: 17.315359 [I 2022-10-09 10:37:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 1400/2304 [I 2022-10-09 10:37:36] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:12:39 [I 2022-10-09 10:37:36] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00292 [I 2022-10-09 10:37:36] ...trap/engines/engine.py.230: loss: 0.12075 [I 2022-10-09 10:37:36] ...optimizers/trijoint.py.151: total_norm: 11.672740 [I 2022-10-09 10:37:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 1410/2304 [I 2022-10-09 10:37:44] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:34 [I 2022-10-09 10:37:44] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00292 [I 2022-10-09 10:37:44] ...trap/engines/engine.py.230: loss: 0.21615 [I 2022-10-09 10:37:44] ...optimizers/trijoint.py.151: total_norm: 29.399750 [I 2022-10-09 10:37:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 1420/2304 [I 2022-10-09 10:37:53] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:25 [I 2022-10-09 10:37:53] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00292 [I 2022-10-09 10:37:53] ...trap/engines/engine.py.230: loss: 0.18005 [I 2022-10-09 10:37:53] ...optimizers/trijoint.py.151: total_norm: 19.965122 [I 2022-10-09 10:38:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 1430/2304 [I 2022-10-09 10:38:01] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:14 [I 2022-10-09 10:38:01] ...trap/engines/engine.py.229: process: 0.82709 | load: 0.00294 [I 2022-10-09 10:38:01] ...trap/engines/engine.py.230: loss: 0.15975 [I 2022-10-09 10:38:01] ...optimizers/trijoint.py.151: total_norm: 16.896856 [I 2022-10-09 10:38:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 1440/2304 [I 2022-10-09 10:38:10] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:03 [I 2022-10-09 10:38:10] ...trap/engines/engine.py.229: process: 0.82837 | load: 0.00294 [I 2022-10-09 10:38:10] ...trap/engines/engine.py.230: loss: 0.12833 [I 2022-10-09 10:38:10] ...optimizers/trijoint.py.151: total_norm: 18.700771 [I 2022-10-09 10:38:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 1450/2304 [I 2022-10-09 10:38:18] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:11:58 [I 2022-10-09 10:38:18] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00302 [I 2022-10-09 10:38:18] ...trap/engines/engine.py.230: loss: 0.18525 [I 2022-10-09 10:38:18] ...optimizers/trijoint.py.151: total_norm: 29.890165 [I 2022-10-09 10:38:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 1460/2304 [I 2022-10-09 10:38:26] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:11:50 [I 2022-10-09 10:38:26] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00294 [I 2022-10-09 10:38:26] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-09 10:38:26] ...optimizers/trijoint.py.151: total_norm: 16.123222 [I 2022-10-09 10:38:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 1470/2304 [I 2022-10-09 10:38:35] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:43 [I 2022-10-09 10:38:35] ...trap/engines/engine.py.229: process: 0.85270 | load: 0.00292 [I 2022-10-09 10:38:35] ...trap/engines/engine.py.230: loss: 0.12731 [I 2022-10-09 10:38:35] ...optimizers/trijoint.py.151: total_norm: 12.638338 [I 2022-10-09 10:38:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 1480/2304 [I 2022-10-09 10:38:43] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:11:34 [I 2022-10-09 10:38:43] ...trap/engines/engine.py.229: process: 0.84103 | load: 0.00294 [I 2022-10-09 10:38:43] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-09 10:38:43] ...optimizers/trijoint.py.151: total_norm: 9.412714 [I 2022-10-09 10:38:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 1490/2304 [I 2022-10-09 10:38:52] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:24 [I 2022-10-09 10:38:52] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00293 [I 2022-10-09 10:38:52] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-09 10:38:52] ...optimizers/trijoint.py.151: total_norm: 20.440058 [I 2022-10-09 10:39:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 1500/2304 [I 2022-10-09 10:39:00] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:17 [I 2022-10-09 10:39:00] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00288 [I 2022-10-09 10:39:00] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-09 10:39:00] ...optimizers/trijoint.py.151: total_norm: 9.875997 [I 2022-10-09 10:39:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 1510/2304 [I 2022-10-09 10:39:08] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:11:06 [I 2022-10-09 10:39:08] ...trap/engines/engine.py.229: process: 0.83067 | load: 0.00294 [I 2022-10-09 10:39:08] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-09 10:39:08] ...optimizers/trijoint.py.151: total_norm: 18.010723 [I 2022-10-09 10:39:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 1520/2304 [I 2022-10-09 10:39:17] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:11:01 [I 2022-10-09 10:39:17] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00294 [I 2022-10-09 10:39:17] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-09 10:39:17] ...optimizers/trijoint.py.151: total_norm: 21.589054 [I 2022-10-09 10:39:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 1530/2304 [I 2022-10-09 10:39:25] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:10:54 [I 2022-10-09 10:39:25] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00294 [I 2022-10-09 10:39:25] ...trap/engines/engine.py.230: loss: 0.14577 [I 2022-10-09 10:39:25] ...optimizers/trijoint.py.151: total_norm: 12.479338 [I 2022-10-09 10:39:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 1540/2304 [I 2022-10-09 10:39:34] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:10:42 [I 2022-10-09 10:39:34] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00291 [I 2022-10-09 10:39:34] ...trap/engines/engine.py.230: loss: 0.18566 [I 2022-10-09 10:39:34] ...optimizers/trijoint.py.151: total_norm: 16.000340 [I 2022-10-09 10:39:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 1550/2304 [I 2022-10-09 10:39:42] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:10:35 [I 2022-10-09 10:39:42] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00294 [I 2022-10-09 10:39:42] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-09 10:39:42] ...optimizers/trijoint.py.151: total_norm: 10.572505 [I 2022-10-09 10:39:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 1560/2304 [I 2022-10-09 10:39:51] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:25 [I 2022-10-09 10:39:51] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00292 [I 2022-10-09 10:39:51] ...trap/engines/engine.py.230: loss: 0.22319 [I 2022-10-09 10:39:51] ...optimizers/trijoint.py.151: total_norm: 13.153052 [I 2022-10-09 10:39:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 1570/2304 [I 2022-10-09 10:39:59] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:10:18 [I 2022-10-09 10:39:59] ...trap/engines/engine.py.229: process: 0.84168 | load: 0.00294 [I 2022-10-09 10:39:59] ...trap/engines/engine.py.230: loss: 0.23144 [I 2022-10-09 10:39:59] ...optimizers/trijoint.py.151: total_norm: 20.959101 [I 2022-10-09 10:40:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 1580/2304 [I 2022-10-09 10:40:07] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:09 [I 2022-10-09 10:40:07] ...trap/engines/engine.py.229: process: 0.84778 | load: 0.00292 [I 2022-10-09 10:40:07] ...trap/engines/engine.py.230: loss: 0.16383 [I 2022-10-09 10:40:07] ...optimizers/trijoint.py.151: total_norm: 17.663528 [I 2022-10-09 10:40:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 1590/2304 [I 2022-10-09 10:40:16] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:10:02 [I 2022-10-09 10:40:16] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00291 [I 2022-10-09 10:40:16] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-09 10:40:16] ...optimizers/trijoint.py.151: total_norm: 11.020097 [I 2022-10-09 10:40:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 1600/2304 [I 2022-10-09 10:40:24] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:09:53 [I 2022-10-09 10:40:24] ...trap/engines/engine.py.229: process: 0.84837 | load: 0.00297 [I 2022-10-09 10:40:24] ...trap/engines/engine.py.230: loss: 0.12269 [I 2022-10-09 10:40:24] ...optimizers/trijoint.py.151: total_norm: 13.376943 [I 2022-10-09 10:40:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 1610/2304 [I 2022-10-09 10:40:33] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:09:44 [I 2022-10-09 10:40:33] ...trap/engines/engine.py.229: process: 0.82764 | load: 0.00297 [I 2022-10-09 10:40:33] ...trap/engines/engine.py.230: loss: 0.13490 [I 2022-10-09 10:40:33] ...optimizers/trijoint.py.151: total_norm: 38.693913 [I 2022-10-09 10:40:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1620/2304 [I 2022-10-09 10:40:41] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:09:32 [I 2022-10-09 10:40:41] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00292 [I 2022-10-09 10:40:41] ...trap/engines/engine.py.230: loss: 0.13110 [I 2022-10-09 10:40:41] ...optimizers/trijoint.py.151: total_norm: 10.720104 [I 2022-10-09 10:40:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 1630/2304 [I 2022-10-09 10:40:50] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:25 [I 2022-10-09 10:40:50] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00290 [I 2022-10-09 10:40:50] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-09 10:40:50] ...optimizers/trijoint.py.151: total_norm: 11.887939 [I 2022-10-09 10:40:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 1640/2304 [I 2022-10-09 10:40:58] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:19 [I 2022-10-09 10:40:58] ...trap/engines/engine.py.229: process: 0.84348 | load: 0.00293 [I 2022-10-09 10:40:58] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-09 10:40:58] ...optimizers/trijoint.py.151: total_norm: 15.686748 [I 2022-10-09 10:41:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 1650/2304 [I 2022-10-09 10:41:06] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:10 [I 2022-10-09 10:41:06] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00298 [I 2022-10-09 10:41:06] ...trap/engines/engine.py.230: loss: 0.17087 [I 2022-10-09 10:41:06] ...optimizers/trijoint.py.151: total_norm: 11.492547 [I 2022-10-09 10:41:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 1660/2304 [I 2022-10-09 10:41:15] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:00 [I 2022-10-09 10:41:15] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00293 [I 2022-10-09 10:41:15] ...trap/engines/engine.py.230: loss: 0.18732 [I 2022-10-09 10:41:15] ...optimizers/trijoint.py.151: total_norm: 28.343685 [I 2022-10-09 10:41:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 1670/2304 [I 2022-10-09 10:41:23] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:08:52 [I 2022-10-09 10:41:23] ...trap/engines/engine.py.229: process: 0.83599 | load: 0.00292 [I 2022-10-09 10:41:23] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-09 10:41:23] ...optimizers/trijoint.py.151: total_norm: 20.478348 [I 2022-10-09 10:41:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 1680/2304 [I 2022-10-09 10:41:32] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:44 [I 2022-10-09 10:41:32] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00293 [I 2022-10-09 10:41:32] ...trap/engines/engine.py.230: loss: 0.16398 [I 2022-10-09 10:41:32] ...optimizers/trijoint.py.151: total_norm: 15.455786 [I 2022-10-09 10:41:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 1690/2304 [I 2022-10-09 10:41:40] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:37 [I 2022-10-09 10:41:40] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00295 [I 2022-10-09 10:41:40] ...trap/engines/engine.py.230: loss: 0.23317 [I 2022-10-09 10:41:40] ...optimizers/trijoint.py.151: total_norm: 23.795820 [I 2022-10-09 10:41:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 1700/2304 [I 2022-10-09 10:41:48] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:28 [I 2022-10-09 10:41:48] ...trap/engines/engine.py.229: process: 0.83689 | load: 0.00300 [I 2022-10-09 10:41:48] ...trap/engines/engine.py.230: loss: 0.19475 [I 2022-10-09 10:41:48] ...optimizers/trijoint.py.151: total_norm: 16.552031 [I 2022-10-09 10:41:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 1710/2304 [I 2022-10-09 10:41:57] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:21 [I 2022-10-09 10:41:57] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00290 [I 2022-10-09 10:41:57] ...trap/engines/engine.py.230: loss: 0.17343 [I 2022-10-09 10:41:57] ...optimizers/trijoint.py.151: total_norm: 14.154968 [I 2022-10-09 10:42:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 1720/2304 [I 2022-10-09 10:42:05] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:08 [I 2022-10-09 10:42:05] ...trap/engines/engine.py.229: process: 0.83222 | load: 0.00292 [I 2022-10-09 10:42:05] ...trap/engines/engine.py.230: loss: 0.17350 [I 2022-10-09 10:42:05] ...optimizers/trijoint.py.151: total_norm: 11.669465 [I 2022-10-09 10:42:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 1730/2304 [I 2022-10-09 10:42:14] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:03 [I 2022-10-09 10:42:14] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00292 [I 2022-10-09 10:42:14] ...trap/engines/engine.py.230: loss: 0.18094 [I 2022-10-09 10:42:14] ...optimizers/trijoint.py.151: total_norm: 19.223719 [I 2022-10-09 10:42:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 1740/2304 [I 2022-10-09 10:42:22] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:07:53 [I 2022-10-09 10:42:22] ...trap/engines/engine.py.229: process: 0.83524 | load: 0.00291 [I 2022-10-09 10:42:22] ...trap/engines/engine.py.230: loss: 0.15964 [I 2022-10-09 10:42:22] ...optimizers/trijoint.py.151: total_norm: 21.841379 [I 2022-10-09 10:42:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 1750/2304 [I 2022-10-09 10:42:31] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:07:45 [I 2022-10-09 10:42:31] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00293 [I 2022-10-09 10:42:31] ...trap/engines/engine.py.230: loss: 0.19576 [I 2022-10-09 10:42:31] ...optimizers/trijoint.py.151: total_norm: 29.888910 [I 2022-10-09 10:42:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 1760/2304 [I 2022-10-09 10:42:39] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:37 [I 2022-10-09 10:42:39] ...trap/engines/engine.py.229: process: 0.84573 | load: 0.00297 [I 2022-10-09 10:42:39] ...trap/engines/engine.py.230: loss: 0.17120 [I 2022-10-09 10:42:39] ...optimizers/trijoint.py.151: total_norm: 18.445263 [I 2022-10-09 10:42:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 1770/2304 [I 2022-10-09 10:42:47] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:30 [I 2022-10-09 10:42:47] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00300 [I 2022-10-09 10:42:47] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-09 10:42:47] ...optimizers/trijoint.py.151: total_norm: 34.571938 [I 2022-10-09 10:42:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 1780/2304 [I 2022-10-09 10:42:56] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:24 [I 2022-10-09 10:42:56] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00289 [I 2022-10-09 10:42:56] ...trap/engines/engine.py.230: loss: 0.16596 [I 2022-10-09 10:42:56] ...optimizers/trijoint.py.151: total_norm: 13.596058 [I 2022-10-09 10:43:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 1790/2304 [I 2022-10-09 10:43:04] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:12 [I 2022-10-09 10:43:04] ...trap/engines/engine.py.229: process: 0.83315 | load: 0.00292 [I 2022-10-09 10:43:04] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-09 10:43:04] ...optimizers/trijoint.py.151: total_norm: 19.987017 [I 2022-10-09 10:43:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1800/2304 [I 2022-10-09 10:43:13] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:05 [I 2022-10-09 10:43:13] ...trap/engines/engine.py.229: process: 0.84112 | load: 0.00290 [I 2022-10-09 10:43:13] ...trap/engines/engine.py.230: loss: 0.20706 [I 2022-10-09 10:43:13] ...optimizers/trijoint.py.151: total_norm: 51.486881 [I 2022-10-09 10:43:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 1810/2304 [I 2022-10-09 10:43:21] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:06:56 [I 2022-10-09 10:43:21] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00293 [I 2022-10-09 10:43:21] ...trap/engines/engine.py.230: loss: 0.11142 [I 2022-10-09 10:43:21] ...optimizers/trijoint.py.151: total_norm: 10.402584 [I 2022-10-09 10:43:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 1820/2304 [I 2022-10-09 10:43:30] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:06:51 [I 2022-10-09 10:43:30] ...trap/engines/engine.py.229: process: 0.86676 | load: 0.00297 [I 2022-10-09 10:43:30] ...trap/engines/engine.py.230: loss: 0.21798 [I 2022-10-09 10:43:30] ...optimizers/trijoint.py.151: total_norm: 30.538712 [I 2022-10-09 10:43:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 1830/2304 [I 2022-10-09 10:43:38] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:39 [I 2022-10-09 10:43:38] ...trap/engines/engine.py.229: process: 0.84008 | load: 0.00289 [I 2022-10-09 10:43:38] ...trap/engines/engine.py.230: loss: 0.12727 [I 2022-10-09 10:43:38] ...optimizers/trijoint.py.151: total_norm: 18.136763 [I 2022-10-09 10:43:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 1840/2304 [I 2022-10-09 10:43:47] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:32 [I 2022-10-09 10:43:47] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00293 [I 2022-10-09 10:43:47] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-09 10:43:47] ...optimizers/trijoint.py.151: total_norm: 12.909716 [I 2022-10-09 10:43:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 1850/2304 [I 2022-10-09 10:43:55] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:24 [I 2022-10-09 10:43:55] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00293 [I 2022-10-09 10:43:55] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-09 10:43:55] ...optimizers/trijoint.py.151: total_norm: 16.654491 [I 2022-10-09 10:44:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 1860/2304 [I 2022-10-09 10:44:04] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:16 [I 2022-10-09 10:44:04] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00304 [I 2022-10-09 10:44:04] ...trap/engines/engine.py.230: loss: 0.19134 [I 2022-10-09 10:44:04] ...optimizers/trijoint.py.151: total_norm: 23.650642 [I 2022-10-09 10:44:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 1870/2304 [I 2022-10-09 10:44:12] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:06 [I 2022-10-09 10:44:12] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00292 [I 2022-10-09 10:44:12] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-09 10:44:12] ...optimizers/trijoint.py.151: total_norm: 10.968371 [I 2022-10-09 10:44:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 1880/2304 [I 2022-10-09 10:44:21] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:00 [I 2022-10-09 10:44:21] ...trap/engines/engine.py.229: process: 0.86121 | load: 0.00291 [I 2022-10-09 10:44:21] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-09 10:44:21] ...optimizers/trijoint.py.151: total_norm: 20.017620 [I 2022-10-09 10:44:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 1890/2304 [I 2022-10-09 10:44:29] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:05:51 [I 2022-10-09 10:44:29] ...trap/engines/engine.py.229: process: 0.84532 | load: 0.00293 [I 2022-10-09 10:44:29] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-09 10:44:29] ...optimizers/trijoint.py.151: total_norm: 16.890848 [I 2022-10-09 10:44:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 1900/2304 [I 2022-10-09 10:44:37] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:05:42 [I 2022-10-09 10:44:37] ...trap/engines/engine.py.229: process: 0.86879 | load: 0.00297 [I 2022-10-09 10:44:37] ...trap/engines/engine.py.230: loss: 0.25900 [I 2022-10-09 10:44:37] ...optimizers/trijoint.py.151: total_norm: 31.585396 [I 2022-10-09 10:44:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 1910/2304 [I 2022-10-09 10:44:46] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:32 [I 2022-10-09 10:44:46] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00291 [I 2022-10-09 10:44:46] ...trap/engines/engine.py.230: loss: 0.22915 [I 2022-10-09 10:44:46] ...optimizers/trijoint.py.151: total_norm: 19.966534 [I 2022-10-09 10:44:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 1920/2304 [I 2022-10-09 10:44:54] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:24 [I 2022-10-09 10:44:54] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00293 [I 2022-10-09 10:44:54] ...trap/engines/engine.py.230: loss: 0.18398 [I 2022-10-09 10:44:54] ...optimizers/trijoint.py.151: total_norm: 24.247507 [I 2022-10-09 10:45:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 1930/2304 [I 2022-10-09 10:45:03] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:17 [I 2022-10-09 10:45:03] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00291 [I 2022-10-09 10:45:03] ...trap/engines/engine.py.230: loss: 0.13823 [I 2022-10-09 10:45:03] ...optimizers/trijoint.py.151: total_norm: 14.389131 [I 2022-10-09 10:45:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 1940/2304 [I 2022-10-09 10:45:11] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:08 [I 2022-10-09 10:45:11] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00291 [I 2022-10-09 10:45:11] ...trap/engines/engine.py.230: loss: 0.19429 [I 2022-10-09 10:45:11] ...optimizers/trijoint.py.151: total_norm: 28.615065 [I 2022-10-09 10:45:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 1950/2304 [I 2022-10-09 10:45:20] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:04:59 [I 2022-10-09 10:45:20] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00296 [I 2022-10-09 10:45:20] ...trap/engines/engine.py.230: loss: 0.19185 [I 2022-10-09 10:45:20] ...optimizers/trijoint.py.151: total_norm: 16.203903 [I 2022-10-09 10:45:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 1960/2304 [I 2022-10-09 10:45:28] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:04:51 [I 2022-10-09 10:45:28] ...trap/engines/engine.py.229: process: 0.86644 | load: 0.00292 [I 2022-10-09 10:45:28] ...trap/engines/engine.py.230: loss: 0.13703 [I 2022-10-09 10:45:28] ...optimizers/trijoint.py.151: total_norm: 12.603574 [I 2022-10-09 10:45:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 1970/2304 [I 2022-10-09 10:45:37] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:43 [I 2022-10-09 10:45:37] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00292 [I 2022-10-09 10:45:37] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-09 10:45:37] ...optimizers/trijoint.py.151: total_norm: 16.312851 [I 2022-10-09 10:45:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 1980/2304 [I 2022-10-09 10:45:45] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:35 [I 2022-10-09 10:45:45] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00290 [I 2022-10-09 10:45:45] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-09 10:45:45] ...optimizers/trijoint.py.151: total_norm: 19.213945 [I 2022-10-09 10:45:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 1990/2304 [I 2022-10-09 10:45:54] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:25 [I 2022-10-09 10:45:54] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00292 [I 2022-10-09 10:45:54] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-09 10:45:54] ...optimizers/trijoint.py.151: total_norm: 9.875916 [I 2022-10-09 10:46:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 2000/2304 [I 2022-10-09 10:46:02] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:17 [I 2022-10-09 10:46:02] ...trap/engines/engine.py.229: process: 0.86494 | load: 0.00292 [I 2022-10-09 10:46:02] ...trap/engines/engine.py.230: loss: 0.15694 [I 2022-10-09 10:46:02] ...optimizers/trijoint.py.151: total_norm: 12.471886 [I 2022-10-09 10:46:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 2010/2304 [I 2022-10-09 10:46:11] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:08 [I 2022-10-09 10:46:11] ...trap/engines/engine.py.229: process: 0.83500 | load: 0.00293 [I 2022-10-09 10:46:11] ...trap/engines/engine.py.230: loss: 0.15513 [I 2022-10-09 10:46:11] ...optimizers/trijoint.py.151: total_norm: 19.212332 [I 2022-10-09 10:46:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 2020/2304 [I 2022-10-09 10:46:19] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:01 [I 2022-10-09 10:46:19] ...trap/engines/engine.py.229: process: 0.86582 | load: 0.00292 [I 2022-10-09 10:46:19] ...trap/engines/engine.py.230: loss: 0.21377 [I 2022-10-09 10:46:19] ...optimizers/trijoint.py.151: total_norm: 21.550655 [I 2022-10-09 10:46:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 2030/2304 [I 2022-10-09 10:46:28] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:03:51 [I 2022-10-09 10:46:28] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00291 [I 2022-10-09 10:46:28] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-09 10:46:28] ...optimizers/trijoint.py.151: total_norm: 8.545206 [I 2022-10-09 10:46:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 2040/2304 [I 2022-10-09 10:46:36] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:03:41 [I 2022-10-09 10:46:36] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00303 [I 2022-10-09 10:46:36] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-09 10:46:36] ...optimizers/trijoint.py.151: total_norm: 15.935240 [I 2022-10-09 10:46:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 2050/2304 [I 2022-10-09 10:46:44] ...trap/engines/engine.py.228: elapsed: 0:29:10 | left: 0:03:33 [I 2022-10-09 10:46:45] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00293 [I 2022-10-09 10:46:45] ...trap/engines/engine.py.230: loss: 0.23415 [I 2022-10-09 10:46:45] ...optimizers/trijoint.py.151: total_norm: 17.173399 [I 2022-10-09 10:46:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 2060/2304 [I 2022-10-09 10:46:53] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:25 [I 2022-10-09 10:46:53] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00292 [I 2022-10-09 10:46:53] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-09 10:46:53] ...optimizers/trijoint.py.151: total_norm: 21.488901 [I 2022-10-09 10:47:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 2070/2304 [I 2022-10-09 10:47:01] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:16 [I 2022-10-09 10:47:01] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00292 [I 2022-10-09 10:47:01] ...trap/engines/engine.py.230: loss: 0.12132 [I 2022-10-09 10:47:01] ...optimizers/trijoint.py.151: total_norm: 25.222786 [I 2022-10-09 10:47:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 2080/2304 [I 2022-10-09 10:47:10] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:08 [I 2022-10-09 10:47:10] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00291 [I 2022-10-09 10:47:10] ...trap/engines/engine.py.230: loss: 0.17104 [I 2022-10-09 10:47:10] ...optimizers/trijoint.py.151: total_norm: 24.296947 [I 2022-10-09 10:47:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 2090/2304 [I 2022-10-09 10:47:18] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:00 [I 2022-10-09 10:47:18] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00297 [I 2022-10-09 10:47:18] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-09 10:47:18] ...optimizers/trijoint.py.151: total_norm: 27.812294 [I 2022-10-09 10:47:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 2100/2304 [I 2022-10-09 10:47:27] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:02:51 [I 2022-10-09 10:47:27] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00297 [I 2022-10-09 10:47:27] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-09 10:47:27] ...optimizers/trijoint.py.151: total_norm: 14.891581 [I 2022-10-09 10:47:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 2110/2304 [I 2022-10-09 10:47:35] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:43 [I 2022-10-09 10:47:35] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00290 [I 2022-10-09 10:47:35] ...trap/engines/engine.py.230: loss: 0.13923 [I 2022-10-09 10:47:35] ...optimizers/trijoint.py.151: total_norm: 9.989217 [I 2022-10-09 10:47:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 2120/2304 [I 2022-10-09 10:47:43] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:35 [I 2022-10-09 10:47:43] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00291 [I 2022-10-09 10:47:43] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-09 10:47:43] ...optimizers/trijoint.py.151: total_norm: 14.978931 [I 2022-10-09 10:47:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 2130/2304 [I 2022-10-09 10:47:52] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:26 [I 2022-10-09 10:47:52] ...trap/engines/engine.py.229: process: 0.82561 | load: 0.00292 [I 2022-10-09 10:47:52] ...trap/engines/engine.py.230: loss: 0.15057 [I 2022-10-09 10:47:52] ...optimizers/trijoint.py.151: total_norm: 22.956823 [I 2022-10-09 10:48:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 2140/2304 [I 2022-10-09 10:48:00] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:17 [I 2022-10-09 10:48:00] ...trap/engines/engine.py.229: process: 0.84960 | load: 0.00294 [I 2022-10-09 10:48:00] ...trap/engines/engine.py.230: loss: 0.20551 [I 2022-10-09 10:48:00] ...optimizers/trijoint.py.151: total_norm: 16.746859 [I 2022-10-09 10:48:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 2150/2304 [I 2022-10-09 10:48:09] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:09 [I 2022-10-09 10:48:09] ...trap/engines/engine.py.229: process: 0.84057 | load: 0.00302 [I 2022-10-09 10:48:09] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-09 10:48:09] ...optimizers/trijoint.py.151: total_norm: 10.118301 [I 2022-10-09 10:48:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 2160/2304 [I 2022-10-09 10:48:17] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:00 [I 2022-10-09 10:48:17] ...trap/engines/engine.py.229: process: 0.82264 | load: 0.00289 [I 2022-10-09 10:48:17] ...trap/engines/engine.py.230: loss: 0.21209 [I 2022-10-09 10:48:17] ...optimizers/trijoint.py.151: total_norm: 14.870542 [I 2022-10-09 10:48:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 2170/2304 [I 2022-10-09 10:48:25] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:01:52 [I 2022-10-09 10:48:25] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00291 [I 2022-10-09 10:48:25] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-09 10:48:25] ...optimizers/trijoint.py.151: total_norm: 17.691582 [I 2022-10-09 10:48:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 2180/2304 [I 2022-10-09 10:48:34] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:01:44 [I 2022-10-09 10:48:34] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00292 [I 2022-10-09 10:48:34] ...trap/engines/engine.py.230: loss: 0.16974 [I 2022-10-09 10:48:34] ...optimizers/trijoint.py.151: total_norm: 12.773581 [I 2022-10-09 10:48:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 2190/2304 [I 2022-10-09 10:48:42] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:36 [I 2022-10-09 10:48:42] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00292 [I 2022-10-09 10:48:42] ...trap/engines/engine.py.230: loss: 0.14929 [I 2022-10-09 10:48:42] ...optimizers/trijoint.py.151: total_norm: 12.158190 [I 2022-10-09 10:48:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 2200/2304 [I 2022-10-09 10:48:51] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:27 [I 2022-10-09 10:48:51] ...trap/engines/engine.py.229: process: 0.82846 | load: 0.00293 [I 2022-10-09 10:48:51] ...trap/engines/engine.py.230: loss: 0.17554 [I 2022-10-09 10:48:51] ...optimizers/trijoint.py.151: total_norm: 21.853167 [I 2022-10-09 10:48:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 2210/2304 [I 2022-10-09 10:48:59] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:19 [I 2022-10-09 10:48:59] ...trap/engines/engine.py.229: process: 0.83986 | load: 0.00301 [I 2022-10-09 10:48:59] ...trap/engines/engine.py.230: loss: 0.14640 [I 2022-10-09 10:48:59] ...optimizers/trijoint.py.151: total_norm: 26.962933 [I 2022-10-09 10:49:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 2220/2304 [I 2022-10-09 10:49:08] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:10 [I 2022-10-09 10:49:08] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00296 [I 2022-10-09 10:49:08] ...trap/engines/engine.py.230: loss: 0.14066 [I 2022-10-09 10:49:08] ...optimizers/trijoint.py.151: total_norm: 11.434024 [I 2022-10-09 10:49:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 2230/2304 [I 2022-10-09 10:49:16] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:01 [I 2022-10-09 10:49:16] ...trap/engines/engine.py.229: process: 0.82410 | load: 0.00292 [I 2022-10-09 10:49:16] ...trap/engines/engine.py.230: loss: 0.24102 [I 2022-10-09 10:49:16] ...optimizers/trijoint.py.151: total_norm: 13.716849 [I 2022-10-09 10:49:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 2240/2304 [I 2022-10-09 10:49:24] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:00:53 [I 2022-10-09 10:49:24] ...trap/engines/engine.py.229: process: 0.84414 | load: 0.00291 [I 2022-10-09 10:49:24] ...trap/engines/engine.py.230: loss: 0.23502 [I 2022-10-09 10:49:24] ...optimizers/trijoint.py.151: total_norm: 20.166506 [I 2022-10-09 10:49:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 2250/2304 [I 2022-10-09 10:49:33] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:00:45 [I 2022-10-09 10:49:33] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00293 [I 2022-10-09 10:49:33] ...trap/engines/engine.py.230: loss: 0.13985 [I 2022-10-09 10:49:33] ...optimizers/trijoint.py.151: total_norm: 17.948984 [I 2022-10-09 10:49:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 2260/2304 [I 2022-10-09 10:49:41] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:37 [I 2022-10-09 10:49:41] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00292 [I 2022-10-09 10:49:41] ...trap/engines/engine.py.230: loss: 0.21263 [I 2022-10-09 10:49:41] ...optimizers/trijoint.py.151: total_norm: 18.195269 [I 2022-10-09 10:49:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 2270/2304 [I 2022-10-09 10:49:50] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:28 [I 2022-10-09 10:49:50] ...trap/engines/engine.py.229: process: 0.82779 | load: 0.00299 [I 2022-10-09 10:49:50] ...trap/engines/engine.py.230: loss: 0.10872 [I 2022-10-09 10:49:50] ...optimizers/trijoint.py.151: total_norm: 13.261316 [I 2022-10-09 10:49:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 2280/2304 [I 2022-10-09 10:49:58] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:20 [I 2022-10-09 10:49:58] ...trap/engines/engine.py.229: process: 0.83594 | load: 0.00291 [I 2022-10-09 10:49:58] ...trap/engines/engine.py.230: loss: 0.15776 [I 2022-10-09 10:49:58] ...optimizers/trijoint.py.151: total_norm: 14.518078 [I 2022-10-09 10:50:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 2290/2304 [I 2022-10-09 10:50:07] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:11 [I 2022-10-09 10:50:07] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00291 [I 2022-10-09 10:50:07] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-09 10:50:07] ...optimizers/trijoint.py.151: total_norm: 12.695775 [I 2022-10-09 10:50:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 2300/2304 [I 2022-10-09 10:50:15] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:03 [I 2022-10-09 10:50:15] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00015 [I 2022-10-09 10:50:15] ...trap/engines/engine.py.230: loss: 0.10840 [I 2022-10-09 10:50:15] ...optimizers/trijoint.py.151: total_norm: 17.851944 [I 2022-10-09 10:50:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 2304/2304 [I 2022-10-09 10:50:18] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:00 [I 2022-10-09 10:50:18] ...trap/engines/engine.py.229: process: 0.82572 | load: 0.00016 [I 2022-10-09 10:50:18] ...trap/engines/engine.py.230: loss: 0.11079 [I 2022-10-09 10:50:18] ...optimizers/trijoint.py.151: total_norm: 14.100554 [S 2022-10-09 10:50:20] ...trap/engines/engine.py.129: train_epoch.epoch: 37 [S 2022-10-09 10:50:20] ...trap/engines/engine.py.129: train_epoch.loss: 0.1601860837635146 [I 2022-10-09 10:50:20] ...s/criterions/triplet.py.84: margin = 0.23500000000000004 [I 2022-10-09 10:50:20] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 37 [I 2022-10-09 10:50:20] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 10:50:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 10:50:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 10:50:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 10:50:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 10:50:23] ...trap/engines/engine.py.321: eval: epoch 37 | batch 0/510 [I 2022-10-09 10:50:23] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:56 [I 2022-10-09 10:50:23] ...trap/engines/engine.py.326: process: 3.28744 | load: 2.79510 [I 2022-10-09 10:50:28] ...trap/engines/engine.py.321: eval: epoch 37 | batch 10/510 [I 2022-10-09 10:50:28] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:25 [I 2022-10-09 10:50:28] ...trap/engines/engine.py.326: process: 0.41509 | load: 0.00017 [I 2022-10-09 10:50:33] ...trap/engines/engine.py.321: eval: epoch 37 | batch 20/510 [I 2022-10-09 10:50:33] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:10 [I 2022-10-09 10:50:33] ...trap/engines/engine.py.326: process: 0.61445 | load: 0.00018 [I 2022-10-09 10:50:39] ...trap/engines/engine.py.321: eval: epoch 37 | batch 30/510 [I 2022-10-09 10:50:39] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:05:00 [I 2022-10-09 10:50:39] ...trap/engines/engine.py.326: process: 0.75843 | load: 0.00017 [I 2022-10-09 10:50:47] ...trap/engines/engine.py.321: eval: epoch 37 | batch 40/510 [I 2022-10-09 10:50:47] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:06:07 [I 2022-10-09 10:50:47] ...trap/engines/engine.py.326: process: 0.61278 | load: 0.00019 [I 2022-10-09 10:50:53] ...trap/engines/engine.py.321: eval: epoch 37 | batch 50/510 [I 2022-10-09 10:50:53] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:04:24 [I 2022-10-09 10:50:53] ...trap/engines/engine.py.326: process: 0.54296 | load: 0.00019 [I 2022-10-09 10:50:59] ...trap/engines/engine.py.321: eval: epoch 37 | batch 60/510 [I 2022-10-09 10:50:59] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:15 [I 2022-10-09 10:50:59] ...trap/engines/engine.py.326: process: 0.59989 | load: 0.00017 [I 2022-10-09 10:51:04] ...trap/engines/engine.py.321: eval: epoch 37 | batch 70/510 [I 2022-10-09 10:51:04] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:56 [I 2022-10-09 10:51:04] ...trap/engines/engine.py.326: process: 0.54224 | load: 0.00016 [I 2022-10-09 10:51:08] ...trap/engines/engine.py.321: eval: epoch 37 | batch 80/510 [I 2022-10-09 10:51:08] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:15 [I 2022-10-09 10:51:08] ...trap/engines/engine.py.326: process: 0.50334 | load: 0.00018 [I 2022-10-09 10:51:13] ...trap/engines/engine.py.321: eval: epoch 37 | batch 90/510 [I 2022-10-09 10:51:13] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:00 [I 2022-10-09 10:51:13] ...trap/engines/engine.py.326: process: 0.42992 | load: 0.00018 [I 2022-10-09 10:51:17] ...trap/engines/engine.py.321: eval: epoch 37 | batch 100/510 [I 2022-10-09 10:51:17] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:47 [I 2022-10-09 10:51:17] ...trap/engines/engine.py.326: process: 0.31271 | load: 0.00018 [I 2022-10-09 10:51:20] ...trap/engines/engine.py.321: eval: epoch 37 | batch 110/510 [I 2022-10-09 10:51:20] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:10 [I 2022-10-09 10:51:20] ...trap/engines/engine.py.326: process: 0.30600 | load: 0.00018 [I 2022-10-09 10:51:23] ...trap/engines/engine.py.321: eval: epoch 37 | batch 120/510 [I 2022-10-09 10:51:23] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-09 10:51:23] ...trap/engines/engine.py.326: process: 0.31458 | load: 0.00020 [I 2022-10-09 10:51:26] ...trap/engines/engine.py.321: eval: epoch 37 | batch 130/510 [I 2022-10-09 10:51:26] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:59 [I 2022-10-09 10:51:26] ...trap/engines/engine.py.326: process: 0.31282 | load: 0.00019 [I 2022-10-09 10:51:30] ...trap/engines/engine.py.321: eval: epoch 37 | batch 140/510 [I 2022-10-09 10:51:30] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:57 [I 2022-10-09 10:51:30] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00018 [I 2022-10-09 10:51:33] ...trap/engines/engine.py.321: eval: epoch 37 | batch 150/510 [I 2022-10-09 10:51:33] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-09 10:51:33] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00019 [I 2022-10-09 10:51:36] ...trap/engines/engine.py.321: eval: epoch 37 | batch 160/510 [I 2022-10-09 10:51:36] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:50 [I 2022-10-09 10:51:36] ...trap/engines/engine.py.326: process: 0.31767 | load: 0.00018 [I 2022-10-09 10:51:39] ...trap/engines/engine.py.321: eval: epoch 37 | batch 170/510 [I 2022-10-09 10:51:39] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:46 [I 2022-10-09 10:51:39] ...trap/engines/engine.py.326: process: 0.31070 | load: 0.00019 [I 2022-10-09 10:51:42] ...trap/engines/engine.py.321: eval: epoch 37 | batch 180/510 [I 2022-10-09 10:51:42] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:43 [I 2022-10-09 10:51:42] ...trap/engines/engine.py.326: process: 0.31265 | load: 0.00019 [I 2022-10-09 10:51:45] ...trap/engines/engine.py.321: eval: epoch 37 | batch 190/510 [I 2022-10-09 10:51:45] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:41 [I 2022-10-09 10:51:45] ...trap/engines/engine.py.326: process: 0.31827 | load: 0.00019 [I 2022-10-09 10:51:49] ...trap/engines/engine.py.321: eval: epoch 37 | batch 200/510 [I 2022-10-09 10:51:49] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:38 [I 2022-10-09 10:51:49] ...trap/engines/engine.py.326: process: 0.31561 | load: 0.00020 [I 2022-10-09 10:51:52] ...trap/engines/engine.py.321: eval: epoch 37 | batch 210/510 [I 2022-10-09 10:51:52] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:34 [I 2022-10-09 10:51:52] ...trap/engines/engine.py.326: process: 0.31462 | load: 0.00019 [I 2022-10-09 10:51:55] ...trap/engines/engine.py.321: eval: epoch 37 | batch 220/510 [I 2022-10-09 10:51:55] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:32 [I 2022-10-09 10:51:55] ...trap/engines/engine.py.326: process: 0.31296 | load: 0.00019 [I 2022-10-09 10:51:58] ...trap/engines/engine.py.321: eval: epoch 37 | batch 230/510 [I 2022-10-09 10:51:58] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:28 [I 2022-10-09 10:51:58] ...trap/engines/engine.py.326: process: 0.31579 | load: 0.00020 [I 2022-10-09 10:52:01] ...trap/engines/engine.py.321: eval: epoch 37 | batch 240/510 [I 2022-10-09 10:52:01] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:25 [I 2022-10-09 10:52:01] ...trap/engines/engine.py.326: process: 0.31558 | load: 0.00020 [I 2022-10-09 10:52:04] ...trap/engines/engine.py.321: eval: epoch 37 | batch 250/510 [I 2022-10-09 10:52:04] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:21 [I 2022-10-09 10:52:04] ...trap/engines/engine.py.326: process: 0.31516 | load: 0.00019 [I 2022-10-09 10:52:08] ...trap/engines/engine.py.321: eval: epoch 37 | batch 260/510 [I 2022-10-09 10:52:08] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:19 [I 2022-10-09 10:52:08] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00019 [I 2022-10-09 10:52:11] ...trap/engines/engine.py.321: eval: epoch 37 | batch 270/510 [I 2022-10-09 10:52:11] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:15 [I 2022-10-09 10:52:11] ...trap/engines/engine.py.326: process: 0.32054 | load: 0.00019 [I 2022-10-09 10:52:14] ...trap/engines/engine.py.321: eval: epoch 37 | batch 280/510 [I 2022-10-09 10:52:14] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:12 [I 2022-10-09 10:52:14] ...trap/engines/engine.py.326: process: 0.31762 | load: 0.00018 [I 2022-10-09 10:52:17] ...trap/engines/engine.py.321: eval: epoch 37 | batch 290/510 [I 2022-10-09 10:52:17] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:09 [I 2022-10-09 10:52:17] ...trap/engines/engine.py.326: process: 0.31602 | load: 0.00019 [I 2022-10-09 10:52:20] ...trap/engines/engine.py.321: eval: epoch 37 | batch 300/510 [I 2022-10-09 10:52:20] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-09 10:52:20] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00019 [I 2022-10-09 10:52:23] ...trap/engines/engine.py.321: eval: epoch 37 | batch 310/510 [I 2022-10-09 10:52:23] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:03 [I 2022-10-09 10:52:23] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00020 [I 2022-10-09 10:52:27] ...trap/engines/engine.py.321: eval: epoch 37 | batch 320/510 [I 2022-10-09 10:52:27] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:59 [I 2022-10-09 10:52:27] ...trap/engines/engine.py.326: process: 0.31304 | load: 0.00019 [I 2022-10-09 10:52:30] ...trap/engines/engine.py.321: eval: epoch 37 | batch 330/510 [I 2022-10-09 10:52:30] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-09 10:52:30] ...trap/engines/engine.py.326: process: 0.31830 | load: 0.00019 [I 2022-10-09 10:52:33] ...trap/engines/engine.py.321: eval: epoch 37 | batch 340/510 [I 2022-10-09 10:52:33] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:53 [I 2022-10-09 10:52:33] ...trap/engines/engine.py.326: process: 0.32516 | load: 0.00019 [I 2022-10-09 10:52:36] ...trap/engines/engine.py.321: eval: epoch 37 | batch 350/510 [I 2022-10-09 10:52:36] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:50 [I 2022-10-09 10:52:36] ...trap/engines/engine.py.326: process: 0.31895 | load: 0.00019 [I 2022-10-09 10:52:39] ...trap/engines/engine.py.321: eval: epoch 37 | batch 360/510 [I 2022-10-09 10:52:39] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-09 10:52:39] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00019 [I 2022-10-09 10:52:42] ...trap/engines/engine.py.321: eval: epoch 37 | batch 370/510 [I 2022-10-09 10:52:42] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-09 10:52:42] ...trap/engines/engine.py.326: process: 0.31385 | load: 0.00019 [I 2022-10-09 10:52:46] ...trap/engines/engine.py.321: eval: epoch 37 | batch 380/510 [I 2022-10-09 10:52:46] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-09 10:52:46] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00018 [I 2022-10-09 10:52:49] ...trap/engines/engine.py.321: eval: epoch 37 | batch 390/510 [I 2022-10-09 10:52:49] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-09 10:52:49] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00019 [I 2022-10-09 10:52:52] ...trap/engines/engine.py.321: eval: epoch 37 | batch 400/510 [I 2022-10-09 10:52:52] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:34 [I 2022-10-09 10:52:52] ...trap/engines/engine.py.326: process: 0.31467 | load: 0.00020 [I 2022-10-09 10:52:55] ...trap/engines/engine.py.321: eval: epoch 37 | batch 410/510 [I 2022-10-09 10:52:55] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-09 10:52:55] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00019 [I 2022-10-09 10:52:58] ...trap/engines/engine.py.321: eval: epoch 37 | batch 420/510 [I 2022-10-09 10:52:58] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-09 10:52:58] ...trap/engines/engine.py.326: process: 0.31557 | load: 0.00020 [I 2022-10-09 10:53:01] ...trap/engines/engine.py.321: eval: epoch 37 | batch 430/510 [I 2022-10-09 10:53:02] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-09 10:53:02] ...trap/engines/engine.py.326: process: 0.31447 | load: 0.00020 [I 2022-10-09 10:53:05] ...trap/engines/engine.py.321: eval: epoch 37 | batch 440/510 [I 2022-10-09 10:53:05] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-09 10:53:05] ...trap/engines/engine.py.326: process: 0.31744 | load: 0.00018 [I 2022-10-09 10:53:08] ...trap/engines/engine.py.321: eval: epoch 37 | batch 450/510 [I 2022-10-09 10:53:08] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 10:53:08] ...trap/engines/engine.py.326: process: 0.31433 | load: 0.00020 [I 2022-10-09 10:53:11] ...trap/engines/engine.py.321: eval: epoch 37 | batch 460/510 [I 2022-10-09 10:53:11] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-09 10:53:11] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00020 [I 2022-10-09 10:53:14] ...trap/engines/engine.py.321: eval: epoch 37 | batch 470/510 [I 2022-10-09 10:53:14] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-09 10:53:14] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00020 [I 2022-10-09 10:53:17] ...trap/engines/engine.py.321: eval: epoch 37 | batch 480/510 [I 2022-10-09 10:53:17] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-09 10:53:17] ...trap/engines/engine.py.326: process: 0.31627 | load: 0.00019 [I 2022-10-09 10:53:21] ...trap/engines/engine.py.321: eval: epoch 37 | batch 490/510 [I 2022-10-09 10:53:21] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 10:53:21] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00020 [I 2022-10-09 10:53:24] ...trap/engines/engine.py.321: eval: epoch 37 | batch 500/510 [I 2022-10-09 10:53:24] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 10:53:24] ...trap/engines/engine.py.326: process: 0.31012 | load: 0.00019 [I 2022-10-09 10:53:27] ...trap/engines/engine.py.321: eval: epoch 37 | batch 510/510 [I 2022-10-09 10:53:27] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 10:53:27] ...trap/engines/engine.py.326: process: 0.30986 | load: 0.00015 [S 2022-10-09 10:53:28] ...trap/engines/engine.py.132: eval_epoch.epoch: 37 [I 2022-10-09 10:53:28] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.701700 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.900400 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936400 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.693000 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897100 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011367 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009319 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008902 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009518 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012724 [S 2022-10-09 10:53:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009741 [I 2022-10-09 10:54:10] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 10:54:10] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 10:54:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 10:54:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 10:54:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 10:54:13] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 10:54:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 10:54:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 10:54:19] ...trap/engines/engine.py.163: Training model on trainset for epoch 38 [I 2022-10-09 10:54:19] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 10:54:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 0/2304 [I 2022-10-09 10:54:22] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:09 [I 2022-10-09 10:54:22] ...trap/engines/engine.py.229: process: 2.71265 | load: 1.81538 [I 2022-10-09 10:54:22] ...trap/engines/engine.py.230: loss: 0.19712 [I 2022-10-09 10:54:22] ...optimizers/trijoint.py.151: total_norm: 22.333683 [I 2022-10-09 10:54:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 10/2304 [I 2022-10-09 10:54:30] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:50 [I 2022-10-09 10:54:30] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00544 [I 2022-10-09 10:54:30] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-09 10:54:30] ...optimizers/trijoint.py.151: total_norm: 16.717747 [I 2022-10-09 10:54:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 20/2304 [I 2022-10-09 10:54:39] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:49 [I 2022-10-09 10:54:39] ...trap/engines/engine.py.229: process: 0.82771 | load: 0.00474 [I 2022-10-09 10:54:39] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-09 10:54:39] ...optimizers/trijoint.py.151: total_norm: 7.109263 [I 2022-10-09 10:54:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 30/2304 [I 2022-10-09 10:54:47] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:31:56 [I 2022-10-09 10:54:47] ...trap/engines/engine.py.229: process: 0.83766 | load: 0.00362 [I 2022-10-09 10:54:47] ...trap/engines/engine.py.230: loss: 0.20542 [I 2022-10-09 10:54:47] ...optimizers/trijoint.py.151: total_norm: 27.265299 [I 2022-10-09 10:54:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 40/2304 [I 2022-10-09 10:54:55] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:42 [I 2022-10-09 10:54:55] ...trap/engines/engine.py.229: process: 0.82864 | load: 0.00344 [I 2022-10-09 10:54:55] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-09 10:54:55] ...optimizers/trijoint.py.151: total_norm: 14.925677 [I 2022-10-09 10:55:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 50/2304 [I 2022-10-09 10:55:04] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:41 [I 2022-10-09 10:55:04] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00416 [I 2022-10-09 10:55:04] ...trap/engines/engine.py.230: loss: 0.18873 [I 2022-10-09 10:55:04] ...optimizers/trijoint.py.151: total_norm: 16.867456 [I 2022-10-09 10:55:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 60/2304 [I 2022-10-09 10:55:12] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:29 [I 2022-10-09 10:55:12] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00334 [I 2022-10-09 10:55:12] ...trap/engines/engine.py.230: loss: 0.19857 [I 2022-10-09 10:55:12] ...optimizers/trijoint.py.151: total_norm: 14.559497 [I 2022-10-09 10:55:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 70/2304 [I 2022-10-09 10:55:21] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:24 [I 2022-10-09 10:55:21] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00368 [I 2022-10-09 10:55:21] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-09 10:55:21] ...optimizers/trijoint.py.151: total_norm: 14.976870 [I 2022-10-09 10:55:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 80/2304 [I 2022-10-09 10:55:29] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:12 [I 2022-10-09 10:55:29] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00292 [I 2022-10-09 10:55:29] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-09 10:55:29] ...optimizers/trijoint.py.151: total_norm: 7.940152 [I 2022-10-09 10:55:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 90/2304 [I 2022-10-09 10:55:38] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:00 [I 2022-10-09 10:55:38] ...trap/engines/engine.py.229: process: 0.83933 | load: 0.00303 [I 2022-10-09 10:55:38] ...trap/engines/engine.py.230: loss: 0.16106 [I 2022-10-09 10:55:38] ...optimizers/trijoint.py.151: total_norm: 17.926485 [I 2022-10-09 10:55:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 100/2304 [I 2022-10-09 10:55:46] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:30:46 [I 2022-10-09 10:55:46] ...trap/engines/engine.py.229: process: 0.84082 | load: 0.00320 [I 2022-10-09 10:55:46] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-09 10:55:46] ...optimizers/trijoint.py.151: total_norm: 9.161309 [I 2022-10-09 10:55:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 110/2304 [I 2022-10-09 10:55:54] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:42 [I 2022-10-09 10:55:54] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00291 [I 2022-10-09 10:55:54] ...trap/engines/engine.py.230: loss: 0.19980 [I 2022-10-09 10:55:54] ...optimizers/trijoint.py.151: total_norm: 21.097853 [I 2022-10-09 10:56:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 120/2304 [I 2022-10-09 10:56:03] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:33 [I 2022-10-09 10:56:03] ...trap/engines/engine.py.229: process: 0.82920 | load: 0.00313 [I 2022-10-09 10:56:03] ...trap/engines/engine.py.230: loss: 0.17586 [I 2022-10-09 10:56:03] ...optimizers/trijoint.py.151: total_norm: 46.364670 [I 2022-10-09 10:56:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 130/2304 [I 2022-10-09 10:56:11] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:21 [I 2022-10-09 10:56:11] ...trap/engines/engine.py.229: process: 0.83624 | load: 0.00346 [I 2022-10-09 10:56:11] ...trap/engines/engine.py.230: loss: 0.13879 [I 2022-10-09 10:56:11] ...optimizers/trijoint.py.151: total_norm: 14.039466 [I 2022-10-09 10:56:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 140/2304 [I 2022-10-09 10:56:20] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:14 [I 2022-10-09 10:56:20] ...trap/engines/engine.py.229: process: 0.83770 | load: 0.00308 [I 2022-10-09 10:56:20] ...trap/engines/engine.py.230: loss: 0.22358 [I 2022-10-09 10:56:20] ...optimizers/trijoint.py.151: total_norm: 28.076481 [I 2022-10-09 10:56:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 150/2304 [I 2022-10-09 10:56:28] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:18 [I 2022-10-09 10:56:28] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00311 [I 2022-10-09 10:56:28] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-09 10:56:28] ...optimizers/trijoint.py.151: total_norm: 19.451666 [I 2022-10-09 10:56:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 160/2304 [I 2022-10-09 10:56:36] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:01 [I 2022-10-09 10:56:36] ...trap/engines/engine.py.229: process: 0.83918 | load: 0.00316 [I 2022-10-09 10:56:36] ...trap/engines/engine.py.230: loss: 0.16731 [I 2022-10-09 10:56:36] ...optimizers/trijoint.py.151: total_norm: 16.385851 [I 2022-10-09 10:56:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 170/2304 [I 2022-10-09 10:56:45] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:29:55 [I 2022-10-09 10:56:45] ...trap/engines/engine.py.229: process: 0.83663 | load: 0.00328 [I 2022-10-09 10:56:45] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-09 10:56:45] ...optimizers/trijoint.py.151: total_norm: 13.737916 [I 2022-10-09 10:56:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 180/2304 [I 2022-10-09 10:56:53] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:45 [I 2022-10-09 10:56:53] ...trap/engines/engine.py.229: process: 0.84680 | load: 0.00303 [I 2022-10-09 10:56:53] ...trap/engines/engine.py.230: loss: 0.09292 [I 2022-10-09 10:56:53] ...optimizers/trijoint.py.151: total_norm: 15.836011 [I 2022-10-09 10:57:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 190/2304 [I 2022-10-09 10:57:02] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:38 [I 2022-10-09 10:57:02] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00299 [I 2022-10-09 10:57:02] ...trap/engines/engine.py.230: loss: 0.17042 [I 2022-10-09 10:57:02] ...optimizers/trijoint.py.151: total_norm: 12.791178 [I 2022-10-09 10:57:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 200/2304 [I 2022-10-09 10:57:10] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:30 [I 2022-10-09 10:57:10] ...trap/engines/engine.py.229: process: 0.83152 | load: 0.00311 [I 2022-10-09 10:57:10] ...trap/engines/engine.py.230: loss: 0.20016 [I 2022-10-09 10:57:10] ...optimizers/trijoint.py.151: total_norm: 21.379251 [I 2022-10-09 10:57:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 210/2304 [I 2022-10-09 10:57:19] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:11 [I 2022-10-09 10:57:19] ...trap/engines/engine.py.229: process: 0.83132 | load: 0.00316 [I 2022-10-09 10:57:19] ...trap/engines/engine.py.230: loss: 0.13569 [I 2022-10-09 10:57:19] ...optimizers/trijoint.py.151: total_norm: 15.295707 [I 2022-10-09 10:57:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 220/2304 [I 2022-10-09 10:57:27] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:29:05 [I 2022-10-09 10:57:27] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00336 [I 2022-10-09 10:57:27] ...trap/engines/engine.py.230: loss: 0.17930 [I 2022-10-09 10:57:27] ...optimizers/trijoint.py.151: total_norm: 11.728386 [I 2022-10-09 10:57:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 230/2304 [I 2022-10-09 10:57:35] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:10 [I 2022-10-09 10:57:35] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00299 [I 2022-10-09 10:57:35] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-09 10:57:35] ...optimizers/trijoint.py.151: total_norm: 21.575575 [I 2022-10-09 10:57:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 240/2304 [I 2022-10-09 10:57:44] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:29:04 [I 2022-10-09 10:57:44] ...trap/engines/engine.py.229: process: 0.85013 | load: 0.00292 [I 2022-10-09 10:57:44] ...trap/engines/engine.py.230: loss: 0.17166 [I 2022-10-09 10:57:44] ...optimizers/trijoint.py.151: total_norm: 12.824835 [I 2022-10-09 10:57:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 250/2304 [I 2022-10-09 10:57:52] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:47 [I 2022-10-09 10:57:52] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00305 [I 2022-10-09 10:57:52] ...trap/engines/engine.py.230: loss: 0.24751 [I 2022-10-09 10:57:52] ...optimizers/trijoint.py.151: total_norm: 28.070751 [I 2022-10-09 10:58:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 260/2304 [I 2022-10-09 10:58:01] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:41 [I 2022-10-09 10:58:01] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00340 [I 2022-10-09 10:58:01] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-09 10:58:01] ...optimizers/trijoint.py.151: total_norm: 48.989746 [I 2022-10-09 10:58:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 270/2304 [I 2022-10-09 10:58:09] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:43 [I 2022-10-09 10:58:09] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00297 [I 2022-10-09 10:58:09] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-09 10:58:09] ...optimizers/trijoint.py.151: total_norm: 9.875059 [I 2022-10-09 10:58:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 280/2304 [I 2022-10-09 10:58:18] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:35 [I 2022-10-09 10:58:18] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00294 [I 2022-10-09 10:58:18] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-09 10:58:18] ...optimizers/trijoint.py.151: total_norm: 19.909483 [I 2022-10-09 10:58:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 290/2304 [I 2022-10-09 10:58:26] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:21 [I 2022-10-09 10:58:26] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00292 [I 2022-10-09 10:58:26] ...trap/engines/engine.py.230: loss: 0.17165 [I 2022-10-09 10:58:26] ...optimizers/trijoint.py.151: total_norm: 20.495070 [I 2022-10-09 10:58:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 300/2304 [I 2022-10-09 10:58:34] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:52 [I 2022-10-09 10:58:34] ...trap/engines/engine.py.229: process: 0.83164 | load: 0.00296 [I 2022-10-09 10:58:34] ...trap/engines/engine.py.230: loss: 0.22542 [I 2022-10-09 10:58:34] ...optimizers/trijoint.py.151: total_norm: 32.120281 [I 2022-10-09 10:58:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 310/2304 [I 2022-10-09 10:58:43] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:27:58 [I 2022-10-09 10:58:43] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00294 [I 2022-10-09 10:58:43] ...trap/engines/engine.py.230: loss: 0.21678 [I 2022-10-09 10:58:43] ...optimizers/trijoint.py.151: total_norm: 20.738708 [I 2022-10-09 10:58:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 320/2304 [I 2022-10-09 10:58:51] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:55 [I 2022-10-09 10:58:51] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00307 [I 2022-10-09 10:58:51] ...trap/engines/engine.py.230: loss: 0.15020 [I 2022-10-09 10:58:51] ...optimizers/trijoint.py.151: total_norm: 14.201682 [I 2022-10-09 10:59:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 330/2304 [I 2022-10-09 10:59:00] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:53 [I 2022-10-09 10:59:00] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00294 [I 2022-10-09 10:59:00] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-09 10:59:00] ...optimizers/trijoint.py.151: total_norm: 98.775146 [I 2022-10-09 10:59:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 340/2304 [I 2022-10-09 10:59:08] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:42 [I 2022-10-09 10:59:08] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00293 [I 2022-10-09 10:59:08] ...trap/engines/engine.py.230: loss: 0.13758 [I 2022-10-09 10:59:08] ...optimizers/trijoint.py.151: total_norm: 11.034685 [I 2022-10-09 10:59:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 350/2304 [I 2022-10-09 10:59:17] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:25 [I 2022-10-09 10:59:17] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00296 [I 2022-10-09 10:59:17] ...trap/engines/engine.py.230: loss: 0.20797 [I 2022-10-09 10:59:17] ...optimizers/trijoint.py.151: total_norm: 12.327870 [I 2022-10-09 10:59:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 360/2304 [I 2022-10-09 10:59:25] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:23 [I 2022-10-09 10:59:25] ...trap/engines/engine.py.229: process: 0.83417 | load: 0.00296 [I 2022-10-09 10:59:25] ...trap/engines/engine.py.230: loss: 0.18074 [I 2022-10-09 10:59:25] ...optimizers/trijoint.py.151: total_norm: 19.152395 [I 2022-10-09 10:59:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 370/2304 [I 2022-10-09 10:59:34] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:06 [I 2022-10-09 10:59:34] ...trap/engines/engine.py.229: process: 0.83340 | load: 0.00302 [I 2022-10-09 10:59:34] ...trap/engines/engine.py.230: loss: 0.20637 [I 2022-10-09 10:59:34] ...optimizers/trijoint.py.151: total_norm: 41.385178 [I 2022-10-09 10:59:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 380/2304 [I 2022-10-09 10:59:42] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:06 [I 2022-10-09 10:59:42] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00299 [I 2022-10-09 10:59:42] ...trap/engines/engine.py.230: loss: 0.17648 [I 2022-10-09 10:59:42] ...optimizers/trijoint.py.151: total_norm: 20.096870 [I 2022-10-09 10:59:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 390/2304 [I 2022-10-09 10:59:51] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:54 [I 2022-10-09 10:59:51] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00305 [I 2022-10-09 10:59:51] ...trap/engines/engine.py.230: loss: 0.18258 [I 2022-10-09 10:59:51] ...optimizers/trijoint.py.151: total_norm: 30.257717 [I 2022-10-09 10:59:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 400/2304 [I 2022-10-09 10:59:59] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:43 [I 2022-10-09 10:59:59] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00295 [I 2022-10-09 10:59:59] ...trap/engines/engine.py.230: loss: 0.18068 [I 2022-10-09 10:59:59] ...optimizers/trijoint.py.151: total_norm: 33.935650 [I 2022-10-09 11:00:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 410/2304 [I 2022-10-09 11:00:07] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:38 [I 2022-10-09 11:00:07] ...trap/engines/engine.py.229: process: 0.84496 | load: 0.00299 [I 2022-10-09 11:00:07] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-09 11:00:07] ...optimizers/trijoint.py.151: total_norm: 9.899074 [I 2022-10-09 11:00:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 420/2304 [I 2022-10-09 11:00:16] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:22 [I 2022-10-09 11:00:16] ...trap/engines/engine.py.229: process: 0.83004 | load: 0.00300 [I 2022-10-09 11:00:16] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-09 11:00:16] ...optimizers/trijoint.py.151: total_norm: 17.227570 [I 2022-10-09 11:00:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 430/2304 [I 2022-10-09 11:00:24] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:21 [I 2022-10-09 11:00:24] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00297 [I 2022-10-09 11:00:24] ...trap/engines/engine.py.230: loss: 0.13069 [I 2022-10-09 11:00:24] ...optimizers/trijoint.py.151: total_norm: 14.477580 [I 2022-10-09 11:00:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 440/2304 [I 2022-10-09 11:00:33] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:14 [I 2022-10-09 11:00:33] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00296 [I 2022-10-09 11:00:33] ...trap/engines/engine.py.230: loss: 0.22363 [I 2022-10-09 11:00:33] ...optimizers/trijoint.py.151: total_norm: 29.397207 [I 2022-10-09 11:00:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 450/2304 [I 2022-10-09 11:00:41] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:57 [I 2022-10-09 11:00:41] ...trap/engines/engine.py.229: process: 0.83147 | load: 0.00298 [I 2022-10-09 11:00:41] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-09 11:00:41] ...optimizers/trijoint.py.151: total_norm: 10.893609 [I 2022-10-09 11:00:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 460/2304 [I 2022-10-09 11:00:50] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:50 [I 2022-10-09 11:00:50] ...trap/engines/engine.py.229: process: 0.83510 | load: 0.00297 [I 2022-10-09 11:00:50] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-09 11:00:50] ...optimizers/trijoint.py.151: total_norm: 16.044575 [I 2022-10-09 11:00:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 470/2304 [I 2022-10-09 11:00:58] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:42 [I 2022-10-09 11:00:58] ...trap/engines/engine.py.229: process: 0.83885 | load: 0.00296 [I 2022-10-09 11:00:58] ...trap/engines/engine.py.230: loss: 0.15315 [I 2022-10-09 11:00:58] ...optimizers/trijoint.py.151: total_norm: 13.374653 [I 2022-10-09 11:01:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 480/2304 [I 2022-10-09 11:01:06] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:32 [I 2022-10-09 11:01:06] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00303 [I 2022-10-09 11:01:06] ...trap/engines/engine.py.230: loss: 0.20312 [I 2022-10-09 11:01:06] ...optimizers/trijoint.py.151: total_norm: 22.532707 [I 2022-10-09 11:01:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 490/2304 [I 2022-10-09 11:01:15] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:26 [I 2022-10-09 11:01:15] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00289 [I 2022-10-09 11:01:15] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-09 11:01:15] ...optimizers/trijoint.py.151: total_norm: 22.880699 [I 2022-10-09 11:01:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 500/2304 [I 2022-10-09 11:01:23] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:23 [I 2022-10-09 11:01:23] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00290 [I 2022-10-09 11:01:23] ...trap/engines/engine.py.230: loss: 0.14479 [I 2022-10-09 11:01:23] ...optimizers/trijoint.py.151: total_norm: 20.230312 [I 2022-10-09 11:01:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 510/2304 [I 2022-10-09 11:01:32] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:07 [I 2022-10-09 11:01:32] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00291 [I 2022-10-09 11:01:32] ...trap/engines/engine.py.230: loss: 0.19652 [I 2022-10-09 11:01:32] ...optimizers/trijoint.py.151: total_norm: 9.056415 [I 2022-10-09 11:01:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 520/2304 [I 2022-10-09 11:01:40] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:00 [I 2022-10-09 11:01:40] ...trap/engines/engine.py.229: process: 0.83662 | load: 0.00295 [I 2022-10-09 11:01:40] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-09 11:01:40] ...optimizers/trijoint.py.151: total_norm: 27.365589 [I 2022-10-09 11:01:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 530/2304 [I 2022-10-09 11:01:49] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:54 [I 2022-10-09 11:01:49] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00291 [I 2022-10-09 11:01:49] ...trap/engines/engine.py.230: loss: 0.13483 [I 2022-10-09 11:01:49] ...optimizers/trijoint.py.151: total_norm: 16.490980 [I 2022-10-09 11:01:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 540/2304 [I 2022-10-09 11:01:57] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:43 [I 2022-10-09 11:01:57] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00299 [I 2022-10-09 11:01:57] ...trap/engines/engine.py.230: loss: 0.19021 [I 2022-10-09 11:01:57] ...optimizers/trijoint.py.151: total_norm: 14.188200 [I 2022-10-09 11:02:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 550/2304 [I 2022-10-09 11:02:05] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:39 [I 2022-10-09 11:02:05] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00298 [I 2022-10-09 11:02:05] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-09 11:02:05] ...optimizers/trijoint.py.151: total_norm: 11.171353 [I 2022-10-09 11:02:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 560/2304 [I 2022-10-09 11:02:14] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:28 [I 2022-10-09 11:02:14] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00292 [I 2022-10-09 11:02:14] ...trap/engines/engine.py.230: loss: 0.12871 [I 2022-10-09 11:02:14] ...optimizers/trijoint.py.151: total_norm: 11.977147 [I 2022-10-09 11:02:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 570/2304 [I 2022-10-09 11:02:22] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:18 [I 2022-10-09 11:02:22] ...trap/engines/engine.py.229: process: 0.84496 | load: 0.00292 [I 2022-10-09 11:02:22] ...trap/engines/engine.py.230: loss: 0.20911 [I 2022-10-09 11:02:22] ...optimizers/trijoint.py.151: total_norm: 31.091602 [I 2022-10-09 11:02:31] ...trap/engines/engine.py.224: train: epoch 38 | batch 580/2304 [I 2022-10-09 11:02:31] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:10 [I 2022-10-09 11:02:31] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00293 [I 2022-10-09 11:02:31] ...trap/engines/engine.py.230: loss: 0.20999 [I 2022-10-09 11:02:31] ...optimizers/trijoint.py.151: total_norm: 24.552731 [I 2022-10-09 11:02:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 590/2304 [I 2022-10-09 11:02:39] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:01 [I 2022-10-09 11:02:39] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00297 [I 2022-10-09 11:02:39] ...trap/engines/engine.py.230: loss: 0.17083 [I 2022-10-09 11:02:39] ...optimizers/trijoint.py.151: total_norm: 21.426455 [I 2022-10-09 11:02:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 600/2304 [I 2022-10-09 11:02:48] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:23:52 [I 2022-10-09 11:02:48] ...trap/engines/engine.py.229: process: 0.83672 | load: 0.00296 [I 2022-10-09 11:02:48] ...trap/engines/engine.py.230: loss: 0.16027 [I 2022-10-09 11:02:48] ...optimizers/trijoint.py.151: total_norm: 11.769945 [I 2022-10-09 11:02:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 610/2304 [I 2022-10-09 11:02:56] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:45 [I 2022-10-09 11:02:56] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00294 [I 2022-10-09 11:02:56] ...trap/engines/engine.py.230: loss: 0.13651 [I 2022-10-09 11:02:56] ...optimizers/trijoint.py.151: total_norm: 13.209516 [I 2022-10-09 11:03:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 620/2304 [I 2022-10-09 11:03:04] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:38 [I 2022-10-09 11:03:04] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00297 [I 2022-10-09 11:03:04] ...trap/engines/engine.py.230: loss: 0.16667 [I 2022-10-09 11:03:04] ...optimizers/trijoint.py.151: total_norm: 17.411598 [I 2022-10-09 11:03:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 630/2304 [I 2022-10-09 11:03:13] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:34 [I 2022-10-09 11:03:13] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00293 [I 2022-10-09 11:03:13] ...trap/engines/engine.py.230: loss: 0.17982 [I 2022-10-09 11:03:13] ...optimizers/trijoint.py.151: total_norm: 34.227592 [I 2022-10-09 11:03:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 640/2304 [I 2022-10-09 11:03:21] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:26 [I 2022-10-09 11:03:21] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00292 [I 2022-10-09 11:03:21] ...trap/engines/engine.py.230: loss: 0.08834 [I 2022-10-09 11:03:21] ...optimizers/trijoint.py.151: total_norm: 11.458233 [I 2022-10-09 11:03:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 650/2304 [I 2022-10-09 11:03:30] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:11 [I 2022-10-09 11:03:30] ...trap/engines/engine.py.229: process: 0.83303 | load: 0.00293 [I 2022-10-09 11:03:30] ...trap/engines/engine.py.230: loss: 0.17426 [I 2022-10-09 11:03:30] ...optimizers/trijoint.py.151: total_norm: 24.857979 [I 2022-10-09 11:03:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 660/2304 [I 2022-10-09 11:03:38] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:04 [I 2022-10-09 11:03:38] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00300 [I 2022-10-09 11:03:38] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-09 11:03:38] ...optimizers/trijoint.py.151: total_norm: 12.735429 [I 2022-10-09 11:03:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 670/2304 [I 2022-10-09 11:03:47] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:52 [I 2022-10-09 11:03:47] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00293 [I 2022-10-09 11:03:47] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-09 11:03:47] ...optimizers/trijoint.py.151: total_norm: 17.289286 [I 2022-10-09 11:03:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 680/2304 [I 2022-10-09 11:03:55] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:51 [I 2022-10-09 11:03:55] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00295 [I 2022-10-09 11:03:55] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-09 11:03:55] ...optimizers/trijoint.py.151: total_norm: 9.783367 [I 2022-10-09 11:04:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 690/2304 [I 2022-10-09 11:04:04] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:33 [I 2022-10-09 11:04:04] ...trap/engines/engine.py.229: process: 0.83595 | load: 0.00294 [I 2022-10-09 11:04:04] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-09 11:04:04] ...optimizers/trijoint.py.151: total_norm: 24.644178 [I 2022-10-09 11:04:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 700/2304 [I 2022-10-09 11:04:12] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:28 [I 2022-10-09 11:04:12] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00296 [I 2022-10-09 11:04:12] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-09 11:04:12] ...optimizers/trijoint.py.151: total_norm: 27.351248 [I 2022-10-09 11:04:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 710/2304 [I 2022-10-09 11:04:20] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:20 [I 2022-10-09 11:04:20] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00292 [I 2022-10-09 11:04:20] ...trap/engines/engine.py.230: loss: 0.15018 [I 2022-10-09 11:04:20] ...optimizers/trijoint.py.151: total_norm: 17.520241 [I 2022-10-09 11:04:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 720/2304 [I 2022-10-09 11:04:29] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:10 [I 2022-10-09 11:04:29] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00295 [I 2022-10-09 11:04:29] ...trap/engines/engine.py.230: loss: 0.18958 [I 2022-10-09 11:04:29] ...optimizers/trijoint.py.151: total_norm: 18.135292 [I 2022-10-09 11:04:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 730/2304 [I 2022-10-09 11:04:37] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:03 [I 2022-10-09 11:04:37] ...trap/engines/engine.py.229: process: 0.83886 | load: 0.00294 [I 2022-10-09 11:04:37] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-09 11:04:37] ...optimizers/trijoint.py.151: total_norm: 20.909040 [I 2022-10-09 11:04:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 740/2304 [I 2022-10-09 11:04:46] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:58 [I 2022-10-09 11:04:46] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00297 [I 2022-10-09 11:04:46] ...trap/engines/engine.py.230: loss: 0.20243 [I 2022-10-09 11:04:46] ...optimizers/trijoint.py.151: total_norm: 24.119936 [I 2022-10-09 11:04:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 750/2304 [I 2022-10-09 11:04:54] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:46 [I 2022-10-09 11:04:54] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00294 [I 2022-10-09 11:04:54] ...trap/engines/engine.py.230: loss: 0.16787 [I 2022-10-09 11:04:54] ...optimizers/trijoint.py.151: total_norm: 20.737268 [I 2022-10-09 11:05:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 760/2304 [I 2022-10-09 11:05:02] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:34 [I 2022-10-09 11:05:02] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00294 [I 2022-10-09 11:05:02] ...trap/engines/engine.py.230: loss: 0.14106 [I 2022-10-09 11:05:02] ...optimizers/trijoint.py.151: total_norm: 14.527865 [I 2022-10-09 11:05:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 770/2304 [I 2022-10-09 11:05:11] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:31 [I 2022-10-09 11:05:11] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00297 [I 2022-10-09 11:05:11] ...trap/engines/engine.py.230: loss: 0.16856 [I 2022-10-09 11:05:11] ...optimizers/trijoint.py.151: total_norm: 24.892574 [I 2022-10-09 11:05:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 780/2304 [I 2022-10-09 11:05:19] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:30 [I 2022-10-09 11:05:19] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00298 [I 2022-10-09 11:05:19] ...trap/engines/engine.py.230: loss: 0.18087 [I 2022-10-09 11:05:19] ...optimizers/trijoint.py.151: total_norm: 10.534805 [I 2022-10-09 11:05:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 790/2304 [I 2022-10-09 11:05:28] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:10 [I 2022-10-09 11:05:28] ...trap/engines/engine.py.229: process: 0.82824 | load: 0.00298 [I 2022-10-09 11:05:28] ...trap/engines/engine.py.230: loss: 0.14915 [I 2022-10-09 11:05:28] ...optimizers/trijoint.py.151: total_norm: 17.253572 [I 2022-10-09 11:05:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 800/2304 [I 2022-10-09 11:05:36] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:13 [I 2022-10-09 11:05:36] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00295 [I 2022-10-09 11:05:36] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-09 11:05:36] ...optimizers/trijoint.py.151: total_norm: 16.798546 [I 2022-10-09 11:05:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 810/2304 [I 2022-10-09 11:05:45] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:55 [I 2022-10-09 11:05:45] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00292 [I 2022-10-09 11:05:45] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-09 11:05:45] ...optimizers/trijoint.py.151: total_norm: 28.139896 [I 2022-10-09 11:05:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 820/2304 [I 2022-10-09 11:05:53] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:48 [I 2022-10-09 11:05:53] ...trap/engines/engine.py.229: process: 0.83345 | load: 0.00293 [I 2022-10-09 11:05:53] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-09 11:05:53] ...optimizers/trijoint.py.151: total_norm: 15.814060 [I 2022-10-09 11:06:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 830/2304 [I 2022-10-09 11:06:02] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:41 [I 2022-10-09 11:06:02] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00292 [I 2022-10-09 11:06:02] ...trap/engines/engine.py.230: loss: 0.21263 [I 2022-10-09 11:06:02] ...optimizers/trijoint.py.151: total_norm: 21.428986 [I 2022-10-09 11:06:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 840/2304 [I 2022-10-09 11:06:10] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:29 [I 2022-10-09 11:06:10] ...trap/engines/engine.py.229: process: 0.82986 | load: 0.00294 [I 2022-10-09 11:06:10] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-09 11:06:10] ...optimizers/trijoint.py.151: total_norm: 14.889850 [I 2022-10-09 11:06:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 850/2304 [I 2022-10-09 11:06:18] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:19 [I 2022-10-09 11:06:18] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00293 [I 2022-10-09 11:06:18] ...trap/engines/engine.py.230: loss: 0.11723 [I 2022-10-09 11:06:18] ...optimizers/trijoint.py.151: total_norm: 9.601521 [I 2022-10-09 11:06:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 860/2304 [I 2022-10-09 11:06:27] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:15 [I 2022-10-09 11:06:27] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00301 [I 2022-10-09 11:06:27] ...trap/engines/engine.py.230: loss: 0.09936 [I 2022-10-09 11:06:27] ...optimizers/trijoint.py.151: total_norm: 9.943370 [I 2022-10-09 11:06:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 870/2304 [I 2022-10-09 11:06:35] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:09 [I 2022-10-09 11:06:35] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00296 [I 2022-10-09 11:06:35] ...trap/engines/engine.py.230: loss: 0.25396 [I 2022-10-09 11:06:35] ...optimizers/trijoint.py.151: total_norm: 31.386806 [I 2022-10-09 11:06:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 880/2304 [I 2022-10-09 11:06:44] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:58 [I 2022-10-09 11:06:44] ...trap/engines/engine.py.229: process: 0.83843 | load: 0.00295 [I 2022-10-09 11:06:44] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-09 11:06:44] ...optimizers/trijoint.py.151: total_norm: 11.737087 [I 2022-10-09 11:06:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 890/2304 [I 2022-10-09 11:06:52] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:50 [I 2022-10-09 11:06:52] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00292 [I 2022-10-09 11:06:52] ...trap/engines/engine.py.230: loss: 0.20617 [I 2022-10-09 11:06:52] ...optimizers/trijoint.py.151: total_norm: 24.458002 [I 2022-10-09 11:07:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 900/2304 [I 2022-10-09 11:07:00] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:37 [I 2022-10-09 11:07:00] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00294 [I 2022-10-09 11:07:00] ...trap/engines/engine.py.230: loss: 0.15155 [I 2022-10-09 11:07:00] ...optimizers/trijoint.py.151: total_norm: 11.021027 [I 2022-10-09 11:07:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 910/2304 [I 2022-10-09 11:07:09] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:25 [I 2022-10-09 11:07:09] ...trap/engines/engine.py.229: process: 0.83101 | load: 0.00294 [I 2022-10-09 11:07:09] ...trap/engines/engine.py.230: loss: 0.18010 [I 2022-10-09 11:07:09] ...optimizers/trijoint.py.151: total_norm: 17.085215 [I 2022-10-09 11:07:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 920/2304 [I 2022-10-09 11:07:17] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:24 [I 2022-10-09 11:07:17] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00295 [I 2022-10-09 11:07:17] ...trap/engines/engine.py.230: loss: 0.25081 [I 2022-10-09 11:07:17] ...optimizers/trijoint.py.151: total_norm: 30.250355 [I 2022-10-09 11:07:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 930/2304 [I 2022-10-09 11:07:26] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:22 [I 2022-10-09 11:07:26] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00299 [I 2022-10-09 11:07:26] ...trap/engines/engine.py.230: loss: 0.18011 [I 2022-10-09 11:07:26] ...optimizers/trijoint.py.151: total_norm: 15.822231 [I 2022-10-09 11:07:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 940/2304 [I 2022-10-09 11:07:34] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:13 [I 2022-10-09 11:07:34] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00295 [I 2022-10-09 11:07:34] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-09 11:07:34] ...optimizers/trijoint.py.151: total_norm: 16.181494 [I 2022-10-09 11:07:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 950/2304 [I 2022-10-09 11:07:43] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:01 [I 2022-10-09 11:07:43] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00292 [I 2022-10-09 11:07:43] ...trap/engines/engine.py.230: loss: 0.13500 [I 2022-10-09 11:07:43] ...optimizers/trijoint.py.151: total_norm: 11.028819 [I 2022-10-09 11:07:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 960/2304 [I 2022-10-09 11:07:51] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:18:50 [I 2022-10-09 11:07:51] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00293 [I 2022-10-09 11:07:51] ...trap/engines/engine.py.230: loss: 0.16008 [I 2022-10-09 11:07:51] ...optimizers/trijoint.py.151: total_norm: 10.924731 [I 2022-10-09 11:07:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 970/2304 [I 2022-10-09 11:08:00] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:44 [I 2022-10-09 11:08:00] ...trap/engines/engine.py.229: process: 0.84660 | load: 0.00293 [I 2022-10-09 11:08:00] ...trap/engines/engine.py.230: loss: 0.18523 [I 2022-10-09 11:08:00] ...optimizers/trijoint.py.151: total_norm: 9.571326 [I 2022-10-09 11:08:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 980/2304 [I 2022-10-09 11:08:08] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:18:32 [I 2022-10-09 11:08:08] ...trap/engines/engine.py.229: process: 0.83327 | load: 0.00297 [I 2022-10-09 11:08:08] ...trap/engines/engine.py.230: loss: 0.11940 [I 2022-10-09 11:08:08] ...optimizers/trijoint.py.151: total_norm: 12.722198 [I 2022-10-09 11:08:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 990/2304 [I 2022-10-09 11:08:16] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:25 [I 2022-10-09 11:08:16] ...trap/engines/engine.py.229: process: 0.83390 | load: 0.00293 [I 2022-10-09 11:08:16] ...trap/engines/engine.py.230: loss: 0.20205 [I 2022-10-09 11:08:16] ...optimizers/trijoint.py.151: total_norm: 26.420919 [I 2022-10-09 11:08:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 1000/2304 [I 2022-10-09 11:08:25] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:17 [I 2022-10-09 11:08:25] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00295 [I 2022-10-09 11:08:25] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-09 11:08:25] ...optimizers/trijoint.py.151: total_norm: 23.249664 [I 2022-10-09 11:08:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 1010/2304 [I 2022-10-09 11:08:33] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:12 [I 2022-10-09 11:08:33] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00298 [I 2022-10-09 11:08:33] ...trap/engines/engine.py.230: loss: 0.19198 [I 2022-10-09 11:08:33] ...optimizers/trijoint.py.151: total_norm: 11.754887 [I 2022-10-09 11:08:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 1020/2304 [I 2022-10-09 11:08:42] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:02 [I 2022-10-09 11:08:42] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00295 [I 2022-10-09 11:08:42] ...trap/engines/engine.py.230: loss: 0.13930 [I 2022-10-09 11:08:42] ...optimizers/trijoint.py.151: total_norm: 12.500003 [I 2022-10-09 11:08:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 1030/2304 [I 2022-10-09 11:08:50] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:17:53 [I 2022-10-09 11:08:50] ...trap/engines/engine.py.229: process: 0.83875 | load: 0.00294 [I 2022-10-09 11:08:50] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-09 11:08:50] ...optimizers/trijoint.py.151: total_norm: 11.911089 [I 2022-10-09 11:08:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 1040/2304 [I 2022-10-09 11:08:59] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:44 [I 2022-10-09 11:08:59] ...trap/engines/engine.py.229: process: 0.83885 | load: 0.00294 [I 2022-10-09 11:08:59] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-09 11:08:59] ...optimizers/trijoint.py.151: total_norm: 16.403675 [I 2022-10-09 11:09:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 1050/2304 [I 2022-10-09 11:09:07] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:17:37 [I 2022-10-09 11:09:07] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00294 [I 2022-10-09 11:09:07] ...trap/engines/engine.py.230: loss: 0.20868 [I 2022-10-09 11:09:07] ...optimizers/trijoint.py.151: total_norm: 44.595680 [I 2022-10-09 11:09:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 1060/2304 [I 2022-10-09 11:09:15] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:27 [I 2022-10-09 11:09:15] ...trap/engines/engine.py.229: process: 0.84680 | load: 0.00297 [I 2022-10-09 11:09:15] ...trap/engines/engine.py.230: loss: 0.14687 [I 2022-10-09 11:09:15] ...optimizers/trijoint.py.151: total_norm: 16.389385 [I 2022-10-09 11:09:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 1070/2304 [I 2022-10-09 11:09:24] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:20 [I 2022-10-09 11:09:24] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00292 [I 2022-10-09 11:09:24] ...trap/engines/engine.py.230: loss: 0.11142 [I 2022-10-09 11:09:24] ...optimizers/trijoint.py.151: total_norm: 12.299133 [I 2022-10-09 11:09:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 1080/2304 [I 2022-10-09 11:09:32] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:08 [I 2022-10-09 11:09:32] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00293 [I 2022-10-09 11:09:32] ...trap/engines/engine.py.230: loss: 0.16009 [I 2022-10-09 11:09:32] ...optimizers/trijoint.py.151: total_norm: 21.136572 [I 2022-10-09 11:09:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 1090/2304 [I 2022-10-09 11:09:41] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:16:58 [I 2022-10-09 11:09:41] ...trap/engines/engine.py.229: process: 0.83713 | load: 0.00296 [I 2022-10-09 11:09:41] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-09 11:09:41] ...optimizers/trijoint.py.151: total_norm: 17.136011 [I 2022-10-09 11:09:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 1100/2304 [I 2022-10-09 11:09:49] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:16:52 [I 2022-10-09 11:09:49] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00307 [I 2022-10-09 11:09:49] ...trap/engines/engine.py.230: loss: 0.13527 [I 2022-10-09 11:09:49] ...optimizers/trijoint.py.151: total_norm: 13.157190 [I 2022-10-09 11:09:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1110/2304 [I 2022-10-09 11:09:57] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:16:42 [I 2022-10-09 11:09:57] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00292 [I 2022-10-09 11:09:57] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-09 11:09:57] ...optimizers/trijoint.py.151: total_norm: 10.076394 [I 2022-10-09 11:10:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1120/2304 [I 2022-10-09 11:10:06] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:16:35 [I 2022-10-09 11:10:06] ...trap/engines/engine.py.229: process: 0.84443 | load: 0.00294 [I 2022-10-09 11:10:06] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-09 11:10:06] ...optimizers/trijoint.py.151: total_norm: 17.575073 [I 2022-10-09 11:10:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 1130/2304 [I 2022-10-09 11:10:14] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:27 [I 2022-10-09 11:10:14] ...trap/engines/engine.py.229: process: 0.83594 | load: 0.00295 [I 2022-10-09 11:10:14] ...trap/engines/engine.py.230: loss: 0.14492 [I 2022-10-09 11:10:14] ...optimizers/trijoint.py.151: total_norm: 22.712036 [I 2022-10-09 11:10:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 1140/2304 [I 2022-10-09 11:10:23] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:16:20 [I 2022-10-09 11:10:23] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00295 [I 2022-10-09 11:10:23] ...trap/engines/engine.py.230: loss: 0.17339 [I 2022-10-09 11:10:23] ...optimizers/trijoint.py.151: total_norm: 9.940729 [I 2022-10-09 11:10:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 1150/2304 [I 2022-10-09 11:10:49] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 1:24:59 [I 2022-10-09 11:10:49] ...trap/engines/engine.py.229: process: 18.74248 | load: 0.00294 [I 2022-10-09 11:10:49] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-09 11:10:49] ...optimizers/trijoint.py.151: total_norm: 14.870624 [I 2022-10-09 11:10:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1160/2304 [I 2022-10-09 11:10:57] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:23:23 [I 2022-10-09 11:10:57] ...trap/engines/engine.py.229: process: 0.84890 | load: 0.00300 [I 2022-10-09 11:10:57] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-09 11:10:57] ...optimizers/trijoint.py.151: total_norm: 11.774715 [I 2022-10-09 11:11:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1170/2304 [I 2022-10-09 11:11:06] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:38 [I 2022-10-09 11:11:06] ...trap/engines/engine.py.229: process: 0.83401 | load: 0.00293 [I 2022-10-09 11:11:06] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-09 11:11:06] ...optimizers/trijoint.py.151: total_norm: 25.062502 [I 2022-10-09 11:11:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 1180/2304 [I 2022-10-09 11:11:14] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:46 [I 2022-10-09 11:11:14] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00294 [I 2022-10-09 11:11:14] ...trap/engines/engine.py.230: loss: 0.17953 [I 2022-10-09 11:11:14] ...optimizers/trijoint.py.151: total_norm: 20.998857 [I 2022-10-09 11:11:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 1190/2304 [I 2022-10-09 11:11:23] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:39 [I 2022-10-09 11:11:23] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00293 [I 2022-10-09 11:11:23] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-09 11:11:23] ...optimizers/trijoint.py.151: total_norm: 21.089132 [I 2022-10-09 11:11:31] ...trap/engines/engine.py.224: train: epoch 38 | batch 1200/2304 [I 2022-10-09 11:11:31] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:29 [I 2022-10-09 11:11:31] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00294 [I 2022-10-09 11:11:31] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-09 11:11:31] ...optimizers/trijoint.py.151: total_norm: 21.642439 [I 2022-10-09 11:11:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 1210/2304 [I 2022-10-09 11:11:40] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:20 [I 2022-10-09 11:11:40] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00292 [I 2022-10-09 11:11:40] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-09 11:11:40] ...optimizers/trijoint.py.151: total_norm: 9.931755 [I 2022-10-09 11:11:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 1220/2304 [I 2022-10-09 11:11:48] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:02 [I 2022-10-09 11:11:48] ...trap/engines/engine.py.229: process: 0.82426 | load: 0.00299 [I 2022-10-09 11:11:48] ...trap/engines/engine.py.230: loss: 0.11579 [I 2022-10-09 11:11:48] ...optimizers/trijoint.py.151: total_norm: 7.951942 [I 2022-10-09 11:11:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 1230/2304 [I 2022-10-09 11:11:56] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:15:02 [I 2022-10-09 11:11:56] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00294 [I 2022-10-09 11:11:56] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-09 11:11:56] ...optimizers/trijoint.py.151: total_norm: 12.549050 [I 2022-10-09 11:12:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 1240/2304 [I 2022-10-09 11:12:05] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:58 [I 2022-10-09 11:12:05] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00299 [I 2022-10-09 11:12:05] ...trap/engines/engine.py.230: loss: 0.21998 [I 2022-10-09 11:12:05] ...optimizers/trijoint.py.151: total_norm: 27.114986 [I 2022-10-09 11:12:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 1250/2304 [I 2022-10-09 11:12:13] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:46 [I 2022-10-09 11:12:13] ...trap/engines/engine.py.229: process: 0.84800 | load: 0.00290 [I 2022-10-09 11:12:13] ...trap/engines/engine.py.230: loss: 0.22756 [I 2022-10-09 11:12:13] ...optimizers/trijoint.py.151: total_norm: 21.092834 [I 2022-10-09 11:12:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 1260/2304 [I 2022-10-09 11:12:22] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:34 [I 2022-10-09 11:12:22] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00295 [I 2022-10-09 11:12:22] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-09 11:12:22] ...optimizers/trijoint.py.151: total_norm: 20.544598 [I 2022-10-09 11:12:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 1270/2304 [I 2022-10-09 11:12:30] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:27 [I 2022-10-09 11:12:30] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00292 [I 2022-10-09 11:12:30] ...trap/engines/engine.py.230: loss: 0.18466 [I 2022-10-09 11:12:30] ...optimizers/trijoint.py.151: total_norm: 13.709556 [I 2022-10-09 11:12:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 1280/2304 [I 2022-10-09 11:12:39] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:27 [I 2022-10-09 11:12:39] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00294 [I 2022-10-09 11:12:39] ...trap/engines/engine.py.230: loss: 0.20027 [I 2022-10-09 11:12:39] ...optimizers/trijoint.py.151: total_norm: 11.510235 [I 2022-10-09 11:12:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 1290/2304 [I 2022-10-09 11:12:47] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:20 [I 2022-10-09 11:12:47] ...trap/engines/engine.py.229: process: 0.85284 | load: 0.00293 [I 2022-10-09 11:12:47] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-09 11:12:47] ...optimizers/trijoint.py.151: total_norm: 24.548359 [I 2022-10-09 11:12:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 1300/2304 [I 2022-10-09 11:12:56] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:10 [I 2022-10-09 11:12:56] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00296 [I 2022-10-09 11:12:56] ...trap/engines/engine.py.230: loss: 0.20547 [I 2022-10-09 11:12:56] ...optimizers/trijoint.py.151: total_norm: 79.291000 [I 2022-10-09 11:13:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 1310/2304 [I 2022-10-09 11:13:04] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:58 [I 2022-10-09 11:13:04] ...trap/engines/engine.py.229: process: 0.83342 | load: 0.00292 [I 2022-10-09 11:13:04] ...trap/engines/engine.py.230: loss: 0.24273 [I 2022-10-09 11:13:04] ...optimizers/trijoint.py.151: total_norm: 12.560338 [I 2022-10-09 11:13:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 1320/2304 [I 2022-10-09 11:13:12] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:49 [I 2022-10-09 11:13:12] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00294 [I 2022-10-09 11:13:12] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-09 11:13:12] ...optimizers/trijoint.py.151: total_norm: 15.331580 [I 2022-10-09 11:13:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 1330/2304 [I 2022-10-09 11:13:21] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:38 [I 2022-10-09 11:13:21] ...trap/engines/engine.py.229: process: 0.83853 | load: 0.00295 [I 2022-10-09 11:13:21] ...trap/engines/engine.py.230: loss: 0.11846 [I 2022-10-09 11:13:21] ...optimizers/trijoint.py.151: total_norm: 16.711813 [I 2022-10-09 11:13:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 1340/2304 [I 2022-10-09 11:13:29] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:29 [I 2022-10-09 11:13:29] ...trap/engines/engine.py.229: process: 0.83558 | load: 0.00292 [I 2022-10-09 11:13:29] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-09 11:13:29] ...optimizers/trijoint.py.151: total_norm: 9.800391 [I 2022-10-09 11:13:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 1350/2304 [I 2022-10-09 11:13:38] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:25 [I 2022-10-09 11:13:38] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00293 [I 2022-10-09 11:13:38] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-09 11:13:38] ...optimizers/trijoint.py.151: total_norm: 22.074501 [I 2022-10-09 11:13:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 1360/2304 [I 2022-10-09 11:13:46] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:12 [I 2022-10-09 11:13:46] ...trap/engines/engine.py.229: process: 0.84334 | load: 0.00292 [I 2022-10-09 11:13:46] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-09 11:13:46] ...optimizers/trijoint.py.151: total_norm: 18.764782 [I 2022-10-09 11:13:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 1370/2304 [I 2022-10-09 11:13:54] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:03 [I 2022-10-09 11:13:54] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00301 [I 2022-10-09 11:13:54] ...trap/engines/engine.py.230: loss: 0.15321 [I 2022-10-09 11:13:54] ...optimizers/trijoint.py.151: total_norm: 23.858965 [I 2022-10-09 11:14:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 1380/2304 [I 2022-10-09 11:14:03] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:12:58 [I 2022-10-09 11:14:03] ...trap/engines/engine.py.229: process: 0.83482 | load: 0.00294 [I 2022-10-09 11:14:03] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-09 11:14:03] ...optimizers/trijoint.py.151: total_norm: 20.749212 [I 2022-10-09 11:14:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 1390/2304 [I 2022-10-09 11:14:11] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:12:46 [I 2022-10-09 11:14:11] ...trap/engines/engine.py.229: process: 0.83233 | load: 0.00291 [I 2022-10-09 11:14:11] ...trap/engines/engine.py.230: loss: 0.18658 [I 2022-10-09 11:14:11] ...optimizers/trijoint.py.151: total_norm: 18.277784 [I 2022-10-09 11:14:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 1400/2304 [I 2022-10-09 11:14:20] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:41 [I 2022-10-09 11:14:20] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00294 [I 2022-10-09 11:14:20] ...trap/engines/engine.py.230: loss: 0.16111 [I 2022-10-09 11:14:20] ...optimizers/trijoint.py.151: total_norm: 12.597352 [I 2022-10-09 11:14:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 1410/2304 [I 2022-10-09 11:14:28] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:31 [I 2022-10-09 11:14:28] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00296 [I 2022-10-09 11:14:28] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-09 11:14:28] ...optimizers/trijoint.py.151: total_norm: 18.469505 [I 2022-10-09 11:14:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 1420/2304 [I 2022-10-09 11:14:37] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:28 [I 2022-10-09 11:14:37] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00291 [I 2022-10-09 11:14:37] ...trap/engines/engine.py.230: loss: 0.16490 [I 2022-10-09 11:14:37] ...optimizers/trijoint.py.151: total_norm: 11.347682 [I 2022-10-09 11:14:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 1430/2304 [I 2022-10-09 11:14:45] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:11 [I 2022-10-09 11:14:45] ...trap/engines/engine.py.229: process: 0.82635 | load: 0.00293 [I 2022-10-09 11:14:45] ...trap/engines/engine.py.230: loss: 0.26472 [I 2022-10-09 11:14:45] ...optimizers/trijoint.py.151: total_norm: 18.209883 [I 2022-10-09 11:14:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 1440/2304 [I 2022-10-09 11:14:53] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:07 [I 2022-10-09 11:14:53] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00292 [I 2022-10-09 11:14:53] ...trap/engines/engine.py.230: loss: 0.16822 [I 2022-10-09 11:14:53] ...optimizers/trijoint.py.151: total_norm: 11.724147 [I 2022-10-09 11:15:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 1450/2304 [I 2022-10-09 11:15:02] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:11:57 [I 2022-10-09 11:15:02] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00294 [I 2022-10-09 11:15:02] ...trap/engines/engine.py.230: loss: 0.23831 [I 2022-10-09 11:15:02] ...optimizers/trijoint.py.151: total_norm: 27.972771 [I 2022-10-09 11:15:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 1460/2304 [I 2022-10-09 11:15:10] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:47 [I 2022-10-09 11:15:10] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00298 [I 2022-10-09 11:15:10] ...trap/engines/engine.py.230: loss: 0.25286 [I 2022-10-09 11:15:10] ...optimizers/trijoint.py.151: total_norm: 21.920895 [I 2022-10-09 11:15:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 1470/2304 [I 2022-10-09 11:15:19] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:11:43 [I 2022-10-09 11:15:19] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00292 [I 2022-10-09 11:15:19] ...trap/engines/engine.py.230: loss: 0.12616 [I 2022-10-09 11:15:19] ...optimizers/trijoint.py.151: total_norm: 18.732786 [I 2022-10-09 11:15:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 1480/2304 [I 2022-10-09 11:15:27] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:35 [I 2022-10-09 11:15:27] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00293 [I 2022-10-09 11:15:27] ...trap/engines/engine.py.230: loss: 0.18064 [I 2022-10-09 11:15:27] ...optimizers/trijoint.py.151: total_norm: 17.135687 [I 2022-10-09 11:15:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 1490/2304 [I 2022-10-09 11:15:36] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:23 [I 2022-10-09 11:15:36] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00293 [I 2022-10-09 11:15:36] ...trap/engines/engine.py.230: loss: 0.14065 [I 2022-10-09 11:15:36] ...optimizers/trijoint.py.151: total_norm: 24.900112 [I 2022-10-09 11:15:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 1500/2304 [I 2022-10-09 11:15:44] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:13 [I 2022-10-09 11:15:44] ...trap/engines/engine.py.229: process: 0.82881 | load: 0.00296 [I 2022-10-09 11:15:44] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-09 11:15:44] ...optimizers/trijoint.py.151: total_norm: 7.509328 [I 2022-10-09 11:15:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 1510/2304 [I 2022-10-09 11:15:52] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:07 [I 2022-10-09 11:15:52] ...trap/engines/engine.py.229: process: 0.84024 | load: 0.00296 [I 2022-10-09 11:15:52] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-09 11:15:52] ...optimizers/trijoint.py.151: total_norm: 19.968527 [I 2022-10-09 11:16:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 1520/2304 [I 2022-10-09 11:16:01] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:10:57 [I 2022-10-09 11:16:01] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00299 [I 2022-10-09 11:16:01] ...trap/engines/engine.py.230: loss: 0.14523 [I 2022-10-09 11:16:01] ...optimizers/trijoint.py.151: total_norm: 15.754070 [I 2022-10-09 11:16:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 1530/2304 [I 2022-10-09 11:16:09] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:52 [I 2022-10-09 11:16:09] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00293 [I 2022-10-09 11:16:09] ...trap/engines/engine.py.230: loss: 0.20498 [I 2022-10-09 11:16:09] ...optimizers/trijoint.py.151: total_norm: 26.624428 [I 2022-10-09 11:16:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 1540/2304 [I 2022-10-09 11:16:18] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:41 [I 2022-10-09 11:16:18] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00293 [I 2022-10-09 11:16:18] ...trap/engines/engine.py.230: loss: 0.23073 [I 2022-10-09 11:16:18] ...optimizers/trijoint.py.151: total_norm: 19.993654 [I 2022-10-09 11:16:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 1550/2304 [I 2022-10-09 11:16:26] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:34 [I 2022-10-09 11:16:26] ...trap/engines/engine.py.229: process: 0.83710 | load: 0.00294 [I 2022-10-09 11:16:26] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-09 11:16:26] ...optimizers/trijoint.py.151: total_norm: 20.045916 [I 2022-10-09 11:16:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 1560/2304 [I 2022-10-09 11:16:35] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:29 [I 2022-10-09 11:16:35] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00295 [I 2022-10-09 11:16:35] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-09 11:16:35] ...optimizers/trijoint.py.151: total_norm: 14.102913 [I 2022-10-09 11:16:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 1570/2304 [I 2022-10-09 11:16:43] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:19 [I 2022-10-09 11:16:43] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00294 [I 2022-10-09 11:16:43] ...trap/engines/engine.py.230: loss: 0.13370 [I 2022-10-09 11:16:43] ...optimizers/trijoint.py.151: total_norm: 21.070807 [I 2022-10-09 11:16:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 1580/2304 [I 2022-10-09 11:16:51] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:09 [I 2022-10-09 11:16:51] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00293 [I 2022-10-09 11:16:51] ...trap/engines/engine.py.230: loss: 0.19353 [I 2022-10-09 11:16:51] ...optimizers/trijoint.py.151: total_norm: 17.703047 [I 2022-10-09 11:17:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 1590/2304 [I 2022-10-09 11:17:00] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:00 [I 2022-10-09 11:17:00] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00298 [I 2022-10-09 11:17:00] ...trap/engines/engine.py.230: loss: 0.14609 [I 2022-10-09 11:17:00] ...optimizers/trijoint.py.151: total_norm: 18.085768 [I 2022-10-09 11:17:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 1600/2304 [I 2022-10-09 11:17:08] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:51 [I 2022-10-09 11:17:08] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00287 [I 2022-10-09 11:17:08] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-09 11:17:08] ...optimizers/trijoint.py.151: total_norm: 20.739559 [I 2022-10-09 11:17:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 1610/2304 [I 2022-10-09 11:17:17] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:45 [I 2022-10-09 11:17:17] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00293 [I 2022-10-09 11:17:17] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-09 11:17:17] ...optimizers/trijoint.py.151: total_norm: 19.041351 [I 2022-10-09 11:17:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 1620/2304 [I 2022-10-09 11:17:25] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:33 [I 2022-10-09 11:17:25] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00291 [I 2022-10-09 11:17:25] ...trap/engines/engine.py.230: loss: 0.13235 [I 2022-10-09 11:17:25] ...optimizers/trijoint.py.151: total_norm: 16.984110 [I 2022-10-09 11:17:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 1630/2304 [I 2022-10-09 11:17:34] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:29 [I 2022-10-09 11:17:34] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00292 [I 2022-10-09 11:17:34] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-09 11:17:34] ...optimizers/trijoint.py.151: total_norm: 11.179033 [I 2022-10-09 11:17:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 1640/2304 [I 2022-10-09 11:17:42] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:16 [I 2022-10-09 11:17:42] ...trap/engines/engine.py.229: process: 0.83010 | load: 0.00295 [I 2022-10-09 11:17:42] ...trap/engines/engine.py.230: loss: 0.17064 [I 2022-10-09 11:17:42] ...optimizers/trijoint.py.151: total_norm: 11.056360 [I 2022-10-09 11:17:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 1650/2304 [I 2022-10-09 11:17:50] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:10 [I 2022-10-09 11:17:50] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00302 [I 2022-10-09 11:17:50] ...trap/engines/engine.py.230: loss: 0.10754 [I 2022-10-09 11:17:50] ...optimizers/trijoint.py.151: total_norm: 12.866191 [I 2022-10-09 11:17:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 1660/2304 [I 2022-10-09 11:17:59] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:08:58 [I 2022-10-09 11:17:59] ...trap/engines/engine.py.229: process: 0.83775 | load: 0.00295 [I 2022-10-09 11:17:59] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-09 11:17:59] ...optimizers/trijoint.py.151: total_norm: 20.501295 [I 2022-10-09 11:18:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 1670/2304 [I 2022-10-09 11:18:07] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:55 [I 2022-10-09 11:18:07] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00291 [I 2022-10-09 11:18:07] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-09 11:18:07] ...optimizers/trijoint.py.151: total_norm: 11.016148 [I 2022-10-09 11:18:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 1680/2304 [I 2022-10-09 11:18:16] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:46 [I 2022-10-09 11:18:16] ...trap/engines/engine.py.229: process: 0.83603 | load: 0.00292 [I 2022-10-09 11:18:16] ...trap/engines/engine.py.230: loss: 0.13368 [I 2022-10-09 11:18:16] ...optimizers/trijoint.py.151: total_norm: 14.244035 [I 2022-10-09 11:18:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 1690/2304 [I 2022-10-09 11:18:24] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:36 [I 2022-10-09 11:18:24] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00292 [I 2022-10-09 11:18:24] ...trap/engines/engine.py.230: loss: 0.16333 [I 2022-10-09 11:18:24] ...optimizers/trijoint.py.151: total_norm: 27.350956 [I 2022-10-09 11:18:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 1700/2304 [I 2022-10-09 11:18:32] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:26 [I 2022-10-09 11:18:32] ...trap/engines/engine.py.229: process: 0.82704 | load: 0.00293 [I 2022-10-09 11:18:32] ...trap/engines/engine.py.230: loss: 0.20020 [I 2022-10-09 11:18:32] ...optimizers/trijoint.py.151: total_norm: 11.786606 [I 2022-10-09 11:18:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 1710/2304 [I 2022-10-09 11:18:41] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:19 [I 2022-10-09 11:18:41] ...trap/engines/engine.py.229: process: 0.84702 | load: 0.00300 [I 2022-10-09 11:18:41] ...trap/engines/engine.py.230: loss: 0.14547 [I 2022-10-09 11:18:41] ...optimizers/trijoint.py.151: total_norm: 11.327448 [I 2022-10-09 11:18:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 1720/2304 [I 2022-10-09 11:18:49] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:12 [I 2022-10-09 11:18:49] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00290 [I 2022-10-09 11:18:49] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-09 11:18:49] ...optimizers/trijoint.py.151: total_norm: 13.210871 [I 2022-10-09 11:18:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 1730/2304 [I 2022-10-09 11:18:58] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:02 [I 2022-10-09 11:18:58] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00291 [I 2022-10-09 11:18:58] ...trap/engines/engine.py.230: loss: 0.19745 [I 2022-10-09 11:18:58] ...optimizers/trijoint.py.151: total_norm: 22.812969 [I 2022-10-09 11:19:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1740/2304 [I 2022-10-09 11:19:06] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:07:54 [I 2022-10-09 11:19:06] ...trap/engines/engine.py.229: process: 0.83410 | load: 0.00293 [I 2022-10-09 11:19:06] ...trap/engines/engine.py.230: loss: 0.11954 [I 2022-10-09 11:19:06] ...optimizers/trijoint.py.151: total_norm: 10.488448 [I 2022-10-09 11:19:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 1750/2304 [I 2022-10-09 11:19:15] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:46 [I 2022-10-09 11:19:15] ...trap/engines/engine.py.229: process: 0.84712 | load: 0.00293 [I 2022-10-09 11:19:15] ...trap/engines/engine.py.230: loss: 0.24465 [I 2022-10-09 11:19:15] ...optimizers/trijoint.py.151: total_norm: 33.822800 [I 2022-10-09 11:19:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 1760/2304 [I 2022-10-09 11:19:23] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:38 [I 2022-10-09 11:19:23] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00298 [I 2022-10-09 11:19:23] ...trap/engines/engine.py.230: loss: 0.13302 [I 2022-10-09 11:19:23] ...optimizers/trijoint.py.151: total_norm: 14.893766 [I 2022-10-09 11:19:31] ...trap/engines/engine.py.224: train: epoch 38 | batch 1770/2304 [I 2022-10-09 11:19:31] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:27 [I 2022-10-09 11:19:31] ...trap/engines/engine.py.229: process: 0.83431 | load: 0.00294 [I 2022-10-09 11:19:31] ...trap/engines/engine.py.230: loss: 0.10235 [I 2022-10-09 11:19:31] ...optimizers/trijoint.py.151: total_norm: 7.297853 [I 2022-10-09 11:19:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 1780/2304 [I 2022-10-09 11:19:40] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:22 [I 2022-10-09 11:19:40] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00292 [I 2022-10-09 11:19:40] ...trap/engines/engine.py.230: loss: 0.09823 [I 2022-10-09 11:19:40] ...optimizers/trijoint.py.151: total_norm: 8.017586 [I 2022-10-09 11:19:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 1790/2304 [I 2022-10-09 11:19:48] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:12 [I 2022-10-09 11:19:48] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00294 [I 2022-10-09 11:19:48] ...trap/engines/engine.py.230: loss: 0.15082 [I 2022-10-09 11:19:48] ...optimizers/trijoint.py.151: total_norm: 41.183327 [I 2022-10-09 11:19:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1800/2304 [I 2022-10-09 11:19:57] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:04 [I 2022-10-09 11:19:57] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00293 [I 2022-10-09 11:19:57] ...trap/engines/engine.py.230: loss: 0.15038 [I 2022-10-09 11:19:57] ...optimizers/trijoint.py.151: total_norm: 17.877903 [I 2022-10-09 11:20:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 1810/2304 [I 2022-10-09 11:20:05] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:06:54 [I 2022-10-09 11:20:05] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00294 [I 2022-10-09 11:20:05] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-09 11:20:05] ...optimizers/trijoint.py.151: total_norm: 16.422958 [I 2022-10-09 11:20:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 1820/2304 [I 2022-10-09 11:20:14] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:47 [I 2022-10-09 11:20:14] ...trap/engines/engine.py.229: process: 0.83617 | load: 0.00300 [I 2022-10-09 11:20:14] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-09 11:20:14] ...optimizers/trijoint.py.151: total_norm: 10.789854 [I 2022-10-09 11:20:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 1830/2304 [I 2022-10-09 11:20:22] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:37 [I 2022-10-09 11:20:22] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00291 [I 2022-10-09 11:20:22] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-09 11:20:22] ...optimizers/trijoint.py.151: total_norm: 17.428095 [I 2022-10-09 11:20:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 1840/2304 [I 2022-10-09 11:20:30] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:30 [I 2022-10-09 11:20:30] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00292 [I 2022-10-09 11:20:30] ...trap/engines/engine.py.230: loss: 0.18457 [I 2022-10-09 11:20:30] ...optimizers/trijoint.py.151: total_norm: 17.931782 [I 2022-10-09 11:20:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 1850/2304 [I 2022-10-09 11:20:39] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:22 [I 2022-10-09 11:20:39] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00294 [I 2022-10-09 11:20:39] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-09 11:20:39] ...optimizers/trijoint.py.151: total_norm: 14.473240 [I 2022-10-09 11:20:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 1860/2304 [I 2022-10-09 11:20:47] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:13 [I 2022-10-09 11:20:47] ...trap/engines/engine.py.229: process: 0.84581 | load: 0.00291 [I 2022-10-09 11:20:47] ...trap/engines/engine.py.230: loss: 0.15962 [I 2022-10-09 11:20:47] ...optimizers/trijoint.py.151: total_norm: 25.555552 [I 2022-10-09 11:20:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 1870/2304 [I 2022-10-09 11:20:56] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:05 [I 2022-10-09 11:20:56] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00304 [I 2022-10-09 11:20:56] ...trap/engines/engine.py.230: loss: 0.15311 [I 2022-10-09 11:20:56] ...optimizers/trijoint.py.151: total_norm: 13.316634 [I 2022-10-09 11:21:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 1880/2304 [I 2022-10-09 11:21:04] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:05:57 [I 2022-10-09 11:21:04] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00291 [I 2022-10-09 11:21:04] ...trap/engines/engine.py.230: loss: 0.16247 [I 2022-10-09 11:21:04] ...optimizers/trijoint.py.151: total_norm: 23.779446 [I 2022-10-09 11:21:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 1890/2304 [I 2022-10-09 11:21:12] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:46 [I 2022-10-09 11:21:12] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00292 [I 2022-10-09 11:21:12] ...trap/engines/engine.py.230: loss: 0.20014 [I 2022-10-09 11:21:12] ...optimizers/trijoint.py.151: total_norm: 26.773323 [I 2022-10-09 11:21:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 1900/2304 [I 2022-10-09 11:21:21] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:39 [I 2022-10-09 11:21:21] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00292 [I 2022-10-09 11:21:21] ...trap/engines/engine.py.230: loss: 0.21906 [I 2022-10-09 11:21:21] ...optimizers/trijoint.py.151: total_norm: 19.394039 [I 2022-10-09 11:21:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 1910/2304 [I 2022-10-09 11:21:29] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:30 [I 2022-10-09 11:21:29] ...trap/engines/engine.py.229: process: 0.84367 | load: 0.00294 [I 2022-10-09 11:21:29] ...trap/engines/engine.py.230: loss: 0.10644 [I 2022-10-09 11:21:29] ...optimizers/trijoint.py.151: total_norm: 10.325638 [I 2022-10-09 11:21:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 1920/2304 [I 2022-10-09 11:21:38] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:23 [I 2022-10-09 11:21:38] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00298 [I 2022-10-09 11:21:38] ...trap/engines/engine.py.230: loss: 0.16172 [I 2022-10-09 11:21:38] ...optimizers/trijoint.py.151: total_norm: 17.061691 [I 2022-10-09 11:21:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 1930/2304 [I 2022-10-09 11:21:46] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:14 [I 2022-10-09 11:21:46] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00289 [I 2022-10-09 11:21:46] ...trap/engines/engine.py.230: loss: 0.14803 [I 2022-10-09 11:21:46] ...optimizers/trijoint.py.151: total_norm: 18.716179 [I 2022-10-09 11:21:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 1940/2304 [I 2022-10-09 11:21:54] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:06 [I 2022-10-09 11:21:54] ...trap/engines/engine.py.229: process: 0.82981 | load: 0.00292 [I 2022-10-09 11:21:54] ...trap/engines/engine.py.230: loss: 0.24137 [I 2022-10-09 11:21:54] ...optimizers/trijoint.py.151: total_norm: 27.272135 [I 2022-10-09 11:22:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 1950/2304 [I 2022-10-09 11:22:03] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:04:59 [I 2022-10-09 11:22:03] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00294 [I 2022-10-09 11:22:03] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-09 11:22:03] ...optimizers/trijoint.py.151: total_norm: 15.671868 [I 2022-10-09 11:22:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 1960/2304 [I 2022-10-09 11:22:11] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:49 [I 2022-10-09 11:22:11] ...trap/engines/engine.py.229: process: 0.83694 | load: 0.00296 [I 2022-10-09 11:22:11] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-09 11:22:11] ...optimizers/trijoint.py.151: total_norm: 10.245016 [I 2022-10-09 11:22:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 1970/2304 [I 2022-10-09 11:22:20] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:41 [I 2022-10-09 11:22:20] ...trap/engines/engine.py.229: process: 0.85359 | load: 0.00295 [I 2022-10-09 11:22:20] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-09 11:22:20] ...optimizers/trijoint.py.151: total_norm: 24.626732 [I 2022-10-09 11:22:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 1980/2304 [I 2022-10-09 11:22:28] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:30 [I 2022-10-09 11:22:28] ...trap/engines/engine.py.229: process: 0.82680 | load: 0.00293 [I 2022-10-09 11:22:28] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-09 11:22:28] ...optimizers/trijoint.py.151: total_norm: 13.978591 [I 2022-10-09 11:22:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 1990/2304 [I 2022-10-09 11:22:37] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:24 [I 2022-10-09 11:22:37] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00294 [I 2022-10-09 11:22:37] ...trap/engines/engine.py.230: loss: 0.26142 [I 2022-10-09 11:22:37] ...optimizers/trijoint.py.151: total_norm: 24.968708 [I 2022-10-09 11:22:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 2000/2304 [I 2022-10-09 11:22:45] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:16 [I 2022-10-09 11:22:45] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00294 [I 2022-10-09 11:22:45] ...trap/engines/engine.py.230: loss: 0.15878 [I 2022-10-09 11:22:45] ...optimizers/trijoint.py.151: total_norm: 24.163054 [I 2022-10-09 11:22:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 2010/2304 [I 2022-10-09 11:22:54] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:08 [I 2022-10-09 11:22:54] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00296 [I 2022-10-09 11:22:54] ...trap/engines/engine.py.230: loss: 0.11033 [I 2022-10-09 11:22:54] ...optimizers/trijoint.py.151: total_norm: 18.534788 [I 2022-10-09 11:23:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 2020/2304 [I 2022-10-09 11:23:02] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:59 [I 2022-10-09 11:23:02] ...trap/engines/engine.py.229: process: 0.83755 | load: 0.00293 [I 2022-10-09 11:23:02] ...trap/engines/engine.py.230: loss: 0.17580 [I 2022-10-09 11:23:02] ...optimizers/trijoint.py.151: total_norm: 24.713833 [I 2022-10-09 11:23:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 2030/2304 [I 2022-10-09 11:23:10] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:50 [I 2022-10-09 11:23:10] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00290 [I 2022-10-09 11:23:10] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-09 11:23:10] ...optimizers/trijoint.py.151: total_norm: 16.073999 [I 2022-10-09 11:23:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 2040/2304 [I 2022-10-09 11:23:19] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:41 [I 2022-10-09 11:23:19] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00294 [I 2022-10-09 11:23:19] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-09 11:23:19] ...optimizers/trijoint.py.151: total_norm: 14.165730 [I 2022-10-09 11:23:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 2050/2304 [I 2022-10-09 11:23:27] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:33 [I 2022-10-09 11:23:27] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00294 [I 2022-10-09 11:23:27] ...trap/engines/engine.py.230: loss: 0.22346 [I 2022-10-09 11:23:27] ...optimizers/trijoint.py.151: total_norm: 16.641365 [I 2022-10-09 11:23:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 2060/2304 [I 2022-10-09 11:23:36] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:24 [I 2022-10-09 11:23:36] ...trap/engines/engine.py.229: process: 0.83177 | load: 0.00291 [I 2022-10-09 11:23:36] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-09 11:23:36] ...optimizers/trijoint.py.151: total_norm: 18.093250 [I 2022-10-09 11:23:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 2070/2304 [I 2022-10-09 11:23:44] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:16 [I 2022-10-09 11:23:44] ...trap/engines/engine.py.229: process: 0.83989 | load: 0.00296 [I 2022-10-09 11:23:44] ...trap/engines/engine.py.230: loss: 0.19886 [I 2022-10-09 11:23:44] ...optimizers/trijoint.py.151: total_norm: 27.450239 [I 2022-10-09 11:23:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 2080/2304 [I 2022-10-09 11:23:52] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:08 [I 2022-10-09 11:23:52] ...trap/engines/engine.py.229: process: 0.83986 | load: 0.00292 [I 2022-10-09 11:23:52] ...trap/engines/engine.py.230: loss: 0.12355 [I 2022-10-09 11:23:52] ...optimizers/trijoint.py.151: total_norm: 10.783348 [I 2022-10-09 11:24:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 2090/2304 [I 2022-10-09 11:24:01] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:02:59 [I 2022-10-09 11:24:01] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00294 [I 2022-10-09 11:24:01] ...trap/engines/engine.py.230: loss: 0.28299 [I 2022-10-09 11:24:01] ...optimizers/trijoint.py.151: total_norm: 17.294334 [I 2022-10-09 11:24:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 2100/2304 [I 2022-10-09 11:24:09] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:51 [I 2022-10-09 11:24:09] ...trap/engines/engine.py.229: process: 0.84877 | load: 0.00294 [I 2022-10-09 11:24:09] ...trap/engines/engine.py.230: loss: 0.17211 [I 2022-10-09 11:24:09] ...optimizers/trijoint.py.151: total_norm: 12.447784 [I 2022-10-09 11:24:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 2110/2304 [I 2022-10-09 11:24:18] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:43 [I 2022-10-09 11:24:18] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00290 [I 2022-10-09 11:24:18] ...trap/engines/engine.py.230: loss: 0.13093 [I 2022-10-09 11:24:18] ...optimizers/trijoint.py.151: total_norm: 12.618912 [I 2022-10-09 11:24:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 2120/2304 [I 2022-10-09 11:24:26] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:34 [I 2022-10-09 11:24:26] ...trap/engines/engine.py.229: process: 0.82947 | load: 0.00293 [I 2022-10-09 11:24:26] ...trap/engines/engine.py.230: loss: 0.12639 [I 2022-10-09 11:24:26] ...optimizers/trijoint.py.151: total_norm: 19.519672 [I 2022-10-09 11:24:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 2130/2304 [I 2022-10-09 11:24:35] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:26 [I 2022-10-09 11:24:35] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00291 [I 2022-10-09 11:24:35] ...trap/engines/engine.py.230: loss: 0.13992 [I 2022-10-09 11:24:35] ...optimizers/trijoint.py.151: total_norm: 9.649564 [I 2022-10-09 11:24:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 2140/2304 [I 2022-10-09 11:24:43] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:17 [I 2022-10-09 11:24:43] ...trap/engines/engine.py.229: process: 0.82627 | load: 0.00292 [I 2022-10-09 11:24:43] ...trap/engines/engine.py.230: loss: 0.12817 [I 2022-10-09 11:24:43] ...optimizers/trijoint.py.151: total_norm: 15.947452 [I 2022-10-09 11:24:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 2150/2304 [I 2022-10-09 11:24:51] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:09 [I 2022-10-09 11:24:51] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00297 [I 2022-10-09 11:24:51] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-09 11:24:51] ...optimizers/trijoint.py.151: total_norm: 21.248602 [I 2022-10-09 11:25:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 2160/2304 [I 2022-10-09 11:25:00] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:01 [I 2022-10-09 11:25:00] ...trap/engines/engine.py.229: process: 0.84107 | load: 0.00308 [I 2022-10-09 11:25:00] ...trap/engines/engine.py.230: loss: 0.14294 [I 2022-10-09 11:25:00] ...optimizers/trijoint.py.151: total_norm: 14.340703 [I 2022-10-09 11:25:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 2170/2304 [I 2022-10-09 11:25:08] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:53 [I 2022-10-09 11:25:08] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00291 [I 2022-10-09 11:25:08] ...trap/engines/engine.py.230: loss: 0.23073 [I 2022-10-09 11:25:08] ...optimizers/trijoint.py.151: total_norm: 25.038435 [I 2022-10-09 11:25:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 2180/2304 [I 2022-10-09 11:25:17] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:44 [I 2022-10-09 11:25:17] ...trap/engines/engine.py.229: process: 0.84051 | load: 0.00292 [I 2022-10-09 11:25:17] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-09 11:25:17] ...optimizers/trijoint.py.151: total_norm: 15.305583 [I 2022-10-09 11:25:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 2190/2304 [I 2022-10-09 11:25:25] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:36 [I 2022-10-09 11:25:25] ...trap/engines/engine.py.229: process: 0.84228 | load: 0.00296 [I 2022-10-09 11:25:25] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-09 11:25:25] ...optimizers/trijoint.py.151: total_norm: 15.477881 [I 2022-10-09 11:25:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 2200/2304 [I 2022-10-09 11:25:34] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:27 [I 2022-10-09 11:25:34] ...trap/engines/engine.py.229: process: 0.82947 | load: 0.00292 [I 2022-10-09 11:25:34] ...trap/engines/engine.py.230: loss: 0.11668 [I 2022-10-09 11:25:34] ...optimizers/trijoint.py.151: total_norm: 9.225473 [I 2022-10-09 11:25:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 2210/2304 [I 2022-10-09 11:25:42] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:19 [I 2022-10-09 11:25:42] ...trap/engines/engine.py.229: process: 0.84414 | load: 0.00290 [I 2022-10-09 11:25:42] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-09 11:25:42] ...optimizers/trijoint.py.151: total_norm: 22.741541 [I 2022-10-09 11:25:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 2220/2304 [I 2022-10-09 11:25:50] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:11 [I 2022-10-09 11:25:50] ...trap/engines/engine.py.229: process: 0.86066 | load: 0.00292 [I 2022-10-09 11:25:50] ...trap/engines/engine.py.230: loss: 0.14427 [I 2022-10-09 11:25:50] ...optimizers/trijoint.py.151: total_norm: 38.845070 [I 2022-10-09 11:25:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 2230/2304 [I 2022-10-09 11:25:59] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:02 [I 2022-10-09 11:25:59] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00295 [I 2022-10-09 11:25:59] ...trap/engines/engine.py.230: loss: 0.14093 [I 2022-10-09 11:25:59] ...optimizers/trijoint.py.151: total_norm: 12.148950 [I 2022-10-09 11:26:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 2240/2304 [I 2022-10-09 11:26:08] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:54 [I 2022-10-09 11:26:08] ...trap/engines/engine.py.229: process: 0.85628 | load: 0.00293 [I 2022-10-09 11:26:08] ...trap/engines/engine.py.230: loss: 0.21351 [I 2022-10-09 11:26:08] ...optimizers/trijoint.py.151: total_norm: 19.179760 [I 2022-10-09 11:26:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 2250/2304 [I 2022-10-09 11:26:16] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:46 [I 2022-10-09 11:26:16] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00297 [I 2022-10-09 11:26:16] ...trap/engines/engine.py.230: loss: 0.13347 [I 2022-10-09 11:26:16] ...optimizers/trijoint.py.151: total_norm: 15.376441 [I 2022-10-09 11:26:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 2260/2304 [I 2022-10-09 11:26:25] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:37 [I 2022-10-09 11:26:25] ...trap/engines/engine.py.229: process: 0.85913 | load: 0.00290 [I 2022-10-09 11:26:25] ...trap/engines/engine.py.230: loss: 0.13095 [I 2022-10-09 11:26:25] ...optimizers/trijoint.py.151: total_norm: 11.961580 [I 2022-10-09 11:26:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 2270/2304 [I 2022-10-09 11:26:33] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:28 [I 2022-10-09 11:26:33] ...trap/engines/engine.py.229: process: 0.83263 | load: 0.00294 [I 2022-10-09 11:26:33] ...trap/engines/engine.py.230: loss: 0.16448 [I 2022-10-09 11:26:33] ...optimizers/trijoint.py.151: total_norm: 43.804478 [I 2022-10-09 11:26:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 2280/2304 [I 2022-10-09 11:26:42] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:20 [I 2022-10-09 11:26:42] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00295 [I 2022-10-09 11:26:42] ...trap/engines/engine.py.230: loss: 0.11915 [I 2022-10-09 11:26:42] ...optimizers/trijoint.py.151: total_norm: 13.516701 [I 2022-10-09 11:26:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 2290/2304 [I 2022-10-09 11:26:50] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:11 [I 2022-10-09 11:26:50] ...trap/engines/engine.py.229: process: 0.83772 | load: 0.00295 [I 2022-10-09 11:26:50] ...trap/engines/engine.py.230: loss: 0.19555 [I 2022-10-09 11:26:50] ...optimizers/trijoint.py.151: total_norm: 21.843557 [I 2022-10-09 11:26:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 2300/2304 [I 2022-10-09 11:26:58] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:03 [I 2022-10-09 11:26:58] ...trap/engines/engine.py.229: process: 0.82538 | load: 0.00015 [I 2022-10-09 11:26:58] ...trap/engines/engine.py.230: loss: 0.12865 [I 2022-10-09 11:26:58] ...optimizers/trijoint.py.151: total_norm: 25.743170 [I 2022-10-09 11:27:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 2304/2304 [I 2022-10-09 11:27:02] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:00 [I 2022-10-09 11:27:02] ...trap/engines/engine.py.229: process: 0.82600 | load: 0.00015 [I 2022-10-09 11:27:02] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-09 11:27:02] ...optimizers/trijoint.py.151: total_norm: 14.899107 [S 2022-10-09 11:27:03] ...trap/engines/engine.py.129: train_epoch.epoch: 38 [S 2022-10-09 11:27:03] ...trap/engines/engine.py.129: train_epoch.loss: 0.1587315001127001 [I 2022-10-09 11:27:03] ...s/criterions/triplet.py.84: margin = 0.24000000000000005 [I 2022-10-09 11:27:03] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 38 [I 2022-10-09 11:27:04] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 11:27:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 11:27:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 11:27:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 11:27:06] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 11:27:07] ...trap/engines/engine.py.321: eval: epoch 38 | batch 0/510 [I 2022-10-09 11:27:07] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:26:45 [I 2022-10-09 11:27:07] ...trap/engines/engine.py.326: process: 3.14776 | load: 2.65389 [I 2022-10-09 11:27:12] ...trap/engines/engine.py.321: eval: epoch 38 | batch 10/510 [I 2022-10-09 11:27:12] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:31 [I 2022-10-09 11:27:12] ...trap/engines/engine.py.326: process: 0.48021 | load: 0.00019 [I 2022-10-09 11:27:20] ...trap/engines/engine.py.321: eval: epoch 38 | batch 20/510 [I 2022-10-09 11:27:20] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:07:42 [I 2022-10-09 11:27:20] ...trap/engines/engine.py.326: process: 0.82599 | load: 0.00017 [I 2022-10-09 11:27:27] ...trap/engines/engine.py.321: eval: epoch 38 | batch 30/510 [I 2022-10-09 11:27:27] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:06:30 [I 2022-10-09 11:27:27] ...trap/engines/engine.py.326: process: 1.04034 | load: 0.00018 [I 2022-10-09 11:27:33] ...trap/engines/engine.py.321: eval: epoch 38 | batch 40/510 [I 2022-10-09 11:27:33] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:50 [I 2022-10-09 11:27:33] ...trap/engines/engine.py.326: process: 0.63778 | load: 0.00018 [I 2022-10-09 11:27:37] ...trap/engines/engine.py.321: eval: epoch 38 | batch 50/510 [I 2022-10-09 11:27:37] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:54 [I 2022-10-09 11:27:37] ...trap/engines/engine.py.326: process: 0.59464 | load: 0.00017 [I 2022-10-09 11:27:42] ...trap/engines/engine.py.321: eval: epoch 38 | batch 60/510 [I 2022-10-09 11:27:42] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:47 [I 2022-10-09 11:27:42] ...trap/engines/engine.py.326: process: 0.58007 | load: 0.00017 [I 2022-10-09 11:27:48] ...trap/engines/engine.py.321: eval: epoch 38 | batch 70/510 [I 2022-10-09 11:27:48] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:15 [I 2022-10-09 11:27:48] ...trap/engines/engine.py.326: process: 0.59349 | load: 0.00018 [I 2022-10-09 11:27:53] ...trap/engines/engine.py.321: eval: epoch 38 | batch 80/510 [I 2022-10-09 11:27:53] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:34 [I 2022-10-09 11:27:53] ...trap/engines/engine.py.326: process: 0.55068 | load: 0.00018 [I 2022-10-09 11:27:57] ...trap/engines/engine.py.321: eval: epoch 38 | batch 90/510 [I 2022-10-09 11:27:57] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:01 [I 2022-10-09 11:27:57] ...trap/engines/engine.py.326: process: 0.43046 | load: 0.00018 [I 2022-10-09 11:28:01] ...trap/engines/engine.py.321: eval: epoch 38 | batch 100/510 [I 2022-10-09 11:28:01] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:47 [I 2022-10-09 11:28:01] ...trap/engines/engine.py.326: process: 0.30823 | load: 0.00018 [I 2022-10-09 11:28:04] ...trap/engines/engine.py.321: eval: epoch 38 | batch 110/510 [I 2022-10-09 11:28:04] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:10 [I 2022-10-09 11:28:04] ...trap/engines/engine.py.326: process: 0.31557 | load: 0.00018 [I 2022-10-09 11:28:08] ...trap/engines/engine.py.321: eval: epoch 38 | batch 120/510 [I 2022-10-09 11:28:08] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:03 [I 2022-10-09 11:28:08] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00018 [I 2022-10-09 11:28:11] ...trap/engines/engine.py.321: eval: epoch 38 | batch 130/510 [I 2022-10-09 11:28:11] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:00 [I 2022-10-09 11:28:11] ...trap/engines/engine.py.326: process: 0.31953 | load: 0.00018 [I 2022-10-09 11:28:14] ...trap/engines/engine.py.321: eval: epoch 38 | batch 140/510 [I 2022-10-09 11:28:14] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:56 [I 2022-10-09 11:28:14] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00018 [I 2022-10-09 11:28:17] ...trap/engines/engine.py.321: eval: epoch 38 | batch 150/510 [I 2022-10-09 11:28:17] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:53 [I 2022-10-09 11:28:17] ...trap/engines/engine.py.326: process: 0.31519 | load: 0.00020 [I 2022-10-09 11:28:20] ...trap/engines/engine.py.321: eval: epoch 38 | batch 160/510 [I 2022-10-09 11:28:20] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:49 [I 2022-10-09 11:28:20] ...trap/engines/engine.py.326: process: 0.30520 | load: 0.00018 [I 2022-10-09 11:28:23] ...trap/engines/engine.py.321: eval: epoch 38 | batch 170/510 [I 2022-10-09 11:28:23] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-09 11:28:23] ...trap/engines/engine.py.326: process: 0.31618 | load: 0.00019 [I 2022-10-09 11:28:27] ...trap/engines/engine.py.321: eval: epoch 38 | batch 180/510 [I 2022-10-09 11:28:27] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:44 [I 2022-10-09 11:28:27] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00019 [I 2022-10-09 11:28:30] ...trap/engines/engine.py.321: eval: epoch 38 | batch 190/510 [I 2022-10-09 11:28:30] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:40 [I 2022-10-09 11:28:30] ...trap/engines/engine.py.326: process: 0.31505 | load: 0.00018 [I 2022-10-09 11:28:33] ...trap/engines/engine.py.321: eval: epoch 38 | batch 200/510 [I 2022-10-09 11:28:33] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:38 [I 2022-10-09 11:28:33] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00019 [I 2022-10-09 11:28:36] ...trap/engines/engine.py.321: eval: epoch 38 | batch 210/510 [I 2022-10-09 11:28:36] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:34 [I 2022-10-09 11:28:36] ...trap/engines/engine.py.326: process: 0.31056 | load: 0.00019 [I 2022-10-09 11:28:39] ...trap/engines/engine.py.321: eval: epoch 38 | batch 220/510 [I 2022-10-09 11:28:39] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:31 [I 2022-10-09 11:28:39] ...trap/engines/engine.py.326: process: 0.31586 | load: 0.00018 [I 2022-10-09 11:28:42] ...trap/engines/engine.py.321: eval: epoch 38 | batch 230/510 [I 2022-10-09 11:28:42] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:28 [I 2022-10-09 11:28:42] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00020 [I 2022-10-09 11:28:45] ...trap/engines/engine.py.321: eval: epoch 38 | batch 240/510 [I 2022-10-09 11:28:45] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:25 [I 2022-10-09 11:28:45] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00019 [I 2022-10-09 11:28:49] ...trap/engines/engine.py.321: eval: epoch 38 | batch 250/510 [I 2022-10-09 11:28:49] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:22 [I 2022-10-09 11:28:49] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00020 [I 2022-10-09 11:28:52] ...trap/engines/engine.py.321: eval: epoch 38 | batch 260/510 [I 2022-10-09 11:28:52] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:19 [I 2022-10-09 11:28:52] ...trap/engines/engine.py.326: process: 0.31691 | load: 0.00019 [I 2022-10-09 11:28:55] ...trap/engines/engine.py.321: eval: epoch 38 | batch 270/510 [I 2022-10-09 11:28:55] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:15 [I 2022-10-09 11:28:55] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00019 [I 2022-10-09 11:28:58] ...trap/engines/engine.py.321: eval: epoch 38 | batch 280/510 [I 2022-10-09 11:28:58] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:12 [I 2022-10-09 11:28:58] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00019 [I 2022-10-09 11:29:01] ...trap/engines/engine.py.321: eval: epoch 38 | batch 290/510 [I 2022-10-09 11:29:01] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:09 [I 2022-10-09 11:29:01] ...trap/engines/engine.py.326: process: 0.31988 | load: 0.00020 [I 2022-10-09 11:29:04] ...trap/engines/engine.py.321: eval: epoch 38 | batch 300/510 [I 2022-10-09 11:29:04] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-09 11:29:04] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00020 [I 2022-10-09 11:29:08] ...trap/engines/engine.py.321: eval: epoch 38 | batch 310/510 [I 2022-10-09 11:29:08] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-09 11:29:08] ...trap/engines/engine.py.326: process: 0.31743 | load: 0.00019 [I 2022-10-09 11:29:11] ...trap/engines/engine.py.321: eval: epoch 38 | batch 320/510 [I 2022-10-09 11:29:11] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-09 11:29:11] ...trap/engines/engine.py.326: process: 0.31774 | load: 0.00020 [I 2022-10-09 11:29:14] ...trap/engines/engine.py.321: eval: epoch 38 | batch 330/510 [I 2022-10-09 11:29:14] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-09 11:29:14] ...trap/engines/engine.py.326: process: 0.32069 | load: 0.00019 [I 2022-10-09 11:29:17] ...trap/engines/engine.py.321: eval: epoch 38 | batch 340/510 [I 2022-10-09 11:29:17] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:53 [I 2022-10-09 11:29:17] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-09 11:29:20] ...trap/engines/engine.py.321: eval: epoch 38 | batch 350/510 [I 2022-10-09 11:29:20] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:50 [I 2022-10-09 11:29:20] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00020 [I 2022-10-09 11:29:23] ...trap/engines/engine.py.321: eval: epoch 38 | batch 360/510 [I 2022-10-09 11:29:23] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-09 11:29:23] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00019 [I 2022-10-09 11:29:27] ...trap/engines/engine.py.321: eval: epoch 38 | batch 370/510 [I 2022-10-09 11:29:27] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-09 11:29:27] ...trap/engines/engine.py.326: process: 0.31340 | load: 0.00019 [I 2022-10-09 11:29:30] ...trap/engines/engine.py.321: eval: epoch 38 | batch 380/510 [I 2022-10-09 11:29:30] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:40 [I 2022-10-09 11:29:30] ...trap/engines/engine.py.326: process: 0.31278 | load: 0.00019 [I 2022-10-09 11:29:33] ...trap/engines/engine.py.321: eval: epoch 38 | batch 390/510 [I 2022-10-09 11:29:33] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:37 [I 2022-10-09 11:29:33] ...trap/engines/engine.py.326: process: 0.31714 | load: 0.00018 [I 2022-10-09 11:29:36] ...trap/engines/engine.py.321: eval: epoch 38 | batch 400/510 [I 2022-10-09 11:29:36] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:34 [I 2022-10-09 11:29:36] ...trap/engines/engine.py.326: process: 0.31697 | load: 0.00020 [I 2022-10-09 11:29:39] ...trap/engines/engine.py.321: eval: epoch 38 | batch 410/510 [I 2022-10-09 11:29:39] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-09 11:29:39] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00021 [I 2022-10-09 11:29:42] ...trap/engines/engine.py.321: eval: epoch 38 | batch 420/510 [I 2022-10-09 11:29:42] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-09 11:29:42] ...trap/engines/engine.py.326: process: 0.32062 | load: 0.00019 [I 2022-10-09 11:29:46] ...trap/engines/engine.py.321: eval: epoch 38 | batch 430/510 [I 2022-10-09 11:29:46] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-09 11:29:46] ...trap/engines/engine.py.326: process: 0.32059 | load: 0.00019 [I 2022-10-09 11:29:49] ...trap/engines/engine.py.321: eval: epoch 38 | batch 440/510 [I 2022-10-09 11:29:49] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-09 11:29:49] ...trap/engines/engine.py.326: process: 0.32061 | load: 0.00020 [I 2022-10-09 11:29:52] ...trap/engines/engine.py.321: eval: epoch 38 | batch 450/510 [I 2022-10-09 11:29:52] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-09 11:29:52] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00019 [I 2022-10-09 11:29:55] ...trap/engines/engine.py.321: eval: epoch 38 | batch 460/510 [I 2022-10-09 11:29:55] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-09 11:29:55] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00020 [I 2022-10-09 11:29:58] ...trap/engines/engine.py.321: eval: epoch 38 | batch 470/510 [I 2022-10-09 11:29:58] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-09 11:29:58] ...trap/engines/engine.py.326: process: 0.31753 | load: 0.00020 [I 2022-10-09 11:30:02] ...trap/engines/engine.py.321: eval: epoch 38 | batch 480/510 [I 2022-10-09 11:30:02] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-09 11:30:02] ...trap/engines/engine.py.326: process: 0.32169 | load: 0.00019 [I 2022-10-09 11:30:05] ...trap/engines/engine.py.321: eval: epoch 38 | batch 490/510 [I 2022-10-09 11:30:05] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-09 11:30:05] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00019 [I 2022-10-09 11:30:08] ...trap/engines/engine.py.321: eval: epoch 38 | batch 500/510 [I 2022-10-09 11:30:08] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-09 11:30:08] ...trap/engines/engine.py.326: process: 0.31586 | load: 0.00019 [I 2022-10-09 11:30:11] ...trap/engines/engine.py.321: eval: epoch 38 | batch 510/510 [I 2022-10-09 11:30:11] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-09 11:30:11] ...trap/engines/engine.py.326: process: 0.31638 | load: 0.00015 [S 2022-10-09 11:30:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 38 [I 2022-10-09 11:30:12] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.710800 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905200 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936300 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.698800 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900700 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933600 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.019025 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009421 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009487 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016564 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010826 [S 2022-10-09 11:30:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009035 [I 2022-10-09 11:30:55] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 11:30:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 11:30:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 11:30:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 11:30:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 11:30:59] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 11:30:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 11:31:05] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 11:31:05] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 11:31:05] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 11:31:08] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 11:31:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 11:31:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 39 [I 2022-10-09 11:31:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 11:31:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 0/2304 [I 2022-10-09 11:31:16] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:27 [I 2022-10-09 11:31:16] ...trap/engines/engine.py.229: process: 2.61612 | load: 1.71230 [I 2022-10-09 11:31:16] ...trap/engines/engine.py.230: loss: 0.13978 [I 2022-10-09 11:31:16] ...optimizers/trijoint.py.151: total_norm: 11.888884 [I 2022-10-09 11:31:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 10/2304 [I 2022-10-09 11:31:25] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:27 [I 2022-10-09 11:31:25] ...trap/engines/engine.py.229: process: 0.83552 | load: 0.00491 [I 2022-10-09 11:31:25] ...trap/engines/engine.py.230: loss: 0.19258 [I 2022-10-09 11:31:25] ...optimizers/trijoint.py.151: total_norm: 53.719143 [I 2022-10-09 11:31:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 20/2304 [I 2022-10-09 11:31:33] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:59 [I 2022-10-09 11:31:33] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00436 [I 2022-10-09 11:31:33] ...trap/engines/engine.py.230: loss: 0.11665 [I 2022-10-09 11:31:33] ...optimizers/trijoint.py.151: total_norm: 21.125614 [I 2022-10-09 11:31:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 30/2304 [I 2022-10-09 11:31:42] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:31:58 [I 2022-10-09 11:31:42] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00426 [I 2022-10-09 11:31:42] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-09 11:31:42] ...optimizers/trijoint.py.151: total_norm: 11.274213 [I 2022-10-09 11:31:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 40/2304 [I 2022-10-09 11:31:50] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:49 [I 2022-10-09 11:31:50] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00480 [I 2022-10-09 11:31:50] ...trap/engines/engine.py.230: loss: 0.19096 [I 2022-10-09 11:31:50] ...optimizers/trijoint.py.151: total_norm: 26.970125 [I 2022-10-09 11:31:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 50/2304 [I 2022-10-09 11:31:58] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:51 [I 2022-10-09 11:31:58] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00377 [I 2022-10-09 11:31:58] ...trap/engines/engine.py.230: loss: 0.15613 [I 2022-10-09 11:31:58] ...optimizers/trijoint.py.151: total_norm: 19.747620 [I 2022-10-09 11:32:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 60/2304 [I 2022-10-09 11:32:07] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:35 [I 2022-10-09 11:32:07] ...trap/engines/engine.py.229: process: 0.83552 | load: 0.00327 [I 2022-10-09 11:32:07] ...trap/engines/engine.py.230: loss: 0.13460 [I 2022-10-09 11:32:07] ...optimizers/trijoint.py.151: total_norm: 37.280563 [I 2022-10-09 11:32:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 70/2304 [I 2022-10-09 11:32:15] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:17 [I 2022-10-09 11:32:15] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00381 [I 2022-10-09 11:32:15] ...trap/engines/engine.py.230: loss: 0.19234 [I 2022-10-09 11:32:15] ...optimizers/trijoint.py.151: total_norm: 17.389477 [I 2022-10-09 11:32:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 80/2304 [I 2022-10-09 11:32:24] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:09 [I 2022-10-09 11:32:24] ...trap/engines/engine.py.229: process: 0.83646 | load: 0.00320 [I 2022-10-09 11:32:24] ...trap/engines/engine.py.230: loss: 0.12696 [I 2022-10-09 11:32:24] ...optimizers/trijoint.py.151: total_norm: 14.707369 [I 2022-10-09 11:32:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 90/2304 [I 2022-10-09 11:32:32] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:11 [I 2022-10-09 11:32:32] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00352 [I 2022-10-09 11:32:32] ...trap/engines/engine.py.230: loss: 0.16457 [I 2022-10-09 11:32:32] ...optimizers/trijoint.py.151: total_norm: 18.089025 [I 2022-10-09 11:32:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 100/2304 [I 2022-10-09 11:32:41] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:51 [I 2022-10-09 11:32:41] ...trap/engines/engine.py.229: process: 0.82893 | load: 0.00327 [I 2022-10-09 11:32:41] ...trap/engines/engine.py.230: loss: 0.12165 [I 2022-10-09 11:32:41] ...optimizers/trijoint.py.151: total_norm: 10.269358 [I 2022-10-09 11:32:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 110/2304 [I 2022-10-09 11:32:49] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:41 [I 2022-10-09 11:32:49] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00329 [I 2022-10-09 11:32:49] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-09 11:32:49] ...optimizers/trijoint.py.151: total_norm: 14.237202 [I 2022-10-09 11:32:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 120/2304 [I 2022-10-09 11:32:57] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:36 [I 2022-10-09 11:32:57] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00291 [I 2022-10-09 11:32:57] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-09 11:32:57] ...optimizers/trijoint.py.151: total_norm: 15.397287 [I 2022-10-09 11:33:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 130/2304 [I 2022-10-09 11:33:06] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:31 [I 2022-10-09 11:33:06] ...trap/engines/engine.py.229: process: 0.84197 | load: 0.00296 [I 2022-10-09 11:33:06] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-09 11:33:06] ...optimizers/trijoint.py.151: total_norm: 22.185774 [I 2022-10-09 11:33:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 140/2304 [I 2022-10-09 11:33:14] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:11 [I 2022-10-09 11:33:14] ...trap/engines/engine.py.229: process: 0.82788 | load: 0.00297 [I 2022-10-09 11:33:14] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-09 11:33:14] ...optimizers/trijoint.py.151: total_norm: 7.480714 [I 2022-10-09 11:33:23] ...trap/engines/engine.py.224: train: epoch 39 | batch 150/2304 [I 2022-10-09 11:33:23] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:11 [I 2022-10-09 11:33:23] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00306 [I 2022-10-09 11:33:23] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-09 11:33:23] ...optimizers/trijoint.py.151: total_norm: 19.897379 [I 2022-10-09 11:33:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 160/2304 [I 2022-10-09 11:33:31] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:13 [I 2022-10-09 11:33:31] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00331 [I 2022-10-09 11:33:31] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-09 11:33:31] ...optimizers/trijoint.py.151: total_norm: 12.191569 [I 2022-10-09 11:33:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 170/2304 [I 2022-10-09 11:33:40] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:29:50 [I 2022-10-09 11:33:40] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00343 [I 2022-10-09 11:33:40] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-09 11:33:40] ...optimizers/trijoint.py.151: total_norm: 17.216747 [I 2022-10-09 11:33:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 180/2304 [I 2022-10-09 11:33:48] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:51 [I 2022-10-09 11:33:48] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00331 [I 2022-10-09 11:33:48] ...trap/engines/engine.py.230: loss: 0.22196 [I 2022-10-09 11:33:48] ...optimizers/trijoint.py.151: total_norm: 13.928897 [I 2022-10-09 11:33:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 190/2304 [I 2022-10-09 11:33:56] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:43 [I 2022-10-09 11:33:56] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00298 [I 2022-10-09 11:33:56] ...trap/engines/engine.py.230: loss: 0.10516 [I 2022-10-09 11:33:56] ...optimizers/trijoint.py.151: total_norm: 10.346173 [I 2022-10-09 11:34:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 200/2304 [I 2022-10-09 11:34:05] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:37 [I 2022-10-09 11:34:05] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00305 [I 2022-10-09 11:34:05] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-09 11:34:05] ...optimizers/trijoint.py.151: total_norm: 12.766221 [I 2022-10-09 11:34:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 210/2304 [I 2022-10-09 11:34:13] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:24 [I 2022-10-09 11:34:13] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00292 [I 2022-10-09 11:34:13] ...trap/engines/engine.py.230: loss: 0.14040 [I 2022-10-09 11:34:13] ...optimizers/trijoint.py.151: total_norm: 12.890800 [I 2022-10-09 11:34:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 220/2304 [I 2022-10-09 11:34:22] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:22 [I 2022-10-09 11:34:22] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00294 [I 2022-10-09 11:34:22] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-09 11:34:22] ...optimizers/trijoint.py.151: total_norm: 11.004899 [I 2022-10-09 11:34:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 230/2304 [I 2022-10-09 11:34:30] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:09 [I 2022-10-09 11:34:30] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00300 [I 2022-10-09 11:34:30] ...trap/engines/engine.py.230: loss: 0.13460 [I 2022-10-09 11:34:30] ...optimizers/trijoint.py.151: total_norm: 17.394274 [I 2022-10-09 11:34:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 240/2304 [I 2022-10-09 11:34:39] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:05 [I 2022-10-09 11:34:39] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00301 [I 2022-10-09 11:34:39] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-09 11:34:39] ...optimizers/trijoint.py.151: total_norm: 17.008598 [I 2022-10-09 11:34:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 250/2304 [I 2022-10-09 11:34:47] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:51 [I 2022-10-09 11:34:47] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00291 [I 2022-10-09 11:34:47] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-09 11:34:47] ...optimizers/trijoint.py.151: total_norm: 33.800598 [I 2022-10-09 11:34:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 260/2304 [I 2022-10-09 11:34:56] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:54 [I 2022-10-09 11:34:56] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00293 [I 2022-10-09 11:34:56] ...trap/engines/engine.py.230: loss: 0.17342 [I 2022-10-09 11:34:56] ...optimizers/trijoint.py.151: total_norm: 26.088686 [I 2022-10-09 11:35:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 270/2304 [I 2022-10-09 11:35:04] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:46 [I 2022-10-09 11:35:04] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00294 [I 2022-10-09 11:35:04] ...trap/engines/engine.py.230: loss: 0.15283 [I 2022-10-09 11:35:04] ...optimizers/trijoint.py.151: total_norm: 12.952350 [I 2022-10-09 11:35:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 280/2304 [I 2022-10-09 11:35:13] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:33 [I 2022-10-09 11:35:13] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00301 [I 2022-10-09 11:35:13] ...trap/engines/engine.py.230: loss: 0.16220 [I 2022-10-09 11:35:13] ...optimizers/trijoint.py.151: total_norm: 11.474585 [I 2022-10-09 11:35:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 290/2304 [I 2022-10-09 11:35:21] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:28 [I 2022-10-09 11:35:21] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00292 [I 2022-10-09 11:35:21] ...trap/engines/engine.py.230: loss: 0.15165 [I 2022-10-09 11:35:21] ...optimizers/trijoint.py.151: total_norm: 18.909935 [I 2022-10-09 11:35:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 300/2304 [I 2022-10-09 11:35:30] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:17 [I 2022-10-09 11:35:30] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00315 [I 2022-10-09 11:35:30] ...trap/engines/engine.py.230: loss: 0.11872 [I 2022-10-09 11:35:30] ...optimizers/trijoint.py.151: total_norm: 8.487880 [I 2022-10-09 11:35:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 310/2304 [I 2022-10-09 11:35:38] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:18 [I 2022-10-09 11:35:38] ...trap/engines/engine.py.229: process: 0.84038 | load: 0.00307 [I 2022-10-09 11:35:38] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-09 11:35:38] ...optimizers/trijoint.py.151: total_norm: 17.929815 [I 2022-10-09 11:35:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 320/2304 [I 2022-10-09 11:35:47] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:14 [I 2022-10-09 11:35:47] ...trap/engines/engine.py.229: process: 0.87440 | load: 0.00294 [I 2022-10-09 11:35:47] ...trap/engines/engine.py.230: loss: 0.16602 [I 2022-10-09 11:35:47] ...optimizers/trijoint.py.151: total_norm: 8.043120 [I 2022-10-09 11:35:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 330/2304 [I 2022-10-09 11:35:55] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:28:03 [I 2022-10-09 11:35:55] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00291 [I 2022-10-09 11:35:55] ...trap/engines/engine.py.230: loss: 0.17036 [I 2022-10-09 11:35:55] ...optimizers/trijoint.py.151: total_norm: 10.277155 [I 2022-10-09 11:36:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 340/2304 [I 2022-10-09 11:36:04] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:57 [I 2022-10-09 11:36:04] ...trap/engines/engine.py.229: process: 0.86314 | load: 0.00293 [I 2022-10-09 11:36:04] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-09 11:36:04] ...optimizers/trijoint.py.151: total_norm: 24.806307 [I 2022-10-09 11:36:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 350/2304 [I 2022-10-09 11:36:12] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:34 [I 2022-10-09 11:36:12] ...trap/engines/engine.py.229: process: 0.83921 | load: 0.00290 [I 2022-10-09 11:36:12] ...trap/engines/engine.py.230: loss: 0.18764 [I 2022-10-09 11:36:12] ...optimizers/trijoint.py.151: total_norm: 16.335382 [I 2022-10-09 11:36:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 360/2304 [I 2022-10-09 11:36:21] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:35 [I 2022-10-09 11:36:21] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00293 [I 2022-10-09 11:36:21] ...trap/engines/engine.py.230: loss: 0.12270 [I 2022-10-09 11:36:21] ...optimizers/trijoint.py.151: total_norm: 24.060951 [I 2022-10-09 11:36:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 370/2304 [I 2022-10-09 11:36:29] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:19 [I 2022-10-09 11:36:29] ...trap/engines/engine.py.229: process: 0.83261 | load: 0.00309 [I 2022-10-09 11:36:29] ...trap/engines/engine.py.230: loss: 0.16445 [I 2022-10-09 11:36:29] ...optimizers/trijoint.py.151: total_norm: 75.880737 [I 2022-10-09 11:36:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 380/2304 [I 2022-10-09 11:36:38] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:15 [I 2022-10-09 11:36:38] ...trap/engines/engine.py.229: process: 0.86458 | load: 0.00295 [I 2022-10-09 11:36:38] ...trap/engines/engine.py.230: loss: 0.20298 [I 2022-10-09 11:36:38] ...optimizers/trijoint.py.151: total_norm: 17.731884 [I 2022-10-09 11:36:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 390/2304 [I 2022-10-09 11:36:46] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:03 [I 2022-10-09 11:36:46] ...trap/engines/engine.py.229: process: 0.83857 | load: 0.00302 [I 2022-10-09 11:36:46] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-09 11:36:46] ...optimizers/trijoint.py.151: total_norm: 15.117807 [I 2022-10-09 11:36:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 400/2304 [I 2022-10-09 11:36:55] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:59 [I 2022-10-09 11:36:55] ...trap/engines/engine.py.229: process: 0.85886 | load: 0.00291 [I 2022-10-09 11:36:55] ...trap/engines/engine.py.230: loss: 0.13222 [I 2022-10-09 11:36:55] ...optimizers/trijoint.py.151: total_norm: 22.413948 [I 2022-10-09 11:37:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 410/2304 [I 2022-10-09 11:37:03] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:48 [I 2022-10-09 11:37:03] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00294 [I 2022-10-09 11:37:03] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-09 11:37:03] ...optimizers/trijoint.py.151: total_norm: 21.014338 [I 2022-10-09 11:37:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 420/2304 [I 2022-10-09 11:37:12] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:39 [I 2022-10-09 11:37:12] ...trap/engines/engine.py.229: process: 0.87365 | load: 0.00301 [I 2022-10-09 11:37:12] ...trap/engines/engine.py.230: loss: 0.13425 [I 2022-10-09 11:37:12] ...optimizers/trijoint.py.151: total_norm: 16.841171 [I 2022-10-09 11:37:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 430/2304 [I 2022-10-09 11:37:20] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:24 [I 2022-10-09 11:37:20] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00295 [I 2022-10-09 11:37:20] ...trap/engines/engine.py.230: loss: 0.15249 [I 2022-10-09 11:37:20] ...optimizers/trijoint.py.151: total_norm: 18.220598 [I 2022-10-09 11:37:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 440/2304 [I 2022-10-09 11:37:29] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:29 [I 2022-10-09 11:37:29] ...trap/engines/engine.py.229: process: 0.86593 | load: 0.00293 [I 2022-10-09 11:37:29] ...trap/engines/engine.py.230: loss: 0.14477 [I 2022-10-09 11:37:29] ...optimizers/trijoint.py.151: total_norm: 20.328114 [I 2022-10-09 11:37:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 450/2304 [I 2022-10-09 11:37:37] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:10 [I 2022-10-09 11:37:37] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00291 [I 2022-10-09 11:37:37] ...trap/engines/engine.py.230: loss: 0.16317 [I 2022-10-09 11:37:37] ...optimizers/trijoint.py.151: total_norm: 21.526518 [I 2022-10-09 11:37:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 460/2304 [I 2022-10-09 11:37:46] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:00 [I 2022-10-09 11:37:46] ...trap/engines/engine.py.229: process: 0.84983 | load: 0.00291 [I 2022-10-09 11:37:46] ...trap/engines/engine.py.230: loss: 0.19035 [I 2022-10-09 11:37:46] ...optimizers/trijoint.py.151: total_norm: 20.258160 [I 2022-10-09 11:37:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 470/2304 [I 2022-10-09 11:37:54] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:00 [I 2022-10-09 11:37:54] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00292 [I 2022-10-09 11:37:54] ...trap/engines/engine.py.230: loss: 0.15964 [I 2022-10-09 11:37:54] ...optimizers/trijoint.py.151: total_norm: 15.930100 [I 2022-10-09 11:38:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 480/2304 [I 2022-10-09 11:38:03] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:45 [I 2022-10-09 11:38:03] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00293 [I 2022-10-09 11:38:03] ...trap/engines/engine.py.230: loss: 0.19846 [I 2022-10-09 11:38:03] ...optimizers/trijoint.py.151: total_norm: 60.513824 [I 2022-10-09 11:38:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 490/2304 [I 2022-10-09 11:38:11] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:40 [I 2022-10-09 11:38:11] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00291 [I 2022-10-09 11:38:11] ...trap/engines/engine.py.230: loss: 0.14167 [I 2022-10-09 11:38:11] ...optimizers/trijoint.py.151: total_norm: 17.629379 [I 2022-10-09 11:38:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 500/2304 [I 2022-10-09 11:38:20] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:25 [I 2022-10-09 11:38:20] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00294 [I 2022-10-09 11:38:20] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-09 11:38:20] ...optimizers/trijoint.py.151: total_norm: 24.668077 [I 2022-10-09 11:38:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 510/2304 [I 2022-10-09 11:38:28] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:13 [I 2022-10-09 11:38:28] ...trap/engines/engine.py.229: process: 0.83368 | load: 0.00297 [I 2022-10-09 11:38:28] ...trap/engines/engine.py.230: loss: 0.17052 [I 2022-10-09 11:38:28] ...optimizers/trijoint.py.151: total_norm: 15.228489 [I 2022-10-09 11:38:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 520/2304 [I 2022-10-09 11:38:36] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:01 [I 2022-10-09 11:38:36] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00293 [I 2022-10-09 11:38:36] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-09 11:38:36] ...optimizers/trijoint.py.151: total_norm: 9.726383 [I 2022-10-09 11:38:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 530/2304 [I 2022-10-09 11:38:45] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:24:54 [I 2022-10-09 11:38:45] ...trap/engines/engine.py.229: process: 0.83694 | load: 0.00302 [I 2022-10-09 11:38:45] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-09 11:38:45] ...optimizers/trijoint.py.151: total_norm: 9.903389 [I 2022-10-09 11:38:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 540/2304 [I 2022-10-09 11:38:53] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:36 [I 2022-10-09 11:38:53] ...trap/engines/engine.py.229: process: 0.82989 | load: 0.00292 [I 2022-10-09 11:38:53] ...trap/engines/engine.py.230: loss: 0.19158 [I 2022-10-09 11:38:53] ...optimizers/trijoint.py.151: total_norm: 21.020540 [I 2022-10-09 11:39:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 550/2304 [I 2022-10-09 11:39:02] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:31 [I 2022-10-09 11:39:02] ...trap/engines/engine.py.229: process: 0.83191 | load: 0.00297 [I 2022-10-09 11:39:02] ...trap/engines/engine.py.230: loss: 0.15097 [I 2022-10-09 11:39:02] ...optimizers/trijoint.py.151: total_norm: 18.465969 [I 2022-10-09 11:39:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 560/2304 [I 2022-10-09 11:39:10] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:28 [I 2022-10-09 11:39:10] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00291 [I 2022-10-09 11:39:10] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-09 11:39:10] ...optimizers/trijoint.py.151: total_norm: 13.615538 [I 2022-10-09 11:39:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 570/2304 [I 2022-10-09 11:39:19] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:25 [I 2022-10-09 11:39:19] ...trap/engines/engine.py.229: process: 0.84864 | load: 0.00293 [I 2022-10-09 11:39:19] ...trap/engines/engine.py.230: loss: 0.10160 [I 2022-10-09 11:39:19] ...optimizers/trijoint.py.151: total_norm: 9.619105 [I 2022-10-09 11:39:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 580/2304 [I 2022-10-09 11:39:27] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:17 [I 2022-10-09 11:39:27] ...trap/engines/engine.py.229: process: 0.84712 | load: 0.00292 [I 2022-10-09 11:39:27] ...trap/engines/engine.py.230: loss: 0.13770 [I 2022-10-09 11:39:27] ...optimizers/trijoint.py.151: total_norm: 15.490688 [I 2022-10-09 11:39:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 590/2304 [I 2022-10-09 11:39:35] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:00 [I 2022-10-09 11:39:35] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00294 [I 2022-10-09 11:39:35] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-09 11:39:35] ...optimizers/trijoint.py.151: total_norm: 13.382212 [I 2022-10-09 11:39:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 600/2304 [I 2022-10-09 11:39:44] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:23:57 [I 2022-10-09 11:39:44] ...trap/engines/engine.py.229: process: 0.83710 | load: 0.00296 [I 2022-10-09 11:39:44] ...trap/engines/engine.py.230: loss: 0.28054 [I 2022-10-09 11:39:44] ...optimizers/trijoint.py.151: total_norm: 22.683088 [I 2022-10-09 11:39:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 610/2304 [I 2022-10-09 11:39:52] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:23:39 [I 2022-10-09 11:39:52] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00296 [I 2022-10-09 11:39:52] ...trap/engines/engine.py.230: loss: 0.15098 [I 2022-10-09 11:39:52] ...optimizers/trijoint.py.151: total_norm: 14.086403 [I 2022-10-09 11:40:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 620/2304 [I 2022-10-09 11:40:01] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:34 [I 2022-10-09 11:40:01] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00296 [I 2022-10-09 11:40:01] ...trap/engines/engine.py.230: loss: 0.15129 [I 2022-10-09 11:40:01] ...optimizers/trijoint.py.151: total_norm: 18.414268 [I 2022-10-09 11:40:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 630/2304 [I 2022-10-09 11:40:09] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:28 [I 2022-10-09 11:40:09] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00291 [I 2022-10-09 11:40:09] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-09 11:40:09] ...optimizers/trijoint.py.151: total_norm: 15.510310 [I 2022-10-09 11:40:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 640/2304 [I 2022-10-09 11:40:18] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:22 [I 2022-10-09 11:40:18] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00292 [I 2022-10-09 11:40:18] ...trap/engines/engine.py.230: loss: 0.17574 [I 2022-10-09 11:40:18] ...optimizers/trijoint.py.151: total_norm: 18.710329 [I 2022-10-09 11:40:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 650/2304 [I 2022-10-09 11:40:26] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:17 [I 2022-10-09 11:40:26] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00291 [I 2022-10-09 11:40:26] ...trap/engines/engine.py.230: loss: 0.15415 [I 2022-10-09 11:40:26] ...optimizers/trijoint.py.151: total_norm: 16.091393 [I 2022-10-09 11:40:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 660/2304 [I 2022-10-09 11:40:34] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:02 [I 2022-10-09 11:40:34] ...trap/engines/engine.py.229: process: 0.85413 | load: 0.00295 [I 2022-10-09 11:40:34] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-09 11:40:34] ...optimizers/trijoint.py.151: total_norm: 17.241377 [I 2022-10-09 11:40:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 670/2304 [I 2022-10-09 11:40:43] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:03 [I 2022-10-09 11:40:43] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00298 [I 2022-10-09 11:40:43] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-09 11:40:43] ...optimizers/trijoint.py.151: total_norm: 17.157806 [I 2022-10-09 11:40:51] ...trap/engines/engine.py.224: train: epoch 39 | batch 680/2304 [I 2022-10-09 11:40:51] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:22:45 [I 2022-10-09 11:40:51] ...trap/engines/engine.py.229: process: 0.83944 | load: 0.00303 [I 2022-10-09 11:40:51] ...trap/engines/engine.py.230: loss: 0.18584 [I 2022-10-09 11:40:51] ...optimizers/trijoint.py.151: total_norm: 16.340977 [I 2022-10-09 11:41:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 690/2304 [I 2022-10-09 11:41:00] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:41 [I 2022-10-09 11:41:00] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00292 [I 2022-10-09 11:41:00] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-09 11:41:00] ...optimizers/trijoint.py.151: total_norm: 13.866799 [I 2022-10-09 11:41:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 700/2304 [I 2022-10-09 11:41:08] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:34 [I 2022-10-09 11:41:08] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00292 [I 2022-10-09 11:41:08] ...trap/engines/engine.py.230: loss: 0.17498 [I 2022-10-09 11:41:08] ...optimizers/trijoint.py.151: total_norm: 10.129436 [I 2022-10-09 11:41:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 710/2304 [I 2022-10-09 11:41:17] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:24 [I 2022-10-09 11:41:17] ...trap/engines/engine.py.229: process: 0.83786 | load: 0.00292 [I 2022-10-09 11:41:17] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-09 11:41:17] ...optimizers/trijoint.py.151: total_norm: 15.535946 [I 2022-10-09 11:41:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 720/2304 [I 2022-10-09 11:41:25] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:17 [I 2022-10-09 11:41:25] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00293 [I 2022-10-09 11:41:25] ...trap/engines/engine.py.230: loss: 0.15996 [I 2022-10-09 11:41:25] ...optimizers/trijoint.py.151: total_norm: 21.114166 [I 2022-10-09 11:41:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 730/2304 [I 2022-10-09 11:41:33] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:07 [I 2022-10-09 11:41:33] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00298 [I 2022-10-09 11:41:33] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-09 11:41:33] ...optimizers/trijoint.py.151: total_norm: 15.130237 [I 2022-10-09 11:41:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 740/2304 [I 2022-10-09 11:41:42] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:21:56 [I 2022-10-09 11:41:42] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00291 [I 2022-10-09 11:41:42] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-09 11:41:42] ...optimizers/trijoint.py.151: total_norm: 41.058041 [I 2022-10-09 11:41:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 750/2304 [I 2022-10-09 11:41:50] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:21:47 [I 2022-10-09 11:41:50] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00293 [I 2022-10-09 11:41:50] ...trap/engines/engine.py.230: loss: 0.23055 [I 2022-10-09 11:41:50] ...optimizers/trijoint.py.151: total_norm: 17.133076 [I 2022-10-09 11:41:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 760/2304 [I 2022-10-09 11:41:59] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:21:43 [I 2022-10-09 11:41:59] ...trap/engines/engine.py.229: process: 0.84998 | load: 0.00298 [I 2022-10-09 11:41:59] ...trap/engines/engine.py.230: loss: 0.17398 [I 2022-10-09 11:41:59] ...optimizers/trijoint.py.151: total_norm: 15.107803 [I 2022-10-09 11:42:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 770/2304 [I 2022-10-09 11:42:07] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:21:29 [I 2022-10-09 11:42:07] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00292 [I 2022-10-09 11:42:07] ...trap/engines/engine.py.230: loss: 0.18674 [I 2022-10-09 11:42:07] ...optimizers/trijoint.py.151: total_norm: 30.920923 [I 2022-10-09 11:42:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 780/2304 [I 2022-10-09 11:42:16] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:26 [I 2022-10-09 11:42:16] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00292 [I 2022-10-09 11:42:16] ...trap/engines/engine.py.230: loss: 0.16387 [I 2022-10-09 11:42:16] ...optimizers/trijoint.py.151: total_norm: 16.151546 [I 2022-10-09 11:42:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 790/2304 [I 2022-10-09 11:42:24] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:12 [I 2022-10-09 11:42:24] ...trap/engines/engine.py.229: process: 0.84550 | load: 0.00294 [I 2022-10-09 11:42:24] ...trap/engines/engine.py.230: loss: 0.19297 [I 2022-10-09 11:42:24] ...optimizers/trijoint.py.151: total_norm: 29.594791 [I 2022-10-09 11:42:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 800/2304 [I 2022-10-09 11:42:32] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:07 [I 2022-10-09 11:42:32] ...trap/engines/engine.py.229: process: 0.83944 | load: 0.00293 [I 2022-10-09 11:42:32] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-09 11:42:32] ...optimizers/trijoint.py.151: total_norm: 15.196383 [I 2022-10-09 11:42:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 810/2304 [I 2022-10-09 11:42:41] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:20:53 [I 2022-10-09 11:42:41] ...trap/engines/engine.py.229: process: 0.83615 | load: 0.00288 [I 2022-10-09 11:42:41] ...trap/engines/engine.py.230: loss: 0.16565 [I 2022-10-09 11:42:41] ...optimizers/trijoint.py.151: total_norm: 23.990244 [I 2022-10-09 11:42:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 820/2304 [I 2022-10-09 11:42:49] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:20:49 [I 2022-10-09 11:42:49] ...trap/engines/engine.py.229: process: 0.83011 | load: 0.00294 [I 2022-10-09 11:42:49] ...trap/engines/engine.py.230: loss: 0.11888 [I 2022-10-09 11:42:49] ...optimizers/trijoint.py.151: total_norm: 9.843056 [I 2022-10-09 11:42:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 830/2304 [I 2022-10-09 11:42:58] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:20:41 [I 2022-10-09 11:42:58] ...trap/engines/engine.py.229: process: 0.84702 | load: 0.00291 [I 2022-10-09 11:42:58] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-09 11:42:58] ...optimizers/trijoint.py.151: total_norm: 17.955614 [I 2022-10-09 11:43:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 840/2304 [I 2022-10-09 11:43:06] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:20:35 [I 2022-10-09 11:43:06] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00293 [I 2022-10-09 11:43:06] ...trap/engines/engine.py.230: loss: 0.13276 [I 2022-10-09 11:43:06] ...optimizers/trijoint.py.151: total_norm: 32.983135 [I 2022-10-09 11:43:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 850/2304 [I 2022-10-09 11:43:15] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:20:24 [I 2022-10-09 11:43:15] ...trap/engines/engine.py.229: process: 0.83858 | load: 0.00291 [I 2022-10-09 11:43:15] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-09 11:43:15] ...optimizers/trijoint.py.151: total_norm: 5.728547 [I 2022-10-09 11:43:23] ...trap/engines/engine.py.224: train: epoch 39 | batch 860/2304 [I 2022-10-09 11:43:23] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:18 [I 2022-10-09 11:43:23] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00295 [I 2022-10-09 11:43:23] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-09 11:43:23] ...optimizers/trijoint.py.151: total_norm: 9.989029 [I 2022-10-09 11:43:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 870/2304 [I 2022-10-09 11:43:31] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:10 [I 2022-10-09 11:43:31] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00292 [I 2022-10-09 11:43:31] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-09 11:43:31] ...optimizers/trijoint.py.151: total_norm: 28.949633 [I 2022-10-09 11:43:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 880/2304 [I 2022-10-09 11:43:40] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:01 [I 2022-10-09 11:43:40] ...trap/engines/engine.py.229: process: 0.84925 | load: 0.00293 [I 2022-10-09 11:43:40] ...trap/engines/engine.py.230: loss: 0.15789 [I 2022-10-09 11:43:40] ...optimizers/trijoint.py.151: total_norm: 22.914320 [I 2022-10-09 11:43:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 890/2304 [I 2022-10-09 11:43:48] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:19:51 [I 2022-10-09 11:43:48] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00292 [I 2022-10-09 11:43:48] ...trap/engines/engine.py.230: loss: 0.19697 [I 2022-10-09 11:43:48] ...optimizers/trijoint.py.151: total_norm: 26.276171 [I 2022-10-09 11:43:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 900/2304 [I 2022-10-09 11:43:57] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:19:43 [I 2022-10-09 11:43:57] ...trap/engines/engine.py.229: process: 0.83886 | load: 0.00294 [I 2022-10-09 11:43:57] ...trap/engines/engine.py.230: loss: 0.11905 [I 2022-10-09 11:43:57] ...optimizers/trijoint.py.151: total_norm: 18.139406 [I 2022-10-09 11:44:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 910/2304 [I 2022-10-09 11:44:05] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:29 [I 2022-10-09 11:44:05] ...trap/engines/engine.py.229: process: 0.83324 | load: 0.00294 [I 2022-10-09 11:44:05] ...trap/engines/engine.py.230: loss: 0.17341 [I 2022-10-09 11:44:05] ...optimizers/trijoint.py.151: total_norm: 11.434031 [I 2022-10-09 11:44:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 920/2304 [I 2022-10-09 11:44:14] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:25 [I 2022-10-09 11:44:14] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00295 [I 2022-10-09 11:44:14] ...trap/engines/engine.py.230: loss: 0.16355 [I 2022-10-09 11:44:14] ...optimizers/trijoint.py.151: total_norm: 15.139902 [I 2022-10-09 11:44:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 930/2304 [I 2022-10-09 11:44:22] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:12 [I 2022-10-09 11:44:22] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00293 [I 2022-10-09 11:44:22] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-09 11:44:22] ...optimizers/trijoint.py.151: total_norm: 20.190144 [I 2022-10-09 11:44:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 940/2304 [I 2022-10-09 11:44:30] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:11 [I 2022-10-09 11:44:30] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00298 [I 2022-10-09 11:44:30] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-09 11:44:30] ...optimizers/trijoint.py.151: total_norm: 16.694862 [I 2022-10-09 11:44:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 950/2304 [I 2022-10-09 11:44:39] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:18:59 [I 2022-10-09 11:44:39] ...trap/engines/engine.py.229: process: 0.84948 | load: 0.00294 [I 2022-10-09 11:44:39] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-09 11:44:39] ...optimizers/trijoint.py.151: total_norm: 18.112034 [I 2022-10-09 11:44:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 960/2304 [I 2022-10-09 11:44:47] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:18:53 [I 2022-10-09 11:44:47] ...trap/engines/engine.py.229: process: 0.84437 | load: 0.00300 [I 2022-10-09 11:44:47] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-09 11:44:47] ...optimizers/trijoint.py.151: total_norm: 15.323978 [I 2022-10-09 11:44:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 970/2304 [I 2022-10-09 11:44:56] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:18:47 [I 2022-10-09 11:44:56] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00293 [I 2022-10-09 11:44:56] ...trap/engines/engine.py.230: loss: 0.17708 [I 2022-10-09 11:44:56] ...optimizers/trijoint.py.151: total_norm: 20.436993 [I 2022-10-09 11:45:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 980/2304 [I 2022-10-09 11:45:04] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:39 [I 2022-10-09 11:45:04] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00298 [I 2022-10-09 11:45:04] ...trap/engines/engine.py.230: loss: 0.17545 [I 2022-10-09 11:45:04] ...optimizers/trijoint.py.151: total_norm: 25.316904 [I 2022-10-09 11:45:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 990/2304 [I 2022-10-09 11:45:13] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:33 [I 2022-10-09 11:45:13] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00291 [I 2022-10-09 11:45:13] ...trap/engines/engine.py.230: loss: 0.18428 [I 2022-10-09 11:45:13] ...optimizers/trijoint.py.151: total_norm: 22.033371 [I 2022-10-09 11:45:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 1000/2304 [I 2022-10-09 11:45:21] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:17 [I 2022-10-09 11:45:21] ...trap/engines/engine.py.229: process: 0.84344 | load: 0.00292 [I 2022-10-09 11:45:21] ...trap/engines/engine.py.230: loss: 0.16204 [I 2022-10-09 11:45:21] ...optimizers/trijoint.py.151: total_norm: 17.533194 [I 2022-10-09 11:45:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1010/2304 [I 2022-10-09 11:45:29] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:07 [I 2022-10-09 11:45:29] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00294 [I 2022-10-09 11:45:29] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-09 11:45:29] ...optimizers/trijoint.py.151: total_norm: 10.341770 [I 2022-10-09 11:45:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 1020/2304 [I 2022-10-09 11:45:38] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:17:55 [I 2022-10-09 11:45:38] ...trap/engines/engine.py.229: process: 0.84305 | load: 0.00300 [I 2022-10-09 11:45:38] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-09 11:45:38] ...optimizers/trijoint.py.151: total_norm: 22.238934 [I 2022-10-09 11:45:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 1030/2304 [I 2022-10-09 11:45:46] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:17:50 [I 2022-10-09 11:45:46] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00295 [I 2022-10-09 11:45:46] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-09 11:45:46] ...optimizers/trijoint.py.151: total_norm: 25.054132 [I 2022-10-09 11:45:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 1040/2304 [I 2022-10-09 11:45:55] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:17:42 [I 2022-10-09 11:45:55] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00293 [I 2022-10-09 11:45:55] ...trap/engines/engine.py.230: loss: 0.10673 [I 2022-10-09 11:45:55] ...optimizers/trijoint.py.151: total_norm: 10.850118 [I 2022-10-09 11:46:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 1050/2304 [I 2022-10-09 11:46:03] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:35 [I 2022-10-09 11:46:03] ...trap/engines/engine.py.229: process: 0.83078 | load: 0.00293 [I 2022-10-09 11:46:03] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-09 11:46:03] ...optimizers/trijoint.py.151: total_norm: 15.887982 [I 2022-10-09 11:46:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 1060/2304 [I 2022-10-09 11:46:12] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:26 [I 2022-10-09 11:46:12] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00293 [I 2022-10-09 11:46:12] ...trap/engines/engine.py.230: loss: 0.20267 [I 2022-10-09 11:46:12] ...optimizers/trijoint.py.151: total_norm: 19.011698 [I 2022-10-09 11:46:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 1070/2304 [I 2022-10-09 11:46:20] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:17 [I 2022-10-09 11:46:20] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00294 [I 2022-10-09 11:46:20] ...trap/engines/engine.py.230: loss: 0.12360 [I 2022-10-09 11:46:20] ...optimizers/trijoint.py.151: total_norm: 9.499047 [I 2022-10-09 11:46:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 1080/2304 [I 2022-10-09 11:46:28] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:14 [I 2022-10-09 11:46:28] ...trap/engines/engine.py.229: process: 0.85436 | load: 0.00302 [I 2022-10-09 11:46:28] ...trap/engines/engine.py.230: loss: 0.16139 [I 2022-10-09 11:46:28] ...optimizers/trijoint.py.151: total_norm: 11.861472 [I 2022-10-09 11:46:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 1090/2304 [I 2022-10-09 11:46:37] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:03 [I 2022-10-09 11:46:37] ...trap/engines/engine.py.229: process: 0.83623 | load: 0.00295 [I 2022-10-09 11:46:37] ...trap/engines/engine.py.230: loss: 0.14189 [I 2022-10-09 11:46:37] ...optimizers/trijoint.py.151: total_norm: 14.080750 [I 2022-10-09 11:46:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 1100/2304 [I 2022-10-09 11:46:45] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:16:54 [I 2022-10-09 11:46:45] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00291 [I 2022-10-09 11:46:45] ...trap/engines/engine.py.230: loss: 0.19948 [I 2022-10-09 11:46:45] ...optimizers/trijoint.py.151: total_norm: 22.909821 [I 2022-10-09 11:46:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 1110/2304 [I 2022-10-09 11:46:54] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:16:41 [I 2022-10-09 11:46:54] ...trap/engines/engine.py.229: process: 0.83123 | load: 0.00292 [I 2022-10-09 11:46:54] ...trap/engines/engine.py.230: loss: 0.11931 [I 2022-10-09 11:46:54] ...optimizers/trijoint.py.151: total_norm: 9.586075 [I 2022-10-09 11:47:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 1120/2304 [I 2022-10-09 11:47:02] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:37 [I 2022-10-09 11:47:02] ...trap/engines/engine.py.229: process: 0.83766 | load: 0.00293 [I 2022-10-09 11:47:02] ...trap/engines/engine.py.230: loss: 0.13513 [I 2022-10-09 11:47:02] ...optimizers/trijoint.py.151: total_norm: 16.980204 [I 2022-10-09 11:47:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 1130/2304 [I 2022-10-09 11:47:11] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:27 [I 2022-10-09 11:47:11] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00293 [I 2022-10-09 11:47:11] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-09 11:47:11] ...optimizers/trijoint.py.151: total_norm: 28.720978 [I 2022-10-09 11:47:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 1140/2304 [I 2022-10-09 11:47:19] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:14 [I 2022-10-09 11:47:19] ...trap/engines/engine.py.229: process: 0.83564 | load: 0.00295 [I 2022-10-09 11:47:19] ...trap/engines/engine.py.230: loss: 0.10298 [I 2022-10-09 11:47:19] ...optimizers/trijoint.py.151: total_norm: 11.893308 [I 2022-10-09 11:47:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 1150/2304 [I 2022-10-09 11:47:27] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:15 [I 2022-10-09 11:47:27] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00300 [I 2022-10-09 11:47:27] ...trap/engines/engine.py.230: loss: 0.14108 [I 2022-10-09 11:47:27] ...optimizers/trijoint.py.151: total_norm: 11.597006 [I 2022-10-09 11:47:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 1160/2304 [I 2022-10-09 11:47:36] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:00 [I 2022-10-09 11:47:36] ...trap/engines/engine.py.229: process: 0.83637 | load: 0.00297 [I 2022-10-09 11:47:36] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-09 11:47:36] ...optimizers/trijoint.py.151: total_norm: 11.646836 [I 2022-10-09 11:47:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 1170/2304 [I 2022-10-09 11:47:44] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:15:55 [I 2022-10-09 11:47:44] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00292 [I 2022-10-09 11:47:44] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-09 11:47:44] ...optimizers/trijoint.py.151: total_norm: 10.151436 [I 2022-10-09 11:47:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 1180/2304 [I 2022-10-09 11:47:53] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:15:47 [I 2022-10-09 11:47:53] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00293 [I 2022-10-09 11:47:53] ...trap/engines/engine.py.230: loss: 0.19305 [I 2022-10-09 11:47:53] ...optimizers/trijoint.py.151: total_norm: 13.395938 [I 2022-10-09 11:48:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 1190/2304 [I 2022-10-09 11:48:01] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:44 [I 2022-10-09 11:48:01] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00293 [I 2022-10-09 11:48:01] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-09 11:48:01] ...optimizers/trijoint.py.151: total_norm: 22.903189 [I 2022-10-09 11:48:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 1200/2304 [I 2022-10-09 11:48:10] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:29 [I 2022-10-09 11:48:10] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00295 [I 2022-10-09 11:48:10] ...trap/engines/engine.py.230: loss: 0.16037 [I 2022-10-09 11:48:10] ...optimizers/trijoint.py.151: total_norm: 41.539745 [I 2022-10-09 11:48:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 1210/2304 [I 2022-10-09 11:48:18] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:21 [I 2022-10-09 11:48:18] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00295 [I 2022-10-09 11:48:18] ...trap/engines/engine.py.230: loss: 0.13861 [I 2022-10-09 11:48:18] ...optimizers/trijoint.py.151: total_norm: 13.576980 [I 2022-10-09 11:48:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 1220/2304 [I 2022-10-09 11:48:26] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:09 [I 2022-10-09 11:48:26] ...trap/engines/engine.py.229: process: 0.83640 | load: 0.00298 [I 2022-10-09 11:48:26] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-09 11:48:26] ...optimizers/trijoint.py.151: total_norm: 10.359359 [I 2022-10-09 11:48:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 1230/2304 [I 2022-10-09 11:48:35] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:07 [I 2022-10-09 11:48:35] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00290 [I 2022-10-09 11:48:35] ...trap/engines/engine.py.230: loss: 0.15535 [I 2022-10-09 11:48:35] ...optimizers/trijoint.py.151: total_norm: 12.310903 [I 2022-10-09 11:48:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 1240/2304 [I 2022-10-09 11:48:43] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:14:57 [I 2022-10-09 11:48:43] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00293 [I 2022-10-09 11:48:43] ...trap/engines/engine.py.230: loss: 0.14695 [I 2022-10-09 11:48:43] ...optimizers/trijoint.py.151: total_norm: 21.304592 [I 2022-10-09 11:48:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 1250/2304 [I 2022-10-09 11:48:52] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:14:46 [I 2022-10-09 11:48:52] ...trap/engines/engine.py.229: process: 0.82260 | load: 0.00291 [I 2022-10-09 11:48:52] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-09 11:48:52] ...optimizers/trijoint.py.151: total_norm: 16.077133 [I 2022-10-09 11:49:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 1260/2304 [I 2022-10-09 11:49:00] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:14:41 [I 2022-10-09 11:49:00] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00293 [I 2022-10-09 11:49:00] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-09 11:49:00] ...optimizers/trijoint.py.151: total_norm: 12.225473 [I 2022-10-09 11:49:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 1270/2304 [I 2022-10-09 11:49:09] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:35 [I 2022-10-09 11:49:09] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00294 [I 2022-10-09 11:49:09] ...trap/engines/engine.py.230: loss: 0.13810 [I 2022-10-09 11:49:09] ...optimizers/trijoint.py.151: total_norm: 9.676857 [I 2022-10-09 11:49:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 1280/2304 [I 2022-10-09 11:49:17] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:14:16 [I 2022-10-09 11:49:17] ...trap/engines/engine.py.229: process: 0.83008 | load: 0.00297 [I 2022-10-09 11:49:17] ...trap/engines/engine.py.230: loss: 0.10996 [I 2022-10-09 11:49:17] ...optimizers/trijoint.py.151: total_norm: 12.975963 [I 2022-10-09 11:49:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 1290/2304 [I 2022-10-09 11:49:26] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:15 [I 2022-10-09 11:49:26] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00296 [I 2022-10-09 11:49:26] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-09 11:49:26] ...optimizers/trijoint.py.151: total_norm: 15.040060 [I 2022-10-09 11:49:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 1300/2304 [I 2022-10-09 11:49:34] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:06 [I 2022-10-09 11:49:34] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00290 [I 2022-10-09 11:49:34] ...trap/engines/engine.py.230: loss: 0.16269 [I 2022-10-09 11:49:34] ...optimizers/trijoint.py.151: total_norm: 16.183069 [I 2022-10-09 11:49:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 1310/2304 [I 2022-10-09 11:49:42] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:13:58 [I 2022-10-09 11:49:42] ...trap/engines/engine.py.229: process: 0.83489 | load: 0.00291 [I 2022-10-09 11:49:42] ...trap/engines/engine.py.230: loss: 0.25090 [I 2022-10-09 11:49:42] ...optimizers/trijoint.py.151: total_norm: 21.228104 [I 2022-10-09 11:49:51] ...trap/engines/engine.py.224: train: epoch 39 | batch 1320/2304 [I 2022-10-09 11:49:51] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:13:46 [I 2022-10-09 11:49:51] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00292 [I 2022-10-09 11:49:51] ...trap/engines/engine.py.230: loss: 0.22319 [I 2022-10-09 11:49:51] ...optimizers/trijoint.py.151: total_norm: 17.898684 [I 2022-10-09 11:49:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 1330/2304 [I 2022-10-09 11:49:59] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:13:38 [I 2022-10-09 11:49:59] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00293 [I 2022-10-09 11:49:59] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-09 11:49:59] ...optimizers/trijoint.py.151: total_norm: 13.209907 [I 2022-10-09 11:50:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 1340/2304 [I 2022-10-09 11:50:08] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:13:26 [I 2022-10-09 11:50:08] ...trap/engines/engine.py.229: process: 0.83290 | load: 0.00295 [I 2022-10-09 11:50:08] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-09 11:50:08] ...optimizers/trijoint.py.151: total_norm: 16.224010 [I 2022-10-09 11:50:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 1350/2304 [I 2022-10-09 11:50:16] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:13:24 [I 2022-10-09 11:50:16] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00296 [I 2022-10-09 11:50:16] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-09 11:50:16] ...optimizers/trijoint.py.151: total_norm: 24.524992 [I 2022-10-09 11:50:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 1360/2304 [I 2022-10-09 11:50:25] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:16 [I 2022-10-09 11:50:25] ...trap/engines/engine.py.229: process: 0.83847 | load: 0.00289 [I 2022-10-09 11:50:25] ...trap/engines/engine.py.230: loss: 0.19470 [I 2022-10-09 11:50:25] ...optimizers/trijoint.py.151: total_norm: 21.944067 [I 2022-10-09 11:50:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 1370/2304 [I 2022-10-09 11:50:33] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:07 [I 2022-10-09 11:50:33] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00293 [I 2022-10-09 11:50:33] ...trap/engines/engine.py.230: loss: 0.19375 [I 2022-10-09 11:50:33] ...optimizers/trijoint.py.151: total_norm: 12.266754 [I 2022-10-09 11:50:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 1380/2304 [I 2022-10-09 11:50:41] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:12:56 [I 2022-10-09 11:50:41] ...trap/engines/engine.py.229: process: 0.82689 | load: 0.00291 [I 2022-10-09 11:50:41] ...trap/engines/engine.py.230: loss: 0.15108 [I 2022-10-09 11:50:41] ...optimizers/trijoint.py.151: total_norm: 14.571066 [I 2022-10-09 11:50:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 1390/2304 [I 2022-10-09 11:50:50] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:12:50 [I 2022-10-09 11:50:50] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00293 [I 2022-10-09 11:50:50] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-09 11:50:50] ...optimizers/trijoint.py.151: total_norm: 17.461519 [I 2022-10-09 11:50:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 1400/2304 [I 2022-10-09 11:50:58] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:12:37 [I 2022-10-09 11:50:58] ...trap/engines/engine.py.229: process: 0.82962 | load: 0.00302 [I 2022-10-09 11:50:58] ...trap/engines/engine.py.230: loss: 0.18795 [I 2022-10-09 11:50:58] ...optimizers/trijoint.py.151: total_norm: 60.864429 [I 2022-10-09 11:51:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 1410/2304 [I 2022-10-09 11:51:07] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:12:32 [I 2022-10-09 11:51:07] ...trap/engines/engine.py.229: process: 0.84851 | load: 0.00291 [I 2022-10-09 11:51:07] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-09 11:51:07] ...optimizers/trijoint.py.151: total_norm: 9.700275 [I 2022-10-09 11:51:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 1420/2304 [I 2022-10-09 11:51:15] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:12:23 [I 2022-10-09 11:51:15] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00291 [I 2022-10-09 11:51:15] ...trap/engines/engine.py.230: loss: 0.14944 [I 2022-10-09 11:51:15] ...optimizers/trijoint.py.151: total_norm: 10.993406 [I 2022-10-09 11:51:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 1430/2304 [I 2022-10-09 11:51:24] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:16 [I 2022-10-09 11:51:24] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00291 [I 2022-10-09 11:51:24] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-09 11:51:24] ...optimizers/trijoint.py.151: total_norm: 33.664375 [I 2022-10-09 11:51:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 1440/2304 [I 2022-10-09 11:51:32] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:04 [I 2022-10-09 11:51:32] ...trap/engines/engine.py.229: process: 0.83681 | load: 0.00298 [I 2022-10-09 11:51:32] ...trap/engines/engine.py.230: loss: 0.19249 [I 2022-10-09 11:51:32] ...optimizers/trijoint.py.151: total_norm: 24.046585 [I 2022-10-09 11:51:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 1450/2304 [I 2022-10-09 11:51:40] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:11:57 [I 2022-10-09 11:51:40] ...trap/engines/engine.py.229: process: 0.83230 | load: 0.00294 [I 2022-10-09 11:51:40] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-09 11:51:40] ...optimizers/trijoint.py.151: total_norm: 14.408798 [I 2022-10-09 11:51:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 1460/2304 [I 2022-10-09 11:51:49] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:11:46 [I 2022-10-09 11:51:49] ...trap/engines/engine.py.229: process: 0.83452 | load: 0.00292 [I 2022-10-09 11:51:49] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-09 11:51:49] ...optimizers/trijoint.py.151: total_norm: 10.676009 [I 2022-10-09 11:51:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 1470/2304 [I 2022-10-09 11:51:57] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:11:40 [I 2022-10-09 11:51:57] ...trap/engines/engine.py.229: process: 0.83896 | load: 0.00291 [I 2022-10-09 11:51:57] ...trap/engines/engine.py.230: loss: 0.20039 [I 2022-10-09 11:51:57] ...optimizers/trijoint.py.151: total_norm: 18.124933 [I 2022-10-09 11:52:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 1480/2304 [I 2022-10-09 11:52:06] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:32 [I 2022-10-09 11:52:06] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00301 [I 2022-10-09 11:52:06] ...trap/engines/engine.py.230: loss: 0.09271 [I 2022-10-09 11:52:06] ...optimizers/trijoint.py.151: total_norm: 11.220278 [I 2022-10-09 11:52:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 1490/2304 [I 2022-10-09 11:52:14] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:27 [I 2022-10-09 11:52:14] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00289 [I 2022-10-09 11:52:14] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-09 11:52:14] ...optimizers/trijoint.py.151: total_norm: 16.233252 [I 2022-10-09 11:52:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 1500/2304 [I 2022-10-09 11:52:22] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:14 [I 2022-10-09 11:52:22] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00292 [I 2022-10-09 11:52:22] ...trap/engines/engine.py.230: loss: 0.12166 [I 2022-10-09 11:52:22] ...optimizers/trijoint.py.151: total_norm: 14.402830 [I 2022-10-09 11:52:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 1510/2304 [I 2022-10-09 11:52:31] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:05 [I 2022-10-09 11:52:31] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00290 [I 2022-10-09 11:52:31] ...trap/engines/engine.py.230: loss: 0.25962 [I 2022-10-09 11:52:31] ...optimizers/trijoint.py.151: total_norm: 20.651402 [I 2022-10-09 11:52:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 1520/2304 [I 2022-10-09 11:52:39] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:10:59 [I 2022-10-09 11:52:39] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00292 [I 2022-10-09 11:52:39] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-09 11:52:39] ...optimizers/trijoint.py.151: total_norm: 12.640438 [I 2022-10-09 11:52:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 1530/2304 [I 2022-10-09 11:52:48] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:10:51 [I 2022-10-09 11:52:48] ...trap/engines/engine.py.229: process: 0.84399 | load: 0.00298 [I 2022-10-09 11:52:48] ...trap/engines/engine.py.230: loss: 0.18548 [I 2022-10-09 11:52:48] ...optimizers/trijoint.py.151: total_norm: 30.716509 [I 2022-10-09 11:52:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 1540/2304 [I 2022-10-09 11:52:56] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:10:43 [I 2022-10-09 11:52:56] ...trap/engines/engine.py.229: process: 0.84309 | load: 0.00298 [I 2022-10-09 11:52:56] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-09 11:52:56] ...optimizers/trijoint.py.151: total_norm: 17.284712 [I 2022-10-09 11:53:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 1550/2304 [I 2022-10-09 11:53:05] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:36 [I 2022-10-09 11:53:05] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00299 [I 2022-10-09 11:53:05] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-09 11:53:05] ...optimizers/trijoint.py.151: total_norm: 26.113625 [I 2022-10-09 11:53:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 1560/2304 [I 2022-10-09 11:53:13] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:10:28 [I 2022-10-09 11:53:13] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00293 [I 2022-10-09 11:53:13] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-09 11:53:13] ...optimizers/trijoint.py.151: total_norm: 14.981776 [I 2022-10-09 11:53:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 1570/2304 [I 2022-10-09 11:53:21] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:19 [I 2022-10-09 11:53:21] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00295 [I 2022-10-09 11:53:21] ...trap/engines/engine.py.230: loss: 0.11038 [I 2022-10-09 11:53:21] ...optimizers/trijoint.py.151: total_norm: 13.178129 [I 2022-10-09 11:53:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 1580/2304 [I 2022-10-09 11:53:30] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:08 [I 2022-10-09 11:53:30] ...trap/engines/engine.py.229: process: 0.84021 | load: 0.00295 [I 2022-10-09 11:53:30] ...trap/engines/engine.py.230: loss: 0.18056 [I 2022-10-09 11:53:30] ...optimizers/trijoint.py.151: total_norm: 20.660511 [I 2022-10-09 11:53:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 1590/2304 [I 2022-10-09 11:53:38] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:02 [I 2022-10-09 11:53:38] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00288 [I 2022-10-09 11:53:38] ...trap/engines/engine.py.230: loss: 0.18700 [I 2022-10-09 11:53:38] ...optimizers/trijoint.py.151: total_norm: 9.259890 [I 2022-10-09 11:53:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 1600/2304 [I 2022-10-09 11:53:47] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:09:48 [I 2022-10-09 11:53:47] ...trap/engines/engine.py.229: process: 0.82884 | load: 0.00292 [I 2022-10-09 11:53:47] ...trap/engines/engine.py.230: loss: 0.14677 [I 2022-10-09 11:53:47] ...optimizers/trijoint.py.151: total_norm: 19.575680 [I 2022-10-09 11:53:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 1610/2304 [I 2022-10-09 11:53:55] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:09:42 [I 2022-10-09 11:53:55] ...trap/engines/engine.py.229: process: 0.83254 | load: 0.00291 [I 2022-10-09 11:53:55] ...trap/engines/engine.py.230: loss: 0.17078 [I 2022-10-09 11:53:55] ...optimizers/trijoint.py.151: total_norm: 32.068256 [I 2022-10-09 11:54:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 1620/2304 [I 2022-10-09 11:54:03] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:35 [I 2022-10-09 11:54:03] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00295 [I 2022-10-09 11:54:03] ...trap/engines/engine.py.230: loss: 0.13064 [I 2022-10-09 11:54:03] ...optimizers/trijoint.py.151: total_norm: 11.367495 [I 2022-10-09 11:54:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 1630/2304 [I 2022-10-09 11:54:12] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:09:27 [I 2022-10-09 11:54:12] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00289 [I 2022-10-09 11:54:12] ...trap/engines/engine.py.230: loss: 0.20640 [I 2022-10-09 11:54:12] ...optimizers/trijoint.py.151: total_norm: 28.200975 [I 2022-10-09 11:54:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 1640/2304 [I 2022-10-09 11:54:20] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:15 [I 2022-10-09 11:54:20] ...trap/engines/engine.py.229: process: 0.82671 | load: 0.00292 [I 2022-10-09 11:54:20] ...trap/engines/engine.py.230: loss: 0.16925 [I 2022-10-09 11:54:20] ...optimizers/trijoint.py.151: total_norm: 10.523102 [I 2022-10-09 11:54:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1650/2304 [I 2022-10-09 11:54:29] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:10 [I 2022-10-09 11:54:29] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00298 [I 2022-10-09 11:54:29] ...trap/engines/engine.py.230: loss: 0.13605 [I 2022-10-09 11:54:29] ...optimizers/trijoint.py.151: total_norm: 37.446690 [I 2022-10-09 11:54:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 1660/2304 [I 2022-10-09 11:54:37] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:00 [I 2022-10-09 11:54:37] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00289 [I 2022-10-09 11:54:37] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-09 11:54:37] ...optimizers/trijoint.py.151: total_norm: 34.017860 [I 2022-10-09 11:54:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 1670/2304 [I 2022-10-09 11:54:45] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:08:51 [I 2022-10-09 11:54:45] ...trap/engines/engine.py.229: process: 0.84732 | load: 0.00291 [I 2022-10-09 11:54:45] ...trap/engines/engine.py.230: loss: 0.10124 [I 2022-10-09 11:54:45] ...optimizers/trijoint.py.151: total_norm: 9.658503 [I 2022-10-09 11:54:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 1680/2304 [I 2022-10-09 11:54:54] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:08:45 [I 2022-10-09 11:54:54] ...trap/engines/engine.py.229: process: 0.84230 | load: 0.00291 [I 2022-10-09 11:54:54] ...trap/engines/engine.py.230: loss: 0.17506 [I 2022-10-09 11:54:54] ...optimizers/trijoint.py.151: total_norm: 21.001347 [I 2022-10-09 11:55:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 1690/2304 [I 2022-10-09 11:55:20] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:27:24 [I 2022-10-09 11:55:20] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00293 [I 2022-10-09 11:55:20] ...trap/engines/engine.py.230: loss: 0.14068 [I 2022-10-09 11:55:20] ...optimizers/trijoint.py.151: total_norm: 11.435685 [I 2022-10-09 11:55:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1700/2304 [I 2022-10-09 11:55:29] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:10:25 [I 2022-10-09 11:55:29] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00293 [I 2022-10-09 11:55:29] ...trap/engines/engine.py.230: loss: 0.10205 [I 2022-10-09 11:55:29] ...optimizers/trijoint.py.151: total_norm: 16.863997 [I 2022-10-09 11:55:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 1710/2304 [I 2022-10-09 11:55:37] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:30 [I 2022-10-09 11:55:37] ...trap/engines/engine.py.229: process: 0.82521 | load: 0.00294 [I 2022-10-09 11:55:37] ...trap/engines/engine.py.230: loss: 0.14266 [I 2022-10-09 11:55:37] ...optimizers/trijoint.py.151: total_norm: 10.394991 [I 2022-10-09 11:55:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 1720/2304 [I 2022-10-09 11:55:45] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:12 [I 2022-10-09 11:55:45] ...trap/engines/engine.py.229: process: 0.83209 | load: 0.00289 [I 2022-10-09 11:55:45] ...trap/engines/engine.py.230: loss: 0.20435 [I 2022-10-09 11:55:45] ...optimizers/trijoint.py.151: total_norm: 35.000427 [I 2022-10-09 11:55:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 1730/2304 [I 2022-10-09 11:55:54] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:07:59 [I 2022-10-09 11:55:54] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00291 [I 2022-10-09 11:55:54] ...trap/engines/engine.py.230: loss: 0.21626 [I 2022-10-09 11:55:54] ...optimizers/trijoint.py.151: total_norm: 26.097441 [I 2022-10-09 11:56:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 1740/2304 [I 2022-10-09 11:56:02] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:07:54 [I 2022-10-09 11:56:02] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00290 [I 2022-10-09 11:56:02] ...trap/engines/engine.py.230: loss: 0.17730 [I 2022-10-09 11:56:02] ...optimizers/trijoint.py.151: total_norm: 30.664579 [I 2022-10-09 11:56:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 1750/2304 [I 2022-10-09 11:56:11] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:07:50 [I 2022-10-09 11:56:11] ...trap/engines/engine.py.229: process: 0.84947 | load: 0.00295 [I 2022-10-09 11:56:11] ...trap/engines/engine.py.230: loss: 0.15712 [I 2022-10-09 11:56:11] ...optimizers/trijoint.py.151: total_norm: 17.954729 [I 2022-10-09 11:56:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 1760/2304 [I 2022-10-09 11:56:19] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:40 [I 2022-10-09 11:56:19] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00300 [I 2022-10-09 11:56:19] ...trap/engines/engine.py.230: loss: 0.12940 [I 2022-10-09 11:56:19] ...optimizers/trijoint.py.151: total_norm: 30.189817 [I 2022-10-09 11:56:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 1770/2304 [I 2022-10-09 11:56:28] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:25 [I 2022-10-09 11:56:28] ...trap/engines/engine.py.229: process: 0.82859 | load: 0.00291 [I 2022-10-09 11:56:28] ...trap/engines/engine.py.230: loss: 0.12996 [I 2022-10-09 11:56:28] ...optimizers/trijoint.py.151: total_norm: 24.596914 [I 2022-10-09 11:56:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 1780/2304 [I 2022-10-09 11:56:36] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:20 [I 2022-10-09 11:56:36] ...trap/engines/engine.py.229: process: 0.83499 | load: 0.00292 [I 2022-10-09 11:56:36] ...trap/engines/engine.py.230: loss: 0.21744 [I 2022-10-09 11:56:36] ...optimizers/trijoint.py.151: total_norm: 26.035101 [I 2022-10-09 11:56:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 1790/2304 [I 2022-10-09 11:56:44] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:10 [I 2022-10-09 11:56:44] ...trap/engines/engine.py.229: process: 0.83336 | load: 0.00293 [I 2022-10-09 11:56:44] ...trap/engines/engine.py.230: loss: 0.21097 [I 2022-10-09 11:56:44] ...optimizers/trijoint.py.151: total_norm: 15.603344 [I 2022-10-09 11:56:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 1800/2304 [I 2022-10-09 11:56:53] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:05 [I 2022-10-09 11:56:53] ...trap/engines/engine.py.229: process: 0.84462 | load: 0.00305 [I 2022-10-09 11:56:53] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-09 11:56:53] ...optimizers/trijoint.py.151: total_norm: 26.293961 [I 2022-10-09 11:57:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 1810/2304 [I 2022-10-09 11:57:01] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:06:56 [I 2022-10-09 11:57:01] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00291 [I 2022-10-09 11:57:01] ...trap/engines/engine.py.230: loss: 0.15352 [I 2022-10-09 11:57:01] ...optimizers/trijoint.py.151: total_norm: 12.683301 [I 2022-10-09 11:57:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 1820/2304 [I 2022-10-09 11:57:10] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:50 [I 2022-10-09 11:57:10] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00293 [I 2022-10-09 11:57:10] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-09 11:57:10] ...optimizers/trijoint.py.151: total_norm: 10.975846 [I 2022-10-09 11:57:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 1830/2304 [I 2022-10-09 11:57:18] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:39 [I 2022-10-09 11:57:18] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00292 [I 2022-10-09 11:57:18] ...trap/engines/engine.py.230: loss: 0.25284 [I 2022-10-09 11:57:18] ...optimizers/trijoint.py.151: total_norm: 28.057720 [I 2022-10-09 11:57:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 1840/2304 [I 2022-10-09 11:57:27] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:30 [I 2022-10-09 11:57:27] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00302 [I 2022-10-09 11:57:27] ...trap/engines/engine.py.230: loss: 0.17493 [I 2022-10-09 11:57:27] ...optimizers/trijoint.py.151: total_norm: 24.971306 [I 2022-10-09 11:57:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 1850/2304 [I 2022-10-09 11:57:35] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:21 [I 2022-10-09 11:57:35] ...trap/engines/engine.py.229: process: 0.82581 | load: 0.00295 [I 2022-10-09 11:57:35] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-09 11:57:35] ...optimizers/trijoint.py.151: total_norm: 13.394374 [I 2022-10-09 11:57:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 1860/2304 [I 2022-10-09 11:57:44] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:14 [I 2022-10-09 11:57:44] ...trap/engines/engine.py.229: process: 0.84021 | load: 0.00291 [I 2022-10-09 11:57:44] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-09 11:57:44] ...optimizers/trijoint.py.151: total_norm: 10.679442 [I 2022-10-09 11:57:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 1870/2304 [I 2022-10-09 11:57:52] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:04 [I 2022-10-09 11:57:52] ...trap/engines/engine.py.229: process: 0.82954 | load: 0.00294 [I 2022-10-09 11:57:52] ...trap/engines/engine.py.230: loss: 0.21531 [I 2022-10-09 11:57:52] ...optimizers/trijoint.py.151: total_norm: 14.942047 [I 2022-10-09 11:58:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 1880/2304 [I 2022-10-09 11:58:00] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:05:56 [I 2022-10-09 11:58:00] ...trap/engines/engine.py.229: process: 0.84574 | load: 0.00296 [I 2022-10-09 11:58:00] ...trap/engines/engine.py.230: loss: 0.16745 [I 2022-10-09 11:58:00] ...optimizers/trijoint.py.151: total_norm: 17.235411 [I 2022-10-09 11:58:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 1890/2304 [I 2022-10-09 11:58:09] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:05:48 [I 2022-10-09 11:58:09] ...trap/engines/engine.py.229: process: 0.83922 | load: 0.00291 [I 2022-10-09 11:58:09] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-09 11:58:09] ...optimizers/trijoint.py.151: total_norm: 15.948849 [I 2022-10-09 11:58:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 1900/2304 [I 2022-10-09 11:58:17] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:05:41 [I 2022-10-09 11:58:17] ...trap/engines/engine.py.229: process: 0.84512 | load: 0.00293 [I 2022-10-09 11:58:17] ...trap/engines/engine.py.230: loss: 0.10946 [I 2022-10-09 11:58:17] ...optimizers/trijoint.py.151: total_norm: 19.084148 [I 2022-10-09 11:58:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 1910/2304 [I 2022-10-09 11:58:26] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:32 [I 2022-10-09 11:58:26] ...trap/engines/engine.py.229: process: 0.83986 | load: 0.00293 [I 2022-10-09 11:58:26] ...trap/engines/engine.py.230: loss: 0.15256 [I 2022-10-09 11:58:26] ...optimizers/trijoint.py.151: total_norm: 14.877110 [I 2022-10-09 11:58:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 1920/2304 [I 2022-10-09 11:58:34] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:24 [I 2022-10-09 11:58:34] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00295 [I 2022-10-09 11:58:34] ...trap/engines/engine.py.230: loss: 0.29534 [I 2022-10-09 11:58:34] ...optimizers/trijoint.py.151: total_norm: 17.176144 [I 2022-10-09 11:58:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 1930/2304 [I 2022-10-09 11:58:43] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:15 [I 2022-10-09 11:58:43] ...trap/engines/engine.py.229: process: 0.84908 | load: 0.00291 [I 2022-10-09 11:58:43] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-09 11:58:43] ...optimizers/trijoint.py.151: total_norm: 25.552542 [I 2022-10-09 11:58:51] ...trap/engines/engine.py.224: train: epoch 39 | batch 1940/2304 [I 2022-10-09 11:58:51] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:05 [I 2022-10-09 11:58:51] ...trap/engines/engine.py.229: process: 0.82496 | load: 0.00291 [I 2022-10-09 11:58:51] ...trap/engines/engine.py.230: loss: 0.11918 [I 2022-10-09 11:58:51] ...optimizers/trijoint.py.151: total_norm: 8.549154 [I 2022-10-09 11:58:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 1950/2304 [I 2022-10-09 11:58:59] ...trap/engines/engine.py.228: elapsed: 0:27:45 | left: 0:04:58 [I 2022-10-09 11:58:59] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00291 [I 2022-10-09 11:58:59] ...trap/engines/engine.py.230: loss: 0.19347 [I 2022-10-09 11:58:59] ...optimizers/trijoint.py.151: total_norm: 15.111513 [I 2022-10-09 11:59:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 1960/2304 [I 2022-10-09 11:59:08] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:04:49 [I 2022-10-09 11:59:08] ...trap/engines/engine.py.229: process: 0.83804 | load: 0.00289 [I 2022-10-09 11:59:08] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-09 11:59:08] ...optimizers/trijoint.py.151: total_norm: 18.844790 [I 2022-10-09 11:59:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 1970/2304 [I 2022-10-09 11:59:16] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:41 [I 2022-10-09 11:59:16] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00292 [I 2022-10-09 11:59:16] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-09 11:59:16] ...optimizers/trijoint.py.151: total_norm: 15.269747 [I 2022-10-09 11:59:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 1980/2304 [I 2022-10-09 11:59:25] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:32 [I 2022-10-09 11:59:25] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00291 [I 2022-10-09 11:59:25] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-09 11:59:25] ...optimizers/trijoint.py.151: total_norm: 11.443706 [I 2022-10-09 11:59:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 1990/2304 [I 2022-10-09 11:59:33] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:24 [I 2022-10-09 11:59:33] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00291 [I 2022-10-09 11:59:33] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-09 11:59:33] ...optimizers/trijoint.py.151: total_norm: 17.501776 [I 2022-10-09 11:59:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 2000/2304 [I 2022-10-09 11:59:42] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:15 [I 2022-10-09 11:59:42] ...trap/engines/engine.py.229: process: 0.82674 | load: 0.00290 [I 2022-10-09 11:59:42] ...trap/engines/engine.py.230: loss: 0.13970 [I 2022-10-09 11:59:42] ...optimizers/trijoint.py.151: total_norm: 28.146233 [I 2022-10-09 11:59:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 2010/2304 [I 2022-10-09 11:59:50] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:06 [I 2022-10-09 11:59:50] ...trap/engines/engine.py.229: process: 0.83751 | load: 0.00299 [I 2022-10-09 11:59:50] ...trap/engines/engine.py.230: loss: 0.16740 [I 2022-10-09 11:59:50] ...optimizers/trijoint.py.151: total_norm: 12.992521 [I 2022-10-09 11:59:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 2020/2304 [I 2022-10-09 11:59:58] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:03:58 [I 2022-10-09 11:59:58] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00292 [I 2022-10-09 11:59:58] ...trap/engines/engine.py.230: loss: 0.14553 [I 2022-10-09 11:59:58] ...optimizers/trijoint.py.151: total_norm: 16.139864 [I 2022-10-09 12:00:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 2030/2304 [I 2022-10-09 12:00:07] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:03:50 [I 2022-10-09 12:00:07] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00294 [I 2022-10-09 12:00:07] ...trap/engines/engine.py.230: loss: 0.15476 [I 2022-10-09 12:00:07] ...optimizers/trijoint.py.151: total_norm: 21.809353 [I 2022-10-09 12:00:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 2040/2304 [I 2022-10-09 12:00:15] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:42 [I 2022-10-09 12:00:15] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00292 [I 2022-10-09 12:00:15] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-09 12:00:15] ...optimizers/trijoint.py.151: total_norm: 10.767951 [I 2022-10-09 12:00:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 2050/2304 [I 2022-10-09 12:00:24] ...trap/engines/engine.py.228: elapsed: 0:29:10 | left: 0:03:32 [I 2022-10-09 12:00:24] ...trap/engines/engine.py.229: process: 0.82350 | load: 0.00290 [I 2022-10-09 12:00:24] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-09 12:00:24] ...optimizers/trijoint.py.151: total_norm: 19.701538 [I 2022-10-09 12:00:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 2060/2304 [I 2022-10-09 12:00:32] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:26 [I 2022-10-09 12:00:32] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00293 [I 2022-10-09 12:00:32] ...trap/engines/engine.py.230: loss: 0.12019 [I 2022-10-09 12:00:32] ...optimizers/trijoint.py.151: total_norm: 13.113142 [I 2022-10-09 12:00:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 2070/2304 [I 2022-10-09 12:00:41] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:15 [I 2022-10-09 12:00:41] ...trap/engines/engine.py.229: process: 0.82413 | load: 0.00292 [I 2022-10-09 12:00:41] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-09 12:00:41] ...optimizers/trijoint.py.151: total_norm: 16.105011 [I 2022-10-09 12:00:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 2080/2304 [I 2022-10-09 12:00:49] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:08 [I 2022-10-09 12:00:49] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00297 [I 2022-10-09 12:00:49] ...trap/engines/engine.py.230: loss: 0.18319 [I 2022-10-09 12:00:49] ...optimizers/trijoint.py.151: total_norm: 15.310618 [I 2022-10-09 12:00:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 2090/2304 [I 2022-10-09 12:00:57] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:00 [I 2022-10-09 12:00:57] ...trap/engines/engine.py.229: process: 0.85151 | load: 0.00290 [I 2022-10-09 12:00:57] ...trap/engines/engine.py.230: loss: 0.10156 [I 2022-10-09 12:00:57] ...optimizers/trijoint.py.151: total_norm: 11.416316 [I 2022-10-09 12:01:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 2100/2304 [I 2022-10-09 12:01:06] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:02:51 [I 2022-10-09 12:01:06] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00297 [I 2022-10-09 12:01:06] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-09 12:01:06] ...optimizers/trijoint.py.151: total_norm: 15.070839 [I 2022-10-09 12:01:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 2110/2304 [I 2022-10-09 12:01:14] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:02:44 [I 2022-10-09 12:01:14] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00292 [I 2022-10-09 12:01:14] ...trap/engines/engine.py.230: loss: 0.17413 [I 2022-10-09 12:01:14] ...optimizers/trijoint.py.151: total_norm: 14.649730 [I 2022-10-09 12:01:23] ...trap/engines/engine.py.224: train: epoch 39 | batch 2120/2304 [I 2022-10-09 12:01:23] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:35 [I 2022-10-09 12:01:23] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00291 [I 2022-10-09 12:01:23] ...trap/engines/engine.py.230: loss: 0.15444 [I 2022-10-09 12:01:23] ...optimizers/trijoint.py.151: total_norm: 64.220955 [I 2022-10-09 12:01:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 2130/2304 [I 2022-10-09 12:01:31] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:26 [I 2022-10-09 12:01:31] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00294 [I 2022-10-09 12:01:31] ...trap/engines/engine.py.230: loss: 0.12171 [I 2022-10-09 12:01:31] ...optimizers/trijoint.py.151: total_norm: 30.513565 [I 2022-10-09 12:01:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 2140/2304 [I 2022-10-09 12:01:40] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:17 [I 2022-10-09 12:01:40] ...trap/engines/engine.py.229: process: 0.83225 | load: 0.00302 [I 2022-10-09 12:01:40] ...trap/engines/engine.py.230: loss: 0.16346 [I 2022-10-09 12:01:40] ...optimizers/trijoint.py.151: total_norm: 14.037135 [I 2022-10-09 12:01:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 2150/2304 [I 2022-10-09 12:01:48] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:10 [I 2022-10-09 12:01:48] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00293 [I 2022-10-09 12:01:48] ...trap/engines/engine.py.230: loss: 0.14017 [I 2022-10-09 12:01:48] ...optimizers/trijoint.py.151: total_norm: 19.557503 [I 2022-10-09 12:01:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 2160/2304 [I 2022-10-09 12:01:57] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:01 [I 2022-10-09 12:01:57] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00293 [I 2022-10-09 12:01:57] ...trap/engines/engine.py.230: loss: 0.11099 [I 2022-10-09 12:01:57] ...optimizers/trijoint.py.151: total_norm: 42.627647 [I 2022-10-09 12:02:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 2170/2304 [I 2022-10-09 12:02:05] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:01:53 [I 2022-10-09 12:02:05] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00293 [I 2022-10-09 12:02:05] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-09 12:02:05] ...optimizers/trijoint.py.151: total_norm: 5.376648 [I 2022-10-09 12:02:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 2180/2304 [I 2022-10-09 12:02:14] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:01:44 [I 2022-10-09 12:02:14] ...trap/engines/engine.py.229: process: 0.83824 | load: 0.00297 [I 2022-10-09 12:02:14] ...trap/engines/engine.py.230: loss: 0.17349 [I 2022-10-09 12:02:14] ...optimizers/trijoint.py.151: total_norm: 11.917880 [I 2022-10-09 12:02:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 2190/2304 [I 2022-10-09 12:02:22] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:36 [I 2022-10-09 12:02:22] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00298 [I 2022-10-09 12:02:22] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-09 12:02:22] ...optimizers/trijoint.py.151: total_norm: 13.162305 [I 2022-10-09 12:02:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 2200/2304 [I 2022-10-09 12:02:30] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:27 [I 2022-10-09 12:02:30] ...trap/engines/engine.py.229: process: 0.85345 | load: 0.00296 [I 2022-10-09 12:02:30] ...trap/engines/engine.py.230: loss: 0.17080 [I 2022-10-09 12:02:30] ...optimizers/trijoint.py.151: total_norm: 22.575708 [I 2022-10-09 12:02:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 2210/2304 [I 2022-10-09 12:02:39] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:18 [I 2022-10-09 12:02:39] ...trap/engines/engine.py.229: process: 0.83172 | load: 0.00295 [I 2022-10-09 12:02:39] ...trap/engines/engine.py.230: loss: 0.21087 [I 2022-10-09 12:02:39] ...optimizers/trijoint.py.151: total_norm: 18.856667 [I 2022-10-09 12:02:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 2220/2304 [I 2022-10-09 12:02:47] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:10 [I 2022-10-09 12:02:47] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00300 [I 2022-10-09 12:02:47] ...trap/engines/engine.py.230: loss: 0.18483 [I 2022-10-09 12:02:47] ...optimizers/trijoint.py.151: total_norm: 12.124692 [I 2022-10-09 12:02:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 2230/2304 [I 2022-10-09 12:02:56] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:02 [I 2022-10-09 12:02:56] ...trap/engines/engine.py.229: process: 0.83642 | load: 0.00295 [I 2022-10-09 12:02:56] ...trap/engines/engine.py.230: loss: 0.22181 [I 2022-10-09 12:02:56] ...optimizers/trijoint.py.151: total_norm: 37.719753 [I 2022-10-09 12:03:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 2240/2304 [I 2022-10-09 12:03:04] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:00:53 [I 2022-10-09 12:03:04] ...trap/engines/engine.py.229: process: 0.84080 | load: 0.00293 [I 2022-10-09 12:03:04] ...trap/engines/engine.py.230: loss: 0.18278 [I 2022-10-09 12:03:04] ...optimizers/trijoint.py.151: total_norm: 23.242317 [I 2022-10-09 12:03:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 2250/2304 [I 2022-10-09 12:03:13] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:00:45 [I 2022-10-09 12:03:13] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00298 [I 2022-10-09 12:03:13] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-09 12:03:13] ...optimizers/trijoint.py.151: total_norm: 21.410501 [I 2022-10-09 12:03:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 2260/2304 [I 2022-10-09 12:03:21] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:36 [I 2022-10-09 12:03:21] ...trap/engines/engine.py.229: process: 0.84539 | load: 0.00296 [I 2022-10-09 12:03:21] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-09 12:03:21] ...optimizers/trijoint.py.151: total_norm: 10.900527 [I 2022-10-09 12:03:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 2270/2304 [I 2022-10-09 12:03:29] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:28 [I 2022-10-09 12:03:29] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00299 [I 2022-10-09 12:03:30] ...trap/engines/engine.py.230: loss: 0.19410 [I 2022-10-09 12:03:30] ...optimizers/trijoint.py.151: total_norm: 24.013168 [I 2022-10-09 12:03:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 2280/2304 [I 2022-10-09 12:03:38] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:20 [I 2022-10-09 12:03:38] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00297 [I 2022-10-09 12:03:38] ...trap/engines/engine.py.230: loss: 0.11423 [I 2022-10-09 12:03:38] ...optimizers/trijoint.py.151: total_norm: 18.026257 [I 2022-10-09 12:03:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 2290/2304 [I 2022-10-09 12:03:46] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:11 [I 2022-10-09 12:03:46] ...trap/engines/engine.py.229: process: 0.82822 | load: 0.00295 [I 2022-10-09 12:03:46] ...trap/engines/engine.py.230: loss: 0.17548 [I 2022-10-09 12:03:46] ...optimizers/trijoint.py.151: total_norm: 11.787363 [I 2022-10-09 12:03:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 2300/2304 [I 2022-10-09 12:03:55] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:03 [I 2022-10-09 12:03:55] ...trap/engines/engine.py.229: process: 0.82879 | load: 0.00016 [I 2022-10-09 12:03:55] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-09 12:03:55] ...optimizers/trijoint.py.151: total_norm: 25.257515 [I 2022-10-09 12:03:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 2304/2304 [I 2022-10-09 12:03:58] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:00 [I 2022-10-09 12:03:58] ...trap/engines/engine.py.229: process: 0.82123 | load: 0.00017 [I 2022-10-09 12:03:58] ...trap/engines/engine.py.230: loss: 0.15301 [I 2022-10-09 12:03:58] ...optimizers/trijoint.py.151: total_norm: 16.033838 [S 2022-10-09 12:03:59] ...trap/engines/engine.py.129: train_epoch.epoch: 39 [S 2022-10-09 12:03:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.16010217936527704 [I 2022-10-09 12:03:59] ...s/criterions/triplet.py.84: margin = 0.24500000000000005 [I 2022-10-09 12:04:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 39 [I 2022-10-09 12:04:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 12:04:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 12:04:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 12:04:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 12:04:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 12:04:03] ...trap/engines/engine.py.321: eval: epoch 39 | batch 0/510 [I 2022-10-09 12:04:03] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:23 [I 2022-10-09 12:04:03] ...trap/engines/engine.py.326: process: 3.22215 | load: 2.71027 [I 2022-10-09 12:04:08] ...trap/engines/engine.py.321: eval: epoch 39 | batch 10/510 [I 2022-10-09 12:04:08] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:08 [I 2022-10-09 12:04:08] ...trap/engines/engine.py.326: process: 0.41524 | load: 0.00017 [I 2022-10-09 12:04:12] ...trap/engines/engine.py.321: eval: epoch 39 | batch 20/510 [I 2022-10-09 12:04:12] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:43 [I 2022-10-09 12:04:12] ...trap/engines/engine.py.326: process: 0.41676 | load: 0.00018 [I 2022-10-09 12:04:16] ...trap/engines/engine.py.321: eval: epoch 39 | batch 30/510 [I 2022-10-09 12:04:16] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:03:25 [I 2022-10-09 12:04:16] ...trap/engines/engine.py.326: process: 0.43049 | load: 0.00018 [I 2022-10-09 12:04:20] ...trap/engines/engine.py.321: eval: epoch 39 | batch 40/510 [I 2022-10-09 12:04:20] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:21 [I 2022-10-09 12:04:20] ...trap/engines/engine.py.326: process: 0.43387 | load: 0.00018 [I 2022-10-09 12:04:25] ...trap/engines/engine.py.321: eval: epoch 39 | batch 50/510 [I 2022-10-09 12:04:25] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:16 [I 2022-10-09 12:04:25] ...trap/engines/engine.py.326: process: 0.41605 | load: 0.00017 [I 2022-10-09 12:04:29] ...trap/engines/engine.py.321: eval: epoch 39 | batch 60/510 [I 2022-10-09 12:04:29] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:21 [I 2022-10-09 12:04:29] ...trap/engines/engine.py.326: process: 0.44805 | load: 0.00017 [I 2022-10-09 12:04:34] ...trap/engines/engine.py.321: eval: epoch 39 | batch 70/510 [I 2022-10-09 12:04:34] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:21 [I 2022-10-09 12:04:34] ...trap/engines/engine.py.326: process: 0.44015 | load: 0.00018 [I 2022-10-09 12:04:39] ...trap/engines/engine.py.321: eval: epoch 39 | batch 80/510 [I 2022-10-09 12:04:39] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:35 [I 2022-10-09 12:04:39] ...trap/engines/engine.py.326: process: 0.49953 | load: 0.00017 [I 2022-10-09 12:04:43] ...trap/engines/engine.py.321: eval: epoch 39 | batch 90/510 [I 2022-10-09 12:04:43] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:03 [I 2022-10-09 12:04:43] ...trap/engines/engine.py.326: process: 0.42759 | load: 0.00017 [I 2022-10-09 12:04:47] ...trap/engines/engine.py.321: eval: epoch 39 | batch 100/510 [I 2022-10-09 12:04:47] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:02:46 [I 2022-10-09 12:04:47] ...trap/engines/engine.py.326: process: 0.29964 | load: 0.00020 [I 2022-10-09 12:04:50] ...trap/engines/engine.py.321: eval: epoch 39 | batch 110/510 [I 2022-10-09 12:04:50] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:02:09 [I 2022-10-09 12:04:50] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00018 [I 2022-10-09 12:04:54] ...trap/engines/engine.py.321: eval: epoch 39 | batch 120/510 [I 2022-10-09 12:04:54] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:02 [I 2022-10-09 12:04:54] ...trap/engines/engine.py.326: process: 0.30854 | load: 0.00019 [I 2022-10-09 12:04:57] ...trap/engines/engine.py.321: eval: epoch 39 | batch 130/510 [I 2022-10-09 12:04:57] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:59 [I 2022-10-09 12:04:57] ...trap/engines/engine.py.326: process: 0.31639 | load: 0.00018 [I 2022-10-09 12:05:00] ...trap/engines/engine.py.321: eval: epoch 39 | batch 140/510 [I 2022-10-09 12:05:00] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:57 [I 2022-10-09 12:05:00] ...trap/engines/engine.py.326: process: 0.31658 | load: 0.00019 [I 2022-10-09 12:05:03] ...trap/engines/engine.py.321: eval: epoch 39 | batch 150/510 [I 2022-10-09 12:05:03] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:54 [I 2022-10-09 12:05:03] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00018 [I 2022-10-09 12:05:06] ...trap/engines/engine.py.321: eval: epoch 39 | batch 160/510 [I 2022-10-09 12:05:06] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:51 [I 2022-10-09 12:05:06] ...trap/engines/engine.py.326: process: 0.31931 | load: 0.00018 [I 2022-10-09 12:05:09] ...trap/engines/engine.py.321: eval: epoch 39 | batch 170/510 [I 2022-10-09 12:05:09] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:47 [I 2022-10-09 12:05:09] ...trap/engines/engine.py.326: process: 0.31873 | load: 0.00018 [I 2022-10-09 12:05:13] ...trap/engines/engine.py.321: eval: epoch 39 | batch 180/510 [I 2022-10-09 12:05:13] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:44 [I 2022-10-09 12:05:13] ...trap/engines/engine.py.326: process: 0.31451 | load: 0.00019 [I 2022-10-09 12:05:16] ...trap/engines/engine.py.321: eval: epoch 39 | batch 190/510 [I 2022-10-09 12:05:16] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:41 [I 2022-10-09 12:05:16] ...trap/engines/engine.py.326: process: 0.31845 | load: 0.00018 [I 2022-10-09 12:05:19] ...trap/engines/engine.py.321: eval: epoch 39 | batch 200/510 [I 2022-10-09 12:05:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:37 [I 2022-10-09 12:05:19] ...trap/engines/engine.py.326: process: 0.30992 | load: 0.00018 [I 2022-10-09 12:05:22] ...trap/engines/engine.py.321: eval: epoch 39 | batch 210/510 [I 2022-10-09 12:05:22] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:34 [I 2022-10-09 12:05:22] ...trap/engines/engine.py.326: process: 0.31697 | load: 0.00019 [I 2022-10-09 12:05:25] ...trap/engines/engine.py.321: eval: epoch 39 | batch 220/510 [I 2022-10-09 12:05:25] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:32 [I 2022-10-09 12:05:25] ...trap/engines/engine.py.326: process: 0.31855 | load: 0.00019 [I 2022-10-09 12:05:28] ...trap/engines/engine.py.321: eval: epoch 39 | batch 230/510 [I 2022-10-09 12:05:28] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:28 [I 2022-10-09 12:05:28] ...trap/engines/engine.py.326: process: 0.31782 | load: 0.00019 [I 2022-10-09 12:05:32] ...trap/engines/engine.py.321: eval: epoch 39 | batch 240/510 [I 2022-10-09 12:05:32] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:25 [I 2022-10-09 12:05:32] ...trap/engines/engine.py.326: process: 0.31571 | load: 0.00019 [I 2022-10-09 12:05:35] ...trap/engines/engine.py.321: eval: epoch 39 | batch 250/510 [I 2022-10-09 12:05:35] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:22 [I 2022-10-09 12:05:35] ...trap/engines/engine.py.326: process: 0.32019 | load: 0.00019 [I 2022-10-09 12:05:38] ...trap/engines/engine.py.321: eval: epoch 39 | batch 260/510 [I 2022-10-09 12:05:38] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:19 [I 2022-10-09 12:05:38] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00019 [I 2022-10-09 12:05:41] ...trap/engines/engine.py.321: eval: epoch 39 | batch 270/510 [I 2022-10-09 12:05:41] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:16 [I 2022-10-09 12:05:41] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00020 [I 2022-10-09 12:05:44] ...trap/engines/engine.py.321: eval: epoch 39 | batch 280/510 [I 2022-10-09 12:05:44] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:13 [I 2022-10-09 12:05:44] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00020 [I 2022-10-09 12:05:47] ...trap/engines/engine.py.321: eval: epoch 39 | batch 290/510 [I 2022-10-09 12:05:47] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:09 [I 2022-10-09 12:05:47] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00019 [I 2022-10-09 12:05:51] ...trap/engines/engine.py.321: eval: epoch 39 | batch 300/510 [I 2022-10-09 12:05:51] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:06 [I 2022-10-09 12:05:51] ...trap/engines/engine.py.326: process: 0.31975 | load: 0.00019 [I 2022-10-09 12:05:54] ...trap/engines/engine.py.321: eval: epoch 39 | batch 310/510 [I 2022-10-09 12:05:54] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:03 [I 2022-10-09 12:05:54] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00019 [I 2022-10-09 12:05:57] ...trap/engines/engine.py.321: eval: epoch 39 | batch 320/510 [I 2022-10-09 12:05:57] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:00 [I 2022-10-09 12:05:57] ...trap/engines/engine.py.326: process: 0.32028 | load: 0.00020 [I 2022-10-09 12:06:00] ...trap/engines/engine.py.321: eval: epoch 39 | batch 330/510 [I 2022-10-09 12:06:00] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:57 [I 2022-10-09 12:06:00] ...trap/engines/engine.py.326: process: 0.31562 | load: 0.00019 [I 2022-10-09 12:06:03] ...trap/engines/engine.py.321: eval: epoch 39 | batch 340/510 [I 2022-10-09 12:06:03] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:53 [I 2022-10-09 12:06:03] ...trap/engines/engine.py.326: process: 0.31007 | load: 0.00019 [I 2022-10-09 12:06:06] ...trap/engines/engine.py.321: eval: epoch 39 | batch 350/510 [I 2022-10-09 12:06:06] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:50 [I 2022-10-09 12:06:06] ...trap/engines/engine.py.326: process: 0.31407 | load: 0.00019 [I 2022-10-09 12:06:10] ...trap/engines/engine.py.321: eval: epoch 39 | batch 360/510 [I 2022-10-09 12:06:10] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:47 [I 2022-10-09 12:06:10] ...trap/engines/engine.py.326: process: 0.31327 | load: 0.00019 [I 2022-10-09 12:06:13] ...trap/engines/engine.py.321: eval: epoch 39 | batch 370/510 [I 2022-10-09 12:06:13] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:44 [I 2022-10-09 12:06:13] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00020 [I 2022-10-09 12:06:16] ...trap/engines/engine.py.321: eval: epoch 39 | batch 380/510 [I 2022-10-09 12:06:16] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:41 [I 2022-10-09 12:06:16] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00021 [I 2022-10-09 12:06:19] ...trap/engines/engine.py.321: eval: epoch 39 | batch 390/510 [I 2022-10-09 12:06:19] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:38 [I 2022-10-09 12:06:19] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00019 [I 2022-10-09 12:06:22] ...trap/engines/engine.py.321: eval: epoch 39 | batch 400/510 [I 2022-10-09 12:06:22] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:35 [I 2022-10-09 12:06:22] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00019 [I 2022-10-09 12:06:26] ...trap/engines/engine.py.321: eval: epoch 39 | batch 410/510 [I 2022-10-09 12:06:26] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:31 [I 2022-10-09 12:06:26] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00018 [I 2022-10-09 12:06:29] ...trap/engines/engine.py.321: eval: epoch 39 | batch 420/510 [I 2022-10-09 12:06:29] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:28 [I 2022-10-09 12:06:29] ...trap/engines/engine.py.326: process: 0.31688 | load: 0.00019 [I 2022-10-09 12:06:32] ...trap/engines/engine.py.321: eval: epoch 39 | batch 430/510 [I 2022-10-09 12:06:32] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:25 [I 2022-10-09 12:06:32] ...trap/engines/engine.py.326: process: 0.31830 | load: 0.00019 [I 2022-10-09 12:06:35] ...trap/engines/engine.py.321: eval: epoch 39 | batch 440/510 [I 2022-10-09 12:06:35] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:22 [I 2022-10-09 12:06:35] ...trap/engines/engine.py.326: process: 0.32134 | load: 0.00018 [I 2022-10-09 12:06:38] ...trap/engines/engine.py.321: eval: epoch 39 | batch 450/510 [I 2022-10-09 12:06:38] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:19 [I 2022-10-09 12:06:38] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00020 [I 2022-10-09 12:06:41] ...trap/engines/engine.py.321: eval: epoch 39 | batch 460/510 [I 2022-10-09 12:06:41] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:15 [I 2022-10-09 12:06:41] ...trap/engines/engine.py.326: process: 0.31209 | load: 0.00018 [I 2022-10-09 12:06:45] ...trap/engines/engine.py.321: eval: epoch 39 | batch 470/510 [I 2022-10-09 12:06:45] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:12 [I 2022-10-09 12:06:45] ...trap/engines/engine.py.326: process: 0.31698 | load: 0.00018 [I 2022-10-09 12:06:48] ...trap/engines/engine.py.321: eval: epoch 39 | batch 480/510 [I 2022-10-09 12:06:48] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:09 [I 2022-10-09 12:06:48] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00019 [I 2022-10-09 12:06:51] ...trap/engines/engine.py.321: eval: epoch 39 | batch 490/510 [I 2022-10-09 12:06:51] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:06 [I 2022-10-09 12:06:51] ...trap/engines/engine.py.326: process: 0.32172 | load: 0.00019 [I 2022-10-09 12:06:54] ...trap/engines/engine.py.321: eval: epoch 39 | batch 500/510 [I 2022-10-09 12:06:54] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:03 [I 2022-10-09 12:06:54] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00019 [I 2022-10-09 12:06:57] ...trap/engines/engine.py.321: eval: epoch 39 | batch 510/510 [I 2022-10-09 12:06:57] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:00 [I 2022-10-09 12:06:57] ...trap/engines/engine.py.326: process: 0.30992 | load: 0.00014 [S 2022-10-09 12:06:58] ...trap/engines/engine.py.132: eval_epoch.epoch: 39 [I 2022-10-09 12:06:58] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.711700 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902000 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935100 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.699500 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897400 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930700 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007457 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008899 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007726 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011413 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009541 [S 2022-10-09 12:07:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008026 [I 2022-10-09 12:07:32] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 12:07:32] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 12:07:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 12:07:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 12:07:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 12:07:36] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 12:07:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 12:07:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 12:07:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 12:07:42] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 12:07:44] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 12:07:50] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 12:07:50] ...trap/engines/engine.py.163: Training model on trainset for epoch 40 [I 2022-10-09 12:07:50] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 12:07:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 0/2304 [I 2022-10-09 12:07:53] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:24 [I 2022-10-09 12:07:53] ...trap/engines/engine.py.229: process: 2.61484 | load: 1.70724 [I 2022-10-09 12:07:53] ...trap/engines/engine.py.230: loss: 0.18671 [I 2022-10-09 12:07:53] ...optimizers/trijoint.py.151: total_norm: 21.256550 [I 2022-10-09 12:08:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 10/2304 [I 2022-10-09 12:08:01] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:21 [I 2022-10-09 12:08:01] ...trap/engines/engine.py.229: process: 0.83142 | load: 0.00428 [I 2022-10-09 12:08:01] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-09 12:08:01] ...optimizers/trijoint.py.151: total_norm: 17.633654 [I 2022-10-09 12:08:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 20/2304 [I 2022-10-09 12:08:10] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:52 [I 2022-10-09 12:08:10] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00521 [I 2022-10-09 12:08:10] ...trap/engines/engine.py.230: loss: 0.12464 [I 2022-10-09 12:08:10] ...optimizers/trijoint.py.151: total_norm: 56.528591 [I 2022-10-09 12:08:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 30/2304 [I 2022-10-09 12:08:18] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:31:59 [I 2022-10-09 12:08:18] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00448 [I 2022-10-09 12:08:18] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-09 12:08:18] ...optimizers/trijoint.py.151: total_norm: 14.007812 [I 2022-10-09 12:08:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 40/2304 [I 2022-10-09 12:08:26] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:40 [I 2022-10-09 12:08:26] ...trap/engines/engine.py.229: process: 0.82924 | load: 0.00413 [I 2022-10-09 12:08:26] ...trap/engines/engine.py.230: loss: 0.13625 [I 2022-10-09 12:08:26] ...optimizers/trijoint.py.151: total_norm: 17.042776 [I 2022-10-09 12:08:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 50/2304 [I 2022-10-09 12:08:35] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:43 [I 2022-10-09 12:08:35] ...trap/engines/engine.py.229: process: 0.85465 | load: 0.00385 [I 2022-10-09 12:08:35] ...trap/engines/engine.py.230: loss: 0.19687 [I 2022-10-09 12:08:35] ...optimizers/trijoint.py.151: total_norm: 24.253025 [I 2022-10-09 12:08:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 60/2304 [I 2022-10-09 12:08:43] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:26 [I 2022-10-09 12:08:43] ...trap/engines/engine.py.229: process: 0.82682 | load: 0.00305 [I 2022-10-09 12:08:43] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-09 12:08:43] ...optimizers/trijoint.py.151: total_norm: 9.794509 [I 2022-10-09 12:08:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 70/2304 [I 2022-10-09 12:08:52] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:31 [I 2022-10-09 12:08:52] ...trap/engines/engine.py.229: process: 0.86185 | load: 0.00336 [I 2022-10-09 12:08:52] ...trap/engines/engine.py.230: loss: 0.12500 [I 2022-10-09 12:08:52] ...optimizers/trijoint.py.151: total_norm: 19.403175 [I 2022-10-09 12:09:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 80/2304 [I 2022-10-09 12:09:00] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:22 [I 2022-10-09 12:09:00] ...trap/engines/engine.py.229: process: 0.84818 | load: 0.00380 [I 2022-10-09 12:09:00] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-09 12:09:00] ...optimizers/trijoint.py.151: total_norm: 27.026320 [I 2022-10-09 12:09:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 90/2304 [I 2022-10-09 12:09:09] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:08 [I 2022-10-09 12:09:09] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00328 [I 2022-10-09 12:09:09] ...trap/engines/engine.py.230: loss: 0.23640 [I 2022-10-09 12:09:09] ...optimizers/trijoint.py.151: total_norm: 21.862749 [I 2022-10-09 12:09:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 100/2304 [I 2022-10-09 12:09:17] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:02 [I 2022-10-09 12:09:17] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00296 [I 2022-10-09 12:09:17] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-09 12:09:17] ...optimizers/trijoint.py.151: total_norm: 9.778131 [I 2022-10-09 12:09:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 110/2304 [I 2022-10-09 12:09:26] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:44 [I 2022-10-09 12:09:26] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00316 [I 2022-10-09 12:09:26] ...trap/engines/engine.py.230: loss: 0.23668 [I 2022-10-09 12:09:26] ...optimizers/trijoint.py.151: total_norm: 28.599510 [I 2022-10-09 12:09:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 120/2304 [I 2022-10-09 12:09:34] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:38 [I 2022-10-09 12:09:34] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00320 [I 2022-10-09 12:09:34] ...trap/engines/engine.py.230: loss: 0.19423 [I 2022-10-09 12:09:34] ...optimizers/trijoint.py.151: total_norm: 33.443394 [I 2022-10-09 12:09:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 130/2304 [I 2022-10-09 12:09:43] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:31 [I 2022-10-09 12:09:43] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00305 [I 2022-10-09 12:09:43] ...trap/engines/engine.py.230: loss: 0.24585 [I 2022-10-09 12:09:43] ...optimizers/trijoint.py.151: total_norm: 19.729568 [I 2022-10-09 12:09:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 140/2304 [I 2022-10-09 12:09:51] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:25 [I 2022-10-09 12:09:51] ...trap/engines/engine.py.229: process: 0.84013 | load: 0.00304 [I 2022-10-09 12:09:51] ...trap/engines/engine.py.230: loss: 0.13711 [I 2022-10-09 12:09:51] ...optimizers/trijoint.py.151: total_norm: 13.700409 [I 2022-10-09 12:09:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 150/2304 [I 2022-10-09 12:09:59] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:21 [I 2022-10-09 12:09:59] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00320 [I 2022-10-09 12:09:59] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-09 12:09:59] ...optimizers/trijoint.py.151: total_norm: 36.939857 [I 2022-10-09 12:10:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 160/2304 [I 2022-10-09 12:10:08] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:00 [I 2022-10-09 12:10:08] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00303 [I 2022-10-09 12:10:08] ...trap/engines/engine.py.230: loss: 0.19121 [I 2022-10-09 12:10:08] ...optimizers/trijoint.py.151: total_norm: 20.891197 [I 2022-10-09 12:10:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 170/2304 [I 2022-10-09 12:10:16] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:55 [I 2022-10-09 12:10:16] ...trap/engines/engine.py.229: process: 0.83824 | load: 0.00306 [I 2022-10-09 12:10:16] ...trap/engines/engine.py.230: loss: 0.17190 [I 2022-10-09 12:10:16] ...optimizers/trijoint.py.151: total_norm: 25.434582 [I 2022-10-09 12:10:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 180/2304 [I 2022-10-09 12:10:25] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:50 [I 2022-10-09 12:10:25] ...trap/engines/engine.py.229: process: 0.83428 | load: 0.00297 [I 2022-10-09 12:10:25] ...trap/engines/engine.py.230: loss: 0.21141 [I 2022-10-09 12:10:25] ...optimizers/trijoint.py.151: total_norm: 22.155218 [I 2022-10-09 12:10:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 190/2304 [I 2022-10-09 12:10:33] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:29:42 [I 2022-10-09 12:10:33] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00293 [I 2022-10-09 12:10:33] ...trap/engines/engine.py.230: loss: 0.14849 [I 2022-10-09 12:10:33] ...optimizers/trijoint.py.151: total_norm: 17.885521 [I 2022-10-09 12:10:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 200/2304 [I 2022-10-09 12:10:42] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:26 [I 2022-10-09 12:10:42] ...trap/engines/engine.py.229: process: 0.83114 | load: 0.00316 [I 2022-10-09 12:10:42] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-09 12:10:42] ...optimizers/trijoint.py.151: total_norm: 11.176354 [I 2022-10-09 12:10:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 210/2304 [I 2022-10-09 12:10:50] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:18 [I 2022-10-09 12:10:50] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00303 [I 2022-10-09 12:10:50] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-09 12:10:50] ...optimizers/trijoint.py.151: total_norm: 10.309124 [I 2022-10-09 12:10:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 220/2304 [I 2022-10-09 12:10:58] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:07 [I 2022-10-09 12:10:58] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00307 [I 2022-10-09 12:10:58] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-09 12:10:58] ...optimizers/trijoint.py.151: total_norm: 12.936414 [I 2022-10-09 12:11:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 230/2304 [I 2022-10-09 12:11:07] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:07 [I 2022-10-09 12:11:07] ...trap/engines/engine.py.229: process: 0.83509 | load: 0.00317 [I 2022-10-09 12:11:07] ...trap/engines/engine.py.230: loss: 0.19674 [I 2022-10-09 12:11:07] ...optimizers/trijoint.py.151: total_norm: 12.123920 [I 2022-10-09 12:11:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 240/2304 [I 2022-10-09 12:11:15] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:57 [I 2022-10-09 12:11:15] ...trap/engines/engine.py.229: process: 0.84057 | load: 0.00294 [I 2022-10-09 12:11:15] ...trap/engines/engine.py.230: loss: 0.11350 [I 2022-10-09 12:11:15] ...optimizers/trijoint.py.151: total_norm: 15.134203 [I 2022-10-09 12:11:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 250/2304 [I 2022-10-09 12:11:24] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:56 [I 2022-10-09 12:11:24] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00293 [I 2022-10-09 12:11:24] ...trap/engines/engine.py.230: loss: 0.15537 [I 2022-10-09 12:11:24] ...optimizers/trijoint.py.151: total_norm: 20.067011 [I 2022-10-09 12:11:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 260/2304 [I 2022-10-09 12:11:32] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:28 [I 2022-10-09 12:11:32] ...trap/engines/engine.py.229: process: 0.82813 | load: 0.00302 [I 2022-10-09 12:11:32] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-09 12:11:32] ...optimizers/trijoint.py.151: total_norm: 22.947977 [I 2022-10-09 12:11:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 270/2304 [I 2022-10-09 12:11:41] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:33 [I 2022-10-09 12:11:41] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00293 [I 2022-10-09 12:11:41] ...trap/engines/engine.py.230: loss: 0.20637 [I 2022-10-09 12:11:41] ...optimizers/trijoint.py.151: total_norm: 27.899624 [I 2022-10-09 12:11:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 280/2304 [I 2022-10-09 12:11:49] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:25 [I 2022-10-09 12:11:49] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00296 [I 2022-10-09 12:11:49] ...trap/engines/engine.py.230: loss: 0.16702 [I 2022-10-09 12:11:49] ...optimizers/trijoint.py.151: total_norm: 17.516701 [I 2022-10-09 12:11:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 290/2304 [I 2022-10-09 12:11:57] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:16 [I 2022-10-09 12:11:57] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00306 [I 2022-10-09 12:11:57] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-09 12:11:57] ...optimizers/trijoint.py.151: total_norm: 17.071949 [I 2022-10-09 12:12:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 300/2304 [I 2022-10-09 12:12:06] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:28:05 [I 2022-10-09 12:12:06] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00292 [I 2022-10-09 12:12:06] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-09 12:12:06] ...optimizers/trijoint.py.151: total_norm: 19.361652 [I 2022-10-09 12:12:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 310/2304 [I 2022-10-09 12:12:14] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:58 [I 2022-10-09 12:12:14] ...trap/engines/engine.py.229: process: 0.84807 | load: 0.00313 [I 2022-10-09 12:12:14] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-09 12:12:14] ...optimizers/trijoint.py.151: total_norm: 8.949874 [I 2022-10-09 12:12:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 320/2304 [I 2022-10-09 12:12:23] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:47 [I 2022-10-09 12:12:23] ...trap/engines/engine.py.229: process: 0.83240 | load: 0.00294 [I 2022-10-09 12:12:23] ...trap/engines/engine.py.230: loss: 0.14167 [I 2022-10-09 12:12:23] ...optimizers/trijoint.py.151: total_norm: 32.111423 [I 2022-10-09 12:12:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 330/2304 [I 2022-10-09 12:12:31] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:46 [I 2022-10-09 12:12:31] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00305 [I 2022-10-09 12:12:31] ...trap/engines/engine.py.230: loss: 0.14669 [I 2022-10-09 12:12:31] ...optimizers/trijoint.py.151: total_norm: 18.188513 [I 2022-10-09 12:12:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 340/2304 [I 2022-10-09 12:12:39] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:27 [I 2022-10-09 12:12:39] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00294 [I 2022-10-09 12:12:39] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-09 12:12:39] ...optimizers/trijoint.py.151: total_norm: 17.414759 [I 2022-10-09 12:12:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 350/2304 [I 2022-10-09 12:12:48] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:21 [I 2022-10-09 12:12:48] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00292 [I 2022-10-09 12:12:48] ...trap/engines/engine.py.230: loss: 0.13590 [I 2022-10-09 12:12:48] ...optimizers/trijoint.py.151: total_norm: 21.765322 [I 2022-10-09 12:12:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 360/2304 [I 2022-10-09 12:12:56] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:12 [I 2022-10-09 12:12:56] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00303 [I 2022-10-09 12:12:56] ...trap/engines/engine.py.230: loss: 0.12723 [I 2022-10-09 12:12:56] ...optimizers/trijoint.py.151: total_norm: 13.765155 [I 2022-10-09 12:13:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 370/2304 [I 2022-10-09 12:13:05] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:08 [I 2022-10-09 12:13:05] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00291 [I 2022-10-09 12:13:05] ...trap/engines/engine.py.230: loss: 0.26293 [I 2022-10-09 12:13:05] ...optimizers/trijoint.py.151: total_norm: 28.375303 [I 2022-10-09 12:13:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 380/2304 [I 2022-10-09 12:13:13] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:58 [I 2022-10-09 12:13:13] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00295 [I 2022-10-09 12:13:13] ...trap/engines/engine.py.230: loss: 0.17373 [I 2022-10-09 12:13:13] ...optimizers/trijoint.py.151: total_norm: 34.870026 [I 2022-10-09 12:13:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 390/2304 [I 2022-10-09 12:13:22] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:49 [I 2022-10-09 12:13:22] ...trap/engines/engine.py.229: process: 0.84465 | load: 0.00301 [I 2022-10-09 12:13:22] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-09 12:13:22] ...optimizers/trijoint.py.151: total_norm: 30.133072 [I 2022-10-09 12:13:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 400/2304 [I 2022-10-09 12:13:30] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:48 [I 2022-10-09 12:13:30] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00291 [I 2022-10-09 12:13:30] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-09 12:13:30] ...optimizers/trijoint.py.151: total_norm: 29.034794 [I 2022-10-09 12:13:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 410/2304 [I 2022-10-09 12:13:38] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:26:34 [I 2022-10-09 12:13:38] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00295 [I 2022-10-09 12:13:38] ...trap/engines/engine.py.230: loss: 0.20573 [I 2022-10-09 12:13:38] ...optimizers/trijoint.py.151: total_norm: 42.525467 [I 2022-10-09 12:13:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 420/2304 [I 2022-10-09 12:13:47] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:30 [I 2022-10-09 12:13:47] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00291 [I 2022-10-09 12:13:47] ...trap/engines/engine.py.230: loss: 0.15726 [I 2022-10-09 12:13:47] ...optimizers/trijoint.py.151: total_norm: 13.431700 [I 2022-10-09 12:13:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 430/2304 [I 2022-10-09 12:13:55] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:21 [I 2022-10-09 12:13:55] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00302 [I 2022-10-09 12:13:55] ...trap/engines/engine.py.230: loss: 0.18301 [I 2022-10-09 12:13:55] ...optimizers/trijoint.py.151: total_norm: 15.164442 [I 2022-10-09 12:14:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 440/2304 [I 2022-10-09 12:14:04] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:13 [I 2022-10-09 12:14:04] ...trap/engines/engine.py.229: process: 0.85436 | load: 0.00290 [I 2022-10-09 12:14:04] ...trap/engines/engine.py.230: loss: 0.13870 [I 2022-10-09 12:14:04] ...optimizers/trijoint.py.151: total_norm: 25.137873 [I 2022-10-09 12:14:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 450/2304 [I 2022-10-09 12:14:12] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:54 [I 2022-10-09 12:14:12] ...trap/engines/engine.py.229: process: 0.83365 | load: 0.00293 [I 2022-10-09 12:14:12] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-09 12:14:12] ...optimizers/trijoint.py.151: total_norm: 17.159767 [I 2022-10-09 12:14:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 460/2304 [I 2022-10-09 12:14:21] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:25:51 [I 2022-10-09 12:14:21] ...trap/engines/engine.py.229: process: 0.83816 | load: 0.00294 [I 2022-10-09 12:14:21] ...trap/engines/engine.py.230: loss: 0.14501 [I 2022-10-09 12:14:21] ...optimizers/trijoint.py.151: total_norm: 26.108719 [I 2022-10-09 12:14:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 470/2304 [I 2022-10-09 12:14:29] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:42 [I 2022-10-09 12:14:29] ...trap/engines/engine.py.229: process: 0.84328 | load: 0.00295 [I 2022-10-09 12:14:29] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-09 12:14:29] ...optimizers/trijoint.py.151: total_norm: 21.254295 [I 2022-10-09 12:14:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 480/2304 [I 2022-10-09 12:14:37] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:39 [I 2022-10-09 12:14:37] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00291 [I 2022-10-09 12:14:37] ...trap/engines/engine.py.230: loss: 0.19250 [I 2022-10-09 12:14:37] ...optimizers/trijoint.py.151: total_norm: 31.579422 [I 2022-10-09 12:14:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 490/2304 [I 2022-10-09 12:14:46] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:25:30 [I 2022-10-09 12:14:46] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00294 [I 2022-10-09 12:14:46] ...trap/engines/engine.py.230: loss: 0.14455 [I 2022-10-09 12:14:46] ...optimizers/trijoint.py.151: total_norm: 17.279831 [I 2022-10-09 12:14:54] ...trap/engines/engine.py.224: train: epoch 40 | batch 500/2304 [I 2022-10-09 12:14:54] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:17 [I 2022-10-09 12:14:54] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00292 [I 2022-10-09 12:14:54] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-09 12:14:54] ...optimizers/trijoint.py.151: total_norm: 16.516647 [I 2022-10-09 12:15:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 510/2304 [I 2022-10-09 12:15:03] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:10 [I 2022-10-09 12:15:03] ...trap/engines/engine.py.229: process: 0.83222 | load: 0.00292 [I 2022-10-09 12:15:03] ...trap/engines/engine.py.230: loss: 0.13295 [I 2022-10-09 12:15:03] ...optimizers/trijoint.py.151: total_norm: 11.067023 [I 2022-10-09 12:15:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 520/2304 [I 2022-10-09 12:15:11] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:05 [I 2022-10-09 12:15:11] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00292 [I 2022-10-09 12:15:11] ...trap/engines/engine.py.230: loss: 0.22533 [I 2022-10-09 12:15:11] ...optimizers/trijoint.py.151: total_norm: 23.955963 [I 2022-10-09 12:15:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 530/2304 [I 2022-10-09 12:15:20] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:53 [I 2022-10-09 12:15:20] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00294 [I 2022-10-09 12:15:20] ...trap/engines/engine.py.230: loss: 0.15626 [I 2022-10-09 12:15:20] ...optimizers/trijoint.py.151: total_norm: 17.689236 [I 2022-10-09 12:15:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 540/2304 [I 2022-10-09 12:15:28] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:41 [I 2022-10-09 12:15:28] ...trap/engines/engine.py.229: process: 0.83955 | load: 0.00290 [I 2022-10-09 12:15:28] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-09 12:15:28] ...optimizers/trijoint.py.151: total_norm: 9.454953 [I 2022-10-09 12:15:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 550/2304 [I 2022-10-09 12:15:37] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:44 [I 2022-10-09 12:15:37] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00297 [I 2022-10-09 12:15:37] ...trap/engines/engine.py.230: loss: 0.18647 [I 2022-10-09 12:15:37] ...optimizers/trijoint.py.151: total_norm: 30.330889 [I 2022-10-09 12:15:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 560/2304 [I 2022-10-09 12:15:45] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:24 [I 2022-10-09 12:15:45] ...trap/engines/engine.py.229: process: 0.82834 | load: 0.00292 [I 2022-10-09 12:15:45] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-09 12:15:45] ...optimizers/trijoint.py.151: total_norm: 8.103731 [I 2022-10-09 12:15:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 570/2304 [I 2022-10-09 12:15:53] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:15 [I 2022-10-09 12:15:53] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00298 [I 2022-10-09 12:15:53] ...trap/engines/engine.py.230: loss: 0.17735 [I 2022-10-09 12:15:53] ...optimizers/trijoint.py.151: total_norm: 39.197849 [I 2022-10-09 12:16:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 580/2304 [I 2022-10-09 12:16:02] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:05 [I 2022-10-09 12:16:02] ...trap/engines/engine.py.229: process: 0.83891 | load: 0.00293 [I 2022-10-09 12:16:02] ...trap/engines/engine.py.230: loss: 0.22429 [I 2022-10-09 12:16:02] ...optimizers/trijoint.py.151: total_norm: 34.216774 [I 2022-10-09 12:16:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 590/2304 [I 2022-10-09 12:16:10] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:23:56 [I 2022-10-09 12:16:10] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00300 [I 2022-10-09 12:16:10] ...trap/engines/engine.py.230: loss: 0.19083 [I 2022-10-09 12:16:10] ...optimizers/trijoint.py.151: total_norm: 23.736618 [I 2022-10-09 12:16:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 600/2304 [I 2022-10-09 12:16:19] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:24:11 [I 2022-10-09 12:16:19] ...trap/engines/engine.py.229: process: 0.85063 | load: 0.00290 [I 2022-10-09 12:16:19] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-09 12:16:19] ...optimizers/trijoint.py.151: total_norm: 15.098796 [I 2022-10-09 12:16:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 610/2304 [I 2022-10-09 12:16:27] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:24:08 [I 2022-10-09 12:16:27] ...trap/engines/engine.py.229: process: 0.85436 | load: 0.00291 [I 2022-10-09 12:16:27] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-09 12:16:27] ...optimizers/trijoint.py.151: total_norm: 28.545145 [I 2022-10-09 12:16:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 620/2304 [I 2022-10-09 12:16:36] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:23:53 [I 2022-10-09 12:16:36] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00292 [I 2022-10-09 12:16:36] ...trap/engines/engine.py.230: loss: 0.28624 [I 2022-10-09 12:16:36] ...optimizers/trijoint.py.151: total_norm: 20.319981 [I 2022-10-09 12:16:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 630/2304 [I 2022-10-09 12:16:44] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:34 [I 2022-10-09 12:16:44] ...trap/engines/engine.py.229: process: 0.84116 | load: 0.00293 [I 2022-10-09 12:16:44] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-09 12:16:44] ...optimizers/trijoint.py.151: total_norm: 13.571774 [I 2022-10-09 12:16:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 640/2304 [I 2022-10-09 12:16:53] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:17 [I 2022-10-09 12:16:53] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00297 [I 2022-10-09 12:16:53] ...trap/engines/engine.py.230: loss: 0.18918 [I 2022-10-09 12:16:53] ...optimizers/trijoint.py.151: total_norm: 12.274200 [I 2022-10-09 12:17:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 650/2304 [I 2022-10-09 12:17:01] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:10 [I 2022-10-09 12:17:01] ...trap/engines/engine.py.229: process: 0.83247 | load: 0.00301 [I 2022-10-09 12:17:01] ...trap/engines/engine.py.230: loss: 0.10017 [I 2022-10-09 12:17:01] ...optimizers/trijoint.py.151: total_norm: 7.705077 [I 2022-10-09 12:17:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 660/2304 [I 2022-10-09 12:17:10] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:04 [I 2022-10-09 12:17:10] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00301 [I 2022-10-09 12:17:10] ...trap/engines/engine.py.230: loss: 0.18189 [I 2022-10-09 12:17:10] ...optimizers/trijoint.py.151: total_norm: 31.199671 [I 2022-10-09 12:17:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 670/2304 [I 2022-10-09 12:17:18] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:57 [I 2022-10-09 12:17:18] ...trap/engines/engine.py.229: process: 0.83320 | load: 0.00292 [I 2022-10-09 12:17:18] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-09 12:17:18] ...optimizers/trijoint.py.151: total_norm: 10.985038 [I 2022-10-09 12:17:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 680/2304 [I 2022-10-09 12:17:26] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:47 [I 2022-10-09 12:17:26] ...trap/engines/engine.py.229: process: 0.83119 | load: 0.00292 [I 2022-10-09 12:17:26] ...trap/engines/engine.py.230: loss: 0.20867 [I 2022-10-09 12:17:26] ...optimizers/trijoint.py.151: total_norm: 18.460030 [I 2022-10-09 12:17:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 690/2304 [I 2022-10-09 12:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:30 [I 2022-10-09 12:17:35] ...trap/engines/engine.py.229: process: 0.82903 | load: 0.00292 [I 2022-10-09 12:17:35] ...trap/engines/engine.py.230: loss: 0.14373 [I 2022-10-09 12:17:35] ...optimizers/trijoint.py.151: total_norm: 14.262552 [I 2022-10-09 12:17:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 700/2304 [I 2022-10-09 12:17:43] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:23 [I 2022-10-09 12:17:43] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00294 [I 2022-10-09 12:17:43] ...trap/engines/engine.py.230: loss: 0.17849 [I 2022-10-09 12:17:43] ...optimizers/trijoint.py.151: total_norm: 15.848042 [I 2022-10-09 12:17:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 710/2304 [I 2022-10-09 12:17:52] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:20 [I 2022-10-09 12:17:52] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00300 [I 2022-10-09 12:17:52] ...trap/engines/engine.py.230: loss: 0.19363 [I 2022-10-09 12:17:52] ...optimizers/trijoint.py.151: total_norm: 15.102908 [I 2022-10-09 12:18:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 720/2304 [I 2022-10-09 12:18:00] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:08 [I 2022-10-09 12:18:00] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00294 [I 2022-10-09 12:18:00] ...trap/engines/engine.py.230: loss: 0.14323 [I 2022-10-09 12:18:00] ...optimizers/trijoint.py.151: total_norm: 18.798119 [I 2022-10-09 12:18:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 730/2304 [I 2022-10-09 12:18:08] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:21:54 [I 2022-10-09 12:18:08] ...trap/engines/engine.py.229: process: 0.82875 | load: 0.00292 [I 2022-10-09 12:18:08] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-09 12:18:08] ...optimizers/trijoint.py.151: total_norm: 17.828259 [I 2022-10-09 12:18:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 740/2304 [I 2022-10-09 12:18:17] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:56 [I 2022-10-09 12:18:17] ...trap/engines/engine.py.229: process: 0.82858 | load: 0.00292 [I 2022-10-09 12:18:17] ...trap/engines/engine.py.230: loss: 0.14862 [I 2022-10-09 12:18:17] ...optimizers/trijoint.py.151: total_norm: 26.171627 [I 2022-10-09 12:18:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 750/2304 [I 2022-10-09 12:18:25] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:47 [I 2022-10-09 12:18:25] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00292 [I 2022-10-09 12:18:25] ...trap/engines/engine.py.230: loss: 0.10531 [I 2022-10-09 12:18:25] ...optimizers/trijoint.py.151: total_norm: 10.530465 [I 2022-10-09 12:18:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 760/2304 [I 2022-10-09 12:18:34] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:35 [I 2022-10-09 12:18:34] ...trap/engines/engine.py.229: process: 0.84024 | load: 0.00295 [I 2022-10-09 12:18:34] ...trap/engines/engine.py.230: loss: 0.10219 [I 2022-10-09 12:18:34] ...optimizers/trijoint.py.151: total_norm: 17.622524 [I 2022-10-09 12:18:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 770/2304 [I 2022-10-09 12:18:42] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:31 [I 2022-10-09 12:18:42] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00299 [I 2022-10-09 12:18:42] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-09 12:18:42] ...optimizers/trijoint.py.151: total_norm: 13.959552 [I 2022-10-09 12:18:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 780/2304 [I 2022-10-09 12:18:51] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:24 [I 2022-10-09 12:18:51] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00296 [I 2022-10-09 12:18:51] ...trap/engines/engine.py.230: loss: 0.18344 [I 2022-10-09 12:18:51] ...optimizers/trijoint.py.151: total_norm: 11.087810 [I 2022-10-09 12:18:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 790/2304 [I 2022-10-09 12:18:59] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:19 [I 2022-10-09 12:18:59] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00298 [I 2022-10-09 12:18:59] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-09 12:18:59] ...optimizers/trijoint.py.151: total_norm: 18.568670 [I 2022-10-09 12:19:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 800/2304 [I 2022-10-09 12:19:07] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:02 [I 2022-10-09 12:19:07] ...trap/engines/engine.py.229: process: 0.83457 | load: 0.00292 [I 2022-10-09 12:19:07] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-09 12:19:07] ...optimizers/trijoint.py.151: total_norm: 20.691183 [I 2022-10-09 12:19:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 810/2304 [I 2022-10-09 12:19:16] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:57 [I 2022-10-09 12:19:16] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00292 [I 2022-10-09 12:19:16] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-09 12:19:16] ...optimizers/trijoint.py.151: total_norm: 13.824135 [I 2022-10-09 12:19:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 820/2304 [I 2022-10-09 12:19:24] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:44 [I 2022-10-09 12:19:24] ...trap/engines/engine.py.229: process: 0.83186 | load: 0.00293 [I 2022-10-09 12:19:24] ...trap/engines/engine.py.230: loss: 0.14217 [I 2022-10-09 12:19:24] ...optimizers/trijoint.py.151: total_norm: 26.989836 [I 2022-10-09 12:19:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 830/2304 [I 2022-10-09 12:19:33] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:32 [I 2022-10-09 12:19:33] ...trap/engines/engine.py.229: process: 0.82599 | load: 0.00296 [I 2022-10-09 12:19:33] ...trap/engines/engine.py.230: loss: 0.13667 [I 2022-10-09 12:19:33] ...optimizers/trijoint.py.151: total_norm: 12.796819 [I 2022-10-09 12:19:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 840/2304 [I 2022-10-09 12:19:41] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:30 [I 2022-10-09 12:19:41] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00294 [I 2022-10-09 12:19:41] ...trap/engines/engine.py.230: loss: 0.17536 [I 2022-10-09 12:19:41] ...optimizers/trijoint.py.151: total_norm: 14.468554 [I 2022-10-09 12:19:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 850/2304 [I 2022-10-09 12:19:50] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:26 [I 2022-10-09 12:19:50] ...trap/engines/engine.py.229: process: 0.82502 | load: 0.00293 [I 2022-10-09 12:19:50] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-09 12:19:50] ...optimizers/trijoint.py.151: total_norm: 10.850599 [I 2022-10-09 12:19:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 860/2304 [I 2022-10-09 12:19:58] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:16 [I 2022-10-09 12:19:58] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00293 [I 2022-10-09 12:19:58] ...trap/engines/engine.py.230: loss: 0.18757 [I 2022-10-09 12:19:58] ...optimizers/trijoint.py.151: total_norm: 14.486120 [I 2022-10-09 12:20:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 870/2304 [I 2022-10-09 12:20:06] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:09 [I 2022-10-09 12:20:06] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00293 [I 2022-10-09 12:20:06] ...trap/engines/engine.py.230: loss: 0.19239 [I 2022-10-09 12:20:06] ...optimizers/trijoint.py.151: total_norm: 26.271416 [I 2022-10-09 12:20:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 880/2304 [I 2022-10-09 12:20:15] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:58 [I 2022-10-09 12:20:15] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00294 [I 2022-10-09 12:20:15] ...trap/engines/engine.py.230: loss: 0.21146 [I 2022-10-09 12:20:15] ...optimizers/trijoint.py.151: total_norm: 27.732063 [I 2022-10-09 12:20:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 890/2304 [I 2022-10-09 12:20:23] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:19:48 [I 2022-10-09 12:20:23] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00295 [I 2022-10-09 12:20:23] ...trap/engines/engine.py.230: loss: 0.17589 [I 2022-10-09 12:20:23] ...optimizers/trijoint.py.151: total_norm: 18.378458 [I 2022-10-09 12:20:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 900/2304 [I 2022-10-09 12:20:32] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:43 [I 2022-10-09 12:20:32] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00296 [I 2022-10-09 12:20:32] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-09 12:20:32] ...optimizers/trijoint.py.151: total_norm: 12.124840 [I 2022-10-09 12:20:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 910/2304 [I 2022-10-09 12:20:40] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:33 [I 2022-10-09 12:20:40] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00302 [I 2022-10-09 12:20:40] ...trap/engines/engine.py.230: loss: 0.17369 [I 2022-10-09 12:20:40] ...optimizers/trijoint.py.151: total_norm: 52.191555 [I 2022-10-09 12:20:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 920/2304 [I 2022-10-09 12:20:49] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:29 [I 2022-10-09 12:20:49] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00296 [I 2022-10-09 12:20:49] ...trap/engines/engine.py.230: loss: 0.18255 [I 2022-10-09 12:20:49] ...optimizers/trijoint.py.151: total_norm: 11.778346 [I 2022-10-09 12:20:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 930/2304 [I 2022-10-09 12:20:57] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:20 [I 2022-10-09 12:20:57] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00297 [I 2022-10-09 12:20:57] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-09 12:20:57] ...optimizers/trijoint.py.151: total_norm: 23.835762 [I 2022-10-09 12:21:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 940/2304 [I 2022-10-09 12:21:06] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:10 [I 2022-10-09 12:21:06] ...trap/engines/engine.py.229: process: 0.85203 | load: 0.00293 [I 2022-10-09 12:21:06] ...trap/engines/engine.py.230: loss: 0.21245 [I 2022-10-09 12:21:06] ...optimizers/trijoint.py.151: total_norm: 15.312451 [I 2022-10-09 12:21:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 950/2304 [I 2022-10-09 12:21:14] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:04 [I 2022-10-09 12:21:14] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00293 [I 2022-10-09 12:21:14] ...trap/engines/engine.py.230: loss: 0.14603 [I 2022-10-09 12:21:14] ...optimizers/trijoint.py.151: total_norm: 11.832555 [I 2022-10-09 12:21:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 960/2304 [I 2022-10-09 12:21:22] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:18:48 [I 2022-10-09 12:21:22] ...trap/engines/engine.py.229: process: 0.82850 | load: 0.00294 [I 2022-10-09 12:21:22] ...trap/engines/engine.py.230: loss: 0.19876 [I 2022-10-09 12:21:22] ...optimizers/trijoint.py.151: total_norm: 24.120554 [I 2022-10-09 12:21:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 970/2304 [I 2022-10-09 12:21:31] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:45 [I 2022-10-09 12:21:31] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00294 [I 2022-10-09 12:21:31] ...trap/engines/engine.py.230: loss: 0.21412 [I 2022-10-09 12:21:31] ...optimizers/trijoint.py.151: total_norm: 18.074224 [I 2022-10-09 12:21:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 980/2304 [I 2022-10-09 12:21:39] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:24 [I 2022-10-09 12:21:39] ...trap/engines/engine.py.229: process: 0.82949 | load: 0.00297 [I 2022-10-09 12:21:39] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-09 12:21:39] ...optimizers/trijoint.py.151: total_norm: 10.570654 [I 2022-10-09 12:21:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 990/2304 [I 2022-10-09 12:21:48] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:26 [I 2022-10-09 12:21:48] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00293 [I 2022-10-09 12:21:48] ...trap/engines/engine.py.230: loss: 0.14327 [I 2022-10-09 12:21:48] ...optimizers/trijoint.py.151: total_norm: 16.522348 [I 2022-10-09 12:21:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 1000/2304 [I 2022-10-09 12:21:56] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:16 [I 2022-10-09 12:21:56] ...trap/engines/engine.py.229: process: 0.83437 | load: 0.00294 [I 2022-10-09 12:21:56] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-09 12:21:56] ...optimizers/trijoint.py.151: total_norm: 11.322659 [I 2022-10-09 12:22:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1010/2304 [I 2022-10-09 12:22:04] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:03 [I 2022-10-09 12:22:04] ...trap/engines/engine.py.229: process: 0.83319 | load: 0.00292 [I 2022-10-09 12:22:04] ...trap/engines/engine.py.230: loss: 0.16236 [I 2022-10-09 12:22:04] ...optimizers/trijoint.py.151: total_norm: 14.164862 [I 2022-10-09 12:22:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 1020/2304 [I 2022-10-09 12:22:13] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:17:59 [I 2022-10-09 12:22:13] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00294 [I 2022-10-09 12:22:13] ...trap/engines/engine.py.230: loss: 0.22416 [I 2022-10-09 12:22:13] ...optimizers/trijoint.py.151: total_norm: 35.815491 [I 2022-10-09 12:22:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1030/2304 [I 2022-10-09 12:22:21] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:17:47 [I 2022-10-09 12:22:21] ...trap/engines/engine.py.229: process: 0.82680 | load: 0.00296 [I 2022-10-09 12:22:21] ...trap/engines/engine.py.230: loss: 0.13363 [I 2022-10-09 12:22:21] ...optimizers/trijoint.py.151: total_norm: 17.810080 [I 2022-10-09 12:22:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 1040/2304 [I 2022-10-09 12:22:30] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:43 [I 2022-10-09 12:22:30] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00293 [I 2022-10-09 12:22:30] ...trap/engines/engine.py.230: loss: 0.13230 [I 2022-10-09 12:22:30] ...optimizers/trijoint.py.151: total_norm: 18.710312 [I 2022-10-09 12:22:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1050/2304 [I 2022-10-09 12:22:38] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:33 [I 2022-10-09 12:22:38] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00298 [I 2022-10-09 12:22:38] ...trap/engines/engine.py.230: loss: 0.18719 [I 2022-10-09 12:22:38] ...optimizers/trijoint.py.151: total_norm: 57.046906 [I 2022-10-09 12:22:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 1060/2304 [I 2022-10-09 12:22:47] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:25 [I 2022-10-09 12:22:47] ...trap/engines/engine.py.229: process: 0.83964 | load: 0.00292 [I 2022-10-09 12:22:47] ...trap/engines/engine.py.230: loss: 0.17895 [I 2022-10-09 12:22:47] ...optimizers/trijoint.py.151: total_norm: 9.320314 [I 2022-10-09 12:22:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1070/2304 [I 2022-10-09 12:22:55] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:21 [I 2022-10-09 12:22:55] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00294 [I 2022-10-09 12:22:55] ...trap/engines/engine.py.230: loss: 0.15452 [I 2022-10-09 12:22:55] ...optimizers/trijoint.py.151: total_norm: 8.878636 [I 2022-10-09 12:23:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1080/2304 [I 2022-10-09 12:23:04] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:20 [I 2022-10-09 12:23:04] ...trap/engines/engine.py.229: process: 0.87328 | load: 0.00295 [I 2022-10-09 12:23:04] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-09 12:23:04] ...optimizers/trijoint.py.151: total_norm: 11.225935 [I 2022-10-09 12:23:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 1090/2304 [I 2022-10-09 12:23:12] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:17:11 [I 2022-10-09 12:23:12] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00297 [I 2022-10-09 12:23:12] ...trap/engines/engine.py.230: loss: 0.19403 [I 2022-10-09 12:23:12] ...optimizers/trijoint.py.151: total_norm: 12.954773 [I 2022-10-09 12:23:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1100/2304 [I 2022-10-09 12:23:21] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:08 [I 2022-10-09 12:23:21] ...trap/engines/engine.py.229: process: 0.87657 | load: 0.00296 [I 2022-10-09 12:23:21] ...trap/engines/engine.py.230: loss: 0.13442 [I 2022-10-09 12:23:21] ...optimizers/trijoint.py.151: total_norm: 11.518828 [I 2022-10-09 12:23:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 1110/2304 [I 2022-10-09 12:23:29] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:16:50 [I 2022-10-09 12:23:29] ...trap/engines/engine.py.229: process: 0.82764 | load: 0.00294 [I 2022-10-09 12:23:29] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-09 12:23:29] ...optimizers/trijoint.py.151: total_norm: 26.695677 [I 2022-10-09 12:23:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1120/2304 [I 2022-10-09 12:23:38] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:45 [I 2022-10-09 12:23:38] ...trap/engines/engine.py.229: process: 0.86621 | load: 0.00305 [I 2022-10-09 12:23:38] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-09 12:23:38] ...optimizers/trijoint.py.151: total_norm: 23.891188 [I 2022-10-09 12:23:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 1130/2304 [I 2022-10-09 12:23:46] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:29 [I 2022-10-09 12:23:46] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00292 [I 2022-10-09 12:23:46] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-09 12:23:46] ...optimizers/trijoint.py.151: total_norm: 10.181301 [I 2022-10-09 12:23:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1140/2304 [I 2022-10-09 12:23:55] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:23 [I 2022-10-09 12:23:55] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00299 [I 2022-10-09 12:23:55] ...trap/engines/engine.py.230: loss: 0.15936 [I 2022-10-09 12:23:55] ...optimizers/trijoint.py.151: total_norm: 14.078107 [I 2022-10-09 12:24:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 1150/2304 [I 2022-10-09 12:24:03] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:09 [I 2022-10-09 12:24:03] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00294 [I 2022-10-09 12:24:03] ...trap/engines/engine.py.230: loss: 0.18695 [I 2022-10-09 12:24:03] ...optimizers/trijoint.py.151: total_norm: 64.564949 [I 2022-10-09 12:24:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 1160/2304 [I 2022-10-09 12:24:11] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:05 [I 2022-10-09 12:24:11] ...trap/engines/engine.py.229: process: 0.84240 | load: 0.00292 [I 2022-10-09 12:24:11] ...trap/engines/engine.py.230: loss: 0.15796 [I 2022-10-09 12:24:11] ...optimizers/trijoint.py.151: total_norm: 18.977678 [I 2022-10-09 12:24:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 1170/2304 [I 2022-10-09 12:24:20] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:15:55 [I 2022-10-09 12:24:20] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00293 [I 2022-10-09 12:24:20] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-09 12:24:20] ...optimizers/trijoint.py.151: total_norm: 16.055889 [I 2022-10-09 12:24:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 1180/2304 [I 2022-10-09 12:24:28] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:15:50 [I 2022-10-09 12:24:28] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00291 [I 2022-10-09 12:24:28] ...trap/engines/engine.py.230: loss: 0.19796 [I 2022-10-09 12:24:28] ...optimizers/trijoint.py.151: total_norm: 13.849062 [I 2022-10-09 12:24:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 1190/2304 [I 2022-10-09 12:24:37] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:40 [I 2022-10-09 12:24:37] ...trap/engines/engine.py.229: process: 0.83945 | load: 0.00295 [I 2022-10-09 12:24:37] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-09 12:24:37] ...optimizers/trijoint.py.151: total_norm: 12.732574 [I 2022-10-09 12:24:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 1200/2304 [I 2022-10-09 12:24:45] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:15:25 [I 2022-10-09 12:24:45] ...trap/engines/engine.py.229: process: 0.82815 | load: 0.00295 [I 2022-10-09 12:24:45] ...trap/engines/engine.py.230: loss: 0.20351 [I 2022-10-09 12:24:45] ...optimizers/trijoint.py.151: total_norm: 18.993223 [I 2022-10-09 12:24:54] ...trap/engines/engine.py.224: train: epoch 40 | batch 1210/2304 [I 2022-10-09 12:24:54] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:25 [I 2022-10-09 12:24:54] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00293 [I 2022-10-09 12:24:54] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-09 12:24:54] ...optimizers/trijoint.py.151: total_norm: 9.698445 [I 2022-10-09 12:25:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 1220/2304 [I 2022-10-09 12:25:02] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:12 [I 2022-10-09 12:25:02] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00292 [I 2022-10-09 12:25:02] ...trap/engines/engine.py.230: loss: 0.18018 [I 2022-10-09 12:25:02] ...optimizers/trijoint.py.151: total_norm: 20.752203 [I 2022-10-09 12:25:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 1230/2304 [I 2022-10-09 12:25:11] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:05 [I 2022-10-09 12:25:11] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00300 [I 2022-10-09 12:25:11] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-09 12:25:11] ...optimizers/trijoint.py.151: total_norm: 9.487484 [I 2022-10-09 12:25:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 1240/2304 [I 2022-10-09 12:25:19] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:14:55 [I 2022-10-09 12:25:19] ...trap/engines/engine.py.229: process: 0.82998 | load: 0.00294 [I 2022-10-09 12:25:19] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-09 12:25:19] ...optimizers/trijoint.py.151: total_norm: 15.470323 [I 2022-10-09 12:25:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 1250/2304 [I 2022-10-09 12:25:27] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:14:43 [I 2022-10-09 12:25:27] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00302 [I 2022-10-09 12:25:27] ...trap/engines/engine.py.230: loss: 0.20822 [I 2022-10-09 12:25:27] ...optimizers/trijoint.py.151: total_norm: 23.833723 [I 2022-10-09 12:25:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 1260/2304 [I 2022-10-09 12:25:36] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:40 [I 2022-10-09 12:25:36] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00293 [I 2022-10-09 12:25:36] ...trap/engines/engine.py.230: loss: 0.17789 [I 2022-10-09 12:25:36] ...optimizers/trijoint.py.151: total_norm: 12.759384 [I 2022-10-09 12:25:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 1270/2304 [I 2022-10-09 12:25:44] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:28 [I 2022-10-09 12:25:44] ...trap/engines/engine.py.229: process: 0.83101 | load: 0.00298 [I 2022-10-09 12:25:44] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-09 12:25:44] ...optimizers/trijoint.py.151: total_norm: 15.140389 [I 2022-10-09 12:25:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 1280/2304 [I 2022-10-09 12:25:53] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:24 [I 2022-10-09 12:25:53] ...trap/engines/engine.py.229: process: 0.84189 | load: 0.00297 [I 2022-10-09 12:25:53] ...trap/engines/engine.py.230: loss: 0.13048 [I 2022-10-09 12:25:53] ...optimizers/trijoint.py.151: total_norm: 14.120207 [I 2022-10-09 12:26:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 1290/2304 [I 2022-10-09 12:26:01] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:11 [I 2022-10-09 12:26:01] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00292 [I 2022-10-09 12:26:01] ...trap/engines/engine.py.230: loss: 0.21745 [I 2022-10-09 12:26:01] ...optimizers/trijoint.py.151: total_norm: 20.218081 [I 2022-10-09 12:26:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 1300/2304 [I 2022-10-09 12:26:10] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:06 [I 2022-10-09 12:26:10] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00294 [I 2022-10-09 12:26:10] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-09 12:26:10] ...optimizers/trijoint.py.151: total_norm: 12.765574 [I 2022-10-09 12:26:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 1310/2304 [I 2022-10-09 12:26:18] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:00 [I 2022-10-09 12:26:18] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00294 [I 2022-10-09 12:26:18] ...trap/engines/engine.py.230: loss: 0.18424 [I 2022-10-09 12:26:18] ...optimizers/trijoint.py.151: total_norm: 18.342787 [I 2022-10-09 12:26:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 1320/2304 [I 2022-10-09 12:26:26] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:13:48 [I 2022-10-09 12:26:26] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00296 [I 2022-10-09 12:26:27] ...trap/engines/engine.py.230: loss: 0.21509 [I 2022-10-09 12:26:27] ...optimizers/trijoint.py.151: total_norm: 11.653272 [I 2022-10-09 12:26:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 1330/2304 [I 2022-10-09 12:26:35] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:38 [I 2022-10-09 12:26:35] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00292 [I 2022-10-09 12:26:35] ...trap/engines/engine.py.230: loss: 0.15650 [I 2022-10-09 12:26:35] ...optimizers/trijoint.py.151: total_norm: 26.680872 [I 2022-10-09 12:26:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 1340/2304 [I 2022-10-09 12:26:43] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:38 [I 2022-10-09 12:26:43] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00292 [I 2022-10-09 12:26:43] ...trap/engines/engine.py.230: loss: 0.20491 [I 2022-10-09 12:26:43] ...optimizers/trijoint.py.151: total_norm: 29.779457 [I 2022-10-09 12:26:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 1350/2304 [I 2022-10-09 12:26:52] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:22 [I 2022-10-09 12:26:52] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00296 [I 2022-10-09 12:26:52] ...trap/engines/engine.py.230: loss: 0.18021 [I 2022-10-09 12:26:52] ...optimizers/trijoint.py.151: total_norm: 17.101065 [I 2022-10-09 12:27:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 1360/2304 [I 2022-10-09 12:27:00] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:21 [I 2022-10-09 12:27:00] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00306 [I 2022-10-09 12:27:00] ...trap/engines/engine.py.230: loss: 0.25040 [I 2022-10-09 12:27:00] ...optimizers/trijoint.py.151: total_norm: 21.135881 [I 2022-10-09 12:27:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 1370/2304 [I 2022-10-09 12:27:09] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:01 [I 2022-10-09 12:27:09] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00298 [I 2022-10-09 12:27:09] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-09 12:27:09] ...optimizers/trijoint.py.151: total_norm: 17.323759 [I 2022-10-09 12:27:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 1380/2304 [I 2022-10-09 12:27:17] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:12:59 [I 2022-10-09 12:27:17] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00297 [I 2022-10-09 12:27:17] ...trap/engines/engine.py.230: loss: 0.12481 [I 2022-10-09 12:27:17] ...optimizers/trijoint.py.151: total_norm: 9.442298 [I 2022-10-09 12:27:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 1390/2304 [I 2022-10-09 12:27:26] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:12:50 [I 2022-10-09 12:27:26] ...trap/engines/engine.py.229: process: 0.85405 | load: 0.00295 [I 2022-10-09 12:27:26] ...trap/engines/engine.py.230: loss: 0.14024 [I 2022-10-09 12:27:26] ...optimizers/trijoint.py.151: total_norm: 10.207928 [I 2022-10-09 12:27:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 1400/2304 [I 2022-10-09 12:27:34] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:12:43 [I 2022-10-09 12:27:34] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00295 [I 2022-10-09 12:27:34] ...trap/engines/engine.py.230: loss: 0.16349 [I 2022-10-09 12:27:34] ...optimizers/trijoint.py.151: total_norm: 13.288292 [I 2022-10-09 12:27:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 1410/2304 [I 2022-10-09 12:27:43] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:12:32 [I 2022-10-09 12:27:43] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00302 [I 2022-10-09 12:27:43] ...trap/engines/engine.py.230: loss: 0.15770 [I 2022-10-09 12:27:43] ...optimizers/trijoint.py.151: total_norm: 24.139347 [I 2022-10-09 12:27:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 1420/2304 [I 2022-10-09 12:27:51] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:25 [I 2022-10-09 12:27:51] ...trap/engines/engine.py.229: process: 0.83588 | load: 0.00297 [I 2022-10-09 12:27:51] ...trap/engines/engine.py.230: loss: 0.15122 [I 2022-10-09 12:27:51] ...optimizers/trijoint.py.151: total_norm: 26.466833 [I 2022-10-09 12:28:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 1430/2304 [I 2022-10-09 12:28:00] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:19 [I 2022-10-09 12:28:00] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00315 [I 2022-10-09 12:28:00] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-09 12:28:00] ...optimizers/trijoint.py.151: total_norm: 11.756763 [I 2022-10-09 12:28:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 1440/2304 [I 2022-10-09 12:28:08] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:09 [I 2022-10-09 12:28:08] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00297 [I 2022-10-09 12:28:08] ...trap/engines/engine.py.230: loss: 0.12779 [I 2022-10-09 12:28:08] ...optimizers/trijoint.py.151: total_norm: 27.866888 [I 2022-10-09 12:28:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 1450/2304 [I 2022-10-09 12:28:16] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:00 [I 2022-10-09 12:28:16] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00299 [I 2022-10-09 12:28:16] ...trap/engines/engine.py.230: loss: 0.15408 [I 2022-10-09 12:28:16] ...optimizers/trijoint.py.151: total_norm: 13.839375 [I 2022-10-09 12:28:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 1460/2304 [I 2022-10-09 12:28:25] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:11:51 [I 2022-10-09 12:28:25] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00295 [I 2022-10-09 12:28:25] ...trap/engines/engine.py.230: loss: 0.12282 [I 2022-10-09 12:28:25] ...optimizers/trijoint.py.151: total_norm: 19.715261 [I 2022-10-09 12:28:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 1470/2304 [I 2022-10-09 12:28:33] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:11:47 [I 2022-10-09 12:28:33] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00294 [I 2022-10-09 12:28:33] ...trap/engines/engine.py.230: loss: 0.10557 [I 2022-10-09 12:28:33] ...optimizers/trijoint.py.151: total_norm: 15.865497 [I 2022-10-09 12:28:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 1480/2304 [I 2022-10-09 12:28:42] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:34 [I 2022-10-09 12:28:42] ...trap/engines/engine.py.229: process: 0.83320 | load: 0.00300 [I 2022-10-09 12:28:42] ...trap/engines/engine.py.230: loss: 0.13647 [I 2022-10-09 12:28:42] ...optimizers/trijoint.py.151: total_norm: 17.978283 [I 2022-10-09 12:28:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 1490/2304 [I 2022-10-09 12:28:50] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:27 [I 2022-10-09 12:28:50] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00294 [I 2022-10-09 12:28:50] ...trap/engines/engine.py.230: loss: 0.14441 [I 2022-10-09 12:28:50] ...optimizers/trijoint.py.151: total_norm: 34.495796 [I 2022-10-09 12:28:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 1500/2304 [I 2022-10-09 12:28:59] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:19 [I 2022-10-09 12:28:59] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00295 [I 2022-10-09 12:28:59] ...trap/engines/engine.py.230: loss: 0.17167 [I 2022-10-09 12:28:59] ...optimizers/trijoint.py.151: total_norm: 22.037260 [I 2022-10-09 12:29:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 1510/2304 [I 2022-10-09 12:29:07] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:11 [I 2022-10-09 12:29:07] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00298 [I 2022-10-09 12:29:07] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-09 12:29:07] ...optimizers/trijoint.py.151: total_norm: 11.856087 [I 2022-10-09 12:29:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 1520/2304 [I 2022-10-09 12:29:16] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:03 [I 2022-10-09 12:29:16] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00299 [I 2022-10-09 12:29:16] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-09 12:29:16] ...optimizers/trijoint.py.151: total_norm: 10.373953 [I 2022-10-09 12:29:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 1530/2304 [I 2022-10-09 12:29:24] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:10:53 [I 2022-10-09 12:29:24] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00292 [I 2022-10-09 12:29:24] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-09 12:29:24] ...optimizers/trijoint.py.151: total_norm: 9.738721 [I 2022-10-09 12:29:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 1540/2304 [I 2022-10-09 12:29:33] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:10:45 [I 2022-10-09 12:29:33] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00295 [I 2022-10-09 12:29:33] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-09 12:29:33] ...optimizers/trijoint.py.151: total_norm: 9.357029 [I 2022-10-09 12:29:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 1550/2304 [I 2022-10-09 12:29:41] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:33 [I 2022-10-09 12:29:41] ...trap/engines/engine.py.229: process: 0.82767 | load: 0.00300 [I 2022-10-09 12:29:41] ...trap/engines/engine.py.230: loss: 0.09751 [I 2022-10-09 12:29:41] ...optimizers/trijoint.py.151: total_norm: 7.512674 [I 2022-10-09 12:29:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 1560/2304 [I 2022-10-09 12:29:49] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:10:28 [I 2022-10-09 12:29:49] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00293 [I 2022-10-09 12:29:50] ...trap/engines/engine.py.230: loss: 0.14075 [I 2022-10-09 12:29:50] ...optimizers/trijoint.py.151: total_norm: 13.626020 [I 2022-10-09 12:29:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 1570/2304 [I 2022-10-09 12:29:58] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:16 [I 2022-10-09 12:29:58] ...trap/engines/engine.py.229: process: 0.82739 | load: 0.00293 [I 2022-10-09 12:29:58] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-09 12:29:58] ...optimizers/trijoint.py.151: total_norm: 7.645281 [I 2022-10-09 12:30:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 1580/2304 [I 2022-10-09 12:30:06] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:06 [I 2022-10-09 12:30:06] ...trap/engines/engine.py.229: process: 0.83152 | load: 0.00296 [I 2022-10-09 12:30:06] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-09 12:30:06] ...optimizers/trijoint.py.151: total_norm: 13.090062 [I 2022-10-09 12:30:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 1590/2304 [I 2022-10-09 12:30:15] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:03 [I 2022-10-09 12:30:15] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00293 [I 2022-10-09 12:30:15] ...trap/engines/engine.py.230: loss: 0.20139 [I 2022-10-09 12:30:15] ...optimizers/trijoint.py.151: total_norm: 26.350529 [I 2022-10-09 12:30:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 1600/2304 [I 2022-10-09 12:30:23] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:09:57 [I 2022-10-09 12:30:23] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00297 [I 2022-10-09 12:30:23] ...trap/engines/engine.py.230: loss: 0.14437 [I 2022-10-09 12:30:23] ...optimizers/trijoint.py.151: total_norm: 14.051622 [I 2022-10-09 12:30:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 1610/2304 [I 2022-10-09 12:30:32] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:09:45 [I 2022-10-09 12:30:32] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00295 [I 2022-10-09 12:30:32] ...trap/engines/engine.py.230: loss: 0.14754 [I 2022-10-09 12:30:32] ...optimizers/trijoint.py.151: total_norm: 8.335894 [I 2022-10-09 12:30:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 1620/2304 [I 2022-10-09 12:30:40] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:33 [I 2022-10-09 12:30:40] ...trap/engines/engine.py.229: process: 0.84091 | load: 0.00295 [I 2022-10-09 12:30:40] ...trap/engines/engine.py.230: loss: 0.17213 [I 2022-10-09 12:30:40] ...optimizers/trijoint.py.151: total_norm: 12.916186 [I 2022-10-09 12:30:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 1630/2304 [I 2022-10-09 12:30:49] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:09:27 [I 2022-10-09 12:30:49] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00302 [I 2022-10-09 12:30:49] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-09 12:30:49] ...optimizers/trijoint.py.151: total_norm: 12.076014 [I 2022-10-09 12:30:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 1640/2304 [I 2022-10-09 12:30:57] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:18 [I 2022-10-09 12:30:57] ...trap/engines/engine.py.229: process: 0.84591 | load: 0.00294 [I 2022-10-09 12:30:57] ...trap/engines/engine.py.230: loss: 0.16545 [I 2022-10-09 12:30:57] ...optimizers/trijoint.py.151: total_norm: 21.140289 [I 2022-10-09 12:31:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 1650/2304 [I 2022-10-09 12:31:05] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:11 [I 2022-10-09 12:31:05] ...trap/engines/engine.py.229: process: 0.82731 | load: 0.00303 [I 2022-10-09 12:31:05] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-09 12:31:05] ...optimizers/trijoint.py.151: total_norm: 7.807024 [I 2022-10-09 12:31:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 1660/2304 [I 2022-10-09 12:31:14] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:03 [I 2022-10-09 12:31:14] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00293 [I 2022-10-09 12:31:14] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-09 12:31:14] ...optimizers/trijoint.py.151: total_norm: 9.734518 [I 2022-10-09 12:31:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 1670/2304 [I 2022-10-09 12:31:22] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:08:55 [I 2022-10-09 12:31:22] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00297 [I 2022-10-09 12:31:22] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-09 12:31:22] ...optimizers/trijoint.py.151: total_norm: 16.979515 [I 2022-10-09 12:31:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 1680/2304 [I 2022-10-09 12:31:31] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:08:45 [I 2022-10-09 12:31:31] ...trap/engines/engine.py.229: process: 0.83411 | load: 0.00308 [I 2022-10-09 12:31:31] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-09 12:31:31] ...optimizers/trijoint.py.151: total_norm: 12.965795 [I 2022-10-09 12:31:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 1690/2304 [I 2022-10-09 12:31:39] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:08:38 [I 2022-10-09 12:31:39] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00293 [I 2022-10-09 12:31:39] ...trap/engines/engine.py.230: loss: 0.22459 [I 2022-10-09 12:31:39] ...optimizers/trijoint.py.151: total_norm: 156.214752 [I 2022-10-09 12:31:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 1700/2304 [I 2022-10-09 12:31:48] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:29 [I 2022-10-09 12:31:48] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00297 [I 2022-10-09 12:31:48] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-09 12:31:48] ...optimizers/trijoint.py.151: total_norm: 15.999030 [I 2022-10-09 12:31:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 1710/2304 [I 2022-10-09 12:31:56] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:24 [I 2022-10-09 12:31:56] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00291 [I 2022-10-09 12:31:56] ...trap/engines/engine.py.230: loss: 0.11555 [I 2022-10-09 12:31:56] ...optimizers/trijoint.py.151: total_norm: 10.982090 [I 2022-10-09 12:32:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 1720/2304 [I 2022-10-09 12:32:05] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:13 [I 2022-10-09 12:32:05] ...trap/engines/engine.py.229: process: 0.83698 | load: 0.00293 [I 2022-10-09 12:32:05] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-09 12:32:05] ...optimizers/trijoint.py.151: total_norm: 10.539139 [I 2022-10-09 12:32:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 1730/2304 [I 2022-10-09 12:32:13] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:07 [I 2022-10-09 12:32:13] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00301 [I 2022-10-09 12:32:13] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-09 12:32:13] ...optimizers/trijoint.py.151: total_norm: 10.619625 [I 2022-10-09 12:32:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 1740/2304 [I 2022-10-09 12:32:22] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:07:57 [I 2022-10-09 12:32:22] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00298 [I 2022-10-09 12:32:22] ...trap/engines/engine.py.230: loss: 0.15679 [I 2022-10-09 12:32:22] ...optimizers/trijoint.py.151: total_norm: 14.501698 [I 2022-10-09 12:32:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 1750/2304 [I 2022-10-09 12:32:30] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:07:49 [I 2022-10-09 12:32:30] ...trap/engines/engine.py.229: process: 0.84015 | load: 0.00293 [I 2022-10-09 12:32:30] ...trap/engines/engine.py.230: loss: 0.20630 [I 2022-10-09 12:32:30] ...optimizers/trijoint.py.151: total_norm: 25.654903 [I 2022-10-09 12:32:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 1760/2304 [I 2022-10-09 12:32:39] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:07:41 [I 2022-10-09 12:32:39] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00294 [I 2022-10-09 12:32:39] ...trap/engines/engine.py.230: loss: 0.11848 [I 2022-10-09 12:32:39] ...optimizers/trijoint.py.151: total_norm: 8.384769 [I 2022-10-09 12:32:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 1770/2304 [I 2022-10-09 12:32:47] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:07:31 [I 2022-10-09 12:32:47] ...trap/engines/engine.py.229: process: 0.83166 | load: 0.00295 [I 2022-10-09 12:32:47] ...trap/engines/engine.py.230: loss: 0.20708 [I 2022-10-09 12:32:47] ...optimizers/trijoint.py.151: total_norm: 24.668955 [I 2022-10-09 12:32:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 1780/2304 [I 2022-10-09 12:32:56] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:24 [I 2022-10-09 12:32:56] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00313 [I 2022-10-09 12:32:56] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-09 12:32:56] ...optimizers/trijoint.py.151: total_norm: 11.011631 [I 2022-10-09 12:33:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1790/2304 [I 2022-10-09 12:33:04] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:16 [I 2022-10-09 12:33:04] ...trap/engines/engine.py.229: process: 0.84240 | load: 0.00293 [I 2022-10-09 12:33:04] ...trap/engines/engine.py.230: loss: 0.16790 [I 2022-10-09 12:33:04] ...optimizers/trijoint.py.151: total_norm: 16.105623 [I 2022-10-09 12:33:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 1800/2304 [I 2022-10-09 12:33:13] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:08 [I 2022-10-09 12:33:13] ...trap/engines/engine.py.229: process: 0.84379 | load: 0.00294 [I 2022-10-09 12:33:13] ...trap/engines/engine.py.230: loss: 0.20045 [I 2022-10-09 12:33:13] ...optimizers/trijoint.py.151: total_norm: 21.035002 [I 2022-10-09 12:33:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1810/2304 [I 2022-10-09 12:33:21] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:01 [I 2022-10-09 12:33:21] ...trap/engines/engine.py.229: process: 0.86020 | load: 0.00297 [I 2022-10-09 12:33:21] ...trap/engines/engine.py.230: loss: 0.19456 [I 2022-10-09 12:33:21] ...optimizers/trijoint.py.151: total_norm: 22.985409 [I 2022-10-09 12:33:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 1820/2304 [I 2022-10-09 12:33:30] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:06:52 [I 2022-10-09 12:33:30] ...trap/engines/engine.py.229: process: 0.86372 | load: 0.00303 [I 2022-10-09 12:33:30] ...trap/engines/engine.py.230: loss: 0.16790 [I 2022-10-09 12:33:30] ...optimizers/trijoint.py.151: total_norm: 22.625648 [I 2022-10-09 12:33:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1830/2304 [I 2022-10-09 12:33:38] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:06:42 [I 2022-10-09 12:33:38] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00298 [I 2022-10-09 12:33:38] ...trap/engines/engine.py.230: loss: 0.14478 [I 2022-10-09 12:33:38] ...optimizers/trijoint.py.151: total_norm: 15.200808 [I 2022-10-09 12:33:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 1840/2304 [I 2022-10-09 12:33:47] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:34 [I 2022-10-09 12:33:47] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00300 [I 2022-10-09 12:33:47] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-09 12:33:47] ...optimizers/trijoint.py.151: total_norm: 19.913464 [I 2022-10-09 12:33:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1850/2304 [I 2022-10-09 12:33:55] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:23 [I 2022-10-09 12:33:55] ...trap/engines/engine.py.229: process: 0.83798 | load: 0.00296 [I 2022-10-09 12:33:55] ...trap/engines/engine.py.230: loss: 0.22814 [I 2022-10-09 12:33:55] ...optimizers/trijoint.py.151: total_norm: 18.598125 [I 2022-10-09 12:34:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1860/2304 [I 2022-10-09 12:34:04] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:17 [I 2022-10-09 12:34:04] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00302 [I 2022-10-09 12:34:04] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-09 12:34:04] ...optimizers/trijoint.py.151: total_norm: 16.268341 [I 2022-10-09 12:34:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 1870/2304 [I 2022-10-09 12:34:12] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:09 [I 2022-10-09 12:34:12] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00298 [I 2022-10-09 12:34:12] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-09 12:34:12] ...optimizers/trijoint.py.151: total_norm: 8.464186 [I 2022-10-09 12:34:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1880/2304 [I 2022-10-09 12:34:21] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:05:59 [I 2022-10-09 12:34:21] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00298 [I 2022-10-09 12:34:21] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-09 12:34:21] ...optimizers/trijoint.py.151: total_norm: 10.426101 [I 2022-10-09 12:34:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 1890/2304 [I 2022-10-09 12:34:29] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:05:52 [I 2022-10-09 12:34:29] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00302 [I 2022-10-09 12:34:29] ...trap/engines/engine.py.230: loss: 0.25714 [I 2022-10-09 12:34:29] ...optimizers/trijoint.py.151: total_norm: 24.396759 [I 2022-10-09 12:34:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1900/2304 [I 2022-10-09 12:34:38] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:05:40 [I 2022-10-09 12:34:38] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00298 [I 2022-10-09 12:34:38] ...trap/engines/engine.py.230: loss: 0.21496 [I 2022-10-09 12:34:38] ...optimizers/trijoint.py.151: total_norm: 19.421988 [I 2022-10-09 12:34:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 1910/2304 [I 2022-10-09 12:34:46] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:34 [I 2022-10-09 12:34:46] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00296 [I 2022-10-09 12:34:46] ...trap/engines/engine.py.230: loss: 0.13804 [I 2022-10-09 12:34:46] ...optimizers/trijoint.py.151: total_norm: 9.476068 [I 2022-10-09 12:34:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1920/2304 [I 2022-10-09 12:34:55] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:26 [I 2022-10-09 12:34:55] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00295 [I 2022-10-09 12:34:55] ...trap/engines/engine.py.230: loss: 0.12571 [I 2022-10-09 12:34:55] ...optimizers/trijoint.py.151: total_norm: 16.345301 [I 2022-10-09 12:35:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 1930/2304 [I 2022-10-09 12:35:03] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:14 [I 2022-10-09 12:35:03] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00296 [I 2022-10-09 12:35:03] ...trap/engines/engine.py.230: loss: 0.10019 [I 2022-10-09 12:35:03] ...optimizers/trijoint.py.151: total_norm: 10.510894 [I 2022-10-09 12:35:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 1940/2304 [I 2022-10-09 12:35:12] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:08 [I 2022-10-09 12:35:12] ...trap/engines/engine.py.229: process: 0.85973 | load: 0.00296 [I 2022-10-09 12:35:12] ...trap/engines/engine.py.230: loss: 0.17472 [I 2022-10-09 12:35:12] ...optimizers/trijoint.py.151: total_norm: 11.570196 [I 2022-10-09 12:35:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 1950/2304 [I 2022-10-09 12:35:20] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:04:58 [I 2022-10-09 12:35:20] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00297 [I 2022-10-09 12:35:20] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-09 12:35:20] ...optimizers/trijoint.py.151: total_norm: 9.211110 [I 2022-10-09 12:35:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 1960/2304 [I 2022-10-09 12:35:29] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:04:49 [I 2022-10-09 12:35:29] ...trap/engines/engine.py.229: process: 0.82836 | load: 0.00291 [I 2022-10-09 12:35:29] ...trap/engines/engine.py.230: loss: 0.23359 [I 2022-10-09 12:35:29] ...optimizers/trijoint.py.151: total_norm: 32.286522 [I 2022-10-09 12:35:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 1970/2304 [I 2022-10-09 12:35:37] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:04:40 [I 2022-10-09 12:35:37] ...trap/engines/engine.py.229: process: 0.84563 | load: 0.00301 [I 2022-10-09 12:35:37] ...trap/engines/engine.py.230: loss: 0.14220 [I 2022-10-09 12:35:37] ...optimizers/trijoint.py.151: total_norm: 23.281998 [I 2022-10-09 12:35:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 1980/2304 [I 2022-10-09 12:35:46] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:33 [I 2022-10-09 12:35:46] ...trap/engines/engine.py.229: process: 0.84341 | load: 0.00297 [I 2022-10-09 12:35:46] ...trap/engines/engine.py.230: loss: 0.14038 [I 2022-10-09 12:35:46] ...optimizers/trijoint.py.151: total_norm: 14.421837 [I 2022-10-09 12:35:54] ...trap/engines/engine.py.224: train: epoch 40 | batch 1990/2304 [I 2022-10-09 12:35:54] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:04:27 [I 2022-10-09 12:35:54] ...trap/engines/engine.py.229: process: 0.86460 | load: 0.00298 [I 2022-10-09 12:35:54] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-09 12:35:54] ...optimizers/trijoint.py.151: total_norm: 12.353299 [I 2022-10-09 12:36:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 2000/2304 [I 2022-10-09 12:36:02] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:17 [I 2022-10-09 12:36:02] ...trap/engines/engine.py.229: process: 0.85592 | load: 0.00299 [I 2022-10-09 12:36:02] ...trap/engines/engine.py.230: loss: 0.11350 [I 2022-10-09 12:36:02] ...optimizers/trijoint.py.151: total_norm: 14.608223 [I 2022-10-09 12:36:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 2010/2304 [I 2022-10-09 12:36:11] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:09 [I 2022-10-09 12:36:11] ...trap/engines/engine.py.229: process: 0.84870 | load: 0.00302 [I 2022-10-09 12:36:11] ...trap/engines/engine.py.230: loss: 0.18896 [I 2022-10-09 12:36:11] ...optimizers/trijoint.py.151: total_norm: 17.063828 [I 2022-10-09 12:36:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 2020/2304 [I 2022-10-09 12:36:19] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:03:59 [I 2022-10-09 12:36:19] ...trap/engines/engine.py.229: process: 0.85550 | load: 0.00300 [I 2022-10-09 12:36:19] ...trap/engines/engine.py.230: loss: 0.14197 [I 2022-10-09 12:36:19] ...optimizers/trijoint.py.151: total_norm: 14.191121 [I 2022-10-09 12:36:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 2030/2304 [I 2022-10-09 12:36:28] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:03:52 [I 2022-10-09 12:36:28] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00294 [I 2022-10-09 12:36:28] ...trap/engines/engine.py.230: loss: 0.20122 [I 2022-10-09 12:36:28] ...optimizers/trijoint.py.151: total_norm: 17.899330 [I 2022-10-09 12:36:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 2040/2304 [I 2022-10-09 12:36:36] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:03:42 [I 2022-10-09 12:36:36] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00301 [I 2022-10-09 12:36:36] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-09 12:36:36] ...optimizers/trijoint.py.151: total_norm: 20.365917 [I 2022-10-09 12:36:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 2050/2304 [I 2022-10-09 12:36:45] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:03:33 [I 2022-10-09 12:36:45] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00292 [I 2022-10-09 12:36:45] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-09 12:36:45] ...optimizers/trijoint.py.151: total_norm: 14.857761 [I 2022-10-09 12:36:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 2060/2304 [I 2022-10-09 12:36:53] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:26 [I 2022-10-09 12:36:53] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00297 [I 2022-10-09 12:36:53] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-09 12:36:53] ...optimizers/trijoint.py.151: total_norm: 10.566610 [I 2022-10-09 12:37:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 2070/2304 [I 2022-10-09 12:37:02] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:17 [I 2022-10-09 12:37:02] ...trap/engines/engine.py.229: process: 0.83631 | load: 0.00295 [I 2022-10-09 12:37:02] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-09 12:37:02] ...optimizers/trijoint.py.151: total_norm: 12.772153 [I 2022-10-09 12:37:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 2080/2304 [I 2022-10-09 12:37:10] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:08 [I 2022-10-09 12:37:10] ...trap/engines/engine.py.229: process: 0.84823 | load: 0.00308 [I 2022-10-09 12:37:10] ...trap/engines/engine.py.230: loss: 0.13271 [I 2022-10-09 12:37:10] ...optimizers/trijoint.py.151: total_norm: 20.334410 [I 2022-10-09 12:37:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 2090/2304 [I 2022-10-09 12:37:19] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:01 [I 2022-10-09 12:37:19] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00300 [I 2022-10-09 12:37:19] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-09 12:37:19] ...optimizers/trijoint.py.151: total_norm: 37.818546 [I 2022-10-09 12:37:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 2100/2304 [I 2022-10-09 12:37:27] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:02:52 [I 2022-10-09 12:37:27] ...trap/engines/engine.py.229: process: 0.83798 | load: 0.00302 [I 2022-10-09 12:37:27] ...trap/engines/engine.py.230: loss: 0.13250 [I 2022-10-09 12:37:27] ...optimizers/trijoint.py.151: total_norm: 11.353590 [I 2022-10-09 12:37:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 2110/2304 [I 2022-10-09 12:37:35] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:02:43 [I 2022-10-09 12:37:35] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00295 [I 2022-10-09 12:37:35] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-09 12:37:35] ...optimizers/trijoint.py.151: total_norm: 11.043464 [I 2022-10-09 12:37:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 2120/2304 [I 2022-10-09 12:37:44] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:02:34 [I 2022-10-09 12:37:44] ...trap/engines/engine.py.229: process: 0.83661 | load: 0.00297 [I 2022-10-09 12:37:44] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-09 12:37:44] ...optimizers/trijoint.py.151: total_norm: 15.811376 [I 2022-10-09 12:37:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 2130/2304 [I 2022-10-09 12:37:52] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:26 [I 2022-10-09 12:37:52] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00304 [I 2022-10-09 12:37:52] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-09 12:37:52] ...optimizers/trijoint.py.151: total_norm: 29.663361 [I 2022-10-09 12:38:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 2140/2304 [I 2022-10-09 12:38:01] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:17 [I 2022-10-09 12:38:01] ...trap/engines/engine.py.229: process: 0.82944 | load: 0.00298 [I 2022-10-09 12:38:01] ...trap/engines/engine.py.230: loss: 0.13615 [I 2022-10-09 12:38:01] ...optimizers/trijoint.py.151: total_norm: 31.721643 [I 2022-10-09 12:38:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 2150/2304 [I 2022-10-09 12:38:09] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:09 [I 2022-10-09 12:38:09] ...trap/engines/engine.py.229: process: 0.84461 | load: 0.00295 [I 2022-10-09 12:38:09] ...trap/engines/engine.py.230: loss: 0.21787 [I 2022-10-09 12:38:09] ...optimizers/trijoint.py.151: total_norm: 24.529648 [I 2022-10-09 12:38:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 2160/2304 [I 2022-10-09 12:38:18] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:00 [I 2022-10-09 12:38:18] ...trap/engines/engine.py.229: process: 0.83067 | load: 0.00295 [I 2022-10-09 12:38:18] ...trap/engines/engine.py.230: loss: 0.10241 [I 2022-10-09 12:38:18] ...optimizers/trijoint.py.151: total_norm: 17.102030 [I 2022-10-09 12:38:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 2170/2304 [I 2022-10-09 12:38:26] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:01:53 [I 2022-10-09 12:38:26] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00295 [I 2022-10-09 12:38:26] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-09 12:38:26] ...optimizers/trijoint.py.151: total_norm: 26.924448 [I 2022-10-09 12:38:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 2180/2304 [I 2022-10-09 12:38:34] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:01:44 [I 2022-10-09 12:38:35] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00303 [I 2022-10-09 12:38:35] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-09 12:38:35] ...optimizers/trijoint.py.151: total_norm: 21.903872 [I 2022-10-09 12:38:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 2190/2304 [I 2022-10-09 12:38:43] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:35 [I 2022-10-09 12:38:43] ...trap/engines/engine.py.229: process: 0.83256 | load: 0.00293 [I 2022-10-09 12:38:43] ...trap/engines/engine.py.230: loss: 0.27142 [I 2022-10-09 12:38:43] ...optimizers/trijoint.py.151: total_norm: 44.043816 [I 2022-10-09 12:38:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 2200/2304 [I 2022-10-09 12:38:51] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:27 [I 2022-10-09 12:38:51] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00293 [I 2022-10-09 12:38:51] ...trap/engines/engine.py.230: loss: 0.11712 [I 2022-10-09 12:38:51] ...optimizers/trijoint.py.151: total_norm: 18.570175 [I 2022-10-09 12:39:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 2210/2304 [I 2022-10-09 12:39:00] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:18 [I 2022-10-09 12:39:00] ...trap/engines/engine.py.229: process: 0.82906 | load: 0.00297 [I 2022-10-09 12:39:00] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-09 12:39:00] ...optimizers/trijoint.py.151: total_norm: 18.592428 [I 2022-10-09 12:39:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 2220/2304 [I 2022-10-09 12:39:08] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:11 [I 2022-10-09 12:39:08] ...trap/engines/engine.py.229: process: 0.85628 | load: 0.00298 [I 2022-10-09 12:39:08] ...trap/engines/engine.py.230: loss: 0.20004 [I 2022-10-09 12:39:08] ...optimizers/trijoint.py.151: total_norm: 19.940813 [I 2022-10-09 12:39:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 2230/2304 [I 2022-10-09 12:39:17] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:02 [I 2022-10-09 12:39:17] ...trap/engines/engine.py.229: process: 0.83056 | load: 0.00296 [I 2022-10-09 12:39:17] ...trap/engines/engine.py.230: loss: 0.16950 [I 2022-10-09 12:39:17] ...optimizers/trijoint.py.151: total_norm: 20.319538 [I 2022-10-09 12:39:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 2240/2304 [I 2022-10-09 12:39:25] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:00:54 [I 2022-10-09 12:39:25] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00302 [I 2022-10-09 12:39:25] ...trap/engines/engine.py.230: loss: 0.19922 [I 2022-10-09 12:39:25] ...optimizers/trijoint.py.151: total_norm: 55.066948 [I 2022-10-09 12:39:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 2250/2304 [I 2022-10-09 12:39:34] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:00:45 [I 2022-10-09 12:39:34] ...trap/engines/engine.py.229: process: 0.82255 | load: 0.00293 [I 2022-10-09 12:39:34] ...trap/engines/engine.py.230: loss: 0.11290 [I 2022-10-09 12:39:34] ...optimizers/trijoint.py.151: total_norm: 10.349745 [I 2022-10-09 12:39:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 2260/2304 [I 2022-10-09 12:39:42] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:00:36 [I 2022-10-09 12:39:42] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00299 [I 2022-10-09 12:39:42] ...trap/engines/engine.py.230: loss: 0.16348 [I 2022-10-09 12:39:42] ...optimizers/trijoint.py.151: total_norm: 12.847721 [I 2022-10-09 12:39:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 2270/2304 [I 2022-10-09 12:39:50] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:28 [I 2022-10-09 12:39:50] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00299 [I 2022-10-09 12:39:50] ...trap/engines/engine.py.230: loss: 0.14011 [I 2022-10-09 12:39:50] ...optimizers/trijoint.py.151: total_norm: 13.325454 [I 2022-10-09 12:39:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 2280/2304 [I 2022-10-09 12:39:59] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:00:20 [I 2022-10-09 12:39:59] ...trap/engines/engine.py.229: process: 0.84418 | load: 0.00300 [I 2022-10-09 12:39:59] ...trap/engines/engine.py.230: loss: 0.19297 [I 2022-10-09 12:39:59] ...optimizers/trijoint.py.151: total_norm: 19.714886 [I 2022-10-09 12:40:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 2290/2304 [I 2022-10-09 12:40:07] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:11 [I 2022-10-09 12:40:07] ...trap/engines/engine.py.229: process: 0.83889 | load: 0.00293 [I 2022-10-09 12:40:07] ...trap/engines/engine.py.230: loss: 0.17894 [I 2022-10-09 12:40:07] ...optimizers/trijoint.py.151: total_norm: 15.567647 [I 2022-10-09 12:40:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 2300/2304 [I 2022-10-09 12:40:16] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:03 [I 2022-10-09 12:40:16] ...trap/engines/engine.py.229: process: 0.82788 | load: 0.00015 [I 2022-10-09 12:40:16] ...trap/engines/engine.py.230: loss: 0.11337 [I 2022-10-09 12:40:16] ...optimizers/trijoint.py.151: total_norm: 13.316070 [I 2022-10-09 12:40:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 2304/2304 [I 2022-10-09 12:40:19] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:00 [I 2022-10-09 12:40:19] ...trap/engines/engine.py.229: process: 0.82906 | load: 0.00016 [I 2022-10-09 12:40:19] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-09 12:40:19] ...optimizers/trijoint.py.151: total_norm: 8.511747 [S 2022-10-09 12:40:20] ...trap/engines/engine.py.129: train_epoch.epoch: 40 [S 2022-10-09 12:40:20] ...trap/engines/engine.py.129: train_epoch.loss: 0.15880039905131252 [I 2022-10-09 12:40:20] ...s/criterions/triplet.py.84: margin = 0.25000000000000006 [I 2022-10-09 12:40:21] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 40 [I 2022-10-09 12:40:21] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 12:40:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 12:40:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 12:40:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 12:40:24] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 12:40:24] ...trap/engines/engine.py.321: eval: epoch 40 | batch 0/510 [I 2022-10-09 12:40:24] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:24 [I 2022-10-09 12:40:24] ...trap/engines/engine.py.326: process: 3.45925 | load: 2.95748 [I 2022-10-09 12:40:30] ...trap/engines/engine.py.321: eval: epoch 40 | batch 10/510 [I 2022-10-09 12:40:30] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:45 [I 2022-10-09 12:40:30] ...trap/engines/engine.py.326: process: 0.55507 | load: 0.00018 [I 2022-10-09 12:40:37] ...trap/engines/engine.py.321: eval: epoch 40 | batch 20/510 [I 2022-10-09 12:40:37] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:42 [I 2022-10-09 12:40:37] ...trap/engines/engine.py.326: process: 0.73721 | load: 0.00021 [I 2022-10-09 12:40:44] ...trap/engines/engine.py.321: eval: epoch 40 | batch 30/510 [I 2022-10-09 12:40:44] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:09 [I 2022-10-09 12:40:44] ...trap/engines/engine.py.326: process: 0.51518 | load: 0.00018 [I 2022-10-09 12:40:52] ...trap/engines/engine.py.321: eval: epoch 40 | batch 40/510 [I 2022-10-09 12:40:52] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:06:11 [I 2022-10-09 12:40:52] ...trap/engines/engine.py.326: process: 0.56854 | load: 0.00018 [I 2022-10-09 12:40:57] ...trap/engines/engine.py.321: eval: epoch 40 | batch 50/510 [I 2022-10-09 12:40:57] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:11 [I 2022-10-09 12:40:57] ...trap/engines/engine.py.326: process: 0.54367 | load: 0.00018 [I 2022-10-09 12:41:02] ...trap/engines/engine.py.321: eval: epoch 40 | batch 60/510 [I 2022-10-09 12:41:02] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:00 [I 2022-10-09 12:41:02] ...trap/engines/engine.py.326: process: 0.58155 | load: 0.00018 [I 2022-10-09 12:41:08] ...trap/engines/engine.py.321: eval: epoch 40 | batch 70/510 [I 2022-10-09 12:41:08] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:02 [I 2022-10-09 12:41:08] ...trap/engines/engine.py.326: process: 0.52102 | load: 0.00017 [I 2022-10-09 12:41:13] ...trap/engines/engine.py.321: eval: epoch 40 | batch 80/510 [I 2022-10-09 12:41:13] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:35 [I 2022-10-09 12:41:13] ...trap/engines/engine.py.326: process: 0.53384 | load: 0.00017 [I 2022-10-09 12:41:18] ...trap/engines/engine.py.321: eval: epoch 40 | batch 90/510 [I 2022-10-09 12:41:18] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:28 [I 2022-10-09 12:41:18] ...trap/engines/engine.py.326: process: 0.49796 | load: 0.00018 [I 2022-10-09 12:41:22] ...trap/engines/engine.py.321: eval: epoch 40 | batch 100/510 [I 2022-10-09 12:41:22] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:03:00 [I 2022-10-09 12:41:22] ...trap/engines/engine.py.326: process: 0.30469 | load: 0.00019 [I 2022-10-09 12:41:26] ...trap/engines/engine.py.321: eval: epoch 40 | batch 110/510 [I 2022-10-09 12:41:26] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:11 [I 2022-10-09 12:41:26] ...trap/engines/engine.py.326: process: 0.30933 | load: 0.00017 [I 2022-10-09 12:41:29] ...trap/engines/engine.py.321: eval: epoch 40 | batch 120/510 [I 2022-10-09 12:41:29] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:03 [I 2022-10-09 12:41:29] ...trap/engines/engine.py.326: process: 0.31284 | load: 0.00019 [I 2022-10-09 12:41:32] ...trap/engines/engine.py.321: eval: epoch 40 | batch 130/510 [I 2022-10-09 12:41:32] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:59 [I 2022-10-09 12:41:32] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00019 [I 2022-10-09 12:41:35] ...trap/engines/engine.py.321: eval: epoch 40 | batch 140/510 [I 2022-10-09 12:41:35] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:56 [I 2022-10-09 12:41:35] ...trap/engines/engine.py.326: process: 0.31534 | load: 0.00020 [I 2022-10-09 12:41:38] ...trap/engines/engine.py.321: eval: epoch 40 | batch 150/510 [I 2022-10-09 12:41:38] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:53 [I 2022-10-09 12:41:38] ...trap/engines/engine.py.326: process: 0.31112 | load: 0.00020 [I 2022-10-09 12:41:41] ...trap/engines/engine.py.321: eval: epoch 40 | batch 160/510 [I 2022-10-09 12:41:41] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:50 [I 2022-10-09 12:41:41] ...trap/engines/engine.py.326: process: 0.31101 | load: 0.00018 [I 2022-10-09 12:41:45] ...trap/engines/engine.py.321: eval: epoch 40 | batch 170/510 [I 2022-10-09 12:41:45] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:47 [I 2022-10-09 12:41:45] ...trap/engines/engine.py.326: process: 0.31677 | load: 0.00019 [I 2022-10-09 12:41:48] ...trap/engines/engine.py.321: eval: epoch 40 | batch 180/510 [I 2022-10-09 12:41:48] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:45 [I 2022-10-09 12:41:48] ...trap/engines/engine.py.326: process: 0.32057 | load: 0.00020 [I 2022-10-09 12:41:51] ...trap/engines/engine.py.321: eval: epoch 40 | batch 190/510 [I 2022-10-09 12:41:51] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:41 [I 2022-10-09 12:41:51] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00019 [I 2022-10-09 12:41:54] ...trap/engines/engine.py.321: eval: epoch 40 | batch 200/510 [I 2022-10-09 12:41:54] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:38 [I 2022-10-09 12:41:54] ...trap/engines/engine.py.326: process: 0.31879 | load: 0.00020 [I 2022-10-09 12:41:57] ...trap/engines/engine.py.321: eval: epoch 40 | batch 210/510 [I 2022-10-09 12:41:57] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:35 [I 2022-10-09 12:41:57] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00018 [I 2022-10-09 12:42:01] ...trap/engines/engine.py.321: eval: epoch 40 | batch 220/510 [I 2022-10-09 12:42:01] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:32 [I 2022-10-09 12:42:01] ...trap/engines/engine.py.326: process: 0.32190 | load: 0.00020 [I 2022-10-09 12:42:04] ...trap/engines/engine.py.321: eval: epoch 40 | batch 230/510 [I 2022-10-09 12:42:04] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:29 [I 2022-10-09 12:42:04] ...trap/engines/engine.py.326: process: 0.32700 | load: 0.00019 [I 2022-10-09 12:42:07] ...trap/engines/engine.py.321: eval: epoch 40 | batch 240/510 [I 2022-10-09 12:42:07] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:26 [I 2022-10-09 12:42:07] ...trap/engines/engine.py.326: process: 0.32102 | load: 0.00020 [I 2022-10-09 12:42:10] ...trap/engines/engine.py.321: eval: epoch 40 | batch 250/510 [I 2022-10-09 12:42:10] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:22 [I 2022-10-09 12:42:10] ...trap/engines/engine.py.326: process: 0.31902 | load: 0.00019 [I 2022-10-09 12:42:13] ...trap/engines/engine.py.321: eval: epoch 40 | batch 260/510 [I 2022-10-09 12:42:13] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:19 [I 2022-10-09 12:42:13] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-09 12:42:16] ...trap/engines/engine.py.321: eval: epoch 40 | batch 270/510 [I 2022-10-09 12:42:16] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:16 [I 2022-10-09 12:42:16] ...trap/engines/engine.py.326: process: 0.31878 | load: 0.00018 [I 2022-10-09 12:42:20] ...trap/engines/engine.py.321: eval: epoch 40 | batch 280/510 [I 2022-10-09 12:42:20] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:13 [I 2022-10-09 12:42:20] ...trap/engines/engine.py.326: process: 0.32212 | load: 0.00020 [I 2022-10-09 12:42:23] ...trap/engines/engine.py.321: eval: epoch 40 | batch 290/510 [I 2022-10-09 12:42:23] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:09 [I 2022-10-09 12:42:23] ...trap/engines/engine.py.326: process: 0.31356 | load: 0.00018 [I 2022-10-09 12:42:26] ...trap/engines/engine.py.321: eval: epoch 40 | batch 300/510 [I 2022-10-09 12:42:26] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:07 [I 2022-10-09 12:42:26] ...trap/engines/engine.py.326: process: 0.32003 | load: 0.00019 [I 2022-10-09 12:42:29] ...trap/engines/engine.py.321: eval: epoch 40 | batch 310/510 [I 2022-10-09 12:42:29] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:03 [I 2022-10-09 12:42:29] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00019 [I 2022-10-09 12:42:32] ...trap/engines/engine.py.321: eval: epoch 40 | batch 320/510 [I 2022-10-09 12:42:32] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:00 [I 2022-10-09 12:42:32] ...trap/engines/engine.py.326: process: 0.32439 | load: 0.00018 [I 2022-10-09 12:42:36] ...trap/engines/engine.py.321: eval: epoch 40 | batch 330/510 [I 2022-10-09 12:42:36] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:57 [I 2022-10-09 12:42:36] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00019 [I 2022-10-09 12:42:39] ...trap/engines/engine.py.321: eval: epoch 40 | batch 340/510 [I 2022-10-09 12:42:39] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:53 [I 2022-10-09 12:42:39] ...trap/engines/engine.py.326: process: 0.31207 | load: 0.00019 [I 2022-10-09 12:42:42] ...trap/engines/engine.py.321: eval: epoch 40 | batch 350/510 [I 2022-10-09 12:42:42] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:50 [I 2022-10-09 12:42:42] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00019 [I 2022-10-09 12:42:45] ...trap/engines/engine.py.321: eval: epoch 40 | batch 360/510 [I 2022-10-09 12:42:45] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:47 [I 2022-10-09 12:42:45] ...trap/engines/engine.py.326: process: 0.31003 | load: 0.00019 [I 2022-10-09 12:42:48] ...trap/engines/engine.py.321: eval: epoch 40 | batch 370/510 [I 2022-10-09 12:42:48] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:44 [I 2022-10-09 12:42:48] ...trap/engines/engine.py.326: process: 0.31895 | load: 0.00018 [I 2022-10-09 12:42:51] ...trap/engines/engine.py.321: eval: epoch 40 | batch 380/510 [I 2022-10-09 12:42:51] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:41 [I 2022-10-09 12:42:51] ...trap/engines/engine.py.326: process: 0.31455 | load: 0.00021 [I 2022-10-09 12:42:55] ...trap/engines/engine.py.321: eval: epoch 40 | batch 390/510 [I 2022-10-09 12:42:55] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:38 [I 2022-10-09 12:42:55] ...trap/engines/engine.py.326: process: 0.31789 | load: 0.00018 [I 2022-10-09 12:42:58] ...trap/engines/engine.py.321: eval: epoch 40 | batch 400/510 [I 2022-10-09 12:42:58] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:35 [I 2022-10-09 12:42:58] ...trap/engines/engine.py.326: process: 0.32009 | load: 0.00020 [I 2022-10-09 12:43:01] ...trap/engines/engine.py.321: eval: epoch 40 | batch 410/510 [I 2022-10-09 12:43:01] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:31 [I 2022-10-09 12:43:01] ...trap/engines/engine.py.326: process: 0.32243 | load: 0.00019 [I 2022-10-09 12:43:04] ...trap/engines/engine.py.321: eval: epoch 40 | batch 420/510 [I 2022-10-09 12:43:04] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:28 [I 2022-10-09 12:43:04] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00020 [I 2022-10-09 12:43:07] ...trap/engines/engine.py.321: eval: epoch 40 | batch 430/510 [I 2022-10-09 12:43:07] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:25 [I 2022-10-09 12:43:07] ...trap/engines/engine.py.326: process: 0.31575 | load: 0.00019 [I 2022-10-09 12:43:11] ...trap/engines/engine.py.321: eval: epoch 40 | batch 440/510 [I 2022-10-09 12:43:11] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:22 [I 2022-10-09 12:43:11] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00019 [I 2022-10-09 12:43:14] ...trap/engines/engine.py.321: eval: epoch 40 | batch 450/510 [I 2022-10-09 12:43:14] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:19 [I 2022-10-09 12:43:14] ...trap/engines/engine.py.326: process: 0.31552 | load: 0.00021 [I 2022-10-09 12:43:17] ...trap/engines/engine.py.321: eval: epoch 40 | batch 460/510 [I 2022-10-09 12:43:17] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:15 [I 2022-10-09 12:43:17] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00019 [I 2022-10-09 12:43:20] ...trap/engines/engine.py.321: eval: epoch 40 | batch 470/510 [I 2022-10-09 12:43:20] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:12 [I 2022-10-09 12:43:20] ...trap/engines/engine.py.326: process: 0.31501 | load: 0.00019 [I 2022-10-09 12:43:23] ...trap/engines/engine.py.321: eval: epoch 40 | batch 480/510 [I 2022-10-09 12:43:23] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:09 [I 2022-10-09 12:43:23] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00019 [I 2022-10-09 12:43:26] ...trap/engines/engine.py.321: eval: epoch 40 | batch 490/510 [I 2022-10-09 12:43:26] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:06 [I 2022-10-09 12:43:26] ...trap/engines/engine.py.326: process: 0.31439 | load: 0.00020 [I 2022-10-09 12:43:30] ...trap/engines/engine.py.321: eval: epoch 40 | batch 500/510 [I 2022-10-09 12:43:30] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:03 [I 2022-10-09 12:43:30] ...trap/engines/engine.py.326: process: 0.31011 | load: 0.00019 [I 2022-10-09 12:43:33] ...trap/engines/engine.py.321: eval: epoch 40 | batch 510/510 [I 2022-10-09 12:43:33] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:00 [I 2022-10-09 12:43:33] ...trap/engines/engine.py.326: process: 0.31387 | load: 0.00015 [S 2022-10-09 12:43:33] ...trap/engines/engine.py.132: eval_epoch.epoch: 40 [I 2022-10-09 12:43:33] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.709500 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905300 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937900 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.699600 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906500 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937000 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015028 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008821 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009181 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011672 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006946 [S 2022-10-09 12:43:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008295 [I 2022-10-09 12:44:11] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 12:44:11] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 12:44:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 12:44:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 12:44:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 12:44:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 12:44:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 12:44:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 12:44:21] ...trap/engines/engine.py.163: Training model on trainset for epoch 41 [I 2022-10-09 12:44:21] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 12:44:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 0/2304 [I 2022-10-09 12:44:23] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:40 [I 2022-10-09 12:44:23] ...trap/engines/engine.py.229: process: 2.85596 | load: 1.92524 [I 2022-10-09 12:44:23] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-09 12:44:23] ...optimizers/trijoint.py.151: total_norm: 8.870871 [I 2022-10-09 12:44:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 10/2304 [I 2022-10-09 12:44:32] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:38 [I 2022-10-09 12:44:32] ...trap/engines/engine.py.229: process: 0.83414 | load: 0.00608 [I 2022-10-09 12:44:32] ...trap/engines/engine.py.230: loss: 0.13551 [I 2022-10-09 12:44:32] ...optimizers/trijoint.py.151: total_norm: 20.526197 [I 2022-10-09 12:44:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 20/2304 [I 2022-10-09 12:44:40] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:12 [I 2022-10-09 12:44:40] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00449 [I 2022-10-09 12:44:40] ...trap/engines/engine.py.230: loss: 0.19727 [I 2022-10-09 12:44:40] ...optimizers/trijoint.py.151: total_norm: 11.752218 [I 2022-10-09 12:44:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 30/2304 [I 2022-10-09 12:44:49] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:59 [I 2022-10-09 12:44:49] ...trap/engines/engine.py.229: process: 0.83833 | load: 0.00383 [I 2022-10-09 12:44:49] ...trap/engines/engine.py.230: loss: 0.18775 [I 2022-10-09 12:44:49] ...optimizers/trijoint.py.151: total_norm: 15.410764 [I 2022-10-09 12:44:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 40/2304 [I 2022-10-09 12:44:57] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:51 [I 2022-10-09 12:44:57] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00347 [I 2022-10-09 12:44:57] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-09 12:44:57] ...optimizers/trijoint.py.151: total_norm: 10.347823 [I 2022-10-09 12:45:06] ...trap/engines/engine.py.224: train: epoch 41 | batch 50/2304 [I 2022-10-09 12:45:06] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:01 [I 2022-10-09 12:45:06] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00322 [I 2022-10-09 12:45:06] ...trap/engines/engine.py.230: loss: 0.11254 [I 2022-10-09 12:45:06] ...optimizers/trijoint.py.151: total_norm: 25.324303 [I 2022-10-09 12:45:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 60/2304 [I 2022-10-09 12:45:14] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:38 [I 2022-10-09 12:45:14] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00351 [I 2022-10-09 12:45:14] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-09 12:45:14] ...optimizers/trijoint.py.151: total_norm: 19.002811 [I 2022-10-09 12:45:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 70/2304 [I 2022-10-09 12:45:23] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:29 [I 2022-10-09 12:45:23] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00352 [I 2022-10-09 12:45:23] ...trap/engines/engine.py.230: loss: 0.14715 [I 2022-10-09 12:45:23] ...optimizers/trijoint.py.151: total_norm: 14.437471 [I 2022-10-09 12:45:31] ...trap/engines/engine.py.224: train: epoch 41 | batch 80/2304 [I 2022-10-09 12:45:31] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:10 [I 2022-10-09 12:45:31] ...trap/engines/engine.py.229: process: 0.83778 | load: 0.00356 [I 2022-10-09 12:45:31] ...trap/engines/engine.py.230: loss: 0.16861 [I 2022-10-09 12:45:31] ...optimizers/trijoint.py.151: total_norm: 42.802216 [I 2022-10-09 12:45:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 90/2304 [I 2022-10-09 12:45:40] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:13 [I 2022-10-09 12:45:40] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00327 [I 2022-10-09 12:45:40] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-09 12:45:40] ...optimizers/trijoint.py.151: total_norm: 24.292097 [I 2022-10-09 12:45:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 100/2304 [I 2022-10-09 12:45:48] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:11 [I 2022-10-09 12:45:48] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00306 [I 2022-10-09 12:45:48] ...trap/engines/engine.py.230: loss: 0.21206 [I 2022-10-09 12:45:48] ...optimizers/trijoint.py.151: total_norm: 33.006355 [I 2022-10-09 12:45:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 110/2304 [I 2022-10-09 12:45:57] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:50 [I 2022-10-09 12:45:57] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00310 [I 2022-10-09 12:45:57] ...trap/engines/engine.py.230: loss: 0.23827 [I 2022-10-09 12:45:57] ...optimizers/trijoint.py.151: total_norm: 19.162733 [I 2022-10-09 12:46:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 120/2304 [I 2022-10-09 12:46:05] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:38 [I 2022-10-09 12:46:05] ...trap/engines/engine.py.229: process: 0.83349 | load: 0.00336 [I 2022-10-09 12:46:05] ...trap/engines/engine.py.230: loss: 0.21275 [I 2022-10-09 12:46:05] ...optimizers/trijoint.py.151: total_norm: 15.028241 [I 2022-10-09 12:46:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 130/2304 [I 2022-10-09 12:46:14] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:39 [I 2022-10-09 12:46:14] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00313 [I 2022-10-09 12:46:14] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-09 12:46:14] ...optimizers/trijoint.py.151: total_norm: 13.288104 [I 2022-10-09 12:46:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 140/2304 [I 2022-10-09 12:46:22] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:25 [I 2022-10-09 12:46:22] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00299 [I 2022-10-09 12:46:22] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-09 12:46:22] ...optimizers/trijoint.py.151: total_norm: 6.827307 [I 2022-10-09 12:46:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 150/2304 [I 2022-10-09 12:46:30] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:16 [I 2022-10-09 12:46:30] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00304 [I 2022-10-09 12:46:30] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-09 12:46:30] ...optimizers/trijoint.py.151: total_norm: 11.505402 [I 2022-10-09 12:46:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 160/2304 [I 2022-10-09 12:46:39] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:02 [I 2022-10-09 12:46:39] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00335 [I 2022-10-09 12:46:39] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-09 12:46:39] ...optimizers/trijoint.py.151: total_norm: 16.036625 [I 2022-10-09 12:46:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 170/2304 [I 2022-10-09 12:46:47] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:55 [I 2022-10-09 12:46:47] ...trap/engines/engine.py.229: process: 0.84623 | load: 0.00311 [I 2022-10-09 12:46:47] ...trap/engines/engine.py.230: loss: 0.14080 [I 2022-10-09 12:46:47] ...optimizers/trijoint.py.151: total_norm: 18.430222 [I 2022-10-09 12:46:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 180/2304 [I 2022-10-09 12:46:56] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:53 [I 2022-10-09 12:46:56] ...trap/engines/engine.py.229: process: 0.84426 | load: 0.00292 [I 2022-10-09 12:46:56] ...trap/engines/engine.py.230: loss: 0.23454 [I 2022-10-09 12:46:56] ...optimizers/trijoint.py.151: total_norm: 26.379610 [I 2022-10-09 12:47:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 190/2304 [I 2022-10-09 12:47:04] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:36 [I 2022-10-09 12:47:04] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00292 [I 2022-10-09 12:47:04] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-09 12:47:04] ...optimizers/trijoint.py.151: total_norm: 23.255735 [I 2022-10-09 12:47:13] ...trap/engines/engine.py.224: train: epoch 41 | batch 200/2304 [I 2022-10-09 12:47:13] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:35 [I 2022-10-09 12:47:13] ...trap/engines/engine.py.229: process: 0.84574 | load: 0.00309 [I 2022-10-09 12:47:13] ...trap/engines/engine.py.230: loss: 0.25594 [I 2022-10-09 12:47:13] ...optimizers/trijoint.py.151: total_norm: 16.826736 [I 2022-10-09 12:47:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 210/2304 [I 2022-10-09 12:47:21] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:23 [I 2022-10-09 12:47:21] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00305 [I 2022-10-09 12:47:21] ...trap/engines/engine.py.230: loss: 0.16897 [I 2022-10-09 12:47:21] ...optimizers/trijoint.py.151: total_norm: 24.700537 [I 2022-10-09 12:47:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 220/2304 [I 2022-10-09 12:47:30] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:30 [I 2022-10-09 12:47:30] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00298 [I 2022-10-09 12:47:30] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-09 12:47:30] ...optimizers/trijoint.py.151: total_norm: 16.008074 [I 2022-10-09 12:47:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 230/2304 [I 2022-10-09 12:47:38] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:10 [I 2022-10-09 12:47:38] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00311 [I 2022-10-09 12:47:38] ...trap/engines/engine.py.230: loss: 0.16464 [I 2022-10-09 12:47:38] ...optimizers/trijoint.py.151: total_norm: 30.905771 [I 2022-10-09 12:47:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 240/2304 [I 2022-10-09 12:47:46] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:00 [I 2022-10-09 12:47:46] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00305 [I 2022-10-09 12:47:46] ...trap/engines/engine.py.230: loss: 0.17115 [I 2022-10-09 12:47:46] ...optimizers/trijoint.py.151: total_norm: 20.148035 [I 2022-10-09 12:47:55] ...trap/engines/engine.py.224: train: epoch 41 | batch 250/2304 [I 2022-10-09 12:47:55] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:51 [I 2022-10-09 12:47:55] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00311 [I 2022-10-09 12:47:55] ...trap/engines/engine.py.230: loss: 0.16183 [I 2022-10-09 12:47:55] ...optimizers/trijoint.py.151: total_norm: 23.265858 [I 2022-10-09 12:48:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 260/2304 [I 2022-10-09 12:48:03] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:40 [I 2022-10-09 12:48:03] ...trap/engines/engine.py.229: process: 0.84549 | load: 0.00319 [I 2022-10-09 12:48:03] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-09 12:48:03] ...optimizers/trijoint.py.151: total_norm: 13.923881 [I 2022-10-09 12:48:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 270/2304 [I 2022-10-09 12:48:12] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:44 [I 2022-10-09 12:48:12] ...trap/engines/engine.py.229: process: 0.84609 | load: 0.00310 [I 2022-10-09 12:48:12] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-09 12:48:12] ...optimizers/trijoint.py.151: total_norm: 26.449030 [I 2022-10-09 12:48:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 280/2304 [I 2022-10-09 12:48:20] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:25 [I 2022-10-09 12:48:20] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00301 [I 2022-10-09 12:48:20] ...trap/engines/engine.py.230: loss: 0.22604 [I 2022-10-09 12:48:20] ...optimizers/trijoint.py.151: total_norm: 28.357195 [I 2022-10-09 12:48:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 290/2304 [I 2022-10-09 12:48:29] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:09 [I 2022-10-09 12:48:29] ...trap/engines/engine.py.229: process: 0.84144 | load: 0.00297 [I 2022-10-09 12:48:29] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-09 12:48:29] ...optimizers/trijoint.py.151: total_norm: 12.997185 [I 2022-10-09 12:48:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 300/2304 [I 2022-10-09 12:48:37] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:08 [I 2022-10-09 12:48:37] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00314 [I 2022-10-09 12:48:37] ...trap/engines/engine.py.230: loss: 0.20475 [I 2022-10-09 12:48:37] ...optimizers/trijoint.py.151: total_norm: 13.520728 [I 2022-10-09 12:48:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 310/2304 [I 2022-10-09 12:48:45] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:47 [I 2022-10-09 12:48:45] ...trap/engines/engine.py.229: process: 0.82494 | load: 0.00307 [I 2022-10-09 12:48:45] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-09 12:48:45] ...optimizers/trijoint.py.151: total_norm: 12.632534 [I 2022-10-09 12:48:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 320/2304 [I 2022-10-09 12:48:54] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:56 [I 2022-10-09 12:48:54] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00312 [I 2022-10-09 12:48:54] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-09 12:48:54] ...optimizers/trijoint.py.151: total_norm: 14.599695 [I 2022-10-09 12:49:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 330/2304 [I 2022-10-09 12:49:02] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:44 [I 2022-10-09 12:49:02] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00299 [I 2022-10-09 12:49:02] ...trap/engines/engine.py.230: loss: 0.21322 [I 2022-10-09 12:49:02] ...optimizers/trijoint.py.151: total_norm: 11.485106 [I 2022-10-09 12:49:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 340/2304 [I 2022-10-09 12:49:11] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:44 [I 2022-10-09 12:49:11] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00295 [I 2022-10-09 12:49:11] ...trap/engines/engine.py.230: loss: 0.14017 [I 2022-10-09 12:49:11] ...optimizers/trijoint.py.151: total_norm: 14.445948 [I 2022-10-09 12:49:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 350/2304 [I 2022-10-09 12:49:19] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:27 [I 2022-10-09 12:49:19] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00303 [I 2022-10-09 12:49:19] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-09 12:49:19] ...optimizers/trijoint.py.151: total_norm: 11.287824 [I 2022-10-09 12:49:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 360/2304 [I 2022-10-09 12:49:28] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:22 [I 2022-10-09 12:49:28] ...trap/engines/engine.py.229: process: 0.84356 | load: 0.00296 [I 2022-10-09 12:49:28] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-09 12:49:28] ...optimizers/trijoint.py.151: total_norm: 38.077236 [I 2022-10-09 12:49:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 370/2304 [I 2022-10-09 12:49:36] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:15 [I 2022-10-09 12:49:36] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00295 [I 2022-10-09 12:49:36] ...trap/engines/engine.py.230: loss: 0.17268 [I 2022-10-09 12:49:36] ...optimizers/trijoint.py.151: total_norm: 11.906273 [I 2022-10-09 12:49:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 380/2304 [I 2022-10-09 12:49:45] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:00 [I 2022-10-09 12:49:45] ...trap/engines/engine.py.229: process: 0.84515 | load: 0.00304 [I 2022-10-09 12:49:45] ...trap/engines/engine.py.230: loss: 0.15071 [I 2022-10-09 12:49:45] ...optimizers/trijoint.py.151: total_norm: 16.459421 [I 2022-10-09 12:49:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 390/2304 [I 2022-10-09 12:49:53] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:49 [I 2022-10-09 12:49:53] ...trap/engines/engine.py.229: process: 0.84275 | load: 0.00294 [I 2022-10-09 12:49:53] ...trap/engines/engine.py.230: loss: 0.18422 [I 2022-10-09 12:49:53] ...optimizers/trijoint.py.151: total_norm: 28.405006 [I 2022-10-09 12:50:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 400/2304 [I 2022-10-09 12:50:02] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:50 [I 2022-10-09 12:50:02] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00298 [I 2022-10-09 12:50:02] ...trap/engines/engine.py.230: loss: 0.18207 [I 2022-10-09 12:50:02] ...optimizers/trijoint.py.151: total_norm: 15.638129 [I 2022-10-09 12:50:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 410/2304 [I 2022-10-09 12:50:10] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:39 [I 2022-10-09 12:50:10] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00300 [I 2022-10-09 12:50:10] ...trap/engines/engine.py.230: loss: 0.16004 [I 2022-10-09 12:50:10] ...optimizers/trijoint.py.151: total_norm: 19.523672 [I 2022-10-09 12:50:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 420/2304 [I 2022-10-09 12:50:18] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:32 [I 2022-10-09 12:50:18] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00295 [I 2022-10-09 12:50:18] ...trap/engines/engine.py.230: loss: 0.15407 [I 2022-10-09 12:50:18] ...optimizers/trijoint.py.151: total_norm: 16.775324 [I 2022-10-09 12:50:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 430/2304 [I 2022-10-09 12:50:27] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:22 [I 2022-10-09 12:50:27] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00294 [I 2022-10-09 12:50:27] ...trap/engines/engine.py.230: loss: 0.15022 [I 2022-10-09 12:50:27] ...optimizers/trijoint.py.151: total_norm: 27.767220 [I 2022-10-09 12:50:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 440/2304 [I 2022-10-09 12:50:35] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:10 [I 2022-10-09 12:50:35] ...trap/engines/engine.py.229: process: 0.84284 | load: 0.00297 [I 2022-10-09 12:50:35] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-09 12:50:35] ...optimizers/trijoint.py.151: total_norm: 18.208557 [I 2022-10-09 12:50:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 450/2304 [I 2022-10-09 12:50:44] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:11 [I 2022-10-09 12:50:44] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00295 [I 2022-10-09 12:50:44] ...trap/engines/engine.py.230: loss: 0.15051 [I 2022-10-09 12:50:44] ...optimizers/trijoint.py.151: total_norm: 26.562700 [I 2022-10-09 12:50:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 460/2304 [I 2022-10-09 12:50:52] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:03 [I 2022-10-09 12:50:52] ...trap/engines/engine.py.229: process: 0.83620 | load: 0.00306 [I 2022-10-09 12:50:52] ...trap/engines/engine.py.230: loss: 0.15334 [I 2022-10-09 12:50:52] ...optimizers/trijoint.py.151: total_norm: 43.455444 [I 2022-10-09 12:51:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 470/2304 [I 2022-10-09 12:51:01] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:56 [I 2022-10-09 12:51:01] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00316 [I 2022-10-09 12:51:01] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-09 12:51:01] ...optimizers/trijoint.py.151: total_norm: 14.181603 [I 2022-10-09 12:51:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 480/2304 [I 2022-10-09 12:51:09] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:50 [I 2022-10-09 12:51:09] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00301 [I 2022-10-09 12:51:09] ...trap/engines/engine.py.230: loss: 0.24182 [I 2022-10-09 12:51:09] ...optimizers/trijoint.py.151: total_norm: 26.154171 [I 2022-10-09 12:51:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 490/2304 [I 2022-10-09 12:51:18] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:32 [I 2022-10-09 12:51:18] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00295 [I 2022-10-09 12:51:18] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-09 12:51:18] ...optimizers/trijoint.py.151: total_norm: 22.526442 [I 2022-10-09 12:51:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 500/2304 [I 2022-10-09 12:51:26] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:24 [I 2022-10-09 12:51:26] ...trap/engines/engine.py.229: process: 0.85063 | load: 0.00297 [I 2022-10-09 12:51:26] ...trap/engines/engine.py.230: loss: 0.16163 [I 2022-10-09 12:51:26] ...optimizers/trijoint.py.151: total_norm: 17.046827 [I 2022-10-09 12:51:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 510/2304 [I 2022-10-09 12:51:35] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:24 [I 2022-10-09 12:51:35] ...trap/engines/engine.py.229: process: 0.86884 | load: 0.00294 [I 2022-10-09 12:51:35] ...trap/engines/engine.py.230: loss: 0.18613 [I 2022-10-09 12:51:35] ...optimizers/trijoint.py.151: total_norm: 14.379209 [I 2022-10-09 12:51:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 520/2304 [I 2022-10-09 12:51:43] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:16 [I 2022-10-09 12:51:43] ...trap/engines/engine.py.229: process: 0.84609 | load: 0.00303 [I 2022-10-09 12:51:43] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-09 12:51:43] ...optimizers/trijoint.py.151: total_norm: 10.060525 [I 2022-10-09 12:51:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 530/2304 [I 2022-10-09 12:51:52] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:02 [I 2022-10-09 12:51:52] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00291 [I 2022-10-09 12:51:52] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-09 12:51:52] ...optimizers/trijoint.py.151: total_norm: 12.651031 [I 2022-10-09 12:52:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 540/2304 [I 2022-10-09 12:52:00] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:52 [I 2022-10-09 12:52:00] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00294 [I 2022-10-09 12:52:00] ...trap/engines/engine.py.230: loss: 0.18240 [I 2022-10-09 12:52:00] ...optimizers/trijoint.py.151: total_norm: 20.565725 [I 2022-10-09 12:52:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 550/2304 [I 2022-10-09 12:52:09] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:48 [I 2022-10-09 12:52:09] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00296 [I 2022-10-09 12:52:09] ...trap/engines/engine.py.230: loss: 0.17995 [I 2022-10-09 12:52:09] ...optimizers/trijoint.py.151: total_norm: 24.165434 [I 2022-10-09 12:52:17] ...trap/engines/engine.py.224: train: epoch 41 | batch 560/2304 [I 2022-10-09 12:52:17] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:43 [I 2022-10-09 12:52:17] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00305 [I 2022-10-09 12:52:17] ...trap/engines/engine.py.230: loss: 0.17603 [I 2022-10-09 12:52:17] ...optimizers/trijoint.py.151: total_norm: 14.247005 [I 2022-10-09 12:52:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 570/2304 [I 2022-10-09 12:52:26] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:24:40 [I 2022-10-09 12:52:26] ...trap/engines/engine.py.229: process: 0.85449 | load: 0.00294 [I 2022-10-09 12:52:26] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-09 12:52:26] ...optimizers/trijoint.py.151: total_norm: 11.672073 [I 2022-10-09 12:52:34] ...trap/engines/engine.py.224: train: epoch 41 | batch 580/2304 [I 2022-10-09 12:52:34] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:18 [I 2022-10-09 12:52:34] ...trap/engines/engine.py.229: process: 0.84669 | load: 0.00296 [I 2022-10-09 12:52:34] ...trap/engines/engine.py.230: loss: 0.20646 [I 2022-10-09 12:52:34] ...optimizers/trijoint.py.151: total_norm: 24.527941 [I 2022-10-09 12:52:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 590/2304 [I 2022-10-09 12:52:43] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:17 [I 2022-10-09 12:52:43] ...trap/engines/engine.py.229: process: 0.83705 | load: 0.00297 [I 2022-10-09 12:52:43] ...trap/engines/engine.py.230: loss: 0.09915 [I 2022-10-09 12:52:43] ...optimizers/trijoint.py.151: total_norm: 23.190599 [I 2022-10-09 12:52:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 600/2304 [I 2022-10-09 12:52:51] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:12 [I 2022-10-09 12:52:51] ...trap/engines/engine.py.229: process: 0.85556 | load: 0.00298 [I 2022-10-09 12:52:51] ...trap/engines/engine.py.230: loss: 0.14716 [I 2022-10-09 12:52:51] ...optimizers/trijoint.py.151: total_norm: 17.789062 [I 2022-10-09 12:53:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 610/2304 [I 2022-10-09 12:53:00] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:23:58 [I 2022-10-09 12:53:00] ...trap/engines/engine.py.229: process: 0.84626 | load: 0.00297 [I 2022-10-09 12:53:00] ...trap/engines/engine.py.230: loss: 0.18649 [I 2022-10-09 12:53:00] ...optimizers/trijoint.py.151: total_norm: 14.353794 [I 2022-10-09 12:53:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 620/2304 [I 2022-10-09 12:53:08] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:51 [I 2022-10-09 12:53:08] ...trap/engines/engine.py.229: process: 0.83085 | load: 0.00315 [I 2022-10-09 12:53:08] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-09 12:53:08] ...optimizers/trijoint.py.151: total_norm: 6.972775 [I 2022-10-09 12:53:17] ...trap/engines/engine.py.224: train: epoch 41 | batch 630/2304 [I 2022-10-09 12:53:17] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:45 [I 2022-10-09 12:53:17] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00298 [I 2022-10-09 12:53:17] ...trap/engines/engine.py.230: loss: 0.18621 [I 2022-10-09 12:53:17] ...optimizers/trijoint.py.151: total_norm: 17.565292 [I 2022-10-09 12:53:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 640/2304 [I 2022-10-09 12:53:25] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:33 [I 2022-10-09 12:53:25] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00309 [I 2022-10-09 12:53:25] ...trap/engines/engine.py.230: loss: 0.14154 [I 2022-10-09 12:53:25] ...optimizers/trijoint.py.151: total_norm: 28.344004 [I 2022-10-09 12:53:34] ...trap/engines/engine.py.224: train: epoch 41 | batch 650/2304 [I 2022-10-09 12:53:34] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:16 [I 2022-10-09 12:53:34] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00297 [I 2022-10-09 12:53:34] ...trap/engines/engine.py.230: loss: 0.22341 [I 2022-10-09 12:53:34] ...optimizers/trijoint.py.151: total_norm: 23.221985 [I 2022-10-09 12:53:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 660/2304 [I 2022-10-09 12:53:42] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:06 [I 2022-10-09 12:53:42] ...trap/engines/engine.py.229: process: 0.83302 | load: 0.00303 [I 2022-10-09 12:53:42] ...trap/engines/engine.py.230: loss: 0.12100 [I 2022-10-09 12:53:42] ...optimizers/trijoint.py.151: total_norm: 10.868287 [I 2022-10-09 12:53:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 670/2304 [I 2022-10-09 12:53:51] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:00 [I 2022-10-09 12:53:51] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00298 [I 2022-10-09 12:53:51] ...trap/engines/engine.py.230: loss: 0.10237 [I 2022-10-09 12:53:51] ...optimizers/trijoint.py.151: total_norm: 14.691350 [I 2022-10-09 12:53:59] ...trap/engines/engine.py.224: train: epoch 41 | batch 680/2304 [I 2022-10-09 12:53:59] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:22:51 [I 2022-10-09 12:53:59] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00301 [I 2022-10-09 12:53:59] ...trap/engines/engine.py.230: loss: 0.14608 [I 2022-10-09 12:53:59] ...optimizers/trijoint.py.151: total_norm: 12.801774 [I 2022-10-09 12:54:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 690/2304 [I 2022-10-09 12:54:08] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:22:41 [I 2022-10-09 12:54:08] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00294 [I 2022-10-09 12:54:08] ...trap/engines/engine.py.230: loss: 0.15006 [I 2022-10-09 12:54:08] ...optimizers/trijoint.py.151: total_norm: 29.740906 [I 2022-10-09 12:54:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 700/2304 [I 2022-10-09 12:54:16] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:22:39 [I 2022-10-09 12:54:16] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00294 [I 2022-10-09 12:54:16] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-09 12:54:16] ...optimizers/trijoint.py.151: total_norm: 10.689103 [I 2022-10-09 12:54:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 710/2304 [I 2022-10-09 12:54:25] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:22:28 [I 2022-10-09 12:54:25] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00295 [I 2022-10-09 12:54:25] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-09 12:54:25] ...optimizers/trijoint.py.151: total_norm: 11.718487 [I 2022-10-09 12:54:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 720/2304 [I 2022-10-09 12:54:33] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:22:17 [I 2022-10-09 12:54:33] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00307 [I 2022-10-09 12:54:33] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-09 12:54:33] ...optimizers/trijoint.py.151: total_norm: 10.898496 [I 2022-10-09 12:54:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 730/2304 [I 2022-10-09 12:54:42] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:06 [I 2022-10-09 12:54:42] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00313 [I 2022-10-09 12:54:42] ...trap/engines/engine.py.230: loss: 0.13344 [I 2022-10-09 12:54:42] ...optimizers/trijoint.py.151: total_norm: 10.029802 [I 2022-10-09 12:54:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 740/2304 [I 2022-10-09 12:54:50] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:22:06 [I 2022-10-09 12:54:50] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00296 [I 2022-10-09 12:54:50] ...trap/engines/engine.py.230: loss: 0.19921 [I 2022-10-09 12:54:50] ...optimizers/trijoint.py.151: total_norm: 12.321710 [I 2022-10-09 12:54:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 750/2304 [I 2022-10-09 12:54:58] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:52 [I 2022-10-09 12:54:58] ...trap/engines/engine.py.229: process: 0.83170 | load: 0.00296 [I 2022-10-09 12:54:58] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-09 12:54:58] ...optimizers/trijoint.py.151: total_norm: 13.837082 [I 2022-10-09 12:55:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 760/2304 [I 2022-10-09 12:55:07] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:21:48 [I 2022-10-09 12:55:07] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00300 [I 2022-10-09 12:55:07] ...trap/engines/engine.py.230: loss: 0.18711 [I 2022-10-09 12:55:07] ...optimizers/trijoint.py.151: total_norm: 15.592144 [I 2022-10-09 12:55:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 770/2304 [I 2022-10-09 12:55:15] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:36 [I 2022-10-09 12:55:15] ...trap/engines/engine.py.229: process: 0.84094 | load: 0.00299 [I 2022-10-09 12:55:15] ...trap/engines/engine.py.230: loss: 0.16160 [I 2022-10-09 12:55:15] ...optimizers/trijoint.py.151: total_norm: 12.683277 [I 2022-10-09 12:55:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 780/2304 [I 2022-10-09 12:55:24] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:21:33 [I 2022-10-09 12:55:24] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00297 [I 2022-10-09 12:55:24] ...trap/engines/engine.py.230: loss: 0.19711 [I 2022-10-09 12:55:24] ...optimizers/trijoint.py.151: total_norm: 18.087879 [I 2022-10-09 12:55:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 790/2304 [I 2022-10-09 12:55:32] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:14 [I 2022-10-09 12:55:32] ...trap/engines/engine.py.229: process: 0.83258 | load: 0.00295 [I 2022-10-09 12:55:32] ...trap/engines/engine.py.230: loss: 0.19911 [I 2022-10-09 12:55:32] ...optimizers/trijoint.py.151: total_norm: 29.388666 [I 2022-10-09 12:55:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 800/2304 [I 2022-10-09 12:55:41] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:09 [I 2022-10-09 12:55:41] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00299 [I 2022-10-09 12:55:41] ...trap/engines/engine.py.230: loss: 0.16094 [I 2022-10-09 12:55:41] ...optimizers/trijoint.py.151: total_norm: 13.666621 [I 2022-10-09 12:55:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 810/2304 [I 2022-10-09 12:55:49] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:20:55 [I 2022-10-09 12:55:49] ...trap/engines/engine.py.229: process: 0.83037 | load: 0.00298 [I 2022-10-09 12:55:49] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-09 12:55:49] ...optimizers/trijoint.py.151: total_norm: 12.147528 [I 2022-10-09 12:55:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 820/2304 [I 2022-10-09 12:55:58] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:20:55 [I 2022-10-09 12:55:58] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00296 [I 2022-10-09 12:55:58] ...trap/engines/engine.py.230: loss: 0.13970 [I 2022-10-09 12:55:58] ...optimizers/trijoint.py.151: total_norm: 15.975626 [I 2022-10-09 12:56:06] ...trap/engines/engine.py.224: train: epoch 41 | batch 830/2304 [I 2022-10-09 12:56:06] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:43 [I 2022-10-09 12:56:06] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00294 [I 2022-10-09 12:56:06] ...trap/engines/engine.py.230: loss: 0.16323 [I 2022-10-09 12:56:06] ...optimizers/trijoint.py.151: total_norm: 12.881041 [I 2022-10-09 12:56:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 840/2304 [I 2022-10-09 12:56:15] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:20:35 [I 2022-10-09 12:56:15] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00293 [I 2022-10-09 12:56:15] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-09 12:56:15] ...optimizers/trijoint.py.151: total_norm: 13.362329 [I 2022-10-09 12:56:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 850/2304 [I 2022-10-09 12:56:23] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:20:26 [I 2022-10-09 12:56:23] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00293 [I 2022-10-09 12:56:23] ...trap/engines/engine.py.230: loss: 0.16071 [I 2022-10-09 12:56:23] ...optimizers/trijoint.py.151: total_norm: 25.635139 [I 2022-10-09 12:56:31] ...trap/engines/engine.py.224: train: epoch 41 | batch 860/2304 [I 2022-10-09 12:56:31] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:16 [I 2022-10-09 12:56:31] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00304 [I 2022-10-09 12:56:31] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-09 12:56:31] ...optimizers/trijoint.py.151: total_norm: 17.020969 [I 2022-10-09 12:56:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 870/2304 [I 2022-10-09 12:56:40] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:10 [I 2022-10-09 12:56:40] ...trap/engines/engine.py.229: process: 0.85195 | load: 0.00293 [I 2022-10-09 12:56:40] ...trap/engines/engine.py.230: loss: 0.21404 [I 2022-10-09 12:56:40] ...optimizers/trijoint.py.151: total_norm: 19.920828 [I 2022-10-09 12:56:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 880/2304 [I 2022-10-09 12:56:48] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:19:55 [I 2022-10-09 12:56:48] ...trap/engines/engine.py.229: process: 0.83302 | load: 0.00292 [I 2022-10-09 12:56:48] ...trap/engines/engine.py.230: loss: 0.17259 [I 2022-10-09 12:56:48] ...optimizers/trijoint.py.151: total_norm: 25.325537 [I 2022-10-09 12:56:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 890/2304 [I 2022-10-09 12:56:57] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:19:55 [I 2022-10-09 12:56:57] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00294 [I 2022-10-09 12:56:57] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-09 12:56:57] ...optimizers/trijoint.py.151: total_norm: 13.221137 [I 2022-10-09 12:57:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 900/2304 [I 2022-10-09 12:57:05] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:19:40 [I 2022-10-09 12:57:05] ...trap/engines/engine.py.229: process: 0.83729 | load: 0.00293 [I 2022-10-09 12:57:05] ...trap/engines/engine.py.230: loss: 0.16566 [I 2022-10-09 12:57:05] ...optimizers/trijoint.py.151: total_norm: 14.945266 [I 2022-10-09 12:57:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 910/2304 [I 2022-10-09 12:57:14] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:39 [I 2022-10-09 12:57:14] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00296 [I 2022-10-09 12:57:14] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-09 12:57:14] ...optimizers/trijoint.py.151: total_norm: 19.837496 [I 2022-10-09 12:57:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 920/2304 [I 2022-10-09 12:57:22] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:22 [I 2022-10-09 12:57:22] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00293 [I 2022-10-09 12:57:22] ...trap/engines/engine.py.230: loss: 0.10148 [I 2022-10-09 12:57:22] ...optimizers/trijoint.py.151: total_norm: 23.126566 [I 2022-10-09 12:57:31] ...trap/engines/engine.py.224: train: epoch 41 | batch 930/2304 [I 2022-10-09 12:57:31] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:20 [I 2022-10-09 12:57:31] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00293 [I 2022-10-09 12:57:31] ...trap/engines/engine.py.230: loss: 0.16562 [I 2022-10-09 12:57:31] ...optimizers/trijoint.py.151: total_norm: 24.421801 [I 2022-10-09 12:57:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 940/2304 [I 2022-10-09 12:57:39] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:18 [I 2022-10-09 12:57:39] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00296 [I 2022-10-09 12:57:39] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-09 12:57:39] ...optimizers/trijoint.py.151: total_norm: 12.175596 [I 2022-10-09 12:57:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 950/2304 [I 2022-10-09 12:57:48] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:18:59 [I 2022-10-09 12:57:48] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00299 [I 2022-10-09 12:57:48] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-09 12:57:48] ...optimizers/trijoint.py.151: total_norm: 14.826488 [I 2022-10-09 12:57:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 960/2304 [I 2022-10-09 12:57:56] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:18:47 [I 2022-10-09 12:57:56] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00299 [I 2022-10-09 12:57:56] ...trap/engines/engine.py.230: loss: 0.12578 [I 2022-10-09 12:57:56] ...optimizers/trijoint.py.151: total_norm: 12.032132 [I 2022-10-09 12:58:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 970/2304 [I 2022-10-09 12:58:04] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:18:47 [I 2022-10-09 12:58:04] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00293 [I 2022-10-09 12:58:04] ...trap/engines/engine.py.230: loss: 0.13666 [I 2022-10-09 12:58:04] ...optimizers/trijoint.py.151: total_norm: 16.506510 [I 2022-10-09 12:58:13] ...trap/engines/engine.py.224: train: epoch 41 | batch 980/2304 [I 2022-10-09 12:58:13] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:29 [I 2022-10-09 12:58:13] ...trap/engines/engine.py.229: process: 0.82568 | load: 0.00293 [I 2022-10-09 12:58:13] ...trap/engines/engine.py.230: loss: 0.16059 [I 2022-10-09 12:58:13] ...optimizers/trijoint.py.151: total_norm: 11.709369 [I 2022-10-09 12:58:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 990/2304 [I 2022-10-09 12:58:21] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:31 [I 2022-10-09 12:58:21] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00294 [I 2022-10-09 12:58:21] ...trap/engines/engine.py.230: loss: 0.18628 [I 2022-10-09 12:58:21] ...optimizers/trijoint.py.151: total_norm: 8.286078 [I 2022-10-09 12:58:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 1000/2304 [I 2022-10-09 12:58:30] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:19 [I 2022-10-09 12:58:30] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00298 [I 2022-10-09 12:58:30] ...trap/engines/engine.py.230: loss: 0.09947 [I 2022-10-09 12:58:30] ...optimizers/trijoint.py.151: total_norm: 7.748670 [I 2022-10-09 12:58:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 1010/2304 [I 2022-10-09 12:58:38] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:12 [I 2022-10-09 12:58:38] ...trap/engines/engine.py.229: process: 0.85616 | load: 0.00295 [I 2022-10-09 12:58:38] ...trap/engines/engine.py.230: loss: 0.15149 [I 2022-10-09 12:58:38] ...optimizers/trijoint.py.151: total_norm: 16.413275 [I 2022-10-09 12:58:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1020/2304 [I 2022-10-09 12:58:47] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:01 [I 2022-10-09 12:58:47] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00297 [I 2022-10-09 12:58:47] ...trap/engines/engine.py.230: loss: 0.18171 [I 2022-10-09 12:58:47] ...optimizers/trijoint.py.151: total_norm: 14.572274 [I 2022-10-09 12:58:55] ...trap/engines/engine.py.224: train: epoch 41 | batch 1030/2304 [I 2022-10-09 12:58:55] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:17:52 [I 2022-10-09 12:58:55] ...trap/engines/engine.py.229: process: 0.83683 | load: 0.00293 [I 2022-10-09 12:58:55] ...trap/engines/engine.py.230: loss: 0.12879 [I 2022-10-09 12:58:55] ...optimizers/trijoint.py.151: total_norm: 13.870106 [I 2022-10-09 12:59:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 1040/2304 [I 2022-10-09 12:59:03] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:17:48 [I 2022-10-09 12:59:03] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00298 [I 2022-10-09 12:59:03] ...trap/engines/engine.py.230: loss: 0.24030 [I 2022-10-09 12:59:03] ...optimizers/trijoint.py.151: total_norm: 13.044727 [I 2022-10-09 12:59:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 1050/2304 [I 2022-10-09 12:59:12] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:48 [I 2022-10-09 12:59:12] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00300 [I 2022-10-09 12:59:12] ...trap/engines/engine.py.230: loss: 0.14844 [I 2022-10-09 12:59:12] ...optimizers/trijoint.py.151: total_norm: 21.215872 [I 2022-10-09 12:59:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 1060/2304 [I 2022-10-09 12:59:21] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:38 [I 2022-10-09 12:59:21] ...trap/engines/engine.py.229: process: 0.85929 | load: 0.00294 [I 2022-10-09 12:59:21] ...trap/engines/engine.py.230: loss: 0.19676 [I 2022-10-09 12:59:21] ...optimizers/trijoint.py.151: total_norm: 19.399632 [I 2022-10-09 12:59:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 1070/2304 [I 2022-10-09 12:59:29] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:33 [I 2022-10-09 12:59:29] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00296 [I 2022-10-09 12:59:29] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-09 12:59:29] ...optimizers/trijoint.py.151: total_norm: 12.235494 [I 2022-10-09 12:59:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 1080/2304 [I 2022-10-09 12:59:38] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:14 [I 2022-10-09 12:59:38] ...trap/engines/engine.py.229: process: 0.83143 | load: 0.00312 [I 2022-10-09 12:59:38] ...trap/engines/engine.py.230: loss: 0.11560 [I 2022-10-09 12:59:38] ...optimizers/trijoint.py.151: total_norm: 8.697197 [I 2022-10-09 12:59:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 1090/2304 [I 2022-10-09 12:59:46] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:08 [I 2022-10-09 12:59:46] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00299 [I 2022-10-09 12:59:46] ...trap/engines/engine.py.230: loss: 0.23780 [I 2022-10-09 12:59:46] ...optimizers/trijoint.py.151: total_norm: 22.813976 [I 2022-10-09 12:59:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1100/2304 [I 2022-10-09 12:59:54] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:16:51 [I 2022-10-09 12:59:54] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00295 [I 2022-10-09 12:59:54] ...trap/engines/engine.py.230: loss: 0.13455 [I 2022-10-09 12:59:54] ...optimizers/trijoint.py.151: total_norm: 17.336048 [I 2022-10-09 13:00:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 1110/2304 [I 2022-10-09 13:00:03] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:46 [I 2022-10-09 13:00:03] ...trap/engines/engine.py.229: process: 0.85046 | load: 0.00300 [I 2022-10-09 13:00:03] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-09 13:00:03] ...optimizers/trijoint.py.151: total_norm: 18.923540 [I 2022-10-09 13:00:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 1120/2304 [I 2022-10-09 13:00:11] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:40 [I 2022-10-09 13:00:11] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00299 [I 2022-10-09 13:00:11] ...trap/engines/engine.py.230: loss: 0.27055 [I 2022-10-09 13:00:11] ...optimizers/trijoint.py.151: total_norm: 19.909285 [I 2022-10-09 13:00:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 1130/2304 [I 2022-10-09 13:00:20] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:30 [I 2022-10-09 13:00:20] ...trap/engines/engine.py.229: process: 0.84418 | load: 0.00295 [I 2022-10-09 13:00:20] ...trap/engines/engine.py.230: loss: 0.09868 [I 2022-10-09 13:00:20] ...optimizers/trijoint.py.151: total_norm: 8.513770 [I 2022-10-09 13:00:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 1140/2304 [I 2022-10-09 13:00:28] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:25 [I 2022-10-09 13:00:28] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00293 [I 2022-10-09 13:00:28] ...trap/engines/engine.py.230: loss: 0.19819 [I 2022-10-09 13:00:28] ...optimizers/trijoint.py.151: total_norm: 44.849552 [I 2022-10-09 13:00:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 1150/2304 [I 2022-10-09 13:00:37] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:17 [I 2022-10-09 13:00:37] ...trap/engines/engine.py.229: process: 0.84206 | load: 0.00297 [I 2022-10-09 13:00:37] ...trap/engines/engine.py.230: loss: 0.17239 [I 2022-10-09 13:00:37] ...optimizers/trijoint.py.151: total_norm: 16.016567 [I 2022-10-09 13:00:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 1160/2304 [I 2022-10-09 13:00:45] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:10 [I 2022-10-09 13:00:45] ...trap/engines/engine.py.229: process: 0.85848 | load: 0.00300 [I 2022-10-09 13:00:45] ...trap/engines/engine.py.230: loss: 0.18096 [I 2022-10-09 13:00:45] ...optimizers/trijoint.py.151: total_norm: 15.269338 [I 2022-10-09 13:00:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1170/2304 [I 2022-10-09 13:00:54] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:15:59 [I 2022-10-09 13:00:54] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00312 [I 2022-10-09 13:00:54] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-09 13:00:54] ...optimizers/trijoint.py.151: total_norm: 8.956592 [I 2022-10-09 13:01:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 1180/2304 [I 2022-10-09 13:01:02] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:47 [I 2022-10-09 13:01:02] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00295 [I 2022-10-09 13:01:02] ...trap/engines/engine.py.230: loss: 0.16512 [I 2022-10-09 13:01:02] ...optimizers/trijoint.py.151: total_norm: 11.211768 [I 2022-10-09 13:01:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 1190/2304 [I 2022-10-09 13:01:11] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:37 [I 2022-10-09 13:01:11] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00297 [I 2022-10-09 13:01:11] ...trap/engines/engine.py.230: loss: 0.17219 [I 2022-10-09 13:01:11] ...optimizers/trijoint.py.151: total_norm: 22.125954 [I 2022-10-09 13:01:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 1200/2304 [I 2022-10-09 13:01:19] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:36 [I 2022-10-09 13:01:19] ...trap/engines/engine.py.229: process: 0.85378 | load: 0.00298 [I 2022-10-09 13:01:19] ...trap/engines/engine.py.230: loss: 0.21002 [I 2022-10-09 13:01:19] ...optimizers/trijoint.py.151: total_norm: 17.957979 [I 2022-10-09 13:01:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 1210/2304 [I 2022-10-09 13:01:28] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:24 [I 2022-10-09 13:01:28] ...trap/engines/engine.py.229: process: 0.84664 | load: 0.00296 [I 2022-10-09 13:01:28] ...trap/engines/engine.py.230: loss: 0.23476 [I 2022-10-09 13:01:28] ...optimizers/trijoint.py.151: total_norm: 21.065079 [I 2022-10-09 13:01:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 1220/2304 [I 2022-10-09 13:01:36] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:15 [I 2022-10-09 13:01:36] ...trap/engines/engine.py.229: process: 0.83305 | load: 0.00299 [I 2022-10-09 13:01:36] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-09 13:01:36] ...optimizers/trijoint.py.151: total_norm: 23.369486 [I 2022-10-09 13:01:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 1230/2304 [I 2022-10-09 13:01:44] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:02 [I 2022-10-09 13:01:44] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00296 [I 2022-10-09 13:01:44] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-09 13:01:44] ...optimizers/trijoint.py.151: total_norm: 12.138420 [I 2022-10-09 13:01:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 1240/2304 [I 2022-10-09 13:01:53] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:58 [I 2022-10-09 13:01:53] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00300 [I 2022-10-09 13:01:53] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-09 13:01:53] ...optimizers/trijoint.py.151: total_norm: 16.011765 [I 2022-10-09 13:02:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 1250/2304 [I 2022-10-09 13:02:01] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:49 [I 2022-10-09 13:02:01] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00297 [I 2022-10-09 13:02:01] ...trap/engines/engine.py.230: loss: 0.13417 [I 2022-10-09 13:02:01] ...optimizers/trijoint.py.151: total_norm: 19.014929 [I 2022-10-09 13:02:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 1260/2304 [I 2022-10-09 13:02:10] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:14:38 [I 2022-10-09 13:02:10] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00297 [I 2022-10-09 13:02:10] ...trap/engines/engine.py.230: loss: 0.08345 [I 2022-10-09 13:02:10] ...optimizers/trijoint.py.151: total_norm: 7.478398 [I 2022-10-09 13:02:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 1270/2304 [I 2022-10-09 13:02:18] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:31 [I 2022-10-09 13:02:18] ...trap/engines/engine.py.229: process: 0.82753 | load: 0.00301 [I 2022-10-09 13:02:18] ...trap/engines/engine.py.230: loss: 0.14803 [I 2022-10-09 13:02:18] ...optimizers/trijoint.py.151: total_norm: 22.781874 [I 2022-10-09 13:02:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 1280/2304 [I 2022-10-09 13:02:27] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:21 [I 2022-10-09 13:02:27] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00296 [I 2022-10-09 13:02:27] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-09 13:02:27] ...optimizers/trijoint.py.151: total_norm: 9.603654 [I 2022-10-09 13:02:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 1290/2304 [I 2022-10-09 13:02:35] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:15 [I 2022-10-09 13:02:35] ...trap/engines/engine.py.229: process: 0.84425 | load: 0.00294 [I 2022-10-09 13:02:35] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-09 13:02:35] ...optimizers/trijoint.py.151: total_norm: 13.558330 [I 2022-10-09 13:02:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 1300/2304 [I 2022-10-09 13:02:44] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:07 [I 2022-10-09 13:02:44] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00309 [I 2022-10-09 13:02:44] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-09 13:02:44] ...optimizers/trijoint.py.151: total_norm: 10.715295 [I 2022-10-09 13:02:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 1310/2304 [I 2022-10-09 13:02:52] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:00 [I 2022-10-09 13:02:52] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00296 [I 2022-10-09 13:02:52] ...trap/engines/engine.py.230: loss: 0.14666 [I 2022-10-09 13:02:52] ...optimizers/trijoint.py.151: total_norm: 17.510456 [I 2022-10-09 13:03:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 1320/2304 [I 2022-10-09 13:03:01] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:52 [I 2022-10-09 13:03:01] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00295 [I 2022-10-09 13:03:01] ...trap/engines/engine.py.230: loss: 0.16446 [I 2022-10-09 13:03:01] ...optimizers/trijoint.py.151: total_norm: 11.684323 [I 2022-10-09 13:03:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 1330/2304 [I 2022-10-09 13:03:09] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:44 [I 2022-10-09 13:03:09] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00299 [I 2022-10-09 13:03:09] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-09 13:03:09] ...optimizers/trijoint.py.151: total_norm: 11.949713 [I 2022-10-09 13:03:17] ...trap/engines/engine.py.224: train: epoch 41 | batch 1340/2304 [I 2022-10-09 13:03:17] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:35 [I 2022-10-09 13:03:17] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00300 [I 2022-10-09 13:03:17] ...trap/engines/engine.py.230: loss: 0.15931 [I 2022-10-09 13:03:17] ...optimizers/trijoint.py.151: total_norm: 18.018719 [I 2022-10-09 13:03:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 1350/2304 [I 2022-10-09 13:03:26] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:25 [I 2022-10-09 13:03:26] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00294 [I 2022-10-09 13:03:26] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-09 13:03:26] ...optimizers/trijoint.py.151: total_norm: 18.861227 [I 2022-10-09 13:03:34] ...trap/engines/engine.py.224: train: epoch 41 | batch 1360/2304 [I 2022-10-09 13:03:34] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:13 [I 2022-10-09 13:03:34] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00294 [I 2022-10-09 13:03:34] ...trap/engines/engine.py.230: loss: 0.09276 [I 2022-10-09 13:03:34] ...optimizers/trijoint.py.151: total_norm: 8.801896 [I 2022-10-09 13:03:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 1370/2304 [I 2022-10-09 13:03:43] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:05 [I 2022-10-09 13:03:43] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00295 [I 2022-10-09 13:03:43] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-09 13:03:43] ...optimizers/trijoint.py.151: total_norm: 10.339970 [I 2022-10-09 13:03:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 1380/2304 [I 2022-10-09 13:03:51] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:12:59 [I 2022-10-09 13:03:51] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00300 [I 2022-10-09 13:03:51] ...trap/engines/engine.py.230: loss: 0.11543 [I 2022-10-09 13:03:51] ...optimizers/trijoint.py.151: total_norm: 11.739589 [I 2022-10-09 13:04:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 1390/2304 [I 2022-10-09 13:04:00] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:12:49 [I 2022-10-09 13:04:00] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00294 [I 2022-10-09 13:04:00] ...trap/engines/engine.py.230: loss: 0.16656 [I 2022-10-09 13:04:00] ...optimizers/trijoint.py.151: total_norm: 17.450771 [I 2022-10-09 13:04:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 1400/2304 [I 2022-10-09 13:04:08] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:45 [I 2022-10-09 13:04:08] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00307 [I 2022-10-09 13:04:08] ...trap/engines/engine.py.230: loss: 0.15378 [I 2022-10-09 13:04:08] ...optimizers/trijoint.py.151: total_norm: 15.039946 [I 2022-10-09 13:04:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 1410/2304 [I 2022-10-09 13:04:16] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:28 [I 2022-10-09 13:04:16] ...trap/engines/engine.py.229: process: 0.82975 | load: 0.00292 [I 2022-10-09 13:04:16] ...trap/engines/engine.py.230: loss: 0.15905 [I 2022-10-09 13:04:16] ...optimizers/trijoint.py.151: total_norm: 22.021095 [I 2022-10-09 13:04:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 1420/2304 [I 2022-10-09 13:04:25] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:24 [I 2022-10-09 13:04:25] ...trap/engines/engine.py.229: process: 0.84641 | load: 0.00302 [I 2022-10-09 13:04:25] ...trap/engines/engine.py.230: loss: 0.13511 [I 2022-10-09 13:04:25] ...optimizers/trijoint.py.151: total_norm: 11.677469 [I 2022-10-09 13:04:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 1430/2304 [I 2022-10-09 13:04:33] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:24 [I 2022-10-09 13:04:33] ...trap/engines/engine.py.229: process: 0.86297 | load: 0.00294 [I 2022-10-09 13:04:33] ...trap/engines/engine.py.230: loss: 0.17058 [I 2022-10-09 13:04:33] ...optimizers/trijoint.py.151: total_norm: 34.766487 [I 2022-10-09 13:04:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 1440/2304 [I 2022-10-09 13:04:42] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:19 [I 2022-10-09 13:04:42] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00294 [I 2022-10-09 13:04:42] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-09 13:04:42] ...optimizers/trijoint.py.151: total_norm: 30.479370 [I 2022-10-09 13:04:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 1450/2304 [I 2022-10-09 13:04:51] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:12 [I 2022-10-09 13:04:51] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00298 [I 2022-10-09 13:04:51] ...trap/engines/engine.py.230: loss: 0.17683 [I 2022-10-09 13:04:51] ...optimizers/trijoint.py.151: total_norm: 14.257465 [I 2022-10-09 13:04:59] ...trap/engines/engine.py.224: train: epoch 41 | batch 1460/2304 [I 2022-10-09 13:04:59] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:11:57 [I 2022-10-09 13:04:59] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00297 [I 2022-10-09 13:04:59] ...trap/engines/engine.py.230: loss: 0.24337 [I 2022-10-09 13:04:59] ...optimizers/trijoint.py.151: total_norm: 28.537716 [I 2022-10-09 13:05:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 1470/2304 [I 2022-10-09 13:05:08] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:11:45 [I 2022-10-09 13:05:08] ...trap/engines/engine.py.229: process: 0.83751 | load: 0.00298 [I 2022-10-09 13:05:08] ...trap/engines/engine.py.230: loss: 0.15006 [I 2022-10-09 13:05:08] ...optimizers/trijoint.py.151: total_norm: 18.090681 [I 2022-10-09 13:05:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 1480/2304 [I 2022-10-09 13:05:16] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:38 [I 2022-10-09 13:05:16] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00294 [I 2022-10-09 13:05:16] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-09 13:05:16] ...optimizers/trijoint.py.151: total_norm: 8.666328 [I 2022-10-09 13:05:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 1490/2304 [I 2022-10-09 13:05:25] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:29 [I 2022-10-09 13:05:25] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00296 [I 2022-10-09 13:05:25] ...trap/engines/engine.py.230: loss: 0.17593 [I 2022-10-09 13:05:25] ...optimizers/trijoint.py.151: total_norm: 32.524826 [I 2022-10-09 13:05:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 1500/2304 [I 2022-10-09 13:05:33] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:18 [I 2022-10-09 13:05:33] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00294 [I 2022-10-09 13:05:33] ...trap/engines/engine.py.230: loss: 0.15933 [I 2022-10-09 13:05:33] ...optimizers/trijoint.py.151: total_norm: 17.674957 [I 2022-10-09 13:05:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 1510/2304 [I 2022-10-09 13:05:41] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:11 [I 2022-10-09 13:05:41] ...trap/engines/engine.py.229: process: 0.87055 | load: 0.00305 [I 2022-10-09 13:05:41] ...trap/engines/engine.py.230: loss: 0.12969 [I 2022-10-09 13:05:41] ...optimizers/trijoint.py.151: total_norm: 20.908857 [I 2022-10-09 13:05:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 1520/2304 [I 2022-10-09 13:05:50] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:06 [I 2022-10-09 13:05:50] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00299 [I 2022-10-09 13:05:50] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-09 13:05:50] ...optimizers/trijoint.py.151: total_norm: 18.932930 [I 2022-10-09 13:05:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 1530/2304 [I 2022-10-09 13:05:58] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:10:56 [I 2022-10-09 13:05:58] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00304 [I 2022-10-09 13:05:58] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-09 13:05:58] ...optimizers/trijoint.py.151: total_norm: 13.557593 [I 2022-10-09 13:06:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 1540/2304 [I 2022-10-09 13:06:07] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:45 [I 2022-10-09 13:06:07] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00298 [I 2022-10-09 13:06:07] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-09 13:06:07] ...optimizers/trijoint.py.151: total_norm: 22.140852 [I 2022-10-09 13:06:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 1550/2304 [I 2022-10-09 13:06:15] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:40 [I 2022-10-09 13:06:15] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00298 [I 2022-10-09 13:06:15] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-09 13:06:15] ...optimizers/trijoint.py.151: total_norm: 14.150444 [I 2022-10-09 13:06:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 1560/2304 [I 2022-10-09 13:06:24] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:33 [I 2022-10-09 13:06:24] ...trap/engines/engine.py.229: process: 0.85922 | load: 0.00298 [I 2022-10-09 13:06:24] ...trap/engines/engine.py.230: loss: 0.19067 [I 2022-10-09 13:06:24] ...optimizers/trijoint.py.151: total_norm: 17.444990 [I 2022-10-09 13:06:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 1570/2304 [I 2022-10-09 13:06:33] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:26 [I 2022-10-09 13:06:33] ...trap/engines/engine.py.229: process: 0.85660 | load: 0.00298 [I 2022-10-09 13:06:33] ...trap/engines/engine.py.230: loss: 0.17960 [I 2022-10-09 13:06:33] ...optimizers/trijoint.py.151: total_norm: 13.233210 [I 2022-10-09 13:06:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 1580/2304 [I 2022-10-09 13:06:41] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:14 [I 2022-10-09 13:06:41] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00295 [I 2022-10-09 13:06:41] ...trap/engines/engine.py.230: loss: 0.18619 [I 2022-10-09 13:06:41] ...optimizers/trijoint.py.151: total_norm: 26.170982 [I 2022-10-09 13:06:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 1590/2304 [I 2022-10-09 13:06:50] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:09 [I 2022-10-09 13:06:50] ...trap/engines/engine.py.229: process: 0.86252 | load: 0.00297 [I 2022-10-09 13:06:50] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-09 13:06:50] ...optimizers/trijoint.py.151: total_norm: 17.802969 [I 2022-10-09 13:06:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 1600/2304 [I 2022-10-09 13:06:58] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:00 [I 2022-10-09 13:06:58] ...trap/engines/engine.py.229: process: 0.84919 | load: 0.00310 [I 2022-10-09 13:06:58] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-09 13:06:58] ...optimizers/trijoint.py.151: total_norm: 12.898386 [I 2022-10-09 13:07:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 1610/2304 [I 2022-10-09 13:07:07] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:09:52 [I 2022-10-09 13:07:07] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00345 [I 2022-10-09 13:07:07] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-09 13:07:07] ...optimizers/trijoint.py.151: total_norm: 14.046708 [I 2022-10-09 13:07:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 1620/2304 [I 2022-10-09 13:07:15] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:36 [I 2022-10-09 13:07:15] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00298 [I 2022-10-09 13:07:15] ...trap/engines/engine.py.230: loss: 0.16947 [I 2022-10-09 13:07:15] ...optimizers/trijoint.py.151: total_norm: 19.265009 [I 2022-10-09 13:07:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 1630/2304 [I 2022-10-09 13:07:24] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:31 [I 2022-10-09 13:07:24] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00296 [I 2022-10-09 13:07:24] ...trap/engines/engine.py.230: loss: 0.15268 [I 2022-10-09 13:07:24] ...optimizers/trijoint.py.151: total_norm: 17.073664 [I 2022-10-09 13:07:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 1640/2304 [I 2022-10-09 13:07:32] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:21 [I 2022-10-09 13:07:32] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00295 [I 2022-10-09 13:07:32] ...trap/engines/engine.py.230: loss: 0.25959 [I 2022-10-09 13:07:32] ...optimizers/trijoint.py.151: total_norm: 24.271950 [I 2022-10-09 13:07:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 1650/2304 [I 2022-10-09 13:07:41] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:15 [I 2022-10-09 13:07:41] ...trap/engines/engine.py.229: process: 0.83898 | load: 0.00303 [I 2022-10-09 13:07:41] ...trap/engines/engine.py.230: loss: 0.16201 [I 2022-10-09 13:07:41] ...optimizers/trijoint.py.151: total_norm: 19.761696 [I 2022-10-09 13:07:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 1660/2304 [I 2022-10-09 13:07:49] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:06 [I 2022-10-09 13:07:49] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00304 [I 2022-10-09 13:07:49] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-09 13:07:49] ...optimizers/trijoint.py.151: total_norm: 16.264601 [I 2022-10-09 13:07:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 1670/2304 [I 2022-10-09 13:07:57] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:08:54 [I 2022-10-09 13:07:57] ...trap/engines/engine.py.229: process: 0.84034 | load: 0.00302 [I 2022-10-09 13:07:57] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-09 13:07:57] ...optimizers/trijoint.py.151: total_norm: 9.140104 [I 2022-10-09 13:08:06] ...trap/engines/engine.py.224: train: epoch 41 | batch 1680/2304 [I 2022-10-09 13:08:06] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:08:46 [I 2022-10-09 13:08:06] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00295 [I 2022-10-09 13:08:06] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-09 13:08:06] ...optimizers/trijoint.py.151: total_norm: 12.530397 [I 2022-10-09 13:08:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 1690/2304 [I 2022-10-09 13:08:14] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:08:36 [I 2022-10-09 13:08:14] ...trap/engines/engine.py.229: process: 0.83672 | load: 0.00298 [I 2022-10-09 13:08:14] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-09 13:08:14] ...optimizers/trijoint.py.151: total_norm: 19.472723 [I 2022-10-09 13:08:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 1700/2304 [I 2022-10-09 13:08:23] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:30 [I 2022-10-09 13:08:23] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00295 [I 2022-10-09 13:08:23] ...trap/engines/engine.py.230: loss: 0.14587 [I 2022-10-09 13:08:23] ...optimizers/trijoint.py.151: total_norm: 16.272442 [I 2022-10-09 13:08:31] ...trap/engines/engine.py.224: train: epoch 41 | batch 1710/2304 [I 2022-10-09 13:08:31] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:19 [I 2022-10-09 13:08:31] ...trap/engines/engine.py.229: process: 0.83364 | load: 0.00302 [I 2022-10-09 13:08:31] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-09 13:08:31] ...optimizers/trijoint.py.151: total_norm: 16.249512 [I 2022-10-09 13:08:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 1720/2304 [I 2022-10-09 13:08:40] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:13 [I 2022-10-09 13:08:40] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00293 [I 2022-10-09 13:08:40] ...trap/engines/engine.py.230: loss: 0.19076 [I 2022-10-09 13:08:40] ...optimizers/trijoint.py.151: total_norm: 19.874004 [I 2022-10-09 13:08:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 1730/2304 [I 2022-10-09 13:08:48] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:03 [I 2022-10-09 13:08:48] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00298 [I 2022-10-09 13:08:48] ...trap/engines/engine.py.230: loss: 0.20242 [I 2022-10-09 13:08:48] ...optimizers/trijoint.py.151: total_norm: 14.024114 [I 2022-10-09 13:08:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 1740/2304 [I 2022-10-09 13:08:57] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:07:57 [I 2022-10-09 13:08:57] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00298 [I 2022-10-09 13:08:57] ...trap/engines/engine.py.230: loss: 0.19772 [I 2022-10-09 13:08:57] ...optimizers/trijoint.py.151: total_norm: 11.816332 [I 2022-10-09 13:09:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 1750/2304 [I 2022-10-09 13:09:05] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:07:48 [I 2022-10-09 13:09:05] ...trap/engines/engine.py.229: process: 0.84715 | load: 0.00298 [I 2022-10-09 13:09:05] ...trap/engines/engine.py.230: loss: 0.11882 [I 2022-10-09 13:09:05] ...optimizers/trijoint.py.151: total_norm: 16.204227 [I 2022-10-09 13:09:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 1760/2304 [I 2022-10-09 13:09:14] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:07:36 [I 2022-10-09 13:09:14] ...trap/engines/engine.py.229: process: 0.82683 | load: 0.00296 [I 2022-10-09 13:09:14] ...trap/engines/engine.py.230: loss: 0.19151 [I 2022-10-09 13:09:14] ...optimizers/trijoint.py.151: total_norm: 13.474185 [I 2022-10-09 13:09:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 1770/2304 [I 2022-10-09 13:09:22] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:28 [I 2022-10-09 13:09:22] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00293 [I 2022-10-09 13:09:22] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-09 13:09:22] ...optimizers/trijoint.py.151: total_norm: 11.617707 [I 2022-10-09 13:09:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 1780/2304 [I 2022-10-09 13:09:30] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:22 [I 2022-10-09 13:09:30] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00294 [I 2022-10-09 13:09:30] ...trap/engines/engine.py.230: loss: 0.11918 [I 2022-10-09 13:09:30] ...optimizers/trijoint.py.151: total_norm: 12.317047 [I 2022-10-09 13:09:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 1790/2304 [I 2022-10-09 13:09:39] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:17 [I 2022-10-09 13:09:39] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00300 [I 2022-10-09 13:09:39] ...trap/engines/engine.py.230: loss: 0.16841 [I 2022-10-09 13:09:39] ...optimizers/trijoint.py.151: total_norm: 12.345312 [I 2022-10-09 13:09:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1800/2304 [I 2022-10-09 13:09:47] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:05 [I 2022-10-09 13:09:47] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00307 [I 2022-10-09 13:09:47] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-09 13:09:47] ...optimizers/trijoint.py.151: total_norm: 12.284040 [I 2022-10-09 13:09:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 1810/2304 [I 2022-10-09 13:09:56] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:06:55 [I 2022-10-09 13:09:56] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00296 [I 2022-10-09 13:09:56] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-09 13:09:56] ...optimizers/trijoint.py.151: total_norm: 15.833996 [I 2022-10-09 13:10:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 1820/2304 [I 2022-10-09 13:10:04] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:06:48 [I 2022-10-09 13:10:04] ...trap/engines/engine.py.229: process: 0.84705 | load: 0.00299 [I 2022-10-09 13:10:04] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-09 13:10:04] ...optimizers/trijoint.py.151: total_norm: 12.787539 [I 2022-10-09 13:10:13] ...trap/engines/engine.py.224: train: epoch 41 | batch 1830/2304 [I 2022-10-09 13:10:13] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:41 [I 2022-10-09 13:10:13] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00303 [I 2022-10-09 13:10:13] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-09 13:10:13] ...optimizers/trijoint.py.151: total_norm: 19.464521 [I 2022-10-09 13:10:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 1840/2304 [I 2022-10-09 13:10:21] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:29 [I 2022-10-09 13:10:21] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00295 [I 2022-10-09 13:10:21] ...trap/engines/engine.py.230: loss: 0.18981 [I 2022-10-09 13:10:21] ...optimizers/trijoint.py.151: total_norm: 32.348431 [I 2022-10-09 13:10:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 1850/2304 [I 2022-10-09 13:10:30] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:21 [I 2022-10-09 13:10:30] ...trap/engines/engine.py.229: process: 0.82828 | load: 0.00292 [I 2022-10-09 13:10:30] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-09 13:10:30] ...optimizers/trijoint.py.151: total_norm: 17.429834 [I 2022-10-09 13:10:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 1860/2304 [I 2022-10-09 13:10:38] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:14 [I 2022-10-09 13:10:38] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00294 [I 2022-10-09 13:10:38] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-09 13:10:38] ...optimizers/trijoint.py.151: total_norm: 14.309617 [I 2022-10-09 13:10:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1870/2304 [I 2022-10-09 13:10:47] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:06 [I 2022-10-09 13:10:47] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00294 [I 2022-10-09 13:10:47] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-09 13:10:47] ...optimizers/trijoint.py.151: total_norm: 18.929480 [I 2022-10-09 13:10:55] ...trap/engines/engine.py.224: train: epoch 41 | batch 1880/2304 [I 2022-10-09 13:10:55] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:05:57 [I 2022-10-09 13:10:55] ...trap/engines/engine.py.229: process: 0.84459 | load: 0.00295 [I 2022-10-09 13:10:55] ...trap/engines/engine.py.230: loss: 0.17995 [I 2022-10-09 13:10:55] ...optimizers/trijoint.py.151: total_norm: 19.474016 [I 2022-10-09 13:11:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 1890/2304 [I 2022-10-09 13:11:03] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:05:50 [I 2022-10-09 13:11:03] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00295 [I 2022-10-09 13:11:03] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-09 13:11:03] ...optimizers/trijoint.py.151: total_norm: 29.416460 [I 2022-10-09 13:11:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 1900/2304 [I 2022-10-09 13:11:12] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:39 [I 2022-10-09 13:11:12] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00304 [I 2022-10-09 13:11:12] ...trap/engines/engine.py.230: loss: 0.15193 [I 2022-10-09 13:11:12] ...optimizers/trijoint.py.151: total_norm: 22.348700 [I 2022-10-09 13:11:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 1910/2304 [I 2022-10-09 13:11:20] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:32 [I 2022-10-09 13:11:20] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00297 [I 2022-10-09 13:11:20] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-09 13:11:20] ...optimizers/trijoint.py.151: total_norm: 19.249298 [I 2022-10-09 13:11:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 1920/2304 [I 2022-10-09 13:11:29] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:26 [I 2022-10-09 13:11:29] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00296 [I 2022-10-09 13:11:29] ...trap/engines/engine.py.230: loss: 0.08760 [I 2022-10-09 13:11:29] ...optimizers/trijoint.py.151: total_norm: 11.435861 [I 2022-10-09 13:11:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 1930/2304 [I 2022-10-09 13:11:37] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:15 [I 2022-10-09 13:11:37] ...trap/engines/engine.py.229: process: 0.83173 | load: 0.00292 [I 2022-10-09 13:11:37] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-09 13:11:37] ...optimizers/trijoint.py.151: total_norm: 16.410971 [I 2022-10-09 13:11:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 1940/2304 [I 2022-10-09 13:11:46] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:08 [I 2022-10-09 13:11:46] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00293 [I 2022-10-09 13:11:46] ...trap/engines/engine.py.230: loss: 0.12278 [I 2022-10-09 13:11:46] ...optimizers/trijoint.py.151: total_norm: 13.909008 [I 2022-10-09 13:11:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1950/2304 [I 2022-10-09 13:11:54] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:00 [I 2022-10-09 13:11:54] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00292 [I 2022-10-09 13:11:54] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-09 13:11:54] ...optimizers/trijoint.py.151: total_norm: 23.314367 [I 2022-10-09 13:12:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 1960/2304 [I 2022-10-09 13:12:03] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:04:51 [I 2022-10-09 13:12:03] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00293 [I 2022-10-09 13:12:03] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-09 13:12:03] ...optimizers/trijoint.py.151: total_norm: 18.173969 [I 2022-10-09 13:12:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 1970/2304 [I 2022-10-09 13:12:11] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:40 [I 2022-10-09 13:12:11] ...trap/engines/engine.py.229: process: 0.83215 | load: 0.00296 [I 2022-10-09 13:12:11] ...trap/engines/engine.py.230: loss: 0.14440 [I 2022-10-09 13:12:11] ...optimizers/trijoint.py.151: total_norm: 13.970433 [I 2022-10-09 13:12:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 1980/2304 [I 2022-10-09 13:12:20] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:34 [I 2022-10-09 13:12:20] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00304 [I 2022-10-09 13:12:20] ...trap/engines/engine.py.230: loss: 0.18882 [I 2022-10-09 13:12:20] ...optimizers/trijoint.py.151: total_norm: 214.844559 [I 2022-10-09 13:12:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 1990/2304 [I 2022-10-09 13:12:28] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:24 [I 2022-10-09 13:12:28] ...trap/engines/engine.py.229: process: 0.83315 | load: 0.00297 [I 2022-10-09 13:12:28] ...trap/engines/engine.py.230: loss: 0.14079 [I 2022-10-09 13:12:28] ...optimizers/trijoint.py.151: total_norm: 18.840673 [I 2022-10-09 13:12:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 2000/2304 [I 2022-10-09 13:12:37] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:18 [I 2022-10-09 13:12:37] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00293 [I 2022-10-09 13:12:37] ...trap/engines/engine.py.230: loss: 0.12211 [I 2022-10-09 13:12:37] ...optimizers/trijoint.py.151: total_norm: 9.790218 [I 2022-10-09 13:12:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 2010/2304 [I 2022-10-09 13:12:45] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:11 [I 2022-10-09 13:12:45] ...trap/engines/engine.py.229: process: 0.86766 | load: 0.00295 [I 2022-10-09 13:12:45] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-09 13:12:45] ...optimizers/trijoint.py.151: total_norm: 10.187793 [I 2022-10-09 13:12:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 2020/2304 [I 2022-10-09 13:12:54] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:01 [I 2022-10-09 13:12:54] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00298 [I 2022-10-09 13:12:54] ...trap/engines/engine.py.230: loss: 0.12557 [I 2022-10-09 13:12:54] ...optimizers/trijoint.py.151: total_norm: 13.181999 [I 2022-10-09 13:13:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 2030/2304 [I 2022-10-09 13:13:02] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:52 [I 2022-10-09 13:13:02] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00299 [I 2022-10-09 13:13:02] ...trap/engines/engine.py.230: loss: 0.15176 [I 2022-10-09 13:13:02] ...optimizers/trijoint.py.151: total_norm: 15.639965 [I 2022-10-09 13:13:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 2040/2304 [I 2022-10-09 13:13:11] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:03:45 [I 2022-10-09 13:13:11] ...trap/engines/engine.py.229: process: 0.86270 | load: 0.00299 [I 2022-10-09 13:13:11] ...trap/engines/engine.py.230: loss: 0.26023 [I 2022-10-09 13:13:11] ...optimizers/trijoint.py.151: total_norm: 18.175381 [I 2022-10-09 13:13:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 2050/2304 [I 2022-10-09 13:13:19] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:36 [I 2022-10-09 13:13:19] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00296 [I 2022-10-09 13:13:19] ...trap/engines/engine.py.230: loss: 0.16927 [I 2022-10-09 13:13:19] ...optimizers/trijoint.py.151: total_norm: 15.078315 [I 2022-10-09 13:13:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 2060/2304 [I 2022-10-09 13:13:28] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:27 [I 2022-10-09 13:13:28] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00297 [I 2022-10-09 13:13:28] ...trap/engines/engine.py.230: loss: 0.23693 [I 2022-10-09 13:13:28] ...optimizers/trijoint.py.151: total_norm: 28.125587 [I 2022-10-09 13:13:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 2070/2304 [I 2022-10-09 13:13:36] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:17 [I 2022-10-09 13:13:36] ...trap/engines/engine.py.229: process: 0.84448 | load: 0.00297 [I 2022-10-09 13:13:36] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-09 13:13:36] ...optimizers/trijoint.py.151: total_norm: 12.248778 [I 2022-10-09 13:13:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 2080/2304 [I 2022-10-09 13:13:45] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:10 [I 2022-10-09 13:13:45] ...trap/engines/engine.py.229: process: 0.86203 | load: 0.00308 [I 2022-10-09 13:13:45] ...trap/engines/engine.py.230: loss: 0.19159 [I 2022-10-09 13:13:45] ...optimizers/trijoint.py.151: total_norm: 11.345530 [I 2022-10-09 13:13:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 2090/2304 [I 2022-10-09 13:13:53] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:02 [I 2022-10-09 13:13:53] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00307 [I 2022-10-09 13:13:53] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-09 13:13:53] ...optimizers/trijoint.py.151: total_norm: 15.760703 [I 2022-10-09 13:14:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 2100/2304 [I 2022-10-09 13:14:02] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:02:51 [I 2022-10-09 13:14:02] ...trap/engines/engine.py.229: process: 0.81614 | load: 0.00298 [I 2022-10-09 13:14:02] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-09 13:14:02] ...optimizers/trijoint.py.151: total_norm: 7.853035 [I 2022-10-09 13:14:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 2110/2304 [I 2022-10-09 13:14:10] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:02:43 [I 2022-10-09 13:14:10] ...trap/engines/engine.py.229: process: 0.83226 | load: 0.00300 [I 2022-10-09 13:14:10] ...trap/engines/engine.py.230: loss: 0.31479 [I 2022-10-09 13:14:10] ...optimizers/trijoint.py.151: total_norm: 40.507351 [I 2022-10-09 13:14:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 2120/2304 [I 2022-10-09 13:14:19] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:35 [I 2022-10-09 13:14:19] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00295 [I 2022-10-09 13:14:19] ...trap/engines/engine.py.230: loss: 0.21976 [I 2022-10-09 13:14:19] ...optimizers/trijoint.py.151: total_norm: 19.259245 [I 2022-10-09 13:14:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 2130/2304 [I 2022-10-09 13:14:27] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:27 [I 2022-10-09 13:14:27] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00296 [I 2022-10-09 13:14:27] ...trap/engines/engine.py.230: loss: 0.20001 [I 2022-10-09 13:14:27] ...optimizers/trijoint.py.151: total_norm: 7.295561 [I 2022-10-09 13:14:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 2140/2304 [I 2022-10-09 13:14:36] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:19 [I 2022-10-09 13:14:36] ...trap/engines/engine.py.229: process: 0.84444 | load: 0.00298 [I 2022-10-09 13:14:36] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-09 13:14:36] ...optimizers/trijoint.py.151: total_norm: 15.395665 [I 2022-10-09 13:14:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 2150/2304 [I 2022-10-09 13:14:44] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:10 [I 2022-10-09 13:14:44] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00297 [I 2022-10-09 13:14:44] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-09 13:14:44] ...optimizers/trijoint.py.151: total_norm: 10.139043 [I 2022-10-09 13:14:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 2160/2304 [I 2022-10-09 13:14:53] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:01 [I 2022-10-09 13:14:53] ...trap/engines/engine.py.229: process: 0.83040 | load: 0.00298 [I 2022-10-09 13:14:53] ...trap/engines/engine.py.230: loss: 0.13809 [I 2022-10-09 13:14:53] ...optimizers/trijoint.py.151: total_norm: 11.585598 [I 2022-10-09 13:15:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 2170/2304 [I 2022-10-09 13:15:01] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:01:53 [I 2022-10-09 13:15:01] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00297 [I 2022-10-09 13:15:01] ...trap/engines/engine.py.230: loss: 0.13160 [I 2022-10-09 13:15:01] ...optimizers/trijoint.py.151: total_norm: 17.598986 [I 2022-10-09 13:15:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 2180/2304 [I 2022-10-09 13:15:10] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:44 [I 2022-10-09 13:15:10] ...trap/engines/engine.py.229: process: 0.83918 | load: 0.00294 [I 2022-10-09 13:15:10] ...trap/engines/engine.py.230: loss: 0.16667 [I 2022-10-09 13:15:10] ...optimizers/trijoint.py.151: total_norm: 18.208714 [I 2022-10-09 13:15:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 2190/2304 [I 2022-10-09 13:15:18] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:36 [I 2022-10-09 13:15:18] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00298 [I 2022-10-09 13:15:18] ...trap/engines/engine.py.230: loss: 0.11141 [I 2022-10-09 13:15:18] ...optimizers/trijoint.py.151: total_norm: 15.404387 [I 2022-10-09 13:15:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 2200/2304 [I 2022-10-09 13:15:27] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:28 [I 2022-10-09 13:15:27] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00301 [I 2022-10-09 13:15:27] ...trap/engines/engine.py.230: loss: 0.13334 [I 2022-10-09 13:15:27] ...optimizers/trijoint.py.151: total_norm: 13.618193 [I 2022-10-09 13:15:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 2210/2304 [I 2022-10-09 13:15:35] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:19 [I 2022-10-09 13:15:35] ...trap/engines/engine.py.229: process: 0.84050 | load: 0.00298 [I 2022-10-09 13:15:35] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-09 13:15:35] ...optimizers/trijoint.py.151: total_norm: 16.177843 [I 2022-10-09 13:15:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 2220/2304 [I 2022-10-09 13:15:44] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:11 [I 2022-10-09 13:15:44] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00300 [I 2022-10-09 13:15:44] ...trap/engines/engine.py.230: loss: 0.17733 [I 2022-10-09 13:15:44] ...optimizers/trijoint.py.151: total_norm: 9.211723 [I 2022-10-09 13:15:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 2230/2304 [I 2022-10-09 13:15:52] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:02 [I 2022-10-09 13:15:52] ...trap/engines/engine.py.229: process: 0.83186 | load: 0.00298 [I 2022-10-09 13:15:52] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-09 13:15:52] ...optimizers/trijoint.py.151: total_norm: 14.780934 [I 2022-10-09 13:16:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 2240/2304 [I 2022-10-09 13:16:01] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:00:54 [I 2022-10-09 13:16:01] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00296 [I 2022-10-09 13:16:01] ...trap/engines/engine.py.230: loss: 0.16659 [I 2022-10-09 13:16:01] ...optimizers/trijoint.py.151: total_norm: 11.783430 [I 2022-10-09 13:16:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 2250/2304 [I 2022-10-09 13:16:09] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:45 [I 2022-10-09 13:16:09] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00293 [I 2022-10-09 13:16:09] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-09 13:16:09] ...optimizers/trijoint.py.151: total_norm: 9.817649 [I 2022-10-09 13:16:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 2260/2304 [I 2022-10-09 13:16:18] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:37 [I 2022-10-09 13:16:18] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00295 [I 2022-10-09 13:16:18] ...trap/engines/engine.py.230: loss: 0.22391 [I 2022-10-09 13:16:18] ...optimizers/trijoint.py.151: total_norm: 34.098297 [I 2022-10-09 13:16:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 2270/2304 [I 2022-10-09 13:16:26] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:28 [I 2022-10-09 13:16:26] ...trap/engines/engine.py.229: process: 0.84003 | load: 0.00296 [I 2022-10-09 13:16:26] ...trap/engines/engine.py.230: loss: 0.25547 [I 2022-10-09 13:16:26] ...optimizers/trijoint.py.151: total_norm: 18.040445 [I 2022-10-09 13:16:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 2280/2304 [I 2022-10-09 13:16:35] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:20 [I 2022-10-09 13:16:35] ...trap/engines/engine.py.229: process: 0.86189 | load: 0.00294 [I 2022-10-09 13:16:35] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-09 13:16:35] ...optimizers/trijoint.py.151: total_norm: 13.390220 [I 2022-10-09 13:16:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 2290/2304 [I 2022-10-09 13:16:43] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:11 [I 2022-10-09 13:16:43] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00297 [I 2022-10-09 13:16:43] ...trap/engines/engine.py.230: loss: 0.17212 [I 2022-10-09 13:16:43] ...optimizers/trijoint.py.151: total_norm: 17.835815 [I 2022-10-09 13:16:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 2300/2304 [I 2022-10-09 13:16:52] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:03 [I 2022-10-09 13:16:52] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00015 [I 2022-10-09 13:16:52] ...trap/engines/engine.py.230: loss: 0.17445 [I 2022-10-09 13:16:52] ...optimizers/trijoint.py.151: total_norm: 22.301041 [I 2022-10-09 13:16:55] ...trap/engines/engine.py.224: train: epoch 41 | batch 2304/2304 [I 2022-10-09 13:16:55] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:00 [I 2022-10-09 13:16:55] ...trap/engines/engine.py.229: process: 0.82558 | load: 0.00016 [I 2022-10-09 13:16:55] ...trap/engines/engine.py.230: loss: 0.12844 [I 2022-10-09 13:16:55] ...optimizers/trijoint.py.151: total_norm: 13.960406 [S 2022-10-09 13:16:56] ...trap/engines/engine.py.129: train_epoch.epoch: 41 [S 2022-10-09 13:16:56] ...trap/engines/engine.py.129: train_epoch.loss: 0.15872129723591297 [I 2022-10-09 13:16:56] ...s/criterions/triplet.py.84: margin = 0.25500000000000006 [I 2022-10-09 13:16:57] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 41 [I 2022-10-09 13:16:57] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 13:16:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 13:16:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 13:16:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 13:17:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 13:17:01] ...trap/engines/engine.py.321: eval: epoch 41 | batch 0/510 [I 2022-10-09 13:17:01] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:31:20 [I 2022-10-09 13:17:01] ...trap/engines/engine.py.326: process: 3.68708 | load: 3.02059 [I 2022-10-09 13:17:05] ...trap/engines/engine.py.321: eval: epoch 41 | batch 10/510 [I 2022-10-09 13:17:05] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:30 [I 2022-10-09 13:17:05] ...trap/engines/engine.py.326: process: 0.43325 | load: 0.00026 [I 2022-10-09 13:17:10] ...trap/engines/engine.py.321: eval: epoch 41 | batch 20/510 [I 2022-10-09 13:17:10] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:50 [I 2022-10-09 13:17:10] ...trap/engines/engine.py.326: process: 0.71047 | load: 0.00019 [I 2022-10-09 13:17:16] ...trap/engines/engine.py.321: eval: epoch 41 | batch 30/510 [I 2022-10-09 13:17:16] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:25 [I 2022-10-09 13:17:16] ...trap/engines/engine.py.326: process: 0.84379 | load: 0.00019 [I 2022-10-09 13:17:22] ...trap/engines/engine.py.321: eval: epoch 41 | batch 40/510 [I 2022-10-09 13:17:22] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:56 [I 2022-10-09 13:17:22] ...trap/engines/engine.py.326: process: 0.43546 | load: 0.00018 [I 2022-10-09 13:17:27] ...trap/engines/engine.py.321: eval: epoch 41 | batch 50/510 [I 2022-10-09 13:17:27] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:20 [I 2022-10-09 13:17:27] ...trap/engines/engine.py.326: process: 0.59849 | load: 0.00018 [I 2022-10-09 13:17:33] ...trap/engines/engine.py.321: eval: epoch 41 | batch 60/510 [I 2022-10-09 13:17:33] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:27 [I 2022-10-09 13:17:33] ...trap/engines/engine.py.326: process: 0.60376 | load: 0.00017 [I 2022-10-09 13:17:39] ...trap/engines/engine.py.321: eval: epoch 41 | batch 70/510 [I 2022-10-09 13:17:39] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:33 [I 2022-10-09 13:17:39] ...trap/engines/engine.py.326: process: 0.67061 | load: 0.00019 [I 2022-10-09 13:17:45] ...trap/engines/engine.py.321: eval: epoch 41 | batch 80/510 [I 2022-10-09 13:17:45] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:47 [I 2022-10-09 13:17:45] ...trap/engines/engine.py.326: process: 0.43690 | load: 0.00018 [I 2022-10-09 13:17:49] ...trap/engines/engine.py.321: eval: epoch 41 | batch 90/510 [I 2022-10-09 13:17:49] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:10 [I 2022-10-09 13:17:49] ...trap/engines/engine.py.326: process: 0.41645 | load: 0.00018 [I 2022-10-09 13:17:53] ...trap/engines/engine.py.321: eval: epoch 41 | batch 100/510 [I 2022-10-09 13:17:53] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:45 [I 2022-10-09 13:17:53] ...trap/engines/engine.py.326: process: 0.31193 | load: 0.00018 [I 2022-10-09 13:17:56] ...trap/engines/engine.py.321: eval: epoch 41 | batch 110/510 [I 2022-10-09 13:17:56] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:09 [I 2022-10-09 13:17:56] ...trap/engines/engine.py.326: process: 0.31338 | load: 0.00019 [I 2022-10-09 13:18:00] ...trap/engines/engine.py.321: eval: epoch 41 | batch 120/510 [I 2022-10-09 13:18:00] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:03 [I 2022-10-09 13:18:00] ...trap/engines/engine.py.326: process: 0.31571 | load: 0.00019 [I 2022-10-09 13:18:03] ...trap/engines/engine.py.321: eval: epoch 41 | batch 130/510 [I 2022-10-09 13:18:03] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:59 [I 2022-10-09 13:18:03] ...trap/engines/engine.py.326: process: 0.31563 | load: 0.00018 [I 2022-10-09 13:18:06] ...trap/engines/engine.py.321: eval: epoch 41 | batch 140/510 [I 2022-10-09 13:18:06] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:56 [I 2022-10-09 13:18:06] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00019 [I 2022-10-09 13:18:09] ...trap/engines/engine.py.321: eval: epoch 41 | batch 150/510 [I 2022-10-09 13:18:09] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:53 [I 2022-10-09 13:18:09] ...trap/engines/engine.py.326: process: 0.31312 | load: 0.00019 [I 2022-10-09 13:18:12] ...trap/engines/engine.py.321: eval: epoch 41 | batch 160/510 [I 2022-10-09 13:18:12] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:50 [I 2022-10-09 13:18:12] ...trap/engines/engine.py.326: process: 0.31544 | load: 0.00020 [I 2022-10-09 13:18:15] ...trap/engines/engine.py.321: eval: epoch 41 | batch 170/510 [I 2022-10-09 13:18:15] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:47 [I 2022-10-09 13:18:15] ...trap/engines/engine.py.326: process: 0.31812 | load: 0.00019 [I 2022-10-09 13:18:19] ...trap/engines/engine.py.321: eval: epoch 41 | batch 180/510 [I 2022-10-09 13:18:19] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:44 [I 2022-10-09 13:18:19] ...trap/engines/engine.py.326: process: 0.31890 | load: 0.00019 [I 2022-10-09 13:18:22] ...trap/engines/engine.py.321: eval: epoch 41 | batch 190/510 [I 2022-10-09 13:18:22] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:42 [I 2022-10-09 13:18:22] ...trap/engines/engine.py.326: process: 0.32539 | load: 0.00019 [I 2022-10-09 13:18:25] ...trap/engines/engine.py.321: eval: epoch 41 | batch 200/510 [I 2022-10-09 13:18:25] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-09 13:18:25] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00021 [I 2022-10-09 13:18:28] ...trap/engines/engine.py.321: eval: epoch 41 | batch 210/510 [I 2022-10-09 13:18:28] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-09 13:18:28] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00018 [I 2022-10-09 13:18:31] ...trap/engines/engine.py.321: eval: epoch 41 | batch 220/510 [I 2022-10-09 13:18:31] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:32 [I 2022-10-09 13:18:31] ...trap/engines/engine.py.326: process: 0.31780 | load: 0.00019 [I 2022-10-09 13:18:35] ...trap/engines/engine.py.321: eval: epoch 41 | batch 230/510 [I 2022-10-09 13:18:35] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:29 [I 2022-10-09 13:18:35] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00019 [I 2022-10-09 13:18:38] ...trap/engines/engine.py.321: eval: epoch 41 | batch 240/510 [I 2022-10-09 13:18:38] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-09 13:18:38] ...trap/engines/engine.py.326: process: 0.32184 | load: 0.00020 [I 2022-10-09 13:18:41] ...trap/engines/engine.py.321: eval: epoch 41 | batch 250/510 [I 2022-10-09 13:18:41] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:22 [I 2022-10-09 13:18:41] ...trap/engines/engine.py.326: process: 0.32054 | load: 0.00019 [I 2022-10-09 13:18:44] ...trap/engines/engine.py.321: eval: epoch 41 | batch 260/510 [I 2022-10-09 13:18:44] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:19 [I 2022-10-09 13:18:44] ...trap/engines/engine.py.326: process: 0.32031 | load: 0.00019 [I 2022-10-09 13:18:47] ...trap/engines/engine.py.321: eval: epoch 41 | batch 270/510 [I 2022-10-09 13:18:47] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:16 [I 2022-10-09 13:18:47] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00021 [I 2022-10-09 13:18:51] ...trap/engines/engine.py.321: eval: epoch 41 | batch 280/510 [I 2022-10-09 13:18:51] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:12 [I 2022-10-09 13:18:51] ...trap/engines/engine.py.326: process: 0.30972 | load: 0.00019 [I 2022-10-09 13:18:54] ...trap/engines/engine.py.321: eval: epoch 41 | batch 290/510 [I 2022-10-09 13:18:54] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:09 [I 2022-10-09 13:18:54] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00019 [I 2022-10-09 13:18:57] ...trap/engines/engine.py.321: eval: epoch 41 | batch 300/510 [I 2022-10-09 13:18:57] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-09 13:18:57] ...trap/engines/engine.py.326: process: 0.31922 | load: 0.00020 [I 2022-10-09 13:19:00] ...trap/engines/engine.py.321: eval: epoch 41 | batch 310/510 [I 2022-10-09 13:19:00] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:02 [I 2022-10-09 13:19:00] ...trap/engines/engine.py.326: process: 0.31210 | load: 0.00019 [I 2022-10-09 13:19:03] ...trap/engines/engine.py.321: eval: epoch 41 | batch 320/510 [I 2022-10-09 13:19:03] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-09 13:19:03] ...trap/engines/engine.py.326: process: 0.31412 | load: 0.00020 [I 2022-10-09 13:19:06] ...trap/engines/engine.py.321: eval: epoch 41 | batch 330/510 [I 2022-10-09 13:19:06] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-09 13:19:06] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00020 [I 2022-10-09 13:19:10] ...trap/engines/engine.py.321: eval: epoch 41 | batch 340/510 [I 2022-10-09 13:19:10] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:53 [I 2022-10-09 13:19:10] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00020 [I 2022-10-09 13:19:13] ...trap/engines/engine.py.321: eval: epoch 41 | batch 350/510 [I 2022-10-09 13:19:13] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:50 [I 2022-10-09 13:19:13] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00019 [I 2022-10-09 13:19:16] ...trap/engines/engine.py.321: eval: epoch 41 | batch 360/510 [I 2022-10-09 13:19:16] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-09 13:19:16] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00020 [I 2022-10-09 13:19:19] ...trap/engines/engine.py.321: eval: epoch 41 | batch 370/510 [I 2022-10-09 13:19:19] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-09 13:19:19] ...trap/engines/engine.py.326: process: 0.31679 | load: 0.00020 [I 2022-10-09 13:19:22] ...trap/engines/engine.py.321: eval: epoch 41 | batch 380/510 [I 2022-10-09 13:19:22] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-09 13:19:22] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00020 [I 2022-10-09 13:19:25] ...trap/engines/engine.py.321: eval: epoch 41 | batch 390/510 [I 2022-10-09 13:19:25] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:37 [I 2022-10-09 13:19:25] ...trap/engines/engine.py.326: process: 0.31621 | load: 0.00019 [I 2022-10-09 13:19:29] ...trap/engines/engine.py.321: eval: epoch 41 | batch 400/510 [I 2022-10-09 13:19:29] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:34 [I 2022-10-09 13:19:29] ...trap/engines/engine.py.326: process: 0.31659 | load: 0.00021 [I 2022-10-09 13:19:32] ...trap/engines/engine.py.321: eval: epoch 41 | batch 410/510 [I 2022-10-09 13:19:32] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-09 13:19:32] ...trap/engines/engine.py.326: process: 0.31202 | load: 0.00020 [I 2022-10-09 13:19:35] ...trap/engines/engine.py.321: eval: epoch 41 | batch 420/510 [I 2022-10-09 13:19:35] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-09 13:19:35] ...trap/engines/engine.py.326: process: 0.31766 | load: 0.00019 [I 2022-10-09 13:19:38] ...trap/engines/engine.py.321: eval: epoch 41 | batch 430/510 [I 2022-10-09 13:19:38] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-09 13:19:38] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00020 [I 2022-10-09 13:19:41] ...trap/engines/engine.py.321: eval: epoch 41 | batch 440/510 [I 2022-10-09 13:19:41] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-09 13:19:41] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00020 [I 2022-10-09 13:19:45] ...trap/engines/engine.py.321: eval: epoch 41 | batch 450/510 [I 2022-10-09 13:19:45] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 13:19:45] ...trap/engines/engine.py.326: process: 0.31232 | load: 0.00019 [I 2022-10-09 13:19:48] ...trap/engines/engine.py.321: eval: epoch 41 | batch 460/510 [I 2022-10-09 13:19:48] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-09 13:19:48] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00019 [I 2022-10-09 13:19:51] ...trap/engines/engine.py.321: eval: epoch 41 | batch 470/510 [I 2022-10-09 13:19:51] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 13:19:51] ...trap/engines/engine.py.326: process: 0.31716 | load: 0.00021 [I 2022-10-09 13:19:54] ...trap/engines/engine.py.321: eval: epoch 41 | batch 480/510 [I 2022-10-09 13:19:54] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-09 13:19:54] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00020 [I 2022-10-09 13:19:57] ...trap/engines/engine.py.321: eval: epoch 41 | batch 490/510 [I 2022-10-09 13:19:57] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 13:19:57] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00019 [I 2022-10-09 13:20:00] ...trap/engines/engine.py.321: eval: epoch 41 | batch 500/510 [I 2022-10-09 13:20:00] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 13:20:00] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00020 [I 2022-10-09 13:20:04] ...trap/engines/engine.py.321: eval: epoch 41 | batch 510/510 [I 2022-10-09 13:20:04] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 13:20:04] ...trap/engines/engine.py.326: process: 0.31223 | load: 0.00020 [S 2022-10-09 13:20:04] ...trap/engines/engine.py.132: eval_epoch.epoch: 41 [I 2022-10-09 13:20:04] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.719600 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908100 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941800 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.710400 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905400 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936400 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016409 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008994 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004308 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013500 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006960 [S 2022-10-09 13:20:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005987 [I 2022-10-09 13:20:40] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 13:20:40] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 13:20:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 13:20:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 13:20:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 13:20:44] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 13:20:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 13:20:50] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 13:20:50] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 13:20:50] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 13:20:53] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 13:20:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 13:20:59] ...trap/engines/engine.py.163: Training model on trainset for epoch 42 [I 2022-10-09 13:20:59] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 13:21:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 0/2304 [I 2022-10-09 13:21:02] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:52 [I 2022-10-09 13:21:02] ...trap/engines/engine.py.229: process: 2.78304 | load: 1.87819 [I 2022-10-09 13:21:02] ...trap/engines/engine.py.230: loss: 0.14590 [I 2022-10-09 13:21:02] ...optimizers/trijoint.py.151: total_norm: 13.353601 [I 2022-10-09 13:21:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 10/2304 [I 2022-10-09 13:21:11] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:15 [I 2022-10-09 13:21:11] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00550 [I 2022-10-09 13:21:11] ...trap/engines/engine.py.230: loss: 0.11983 [I 2022-10-09 13:21:11] ...optimizers/trijoint.py.151: total_norm: 7.519316 [I 2022-10-09 13:21:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 20/2304 [I 2022-10-09 13:21:19] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-09 13:21:19] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00497 [I 2022-10-09 13:21:19] ...trap/engines/engine.py.230: loss: 0.10992 [I 2022-10-09 13:21:19] ...optimizers/trijoint.py.151: total_norm: 17.254385 [I 2022-10-09 13:21:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 30/2304 [I 2022-10-09 13:21:27] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:14 [I 2022-10-09 13:21:27] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00413 [I 2022-10-09 13:21:27] ...trap/engines/engine.py.230: loss: 0.19592 [I 2022-10-09 13:21:27] ...optimizers/trijoint.py.151: total_norm: 20.822699 [I 2022-10-09 13:21:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 40/2304 [I 2022-10-09 13:21:36] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:56 [I 2022-10-09 13:21:36] ...trap/engines/engine.py.229: process: 0.85403 | load: 0.00334 [I 2022-10-09 13:21:36] ...trap/engines/engine.py.230: loss: 0.11325 [I 2022-10-09 13:21:36] ...optimizers/trijoint.py.151: total_norm: 14.067651 [I 2022-10-09 13:21:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 50/2304 [I 2022-10-09 13:21:44] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:49 [I 2022-10-09 13:21:44] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00390 [I 2022-10-09 13:21:44] ...trap/engines/engine.py.230: loss: 0.15664 [I 2022-10-09 13:21:44] ...optimizers/trijoint.py.151: total_norm: 23.164238 [I 2022-10-09 13:21:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 60/2304 [I 2022-10-09 13:21:53] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:46 [I 2022-10-09 13:21:53] ...trap/engines/engine.py.229: process: 0.87434 | load: 0.00358 [I 2022-10-09 13:21:53] ...trap/engines/engine.py.230: loss: 0.13485 [I 2022-10-09 13:21:53] ...optimizers/trijoint.py.151: total_norm: 10.989673 [I 2022-10-09 13:22:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 70/2304 [I 2022-10-09 13:22:01] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:42 [I 2022-10-09 13:22:01] ...trap/engines/engine.py.229: process: 0.86156 | load: 0.00319 [I 2022-10-09 13:22:01] ...trap/engines/engine.py.230: loss: 0.15501 [I 2022-10-09 13:22:01] ...optimizers/trijoint.py.151: total_norm: 15.960342 [I 2022-10-09 13:22:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 80/2304 [I 2022-10-09 13:22:10] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:34 [I 2022-10-09 13:22:10] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00328 [I 2022-10-09 13:22:10] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-09 13:22:10] ...optimizers/trijoint.py.151: total_norm: 16.638889 [I 2022-10-09 13:22:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 90/2304 [I 2022-10-09 13:22:18] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:12 [I 2022-10-09 13:22:18] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00333 [I 2022-10-09 13:22:18] ...trap/engines/engine.py.230: loss: 0.18594 [I 2022-10-09 13:22:18] ...optimizers/trijoint.py.151: total_norm: 13.600913 [I 2022-10-09 13:22:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 100/2304 [I 2022-10-09 13:22:27] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:05 [I 2022-10-09 13:22:27] ...trap/engines/engine.py.229: process: 0.86346 | load: 0.00326 [I 2022-10-09 13:22:27] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-09 13:22:27] ...optimizers/trijoint.py.151: total_norm: 28.189785 [I 2022-10-09 13:22:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 110/2304 [I 2022-10-09 13:22:35] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:01 [I 2022-10-09 13:22:35] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00334 [I 2022-10-09 13:22:35] ...trap/engines/engine.py.230: loss: 0.11207 [I 2022-10-09 13:22:35] ...optimizers/trijoint.py.151: total_norm: 9.701085 [I 2022-10-09 13:22:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 120/2304 [I 2022-10-09 13:22:44] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:31:02 [I 2022-10-09 13:22:44] ...trap/engines/engine.py.229: process: 0.86542 | load: 0.00313 [I 2022-10-09 13:22:44] ...trap/engines/engine.py.230: loss: 0.20333 [I 2022-10-09 13:22:44] ...optimizers/trijoint.py.151: total_norm: 16.283123 [I 2022-10-09 13:22:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 130/2304 [I 2022-10-09 13:22:52] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:39 [I 2022-10-09 13:22:52] ...trap/engines/engine.py.229: process: 0.83850 | load: 0.00331 [I 2022-10-09 13:22:52] ...trap/engines/engine.py.230: loss: 0.21457 [I 2022-10-09 13:22:52] ...optimizers/trijoint.py.151: total_norm: 18.888073 [I 2022-10-09 13:23:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 140/2304 [I 2022-10-09 13:23:01] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:38 [I 2022-10-09 13:23:01] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00307 [I 2022-10-09 13:23:01] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-09 13:23:01] ...optimizers/trijoint.py.151: total_norm: 10.025252 [I 2022-10-09 13:23:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 150/2304 [I 2022-10-09 13:23:09] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:18 [I 2022-10-09 13:23:09] ...trap/engines/engine.py.229: process: 0.82812 | load: 0.00295 [I 2022-10-09 13:23:09] ...trap/engines/engine.py.230: loss: 0.23443 [I 2022-10-09 13:23:09] ...optimizers/trijoint.py.151: total_norm: 31.953415 [I 2022-10-09 13:23:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 160/2304 [I 2022-10-09 13:23:18] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:28 [I 2022-10-09 13:23:18] ...trap/engines/engine.py.229: process: 0.86807 | load: 0.00331 [I 2022-10-09 13:23:18] ...trap/engines/engine.py.230: loss: 0.18514 [I 2022-10-09 13:23:18] ...optimizers/trijoint.py.151: total_norm: 22.469959 [I 2022-10-09 13:23:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 170/2304 [I 2022-10-09 13:23:26] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:05 [I 2022-10-09 13:23:26] ...trap/engines/engine.py.229: process: 0.82419 | load: 0.00296 [I 2022-10-09 13:23:26] ...trap/engines/engine.py.230: loss: 0.12601 [I 2022-10-09 13:23:26] ...optimizers/trijoint.py.151: total_norm: 8.465665 [I 2022-10-09 13:23:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 180/2304 [I 2022-10-09 13:23:35] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:30:06 [I 2022-10-09 13:23:35] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00305 [I 2022-10-09 13:23:35] ...trap/engines/engine.py.230: loss: 0.29715 [I 2022-10-09 13:23:35] ...optimizers/trijoint.py.151: total_norm: 11.566459 [I 2022-10-09 13:23:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 190/2304 [I 2022-10-09 13:23:43] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:46 [I 2022-10-09 13:23:43] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00314 [I 2022-10-09 13:23:43] ...trap/engines/engine.py.230: loss: 0.19265 [I 2022-10-09 13:23:43] ...optimizers/trijoint.py.151: total_norm: 23.095434 [I 2022-10-09 13:23:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 200/2304 [I 2022-10-09 13:23:52] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:31 [I 2022-10-09 13:23:52] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00311 [I 2022-10-09 13:23:52] ...trap/engines/engine.py.230: loss: 0.19002 [I 2022-10-09 13:23:52] ...optimizers/trijoint.py.151: total_norm: 24.239090 [I 2022-10-09 13:24:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 210/2304 [I 2022-10-09 13:24:00] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:16 [I 2022-10-09 13:24:00] ...trap/engines/engine.py.229: process: 0.82467 | load: 0.00308 [I 2022-10-09 13:24:00] ...trap/engines/engine.py.230: loss: 0.13956 [I 2022-10-09 13:24:00] ...optimizers/trijoint.py.151: total_norm: 10.053423 [I 2022-10-09 13:24:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 220/2304 [I 2022-10-09 13:24:09] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:23 [I 2022-10-09 13:24:09] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00316 [I 2022-10-09 13:24:09] ...trap/engines/engine.py.230: loss: 0.20603 [I 2022-10-09 13:24:09] ...optimizers/trijoint.py.151: total_norm: 22.923691 [I 2022-10-09 13:24:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 230/2304 [I 2022-10-09 13:24:17] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:22 [I 2022-10-09 13:24:17] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00292 [I 2022-10-09 13:24:17] ...trap/engines/engine.py.230: loss: 0.11948 [I 2022-10-09 13:24:17] ...optimizers/trijoint.py.151: total_norm: 11.515500 [I 2022-10-09 13:24:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 240/2304 [I 2022-10-09 13:24:26] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:09 [I 2022-10-09 13:24:26] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00295 [I 2022-10-09 13:24:26] ...trap/engines/engine.py.230: loss: 0.12076 [I 2022-10-09 13:24:26] ...optimizers/trijoint.py.151: total_norm: 15.589643 [I 2022-10-09 13:24:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 250/2304 [I 2022-10-09 13:24:34] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:00 [I 2022-10-09 13:24:34] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00295 [I 2022-10-09 13:24:34] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-09 13:24:34] ...optimizers/trijoint.py.151: total_norm: 11.768313 [I 2022-10-09 13:24:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 260/2304 [I 2022-10-09 13:24:43] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:44 [I 2022-10-09 13:24:43] ...trap/engines/engine.py.229: process: 0.83469 | load: 0.00299 [I 2022-10-09 13:24:43] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-09 13:24:43] ...optimizers/trijoint.py.151: total_norm: 15.385905 [I 2022-10-09 13:24:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 270/2304 [I 2022-10-09 13:24:51] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:35 [I 2022-10-09 13:24:51] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00303 [I 2022-10-09 13:24:51] ...trap/engines/engine.py.230: loss: 0.15594 [I 2022-10-09 13:24:51] ...optimizers/trijoint.py.151: total_norm: 24.341429 [I 2022-10-09 13:24:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 280/2304 [I 2022-10-09 13:24:59] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:19 [I 2022-10-09 13:24:59] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00303 [I 2022-10-09 13:24:59] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-09 13:24:59] ...optimizers/trijoint.py.151: total_norm: 16.692917 [I 2022-10-09 13:25:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 290/2304 [I 2022-10-09 13:25:08] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:18 [I 2022-10-09 13:25:08] ...trap/engines/engine.py.229: process: 0.83707 | load: 0.00301 [I 2022-10-09 13:25:08] ...trap/engines/engine.py.230: loss: 0.22180 [I 2022-10-09 13:25:08] ...optimizers/trijoint.py.151: total_norm: 50.150261 [I 2022-10-09 13:25:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 300/2304 [I 2022-10-09 13:25:16] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:27:59 [I 2022-10-09 13:25:16] ...trap/engines/engine.py.229: process: 0.82921 | load: 0.00294 [I 2022-10-09 13:25:16] ...trap/engines/engine.py.230: loss: 0.12919 [I 2022-10-09 13:25:16] ...optimizers/trijoint.py.151: total_norm: 18.041920 [I 2022-10-09 13:25:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 310/2304 [I 2022-10-09 13:25:25] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:27:57 [I 2022-10-09 13:25:25] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00310 [I 2022-10-09 13:25:25] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-09 13:25:25] ...optimizers/trijoint.py.151: total_norm: 12.904061 [I 2022-10-09 13:25:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 320/2304 [I 2022-10-09 13:25:33] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:51 [I 2022-10-09 13:25:33] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00295 [I 2022-10-09 13:25:33] ...trap/engines/engine.py.230: loss: 0.23269 [I 2022-10-09 13:25:33] ...optimizers/trijoint.py.151: total_norm: 12.858766 [I 2022-10-09 13:25:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 330/2304 [I 2022-10-09 13:25:42] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:40 [I 2022-10-09 13:25:42] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00290 [I 2022-10-09 13:25:42] ...trap/engines/engine.py.230: loss: 0.15990 [I 2022-10-09 13:25:42] ...optimizers/trijoint.py.151: total_norm: 15.776663 [I 2022-10-09 13:25:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 340/2304 [I 2022-10-09 13:25:50] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:37 [I 2022-10-09 13:25:50] ...trap/engines/engine.py.229: process: 0.84775 | load: 0.00322 [I 2022-10-09 13:25:50] ...trap/engines/engine.py.230: loss: 0.11598 [I 2022-10-09 13:25:50] ...optimizers/trijoint.py.151: total_norm: 20.167772 [I 2022-10-09 13:25:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 350/2304 [I 2022-10-09 13:25:58] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:32 [I 2022-10-09 13:25:58] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00293 [I 2022-10-09 13:25:58] ...trap/engines/engine.py.230: loss: 0.17885 [I 2022-10-09 13:25:58] ...optimizers/trijoint.py.151: total_norm: 28.891750 [I 2022-10-09 13:26:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 360/2304 [I 2022-10-09 13:26:07] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:18 [I 2022-10-09 13:26:07] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00298 [I 2022-10-09 13:26:07] ...trap/engines/engine.py.230: loss: 0.23571 [I 2022-10-09 13:26:07] ...optimizers/trijoint.py.151: total_norm: 15.218102 [I 2022-10-09 13:26:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 370/2304 [I 2022-10-09 13:26:15] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:26:55 [I 2022-10-09 13:26:15] ...trap/engines/engine.py.229: process: 0.83193 | load: 0.00294 [I 2022-10-09 13:26:15] ...trap/engines/engine.py.230: loss: 0.22653 [I 2022-10-09 13:26:15] ...optimizers/trijoint.py.151: total_norm: 58.025196 [I 2022-10-09 13:26:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 380/2304 [I 2022-10-09 13:26:24] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:26:53 [I 2022-10-09 13:26:24] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00314 [I 2022-10-09 13:26:24] ...trap/engines/engine.py.230: loss: 0.24447 [I 2022-10-09 13:26:24] ...optimizers/trijoint.py.151: total_norm: 20.340439 [I 2022-10-09 13:26:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 390/2304 [I 2022-10-09 13:26:32] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:58 [I 2022-10-09 13:26:32] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00307 [I 2022-10-09 13:26:32] ...trap/engines/engine.py.230: loss: 0.17818 [I 2022-10-09 13:26:32] ...optimizers/trijoint.py.151: total_norm: 9.581172 [I 2022-10-09 13:26:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 400/2304 [I 2022-10-09 13:26:41] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:48 [I 2022-10-09 13:26:41] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00290 [I 2022-10-09 13:26:41] ...trap/engines/engine.py.230: loss: 0.18283 [I 2022-10-09 13:26:41] ...optimizers/trijoint.py.151: total_norm: 21.399324 [I 2022-10-09 13:26:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 410/2304 [I 2022-10-09 13:26:49] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:31 [I 2022-10-09 13:26:49] ...trap/engines/engine.py.229: process: 0.82695 | load: 0.00318 [I 2022-10-09 13:26:49] ...trap/engines/engine.py.230: loss: 0.14402 [I 2022-10-09 13:26:49] ...optimizers/trijoint.py.151: total_norm: 29.677204 [I 2022-10-09 13:26:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 420/2304 [I 2022-10-09 13:26:57] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:28 [I 2022-10-09 13:26:57] ...trap/engines/engine.py.229: process: 0.83920 | load: 0.00293 [I 2022-10-09 13:26:57] ...trap/engines/engine.py.230: loss: 0.19696 [I 2022-10-09 13:26:57] ...optimizers/trijoint.py.151: total_norm: 28.663033 [I 2022-10-09 13:27:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 430/2304 [I 2022-10-09 13:27:06] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:21 [I 2022-10-09 13:27:06] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00295 [I 2022-10-09 13:27:06] ...trap/engines/engine.py.230: loss: 0.18399 [I 2022-10-09 13:27:06] ...optimizers/trijoint.py.151: total_norm: 9.411531 [I 2022-10-09 13:27:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 440/2304 [I 2022-10-09 13:27:14] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:01 [I 2022-10-09 13:27:14] ...trap/engines/engine.py.229: process: 0.83378 | load: 0.00297 [I 2022-10-09 13:27:14] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-09 13:27:14] ...optimizers/trijoint.py.151: total_norm: 9.983604 [I 2022-10-09 13:27:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 450/2304 [I 2022-10-09 13:27:23] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:05 [I 2022-10-09 13:27:23] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00305 [I 2022-10-09 13:27:23] ...trap/engines/engine.py.230: loss: 0.17028 [I 2022-10-09 13:27:23] ...optimizers/trijoint.py.151: total_norm: 17.354998 [I 2022-10-09 13:27:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 460/2304 [I 2022-10-09 13:27:31] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:25:58 [I 2022-10-09 13:27:31] ...trap/engines/engine.py.229: process: 0.85053 | load: 0.00297 [I 2022-10-09 13:27:31] ...trap/engines/engine.py.230: loss: 0.13701 [I 2022-10-09 13:27:31] ...optimizers/trijoint.py.151: total_norm: 15.358036 [I 2022-10-09 13:27:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 470/2304 [I 2022-10-09 13:27:40] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:51 [I 2022-10-09 13:27:40] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00297 [I 2022-10-09 13:27:40] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-09 13:27:40] ...optimizers/trijoint.py.151: total_norm: 33.739319 [I 2022-10-09 13:27:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 480/2304 [I 2022-10-09 13:27:48] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:29 [I 2022-10-09 13:27:48] ...trap/engines/engine.py.229: process: 0.83429 | load: 0.00329 [I 2022-10-09 13:27:48] ...trap/engines/engine.py.230: loss: 0.20312 [I 2022-10-09 13:27:48] ...optimizers/trijoint.py.151: total_norm: 13.871243 [I 2022-10-09 13:27:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 490/2304 [I 2022-10-09 13:27:57] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:32 [I 2022-10-09 13:27:57] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00295 [I 2022-10-09 13:27:57] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-09 13:27:57] ...optimizers/trijoint.py.151: total_norm: 14.991299 [I 2022-10-09 13:28:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 500/2304 [I 2022-10-09 13:28:05] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:18 [I 2022-10-09 13:28:05] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00323 [I 2022-10-09 13:28:05] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-09 13:28:05] ...optimizers/trijoint.py.151: total_norm: 11.041660 [I 2022-10-09 13:28:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 510/2304 [I 2022-10-09 13:28:14] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:19 [I 2022-10-09 13:28:14] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00297 [I 2022-10-09 13:28:14] ...trap/engines/engine.py.230: loss: 0.11209 [I 2022-10-09 13:28:14] ...optimizers/trijoint.py.151: total_norm: 40.187336 [I 2022-10-09 13:28:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 520/2304 [I 2022-10-09 13:28:22] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:24:59 [I 2022-10-09 13:28:22] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00294 [I 2022-10-09 13:28:22] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-09 13:28:22] ...optimizers/trijoint.py.151: total_norm: 12.586258 [I 2022-10-09 13:28:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 530/2304 [I 2022-10-09 13:28:30] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:24:53 [I 2022-10-09 13:28:30] ...trap/engines/engine.py.229: process: 0.84403 | load: 0.00295 [I 2022-10-09 13:28:30] ...trap/engines/engine.py.230: loss: 0.14391 [I 2022-10-09 13:28:30] ...optimizers/trijoint.py.151: total_norm: 14.085137 [I 2022-10-09 13:28:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 540/2304 [I 2022-10-09 13:28:39] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:51 [I 2022-10-09 13:28:39] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00301 [I 2022-10-09 13:28:39] ...trap/engines/engine.py.230: loss: 0.17476 [I 2022-10-09 13:28:39] ...optimizers/trijoint.py.151: total_norm: 9.559384 [I 2022-10-09 13:28:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 550/2304 [I 2022-10-09 13:28:47] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:43 [I 2022-10-09 13:28:47] ...trap/engines/engine.py.229: process: 0.85578 | load: 0.00295 [I 2022-10-09 13:28:47] ...trap/engines/engine.py.230: loss: 0.23204 [I 2022-10-09 13:28:47] ...optimizers/trijoint.py.151: total_norm: 18.795286 [I 2022-10-09 13:28:56] ...trap/engines/engine.py.224: train: epoch 42 | batch 560/2304 [I 2022-10-09 13:28:56] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:34 [I 2022-10-09 13:28:56] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00293 [I 2022-10-09 13:28:56] ...trap/engines/engine.py.230: loss: 0.17565 [I 2022-10-09 13:28:56] ...optimizers/trijoint.py.151: total_norm: 17.215828 [I 2022-10-09 13:29:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 570/2304 [I 2022-10-09 13:29:04] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:17 [I 2022-10-09 13:29:04] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00298 [I 2022-10-09 13:29:04] ...trap/engines/engine.py.230: loss: 0.19747 [I 2022-10-09 13:29:04] ...optimizers/trijoint.py.151: total_norm: 16.945278 [I 2022-10-09 13:29:13] ...trap/engines/engine.py.224: train: epoch 42 | batch 580/2304 [I 2022-10-09 13:29:13] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:21 [I 2022-10-09 13:29:13] ...trap/engines/engine.py.229: process: 0.84558 | load: 0.00299 [I 2022-10-09 13:29:13] ...trap/engines/engine.py.230: loss: 0.16659 [I 2022-10-09 13:29:13] ...optimizers/trijoint.py.151: total_norm: 27.191906 [I 2022-10-09 13:29:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 590/2304 [I 2022-10-09 13:29:21] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:13 [I 2022-10-09 13:29:21] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00302 [I 2022-10-09 13:29:21] ...trap/engines/engine.py.230: loss: 0.24660 [I 2022-10-09 13:29:21] ...optimizers/trijoint.py.151: total_norm: 21.048763 [I 2022-10-09 13:29:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 600/2304 [I 2022-10-09 13:29:30] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:23:57 [I 2022-10-09 13:29:30] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00331 [I 2022-10-09 13:29:30] ...trap/engines/engine.py.230: loss: 0.14081 [I 2022-10-09 13:29:30] ...optimizers/trijoint.py.151: total_norm: 11.906940 [I 2022-10-09 13:29:38] ...trap/engines/engine.py.224: train: epoch 42 | batch 610/2304 [I 2022-10-09 13:29:38] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:23:56 [I 2022-10-09 13:29:38] ...trap/engines/engine.py.229: process: 0.85132 | load: 0.00306 [I 2022-10-09 13:29:38] ...trap/engines/engine.py.230: loss: 0.20705 [I 2022-10-09 13:29:38] ...optimizers/trijoint.py.151: total_norm: 17.545702 [I 2022-10-09 13:29:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 620/2304 [I 2022-10-09 13:29:47] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:39 [I 2022-10-09 13:29:47] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00302 [I 2022-10-09 13:29:47] ...trap/engines/engine.py.230: loss: 0.14823 [I 2022-10-09 13:29:47] ...optimizers/trijoint.py.151: total_norm: 15.851384 [I 2022-10-09 13:29:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 630/2304 [I 2022-10-09 13:29:55] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:30 [I 2022-10-09 13:29:55] ...trap/engines/engine.py.229: process: 0.84223 | load: 0.00301 [I 2022-10-09 13:29:55] ...trap/engines/engine.py.230: loss: 0.13940 [I 2022-10-09 13:29:55] ...optimizers/trijoint.py.151: total_norm: 16.080490 [I 2022-10-09 13:30:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 640/2304 [I 2022-10-09 13:30:03] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:21 [I 2022-10-09 13:30:03] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00297 [I 2022-10-09 13:30:03] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-09 13:30:03] ...optimizers/trijoint.py.151: total_norm: 17.095795 [I 2022-10-09 13:30:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 650/2304 [I 2022-10-09 13:30:12] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:20 [I 2022-10-09 13:30:12] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00296 [I 2022-10-09 13:30:12] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-09 13:30:12] ...optimizers/trijoint.py.151: total_norm: 14.176433 [I 2022-10-09 13:30:20] ...trap/engines/engine.py.224: train: epoch 42 | batch 660/2304 [I 2022-10-09 13:30:20] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:04 [I 2022-10-09 13:30:20] ...trap/engines/engine.py.229: process: 0.84430 | load: 0.00297 [I 2022-10-09 13:30:20] ...trap/engines/engine.py.230: loss: 0.21801 [I 2022-10-09 13:30:20] ...optimizers/trijoint.py.151: total_norm: 49.379730 [I 2022-10-09 13:30:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 670/2304 [I 2022-10-09 13:30:29] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:22:51 [I 2022-10-09 13:30:29] ...trap/engines/engine.py.229: process: 0.82929 | load: 0.00294 [I 2022-10-09 13:30:29] ...trap/engines/engine.py.230: loss: 0.26158 [I 2022-10-09 13:30:29] ...optimizers/trijoint.py.151: total_norm: 55.569904 [I 2022-10-09 13:30:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 680/2304 [I 2022-10-09 13:30:37] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:22:57 [I 2022-10-09 13:30:37] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00291 [I 2022-10-09 13:30:37] ...trap/engines/engine.py.230: loss: 0.20948 [I 2022-10-09 13:30:37] ...optimizers/trijoint.py.151: total_norm: 22.975729 [I 2022-10-09 13:30:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 690/2304 [I 2022-10-09 13:30:46] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:45 [I 2022-10-09 13:30:46] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00295 [I 2022-10-09 13:30:46] ...trap/engines/engine.py.230: loss: 0.12085 [I 2022-10-09 13:30:46] ...optimizers/trijoint.py.151: total_norm: 11.427160 [I 2022-10-09 13:30:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 700/2304 [I 2022-10-09 13:30:54] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:30 [I 2022-10-09 13:30:54] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00303 [I 2022-10-09 13:30:54] ...trap/engines/engine.py.230: loss: 0.18918 [I 2022-10-09 13:30:54] ...optimizers/trijoint.py.151: total_norm: 24.644638 [I 2022-10-09 13:31:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 710/2304 [I 2022-10-09 13:31:03] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:24 [I 2022-10-09 13:31:03] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00295 [I 2022-10-09 13:31:03] ...trap/engines/engine.py.230: loss: 0.24148 [I 2022-10-09 13:31:03] ...optimizers/trijoint.py.151: total_norm: 23.758484 [I 2022-10-09 13:31:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 720/2304 [I 2022-10-09 13:31:11] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:21 [I 2022-10-09 13:31:11] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00297 [I 2022-10-09 13:31:11] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-09 13:31:11] ...optimizers/trijoint.py.151: total_norm: 12.785896 [I 2022-10-09 13:31:20] ...trap/engines/engine.py.224: train: epoch 42 | batch 730/2304 [I 2022-10-09 13:31:20] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:13 [I 2022-10-09 13:31:20] ...trap/engines/engine.py.229: process: 0.85307 | load: 0.00305 [I 2022-10-09 13:31:20] ...trap/engines/engine.py.230: loss: 0.13222 [I 2022-10-09 13:31:20] ...optimizers/trijoint.py.151: total_norm: 14.725368 [I 2022-10-09 13:31:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 740/2304 [I 2022-10-09 13:31:28] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:22:07 [I 2022-10-09 13:31:28] ...trap/engines/engine.py.229: process: 0.85156 | load: 0.00295 [I 2022-10-09 13:31:28] ...trap/engines/engine.py.230: loss: 0.20370 [I 2022-10-09 13:31:28] ...optimizers/trijoint.py.151: total_norm: 33.565529 [I 2022-10-09 13:31:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 750/2304 [I 2022-10-09 13:31:37] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:53 [I 2022-10-09 13:31:37] ...trap/engines/engine.py.229: process: 0.83849 | load: 0.00297 [I 2022-10-09 13:31:37] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-09 13:31:37] ...optimizers/trijoint.py.151: total_norm: 20.329990 [I 2022-10-09 13:31:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 760/2304 [I 2022-10-09 13:31:45] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:21:35 [I 2022-10-09 13:31:45] ...trap/engines/engine.py.229: process: 0.83046 | load: 0.00298 [I 2022-10-09 13:31:45] ...trap/engines/engine.py.230: loss: 0.22788 [I 2022-10-09 13:31:45] ...optimizers/trijoint.py.151: total_norm: 22.336226 [I 2022-10-09 13:31:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 770/2304 [I 2022-10-09 13:31:53] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:27 [I 2022-10-09 13:31:53] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00295 [I 2022-10-09 13:31:53] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-09 13:31:53] ...optimizers/trijoint.py.151: total_norm: 16.974251 [I 2022-10-09 13:32:20] ...trap/engines/engine.py.224: train: epoch 42 | batch 780/2304 [I 2022-10-09 13:32:20] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:33:54 [I 2022-10-09 13:32:20] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00302 [I 2022-10-09 13:32:20] ...trap/engines/engine.py.230: loss: 0.24006 [I 2022-10-09 13:32:20] ...optimizers/trijoint.py.151: total_norm: 35.826118 [I 2022-10-09 13:32:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 790/2304 [I 2022-10-09 13:32:28] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:22:36 [I 2022-10-09 13:32:28] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00292 [I 2022-10-09 13:32:28] ...trap/engines/engine.py.230: loss: 0.21842 [I 2022-10-09 13:32:28] ...optimizers/trijoint.py.151: total_norm: 19.813166 [I 2022-10-09 13:32:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 800/2304 [I 2022-10-09 13:32:37] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:21:15 [I 2022-10-09 13:32:37] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00293 [I 2022-10-09 13:32:37] ...trap/engines/engine.py.230: loss: 0.12302 [I 2022-10-09 13:32:37] ...optimizers/trijoint.py.151: total_norm: 12.620681 [I 2022-10-09 13:32:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 810/2304 [I 2022-10-09 13:32:45] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:51 [I 2022-10-09 13:32:45] ...trap/engines/engine.py.229: process: 0.82735 | load: 0.00293 [I 2022-10-09 13:32:45] ...trap/engines/engine.py.230: loss: 0.22567 [I 2022-10-09 13:32:45] ...optimizers/trijoint.py.151: total_norm: 14.144454 [I 2022-10-09 13:32:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 820/2304 [I 2022-10-09 13:32:54] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:20:53 [I 2022-10-09 13:32:54] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00304 [I 2022-10-09 13:32:54] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-09 13:32:54] ...optimizers/trijoint.py.151: total_norm: 12.267707 [I 2022-10-09 13:33:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 830/2304 [I 2022-10-09 13:33:02] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:20:41 [I 2022-10-09 13:33:02] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00294 [I 2022-10-09 13:33:02] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-09 13:33:02] ...optimizers/trijoint.py.151: total_norm: 11.605305 [I 2022-10-09 13:33:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 840/2304 [I 2022-10-09 13:33:11] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:20:38 [I 2022-10-09 13:33:11] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00294 [I 2022-10-09 13:33:11] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-09 13:33:11] ...optimizers/trijoint.py.151: total_norm: 16.816429 [I 2022-10-09 13:33:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 850/2304 [I 2022-10-09 13:33:19] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:23 [I 2022-10-09 13:33:19] ...trap/engines/engine.py.229: process: 0.83936 | load: 0.00299 [I 2022-10-09 13:33:19] ...trap/engines/engine.py.230: loss: 0.13377 [I 2022-10-09 13:33:19] ...optimizers/trijoint.py.151: total_norm: 17.206244 [I 2022-10-09 13:33:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 860/2304 [I 2022-10-09 13:33:28] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:20:17 [I 2022-10-09 13:33:28] ...trap/engines/engine.py.229: process: 0.84338 | load: 0.00291 [I 2022-10-09 13:33:28] ...trap/engines/engine.py.230: loss: 0.15091 [I 2022-10-09 13:33:28] ...optimizers/trijoint.py.151: total_norm: 14.157448 [I 2022-10-09 13:33:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 870/2304 [I 2022-10-09 13:33:36] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:20:01 [I 2022-10-09 13:33:36] ...trap/engines/engine.py.229: process: 0.83996 | load: 0.00292 [I 2022-10-09 13:33:36] ...trap/engines/engine.py.230: loss: 0.19519 [I 2022-10-09 13:33:36] ...optimizers/trijoint.py.151: total_norm: 9.815845 [I 2022-10-09 13:33:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 880/2304 [I 2022-10-09 13:33:44] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:20:02 [I 2022-10-09 13:33:44] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00297 [I 2022-10-09 13:33:44] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-09 13:33:44] ...optimizers/trijoint.py.151: total_norm: 9.579787 [I 2022-10-09 13:33:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 890/2304 [I 2022-10-09 13:33:53] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:48 [I 2022-10-09 13:33:53] ...trap/engines/engine.py.229: process: 0.83187 | load: 0.00296 [I 2022-10-09 13:33:53] ...trap/engines/engine.py.230: loss: 0.16518 [I 2022-10-09 13:33:53] ...optimizers/trijoint.py.151: total_norm: 15.293853 [I 2022-10-09 13:34:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 900/2304 [I 2022-10-09 13:34:01] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:19:46 [I 2022-10-09 13:34:01] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00291 [I 2022-10-09 13:34:01] ...trap/engines/engine.py.230: loss: 0.20710 [I 2022-10-09 13:34:01] ...optimizers/trijoint.py.151: total_norm: 14.602202 [I 2022-10-09 13:34:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 910/2304 [I 2022-10-09 13:34:10] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:40 [I 2022-10-09 13:34:10] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00292 [I 2022-10-09 13:34:10] ...trap/engines/engine.py.230: loss: 0.14324 [I 2022-10-09 13:34:10] ...optimizers/trijoint.py.151: total_norm: 11.561287 [I 2022-10-09 13:34:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 920/2304 [I 2022-10-09 13:34:18] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:32 [I 2022-10-09 13:34:18] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00299 [I 2022-10-09 13:34:18] ...trap/engines/engine.py.230: loss: 0.18129 [I 2022-10-09 13:34:18] ...optimizers/trijoint.py.151: total_norm: 18.133923 [I 2022-10-09 13:34:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 930/2304 [I 2022-10-09 13:34:27] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:23 [I 2022-10-09 13:34:27] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00297 [I 2022-10-09 13:34:27] ...trap/engines/engine.py.230: loss: 0.17949 [I 2022-10-09 13:34:27] ...optimizers/trijoint.py.151: total_norm: 11.134113 [I 2022-10-09 13:34:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 940/2304 [I 2022-10-09 13:34:35] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:14 [I 2022-10-09 13:34:35] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00297 [I 2022-10-09 13:34:35] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-09 13:34:35] ...optimizers/trijoint.py.151: total_norm: 10.656427 [I 2022-10-09 13:34:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 950/2304 [I 2022-10-09 13:34:44] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:05 [I 2022-10-09 13:34:44] ...trap/engines/engine.py.229: process: 0.84050 | load: 0.00296 [I 2022-10-09 13:34:44] ...trap/engines/engine.py.230: loss: 0.12170 [I 2022-10-09 13:34:44] ...optimizers/trijoint.py.151: total_norm: 10.397112 [I 2022-10-09 13:34:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 960/2304 [I 2022-10-09 13:34:52] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:54 [I 2022-10-09 13:34:52] ...trap/engines/engine.py.229: process: 0.84923 | load: 0.00304 [I 2022-10-09 13:34:52] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-09 13:34:52] ...optimizers/trijoint.py.151: total_norm: 38.922840 [I 2022-10-09 13:35:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 970/2304 [I 2022-10-09 13:35:01] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:18:50 [I 2022-10-09 13:35:01] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00295 [I 2022-10-09 13:35:01] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-09 13:35:01] ...optimizers/trijoint.py.151: total_norm: 10.908611 [I 2022-10-09 13:35:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 980/2304 [I 2022-10-09 13:35:09] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:29 [I 2022-10-09 13:35:09] ...trap/engines/engine.py.229: process: 0.83036 | load: 0.00303 [I 2022-10-09 13:35:09] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-09 13:35:09] ...optimizers/trijoint.py.151: total_norm: 11.173578 [I 2022-10-09 13:35:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 990/2304 [I 2022-10-09 13:35:17] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:29 [I 2022-10-09 13:35:17] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00295 [I 2022-10-09 13:35:17] ...trap/engines/engine.py.230: loss: 0.14539 [I 2022-10-09 13:35:17] ...optimizers/trijoint.py.151: total_norm: 16.733181 [I 2022-10-09 13:35:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 1000/2304 [I 2022-10-09 13:35:26] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:17 [I 2022-10-09 13:35:26] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00305 [I 2022-10-09 13:35:26] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-09 13:35:26] ...optimizers/trijoint.py.151: total_norm: 13.723071 [I 2022-10-09 13:35:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 1010/2304 [I 2022-10-09 13:35:34] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:12 [I 2022-10-09 13:35:34] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00298 [I 2022-10-09 13:35:34] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-09 13:35:34] ...optimizers/trijoint.py.151: total_norm: 10.419161 [I 2022-10-09 13:35:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 1020/2304 [I 2022-10-09 13:35:43] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:05 [I 2022-10-09 13:35:43] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00293 [I 2022-10-09 13:35:43] ...trap/engines/engine.py.230: loss: 0.25440 [I 2022-10-09 13:35:43] ...optimizers/trijoint.py.151: total_norm: 18.093887 [I 2022-10-09 13:35:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 1030/2304 [I 2022-10-09 13:35:51] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:58 [I 2022-10-09 13:35:51] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00302 [I 2022-10-09 13:35:51] ...trap/engines/engine.py.230: loss: 0.14657 [I 2022-10-09 13:35:51] ...optimizers/trijoint.py.151: total_norm: 22.418270 [I 2022-10-09 13:36:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 1040/2304 [I 2022-10-09 13:36:00] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:47 [I 2022-10-09 13:36:00] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00299 [I 2022-10-09 13:36:00] ...trap/engines/engine.py.230: loss: 0.19703 [I 2022-10-09 13:36:00] ...optimizers/trijoint.py.151: total_norm: 25.808723 [I 2022-10-09 13:36:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1050/2304 [I 2022-10-09 13:36:08] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:42 [I 2022-10-09 13:36:08] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00303 [I 2022-10-09 13:36:08] ...trap/engines/engine.py.230: loss: 0.18277 [I 2022-10-09 13:36:08] ...optimizers/trijoint.py.151: total_norm: 16.653791 [I 2022-10-09 13:36:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 1060/2304 [I 2022-10-09 13:36:17] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:32 [I 2022-10-09 13:36:17] ...trap/engines/engine.py.229: process: 0.82928 | load: 0.00298 [I 2022-10-09 13:36:17] ...trap/engines/engine.py.230: loss: 0.14000 [I 2022-10-09 13:36:17] ...optimizers/trijoint.py.151: total_norm: 23.601837 [I 2022-10-09 13:36:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1070/2304 [I 2022-10-09 13:36:25] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:21 [I 2022-10-09 13:36:25] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00303 [I 2022-10-09 13:36:25] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-09 13:36:25] ...optimizers/trijoint.py.151: total_norm: 22.368910 [I 2022-10-09 13:36:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 1080/2304 [I 2022-10-09 13:36:34] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:14 [I 2022-10-09 13:36:34] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00302 [I 2022-10-09 13:36:34] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-09 13:36:34] ...optimizers/trijoint.py.151: total_norm: 10.169552 [I 2022-10-09 13:36:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 1090/2304 [I 2022-10-09 13:36:42] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:17 [I 2022-10-09 13:36:42] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00387 [I 2022-10-09 13:36:42] ...trap/engines/engine.py.230: loss: 0.18462 [I 2022-10-09 13:36:42] ...optimizers/trijoint.py.151: total_norm: 21.632755 [I 2022-10-09 13:36:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 1100/2304 [I 2022-10-09 13:36:51] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:17:04 [I 2022-10-09 13:36:51] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00311 [I 2022-10-09 13:36:51] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-09 13:36:51] ...optimizers/trijoint.py.151: total_norm: 10.399991 [I 2022-10-09 13:36:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 1110/2304 [I 2022-10-09 13:36:59] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:54 [I 2022-10-09 13:36:59] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00295 [I 2022-10-09 13:36:59] ...trap/engines/engine.py.230: loss: 0.14670 [I 2022-10-09 13:36:59] ...optimizers/trijoint.py.151: total_norm: 14.592196 [I 2022-10-09 13:37:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1120/2304 [I 2022-10-09 13:37:08] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:45 [I 2022-10-09 13:37:08] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00313 [I 2022-10-09 13:37:08] ...trap/engines/engine.py.230: loss: 0.15294 [I 2022-10-09 13:37:08] ...optimizers/trijoint.py.151: total_norm: 30.934940 [I 2022-10-09 13:37:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 1130/2304 [I 2022-10-09 13:37:16] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:39 [I 2022-10-09 13:37:16] ...trap/engines/engine.py.229: process: 0.85957 | load: 0.00299 [I 2022-10-09 13:37:16] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-09 13:37:16] ...optimizers/trijoint.py.151: total_norm: 12.970649 [I 2022-10-09 13:37:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1140/2304 [I 2022-10-09 13:37:25] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:33 [I 2022-10-09 13:37:25] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00296 [I 2022-10-09 13:37:25] ...trap/engines/engine.py.230: loss: 0.23286 [I 2022-10-09 13:37:25] ...optimizers/trijoint.py.151: total_norm: 11.840314 [I 2022-10-09 13:37:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 1150/2304 [I 2022-10-09 13:37:33] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:16 [I 2022-10-09 13:37:33] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00302 [I 2022-10-09 13:37:33] ...trap/engines/engine.py.230: loss: 0.22008 [I 2022-10-09 13:37:33] ...optimizers/trijoint.py.151: total_norm: 15.105113 [I 2022-10-09 13:37:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 1160/2304 [I 2022-10-09 13:37:42] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:12 [I 2022-10-09 13:37:42] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00297 [I 2022-10-09 13:37:42] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-09 13:37:42] ...optimizers/trijoint.py.151: total_norm: 14.351649 [I 2022-10-09 13:37:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 1170/2304 [I 2022-10-09 13:37:50] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:01 [I 2022-10-09 13:37:50] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00298 [I 2022-10-09 13:37:50] ...trap/engines/engine.py.230: loss: 0.14808 [I 2022-10-09 13:37:50] ...optimizers/trijoint.py.151: total_norm: 16.126532 [I 2022-10-09 13:37:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 1180/2304 [I 2022-10-09 13:37:59] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:52 [I 2022-10-09 13:37:59] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00302 [I 2022-10-09 13:37:59] ...trap/engines/engine.py.230: loss: 0.16513 [I 2022-10-09 13:37:59] ...optimizers/trijoint.py.151: total_norm: 17.036957 [I 2022-10-09 13:38:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 1190/2304 [I 2022-10-09 13:38:07] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:46 [I 2022-10-09 13:38:07] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00304 [I 2022-10-09 13:38:07] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-09 13:38:07] ...optimizers/trijoint.py.151: total_norm: 9.940069 [I 2022-10-09 13:38:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 1200/2304 [I 2022-10-09 13:38:16] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:34 [I 2022-10-09 13:38:16] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00300 [I 2022-10-09 13:38:16] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-09 13:38:16] ...optimizers/trijoint.py.151: total_norm: 27.300636 [I 2022-10-09 13:38:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 1210/2304 [I 2022-10-09 13:38:24] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:22 [I 2022-10-09 13:38:24] ...trap/engines/engine.py.229: process: 0.83356 | load: 0.00299 [I 2022-10-09 13:38:24] ...trap/engines/engine.py.230: loss: 0.13036 [I 2022-10-09 13:38:24] ...optimizers/trijoint.py.151: total_norm: 15.535122 [I 2022-10-09 13:38:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 1220/2304 [I 2022-10-09 13:38:33] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:09 [I 2022-10-09 13:38:33] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00297 [I 2022-10-09 13:38:33] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-09 13:38:33] ...optimizers/trijoint.py.151: total_norm: 16.292240 [I 2022-10-09 13:38:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 1230/2304 [I 2022-10-09 13:38:41] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:15:09 [I 2022-10-09 13:38:41] ...trap/engines/engine.py.229: process: 0.83377 | load: 0.00297 [I 2022-10-09 13:38:41] ...trap/engines/engine.py.230: loss: 0.17759 [I 2022-10-09 13:38:41] ...optimizers/trijoint.py.151: total_norm: 34.306747 [I 2022-10-09 13:38:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 1240/2304 [I 2022-10-09 13:38:50] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:06 [I 2022-10-09 13:38:50] ...trap/engines/engine.py.229: process: 0.85910 | load: 0.00305 [I 2022-10-09 13:38:50] ...trap/engines/engine.py.230: loss: 0.14841 [I 2022-10-09 13:38:50] ...optimizers/trijoint.py.151: total_norm: 18.473219 [I 2022-10-09 13:38:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 1250/2304 [I 2022-10-09 13:38:58] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:43 [I 2022-10-09 13:38:58] ...trap/engines/engine.py.229: process: 0.82899 | load: 0.00293 [I 2022-10-09 13:38:58] ...trap/engines/engine.py.230: loss: 0.15664 [I 2022-10-09 13:38:58] ...optimizers/trijoint.py.151: total_norm: 15.156030 [I 2022-10-09 13:39:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 1260/2304 [I 2022-10-09 13:39:07] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:42 [I 2022-10-09 13:39:07] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00299 [I 2022-10-09 13:39:07] ...trap/engines/engine.py.230: loss: 0.15615 [I 2022-10-09 13:39:07] ...optimizers/trijoint.py.151: total_norm: 8.255808 [I 2022-10-09 13:39:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 1270/2304 [I 2022-10-09 13:39:15] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:32 [I 2022-10-09 13:39:15] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00293 [I 2022-10-09 13:39:15] ...trap/engines/engine.py.230: loss: 0.13633 [I 2022-10-09 13:39:15] ...optimizers/trijoint.py.151: total_norm: 20.996050 [I 2022-10-09 13:39:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 1280/2304 [I 2022-10-09 13:39:24] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:29 [I 2022-10-09 13:39:24] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00292 [I 2022-10-09 13:39:24] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-09 13:39:24] ...optimizers/trijoint.py.151: total_norm: 17.783691 [I 2022-10-09 13:39:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 1290/2304 [I 2022-10-09 13:39:32] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:20 [I 2022-10-09 13:39:32] ...trap/engines/engine.py.229: process: 0.85578 | load: 0.00304 [I 2022-10-09 13:39:32] ...trap/engines/engine.py.230: loss: 0.17475 [I 2022-10-09 13:39:32] ...optimizers/trijoint.py.151: total_norm: 20.055948 [I 2022-10-09 13:39:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 1300/2304 [I 2022-10-09 13:39:40] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:09 [I 2022-10-09 13:39:40] ...trap/engines/engine.py.229: process: 0.85616 | load: 0.00295 [I 2022-10-09 13:39:40] ...trap/engines/engine.py.230: loss: 0.18474 [I 2022-10-09 13:39:40] ...optimizers/trijoint.py.151: total_norm: 13.813292 [I 2022-10-09 13:39:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 1310/2304 [I 2022-10-09 13:39:49] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:05 [I 2022-10-09 13:39:49] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00294 [I 2022-10-09 13:39:49] ...trap/engines/engine.py.230: loss: 0.17691 [I 2022-10-09 13:39:49] ...optimizers/trijoint.py.151: total_norm: 11.760229 [I 2022-10-09 13:39:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 1320/2304 [I 2022-10-09 13:39:58] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:13:57 [I 2022-10-09 13:39:58] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00293 [I 2022-10-09 13:39:58] ...trap/engines/engine.py.230: loss: 0.17021 [I 2022-10-09 13:39:58] ...optimizers/trijoint.py.151: total_norm: 13.834929 [I 2022-10-09 13:40:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 1330/2304 [I 2022-10-09 13:40:06] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:49 [I 2022-10-09 13:40:06] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00294 [I 2022-10-09 13:40:06] ...trap/engines/engine.py.230: loss: 0.21973 [I 2022-10-09 13:40:06] ...optimizers/trijoint.py.151: total_norm: 25.952541 [I 2022-10-09 13:40:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 1340/2304 [I 2022-10-09 13:40:15] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:34 [I 2022-10-09 13:40:15] ...trap/engines/engine.py.229: process: 0.83858 | load: 0.00299 [I 2022-10-09 13:40:15] ...trap/engines/engine.py.230: loss: 0.20596 [I 2022-10-09 13:40:15] ...optimizers/trijoint.py.151: total_norm: 23.694750 [I 2022-10-09 13:40:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 1350/2304 [I 2022-10-09 13:40:23] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:31 [I 2022-10-09 13:40:23] ...trap/engines/engine.py.229: process: 0.85844 | load: 0.00294 [I 2022-10-09 13:40:23] ...trap/engines/engine.py.230: loss: 0.17657 [I 2022-10-09 13:40:23] ...optimizers/trijoint.py.151: total_norm: 10.883445 [I 2022-10-09 13:40:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 1360/2304 [I 2022-10-09 13:40:32] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:18 [I 2022-10-09 13:40:32] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00296 [I 2022-10-09 13:40:32] ...trap/engines/engine.py.230: loss: 0.14322 [I 2022-10-09 13:40:32] ...optimizers/trijoint.py.151: total_norm: 23.205032 [I 2022-10-09 13:40:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 1370/2304 [I 2022-10-09 13:40:40] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:14 [I 2022-10-09 13:40:40] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00293 [I 2022-10-09 13:40:40] ...trap/engines/engine.py.230: loss: 0.10819 [I 2022-10-09 13:40:40] ...optimizers/trijoint.py.151: total_norm: 7.826347 [I 2022-10-09 13:40:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 1380/2304 [I 2022-10-09 13:40:49] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:12:59 [I 2022-10-09 13:40:49] ...trap/engines/engine.py.229: process: 0.85506 | load: 0.00294 [I 2022-10-09 13:40:49] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-09 13:40:49] ...optimizers/trijoint.py.151: total_norm: 15.440629 [I 2022-10-09 13:40:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 1390/2304 [I 2022-10-09 13:40:57] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:49 [I 2022-10-09 13:40:57] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00296 [I 2022-10-09 13:40:57] ...trap/engines/engine.py.230: loss: 0.20096 [I 2022-10-09 13:40:57] ...optimizers/trijoint.py.151: total_norm: 11.784554 [I 2022-10-09 13:41:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 1400/2304 [I 2022-10-09 13:41:05] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:39 [I 2022-10-09 13:41:05] ...trap/engines/engine.py.229: process: 0.83955 | load: 0.00303 [I 2022-10-09 13:41:05] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-09 13:41:05] ...optimizers/trijoint.py.151: total_norm: 17.629454 [I 2022-10-09 13:41:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 1410/2304 [I 2022-10-09 13:41:14] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:35 [I 2022-10-09 13:41:14] ...trap/engines/engine.py.229: process: 0.82734 | load: 0.00308 [I 2022-10-09 13:41:14] ...trap/engines/engine.py.230: loss: 0.24675 [I 2022-10-09 13:41:14] ...optimizers/trijoint.py.151: total_norm: 19.469282 [I 2022-10-09 13:41:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 1420/2304 [I 2022-10-09 13:41:22] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:25 [I 2022-10-09 13:41:22] ...trap/engines/engine.py.229: process: 0.83237 | load: 0.00302 [I 2022-10-09 13:41:22] ...trap/engines/engine.py.230: loss: 0.17987 [I 2022-10-09 13:41:22] ...optimizers/trijoint.py.151: total_norm: 8.466493 [I 2022-10-09 13:41:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 1430/2304 [I 2022-10-09 13:41:31] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:20 [I 2022-10-09 13:41:31] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00298 [I 2022-10-09 13:41:31] ...trap/engines/engine.py.230: loss: 0.16153 [I 2022-10-09 13:41:31] ...optimizers/trijoint.py.151: total_norm: 15.297905 [I 2022-10-09 13:41:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 1440/2304 [I 2022-10-09 13:41:39] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:09 [I 2022-10-09 13:41:39] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00336 [I 2022-10-09 13:41:39] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-09 13:41:39] ...optimizers/trijoint.py.151: total_norm: 12.174783 [I 2022-10-09 13:41:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 1450/2304 [I 2022-10-09 13:41:48] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:00 [I 2022-10-09 13:41:48] ...trap/engines/engine.py.229: process: 0.82076 | load: 0.00297 [I 2022-10-09 13:41:48] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-09 13:41:48] ...optimizers/trijoint.py.151: total_norm: 11.777177 [I 2022-10-09 13:41:56] ...trap/engines/engine.py.224: train: epoch 42 | batch 1460/2304 [I 2022-10-09 13:41:56] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:50 [I 2022-10-09 13:41:56] ...trap/engines/engine.py.229: process: 0.82535 | load: 0.00300 [I 2022-10-09 13:41:56] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-09 13:41:56] ...optimizers/trijoint.py.151: total_norm: 19.989483 [I 2022-10-09 13:42:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 1470/2304 [I 2022-10-09 13:42:05] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:46 [I 2022-10-09 13:42:05] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00304 [I 2022-10-09 13:42:05] ...trap/engines/engine.py.230: loss: 0.16461 [I 2022-10-09 13:42:05] ...optimizers/trijoint.py.151: total_norm: 22.806349 [I 2022-10-09 13:42:13] ...trap/engines/engine.py.224: train: epoch 42 | batch 1480/2304 [I 2022-10-09 13:42:13] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:40 [I 2022-10-09 13:42:13] ...trap/engines/engine.py.229: process: 0.84887 | load: 0.00296 [I 2022-10-09 13:42:13] ...trap/engines/engine.py.230: loss: 0.13982 [I 2022-10-09 13:42:13] ...optimizers/trijoint.py.151: total_norm: 16.933084 [I 2022-10-09 13:42:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 1490/2304 [I 2022-10-09 13:42:22] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:32 [I 2022-10-09 13:42:22] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00299 [I 2022-10-09 13:42:22] ...trap/engines/engine.py.230: loss: 0.21088 [I 2022-10-09 13:42:22] ...optimizers/trijoint.py.151: total_norm: 29.335489 [I 2022-10-09 13:42:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 1500/2304 [I 2022-10-09 13:42:30] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:19 [I 2022-10-09 13:42:30] ...trap/engines/engine.py.229: process: 0.86037 | load: 0.00297 [I 2022-10-09 13:42:30] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-09 13:42:30] ...optimizers/trijoint.py.151: total_norm: 49.262707 [I 2022-10-09 13:42:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 1510/2304 [I 2022-10-09 13:42:39] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:08 [I 2022-10-09 13:42:39] ...trap/engines/engine.py.229: process: 0.82912 | load: 0.00300 [I 2022-10-09 13:42:39] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-09 13:42:39] ...optimizers/trijoint.py.151: total_norm: 25.527784 [I 2022-10-09 13:42:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 1520/2304 [I 2022-10-09 13:42:47] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:01 [I 2022-10-09 13:42:47] ...trap/engines/engine.py.229: process: 0.84013 | load: 0.00298 [I 2022-10-09 13:42:47] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-09 13:42:47] ...optimizers/trijoint.py.151: total_norm: 17.617218 [I 2022-10-09 13:42:56] ...trap/engines/engine.py.224: train: epoch 42 | batch 1530/2304 [I 2022-10-09 13:42:56] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:54 [I 2022-10-09 13:42:56] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00297 [I 2022-10-09 13:42:56] ...trap/engines/engine.py.230: loss: 0.17539 [I 2022-10-09 13:42:56] ...optimizers/trijoint.py.151: total_norm: 15.533335 [I 2022-10-09 13:43:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 1540/2304 [I 2022-10-09 13:43:04] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:42 [I 2022-10-09 13:43:04] ...trap/engines/engine.py.229: process: 0.83312 | load: 0.00302 [I 2022-10-09 13:43:04] ...trap/engines/engine.py.230: loss: 0.15342 [I 2022-10-09 13:43:04] ...optimizers/trijoint.py.151: total_norm: 12.553336 [I 2022-10-09 13:43:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 1550/2304 [I 2022-10-09 13:43:12] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:37 [I 2022-10-09 13:43:12] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00304 [I 2022-10-09 13:43:12] ...trap/engines/engine.py.230: loss: 0.29652 [I 2022-10-09 13:43:12] ...optimizers/trijoint.py.151: total_norm: 29.089073 [I 2022-10-09 13:43:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 1560/2304 [I 2022-10-09 13:43:21] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:24 [I 2022-10-09 13:43:21] ...trap/engines/engine.py.229: process: 0.82618 | load: 0.00301 [I 2022-10-09 13:43:21] ...trap/engines/engine.py.230: loss: 0.22659 [I 2022-10-09 13:43:21] ...optimizers/trijoint.py.151: total_norm: 21.085493 [I 2022-10-09 13:43:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 1570/2304 [I 2022-10-09 13:43:29] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:20 [I 2022-10-09 13:43:29] ...trap/engines/engine.py.229: process: 0.85206 | load: 0.00296 [I 2022-10-09 13:43:29] ...trap/engines/engine.py.230: loss: 0.20880 [I 2022-10-09 13:43:29] ...optimizers/trijoint.py.151: total_norm: 13.018029 [I 2022-10-09 13:43:38] ...trap/engines/engine.py.224: train: epoch 42 | batch 1580/2304 [I 2022-10-09 13:43:38] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:14 [I 2022-10-09 13:43:38] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00296 [I 2022-10-09 13:43:38] ...trap/engines/engine.py.230: loss: 0.11697 [I 2022-10-09 13:43:38] ...optimizers/trijoint.py.151: total_norm: 9.998739 [I 2022-10-09 13:43:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 1590/2304 [I 2022-10-09 13:43:46] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:04 [I 2022-10-09 13:43:46] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00300 [I 2022-10-09 13:43:46] ...trap/engines/engine.py.230: loss: 0.16809 [I 2022-10-09 13:43:46] ...optimizers/trijoint.py.151: total_norm: 16.269249 [I 2022-10-09 13:43:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 1600/2304 [I 2022-10-09 13:43:55] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:57 [I 2022-10-09 13:43:55] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00300 [I 2022-10-09 13:43:55] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-09 13:43:55] ...optimizers/trijoint.py.151: total_norm: 30.299400 [I 2022-10-09 13:44:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 1610/2304 [I 2022-10-09 13:44:03] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:48 [I 2022-10-09 13:44:03] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00294 [I 2022-10-09 13:44:03] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-09 13:44:03] ...optimizers/trijoint.py.151: total_norm: 11.756331 [I 2022-10-09 13:44:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 1620/2304 [I 2022-10-09 13:44:12] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:41 [I 2022-10-09 13:44:12] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00310 [I 2022-10-09 13:44:12] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-09 13:44:12] ...optimizers/trijoint.py.151: total_norm: 10.068318 [I 2022-10-09 13:44:20] ...trap/engines/engine.py.224: train: epoch 42 | batch 1630/2304 [I 2022-10-09 13:44:20] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:30 [I 2022-10-09 13:44:20] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00297 [I 2022-10-09 13:44:20] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-09 13:44:20] ...optimizers/trijoint.py.151: total_norm: 22.154894 [I 2022-10-09 13:44:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 1640/2304 [I 2022-10-09 13:44:29] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:20 [I 2022-10-09 13:44:29] ...trap/engines/engine.py.229: process: 0.83810 | load: 0.00296 [I 2022-10-09 13:44:29] ...trap/engines/engine.py.230: loss: 0.16887 [I 2022-10-09 13:44:29] ...optimizers/trijoint.py.151: total_norm: 9.865841 [I 2022-10-09 13:44:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 1650/2304 [I 2022-10-09 13:44:37] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:10 [I 2022-10-09 13:44:37] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00299 [I 2022-10-09 13:44:37] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-09 13:44:37] ...optimizers/trijoint.py.151: total_norm: 22.843893 [I 2022-10-09 13:44:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 1660/2304 [I 2022-10-09 13:44:46] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:04 [I 2022-10-09 13:44:46] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00297 [I 2022-10-09 13:44:46] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-09 13:44:46] ...optimizers/trijoint.py.151: total_norm: 17.238914 [I 2022-10-09 13:44:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 1670/2304 [I 2022-10-09 13:44:54] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:56 [I 2022-10-09 13:44:54] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00295 [I 2022-10-09 13:44:54] ...trap/engines/engine.py.230: loss: 0.12709 [I 2022-10-09 13:44:54] ...optimizers/trijoint.py.151: total_norm: 17.081182 [I 2022-10-09 13:45:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 1680/2304 [I 2022-10-09 13:45:02] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:46 [I 2022-10-09 13:45:02] ...trap/engines/engine.py.229: process: 0.83687 | load: 0.00294 [I 2022-10-09 13:45:02] ...trap/engines/engine.py.230: loss: 0.15338 [I 2022-10-09 13:45:02] ...optimizers/trijoint.py.151: total_norm: 27.089003 [I 2022-10-09 13:45:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 1690/2304 [I 2022-10-09 13:45:11] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:38 [I 2022-10-09 13:45:11] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00298 [I 2022-10-09 13:45:11] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-09 13:45:11] ...optimizers/trijoint.py.151: total_norm: 13.548719 [I 2022-10-09 13:45:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 1700/2304 [I 2022-10-09 13:45:19] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:28 [I 2022-10-09 13:45:19] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00295 [I 2022-10-09 13:45:19] ...trap/engines/engine.py.230: loss: 0.11735 [I 2022-10-09 13:45:19] ...optimizers/trijoint.py.151: total_norm: 12.042997 [I 2022-10-09 13:45:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 1710/2304 [I 2022-10-09 13:45:28] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:23 [I 2022-10-09 13:45:28] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00294 [I 2022-10-09 13:45:28] ...trap/engines/engine.py.230: loss: 0.24191 [I 2022-10-09 13:45:28] ...optimizers/trijoint.py.151: total_norm: 30.553627 [I 2022-10-09 13:45:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 1720/2304 [I 2022-10-09 13:45:36] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:15 [I 2022-10-09 13:45:36] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00307 [I 2022-10-09 13:45:36] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-09 13:45:36] ...optimizers/trijoint.py.151: total_norm: 16.022867 [I 2022-10-09 13:45:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 1730/2304 [I 2022-10-09 13:45:45] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:05 [I 2022-10-09 13:45:45] ...trap/engines/engine.py.229: process: 0.83734 | load: 0.00299 [I 2022-10-09 13:45:45] ...trap/engines/engine.py.230: loss: 0.14269 [I 2022-10-09 13:45:45] ...optimizers/trijoint.py.151: total_norm: 15.098239 [I 2022-10-09 13:45:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 1740/2304 [I 2022-10-09 13:45:53] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:57 [I 2022-10-09 13:45:53] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00298 [I 2022-10-09 13:45:53] ...trap/engines/engine.py.230: loss: 0.15601 [I 2022-10-09 13:45:53] ...optimizers/trijoint.py.151: total_norm: 19.806482 [I 2022-10-09 13:46:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 1750/2304 [I 2022-10-09 13:46:02] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:48 [I 2022-10-09 13:46:02] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00306 [I 2022-10-09 13:46:02] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-09 13:46:02] ...optimizers/trijoint.py.151: total_norm: 17.391537 [I 2022-10-09 13:46:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 1760/2304 [I 2022-10-09 13:46:10] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:36 [I 2022-10-09 13:46:10] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00292 [I 2022-10-09 13:46:10] ...trap/engines/engine.py.230: loss: 0.12469 [I 2022-10-09 13:46:10] ...optimizers/trijoint.py.151: total_norm: 13.993315 [I 2022-10-09 13:46:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 1770/2304 [I 2022-10-09 13:46:19] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:30 [I 2022-10-09 13:46:19] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00295 [I 2022-10-09 13:46:19] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-09 13:46:19] ...optimizers/trijoint.py.151: total_norm: 14.901134 [I 2022-10-09 13:46:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 1780/2304 [I 2022-10-09 13:46:27] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:20 [I 2022-10-09 13:46:27] ...trap/engines/engine.py.229: process: 0.82925 | load: 0.00294 [I 2022-10-09 13:46:27] ...trap/engines/engine.py.230: loss: 0.20177 [I 2022-10-09 13:46:27] ...optimizers/trijoint.py.151: total_norm: 26.957315 [I 2022-10-09 13:46:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 1790/2304 [I 2022-10-09 13:46:35] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:13 [I 2022-10-09 13:46:35] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00297 [I 2022-10-09 13:46:35] ...trap/engines/engine.py.230: loss: 0.16094 [I 2022-10-09 13:46:35] ...optimizers/trijoint.py.151: total_norm: 12.709073 [I 2022-10-09 13:46:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 1800/2304 [I 2022-10-09 13:46:44] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:04 [I 2022-10-09 13:46:44] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00301 [I 2022-10-09 13:46:44] ...trap/engines/engine.py.230: loss: 0.21595 [I 2022-10-09 13:46:44] ...optimizers/trijoint.py.151: total_norm: 19.021967 [I 2022-10-09 13:46:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 1810/2304 [I 2022-10-09 13:46:52] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:57 [I 2022-10-09 13:46:52] ...trap/engines/engine.py.229: process: 0.85422 | load: 0.00299 [I 2022-10-09 13:46:52] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-09 13:46:52] ...optimizers/trijoint.py.151: total_norm: 10.779724 [I 2022-10-09 13:47:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 1820/2304 [I 2022-10-09 13:47:01] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:49 [I 2022-10-09 13:47:01] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00294 [I 2022-10-09 13:47:01] ...trap/engines/engine.py.230: loss: 0.14477 [I 2022-10-09 13:47:01] ...optimizers/trijoint.py.151: total_norm: 17.095711 [I 2022-10-09 13:47:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 1830/2304 [I 2022-10-09 13:47:09] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:41 [I 2022-10-09 13:47:09] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00294 [I 2022-10-09 13:47:09] ...trap/engines/engine.py.230: loss: 0.17561 [I 2022-10-09 13:47:09] ...optimizers/trijoint.py.151: total_norm: 15.288911 [I 2022-10-09 13:47:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 1840/2304 [I 2022-10-09 13:47:18] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:30 [I 2022-10-09 13:47:18] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00293 [I 2022-10-09 13:47:18] ...trap/engines/engine.py.230: loss: 0.16010 [I 2022-10-09 13:47:18] ...optimizers/trijoint.py.151: total_norm: 36.081581 [I 2022-10-09 13:47:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 1850/2304 [I 2022-10-09 13:47:26] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:23 [I 2022-10-09 13:47:26] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00298 [I 2022-10-09 13:47:26] ...trap/engines/engine.py.230: loss: 0.17502 [I 2022-10-09 13:47:26] ...optimizers/trijoint.py.151: total_norm: 17.989351 [I 2022-10-09 13:47:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 1860/2304 [I 2022-10-09 13:47:35] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:12 [I 2022-10-09 13:47:35] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00297 [I 2022-10-09 13:47:35] ...trap/engines/engine.py.230: loss: 0.12819 [I 2022-10-09 13:47:35] ...optimizers/trijoint.py.151: total_norm: 18.453070 [I 2022-10-09 13:47:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 1870/2304 [I 2022-10-09 13:47:43] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:06 [I 2022-10-09 13:47:43] ...trap/engines/engine.py.229: process: 0.84018 | load: 0.00294 [I 2022-10-09 13:47:43] ...trap/engines/engine.py.230: loss: 0.19986 [I 2022-10-09 13:47:43] ...optimizers/trijoint.py.151: total_norm: 20.176186 [I 2022-10-09 13:47:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 1880/2304 [I 2022-10-09 13:47:52] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:59 [I 2022-10-09 13:47:52] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00295 [I 2022-10-09 13:47:52] ...trap/engines/engine.py.230: loss: 0.18506 [I 2022-10-09 13:47:52] ...optimizers/trijoint.py.151: total_norm: 20.049988 [I 2022-10-09 13:48:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 1890/2304 [I 2022-10-09 13:48:00] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:49 [I 2022-10-09 13:48:00] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00294 [I 2022-10-09 13:48:00] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-09 13:48:00] ...optimizers/trijoint.py.151: total_norm: 14.013568 [I 2022-10-09 13:48:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1900/2304 [I 2022-10-09 13:48:08] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:38 [I 2022-10-09 13:48:08] ...trap/engines/engine.py.229: process: 0.83034 | load: 0.00305 [I 2022-10-09 13:48:08] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-09 13:48:08] ...optimizers/trijoint.py.151: total_norm: 29.687717 [I 2022-10-09 13:48:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 1910/2304 [I 2022-10-09 13:48:17] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:34 [I 2022-10-09 13:48:17] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00293 [I 2022-10-09 13:48:17] ...trap/engines/engine.py.230: loss: 0.21345 [I 2022-10-09 13:48:17] ...optimizers/trijoint.py.151: total_norm: 20.399069 [I 2022-10-09 13:48:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1920/2304 [I 2022-10-09 13:48:25] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:22 [I 2022-10-09 13:48:25] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00293 [I 2022-10-09 13:48:25] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-09 13:48:25] ...optimizers/trijoint.py.151: total_norm: 20.300676 [I 2022-10-09 13:48:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 1930/2304 [I 2022-10-09 13:48:34] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:14 [I 2022-10-09 13:48:34] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00299 [I 2022-10-09 13:48:34] ...trap/engines/engine.py.230: loss: 0.14820 [I 2022-10-09 13:48:34] ...optimizers/trijoint.py.151: total_norm: 25.086126 [I 2022-10-09 13:48:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 1940/2304 [I 2022-10-09 13:48:42] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:07 [I 2022-10-09 13:48:42] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00297 [I 2022-10-09 13:48:42] ...trap/engines/engine.py.230: loss: 0.20871 [I 2022-10-09 13:48:42] ...optimizers/trijoint.py.151: total_norm: 13.644025 [I 2022-10-09 13:48:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 1950/2304 [I 2022-10-09 13:48:51] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:58 [I 2022-10-09 13:48:51] ...trap/engines/engine.py.229: process: 0.83397 | load: 0.00294 [I 2022-10-09 13:48:51] ...trap/engines/engine.py.230: loss: 0.14479 [I 2022-10-09 13:48:51] ...optimizers/trijoint.py.151: total_norm: 13.291524 [I 2022-10-09 13:48:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 1960/2304 [I 2022-10-09 13:48:59] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:51 [I 2022-10-09 13:48:59] ...trap/engines/engine.py.229: process: 0.86146 | load: 0.00296 [I 2022-10-09 13:48:59] ...trap/engines/engine.py.230: loss: 0.18375 [I 2022-10-09 13:48:59] ...optimizers/trijoint.py.151: total_norm: 34.479168 [I 2022-10-09 13:49:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1970/2304 [I 2022-10-09 13:49:08] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:41 [I 2022-10-09 13:49:08] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00297 [I 2022-10-09 13:49:08] ...trap/engines/engine.py.230: loss: 0.26873 [I 2022-10-09 13:49:08] ...optimizers/trijoint.py.151: total_norm: 24.561415 [I 2022-10-09 13:49:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 1980/2304 [I 2022-10-09 13:49:16] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:33 [I 2022-10-09 13:49:16] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00310 [I 2022-10-09 13:49:16] ...trap/engines/engine.py.230: loss: 0.09789 [I 2022-10-09 13:49:16] ...optimizers/trijoint.py.151: total_norm: 20.138983 [I 2022-10-09 13:49:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1990/2304 [I 2022-10-09 13:49:25] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:25 [I 2022-10-09 13:49:25] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00295 [I 2022-10-09 13:49:25] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-09 13:49:25] ...optimizers/trijoint.py.151: total_norm: 14.606061 [I 2022-10-09 13:49:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 2000/2304 [I 2022-10-09 13:49:33] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:16 [I 2022-10-09 13:49:33] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00511 [I 2022-10-09 13:49:33] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-09 13:49:33] ...optimizers/trijoint.py.151: total_norm: 9.471970 [I 2022-10-09 13:49:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 2010/2304 [I 2022-10-09 13:49:41] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:07 [I 2022-10-09 13:49:41] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00295 [I 2022-10-09 13:49:41] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-09 13:49:41] ...optimizers/trijoint.py.151: total_norm: 8.075798 [I 2022-10-09 13:49:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 2020/2304 [I 2022-10-09 13:49:50] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:03:59 [I 2022-10-09 13:49:50] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00294 [I 2022-10-09 13:49:50] ...trap/engines/engine.py.230: loss: 0.16157 [I 2022-10-09 13:49:50] ...optimizers/trijoint.py.151: total_norm: 11.447682 [I 2022-10-09 13:49:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 2030/2304 [I 2022-10-09 13:49:58] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:51 [I 2022-10-09 13:49:58] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00302 [I 2022-10-09 13:49:58] ...trap/engines/engine.py.230: loss: 0.14417 [I 2022-10-09 13:49:58] ...optimizers/trijoint.py.151: total_norm: 12.181078 [I 2022-10-09 13:50:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 2040/2304 [I 2022-10-09 13:50:07] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:42 [I 2022-10-09 13:50:07] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00295 [I 2022-10-09 13:50:07] ...trap/engines/engine.py.230: loss: 0.16346 [I 2022-10-09 13:50:07] ...optimizers/trijoint.py.151: total_norm: 28.405809 [I 2022-10-09 13:50:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 2050/2304 [I 2022-10-09 13:50:15] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:35 [I 2022-10-09 13:50:15] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00294 [I 2022-10-09 13:50:15] ...trap/engines/engine.py.230: loss: 0.13924 [I 2022-10-09 13:50:15] ...optimizers/trijoint.py.151: total_norm: 11.647413 [I 2022-10-09 13:50:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 2060/2304 [I 2022-10-09 13:50:24] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:25 [I 2022-10-09 13:50:24] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00300 [I 2022-10-09 13:50:24] ...trap/engines/engine.py.230: loss: 0.18279 [I 2022-10-09 13:50:24] ...optimizers/trijoint.py.151: total_norm: 15.047451 [I 2022-10-09 13:50:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 2070/2304 [I 2022-10-09 13:50:32] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:17 [I 2022-10-09 13:50:32] ...trap/engines/engine.py.229: process: 0.83149 | load: 0.00294 [I 2022-10-09 13:50:32] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-09 13:50:32] ...optimizers/trijoint.py.151: total_norm: 18.133221 [I 2022-10-09 13:50:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 2080/2304 [I 2022-10-09 13:50:41] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:09 [I 2022-10-09 13:50:41] ...trap/engines/engine.py.229: process: 0.85206 | load: 0.00300 [I 2022-10-09 13:50:41] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-09 13:50:41] ...optimizers/trijoint.py.151: total_norm: 11.352787 [I 2022-10-09 13:50:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 2090/2304 [I 2022-10-09 13:50:49] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:01 [I 2022-10-09 13:50:49] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00296 [I 2022-10-09 13:50:49] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-09 13:50:49] ...optimizers/trijoint.py.151: total_norm: 15.395829 [I 2022-10-09 13:50:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 2100/2304 [I 2022-10-09 13:50:58] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:54 [I 2022-10-09 13:50:58] ...trap/engines/engine.py.229: process: 0.86215 | load: 0.00297 [I 2022-10-09 13:50:58] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-09 13:50:58] ...optimizers/trijoint.py.151: total_norm: 23.833084 [I 2022-10-09 13:51:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 2110/2304 [I 2022-10-09 13:51:06] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:45 [I 2022-10-09 13:51:06] ...trap/engines/engine.py.229: process: 0.84625 | load: 0.00299 [I 2022-10-09 13:51:06] ...trap/engines/engine.py.230: loss: 0.14029 [I 2022-10-09 13:51:06] ...optimizers/trijoint.py.151: total_norm: 19.816236 [I 2022-10-09 13:51:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 2120/2304 [I 2022-10-09 13:51:15] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:37 [I 2022-10-09 13:51:15] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00312 [I 2022-10-09 13:51:15] ...trap/engines/engine.py.230: loss: 0.21202 [I 2022-10-09 13:51:15] ...optimizers/trijoint.py.151: total_norm: 18.842707 [I 2022-10-09 13:51:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 2130/2304 [I 2022-10-09 13:51:23] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:27 [I 2022-10-09 13:51:23] ...trap/engines/engine.py.229: process: 0.84953 | load: 0.00299 [I 2022-10-09 13:51:23] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-09 13:51:23] ...optimizers/trijoint.py.151: total_norm: 11.642503 [I 2022-10-09 13:51:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 2140/2304 [I 2022-10-09 13:51:32] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:19 [I 2022-10-09 13:51:32] ...trap/engines/engine.py.229: process: 0.85876 | load: 0.00300 [I 2022-10-09 13:51:32] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-09 13:51:32] ...optimizers/trijoint.py.151: total_norm: 14.177062 [I 2022-10-09 13:51:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 2150/2304 [I 2022-10-09 13:51:40] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:11 [I 2022-10-09 13:51:40] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00298 [I 2022-10-09 13:51:40] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-09 13:51:40] ...optimizers/trijoint.py.151: total_norm: 24.347883 [I 2022-10-09 13:51:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 2160/2304 [I 2022-10-09 13:51:49] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:01 [I 2022-10-09 13:51:49] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00298 [I 2022-10-09 13:51:49] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-09 13:51:49] ...optimizers/trijoint.py.151: total_norm: 22.404413 [I 2022-10-09 13:51:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 2170/2304 [I 2022-10-09 13:51:57] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:01:53 [I 2022-10-09 13:51:57] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00301 [I 2022-10-09 13:51:57] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-09 13:51:57] ...optimizers/trijoint.py.151: total_norm: 12.005187 [I 2022-10-09 13:52:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 2180/2304 [I 2022-10-09 13:52:06] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:45 [I 2022-10-09 13:52:06] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00296 [I 2022-10-09 13:52:06] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-09 13:52:06] ...optimizers/trijoint.py.151: total_norm: 13.397753 [I 2022-10-09 13:52:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 2190/2304 [I 2022-10-09 13:52:14] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:36 [I 2022-10-09 13:52:14] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00298 [I 2022-10-09 13:52:14] ...trap/engines/engine.py.230: loss: 0.24675 [I 2022-10-09 13:52:14] ...optimizers/trijoint.py.151: total_norm: 26.553267 [I 2022-10-09 13:52:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 2200/2304 [I 2022-10-09 13:52:23] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:28 [I 2022-10-09 13:52:23] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00296 [I 2022-10-09 13:52:23] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-09 13:52:23] ...optimizers/trijoint.py.151: total_norm: 19.812023 [I 2022-10-09 13:52:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 2210/2304 [I 2022-10-09 13:52:31] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:19 [I 2022-10-09 13:52:31] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00309 [I 2022-10-09 13:52:31] ...trap/engines/engine.py.230: loss: 0.22941 [I 2022-10-09 13:52:31] ...optimizers/trijoint.py.151: total_norm: 23.505478 [I 2022-10-09 13:52:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 2220/2304 [I 2022-10-09 13:52:40] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:10 [I 2022-10-09 13:52:40] ...trap/engines/engine.py.229: process: 0.84649 | load: 0.00290 [I 2022-10-09 13:52:40] ...trap/engines/engine.py.230: loss: 0.13990 [I 2022-10-09 13:52:40] ...optimizers/trijoint.py.151: total_norm: 31.977781 [I 2022-10-09 13:52:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 2230/2304 [I 2022-10-09 13:52:48] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:02 [I 2022-10-09 13:52:48] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00298 [I 2022-10-09 13:52:48] ...trap/engines/engine.py.230: loss: 0.14666 [I 2022-10-09 13:52:48] ...optimizers/trijoint.py.151: total_norm: 10.161481 [I 2022-10-09 13:52:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 2240/2304 [I 2022-10-09 13:52:57] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:53 [I 2022-10-09 13:52:57] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00297 [I 2022-10-09 13:52:57] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-09 13:52:57] ...optimizers/trijoint.py.151: total_norm: 18.079161 [I 2022-10-09 13:53:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 2250/2304 [I 2022-10-09 13:53:05] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:45 [I 2022-10-09 13:53:05] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00309 [I 2022-10-09 13:53:05] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-09 13:53:05] ...optimizers/trijoint.py.151: total_norm: 16.745167 [I 2022-10-09 13:53:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 2260/2304 [I 2022-10-09 13:53:14] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:37 [I 2022-10-09 13:53:14] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00294 [I 2022-10-09 13:53:14] ...trap/engines/engine.py.230: loss: 0.12785 [I 2022-10-09 13:53:14] ...optimizers/trijoint.py.151: total_norm: 13.824516 [I 2022-10-09 13:53:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 2270/2304 [I 2022-10-09 13:53:22] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:28 [I 2022-10-09 13:53:22] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00297 [I 2022-10-09 13:53:22] ...trap/engines/engine.py.230: loss: 0.21350 [I 2022-10-09 13:53:22] ...optimizers/trijoint.py.151: total_norm: 25.710814 [I 2022-10-09 13:53:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 2280/2304 [I 2022-10-09 13:53:31] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:20 [I 2022-10-09 13:53:31] ...trap/engines/engine.py.229: process: 0.84488 | load: 0.00301 [I 2022-10-09 13:53:31] ...trap/engines/engine.py.230: loss: 0.16918 [I 2022-10-09 13:53:31] ...optimizers/trijoint.py.151: total_norm: 18.641582 [I 2022-10-09 13:53:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 2290/2304 [I 2022-10-09 13:53:39] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:11 [I 2022-10-09 13:53:39] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00294 [I 2022-10-09 13:53:39] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-09 13:53:39] ...optimizers/trijoint.py.151: total_norm: 17.168119 [I 2022-10-09 13:53:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 2300/2304 [I 2022-10-09 13:53:47] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:03 [I 2022-10-09 13:53:47] ...trap/engines/engine.py.229: process: 0.83224 | load: 0.00015 [I 2022-10-09 13:53:47] ...trap/engines/engine.py.230: loss: 0.13532 [I 2022-10-09 13:53:47] ...optimizers/trijoint.py.151: total_norm: 13.957819 [I 2022-10-09 13:53:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 2304/2304 [I 2022-10-09 13:53:51] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:00 [I 2022-10-09 13:53:51] ...trap/engines/engine.py.229: process: 0.82739 | load: 0.00016 [I 2022-10-09 13:53:51] ...trap/engines/engine.py.230: loss: 0.17868 [I 2022-10-09 13:53:51] ...optimizers/trijoint.py.151: total_norm: 21.742599 [S 2022-10-09 13:53:52] ...trap/engines/engine.py.129: train_epoch.epoch: 42 [S 2022-10-09 13:53:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.15865773717002082 [I 2022-10-09 13:53:52] ...s/criterions/triplet.py.84: margin = 0.26000000000000006 [I 2022-10-09 13:53:53] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 42 [I 2022-10-09 13:53:53] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 13:53:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 13:53:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 13:53:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 13:53:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 13:53:56] ...trap/engines/engine.py.321: eval: epoch 42 | batch 0/510 [I 2022-10-09 13:53:56] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:00 [I 2022-10-09 13:53:56] ...trap/engines/engine.py.326: process: 3.41325 | load: 2.84919 [I 2022-10-09 13:54:02] ...trap/engines/engine.py.321: eval: epoch 42 | batch 10/510 [I 2022-10-09 13:54:02] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:47 [I 2022-10-09 13:54:02] ...trap/engines/engine.py.326: process: 0.68988 | load: 0.00017 [I 2022-10-09 13:54:07] ...trap/engines/engine.py.321: eval: epoch 42 | batch 20/510 [I 2022-10-09 13:54:07] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:04:29 [I 2022-10-09 13:54:07] ...trap/engines/engine.py.326: process: 0.65032 | load: 0.00018 [I 2022-10-09 13:54:12] ...trap/engines/engine.py.321: eval: epoch 42 | batch 30/510 [I 2022-10-09 13:54:12] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:03:54 [I 2022-10-09 13:54:12] ...trap/engines/engine.py.326: process: 0.42383 | load: 0.00018 [I 2022-10-09 13:54:17] ...trap/engines/engine.py.321: eval: epoch 42 | batch 40/510 [I 2022-10-09 13:54:17] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:44 [I 2022-10-09 13:54:17] ...trap/engines/engine.py.326: process: 0.43681 | load: 0.00019 [I 2022-10-09 13:54:21] ...trap/engines/engine.py.321: eval: epoch 42 | batch 50/510 [I 2022-10-09 13:54:21] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:16 [I 2022-10-09 13:54:21] ...trap/engines/engine.py.326: process: 0.42229 | load: 0.00017 [I 2022-10-09 13:54:26] ...trap/engines/engine.py.321: eval: epoch 42 | batch 60/510 [I 2022-10-09 13:54:26] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:44 [I 2022-10-09 13:54:26] ...trap/engines/engine.py.326: process: 0.52835 | load: 0.00018 [I 2022-10-09 13:54:31] ...trap/engines/engine.py.321: eval: epoch 42 | batch 70/510 [I 2022-10-09 13:54:31] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:35 [I 2022-10-09 13:54:31] ...trap/engines/engine.py.326: process: 0.44734 | load: 0.00018 [I 2022-10-09 13:54:35] ...trap/engines/engine.py.321: eval: epoch 42 | batch 80/510 [I 2022-10-09 13:54:35] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:03 [I 2022-10-09 13:54:35] ...trap/engines/engine.py.326: process: 0.42168 | load: 0.00019 [I 2022-10-09 13:54:40] ...trap/engines/engine.py.321: eval: epoch 42 | batch 90/510 [I 2022-10-09 13:54:40] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:02:55 [I 2022-10-09 13:54:40] ...trap/engines/engine.py.326: process: 0.42342 | load: 0.00019 [I 2022-10-09 13:54:44] ...trap/engines/engine.py.321: eval: epoch 42 | batch 100/510 [I 2022-10-09 13:54:44] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:02:43 [I 2022-10-09 13:54:44] ...trap/engines/engine.py.326: process: 0.30811 | load: 0.00018 [I 2022-10-09 13:54:47] ...trap/engines/engine.py.321: eval: epoch 42 | batch 110/510 [I 2022-10-09 13:54:47] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:09 [I 2022-10-09 13:54:47] ...trap/engines/engine.py.326: process: 0.30926 | load: 0.00018 [I 2022-10-09 13:54:50] ...trap/engines/engine.py.321: eval: epoch 42 | batch 120/510 [I 2022-10-09 13:54:50] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:04 [I 2022-10-09 13:54:50] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00018 [I 2022-10-09 13:54:53] ...trap/engines/engine.py.321: eval: epoch 42 | batch 130/510 [I 2022-10-09 13:54:53] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:00 [I 2022-10-09 13:54:53] ...trap/engines/engine.py.326: process: 0.31690 | load: 0.00019 [I 2022-10-09 13:54:56] ...trap/engines/engine.py.321: eval: epoch 42 | batch 140/510 [I 2022-10-09 13:54:56] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:57 [I 2022-10-09 13:54:56] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00020 [I 2022-10-09 13:55:00] ...trap/engines/engine.py.321: eval: epoch 42 | batch 150/510 [I 2022-10-09 13:55:00] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:54 [I 2022-10-09 13:55:00] ...trap/engines/engine.py.326: process: 0.31972 | load: 0.00019 [I 2022-10-09 13:55:03] ...trap/engines/engine.py.321: eval: epoch 42 | batch 160/510 [I 2022-10-09 13:55:03] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:51 [I 2022-10-09 13:55:03] ...trap/engines/engine.py.326: process: 0.32115 | load: 0.00019 [I 2022-10-09 13:55:06] ...trap/engines/engine.py.321: eval: epoch 42 | batch 170/510 [I 2022-10-09 13:55:06] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:48 [I 2022-10-09 13:55:06] ...trap/engines/engine.py.326: process: 0.32514 | load: 0.00019 [I 2022-10-09 13:55:09] ...trap/engines/engine.py.321: eval: epoch 42 | batch 180/510 [I 2022-10-09 13:55:09] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:45 [I 2022-10-09 13:55:09] ...trap/engines/engine.py.326: process: 0.31721 | load: 0.00020 [I 2022-10-09 13:55:12] ...trap/engines/engine.py.321: eval: epoch 42 | batch 190/510 [I 2022-10-09 13:55:12] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:42 [I 2022-10-09 13:55:12] ...trap/engines/engine.py.326: process: 0.31916 | load: 0.00020 [I 2022-10-09 13:55:15] ...trap/engines/engine.py.321: eval: epoch 42 | batch 200/510 [I 2022-10-09 13:55:15] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:38 [I 2022-10-09 13:55:15] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00020 [I 2022-10-09 13:55:19] ...trap/engines/engine.py.321: eval: epoch 42 | batch 210/510 [I 2022-10-09 13:55:19] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:35 [I 2022-10-09 13:55:19] ...trap/engines/engine.py.326: process: 0.32416 | load: 0.00019 [I 2022-10-09 13:55:22] ...trap/engines/engine.py.321: eval: epoch 42 | batch 220/510 [I 2022-10-09 13:55:22] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:32 [I 2022-10-09 13:55:22] ...trap/engines/engine.py.326: process: 0.32032 | load: 0.00018 [I 2022-10-09 13:55:25] ...trap/engines/engine.py.321: eval: epoch 42 | batch 230/510 [I 2022-10-09 13:55:25] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:29 [I 2022-10-09 13:55:25] ...trap/engines/engine.py.326: process: 0.32472 | load: 0.00020 [I 2022-10-09 13:55:28] ...trap/engines/engine.py.321: eval: epoch 42 | batch 240/510 [I 2022-10-09 13:55:28] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:26 [I 2022-10-09 13:55:28] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00020 [I 2022-10-09 13:55:31] ...trap/engines/engine.py.321: eval: epoch 42 | batch 250/510 [I 2022-10-09 13:55:31] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:22 [I 2022-10-09 13:55:31] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00019 [I 2022-10-09 13:55:35] ...trap/engines/engine.py.321: eval: epoch 42 | batch 260/510 [I 2022-10-09 13:55:35] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:18 [I 2022-10-09 13:55:35] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00019 [I 2022-10-09 13:55:38] ...trap/engines/engine.py.321: eval: epoch 42 | batch 270/510 [I 2022-10-09 13:55:38] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:16 [I 2022-10-09 13:55:38] ...trap/engines/engine.py.326: process: 0.31802 | load: 0.00021 [I 2022-10-09 13:55:41] ...trap/engines/engine.py.321: eval: epoch 42 | batch 280/510 [I 2022-10-09 13:55:41] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:12 [I 2022-10-09 13:55:41] ...trap/engines/engine.py.326: process: 0.31540 | load: 0.00020 [I 2022-10-09 13:55:44] ...trap/engines/engine.py.321: eval: epoch 42 | batch 290/510 [I 2022-10-09 13:55:44] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:09 [I 2022-10-09 13:55:44] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00019 [I 2022-10-09 13:55:47] ...trap/engines/engine.py.321: eval: epoch 42 | batch 300/510 [I 2022-10-09 13:55:47] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:07 [I 2022-10-09 13:55:47] ...trap/engines/engine.py.326: process: 0.32004 | load: 0.00019 [I 2022-10-09 13:55:50] ...trap/engines/engine.py.321: eval: epoch 42 | batch 310/510 [I 2022-10-09 13:55:50] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:03 [I 2022-10-09 13:55:50] ...trap/engines/engine.py.326: process: 0.31546 | load: 0.00021 [I 2022-10-09 13:55:54] ...trap/engines/engine.py.321: eval: epoch 42 | batch 320/510 [I 2022-10-09 13:55:54] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-09 13:55:54] ...trap/engines/engine.py.326: process: 0.32097 | load: 0.00018 [I 2022-10-09 13:55:57] ...trap/engines/engine.py.321: eval: epoch 42 | batch 330/510 [I 2022-10-09 13:55:57] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:57 [I 2022-10-09 13:55:57] ...trap/engines/engine.py.326: process: 0.31445 | load: 0.00021 [I 2022-10-09 13:56:00] ...trap/engines/engine.py.321: eval: epoch 42 | batch 340/510 [I 2022-10-09 13:56:00] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:54 [I 2022-10-09 13:56:00] ...trap/engines/engine.py.326: process: 0.31724 | load: 0.00021 [I 2022-10-09 13:56:03] ...trap/engines/engine.py.321: eval: epoch 42 | batch 350/510 [I 2022-10-09 13:56:03] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:50 [I 2022-10-09 13:56:03] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00019 [I 2022-10-09 13:56:06] ...trap/engines/engine.py.321: eval: epoch 42 | batch 360/510 [I 2022-10-09 13:56:06] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:47 [I 2022-10-09 13:56:06] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00020 [I 2022-10-09 13:56:10] ...trap/engines/engine.py.321: eval: epoch 42 | batch 370/510 [I 2022-10-09 13:56:10] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:44 [I 2022-10-09 13:56:10] ...trap/engines/engine.py.326: process: 0.32103 | load: 0.00020 [I 2022-10-09 13:56:13] ...trap/engines/engine.py.321: eval: epoch 42 | batch 380/510 [I 2022-10-09 13:56:13] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:41 [I 2022-10-09 13:56:13] ...trap/engines/engine.py.326: process: 0.31431 | load: 0.00020 [I 2022-10-09 13:56:16] ...trap/engines/engine.py.321: eval: epoch 42 | batch 390/510 [I 2022-10-09 13:56:16] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:38 [I 2022-10-09 13:56:16] ...trap/engines/engine.py.326: process: 0.31160 | load: 0.00020 [I 2022-10-09 13:56:19] ...trap/engines/engine.py.321: eval: epoch 42 | batch 400/510 [I 2022-10-09 13:56:19] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:35 [I 2022-10-09 13:56:19] ...trap/engines/engine.py.326: process: 0.31167 | load: 0.00019 [I 2022-10-09 13:56:22] ...trap/engines/engine.py.321: eval: epoch 42 | batch 410/510 [I 2022-10-09 13:56:22] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:32 [I 2022-10-09 13:56:22] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00025 [I 2022-10-09 13:56:26] ...trap/engines/engine.py.321: eval: epoch 42 | batch 420/510 [I 2022-10-09 13:56:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:28 [I 2022-10-09 13:56:26] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00020 [I 2022-10-09 13:56:29] ...trap/engines/engine.py.321: eval: epoch 42 | batch 430/510 [I 2022-10-09 13:56:29] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:25 [I 2022-10-09 13:56:29] ...trap/engines/engine.py.326: process: 0.31787 | load: 0.00022 [I 2022-10-09 13:56:32] ...trap/engines/engine.py.321: eval: epoch 42 | batch 440/510 [I 2022-10-09 13:56:32] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:22 [I 2022-10-09 13:56:32] ...trap/engines/engine.py.326: process: 0.31986 | load: 0.00019 [I 2022-10-09 13:56:35] ...trap/engines/engine.py.321: eval: epoch 42 | batch 450/510 [I 2022-10-09 13:56:35] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:19 [I 2022-10-09 13:56:35] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00025 [I 2022-10-09 13:56:38] ...trap/engines/engine.py.321: eval: epoch 42 | batch 460/510 [I 2022-10-09 13:56:38] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:15 [I 2022-10-09 13:56:38] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00020 [I 2022-10-09 13:56:41] ...trap/engines/engine.py.321: eval: epoch 42 | batch 470/510 [I 2022-10-09 13:56:41] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:12 [I 2022-10-09 13:56:41] ...trap/engines/engine.py.326: process: 0.32287 | load: 0.00020 [I 2022-10-09 13:56:45] ...trap/engines/engine.py.321: eval: epoch 42 | batch 480/510 [I 2022-10-09 13:56:45] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-09 13:56:45] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00020 [I 2022-10-09 13:56:48] ...trap/engines/engine.py.321: eval: epoch 42 | batch 490/510 [I 2022-10-09 13:56:48] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:06 [I 2022-10-09 13:56:48] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00024 [I 2022-10-09 13:56:51] ...trap/engines/engine.py.321: eval: epoch 42 | batch 500/510 [I 2022-10-09 13:56:51] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:03 [I 2022-10-09 13:56:51] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00020 [I 2022-10-09 13:56:54] ...trap/engines/engine.py.321: eval: epoch 42 | batch 510/510 [I 2022-10-09 13:56:54] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:00 [I 2022-10-09 13:56:54] ...trap/engines/engine.py.326: process: 0.31741 | load: 0.00013 [S 2022-10-09 13:56:55] ...trap/engines/engine.py.132: eval_epoch.epoch: 42 [I 2022-10-09 13:56:55] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.718000 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903700 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934600 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.707200 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900700 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933300 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016012 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008100 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009178 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012734 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010364 [S 2022-10-09 13:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008235 [I 2022-10-09 13:57:33] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 13:57:33] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 13:57:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 13:57:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 13:57:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 13:57:37] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 13:57:38] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 13:57:43] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 13:57:43] ...trap/engines/engine.py.163: Training model on trainset for epoch 43 [I 2022-10-09 13:57:43] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 13:57:46] ...trap/engines/engine.py.224: train: epoch 43 | batch 0/2304 [I 2022-10-09 13:57:46] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:44 [I 2022-10-09 13:57:46] ...trap/engines/engine.py.229: process: 2.72761 | load: 1.81564 [I 2022-10-09 13:57:46] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-09 13:57:46] ...optimizers/trijoint.py.151: total_norm: 13.102454 [I 2022-10-09 13:57:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 10/2304 [I 2022-10-09 13:57:55] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:05 [I 2022-10-09 13:57:55] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00683 [I 2022-10-09 13:57:55] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-09 13:57:55] ...optimizers/trijoint.py.151: total_norm: 13.166874 [I 2022-10-09 13:58:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 20/2304 [I 2022-10-09 13:58:03] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:06 [I 2022-10-09 13:58:03] ...trap/engines/engine.py.229: process: 0.85762 | load: 0.00443 [I 2022-10-09 13:58:03] ...trap/engines/engine.py.230: loss: 0.22494 [I 2022-10-09 13:58:03] ...optimizers/trijoint.py.151: total_norm: 18.744818 [I 2022-10-09 13:58:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 30/2304 [I 2022-10-09 13:58:12] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:09 [I 2022-10-09 13:58:12] ...trap/engines/engine.py.229: process: 0.84999 | load: 0.00441 [I 2022-10-09 13:58:12] ...trap/engines/engine.py.230: loss: 0.18706 [I 2022-10-09 13:58:12] ...optimizers/trijoint.py.151: total_norm: 19.161724 [I 2022-10-09 13:58:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 40/2304 [I 2022-10-09 13:58:20] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:50 [I 2022-10-09 13:58:20] ...trap/engines/engine.py.229: process: 0.83962 | load: 0.00446 [I 2022-10-09 13:58:20] ...trap/engines/engine.py.230: loss: 0.18129 [I 2022-10-09 13:58:20] ...optimizers/trijoint.py.151: total_norm: 17.590462 [I 2022-10-09 13:58:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 50/2304 [I 2022-10-09 13:58:28] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:35 [I 2022-10-09 13:58:28] ...trap/engines/engine.py.229: process: 0.84710 | load: 0.00369 [I 2022-10-09 13:58:28] ...trap/engines/engine.py.230: loss: 0.11913 [I 2022-10-09 13:58:28] ...optimizers/trijoint.py.151: total_norm: 16.925571 [I 2022-10-09 13:58:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 60/2304 [I 2022-10-09 13:58:37] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:29 [I 2022-10-09 13:58:37] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00401 [I 2022-10-09 13:58:37] ...trap/engines/engine.py.230: loss: 0.17286 [I 2022-10-09 13:58:37] ...optimizers/trijoint.py.151: total_norm: 15.013153 [I 2022-10-09 13:58:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 70/2304 [I 2022-10-09 13:58:45] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:34 [I 2022-10-09 13:58:45] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00335 [I 2022-10-09 13:58:45] ...trap/engines/engine.py.230: loss: 0.13606 [I 2022-10-09 13:58:45] ...optimizers/trijoint.py.151: total_norm: 24.323271 [I 2022-10-09 13:58:54] ...trap/engines/engine.py.224: train: epoch 43 | batch 80/2304 [I 2022-10-09 13:58:54] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:35 [I 2022-10-09 13:58:54] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00323 [I 2022-10-09 13:58:54] ...trap/engines/engine.py.230: loss: 0.15876 [I 2022-10-09 13:58:54] ...optimizers/trijoint.py.151: total_norm: 13.733413 [I 2022-10-09 13:59:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 90/2304 [I 2022-10-09 13:59:02] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:09 [I 2022-10-09 13:59:02] ...trap/engines/engine.py.229: process: 0.83995 | load: 0.00320 [I 2022-10-09 13:59:02] ...trap/engines/engine.py.230: loss: 0.13868 [I 2022-10-09 13:59:02] ...optimizers/trijoint.py.151: total_norm: 19.339495 [I 2022-10-09 13:59:11] ...trap/engines/engine.py.224: train: epoch 43 | batch 100/2304 [I 2022-10-09 13:59:11] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:55 [I 2022-10-09 13:59:11] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00310 [I 2022-10-09 13:59:11] ...trap/engines/engine.py.230: loss: 0.14280 [I 2022-10-09 13:59:11] ...optimizers/trijoint.py.151: total_norm: 12.731626 [I 2022-10-09 13:59:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 110/2304 [I 2022-10-09 13:59:19] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:45 [I 2022-10-09 13:59:19] ...trap/engines/engine.py.229: process: 0.83208 | load: 0.00335 [I 2022-10-09 13:59:19] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-09 13:59:19] ...optimizers/trijoint.py.151: total_norm: 16.670031 [I 2022-10-09 13:59:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 120/2304 [I 2022-10-09 13:59:28] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:46 [I 2022-10-09 13:59:28] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00318 [I 2022-10-09 13:59:28] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-09 13:59:28] ...optimizers/trijoint.py.151: total_norm: 16.436071 [I 2022-10-09 13:59:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 130/2304 [I 2022-10-09 13:59:36] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:36 [I 2022-10-09 13:59:36] ...trap/engines/engine.py.229: process: 0.83528 | load: 0.00295 [I 2022-10-09 13:59:36] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-09 13:59:36] ...optimizers/trijoint.py.151: total_norm: 23.195421 [I 2022-10-09 13:59:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 140/2304 [I 2022-10-09 13:59:45] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:31 [I 2022-10-09 13:59:45] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00317 [I 2022-10-09 13:59:45] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-09 13:59:45] ...optimizers/trijoint.py.151: total_norm: 11.475093 [I 2022-10-09 13:59:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 150/2304 [I 2022-10-09 13:59:53] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:22 [I 2022-10-09 13:59:53] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00302 [I 2022-10-09 13:59:53] ...trap/engines/engine.py.230: loss: 0.18371 [I 2022-10-09 13:59:53] ...optimizers/trijoint.py.151: total_norm: 22.889797 [I 2022-10-09 14:00:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 160/2304 [I 2022-10-09 14:00:01] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:12 [I 2022-10-09 14:00:01] ...trap/engines/engine.py.229: process: 0.83360 | load: 0.00339 [I 2022-10-09 14:00:01] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-09 14:00:01] ...optimizers/trijoint.py.151: total_norm: 11.239018 [I 2022-10-09 14:00:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 170/2304 [I 2022-10-09 14:00:10] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:52 [I 2022-10-09 14:00:10] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00318 [I 2022-10-09 14:00:10] ...trap/engines/engine.py.230: loss: 0.16979 [I 2022-10-09 14:00:10] ...optimizers/trijoint.py.151: total_norm: 19.578363 [I 2022-10-09 14:00:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 180/2304 [I 2022-10-09 14:00:18] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:55 [I 2022-10-09 14:00:18] ...trap/engines/engine.py.229: process: 0.85615 | load: 0.00295 [I 2022-10-09 14:00:18] ...trap/engines/engine.py.230: loss: 0.10793 [I 2022-10-09 14:00:18] ...optimizers/trijoint.py.151: total_norm: 21.850454 [I 2022-10-09 14:00:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 190/2304 [I 2022-10-09 14:00:27] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:40 [I 2022-10-09 14:00:27] ...trap/engines/engine.py.229: process: 0.83862 | load: 0.00339 [I 2022-10-09 14:00:27] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-09 14:00:27] ...optimizers/trijoint.py.151: total_norm: 13.822678 [I 2022-10-09 14:00:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 200/2304 [I 2022-10-09 14:00:35] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:44 [I 2022-10-09 14:00:35] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00311 [I 2022-10-09 14:00:35] ...trap/engines/engine.py.230: loss: 0.20510 [I 2022-10-09 14:00:35] ...optimizers/trijoint.py.151: total_norm: 17.812183 [I 2022-10-09 14:00:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 210/2304 [I 2022-10-09 14:00:44] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:24 [I 2022-10-09 14:00:44] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00317 [I 2022-10-09 14:00:44] ...trap/engines/engine.py.230: loss: 0.12830 [I 2022-10-09 14:00:44] ...optimizers/trijoint.py.151: total_norm: 12.895107 [I 2022-10-09 14:00:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 220/2304 [I 2022-10-09 14:00:52] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:09 [I 2022-10-09 14:00:52] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00292 [I 2022-10-09 14:00:52] ...trap/engines/engine.py.230: loss: 0.14883 [I 2022-10-09 14:00:52] ...optimizers/trijoint.py.151: total_norm: 13.470831 [I 2022-10-09 14:01:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 230/2304 [I 2022-10-09 14:01:01] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:04 [I 2022-10-09 14:01:01] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00292 [I 2022-10-09 14:01:01] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-09 14:01:01] ...optimizers/trijoint.py.151: total_norm: 18.291407 [I 2022-10-09 14:01:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 240/2304 [I 2022-10-09 14:01:09] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:55 [I 2022-10-09 14:01:09] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00310 [I 2022-10-09 14:01:09] ...trap/engines/engine.py.230: loss: 0.10785 [I 2022-10-09 14:01:09] ...optimizers/trijoint.py.151: total_norm: 16.043613 [I 2022-10-09 14:01:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 250/2304 [I 2022-10-09 14:01:18] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:49 [I 2022-10-09 14:01:18] ...trap/engines/engine.py.229: process: 0.84467 | load: 0.00316 [I 2022-10-09 14:01:18] ...trap/engines/engine.py.230: loss: 0.19335 [I 2022-10-09 14:01:18] ...optimizers/trijoint.py.151: total_norm: 11.501575 [I 2022-10-09 14:01:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 260/2304 [I 2022-10-09 14:01:26] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:45 [I 2022-10-09 14:01:26] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00291 [I 2022-10-09 14:01:26] ...trap/engines/engine.py.230: loss: 0.22686 [I 2022-10-09 14:01:26] ...optimizers/trijoint.py.151: total_norm: 13.319118 [I 2022-10-09 14:01:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 270/2304 [I 2022-10-09 14:01:34] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:32 [I 2022-10-09 14:01:34] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00292 [I 2022-10-09 14:01:34] ...trap/engines/engine.py.230: loss: 0.13042 [I 2022-10-09 14:01:34] ...optimizers/trijoint.py.151: total_norm: 20.192204 [I 2022-10-09 14:01:43] ...trap/engines/engine.py.224: train: epoch 43 | batch 280/2304 [I 2022-10-09 14:01:43] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:28 [I 2022-10-09 14:01:43] ...trap/engines/engine.py.229: process: 0.84660 | load: 0.00293 [I 2022-10-09 14:01:43] ...trap/engines/engine.py.230: loss: 0.22191 [I 2022-10-09 14:01:43] ...optimizers/trijoint.py.151: total_norm: 13.563102 [I 2022-10-09 14:01:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 290/2304 [I 2022-10-09 14:01:51] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:23 [I 2022-10-09 14:01:51] ...trap/engines/engine.py.229: process: 0.84197 | load: 0.00300 [I 2022-10-09 14:01:51] ...trap/engines/engine.py.230: loss: 0.14107 [I 2022-10-09 14:01:51] ...optimizers/trijoint.py.151: total_norm: 10.222596 [I 2022-10-09 14:02:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 300/2304 [I 2022-10-09 14:02:00] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:13 [I 2022-10-09 14:02:00] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00292 [I 2022-10-09 14:02:00] ...trap/engines/engine.py.230: loss: 0.13181 [I 2022-10-09 14:02:00] ...optimizers/trijoint.py.151: total_norm: 13.588471 [I 2022-10-09 14:02:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 310/2304 [I 2022-10-09 14:02:08] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:27:54 [I 2022-10-09 14:02:08] ...trap/engines/engine.py.229: process: 0.83642 | load: 0.00291 [I 2022-10-09 14:02:08] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-09 14:02:08] ...optimizers/trijoint.py.151: total_norm: 12.358994 [I 2022-10-09 14:02:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 320/2304 [I 2022-10-09 14:02:17] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:48 [I 2022-10-09 14:02:17] ...trap/engines/engine.py.229: process: 0.83126 | load: 0.00293 [I 2022-10-09 14:02:17] ...trap/engines/engine.py.230: loss: 0.15807 [I 2022-10-09 14:02:17] ...optimizers/trijoint.py.151: total_norm: 10.257985 [I 2022-10-09 14:02:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 330/2304 [I 2022-10-09 14:02:25] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:44 [I 2022-10-09 14:02:25] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00310 [I 2022-10-09 14:02:25] ...trap/engines/engine.py.230: loss: 0.15647 [I 2022-10-09 14:02:25] ...optimizers/trijoint.py.151: total_norm: 9.031963 [I 2022-10-09 14:02:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 340/2304 [I 2022-10-09 14:02:34] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:41 [I 2022-10-09 14:02:34] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00298 [I 2022-10-09 14:02:34] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-09 14:02:34] ...optimizers/trijoint.py.151: total_norm: 24.905817 [I 2022-10-09 14:02:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 350/2304 [I 2022-10-09 14:02:42] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:34 [I 2022-10-09 14:02:42] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00291 [I 2022-10-09 14:02:42] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-09 14:02:42] ...optimizers/trijoint.py.151: total_norm: 12.773953 [I 2022-10-09 14:02:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 360/2304 [I 2022-10-09 14:02:51] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:17 [I 2022-10-09 14:02:51] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00293 [I 2022-10-09 14:02:51] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-09 14:02:51] ...optimizers/trijoint.py.151: total_norm: 26.988495 [I 2022-10-09 14:02:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 370/2304 [I 2022-10-09 14:02:59] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:00 [I 2022-10-09 14:02:59] ...trap/engines/engine.py.229: process: 0.82899 | load: 0.00295 [I 2022-10-09 14:02:59] ...trap/engines/engine.py.230: loss: 0.14587 [I 2022-10-09 14:02:59] ...optimizers/trijoint.py.151: total_norm: 10.360231 [I 2022-10-09 14:03:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 380/2304 [I 2022-10-09 14:03:07] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:06 [I 2022-10-09 14:03:07] ...trap/engines/engine.py.229: process: 0.84846 | load: 0.00301 [I 2022-10-09 14:03:07] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-09 14:03:07] ...optimizers/trijoint.py.151: total_norm: 53.554230 [I 2022-10-09 14:03:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 390/2304 [I 2022-10-09 14:03:16] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:04 [I 2022-10-09 14:03:16] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00306 [I 2022-10-09 14:03:16] ...trap/engines/engine.py.230: loss: 0.15013 [I 2022-10-09 14:03:16] ...optimizers/trijoint.py.151: total_norm: 19.537144 [I 2022-10-09 14:03:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 400/2304 [I 2022-10-09 14:03:24] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:44 [I 2022-10-09 14:03:24] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00294 [I 2022-10-09 14:03:24] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-09 14:03:24] ...optimizers/trijoint.py.151: total_norm: 8.233404 [I 2022-10-09 14:03:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 410/2304 [I 2022-10-09 14:03:33] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:40 [I 2022-10-09 14:03:33] ...trap/engines/engine.py.229: process: 0.84275 | load: 0.00310 [I 2022-10-09 14:03:33] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-09 14:03:33] ...optimizers/trijoint.py.151: total_norm: 13.050498 [I 2022-10-09 14:03:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 420/2304 [I 2022-10-09 14:03:41] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:30 [I 2022-10-09 14:03:41] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00295 [I 2022-10-09 14:03:41] ...trap/engines/engine.py.230: loss: 0.19538 [I 2022-10-09 14:03:41] ...optimizers/trijoint.py.151: total_norm: 24.106758 [I 2022-10-09 14:03:50] ...trap/engines/engine.py.224: train: epoch 43 | batch 430/2304 [I 2022-10-09 14:03:50] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:22 [I 2022-10-09 14:03:50] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00298 [I 2022-10-09 14:03:50] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-09 14:03:50] ...optimizers/trijoint.py.151: total_norm: 30.785654 [I 2022-10-09 14:03:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 440/2304 [I 2022-10-09 14:03:58] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:13 [I 2022-10-09 14:03:58] ...trap/engines/engine.py.229: process: 0.83957 | load: 0.00306 [I 2022-10-09 14:03:58] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-09 14:03:58] ...optimizers/trijoint.py.151: total_norm: 8.327087 [I 2022-10-09 14:04:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 450/2304 [I 2022-10-09 14:04:07] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:08 [I 2022-10-09 14:04:07] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00297 [I 2022-10-09 14:04:07] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-09 14:04:07] ...optimizers/trijoint.py.151: total_norm: 10.237158 [I 2022-10-09 14:04:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 460/2304 [I 2022-10-09 14:04:15] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:00 [I 2022-10-09 14:04:15] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00295 [I 2022-10-09 14:04:15] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-09 14:04:15] ...optimizers/trijoint.py.151: total_norm: 43.368774 [I 2022-10-09 14:04:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 470/2304 [I 2022-10-09 14:04:24] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:59 [I 2022-10-09 14:04:24] ...trap/engines/engine.py.229: process: 0.85849 | load: 0.00295 [I 2022-10-09 14:04:24] ...trap/engines/engine.py.230: loss: 0.17394 [I 2022-10-09 14:04:24] ...optimizers/trijoint.py.151: total_norm: 11.300967 [I 2022-10-09 14:04:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 480/2304 [I 2022-10-09 14:04:32] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:38 [I 2022-10-09 14:04:32] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00297 [I 2022-10-09 14:04:32] ...trap/engines/engine.py.230: loss: 0.19818 [I 2022-10-09 14:04:32] ...optimizers/trijoint.py.151: total_norm: 12.756401 [I 2022-10-09 14:04:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 490/2304 [I 2022-10-09 14:04:41] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:37 [I 2022-10-09 14:04:41] ...trap/engines/engine.py.229: process: 0.86120 | load: 0.00305 [I 2022-10-09 14:04:41] ...trap/engines/engine.py.230: loss: 0.18291 [I 2022-10-09 14:04:41] ...optimizers/trijoint.py.151: total_norm: 23.121191 [I 2022-10-09 14:04:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 500/2304 [I 2022-10-09 14:04:49] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:24 [I 2022-10-09 14:04:49] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00294 [I 2022-10-09 14:04:49] ...trap/engines/engine.py.230: loss: 0.16890 [I 2022-10-09 14:04:49] ...optimizers/trijoint.py.151: total_norm: 12.161157 [I 2022-10-09 14:04:57] ...trap/engines/engine.py.224: train: epoch 43 | batch 510/2304 [I 2022-10-09 14:04:57] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:09 [I 2022-10-09 14:04:57] ...trap/engines/engine.py.229: process: 0.83353 | load: 0.00293 [I 2022-10-09 14:04:57] ...trap/engines/engine.py.230: loss: 0.18047 [I 2022-10-09 14:04:57] ...optimizers/trijoint.py.151: total_norm: 16.709532 [I 2022-10-09 14:05:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 520/2304 [I 2022-10-09 14:05:06] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:07 [I 2022-10-09 14:05:06] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00304 [I 2022-10-09 14:05:06] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-09 14:05:06] ...optimizers/trijoint.py.151: total_norm: 12.727113 [I 2022-10-09 14:05:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 530/2304 [I 2022-10-09 14:05:14] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:24:53 [I 2022-10-09 14:05:14] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00296 [I 2022-10-09 14:05:14] ...trap/engines/engine.py.230: loss: 0.15560 [I 2022-10-09 14:05:14] ...optimizers/trijoint.py.151: total_norm: 19.513006 [I 2022-10-09 14:05:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 540/2304 [I 2022-10-09 14:05:23] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:44 [I 2022-10-09 14:05:23] ...trap/engines/engine.py.229: process: 0.84356 | load: 0.00294 [I 2022-10-09 14:05:23] ...trap/engines/engine.py.230: loss: 0.14544 [I 2022-10-09 14:05:23] ...optimizers/trijoint.py.151: total_norm: 19.143206 [I 2022-10-09 14:05:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 550/2304 [I 2022-10-09 14:05:31] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:37 [I 2022-10-09 14:05:31] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00300 [I 2022-10-09 14:05:31] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-09 14:05:31] ...optimizers/trijoint.py.151: total_norm: 9.627373 [I 2022-10-09 14:05:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 560/2304 [I 2022-10-09 14:05:40] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:33 [I 2022-10-09 14:05:40] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00293 [I 2022-10-09 14:05:40] ...trap/engines/engine.py.230: loss: 0.17486 [I 2022-10-09 14:05:40] ...optimizers/trijoint.py.151: total_norm: 19.398535 [I 2022-10-09 14:05:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 570/2304 [I 2022-10-09 14:05:48] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:28 [I 2022-10-09 14:05:48] ...trap/engines/engine.py.229: process: 0.85842 | load: 0.00294 [I 2022-10-09 14:05:48] ...trap/engines/engine.py.230: loss: 0.19775 [I 2022-10-09 14:05:48] ...optimizers/trijoint.py.151: total_norm: 54.564232 [I 2022-10-09 14:05:57] ...trap/engines/engine.py.224: train: epoch 43 | batch 580/2304 [I 2022-10-09 14:05:57] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:15 [I 2022-10-09 14:05:57] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00296 [I 2022-10-09 14:05:57] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-09 14:05:57] ...optimizers/trijoint.py.151: total_norm: 36.477581 [I 2022-10-09 14:06:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 590/2304 [I 2022-10-09 14:06:05] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:06 [I 2022-10-09 14:06:05] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00297 [I 2022-10-09 14:06:05] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-09 14:06:05] ...optimizers/trijoint.py.151: total_norm: 20.732258 [I 2022-10-09 14:06:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 600/2304 [I 2022-10-09 14:06:13] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:23:57 [I 2022-10-09 14:06:13] ...trap/engines/engine.py.229: process: 0.84067 | load: 0.00298 [I 2022-10-09 14:06:13] ...trap/engines/engine.py.230: loss: 0.22474 [I 2022-10-09 14:06:13] ...optimizers/trijoint.py.151: total_norm: 29.515747 [I 2022-10-09 14:06:22] ...trap/engines/engine.py.224: train: epoch 43 | batch 610/2304 [I 2022-10-09 14:06:22] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:23:55 [I 2022-10-09 14:06:22] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00304 [I 2022-10-09 14:06:22] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-09 14:06:22] ...optimizers/trijoint.py.151: total_norm: 19.618734 [I 2022-10-09 14:06:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 620/2304 [I 2022-10-09 14:06:30] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:41 [I 2022-10-09 14:06:30] ...trap/engines/engine.py.229: process: 0.83666 | load: 0.00297 [I 2022-10-09 14:06:30] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-09 14:06:30] ...optimizers/trijoint.py.151: total_norm: 8.821955 [I 2022-10-09 14:06:39] ...trap/engines/engine.py.224: train: epoch 43 | batch 630/2304 [I 2022-10-09 14:06:39] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:25 [I 2022-10-09 14:06:39] ...trap/engines/engine.py.229: process: 0.83620 | load: 0.00297 [I 2022-10-09 14:06:39] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-09 14:06:39] ...optimizers/trijoint.py.151: total_norm: 8.834506 [I 2022-10-09 14:06:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 640/2304 [I 2022-10-09 14:06:47] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:29 [I 2022-10-09 14:06:47] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00292 [I 2022-10-09 14:06:47] ...trap/engines/engine.py.230: loss: 0.10792 [I 2022-10-09 14:06:47] ...optimizers/trijoint.py.151: total_norm: 9.390580 [I 2022-10-09 14:06:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 650/2304 [I 2022-10-09 14:06:56] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:20 [I 2022-10-09 14:06:56] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00303 [I 2022-10-09 14:06:56] ...trap/engines/engine.py.230: loss: 0.12274 [I 2022-10-09 14:06:56] ...optimizers/trijoint.py.151: total_norm: 15.345118 [I 2022-10-09 14:07:04] ...trap/engines/engine.py.224: train: epoch 43 | batch 660/2304 [I 2022-10-09 14:07:04] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:11 [I 2022-10-09 14:07:04] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00308 [I 2022-10-09 14:07:04] ...trap/engines/engine.py.230: loss: 0.11251 [I 2022-10-09 14:07:04] ...optimizers/trijoint.py.151: total_norm: 12.884033 [I 2022-10-09 14:07:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 670/2304 [I 2022-10-09 14:07:13] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:01 [I 2022-10-09 14:07:13] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00302 [I 2022-10-09 14:07:13] ...trap/engines/engine.py.230: loss: 0.13726 [I 2022-10-09 14:07:13] ...optimizers/trijoint.py.151: total_norm: 20.328478 [I 2022-10-09 14:07:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 680/2304 [I 2022-10-09 14:07:21] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:22:51 [I 2022-10-09 14:07:21] ...trap/engines/engine.py.229: process: 0.84591 | load: 0.00296 [I 2022-10-09 14:07:21] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-09 14:07:21] ...optimizers/trijoint.py.151: total_norm: 12.684002 [I 2022-10-09 14:07:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 690/2304 [I 2022-10-09 14:07:30] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:41 [I 2022-10-09 14:07:30] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00297 [I 2022-10-09 14:07:30] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-09 14:07:30] ...optimizers/trijoint.py.151: total_norm: 19.063221 [I 2022-10-09 14:07:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 700/2304 [I 2022-10-09 14:07:38] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:32 [I 2022-10-09 14:07:38] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00317 [I 2022-10-09 14:07:38] ...trap/engines/engine.py.230: loss: 0.17622 [I 2022-10-09 14:07:38] ...optimizers/trijoint.py.151: total_norm: 11.499701 [I 2022-10-09 14:07:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 710/2304 [I 2022-10-09 14:07:47] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:24 [I 2022-10-09 14:07:47] ...trap/engines/engine.py.229: process: 0.83971 | load: 0.00297 [I 2022-10-09 14:07:47] ...trap/engines/engine.py.230: loss: 0.19637 [I 2022-10-09 14:07:47] ...optimizers/trijoint.py.151: total_norm: 23.160234 [I 2022-10-09 14:07:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 720/2304 [I 2022-10-09 14:07:55] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:15 [I 2022-10-09 14:07:55] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00307 [I 2022-10-09 14:07:55] ...trap/engines/engine.py.230: loss: 0.12544 [I 2022-10-09 14:07:55] ...optimizers/trijoint.py.151: total_norm: 19.536821 [I 2022-10-09 14:08:04] ...trap/engines/engine.py.224: train: epoch 43 | batch 730/2304 [I 2022-10-09 14:08:04] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:20 [I 2022-10-09 14:08:04] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00295 [I 2022-10-09 14:08:04] ...trap/engines/engine.py.230: loss: 0.24102 [I 2022-10-09 14:08:04] ...optimizers/trijoint.py.151: total_norm: 16.703140 [I 2022-10-09 14:08:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 740/2304 [I 2022-10-09 14:08:12] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:22:09 [I 2022-10-09 14:08:12] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00294 [I 2022-10-09 14:08:12] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-09 14:08:12] ...optimizers/trijoint.py.151: total_norm: 21.742649 [I 2022-10-09 14:08:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 750/2304 [I 2022-10-09 14:08:21] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:56 [I 2022-10-09 14:08:21] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00295 [I 2022-10-09 14:08:21] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-09 14:08:21] ...optimizers/trijoint.py.151: total_norm: 6.030351 [I 2022-10-09 14:08:29] ...trap/engines/engine.py.224: train: epoch 43 | batch 760/2304 [I 2022-10-09 14:08:29] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:21:49 [I 2022-10-09 14:08:29] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00306 [I 2022-10-09 14:08:29] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-09 14:08:29] ...optimizers/trijoint.py.151: total_norm: 12.713038 [I 2022-10-09 14:08:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 770/2304 [I 2022-10-09 14:08:38] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:38 [I 2022-10-09 14:08:38] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00297 [I 2022-10-09 14:08:38] ...trap/engines/engine.py.230: loss: 0.19071 [I 2022-10-09 14:08:38] ...optimizers/trijoint.py.151: total_norm: 23.570627 [I 2022-10-09 14:08:46] ...trap/engines/engine.py.224: train: epoch 43 | batch 780/2304 [I 2022-10-09 14:08:46] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:21:45 [I 2022-10-09 14:08:46] ...trap/engines/engine.py.229: process: 0.86989 | load: 0.00294 [I 2022-10-09 14:08:46] ...trap/engines/engine.py.230: loss: 0.14125 [I 2022-10-09 14:08:46] ...optimizers/trijoint.py.151: total_norm: 15.484078 [I 2022-10-09 14:08:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 790/2304 [I 2022-10-09 14:08:55] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:27 [I 2022-10-09 14:08:55] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00297 [I 2022-10-09 14:08:55] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-09 14:08:55] ...optimizers/trijoint.py.151: total_norm: 17.941294 [I 2022-10-09 14:09:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 800/2304 [I 2022-10-09 14:09:03] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:21:15 [I 2022-10-09 14:09:03] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00295 [I 2022-10-09 14:09:03] ...trap/engines/engine.py.230: loss: 0.19255 [I 2022-10-09 14:09:03] ...optimizers/trijoint.py.151: total_norm: 22.295885 [I 2022-10-09 14:09:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 810/2304 [I 2022-10-09 14:09:12] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:20:59 [I 2022-10-09 14:09:12] ...trap/engines/engine.py.229: process: 0.84468 | load: 0.00292 [I 2022-10-09 14:09:12] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-09 14:09:12] ...optimizers/trijoint.py.151: total_norm: 18.852055 [I 2022-10-09 14:09:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 820/2304 [I 2022-10-09 14:09:20] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:20:49 [I 2022-10-09 14:09:20] ...trap/engines/engine.py.229: process: 0.83274 | load: 0.00297 [I 2022-10-09 14:09:20] ...trap/engines/engine.py.230: loss: 0.10477 [I 2022-10-09 14:09:20] ...optimizers/trijoint.py.151: total_norm: 9.130733 [I 2022-10-09 14:09:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 830/2304 [I 2022-10-09 14:09:28] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:50 [I 2022-10-09 14:09:28] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00298 [I 2022-10-09 14:09:28] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-09 14:09:28] ...optimizers/trijoint.py.151: total_norm: 21.316887 [I 2022-10-09 14:09:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 840/2304 [I 2022-10-09 14:09:37] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:39 [I 2022-10-09 14:09:37] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00301 [I 2022-10-09 14:09:37] ...trap/engines/engine.py.230: loss: 0.21057 [I 2022-10-09 14:09:37] ...optimizers/trijoint.py.151: total_norm: 18.104849 [I 2022-10-09 14:09:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 850/2304 [I 2022-10-09 14:09:45] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:20:28 [I 2022-10-09 14:09:45] ...trap/engines/engine.py.229: process: 0.85261 | load: 0.00300 [I 2022-10-09 14:09:45] ...trap/engines/engine.py.230: loss: 0.16668 [I 2022-10-09 14:09:45] ...optimizers/trijoint.py.151: total_norm: 49.369770 [I 2022-10-09 14:09:54] ...trap/engines/engine.py.224: train: epoch 43 | batch 860/2304 [I 2022-10-09 14:09:54] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:18 [I 2022-10-09 14:09:54] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00296 [I 2022-10-09 14:09:54] ...trap/engines/engine.py.230: loss: 0.20950 [I 2022-10-09 14:09:54] ...optimizers/trijoint.py.151: total_norm: 24.588856 [I 2022-10-09 14:10:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 870/2304 [I 2022-10-09 14:10:02] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:09 [I 2022-10-09 14:10:02] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00294 [I 2022-10-09 14:10:02] ...trap/engines/engine.py.230: loss: 0.18525 [I 2022-10-09 14:10:02] ...optimizers/trijoint.py.151: total_norm: 19.225325 [I 2022-10-09 14:10:11] ...trap/engines/engine.py.224: train: epoch 43 | batch 880/2304 [I 2022-10-09 14:10:11] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:03 [I 2022-10-09 14:10:11] ...trap/engines/engine.py.229: process: 0.83245 | load: 0.00296 [I 2022-10-09 14:10:11] ...trap/engines/engine.py.230: loss: 0.17258 [I 2022-10-09 14:10:11] ...optimizers/trijoint.py.151: total_norm: 18.620050 [I 2022-10-09 14:10:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 890/2304 [I 2022-10-09 14:10:19] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:19:54 [I 2022-10-09 14:10:19] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00296 [I 2022-10-09 14:10:19] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-09 14:10:19] ...optimizers/trijoint.py.151: total_norm: 13.171527 [I 2022-10-09 14:10:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 900/2304 [I 2022-10-09 14:10:28] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:19:48 [I 2022-10-09 14:10:28] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00293 [I 2022-10-09 14:10:28] ...trap/engines/engine.py.230: loss: 0.19358 [I 2022-10-09 14:10:28] ...optimizers/trijoint.py.151: total_norm: 14.513803 [I 2022-10-09 14:10:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 910/2304 [I 2022-10-09 14:10:36] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:19:40 [I 2022-10-09 14:10:36] ...trap/engines/engine.py.229: process: 0.86369 | load: 0.00292 [I 2022-10-09 14:10:36] ...trap/engines/engine.py.230: loss: 0.13943 [I 2022-10-09 14:10:36] ...optimizers/trijoint.py.151: total_norm: 14.603411 [I 2022-10-09 14:10:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 920/2304 [I 2022-10-09 14:10:45] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:26 [I 2022-10-09 14:10:45] ...trap/engines/engine.py.229: process: 0.83058 | load: 0.00291 [I 2022-10-09 14:10:45] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-09 14:10:45] ...optimizers/trijoint.py.151: total_norm: 22.469934 [I 2022-10-09 14:10:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 930/2304 [I 2022-10-09 14:10:53] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:29 [I 2022-10-09 14:10:53] ...trap/engines/engine.py.229: process: 0.86378 | load: 0.00303 [I 2022-10-09 14:10:53] ...trap/engines/engine.py.230: loss: 0.13803 [I 2022-10-09 14:10:53] ...optimizers/trijoint.py.151: total_norm: 17.099192 [I 2022-10-09 14:11:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 940/2304 [I 2022-10-09 14:11:02] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:16 [I 2022-10-09 14:11:02] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00301 [I 2022-10-09 14:11:02] ...trap/engines/engine.py.230: loss: 0.21088 [I 2022-10-09 14:11:02] ...optimizers/trijoint.py.151: total_norm: 45.828442 [I 2022-10-09 14:11:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 950/2304 [I 2022-10-09 14:11:10] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:07 [I 2022-10-09 14:11:10] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00292 [I 2022-10-09 14:11:10] ...trap/engines/engine.py.230: loss: 0.12846 [I 2022-10-09 14:11:10] ...optimizers/trijoint.py.151: total_norm: 9.529903 [I 2022-10-09 14:11:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 960/2304 [I 2022-10-09 14:11:19] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:18:51 [I 2022-10-09 14:11:19] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00294 [I 2022-10-09 14:11:19] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-09 14:11:19] ...optimizers/trijoint.py.151: total_norm: 10.108126 [I 2022-10-09 14:11:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 970/2304 [I 2022-10-09 14:11:27] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:18:55 [I 2022-10-09 14:11:27] ...trap/engines/engine.py.229: process: 0.86375 | load: 0.00302 [I 2022-10-09 14:11:27] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-09 14:11:27] ...optimizers/trijoint.py.151: total_norm: 27.389599 [I 2022-10-09 14:11:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 980/2304 [I 2022-10-09 14:11:36] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:45 [I 2022-10-09 14:11:36] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00298 [I 2022-10-09 14:11:36] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-09 14:11:36] ...optimizers/trijoint.py.151: total_norm: 14.146429 [I 2022-10-09 14:11:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 990/2304 [I 2022-10-09 14:11:44] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:35 [I 2022-10-09 14:11:44] ...trap/engines/engine.py.229: process: 0.86290 | load: 0.00299 [I 2022-10-09 14:11:44] ...trap/engines/engine.py.230: loss: 0.17709 [I 2022-10-09 14:11:44] ...optimizers/trijoint.py.151: total_norm: 16.480974 [I 2022-10-09 14:11:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 1000/2304 [I 2022-10-09 14:11:53] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:26 [I 2022-10-09 14:11:53] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00296 [I 2022-10-09 14:11:53] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-09 14:11:53] ...optimizers/trijoint.py.151: total_norm: 27.008966 [I 2022-10-09 14:12:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 1010/2304 [I 2022-10-09 14:12:01] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:15 [I 2022-10-09 14:12:01] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00300 [I 2022-10-09 14:12:01] ...trap/engines/engine.py.230: loss: 0.18878 [I 2022-10-09 14:12:01] ...optimizers/trijoint.py.151: total_norm: 28.088469 [I 2022-10-09 14:12:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 1020/2304 [I 2022-10-09 14:12:10] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:17 [I 2022-10-09 14:12:10] ...trap/engines/engine.py.229: process: 0.86010 | load: 0.00304 [I 2022-10-09 14:12:10] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-09 14:12:10] ...optimizers/trijoint.py.151: total_norm: 25.716009 [I 2022-10-09 14:12:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 1030/2304 [I 2022-10-09 14:12:18] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:17:58 [I 2022-10-09 14:12:18] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00313 [I 2022-10-09 14:12:18] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-09 14:12:18] ...optimizers/trijoint.py.151: total_norm: 15.432741 [I 2022-10-09 14:12:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 1040/2304 [I 2022-10-09 14:12:27] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:47 [I 2022-10-09 14:12:27] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00294 [I 2022-10-09 14:12:27] ...trap/engines/engine.py.230: loss: 0.19779 [I 2022-10-09 14:12:27] ...optimizers/trijoint.py.151: total_norm: 20.143345 [I 2022-10-09 14:12:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 1050/2304 [I 2022-10-09 14:12:35] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:32 [I 2022-10-09 14:12:35] ...trap/engines/engine.py.229: process: 0.84158 | load: 0.00296 [I 2022-10-09 14:12:35] ...trap/engines/engine.py.230: loss: 0.14752 [I 2022-10-09 14:12:35] ...optimizers/trijoint.py.151: total_norm: 35.282833 [I 2022-10-09 14:13:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 1060/2304 [I 2022-10-09 14:13:02] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:48:27 [I 2022-10-09 14:13:02] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00295 [I 2022-10-09 14:13:02] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-09 14:13:02] ...optimizers/trijoint.py.151: total_norm: 14.632566 [I 2022-10-09 14:13:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 1070/2304 [I 2022-10-09 14:13:10] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:20:44 [I 2022-10-09 14:13:10] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00296 [I 2022-10-09 14:13:10] ...trap/engines/engine.py.230: loss: 0.26842 [I 2022-10-09 14:13:10] ...optimizers/trijoint.py.151: total_norm: 20.390875 [I 2022-10-09 14:13:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 1080/2304 [I 2022-10-09 14:13:19] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:32 [I 2022-10-09 14:13:19] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00301 [I 2022-10-09 14:13:19] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-09 14:13:19] ...optimizers/trijoint.py.151: total_norm: 22.357218 [I 2022-10-09 14:13:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 1090/2304 [I 2022-10-09 14:13:27] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:06 [I 2022-10-09 14:13:27] ...trap/engines/engine.py.229: process: 0.83492 | load: 0.00297 [I 2022-10-09 14:13:27] ...trap/engines/engine.py.230: loss: 0.24126 [I 2022-10-09 14:13:27] ...optimizers/trijoint.py.151: total_norm: 20.147602 [I 2022-10-09 14:13:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 1100/2304 [I 2022-10-09 14:13:36] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:01 [I 2022-10-09 14:13:36] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00298 [I 2022-10-09 14:13:36] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-09 14:13:36] ...optimizers/trijoint.py.151: total_norm: 14.966002 [I 2022-10-09 14:13:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 1110/2304 [I 2022-10-09 14:13:44] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:51 [I 2022-10-09 14:13:44] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00298 [I 2022-10-09 14:13:44] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-09 14:13:44] ...optimizers/trijoint.py.151: total_norm: 17.731585 [I 2022-10-09 14:13:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 1120/2304 [I 2022-10-09 14:13:53] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:42 [I 2022-10-09 14:13:53] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00301 [I 2022-10-09 14:13:53] ...trap/engines/engine.py.230: loss: 0.22380 [I 2022-10-09 14:13:53] ...optimizers/trijoint.py.151: total_norm: 26.002005 [I 2022-10-09 14:14:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 1130/2304 [I 2022-10-09 14:14:01] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:32 [I 2022-10-09 14:14:01] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00299 [I 2022-10-09 14:14:01] ...trap/engines/engine.py.230: loss: 0.19519 [I 2022-10-09 14:14:01] ...optimizers/trijoint.py.151: total_norm: 10.973228 [I 2022-10-09 14:14:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 1140/2304 [I 2022-10-09 14:14:09] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:20 [I 2022-10-09 14:14:09] ...trap/engines/engine.py.229: process: 0.82643 | load: 0.00294 [I 2022-10-09 14:14:09] ...trap/engines/engine.py.230: loss: 0.15055 [I 2022-10-09 14:14:09] ...optimizers/trijoint.py.151: total_norm: 18.979300 [I 2022-10-09 14:14:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 1150/2304 [I 2022-10-09 14:14:18] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:14 [I 2022-10-09 14:14:18] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00294 [I 2022-10-09 14:14:18] ...trap/engines/engine.py.230: loss: 0.17948 [I 2022-10-09 14:14:18] ...optimizers/trijoint.py.151: total_norm: 14.015970 [I 2022-10-09 14:14:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 1160/2304 [I 2022-10-09 14:14:26] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:12 [I 2022-10-09 14:14:26] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00296 [I 2022-10-09 14:14:26] ...trap/engines/engine.py.230: loss: 0.16585 [I 2022-10-09 14:14:26] ...optimizers/trijoint.py.151: total_norm: 15.523714 [I 2022-10-09 14:14:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 1170/2304 [I 2022-10-09 14:14:35] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:00 [I 2022-10-09 14:14:35] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00297 [I 2022-10-09 14:14:35] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-09 14:14:35] ...optimizers/trijoint.py.151: total_norm: 12.633500 [I 2022-10-09 14:14:43] ...trap/engines/engine.py.224: train: epoch 43 | batch 1180/2304 [I 2022-10-09 14:14:43] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:51 [I 2022-10-09 14:14:43] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00302 [I 2022-10-09 14:14:43] ...trap/engines/engine.py.230: loss: 0.14240 [I 2022-10-09 14:14:43] ...optimizers/trijoint.py.151: total_norm: 22.981838 [I 2022-10-09 14:14:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 1190/2304 [I 2022-10-09 14:14:52] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:41 [I 2022-10-09 14:14:52] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00298 [I 2022-10-09 14:14:52] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-09 14:14:52] ...optimizers/trijoint.py.151: total_norm: 15.261260 [I 2022-10-09 14:15:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 1200/2304 [I 2022-10-09 14:15:00] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:34 [I 2022-10-09 14:15:00] ...trap/engines/engine.py.229: process: 0.86347 | load: 0.00296 [I 2022-10-09 14:15:00] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-09 14:15:00] ...optimizers/trijoint.py.151: total_norm: 9.834223 [I 2022-10-09 14:15:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 1210/2304 [I 2022-10-09 14:15:09] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:30 [I 2022-10-09 14:15:09] ...trap/engines/engine.py.229: process: 0.86620 | load: 0.00296 [I 2022-10-09 14:15:09] ...trap/engines/engine.py.230: loss: 0.20468 [I 2022-10-09 14:15:09] ...optimizers/trijoint.py.151: total_norm: 16.267494 [I 2022-10-09 14:15:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 1220/2304 [I 2022-10-09 14:15:17] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:16 [I 2022-10-09 14:15:17] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00305 [I 2022-10-09 14:15:17] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-09 14:15:17] ...optimizers/trijoint.py.151: total_norm: 13.692285 [I 2022-10-09 14:15:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 1230/2304 [I 2022-10-09 14:15:26] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:02 [I 2022-10-09 14:15:26] ...trap/engines/engine.py.229: process: 0.83366 | load: 0.00296 [I 2022-10-09 14:15:26] ...trap/engines/engine.py.230: loss: 0.17103 [I 2022-10-09 14:15:26] ...optimizers/trijoint.py.151: total_norm: 10.553267 [I 2022-10-09 14:15:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 1240/2304 [I 2022-10-09 14:15:34] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:14:56 [I 2022-10-09 14:15:34] ...trap/engines/engine.py.229: process: 0.83114 | load: 0.00295 [I 2022-10-09 14:15:34] ...trap/engines/engine.py.230: loss: 0.21510 [I 2022-10-09 14:15:34] ...optimizers/trijoint.py.151: total_norm: 25.379427 [I 2022-10-09 14:15:43] ...trap/engines/engine.py.224: train: epoch 43 | batch 1250/2304 [I 2022-10-09 14:15:43] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:49 [I 2022-10-09 14:15:43] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00294 [I 2022-10-09 14:15:43] ...trap/engines/engine.py.230: loss: 0.14968 [I 2022-10-09 14:15:43] ...optimizers/trijoint.py.151: total_norm: 20.988216 [I 2022-10-09 14:15:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 1260/2304 [I 2022-10-09 14:15:51] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:40 [I 2022-10-09 14:15:51] ...trap/engines/engine.py.229: process: 0.82937 | load: 0.00296 [I 2022-10-09 14:15:51] ...trap/engines/engine.py.230: loss: 0.18210 [I 2022-10-09 14:15:51] ...optimizers/trijoint.py.151: total_norm: 16.625858 [I 2022-10-09 14:15:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 1270/2304 [I 2022-10-09 14:15:59] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:36 [I 2022-10-09 14:15:59] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00305 [I 2022-10-09 14:15:59] ...trap/engines/engine.py.230: loss: 0.20207 [I 2022-10-09 14:15:59] ...optimizers/trijoint.py.151: total_norm: 18.711344 [I 2022-10-09 14:16:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 1280/2304 [I 2022-10-09 14:16:08] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:27 [I 2022-10-09 14:16:08] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00296 [I 2022-10-09 14:16:08] ...trap/engines/engine.py.230: loss: 0.18621 [I 2022-10-09 14:16:08] ...optimizers/trijoint.py.151: total_norm: 12.958221 [I 2022-10-09 14:16:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 1290/2304 [I 2022-10-09 14:16:16] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:20 [I 2022-10-09 14:16:16] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00292 [I 2022-10-09 14:16:16] ...trap/engines/engine.py.230: loss: 0.15736 [I 2022-10-09 14:16:16] ...optimizers/trijoint.py.151: total_norm: 22.247311 [I 2022-10-09 14:16:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 1300/2304 [I 2022-10-09 14:16:25] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:14 [I 2022-10-09 14:16:25] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00292 [I 2022-10-09 14:16:25] ...trap/engines/engine.py.230: loss: 0.21179 [I 2022-10-09 14:16:25] ...optimizers/trijoint.py.151: total_norm: 33.566078 [I 2022-10-09 14:16:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 1310/2304 [I 2022-10-09 14:16:33] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:14:01 [I 2022-10-09 14:16:33] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00293 [I 2022-10-09 14:16:33] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-09 14:16:33] ...optimizers/trijoint.py.151: total_norm: 18.654579 [I 2022-10-09 14:16:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 1320/2304 [I 2022-10-09 14:16:42] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:13:57 [I 2022-10-09 14:16:42] ...trap/engines/engine.py.229: process: 0.87199 | load: 0.00294 [I 2022-10-09 14:16:42] ...trap/engines/engine.py.230: loss: 0.19839 [I 2022-10-09 14:16:42] ...optimizers/trijoint.py.151: total_norm: 19.819365 [I 2022-10-09 14:16:50] ...trap/engines/engine.py.224: train: epoch 43 | batch 1330/2304 [I 2022-10-09 14:16:50] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:44 [I 2022-10-09 14:16:50] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00303 [I 2022-10-09 14:16:50] ...trap/engines/engine.py.230: loss: 0.17686 [I 2022-10-09 14:16:50] ...optimizers/trijoint.py.151: total_norm: 15.266433 [I 2022-10-09 14:16:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 1340/2304 [I 2022-10-09 14:16:59] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:33 [I 2022-10-09 14:16:59] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00293 [I 2022-10-09 14:16:59] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-09 14:16:59] ...optimizers/trijoint.py.151: total_norm: 31.843174 [I 2022-10-09 14:17:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 1350/2304 [I 2022-10-09 14:17:07] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:27 [I 2022-10-09 14:17:07] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00293 [I 2022-10-09 14:17:07] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-09 14:17:07] ...optimizers/trijoint.py.151: total_norm: 15.283587 [I 2022-10-09 14:17:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 1360/2304 [I 2022-10-09 14:17:16] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:20 [I 2022-10-09 14:17:16] ...trap/engines/engine.py.229: process: 0.86333 | load: 0.00294 [I 2022-10-09 14:17:16] ...trap/engines/engine.py.230: loss: 0.18167 [I 2022-10-09 14:17:16] ...optimizers/trijoint.py.151: total_norm: 13.947681 [I 2022-10-09 14:17:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 1370/2304 [I 2022-10-09 14:17:24] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:08 [I 2022-10-09 14:17:24] ...trap/engines/engine.py.229: process: 0.82948 | load: 0.00297 [I 2022-10-09 14:17:24] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-09 14:17:24] ...optimizers/trijoint.py.151: total_norm: 21.415564 [I 2022-10-09 14:17:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 1380/2304 [I 2022-10-09 14:17:33] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:03 [I 2022-10-09 14:17:33] ...trap/engines/engine.py.229: process: 0.86357 | load: 0.00302 [I 2022-10-09 14:17:33] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-09 14:17:33] ...optimizers/trijoint.py.151: total_norm: 18.303143 [I 2022-10-09 14:17:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 1390/2304 [I 2022-10-09 14:17:41] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:48 [I 2022-10-09 14:17:41] ...trap/engines/engine.py.229: process: 0.81917 | load: 0.00293 [I 2022-10-09 14:17:41] ...trap/engines/engine.py.230: loss: 0.16407 [I 2022-10-09 14:17:41] ...optimizers/trijoint.py.151: total_norm: 18.620203 [I 2022-10-09 14:17:50] ...trap/engines/engine.py.224: train: epoch 43 | batch 1400/2304 [I 2022-10-09 14:17:50] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:42 [I 2022-10-09 14:17:50] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00295 [I 2022-10-09 14:17:50] ...trap/engines/engine.py.230: loss: 0.16397 [I 2022-10-09 14:17:50] ...optimizers/trijoint.py.151: total_norm: 17.482979 [I 2022-10-09 14:17:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 1410/2304 [I 2022-10-09 14:17:58] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:35 [I 2022-10-09 14:17:58] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00294 [I 2022-10-09 14:17:58] ...trap/engines/engine.py.230: loss: 0.19478 [I 2022-10-09 14:17:58] ...optimizers/trijoint.py.151: total_norm: 13.055371 [I 2022-10-09 14:18:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 1420/2304 [I 2022-10-09 14:18:07] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:32 [I 2022-10-09 14:18:07] ...trap/engines/engine.py.229: process: 0.86042 | load: 0.00295 [I 2022-10-09 14:18:07] ...trap/engines/engine.py.230: loss: 0.12723 [I 2022-10-09 14:18:07] ...optimizers/trijoint.py.151: total_norm: 26.515097 [I 2022-10-09 14:18:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 1430/2304 [I 2022-10-09 14:18:15] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:20 [I 2022-10-09 14:18:15] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00293 [I 2022-10-09 14:18:15] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-09 14:18:15] ...optimizers/trijoint.py.151: total_norm: 12.614119 [I 2022-10-09 14:18:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 1440/2304 [I 2022-10-09 14:18:24] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:14 [I 2022-10-09 14:18:24] ...trap/engines/engine.py.229: process: 0.86820 | load: 0.00297 [I 2022-10-09 14:18:24] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-09 14:18:24] ...optimizers/trijoint.py.151: total_norm: 14.107819 [I 2022-10-09 14:18:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 1450/2304 [I 2022-10-09 14:18:32] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:05 [I 2022-10-09 14:18:32] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00293 [I 2022-10-09 14:18:32] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-09 14:18:32] ...optimizers/trijoint.py.151: total_norm: 10.016376 [I 2022-10-09 14:18:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 1460/2304 [I 2022-10-09 14:18:41] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:11:58 [I 2022-10-09 14:18:41] ...trap/engines/engine.py.229: process: 0.87139 | load: 0.00292 [I 2022-10-09 14:18:41] ...trap/engines/engine.py.230: loss: 0.18531 [I 2022-10-09 14:18:41] ...optimizers/trijoint.py.151: total_norm: 13.714771 [I 2022-10-09 14:18:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 1470/2304 [I 2022-10-09 14:18:49] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:43 [I 2022-10-09 14:18:49] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00292 [I 2022-10-09 14:18:49] ...trap/engines/engine.py.230: loss: 0.16040 [I 2022-10-09 14:18:49] ...optimizers/trijoint.py.151: total_norm: 20.006659 [I 2022-10-09 14:18:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 1480/2304 [I 2022-10-09 14:18:58] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:36 [I 2022-10-09 14:18:58] ...trap/engines/engine.py.229: process: 0.85102 | load: 0.00295 [I 2022-10-09 14:18:58] ...trap/engines/engine.py.230: loss: 0.17597 [I 2022-10-09 14:18:58] ...optimizers/trijoint.py.151: total_norm: 12.541652 [I 2022-10-09 14:19:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 1490/2304 [I 2022-10-09 14:19:06] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:28 [I 2022-10-09 14:19:06] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00314 [I 2022-10-09 14:19:06] ...trap/engines/engine.py.230: loss: 0.20791 [I 2022-10-09 14:19:06] ...optimizers/trijoint.py.151: total_norm: 14.086017 [I 2022-10-09 14:19:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 1500/2304 [I 2022-10-09 14:19:15] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:20 [I 2022-10-09 14:19:15] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00296 [I 2022-10-09 14:19:15] ...trap/engines/engine.py.230: loss: 0.17932 [I 2022-10-09 14:19:15] ...optimizers/trijoint.py.151: total_norm: 26.025959 [I 2022-10-09 14:19:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 1510/2304 [I 2022-10-09 14:19:23] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:11 [I 2022-10-09 14:19:23] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00295 [I 2022-10-09 14:19:23] ...trap/engines/engine.py.230: loss: 0.19317 [I 2022-10-09 14:19:23] ...optimizers/trijoint.py.151: total_norm: 21.097456 [I 2022-10-09 14:19:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 1520/2304 [I 2022-10-09 14:19:31] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:11:04 [I 2022-10-09 14:19:31] ...trap/engines/engine.py.229: process: 0.85654 | load: 0.00299 [I 2022-10-09 14:19:31] ...trap/engines/engine.py.230: loss: 0.14458 [I 2022-10-09 14:19:31] ...optimizers/trijoint.py.151: total_norm: 20.689911 [I 2022-10-09 14:19:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 1530/2304 [I 2022-10-09 14:19:40] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:53 [I 2022-10-09 14:19:40] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00302 [I 2022-10-09 14:19:40] ...trap/engines/engine.py.230: loss: 0.20571 [I 2022-10-09 14:19:40] ...optimizers/trijoint.py.151: total_norm: 21.829571 [I 2022-10-09 14:19:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 1540/2304 [I 2022-10-09 14:19:48] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:45 [I 2022-10-09 14:19:48] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00300 [I 2022-10-09 14:19:48] ...trap/engines/engine.py.230: loss: 0.18291 [I 2022-10-09 14:19:48] ...optimizers/trijoint.py.151: total_norm: 13.918376 [I 2022-10-09 14:19:57] ...trap/engines/engine.py.224: train: epoch 43 | batch 1550/2304 [I 2022-10-09 14:19:57] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:36 [I 2022-10-09 14:19:57] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00300 [I 2022-10-09 14:19:57] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-09 14:19:57] ...optimizers/trijoint.py.151: total_norm: 14.556589 [I 2022-10-09 14:20:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 1560/2304 [I 2022-10-09 14:20:05] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:23 [I 2022-10-09 14:20:05] ...trap/engines/engine.py.229: process: 0.82430 | load: 0.00300 [I 2022-10-09 14:20:05] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-09 14:20:05] ...optimizers/trijoint.py.151: total_norm: 13.694405 [I 2022-10-09 14:20:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 1570/2304 [I 2022-10-09 14:20:14] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:19 [I 2022-10-09 14:20:14] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00305 [I 2022-10-09 14:20:14] ...trap/engines/engine.py.230: loss: 0.18819 [I 2022-10-09 14:20:14] ...optimizers/trijoint.py.151: total_norm: 23.422613 [I 2022-10-09 14:20:22] ...trap/engines/engine.py.224: train: epoch 43 | batch 1580/2304 [I 2022-10-09 14:20:22] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:10 [I 2022-10-09 14:20:22] ...trap/engines/engine.py.229: process: 0.83643 | load: 0.00298 [I 2022-10-09 14:20:22] ...trap/engines/engine.py.230: loss: 0.14353 [I 2022-10-09 14:20:22] ...optimizers/trijoint.py.151: total_norm: 7.751420 [I 2022-10-09 14:20:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 1590/2304 [I 2022-10-09 14:20:31] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:59 [I 2022-10-09 14:20:31] ...trap/engines/engine.py.229: process: 0.82827 | load: 0.00295 [I 2022-10-09 14:20:31] ...trap/engines/engine.py.230: loss: 0.10431 [I 2022-10-09 14:20:31] ...optimizers/trijoint.py.151: total_norm: 8.391186 [I 2022-10-09 14:20:39] ...trap/engines/engine.py.224: train: epoch 43 | batch 1600/2304 [I 2022-10-09 14:20:39] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:53 [I 2022-10-09 14:20:39] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00298 [I 2022-10-09 14:20:39] ...trap/engines/engine.py.230: loss: 0.19473 [I 2022-10-09 14:20:39] ...optimizers/trijoint.py.151: total_norm: 17.647152 [I 2022-10-09 14:20:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 1610/2304 [I 2022-10-09 14:20:47] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:42 [I 2022-10-09 14:20:47] ...trap/engines/engine.py.229: process: 0.83015 | load: 0.00301 [I 2022-10-09 14:20:47] ...trap/engines/engine.py.230: loss: 0.20702 [I 2022-10-09 14:20:47] ...optimizers/trijoint.py.151: total_norm: 18.020397 [I 2022-10-09 14:20:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 1620/2304 [I 2022-10-09 14:20:56] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:36 [I 2022-10-09 14:20:56] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00296 [I 2022-10-09 14:20:56] ...trap/engines/engine.py.230: loss: 0.11361 [I 2022-10-09 14:20:56] ...optimizers/trijoint.py.151: total_norm: 30.044725 [I 2022-10-09 14:21:04] ...trap/engines/engine.py.224: train: epoch 43 | batch 1630/2304 [I 2022-10-09 14:21:04] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:26 [I 2022-10-09 14:21:04] ...trap/engines/engine.py.229: process: 0.83351 | load: 0.00295 [I 2022-10-09 14:21:04] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-09 14:21:04] ...optimizers/trijoint.py.151: total_norm: 20.021791 [I 2022-10-09 14:21:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 1640/2304 [I 2022-10-09 14:21:13] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:17 [I 2022-10-09 14:21:13] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00303 [I 2022-10-09 14:21:13] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-09 14:21:13] ...optimizers/trijoint.py.151: total_norm: 18.571314 [I 2022-10-09 14:21:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 1650/2304 [I 2022-10-09 14:21:21] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:11 [I 2022-10-09 14:21:21] ...trap/engines/engine.py.229: process: 0.85660 | load: 0.00296 [I 2022-10-09 14:21:21] ...trap/engines/engine.py.230: loss: 0.15234 [I 2022-10-09 14:21:21] ...optimizers/trijoint.py.151: total_norm: 47.786324 [I 2022-10-09 14:21:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 1660/2304 [I 2022-10-09 14:21:30] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:00 [I 2022-10-09 14:21:30] ...trap/engines/engine.py.229: process: 0.82753 | load: 0.00293 [I 2022-10-09 14:21:30] ...trap/engines/engine.py.230: loss: 0.15135 [I 2022-10-09 14:21:30] ...optimizers/trijoint.py.151: total_norm: 18.108868 [I 2022-10-09 14:21:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 1670/2304 [I 2022-10-09 14:21:38] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:54 [I 2022-10-09 14:21:38] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00299 [I 2022-10-09 14:21:38] ...trap/engines/engine.py.230: loss: 0.15276 [I 2022-10-09 14:21:38] ...optimizers/trijoint.py.151: total_norm: 12.797237 [I 2022-10-09 14:21:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 1680/2304 [I 2022-10-09 14:21:47] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:46 [I 2022-10-09 14:21:47] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00296 [I 2022-10-09 14:21:47] ...trap/engines/engine.py.230: loss: 0.18682 [I 2022-10-09 14:21:47] ...optimizers/trijoint.py.151: total_norm: 17.488594 [I 2022-10-09 14:21:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 1690/2304 [I 2022-10-09 14:21:55] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:39 [I 2022-10-09 14:21:55] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00306 [I 2022-10-09 14:21:55] ...trap/engines/engine.py.230: loss: 0.21306 [I 2022-10-09 14:21:55] ...optimizers/trijoint.py.151: total_norm: 47.991608 [I 2022-10-09 14:22:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 1700/2304 [I 2022-10-09 14:22:03] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:32 [I 2022-10-09 14:22:03] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00300 [I 2022-10-09 14:22:03] ...trap/engines/engine.py.230: loss: 0.18177 [I 2022-10-09 14:22:03] ...optimizers/trijoint.py.151: total_norm: 24.025249 [I 2022-10-09 14:22:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 1710/2304 [I 2022-10-09 14:22:12] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:21 [I 2022-10-09 14:22:12] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00297 [I 2022-10-09 14:22:12] ...trap/engines/engine.py.230: loss: 0.13780 [I 2022-10-09 14:22:12] ...optimizers/trijoint.py.151: total_norm: 14.184271 [I 2022-10-09 14:22:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 1720/2304 [I 2022-10-09 14:22:20] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:13 [I 2022-10-09 14:22:20] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00299 [I 2022-10-09 14:22:20] ...trap/engines/engine.py.230: loss: 0.16433 [I 2022-10-09 14:22:20] ...optimizers/trijoint.py.151: total_norm: 17.942982 [I 2022-10-09 14:22:29] ...trap/engines/engine.py.224: train: epoch 43 | batch 1730/2304 [I 2022-10-09 14:22:29] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:03 [I 2022-10-09 14:22:29] ...trap/engines/engine.py.229: process: 0.82540 | load: 0.00298 [I 2022-10-09 14:22:29] ...trap/engines/engine.py.230: loss: 0.18680 [I 2022-10-09 14:22:29] ...optimizers/trijoint.py.151: total_norm: 10.932623 [I 2022-10-09 14:22:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 1740/2304 [I 2022-10-09 14:22:37] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:54 [I 2022-10-09 14:22:37] ...trap/engines/engine.py.229: process: 0.84418 | load: 0.00296 [I 2022-10-09 14:22:37] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-09 14:22:37] ...optimizers/trijoint.py.151: total_norm: 31.180756 [I 2022-10-09 14:22:46] ...trap/engines/engine.py.224: train: epoch 43 | batch 1750/2304 [I 2022-10-09 14:22:46] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:46 [I 2022-10-09 14:22:46] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00304 [I 2022-10-09 14:22:46] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-09 14:22:46] ...optimizers/trijoint.py.151: total_norm: 15.254649 [I 2022-10-09 14:22:54] ...trap/engines/engine.py.224: train: epoch 43 | batch 1760/2304 [I 2022-10-09 14:22:54] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:39 [I 2022-10-09 14:22:54] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00303 [I 2022-10-09 14:22:54] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-09 14:22:54] ...optimizers/trijoint.py.151: total_norm: 12.327531 [I 2022-10-09 14:23:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 1770/2304 [I 2022-10-09 14:23:03] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:30 [I 2022-10-09 14:23:03] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00295 [I 2022-10-09 14:23:03] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-09 14:23:03] ...optimizers/trijoint.py.151: total_norm: 19.825451 [I 2022-10-09 14:23:11] ...trap/engines/engine.py.224: train: epoch 43 | batch 1780/2304 [I 2022-10-09 14:23:11] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:23 [I 2022-10-09 14:23:11] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00296 [I 2022-10-09 14:23:11] ...trap/engines/engine.py.230: loss: 0.22530 [I 2022-10-09 14:23:11] ...optimizers/trijoint.py.151: total_norm: 32.422318 [I 2022-10-09 14:23:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 1790/2304 [I 2022-10-09 14:23:20] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:13 [I 2022-10-09 14:23:20] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00305 [I 2022-10-09 14:23:20] ...trap/engines/engine.py.230: loss: 0.10255 [I 2022-10-09 14:23:20] ...optimizers/trijoint.py.151: total_norm: 9.682166 [I 2022-10-09 14:23:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 1800/2304 [I 2022-10-09 14:23:28] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:05 [I 2022-10-09 14:23:28] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00296 [I 2022-10-09 14:23:28] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-09 14:23:28] ...optimizers/trijoint.py.151: total_norm: 19.637627 [I 2022-10-09 14:23:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 1810/2304 [I 2022-10-09 14:23:36] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:56 [I 2022-10-09 14:23:36] ...trap/engines/engine.py.229: process: 0.84373 | load: 0.00298 [I 2022-10-09 14:23:36] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-09 14:23:36] ...optimizers/trijoint.py.151: total_norm: 8.182491 [I 2022-10-09 14:23:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 1820/2304 [I 2022-10-09 14:23:45] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:47 [I 2022-10-09 14:23:45] ...trap/engines/engine.py.229: process: 0.83238 | load: 0.00303 [I 2022-10-09 14:23:45] ...trap/engines/engine.py.230: loss: 0.16167 [I 2022-10-09 14:23:45] ...optimizers/trijoint.py.151: total_norm: 11.282259 [I 2022-10-09 14:23:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 1830/2304 [I 2022-10-09 14:23:53] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:38 [I 2022-10-09 14:23:53] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00307 [I 2022-10-09 14:23:53] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-09 14:23:53] ...optimizers/trijoint.py.151: total_norm: 11.243679 [I 2022-10-09 14:24:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 1840/2304 [I 2022-10-09 14:24:02] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:32 [I 2022-10-09 14:24:02] ...trap/engines/engine.py.229: process: 0.83231 | load: 0.00297 [I 2022-10-09 14:24:02] ...trap/engines/engine.py.230: loss: 0.13701 [I 2022-10-09 14:24:02] ...optimizers/trijoint.py.151: total_norm: 22.610641 [I 2022-10-09 14:24:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 1850/2304 [I 2022-10-09 14:24:10] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:19 [I 2022-10-09 14:24:10] ...trap/engines/engine.py.229: process: 0.83724 | load: 0.00303 [I 2022-10-09 14:24:10] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-09 14:24:10] ...optimizers/trijoint.py.151: total_norm: 15.659192 [I 2022-10-09 14:24:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 1860/2304 [I 2022-10-09 14:24:19] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:14 [I 2022-10-09 14:24:19] ...trap/engines/engine.py.229: process: 0.84832 | load: 0.00298 [I 2022-10-09 14:24:19] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-09 14:24:19] ...optimizers/trijoint.py.151: total_norm: 9.488467 [I 2022-10-09 14:24:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 1870/2304 [I 2022-10-09 14:24:27] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:04 [I 2022-10-09 14:24:27] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00292 [I 2022-10-09 14:24:27] ...trap/engines/engine.py.230: loss: 0.20720 [I 2022-10-09 14:24:27] ...optimizers/trijoint.py.151: total_norm: 14.189817 [I 2022-10-09 14:24:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 1880/2304 [I 2022-10-09 14:24:36] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:58 [I 2022-10-09 14:24:36] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00294 [I 2022-10-09 14:24:36] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-09 14:24:36] ...optimizers/trijoint.py.151: total_norm: 22.203924 [I 2022-10-09 14:24:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 1890/2304 [I 2022-10-09 14:24:44] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:48 [I 2022-10-09 14:24:44] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00294 [I 2022-10-09 14:24:44] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-09 14:24:44] ...optimizers/trijoint.py.151: total_norm: 18.374043 [I 2022-10-09 14:24:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 1900/2304 [I 2022-10-09 14:24:52] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:40 [I 2022-10-09 14:24:52] ...trap/engines/engine.py.229: process: 0.83381 | load: 0.00297 [I 2022-10-09 14:24:52] ...trap/engines/engine.py.230: loss: 0.18941 [I 2022-10-09 14:24:52] ...optimizers/trijoint.py.151: total_norm: 20.581276 [I 2022-10-09 14:25:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 1910/2304 [I 2022-10-09 14:25:01] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:34 [I 2022-10-09 14:25:01] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00298 [I 2022-10-09 14:25:01] ...trap/engines/engine.py.230: loss: 0.13747 [I 2022-10-09 14:25:01] ...optimizers/trijoint.py.151: total_norm: 13.178183 [I 2022-10-09 14:25:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 1920/2304 [I 2022-10-09 14:25:09] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:23 [I 2022-10-09 14:25:09] ...trap/engines/engine.py.229: process: 0.82939 | load: 0.00293 [I 2022-10-09 14:25:09] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-09 14:25:09] ...optimizers/trijoint.py.151: total_norm: 9.789283 [I 2022-10-09 14:25:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 1930/2304 [I 2022-10-09 14:25:18] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:16 [I 2022-10-09 14:25:18] ...trap/engines/engine.py.229: process: 0.84515 | load: 0.00294 [I 2022-10-09 14:25:18] ...trap/engines/engine.py.230: loss: 0.17729 [I 2022-10-09 14:25:18] ...optimizers/trijoint.py.151: total_norm: 15.186334 [I 2022-10-09 14:25:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 1940/2304 [I 2022-10-09 14:25:26] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:06 [I 2022-10-09 14:25:26] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00293 [I 2022-10-09 14:25:26] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-09 14:25:26] ...optimizers/trijoint.py.151: total_norm: 21.182030 [I 2022-10-09 14:25:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 1950/2304 [I 2022-10-09 14:25:35] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:58 [I 2022-10-09 14:25:35] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00298 [I 2022-10-09 14:25:35] ...trap/engines/engine.py.230: loss: 0.14884 [I 2022-10-09 14:25:35] ...optimizers/trijoint.py.151: total_norm: 16.592628 [I 2022-10-09 14:25:43] ...trap/engines/engine.py.224: train: epoch 43 | batch 1960/2304 [I 2022-10-09 14:25:43] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:50 [I 2022-10-09 14:25:43] ...trap/engines/engine.py.229: process: 0.83657 | load: 0.00300 [I 2022-10-09 14:25:43] ...trap/engines/engine.py.230: loss: 0.14543 [I 2022-10-09 14:25:43] ...optimizers/trijoint.py.151: total_norm: 10.635422 [I 2022-10-09 14:25:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 1970/2304 [I 2022-10-09 14:25:52] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:43 [I 2022-10-09 14:25:52] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00298 [I 2022-10-09 14:25:52] ...trap/engines/engine.py.230: loss: 0.13251 [I 2022-10-09 14:25:52] ...optimizers/trijoint.py.151: total_norm: 17.650408 [I 2022-10-09 14:26:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 1980/2304 [I 2022-10-09 14:26:00] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:31 [I 2022-10-09 14:26:00] ...trap/engines/engine.py.229: process: 0.81812 | load: 0.00294 [I 2022-10-09 14:26:00] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-09 14:26:00] ...optimizers/trijoint.py.151: total_norm: 31.010780 [I 2022-10-09 14:26:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 1990/2304 [I 2022-10-09 14:26:08] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:23 [I 2022-10-09 14:26:08] ...trap/engines/engine.py.229: process: 0.83143 | load: 0.00294 [I 2022-10-09 14:26:08] ...trap/engines/engine.py.230: loss: 0.16853 [I 2022-10-09 14:26:08] ...optimizers/trijoint.py.151: total_norm: 36.215214 [I 2022-10-09 14:26:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 2000/2304 [I 2022-10-09 14:26:17] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:16 [I 2022-10-09 14:26:17] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00297 [I 2022-10-09 14:26:17] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-09 14:26:17] ...optimizers/trijoint.py.151: total_norm: 8.535699 [I 2022-10-09 14:26:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 2010/2304 [I 2022-10-09 14:26:25] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:09 [I 2022-10-09 14:26:25] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00299 [I 2022-10-09 14:26:25] ...trap/engines/engine.py.230: loss: 0.16195 [I 2022-10-09 14:26:25] ...optimizers/trijoint.py.151: total_norm: 10.678175 [I 2022-10-09 14:26:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 2020/2304 [I 2022-10-09 14:26:34] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:03:59 [I 2022-10-09 14:26:34] ...trap/engines/engine.py.229: process: 0.83434 | load: 0.00294 [I 2022-10-09 14:26:34] ...trap/engines/engine.py.230: loss: 0.12558 [I 2022-10-09 14:26:34] ...optimizers/trijoint.py.151: total_norm: 10.619454 [I 2022-10-09 14:26:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 2030/2304 [I 2022-10-09 14:26:42] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:51 [I 2022-10-09 14:26:42] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00299 [I 2022-10-09 14:26:42] ...trap/engines/engine.py.230: loss: 0.15054 [I 2022-10-09 14:26:42] ...optimizers/trijoint.py.151: total_norm: 85.862129 [I 2022-10-09 14:26:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 2040/2304 [I 2022-10-09 14:26:51] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:42 [I 2022-10-09 14:26:51] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00295 [I 2022-10-09 14:26:51] ...trap/engines/engine.py.230: loss: 0.22125 [I 2022-10-09 14:26:51] ...optimizers/trijoint.py.151: total_norm: 23.825241 [I 2022-10-09 14:26:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 2050/2304 [I 2022-10-09 14:26:59] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:33 [I 2022-10-09 14:26:59] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00300 [I 2022-10-09 14:26:59] ...trap/engines/engine.py.230: loss: 0.22466 [I 2022-10-09 14:26:59] ...optimizers/trijoint.py.151: total_norm: 49.491421 [I 2022-10-09 14:27:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 2060/2304 [I 2022-10-09 14:27:08] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:25 [I 2022-10-09 14:27:08] ...trap/engines/engine.py.229: process: 0.84939 | load: 0.00295 [I 2022-10-09 14:27:08] ...trap/engines/engine.py.230: loss: 0.22310 [I 2022-10-09 14:27:08] ...optimizers/trijoint.py.151: total_norm: 15.594167 [I 2022-10-09 14:27:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 2070/2304 [I 2022-10-09 14:27:16] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:17 [I 2022-10-09 14:27:16] ...trap/engines/engine.py.229: process: 0.83726 | load: 0.00294 [I 2022-10-09 14:27:16] ...trap/engines/engine.py.230: loss: 0.14826 [I 2022-10-09 14:27:16] ...optimizers/trijoint.py.151: total_norm: 23.760447 [I 2022-10-09 14:27:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 2080/2304 [I 2022-10-09 14:27:25] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:08 [I 2022-10-09 14:27:25] ...trap/engines/engine.py.229: process: 0.82742 | load: 0.00295 [I 2022-10-09 14:27:25] ...trap/engines/engine.py.230: loss: 0.15896 [I 2022-10-09 14:27:25] ...optimizers/trijoint.py.151: total_norm: 22.913694 [I 2022-10-09 14:27:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 2090/2304 [I 2022-10-09 14:27:33] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:00 [I 2022-10-09 14:27:33] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00294 [I 2022-10-09 14:27:33] ...trap/engines/engine.py.230: loss: 0.16100 [I 2022-10-09 14:27:33] ...optimizers/trijoint.py.151: total_norm: 24.372345 [I 2022-10-09 14:27:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 2100/2304 [I 2022-10-09 14:27:41] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:51 [I 2022-10-09 14:27:41] ...trap/engines/engine.py.229: process: 0.83093 | load: 0.00299 [I 2022-10-09 14:27:41] ...trap/engines/engine.py.230: loss: 0.17074 [I 2022-10-09 14:27:41] ...optimizers/trijoint.py.151: total_norm: 22.293034 [I 2022-10-09 14:27:50] ...trap/engines/engine.py.224: train: epoch 43 | batch 2110/2304 [I 2022-10-09 14:27:50] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:43 [I 2022-10-09 14:27:50] ...trap/engines/engine.py.229: process: 0.83619 | load: 0.00296 [I 2022-10-09 14:27:50] ...trap/engines/engine.py.230: loss: 0.26336 [I 2022-10-09 14:27:50] ...optimizers/trijoint.py.151: total_norm: 17.193048 [I 2022-10-09 14:27:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 2120/2304 [I 2022-10-09 14:27:58] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:34 [I 2022-10-09 14:27:58] ...trap/engines/engine.py.229: process: 0.82851 | load: 0.00308 [I 2022-10-09 14:27:58] ...trap/engines/engine.py.230: loss: 0.10917 [I 2022-10-09 14:27:58] ...optimizers/trijoint.py.151: total_norm: 10.991271 [I 2022-10-09 14:28:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 2130/2304 [I 2022-10-09 14:28:07] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:27 [I 2022-10-09 14:28:07] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00295 [I 2022-10-09 14:28:07] ...trap/engines/engine.py.230: loss: 0.17933 [I 2022-10-09 14:28:07] ...optimizers/trijoint.py.151: total_norm: 21.211998 [I 2022-10-09 14:28:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 2140/2304 [I 2022-10-09 14:28:15] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:18 [I 2022-10-09 14:28:15] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00296 [I 2022-10-09 14:28:15] ...trap/engines/engine.py.230: loss: 0.18699 [I 2022-10-09 14:28:15] ...optimizers/trijoint.py.151: total_norm: 13.006802 [I 2022-10-09 14:28:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 2150/2304 [I 2022-10-09 14:28:24] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:10 [I 2022-10-09 14:28:24] ...trap/engines/engine.py.229: process: 0.83707 | load: 0.00297 [I 2022-10-09 14:28:24] ...trap/engines/engine.py.230: loss: 0.13873 [I 2022-10-09 14:28:24] ...optimizers/trijoint.py.151: total_norm: 12.718610 [I 2022-10-09 14:28:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 2160/2304 [I 2022-10-09 14:28:32] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:00 [I 2022-10-09 14:28:32] ...trap/engines/engine.py.229: process: 0.84303 | load: 0.00298 [I 2022-10-09 14:28:32] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-09 14:28:32] ...optimizers/trijoint.py.151: total_norm: 17.863794 [I 2022-10-09 14:28:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 2170/2304 [I 2022-10-09 14:28:40] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:53 [I 2022-10-09 14:28:40] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00298 [I 2022-10-09 14:28:40] ...trap/engines/engine.py.230: loss: 0.09904 [I 2022-10-09 14:28:40] ...optimizers/trijoint.py.151: total_norm: 9.784633 [I 2022-10-09 14:28:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 2180/2304 [I 2022-10-09 14:28:49] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:01:44 [I 2022-10-09 14:28:49] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00297 [I 2022-10-09 14:28:49] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-09 14:28:49] ...optimizers/trijoint.py.151: total_norm: 9.377229 [I 2022-10-09 14:28:57] ...trap/engines/engine.py.224: train: epoch 43 | batch 2190/2304 [I 2022-10-09 14:28:57] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:36 [I 2022-10-09 14:28:57] ...trap/engines/engine.py.229: process: 0.85465 | load: 0.00297 [I 2022-10-09 14:28:57] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-09 14:28:57] ...optimizers/trijoint.py.151: total_norm: 10.217092 [I 2022-10-09 14:29:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 2200/2304 [I 2022-10-09 14:29:06] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:27 [I 2022-10-09 14:29:06] ...trap/engines/engine.py.229: process: 0.83179 | load: 0.00299 [I 2022-10-09 14:29:06] ...trap/engines/engine.py.230: loss: 0.19292 [I 2022-10-09 14:29:06] ...optimizers/trijoint.py.151: total_norm: 14.081031 [I 2022-10-09 14:29:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 2210/2304 [I 2022-10-09 14:29:14] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:19 [I 2022-10-09 14:29:14] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00294 [I 2022-10-09 14:29:14] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-09 14:29:14] ...optimizers/trijoint.py.151: total_norm: 9.031569 [I 2022-10-09 14:29:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 2220/2304 [I 2022-10-09 14:29:23] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:11 [I 2022-10-09 14:29:23] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00296 [I 2022-10-09 14:29:23] ...trap/engines/engine.py.230: loss: 0.15839 [I 2022-10-09 14:29:23] ...optimizers/trijoint.py.151: total_norm: 17.344574 [I 2022-10-09 14:29:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 2230/2304 [I 2022-10-09 14:29:31] ...trap/engines/engine.py.228: elapsed: 0:31:47 | left: 0:01:02 [I 2022-10-09 14:29:31] ...trap/engines/engine.py.229: process: 0.83054 | load: 0.00297 [I 2022-10-09 14:29:31] ...trap/engines/engine.py.230: loss: 0.17994 [I 2022-10-09 14:29:31] ...optimizers/trijoint.py.151: total_norm: 11.149763 [I 2022-10-09 14:29:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 2240/2304 [I 2022-10-09 14:29:40] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:00:53 [I 2022-10-09 14:29:40] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00298 [I 2022-10-09 14:29:40] ...trap/engines/engine.py.230: loss: 0.12234 [I 2022-10-09 14:29:40] ...optimizers/trijoint.py.151: total_norm: 13.184175 [I 2022-10-09 14:29:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 2250/2304 [I 2022-10-09 14:29:48] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:45 [I 2022-10-09 14:29:48] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00296 [I 2022-10-09 14:29:48] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-09 14:29:48] ...optimizers/trijoint.py.151: total_norm: 12.968170 [I 2022-10-09 14:29:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 2260/2304 [I 2022-10-09 14:29:56] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:37 [I 2022-10-09 14:29:56] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00300 [I 2022-10-09 14:29:56] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-09 14:29:56] ...optimizers/trijoint.py.151: total_norm: 16.932444 [I 2022-10-09 14:30:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 2270/2304 [I 2022-10-09 14:30:05] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:28 [I 2022-10-09 14:30:05] ...trap/engines/engine.py.229: process: 0.83968 | load: 0.00297 [I 2022-10-09 14:30:05] ...trap/engines/engine.py.230: loss: 0.13605 [I 2022-10-09 14:30:05] ...optimizers/trijoint.py.151: total_norm: 12.382833 [I 2022-10-09 14:30:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 2280/2304 [I 2022-10-09 14:30:13] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:20 [I 2022-10-09 14:30:13] ...trap/engines/engine.py.229: process: 0.83069 | load: 0.00307 [I 2022-10-09 14:30:13] ...trap/engines/engine.py.230: loss: 0.22787 [I 2022-10-09 14:30:13] ...optimizers/trijoint.py.151: total_norm: 19.700764 [I 2022-10-09 14:30:22] ...trap/engines/engine.py.224: train: epoch 43 | batch 2290/2304 [I 2022-10-09 14:30:22] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:11 [I 2022-10-09 14:30:22] ...trap/engines/engine.py.229: process: 0.83698 | load: 0.00299 [I 2022-10-09 14:30:22] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-09 14:30:22] ...optimizers/trijoint.py.151: total_norm: 12.451202 [I 2022-10-09 14:30:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 2300/2304 [I 2022-10-09 14:30:30] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:03 [I 2022-10-09 14:30:30] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00016 [I 2022-10-09 14:30:30] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-09 14:30:30] ...optimizers/trijoint.py.151: total_norm: 15.518341 [I 2022-10-09 14:30:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 2304/2304 [I 2022-10-09 14:30:34] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:00 [I 2022-10-09 14:30:34] ...trap/engines/engine.py.229: process: 0.82757 | load: 0.00016 [I 2022-10-09 14:30:34] ...trap/engines/engine.py.230: loss: 0.15986 [I 2022-10-09 14:30:34] ...optimizers/trijoint.py.151: total_norm: 19.407162 [S 2022-10-09 14:30:35] ...trap/engines/engine.py.129: train_epoch.epoch: 43 [S 2022-10-09 14:30:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.1573415666430736 [I 2022-10-09 14:30:35] ...s/criterions/triplet.py.84: margin = 0.26500000000000007 [I 2022-10-09 14:30:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 43 [I 2022-10-09 14:30:36] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 14:30:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 14:30:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 14:30:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 14:30:39] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 14:30:39] ...trap/engines/engine.py.321: eval: epoch 43 | batch 0/510 [I 2022-10-09 14:30:39] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:30:07 [I 2022-10-09 14:30:39] ...trap/engines/engine.py.326: process: 3.54443 | load: 3.00714 [I 2022-10-09 14:30:45] ...trap/engines/engine.py.321: eval: epoch 43 | batch 10/510 [I 2022-10-09 14:30:45] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:24 [I 2022-10-09 14:30:45] ...trap/engines/engine.py.326: process: 0.49166 | load: 0.00018 [I 2022-10-09 14:30:51] ...trap/engines/engine.py.321: eval: epoch 43 | batch 20/510 [I 2022-10-09 14:30:51] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:51 [I 2022-10-09 14:30:51] ...trap/engines/engine.py.326: process: 0.63316 | load: 0.00019 [I 2022-10-09 14:30:57] ...trap/engines/engine.py.321: eval: epoch 43 | batch 30/510 [I 2022-10-09 14:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:15 [I 2022-10-09 14:30:57] ...trap/engines/engine.py.326: process: 0.67713 | load: 0.00018 [I 2022-10-09 14:31:04] ...trap/engines/engine.py.321: eval: epoch 43 | batch 40/510 [I 2022-10-09 14:31:04] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:05:13 [I 2022-10-09 14:31:04] ...trap/engines/engine.py.326: process: 0.70052 | load: 0.00018 [I 2022-10-09 14:31:09] ...trap/engines/engine.py.321: eval: epoch 43 | batch 50/510 [I 2022-10-09 14:31:09] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:04:00 [I 2022-10-09 14:31:09] ...trap/engines/engine.py.326: process: 0.58369 | load: 0.00018 [I 2022-10-09 14:31:14] ...trap/engines/engine.py.321: eval: epoch 43 | batch 60/510 [I 2022-10-09 14:31:14] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:23 [I 2022-10-09 14:31:14] ...trap/engines/engine.py.326: process: 0.42680 | load: 0.00018 [I 2022-10-09 14:31:19] ...trap/engines/engine.py.321: eval: epoch 43 | batch 70/510 [I 2022-10-09 14:31:19] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:40 [I 2022-10-09 14:31:19] ...trap/engines/engine.py.326: process: 0.52390 | load: 0.00018 [I 2022-10-09 14:31:24] ...trap/engines/engine.py.321: eval: epoch 43 | batch 80/510 [I 2022-10-09 14:31:24] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:33 [I 2022-10-09 14:31:24] ...trap/engines/engine.py.326: process: 0.45949 | load: 0.00019 [I 2022-10-09 14:31:28] ...trap/engines/engine.py.321: eval: epoch 43 | batch 90/510 [I 2022-10-09 14:31:28] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:00 [I 2022-10-09 14:31:28] ...trap/engines/engine.py.326: process: 0.43598 | load: 0.00018 [I 2022-10-09 14:31:32] ...trap/engines/engine.py.321: eval: epoch 43 | batch 100/510 [I 2022-10-09 14:31:32] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:47 [I 2022-10-09 14:31:32] ...trap/engines/engine.py.326: process: 0.31151 | load: 0.00019 [I 2022-10-09 14:31:35] ...trap/engines/engine.py.321: eval: epoch 43 | batch 110/510 [I 2022-10-09 14:31:35] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:09 [I 2022-10-09 14:31:35] ...trap/engines/engine.py.326: process: 0.30403 | load: 0.00019 [I 2022-10-09 14:31:38] ...trap/engines/engine.py.321: eval: epoch 43 | batch 120/510 [I 2022-10-09 14:31:38] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:03 [I 2022-10-09 14:31:38] ...trap/engines/engine.py.326: process: 0.31776 | load: 0.00019 [I 2022-10-09 14:31:42] ...trap/engines/engine.py.321: eval: epoch 43 | batch 130/510 [I 2022-10-09 14:31:42] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:00 [I 2022-10-09 14:31:42] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00019 [I 2022-10-09 14:31:45] ...trap/engines/engine.py.321: eval: epoch 43 | batch 140/510 [I 2022-10-09 14:31:45] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:56 [I 2022-10-09 14:31:45] ...trap/engines/engine.py.326: process: 0.31008 | load: 0.00018 [I 2022-10-09 14:31:48] ...trap/engines/engine.py.321: eval: epoch 43 | batch 150/510 [I 2022-10-09 14:31:48] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:52 [I 2022-10-09 14:31:48] ...trap/engines/engine.py.326: process: 0.31109 | load: 0.00019 [I 2022-10-09 14:31:51] ...trap/engines/engine.py.321: eval: epoch 43 | batch 160/510 [I 2022-10-09 14:31:51] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:51 [I 2022-10-09 14:31:51] ...trap/engines/engine.py.326: process: 0.31803 | load: 0.00020 [I 2022-10-09 14:31:54] ...trap/engines/engine.py.321: eval: epoch 43 | batch 170/510 [I 2022-10-09 14:31:54] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:48 [I 2022-10-09 14:31:54] ...trap/engines/engine.py.326: process: 0.32004 | load: 0.00020 [I 2022-10-09 14:31:57] ...trap/engines/engine.py.321: eval: epoch 43 | batch 180/510 [I 2022-10-09 14:31:57] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:45 [I 2022-10-09 14:31:57] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00020 [I 2022-10-09 14:32:01] ...trap/engines/engine.py.321: eval: epoch 43 | batch 190/510 [I 2022-10-09 14:32:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:41 [I 2022-10-09 14:32:01] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00020 [I 2022-10-09 14:32:04] ...trap/engines/engine.py.321: eval: epoch 43 | batch 200/510 [I 2022-10-09 14:32:04] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:38 [I 2022-10-09 14:32:04] ...trap/engines/engine.py.326: process: 0.31718 | load: 0.00019 [I 2022-10-09 14:32:07] ...trap/engines/engine.py.321: eval: epoch 43 | batch 210/510 [I 2022-10-09 14:32:07] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-09 14:32:07] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00020 [I 2022-10-09 14:32:10] ...trap/engines/engine.py.321: eval: epoch 43 | batch 220/510 [I 2022-10-09 14:32:10] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:32 [I 2022-10-09 14:32:10] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00020 [I 2022-10-09 14:32:13] ...trap/engines/engine.py.321: eval: epoch 43 | batch 230/510 [I 2022-10-09 14:32:13] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:29 [I 2022-10-09 14:32:13] ...trap/engines/engine.py.326: process: 0.32075 | load: 0.00020 [I 2022-10-09 14:32:17] ...trap/engines/engine.py.321: eval: epoch 43 | batch 240/510 [I 2022-10-09 14:32:17] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:25 [I 2022-10-09 14:32:17] ...trap/engines/engine.py.326: process: 0.31100 | load: 0.00020 [I 2022-10-09 14:32:20] ...trap/engines/engine.py.321: eval: epoch 43 | batch 250/510 [I 2022-10-09 14:32:20] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:22 [I 2022-10-09 14:32:20] ...trap/engines/engine.py.326: process: 0.31775 | load: 0.00020 [I 2022-10-09 14:32:23] ...trap/engines/engine.py.321: eval: epoch 43 | batch 260/510 [I 2022-10-09 14:32:23] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:19 [I 2022-10-09 14:32:23] ...trap/engines/engine.py.326: process: 0.31084 | load: 0.00019 [I 2022-10-09 14:32:26] ...trap/engines/engine.py.321: eval: epoch 43 | batch 270/510 [I 2022-10-09 14:32:26] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:15 [I 2022-10-09 14:32:26] ...trap/engines/engine.py.326: process: 0.31148 | load: 0.00020 [I 2022-10-09 14:32:29] ...trap/engines/engine.py.321: eval: epoch 43 | batch 280/510 [I 2022-10-09 14:32:29] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-09 14:32:29] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00019 [I 2022-10-09 14:32:32] ...trap/engines/engine.py.321: eval: epoch 43 | batch 290/510 [I 2022-10-09 14:32:32] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:09 [I 2022-10-09 14:32:32] ...trap/engines/engine.py.326: process: 0.31856 | load: 0.00020 [I 2022-10-09 14:32:36] ...trap/engines/engine.py.321: eval: epoch 43 | batch 300/510 [I 2022-10-09 14:32:36] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-09 14:32:36] ...trap/engines/engine.py.326: process: 0.31971 | load: 0.00019 [I 2022-10-09 14:32:39] ...trap/engines/engine.py.321: eval: epoch 43 | batch 310/510 [I 2022-10-09 14:32:39] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:03 [I 2022-10-09 14:32:39] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00020 [I 2022-10-09 14:32:42] ...trap/engines/engine.py.321: eval: epoch 43 | batch 320/510 [I 2022-10-09 14:32:42] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:00 [I 2022-10-09 14:32:42] ...trap/engines/engine.py.326: process: 0.31749 | load: 0.00019 [I 2022-10-09 14:32:45] ...trap/engines/engine.py.321: eval: epoch 43 | batch 330/510 [I 2022-10-09 14:32:45] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-09 14:32:45] ...trap/engines/engine.py.326: process: 0.31909 | load: 0.00021 [I 2022-10-09 14:32:48] ...trap/engines/engine.py.321: eval: epoch 43 | batch 340/510 [I 2022-10-09 14:32:48] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:53 [I 2022-10-09 14:32:48] ...trap/engines/engine.py.326: process: 0.31660 | load: 0.00020 [I 2022-10-09 14:32:52] ...trap/engines/engine.py.321: eval: epoch 43 | batch 350/510 [I 2022-10-09 14:32:52] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:50 [I 2022-10-09 14:32:52] ...trap/engines/engine.py.326: process: 0.31404 | load: 0.00020 [I 2022-10-09 14:32:55] ...trap/engines/engine.py.321: eval: epoch 43 | batch 360/510 [I 2022-10-09 14:32:55] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:47 [I 2022-10-09 14:32:55] ...trap/engines/engine.py.326: process: 0.31197 | load: 0.00021 [I 2022-10-09 14:32:58] ...trap/engines/engine.py.321: eval: epoch 43 | batch 370/510 [I 2022-10-09 14:32:58] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-09 14:32:58] ...trap/engines/engine.py.326: process: 0.31206 | load: 0.00020 [I 2022-10-09 14:33:01] ...trap/engines/engine.py.321: eval: epoch 43 | batch 380/510 [I 2022-10-09 14:33:01] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-09 14:33:01] ...trap/engines/engine.py.326: process: 0.31486 | load: 0.00020 [I 2022-10-09 14:33:04] ...trap/engines/engine.py.321: eval: epoch 43 | batch 390/510 [I 2022-10-09 14:33:04] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-09 14:33:04] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00021 [I 2022-10-09 14:33:07] ...trap/engines/engine.py.321: eval: epoch 43 | batch 400/510 [I 2022-10-09 14:33:07] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-09 14:33:07] ...trap/engines/engine.py.326: process: 0.31909 | load: 0.00019 [I 2022-10-09 14:33:11] ...trap/engines/engine.py.321: eval: epoch 43 | batch 410/510 [I 2022-10-09 14:33:11] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:31 [I 2022-10-09 14:33:11] ...trap/engines/engine.py.326: process: 0.32027 | load: 0.00020 [I 2022-10-09 14:33:14] ...trap/engines/engine.py.321: eval: epoch 43 | batch 420/510 [I 2022-10-09 14:33:14] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-09 14:33:14] ...trap/engines/engine.py.326: process: 0.31262 | load: 0.00019 [I 2022-10-09 14:33:17] ...trap/engines/engine.py.321: eval: epoch 43 | batch 430/510 [I 2022-10-09 14:33:17] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-09 14:33:17] ...trap/engines/engine.py.326: process: 0.31931 | load: 0.00020 [I 2022-10-09 14:33:20] ...trap/engines/engine.py.321: eval: epoch 43 | batch 440/510 [I 2022-10-09 14:33:20] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-09 14:33:20] ...trap/engines/engine.py.326: process: 0.32418 | load: 0.00020 [I 2022-10-09 14:33:23] ...trap/engines/engine.py.321: eval: epoch 43 | batch 450/510 [I 2022-10-09 14:33:23] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 14:33:23] ...trap/engines/engine.py.326: process: 0.32800 | load: 0.00020 [I 2022-10-09 14:33:27] ...trap/engines/engine.py.321: eval: epoch 43 | batch 460/510 [I 2022-10-09 14:33:27] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-09 14:33:27] ...trap/engines/engine.py.326: process: 0.32268 | load: 0.00020 [I 2022-10-09 14:33:30] ...trap/engines/engine.py.321: eval: epoch 43 | batch 470/510 [I 2022-10-09 14:33:30] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-09 14:33:30] ...trap/engines/engine.py.326: process: 0.31699 | load: 0.00019 [I 2022-10-09 14:33:33] ...trap/engines/engine.py.321: eval: epoch 43 | batch 480/510 [I 2022-10-09 14:33:33] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-09 14:33:33] ...trap/engines/engine.py.326: process: 0.31358 | load: 0.00019 [I 2022-10-09 14:33:36] ...trap/engines/engine.py.321: eval: epoch 43 | batch 490/510 [I 2022-10-09 14:33:36] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 14:33:36] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00020 [I 2022-10-09 14:33:39] ...trap/engines/engine.py.321: eval: epoch 43 | batch 500/510 [I 2022-10-09 14:33:39] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 14:33:39] ...trap/engines/engine.py.326: process: 0.31969 | load: 0.00022 [I 2022-10-09 14:33:42] ...trap/engines/engine.py.321: eval: epoch 43 | batch 510/510 [I 2022-10-09 14:33:42] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 14:33:42] ...trap/engines/engine.py.326: process: 0.31610 | load: 0.00016 [S 2022-10-09 14:33:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 43 [I 2022-10-09 14:33:43] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.712100 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905500 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940400 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.705900 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.899800 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936700 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017050 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007256 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008417 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017172 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010400 [S 2022-10-09 14:34:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009477 [I 2022-10-09 14:34:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 14:34:19] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 14:34:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 14:34:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 14:34:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 14:34:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 14:34:23] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 14:34:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 14:34:29] ...trap/engines/engine.py.163: Training model on trainset for epoch 44 [I 2022-10-09 14:34:29] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 14:34:32] ...trap/engines/engine.py.224: train: epoch 44 | batch 0/2304 [I 2022-10-09 14:34:32] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:53 [I 2022-10-09 14:34:32] ...trap/engines/engine.py.229: process: 2.78365 | load: 1.87268 [I 2022-10-09 14:34:32] ...trap/engines/engine.py.230: loss: 0.16066 [I 2022-10-09 14:34:32] ...optimizers/trijoint.py.151: total_norm: 25.622561 [I 2022-10-09 14:34:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 10/2304 [I 2022-10-09 14:34:41] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:14 [I 2022-10-09 14:34:41] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00504 [I 2022-10-09 14:34:41] ...trap/engines/engine.py.230: loss: 0.11753 [I 2022-10-09 14:34:41] ...optimizers/trijoint.py.151: total_norm: 10.735109 [I 2022-10-09 14:34:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 20/2304 [I 2022-10-09 14:34:49] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:58 [I 2022-10-09 14:34:49] ...trap/engines/engine.py.229: process: 0.83935 | load: 0.00372 [I 2022-10-09 14:34:49] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-09 14:34:49] ...optimizers/trijoint.py.151: total_norm: 10.111835 [I 2022-10-09 14:34:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 30/2304 [I 2022-10-09 14:34:57] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:08 [I 2022-10-09 14:34:57] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00370 [I 2022-10-09 14:34:57] ...trap/engines/engine.py.230: loss: 0.14402 [I 2022-10-09 14:34:57] ...optimizers/trijoint.py.151: total_norm: 18.087055 [I 2022-10-09 14:35:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 40/2304 [I 2022-10-09 14:35:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:58 [I 2022-10-09 14:35:06] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00416 [I 2022-10-09 14:35:06] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-09 14:35:06] ...optimizers/trijoint.py.151: total_norm: 20.043411 [I 2022-10-09 14:35:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 50/2304 [I 2022-10-09 14:35:14] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:38 [I 2022-10-09 14:35:14] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00338 [I 2022-10-09 14:35:14] ...trap/engines/engine.py.230: loss: 0.13505 [I 2022-10-09 14:35:14] ...optimizers/trijoint.py.151: total_norm: 40.652546 [I 2022-10-09 14:35:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 60/2304 [I 2022-10-09 14:35:23] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:39 [I 2022-10-09 14:35:23] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00340 [I 2022-10-09 14:35:23] ...trap/engines/engine.py.230: loss: 0.10704 [I 2022-10-09 14:35:23] ...optimizers/trijoint.py.151: total_norm: 9.930961 [I 2022-10-09 14:35:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 70/2304 [I 2022-10-09 14:35:31] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:36 [I 2022-10-09 14:35:31] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00383 [I 2022-10-09 14:35:31] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-09 14:35:31] ...optimizers/trijoint.py.151: total_norm: 10.143216 [I 2022-10-09 14:35:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 80/2304 [I 2022-10-09 14:35:40] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:34 [I 2022-10-09 14:35:40] ...trap/engines/engine.py.229: process: 0.85669 | load: 0.00319 [I 2022-10-09 14:35:40] ...trap/engines/engine.py.230: loss: 0.16887 [I 2022-10-09 14:35:40] ...optimizers/trijoint.py.151: total_norm: 16.476154 [I 2022-10-09 14:35:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 90/2304 [I 2022-10-09 14:35:48] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:17 [I 2022-10-09 14:35:48] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00313 [I 2022-10-09 14:35:48] ...trap/engines/engine.py.230: loss: 0.15810 [I 2022-10-09 14:35:48] ...optimizers/trijoint.py.151: total_norm: 21.491419 [I 2022-10-09 14:35:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 100/2304 [I 2022-10-09 14:35:57] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:08 [I 2022-10-09 14:35:57] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00324 [I 2022-10-09 14:35:57] ...trap/engines/engine.py.230: loss: 0.11279 [I 2022-10-09 14:35:57] ...optimizers/trijoint.py.151: total_norm: 11.998907 [I 2022-10-09 14:36:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 110/2304 [I 2022-10-09 14:36:05] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:58 [I 2022-10-09 14:36:05] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00337 [I 2022-10-09 14:36:05] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-09 14:36:05] ...optimizers/trijoint.py.151: total_norm: 20.632322 [I 2022-10-09 14:36:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 120/2304 [I 2022-10-09 14:36:14] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:48 [I 2022-10-09 14:36:14] ...trap/engines/engine.py.229: process: 0.84040 | load: 0.00330 [I 2022-10-09 14:36:14] ...trap/engines/engine.py.230: loss: 0.09975 [I 2022-10-09 14:36:14] ...optimizers/trijoint.py.151: total_norm: 12.627909 [I 2022-10-09 14:36:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 130/2304 [I 2022-10-09 14:36:22] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:41 [I 2022-10-09 14:36:22] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00330 [I 2022-10-09 14:36:22] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-09 14:36:22] ...optimizers/trijoint.py.151: total_norm: 16.769045 [I 2022-10-09 14:36:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 140/2304 [I 2022-10-09 14:36:31] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:36 [I 2022-10-09 14:36:31] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00296 [I 2022-10-09 14:36:31] ...trap/engines/engine.py.230: loss: 0.18439 [I 2022-10-09 14:36:31] ...optimizers/trijoint.py.151: total_norm: 55.956436 [I 2022-10-09 14:36:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 150/2304 [I 2022-10-09 14:36:39] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:20 [I 2022-10-09 14:36:39] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00315 [I 2022-10-09 14:36:39] ...trap/engines/engine.py.230: loss: 0.14106 [I 2022-10-09 14:36:39] ...optimizers/trijoint.py.151: total_norm: 29.183680 [I 2022-10-09 14:36:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 160/2304 [I 2022-10-09 14:36:48] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:14 [I 2022-10-09 14:36:48] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00305 [I 2022-10-09 14:36:48] ...trap/engines/engine.py.230: loss: 0.14262 [I 2022-10-09 14:36:48] ...optimizers/trijoint.py.151: total_norm: 21.884230 [I 2022-10-09 14:36:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 170/2304 [I 2022-10-09 14:36:56] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:13 [I 2022-10-09 14:36:56] ...trap/engines/engine.py.229: process: 0.85284 | load: 0.00321 [I 2022-10-09 14:36:56] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-09 14:36:56] ...optimizers/trijoint.py.151: total_norm: 18.837145 [I 2022-10-09 14:37:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 180/2304 [I 2022-10-09 14:37:05] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:57 [I 2022-10-09 14:37:05] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00300 [I 2022-10-09 14:37:05] ...trap/engines/engine.py.230: loss: 0.16277 [I 2022-10-09 14:37:05] ...optimizers/trijoint.py.151: total_norm: 21.251205 [I 2022-10-09 14:37:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 190/2304 [I 2022-10-09 14:37:13] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:43 [I 2022-10-09 14:37:13] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00323 [I 2022-10-09 14:37:13] ...trap/engines/engine.py.230: loss: 0.16101 [I 2022-10-09 14:37:13] ...optimizers/trijoint.py.151: total_norm: 14.223453 [I 2022-10-09 14:37:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 200/2304 [I 2022-10-09 14:37:22] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:41 [I 2022-10-09 14:37:22] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00321 [I 2022-10-09 14:37:22] ...trap/engines/engine.py.230: loss: 0.17689 [I 2022-10-09 14:37:22] ...optimizers/trijoint.py.151: total_norm: 33.291122 [I 2022-10-09 14:37:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 210/2304 [I 2022-10-09 14:37:30] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:29 [I 2022-10-09 14:37:30] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00327 [I 2022-10-09 14:37:30] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-09 14:37:30] ...optimizers/trijoint.py.151: total_norm: 16.232908 [I 2022-10-09 14:37:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 220/2304 [I 2022-10-09 14:37:39] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:20 [I 2022-10-09 14:37:39] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00296 [I 2022-10-09 14:37:39] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-09 14:37:39] ...optimizers/trijoint.py.151: total_norm: 28.816135 [I 2022-10-09 14:37:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 230/2304 [I 2022-10-09 14:37:47] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:06 [I 2022-10-09 14:37:47] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00309 [I 2022-10-09 14:37:47] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-09 14:37:47] ...optimizers/trijoint.py.151: total_norm: 10.358013 [I 2022-10-09 14:37:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 240/2304 [I 2022-10-09 14:37:55] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:01 [I 2022-10-09 14:37:55] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00307 [I 2022-10-09 14:37:55] ...trap/engines/engine.py.230: loss: 0.13924 [I 2022-10-09 14:37:55] ...optimizers/trijoint.py.151: total_norm: 11.229288 [I 2022-10-09 14:38:04] ...trap/engines/engine.py.224: train: epoch 44 | batch 250/2304 [I 2022-10-09 14:38:04] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:03 [I 2022-10-09 14:38:04] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00296 [I 2022-10-09 14:38:04] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-09 14:38:04] ...optimizers/trijoint.py.151: total_norm: 12.482734 [I 2022-10-09 14:38:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 260/2304 [I 2022-10-09 14:38:12] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:51 [I 2022-10-09 14:38:12] ...trap/engines/engine.py.229: process: 0.85839 | load: 0.00312 [I 2022-10-09 14:38:12] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-09 14:38:12] ...optimizers/trijoint.py.151: total_norm: 11.811040 [I 2022-10-09 14:38:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 270/2304 [I 2022-10-09 14:38:21] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:32 [I 2022-10-09 14:38:21] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00300 [I 2022-10-09 14:38:21] ...trap/engines/engine.py.230: loss: 0.15163 [I 2022-10-09 14:38:21] ...optimizers/trijoint.py.151: total_norm: 17.105726 [I 2022-10-09 14:38:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 280/2304 [I 2022-10-09 14:38:29] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:21 [I 2022-10-09 14:38:29] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00305 [I 2022-10-09 14:38:29] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-09 14:38:29] ...optimizers/trijoint.py.151: total_norm: 30.638754 [I 2022-10-09 14:38:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 290/2304 [I 2022-10-09 14:38:38] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:15 [I 2022-10-09 14:38:38] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00301 [I 2022-10-09 14:38:38] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-09 14:38:38] ...optimizers/trijoint.py.151: total_norm: 20.444544 [I 2022-10-09 14:38:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 300/2304 [I 2022-10-09 14:38:46] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:27:58 [I 2022-10-09 14:38:46] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00293 [I 2022-10-09 14:38:46] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-09 14:38:46] ...optimizers/trijoint.py.151: total_norm: 14.075187 [I 2022-10-09 14:38:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 310/2304 [I 2022-10-09 14:38:55] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:02 [I 2022-10-09 14:38:55] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00295 [I 2022-10-09 14:38:55] ...trap/engines/engine.py.230: loss: 0.23458 [I 2022-10-09 14:38:55] ...optimizers/trijoint.py.151: total_norm: 18.495314 [I 2022-10-09 14:39:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 320/2304 [I 2022-10-09 14:39:03] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:56 [I 2022-10-09 14:39:03] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00294 [I 2022-10-09 14:39:03] ...trap/engines/engine.py.230: loss: 0.13886 [I 2022-10-09 14:39:03] ...optimizers/trijoint.py.151: total_norm: 13.515325 [I 2022-10-09 14:39:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 330/2304 [I 2022-10-09 14:39:11] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:41 [I 2022-10-09 14:39:11] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00309 [I 2022-10-09 14:39:11] ...trap/engines/engine.py.230: loss: 0.11377 [I 2022-10-09 14:39:11] ...optimizers/trijoint.py.151: total_norm: 12.965800 [I 2022-10-09 14:39:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 340/2304 [I 2022-10-09 14:39:20] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:44 [I 2022-10-09 14:39:20] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00296 [I 2022-10-09 14:39:20] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-09 14:39:20] ...optimizers/trijoint.py.151: total_norm: 13.088322 [I 2022-10-09 14:39:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 350/2304 [I 2022-10-09 14:39:28] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:39 [I 2022-10-09 14:39:28] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00299 [I 2022-10-09 14:39:28] ...trap/engines/engine.py.230: loss: 0.09839 [I 2022-10-09 14:39:28] ...optimizers/trijoint.py.151: total_norm: 15.372224 [I 2022-10-09 14:39:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 360/2304 [I 2022-10-09 14:39:37] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:13 [I 2022-10-09 14:39:37] ...trap/engines/engine.py.229: process: 0.83380 | load: 0.00304 [I 2022-10-09 14:39:37] ...trap/engines/engine.py.230: loss: 0.15364 [I 2022-10-09 14:39:37] ...optimizers/trijoint.py.151: total_norm: 16.305759 [I 2022-10-09 14:39:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 370/2304 [I 2022-10-09 14:39:45] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:27:25 [I 2022-10-09 14:39:45] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00298 [I 2022-10-09 14:39:45] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-09 14:39:45] ...optimizers/trijoint.py.151: total_norm: 12.851034 [I 2022-10-09 14:39:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 380/2304 [I 2022-10-09 14:39:54] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:09 [I 2022-10-09 14:39:54] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00298 [I 2022-10-09 14:39:54] ...trap/engines/engine.py.230: loss: 0.20048 [I 2022-10-09 14:39:54] ...optimizers/trijoint.py.151: total_norm: 23.548782 [I 2022-10-09 14:40:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 390/2304 [I 2022-10-09 14:40:02] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:08 [I 2022-10-09 14:40:02] ...trap/engines/engine.py.229: process: 0.85641 | load: 0.00298 [I 2022-10-09 14:40:02] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-09 14:40:02] ...optimizers/trijoint.py.151: total_norm: 8.058020 [I 2022-10-09 14:40:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 400/2304 [I 2022-10-09 14:40:11] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:27:03 [I 2022-10-09 14:40:11] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00299 [I 2022-10-09 14:40:11] ...trap/engines/engine.py.230: loss: 0.14850 [I 2022-10-09 14:40:11] ...optimizers/trijoint.py.151: total_norm: 19.550800 [I 2022-10-09 14:40:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 410/2304 [I 2022-10-09 14:40:19] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:26:47 [I 2022-10-09 14:40:19] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00298 [I 2022-10-09 14:40:19] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-09 14:40:19] ...optimizers/trijoint.py.151: total_norm: 11.555593 [I 2022-10-09 14:40:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 420/2304 [I 2022-10-09 14:40:28] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:39 [I 2022-10-09 14:40:28] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00297 [I 2022-10-09 14:40:28] ...trap/engines/engine.py.230: loss: 0.14189 [I 2022-10-09 14:40:28] ...optimizers/trijoint.py.151: total_norm: 11.851549 [I 2022-10-09 14:40:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 430/2304 [I 2022-10-09 14:40:36] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:23 [I 2022-10-09 14:40:36] ...trap/engines/engine.py.229: process: 0.83856 | load: 0.00303 [I 2022-10-09 14:40:36] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-09 14:40:36] ...optimizers/trijoint.py.151: total_norm: 29.591105 [I 2022-10-09 14:40:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 440/2304 [I 2022-10-09 14:40:45] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:07 [I 2022-10-09 14:40:45] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00297 [I 2022-10-09 14:40:45] ...trap/engines/engine.py.230: loss: 0.16080 [I 2022-10-09 14:40:45] ...optimizers/trijoint.py.151: total_norm: 20.180292 [I 2022-10-09 14:40:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 450/2304 [I 2022-10-09 14:40:53] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:10 [I 2022-10-09 14:40:53] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00298 [I 2022-10-09 14:40:53] ...trap/engines/engine.py.230: loss: 0.15449 [I 2022-10-09 14:40:53] ...optimizers/trijoint.py.151: total_norm: 7.773717 [I 2022-10-09 14:41:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 460/2304 [I 2022-10-09 14:41:02] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:04 [I 2022-10-09 14:41:02] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00299 [I 2022-10-09 14:41:02] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-09 14:41:02] ...optimizers/trijoint.py.151: total_norm: 8.850749 [I 2022-10-09 14:41:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 470/2304 [I 2022-10-09 14:41:10] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:25:57 [I 2022-10-09 14:41:10] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00302 [I 2022-10-09 14:41:10] ...trap/engines/engine.py.230: loss: 0.15475 [I 2022-10-09 14:41:10] ...optimizers/trijoint.py.151: total_norm: 23.570387 [I 2022-10-09 14:41:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 480/2304 [I 2022-10-09 14:41:19] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:25:48 [I 2022-10-09 14:41:19] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00298 [I 2022-10-09 14:41:19] ...trap/engines/engine.py.230: loss: 0.13525 [I 2022-10-09 14:41:19] ...optimizers/trijoint.py.151: total_norm: 12.707829 [I 2022-10-09 14:41:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 490/2304 [I 2022-10-09 14:41:27] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:25:37 [I 2022-10-09 14:41:27] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00302 [I 2022-10-09 14:41:27] ...trap/engines/engine.py.230: loss: 0.10995 [I 2022-10-09 14:41:27] ...optimizers/trijoint.py.151: total_norm: 14.539501 [I 2022-10-09 14:41:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 500/2304 [I 2022-10-09 14:41:36] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:24 [I 2022-10-09 14:41:36] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00302 [I 2022-10-09 14:41:36] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-09 14:41:36] ...optimizers/trijoint.py.151: total_norm: 30.432274 [I 2022-10-09 14:41:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 510/2304 [I 2022-10-09 14:41:44] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:20 [I 2022-10-09 14:41:44] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00313 [I 2022-10-09 14:41:44] ...trap/engines/engine.py.230: loss: 0.19155 [I 2022-10-09 14:41:44] ...optimizers/trijoint.py.151: total_norm: 22.200262 [I 2022-10-09 14:41:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 520/2304 [I 2022-10-09 14:41:53] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:16 [I 2022-10-09 14:41:53] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00296 [I 2022-10-09 14:41:53] ...trap/engines/engine.py.230: loss: 0.11755 [I 2022-10-09 14:41:53] ...optimizers/trijoint.py.151: total_norm: 19.954071 [I 2022-10-09 14:42:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 530/2304 [I 2022-10-09 14:42:01] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:00 [I 2022-10-09 14:42:01] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00311 [I 2022-10-09 14:42:01] ...trap/engines/engine.py.230: loss: 0.15795 [I 2022-10-09 14:42:01] ...optimizers/trijoint.py.151: total_norm: 14.360938 [I 2022-10-09 14:42:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 540/2304 [I 2022-10-09 14:42:10] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:04 [I 2022-10-09 14:42:10] ...trap/engines/engine.py.229: process: 0.85853 | load: 0.00301 [I 2022-10-09 14:42:10] ...trap/engines/engine.py.230: loss: 0.29516 [I 2022-10-09 14:42:10] ...optimizers/trijoint.py.151: total_norm: 42.964252 [I 2022-10-09 14:42:18] ...trap/engines/engine.py.224: train: epoch 44 | batch 550/2304 [I 2022-10-09 14:42:18] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:24:39 [I 2022-10-09 14:42:18] ...trap/engines/engine.py.229: process: 0.83500 | load: 0.00305 [I 2022-10-09 14:42:18] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-09 14:42:18] ...optimizers/trijoint.py.151: total_norm: 10.372503 [I 2022-10-09 14:42:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 560/2304 [I 2022-10-09 14:42:27] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:24:41 [I 2022-10-09 14:42:27] ...trap/engines/engine.py.229: process: 0.85394 | load: 0.00305 [I 2022-10-09 14:42:27] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-09 14:42:27] ...optimizers/trijoint.py.151: total_norm: 12.274524 [I 2022-10-09 14:42:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 570/2304 [I 2022-10-09 14:42:35] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:29 [I 2022-10-09 14:42:35] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00303 [I 2022-10-09 14:42:35] ...trap/engines/engine.py.230: loss: 0.13079 [I 2022-10-09 14:42:35] ...optimizers/trijoint.py.151: total_norm: 17.049946 [I 2022-10-09 14:42:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 580/2304 [I 2022-10-09 14:42:44] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:25 [I 2022-10-09 14:42:44] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00304 [I 2022-10-09 14:42:44] ...trap/engines/engine.py.230: loss: 0.13150 [I 2022-10-09 14:42:44] ...optimizers/trijoint.py.151: total_norm: 10.712019 [I 2022-10-09 14:42:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 590/2304 [I 2022-10-09 14:42:52] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:17 [I 2022-10-09 14:42:52] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00315 [I 2022-10-09 14:42:52] ...trap/engines/engine.py.230: loss: 0.13279 [I 2022-10-09 14:42:52] ...optimizers/trijoint.py.151: total_norm: 8.409334 [I 2022-10-09 14:43:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 600/2304 [I 2022-10-09 14:43:01] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:01 [I 2022-10-09 14:43:01] ...trap/engines/engine.py.229: process: 0.85015 | load: 0.00300 [I 2022-10-09 14:43:01] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-09 14:43:01] ...optimizers/trijoint.py.151: total_norm: 22.120884 [I 2022-10-09 14:43:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 610/2304 [I 2022-10-09 14:43:09] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:23:54 [I 2022-10-09 14:43:09] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00296 [I 2022-10-09 14:43:09] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-09 14:43:09] ...optimizers/trijoint.py.151: total_norm: 8.629211 [I 2022-10-09 14:43:18] ...trap/engines/engine.py.224: train: epoch 44 | batch 620/2304 [I 2022-10-09 14:43:18] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:23:38 [I 2022-10-09 14:43:18] ...trap/engines/engine.py.229: process: 0.82873 | load: 0.00310 [I 2022-10-09 14:43:18] ...trap/engines/engine.py.230: loss: 0.16310 [I 2022-10-09 14:43:18] ...optimizers/trijoint.py.151: total_norm: 22.805380 [I 2022-10-09 14:43:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 630/2304 [I 2022-10-09 14:43:26] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:41 [I 2022-10-09 14:43:26] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00299 [I 2022-10-09 14:43:26] ...trap/engines/engine.py.230: loss: 0.25606 [I 2022-10-09 14:43:26] ...optimizers/trijoint.py.151: total_norm: 22.917103 [I 2022-10-09 14:43:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 640/2304 [I 2022-10-09 14:43:35] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:16 [I 2022-10-09 14:43:35] ...trap/engines/engine.py.229: process: 0.83141 | load: 0.00296 [I 2022-10-09 14:43:35] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-09 14:43:35] ...optimizers/trijoint.py.151: total_norm: 17.955486 [I 2022-10-09 14:43:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 650/2304 [I 2022-10-09 14:43:43] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:17 [I 2022-10-09 14:43:43] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00296 [I 2022-10-09 14:43:43] ...trap/engines/engine.py.230: loss: 0.17627 [I 2022-10-09 14:43:43] ...optimizers/trijoint.py.151: total_norm: 19.793703 [I 2022-10-09 14:43:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 660/2304 [I 2022-10-09 14:43:52] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:01 [I 2022-10-09 14:43:52] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00308 [I 2022-10-09 14:43:52] ...trap/engines/engine.py.230: loss: 0.13583 [I 2022-10-09 14:43:52] ...optimizers/trijoint.py.151: total_norm: 19.194107 [I 2022-10-09 14:44:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 670/2304 [I 2022-10-09 14:44:00] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:01 [I 2022-10-09 14:44:00] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00298 [I 2022-10-09 14:44:00] ...trap/engines/engine.py.230: loss: 0.14366 [I 2022-10-09 14:44:00] ...optimizers/trijoint.py.151: total_norm: 6.803374 [I 2022-10-09 14:44:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 680/2304 [I 2022-10-09 14:44:08] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:22:46 [I 2022-10-09 14:44:08] ...trap/engines/engine.py.229: process: 0.82860 | load: 0.00295 [I 2022-10-09 14:44:08] ...trap/engines/engine.py.230: loss: 0.18099 [I 2022-10-09 14:44:08] ...optimizers/trijoint.py.151: total_norm: 13.680272 [I 2022-10-09 14:44:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 690/2304 [I 2022-10-09 14:44:17] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:22:39 [I 2022-10-09 14:44:17] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00298 [I 2022-10-09 14:44:17] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-09 14:44:17] ...optimizers/trijoint.py.151: total_norm: 18.008966 [I 2022-10-09 14:44:25] ...trap/engines/engine.py.224: train: epoch 44 | batch 700/2304 [I 2022-10-09 14:44:25] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:22:43 [I 2022-10-09 14:44:25] ...trap/engines/engine.py.229: process: 0.85151 | load: 0.00315 [I 2022-10-09 14:44:25] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-09 14:44:25] ...optimizers/trijoint.py.151: total_norm: 16.633228 [I 2022-10-09 14:44:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 710/2304 [I 2022-10-09 14:44:52] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:38:36 [I 2022-10-09 14:44:52] ...trap/engines/engine.py.229: process: 0.83320 | load: 0.00298 [I 2022-10-09 14:44:52] ...trap/engines/engine.py.230: loss: 0.17521 [I 2022-10-09 14:44:52] ...optimizers/trijoint.py.151: total_norm: 58.856533 [I 2022-10-09 14:45:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 720/2304 [I 2022-10-09 14:45:01] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:24:01 [I 2022-10-09 14:45:01] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00295 [I 2022-10-09 14:45:01] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-09 14:45:01] ...optimizers/trijoint.py.151: total_norm: 35.819359 [I 2022-10-09 14:45:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 730/2304 [I 2022-10-09 14:45:09] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:22:13 [I 2022-10-09 14:45:09] ...trap/engines/engine.py.229: process: 0.82412 | load: 0.00298 [I 2022-10-09 14:45:09] ...trap/engines/engine.py.230: loss: 0.18435 [I 2022-10-09 14:45:09] ...optimizers/trijoint.py.151: total_norm: 15.846377 [I 2022-10-09 14:45:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 740/2304 [I 2022-10-09 14:45:17] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:22:01 [I 2022-10-09 14:45:17] ...trap/engines/engine.py.229: process: 0.83255 | load: 0.00298 [I 2022-10-09 14:45:17] ...trap/engines/engine.py.230: loss: 0.16060 [I 2022-10-09 14:45:17] ...optimizers/trijoint.py.151: total_norm: 16.051167 [I 2022-10-09 14:45:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 750/2304 [I 2022-10-09 14:45:26] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:22:02 [I 2022-10-09 14:45:26] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00296 [I 2022-10-09 14:45:26] ...trap/engines/engine.py.230: loss: 0.18802 [I 2022-10-09 14:45:26] ...optimizers/trijoint.py.151: total_norm: 15.027269 [I 2022-10-09 14:45:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 760/2304 [I 2022-10-09 14:45:34] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:50 [I 2022-10-09 14:45:34] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00292 [I 2022-10-09 14:45:34] ...trap/engines/engine.py.230: loss: 0.20513 [I 2022-10-09 14:45:34] ...optimizers/trijoint.py.151: total_norm: 14.879788 [I 2022-10-09 14:45:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 770/2304 [I 2022-10-09 14:45:43] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:40 [I 2022-10-09 14:45:43] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00299 [I 2022-10-09 14:45:43] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-09 14:45:43] ...optimizers/trijoint.py.151: total_norm: 12.532517 [I 2022-10-09 14:45:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 780/2304 [I 2022-10-09 14:45:51] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:28 [I 2022-10-09 14:45:51] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00306 [I 2022-10-09 14:45:51] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-09 14:45:51] ...optimizers/trijoint.py.151: total_norm: 13.822704 [I 2022-10-09 14:46:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 790/2304 [I 2022-10-09 14:46:00] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:14 [I 2022-10-09 14:46:00] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00295 [I 2022-10-09 14:46:00] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-09 14:46:00] ...optimizers/trijoint.py.151: total_norm: 12.066042 [I 2022-10-09 14:46:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 800/2304 [I 2022-10-09 14:46:08] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:11 [I 2022-10-09 14:46:08] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00294 [I 2022-10-09 14:46:08] ...trap/engines/engine.py.230: loss: 0.17984 [I 2022-10-09 14:46:08] ...optimizers/trijoint.py.151: total_norm: 19.923395 [I 2022-10-09 14:46:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 810/2304 [I 2022-10-09 14:46:17] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:55 [I 2022-10-09 14:46:17] ...trap/engines/engine.py.229: process: 0.83900 | load: 0.00294 [I 2022-10-09 14:46:17] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-09 14:46:17] ...optimizers/trijoint.py.151: total_norm: 21.944452 [I 2022-10-09 14:46:25] ...trap/engines/engine.py.224: train: epoch 44 | batch 820/2304 [I 2022-10-09 14:46:25] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:48 [I 2022-10-09 14:46:25] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00292 [I 2022-10-09 14:46:25] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-09 14:46:25] ...optimizers/trijoint.py.151: total_norm: 10.619181 [I 2022-10-09 14:46:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 830/2304 [I 2022-10-09 14:46:34] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:34 [I 2022-10-09 14:46:34] ...trap/engines/engine.py.229: process: 0.84157 | load: 0.00294 [I 2022-10-09 14:46:34] ...trap/engines/engine.py.230: loss: 0.21107 [I 2022-10-09 14:46:34] ...optimizers/trijoint.py.151: total_norm: 19.348839 [I 2022-10-09 14:46:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 840/2304 [I 2022-10-09 14:46:42] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:38 [I 2022-10-09 14:46:42] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00300 [I 2022-10-09 14:46:42] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-09 14:46:42] ...optimizers/trijoint.py.151: total_norm: 12.969586 [I 2022-10-09 14:46:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 850/2304 [I 2022-10-09 14:46:51] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:28 [I 2022-10-09 14:46:51] ...trap/engines/engine.py.229: process: 0.85085 | load: 0.00297 [I 2022-10-09 14:46:51] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-09 14:46:51] ...optimizers/trijoint.py.151: total_norm: 13.310785 [I 2022-10-09 14:46:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 860/2304 [I 2022-10-09 14:46:59] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:20:20 [I 2022-10-09 14:46:59] ...trap/engines/engine.py.229: process: 0.83620 | load: 0.00292 [I 2022-10-09 14:46:59] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-09 14:46:59] ...optimizers/trijoint.py.151: total_norm: 15.500889 [I 2022-10-09 14:47:07] ...trap/engines/engine.py.224: train: epoch 44 | batch 870/2304 [I 2022-10-09 14:47:07] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:20:07 [I 2022-10-09 14:47:07] ...trap/engines/engine.py.229: process: 0.83663 | load: 0.00293 [I 2022-10-09 14:47:07] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-09 14:47:07] ...optimizers/trijoint.py.151: total_norm: 57.469696 [I 2022-10-09 14:47:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 880/2304 [I 2022-10-09 14:47:16] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:51 [I 2022-10-09 14:47:16] ...trap/engines/engine.py.229: process: 0.82734 | load: 0.00293 [I 2022-10-09 14:47:16] ...trap/engines/engine.py.230: loss: 0.20641 [I 2022-10-09 14:47:16] ...optimizers/trijoint.py.151: total_norm: 40.213650 [I 2022-10-09 14:47:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 890/2304 [I 2022-10-09 14:47:24] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:19:54 [I 2022-10-09 14:47:24] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00294 [I 2022-10-09 14:47:24] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-09 14:47:24] ...optimizers/trijoint.py.151: total_norm: 30.134798 [I 2022-10-09 14:47:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 900/2304 [I 2022-10-09 14:47:33] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:49 [I 2022-10-09 14:47:33] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00299 [I 2022-10-09 14:47:33] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-09 14:47:33] ...optimizers/trijoint.py.151: total_norm: 13.555195 [I 2022-10-09 14:47:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 910/2304 [I 2022-10-09 14:47:41] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:36 [I 2022-10-09 14:47:41] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00314 [I 2022-10-09 14:47:41] ...trap/engines/engine.py.230: loss: 0.09470 [I 2022-10-09 14:47:41] ...optimizers/trijoint.py.151: total_norm: 11.389625 [I 2022-10-09 14:47:50] ...trap/engines/engine.py.224: train: epoch 44 | batch 920/2304 [I 2022-10-09 14:47:50] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:30 [I 2022-10-09 14:47:50] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00297 [I 2022-10-09 14:47:50] ...trap/engines/engine.py.230: loss: 0.16434 [I 2022-10-09 14:47:50] ...optimizers/trijoint.py.151: total_norm: 17.816357 [I 2022-10-09 14:47:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 930/2304 [I 2022-10-09 14:47:58] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:18 [I 2022-10-09 14:47:58] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00292 [I 2022-10-09 14:47:58] ...trap/engines/engine.py.230: loss: 0.17422 [I 2022-10-09 14:47:58] ...optimizers/trijoint.py.151: total_norm: 53.562634 [I 2022-10-09 14:48:07] ...trap/engines/engine.py.224: train: epoch 44 | batch 940/2304 [I 2022-10-09 14:48:07] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:19:05 [I 2022-10-09 14:48:07] ...trap/engines/engine.py.229: process: 0.83352 | load: 0.00293 [I 2022-10-09 14:48:07] ...trap/engines/engine.py.230: loss: 0.13799 [I 2022-10-09 14:48:07] ...optimizers/trijoint.py.151: total_norm: 17.621935 [I 2022-10-09 14:48:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 950/2304 [I 2022-10-09 14:48:15] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:19:04 [I 2022-10-09 14:48:15] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00300 [I 2022-10-09 14:48:15] ...trap/engines/engine.py.230: loss: 0.11792 [I 2022-10-09 14:48:15] ...optimizers/trijoint.py.151: total_norm: 27.036335 [I 2022-10-09 14:48:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 960/2304 [I 2022-10-09 14:48:24] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:59 [I 2022-10-09 14:48:24] ...trap/engines/engine.py.229: process: 0.84665 | load: 0.00299 [I 2022-10-09 14:48:24] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-09 14:48:24] ...optimizers/trijoint.py.151: total_norm: 8.459600 [I 2022-10-09 14:48:32] ...trap/engines/engine.py.224: train: epoch 44 | batch 970/2304 [I 2022-10-09 14:48:32] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:55 [I 2022-10-09 14:48:32] ...trap/engines/engine.py.229: process: 0.83982 | load: 0.00310 [I 2022-10-09 14:48:32] ...trap/engines/engine.py.230: loss: 0.15707 [I 2022-10-09 14:48:32] ...optimizers/trijoint.py.151: total_norm: 17.132620 [I 2022-10-09 14:48:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 980/2304 [I 2022-10-09 14:48:41] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:44 [I 2022-10-09 14:48:41] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00295 [I 2022-10-09 14:48:41] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-09 14:48:41] ...optimizers/trijoint.py.151: total_norm: 12.048957 [I 2022-10-09 14:48:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 990/2304 [I 2022-10-09 14:48:49] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:31 [I 2022-10-09 14:48:49] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00296 [I 2022-10-09 14:48:49] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-09 14:48:49] ...optimizers/trijoint.py.151: total_norm: 17.463314 [I 2022-10-09 14:48:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 1000/2304 [I 2022-10-09 14:48:57] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:24 [I 2022-10-09 14:48:57] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00296 [I 2022-10-09 14:48:57] ...trap/engines/engine.py.230: loss: 0.16822 [I 2022-10-09 14:48:57] ...optimizers/trijoint.py.151: total_norm: 22.481695 [I 2022-10-09 14:49:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 1010/2304 [I 2022-10-09 14:49:06] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:11 [I 2022-10-09 14:49:06] ...trap/engines/engine.py.229: process: 0.83459 | load: 0.00295 [I 2022-10-09 14:49:06] ...trap/engines/engine.py.230: loss: 0.11164 [I 2022-10-09 14:49:06] ...optimizers/trijoint.py.151: total_norm: 14.806027 [I 2022-10-09 14:49:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 1020/2304 [I 2022-10-09 14:49:14] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:18:08 [I 2022-10-09 14:49:14] ...trap/engines/engine.py.229: process: 0.85013 | load: 0.00300 [I 2022-10-09 14:49:14] ...trap/engines/engine.py.230: loss: 0.18283 [I 2022-10-09 14:49:14] ...optimizers/trijoint.py.151: total_norm: 12.235458 [I 2022-10-09 14:49:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 1030/2304 [I 2022-10-09 14:49:23] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:17:53 [I 2022-10-09 14:49:23] ...trap/engines/engine.py.229: process: 0.84500 | load: 0.00300 [I 2022-10-09 14:49:23] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-09 14:49:23] ...optimizers/trijoint.py.151: total_norm: 16.064671 [I 2022-10-09 14:49:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 1040/2304 [I 2022-10-09 14:49:31] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:46 [I 2022-10-09 14:49:31] ...trap/engines/engine.py.229: process: 0.83219 | load: 0.00293 [I 2022-10-09 14:49:31] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-09 14:49:31] ...optimizers/trijoint.py.151: total_norm: 9.407825 [I 2022-10-09 14:49:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 1050/2304 [I 2022-10-09 14:49:40] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:17:44 [I 2022-10-09 14:49:40] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00296 [I 2022-10-09 14:49:40] ...trap/engines/engine.py.230: loss: 0.11269 [I 2022-10-09 14:49:40] ...optimizers/trijoint.py.151: total_norm: 9.761906 [I 2022-10-09 14:49:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 1060/2304 [I 2022-10-09 14:49:48] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:32 [I 2022-10-09 14:49:48] ...trap/engines/engine.py.229: process: 0.84231 | load: 0.00293 [I 2022-10-09 14:49:48] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-09 14:49:48] ...optimizers/trijoint.py.151: total_norm: 15.336576 [I 2022-10-09 14:49:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 1070/2304 [I 2022-10-09 14:49:57] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:15 [I 2022-10-09 14:49:57] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00299 [I 2022-10-09 14:49:57] ...trap/engines/engine.py.230: loss: 0.22199 [I 2022-10-09 14:49:57] ...optimizers/trijoint.py.151: total_norm: 30.376020 [I 2022-10-09 14:50:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 1080/2304 [I 2022-10-09 14:50:05] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:05 [I 2022-10-09 14:50:05] ...trap/engines/engine.py.229: process: 0.82323 | load: 0.00297 [I 2022-10-09 14:50:05] ...trap/engines/engine.py.230: loss: 0.20439 [I 2022-10-09 14:50:05] ...optimizers/trijoint.py.151: total_norm: 21.381937 [I 2022-10-09 14:50:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 1090/2304 [I 2022-10-09 14:50:14] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:06 [I 2022-10-09 14:50:14] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00295 [I 2022-10-09 14:50:14] ...trap/engines/engine.py.230: loss: 0.23671 [I 2022-10-09 14:50:14] ...optimizers/trijoint.py.151: total_norm: 14.616924 [I 2022-10-09 14:50:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 1100/2304 [I 2022-10-09 14:50:22] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:16:54 [I 2022-10-09 14:50:22] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00301 [I 2022-10-09 14:50:22] ...trap/engines/engine.py.230: loss: 0.14182 [I 2022-10-09 14:50:22] ...optimizers/trijoint.py.151: total_norm: 18.868685 [I 2022-10-09 14:50:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 1110/2304 [I 2022-10-09 14:50:30] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:44 [I 2022-10-09 14:50:30] ...trap/engines/engine.py.229: process: 0.83511 | load: 0.00296 [I 2022-10-09 14:50:30] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-09 14:50:30] ...optimizers/trijoint.py.151: total_norm: 11.295549 [I 2022-10-09 14:50:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 1120/2304 [I 2022-10-09 14:50:39] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:39 [I 2022-10-09 14:50:39] ...trap/engines/engine.py.229: process: 0.83985 | load: 0.00298 [I 2022-10-09 14:50:39] ...trap/engines/engine.py.230: loss: 0.17054 [I 2022-10-09 14:50:39] ...optimizers/trijoint.py.151: total_norm: 18.363211 [I 2022-10-09 14:50:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 1130/2304 [I 2022-10-09 14:50:47] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:31 [I 2022-10-09 14:50:47] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00298 [I 2022-10-09 14:50:47] ...trap/engines/engine.py.230: loss: 0.16427 [I 2022-10-09 14:50:47] ...optimizers/trijoint.py.151: total_norm: 15.729867 [I 2022-10-09 14:50:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 1140/2304 [I 2022-10-09 14:50:56] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:27 [I 2022-10-09 14:50:56] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00300 [I 2022-10-09 14:50:56] ...trap/engines/engine.py.230: loss: 0.12962 [I 2022-10-09 14:50:56] ...optimizers/trijoint.py.151: total_norm: 24.877321 [I 2022-10-09 14:51:04] ...trap/engines/engine.py.224: train: epoch 44 | batch 1150/2304 [I 2022-10-09 14:51:04] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:13 [I 2022-10-09 14:51:04] ...trap/engines/engine.py.229: process: 0.84461 | load: 0.00296 [I 2022-10-09 14:51:04] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-09 14:51:04] ...optimizers/trijoint.py.151: total_norm: 16.821098 [I 2022-10-09 14:51:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 1160/2304 [I 2022-10-09 14:51:13] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:13 [I 2022-10-09 14:51:13] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00298 [I 2022-10-09 14:51:13] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-09 14:51:13] ...optimizers/trijoint.py.151: total_norm: 10.677562 [I 2022-10-09 14:51:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 1170/2304 [I 2022-10-09 14:51:21] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:15:59 [I 2022-10-09 14:51:21] ...trap/engines/engine.py.229: process: 0.86159 | load: 0.00296 [I 2022-10-09 14:51:21] ...trap/engines/engine.py.230: loss: 0.18293 [I 2022-10-09 14:51:21] ...optimizers/trijoint.py.151: total_norm: 12.702379 [I 2022-10-09 14:51:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 1180/2304 [I 2022-10-09 14:51:30] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:15:47 [I 2022-10-09 14:51:30] ...trap/engines/engine.py.229: process: 0.83988 | load: 0.00304 [I 2022-10-09 14:51:30] ...trap/engines/engine.py.230: loss: 0.18551 [I 2022-10-09 14:51:30] ...optimizers/trijoint.py.151: total_norm: 23.916079 [I 2022-10-09 14:51:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 1190/2304 [I 2022-10-09 14:51:38] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:37 [I 2022-10-09 14:51:38] ...trap/engines/engine.py.229: process: 0.84010 | load: 0.00300 [I 2022-10-09 14:51:38] ...trap/engines/engine.py.230: loss: 0.21260 [I 2022-10-09 14:51:38] ...optimizers/trijoint.py.151: total_norm: 13.884937 [I 2022-10-09 14:51:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 1200/2304 [I 2022-10-09 14:51:47] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:31 [I 2022-10-09 14:51:47] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00294 [I 2022-10-09 14:51:47] ...trap/engines/engine.py.230: loss: 0.18661 [I 2022-10-09 14:51:47] ...optimizers/trijoint.py.151: total_norm: 21.926517 [I 2022-10-09 14:51:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 1210/2304 [I 2022-10-09 14:51:55] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:29 [I 2022-10-09 14:51:55] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00295 [I 2022-10-09 14:51:55] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-09 14:51:55] ...optimizers/trijoint.py.151: total_norm: 9.529742 [I 2022-10-09 14:52:04] ...trap/engines/engine.py.224: train: epoch 44 | batch 1220/2304 [I 2022-10-09 14:52:04] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:19 [I 2022-10-09 14:52:04] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00302 [I 2022-10-09 14:52:04] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-09 14:52:04] ...optimizers/trijoint.py.151: total_norm: 22.049536 [I 2022-10-09 14:52:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 1230/2304 [I 2022-10-09 14:52:12] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:06 [I 2022-10-09 14:52:12] ...trap/engines/engine.py.229: process: 0.83744 | load: 0.00299 [I 2022-10-09 14:52:12] ...trap/engines/engine.py.230: loss: 0.13898 [I 2022-10-09 14:52:12] ...optimizers/trijoint.py.151: total_norm: 44.025806 [I 2022-10-09 14:52:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 1240/2304 [I 2022-10-09 14:52:21] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:02 [I 2022-10-09 14:52:21] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00296 [I 2022-10-09 14:52:21] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-09 14:52:21] ...optimizers/trijoint.py.151: total_norm: 10.309162 [I 2022-10-09 14:52:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 1250/2304 [I 2022-10-09 14:52:29] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:53 [I 2022-10-09 14:52:29] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00302 [I 2022-10-09 14:52:29] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-09 14:52:29] ...optimizers/trijoint.py.151: total_norm: 10.213907 [I 2022-10-09 14:52:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 1260/2304 [I 2022-10-09 14:52:38] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:42 [I 2022-10-09 14:52:38] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00297 [I 2022-10-09 14:52:38] ...trap/engines/engine.py.230: loss: 0.15043 [I 2022-10-09 14:52:38] ...optimizers/trijoint.py.151: total_norm: 13.724539 [I 2022-10-09 14:52:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 1270/2304 [I 2022-10-09 14:52:46] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:32 [I 2022-10-09 14:52:46] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00297 [I 2022-10-09 14:52:46] ...trap/engines/engine.py.230: loss: 0.16764 [I 2022-10-09 14:52:46] ...optimizers/trijoint.py.151: total_norm: 9.966650 [I 2022-10-09 14:52:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 1280/2304 [I 2022-10-09 14:52:54] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:30 [I 2022-10-09 14:52:54] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00294 [I 2022-10-09 14:52:54] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-09 14:52:54] ...optimizers/trijoint.py.151: total_norm: 15.783773 [I 2022-10-09 14:53:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 1290/2304 [I 2022-10-09 14:53:03] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:23 [I 2022-10-09 14:53:03] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00292 [I 2022-10-09 14:53:03] ...trap/engines/engine.py.230: loss: 0.13189 [I 2022-10-09 14:53:03] ...optimizers/trijoint.py.151: total_norm: 11.918086 [I 2022-10-09 14:53:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 1300/2304 [I 2022-10-09 14:53:12] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:14:12 [I 2022-10-09 14:53:12] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00294 [I 2022-10-09 14:53:12] ...trap/engines/engine.py.230: loss: 0.15175 [I 2022-10-09 14:53:12] ...optimizers/trijoint.py.151: total_norm: 17.145758 [I 2022-10-09 14:53:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 1310/2304 [I 2022-10-09 14:53:20] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:14:10 [I 2022-10-09 14:53:20] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00295 [I 2022-10-09 14:53:20] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-09 14:53:20] ...optimizers/trijoint.py.151: total_norm: 15.412956 [I 2022-10-09 14:53:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 1320/2304 [I 2022-10-09 14:53:29] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:57 [I 2022-10-09 14:53:29] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00291 [I 2022-10-09 14:53:29] ...trap/engines/engine.py.230: loss: 0.14500 [I 2022-10-09 14:53:29] ...optimizers/trijoint.py.151: total_norm: 33.708870 [I 2022-10-09 14:53:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 1330/2304 [I 2022-10-09 14:53:37] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:45 [I 2022-10-09 14:53:37] ...trap/engines/engine.py.229: process: 0.86795 | load: 0.00294 [I 2022-10-09 14:53:37] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-09 14:53:37] ...optimizers/trijoint.py.151: total_norm: 21.737761 [I 2022-10-09 14:53:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 1340/2304 [I 2022-10-09 14:53:46] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:40 [I 2022-10-09 14:53:46] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00298 [I 2022-10-09 14:53:46] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-09 14:53:46] ...optimizers/trijoint.py.151: total_norm: 9.359870 [I 2022-10-09 14:53:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 1350/2304 [I 2022-10-09 14:53:54] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:24 [I 2022-10-09 14:53:54] ...trap/engines/engine.py.229: process: 0.83723 | load: 0.00290 [I 2022-10-09 14:53:54] ...trap/engines/engine.py.230: loss: 0.14483 [I 2022-10-09 14:53:54] ...optimizers/trijoint.py.151: total_norm: 10.810057 [I 2022-10-09 14:54:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 1360/2304 [I 2022-10-09 14:54:03] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:13:19 [I 2022-10-09 14:54:03] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00294 [I 2022-10-09 14:54:03] ...trap/engines/engine.py.230: loss: 0.16701 [I 2022-10-09 14:54:03] ...optimizers/trijoint.py.151: total_norm: 13.293933 [I 2022-10-09 14:54:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 1370/2304 [I 2022-10-09 14:54:11] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:07 [I 2022-10-09 14:54:11] ...trap/engines/engine.py.229: process: 0.85434 | load: 0.00294 [I 2022-10-09 14:54:11] ...trap/engines/engine.py.230: loss: 0.12852 [I 2022-10-09 14:54:11] ...optimizers/trijoint.py.151: total_norm: 9.311776 [I 2022-10-09 14:54:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 1380/2304 [I 2022-10-09 14:54:20] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:13:05 [I 2022-10-09 14:54:20] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00302 [I 2022-10-09 14:54:20] ...trap/engines/engine.py.230: loss: 0.26893 [I 2022-10-09 14:54:20] ...optimizers/trijoint.py.151: total_norm: 16.715302 [I 2022-10-09 14:54:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 1390/2304 [I 2022-10-09 14:54:28] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:12:49 [I 2022-10-09 14:54:28] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00293 [I 2022-10-09 14:54:28] ...trap/engines/engine.py.230: loss: 0.16363 [I 2022-10-09 14:54:28] ...optimizers/trijoint.py.151: total_norm: 23.905785 [I 2022-10-09 14:54:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 1400/2304 [I 2022-10-09 14:54:36] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:12:43 [I 2022-10-09 14:54:36] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00296 [I 2022-10-09 14:54:36] ...trap/engines/engine.py.230: loss: 0.12091 [I 2022-10-09 14:54:36] ...optimizers/trijoint.py.151: total_norm: 13.778562 [I 2022-10-09 14:54:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 1410/2304 [I 2022-10-09 14:54:45] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:30 [I 2022-10-09 14:54:45] ...trap/engines/engine.py.229: process: 0.83119 | load: 0.00295 [I 2022-10-09 14:54:45] ...trap/engines/engine.py.230: loss: 0.22202 [I 2022-10-09 14:54:45] ...optimizers/trijoint.py.151: total_norm: 16.048090 [I 2022-10-09 14:54:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 1420/2304 [I 2022-10-09 14:54:53] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:25 [I 2022-10-09 14:54:53] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00304 [I 2022-10-09 14:54:53] ...trap/engines/engine.py.230: loss: 0.21080 [I 2022-10-09 14:54:53] ...optimizers/trijoint.py.151: total_norm: 18.077646 [I 2022-10-09 14:55:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 1430/2304 [I 2022-10-09 14:55:02] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:12:16 [I 2022-10-09 14:55:02] ...trap/engines/engine.py.229: process: 0.85554 | load: 0.00294 [I 2022-10-09 14:55:02] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-09 14:55:02] ...optimizers/trijoint.py.151: total_norm: 47.202240 [I 2022-10-09 14:55:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 1440/2304 [I 2022-10-09 14:55:10] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:05 [I 2022-10-09 14:55:10] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00306 [I 2022-10-09 14:55:10] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-09 14:55:10] ...optimizers/trijoint.py.151: total_norm: 14.935294 [I 2022-10-09 14:55:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 1450/2304 [I 2022-10-09 14:55:19] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:00 [I 2022-10-09 14:55:19] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00294 [I 2022-10-09 14:55:19] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-09 14:55:19] ...optimizers/trijoint.py.151: total_norm: 22.942225 [I 2022-10-09 14:55:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 1460/2304 [I 2022-10-09 14:55:27] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:11:54 [I 2022-10-09 14:55:27] ...trap/engines/engine.py.229: process: 0.85876 | load: 0.00299 [I 2022-10-09 14:55:27] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-09 14:55:27] ...optimizers/trijoint.py.151: total_norm: 16.113361 [I 2022-10-09 14:55:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 1470/2304 [I 2022-10-09 14:55:36] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:44 [I 2022-10-09 14:55:36] ...trap/engines/engine.py.229: process: 0.83797 | load: 0.00293 [I 2022-10-09 14:55:36] ...trap/engines/engine.py.230: loss: 0.11687 [I 2022-10-09 14:55:36] ...optimizers/trijoint.py.151: total_norm: 8.542352 [I 2022-10-09 14:55:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 1480/2304 [I 2022-10-09 14:55:44] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:31 [I 2022-10-09 14:55:44] ...trap/engines/engine.py.229: process: 0.82711 | load: 0.00304 [I 2022-10-09 14:55:44] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-09 14:55:44] ...optimizers/trijoint.py.151: total_norm: 13.395371 [I 2022-10-09 14:55:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 1490/2304 [I 2022-10-09 14:55:52] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:24 [I 2022-10-09 14:55:52] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00297 [I 2022-10-09 14:55:52] ...trap/engines/engine.py.230: loss: 0.13235 [I 2022-10-09 14:55:52] ...optimizers/trijoint.py.151: total_norm: 19.287987 [I 2022-10-09 14:56:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 1500/2304 [I 2022-10-09 14:56:01] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:20 [I 2022-10-09 14:56:01] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00298 [I 2022-10-09 14:56:01] ...trap/engines/engine.py.230: loss: 0.20338 [I 2022-10-09 14:56:01] ...optimizers/trijoint.py.151: total_norm: 11.564967 [I 2022-10-09 14:56:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 1510/2304 [I 2022-10-09 14:56:09] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:15 [I 2022-10-09 14:56:09] ...trap/engines/engine.py.229: process: 0.86684 | load: 0.00315 [I 2022-10-09 14:56:09] ...trap/engines/engine.py.230: loss: 0.23359 [I 2022-10-09 14:56:09] ...optimizers/trijoint.py.151: total_norm: 27.056036 [I 2022-10-09 14:56:18] ...trap/engines/engine.py.224: train: epoch 44 | batch 1520/2304 [I 2022-10-09 14:56:18] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:11:02 [I 2022-10-09 14:56:18] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00297 [I 2022-10-09 14:56:18] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-09 14:56:18] ...optimizers/trijoint.py.151: total_norm: 8.940898 [I 2022-10-09 14:56:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 1530/2304 [I 2022-10-09 14:56:26] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:10:55 [I 2022-10-09 14:56:26] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00303 [I 2022-10-09 14:56:26] ...trap/engines/engine.py.230: loss: 0.17920 [I 2022-10-09 14:56:26] ...optimizers/trijoint.py.151: total_norm: 14.498769 [I 2022-10-09 14:56:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 1540/2304 [I 2022-10-09 14:56:35] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:46 [I 2022-10-09 14:56:35] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00299 [I 2022-10-09 14:56:35] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-09 14:56:35] ...optimizers/trijoint.py.151: total_norm: 15.187875 [I 2022-10-09 14:56:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 1550/2304 [I 2022-10-09 14:56:43] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:36 [I 2022-10-09 14:56:43] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00305 [I 2022-10-09 14:56:43] ...trap/engines/engine.py.230: loss: 0.13584 [I 2022-10-09 14:56:43] ...optimizers/trijoint.py.151: total_norm: 16.770372 [I 2022-10-09 14:56:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 1560/2304 [I 2022-10-09 14:56:52] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:29 [I 2022-10-09 14:56:52] ...trap/engines/engine.py.229: process: 0.85063 | load: 0.00299 [I 2022-10-09 14:56:52] ...trap/engines/engine.py.230: loss: 0.20106 [I 2022-10-09 14:56:52] ...optimizers/trijoint.py.151: total_norm: 11.607095 [I 2022-10-09 14:57:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 1570/2304 [I 2022-10-09 14:57:00] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:17 [I 2022-10-09 14:57:00] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00297 [I 2022-10-09 14:57:00] ...trap/engines/engine.py.230: loss: 0.14584 [I 2022-10-09 14:57:00] ...optimizers/trijoint.py.151: total_norm: 18.708412 [I 2022-10-09 14:57:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 1580/2304 [I 2022-10-09 14:57:09] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:12 [I 2022-10-09 14:57:09] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00305 [I 2022-10-09 14:57:09] ...trap/engines/engine.py.230: loss: 0.11946 [I 2022-10-09 14:57:09] ...optimizers/trijoint.py.151: total_norm: 14.111986 [I 2022-10-09 14:57:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 1590/2304 [I 2022-10-09 14:57:17] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:06 [I 2022-10-09 14:57:17] ...trap/engines/engine.py.229: process: 0.85085 | load: 0.00302 [I 2022-10-09 14:57:17] ...trap/engines/engine.py.230: loss: 0.13680 [I 2022-10-09 14:57:17] ...optimizers/trijoint.py.151: total_norm: 12.259752 [I 2022-10-09 14:57:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 1600/2304 [I 2022-10-09 14:57:26] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:54 [I 2022-10-09 14:57:26] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00298 [I 2022-10-09 14:57:26] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-09 14:57:26] ...optimizers/trijoint.py.151: total_norm: 10.016862 [I 2022-10-09 14:57:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 1610/2304 [I 2022-10-09 14:57:34] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:41 [I 2022-10-09 14:57:34] ...trap/engines/engine.py.229: process: 0.83478 | load: 0.00300 [I 2022-10-09 14:57:34] ...trap/engines/engine.py.230: loss: 0.23776 [I 2022-10-09 14:57:34] ...optimizers/trijoint.py.151: total_norm: 27.439007 [I 2022-10-09 14:57:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 1620/2304 [I 2022-10-09 14:57:43] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:37 [I 2022-10-09 14:57:43] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00300 [I 2022-10-09 14:57:43] ...trap/engines/engine.py.230: loss: 0.24500 [I 2022-10-09 14:57:43] ...optimizers/trijoint.py.151: total_norm: 19.070477 [I 2022-10-09 14:57:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 1630/2304 [I 2022-10-09 14:57:51] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:33 [I 2022-10-09 14:57:51] ...trap/engines/engine.py.229: process: 0.85863 | load: 0.00300 [I 2022-10-09 14:57:51] ...trap/engines/engine.py.230: loss: 0.13974 [I 2022-10-09 14:57:51] ...optimizers/trijoint.py.151: total_norm: 17.600740 [I 2022-10-09 14:57:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 1640/2304 [I 2022-10-09 14:57:59] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:20 [I 2022-10-09 14:57:59] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00300 [I 2022-10-09 14:57:59] ...trap/engines/engine.py.230: loss: 0.16449 [I 2022-10-09 14:57:59] ...optimizers/trijoint.py.151: total_norm: 10.710043 [I 2022-10-09 14:58:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 1650/2304 [I 2022-10-09 14:58:08] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:13 [I 2022-10-09 14:58:08] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00297 [I 2022-10-09 14:58:08] ...trap/engines/engine.py.230: loss: 0.17773 [I 2022-10-09 14:58:08] ...optimizers/trijoint.py.151: total_norm: 17.485748 [I 2022-10-09 14:58:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 1660/2304 [I 2022-10-09 14:58:16] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:01 [I 2022-10-09 14:58:16] ...trap/engines/engine.py.229: process: 0.82529 | load: 0.00298 [I 2022-10-09 14:58:16] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-09 14:58:16] ...optimizers/trijoint.py.151: total_norm: 7.883954 [I 2022-10-09 14:58:25] ...trap/engines/engine.py.224: train: epoch 44 | batch 1670/2304 [I 2022-10-09 14:58:25] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:58 [I 2022-10-09 14:58:25] ...trap/engines/engine.py.229: process: 0.86333 | load: 0.00296 [I 2022-10-09 14:58:25] ...trap/engines/engine.py.230: loss: 0.20092 [I 2022-10-09 14:58:25] ...optimizers/trijoint.py.151: total_norm: 38.821777 [I 2022-10-09 14:58:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 1680/2304 [I 2022-10-09 14:58:33] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:51 [I 2022-10-09 14:58:33] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00295 [I 2022-10-09 14:58:33] ...trap/engines/engine.py.230: loss: 0.13470 [I 2022-10-09 14:58:33] ...optimizers/trijoint.py.151: total_norm: 14.892365 [I 2022-10-09 14:58:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 1690/2304 [I 2022-10-09 14:58:42] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:44 [I 2022-10-09 14:58:42] ...trap/engines/engine.py.229: process: 0.85978 | load: 0.00299 [I 2022-10-09 14:58:42] ...trap/engines/engine.py.230: loss: 0.14009 [I 2022-10-09 14:58:42] ...optimizers/trijoint.py.151: total_norm: 11.105214 [I 2022-10-09 14:58:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 1700/2304 [I 2022-10-09 14:58:51] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:32 [I 2022-10-09 14:58:51] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00296 [I 2022-10-09 14:58:51] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-09 14:58:51] ...optimizers/trijoint.py.151: total_norm: 37.924545 [I 2022-10-09 14:58:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 1710/2304 [I 2022-10-09 14:58:59] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:24 [I 2022-10-09 14:58:59] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00295 [I 2022-10-09 14:58:59] ...trap/engines/engine.py.230: loss: 0.15307 [I 2022-10-09 14:58:59] ...optimizers/trijoint.py.151: total_norm: 33.622540 [I 2022-10-09 14:59:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 1720/2304 [I 2022-10-09 14:59:08] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:18 [I 2022-10-09 14:59:08] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00306 [I 2022-10-09 14:59:08] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-09 14:59:08] ...optimizers/trijoint.py.151: total_norm: 15.896890 [I 2022-10-09 14:59:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 1730/2304 [I 2022-10-09 14:59:16] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:08 [I 2022-10-09 14:59:16] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00302 [I 2022-10-09 14:59:16] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-09 14:59:16] ...optimizers/trijoint.py.151: total_norm: 14.106662 [I 2022-10-09 14:59:25] ...trap/engines/engine.py.224: train: epoch 44 | batch 1740/2304 [I 2022-10-09 14:59:25] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:57 [I 2022-10-09 14:59:25] ...trap/engines/engine.py.229: process: 0.84014 | load: 0.00300 [I 2022-10-09 14:59:25] ...trap/engines/engine.py.230: loss: 0.19330 [I 2022-10-09 14:59:25] ...optimizers/trijoint.py.151: total_norm: 14.615908 [I 2022-10-09 14:59:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 1750/2304 [I 2022-10-09 14:59:33] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:51 [I 2022-10-09 14:59:33] ...trap/engines/engine.py.229: process: 0.86578 | load: 0.00296 [I 2022-10-09 14:59:33] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-09 14:59:33] ...optimizers/trijoint.py.151: total_norm: 10.319880 [I 2022-10-09 14:59:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 1760/2304 [I 2022-10-09 14:59:42] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:39 [I 2022-10-09 14:59:42] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00297 [I 2022-10-09 14:59:42] ...trap/engines/engine.py.230: loss: 0.15387 [I 2022-10-09 14:59:42] ...optimizers/trijoint.py.151: total_norm: 28.733757 [I 2022-10-09 14:59:50] ...trap/engines/engine.py.224: train: epoch 44 | batch 1770/2304 [I 2022-10-09 14:59:50] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:32 [I 2022-10-09 14:59:50] ...trap/engines/engine.py.229: process: 0.86316 | load: 0.00300 [I 2022-10-09 14:59:50] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-09 14:59:50] ...optimizers/trijoint.py.151: total_norm: 15.401937 [I 2022-10-09 14:59:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 1780/2304 [I 2022-10-09 14:59:58] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:24 [I 2022-10-09 14:59:58] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00299 [I 2022-10-09 14:59:58] ...trap/engines/engine.py.230: loss: 0.17203 [I 2022-10-09 14:59:58] ...optimizers/trijoint.py.151: total_norm: 25.256227 [I 2022-10-09 15:00:07] ...trap/engines/engine.py.224: train: epoch 44 | batch 1790/2304 [I 2022-10-09 15:00:07] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:16 [I 2022-10-09 15:00:07] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00292 [I 2022-10-09 15:00:07] ...trap/engines/engine.py.230: loss: 0.22700 [I 2022-10-09 15:00:07] ...optimizers/trijoint.py.151: total_norm: 16.275459 [I 2022-10-09 15:00:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 1800/2304 [I 2022-10-09 15:00:15] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:07:08 [I 2022-10-09 15:00:15] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00295 [I 2022-10-09 15:00:15] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-09 15:00:15] ...optimizers/trijoint.py.151: total_norm: 19.850313 [I 2022-10-09 15:00:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 1810/2304 [I 2022-10-09 15:00:24] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:57 [I 2022-10-09 15:00:24] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00297 [I 2022-10-09 15:00:24] ...trap/engines/engine.py.230: loss: 0.18087 [I 2022-10-09 15:00:24] ...optimizers/trijoint.py.151: total_norm: 9.690759 [I 2022-10-09 15:00:32] ...trap/engines/engine.py.224: train: epoch 44 | batch 1820/2304 [I 2022-10-09 15:00:32] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:06:49 [I 2022-10-09 15:00:32] ...trap/engines/engine.py.229: process: 0.84530 | load: 0.00301 [I 2022-10-09 15:00:32] ...trap/engines/engine.py.230: loss: 0.20665 [I 2022-10-09 15:00:32] ...optimizers/trijoint.py.151: total_norm: 30.703836 [I 2022-10-09 15:00:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 1830/2304 [I 2022-10-09 15:00:41] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:39 [I 2022-10-09 15:00:41] ...trap/engines/engine.py.229: process: 0.83278 | load: 0.00299 [I 2022-10-09 15:00:41] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-09 15:00:41] ...optimizers/trijoint.py.151: total_norm: 8.958582 [I 2022-10-09 15:00:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 1840/2304 [I 2022-10-09 15:00:49] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:30 [I 2022-10-09 15:00:49] ...trap/engines/engine.py.229: process: 0.83479 | load: 0.00293 [I 2022-10-09 15:00:49] ...trap/engines/engine.py.230: loss: 0.19600 [I 2022-10-09 15:00:49] ...optimizers/trijoint.py.151: total_norm: 36.283833 [I 2022-10-09 15:00:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 1850/2304 [I 2022-10-09 15:00:58] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:26 [I 2022-10-09 15:00:58] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00292 [I 2022-10-09 15:00:58] ...trap/engines/engine.py.230: loss: 0.18104 [I 2022-10-09 15:00:58] ...optimizers/trijoint.py.151: total_norm: 13.299371 [I 2022-10-09 15:01:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 1860/2304 [I 2022-10-09 15:01:06] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:16 [I 2022-10-09 15:01:06] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00293 [I 2022-10-09 15:01:06] ...trap/engines/engine.py.230: loss: 0.17949 [I 2022-10-09 15:01:06] ...optimizers/trijoint.py.151: total_norm: 11.360276 [I 2022-10-09 15:01:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 1870/2304 [I 2022-10-09 15:01:15] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:05 [I 2022-10-09 15:01:15] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00292 [I 2022-10-09 15:01:15] ...trap/engines/engine.py.230: loss: 0.16542 [I 2022-10-09 15:01:15] ...optimizers/trijoint.py.151: total_norm: 15.074370 [I 2022-10-09 15:01:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 1880/2304 [I 2022-10-09 15:01:23] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:57 [I 2022-10-09 15:01:23] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00296 [I 2022-10-09 15:01:23] ...trap/engines/engine.py.230: loss: 0.22017 [I 2022-10-09 15:01:23] ...optimizers/trijoint.py.151: total_norm: 21.557455 [I 2022-10-09 15:01:32] ...trap/engines/engine.py.224: train: epoch 44 | batch 1890/2304 [I 2022-10-09 15:01:32] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:50 [I 2022-10-09 15:01:32] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00298 [I 2022-10-09 15:01:32] ...trap/engines/engine.py.230: loss: 0.20835 [I 2022-10-09 15:01:32] ...optimizers/trijoint.py.151: total_norm: 14.300388 [I 2022-10-09 15:01:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 1900/2304 [I 2022-10-09 15:01:40] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:39 [I 2022-10-09 15:01:40] ...trap/engines/engine.py.229: process: 0.83935 | load: 0.00294 [I 2022-10-09 15:01:40] ...trap/engines/engine.py.230: loss: 0.14717 [I 2022-10-09 15:01:40] ...optimizers/trijoint.py.151: total_norm: 13.658216 [I 2022-10-09 15:01:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 1910/2304 [I 2022-10-09 15:01:48] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:31 [I 2022-10-09 15:01:48] ...trap/engines/engine.py.229: process: 0.85124 | load: 0.00292 [I 2022-10-09 15:01:48] ...trap/engines/engine.py.230: loss: 0.20627 [I 2022-10-09 15:01:48] ...optimizers/trijoint.py.151: total_norm: 16.302526 [I 2022-10-09 15:01:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 1920/2304 [I 2022-10-09 15:01:57] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:24 [I 2022-10-09 15:01:57] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00294 [I 2022-10-09 15:01:57] ...trap/engines/engine.py.230: loss: 0.11409 [I 2022-10-09 15:01:57] ...optimizers/trijoint.py.151: total_norm: 14.926710 [I 2022-10-09 15:02:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 1930/2304 [I 2022-10-09 15:02:05] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:15 [I 2022-10-09 15:02:05] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00297 [I 2022-10-09 15:02:05] ...trap/engines/engine.py.230: loss: 0.15196 [I 2022-10-09 15:02:05] ...optimizers/trijoint.py.151: total_norm: 39.096462 [I 2022-10-09 15:02:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 1940/2304 [I 2022-10-09 15:02:14] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:08 [I 2022-10-09 15:02:14] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00307 [I 2022-10-09 15:02:14] ...trap/engines/engine.py.230: loss: 0.12129 [I 2022-10-09 15:02:14] ...optimizers/trijoint.py.151: total_norm: 8.844859 [I 2022-10-09 15:02:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 1950/2304 [I 2022-10-09 15:02:22] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:58 [I 2022-10-09 15:02:22] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00292 [I 2022-10-09 15:02:22] ...trap/engines/engine.py.230: loss: 0.22497 [I 2022-10-09 15:02:22] ...optimizers/trijoint.py.151: total_norm: 15.553944 [I 2022-10-09 15:02:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 1960/2304 [I 2022-10-09 15:02:31] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:48 [I 2022-10-09 15:02:31] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00308 [I 2022-10-09 15:02:31] ...trap/engines/engine.py.230: loss: 0.16592 [I 2022-10-09 15:02:31] ...optimizers/trijoint.py.151: total_norm: 20.428364 [I 2022-10-09 15:02:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 1970/2304 [I 2022-10-09 15:02:39] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:40 [I 2022-10-09 15:02:39] ...trap/engines/engine.py.229: process: 0.83268 | load: 0.00305 [I 2022-10-09 15:02:39] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-09 15:02:39] ...optimizers/trijoint.py.151: total_norm: 25.874949 [I 2022-10-09 15:02:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 1980/2304 [I 2022-10-09 15:02:48] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:34 [I 2022-10-09 15:02:48] ...trap/engines/engine.py.229: process: 0.85744 | load: 0.00290 [I 2022-10-09 15:02:48] ...trap/engines/engine.py.230: loss: 0.25247 [I 2022-10-09 15:02:48] ...optimizers/trijoint.py.151: total_norm: 22.371447 [I 2022-10-09 15:02:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 1990/2304 [I 2022-10-09 15:02:56] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:24 [I 2022-10-09 15:02:56] ...trap/engines/engine.py.229: process: 0.83833 | load: 0.00296 [I 2022-10-09 15:02:56] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-09 15:02:56] ...optimizers/trijoint.py.151: total_norm: 17.637394 [I 2022-10-09 15:03:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 2000/2304 [I 2022-10-09 15:03:05] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:16 [I 2022-10-09 15:03:05] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00295 [I 2022-10-09 15:03:05] ...trap/engines/engine.py.230: loss: 0.19144 [I 2022-10-09 15:03:05] ...optimizers/trijoint.py.151: total_norm: 27.484365 [I 2022-10-09 15:03:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 2010/2304 [I 2022-10-09 15:03:13] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:06 [I 2022-10-09 15:03:13] ...trap/engines/engine.py.229: process: 0.83378 | load: 0.00298 [I 2022-10-09 15:03:13] ...trap/engines/engine.py.230: loss: 0.15418 [I 2022-10-09 15:03:13] ...optimizers/trijoint.py.151: total_norm: 17.982300 [I 2022-10-09 15:03:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 2020/2304 [I 2022-10-09 15:03:21] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:58 [I 2022-10-09 15:03:21] ...trap/engines/engine.py.229: process: 0.82730 | load: 0.00294 [I 2022-10-09 15:03:21] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-09 15:03:21] ...optimizers/trijoint.py.151: total_norm: 10.424068 [I 2022-10-09 15:03:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 2030/2304 [I 2022-10-09 15:03:30] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:52 [I 2022-10-09 15:03:30] ...trap/engines/engine.py.229: process: 0.84242 | load: 0.00294 [I 2022-10-09 15:03:30] ...trap/engines/engine.py.230: loss: 0.18690 [I 2022-10-09 15:03:30] ...optimizers/trijoint.py.151: total_norm: 11.628828 [I 2022-10-09 15:03:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 2040/2304 [I 2022-10-09 15:03:38] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:44 [I 2022-10-09 15:03:38] ...trap/engines/engine.py.229: process: 0.85210 | load: 0.00300 [I 2022-10-09 15:03:38] ...trap/engines/engine.py.230: loss: 0.16877 [I 2022-10-09 15:03:38] ...optimizers/trijoint.py.151: total_norm: 12.706618 [I 2022-10-09 15:03:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 2050/2304 [I 2022-10-09 15:03:47] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:35 [I 2022-10-09 15:03:47] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00297 [I 2022-10-09 15:03:47] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-09 15:03:47] ...optimizers/trijoint.py.151: total_norm: 13.973394 [I 2022-10-09 15:03:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 2060/2304 [I 2022-10-09 15:03:55] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:26 [I 2022-10-09 15:03:55] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00300 [I 2022-10-09 15:03:55] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-09 15:03:55] ...optimizers/trijoint.py.151: total_norm: 25.658747 [I 2022-10-09 15:04:04] ...trap/engines/engine.py.224: train: epoch 44 | batch 2070/2304 [I 2022-10-09 15:04:04] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:19 [I 2022-10-09 15:04:04] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00297 [I 2022-10-09 15:04:04] ...trap/engines/engine.py.230: loss: 0.17657 [I 2022-10-09 15:04:04] ...optimizers/trijoint.py.151: total_norm: 31.444803 [I 2022-10-09 15:04:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 2080/2304 [I 2022-10-09 15:04:12] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:09 [I 2022-10-09 15:04:12] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00311 [I 2022-10-09 15:04:12] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-09 15:04:12] ...optimizers/trijoint.py.151: total_norm: 13.837977 [I 2022-10-09 15:04:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 2090/2304 [I 2022-10-09 15:04:21] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:01 [I 2022-10-09 15:04:21] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00297 [I 2022-10-09 15:04:21] ...trap/engines/engine.py.230: loss: 0.20372 [I 2022-10-09 15:04:21] ...optimizers/trijoint.py.151: total_norm: 20.857840 [I 2022-10-09 15:04:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 2100/2304 [I 2022-10-09 15:04:29] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:52 [I 2022-10-09 15:04:29] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00310 [I 2022-10-09 15:04:29] ...trap/engines/engine.py.230: loss: 0.24612 [I 2022-10-09 15:04:29] ...optimizers/trijoint.py.151: total_norm: 15.106272 [I 2022-10-09 15:04:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 2110/2304 [I 2022-10-09 15:04:38] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:44 [I 2022-10-09 15:04:38] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00294 [I 2022-10-09 15:04:38] ...trap/engines/engine.py.230: loss: 0.22576 [I 2022-10-09 15:04:38] ...optimizers/trijoint.py.151: total_norm: 18.982834 [I 2022-10-09 15:04:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 2120/2304 [I 2022-10-09 15:04:46] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:35 [I 2022-10-09 15:04:46] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00311 [I 2022-10-09 15:04:46] ...trap/engines/engine.py.230: loss: 0.16439 [I 2022-10-09 15:04:46] ...optimizers/trijoint.py.151: total_norm: 11.936044 [I 2022-10-09 15:04:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 2130/2304 [I 2022-10-09 15:04:55] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:27 [I 2022-10-09 15:04:55] ...trap/engines/engine.py.229: process: 0.85095 | load: 0.00300 [I 2022-10-09 15:04:55] ...trap/engines/engine.py.230: loss: 0.15222 [I 2022-10-09 15:04:55] ...optimizers/trijoint.py.151: total_norm: 42.334946 [I 2022-10-09 15:05:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 2140/2304 [I 2022-10-09 15:05:03] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:18 [I 2022-10-09 15:05:03] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00300 [I 2022-10-09 15:05:03] ...trap/engines/engine.py.230: loss: 0.14488 [I 2022-10-09 15:05:03] ...optimizers/trijoint.py.151: total_norm: 13.937396 [I 2022-10-09 15:05:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 2150/2304 [I 2022-10-09 15:05:12] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:10 [I 2022-10-09 15:05:12] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00297 [I 2022-10-09 15:05:12] ...trap/engines/engine.py.230: loss: 0.20807 [I 2022-10-09 15:05:12] ...optimizers/trijoint.py.151: total_norm: 24.835649 [I 2022-10-09 15:05:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 2160/2304 [I 2022-10-09 15:05:20] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:02 [I 2022-10-09 15:05:20] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00303 [I 2022-10-09 15:05:20] ...trap/engines/engine.py.230: loss: 0.10935 [I 2022-10-09 15:05:20] ...optimizers/trijoint.py.151: total_norm: 6.615655 [I 2022-10-09 15:05:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 2170/2304 [I 2022-10-09 15:05:29] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:01:52 [I 2022-10-09 15:05:29] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00299 [I 2022-10-09 15:05:29] ...trap/engines/engine.py.230: loss: 0.16889 [I 2022-10-09 15:05:29] ...optimizers/trijoint.py.151: total_norm: 12.772552 [I 2022-10-09 15:05:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 2180/2304 [I 2022-10-09 15:05:37] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:44 [I 2022-10-09 15:05:37] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00299 [I 2022-10-09 15:05:37] ...trap/engines/engine.py.230: loss: 0.15387 [I 2022-10-09 15:05:37] ...optimizers/trijoint.py.151: total_norm: 19.681007 [I 2022-10-09 15:05:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 2190/2304 [I 2022-10-09 15:05:45] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:36 [I 2022-10-09 15:05:45] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00295 [I 2022-10-09 15:05:45] ...trap/engines/engine.py.230: loss: 0.12715 [I 2022-10-09 15:05:45] ...optimizers/trijoint.py.151: total_norm: 17.754475 [I 2022-10-09 15:05:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 2200/2304 [I 2022-10-09 15:05:54] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:28 [I 2022-10-09 15:05:54] ...trap/engines/engine.py.229: process: 0.85427 | load: 0.00313 [I 2022-10-09 15:05:54] ...trap/engines/engine.py.230: loss: 0.19268 [I 2022-10-09 15:05:54] ...optimizers/trijoint.py.151: total_norm: 26.370470 [I 2022-10-09 15:06:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 2210/2304 [I 2022-10-09 15:06:02] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:19 [I 2022-10-09 15:06:02] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00295 [I 2022-10-09 15:06:02] ...trap/engines/engine.py.230: loss: 0.19567 [I 2022-10-09 15:06:02] ...optimizers/trijoint.py.151: total_norm: 16.966543 [I 2022-10-09 15:06:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 2220/2304 [I 2022-10-09 15:06:11] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:11 [I 2022-10-09 15:06:11] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00297 [I 2022-10-09 15:06:11] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-09 15:06:11] ...optimizers/trijoint.py.151: total_norm: 8.733294 [I 2022-10-09 15:06:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 2230/2304 [I 2022-10-09 15:06:19] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:02 [I 2022-10-09 15:06:19] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00296 [I 2022-10-09 15:06:19] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-09 15:06:19] ...optimizers/trijoint.py.151: total_norm: 14.306679 [I 2022-10-09 15:06:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 2240/2304 [I 2022-10-09 15:06:28] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:00:54 [I 2022-10-09 15:06:28] ...trap/engines/engine.py.229: process: 0.84322 | load: 0.00295 [I 2022-10-09 15:06:28] ...trap/engines/engine.py.230: loss: 0.17825 [I 2022-10-09 15:06:28] ...optimizers/trijoint.py.151: total_norm: 29.079842 [I 2022-10-09 15:06:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 2250/2304 [I 2022-10-09 15:06:36] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:45 [I 2022-10-09 15:06:36] ...trap/engines/engine.py.229: process: 0.83969 | load: 0.00300 [I 2022-10-09 15:06:36] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-09 15:06:36] ...optimizers/trijoint.py.151: total_norm: 19.399853 [I 2022-10-09 15:06:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 2260/2304 [I 2022-10-09 15:06:45] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:37 [I 2022-10-09 15:06:45] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00301 [I 2022-10-09 15:06:45] ...trap/engines/engine.py.230: loss: 0.15988 [I 2022-10-09 15:06:45] ...optimizers/trijoint.py.151: total_norm: 31.700415 [I 2022-10-09 15:06:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 2270/2304 [I 2022-10-09 15:06:53] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:28 [I 2022-10-09 15:06:53] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00298 [I 2022-10-09 15:06:53] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-09 15:06:53] ...optimizers/trijoint.py.151: total_norm: 30.437366 [I 2022-10-09 15:07:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 2280/2304 [I 2022-10-09 15:07:02] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:20 [I 2022-10-09 15:07:02] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00295 [I 2022-10-09 15:07:02] ...trap/engines/engine.py.230: loss: 0.16601 [I 2022-10-09 15:07:02] ...optimizers/trijoint.py.151: total_norm: 30.327452 [I 2022-10-09 15:07:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 2290/2304 [I 2022-10-09 15:07:10] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:11 [I 2022-10-09 15:07:10] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00294 [I 2022-10-09 15:07:10] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-09 15:07:10] ...optimizers/trijoint.py.151: total_norm: 16.349800 [I 2022-10-09 15:07:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 2300/2304 [I 2022-10-09 15:07:19] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:03 [I 2022-10-09 15:07:19] ...trap/engines/engine.py.229: process: 0.82809 | load: 0.00016 [I 2022-10-09 15:07:19] ...trap/engines/engine.py.230: loss: 0.19726 [I 2022-10-09 15:07:19] ...optimizers/trijoint.py.151: total_norm: 12.351441 [I 2022-10-09 15:07:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 2304/2304 [I 2022-10-09 15:07:22] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:00 [I 2022-10-09 15:07:22] ...trap/engines/engine.py.229: process: 0.82439 | load: 0.00015 [I 2022-10-09 15:07:22] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-09 15:07:22] ...optimizers/trijoint.py.151: total_norm: 11.064383 [S 2022-10-09 15:07:23] ...trap/engines/engine.py.129: train_epoch.epoch: 44 [S 2022-10-09 15:07:23] ...trap/engines/engine.py.129: train_epoch.loss: 0.15654616386926615 [I 2022-10-09 15:07:23] ...s/criterions/triplet.py.84: margin = 0.2700000000000001 [I 2022-10-09 15:07:24] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 44 [I 2022-10-09 15:07:24] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 15:07:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 15:07:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 15:07:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 15:07:27] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 15:07:28] ...trap/engines/engine.py.321: eval: epoch 44 | batch 0/510 [I 2022-10-09 15:07:28] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:55 [I 2022-10-09 15:07:28] ...trap/engines/engine.py.326: process: 3.52053 | load: 2.81030 [I 2022-10-09 15:07:33] ...trap/engines/engine.py.321: eval: epoch 44 | batch 10/510 [I 2022-10-09 15:07:33] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:22 [I 2022-10-09 15:07:33] ...trap/engines/engine.py.326: process: 0.47554 | load: 0.00018 [I 2022-10-09 15:07:39] ...trap/engines/engine.py.321: eval: epoch 44 | batch 20/510 [I 2022-10-09 15:07:39] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:13 [I 2022-10-09 15:07:39] ...trap/engines/engine.py.326: process: 0.54469 | load: 0.00020 [I 2022-10-09 15:07:46] ...trap/engines/engine.py.321: eval: epoch 44 | batch 30/510 [I 2022-10-09 15:07:46] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:13 [I 2022-10-09 15:07:46] ...trap/engines/engine.py.326: process: 0.42370 | load: 0.00019 [I 2022-10-09 15:07:52] ...trap/engines/engine.py.321: eval: epoch 44 | batch 40/510 [I 2022-10-09 15:07:52] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:53 [I 2022-10-09 15:07:52] ...trap/engines/engine.py.326: process: 0.63133 | load: 0.00019 [I 2022-10-09 15:07:56] ...trap/engines/engine.py.321: eval: epoch 44 | batch 50/510 [I 2022-10-09 15:07:56] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:27 [I 2022-10-09 15:07:56] ...trap/engines/engine.py.326: process: 0.42347 | load: 0.00017 [I 2022-10-09 15:08:00] ...trap/engines/engine.py.321: eval: epoch 44 | batch 60/510 [I 2022-10-09 15:08:00] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:25 [I 2022-10-09 15:08:00] ...trap/engines/engine.py.326: process: 0.55738 | load: 0.00018 [I 2022-10-09 15:08:06] ...trap/engines/engine.py.321: eval: epoch 44 | batch 70/510 [I 2022-10-09 15:08:06] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:39 [I 2022-10-09 15:08:06] ...trap/engines/engine.py.326: process: 0.43721 | load: 0.00017 [I 2022-10-09 15:08:11] ...trap/engines/engine.py.321: eval: epoch 44 | batch 80/510 [I 2022-10-09 15:08:11] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:03 [I 2022-10-09 15:08:11] ...trap/engines/engine.py.326: process: 0.61033 | load: 0.00018 [I 2022-10-09 15:08:16] ...trap/engines/engine.py.321: eval: epoch 44 | batch 90/510 [I 2022-10-09 15:08:16] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:15 [I 2022-10-09 15:08:16] ...trap/engines/engine.py.326: process: 0.47223 | load: 0.00022 [I 2022-10-09 15:08:20] ...trap/engines/engine.py.321: eval: epoch 44 | batch 100/510 [I 2022-10-09 15:08:20] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:49 [I 2022-10-09 15:08:20] ...trap/engines/engine.py.326: process: 0.30800 | load: 0.00018 [I 2022-10-09 15:08:23] ...trap/engines/engine.py.321: eval: epoch 44 | batch 110/510 [I 2022-10-09 15:08:23] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:10 [I 2022-10-09 15:08:23] ...trap/engines/engine.py.326: process: 0.31516 | load: 0.00018 [I 2022-10-09 15:08:27] ...trap/engines/engine.py.321: eval: epoch 44 | batch 120/510 [I 2022-10-09 15:08:27] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:03 [I 2022-10-09 15:08:27] ...trap/engines/engine.py.326: process: 0.30998 | load: 0.00018 [I 2022-10-09 15:08:30] ...trap/engines/engine.py.321: eval: epoch 44 | batch 130/510 [I 2022-10-09 15:08:30] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:00 [I 2022-10-09 15:08:30] ...trap/engines/engine.py.326: process: 0.31867 | load: 0.00019 [I 2022-10-09 15:08:33] ...trap/engines/engine.py.321: eval: epoch 44 | batch 140/510 [I 2022-10-09 15:08:33] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:56 [I 2022-10-09 15:08:33] ...trap/engines/engine.py.326: process: 0.31654 | load: 0.00018 [I 2022-10-09 15:08:36] ...trap/engines/engine.py.321: eval: epoch 44 | batch 150/510 [I 2022-10-09 15:08:36] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:53 [I 2022-10-09 15:08:36] ...trap/engines/engine.py.326: process: 0.30861 | load: 0.00019 [I 2022-10-09 15:08:39] ...trap/engines/engine.py.321: eval: epoch 44 | batch 160/510 [I 2022-10-09 15:08:39] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-09 15:08:39] ...trap/engines/engine.py.326: process: 0.31718 | load: 0.00019 [I 2022-10-09 15:08:42] ...trap/engines/engine.py.321: eval: epoch 44 | batch 170/510 [I 2022-10-09 15:08:42] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:47 [I 2022-10-09 15:08:42] ...trap/engines/engine.py.326: process: 0.31552 | load: 0.00018 [I 2022-10-09 15:08:46] ...trap/engines/engine.py.321: eval: epoch 44 | batch 180/510 [I 2022-10-09 15:08:46] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:44 [I 2022-10-09 15:08:46] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00019 [I 2022-10-09 15:08:49] ...trap/engines/engine.py.321: eval: epoch 44 | batch 190/510 [I 2022-10-09 15:08:49] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:42 [I 2022-10-09 15:08:49] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00018 [I 2022-10-09 15:08:52] ...trap/engines/engine.py.321: eval: epoch 44 | batch 200/510 [I 2022-10-09 15:08:52] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-09 15:08:52] ...trap/engines/engine.py.326: process: 0.31407 | load: 0.00021 [I 2022-10-09 15:08:55] ...trap/engines/engine.py.321: eval: epoch 44 | batch 210/510 [I 2022-10-09 15:08:55] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-09 15:08:55] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00019 [I 2022-10-09 15:08:58] ...trap/engines/engine.py.321: eval: epoch 44 | batch 220/510 [I 2022-10-09 15:08:58] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:32 [I 2022-10-09 15:08:58] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00019 [I 2022-10-09 15:09:01] ...trap/engines/engine.py.321: eval: epoch 44 | batch 230/510 [I 2022-10-09 15:09:01] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:29 [I 2022-10-09 15:09:01] ...trap/engines/engine.py.326: process: 0.31461 | load: 0.00018 [I 2022-10-09 15:09:05] ...trap/engines/engine.py.321: eval: epoch 44 | batch 240/510 [I 2022-10-09 15:09:05] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:25 [I 2022-10-09 15:09:05] ...trap/engines/engine.py.326: process: 0.32124 | load: 0.00020 [I 2022-10-09 15:09:08] ...trap/engines/engine.py.321: eval: epoch 44 | batch 250/510 [I 2022-10-09 15:09:08] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:22 [I 2022-10-09 15:09:08] ...trap/engines/engine.py.326: process: 0.31902 | load: 0.00018 [I 2022-10-09 15:09:11] ...trap/engines/engine.py.321: eval: epoch 44 | batch 260/510 [I 2022-10-09 15:09:11] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:18 [I 2022-10-09 15:09:11] ...trap/engines/engine.py.326: process: 0.30879 | load: 0.00018 [I 2022-10-09 15:09:14] ...trap/engines/engine.py.321: eval: epoch 44 | batch 270/510 [I 2022-10-09 15:09:14] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 15:09:14] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00019 [I 2022-10-09 15:09:17] ...trap/engines/engine.py.321: eval: epoch 44 | batch 280/510 [I 2022-10-09 15:09:17] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-09 15:09:17] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00018 [I 2022-10-09 15:09:21] ...trap/engines/engine.py.321: eval: epoch 44 | batch 290/510 [I 2022-10-09 15:09:21] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:09 [I 2022-10-09 15:09:21] ...trap/engines/engine.py.326: process: 0.31691 | load: 0.00020 [I 2022-10-09 15:09:24] ...trap/engines/engine.py.321: eval: epoch 44 | batch 300/510 [I 2022-10-09 15:09:24] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:06 [I 2022-10-09 15:09:24] ...trap/engines/engine.py.326: process: 0.31563 | load: 0.00019 [I 2022-10-09 15:09:27] ...trap/engines/engine.py.321: eval: epoch 44 | batch 310/510 [I 2022-10-09 15:09:27] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-09 15:09:27] ...trap/engines/engine.py.326: process: 0.31525 | load: 0.00019 [I 2022-10-09 15:09:30] ...trap/engines/engine.py.321: eval: epoch 44 | batch 320/510 [I 2022-10-09 15:09:30] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-09 15:09:30] ...trap/engines/engine.py.326: process: 0.31833 | load: 0.00020 [I 2022-10-09 15:09:33] ...trap/engines/engine.py.321: eval: epoch 44 | batch 330/510 [I 2022-10-09 15:09:33] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-09 15:09:33] ...trap/engines/engine.py.326: process: 0.31581 | load: 0.00020 [I 2022-10-09 15:09:36] ...trap/engines/engine.py.321: eval: epoch 44 | batch 340/510 [I 2022-10-09 15:09:36] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-09 15:09:36] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00020 [I 2022-10-09 15:09:40] ...trap/engines/engine.py.321: eval: epoch 44 | batch 350/510 [I 2022-10-09 15:09:40] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:50 [I 2022-10-09 15:09:40] ...trap/engines/engine.py.326: process: 0.31517 | load: 0.00019 [I 2022-10-09 15:09:43] ...trap/engines/engine.py.321: eval: epoch 44 | batch 360/510 [I 2022-10-09 15:09:43] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-09 15:09:43] ...trap/engines/engine.py.326: process: 0.31731 | load: 0.00018 [I 2022-10-09 15:09:46] ...trap/engines/engine.py.321: eval: epoch 44 | batch 370/510 [I 2022-10-09 15:09:46] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 15:09:46] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00019 [I 2022-10-09 15:09:49] ...trap/engines/engine.py.321: eval: epoch 44 | batch 380/510 [I 2022-10-09 15:09:49] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 15:09:49] ...trap/engines/engine.py.326: process: 0.31893 | load: 0.00020 [I 2022-10-09 15:09:52] ...trap/engines/engine.py.321: eval: epoch 44 | batch 390/510 [I 2022-10-09 15:09:52] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:37 [I 2022-10-09 15:09:52] ...trap/engines/engine.py.326: process: 0.31807 | load: 0.00020 [I 2022-10-09 15:09:56] ...trap/engines/engine.py.321: eval: epoch 44 | batch 400/510 [I 2022-10-09 15:09:56] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:34 [I 2022-10-09 15:09:56] ...trap/engines/engine.py.326: process: 0.31584 | load: 0.00018 [I 2022-10-09 15:09:59] ...trap/engines/engine.py.321: eval: epoch 44 | batch 410/510 [I 2022-10-09 15:09:59] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-09 15:09:59] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00020 [I 2022-10-09 15:10:02] ...trap/engines/engine.py.321: eval: epoch 44 | batch 420/510 [I 2022-10-09 15:10:02] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 15:10:02] ...trap/engines/engine.py.326: process: 0.31617 | load: 0.00019 [I 2022-10-09 15:10:05] ...trap/engines/engine.py.321: eval: epoch 44 | batch 430/510 [I 2022-10-09 15:10:05] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 15:10:05] ...trap/engines/engine.py.326: process: 0.32325 | load: 0.00019 [I 2022-10-09 15:10:08] ...trap/engines/engine.py.321: eval: epoch 44 | batch 440/510 [I 2022-10-09 15:10:08] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 15:10:08] ...trap/engines/engine.py.326: process: 0.31544 | load: 0.00019 [I 2022-10-09 15:10:11] ...trap/engines/engine.py.321: eval: epoch 44 | batch 450/510 [I 2022-10-09 15:10:11] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 15:10:11] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00018 [I 2022-10-09 15:10:15] ...trap/engines/engine.py.321: eval: epoch 44 | batch 460/510 [I 2022-10-09 15:10:15] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-09 15:10:15] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00018 [I 2022-10-09 15:10:18] ...trap/engines/engine.py.321: eval: epoch 44 | batch 470/510 [I 2022-10-09 15:10:18] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 15:10:18] ...trap/engines/engine.py.326: process: 0.31348 | load: 0.00019 [I 2022-10-09 15:10:21] ...trap/engines/engine.py.321: eval: epoch 44 | batch 480/510 [I 2022-10-09 15:10:21] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 15:10:21] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00019 [I 2022-10-09 15:10:24] ...trap/engines/engine.py.321: eval: epoch 44 | batch 490/510 [I 2022-10-09 15:10:24] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-09 15:10:24] ...trap/engines/engine.py.326: process: 0.30548 | load: 0.00020 [I 2022-10-09 15:10:27] ...trap/engines/engine.py.321: eval: epoch 44 | batch 500/510 [I 2022-10-09 15:10:27] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 15:10:27] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00020 [I 2022-10-09 15:10:30] ...trap/engines/engine.py.321: eval: epoch 44 | batch 510/510 [I 2022-10-09 15:10:30] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 15:10:30] ...trap/engines/engine.py.326: process: 0.30834 | load: 0.00015 [S 2022-10-09 15:10:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 44 [I 2022-10-09 15:10:31] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.697800 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.898700 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934800 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.706100 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897900 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933600 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.003763 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008001 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007427 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011954 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008814 [S 2022-10-09 15:10:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006453 [I 2022-10-09 15:11:06] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 15:11:06] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 15:11:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 15:11:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 15:11:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 15:11:09] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 15:11:10] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 15:11:15] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 15:11:15] ...trap/engines/engine.py.163: Training model on trainset for epoch 45 [I 2022-10-09 15:11:15] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 15:11:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 0/2304 [I 2022-10-09 15:11:18] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:43 [I 2022-10-09 15:11:18] ...trap/engines/engine.py.229: process: 2.80516 | load: 1.90025 [I 2022-10-09 15:11:18] ...trap/engines/engine.py.230: loss: 0.16540 [I 2022-10-09 15:11:18] ...optimizers/trijoint.py.151: total_norm: 23.346279 [I 2022-10-09 15:11:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 10/2304 [I 2022-10-09 15:11:27] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:18 [I 2022-10-09 15:11:27] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00607 [I 2022-10-09 15:11:27] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-09 15:11:27] ...optimizers/trijoint.py.151: total_norm: 17.999132 [I 2022-10-09 15:11:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 20/2304 [I 2022-10-09 15:11:35] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:49 [I 2022-10-09 15:11:35] ...trap/engines/engine.py.229: process: 0.82671 | load: 0.00481 [I 2022-10-09 15:11:35] ...trap/engines/engine.py.230: loss: 0.26709 [I 2022-10-09 15:11:35] ...optimizers/trijoint.py.151: total_norm: 40.592224 [I 2022-10-09 15:11:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 30/2304 [I 2022-10-09 15:11:44] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:10 [I 2022-10-09 15:11:44] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00392 [I 2022-10-09 15:11:44] ...trap/engines/engine.py.230: loss: 0.14020 [I 2022-10-09 15:11:44] ...optimizers/trijoint.py.151: total_norm: 18.579432 [I 2022-10-09 15:11:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 40/2304 [I 2022-10-09 15:11:52] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:01 [I 2022-10-09 15:11:52] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00409 [I 2022-10-09 15:11:52] ...trap/engines/engine.py.230: loss: 0.13643 [I 2022-10-09 15:11:52] ...optimizers/trijoint.py.151: total_norm: 11.341733 [I 2022-10-09 15:12:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 50/2304 [I 2022-10-09 15:12:00] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:42 [I 2022-10-09 15:12:00] ...trap/engines/engine.py.229: process: 0.83530 | load: 0.00365 [I 2022-10-09 15:12:00] ...trap/engines/engine.py.230: loss: 0.16705 [I 2022-10-09 15:12:00] ...optimizers/trijoint.py.151: total_norm: 12.221989 [I 2022-10-09 15:12:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 60/2304 [I 2022-10-09 15:12:09] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:43 [I 2022-10-09 15:12:09] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00368 [I 2022-10-09 15:12:09] ...trap/engines/engine.py.230: loss: 0.09457 [I 2022-10-09 15:12:09] ...optimizers/trijoint.py.151: total_norm: 9.818988 [I 2022-10-09 15:12:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 70/2304 [I 2022-10-09 15:12:17] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:41 [I 2022-10-09 15:12:17] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00328 [I 2022-10-09 15:12:17] ...trap/engines/engine.py.230: loss: 0.12026 [I 2022-10-09 15:12:17] ...optimizers/trijoint.py.151: total_norm: 17.579111 [I 2022-10-09 15:12:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 80/2304 [I 2022-10-09 15:12:26] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:24 [I 2022-10-09 15:12:26] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00340 [I 2022-10-09 15:12:26] ...trap/engines/engine.py.230: loss: 0.16655 [I 2022-10-09 15:12:26] ...optimizers/trijoint.py.151: total_norm: 26.240616 [I 2022-10-09 15:12:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 90/2304 [I 2022-10-09 15:12:34] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:10 [I 2022-10-09 15:12:34] ...trap/engines/engine.py.229: process: 0.83058 | load: 0.00371 [I 2022-10-09 15:12:34] ...trap/engines/engine.py.230: loss: 0.12102 [I 2022-10-09 15:12:34] ...optimizers/trijoint.py.151: total_norm: 8.845606 [I 2022-10-09 15:12:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 100/2304 [I 2022-10-09 15:12:43] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:06 [I 2022-10-09 15:12:43] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00355 [I 2022-10-09 15:12:43] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-09 15:12:43] ...optimizers/trijoint.py.151: total_norm: 20.218210 [I 2022-10-09 15:12:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 110/2304 [I 2022-10-09 15:12:51] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:52 [I 2022-10-09 15:12:51] ...trap/engines/engine.py.229: process: 0.83056 | load: 0.00313 [I 2022-10-09 15:12:51] ...trap/engines/engine.py.230: loss: 0.15587 [I 2022-10-09 15:12:51] ...optimizers/trijoint.py.151: total_norm: 12.560852 [I 2022-10-09 15:13:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 120/2304 [I 2022-10-09 15:13:00] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:39 [I 2022-10-09 15:13:00] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00321 [I 2022-10-09 15:13:00] ...trap/engines/engine.py.230: loss: 0.23575 [I 2022-10-09 15:13:00] ...optimizers/trijoint.py.151: total_norm: 23.193716 [I 2022-10-09 15:13:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 130/2304 [I 2022-10-09 15:13:08] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:35 [I 2022-10-09 15:13:08] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00294 [I 2022-10-09 15:13:08] ...trap/engines/engine.py.230: loss: 0.20498 [I 2022-10-09 15:13:08] ...optimizers/trijoint.py.151: total_norm: 24.692707 [I 2022-10-09 15:13:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 140/2304 [I 2022-10-09 15:13:17] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:38 [I 2022-10-09 15:13:17] ...trap/engines/engine.py.229: process: 0.86007 | load: 0.00332 [I 2022-10-09 15:13:17] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-09 15:13:17] ...optimizers/trijoint.py.151: total_norm: 19.336315 [I 2022-10-09 15:13:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 150/2304 [I 2022-10-09 15:13:25] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:20 [I 2022-10-09 15:13:25] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00317 [I 2022-10-09 15:13:25] ...trap/engines/engine.py.230: loss: 0.18537 [I 2022-10-09 15:13:25] ...optimizers/trijoint.py.151: total_norm: 15.210219 [I 2022-10-09 15:13:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 160/2304 [I 2022-10-09 15:13:34] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:06 [I 2022-10-09 15:13:34] ...trap/engines/engine.py.229: process: 0.84456 | load: 0.00343 [I 2022-10-09 15:13:34] ...trap/engines/engine.py.230: loss: 0.23577 [I 2022-10-09 15:13:34] ...optimizers/trijoint.py.151: total_norm: 25.546644 [I 2022-10-09 15:13:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 170/2304 [I 2022-10-09 15:13:42] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:07 [I 2022-10-09 15:13:42] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00325 [I 2022-10-09 15:13:42] ...trap/engines/engine.py.230: loss: 0.15746 [I 2022-10-09 15:13:42] ...optimizers/trijoint.py.151: total_norm: 13.216587 [I 2022-10-09 15:13:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 180/2304 [I 2022-10-09 15:13:51] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:56 [I 2022-10-09 15:13:51] ...trap/engines/engine.py.229: process: 0.84227 | load: 0.00296 [I 2022-10-09 15:13:51] ...trap/engines/engine.py.230: loss: 0.18178 [I 2022-10-09 15:13:51] ...optimizers/trijoint.py.151: total_norm: 17.665440 [I 2022-10-09 15:13:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 190/2304 [I 2022-10-09 15:13:59] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:51 [I 2022-10-09 15:13:59] ...trap/engines/engine.py.229: process: 0.83556 | load: 0.00315 [I 2022-10-09 15:13:59] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-09 15:13:59] ...optimizers/trijoint.py.151: total_norm: 7.924824 [I 2022-10-09 15:14:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 200/2304 [I 2022-10-09 15:14:08] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:32 [I 2022-10-09 15:14:08] ...trap/engines/engine.py.229: process: 0.83791 | load: 0.00316 [I 2022-10-09 15:14:08] ...trap/engines/engine.py.230: loss: 0.12114 [I 2022-10-09 15:14:08] ...optimizers/trijoint.py.151: total_norm: 10.215115 [I 2022-10-09 15:14:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 210/2304 [I 2022-10-09 15:14:16] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:31 [I 2022-10-09 15:14:16] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00314 [I 2022-10-09 15:14:16] ...trap/engines/engine.py.230: loss: 0.19236 [I 2022-10-09 15:14:16] ...optimizers/trijoint.py.151: total_norm: 14.285140 [I 2022-10-09 15:14:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 220/2304 [I 2022-10-09 15:14:24] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:22 [I 2022-10-09 15:14:24] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00295 [I 2022-10-09 15:14:24] ...trap/engines/engine.py.230: loss: 0.23517 [I 2022-10-09 15:14:24] ...optimizers/trijoint.py.151: total_norm: 12.538455 [I 2022-10-09 15:14:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 230/2304 [I 2022-10-09 15:14:33] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:17 [I 2022-10-09 15:14:33] ...trap/engines/engine.py.229: process: 0.87372 | load: 0.00296 [I 2022-10-09 15:14:33] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-09 15:14:33] ...optimizers/trijoint.py.151: total_norm: 13.474464 [I 2022-10-09 15:14:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 240/2304 [I 2022-10-09 15:14:41] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:28:59 [I 2022-10-09 15:14:41] ...trap/engines/engine.py.229: process: 0.83482 | load: 0.00329 [I 2022-10-09 15:14:41] ...trap/engines/engine.py.230: loss: 0.18886 [I 2022-10-09 15:14:41] ...optimizers/trijoint.py.151: total_norm: 17.652578 [I 2022-10-09 15:14:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 250/2304 [I 2022-10-09 15:14:50] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:59 [I 2022-10-09 15:14:50] ...trap/engines/engine.py.229: process: 0.83850 | load: 0.00296 [I 2022-10-09 15:14:50] ...trap/engines/engine.py.230: loss: 0.10638 [I 2022-10-09 15:14:50] ...optimizers/trijoint.py.151: total_norm: 7.972839 [I 2022-10-09 15:14:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 260/2304 [I 2022-10-09 15:14:58] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:50 [I 2022-10-09 15:14:58] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00297 [I 2022-10-09 15:14:58] ...trap/engines/engine.py.230: loss: 0.22726 [I 2022-10-09 15:14:58] ...optimizers/trijoint.py.151: total_norm: 19.418583 [I 2022-10-09 15:15:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 270/2304 [I 2022-10-09 15:15:07] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:30 [I 2022-10-09 15:15:07] ...trap/engines/engine.py.229: process: 0.83336 | load: 0.00324 [I 2022-10-09 15:15:07] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-09 15:15:07] ...optimizers/trijoint.py.151: total_norm: 12.952394 [I 2022-10-09 15:15:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 280/2304 [I 2022-10-09 15:15:15] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:23 [I 2022-10-09 15:15:15] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00294 [I 2022-10-09 15:15:15] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-09 15:15:15] ...optimizers/trijoint.py.151: total_norm: 18.180368 [I 2022-10-09 15:15:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 290/2304 [I 2022-10-09 15:15:24] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:24 [I 2022-10-09 15:15:24] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00295 [I 2022-10-09 15:15:24] ...trap/engines/engine.py.230: loss: 0.20278 [I 2022-10-09 15:15:24] ...optimizers/trijoint.py.151: total_norm: 16.154181 [I 2022-10-09 15:15:32] ...trap/engines/engine.py.224: train: epoch 45 | batch 300/2304 [I 2022-10-09 15:15:32] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:15 [I 2022-10-09 15:15:32] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00296 [I 2022-10-09 15:15:32] ...trap/engines/engine.py.230: loss: 0.14417 [I 2022-10-09 15:15:32] ...optimizers/trijoint.py.151: total_norm: 20.159981 [I 2022-10-09 15:15:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 310/2304 [I 2022-10-09 15:15:41] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:07 [I 2022-10-09 15:15:41] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00305 [I 2022-10-09 15:15:41] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-09 15:15:41] ...optimizers/trijoint.py.151: total_norm: 15.605313 [I 2022-10-09 15:15:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 320/2304 [I 2022-10-09 15:15:49] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:59 [I 2022-10-09 15:15:49] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00296 [I 2022-10-09 15:15:49] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-09 15:15:49] ...optimizers/trijoint.py.151: total_norm: 8.662212 [I 2022-10-09 15:15:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 330/2304 [I 2022-10-09 15:15:58] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:47 [I 2022-10-09 15:15:58] ...trap/engines/engine.py.229: process: 0.83921 | load: 0.00307 [I 2022-10-09 15:15:58] ...trap/engines/engine.py.230: loss: 0.10338 [I 2022-10-09 15:15:58] ...optimizers/trijoint.py.151: total_norm: 23.549379 [I 2022-10-09 15:16:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 340/2304 [I 2022-10-09 15:16:06] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:42 [I 2022-10-09 15:16:06] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00306 [I 2022-10-09 15:16:06] ...trap/engines/engine.py.230: loss: 0.24043 [I 2022-10-09 15:16:06] ...optimizers/trijoint.py.151: total_norm: 20.939035 [I 2022-10-09 15:16:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 350/2304 [I 2022-10-09 15:16:15] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:29 [I 2022-10-09 15:16:15] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00301 [I 2022-10-09 15:16:15] ...trap/engines/engine.py.230: loss: 0.23469 [I 2022-10-09 15:16:15] ...optimizers/trijoint.py.151: total_norm: 13.302125 [I 2022-10-09 15:16:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 360/2304 [I 2022-10-09 15:16:23] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:26 [I 2022-10-09 15:16:23] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00296 [I 2022-10-09 15:16:23] ...trap/engines/engine.py.230: loss: 0.12769 [I 2022-10-09 15:16:23] ...optimizers/trijoint.py.151: total_norm: 10.927137 [I 2022-10-09 15:16:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 370/2304 [I 2022-10-09 15:16:31] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:27:15 [I 2022-10-09 15:16:31] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00297 [I 2022-10-09 15:16:31] ...trap/engines/engine.py.230: loss: 0.15694 [I 2022-10-09 15:16:31] ...optimizers/trijoint.py.151: total_norm: 20.769041 [I 2022-10-09 15:16:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 380/2304 [I 2022-10-09 15:16:40] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:09 [I 2022-10-09 15:16:40] ...trap/engines/engine.py.229: process: 0.84890 | load: 0.00307 [I 2022-10-09 15:16:40] ...trap/engines/engine.py.230: loss: 0.16224 [I 2022-10-09 15:16:40] ...optimizers/trijoint.py.151: total_norm: 14.448716 [I 2022-10-09 15:16:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 390/2304 [I 2022-10-09 15:16:48] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:26:55 [I 2022-10-09 15:16:48] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00301 [I 2022-10-09 15:16:48] ...trap/engines/engine.py.230: loss: 0.19091 [I 2022-10-09 15:16:48] ...optimizers/trijoint.py.151: total_norm: 20.977882 [I 2022-10-09 15:16:57] ...trap/engines/engine.py.224: train: epoch 45 | batch 400/2304 [I 2022-10-09 15:16:57] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:38 [I 2022-10-09 15:16:57] ...trap/engines/engine.py.229: process: 0.84145 | load: 0.00300 [I 2022-10-09 15:16:57] ...trap/engines/engine.py.230: loss: 0.10763 [I 2022-10-09 15:16:57] ...optimizers/trijoint.py.151: total_norm: 13.063787 [I 2022-10-09 15:17:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 410/2304 [I 2022-10-09 15:17:05] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:38 [I 2022-10-09 15:17:05] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00294 [I 2022-10-09 15:17:05] ...trap/engines/engine.py.230: loss: 0.15349 [I 2022-10-09 15:17:05] ...optimizers/trijoint.py.151: total_norm: 14.294502 [I 2022-10-09 15:17:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 420/2304 [I 2022-10-09 15:17:14] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:28 [I 2022-10-09 15:17:14] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00296 [I 2022-10-09 15:17:14] ...trap/engines/engine.py.230: loss: 0.20081 [I 2022-10-09 15:17:14] ...optimizers/trijoint.py.151: total_norm: 37.242329 [I 2022-10-09 15:17:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 430/2304 [I 2022-10-09 15:17:22] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:15 [I 2022-10-09 15:17:22] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00302 [I 2022-10-09 15:17:22] ...trap/engines/engine.py.230: loss: 0.23534 [I 2022-10-09 15:17:22] ...optimizers/trijoint.py.151: total_norm: 29.090208 [I 2022-10-09 15:17:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 440/2304 [I 2022-10-09 15:17:31] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:09 [I 2022-10-09 15:17:31] ...trap/engines/engine.py.229: process: 0.84477 | load: 0.00297 [I 2022-10-09 15:17:31] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-09 15:17:31] ...optimizers/trijoint.py.151: total_norm: 12.275548 [I 2022-10-09 15:17:39] ...trap/engines/engine.py.224: train: epoch 45 | batch 450/2304 [I 2022-10-09 15:17:39] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:25:54 [I 2022-10-09 15:17:39] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00308 [I 2022-10-09 15:17:39] ...trap/engines/engine.py.230: loss: 0.16264 [I 2022-10-09 15:17:39] ...optimizers/trijoint.py.151: total_norm: 13.157338 [I 2022-10-09 15:17:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 460/2304 [I 2022-10-09 15:17:47] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:25:52 [I 2022-10-09 15:17:47] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00295 [I 2022-10-09 15:17:47] ...trap/engines/engine.py.230: loss: 0.27094 [I 2022-10-09 15:17:47] ...optimizers/trijoint.py.151: total_norm: 26.531433 [I 2022-10-09 15:17:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 470/2304 [I 2022-10-09 15:17:56] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:42 [I 2022-10-09 15:17:56] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00297 [I 2022-10-09 15:17:56] ...trap/engines/engine.py.230: loss: 0.17389 [I 2022-10-09 15:17:56] ...optimizers/trijoint.py.151: total_norm: 22.655418 [I 2022-10-09 15:18:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 480/2304 [I 2022-10-09 15:18:04] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:34 [I 2022-10-09 15:18:04] ...trap/engines/engine.py.229: process: 0.83004 | load: 0.00302 [I 2022-10-09 15:18:04] ...trap/engines/engine.py.230: loss: 0.18041 [I 2022-10-09 15:18:04] ...optimizers/trijoint.py.151: total_norm: 13.521359 [I 2022-10-09 15:18:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 490/2304 [I 2022-10-09 15:18:13] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:23 [I 2022-10-09 15:18:13] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00307 [I 2022-10-09 15:18:13] ...trap/engines/engine.py.230: loss: 0.15102 [I 2022-10-09 15:18:13] ...optimizers/trijoint.py.151: total_norm: 13.815189 [I 2022-10-09 15:18:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 500/2304 [I 2022-10-09 15:18:21] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:19 [I 2022-10-09 15:18:21] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00311 [I 2022-10-09 15:18:21] ...trap/engines/engine.py.230: loss: 0.23444 [I 2022-10-09 15:18:21] ...optimizers/trijoint.py.151: total_norm: 22.774031 [I 2022-10-09 15:18:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 510/2304 [I 2022-10-09 15:18:30] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:10 [I 2022-10-09 15:18:30] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00302 [I 2022-10-09 15:18:30] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-09 15:18:30] ...optimizers/trijoint.py.151: total_norm: 20.296860 [I 2022-10-09 15:18:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 520/2304 [I 2022-10-09 15:18:38] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:02 [I 2022-10-09 15:18:38] ...trap/engines/engine.py.229: process: 0.82726 | load: 0.00295 [I 2022-10-09 15:18:38] ...trap/engines/engine.py.230: loss: 0.18980 [I 2022-10-09 15:18:38] ...optimizers/trijoint.py.151: total_norm: 33.656399 [I 2022-10-09 15:18:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 530/2304 [I 2022-10-09 15:18:46] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:24:54 [I 2022-10-09 15:18:46] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00296 [I 2022-10-09 15:18:46] ...trap/engines/engine.py.230: loss: 0.15292 [I 2022-10-09 15:18:46] ...optimizers/trijoint.py.151: total_norm: 17.590317 [I 2022-10-09 15:18:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 540/2304 [I 2022-10-09 15:18:55] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:56 [I 2022-10-09 15:18:55] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00301 [I 2022-10-09 15:18:55] ...trap/engines/engine.py.230: loss: 0.12063 [I 2022-10-09 15:18:55] ...optimizers/trijoint.py.151: total_norm: 7.931533 [I 2022-10-09 15:19:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 550/2304 [I 2022-10-09 15:19:03] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:49 [I 2022-10-09 15:19:03] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00300 [I 2022-10-09 15:19:03] ...trap/engines/engine.py.230: loss: 0.16426 [I 2022-10-09 15:19:03] ...optimizers/trijoint.py.151: total_norm: 12.881588 [I 2022-10-09 15:19:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 560/2304 [I 2022-10-09 15:19:12] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:44 [I 2022-10-09 15:19:12] ...trap/engines/engine.py.229: process: 0.86945 | load: 0.00291 [I 2022-10-09 15:19:12] ...trap/engines/engine.py.230: loss: 0.13093 [I 2022-10-09 15:19:12] ...optimizers/trijoint.py.151: total_norm: 13.377256 [I 2022-10-09 15:19:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 570/2304 [I 2022-10-09 15:19:20] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:24:26 [I 2022-10-09 15:19:20] ...trap/engines/engine.py.229: process: 0.82781 | load: 0.00294 [I 2022-10-09 15:19:20] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-09 15:19:20] ...optimizers/trijoint.py.151: total_norm: 9.194736 [I 2022-10-09 15:19:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 580/2304 [I 2022-10-09 15:19:29] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:20 [I 2022-10-09 15:19:29] ...trap/engines/engine.py.229: process: 0.85703 | load: 0.00297 [I 2022-10-09 15:19:29] ...trap/engines/engine.py.230: loss: 0.12794 [I 2022-10-09 15:19:29] ...optimizers/trijoint.py.151: total_norm: 17.148504 [I 2022-10-09 15:19:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 590/2304 [I 2022-10-09 15:19:37] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:24 [I 2022-10-09 15:19:37] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00294 [I 2022-10-09 15:19:37] ...trap/engines/engine.py.230: loss: 0.18517 [I 2022-10-09 15:19:37] ...optimizers/trijoint.py.151: total_norm: 13.135371 [I 2022-10-09 15:19:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 600/2304 [I 2022-10-09 15:19:46] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:23:56 [I 2022-10-09 15:19:46] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00295 [I 2022-10-09 15:19:46] ...trap/engines/engine.py.230: loss: 0.10393 [I 2022-10-09 15:19:46] ...optimizers/trijoint.py.151: total_norm: 7.799104 [I 2022-10-09 15:19:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 610/2304 [I 2022-10-09 15:19:54] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:23:54 [I 2022-10-09 15:19:54] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00295 [I 2022-10-09 15:19:54] ...trap/engines/engine.py.230: loss: 0.14323 [I 2022-10-09 15:19:54] ...optimizers/trijoint.py.151: total_norm: 15.386447 [I 2022-10-09 15:20:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 620/2304 [I 2022-10-09 15:20:03] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:37 [I 2022-10-09 15:20:03] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00301 [I 2022-10-09 15:20:03] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-09 15:20:03] ...optimizers/trijoint.py.151: total_norm: 13.487097 [I 2022-10-09 15:20:11] ...trap/engines/engine.py.224: train: epoch 45 | batch 630/2304 [I 2022-10-09 15:20:11] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:33 [I 2022-10-09 15:20:11] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00293 [I 2022-10-09 15:20:11] ...trap/engines/engine.py.230: loss: 0.18153 [I 2022-10-09 15:20:11] ...optimizers/trijoint.py.151: total_norm: 25.115784 [I 2022-10-09 15:20:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 640/2304 [I 2022-10-09 15:20:20] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:20 [I 2022-10-09 15:20:20] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00296 [I 2022-10-09 15:20:20] ...trap/engines/engine.py.230: loss: 0.17325 [I 2022-10-09 15:20:20] ...optimizers/trijoint.py.151: total_norm: 23.582850 [I 2022-10-09 15:20:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 650/2304 [I 2022-10-09 15:20:28] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:10 [I 2022-10-09 15:20:28] ...trap/engines/engine.py.229: process: 0.83211 | load: 0.00299 [I 2022-10-09 15:20:28] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-09 15:20:28] ...optimizers/trijoint.py.151: total_norm: 10.885143 [I 2022-10-09 15:20:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 660/2304 [I 2022-10-09 15:20:37] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:08 [I 2022-10-09 15:20:37] ...trap/engines/engine.py.229: process: 0.83207 | load: 0.00335 [I 2022-10-09 15:20:37] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-09 15:20:37] ...optimizers/trijoint.py.151: total_norm: 10.637465 [I 2022-10-09 15:20:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 670/2304 [I 2022-10-09 15:20:45] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:00 [I 2022-10-09 15:20:45] ...trap/engines/engine.py.229: process: 0.84702 | load: 0.00299 [I 2022-10-09 15:20:45] ...trap/engines/engine.py.230: loss: 0.21165 [I 2022-10-09 15:20:45] ...optimizers/trijoint.py.151: total_norm: 24.921722 [I 2022-10-09 15:20:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 680/2304 [I 2022-10-09 15:20:54] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:22:52 [I 2022-10-09 15:20:54] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00297 [I 2022-10-09 15:20:54] ...trap/engines/engine.py.230: loss: 0.16686 [I 2022-10-09 15:20:54] ...optimizers/trijoint.py.151: total_norm: 26.270206 [I 2022-10-09 15:21:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 690/2304 [I 2022-10-09 15:21:02] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:39 [I 2022-10-09 15:21:02] ...trap/engines/engine.py.229: process: 0.85053 | load: 0.00293 [I 2022-10-09 15:21:02] ...trap/engines/engine.py.230: loss: 0.19801 [I 2022-10-09 15:21:02] ...optimizers/trijoint.py.151: total_norm: 19.268806 [I 2022-10-09 15:21:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 700/2304 [I 2022-10-09 15:21:10] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:22:29 [I 2022-10-09 15:21:10] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00294 [I 2022-10-09 15:21:10] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-09 15:21:10] ...optimizers/trijoint.py.151: total_norm: 16.354622 [I 2022-10-09 15:21:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 710/2304 [I 2022-10-09 15:21:19] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:26 [I 2022-10-09 15:21:19] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00295 [I 2022-10-09 15:21:19] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-09 15:21:19] ...optimizers/trijoint.py.151: total_norm: 23.414997 [I 2022-10-09 15:21:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 720/2304 [I 2022-10-09 15:21:27] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:22:26 [I 2022-10-09 15:21:27] ...trap/engines/engine.py.229: process: 0.83261 | load: 0.00297 [I 2022-10-09 15:21:27] ...trap/engines/engine.py.230: loss: 0.13431 [I 2022-10-09 15:21:27] ...optimizers/trijoint.py.151: total_norm: 12.458581 [I 2022-10-09 15:21:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 730/2304 [I 2022-10-09 15:21:36] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:21 [I 2022-10-09 15:21:36] ...trap/engines/engine.py.229: process: 0.85946 | load: 0.00300 [I 2022-10-09 15:21:36] ...trap/engines/engine.py.230: loss: 0.12937 [I 2022-10-09 15:21:36] ...optimizers/trijoint.py.151: total_norm: 16.254789 [I 2022-10-09 15:21:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 740/2304 [I 2022-10-09 15:21:45] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:22:16 [I 2022-10-09 15:21:45] ...trap/engines/engine.py.229: process: 0.87901 | load: 0.00301 [I 2022-10-09 15:21:45] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-09 15:21:45] ...optimizers/trijoint.py.151: total_norm: 21.233889 [I 2022-10-09 15:21:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 750/2304 [I 2022-10-09 15:21:53] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:59 [I 2022-10-09 15:21:53] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00294 [I 2022-10-09 15:21:53] ...trap/engines/engine.py.230: loss: 0.12315 [I 2022-10-09 15:21:53] ...optimizers/trijoint.py.151: total_norm: 76.082870 [I 2022-10-09 15:22:01] ...trap/engines/engine.py.224: train: epoch 45 | batch 760/2304 [I 2022-10-09 15:22:01] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:21:43 [I 2022-10-09 15:22:01] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00314 [I 2022-10-09 15:22:01] ...trap/engines/engine.py.230: loss: 0.14375 [I 2022-10-09 15:22:01] ...optimizers/trijoint.py.151: total_norm: 13.942793 [I 2022-10-09 15:22:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 770/2304 [I 2022-10-09 15:22:10] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:43 [I 2022-10-09 15:22:10] ...trap/engines/engine.py.229: process: 0.86453 | load: 0.00294 [I 2022-10-09 15:22:10] ...trap/engines/engine.py.230: loss: 0.26553 [I 2022-10-09 15:22:10] ...optimizers/trijoint.py.151: total_norm: 19.381279 [I 2022-10-09 15:22:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 780/2304 [I 2022-10-09 15:22:18] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:21:32 [I 2022-10-09 15:22:18] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00298 [I 2022-10-09 15:22:18] ...trap/engines/engine.py.230: loss: 0.17753 [I 2022-10-09 15:22:18] ...optimizers/trijoint.py.151: total_norm: 14.159927 [I 2022-10-09 15:22:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 790/2304 [I 2022-10-09 15:22:27] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:24 [I 2022-10-09 15:22:27] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00295 [I 2022-10-09 15:22:27] ...trap/engines/engine.py.230: loss: 0.16340 [I 2022-10-09 15:22:27] ...optimizers/trijoint.py.151: total_norm: 20.591059 [I 2022-10-09 15:22:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 800/2304 [I 2022-10-09 15:22:35] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:12 [I 2022-10-09 15:22:35] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00299 [I 2022-10-09 15:22:35] ...trap/engines/engine.py.230: loss: 0.26425 [I 2022-10-09 15:22:35] ...optimizers/trijoint.py.151: total_norm: 34.158459 [I 2022-10-09 15:22:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 810/2304 [I 2022-10-09 15:22:44] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:20:59 [I 2022-10-09 15:22:44] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00294 [I 2022-10-09 15:22:44] ...trap/engines/engine.py.230: loss: 0.20101 [I 2022-10-09 15:22:44] ...optimizers/trijoint.py.151: total_norm: 15.803882 [I 2022-10-09 15:22:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 820/2304 [I 2022-10-09 15:22:52] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:20:51 [I 2022-10-09 15:22:52] ...trap/engines/engine.py.229: process: 0.85477 | load: 0.00298 [I 2022-10-09 15:22:52] ...trap/engines/engine.py.230: loss: 0.14957 [I 2022-10-09 15:22:52] ...optimizers/trijoint.py.151: total_norm: 29.616884 [I 2022-10-09 15:23:01] ...trap/engines/engine.py.224: train: epoch 45 | batch 830/2304 [I 2022-10-09 15:23:01] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:42 [I 2022-10-09 15:23:01] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00295 [I 2022-10-09 15:23:01] ...trap/engines/engine.py.230: loss: 0.12763 [I 2022-10-09 15:23:01] ...optimizers/trijoint.py.151: total_norm: 14.993349 [I 2022-10-09 15:23:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 840/2304 [I 2022-10-09 15:23:09] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:42 [I 2022-10-09 15:23:09] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00304 [I 2022-10-09 15:23:09] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-09 15:23:09] ...optimizers/trijoint.py.151: total_norm: 10.820351 [I 2022-10-09 15:23:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 850/2304 [I 2022-10-09 15:23:18] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:20:23 [I 2022-10-09 15:23:18] ...trap/engines/engine.py.229: process: 0.83633 | load: 0.00292 [I 2022-10-09 15:23:18] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-09 15:23:18] ...optimizers/trijoint.py.151: total_norm: 22.976088 [I 2022-10-09 15:23:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 860/2304 [I 2022-10-09 15:23:26] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:19 [I 2022-10-09 15:23:26] ...trap/engines/engine.py.229: process: 0.84546 | load: 0.00294 [I 2022-10-09 15:23:26] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-09 15:23:26] ...optimizers/trijoint.py.151: total_norm: 12.998876 [I 2022-10-09 15:23:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 870/2304 [I 2022-10-09 15:23:35] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:02 [I 2022-10-09 15:23:35] ...trap/engines/engine.py.229: process: 0.83300 | load: 0.00294 [I 2022-10-09 15:23:35] ...trap/engines/engine.py.230: loss: 0.15108 [I 2022-10-09 15:23:35] ...optimizers/trijoint.py.151: total_norm: 21.157822 [I 2022-10-09 15:23:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 880/2304 [I 2022-10-09 15:23:43] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:01 [I 2022-10-09 15:23:43] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00294 [I 2022-10-09 15:23:43] ...trap/engines/engine.py.230: loss: 0.17098 [I 2022-10-09 15:23:43] ...optimizers/trijoint.py.151: total_norm: 21.279039 [I 2022-10-09 15:23:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 890/2304 [I 2022-10-09 15:23:51] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:19:55 [I 2022-10-09 15:23:51] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00306 [I 2022-10-09 15:23:51] ...trap/engines/engine.py.230: loss: 0.17012 [I 2022-10-09 15:23:51] ...optimizers/trijoint.py.151: total_norm: 18.565905 [I 2022-10-09 15:24:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 900/2304 [I 2022-10-09 15:24:00] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:19:43 [I 2022-10-09 15:24:00] ...trap/engines/engine.py.229: process: 0.84101 | load: 0.00292 [I 2022-10-09 15:24:00] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-09 15:24:00] ...optimizers/trijoint.py.151: total_norm: 17.457657 [I 2022-10-09 15:24:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 910/2304 [I 2022-10-09 15:24:08] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:40 [I 2022-10-09 15:24:08] ...trap/engines/engine.py.229: process: 0.84947 | load: 0.00294 [I 2022-10-09 15:24:08] ...trap/engines/engine.py.230: loss: 0.21113 [I 2022-10-09 15:24:08] ...optimizers/trijoint.py.151: total_norm: 25.838957 [I 2022-10-09 15:24:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 920/2304 [I 2022-10-09 15:24:17] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:27 [I 2022-10-09 15:24:17] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00293 [I 2022-10-09 15:24:17] ...trap/engines/engine.py.230: loss: 0.17819 [I 2022-10-09 15:24:17] ...optimizers/trijoint.py.151: total_norm: 34.558537 [I 2022-10-09 15:24:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 930/2304 [I 2022-10-09 15:24:25] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:21 [I 2022-10-09 15:24:25] ...trap/engines/engine.py.229: process: 0.83876 | load: 0.00295 [I 2022-10-09 15:24:25] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-09 15:24:25] ...optimizers/trijoint.py.151: total_norm: 20.514830 [I 2022-10-09 15:24:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 940/2304 [I 2022-10-09 15:24:34] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:08 [I 2022-10-09 15:24:34] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00301 [I 2022-10-09 15:24:34] ...trap/engines/engine.py.230: loss: 0.22235 [I 2022-10-09 15:24:34] ...optimizers/trijoint.py.151: total_norm: 20.340691 [I 2022-10-09 15:24:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 950/2304 [I 2022-10-09 15:24:42] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:01 [I 2022-10-09 15:24:42] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00293 [I 2022-10-09 15:24:42] ...trap/engines/engine.py.230: loss: 0.18870 [I 2022-10-09 15:24:42] ...optimizers/trijoint.py.151: total_norm: 13.561278 [I 2022-10-09 15:24:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 960/2304 [I 2022-10-09 15:24:51] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:18:59 [I 2022-10-09 15:24:51] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00298 [I 2022-10-09 15:24:51] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-09 15:24:51] ...optimizers/trijoint.py.151: total_norm: 12.488068 [I 2022-10-09 15:24:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 970/2304 [I 2022-10-09 15:24:59] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:18:45 [I 2022-10-09 15:24:59] ...trap/engines/engine.py.229: process: 0.83026 | load: 0.00294 [I 2022-10-09 15:24:59] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-09 15:24:59] ...optimizers/trijoint.py.151: total_norm: 8.771734 [I 2022-10-09 15:25:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 980/2304 [I 2022-10-09 15:25:08] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:36 [I 2022-10-09 15:25:08] ...trap/engines/engine.py.229: process: 0.83223 | load: 0.00296 [I 2022-10-09 15:25:08] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-09 15:25:08] ...optimizers/trijoint.py.151: total_norm: 16.951799 [I 2022-10-09 15:25:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 990/2304 [I 2022-10-09 15:25:16] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:32 [I 2022-10-09 15:25:16] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00296 [I 2022-10-09 15:25:16] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-09 15:25:16] ...optimizers/trijoint.py.151: total_norm: 13.985467 [I 2022-10-09 15:25:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 1000/2304 [I 2022-10-09 15:25:25] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:26 [I 2022-10-09 15:25:25] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00308 [I 2022-10-09 15:25:25] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-09 15:25:25] ...optimizers/trijoint.py.151: total_norm: 11.441908 [I 2022-10-09 15:25:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 1010/2304 [I 2022-10-09 15:25:33] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:20 [I 2022-10-09 15:25:33] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00293 [I 2022-10-09 15:25:33] ...trap/engines/engine.py.230: loss: 0.18867 [I 2022-10-09 15:25:33] ...optimizers/trijoint.py.151: total_norm: 34.989525 [I 2022-10-09 15:25:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 1020/2304 [I 2022-10-09 15:25:41] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:03 [I 2022-10-09 15:25:41] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00293 [I 2022-10-09 15:25:41] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-09 15:25:41] ...optimizers/trijoint.py.151: total_norm: 20.275227 [I 2022-10-09 15:25:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 1030/2304 [I 2022-10-09 15:25:50] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:17:54 [I 2022-10-09 15:25:50] ...trap/engines/engine.py.229: process: 0.83580 | load: 0.00295 [I 2022-10-09 15:25:50] ...trap/engines/engine.py.230: loss: 0.18178 [I 2022-10-09 15:25:50] ...optimizers/trijoint.py.151: total_norm: 19.545481 [I 2022-10-09 15:25:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 1040/2304 [I 2022-10-09 15:25:58] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:46 [I 2022-10-09 15:25:58] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00307 [I 2022-10-09 15:25:58] ...trap/engines/engine.py.230: loss: 0.12717 [I 2022-10-09 15:25:58] ...optimizers/trijoint.py.151: total_norm: 22.451845 [I 2022-10-09 15:26:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 1050/2304 [I 2022-10-09 15:26:07] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:34 [I 2022-10-09 15:26:07] ...trap/engines/engine.py.229: process: 0.82468 | load: 0.00293 [I 2022-10-09 15:26:07] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-09 15:26:07] ...optimizers/trijoint.py.151: total_norm: 10.833715 [I 2022-10-09 15:26:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 1060/2304 [I 2022-10-09 15:26:15] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:34 [I 2022-10-09 15:26:15] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00297 [I 2022-10-09 15:26:15] ...trap/engines/engine.py.230: loss: 0.16198 [I 2022-10-09 15:26:15] ...optimizers/trijoint.py.151: total_norm: 22.427650 [I 2022-10-09 15:26:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 1070/2304 [I 2022-10-09 15:26:24] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:21 [I 2022-10-09 15:26:24] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00295 [I 2022-10-09 15:26:24] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-09 15:26:24] ...optimizers/trijoint.py.151: total_norm: 18.699207 [I 2022-10-09 15:26:32] ...trap/engines/engine.py.224: train: epoch 45 | batch 1080/2304 [I 2022-10-09 15:26:32] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:13 [I 2022-10-09 15:26:32] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00299 [I 2022-10-09 15:26:32] ...trap/engines/engine.py.230: loss: 0.19674 [I 2022-10-09 15:26:32] ...optimizers/trijoint.py.151: total_norm: 10.813891 [I 2022-10-09 15:26:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 1090/2304 [I 2022-10-09 15:26:41] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:06 [I 2022-10-09 15:26:41] ...trap/engines/engine.py.229: process: 0.85265 | load: 0.00296 [I 2022-10-09 15:26:41] ...trap/engines/engine.py.230: loss: 0.29616 [I 2022-10-09 15:26:41] ...optimizers/trijoint.py.151: total_norm: 37.466911 [I 2022-10-09 15:26:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 1100/2304 [I 2022-10-09 15:26:49] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:16:57 [I 2022-10-09 15:26:49] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00299 [I 2022-10-09 15:26:49] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-09 15:26:49] ...optimizers/trijoint.py.151: total_norm: 13.285974 [I 2022-10-09 15:26:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 1110/2304 [I 2022-10-09 15:26:58] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:47 [I 2022-10-09 15:26:58] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00309 [I 2022-10-09 15:26:58] ...trap/engines/engine.py.230: loss: 0.20073 [I 2022-10-09 15:26:58] ...optimizers/trijoint.py.151: total_norm: 26.857792 [I 2022-10-09 15:27:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 1120/2304 [I 2022-10-09 15:27:06] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:37 [I 2022-10-09 15:27:06] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00293 [I 2022-10-09 15:27:06] ...trap/engines/engine.py.230: loss: 0.18244 [I 2022-10-09 15:27:06] ...optimizers/trijoint.py.151: total_norm: 14.225906 [I 2022-10-09 15:27:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 1130/2304 [I 2022-10-09 15:27:14] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:28 [I 2022-10-09 15:27:14] ...trap/engines/engine.py.229: process: 0.83702 | load: 0.00296 [I 2022-10-09 15:27:14] ...trap/engines/engine.py.230: loss: 0.17731 [I 2022-10-09 15:27:14] ...optimizers/trijoint.py.151: total_norm: 11.606994 [I 2022-10-09 15:27:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 1140/2304 [I 2022-10-09 15:27:23] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:22 [I 2022-10-09 15:27:23] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00295 [I 2022-10-09 15:27:23] ...trap/engines/engine.py.230: loss: 0.12868 [I 2022-10-09 15:27:23] ...optimizers/trijoint.py.151: total_norm: 18.886921 [I 2022-10-09 15:27:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 1150/2304 [I 2022-10-09 15:27:31] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:16 [I 2022-10-09 15:27:31] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00300 [I 2022-10-09 15:27:31] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-09 15:27:31] ...optimizers/trijoint.py.151: total_norm: 12.835126 [I 2022-10-09 15:27:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 1160/2304 [I 2022-10-09 15:27:40] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:06 [I 2022-10-09 15:27:40] ...trap/engines/engine.py.229: process: 0.83862 | load: 0.00299 [I 2022-10-09 15:27:40] ...trap/engines/engine.py.230: loss: 0.29074 [I 2022-10-09 15:27:40] ...optimizers/trijoint.py.151: total_norm: 18.532433 [I 2022-10-09 15:27:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 1170/2304 [I 2022-10-09 15:27:48] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:15:58 [I 2022-10-09 15:27:48] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00294 [I 2022-10-09 15:27:48] ...trap/engines/engine.py.230: loss: 0.16990 [I 2022-10-09 15:27:48] ...optimizers/trijoint.py.151: total_norm: 11.737264 [I 2022-10-09 15:27:57] ...trap/engines/engine.py.224: train: epoch 45 | batch 1180/2304 [I 2022-10-09 15:27:57] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:52 [I 2022-10-09 15:27:57] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00308 [I 2022-10-09 15:27:57] ...trap/engines/engine.py.230: loss: 0.11834 [I 2022-10-09 15:27:57] ...optimizers/trijoint.py.151: total_norm: 8.934521 [I 2022-10-09 15:28:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 1190/2304 [I 2022-10-09 15:28:05] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:41 [I 2022-10-09 15:28:05] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00301 [I 2022-10-09 15:28:05] ...trap/engines/engine.py.230: loss: 0.22204 [I 2022-10-09 15:28:05] ...optimizers/trijoint.py.151: total_norm: 23.578379 [I 2022-10-09 15:28:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 1200/2304 [I 2022-10-09 15:28:14] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:37 [I 2022-10-09 15:28:14] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00294 [I 2022-10-09 15:28:14] ...trap/engines/engine.py.230: loss: 0.19430 [I 2022-10-09 15:28:14] ...optimizers/trijoint.py.151: total_norm: 8.544190 [I 2022-10-09 15:28:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 1210/2304 [I 2022-10-09 15:28:22] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:19 [I 2022-10-09 15:28:22] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00293 [I 2022-10-09 15:28:22] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-09 15:28:22] ...optimizers/trijoint.py.151: total_norm: 9.497830 [I 2022-10-09 15:28:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 1220/2304 [I 2022-10-09 15:28:31] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:14 [I 2022-10-09 15:28:31] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00303 [I 2022-10-09 15:28:31] ...trap/engines/engine.py.230: loss: 0.23288 [I 2022-10-09 15:28:31] ...optimizers/trijoint.py.151: total_norm: 16.411165 [I 2022-10-09 15:28:39] ...trap/engines/engine.py.224: train: epoch 45 | batch 1230/2304 [I 2022-10-09 15:28:39] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:09 [I 2022-10-09 15:28:39] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00300 [I 2022-10-09 15:28:39] ...trap/engines/engine.py.230: loss: 0.23760 [I 2022-10-09 15:28:39] ...optimizers/trijoint.py.151: total_norm: 25.058329 [I 2022-10-09 15:28:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 1240/2304 [I 2022-10-09 15:28:47] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:56 [I 2022-10-09 15:28:47] ...trap/engines/engine.py.229: process: 0.82813 | load: 0.00299 [I 2022-10-09 15:28:47] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-09 15:28:47] ...optimizers/trijoint.py.151: total_norm: 27.183205 [I 2022-10-09 15:28:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 1250/2304 [I 2022-10-09 15:28:56] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:47 [I 2022-10-09 15:28:56] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00296 [I 2022-10-09 15:28:56] ...trap/engines/engine.py.230: loss: 0.14676 [I 2022-10-09 15:28:56] ...optimizers/trijoint.py.151: total_norm: 15.797695 [I 2022-10-09 15:29:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 1260/2304 [I 2022-10-09 15:29:04] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:41 [I 2022-10-09 15:29:04] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00296 [I 2022-10-09 15:29:04] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-09 15:29:04] ...optimizers/trijoint.py.151: total_norm: 15.600227 [I 2022-10-09 15:29:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 1270/2304 [I 2022-10-09 15:29:13] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:34 [I 2022-10-09 15:29:13] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00294 [I 2022-10-09 15:29:13] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-09 15:29:13] ...optimizers/trijoint.py.151: total_norm: 11.999983 [I 2022-10-09 15:29:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 1280/2304 [I 2022-10-09 15:29:21] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:23 [I 2022-10-09 15:29:21] ...trap/engines/engine.py.229: process: 0.83284 | load: 0.00296 [I 2022-10-09 15:29:21] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-09 15:29:21] ...optimizers/trijoint.py.151: total_norm: 13.292345 [I 2022-10-09 15:29:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 1290/2304 [I 2022-10-09 15:29:30] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:18 [I 2022-10-09 15:29:30] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00307 [I 2022-10-09 15:29:30] ...trap/engines/engine.py.230: loss: 0.16691 [I 2022-10-09 15:29:30] ...optimizers/trijoint.py.151: total_norm: 14.469322 [I 2022-10-09 15:29:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 1300/2304 [I 2022-10-09 15:29:38] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:08 [I 2022-10-09 15:29:38] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00299 [I 2022-10-09 15:29:38] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-09 15:29:38] ...optimizers/trijoint.py.151: total_norm: 10.144320 [I 2022-10-09 15:29:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 1310/2304 [I 2022-10-09 15:29:47] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:13:58 [I 2022-10-09 15:29:47] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00300 [I 2022-10-09 15:29:47] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-09 15:29:47] ...optimizers/trijoint.py.151: total_norm: 17.322550 [I 2022-10-09 15:29:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 1320/2304 [I 2022-10-09 15:29:55] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:50 [I 2022-10-09 15:29:55] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00296 [I 2022-10-09 15:29:55] ...trap/engines/engine.py.230: loss: 0.10017 [I 2022-10-09 15:29:55] ...optimizers/trijoint.py.151: total_norm: 12.099494 [I 2022-10-09 15:30:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 1330/2304 [I 2022-10-09 15:30:04] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:40 [I 2022-10-09 15:30:04] ...trap/engines/engine.py.229: process: 0.83020 | load: 0.00295 [I 2022-10-09 15:30:04] ...trap/engines/engine.py.230: loss: 0.09063 [I 2022-10-09 15:30:04] ...optimizers/trijoint.py.151: total_norm: 7.298129 [I 2022-10-09 15:30:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 1340/2304 [I 2022-10-09 15:30:12] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:33 [I 2022-10-09 15:30:12] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00296 [I 2022-10-09 15:30:12] ...trap/engines/engine.py.230: loss: 0.20027 [I 2022-10-09 15:30:12] ...optimizers/trijoint.py.151: total_norm: 19.833086 [I 2022-10-09 15:30:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 1350/2304 [I 2022-10-09 15:30:20] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:27 [I 2022-10-09 15:30:20] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00301 [I 2022-10-09 15:30:20] ...trap/engines/engine.py.230: loss: 0.14427 [I 2022-10-09 15:30:20] ...optimizers/trijoint.py.151: total_norm: 17.303072 [I 2022-10-09 15:30:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 1360/2304 [I 2022-10-09 15:30:29] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:16 [I 2022-10-09 15:30:29] ...trap/engines/engine.py.229: process: 0.84759 | load: 0.00300 [I 2022-10-09 15:30:29] ...trap/engines/engine.py.230: loss: 0.12966 [I 2022-10-09 15:30:29] ...optimizers/trijoint.py.151: total_norm: 17.337751 [I 2022-10-09 15:30:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 1370/2304 [I 2022-10-09 15:30:37] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:09 [I 2022-10-09 15:30:37] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00299 [I 2022-10-09 15:30:37] ...trap/engines/engine.py.230: loss: 0.16619 [I 2022-10-09 15:30:37] ...optimizers/trijoint.py.151: total_norm: 19.319468 [I 2022-10-09 15:30:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 1380/2304 [I 2022-10-09 15:30:46] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:03 [I 2022-10-09 15:30:46] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00294 [I 2022-10-09 15:30:46] ...trap/engines/engine.py.230: loss: 0.18870 [I 2022-10-09 15:30:46] ...optimizers/trijoint.py.151: total_norm: 23.766680 [I 2022-10-09 15:30:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 1390/2304 [I 2022-10-09 15:30:54] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:50 [I 2022-10-09 15:30:54] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00294 [I 2022-10-09 15:30:54] ...trap/engines/engine.py.230: loss: 0.19770 [I 2022-10-09 15:30:54] ...optimizers/trijoint.py.151: total_norm: 16.356251 [I 2022-10-09 15:31:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 1400/2304 [I 2022-10-09 15:31:03] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:36 [I 2022-10-09 15:31:03] ...trap/engines/engine.py.229: process: 0.84284 | load: 0.00296 [I 2022-10-09 15:31:03] ...trap/engines/engine.py.230: loss: 0.08102 [I 2022-10-09 15:31:03] ...optimizers/trijoint.py.151: total_norm: 7.913653 [I 2022-10-09 15:31:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 1410/2304 [I 2022-10-09 15:31:29] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:19:48 [I 2022-10-09 15:31:29] ...trap/engines/engine.py.229: process: 0.83525 | load: 0.00299 [I 2022-10-09 15:31:29] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-09 15:31:29] ...optimizers/trijoint.py.151: total_norm: 17.317026 [I 2022-10-09 15:31:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 1420/2304 [I 2022-10-09 15:31:38] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:13:10 [I 2022-10-09 15:31:38] ...trap/engines/engine.py.229: process: 0.84761 | load: 0.00300 [I 2022-10-09 15:31:38] ...trap/engines/engine.py.230: loss: 0.10431 [I 2022-10-09 15:31:38] ...optimizers/trijoint.py.151: total_norm: 8.789941 [I 2022-10-09 15:31:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 1430/2304 [I 2022-10-09 15:31:46] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:22 [I 2022-10-09 15:31:46] ...trap/engines/engine.py.229: process: 0.83569 | load: 0.00294 [I 2022-10-09 15:31:46] ...trap/engines/engine.py.230: loss: 0.13111 [I 2022-10-09 15:31:46] ...optimizers/trijoint.py.151: total_norm: 10.976670 [I 2022-10-09 15:31:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 1440/2304 [I 2022-10-09 15:31:55] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:06 [I 2022-10-09 15:31:55] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00296 [I 2022-10-09 15:31:55] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-09 15:31:55] ...optimizers/trijoint.py.151: total_norm: 10.928267 [I 2022-10-09 15:32:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 1450/2304 [I 2022-10-09 15:32:03] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:01 [I 2022-10-09 15:32:03] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00296 [I 2022-10-09 15:32:03] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-09 15:32:03] ...optimizers/trijoint.py.151: total_norm: 17.323816 [I 2022-10-09 15:32:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 1460/2304 [I 2022-10-09 15:32:12] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:54 [I 2022-10-09 15:32:12] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00304 [I 2022-10-09 15:32:12] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-09 15:32:12] ...optimizers/trijoint.py.151: total_norm: 13.782615 [I 2022-10-09 15:32:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 1470/2304 [I 2022-10-09 15:32:20] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:49 [I 2022-10-09 15:32:20] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00309 [I 2022-10-09 15:32:20] ...trap/engines/engine.py.230: loss: 0.18233 [I 2022-10-09 15:32:20] ...optimizers/trijoint.py.151: total_norm: 26.110538 [I 2022-10-09 15:32:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 1480/2304 [I 2022-10-09 15:32:29] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:38 [I 2022-10-09 15:32:29] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00295 [I 2022-10-09 15:32:29] ...trap/engines/engine.py.230: loss: 0.10136 [I 2022-10-09 15:32:29] ...optimizers/trijoint.py.151: total_norm: 9.640037 [I 2022-10-09 15:32:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 1490/2304 [I 2022-10-09 15:32:37] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:29 [I 2022-10-09 15:32:37] ...trap/engines/engine.py.229: process: 0.84082 | load: 0.00292 [I 2022-10-09 15:32:37] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-09 15:32:37] ...optimizers/trijoint.py.151: total_norm: 11.495946 [I 2022-10-09 15:32:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 1500/2304 [I 2022-10-09 15:32:45] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:21 [I 2022-10-09 15:32:45] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00295 [I 2022-10-09 15:32:45] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-09 15:32:45] ...optimizers/trijoint.py.151: total_norm: 16.827576 [I 2022-10-09 15:32:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 1510/2304 [I 2022-10-09 15:32:54] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:12 [I 2022-10-09 15:32:54] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00295 [I 2022-10-09 15:32:54] ...trap/engines/engine.py.230: loss: 0.23312 [I 2022-10-09 15:32:54] ...optimizers/trijoint.py.151: total_norm: 43.002033 [I 2022-10-09 15:33:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 1520/2304 [I 2022-10-09 15:33:02] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:06 [I 2022-10-09 15:33:02] ...trap/engines/engine.py.229: process: 0.87712 | load: 0.00302 [I 2022-10-09 15:33:02] ...trap/engines/engine.py.230: loss: 0.14839 [I 2022-10-09 15:33:02] ...optimizers/trijoint.py.151: total_norm: 19.842419 [I 2022-10-09 15:33:11] ...trap/engines/engine.py.224: train: epoch 45 | batch 1530/2304 [I 2022-10-09 15:33:11] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:56 [I 2022-10-09 15:33:11] ...trap/engines/engine.py.229: process: 0.84511 | load: 0.00305 [I 2022-10-09 15:33:11] ...trap/engines/engine.py.230: loss: 0.18422 [I 2022-10-09 15:33:11] ...optimizers/trijoint.py.151: total_norm: 10.102691 [I 2022-10-09 15:33:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 1540/2304 [I 2022-10-09 15:33:19] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:52 [I 2022-10-09 15:33:19] ...trap/engines/engine.py.229: process: 0.86353 | load: 0.00298 [I 2022-10-09 15:33:19] ...trap/engines/engine.py.230: loss: 0.20172 [I 2022-10-09 15:33:19] ...optimizers/trijoint.py.151: total_norm: 22.705162 [I 2022-10-09 15:33:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 1550/2304 [I 2022-10-09 15:33:28] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:39 [I 2022-10-09 15:33:28] ...trap/engines/engine.py.229: process: 0.82954 | load: 0.00293 [I 2022-10-09 15:33:28] ...trap/engines/engine.py.230: loss: 0.21292 [I 2022-10-09 15:33:28] ...optimizers/trijoint.py.151: total_norm: 25.613686 [I 2022-10-09 15:33:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 1560/2304 [I 2022-10-09 15:33:36] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:32 [I 2022-10-09 15:33:36] ...trap/engines/engine.py.229: process: 0.87057 | load: 0.00294 [I 2022-10-09 15:33:36] ...trap/engines/engine.py.230: loss: 0.20001 [I 2022-10-09 15:33:36] ...optimizers/trijoint.py.151: total_norm: 18.479511 [I 2022-10-09 15:33:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 1570/2304 [I 2022-10-09 15:33:45] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:20 [I 2022-10-09 15:33:45] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00304 [I 2022-10-09 15:33:45] ...trap/engines/engine.py.230: loss: 0.18482 [I 2022-10-09 15:33:45] ...optimizers/trijoint.py.151: total_norm: 63.148563 [I 2022-10-09 15:33:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 1580/2304 [I 2022-10-09 15:33:53] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:12 [I 2022-10-09 15:33:53] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00294 [I 2022-10-09 15:33:53] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-09 15:33:53] ...optimizers/trijoint.py.151: total_norm: 15.536361 [I 2022-10-09 15:34:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 1590/2304 [I 2022-10-09 15:34:02] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:03 [I 2022-10-09 15:34:02] ...trap/engines/engine.py.229: process: 0.85771 | load: 0.00299 [I 2022-10-09 15:34:02] ...trap/engines/engine.py.230: loss: 0.16634 [I 2022-10-09 15:34:02] ...optimizers/trijoint.py.151: total_norm: 18.816788 [I 2022-10-09 15:34:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 1600/2304 [I 2022-10-09 15:34:10] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:55 [I 2022-10-09 15:34:10] ...trap/engines/engine.py.229: process: 0.85068 | load: 0.00293 [I 2022-10-09 15:34:10] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-09 15:34:10] ...optimizers/trijoint.py.151: total_norm: 33.891514 [I 2022-10-09 15:34:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 1610/2304 [I 2022-10-09 15:34:19] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:45 [I 2022-10-09 15:34:19] ...trap/engines/engine.py.229: process: 0.83954 | load: 0.00297 [I 2022-10-09 15:34:19] ...trap/engines/engine.py.230: loss: 0.16253 [I 2022-10-09 15:34:19] ...optimizers/trijoint.py.151: total_norm: 21.651434 [I 2022-10-09 15:34:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 1620/2304 [I 2022-10-09 15:34:27] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:36 [I 2022-10-09 15:34:27] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00295 [I 2022-10-09 15:34:27] ...trap/engines/engine.py.230: loss: 0.23524 [I 2022-10-09 15:34:27] ...optimizers/trijoint.py.151: total_norm: 13.036194 [I 2022-10-09 15:34:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 1630/2304 [I 2022-10-09 15:34:36] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:28 [I 2022-10-09 15:34:36] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00294 [I 2022-10-09 15:34:36] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-09 15:34:36] ...optimizers/trijoint.py.151: total_norm: 20.061806 [I 2022-10-09 15:34:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 1640/2304 [I 2022-10-09 15:34:44] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:23 [I 2022-10-09 15:34:44] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00294 [I 2022-10-09 15:34:44] ...trap/engines/engine.py.230: loss: 0.20914 [I 2022-10-09 15:34:44] ...optimizers/trijoint.py.151: total_norm: 12.326397 [I 2022-10-09 15:34:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 1650/2304 [I 2022-10-09 15:34:53] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:16 [I 2022-10-09 15:34:53] ...trap/engines/engine.py.229: process: 0.85485 | load: 0.00297 [I 2022-10-09 15:34:53] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-09 15:34:53] ...optimizers/trijoint.py.151: total_norm: 28.639877 [I 2022-10-09 15:35:01] ...trap/engines/engine.py.224: train: epoch 45 | batch 1660/2304 [I 2022-10-09 15:35:01] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:03 [I 2022-10-09 15:35:01] ...trap/engines/engine.py.229: process: 0.83871 | load: 0.00296 [I 2022-10-09 15:35:01] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-09 15:35:01] ...optimizers/trijoint.py.151: total_norm: 13.495085 [I 2022-10-09 15:35:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 1670/2304 [I 2022-10-09 15:35:10] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:53 [I 2022-10-09 15:35:10] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00304 [I 2022-10-09 15:35:10] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-09 15:35:10] ...optimizers/trijoint.py.151: total_norm: 14.907804 [I 2022-10-09 15:35:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 1680/2304 [I 2022-10-09 15:35:18] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:45 [I 2022-10-09 15:35:18] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00291 [I 2022-10-09 15:35:18] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-09 15:35:18] ...optimizers/trijoint.py.151: total_norm: 16.029119 [I 2022-10-09 15:35:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 1690/2304 [I 2022-10-09 15:35:26] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:37 [I 2022-10-09 15:35:26] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00294 [I 2022-10-09 15:35:26] ...trap/engines/engine.py.230: loss: 0.14894 [I 2022-10-09 15:35:26] ...optimizers/trijoint.py.151: total_norm: 24.707361 [I 2022-10-09 15:35:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 1700/2304 [I 2022-10-09 15:35:35] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:27 [I 2022-10-09 15:35:35] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00293 [I 2022-10-09 15:35:35] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-09 15:35:35] ...optimizers/trijoint.py.151: total_norm: 17.714975 [I 2022-10-09 15:35:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 1710/2304 [I 2022-10-09 15:35:43] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:18 [I 2022-10-09 15:35:43] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00294 [I 2022-10-09 15:35:43] ...trap/engines/engine.py.230: loss: 0.20076 [I 2022-10-09 15:35:43] ...optimizers/trijoint.py.151: total_norm: 26.128027 [I 2022-10-09 15:35:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 1720/2304 [I 2022-10-09 15:35:52] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:12 [I 2022-10-09 15:35:52] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00293 [I 2022-10-09 15:35:52] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-09 15:35:52] ...optimizers/trijoint.py.151: total_norm: 17.802324 [I 2022-10-09 15:36:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 1730/2304 [I 2022-10-09 15:36:00] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:04 [I 2022-10-09 15:36:00] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00298 [I 2022-10-09 15:36:00] ...trap/engines/engine.py.230: loss: 0.15667 [I 2022-10-09 15:36:00] ...optimizers/trijoint.py.151: total_norm: 16.959776 [I 2022-10-09 15:36:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 1740/2304 [I 2022-10-09 15:36:09] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:55 [I 2022-10-09 15:36:09] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00306 [I 2022-10-09 15:36:09] ...trap/engines/engine.py.230: loss: 0.18373 [I 2022-10-09 15:36:09] ...optimizers/trijoint.py.151: total_norm: 23.578434 [I 2022-10-09 15:36:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 1750/2304 [I 2022-10-09 15:36:17] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:45 [I 2022-10-09 15:36:17] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00295 [I 2022-10-09 15:36:17] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-09 15:36:17] ...optimizers/trijoint.py.151: total_norm: 15.766530 [I 2022-10-09 15:36:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 1760/2304 [I 2022-10-09 15:36:25] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:37 [I 2022-10-09 15:36:26] ...trap/engines/engine.py.229: process: 0.82966 | load: 0.00297 [I 2022-10-09 15:36:26] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-09 15:36:26] ...optimizers/trijoint.py.151: total_norm: 11.621515 [I 2022-10-09 15:36:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 1770/2304 [I 2022-10-09 15:36:34] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:29 [I 2022-10-09 15:36:34] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00301 [I 2022-10-09 15:36:34] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-09 15:36:34] ...optimizers/trijoint.py.151: total_norm: 13.359276 [I 2022-10-09 15:36:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 1780/2304 [I 2022-10-09 15:36:42] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:22 [I 2022-10-09 15:36:42] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00295 [I 2022-10-09 15:36:42] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-09 15:36:42] ...optimizers/trijoint.py.151: total_norm: 24.831108 [I 2022-10-09 15:36:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 1790/2304 [I 2022-10-09 15:36:51] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:15 [I 2022-10-09 15:36:51] ...trap/engines/engine.py.229: process: 0.83173 | load: 0.00296 [I 2022-10-09 15:36:51] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-09 15:36:51] ...optimizers/trijoint.py.151: total_norm: 9.992759 [I 2022-10-09 15:36:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 1800/2304 [I 2022-10-09 15:36:59] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:05 [I 2022-10-09 15:36:59] ...trap/engines/engine.py.229: process: 0.84580 | load: 0.00300 [I 2022-10-09 15:36:59] ...trap/engines/engine.py.230: loss: 0.12073 [I 2022-10-09 15:36:59] ...optimizers/trijoint.py.151: total_norm: 8.655767 [I 2022-10-09 15:37:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 1810/2304 [I 2022-10-09 15:37:08] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:57 [I 2022-10-09 15:37:08] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00293 [I 2022-10-09 15:37:08] ...trap/engines/engine.py.230: loss: 0.20402 [I 2022-10-09 15:37:08] ...optimizers/trijoint.py.151: total_norm: 46.900719 [I 2022-10-09 15:37:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 1820/2304 [I 2022-10-09 15:37:16] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:48 [I 2022-10-09 15:37:16] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00293 [I 2022-10-09 15:37:16] ...trap/engines/engine.py.230: loss: 0.13303 [I 2022-10-09 15:37:16] ...optimizers/trijoint.py.151: total_norm: 16.529869 [I 2022-10-09 15:37:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 1830/2304 [I 2022-10-09 15:37:25] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:40 [I 2022-10-09 15:37:25] ...trap/engines/engine.py.229: process: 0.84231 | load: 0.00294 [I 2022-10-09 15:37:25] ...trap/engines/engine.py.230: loss: 0.21441 [I 2022-10-09 15:37:25] ...optimizers/trijoint.py.151: total_norm: 223.785736 [I 2022-10-09 15:37:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 1840/2304 [I 2022-10-09 15:37:33] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:32 [I 2022-10-09 15:37:33] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00298 [I 2022-10-09 15:37:33] ...trap/engines/engine.py.230: loss: 0.17698 [I 2022-10-09 15:37:33] ...optimizers/trijoint.py.151: total_norm: 18.800474 [I 2022-10-09 15:37:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 1850/2304 [I 2022-10-09 15:37:42] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:23 [I 2022-10-09 15:37:42] ...trap/engines/engine.py.229: process: 0.82950 | load: 0.00290 [I 2022-10-09 15:37:42] ...trap/engines/engine.py.230: loss: 0.14580 [I 2022-10-09 15:37:42] ...optimizers/trijoint.py.151: total_norm: 15.859139 [I 2022-10-09 15:37:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 1860/2304 [I 2022-10-09 15:37:50] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:16 [I 2022-10-09 15:37:50] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00294 [I 2022-10-09 15:37:50] ...trap/engines/engine.py.230: loss: 0.20163 [I 2022-10-09 15:37:50] ...optimizers/trijoint.py.151: total_norm: 30.044155 [I 2022-10-09 15:37:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 1870/2304 [I 2022-10-09 15:37:59] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:08 [I 2022-10-09 15:37:59] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00295 [I 2022-10-09 15:37:59] ...trap/engines/engine.py.230: loss: 0.15839 [I 2022-10-09 15:37:59] ...optimizers/trijoint.py.151: total_norm: 16.111101 [I 2022-10-09 15:38:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 1880/2304 [I 2022-10-09 15:38:07] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:59 [I 2022-10-09 15:38:07] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00299 [I 2022-10-09 15:38:07] ...trap/engines/engine.py.230: loss: 0.18797 [I 2022-10-09 15:38:07] ...optimizers/trijoint.py.151: total_norm: 42.668560 [I 2022-10-09 15:38:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 1890/2304 [I 2022-10-09 15:38:16] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:47 [I 2022-10-09 15:38:16] ...trap/engines/engine.py.229: process: 0.82291 | load: 0.00294 [I 2022-10-09 15:38:16] ...trap/engines/engine.py.230: loss: 0.12601 [I 2022-10-09 15:38:16] ...optimizers/trijoint.py.151: total_norm: 8.479284 [I 2022-10-09 15:38:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 1900/2304 [I 2022-10-09 15:38:24] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:42 [I 2022-10-09 15:38:24] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00295 [I 2022-10-09 15:38:24] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-09 15:38:24] ...optimizers/trijoint.py.151: total_norm: 15.178398 [I 2022-10-09 15:38:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 1910/2304 [I 2022-10-09 15:38:33] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:32 [I 2022-10-09 15:38:33] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00296 [I 2022-10-09 15:38:33] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-09 15:38:33] ...optimizers/trijoint.py.151: total_norm: 18.860352 [I 2022-10-09 15:38:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 1920/2304 [I 2022-10-09 15:38:41] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:24 [I 2022-10-09 15:38:41] ...trap/engines/engine.py.229: process: 0.82603 | load: 0.00300 [I 2022-10-09 15:38:41] ...trap/engines/engine.py.230: loss: 0.18098 [I 2022-10-09 15:38:41] ...optimizers/trijoint.py.151: total_norm: 27.553391 [I 2022-10-09 15:38:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 1930/2304 [I 2022-10-09 15:38:50] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:16 [I 2022-10-09 15:38:50] ...trap/engines/engine.py.229: process: 0.82183 | load: 0.00296 [I 2022-10-09 15:38:50] ...trap/engines/engine.py.230: loss: 0.12278 [I 2022-10-09 15:38:50] ...optimizers/trijoint.py.151: total_norm: 11.988626 [I 2022-10-09 15:38:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 1940/2304 [I 2022-10-09 15:38:58] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:08 [I 2022-10-09 15:38:58] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00293 [I 2022-10-09 15:38:58] ...trap/engines/engine.py.230: loss: 0.21983 [I 2022-10-09 15:38:58] ...optimizers/trijoint.py.151: total_norm: 21.173006 [I 2022-10-09 15:39:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 1950/2304 [I 2022-10-09 15:39:06] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:56 [I 2022-10-09 15:39:06] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00295 [I 2022-10-09 15:39:06] ...trap/engines/engine.py.230: loss: 0.14028 [I 2022-10-09 15:39:06] ...optimizers/trijoint.py.151: total_norm: 14.505645 [I 2022-10-09 15:39:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 1960/2304 [I 2022-10-09 15:39:15] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:49 [I 2022-10-09 15:39:15] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00297 [I 2022-10-09 15:39:15] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-09 15:39:15] ...optimizers/trijoint.py.151: total_norm: 11.782538 [I 2022-10-09 15:39:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 1970/2304 [I 2022-10-09 15:39:23] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:43 [I 2022-10-09 15:39:23] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00295 [I 2022-10-09 15:39:23] ...trap/engines/engine.py.230: loss: 0.16281 [I 2022-10-09 15:39:23] ...optimizers/trijoint.py.151: total_norm: 20.919619 [I 2022-10-09 15:39:32] ...trap/engines/engine.py.224: train: epoch 45 | batch 1980/2304 [I 2022-10-09 15:39:32] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:32 [I 2022-10-09 15:39:32] ...trap/engines/engine.py.229: process: 0.83203 | load: 0.00297 [I 2022-10-09 15:39:32] ...trap/engines/engine.py.230: loss: 0.17462 [I 2022-10-09 15:39:32] ...optimizers/trijoint.py.151: total_norm: 25.640553 [I 2022-10-09 15:39:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 1990/2304 [I 2022-10-09 15:39:40] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:26 [I 2022-10-09 15:39:40] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00312 [I 2022-10-09 15:39:40] ...trap/engines/engine.py.230: loss: 0.23486 [I 2022-10-09 15:39:40] ...optimizers/trijoint.py.151: total_norm: 22.254036 [I 2022-10-09 15:39:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 2000/2304 [I 2022-10-09 15:39:49] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:16 [I 2022-10-09 15:39:49] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00298 [I 2022-10-09 15:39:49] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-09 15:39:49] ...optimizers/trijoint.py.151: total_norm: 17.155365 [I 2022-10-09 15:39:57] ...trap/engines/engine.py.224: train: epoch 45 | batch 2010/2304 [I 2022-10-09 15:39:57] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:09 [I 2022-10-09 15:39:57] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00294 [I 2022-10-09 15:39:57] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-09 15:39:57] ...optimizers/trijoint.py.151: total_norm: 10.637875 [I 2022-10-09 15:40:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 2020/2304 [I 2022-10-09 15:40:06] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:04:01 [I 2022-10-09 15:40:06] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00293 [I 2022-10-09 15:40:06] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-09 15:40:06] ...optimizers/trijoint.py.151: total_norm: 22.316551 [I 2022-10-09 15:40:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 2030/2304 [I 2022-10-09 15:40:14] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:03:50 [I 2022-10-09 15:40:14] ...trap/engines/engine.py.229: process: 0.84229 | load: 0.00295 [I 2022-10-09 15:40:14] ...trap/engines/engine.py.230: loss: 0.14125 [I 2022-10-09 15:40:14] ...optimizers/trijoint.py.151: total_norm: 22.883211 [I 2022-10-09 15:40:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 2040/2304 [I 2022-10-09 15:40:23] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:42 [I 2022-10-09 15:40:23] ...trap/engines/engine.py.229: process: 0.83376 | load: 0.00298 [I 2022-10-09 15:40:23] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-09 15:40:23] ...optimizers/trijoint.py.151: total_norm: 6.847600 [I 2022-10-09 15:40:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 2050/2304 [I 2022-10-09 15:40:31] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:35 [I 2022-10-09 15:40:31] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00299 [I 2022-10-09 15:40:31] ...trap/engines/engine.py.230: loss: 0.16721 [I 2022-10-09 15:40:31] ...optimizers/trijoint.py.151: total_norm: 38.345127 [I 2022-10-09 15:40:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 2060/2304 [I 2022-10-09 15:40:40] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:25 [I 2022-10-09 15:40:40] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00297 [I 2022-10-09 15:40:40] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-09 15:40:40] ...optimizers/trijoint.py.151: total_norm: 20.844269 [I 2022-10-09 15:40:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 2070/2304 [I 2022-10-09 15:40:48] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:16 [I 2022-10-09 15:40:48] ...trap/engines/engine.py.229: process: 0.83321 | load: 0.00296 [I 2022-10-09 15:40:48] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-09 15:40:48] ...optimizers/trijoint.py.151: total_norm: 18.185452 [I 2022-10-09 15:40:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 2080/2304 [I 2022-10-09 15:40:56] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:07 [I 2022-10-09 15:40:56] ...trap/engines/engine.py.229: process: 0.82633 | load: 0.00301 [I 2022-10-09 15:40:56] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-09 15:40:56] ...optimizers/trijoint.py.151: total_norm: 18.724901 [I 2022-10-09 15:41:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 2090/2304 [I 2022-10-09 15:41:05] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:00 [I 2022-10-09 15:41:05] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00295 [I 2022-10-09 15:41:05] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-09 15:41:05] ...optimizers/trijoint.py.151: total_norm: 15.871391 [I 2022-10-09 15:41:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 2100/2304 [I 2022-10-09 15:41:13] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:53 [I 2022-10-09 15:41:13] ...trap/engines/engine.py.229: process: 0.83600 | load: 0.00295 [I 2022-10-09 15:41:13] ...trap/engines/engine.py.230: loss: 0.17232 [I 2022-10-09 15:41:13] ...optimizers/trijoint.py.151: total_norm: 45.201485 [I 2022-10-09 15:41:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 2110/2304 [I 2022-10-09 15:41:22] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:45 [I 2022-10-09 15:41:22] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00296 [I 2022-10-09 15:41:22] ...trap/engines/engine.py.230: loss: 0.14696 [I 2022-10-09 15:41:22] ...optimizers/trijoint.py.151: total_norm: 16.425793 [I 2022-10-09 15:41:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 2120/2304 [I 2022-10-09 15:41:30] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:36 [I 2022-10-09 15:41:30] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00300 [I 2022-10-09 15:41:30] ...trap/engines/engine.py.230: loss: 0.22169 [I 2022-10-09 15:41:30] ...optimizers/trijoint.py.151: total_norm: 17.144327 [I 2022-10-09 15:41:39] ...trap/engines/engine.py.224: train: epoch 45 | batch 2130/2304 [I 2022-10-09 15:41:39] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:28 [I 2022-10-09 15:41:39] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00295 [I 2022-10-09 15:41:39] ...trap/engines/engine.py.230: loss: 0.15858 [I 2022-10-09 15:41:39] ...optimizers/trijoint.py.151: total_norm: 19.601099 [I 2022-10-09 15:41:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 2140/2304 [I 2022-10-09 15:41:47] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:18 [I 2022-10-09 15:41:47] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00297 [I 2022-10-09 15:41:47] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-09 15:41:47] ...optimizers/trijoint.py.151: total_norm: 46.721043 [I 2022-10-09 15:41:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 2150/2304 [I 2022-10-09 15:41:56] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:11 [I 2022-10-09 15:41:56] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00296 [I 2022-10-09 15:41:56] ...trap/engines/engine.py.230: loss: 0.14731 [I 2022-10-09 15:41:56] ...optimizers/trijoint.py.151: total_norm: 12.776706 [I 2022-10-09 15:42:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 2160/2304 [I 2022-10-09 15:42:04] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:01 [I 2022-10-09 15:42:04] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00295 [I 2022-10-09 15:42:04] ...trap/engines/engine.py.230: loss: 0.18740 [I 2022-10-09 15:42:04] ...optimizers/trijoint.py.151: total_norm: 21.506588 [I 2022-10-09 15:42:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 2170/2304 [I 2022-10-09 15:42:13] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:53 [I 2022-10-09 15:42:13] ...trap/engines/engine.py.229: process: 0.84908 | load: 0.00296 [I 2022-10-09 15:42:13] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-09 15:42:13] ...optimizers/trijoint.py.151: total_norm: 12.557850 [I 2022-10-09 15:42:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 2180/2304 [I 2022-10-09 15:42:21] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:45 [I 2022-10-09 15:42:21] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00299 [I 2022-10-09 15:42:21] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-09 15:42:21] ...optimizers/trijoint.py.151: total_norm: 15.329185 [I 2022-10-09 15:42:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 2190/2304 [I 2022-10-09 15:42:30] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:36 [I 2022-10-09 15:42:30] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00296 [I 2022-10-09 15:42:30] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-09 15:42:30] ...optimizers/trijoint.py.151: total_norm: 8.457120 [I 2022-10-09 15:42:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 2200/2304 [I 2022-10-09 15:42:38] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:27 [I 2022-10-09 15:42:38] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00298 [I 2022-10-09 15:42:38] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-09 15:42:38] ...optimizers/trijoint.py.151: total_norm: 11.956122 [I 2022-10-09 15:42:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 2210/2304 [I 2022-10-09 15:42:47] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:19 [I 2022-10-09 15:42:47] ...trap/engines/engine.py.229: process: 0.86328 | load: 0.00298 [I 2022-10-09 15:42:47] ...trap/engines/engine.py.230: loss: 0.19451 [I 2022-10-09 15:42:47] ...optimizers/trijoint.py.151: total_norm: 33.211922 [I 2022-10-09 15:42:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 2220/2304 [I 2022-10-09 15:42:55] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:11 [I 2022-10-09 15:42:55] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00297 [I 2022-10-09 15:42:55] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-09 15:42:55] ...optimizers/trijoint.py.151: total_norm: 16.596889 [I 2022-10-09 15:43:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 2230/2304 [I 2022-10-09 15:43:04] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:02 [I 2022-10-09 15:43:04] ...trap/engines/engine.py.229: process: 0.85337 | load: 0.00300 [I 2022-10-09 15:43:04] ...trap/engines/engine.py.230: loss: 0.15647 [I 2022-10-09 15:43:04] ...optimizers/trijoint.py.151: total_norm: 57.103451 [I 2022-10-09 15:43:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 2240/2304 [I 2022-10-09 15:43:12] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:54 [I 2022-10-09 15:43:12] ...trap/engines/engine.py.229: process: 0.83527 | load: 0.00303 [I 2022-10-09 15:43:12] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-09 15:43:12] ...optimizers/trijoint.py.151: total_norm: 11.433006 [I 2022-10-09 15:43:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 2250/2304 [I 2022-10-09 15:43:21] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:45 [I 2022-10-09 15:43:21] ...trap/engines/engine.py.229: process: 0.84443 | load: 0.00300 [I 2022-10-09 15:43:21] ...trap/engines/engine.py.230: loss: 0.18397 [I 2022-10-09 15:43:21] ...optimizers/trijoint.py.151: total_norm: 28.488380 [I 2022-10-09 15:43:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 2260/2304 [I 2022-10-09 15:43:29] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:37 [I 2022-10-09 15:43:29] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00300 [I 2022-10-09 15:43:29] ...trap/engines/engine.py.230: loss: 0.10981 [I 2022-10-09 15:43:29] ...optimizers/trijoint.py.151: total_norm: 7.711719 [I 2022-10-09 15:43:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 2270/2304 [I 2022-10-09 15:43:38] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:28 [I 2022-10-09 15:43:38] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00300 [I 2022-10-09 15:43:38] ...trap/engines/engine.py.230: loss: 0.19370 [I 2022-10-09 15:43:38] ...optimizers/trijoint.py.151: total_norm: 14.980921 [I 2022-10-09 15:43:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 2280/2304 [I 2022-10-09 15:43:46] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:20 [I 2022-10-09 15:43:46] ...trap/engines/engine.py.229: process: 0.83560 | load: 0.00303 [I 2022-10-09 15:43:46] ...trap/engines/engine.py.230: loss: 0.19178 [I 2022-10-09 15:43:46] ...optimizers/trijoint.py.151: total_norm: 18.186092 [I 2022-10-09 15:43:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 2290/2304 [I 2022-10-09 15:43:55] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:11 [I 2022-10-09 15:43:55] ...trap/engines/engine.py.229: process: 0.85648 | load: 0.00306 [I 2022-10-09 15:43:55] ...trap/engines/engine.py.230: loss: 0.16623 [I 2022-10-09 15:43:55] ...optimizers/trijoint.py.151: total_norm: 12.584684 [I 2022-10-09 15:44:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 2300/2304 [I 2022-10-09 15:44:04] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:03 [I 2022-10-09 15:44:04] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00017 [I 2022-10-09 15:44:04] ...trap/engines/engine.py.230: loss: 0.32189 [I 2022-10-09 15:44:04] ...optimizers/trijoint.py.151: total_norm: 33.252632 [I 2022-10-09 15:44:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 2304/2304 [I 2022-10-09 15:44:07] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:00 [I 2022-10-09 15:44:07] ...trap/engines/engine.py.229: process: 0.82475 | load: 0.00018 [I 2022-10-09 15:44:07] ...trap/engines/engine.py.230: loss: 0.11703 [I 2022-10-09 15:44:07] ...optimizers/trijoint.py.151: total_norm: 10.233543 [S 2022-10-09 15:44:08] ...trap/engines/engine.py.129: train_epoch.epoch: 45 [S 2022-10-09 15:44:08] ...trap/engines/engine.py.129: train_epoch.loss: 0.15732256973449166 [I 2022-10-09 15:44:08] ...s/criterions/triplet.py.84: margin = 0.2750000000000001 [I 2022-10-09 15:44:09] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 45 [I 2022-10-09 15:44:09] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 15:44:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 15:44:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 15:44:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 15:44:12] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 15:44:12] ...trap/engines/engine.py.321: eval: epoch 45 | batch 0/510 [I 2022-10-09 15:44:12] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:57 [I 2022-10-09 15:44:12] ...trap/engines/engine.py.326: process: 3.52498 | load: 2.99863 [I 2022-10-09 15:44:17] ...trap/engines/engine.py.321: eval: epoch 45 | batch 10/510 [I 2022-10-09 15:44:17] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:16 [I 2022-10-09 15:44:17] ...trap/engines/engine.py.326: process: 0.42212 | load: 0.00018 [I 2022-10-09 15:44:21] ...trap/engines/engine.py.321: eval: epoch 45 | batch 20/510 [I 2022-10-09 15:44:21] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:24 [I 2022-10-09 15:44:21] ...trap/engines/engine.py.326: process: 0.60977 | load: 0.00019 [I 2022-10-09 15:44:27] ...trap/engines/engine.py.321: eval: epoch 45 | batch 30/510 [I 2022-10-09 15:44:27] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:04:10 [I 2022-10-09 15:44:27] ...trap/engines/engine.py.326: process: 0.48353 | load: 0.00018 [I 2022-10-09 15:44:32] ...trap/engines/engine.py.321: eval: epoch 45 | batch 40/510 [I 2022-10-09 15:44:32] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:04:22 [I 2022-10-09 15:44:32] ...trap/engines/engine.py.326: process: 0.54745 | load: 0.00020 [I 2022-10-09 15:44:36] ...trap/engines/engine.py.321: eval: epoch 45 | batch 50/510 [I 2022-10-09 15:44:36] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:03:30 [I 2022-10-09 15:44:36] ...trap/engines/engine.py.326: process: 0.45439 | load: 0.00018 [I 2022-10-09 15:44:41] ...trap/engines/engine.py.321: eval: epoch 45 | batch 60/510 [I 2022-10-09 15:44:41] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:21 [I 2022-10-09 15:44:41] ...trap/engines/engine.py.326: process: 0.44998 | load: 0.00019 [I 2022-10-09 15:44:45] ...trap/engines/engine.py.321: eval: epoch 45 | batch 70/510 [I 2022-10-09 15:44:45] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:07 [I 2022-10-09 15:44:45] ...trap/engines/engine.py.326: process: 0.41478 | load: 0.00019 [I 2022-10-09 15:44:49] ...trap/engines/engine.py.321: eval: epoch 45 | batch 80/510 [I 2022-10-09 15:44:49] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:09 [I 2022-10-09 15:44:49] ...trap/engines/engine.py.326: process: 0.44858 | load: 0.00018 [I 2022-10-09 15:44:54] ...trap/engines/engine.py.321: eval: epoch 45 | batch 90/510 [I 2022-10-09 15:44:54] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:02:57 [I 2022-10-09 15:44:54] ...trap/engines/engine.py.326: process: 0.41508 | load: 0.00017 [I 2022-10-09 15:44:58] ...trap/engines/engine.py.321: eval: epoch 45 | batch 100/510 [I 2022-10-09 15:44:58] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:02:45 [I 2022-10-09 15:44:58] ...trap/engines/engine.py.326: process: 0.31280 | load: 0.00019 [I 2022-10-09 15:45:01] ...trap/engines/engine.py.321: eval: epoch 45 | batch 110/510 [I 2022-10-09 15:45:01] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:10 [I 2022-10-09 15:45:01] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00018 [I 2022-10-09 15:45:04] ...trap/engines/engine.py.321: eval: epoch 45 | batch 120/510 [I 2022-10-09 15:45:04] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:04 [I 2022-10-09 15:45:04] ...trap/engines/engine.py.326: process: 0.31507 | load: 0.00019 [I 2022-10-09 15:45:07] ...trap/engines/engine.py.321: eval: epoch 45 | batch 130/510 [I 2022-10-09 15:45:07] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:01 [I 2022-10-09 15:45:07] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00027 [I 2022-10-09 15:45:11] ...trap/engines/engine.py.321: eval: epoch 45 | batch 140/510 [I 2022-10-09 15:45:11] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:57 [I 2022-10-09 15:45:11] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00019 [I 2022-10-09 15:45:14] ...trap/engines/engine.py.321: eval: epoch 45 | batch 150/510 [I 2022-10-09 15:45:14] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:55 [I 2022-10-09 15:45:14] ...trap/engines/engine.py.326: process: 0.32385 | load: 0.00021 [I 2022-10-09 15:45:17] ...trap/engines/engine.py.321: eval: epoch 45 | batch 160/510 [I 2022-10-09 15:45:17] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:50 [I 2022-10-09 15:45:17] ...trap/engines/engine.py.326: process: 0.32244 | load: 0.00019 [I 2022-10-09 15:45:20] ...trap/engines/engine.py.321: eval: epoch 45 | batch 170/510 [I 2022-10-09 15:45:20] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:48 [I 2022-10-09 15:45:20] ...trap/engines/engine.py.326: process: 0.32001 | load: 0.00020 [I 2022-10-09 15:45:23] ...trap/engines/engine.py.321: eval: epoch 45 | batch 180/510 [I 2022-10-09 15:45:23] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:44 [I 2022-10-09 15:45:23] ...trap/engines/engine.py.326: process: 0.31682 | load: 0.00019 [I 2022-10-09 15:45:26] ...trap/engines/engine.py.321: eval: epoch 45 | batch 190/510 [I 2022-10-09 15:45:26] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:41 [I 2022-10-09 15:45:26] ...trap/engines/engine.py.326: process: 0.32575 | load: 0.00018 [I 2022-10-09 15:45:30] ...trap/engines/engine.py.321: eval: epoch 45 | batch 200/510 [I 2022-10-09 15:45:30] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:38 [I 2022-10-09 15:45:30] ...trap/engines/engine.py.326: process: 0.31960 | load: 0.00020 [I 2022-10-09 15:45:33] ...trap/engines/engine.py.321: eval: epoch 45 | batch 210/510 [I 2022-10-09 15:45:33] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:35 [I 2022-10-09 15:45:33] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00019 [I 2022-10-09 15:45:36] ...trap/engines/engine.py.321: eval: epoch 45 | batch 220/510 [I 2022-10-09 15:45:36] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:31 [I 2022-10-09 15:45:36] ...trap/engines/engine.py.326: process: 0.31164 | load: 0.00019 [I 2022-10-09 15:45:39] ...trap/engines/engine.py.321: eval: epoch 45 | batch 230/510 [I 2022-10-09 15:45:39] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:28 [I 2022-10-09 15:45:39] ...trap/engines/engine.py.326: process: 0.31796 | load: 0.00020 [I 2022-10-09 15:45:42] ...trap/engines/engine.py.321: eval: epoch 45 | batch 240/510 [I 2022-10-09 15:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:25 [I 2022-10-09 15:45:42] ...trap/engines/engine.py.326: process: 0.31996 | load: 0.00020 [I 2022-10-09 15:45:45] ...trap/engines/engine.py.321: eval: epoch 45 | batch 250/510 [I 2022-10-09 15:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:22 [I 2022-10-09 15:45:45] ...trap/engines/engine.py.326: process: 0.32147 | load: 0.00020 [I 2022-10-09 15:45:49] ...trap/engines/engine.py.321: eval: epoch 45 | batch 260/510 [I 2022-10-09 15:45:49] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:19 [I 2022-10-09 15:45:49] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00020 [I 2022-10-09 15:45:52] ...trap/engines/engine.py.321: eval: epoch 45 | batch 270/510 [I 2022-10-09 15:45:52] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:15 [I 2022-10-09 15:45:52] ...trap/engines/engine.py.326: process: 0.31604 | load: 0.00020 [I 2022-10-09 15:45:55] ...trap/engines/engine.py.321: eval: epoch 45 | batch 280/510 [I 2022-10-09 15:45:55] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:13 [I 2022-10-09 15:45:55] ...trap/engines/engine.py.326: process: 0.31933 | load: 0.00021 [I 2022-10-09 15:45:58] ...trap/engines/engine.py.321: eval: epoch 45 | batch 290/510 [I 2022-10-09 15:45:58] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:09 [I 2022-10-09 15:45:58] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00019 [I 2022-10-09 15:46:01] ...trap/engines/engine.py.321: eval: epoch 45 | batch 300/510 [I 2022-10-09 15:46:01] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:06 [I 2022-10-09 15:46:01] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00019 [I 2022-10-09 15:46:05] ...trap/engines/engine.py.321: eval: epoch 45 | batch 310/510 [I 2022-10-09 15:46:05] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:03 [I 2022-10-09 15:46:05] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00021 [I 2022-10-09 15:46:08] ...trap/engines/engine.py.321: eval: epoch 45 | batch 320/510 [I 2022-10-09 15:46:08] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:00 [I 2022-10-09 15:46:08] ...trap/engines/engine.py.326: process: 0.31350 | load: 0.00020 [I 2022-10-09 15:46:11] ...trap/engines/engine.py.321: eval: epoch 45 | batch 330/510 [I 2022-10-09 15:46:11] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:57 [I 2022-10-09 15:46:11] ...trap/engines/engine.py.326: process: 0.31645 | load: 0.00021 [I 2022-10-09 15:46:14] ...trap/engines/engine.py.321: eval: epoch 45 | batch 340/510 [I 2022-10-09 15:46:14] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:54 [I 2022-10-09 15:46:14] ...trap/engines/engine.py.326: process: 0.32030 | load: 0.00020 [I 2022-10-09 15:46:17] ...trap/engines/engine.py.321: eval: epoch 45 | batch 350/510 [I 2022-10-09 15:46:17] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:51 [I 2022-10-09 15:46:17] ...trap/engines/engine.py.326: process: 0.32363 | load: 0.00020 [I 2022-10-09 15:46:21] ...trap/engines/engine.py.321: eval: epoch 45 | batch 360/510 [I 2022-10-09 15:46:21] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:47 [I 2022-10-09 15:46:21] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00022 [I 2022-10-09 15:46:24] ...trap/engines/engine.py.321: eval: epoch 45 | batch 370/510 [I 2022-10-09 15:46:24] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:44 [I 2022-10-09 15:46:24] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00021 [I 2022-10-09 15:46:27] ...trap/engines/engine.py.321: eval: epoch 45 | batch 380/510 [I 2022-10-09 15:46:27] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:41 [I 2022-10-09 15:46:27] ...trap/engines/engine.py.326: process: 0.32280 | load: 0.00021 [I 2022-10-09 15:46:30] ...trap/engines/engine.py.321: eval: epoch 45 | batch 390/510 [I 2022-10-09 15:46:30] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:38 [I 2022-10-09 15:46:30] ...trap/engines/engine.py.326: process: 0.31457 | load: 0.00020 [I 2022-10-09 15:46:33] ...trap/engines/engine.py.321: eval: epoch 45 | batch 400/510 [I 2022-10-09 15:46:33] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:35 [I 2022-10-09 15:46:33] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00021 [I 2022-10-09 15:46:36] ...trap/engines/engine.py.321: eval: epoch 45 | batch 410/510 [I 2022-10-09 15:46:36] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:31 [I 2022-10-09 15:46:36] ...trap/engines/engine.py.326: process: 0.31835 | load: 0.00019 [I 2022-10-09 15:46:40] ...trap/engines/engine.py.321: eval: epoch 45 | batch 420/510 [I 2022-10-09 15:46:40] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:28 [I 2022-10-09 15:46:40] ...trap/engines/engine.py.326: process: 0.32432 | load: 0.00021 [I 2022-10-09 15:46:43] ...trap/engines/engine.py.321: eval: epoch 45 | batch 430/510 [I 2022-10-09 15:46:43] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:25 [I 2022-10-09 15:46:43] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00025 [I 2022-10-09 15:46:46] ...trap/engines/engine.py.321: eval: epoch 45 | batch 440/510 [I 2022-10-09 15:46:46] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:22 [I 2022-10-09 15:46:46] ...trap/engines/engine.py.326: process: 0.31965 | load: 0.00019 [I 2022-10-09 15:46:49] ...trap/engines/engine.py.321: eval: epoch 45 | batch 450/510 [I 2022-10-09 15:46:49] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:19 [I 2022-10-09 15:46:49] ...trap/engines/engine.py.326: process: 0.31469 | load: 0.00021 [I 2022-10-09 15:46:52] ...trap/engines/engine.py.321: eval: epoch 45 | batch 460/510 [I 2022-10-09 15:46:52] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:15 [I 2022-10-09 15:46:52] ...trap/engines/engine.py.326: process: 0.31704 | load: 0.00020 [I 2022-10-09 15:46:56] ...trap/engines/engine.py.321: eval: epoch 45 | batch 470/510 [I 2022-10-09 15:46:56] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:12 [I 2022-10-09 15:46:56] ...trap/engines/engine.py.326: process: 0.31865 | load: 0.00019 [I 2022-10-09 15:46:59] ...trap/engines/engine.py.321: eval: epoch 45 | batch 480/510 [I 2022-10-09 15:46:59] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:09 [I 2022-10-09 15:46:59] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00021 [I 2022-10-09 15:47:02] ...trap/engines/engine.py.321: eval: epoch 45 | batch 490/510 [I 2022-10-09 15:47:02] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:06 [I 2022-10-09 15:47:02] ...trap/engines/engine.py.326: process: 0.32196 | load: 0.00020 [I 2022-10-09 15:47:05] ...trap/engines/engine.py.321: eval: epoch 45 | batch 500/510 [I 2022-10-09 15:47:05] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:03 [I 2022-10-09 15:47:05] ...trap/engines/engine.py.326: process: 0.31672 | load: 0.00019 [I 2022-10-09 15:47:08] ...trap/engines/engine.py.321: eval: epoch 45 | batch 510/510 [I 2022-10-09 15:47:08] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:00 [I 2022-10-09 15:47:08] ...trap/engines/engine.py.326: process: 0.31706 | load: 0.00016 [S 2022-10-09 15:47:09] ...trap/engines/engine.py.132: eval_epoch.epoch: 45 [I 2022-10-09 15:47:09] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.717600 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909700 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.942300 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.705100 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905000 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938600 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011612 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007785 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008100 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011476 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007785 [S 2022-10-09 15:47:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009102 [I 2022-10-09 15:47:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 15:47:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 15:47:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 15:47:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 15:47:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 15:47:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 15:47:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 15:47:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 15:47:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 46 [I 2022-10-09 15:47:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 15:47:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 0/2304 [I 2022-10-09 15:47:57] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:47 [I 2022-10-09 15:47:57] ...trap/engines/engine.py.229: process: 2.72887 | load: 1.80321 [I 2022-10-09 15:47:57] ...trap/engines/engine.py.230: loss: 0.20114 [I 2022-10-09 15:47:57] ...optimizers/trijoint.py.151: total_norm: 12.590463 [I 2022-10-09 15:48:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 10/2304 [I 2022-10-09 15:48:06] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:39:57 [I 2022-10-09 15:48:06] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00582 [I 2022-10-09 15:48:06] ...trap/engines/engine.py.230: loss: 0.18648 [I 2022-10-09 15:48:06] ...optimizers/trijoint.py.151: total_norm: 14.921669 [I 2022-10-09 15:48:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 20/2304 [I 2022-10-09 15:48:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:48 [I 2022-10-09 15:48:14] ...trap/engines/engine.py.229: process: 0.82800 | load: 0.00488 [I 2022-10-09 15:48:14] ...trap/engines/engine.py.230: loss: 0.14656 [I 2022-10-09 15:48:14] ...optimizers/trijoint.py.151: total_norm: 14.117738 [I 2022-10-09 15:48:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 30/2304 [I 2022-10-09 15:48:22] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:06 [I 2022-10-09 15:48:22] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00414 [I 2022-10-09 15:48:22] ...trap/engines/engine.py.230: loss: 0.16276 [I 2022-10-09 15:48:22] ...optimizers/trijoint.py.151: total_norm: 14.458816 [I 2022-10-09 15:48:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 40/2304 [I 2022-10-09 15:48:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:57 [I 2022-10-09 15:48:31] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00372 [I 2022-10-09 15:48:31] ...trap/engines/engine.py.230: loss: 0.15470 [I 2022-10-09 15:48:31] ...optimizers/trijoint.py.151: total_norm: 15.298096 [I 2022-10-09 15:48:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 50/2304 [I 2022-10-09 15:48:39] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:52 [I 2022-10-09 15:48:39] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00391 [I 2022-10-09 15:48:39] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-09 15:48:39] ...optimizers/trijoint.py.151: total_norm: 25.350506 [I 2022-10-09 15:48:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 60/2304 [I 2022-10-09 15:48:48] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:20 [I 2022-10-09 15:48:48] ...trap/engines/engine.py.229: process: 0.82306 | load: 0.00350 [I 2022-10-09 15:48:48] ...trap/engines/engine.py.230: loss: 0.13722 [I 2022-10-09 15:48:48] ...optimizers/trijoint.py.151: total_norm: 15.541886 [I 2022-10-09 15:48:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 70/2304 [I 2022-10-09 15:48:56] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:29 [I 2022-10-09 15:48:56] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00360 [I 2022-10-09 15:48:56] ...trap/engines/engine.py.230: loss: 0.22654 [I 2022-10-09 15:48:56] ...optimizers/trijoint.py.151: total_norm: 19.499393 [I 2022-10-09 15:49:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 80/2304 [I 2022-10-09 15:49:05] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:19 [I 2022-10-09 15:49:05] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00372 [I 2022-10-09 15:49:05] ...trap/engines/engine.py.230: loss: 0.13792 [I 2022-10-09 15:49:05] ...optimizers/trijoint.py.151: total_norm: 13.385786 [I 2022-10-09 15:49:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 90/2304 [I 2022-10-09 15:49:13] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:09 [I 2022-10-09 15:49:13] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00325 [I 2022-10-09 15:49:13] ...trap/engines/engine.py.230: loss: 0.12239 [I 2022-10-09 15:49:13] ...optimizers/trijoint.py.151: total_norm: 10.848595 [I 2022-10-09 15:49:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 100/2304 [I 2022-10-09 15:49:22] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:57 [I 2022-10-09 15:49:22] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00319 [I 2022-10-09 15:49:22] ...trap/engines/engine.py.230: loss: 0.21240 [I 2022-10-09 15:49:22] ...optimizers/trijoint.py.151: total_norm: 19.323013 [I 2022-10-09 15:49:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 110/2304 [I 2022-10-09 15:49:30] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:57 [I 2022-10-09 15:49:30] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00332 [I 2022-10-09 15:49:30] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-09 15:49:30] ...optimizers/trijoint.py.151: total_norm: 12.272647 [I 2022-10-09 15:49:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 120/2304 [I 2022-10-09 15:49:39] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:54 [I 2022-10-09 15:49:39] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00306 [I 2022-10-09 15:49:39] ...trap/engines/engine.py.230: loss: 0.21506 [I 2022-10-09 15:49:39] ...optimizers/trijoint.py.151: total_norm: 12.354667 [I 2022-10-09 15:49:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 130/2304 [I 2022-10-09 15:49:47] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:39 [I 2022-10-09 15:49:47] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00299 [I 2022-10-09 15:49:47] ...trap/engines/engine.py.230: loss: 0.09774 [I 2022-10-09 15:49:47] ...optimizers/trijoint.py.151: total_norm: 10.531008 [I 2022-10-09 15:49:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 140/2304 [I 2022-10-09 15:49:56] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:15 [I 2022-10-09 15:49:56] ...trap/engines/engine.py.229: process: 0.82339 | load: 0.00353 [I 2022-10-09 15:49:56] ...trap/engines/engine.py.230: loss: 0.14943 [I 2022-10-09 15:49:56] ...optimizers/trijoint.py.151: total_norm: 20.742496 [I 2022-10-09 15:50:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 150/2304 [I 2022-10-09 15:50:04] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:25 [I 2022-10-09 15:50:04] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00296 [I 2022-10-09 15:50:04] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-09 15:50:04] ...optimizers/trijoint.py.151: total_norm: 13.520910 [I 2022-10-09 15:50:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 160/2304 [I 2022-10-09 15:50:12] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:14 [I 2022-10-09 15:50:12] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00306 [I 2022-10-09 15:50:12] ...trap/engines/engine.py.230: loss: 0.15546 [I 2022-10-09 15:50:12] ...optimizers/trijoint.py.151: total_norm: 10.326838 [I 2022-10-09 15:50:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 170/2304 [I 2022-10-09 15:50:21] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:58 [I 2022-10-09 15:50:21] ...trap/engines/engine.py.229: process: 0.83464 | load: 0.00296 [I 2022-10-09 15:50:21] ...trap/engines/engine.py.230: loss: 0.13893 [I 2022-10-09 15:50:21] ...optimizers/trijoint.py.151: total_norm: 15.679780 [I 2022-10-09 15:50:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 180/2304 [I 2022-10-09 15:50:29] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:55 [I 2022-10-09 15:50:29] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00311 [I 2022-10-09 15:50:29] ...trap/engines/engine.py.230: loss: 0.14774 [I 2022-10-09 15:50:29] ...optimizers/trijoint.py.151: total_norm: 21.547939 [I 2022-10-09 15:50:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 190/2304 [I 2022-10-09 15:50:38] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:37 [I 2022-10-09 15:50:38] ...trap/engines/engine.py.229: process: 0.83030 | load: 0.00297 [I 2022-10-09 15:50:38] ...trap/engines/engine.py.230: loss: 0.16100 [I 2022-10-09 15:50:38] ...optimizers/trijoint.py.151: total_norm: 14.565033 [I 2022-10-09 15:50:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 200/2304 [I 2022-10-09 15:50:46] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:39 [I 2022-10-09 15:50:46] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00324 [I 2022-10-09 15:50:46] ...trap/engines/engine.py.230: loss: 0.19091 [I 2022-10-09 15:50:46] ...optimizers/trijoint.py.151: total_norm: 19.109064 [I 2022-10-09 15:50:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 210/2304 [I 2022-10-09 15:50:55] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:37 [I 2022-10-09 15:50:55] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00312 [I 2022-10-09 15:50:55] ...trap/engines/engine.py.230: loss: 0.09975 [I 2022-10-09 15:50:55] ...optimizers/trijoint.py.151: total_norm: 9.154358 [I 2022-10-09 15:51:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 220/2304 [I 2022-10-09 15:51:03] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:28 [I 2022-10-09 15:51:03] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00298 [I 2022-10-09 15:51:03] ...trap/engines/engine.py.230: loss: 0.13131 [I 2022-10-09 15:51:03] ...optimizers/trijoint.py.151: total_norm: 15.953691 [I 2022-10-09 15:51:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 230/2304 [I 2022-10-09 15:51:12] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:21 [I 2022-10-09 15:51:12] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00318 [I 2022-10-09 15:51:12] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-09 15:51:12] ...optimizers/trijoint.py.151: total_norm: 13.791709 [I 2022-10-09 15:51:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 240/2304 [I 2022-10-09 15:51:20] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:59 [I 2022-10-09 15:51:20] ...trap/engines/engine.py.229: process: 0.83086 | load: 0.00304 [I 2022-10-09 15:51:20] ...trap/engines/engine.py.230: loss: 0.18936 [I 2022-10-09 15:51:20] ...optimizers/trijoint.py.151: total_norm: 26.823441 [I 2022-10-09 15:51:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 250/2304 [I 2022-10-09 15:51:29] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:04 [I 2022-10-09 15:51:29] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00295 [I 2022-10-09 15:51:29] ...trap/engines/engine.py.230: loss: 0.15695 [I 2022-10-09 15:51:29] ...optimizers/trijoint.py.151: total_norm: 17.600183 [I 2022-10-09 15:51:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 260/2304 [I 2022-10-09 15:51:37] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:50 [I 2022-10-09 15:51:37] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00310 [I 2022-10-09 15:51:37] ...trap/engines/engine.py.230: loss: 0.17632 [I 2022-10-09 15:51:37] ...optimizers/trijoint.py.151: total_norm: 9.342675 [I 2022-10-09 15:51:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 270/2304 [I 2022-10-09 15:51:46] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:53 [I 2022-10-09 15:51:46] ...trap/engines/engine.py.229: process: 0.85469 | load: 0.00302 [I 2022-10-09 15:51:46] ...trap/engines/engine.py.230: loss: 0.13325 [I 2022-10-09 15:51:46] ...optimizers/trijoint.py.151: total_norm: 14.586694 [I 2022-10-09 15:51:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 280/2304 [I 2022-10-09 15:51:54] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:26 [I 2022-10-09 15:51:54] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00290 [I 2022-10-09 15:51:54] ...trap/engines/engine.py.230: loss: 0.18235 [I 2022-10-09 15:51:54] ...optimizers/trijoint.py.151: total_norm: 62.523159 [I 2022-10-09 15:52:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 290/2304 [I 2022-10-09 15:52:03] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:29 [I 2022-10-09 15:52:03] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00303 [I 2022-10-09 15:52:03] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-09 15:52:03] ...optimizers/trijoint.py.151: total_norm: 17.430635 [I 2022-10-09 15:52:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 300/2304 [I 2022-10-09 15:52:11] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:23 [I 2022-10-09 15:52:11] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00294 [I 2022-10-09 15:52:11] ...trap/engines/engine.py.230: loss: 0.19436 [I 2022-10-09 15:52:11] ...optimizers/trijoint.py.151: total_norm: 16.706961 [I 2022-10-09 15:52:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 310/2304 [I 2022-10-09 15:52:20] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:14 [I 2022-10-09 15:52:20] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00295 [I 2022-10-09 15:52:20] ...trap/engines/engine.py.230: loss: 0.12682 [I 2022-10-09 15:52:20] ...optimizers/trijoint.py.151: total_norm: 12.962866 [I 2022-10-09 15:52:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 320/2304 [I 2022-10-09 15:52:28] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:59 [I 2022-10-09 15:52:28] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00298 [I 2022-10-09 15:52:28] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-09 15:52:28] ...optimizers/trijoint.py.151: total_norm: 9.145760 [I 2022-10-09 15:52:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 330/2304 [I 2022-10-09 15:52:37] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:53 [I 2022-10-09 15:52:37] ...trap/engines/engine.py.229: process: 0.83541 | load: 0.00312 [I 2022-10-09 15:52:37] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-09 15:52:37] ...optimizers/trijoint.py.151: total_norm: 15.074846 [I 2022-10-09 15:52:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 340/2304 [I 2022-10-09 15:52:45] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:42 [I 2022-10-09 15:52:45] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00302 [I 2022-10-09 15:52:45] ...trap/engines/engine.py.230: loss: 0.26121 [I 2022-10-09 15:52:45] ...optimizers/trijoint.py.151: total_norm: 31.805235 [I 2022-10-09 15:52:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 350/2304 [I 2022-10-09 15:52:54] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:48 [I 2022-10-09 15:52:54] ...trap/engines/engine.py.229: process: 0.84825 | load: 0.00293 [I 2022-10-09 15:52:54] ...trap/engines/engine.py.230: loss: 0.15627 [I 2022-10-09 15:52:54] ...optimizers/trijoint.py.151: total_norm: 19.479057 [I 2022-10-09 15:53:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 360/2304 [I 2022-10-09 15:53:02] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:21 [I 2022-10-09 15:53:02] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00326 [I 2022-10-09 15:53:02] ...trap/engines/engine.py.230: loss: 0.21427 [I 2022-10-09 15:53:02] ...optimizers/trijoint.py.151: total_norm: 37.142948 [I 2022-10-09 15:53:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 370/2304 [I 2022-10-09 15:53:11] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:27:13 [I 2022-10-09 15:53:11] ...trap/engines/engine.py.229: process: 0.83869 | load: 0.00296 [I 2022-10-09 15:53:11] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-09 15:53:11] ...optimizers/trijoint.py.151: total_norm: 19.277636 [I 2022-10-09 15:53:19] ...trap/engines/engine.py.224: train: epoch 46 | batch 380/2304 [I 2022-10-09 15:53:19] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:04 [I 2022-10-09 15:53:19] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00300 [I 2022-10-09 15:53:19] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-09 15:53:19] ...optimizers/trijoint.py.151: total_norm: 9.821283 [I 2022-10-09 15:53:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 390/2304 [I 2022-10-09 15:53:27] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:26:51 [I 2022-10-09 15:53:27] ...trap/engines/engine.py.229: process: 0.82075 | load: 0.00296 [I 2022-10-09 15:53:27] ...trap/engines/engine.py.230: loss: 0.12220 [I 2022-10-09 15:53:27] ...optimizers/trijoint.py.151: total_norm: 6.951917 [I 2022-10-09 15:53:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 400/2304 [I 2022-10-09 15:53:36] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:44 [I 2022-10-09 15:53:36] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00304 [I 2022-10-09 15:53:36] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-09 15:53:36] ...optimizers/trijoint.py.151: total_norm: 39.677452 [I 2022-10-09 15:53:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 410/2304 [I 2022-10-09 15:53:44] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:36 [I 2022-10-09 15:53:44] ...trap/engines/engine.py.229: process: 0.84085 | load: 0.00296 [I 2022-10-09 15:53:44] ...trap/engines/engine.py.230: loss: 0.14986 [I 2022-10-09 15:53:44] ...optimizers/trijoint.py.151: total_norm: 12.640799 [I 2022-10-09 15:53:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 420/2304 [I 2022-10-09 15:53:53] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:26 [I 2022-10-09 15:53:53] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00306 [I 2022-10-09 15:53:53] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-09 15:53:53] ...optimizers/trijoint.py.151: total_norm: 8.199173 [I 2022-10-09 15:54:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 430/2304 [I 2022-10-09 15:54:01] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:19 [I 2022-10-09 15:54:01] ...trap/engines/engine.py.229: process: 0.84696 | load: 0.00297 [I 2022-10-09 15:54:01] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-09 15:54:01] ...optimizers/trijoint.py.151: total_norm: 14.190898 [I 2022-10-09 15:54:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 440/2304 [I 2022-10-09 15:54:10] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:18 [I 2022-10-09 15:54:10] ...trap/engines/engine.py.229: process: 0.83957 | load: 0.00300 [I 2022-10-09 15:54:10] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-09 15:54:10] ...optimizers/trijoint.py.151: total_norm: 21.667459 [I 2022-10-09 15:54:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 450/2304 [I 2022-10-09 15:54:18] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:21 [I 2022-10-09 15:54:18] ...trap/engines/engine.py.229: process: 0.87528 | load: 0.00297 [I 2022-10-09 15:54:18] ...trap/engines/engine.py.230: loss: 0.15383 [I 2022-10-09 15:54:18] ...optimizers/trijoint.py.151: total_norm: 16.151121 [I 2022-10-09 15:54:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 460/2304 [I 2022-10-09 15:54:27] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:25:55 [I 2022-10-09 15:54:27] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00295 [I 2022-10-09 15:54:27] ...trap/engines/engine.py.230: loss: 0.11752 [I 2022-10-09 15:54:27] ...optimizers/trijoint.py.151: total_norm: 10.686211 [I 2022-10-09 15:54:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 470/2304 [I 2022-10-09 15:54:35] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:44 [I 2022-10-09 15:54:35] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00301 [I 2022-10-09 15:54:35] ...trap/engines/engine.py.230: loss: 0.16258 [I 2022-10-09 15:54:35] ...optimizers/trijoint.py.151: total_norm: 20.933012 [I 2022-10-09 15:54:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 480/2304 [I 2022-10-09 15:54:44] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:25:29 [I 2022-10-09 15:54:44] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00298 [I 2022-10-09 15:54:44] ...trap/engines/engine.py.230: loss: 0.23469 [I 2022-10-09 15:54:44] ...optimizers/trijoint.py.151: total_norm: 27.248495 [I 2022-10-09 15:54:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 490/2304 [I 2022-10-09 15:54:52] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:39 [I 2022-10-09 15:54:52] ...trap/engines/engine.py.229: process: 0.83915 | load: 0.00295 [I 2022-10-09 15:54:52] ...trap/engines/engine.py.230: loss: 0.11612 [I 2022-10-09 15:54:52] ...optimizers/trijoint.py.151: total_norm: 9.547003 [I 2022-10-09 15:55:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 500/2304 [I 2022-10-09 15:55:01] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:21 [I 2022-10-09 15:55:01] ...trap/engines/engine.py.229: process: 0.84573 | load: 0.00299 [I 2022-10-09 15:55:01] ...trap/engines/engine.py.230: loss: 0.20668 [I 2022-10-09 15:55:01] ...optimizers/trijoint.py.151: total_norm: 18.134340 [I 2022-10-09 15:55:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 510/2304 [I 2022-10-09 15:55:09] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:20 [I 2022-10-09 15:55:09] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00311 [I 2022-10-09 15:55:09] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-09 15:55:09] ...optimizers/trijoint.py.151: total_norm: 12.988731 [I 2022-10-09 15:55:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 520/2304 [I 2022-10-09 15:55:17] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:24:57 [I 2022-10-09 15:55:17] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00297 [I 2022-10-09 15:55:17] ...trap/engines/engine.py.230: loss: 0.11420 [I 2022-10-09 15:55:17] ...optimizers/trijoint.py.151: total_norm: 13.276451 [I 2022-10-09 15:55:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 530/2304 [I 2022-10-09 15:55:26] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:12 [I 2022-10-09 15:55:26] ...trap/engines/engine.py.229: process: 0.87427 | load: 0.00299 [I 2022-10-09 15:55:26] ...trap/engines/engine.py.230: loss: 0.15970 [I 2022-10-09 15:55:26] ...optimizers/trijoint.py.151: total_norm: 21.353935 [I 2022-10-09 15:55:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 540/2304 [I 2022-10-09 15:55:34] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:24:57 [I 2022-10-09 15:55:34] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00297 [I 2022-10-09 15:55:34] ...trap/engines/engine.py.230: loss: 0.21993 [I 2022-10-09 15:55:34] ...optimizers/trijoint.py.151: total_norm: 28.206343 [I 2022-10-09 15:55:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 550/2304 [I 2022-10-09 15:55:43] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:42 [I 2022-10-09 15:55:43] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00298 [I 2022-10-09 15:55:43] ...trap/engines/engine.py.230: loss: 0.17324 [I 2022-10-09 15:55:43] ...optimizers/trijoint.py.151: total_norm: 29.234121 [I 2022-10-09 15:55:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 560/2304 [I 2022-10-09 15:55:51] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:31 [I 2022-10-09 15:55:51] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00298 [I 2022-10-09 15:55:51] ...trap/engines/engine.py.230: loss: 0.14630 [I 2022-10-09 15:55:51] ...optimizers/trijoint.py.151: total_norm: 20.431107 [I 2022-10-09 15:56:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 570/2304 [I 2022-10-09 15:56:00] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:24:23 [I 2022-10-09 15:56:00] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00294 [I 2022-10-09 15:56:00] ...trap/engines/engine.py.230: loss: 0.14468 [I 2022-10-09 15:56:00] ...optimizers/trijoint.py.151: total_norm: 19.409098 [I 2022-10-09 15:56:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 580/2304 [I 2022-10-09 15:56:08] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:15 [I 2022-10-09 15:56:08] ...trap/engines/engine.py.229: process: 0.83884 | load: 0.00300 [I 2022-10-09 15:56:08] ...trap/engines/engine.py.230: loss: 0.16562 [I 2022-10-09 15:56:08] ...optimizers/trijoint.py.151: total_norm: 19.791731 [I 2022-10-09 15:56:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 590/2304 [I 2022-10-09 15:56:17] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:06 [I 2022-10-09 15:56:17] ...trap/engines/engine.py.229: process: 0.83056 | load: 0.00297 [I 2022-10-09 15:56:17] ...trap/engines/engine.py.230: loss: 0.11907 [I 2022-10-09 15:56:17] ...optimizers/trijoint.py.151: total_norm: 8.729229 [I 2022-10-09 15:56:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 600/2304 [I 2022-10-09 15:56:25] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:23:59 [I 2022-10-09 15:56:25] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00298 [I 2022-10-09 15:56:25] ...trap/engines/engine.py.230: loss: 0.17031 [I 2022-10-09 15:56:25] ...optimizers/trijoint.py.151: total_norm: 27.726587 [I 2022-10-09 15:56:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 610/2304 [I 2022-10-09 15:56:34] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:23:56 [I 2022-10-09 15:56:34] ...trap/engines/engine.py.229: process: 0.85604 | load: 0.00304 [I 2022-10-09 15:56:34] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-09 15:56:34] ...optimizers/trijoint.py.151: total_norm: 8.466928 [I 2022-10-09 15:56:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 620/2304 [I 2022-10-09 15:56:42] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:45 [I 2022-10-09 15:56:42] ...trap/engines/engine.py.229: process: 0.83531 | load: 0.00296 [I 2022-10-09 15:56:42] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-09 15:56:42] ...optimizers/trijoint.py.151: total_norm: 15.007326 [I 2022-10-09 15:56:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 630/2304 [I 2022-10-09 15:56:51] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:36 [I 2022-10-09 15:56:51] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00299 [I 2022-10-09 15:56:51] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-09 15:56:51] ...optimizers/trijoint.py.151: total_norm: 42.228069 [I 2022-10-09 15:56:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 640/2304 [I 2022-10-09 15:56:59] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:30 [I 2022-10-09 15:56:59] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00296 [I 2022-10-09 15:56:59] ...trap/engines/engine.py.230: loss: 0.18286 [I 2022-10-09 15:56:59] ...optimizers/trijoint.py.151: total_norm: 15.468429 [I 2022-10-09 15:57:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 650/2304 [I 2022-10-09 15:57:08] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:18 [I 2022-10-09 15:57:08] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00296 [I 2022-10-09 15:57:08] ...trap/engines/engine.py.230: loss: 0.18507 [I 2022-10-09 15:57:08] ...optimizers/trijoint.py.151: total_norm: 25.866625 [I 2022-10-09 15:57:16] ...trap/engines/engine.py.224: train: epoch 46 | batch 660/2304 [I 2022-10-09 15:57:16] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:04 [I 2022-10-09 15:57:16] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00302 [I 2022-10-09 15:57:16] ...trap/engines/engine.py.230: loss: 0.19096 [I 2022-10-09 15:57:16] ...optimizers/trijoint.py.151: total_norm: 10.811520 [I 2022-10-09 15:57:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 670/2304 [I 2022-10-09 15:57:24] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:22:53 [I 2022-10-09 15:57:24] ...trap/engines/engine.py.229: process: 0.83420 | load: 0.00294 [I 2022-10-09 15:57:25] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-09 15:57:25] ...optimizers/trijoint.py.151: total_norm: 14.619770 [I 2022-10-09 15:57:33] ...trap/engines/engine.py.224: train: epoch 46 | batch 680/2304 [I 2022-10-09 15:57:33] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:22:40 [I 2022-10-09 15:57:33] ...trap/engines/engine.py.229: process: 0.82783 | load: 0.00299 [I 2022-10-09 15:57:33] ...trap/engines/engine.py.230: loss: 0.14123 [I 2022-10-09 15:57:33] ...optimizers/trijoint.py.151: total_norm: 16.873453 [I 2022-10-09 15:57:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 690/2304 [I 2022-10-09 15:57:41] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:50 [I 2022-10-09 15:57:41] ...trap/engines/engine.py.229: process: 0.85337 | load: 0.00301 [I 2022-10-09 15:57:41] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-09 15:57:41] ...optimizers/trijoint.py.151: total_norm: 13.122565 [I 2022-10-09 15:57:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 700/2304 [I 2022-10-09 15:57:50] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:22:37 [I 2022-10-09 15:57:50] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00297 [I 2022-10-09 15:57:50] ...trap/engines/engine.py.230: loss: 0.17658 [I 2022-10-09 15:57:50] ...optimizers/trijoint.py.151: total_norm: 22.811041 [I 2022-10-09 15:57:58] ...trap/engines/engine.py.224: train: epoch 46 | batch 710/2304 [I 2022-10-09 15:57:58] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:23 [I 2022-10-09 15:57:58] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00298 [I 2022-10-09 15:57:58] ...trap/engines/engine.py.230: loss: 0.11754 [I 2022-10-09 15:57:58] ...optimizers/trijoint.py.151: total_norm: 17.507355 [I 2022-10-09 15:58:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 720/2304 [I 2022-10-09 15:58:07] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:22:18 [I 2022-10-09 15:58:07] ...trap/engines/engine.py.229: process: 0.84836 | load: 0.00297 [I 2022-10-09 15:58:07] ...trap/engines/engine.py.230: loss: 0.23287 [I 2022-10-09 15:58:07] ...optimizers/trijoint.py.151: total_norm: 17.868837 [I 2022-10-09 15:58:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 730/2304 [I 2022-10-09 15:58:15] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:04 [I 2022-10-09 15:58:15] ...trap/engines/engine.py.229: process: 0.82431 | load: 0.00298 [I 2022-10-09 15:58:15] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-09 15:58:15] ...optimizers/trijoint.py.151: total_norm: 14.526827 [I 2022-10-09 15:58:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 740/2304 [I 2022-10-09 15:58:24] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:22:03 [I 2022-10-09 15:58:24] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00296 [I 2022-10-09 15:58:24] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-09 15:58:24] ...optimizers/trijoint.py.151: total_norm: 18.712486 [I 2022-10-09 15:58:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 750/2304 [I 2022-10-09 15:58:32] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:57 [I 2022-10-09 15:58:32] ...trap/engines/engine.py.229: process: 0.84127 | load: 0.00297 [I 2022-10-09 15:58:32] ...trap/engines/engine.py.230: loss: 0.16961 [I 2022-10-09 15:58:32] ...optimizers/trijoint.py.151: total_norm: 12.548827 [I 2022-10-09 15:58:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 760/2304 [I 2022-10-09 15:58:41] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:21:42 [I 2022-10-09 15:58:41] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00294 [I 2022-10-09 15:58:41] ...trap/engines/engine.py.230: loss: 0.13273 [I 2022-10-09 15:58:41] ...optimizers/trijoint.py.151: total_norm: 17.808861 [I 2022-10-09 15:58:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 770/2304 [I 2022-10-09 15:58:49] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:42 [I 2022-10-09 15:58:49] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00298 [I 2022-10-09 15:58:49] ...trap/engines/engine.py.230: loss: 0.15796 [I 2022-10-09 15:58:49] ...optimizers/trijoint.py.151: total_norm: 22.533779 [I 2022-10-09 15:58:58] ...trap/engines/engine.py.224: train: epoch 46 | batch 780/2304 [I 2022-10-09 15:58:58] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:21:30 [I 2022-10-09 15:58:58] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00294 [I 2022-10-09 15:58:58] ...trap/engines/engine.py.230: loss: 0.13113 [I 2022-10-09 15:58:58] ...optimizers/trijoint.py.151: total_norm: 14.477176 [I 2022-10-09 15:59:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 790/2304 [I 2022-10-09 15:59:06] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:25 [I 2022-10-09 15:59:06] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00303 [I 2022-10-09 15:59:06] ...trap/engines/engine.py.230: loss: 0.13663 [I 2022-10-09 15:59:06] ...optimizers/trijoint.py.151: total_norm: 13.505713 [I 2022-10-09 15:59:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 800/2304 [I 2022-10-09 15:59:15] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:09 [I 2022-10-09 15:59:15] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00295 [I 2022-10-09 15:59:15] ...trap/engines/engine.py.230: loss: 0.17023 [I 2022-10-09 15:59:15] ...optimizers/trijoint.py.151: total_norm: 23.238937 [I 2022-10-09 15:59:23] ...trap/engines/engine.py.224: train: epoch 46 | batch 810/2304 [I 2022-10-09 15:59:23] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:21:04 [I 2022-10-09 15:59:23] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00300 [I 2022-10-09 15:59:23] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-09 15:59:23] ...optimizers/trijoint.py.151: total_norm: 25.877956 [I 2022-10-09 15:59:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 820/2304 [I 2022-10-09 15:59:32] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:20:53 [I 2022-10-09 15:59:32] ...trap/engines/engine.py.229: process: 0.84540 | load: 0.00293 [I 2022-10-09 15:59:32] ...trap/engines/engine.py.230: loss: 0.14104 [I 2022-10-09 15:59:32] ...optimizers/trijoint.py.151: total_norm: 33.795345 [I 2022-10-09 15:59:40] ...trap/engines/engine.py.224: train: epoch 46 | batch 830/2304 [I 2022-10-09 15:59:40] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:45 [I 2022-10-09 15:59:40] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00305 [I 2022-10-09 15:59:40] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-09 15:59:40] ...optimizers/trijoint.py.151: total_norm: 16.769403 [I 2022-10-09 15:59:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 840/2304 [I 2022-10-09 15:59:48] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:32 [I 2022-10-09 15:59:48] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00297 [I 2022-10-09 15:59:48] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-09 15:59:48] ...optimizers/trijoint.py.151: total_norm: 18.936075 [I 2022-10-09 15:59:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 850/2304 [I 2022-10-09 15:59:57] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:20:33 [I 2022-10-09 15:59:57] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00293 [I 2022-10-09 15:59:57] ...trap/engines/engine.py.230: loss: 0.16227 [I 2022-10-09 15:59:57] ...optimizers/trijoint.py.151: total_norm: 27.660044 [I 2022-10-09 16:00:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 860/2304 [I 2022-10-09 16:00:05] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:17 [I 2022-10-09 16:00:05] ...trap/engines/engine.py.229: process: 0.84471 | load: 0.00292 [I 2022-10-09 16:00:05] ...trap/engines/engine.py.230: loss: 0.19855 [I 2022-10-09 16:00:05] ...optimizers/trijoint.py.151: total_norm: 37.681866 [I 2022-10-09 16:00:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 870/2304 [I 2022-10-09 16:00:14] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:07 [I 2022-10-09 16:00:14] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00294 [I 2022-10-09 16:00:14] ...trap/engines/engine.py.230: loss: 0.15755 [I 2022-10-09 16:00:14] ...optimizers/trijoint.py.151: total_norm: 17.222088 [I 2022-10-09 16:00:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 880/2304 [I 2022-10-09 16:00:22] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:19:58 [I 2022-10-09 16:00:22] ...trap/engines/engine.py.229: process: 0.83298 | load: 0.00293 [I 2022-10-09 16:00:22] ...trap/engines/engine.py.230: loss: 0.15478 [I 2022-10-09 16:00:22] ...optimizers/trijoint.py.151: total_norm: 13.165437 [I 2022-10-09 16:00:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 890/2304 [I 2022-10-09 16:00:31] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:19:52 [I 2022-10-09 16:00:31] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00302 [I 2022-10-09 16:00:31] ...trap/engines/engine.py.230: loss: 0.09536 [I 2022-10-09 16:00:31] ...optimizers/trijoint.py.151: total_norm: 12.753167 [I 2022-10-09 16:00:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 900/2304 [I 2022-10-09 16:00:39] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:19:43 [I 2022-10-09 16:00:39] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00296 [I 2022-10-09 16:00:39] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-09 16:00:39] ...optimizers/trijoint.py.151: total_norm: 18.969330 [I 2022-10-09 16:00:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 910/2304 [I 2022-10-09 16:00:48] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:40 [I 2022-10-09 16:00:48] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00291 [I 2022-10-09 16:00:48] ...trap/engines/engine.py.230: loss: 0.14120 [I 2022-10-09 16:00:48] ...optimizers/trijoint.py.151: total_norm: 16.532434 [I 2022-10-09 16:00:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 920/2304 [I 2022-10-09 16:00:56] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:29 [I 2022-10-09 16:00:56] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00293 [I 2022-10-09 16:00:56] ...trap/engines/engine.py.230: loss: 0.11129 [I 2022-10-09 16:00:56] ...optimizers/trijoint.py.151: total_norm: 8.865885 [I 2022-10-09 16:01:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 930/2304 [I 2022-10-09 16:01:05] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:23 [I 2022-10-09 16:01:05] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00293 [I 2022-10-09 16:01:05] ...trap/engines/engine.py.230: loss: 0.11736 [I 2022-10-09 16:01:05] ...optimizers/trijoint.py.151: total_norm: 18.888439 [I 2022-10-09 16:01:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 940/2304 [I 2022-10-09 16:01:13] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:09 [I 2022-10-09 16:01:13] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00294 [I 2022-10-09 16:01:13] ...trap/engines/engine.py.230: loss: 0.20877 [I 2022-10-09 16:01:13] ...optimizers/trijoint.py.151: total_norm: 12.029911 [I 2022-10-09 16:01:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 950/2304 [I 2022-10-09 16:01:22] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:04 [I 2022-10-09 16:01:22] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00297 [I 2022-10-09 16:01:22] ...trap/engines/engine.py.230: loss: 0.17870 [I 2022-10-09 16:01:22] ...optimizers/trijoint.py.151: total_norm: 13.471573 [I 2022-10-09 16:01:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 960/2304 [I 2022-10-09 16:01:30] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:18:54 [I 2022-10-09 16:01:30] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00292 [I 2022-10-09 16:01:30] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-09 16:01:30] ...optimizers/trijoint.py.151: total_norm: 29.441658 [I 2022-10-09 16:01:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 970/2304 [I 2022-10-09 16:01:38] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:18:48 [I 2022-10-09 16:01:38] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00295 [I 2022-10-09 16:01:38] ...trap/engines/engine.py.230: loss: 0.18410 [I 2022-10-09 16:01:38] ...optimizers/trijoint.py.151: total_norm: 18.865492 [I 2022-10-09 16:01:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 980/2304 [I 2022-10-09 16:01:47] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:36 [I 2022-10-09 16:01:47] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00311 [I 2022-10-09 16:01:47] ...trap/engines/engine.py.230: loss: 0.15961 [I 2022-10-09 16:01:47] ...optimizers/trijoint.py.151: total_norm: 19.741272 [I 2022-10-09 16:01:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 990/2304 [I 2022-10-09 16:01:55] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:33 [I 2022-10-09 16:01:55] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00292 [I 2022-10-09 16:01:55] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-09 16:01:55] ...optimizers/trijoint.py.151: total_norm: 16.907764 [I 2022-10-09 16:02:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 1000/2304 [I 2022-10-09 16:02:04] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:22 [I 2022-10-09 16:02:04] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00301 [I 2022-10-09 16:02:04] ...trap/engines/engine.py.230: loss: 0.12177 [I 2022-10-09 16:02:04] ...optimizers/trijoint.py.151: total_norm: 15.074225 [I 2022-10-09 16:02:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 1010/2304 [I 2022-10-09 16:02:12] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:06 [I 2022-10-09 16:02:12] ...trap/engines/engine.py.229: process: 0.82983 | load: 0.00296 [I 2022-10-09 16:02:12] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-09 16:02:12] ...optimizers/trijoint.py.151: total_norm: 27.706133 [I 2022-10-09 16:02:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 1020/2304 [I 2022-10-09 16:02:21] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:01 [I 2022-10-09 16:02:21] ...trap/engines/engine.py.229: process: 0.83725 | load: 0.00307 [I 2022-10-09 16:02:21] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-09 16:02:21] ...optimizers/trijoint.py.151: total_norm: 9.431585 [I 2022-10-09 16:02:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 1030/2304 [I 2022-10-09 16:02:29] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:17:55 [I 2022-10-09 16:02:29] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00296 [I 2022-10-09 16:02:29] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-09 16:02:29] ...optimizers/trijoint.py.151: total_norm: 19.255299 [I 2022-10-09 16:02:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 1040/2304 [I 2022-10-09 16:02:38] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:55 [I 2022-10-09 16:02:38] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00298 [I 2022-10-09 16:02:38] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-09 16:02:38] ...optimizers/trijoint.py.151: total_norm: 10.855591 [I 2022-10-09 16:02:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1050/2304 [I 2022-10-09 16:02:46] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:48 [I 2022-10-09 16:02:46] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00298 [I 2022-10-09 16:02:46] ...trap/engines/engine.py.230: loss: 0.14677 [I 2022-10-09 16:02:46] ...optimizers/trijoint.py.151: total_norm: 8.215175 [I 2022-10-09 16:02:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 1060/2304 [I 2022-10-09 16:02:55] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:37 [I 2022-10-09 16:02:55] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00309 [I 2022-10-09 16:02:55] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-09 16:02:55] ...optimizers/trijoint.py.151: total_norm: 15.783922 [I 2022-10-09 16:03:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 1070/2304 [I 2022-10-09 16:03:03] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:32 [I 2022-10-09 16:03:03] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00302 [I 2022-10-09 16:03:03] ...trap/engines/engine.py.230: loss: 0.20502 [I 2022-10-09 16:03:03] ...optimizers/trijoint.py.151: total_norm: 22.656179 [I 2022-10-09 16:03:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 1080/2304 [I 2022-10-09 16:03:12] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:09 [I 2022-10-09 16:03:12] ...trap/engines/engine.py.229: process: 0.83432 | load: 0.00302 [I 2022-10-09 16:03:12] ...trap/engines/engine.py.230: loss: 0.18411 [I 2022-10-09 16:03:12] ...optimizers/trijoint.py.151: total_norm: 15.101568 [I 2022-10-09 16:03:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 1090/2304 [I 2022-10-09 16:03:20] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:06 [I 2022-10-09 16:03:20] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00295 [I 2022-10-09 16:03:20] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-09 16:03:20] ...optimizers/trijoint.py.151: total_norm: 17.332867 [I 2022-10-09 16:03:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 1100/2304 [I 2022-10-09 16:03:29] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:16:53 [I 2022-10-09 16:03:29] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00305 [I 2022-10-09 16:03:29] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-09 16:03:29] ...optimizers/trijoint.py.151: total_norm: 30.752592 [I 2022-10-09 16:03:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 1110/2304 [I 2022-10-09 16:03:37] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:48 [I 2022-10-09 16:03:37] ...trap/engines/engine.py.229: process: 0.83807 | load: 0.00296 [I 2022-10-09 16:03:37] ...trap/engines/engine.py.230: loss: 0.13849 [I 2022-10-09 16:03:37] ...optimizers/trijoint.py.151: total_norm: 13.504070 [I 2022-10-09 16:03:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1120/2304 [I 2022-10-09 16:03:46] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:33 [I 2022-10-09 16:03:46] ...trap/engines/engine.py.229: process: 0.82579 | load: 0.00294 [I 2022-10-09 16:03:46] ...trap/engines/engine.py.230: loss: 0.16397 [I 2022-10-09 16:03:46] ...optimizers/trijoint.py.151: total_norm: 15.351032 [I 2022-10-09 16:03:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 1130/2304 [I 2022-10-09 16:03:54] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:31 [I 2022-10-09 16:03:54] ...trap/engines/engine.py.229: process: 0.85430 | load: 0.00293 [I 2022-10-09 16:03:54] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-09 16:03:54] ...optimizers/trijoint.py.151: total_norm: 13.042797 [I 2022-10-09 16:04:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 1140/2304 [I 2022-10-09 16:04:02] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:18 [I 2022-10-09 16:04:02] ...trap/engines/engine.py.229: process: 0.83269 | load: 0.00307 [I 2022-10-09 16:04:02] ...trap/engines/engine.py.230: loss: 0.17362 [I 2022-10-09 16:04:02] ...optimizers/trijoint.py.151: total_norm: 23.007624 [I 2022-10-09 16:04:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 1150/2304 [I 2022-10-09 16:04:11] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:10 [I 2022-10-09 16:04:11] ...trap/engines/engine.py.229: process: 0.82984 | load: 0.00297 [I 2022-10-09 16:04:11] ...trap/engines/engine.py.230: loss: 0.17892 [I 2022-10-09 16:04:11] ...optimizers/trijoint.py.151: total_norm: 18.705194 [I 2022-10-09 16:04:19] ...trap/engines/engine.py.224: train: epoch 46 | batch 1160/2304 [I 2022-10-09 16:04:19] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:05 [I 2022-10-09 16:04:19] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00297 [I 2022-10-09 16:04:19] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-09 16:04:19] ...optimizers/trijoint.py.151: total_norm: 18.445881 [I 2022-10-09 16:04:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 1170/2304 [I 2022-10-09 16:04:28] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:15:51 [I 2022-10-09 16:04:28] ...trap/engines/engine.py.229: process: 0.83431 | load: 0.00298 [I 2022-10-09 16:04:28] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-09 16:04:28] ...optimizers/trijoint.py.151: total_norm: 10.178900 [I 2022-10-09 16:04:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 1180/2304 [I 2022-10-09 16:04:36] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:50 [I 2022-10-09 16:04:36] ...trap/engines/engine.py.229: process: 0.86504 | load: 0.00305 [I 2022-10-09 16:04:36] ...trap/engines/engine.py.230: loss: 0.17590 [I 2022-10-09 16:04:36] ...optimizers/trijoint.py.151: total_norm: 48.238342 [I 2022-10-09 16:04:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 1190/2304 [I 2022-10-09 16:04:45] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:45 [I 2022-10-09 16:04:45] ...trap/engines/engine.py.229: process: 0.86098 | load: 0.00302 [I 2022-10-09 16:04:45] ...trap/engines/engine.py.230: loss: 0.27855 [I 2022-10-09 16:04:45] ...optimizers/trijoint.py.151: total_norm: 17.693758 [I 2022-10-09 16:04:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 1200/2304 [I 2022-10-09 16:04:53] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:30 [I 2022-10-09 16:04:53] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00295 [I 2022-10-09 16:04:53] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-09 16:04:53] ...optimizers/trijoint.py.151: total_norm: 7.108878 [I 2022-10-09 16:05:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 1210/2304 [I 2022-10-09 16:05:02] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:22 [I 2022-10-09 16:05:02] ...trap/engines/engine.py.229: process: 0.83349 | load: 0.00295 [I 2022-10-09 16:05:02] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-09 16:05:02] ...optimizers/trijoint.py.151: total_norm: 22.634674 [I 2022-10-09 16:05:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 1220/2304 [I 2022-10-09 16:05:10] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:13 [I 2022-10-09 16:05:10] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00299 [I 2022-10-09 16:05:10] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-09 16:05:10] ...optimizers/trijoint.py.151: total_norm: 14.759346 [I 2022-10-09 16:05:19] ...trap/engines/engine.py.224: train: epoch 46 | batch 1230/2304 [I 2022-10-09 16:05:19] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:09 [I 2022-10-09 16:05:19] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00296 [I 2022-10-09 16:05:19] ...trap/engines/engine.py.230: loss: 0.11741 [I 2022-10-09 16:05:19] ...optimizers/trijoint.py.151: total_norm: 14.103160 [I 2022-10-09 16:05:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 1240/2304 [I 2022-10-09 16:05:27] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:56 [I 2022-10-09 16:05:27] ...trap/engines/engine.py.229: process: 0.82770 | load: 0.00296 [I 2022-10-09 16:05:27] ...trap/engines/engine.py.230: loss: 0.10626 [I 2022-10-09 16:05:27] ...optimizers/trijoint.py.151: total_norm: 10.659162 [I 2022-10-09 16:05:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 1250/2304 [I 2022-10-09 16:05:35] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:49 [I 2022-10-09 16:05:35] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00295 [I 2022-10-09 16:05:35] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-09 16:05:35] ...optimizers/trijoint.py.151: total_norm: 11.859202 [I 2022-10-09 16:05:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 1260/2304 [I 2022-10-09 16:05:44] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:14:37 [I 2022-10-09 16:05:44] ...trap/engines/engine.py.229: process: 0.83850 | load: 0.00294 [I 2022-10-09 16:05:44] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-09 16:05:44] ...optimizers/trijoint.py.151: total_norm: 11.358443 [I 2022-10-09 16:05:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 1270/2304 [I 2022-10-09 16:05:52] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:36 [I 2022-10-09 16:05:52] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00298 [I 2022-10-09 16:05:52] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-09 16:05:52] ...optimizers/trijoint.py.151: total_norm: 15.292560 [I 2022-10-09 16:06:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 1280/2304 [I 2022-10-09 16:06:01] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:21 [I 2022-10-09 16:06:01] ...trap/engines/engine.py.229: process: 0.82283 | load: 0.00294 [I 2022-10-09 16:06:01] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-09 16:06:01] ...optimizers/trijoint.py.151: total_norm: 9.870511 [I 2022-10-09 16:06:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 1290/2304 [I 2022-10-09 16:06:09] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:16 [I 2022-10-09 16:06:09] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00306 [I 2022-10-09 16:06:09] ...trap/engines/engine.py.230: loss: 0.13915 [I 2022-10-09 16:06:09] ...optimizers/trijoint.py.151: total_norm: 21.177252 [I 2022-10-09 16:06:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 1300/2304 [I 2022-10-09 16:06:18] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:05 [I 2022-10-09 16:06:18] ...trap/engines/engine.py.229: process: 0.84939 | load: 0.00295 [I 2022-10-09 16:06:18] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-09 16:06:18] ...optimizers/trijoint.py.151: total_norm: 29.445585 [I 2022-10-09 16:06:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 1310/2304 [I 2022-10-09 16:06:26] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:01 [I 2022-10-09 16:06:26] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00297 [I 2022-10-09 16:06:26] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-09 16:06:26] ...optimizers/trijoint.py.151: total_norm: 18.143133 [I 2022-10-09 16:06:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 1320/2304 [I 2022-10-09 16:06:35] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:13:47 [I 2022-10-09 16:06:35] ...trap/engines/engine.py.229: process: 0.83506 | load: 0.00293 [I 2022-10-09 16:06:35] ...trap/engines/engine.py.230: loss: 0.10034 [I 2022-10-09 16:06:35] ...optimizers/trijoint.py.151: total_norm: 7.081726 [I 2022-10-09 16:06:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 1330/2304 [I 2022-10-09 16:06:43] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:40 [I 2022-10-09 16:06:43] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00294 [I 2022-10-09 16:06:43] ...trap/engines/engine.py.230: loss: 0.16260 [I 2022-10-09 16:06:43] ...optimizers/trijoint.py.151: total_norm: 19.575224 [I 2022-10-09 16:06:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 1340/2304 [I 2022-10-09 16:06:51] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:30 [I 2022-10-09 16:06:51] ...trap/engines/engine.py.229: process: 0.83474 | load: 0.00299 [I 2022-10-09 16:06:51] ...trap/engines/engine.py.230: loss: 0.13756 [I 2022-10-09 16:06:51] ...optimizers/trijoint.py.151: total_norm: 19.893597 [I 2022-10-09 16:07:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 1350/2304 [I 2022-10-09 16:07:00] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:26 [I 2022-10-09 16:07:00] ...trap/engines/engine.py.229: process: 0.84862 | load: 0.00302 [I 2022-10-09 16:07:00] ...trap/engines/engine.py.230: loss: 0.13242 [I 2022-10-09 16:07:00] ...optimizers/trijoint.py.151: total_norm: 11.402226 [I 2022-10-09 16:07:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 1360/2304 [I 2022-10-09 16:07:08] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:13 [I 2022-10-09 16:07:08] ...trap/engines/engine.py.229: process: 0.83859 | load: 0.00297 [I 2022-10-09 16:07:08] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-09 16:07:08] ...optimizers/trijoint.py.151: total_norm: 8.896601 [I 2022-10-09 16:07:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 1370/2304 [I 2022-10-09 16:07:17] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:11 [I 2022-10-09 16:07:17] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00295 [I 2022-10-09 16:07:17] ...trap/engines/engine.py.230: loss: 0.11104 [I 2022-10-09 16:07:17] ...optimizers/trijoint.py.151: total_norm: 13.828048 [I 2022-10-09 16:07:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 1380/2304 [I 2022-10-09 16:07:25] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:12:57 [I 2022-10-09 16:07:25] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00294 [I 2022-10-09 16:07:25] ...trap/engines/engine.py.230: loss: 0.19220 [I 2022-10-09 16:07:25] ...optimizers/trijoint.py.151: total_norm: 58.302242 [I 2022-10-09 16:07:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 1390/2304 [I 2022-10-09 16:07:34] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:12:50 [I 2022-10-09 16:07:34] ...trap/engines/engine.py.229: process: 0.83961 | load: 0.00293 [I 2022-10-09 16:07:34] ...trap/engines/engine.py.230: loss: 0.17885 [I 2022-10-09 16:07:34] ...optimizers/trijoint.py.151: total_norm: 21.743359 [I 2022-10-09 16:07:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 1400/2304 [I 2022-10-09 16:07:42] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:41 [I 2022-10-09 16:07:42] ...trap/engines/engine.py.229: process: 0.83567 | load: 0.00291 [I 2022-10-09 16:07:42] ...trap/engines/engine.py.230: loss: 0.14403 [I 2022-10-09 16:07:42] ...optimizers/trijoint.py.151: total_norm: 9.654390 [I 2022-10-09 16:07:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 1410/2304 [I 2022-10-09 16:07:51] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:32 [I 2022-10-09 16:07:51] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00294 [I 2022-10-09 16:07:51] ...trap/engines/engine.py.230: loss: 0.16814 [I 2022-10-09 16:07:51] ...optimizers/trijoint.py.151: total_norm: 21.792980 [I 2022-10-09 16:07:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 1420/2304 [I 2022-10-09 16:07:59] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:24 [I 2022-10-09 16:07:59] ...trap/engines/engine.py.229: process: 0.82724 | load: 0.00303 [I 2022-10-09 16:07:59] ...trap/engines/engine.py.230: loss: 0.23553 [I 2022-10-09 16:07:59] ...optimizers/trijoint.py.151: total_norm: 28.431589 [I 2022-10-09 16:08:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 1430/2304 [I 2022-10-09 16:08:08] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:18 [I 2022-10-09 16:08:08] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00292 [I 2022-10-09 16:08:08] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-09 16:08:08] ...optimizers/trijoint.py.151: total_norm: 16.865602 [I 2022-10-09 16:08:16] ...trap/engines/engine.py.224: train: epoch 46 | batch 1440/2304 [I 2022-10-09 16:08:16] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:09 [I 2022-10-09 16:08:16] ...trap/engines/engine.py.229: process: 0.85134 | load: 0.00293 [I 2022-10-09 16:08:16] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-09 16:08:16] ...optimizers/trijoint.py.151: total_norm: 9.202018 [I 2022-10-09 16:08:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 1450/2304 [I 2022-10-09 16:08:24] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:11:58 [I 2022-10-09 16:08:24] ...trap/engines/engine.py.229: process: 0.83256 | load: 0.00295 [I 2022-10-09 16:08:24] ...trap/engines/engine.py.230: loss: 0.15777 [I 2022-10-09 16:08:24] ...optimizers/trijoint.py.151: total_norm: 11.629564 [I 2022-10-09 16:08:33] ...trap/engines/engine.py.224: train: epoch 46 | batch 1460/2304 [I 2022-10-09 16:08:33] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:11:45 [I 2022-10-09 16:08:33] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00295 [I 2022-10-09 16:08:33] ...trap/engines/engine.py.230: loss: 0.08320 [I 2022-10-09 16:08:33] ...optimizers/trijoint.py.151: total_norm: 8.113251 [I 2022-10-09 16:08:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 1470/2304 [I 2022-10-09 16:08:41] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:41 [I 2022-10-09 16:08:41] ...trap/engines/engine.py.229: process: 0.84998 | load: 0.00294 [I 2022-10-09 16:08:41] ...trap/engines/engine.py.230: loss: 0.29977 [I 2022-10-09 16:08:41] ...optimizers/trijoint.py.151: total_norm: 20.840458 [I 2022-10-09 16:08:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 1480/2304 [I 2022-10-09 16:08:50] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:32 [I 2022-10-09 16:08:50] ...trap/engines/engine.py.229: process: 0.82535 | load: 0.00295 [I 2022-10-09 16:08:50] ...trap/engines/engine.py.230: loss: 0.15059 [I 2022-10-09 16:08:50] ...optimizers/trijoint.py.151: total_norm: 10.145577 [I 2022-10-09 16:08:58] ...trap/engines/engine.py.224: train: epoch 46 | batch 1490/2304 [I 2022-10-09 16:08:58] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:23 [I 2022-10-09 16:08:58] ...trap/engines/engine.py.229: process: 0.82745 | load: 0.00296 [I 2022-10-09 16:08:58] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-09 16:08:58] ...optimizers/trijoint.py.151: total_norm: 10.334212 [I 2022-10-09 16:09:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 1500/2304 [I 2022-10-09 16:09:07] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:16 [I 2022-10-09 16:09:07] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00296 [I 2022-10-09 16:09:07] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-09 16:09:07] ...optimizers/trijoint.py.151: total_norm: 19.834980 [I 2022-10-09 16:09:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 1510/2304 [I 2022-10-09 16:09:15] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:10 [I 2022-10-09 16:09:15] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00299 [I 2022-10-09 16:09:15] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-09 16:09:15] ...optimizers/trijoint.py.151: total_norm: 15.375160 [I 2022-10-09 16:09:23] ...trap/engines/engine.py.224: train: epoch 46 | batch 1520/2304 [I 2022-10-09 16:09:23] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:03 [I 2022-10-09 16:09:23] ...trap/engines/engine.py.229: process: 0.83896 | load: 0.00306 [I 2022-10-09 16:09:23] ...trap/engines/engine.py.230: loss: 0.14672 [I 2022-10-09 16:09:23] ...optimizers/trijoint.py.151: total_norm: 12.636621 [I 2022-10-09 16:09:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 1530/2304 [I 2022-10-09 16:09:32] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:10:52 [I 2022-10-09 16:09:32] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00299 [I 2022-10-09 16:09:32] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-09 16:09:32] ...optimizers/trijoint.py.151: total_norm: 19.733164 [I 2022-10-09 16:09:40] ...trap/engines/engine.py.224: train: epoch 46 | batch 1540/2304 [I 2022-10-09 16:09:40] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:10:45 [I 2022-10-09 16:09:40] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00296 [I 2022-10-09 16:09:40] ...trap/engines/engine.py.230: loss: 0.15496 [I 2022-10-09 16:09:40] ...optimizers/trijoint.py.151: total_norm: 9.591530 [I 2022-10-09 16:09:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 1550/2304 [I 2022-10-09 16:09:49] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:40 [I 2022-10-09 16:09:49] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00295 [I 2022-10-09 16:09:49] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-09 16:09:49] ...optimizers/trijoint.py.151: total_norm: 15.121653 [I 2022-10-09 16:09:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 1560/2304 [I 2022-10-09 16:09:57] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:31 [I 2022-10-09 16:09:57] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00300 [I 2022-10-09 16:09:57] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-09 16:09:57] ...optimizers/trijoint.py.151: total_norm: 10.725473 [I 2022-10-09 16:10:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 1570/2304 [I 2022-10-09 16:10:06] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:23 [I 2022-10-09 16:10:06] ...trap/engines/engine.py.229: process: 0.84168 | load: 0.00306 [I 2022-10-09 16:10:06] ...trap/engines/engine.py.230: loss: 0.11277 [I 2022-10-09 16:10:06] ...optimizers/trijoint.py.151: total_norm: 12.194131 [I 2022-10-09 16:10:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 1580/2304 [I 2022-10-09 16:10:14] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:10 [I 2022-10-09 16:10:14] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00295 [I 2022-10-09 16:10:14] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-09 16:10:14] ...optimizers/trijoint.py.151: total_norm: 11.764606 [I 2022-10-09 16:10:23] ...trap/engines/engine.py.224: train: epoch 46 | batch 1590/2304 [I 2022-10-09 16:10:23] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:05 [I 2022-10-09 16:10:23] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00298 [I 2022-10-09 16:10:23] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-09 16:10:23] ...optimizers/trijoint.py.151: total_norm: 19.489967 [I 2022-10-09 16:10:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 1600/2304 [I 2022-10-09 16:10:31] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:09:56 [I 2022-10-09 16:10:31] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00295 [I 2022-10-09 16:10:31] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-09 16:10:31] ...optimizers/trijoint.py.151: total_norm: 16.202333 [I 2022-10-09 16:10:40] ...trap/engines/engine.py.224: train: epoch 46 | batch 1610/2304 [I 2022-10-09 16:10:40] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:09:48 [I 2022-10-09 16:10:40] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00307 [I 2022-10-09 16:10:40] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-09 16:10:40] ...optimizers/trijoint.py.151: total_norm: 10.051041 [I 2022-10-09 16:10:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 1620/2304 [I 2022-10-09 16:10:48] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:09:36 [I 2022-10-09 16:10:48] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00295 [I 2022-10-09 16:10:48] ...trap/engines/engine.py.230: loss: 0.18405 [I 2022-10-09 16:10:48] ...optimizers/trijoint.py.151: total_norm: 18.742418 [I 2022-10-09 16:10:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 1630/2304 [I 2022-10-09 16:10:57] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:27 [I 2022-10-09 16:10:57] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00297 [I 2022-10-09 16:10:57] ...trap/engines/engine.py.230: loss: 0.14128 [I 2022-10-09 16:10:57] ...optimizers/trijoint.py.151: total_norm: 19.198729 [I 2022-10-09 16:11:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 1640/2304 [I 2022-10-09 16:11:05] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:21 [I 2022-10-09 16:11:05] ...trap/engines/engine.py.229: process: 0.83681 | load: 0.00306 [I 2022-10-09 16:11:05] ...trap/engines/engine.py.230: loss: 0.09123 [I 2022-10-09 16:11:05] ...optimizers/trijoint.py.151: total_norm: 8.594551 [I 2022-10-09 16:11:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 1650/2304 [I 2022-10-09 16:11:14] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:08 [I 2022-10-09 16:11:14] ...trap/engines/engine.py.229: process: 0.83267 | load: 0.00297 [I 2022-10-09 16:11:14] ...trap/engines/engine.py.230: loss: 0.11229 [I 2022-10-09 16:11:14] ...optimizers/trijoint.py.151: total_norm: 18.087681 [I 2022-10-09 16:11:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 1660/2304 [I 2022-10-09 16:11:22] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:07 [I 2022-10-09 16:11:22] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00295 [I 2022-10-09 16:11:22] ...trap/engines/engine.py.230: loss: 0.20057 [I 2022-10-09 16:11:22] ...optimizers/trijoint.py.151: total_norm: 30.887728 [I 2022-10-09 16:11:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 1670/2304 [I 2022-10-09 16:11:31] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:08:55 [I 2022-10-09 16:11:31] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00298 [I 2022-10-09 16:11:31] ...trap/engines/engine.py.230: loss: 0.28360 [I 2022-10-09 16:11:31] ...optimizers/trijoint.py.151: total_norm: 50.588131 [I 2022-10-09 16:11:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 1680/2304 [I 2022-10-09 16:11:39] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:08:48 [I 2022-10-09 16:11:39] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00301 [I 2022-10-09 16:11:39] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-09 16:11:39] ...optimizers/trijoint.py.151: total_norm: 13.791931 [I 2022-10-09 16:11:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 1690/2304 [I 2022-10-09 16:11:47] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:08:39 [I 2022-10-09 16:11:47] ...trap/engines/engine.py.229: process: 0.84651 | load: 0.00295 [I 2022-10-09 16:11:47] ...trap/engines/engine.py.230: loss: 0.16240 [I 2022-10-09 16:11:47] ...optimizers/trijoint.py.151: total_norm: 11.503677 [I 2022-10-09 16:11:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 1700/2304 [I 2022-10-09 16:11:56] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:28 [I 2022-10-09 16:11:56] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00300 [I 2022-10-09 16:11:56] ...trap/engines/engine.py.230: loss: 0.09641 [I 2022-10-09 16:11:56] ...optimizers/trijoint.py.151: total_norm: 7.860853 [I 2022-10-09 16:12:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 1710/2304 [I 2022-10-09 16:12:04] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:24 [I 2022-10-09 16:12:04] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00309 [I 2022-10-09 16:12:04] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-09 16:12:04] ...optimizers/trijoint.py.151: total_norm: 13.316045 [I 2022-10-09 16:12:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 1720/2304 [I 2022-10-09 16:12:13] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:14 [I 2022-10-09 16:12:13] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00297 [I 2022-10-09 16:12:13] ...trap/engines/engine.py.230: loss: 0.14200 [I 2022-10-09 16:12:13] ...optimizers/trijoint.py.151: total_norm: 72.087646 [I 2022-10-09 16:12:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 1730/2304 [I 2022-10-09 16:12:21] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:03 [I 2022-10-09 16:12:21] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00292 [I 2022-10-09 16:12:21] ...trap/engines/engine.py.230: loss: 0.18517 [I 2022-10-09 16:12:21] ...optimizers/trijoint.py.151: total_norm: 17.251745 [I 2022-10-09 16:12:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 1740/2304 [I 2022-10-09 16:12:30] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:07:56 [I 2022-10-09 16:12:30] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00295 [I 2022-10-09 16:12:30] ...trap/engines/engine.py.230: loss: 0.17399 [I 2022-10-09 16:12:30] ...optimizers/trijoint.py.151: total_norm: 10.921295 [I 2022-10-09 16:12:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 1750/2304 [I 2022-10-09 16:12:38] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:07:50 [I 2022-10-09 16:12:38] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00297 [I 2022-10-09 16:12:38] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-09 16:12:38] ...optimizers/trijoint.py.151: total_norm: 9.824717 [I 2022-10-09 16:12:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 1760/2304 [I 2022-10-09 16:12:47] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:07:40 [I 2022-10-09 16:12:47] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00300 [I 2022-10-09 16:12:47] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-09 16:12:47] ...optimizers/trijoint.py.151: total_norm: 24.327490 [I 2022-10-09 16:12:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 1770/2304 [I 2022-10-09 16:12:55] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:30 [I 2022-10-09 16:12:55] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00295 [I 2022-10-09 16:12:55] ...trap/engines/engine.py.230: loss: 0.26640 [I 2022-10-09 16:12:55] ...optimizers/trijoint.py.151: total_norm: 17.244553 [I 2022-10-09 16:13:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 1780/2304 [I 2022-10-09 16:13:04] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:23 [I 2022-10-09 16:13:04] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00294 [I 2022-10-09 16:13:04] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-09 16:13:04] ...optimizers/trijoint.py.151: total_norm: 12.902522 [I 2022-10-09 16:13:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 1790/2304 [I 2022-10-09 16:13:12] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:11 [I 2022-10-09 16:13:12] ...trap/engines/engine.py.229: process: 0.83021 | load: 0.00296 [I 2022-10-09 16:13:12] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-09 16:13:12] ...optimizers/trijoint.py.151: total_norm: 15.282275 [I 2022-10-09 16:13:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 1800/2304 [I 2022-10-09 16:13:21] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:07 [I 2022-10-09 16:13:21] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00296 [I 2022-10-09 16:13:21] ...trap/engines/engine.py.230: loss: 0.20318 [I 2022-10-09 16:13:21] ...optimizers/trijoint.py.151: total_norm: 22.500149 [I 2022-10-09 16:13:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 1810/2304 [I 2022-10-09 16:13:29] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:06:56 [I 2022-10-09 16:13:29] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00295 [I 2022-10-09 16:13:29] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-09 16:13:29] ...optimizers/trijoint.py.151: total_norm: 11.964933 [I 2022-10-09 16:13:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 1820/2304 [I 2022-10-09 16:13:37] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:06:48 [I 2022-10-09 16:13:37] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00298 [I 2022-10-09 16:13:37] ...trap/engines/engine.py.230: loss: 0.13844 [I 2022-10-09 16:13:37] ...optimizers/trijoint.py.151: total_norm: 11.275156 [I 2022-10-09 16:13:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1830/2304 [I 2022-10-09 16:13:46] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:06:40 [I 2022-10-09 16:13:46] ...trap/engines/engine.py.229: process: 0.85148 | load: 0.00297 [I 2022-10-09 16:13:46] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-09 16:13:46] ...optimizers/trijoint.py.151: total_norm: 14.369704 [I 2022-10-09 16:13:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 1840/2304 [I 2022-10-09 16:13:54] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:06:30 [I 2022-10-09 16:13:54] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00299 [I 2022-10-09 16:13:54] ...trap/engines/engine.py.230: loss: 0.14991 [I 2022-10-09 16:13:54] ...optimizers/trijoint.py.151: total_norm: 12.305669 [I 2022-10-09 16:14:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 1850/2304 [I 2022-10-09 16:14:03] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:06:23 [I 2022-10-09 16:14:03] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00303 [I 2022-10-09 16:14:03] ...trap/engines/engine.py.230: loss: 0.22788 [I 2022-10-09 16:14:03] ...optimizers/trijoint.py.151: total_norm: 16.606108 [I 2022-10-09 16:14:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 1860/2304 [I 2022-10-09 16:14:11] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:11 [I 2022-10-09 16:14:11] ...trap/engines/engine.py.229: process: 0.82912 | load: 0.00296 [I 2022-10-09 16:14:11] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-09 16:14:11] ...optimizers/trijoint.py.151: total_norm: 19.981276 [I 2022-10-09 16:14:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 1870/2304 [I 2022-10-09 16:14:20] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:06 [I 2022-10-09 16:14:20] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00296 [I 2022-10-09 16:14:20] ...trap/engines/engine.py.230: loss: 0.26412 [I 2022-10-09 16:14:20] ...optimizers/trijoint.py.151: total_norm: 22.228664 [I 2022-10-09 16:14:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 1880/2304 [I 2022-10-09 16:14:28] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:05:59 [I 2022-10-09 16:14:28] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00300 [I 2022-10-09 16:14:28] ...trap/engines/engine.py.230: loss: 0.15286 [I 2022-10-09 16:14:28] ...optimizers/trijoint.py.151: total_norm: 13.257111 [I 2022-10-09 16:14:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 1890/2304 [I 2022-10-09 16:14:37] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:05:51 [I 2022-10-09 16:14:37] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00298 [I 2022-10-09 16:14:37] ...trap/engines/engine.py.230: loss: 0.15217 [I 2022-10-09 16:14:37] ...optimizers/trijoint.py.151: total_norm: 18.348146 [I 2022-10-09 16:14:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 1900/2304 [I 2022-10-09 16:14:45] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:05:40 [I 2022-10-09 16:14:45] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00298 [I 2022-10-09 16:14:45] ...trap/engines/engine.py.230: loss: 0.14387 [I 2022-10-09 16:14:45] ...optimizers/trijoint.py.151: total_norm: 11.047574 [I 2022-10-09 16:14:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 1910/2304 [I 2022-10-09 16:14:54] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:32 [I 2022-10-09 16:14:54] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00301 [I 2022-10-09 16:14:54] ...trap/engines/engine.py.230: loss: 0.11972 [I 2022-10-09 16:14:54] ...optimizers/trijoint.py.151: total_norm: 25.990601 [I 2022-10-09 16:15:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 1920/2304 [I 2022-10-09 16:15:02] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:05:25 [I 2022-10-09 16:15:02] ...trap/engines/engine.py.229: process: 0.85581 | load: 0.00297 [I 2022-10-09 16:15:02] ...trap/engines/engine.py.230: loss: 0.17081 [I 2022-10-09 16:15:02] ...optimizers/trijoint.py.151: total_norm: 11.200864 [I 2022-10-09 16:15:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 1930/2304 [I 2022-10-09 16:15:11] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:15 [I 2022-10-09 16:15:11] ...trap/engines/engine.py.229: process: 0.84837 | load: 0.00296 [I 2022-10-09 16:15:11] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-09 16:15:11] ...optimizers/trijoint.py.151: total_norm: 11.661479 [I 2022-10-09 16:15:19] ...trap/engines/engine.py.224: train: epoch 46 | batch 1940/2304 [I 2022-10-09 16:15:19] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:08 [I 2022-10-09 16:15:19] ...trap/engines/engine.py.229: process: 0.83822 | load: 0.00297 [I 2022-10-09 16:15:19] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-09 16:15:19] ...optimizers/trijoint.py.151: total_norm: 35.666637 [I 2022-10-09 16:15:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 1950/2304 [I 2022-10-09 16:15:27] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:04:58 [I 2022-10-09 16:15:27] ...trap/engines/engine.py.229: process: 0.84017 | load: 0.00299 [I 2022-10-09 16:15:27] ...trap/engines/engine.py.230: loss: 0.14731 [I 2022-10-09 16:15:27] ...optimizers/trijoint.py.151: total_norm: 11.246865 [I 2022-10-09 16:15:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 1960/2304 [I 2022-10-09 16:15:36] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:04:49 [I 2022-10-09 16:15:36] ...trap/engines/engine.py.229: process: 0.84047 | load: 0.00308 [I 2022-10-09 16:15:36] ...trap/engines/engine.py.230: loss: 0.14427 [I 2022-10-09 16:15:36] ...optimizers/trijoint.py.151: total_norm: 18.609465 [I 2022-10-09 16:15:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 1970/2304 [I 2022-10-09 16:15:44] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:04:41 [I 2022-10-09 16:15:44] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00305 [I 2022-10-09 16:15:44] ...trap/engines/engine.py.230: loss: 0.22394 [I 2022-10-09 16:15:44] ...optimizers/trijoint.py.151: total_norm: 13.127386 [I 2022-10-09 16:15:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 1980/2304 [I 2022-10-09 16:15:53] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:04:34 [I 2022-10-09 16:15:53] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00300 [I 2022-10-09 16:15:53] ...trap/engines/engine.py.230: loss: 0.13200 [I 2022-10-09 16:15:53] ...optimizers/trijoint.py.151: total_norm: 13.599828 [I 2022-10-09 16:16:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 1990/2304 [I 2022-10-09 16:16:01] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:25 [I 2022-10-09 16:16:01] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00295 [I 2022-10-09 16:16:01] ...trap/engines/engine.py.230: loss: 0.14502 [I 2022-10-09 16:16:01] ...optimizers/trijoint.py.151: total_norm: 14.538823 [I 2022-10-09 16:16:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 2000/2304 [I 2022-10-09 16:16:10] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:15 [I 2022-10-09 16:16:10] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00294 [I 2022-10-09 16:16:10] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-09 16:16:10] ...optimizers/trijoint.py.151: total_norm: 38.591042 [I 2022-10-09 16:16:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 2010/2304 [I 2022-10-09 16:16:18] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:07 [I 2022-10-09 16:16:18] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00291 [I 2022-10-09 16:16:18] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-09 16:16:18] ...optimizers/trijoint.py.151: total_norm: 15.400624 [I 2022-10-09 16:16:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 2020/2304 [I 2022-10-09 16:16:27] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:03:59 [I 2022-10-09 16:16:27] ...trap/engines/engine.py.229: process: 0.84052 | load: 0.00293 [I 2022-10-09 16:16:27] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-09 16:16:27] ...optimizers/trijoint.py.151: total_norm: 12.881311 [I 2022-10-09 16:16:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 2030/2304 [I 2022-10-09 16:16:35] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:03:51 [I 2022-10-09 16:16:35] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00293 [I 2022-10-09 16:16:35] ...trap/engines/engine.py.230: loss: 0.18770 [I 2022-10-09 16:16:35] ...optimizers/trijoint.py.151: total_norm: 14.952940 [I 2022-10-09 16:16:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 2040/2304 [I 2022-10-09 16:16:44] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:43 [I 2022-10-09 16:16:44] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00302 [I 2022-10-09 16:16:44] ...trap/engines/engine.py.230: loss: 0.15637 [I 2022-10-09 16:16:44] ...optimizers/trijoint.py.151: total_norm: 14.785048 [I 2022-10-09 16:16:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 2050/2304 [I 2022-10-09 16:16:52] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:03:36 [I 2022-10-09 16:16:52] ...trap/engines/engine.py.229: process: 0.86602 | load: 0.00301 [I 2022-10-09 16:16:52] ...trap/engines/engine.py.230: loss: 0.14925 [I 2022-10-09 16:16:52] ...optimizers/trijoint.py.151: total_norm: 18.776335 [I 2022-10-09 16:17:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 2060/2304 [I 2022-10-09 16:17:01] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:27 [I 2022-10-09 16:17:01] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00293 [I 2022-10-09 16:17:01] ...trap/engines/engine.py.230: loss: 0.16740 [I 2022-10-09 16:17:01] ...optimizers/trijoint.py.151: total_norm: 15.689666 [I 2022-10-09 16:17:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 2070/2304 [I 2022-10-09 16:17:09] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:18 [I 2022-10-09 16:17:09] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00293 [I 2022-10-09 16:17:09] ...trap/engines/engine.py.230: loss: 0.11293 [I 2022-10-09 16:17:09] ...optimizers/trijoint.py.151: total_norm: 13.814083 [I 2022-10-09 16:17:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 2080/2304 [I 2022-10-09 16:17:18] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:10 [I 2022-10-09 16:17:18] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00295 [I 2022-10-09 16:17:18] ...trap/engines/engine.py.230: loss: 0.10175 [I 2022-10-09 16:17:18] ...optimizers/trijoint.py.151: total_norm: 10.135016 [I 2022-10-09 16:17:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 2090/2304 [I 2022-10-09 16:17:26] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:01 [I 2022-10-09 16:17:26] ...trap/engines/engine.py.229: process: 0.87321 | load: 0.00293 [I 2022-10-09 16:17:26] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-09 16:17:26] ...optimizers/trijoint.py.151: total_norm: 11.119699 [I 2022-10-09 16:17:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 2100/2304 [I 2022-10-09 16:17:35] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:53 [I 2022-10-09 16:17:35] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00298 [I 2022-10-09 16:17:35] ...trap/engines/engine.py.230: loss: 0.19472 [I 2022-10-09 16:17:35] ...optimizers/trijoint.py.151: total_norm: 18.262796 [I 2022-10-09 16:17:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 2110/2304 [I 2022-10-09 16:17:43] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:43 [I 2022-10-09 16:17:43] ...trap/engines/engine.py.229: process: 0.86337 | load: 0.00304 [I 2022-10-09 16:17:43] ...trap/engines/engine.py.230: loss: 0.12403 [I 2022-10-09 16:17:43] ...optimizers/trijoint.py.151: total_norm: 20.233803 [I 2022-10-09 16:17:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 2120/2304 [I 2022-10-09 16:17:51] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:02:36 [I 2022-10-09 16:17:51] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00297 [I 2022-10-09 16:17:51] ...trap/engines/engine.py.230: loss: 0.20355 [I 2022-10-09 16:17:51] ...optimizers/trijoint.py.151: total_norm: 16.149588 [I 2022-10-09 16:18:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 2130/2304 [I 2022-10-09 16:18:00] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:28 [I 2022-10-09 16:18:00] ...trap/engines/engine.py.229: process: 0.86858 | load: 0.00295 [I 2022-10-09 16:18:00] ...trap/engines/engine.py.230: loss: 0.17427 [I 2022-10-09 16:18:00] ...optimizers/trijoint.py.151: total_norm: 42.230125 [I 2022-10-09 16:18:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 2140/2304 [I 2022-10-09 16:18:09] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:20 [I 2022-10-09 16:18:09] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00291 [I 2022-10-09 16:18:09] ...trap/engines/engine.py.230: loss: 0.13406 [I 2022-10-09 16:18:09] ...optimizers/trijoint.py.151: total_norm: 8.798782 [I 2022-10-09 16:18:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 2150/2304 [I 2022-10-09 16:18:17] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:11 [I 2022-10-09 16:18:17] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00293 [I 2022-10-09 16:18:17] ...trap/engines/engine.py.230: loss: 0.21174 [I 2022-10-09 16:18:17] ...optimizers/trijoint.py.151: total_norm: 40.877476 [I 2022-10-09 16:18:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 2160/2304 [I 2022-10-09 16:18:26] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:02 [I 2022-10-09 16:18:26] ...trap/engines/engine.py.229: process: 0.84024 | load: 0.00298 [I 2022-10-09 16:18:26] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-09 16:18:26] ...optimizers/trijoint.py.151: total_norm: 16.767986 [I 2022-10-09 16:18:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 2170/2304 [I 2022-10-09 16:18:34] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:01:53 [I 2022-10-09 16:18:34] ...trap/engines/engine.py.229: process: 0.82973 | load: 0.00311 [I 2022-10-09 16:18:34] ...trap/engines/engine.py.230: loss: 0.13121 [I 2022-10-09 16:18:34] ...optimizers/trijoint.py.151: total_norm: 36.551659 [I 2022-10-09 16:18:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 2180/2304 [I 2022-10-09 16:18:43] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:01:44 [I 2022-10-09 16:18:43] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00309 [I 2022-10-09 16:18:43] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-09 16:18:43] ...optimizers/trijoint.py.151: total_norm: 13.477095 [I 2022-10-09 16:18:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 2190/2304 [I 2022-10-09 16:18:51] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:36 [I 2022-10-09 16:18:51] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00297 [I 2022-10-09 16:18:51] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-09 16:18:51] ...optimizers/trijoint.py.151: total_norm: 14.001513 [I 2022-10-09 16:19:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 2200/2304 [I 2022-10-09 16:19:00] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:01:28 [I 2022-10-09 16:19:00] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00294 [I 2022-10-09 16:19:00] ...trap/engines/engine.py.230: loss: 0.19795 [I 2022-10-09 16:19:00] ...optimizers/trijoint.py.151: total_norm: 18.601959 [I 2022-10-09 16:19:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 2210/2304 [I 2022-10-09 16:19:08] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:19 [I 2022-10-09 16:19:08] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00295 [I 2022-10-09 16:19:08] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-09 16:19:08] ...optimizers/trijoint.py.151: total_norm: 17.614355 [I 2022-10-09 16:19:16] ...trap/engines/engine.py.224: train: epoch 46 | batch 2220/2304 [I 2022-10-09 16:19:16] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:10 [I 2022-10-09 16:19:16] ...trap/engines/engine.py.229: process: 0.83023 | load: 0.00298 [I 2022-10-09 16:19:16] ...trap/engines/engine.py.230: loss: 0.18875 [I 2022-10-09 16:19:16] ...optimizers/trijoint.py.151: total_norm: 12.512242 [I 2022-10-09 16:19:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 2230/2304 [I 2022-10-09 16:19:25] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:02 [I 2022-10-09 16:19:25] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00296 [I 2022-10-09 16:19:25] ...trap/engines/engine.py.230: loss: 0.13355 [I 2022-10-09 16:19:25] ...optimizers/trijoint.py.151: total_norm: 25.086214 [I 2022-10-09 16:19:33] ...trap/engines/engine.py.224: train: epoch 46 | batch 2240/2304 [I 2022-10-09 16:19:33] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:00:54 [I 2022-10-09 16:19:33] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00298 [I 2022-10-09 16:19:33] ...trap/engines/engine.py.230: loss: 0.16618 [I 2022-10-09 16:19:33] ...optimizers/trijoint.py.151: total_norm: 26.347330 [I 2022-10-09 16:20:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 2250/2304 [I 2022-10-09 16:20:00] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:26 [I 2022-10-09 16:20:00] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00295 [I 2022-10-09 16:20:00] ...trap/engines/engine.py.230: loss: 0.15028 [I 2022-10-09 16:20:00] ...optimizers/trijoint.py.151: total_norm: 9.049372 [I 2022-10-09 16:20:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 2260/2304 [I 2022-10-09 16:20:08] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:40 [I 2022-10-09 16:20:08] ...trap/engines/engine.py.229: process: 0.83257 | load: 0.00299 [I 2022-10-09 16:20:08] ...trap/engines/engine.py.230: loss: 0.21902 [I 2022-10-09 16:20:08] ...optimizers/trijoint.py.151: total_norm: 19.379713 [I 2022-10-09 16:20:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 2270/2304 [I 2022-10-09 16:20:17] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:29 [I 2022-10-09 16:20:17] ...trap/engines/engine.py.229: process: 0.85702 | load: 0.00301 [I 2022-10-09 16:20:17] ...trap/engines/engine.py.230: loss: 0.11078 [I 2022-10-09 16:20:17] ...optimizers/trijoint.py.151: total_norm: 22.102264 [I 2022-10-09 16:20:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 2280/2304 [I 2022-10-09 16:20:25] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:20 [I 2022-10-09 16:20:25] ...trap/engines/engine.py.229: process: 0.83158 | load: 0.00298 [I 2022-10-09 16:20:25] ...trap/engines/engine.py.230: loss: 0.11549 [I 2022-10-09 16:20:25] ...optimizers/trijoint.py.151: total_norm: 21.899420 [I 2022-10-09 16:20:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 2290/2304 [I 2022-10-09 16:20:34] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:11 [I 2022-10-09 16:20:34] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00295 [I 2022-10-09 16:20:34] ...trap/engines/engine.py.230: loss: 0.21466 [I 2022-10-09 16:20:34] ...optimizers/trijoint.py.151: total_norm: 17.518394 [I 2022-10-09 16:20:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 2300/2304 [I 2022-10-09 16:20:42] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:03 [I 2022-10-09 16:20:42] ...trap/engines/engine.py.229: process: 0.82811 | load: 0.00016 [I 2022-10-09 16:20:42] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-09 16:20:42] ...optimizers/trijoint.py.151: total_norm: 21.039261 [I 2022-10-09 16:20:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 2304/2304 [I 2022-10-09 16:20:45] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:00 [I 2022-10-09 16:20:46] ...trap/engines/engine.py.229: process: 0.82683 | load: 0.00016 [I 2022-10-09 16:20:46] ...trap/engines/engine.py.230: loss: 0.16971 [I 2022-10-09 16:20:46] ...optimizers/trijoint.py.151: total_norm: 28.430880 [S 2022-10-09 16:20:47] ...trap/engines/engine.py.129: train_epoch.epoch: 46 [S 2022-10-09 16:20:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.15671810289267088 [I 2022-10-09 16:20:47] ...s/criterions/triplet.py.84: margin = 0.2800000000000001 [I 2022-10-09 16:20:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 46 [I 2022-10-09 16:20:48] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 16:20:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 16:20:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 16:20:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 16:20:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 16:20:51] ...trap/engines/engine.py.321: eval: epoch 46 | batch 0/510 [I 2022-10-09 16:20:51] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:09 [I 2022-10-09 16:20:51] ...trap/engines/engine.py.326: process: 3.90152 | load: 3.38363 [I 2022-10-09 16:20:56] ...trap/engines/engine.py.321: eval: epoch 46 | batch 10/510 [I 2022-10-09 16:20:56] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:45 [I 2022-10-09 16:20:56] ...trap/engines/engine.py.326: process: 0.42296 | load: 0.00019 [I 2022-10-09 16:21:00] ...trap/engines/engine.py.321: eval: epoch 46 | batch 20/510 [I 2022-10-09 16:21:00] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:00 [I 2022-10-09 16:21:00] ...trap/engines/engine.py.326: process: 0.47070 | load: 0.00019 [I 2022-10-09 16:21:07] ...trap/engines/engine.py.321: eval: epoch 46 | batch 30/510 [I 2022-10-09 16:21:07] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:35 [I 2022-10-09 16:21:07] ...trap/engines/engine.py.326: process: 0.83093 | load: 0.00019 [I 2022-10-09 16:21:14] ...trap/engines/engine.py.321: eval: epoch 46 | batch 40/510 [I 2022-10-09 16:21:14] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:16 [I 2022-10-09 16:21:14] ...trap/engines/engine.py.326: process: 0.56073 | load: 0.00018 [I 2022-10-09 16:21:19] ...trap/engines/engine.py.321: eval: epoch 46 | batch 50/510 [I 2022-10-09 16:21:19] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:17 [I 2022-10-09 16:21:19] ...trap/engines/engine.py.326: process: 0.60964 | load: 0.00018 [I 2022-10-09 16:21:24] ...trap/engines/engine.py.321: eval: epoch 46 | batch 60/510 [I 2022-10-09 16:21:24] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:44 [I 2022-10-09 16:21:24] ...trap/engines/engine.py.326: process: 0.41652 | load: 0.00022 [I 2022-10-09 16:21:28] ...trap/engines/engine.py.321: eval: epoch 46 | batch 70/510 [I 2022-10-09 16:21:28] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:24 [I 2022-10-09 16:21:28] ...trap/engines/engine.py.326: process: 0.45947 | load: 0.00018 [I 2022-10-09 16:21:33] ...trap/engines/engine.py.321: eval: epoch 46 | batch 80/510 [I 2022-10-09 16:21:33] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:04 [I 2022-10-09 16:21:33] ...trap/engines/engine.py.326: process: 0.45154 | load: 0.00019 [I 2022-10-09 16:21:37] ...trap/engines/engine.py.321: eval: epoch 46 | batch 90/510 [I 2022-10-09 16:21:37] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:05 [I 2022-10-09 16:21:37] ...trap/engines/engine.py.326: process: 0.45236 | load: 0.00019 [I 2022-10-09 16:21:41] ...trap/engines/engine.py.321: eval: epoch 46 | batch 100/510 [I 2022-10-09 16:21:41] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:54 [I 2022-10-09 16:21:41] ...trap/engines/engine.py.326: process: 0.30525 | load: 0.00019 [I 2022-10-09 16:21:45] ...trap/engines/engine.py.321: eval: epoch 46 | batch 110/510 [I 2022-10-09 16:21:45] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:10 [I 2022-10-09 16:21:45] ...trap/engines/engine.py.326: process: 0.31395 | load: 0.00018 [I 2022-10-09 16:21:48] ...trap/engines/engine.py.321: eval: epoch 46 | batch 120/510 [I 2022-10-09 16:21:48] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-09 16:21:48] ...trap/engines/engine.py.326: process: 0.31040 | load: 0.00018 [I 2022-10-09 16:21:51] ...trap/engines/engine.py.321: eval: epoch 46 | batch 130/510 [I 2022-10-09 16:21:51] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-09 16:21:51] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00018 [I 2022-10-09 16:21:54] ...trap/engines/engine.py.321: eval: epoch 46 | batch 140/510 [I 2022-10-09 16:21:54] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:56 [I 2022-10-09 16:21:54] ...trap/engines/engine.py.326: process: 0.31367 | load: 0.00019 [I 2022-10-09 16:21:57] ...trap/engines/engine.py.321: eval: epoch 46 | batch 150/510 [I 2022-10-09 16:21:57] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:53 [I 2022-10-09 16:21:57] ...trap/engines/engine.py.326: process: 0.31806 | load: 0.00018 [I 2022-10-09 16:22:00] ...trap/engines/engine.py.321: eval: epoch 46 | batch 160/510 [I 2022-10-09 16:22:00] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:49 [I 2022-10-09 16:22:00] ...trap/engines/engine.py.326: process: 0.31171 | load: 0.00018 [I 2022-10-09 16:22:04] ...trap/engines/engine.py.321: eval: epoch 46 | batch 170/510 [I 2022-10-09 16:22:04] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-09 16:22:04] ...trap/engines/engine.py.326: process: 0.31007 | load: 0.00018 [I 2022-10-09 16:22:07] ...trap/engines/engine.py.321: eval: epoch 46 | batch 180/510 [I 2022-10-09 16:22:07] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-09 16:22:07] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00018 [I 2022-10-09 16:22:10] ...trap/engines/engine.py.321: eval: epoch 46 | batch 190/510 [I 2022-10-09 16:22:10] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:41 [I 2022-10-09 16:22:10] ...trap/engines/engine.py.326: process: 0.31151 | load: 0.00020 [I 2022-10-09 16:22:13] ...trap/engines/engine.py.321: eval: epoch 46 | batch 200/510 [I 2022-10-09 16:22:13] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:38 [I 2022-10-09 16:22:13] ...trap/engines/engine.py.326: process: 0.31342 | load: 0.00019 [I 2022-10-09 16:22:16] ...trap/engines/engine.py.321: eval: epoch 46 | batch 210/510 [I 2022-10-09 16:22:16] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:34 [I 2022-10-09 16:22:16] ...trap/engines/engine.py.326: process: 0.31761 | load: 0.00019 [I 2022-10-09 16:22:19] ...trap/engines/engine.py.321: eval: epoch 46 | batch 220/510 [I 2022-10-09 16:22:19] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:31 [I 2022-10-09 16:22:19] ...trap/engines/engine.py.326: process: 0.31786 | load: 0.00020 [I 2022-10-09 16:22:23] ...trap/engines/engine.py.321: eval: epoch 46 | batch 230/510 [I 2022-10-09 16:22:23] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:29 [I 2022-10-09 16:22:23] ...trap/engines/engine.py.326: process: 0.31999 | load: 0.00020 [I 2022-10-09 16:22:26] ...trap/engines/engine.py.321: eval: epoch 46 | batch 240/510 [I 2022-10-09 16:22:26] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:25 [I 2022-10-09 16:22:26] ...trap/engines/engine.py.326: process: 0.31520 | load: 0.00019 [I 2022-10-09 16:22:29] ...trap/engines/engine.py.321: eval: epoch 46 | batch 250/510 [I 2022-10-09 16:22:29] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:22 [I 2022-10-09 16:22:29] ...trap/engines/engine.py.326: process: 0.31763 | load: 0.00019 [I 2022-10-09 16:22:32] ...trap/engines/engine.py.321: eval: epoch 46 | batch 260/510 [I 2022-10-09 16:22:32] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-09 16:22:32] ...trap/engines/engine.py.326: process: 0.31681 | load: 0.00021 [I 2022-10-09 16:22:35] ...trap/engines/engine.py.321: eval: epoch 46 | batch 270/510 [I 2022-10-09 16:22:35] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:15 [I 2022-10-09 16:22:35] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00019 [I 2022-10-09 16:22:39] ...trap/engines/engine.py.321: eval: epoch 46 | batch 280/510 [I 2022-10-09 16:22:39] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-09 16:22:39] ...trap/engines/engine.py.326: process: 0.32154 | load: 0.00019 [I 2022-10-09 16:22:42] ...trap/engines/engine.py.321: eval: epoch 46 | batch 290/510 [I 2022-10-09 16:22:42] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:09 [I 2022-10-09 16:22:42] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00020 [I 2022-10-09 16:22:45] ...trap/engines/engine.py.321: eval: epoch 46 | batch 300/510 [I 2022-10-09 16:22:45] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:06 [I 2022-10-09 16:22:45] ...trap/engines/engine.py.326: process: 0.31836 | load: 0.00019 [I 2022-10-09 16:22:48] ...trap/engines/engine.py.321: eval: epoch 46 | batch 310/510 [I 2022-10-09 16:22:48] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-09 16:22:48] ...trap/engines/engine.py.326: process: 0.31199 | load: 0.00020 [I 2022-10-09 16:22:51] ...trap/engines/engine.py.321: eval: epoch 46 | batch 320/510 [I 2022-10-09 16:22:51] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-09 16:22:51] ...trap/engines/engine.py.326: process: 0.31923 | load: 0.00019 [I 2022-10-09 16:22:54] ...trap/engines/engine.py.321: eval: epoch 46 | batch 330/510 [I 2022-10-09 16:22:54] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:57 [I 2022-10-09 16:22:54] ...trap/engines/engine.py.326: process: 0.31880 | load: 0.00019 [I 2022-10-09 16:22:58] ...trap/engines/engine.py.321: eval: epoch 46 | batch 340/510 [I 2022-10-09 16:22:58] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-09 16:22:58] ...trap/engines/engine.py.326: process: 0.31565 | load: 0.00019 [I 2022-10-09 16:23:01] ...trap/engines/engine.py.321: eval: epoch 46 | batch 350/510 [I 2022-10-09 16:23:01] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:50 [I 2022-10-09 16:23:01] ...trap/engines/engine.py.326: process: 0.31814 | load: 0.00020 [I 2022-10-09 16:23:04] ...trap/engines/engine.py.321: eval: epoch 46 | batch 360/510 [I 2022-10-09 16:23:04] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-09 16:23:04] ...trap/engines/engine.py.326: process: 0.31714 | load: 0.00020 [I 2022-10-09 16:23:07] ...trap/engines/engine.py.321: eval: epoch 46 | batch 370/510 [I 2022-10-09 16:23:07] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:44 [I 2022-10-09 16:23:07] ...trap/engines/engine.py.326: process: 0.30898 | load: 0.00020 [I 2022-10-09 16:23:10] ...trap/engines/engine.py.321: eval: epoch 46 | batch 380/510 [I 2022-10-09 16:23:10] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:41 [I 2022-10-09 16:23:10] ...trap/engines/engine.py.326: process: 0.32239 | load: 0.00021 [I 2022-10-09 16:23:13] ...trap/engines/engine.py.321: eval: epoch 46 | batch 390/510 [I 2022-10-09 16:23:13] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:38 [I 2022-10-09 16:23:13] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00020 [I 2022-10-09 16:23:17] ...trap/engines/engine.py.321: eval: epoch 46 | batch 400/510 [I 2022-10-09 16:23:17] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:34 [I 2022-10-09 16:23:17] ...trap/engines/engine.py.326: process: 0.31903 | load: 0.00020 [I 2022-10-09 16:23:20] ...trap/engines/engine.py.321: eval: epoch 46 | batch 410/510 [I 2022-10-09 16:23:20] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:31 [I 2022-10-09 16:23:20] ...trap/engines/engine.py.326: process: 0.31865 | load: 0.00020 [I 2022-10-09 16:23:23] ...trap/engines/engine.py.321: eval: epoch 46 | batch 420/510 [I 2022-10-09 16:23:23] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:28 [I 2022-10-09 16:23:23] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00018 [I 2022-10-09 16:23:26] ...trap/engines/engine.py.321: eval: epoch 46 | batch 430/510 [I 2022-10-09 16:23:26] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-09 16:23:26] ...trap/engines/engine.py.326: process: 0.31756 | load: 0.00019 [I 2022-10-09 16:23:29] ...trap/engines/engine.py.321: eval: epoch 46 | batch 440/510 [I 2022-10-09 16:23:29] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-09 16:23:29] ...trap/engines/engine.py.326: process: 0.32095 | load: 0.00020 [I 2022-10-09 16:23:33] ...trap/engines/engine.py.321: eval: epoch 46 | batch 450/510 [I 2022-10-09 16:23:33] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-09 16:23:33] ...trap/engines/engine.py.326: process: 0.31617 | load: 0.00019 [I 2022-10-09 16:23:36] ...trap/engines/engine.py.321: eval: epoch 46 | batch 460/510 [I 2022-10-09 16:23:36] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:16 [I 2022-10-09 16:23:36] ...trap/engines/engine.py.326: process: 0.32491 | load: 0.00019 [I 2022-10-09 16:23:39] ...trap/engines/engine.py.321: eval: epoch 46 | batch 470/510 [I 2022-10-09 16:23:39] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:12 [I 2022-10-09 16:23:39] ...trap/engines/engine.py.326: process: 0.31588 | load: 0.00020 [I 2022-10-09 16:23:42] ...trap/engines/engine.py.321: eval: epoch 46 | batch 480/510 [I 2022-10-09 16:23:42] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-09 16:23:42] ...trap/engines/engine.py.326: process: 0.31509 | load: 0.00019 [I 2022-10-09 16:23:45] ...trap/engines/engine.py.321: eval: epoch 46 | batch 490/510 [I 2022-10-09 16:23:45] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-09 16:23:45] ...trap/engines/engine.py.326: process: 0.31856 | load: 0.00020 [I 2022-10-09 16:23:49] ...trap/engines/engine.py.321: eval: epoch 46 | batch 500/510 [I 2022-10-09 16:23:49] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-09 16:23:49] ...trap/engines/engine.py.326: process: 0.32026 | load: 0.00019 [I 2022-10-09 16:23:52] ...trap/engines/engine.py.321: eval: epoch 46 | batch 510/510 [I 2022-10-09 16:23:52] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-09 16:23:52] ...trap/engines/engine.py.326: process: 0.31463 | load: 0.00015 [S 2022-10-09 16:23:52] ...trap/engines/engine.py.132: eval_epoch.epoch: 46 [I 2022-10-09 16:23:52] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.716800 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907100 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939600 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.706700 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903300 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.940300 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014627 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007687 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006576 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016823 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005405 [S 2022-10-09 16:24:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004051 [I 2022-10-09 16:24:27] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 16:24:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 16:24:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 16:24:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 16:24:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 16:24:31] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 16:24:31] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 16:24:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 16:24:37] ...trap/engines/engine.py.163: Training model on trainset for epoch 47 [I 2022-10-09 16:24:37] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 16:24:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 0/2304 [I 2022-10-09 16:24:40] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:49 [I 2022-10-09 16:24:40] ...trap/engines/engine.py.229: process: 2.73002 | load: 1.84401 [I 2022-10-09 16:24:40] ...trap/engines/engine.py.230: loss: 0.11620 [I 2022-10-09 16:24:40] ...optimizers/trijoint.py.151: total_norm: 14.932726 [I 2022-10-09 16:24:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 10/2304 [I 2022-10-09 16:24:48] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:08 [I 2022-10-09 16:24:48] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00673 [I 2022-10-09 16:24:48] ...trap/engines/engine.py.230: loss: 0.17441 [I 2022-10-09 16:24:48] ...optimizers/trijoint.py.151: total_norm: 21.665323 [I 2022-10-09 16:24:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 20/2304 [I 2022-10-09 16:24:57] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:02 [I 2022-10-09 16:24:57] ...trap/engines/engine.py.229: process: 0.85327 | load: 0.00561 [I 2022-10-09 16:24:57] ...trap/engines/engine.py.230: loss: 0.10013 [I 2022-10-09 16:24:57] ...optimizers/trijoint.py.151: total_norm: 9.022896 [I 2022-10-09 16:25:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 30/2304 [I 2022-10-09 16:25:05] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:14 [I 2022-10-09 16:25:05] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00376 [I 2022-10-09 16:25:05] ...trap/engines/engine.py.230: loss: 0.13032 [I 2022-10-09 16:25:05] ...optimizers/trijoint.py.151: total_norm: 16.460888 [I 2022-10-09 16:25:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 40/2304 [I 2022-10-09 16:25:14] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:00 [I 2022-10-09 16:25:14] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00445 [I 2022-10-09 16:25:14] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-09 16:25:14] ...optimizers/trijoint.py.151: total_norm: 13.664731 [I 2022-10-09 16:25:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 50/2304 [I 2022-10-09 16:25:22] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:48 [I 2022-10-09 16:25:22] ...trap/engines/engine.py.229: process: 0.83915 | load: 0.00357 [I 2022-10-09 16:25:22] ...trap/engines/engine.py.230: loss: 0.21873 [I 2022-10-09 16:25:22] ...optimizers/trijoint.py.151: total_norm: 20.655136 [I 2022-10-09 16:25:31] ...trap/engines/engine.py.224: train: epoch 47 | batch 60/2304 [I 2022-10-09 16:25:31] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:42 [I 2022-10-09 16:25:31] ...trap/engines/engine.py.229: process: 0.84257 | load: 0.00386 [I 2022-10-09 16:25:31] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-09 16:25:31] ...optimizers/trijoint.py.151: total_norm: 13.246769 [I 2022-10-09 16:25:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 70/2304 [I 2022-10-09 16:25:39] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:32 [I 2022-10-09 16:25:39] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00349 [I 2022-10-09 16:25:39] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-09 16:25:39] ...optimizers/trijoint.py.151: total_norm: 7.771361 [I 2022-10-09 16:25:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 80/2304 [I 2022-10-09 16:25:48] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:28 [I 2022-10-09 16:25:48] ...trap/engines/engine.py.229: process: 0.85671 | load: 0.00303 [I 2022-10-09 16:25:48] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-09 16:25:48] ...optimizers/trijoint.py.151: total_norm: 13.449220 [I 2022-10-09 16:25:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 90/2304 [I 2022-10-09 16:25:56] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:18 [I 2022-10-09 16:25:56] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00339 [I 2022-10-09 16:25:56] ...trap/engines/engine.py.230: loss: 0.15224 [I 2022-10-09 16:25:56] ...optimizers/trijoint.py.151: total_norm: 45.553383 [I 2022-10-09 16:26:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 100/2304 [I 2022-10-09 16:26:05] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:54 [I 2022-10-09 16:26:05] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00333 [I 2022-10-09 16:26:05] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-09 16:26:05] ...optimizers/trijoint.py.151: total_norm: 23.211828 [I 2022-10-09 16:26:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 110/2304 [I 2022-10-09 16:26:13] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:55 [I 2022-10-09 16:26:13] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00328 [I 2022-10-09 16:26:13] ...trap/engines/engine.py.230: loss: 0.16089 [I 2022-10-09 16:26:13] ...optimizers/trijoint.py.151: total_norm: 9.201887 [I 2022-10-09 16:26:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 120/2304 [I 2022-10-09 16:26:22] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:49 [I 2022-10-09 16:26:22] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00345 [I 2022-10-09 16:26:22] ...trap/engines/engine.py.230: loss: 0.18134 [I 2022-10-09 16:26:22] ...optimizers/trijoint.py.151: total_norm: 19.507267 [I 2022-10-09 16:26:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 130/2304 [I 2022-10-09 16:26:30] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:46 [I 2022-10-09 16:26:30] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00333 [I 2022-10-09 16:26:30] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-09 16:26:30] ...optimizers/trijoint.py.151: total_norm: 10.621248 [I 2022-10-09 16:26:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 140/2304 [I 2022-10-09 16:26:39] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:33 [I 2022-10-09 16:26:39] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00323 [I 2022-10-09 16:26:39] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-09 16:26:39] ...optimizers/trijoint.py.151: total_norm: 15.447545 [I 2022-10-09 16:26:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 150/2304 [I 2022-10-09 16:26:47] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:36 [I 2022-10-09 16:26:47] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00324 [I 2022-10-09 16:26:47] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-09 16:26:47] ...optimizers/trijoint.py.151: total_norm: 22.538395 [I 2022-10-09 16:26:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 160/2304 [I 2022-10-09 16:26:56] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:27 [I 2022-10-09 16:26:56] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00329 [I 2022-10-09 16:26:56] ...trap/engines/engine.py.230: loss: 0.20041 [I 2022-10-09 16:26:56] ...optimizers/trijoint.py.151: total_norm: 46.502228 [I 2022-10-09 16:27:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 170/2304 [I 2022-10-09 16:27:04] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:26 [I 2022-10-09 16:27:04] ...trap/engines/engine.py.229: process: 0.87155 | load: 0.00311 [I 2022-10-09 16:27:04] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-09 16:27:04] ...optimizers/trijoint.py.151: total_norm: 24.969078 [I 2022-10-09 16:27:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 180/2304 [I 2022-10-09 16:27:13] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:30:02 [I 2022-10-09 16:27:13] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00332 [I 2022-10-09 16:27:13] ...trap/engines/engine.py.230: loss: 0.13810 [I 2022-10-09 16:27:13] ...optimizers/trijoint.py.151: total_norm: 10.474836 [I 2022-10-09 16:27:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 190/2304 [I 2022-10-09 16:27:21] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:56 [I 2022-10-09 16:27:21] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00315 [I 2022-10-09 16:27:21] ...trap/engines/engine.py.230: loss: 0.18497 [I 2022-10-09 16:27:21] ...optimizers/trijoint.py.151: total_norm: 34.358036 [I 2022-10-09 16:27:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 200/2304 [I 2022-10-09 16:27:30] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:56 [I 2022-10-09 16:27:30] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00310 [I 2022-10-09 16:27:30] ...trap/engines/engine.py.230: loss: 0.15557 [I 2022-10-09 16:27:30] ...optimizers/trijoint.py.151: total_norm: 41.128357 [I 2022-10-09 16:27:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 210/2304 [I 2022-10-09 16:27:38] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:45 [I 2022-10-09 16:27:38] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00342 [I 2022-10-09 16:27:38] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-09 16:27:38] ...optimizers/trijoint.py.151: total_norm: 11.994961 [I 2022-10-09 16:27:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 220/2304 [I 2022-10-09 16:27:47] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:37 [I 2022-10-09 16:27:47] ...trap/engines/engine.py.229: process: 0.84772 | load: 0.00308 [I 2022-10-09 16:27:47] ...trap/engines/engine.py.230: loss: 0.12218 [I 2022-10-09 16:27:47] ...optimizers/trijoint.py.151: total_norm: 10.644129 [I 2022-10-09 16:27:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 230/2304 [I 2022-10-09 16:27:56] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:35 [I 2022-10-09 16:27:56] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00319 [I 2022-10-09 16:27:56] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-09 16:27:56] ...optimizers/trijoint.py.151: total_norm: 10.849493 [I 2022-10-09 16:28:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 240/2304 [I 2022-10-09 16:28:04] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:24 [I 2022-10-09 16:28:04] ...trap/engines/engine.py.229: process: 0.85616 | load: 0.00325 [I 2022-10-09 16:28:04] ...trap/engines/engine.py.230: loss: 0.19182 [I 2022-10-09 16:28:04] ...optimizers/trijoint.py.151: total_norm: 18.736546 [I 2022-10-09 16:28:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 250/2304 [I 2022-10-09 16:28:13] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:19 [I 2022-10-09 16:28:13] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00311 [I 2022-10-09 16:28:13] ...trap/engines/engine.py.230: loss: 0.16461 [I 2022-10-09 16:28:13] ...optimizers/trijoint.py.151: total_norm: 38.638538 [I 2022-10-09 16:28:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 260/2304 [I 2022-10-09 16:28:21] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:20 [I 2022-10-09 16:28:21] ...trap/engines/engine.py.229: process: 0.86834 | load: 0.00312 [I 2022-10-09 16:28:21] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-09 16:28:21] ...optimizers/trijoint.py.151: total_norm: 39.219566 [I 2022-10-09 16:28:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 270/2304 [I 2022-10-09 16:28:30] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:29:03 [I 2022-10-09 16:28:30] ...trap/engines/engine.py.229: process: 0.86731 | load: 0.00310 [I 2022-10-09 16:28:30] ...trap/engines/engine.py.230: loss: 0.15442 [I 2022-10-09 16:28:30] ...optimizers/trijoint.py.151: total_norm: 18.558382 [I 2022-10-09 16:28:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 280/2304 [I 2022-10-09 16:28:38] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:57 [I 2022-10-09 16:28:38] ...trap/engines/engine.py.229: process: 0.85918 | load: 0.00315 [I 2022-10-09 16:28:38] ...trap/engines/engine.py.230: loss: 0.18641 [I 2022-10-09 16:28:38] ...optimizers/trijoint.py.151: total_norm: 19.173080 [I 2022-10-09 16:28:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 290/2304 [I 2022-10-09 16:28:47] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:37 [I 2022-10-09 16:28:47] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00305 [I 2022-10-09 16:28:47] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-09 16:28:47] ...optimizers/trijoint.py.151: total_norm: 18.945101 [I 2022-10-09 16:28:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 300/2304 [I 2022-10-09 16:28:55] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:19 [I 2022-10-09 16:28:55] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00314 [I 2022-10-09 16:28:55] ...trap/engines/engine.py.230: loss: 0.24382 [I 2022-10-09 16:28:55] ...optimizers/trijoint.py.151: total_norm: 14.375969 [I 2022-10-09 16:29:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 310/2304 [I 2022-10-09 16:29:04] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:12 [I 2022-10-09 16:29:04] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00314 [I 2022-10-09 16:29:04] ...trap/engines/engine.py.230: loss: 0.17632 [I 2022-10-09 16:29:04] ...optimizers/trijoint.py.151: total_norm: 18.805471 [I 2022-10-09 16:29:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 320/2304 [I 2022-10-09 16:29:13] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:12 [I 2022-10-09 16:29:13] ...trap/engines/engine.py.229: process: 0.84284 | load: 0.00311 [I 2022-10-09 16:29:13] ...trap/engines/engine.py.230: loss: 0.19496 [I 2022-10-09 16:29:13] ...optimizers/trijoint.py.151: total_norm: 29.461138 [I 2022-10-09 16:29:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 330/2304 [I 2022-10-09 16:29:21] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:54 [I 2022-10-09 16:29:21] ...trap/engines/engine.py.229: process: 0.83969 | load: 0.00312 [I 2022-10-09 16:29:21] ...trap/engines/engine.py.230: loss: 0.13202 [I 2022-10-09 16:29:21] ...optimizers/trijoint.py.151: total_norm: 25.079924 [I 2022-10-09 16:29:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 340/2304 [I 2022-10-09 16:29:30] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:28:00 [I 2022-10-09 16:29:30] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00324 [I 2022-10-09 16:29:30] ...trap/engines/engine.py.230: loss: 0.18981 [I 2022-10-09 16:29:30] ...optimizers/trijoint.py.151: total_norm: 27.619978 [I 2022-10-09 16:29:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 350/2304 [I 2022-10-09 16:29:38] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:52 [I 2022-10-09 16:29:38] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00334 [I 2022-10-09 16:29:38] ...trap/engines/engine.py.230: loss: 0.12589 [I 2022-10-09 16:29:38] ...optimizers/trijoint.py.151: total_norm: 10.936801 [I 2022-10-09 16:29:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 360/2304 [I 2022-10-09 16:29:47] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:45 [I 2022-10-09 16:29:47] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00312 [I 2022-10-09 16:29:47] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-09 16:29:47] ...optimizers/trijoint.py.151: total_norm: 18.130985 [I 2022-10-09 16:29:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 370/2304 [I 2022-10-09 16:29:55] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:33 [I 2022-10-09 16:29:55] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00329 [I 2022-10-09 16:29:55] ...trap/engines/engine.py.230: loss: 0.11586 [I 2022-10-09 16:29:55] ...optimizers/trijoint.py.151: total_norm: 19.295195 [I 2022-10-09 16:30:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 380/2304 [I 2022-10-09 16:30:04] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:22 [I 2022-10-09 16:30:04] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00311 [I 2022-10-09 16:30:04] ...trap/engines/engine.py.230: loss: 0.16582 [I 2022-10-09 16:30:04] ...optimizers/trijoint.py.151: total_norm: 22.542042 [I 2022-10-09 16:30:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 390/2304 [I 2022-10-09 16:30:12] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:08 [I 2022-10-09 16:30:12] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00313 [I 2022-10-09 16:30:12] ...trap/engines/engine.py.230: loss: 0.15635 [I 2022-10-09 16:30:12] ...optimizers/trijoint.py.151: total_norm: 20.870197 [I 2022-10-09 16:30:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 400/2304 [I 2022-10-09 16:30:21] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:27:04 [I 2022-10-09 16:30:21] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00312 [I 2022-10-09 16:30:21] ...trap/engines/engine.py.230: loss: 0.17237 [I 2022-10-09 16:30:21] ...optimizers/trijoint.py.151: total_norm: 14.646312 [I 2022-10-09 16:30:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 410/2304 [I 2022-10-09 16:30:30] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:57 [I 2022-10-09 16:30:30] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00319 [I 2022-10-09 16:30:30] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-09 16:30:30] ...optimizers/trijoint.py.151: total_norm: 15.085480 [I 2022-10-09 16:30:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 420/2304 [I 2022-10-09 16:30:38] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:39 [I 2022-10-09 16:30:38] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00308 [I 2022-10-09 16:30:38] ...trap/engines/engine.py.230: loss: 0.18044 [I 2022-10-09 16:30:38] ...optimizers/trijoint.py.151: total_norm: 19.317265 [I 2022-10-09 16:30:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 430/2304 [I 2022-10-09 16:30:47] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:39 [I 2022-10-09 16:30:47] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00310 [I 2022-10-09 16:30:47] ...trap/engines/engine.py.230: loss: 0.25820 [I 2022-10-09 16:30:47] ...optimizers/trijoint.py.151: total_norm: 35.847321 [I 2022-10-09 16:30:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 440/2304 [I 2022-10-09 16:30:55] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:41 [I 2022-10-09 16:30:55] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00348 [I 2022-10-09 16:30:55] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-09 16:30:55] ...optimizers/trijoint.py.151: total_norm: 12.676044 [I 2022-10-09 16:31:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 450/2304 [I 2022-10-09 16:31:04] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:32 [I 2022-10-09 16:31:04] ...trap/engines/engine.py.229: process: 0.87474 | load: 0.00312 [I 2022-10-09 16:31:04] ...trap/engines/engine.py.230: loss: 0.14321 [I 2022-10-09 16:31:04] ...optimizers/trijoint.py.151: total_norm: 18.084459 [I 2022-10-09 16:31:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 460/2304 [I 2022-10-09 16:31:12] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:59 [I 2022-10-09 16:31:12] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00298 [I 2022-10-09 16:31:12] ...trap/engines/engine.py.230: loss: 0.14386 [I 2022-10-09 16:31:12] ...optimizers/trijoint.py.151: total_norm: 18.645962 [I 2022-10-09 16:31:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 470/2304 [I 2022-10-09 16:31:21] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:48 [I 2022-10-09 16:31:21] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00296 [I 2022-10-09 16:31:21] ...trap/engines/engine.py.230: loss: 0.10144 [I 2022-10-09 16:31:21] ...optimizers/trijoint.py.151: total_norm: 7.149147 [I 2022-10-09 16:31:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 480/2304 [I 2022-10-09 16:31:29] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:38 [I 2022-10-09 16:31:29] ...trap/engines/engine.py.229: process: 0.83760 | load: 0.00299 [I 2022-10-09 16:31:29] ...trap/engines/engine.py.230: loss: 0.13276 [I 2022-10-09 16:31:29] ...optimizers/trijoint.py.151: total_norm: 16.225965 [I 2022-10-09 16:31:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 490/2304 [I 2022-10-09 16:31:38] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:36 [I 2022-10-09 16:31:38] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00298 [I 2022-10-09 16:31:38] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-09 16:31:38] ...optimizers/trijoint.py.151: total_norm: 8.375036 [I 2022-10-09 16:31:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 500/2304 [I 2022-10-09 16:31:46] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:31 [I 2022-10-09 16:31:46] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00299 [I 2022-10-09 16:31:46] ...trap/engines/engine.py.230: loss: 0.15380 [I 2022-10-09 16:31:46] ...optimizers/trijoint.py.151: total_norm: 18.843460 [I 2022-10-09 16:31:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 510/2304 [I 2022-10-09 16:31:55] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:25 [I 2022-10-09 16:31:55] ...trap/engines/engine.py.229: process: 0.85072 | load: 0.00311 [I 2022-10-09 16:31:55] ...trap/engines/engine.py.230: loss: 0.13629 [I 2022-10-09 16:31:55] ...optimizers/trijoint.py.151: total_norm: 13.708643 [I 2022-10-09 16:32:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 520/2304 [I 2022-10-09 16:32:03] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:13 [I 2022-10-09 16:32:03] ...trap/engines/engine.py.229: process: 0.85222 | load: 0.00306 [I 2022-10-09 16:32:03] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-09 16:32:03] ...optimizers/trijoint.py.151: total_norm: 16.228617 [I 2022-10-09 16:32:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 530/2304 [I 2022-10-09 16:32:12] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:53 [I 2022-10-09 16:32:12] ...trap/engines/engine.py.229: process: 0.83550 | load: 0.00302 [I 2022-10-09 16:32:12] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-09 16:32:12] ...optimizers/trijoint.py.151: total_norm: 19.830360 [I 2022-10-09 16:32:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 540/2304 [I 2022-10-09 16:32:20] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:59 [I 2022-10-09 16:32:20] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00298 [I 2022-10-09 16:32:20] ...trap/engines/engine.py.230: loss: 0.25101 [I 2022-10-09 16:32:20] ...optimizers/trijoint.py.151: total_norm: 34.913364 [I 2022-10-09 16:32:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 550/2304 [I 2022-10-09 16:32:29] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:37 [I 2022-10-09 16:32:29] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00315 [I 2022-10-09 16:32:29] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-09 16:32:29] ...optimizers/trijoint.py.151: total_norm: 16.764360 [I 2022-10-09 16:32:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 560/2304 [I 2022-10-09 16:32:37] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:48 [I 2022-10-09 16:32:37] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00298 [I 2022-10-09 16:32:37] ...trap/engines/engine.py.230: loss: 0.17224 [I 2022-10-09 16:32:37] ...optimizers/trijoint.py.151: total_norm: 7.501864 [I 2022-10-09 16:32:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 570/2304 [I 2022-10-09 16:32:46] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:28 [I 2022-10-09 16:32:46] ...trap/engines/engine.py.229: process: 0.85714 | load: 0.00300 [I 2022-10-09 16:32:46] ...trap/engines/engine.py.230: loss: 0.10618 [I 2022-10-09 16:32:46] ...optimizers/trijoint.py.151: total_norm: 10.519666 [I 2022-10-09 16:32:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 580/2304 [I 2022-10-09 16:32:54] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:14 [I 2022-10-09 16:32:54] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00304 [I 2022-10-09 16:32:54] ...trap/engines/engine.py.230: loss: 0.19741 [I 2022-10-09 16:32:54] ...optimizers/trijoint.py.151: total_norm: 15.891542 [I 2022-10-09 16:33:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 590/2304 [I 2022-10-09 16:33:03] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:10 [I 2022-10-09 16:33:03] ...trap/engines/engine.py.229: process: 0.84574 | load: 0.00303 [I 2022-10-09 16:33:03] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-09 16:33:03] ...optimizers/trijoint.py.151: total_norm: 11.126935 [I 2022-10-09 16:33:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 600/2304 [I 2022-10-09 16:33:11] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:01 [I 2022-10-09 16:33:11] ...trap/engines/engine.py.229: process: 0.85345 | load: 0.00313 [I 2022-10-09 16:33:11] ...trap/engines/engine.py.230: loss: 0.12747 [I 2022-10-09 16:33:11] ...optimizers/trijoint.py.151: total_norm: 18.025604 [I 2022-10-09 16:33:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 610/2304 [I 2022-10-09 16:33:20] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:23:55 [I 2022-10-09 16:33:20] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00300 [I 2022-10-09 16:33:20] ...trap/engines/engine.py.230: loss: 0.11880 [I 2022-10-09 16:33:20] ...optimizers/trijoint.py.151: total_norm: 13.168747 [I 2022-10-09 16:33:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 620/2304 [I 2022-10-09 16:33:28] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:53 [I 2022-10-09 16:33:28] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00300 [I 2022-10-09 16:33:28] ...trap/engines/engine.py.230: loss: 0.17320 [I 2022-10-09 16:33:28] ...optimizers/trijoint.py.151: total_norm: 12.817995 [I 2022-10-09 16:33:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 630/2304 [I 2022-10-09 16:33:37] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:33 [I 2022-10-09 16:33:37] ...trap/engines/engine.py.229: process: 0.82563 | load: 0.00301 [I 2022-10-09 16:33:37] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-09 16:33:37] ...optimizers/trijoint.py.151: total_norm: 7.986524 [I 2022-10-09 16:33:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 640/2304 [I 2022-10-09 16:33:45] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:31 [I 2022-10-09 16:33:45] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00304 [I 2022-10-09 16:33:45] ...trap/engines/engine.py.230: loss: 0.21279 [I 2022-10-09 16:33:45] ...optimizers/trijoint.py.151: total_norm: 15.268864 [I 2022-10-09 16:33:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 650/2304 [I 2022-10-09 16:33:54] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:20 [I 2022-10-09 16:33:54] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00300 [I 2022-10-09 16:33:54] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-09 16:33:54] ...optimizers/trijoint.py.151: total_norm: 11.611482 [I 2022-10-09 16:34:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 660/2304 [I 2022-10-09 16:34:02] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:19 [I 2022-10-09 16:34:02] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00314 [I 2022-10-09 16:34:02] ...trap/engines/engine.py.230: loss: 0.18756 [I 2022-10-09 16:34:02] ...optimizers/trijoint.py.151: total_norm: 18.381010 [I 2022-10-09 16:34:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 670/2304 [I 2022-10-09 16:34:11] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:11 [I 2022-10-09 16:34:11] ...trap/engines/engine.py.229: process: 0.85680 | load: 0.00309 [I 2022-10-09 16:34:11] ...trap/engines/engine.py.230: loss: 0.16621 [I 2022-10-09 16:34:11] ...optimizers/trijoint.py.151: total_norm: 22.921158 [I 2022-10-09 16:34:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 680/2304 [I 2022-10-09 16:34:19] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:03 [I 2022-10-09 16:34:19] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00311 [I 2022-10-09 16:34:19] ...trap/engines/engine.py.230: loss: 0.17226 [I 2022-10-09 16:34:19] ...optimizers/trijoint.py.151: total_norm: 24.693686 [I 2022-10-09 16:34:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 690/2304 [I 2022-10-09 16:34:28] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:51 [I 2022-10-09 16:34:28] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00308 [I 2022-10-09 16:34:28] ...trap/engines/engine.py.230: loss: 0.16725 [I 2022-10-09 16:34:28] ...optimizers/trijoint.py.151: total_norm: 16.372562 [I 2022-10-09 16:34:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 700/2304 [I 2022-10-09 16:34:36] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:48 [I 2022-10-09 16:34:36] ...trap/engines/engine.py.229: process: 0.85763 | load: 0.00310 [I 2022-10-09 16:34:36] ...trap/engines/engine.py.230: loss: 0.19237 [I 2022-10-09 16:34:36] ...optimizers/trijoint.py.151: total_norm: 22.282972 [I 2022-10-09 16:34:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 710/2304 [I 2022-10-09 16:34:45] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:37 [I 2022-10-09 16:34:45] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00313 [I 2022-10-09 16:34:45] ...trap/engines/engine.py.230: loss: 0.16570 [I 2022-10-09 16:34:45] ...optimizers/trijoint.py.151: total_norm: 8.342765 [I 2022-10-09 16:34:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 720/2304 [I 2022-10-09 16:34:53] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:33 [I 2022-10-09 16:34:53] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00331 [I 2022-10-09 16:34:53] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-09 16:34:53] ...optimizers/trijoint.py.151: total_norm: 14.686167 [I 2022-10-09 16:35:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 730/2304 [I 2022-10-09 16:35:02] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:18 [I 2022-10-09 16:35:02] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00316 [I 2022-10-09 16:35:02] ...trap/engines/engine.py.230: loss: 0.14861 [I 2022-10-09 16:35:02] ...optimizers/trijoint.py.151: total_norm: 22.502056 [I 2022-10-09 16:35:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 740/2304 [I 2022-10-09 16:35:10] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:10 [I 2022-10-09 16:35:10] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00313 [I 2022-10-09 16:35:10] ...trap/engines/engine.py.230: loss: 0.18002 [I 2022-10-09 16:35:11] ...optimizers/trijoint.py.151: total_norm: 21.784924 [I 2022-10-09 16:35:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 750/2304 [I 2022-10-09 16:35:19] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:05 [I 2022-10-09 16:35:19] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00310 [I 2022-10-09 16:35:19] ...trap/engines/engine.py.230: loss: 0.09353 [I 2022-10-09 16:35:19] ...optimizers/trijoint.py.151: total_norm: 8.436313 [I 2022-10-09 16:35:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 760/2304 [I 2022-10-09 16:35:28] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:55 [I 2022-10-09 16:35:28] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00314 [I 2022-10-09 16:35:28] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-09 16:35:28] ...optimizers/trijoint.py.151: total_norm: 12.571429 [I 2022-10-09 16:35:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 770/2304 [I 2022-10-09 16:35:36] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:49 [I 2022-10-09 16:35:36] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00314 [I 2022-10-09 16:35:36] ...trap/engines/engine.py.230: loss: 0.11528 [I 2022-10-09 16:35:36] ...optimizers/trijoint.py.151: total_norm: 8.764891 [I 2022-10-09 16:35:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 780/2304 [I 2022-10-09 16:35:45] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:35 [I 2022-10-09 16:35:45] ...trap/engines/engine.py.229: process: 0.86262 | load: 0.00311 [I 2022-10-09 16:35:45] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-09 16:35:45] ...optimizers/trijoint.py.151: total_norm: 10.057582 [I 2022-10-09 16:35:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 790/2304 [I 2022-10-09 16:35:53] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:30 [I 2022-10-09 16:35:53] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00312 [I 2022-10-09 16:35:53] ...trap/engines/engine.py.230: loss: 0.11521 [I 2022-10-09 16:35:53] ...optimizers/trijoint.py.151: total_norm: 11.325755 [I 2022-10-09 16:36:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 800/2304 [I 2022-10-09 16:36:02] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:22 [I 2022-10-09 16:36:02] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00304 [I 2022-10-09 16:36:02] ...trap/engines/engine.py.230: loss: 0.21338 [I 2022-10-09 16:36:02] ...optimizers/trijoint.py.151: total_norm: 22.616192 [I 2022-10-09 16:36:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 810/2304 [I 2022-10-09 16:36:10] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:14 [I 2022-10-09 16:36:10] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00297 [I 2022-10-09 16:36:10] ...trap/engines/engine.py.230: loss: 0.22508 [I 2022-10-09 16:36:10] ...optimizers/trijoint.py.151: total_norm: 16.039797 [I 2022-10-09 16:36:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 820/2304 [I 2022-10-09 16:36:19] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:03 [I 2022-10-09 16:36:19] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00311 [I 2022-10-09 16:36:19] ...trap/engines/engine.py.230: loss: 0.16960 [I 2022-10-09 16:36:19] ...optimizers/trijoint.py.151: total_norm: 28.015192 [I 2022-10-09 16:36:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 830/2304 [I 2022-10-09 16:36:27] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:55 [I 2022-10-09 16:36:27] ...trap/engines/engine.py.229: process: 0.85590 | load: 0.00310 [I 2022-10-09 16:36:27] ...trap/engines/engine.py.230: loss: 0.17404 [I 2022-10-09 16:36:27] ...optimizers/trijoint.py.151: total_norm: 12.525327 [I 2022-10-09 16:36:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 840/2304 [I 2022-10-09 16:36:36] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:56 [I 2022-10-09 16:36:36] ...trap/engines/engine.py.229: process: 0.87232 | load: 0.00310 [I 2022-10-09 16:36:36] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-09 16:36:36] ...optimizers/trijoint.py.151: total_norm: 18.826185 [I 2022-10-09 16:36:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 850/2304 [I 2022-10-09 16:36:44] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:40 [I 2022-10-09 16:36:44] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00312 [I 2022-10-09 16:36:44] ...trap/engines/engine.py.230: loss: 0.15513 [I 2022-10-09 16:36:44] ...optimizers/trijoint.py.151: total_norm: 11.800056 [I 2022-10-09 16:36:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 860/2304 [I 2022-10-09 16:36:53] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:29 [I 2022-10-09 16:36:53] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00310 [I 2022-10-09 16:36:53] ...trap/engines/engine.py.230: loss: 0.09864 [I 2022-10-09 16:36:53] ...optimizers/trijoint.py.151: total_norm: 6.357249 [I 2022-10-09 16:37:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 870/2304 [I 2022-10-09 16:37:01] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:26 [I 2022-10-09 16:37:01] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00330 [I 2022-10-09 16:37:01] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-09 16:37:01] ...optimizers/trijoint.py.151: total_norm: 10.412719 [I 2022-10-09 16:37:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 880/2304 [I 2022-10-09 16:37:10] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:18 [I 2022-10-09 16:37:10] ...trap/engines/engine.py.229: process: 0.86049 | load: 0.00313 [I 2022-10-09 16:37:10] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-09 16:37:10] ...optimizers/trijoint.py.151: total_norm: 23.436853 [I 2022-10-09 16:37:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 890/2304 [I 2022-10-09 16:37:19] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:02 [I 2022-10-09 16:37:19] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00312 [I 2022-10-09 16:37:19] ...trap/engines/engine.py.230: loss: 0.16565 [I 2022-10-09 16:37:19] ...optimizers/trijoint.py.151: total_norm: 22.733780 [I 2022-10-09 16:37:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 900/2304 [I 2022-10-09 16:37:27] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:52 [I 2022-10-09 16:37:27] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00310 [I 2022-10-09 16:37:27] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-09 16:37:27] ...optimizers/trijoint.py.151: total_norm: 7.686289 [I 2022-10-09 16:37:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 910/2304 [I 2022-10-09 16:37:36] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:44 [I 2022-10-09 16:37:36] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00311 [I 2022-10-09 16:37:36] ...trap/engines/engine.py.230: loss: 0.15442 [I 2022-10-09 16:37:36] ...optimizers/trijoint.py.151: total_norm: 13.234822 [I 2022-10-09 16:37:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 920/2304 [I 2022-10-09 16:37:44] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:38 [I 2022-10-09 16:37:44] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00312 [I 2022-10-09 16:37:44] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-09 16:37:44] ...optimizers/trijoint.py.151: total_norm: 8.127872 [I 2022-10-09 16:37:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 930/2304 [I 2022-10-09 16:37:53] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:25 [I 2022-10-09 16:37:53] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00311 [I 2022-10-09 16:37:53] ...trap/engines/engine.py.230: loss: 0.15961 [I 2022-10-09 16:37:53] ...optimizers/trijoint.py.151: total_norm: 14.570460 [I 2022-10-09 16:38:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 940/2304 [I 2022-10-09 16:38:01] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:20 [I 2022-10-09 16:38:01] ...trap/engines/engine.py.229: process: 0.85852 | load: 0.00313 [I 2022-10-09 16:38:01] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-09 16:38:01] ...optimizers/trijoint.py.151: total_norm: 13.076720 [I 2022-10-09 16:38:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 950/2304 [I 2022-10-09 16:38:10] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:11 [I 2022-10-09 16:38:10] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00308 [I 2022-10-09 16:38:10] ...trap/engines/engine.py.230: loss: 0.20909 [I 2022-10-09 16:38:10] ...optimizers/trijoint.py.151: total_norm: 27.444818 [I 2022-10-09 16:38:18] ...trap/engines/engine.py.224: train: epoch 47 | batch 960/2304 [I 2022-10-09 16:38:18] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:06 [I 2022-10-09 16:38:18] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00310 [I 2022-10-09 16:38:18] ...trap/engines/engine.py.230: loss: 0.12198 [I 2022-10-09 16:38:18] ...optimizers/trijoint.py.151: total_norm: 13.214550 [I 2022-10-09 16:38:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 970/2304 [I 2022-10-09 16:38:27] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:51 [I 2022-10-09 16:38:27] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00309 [I 2022-10-09 16:38:27] ...trap/engines/engine.py.230: loss: 0.15428 [I 2022-10-09 16:38:27] ...optimizers/trijoint.py.151: total_norm: 17.443462 [I 2022-10-09 16:38:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 980/2304 [I 2022-10-09 16:38:35] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:38 [I 2022-10-09 16:38:35] ...trap/engines/engine.py.229: process: 0.85359 | load: 0.00292 [I 2022-10-09 16:38:35] ...trap/engines/engine.py.230: loss: 0.17464 [I 2022-10-09 16:38:35] ...optimizers/trijoint.py.151: total_norm: 17.527237 [I 2022-10-09 16:38:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 990/2304 [I 2022-10-09 16:38:44] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:26 [I 2022-10-09 16:38:44] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00295 [I 2022-10-09 16:38:44] ...trap/engines/engine.py.230: loss: 0.16594 [I 2022-10-09 16:38:44] ...optimizers/trijoint.py.151: total_norm: 25.624075 [I 2022-10-09 16:38:52] ...trap/engines/engine.py.224: train: epoch 47 | batch 1000/2304 [I 2022-10-09 16:38:52] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:15 [I 2022-10-09 16:38:52] ...trap/engines/engine.py.229: process: 0.82927 | load: 0.00297 [I 2022-10-09 16:38:52] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-09 16:38:52] ...optimizers/trijoint.py.151: total_norm: 11.603918 [I 2022-10-09 16:39:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 1010/2304 [I 2022-10-09 16:39:01] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:11 [I 2022-10-09 16:39:01] ...trap/engines/engine.py.229: process: 0.84745 | load: 0.00299 [I 2022-10-09 16:39:01] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-09 16:39:01] ...optimizers/trijoint.py.151: total_norm: 16.029310 [I 2022-10-09 16:39:09] ...trap/engines/engine.py.224: train: epoch 47 | batch 1020/2304 [I 2022-10-09 16:39:09] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:04 [I 2022-10-09 16:39:09] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00295 [I 2022-10-09 16:39:09] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-09 16:39:09] ...optimizers/trijoint.py.151: total_norm: 12.968988 [I 2022-10-09 16:39:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 1030/2304 [I 2022-10-09 16:39:17] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:55 [I 2022-10-09 16:39:17] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00299 [I 2022-10-09 16:39:17] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-09 16:39:17] ...optimizers/trijoint.py.151: total_norm: 16.281456 [I 2022-10-09 16:39:26] ...trap/engines/engine.py.224: train: epoch 47 | batch 1040/2304 [I 2022-10-09 16:39:26] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:51 [I 2022-10-09 16:39:26] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00297 [I 2022-10-09 16:39:26] ...trap/engines/engine.py.230: loss: 0.21672 [I 2022-10-09 16:39:26] ...optimizers/trijoint.py.151: total_norm: 16.843569 [I 2022-10-09 16:39:34] ...trap/engines/engine.py.224: train: epoch 47 | batch 1050/2304 [I 2022-10-09 16:39:34] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:37 [I 2022-10-09 16:39:34] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00301 [I 2022-10-09 16:39:34] ...trap/engines/engine.py.230: loss: 0.15514 [I 2022-10-09 16:39:34] ...optimizers/trijoint.py.151: total_norm: 45.195786 [I 2022-10-09 16:39:43] ...trap/engines/engine.py.224: train: epoch 47 | batch 1060/2304 [I 2022-10-09 16:39:43] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:23 [I 2022-10-09 16:39:43] ...trap/engines/engine.py.229: process: 0.83802 | load: 0.00297 [I 2022-10-09 16:39:43] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-09 16:39:43] ...optimizers/trijoint.py.151: total_norm: 21.780312 [I 2022-10-09 16:39:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 1070/2304 [I 2022-10-09 16:39:51] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:19 [I 2022-10-09 16:39:51] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00297 [I 2022-10-09 16:39:51] ...trap/engines/engine.py.230: loss: 0.11752 [I 2022-10-09 16:39:51] ...optimizers/trijoint.py.151: total_norm: 10.825738 [I 2022-10-09 16:40:00] ...trap/engines/engine.py.224: train: epoch 47 | batch 1080/2304 [I 2022-10-09 16:40:00] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:10 [I 2022-10-09 16:40:00] ...trap/engines/engine.py.229: process: 0.83406 | load: 0.00295 [I 2022-10-09 16:40:00] ...trap/engines/engine.py.230: loss: 0.17240 [I 2022-10-09 16:40:00] ...optimizers/trijoint.py.151: total_norm: 12.241312 [I 2022-10-09 16:40:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 1090/2304 [I 2022-10-09 16:40:08] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:09 [I 2022-10-09 16:40:08] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00297 [I 2022-10-09 16:40:08] ...trap/engines/engine.py.230: loss: 0.18477 [I 2022-10-09 16:40:08] ...optimizers/trijoint.py.151: total_norm: 17.936886 [I 2022-10-09 16:40:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 1100/2304 [I 2022-10-09 16:40:17] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:55 [I 2022-10-09 16:40:17] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00297 [I 2022-10-09 16:40:17] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-09 16:40:17] ...optimizers/trijoint.py.151: total_norm: 14.281011 [I 2022-10-09 16:40:25] ...trap/engines/engine.py.224: train: epoch 47 | batch 1110/2304 [I 2022-10-09 16:40:25] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:46 [I 2022-10-09 16:40:25] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00302 [I 2022-10-09 16:40:25] ...trap/engines/engine.py.230: loss: 0.22242 [I 2022-10-09 16:40:25] ...optimizers/trijoint.py.151: total_norm: 18.601320 [I 2022-10-09 16:40:34] ...trap/engines/engine.py.224: train: epoch 47 | batch 1120/2304 [I 2022-10-09 16:40:34] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:42 [I 2022-10-09 16:40:34] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00301 [I 2022-10-09 16:40:34] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-09 16:40:34] ...optimizers/trijoint.py.151: total_norm: 12.900455 [I 2022-10-09 16:40:42] ...trap/engines/engine.py.224: train: epoch 47 | batch 1130/2304 [I 2022-10-09 16:40:42] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:35 [I 2022-10-09 16:40:42] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00313 [I 2022-10-09 16:40:42] ...trap/engines/engine.py.230: loss: 0.18867 [I 2022-10-09 16:40:42] ...optimizers/trijoint.py.151: total_norm: 17.146301 [I 2022-10-09 16:40:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 1140/2304 [I 2022-10-09 16:40:51] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:31 [I 2022-10-09 16:40:51] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00312 [I 2022-10-09 16:40:51] ...trap/engines/engine.py.230: loss: 0.22599 [I 2022-10-09 16:40:51] ...optimizers/trijoint.py.151: total_norm: 32.449612 [I 2022-10-09 16:40:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 1150/2304 [I 2022-10-09 16:40:59] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:22 [I 2022-10-09 16:40:59] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00314 [I 2022-10-09 16:40:59] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-09 16:40:59] ...optimizers/trijoint.py.151: total_norm: 42.787697 [I 2022-10-09 16:41:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 1160/2304 [I 2022-10-09 16:41:08] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:16 [I 2022-10-09 16:41:08] ...trap/engines/engine.py.229: process: 0.86124 | load: 0.00312 [I 2022-10-09 16:41:08] ...trap/engines/engine.py.230: loss: 0.21849 [I 2022-10-09 16:41:08] ...optimizers/trijoint.py.151: total_norm: 30.697340 [I 2022-10-09 16:41:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1170/2304 [I 2022-10-09 16:41:16] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:08 [I 2022-10-09 16:41:16] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00312 [I 2022-10-09 16:41:16] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-09 16:41:16] ...optimizers/trijoint.py.151: total_norm: 30.793699 [I 2022-10-09 16:41:25] ...trap/engines/engine.py.224: train: epoch 47 | batch 1180/2304 [I 2022-10-09 16:41:25] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:57 [I 2022-10-09 16:41:25] ...trap/engines/engine.py.229: process: 0.85266 | load: 0.00312 [I 2022-10-09 16:41:25] ...trap/engines/engine.py.230: loss: 0.12975 [I 2022-10-09 16:41:25] ...optimizers/trijoint.py.151: total_norm: 14.749655 [I 2022-10-09 16:41:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1190/2304 [I 2022-10-09 16:41:33] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:41 [I 2022-10-09 16:41:33] ...trap/engines/engine.py.229: process: 0.86089 | load: 0.00314 [I 2022-10-09 16:41:33] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-09 16:41:33] ...optimizers/trijoint.py.151: total_norm: 12.259531 [I 2022-10-09 16:41:42] ...trap/engines/engine.py.224: train: epoch 47 | batch 1200/2304 [I 2022-10-09 16:41:42] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:40 [I 2022-10-09 16:41:42] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00314 [I 2022-10-09 16:41:42] ...trap/engines/engine.py.230: loss: 0.12578 [I 2022-10-09 16:41:42] ...optimizers/trijoint.py.151: total_norm: 17.322910 [I 2022-10-09 16:41:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 1210/2304 [I 2022-10-09 16:41:50] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:33 [I 2022-10-09 16:41:50] ...trap/engines/engine.py.229: process: 0.86364 | load: 0.00312 [I 2022-10-09 16:41:50] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-09 16:41:50] ...optimizers/trijoint.py.151: total_norm: 33.585999 [I 2022-10-09 16:41:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 1220/2304 [I 2022-10-09 16:41:59] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:28 [I 2022-10-09 16:41:59] ...trap/engines/engine.py.229: process: 0.86338 | load: 0.00315 [I 2022-10-09 16:41:59] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-09 16:41:59] ...optimizers/trijoint.py.151: total_norm: 10.417613 [I 2022-10-09 16:42:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 1230/2304 [I 2022-10-09 16:42:07] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:12 [I 2022-10-09 16:42:07] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00312 [I 2022-10-09 16:42:07] ...trap/engines/engine.py.230: loss: 0.12158 [I 2022-10-09 16:42:07] ...optimizers/trijoint.py.151: total_norm: 11.570446 [I 2022-10-09 16:42:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1240/2304 [I 2022-10-09 16:42:16] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:10 [I 2022-10-09 16:42:16] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00312 [I 2022-10-09 16:42:16] ...trap/engines/engine.py.230: loss: 0.11764 [I 2022-10-09 16:42:16] ...optimizers/trijoint.py.151: total_norm: 11.311055 [I 2022-10-09 16:42:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1250/2304 [I 2022-10-09 16:42:24] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:56 [I 2022-10-09 16:42:24] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00298 [I 2022-10-09 16:42:24] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-09 16:42:24] ...optimizers/trijoint.py.151: total_norm: 13.888864 [I 2022-10-09 16:42:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1260/2304 [I 2022-10-09 16:42:33] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:42 [I 2022-10-09 16:42:33] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00314 [I 2022-10-09 16:42:33] ...trap/engines/engine.py.230: loss: 0.11808 [I 2022-10-09 16:42:33] ...optimizers/trijoint.py.151: total_norm: 27.150324 [I 2022-10-09 16:42:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1270/2304 [I 2022-10-09 16:42:41] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:39 [I 2022-10-09 16:42:41] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00303 [I 2022-10-09 16:42:41] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-09 16:42:41] ...optimizers/trijoint.py.151: total_norm: 7.403239 [I 2022-10-09 16:42:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 1280/2304 [I 2022-10-09 16:42:50] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:35 [I 2022-10-09 16:42:50] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00311 [I 2022-10-09 16:42:50] ...trap/engines/engine.py.230: loss: 0.20668 [I 2022-10-09 16:42:50] ...optimizers/trijoint.py.151: total_norm: 23.320520 [I 2022-10-09 16:42:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 1290/2304 [I 2022-10-09 16:42:59] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:24 [I 2022-10-09 16:42:59] ...trap/engines/engine.py.229: process: 0.84626 | load: 0.00314 [I 2022-10-09 16:42:59] ...trap/engines/engine.py.230: loss: 0.21210 [I 2022-10-09 16:42:59] ...optimizers/trijoint.py.151: total_norm: 13.489373 [I 2022-10-09 16:43:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 1300/2304 [I 2022-10-09 16:43:07] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:20 [I 2022-10-09 16:43:07] ...trap/engines/engine.py.229: process: 0.86394 | load: 0.00312 [I 2022-10-09 16:43:07] ...trap/engines/engine.py.230: loss: 0.14286 [I 2022-10-09 16:43:07] ...optimizers/trijoint.py.151: total_norm: 11.811142 [I 2022-10-09 16:43:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1310/2304 [I 2022-10-09 16:43:16] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:13 [I 2022-10-09 16:43:16] ...trap/engines/engine.py.229: process: 0.85451 | load: 0.00311 [I 2022-10-09 16:43:16] ...trap/engines/engine.py.230: loss: 0.17620 [I 2022-10-09 16:43:16] ...optimizers/trijoint.py.151: total_norm: 12.077686 [I 2022-10-09 16:43:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1320/2304 [I 2022-10-09 16:43:24] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:01 [I 2022-10-09 16:43:24] ...trap/engines/engine.py.229: process: 0.87023 | load: 0.00308 [I 2022-10-09 16:43:24] ...trap/engines/engine.py.230: loss: 0.16781 [I 2022-10-09 16:43:24] ...optimizers/trijoint.py.151: total_norm: 18.712482 [I 2022-10-09 16:43:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1330/2304 [I 2022-10-09 16:43:33] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:51 [I 2022-10-09 16:43:33] ...trap/engines/engine.py.229: process: 0.83716 | load: 0.00313 [I 2022-10-09 16:43:33] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-09 16:43:33] ...optimizers/trijoint.py.151: total_norm: 15.188015 [I 2022-10-09 16:43:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1340/2304 [I 2022-10-09 16:43:41] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:42 [I 2022-10-09 16:43:41] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00313 [I 2022-10-09 16:43:41] ...trap/engines/engine.py.230: loss: 0.17031 [I 2022-10-09 16:43:41] ...optimizers/trijoint.py.151: total_norm: 15.132652 [I 2022-10-09 16:43:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 1350/2304 [I 2022-10-09 16:43:50] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:34 [I 2022-10-09 16:43:50] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00317 [I 2022-10-09 16:43:50] ...trap/engines/engine.py.230: loss: 0.13591 [I 2022-10-09 16:43:50] ...optimizers/trijoint.py.151: total_norm: 26.377342 [I 2022-10-09 16:43:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1360/2304 [I 2022-10-09 16:43:58] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:22 [I 2022-10-09 16:43:58] ...trap/engines/engine.py.229: process: 0.83409 | load: 0.00334 [I 2022-10-09 16:43:58] ...trap/engines/engine.py.230: loss: 0.09123 [I 2022-10-09 16:43:58] ...optimizers/trijoint.py.151: total_norm: 10.779816 [I 2022-10-09 16:44:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 1370/2304 [I 2022-10-09 16:44:07] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:12 [I 2022-10-09 16:44:07] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00315 [I 2022-10-09 16:44:07] ...trap/engines/engine.py.230: loss: 0.17062 [I 2022-10-09 16:44:07] ...optimizers/trijoint.py.151: total_norm: 16.562141 [I 2022-10-09 16:44:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1380/2304 [I 2022-10-09 16:44:16] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:03 [I 2022-10-09 16:44:16] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00316 [I 2022-10-09 16:44:16] ...trap/engines/engine.py.230: loss: 0.21332 [I 2022-10-09 16:44:16] ...optimizers/trijoint.py.151: total_norm: 38.126411 [I 2022-10-09 16:44:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1390/2304 [I 2022-10-09 16:44:24] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:54 [I 2022-10-09 16:44:24] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00328 [I 2022-10-09 16:44:24] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-09 16:44:24] ...optimizers/trijoint.py.151: total_norm: 7.834773 [I 2022-10-09 16:44:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1400/2304 [I 2022-10-09 16:44:33] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:56 [I 2022-10-09 16:44:33] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00312 [I 2022-10-09 16:44:33] ...trap/engines/engine.py.230: loss: 0.13242 [I 2022-10-09 16:44:33] ...optimizers/trijoint.py.151: total_norm: 17.005112 [I 2022-10-09 16:44:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1410/2304 [I 2022-10-09 16:44:41] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:43 [I 2022-10-09 16:44:41] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00313 [I 2022-10-09 16:44:41] ...trap/engines/engine.py.230: loss: 0.16261 [I 2022-10-09 16:44:41] ...optimizers/trijoint.py.151: total_norm: 11.888474 [I 2022-10-09 16:44:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 1420/2304 [I 2022-10-09 16:44:50] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:32 [I 2022-10-09 16:44:50] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00316 [I 2022-10-09 16:44:50] ...trap/engines/engine.py.230: loss: 0.20699 [I 2022-10-09 16:44:50] ...optimizers/trijoint.py.151: total_norm: 30.426765 [I 2022-10-09 16:44:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1430/2304 [I 2022-10-09 16:44:58] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:16 [I 2022-10-09 16:44:58] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00310 [I 2022-10-09 16:44:58] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-09 16:44:58] ...optimizers/trijoint.py.151: total_norm: 18.844730 [I 2022-10-09 16:45:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 1440/2304 [I 2022-10-09 16:45:07] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:17 [I 2022-10-09 16:45:07] ...trap/engines/engine.py.229: process: 0.86948 | load: 0.00313 [I 2022-10-09 16:45:07] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-09 16:45:07] ...optimizers/trijoint.py.151: total_norm: 11.399694 [I 2022-10-09 16:45:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 1450/2304 [I 2022-10-09 16:45:15] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:12 [I 2022-10-09 16:45:15] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00313 [I 2022-10-09 16:45:15] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-09 16:45:15] ...optimizers/trijoint.py.151: total_norm: 20.250338 [I 2022-10-09 16:45:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1460/2304 [I 2022-10-09 16:45:24] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:01 [I 2022-10-09 16:45:24] ...trap/engines/engine.py.229: process: 0.85660 | load: 0.00313 [I 2022-10-09 16:45:24] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-09 16:45:24] ...optimizers/trijoint.py.151: total_norm: 11.115870 [I 2022-10-09 16:45:32] ...trap/engines/engine.py.224: train: epoch 47 | batch 1470/2304 [I 2022-10-09 16:45:32] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:41 [I 2022-10-09 16:45:32] ...trap/engines/engine.py.229: process: 0.83401 | load: 0.00293 [I 2022-10-09 16:45:32] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-09 16:45:32] ...optimizers/trijoint.py.151: total_norm: 11.078979 [I 2022-10-09 16:45:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1480/2304 [I 2022-10-09 16:45:41] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:35 [I 2022-10-09 16:45:41] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00293 [I 2022-10-09 16:45:41] ...trap/engines/engine.py.230: loss: 0.20213 [I 2022-10-09 16:45:41] ...optimizers/trijoint.py.151: total_norm: 23.357349 [I 2022-10-09 16:45:49] ...trap/engines/engine.py.224: train: epoch 47 | batch 1490/2304 [I 2022-10-09 16:45:49] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:27 [I 2022-10-09 16:45:49] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00299 [I 2022-10-09 16:45:49] ...trap/engines/engine.py.230: loss: 0.11385 [I 2022-10-09 16:45:49] ...optimizers/trijoint.py.151: total_norm: 11.715019 [I 2022-10-09 16:45:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1500/2304 [I 2022-10-09 16:45:58] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:18 [I 2022-10-09 16:45:58] ...trap/engines/engine.py.229: process: 0.84374 | load: 0.00299 [I 2022-10-09 16:45:58] ...trap/engines/engine.py.230: loss: 0.20035 [I 2022-10-09 16:45:58] ...optimizers/trijoint.py.151: total_norm: 23.127678 [I 2022-10-09 16:46:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 1510/2304 [I 2022-10-09 16:46:06] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:08 [I 2022-10-09 16:46:06] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00291 [I 2022-10-09 16:46:06] ...trap/engines/engine.py.230: loss: 0.13904 [I 2022-10-09 16:46:06] ...optimizers/trijoint.py.151: total_norm: 12.380448 [I 2022-10-09 16:46:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 1520/2304 [I 2022-10-09 16:46:15] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:05 [I 2022-10-09 16:46:15] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00294 [I 2022-10-09 16:46:15] ...trap/engines/engine.py.230: loss: 0.18713 [I 2022-10-09 16:46:15] ...optimizers/trijoint.py.151: total_norm: 18.197710 [I 2022-10-09 16:46:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 1530/2304 [I 2022-10-09 16:46:23] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:50 [I 2022-10-09 16:46:23] ...trap/engines/engine.py.229: process: 0.83543 | load: 0.00293 [I 2022-10-09 16:46:23] ...trap/engines/engine.py.230: loss: 0.18743 [I 2022-10-09 16:46:23] ...optimizers/trijoint.py.151: total_norm: 35.197426 [I 2022-10-09 16:46:31] ...trap/engines/engine.py.224: train: epoch 47 | batch 1540/2304 [I 2022-10-09 16:46:31] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:42 [I 2022-10-09 16:46:31] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00292 [I 2022-10-09 16:46:31] ...trap/engines/engine.py.230: loss: 0.22125 [I 2022-10-09 16:46:31] ...optimizers/trijoint.py.151: total_norm: 20.384892 [I 2022-10-09 16:46:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 1550/2304 [I 2022-10-09 16:46:40] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:36 [I 2022-10-09 16:46:40] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00295 [I 2022-10-09 16:46:40] ...trap/engines/engine.py.230: loss: 0.22396 [I 2022-10-09 16:46:40] ...optimizers/trijoint.py.151: total_norm: 39.907383 [I 2022-10-09 16:46:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 1560/2304 [I 2022-10-09 16:46:48] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:22 [I 2022-10-09 16:46:48] ...trap/engines/engine.py.229: process: 0.81717 | load: 0.00291 [I 2022-10-09 16:46:48] ...trap/engines/engine.py.230: loss: 0.11934 [I 2022-10-09 16:46:48] ...optimizers/trijoint.py.151: total_norm: 7.275253 [I 2022-10-09 16:46:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 1570/2304 [I 2022-10-09 16:46:57] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:16 [I 2022-10-09 16:46:57] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00301 [I 2022-10-09 16:46:57] ...trap/engines/engine.py.230: loss: 0.15912 [I 2022-10-09 16:46:57] ...optimizers/trijoint.py.151: total_norm: 23.011061 [I 2022-10-09 16:47:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 1580/2304 [I 2022-10-09 16:47:05] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:11 [I 2022-10-09 16:47:05] ...trap/engines/engine.py.229: process: 0.84711 | load: 0.00298 [I 2022-10-09 16:47:05] ...trap/engines/engine.py.230: loss: 0.25167 [I 2022-10-09 16:47:05] ...optimizers/trijoint.py.151: total_norm: 19.706053 [I 2022-10-09 16:47:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 1590/2304 [I 2022-10-09 16:47:14] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:09:59 [I 2022-10-09 16:47:14] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00298 [I 2022-10-09 16:47:14] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-09 16:47:14] ...optimizers/trijoint.py.151: total_norm: 14.932203 [I 2022-10-09 16:47:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 1600/2304 [I 2022-10-09 16:47:22] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:09:52 [I 2022-10-09 16:47:22] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00296 [I 2022-10-09 16:47:22] ...trap/engines/engine.py.230: loss: 0.19719 [I 2022-10-09 16:47:22] ...optimizers/trijoint.py.151: total_norm: 11.065407 [I 2022-10-09 16:47:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 1610/2304 [I 2022-10-09 16:47:30] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:09:47 [I 2022-10-09 16:47:30] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00302 [I 2022-10-09 16:47:30] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-09 16:47:30] ...optimizers/trijoint.py.151: total_norm: 16.439432 [I 2022-10-09 16:47:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 1620/2304 [I 2022-10-09 16:47:39] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:09:36 [I 2022-10-09 16:47:39] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00298 [I 2022-10-09 16:47:39] ...trap/engines/engine.py.230: loss: 0.17994 [I 2022-10-09 16:47:39] ...optimizers/trijoint.py.151: total_norm: 14.709977 [I 2022-10-09 16:47:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 1630/2304 [I 2022-10-09 16:47:47] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:28 [I 2022-10-09 16:47:47] ...trap/engines/engine.py.229: process: 0.84641 | load: 0.00297 [I 2022-10-09 16:47:47] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-09 16:47:47] ...optimizers/trijoint.py.151: total_norm: 14.430216 [I 2022-10-09 16:47:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 1640/2304 [I 2022-10-09 16:47:56] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:19 [I 2022-10-09 16:47:56] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00298 [I 2022-10-09 16:47:56] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-09 16:47:56] ...optimizers/trijoint.py.151: total_norm: 12.611938 [I 2022-10-09 16:48:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 1650/2304 [I 2022-10-09 16:48:04] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:12 [I 2022-10-09 16:48:04] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00303 [I 2022-10-09 16:48:04] ...trap/engines/engine.py.230: loss: 0.13617 [I 2022-10-09 16:48:04] ...optimizers/trijoint.py.151: total_norm: 6.335104 [I 2022-10-09 16:48:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 1660/2304 [I 2022-10-09 16:48:13] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:03 [I 2022-10-09 16:48:13] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00296 [I 2022-10-09 16:48:13] ...trap/engines/engine.py.230: loss: 0.25927 [I 2022-10-09 16:48:13] ...optimizers/trijoint.py.151: total_norm: 21.684580 [I 2022-10-09 16:48:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 1670/2304 [I 2022-10-09 16:48:21] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:08:59 [I 2022-10-09 16:48:21] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00297 [I 2022-10-09 16:48:21] ...trap/engines/engine.py.230: loss: 0.19683 [I 2022-10-09 16:48:21] ...optimizers/trijoint.py.151: total_norm: 18.221426 [I 2022-10-09 16:48:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 1680/2304 [I 2022-10-09 16:48:30] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:08:47 [I 2022-10-09 16:48:30] ...trap/engines/engine.py.229: process: 0.85726 | load: 0.00297 [I 2022-10-09 16:48:30] ...trap/engines/engine.py.230: loss: 0.15789 [I 2022-10-09 16:48:30] ...optimizers/trijoint.py.151: total_norm: 11.037099 [I 2022-10-09 16:48:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1690/2304 [I 2022-10-09 16:48:38] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:41 [I 2022-10-09 16:48:38] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00296 [I 2022-10-09 16:48:38] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-09 16:48:38] ...optimizers/trijoint.py.151: total_norm: 50.633156 [I 2022-10-09 16:48:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 1700/2304 [I 2022-10-09 16:48:47] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:33 [I 2022-10-09 16:48:47] ...trap/engines/engine.py.229: process: 0.85420 | load: 0.00312 [I 2022-10-09 16:48:47] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-09 16:48:47] ...optimizers/trijoint.py.151: total_norm: 22.398781 [I 2022-10-09 16:48:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1710/2304 [I 2022-10-09 16:48:55] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:26 [I 2022-10-09 16:48:55] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00323 [I 2022-10-09 16:48:55] ...trap/engines/engine.py.230: loss: 0.21144 [I 2022-10-09 16:48:55] ...optimizers/trijoint.py.151: total_norm: 14.307961 [I 2022-10-09 16:49:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 1720/2304 [I 2022-10-09 16:49:04] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:17 [I 2022-10-09 16:49:04] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00312 [I 2022-10-09 16:49:04] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-09 16:49:04] ...optimizers/trijoint.py.151: total_norm: 21.306404 [I 2022-10-09 16:49:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1730/2304 [I 2022-10-09 16:49:12] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:08 [I 2022-10-09 16:49:12] ...trap/engines/engine.py.229: process: 0.84483 | load: 0.00316 [I 2022-10-09 16:49:12] ...trap/engines/engine.py.230: loss: 0.12709 [I 2022-10-09 16:49:12] ...optimizers/trijoint.py.151: total_norm: 15.017784 [I 2022-10-09 16:49:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 1740/2304 [I 2022-10-09 16:49:21] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:01 [I 2022-10-09 16:49:21] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00312 [I 2022-10-09 16:49:21] ...trap/engines/engine.py.230: loss: 0.21062 [I 2022-10-09 16:49:21] ...optimizers/trijoint.py.151: total_norm: 11.828325 [I 2022-10-09 16:49:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 1750/2304 [I 2022-10-09 16:49:29] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:07:52 [I 2022-10-09 16:49:29] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00313 [I 2022-10-09 16:49:29] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-09 16:49:29] ...optimizers/trijoint.py.151: total_norm: 24.667173 [I 2022-10-09 16:49:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1760/2304 [I 2022-10-09 16:49:38] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:43 [I 2022-10-09 16:49:38] ...trap/engines/engine.py.229: process: 0.85666 | load: 0.00315 [I 2022-10-09 16:49:38] ...trap/engines/engine.py.230: loss: 0.21518 [I 2022-10-09 16:49:38] ...optimizers/trijoint.py.151: total_norm: 16.373657 [I 2022-10-09 16:49:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 1770/2304 [I 2022-10-09 16:49:47] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:34 [I 2022-10-09 16:49:47] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00311 [I 2022-10-09 16:49:47] ...trap/engines/engine.py.230: loss: 0.10881 [I 2022-10-09 16:49:47] ...optimizers/trijoint.py.151: total_norm: 11.329556 [I 2022-10-09 16:49:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1780/2304 [I 2022-10-09 16:49:55] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:29 [I 2022-10-09 16:49:55] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00312 [I 2022-10-09 16:49:55] ...trap/engines/engine.py.230: loss: 0.17297 [I 2022-10-09 16:49:55] ...optimizers/trijoint.py.151: total_norm: 13.273676 [I 2022-10-09 16:50:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 1790/2304 [I 2022-10-09 16:50:04] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:19 [I 2022-10-09 16:50:04] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00315 [I 2022-10-09 16:50:04] ...trap/engines/engine.py.230: loss: 0.15560 [I 2022-10-09 16:50:04] ...optimizers/trijoint.py.151: total_norm: 14.539249 [I 2022-10-09 16:50:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1800/2304 [I 2022-10-09 16:50:12] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:08 [I 2022-10-09 16:50:12] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00317 [I 2022-10-09 16:50:12] ...trap/engines/engine.py.230: loss: 0.12016 [I 2022-10-09 16:50:12] ...optimizers/trijoint.py.151: total_norm: 13.526095 [I 2022-10-09 16:50:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 1810/2304 [I 2022-10-09 16:50:21] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:01 [I 2022-10-09 16:50:21] ...trap/engines/engine.py.229: process: 0.83836 | load: 0.00313 [I 2022-10-09 16:50:21] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-09 16:50:21] ...optimizers/trijoint.py.151: total_norm: 9.598665 [I 2022-10-09 16:50:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 1820/2304 [I 2022-10-09 16:50:29] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:53 [I 2022-10-09 16:50:29] ...trap/engines/engine.py.229: process: 0.85978 | load: 0.00314 [I 2022-10-09 16:50:29] ...trap/engines/engine.py.230: loss: 0.13026 [I 2022-10-09 16:50:29] ...optimizers/trijoint.py.151: total_norm: 16.357182 [I 2022-10-09 16:50:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1830/2304 [I 2022-10-09 16:50:38] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:43 [I 2022-10-09 16:50:38] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00337 [I 2022-10-09 16:50:38] ...trap/engines/engine.py.230: loss: 0.12238 [I 2022-10-09 16:50:38] ...optimizers/trijoint.py.151: total_norm: 11.642455 [I 2022-10-09 16:50:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 1840/2304 [I 2022-10-09 16:50:46] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:34 [I 2022-10-09 16:50:46] ...trap/engines/engine.py.229: process: 0.85589 | load: 0.00309 [I 2022-10-09 16:50:46] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-09 16:50:46] ...optimizers/trijoint.py.151: total_norm: 10.107038 [I 2022-10-09 16:50:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1850/2304 [I 2022-10-09 16:50:55] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:27 [I 2022-10-09 16:50:55] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00338 [I 2022-10-09 16:50:55] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-09 16:50:55] ...optimizers/trijoint.py.151: total_norm: 11.060687 [I 2022-10-09 16:51:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 1860/2304 [I 2022-10-09 16:51:03] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:19 [I 2022-10-09 16:51:03] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00313 [I 2022-10-09 16:51:03] ...trap/engines/engine.py.230: loss: 0.15918 [I 2022-10-09 16:51:03] ...optimizers/trijoint.py.151: total_norm: 17.531355 [I 2022-10-09 16:51:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1870/2304 [I 2022-10-09 16:51:12] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:07 [I 2022-10-09 16:51:12] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00312 [I 2022-10-09 16:51:12] ...trap/engines/engine.py.230: loss: 0.15650 [I 2022-10-09 16:51:12] ...optimizers/trijoint.py.151: total_norm: 11.623190 [I 2022-10-09 16:51:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 1880/2304 [I 2022-10-09 16:51:20] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:00 [I 2022-10-09 16:51:20] ...trap/engines/engine.py.229: process: 0.83203 | load: 0.00318 [I 2022-10-09 16:51:20] ...trap/engines/engine.py.230: loss: 0.12666 [I 2022-10-09 16:51:20] ...optimizers/trijoint.py.151: total_norm: 10.818336 [I 2022-10-09 16:51:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 1890/2304 [I 2022-10-09 16:51:29] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:54 [I 2022-10-09 16:51:29] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00319 [I 2022-10-09 16:51:29] ...trap/engines/engine.py.230: loss: 0.16687 [I 2022-10-09 16:51:29] ...optimizers/trijoint.py.151: total_norm: 35.591770 [I 2022-10-09 16:51:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1900/2304 [I 2022-10-09 16:51:38] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:43 [I 2022-10-09 16:51:38] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00306 [I 2022-10-09 16:51:38] ...trap/engines/engine.py.230: loss: 0.13519 [I 2022-10-09 16:51:38] ...optimizers/trijoint.py.151: total_norm: 8.352277 [I 2022-10-09 16:51:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 1910/2304 [I 2022-10-09 16:51:46] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:35 [I 2022-10-09 16:51:46] ...trap/engines/engine.py.229: process: 0.85482 | load: 0.00309 [I 2022-10-09 16:51:46] ...trap/engines/engine.py.230: loss: 0.19159 [I 2022-10-09 16:51:46] ...optimizers/trijoint.py.151: total_norm: 15.991779 [I 2022-10-09 16:51:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1920/2304 [I 2022-10-09 16:51:55] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:27 [I 2022-10-09 16:51:55] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00312 [I 2022-10-09 16:51:55] ...trap/engines/engine.py.230: loss: 0.16418 [I 2022-10-09 16:51:55] ...optimizers/trijoint.py.151: total_norm: 14.937901 [I 2022-10-09 16:52:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 1930/2304 [I 2022-10-09 16:52:03] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:18 [I 2022-10-09 16:52:03] ...trap/engines/engine.py.229: process: 0.85846 | load: 0.00310 [I 2022-10-09 16:52:03] ...trap/engines/engine.py.230: loss: 0.15546 [I 2022-10-09 16:52:03] ...optimizers/trijoint.py.151: total_norm: 25.850756 [I 2022-10-09 16:52:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1940/2304 [I 2022-10-09 16:52:12] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:10 [I 2022-10-09 16:52:12] ...trap/engines/engine.py.229: process: 0.85059 | load: 0.00312 [I 2022-10-09 16:52:12] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-09 16:52:12] ...optimizers/trijoint.py.151: total_norm: 15.168300 [I 2022-10-09 16:52:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 1950/2304 [I 2022-10-09 16:52:20] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:01 [I 2022-10-09 16:52:20] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00316 [I 2022-10-09 16:52:20] ...trap/engines/engine.py.230: loss: 0.20467 [I 2022-10-09 16:52:20] ...optimizers/trijoint.py.151: total_norm: 12.987300 [I 2022-10-09 16:52:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 1960/2304 [I 2022-10-09 16:52:29] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:52 [I 2022-10-09 16:52:29] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00319 [I 2022-10-09 16:52:29] ...trap/engines/engine.py.230: loss: 0.15271 [I 2022-10-09 16:52:29] ...optimizers/trijoint.py.151: total_norm: 22.047649 [I 2022-10-09 16:52:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 1970/2304 [I 2022-10-09 16:52:37] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:44 [I 2022-10-09 16:52:37] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00313 [I 2022-10-09 16:52:37] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-09 16:52:37] ...optimizers/trijoint.py.151: total_norm: 20.548506 [I 2022-10-09 16:52:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 1980/2304 [I 2022-10-09 16:52:46] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:35 [I 2022-10-09 16:52:46] ...trap/engines/engine.py.229: process: 0.84051 | load: 0.00315 [I 2022-10-09 16:52:46] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-09 16:52:46] ...optimizers/trijoint.py.151: total_norm: 11.716127 [I 2022-10-09 16:52:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 1990/2304 [I 2022-10-09 16:52:54] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:27 [I 2022-10-09 16:52:54] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00312 [I 2022-10-09 16:52:54] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-09 16:52:54] ...optimizers/trijoint.py.151: total_norm: 10.339986 [I 2022-10-09 16:53:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 2000/2304 [I 2022-10-09 16:53:03] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:21 [I 2022-10-09 16:53:03] ...trap/engines/engine.py.229: process: 0.87520 | load: 0.00315 [I 2022-10-09 16:53:03] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-09 16:53:03] ...optimizers/trijoint.py.151: total_norm: 12.122929 [I 2022-10-09 16:53:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 2010/2304 [I 2022-10-09 16:53:11] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:11 [I 2022-10-09 16:53:11] ...trap/engines/engine.py.229: process: 0.86831 | load: 0.00311 [I 2022-10-09 16:53:11] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-09 16:53:11] ...optimizers/trijoint.py.151: total_norm: 11.380106 [I 2022-10-09 16:53:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 2020/2304 [I 2022-10-09 16:53:20] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:01 [I 2022-10-09 16:53:20] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00303 [I 2022-10-09 16:53:20] ...trap/engines/engine.py.230: loss: 0.19107 [I 2022-10-09 16:53:20] ...optimizers/trijoint.py.151: total_norm: 17.830280 [I 2022-10-09 16:53:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 2030/2304 [I 2022-10-09 16:53:28] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:52 [I 2022-10-09 16:53:28] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00302 [I 2022-10-09 16:53:28] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-09 16:53:28] ...optimizers/trijoint.py.151: total_norm: 17.649412 [I 2022-10-09 16:53:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 2040/2304 [I 2022-10-09 16:53:37] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:43 [I 2022-10-09 16:53:37] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00299 [I 2022-10-09 16:53:37] ...trap/engines/engine.py.230: loss: 0.16302 [I 2022-10-09 16:53:37] ...optimizers/trijoint.py.151: total_norm: 16.051617 [I 2022-10-09 16:53:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 2050/2304 [I 2022-10-09 16:53:45] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:33 [I 2022-10-09 16:53:45] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00296 [I 2022-10-09 16:53:45] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-09 16:53:45] ...optimizers/trijoint.py.151: total_norm: 34.340931 [I 2022-10-09 16:53:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 2060/2304 [I 2022-10-09 16:53:54] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:25 [I 2022-10-09 16:53:54] ...trap/engines/engine.py.229: process: 0.83449 | load: 0.00298 [I 2022-10-09 16:53:54] ...trap/engines/engine.py.230: loss: 0.14170 [I 2022-10-09 16:53:54] ...optimizers/trijoint.py.151: total_norm: 10.533185 [I 2022-10-09 16:54:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 2070/2304 [I 2022-10-09 16:54:02] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:16 [I 2022-10-09 16:54:02] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00292 [I 2022-10-09 16:54:02] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-09 16:54:02] ...optimizers/trijoint.py.151: total_norm: 14.689115 [I 2022-10-09 16:54:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 2080/2304 [I 2022-10-09 16:54:11] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:09 [I 2022-10-09 16:54:11] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00292 [I 2022-10-09 16:54:11] ...trap/engines/engine.py.230: loss: 0.15493 [I 2022-10-09 16:54:11] ...optimizers/trijoint.py.151: total_norm: 12.761887 [I 2022-10-09 16:54:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 2090/2304 [I 2022-10-09 16:54:19] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:00 [I 2022-10-09 16:54:19] ...trap/engines/engine.py.229: process: 0.84244 | load: 0.00294 [I 2022-10-09 16:54:19] ...trap/engines/engine.py.230: loss: 0.13872 [I 2022-10-09 16:54:19] ...optimizers/trijoint.py.151: total_norm: 14.095918 [I 2022-10-09 16:54:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 2100/2304 [I 2022-10-09 16:54:28] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:52 [I 2022-10-09 16:54:28] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00293 [I 2022-10-09 16:54:28] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-09 16:54:28] ...optimizers/trijoint.py.151: total_norm: 9.557901 [I 2022-10-09 16:54:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 2110/2304 [I 2022-10-09 16:54:36] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:43 [I 2022-10-09 16:54:36] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00295 [I 2022-10-09 16:54:36] ...trap/engines/engine.py.230: loss: 0.15426 [I 2022-10-09 16:54:36] ...optimizers/trijoint.py.151: total_norm: 26.146366 [I 2022-10-09 16:54:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 2120/2304 [I 2022-10-09 16:54:45] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:35 [I 2022-10-09 16:54:45] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00297 [I 2022-10-09 16:54:45] ...trap/engines/engine.py.230: loss: 0.14821 [I 2022-10-09 16:54:45] ...optimizers/trijoint.py.151: total_norm: 13.328587 [I 2022-10-09 16:54:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 2130/2304 [I 2022-10-09 16:54:53] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:26 [I 2022-10-09 16:54:53] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00296 [I 2022-10-09 16:54:53] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-09 16:54:53] ...optimizers/trijoint.py.151: total_norm: 11.314567 [I 2022-10-09 16:55:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 2140/2304 [I 2022-10-09 16:55:02] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:18 [I 2022-10-09 16:55:02] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00298 [I 2022-10-09 16:55:02] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-09 16:55:02] ...optimizers/trijoint.py.151: total_norm: 30.930096 [I 2022-10-09 16:55:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 2150/2304 [I 2022-10-09 16:55:28] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:03:23 [I 2022-10-09 16:55:28] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00300 [I 2022-10-09 16:55:28] ...trap/engines/engine.py.230: loss: 0.09726 [I 2022-10-09 16:55:28] ...optimizers/trijoint.py.151: total_norm: 8.124889 [I 2022-10-09 16:55:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 2160/2304 [I 2022-10-09 16:55:36] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:09 [I 2022-10-09 16:55:36] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00296 [I 2022-10-09 16:55:36] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-09 16:55:36] ...optimizers/trijoint.py.151: total_norm: 19.543997 [I 2022-10-09 16:55:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 2170/2304 [I 2022-10-09 16:55:45] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:53 [I 2022-10-09 16:55:45] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00298 [I 2022-10-09 16:55:45] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-09 16:55:45] ...optimizers/trijoint.py.151: total_norm: 15.843350 [I 2022-10-09 16:55:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 2180/2304 [I 2022-10-09 16:55:53] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:45 [I 2022-10-09 16:55:53] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00300 [I 2022-10-09 16:55:53] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-09 16:55:53] ...optimizers/trijoint.py.151: total_norm: 16.004242 [I 2022-10-09 16:56:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 2190/2304 [I 2022-10-09 16:56:02] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:35 [I 2022-10-09 16:56:02] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00313 [I 2022-10-09 16:56:02] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-09 16:56:02] ...optimizers/trijoint.py.151: total_norm: 20.681147 [I 2022-10-09 16:56:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 2200/2304 [I 2022-10-09 16:56:10] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:27 [I 2022-10-09 16:56:10] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00300 [I 2022-10-09 16:56:10] ...trap/engines/engine.py.230: loss: 0.23092 [I 2022-10-09 16:56:10] ...optimizers/trijoint.py.151: total_norm: 30.774586 [I 2022-10-09 16:56:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 2210/2304 [I 2022-10-09 16:56:19] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:20 [I 2022-10-09 16:56:19] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00312 [I 2022-10-09 16:56:19] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-09 16:56:19] ...optimizers/trijoint.py.151: total_norm: 14.779524 [I 2022-10-09 16:56:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 2220/2304 [I 2022-10-09 16:56:27] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:11 [I 2022-10-09 16:56:27] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00314 [I 2022-10-09 16:56:27] ...trap/engines/engine.py.230: loss: 0.12685 [I 2022-10-09 16:56:27] ...optimizers/trijoint.py.151: total_norm: 21.477360 [I 2022-10-09 16:56:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 2230/2304 [I 2022-10-09 16:56:36] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:03 [I 2022-10-09 16:56:36] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00318 [I 2022-10-09 16:56:36] ...trap/engines/engine.py.230: loss: 0.17029 [I 2022-10-09 16:56:36] ...optimizers/trijoint.py.151: total_norm: 16.828445 [I 2022-10-09 16:56:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 2240/2304 [I 2022-10-09 16:56:44] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:54 [I 2022-10-09 16:56:44] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00319 [I 2022-10-09 16:56:44] ...trap/engines/engine.py.230: loss: 0.19406 [I 2022-10-09 16:56:44] ...optimizers/trijoint.py.151: total_norm: 22.735184 [I 2022-10-09 16:56:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 2250/2304 [I 2022-10-09 16:56:53] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:46 [I 2022-10-09 16:56:53] ...trap/engines/engine.py.229: process: 0.83504 | load: 0.00312 [I 2022-10-09 16:56:53] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-09 16:56:53] ...optimizers/trijoint.py.151: total_norm: 7.618401 [I 2022-10-09 16:57:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 2260/2304 [I 2022-10-09 16:57:01] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:37 [I 2022-10-09 16:57:01] ...trap/engines/engine.py.229: process: 0.85960 | load: 0.00308 [I 2022-10-09 16:57:01] ...trap/engines/engine.py.230: loss: 0.16392 [I 2022-10-09 16:57:01] ...optimizers/trijoint.py.151: total_norm: 14.642234 [I 2022-10-09 16:57:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 2270/2304 [I 2022-10-09 16:57:10] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:28 [I 2022-10-09 16:57:10] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00313 [I 2022-10-09 16:57:10] ...trap/engines/engine.py.230: loss: 0.20054 [I 2022-10-09 16:57:10] ...optimizers/trijoint.py.151: total_norm: 13.218697 [I 2022-10-09 16:57:18] ...trap/engines/engine.py.224: train: epoch 47 | batch 2280/2304 [I 2022-10-09 16:57:18] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:20 [I 2022-10-09 16:57:18] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00313 [I 2022-10-09 16:57:18] ...trap/engines/engine.py.230: loss: 0.10830 [I 2022-10-09 16:57:18] ...optimizers/trijoint.py.151: total_norm: 9.095587 [I 2022-10-09 16:57:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 2290/2304 [I 2022-10-09 16:57:27] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:11 [I 2022-10-09 16:57:27] ...trap/engines/engine.py.229: process: 0.86476 | load: 0.00310 [I 2022-10-09 16:57:27] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-09 16:57:27] ...optimizers/trijoint.py.151: total_norm: 12.759691 [I 2022-10-09 16:57:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 2300/2304 [I 2022-10-09 16:57:35] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:03 [I 2022-10-09 16:57:35] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00019 [I 2022-10-09 16:57:35] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-09 16:57:35] ...optimizers/trijoint.py.151: total_norm: 15.382203 [I 2022-10-09 16:57:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 2304/2304 [I 2022-10-09 16:57:39] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:00 [I 2022-10-09 16:57:39] ...trap/engines/engine.py.229: process: 0.83386 | load: 0.00018 [I 2022-10-09 16:57:39] ...trap/engines/engine.py.230: loss: 0.21099 [I 2022-10-09 16:57:39] ...optimizers/trijoint.py.151: total_norm: 18.439539 [S 2022-10-09 16:57:40] ...trap/engines/engine.py.129: train_epoch.epoch: 47 [S 2022-10-09 16:57:40] ...trap/engines/engine.py.129: train_epoch.loss: 0.1566539296197529 [I 2022-10-09 16:57:40] ...s/criterions/triplet.py.84: margin = 0.2850000000000001 [I 2022-10-09 16:57:41] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 47 [I 2022-10-09 16:57:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 16:57:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 16:57:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 16:57:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 16:57:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 16:57:45] ...trap/engines/engine.py.321: eval: epoch 47 | batch 0/510 [I 2022-10-09 16:57:45] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:47 [I 2022-10-09 16:57:45] ...trap/engines/engine.py.326: process: 4.32816 | load: 3.66910 [I 2022-10-09 16:57:50] ...trap/engines/engine.py.321: eval: epoch 47 | batch 10/510 [I 2022-10-09 16:57:50] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:17 [I 2022-10-09 16:57:50] ...trap/engines/engine.py.326: process: 0.47926 | load: 0.00019 [I 2022-10-09 16:57:55] ...trap/engines/engine.py.321: eval: epoch 47 | batch 20/510 [I 2022-10-09 16:57:55] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:24 [I 2022-10-09 16:57:55] ...trap/engines/engine.py.326: process: 0.48933 | load: 0.00022 [I 2022-10-09 16:58:00] ...trap/engines/engine.py.321: eval: epoch 47 | batch 30/510 [I 2022-10-09 16:58:00] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:12 [I 2022-10-09 16:58:00] ...trap/engines/engine.py.326: process: 0.43134 | load: 0.00019 [I 2022-10-09 16:58:05] ...trap/engines/engine.py.321: eval: epoch 47 | batch 40/510 [I 2022-10-09 16:58:05] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:03:57 [I 2022-10-09 16:58:05] ...trap/engines/engine.py.326: process: 0.42803 | load: 0.00023 [I 2022-10-09 16:58:10] ...trap/engines/engine.py.321: eval: epoch 47 | batch 50/510 [I 2022-10-09 16:58:10] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:31 [I 2022-10-09 16:58:10] ...trap/engines/engine.py.326: process: 0.59393 | load: 0.00021 [I 2022-10-09 16:58:16] ...trap/engines/engine.py.321: eval: epoch 47 | batch 60/510 [I 2022-10-09 16:58:16] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:37 [I 2022-10-09 16:58:16] ...trap/engines/engine.py.326: process: 0.63919 | load: 0.00020 [I 2022-10-09 16:58:22] ...trap/engines/engine.py.321: eval: epoch 47 | batch 70/510 [I 2022-10-09 16:58:22] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:01 [I 2022-10-09 16:58:22] ...trap/engines/engine.py.326: process: 0.48088 | load: 0.00020 [I 2022-10-09 16:58:27] ...trap/engines/engine.py.321: eval: epoch 47 | batch 80/510 [I 2022-10-09 16:58:27] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:25 [I 2022-10-09 16:58:27] ...trap/engines/engine.py.326: process: 0.48606 | load: 0.00020 [I 2022-10-09 16:58:32] ...trap/engines/engine.py.321: eval: epoch 47 | batch 90/510 [I 2022-10-09 16:58:32] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:11 [I 2022-10-09 16:58:32] ...trap/engines/engine.py.326: process: 0.45630 | load: 0.00019 [I 2022-10-09 16:58:36] ...trap/engines/engine.py.321: eval: epoch 47 | batch 100/510 [I 2022-10-09 16:58:36] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:55 [I 2022-10-09 16:58:36] ...trap/engines/engine.py.326: process: 0.31030 | load: 0.00021 [I 2022-10-09 16:58:39] ...trap/engines/engine.py.321: eval: epoch 47 | batch 110/510 [I 2022-10-09 16:58:39] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-09 16:58:39] ...trap/engines/engine.py.326: process: 0.31670 | load: 0.00020 [I 2022-10-09 16:58:42] ...trap/engines/engine.py.321: eval: epoch 47 | batch 120/510 [I 2022-10-09 16:58:42] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:05 [I 2022-10-09 16:58:42] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00020 [I 2022-10-09 16:58:46] ...trap/engines/engine.py.321: eval: epoch 47 | batch 130/510 [I 2022-10-09 16:58:46] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-09 16:58:46] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00022 [I 2022-10-09 16:58:49] ...trap/engines/engine.py.321: eval: epoch 47 | batch 140/510 [I 2022-10-09 16:58:49] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-09 16:58:49] ...trap/engines/engine.py.326: process: 0.32131 | load: 0.00020 [I 2022-10-09 16:58:52] ...trap/engines/engine.py.321: eval: epoch 47 | batch 150/510 [I 2022-10-09 16:58:52] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:55 [I 2022-10-09 16:58:52] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00020 [I 2022-10-09 16:58:55] ...trap/engines/engine.py.321: eval: epoch 47 | batch 160/510 [I 2022-10-09 16:58:55] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-09 16:58:55] ...trap/engines/engine.py.326: process: 0.32140 | load: 0.00020 [I 2022-10-09 16:58:58] ...trap/engines/engine.py.321: eval: epoch 47 | batch 170/510 [I 2022-10-09 16:58:58] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:49 [I 2022-10-09 16:58:58] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00022 [I 2022-10-09 16:59:02] ...trap/engines/engine.py.321: eval: epoch 47 | batch 180/510 [I 2022-10-09 16:59:02] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:44 [I 2022-10-09 16:59:02] ...trap/engines/engine.py.326: process: 0.31530 | load: 0.00020 [I 2022-10-09 16:59:05] ...trap/engines/engine.py.321: eval: epoch 47 | batch 190/510 [I 2022-10-09 16:59:05] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:42 [I 2022-10-09 16:59:05] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00020 [I 2022-10-09 16:59:08] ...trap/engines/engine.py.321: eval: epoch 47 | batch 200/510 [I 2022-10-09 16:59:08] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-09 16:59:08] ...trap/engines/engine.py.326: process: 0.31624 | load: 0.00019 [I 2022-10-09 16:59:11] ...trap/engines/engine.py.321: eval: epoch 47 | batch 210/510 [I 2022-10-09 16:59:11] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-09 16:59:11] ...trap/engines/engine.py.326: process: 0.31739 | load: 0.00020 [I 2022-10-09 16:59:14] ...trap/engines/engine.py.321: eval: epoch 47 | batch 220/510 [I 2022-10-09 16:59:14] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-09 16:59:14] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00020 [I 2022-10-09 16:59:18] ...trap/engines/engine.py.321: eval: epoch 47 | batch 230/510 [I 2022-10-09 16:59:18] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:30 [I 2022-10-09 16:59:18] ...trap/engines/engine.py.326: process: 0.32264 | load: 0.00020 [I 2022-10-09 16:59:21] ...trap/engines/engine.py.321: eval: epoch 47 | batch 240/510 [I 2022-10-09 16:59:21] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-09 16:59:21] ...trap/engines/engine.py.326: process: 0.31693 | load: 0.00021 [I 2022-10-09 16:59:24] ...trap/engines/engine.py.321: eval: epoch 47 | batch 250/510 [I 2022-10-09 16:59:24] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-09 16:59:24] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00021 [I 2022-10-09 16:59:27] ...trap/engines/engine.py.321: eval: epoch 47 | batch 260/510 [I 2022-10-09 16:59:27] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-09 16:59:27] ...trap/engines/engine.py.326: process: 0.32181 | load: 0.00020 [I 2022-10-09 16:59:30] ...trap/engines/engine.py.321: eval: epoch 47 | batch 270/510 [I 2022-10-09 16:59:30] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-09 16:59:30] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00021 [I 2022-10-09 16:59:34] ...trap/engines/engine.py.321: eval: epoch 47 | batch 280/510 [I 2022-10-09 16:59:34] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 16:59:34] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00020 [I 2022-10-09 16:59:37] ...trap/engines/engine.py.321: eval: epoch 47 | batch 290/510 [I 2022-10-09 16:59:37] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:10 [I 2022-10-09 16:59:37] ...trap/engines/engine.py.326: process: 0.32220 | load: 0.00020 [I 2022-10-09 16:59:40] ...trap/engines/engine.py.321: eval: epoch 47 | batch 300/510 [I 2022-10-09 16:59:40] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:07 [I 2022-10-09 16:59:40] ...trap/engines/engine.py.326: process: 0.32351 | load: 0.00020 [I 2022-10-09 16:59:43] ...trap/engines/engine.py.321: eval: epoch 47 | batch 310/510 [I 2022-10-09 16:59:43] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-09 16:59:43] ...trap/engines/engine.py.326: process: 0.32476 | load: 0.00020 [I 2022-10-09 16:59:46] ...trap/engines/engine.py.321: eval: epoch 47 | batch 320/510 [I 2022-10-09 16:59:46] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-09 16:59:46] ...trap/engines/engine.py.326: process: 0.31538 | load: 0.00020 [I 2022-10-09 16:59:50] ...trap/engines/engine.py.321: eval: epoch 47 | batch 330/510 [I 2022-10-09 16:59:50] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:58 [I 2022-10-09 16:59:50] ...trap/engines/engine.py.326: process: 0.32484 | load: 0.00022 [I 2022-10-09 16:59:53] ...trap/engines/engine.py.321: eval: epoch 47 | batch 340/510 [I 2022-10-09 16:59:53] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-09 16:59:53] ...trap/engines/engine.py.326: process: 0.32388 | load: 0.00021 [I 2022-10-09 16:59:56] ...trap/engines/engine.py.321: eval: epoch 47 | batch 350/510 [I 2022-10-09 16:59:56] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-09 16:59:56] ...trap/engines/engine.py.326: process: 0.32877 | load: 0.00020 [I 2022-10-09 16:59:59] ...trap/engines/engine.py.321: eval: epoch 47 | batch 360/510 [I 2022-10-09 16:59:59] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-09 16:59:59] ...trap/engines/engine.py.326: process: 0.31450 | load: 0.00021 [I 2022-10-09 17:00:03] ...trap/engines/engine.py.321: eval: epoch 47 | batch 370/510 [I 2022-10-09 17:00:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 17:00:03] ...trap/engines/engine.py.326: process: 0.32015 | load: 0.00020 [I 2022-10-09 17:00:06] ...trap/engines/engine.py.321: eval: epoch 47 | batch 380/510 [I 2022-10-09 17:00:06] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 17:00:06] ...trap/engines/engine.py.326: process: 0.32680 | load: 0.00021 [I 2022-10-09 17:00:09] ...trap/engines/engine.py.321: eval: epoch 47 | batch 390/510 [I 2022-10-09 17:00:09] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-09 17:00:09] ...trap/engines/engine.py.326: process: 0.31363 | load: 0.00022 [I 2022-10-09 17:00:12] ...trap/engines/engine.py.321: eval: epoch 47 | batch 400/510 [I 2022-10-09 17:00:12] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-09 17:00:12] ...trap/engines/engine.py.326: process: 0.32419 | load: 0.00020 [I 2022-10-09 17:00:15] ...trap/engines/engine.py.321: eval: epoch 47 | batch 410/510 [I 2022-10-09 17:00:15] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-09 17:00:15] ...trap/engines/engine.py.326: process: 0.32299 | load: 0.00021 [I 2022-10-09 17:00:19] ...trap/engines/engine.py.321: eval: epoch 47 | batch 420/510 [I 2022-10-09 17:00:19] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 17:00:19] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00021 [I 2022-10-09 17:00:22] ...trap/engines/engine.py.321: eval: epoch 47 | batch 430/510 [I 2022-10-09 17:00:22] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 17:00:22] ...trap/engines/engine.py.326: process: 0.32395 | load: 0.00021 [I 2022-10-09 17:00:25] ...trap/engines/engine.py.321: eval: epoch 47 | batch 440/510 [I 2022-10-09 17:00:25] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-09 17:00:25] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00021 [I 2022-10-09 17:00:28] ...trap/engines/engine.py.321: eval: epoch 47 | batch 450/510 [I 2022-10-09 17:00:28] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 17:00:28] ...trap/engines/engine.py.326: process: 0.32142 | load: 0.00021 [I 2022-10-09 17:00:31] ...trap/engines/engine.py.321: eval: epoch 47 | batch 460/510 [I 2022-10-09 17:00:31] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-09 17:00:31] ...trap/engines/engine.py.326: process: 0.32579 | load: 0.00020 [I 2022-10-09 17:00:35] ...trap/engines/engine.py.321: eval: epoch 47 | batch 470/510 [I 2022-10-09 17:00:35] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 17:00:35] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00021 [I 2022-10-09 17:00:38] ...trap/engines/engine.py.321: eval: epoch 47 | batch 480/510 [I 2022-10-09 17:00:38] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-09 17:00:38] ...trap/engines/engine.py.326: process: 0.33377 | load: 0.00019 [I 2022-10-09 17:00:41] ...trap/engines/engine.py.321: eval: epoch 47 | batch 490/510 [I 2022-10-09 17:00:41] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 17:00:41] ...trap/engines/engine.py.326: process: 0.32409 | load: 0.00022 [I 2022-10-09 17:00:44] ...trap/engines/engine.py.321: eval: epoch 47 | batch 500/510 [I 2022-10-09 17:00:44] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 17:00:44] ...trap/engines/engine.py.326: process: 0.31792 | load: 0.00020 [I 2022-10-09 17:00:48] ...trap/engines/engine.py.321: eval: epoch 47 | batch 510/510 [I 2022-10-09 17:00:48] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 17:00:48] ...trap/engines/engine.py.326: process: 0.31831 | load: 0.00018 [S 2022-10-09 17:00:48] ...trap/engines/engine.py.132: eval_epoch.epoch: 47 [I 2022-10-09 17:00:48] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.716900 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906800 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941600 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.711200 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906500 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937900 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015890 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005363 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006312 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.019585 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006652 [S 2022-10-09 17:01:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.003885 [I 2022-10-09 17:01:24] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 17:01:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 17:01:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 17:01:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 17:01:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 17:01:27] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 17:01:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 17:01:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 17:01:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 48 [I 2022-10-09 17:01:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 17:01:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 0/2304 [I 2022-10-09 17:01:37] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:40 [I 2022-10-09 17:01:37] ...trap/engines/engine.py.229: process: 2.77797 | load: 1.87341 [I 2022-10-09 17:01:37] ...trap/engines/engine.py.230: loss: 0.13779 [I 2022-10-09 17:01:37] ...optimizers/trijoint.py.151: total_norm: 14.927041 [I 2022-10-09 17:01:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 10/2304 [I 2022-10-09 17:01:45] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:03 [I 2022-10-09 17:01:45] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00546 [I 2022-10-09 17:01:45] ...trap/engines/engine.py.230: loss: 0.12777 [I 2022-10-09 17:01:45] ...optimizers/trijoint.py.151: total_norm: 28.164639 [I 2022-10-09 17:01:54] ...trap/engines/engine.py.224: train: epoch 48 | batch 20/2304 [I 2022-10-09 17:01:54] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:54 [I 2022-10-09 17:01:54] ...trap/engines/engine.py.229: process: 0.84837 | load: 0.00450 [I 2022-10-09 17:01:54] ...trap/engines/engine.py.230: loss: 0.17290 [I 2022-10-09 17:01:54] ...optimizers/trijoint.py.151: total_norm: 16.173338 [I 2022-10-09 17:02:02] ...trap/engines/engine.py.224: train: epoch 48 | batch 30/2304 [I 2022-10-09 17:02:02] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:48 [I 2022-10-09 17:02:02] ...trap/engines/engine.py.229: process: 0.83137 | load: 0.00362 [I 2022-10-09 17:02:02] ...trap/engines/engine.py.230: loss: 0.21946 [I 2022-10-09 17:02:02] ...optimizers/trijoint.py.151: total_norm: 16.341602 [I 2022-10-09 17:02:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 40/2304 [I 2022-10-09 17:02:10] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:34 [I 2022-10-09 17:02:10] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00342 [I 2022-10-09 17:02:10] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-09 17:02:10] ...optimizers/trijoint.py.151: total_norm: 16.554548 [I 2022-10-09 17:02:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 50/2304 [I 2022-10-09 17:02:19] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:42 [I 2022-10-09 17:02:19] ...trap/engines/engine.py.229: process: 0.83983 | load: 0.00433 [I 2022-10-09 17:02:19] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-09 17:02:19] ...optimizers/trijoint.py.151: total_norm: 23.223156 [I 2022-10-09 17:02:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 60/2304 [I 2022-10-09 17:02:27] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:26 [I 2022-10-09 17:02:27] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00365 [I 2022-10-09 17:02:27] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-09 17:02:27] ...optimizers/trijoint.py.151: total_norm: 13.572295 [I 2022-10-09 17:02:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 70/2304 [I 2022-10-09 17:02:36] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:16 [I 2022-10-09 17:02:36] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00332 [I 2022-10-09 17:02:36] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-09 17:02:36] ...optimizers/trijoint.py.151: total_norm: 13.750629 [I 2022-10-09 17:02:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 80/2304 [I 2022-10-09 17:02:44] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:17 [I 2022-10-09 17:02:44] ...trap/engines/engine.py.229: process: 0.84430 | load: 0.00347 [I 2022-10-09 17:02:44] ...trap/engines/engine.py.230: loss: 0.10217 [I 2022-10-09 17:02:44] ...optimizers/trijoint.py.151: total_norm: 10.292353 [I 2022-10-09 17:02:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 90/2304 [I 2022-10-09 17:02:52] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:07 [I 2022-10-09 17:02:52] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00306 [I 2022-10-09 17:02:52] ...trap/engines/engine.py.230: loss: 0.11471 [I 2022-10-09 17:02:52] ...optimizers/trijoint.py.151: total_norm: 10.358160 [I 2022-10-09 17:03:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 100/2304 [I 2022-10-09 17:03:01] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:30:53 [I 2022-10-09 17:03:01] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00303 [I 2022-10-09 17:03:01] ...trap/engines/engine.py.230: loss: 0.10757 [I 2022-10-09 17:03:01] ...optimizers/trijoint.py.151: total_norm: 16.247265 [I 2022-10-09 17:03:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 110/2304 [I 2022-10-09 17:03:09] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:56 [I 2022-10-09 17:03:09] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00315 [I 2022-10-09 17:03:09] ...trap/engines/engine.py.230: loss: 0.11452 [I 2022-10-09 17:03:09] ...optimizers/trijoint.py.151: total_norm: 14.986731 [I 2022-10-09 17:03:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 120/2304 [I 2022-10-09 17:03:18] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:42 [I 2022-10-09 17:03:18] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00296 [I 2022-10-09 17:03:18] ...trap/engines/engine.py.230: loss: 0.15144 [I 2022-10-09 17:03:18] ...optimizers/trijoint.py.151: total_norm: 8.712432 [I 2022-10-09 17:03:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 130/2304 [I 2022-10-09 17:03:26] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:33 [I 2022-10-09 17:03:26] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00319 [I 2022-10-09 17:03:26] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-09 17:03:26] ...optimizers/trijoint.py.151: total_norm: 12.601983 [I 2022-10-09 17:03:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 140/2304 [I 2022-10-09 17:03:35] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:30:25 [I 2022-10-09 17:03:35] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00329 [I 2022-10-09 17:03:35] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-09 17:03:35] ...optimizers/trijoint.py.151: total_norm: 14.120620 [I 2022-10-09 17:03:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 150/2304 [I 2022-10-09 17:03:43] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:14 [I 2022-10-09 17:03:43] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00306 [I 2022-10-09 17:03:43] ...trap/engines/engine.py.230: loss: 0.19635 [I 2022-10-09 17:03:43] ...optimizers/trijoint.py.151: total_norm: 46.400257 [I 2022-10-09 17:03:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 160/2304 [I 2022-10-09 17:03:52] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:29:53 [I 2022-10-09 17:03:52] ...trap/engines/engine.py.229: process: 0.82795 | load: 0.00310 [I 2022-10-09 17:03:52] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-09 17:03:52] ...optimizers/trijoint.py.151: total_norm: 20.365221 [I 2022-10-09 17:04:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 170/2304 [I 2022-10-09 17:04:00] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:05 [I 2022-10-09 17:04:00] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00307 [I 2022-10-09 17:04:00] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-09 17:04:00] ...optimizers/trijoint.py.151: total_norm: 10.293572 [I 2022-10-09 17:04:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 180/2304 [I 2022-10-09 17:04:08] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:34 [I 2022-10-09 17:04:08] ...trap/engines/engine.py.229: process: 0.82817 | load: 0.00297 [I 2022-10-09 17:04:08] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-09 17:04:08] ...optimizers/trijoint.py.151: total_norm: 22.024956 [I 2022-10-09 17:04:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 190/2304 [I 2022-10-09 17:04:17] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:52 [I 2022-10-09 17:04:17] ...trap/engines/engine.py.229: process: 0.85829 | load: 0.00308 [I 2022-10-09 17:04:17] ...trap/engines/engine.py.230: loss: 0.11064 [I 2022-10-09 17:04:17] ...optimizers/trijoint.py.151: total_norm: 10.795911 [I 2022-10-09 17:04:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 200/2304 [I 2022-10-09 17:04:25] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:41 [I 2022-10-09 17:04:25] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00320 [I 2022-10-09 17:04:25] ...trap/engines/engine.py.230: loss: 0.10481 [I 2022-10-09 17:04:25] ...optimizers/trijoint.py.151: total_norm: 11.457780 [I 2022-10-09 17:04:34] ...trap/engines/engine.py.224: train: epoch 48 | batch 210/2304 [I 2022-10-09 17:04:34] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:40 [I 2022-10-09 17:04:34] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00295 [I 2022-10-09 17:04:34] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-09 17:04:34] ...optimizers/trijoint.py.151: total_norm: 13.860828 [I 2022-10-09 17:04:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 220/2304 [I 2022-10-09 17:04:42] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:17 [I 2022-10-09 17:04:42] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00321 [I 2022-10-09 17:04:42] ...trap/engines/engine.py.230: loss: 0.13108 [I 2022-10-09 17:04:42] ...optimizers/trijoint.py.151: total_norm: 9.609619 [I 2022-10-09 17:04:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 230/2304 [I 2022-10-09 17:04:51] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:13 [I 2022-10-09 17:04:51] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00306 [I 2022-10-09 17:04:51] ...trap/engines/engine.py.230: loss: 0.20755 [I 2022-10-09 17:04:51] ...optimizers/trijoint.py.151: total_norm: 28.398027 [I 2022-10-09 17:04:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 240/2304 [I 2022-10-09 17:04:59] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:00 [I 2022-10-09 17:04:59] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00296 [I 2022-10-09 17:04:59] ...trap/engines/engine.py.230: loss: 0.12735 [I 2022-10-09 17:04:59] ...optimizers/trijoint.py.151: total_norm: 17.675335 [I 2022-10-09 17:05:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 250/2304 [I 2022-10-09 17:05:08] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:44 [I 2022-10-09 17:05:08] ...trap/engines/engine.py.229: process: 0.83019 | load: 0.00293 [I 2022-10-09 17:05:08] ...trap/engines/engine.py.230: loss: 0.15620 [I 2022-10-09 17:05:08] ...optimizers/trijoint.py.151: total_norm: 16.310049 [I 2022-10-09 17:05:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 260/2304 [I 2022-10-09 17:05:16] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:44 [I 2022-10-09 17:05:16] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00295 [I 2022-10-09 17:05:16] ...trap/engines/engine.py.230: loss: 0.17974 [I 2022-10-09 17:05:16] ...optimizers/trijoint.py.151: total_norm: 44.605953 [I 2022-10-09 17:05:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 270/2304 [I 2022-10-09 17:05:25] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:46 [I 2022-10-09 17:05:25] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00339 [I 2022-10-09 17:05:25] ...trap/engines/engine.py.230: loss: 0.19249 [I 2022-10-09 17:05:25] ...optimizers/trijoint.py.151: total_norm: 20.731874 [I 2022-10-09 17:05:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 280/2304 [I 2022-10-09 17:05:33] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:44 [I 2022-10-09 17:05:33] ...trap/engines/engine.py.229: process: 0.85762 | load: 0.00314 [I 2022-10-09 17:05:33] ...trap/engines/engine.py.230: loss: 0.12839 [I 2022-10-09 17:05:33] ...optimizers/trijoint.py.151: total_norm: 15.429834 [I 2022-10-09 17:05:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 290/2304 [I 2022-10-09 17:05:42] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:49 [I 2022-10-09 17:05:42] ...trap/engines/engine.py.229: process: 0.86783 | load: 0.00328 [I 2022-10-09 17:05:42] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-09 17:05:42] ...optimizers/trijoint.py.151: total_norm: 18.282633 [I 2022-10-09 17:05:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 300/2304 [I 2022-10-09 17:05:50] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:36 [I 2022-10-09 17:05:50] ...trap/engines/engine.py.229: process: 0.85855 | load: 0.00327 [I 2022-10-09 17:05:50] ...trap/engines/engine.py.230: loss: 0.22567 [I 2022-10-09 17:05:50] ...optimizers/trijoint.py.151: total_norm: 19.422823 [I 2022-10-09 17:05:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 310/2304 [I 2022-10-09 17:05:59] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:14 [I 2022-10-09 17:05:59] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00331 [I 2022-10-09 17:05:59] ...trap/engines/engine.py.230: loss: 0.16672 [I 2022-10-09 17:05:59] ...optimizers/trijoint.py.151: total_norm: 16.990023 [I 2022-10-09 17:06:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 320/2304 [I 2022-10-09 17:06:07] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:00 [I 2022-10-09 17:06:07] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00298 [I 2022-10-09 17:06:07] ...trap/engines/engine.py.230: loss: 0.14283 [I 2022-10-09 17:06:07] ...optimizers/trijoint.py.151: total_norm: 9.589437 [I 2022-10-09 17:06:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 330/2304 [I 2022-10-09 17:06:16] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:54 [I 2022-10-09 17:06:16] ...trap/engines/engine.py.229: process: 0.84303 | load: 0.00312 [I 2022-10-09 17:06:16] ...trap/engines/engine.py.230: loss: 0.20382 [I 2022-10-09 17:06:16] ...optimizers/trijoint.py.151: total_norm: 27.473076 [I 2022-10-09 17:06:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 340/2304 [I 2022-10-09 17:06:25] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:57 [I 2022-10-09 17:06:25] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00327 [I 2022-10-09 17:06:25] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-09 17:06:25] ...optimizers/trijoint.py.151: total_norm: 15.126235 [I 2022-10-09 17:06:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 350/2304 [I 2022-10-09 17:06:33] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:27:45 [I 2022-10-09 17:06:33] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00313 [I 2022-10-09 17:06:33] ...trap/engines/engine.py.230: loss: 0.14417 [I 2022-10-09 17:06:33] ...optimizers/trijoint.py.151: total_norm: 10.276706 [I 2022-10-09 17:06:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 360/2304 [I 2022-10-09 17:06:42] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:45 [I 2022-10-09 17:06:42] ...trap/engines/engine.py.229: process: 0.86972 | load: 0.00336 [I 2022-10-09 17:06:42] ...trap/engines/engine.py.230: loss: 0.14346 [I 2022-10-09 17:06:42] ...optimizers/trijoint.py.151: total_norm: 24.302603 [I 2022-10-09 17:06:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 370/2304 [I 2022-10-09 17:06:50] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:27:34 [I 2022-10-09 17:06:50] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00311 [I 2022-10-09 17:06:50] ...trap/engines/engine.py.230: loss: 0.15983 [I 2022-10-09 17:06:50] ...optimizers/trijoint.py.151: total_norm: 8.001770 [I 2022-10-09 17:06:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 380/2304 [I 2022-10-09 17:06:59] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:25 [I 2022-10-09 17:06:59] ...trap/engines/engine.py.229: process: 0.85838 | load: 0.00315 [I 2022-10-09 17:06:59] ...trap/engines/engine.py.230: loss: 0.20421 [I 2022-10-09 17:06:59] ...optimizers/trijoint.py.151: total_norm: 24.747019 [I 2022-10-09 17:07:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 390/2304 [I 2022-10-09 17:07:07] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:20 [I 2022-10-09 17:07:07] ...trap/engines/engine.py.229: process: 0.86474 | load: 0.00315 [I 2022-10-09 17:07:07] ...trap/engines/engine.py.230: loss: 0.12376 [I 2022-10-09 17:07:07] ...optimizers/trijoint.py.151: total_norm: 15.033982 [I 2022-10-09 17:07:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 400/2304 [I 2022-10-09 17:07:16] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:27:15 [I 2022-10-09 17:07:16] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00338 [I 2022-10-09 17:07:16] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-09 17:07:16] ...optimizers/trijoint.py.151: total_norm: 10.238843 [I 2022-10-09 17:07:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 410/2304 [I 2022-10-09 17:07:24] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:27:01 [I 2022-10-09 17:07:24] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00326 [I 2022-10-09 17:07:24] ...trap/engines/engine.py.230: loss: 0.13737 [I 2022-10-09 17:07:24] ...optimizers/trijoint.py.151: total_norm: 11.678552 [I 2022-10-09 17:07:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 420/2304 [I 2022-10-09 17:07:33] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:26:48 [I 2022-10-09 17:07:33] ...trap/engines/engine.py.229: process: 0.85102 | load: 0.00317 [I 2022-10-09 17:07:33] ...trap/engines/engine.py.230: loss: 0.10256 [I 2022-10-09 17:07:33] ...optimizers/trijoint.py.151: total_norm: 6.652769 [I 2022-10-09 17:07:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 430/2304 [I 2022-10-09 17:07:42] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:34 [I 2022-10-09 17:07:42] ...trap/engines/engine.py.229: process: 0.85212 | load: 0.00314 [I 2022-10-09 17:07:42] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-09 17:07:42] ...optimizers/trijoint.py.151: total_norm: 21.503376 [I 2022-10-09 17:07:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 440/2304 [I 2022-10-09 17:07:50] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:27 [I 2022-10-09 17:07:50] ...trap/engines/engine.py.229: process: 0.84292 | load: 0.00311 [I 2022-10-09 17:07:50] ...trap/engines/engine.py.230: loss: 0.14476 [I 2022-10-09 17:07:50] ...optimizers/trijoint.py.151: total_norm: 9.855662 [I 2022-10-09 17:07:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 450/2304 [I 2022-10-09 17:07:59] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:26 [I 2022-10-09 17:07:59] ...trap/engines/engine.py.229: process: 0.87008 | load: 0.00315 [I 2022-10-09 17:07:59] ...trap/engines/engine.py.230: loss: 0.13757 [I 2022-10-09 17:07:59] ...optimizers/trijoint.py.151: total_norm: 8.954793 [I 2022-10-09 17:08:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 460/2304 [I 2022-10-09 17:08:07] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:26:22 [I 2022-10-09 17:08:07] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00325 [I 2022-10-09 17:08:07] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-09 17:08:07] ...optimizers/trijoint.py.151: total_norm: 10.731118 [I 2022-10-09 17:08:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 470/2304 [I 2022-10-09 17:08:16] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:26:01 [I 2022-10-09 17:08:16] ...trap/engines/engine.py.229: process: 0.84853 | load: 0.00305 [I 2022-10-09 17:08:16] ...trap/engines/engine.py.230: loss: 0.19460 [I 2022-10-09 17:08:16] ...optimizers/trijoint.py.151: total_norm: 34.170731 [I 2022-10-09 17:08:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 480/2304 [I 2022-10-09 17:08:24] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:50 [I 2022-10-09 17:08:24] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00310 [I 2022-10-09 17:08:24] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-09 17:08:24] ...optimizers/trijoint.py.151: total_norm: 14.235839 [I 2022-10-09 17:08:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 490/2304 [I 2022-10-09 17:08:33] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:25:42 [I 2022-10-09 17:08:33] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00309 [I 2022-10-09 17:08:33] ...trap/engines/engine.py.230: loss: 0.14929 [I 2022-10-09 17:08:33] ...optimizers/trijoint.py.151: total_norm: 12.219632 [I 2022-10-09 17:08:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 500/2304 [I 2022-10-09 17:08:41] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:35 [I 2022-10-09 17:08:41] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00311 [I 2022-10-09 17:08:41] ...trap/engines/engine.py.230: loss: 0.21469 [I 2022-10-09 17:08:41] ...optimizers/trijoint.py.151: total_norm: 29.268063 [I 2022-10-09 17:08:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 510/2304 [I 2022-10-09 17:08:50] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:29 [I 2022-10-09 17:08:50] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00319 [I 2022-10-09 17:08:50] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-09 17:08:50] ...optimizers/trijoint.py.151: total_norm: 14.546328 [I 2022-10-09 17:08:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 520/2304 [I 2022-10-09 17:08:58] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:24 [I 2022-10-09 17:08:58] ...trap/engines/engine.py.229: process: 0.86231 | load: 0.00311 [I 2022-10-09 17:08:58] ...trap/engines/engine.py.230: loss: 0.22868 [I 2022-10-09 17:08:58] ...optimizers/trijoint.py.151: total_norm: 17.196875 [I 2022-10-09 17:09:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 530/2304 [I 2022-10-09 17:09:07] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:09 [I 2022-10-09 17:09:07] ...trap/engines/engine.py.229: process: 0.84302 | load: 0.00311 [I 2022-10-09 17:09:07] ...trap/engines/engine.py.230: loss: 0.18730 [I 2022-10-09 17:09:07] ...optimizers/trijoint.py.151: total_norm: 28.373552 [I 2022-10-09 17:09:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 540/2304 [I 2022-10-09 17:09:15] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:24:59 [I 2022-10-09 17:09:15] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00310 [I 2022-10-09 17:09:15] ...trap/engines/engine.py.230: loss: 0.15952 [I 2022-10-09 17:09:15] ...optimizers/trijoint.py.151: total_norm: 23.174265 [I 2022-10-09 17:09:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 550/2304 [I 2022-10-09 17:09:24] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:46 [I 2022-10-09 17:09:24] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00314 [I 2022-10-09 17:09:24] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-09 17:09:24] ...optimizers/trijoint.py.151: total_norm: 10.725023 [I 2022-10-09 17:09:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 560/2304 [I 2022-10-09 17:09:32] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:38 [I 2022-10-09 17:09:32] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00311 [I 2022-10-09 17:09:32] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-09 17:09:32] ...optimizers/trijoint.py.151: total_norm: 20.863817 [I 2022-10-09 17:09:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 570/2304 [I 2022-10-09 17:09:41] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:47 [I 2022-10-09 17:09:41] ...trap/engines/engine.py.229: process: 0.86890 | load: 0.00309 [I 2022-10-09 17:09:41] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-09 17:09:41] ...optimizers/trijoint.py.151: total_norm: 11.637753 [I 2022-10-09 17:09:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 580/2304 [I 2022-10-09 17:09:50] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:32 [I 2022-10-09 17:09:50] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00308 [I 2022-10-09 17:09:50] ...trap/engines/engine.py.230: loss: 0.12896 [I 2022-10-09 17:09:50] ...optimizers/trijoint.py.151: total_norm: 20.115286 [I 2022-10-09 17:09:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 590/2304 [I 2022-10-09 17:09:58] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:22 [I 2022-10-09 17:09:58] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00298 [I 2022-10-09 17:09:58] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-09 17:09:58] ...optimizers/trijoint.py.151: total_norm: 50.950760 [I 2022-10-09 17:10:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 600/2304 [I 2022-10-09 17:10:07] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:15 [I 2022-10-09 17:10:07] ...trap/engines/engine.py.229: process: 0.86651 | load: 0.00305 [I 2022-10-09 17:10:07] ...trap/engines/engine.py.230: loss: 0.14800 [I 2022-10-09 17:10:07] ...optimizers/trijoint.py.151: total_norm: 17.952234 [I 2022-10-09 17:10:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 610/2304 [I 2022-10-09 17:10:15] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:23:56 [I 2022-10-09 17:10:15] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00300 [I 2022-10-09 17:10:15] ...trap/engines/engine.py.230: loss: 0.21419 [I 2022-10-09 17:10:15] ...optimizers/trijoint.py.151: total_norm: 40.366249 [I 2022-10-09 17:10:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 620/2304 [I 2022-10-09 17:10:24] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:23:43 [I 2022-10-09 17:10:24] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00300 [I 2022-10-09 17:10:24] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-09 17:10:24] ...optimizers/trijoint.py.151: total_norm: 16.270821 [I 2022-10-09 17:10:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 630/2304 [I 2022-10-09 17:10:32] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:44 [I 2022-10-09 17:10:32] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00307 [I 2022-10-09 17:10:32] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-09 17:10:32] ...optimizers/trijoint.py.151: total_norm: 15.300749 [I 2022-10-09 17:10:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 640/2304 [I 2022-10-09 17:10:41] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:25 [I 2022-10-09 17:10:41] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00301 [I 2022-10-09 17:10:41] ...trap/engines/engine.py.230: loss: 0.20655 [I 2022-10-09 17:10:41] ...optimizers/trijoint.py.151: total_norm: 13.302906 [I 2022-10-09 17:10:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 650/2304 [I 2022-10-09 17:10:49] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:19 [I 2022-10-09 17:10:49] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00292 [I 2022-10-09 17:10:49] ...trap/engines/engine.py.230: loss: 0.15354 [I 2022-10-09 17:10:49] ...optimizers/trijoint.py.151: total_norm: 14.022754 [I 2022-10-09 17:10:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 660/2304 [I 2022-10-09 17:10:58] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:00 [I 2022-10-09 17:10:58] ...trap/engines/engine.py.229: process: 0.82709 | load: 0.00290 [I 2022-10-09 17:10:58] ...trap/engines/engine.py.230: loss: 0.10877 [I 2022-10-09 17:10:58] ...optimizers/trijoint.py.151: total_norm: 7.026793 [I 2022-10-09 17:11:06] ...trap/engines/engine.py.224: train: epoch 48 | batch 670/2304 [I 2022-10-09 17:11:06] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:22:59 [I 2022-10-09 17:11:06] ...trap/engines/engine.py.229: process: 0.85506 | load: 0.00292 [I 2022-10-09 17:11:06] ...trap/engines/engine.py.230: loss: 0.13490 [I 2022-10-09 17:11:06] ...optimizers/trijoint.py.151: total_norm: 15.200771 [I 2022-10-09 17:11:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 680/2304 [I 2022-10-09 17:11:14] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:22:50 [I 2022-10-09 17:11:14] ...trap/engines/engine.py.229: process: 0.84488 | load: 0.00293 [I 2022-10-09 17:11:14] ...trap/engines/engine.py.230: loss: 0.14081 [I 2022-10-09 17:11:14] ...optimizers/trijoint.py.151: total_norm: 16.379145 [I 2022-10-09 17:11:23] ...trap/engines/engine.py.224: train: epoch 48 | batch 690/2304 [I 2022-10-09 17:11:23] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:22:47 [I 2022-10-09 17:11:23] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00291 [I 2022-10-09 17:11:23] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-09 17:11:23] ...optimizers/trijoint.py.151: total_norm: 19.812441 [I 2022-10-09 17:11:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 700/2304 [I 2022-10-09 17:11:31] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:22:36 [I 2022-10-09 17:11:31] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00297 [I 2022-10-09 17:11:31] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-09 17:11:31] ...optimizers/trijoint.py.151: total_norm: 15.942246 [I 2022-10-09 17:11:40] ...trap/engines/engine.py.224: train: epoch 48 | batch 710/2304 [I 2022-10-09 17:11:40] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:22 [I 2022-10-09 17:11:40] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00306 [I 2022-10-09 17:11:40] ...trap/engines/engine.py.230: loss: 0.17523 [I 2022-10-09 17:11:40] ...optimizers/trijoint.py.151: total_norm: 13.452449 [I 2022-10-09 17:11:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 720/2304 [I 2022-10-09 17:11:48] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:23 [I 2022-10-09 17:11:48] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00297 [I 2022-10-09 17:11:48] ...trap/engines/engine.py.230: loss: 0.16024 [I 2022-10-09 17:11:48] ...optimizers/trijoint.py.151: total_norm: 8.283153 [I 2022-10-09 17:11:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 730/2304 [I 2022-10-09 17:11:57] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:11 [I 2022-10-09 17:11:57] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00297 [I 2022-10-09 17:11:57] ...trap/engines/engine.py.230: loss: 0.13420 [I 2022-10-09 17:11:57] ...optimizers/trijoint.py.151: total_norm: 12.777188 [I 2022-10-09 17:12:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 740/2304 [I 2022-10-09 17:12:05] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:03 [I 2022-10-09 17:12:05] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00296 [I 2022-10-09 17:12:05] ...trap/engines/engine.py.230: loss: 0.22077 [I 2022-10-09 17:12:05] ...optimizers/trijoint.py.151: total_norm: 20.531515 [I 2022-10-09 17:12:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 750/2304 [I 2022-10-09 17:12:14] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:21:57 [I 2022-10-09 17:12:14] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00296 [I 2022-10-09 17:12:14] ...trap/engines/engine.py.230: loss: 0.14342 [I 2022-10-09 17:12:14] ...optimizers/trijoint.py.151: total_norm: 17.136845 [I 2022-10-09 17:12:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 760/2304 [I 2022-10-09 17:12:22] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:47 [I 2022-10-09 17:12:22] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00308 [I 2022-10-09 17:12:22] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-09 17:12:22] ...optimizers/trijoint.py.151: total_norm: 12.235263 [I 2022-10-09 17:12:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 770/2304 [I 2022-10-09 17:12:31] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:21:37 [I 2022-10-09 17:12:31] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00297 [I 2022-10-09 17:12:31] ...trap/engines/engine.py.230: loss: 0.15990 [I 2022-10-09 17:12:31] ...optimizers/trijoint.py.151: total_norm: 30.493519 [I 2022-10-09 17:12:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 780/2304 [I 2022-10-09 17:12:39] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:32 [I 2022-10-09 17:12:39] ...trap/engines/engine.py.229: process: 0.85149 | load: 0.00297 [I 2022-10-09 17:12:39] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-09 17:12:39] ...optimizers/trijoint.py.151: total_norm: 17.807608 [I 2022-10-09 17:12:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 790/2304 [I 2022-10-09 17:12:48] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:23 [I 2022-10-09 17:12:48] ...trap/engines/engine.py.229: process: 0.85157 | load: 0.00298 [I 2022-10-09 17:12:48] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-09 17:12:48] ...optimizers/trijoint.py.151: total_norm: 8.187638 [I 2022-10-09 17:12:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 800/2304 [I 2022-10-09 17:12:56] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:08 [I 2022-10-09 17:12:56] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00304 [I 2022-10-09 17:12:56] ...trap/engines/engine.py.230: loss: 0.11238 [I 2022-10-09 17:12:56] ...optimizers/trijoint.py.151: total_norm: 12.531620 [I 2022-10-09 17:13:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 810/2304 [I 2022-10-09 17:13:05] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:15 [I 2022-10-09 17:13:05] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00312 [I 2022-10-09 17:13:05] ...trap/engines/engine.py.230: loss: 0.11650 [I 2022-10-09 17:13:05] ...optimizers/trijoint.py.151: total_norm: 9.910697 [I 2022-10-09 17:13:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 820/2304 [I 2022-10-09 17:13:13] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:01 [I 2022-10-09 17:13:13] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00312 [I 2022-10-09 17:13:13] ...trap/engines/engine.py.230: loss: 0.23749 [I 2022-10-09 17:13:13] ...optimizers/trijoint.py.151: total_norm: 26.522327 [I 2022-10-09 17:13:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 830/2304 [I 2022-10-09 17:13:22] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:21:02 [I 2022-10-09 17:13:22] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00314 [I 2022-10-09 17:13:22] ...trap/engines/engine.py.230: loss: 0.18087 [I 2022-10-09 17:13:22] ...optimizers/trijoint.py.151: total_norm: 16.713984 [I 2022-10-09 17:13:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 840/2304 [I 2022-10-09 17:13:30] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:48 [I 2022-10-09 17:13:30] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00319 [I 2022-10-09 17:13:30] ...trap/engines/engine.py.230: loss: 0.21393 [I 2022-10-09 17:13:30] ...optimizers/trijoint.py.151: total_norm: 23.959164 [I 2022-10-09 17:13:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 850/2304 [I 2022-10-09 17:13:39] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:20:37 [I 2022-10-09 17:13:39] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00312 [I 2022-10-09 17:13:39] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-09 17:13:39] ...optimizers/trijoint.py.151: total_norm: 12.973899 [I 2022-10-09 17:13:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 860/2304 [I 2022-10-09 17:13:48] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:35 [I 2022-10-09 17:13:48] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00313 [I 2022-10-09 17:13:48] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-09 17:13:48] ...optimizers/trijoint.py.151: total_norm: 19.278971 [I 2022-10-09 17:13:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 870/2304 [I 2022-10-09 17:13:56] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:20:13 [I 2022-10-09 17:13:56] ...trap/engines/engine.py.229: process: 0.83821 | load: 0.00316 [I 2022-10-09 17:13:56] ...trap/engines/engine.py.230: loss: 0.12774 [I 2022-10-09 17:13:56] ...optimizers/trijoint.py.151: total_norm: 13.221710 [I 2022-10-09 17:14:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 880/2304 [I 2022-10-09 17:14:05] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:10 [I 2022-10-09 17:14:05] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00302 [I 2022-10-09 17:14:05] ...trap/engines/engine.py.230: loss: 0.22580 [I 2022-10-09 17:14:05] ...optimizers/trijoint.py.151: total_norm: 15.455808 [I 2022-10-09 17:14:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 890/2304 [I 2022-10-09 17:14:13] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:20:02 [I 2022-10-09 17:14:13] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00302 [I 2022-10-09 17:14:13] ...trap/engines/engine.py.230: loss: 0.15384 [I 2022-10-09 17:14:13] ...optimizers/trijoint.py.151: total_norm: 10.317180 [I 2022-10-09 17:14:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 900/2304 [I 2022-10-09 17:14:22] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:19:59 [I 2022-10-09 17:14:22] ...trap/engines/engine.py.229: process: 0.87471 | load: 0.00321 [I 2022-10-09 17:14:22] ...trap/engines/engine.py.230: loss: 0.15331 [I 2022-10-09 17:14:22] ...optimizers/trijoint.py.151: total_norm: 17.760086 [I 2022-10-09 17:14:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 910/2304 [I 2022-10-09 17:14:30] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:19:47 [I 2022-10-09 17:14:30] ...trap/engines/engine.py.229: process: 0.84916 | load: 0.00315 [I 2022-10-09 17:14:30] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-09 17:14:30] ...optimizers/trijoint.py.151: total_norm: 17.906025 [I 2022-10-09 17:14:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 920/2304 [I 2022-10-09 17:14:39] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:42 [I 2022-10-09 17:14:39] ...trap/engines/engine.py.229: process: 0.86684 | load: 0.00320 [I 2022-10-09 17:14:39] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-09 17:14:39] ...optimizers/trijoint.py.151: total_norm: 18.260021 [I 2022-10-09 17:14:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 930/2304 [I 2022-10-09 17:14:47] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:19:25 [I 2022-10-09 17:14:47] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00317 [I 2022-10-09 17:14:47] ...trap/engines/engine.py.230: loss: 0.14609 [I 2022-10-09 17:14:47] ...optimizers/trijoint.py.151: total_norm: 9.332438 [I 2022-10-09 17:14:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 940/2304 [I 2022-10-09 17:14:56] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:23 [I 2022-10-09 17:14:56] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00311 [I 2022-10-09 17:14:56] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-09 17:14:56] ...optimizers/trijoint.py.151: total_norm: 13.885468 [I 2022-10-09 17:15:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 950/2304 [I 2022-10-09 17:15:04] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:19:20 [I 2022-10-09 17:15:04] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00311 [I 2022-10-09 17:15:04] ...trap/engines/engine.py.230: loss: 0.20625 [I 2022-10-09 17:15:04] ...optimizers/trijoint.py.151: total_norm: 18.470734 [I 2022-10-09 17:15:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 960/2304 [I 2022-10-09 17:15:13] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:19:04 [I 2022-10-09 17:15:13] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00312 [I 2022-10-09 17:15:13] ...trap/engines/engine.py.230: loss: 0.16732 [I 2022-10-09 17:15:13] ...optimizers/trijoint.py.151: total_norm: 18.644533 [I 2022-10-09 17:15:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 970/2304 [I 2022-10-09 17:15:21] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:18:51 [I 2022-10-09 17:15:21] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00312 [I 2022-10-09 17:15:21] ...trap/engines/engine.py.230: loss: 0.13217 [I 2022-10-09 17:15:21] ...optimizers/trijoint.py.151: total_norm: 7.489369 [I 2022-10-09 17:15:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 980/2304 [I 2022-10-09 17:15:30] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:47 [I 2022-10-09 17:15:30] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00308 [I 2022-10-09 17:15:30] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-09 17:15:30] ...optimizers/trijoint.py.151: total_norm: 18.958591 [I 2022-10-09 17:15:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 990/2304 [I 2022-10-09 17:15:38] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:43 [I 2022-10-09 17:15:38] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00313 [I 2022-10-09 17:15:38] ...trap/engines/engine.py.230: loss: 0.16602 [I 2022-10-09 17:15:38] ...optimizers/trijoint.py.151: total_norm: 20.013378 [I 2022-10-09 17:15:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 1000/2304 [I 2022-10-09 17:15:47] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:31 [I 2022-10-09 17:15:47] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00312 [I 2022-10-09 17:15:47] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-09 17:15:47] ...optimizers/trijoint.py.151: total_norm: 20.117275 [I 2022-10-09 17:15:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 1010/2304 [I 2022-10-09 17:15:56] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:18:28 [I 2022-10-09 17:15:56] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00324 [I 2022-10-09 17:15:56] ...trap/engines/engine.py.230: loss: 0.23112 [I 2022-10-09 17:15:56] ...optimizers/trijoint.py.151: total_norm: 29.365238 [I 2022-10-09 17:16:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 1020/2304 [I 2022-10-09 17:16:04] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:18:10 [I 2022-10-09 17:16:04] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00315 [I 2022-10-09 17:16:04] ...trap/engines/engine.py.230: loss: 0.21130 [I 2022-10-09 17:16:04] ...optimizers/trijoint.py.151: total_norm: 17.126226 [I 2022-10-09 17:16:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 1030/2304 [I 2022-10-09 17:16:13] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:18:05 [I 2022-10-09 17:16:13] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00309 [I 2022-10-09 17:16:13] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-09 17:16:13] ...optimizers/trijoint.py.151: total_norm: 13.541956 [I 2022-10-09 17:16:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1040/2304 [I 2022-10-09 17:16:21] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:17:59 [I 2022-10-09 17:16:21] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00315 [I 2022-10-09 17:16:21] ...trap/engines/engine.py.230: loss: 0.15504 [I 2022-10-09 17:16:21] ...optimizers/trijoint.py.151: total_norm: 40.679047 [I 2022-10-09 17:16:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 1050/2304 [I 2022-10-09 17:16:30] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:17:49 [I 2022-10-09 17:16:30] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00312 [I 2022-10-09 17:16:30] ...trap/engines/engine.py.230: loss: 0.14960 [I 2022-10-09 17:16:30] ...optimizers/trijoint.py.151: total_norm: 12.957970 [I 2022-10-09 17:16:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 1060/2304 [I 2022-10-09 17:16:38] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:17:36 [I 2022-10-09 17:16:38] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00311 [I 2022-10-09 17:16:38] ...trap/engines/engine.py.230: loss: 0.15256 [I 2022-10-09 17:16:38] ...optimizers/trijoint.py.151: total_norm: 34.727524 [I 2022-10-09 17:16:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 1070/2304 [I 2022-10-09 17:16:47] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:48 [I 2022-10-09 17:16:47] ...trap/engines/engine.py.229: process: 0.88765 | load: 0.00311 [I 2022-10-09 17:16:47] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-09 17:16:47] ...optimizers/trijoint.py.151: total_norm: 11.365255 [I 2022-10-09 17:16:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 1080/2304 [I 2022-10-09 17:16:55] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:17:25 [I 2022-10-09 17:16:55] ...trap/engines/engine.py.229: process: 0.85033 | load: 0.00312 [I 2022-10-09 17:16:55] ...trap/engines/engine.py.230: loss: 0.14203 [I 2022-10-09 17:16:55] ...optimizers/trijoint.py.151: total_norm: 11.406132 [I 2022-10-09 17:17:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 1090/2304 [I 2022-10-09 17:17:04] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:16 [I 2022-10-09 17:17:04] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00316 [I 2022-10-09 17:17:04] ...trap/engines/engine.py.230: loss: 0.12454 [I 2022-10-09 17:17:04] ...optimizers/trijoint.py.151: total_norm: 22.083778 [I 2022-10-09 17:17:12] ...trap/engines/engine.py.224: train: epoch 48 | batch 1100/2304 [I 2022-10-09 17:17:12] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:17:03 [I 2022-10-09 17:17:12] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00314 [I 2022-10-09 17:17:12] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-09 17:17:12] ...optimizers/trijoint.py.151: total_norm: 15.235650 [I 2022-10-09 17:17:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1110/2304 [I 2022-10-09 17:17:21] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:59 [I 2022-10-09 17:17:21] ...trap/engines/engine.py.229: process: 0.85591 | load: 0.00310 [I 2022-10-09 17:17:21] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-09 17:17:21] ...optimizers/trijoint.py.151: total_norm: 11.488759 [I 2022-10-09 17:17:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 1120/2304 [I 2022-10-09 17:17:30] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:16:53 [I 2022-10-09 17:17:30] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00309 [I 2022-10-09 17:17:30] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-09 17:17:30] ...optimizers/trijoint.py.151: total_norm: 14.105573 [I 2022-10-09 17:17:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 1130/2304 [I 2022-10-09 17:17:38] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:35 [I 2022-10-09 17:17:38] ...trap/engines/engine.py.229: process: 0.84701 | load: 0.00300 [I 2022-10-09 17:17:38] ...trap/engines/engine.py.230: loss: 0.17132 [I 2022-10-09 17:17:38] ...optimizers/trijoint.py.151: total_norm: 61.048996 [I 2022-10-09 17:17:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 1140/2304 [I 2022-10-09 17:17:47] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:29 [I 2022-10-09 17:17:47] ...trap/engines/engine.py.229: process: 0.85963 | load: 0.00313 [I 2022-10-09 17:17:47] ...trap/engines/engine.py.230: loss: 0.16307 [I 2022-10-09 17:17:47] ...optimizers/trijoint.py.151: total_norm: 26.205286 [I 2022-10-09 17:17:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 1150/2304 [I 2022-10-09 17:17:55] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:23 [I 2022-10-09 17:17:55] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00308 [I 2022-10-09 17:17:55] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-09 17:17:55] ...optimizers/trijoint.py.151: total_norm: 10.918059 [I 2022-10-09 17:18:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 1160/2304 [I 2022-10-09 17:18:04] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:16:08 [I 2022-10-09 17:18:04] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00297 [I 2022-10-09 17:18:04] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-09 17:18:04] ...optimizers/trijoint.py.151: total_norm: 17.895773 [I 2022-10-09 17:18:12] ...trap/engines/engine.py.224: train: epoch 48 | batch 1170/2304 [I 2022-10-09 17:18:12] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:15:59 [I 2022-10-09 17:18:12] ...trap/engines/engine.py.229: process: 0.84928 | load: 0.00293 [I 2022-10-09 17:18:12] ...trap/engines/engine.py.230: loss: 0.12301 [I 2022-10-09 17:18:12] ...optimizers/trijoint.py.151: total_norm: 13.009781 [I 2022-10-09 17:18:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1180/2304 [I 2022-10-09 17:18:21] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:46 [I 2022-10-09 17:18:21] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00292 [I 2022-10-09 17:18:21] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-09 17:18:21] ...optimizers/trijoint.py.151: total_norm: 9.193037 [I 2022-10-09 17:18:29] ...trap/engines/engine.py.224: train: epoch 48 | batch 1190/2304 [I 2022-10-09 17:18:29] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:39 [I 2022-10-09 17:18:29] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00293 [I 2022-10-09 17:18:29] ...trap/engines/engine.py.230: loss: 0.19325 [I 2022-10-09 17:18:29] ...optimizers/trijoint.py.151: total_norm: 20.584728 [I 2022-10-09 17:18:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 1200/2304 [I 2022-10-09 17:18:38] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:36 [I 2022-10-09 17:18:38] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00309 [I 2022-10-09 17:18:38] ...trap/engines/engine.py.230: loss: 0.15927 [I 2022-10-09 17:18:38] ...optimizers/trijoint.py.151: total_norm: 23.576206 [I 2022-10-09 17:18:46] ...trap/engines/engine.py.224: train: epoch 48 | batch 1210/2304 [I 2022-10-09 17:18:46] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:26 [I 2022-10-09 17:18:46] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00291 [I 2022-10-09 17:18:46] ...trap/engines/engine.py.230: loss: 0.13271 [I 2022-10-09 17:18:46] ...optimizers/trijoint.py.151: total_norm: 13.067643 [I 2022-10-09 17:18:54] ...trap/engines/engine.py.224: train: epoch 48 | batch 1220/2304 [I 2022-10-09 17:18:54] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:14 [I 2022-10-09 17:18:54] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00292 [I 2022-10-09 17:18:54] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-09 17:18:54] ...optimizers/trijoint.py.151: total_norm: 13.406713 [I 2022-10-09 17:19:03] ...trap/engines/engine.py.224: train: epoch 48 | batch 1230/2304 [I 2022-10-09 17:19:03] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:05 [I 2022-10-09 17:19:03] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00304 [I 2022-10-09 17:19:03] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-09 17:19:03] ...optimizers/trijoint.py.151: total_norm: 8.261294 [I 2022-10-09 17:19:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 1240/2304 [I 2022-10-09 17:19:11] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:14:55 [I 2022-10-09 17:19:11] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00300 [I 2022-10-09 17:19:11] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-09 17:19:11] ...optimizers/trijoint.py.151: total_norm: 9.235271 [I 2022-10-09 17:19:20] ...trap/engines/engine.py.224: train: epoch 48 | batch 1250/2304 [I 2022-10-09 17:19:20] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:48 [I 2022-10-09 17:19:20] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00292 [I 2022-10-09 17:19:20] ...trap/engines/engine.py.230: loss: 0.17619 [I 2022-10-09 17:19:20] ...optimizers/trijoint.py.151: total_norm: 14.408692 [I 2022-10-09 17:19:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 1260/2304 [I 2022-10-09 17:19:28] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:39 [I 2022-10-09 17:19:28] ...trap/engines/engine.py.229: process: 0.85204 | load: 0.00296 [I 2022-10-09 17:19:28] ...trap/engines/engine.py.230: loss: 0.15142 [I 2022-10-09 17:19:28] ...optimizers/trijoint.py.151: total_norm: 12.671435 [I 2022-10-09 17:19:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 1270/2304 [I 2022-10-09 17:19:37] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:32 [I 2022-10-09 17:19:37] ...trap/engines/engine.py.229: process: 0.84472 | load: 0.00297 [I 2022-10-09 17:19:37] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-09 17:19:37] ...optimizers/trijoint.py.151: total_norm: 15.331629 [I 2022-10-09 17:19:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 1280/2304 [I 2022-10-09 17:19:45] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:26 [I 2022-10-09 17:19:45] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00311 [I 2022-10-09 17:19:45] ...trap/engines/engine.py.230: loss: 0.18988 [I 2022-10-09 17:19:45] ...optimizers/trijoint.py.151: total_norm: 21.397295 [I 2022-10-09 17:19:54] ...trap/engines/engine.py.224: train: epoch 48 | batch 1290/2304 [I 2022-10-09 17:19:54] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:21 [I 2022-10-09 17:19:54] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00298 [I 2022-10-09 17:19:54] ...trap/engines/engine.py.230: loss: 0.18414 [I 2022-10-09 17:19:54] ...optimizers/trijoint.py.151: total_norm: 41.781914 [I 2022-10-09 17:20:02] ...trap/engines/engine.py.224: train: epoch 48 | batch 1300/2304 [I 2022-10-09 17:20:02] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:06 [I 2022-10-09 17:20:02] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00300 [I 2022-10-09 17:20:02] ...trap/engines/engine.py.230: loss: 0.19005 [I 2022-10-09 17:20:02] ...optimizers/trijoint.py.151: total_norm: 33.210518 [I 2022-10-09 17:20:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 1310/2304 [I 2022-10-09 17:20:11] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:13:59 [I 2022-10-09 17:20:11] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00300 [I 2022-10-09 17:20:11] ...trap/engines/engine.py.230: loss: 0.20035 [I 2022-10-09 17:20:11] ...optimizers/trijoint.py.151: total_norm: 17.005318 [I 2022-10-09 17:20:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 1320/2304 [I 2022-10-09 17:20:19] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:13:54 [I 2022-10-09 17:20:19] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00303 [I 2022-10-09 17:20:19] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-09 17:20:19] ...optimizers/trijoint.py.151: total_norm: 10.186662 [I 2022-10-09 17:20:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 1330/2304 [I 2022-10-09 17:20:28] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:41 [I 2022-10-09 17:20:28] ...trap/engines/engine.py.229: process: 0.85035 | load: 0.00296 [I 2022-10-09 17:20:28] ...trap/engines/engine.py.230: loss: 0.11669 [I 2022-10-09 17:20:28] ...optimizers/trijoint.py.151: total_norm: 12.090897 [I 2022-10-09 17:20:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 1340/2304 [I 2022-10-09 17:20:36] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:13:32 [I 2022-10-09 17:20:36] ...trap/engines/engine.py.229: process: 0.83914 | load: 0.00298 [I 2022-10-09 17:20:36] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-09 17:20:36] ...optimizers/trijoint.py.151: total_norm: 11.479098 [I 2022-10-09 17:20:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 1350/2304 [I 2022-10-09 17:20:45] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:32 [I 2022-10-09 17:20:45] ...trap/engines/engine.py.229: process: 0.84983 | load: 0.00298 [I 2022-10-09 17:20:45] ...trap/engines/engine.py.230: loss: 0.20641 [I 2022-10-09 17:20:45] ...optimizers/trijoint.py.151: total_norm: 22.165136 [I 2022-10-09 17:20:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1360/2304 [I 2022-10-09 17:20:53] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:16 [I 2022-10-09 17:20:53] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00296 [I 2022-10-09 17:20:53] ...trap/engines/engine.py.230: loss: 0.14176 [I 2022-10-09 17:20:53] ...optimizers/trijoint.py.151: total_norm: 11.728414 [I 2022-10-09 17:21:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1370/2304 [I 2022-10-09 17:21:01] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:08 [I 2022-10-09 17:21:01] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00296 [I 2022-10-09 17:21:01] ...trap/engines/engine.py.230: loss: 0.13524 [I 2022-10-09 17:21:01] ...optimizers/trijoint.py.151: total_norm: 17.940683 [I 2022-10-09 17:21:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1380/2304 [I 2022-10-09 17:21:10] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:05 [I 2022-10-09 17:21:10] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00314 [I 2022-10-09 17:21:10] ...trap/engines/engine.py.230: loss: 0.17692 [I 2022-10-09 17:21:10] ...optimizers/trijoint.py.151: total_norm: 10.169694 [I 2022-10-09 17:21:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 1390/2304 [I 2022-10-09 17:21:19] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:12:57 [I 2022-10-09 17:21:19] ...trap/engines/engine.py.229: process: 0.84818 | load: 0.00313 [I 2022-10-09 17:21:19] ...trap/engines/engine.py.230: loss: 0.17824 [I 2022-10-09 17:21:19] ...optimizers/trijoint.py.151: total_norm: 18.538675 [I 2022-10-09 17:21:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 1400/2304 [I 2022-10-09 17:21:27] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:12:52 [I 2022-10-09 17:21:27] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00316 [I 2022-10-09 17:21:27] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-09 17:21:27] ...optimizers/trijoint.py.151: total_norm: 14.242541 [I 2022-10-09 17:21:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 1410/2304 [I 2022-10-09 17:21:36] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:12:40 [I 2022-10-09 17:21:36] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00313 [I 2022-10-09 17:21:36] ...trap/engines/engine.py.230: loss: 0.18668 [I 2022-10-09 17:21:36] ...optimizers/trijoint.py.151: total_norm: 20.981716 [I 2022-10-09 17:21:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 1420/2304 [I 2022-10-09 17:21:44] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:30 [I 2022-10-09 17:21:44] ...trap/engines/engine.py.229: process: 0.83987 | load: 0.00314 [I 2022-10-09 17:21:44] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-09 17:21:44] ...optimizers/trijoint.py.151: total_norm: 11.262043 [I 2022-10-09 17:21:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1430/2304 [I 2022-10-09 17:21:53] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:28 [I 2022-10-09 17:21:53] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00313 [I 2022-10-09 17:21:53] ...trap/engines/engine.py.230: loss: 0.16605 [I 2022-10-09 17:21:53] ...optimizers/trijoint.py.151: total_norm: 13.670996 [I 2022-10-09 17:22:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1440/2304 [I 2022-10-09 17:22:01] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:17 [I 2022-10-09 17:22:01] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00313 [I 2022-10-09 17:22:01] ...trap/engines/engine.py.230: loss: 0.15392 [I 2022-10-09 17:22:01] ...optimizers/trijoint.py.151: total_norm: 17.168322 [I 2022-10-09 17:22:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1450/2304 [I 2022-10-09 17:22:10] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:11 [I 2022-10-09 17:22:10] ...trap/engines/engine.py.229: process: 0.85826 | load: 0.00313 [I 2022-10-09 17:22:10] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-09 17:22:10] ...optimizers/trijoint.py.151: total_norm: 17.382740 [I 2022-10-09 17:22:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 1460/2304 [I 2022-10-09 17:22:18] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:11:59 [I 2022-10-09 17:22:18] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00310 [I 2022-10-09 17:22:18] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-09 17:22:18] ...optimizers/trijoint.py.151: total_norm: 10.029476 [I 2022-10-09 17:22:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 1470/2304 [I 2022-10-09 17:22:27] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:11:54 [I 2022-10-09 17:22:27] ...trap/engines/engine.py.229: process: 0.86680 | load: 0.00319 [I 2022-10-09 17:22:27] ...trap/engines/engine.py.230: loss: 0.15029 [I 2022-10-09 17:22:27] ...optimizers/trijoint.py.151: total_norm: 11.927943 [I 2022-10-09 17:22:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 1480/2304 [I 2022-10-09 17:22:36] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:11:43 [I 2022-10-09 17:22:36] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00305 [I 2022-10-09 17:22:36] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-09 17:22:36] ...optimizers/trijoint.py.151: total_norm: 12.734172 [I 2022-10-09 17:22:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 1490/2304 [I 2022-10-09 17:22:44] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:35 [I 2022-10-09 17:22:44] ...trap/engines/engine.py.229: process: 0.85124 | load: 0.00312 [I 2022-10-09 17:22:44] ...trap/engines/engine.py.230: loss: 0.10435 [I 2022-10-09 17:22:44] ...optimizers/trijoint.py.151: total_norm: 12.649270 [I 2022-10-09 17:22:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1500/2304 [I 2022-10-09 17:22:53] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:29 [I 2022-10-09 17:22:53] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00301 [I 2022-10-09 17:22:53] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-09 17:22:53] ...optimizers/trijoint.py.151: total_norm: 21.171854 [I 2022-10-09 17:23:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1510/2304 [I 2022-10-09 17:23:01] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:17 [I 2022-10-09 17:23:01] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00314 [I 2022-10-09 17:23:01] ...trap/engines/engine.py.230: loss: 0.20234 [I 2022-10-09 17:23:01] ...optimizers/trijoint.py.151: total_norm: 11.752502 [I 2022-10-09 17:23:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1520/2304 [I 2022-10-09 17:23:10] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:08 [I 2022-10-09 17:23:10] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00312 [I 2022-10-09 17:23:10] ...trap/engines/engine.py.230: loss: 0.11302 [I 2022-10-09 17:23:10] ...optimizers/trijoint.py.151: total_norm: 12.909566 [I 2022-10-09 17:23:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 1530/2304 [I 2022-10-09 17:23:18] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:10:57 [I 2022-10-09 17:23:18] ...trap/engines/engine.py.229: process: 0.84863 | load: 0.00312 [I 2022-10-09 17:23:18] ...trap/engines/engine.py.230: loss: 0.12355 [I 2022-10-09 17:23:18] ...optimizers/trijoint.py.151: total_norm: 13.400470 [I 2022-10-09 17:23:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 1540/2304 [I 2022-10-09 17:23:27] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:10:54 [I 2022-10-09 17:23:27] ...trap/engines/engine.py.229: process: 0.85972 | load: 0.00321 [I 2022-10-09 17:23:27] ...trap/engines/engine.py.230: loss: 0.17580 [I 2022-10-09 17:23:27] ...optimizers/trijoint.py.151: total_norm: 14.435359 [I 2022-10-09 17:23:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 1550/2304 [I 2022-10-09 17:23:35] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:43 [I 2022-10-09 17:23:35] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00314 [I 2022-10-09 17:23:35] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-09 17:23:35] ...optimizers/trijoint.py.151: total_norm: 21.311691 [I 2022-10-09 17:23:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 1560/2304 [I 2022-10-09 17:23:44] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:10:35 [I 2022-10-09 17:23:44] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00313 [I 2022-10-09 17:23:44] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-09 17:23:44] ...optimizers/trijoint.py.151: total_norm: 7.608707 [I 2022-10-09 17:23:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1570/2304 [I 2022-10-09 17:23:53] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:28 [I 2022-10-09 17:23:53] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00312 [I 2022-10-09 17:23:53] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-09 17:23:53] ...optimizers/trijoint.py.151: total_norm: 16.002735 [I 2022-10-09 17:24:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1580/2304 [I 2022-10-09 17:24:01] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:14 [I 2022-10-09 17:24:01] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00313 [I 2022-10-09 17:24:01] ...trap/engines/engine.py.230: loss: 0.15886 [I 2022-10-09 17:24:01] ...optimizers/trijoint.py.151: total_norm: 16.291006 [I 2022-10-09 17:24:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1590/2304 [I 2022-10-09 17:24:10] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:07 [I 2022-10-09 17:24:10] ...trap/engines/engine.py.229: process: 0.84003 | load: 0.00313 [I 2022-10-09 17:24:10] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-09 17:24:10] ...optimizers/trijoint.py.151: total_norm: 16.652174 [I 2022-10-09 17:24:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 1600/2304 [I 2022-10-09 17:24:18] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:00 [I 2022-10-09 17:24:18] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00314 [I 2022-10-09 17:24:18] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-09 17:24:18] ...optimizers/trijoint.py.151: total_norm: 36.630749 [I 2022-10-09 17:24:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 1610/2304 [I 2022-10-09 17:24:27] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:09:52 [I 2022-10-09 17:24:27] ...trap/engines/engine.py.229: process: 0.86354 | load: 0.00313 [I 2022-10-09 17:24:27] ...trap/engines/engine.py.230: loss: 0.15726 [I 2022-10-09 17:24:27] ...optimizers/trijoint.py.151: total_norm: 15.890166 [I 2022-10-09 17:24:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 1620/2304 [I 2022-10-09 17:24:35] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:09:43 [I 2022-10-09 17:24:35] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00311 [I 2022-10-09 17:24:35] ...trap/engines/engine.py.230: loss: 0.16270 [I 2022-10-09 17:24:35] ...optimizers/trijoint.py.151: total_norm: 12.026459 [I 2022-10-09 17:24:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 1630/2304 [I 2022-10-09 17:24:44] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:35 [I 2022-10-09 17:24:44] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00312 [I 2022-10-09 17:24:44] ...trap/engines/engine.py.230: loss: 0.12628 [I 2022-10-09 17:24:44] ...optimizers/trijoint.py.151: total_norm: 11.726897 [I 2022-10-09 17:24:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 1640/2304 [I 2022-10-09 17:24:52] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:23 [I 2022-10-09 17:24:52] ...trap/engines/engine.py.229: process: 0.83789 | load: 0.00309 [I 2022-10-09 17:24:52] ...trap/engines/engine.py.230: loss: 0.18972 [I 2022-10-09 17:24:52] ...optimizers/trijoint.py.151: total_norm: 37.341564 [I 2022-10-09 17:25:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1650/2304 [I 2022-10-09 17:25:01] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:13 [I 2022-10-09 17:25:01] ...trap/engines/engine.py.229: process: 0.85312 | load: 0.00301 [I 2022-10-09 17:25:01] ...trap/engines/engine.py.230: loss: 0.17107 [I 2022-10-09 17:25:01] ...optimizers/trijoint.py.151: total_norm: 9.306121 [I 2022-10-09 17:25:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 1660/2304 [I 2022-10-09 17:25:09] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:08 [I 2022-10-09 17:25:09] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00314 [I 2022-10-09 17:25:09] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-09 17:25:09] ...optimizers/trijoint.py.151: total_norm: 17.132421 [I 2022-10-09 17:25:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 1670/2304 [I 2022-10-09 17:25:18] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:09:01 [I 2022-10-09 17:25:18] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00311 [I 2022-10-09 17:25:18] ...trap/engines/engine.py.230: loss: 0.14322 [I 2022-10-09 17:25:18] ...optimizers/trijoint.py.151: total_norm: 11.416053 [I 2022-10-09 17:25:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 1680/2304 [I 2022-10-09 17:25:26] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:08:52 [I 2022-10-09 17:25:26] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00310 [I 2022-10-09 17:25:26] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-09 17:25:26] ...optimizers/trijoint.py.151: total_norm: 25.055368 [I 2022-10-09 17:25:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 1690/2304 [I 2022-10-09 17:25:35] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:08:41 [I 2022-10-09 17:25:35] ...trap/engines/engine.py.229: process: 0.85441 | load: 0.00309 [I 2022-10-09 17:25:35] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-09 17:25:35] ...optimizers/trijoint.py.151: total_norm: 15.012591 [I 2022-10-09 17:25:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 1700/2304 [I 2022-10-09 17:25:43] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:31 [I 2022-10-09 17:25:43] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00310 [I 2022-10-09 17:25:43] ...trap/engines/engine.py.230: loss: 0.14468 [I 2022-10-09 17:25:43] ...optimizers/trijoint.py.151: total_norm: 12.767358 [I 2022-10-09 17:25:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 1710/2304 [I 2022-10-09 17:25:52] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:23 [I 2022-10-09 17:25:52] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00301 [I 2022-10-09 17:25:52] ...trap/engines/engine.py.230: loss: 0.14956 [I 2022-10-09 17:25:52] ...optimizers/trijoint.py.151: total_norm: 17.186865 [I 2022-10-09 17:26:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 1720/2304 [I 2022-10-09 17:26:00] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:15 [I 2022-10-09 17:26:00] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00298 [I 2022-10-09 17:26:00] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-09 17:26:00] ...optimizers/trijoint.py.151: total_norm: 15.731613 [I 2022-10-09 17:26:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 1730/2304 [I 2022-10-09 17:26:09] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:08:07 [I 2022-10-09 17:26:09] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00300 [I 2022-10-09 17:26:09] ...trap/engines/engine.py.230: loss: 0.12163 [I 2022-10-09 17:26:09] ...optimizers/trijoint.py.151: total_norm: 9.938307 [I 2022-10-09 17:26:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 1740/2304 [I 2022-10-09 17:26:17] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:07:55 [I 2022-10-09 17:26:17] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00291 [I 2022-10-09 17:26:17] ...trap/engines/engine.py.230: loss: 0.15625 [I 2022-10-09 17:26:17] ...optimizers/trijoint.py.151: total_norm: 19.741928 [I 2022-10-09 17:26:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 1750/2304 [I 2022-10-09 17:26:26] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:07:49 [I 2022-10-09 17:26:26] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00294 [I 2022-10-09 17:26:26] ...trap/engines/engine.py.230: loss: 0.10534 [I 2022-10-09 17:26:26] ...optimizers/trijoint.py.151: total_norm: 23.428610 [I 2022-10-09 17:26:34] ...trap/engines/engine.py.224: train: epoch 48 | batch 1760/2304 [I 2022-10-09 17:26:34] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:38 [I 2022-10-09 17:26:34] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00296 [I 2022-10-09 17:26:34] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-09 17:26:34] ...optimizers/trijoint.py.151: total_norm: 30.315084 [I 2022-10-09 17:26:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 1770/2304 [I 2022-10-09 17:26:43] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:07:31 [I 2022-10-09 17:26:43] ...trap/engines/engine.py.229: process: 0.85872 | load: 0.00294 [I 2022-10-09 17:26:43] ...trap/engines/engine.py.230: loss: 0.16918 [I 2022-10-09 17:26:43] ...optimizers/trijoint.py.151: total_norm: 20.869078 [I 2022-10-09 17:26:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 1780/2304 [I 2022-10-09 17:26:51] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:22 [I 2022-10-09 17:26:51] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00298 [I 2022-10-09 17:26:51] ...trap/engines/engine.py.230: loss: 0.10213 [I 2022-10-09 17:26:51] ...optimizers/trijoint.py.151: total_norm: 10.588058 [I 2022-10-09 17:27:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 1790/2304 [I 2022-10-09 17:27:00] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:15 [I 2022-10-09 17:27:00] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00298 [I 2022-10-09 17:27:00] ...trap/engines/engine.py.230: loss: 0.15374 [I 2022-10-09 17:27:00] ...optimizers/trijoint.py.151: total_norm: 11.387277 [I 2022-10-09 17:27:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 1800/2304 [I 2022-10-09 17:27:08] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:06 [I 2022-10-09 17:27:08] ...trap/engines/engine.py.229: process: 0.84500 | load: 0.00299 [I 2022-10-09 17:27:08] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-09 17:27:08] ...optimizers/trijoint.py.151: total_norm: 29.082939 [I 2022-10-09 17:27:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 1810/2304 [I 2022-10-09 17:27:17] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:06:55 [I 2022-10-09 17:27:17] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00301 [I 2022-10-09 17:27:17] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-09 17:27:17] ...optimizers/trijoint.py.151: total_norm: 15.916138 [I 2022-10-09 17:27:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1820/2304 [I 2022-10-09 17:27:25] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:06:46 [I 2022-10-09 17:27:25] ...trap/engines/engine.py.229: process: 0.82939 | load: 0.00304 [I 2022-10-09 17:27:25] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-09 17:27:25] ...optimizers/trijoint.py.151: total_norm: 23.321217 [I 2022-10-09 17:27:34] ...trap/engines/engine.py.224: train: epoch 48 | batch 1830/2304 [I 2022-10-09 17:27:34] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:06:44 [I 2022-10-09 17:27:34] ...trap/engines/engine.py.229: process: 0.87771 | load: 0.00302 [I 2022-10-09 17:27:34] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-09 17:27:34] ...optimizers/trijoint.py.151: total_norm: 25.948303 [I 2022-10-09 17:27:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 1840/2304 [I 2022-10-09 17:27:42] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:06:33 [I 2022-10-09 17:27:42] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00298 [I 2022-10-09 17:27:42] ...trap/engines/engine.py.230: loss: 0.14314 [I 2022-10-09 17:27:42] ...optimizers/trijoint.py.151: total_norm: 24.512638 [I 2022-10-09 17:27:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 1850/2304 [I 2022-10-09 17:27:51] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:26 [I 2022-10-09 17:27:51] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00299 [I 2022-10-09 17:27:51] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-09 17:27:51] ...optimizers/trijoint.py.151: total_norm: 15.170985 [I 2022-10-09 17:27:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 1860/2304 [I 2022-10-09 17:27:59] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:17 [I 2022-10-09 17:27:59] ...trap/engines/engine.py.229: process: 0.85250 | load: 0.00323 [I 2022-10-09 17:27:59] ...trap/engines/engine.py.230: loss: 0.22798 [I 2022-10-09 17:27:59] ...optimizers/trijoint.py.151: total_norm: 11.078460 [I 2022-10-09 17:28:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 1870/2304 [I 2022-10-09 17:28:07] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:06 [I 2022-10-09 17:28:07] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00299 [I 2022-10-09 17:28:07] ...trap/engines/engine.py.230: loss: 0.16146 [I 2022-10-09 17:28:07] ...optimizers/trijoint.py.151: total_norm: 14.629210 [I 2022-10-09 17:28:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 1880/2304 [I 2022-10-09 17:28:16] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:01 [I 2022-10-09 17:28:16] ...trap/engines/engine.py.229: process: 0.85214 | load: 0.00297 [I 2022-10-09 17:28:16] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-09 17:28:16] ...optimizers/trijoint.py.151: total_norm: 9.687495 [I 2022-10-09 17:28:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1890/2304 [I 2022-10-09 17:28:25] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:05:52 [I 2022-10-09 17:28:25] ...trap/engines/engine.py.229: process: 0.85092 | load: 0.00298 [I 2022-10-09 17:28:25] ...trap/engines/engine.py.230: loss: 0.22531 [I 2022-10-09 17:28:25] ...optimizers/trijoint.py.151: total_norm: 17.551128 [I 2022-10-09 17:28:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 1900/2304 [I 2022-10-09 17:28:33] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:46 [I 2022-10-09 17:28:33] ...trap/engines/engine.py.229: process: 0.86423 | load: 0.00313 [I 2022-10-09 17:28:33] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-09 17:28:33] ...optimizers/trijoint.py.151: total_norm: 16.642267 [I 2022-10-09 17:28:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 1910/2304 [I 2022-10-09 17:28:42] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:05:34 [I 2022-10-09 17:28:42] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00324 [I 2022-10-09 17:28:42] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-09 17:28:42] ...optimizers/trijoint.py.151: total_norm: 15.111817 [I 2022-10-09 17:28:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 1920/2304 [I 2022-10-09 17:28:50] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:27 [I 2022-10-09 17:28:50] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00314 [I 2022-10-09 17:28:50] ...trap/engines/engine.py.230: loss: 0.17630 [I 2022-10-09 17:28:50] ...optimizers/trijoint.py.151: total_norm: 13.850254 [I 2022-10-09 17:28:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 1930/2304 [I 2022-10-09 17:28:59] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:19 [I 2022-10-09 17:28:59] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00312 [I 2022-10-09 17:28:59] ...trap/engines/engine.py.230: loss: 0.13431 [I 2022-10-09 17:28:59] ...optimizers/trijoint.py.151: total_norm: 19.989019 [I 2022-10-09 17:29:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 1940/2304 [I 2022-10-09 17:29:07] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:14 [I 2022-10-09 17:29:07] ...trap/engines/engine.py.229: process: 0.86466 | load: 0.00316 [I 2022-10-09 17:29:07] ...trap/engines/engine.py.230: loss: 0.13724 [I 2022-10-09 17:29:07] ...optimizers/trijoint.py.151: total_norm: 28.982590 [I 2022-10-09 17:29:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 1950/2304 [I 2022-10-09 17:29:16] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:04 [I 2022-10-09 17:29:16] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00314 [I 2022-10-09 17:29:16] ...trap/engines/engine.py.230: loss: 0.14718 [I 2022-10-09 17:29:16] ...optimizers/trijoint.py.151: total_norm: 8.734509 [I 2022-10-09 17:29:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1960/2304 [I 2022-10-09 17:29:25] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:56 [I 2022-10-09 17:29:25] ...trap/engines/engine.py.229: process: 0.87237 | load: 0.00313 [I 2022-10-09 17:29:25] ...trap/engines/engine.py.230: loss: 0.19195 [I 2022-10-09 17:29:25] ...optimizers/trijoint.py.151: total_norm: 19.663872 [I 2022-10-09 17:29:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 1970/2304 [I 2022-10-09 17:29:33] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:44 [I 2022-10-09 17:29:33] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00311 [I 2022-10-09 17:29:33] ...trap/engines/engine.py.230: loss: 0.09771 [I 2022-10-09 17:29:33] ...optimizers/trijoint.py.151: total_norm: 15.826103 [I 2022-10-09 17:29:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 1980/2304 [I 2022-10-09 17:29:42] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:36 [I 2022-10-09 17:29:42] ...trap/engines/engine.py.229: process: 0.86047 | load: 0.00312 [I 2022-10-09 17:29:42] ...trap/engines/engine.py.230: loss: 0.22748 [I 2022-10-09 17:29:42] ...optimizers/trijoint.py.151: total_norm: 26.847349 [I 2022-10-09 17:29:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 1990/2304 [I 2022-10-09 17:29:50] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:26 [I 2022-10-09 17:29:50] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00319 [I 2022-10-09 17:29:50] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-09 17:29:50] ...optimizers/trijoint.py.151: total_norm: 12.244564 [I 2022-10-09 17:29:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 2000/2304 [I 2022-10-09 17:29:59] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:19 [I 2022-10-09 17:29:59] ...trap/engines/engine.py.229: process: 0.85117 | load: 0.00315 [I 2022-10-09 17:29:59] ...trap/engines/engine.py.230: loss: 0.15044 [I 2022-10-09 17:29:59] ...optimizers/trijoint.py.151: total_norm: 12.093333 [I 2022-10-09 17:30:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 2010/2304 [I 2022-10-09 17:30:07] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:12 [I 2022-10-09 17:30:07] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00313 [I 2022-10-09 17:30:07] ...trap/engines/engine.py.230: loss: 0.21249 [I 2022-10-09 17:30:07] ...optimizers/trijoint.py.151: total_norm: 15.586141 [I 2022-10-09 17:30:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 2020/2304 [I 2022-10-09 17:30:16] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:02 [I 2022-10-09 17:30:16] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00314 [I 2022-10-09 17:30:16] ...trap/engines/engine.py.230: loss: 0.18160 [I 2022-10-09 17:30:16] ...optimizers/trijoint.py.151: total_norm: 20.261229 [I 2022-10-09 17:30:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 2030/2304 [I 2022-10-09 17:30:25] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:03:55 [I 2022-10-09 17:30:25] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00315 [I 2022-10-09 17:30:25] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-09 17:30:25] ...optimizers/trijoint.py.151: total_norm: 28.661749 [I 2022-10-09 17:30:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 2040/2304 [I 2022-10-09 17:30:33] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:45 [I 2022-10-09 17:30:33] ...trap/engines/engine.py.229: process: 0.85736 | load: 0.00315 [I 2022-10-09 17:30:33] ...trap/engines/engine.py.230: loss: 0.18582 [I 2022-10-09 17:30:33] ...optimizers/trijoint.py.151: total_norm: 16.639698 [I 2022-10-09 17:30:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 2050/2304 [I 2022-10-09 17:30:42] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:38 [I 2022-10-09 17:30:42] ...trap/engines/engine.py.229: process: 0.85870 | load: 0.00312 [I 2022-10-09 17:30:42] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-09 17:30:42] ...optimizers/trijoint.py.151: total_norm: 15.141867 [I 2022-10-09 17:30:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 2060/2304 [I 2022-10-09 17:30:50] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:27 [I 2022-10-09 17:30:50] ...trap/engines/engine.py.229: process: 0.83517 | load: 0.00321 [I 2022-10-09 17:30:50] ...trap/engines/engine.py.230: loss: 0.18607 [I 2022-10-09 17:30:50] ...optimizers/trijoint.py.151: total_norm: 79.632942 [I 2022-10-09 17:30:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 2070/2304 [I 2022-10-09 17:30:59] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:18 [I 2022-10-09 17:30:59] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00316 [I 2022-10-09 17:30:59] ...trap/engines/engine.py.230: loss: 0.15578 [I 2022-10-09 17:30:59] ...optimizers/trijoint.py.151: total_norm: 11.829250 [I 2022-10-09 17:31:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 2080/2304 [I 2022-10-09 17:31:07] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:11 [I 2022-10-09 17:31:07] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00314 [I 2022-10-09 17:31:07] ...trap/engines/engine.py.230: loss: 0.10856 [I 2022-10-09 17:31:07] ...optimizers/trijoint.py.151: total_norm: 6.726292 [I 2022-10-09 17:31:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 2090/2304 [I 2022-10-09 17:31:16] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:03 [I 2022-10-09 17:31:16] ...trap/engines/engine.py.229: process: 0.87395 | load: 0.00315 [I 2022-10-09 17:31:16] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-09 17:31:16] ...optimizers/trijoint.py.151: total_norm: 38.021130 [I 2022-10-09 17:31:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 2100/2304 [I 2022-10-09 17:31:24] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:54 [I 2022-10-09 17:31:24] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00317 [I 2022-10-09 17:31:24] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-09 17:31:24] ...optimizers/trijoint.py.151: total_norm: 10.988256 [I 2022-10-09 17:31:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 2110/2304 [I 2022-10-09 17:31:33] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:45 [I 2022-10-09 17:31:33] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00312 [I 2022-10-09 17:31:33] ...trap/engines/engine.py.230: loss: 0.09875 [I 2022-10-09 17:31:33] ...optimizers/trijoint.py.151: total_norm: 7.056009 [I 2022-10-09 17:31:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 2120/2304 [I 2022-10-09 17:31:42] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:38 [I 2022-10-09 17:31:42] ...trap/engines/engine.py.229: process: 0.85695 | load: 0.00314 [I 2022-10-09 17:31:42] ...trap/engines/engine.py.230: loss: 0.16747 [I 2022-10-09 17:31:42] ...optimizers/trijoint.py.151: total_norm: 16.490746 [I 2022-10-09 17:31:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 2130/2304 [I 2022-10-09 17:31:50] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:27 [I 2022-10-09 17:31:50] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00312 [I 2022-10-09 17:31:50] ...trap/engines/engine.py.230: loss: 0.16633 [I 2022-10-09 17:31:50] ...optimizers/trijoint.py.151: total_norm: 23.502794 [I 2022-10-09 17:31:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 2140/2304 [I 2022-10-09 17:31:59] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:19 [I 2022-10-09 17:31:59] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00312 [I 2022-10-09 17:31:59] ...trap/engines/engine.py.230: loss: 0.14441 [I 2022-10-09 17:31:59] ...optimizers/trijoint.py.151: total_norm: 11.176181 [I 2022-10-09 17:32:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 2150/2304 [I 2022-10-09 17:32:07] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:11 [I 2022-10-09 17:32:07] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00313 [I 2022-10-09 17:32:07] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-09 17:32:07] ...optimizers/trijoint.py.151: total_norm: 14.261772 [I 2022-10-09 17:32:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 2160/2304 [I 2022-10-09 17:32:16] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:02 [I 2022-10-09 17:32:16] ...trap/engines/engine.py.229: process: 0.86120 | load: 0.00320 [I 2022-10-09 17:32:16] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-09 17:32:16] ...optimizers/trijoint.py.151: total_norm: 11.052973 [I 2022-10-09 17:32:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 2170/2304 [I 2022-10-09 17:32:24] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:01:54 [I 2022-10-09 17:32:24] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00313 [I 2022-10-09 17:32:24] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-09 17:32:24] ...optimizers/trijoint.py.151: total_norm: 13.329052 [I 2022-10-09 17:32:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 2180/2304 [I 2022-10-09 17:32:33] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:01:45 [I 2022-10-09 17:32:33] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00312 [I 2022-10-09 17:32:33] ...trap/engines/engine.py.230: loss: 0.13995 [I 2022-10-09 17:32:33] ...optimizers/trijoint.py.151: total_norm: 13.608978 [I 2022-10-09 17:32:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 2190/2304 [I 2022-10-09 17:32:41] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:37 [I 2022-10-09 17:32:41] ...trap/engines/engine.py.229: process: 0.84579 | load: 0.00312 [I 2022-10-09 17:32:41] ...trap/engines/engine.py.230: loss: 0.15334 [I 2022-10-09 17:32:41] ...optimizers/trijoint.py.151: total_norm: 11.139177 [I 2022-10-09 17:32:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 2200/2304 [I 2022-10-09 17:32:50] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:28 [I 2022-10-09 17:32:50] ...trap/engines/engine.py.229: process: 0.84108 | load: 0.00301 [I 2022-10-09 17:32:50] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-09 17:32:50] ...optimizers/trijoint.py.151: total_norm: 9.174039 [I 2022-10-09 17:32:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 2210/2304 [I 2022-10-09 17:32:58] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:19 [I 2022-10-09 17:32:58] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00299 [I 2022-10-09 17:32:58] ...trap/engines/engine.py.230: loss: 0.22830 [I 2022-10-09 17:32:58] ...optimizers/trijoint.py.151: total_norm: 43.066956 [I 2022-10-09 17:33:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 2220/2304 [I 2022-10-09 17:33:07] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:10 [I 2022-10-09 17:33:07] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00295 [I 2022-10-09 17:33:07] ...trap/engines/engine.py.230: loss: 0.15583 [I 2022-10-09 17:33:07] ...optimizers/trijoint.py.151: total_norm: 22.677990 [I 2022-10-09 17:33:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 2230/2304 [I 2022-10-09 17:33:15] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:03 [I 2022-10-09 17:33:15] ...trap/engines/engine.py.229: process: 0.85852 | load: 0.00299 [I 2022-10-09 17:33:15] ...trap/engines/engine.py.230: loss: 0.12043 [I 2022-10-09 17:33:15] ...optimizers/trijoint.py.151: total_norm: 11.810751 [I 2022-10-09 17:33:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 2240/2304 [I 2022-10-09 17:33:24] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:00:54 [I 2022-10-09 17:33:24] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00293 [I 2022-10-09 17:33:24] ...trap/engines/engine.py.230: loss: 0.15826 [I 2022-10-09 17:33:24] ...optimizers/trijoint.py.151: total_norm: 17.669903 [I 2022-10-09 17:33:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 2250/2304 [I 2022-10-09 17:33:32] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:00:45 [I 2022-10-09 17:33:32] ...trap/engines/engine.py.229: process: 0.83445 | load: 0.00295 [I 2022-10-09 17:33:32] ...trap/engines/engine.py.230: loss: 0.16593 [I 2022-10-09 17:33:32] ...optimizers/trijoint.py.151: total_norm: 12.214584 [I 2022-10-09 17:33:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 2260/2304 [I 2022-10-09 17:33:41] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:00:36 [I 2022-10-09 17:33:41] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00294 [I 2022-10-09 17:33:41] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-09 17:33:41] ...optimizers/trijoint.py.151: total_norm: 9.807963 [I 2022-10-09 17:33:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 2270/2304 [I 2022-10-09 17:33:49] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:28 [I 2022-10-09 17:33:49] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00298 [I 2022-10-09 17:33:49] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-09 17:33:49] ...optimizers/trijoint.py.151: total_norm: 22.192348 [I 2022-10-09 17:33:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 2280/2304 [I 2022-10-09 17:33:58] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:20 [I 2022-10-09 17:33:58] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00294 [I 2022-10-09 17:33:58] ...trap/engines/engine.py.230: loss: 0.20754 [I 2022-10-09 17:33:58] ...optimizers/trijoint.py.151: total_norm: 18.767603 [I 2022-10-09 17:34:06] ...trap/engines/engine.py.224: train: epoch 48 | batch 2290/2304 [I 2022-10-09 17:34:06] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:11 [I 2022-10-09 17:34:06] ...trap/engines/engine.py.229: process: 0.81799 | load: 0.00302 [I 2022-10-09 17:34:06] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-09 17:34:06] ...optimizers/trijoint.py.151: total_norm: 14.400485 [I 2022-10-09 17:34:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 2300/2304 [I 2022-10-09 17:34:15] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:03 [I 2022-10-09 17:34:15] ...trap/engines/engine.py.229: process: 0.83147 | load: 0.00017 [I 2022-10-09 17:34:15] ...trap/engines/engine.py.230: loss: 0.24384 [I 2022-10-09 17:34:15] ...optimizers/trijoint.py.151: total_norm: 48.993423 [I 2022-10-09 17:34:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 2304/2304 [I 2022-10-09 17:34:18] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:00 [I 2022-10-09 17:34:18] ...trap/engines/engine.py.229: process: 0.81465 | load: 0.00017 [I 2022-10-09 17:34:18] ...trap/engines/engine.py.230: loss: 0.12280 [I 2022-10-09 17:34:18] ...optimizers/trijoint.py.151: total_norm: 13.662761 [S 2022-10-09 17:34:19] ...trap/engines/engine.py.129: train_epoch.epoch: 48 [S 2022-10-09 17:34:19] ...trap/engines/engine.py.129: train_epoch.loss: 0.155302806017585 [I 2022-10-09 17:34:19] ...s/criterions/triplet.py.84: margin = 0.2900000000000001 [I 2022-10-09 17:34:20] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 48 [I 2022-10-09 17:34:20] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 17:34:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 17:34:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 17:34:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 17:34:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 17:34:24] ...trap/engines/engine.py.321: eval: epoch 48 | batch 0/510 [I 2022-10-09 17:34:24] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:31:55 [I 2022-10-09 17:34:24] ...trap/engines/engine.py.326: process: 3.75502 | load: 2.89313 [I 2022-10-09 17:34:28] ...trap/engines/engine.py.321: eval: epoch 48 | batch 10/510 [I 2022-10-09 17:34:28] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:51 [I 2022-10-09 17:34:28] ...trap/engines/engine.py.326: process: 0.41762 | load: 0.00018 [I 2022-10-09 17:34:34] ...trap/engines/engine.py.321: eval: epoch 48 | batch 20/510 [I 2022-10-09 17:34:34] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:44 [I 2022-10-09 17:34:34] ...trap/engines/engine.py.326: process: 0.60310 | load: 0.00019 [I 2022-10-09 17:34:40] ...trap/engines/engine.py.321: eval: epoch 48 | batch 30/510 [I 2022-10-09 17:34:40] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:03 [I 2022-10-09 17:34:40] ...trap/engines/engine.py.326: process: 0.71491 | load: 0.00019 [I 2022-10-09 17:34:45] ...trap/engines/engine.py.321: eval: epoch 48 | batch 40/510 [I 2022-10-09 17:34:45] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:49 [I 2022-10-09 17:34:45] ...trap/engines/engine.py.326: process: 0.43890 | load: 0.00021 [I 2022-10-09 17:34:49] ...trap/engines/engine.py.321: eval: epoch 48 | batch 50/510 [I 2022-10-09 17:34:49] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:32 [I 2022-10-09 17:34:49] ...trap/engines/engine.py.326: process: 0.50835 | load: 0.00018 [I 2022-10-09 17:34:54] ...trap/engines/engine.py.321: eval: epoch 48 | batch 60/510 [I 2022-10-09 17:34:54] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:08 [I 2022-10-09 17:34:54] ...trap/engines/engine.py.326: process: 0.40230 | load: 0.00018 [I 2022-10-09 17:34:58] ...trap/engines/engine.py.321: eval: epoch 48 | batch 70/510 [I 2022-10-09 17:34:58] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:30 [I 2022-10-09 17:34:58] ...trap/engines/engine.py.326: process: 0.50611 | load: 0.00018 [I 2022-10-09 17:35:03] ...trap/engines/engine.py.321: eval: epoch 48 | batch 80/510 [I 2022-10-09 17:35:03] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:11 [I 2022-10-09 17:35:03] ...trap/engines/engine.py.326: process: 0.46928 | load: 0.00020 [I 2022-10-09 17:35:07] ...trap/engines/engine.py.321: eval: epoch 48 | batch 90/510 [I 2022-10-09 17:35:07] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:07 [I 2022-10-09 17:35:07] ...trap/engines/engine.py.326: process: 0.46423 | load: 0.00018 [I 2022-10-09 17:35:12] ...trap/engines/engine.py.321: eval: epoch 48 | batch 100/510 [I 2022-10-09 17:35:12] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:02:49 [I 2022-10-09 17:35:12] ...trap/engines/engine.py.326: process: 0.31369 | load: 0.00018 [I 2022-10-09 17:35:15] ...trap/engines/engine.py.321: eval: epoch 48 | batch 110/510 [I 2022-10-09 17:35:15] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:10 [I 2022-10-09 17:35:15] ...trap/engines/engine.py.326: process: 0.31539 | load: 0.00018 [I 2022-10-09 17:35:18] ...trap/engines/engine.py.321: eval: epoch 48 | batch 120/510 [I 2022-10-09 17:35:18] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:03 [I 2022-10-09 17:35:18] ...trap/engines/engine.py.326: process: 0.31581 | load: 0.00019 [I 2022-10-09 17:35:21] ...trap/engines/engine.py.321: eval: epoch 48 | batch 130/510 [I 2022-10-09 17:35:21] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:59 [I 2022-10-09 17:35:21] ...trap/engines/engine.py.326: process: 0.31090 | load: 0.00020 [I 2022-10-09 17:35:24] ...trap/engines/engine.py.321: eval: epoch 48 | batch 140/510 [I 2022-10-09 17:35:24] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:57 [I 2022-10-09 17:35:24] ...trap/engines/engine.py.326: process: 0.31594 | load: 0.00020 [I 2022-10-09 17:35:27] ...trap/engines/engine.py.321: eval: epoch 48 | batch 150/510 [I 2022-10-09 17:35:27] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:54 [I 2022-10-09 17:35:27] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00021 [I 2022-10-09 17:35:31] ...trap/engines/engine.py.321: eval: epoch 48 | batch 160/510 [I 2022-10-09 17:35:31] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:51 [I 2022-10-09 17:35:31] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00019 [I 2022-10-09 17:35:34] ...trap/engines/engine.py.321: eval: epoch 48 | batch 170/510 [I 2022-10-09 17:35:34] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:49 [I 2022-10-09 17:35:34] ...trap/engines/engine.py.326: process: 0.32261 | load: 0.00019 [I 2022-10-09 17:35:37] ...trap/engines/engine.py.321: eval: epoch 48 | batch 180/510 [I 2022-10-09 17:35:37] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:45 [I 2022-10-09 17:35:37] ...trap/engines/engine.py.326: process: 0.32037 | load: 0.00020 [I 2022-10-09 17:35:40] ...trap/engines/engine.py.321: eval: epoch 48 | batch 190/510 [I 2022-10-09 17:35:40] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:41 [I 2022-10-09 17:35:40] ...trap/engines/engine.py.326: process: 0.30938 | load: 0.00019 [I 2022-10-09 17:35:43] ...trap/engines/engine.py.321: eval: epoch 48 | batch 200/510 [I 2022-10-09 17:35:43] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:38 [I 2022-10-09 17:35:43] ...trap/engines/engine.py.326: process: 0.32026 | load: 0.00020 [I 2022-10-09 17:35:47] ...trap/engines/engine.py.321: eval: epoch 48 | batch 210/510 [I 2022-10-09 17:35:47] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:34 [I 2022-10-09 17:35:47] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-09 17:35:50] ...trap/engines/engine.py.321: eval: epoch 48 | batch 220/510 [I 2022-10-09 17:35:50] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:32 [I 2022-10-09 17:35:50] ...trap/engines/engine.py.326: process: 0.32771 | load: 0.00020 [I 2022-10-09 17:35:53] ...trap/engines/engine.py.321: eval: epoch 48 | batch 230/510 [I 2022-10-09 17:35:53] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-09 17:35:53] ...trap/engines/engine.py.326: process: 0.30845 | load: 0.00019 [I 2022-10-09 17:35:56] ...trap/engines/engine.py.321: eval: epoch 48 | batch 240/510 [I 2022-10-09 17:35:56] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:26 [I 2022-10-09 17:35:56] ...trap/engines/engine.py.326: process: 0.32082 | load: 0.00019 [I 2022-10-09 17:35:59] ...trap/engines/engine.py.321: eval: epoch 48 | batch 250/510 [I 2022-10-09 17:35:59] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-09 17:35:59] ...trap/engines/engine.py.326: process: 0.31218 | load: 0.00019 [I 2022-10-09 17:36:02] ...trap/engines/engine.py.321: eval: epoch 48 | batch 260/510 [I 2022-10-09 17:36:02] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-09 17:36:02] ...trap/engines/engine.py.326: process: 0.32182 | load: 0.00020 [I 2022-10-09 17:36:06] ...trap/engines/engine.py.321: eval: epoch 48 | batch 270/510 [I 2022-10-09 17:36:06] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:16 [I 2022-10-09 17:36:06] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00020 [I 2022-10-09 17:36:09] ...trap/engines/engine.py.321: eval: epoch 48 | batch 280/510 [I 2022-10-09 17:36:09] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-09 17:36:09] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00021 [I 2022-10-09 17:36:12] ...trap/engines/engine.py.321: eval: epoch 48 | batch 290/510 [I 2022-10-09 17:36:12] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:09 [I 2022-10-09 17:36:12] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00020 [I 2022-10-09 17:36:15] ...trap/engines/engine.py.321: eval: epoch 48 | batch 300/510 [I 2022-10-09 17:36:15] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-09 17:36:15] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00020 [I 2022-10-09 17:36:18] ...trap/engines/engine.py.321: eval: epoch 48 | batch 310/510 [I 2022-10-09 17:36:18] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-09 17:36:18] ...trap/engines/engine.py.326: process: 0.31429 | load: 0.00019 [I 2022-10-09 17:36:22] ...trap/engines/engine.py.321: eval: epoch 48 | batch 320/510 [I 2022-10-09 17:36:22] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-09 17:36:22] ...trap/engines/engine.py.326: process: 0.32299 | load: 0.00020 [I 2022-10-09 17:36:25] ...trap/engines/engine.py.321: eval: epoch 48 | batch 330/510 [I 2022-10-09 17:36:25] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:57 [I 2022-10-09 17:36:25] ...trap/engines/engine.py.326: process: 0.31736 | load: 0.00020 [I 2022-10-09 17:36:28] ...trap/engines/engine.py.321: eval: epoch 48 | batch 340/510 [I 2022-10-09 17:36:28] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-09 17:36:28] ...trap/engines/engine.py.326: process: 0.31307 | load: 0.00019 [I 2022-10-09 17:36:31] ...trap/engines/engine.py.321: eval: epoch 48 | batch 350/510 [I 2022-10-09 17:36:31] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:51 [I 2022-10-09 17:36:31] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00020 [I 2022-10-09 17:36:34] ...trap/engines/engine.py.321: eval: epoch 48 | batch 360/510 [I 2022-10-09 17:36:34] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-09 17:36:34] ...trap/engines/engine.py.326: process: 0.31898 | load: 0.00020 [I 2022-10-09 17:36:38] ...trap/engines/engine.py.321: eval: epoch 48 | batch 370/510 [I 2022-10-09 17:36:38] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:44 [I 2022-10-09 17:36:38] ...trap/engines/engine.py.326: process: 0.31065 | load: 0.00020 [I 2022-10-09 17:36:41] ...trap/engines/engine.py.321: eval: epoch 48 | batch 380/510 [I 2022-10-09 17:36:41] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-09 17:36:41] ...trap/engines/engine.py.326: process: 0.32421 | load: 0.00020 [I 2022-10-09 17:36:44] ...trap/engines/engine.py.321: eval: epoch 48 | batch 390/510 [I 2022-10-09 17:36:44] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-09 17:36:44] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00021 [I 2022-10-09 17:36:47] ...trap/engines/engine.py.321: eval: epoch 48 | batch 400/510 [I 2022-10-09 17:36:47] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:35 [I 2022-10-09 17:36:47] ...trap/engines/engine.py.326: process: 0.31632 | load: 0.00021 [I 2022-10-09 17:36:51] ...trap/engines/engine.py.321: eval: epoch 48 | batch 410/510 [I 2022-10-09 17:36:51] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-09 17:36:51] ...trap/engines/engine.py.326: process: 0.32226 | load: 0.00020 [I 2022-10-09 17:36:54] ...trap/engines/engine.py.321: eval: epoch 48 | batch 420/510 [I 2022-10-09 17:36:54] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-09 17:36:54] ...trap/engines/engine.py.326: process: 0.32039 | load: 0.00019 [I 2022-10-09 17:36:57] ...trap/engines/engine.py.321: eval: epoch 48 | batch 430/510 [I 2022-10-09 17:36:57] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-09 17:36:57] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00021 [I 2022-10-09 17:37:00] ...trap/engines/engine.py.321: eval: epoch 48 | batch 440/510 [I 2022-10-09 17:37:00] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-09 17:37:00] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00020 [I 2022-10-09 17:37:03] ...trap/engines/engine.py.321: eval: epoch 48 | batch 450/510 [I 2022-10-09 17:37:03] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-09 17:37:03] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00020 [I 2022-10-09 17:37:06] ...trap/engines/engine.py.321: eval: epoch 48 | batch 460/510 [I 2022-10-09 17:37:06] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-09 17:37:06] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00020 [I 2022-10-09 17:37:10] ...trap/engines/engine.py.321: eval: epoch 48 | batch 470/510 [I 2022-10-09 17:37:10] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-09 17:37:10] ...trap/engines/engine.py.326: process: 0.31817 | load: 0.00020 [I 2022-10-09 17:37:13] ...trap/engines/engine.py.321: eval: epoch 48 | batch 480/510 [I 2022-10-09 17:37:13] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-09 17:37:13] ...trap/engines/engine.py.326: process: 0.32031 | load: 0.00021 [I 2022-10-09 17:37:16] ...trap/engines/engine.py.321: eval: epoch 48 | batch 490/510 [I 2022-10-09 17:37:16] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-09 17:37:16] ...trap/engines/engine.py.326: process: 0.31584 | load: 0.00020 [I 2022-10-09 17:37:19] ...trap/engines/engine.py.321: eval: epoch 48 | batch 500/510 [I 2022-10-09 17:37:19] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-09 17:37:19] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00020 [I 2022-10-09 17:37:22] ...trap/engines/engine.py.321: eval: epoch 48 | batch 510/510 [I 2022-10-09 17:37:22] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-09 17:37:22] ...trap/engines/engine.py.326: process: 0.31915 | load: 0.00015 [S 2022-10-09 17:37:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 48 [I 2022-10-09 17:37:23] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.725200 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908200 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939200 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.715800 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906300 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936600 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012392 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008035 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006478 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014344 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008798 [S 2022-10-09 17:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007228 [I 2022-10-09 17:38:00] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 17:38:00] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 17:38:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 17:38:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 17:38:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 17:38:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 17:38:04] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 17:38:10] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 17:38:10] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 17:38:10] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 17:38:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 17:38:20] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 17:38:20] ...trap/engines/engine.py.163: Training model on trainset for epoch 49 [I 2022-10-09 17:38:20] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 17:38:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 0/2304 [I 2022-10-09 17:38:24] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:18:15 [I 2022-10-09 17:38:24] ...trap/engines/engine.py.229: process: 3.60067 | load: 2.69543 [I 2022-10-09 17:38:24] ...trap/engines/engine.py.230: loss: 0.24419 [I 2022-10-09 17:38:24] ...optimizers/trijoint.py.151: total_norm: 19.093451 [I 2022-10-09 17:38:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 10/2304 [I 2022-10-09 17:38:32] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:37 [I 2022-10-09 17:38:32] ...trap/engines/engine.py.229: process: 0.83363 | load: 0.00512 [I 2022-10-09 17:38:32] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-09 17:38:32] ...optimizers/trijoint.py.151: total_norm: 12.689178 [I 2022-10-09 17:38:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 20/2304 [I 2022-10-09 17:38:41] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:29 [I 2022-10-09 17:38:41] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00436 [I 2022-10-09 17:38:41] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-09 17:38:41] ...optimizers/trijoint.py.151: total_norm: 12.175850 [I 2022-10-09 17:38:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 30/2304 [I 2022-10-09 17:38:49] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:19 [I 2022-10-09 17:38:49] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00365 [I 2022-10-09 17:38:49] ...trap/engines/engine.py.230: loss: 0.16626 [I 2022-10-09 17:38:49] ...optimizers/trijoint.py.151: total_norm: 15.928393 [I 2022-10-09 17:38:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 40/2304 [I 2022-10-09 17:38:58] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:58 [I 2022-10-09 17:38:58] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00369 [I 2022-10-09 17:38:58] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-09 17:38:58] ...optimizers/trijoint.py.151: total_norm: 16.906542 [I 2022-10-09 17:39:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 50/2304 [I 2022-10-09 17:39:06] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:31:53 [I 2022-10-09 17:39:06] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00333 [I 2022-10-09 17:39:06] ...trap/engines/engine.py.230: loss: 0.20236 [I 2022-10-09 17:39:06] ...optimizers/trijoint.py.151: total_norm: 29.231825 [I 2022-10-09 17:39:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 60/2304 [I 2022-10-09 17:39:15] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:49 [I 2022-10-09 17:39:15] ...trap/engines/engine.py.229: process: 0.87327 | load: 0.00371 [I 2022-10-09 17:39:15] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-09 17:39:15] ...optimizers/trijoint.py.151: total_norm: 9.728768 [I 2022-10-09 17:39:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 70/2304 [I 2022-10-09 17:39:23] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:46 [I 2022-10-09 17:39:23] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00338 [I 2022-10-09 17:39:23] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-09 17:39:23] ...optimizers/trijoint.py.151: total_norm: 14.677158 [I 2022-10-09 17:39:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 80/2304 [I 2022-10-09 17:39:32] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:39 [I 2022-10-09 17:39:32] ...trap/engines/engine.py.229: process: 0.86634 | load: 0.00323 [I 2022-10-09 17:39:32] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-09 17:39:32] ...optimizers/trijoint.py.151: total_norm: 16.337866 [I 2022-10-09 17:39:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 90/2304 [I 2022-10-09 17:39:40] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:26 [I 2022-10-09 17:39:40] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00340 [I 2022-10-09 17:39:40] ...trap/engines/engine.py.230: loss: 0.22156 [I 2022-10-09 17:39:40] ...optimizers/trijoint.py.151: total_norm: 28.387724 [I 2022-10-09 17:39:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 100/2304 [I 2022-10-09 17:39:49] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:18 [I 2022-10-09 17:39:49] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00317 [I 2022-10-09 17:39:49] ...trap/engines/engine.py.230: loss: 0.14861 [I 2022-10-09 17:39:49] ...optimizers/trijoint.py.151: total_norm: 54.691738 [I 2022-10-09 17:39:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 110/2304 [I 2022-10-09 17:39:57] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:06 [I 2022-10-09 17:39:57] ...trap/engines/engine.py.229: process: 0.85250 | load: 0.00349 [I 2022-10-09 17:39:57] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-09 17:39:57] ...optimizers/trijoint.py.151: total_norm: 16.048117 [I 2022-10-09 17:40:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 120/2304 [I 2022-10-09 17:40:06] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:49 [I 2022-10-09 17:40:06] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00346 [I 2022-10-09 17:40:06] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-09 17:40:06] ...optimizers/trijoint.py.151: total_norm: 20.286890 [I 2022-10-09 17:40:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 130/2304 [I 2022-10-09 17:40:15] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:59 [I 2022-10-09 17:40:15] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00325 [I 2022-10-09 17:40:15] ...trap/engines/engine.py.230: loss: 0.18309 [I 2022-10-09 17:40:15] ...optimizers/trijoint.py.151: total_norm: 31.009388 [I 2022-10-09 17:40:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 140/2304 [I 2022-10-09 17:40:23] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:50 [I 2022-10-09 17:40:23] ...trap/engines/engine.py.229: process: 0.87421 | load: 0.00337 [I 2022-10-09 17:40:23] ...trap/engines/engine.py.230: loss: 0.11657 [I 2022-10-09 17:40:23] ...optimizers/trijoint.py.151: total_norm: 28.595985 [I 2022-10-09 17:40:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 150/2304 [I 2022-10-09 17:40:32] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:49 [I 2022-10-09 17:40:32] ...trap/engines/engine.py.229: process: 0.85865 | load: 0.00359 [I 2022-10-09 17:40:32] ...trap/engines/engine.py.230: loss: 0.20868 [I 2022-10-09 17:40:32] ...optimizers/trijoint.py.151: total_norm: 26.775175 [I 2022-10-09 17:40:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 160/2304 [I 2022-10-09 17:40:40] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:41 [I 2022-10-09 17:40:40] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00310 [I 2022-10-09 17:40:40] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-09 17:40:40] ...optimizers/trijoint.py.151: total_norm: 23.137251 [I 2022-10-09 17:40:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 170/2304 [I 2022-10-09 17:40:49] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:18 [I 2022-10-09 17:40:49] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00347 [I 2022-10-09 17:40:49] ...trap/engines/engine.py.230: loss: 0.23071 [I 2022-10-09 17:40:49] ...optimizers/trijoint.py.151: total_norm: 22.542206 [I 2022-10-09 17:40:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 180/2304 [I 2022-10-09 17:40:57] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:18 [I 2022-10-09 17:40:57] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00336 [I 2022-10-09 17:40:57] ...trap/engines/engine.py.230: loss: 0.21451 [I 2022-10-09 17:40:57] ...optimizers/trijoint.py.151: total_norm: 24.593336 [I 2022-10-09 17:41:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 190/2304 [I 2022-10-09 17:41:06] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:01 [I 2022-10-09 17:41:06] ...trap/engines/engine.py.229: process: 0.85059 | load: 0.00349 [I 2022-10-09 17:41:06] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-09 17:41:06] ...optimizers/trijoint.py.151: total_norm: 14.073275 [I 2022-10-09 17:41:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 200/2304 [I 2022-10-09 17:41:15] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:05 [I 2022-10-09 17:41:15] ...trap/engines/engine.py.229: process: 0.85890 | load: 0.00311 [I 2022-10-09 17:41:15] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-09 17:41:15] ...optimizers/trijoint.py.151: total_norm: 18.403036 [I 2022-10-09 17:41:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 210/2304 [I 2022-10-09 17:41:23] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:52 [I 2022-10-09 17:41:23] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00311 [I 2022-10-09 17:41:23] ...trap/engines/engine.py.230: loss: 0.16783 [I 2022-10-09 17:41:23] ...optimizers/trijoint.py.151: total_norm: 23.600830 [I 2022-10-09 17:41:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 220/2304 [I 2022-10-09 17:41:32] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:46 [I 2022-10-09 17:41:32] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00326 [I 2022-10-09 17:41:32] ...trap/engines/engine.py.230: loss: 0.24506 [I 2022-10-09 17:41:32] ...optimizers/trijoint.py.151: total_norm: 26.783403 [I 2022-10-09 17:41:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 230/2304 [I 2022-10-09 17:41:40] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:49 [I 2022-10-09 17:41:40] ...trap/engines/engine.py.229: process: 0.85758 | load: 0.00351 [I 2022-10-09 17:41:40] ...trap/engines/engine.py.230: loss: 0.14753 [I 2022-10-09 17:41:40] ...optimizers/trijoint.py.151: total_norm: 10.417789 [I 2022-10-09 17:41:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 240/2304 [I 2022-10-09 17:41:49] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:21 [I 2022-10-09 17:41:49] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00324 [I 2022-10-09 17:41:49] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-09 17:41:49] ...optimizers/trijoint.py.151: total_norm: 16.658697 [I 2022-10-09 17:41:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 250/2304 [I 2022-10-09 17:41:58] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:24 [I 2022-10-09 17:41:58] ...trap/engines/engine.py.229: process: 0.86093 | load: 0.00346 [I 2022-10-09 17:41:58] ...trap/engines/engine.py.230: loss: 0.15076 [I 2022-10-09 17:41:58] ...optimizers/trijoint.py.151: total_norm: 12.076679 [I 2022-10-09 17:42:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 260/2304 [I 2022-10-09 17:42:06] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:07 [I 2022-10-09 17:42:06] ...trap/engines/engine.py.229: process: 0.86955 | load: 0.00311 [I 2022-10-09 17:42:06] ...trap/engines/engine.py.230: loss: 0.21004 [I 2022-10-09 17:42:06] ...optimizers/trijoint.py.151: total_norm: 23.125317 [I 2022-10-09 17:42:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 270/2304 [I 2022-10-09 17:42:15] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:59 [I 2022-10-09 17:42:15] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00310 [I 2022-10-09 17:42:15] ...trap/engines/engine.py.230: loss: 0.21163 [I 2022-10-09 17:42:15] ...optimizers/trijoint.py.151: total_norm: 25.101936 [I 2022-10-09 17:42:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 280/2304 [I 2022-10-09 17:42:23] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:31 [I 2022-10-09 17:42:23] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00310 [I 2022-10-09 17:42:23] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-09 17:42:23] ...optimizers/trijoint.py.151: total_norm: 17.730106 [I 2022-10-09 17:42:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 290/2304 [I 2022-10-09 17:42:32] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:41 [I 2022-10-09 17:42:32] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00347 [I 2022-10-09 17:42:32] ...trap/engines/engine.py.230: loss: 0.12146 [I 2022-10-09 17:42:32] ...optimizers/trijoint.py.151: total_norm: 13.071725 [I 2022-10-09 17:42:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 300/2304 [I 2022-10-09 17:42:40] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:29 [I 2022-10-09 17:42:40] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00325 [I 2022-10-09 17:42:40] ...trap/engines/engine.py.230: loss: 0.21764 [I 2022-10-09 17:42:40] ...optimizers/trijoint.py.151: total_norm: 10.029741 [I 2022-10-09 17:42:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 310/2304 [I 2022-10-09 17:42:49] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:16 [I 2022-10-09 17:42:49] ...trap/engines/engine.py.229: process: 0.85433 | load: 0.00306 [I 2022-10-09 17:42:49] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-09 17:42:49] ...optimizers/trijoint.py.151: total_norm: 15.152005 [I 2022-10-09 17:42:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 320/2304 [I 2022-10-09 17:42:57] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:13 [I 2022-10-09 17:42:57] ...trap/engines/engine.py.229: process: 0.84793 | load: 0.00304 [I 2022-10-09 17:42:57] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-09 17:42:57] ...optimizers/trijoint.py.151: total_norm: 14.253153 [I 2022-10-09 17:43:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 330/2304 [I 2022-10-09 17:43:06] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:27:47 [I 2022-10-09 17:43:06] ...trap/engines/engine.py.229: process: 0.83721 | load: 0.00298 [I 2022-10-09 17:43:06] ...trap/engines/engine.py.230: loss: 0.17854 [I 2022-10-09 17:43:06] ...optimizers/trijoint.py.151: total_norm: 25.192463 [I 2022-10-09 17:43:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 340/2304 [I 2022-10-09 17:43:14] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:27:41 [I 2022-10-09 17:43:14] ...trap/engines/engine.py.229: process: 0.85736 | load: 0.00308 [I 2022-10-09 17:43:14] ...trap/engines/engine.py.230: loss: 0.17805 [I 2022-10-09 17:43:14] ...optimizers/trijoint.py.151: total_norm: 10.880469 [I 2022-10-09 17:43:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 350/2304 [I 2022-10-09 17:43:23] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:39 [I 2022-10-09 17:43:23] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00299 [I 2022-10-09 17:43:23] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-09 17:43:23] ...optimizers/trijoint.py.151: total_norm: 15.696063 [I 2022-10-09 17:43:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 360/2304 [I 2022-10-09 17:43:31] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:28 [I 2022-10-09 17:43:31] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00311 [I 2022-10-09 17:43:31] ...trap/engines/engine.py.230: loss: 0.18369 [I 2022-10-09 17:43:31] ...optimizers/trijoint.py.151: total_norm: 15.138580 [I 2022-10-09 17:43:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 370/2304 [I 2022-10-09 17:43:40] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:14 [I 2022-10-09 17:43:40] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00310 [I 2022-10-09 17:43:40] ...trap/engines/engine.py.230: loss: 0.25427 [I 2022-10-09 17:43:40] ...optimizers/trijoint.py.151: total_norm: 25.508471 [I 2022-10-09 17:43:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 380/2304 [I 2022-10-09 17:43:48] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:09 [I 2022-10-09 17:43:48] ...trap/engines/engine.py.229: process: 0.83818 | load: 0.00311 [I 2022-10-09 17:43:48] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-09 17:43:48] ...optimizers/trijoint.py.151: total_norm: 13.719938 [I 2022-10-09 17:43:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 390/2304 [I 2022-10-09 17:43:57] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:26:59 [I 2022-10-09 17:43:57] ...trap/engines/engine.py.229: process: 0.86141 | load: 0.00336 [I 2022-10-09 17:43:57] ...trap/engines/engine.py.230: loss: 0.14242 [I 2022-10-09 17:43:57] ...optimizers/trijoint.py.151: total_norm: 12.625155 [I 2022-10-09 17:44:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 400/2304 [I 2022-10-09 17:44:05] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:05 [I 2022-10-09 17:44:05] ...trap/engines/engine.py.229: process: 0.87630 | load: 0.00321 [I 2022-10-09 17:44:05] ...trap/engines/engine.py.230: loss: 0.10104 [I 2022-10-09 17:44:05] ...optimizers/trijoint.py.151: total_norm: 8.370757 [I 2022-10-09 17:44:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 410/2304 [I 2022-10-09 17:44:14] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:42 [I 2022-10-09 17:44:14] ...trap/engines/engine.py.229: process: 0.84437 | load: 0.00312 [I 2022-10-09 17:44:14] ...trap/engines/engine.py.230: loss: 0.20691 [I 2022-10-09 17:44:14] ...optimizers/trijoint.py.151: total_norm: 20.933193 [I 2022-10-09 17:44:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 420/2304 [I 2022-10-09 17:44:22] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:26:33 [I 2022-10-09 17:44:22] ...trap/engines/engine.py.229: process: 0.85977 | load: 0.00308 [I 2022-10-09 17:44:22] ...trap/engines/engine.py.230: loss: 0.17167 [I 2022-10-09 17:44:22] ...optimizers/trijoint.py.151: total_norm: 24.156252 [I 2022-10-09 17:44:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 430/2304 [I 2022-10-09 17:44:31] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:22 [I 2022-10-09 17:44:31] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00302 [I 2022-10-09 17:44:31] ...trap/engines/engine.py.230: loss: 0.11001 [I 2022-10-09 17:44:31] ...optimizers/trijoint.py.151: total_norm: 7.615691 [I 2022-10-09 17:44:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 440/2304 [I 2022-10-09 17:44:39] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:26:29 [I 2022-10-09 17:44:39] ...trap/engines/engine.py.229: process: 0.86137 | load: 0.00323 [I 2022-10-09 17:44:39] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-09 17:44:39] ...optimizers/trijoint.py.151: total_norm: 10.056807 [I 2022-10-09 17:44:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 450/2304 [I 2022-10-09 17:44:48] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:20 [I 2022-10-09 17:44:48] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00317 [I 2022-10-09 17:44:48] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-09 17:44:48] ...optimizers/trijoint.py.151: total_norm: 22.437111 [I 2022-10-09 17:44:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 460/2304 [I 2022-10-09 17:44:56] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:26:08 [I 2022-10-09 17:44:56] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00317 [I 2022-10-09 17:44:56] ...trap/engines/engine.py.230: loss: 0.14251 [I 2022-10-09 17:44:56] ...optimizers/trijoint.py.151: total_norm: 20.869337 [I 2022-10-09 17:45:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 470/2304 [I 2022-10-09 17:45:05] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:55 [I 2022-10-09 17:45:05] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00319 [I 2022-10-09 17:45:05] ...trap/engines/engine.py.230: loss: 0.18101 [I 2022-10-09 17:45:05] ...optimizers/trijoint.py.151: total_norm: 27.417917 [I 2022-10-09 17:45:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 480/2304 [I 2022-10-09 17:45:13] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:25:58 [I 2022-10-09 17:45:13] ...trap/engines/engine.py.229: process: 0.87329 | load: 0.00299 [I 2022-10-09 17:45:13] ...trap/engines/engine.py.230: loss: 0.17427 [I 2022-10-09 17:45:13] ...optimizers/trijoint.py.151: total_norm: 20.975412 [I 2022-10-09 17:45:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 490/2304 [I 2022-10-09 17:45:22] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:48 [I 2022-10-09 17:45:22] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00303 [I 2022-10-09 17:45:22] ...trap/engines/engine.py.230: loss: 0.12751 [I 2022-10-09 17:45:22] ...optimizers/trijoint.py.151: total_norm: 16.081247 [I 2022-10-09 17:45:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 500/2304 [I 2022-10-09 17:45:30] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:21 [I 2022-10-09 17:45:30] ...trap/engines/engine.py.229: process: 0.83528 | load: 0.00298 [I 2022-10-09 17:45:30] ...trap/engines/engine.py.230: loss: 0.18542 [I 2022-10-09 17:45:30] ...optimizers/trijoint.py.151: total_norm: 13.604771 [I 2022-10-09 17:45:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 510/2304 [I 2022-10-09 17:45:39] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:22 [I 2022-10-09 17:45:39] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00296 [I 2022-10-09 17:45:39] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-09 17:45:39] ...optimizers/trijoint.py.151: total_norm: 15.373007 [I 2022-10-09 17:45:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 520/2304 [I 2022-10-09 17:45:48] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:25:22 [I 2022-10-09 17:45:48] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00298 [I 2022-10-09 17:45:48] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-09 17:45:48] ...optimizers/trijoint.py.151: total_norm: 28.728064 [I 2022-10-09 17:45:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 530/2304 [I 2022-10-09 17:45:56] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:05 [I 2022-10-09 17:45:56] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00307 [I 2022-10-09 17:45:56] ...trap/engines/engine.py.230: loss: 0.16175 [I 2022-10-09 17:45:56] ...optimizers/trijoint.py.151: total_norm: 12.255427 [I 2022-10-09 17:46:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 540/2304 [I 2022-10-09 17:46:05] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:25:03 [I 2022-10-09 17:46:05] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00295 [I 2022-10-09 17:46:05] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-09 17:46:05] ...optimizers/trijoint.py.151: total_norm: 12.438058 [I 2022-10-09 17:46:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 550/2304 [I 2022-10-09 17:46:13] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:01 [I 2022-10-09 17:46:13] ...trap/engines/engine.py.229: process: 0.85419 | load: 0.00307 [I 2022-10-09 17:46:13] ...trap/engines/engine.py.230: loss: 0.21537 [I 2022-10-09 17:46:13] ...optimizers/trijoint.py.151: total_norm: 45.811691 [I 2022-10-09 17:46:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 560/2304 [I 2022-10-09 17:46:22] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:24:41 [I 2022-10-09 17:46:22] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00308 [I 2022-10-09 17:46:22] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-09 17:46:22] ...optimizers/trijoint.py.151: total_norm: 21.933912 [I 2022-10-09 17:46:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 570/2304 [I 2022-10-09 17:46:30] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:36 [I 2022-10-09 17:46:30] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00300 [I 2022-10-09 17:46:30] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-09 17:46:30] ...optimizers/trijoint.py.151: total_norm: 10.750507 [I 2022-10-09 17:46:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 580/2304 [I 2022-10-09 17:46:39] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:24 [I 2022-10-09 17:46:39] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00304 [I 2022-10-09 17:46:39] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-09 17:46:39] ...optimizers/trijoint.py.151: total_norm: 12.485357 [I 2022-10-09 17:46:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 590/2304 [I 2022-10-09 17:46:47] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:21 [I 2022-10-09 17:46:47] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00308 [I 2022-10-09 17:46:47] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-09 17:46:47] ...optimizers/trijoint.py.151: total_norm: 9.277979 [I 2022-10-09 17:46:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 600/2304 [I 2022-10-09 17:46:56] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:10 [I 2022-10-09 17:46:56] ...trap/engines/engine.py.229: process: 0.86326 | load: 0.00301 [I 2022-10-09 17:46:56] ...trap/engines/engine.py.230: loss: 0.14797 [I 2022-10-09 17:46:56] ...optimizers/trijoint.py.151: total_norm: 13.012304 [I 2022-10-09 17:47:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 610/2304 [I 2022-10-09 17:47:04] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:57 [I 2022-10-09 17:47:04] ...trap/engines/engine.py.229: process: 0.83322 | load: 0.00300 [I 2022-10-09 17:47:04] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-09 17:47:04] ...optimizers/trijoint.py.151: total_norm: 14.937888 [I 2022-10-09 17:47:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 620/2304 [I 2022-10-09 17:47:13] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:44 [I 2022-10-09 17:47:13] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00296 [I 2022-10-09 17:47:13] ...trap/engines/engine.py.230: loss: 0.21166 [I 2022-10-09 17:47:13] ...optimizers/trijoint.py.151: total_norm: 16.843536 [I 2022-10-09 17:47:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 630/2304 [I 2022-10-09 17:47:21] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:54 [I 2022-10-09 17:47:21] ...trap/engines/engine.py.229: process: 0.85506 | load: 0.00296 [I 2022-10-09 17:47:21] ...trap/engines/engine.py.230: loss: 0.13430 [I 2022-10-09 17:47:21] ...optimizers/trijoint.py.151: total_norm: 14.570581 [I 2022-10-09 17:47:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 640/2304 [I 2022-10-09 17:47:30] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:38 [I 2022-10-09 17:47:30] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00320 [I 2022-10-09 17:47:30] ...trap/engines/engine.py.230: loss: 0.18734 [I 2022-10-09 17:47:30] ...optimizers/trijoint.py.151: total_norm: 18.815928 [I 2022-10-09 17:47:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 650/2304 [I 2022-10-09 17:47:38] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:29 [I 2022-10-09 17:47:38] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00313 [I 2022-10-09 17:47:38] ...trap/engines/engine.py.230: loss: 0.18374 [I 2022-10-09 17:47:38] ...optimizers/trijoint.py.151: total_norm: 19.421812 [I 2022-10-09 17:47:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 660/2304 [I 2022-10-09 17:47:47] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:22 [I 2022-10-09 17:47:47] ...trap/engines/engine.py.229: process: 0.85143 | load: 0.00310 [I 2022-10-09 17:47:47] ...trap/engines/engine.py.230: loss: 0.15733 [I 2022-10-09 17:47:47] ...optimizers/trijoint.py.151: total_norm: 18.746359 [I 2022-10-09 17:47:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 670/2304 [I 2022-10-09 17:47:56] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:15 [I 2022-10-09 17:47:56] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00310 [I 2022-10-09 17:47:56] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-09 17:47:56] ...optimizers/trijoint.py.151: total_norm: 10.307908 [I 2022-10-09 17:48:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 680/2304 [I 2022-10-09 17:48:04] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:14 [I 2022-10-09 17:48:04] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00310 [I 2022-10-09 17:48:04] ...trap/engines/engine.py.230: loss: 0.19499 [I 2022-10-09 17:48:04] ...optimizers/trijoint.py.151: total_norm: 23.117445 [I 2022-10-09 17:48:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 690/2304 [I 2022-10-09 17:48:13] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:23:00 [I 2022-10-09 17:48:13] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00311 [I 2022-10-09 17:48:13] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-09 17:48:13] ...optimizers/trijoint.py.151: total_norm: 9.810611 [I 2022-10-09 17:48:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 700/2304 [I 2022-10-09 17:48:21] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:50 [I 2022-10-09 17:48:21] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00311 [I 2022-10-09 17:48:21] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-09 17:48:21] ...optimizers/trijoint.py.151: total_norm: 14.481781 [I 2022-10-09 17:48:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 710/2304 [I 2022-10-09 17:48:30] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:44 [I 2022-10-09 17:48:30] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00311 [I 2022-10-09 17:48:30] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-09 17:48:30] ...optimizers/trijoint.py.151: total_norm: 16.747629 [I 2022-10-09 17:48:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 720/2304 [I 2022-10-09 17:48:39] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:43 [I 2022-10-09 17:48:39] ...trap/engines/engine.py.229: process: 0.85586 | load: 0.00314 [I 2022-10-09 17:48:39] ...trap/engines/engine.py.230: loss: 0.17645 [I 2022-10-09 17:48:39] ...optimizers/trijoint.py.151: total_norm: 19.432482 [I 2022-10-09 17:48:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 730/2304 [I 2022-10-09 17:48:47] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:36 [I 2022-10-09 17:48:47] ...trap/engines/engine.py.229: process: 0.87519 | load: 0.00310 [I 2022-10-09 17:48:47] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-09 17:48:47] ...optimizers/trijoint.py.151: total_norm: 12.801000 [I 2022-10-09 17:48:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 740/2304 [I 2022-10-09 17:48:56] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:20 [I 2022-10-09 17:48:56] ...trap/engines/engine.py.229: process: 0.83600 | load: 0.00309 [I 2022-10-09 17:48:56] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-09 17:48:56] ...optimizers/trijoint.py.151: total_norm: 12.415803 [I 2022-10-09 17:49:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 750/2304 [I 2022-10-09 17:49:04] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:22:09 [I 2022-10-09 17:49:04] ...trap/engines/engine.py.229: process: 0.84272 | load: 0.00312 [I 2022-10-09 17:49:04] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-09 17:49:04] ...optimizers/trijoint.py.151: total_norm: 14.218229 [I 2022-10-09 17:49:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 760/2304 [I 2022-10-09 17:49:13] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:59 [I 2022-10-09 17:49:13] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00311 [I 2022-10-09 17:49:13] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-09 17:49:13] ...optimizers/trijoint.py.151: total_norm: 12.543087 [I 2022-10-09 17:49:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 770/2304 [I 2022-10-09 17:49:21] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:55 [I 2022-10-09 17:49:21] ...trap/engines/engine.py.229: process: 0.86220 | load: 0.00313 [I 2022-10-09 17:49:21] ...trap/engines/engine.py.230: loss: 0.09917 [I 2022-10-09 17:49:21] ...optimizers/trijoint.py.151: total_norm: 12.331514 [I 2022-10-09 17:49:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 780/2304 [I 2022-10-09 17:49:30] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:45 [I 2022-10-09 17:49:30] ...trap/engines/engine.py.229: process: 0.86818 | load: 0.00311 [I 2022-10-09 17:49:30] ...trap/engines/engine.py.230: loss: 0.14652 [I 2022-10-09 17:49:30] ...optimizers/trijoint.py.151: total_norm: 22.926773 [I 2022-10-09 17:49:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 790/2304 [I 2022-10-09 17:49:39] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:33 [I 2022-10-09 17:49:39] ...trap/engines/engine.py.229: process: 0.85991 | load: 0.00309 [I 2022-10-09 17:49:39] ...trap/engines/engine.py.230: loss: 0.15045 [I 2022-10-09 17:49:39] ...optimizers/trijoint.py.151: total_norm: 9.820752 [I 2022-10-09 17:49:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 800/2304 [I 2022-10-09 17:49:47] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:24 [I 2022-10-09 17:49:47] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00311 [I 2022-10-09 17:49:47] ...trap/engines/engine.py.230: loss: 0.10426 [I 2022-10-09 17:49:47] ...optimizers/trijoint.py.151: total_norm: 13.637564 [I 2022-10-09 17:49:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 810/2304 [I 2022-10-09 17:49:56] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:09 [I 2022-10-09 17:49:56] ...trap/engines/engine.py.229: process: 0.84515 | load: 0.00312 [I 2022-10-09 17:49:56] ...trap/engines/engine.py.230: loss: 0.13270 [I 2022-10-09 17:49:56] ...optimizers/trijoint.py.151: total_norm: 12.199169 [I 2022-10-09 17:50:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 820/2304 [I 2022-10-09 17:50:04] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:21:05 [I 2022-10-09 17:50:04] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00311 [I 2022-10-09 17:50:04] ...trap/engines/engine.py.230: loss: 0.17615 [I 2022-10-09 17:50:04] ...optimizers/trijoint.py.151: total_norm: 28.352829 [I 2022-10-09 17:50:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 830/2304 [I 2022-10-09 17:50:13] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:21:12 [I 2022-10-09 17:50:13] ...trap/engines/engine.py.229: process: 0.86532 | load: 0.00300 [I 2022-10-09 17:50:13] ...trap/engines/engine.py.230: loss: 0.14089 [I 2022-10-09 17:50:13] ...optimizers/trijoint.py.151: total_norm: 18.817682 [I 2022-10-09 17:50:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 840/2304 [I 2022-10-09 17:50:21] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:20:42 [I 2022-10-09 17:50:21] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00311 [I 2022-10-09 17:50:21] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-09 17:50:21] ...optimizers/trijoint.py.151: total_norm: 19.586052 [I 2022-10-09 17:50:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 850/2304 [I 2022-10-09 17:50:30] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:32 [I 2022-10-09 17:50:30] ...trap/engines/engine.py.229: process: 0.84928 | load: 0.00311 [I 2022-10-09 17:50:30] ...trap/engines/engine.py.230: loss: 0.15051 [I 2022-10-09 17:50:30] ...optimizers/trijoint.py.151: total_norm: 20.659391 [I 2022-10-09 17:50:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 860/2304 [I 2022-10-09 17:50:38] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:23 [I 2022-10-09 17:50:38] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00313 [I 2022-10-09 17:50:38] ...trap/engines/engine.py.230: loss: 0.09803 [I 2022-10-09 17:50:38] ...optimizers/trijoint.py.151: total_norm: 9.856504 [I 2022-10-09 17:50:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 870/2304 [I 2022-10-09 17:50:47] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:10 [I 2022-10-09 17:50:47] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00305 [I 2022-10-09 17:50:47] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-09 17:50:47] ...optimizers/trijoint.py.151: total_norm: 12.618664 [I 2022-10-09 17:50:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 880/2304 [I 2022-10-09 17:50:55] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:03 [I 2022-10-09 17:50:55] ...trap/engines/engine.py.229: process: 0.83550 | load: 0.00305 [I 2022-10-09 17:50:55] ...trap/engines/engine.py.230: loss: 0.12743 [I 2022-10-09 17:50:55] ...optimizers/trijoint.py.151: total_norm: 15.279242 [I 2022-10-09 17:51:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 890/2304 [I 2022-10-09 17:51:04] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:19:57 [I 2022-10-09 17:51:04] ...trap/engines/engine.py.229: process: 0.85556 | load: 0.00313 [I 2022-10-09 17:51:04] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-09 17:51:04] ...optimizers/trijoint.py.151: total_norm: 16.445644 [I 2022-10-09 17:51:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 900/2304 [I 2022-10-09 17:51:12] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:19:46 [I 2022-10-09 17:51:12] ...trap/engines/engine.py.229: process: 0.83439 | load: 0.00310 [I 2022-10-09 17:51:12] ...trap/engines/engine.py.230: loss: 0.26622 [I 2022-10-09 17:51:12] ...optimizers/trijoint.py.151: total_norm: 34.043781 [I 2022-10-09 17:51:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 910/2304 [I 2022-10-09 17:51:21] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:19:32 [I 2022-10-09 17:51:21] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00294 [I 2022-10-09 17:51:21] ...trap/engines/engine.py.230: loss: 0.18878 [I 2022-10-09 17:51:21] ...optimizers/trijoint.py.151: total_norm: 31.228189 [I 2022-10-09 17:51:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 920/2304 [I 2022-10-09 17:51:29] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:32 [I 2022-10-09 17:51:29] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00301 [I 2022-10-09 17:51:29] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-09 17:51:29] ...optimizers/trijoint.py.151: total_norm: 15.653015 [I 2022-10-09 17:51:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 930/2304 [I 2022-10-09 17:51:38] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:25 [I 2022-10-09 17:51:38] ...trap/engines/engine.py.229: process: 0.84623 | load: 0.00302 [I 2022-10-09 17:51:38] ...trap/engines/engine.py.230: loss: 0.11841 [I 2022-10-09 17:51:38] ...optimizers/trijoint.py.151: total_norm: 11.679041 [I 2022-10-09 17:51:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 940/2304 [I 2022-10-09 17:51:46] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:13 [I 2022-10-09 17:51:46] ...trap/engines/engine.py.229: process: 0.85076 | load: 0.00298 [I 2022-10-09 17:51:46] ...trap/engines/engine.py.230: loss: 0.15850 [I 2022-10-09 17:51:46] ...optimizers/trijoint.py.151: total_norm: 21.558397 [I 2022-10-09 17:51:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 950/2304 [I 2022-10-09 17:51:55] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:15 [I 2022-10-09 17:51:55] ...trap/engines/engine.py.229: process: 0.85665 | load: 0.00302 [I 2022-10-09 17:51:55] ...trap/engines/engine.py.230: loss: 0.11030 [I 2022-10-09 17:51:55] ...optimizers/trijoint.py.151: total_norm: 14.714455 [I 2022-10-09 17:52:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 960/2304 [I 2022-10-09 17:52:03] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:18:56 [I 2022-10-09 17:52:03] ...trap/engines/engine.py.229: process: 0.83188 | load: 0.00310 [I 2022-10-09 17:52:03] ...trap/engines/engine.py.230: loss: 0.12775 [I 2022-10-09 17:52:03] ...optimizers/trijoint.py.151: total_norm: 7.221196 [I 2022-10-09 17:52:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 970/2304 [I 2022-10-09 17:52:12] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:18:56 [I 2022-10-09 17:52:12] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00326 [I 2022-10-09 17:52:12] ...trap/engines/engine.py.230: loss: 0.20042 [I 2022-10-09 17:52:12] ...optimizers/trijoint.py.151: total_norm: 23.829781 [I 2022-10-09 17:52:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 980/2304 [I 2022-10-09 17:52:20] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:52 [I 2022-10-09 17:52:20] ...trap/engines/engine.py.229: process: 0.86972 | load: 0.00309 [I 2022-10-09 17:52:20] ...trap/engines/engine.py.230: loss: 0.16308 [I 2022-10-09 17:52:20] ...optimizers/trijoint.py.151: total_norm: 9.360038 [I 2022-10-09 17:52:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 990/2304 [I 2022-10-09 17:52:29] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:32 [I 2022-10-09 17:52:29] ...trap/engines/engine.py.229: process: 0.83461 | load: 0.00312 [I 2022-10-09 17:52:29] ...trap/engines/engine.py.230: loss: 0.16398 [I 2022-10-09 17:52:29] ...optimizers/trijoint.py.151: total_norm: 15.729952 [I 2022-10-09 17:52:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1000/2304 [I 2022-10-09 17:52:37] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:28 [I 2022-10-09 17:52:37] ...trap/engines/engine.py.229: process: 0.85188 | load: 0.00307 [I 2022-10-09 17:52:37] ...trap/engines/engine.py.230: loss: 0.14149 [I 2022-10-09 17:52:37] ...optimizers/trijoint.py.151: total_norm: 33.730221 [I 2022-10-09 17:52:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 1010/2304 [I 2022-10-09 17:52:46] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:22 [I 2022-10-09 17:52:46] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00297 [I 2022-10-09 17:52:46] ...trap/engines/engine.py.230: loss: 0.12463 [I 2022-10-09 17:52:46] ...optimizers/trijoint.py.151: total_norm: 6.660469 [I 2022-10-09 17:52:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1020/2304 [I 2022-10-09 17:52:54] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:09 [I 2022-10-09 17:52:54] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00310 [I 2022-10-09 17:52:54] ...trap/engines/engine.py.230: loss: 0.11647 [I 2022-10-09 17:52:54] ...optimizers/trijoint.py.151: total_norm: 7.372606 [I 2022-10-09 17:53:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 1030/2304 [I 2022-10-09 17:53:03] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:10 [I 2022-10-09 17:53:03] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00307 [I 2022-10-09 17:53:03] ...trap/engines/engine.py.230: loss: 0.25139 [I 2022-10-09 17:53:03] ...optimizers/trijoint.py.151: total_norm: 24.787998 [I 2022-10-09 17:53:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 1040/2304 [I 2022-10-09 17:53:12] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:59 [I 2022-10-09 17:53:12] ...trap/engines/engine.py.229: process: 0.85822 | load: 0.00298 [I 2022-10-09 17:53:12] ...trap/engines/engine.py.230: loss: 0.09588 [I 2022-10-09 17:53:12] ...optimizers/trijoint.py.151: total_norm: 23.338015 [I 2022-10-09 17:53:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 1050/2304 [I 2022-10-09 17:53:20] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:48 [I 2022-10-09 17:53:20] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00311 [I 2022-10-09 17:53:20] ...trap/engines/engine.py.230: loss: 0.12770 [I 2022-10-09 17:53:20] ...optimizers/trijoint.py.151: total_norm: 11.026039 [I 2022-10-09 17:53:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 1060/2304 [I 2022-10-09 17:53:29] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:40 [I 2022-10-09 17:53:29] ...trap/engines/engine.py.229: process: 0.85889 | load: 0.00313 [I 2022-10-09 17:53:29] ...trap/engines/engine.py.230: loss: 0.13772 [I 2022-10-09 17:53:29] ...optimizers/trijoint.py.151: total_norm: 13.506605 [I 2022-10-09 17:53:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1070/2304 [I 2022-10-09 17:53:37] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:26 [I 2022-10-09 17:53:37] ...trap/engines/engine.py.229: process: 0.85063 | load: 0.00308 [I 2022-10-09 17:53:37] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-09 17:53:37] ...optimizers/trijoint.py.151: total_norm: 18.514059 [I 2022-10-09 17:53:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 1080/2304 [I 2022-10-09 17:53:46] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:19 [I 2022-10-09 17:53:46] ...trap/engines/engine.py.229: process: 0.84231 | load: 0.00304 [I 2022-10-09 17:53:46] ...trap/engines/engine.py.230: loss: 0.13033 [I 2022-10-09 17:53:46] ...optimizers/trijoint.py.151: total_norm: 11.633791 [I 2022-10-09 17:53:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1090/2304 [I 2022-10-09 17:53:54] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:14 [I 2022-10-09 17:53:54] ...trap/engines/engine.py.229: process: 0.85403 | load: 0.00299 [I 2022-10-09 17:53:54] ...trap/engines/engine.py.230: loss: 0.21096 [I 2022-10-09 17:53:54] ...optimizers/trijoint.py.151: total_norm: 20.905109 [I 2022-10-09 17:54:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 1100/2304 [I 2022-10-09 17:54:03] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:07 [I 2022-10-09 17:54:03] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00310 [I 2022-10-09 17:54:03] ...trap/engines/engine.py.230: loss: 0.19838 [I 2022-10-09 17:54:03] ...optimizers/trijoint.py.151: total_norm: 19.962383 [I 2022-10-09 17:54:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1110/2304 [I 2022-10-09 17:54:11] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:58 [I 2022-10-09 17:54:11] ...trap/engines/engine.py.229: process: 0.84800 | load: 0.00313 [I 2022-10-09 17:54:11] ...trap/engines/engine.py.230: loss: 0.17068 [I 2022-10-09 17:54:11] ...optimizers/trijoint.py.151: total_norm: 21.340160 [I 2022-10-09 17:54:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 1120/2304 [I 2022-10-09 17:54:20] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:46 [I 2022-10-09 17:54:20] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00297 [I 2022-10-09 17:54:20] ...trap/engines/engine.py.230: loss: 0.17473 [I 2022-10-09 17:54:20] ...optimizers/trijoint.py.151: total_norm: 13.349348 [I 2022-10-09 17:54:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1130/2304 [I 2022-10-09 17:54:28] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:35 [I 2022-10-09 17:54:28] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00297 [I 2022-10-09 17:54:28] ...trap/engines/engine.py.230: loss: 0.12481 [I 2022-10-09 17:54:28] ...optimizers/trijoint.py.151: total_norm: 8.951870 [I 2022-10-09 17:54:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1140/2304 [I 2022-10-09 17:54:37] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:30 [I 2022-10-09 17:54:37] ...trap/engines/engine.py.229: process: 0.85394 | load: 0.00300 [I 2022-10-09 17:54:37] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-09 17:54:37] ...optimizers/trijoint.py.151: total_norm: 9.743102 [I 2022-10-09 17:54:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1150/2304 [I 2022-10-09 17:54:45] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:17 [I 2022-10-09 17:54:45] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00300 [I 2022-10-09 17:54:45] ...trap/engines/engine.py.230: loss: 0.14755 [I 2022-10-09 17:54:45] ...optimizers/trijoint.py.151: total_norm: 23.965673 [I 2022-10-09 17:54:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1160/2304 [I 2022-10-09 17:54:54] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:12 [I 2022-10-09 17:54:54] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00303 [I 2022-10-09 17:54:54] ...trap/engines/engine.py.230: loss: 0.10817 [I 2022-10-09 17:54:54] ...optimizers/trijoint.py.151: total_norm: 13.058064 [I 2022-10-09 17:55:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1170/2304 [I 2022-10-09 17:55:02] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:04 [I 2022-10-09 17:55:02] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00300 [I 2022-10-09 17:55:02] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-09 17:55:02] ...optimizers/trijoint.py.151: total_norm: 12.810495 [I 2022-10-09 17:55:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1180/2304 [I 2022-10-09 17:55:11] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:59 [I 2022-10-09 17:55:11] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00312 [I 2022-10-09 17:55:11] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-09 17:55:11] ...optimizers/trijoint.py.151: total_norm: 17.443335 [I 2022-10-09 17:55:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1190/2304 [I 2022-10-09 17:55:19] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:51 [I 2022-10-09 17:55:19] ...trap/engines/engine.py.229: process: 0.86336 | load: 0.00311 [I 2022-10-09 17:55:19] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-09 17:55:19] ...optimizers/trijoint.py.151: total_norm: 13.956573 [I 2022-10-09 17:55:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1200/2304 [I 2022-10-09 17:55:28] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:51 [I 2022-10-09 17:55:28] ...trap/engines/engine.py.229: process: 0.86453 | load: 0.00312 [I 2022-10-09 17:55:28] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-09 17:55:28] ...optimizers/trijoint.py.151: total_norm: 16.335135 [I 2022-10-09 17:55:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1210/2304 [I 2022-10-09 17:55:37] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:38 [I 2022-10-09 17:55:37] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00312 [I 2022-10-09 17:55:37] ...trap/engines/engine.py.230: loss: 0.19439 [I 2022-10-09 17:55:37] ...optimizers/trijoint.py.151: total_norm: 18.024891 [I 2022-10-09 17:55:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1220/2304 [I 2022-10-09 17:55:45] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:33 [I 2022-10-09 17:55:45] ...trap/engines/engine.py.229: process: 0.85876 | load: 0.00321 [I 2022-10-09 17:55:45] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-09 17:55:45] ...optimizers/trijoint.py.151: total_norm: 11.453416 [I 2022-10-09 17:55:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1230/2304 [I 2022-10-09 17:55:54] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:23 [I 2022-10-09 17:55:54] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00308 [I 2022-10-09 17:55:54] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-09 17:55:54] ...optimizers/trijoint.py.151: total_norm: 14.056348 [I 2022-10-09 17:56:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1240/2304 [I 2022-10-09 17:56:02] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:09 [I 2022-10-09 17:56:02] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00309 [I 2022-10-09 17:56:02] ...trap/engines/engine.py.230: loss: 0.19135 [I 2022-10-09 17:56:02] ...optimizers/trijoint.py.151: total_norm: 20.068756 [I 2022-10-09 17:56:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1250/2304 [I 2022-10-09 17:56:11] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:05 [I 2022-10-09 17:56:11] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00313 [I 2022-10-09 17:56:11] ...trap/engines/engine.py.230: loss: 0.14309 [I 2022-10-09 17:56:11] ...optimizers/trijoint.py.151: total_norm: 27.209692 [I 2022-10-09 17:56:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 1260/2304 [I 2022-10-09 17:56:20] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:52 [I 2022-10-09 17:56:20] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00313 [I 2022-10-09 17:56:20] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-09 17:56:20] ...optimizers/trijoint.py.151: total_norm: 10.225280 [I 2022-10-09 17:56:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1270/2304 [I 2022-10-09 17:56:28] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:45 [I 2022-10-09 17:56:28] ...trap/engines/engine.py.229: process: 0.86322 | load: 0.00310 [I 2022-10-09 17:56:28] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-09 17:56:28] ...optimizers/trijoint.py.151: total_norm: 14.580934 [I 2022-10-09 17:56:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1280/2304 [I 2022-10-09 17:56:37] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:33 [I 2022-10-09 17:56:37] ...trap/engines/engine.py.229: process: 0.84732 | load: 0.00314 [I 2022-10-09 17:56:37] ...trap/engines/engine.py.230: loss: 0.09819 [I 2022-10-09 17:56:37] ...optimizers/trijoint.py.151: total_norm: 9.715988 [I 2022-10-09 17:56:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1290/2304 [I 2022-10-09 17:56:45] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:27 [I 2022-10-09 17:56:45] ...trap/engines/engine.py.229: process: 0.85821 | load: 0.00311 [I 2022-10-09 17:56:45] ...trap/engines/engine.py.230: loss: 0.12954 [I 2022-10-09 17:56:45] ...optimizers/trijoint.py.151: total_norm: 9.145259 [I 2022-10-09 17:56:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1300/2304 [I 2022-10-09 17:56:54] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:19 [I 2022-10-09 17:56:54] ...trap/engines/engine.py.229: process: 0.86148 | load: 0.00310 [I 2022-10-09 17:56:54] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-09 17:56:54] ...optimizers/trijoint.py.151: total_norm: 46.861584 [I 2022-10-09 17:57:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1310/2304 [I 2022-10-09 17:57:02] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:14:07 [I 2022-10-09 17:57:02] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00312 [I 2022-10-09 17:57:02] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-09 17:57:02] ...optimizers/trijoint.py.151: total_norm: 20.535086 [I 2022-10-09 17:57:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1320/2304 [I 2022-10-09 17:57:11] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:13:56 [I 2022-10-09 17:57:11] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00310 [I 2022-10-09 17:57:11] ...trap/engines/engine.py.230: loss: 0.15066 [I 2022-10-09 17:57:11] ...optimizers/trijoint.py.151: total_norm: 18.933609 [I 2022-10-09 17:57:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 1330/2304 [I 2022-10-09 17:57:20] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:54 [I 2022-10-09 17:57:20] ...trap/engines/engine.py.229: process: 0.86219 | load: 0.00309 [I 2022-10-09 17:57:20] ...trap/engines/engine.py.230: loss: 0.12091 [I 2022-10-09 17:57:20] ...optimizers/trijoint.py.151: total_norm: 8.629780 [I 2022-10-09 17:57:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1340/2304 [I 2022-10-09 17:57:28] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:48 [I 2022-10-09 17:57:28] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00311 [I 2022-10-09 17:57:28] ...trap/engines/engine.py.230: loss: 0.18649 [I 2022-10-09 17:57:28] ...optimizers/trijoint.py.151: total_norm: 21.888283 [I 2022-10-09 17:57:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1350/2304 [I 2022-10-09 17:57:37] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:37 [I 2022-10-09 17:57:37] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00307 [I 2022-10-09 17:57:37] ...trap/engines/engine.py.230: loss: 0.15159 [I 2022-10-09 17:57:37] ...optimizers/trijoint.py.151: total_norm: 19.041981 [I 2022-10-09 17:57:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1360/2304 [I 2022-10-09 17:57:45] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:23 [I 2022-10-09 17:57:45] ...trap/engines/engine.py.229: process: 0.86186 | load: 0.00315 [I 2022-10-09 17:57:45] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-09 17:57:45] ...optimizers/trijoint.py.151: total_norm: 15.026506 [I 2022-10-09 17:57:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1370/2304 [I 2022-10-09 17:57:54] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:13:10 [I 2022-10-09 17:57:54] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00309 [I 2022-10-09 17:57:54] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-09 17:57:54] ...optimizers/trijoint.py.151: total_norm: 18.691225 [I 2022-10-09 17:58:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1380/2304 [I 2022-10-09 17:58:02] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:07 [I 2022-10-09 17:58:02] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00308 [I 2022-10-09 17:58:02] ...trap/engines/engine.py.230: loss: 0.18159 [I 2022-10-09 17:58:02] ...optimizers/trijoint.py.151: total_norm: 19.734653 [I 2022-10-09 17:58:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1390/2304 [I 2022-10-09 17:58:11] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:13:00 [I 2022-10-09 17:58:11] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00308 [I 2022-10-09 17:58:11] ...trap/engines/engine.py.230: loss: 0.13809 [I 2022-10-09 17:58:11] ...optimizers/trijoint.py.151: total_norm: 9.422643 [I 2022-10-09 17:58:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 1400/2304 [I 2022-10-09 17:58:20] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:52 [I 2022-10-09 17:58:20] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00304 [I 2022-10-09 17:58:20] ...trap/engines/engine.py.230: loss: 0.13451 [I 2022-10-09 17:58:20] ...optimizers/trijoint.py.151: total_norm: 25.201519 [I 2022-10-09 17:58:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1410/2304 [I 2022-10-09 17:58:28] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:12:36 [I 2022-10-09 17:58:28] ...trap/engines/engine.py.229: process: 0.82831 | load: 0.00298 [I 2022-10-09 17:58:28] ...trap/engines/engine.py.230: loss: 0.10056 [I 2022-10-09 17:58:28] ...optimizers/trijoint.py.151: total_norm: 7.621076 [I 2022-10-09 17:58:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 1420/2304 [I 2022-10-09 17:58:36] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:28 [I 2022-10-09 17:58:36] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00299 [I 2022-10-09 17:58:36] ...trap/engines/engine.py.230: loss: 0.13701 [I 2022-10-09 17:58:36] ...optimizers/trijoint.py.151: total_norm: 17.314749 [I 2022-10-09 17:58:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1430/2304 [I 2022-10-09 17:58:45] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:22 [I 2022-10-09 17:58:45] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00299 [I 2022-10-09 17:58:45] ...trap/engines/engine.py.230: loss: 0.14447 [I 2022-10-09 17:58:45] ...optimizers/trijoint.py.151: total_norm: 28.489407 [I 2022-10-09 17:58:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 1440/2304 [I 2022-10-09 17:58:53] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:12 [I 2022-10-09 17:58:53] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00312 [I 2022-10-09 17:58:53] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-09 17:58:53] ...optimizers/trijoint.py.151: total_norm: 19.627291 [I 2022-10-09 17:59:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1450/2304 [I 2022-10-09 17:59:02] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:04 [I 2022-10-09 17:59:02] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00304 [I 2022-10-09 17:59:02] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-09 17:59:02] ...optimizers/trijoint.py.151: total_norm: 14.717996 [I 2022-10-09 17:59:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 1460/2304 [I 2022-10-09 17:59:10] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:52 [I 2022-10-09 17:59:10] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00313 [I 2022-10-09 17:59:10] ...trap/engines/engine.py.230: loss: 0.16212 [I 2022-10-09 17:59:10] ...optimizers/trijoint.py.151: total_norm: 25.723356 [I 2022-10-09 17:59:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1470/2304 [I 2022-10-09 17:59:19] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:52 [I 2022-10-09 17:59:19] ...trap/engines/engine.py.229: process: 0.84302 | load: 0.00302 [I 2022-10-09 17:59:19] ...trap/engines/engine.py.230: loss: 0.21943 [I 2022-10-09 17:59:19] ...optimizers/trijoint.py.151: total_norm: 42.692310 [I 2022-10-09 17:59:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1480/2304 [I 2022-10-09 17:59:28] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:11:35 [I 2022-10-09 17:59:28] ...trap/engines/engine.py.229: process: 0.83912 | load: 0.00304 [I 2022-10-09 17:59:28] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-09 17:59:28] ...optimizers/trijoint.py.151: total_norm: 9.000485 [I 2022-10-09 17:59:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 1490/2304 [I 2022-10-09 17:59:36] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:31 [I 2022-10-09 17:59:36] ...trap/engines/engine.py.229: process: 0.83191 | load: 0.00300 [I 2022-10-09 17:59:36] ...trap/engines/engine.py.230: loss: 0.11866 [I 2022-10-09 17:59:36] ...optimizers/trijoint.py.151: total_norm: 7.999032 [I 2022-10-09 17:59:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1500/2304 [I 2022-10-09 17:59:45] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:26 [I 2022-10-09 17:59:45] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00311 [I 2022-10-09 17:59:45] ...trap/engines/engine.py.230: loss: 0.23273 [I 2022-10-09 17:59:45] ...optimizers/trijoint.py.151: total_norm: 22.552395 [I 2022-10-09 17:59:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 1510/2304 [I 2022-10-09 17:59:53] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:12 [I 2022-10-09 17:59:53] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00311 [I 2022-10-09 17:59:53] ...trap/engines/engine.py.230: loss: 0.12903 [I 2022-10-09 17:59:53] ...optimizers/trijoint.py.151: total_norm: 11.382998 [I 2022-10-09 18:00:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1520/2304 [I 2022-10-09 18:00:02] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:07 [I 2022-10-09 18:00:02] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00301 [I 2022-10-09 18:00:02] ...trap/engines/engine.py.230: loss: 0.16729 [I 2022-10-09 18:00:02] ...optimizers/trijoint.py.151: total_norm: 24.178902 [I 2022-10-09 18:00:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 1530/2304 [I 2022-10-09 18:00:10] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:10:58 [I 2022-10-09 18:00:10] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00310 [I 2022-10-09 18:00:10] ...trap/engines/engine.py.230: loss: 0.14212 [I 2022-10-09 18:00:10] ...optimizers/trijoint.py.151: total_norm: 16.473383 [I 2022-10-09 18:00:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1540/2304 [I 2022-10-09 18:00:19] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:49 [I 2022-10-09 18:00:19] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00298 [I 2022-10-09 18:00:19] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-09 18:00:19] ...optimizers/trijoint.py.151: total_norm: 23.727119 [I 2022-10-09 18:00:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 1550/2304 [I 2022-10-09 18:00:27] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:10:44 [I 2022-10-09 18:00:27] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00301 [I 2022-10-09 18:00:27] ...trap/engines/engine.py.230: loss: 0.20416 [I 2022-10-09 18:00:27] ...optimizers/trijoint.py.151: total_norm: 16.698063 [I 2022-10-09 18:00:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 1560/2304 [I 2022-10-09 18:00:36] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:31 [I 2022-10-09 18:00:36] ...trap/engines/engine.py.229: process: 0.84577 | load: 0.00300 [I 2022-10-09 18:00:36] ...trap/engines/engine.py.230: loss: 0.10902 [I 2022-10-09 18:00:36] ...optimizers/trijoint.py.151: total_norm: 10.118563 [I 2022-10-09 18:00:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 1570/2304 [I 2022-10-09 18:00:44] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:19 [I 2022-10-09 18:00:44] ...trap/engines/engine.py.229: process: 0.83522 | load: 0.00299 [I 2022-10-09 18:00:44] ...trap/engines/engine.py.230: loss: 0.19216 [I 2022-10-09 18:00:44] ...optimizers/trijoint.py.151: total_norm: 21.254778 [I 2022-10-09 18:00:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 1580/2304 [I 2022-10-09 18:00:53] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:13 [I 2022-10-09 18:00:53] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00304 [I 2022-10-09 18:00:53] ...trap/engines/engine.py.230: loss: 0.16423 [I 2022-10-09 18:00:53] ...optimizers/trijoint.py.151: total_norm: 18.123363 [I 2022-10-09 18:01:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 1590/2304 [I 2022-10-09 18:01:01] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:07 [I 2022-10-09 18:01:01] ...trap/engines/engine.py.229: process: 0.85626 | load: 0.00310 [I 2022-10-09 18:01:01] ...trap/engines/engine.py.230: loss: 0.13798 [I 2022-10-09 18:01:01] ...optimizers/trijoint.py.151: total_norm: 24.885027 [I 2022-10-09 18:01:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 1600/2304 [I 2022-10-09 18:01:10] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:53 [I 2022-10-09 18:01:10] ...trap/engines/engine.py.229: process: 0.82552 | load: 0.00302 [I 2022-10-09 18:01:10] ...trap/engines/engine.py.230: loss: 0.18448 [I 2022-10-09 18:01:10] ...optimizers/trijoint.py.151: total_norm: 7.121540 [I 2022-10-09 18:01:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1610/2304 [I 2022-10-09 18:01:18] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:47 [I 2022-10-09 18:01:18] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00309 [I 2022-10-09 18:01:18] ...trap/engines/engine.py.230: loss: 0.16803 [I 2022-10-09 18:01:18] ...optimizers/trijoint.py.151: total_norm: 15.769243 [I 2022-10-09 18:01:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 1620/2304 [I 2022-10-09 18:01:26] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:34 [I 2022-10-09 18:01:26] ...trap/engines/engine.py.229: process: 0.82985 | load: 0.00300 [I 2022-10-09 18:01:26] ...trap/engines/engine.py.230: loss: 0.19326 [I 2022-10-09 18:01:26] ...optimizers/trijoint.py.151: total_norm: 26.768953 [I 2022-10-09 18:01:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 1630/2304 [I 2022-10-09 18:01:35] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:36 [I 2022-10-09 18:01:35] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00307 [I 2022-10-09 18:01:35] ...trap/engines/engine.py.230: loss: 0.18640 [I 2022-10-09 18:01:35] ...optimizers/trijoint.py.151: total_norm: 22.759153 [I 2022-10-09 18:01:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 1640/2304 [I 2022-10-09 18:01:44] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:22 [I 2022-10-09 18:01:44] ...trap/engines/engine.py.229: process: 0.84302 | load: 0.00309 [I 2022-10-09 18:01:44] ...trap/engines/engine.py.230: loss: 0.19479 [I 2022-10-09 18:01:44] ...optimizers/trijoint.py.151: total_norm: 51.481136 [I 2022-10-09 18:01:52] ...trap/engines/engine.py.224: train: epoch 49 | batch 1650/2304 [I 2022-10-09 18:01:52] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:11 [I 2022-10-09 18:01:52] ...trap/engines/engine.py.229: process: 0.83695 | load: 0.00309 [I 2022-10-09 18:01:52] ...trap/engines/engine.py.230: loss: 0.13679 [I 2022-10-09 18:01:52] ...optimizers/trijoint.py.151: total_norm: 15.589284 [I 2022-10-09 18:02:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 1660/2304 [I 2022-10-09 18:02:01] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:06 [I 2022-10-09 18:02:01] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00307 [I 2022-10-09 18:02:01] ...trap/engines/engine.py.230: loss: 0.12996 [I 2022-10-09 18:02:01] ...optimizers/trijoint.py.151: total_norm: 12.553599 [I 2022-10-09 18:02:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 1670/2304 [I 2022-10-09 18:02:09] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:57 [I 2022-10-09 18:02:09] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00309 [I 2022-10-09 18:02:09] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-09 18:02:09] ...optimizers/trijoint.py.151: total_norm: 10.440186 [I 2022-10-09 18:02:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1680/2304 [I 2022-10-09 18:02:18] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:51 [I 2022-10-09 18:02:18] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00307 [I 2022-10-09 18:02:18] ...trap/engines/engine.py.230: loss: 0.18002 [I 2022-10-09 18:02:18] ...optimizers/trijoint.py.151: total_norm: 19.085129 [I 2022-10-09 18:02:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 1690/2304 [I 2022-10-09 18:02:26] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:42 [I 2022-10-09 18:02:26] ...trap/engines/engine.py.229: process: 0.84680 | load: 0.00312 [I 2022-10-09 18:02:26] ...trap/engines/engine.py.230: loss: 0.16607 [I 2022-10-09 18:02:26] ...optimizers/trijoint.py.151: total_norm: 8.819226 [I 2022-10-09 18:02:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 1700/2304 [I 2022-10-09 18:02:35] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:35 [I 2022-10-09 18:02:35] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00309 [I 2022-10-09 18:02:35] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-09 18:02:35] ...optimizers/trijoint.py.151: total_norm: 23.259607 [I 2022-10-09 18:02:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 1710/2304 [I 2022-10-09 18:02:43] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:29 [I 2022-10-09 18:02:43] ...trap/engines/engine.py.229: process: 0.85662 | load: 0.00312 [I 2022-10-09 18:02:43] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-09 18:02:43] ...optimizers/trijoint.py.151: total_norm: 17.576300 [I 2022-10-09 18:02:52] ...trap/engines/engine.py.224: train: epoch 49 | batch 1720/2304 [I 2022-10-09 18:02:52] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:19 [I 2022-10-09 18:02:52] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00315 [I 2022-10-09 18:02:52] ...trap/engines/engine.py.230: loss: 0.22836 [I 2022-10-09 18:02:52] ...optimizers/trijoint.py.151: total_norm: 19.153255 [I 2022-10-09 18:03:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1730/2304 [I 2022-10-09 18:03:00] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:12 [I 2022-10-09 18:03:00] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00309 [I 2022-10-09 18:03:00] ...trap/engines/engine.py.230: loss: 0.11955 [I 2022-10-09 18:03:00] ...optimizers/trijoint.py.151: total_norm: 10.149261 [I 2022-10-09 18:03:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 1740/2304 [I 2022-10-09 18:03:09] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:03 [I 2022-10-09 18:03:09] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00310 [I 2022-10-09 18:03:09] ...trap/engines/engine.py.230: loss: 0.18085 [I 2022-10-09 18:03:09] ...optimizers/trijoint.py.151: total_norm: 23.047663 [I 2022-10-09 18:03:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1750/2304 [I 2022-10-09 18:03:18] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:07:55 [I 2022-10-09 18:03:18] ...trap/engines/engine.py.229: process: 0.87246 | load: 0.00313 [I 2022-10-09 18:03:18] ...trap/engines/engine.py.230: loss: 0.25031 [I 2022-10-09 18:03:18] ...optimizers/trijoint.py.151: total_norm: 15.373367 [I 2022-10-09 18:03:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 1760/2304 [I 2022-10-09 18:03:26] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:45 [I 2022-10-09 18:03:26] ...trap/engines/engine.py.229: process: 0.85530 | load: 0.00344 [I 2022-10-09 18:03:26] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-09 18:03:26] ...optimizers/trijoint.py.151: total_norm: 9.936569 [I 2022-10-09 18:03:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 1770/2304 [I 2022-10-09 18:03:35] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:35 [I 2022-10-09 18:03:35] ...trap/engines/engine.py.229: process: 0.85737 | load: 0.00309 [I 2022-10-09 18:03:35] ...trap/engines/engine.py.230: loss: 0.18758 [I 2022-10-09 18:03:35] ...optimizers/trijoint.py.151: total_norm: 17.577259 [I 2022-10-09 18:03:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 1780/2304 [I 2022-10-09 18:03:43] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:25 [I 2022-10-09 18:03:43] ...trap/engines/engine.py.229: process: 0.86223 | load: 0.00311 [I 2022-10-09 18:03:43] ...trap/engines/engine.py.230: loss: 0.13935 [I 2022-10-09 18:03:43] ...optimizers/trijoint.py.151: total_norm: 20.400415 [I 2022-10-09 18:03:52] ...trap/engines/engine.py.224: train: epoch 49 | batch 1790/2304 [I 2022-10-09 18:03:52] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:19 [I 2022-10-09 18:03:52] ...trap/engines/engine.py.229: process: 0.85538 | load: 0.00311 [I 2022-10-09 18:03:52] ...trap/engines/engine.py.230: loss: 0.12017 [I 2022-10-09 18:03:52] ...optimizers/trijoint.py.151: total_norm: 10.564605 [I 2022-10-09 18:04:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1800/2304 [I 2022-10-09 18:04:00] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:09 [I 2022-10-09 18:04:00] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00313 [I 2022-10-09 18:04:00] ...trap/engines/engine.py.230: loss: 0.16027 [I 2022-10-09 18:04:00] ...optimizers/trijoint.py.151: total_norm: 10.220000 [I 2022-10-09 18:04:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 1810/2304 [I 2022-10-09 18:04:09] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:04 [I 2022-10-09 18:04:09] ...trap/engines/engine.py.229: process: 0.85743 | load: 0.00312 [I 2022-10-09 18:04:09] ...trap/engines/engine.py.230: loss: 0.26341 [I 2022-10-09 18:04:09] ...optimizers/trijoint.py.151: total_norm: 24.307795 [I 2022-10-09 18:04:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1820/2304 [I 2022-10-09 18:04:18] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:06:54 [I 2022-10-09 18:04:18] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00313 [I 2022-10-09 18:04:18] ...trap/engines/engine.py.230: loss: 0.14401 [I 2022-10-09 18:04:18] ...optimizers/trijoint.py.151: total_norm: 11.561088 [I 2022-10-09 18:04:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 1830/2304 [I 2022-10-09 18:04:26] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:44 [I 2022-10-09 18:04:26] ...trap/engines/engine.py.229: process: 0.85081 | load: 0.00313 [I 2022-10-09 18:04:26] ...trap/engines/engine.py.230: loss: 0.19243 [I 2022-10-09 18:04:26] ...optimizers/trijoint.py.151: total_norm: 70.141434 [I 2022-10-09 18:04:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 1840/2304 [I 2022-10-09 18:04:35] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:37 [I 2022-10-09 18:04:35] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00307 [I 2022-10-09 18:04:35] ...trap/engines/engine.py.230: loss: 0.16761 [I 2022-10-09 18:04:35] ...optimizers/trijoint.py.151: total_norm: 17.604887 [I 2022-10-09 18:04:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 1850/2304 [I 2022-10-09 18:04:43] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:21 [I 2022-10-09 18:04:43] ...trap/engines/engine.py.229: process: 0.83126 | load: 0.00295 [I 2022-10-09 18:04:43] ...trap/engines/engine.py.230: loss: 0.15123 [I 2022-10-09 18:04:43] ...optimizers/trijoint.py.151: total_norm: 22.034317 [I 2022-10-09 18:04:52] ...trap/engines/engine.py.224: train: epoch 49 | batch 1860/2304 [I 2022-10-09 18:04:52] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:11 [I 2022-10-09 18:04:52] ...trap/engines/engine.py.229: process: 0.83797 | load: 0.00293 [I 2022-10-09 18:04:52] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-09 18:04:52] ...optimizers/trijoint.py.151: total_norm: 14.649472 [I 2022-10-09 18:05:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1870/2304 [I 2022-10-09 18:05:00] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:09 [I 2022-10-09 18:05:00] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00292 [I 2022-10-09 18:05:00] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-09 18:05:00] ...optimizers/trijoint.py.151: total_norm: 40.246746 [I 2022-10-09 18:05:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 1880/2304 [I 2022-10-09 18:05:09] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:05:56 [I 2022-10-09 18:05:09] ...trap/engines/engine.py.229: process: 0.83578 | load: 0.00293 [I 2022-10-09 18:05:09] ...trap/engines/engine.py.230: loss: 0.14457 [I 2022-10-09 18:05:09] ...optimizers/trijoint.py.151: total_norm: 28.363583 [I 2022-10-09 18:05:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 1890/2304 [I 2022-10-09 18:05:17] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:49 [I 2022-10-09 18:05:17] ...trap/engines/engine.py.229: process: 0.83480 | load: 0.00301 [I 2022-10-09 18:05:17] ...trap/engines/engine.py.230: loss: 0.18067 [I 2022-10-09 18:05:17] ...optimizers/trijoint.py.151: total_norm: 10.894610 [I 2022-10-09 18:05:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 1900/2304 [I 2022-10-09 18:05:25] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:41 [I 2022-10-09 18:05:25] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00296 [I 2022-10-09 18:05:25] ...trap/engines/engine.py.230: loss: 0.17411 [I 2022-10-09 18:05:25] ...optimizers/trijoint.py.151: total_norm: 15.834501 [I 2022-10-09 18:05:34] ...trap/engines/engine.py.224: train: epoch 49 | batch 1910/2304 [I 2022-10-09 18:05:34] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:35 [I 2022-10-09 18:05:34] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00296 [I 2022-10-09 18:05:34] ...trap/engines/engine.py.230: loss: 0.22468 [I 2022-10-09 18:05:34] ...optimizers/trijoint.py.151: total_norm: 26.108541 [I 2022-10-09 18:05:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 1920/2304 [I 2022-10-09 18:05:43] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:26 [I 2022-10-09 18:05:43] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00300 [I 2022-10-09 18:05:43] ...trap/engines/engine.py.230: loss: 0.12025 [I 2022-10-09 18:05:43] ...optimizers/trijoint.py.151: total_norm: 14.010266 [I 2022-10-09 18:05:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 1930/2304 [I 2022-10-09 18:05:51] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:14 [I 2022-10-09 18:05:51] ...trap/engines/engine.py.229: process: 0.82892 | load: 0.00313 [I 2022-10-09 18:05:51] ...trap/engines/engine.py.230: loss: 0.13590 [I 2022-10-09 18:05:51] ...optimizers/trijoint.py.151: total_norm: 23.085426 [I 2022-10-09 18:06:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1940/2304 [I 2022-10-09 18:06:00] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:08 [I 2022-10-09 18:06:00] ...trap/engines/engine.py.229: process: 0.83968 | load: 0.00300 [I 2022-10-09 18:06:00] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-09 18:06:00] ...optimizers/trijoint.py.151: total_norm: 13.216338 [I 2022-10-09 18:06:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 1950/2304 [I 2022-10-09 18:06:08] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:04:59 [I 2022-10-09 18:06:08] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00300 [I 2022-10-09 18:06:08] ...trap/engines/engine.py.230: loss: 0.16149 [I 2022-10-09 18:06:08] ...optimizers/trijoint.py.151: total_norm: 19.730574 [I 2022-10-09 18:06:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 1960/2304 [I 2022-10-09 18:06:17] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:04:51 [I 2022-10-09 18:06:17] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00302 [I 2022-10-09 18:06:17] ...trap/engines/engine.py.230: loss: 0.16150 [I 2022-10-09 18:06:17] ...optimizers/trijoint.py.151: total_norm: 11.323028 [I 2022-10-09 18:06:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 1970/2304 [I 2022-10-09 18:06:25] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:41 [I 2022-10-09 18:06:25] ...trap/engines/engine.py.229: process: 0.83927 | load: 0.00301 [I 2022-10-09 18:06:25] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-09 18:06:25] ...optimizers/trijoint.py.151: total_norm: 15.773289 [I 2022-10-09 18:06:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 1980/2304 [I 2022-10-09 18:06:33] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:33 [I 2022-10-09 18:06:33] ...trap/engines/engine.py.229: process: 0.84008 | load: 0.00295 [I 2022-10-09 18:06:33] ...trap/engines/engine.py.230: loss: 0.22368 [I 2022-10-09 18:06:33] ...optimizers/trijoint.py.151: total_norm: 27.034716 [I 2022-10-09 18:06:42] ...trap/engines/engine.py.224: train: epoch 49 | batch 1990/2304 [I 2022-10-09 18:06:42] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:23 [I 2022-10-09 18:06:42] ...trap/engines/engine.py.229: process: 0.83186 | load: 0.00336 [I 2022-10-09 18:06:42] ...trap/engines/engine.py.230: loss: 0.18278 [I 2022-10-09 18:06:42] ...optimizers/trijoint.py.151: total_norm: 10.388189 [I 2022-10-09 18:06:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 2000/2304 [I 2022-10-09 18:06:50] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:15 [I 2022-10-09 18:06:50] ...trap/engines/engine.py.229: process: 0.83491 | load: 0.00298 [I 2022-10-09 18:06:50] ...trap/engines/engine.py.230: loss: 0.12188 [I 2022-10-09 18:06:50] ...optimizers/trijoint.py.151: total_norm: 12.910055 [I 2022-10-09 18:06:59] ...trap/engines/engine.py.224: train: epoch 49 | batch 2010/2304 [I 2022-10-09 18:06:59] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:08 [I 2022-10-09 18:06:59] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00300 [I 2022-10-09 18:06:59] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-09 18:06:59] ...optimizers/trijoint.py.151: total_norm: 14.493916 [I 2022-10-09 18:07:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 2020/2304 [I 2022-10-09 18:07:07] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:00 [I 2022-10-09 18:07:07] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00297 [I 2022-10-09 18:07:07] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-09 18:07:07] ...optimizers/trijoint.py.151: total_norm: 7.594838 [I 2022-10-09 18:07:16] ...trap/engines/engine.py.224: train: epoch 49 | batch 2030/2304 [I 2022-10-09 18:07:16] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:03:52 [I 2022-10-09 18:07:16] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00311 [I 2022-10-09 18:07:16] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-09 18:07:16] ...optimizers/trijoint.py.151: total_norm: 9.287418 [I 2022-10-09 18:07:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 2040/2304 [I 2022-10-09 18:07:24] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:44 [I 2022-10-09 18:07:24] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00311 [I 2022-10-09 18:07:24] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-09 18:07:24] ...optimizers/trijoint.py.151: total_norm: 17.342764 [I 2022-10-09 18:07:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 2050/2304 [I 2022-10-09 18:07:33] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:35 [I 2022-10-09 18:07:33] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00315 [I 2022-10-09 18:07:33] ...trap/engines/engine.py.230: loss: 0.23004 [I 2022-10-09 18:07:33] ...optimizers/trijoint.py.151: total_norm: 20.994007 [I 2022-10-09 18:07:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 2060/2304 [I 2022-10-09 18:07:41] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:27 [I 2022-10-09 18:07:41] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00312 [I 2022-10-09 18:07:41] ...trap/engines/engine.py.230: loss: 0.09961 [I 2022-10-09 18:07:41] ...optimizers/trijoint.py.151: total_norm: 6.702210 [I 2022-10-09 18:07:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 2070/2304 [I 2022-10-09 18:07:50] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:19 [I 2022-10-09 18:07:50] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00312 [I 2022-10-09 18:07:50] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-09 18:07:50] ...optimizers/trijoint.py.151: total_norm: 19.261646 [I 2022-10-09 18:07:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 2080/2304 [I 2022-10-09 18:07:58] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:10 [I 2022-10-09 18:07:58] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00312 [I 2022-10-09 18:07:58] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-09 18:07:59] ...optimizers/trijoint.py.151: total_norm: 16.946409 [I 2022-10-09 18:08:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 2090/2304 [I 2022-10-09 18:08:07] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:03:01 [I 2022-10-09 18:08:07] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00319 [I 2022-10-09 18:08:07] ...trap/engines/engine.py.230: loss: 0.19331 [I 2022-10-09 18:08:07] ...optimizers/trijoint.py.151: total_norm: 10.637554 [I 2022-10-09 18:08:16] ...trap/engines/engine.py.224: train: epoch 49 | batch 2100/2304 [I 2022-10-09 18:08:16] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:02:53 [I 2022-10-09 18:08:16] ...trap/engines/engine.py.229: process: 0.85368 | load: 0.00313 [I 2022-10-09 18:08:16] ...trap/engines/engine.py.230: loss: 0.14413 [I 2022-10-09 18:08:16] ...optimizers/trijoint.py.151: total_norm: 15.360473 [I 2022-10-09 18:08:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 2110/2304 [I 2022-10-09 18:08:24] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:02:45 [I 2022-10-09 18:08:24] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00310 [I 2022-10-09 18:08:24] ...trap/engines/engine.py.230: loss: 0.15535 [I 2022-10-09 18:08:24] ...optimizers/trijoint.py.151: total_norm: 8.379644 [I 2022-10-09 18:08:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 2120/2304 [I 2022-10-09 18:08:33] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:36 [I 2022-10-09 18:08:33] ...trap/engines/engine.py.229: process: 0.84345 | load: 0.00311 [I 2022-10-09 18:08:33] ...trap/engines/engine.py.230: loss: 0.17702 [I 2022-10-09 18:08:33] ...optimizers/trijoint.py.151: total_norm: 8.913830 [I 2022-10-09 18:08:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 2130/2304 [I 2022-10-09 18:08:41] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:28 [I 2022-10-09 18:08:41] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00312 [I 2022-10-09 18:08:41] ...trap/engines/engine.py.230: loss: 0.19926 [I 2022-10-09 18:08:41] ...optimizers/trijoint.py.151: total_norm: 13.757977 [I 2022-10-09 18:08:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 2140/2304 [I 2022-10-09 18:08:50] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:20 [I 2022-10-09 18:08:50] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00310 [I 2022-10-09 18:08:50] ...trap/engines/engine.py.230: loss: 0.13507 [I 2022-10-09 18:08:50] ...optimizers/trijoint.py.151: total_norm: 11.508588 [I 2022-10-09 18:08:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 2150/2304 [I 2022-10-09 18:08:58] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:02:11 [I 2022-10-09 18:08:58] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00310 [I 2022-10-09 18:08:58] ...trap/engines/engine.py.230: loss: 0.20566 [I 2022-10-09 18:08:58] ...optimizers/trijoint.py.151: total_norm: 19.829865 [I 2022-10-09 18:09:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 2160/2304 [I 2022-10-09 18:09:07] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:02:02 [I 2022-10-09 18:09:07] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00302 [I 2022-10-09 18:09:07] ...trap/engines/engine.py.230: loss: 0.20086 [I 2022-10-09 18:09:07] ...optimizers/trijoint.py.151: total_norm: 22.365225 [I 2022-10-09 18:09:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 2170/2304 [I 2022-10-09 18:09:15] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:01:54 [I 2022-10-09 18:09:15] ...trap/engines/engine.py.229: process: 0.86735 | load: 0.00316 [I 2022-10-09 18:09:15] ...trap/engines/engine.py.230: loss: 0.16770 [I 2022-10-09 18:09:15] ...optimizers/trijoint.py.151: total_norm: 23.332573 [I 2022-10-09 18:09:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 2180/2304 [I 2022-10-09 18:09:24] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:01:46 [I 2022-10-09 18:09:24] ...trap/engines/engine.py.229: process: 0.85968 | load: 0.00310 [I 2022-10-09 18:09:24] ...trap/engines/engine.py.230: loss: 0.25305 [I 2022-10-09 18:09:24] ...optimizers/trijoint.py.151: total_norm: 19.128460 [I 2022-10-09 18:09:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 2190/2304 [I 2022-10-09 18:09:32] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:36 [I 2022-10-09 18:09:32] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00312 [I 2022-10-09 18:09:32] ...trap/engines/engine.py.230: loss: 0.20379 [I 2022-10-09 18:09:32] ...optimizers/trijoint.py.151: total_norm: 28.755314 [I 2022-10-09 18:09:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 2200/2304 [I 2022-10-09 18:09:41] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:28 [I 2022-10-09 18:09:41] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00313 [I 2022-10-09 18:09:41] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-09 18:09:41] ...optimizers/trijoint.py.151: total_norm: 18.148584 [I 2022-10-09 18:09:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 2210/2304 [I 2022-10-09 18:09:50] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:20 [I 2022-10-09 18:09:50] ...trap/engines/engine.py.229: process: 0.85444 | load: 0.00317 [I 2022-10-09 18:09:50] ...trap/engines/engine.py.230: loss: 0.14364 [I 2022-10-09 18:09:50] ...optimizers/trijoint.py.151: total_norm: 14.780684 [I 2022-10-09 18:09:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 2220/2304 [I 2022-10-09 18:09:58] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:11 [I 2022-10-09 18:09:58] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00308 [I 2022-10-09 18:09:58] ...trap/engines/engine.py.230: loss: 0.20522 [I 2022-10-09 18:09:58] ...optimizers/trijoint.py.151: total_norm: 15.023252 [I 2022-10-09 18:10:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 2230/2304 [I 2022-10-09 18:10:07] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:02 [I 2022-10-09 18:10:07] ...trap/engines/engine.py.229: process: 0.83452 | load: 0.00306 [I 2022-10-09 18:10:07] ...trap/engines/engine.py.230: loss: 0.15807 [I 2022-10-09 18:10:07] ...optimizers/trijoint.py.151: total_norm: 17.937653 [I 2022-10-09 18:10:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 2240/2304 [I 2022-10-09 18:10:15] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:00:54 [I 2022-10-09 18:10:15] ...trap/engines/engine.py.229: process: 0.84877 | load: 0.00312 [I 2022-10-09 18:10:15] ...trap/engines/engine.py.230: loss: 0.16383 [I 2022-10-09 18:10:15] ...optimizers/trijoint.py.151: total_norm: 12.357670 [I 2022-10-09 18:10:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 2250/2304 [I 2022-10-09 18:10:24] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:45 [I 2022-10-09 18:10:24] ...trap/engines/engine.py.229: process: 0.85571 | load: 0.00320 [I 2022-10-09 18:10:24] ...trap/engines/engine.py.230: loss: 0.17887 [I 2022-10-09 18:10:24] ...optimizers/trijoint.py.151: total_norm: 16.049908 [I 2022-10-09 18:10:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 2260/2304 [I 2022-10-09 18:10:32] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:37 [I 2022-10-09 18:10:32] ...trap/engines/engine.py.229: process: 0.85158 | load: 0.00313 [I 2022-10-09 18:10:32] ...trap/engines/engine.py.230: loss: 0.22585 [I 2022-10-09 18:10:32] ...optimizers/trijoint.py.151: total_norm: 18.321026 [I 2022-10-09 18:10:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 2270/2304 [I 2022-10-09 18:10:41] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:28 [I 2022-10-09 18:10:41] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00313 [I 2022-10-09 18:10:41] ...trap/engines/engine.py.230: loss: 0.16046 [I 2022-10-09 18:10:41] ...optimizers/trijoint.py.151: total_norm: 21.887775 [I 2022-10-09 18:10:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 2280/2304 [I 2022-10-09 18:10:49] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:20 [I 2022-10-09 18:10:49] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00312 [I 2022-10-09 18:10:49] ...trap/engines/engine.py.230: loss: 0.18132 [I 2022-10-09 18:10:49] ...optimizers/trijoint.py.151: total_norm: 33.288994 [I 2022-10-09 18:10:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 2290/2304 [I 2022-10-09 18:10:58] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:11 [I 2022-10-09 18:10:58] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00309 [I 2022-10-09 18:10:58] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-09 18:10:58] ...optimizers/trijoint.py.151: total_norm: 13.685581 [I 2022-10-09 18:11:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 2300/2304 [I 2022-10-09 18:11:06] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:03 [I 2022-10-09 18:11:06] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00023 [I 2022-10-09 18:11:06] ...trap/engines/engine.py.230: loss: 0.12331 [I 2022-10-09 18:11:06] ...optimizers/trijoint.py.151: total_norm: 11.514709 [I 2022-10-09 18:11:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 2304/2304 [I 2022-10-09 18:11:10] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:00 [I 2022-10-09 18:11:10] ...trap/engines/engine.py.229: process: 0.83909 | load: 0.00019 [I 2022-10-09 18:11:10] ...trap/engines/engine.py.230: loss: 0.16713 [I 2022-10-09 18:11:10] ...optimizers/trijoint.py.151: total_norm: 23.459049 [S 2022-10-09 18:11:11] ...trap/engines/engine.py.129: train_epoch.epoch: 49 [S 2022-10-09 18:11:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.1571333704827406 [I 2022-10-09 18:11:11] ...s/criterions/triplet.py.84: margin = 0.2950000000000001 [I 2022-10-09 18:11:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 49 [I 2022-10-09 18:11:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 18:11:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 18:11:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 18:11:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 18:11:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 18:11:16] ...trap/engines/engine.py.321: eval: epoch 49 | batch 0/510 [I 2022-10-09 18:11:16] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:38 [I 2022-10-09 18:11:16] ...trap/engines/engine.py.326: process: 3.95792 | load: 3.41863 [I 2022-10-09 18:11:21] ...trap/engines/engine.py.321: eval: epoch 49 | batch 10/510 [I 2022-10-09 18:11:21] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:09 [I 2022-10-09 18:11:21] ...trap/engines/engine.py.326: process: 0.51067 | load: 0.00019 [I 2022-10-09 18:11:26] ...trap/engines/engine.py.321: eval: epoch 49 | batch 20/510 [I 2022-10-09 18:11:26] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:17 [I 2022-10-09 18:11:26] ...trap/engines/engine.py.326: process: 0.50284 | load: 0.00020 [I 2022-10-09 18:11:32] ...trap/engines/engine.py.321: eval: epoch 49 | batch 30/510 [I 2022-10-09 18:11:32] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:05:05 [I 2022-10-09 18:11:32] ...trap/engines/engine.py.326: process: 0.74809 | load: 0.00020 [I 2022-10-09 18:11:39] ...trap/engines/engine.py.321: eval: epoch 49 | batch 40/510 [I 2022-10-09 18:11:39] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:21 [I 2022-10-09 18:11:39] ...trap/engines/engine.py.326: process: 0.53808 | load: 0.00020 [I 2022-10-09 18:11:43] ...trap/engines/engine.py.321: eval: epoch 49 | batch 50/510 [I 2022-10-09 18:11:43] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:53 [I 2022-10-09 18:11:43] ...trap/engines/engine.py.326: process: 0.41710 | load: 0.00019 [I 2022-10-09 18:11:48] ...trap/engines/engine.py.321: eval: epoch 49 | batch 60/510 [I 2022-10-09 18:11:48] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:14 [I 2022-10-09 18:11:48] ...trap/engines/engine.py.326: process: 0.41653 | load: 0.00018 [I 2022-10-09 18:11:53] ...trap/engines/engine.py.321: eval: epoch 49 | batch 70/510 [I 2022-10-09 18:11:53] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:38 [I 2022-10-09 18:11:53] ...trap/engines/engine.py.326: process: 0.56089 | load: 0.00018 [I 2022-10-09 18:11:57] ...trap/engines/engine.py.321: eval: epoch 49 | batch 80/510 [I 2022-10-09 18:11:57] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:21 [I 2022-10-09 18:11:57] ...trap/engines/engine.py.326: process: 0.45865 | load: 0.00019 [I 2022-10-09 18:12:02] ...trap/engines/engine.py.321: eval: epoch 49 | batch 90/510 [I 2022-10-09 18:12:02] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:00 [I 2022-10-09 18:12:02] ...trap/engines/engine.py.326: process: 0.41863 | load: 0.00018 [I 2022-10-09 18:12:06] ...trap/engines/engine.py.321: eval: epoch 49 | batch 100/510 [I 2022-10-09 18:12:06] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:43 [I 2022-10-09 18:12:06] ...trap/engines/engine.py.326: process: 0.30823 | load: 0.00019 [I 2022-10-09 18:12:09] ...trap/engines/engine.py.321: eval: epoch 49 | batch 110/510 [I 2022-10-09 18:12:09] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:09 [I 2022-10-09 18:12:09] ...trap/engines/engine.py.326: process: 0.31683 | load: 0.00019 [I 2022-10-09 18:12:12] ...trap/engines/engine.py.321: eval: epoch 49 | batch 120/510 [I 2022-10-09 18:12:12] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:04 [I 2022-10-09 18:12:12] ...trap/engines/engine.py.326: process: 0.31882 | load: 0.00018 [I 2022-10-09 18:12:15] ...trap/engines/engine.py.321: eval: epoch 49 | batch 130/510 [I 2022-10-09 18:12:15] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-09 18:12:15] ...trap/engines/engine.py.326: process: 0.32040 | load: 0.00020 [I 2022-10-09 18:12:18] ...trap/engines/engine.py.321: eval: epoch 49 | batch 140/510 [I 2022-10-09 18:12:18] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:56 [I 2022-10-09 18:12:18] ...trap/engines/engine.py.326: process: 0.31696 | load: 0.00020 [I 2022-10-09 18:12:22] ...trap/engines/engine.py.321: eval: epoch 49 | batch 150/510 [I 2022-10-09 18:12:22] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:53 [I 2022-10-09 18:12:22] ...trap/engines/engine.py.326: process: 0.31723 | load: 0.00021 [I 2022-10-09 18:12:25] ...trap/engines/engine.py.321: eval: epoch 49 | batch 160/510 [I 2022-10-09 18:12:25] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-09 18:12:25] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00019 [I 2022-10-09 18:12:28] ...trap/engines/engine.py.321: eval: epoch 49 | batch 170/510 [I 2022-10-09 18:12:28] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:47 [I 2022-10-09 18:12:28] ...trap/engines/engine.py.326: process: 0.31722 | load: 0.00021 [I 2022-10-09 18:12:31] ...trap/engines/engine.py.321: eval: epoch 49 | batch 180/510 [I 2022-10-09 18:12:31] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-09 18:12:31] ...trap/engines/engine.py.326: process: 0.31786 | load: 0.00020 [I 2022-10-09 18:12:34] ...trap/engines/engine.py.321: eval: epoch 49 | batch 190/510 [I 2022-10-09 18:12:34] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:41 [I 2022-10-09 18:12:34] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00020 [I 2022-10-09 18:12:37] ...trap/engines/engine.py.321: eval: epoch 49 | batch 200/510 [I 2022-10-09 18:12:37] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:37 [I 2022-10-09 18:12:37] ...trap/engines/engine.py.326: process: 0.32032 | load: 0.00019 [I 2022-10-09 18:12:41] ...trap/engines/engine.py.321: eval: epoch 49 | batch 210/510 [I 2022-10-09 18:12:41] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:35 [I 2022-10-09 18:12:41] ...trap/engines/engine.py.326: process: 0.32307 | load: 0.00023 [I 2022-10-09 18:12:44] ...trap/engines/engine.py.321: eval: epoch 49 | batch 220/510 [I 2022-10-09 18:12:44] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:32 [I 2022-10-09 18:12:44] ...trap/engines/engine.py.326: process: 0.31622 | load: 0.00020 [I 2022-10-09 18:12:47] ...trap/engines/engine.py.321: eval: epoch 49 | batch 230/510 [I 2022-10-09 18:12:47] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:29 [I 2022-10-09 18:12:47] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00020 [I 2022-10-09 18:12:50] ...trap/engines/engine.py.321: eval: epoch 49 | batch 240/510 [I 2022-10-09 18:12:50] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:25 [I 2022-10-09 18:12:50] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00020 [I 2022-10-09 18:12:53] ...trap/engines/engine.py.321: eval: epoch 49 | batch 250/510 [I 2022-10-09 18:12:53] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:22 [I 2022-10-09 18:12:53] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00021 [I 2022-10-09 18:12:57] ...trap/engines/engine.py.321: eval: epoch 49 | batch 260/510 [I 2022-10-09 18:12:57] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-09 18:12:57] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00020 [I 2022-10-09 18:13:00] ...trap/engines/engine.py.321: eval: epoch 49 | batch 270/510 [I 2022-10-09 18:13:00] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-09 18:13:00] ...trap/engines/engine.py.326: process: 0.32071 | load: 0.00020 [I 2022-10-09 18:13:03] ...trap/engines/engine.py.321: eval: epoch 49 | batch 280/510 [I 2022-10-09 18:13:03] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-09 18:13:03] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00021 [I 2022-10-09 18:13:06] ...trap/engines/engine.py.321: eval: epoch 49 | batch 290/510 [I 2022-10-09 18:13:06] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:09 [I 2022-10-09 18:13:06] ...trap/engines/engine.py.326: process: 0.30391 | load: 0.00019 [I 2022-10-09 18:13:09] ...trap/engines/engine.py.321: eval: epoch 49 | batch 300/510 [I 2022-10-09 18:13:09] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:06 [I 2022-10-09 18:13:09] ...trap/engines/engine.py.326: process: 0.31187 | load: 0.00019 [I 2022-10-09 18:13:13] ...trap/engines/engine.py.321: eval: epoch 49 | batch 310/510 [I 2022-10-09 18:13:13] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-09 18:13:13] ...trap/engines/engine.py.326: process: 0.31918 | load: 0.00020 [I 2022-10-09 18:13:16] ...trap/engines/engine.py.321: eval: epoch 49 | batch 320/510 [I 2022-10-09 18:13:16] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 18:13:16] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00019 [I 2022-10-09 18:13:19] ...trap/engines/engine.py.321: eval: epoch 49 | batch 330/510 [I 2022-10-09 18:13:19] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-09 18:13:19] ...trap/engines/engine.py.326: process: 0.31572 | load: 0.00020 [I 2022-10-09 18:13:22] ...trap/engines/engine.py.321: eval: epoch 49 | batch 340/510 [I 2022-10-09 18:13:22] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-09 18:13:22] ...trap/engines/engine.py.326: process: 0.31807 | load: 0.00020 [I 2022-10-09 18:13:25] ...trap/engines/engine.py.321: eval: epoch 49 | batch 350/510 [I 2022-10-09 18:13:25] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-09 18:13:25] ...trap/engines/engine.py.326: process: 0.32097 | load: 0.00019 [I 2022-10-09 18:13:29] ...trap/engines/engine.py.321: eval: epoch 49 | batch 360/510 [I 2022-10-09 18:13:29] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-09 18:13:29] ...trap/engines/engine.py.326: process: 0.31054 | load: 0.00019 [I 2022-10-09 18:13:32] ...trap/engines/engine.py.321: eval: epoch 49 | batch 370/510 [I 2022-10-09 18:13:32] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:44 [I 2022-10-09 18:13:32] ...trap/engines/engine.py.326: process: 0.32234 | load: 0.00019 [I 2022-10-09 18:13:35] ...trap/engines/engine.py.321: eval: epoch 49 | batch 380/510 [I 2022-10-09 18:13:35] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-09 18:13:35] ...trap/engines/engine.py.326: process: 0.31321 | load: 0.00020 [I 2022-10-09 18:13:38] ...trap/engines/engine.py.321: eval: epoch 49 | batch 390/510 [I 2022-10-09 18:13:38] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-09 18:13:38] ...trap/engines/engine.py.326: process: 0.31844 | load: 0.00021 [I 2022-10-09 18:13:41] ...trap/engines/engine.py.321: eval: epoch 49 | batch 400/510 [I 2022-10-09 18:13:41] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-09 18:13:41] ...trap/engines/engine.py.326: process: 0.32233 | load: 0.00021 [I 2022-10-09 18:13:45] ...trap/engines/engine.py.321: eval: epoch 49 | batch 410/510 [I 2022-10-09 18:13:45] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:31 [I 2022-10-09 18:13:45] ...trap/engines/engine.py.326: process: 0.31434 | load: 0.00020 [I 2022-10-09 18:13:48] ...trap/engines/engine.py.321: eval: epoch 49 | batch 420/510 [I 2022-10-09 18:13:48] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:28 [I 2022-10-09 18:13:48] ...trap/engines/engine.py.326: process: 0.31175 | load: 0.00020 [I 2022-10-09 18:13:51] ...trap/engines/engine.py.321: eval: epoch 49 | batch 430/510 [I 2022-10-09 18:13:51] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-09 18:13:51] ...trap/engines/engine.py.326: process: 0.31450 | load: 0.00020 [I 2022-10-09 18:13:54] ...trap/engines/engine.py.321: eval: epoch 49 | batch 440/510 [I 2022-10-09 18:13:54] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-09 18:13:54] ...trap/engines/engine.py.326: process: 0.31284 | load: 0.00021 [I 2022-10-09 18:13:57] ...trap/engines/engine.py.321: eval: epoch 49 | batch 450/510 [I 2022-10-09 18:13:57] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-09 18:13:57] ...trap/engines/engine.py.326: process: 0.33048 | load: 0.00022 [I 2022-10-09 18:14:01] ...trap/engines/engine.py.321: eval: epoch 49 | batch 460/510 [I 2022-10-09 18:14:01] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:16 [I 2022-10-09 18:14:01] ...trap/engines/engine.py.326: process: 0.31486 | load: 0.00020 [I 2022-10-09 18:14:04] ...trap/engines/engine.py.321: eval: epoch 49 | batch 470/510 [I 2022-10-09 18:14:04] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 18:14:04] ...trap/engines/engine.py.326: process: 0.32590 | load: 0.00022 [I 2022-10-09 18:14:07] ...trap/engines/engine.py.321: eval: epoch 49 | batch 480/510 [I 2022-10-09 18:14:07] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-09 18:14:07] ...trap/engines/engine.py.326: process: 0.32327 | load: 0.00021 [I 2022-10-09 18:14:10] ...trap/engines/engine.py.321: eval: epoch 49 | batch 490/510 [I 2022-10-09 18:14:10] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-09 18:14:10] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00020 [I 2022-10-09 18:14:13] ...trap/engines/engine.py.321: eval: epoch 49 | batch 500/510 [I 2022-10-09 18:14:13] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-09 18:14:13] ...trap/engines/engine.py.326: process: 0.32183 | load: 0.00022 [I 2022-10-09 18:14:17] ...trap/engines/engine.py.321: eval: epoch 49 | batch 510/510 [I 2022-10-09 18:14:17] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-09 18:14:17] ...trap/engines/engine.py.326: process: 0.31292 | load: 0.00018 [S 2022-10-09 18:14:17] ...trap/engines/engine.py.132: eval_epoch.epoch: 49 [I 2022-10-09 18:14:17] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.717300 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901600 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933700 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.709200 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901900 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932600 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016150 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009024 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006679 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014435 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008502 [S 2022-10-09 18:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006576 [I 2022-10-09 18:14:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 18:14:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 18:14:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 18:14:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 18:14:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 18:14:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 18:14:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 18:15:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 18:15:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 50 [I 2022-10-09 18:15:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 18:15:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 0/2304 [I 2022-10-09 18:15:08] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:17:16 [I 2022-10-09 18:15:08] ...trap/engines/engine.py.229: process: 3.57504 | load: 2.61102 [I 2022-10-09 18:15:08] ...trap/engines/engine.py.230: loss: 0.15797 [I 2022-10-09 18:15:08] ...optimizers/trijoint.py.151: total_norm: 15.576317 [I 2022-10-09 18:15:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 10/2304 [I 2022-10-09 18:15:16] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:56 [I 2022-10-09 18:15:16] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00441 [I 2022-10-09 18:15:16] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-09 18:15:16] ...optimizers/trijoint.py.151: total_norm: 10.084547 [I 2022-10-09 18:15:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 20/2304 [I 2022-10-09 18:15:25] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:44 [I 2022-10-09 18:15:25] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00508 [I 2022-10-09 18:15:25] ...trap/engines/engine.py.230: loss: 0.16556 [I 2022-10-09 18:15:25] ...optimizers/trijoint.py.151: total_norm: 12.924215 [I 2022-10-09 18:15:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 30/2304 [I 2022-10-09 18:15:34] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:40 [I 2022-10-09 18:15:34] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00477 [I 2022-10-09 18:15:34] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-09 18:15:34] ...optimizers/trijoint.py.151: total_norm: 9.404189 [I 2022-10-09 18:15:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 40/2304 [I 2022-10-09 18:15:42] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:10 [I 2022-10-09 18:15:42] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00485 [I 2022-10-09 18:15:42] ...trap/engines/engine.py.230: loss: 0.24804 [I 2022-10-09 18:15:42] ...optimizers/trijoint.py.151: total_norm: 61.931202 [I 2022-10-09 18:15:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 50/2304 [I 2022-10-09 18:15:51] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:31:58 [I 2022-10-09 18:15:51] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00390 [I 2022-10-09 18:15:51] ...trap/engines/engine.py.230: loss: 0.12260 [I 2022-10-09 18:15:51] ...optimizers/trijoint.py.151: total_norm: 9.125179 [I 2022-10-09 18:15:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 60/2304 [I 2022-10-09 18:15:59] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:56 [I 2022-10-09 18:15:59] ...trap/engines/engine.py.229: process: 0.85427 | load: 0.00360 [I 2022-10-09 18:15:59] ...trap/engines/engine.py.230: loss: 0.18883 [I 2022-10-09 18:15:59] ...optimizers/trijoint.py.151: total_norm: 11.769636 [I 2022-10-09 18:16:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 70/2304 [I 2022-10-09 18:16:08] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:39 [I 2022-10-09 18:16:08] ...trap/engines/engine.py.229: process: 0.86074 | load: 0.00396 [I 2022-10-09 18:16:08] ...trap/engines/engine.py.230: loss: 0.19493 [I 2022-10-09 18:16:08] ...optimizers/trijoint.py.151: total_norm: 10.404733 [I 2022-10-09 18:16:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 80/2304 [I 2022-10-09 18:16:16] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:30 [I 2022-10-09 18:16:16] ...trap/engines/engine.py.229: process: 0.83278 | load: 0.00311 [I 2022-10-09 18:16:16] ...trap/engines/engine.py.230: loss: 0.15463 [I 2022-10-09 18:16:16] ...optimizers/trijoint.py.151: total_norm: 39.788479 [I 2022-10-09 18:16:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 90/2304 [I 2022-10-09 18:16:25] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:26 [I 2022-10-09 18:16:25] ...trap/engines/engine.py.229: process: 0.85345 | load: 0.00309 [I 2022-10-09 18:16:25] ...trap/engines/engine.py.230: loss: 0.13667 [I 2022-10-09 18:16:25] ...optimizers/trijoint.py.151: total_norm: 10.107715 [I 2022-10-09 18:16:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 100/2304 [I 2022-10-09 18:16:33] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:08 [I 2022-10-09 18:16:33] ...trap/engines/engine.py.229: process: 0.82835 | load: 0.00369 [I 2022-10-09 18:16:33] ...trap/engines/engine.py.230: loss: 0.10896 [I 2022-10-09 18:16:33] ...optimizers/trijoint.py.151: total_norm: 7.299978 [I 2022-10-09 18:16:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 110/2304 [I 2022-10-09 18:16:42] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:09 [I 2022-10-09 18:16:42] ...trap/engines/engine.py.229: process: 0.86224 | load: 0.00325 [I 2022-10-09 18:16:42] ...trap/engines/engine.py.230: loss: 0.11252 [I 2022-10-09 18:16:42] ...optimizers/trijoint.py.151: total_norm: 9.455944 [I 2022-10-09 18:16:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 120/2304 [I 2022-10-09 18:16:50] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:30:54 [I 2022-10-09 18:16:50] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00388 [I 2022-10-09 18:16:50] ...trap/engines/engine.py.230: loss: 0.14233 [I 2022-10-09 18:16:50] ...optimizers/trijoint.py.151: total_norm: 13.197048 [I 2022-10-09 18:16:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 130/2304 [I 2022-10-09 18:16:59] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:54 [I 2022-10-09 18:16:59] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00328 [I 2022-10-09 18:16:59] ...trap/engines/engine.py.230: loss: 0.16993 [I 2022-10-09 18:16:59] ...optimizers/trijoint.py.151: total_norm: 16.459881 [I 2022-10-09 18:17:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 140/2304 [I 2022-10-09 18:17:08] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:48 [I 2022-10-09 18:17:08] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00332 [I 2022-10-09 18:17:08] ...trap/engines/engine.py.230: loss: 0.12441 [I 2022-10-09 18:17:08] ...optimizers/trijoint.py.151: total_norm: 21.041908 [I 2022-10-09 18:17:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 150/2304 [I 2022-10-09 18:17:16] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:40 [I 2022-10-09 18:17:16] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00324 [I 2022-10-09 18:17:16] ...trap/engines/engine.py.230: loss: 0.15471 [I 2022-10-09 18:17:16] ...optimizers/trijoint.py.151: total_norm: 15.285174 [I 2022-10-09 18:17:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 160/2304 [I 2022-10-09 18:17:25] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:27 [I 2022-10-09 18:17:25] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00331 [I 2022-10-09 18:17:25] ...trap/engines/engine.py.230: loss: 0.18819 [I 2022-10-09 18:17:25] ...optimizers/trijoint.py.151: total_norm: 14.047379 [I 2022-10-09 18:17:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 170/2304 [I 2022-10-09 18:17:33] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:05 [I 2022-10-09 18:17:33] ...trap/engines/engine.py.229: process: 0.85146 | load: 0.00321 [I 2022-10-09 18:17:33] ...trap/engines/engine.py.230: loss: 0.16079 [I 2022-10-09 18:17:33] ...optimizers/trijoint.py.151: total_norm: 57.926636 [I 2022-10-09 18:17:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 180/2304 [I 2022-10-09 18:17:42] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:12 [I 2022-10-09 18:17:42] ...trap/engines/engine.py.229: process: 0.86540 | load: 0.00312 [I 2022-10-09 18:17:42] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-09 18:17:42] ...optimizers/trijoint.py.151: total_norm: 30.102446 [I 2022-10-09 18:17:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 190/2304 [I 2022-10-09 18:17:50] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:53 [I 2022-10-09 18:17:50] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00311 [I 2022-10-09 18:17:50] ...trap/engines/engine.py.230: loss: 0.29629 [I 2022-10-09 18:17:50] ...optimizers/trijoint.py.151: total_norm: 42.223862 [I 2022-10-09 18:17:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 200/2304 [I 2022-10-09 18:17:59] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:48 [I 2022-10-09 18:17:59] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00328 [I 2022-10-09 18:17:59] ...trap/engines/engine.py.230: loss: 0.15675 [I 2022-10-09 18:17:59] ...optimizers/trijoint.py.151: total_norm: 13.220363 [I 2022-10-09 18:18:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 210/2304 [I 2022-10-09 18:18:07] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:51 [I 2022-10-09 18:18:07] ...trap/engines/engine.py.229: process: 0.86309 | load: 0.00334 [I 2022-10-09 18:18:07] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-09 18:18:07] ...optimizers/trijoint.py.151: total_norm: 10.951564 [I 2022-10-09 18:18:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 220/2304 [I 2022-10-09 18:18:16] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:35 [I 2022-10-09 18:18:16] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00332 [I 2022-10-09 18:18:16] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-09 18:18:16] ...optimizers/trijoint.py.151: total_norm: 7.560517 [I 2022-10-09 18:18:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 230/2304 [I 2022-10-09 18:18:24] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:21 [I 2022-10-09 18:18:24] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00320 [I 2022-10-09 18:18:24] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-09 18:18:24] ...optimizers/trijoint.py.151: total_norm: 24.534725 [I 2022-10-09 18:18:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 240/2304 [I 2022-10-09 18:18:33] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:05 [I 2022-10-09 18:18:33] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00297 [I 2022-10-09 18:18:33] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-09 18:18:33] ...optimizers/trijoint.py.151: total_norm: 7.928850 [I 2022-10-09 18:18:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 250/2304 [I 2022-10-09 18:18:41] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:28:58 [I 2022-10-09 18:18:41] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00309 [I 2022-10-09 18:18:41] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-09 18:18:41] ...optimizers/trijoint.py.151: total_norm: 11.763373 [I 2022-10-09 18:18:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 260/2304 [I 2022-10-09 18:18:50] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:58 [I 2022-10-09 18:18:50] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00305 [I 2022-10-09 18:18:50] ...trap/engines/engine.py.230: loss: 0.11520 [I 2022-10-09 18:18:50] ...optimizers/trijoint.py.151: total_norm: 10.094398 [I 2022-10-09 18:18:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 270/2304 [I 2022-10-09 18:18:58] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:29 [I 2022-10-09 18:18:58] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00295 [I 2022-10-09 18:18:58] ...trap/engines/engine.py.230: loss: 0.10483 [I 2022-10-09 18:18:58] ...optimizers/trijoint.py.151: total_norm: 10.764757 [I 2022-10-09 18:19:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 280/2304 [I 2022-10-09 18:19:07] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:13 [I 2022-10-09 18:19:07] ...trap/engines/engine.py.229: process: 0.82895 | load: 0.00297 [I 2022-10-09 18:19:07] ...trap/engines/engine.py.230: loss: 0.20230 [I 2022-10-09 18:19:07] ...optimizers/trijoint.py.151: total_norm: 25.333723 [I 2022-10-09 18:19:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 290/2304 [I 2022-10-09 18:19:15] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:10 [I 2022-10-09 18:19:15] ...trap/engines/engine.py.229: process: 0.83523 | load: 0.00293 [I 2022-10-09 18:19:15] ...trap/engines/engine.py.230: loss: 0.22999 [I 2022-10-09 18:19:15] ...optimizers/trijoint.py.151: total_norm: 33.208660 [I 2022-10-09 18:19:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 300/2304 [I 2022-10-09 18:19:23] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:13 [I 2022-10-09 18:19:23] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00292 [I 2022-10-09 18:19:23] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-09 18:19:23] ...optimizers/trijoint.py.151: total_norm: 14.618253 [I 2022-10-09 18:19:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 310/2304 [I 2022-10-09 18:19:32] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:27:54 [I 2022-10-09 18:19:32] ...trap/engines/engine.py.229: process: 0.83262 | load: 0.00293 [I 2022-10-09 18:19:32] ...trap/engines/engine.py.230: loss: 0.13711 [I 2022-10-09 18:19:32] ...optimizers/trijoint.py.151: total_norm: 9.753991 [I 2022-10-09 18:19:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 320/2304 [I 2022-10-09 18:19:40] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:27:54 [I 2022-10-09 18:19:40] ...trap/engines/engine.py.229: process: 0.85146 | load: 0.00298 [I 2022-10-09 18:19:40] ...trap/engines/engine.py.230: loss: 0.18431 [I 2022-10-09 18:19:40] ...optimizers/trijoint.py.151: total_norm: 24.498762 [I 2022-10-09 18:19:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 330/2304 [I 2022-10-09 18:19:49] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:52 [I 2022-10-09 18:19:49] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00295 [I 2022-10-09 18:19:49] ...trap/engines/engine.py.230: loss: 0.19231 [I 2022-10-09 18:19:49] ...optimizers/trijoint.py.151: total_norm: 16.756821 [I 2022-10-09 18:19:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 340/2304 [I 2022-10-09 18:19:57] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:41 [I 2022-10-09 18:19:57] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00296 [I 2022-10-09 18:19:57] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-09 18:19:57] ...optimizers/trijoint.py.151: total_norm: 15.714688 [I 2022-10-09 18:20:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 350/2304 [I 2022-10-09 18:20:06] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:24 [I 2022-10-09 18:20:06] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00290 [I 2022-10-09 18:20:06] ...trap/engines/engine.py.230: loss: 0.18905 [I 2022-10-09 18:20:06] ...optimizers/trijoint.py.151: total_norm: 12.472969 [I 2022-10-09 18:20:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 360/2304 [I 2022-10-09 18:20:14] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:22 [I 2022-10-09 18:20:14] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00294 [I 2022-10-09 18:20:14] ...trap/engines/engine.py.230: loss: 0.12600 [I 2022-10-09 18:20:14] ...optimizers/trijoint.py.151: total_norm: 11.785755 [I 2022-10-09 18:20:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 370/2304 [I 2022-10-09 18:20:23] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:16 [I 2022-10-09 18:20:23] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00293 [I 2022-10-09 18:20:23] ...trap/engines/engine.py.230: loss: 0.23194 [I 2022-10-09 18:20:23] ...optimizers/trijoint.py.151: total_norm: 21.689535 [I 2022-10-09 18:20:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 380/2304 [I 2022-10-09 18:20:31] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:04 [I 2022-10-09 18:20:31] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00307 [I 2022-10-09 18:20:31] ...trap/engines/engine.py.230: loss: 0.17030 [I 2022-10-09 18:20:31] ...optimizers/trijoint.py.151: total_norm: 17.545158 [I 2022-10-09 18:20:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 390/2304 [I 2022-10-09 18:20:40] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:26:55 [I 2022-10-09 18:20:40] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00322 [I 2022-10-09 18:20:40] ...trap/engines/engine.py.230: loss: 0.17284 [I 2022-10-09 18:20:40] ...optimizers/trijoint.py.151: total_norm: 21.581251 [I 2022-10-09 18:20:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 400/2304 [I 2022-10-09 18:20:48] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:53 [I 2022-10-09 18:20:48] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00301 [I 2022-10-09 18:20:48] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-09 18:20:48] ...optimizers/trijoint.py.151: total_norm: 12.552825 [I 2022-10-09 18:20:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 410/2304 [I 2022-10-09 18:20:57] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:47 [I 2022-10-09 18:20:57] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00313 [I 2022-10-09 18:20:57] ...trap/engines/engine.py.230: loss: 0.08110 [I 2022-10-09 18:20:57] ...optimizers/trijoint.py.151: total_norm: 7.196619 [I 2022-10-09 18:21:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 420/2304 [I 2022-10-09 18:21:05] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:34 [I 2022-10-09 18:21:05] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00297 [I 2022-10-09 18:21:05] ...trap/engines/engine.py.230: loss: 0.12775 [I 2022-10-09 18:21:05] ...optimizers/trijoint.py.151: total_norm: 17.660206 [I 2022-10-09 18:21:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 430/2304 [I 2022-10-09 18:21:14] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:21 [I 2022-10-09 18:21:14] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00316 [I 2022-10-09 18:21:14] ...trap/engines/engine.py.230: loss: 0.22154 [I 2022-10-09 18:21:14] ...optimizers/trijoint.py.151: total_norm: 29.162279 [I 2022-10-09 18:21:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 440/2304 [I 2022-10-09 18:21:22] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:14 [I 2022-10-09 18:21:22] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00300 [I 2022-10-09 18:21:22] ...trap/engines/engine.py.230: loss: 0.16152 [I 2022-10-09 18:21:22] ...optimizers/trijoint.py.151: total_norm: 12.177864 [I 2022-10-09 18:21:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 450/2304 [I 2022-10-09 18:21:30] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:09 [I 2022-10-09 18:21:30] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00299 [I 2022-10-09 18:21:30] ...trap/engines/engine.py.230: loss: 0.16565 [I 2022-10-09 18:21:30] ...optimizers/trijoint.py.151: total_norm: 19.736191 [I 2022-10-09 18:21:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 460/2304 [I 2022-10-09 18:21:39] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:25:53 [I 2022-10-09 18:21:39] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00300 [I 2022-10-09 18:21:39] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-09 18:21:39] ...optimizers/trijoint.py.151: total_norm: 11.106886 [I 2022-10-09 18:21:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 470/2304 [I 2022-10-09 18:21:47] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:59 [I 2022-10-09 18:21:47] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00307 [I 2022-10-09 18:21:47] ...trap/engines/engine.py.230: loss: 0.17551 [I 2022-10-09 18:21:47] ...optimizers/trijoint.py.151: total_norm: 14.887588 [I 2022-10-09 18:21:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 480/2304 [I 2022-10-09 18:21:56] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:25:50 [I 2022-10-09 18:21:56] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00311 [I 2022-10-09 18:21:56] ...trap/engines/engine.py.230: loss: 0.12886 [I 2022-10-09 18:21:56] ...optimizers/trijoint.py.151: total_norm: 14.497246 [I 2022-10-09 18:22:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 490/2304 [I 2022-10-09 18:22:05] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:52 [I 2022-10-09 18:22:05] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00321 [I 2022-10-09 18:22:05] ...trap/engines/engine.py.230: loss: 0.16595 [I 2022-10-09 18:22:05] ...optimizers/trijoint.py.151: total_norm: 9.618567 [I 2022-10-09 18:22:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 500/2304 [I 2022-10-09 18:22:13] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:41 [I 2022-10-09 18:22:13] ...trap/engines/engine.py.229: process: 0.85826 | load: 0.00319 [I 2022-10-09 18:22:13] ...trap/engines/engine.py.230: loss: 0.20424 [I 2022-10-09 18:22:13] ...optimizers/trijoint.py.151: total_norm: 21.010767 [I 2022-10-09 18:22:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 510/2304 [I 2022-10-09 18:22:22] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:34 [I 2022-10-09 18:22:22] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00315 [I 2022-10-09 18:22:22] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-09 18:22:22] ...optimizers/trijoint.py.151: total_norm: 18.270451 [I 2022-10-09 18:22:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 520/2304 [I 2022-10-09 18:22:30] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:30 [I 2022-10-09 18:22:30] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00336 [I 2022-10-09 18:22:30] ...trap/engines/engine.py.230: loss: 0.19532 [I 2022-10-09 18:22:30] ...optimizers/trijoint.py.151: total_norm: 42.622219 [I 2022-10-09 18:22:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 530/2304 [I 2022-10-09 18:22:39] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:23 [I 2022-10-09 18:22:39] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00312 [I 2022-10-09 18:22:39] ...trap/engines/engine.py.230: loss: 0.13500 [I 2022-10-09 18:22:39] ...optimizers/trijoint.py.151: total_norm: 14.832297 [I 2022-10-09 18:22:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 540/2304 [I 2022-10-09 18:22:47] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:04 [I 2022-10-09 18:22:47] ...trap/engines/engine.py.229: process: 0.84418 | load: 0.00310 [I 2022-10-09 18:22:47] ...trap/engines/engine.py.230: loss: 0.15410 [I 2022-10-09 18:22:47] ...optimizers/trijoint.py.151: total_norm: 12.056180 [I 2022-10-09 18:22:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 550/2304 [I 2022-10-09 18:22:56] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:51 [I 2022-10-09 18:22:56] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00312 [I 2022-10-09 18:22:56] ...trap/engines/engine.py.230: loss: 0.22455 [I 2022-10-09 18:22:56] ...optimizers/trijoint.py.151: total_norm: 16.164444 [I 2022-10-09 18:23:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 560/2304 [I 2022-10-09 18:23:04] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:44 [I 2022-10-09 18:23:04] ...trap/engines/engine.py.229: process: 0.86097 | load: 0.00312 [I 2022-10-09 18:23:04] ...trap/engines/engine.py.230: loss: 0.22787 [I 2022-10-09 18:23:04] ...optimizers/trijoint.py.151: total_norm: 11.004539 [I 2022-10-09 18:23:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 570/2304 [I 2022-10-09 18:23:13] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:34 [I 2022-10-09 18:23:13] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00313 [I 2022-10-09 18:23:13] ...trap/engines/engine.py.230: loss: 0.12783 [I 2022-10-09 18:23:13] ...optimizers/trijoint.py.151: total_norm: 9.969872 [I 2022-10-09 18:23:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 580/2304 [I 2022-10-09 18:23:22] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:33 [I 2022-10-09 18:23:22] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00311 [I 2022-10-09 18:23:22] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-09 18:23:22] ...optimizers/trijoint.py.151: total_norm: 12.959631 [I 2022-10-09 18:23:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 590/2304 [I 2022-10-09 18:23:30] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:26 [I 2022-10-09 18:23:30] ...trap/engines/engine.py.229: process: 0.86182 | load: 0.00321 [I 2022-10-09 18:23:30] ...trap/engines/engine.py.230: loss: 0.18021 [I 2022-10-09 18:23:30] ...optimizers/trijoint.py.151: total_norm: 13.541975 [I 2022-10-09 18:23:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 600/2304 [I 2022-10-09 18:23:39] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:23 [I 2022-10-09 18:23:39] ...trap/engines/engine.py.229: process: 0.87089 | load: 0.00311 [I 2022-10-09 18:23:39] ...trap/engines/engine.py.230: loss: 0.20570 [I 2022-10-09 18:23:39] ...optimizers/trijoint.py.151: total_norm: 24.799925 [I 2022-10-09 18:23:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 610/2304 [I 2022-10-09 18:23:47] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:12 [I 2022-10-09 18:23:47] ...trap/engines/engine.py.229: process: 0.84877 | load: 0.00315 [I 2022-10-09 18:23:47] ...trap/engines/engine.py.230: loss: 0.14549 [I 2022-10-09 18:23:47] ...optimizers/trijoint.py.151: total_norm: 11.264342 [I 2022-10-09 18:23:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 620/2304 [I 2022-10-09 18:23:56] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:51 [I 2022-10-09 18:23:56] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00310 [I 2022-10-09 18:23:56] ...trap/engines/engine.py.230: loss: 0.20515 [I 2022-10-09 18:23:56] ...optimizers/trijoint.py.151: total_norm: 27.932178 [I 2022-10-09 18:24:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 630/2304 [I 2022-10-09 18:24:04] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:42 [I 2022-10-09 18:24:04] ...trap/engines/engine.py.229: process: 0.85352 | load: 0.00310 [I 2022-10-09 18:24:04] ...trap/engines/engine.py.230: loss: 0.21074 [I 2022-10-09 18:24:04] ...optimizers/trijoint.py.151: total_norm: 20.215473 [I 2022-10-09 18:24:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 640/2304 [I 2022-10-09 18:24:13] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:36 [I 2022-10-09 18:24:13] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00319 [I 2022-10-09 18:24:13] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-09 18:24:13] ...optimizers/trijoint.py.151: total_norm: 8.898833 [I 2022-10-09 18:24:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 650/2304 [I 2022-10-09 18:24:22] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:35 [I 2022-10-09 18:24:22] ...trap/engines/engine.py.229: process: 0.85323 | load: 0.00312 [I 2022-10-09 18:24:22] ...trap/engines/engine.py.230: loss: 0.16717 [I 2022-10-09 18:24:22] ...optimizers/trijoint.py.151: total_norm: 12.850512 [I 2022-10-09 18:24:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 660/2304 [I 2022-10-09 18:24:30] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:23 [I 2022-10-09 18:24:30] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00316 [I 2022-10-09 18:24:30] ...trap/engines/engine.py.230: loss: 0.18136 [I 2022-10-09 18:24:30] ...optimizers/trijoint.py.151: total_norm: 15.187741 [I 2022-10-09 18:24:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 670/2304 [I 2022-10-09 18:24:39] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:13 [I 2022-10-09 18:24:39] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00313 [I 2022-10-09 18:24:39] ...trap/engines/engine.py.230: loss: 0.14174 [I 2022-10-09 18:24:39] ...optimizers/trijoint.py.151: total_norm: 9.085085 [I 2022-10-09 18:24:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 680/2304 [I 2022-10-09 18:24:47] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:13 [I 2022-10-09 18:24:47] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00317 [I 2022-10-09 18:24:47] ...trap/engines/engine.py.230: loss: 0.19751 [I 2022-10-09 18:24:47] ...optimizers/trijoint.py.151: total_norm: 30.153234 [I 2022-10-09 18:24:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 690/2304 [I 2022-10-09 18:24:56] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:23:01 [I 2022-10-09 18:24:56] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00312 [I 2022-10-09 18:24:56] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-09 18:24:56] ...optimizers/trijoint.py.151: total_norm: 6.319338 [I 2022-10-09 18:25:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 700/2304 [I 2022-10-09 18:25:05] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:59 [I 2022-10-09 18:25:05] ...trap/engines/engine.py.229: process: 0.86493 | load: 0.00311 [I 2022-10-09 18:25:05] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-09 18:25:05] ...optimizers/trijoint.py.151: total_norm: 12.117100 [I 2022-10-09 18:25:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 710/2304 [I 2022-10-09 18:25:13] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:42 [I 2022-10-09 18:25:13] ...trap/engines/engine.py.229: process: 0.86407 | load: 0.00312 [I 2022-10-09 18:25:13] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-09 18:25:13] ...optimizers/trijoint.py.151: total_norm: 25.100109 [I 2022-10-09 18:25:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 720/2304 [I 2022-10-09 18:25:22] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:29 [I 2022-10-09 18:25:22] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00311 [I 2022-10-09 18:25:22] ...trap/engines/engine.py.230: loss: 0.10832 [I 2022-10-09 18:25:22] ...optimizers/trijoint.py.151: total_norm: 7.993517 [I 2022-10-09 18:25:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 730/2304 [I 2022-10-09 18:25:30] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:22 [I 2022-10-09 18:25:30] ...trap/engines/engine.py.229: process: 0.83798 | load: 0.00324 [I 2022-10-09 18:25:30] ...trap/engines/engine.py.230: loss: 0.17417 [I 2022-10-09 18:25:30] ...optimizers/trijoint.py.151: total_norm: 15.018663 [I 2022-10-09 18:25:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 740/2304 [I 2022-10-09 18:25:39] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:13 [I 2022-10-09 18:25:39] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00296 [I 2022-10-09 18:25:39] ...trap/engines/engine.py.230: loss: 0.20442 [I 2022-10-09 18:25:39] ...optimizers/trijoint.py.151: total_norm: 21.046879 [I 2022-10-09 18:25:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 750/2304 [I 2022-10-09 18:25:47] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:57 [I 2022-10-09 18:25:47] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00302 [I 2022-10-09 18:25:47] ...trap/engines/engine.py.230: loss: 0.17808 [I 2022-10-09 18:25:47] ...optimizers/trijoint.py.151: total_norm: 32.389744 [I 2022-10-09 18:25:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 760/2304 [I 2022-10-09 18:25:56] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:41 [I 2022-10-09 18:25:56] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00297 [I 2022-10-09 18:25:56] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-09 18:25:56] ...optimizers/trijoint.py.151: total_norm: 8.066204 [I 2022-10-09 18:26:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 770/2304 [I 2022-10-09 18:26:04] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:43 [I 2022-10-09 18:26:04] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00315 [I 2022-10-09 18:26:04] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-09 18:26:04] ...optimizers/trijoint.py.151: total_norm: 15.923856 [I 2022-10-09 18:26:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 780/2304 [I 2022-10-09 18:26:13] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:24 [I 2022-10-09 18:26:13] ...trap/engines/engine.py.229: process: 0.83115 | load: 0.00300 [I 2022-10-09 18:26:13] ...trap/engines/engine.py.230: loss: 0.13981 [I 2022-10-09 18:26:13] ...optimizers/trijoint.py.151: total_norm: 14.423230 [I 2022-10-09 18:26:21] ...trap/engines/engine.py.224: train: epoch 50 | batch 790/2304 [I 2022-10-09 18:26:21] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:23 [I 2022-10-09 18:26:21] ...trap/engines/engine.py.229: process: 0.85248 | load: 0.00298 [I 2022-10-09 18:26:21] ...trap/engines/engine.py.230: loss: 0.14792 [I 2022-10-09 18:26:21] ...optimizers/trijoint.py.151: total_norm: 12.512824 [I 2022-10-09 18:26:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 800/2304 [I 2022-10-09 18:26:30] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:14 [I 2022-10-09 18:26:30] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00299 [I 2022-10-09 18:26:30] ...trap/engines/engine.py.230: loss: 0.15037 [I 2022-10-09 18:26:30] ...optimizers/trijoint.py.151: total_norm: 16.772778 [I 2022-10-09 18:26:38] ...trap/engines/engine.py.224: train: epoch 50 | batch 810/2304 [I 2022-10-09 18:26:38] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:00 [I 2022-10-09 18:26:38] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00298 [I 2022-10-09 18:26:38] ...trap/engines/engine.py.230: loss: 0.14108 [I 2022-10-09 18:26:38] ...optimizers/trijoint.py.151: total_norm: 11.897099 [I 2022-10-09 18:26:46] ...trap/engines/engine.py.224: train: epoch 50 | batch 820/2304 [I 2022-10-09 18:26:46] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:51 [I 2022-10-09 18:26:46] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00302 [I 2022-10-09 18:26:46] ...trap/engines/engine.py.230: loss: 0.21278 [I 2022-10-09 18:26:46] ...optimizers/trijoint.py.151: total_norm: 14.572054 [I 2022-10-09 18:26:55] ...trap/engines/engine.py.224: train: epoch 50 | batch 830/2304 [I 2022-10-09 18:26:55] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:45 [I 2022-10-09 18:26:55] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00300 [I 2022-10-09 18:26:55] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-09 18:26:55] ...optimizers/trijoint.py.151: total_norm: 10.873253 [I 2022-10-09 18:27:03] ...trap/engines/engine.py.224: train: epoch 50 | batch 840/2304 [I 2022-10-09 18:27:03] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:33 [I 2022-10-09 18:27:03] ...trap/engines/engine.py.229: process: 0.84887 | load: 0.00303 [I 2022-10-09 18:27:03] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-09 18:27:03] ...optimizers/trijoint.py.151: total_norm: 8.875284 [I 2022-10-09 18:27:12] ...trap/engines/engine.py.224: train: epoch 50 | batch 850/2304 [I 2022-10-09 18:27:12] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:32 [I 2022-10-09 18:27:12] ...trap/engines/engine.py.229: process: 0.84953 | load: 0.00298 [I 2022-10-09 18:27:12] ...trap/engines/engine.py.230: loss: 0.10088 [I 2022-10-09 18:27:12] ...optimizers/trijoint.py.151: total_norm: 6.342680 [I 2022-10-09 18:27:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 860/2304 [I 2022-10-09 18:27:20] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:20 [I 2022-10-09 18:27:20] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00301 [I 2022-10-09 18:27:20] ...trap/engines/engine.py.230: loss: 0.16122 [I 2022-10-09 18:27:20] ...optimizers/trijoint.py.151: total_norm: 17.521183 [I 2022-10-09 18:27:29] ...trap/engines/engine.py.224: train: epoch 50 | batch 870/2304 [I 2022-10-09 18:27:29] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:19 [I 2022-10-09 18:27:29] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00299 [I 2022-10-09 18:27:29] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-09 18:27:29] ...optimizers/trijoint.py.151: total_norm: 9.867164 [I 2022-10-09 18:27:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 880/2304 [I 2022-10-09 18:27:37] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:06 [I 2022-10-09 18:27:37] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00310 [I 2022-10-09 18:27:37] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-09 18:27:37] ...optimizers/trijoint.py.151: total_norm: 19.416859 [I 2022-10-09 18:27:46] ...trap/engines/engine.py.224: train: epoch 50 | batch 890/2304 [I 2022-10-09 18:27:46] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:55 [I 2022-10-09 18:27:46] ...trap/engines/engine.py.229: process: 0.83164 | load: 0.00299 [I 2022-10-09 18:27:46] ...trap/engines/engine.py.230: loss: 0.10502 [I 2022-10-09 18:27:46] ...optimizers/trijoint.py.151: total_norm: 12.389124 [I 2022-10-09 18:27:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 900/2304 [I 2022-10-09 18:27:54] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:47 [I 2022-10-09 18:27:54] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00294 [I 2022-10-09 18:27:54] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-09 18:27:54] ...optimizers/trijoint.py.151: total_norm: 11.534723 [I 2022-10-09 18:28:03] ...trap/engines/engine.py.224: train: epoch 50 | batch 910/2304 [I 2022-10-09 18:28:03] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:37 [I 2022-10-09 18:28:03] ...trap/engines/engine.py.229: process: 0.83212 | load: 0.00298 [I 2022-10-09 18:28:03] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-09 18:28:03] ...optimizers/trijoint.py.151: total_norm: 16.634853 [I 2022-10-09 18:28:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 920/2304 [I 2022-10-09 18:28:11] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:32 [I 2022-10-09 18:28:11] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00293 [I 2022-10-09 18:28:11] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-09 18:28:11] ...optimizers/trijoint.py.151: total_norm: 12.381293 [I 2022-10-09 18:28:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 930/2304 [I 2022-10-09 18:28:20] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:24 [I 2022-10-09 18:28:20] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00299 [I 2022-10-09 18:28:20] ...trap/engines/engine.py.230: loss: 0.22354 [I 2022-10-09 18:28:20] ...optimizers/trijoint.py.151: total_norm: 25.360893 [I 2022-10-09 18:28:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 940/2304 [I 2022-10-09 18:28:28] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:24 [I 2022-10-09 18:28:28] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00300 [I 2022-10-09 18:28:28] ...trap/engines/engine.py.230: loss: 0.19464 [I 2022-10-09 18:28:28] ...optimizers/trijoint.py.151: total_norm: 27.486973 [I 2022-10-09 18:28:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 950/2304 [I 2022-10-09 18:28:37] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:09 [I 2022-10-09 18:28:37] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00315 [I 2022-10-09 18:28:37] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-09 18:28:37] ...optimizers/trijoint.py.151: total_norm: 12.022431 [I 2022-10-09 18:28:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 960/2304 [I 2022-10-09 18:28:45] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:57 [I 2022-10-09 18:28:45] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00313 [I 2022-10-09 18:28:45] ...trap/engines/engine.py.230: loss: 0.17832 [I 2022-10-09 18:28:45] ...optimizers/trijoint.py.151: total_norm: 15.965535 [I 2022-10-09 18:28:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 970/2304 [I 2022-10-09 18:28:54] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:58 [I 2022-10-09 18:28:54] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00313 [I 2022-10-09 18:28:54] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-09 18:28:54] ...optimizers/trijoint.py.151: total_norm: 18.896631 [I 2022-10-09 18:29:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 980/2304 [I 2022-10-09 18:29:02] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:50 [I 2022-10-09 18:29:02] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00310 [I 2022-10-09 18:29:02] ...trap/engines/engine.py.230: loss: 0.12520 [I 2022-10-09 18:29:02] ...optimizers/trijoint.py.151: total_norm: 14.725615 [I 2022-10-09 18:29:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 990/2304 [I 2022-10-09 18:29:11] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:44 [I 2022-10-09 18:29:11] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00310 [I 2022-10-09 18:29:11] ...trap/engines/engine.py.230: loss: 0.11296 [I 2022-10-09 18:29:11] ...optimizers/trijoint.py.151: total_norm: 13.432988 [I 2022-10-09 18:29:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 1000/2304 [I 2022-10-09 18:29:20] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:32 [I 2022-10-09 18:29:20] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00313 [I 2022-10-09 18:29:20] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-09 18:29:20] ...optimizers/trijoint.py.151: total_norm: 8.961158 [I 2022-10-09 18:29:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 1010/2304 [I 2022-10-09 18:29:28] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:25 [I 2022-10-09 18:29:28] ...trap/engines/engine.py.229: process: 0.86214 | load: 0.00313 [I 2022-10-09 18:29:28] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-09 18:29:28] ...optimizers/trijoint.py.151: total_norm: 20.499952 [I 2022-10-09 18:29:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 1020/2304 [I 2022-10-09 18:29:37] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:14 [I 2022-10-09 18:29:37] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00311 [I 2022-10-09 18:29:37] ...trap/engines/engine.py.230: loss: 0.18104 [I 2022-10-09 18:29:37] ...optimizers/trijoint.py.151: total_norm: 25.767660 [I 2022-10-09 18:29:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1030/2304 [I 2022-10-09 18:29:45] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:09 [I 2022-10-09 18:29:45] ...trap/engines/engine.py.229: process: 0.85027 | load: 0.00312 [I 2022-10-09 18:29:45] ...trap/engines/engine.py.230: loss: 0.14075 [I 2022-10-09 18:29:45] ...optimizers/trijoint.py.151: total_norm: 15.725206 [I 2022-10-09 18:29:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 1040/2304 [I 2022-10-09 18:29:54] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:59 [I 2022-10-09 18:29:54] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00312 [I 2022-10-09 18:29:54] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-09 18:29:54] ...optimizers/trijoint.py.151: total_norm: 9.990447 [I 2022-10-09 18:30:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 1050/2304 [I 2022-10-09 18:30:02] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:49 [I 2022-10-09 18:30:02] ...trap/engines/engine.py.229: process: 0.84530 | load: 0.00313 [I 2022-10-09 18:30:02] ...trap/engines/engine.py.230: loss: 0.15796 [I 2022-10-09 18:30:02] ...optimizers/trijoint.py.151: total_norm: 13.626236 [I 2022-10-09 18:30:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 1060/2304 [I 2022-10-09 18:30:11] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:45 [I 2022-10-09 18:30:11] ...trap/engines/engine.py.229: process: 0.86645 | load: 0.00316 [I 2022-10-09 18:30:11] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-09 18:30:11] ...optimizers/trijoint.py.151: total_norm: 13.431052 [I 2022-10-09 18:30:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 1070/2304 [I 2022-10-09 18:30:20] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:40 [I 2022-10-09 18:30:20] ...trap/engines/engine.py.229: process: 0.87284 | load: 0.00316 [I 2022-10-09 18:30:20] ...trap/engines/engine.py.230: loss: 0.13558 [I 2022-10-09 18:30:20] ...optimizers/trijoint.py.151: total_norm: 8.391591 [I 2022-10-09 18:30:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 1080/2304 [I 2022-10-09 18:30:28] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:25 [I 2022-10-09 18:30:28] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00312 [I 2022-10-09 18:30:28] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-09 18:30:28] ...optimizers/trijoint.py.151: total_norm: 19.333715 [I 2022-10-09 18:30:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 1090/2304 [I 2022-10-09 18:30:37] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:16 [I 2022-10-09 18:30:37] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00313 [I 2022-10-09 18:30:37] ...trap/engines/engine.py.230: loss: 0.10397 [I 2022-10-09 18:30:37] ...optimizers/trijoint.py.151: total_norm: 11.603603 [I 2022-10-09 18:30:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1100/2304 [I 2022-10-09 18:30:45] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:05 [I 2022-10-09 18:30:45] ...trap/engines/engine.py.229: process: 0.84163 | load: 0.00315 [I 2022-10-09 18:30:45] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-09 18:30:45] ...optimizers/trijoint.py.151: total_norm: 10.385903 [I 2022-10-09 18:30:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 1110/2304 [I 2022-10-09 18:30:54] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:58 [I 2022-10-09 18:30:54] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00311 [I 2022-10-09 18:30:54] ...trap/engines/engine.py.230: loss: 0.13017 [I 2022-10-09 18:30:54] ...optimizers/trijoint.py.151: total_norm: 10.577372 [I 2022-10-09 18:31:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 1120/2304 [I 2022-10-09 18:31:02] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:50 [I 2022-10-09 18:31:02] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00311 [I 2022-10-09 18:31:02] ...trap/engines/engine.py.230: loss: 0.11378 [I 2022-10-09 18:31:02] ...optimizers/trijoint.py.151: total_norm: 7.495520 [I 2022-10-09 18:31:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 1130/2304 [I 2022-10-09 18:31:11] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:42 [I 2022-10-09 18:31:11] ...trap/engines/engine.py.229: process: 0.85108 | load: 0.00311 [I 2022-10-09 18:31:11] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-09 18:31:11] ...optimizers/trijoint.py.151: total_norm: 13.544238 [I 2022-10-09 18:31:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 1140/2304 [I 2022-10-09 18:31:19] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:33 [I 2022-10-09 18:31:19] ...trap/engines/engine.py.229: process: 0.86500 | load: 0.00314 [I 2022-10-09 18:31:19] ...trap/engines/engine.py.230: loss: 0.12847 [I 2022-10-09 18:31:19] ...optimizers/trijoint.py.151: total_norm: 18.593828 [I 2022-10-09 18:31:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 1150/2304 [I 2022-10-09 18:31:28] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:23 [I 2022-10-09 18:31:28] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00316 [I 2022-10-09 18:31:28] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-09 18:31:28] ...optimizers/trijoint.py.151: total_norm: 10.789115 [I 2022-10-09 18:31:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 1160/2304 [I 2022-10-09 18:31:36] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:13 [I 2022-10-09 18:31:36] ...trap/engines/engine.py.229: process: 0.85648 | load: 0.00320 [I 2022-10-09 18:31:36] ...trap/engines/engine.py.230: loss: 0.20863 [I 2022-10-09 18:31:36] ...optimizers/trijoint.py.151: total_norm: 27.341665 [I 2022-10-09 18:31:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1170/2304 [I 2022-10-09 18:31:45] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:04 [I 2022-10-09 18:31:45] ...trap/engines/engine.py.229: process: 0.86489 | load: 0.00308 [I 2022-10-09 18:31:45] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-09 18:31:45] ...optimizers/trijoint.py.151: total_norm: 30.971432 [I 2022-10-09 18:31:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 1180/2304 [I 2022-10-09 18:31:53] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:57 [I 2022-10-09 18:31:53] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00311 [I 2022-10-09 18:31:53] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-09 18:31:53] ...optimizers/trijoint.py.151: total_norm: 12.109714 [I 2022-10-09 18:32:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 1190/2304 [I 2022-10-09 18:32:02] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:51 [I 2022-10-09 18:32:02] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00316 [I 2022-10-09 18:32:02] ...trap/engines/engine.py.230: loss: 0.11020 [I 2022-10-09 18:32:02] ...optimizers/trijoint.py.151: total_norm: 8.203716 [I 2022-10-09 18:32:10] ...trap/engines/engine.py.224: train: epoch 50 | batch 1200/2304 [I 2022-10-09 18:32:10] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:45 [I 2022-10-09 18:32:10] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00312 [I 2022-10-09 18:32:10] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-09 18:32:10] ...optimizers/trijoint.py.151: total_norm: 16.710253 [I 2022-10-09 18:32:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 1210/2304 [I 2022-10-09 18:32:19] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:31 [I 2022-10-09 18:32:19] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00312 [I 2022-10-09 18:32:19] ...trap/engines/engine.py.230: loss: 0.11677 [I 2022-10-09 18:32:19] ...optimizers/trijoint.py.151: total_norm: 8.427946 [I 2022-10-09 18:32:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 1220/2304 [I 2022-10-09 18:32:28] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:22 [I 2022-10-09 18:32:28] ...trap/engines/engine.py.229: process: 0.86092 | load: 0.00311 [I 2022-10-09 18:32:28] ...trap/engines/engine.py.230: loss: 0.31028 [I 2022-10-09 18:32:28] ...optimizers/trijoint.py.151: total_norm: 31.125727 [I 2022-10-09 18:32:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 1230/2304 [I 2022-10-09 18:32:36] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:09 [I 2022-10-09 18:32:36] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00314 [I 2022-10-09 18:32:36] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-09 18:32:36] ...optimizers/trijoint.py.151: total_norm: 14.304214 [I 2022-10-09 18:32:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1240/2304 [I 2022-10-09 18:32:45] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:59 [I 2022-10-09 18:32:45] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00300 [I 2022-10-09 18:32:45] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-09 18:32:45] ...optimizers/trijoint.py.151: total_norm: 14.372988 [I 2022-10-09 18:32:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 1250/2304 [I 2022-10-09 18:32:53] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:51 [I 2022-10-09 18:32:53] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00298 [I 2022-10-09 18:32:53] ...trap/engines/engine.py.230: loss: 0.12571 [I 2022-10-09 18:32:53] ...optimizers/trijoint.py.151: total_norm: 15.046902 [I 2022-10-09 18:33:01] ...trap/engines/engine.py.224: train: epoch 50 | batch 1260/2304 [I 2022-10-09 18:33:01] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:42 [I 2022-10-09 18:33:01] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00296 [I 2022-10-09 18:33:01] ...trap/engines/engine.py.230: loss: 0.21927 [I 2022-10-09 18:33:01] ...optimizers/trijoint.py.151: total_norm: 12.608314 [I 2022-10-09 18:33:10] ...trap/engines/engine.py.224: train: epoch 50 | batch 1270/2304 [I 2022-10-09 18:33:10] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:38 [I 2022-10-09 18:33:10] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00299 [I 2022-10-09 18:33:10] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-09 18:33:10] ...optimizers/trijoint.py.151: total_norm: 15.155863 [I 2022-10-09 18:33:18] ...trap/engines/engine.py.224: train: epoch 50 | batch 1280/2304 [I 2022-10-09 18:33:18] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:22 [I 2022-10-09 18:33:18] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00295 [I 2022-10-09 18:33:18] ...trap/engines/engine.py.230: loss: 0.18700 [I 2022-10-09 18:33:18] ...optimizers/trijoint.py.151: total_norm: 22.416964 [I 2022-10-09 18:33:27] ...trap/engines/engine.py.224: train: epoch 50 | batch 1290/2304 [I 2022-10-09 18:33:27] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:14 [I 2022-10-09 18:33:27] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00293 [I 2022-10-09 18:33:27] ...trap/engines/engine.py.230: loss: 0.14892 [I 2022-10-09 18:33:27] ...optimizers/trijoint.py.151: total_norm: 26.364309 [I 2022-10-09 18:33:35] ...trap/engines/engine.py.224: train: epoch 50 | batch 1300/2304 [I 2022-10-09 18:33:35] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:05 [I 2022-10-09 18:33:35] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00299 [I 2022-10-09 18:33:35] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-09 18:33:35] ...optimizers/trijoint.py.151: total_norm: 10.456410 [I 2022-10-09 18:33:44] ...trap/engines/engine.py.224: train: epoch 50 | batch 1310/2304 [I 2022-10-09 18:33:44] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:01 [I 2022-10-09 18:33:44] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00296 [I 2022-10-09 18:33:44] ...trap/engines/engine.py.230: loss: 0.15691 [I 2022-10-09 18:33:44] ...optimizers/trijoint.py.151: total_norm: 11.768174 [I 2022-10-09 18:33:52] ...trap/engines/engine.py.224: train: epoch 50 | batch 1320/2304 [I 2022-10-09 18:33:52] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:47 [I 2022-10-09 18:33:52] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00298 [I 2022-10-09 18:33:52] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-09 18:33:52] ...optimizers/trijoint.py.151: total_norm: 12.358233 [I 2022-10-09 18:34:01] ...trap/engines/engine.py.224: train: epoch 50 | batch 1330/2304 [I 2022-10-09 18:34:01] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:38 [I 2022-10-09 18:34:01] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00301 [I 2022-10-09 18:34:01] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-09 18:34:01] ...optimizers/trijoint.py.151: total_norm: 12.251509 [I 2022-10-09 18:34:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 1340/2304 [I 2022-10-09 18:34:09] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:35 [I 2022-10-09 18:34:09] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00299 [I 2022-10-09 18:34:09] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-09 18:34:09] ...optimizers/trijoint.py.151: total_norm: 10.347302 [I 2022-10-09 18:34:18] ...trap/engines/engine.py.224: train: epoch 50 | batch 1350/2304 [I 2022-10-09 18:34:18] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:23 [I 2022-10-09 18:34:18] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00317 [I 2022-10-09 18:34:18] ...trap/engines/engine.py.230: loss: 0.21105 [I 2022-10-09 18:34:18] ...optimizers/trijoint.py.151: total_norm: 16.984224 [I 2022-10-09 18:34:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 1360/2304 [I 2022-10-09 18:34:26] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:20 [I 2022-10-09 18:34:26] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00311 [I 2022-10-09 18:34:26] ...trap/engines/engine.py.230: loss: 0.15057 [I 2022-10-09 18:34:26] ...optimizers/trijoint.py.151: total_norm: 13.018537 [I 2022-10-09 18:34:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 1370/2304 [I 2022-10-09 18:34:34] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:02 [I 2022-10-09 18:34:34] ...trap/engines/engine.py.229: process: 0.83633 | load: 0.00305 [I 2022-10-09 18:34:34] ...trap/engines/engine.py.230: loss: 0.18982 [I 2022-10-09 18:34:34] ...optimizers/trijoint.py.151: total_norm: 14.260084 [I 2022-10-09 18:34:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 1380/2304 [I 2022-10-09 18:34:43] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:56 [I 2022-10-09 18:34:43] ...trap/engines/engine.py.229: process: 0.83446 | load: 0.00296 [I 2022-10-09 18:34:43] ...trap/engines/engine.py.230: loss: 0.10139 [I 2022-10-09 18:34:43] ...optimizers/trijoint.py.151: total_norm: 8.331985 [I 2022-10-09 18:34:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 1390/2304 [I 2022-10-09 18:34:51] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:52 [I 2022-10-09 18:34:51] ...trap/engines/engine.py.229: process: 0.84057 | load: 0.00300 [I 2022-10-09 18:34:51] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-09 18:34:51] ...optimizers/trijoint.py.151: total_norm: 10.704513 [I 2022-10-09 18:35:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 1400/2304 [I 2022-10-09 18:35:00] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:42 [I 2022-10-09 18:35:00] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00297 [I 2022-10-09 18:35:00] ...trap/engines/engine.py.230: loss: 0.15749 [I 2022-10-09 18:35:00] ...optimizers/trijoint.py.151: total_norm: 12.969578 [I 2022-10-09 18:35:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 1410/2304 [I 2022-10-09 18:35:08] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:33 [I 2022-10-09 18:35:08] ...trap/engines/engine.py.229: process: 0.83945 | load: 0.00298 [I 2022-10-09 18:35:08] ...trap/engines/engine.py.230: loss: 0.10755 [I 2022-10-09 18:35:08] ...optimizers/trijoint.py.151: total_norm: 9.876590 [I 2022-10-09 18:35:17] ...trap/engines/engine.py.224: train: epoch 50 | batch 1420/2304 [I 2022-10-09 18:35:17] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:24 [I 2022-10-09 18:35:17] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00302 [I 2022-10-09 18:35:17] ...trap/engines/engine.py.230: loss: 0.20897 [I 2022-10-09 18:35:17] ...optimizers/trijoint.py.151: total_norm: 14.656883 [I 2022-10-09 18:35:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 1430/2304 [I 2022-10-09 18:35:25] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:20 [I 2022-10-09 18:35:25] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00299 [I 2022-10-09 18:35:25] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-09 18:35:25] ...optimizers/trijoint.py.151: total_norm: 13.275346 [I 2022-10-09 18:35:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 1440/2304 [I 2022-10-09 18:35:34] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:11 [I 2022-10-09 18:35:34] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00297 [I 2022-10-09 18:35:34] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-09 18:35:34] ...optimizers/trijoint.py.151: total_norm: 13.499322 [I 2022-10-09 18:35:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1450/2304 [I 2022-10-09 18:35:42] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:11:59 [I 2022-10-09 18:35:42] ...trap/engines/engine.py.229: process: 0.82644 | load: 0.00305 [I 2022-10-09 18:35:42] ...trap/engines/engine.py.230: loss: 0.16470 [I 2022-10-09 18:35:42] ...optimizers/trijoint.py.151: total_norm: 11.441263 [I 2022-10-09 18:35:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 1460/2304 [I 2022-10-09 18:35:51] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:57 [I 2022-10-09 18:35:51] ...trap/engines/engine.py.229: process: 0.85420 | load: 0.00298 [I 2022-10-09 18:35:51] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-09 18:35:51] ...optimizers/trijoint.py.151: total_norm: 13.975763 [I 2022-10-09 18:35:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1470/2304 [I 2022-10-09 18:35:59] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:46 [I 2022-10-09 18:35:59] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00299 [I 2022-10-09 18:35:59] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-09 18:35:59] ...optimizers/trijoint.py.151: total_norm: 16.314863 [I 2022-10-09 18:36:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 1480/2304 [I 2022-10-09 18:36:08] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:37 [I 2022-10-09 18:36:08] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00297 [I 2022-10-09 18:36:08] ...trap/engines/engine.py.230: loss: 0.21730 [I 2022-10-09 18:36:08] ...optimizers/trijoint.py.151: total_norm: 18.447584 [I 2022-10-09 18:36:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1490/2304 [I 2022-10-09 18:36:16] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:30 [I 2022-10-09 18:36:16] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00300 [I 2022-10-09 18:36:16] ...trap/engines/engine.py.230: loss: 0.17769 [I 2022-10-09 18:36:16] ...optimizers/trijoint.py.151: total_norm: 21.654963 [I 2022-10-09 18:36:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 1500/2304 [I 2022-10-09 18:36:25] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:28 [I 2022-10-09 18:36:25] ...trap/engines/engine.py.229: process: 0.85435 | load: 0.00311 [I 2022-10-09 18:36:25] ...trap/engines/engine.py.230: loss: 0.18507 [I 2022-10-09 18:36:25] ...optimizers/trijoint.py.151: total_norm: 24.067116 [I 2022-10-09 18:36:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1510/2304 [I 2022-10-09 18:36:33] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:18 [I 2022-10-09 18:36:33] ...trap/engines/engine.py.229: process: 0.84399 | load: 0.00309 [I 2022-10-09 18:36:33] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-09 18:36:33] ...optimizers/trijoint.py.151: total_norm: 15.768820 [I 2022-10-09 18:36:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1520/2304 [I 2022-10-09 18:36:42] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:03 [I 2022-10-09 18:36:42] ...trap/engines/engine.py.229: process: 0.83201 | load: 0.00313 [I 2022-10-09 18:36:42] ...trap/engines/engine.py.230: loss: 0.11761 [I 2022-10-09 18:36:42] ...optimizers/trijoint.py.151: total_norm: 9.411192 [I 2022-10-09 18:36:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1530/2304 [I 2022-10-09 18:36:50] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:58 [I 2022-10-09 18:36:50] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00393 [I 2022-10-09 18:36:50] ...trap/engines/engine.py.230: loss: 0.15135 [I 2022-10-09 18:36:50] ...optimizers/trijoint.py.151: total_norm: 11.198726 [I 2022-10-09 18:36:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1540/2304 [I 2022-10-09 18:36:59] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:54 [I 2022-10-09 18:36:59] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00311 [I 2022-10-09 18:36:59] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-09 18:36:59] ...optimizers/trijoint.py.151: total_norm: 12.109566 [I 2022-10-09 18:37:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 1550/2304 [I 2022-10-09 18:37:08] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:44 [I 2022-10-09 18:37:08] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00312 [I 2022-10-09 18:37:08] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-09 18:37:08] ...optimizers/trijoint.py.151: total_norm: 18.353008 [I 2022-10-09 18:37:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1560/2304 [I 2022-10-09 18:37:16] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:38 [I 2022-10-09 18:37:16] ...trap/engines/engine.py.229: process: 0.86811 | load: 0.00319 [I 2022-10-09 18:37:16] ...trap/engines/engine.py.230: loss: 0.12727 [I 2022-10-09 18:37:16] ...optimizers/trijoint.py.151: total_norm: 24.588406 [I 2022-10-09 18:37:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 1570/2304 [I 2022-10-09 18:37:25] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:26 [I 2022-10-09 18:37:25] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00313 [I 2022-10-09 18:37:25] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-09 18:37:25] ...optimizers/trijoint.py.151: total_norm: 9.666671 [I 2022-10-09 18:37:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1580/2304 [I 2022-10-09 18:37:33] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:19 [I 2022-10-09 18:37:33] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00315 [I 2022-10-09 18:37:33] ...trap/engines/engine.py.230: loss: 0.25894 [I 2022-10-09 18:37:33] ...optimizers/trijoint.py.151: total_norm: 14.565708 [I 2022-10-09 18:37:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1590/2304 [I 2022-10-09 18:37:42] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:10 [I 2022-10-09 18:37:42] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00310 [I 2022-10-09 18:37:42] ...trap/engines/engine.py.230: loss: 0.22626 [I 2022-10-09 18:37:42] ...optimizers/trijoint.py.151: total_norm: 34.251930 [I 2022-10-09 18:37:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1600/2304 [I 2022-10-09 18:37:50] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:00 [I 2022-10-09 18:37:50] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00322 [I 2022-10-09 18:37:50] ...trap/engines/engine.py.230: loss: 0.14484 [I 2022-10-09 18:37:50] ...optimizers/trijoint.py.151: total_norm: 15.424553 [I 2022-10-09 18:37:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1610/2304 [I 2022-10-09 18:37:59] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:51 [I 2022-10-09 18:37:59] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00313 [I 2022-10-09 18:37:59] ...trap/engines/engine.py.230: loss: 0.18100 [I 2022-10-09 18:37:59] ...optimizers/trijoint.py.151: total_norm: 9.133405 [I 2022-10-09 18:38:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 1620/2304 [I 2022-10-09 18:38:07] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:41 [I 2022-10-09 18:38:07] ...trap/engines/engine.py.229: process: 0.85550 | load: 0.00311 [I 2022-10-09 18:38:07] ...trap/engines/engine.py.230: loss: 0.18919 [I 2022-10-09 18:38:07] ...optimizers/trijoint.py.151: total_norm: 12.423633 [I 2022-10-09 18:38:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1630/2304 [I 2022-10-09 18:38:16] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:32 [I 2022-10-09 18:38:16] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00310 [I 2022-10-09 18:38:16] ...trap/engines/engine.py.230: loss: 0.17696 [I 2022-10-09 18:38:16] ...optimizers/trijoint.py.151: total_norm: 20.137554 [I 2022-10-09 18:38:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1640/2304 [I 2022-10-09 18:38:24] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:25 [I 2022-10-09 18:38:24] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00312 [I 2022-10-09 18:38:24] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-09 18:38:24] ...optimizers/trijoint.py.151: total_norm: 25.699318 [I 2022-10-09 18:38:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1650/2304 [I 2022-10-09 18:38:33] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:19 [I 2022-10-09 18:38:33] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00316 [I 2022-10-09 18:38:33] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-09 18:38:33] ...optimizers/trijoint.py.151: total_norm: 31.308168 [I 2022-10-09 18:38:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1660/2304 [I 2022-10-09 18:38:42] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:13 [I 2022-10-09 18:38:42] ...trap/engines/engine.py.229: process: 0.86050 | load: 0.00312 [I 2022-10-09 18:38:42] ...trap/engines/engine.py.230: loss: 0.15016 [I 2022-10-09 18:38:42] ...optimizers/trijoint.py.151: total_norm: 16.491856 [I 2022-10-09 18:38:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1670/2304 [I 2022-10-09 18:38:50] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:01 [I 2022-10-09 18:38:50] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00310 [I 2022-10-09 18:38:50] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-09 18:38:50] ...optimizers/trijoint.py.151: total_norm: 14.102257 [I 2022-10-09 18:38:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1680/2304 [I 2022-10-09 18:38:59] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:56 [I 2022-10-09 18:38:59] ...trap/engines/engine.py.229: process: 0.87684 | load: 0.00313 [I 2022-10-09 18:38:59] ...trap/engines/engine.py.230: loss: 0.23724 [I 2022-10-09 18:38:59] ...optimizers/trijoint.py.151: total_norm: 16.260687 [I 2022-10-09 18:39:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 1690/2304 [I 2022-10-09 18:39:07] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:42 [I 2022-10-09 18:39:07] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00311 [I 2022-10-09 18:39:07] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-09 18:39:07] ...optimizers/trijoint.py.151: total_norm: 16.560555 [I 2022-10-09 18:39:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1700/2304 [I 2022-10-09 18:39:16] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:35 [I 2022-10-09 18:39:16] ...trap/engines/engine.py.229: process: 0.85591 | load: 0.00311 [I 2022-10-09 18:39:16] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-09 18:39:16] ...optimizers/trijoint.py.151: total_norm: 13.514969 [I 2022-10-09 18:39:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1710/2304 [I 2022-10-09 18:39:24] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:27 [I 2022-10-09 18:39:24] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00309 [I 2022-10-09 18:39:24] ...trap/engines/engine.py.230: loss: 0.13442 [I 2022-10-09 18:39:24] ...optimizers/trijoint.py.151: total_norm: 17.834532 [I 2022-10-09 18:39:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1720/2304 [I 2022-10-09 18:39:33] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:19 [I 2022-10-09 18:39:33] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00314 [I 2022-10-09 18:39:33] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-09 18:39:33] ...optimizers/trijoint.py.151: total_norm: 22.887455 [I 2022-10-09 18:39:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1730/2304 [I 2022-10-09 18:39:42] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:08 [I 2022-10-09 18:39:42] ...trap/engines/engine.py.229: process: 0.83752 | load: 0.00319 [I 2022-10-09 18:39:42] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-09 18:39:42] ...optimizers/trijoint.py.151: total_norm: 37.936291 [I 2022-10-09 18:39:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1740/2304 [I 2022-10-09 18:39:50] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:01 [I 2022-10-09 18:39:50] ...trap/engines/engine.py.229: process: 0.86618 | load: 0.00316 [I 2022-10-09 18:39:50] ...trap/engines/engine.py.230: loss: 0.15973 [I 2022-10-09 18:39:50] ...optimizers/trijoint.py.151: total_norm: 9.830447 [I 2022-10-09 18:39:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1750/2304 [I 2022-10-09 18:39:59] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:49 [I 2022-10-09 18:39:59] ...trap/engines/engine.py.229: process: 0.83571 | load: 0.00310 [I 2022-10-09 18:39:59] ...trap/engines/engine.py.230: loss: 0.14243 [I 2022-10-09 18:39:59] ...optimizers/trijoint.py.151: total_norm: 6.654891 [I 2022-10-09 18:40:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 1760/2304 [I 2022-10-09 18:40:07] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:42 [I 2022-10-09 18:40:07] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00313 [I 2022-10-09 18:40:07] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-09 18:40:07] ...optimizers/trijoint.py.151: total_norm: 17.703030 [I 2022-10-09 18:40:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1770/2304 [I 2022-10-09 18:40:16] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:33 [I 2022-10-09 18:40:16] ...trap/engines/engine.py.229: process: 0.83309 | load: 0.00311 [I 2022-10-09 18:40:16] ...trap/engines/engine.py.230: loss: 0.18849 [I 2022-10-09 18:40:16] ...optimizers/trijoint.py.151: total_norm: 25.285923 [I 2022-10-09 18:40:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1780/2304 [I 2022-10-09 18:40:24] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:24 [I 2022-10-09 18:40:24] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00302 [I 2022-10-09 18:40:24] ...trap/engines/engine.py.230: loss: 0.15334 [I 2022-10-09 18:40:24] ...optimizers/trijoint.py.151: total_norm: 27.269073 [I 2022-10-09 18:40:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1790/2304 [I 2022-10-09 18:40:33] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:17 [I 2022-10-09 18:40:33] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00298 [I 2022-10-09 18:40:33] ...trap/engines/engine.py.230: loss: 0.15717 [I 2022-10-09 18:40:33] ...optimizers/trijoint.py.151: total_norm: 13.427487 [I 2022-10-09 18:40:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 1800/2304 [I 2022-10-09 18:40:41] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:07 [I 2022-10-09 18:40:41] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00293 [I 2022-10-09 18:40:41] ...trap/engines/engine.py.230: loss: 0.13751 [I 2022-10-09 18:40:41] ...optimizers/trijoint.py.151: total_norm: 15.070654 [I 2022-10-09 18:41:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 1810/2304 [I 2022-10-09 18:41:08] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:10:57 [I 2022-10-09 18:41:08] ...trap/engines/engine.py.229: process: 0.83568 | load: 0.00291 [I 2022-10-09 18:41:08] ...trap/engines/engine.py.230: loss: 0.14591 [I 2022-10-09 18:41:08] ...optimizers/trijoint.py.151: total_norm: 11.784863 [I 2022-10-09 18:41:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1820/2304 [I 2022-10-09 18:41:16] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:07:11 [I 2022-10-09 18:41:16] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00293 [I 2022-10-09 18:41:16] ...trap/engines/engine.py.230: loss: 0.10283 [I 2022-10-09 18:41:16] ...optimizers/trijoint.py.151: total_norm: 8.636565 [I 2022-10-09 18:41:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 1830/2304 [I 2022-10-09 18:41:25] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:42 [I 2022-10-09 18:41:25] ...trap/engines/engine.py.229: process: 0.83824 | load: 0.00294 [I 2022-10-09 18:41:25] ...trap/engines/engine.py.230: loss: 0.19733 [I 2022-10-09 18:41:25] ...optimizers/trijoint.py.151: total_norm: 28.269333 [I 2022-10-09 18:41:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1840/2304 [I 2022-10-09 18:41:33] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:31 [I 2022-10-09 18:41:33] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00296 [I 2022-10-09 18:41:33] ...trap/engines/engine.py.230: loss: 0.12465 [I 2022-10-09 18:41:33] ...optimizers/trijoint.py.151: total_norm: 24.674606 [I 2022-10-09 18:41:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1850/2304 [I 2022-10-09 18:41:42] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:20 [I 2022-10-09 18:41:42] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00295 [I 2022-10-09 18:41:42] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-09 18:41:42] ...optimizers/trijoint.py.151: total_norm: 9.475917 [I 2022-10-09 18:41:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1860/2304 [I 2022-10-09 18:41:50] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:16 [I 2022-10-09 18:41:50] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00297 [I 2022-10-09 18:41:50] ...trap/engines/engine.py.230: loss: 0.16373 [I 2022-10-09 18:41:50] ...optimizers/trijoint.py.151: total_norm: 10.357588 [I 2022-10-09 18:41:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 1870/2304 [I 2022-10-09 18:41:58] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:05 [I 2022-10-09 18:41:58] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00296 [I 2022-10-09 18:41:58] ...trap/engines/engine.py.230: loss: 0.14837 [I 2022-10-09 18:41:58] ...optimizers/trijoint.py.151: total_norm: 25.670118 [I 2022-10-09 18:42:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 1880/2304 [I 2022-10-09 18:42:07] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:58 [I 2022-10-09 18:42:07] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00315 [I 2022-10-09 18:42:07] ...trap/engines/engine.py.230: loss: 0.11480 [I 2022-10-09 18:42:07] ...optimizers/trijoint.py.151: total_norm: 7.855422 [I 2022-10-09 18:42:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 1890/2304 [I 2022-10-09 18:42:15] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:50 [I 2022-10-09 18:42:15] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00298 [I 2022-10-09 18:42:15] ...trap/engines/engine.py.230: loss: 0.21129 [I 2022-10-09 18:42:15] ...optimizers/trijoint.py.151: total_norm: 18.929667 [I 2022-10-09 18:42:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1900/2304 [I 2022-10-09 18:42:24] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:41 [I 2022-10-09 18:42:24] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00309 [I 2022-10-09 18:42:24] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-09 18:42:24] ...optimizers/trijoint.py.151: total_norm: 9.148194 [I 2022-10-09 18:42:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 1910/2304 [I 2022-10-09 18:42:32] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:31 [I 2022-10-09 18:42:32] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00297 [I 2022-10-09 18:42:32] ...trap/engines/engine.py.230: loss: 0.17228 [I 2022-10-09 18:42:32] ...optimizers/trijoint.py.151: total_norm: 41.316875 [I 2022-10-09 18:42:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 1920/2304 [I 2022-10-09 18:42:41] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:24 [I 2022-10-09 18:42:41] ...trap/engines/engine.py.229: process: 0.84825 | load: 0.00299 [I 2022-10-09 18:42:41] ...trap/engines/engine.py.230: loss: 0.15244 [I 2022-10-09 18:42:41] ...optimizers/trijoint.py.151: total_norm: 16.202755 [I 2022-10-09 18:42:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 1930/2304 [I 2022-10-09 18:42:49] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:15 [I 2022-10-09 18:42:49] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00294 [I 2022-10-09 18:42:49] ...trap/engines/engine.py.230: loss: 0.10663 [I 2022-10-09 18:42:49] ...optimizers/trijoint.py.151: total_norm: 9.775105 [I 2022-10-09 18:42:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 1940/2304 [I 2022-10-09 18:42:58] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:05:06 [I 2022-10-09 18:42:58] ...trap/engines/engine.py.229: process: 0.83838 | load: 0.00298 [I 2022-10-09 18:42:58] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-09 18:42:58] ...optimizers/trijoint.py.151: total_norm: 22.490103 [I 2022-10-09 18:43:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 1950/2304 [I 2022-10-09 18:43:06] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:57 [I 2022-10-09 18:43:06] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00303 [I 2022-10-09 18:43:06] ...trap/engines/engine.py.230: loss: 0.15924 [I 2022-10-09 18:43:06] ...optimizers/trijoint.py.151: total_norm: 32.176243 [I 2022-10-09 18:43:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 1960/2304 [I 2022-10-09 18:43:15] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:51 [I 2022-10-09 18:43:15] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00306 [I 2022-10-09 18:43:15] ...trap/engines/engine.py.230: loss: 0.11467 [I 2022-10-09 18:43:15] ...optimizers/trijoint.py.151: total_norm: 10.777297 [I 2022-10-09 18:43:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 1970/2304 [I 2022-10-09 18:43:23] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:41 [I 2022-10-09 18:43:23] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00305 [I 2022-10-09 18:43:23] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-09 18:43:23] ...optimizers/trijoint.py.151: total_norm: 12.078332 [I 2022-10-09 18:43:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 1980/2304 [I 2022-10-09 18:43:32] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:34 [I 2022-10-09 18:43:32] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00295 [I 2022-10-09 18:43:32] ...trap/engines/engine.py.230: loss: 0.12387 [I 2022-10-09 18:43:32] ...optimizers/trijoint.py.151: total_norm: 12.640105 [I 2022-10-09 18:43:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 1990/2304 [I 2022-10-09 18:43:40] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:24 [I 2022-10-09 18:43:40] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00301 [I 2022-10-09 18:43:40] ...trap/engines/engine.py.230: loss: 0.18129 [I 2022-10-09 18:43:40] ...optimizers/trijoint.py.151: total_norm: 15.642142 [I 2022-10-09 18:43:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 2000/2304 [I 2022-10-09 18:43:49] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:18 [I 2022-10-09 18:43:49] ...trap/engines/engine.py.229: process: 0.85801 | load: 0.00312 [I 2022-10-09 18:43:49] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-09 18:43:49] ...optimizers/trijoint.py.151: total_norm: 18.022182 [I 2022-10-09 18:43:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 2010/2304 [I 2022-10-09 18:43:57] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:10 [I 2022-10-09 18:43:57] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00312 [I 2022-10-09 18:43:57] ...trap/engines/engine.py.230: loss: 0.18562 [I 2022-10-09 18:43:57] ...optimizers/trijoint.py.151: total_norm: 27.238548 [I 2022-10-09 18:44:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 2020/2304 [I 2022-10-09 18:44:06] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:04:02 [I 2022-10-09 18:44:06] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00312 [I 2022-10-09 18:44:06] ...trap/engines/engine.py.230: loss: 0.12766 [I 2022-10-09 18:44:06] ...optimizers/trijoint.py.151: total_norm: 12.271266 [I 2022-10-09 18:44:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 2030/2304 [I 2022-10-09 18:44:14] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:55 [I 2022-10-09 18:44:14] ...trap/engines/engine.py.229: process: 0.86886 | load: 0.00317 [I 2022-10-09 18:44:14] ...trap/engines/engine.py.230: loss: 0.18037 [I 2022-10-09 18:44:14] ...optimizers/trijoint.py.151: total_norm: 16.722778 [I 2022-10-09 18:44:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 2040/2304 [I 2022-10-09 18:44:23] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:46 [I 2022-10-09 18:44:23] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00311 [I 2022-10-09 18:44:23] ...trap/engines/engine.py.230: loss: 0.13865 [I 2022-10-09 18:44:23] ...optimizers/trijoint.py.151: total_norm: 12.160403 [I 2022-10-09 18:44:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 2050/2304 [I 2022-10-09 18:44:31] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:38 [I 2022-10-09 18:44:31] ...trap/engines/engine.py.229: process: 0.87650 | load: 0.00311 [I 2022-10-09 18:44:31] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-09 18:44:31] ...optimizers/trijoint.py.151: total_norm: 18.127066 [I 2022-10-09 18:44:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 2060/2304 [I 2022-10-09 18:44:40] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:29 [I 2022-10-09 18:44:40] ...trap/engines/engine.py.229: process: 0.86425 | load: 0.00314 [I 2022-10-09 18:44:40] ...trap/engines/engine.py.230: loss: 0.18804 [I 2022-10-09 18:44:40] ...optimizers/trijoint.py.151: total_norm: 16.259064 [I 2022-10-09 18:44:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 2070/2304 [I 2022-10-09 18:44:48] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:19 [I 2022-10-09 18:44:48] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00313 [I 2022-10-09 18:44:48] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-09 18:44:48] ...optimizers/trijoint.py.151: total_norm: 17.337740 [I 2022-10-09 18:44:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 2080/2304 [I 2022-10-09 18:44:57] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:11 [I 2022-10-09 18:44:57] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00312 [I 2022-10-09 18:44:57] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-09 18:44:57] ...optimizers/trijoint.py.151: total_norm: 17.282867 [I 2022-10-09 18:45:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 2090/2304 [I 2022-10-09 18:45:06] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:02 [I 2022-10-09 18:45:06] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00312 [I 2022-10-09 18:45:06] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-09 18:45:06] ...optimizers/trijoint.py.151: total_norm: 29.095797 [I 2022-10-09 18:45:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 2100/2304 [I 2022-10-09 18:45:14] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:54 [I 2022-10-09 18:45:14] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00312 [I 2022-10-09 18:45:14] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-09 18:45:14] ...optimizers/trijoint.py.151: total_norm: 15.042253 [I 2022-10-09 18:45:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 2110/2304 [I 2022-10-09 18:45:23] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:45 [I 2022-10-09 18:45:23] ...trap/engines/engine.py.229: process: 0.85402 | load: 0.00324 [I 2022-10-09 18:45:23] ...trap/engines/engine.py.230: loss: 0.19460 [I 2022-10-09 18:45:23] ...optimizers/trijoint.py.151: total_norm: 17.574394 [I 2022-10-09 18:45:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 2120/2304 [I 2022-10-09 18:45:31] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:37 [I 2022-10-09 18:45:31] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00311 [I 2022-10-09 18:45:31] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-09 18:45:31] ...optimizers/trijoint.py.151: total_norm: 13.259475 [I 2022-10-09 18:45:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 2130/2304 [I 2022-10-09 18:45:40] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:28 [I 2022-10-09 18:45:40] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00312 [I 2022-10-09 18:45:40] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-09 18:45:40] ...optimizers/trijoint.py.151: total_norm: 12.637177 [I 2022-10-09 18:45:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 2140/2304 [I 2022-10-09 18:45:48] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:19 [I 2022-10-09 18:45:48] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00312 [I 2022-10-09 18:45:48] ...trap/engines/engine.py.230: loss: 0.19974 [I 2022-10-09 18:45:48] ...optimizers/trijoint.py.151: total_norm: 15.600454 [I 2022-10-09 18:45:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 2150/2304 [I 2022-10-09 18:45:57] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:11 [I 2022-10-09 18:45:57] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00314 [I 2022-10-09 18:45:57] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-09 18:45:57] ...optimizers/trijoint.py.151: total_norm: 21.064917 [I 2022-10-09 18:46:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 2160/2304 [I 2022-10-09 18:46:05] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:02 [I 2022-10-09 18:46:05] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00312 [I 2022-10-09 18:46:05] ...trap/engines/engine.py.230: loss: 0.19341 [I 2022-10-09 18:46:05] ...optimizers/trijoint.py.151: total_norm: 24.452448 [I 2022-10-09 18:46:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 2170/2304 [I 2022-10-09 18:46:14] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:54 [I 2022-10-09 18:46:14] ...trap/engines/engine.py.229: process: 0.85428 | load: 0.00311 [I 2022-10-09 18:46:14] ...trap/engines/engine.py.230: loss: 0.17061 [I 2022-10-09 18:46:14] ...optimizers/trijoint.py.151: total_norm: 16.859865 [I 2022-10-09 18:46:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 2180/2304 [I 2022-10-09 18:46:23] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:45 [I 2022-10-09 18:46:23] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00315 [I 2022-10-09 18:46:23] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-09 18:46:23] ...optimizers/trijoint.py.151: total_norm: 12.439607 [I 2022-10-09 18:46:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 2190/2304 [I 2022-10-09 18:46:31] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:37 [I 2022-10-09 18:46:31] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00312 [I 2022-10-09 18:46:31] ...trap/engines/engine.py.230: loss: 0.17178 [I 2022-10-09 18:46:31] ...optimizers/trijoint.py.151: total_norm: 9.576620 [I 2022-10-09 18:46:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 2200/2304 [I 2022-10-09 18:46:40] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:28 [I 2022-10-09 18:46:40] ...trap/engines/engine.py.229: process: 0.86209 | load: 0.00315 [I 2022-10-09 18:46:40] ...trap/engines/engine.py.230: loss: 0.20976 [I 2022-10-09 18:46:40] ...optimizers/trijoint.py.151: total_norm: 10.888107 [I 2022-10-09 18:46:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 2210/2304 [I 2022-10-09 18:46:48] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:20 [I 2022-10-09 18:46:48] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00311 [I 2022-10-09 18:46:48] ...trap/engines/engine.py.230: loss: 0.15097 [I 2022-10-09 18:46:48] ...optimizers/trijoint.py.151: total_norm: 16.930433 [I 2022-10-09 18:46:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 2220/2304 [I 2022-10-09 18:46:57] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:11 [I 2022-10-09 18:46:57] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00312 [I 2022-10-09 18:46:57] ...trap/engines/engine.py.230: loss: 0.13926 [I 2022-10-09 18:46:57] ...optimizers/trijoint.py.151: total_norm: 17.560562 [I 2022-10-09 18:47:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 2230/2304 [I 2022-10-09 18:47:05] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:03 [I 2022-10-09 18:47:05] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00314 [I 2022-10-09 18:47:05] ...trap/engines/engine.py.230: loss: 0.14990 [I 2022-10-09 18:47:05] ...optimizers/trijoint.py.151: total_norm: 13.647470 [I 2022-10-09 18:47:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 2240/2304 [I 2022-10-09 18:47:14] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:54 [I 2022-10-09 18:47:14] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00314 [I 2022-10-09 18:47:14] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-09 18:47:14] ...optimizers/trijoint.py.151: total_norm: 11.131679 [I 2022-10-09 18:47:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 2250/2304 [I 2022-10-09 18:47:22] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:46 [I 2022-10-09 18:47:22] ...trap/engines/engine.py.229: process: 0.85506 | load: 0.00313 [I 2022-10-09 18:47:22] ...trap/engines/engine.py.230: loss: 0.15440 [I 2022-10-09 18:47:22] ...optimizers/trijoint.py.151: total_norm: 19.628654 [I 2022-10-09 18:47:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 2260/2304 [I 2022-10-09 18:47:31] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:37 [I 2022-10-09 18:47:31] ...trap/engines/engine.py.229: process: 0.83528 | load: 0.00312 [I 2022-10-09 18:47:31] ...trap/engines/engine.py.230: loss: 0.14861 [I 2022-10-09 18:47:31] ...optimizers/trijoint.py.151: total_norm: 7.657654 [I 2022-10-09 18:47:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 2270/2304 [I 2022-10-09 18:47:39] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:28 [I 2022-10-09 18:47:39] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00305 [I 2022-10-09 18:47:39] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-09 18:47:39] ...optimizers/trijoint.py.151: total_norm: 9.596982 [I 2022-10-09 18:47:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 2280/2304 [I 2022-10-09 18:47:48] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:20 [I 2022-10-09 18:47:48] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00294 [I 2022-10-09 18:47:48] ...trap/engines/engine.py.230: loss: 0.17874 [I 2022-10-09 18:47:48] ...optimizers/trijoint.py.151: total_norm: 12.786534 [I 2022-10-09 18:47:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 2290/2304 [I 2022-10-09 18:47:56] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:11 [I 2022-10-09 18:47:56] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00304 [I 2022-10-09 18:47:56] ...trap/engines/engine.py.230: loss: 0.16168 [I 2022-10-09 18:47:56] ...optimizers/trijoint.py.151: total_norm: 23.115221 [I 2022-10-09 18:48:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 2300/2304 [I 2022-10-09 18:48:05] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:03 [I 2022-10-09 18:48:05] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00021 [I 2022-10-09 18:48:05] ...trap/engines/engine.py.230: loss: 0.13718 [I 2022-10-09 18:48:05] ...optimizers/trijoint.py.151: total_norm: 9.638079 [I 2022-10-09 18:48:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 2304/2304 [I 2022-10-09 18:48:08] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:00 [I 2022-10-09 18:48:08] ...trap/engines/engine.py.229: process: 0.83234 | load: 0.00018 [I 2022-10-09 18:48:08] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-09 18:48:08] ...optimizers/trijoint.py.151: total_norm: 11.805467 [S 2022-10-09 18:48:09] ...trap/engines/engine.py.129: train_epoch.epoch: 50 [S 2022-10-09 18:48:09] ...trap/engines/engine.py.129: train_epoch.loss: 0.1553057948091025 [I 2022-10-09 18:48:09] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 18:48:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 50 [I 2022-10-09 18:48:11] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 18:48:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 18:48:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 18:48:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 18:48:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 18:48:14] ...trap/engines/engine.py.321: eval: epoch 50 | batch 0/510 [I 2022-10-09 18:48:14] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:41 [I 2022-10-09 18:48:14] ...trap/engines/engine.py.326: process: 3.96461 | load: 3.05782 [I 2022-10-09 18:48:19] ...trap/engines/engine.py.321: eval: epoch 50 | batch 10/510 [I 2022-10-09 18:48:19] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:56 [I 2022-10-09 18:48:19] ...trap/engines/engine.py.326: process: 0.55665 | load: 0.00018 [I 2022-10-09 18:48:25] ...trap/engines/engine.py.321: eval: epoch 50 | batch 20/510 [I 2022-10-09 18:48:25] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:16 [I 2022-10-09 18:48:25] ...trap/engines/engine.py.326: process: 0.58506 | load: 0.00018 [I 2022-10-09 18:48:30] ...trap/engines/engine.py.321: eval: epoch 50 | batch 30/510 [I 2022-10-09 18:48:30] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:19 [I 2022-10-09 18:48:30] ...trap/engines/engine.py.326: process: 0.63974 | load: 0.00018 [I 2022-10-09 18:48:37] ...trap/engines/engine.py.321: eval: epoch 50 | batch 40/510 [I 2022-10-09 18:48:37] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:57 [I 2022-10-09 18:48:37] ...trap/engines/engine.py.326: process: 0.61682 | load: 0.00019 [I 2022-10-09 18:48:42] ...trap/engines/engine.py.321: eval: epoch 50 | batch 50/510 [I 2022-10-09 18:48:42] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:04 [I 2022-10-09 18:48:42] ...trap/engines/engine.py.326: process: 0.47987 | load: 0.00019 [I 2022-10-09 18:48:47] ...trap/engines/engine.py.321: eval: epoch 50 | batch 60/510 [I 2022-10-09 18:48:47] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:44 [I 2022-10-09 18:48:47] ...trap/engines/engine.py.326: process: 0.53610 | load: 0.00023 [I 2022-10-09 18:48:52] ...trap/engines/engine.py.321: eval: epoch 50 | batch 70/510 [I 2022-10-09 18:48:52] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:34 [I 2022-10-09 18:48:52] ...trap/engines/engine.py.326: process: 0.43078 | load: 0.00019 [I 2022-10-09 18:48:57] ...trap/engines/engine.py.321: eval: epoch 50 | batch 80/510 [I 2022-10-09 18:48:57] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:42 [I 2022-10-09 18:48:57] ...trap/engines/engine.py.326: process: 0.54932 | load: 0.00020 [I 2022-10-09 18:49:02] ...trap/engines/engine.py.321: eval: epoch 50 | batch 90/510 [I 2022-10-09 18:49:02] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:13 [I 2022-10-09 18:49:02] ...trap/engines/engine.py.326: process: 0.43376 | load: 0.00018 [I 2022-10-09 18:49:06] ...trap/engines/engine.py.321: eval: epoch 50 | batch 100/510 [I 2022-10-09 18:49:06] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:51 [I 2022-10-09 18:49:06] ...trap/engines/engine.py.326: process: 0.30168 | load: 0.00018 [I 2022-10-09 18:49:09] ...trap/engines/engine.py.321: eval: epoch 50 | batch 110/510 [I 2022-10-09 18:49:09] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:09 [I 2022-10-09 18:49:09] ...trap/engines/engine.py.326: process: 0.31602 | load: 0.00019 [I 2022-10-09 18:49:12] ...trap/engines/engine.py.321: eval: epoch 50 | batch 120/510 [I 2022-10-09 18:49:12] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-09 18:49:12] ...trap/engines/engine.py.326: process: 0.31809 | load: 0.00020 [I 2022-10-09 18:49:15] ...trap/engines/engine.py.321: eval: epoch 50 | batch 130/510 [I 2022-10-09 18:49:15] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:00 [I 2022-10-09 18:49:15] ...trap/engines/engine.py.326: process: 0.31892 | load: 0.00020 [I 2022-10-09 18:49:19] ...trap/engines/engine.py.321: eval: epoch 50 | batch 140/510 [I 2022-10-09 18:49:19] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:56 [I 2022-10-09 18:49:19] ...trap/engines/engine.py.326: process: 0.31118 | load: 0.00019 [I 2022-10-09 18:49:22] ...trap/engines/engine.py.321: eval: epoch 50 | batch 150/510 [I 2022-10-09 18:49:22] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:53 [I 2022-10-09 18:49:22] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00020 [I 2022-10-09 18:49:25] ...trap/engines/engine.py.321: eval: epoch 50 | batch 160/510 [I 2022-10-09 18:49:25] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-09 18:49:25] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00020 [I 2022-10-09 18:49:28] ...trap/engines/engine.py.321: eval: epoch 50 | batch 170/510 [I 2022-10-09 18:49:28] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-09 18:49:28] ...trap/engines/engine.py.326: process: 0.31947 | load: 0.00020 [I 2022-10-09 18:49:31] ...trap/engines/engine.py.321: eval: epoch 50 | batch 180/510 [I 2022-10-09 18:49:31] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:44 [I 2022-10-09 18:49:31] ...trap/engines/engine.py.326: process: 0.31172 | load: 0.00020 [I 2022-10-09 18:49:34] ...trap/engines/engine.py.321: eval: epoch 50 | batch 190/510 [I 2022-10-09 18:49:34] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:41 [I 2022-10-09 18:49:34] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-09 18:49:38] ...trap/engines/engine.py.321: eval: epoch 50 | batch 200/510 [I 2022-10-09 18:49:38] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:38 [I 2022-10-09 18:49:38] ...trap/engines/engine.py.326: process: 0.31525 | load: 0.00019 [I 2022-10-09 18:49:41] ...trap/engines/engine.py.321: eval: epoch 50 | batch 210/510 [I 2022-10-09 18:49:41] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:34 [I 2022-10-09 18:49:41] ...trap/engines/engine.py.326: process: 0.31355 | load: 0.00019 [I 2022-10-09 18:49:44] ...trap/engines/engine.py.321: eval: epoch 50 | batch 220/510 [I 2022-10-09 18:49:44] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-09 18:49:44] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00020 [I 2022-10-09 18:49:47] ...trap/engines/engine.py.321: eval: epoch 50 | batch 230/510 [I 2022-10-09 18:49:47] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-09 18:49:47] ...trap/engines/engine.py.326: process: 0.31504 | load: 0.00019 [I 2022-10-09 18:49:50] ...trap/engines/engine.py.321: eval: epoch 50 | batch 240/510 [I 2022-10-09 18:49:50] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:25 [I 2022-10-09 18:49:50] ...trap/engines/engine.py.326: process: 0.31482 | load: 0.00020 [I 2022-10-09 18:49:54] ...trap/engines/engine.py.321: eval: epoch 50 | batch 250/510 [I 2022-10-09 18:49:54] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:21 [I 2022-10-09 18:49:54] ...trap/engines/engine.py.326: process: 0.31763 | load: 0.00019 [I 2022-10-09 18:49:57] ...trap/engines/engine.py.321: eval: epoch 50 | batch 260/510 [I 2022-10-09 18:49:57] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:18 [I 2022-10-09 18:49:57] ...trap/engines/engine.py.326: process: 0.31750 | load: 0.00020 [I 2022-10-09 18:50:00] ...trap/engines/engine.py.321: eval: epoch 50 | batch 270/510 [I 2022-10-09 18:50:00] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:15 [I 2022-10-09 18:50:00] ...trap/engines/engine.py.326: process: 0.31413 | load: 0.00019 [I 2022-10-09 18:50:03] ...trap/engines/engine.py.321: eval: epoch 50 | batch 280/510 [I 2022-10-09 18:50:03] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-09 18:50:03] ...trap/engines/engine.py.326: process: 0.31939 | load: 0.00021 [I 2022-10-09 18:50:06] ...trap/engines/engine.py.321: eval: epoch 50 | batch 290/510 [I 2022-10-09 18:50:06] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-09 18:50:06] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00021 [I 2022-10-09 18:50:09] ...trap/engines/engine.py.321: eval: epoch 50 | batch 300/510 [I 2022-10-09 18:50:09] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:06 [I 2022-10-09 18:50:09] ...trap/engines/engine.py.326: process: 0.31758 | load: 0.00020 [I 2022-10-09 18:50:13] ...trap/engines/engine.py.321: eval: epoch 50 | batch 310/510 [I 2022-10-09 18:50:13] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-09 18:50:13] ...trap/engines/engine.py.326: process: 0.31768 | load: 0.00020 [I 2022-10-09 18:50:16] ...trap/engines/engine.py.321: eval: epoch 50 | batch 320/510 [I 2022-10-09 18:50:16] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:59 [I 2022-10-09 18:50:16] ...trap/engines/engine.py.326: process: 0.31236 | load: 0.00021 [I 2022-10-09 18:50:19] ...trap/engines/engine.py.321: eval: epoch 50 | batch 330/510 [I 2022-10-09 18:50:19] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-09 18:50:19] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00020 [I 2022-10-09 18:50:22] ...trap/engines/engine.py.321: eval: epoch 50 | batch 340/510 [I 2022-10-09 18:50:22] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:53 [I 2022-10-09 18:50:22] ...trap/engines/engine.py.326: process: 0.31771 | load: 0.00021 [I 2022-10-09 18:50:25] ...trap/engines/engine.py.321: eval: epoch 50 | batch 350/510 [I 2022-10-09 18:50:25] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-09 18:50:25] ...trap/engines/engine.py.326: process: 0.32060 | load: 0.00021 [I 2022-10-09 18:50:29] ...trap/engines/engine.py.321: eval: epoch 50 | batch 360/510 [I 2022-10-09 18:50:29] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-09 18:50:29] ...trap/engines/engine.py.326: process: 0.32888 | load: 0.00020 [I 2022-10-09 18:50:32] ...trap/engines/engine.py.321: eval: epoch 50 | batch 370/510 [I 2022-10-09 18:50:32] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-09 18:50:32] ...trap/engines/engine.py.326: process: 0.31849 | load: 0.00021 [I 2022-10-09 18:50:35] ...trap/engines/engine.py.321: eval: epoch 50 | batch 380/510 [I 2022-10-09 18:50:35] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-09 18:50:35] ...trap/engines/engine.py.326: process: 0.31843 | load: 0.00021 [I 2022-10-09 18:50:38] ...trap/engines/engine.py.321: eval: epoch 50 | batch 390/510 [I 2022-10-09 18:50:38] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-09 18:50:38] ...trap/engines/engine.py.326: process: 0.32102 | load: 0.00021 [I 2022-10-09 18:50:41] ...trap/engines/engine.py.321: eval: epoch 50 | batch 400/510 [I 2022-10-09 18:50:41] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-09 18:50:41] ...trap/engines/engine.py.326: process: 0.32003 | load: 0.00022 [I 2022-10-09 18:50:45] ...trap/engines/engine.py.321: eval: epoch 50 | batch 410/510 [I 2022-10-09 18:50:45] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-09 18:50:45] ...trap/engines/engine.py.326: process: 0.32217 | load: 0.00020 [I 2022-10-09 18:50:48] ...trap/engines/engine.py.321: eval: epoch 50 | batch 420/510 [I 2022-10-09 18:50:48] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-09 18:50:48] ...trap/engines/engine.py.326: process: 0.32585 | load: 0.00021 [I 2022-10-09 18:50:51] ...trap/engines/engine.py.321: eval: epoch 50 | batch 430/510 [I 2022-10-09 18:50:51] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-09 18:50:51] ...trap/engines/engine.py.326: process: 0.32204 | load: 0.00020 [I 2022-10-09 18:50:54] ...trap/engines/engine.py.321: eval: epoch 50 | batch 440/510 [I 2022-10-09 18:50:54] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-09 18:50:54] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00021 [I 2022-10-09 18:50:58] ...trap/engines/engine.py.321: eval: epoch 50 | batch 450/510 [I 2022-10-09 18:50:58] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-09 18:50:58] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00021 [I 2022-10-09 18:51:01] ...trap/engines/engine.py.321: eval: epoch 50 | batch 460/510 [I 2022-10-09 18:51:01] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-09 18:51:01] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00021 [I 2022-10-09 18:51:04] ...trap/engines/engine.py.321: eval: epoch 50 | batch 470/510 [I 2022-10-09 18:51:04] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-09 18:51:04] ...trap/engines/engine.py.326: process: 0.32165 | load: 0.00021 [I 2022-10-09 18:51:07] ...trap/engines/engine.py.321: eval: epoch 50 | batch 480/510 [I 2022-10-09 18:51:07] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-09 18:51:07] ...trap/engines/engine.py.326: process: 0.32448 | load: 0.00021 [I 2022-10-09 18:51:10] ...trap/engines/engine.py.321: eval: epoch 50 | batch 490/510 [I 2022-10-09 18:51:10] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-09 18:51:10] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00021 [I 2022-10-09 18:51:14] ...trap/engines/engine.py.321: eval: epoch 50 | batch 500/510 [I 2022-10-09 18:51:14] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-09 18:51:14] ...trap/engines/engine.py.326: process: 0.31739 | load: 0.00020 [I 2022-10-09 18:51:17] ...trap/engines/engine.py.321: eval: epoch 50 | batch 510/510 [I 2022-10-09 18:51:17] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-09 18:51:17] ...trap/engines/engine.py.326: process: 0.31799 | load: 0.00016 [S 2022-10-09 18:51:18] ...trap/engines/engine.py.132: eval_epoch.epoch: 50 [I 2022-10-09 18:51:18] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.718300 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901900 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937500 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.712300 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900700 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934700 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010546 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010084 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006152 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013986 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007417 [S 2022-10-09 18:51:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006325 [I 2022-10-09 18:51:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 18:51:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 18:51:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 18:51:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 18:51:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 18:51:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 18:51:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 18:52:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 18:52:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 51 [I 2022-10-09 18:52:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 18:52:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 0/2304 [I 2022-10-09 18:52:07] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:32 [I 2022-10-09 18:52:07] ...trap/engines/engine.py.229: process: 3.03511 | load: 2.13898 [I 2022-10-09 18:52:07] ...trap/engines/engine.py.230: loss: 0.15038 [I 2022-10-09 18:52:07] ...optimizers/trijoint.py.151: total_norm: 16.924870 [I 2022-10-09 18:52:16] ...trap/engines/engine.py.224: train: epoch 51 | batch 10/2304 [I 2022-10-09 18:52:16] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:28 [I 2022-10-09 18:52:16] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00529 [I 2022-10-09 18:52:16] ...trap/engines/engine.py.230: loss: 0.16524 [I 2022-10-09 18:52:16] ...optimizers/trijoint.py.151: total_norm: 10.541533 [I 2022-10-09 18:52:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 20/2304 [I 2022-10-09 18:52:24] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:21 [I 2022-10-09 18:52:24] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00425 [I 2022-10-09 18:52:24] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-09 18:52:24] ...optimizers/trijoint.py.151: total_norm: 9.042325 [I 2022-10-09 18:52:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 30/2304 [I 2022-10-09 18:52:33] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:24 [I 2022-10-09 18:52:33] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00504 [I 2022-10-09 18:52:33] ...trap/engines/engine.py.230: loss: 0.13755 [I 2022-10-09 18:52:33] ...optimizers/trijoint.py.151: total_norm: 14.981963 [I 2022-10-09 18:52:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 40/2304 [I 2022-10-09 18:52:41] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:06 [I 2022-10-09 18:52:41] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00430 [I 2022-10-09 18:52:42] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-09 18:52:42] ...optimizers/trijoint.py.151: total_norm: 14.892883 [I 2022-10-09 18:52:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 50/2304 [I 2022-10-09 18:52:50] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:44 [I 2022-10-09 18:52:50] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00384 [I 2022-10-09 18:52:50] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-09 18:52:50] ...optimizers/trijoint.py.151: total_norm: 13.997272 [I 2022-10-09 18:52:59] ...trap/engines/engine.py.224: train: epoch 51 | batch 60/2304 [I 2022-10-09 18:52:59] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:57 [I 2022-10-09 18:52:59] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00358 [I 2022-10-09 18:52:59] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-09 18:52:59] ...optimizers/trijoint.py.151: total_norm: 27.809752 [I 2022-10-09 18:53:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 70/2304 [I 2022-10-09 18:53:07] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:52 [I 2022-10-09 18:53:07] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00382 [I 2022-10-09 18:53:07] ...trap/engines/engine.py.230: loss: 0.18592 [I 2022-10-09 18:53:07] ...optimizers/trijoint.py.151: total_norm: 16.977839 [I 2022-10-09 18:53:16] ...trap/engines/engine.py.224: train: epoch 51 | batch 80/2304 [I 2022-10-09 18:53:16] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:37 [I 2022-10-09 18:53:16] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00358 [I 2022-10-09 18:53:16] ...trap/engines/engine.py.230: loss: 0.14534 [I 2022-10-09 18:53:16] ...optimizers/trijoint.py.151: total_norm: 15.555417 [I 2022-10-09 18:53:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 90/2304 [I 2022-10-09 18:53:24] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:27 [I 2022-10-09 18:53:24] ...trap/engines/engine.py.229: process: 0.86630 | load: 0.00357 [I 2022-10-09 18:53:24] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-09 18:53:24] ...optimizers/trijoint.py.151: total_norm: 16.228642 [I 2022-10-09 18:53:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 100/2304 [I 2022-10-09 18:53:33] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:20 [I 2022-10-09 18:53:33] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00371 [I 2022-10-09 18:53:33] ...trap/engines/engine.py.230: loss: 0.15910 [I 2022-10-09 18:53:33] ...optimizers/trijoint.py.151: total_norm: 22.702093 [I 2022-10-09 18:53:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 110/2304 [I 2022-10-09 18:53:41] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:13 [I 2022-10-09 18:53:41] ...trap/engines/engine.py.229: process: 0.86715 | load: 0.00363 [I 2022-10-09 18:53:41] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-09 18:53:41] ...optimizers/trijoint.py.151: total_norm: 8.798838 [I 2022-10-09 18:53:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 120/2304 [I 2022-10-09 18:53:50] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:02 [I 2022-10-09 18:53:50] ...trap/engines/engine.py.229: process: 0.84163 | load: 0.00296 [I 2022-10-09 18:53:50] ...trap/engines/engine.py.230: loss: 0.15831 [I 2022-10-09 18:53:50] ...optimizers/trijoint.py.151: total_norm: 42.917984 [I 2022-10-09 18:53:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 130/2304 [I 2022-10-09 18:53:58] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:49 [I 2022-10-09 18:53:58] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00350 [I 2022-10-09 18:53:58] ...trap/engines/engine.py.230: loss: 0.11473 [I 2022-10-09 18:53:58] ...optimizers/trijoint.py.151: total_norm: 13.560018 [I 2022-10-09 18:54:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 140/2304 [I 2022-10-09 18:54:07] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:37 [I 2022-10-09 18:54:07] ...trap/engines/engine.py.229: process: 0.84309 | load: 0.00361 [I 2022-10-09 18:54:07] ...trap/engines/engine.py.230: loss: 0.16948 [I 2022-10-09 18:54:07] ...optimizers/trijoint.py.151: total_norm: 9.740307 [I 2022-10-09 18:54:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 150/2304 [I 2022-10-09 18:54:15] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:39 [I 2022-10-09 18:54:15] ...trap/engines/engine.py.229: process: 0.86351 | load: 0.00314 [I 2022-10-09 18:54:15] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-09 18:54:15] ...optimizers/trijoint.py.151: total_norm: 9.874611 [I 2022-10-09 18:54:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 160/2304 [I 2022-10-09 18:54:24] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:29 [I 2022-10-09 18:54:24] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00312 [I 2022-10-09 18:54:24] ...trap/engines/engine.py.230: loss: 0.26656 [I 2022-10-09 18:54:24] ...optimizers/trijoint.py.151: total_norm: 17.230085 [I 2022-10-09 18:54:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 170/2304 [I 2022-10-09 18:54:32] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:13 [I 2022-10-09 18:54:32] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00331 [I 2022-10-09 18:54:32] ...trap/engines/engine.py.230: loss: 0.20903 [I 2022-10-09 18:54:32] ...optimizers/trijoint.py.151: total_norm: 23.479404 [I 2022-10-09 18:54:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 180/2304 [I 2022-10-09 18:54:41] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:03 [I 2022-10-09 18:54:41] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00323 [I 2022-10-09 18:54:41] ...trap/engines/engine.py.230: loss: 0.18388 [I 2022-10-09 18:54:41] ...optimizers/trijoint.py.151: total_norm: 33.131676 [I 2022-10-09 18:54:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 190/2304 [I 2022-10-09 18:54:50] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:10 [I 2022-10-09 18:54:50] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00314 [I 2022-10-09 18:54:50] ...trap/engines/engine.py.230: loss: 0.15361 [I 2022-10-09 18:54:50] ...optimizers/trijoint.py.151: total_norm: 14.433907 [I 2022-10-09 18:54:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 200/2304 [I 2022-10-09 18:54:58] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:49 [I 2022-10-09 18:54:58] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00315 [I 2022-10-09 18:54:58] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-09 18:54:58] ...optimizers/trijoint.py.151: total_norm: 10.090283 [I 2022-10-09 18:55:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 210/2304 [I 2022-10-09 18:55:07] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:33 [I 2022-10-09 18:55:07] ...trap/engines/engine.py.229: process: 0.82657 | load: 0.00307 [I 2022-10-09 18:55:07] ...trap/engines/engine.py.230: loss: 0.08987 [I 2022-10-09 18:55:07] ...optimizers/trijoint.py.151: total_norm: 7.405520 [I 2022-10-09 18:55:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 220/2304 [I 2022-10-09 18:55:15] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:22 [I 2022-10-09 18:55:15] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00317 [I 2022-10-09 18:55:15] ...trap/engines/engine.py.230: loss: 0.15920 [I 2022-10-09 18:55:15] ...optimizers/trijoint.py.151: total_norm: 14.356327 [I 2022-10-09 18:55:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 230/2304 [I 2022-10-09 18:55:24] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:16 [I 2022-10-09 18:55:24] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00304 [I 2022-10-09 18:55:24] ...trap/engines/engine.py.230: loss: 0.17758 [I 2022-10-09 18:55:24] ...optimizers/trijoint.py.151: total_norm: 21.023178 [I 2022-10-09 18:55:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 240/2304 [I 2022-10-09 18:55:32] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:00 [I 2022-10-09 18:55:32] ...trap/engines/engine.py.229: process: 0.85352 | load: 0.00299 [I 2022-10-09 18:55:32] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-09 18:55:32] ...optimizers/trijoint.py.151: total_norm: 9.096496 [I 2022-10-09 18:55:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 250/2304 [I 2022-10-09 18:55:40] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:09 [I 2022-10-09 18:55:40] ...trap/engines/engine.py.229: process: 0.86770 | load: 0.00304 [I 2022-10-09 18:55:40] ...trap/engines/engine.py.230: loss: 0.16238 [I 2022-10-09 18:55:40] ...optimizers/trijoint.py.151: total_norm: 15.586427 [I 2022-10-09 18:55:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 260/2304 [I 2022-10-09 18:55:49] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:28:57 [I 2022-10-09 18:55:49] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00322 [I 2022-10-09 18:55:49] ...trap/engines/engine.py.230: loss: 0.13845 [I 2022-10-09 18:55:49] ...optimizers/trijoint.py.151: total_norm: 19.988152 [I 2022-10-09 18:55:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 270/2304 [I 2022-10-09 18:55:57] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:47 [I 2022-10-09 18:55:57] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00300 [I 2022-10-09 18:55:57] ...trap/engines/engine.py.230: loss: 0.11888 [I 2022-10-09 18:55:57] ...optimizers/trijoint.py.151: total_norm: 17.499054 [I 2022-10-09 18:56:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 280/2304 [I 2022-10-09 18:56:06] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:36 [I 2022-10-09 18:56:06] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00300 [I 2022-10-09 18:56:06] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-09 18:56:06] ...optimizers/trijoint.py.151: total_norm: 8.337788 [I 2022-10-09 18:56:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 290/2304 [I 2022-10-09 18:56:14] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:29 [I 2022-10-09 18:56:14] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00301 [I 2022-10-09 18:56:14] ...trap/engines/engine.py.230: loss: 0.14870 [I 2022-10-09 18:56:14] ...optimizers/trijoint.py.151: total_norm: 9.446468 [I 2022-10-09 18:56:23] ...trap/engines/engine.py.224: train: epoch 51 | batch 300/2304 [I 2022-10-09 18:56:23] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:22 [I 2022-10-09 18:56:23] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00302 [I 2022-10-09 18:56:23] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-09 18:56:23] ...optimizers/trijoint.py.151: total_norm: 9.450553 [I 2022-10-09 18:56:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 310/2304 [I 2022-10-09 18:56:31] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:03 [I 2022-10-09 18:56:31] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00298 [I 2022-10-09 18:56:31] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-09 18:56:31] ...optimizers/trijoint.py.151: total_norm: 14.911931 [I 2022-10-09 18:56:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 320/2304 [I 2022-10-09 18:56:40] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:27:48 [I 2022-10-09 18:56:40] ...trap/engines/engine.py.229: process: 0.82646 | load: 0.00311 [I 2022-10-09 18:56:40] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-09 18:56:40] ...optimizers/trijoint.py.151: total_norm: 15.094419 [I 2022-10-09 18:56:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 330/2304 [I 2022-10-09 18:56:48] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:01 [I 2022-10-09 18:56:48] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00333 [I 2022-10-09 18:56:48] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-09 18:56:48] ...optimizers/trijoint.py.151: total_norm: 13.620761 [I 2022-10-09 18:56:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 340/2304 [I 2022-10-09 18:56:57] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:53 [I 2022-10-09 18:56:57] ...trap/engines/engine.py.229: process: 0.86126 | load: 0.00301 [I 2022-10-09 18:56:57] ...trap/engines/engine.py.230: loss: 0.12247 [I 2022-10-09 18:56:57] ...optimizers/trijoint.py.151: total_norm: 18.181000 [I 2022-10-09 18:57:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 350/2304 [I 2022-10-09 18:57:05] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:34 [I 2022-10-09 18:57:05] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00297 [I 2022-10-09 18:57:05] ...trap/engines/engine.py.230: loss: 0.20219 [I 2022-10-09 18:57:05] ...optimizers/trijoint.py.151: total_norm: 45.296989 [I 2022-10-09 18:57:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 360/2304 [I 2022-10-09 18:57:14] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:30 [I 2022-10-09 18:57:14] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00299 [I 2022-10-09 18:57:14] ...trap/engines/engine.py.230: loss: 0.12794 [I 2022-10-09 18:57:14] ...optimizers/trijoint.py.151: total_norm: 8.694216 [I 2022-10-09 18:57:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 370/2304 [I 2022-10-09 18:57:22] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:11 [I 2022-10-09 18:57:22] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00304 [I 2022-10-09 18:57:22] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-09 18:57:22] ...optimizers/trijoint.py.151: total_norm: 63.068367 [I 2022-10-09 18:57:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 380/2304 [I 2022-10-09 18:57:31] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:08 [I 2022-10-09 18:57:31] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00300 [I 2022-10-09 18:57:31] ...trap/engines/engine.py.230: loss: 0.10996 [I 2022-10-09 18:57:31] ...optimizers/trijoint.py.151: total_norm: 12.428509 [I 2022-10-09 18:57:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 390/2304 [I 2022-10-09 18:57:39] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:26:59 [I 2022-10-09 18:57:39] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00299 [I 2022-10-09 18:57:39] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-09 18:57:39] ...optimizers/trijoint.py.151: total_norm: 14.322564 [I 2022-10-09 18:57:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 400/2304 [I 2022-10-09 18:57:48] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:38 [I 2022-10-09 18:57:48] ...trap/engines/engine.py.229: process: 0.82782 | load: 0.00297 [I 2022-10-09 18:57:48] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-09 18:57:48] ...optimizers/trijoint.py.151: total_norm: 20.698671 [I 2022-10-09 18:57:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 410/2304 [I 2022-10-09 18:57:56] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:50 [I 2022-10-09 18:57:56] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00311 [I 2022-10-09 18:57:56] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-09 18:57:56] ...optimizers/trijoint.py.151: total_norm: 13.279960 [I 2022-10-09 18:58:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 420/2304 [I 2022-10-09 18:58:05] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:45 [I 2022-10-09 18:58:05] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00310 [I 2022-10-09 18:58:05] ...trap/engines/engine.py.230: loss: 0.16364 [I 2022-10-09 18:58:05] ...optimizers/trijoint.py.151: total_norm: 19.414280 [I 2022-10-09 18:58:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 430/2304 [I 2022-10-09 18:58:13] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:53 [I 2022-10-09 18:58:13] ...trap/engines/engine.py.229: process: 0.86462 | load: 0.00342 [I 2022-10-09 18:58:13] ...trap/engines/engine.py.230: loss: 0.19500 [I 2022-10-09 18:58:13] ...optimizers/trijoint.py.151: total_norm: 15.452971 [I 2022-10-09 18:58:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 440/2304 [I 2022-10-09 18:58:22] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:38 [I 2022-10-09 18:58:22] ...trap/engines/engine.py.229: process: 0.86716 | load: 0.00308 [I 2022-10-09 18:58:22] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-09 18:58:22] ...optimizers/trijoint.py.151: total_norm: 12.279336 [I 2022-10-09 18:58:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 450/2304 [I 2022-10-09 18:58:31] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:20 [I 2022-10-09 18:58:31] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00311 [I 2022-10-09 18:58:31] ...trap/engines/engine.py.230: loss: 0.21048 [I 2022-10-09 18:58:31] ...optimizers/trijoint.py.151: total_norm: 17.538490 [I 2022-10-09 18:58:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 460/2304 [I 2022-10-09 18:58:39] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:15 [I 2022-10-09 18:58:39] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00311 [I 2022-10-09 18:58:39] ...trap/engines/engine.py.230: loss: 0.17463 [I 2022-10-09 18:58:39] ...optimizers/trijoint.py.151: total_norm: 9.182952 [I 2022-10-09 18:58:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 470/2304 [I 2022-10-09 18:58:48] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:06 [I 2022-10-09 18:58:48] ...trap/engines/engine.py.229: process: 0.85746 | load: 0.00312 [I 2022-10-09 18:58:48] ...trap/engines/engine.py.230: loss: 0.20281 [I 2022-10-09 18:58:48] ...optimizers/trijoint.py.151: total_norm: 16.447773 [I 2022-10-09 18:58:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 480/2304 [I 2022-10-09 18:58:56] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:26:06 [I 2022-10-09 18:58:56] ...trap/engines/engine.py.229: process: 0.85985 | load: 0.00312 [I 2022-10-09 18:58:56] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-09 18:58:56] ...optimizers/trijoint.py.151: total_norm: 16.161266 [I 2022-10-09 18:59:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 490/2304 [I 2022-10-09 18:59:05] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:49 [I 2022-10-09 18:59:05] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00309 [I 2022-10-09 18:59:05] ...trap/engines/engine.py.230: loss: 0.11418 [I 2022-10-09 18:59:05] ...optimizers/trijoint.py.151: total_norm: 7.658311 [I 2022-10-09 18:59:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 500/2304 [I 2022-10-09 18:59:13] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:41 [I 2022-10-09 18:59:13] ...trap/engines/engine.py.229: process: 0.86482 | load: 0.00317 [I 2022-10-09 18:59:13] ...trap/engines/engine.py.230: loss: 0.16874 [I 2022-10-09 18:59:13] ...optimizers/trijoint.py.151: total_norm: 10.489305 [I 2022-10-09 18:59:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 510/2304 [I 2022-10-09 18:59:22] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:29 [I 2022-10-09 18:59:22] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00327 [I 2022-10-09 18:59:22] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-09 18:59:22] ...optimizers/trijoint.py.151: total_norm: 11.032076 [I 2022-10-09 18:59:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 520/2304 [I 2022-10-09 18:59:30] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:34 [I 2022-10-09 18:59:30] ...trap/engines/engine.py.229: process: 0.87393 | load: 0.00323 [I 2022-10-09 18:59:30] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-09 18:59:30] ...optimizers/trijoint.py.151: total_norm: 23.225023 [I 2022-10-09 18:59:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 530/2304 [I 2022-10-09 18:59:39] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:12 [I 2022-10-09 18:59:39] ...trap/engines/engine.py.229: process: 0.83377 | load: 0.00313 [I 2022-10-09 18:59:39] ...trap/engines/engine.py.230: loss: 0.21807 [I 2022-10-09 18:59:39] ...optimizers/trijoint.py.151: total_norm: 20.315325 [I 2022-10-09 18:59:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 540/2304 [I 2022-10-09 18:59:48] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:18 [I 2022-10-09 18:59:48] ...trap/engines/engine.py.229: process: 0.87586 | load: 0.00311 [I 2022-10-09 18:59:48] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-09 18:59:48] ...optimizers/trijoint.py.151: total_norm: 18.422049 [I 2022-10-09 18:59:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 550/2304 [I 2022-10-09 18:59:56] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:54 [I 2022-10-09 18:59:56] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00308 [I 2022-10-09 18:59:56] ...trap/engines/engine.py.230: loss: 0.19771 [I 2022-10-09 18:59:56] ...optimizers/trijoint.py.151: total_norm: 20.446575 [I 2022-10-09 19:00:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 560/2304 [I 2022-10-09 19:00:05] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:49 [I 2022-10-09 19:00:05] ...trap/engines/engine.py.229: process: 0.86131 | load: 0.00325 [I 2022-10-09 19:00:05] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-09 19:00:05] ...optimizers/trijoint.py.151: total_norm: 13.989801 [I 2022-10-09 19:00:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 570/2304 [I 2022-10-09 19:00:13] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:37 [I 2022-10-09 19:00:13] ...trap/engines/engine.py.229: process: 0.85392 | load: 0.00321 [I 2022-10-09 19:00:13] ...trap/engines/engine.py.230: loss: 0.11275 [I 2022-10-09 19:00:13] ...optimizers/trijoint.py.151: total_norm: 6.319105 [I 2022-10-09 19:00:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 580/2304 [I 2022-10-09 19:00:22] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:26 [I 2022-10-09 19:00:22] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00312 [I 2022-10-09 19:00:22] ...trap/engines/engine.py.230: loss: 0.09957 [I 2022-10-09 19:00:22] ...optimizers/trijoint.py.151: total_norm: 7.037679 [I 2022-10-09 19:00:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 590/2304 [I 2022-10-09 19:00:30] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:23 [I 2022-10-09 19:00:30] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00312 [I 2022-10-09 19:00:30] ...trap/engines/engine.py.230: loss: 0.11919 [I 2022-10-09 19:00:30] ...optimizers/trijoint.py.151: total_norm: 17.819105 [I 2022-10-09 19:00:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 600/2304 [I 2022-10-09 19:00:39] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:18 [I 2022-10-09 19:00:39] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00323 [I 2022-10-09 19:00:39] ...trap/engines/engine.py.230: loss: 0.09785 [I 2022-10-09 19:00:39] ...optimizers/trijoint.py.151: total_norm: 7.679947 [I 2022-10-09 19:00:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 610/2304 [I 2022-10-09 19:00:47] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:23:56 [I 2022-10-09 19:00:47] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00312 [I 2022-10-09 19:00:47] ...trap/engines/engine.py.230: loss: 0.18679 [I 2022-10-09 19:00:47] ...optimizers/trijoint.py.151: total_norm: 13.714811 [I 2022-10-09 19:00:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 620/2304 [I 2022-10-09 19:00:56] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:55 [I 2022-10-09 19:00:56] ...trap/engines/engine.py.229: process: 0.86290 | load: 0.00312 [I 2022-10-09 19:00:56] ...trap/engines/engine.py.230: loss: 0.13536 [I 2022-10-09 19:00:56] ...optimizers/trijoint.py.151: total_norm: 11.475729 [I 2022-10-09 19:01:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 630/2304 [I 2022-10-09 19:01:04] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:49 [I 2022-10-09 19:01:04] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00307 [I 2022-10-09 19:01:04] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-09 19:01:04] ...optimizers/trijoint.py.151: total_norm: 19.874582 [I 2022-10-09 19:01:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 640/2304 [I 2022-10-09 19:01:13] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:38 [I 2022-10-09 19:01:13] ...trap/engines/engine.py.229: process: 0.84998 | load: 0.00312 [I 2022-10-09 19:01:13] ...trap/engines/engine.py.230: loss: 0.11369 [I 2022-10-09 19:01:13] ...optimizers/trijoint.py.151: total_norm: 14.152627 [I 2022-10-09 19:01:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 650/2304 [I 2022-10-09 19:01:22] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:39 [I 2022-10-09 19:01:22] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00318 [I 2022-10-09 19:01:22] ...trap/engines/engine.py.230: loss: 0.10357 [I 2022-10-09 19:01:22] ...optimizers/trijoint.py.151: total_norm: 9.965497 [I 2022-10-09 19:01:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 660/2304 [I 2022-10-09 19:01:30] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:27 [I 2022-10-09 19:01:30] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00314 [I 2022-10-09 19:01:30] ...trap/engines/engine.py.230: loss: 0.09474 [I 2022-10-09 19:01:30] ...optimizers/trijoint.py.151: total_norm: 7.086511 [I 2022-10-09 19:01:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 670/2304 [I 2022-10-09 19:01:39] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:12 [I 2022-10-09 19:01:39] ...trap/engines/engine.py.229: process: 0.86581 | load: 0.00312 [I 2022-10-09 19:01:39] ...trap/engines/engine.py.230: loss: 0.18899 [I 2022-10-09 19:01:39] ...optimizers/trijoint.py.151: total_norm: 32.914471 [I 2022-10-09 19:01:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 680/2304 [I 2022-10-09 19:01:47] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:59 [I 2022-10-09 19:01:47] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00313 [I 2022-10-09 19:01:47] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-09 19:01:47] ...optimizers/trijoint.py.151: total_norm: 8.978126 [I 2022-10-09 19:01:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 690/2304 [I 2022-10-09 19:01:56] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:53 [I 2022-10-09 19:01:56] ...trap/engines/engine.py.229: process: 0.86315 | load: 0.00313 [I 2022-10-09 19:01:56] ...trap/engines/engine.py.230: loss: 0.16603 [I 2022-10-09 19:01:56] ...optimizers/trijoint.py.151: total_norm: 13.749498 [I 2022-10-09 19:02:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 700/2304 [I 2022-10-09 19:02:04] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:53 [I 2022-10-09 19:02:04] ...trap/engines/engine.py.229: process: 0.85839 | load: 0.00320 [I 2022-10-09 19:02:04] ...trap/engines/engine.py.230: loss: 0.11392 [I 2022-10-09 19:02:04] ...optimizers/trijoint.py.151: total_norm: 13.218266 [I 2022-10-09 19:02:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 710/2304 [I 2022-10-09 19:02:13] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:42 [I 2022-10-09 19:02:13] ...trap/engines/engine.py.229: process: 0.85516 | load: 0.00314 [I 2022-10-09 19:02:13] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-09 19:02:13] ...optimizers/trijoint.py.151: total_norm: 10.916437 [I 2022-10-09 19:02:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 720/2304 [I 2022-10-09 19:02:21] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:36 [I 2022-10-09 19:02:21] ...trap/engines/engine.py.229: process: 0.86086 | load: 0.00312 [I 2022-10-09 19:02:21] ...trap/engines/engine.py.230: loss: 0.18834 [I 2022-10-09 19:02:21] ...optimizers/trijoint.py.151: total_norm: 23.306616 [I 2022-10-09 19:02:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 730/2304 [I 2022-10-09 19:02:30] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:20 [I 2022-10-09 19:02:30] ...trap/engines/engine.py.229: process: 0.84844 | load: 0.00303 [I 2022-10-09 19:02:30] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-09 19:02:30] ...optimizers/trijoint.py.151: total_norm: 15.274423 [I 2022-10-09 19:02:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 740/2304 [I 2022-10-09 19:02:38] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:21:58 [I 2022-10-09 19:02:38] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00298 [I 2022-10-09 19:02:38] ...trap/engines/engine.py.230: loss: 0.15456 [I 2022-10-09 19:02:38] ...optimizers/trijoint.py.151: total_norm: 34.392971 [I 2022-10-09 19:02:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 750/2304 [I 2022-10-09 19:02:47] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:46 [I 2022-10-09 19:02:47] ...trap/engines/engine.py.229: process: 0.83856 | load: 0.00299 [I 2022-10-09 19:02:47] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-09 19:02:47] ...optimizers/trijoint.py.151: total_norm: 7.442929 [I 2022-10-09 19:02:55] ...trap/engines/engine.py.224: train: epoch 51 | batch 760/2304 [I 2022-10-09 19:02:55] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:47 [I 2022-10-09 19:02:55] ...trap/engines/engine.py.229: process: 0.85347 | load: 0.00298 [I 2022-10-09 19:02:55] ...trap/engines/engine.py.230: loss: 0.18351 [I 2022-10-09 19:02:55] ...optimizers/trijoint.py.151: total_norm: 14.621804 [I 2022-10-09 19:03:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 770/2304 [I 2022-10-09 19:03:04] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:38 [I 2022-10-09 19:03:04] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00296 [I 2022-10-09 19:03:04] ...trap/engines/engine.py.230: loss: 0.18529 [I 2022-10-09 19:03:04] ...optimizers/trijoint.py.151: total_norm: 23.644970 [I 2022-10-09 19:03:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 780/2304 [I 2022-10-09 19:03:12] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:29 [I 2022-10-09 19:03:12] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00303 [I 2022-10-09 19:03:12] ...trap/engines/engine.py.230: loss: 0.12154 [I 2022-10-09 19:03:12] ...optimizers/trijoint.py.151: total_norm: 16.526865 [I 2022-10-09 19:03:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 790/2304 [I 2022-10-09 19:03:21] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:23 [I 2022-10-09 19:03:21] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00302 [I 2022-10-09 19:03:21] ...trap/engines/engine.py.230: loss: 0.29993 [I 2022-10-09 19:03:21] ...optimizers/trijoint.py.151: total_norm: 24.777588 [I 2022-10-09 19:03:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 800/2304 [I 2022-10-09 19:03:29] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:15 [I 2022-10-09 19:03:29] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00298 [I 2022-10-09 19:03:29] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-09 19:03:29] ...optimizers/trijoint.py.151: total_norm: 9.268888 [I 2022-10-09 19:03:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 810/2304 [I 2022-10-09 19:03:38] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:02 [I 2022-10-09 19:03:38] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00293 [I 2022-10-09 19:03:38] ...trap/engines/engine.py.230: loss: 0.15055 [I 2022-10-09 19:03:38] ...optimizers/trijoint.py.151: total_norm: 28.634607 [I 2022-10-09 19:03:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 820/2304 [I 2022-10-09 19:03:46] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:54 [I 2022-10-09 19:03:46] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00303 [I 2022-10-09 19:03:46] ...trap/engines/engine.py.230: loss: 0.13514 [I 2022-10-09 19:03:46] ...optimizers/trijoint.py.151: total_norm: 11.165687 [I 2022-10-09 19:03:55] ...trap/engines/engine.py.224: train: epoch 51 | batch 830/2304 [I 2022-10-09 19:03:55] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:45 [I 2022-10-09 19:03:55] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00295 [I 2022-10-09 19:03:55] ...trap/engines/engine.py.230: loss: 0.14183 [I 2022-10-09 19:03:55] ...optimizers/trijoint.py.151: total_norm: 10.139660 [I 2022-10-09 19:04:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 840/2304 [I 2022-10-09 19:04:03] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:33 [I 2022-10-09 19:04:03] ...trap/engines/engine.py.229: process: 0.84594 | load: 0.00293 [I 2022-10-09 19:04:03] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-09 19:04:03] ...optimizers/trijoint.py.151: total_norm: 10.288762 [I 2022-10-09 19:04:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 850/2304 [I 2022-10-09 19:04:12] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:33 [I 2022-10-09 19:04:12] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00315 [I 2022-10-09 19:04:12] ...trap/engines/engine.py.230: loss: 0.19525 [I 2022-10-09 19:04:12] ...optimizers/trijoint.py.151: total_norm: 18.541855 [I 2022-10-09 19:04:20] ...trap/engines/engine.py.224: train: epoch 51 | batch 860/2304 [I 2022-10-09 19:04:20] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:18 [I 2022-10-09 19:04:20] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00300 [I 2022-10-09 19:04:20] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-09 19:04:20] ...optimizers/trijoint.py.151: total_norm: 13.435501 [I 2022-10-09 19:04:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 870/2304 [I 2022-10-09 19:04:29] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:06 [I 2022-10-09 19:04:29] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00299 [I 2022-10-09 19:04:29] ...trap/engines/engine.py.230: loss: 0.13833 [I 2022-10-09 19:04:29] ...optimizers/trijoint.py.151: total_norm: 40.562298 [I 2022-10-09 19:04:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 880/2304 [I 2022-10-09 19:04:37] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:03 [I 2022-10-09 19:04:37] ...trap/engines/engine.py.229: process: 0.85523 | load: 0.00299 [I 2022-10-09 19:04:37] ...trap/engines/engine.py.230: loss: 0.13388 [I 2022-10-09 19:04:37] ...optimizers/trijoint.py.151: total_norm: 20.248503 [I 2022-10-09 19:04:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 890/2304 [I 2022-10-09 19:04:46] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:58 [I 2022-10-09 19:04:46] ...trap/engines/engine.py.229: process: 0.85115 | load: 0.00300 [I 2022-10-09 19:04:46] ...trap/engines/engine.py.230: loss: 0.10458 [I 2022-10-09 19:04:46] ...optimizers/trijoint.py.151: total_norm: 9.884002 [I 2022-10-09 19:04:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 900/2304 [I 2022-10-09 19:04:54] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:52 [I 2022-10-09 19:04:54] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00299 [I 2022-10-09 19:04:54] ...trap/engines/engine.py.230: loss: 0.19905 [I 2022-10-09 19:04:54] ...optimizers/trijoint.py.151: total_norm: 18.033234 [I 2022-10-09 19:05:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 910/2304 [I 2022-10-09 19:05:02] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:41 [I 2022-10-09 19:05:02] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00299 [I 2022-10-09 19:05:02] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-09 19:05:02] ...optimizers/trijoint.py.151: total_norm: 9.564515 [I 2022-10-09 19:05:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 920/2304 [I 2022-10-09 19:05:11] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:30 [I 2022-10-09 19:05:11] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00321 [I 2022-10-09 19:05:11] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-09 19:05:11] ...optimizers/trijoint.py.151: total_norm: 11.608763 [I 2022-10-09 19:05:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 930/2304 [I 2022-10-09 19:05:19] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:19 [I 2022-10-09 19:05:19] ...trap/engines/engine.py.229: process: 0.83879 | load: 0.00302 [I 2022-10-09 19:05:19] ...trap/engines/engine.py.230: loss: 0.25719 [I 2022-10-09 19:05:19] ...optimizers/trijoint.py.151: total_norm: 23.409760 [I 2022-10-09 19:05:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 940/2304 [I 2022-10-09 19:05:28] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:16 [I 2022-10-09 19:05:28] ...trap/engines/engine.py.229: process: 0.85033 | load: 0.00301 [I 2022-10-09 19:05:28] ...trap/engines/engine.py.230: loss: 0.15315 [I 2022-10-09 19:05:28] ...optimizers/trijoint.py.151: total_norm: 19.732752 [I 2022-10-09 19:05:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 950/2304 [I 2022-10-09 19:05:36] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:03 [I 2022-10-09 19:05:36] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00303 [I 2022-10-09 19:05:36] ...trap/engines/engine.py.230: loss: 0.14778 [I 2022-10-09 19:05:36] ...optimizers/trijoint.py.151: total_norm: 9.694777 [I 2022-10-09 19:05:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 960/2304 [I 2022-10-09 19:05:45] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:04 [I 2022-10-09 19:05:45] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00323 [I 2022-10-09 19:05:45] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-09 19:05:45] ...optimizers/trijoint.py.151: total_norm: 11.931266 [I 2022-10-09 19:05:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 970/2304 [I 2022-10-09 19:05:54] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:58 [I 2022-10-09 19:05:54] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00310 [I 2022-10-09 19:05:54] ...trap/engines/engine.py.230: loss: 0.14950 [I 2022-10-09 19:05:54] ...optimizers/trijoint.py.151: total_norm: 23.857323 [I 2022-10-09 19:06:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 980/2304 [I 2022-10-09 19:06:02] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:42 [I 2022-10-09 19:06:02] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00317 [I 2022-10-09 19:06:02] ...trap/engines/engine.py.230: loss: 0.17751 [I 2022-10-09 19:06:02] ...optimizers/trijoint.py.151: total_norm: 19.523481 [I 2022-10-09 19:06:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 990/2304 [I 2022-10-09 19:06:11] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:47 [I 2022-10-09 19:06:11] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00491 [I 2022-10-09 19:06:11] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-09 19:06:11] ...optimizers/trijoint.py.151: total_norm: 11.981648 [I 2022-10-09 19:06:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 1000/2304 [I 2022-10-09 19:06:19] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:34 [I 2022-10-09 19:06:19] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00315 [I 2022-10-09 19:06:19] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-09 19:06:19] ...optimizers/trijoint.py.151: total_norm: 20.603849 [I 2022-10-09 19:06:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 1010/2304 [I 2022-10-09 19:06:28] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:19 [I 2022-10-09 19:06:28] ...trap/engines/engine.py.229: process: 0.84278 | load: 0.00317 [I 2022-10-09 19:06:28] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-09 19:06:28] ...optimizers/trijoint.py.151: total_norm: 13.819403 [I 2022-10-09 19:06:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 1020/2304 [I 2022-10-09 19:06:36] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:12 [I 2022-10-09 19:06:36] ...trap/engines/engine.py.229: process: 0.85488 | load: 0.00316 [I 2022-10-09 19:06:36] ...trap/engines/engine.py.230: loss: 0.17645 [I 2022-10-09 19:06:36] ...optimizers/trijoint.py.151: total_norm: 27.199472 [I 2022-10-09 19:06:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 1030/2304 [I 2022-10-09 19:06:45] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:01 [I 2022-10-09 19:06:45] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00311 [I 2022-10-09 19:06:45] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-09 19:06:45] ...optimizers/trijoint.py.151: total_norm: 18.690069 [I 2022-10-09 19:06:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1040/2304 [I 2022-10-09 19:06:53] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:57 [I 2022-10-09 19:06:53] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00315 [I 2022-10-09 19:06:53] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-09 19:06:53] ...optimizers/trijoint.py.151: total_norm: 14.904576 [I 2022-10-09 19:07:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 1050/2304 [I 2022-10-09 19:07:02] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:52 [I 2022-10-09 19:07:02] ...trap/engines/engine.py.229: process: 0.85263 | load: 0.00312 [I 2022-10-09 19:07:02] ...trap/engines/engine.py.230: loss: 0.14906 [I 2022-10-09 19:07:02] ...optimizers/trijoint.py.151: total_norm: 13.338819 [I 2022-10-09 19:07:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 1060/2304 [I 2022-10-09 19:07:10] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:40 [I 2022-10-09 19:07:10] ...trap/engines/engine.py.229: process: 0.85669 | load: 0.00319 [I 2022-10-09 19:07:10] ...trap/engines/engine.py.230: loss: 0.18995 [I 2022-10-09 19:07:10] ...optimizers/trijoint.py.151: total_norm: 28.611240 [I 2022-10-09 19:07:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 1070/2304 [I 2022-10-09 19:07:19] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:35 [I 2022-10-09 19:07:19] ...trap/engines/engine.py.229: process: 0.86088 | load: 0.00314 [I 2022-10-09 19:07:19] ...trap/engines/engine.py.230: loss: 0.19552 [I 2022-10-09 19:07:19] ...optimizers/trijoint.py.151: total_norm: 16.247416 [I 2022-10-09 19:07:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 1080/2304 [I 2022-10-09 19:07:27] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:23 [I 2022-10-09 19:07:27] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00309 [I 2022-10-09 19:07:27] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-09 19:07:27] ...optimizers/trijoint.py.151: total_norm: 22.530649 [I 2022-10-09 19:07:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 1090/2304 [I 2022-10-09 19:07:36] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:15 [I 2022-10-09 19:07:36] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00302 [I 2022-10-09 19:07:36] ...trap/engines/engine.py.230: loss: 0.26964 [I 2022-10-09 19:07:36] ...optimizers/trijoint.py.151: total_norm: 14.889248 [I 2022-10-09 19:07:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 1100/2304 [I 2022-10-09 19:07:45] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:06 [I 2022-10-09 19:07:45] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00310 [I 2022-10-09 19:07:45] ...trap/engines/engine.py.230: loss: 0.15131 [I 2022-10-09 19:07:45] ...optimizers/trijoint.py.151: total_norm: 9.061026 [I 2022-10-09 19:07:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1110/2304 [I 2022-10-09 19:07:53] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:55 [I 2022-10-09 19:07:53] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00311 [I 2022-10-09 19:07:53] ...trap/engines/engine.py.230: loss: 0.18541 [I 2022-10-09 19:07:53] ...optimizers/trijoint.py.151: total_norm: 20.678585 [I 2022-10-09 19:08:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 1120/2304 [I 2022-10-09 19:08:02] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:45 [I 2022-10-09 19:08:02] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00317 [I 2022-10-09 19:08:02] ...trap/engines/engine.py.230: loss: 0.18000 [I 2022-10-09 19:08:02] ...optimizers/trijoint.py.151: total_norm: 23.367332 [I 2022-10-09 19:08:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 1130/2304 [I 2022-10-09 19:08:10] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:36 [I 2022-10-09 19:08:10] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00313 [I 2022-10-09 19:08:10] ...trap/engines/engine.py.230: loss: 0.12073 [I 2022-10-09 19:08:10] ...optimizers/trijoint.py.151: total_norm: 16.572800 [I 2022-10-09 19:08:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 1140/2304 [I 2022-10-09 19:08:19] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:29 [I 2022-10-09 19:08:19] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00314 [I 2022-10-09 19:08:19] ...trap/engines/engine.py.230: loss: 0.13097 [I 2022-10-09 19:08:19] ...optimizers/trijoint.py.151: total_norm: 16.009104 [I 2022-10-09 19:08:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 1150/2304 [I 2022-10-09 19:08:27] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:28 [I 2022-10-09 19:08:27] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00313 [I 2022-10-09 19:08:27] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-09 19:08:27] ...optimizers/trijoint.py.151: total_norm: 12.469156 [I 2022-10-09 19:08:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 1160/2304 [I 2022-10-09 19:08:36] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:16 [I 2022-10-09 19:08:36] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00313 [I 2022-10-09 19:08:36] ...trap/engines/engine.py.230: loss: 0.18513 [I 2022-10-09 19:08:36] ...optimizers/trijoint.py.151: total_norm: 14.677400 [I 2022-10-09 19:08:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 1170/2304 [I 2022-10-09 19:08:44] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:09 [I 2022-10-09 19:08:44] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00313 [I 2022-10-09 19:08:44] ...trap/engines/engine.py.230: loss: 0.13156 [I 2022-10-09 19:08:44] ...optimizers/trijoint.py.151: total_norm: 13.068228 [I 2022-10-09 19:08:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1180/2304 [I 2022-10-09 19:08:53] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:16:01 [I 2022-10-09 19:08:53] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00316 [I 2022-10-09 19:08:53] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-09 19:08:53] ...optimizers/trijoint.py.151: total_norm: 11.656302 [I 2022-10-09 19:09:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1190/2304 [I 2022-10-09 19:09:01] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:42 [I 2022-10-09 19:09:01] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00316 [I 2022-10-09 19:09:01] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-09 19:09:01] ...optimizers/trijoint.py.151: total_norm: 11.604067 [I 2022-10-09 19:09:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 1200/2304 [I 2022-10-09 19:09:10] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:39 [I 2022-10-09 19:09:10] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00312 [I 2022-10-09 19:09:10] ...trap/engines/engine.py.230: loss: 0.20305 [I 2022-10-09 19:09:10] ...optimizers/trijoint.py.151: total_norm: 25.338968 [I 2022-10-09 19:09:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 1210/2304 [I 2022-10-09 19:09:18] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:29 [I 2022-10-09 19:09:18] ...trap/engines/engine.py.229: process: 0.84854 | load: 0.00312 [I 2022-10-09 19:09:18] ...trap/engines/engine.py.230: loss: 0.22280 [I 2022-10-09 19:09:18] ...optimizers/trijoint.py.151: total_norm: 41.524792 [I 2022-10-09 19:09:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 1220/2304 [I 2022-10-09 19:09:27] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:23 [I 2022-10-09 19:09:27] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00305 [I 2022-10-09 19:09:27] ...trap/engines/engine.py.230: loss: 0.14470 [I 2022-10-09 19:09:27] ...optimizers/trijoint.py.151: total_norm: 12.565461 [I 2022-10-09 19:09:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 1230/2304 [I 2022-10-09 19:09:35] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:17 [I 2022-10-09 19:09:35] ...trap/engines/engine.py.229: process: 0.86041 | load: 0.00312 [I 2022-10-09 19:09:35] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-09 19:09:35] ...optimizers/trijoint.py.151: total_norm: 17.251068 [I 2022-10-09 19:09:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 1240/2304 [I 2022-10-09 19:09:44] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:09 [I 2022-10-09 19:09:44] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00320 [I 2022-10-09 19:09:44] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-09 19:09:44] ...optimizers/trijoint.py.151: total_norm: 9.612950 [I 2022-10-09 19:09:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1250/2304 [I 2022-10-09 19:09:53] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:51 [I 2022-10-09 19:09:53] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00301 [I 2022-10-09 19:09:53] ...trap/engines/engine.py.230: loss: 0.24709 [I 2022-10-09 19:09:53] ...optimizers/trijoint.py.151: total_norm: 18.694103 [I 2022-10-09 19:10:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1260/2304 [I 2022-10-09 19:10:01] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:49 [I 2022-10-09 19:10:01] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00317 [I 2022-10-09 19:10:01] ...trap/engines/engine.py.230: loss: 0.16053 [I 2022-10-09 19:10:01] ...optimizers/trijoint.py.151: total_norm: 29.860725 [I 2022-10-09 19:10:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 1270/2304 [I 2022-10-09 19:10:10] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:39 [I 2022-10-09 19:10:10] ...trap/engines/engine.py.229: process: 0.85676 | load: 0.00320 [I 2022-10-09 19:10:10] ...trap/engines/engine.py.230: loss: 0.14581 [I 2022-10-09 19:10:10] ...optimizers/trijoint.py.151: total_norm: 23.800762 [I 2022-10-09 19:10:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 1280/2304 [I 2022-10-09 19:10:18] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:32 [I 2022-10-09 19:10:18] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00311 [I 2022-10-09 19:10:18] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-09 19:10:18] ...optimizers/trijoint.py.151: total_norm: 15.630326 [I 2022-10-09 19:10:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 1290/2304 [I 2022-10-09 19:10:27] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:20 [I 2022-10-09 19:10:27] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00301 [I 2022-10-09 19:10:27] ...trap/engines/engine.py.230: loss: 0.14033 [I 2022-10-09 19:10:27] ...optimizers/trijoint.py.151: total_norm: 21.319435 [I 2022-10-09 19:10:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 1300/2304 [I 2022-10-09 19:10:35] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:12 [I 2022-10-09 19:10:35] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00297 [I 2022-10-09 19:10:35] ...trap/engines/engine.py.230: loss: 0.18153 [I 2022-10-09 19:10:35] ...optimizers/trijoint.py.151: total_norm: 15.585296 [I 2022-10-09 19:10:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 1310/2304 [I 2022-10-09 19:10:44] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:03 [I 2022-10-09 19:10:44] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00305 [I 2022-10-09 19:10:44] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-09 19:10:44] ...optimizers/trijoint.py.151: total_norm: 18.346958 [I 2022-10-09 19:10:52] ...trap/engines/engine.py.224: train: epoch 51 | batch 1320/2304 [I 2022-10-09 19:10:52] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:51 [I 2022-10-09 19:10:52] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00299 [I 2022-10-09 19:10:52] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-09 19:10:52] ...optimizers/trijoint.py.151: total_norm: 8.133822 [I 2022-10-09 19:11:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1330/2304 [I 2022-10-09 19:11:01] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:39 [I 2022-10-09 19:11:01] ...trap/engines/engine.py.229: process: 0.83920 | load: 0.00293 [I 2022-10-09 19:11:01] ...trap/engines/engine.py.230: loss: 0.12560 [I 2022-10-09 19:11:01] ...optimizers/trijoint.py.151: total_norm: 8.106590 [I 2022-10-09 19:11:09] ...trap/engines/engine.py.224: train: epoch 51 | batch 1340/2304 [I 2022-10-09 19:11:09] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:38 [I 2022-10-09 19:11:09] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00291 [I 2022-10-09 19:11:09] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-09 19:11:09] ...optimizers/trijoint.py.151: total_norm: 19.697273 [I 2022-10-09 19:11:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 1350/2304 [I 2022-10-09 19:11:17] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:25 [I 2022-10-09 19:11:17] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00294 [I 2022-10-09 19:11:17] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-09 19:11:17] ...optimizers/trijoint.py.151: total_norm: 26.086115 [I 2022-10-09 19:11:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 1360/2304 [I 2022-10-09 19:11:26] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:18 [I 2022-10-09 19:11:26] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00294 [I 2022-10-09 19:11:26] ...trap/engines/engine.py.230: loss: 0.24168 [I 2022-10-09 19:11:26] ...optimizers/trijoint.py.151: total_norm: 26.724485 [I 2022-10-09 19:11:34] ...trap/engines/engine.py.224: train: epoch 51 | batch 1370/2304 [I 2022-10-09 19:11:34] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:07 [I 2022-10-09 19:11:34] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00293 [I 2022-10-09 19:11:34] ...trap/engines/engine.py.230: loss: 0.13345 [I 2022-10-09 19:11:34] ...optimizers/trijoint.py.151: total_norm: 11.637071 [I 2022-10-09 19:11:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 1380/2304 [I 2022-10-09 19:11:43] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:57 [I 2022-10-09 19:11:43] ...trap/engines/engine.py.229: process: 0.84430 | load: 0.00294 [I 2022-10-09 19:11:43] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-09 19:11:43] ...optimizers/trijoint.py.151: total_norm: 22.647331 [I 2022-10-09 19:11:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 1390/2304 [I 2022-10-09 19:11:51] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:49 [I 2022-10-09 19:11:51] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00296 [I 2022-10-09 19:11:51] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-09 19:11:51] ...optimizers/trijoint.py.151: total_norm: 16.300571 [I 2022-10-09 19:12:00] ...trap/engines/engine.py.224: train: epoch 51 | batch 1400/2304 [I 2022-10-09 19:12:00] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:37 [I 2022-10-09 19:12:00] ...trap/engines/engine.py.229: process: 0.83455 | load: 0.00292 [I 2022-10-09 19:12:00] ...trap/engines/engine.py.230: loss: 0.24262 [I 2022-10-09 19:12:00] ...optimizers/trijoint.py.151: total_norm: 21.564199 [I 2022-10-09 19:12:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 1410/2304 [I 2022-10-09 19:12:08] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:34 [I 2022-10-09 19:12:08] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00294 [I 2022-10-09 19:12:08] ...trap/engines/engine.py.230: loss: 0.14345 [I 2022-10-09 19:12:08] ...optimizers/trijoint.py.151: total_norm: 16.779196 [I 2022-10-09 19:12:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 1420/2304 [I 2022-10-09 19:12:17] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:28 [I 2022-10-09 19:12:17] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00291 [I 2022-10-09 19:12:17] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-09 19:12:17] ...optimizers/trijoint.py.151: total_norm: 21.839344 [I 2022-10-09 19:12:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 1430/2304 [I 2022-10-09 19:12:25] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:20 [I 2022-10-09 19:12:25] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00294 [I 2022-10-09 19:12:25] ...trap/engines/engine.py.230: loss: 0.16592 [I 2022-10-09 19:12:25] ...optimizers/trijoint.py.151: total_norm: 36.285755 [I 2022-10-09 19:12:34] ...trap/engines/engine.py.224: train: epoch 51 | batch 1440/2304 [I 2022-10-09 19:12:34] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:09 [I 2022-10-09 19:12:34] ...trap/engines/engine.py.229: process: 0.84054 | load: 0.00292 [I 2022-10-09 19:12:34] ...trap/engines/engine.py.230: loss: 0.14916 [I 2022-10-09 19:12:34] ...optimizers/trijoint.py.151: total_norm: 12.681171 [I 2022-10-09 19:12:42] ...trap/engines/engine.py.224: train: epoch 51 | batch 1450/2304 [I 2022-10-09 19:12:42] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:02 [I 2022-10-09 19:12:42] ...trap/engines/engine.py.229: process: 0.84967 | load: 0.00294 [I 2022-10-09 19:12:42] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-09 19:12:42] ...optimizers/trijoint.py.151: total_norm: 9.994204 [I 2022-10-09 19:12:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 1460/2304 [I 2022-10-09 19:12:51] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:51 [I 2022-10-09 19:12:51] ...trap/engines/engine.py.229: process: 0.83784 | load: 0.00294 [I 2022-10-09 19:12:51] ...trap/engines/engine.py.230: loss: 0.21551 [I 2022-10-09 19:12:51] ...optimizers/trijoint.py.151: total_norm: 18.718657 [I 2022-10-09 19:12:59] ...trap/engines/engine.py.224: train: epoch 51 | batch 1470/2304 [I 2022-10-09 19:12:59] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:45 [I 2022-10-09 19:12:59] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00293 [I 2022-10-09 19:12:59] ...trap/engines/engine.py.230: loss: 0.16835 [I 2022-10-09 19:12:59] ...optimizers/trijoint.py.151: total_norm: 13.068244 [I 2022-10-09 19:13:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 1480/2304 [I 2022-10-09 19:13:07] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:38 [I 2022-10-09 19:13:07] ...trap/engines/engine.py.229: process: 0.86123 | load: 0.00298 [I 2022-10-09 19:13:07] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-09 19:13:07] ...optimizers/trijoint.py.151: total_norm: 10.125364 [I 2022-10-09 19:13:16] ...trap/engines/engine.py.224: train: epoch 51 | batch 1490/2304 [I 2022-10-09 19:13:16] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:29 [I 2022-10-09 19:13:16] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00293 [I 2022-10-09 19:13:16] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-09 19:13:16] ...optimizers/trijoint.py.151: total_norm: 8.901447 [I 2022-10-09 19:13:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 1500/2304 [I 2022-10-09 19:13:24] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:22 [I 2022-10-09 19:13:24] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00296 [I 2022-10-09 19:13:24] ...trap/engines/engine.py.230: loss: 0.16257 [I 2022-10-09 19:13:24] ...optimizers/trijoint.py.151: total_norm: 14.839126 [I 2022-10-09 19:13:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 1510/2304 [I 2022-10-09 19:13:33] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:14 [I 2022-10-09 19:13:33] ...trap/engines/engine.py.229: process: 0.85990 | load: 0.00297 [I 2022-10-09 19:13:33] ...trap/engines/engine.py.230: loss: 0.15037 [I 2022-10-09 19:13:33] ...optimizers/trijoint.py.151: total_norm: 15.123141 [I 2022-10-09 19:13:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 1520/2304 [I 2022-10-09 19:13:41] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:03 [I 2022-10-09 19:13:41] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00295 [I 2022-10-09 19:13:41] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-09 19:13:41] ...optimizers/trijoint.py.151: total_norm: 11.013211 [I 2022-10-09 19:13:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 1530/2304 [I 2022-10-09 19:13:50] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:10:51 [I 2022-10-09 19:13:50] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00292 [I 2022-10-09 19:13:50] ...trap/engines/engine.py.230: loss: 0.16322 [I 2022-10-09 19:13:50] ...optimizers/trijoint.py.151: total_norm: 15.028969 [I 2022-10-09 19:13:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 1540/2304 [I 2022-10-09 19:13:58] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:10:45 [I 2022-10-09 19:13:58] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00294 [I 2022-10-09 19:13:58] ...trap/engines/engine.py.230: loss: 0.21578 [I 2022-10-09 19:13:58] ...optimizers/trijoint.py.151: total_norm: 13.243481 [I 2022-10-09 19:14:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 1550/2304 [I 2022-10-09 19:14:07] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:37 [I 2022-10-09 19:14:07] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00295 [I 2022-10-09 19:14:07] ...trap/engines/engine.py.230: loss: 0.11491 [I 2022-10-09 19:14:07] ...optimizers/trijoint.py.151: total_norm: 7.174326 [I 2022-10-09 19:14:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 1560/2304 [I 2022-10-09 19:14:15] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:10:26 [I 2022-10-09 19:14:15] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00300 [I 2022-10-09 19:14:15] ...trap/engines/engine.py.230: loss: 0.19421 [I 2022-10-09 19:14:15] ...optimizers/trijoint.py.151: total_norm: 28.626293 [I 2022-10-09 19:14:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 1570/2304 [I 2022-10-09 19:14:24] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:20 [I 2022-10-09 19:14:24] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00295 [I 2022-10-09 19:14:24] ...trap/engines/engine.py.230: loss: 0.15569 [I 2022-10-09 19:14:24] ...optimizers/trijoint.py.151: total_norm: 17.330713 [I 2022-10-09 19:14:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 1580/2304 [I 2022-10-09 19:14:32] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:15 [I 2022-10-09 19:14:32] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00293 [I 2022-10-09 19:14:32] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-09 19:14:32] ...optimizers/trijoint.py.151: total_norm: 8.120572 [I 2022-10-09 19:14:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 1590/2304 [I 2022-10-09 19:14:41] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:10:04 [I 2022-10-09 19:14:41] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00298 [I 2022-10-09 19:14:41] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-09 19:14:41] ...optimizers/trijoint.py.151: total_norm: 10.173285 [I 2022-10-09 19:14:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 1600/2304 [I 2022-10-09 19:14:49] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:09:56 [I 2022-10-09 19:14:49] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00301 [I 2022-10-09 19:14:49] ...trap/engines/engine.py.230: loss: 0.20138 [I 2022-10-09 19:14:49] ...optimizers/trijoint.py.151: total_norm: 18.017843 [I 2022-10-09 19:14:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 1610/2304 [I 2022-10-09 19:14:58] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:09:43 [I 2022-10-09 19:14:58] ...trap/engines/engine.py.229: process: 0.83374 | load: 0.00294 [I 2022-10-09 19:14:58] ...trap/engines/engine.py.230: loss: 0.13942 [I 2022-10-09 19:14:58] ...optimizers/trijoint.py.151: total_norm: 50.323383 [I 2022-10-09 19:15:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 1620/2304 [I 2022-10-09 19:15:06] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:09:40 [I 2022-10-09 19:15:06] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00301 [I 2022-10-09 19:15:06] ...trap/engines/engine.py.230: loss: 0.14910 [I 2022-10-09 19:15:06] ...optimizers/trijoint.py.151: total_norm: 28.882759 [I 2022-10-09 19:15:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 1630/2304 [I 2022-10-09 19:15:15] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:30 [I 2022-10-09 19:15:15] ...trap/engines/engine.py.229: process: 0.83772 | load: 0.00298 [I 2022-10-09 19:15:15] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-09 19:15:15] ...optimizers/trijoint.py.151: total_norm: 8.070598 [I 2022-10-09 19:15:23] ...trap/engines/engine.py.224: train: epoch 51 | batch 1640/2304 [I 2022-10-09 19:15:23] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:21 [I 2022-10-09 19:15:23] ...trap/engines/engine.py.229: process: 0.84082 | load: 0.00298 [I 2022-10-09 19:15:23] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-09 19:15:23] ...optimizers/trijoint.py.151: total_norm: 8.211617 [I 2022-10-09 19:15:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 1650/2304 [I 2022-10-09 19:15:32] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:12 [I 2022-10-09 19:15:32] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00299 [I 2022-10-09 19:15:32] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-09 19:15:32] ...optimizers/trijoint.py.151: total_norm: 20.680836 [I 2022-10-09 19:15:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 1660/2304 [I 2022-10-09 19:15:40] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:07 [I 2022-10-09 19:15:40] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00309 [I 2022-10-09 19:15:40] ...trap/engines/engine.py.230: loss: 0.18252 [I 2022-10-09 19:15:40] ...optimizers/trijoint.py.151: total_norm: 32.135929 [I 2022-10-09 19:15:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 1670/2304 [I 2022-10-09 19:15:49] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:08:59 [I 2022-10-09 19:15:49] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00311 [I 2022-10-09 19:15:49] ...trap/engines/engine.py.230: loss: 0.19377 [I 2022-10-09 19:15:49] ...optimizers/trijoint.py.151: total_norm: 17.006077 [I 2022-10-09 19:15:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 1680/2304 [I 2022-10-09 19:15:57] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:08:51 [I 2022-10-09 19:15:57] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00317 [I 2022-10-09 19:15:57] ...trap/engines/engine.py.230: loss: 0.24208 [I 2022-10-09 19:15:57] ...optimizers/trijoint.py.151: total_norm: 17.834948 [I 2022-10-09 19:16:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 1690/2304 [I 2022-10-09 19:16:06] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:44 [I 2022-10-09 19:16:06] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00327 [I 2022-10-09 19:16:06] ...trap/engines/engine.py.230: loss: 0.18824 [I 2022-10-09 19:16:06] ...optimizers/trijoint.py.151: total_norm: 25.053793 [I 2022-10-09 19:16:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 1700/2304 [I 2022-10-09 19:16:14] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:34 [I 2022-10-09 19:16:14] ...trap/engines/engine.py.229: process: 0.85483 | load: 0.00313 [I 2022-10-09 19:16:14] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-09 19:16:14] ...optimizers/trijoint.py.151: total_norm: 14.298759 [I 2022-10-09 19:16:23] ...trap/engines/engine.py.224: train: epoch 51 | batch 1710/2304 [I 2022-10-09 19:16:23] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:24 [I 2022-10-09 19:16:23] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00323 [I 2022-10-09 19:16:23] ...trap/engines/engine.py.230: loss: 0.17351 [I 2022-10-09 19:16:23] ...optimizers/trijoint.py.151: total_norm: 45.378353 [I 2022-10-09 19:16:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 1720/2304 [I 2022-10-09 19:16:31] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:18 [I 2022-10-09 19:16:31] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00314 [I 2022-10-09 19:16:31] ...trap/engines/engine.py.230: loss: 0.12318 [I 2022-10-09 19:16:31] ...optimizers/trijoint.py.151: total_norm: 14.405396 [I 2022-10-09 19:16:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 1730/2304 [I 2022-10-09 19:16:40] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:08 [I 2022-10-09 19:16:40] ...trap/engines/engine.py.229: process: 0.85616 | load: 0.00315 [I 2022-10-09 19:16:40] ...trap/engines/engine.py.230: loss: 0.11067 [I 2022-10-09 19:16:40] ...optimizers/trijoint.py.151: total_norm: 13.407692 [I 2022-10-09 19:16:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 1740/2304 [I 2022-10-09 19:16:48] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:00 [I 2022-10-09 19:16:48] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00309 [I 2022-10-09 19:16:48] ...trap/engines/engine.py.230: loss: 0.11100 [I 2022-10-09 19:16:48] ...optimizers/trijoint.py.151: total_norm: 18.059893 [I 2022-10-09 19:16:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 1750/2304 [I 2022-10-09 19:16:57] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:07:52 [I 2022-10-09 19:16:57] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00305 [I 2022-10-09 19:16:57] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-09 19:16:57] ...optimizers/trijoint.py.151: total_norm: 16.960016 [I 2022-10-09 19:17:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 1760/2304 [I 2022-10-09 19:17:05] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:07:43 [I 2022-10-09 19:17:05] ...trap/engines/engine.py.229: process: 0.85284 | load: 0.00313 [I 2022-10-09 19:17:05] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-09 19:17:05] ...optimizers/trijoint.py.151: total_norm: 10.745502 [I 2022-10-09 19:17:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 1770/2304 [I 2022-10-09 19:17:14] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:36 [I 2022-10-09 19:17:14] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00314 [I 2022-10-09 19:17:14] ...trap/engines/engine.py.230: loss: 0.16954 [I 2022-10-09 19:17:14] ...optimizers/trijoint.py.151: total_norm: 20.832611 [I 2022-10-09 19:17:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1780/2304 [I 2022-10-09 19:17:22] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:25 [I 2022-10-09 19:17:22] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00320 [I 2022-10-09 19:17:22] ...trap/engines/engine.py.230: loss: 0.13357 [I 2022-10-09 19:17:22] ...optimizers/trijoint.py.151: total_norm: 13.443527 [I 2022-10-09 19:17:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 1790/2304 [I 2022-10-09 19:17:31] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:18 [I 2022-10-09 19:17:31] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00312 [I 2022-10-09 19:17:31] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-09 19:17:31] ...optimizers/trijoint.py.151: total_norm: 10.844332 [I 2022-10-09 19:17:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 1800/2304 [I 2022-10-09 19:17:39] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:07 [I 2022-10-09 19:17:39] ...trap/engines/engine.py.229: process: 0.83485 | load: 0.00315 [I 2022-10-09 19:17:39] ...trap/engines/engine.py.230: loss: 0.12761 [I 2022-10-09 19:17:39] ...optimizers/trijoint.py.151: total_norm: 18.802048 [I 2022-10-09 19:17:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 1810/2304 [I 2022-10-09 19:17:48] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:01 [I 2022-10-09 19:17:48] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00313 [I 2022-10-09 19:17:48] ...trap/engines/engine.py.230: loss: 0.14166 [I 2022-10-09 19:17:48] ...optimizers/trijoint.py.151: total_norm: 19.282715 [I 2022-10-09 19:17:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 1820/2304 [I 2022-10-09 19:17:56] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:06:50 [I 2022-10-09 19:17:56] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00312 [I 2022-10-09 19:17:56] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-09 19:17:56] ...optimizers/trijoint.py.151: total_norm: 8.309997 [I 2022-10-09 19:18:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 1830/2304 [I 2022-10-09 19:18:05] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:41 [I 2022-10-09 19:18:05] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00312 [I 2022-10-09 19:18:05] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-09 19:18:05] ...optimizers/trijoint.py.151: total_norm: 6.841907 [I 2022-10-09 19:18:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 1840/2304 [I 2022-10-09 19:18:13] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:33 [I 2022-10-09 19:18:13] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00311 [I 2022-10-09 19:18:13] ...trap/engines/engine.py.230: loss: 0.16280 [I 2022-10-09 19:18:13] ...optimizers/trijoint.py.151: total_norm: 19.190624 [I 2022-10-09 19:18:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1850/2304 [I 2022-10-09 19:18:22] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:24 [I 2022-10-09 19:18:22] ...trap/engines/engine.py.229: process: 0.82720 | load: 0.00305 [I 2022-10-09 19:18:22] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-09 19:18:22] ...optimizers/trijoint.py.151: total_norm: 10.047345 [I 2022-10-09 19:18:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 1860/2304 [I 2022-10-09 19:18:30] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:17 [I 2022-10-09 19:18:30] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00313 [I 2022-10-09 19:18:30] ...trap/engines/engine.py.230: loss: 0.13783 [I 2022-10-09 19:18:30] ...optimizers/trijoint.py.151: total_norm: 13.863194 [I 2022-10-09 19:18:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 1870/2304 [I 2022-10-09 19:18:39] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:10 [I 2022-10-09 19:18:39] ...trap/engines/engine.py.229: process: 0.87536 | load: 0.00312 [I 2022-10-09 19:18:39] ...trap/engines/engine.py.230: loss: 0.15514 [I 2022-10-09 19:18:39] ...optimizers/trijoint.py.151: total_norm: 14.095173 [I 2022-10-09 19:18:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 1880/2304 [I 2022-10-09 19:18:48] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:05:58 [I 2022-10-09 19:18:48] ...trap/engines/engine.py.229: process: 0.83134 | load: 0.00306 [I 2022-10-09 19:18:48] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-09 19:18:48] ...optimizers/trijoint.py.151: total_norm: 9.303305 [I 2022-10-09 19:18:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 1890/2304 [I 2022-10-09 19:18:56] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:05:53 [I 2022-10-09 19:18:56] ...trap/engines/engine.py.229: process: 0.85745 | load: 0.00315 [I 2022-10-09 19:18:56] ...trap/engines/engine.py.230: loss: 0.15870 [I 2022-10-09 19:18:56] ...optimizers/trijoint.py.151: total_norm: 24.967224 [I 2022-10-09 19:19:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 1900/2304 [I 2022-10-09 19:19:05] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:43 [I 2022-10-09 19:19:05] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00311 [I 2022-10-09 19:19:05] ...trap/engines/engine.py.230: loss: 0.19381 [I 2022-10-09 19:19:05] ...optimizers/trijoint.py.151: total_norm: 19.261024 [I 2022-10-09 19:19:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 1910/2304 [I 2022-10-09 19:19:13] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:35 [I 2022-10-09 19:19:13] ...trap/engines/engine.py.229: process: 0.85483 | load: 0.00314 [I 2022-10-09 19:19:13] ...trap/engines/engine.py.230: loss: 0.13496 [I 2022-10-09 19:19:13] ...optimizers/trijoint.py.151: total_norm: 21.887959 [I 2022-10-09 19:19:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1920/2304 [I 2022-10-09 19:19:22] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:26 [I 2022-10-09 19:19:22] ...trap/engines/engine.py.229: process: 0.84235 | load: 0.00305 [I 2022-10-09 19:19:22] ...trap/engines/engine.py.230: loss: 0.12240 [I 2022-10-09 19:19:22] ...optimizers/trijoint.py.151: total_norm: 9.288390 [I 2022-10-09 19:19:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 1930/2304 [I 2022-10-09 19:19:30] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:18 [I 2022-10-09 19:19:30] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00314 [I 2022-10-09 19:19:30] ...trap/engines/engine.py.230: loss: 0.17756 [I 2022-10-09 19:19:30] ...optimizers/trijoint.py.151: total_norm: 13.221920 [I 2022-10-09 19:19:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 1940/2304 [I 2022-10-09 19:19:39] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:11 [I 2022-10-09 19:19:39] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00315 [I 2022-10-09 19:19:39] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-09 19:19:39] ...optimizers/trijoint.py.151: total_norm: 11.533966 [I 2022-10-09 19:19:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 1950/2304 [I 2022-10-09 19:19:47] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:03 [I 2022-10-09 19:19:47] ...trap/engines/engine.py.229: process: 0.86177 | load: 0.00312 [I 2022-10-09 19:19:47] ...trap/engines/engine.py.230: loss: 0.17819 [I 2022-10-09 19:19:47] ...optimizers/trijoint.py.151: total_norm: 14.143640 [I 2022-10-09 19:19:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 1960/2304 [I 2022-10-09 19:19:56] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:04:54 [I 2022-10-09 19:19:56] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00333 [I 2022-10-09 19:19:56] ...trap/engines/engine.py.230: loss: 0.18089 [I 2022-10-09 19:19:56] ...optimizers/trijoint.py.151: total_norm: 23.154686 [I 2022-10-09 19:20:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 1970/2304 [I 2022-10-09 19:20:04] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:44 [I 2022-10-09 19:20:04] ...trap/engines/engine.py.229: process: 0.85327 | load: 0.00313 [I 2022-10-09 19:20:04] ...trap/engines/engine.py.230: loss: 0.16871 [I 2022-10-09 19:20:04] ...optimizers/trijoint.py.151: total_norm: 25.257475 [I 2022-10-09 19:20:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 1980/2304 [I 2022-10-09 19:20:13] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:37 [I 2022-10-09 19:20:13] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00309 [I 2022-10-09 19:20:13] ...trap/engines/engine.py.230: loss: 0.17434 [I 2022-10-09 19:20:13] ...optimizers/trijoint.py.151: total_norm: 20.012653 [I 2022-10-09 19:20:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1990/2304 [I 2022-10-09 19:20:22] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:27 [I 2022-10-09 19:20:22] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00312 [I 2022-10-09 19:20:22] ...trap/engines/engine.py.230: loss: 0.15059 [I 2022-10-09 19:20:22] ...optimizers/trijoint.py.151: total_norm: 14.089308 [I 2022-10-09 19:20:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 2000/2304 [I 2022-10-09 19:20:30] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:17 [I 2022-10-09 19:20:30] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00311 [I 2022-10-09 19:20:30] ...trap/engines/engine.py.230: loss: 0.17269 [I 2022-10-09 19:20:30] ...optimizers/trijoint.py.151: total_norm: 21.001406 [I 2022-10-09 19:20:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 2010/2304 [I 2022-10-09 19:20:39] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:10 [I 2022-10-09 19:20:39] ...trap/engines/engine.py.229: process: 0.85968 | load: 0.00302 [I 2022-10-09 19:20:39] ...trap/engines/engine.py.230: loss: 0.16577 [I 2022-10-09 19:20:39] ...optimizers/trijoint.py.151: total_norm: 11.534328 [I 2022-10-09 19:20:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 2020/2304 [I 2022-10-09 19:20:47] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:58 [I 2022-10-09 19:20:47] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00299 [I 2022-10-09 19:20:47] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-09 19:20:47] ...optimizers/trijoint.py.151: total_norm: 11.700199 [I 2022-10-09 19:20:55] ...trap/engines/engine.py.224: train: epoch 51 | batch 2030/2304 [I 2022-10-09 19:20:55] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:51 [I 2022-10-09 19:20:55] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00296 [I 2022-10-09 19:20:55] ...trap/engines/engine.py.230: loss: 0.14125 [I 2022-10-09 19:20:55] ...optimizers/trijoint.py.151: total_norm: 36.865490 [I 2022-10-09 19:21:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 2040/2304 [I 2022-10-09 19:21:04] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:44 [I 2022-10-09 19:21:04] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00297 [I 2022-10-09 19:21:04] ...trap/engines/engine.py.230: loss: 0.17341 [I 2022-10-09 19:21:04] ...optimizers/trijoint.py.151: total_norm: 9.992440 [I 2022-10-09 19:21:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 2050/2304 [I 2022-10-09 19:21:12] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:34 [I 2022-10-09 19:21:12] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00294 [I 2022-10-09 19:21:12] ...trap/engines/engine.py.230: loss: 0.18866 [I 2022-10-09 19:21:12] ...optimizers/trijoint.py.151: total_norm: 9.928152 [I 2022-10-09 19:21:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 2060/2304 [I 2022-10-09 19:21:21] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:26 [I 2022-10-09 19:21:21] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00297 [I 2022-10-09 19:21:21] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-09 19:21:21] ...optimizers/trijoint.py.151: total_norm: 10.169921 [I 2022-10-09 19:21:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 2070/2304 [I 2022-10-09 19:21:29] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:15 [I 2022-10-09 19:21:29] ...trap/engines/engine.py.229: process: 0.82132 | load: 0.00295 [I 2022-10-09 19:21:29] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-09 19:21:29] ...optimizers/trijoint.py.151: total_norm: 12.057261 [I 2022-10-09 19:21:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 2080/2304 [I 2022-10-09 19:21:38] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:08 [I 2022-10-09 19:21:38] ...trap/engines/engine.py.229: process: 0.83801 | load: 0.00300 [I 2022-10-09 19:21:38] ...trap/engines/engine.py.230: loss: 0.14186 [I 2022-10-09 19:21:38] ...optimizers/trijoint.py.151: total_norm: 15.636535 [I 2022-10-09 19:21:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 2090/2304 [I 2022-10-09 19:21:46] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:00 [I 2022-10-09 19:21:46] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00296 [I 2022-10-09 19:21:46] ...trap/engines/engine.py.230: loss: 0.20096 [I 2022-10-09 19:21:46] ...optimizers/trijoint.py.151: total_norm: 30.852659 [I 2022-10-09 19:21:55] ...trap/engines/engine.py.224: train: epoch 51 | batch 2100/2304 [I 2022-10-09 19:21:55] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:51 [I 2022-10-09 19:21:55] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00298 [I 2022-10-09 19:21:55] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-09 19:21:55] ...optimizers/trijoint.py.151: total_norm: 34.819630 [I 2022-10-09 19:22:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 2110/2304 [I 2022-10-09 19:22:03] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:43 [I 2022-10-09 19:22:03] ...trap/engines/engine.py.229: process: 0.83174 | load: 0.00296 [I 2022-10-09 19:22:03] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-09 19:22:03] ...optimizers/trijoint.py.151: total_norm: 16.654015 [I 2022-10-09 19:22:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 2120/2304 [I 2022-10-09 19:22:12] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:35 [I 2022-10-09 19:22:12] ...trap/engines/engine.py.229: process: 0.82861 | load: 0.00298 [I 2022-10-09 19:22:12] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-09 19:22:12] ...optimizers/trijoint.py.151: total_norm: 8.520763 [I 2022-10-09 19:22:20] ...trap/engines/engine.py.224: train: epoch 51 | batch 2130/2304 [I 2022-10-09 19:22:20] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:26 [I 2022-10-09 19:22:20] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00299 [I 2022-10-09 19:22:20] ...trap/engines/engine.py.230: loss: 0.18201 [I 2022-10-09 19:22:20] ...optimizers/trijoint.py.151: total_norm: 18.133011 [I 2022-10-09 19:22:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 2140/2304 [I 2022-10-09 19:22:28] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:18 [I 2022-10-09 19:22:28] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00314 [I 2022-10-09 19:22:28] ...trap/engines/engine.py.230: loss: 0.14099 [I 2022-10-09 19:22:28] ...optimizers/trijoint.py.151: total_norm: 17.910948 [I 2022-10-09 19:22:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 2150/2304 [I 2022-10-09 19:22:37] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:09 [I 2022-10-09 19:22:37] ...trap/engines/engine.py.229: process: 0.83303 | load: 0.00298 [I 2022-10-09 19:22:37] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-09 19:22:37] ...optimizers/trijoint.py.151: total_norm: 13.257784 [I 2022-10-09 19:22:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 2160/2304 [I 2022-10-09 19:22:45] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:01 [I 2022-10-09 19:22:45] ...trap/engines/engine.py.229: process: 0.82933 | load: 0.00295 [I 2022-10-09 19:22:45] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-09 19:22:45] ...optimizers/trijoint.py.151: total_norm: 17.458206 [I 2022-10-09 19:22:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 2170/2304 [I 2022-10-09 19:22:54] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:52 [I 2022-10-09 19:22:54] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00297 [I 2022-10-09 19:22:54] ...trap/engines/engine.py.230: loss: 0.15585 [I 2022-10-09 19:22:54] ...optimizers/trijoint.py.151: total_norm: 12.342090 [I 2022-10-09 19:23:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 2180/2304 [I 2022-10-09 19:23:02] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:44 [I 2022-10-09 19:23:02] ...trap/engines/engine.py.229: process: 0.83737 | load: 0.00296 [I 2022-10-09 19:23:02] ...trap/engines/engine.py.230: loss: 0.17457 [I 2022-10-09 19:23:02] ...optimizers/trijoint.py.151: total_norm: 15.488225 [I 2022-10-09 19:23:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 2190/2304 [I 2022-10-09 19:23:11] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:36 [I 2022-10-09 19:23:11] ...trap/engines/engine.py.229: process: 0.83612 | load: 0.00297 [I 2022-10-09 19:23:11] ...trap/engines/engine.py.230: loss: 0.30471 [I 2022-10-09 19:23:11] ...optimizers/trijoint.py.151: total_norm: 25.091211 [I 2022-10-09 19:23:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 2200/2304 [I 2022-10-09 19:23:19] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:27 [I 2022-10-09 19:23:19] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00300 [I 2022-10-09 19:23:19] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-09 19:23:19] ...optimizers/trijoint.py.151: total_norm: 13.647545 [I 2022-10-09 19:23:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 2210/2304 [I 2022-10-09 19:23:28] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:19 [I 2022-10-09 19:23:28] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00297 [I 2022-10-09 19:23:28] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-09 19:23:28] ...optimizers/trijoint.py.151: total_norm: 19.897907 [I 2022-10-09 19:23:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 2220/2304 [I 2022-10-09 19:23:36] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:10 [I 2022-10-09 19:23:36] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00302 [I 2022-10-09 19:23:36] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-09 19:23:36] ...optimizers/trijoint.py.151: total_norm: 10.521543 [I 2022-10-09 19:23:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 2230/2304 [I 2022-10-09 19:23:44] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:02 [I 2022-10-09 19:23:44] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00294 [I 2022-10-09 19:23:44] ...trap/engines/engine.py.230: loss: 0.18903 [I 2022-10-09 19:23:44] ...optimizers/trijoint.py.151: total_norm: 23.372263 [I 2022-10-09 19:23:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 2240/2304 [I 2022-10-09 19:23:53] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:54 [I 2022-10-09 19:23:53] ...trap/engines/engine.py.229: process: 0.83761 | load: 0.00295 [I 2022-10-09 19:23:53] ...trap/engines/engine.py.230: loss: 0.18376 [I 2022-10-09 19:23:53] ...optimizers/trijoint.py.151: total_norm: 10.525897 [I 2022-10-09 19:24:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 2250/2304 [I 2022-10-09 19:24:01] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:45 [I 2022-10-09 19:24:01] ...trap/engines/engine.py.229: process: 0.86302 | load: 0.00298 [I 2022-10-09 19:24:01] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-09 19:24:01] ...optimizers/trijoint.py.151: total_norm: 19.174486 [I 2022-10-09 19:24:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 2260/2304 [I 2022-10-09 19:24:10] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:37 [I 2022-10-09 19:24:10] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00302 [I 2022-10-09 19:24:10] ...trap/engines/engine.py.230: loss: 0.16640 [I 2022-10-09 19:24:10] ...optimizers/trijoint.py.151: total_norm: 12.459213 [I 2022-10-09 19:24:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 2270/2304 [I 2022-10-09 19:24:18] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:28 [I 2022-10-09 19:24:18] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00298 [I 2022-10-09 19:24:18] ...trap/engines/engine.py.230: loss: 0.13372 [I 2022-10-09 19:24:18] ...optimizers/trijoint.py.151: total_norm: 36.620525 [I 2022-10-09 19:24:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 2280/2304 [I 2022-10-09 19:24:27] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:20 [I 2022-10-09 19:24:27] ...trap/engines/engine.py.229: process: 0.83768 | load: 0.00294 [I 2022-10-09 19:24:27] ...trap/engines/engine.py.230: loss: 0.20023 [I 2022-10-09 19:24:27] ...optimizers/trijoint.py.151: total_norm: 14.920276 [I 2022-10-09 19:24:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 2290/2304 [I 2022-10-09 19:24:35] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:11 [I 2022-10-09 19:24:35] ...trap/engines/engine.py.229: process: 0.85824 | load: 0.00323 [I 2022-10-09 19:24:35] ...trap/engines/engine.py.230: loss: 0.19527 [I 2022-10-09 19:24:35] ...optimizers/trijoint.py.151: total_norm: 13.847182 [I 2022-10-09 19:24:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 2300/2304 [I 2022-10-09 19:24:44] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:03 [I 2022-10-09 19:24:44] ...trap/engines/engine.py.229: process: 0.83152 | load: 0.00019 [I 2022-10-09 19:24:44] ...trap/engines/engine.py.230: loss: 0.10675 [I 2022-10-09 19:24:44] ...optimizers/trijoint.py.151: total_norm: 6.886589 [I 2022-10-09 19:24:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 2304/2304 [I 2022-10-09 19:24:47] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:00 [I 2022-10-09 19:24:47] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00018 [I 2022-10-09 19:24:47] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-09 19:24:47] ...optimizers/trijoint.py.151: total_norm: 24.909824 [S 2022-10-09 19:24:48] ...trap/engines/engine.py.129: train_epoch.epoch: 51 [S 2022-10-09 19:24:48] ...trap/engines/engine.py.129: train_epoch.loss: 0.1558489824227035 [I 2022-10-09 19:24:48] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 19:24:49] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 51 [I 2022-10-09 19:24:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 19:24:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 19:24:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 19:24:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 19:24:53] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 19:24:53] ...trap/engines/engine.py.321: eval: epoch 51 | batch 0/510 [I 2022-10-09 19:24:53] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:21 [I 2022-10-09 19:24:53] ...trap/engines/engine.py.326: process: 4.15902 | load: 3.57356 [I 2022-10-09 19:24:59] ...trap/engines/engine.py.321: eval: epoch 51 | batch 10/510 [I 2022-10-09 19:24:59] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:01 [I 2022-10-09 19:24:59] ...trap/engines/engine.py.326: process: 0.68562 | load: 0.00020 [I 2022-10-09 19:25:05] ...trap/engines/engine.py.321: eval: epoch 51 | batch 20/510 [I 2022-10-09 19:25:05] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:09 [I 2022-10-09 19:25:05] ...trap/engines/engine.py.326: process: 0.59048 | load: 0.00022 [I 2022-10-09 19:25:12] ...trap/engines/engine.py.321: eval: epoch 51 | batch 30/510 [I 2022-10-09 19:25:12] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:22 [I 2022-10-09 19:25:12] ...trap/engines/engine.py.326: process: 0.85097 | load: 0.00020 [I 2022-10-09 19:25:19] ...trap/engines/engine.py.321: eval: epoch 51 | batch 40/510 [I 2022-10-09 19:25:19] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:05:19 [I 2022-10-09 19:25:19] ...trap/engines/engine.py.326: process: 0.50240 | load: 0.00020 [I 2022-10-09 19:25:23] ...trap/engines/engine.py.321: eval: epoch 51 | batch 50/510 [I 2022-10-09 19:25:23] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:42 [I 2022-10-09 19:25:23] ...trap/engines/engine.py.326: process: 0.44796 | load: 0.00020 [I 2022-10-09 19:25:28] ...trap/engines/engine.py.321: eval: epoch 51 | batch 60/510 [I 2022-10-09 19:25:28] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:43 [I 2022-10-09 19:25:28] ...trap/engines/engine.py.326: process: 0.47488 | load: 0.00021 [I 2022-10-09 19:25:34] ...trap/engines/engine.py.321: eval: epoch 51 | batch 70/510 [I 2022-10-09 19:25:34] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:04:28 [I 2022-10-09 19:25:34] ...trap/engines/engine.py.326: process: 0.65162 | load: 0.00020 [I 2022-10-09 19:25:40] ...trap/engines/engine.py.321: eval: epoch 51 | batch 80/510 [I 2022-10-09 19:25:40] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:56 [I 2022-10-09 19:25:40] ...trap/engines/engine.py.326: process: 0.48096 | load: 0.00023 [I 2022-10-09 19:25:45] ...trap/engines/engine.py.321: eval: epoch 51 | batch 90/510 [I 2022-10-09 19:25:45] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:14 [I 2022-10-09 19:25:45] ...trap/engines/engine.py.326: process: 0.42600 | load: 0.00021 [I 2022-10-09 19:25:49] ...trap/engines/engine.py.321: eval: epoch 51 | batch 100/510 [I 2022-10-09 19:25:49] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:03:07 [I 2022-10-09 19:25:49] ...trap/engines/engine.py.326: process: 0.31644 | load: 0.00021 [I 2022-10-09 19:25:52] ...trap/engines/engine.py.321: eval: epoch 51 | batch 110/510 [I 2022-10-09 19:25:52] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:13 [I 2022-10-09 19:25:52] ...trap/engines/engine.py.326: process: 0.31925 | load: 0.00023 [I 2022-10-09 19:25:56] ...trap/engines/engine.py.321: eval: epoch 51 | batch 120/510 [I 2022-10-09 19:25:56] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:04 [I 2022-10-09 19:25:56] ...trap/engines/engine.py.326: process: 0.31702 | load: 0.00020 [I 2022-10-09 19:25:59] ...trap/engines/engine.py.321: eval: epoch 51 | batch 130/510 [I 2022-10-09 19:25:59] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:01 [I 2022-10-09 19:25:59] ...trap/engines/engine.py.326: process: 0.32222 | load: 0.00019 [I 2022-10-09 19:26:02] ...trap/engines/engine.py.321: eval: epoch 51 | batch 140/510 [I 2022-10-09 19:26:02] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:58 [I 2022-10-09 19:26:02] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00019 [I 2022-10-09 19:26:05] ...trap/engines/engine.py.321: eval: epoch 51 | batch 150/510 [I 2022-10-09 19:26:05] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:54 [I 2022-10-09 19:26:05] ...trap/engines/engine.py.326: process: 0.31557 | load: 0.00020 [I 2022-10-09 19:26:08] ...trap/engines/engine.py.321: eval: epoch 51 | batch 160/510 [I 2022-10-09 19:26:08] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:51 [I 2022-10-09 19:26:08] ...trap/engines/engine.py.326: process: 0.32165 | load: 0.00020 [I 2022-10-09 19:26:12] ...trap/engines/engine.py.321: eval: epoch 51 | batch 170/510 [I 2022-10-09 19:26:12] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:48 [I 2022-10-09 19:26:12] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00025 [I 2022-10-09 19:26:15] ...trap/engines/engine.py.321: eval: epoch 51 | batch 180/510 [I 2022-10-09 19:26:15] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:45 [I 2022-10-09 19:26:15] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00019 [I 2022-10-09 19:26:18] ...trap/engines/engine.py.321: eval: epoch 51 | batch 190/510 [I 2022-10-09 19:26:18] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:41 [I 2022-10-09 19:26:18] ...trap/engines/engine.py.326: process: 0.31224 | load: 0.00020 [I 2022-10-09 19:26:21] ...trap/engines/engine.py.321: eval: epoch 51 | batch 200/510 [I 2022-10-09 19:26:21] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:39 [I 2022-10-09 19:26:21] ...trap/engines/engine.py.326: process: 0.31731 | load: 0.00020 [I 2022-10-09 19:26:24] ...trap/engines/engine.py.321: eval: epoch 51 | batch 210/510 [I 2022-10-09 19:26:24] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:36 [I 2022-10-09 19:26:24] ...trap/engines/engine.py.326: process: 0.31953 | load: 0.00021 [I 2022-10-09 19:26:28] ...trap/engines/engine.py.321: eval: epoch 51 | batch 220/510 [I 2022-10-09 19:26:28] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:32 [I 2022-10-09 19:26:28] ...trap/engines/engine.py.326: process: 0.31111 | load: 0.00020 [I 2022-10-09 19:26:31] ...trap/engines/engine.py.321: eval: epoch 51 | batch 230/510 [I 2022-10-09 19:26:31] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:29 [I 2022-10-09 19:26:31] ...trap/engines/engine.py.326: process: 0.32149 | load: 0.00021 [I 2022-10-09 19:26:34] ...trap/engines/engine.py.321: eval: epoch 51 | batch 240/510 [I 2022-10-09 19:26:34] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:27 [I 2022-10-09 19:26:34] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00021 [I 2022-10-09 19:26:37] ...trap/engines/engine.py.321: eval: epoch 51 | batch 250/510 [I 2022-10-09 19:26:37] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:23 [I 2022-10-09 19:26:37] ...trap/engines/engine.py.326: process: 0.32036 | load: 0.00021 [I 2022-10-09 19:26:40] ...trap/engines/engine.py.321: eval: epoch 51 | batch 260/510 [I 2022-10-09 19:26:40] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:20 [I 2022-10-09 19:26:40] ...trap/engines/engine.py.326: process: 0.32097 | load: 0.00021 [I 2022-10-09 19:26:44] ...trap/engines/engine.py.321: eval: epoch 51 | batch 270/510 [I 2022-10-09 19:26:44] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:17 [I 2022-10-09 19:26:44] ...trap/engines/engine.py.326: process: 0.32007 | load: 0.00020 [I 2022-10-09 19:26:47] ...trap/engines/engine.py.321: eval: epoch 51 | batch 280/510 [I 2022-10-09 19:26:47] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:13 [I 2022-10-09 19:26:47] ...trap/engines/engine.py.326: process: 0.32123 | load: 0.00022 [I 2022-10-09 19:26:50] ...trap/engines/engine.py.321: eval: epoch 51 | batch 290/510 [I 2022-10-09 19:26:50] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:10 [I 2022-10-09 19:26:50] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00022 [I 2022-10-09 19:26:53] ...trap/engines/engine.py.321: eval: epoch 51 | batch 300/510 [I 2022-10-09 19:26:53] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:07 [I 2022-10-09 19:26:53] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00021 [I 2022-10-09 19:26:57] ...trap/engines/engine.py.321: eval: epoch 51 | batch 310/510 [I 2022-10-09 19:26:57] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:04 [I 2022-10-09 19:26:57] ...trap/engines/engine.py.326: process: 0.32252 | load: 0.00020 [I 2022-10-09 19:27:00] ...trap/engines/engine.py.321: eval: epoch 51 | batch 320/510 [I 2022-10-09 19:27:00] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:01 [I 2022-10-09 19:27:00] ...trap/engines/engine.py.326: process: 0.31879 | load: 0.00020 [I 2022-10-09 19:27:03] ...trap/engines/engine.py.321: eval: epoch 51 | batch 330/510 [I 2022-10-09 19:27:03] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:57 [I 2022-10-09 19:27:03] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00020 [I 2022-10-09 19:27:06] ...trap/engines/engine.py.321: eval: epoch 51 | batch 340/510 [I 2022-10-09 19:27:06] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:54 [I 2022-10-09 19:27:06] ...trap/engines/engine.py.326: process: 0.32338 | load: 0.00022 [I 2022-10-09 19:27:09] ...trap/engines/engine.py.321: eval: epoch 51 | batch 350/510 [I 2022-10-09 19:27:09] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:51 [I 2022-10-09 19:27:09] ...trap/engines/engine.py.326: process: 0.32059 | load: 0.00020 [I 2022-10-09 19:27:13] ...trap/engines/engine.py.321: eval: epoch 51 | batch 360/510 [I 2022-10-09 19:27:13] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:48 [I 2022-10-09 19:27:13] ...trap/engines/engine.py.326: process: 0.32265 | load: 0.00021 [I 2022-10-09 19:27:16] ...trap/engines/engine.py.321: eval: epoch 51 | batch 370/510 [I 2022-10-09 19:27:16] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:44 [I 2022-10-09 19:27:16] ...trap/engines/engine.py.326: process: 0.31298 | load: 0.00020 [I 2022-10-09 19:27:19] ...trap/engines/engine.py.321: eval: epoch 51 | batch 380/510 [I 2022-10-09 19:27:19] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:41 [I 2022-10-09 19:27:19] ...trap/engines/engine.py.326: process: 0.32283 | load: 0.00021 [I 2022-10-09 19:27:22] ...trap/engines/engine.py.321: eval: epoch 51 | batch 390/510 [I 2022-10-09 19:27:22] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-09 19:27:22] ...trap/engines/engine.py.326: process: 0.32403 | load: 0.00021 [I 2022-10-09 19:27:25] ...trap/engines/engine.py.321: eval: epoch 51 | batch 400/510 [I 2022-10-09 19:27:25] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:35 [I 2022-10-09 19:27:25] ...trap/engines/engine.py.326: process: 0.32255 | load: 0.00021 [I 2022-10-09 19:27:29] ...trap/engines/engine.py.321: eval: epoch 51 | batch 410/510 [I 2022-10-09 19:27:29] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:32 [I 2022-10-09 19:27:29] ...trap/engines/engine.py.326: process: 0.31497 | load: 0.00022 [I 2022-10-09 19:27:32] ...trap/engines/engine.py.321: eval: epoch 51 | batch 420/510 [I 2022-10-09 19:27:32] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:28 [I 2022-10-09 19:27:32] ...trap/engines/engine.py.326: process: 0.31531 | load: 0.00020 [I 2022-10-09 19:27:35] ...trap/engines/engine.py.321: eval: epoch 51 | batch 430/510 [I 2022-10-09 19:27:35] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:25 [I 2022-10-09 19:27:35] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00020 [I 2022-10-09 19:27:38] ...trap/engines/engine.py.321: eval: epoch 51 | batch 440/510 [I 2022-10-09 19:27:38] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:22 [I 2022-10-09 19:27:38] ...trap/engines/engine.py.326: process: 0.32719 | load: 0.00021 [I 2022-10-09 19:27:41] ...trap/engines/engine.py.321: eval: epoch 51 | batch 450/510 [I 2022-10-09 19:27:42] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:19 [I 2022-10-09 19:27:42] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00021 [I 2022-10-09 19:27:45] ...trap/engines/engine.py.321: eval: epoch 51 | batch 460/510 [I 2022-10-09 19:27:45] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:16 [I 2022-10-09 19:27:45] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00021 [I 2022-10-09 19:27:48] ...trap/engines/engine.py.321: eval: epoch 51 | batch 470/510 [I 2022-10-09 19:27:48] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:12 [I 2022-10-09 19:27:48] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00021 [I 2022-10-09 19:27:51] ...trap/engines/engine.py.321: eval: epoch 51 | batch 480/510 [I 2022-10-09 19:27:51] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:09 [I 2022-10-09 19:27:51] ...trap/engines/engine.py.326: process: 0.32265 | load: 0.00020 [I 2022-10-09 19:27:54] ...trap/engines/engine.py.321: eval: epoch 51 | batch 490/510 [I 2022-10-09 19:27:54] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:06 [I 2022-10-09 19:27:54] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00021 [I 2022-10-09 19:27:58] ...trap/engines/engine.py.321: eval: epoch 51 | batch 500/510 [I 2022-10-09 19:27:58] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:03 [I 2022-10-09 19:27:58] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00021 [I 2022-10-09 19:28:01] ...trap/engines/engine.py.321: eval: epoch 51 | batch 510/510 [I 2022-10-09 19:28:01] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:00 [I 2022-10-09 19:28:01] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00018 [S 2022-10-09 19:28:01] ...trap/engines/engine.py.132: eval_epoch.epoch: 51 [I 2022-10-09 19:28:01] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.718900 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903800 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935600 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.706100 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901000 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932200 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008677 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007947 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006800 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010064 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007563 [S 2022-10-09 19:28:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006809 [I 2022-10-09 19:28:36] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 19:28:36] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 19:28:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 19:28:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 19:28:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 19:28:41] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 19:28:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 19:28:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 19:28:47] ...trap/engines/engine.py.163: Training model on trainset for epoch 52 [I 2022-10-09 19:28:47] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 19:28:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 0/2304 [I 2022-10-09 19:28:50] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:03 [I 2022-10-09 19:28:50] ...trap/engines/engine.py.229: process: 3.02222 | load: 2.10346 [I 2022-10-09 19:28:50] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-09 19:28:50] ...optimizers/trijoint.py.151: total_norm: 31.351408 [I 2022-10-09 19:28:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 10/2304 [I 2022-10-09 19:28:58] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:26 [I 2022-10-09 19:28:58] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00538 [I 2022-10-09 19:28:58] ...trap/engines/engine.py.230: loss: 0.15429 [I 2022-10-09 19:28:58] ...optimizers/trijoint.py.151: total_norm: 9.794886 [I 2022-10-09 19:29:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 20/2304 [I 2022-10-09 19:29:07] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:23 [I 2022-10-09 19:29:07] ...trap/engines/engine.py.229: process: 0.85553 | load: 0.00429 [I 2022-10-09 19:29:07] ...trap/engines/engine.py.230: loss: 0.20409 [I 2022-10-09 19:29:07] ...optimizers/trijoint.py.151: total_norm: 17.868231 [I 2022-10-09 19:29:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 30/2304 [I 2022-10-09 19:29:15] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:15 [I 2022-10-09 19:29:15] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00426 [I 2022-10-09 19:29:15] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-09 19:29:15] ...optimizers/trijoint.py.151: total_norm: 8.025180 [I 2022-10-09 19:29:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 40/2304 [I 2022-10-09 19:29:24] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:51 [I 2022-10-09 19:29:24] ...trap/engines/engine.py.229: process: 0.82545 | load: 0.00392 [I 2022-10-09 19:29:24] ...trap/engines/engine.py.230: loss: 0.09942 [I 2022-10-09 19:29:24] ...optimizers/trijoint.py.151: total_norm: 6.542696 [I 2022-10-09 19:29:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 50/2304 [I 2022-10-09 19:29:32] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:41 [I 2022-10-09 19:29:32] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00375 [I 2022-10-09 19:29:32] ...trap/engines/engine.py.230: loss: 0.21850 [I 2022-10-09 19:29:32] ...optimizers/trijoint.py.151: total_norm: 15.433718 [I 2022-10-09 19:29:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 60/2304 [I 2022-10-09 19:29:41] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:41 [I 2022-10-09 19:29:41] ...trap/engines/engine.py.229: process: 0.84366 | load: 0.00367 [I 2022-10-09 19:29:41] ...trap/engines/engine.py.230: loss: 0.21051 [I 2022-10-09 19:29:41] ...optimizers/trijoint.py.151: total_norm: 15.212911 [I 2022-10-09 19:29:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 70/2304 [I 2022-10-09 19:29:49] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:40 [I 2022-10-09 19:29:49] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00333 [I 2022-10-09 19:29:49] ...trap/engines/engine.py.230: loss: 0.11830 [I 2022-10-09 19:29:49] ...optimizers/trijoint.py.151: total_norm: 11.686893 [I 2022-10-09 19:29:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 80/2304 [I 2022-10-09 19:29:58] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:10 [I 2022-10-09 19:29:58] ...trap/engines/engine.py.229: process: 0.83433 | load: 0.00318 [I 2022-10-09 19:29:58] ...trap/engines/engine.py.230: loss: 0.13886 [I 2022-10-09 19:29:58] ...optimizers/trijoint.py.151: total_norm: 43.558773 [I 2022-10-09 19:30:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 90/2304 [I 2022-10-09 19:30:06] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:00 [I 2022-10-09 19:30:06] ...trap/engines/engine.py.229: process: 0.83964 | load: 0.00293 [I 2022-10-09 19:30:06] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-09 19:30:06] ...optimizers/trijoint.py.151: total_norm: 43.341381 [I 2022-10-09 19:30:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 100/2304 [I 2022-10-09 19:30:15] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:51 [I 2022-10-09 19:30:15] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00309 [I 2022-10-09 19:30:15] ...trap/engines/engine.py.230: loss: 0.15810 [I 2022-10-09 19:30:15] ...optimizers/trijoint.py.151: total_norm: 8.249638 [I 2022-10-09 19:30:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 110/2304 [I 2022-10-09 19:30:23] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:48 [I 2022-10-09 19:30:23] ...trap/engines/engine.py.229: process: 0.83485 | load: 0.00329 [I 2022-10-09 19:30:23] ...trap/engines/engine.py.230: loss: 0.08589 [I 2022-10-09 19:30:23] ...optimizers/trijoint.py.151: total_norm: 7.217000 [I 2022-10-09 19:30:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 120/2304 [I 2022-10-09 19:30:31] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:45 [I 2022-10-09 19:30:31] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00347 [I 2022-10-09 19:30:31] ...trap/engines/engine.py.230: loss: 0.18692 [I 2022-10-09 19:30:31] ...optimizers/trijoint.py.151: total_norm: 13.360995 [I 2022-10-09 19:30:40] ...trap/engines/engine.py.224: train: epoch 52 | batch 130/2304 [I 2022-10-09 19:30:40] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:18 [I 2022-10-09 19:30:40] ...trap/engines/engine.py.229: process: 0.83599 | load: 0.00331 [I 2022-10-09 19:30:40] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-09 19:30:40] ...optimizers/trijoint.py.151: total_norm: 12.921431 [I 2022-10-09 19:30:48] ...trap/engines/engine.py.224: train: epoch 52 | batch 140/2304 [I 2022-10-09 19:30:48] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:21 [I 2022-10-09 19:30:48] ...trap/engines/engine.py.229: process: 0.82598 | load: 0.00297 [I 2022-10-09 19:30:48] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-09 19:30:48] ...optimizers/trijoint.py.151: total_norm: 19.394258 [I 2022-10-09 19:30:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 150/2304 [I 2022-10-09 19:30:57] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:13 [I 2022-10-09 19:30:57] ...trap/engines/engine.py.229: process: 0.84242 | load: 0.00295 [I 2022-10-09 19:30:57] ...trap/engines/engine.py.230: loss: 0.22543 [I 2022-10-09 19:30:57] ...optimizers/trijoint.py.151: total_norm: 15.839190 [I 2022-10-09 19:31:05] ...trap/engines/engine.py.224: train: epoch 52 | batch 160/2304 [I 2022-10-09 19:31:05] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:10 [I 2022-10-09 19:31:05] ...trap/engines/engine.py.229: process: 0.84338 | load: 0.00303 [I 2022-10-09 19:31:05] ...trap/engines/engine.py.230: loss: 0.15877 [I 2022-10-09 19:31:05] ...optimizers/trijoint.py.151: total_norm: 20.965771 [I 2022-10-09 19:31:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 170/2304 [I 2022-10-09 19:31:14] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:53 [I 2022-10-09 19:31:14] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00294 [I 2022-10-09 19:31:14] ...trap/engines/engine.py.230: loss: 0.16982 [I 2022-10-09 19:31:14] ...optimizers/trijoint.py.151: total_norm: 32.108215 [I 2022-10-09 19:31:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 180/2304 [I 2022-10-09 19:31:22] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:51 [I 2022-10-09 19:31:22] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00294 [I 2022-10-09 19:31:22] ...trap/engines/engine.py.230: loss: 0.19146 [I 2022-10-09 19:31:22] ...optimizers/trijoint.py.151: total_norm: 20.762871 [I 2022-10-09 19:31:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 190/2304 [I 2022-10-09 19:31:30] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:31 [I 2022-10-09 19:31:30] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00312 [I 2022-10-09 19:31:30] ...trap/engines/engine.py.230: loss: 0.11475 [I 2022-10-09 19:31:30] ...optimizers/trijoint.py.151: total_norm: 22.191849 [I 2022-10-09 19:31:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 200/2304 [I 2022-10-09 19:31:39] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:39 [I 2022-10-09 19:31:39] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00295 [I 2022-10-09 19:31:39] ...trap/engines/engine.py.230: loss: 0.17083 [I 2022-10-09 19:31:39] ...optimizers/trijoint.py.151: total_norm: 27.335760 [I 2022-10-09 19:31:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 210/2304 [I 2022-10-09 19:31:47] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:31 [I 2022-10-09 19:31:47] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00297 [I 2022-10-09 19:31:47] ...trap/engines/engine.py.230: loss: 0.27695 [I 2022-10-09 19:31:47] ...optimizers/trijoint.py.151: total_norm: 16.141233 [I 2022-10-09 19:31:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 220/2304 [I 2022-10-09 19:31:56] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:15 [I 2022-10-09 19:31:56] ...trap/engines/engine.py.229: process: 0.84129 | load: 0.00296 [I 2022-10-09 19:31:56] ...trap/engines/engine.py.230: loss: 0.16960 [I 2022-10-09 19:31:56] ...optimizers/trijoint.py.151: total_norm: 11.015591 [I 2022-10-09 19:32:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 230/2304 [I 2022-10-09 19:32:04] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:06 [I 2022-10-09 19:32:04] ...trap/engines/engine.py.229: process: 0.83504 | load: 0.00307 [I 2022-10-09 19:32:04] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-09 19:32:04] ...optimizers/trijoint.py.151: total_norm: 11.277464 [I 2022-10-09 19:32:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 240/2304 [I 2022-10-09 19:32:13] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:50 [I 2022-10-09 19:32:13] ...trap/engines/engine.py.229: process: 0.83464 | load: 0.00312 [I 2022-10-09 19:32:13] ...trap/engines/engine.py.230: loss: 0.17696 [I 2022-10-09 19:32:13] ...optimizers/trijoint.py.151: total_norm: 22.064930 [I 2022-10-09 19:32:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 250/2304 [I 2022-10-09 19:32:21] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:46 [I 2022-10-09 19:32:21] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00302 [I 2022-10-09 19:32:21] ...trap/engines/engine.py.230: loss: 0.12388 [I 2022-10-09 19:32:21] ...optimizers/trijoint.py.151: total_norm: 21.189936 [I 2022-10-09 19:32:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 260/2304 [I 2022-10-09 19:32:30] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:44 [I 2022-10-09 19:32:30] ...trap/engines/engine.py.229: process: 0.84135 | load: 0.00312 [I 2022-10-09 19:32:30] ...trap/engines/engine.py.230: loss: 0.18398 [I 2022-10-09 19:32:30] ...optimizers/trijoint.py.151: total_norm: 25.813065 [I 2022-10-09 19:32:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 270/2304 [I 2022-10-09 19:32:38] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:45 [I 2022-10-09 19:32:38] ...trap/engines/engine.py.229: process: 0.84975 | load: 0.00314 [I 2022-10-09 19:32:38] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-09 19:32:38] ...optimizers/trijoint.py.151: total_norm: 8.556705 [I 2022-10-09 19:32:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 280/2304 [I 2022-10-09 19:32:47] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:26 [I 2022-10-09 19:32:47] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00300 [I 2022-10-09 19:32:47] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-09 19:32:47] ...optimizers/trijoint.py.151: total_norm: 8.326003 [I 2022-10-09 19:32:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 290/2304 [I 2022-10-09 19:32:55] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:11 [I 2022-10-09 19:32:55] ...trap/engines/engine.py.229: process: 0.82789 | load: 0.00305 [I 2022-10-09 19:32:55] ...trap/engines/engine.py.230: loss: 0.22359 [I 2022-10-09 19:32:55] ...optimizers/trijoint.py.151: total_norm: 17.624887 [I 2022-10-09 19:33:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 300/2304 [I 2022-10-09 19:33:03] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:15 [I 2022-10-09 19:33:03] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00315 [I 2022-10-09 19:33:03] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-09 19:33:03] ...optimizers/trijoint.py.151: total_norm: 19.153790 [I 2022-10-09 19:33:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 310/2304 [I 2022-10-09 19:33:12] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:27:59 [I 2022-10-09 19:33:12] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00305 [I 2022-10-09 19:33:12] ...trap/engines/engine.py.230: loss: 0.14440 [I 2022-10-09 19:33:12] ...optimizers/trijoint.py.151: total_norm: 18.408659 [I 2022-10-09 19:33:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 320/2304 [I 2022-10-09 19:33:20] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:52 [I 2022-10-09 19:33:20] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00296 [I 2022-10-09 19:33:20] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-09 19:33:20] ...optimizers/trijoint.py.151: total_norm: 20.910305 [I 2022-10-09 19:33:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 330/2304 [I 2022-10-09 19:33:29] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:44 [I 2022-10-09 19:33:29] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00294 [I 2022-10-09 19:33:29] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-09 19:33:29] ...optimizers/trijoint.py.151: total_norm: 10.987813 [I 2022-10-09 19:33:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 340/2304 [I 2022-10-09 19:33:37] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:32 [I 2022-10-09 19:33:37] ...trap/engines/engine.py.229: process: 0.83587 | load: 0.00297 [I 2022-10-09 19:33:37] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-09 19:33:37] ...optimizers/trijoint.py.151: total_norm: 15.821898 [I 2022-10-09 19:33:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 350/2304 [I 2022-10-09 19:33:46] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:32 [I 2022-10-09 19:33:46] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00296 [I 2022-10-09 19:33:46] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-09 19:33:46] ...optimizers/trijoint.py.151: total_norm: 34.584785 [I 2022-10-09 19:33:54] ...trap/engines/engine.py.224: train: epoch 52 | batch 360/2304 [I 2022-10-09 19:33:54] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:13 [I 2022-10-09 19:33:54] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00306 [I 2022-10-09 19:33:54] ...trap/engines/engine.py.230: loss: 0.22371 [I 2022-10-09 19:33:54] ...optimizers/trijoint.py.151: total_norm: 11.995845 [I 2022-10-09 19:34:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 370/2304 [I 2022-10-09 19:34:03] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:09 [I 2022-10-09 19:34:03] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00304 [I 2022-10-09 19:34:03] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-09 19:34:03] ...optimizers/trijoint.py.151: total_norm: 23.887674 [I 2022-10-09 19:34:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 380/2304 [I 2022-10-09 19:34:11] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:26:59 [I 2022-10-09 19:34:11] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00296 [I 2022-10-09 19:34:11] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-09 19:34:11] ...optimizers/trijoint.py.151: total_norm: 14.263400 [I 2022-10-09 19:34:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 390/2304 [I 2022-10-09 19:34:19] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:51 [I 2022-10-09 19:34:19] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00294 [I 2022-10-09 19:34:19] ...trap/engines/engine.py.230: loss: 0.14741 [I 2022-10-09 19:34:19] ...optimizers/trijoint.py.151: total_norm: 12.282628 [I 2022-10-09 19:34:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 400/2304 [I 2022-10-09 19:34:28] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:50 [I 2022-10-09 19:34:28] ...trap/engines/engine.py.229: process: 0.86251 | load: 0.00305 [I 2022-10-09 19:34:28] ...trap/engines/engine.py.230: loss: 0.16660 [I 2022-10-09 19:34:28] ...optimizers/trijoint.py.151: total_norm: 133.018784 [I 2022-10-09 19:34:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 410/2304 [I 2022-10-09 19:34:36] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:50 [I 2022-10-09 19:34:36] ...trap/engines/engine.py.229: process: 0.84928 | load: 0.00315 [I 2022-10-09 19:34:36] ...trap/engines/engine.py.230: loss: 0.14249 [I 2022-10-09 19:34:36] ...optimizers/trijoint.py.151: total_norm: 21.258928 [I 2022-10-09 19:34:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 420/2304 [I 2022-10-09 19:34:45] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:46 [I 2022-10-09 19:34:45] ...trap/engines/engine.py.229: process: 0.85714 | load: 0.00312 [I 2022-10-09 19:34:45] ...trap/engines/engine.py.230: loss: 0.16805 [I 2022-10-09 19:34:45] ...optimizers/trijoint.py.151: total_norm: 8.669956 [I 2022-10-09 19:34:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 430/2304 [I 2022-10-09 19:34:53] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:24 [I 2022-10-09 19:34:53] ...trap/engines/engine.py.229: process: 0.83120 | load: 0.00313 [I 2022-10-09 19:34:53] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-09 19:34:53] ...optimizers/trijoint.py.151: total_norm: 11.432042 [I 2022-10-09 19:35:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 440/2304 [I 2022-10-09 19:35:02] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:34 [I 2022-10-09 19:35:02] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00310 [I 2022-10-09 19:35:02] ...trap/engines/engine.py.230: loss: 0.14448 [I 2022-10-09 19:35:02] ...optimizers/trijoint.py.151: total_norm: 10.902072 [I 2022-10-09 19:35:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 450/2304 [I 2022-10-09 19:35:11] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:20 [I 2022-10-09 19:35:11] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00297 [I 2022-10-09 19:35:11] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-09 19:35:11] ...optimizers/trijoint.py.151: total_norm: 9.888718 [I 2022-10-09 19:35:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 460/2304 [I 2022-10-09 19:35:19] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:08 [I 2022-10-09 19:35:19] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00312 [I 2022-10-09 19:35:19] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-09 19:35:19] ...optimizers/trijoint.py.151: total_norm: 13.776456 [I 2022-10-09 19:35:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 470/2304 [I 2022-10-09 19:35:28] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:00 [I 2022-10-09 19:35:28] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00327 [I 2022-10-09 19:35:28] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-09 19:35:28] ...optimizers/trijoint.py.151: total_norm: 15.981248 [I 2022-10-09 19:35:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 480/2304 [I 2022-10-09 19:35:36] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:25:54 [I 2022-10-09 19:35:36] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00312 [I 2022-10-09 19:35:36] ...trap/engines/engine.py.230: loss: 0.20707 [I 2022-10-09 19:35:36] ...optimizers/trijoint.py.151: total_norm: 8.738449 [I 2022-10-09 19:35:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 490/2304 [I 2022-10-09 19:35:45] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:41 [I 2022-10-09 19:35:45] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00330 [I 2022-10-09 19:35:45] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-09 19:35:45] ...optimizers/trijoint.py.151: total_norm: 8.818185 [I 2022-10-09 19:35:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 500/2304 [I 2022-10-09 19:35:53] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:30 [I 2022-10-09 19:35:53] ...trap/engines/engine.py.229: process: 0.84426 | load: 0.00315 [I 2022-10-09 19:35:53] ...trap/engines/engine.py.230: loss: 0.16198 [I 2022-10-09 19:35:53] ...optimizers/trijoint.py.151: total_norm: 16.870583 [I 2022-10-09 19:36:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 510/2304 [I 2022-10-09 19:36:02] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:17 [I 2022-10-09 19:36:02] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00312 [I 2022-10-09 19:36:02] ...trap/engines/engine.py.230: loss: 0.16903 [I 2022-10-09 19:36:02] ...optimizers/trijoint.py.151: total_norm: 8.971934 [I 2022-10-09 19:36:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 520/2304 [I 2022-10-09 19:36:10] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:22 [I 2022-10-09 19:36:10] ...trap/engines/engine.py.229: process: 0.86429 | load: 0.00309 [I 2022-10-09 19:36:10] ...trap/engines/engine.py.230: loss: 0.21544 [I 2022-10-09 19:36:10] ...optimizers/trijoint.py.151: total_norm: 15.625516 [I 2022-10-09 19:36:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 530/2304 [I 2022-10-09 19:36:19] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:23 [I 2022-10-09 19:36:19] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00314 [I 2022-10-09 19:36:19] ...trap/engines/engine.py.230: loss: 0.16821 [I 2022-10-09 19:36:19] ...optimizers/trijoint.py.151: total_norm: 9.976852 [I 2022-10-09 19:36:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 540/2304 [I 2022-10-09 19:36:27] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:24:56 [I 2022-10-09 19:36:27] ...trap/engines/engine.py.229: process: 0.83566 | load: 0.00314 [I 2022-10-09 19:36:27] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-09 19:36:27] ...optimizers/trijoint.py.151: total_norm: 7.841067 [I 2022-10-09 19:36:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 550/2304 [I 2022-10-09 19:36:36] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:00 [I 2022-10-09 19:36:36] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00313 [I 2022-10-09 19:36:36] ...trap/engines/engine.py.230: loss: 0.11346 [I 2022-10-09 19:36:36] ...optimizers/trijoint.py.151: total_norm: 16.402018 [I 2022-10-09 19:36:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 560/2304 [I 2022-10-09 19:36:44] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:24:56 [I 2022-10-09 19:36:44] ...trap/engines/engine.py.229: process: 0.85796 | load: 0.00312 [I 2022-10-09 19:36:44] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-09 19:36:44] ...optimizers/trijoint.py.151: total_norm: 20.266495 [I 2022-10-09 19:36:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 570/2304 [I 2022-10-09 19:36:53] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:44 [I 2022-10-09 19:36:53] ...trap/engines/engine.py.229: process: 0.86715 | load: 0.00313 [I 2022-10-09 19:36:53] ...trap/engines/engine.py.230: loss: 0.16880 [I 2022-10-09 19:36:53] ...optimizers/trijoint.py.151: total_norm: 34.506367 [I 2022-10-09 19:37:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 580/2304 [I 2022-10-09 19:37:02] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:32 [I 2022-10-09 19:37:02] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00311 [I 2022-10-09 19:37:02] ...trap/engines/engine.py.230: loss: 0.10508 [I 2022-10-09 19:37:02] ...optimizers/trijoint.py.151: total_norm: 9.129194 [I 2022-10-09 19:37:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 590/2304 [I 2022-10-09 19:37:10] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:20 [I 2022-10-09 19:37:10] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00318 [I 2022-10-09 19:37:10] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-09 19:37:10] ...optimizers/trijoint.py.151: total_norm: 12.434166 [I 2022-10-09 19:37:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 600/2304 [I 2022-10-09 19:37:19] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:11 [I 2022-10-09 19:37:19] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00303 [I 2022-10-09 19:37:19] ...trap/engines/engine.py.230: loss: 0.13579 [I 2022-10-09 19:37:19] ...optimizers/trijoint.py.151: total_norm: 17.411160 [I 2022-10-09 19:37:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 610/2304 [I 2022-10-09 19:37:27] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:13 [I 2022-10-09 19:37:27] ...trap/engines/engine.py.229: process: 0.85383 | load: 0.00304 [I 2022-10-09 19:37:27] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-09 19:37:27] ...optimizers/trijoint.py.151: total_norm: 15.397058 [I 2022-10-09 19:37:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 620/2304 [I 2022-10-09 19:37:36] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:23:54 [I 2022-10-09 19:37:36] ...trap/engines/engine.py.229: process: 0.86011 | load: 0.00318 [I 2022-10-09 19:37:36] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-09 19:37:36] ...optimizers/trijoint.py.151: total_norm: 18.322325 [I 2022-10-09 19:37:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 630/2304 [I 2022-10-09 19:37:44] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:48 [I 2022-10-09 19:37:44] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00308 [I 2022-10-09 19:37:44] ...trap/engines/engine.py.230: loss: 0.26055 [I 2022-10-09 19:37:44] ...optimizers/trijoint.py.151: total_norm: 21.318029 [I 2022-10-09 19:37:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 640/2304 [I 2022-10-09 19:37:53] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:36 [I 2022-10-09 19:37:53] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00316 [I 2022-10-09 19:37:53] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-09 19:37:53] ...optimizers/trijoint.py.151: total_norm: 10.953509 [I 2022-10-09 19:38:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 650/2304 [I 2022-10-09 19:38:01] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:29 [I 2022-10-09 19:38:01] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00314 [I 2022-10-09 19:38:01] ...trap/engines/engine.py.230: loss: 0.18464 [I 2022-10-09 19:38:01] ...optimizers/trijoint.py.151: total_norm: 40.617836 [I 2022-10-09 19:38:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 660/2304 [I 2022-10-09 19:38:10] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:16 [I 2022-10-09 19:38:10] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00296 [I 2022-10-09 19:38:10] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-09 19:38:10] ...optimizers/trijoint.py.151: total_norm: 12.912705 [I 2022-10-09 19:38:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 670/2304 [I 2022-10-09 19:38:18] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:08 [I 2022-10-09 19:38:18] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00313 [I 2022-10-09 19:38:18] ...trap/engines/engine.py.230: loss: 0.11379 [I 2022-10-09 19:38:18] ...optimizers/trijoint.py.151: total_norm: 8.573608 [I 2022-10-09 19:38:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 680/2304 [I 2022-10-09 19:38:27] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:00 [I 2022-10-09 19:38:27] ...trap/engines/engine.py.229: process: 0.83413 | load: 0.00317 [I 2022-10-09 19:38:27] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-09 19:38:27] ...optimizers/trijoint.py.151: total_norm: 19.375851 [I 2022-10-09 19:38:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 690/2304 [I 2022-10-09 19:38:36] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:22:54 [I 2022-10-09 19:38:36] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00328 [I 2022-10-09 19:38:36] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-09 19:38:36] ...optimizers/trijoint.py.151: total_norm: 13.316734 [I 2022-10-09 19:38:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 700/2304 [I 2022-10-09 19:38:44] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:22:50 [I 2022-10-09 19:38:44] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00302 [I 2022-10-09 19:38:44] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-09 19:38:44] ...optimizers/trijoint.py.151: total_norm: 12.378963 [I 2022-10-09 19:38:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 710/2304 [I 2022-10-09 19:38:53] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:33 [I 2022-10-09 19:38:53] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00312 [I 2022-10-09 19:38:53] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-09 19:38:53] ...optimizers/trijoint.py.151: total_norm: 9.815089 [I 2022-10-09 19:39:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 720/2304 [I 2022-10-09 19:39:01] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:24 [I 2022-10-09 19:39:01] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00312 [I 2022-10-09 19:39:01] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-09 19:39:01] ...optimizers/trijoint.py.151: total_norm: 12.474847 [I 2022-10-09 19:39:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 730/2304 [I 2022-10-09 19:39:10] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:19 [I 2022-10-09 19:39:10] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00309 [I 2022-10-09 19:39:10] ...trap/engines/engine.py.230: loss: 0.13200 [I 2022-10-09 19:39:10] ...optimizers/trijoint.py.151: total_norm: 13.684690 [I 2022-10-09 19:39:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 740/2304 [I 2022-10-09 19:39:18] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:16 [I 2022-10-09 19:39:18] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00310 [I 2022-10-09 19:39:18] ...trap/engines/engine.py.230: loss: 0.18871 [I 2022-10-09 19:39:18] ...optimizers/trijoint.py.151: total_norm: 17.592543 [I 2022-10-09 19:39:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 750/2304 [I 2022-10-09 19:39:27] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:22:03 [I 2022-10-09 19:39:27] ...trap/engines/engine.py.229: process: 0.84423 | load: 0.00296 [I 2022-10-09 19:39:27] ...trap/engines/engine.py.230: loss: 0.20465 [I 2022-10-09 19:39:27] ...optimizers/trijoint.py.151: total_norm: 20.304853 [I 2022-10-09 19:39:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 760/2304 [I 2022-10-09 19:39:35] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:47 [I 2022-10-09 19:39:35] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00301 [I 2022-10-09 19:39:35] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-09 19:39:35] ...optimizers/trijoint.py.151: total_norm: 13.641072 [I 2022-10-09 19:39:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 770/2304 [I 2022-10-09 19:39:44] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:21:45 [I 2022-10-09 19:39:44] ...trap/engines/engine.py.229: process: 0.85042 | load: 0.00308 [I 2022-10-09 19:39:44] ...trap/engines/engine.py.230: loss: 0.22480 [I 2022-10-09 19:39:44] ...optimizers/trijoint.py.151: total_norm: 20.859312 [I 2022-10-09 19:39:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 780/2304 [I 2022-10-09 19:39:52] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:30 [I 2022-10-09 19:39:52] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00294 [I 2022-10-09 19:39:52] ...trap/engines/engine.py.230: loss: 0.16355 [I 2022-10-09 19:39:52] ...optimizers/trijoint.py.151: total_norm: 20.857328 [I 2022-10-09 19:40:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 790/2304 [I 2022-10-09 19:40:01] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:16 [I 2022-10-09 19:40:01] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00299 [I 2022-10-09 19:40:01] ...trap/engines/engine.py.230: loss: 0.15530 [I 2022-10-09 19:40:01] ...optimizers/trijoint.py.151: total_norm: 26.192865 [I 2022-10-09 19:40:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 800/2304 [I 2022-10-09 19:40:09] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:08 [I 2022-10-09 19:40:09] ...trap/engines/engine.py.229: process: 0.83563 | load: 0.00291 [I 2022-10-09 19:40:09] ...trap/engines/engine.py.230: loss: 0.16070 [I 2022-10-09 19:40:09] ...optimizers/trijoint.py.151: total_norm: 21.313902 [I 2022-10-09 19:40:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 810/2304 [I 2022-10-09 19:40:17] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:20:56 [I 2022-10-09 19:40:17] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00297 [I 2022-10-09 19:40:17] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-09 19:40:17] ...optimizers/trijoint.py.151: total_norm: 13.218294 [I 2022-10-09 19:40:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 820/2304 [I 2022-10-09 19:40:26] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:20:55 [I 2022-10-09 19:40:26] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00294 [I 2022-10-09 19:40:26] ...trap/engines/engine.py.230: loss: 0.18426 [I 2022-10-09 19:40:26] ...optimizers/trijoint.py.151: total_norm: 14.512520 [I 2022-10-09 19:40:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 830/2304 [I 2022-10-09 19:40:34] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:36 [I 2022-10-09 19:40:34] ...trap/engines/engine.py.229: process: 0.83459 | load: 0.00291 [I 2022-10-09 19:40:34] ...trap/engines/engine.py.230: loss: 0.12509 [I 2022-10-09 19:40:34] ...optimizers/trijoint.py.151: total_norm: 14.119809 [I 2022-10-09 19:40:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 840/2304 [I 2022-10-09 19:40:43] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:30 [I 2022-10-09 19:40:43] ...trap/engines/engine.py.229: process: 0.84549 | load: 0.00293 [I 2022-10-09 19:40:43] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-09 19:40:43] ...optimizers/trijoint.py.151: total_norm: 16.239449 [I 2022-10-09 19:40:51] ...trap/engines/engine.py.224: train: epoch 52 | batch 850/2304 [I 2022-10-09 19:40:51] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:22 [I 2022-10-09 19:40:51] ...trap/engines/engine.py.229: process: 0.82784 | load: 0.00293 [I 2022-10-09 19:40:51] ...trap/engines/engine.py.230: loss: 0.17998 [I 2022-10-09 19:40:51] ...optimizers/trijoint.py.151: total_norm: 20.673018 [I 2022-10-09 19:41:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 860/2304 [I 2022-10-09 19:41:00] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:15 [I 2022-10-09 19:41:00] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00293 [I 2022-10-09 19:41:00] ...trap/engines/engine.py.230: loss: 0.16297 [I 2022-10-09 19:41:00] ...optimizers/trijoint.py.151: total_norm: 19.969547 [I 2022-10-09 19:41:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 870/2304 [I 2022-10-09 19:41:08] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:08 [I 2022-10-09 19:41:08] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00292 [I 2022-10-09 19:41:08] ...trap/engines/engine.py.230: loss: 0.14124 [I 2022-10-09 19:41:08] ...optimizers/trijoint.py.151: total_norm: 20.710306 [I 2022-10-09 19:41:16] ...trap/engines/engine.py.224: train: epoch 52 | batch 880/2304 [I 2022-10-09 19:41:16] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:19:53 [I 2022-10-09 19:41:16] ...trap/engines/engine.py.229: process: 0.82833 | load: 0.00318 [I 2022-10-09 19:41:16] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-09 19:41:16] ...optimizers/trijoint.py.151: total_norm: 13.601871 [I 2022-10-09 19:41:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 890/2304 [I 2022-10-09 19:41:25] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:19:51 [I 2022-10-09 19:41:25] ...trap/engines/engine.py.229: process: 0.83631 | load: 0.00293 [I 2022-10-09 19:41:25] ...trap/engines/engine.py.230: loss: 0.14224 [I 2022-10-09 19:41:25] ...optimizers/trijoint.py.151: total_norm: 8.592841 [I 2022-10-09 19:41:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 900/2304 [I 2022-10-09 19:41:33] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:41 [I 2022-10-09 19:41:33] ...trap/engines/engine.py.229: process: 0.83153 | load: 0.00300 [I 2022-10-09 19:41:33] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-09 19:41:33] ...optimizers/trijoint.py.151: total_norm: 17.871502 [I 2022-10-09 19:41:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 910/2304 [I 2022-10-09 19:41:42] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:19:32 [I 2022-10-09 19:41:42] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00294 [I 2022-10-09 19:41:42] ...trap/engines/engine.py.230: loss: 0.23587 [I 2022-10-09 19:41:42] ...optimizers/trijoint.py.151: total_norm: 18.313267 [I 2022-10-09 19:41:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 920/2304 [I 2022-10-09 19:41:50] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:26 [I 2022-10-09 19:41:50] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00298 [I 2022-10-09 19:41:50] ...trap/engines/engine.py.230: loss: 0.17977 [I 2022-10-09 19:41:50] ...optimizers/trijoint.py.151: total_norm: 13.543008 [I 2022-10-09 19:41:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 930/2304 [I 2022-10-09 19:41:59] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:23 [I 2022-10-09 19:41:59] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00294 [I 2022-10-09 19:41:59] ...trap/engines/engine.py.230: loss: 0.16963 [I 2022-10-09 19:41:59] ...optimizers/trijoint.py.151: total_norm: 10.834374 [I 2022-10-09 19:42:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 940/2304 [I 2022-10-09 19:42:07] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:08 [I 2022-10-09 19:42:07] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00294 [I 2022-10-09 19:42:07] ...trap/engines/engine.py.230: loss: 0.31298 [I 2022-10-09 19:42:07] ...optimizers/trijoint.py.151: total_norm: 20.546267 [I 2022-10-09 19:42:16] ...trap/engines/engine.py.224: train: epoch 52 | batch 950/2304 [I 2022-10-09 19:42:16] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:02 [I 2022-10-09 19:42:16] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00295 [I 2022-10-09 19:42:16] ...trap/engines/engine.py.230: loss: 0.17611 [I 2022-10-09 19:42:16] ...optimizers/trijoint.py.151: total_norm: 15.493402 [I 2022-10-09 19:42:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 960/2304 [I 2022-10-09 19:42:24] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:18:48 [I 2022-10-09 19:42:24] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00301 [I 2022-10-09 19:42:24] ...trap/engines/engine.py.230: loss: 0.13974 [I 2022-10-09 19:42:24] ...optimizers/trijoint.py.151: total_norm: 8.889833 [I 2022-10-09 19:42:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 970/2304 [I 2022-10-09 19:42:33] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:18:45 [I 2022-10-09 19:42:33] ...trap/engines/engine.py.229: process: 0.84468 | load: 0.00293 [I 2022-10-09 19:42:33] ...trap/engines/engine.py.230: loss: 0.14700 [I 2022-10-09 19:42:33] ...optimizers/trijoint.py.151: total_norm: 19.849451 [I 2022-10-09 19:42:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 980/2304 [I 2022-10-09 19:42:41] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:33 [I 2022-10-09 19:42:41] ...trap/engines/engine.py.229: process: 0.83891 | load: 0.00295 [I 2022-10-09 19:42:41] ...trap/engines/engine.py.230: loss: 0.15167 [I 2022-10-09 19:42:41] ...optimizers/trijoint.py.151: total_norm: 12.742987 [I 2022-10-09 19:42:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 990/2304 [I 2022-10-09 19:42:49] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:33 [I 2022-10-09 19:42:49] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00293 [I 2022-10-09 19:42:49] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-09 19:42:49] ...optimizers/trijoint.py.151: total_norm: 15.680813 [I 2022-10-09 19:42:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 1000/2304 [I 2022-10-09 19:42:58] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:18 [I 2022-10-09 19:42:58] ...trap/engines/engine.py.229: process: 0.83624 | load: 0.00296 [I 2022-10-09 19:42:58] ...trap/engines/engine.py.230: loss: 0.12331 [I 2022-10-09 19:42:58] ...optimizers/trijoint.py.151: total_norm: 13.625163 [I 2022-10-09 19:43:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 1010/2304 [I 2022-10-09 19:43:06] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:09 [I 2022-10-09 19:43:06] ...trap/engines/engine.py.229: process: 0.83717 | load: 0.00301 [I 2022-10-09 19:43:06] ...trap/engines/engine.py.230: loss: 0.16318 [I 2022-10-09 19:43:06] ...optimizers/trijoint.py.151: total_norm: 10.071598 [I 2022-10-09 19:43:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 1020/2304 [I 2022-10-09 19:43:15] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:03 [I 2022-10-09 19:43:15] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00296 [I 2022-10-09 19:43:15] ...trap/engines/engine.py.230: loss: 0.17671 [I 2022-10-09 19:43:15] ...optimizers/trijoint.py.151: total_norm: 9.192761 [I 2022-10-09 19:43:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 1030/2304 [I 2022-10-09 19:43:23] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:17:48 [I 2022-10-09 19:43:23] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00296 [I 2022-10-09 19:43:23] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-09 19:43:23] ...optimizers/trijoint.py.151: total_norm: 20.630424 [I 2022-10-09 19:43:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 1040/2304 [I 2022-10-09 19:43:32] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:17:42 [I 2022-10-09 19:43:32] ...trap/engines/engine.py.229: process: 0.83499 | load: 0.00294 [I 2022-10-09 19:43:32] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-09 19:43:32] ...optimizers/trijoint.py.151: total_norm: 12.424634 [I 2022-10-09 19:43:40] ...trap/engines/engine.py.224: train: epoch 52 | batch 1050/2304 [I 2022-10-09 19:43:40] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:17:38 [I 2022-10-09 19:43:40] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00293 [I 2022-10-09 19:43:40] ...trap/engines/engine.py.230: loss: 0.12090 [I 2022-10-09 19:43:40] ...optimizers/trijoint.py.151: total_norm: 14.070923 [I 2022-10-09 19:43:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 1060/2304 [I 2022-10-09 19:43:49] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:29 [I 2022-10-09 19:43:49] ...trap/engines/engine.py.229: process: 0.85567 | load: 0.00293 [I 2022-10-09 19:43:49] ...trap/engines/engine.py.230: loss: 0.12622 [I 2022-10-09 19:43:49] ...optimizers/trijoint.py.151: total_norm: 13.757448 [I 2022-10-09 19:43:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 1070/2304 [I 2022-10-09 19:43:57] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:17:19 [I 2022-10-09 19:43:57] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00292 [I 2022-10-09 19:43:57] ...trap/engines/engine.py.230: loss: 0.19336 [I 2022-10-09 19:43:57] ...optimizers/trijoint.py.151: total_norm: 14.321886 [I 2022-10-09 19:44:05] ...trap/engines/engine.py.224: train: epoch 52 | batch 1080/2304 [I 2022-10-09 19:44:05] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:11 [I 2022-10-09 19:44:05] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00294 [I 2022-10-09 19:44:05] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-09 19:44:05] ...optimizers/trijoint.py.151: total_norm: 19.902792 [I 2022-10-09 19:44:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 1090/2304 [I 2022-10-09 19:44:14] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:05 [I 2022-10-09 19:44:14] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00291 [I 2022-10-09 19:44:14] ...trap/engines/engine.py.230: loss: 0.13595 [I 2022-10-09 19:44:14] ...optimizers/trijoint.py.151: total_norm: 12.991444 [I 2022-10-09 19:44:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 1100/2304 [I 2022-10-09 19:44:22] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:16:58 [I 2022-10-09 19:44:22] ...trap/engines/engine.py.229: process: 0.83818 | load: 0.00297 [I 2022-10-09 19:44:22] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-09 19:44:22] ...optimizers/trijoint.py.151: total_norm: 10.293762 [I 2022-10-09 19:44:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 1110/2304 [I 2022-10-09 19:44:31] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:16:51 [I 2022-10-09 19:44:31] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00305 [I 2022-10-09 19:44:31] ...trap/engines/engine.py.230: loss: 0.10877 [I 2022-10-09 19:44:31] ...optimizers/trijoint.py.151: total_norm: 17.771090 [I 2022-10-09 19:44:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 1120/2304 [I 2022-10-09 19:44:39] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:16:38 [I 2022-10-09 19:44:39] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00296 [I 2022-10-09 19:44:39] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-09 19:44:39] ...optimizers/trijoint.py.151: total_norm: 31.376972 [I 2022-10-09 19:44:48] ...trap/engines/engine.py.224: train: epoch 52 | batch 1130/2304 [I 2022-10-09 19:44:48] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:33 [I 2022-10-09 19:44:48] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00296 [I 2022-10-09 19:44:48] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-09 19:44:48] ...optimizers/trijoint.py.151: total_norm: 19.630346 [I 2022-10-09 19:44:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 1140/2304 [I 2022-10-09 19:44:56] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:23 [I 2022-10-09 19:44:56] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00301 [I 2022-10-09 19:44:56] ...trap/engines/engine.py.230: loss: 0.10656 [I 2022-10-09 19:44:56] ...optimizers/trijoint.py.151: total_norm: 11.994008 [I 2022-10-09 19:45:05] ...trap/engines/engine.py.224: train: epoch 52 | batch 1150/2304 [I 2022-10-09 19:45:05] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:12 [I 2022-10-09 19:45:05] ...trap/engines/engine.py.229: process: 0.83106 | load: 0.00295 [I 2022-10-09 19:45:05] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-09 19:45:05] ...optimizers/trijoint.py.151: total_norm: 19.024384 [I 2022-10-09 19:45:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 1160/2304 [I 2022-10-09 19:45:13] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:00 [I 2022-10-09 19:45:13] ...trap/engines/engine.py.229: process: 0.84292 | load: 0.00294 [I 2022-10-09 19:45:13] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-09 19:45:13] ...optimizers/trijoint.py.151: total_norm: 30.369522 [I 2022-10-09 19:45:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 1170/2304 [I 2022-10-09 19:45:22] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:15:58 [I 2022-10-09 19:45:22] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00291 [I 2022-10-09 19:45:22] ...trap/engines/engine.py.230: loss: 0.11585 [I 2022-10-09 19:45:22] ...optimizers/trijoint.py.151: total_norm: 14.823347 [I 2022-10-09 19:45:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 1180/2304 [I 2022-10-09 19:45:30] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:15:45 [I 2022-10-09 19:45:30] ...trap/engines/engine.py.229: process: 0.84719 | load: 0.00291 [I 2022-10-09 19:45:30] ...trap/engines/engine.py.230: loss: 0.10981 [I 2022-10-09 19:45:30] ...optimizers/trijoint.py.151: total_norm: 8.512341 [I 2022-10-09 19:45:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1190/2304 [I 2022-10-09 19:45:38] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:15:40 [I 2022-10-09 19:45:38] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00296 [I 2022-10-09 19:45:38] ...trap/engines/engine.py.230: loss: 0.21322 [I 2022-10-09 19:45:38] ...optimizers/trijoint.py.151: total_norm: 24.835825 [I 2022-10-09 19:45:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 1200/2304 [I 2022-10-09 19:45:47] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:15:36 [I 2022-10-09 19:45:47] ...trap/engines/engine.py.229: process: 0.85391 | load: 0.00292 [I 2022-10-09 19:45:47] ...trap/engines/engine.py.230: loss: 0.13523 [I 2022-10-09 19:45:47] ...optimizers/trijoint.py.151: total_norm: 16.439020 [I 2022-10-09 19:45:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1210/2304 [I 2022-10-09 19:45:55] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:30 [I 2022-10-09 19:45:55] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00310 [I 2022-10-09 19:45:55] ...trap/engines/engine.py.230: loss: 0.15968 [I 2022-10-09 19:45:55] ...optimizers/trijoint.py.151: total_norm: 18.421085 [I 2022-10-09 19:46:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 1220/2304 [I 2022-10-09 19:46:04] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:22 [I 2022-10-09 19:46:04] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00316 [I 2022-10-09 19:46:04] ...trap/engines/engine.py.230: loss: 0.20912 [I 2022-10-09 19:46:04] ...optimizers/trijoint.py.151: total_norm: 15.850239 [I 2022-10-09 19:46:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 1230/2304 [I 2022-10-09 19:46:13] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:19 [I 2022-10-09 19:46:13] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00315 [I 2022-10-09 19:46:13] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-09 19:46:13] ...optimizers/trijoint.py.151: total_norm: 8.736109 [I 2022-10-09 19:46:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 1240/2304 [I 2022-10-09 19:46:21] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:10 [I 2022-10-09 19:46:21] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00316 [I 2022-10-09 19:46:21] ...trap/engines/engine.py.230: loss: 0.13446 [I 2022-10-09 19:46:21] ...optimizers/trijoint.py.151: total_norm: 36.809887 [I 2022-10-09 19:46:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 1250/2304 [I 2022-10-09 19:46:30] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:14:59 [I 2022-10-09 19:46:30] ...trap/engines/engine.py.229: process: 0.86366 | load: 0.00314 [I 2022-10-09 19:46:30] ...trap/engines/engine.py.230: loss: 0.13434 [I 2022-10-09 19:46:30] ...optimizers/trijoint.py.151: total_norm: 18.794937 [I 2022-10-09 19:46:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1260/2304 [I 2022-10-09 19:46:38] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:14:41 [I 2022-10-09 19:46:38] ...trap/engines/engine.py.229: process: 0.82958 | load: 0.00313 [I 2022-10-09 19:46:38] ...trap/engines/engine.py.230: loss: 0.19726 [I 2022-10-09 19:46:38] ...optimizers/trijoint.py.151: total_norm: 13.342687 [I 2022-10-09 19:46:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 1270/2304 [I 2022-10-09 19:46:47] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:42 [I 2022-10-09 19:46:47] ...trap/engines/engine.py.229: process: 0.85743 | load: 0.00313 [I 2022-10-09 19:46:47] ...trap/engines/engine.py.230: loss: 0.22101 [I 2022-10-09 19:46:47] ...optimizers/trijoint.py.151: total_norm: 17.352131 [I 2022-10-09 19:46:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1280/2304 [I 2022-10-09 19:46:55] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:30 [I 2022-10-09 19:46:55] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00311 [I 2022-10-09 19:46:55] ...trap/engines/engine.py.230: loss: 0.13429 [I 2022-10-09 19:46:55] ...optimizers/trijoint.py.151: total_norm: 24.221859 [I 2022-10-09 19:47:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 1290/2304 [I 2022-10-09 19:47:04] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:27 [I 2022-10-09 19:47:04] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00310 [I 2022-10-09 19:47:04] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-09 19:47:04] ...optimizers/trijoint.py.151: total_norm: 21.775757 [I 2022-10-09 19:47:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 1300/2304 [I 2022-10-09 19:47:12] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:17 [I 2022-10-09 19:47:12] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00312 [I 2022-10-09 19:47:12] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-09 19:47:12] ...optimizers/trijoint.py.151: total_norm: 10.015606 [I 2022-10-09 19:47:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 1310/2304 [I 2022-10-09 19:47:21] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:04 [I 2022-10-09 19:47:21] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00313 [I 2022-10-09 19:47:21] ...trap/engines/engine.py.230: loss: 0.19632 [I 2022-10-09 19:47:21] ...optimizers/trijoint.py.151: total_norm: 11.885269 [I 2022-10-09 19:47:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1320/2304 [I 2022-10-09 19:47:29] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:13:57 [I 2022-10-09 19:47:29] ...trap/engines/engine.py.229: process: 0.85024 | load: 0.00312 [I 2022-10-09 19:47:29] ...trap/engines/engine.py.230: loss: 0.12479 [I 2022-10-09 19:47:29] ...optimizers/trijoint.py.151: total_norm: 17.339218 [I 2022-10-09 19:47:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1330/2304 [I 2022-10-09 19:47:38] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:13:48 [I 2022-10-09 19:47:38] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00324 [I 2022-10-09 19:47:38] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-09 19:47:38] ...optimizers/trijoint.py.151: total_norm: 18.818487 [I 2022-10-09 19:47:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1340/2304 [I 2022-10-09 19:47:46] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:41 [I 2022-10-09 19:47:46] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00309 [I 2022-10-09 19:47:46] ...trap/engines/engine.py.230: loss: 0.20034 [I 2022-10-09 19:47:46] ...optimizers/trijoint.py.151: total_norm: 21.865759 [I 2022-10-09 19:47:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1350/2304 [I 2022-10-09 19:47:55] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:13:34 [I 2022-10-09 19:47:55] ...trap/engines/engine.py.229: process: 0.86284 | load: 0.00315 [I 2022-10-09 19:47:55] ...trap/engines/engine.py.230: loss: 0.12511 [I 2022-10-09 19:47:55] ...optimizers/trijoint.py.151: total_norm: 12.569281 [I 2022-10-09 19:48:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1360/2304 [I 2022-10-09 19:48:03] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:21 [I 2022-10-09 19:48:03] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00327 [I 2022-10-09 19:48:03] ...trap/engines/engine.py.230: loss: 0.10859 [I 2022-10-09 19:48:03] ...optimizers/trijoint.py.151: total_norm: 8.803017 [I 2022-10-09 19:48:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 1370/2304 [I 2022-10-09 19:48:12] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:10 [I 2022-10-09 19:48:12] ...trap/engines/engine.py.229: process: 0.83297 | load: 0.00306 [I 2022-10-09 19:48:12] ...trap/engines/engine.py.230: loss: 0.19418 [I 2022-10-09 19:48:12] ...optimizers/trijoint.py.151: total_norm: 17.243813 [I 2022-10-09 19:48:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 1380/2304 [I 2022-10-09 19:48:21] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:13:07 [I 2022-10-09 19:48:21] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00315 [I 2022-10-09 19:48:21] ...trap/engines/engine.py.230: loss: 0.19628 [I 2022-10-09 19:48:21] ...optimizers/trijoint.py.151: total_norm: 20.510794 [I 2022-10-09 19:48:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1390/2304 [I 2022-10-09 19:48:29] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:00 [I 2022-10-09 19:48:29] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00309 [I 2022-10-09 19:48:29] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-09 19:48:29] ...optimizers/trijoint.py.151: total_norm: 18.514574 [I 2022-10-09 19:48:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1400/2304 [I 2022-10-09 19:48:38] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:12:47 [I 2022-10-09 19:48:38] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00315 [I 2022-10-09 19:48:38] ...trap/engines/engine.py.230: loss: 0.15874 [I 2022-10-09 19:48:38] ...optimizers/trijoint.py.151: total_norm: 16.789089 [I 2022-10-09 19:48:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1410/2304 [I 2022-10-09 19:48:46] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:12:42 [I 2022-10-09 19:48:46] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00311 [I 2022-10-09 19:48:46] ...trap/engines/engine.py.230: loss: 0.18820 [I 2022-10-09 19:48:46] ...optimizers/trijoint.py.151: total_norm: 9.588161 [I 2022-10-09 19:48:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1420/2304 [I 2022-10-09 19:48:55] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:12:33 [I 2022-10-09 19:48:55] ...trap/engines/engine.py.229: process: 0.86711 | load: 0.00310 [I 2022-10-09 19:48:55] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-09 19:48:55] ...optimizers/trijoint.py.151: total_norm: 13.210618 [I 2022-10-09 19:49:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1430/2304 [I 2022-10-09 19:49:03] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:21 [I 2022-10-09 19:49:03] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00311 [I 2022-10-09 19:49:03] ...trap/engines/engine.py.230: loss: 0.17345 [I 2022-10-09 19:49:03] ...optimizers/trijoint.py.151: total_norm: 16.738026 [I 2022-10-09 19:49:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 1440/2304 [I 2022-10-09 19:49:12] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:16 [I 2022-10-09 19:49:12] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00324 [I 2022-10-09 19:49:12] ...trap/engines/engine.py.230: loss: 0.18709 [I 2022-10-09 19:49:12] ...optimizers/trijoint.py.151: total_norm: 23.054171 [I 2022-10-09 19:49:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 1450/2304 [I 2022-10-09 19:49:20] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:03 [I 2022-10-09 19:49:20] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00313 [I 2022-10-09 19:49:20] ...trap/engines/engine.py.230: loss: 0.12260 [I 2022-10-09 19:49:20] ...optimizers/trijoint.py.151: total_norm: 17.248423 [I 2022-10-09 19:49:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1460/2304 [I 2022-10-09 19:49:29] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:11:57 [I 2022-10-09 19:49:29] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00305 [I 2022-10-09 19:49:29] ...trap/engines/engine.py.230: loss: 0.15436 [I 2022-10-09 19:49:29] ...optimizers/trijoint.py.151: total_norm: 24.953442 [I 2022-10-09 19:49:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 1470/2304 [I 2022-10-09 19:49:37] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:48 [I 2022-10-09 19:49:37] ...trap/engines/engine.py.229: process: 0.84334 | load: 0.00326 [I 2022-10-09 19:49:37] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-09 19:49:37] ...optimizers/trijoint.py.151: total_norm: 11.244321 [I 2022-10-09 19:49:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1480/2304 [I 2022-10-09 19:49:46] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:11:45 [I 2022-10-09 19:49:46] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00314 [I 2022-10-09 19:49:46] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-09 19:49:46] ...optimizers/trijoint.py.151: total_norm: 8.696162 [I 2022-10-09 19:49:54] ...trap/engines/engine.py.224: train: epoch 52 | batch 1490/2304 [I 2022-10-09 19:49:54] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:11:30 [I 2022-10-09 19:49:54] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00317 [I 2022-10-09 19:49:54] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-09 19:49:54] ...optimizers/trijoint.py.151: total_norm: 7.992536 [I 2022-10-09 19:50:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1500/2304 [I 2022-10-09 19:50:03] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:25 [I 2022-10-09 19:50:03] ...trap/engines/engine.py.229: process: 0.86677 | load: 0.00314 [I 2022-10-09 19:50:03] ...trap/engines/engine.py.230: loss: 0.17120 [I 2022-10-09 19:50:03] ...optimizers/trijoint.py.151: total_norm: 17.910809 [I 2022-10-09 19:50:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 1510/2304 [I 2022-10-09 19:50:11] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:23 [I 2022-10-09 19:50:11] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00312 [I 2022-10-09 19:50:11] ...trap/engines/engine.py.230: loss: 0.13948 [I 2022-10-09 19:50:11] ...optimizers/trijoint.py.151: total_norm: 50.692871 [I 2022-10-09 19:50:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 1520/2304 [I 2022-10-09 19:50:20] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:08 [I 2022-10-09 19:50:20] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00316 [I 2022-10-09 19:50:20] ...trap/engines/engine.py.230: loss: 0.11718 [I 2022-10-09 19:50:20] ...optimizers/trijoint.py.151: total_norm: 9.550692 [I 2022-10-09 19:50:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1530/2304 [I 2022-10-09 19:50:29] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:03 [I 2022-10-09 19:50:29] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00314 [I 2022-10-09 19:50:29] ...trap/engines/engine.py.230: loss: 0.16529 [I 2022-10-09 19:50:29] ...optimizers/trijoint.py.151: total_norm: 16.062544 [I 2022-10-09 19:50:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 1540/2304 [I 2022-10-09 19:50:37] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:44 [I 2022-10-09 19:50:37] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00312 [I 2022-10-09 19:50:37] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-09 19:50:37] ...optimizers/trijoint.py.151: total_norm: 11.650464 [I 2022-10-09 19:50:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1550/2304 [I 2022-10-09 19:50:46] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:35 [I 2022-10-09 19:50:46] ...trap/engines/engine.py.229: process: 0.82485 | load: 0.00305 [I 2022-10-09 19:50:46] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-09 19:50:46] ...optimizers/trijoint.py.151: total_norm: 7.986301 [I 2022-10-09 19:50:54] ...trap/engines/engine.py.224: train: epoch 52 | batch 1560/2304 [I 2022-10-09 19:50:54] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:26 [I 2022-10-09 19:50:54] ...trap/engines/engine.py.229: process: 0.82992 | load: 0.00304 [I 2022-10-09 19:50:54] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-09 19:50:54] ...optimizers/trijoint.py.151: total_norm: 13.860460 [I 2022-10-09 19:51:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1570/2304 [I 2022-10-09 19:51:03] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:23 [I 2022-10-09 19:51:03] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00305 [I 2022-10-09 19:51:03] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-09 19:51:03] ...optimizers/trijoint.py.151: total_norm: 13.803823 [I 2022-10-09 19:51:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 1580/2304 [I 2022-10-09 19:51:11] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:10 [I 2022-10-09 19:51:11] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00290 [I 2022-10-09 19:51:11] ...trap/engines/engine.py.230: loss: 0.21091 [I 2022-10-09 19:51:11] ...optimizers/trijoint.py.151: total_norm: 14.787502 [I 2022-10-09 19:51:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 1590/2304 [I 2022-10-09 19:51:19] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:00 [I 2022-10-09 19:51:19] ...trap/engines/engine.py.229: process: 0.83135 | load: 0.00294 [I 2022-10-09 19:51:19] ...trap/engines/engine.py.230: loss: 0.22501 [I 2022-10-09 19:51:19] ...optimizers/trijoint.py.151: total_norm: 30.542368 [I 2022-10-09 19:51:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 1600/2304 [I 2022-10-09 19:51:28] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:09:57 [I 2022-10-09 19:51:28] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00302 [I 2022-10-09 19:51:28] ...trap/engines/engine.py.230: loss: 0.14290 [I 2022-10-09 19:51:28] ...optimizers/trijoint.py.151: total_norm: 8.002302 [I 2022-10-09 19:51:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 1610/2304 [I 2022-10-09 19:51:36] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:46 [I 2022-10-09 19:51:36] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00300 [I 2022-10-09 19:51:36] ...trap/engines/engine.py.230: loss: 0.18831 [I 2022-10-09 19:51:36] ...optimizers/trijoint.py.151: total_norm: 17.754522 [I 2022-10-09 19:51:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 1620/2304 [I 2022-10-09 19:51:45] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:09:37 [I 2022-10-09 19:51:45] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00293 [I 2022-10-09 19:51:45] ...trap/engines/engine.py.230: loss: 0.11585 [I 2022-10-09 19:51:45] ...optimizers/trijoint.py.151: total_norm: 11.278596 [I 2022-10-09 19:51:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 1630/2304 [I 2022-10-09 19:51:53] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:28 [I 2022-10-09 19:51:53] ...trap/engines/engine.py.229: process: 0.84247 | load: 0.00295 [I 2022-10-09 19:51:53] ...trap/engines/engine.py.230: loss: 0.17742 [I 2022-10-09 19:51:53] ...optimizers/trijoint.py.151: total_norm: 13.289779 [I 2022-10-09 19:52:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 1640/2304 [I 2022-10-09 19:52:02] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:21 [I 2022-10-09 19:52:02] ...trap/engines/engine.py.229: process: 0.84429 | load: 0.00296 [I 2022-10-09 19:52:02] ...trap/engines/engine.py.230: loss: 0.17897 [I 2022-10-09 19:52:02] ...optimizers/trijoint.py.151: total_norm: 20.282467 [I 2022-10-09 19:52:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 1650/2304 [I 2022-10-09 19:52:10] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:10 [I 2022-10-09 19:52:10] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00329 [I 2022-10-09 19:52:10] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-09 19:52:10] ...optimizers/trijoint.py.151: total_norm: 9.013045 [I 2022-10-09 19:52:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 1660/2304 [I 2022-10-09 19:52:19] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:05 [I 2022-10-09 19:52:19] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00298 [I 2022-10-09 19:52:19] ...trap/engines/engine.py.230: loss: 0.11185 [I 2022-10-09 19:52:19] ...optimizers/trijoint.py.151: total_norm: 17.249578 [I 2022-10-09 19:52:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 1670/2304 [I 2022-10-09 19:52:27] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:08:57 [I 2022-10-09 19:52:27] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00298 [I 2022-10-09 19:52:27] ...trap/engines/engine.py.230: loss: 0.15636 [I 2022-10-09 19:52:27] ...optimizers/trijoint.py.151: total_norm: 13.283843 [I 2022-10-09 19:52:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 1680/2304 [I 2022-10-09 19:52:36] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:47 [I 2022-10-09 19:52:36] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00303 [I 2022-10-09 19:52:36] ...trap/engines/engine.py.230: loss: 0.25485 [I 2022-10-09 19:52:36] ...optimizers/trijoint.py.151: total_norm: 20.582464 [I 2022-10-09 19:52:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 1690/2304 [I 2022-10-09 19:52:44] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:37 [I 2022-10-09 19:52:44] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00296 [I 2022-10-09 19:52:44] ...trap/engines/engine.py.230: loss: 0.19344 [I 2022-10-09 19:52:44] ...optimizers/trijoint.py.151: total_norm: 25.310787 [I 2022-10-09 19:52:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 1700/2304 [I 2022-10-09 19:52:53] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:33 [I 2022-10-09 19:52:53] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00297 [I 2022-10-09 19:52:53] ...trap/engines/engine.py.230: loss: 0.18197 [I 2022-10-09 19:52:53] ...optimizers/trijoint.py.151: total_norm: 18.816982 [I 2022-10-09 19:53:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 1710/2304 [I 2022-10-09 19:53:01] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:20 [I 2022-10-09 19:53:01] ...trap/engines/engine.py.229: process: 0.83751 | load: 0.00297 [I 2022-10-09 19:53:01] ...trap/engines/engine.py.230: loss: 0.17208 [I 2022-10-09 19:53:01] ...optimizers/trijoint.py.151: total_norm: 15.686689 [I 2022-10-09 19:53:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 1720/2304 [I 2022-10-09 19:53:09] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:10 [I 2022-10-09 19:53:09] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00298 [I 2022-10-09 19:53:09] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-09 19:53:09] ...optimizers/trijoint.py.151: total_norm: 14.346069 [I 2022-10-09 19:53:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 1730/2304 [I 2022-10-09 19:53:18] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:02 [I 2022-10-09 19:53:18] ...trap/engines/engine.py.229: process: 0.82562 | load: 0.00300 [I 2022-10-09 19:53:18] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-09 19:53:18] ...optimizers/trijoint.py.151: total_norm: 9.904569 [I 2022-10-09 19:53:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1740/2304 [I 2022-10-09 19:53:26] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:07:54 [I 2022-10-09 19:53:26] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00297 [I 2022-10-09 19:53:26] ...trap/engines/engine.py.230: loss: 0.09822 [I 2022-10-09 19:53:26] ...optimizers/trijoint.py.151: total_norm: 10.029804 [I 2022-10-09 19:53:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 1750/2304 [I 2022-10-09 19:53:35] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:07:46 [I 2022-10-09 19:53:35] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00299 [I 2022-10-09 19:53:35] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-09 19:53:35] ...optimizers/trijoint.py.151: total_norm: 9.111610 [I 2022-10-09 19:53:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1760/2304 [I 2022-10-09 19:53:43] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:07:42 [I 2022-10-09 19:53:43] ...trap/engines/engine.py.229: process: 0.86703 | load: 0.00294 [I 2022-10-09 19:53:43] ...trap/engines/engine.py.230: loss: 0.21448 [I 2022-10-09 19:53:43] ...optimizers/trijoint.py.151: total_norm: 24.593946 [I 2022-10-09 19:53:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 1770/2304 [I 2022-10-09 19:53:52] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:35 [I 2022-10-09 19:53:52] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00318 [I 2022-10-09 19:53:52] ...trap/engines/engine.py.230: loss: 0.18796 [I 2022-10-09 19:53:52] ...optimizers/trijoint.py.151: total_norm: 12.665464 [I 2022-10-09 19:54:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1780/2304 [I 2022-10-09 19:54:00] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:26 [I 2022-10-09 19:54:00] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00339 [I 2022-10-09 19:54:00] ...trap/engines/engine.py.230: loss: 0.15449 [I 2022-10-09 19:54:00] ...optimizers/trijoint.py.151: total_norm: 13.917094 [I 2022-10-09 19:54:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 1790/2304 [I 2022-10-09 19:54:09] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:17 [I 2022-10-09 19:54:09] ...trap/engines/engine.py.229: process: 0.83751 | load: 0.00302 [I 2022-10-09 19:54:09] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-09 19:54:09] ...optimizers/trijoint.py.151: total_norm: 14.396364 [I 2022-10-09 19:54:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 1800/2304 [I 2022-10-09 19:54:18] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:11 [I 2022-10-09 19:54:18] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00316 [I 2022-10-09 19:54:18] ...trap/engines/engine.py.230: loss: 0.15909 [I 2022-10-09 19:54:18] ...optimizers/trijoint.py.151: total_norm: 14.189666 [I 2022-10-09 19:54:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1810/2304 [I 2022-10-09 19:54:26] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:04 [I 2022-10-09 19:54:26] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00312 [I 2022-10-09 19:54:26] ...trap/engines/engine.py.230: loss: 0.17522 [I 2022-10-09 19:54:26] ...optimizers/trijoint.py.151: total_norm: 17.383892 [I 2022-10-09 19:54:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 1820/2304 [I 2022-10-09 19:54:35] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:06:51 [I 2022-10-09 19:54:35] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00313 [I 2022-10-09 19:54:35] ...trap/engines/engine.py.230: loss: 0.18732 [I 2022-10-09 19:54:35] ...optimizers/trijoint.py.151: total_norm: 12.616389 [I 2022-10-09 19:54:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1830/2304 [I 2022-10-09 19:54:43] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:44 [I 2022-10-09 19:54:43] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00322 [I 2022-10-09 19:54:43] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-09 19:54:43] ...optimizers/trijoint.py.151: total_norm: 14.468474 [I 2022-10-09 19:54:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 1840/2304 [I 2022-10-09 19:54:52] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:36 [I 2022-10-09 19:54:52] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00310 [I 2022-10-09 19:54:52] ...trap/engines/engine.py.230: loss: 0.14449 [I 2022-10-09 19:54:52] ...optimizers/trijoint.py.151: total_norm: 22.842829 [I 2022-10-09 19:55:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1850/2304 [I 2022-10-09 19:55:00] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:27 [I 2022-10-09 19:55:00] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00315 [I 2022-10-09 19:55:00] ...trap/engines/engine.py.230: loss: 0.15195 [I 2022-10-09 19:55:00] ...optimizers/trijoint.py.151: total_norm: 13.661448 [I 2022-10-09 19:55:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 1860/2304 [I 2022-10-09 19:55:09] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:19 [I 2022-10-09 19:55:09] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00314 [I 2022-10-09 19:55:09] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-09 19:55:09] ...optimizers/trijoint.py.151: total_norm: 8.445369 [I 2022-10-09 19:55:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 1870/2304 [I 2022-10-09 19:55:17] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:10 [I 2022-10-09 19:55:17] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00314 [I 2022-10-09 19:55:17] ...trap/engines/engine.py.230: loss: 0.22805 [I 2022-10-09 19:55:17] ...optimizers/trijoint.py.151: total_norm: 21.280184 [I 2022-10-09 19:55:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1880/2304 [I 2022-10-09 19:55:26] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:02 [I 2022-10-09 19:55:26] ...trap/engines/engine.py.229: process: 0.86223 | load: 0.00319 [I 2022-10-09 19:55:26] ...trap/engines/engine.py.230: loss: 0.17259 [I 2022-10-09 19:55:26] ...optimizers/trijoint.py.151: total_norm: 17.236444 [I 2022-10-09 19:55:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 1890/2304 [I 2022-10-09 19:55:35] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:05:53 [I 2022-10-09 19:55:35] ...trap/engines/engine.py.229: process: 0.86550 | load: 0.00308 [I 2022-10-09 19:55:35] ...trap/engines/engine.py.230: loss: 0.25460 [I 2022-10-09 19:55:35] ...optimizers/trijoint.py.151: total_norm: 21.219868 [I 2022-10-09 19:55:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1900/2304 [I 2022-10-09 19:55:43] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:45 [I 2022-10-09 19:55:43] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00317 [I 2022-10-09 19:55:43] ...trap/engines/engine.py.230: loss: 0.15196 [I 2022-10-09 19:55:43] ...optimizers/trijoint.py.151: total_norm: 9.267409 [I 2022-10-09 19:55:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 1910/2304 [I 2022-10-09 19:55:52] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:36 [I 2022-10-09 19:55:52] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00315 [I 2022-10-09 19:55:52] ...trap/engines/engine.py.230: loss: 0.20215 [I 2022-10-09 19:55:52] ...optimizers/trijoint.py.151: total_norm: 22.003710 [I 2022-10-09 19:56:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1920/2304 [I 2022-10-09 19:56:00] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:27 [I 2022-10-09 19:56:00] ...trap/engines/engine.py.229: process: 0.84870 | load: 0.00313 [I 2022-10-09 19:56:00] ...trap/engines/engine.py.230: loss: 0.23493 [I 2022-10-09 19:56:00] ...optimizers/trijoint.py.151: total_norm: 27.327169 [I 2022-10-09 19:56:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 1930/2304 [I 2022-10-09 19:56:09] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:19 [I 2022-10-09 19:56:09] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00314 [I 2022-10-09 19:56:09] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-09 19:56:09] ...optimizers/trijoint.py.151: total_norm: 27.950523 [I 2022-10-09 19:56:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 1940/2304 [I 2022-10-09 19:56:17] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:11 [I 2022-10-09 19:56:17] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00313 [I 2022-10-09 19:56:17] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-09 19:56:17] ...optimizers/trijoint.py.151: total_norm: 15.355487 [I 2022-10-09 19:56:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1950/2304 [I 2022-10-09 19:56:26] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:01 [I 2022-10-09 19:56:26] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00312 [I 2022-10-09 19:56:26] ...trap/engines/engine.py.230: loss: 0.15517 [I 2022-10-09 19:56:26] ...optimizers/trijoint.py.151: total_norm: 15.347930 [I 2022-10-09 19:56:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 1960/2304 [I 2022-10-09 19:56:34] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:04:51 [I 2022-10-09 19:56:34] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00309 [I 2022-10-09 19:56:34] ...trap/engines/engine.py.230: loss: 0.16123 [I 2022-10-09 19:56:34] ...optimizers/trijoint.py.151: total_norm: 23.485535 [I 2022-10-09 19:56:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1970/2304 [I 2022-10-09 19:56:43] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:43 [I 2022-10-09 19:56:43] ...trap/engines/engine.py.229: process: 0.84032 | load: 0.00314 [I 2022-10-09 19:56:43] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-09 19:56:43] ...optimizers/trijoint.py.151: total_norm: 10.958667 [I 2022-10-09 19:56:51] ...trap/engines/engine.py.224: train: epoch 52 | batch 1980/2304 [I 2022-10-09 19:56:51] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:34 [I 2022-10-09 19:56:51] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00313 [I 2022-10-09 19:56:51] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-09 19:56:51] ...optimizers/trijoint.py.151: total_norm: 13.566545 [I 2022-10-09 19:57:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1990/2304 [I 2022-10-09 19:57:00] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:27 [I 2022-10-09 19:57:00] ...trap/engines/engine.py.229: process: 0.85226 | load: 0.00306 [I 2022-10-09 19:57:00] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-09 19:57:00] ...optimizers/trijoint.py.151: total_norm: 9.160741 [I 2022-10-09 19:57:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 2000/2304 [I 2022-10-09 19:57:08] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:18 [I 2022-10-09 19:57:08] ...trap/engines/engine.py.229: process: 0.85811 | load: 0.00303 [I 2022-10-09 19:57:08] ...trap/engines/engine.py.230: loss: 0.13908 [I 2022-10-09 19:57:08] ...optimizers/trijoint.py.151: total_norm: 15.018535 [I 2022-10-09 19:57:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 2010/2304 [I 2022-10-09 19:57:17] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:09 [I 2022-10-09 19:57:17] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00312 [I 2022-10-09 19:57:17] ...trap/engines/engine.py.230: loss: 0.13715 [I 2022-10-09 19:57:17] ...optimizers/trijoint.py.151: total_norm: 10.991729 [I 2022-10-09 19:57:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 2020/2304 [I 2022-10-09 19:57:25] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:01 [I 2022-10-09 19:57:25] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00316 [I 2022-10-09 19:57:25] ...trap/engines/engine.py.230: loss: 0.15381 [I 2022-10-09 19:57:25] ...optimizers/trijoint.py.151: total_norm: 25.613295 [I 2022-10-09 19:57:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 2030/2304 [I 2022-10-09 19:57:34] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:03:55 [I 2022-10-09 19:57:34] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00317 [I 2022-10-09 19:57:34] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-09 19:57:34] ...optimizers/trijoint.py.151: total_norm: 13.440518 [I 2022-10-09 19:57:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 2040/2304 [I 2022-10-09 19:57:43] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:03:46 [I 2022-10-09 19:57:43] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00312 [I 2022-10-09 19:57:43] ...trap/engines/engine.py.230: loss: 0.15770 [I 2022-10-09 19:57:43] ...optimizers/trijoint.py.151: total_norm: 11.626089 [I 2022-10-09 19:57:51] ...trap/engines/engine.py.224: train: epoch 52 | batch 2050/2304 [I 2022-10-09 19:57:51] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:36 [I 2022-10-09 19:57:51] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00317 [I 2022-10-09 19:57:51] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-09 19:57:51] ...optimizers/trijoint.py.151: total_norm: 17.238461 [I 2022-10-09 19:58:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 2060/2304 [I 2022-10-09 19:58:00] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:27 [I 2022-10-09 19:58:00] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00313 [I 2022-10-09 19:58:00] ...trap/engines/engine.py.230: loss: 0.19782 [I 2022-10-09 19:58:00] ...optimizers/trijoint.py.151: total_norm: 20.504929 [I 2022-10-09 19:58:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 2070/2304 [I 2022-10-09 19:58:08] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:19 [I 2022-10-09 19:58:08] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00315 [I 2022-10-09 19:58:08] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-09 19:58:08] ...optimizers/trijoint.py.151: total_norm: 13.272587 [I 2022-10-09 19:58:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 2080/2304 [I 2022-10-09 19:58:17] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:11 [I 2022-10-09 19:58:17] ...trap/engines/engine.py.229: process: 0.85777 | load: 0.00313 [I 2022-10-09 19:58:17] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-09 19:58:17] ...optimizers/trijoint.py.151: total_norm: 11.630518 [I 2022-10-09 19:58:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 2090/2304 [I 2022-10-09 19:58:25] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:01 [I 2022-10-09 19:58:25] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00315 [I 2022-10-09 19:58:25] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-09 19:58:25] ...optimizers/trijoint.py.151: total_norm: 12.611461 [I 2022-10-09 19:58:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 2100/2304 [I 2022-10-09 19:58:34] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:02:51 [I 2022-10-09 19:58:34] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00310 [I 2022-10-09 19:58:34] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-09 19:58:34] ...optimizers/trijoint.py.151: total_norm: 7.646509 [I 2022-10-09 19:58:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 2110/2304 [I 2022-10-09 19:58:42] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:02:44 [I 2022-10-09 19:58:42] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00296 [I 2022-10-09 19:58:42] ...trap/engines/engine.py.230: loss: 0.15032 [I 2022-10-09 19:58:42] ...optimizers/trijoint.py.151: total_norm: 12.112365 [I 2022-10-09 19:58:51] ...trap/engines/engine.py.224: train: epoch 52 | batch 2120/2304 [I 2022-10-09 19:58:51] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:02:36 [I 2022-10-09 19:58:51] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00294 [I 2022-10-09 19:58:51] ...trap/engines/engine.py.230: loss: 0.14994 [I 2022-10-09 19:58:51] ...optimizers/trijoint.py.151: total_norm: 11.844770 [I 2022-10-09 19:58:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 2130/2304 [I 2022-10-09 19:58:59] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:27 [I 2022-10-09 19:58:59] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00297 [I 2022-10-09 19:58:59] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-09 19:58:59] ...optimizers/trijoint.py.151: total_norm: 20.221560 [I 2022-10-09 19:59:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 2140/2304 [I 2022-10-09 19:59:08] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:18 [I 2022-10-09 19:59:08] ...trap/engines/engine.py.229: process: 0.83998 | load: 0.00298 [I 2022-10-09 19:59:08] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-09 19:59:08] ...optimizers/trijoint.py.151: total_norm: 24.311058 [I 2022-10-09 19:59:16] ...trap/engines/engine.py.224: train: epoch 52 | batch 2150/2304 [I 2022-10-09 19:59:16] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:09 [I 2022-10-09 19:59:16] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00296 [I 2022-10-09 19:59:16] ...trap/engines/engine.py.230: loss: 0.18715 [I 2022-10-09 19:59:16] ...optimizers/trijoint.py.151: total_norm: 21.046471 [I 2022-10-09 19:59:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 2160/2304 [I 2022-10-09 19:59:25] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:01 [I 2022-10-09 19:59:25] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00291 [I 2022-10-09 19:59:25] ...trap/engines/engine.py.230: loss: 0.12907 [I 2022-10-09 19:59:25] ...optimizers/trijoint.py.151: total_norm: 22.190903 [I 2022-10-09 19:59:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 2170/2304 [I 2022-10-09 19:59:33] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:01:53 [I 2022-10-09 19:59:33] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00297 [I 2022-10-09 19:59:33] ...trap/engines/engine.py.230: loss: 0.13267 [I 2022-10-09 19:59:33] ...optimizers/trijoint.py.151: total_norm: 13.539681 [I 2022-10-09 19:59:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 2180/2304 [I 2022-10-09 19:59:41] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:01:44 [I 2022-10-09 19:59:41] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00298 [I 2022-10-09 19:59:41] ...trap/engines/engine.py.230: loss: 0.16708 [I 2022-10-09 19:59:41] ...optimizers/trijoint.py.151: total_norm: 17.833311 [I 2022-10-09 19:59:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 2190/2304 [I 2022-10-09 19:59:50] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:01:36 [I 2022-10-09 19:59:50] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00298 [I 2022-10-09 19:59:50] ...trap/engines/engine.py.230: loss: 0.12053 [I 2022-10-09 19:59:50] ...optimizers/trijoint.py.151: total_norm: 11.367906 [I 2022-10-09 19:59:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 2200/2304 [I 2022-10-09 19:59:58] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:27 [I 2022-10-09 19:59:58] ...trap/engines/engine.py.229: process: 0.83294 | load: 0.00296 [I 2022-10-09 19:59:58] ...trap/engines/engine.py.230: loss: 0.14068 [I 2022-10-09 19:59:58] ...optimizers/trijoint.py.151: total_norm: 11.441888 [I 2022-10-09 20:00:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 2210/2304 [I 2022-10-09 20:00:07] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:19 [I 2022-10-09 20:00:07] ...trap/engines/engine.py.229: process: 0.84180 | load: 0.00295 [I 2022-10-09 20:00:07] ...trap/engines/engine.py.230: loss: 0.16833 [I 2022-10-09 20:00:07] ...optimizers/trijoint.py.151: total_norm: 19.968424 [I 2022-10-09 20:00:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 2220/2304 [I 2022-10-09 20:00:15] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:10 [I 2022-10-09 20:00:15] ...trap/engines/engine.py.229: process: 0.83730 | load: 0.00295 [I 2022-10-09 20:00:15] ...trap/engines/engine.py.230: loss: 0.22423 [I 2022-10-09 20:00:15] ...optimizers/trijoint.py.151: total_norm: 14.496729 [I 2022-10-09 20:00:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 2230/2304 [I 2022-10-09 20:00:24] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:02 [I 2022-10-09 20:00:24] ...trap/engines/engine.py.229: process: 0.83486 | load: 0.00295 [I 2022-10-09 20:00:24] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-09 20:00:24] ...optimizers/trijoint.py.151: total_norm: 9.281578 [I 2022-10-09 20:00:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 2240/2304 [I 2022-10-09 20:00:32] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:00:54 [I 2022-10-09 20:00:32] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00298 [I 2022-10-09 20:00:32] ...trap/engines/engine.py.230: loss: 0.22089 [I 2022-10-09 20:00:32] ...optimizers/trijoint.py.151: total_norm: 17.931164 [I 2022-10-09 20:00:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 2250/2304 [I 2022-10-09 20:00:41] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:00:45 [I 2022-10-09 20:00:41] ...trap/engines/engine.py.229: process: 0.83770 | load: 0.00298 [I 2022-10-09 20:00:41] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-09 20:00:41] ...optimizers/trijoint.py.151: total_norm: 10.457971 [I 2022-10-09 20:00:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 2260/2304 [I 2022-10-09 20:00:49] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:37 [I 2022-10-09 20:00:49] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00300 [I 2022-10-09 20:00:49] ...trap/engines/engine.py.230: loss: 0.12282 [I 2022-10-09 20:00:49] ...optimizers/trijoint.py.151: total_norm: 12.928651 [I 2022-10-09 20:00:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 2270/2304 [I 2022-10-09 20:00:58] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:00:28 [I 2022-10-09 20:00:58] ...trap/engines/engine.py.229: process: 0.85076 | load: 0.00297 [I 2022-10-09 20:00:58] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-09 20:00:58] ...optimizers/trijoint.py.151: total_norm: 11.379910 [I 2022-10-09 20:01:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 2280/2304 [I 2022-10-09 20:01:06] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:20 [I 2022-10-09 20:01:06] ...trap/engines/engine.py.229: process: 0.84157 | load: 0.00295 [I 2022-10-09 20:01:06] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-09 20:01:06] ...optimizers/trijoint.py.151: total_norm: 7.615847 [I 2022-10-09 20:01:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 2290/2304 [I 2022-10-09 20:01:14] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:11 [I 2022-10-09 20:01:14] ...trap/engines/engine.py.229: process: 0.84322 | load: 0.00300 [I 2022-10-09 20:01:14] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-09 20:01:14] ...optimizers/trijoint.py.151: total_norm: 25.820366 [I 2022-10-09 20:01:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 2300/2304 [I 2022-10-09 20:01:23] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:03 [I 2022-10-09 20:01:23] ...trap/engines/engine.py.229: process: 0.81628 | load: 0.00016 [I 2022-10-09 20:01:23] ...trap/engines/engine.py.230: loss: 0.18978 [I 2022-10-09 20:01:23] ...optimizers/trijoint.py.151: total_norm: 16.231602 [I 2022-10-09 20:01:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 2304/2304 [I 2022-10-09 20:01:26] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:00 [I 2022-10-09 20:01:26] ...trap/engines/engine.py.229: process: 0.82549 | load: 0.00017 [I 2022-10-09 20:01:26] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-09 20:01:26] ...optimizers/trijoint.py.151: total_norm: 17.075245 [S 2022-10-09 20:01:27] ...trap/engines/engine.py.129: train_epoch.epoch: 52 [S 2022-10-09 20:01:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.1556216046472692 [I 2022-10-09 20:01:27] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 20:01:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 52 [I 2022-10-09 20:01:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 20:01:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 20:01:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 20:01:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 20:01:32] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 20:01:32] ...trap/engines/engine.py.321: eval: epoch 52 | batch 0/510 [I 2022-10-09 20:01:32] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:31:33 [I 2022-10-09 20:01:32] ...trap/engines/engine.py.326: process: 3.71230 | load: 3.17587 [I 2022-10-09 20:01:36] ...trap/engines/engine.py.321: eval: epoch 52 | batch 10/510 [I 2022-10-09 20:01:36] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:28 [I 2022-10-09 20:01:36] ...trap/engines/engine.py.326: process: 0.42033 | load: 0.00018 [I 2022-10-09 20:01:43] ...trap/engines/engine.py.321: eval: epoch 52 | batch 20/510 [I 2022-10-09 20:01:43] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:05:31 [I 2022-10-09 20:01:43] ...trap/engines/engine.py.326: process: 0.44171 | load: 0.00020 [I 2022-10-09 20:01:50] ...trap/engines/engine.py.321: eval: epoch 52 | batch 30/510 [I 2022-10-09 20:01:50] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:53 [I 2022-10-09 20:01:50] ...trap/engines/engine.py.326: process: 0.41242 | load: 0.00020 [I 2022-10-09 20:01:57] ...trap/engines/engine.py.321: eval: epoch 52 | batch 40/510 [I 2022-10-09 20:01:57] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:05:56 [I 2022-10-09 20:01:57] ...trap/engines/engine.py.326: process: 0.71385 | load: 0.00027 [I 2022-10-09 20:02:03] ...trap/engines/engine.py.321: eval: epoch 52 | batch 50/510 [I 2022-10-09 20:02:03] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:04:33 [I 2022-10-09 20:02:03] ...trap/engines/engine.py.326: process: 0.59264 | load: 0.00022 [I 2022-10-09 20:02:08] ...trap/engines/engine.py.321: eval: epoch 52 | batch 60/510 [I 2022-10-09 20:02:08] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:44 [I 2022-10-09 20:02:08] ...trap/engines/engine.py.326: process: 0.43676 | load: 0.00019 [I 2022-10-09 20:02:13] ...trap/engines/engine.py.321: eval: epoch 52 | batch 70/510 [I 2022-10-09 20:02:13] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:41 [I 2022-10-09 20:02:13] ...trap/engines/engine.py.326: process: 0.41795 | load: 0.00021 [I 2022-10-09 20:02:18] ...trap/engines/engine.py.321: eval: epoch 52 | batch 80/510 [I 2022-10-09 20:02:18] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:20 [I 2022-10-09 20:02:18] ...trap/engines/engine.py.326: process: 0.46866 | load: 0.00020 [I 2022-10-09 20:02:23] ...trap/engines/engine.py.321: eval: epoch 52 | batch 90/510 [I 2022-10-09 20:02:23] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:10 [I 2022-10-09 20:02:23] ...trap/engines/engine.py.326: process: 0.44085 | load: 0.00019 [I 2022-10-09 20:02:27] ...trap/engines/engine.py.321: eval: epoch 52 | batch 100/510 [I 2022-10-09 20:02:27] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:00 [I 2022-10-09 20:02:27] ...trap/engines/engine.py.326: process: 0.31116 | load: 0.00020 [I 2022-10-09 20:02:31] ...trap/engines/engine.py.321: eval: epoch 52 | batch 110/510 [I 2022-10-09 20:02:31] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:13 [I 2022-10-09 20:02:31] ...trap/engines/engine.py.326: process: 0.32149 | load: 0.00021 [I 2022-10-09 20:02:34] ...trap/engines/engine.py.321: eval: epoch 52 | batch 120/510 [I 2022-10-09 20:02:34] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:05 [I 2022-10-09 20:02:34] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00018 [I 2022-10-09 20:02:37] ...trap/engines/engine.py.321: eval: epoch 52 | batch 130/510 [I 2022-10-09 20:02:37] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:02 [I 2022-10-09 20:02:37] ...trap/engines/engine.py.326: process: 0.32060 | load: 0.00034 [I 2022-10-09 20:02:40] ...trap/engines/engine.py.321: eval: epoch 52 | batch 140/510 [I 2022-10-09 20:02:40] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:58 [I 2022-10-09 20:02:40] ...trap/engines/engine.py.326: process: 0.31636 | load: 0.00023 [I 2022-10-09 20:02:43] ...trap/engines/engine.py.321: eval: epoch 52 | batch 150/510 [I 2022-10-09 20:02:43] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:55 [I 2022-10-09 20:02:43] ...trap/engines/engine.py.326: process: 0.32328 | load: 0.00021 [I 2022-10-09 20:02:47] ...trap/engines/engine.py.321: eval: epoch 52 | batch 160/510 [I 2022-10-09 20:02:47] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:51 [I 2022-10-09 20:02:47] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00019 [I 2022-10-09 20:02:50] ...trap/engines/engine.py.321: eval: epoch 52 | batch 170/510 [I 2022-10-09 20:02:50] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:49 [I 2022-10-09 20:02:50] ...trap/engines/engine.py.326: process: 0.33151 | load: 0.00021 [I 2022-10-09 20:02:53] ...trap/engines/engine.py.321: eval: epoch 52 | batch 180/510 [I 2022-10-09 20:02:53] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:45 [I 2022-10-09 20:02:53] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00021 [I 2022-10-09 20:02:56] ...trap/engines/engine.py.321: eval: epoch 52 | batch 190/510 [I 2022-10-09 20:02:56] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:42 [I 2022-10-09 20:02:56] ...trap/engines/engine.py.326: process: 0.32074 | load: 0.00021 [I 2022-10-09 20:02:59] ...trap/engines/engine.py.321: eval: epoch 52 | batch 200/510 [I 2022-10-09 20:02:59] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:39 [I 2022-10-09 20:02:59] ...trap/engines/engine.py.326: process: 0.32465 | load: 0.00021 [I 2022-10-09 20:03:03] ...trap/engines/engine.py.321: eval: epoch 52 | batch 210/510 [I 2022-10-09 20:03:03] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:35 [I 2022-10-09 20:03:03] ...trap/engines/engine.py.326: process: 0.31719 | load: 0.00020 [I 2022-10-09 20:03:06] ...trap/engines/engine.py.321: eval: epoch 52 | batch 220/510 [I 2022-10-09 20:03:06] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:32 [I 2022-10-09 20:03:06] ...trap/engines/engine.py.326: process: 0.32024 | load: 0.00021 [I 2022-10-09 20:03:09] ...trap/engines/engine.py.321: eval: epoch 52 | batch 230/510 [I 2022-10-09 20:03:09] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:30 [I 2022-10-09 20:03:09] ...trap/engines/engine.py.326: process: 0.32254 | load: 0.00021 [I 2022-10-09 20:03:12] ...trap/engines/engine.py.321: eval: epoch 52 | batch 240/510 [I 2022-10-09 20:03:12] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:26 [I 2022-10-09 20:03:12] ...trap/engines/engine.py.326: process: 0.31704 | load: 0.00021 [I 2022-10-09 20:03:15] ...trap/engines/engine.py.321: eval: epoch 52 | batch 250/510 [I 2022-10-09 20:03:15] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:23 [I 2022-10-09 20:03:15] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00020 [I 2022-10-09 20:03:19] ...trap/engines/engine.py.321: eval: epoch 52 | batch 260/510 [I 2022-10-09 20:03:19] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:20 [I 2022-10-09 20:03:19] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00018 [I 2022-10-09 20:03:22] ...trap/engines/engine.py.321: eval: epoch 52 | batch 270/510 [I 2022-10-09 20:03:22] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:15 [I 2022-10-09 20:03:22] ...trap/engines/engine.py.326: process: 0.31210 | load: 0.00021 [I 2022-10-09 20:03:25] ...trap/engines/engine.py.321: eval: epoch 52 | batch 280/510 [I 2022-10-09 20:03:25] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:13 [I 2022-10-09 20:03:25] ...trap/engines/engine.py.326: process: 0.32442 | load: 0.00020 [I 2022-10-09 20:03:28] ...trap/engines/engine.py.321: eval: epoch 52 | batch 290/510 [I 2022-10-09 20:03:28] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:10 [I 2022-10-09 20:03:28] ...trap/engines/engine.py.326: process: 0.32470 | load: 0.00021 [I 2022-10-09 20:03:31] ...trap/engines/engine.py.321: eval: epoch 52 | batch 300/510 [I 2022-10-09 20:03:31] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:07 [I 2022-10-09 20:03:31] ...trap/engines/engine.py.326: process: 0.31923 | load: 0.00021 [I 2022-10-09 20:03:35] ...trap/engines/engine.py.321: eval: epoch 52 | batch 310/510 [I 2022-10-09 20:03:35] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:04 [I 2022-10-09 20:03:35] ...trap/engines/engine.py.326: process: 0.32428 | load: 0.00021 [I 2022-10-09 20:03:38] ...trap/engines/engine.py.321: eval: epoch 52 | batch 320/510 [I 2022-10-09 20:03:38] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:01 [I 2022-10-09 20:03:38] ...trap/engines/engine.py.326: process: 0.31722 | load: 0.00021 [I 2022-10-09 20:03:41] ...trap/engines/engine.py.321: eval: epoch 52 | batch 330/510 [I 2022-10-09 20:03:41] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:57 [I 2022-10-09 20:03:41] ...trap/engines/engine.py.326: process: 0.32027 | load: 0.00020 [I 2022-10-09 20:03:44] ...trap/engines/engine.py.321: eval: epoch 52 | batch 340/510 [I 2022-10-09 20:03:44] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:54 [I 2022-10-09 20:03:44] ...trap/engines/engine.py.326: process: 0.31560 | load: 0.00021 [I 2022-10-09 20:03:47] ...trap/engines/engine.py.321: eval: epoch 52 | batch 350/510 [I 2022-10-09 20:03:47] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:51 [I 2022-10-09 20:03:47] ...trap/engines/engine.py.326: process: 0.31967 | load: 0.00021 [I 2022-10-09 20:03:51] ...trap/engines/engine.py.321: eval: epoch 52 | batch 360/510 [I 2022-10-09 20:03:51] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:47 [I 2022-10-09 20:03:51] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00020 [I 2022-10-09 20:03:54] ...trap/engines/engine.py.321: eval: epoch 52 | batch 370/510 [I 2022-10-09 20:03:54] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-09 20:03:54] ...trap/engines/engine.py.326: process: 0.31549 | load: 0.00021 [I 2022-10-09 20:03:57] ...trap/engines/engine.py.321: eval: epoch 52 | batch 380/510 [I 2022-10-09 20:03:57] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-09 20:03:57] ...trap/engines/engine.py.326: process: 0.32054 | load: 0.00022 [I 2022-10-09 20:04:00] ...trap/engines/engine.py.321: eval: epoch 52 | batch 390/510 [I 2022-10-09 20:04:00] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:38 [I 2022-10-09 20:04:00] ...trap/engines/engine.py.326: process: 0.32105 | load: 0.00021 [I 2022-10-09 20:04:03] ...trap/engines/engine.py.321: eval: epoch 52 | batch 400/510 [I 2022-10-09 20:04:03] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:35 [I 2022-10-09 20:04:03] ...trap/engines/engine.py.326: process: 0.31582 | load: 0.00020 [I 2022-10-09 20:04:07] ...trap/engines/engine.py.321: eval: epoch 52 | batch 410/510 [I 2022-10-09 20:04:07] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-09 20:04:07] ...trap/engines/engine.py.326: process: 0.31866 | load: 0.00022 [I 2022-10-09 20:04:10] ...trap/engines/engine.py.321: eval: epoch 52 | batch 420/510 [I 2022-10-09 20:04:10] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-09 20:04:10] ...trap/engines/engine.py.326: process: 0.32259 | load: 0.00022 [I 2022-10-09 20:04:13] ...trap/engines/engine.py.321: eval: epoch 52 | batch 430/510 [I 2022-10-09 20:04:13] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-09 20:04:13] ...trap/engines/engine.py.326: process: 0.32352 | load: 0.00022 [I 2022-10-09 20:04:16] ...trap/engines/engine.py.321: eval: epoch 52 | batch 440/510 [I 2022-10-09 20:04:16] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:22 [I 2022-10-09 20:04:16] ...trap/engines/engine.py.326: process: 0.31428 | load: 0.00021 [I 2022-10-09 20:04:20] ...trap/engines/engine.py.321: eval: epoch 52 | batch 450/510 [I 2022-10-09 20:04:20] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:19 [I 2022-10-09 20:04:20] ...trap/engines/engine.py.326: process: 0.32371 | load: 0.00021 [I 2022-10-09 20:04:23] ...trap/engines/engine.py.321: eval: epoch 52 | batch 460/510 [I 2022-10-09 20:04:23] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:16 [I 2022-10-09 20:04:23] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00021 [I 2022-10-09 20:04:26] ...trap/engines/engine.py.321: eval: epoch 52 | batch 470/510 [I 2022-10-09 20:04:26] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-09 20:04:26] ...trap/engines/engine.py.326: process: 0.32193 | load: 0.00021 [I 2022-10-09 20:04:29] ...trap/engines/engine.py.321: eval: epoch 52 | batch 480/510 [I 2022-10-09 20:04:29] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-09 20:04:29] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00020 [I 2022-10-09 20:04:32] ...trap/engines/engine.py.321: eval: epoch 52 | batch 490/510 [I 2022-10-09 20:04:32] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-09 20:04:32] ...trap/engines/engine.py.326: process: 0.32492 | load: 0.00020 [I 2022-10-09 20:04:36] ...trap/engines/engine.py.321: eval: epoch 52 | batch 500/510 [I 2022-10-09 20:04:36] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:03 [I 2022-10-09 20:04:36] ...trap/engines/engine.py.326: process: 0.31838 | load: 0.00021 [I 2022-10-09 20:04:39] ...trap/engines/engine.py.321: eval: epoch 52 | batch 510/510 [I 2022-10-09 20:04:39] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:00 [I 2022-10-09 20:04:39] ...trap/engines/engine.py.326: process: 0.31824 | load: 0.00018 [S 2022-10-09 20:04:39] ...trap/engines/engine.py.132: eval_epoch.epoch: 52 [I 2022-10-09 20:04:39] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.720400 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907200 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940500 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.713800 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906100 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937300 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008991 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.004490 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005201 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014469 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005224 [S 2022-10-09 20:05:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004196 [I 2022-10-09 20:05:15] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 20:05:15] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 20:05:15] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 20:05:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 20:05:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 20:05:20] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 20:05:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 20:05:26] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 20:05:26] ...trap/engines/engine.py.163: Training model on trainset for epoch 53 [I 2022-10-09 20:05:26] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 20:05:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 0/2304 [I 2022-10-09 20:05:29] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:53:41 [I 2022-10-09 20:05:29] ...trap/engines/engine.py.229: process: 2.96072 | load: 2.06264 [I 2022-10-09 20:05:29] ...trap/engines/engine.py.230: loss: 0.11173 [I 2022-10-09 20:05:29] ...optimizers/trijoint.py.151: total_norm: 5.365979 [I 2022-10-09 20:05:37] ...trap/engines/engine.py.224: train: epoch 53 | batch 10/2304 [I 2022-10-09 20:05:37] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:00 [I 2022-10-09 20:05:37] ...trap/engines/engine.py.229: process: 0.83730 | load: 0.00715 [I 2022-10-09 20:05:37] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-09 20:05:37] ...optimizers/trijoint.py.151: total_norm: 7.314027 [I 2022-10-09 20:05:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 20/2304 [I 2022-10-09 20:05:46] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:16 [I 2022-10-09 20:05:46] ...trap/engines/engine.py.229: process: 0.85095 | load: 0.00384 [I 2022-10-09 20:05:46] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-09 20:05:46] ...optimizers/trijoint.py.151: total_norm: 9.202106 [I 2022-10-09 20:05:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 30/2304 [I 2022-10-09 20:05:54] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:24 [I 2022-10-09 20:05:54] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00468 [I 2022-10-09 20:05:54] ...trap/engines/engine.py.230: loss: 0.13476 [I 2022-10-09 20:05:54] ...optimizers/trijoint.py.151: total_norm: 15.850408 [I 2022-10-09 20:06:03] ...trap/engines/engine.py.224: train: epoch 53 | batch 40/2304 [I 2022-10-09 20:06:03] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:08 [I 2022-10-09 20:06:03] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00375 [I 2022-10-09 20:06:03] ...trap/engines/engine.py.230: loss: 0.21120 [I 2022-10-09 20:06:03] ...optimizers/trijoint.py.151: total_norm: 14.107574 [I 2022-10-09 20:06:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 50/2304 [I 2022-10-09 20:06:11] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:59 [I 2022-10-09 20:06:11] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00379 [I 2022-10-09 20:06:11] ...trap/engines/engine.py.230: loss: 0.13860 [I 2022-10-09 20:06:11] ...optimizers/trijoint.py.151: total_norm: 7.877437 [I 2022-10-09 20:06:20] ...trap/engines/engine.py.224: train: epoch 53 | batch 60/2304 [I 2022-10-09 20:06:20] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:57 [I 2022-10-09 20:06:20] ...trap/engines/engine.py.229: process: 0.86520 | load: 0.00349 [I 2022-10-09 20:06:20] ...trap/engines/engine.py.230: loss: 0.11395 [I 2022-10-09 20:06:20] ...optimizers/trijoint.py.151: total_norm: 12.487227 [I 2022-10-09 20:06:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 70/2304 [I 2022-10-09 20:06:28] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:39 [I 2022-10-09 20:06:28] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00358 [I 2022-10-09 20:06:28] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-09 20:06:28] ...optimizers/trijoint.py.151: total_norm: 20.984131 [I 2022-10-09 20:06:37] ...trap/engines/engine.py.224: train: epoch 53 | batch 80/2304 [I 2022-10-09 20:06:37] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:21 [I 2022-10-09 20:06:37] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00330 [I 2022-10-09 20:06:37] ...trap/engines/engine.py.230: loss: 0.12435 [I 2022-10-09 20:06:37] ...optimizers/trijoint.py.151: total_norm: 8.476567 [I 2022-10-09 20:06:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 90/2304 [I 2022-10-09 20:06:45] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:16 [I 2022-10-09 20:06:45] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00317 [I 2022-10-09 20:06:45] ...trap/engines/engine.py.230: loss: 0.16845 [I 2022-10-09 20:06:45] ...optimizers/trijoint.py.151: total_norm: 24.113459 [I 2022-10-09 20:06:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 100/2304 [I 2022-10-09 20:06:54] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:12 [I 2022-10-09 20:06:54] ...trap/engines/engine.py.229: process: 0.84811 | load: 0.00324 [I 2022-10-09 20:06:54] ...trap/engines/engine.py.230: loss: 0.21284 [I 2022-10-09 20:06:54] ...optimizers/trijoint.py.151: total_norm: 18.409264 [I 2022-10-09 20:07:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 110/2304 [I 2022-10-09 20:07:02] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:47 [I 2022-10-09 20:07:02] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00382 [I 2022-10-09 20:07:02] ...trap/engines/engine.py.230: loss: 0.10693 [I 2022-10-09 20:07:02] ...optimizers/trijoint.py.151: total_norm: 14.440616 [I 2022-10-09 20:07:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 120/2304 [I 2022-10-09 20:07:11] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:43 [I 2022-10-09 20:07:11] ...trap/engines/engine.py.229: process: 0.84184 | load: 0.00344 [I 2022-10-09 20:07:11] ...trap/engines/engine.py.230: loss: 0.15785 [I 2022-10-09 20:07:11] ...optimizers/trijoint.py.151: total_norm: 26.638283 [I 2022-10-09 20:07:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 130/2304 [I 2022-10-09 20:07:19] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:30 [I 2022-10-09 20:07:19] ...trap/engines/engine.py.229: process: 0.84574 | load: 0.00338 [I 2022-10-09 20:07:19] ...trap/engines/engine.py.230: loss: 0.17173 [I 2022-10-09 20:07:19] ...optimizers/trijoint.py.151: total_norm: 22.702583 [I 2022-10-09 20:07:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 140/2304 [I 2022-10-09 20:07:28] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:25 [I 2022-10-09 20:07:28] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00318 [I 2022-10-09 20:07:28] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-09 20:07:28] ...optimizers/trijoint.py.151: total_norm: 79.819084 [I 2022-10-09 20:07:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 150/2304 [I 2022-10-09 20:07:36] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:15 [I 2022-10-09 20:07:36] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00297 [I 2022-10-09 20:07:36] ...trap/engines/engine.py.230: loss: 0.15819 [I 2022-10-09 20:07:36] ...optimizers/trijoint.py.151: total_norm: 16.322556 [I 2022-10-09 20:07:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 160/2304 [I 2022-10-09 20:07:45] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:12 [I 2022-10-09 20:07:45] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00321 [I 2022-10-09 20:07:45] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-09 20:07:45] ...optimizers/trijoint.py.151: total_norm: 7.681848 [I 2022-10-09 20:07:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 170/2304 [I 2022-10-09 20:07:53] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:06 [I 2022-10-09 20:07:53] ...trap/engines/engine.py.229: process: 0.83595 | load: 0.00295 [I 2022-10-09 20:07:53] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-09 20:07:53] ...optimizers/trijoint.py.151: total_norm: 7.741709 [I 2022-10-09 20:08:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 180/2304 [I 2022-10-09 20:08:02] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:56 [I 2022-10-09 20:08:02] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00316 [I 2022-10-09 20:08:02] ...trap/engines/engine.py.230: loss: 0.11735 [I 2022-10-09 20:08:02] ...optimizers/trijoint.py.151: total_norm: 8.946019 [I 2022-10-09 20:08:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 190/2304 [I 2022-10-09 20:08:10] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:57 [I 2022-10-09 20:08:10] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00294 [I 2022-10-09 20:08:10] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-09 20:08:10] ...optimizers/trijoint.py.151: total_norm: 10.503039 [I 2022-10-09 20:08:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 200/2304 [I 2022-10-09 20:08:19] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:36 [I 2022-10-09 20:08:19] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00337 [I 2022-10-09 20:08:19] ...trap/engines/engine.py.230: loss: 0.16785 [I 2022-10-09 20:08:19] ...optimizers/trijoint.py.151: total_norm: 16.506996 [I 2022-10-09 20:08:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 210/2304 [I 2022-10-09 20:08:27] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:35 [I 2022-10-09 20:08:27] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00297 [I 2022-10-09 20:08:27] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-09 20:08:27] ...optimizers/trijoint.py.151: total_norm: 9.469312 [I 2022-10-09 20:08:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 220/2304 [I 2022-10-09 20:08:36] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:18 [I 2022-10-09 20:08:36] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00319 [I 2022-10-09 20:08:36] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-09 20:08:36] ...optimizers/trijoint.py.151: total_norm: 20.823452 [I 2022-10-09 20:08:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 230/2304 [I 2022-10-09 20:08:44] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:04 [I 2022-10-09 20:08:44] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00322 [I 2022-10-09 20:08:44] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-09 20:08:44] ...optimizers/trijoint.py.151: total_norm: 18.027184 [I 2022-10-09 20:08:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 240/2304 [I 2022-10-09 20:08:52] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:09 [I 2022-10-09 20:08:52] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00292 [I 2022-10-09 20:08:52] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-09 20:08:52] ...optimizers/trijoint.py.151: total_norm: 11.083788 [I 2022-10-09 20:09:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 250/2304 [I 2022-10-09 20:09:01] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:28:38 [I 2022-10-09 20:09:01] ...trap/engines/engine.py.229: process: 0.82852 | load: 0.00295 [I 2022-10-09 20:09:01] ...trap/engines/engine.py.230: loss: 0.10817 [I 2022-10-09 20:09:01] ...optimizers/trijoint.py.151: total_norm: 7.738636 [I 2022-10-09 20:09:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 260/2304 [I 2022-10-09 20:09:09] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:51 [I 2022-10-09 20:09:09] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00297 [I 2022-10-09 20:09:09] ...trap/engines/engine.py.230: loss: 0.12148 [I 2022-10-09 20:09:09] ...optimizers/trijoint.py.151: total_norm: 12.068739 [I 2022-10-09 20:09:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 270/2304 [I 2022-10-09 20:09:18] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:35 [I 2022-10-09 20:09:18] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00312 [I 2022-10-09 20:09:18] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-09 20:09:18] ...optimizers/trijoint.py.151: total_norm: 12.533044 [I 2022-10-09 20:09:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 280/2304 [I 2022-10-09 20:09:26] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:24 [I 2022-10-09 20:09:26] ...trap/engines/engine.py.229: process: 0.84019 | load: 0.00296 [I 2022-10-09 20:09:26] ...trap/engines/engine.py.230: loss: 0.10914 [I 2022-10-09 20:09:26] ...optimizers/trijoint.py.151: total_norm: 7.080393 [I 2022-10-09 20:09:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 290/2304 [I 2022-10-09 20:09:35] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:28 [I 2022-10-09 20:09:35] ...trap/engines/engine.py.229: process: 0.83385 | load: 0.00297 [I 2022-10-09 20:09:35] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-09 20:09:35] ...optimizers/trijoint.py.151: total_norm: 23.798580 [I 2022-10-09 20:09:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 300/2304 [I 2022-10-09 20:09:43] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:37 [I 2022-10-09 20:09:43] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00310 [I 2022-10-09 20:09:43] ...trap/engines/engine.py.230: loss: 0.19384 [I 2022-10-09 20:09:43] ...optimizers/trijoint.py.151: total_norm: 16.495203 [I 2022-10-09 20:09:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 310/2304 [I 2022-10-09 20:09:52] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:28 [I 2022-10-09 20:09:52] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00328 [I 2022-10-09 20:09:52] ...trap/engines/engine.py.230: loss: 0.14608 [I 2022-10-09 20:09:52] ...optimizers/trijoint.py.151: total_norm: 17.468189 [I 2022-10-09 20:10:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 320/2304 [I 2022-10-09 20:10:00] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:28:05 [I 2022-10-09 20:10:00] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00312 [I 2022-10-09 20:10:00] ...trap/engines/engine.py.230: loss: 0.11098 [I 2022-10-09 20:10:00] ...optimizers/trijoint.py.151: total_norm: 10.940367 [I 2022-10-09 20:10:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 330/2304 [I 2022-10-09 20:10:09] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:11 [I 2022-10-09 20:10:09] ...trap/engines/engine.py.229: process: 0.86370 | load: 0.00314 [I 2022-10-09 20:10:09] ...trap/engines/engine.py.230: loss: 0.13704 [I 2022-10-09 20:10:09] ...optimizers/trijoint.py.151: total_norm: 15.106056 [I 2022-10-09 20:10:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 340/2304 [I 2022-10-09 20:10:18] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:27:59 [I 2022-10-09 20:10:18] ...trap/engines/engine.py.229: process: 0.86955 | load: 0.00311 [I 2022-10-09 20:10:18] ...trap/engines/engine.py.230: loss: 0.17545 [I 2022-10-09 20:10:18] ...optimizers/trijoint.py.151: total_norm: 11.860184 [I 2022-10-09 20:10:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 350/2304 [I 2022-10-09 20:10:26] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:49 [I 2022-10-09 20:10:26] ...trap/engines/engine.py.229: process: 0.86374 | load: 0.00312 [I 2022-10-09 20:10:26] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-09 20:10:26] ...optimizers/trijoint.py.151: total_norm: 23.581461 [I 2022-10-09 20:10:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 360/2304 [I 2022-10-09 20:10:35] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:27:36 [I 2022-10-09 20:10:35] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00322 [I 2022-10-09 20:10:35] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-09 20:10:35] ...optimizers/trijoint.py.151: total_norm: 27.046391 [I 2022-10-09 20:10:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 370/2304 [I 2022-10-09 20:10:43] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:32 [I 2022-10-09 20:10:43] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00310 [I 2022-10-09 20:10:43] ...trap/engines/engine.py.230: loss: 0.17126 [I 2022-10-09 20:10:43] ...optimizers/trijoint.py.151: total_norm: 19.457838 [I 2022-10-09 20:10:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 380/2304 [I 2022-10-09 20:10:52] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:33 [I 2022-10-09 20:10:52] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00312 [I 2022-10-09 20:10:52] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-09 20:10:52] ...optimizers/trijoint.py.151: total_norm: 8.706698 [I 2022-10-09 20:11:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 390/2304 [I 2022-10-09 20:11:00] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:11 [I 2022-10-09 20:11:00] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00325 [I 2022-10-09 20:11:00] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-09 20:11:00] ...optimizers/trijoint.py.151: total_norm: 23.125435 [I 2022-10-09 20:11:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 400/2304 [I 2022-10-09 20:11:09] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:55 [I 2022-10-09 20:11:09] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00312 [I 2022-10-09 20:11:09] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-09 20:11:09] ...optimizers/trijoint.py.151: total_norm: 15.794541 [I 2022-10-09 20:11:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 410/2304 [I 2022-10-09 20:11:17] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:52 [I 2022-10-09 20:11:17] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00313 [I 2022-10-09 20:11:17] ...trap/engines/engine.py.230: loss: 0.19180 [I 2022-10-09 20:11:17] ...optimizers/trijoint.py.151: total_norm: 23.365475 [I 2022-10-09 20:11:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 420/2304 [I 2022-10-09 20:11:26] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:45 [I 2022-10-09 20:11:26] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00314 [I 2022-10-09 20:11:26] ...trap/engines/engine.py.230: loss: 0.14867 [I 2022-10-09 20:11:26] ...optimizers/trijoint.py.151: total_norm: 9.728723 [I 2022-10-09 20:11:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 430/2304 [I 2022-10-09 20:11:34] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:40 [I 2022-10-09 20:11:34] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00315 [I 2022-10-09 20:11:34] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-09 20:11:34] ...optimizers/trijoint.py.151: total_norm: 13.526160 [I 2022-10-09 20:11:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 440/2304 [I 2022-10-09 20:11:43] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:41 [I 2022-10-09 20:11:43] ...trap/engines/engine.py.229: process: 0.86425 | load: 0.00321 [I 2022-10-09 20:11:43] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-09 20:11:43] ...optimizers/trijoint.py.151: total_norm: 27.521730 [I 2022-10-09 20:11:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 450/2304 [I 2022-10-09 20:11:52] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:22 [I 2022-10-09 20:11:52] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00313 [I 2022-10-09 20:11:52] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-09 20:11:52] ...optimizers/trijoint.py.151: total_norm: 12.464025 [I 2022-10-09 20:12:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 460/2304 [I 2022-10-09 20:12:00] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:31 [I 2022-10-09 20:12:00] ...trap/engines/engine.py.229: process: 0.86597 | load: 0.00313 [I 2022-10-09 20:12:00] ...trap/engines/engine.py.230: loss: 0.15316 [I 2022-10-09 20:12:00] ...optimizers/trijoint.py.151: total_norm: 14.839073 [I 2022-10-09 20:12:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 470/2304 [I 2022-10-09 20:12:09] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:17 [I 2022-10-09 20:12:09] ...trap/engines/engine.py.229: process: 0.86719 | load: 0.00312 [I 2022-10-09 20:12:09] ...trap/engines/engine.py.230: loss: 0.15592 [I 2022-10-09 20:12:09] ...optimizers/trijoint.py.151: total_norm: 14.803388 [I 2022-10-09 20:12:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 480/2304 [I 2022-10-09 20:12:17] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:26:06 [I 2022-10-09 20:12:17] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00324 [I 2022-10-09 20:12:17] ...trap/engines/engine.py.230: loss: 0.13677 [I 2022-10-09 20:12:17] ...optimizers/trijoint.py.151: total_norm: 15.380118 [I 2022-10-09 20:12:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 490/2304 [I 2022-10-09 20:12:26] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:48 [I 2022-10-09 20:12:26] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00320 [I 2022-10-09 20:12:26] ...trap/engines/engine.py.230: loss: 0.16798 [I 2022-10-09 20:12:26] ...optimizers/trijoint.py.151: total_norm: 7.406855 [I 2022-10-09 20:12:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 500/2304 [I 2022-10-09 20:12:35] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:39 [I 2022-10-09 20:12:35] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00313 [I 2022-10-09 20:12:35] ...trap/engines/engine.py.230: loss: 0.09675 [I 2022-10-09 20:12:35] ...optimizers/trijoint.py.151: total_norm: 6.346994 [I 2022-10-09 20:12:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 510/2304 [I 2022-10-09 20:12:43] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:42 [I 2022-10-09 20:12:43] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00309 [I 2022-10-09 20:12:43] ...trap/engines/engine.py.230: loss: 0.12784 [I 2022-10-09 20:12:43] ...optimizers/trijoint.py.151: total_norm: 18.832485 [I 2022-10-09 20:12:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 520/2304 [I 2022-10-09 20:12:52] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:28 [I 2022-10-09 20:12:52] ...trap/engines/engine.py.229: process: 0.86438 | load: 0.00301 [I 2022-10-09 20:12:52] ...trap/engines/engine.py.230: loss: 0.18417 [I 2022-10-09 20:12:52] ...optimizers/trijoint.py.151: total_norm: 46.068951 [I 2022-10-09 20:13:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 530/2304 [I 2022-10-09 20:13:00] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:04 [I 2022-10-09 20:13:00] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00316 [I 2022-10-09 20:13:00] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-09 20:13:00] ...optimizers/trijoint.py.151: total_norm: 12.043947 [I 2022-10-09 20:13:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 540/2304 [I 2022-10-09 20:13:09] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:59 [I 2022-10-09 20:13:09] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00311 [I 2022-10-09 20:13:09] ...trap/engines/engine.py.230: loss: 0.14589 [I 2022-10-09 20:13:09] ...optimizers/trijoint.py.151: total_norm: 9.163072 [I 2022-10-09 20:13:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 550/2304 [I 2022-10-09 20:13:17] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:01 [I 2022-10-09 20:13:17] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00320 [I 2022-10-09 20:13:17] ...trap/engines/engine.py.230: loss: 0.17360 [I 2022-10-09 20:13:17] ...optimizers/trijoint.py.151: total_norm: 18.927235 [I 2022-10-09 20:13:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 560/2304 [I 2022-10-09 20:13:26] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:43 [I 2022-10-09 20:13:26] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00332 [I 2022-10-09 20:13:26] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-09 20:13:26] ...optimizers/trijoint.py.151: total_norm: 22.709227 [I 2022-10-09 20:13:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 570/2304 [I 2022-10-09 20:13:34] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:43 [I 2022-10-09 20:13:34] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00312 [I 2022-10-09 20:13:34] ...trap/engines/engine.py.230: loss: 0.15351 [I 2022-10-09 20:13:34] ...optimizers/trijoint.py.151: total_norm: 11.672653 [I 2022-10-09 20:13:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 580/2304 [I 2022-10-09 20:13:43] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:23 [I 2022-10-09 20:13:43] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00314 [I 2022-10-09 20:13:43] ...trap/engines/engine.py.230: loss: 0.11185 [I 2022-10-09 20:13:43] ...optimizers/trijoint.py.151: total_norm: 20.291737 [I 2022-10-09 20:13:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 590/2304 [I 2022-10-09 20:13:52] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:28 [I 2022-10-09 20:13:52] ...trap/engines/engine.py.229: process: 0.85756 | load: 0.00314 [I 2022-10-09 20:13:52] ...trap/engines/engine.py.230: loss: 0.15984 [I 2022-10-09 20:13:52] ...optimizers/trijoint.py.151: total_norm: 11.567428 [I 2022-10-09 20:14:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 600/2304 [I 2022-10-09 20:14:00] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:16 [I 2022-10-09 20:14:00] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00314 [I 2022-10-09 20:14:00] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-09 20:14:00] ...optimizers/trijoint.py.151: total_norm: 13.242247 [I 2022-10-09 20:14:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 610/2304 [I 2022-10-09 20:14:09] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:04 [I 2022-10-09 20:14:09] ...trap/engines/engine.py.229: process: 0.85474 | load: 0.00310 [I 2022-10-09 20:14:09] ...trap/engines/engine.py.230: loss: 0.13546 [I 2022-10-09 20:14:09] ...optimizers/trijoint.py.151: total_norm: 17.800282 [I 2022-10-09 20:14:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 620/2304 [I 2022-10-09 20:14:17] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:04 [I 2022-10-09 20:14:17] ...trap/engines/engine.py.229: process: 0.86317 | load: 0.00317 [I 2022-10-09 20:14:17] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-09 20:14:17] ...optimizers/trijoint.py.151: total_norm: 14.978255 [I 2022-10-09 20:14:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 630/2304 [I 2022-10-09 20:14:26] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:45 [I 2022-10-09 20:14:26] ...trap/engines/engine.py.229: process: 0.83270 | load: 0.00309 [I 2022-10-09 20:14:26] ...trap/engines/engine.py.230: loss: 0.10862 [I 2022-10-09 20:14:26] ...optimizers/trijoint.py.151: total_norm: 10.257670 [I 2022-10-09 20:14:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 640/2304 [I 2022-10-09 20:14:34] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:32 [I 2022-10-09 20:14:34] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00298 [I 2022-10-09 20:14:34] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-09 20:14:34] ...optimizers/trijoint.py.151: total_norm: 7.480540 [I 2022-10-09 20:14:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 650/2304 [I 2022-10-09 20:14:43] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:22 [I 2022-10-09 20:14:43] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00300 [I 2022-10-09 20:14:43] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-09 20:14:43] ...optimizers/trijoint.py.151: total_norm: 14.212011 [I 2022-10-09 20:14:51] ...trap/engines/engine.py.224: train: epoch 53 | batch 660/2304 [I 2022-10-09 20:14:51] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:05 [I 2022-10-09 20:14:51] ...trap/engines/engine.py.229: process: 0.83413 | load: 0.00293 [I 2022-10-09 20:14:51] ...trap/engines/engine.py.230: loss: 0.12018 [I 2022-10-09 20:14:51] ...optimizers/trijoint.py.151: total_norm: 16.198143 [I 2022-10-09 20:15:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 670/2304 [I 2022-10-09 20:15:00] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:04 [I 2022-10-09 20:15:00] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00298 [I 2022-10-09 20:15:00] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-09 20:15:00] ...optimizers/trijoint.py.151: total_norm: 14.932532 [I 2022-10-09 20:15:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 680/2304 [I 2022-10-09 20:15:08] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:44 [I 2022-10-09 20:15:08] ...trap/engines/engine.py.229: process: 0.84138 | load: 0.00291 [I 2022-10-09 20:15:08] ...trap/engines/engine.py.230: loss: 0.10830 [I 2022-10-09 20:15:08] ...optimizers/trijoint.py.151: total_norm: 16.916180 [I 2022-10-09 20:15:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 690/2304 [I 2022-10-09 20:15:17] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:44 [I 2022-10-09 20:15:17] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00299 [I 2022-10-09 20:15:17] ...trap/engines/engine.py.230: loss: 0.10320 [I 2022-10-09 20:15:17] ...optimizers/trijoint.py.151: total_norm: 8.767653 [I 2022-10-09 20:15:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 700/2304 [I 2022-10-09 20:15:25] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:34 [I 2022-10-09 20:15:25] ...trap/engines/engine.py.229: process: 0.85257 | load: 0.00296 [I 2022-10-09 20:15:25] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-09 20:15:25] ...optimizers/trijoint.py.151: total_norm: 22.787374 [I 2022-10-09 20:15:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 710/2304 [I 2022-10-09 20:15:34] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:24 [I 2022-10-09 20:15:34] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00297 [I 2022-10-09 20:15:34] ...trap/engines/engine.py.230: loss: 0.22529 [I 2022-10-09 20:15:34] ...optimizers/trijoint.py.151: total_norm: 23.841553 [I 2022-10-09 20:15:42] ...trap/engines/engine.py.224: train: epoch 53 | batch 720/2304 [I 2022-10-09 20:15:42] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:14 [I 2022-10-09 20:15:42] ...trap/engines/engine.py.229: process: 0.84323 | load: 0.00298 [I 2022-10-09 20:15:42] ...trap/engines/engine.py.230: loss: 0.15714 [I 2022-10-09 20:15:42] ...optimizers/trijoint.py.151: total_norm: 33.851749 [I 2022-10-09 20:15:51] ...trap/engines/engine.py.224: train: epoch 53 | batch 730/2304 [I 2022-10-09 20:15:51] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:12 [I 2022-10-09 20:15:51] ...trap/engines/engine.py.229: process: 0.83240 | load: 0.00299 [I 2022-10-09 20:15:51] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-09 20:15:51] ...optimizers/trijoint.py.151: total_norm: 11.136132 [I 2022-10-09 20:15:59] ...trap/engines/engine.py.224: train: epoch 53 | batch 740/2304 [I 2022-10-09 20:15:59] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:04 [I 2022-10-09 20:15:59] ...trap/engines/engine.py.229: process: 0.85374 | load: 0.00301 [I 2022-10-09 20:15:59] ...trap/engines/engine.py.230: loss: 0.11113 [I 2022-10-09 20:15:59] ...optimizers/trijoint.py.151: total_norm: 8.435051 [I 2022-10-09 20:16:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 750/2304 [I 2022-10-09 20:16:07] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:55 [I 2022-10-09 20:16:07] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00299 [I 2022-10-09 20:16:07] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-09 20:16:07] ...optimizers/trijoint.py.151: total_norm: 13.829561 [I 2022-10-09 20:16:16] ...trap/engines/engine.py.224: train: epoch 53 | batch 760/2304 [I 2022-10-09 20:16:16] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:42 [I 2022-10-09 20:16:16] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00297 [I 2022-10-09 20:16:16] ...trap/engines/engine.py.230: loss: 0.14423 [I 2022-10-09 20:16:16] ...optimizers/trijoint.py.151: total_norm: 18.193602 [I 2022-10-09 20:16:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 770/2304 [I 2022-10-09 20:16:24] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:33 [I 2022-10-09 20:16:24] ...trap/engines/engine.py.229: process: 0.82644 | load: 0.00294 [I 2022-10-09 20:16:24] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-09 20:16:24] ...optimizers/trijoint.py.151: total_norm: 7.349656 [I 2022-10-09 20:16:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 780/2304 [I 2022-10-09 20:16:33] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:31 [I 2022-10-09 20:16:33] ...trap/engines/engine.py.229: process: 0.85046 | load: 0.00298 [I 2022-10-09 20:16:33] ...trap/engines/engine.py.230: loss: 0.21444 [I 2022-10-09 20:16:33] ...optimizers/trijoint.py.151: total_norm: 36.845955 [I 2022-10-09 20:16:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 790/2304 [I 2022-10-09 20:16:41] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:10 [I 2022-10-09 20:16:41] ...trap/engines/engine.py.229: process: 0.83636 | load: 0.00295 [I 2022-10-09 20:16:41] ...trap/engines/engine.py.230: loss: 0.14655 [I 2022-10-09 20:16:41] ...optimizers/trijoint.py.151: total_norm: 28.713718 [I 2022-10-09 20:16:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 800/2304 [I 2022-10-09 20:16:50] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:06 [I 2022-10-09 20:16:50] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00300 [I 2022-10-09 20:16:50] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-09 20:16:50] ...optimizers/trijoint.py.151: total_norm: 11.284060 [I 2022-10-09 20:16:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 810/2304 [I 2022-10-09 20:16:58] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:20:57 [I 2022-10-09 20:16:58] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00298 [I 2022-10-09 20:16:58] ...trap/engines/engine.py.230: loss: 0.08881 [I 2022-10-09 20:16:58] ...optimizers/trijoint.py.151: total_norm: 16.438656 [I 2022-10-09 20:17:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 820/2304 [I 2022-10-09 20:17:07] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:20:56 [I 2022-10-09 20:17:07] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00299 [I 2022-10-09 20:17:07] ...trap/engines/engine.py.230: loss: 0.15888 [I 2022-10-09 20:17:07] ...optimizers/trijoint.py.151: total_norm: 15.554027 [I 2022-10-09 20:17:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 830/2304 [I 2022-10-09 20:17:15] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:42 [I 2022-10-09 20:17:15] ...trap/engines/engine.py.229: process: 0.83291 | load: 0.00305 [I 2022-10-09 20:17:15] ...trap/engines/engine.py.230: loss: 0.15433 [I 2022-10-09 20:17:15] ...optimizers/trijoint.py.151: total_norm: 16.834948 [I 2022-10-09 20:17:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 840/2304 [I 2022-10-09 20:17:24] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:20:35 [I 2022-10-09 20:17:24] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00296 [I 2022-10-09 20:17:24] ...trap/engines/engine.py.230: loss: 0.15899 [I 2022-10-09 20:17:24] ...optimizers/trijoint.py.151: total_norm: 25.461695 [I 2022-10-09 20:17:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 850/2304 [I 2022-10-09 20:17:32] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:42 [I 2022-10-09 20:17:32] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00314 [I 2022-10-09 20:17:32] ...trap/engines/engine.py.230: loss: 0.18983 [I 2022-10-09 20:17:32] ...optimizers/trijoint.py.151: total_norm: 16.611111 [I 2022-10-09 20:17:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 860/2304 [I 2022-10-09 20:17:41] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:33 [I 2022-10-09 20:17:41] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00311 [I 2022-10-09 20:17:41] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-09 20:17:41] ...optimizers/trijoint.py.151: total_norm: 13.040913 [I 2022-10-09 20:17:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 870/2304 [I 2022-10-09 20:17:49] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:24 [I 2022-10-09 20:17:49] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00297 [I 2022-10-09 20:17:49] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-09 20:17:49] ...optimizers/trijoint.py.151: total_norm: 14.204530 [I 2022-10-09 20:17:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 880/2304 [I 2022-10-09 20:17:58] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:18 [I 2022-10-09 20:17:58] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00312 [I 2022-10-09 20:17:58] ...trap/engines/engine.py.230: loss: 0.21830 [I 2022-10-09 20:17:58] ...optimizers/trijoint.py.151: total_norm: 23.495752 [I 2022-10-09 20:18:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 890/2304 [I 2022-10-09 20:18:07] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:11 [I 2022-10-09 20:18:07] ...trap/engines/engine.py.229: process: 0.85226 | load: 0.00317 [I 2022-10-09 20:18:07] ...trap/engines/engine.py.230: loss: 0.16432 [I 2022-10-09 20:18:07] ...optimizers/trijoint.py.151: total_norm: 10.871103 [I 2022-10-09 20:18:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 900/2304 [I 2022-10-09 20:18:15] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:53 [I 2022-10-09 20:18:15] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00313 [I 2022-10-09 20:18:15] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-09 20:18:15] ...optimizers/trijoint.py.151: total_norm: 9.433582 [I 2022-10-09 20:18:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 910/2304 [I 2022-10-09 20:18:24] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:52 [I 2022-10-09 20:18:24] ...trap/engines/engine.py.229: process: 0.86336 | load: 0.00312 [I 2022-10-09 20:18:24] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-09 20:18:24] ...optimizers/trijoint.py.151: total_norm: 9.957134 [I 2022-10-09 20:18:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 920/2304 [I 2022-10-09 20:18:32] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:40 [I 2022-10-09 20:18:32] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00311 [I 2022-10-09 20:18:32] ...trap/engines/engine.py.230: loss: 0.15878 [I 2022-10-09 20:18:32] ...optimizers/trijoint.py.151: total_norm: 11.340130 [I 2022-10-09 20:18:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 930/2304 [I 2022-10-09 20:18:41] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:26 [I 2022-10-09 20:18:41] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00299 [I 2022-10-09 20:18:41] ...trap/engines/engine.py.230: loss: 0.13222 [I 2022-10-09 20:18:41] ...optimizers/trijoint.py.151: total_norm: 11.115362 [I 2022-10-09 20:18:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 940/2304 [I 2022-10-09 20:18:49] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:23 [I 2022-10-09 20:18:49] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00315 [I 2022-10-09 20:18:49] ...trap/engines/engine.py.230: loss: 0.16976 [I 2022-10-09 20:18:49] ...optimizers/trijoint.py.151: total_norm: 19.801964 [I 2022-10-09 20:18:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 950/2304 [I 2022-10-09 20:18:58] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:13 [I 2022-10-09 20:18:58] ...trap/engines/engine.py.229: process: 0.84892 | load: 0.00314 [I 2022-10-09 20:18:58] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-09 20:18:58] ...optimizers/trijoint.py.151: total_norm: 14.002921 [I 2022-10-09 20:19:06] ...trap/engines/engine.py.224: train: epoch 53 | batch 960/2304 [I 2022-10-09 20:19:06] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:14 [I 2022-10-09 20:19:06] ...trap/engines/engine.py.229: process: 0.88434 | load: 0.00310 [I 2022-10-09 20:19:06] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-09 20:19:06] ...optimizers/trijoint.py.151: total_norm: 14.127770 [I 2022-10-09 20:19:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 970/2304 [I 2022-10-09 20:19:15] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:59 [I 2022-10-09 20:19:15] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00309 [I 2022-10-09 20:19:15] ...trap/engines/engine.py.230: loss: 0.18577 [I 2022-10-09 20:19:15] ...optimizers/trijoint.py.151: total_norm: 18.404387 [I 2022-10-09 20:19:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 980/2304 [I 2022-10-09 20:19:24] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:55 [I 2022-10-09 20:19:24] ...trap/engines/engine.py.229: process: 0.87720 | load: 0.00300 [I 2022-10-09 20:19:24] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-09 20:19:24] ...optimizers/trijoint.py.151: total_norm: 16.122116 [I 2022-10-09 20:19:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 990/2304 [I 2022-10-09 20:19:32] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:35 [I 2022-10-09 20:19:32] ...trap/engines/engine.py.229: process: 0.84178 | load: 0.00311 [I 2022-10-09 20:19:32] ...trap/engines/engine.py.230: loss: 0.13749 [I 2022-10-09 20:19:32] ...optimizers/trijoint.py.151: total_norm: 18.870970 [I 2022-10-09 20:19:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1000/2304 [I 2022-10-09 20:19:41] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:34 [I 2022-10-09 20:19:41] ...trap/engines/engine.py.229: process: 0.86183 | load: 0.00313 [I 2022-10-09 20:19:41] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-09 20:19:41] ...optimizers/trijoint.py.151: total_norm: 9.664915 [I 2022-10-09 20:19:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 1010/2304 [I 2022-10-09 20:19:49] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:25 [I 2022-10-09 20:19:49] ...trap/engines/engine.py.229: process: 0.85669 | load: 0.00314 [I 2022-10-09 20:19:49] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-09 20:19:49] ...optimizers/trijoint.py.151: total_norm: 11.679910 [I 2022-10-09 20:19:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 1020/2304 [I 2022-10-09 20:19:58] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:15 [I 2022-10-09 20:19:58] ...trap/engines/engine.py.229: process: 0.86202 | load: 0.00313 [I 2022-10-09 20:19:58] ...trap/engines/engine.py.230: loss: 0.17482 [I 2022-10-09 20:19:58] ...optimizers/trijoint.py.151: total_norm: 24.936714 [I 2022-10-09 20:20:06] ...trap/engines/engine.py.224: train: epoch 53 | batch 1030/2304 [I 2022-10-09 20:20:06] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:02 [I 2022-10-09 20:20:06] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00308 [I 2022-10-09 20:20:06] ...trap/engines/engine.py.230: loss: 0.19732 [I 2022-10-09 20:20:06] ...optimizers/trijoint.py.151: total_norm: 15.385788 [I 2022-10-09 20:20:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 1040/2304 [I 2022-10-09 20:20:15] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:18:04 [I 2022-10-09 20:20:15] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00310 [I 2022-10-09 20:20:15] ...trap/engines/engine.py.230: loss: 0.09791 [I 2022-10-09 20:20:15] ...optimizers/trijoint.py.151: total_norm: 11.687303 [I 2022-10-09 20:20:23] ...trap/engines/engine.py.224: train: epoch 53 | batch 1050/2304 [I 2022-10-09 20:20:23] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:46 [I 2022-10-09 20:20:23] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00309 [I 2022-10-09 20:20:23] ...trap/engines/engine.py.230: loss: 0.12261 [I 2022-10-09 20:20:23] ...optimizers/trijoint.py.151: total_norm: 11.811674 [I 2022-10-09 20:20:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 1060/2304 [I 2022-10-09 20:20:32] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:43 [I 2022-10-09 20:20:32] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00310 [I 2022-10-09 20:20:32] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-09 20:20:32] ...optimizers/trijoint.py.151: total_norm: 18.176289 [I 2022-10-09 20:20:40] ...trap/engines/engine.py.224: train: epoch 53 | batch 1070/2304 [I 2022-10-09 20:20:40] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:37 [I 2022-10-09 20:20:40] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00312 [I 2022-10-09 20:20:40] ...trap/engines/engine.py.230: loss: 0.22826 [I 2022-10-09 20:20:40] ...optimizers/trijoint.py.151: total_norm: 32.795887 [I 2022-10-09 20:20:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 1080/2304 [I 2022-10-09 20:20:49] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:21 [I 2022-10-09 20:20:49] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00311 [I 2022-10-09 20:20:49] ...trap/engines/engine.py.230: loss: 0.18893 [I 2022-10-09 20:20:49] ...optimizers/trijoint.py.151: total_norm: 15.546459 [I 2022-10-09 20:21:16] ...trap/engines/engine.py.224: train: epoch 53 | batch 1090/2304 [I 2022-10-09 20:21:16] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 1:33:19 [I 2022-10-09 20:21:16] ...trap/engines/engine.py.229: process: 19.64753 | load: 0.00312 [I 2022-10-09 20:21:16] ...trap/engines/engine.py.230: loss: 0.22217 [I 2022-10-09 20:21:16] ...optimizers/trijoint.py.151: total_norm: 19.250351 [I 2022-10-09 20:21:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 1100/2304 [I 2022-10-09 20:21:25] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:25:13 [I 2022-10-09 20:21:25] ...trap/engines/engine.py.229: process: 0.86557 | load: 0.00313 [I 2022-10-09 20:21:25] ...trap/engines/engine.py.230: loss: 0.15082 [I 2022-10-09 20:21:25] ...optimizers/trijoint.py.151: total_norm: 38.554401 [I 2022-10-09 20:21:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 1110/2304 [I 2022-10-09 20:21:33] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:17:46 [I 2022-10-09 20:21:33] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00312 [I 2022-10-09 20:21:33] ...trap/engines/engine.py.230: loss: 0.15864 [I 2022-10-09 20:21:33] ...optimizers/trijoint.py.151: total_norm: 43.528622 [I 2022-10-09 20:21:42] ...trap/engines/engine.py.224: train: epoch 53 | batch 1120/2304 [I 2022-10-09 20:21:42] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:55 [I 2022-10-09 20:21:42] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00312 [I 2022-10-09 20:21:42] ...trap/engines/engine.py.230: loss: 0.19709 [I 2022-10-09 20:21:42] ...optimizers/trijoint.py.151: total_norm: 40.308712 [I 2022-10-09 20:21:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 1130/2304 [I 2022-10-09 20:21:50] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:43 [I 2022-10-09 20:21:50] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00312 [I 2022-10-09 20:21:50] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-09 20:21:50] ...optimizers/trijoint.py.151: total_norm: 12.782218 [I 2022-10-09 20:21:59] ...trap/engines/engine.py.224: train: epoch 53 | batch 1140/2304 [I 2022-10-09 20:21:59] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:24 [I 2022-10-09 20:21:59] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00309 [I 2022-10-09 20:21:59] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-09 20:21:59] ...optimizers/trijoint.py.151: total_norm: 21.441732 [I 2022-10-09 20:22:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 1150/2304 [I 2022-10-09 20:22:07] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:21 [I 2022-10-09 20:22:07] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00313 [I 2022-10-09 20:22:07] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-09 20:22:07] ...optimizers/trijoint.py.151: total_norm: 9.385153 [I 2022-10-09 20:22:16] ...trap/engines/engine.py.224: train: epoch 53 | batch 1160/2304 [I 2022-10-09 20:22:16] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:08 [I 2022-10-09 20:22:16] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00295 [I 2022-10-09 20:22:16] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-09 20:22:16] ...optimizers/trijoint.py.151: total_norm: 42.283123 [I 2022-10-09 20:22:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 1170/2304 [I 2022-10-09 20:22:24] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:16:01 [I 2022-10-09 20:22:24] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00299 [I 2022-10-09 20:22:24] ...trap/engines/engine.py.230: loss: 0.15905 [I 2022-10-09 20:22:24] ...optimizers/trijoint.py.151: total_norm: 15.468185 [I 2022-10-09 20:22:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 1180/2304 [I 2022-10-09 20:22:33] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:49 [I 2022-10-09 20:22:33] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00322 [I 2022-10-09 20:22:33] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-09 20:22:33] ...optimizers/trijoint.py.151: total_norm: 8.251843 [I 2022-10-09 20:22:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1190/2304 [I 2022-10-09 20:22:41] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:46 [I 2022-10-09 20:22:41] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00300 [I 2022-10-09 20:22:41] ...trap/engines/engine.py.230: loss: 0.12064 [I 2022-10-09 20:22:41] ...optimizers/trijoint.py.151: total_norm: 13.248002 [I 2022-10-09 20:22:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 1200/2304 [I 2022-10-09 20:22:50] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:32 [I 2022-10-09 20:22:50] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00308 [I 2022-10-09 20:22:50] ...trap/engines/engine.py.230: loss: 0.17955 [I 2022-10-09 20:22:50] ...optimizers/trijoint.py.151: total_norm: 61.539261 [I 2022-10-09 20:22:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 1210/2304 [I 2022-10-09 20:22:58] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:15:27 [I 2022-10-09 20:22:58] ...trap/engines/engine.py.229: process: 0.85323 | load: 0.00293 [I 2022-10-09 20:22:58] ...trap/engines/engine.py.230: loss: 0.16324 [I 2022-10-09 20:22:58] ...optimizers/trijoint.py.151: total_norm: 11.699557 [I 2022-10-09 20:23:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 1220/2304 [I 2022-10-09 20:23:07] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:15:12 [I 2022-10-09 20:23:07] ...trap/engines/engine.py.229: process: 0.84211 | load: 0.00292 [I 2022-10-09 20:23:07] ...trap/engines/engine.py.230: loss: 0.19494 [I 2022-10-09 20:23:07] ...optimizers/trijoint.py.151: total_norm: 31.525961 [I 2022-10-09 20:23:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 1230/2304 [I 2022-10-09 20:23:15] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:05 [I 2022-10-09 20:23:15] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00297 [I 2022-10-09 20:23:15] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-09 20:23:15] ...optimizers/trijoint.py.151: total_norm: 13.419170 [I 2022-10-09 20:23:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 1240/2304 [I 2022-10-09 20:23:24] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:54 [I 2022-10-09 20:23:24] ...trap/engines/engine.py.229: process: 0.83985 | load: 0.00294 [I 2022-10-09 20:23:24] ...trap/engines/engine.py.230: loss: 0.19060 [I 2022-10-09 20:23:24] ...optimizers/trijoint.py.151: total_norm: 17.944323 [I 2022-10-09 20:23:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 1250/2304 [I 2022-10-09 20:23:32] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:50 [I 2022-10-09 20:23:32] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00296 [I 2022-10-09 20:23:32] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-09 20:23:32] ...optimizers/trijoint.py.151: total_norm: 15.918723 [I 2022-10-09 20:23:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1260/2304 [I 2022-10-09 20:23:41] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:41 [I 2022-10-09 20:23:41] ...trap/engines/engine.py.229: process: 0.84556 | load: 0.00296 [I 2022-10-09 20:23:41] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-09 20:23:41] ...optimizers/trijoint.py.151: total_norm: 10.071751 [I 2022-10-09 20:23:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 1270/2304 [I 2022-10-09 20:23:49] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:30 [I 2022-10-09 20:23:49] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00293 [I 2022-10-09 20:23:49] ...trap/engines/engine.py.230: loss: 0.13809 [I 2022-10-09 20:23:49] ...optimizers/trijoint.py.151: total_norm: 15.594908 [I 2022-10-09 20:23:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 1280/2304 [I 2022-10-09 20:23:58] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:30 [I 2022-10-09 20:23:58] ...trap/engines/engine.py.229: process: 0.86144 | load: 0.00298 [I 2022-10-09 20:23:58] ...trap/engines/engine.py.230: loss: 0.16212 [I 2022-10-09 20:23:58] ...optimizers/trijoint.py.151: total_norm: 13.970103 [I 2022-10-09 20:24:06] ...trap/engines/engine.py.224: train: epoch 53 | batch 1290/2304 [I 2022-10-09 20:24:06] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:12 [I 2022-10-09 20:24:06] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00299 [I 2022-10-09 20:24:06] ...trap/engines/engine.py.230: loss: 0.15085 [I 2022-10-09 20:24:06] ...optimizers/trijoint.py.151: total_norm: 40.466572 [I 2022-10-09 20:24:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 1300/2304 [I 2022-10-09 20:24:15] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:10 [I 2022-10-09 20:24:15] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00296 [I 2022-10-09 20:24:15] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-09 20:24:15] ...optimizers/trijoint.py.151: total_norm: 17.601437 [I 2022-10-09 20:24:23] ...trap/engines/engine.py.224: train: epoch 53 | batch 1310/2304 [I 2022-10-09 20:24:23] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:59 [I 2022-10-09 20:24:23] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00296 [I 2022-10-09 20:24:23] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-09 20:24:23] ...optimizers/trijoint.py.151: total_norm: 12.476297 [I 2022-10-09 20:24:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 1320/2304 [I 2022-10-09 20:24:31] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:53 [I 2022-10-09 20:24:31] ...trap/engines/engine.py.229: process: 0.83615 | load: 0.00301 [I 2022-10-09 20:24:31] ...trap/engines/engine.py.230: loss: 0.11508 [I 2022-10-09 20:24:31] ...optimizers/trijoint.py.151: total_norm: 12.311423 [I 2022-10-09 20:24:40] ...trap/engines/engine.py.224: train: epoch 53 | batch 1330/2304 [I 2022-10-09 20:24:40] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:13:41 [I 2022-10-09 20:24:40] ...trap/engines/engine.py.229: process: 0.84097 | load: 0.00295 [I 2022-10-09 20:24:40] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-09 20:24:40] ...optimizers/trijoint.py.151: total_norm: 17.982708 [I 2022-10-09 20:24:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 1340/2304 [I 2022-10-09 20:24:48] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:37 [I 2022-10-09 20:24:48] ...trap/engines/engine.py.229: process: 0.86354 | load: 0.00298 [I 2022-10-09 20:24:48] ...trap/engines/engine.py.230: loss: 0.14299 [I 2022-10-09 20:24:48] ...optimizers/trijoint.py.151: total_norm: 18.359194 [I 2022-10-09 20:24:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 1350/2304 [I 2022-10-09 20:24:57] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:22 [I 2022-10-09 20:24:57] ...trap/engines/engine.py.229: process: 0.83547 | load: 0.00299 [I 2022-10-09 20:24:57] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-09 20:24:57] ...optimizers/trijoint.py.151: total_norm: 12.844378 [I 2022-10-09 20:25:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 1360/2304 [I 2022-10-09 20:25:05] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:17 [I 2022-10-09 20:25:05] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00298 [I 2022-10-09 20:25:05] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-09 20:25:05] ...optimizers/trijoint.py.151: total_norm: 16.386789 [I 2022-10-09 20:25:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 1370/2304 [I 2022-10-09 20:25:14] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:09 [I 2022-10-09 20:25:14] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00301 [I 2022-10-09 20:25:14] ...trap/engines/engine.py.230: loss: 0.09864 [I 2022-10-09 20:25:14] ...optimizers/trijoint.py.151: total_norm: 9.751606 [I 2022-10-09 20:25:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1380/2304 [I 2022-10-09 20:25:22] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:02 [I 2022-10-09 20:25:22] ...trap/engines/engine.py.229: process: 0.85774 | load: 0.00297 [I 2022-10-09 20:25:22] ...trap/engines/engine.py.230: loss: 0.22665 [I 2022-10-09 20:25:22] ...optimizers/trijoint.py.151: total_norm: 19.832306 [I 2022-10-09 20:25:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 1390/2304 [I 2022-10-09 20:25:31] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:01 [I 2022-10-09 20:25:31] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00313 [I 2022-10-09 20:25:31] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-09 20:25:31] ...optimizers/trijoint.py.151: total_norm: 39.929958 [I 2022-10-09 20:25:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1400/2304 [I 2022-10-09 20:25:39] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:51 [I 2022-10-09 20:25:39] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00313 [I 2022-10-09 20:25:39] ...trap/engines/engine.py.230: loss: 0.20057 [I 2022-10-09 20:25:39] ...optimizers/trijoint.py.151: total_norm: 12.105174 [I 2022-10-09 20:25:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 1410/2304 [I 2022-10-09 20:25:48] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:44 [I 2022-10-09 20:25:48] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00313 [I 2022-10-09 20:25:48] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-09 20:25:48] ...optimizers/trijoint.py.151: total_norm: 6.792087 [I 2022-10-09 20:25:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 1420/2304 [I 2022-10-09 20:25:57] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:37 [I 2022-10-09 20:25:57] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00312 [I 2022-10-09 20:25:57] ...trap/engines/engine.py.230: loss: 0.12914 [I 2022-10-09 20:25:57] ...optimizers/trijoint.py.151: total_norm: 20.296495 [I 2022-10-09 20:26:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 1430/2304 [I 2022-10-09 20:26:05] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:26 [I 2022-10-09 20:26:05] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00310 [I 2022-10-09 20:26:05] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-09 20:26:05] ...optimizers/trijoint.py.151: total_norm: 11.195847 [I 2022-10-09 20:26:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 1440/2304 [I 2022-10-09 20:26:14] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:12 [I 2022-10-09 20:26:14] ...trap/engines/engine.py.229: process: 0.85259 | load: 0.00317 [I 2022-10-09 20:26:14] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-09 20:26:14] ...optimizers/trijoint.py.151: total_norm: 17.817387 [I 2022-10-09 20:26:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1450/2304 [I 2022-10-09 20:26:22] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:08 [I 2022-10-09 20:26:22] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00308 [I 2022-10-09 20:26:22] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-09 20:26:22] ...optimizers/trijoint.py.151: total_norm: 15.143483 [I 2022-10-09 20:26:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 1460/2304 [I 2022-10-09 20:26:31] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:58 [I 2022-10-09 20:26:31] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00323 [I 2022-10-09 20:26:31] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-09 20:26:31] ...optimizers/trijoint.py.151: total_norm: 19.656263 [I 2022-10-09 20:26:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1470/2304 [I 2022-10-09 20:26:39] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:53 [I 2022-10-09 20:26:39] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00313 [I 2022-10-09 20:26:39] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-09 20:26:39] ...optimizers/trijoint.py.151: total_norm: 15.166150 [I 2022-10-09 20:26:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 1480/2304 [I 2022-10-09 20:26:48] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:41 [I 2022-10-09 20:26:48] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00313 [I 2022-10-09 20:26:48] ...trap/engines/engine.py.230: loss: 0.10155 [I 2022-10-09 20:26:48] ...optimizers/trijoint.py.151: total_norm: 12.265722 [I 2022-10-09 20:26:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1490/2304 [I 2022-10-09 20:26:56] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:32 [I 2022-10-09 20:26:56] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00313 [I 2022-10-09 20:26:56] ...trap/engines/engine.py.230: loss: 0.18242 [I 2022-10-09 20:26:56] ...optimizers/trijoint.py.151: total_norm: 18.891102 [I 2022-10-09 20:27:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 1500/2304 [I 2022-10-09 20:27:05] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:29 [I 2022-10-09 20:27:05] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00314 [I 2022-10-09 20:27:05] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-09 20:27:05] ...optimizers/trijoint.py.151: total_norm: 15.304744 [I 2022-10-09 20:27:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1510/2304 [I 2022-10-09 20:27:13] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:12 [I 2022-10-09 20:27:13] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00307 [I 2022-10-09 20:27:13] ...trap/engines/engine.py.230: loss: 0.10056 [I 2022-10-09 20:27:13] ...optimizers/trijoint.py.151: total_norm: 15.225963 [I 2022-10-09 20:27:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1520/2304 [I 2022-10-09 20:27:22] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:06 [I 2022-10-09 20:27:22] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00297 [I 2022-10-09 20:27:22] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-09 20:27:22] ...optimizers/trijoint.py.151: total_norm: 11.931468 [I 2022-10-09 20:27:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 1530/2304 [I 2022-10-09 20:27:30] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:57 [I 2022-10-09 20:27:30] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00309 [I 2022-10-09 20:27:30] ...trap/engines/engine.py.230: loss: 0.14930 [I 2022-10-09 20:27:30] ...optimizers/trijoint.py.151: total_norm: 11.254869 [I 2022-10-09 20:27:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1540/2304 [I 2022-10-09 20:27:39] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:52 [I 2022-10-09 20:27:39] ...trap/engines/engine.py.229: process: 0.86070 | load: 0.00303 [I 2022-10-09 20:27:39] ...trap/engines/engine.py.230: loss: 0.18175 [I 2022-10-09 20:27:39] ...optimizers/trijoint.py.151: total_norm: 14.878960 [I 2022-10-09 20:27:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 1550/2304 [I 2022-10-09 20:27:47] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:39 [I 2022-10-09 20:27:47] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00313 [I 2022-10-09 20:27:47] ...trap/engines/engine.py.230: loss: 0.15995 [I 2022-10-09 20:27:47] ...optimizers/trijoint.py.151: total_norm: 22.134567 [I 2022-10-09 20:27:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1560/2304 [I 2022-10-09 20:27:56] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:29 [I 2022-10-09 20:27:56] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00317 [I 2022-10-09 20:27:56] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-09 20:27:56] ...optimizers/trijoint.py.151: total_norm: 18.777113 [I 2022-10-09 20:28:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 1570/2304 [I 2022-10-09 20:28:05] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:23 [I 2022-10-09 20:28:05] ...trap/engines/engine.py.229: process: 0.83755 | load: 0.00313 [I 2022-10-09 20:28:05] ...trap/engines/engine.py.230: loss: 0.16364 [I 2022-10-09 20:28:05] ...optimizers/trijoint.py.151: total_norm: 25.775856 [I 2022-10-09 20:28:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1580/2304 [I 2022-10-09 20:28:13] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:17 [I 2022-10-09 20:28:13] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00312 [I 2022-10-09 20:28:13] ...trap/engines/engine.py.230: loss: 0.13976 [I 2022-10-09 20:28:13] ...optimizers/trijoint.py.151: total_norm: 18.210842 [I 2022-10-09 20:28:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1590/2304 [I 2022-10-09 20:28:22] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:09 [I 2022-10-09 20:28:22] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00311 [I 2022-10-09 20:28:22] ...trap/engines/engine.py.230: loss: 0.13101 [I 2022-10-09 20:28:22] ...optimizers/trijoint.py.151: total_norm: 16.064932 [I 2022-10-09 20:28:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 1600/2304 [I 2022-10-09 20:28:30] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:58 [I 2022-10-09 20:28:30] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00315 [I 2022-10-09 20:28:30] ...trap/engines/engine.py.230: loss: 0.15598 [I 2022-10-09 20:28:30] ...optimizers/trijoint.py.151: total_norm: 16.004854 [I 2022-10-09 20:28:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1610/2304 [I 2022-10-09 20:28:39] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:50 [I 2022-10-09 20:28:39] ...trap/engines/engine.py.229: process: 0.84244 | load: 0.00312 [I 2022-10-09 20:28:39] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-09 20:28:39] ...optimizers/trijoint.py.151: total_norm: 10.550131 [I 2022-10-09 20:28:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 1620/2304 [I 2022-10-09 20:28:47] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:44 [I 2022-10-09 20:28:47] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00313 [I 2022-10-09 20:28:47] ...trap/engines/engine.py.230: loss: 0.21440 [I 2022-10-09 20:28:47] ...optimizers/trijoint.py.151: total_norm: 28.158426 [I 2022-10-09 20:28:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1630/2304 [I 2022-10-09 20:28:56] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:36 [I 2022-10-09 20:28:56] ...trap/engines/engine.py.229: process: 0.85781 | load: 0.00313 [I 2022-10-09 20:28:56] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-09 20:28:56] ...optimizers/trijoint.py.151: total_norm: 15.567404 [I 2022-10-09 20:29:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 1640/2304 [I 2022-10-09 20:29:04] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:27 [I 2022-10-09 20:29:04] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00312 [I 2022-10-09 20:29:04] ...trap/engines/engine.py.230: loss: 0.12119 [I 2022-10-09 20:29:04] ...optimizers/trijoint.py.151: total_norm: 12.351869 [I 2022-10-09 20:29:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1650/2304 [I 2022-10-09 20:29:13] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:14 [I 2022-10-09 20:29:13] ...trap/engines/engine.py.229: process: 0.86017 | load: 0.00299 [I 2022-10-09 20:29:13] ...trap/engines/engine.py.230: loss: 0.11133 [I 2022-10-09 20:29:13] ...optimizers/trijoint.py.151: total_norm: 9.803061 [I 2022-10-09 20:29:21] ...trap/engines/engine.py.224: train: epoch 53 | batch 1660/2304 [I 2022-10-09 20:29:21] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:07 [I 2022-10-09 20:29:21] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00313 [I 2022-10-09 20:29:21] ...trap/engines/engine.py.230: loss: 0.14300 [I 2022-10-09 20:29:21] ...optimizers/trijoint.py.151: total_norm: 55.792675 [I 2022-10-09 20:29:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 1670/2304 [I 2022-10-09 20:29:30] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:09:01 [I 2022-10-09 20:29:30] ...trap/engines/engine.py.229: process: 0.85671 | load: 0.00314 [I 2022-10-09 20:29:30] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-09 20:29:30] ...optimizers/trijoint.py.151: total_norm: 34.505630 [I 2022-10-09 20:29:38] ...trap/engines/engine.py.224: train: epoch 53 | batch 1680/2304 [I 2022-10-09 20:29:38] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:53 [I 2022-10-09 20:29:38] ...trap/engines/engine.py.229: process: 0.86350 | load: 0.00314 [I 2022-10-09 20:29:38] ...trap/engines/engine.py.230: loss: 0.20216 [I 2022-10-09 20:29:38] ...optimizers/trijoint.py.151: total_norm: 15.763365 [I 2022-10-09 20:29:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 1690/2304 [I 2022-10-09 20:29:47] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:39 [I 2022-10-09 20:29:47] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00311 [I 2022-10-09 20:29:47] ...trap/engines/engine.py.230: loss: 0.18937 [I 2022-10-09 20:29:47] ...optimizers/trijoint.py.151: total_norm: 20.356861 [I 2022-10-09 20:29:55] ...trap/engines/engine.py.224: train: epoch 53 | batch 1700/2304 [I 2022-10-09 20:29:55] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:37 [I 2022-10-09 20:29:55] ...trap/engines/engine.py.229: process: 0.86681 | load: 0.00305 [I 2022-10-09 20:29:55] ...trap/engines/engine.py.230: loss: 0.12041 [I 2022-10-09 20:29:55] ...optimizers/trijoint.py.151: total_norm: 10.163741 [I 2022-10-09 20:30:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 1710/2304 [I 2022-10-09 20:30:04] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:26 [I 2022-10-09 20:30:04] ...trap/engines/engine.py.229: process: 0.86829 | load: 0.00313 [I 2022-10-09 20:30:04] ...trap/engines/engine.py.230: loss: 0.19659 [I 2022-10-09 20:30:04] ...optimizers/trijoint.py.151: total_norm: 22.187609 [I 2022-10-09 20:30:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1720/2304 [I 2022-10-09 20:30:13] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:17 [I 2022-10-09 20:30:13] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00306 [I 2022-10-09 20:30:13] ...trap/engines/engine.py.230: loss: 0.14870 [I 2022-10-09 20:30:13] ...optimizers/trijoint.py.151: total_norm: 13.739504 [I 2022-10-09 20:30:21] ...trap/engines/engine.py.224: train: epoch 53 | batch 1730/2304 [I 2022-10-09 20:30:21] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:08 [I 2022-10-09 20:30:21] ...trap/engines/engine.py.229: process: 0.85160 | load: 0.00303 [I 2022-10-09 20:30:21] ...trap/engines/engine.py.230: loss: 0.10899 [I 2022-10-09 20:30:21] ...optimizers/trijoint.py.151: total_norm: 21.360880 [I 2022-10-09 20:30:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 1740/2304 [I 2022-10-09 20:30:29] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:57 [I 2022-10-09 20:30:29] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00300 [I 2022-10-09 20:30:29] ...trap/engines/engine.py.230: loss: 0.15523 [I 2022-10-09 20:30:29] ...optimizers/trijoint.py.151: total_norm: 17.042986 [I 2022-10-09 20:30:38] ...trap/engines/engine.py.224: train: epoch 53 | batch 1750/2304 [I 2022-10-09 20:30:38] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:46 [I 2022-10-09 20:30:38] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00297 [I 2022-10-09 20:30:38] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-09 20:30:38] ...optimizers/trijoint.py.151: total_norm: 13.329560 [I 2022-10-09 20:30:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 1760/2304 [I 2022-10-09 20:30:46] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:38 [I 2022-10-09 20:30:46] ...trap/engines/engine.py.229: process: 0.84413 | load: 0.00298 [I 2022-10-09 20:30:46] ...trap/engines/engine.py.230: loss: 0.18245 [I 2022-10-09 20:30:46] ...optimizers/trijoint.py.151: total_norm: 31.324446 [I 2022-10-09 20:30:55] ...trap/engines/engine.py.224: train: epoch 53 | batch 1770/2304 [I 2022-10-09 20:30:55] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:27 [I 2022-10-09 20:30:55] ...trap/engines/engine.py.229: process: 0.81632 | load: 0.00291 [I 2022-10-09 20:30:55] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-09 20:30:55] ...optimizers/trijoint.py.151: total_norm: 13.140143 [I 2022-10-09 20:31:03] ...trap/engines/engine.py.224: train: epoch 53 | batch 1780/2304 [I 2022-10-09 20:31:03] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:21 [I 2022-10-09 20:31:03] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00289 [I 2022-10-09 20:31:03] ...trap/engines/engine.py.230: loss: 0.14946 [I 2022-10-09 20:31:03] ...optimizers/trijoint.py.151: total_norm: 28.888781 [I 2022-10-09 20:31:12] ...trap/engines/engine.py.224: train: epoch 53 | batch 1790/2304 [I 2022-10-09 20:31:12] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:11 [I 2022-10-09 20:31:12] ...trap/engines/engine.py.229: process: 0.85081 | load: 0.00295 [I 2022-10-09 20:31:12] ...trap/engines/engine.py.230: loss: 0.11797 [I 2022-10-09 20:31:12] ...optimizers/trijoint.py.151: total_norm: 17.019358 [I 2022-10-09 20:31:20] ...trap/engines/engine.py.224: train: epoch 53 | batch 1800/2304 [I 2022-10-09 20:31:20] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:05 [I 2022-10-09 20:31:20] ...trap/engines/engine.py.229: process: 0.83536 | load: 0.00291 [I 2022-10-09 20:31:20] ...trap/engines/engine.py.230: loss: 0.21443 [I 2022-10-09 20:31:20] ...optimizers/trijoint.py.151: total_norm: 29.118462 [I 2022-10-09 20:31:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 1810/2304 [I 2022-10-09 20:31:29] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:57 [I 2022-10-09 20:31:29] ...trap/engines/engine.py.229: process: 0.85213 | load: 0.00291 [I 2022-10-09 20:31:29] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-09 20:31:29] ...optimizers/trijoint.py.151: total_norm: 14.467936 [I 2022-10-09 20:31:37] ...trap/engines/engine.py.224: train: epoch 53 | batch 1820/2304 [I 2022-10-09 20:31:37] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:47 [I 2022-10-09 20:31:37] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00295 [I 2022-10-09 20:31:37] ...trap/engines/engine.py.230: loss: 0.22558 [I 2022-10-09 20:31:37] ...optimizers/trijoint.py.151: total_norm: 14.303986 [I 2022-10-09 20:31:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 1830/2304 [I 2022-10-09 20:31:45] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:38 [I 2022-10-09 20:31:45] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00290 [I 2022-10-09 20:31:45] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-09 20:31:45] ...optimizers/trijoint.py.151: total_norm: 11.383440 [I 2022-10-09 20:31:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 1840/2304 [I 2022-10-09 20:31:54] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:31 [I 2022-10-09 20:31:54] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00303 [I 2022-10-09 20:31:54] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-09 20:31:54] ...optimizers/trijoint.py.151: total_norm: 26.299850 [I 2022-10-09 20:32:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 1850/2304 [I 2022-10-09 20:32:02] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:23 [I 2022-10-09 20:32:02] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00299 [I 2022-10-09 20:32:02] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-09 20:32:02] ...optimizers/trijoint.py.151: total_norm: 14.530883 [I 2022-10-09 20:32:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 1860/2304 [I 2022-10-09 20:32:11] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:14 [I 2022-10-09 20:32:11] ...trap/engines/engine.py.229: process: 0.83016 | load: 0.00298 [I 2022-10-09 20:32:11] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-09 20:32:11] ...optimizers/trijoint.py.151: total_norm: 13.353157 [I 2022-10-09 20:32:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 1870/2304 [I 2022-10-09 20:32:19] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:06:07 [I 2022-10-09 20:32:19] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00296 [I 2022-10-09 20:32:19] ...trap/engines/engine.py.230: loss: 0.13129 [I 2022-10-09 20:32:19] ...optimizers/trijoint.py.151: total_norm: 11.577385 [I 2022-10-09 20:32:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 1880/2304 [I 2022-10-09 20:32:28] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:59 [I 2022-10-09 20:32:28] ...trap/engines/engine.py.229: process: 0.85728 | load: 0.00298 [I 2022-10-09 20:32:28] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-09 20:32:28] ...optimizers/trijoint.py.151: total_norm: 34.924316 [I 2022-10-09 20:32:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 1890/2304 [I 2022-10-09 20:32:36] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:48 [I 2022-10-09 20:32:36] ...trap/engines/engine.py.229: process: 0.83294 | load: 0.00298 [I 2022-10-09 20:32:36] ...trap/engines/engine.py.230: loss: 0.19118 [I 2022-10-09 20:32:36] ...optimizers/trijoint.py.151: total_norm: 18.914328 [I 2022-10-09 20:32:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 1900/2304 [I 2022-10-09 20:32:45] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:41 [I 2022-10-09 20:32:45] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00301 [I 2022-10-09 20:32:45] ...trap/engines/engine.py.230: loss: 0.13305 [I 2022-10-09 20:32:45] ...optimizers/trijoint.py.151: total_norm: 13.864022 [I 2022-10-09 20:32:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 1910/2304 [I 2022-10-09 20:32:53] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:31 [I 2022-10-09 20:32:53] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00295 [I 2022-10-09 20:32:53] ...trap/engines/engine.py.230: loss: 0.13430 [I 2022-10-09 20:32:53] ...optimizers/trijoint.py.151: total_norm: 27.335211 [I 2022-10-09 20:33:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 1920/2304 [I 2022-10-09 20:33:02] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:24 [I 2022-10-09 20:33:02] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00300 [I 2022-10-09 20:33:02] ...trap/engines/engine.py.230: loss: 0.17133 [I 2022-10-09 20:33:02] ...optimizers/trijoint.py.151: total_norm: 16.383968 [I 2022-10-09 20:33:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 1930/2304 [I 2022-10-09 20:33:10] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:16 [I 2022-10-09 20:33:10] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00296 [I 2022-10-09 20:33:10] ...trap/engines/engine.py.230: loss: 0.13566 [I 2022-10-09 20:33:10] ...optimizers/trijoint.py.151: total_norm: 15.578777 [I 2022-10-09 20:33:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 1940/2304 [I 2022-10-09 20:33:19] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:09 [I 2022-10-09 20:33:19] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00330 [I 2022-10-09 20:33:19] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-09 20:33:19] ...optimizers/trijoint.py.151: total_norm: 8.841479 [I 2022-10-09 20:33:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 1950/2304 [I 2022-10-09 20:33:27] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:01 [I 2022-10-09 20:33:27] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00312 [I 2022-10-09 20:33:27] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-09 20:33:27] ...optimizers/trijoint.py.151: total_norm: 20.436560 [I 2022-10-09 20:33:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 1960/2304 [I 2022-10-09 20:33:36] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:53 [I 2022-10-09 20:33:36] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00317 [I 2022-10-09 20:33:36] ...trap/engines/engine.py.230: loss: 0.11702 [I 2022-10-09 20:33:36] ...optimizers/trijoint.py.151: total_norm: 9.176145 [I 2022-10-09 20:33:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 1970/2304 [I 2022-10-09 20:33:44] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:45 [I 2022-10-09 20:33:44] ...trap/engines/engine.py.229: process: 0.86085 | load: 0.00312 [I 2022-10-09 20:33:44] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-09 20:33:44] ...optimizers/trijoint.py.151: total_norm: 14.120172 [I 2022-10-09 20:33:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 1980/2304 [I 2022-10-09 20:33:53] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:36 [I 2022-10-09 20:33:53] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00314 [I 2022-10-09 20:33:53] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-09 20:33:53] ...optimizers/trijoint.py.151: total_norm: 16.433714 [I 2022-10-09 20:34:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 1990/2304 [I 2022-10-09 20:34:01] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:27 [I 2022-10-09 20:34:01] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00309 [I 2022-10-09 20:34:01] ...trap/engines/engine.py.230: loss: 0.16579 [I 2022-10-09 20:34:01] ...optimizers/trijoint.py.151: total_norm: 19.382973 [I 2022-10-09 20:34:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 2000/2304 [I 2022-10-09 20:34:10] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:20 [I 2022-10-09 20:34:10] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00316 [I 2022-10-09 20:34:10] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-09 20:34:10] ...optimizers/trijoint.py.151: total_norm: 20.525700 [I 2022-10-09 20:34:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2010/2304 [I 2022-10-09 20:34:18] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:11 [I 2022-10-09 20:34:18] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00313 [I 2022-10-09 20:34:18] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-09 20:34:18] ...optimizers/trijoint.py.151: total_norm: 9.526777 [I 2022-10-09 20:34:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 2020/2304 [I 2022-10-09 20:34:27] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:04:00 [I 2022-10-09 20:34:27] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00310 [I 2022-10-09 20:34:27] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-09 20:34:27] ...optimizers/trijoint.py.151: total_norm: 8.038388 [I 2022-10-09 20:34:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 2030/2304 [I 2022-10-09 20:34:36] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:53 [I 2022-10-09 20:34:36] ...trap/engines/engine.py.229: process: 0.83247 | load: 0.00299 [I 2022-10-09 20:34:36] ...trap/engines/engine.py.230: loss: 0.10329 [I 2022-10-09 20:34:36] ...optimizers/trijoint.py.151: total_norm: 14.978643 [I 2022-10-09 20:34:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 2040/2304 [I 2022-10-09 20:34:44] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:45 [I 2022-10-09 20:34:44] ...trap/engines/engine.py.229: process: 0.86765 | load: 0.00331 [I 2022-10-09 20:34:44] ...trap/engines/engine.py.230: loss: 0.20317 [I 2022-10-09 20:34:44] ...optimizers/trijoint.py.151: total_norm: 29.961857 [I 2022-10-09 20:34:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 2050/2304 [I 2022-10-09 20:34:53] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:36 [I 2022-10-09 20:34:53] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00313 [I 2022-10-09 20:34:53] ...trap/engines/engine.py.230: loss: 0.18413 [I 2022-10-09 20:34:53] ...optimizers/trijoint.py.151: total_norm: 17.262558 [I 2022-10-09 20:35:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 2060/2304 [I 2022-10-09 20:35:01] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:28 [I 2022-10-09 20:35:01] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00312 [I 2022-10-09 20:35:01] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-09 20:35:01] ...optimizers/trijoint.py.151: total_norm: 17.576132 [I 2022-10-09 20:35:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 2070/2304 [I 2022-10-09 20:35:10] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:20 [I 2022-10-09 20:35:10] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00313 [I 2022-10-09 20:35:10] ...trap/engines/engine.py.230: loss: 0.16278 [I 2022-10-09 20:35:10] ...optimizers/trijoint.py.151: total_norm: 9.896669 [I 2022-10-09 20:35:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2080/2304 [I 2022-10-09 20:35:18] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:11 [I 2022-10-09 20:35:18] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00311 [I 2022-10-09 20:35:18] ...trap/engines/engine.py.230: loss: 0.20853 [I 2022-10-09 20:35:18] ...optimizers/trijoint.py.151: total_norm: 17.974497 [I 2022-10-09 20:35:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 2090/2304 [I 2022-10-09 20:35:27] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:03 [I 2022-10-09 20:35:27] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00307 [I 2022-10-09 20:35:27] ...trap/engines/engine.py.230: loss: 0.19036 [I 2022-10-09 20:35:27] ...optimizers/trijoint.py.151: total_norm: 15.459588 [I 2022-10-09 20:35:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 2100/2304 [I 2022-10-09 20:35:35] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:54 [I 2022-10-09 20:35:35] ...trap/engines/engine.py.229: process: 0.86796 | load: 0.00316 [I 2022-10-09 20:35:35] ...trap/engines/engine.py.230: loss: 0.19230 [I 2022-10-09 20:35:35] ...optimizers/trijoint.py.151: total_norm: 8.804340 [I 2022-10-09 20:35:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 2110/2304 [I 2022-10-09 20:35:44] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:45 [I 2022-10-09 20:35:44] ...trap/engines/engine.py.229: process: 0.85577 | load: 0.00320 [I 2022-10-09 20:35:44] ...trap/engines/engine.py.230: loss: 0.13660 [I 2022-10-09 20:35:44] ...optimizers/trijoint.py.151: total_norm: 46.418060 [I 2022-10-09 20:35:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 2120/2304 [I 2022-10-09 20:35:53] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:37 [I 2022-10-09 20:35:53] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00315 [I 2022-10-09 20:35:53] ...trap/engines/engine.py.230: loss: 0.15631 [I 2022-10-09 20:35:53] ...optimizers/trijoint.py.151: total_norm: 16.320745 [I 2022-10-09 20:36:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 2130/2304 [I 2022-10-09 20:36:01] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:28 [I 2022-10-09 20:36:01] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00311 [I 2022-10-09 20:36:01] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-09 20:36:01] ...optimizers/trijoint.py.151: total_norm: 12.875451 [I 2022-10-09 20:36:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 2140/2304 [I 2022-10-09 20:36:10] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:20 [I 2022-10-09 20:36:10] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00312 [I 2022-10-09 20:36:10] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-09 20:36:10] ...optimizers/trijoint.py.151: total_norm: 12.239169 [I 2022-10-09 20:36:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2150/2304 [I 2022-10-09 20:36:18] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:11 [I 2022-10-09 20:36:18] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00313 [I 2022-10-09 20:36:18] ...trap/engines/engine.py.230: loss: 0.12638 [I 2022-10-09 20:36:18] ...optimizers/trijoint.py.151: total_norm: 12.165060 [I 2022-10-09 20:36:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 2160/2304 [I 2022-10-09 20:36:27] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:02 [I 2022-10-09 20:36:27] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00311 [I 2022-10-09 20:36:27] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-09 20:36:27] ...optimizers/trijoint.py.151: total_norm: 14.710284 [I 2022-10-09 20:36:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 2170/2304 [I 2022-10-09 20:36:35] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:54 [I 2022-10-09 20:36:35] ...trap/engines/engine.py.229: process: 0.85771 | load: 0.00313 [I 2022-10-09 20:36:35] ...trap/engines/engine.py.230: loss: 0.13992 [I 2022-10-09 20:36:35] ...optimizers/trijoint.py.151: total_norm: 18.618715 [I 2022-10-09 20:36:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 2180/2304 [I 2022-10-09 20:36:44] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:45 [I 2022-10-09 20:36:44] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00311 [I 2022-10-09 20:36:44] ...trap/engines/engine.py.230: loss: 0.14586 [I 2022-10-09 20:36:44] ...optimizers/trijoint.py.151: total_norm: 15.223876 [I 2022-10-09 20:36:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 2190/2304 [I 2022-10-09 20:36:52] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:37 [I 2022-10-09 20:36:52] ...trap/engines/engine.py.229: process: 0.85964 | load: 0.00312 [I 2022-10-09 20:36:52] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-09 20:36:52] ...optimizers/trijoint.py.151: total_norm: 9.346500 [I 2022-10-09 20:37:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 2200/2304 [I 2022-10-09 20:37:01] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:28 [I 2022-10-09 20:37:01] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00299 [I 2022-10-09 20:37:01] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-09 20:37:01] ...optimizers/trijoint.py.151: total_norm: 8.931110 [I 2022-10-09 20:37:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 2210/2304 [I 2022-10-09 20:37:09] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:20 [I 2022-10-09 20:37:09] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00316 [I 2022-10-09 20:37:09] ...trap/engines/engine.py.230: loss: 0.21351 [I 2022-10-09 20:37:09] ...optimizers/trijoint.py.151: total_norm: 12.709765 [I 2022-10-09 20:37:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2220/2304 [I 2022-10-09 20:37:18] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:11 [I 2022-10-09 20:37:18] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00312 [I 2022-10-09 20:37:18] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-09 20:37:18] ...optimizers/trijoint.py.151: total_norm: 10.813403 [I 2022-10-09 20:37:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 2230/2304 [I 2022-10-09 20:37:27] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:01:03 [I 2022-10-09 20:37:27] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00311 [I 2022-10-09 20:37:27] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-09 20:37:27] ...optimizers/trijoint.py.151: total_norm: 15.626069 [I 2022-10-09 20:37:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 2240/2304 [I 2022-10-09 20:37:35] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:54 [I 2022-10-09 20:37:35] ...trap/engines/engine.py.229: process: 0.85212 | load: 0.00318 [I 2022-10-09 20:37:35] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-09 20:37:35] ...optimizers/trijoint.py.151: total_norm: 12.642682 [I 2022-10-09 20:37:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 2250/2304 [I 2022-10-09 20:37:44] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:45 [I 2022-10-09 20:37:44] ...trap/engines/engine.py.229: process: 0.83520 | load: 0.00310 [I 2022-10-09 20:37:44] ...trap/engines/engine.py.230: loss: 0.16426 [I 2022-10-09 20:37:44] ...optimizers/trijoint.py.151: total_norm: 13.326897 [I 2022-10-09 20:37:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 2260/2304 [I 2022-10-09 20:37:52] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:37 [I 2022-10-09 20:37:52] ...trap/engines/engine.py.229: process: 0.83854 | load: 0.00312 [I 2022-10-09 20:37:52] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-09 20:37:52] ...optimizers/trijoint.py.151: total_norm: 9.357747 [I 2022-10-09 20:38:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 2270/2304 [I 2022-10-09 20:38:01] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:28 [I 2022-10-09 20:38:01] ...trap/engines/engine.py.229: process: 0.82812 | load: 0.00314 [I 2022-10-09 20:38:01] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-09 20:38:01] ...optimizers/trijoint.py.151: total_norm: 18.786024 [I 2022-10-09 20:38:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 2280/2304 [I 2022-10-09 20:38:09] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:20 [I 2022-10-09 20:38:09] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00309 [I 2022-10-09 20:38:09] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-09 20:38:09] ...optimizers/trijoint.py.151: total_norm: 12.225450 [I 2022-10-09 20:38:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2290/2304 [I 2022-10-09 20:38:18] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:11 [I 2022-10-09 20:38:18] ...trap/engines/engine.py.229: process: 0.83094 | load: 0.00299 [I 2022-10-09 20:38:18] ...trap/engines/engine.py.230: loss: 0.17145 [I 2022-10-09 20:38:18] ...optimizers/trijoint.py.151: total_norm: 11.482694 [I 2022-10-09 20:38:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 2300/2304 [I 2022-10-09 20:38:26] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:03 [I 2022-10-09 20:38:26] ...trap/engines/engine.py.229: process: 0.82345 | load: 0.00016 [I 2022-10-09 20:38:26] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-09 20:38:26] ...optimizers/trijoint.py.151: total_norm: 8.340809 [I 2022-10-09 20:38:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 2304/2304 [I 2022-10-09 20:38:29] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:00 [I 2022-10-09 20:38:29] ...trap/engines/engine.py.229: process: 0.82820 | load: 0.00016 [I 2022-10-09 20:38:29] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-09 20:38:29] ...optimizers/trijoint.py.151: total_norm: 19.802477 [S 2022-10-09 20:38:31] ...trap/engines/engine.py.129: train_epoch.epoch: 53 [S 2022-10-09 20:38:31] ...trap/engines/engine.py.129: train_epoch.loss: 0.1523850921501048 [I 2022-10-09 20:38:31] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 20:38:31] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 53 [I 2022-10-09 20:38:32] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 20:38:33] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 20:38:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 20:38:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 20:38:35] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 20:38:36] ...trap/engines/engine.py.321: eval: epoch 53 | batch 0/510 [I 2022-10-09 20:38:36] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:26 [I 2022-10-09 20:38:36] ...trap/engines/engine.py.326: process: 4.16969 | load: 3.55075 [I 2022-10-09 20:38:41] ...trap/engines/engine.py.321: eval: epoch 53 | batch 10/510 [I 2022-10-09 20:38:41] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:11 [I 2022-10-09 20:38:41] ...trap/engines/engine.py.326: process: 0.64687 | load: 0.00018 [I 2022-10-09 20:38:47] ...trap/engines/engine.py.321: eval: epoch 53 | batch 20/510 [I 2022-10-09 20:38:47] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:23 [I 2022-10-09 20:38:47] ...trap/engines/engine.py.326: process: 0.71172 | load: 0.00019 [I 2022-10-09 20:38:53] ...trap/engines/engine.py.321: eval: epoch 53 | batch 30/510 [I 2022-10-09 20:38:53] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:08 [I 2022-10-09 20:38:53] ...trap/engines/engine.py.326: process: 0.42434 | load: 0.00018 [I 2022-10-09 20:38:58] ...trap/engines/engine.py.321: eval: epoch 53 | batch 40/510 [I 2022-10-09 20:38:58] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:52 [I 2022-10-09 20:38:58] ...trap/engines/engine.py.326: process: 0.57266 | load: 0.00021 [I 2022-10-09 20:39:02] ...trap/engines/engine.py.321: eval: epoch 53 | batch 50/510 [I 2022-10-09 20:39:02] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:14 [I 2022-10-09 20:39:02] ...trap/engines/engine.py.326: process: 0.41306 | load: 0.00019 [I 2022-10-09 20:39:06] ...trap/engines/engine.py.321: eval: epoch 53 | batch 60/510 [I 2022-10-09 20:39:06] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:07 [I 2022-10-09 20:39:06] ...trap/engines/engine.py.326: process: 0.41612 | load: 0.00018 [I 2022-10-09 20:39:10] ...trap/engines/engine.py.321: eval: epoch 53 | batch 70/510 [I 2022-10-09 20:39:10] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:08 [I 2022-10-09 20:39:10] ...trap/engines/engine.py.326: process: 0.47842 | load: 0.00018 [I 2022-10-09 20:39:15] ...trap/engines/engine.py.321: eval: epoch 53 | batch 80/510 [I 2022-10-09 20:39:15] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:03 [I 2022-10-09 20:39:15] ...trap/engines/engine.py.326: process: 0.42220 | load: 0.00018 [I 2022-10-09 20:39:19] ...trap/engines/engine.py.321: eval: epoch 53 | batch 90/510 [I 2022-10-09 20:39:19] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:09 [I 2022-10-09 20:39:19] ...trap/engines/engine.py.326: process: 0.44082 | load: 0.00018 [I 2022-10-09 20:39:24] ...trap/engines/engine.py.321: eval: epoch 53 | batch 100/510 [I 2022-10-09 20:39:24] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:48 [I 2022-10-09 20:39:24] ...trap/engines/engine.py.326: process: 0.30282 | load: 0.00018 [I 2022-10-09 20:39:27] ...trap/engines/engine.py.321: eval: epoch 53 | batch 110/510 [I 2022-10-09 20:39:27] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-09 20:39:27] ...trap/engines/engine.py.326: process: 0.31543 | load: 0.00018 [I 2022-10-09 20:39:30] ...trap/engines/engine.py.321: eval: epoch 53 | batch 120/510 [I 2022-10-09 20:39:30] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:03 [I 2022-10-09 20:39:30] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00019 [I 2022-10-09 20:39:33] ...trap/engines/engine.py.321: eval: epoch 53 | batch 130/510 [I 2022-10-09 20:39:33] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:59 [I 2022-10-09 20:39:33] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00018 [I 2022-10-09 20:39:36] ...trap/engines/engine.py.321: eval: epoch 53 | batch 140/510 [I 2022-10-09 20:39:36] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:56 [I 2022-10-09 20:39:36] ...trap/engines/engine.py.326: process: 0.31059 | load: 0.00021 [I 2022-10-09 20:39:39] ...trap/engines/engine.py.321: eval: epoch 53 | batch 150/510 [I 2022-10-09 20:39:39] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:53 [I 2022-10-09 20:39:39] ...trap/engines/engine.py.326: process: 0.30983 | load: 0.00020 [I 2022-10-09 20:39:42] ...trap/engines/engine.py.321: eval: epoch 53 | batch 160/510 [I 2022-10-09 20:39:42] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:51 [I 2022-10-09 20:39:42] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00021 [I 2022-10-09 20:39:46] ...trap/engines/engine.py.321: eval: epoch 53 | batch 170/510 [I 2022-10-09 20:39:46] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-09 20:39:46] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00019 [I 2022-10-09 20:39:49] ...trap/engines/engine.py.321: eval: epoch 53 | batch 180/510 [I 2022-10-09 20:39:49] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:44 [I 2022-10-09 20:39:49] ...trap/engines/engine.py.326: process: 0.32227 | load: 0.00021 [I 2022-10-09 20:39:52] ...trap/engines/engine.py.321: eval: epoch 53 | batch 190/510 [I 2022-10-09 20:39:52] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:41 [I 2022-10-09 20:39:52] ...trap/engines/engine.py.326: process: 0.32111 | load: 0.00021 [I 2022-10-09 20:39:55] ...trap/engines/engine.py.321: eval: epoch 53 | batch 200/510 [I 2022-10-09 20:39:55] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:37 [I 2022-10-09 20:39:55] ...trap/engines/engine.py.326: process: 0.31144 | load: 0.00020 [I 2022-10-09 20:39:58] ...trap/engines/engine.py.321: eval: epoch 53 | batch 210/510 [I 2022-10-09 20:39:58] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:34 [I 2022-10-09 20:39:58] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00019 [I 2022-10-09 20:40:02] ...trap/engines/engine.py.321: eval: epoch 53 | batch 220/510 [I 2022-10-09 20:40:02] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:31 [I 2022-10-09 20:40:02] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00019 [I 2022-10-09 20:40:05] ...trap/engines/engine.py.321: eval: epoch 53 | batch 230/510 [I 2022-10-09 20:40:05] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-09 20:40:05] ...trap/engines/engine.py.326: process: 0.31346 | load: 0.00019 [I 2022-10-09 20:40:08] ...trap/engines/engine.py.321: eval: epoch 53 | batch 240/510 [I 2022-10-09 20:40:08] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:25 [I 2022-10-09 20:40:08] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00019 [I 2022-10-09 20:40:11] ...trap/engines/engine.py.321: eval: epoch 53 | batch 250/510 [I 2022-10-09 20:40:11] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-09 20:40:11] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00020 [I 2022-10-09 20:40:14] ...trap/engines/engine.py.321: eval: epoch 53 | batch 260/510 [I 2022-10-09 20:40:14] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-09 20:40:14] ...trap/engines/engine.py.326: process: 0.31779 | load: 0.00020 [I 2022-10-09 20:40:17] ...trap/engines/engine.py.321: eval: epoch 53 | batch 270/510 [I 2022-10-09 20:40:17] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:16 [I 2022-10-09 20:40:17] ...trap/engines/engine.py.326: process: 0.31581 | load: 0.00019 [I 2022-10-09 20:40:21] ...trap/engines/engine.py.321: eval: epoch 53 | batch 280/510 [I 2022-10-09 20:40:21] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-09 20:40:21] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00020 [I 2022-10-09 20:40:24] ...trap/engines/engine.py.321: eval: epoch 53 | batch 290/510 [I 2022-10-09 20:40:24] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:09 [I 2022-10-09 20:40:24] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00019 [I 2022-10-09 20:40:27] ...trap/engines/engine.py.321: eval: epoch 53 | batch 300/510 [I 2022-10-09 20:40:27] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-09 20:40:27] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00019 [I 2022-10-09 20:40:30] ...trap/engines/engine.py.321: eval: epoch 53 | batch 310/510 [I 2022-10-09 20:40:30] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-09 20:40:30] ...trap/engines/engine.py.326: process: 0.31672 | load: 0.00020 [I 2022-10-09 20:40:33] ...trap/engines/engine.py.321: eval: epoch 53 | batch 320/510 [I 2022-10-09 20:40:33] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-09 20:40:33] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00020 [I 2022-10-09 20:40:36] ...trap/engines/engine.py.321: eval: epoch 53 | batch 330/510 [I 2022-10-09 20:40:36] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-09 20:40:36] ...trap/engines/engine.py.326: process: 0.31736 | load: 0.00021 [I 2022-10-09 20:40:40] ...trap/engines/engine.py.321: eval: epoch 53 | batch 340/510 [I 2022-10-09 20:40:40] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:53 [I 2022-10-09 20:40:40] ...trap/engines/engine.py.326: process: 0.31414 | load: 0.00019 [I 2022-10-09 20:40:43] ...trap/engines/engine.py.321: eval: epoch 53 | batch 350/510 [I 2022-10-09 20:40:43] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-09 20:40:43] ...trap/engines/engine.py.326: process: 0.32114 | load: 0.00020 [I 2022-10-09 20:40:46] ...trap/engines/engine.py.321: eval: epoch 53 | batch 360/510 [I 2022-10-09 20:40:46] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-09 20:40:46] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00020 [I 2022-10-09 20:40:49] ...trap/engines/engine.py.321: eval: epoch 53 | batch 370/510 [I 2022-10-09 20:40:49] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:44 [I 2022-10-09 20:40:49] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00020 [I 2022-10-09 20:40:52] ...trap/engines/engine.py.321: eval: epoch 53 | batch 380/510 [I 2022-10-09 20:40:52] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:41 [I 2022-10-09 20:40:52] ...trap/engines/engine.py.326: process: 0.33268 | load: 0.00019 [I 2022-10-09 20:40:56] ...trap/engines/engine.py.321: eval: epoch 53 | batch 390/510 [I 2022-10-09 20:40:56] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-09 20:40:56] ...trap/engines/engine.py.326: process: 0.32362 | load: 0.00019 [I 2022-10-09 20:40:59] ...trap/engines/engine.py.321: eval: epoch 53 | batch 400/510 [I 2022-10-09 20:40:59] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:35 [I 2022-10-09 20:40:59] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00020 [I 2022-10-09 20:41:02] ...trap/engines/engine.py.321: eval: epoch 53 | batch 410/510 [I 2022-10-09 20:41:02] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-09 20:41:02] ...trap/engines/engine.py.326: process: 0.32575 | load: 0.00020 [I 2022-10-09 20:41:05] ...trap/engines/engine.py.321: eval: epoch 53 | batch 420/510 [I 2022-10-09 20:41:05] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-09 20:41:05] ...trap/engines/engine.py.326: process: 0.31688 | load: 0.00021 [I 2022-10-09 20:41:08] ...trap/engines/engine.py.321: eval: epoch 53 | batch 430/510 [I 2022-10-09 20:41:08] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:25 [I 2022-10-09 20:41:08] ...trap/engines/engine.py.326: process: 0.32678 | load: 0.00020 [I 2022-10-09 20:41:12] ...trap/engines/engine.py.321: eval: epoch 53 | batch 440/510 [I 2022-10-09 20:41:12] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-09 20:41:12] ...trap/engines/engine.py.326: process: 0.32144 | load: 0.00021 [I 2022-10-09 20:41:15] ...trap/engines/engine.py.321: eval: epoch 53 | batch 450/510 [I 2022-10-09 20:41:15] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-09 20:41:15] ...trap/engines/engine.py.326: process: 0.31219 | load: 0.00021 [I 2022-10-09 20:41:18] ...trap/engines/engine.py.321: eval: epoch 53 | batch 460/510 [I 2022-10-09 20:41:18] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-09 20:41:18] ...trap/engines/engine.py.326: process: 0.31762 | load: 0.00020 [I 2022-10-09 20:41:21] ...trap/engines/engine.py.321: eval: epoch 53 | batch 470/510 [I 2022-10-09 20:41:21] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-09 20:41:21] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00021 [I 2022-10-09 20:41:24] ...trap/engines/engine.py.321: eval: epoch 53 | batch 480/510 [I 2022-10-09 20:41:24] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-09 20:41:24] ...trap/engines/engine.py.326: process: 0.31213 | load: 0.00020 [I 2022-10-09 20:41:28] ...trap/engines/engine.py.321: eval: epoch 53 | batch 490/510 [I 2022-10-09 20:41:28] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-09 20:41:28] ...trap/engines/engine.py.326: process: 0.31512 | load: 0.00021 [I 2022-10-09 20:41:31] ...trap/engines/engine.py.321: eval: epoch 53 | batch 500/510 [I 2022-10-09 20:41:31] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-09 20:41:31] ...trap/engines/engine.py.326: process: 0.32055 | load: 0.00021 [I 2022-10-09 20:41:34] ...trap/engines/engine.py.321: eval: epoch 53 | batch 510/510 [I 2022-10-09 20:41:34] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-09 20:41:34] ...trap/engines/engine.py.326: process: 0.31377 | load: 0.00015 [S 2022-10-09 20:41:34] ...trap/engines/engine.py.132: eval_epoch.epoch: 53 [I 2022-10-09 20:41:34] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.725000 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907300 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938500 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.716400 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905100 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936800 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011153 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007537 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006185 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014044 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009617 [S 2022-10-09 20:41:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006242 [I 2022-10-09 20:42:09] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 20:42:09] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 20:42:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 20:42:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 20:42:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 20:42:12] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 20:42:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 20:42:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 20:42:19] ...trap/engines/engine.py.163: Training model on trainset for epoch 54 [I 2022-10-09 20:42:19] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 20:42:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 0/2304 [I 2022-10-09 20:42:22] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:57 [I 2022-10-09 20:42:22] ...trap/engines/engine.py.229: process: 3.04561 | load: 2.13248 [I 2022-10-09 20:42:22] ...trap/engines/engine.py.230: loss: 0.13077 [I 2022-10-09 20:42:22] ...optimizers/trijoint.py.151: total_norm: 20.672234 [I 2022-10-09 20:42:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 10/2304 [I 2022-10-09 20:42:31] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:36 [I 2022-10-09 20:42:31] ...trap/engines/engine.py.229: process: 0.86971 | load: 0.00643 [I 2022-10-09 20:42:31] ...trap/engines/engine.py.230: loss: 0.12263 [I 2022-10-09 20:42:31] ...optimizers/trijoint.py.151: total_norm: 12.860163 [I 2022-10-09 20:42:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 20/2304 [I 2022-10-09 20:42:39] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:28 [I 2022-10-09 20:42:39] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00514 [I 2022-10-09 20:42:39] ...trap/engines/engine.py.230: loss: 0.27591 [I 2022-10-09 20:42:39] ...optimizers/trijoint.py.151: total_norm: 30.744659 [I 2022-10-09 20:42:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 30/2304 [I 2022-10-09 20:42:48] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:23 [I 2022-10-09 20:42:48] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00387 [I 2022-10-09 20:42:48] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-09 20:42:48] ...optimizers/trijoint.py.151: total_norm: 8.529337 [I 2022-10-09 20:42:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 40/2304 [I 2022-10-09 20:42:57] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:17 [I 2022-10-09 20:42:57] ...trap/engines/engine.py.229: process: 0.86057 | load: 0.00421 [I 2022-10-09 20:42:57] ...trap/engines/engine.py.230: loss: 0.16524 [I 2022-10-09 20:42:57] ...optimizers/trijoint.py.151: total_norm: 10.674625 [I 2022-10-09 20:43:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 50/2304 [I 2022-10-09 20:43:05] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:08 [I 2022-10-09 20:43:05] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00335 [I 2022-10-09 20:43:05] ...trap/engines/engine.py.230: loss: 0.13077 [I 2022-10-09 20:43:05] ...optimizers/trijoint.py.151: total_norm: 11.580376 [I 2022-10-09 20:43:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 60/2304 [I 2022-10-09 20:43:14] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:56 [I 2022-10-09 20:43:14] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00370 [I 2022-10-09 20:43:14] ...trap/engines/engine.py.230: loss: 0.17626 [I 2022-10-09 20:43:14] ...optimizers/trijoint.py.151: total_norm: 22.340052 [I 2022-10-09 20:43:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 70/2304 [I 2022-10-09 20:43:22] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:40 [I 2022-10-09 20:43:22] ...trap/engines/engine.py.229: process: 0.86190 | load: 0.00417 [I 2022-10-09 20:43:22] ...trap/engines/engine.py.230: loss: 0.15830 [I 2022-10-09 20:43:22] ...optimizers/trijoint.py.151: total_norm: 20.102770 [I 2022-10-09 20:43:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 80/2304 [I 2022-10-09 20:43:31] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:48 [I 2022-10-09 20:43:31] ...trap/engines/engine.py.229: process: 0.85961 | load: 0.00426 [I 2022-10-09 20:43:31] ...trap/engines/engine.py.230: loss: 0.20206 [I 2022-10-09 20:43:31] ...optimizers/trijoint.py.151: total_norm: 18.890850 [I 2022-10-09 20:43:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 90/2304 [I 2022-10-09 20:43:39] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:41 [I 2022-10-09 20:43:39] ...trap/engines/engine.py.229: process: 0.84898 | load: 0.00354 [I 2022-10-09 20:43:39] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-09 20:43:39] ...optimizers/trijoint.py.151: total_norm: 10.421576 [I 2022-10-09 20:43:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 100/2304 [I 2022-10-09 20:43:48] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:30 [I 2022-10-09 20:43:48] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00336 [I 2022-10-09 20:43:48] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-09 20:43:48] ...optimizers/trijoint.py.151: total_norm: 14.276506 [I 2022-10-09 20:43:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 110/2304 [I 2022-10-09 20:43:57] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:01 [I 2022-10-09 20:43:57] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00325 [I 2022-10-09 20:43:57] ...trap/engines/engine.py.230: loss: 0.13321 [I 2022-10-09 20:43:57] ...optimizers/trijoint.py.151: total_norm: 15.039240 [I 2022-10-09 20:44:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 120/2304 [I 2022-10-09 20:44:05] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:56 [I 2022-10-09 20:44:05] ...trap/engines/engine.py.229: process: 0.86826 | load: 0.00336 [I 2022-10-09 20:44:05] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-09 20:44:05] ...optimizers/trijoint.py.151: total_norm: 11.757661 [I 2022-10-09 20:44:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 130/2304 [I 2022-10-09 20:44:14] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:08 [I 2022-10-09 20:44:14] ...trap/engines/engine.py.229: process: 0.87121 | load: 0.00312 [I 2022-10-09 20:44:14] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-09 20:44:14] ...optimizers/trijoint.py.151: total_norm: 22.949629 [I 2022-10-09 20:44:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 140/2304 [I 2022-10-09 20:44:22] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:36 [I 2022-10-09 20:44:22] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00330 [I 2022-10-09 20:44:22] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-09 20:44:22] ...optimizers/trijoint.py.151: total_norm: 17.624731 [I 2022-10-09 20:44:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 150/2304 [I 2022-10-09 20:44:31] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:37 [I 2022-10-09 20:44:31] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00324 [I 2022-10-09 20:44:31] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-09 20:44:31] ...optimizers/trijoint.py.151: total_norm: 18.068617 [I 2022-10-09 20:44:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 160/2304 [I 2022-10-09 20:44:39] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:26 [I 2022-10-09 20:44:39] ...trap/engines/engine.py.229: process: 0.85995 | load: 0.00356 [I 2022-10-09 20:44:39] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-09 20:44:39] ...optimizers/trijoint.py.151: total_norm: 10.501652 [I 2022-10-09 20:44:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 170/2304 [I 2022-10-09 20:44:48] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:09 [I 2022-10-09 20:44:48] ...trap/engines/engine.py.229: process: 0.82961 | load: 0.00332 [I 2022-10-09 20:44:48] ...trap/engines/engine.py.230: loss: 0.13575 [I 2022-10-09 20:44:48] ...optimizers/trijoint.py.151: total_norm: 20.765865 [I 2022-10-09 20:44:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 180/2304 [I 2022-10-09 20:44:56] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:02 [I 2022-10-09 20:44:56] ...trap/engines/engine.py.229: process: 0.84775 | load: 0.00310 [I 2022-10-09 20:44:56] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-09 20:44:56] ...optimizers/trijoint.py.151: total_norm: 13.568669 [I 2022-10-09 20:45:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 190/2304 [I 2022-10-09 20:45:05] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:02 [I 2022-10-09 20:45:05] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00332 [I 2022-10-09 20:45:05] ...trap/engines/engine.py.230: loss: 0.15626 [I 2022-10-09 20:45:05] ...optimizers/trijoint.py.151: total_norm: 19.272722 [I 2022-10-09 20:45:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 200/2304 [I 2022-10-09 20:45:13] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:46 [I 2022-10-09 20:45:13] ...trap/engines/engine.py.229: process: 0.84004 | load: 0.00321 [I 2022-10-09 20:45:13] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-09 20:45:13] ...optimizers/trijoint.py.151: total_norm: 14.495300 [I 2022-10-09 20:45:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 210/2304 [I 2022-10-09 20:45:22] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:55 [I 2022-10-09 20:45:22] ...trap/engines/engine.py.229: process: 0.85775 | load: 0.00335 [I 2022-10-09 20:45:22] ...trap/engines/engine.py.230: loss: 0.25067 [I 2022-10-09 20:45:22] ...optimizers/trijoint.py.151: total_norm: 22.124809 [I 2022-10-09 20:45:30] ...trap/engines/engine.py.224: train: epoch 54 | batch 220/2304 [I 2022-10-09 20:45:30] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:39 [I 2022-10-09 20:45:30] ...trap/engines/engine.py.229: process: 0.86250 | load: 0.00312 [I 2022-10-09 20:45:30] ...trap/engines/engine.py.230: loss: 0.20898 [I 2022-10-09 20:45:30] ...optimizers/trijoint.py.151: total_norm: 27.535568 [I 2022-10-09 20:45:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 230/2304 [I 2022-10-09 20:45:39] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:34 [I 2022-10-09 20:45:39] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00343 [I 2022-10-09 20:45:39] ...trap/engines/engine.py.230: loss: 0.20150 [I 2022-10-09 20:45:39] ...optimizers/trijoint.py.151: total_norm: 17.921652 [I 2022-10-09 20:45:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 240/2304 [I 2022-10-09 20:45:48] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:28 [I 2022-10-09 20:45:48] ...trap/engines/engine.py.229: process: 0.86323 | load: 0.00325 [I 2022-10-09 20:45:48] ...trap/engines/engine.py.230: loss: 0.12071 [I 2022-10-09 20:45:48] ...optimizers/trijoint.py.151: total_norm: 12.807606 [I 2022-10-09 20:45:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 250/2304 [I 2022-10-09 20:45:56] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:21 [I 2022-10-09 20:45:56] ...trap/engines/engine.py.229: process: 0.86553 | load: 0.00338 [I 2022-10-09 20:45:56] ...trap/engines/engine.py.230: loss: 0.10819 [I 2022-10-09 20:45:56] ...optimizers/trijoint.py.151: total_norm: 8.098390 [I 2022-10-09 20:46:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 260/2304 [I 2022-10-09 20:46:05] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:02 [I 2022-10-09 20:46:05] ...trap/engines/engine.py.229: process: 0.86227 | load: 0.00336 [I 2022-10-09 20:46:05] ...trap/engines/engine.py.230: loss: 0.15545 [I 2022-10-09 20:46:05] ...optimizers/trijoint.py.151: total_norm: 15.327578 [I 2022-10-09 20:46:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 270/2304 [I 2022-10-09 20:46:13] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:51 [I 2022-10-09 20:46:13] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00312 [I 2022-10-09 20:46:13] ...trap/engines/engine.py.230: loss: 0.23967 [I 2022-10-09 20:46:13] ...optimizers/trijoint.py.151: total_norm: 17.966072 [I 2022-10-09 20:46:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 280/2304 [I 2022-10-09 20:46:22] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:47 [I 2022-10-09 20:46:22] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00327 [I 2022-10-09 20:46:22] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-09 20:46:22] ...optimizers/trijoint.py.151: total_norm: 20.863985 [I 2022-10-09 20:46:30] ...trap/engines/engine.py.224: train: epoch 54 | batch 290/2304 [I 2022-10-09 20:46:30] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:32 [I 2022-10-09 20:46:30] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00314 [I 2022-10-09 20:46:30] ...trap/engines/engine.py.230: loss: 0.21907 [I 2022-10-09 20:46:30] ...optimizers/trijoint.py.151: total_norm: 13.460134 [I 2022-10-09 20:46:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 300/2304 [I 2022-10-09 20:46:39] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:36 [I 2022-10-09 20:46:39] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00313 [I 2022-10-09 20:46:39] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-09 20:46:39] ...optimizers/trijoint.py.151: total_norm: 11.569555 [I 2022-10-09 20:46:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 310/2304 [I 2022-10-09 20:46:47] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:16 [I 2022-10-09 20:46:47] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00323 [I 2022-10-09 20:46:47] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-09 20:46:47] ...optimizers/trijoint.py.151: total_norm: 21.176998 [I 2022-10-09 20:46:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 320/2304 [I 2022-10-09 20:46:56] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:02 [I 2022-10-09 20:46:56] ...trap/engines/engine.py.229: process: 0.83747 | load: 0.00300 [I 2022-10-09 20:46:56] ...trap/engines/engine.py.230: loss: 0.20248 [I 2022-10-09 20:46:56] ...optimizers/trijoint.py.151: total_norm: 28.068230 [I 2022-10-09 20:47:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 330/2304 [I 2022-10-09 20:47:04] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:27:48 [I 2022-10-09 20:47:04] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00306 [I 2022-10-09 20:47:04] ...trap/engines/engine.py.230: loss: 0.10676 [I 2022-10-09 20:47:04] ...optimizers/trijoint.py.151: total_norm: 8.227419 [I 2022-10-09 20:47:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 340/2304 [I 2022-10-09 20:47:13] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:32 [I 2022-10-09 20:47:13] ...trap/engines/engine.py.229: process: 0.84154 | load: 0.00299 [I 2022-10-09 20:47:13] ...trap/engines/engine.py.230: loss: 0.21068 [I 2022-10-09 20:47:13] ...optimizers/trijoint.py.151: total_norm: 21.282764 [I 2022-10-09 20:47:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 350/2304 [I 2022-10-09 20:47:21] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:31 [I 2022-10-09 20:47:21] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00296 [I 2022-10-09 20:47:21] ...trap/engines/engine.py.230: loss: 0.19964 [I 2022-10-09 20:47:21] ...optimizers/trijoint.py.151: total_norm: 27.628485 [I 2022-10-09 20:47:30] ...trap/engines/engine.py.224: train: epoch 54 | batch 360/2304 [I 2022-10-09 20:47:30] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:19 [I 2022-10-09 20:47:30] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00293 [I 2022-10-09 20:47:30] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-09 20:47:30] ...optimizers/trijoint.py.151: total_norm: 21.540634 [I 2022-10-09 20:47:38] ...trap/engines/engine.py.224: train: epoch 54 | batch 370/2304 [I 2022-10-09 20:47:38] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:13 [I 2022-10-09 20:47:38] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00298 [I 2022-10-09 20:47:38] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-09 20:47:38] ...optimizers/trijoint.py.151: total_norm: 19.222322 [I 2022-10-09 20:47:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 380/2304 [I 2022-10-09 20:47:47] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:02 [I 2022-10-09 20:47:47] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00294 [I 2022-10-09 20:47:47] ...trap/engines/engine.py.230: loss: 0.11386 [I 2022-10-09 20:47:47] ...optimizers/trijoint.py.151: total_norm: 10.238289 [I 2022-10-09 20:47:55] ...trap/engines/engine.py.224: train: epoch 54 | batch 390/2304 [I 2022-10-09 20:47:55] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:26:53 [I 2022-10-09 20:47:55] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00298 [I 2022-10-09 20:47:55] ...trap/engines/engine.py.230: loss: 0.23014 [I 2022-10-09 20:47:55] ...optimizers/trijoint.py.151: total_norm: 14.746992 [I 2022-10-09 20:48:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 400/2304 [I 2022-10-09 20:48:04] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:54 [I 2022-10-09 20:48:04] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00297 [I 2022-10-09 20:48:04] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-09 20:48:04] ...optimizers/trijoint.py.151: total_norm: 13.476671 [I 2022-10-09 20:48:12] ...trap/engines/engine.py.224: train: epoch 54 | batch 410/2304 [I 2022-10-09 20:48:12] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:39 [I 2022-10-09 20:48:12] ...trap/engines/engine.py.229: process: 0.83676 | load: 0.00303 [I 2022-10-09 20:48:12] ...trap/engines/engine.py.230: loss: 0.13321 [I 2022-10-09 20:48:12] ...optimizers/trijoint.py.151: total_norm: 14.937418 [I 2022-10-09 20:48:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 420/2304 [I 2022-10-09 20:48:21] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:29 [I 2022-10-09 20:48:21] ...trap/engines/engine.py.229: process: 0.84137 | load: 0.00299 [I 2022-10-09 20:48:21] ...trap/engines/engine.py.230: loss: 0.15643 [I 2022-10-09 20:48:21] ...optimizers/trijoint.py.151: total_norm: 18.653038 [I 2022-10-09 20:48:29] ...trap/engines/engine.py.224: train: epoch 54 | batch 430/2304 [I 2022-10-09 20:48:29] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:19 [I 2022-10-09 20:48:29] ...trap/engines/engine.py.229: process: 0.84001 | load: 0.00299 [I 2022-10-09 20:48:29] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-09 20:48:29] ...optimizers/trijoint.py.151: total_norm: 10.180532 [I 2022-10-09 20:48:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 440/2304 [I 2022-10-09 20:48:37] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:28 [I 2022-10-09 20:48:37] ...trap/engines/engine.py.229: process: 0.86415 | load: 0.00301 [I 2022-10-09 20:48:37] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-09 20:48:37] ...optimizers/trijoint.py.151: total_norm: 27.508814 [I 2022-10-09 20:48:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 450/2304 [I 2022-10-09 20:48:46] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:18 [I 2022-10-09 20:48:46] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00301 [I 2022-10-09 20:48:46] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-09 20:48:46] ...optimizers/trijoint.py.151: total_norm: 18.529602 [I 2022-10-09 20:48:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 460/2304 [I 2022-10-09 20:48:54] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:57 [I 2022-10-09 20:48:54] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00300 [I 2022-10-09 20:48:54] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-09 20:48:54] ...optimizers/trijoint.py.151: total_norm: 20.192493 [I 2022-10-09 20:49:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 470/2304 [I 2022-10-09 20:49:03] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:58 [I 2022-10-09 20:49:03] ...trap/engines/engine.py.229: process: 0.85647 | load: 0.00305 [I 2022-10-09 20:49:03] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-09 20:49:03] ...optimizers/trijoint.py.151: total_norm: 16.892380 [I 2022-10-09 20:49:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 480/2304 [I 2022-10-09 20:49:11] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:41 [I 2022-10-09 20:49:11] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00318 [I 2022-10-09 20:49:11] ...trap/engines/engine.py.230: loss: 0.15451 [I 2022-10-09 20:49:11] ...optimizers/trijoint.py.151: total_norm: 43.231647 [I 2022-10-09 20:49:20] ...trap/engines/engine.py.224: train: epoch 54 | batch 490/2304 [I 2022-10-09 20:49:20] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:31 [I 2022-10-09 20:49:20] ...trap/engines/engine.py.229: process: 0.83372 | load: 0.00296 [I 2022-10-09 20:49:20] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-09 20:49:20] ...optimizers/trijoint.py.151: total_norm: 24.072491 [I 2022-10-09 20:49:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 500/2304 [I 2022-10-09 20:49:28] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:24 [I 2022-10-09 20:49:28] ...trap/engines/engine.py.229: process: 0.82882 | load: 0.00317 [I 2022-10-09 20:49:28] ...trap/engines/engine.py.230: loss: 0.11056 [I 2022-10-09 20:49:28] ...optimizers/trijoint.py.151: total_norm: 13.794403 [I 2022-10-09 20:49:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 510/2304 [I 2022-10-09 20:49:37] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:24 [I 2022-10-09 20:49:37] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00306 [I 2022-10-09 20:49:37] ...trap/engines/engine.py.230: loss: 0.14249 [I 2022-10-09 20:49:37] ...optimizers/trijoint.py.151: total_norm: 13.889803 [I 2022-10-09 20:49:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 520/2304 [I 2022-10-09 20:49:46] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:23 [I 2022-10-09 20:49:46] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00303 [I 2022-10-09 20:49:46] ...trap/engines/engine.py.230: loss: 0.15224 [I 2022-10-09 20:49:46] ...optimizers/trijoint.py.151: total_norm: 8.506786 [I 2022-10-09 20:49:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 530/2304 [I 2022-10-09 20:49:54] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:55 [I 2022-10-09 20:49:54] ...trap/engines/engine.py.229: process: 0.83612 | load: 0.00305 [I 2022-10-09 20:49:54] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-09 20:49:54] ...optimizers/trijoint.py.151: total_norm: 16.856230 [I 2022-10-09 20:50:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 540/2304 [I 2022-10-09 20:50:03] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:55 [I 2022-10-09 20:50:03] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00299 [I 2022-10-09 20:50:03] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-09 20:50:03] ...optimizers/trijoint.py.151: total_norm: 24.222174 [I 2022-10-09 20:50:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 550/2304 [I 2022-10-09 20:50:11] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:49 [I 2022-10-09 20:50:11] ...trap/engines/engine.py.229: process: 0.84015 | load: 0.00304 [I 2022-10-09 20:50:11] ...trap/engines/engine.py.230: loss: 0.15985 [I 2022-10-09 20:50:11] ...optimizers/trijoint.py.151: total_norm: 28.148773 [I 2022-10-09 20:50:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 560/2304 [I 2022-10-09 20:50:19] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:30 [I 2022-10-09 20:50:19] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00300 [I 2022-10-09 20:50:19] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-09 20:50:19] ...optimizers/trijoint.py.151: total_norm: 11.643809 [I 2022-10-09 20:50:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 570/2304 [I 2022-10-09 20:50:28] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:29 [I 2022-10-09 20:50:28] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00307 [I 2022-10-09 20:50:28] ...trap/engines/engine.py.230: loss: 0.11232 [I 2022-10-09 20:50:28] ...optimizers/trijoint.py.151: total_norm: 7.737970 [I 2022-10-09 20:50:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 580/2304 [I 2022-10-09 20:50:37] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:23 [I 2022-10-09 20:50:37] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00303 [I 2022-10-09 20:50:37] ...trap/engines/engine.py.230: loss: 0.19140 [I 2022-10-09 20:50:37] ...optimizers/trijoint.py.151: total_norm: 20.976183 [I 2022-10-09 20:50:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 590/2304 [I 2022-10-09 20:50:45] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:08 [I 2022-10-09 20:50:45] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00302 [I 2022-10-09 20:50:45] ...trap/engines/engine.py.230: loss: 0.21837 [I 2022-10-09 20:50:45] ...optimizers/trijoint.py.151: total_norm: 16.903181 [I 2022-10-09 20:50:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 600/2304 [I 2022-10-09 20:50:54] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:07 [I 2022-10-09 20:50:54] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00307 [I 2022-10-09 20:50:54] ...trap/engines/engine.py.230: loss: 0.13189 [I 2022-10-09 20:50:54] ...optimizers/trijoint.py.151: total_norm: 31.171453 [I 2022-10-09 20:51:02] ...trap/engines/engine.py.224: train: epoch 54 | batch 610/2304 [I 2022-10-09 20:51:02] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:00 [I 2022-10-09 20:51:02] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00302 [I 2022-10-09 20:51:02] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-09 20:51:02] ...optimizers/trijoint.py.151: total_norm: 7.448265 [I 2022-10-09 20:51:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 620/2304 [I 2022-10-09 20:51:11] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:49 [I 2022-10-09 20:51:11] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00304 [I 2022-10-09 20:51:11] ...trap/engines/engine.py.230: loss: 0.17460 [I 2022-10-09 20:51:11] ...optimizers/trijoint.py.151: total_norm: 22.645767 [I 2022-10-09 20:51:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 630/2304 [I 2022-10-09 20:51:19] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:35 [I 2022-10-09 20:51:19] ...trap/engines/engine.py.229: process: 0.84947 | load: 0.00298 [I 2022-10-09 20:51:19] ...trap/engines/engine.py.230: loss: 0.18245 [I 2022-10-09 20:51:19] ...optimizers/trijoint.py.151: total_norm: 10.333482 [I 2022-10-09 20:51:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 640/2304 [I 2022-10-09 20:51:28] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:28 [I 2022-10-09 20:51:28] ...trap/engines/engine.py.229: process: 0.85213 | load: 0.00311 [I 2022-10-09 20:51:28] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-09 20:51:28] ...optimizers/trijoint.py.151: total_norm: 28.128708 [I 2022-10-09 20:51:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 650/2304 [I 2022-10-09 20:51:36] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:25 [I 2022-10-09 20:51:36] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00302 [I 2022-10-09 20:51:36] ...trap/engines/engine.py.230: loss: 0.15465 [I 2022-10-09 20:51:36] ...optimizers/trijoint.py.151: total_norm: 14.689353 [I 2022-10-09 20:51:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 660/2304 [I 2022-10-09 20:51:45] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:17 [I 2022-10-09 20:51:45] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00297 [I 2022-10-09 20:51:45] ...trap/engines/engine.py.230: loss: 0.14471 [I 2022-10-09 20:51:45] ...optimizers/trijoint.py.151: total_norm: 8.884026 [I 2022-10-09 20:51:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 670/2304 [I 2022-10-09 20:51:53] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:08 [I 2022-10-09 20:51:53] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00304 [I 2022-10-09 20:51:53] ...trap/engines/engine.py.230: loss: 0.21184 [I 2022-10-09 20:51:53] ...optimizers/trijoint.py.151: total_norm: 29.864399 [I 2022-10-09 20:52:02] ...trap/engines/engine.py.224: train: epoch 54 | batch 680/2304 [I 2022-10-09 20:52:02] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:59 [I 2022-10-09 20:52:02] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00299 [I 2022-10-09 20:52:02] ...trap/engines/engine.py.230: loss: 0.11902 [I 2022-10-09 20:52:02] ...optimizers/trijoint.py.151: total_norm: 9.701614 [I 2022-10-09 20:52:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 690/2304 [I 2022-10-09 20:52:10] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:52 [I 2022-10-09 20:52:10] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00318 [I 2022-10-09 20:52:10] ...trap/engines/engine.py.230: loss: 0.16228 [I 2022-10-09 20:52:10] ...optimizers/trijoint.py.151: total_norm: 9.512956 [I 2022-10-09 20:52:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 700/2304 [I 2022-10-09 20:52:19] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:43 [I 2022-10-09 20:52:19] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00314 [I 2022-10-09 20:52:19] ...trap/engines/engine.py.230: loss: 0.16950 [I 2022-10-09 20:52:19] ...optimizers/trijoint.py.151: total_norm: 25.649920 [I 2022-10-09 20:52:27] ...trap/engines/engine.py.224: train: epoch 54 | batch 710/2304 [I 2022-10-09 20:52:27] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:37 [I 2022-10-09 20:52:27] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00311 [I 2022-10-09 20:52:27] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-09 20:52:27] ...optimizers/trijoint.py.151: total_norm: 23.724257 [I 2022-10-09 20:52:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 720/2304 [I 2022-10-09 20:52:36] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:26 [I 2022-10-09 20:52:36] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00306 [I 2022-10-09 20:52:36] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-09 20:52:36] ...optimizers/trijoint.py.151: total_norm: 13.487906 [I 2022-10-09 20:52:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 730/2304 [I 2022-10-09 20:52:44] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:16 [I 2022-10-09 20:52:44] ...trap/engines/engine.py.229: process: 0.85784 | load: 0.00307 [I 2022-10-09 20:52:44] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-09 20:52:44] ...optimizers/trijoint.py.151: total_norm: 13.863252 [I 2022-10-09 20:52:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 740/2304 [I 2022-10-09 20:52:53] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:21:59 [I 2022-10-09 20:52:53] ...trap/engines/engine.py.229: process: 0.83681 | load: 0.00295 [I 2022-10-09 20:52:53] ...trap/engines/engine.py.230: loss: 0.18741 [I 2022-10-09 20:52:53] ...optimizers/trijoint.py.151: total_norm: 16.714369 [I 2022-10-09 20:53:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 750/2304 [I 2022-10-09 20:53:01] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:58 [I 2022-10-09 20:53:01] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00301 [I 2022-10-09 20:53:01] ...trap/engines/engine.py.230: loss: 0.14763 [I 2022-10-09 20:53:01] ...optimizers/trijoint.py.151: total_norm: 23.495310 [I 2022-10-09 20:53:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 760/2304 [I 2022-10-09 20:53:10] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:52 [I 2022-10-09 20:53:10] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00303 [I 2022-10-09 20:53:10] ...trap/engines/engine.py.230: loss: 0.09240 [I 2022-10-09 20:53:10] ...optimizers/trijoint.py.151: total_norm: 8.148166 [I 2022-10-09 20:53:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 770/2304 [I 2022-10-09 20:53:18] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:41 [I 2022-10-09 20:53:18] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00301 [I 2022-10-09 20:53:18] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-09 20:53:18] ...optimizers/trijoint.py.151: total_norm: 7.730148 [I 2022-10-09 20:53:27] ...trap/engines/engine.py.224: train: epoch 54 | batch 780/2304 [I 2022-10-09 20:53:27] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:28 [I 2022-10-09 20:53:27] ...trap/engines/engine.py.229: process: 0.83581 | load: 0.00302 [I 2022-10-09 20:53:27] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-09 20:53:27] ...optimizers/trijoint.py.151: total_norm: 14.164850 [I 2022-10-09 20:53:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 790/2304 [I 2022-10-09 20:53:35] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:19 [I 2022-10-09 20:53:35] ...trap/engines/engine.py.229: process: 0.83436 | load: 0.00307 [I 2022-10-09 20:53:35] ...trap/engines/engine.py.230: loss: 0.17928 [I 2022-10-09 20:53:35] ...optimizers/trijoint.py.151: total_norm: 12.243181 [I 2022-10-09 20:53:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 800/2304 [I 2022-10-09 20:53:44] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:23 [I 2022-10-09 20:53:44] ...trap/engines/engine.py.229: process: 0.85873 | load: 0.00301 [I 2022-10-09 20:53:44] ...trap/engines/engine.py.230: loss: 0.13818 [I 2022-10-09 20:53:44] ...optimizers/trijoint.py.151: total_norm: 47.845749 [I 2022-10-09 20:53:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 810/2304 [I 2022-10-09 20:53:52] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:19 [I 2022-10-09 20:53:52] ...trap/engines/engine.py.229: process: 0.86377 | load: 0.00311 [I 2022-10-09 20:53:52] ...trap/engines/engine.py.230: loss: 0.11748 [I 2022-10-09 20:53:52] ...optimizers/trijoint.py.151: total_norm: 13.001641 [I 2022-10-09 20:54:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 820/2304 [I 2022-10-09 20:54:01] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:57 [I 2022-10-09 20:54:01] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00314 [I 2022-10-09 20:54:01] ...trap/engines/engine.py.230: loss: 0.18577 [I 2022-10-09 20:54:01] ...optimizers/trijoint.py.151: total_norm: 26.240856 [I 2022-10-09 20:54:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 830/2304 [I 2022-10-09 20:54:09] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:58 [I 2022-10-09 20:54:09] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00316 [I 2022-10-09 20:54:09] ...trap/engines/engine.py.230: loss: 0.10285 [I 2022-10-09 20:54:09] ...optimizers/trijoint.py.151: total_norm: 7.770012 [I 2022-10-09 20:54:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 840/2304 [I 2022-10-09 20:54:18] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:51 [I 2022-10-09 20:54:18] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00312 [I 2022-10-09 20:54:18] ...trap/engines/engine.py.230: loss: 0.14967 [I 2022-10-09 20:54:18] ...optimizers/trijoint.py.151: total_norm: 15.634623 [I 2022-10-09 20:54:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 850/2304 [I 2022-10-09 20:54:26] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:38 [I 2022-10-09 20:54:26] ...trap/engines/engine.py.229: process: 0.85889 | load: 0.00309 [I 2022-10-09 20:54:26] ...trap/engines/engine.py.230: loss: 0.21660 [I 2022-10-09 20:54:26] ...optimizers/trijoint.py.151: total_norm: 21.392473 [I 2022-10-09 20:54:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 860/2304 [I 2022-10-09 20:54:35] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:32 [I 2022-10-09 20:54:35] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00311 [I 2022-10-09 20:54:35] ...trap/engines/engine.py.230: loss: 0.18996 [I 2022-10-09 20:54:35] ...optimizers/trijoint.py.151: total_norm: 28.706749 [I 2022-10-09 20:54:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 870/2304 [I 2022-10-09 20:54:44] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:22 [I 2022-10-09 20:54:44] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00313 [I 2022-10-09 20:54:44] ...trap/engines/engine.py.230: loss: 0.12816 [I 2022-10-09 20:54:44] ...optimizers/trijoint.py.151: total_norm: 22.088652 [I 2022-10-09 20:54:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 880/2304 [I 2022-10-09 20:54:52] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:24 [I 2022-10-09 20:54:52] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00315 [I 2022-10-09 20:54:52] ...trap/engines/engine.py.230: loss: 0.20538 [I 2022-10-09 20:54:52] ...optimizers/trijoint.py.151: total_norm: 16.767832 [I 2022-10-09 20:55:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 890/2304 [I 2022-10-09 20:55:01] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:07 [I 2022-10-09 20:55:01] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00311 [I 2022-10-09 20:55:01] ...trap/engines/engine.py.230: loss: 0.25140 [I 2022-10-09 20:55:01] ...optimizers/trijoint.py.151: total_norm: 19.991308 [I 2022-10-09 20:55:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 900/2304 [I 2022-10-09 20:55:09] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:06 [I 2022-10-09 20:55:09] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00311 [I 2022-10-09 20:55:09] ...trap/engines/engine.py.230: loss: 0.22391 [I 2022-10-09 20:55:09] ...optimizers/trijoint.py.151: total_norm: 21.431467 [I 2022-10-09 20:55:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 910/2304 [I 2022-10-09 20:55:18] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:53 [I 2022-10-09 20:55:18] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00315 [I 2022-10-09 20:55:18] ...trap/engines/engine.py.230: loss: 0.27557 [I 2022-10-09 20:55:18] ...optimizers/trijoint.py.151: total_norm: 18.873228 [I 2022-10-09 20:55:27] ...trap/engines/engine.py.224: train: epoch 54 | batch 920/2304 [I 2022-10-09 20:55:27] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:46 [I 2022-10-09 20:55:27] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00311 [I 2022-10-09 20:55:27] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-09 20:55:27] ...optimizers/trijoint.py.151: total_norm: 11.372595 [I 2022-10-09 20:55:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 930/2304 [I 2022-10-09 20:55:35] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:33 [I 2022-10-09 20:55:35] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00312 [I 2022-10-09 20:55:35] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-09 20:55:35] ...optimizers/trijoint.py.151: total_norm: 12.965153 [I 2022-10-09 20:55:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 940/2304 [I 2022-10-09 20:55:44] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:27 [I 2022-10-09 20:55:44] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00310 [I 2022-10-09 20:55:44] ...trap/engines/engine.py.230: loss: 0.11910 [I 2022-10-09 20:55:44] ...optimizers/trijoint.py.151: total_norm: 46.405262 [I 2022-10-09 20:55:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 950/2304 [I 2022-10-09 20:55:52] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:16 [I 2022-10-09 20:55:52] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00316 [I 2022-10-09 20:55:52] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-09 20:55:52] ...optimizers/trijoint.py.151: total_norm: 15.744473 [I 2022-10-09 20:56:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 960/2304 [I 2022-10-09 20:56:01] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:01 [I 2022-10-09 20:56:01] ...trap/engines/engine.py.229: process: 0.84511 | load: 0.00309 [I 2022-10-09 20:56:01] ...trap/engines/engine.py.230: loss: 0.20243 [I 2022-10-09 20:56:01] ...optimizers/trijoint.py.151: total_norm: 20.370548 [I 2022-10-09 20:56:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 970/2304 [I 2022-10-09 20:56:09] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:51 [I 2022-10-09 20:56:09] ...trap/engines/engine.py.229: process: 0.83890 | load: 0.00349 [I 2022-10-09 20:56:09] ...trap/engines/engine.py.230: loss: 0.17043 [I 2022-10-09 20:56:09] ...optimizers/trijoint.py.151: total_norm: 24.308044 [I 2022-10-09 20:56:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 980/2304 [I 2022-10-09 20:56:18] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:48 [I 2022-10-09 20:56:18] ...trap/engines/engine.py.229: process: 0.84953 | load: 0.00311 [I 2022-10-09 20:56:18] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-09 20:56:18] ...optimizers/trijoint.py.151: total_norm: 16.355434 [I 2022-10-09 20:56:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 990/2304 [I 2022-10-09 20:56:26] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:38 [I 2022-10-09 20:56:26] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00314 [I 2022-10-09 20:56:26] ...trap/engines/engine.py.230: loss: 0.12278 [I 2022-10-09 20:56:26] ...optimizers/trijoint.py.151: total_norm: 14.328547 [I 2022-10-09 20:56:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 1000/2304 [I 2022-10-09 20:56:35] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:37 [I 2022-10-09 20:56:35] ...trap/engines/engine.py.229: process: 0.85696 | load: 0.00312 [I 2022-10-09 20:56:35] ...trap/engines/engine.py.230: loss: 0.18573 [I 2022-10-09 20:56:35] ...optimizers/trijoint.py.151: total_norm: 21.732641 [I 2022-10-09 20:56:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 1010/2304 [I 2022-10-09 20:56:44] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:31 [I 2022-10-09 20:56:44] ...trap/engines/engine.py.229: process: 0.87196 | load: 0.00313 [I 2022-10-09 20:56:44] ...trap/engines/engine.py.230: loss: 0.18790 [I 2022-10-09 20:56:44] ...optimizers/trijoint.py.151: total_norm: 17.881937 [I 2022-10-09 20:56:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 1020/2304 [I 2022-10-09 20:56:52] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:13 [I 2022-10-09 20:56:52] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00312 [I 2022-10-09 20:56:52] ...trap/engines/engine.py.230: loss: 0.14095 [I 2022-10-09 20:56:52] ...optimizers/trijoint.py.151: total_norm: 7.838603 [I 2022-10-09 20:57:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 1030/2304 [I 2022-10-09 20:57:01] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:08 [I 2022-10-09 20:57:01] ...trap/engines/engine.py.229: process: 0.86085 | load: 0.00316 [I 2022-10-09 20:57:01] ...trap/engines/engine.py.230: loss: 0.14831 [I 2022-10-09 20:57:01] ...optimizers/trijoint.py.151: total_norm: 13.325971 [I 2022-10-09 20:57:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 1040/2304 [I 2022-10-09 20:57:09] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:02 [I 2022-10-09 20:57:09] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00317 [I 2022-10-09 20:57:09] ...trap/engines/engine.py.230: loss: 0.28000 [I 2022-10-09 20:57:09] ...optimizers/trijoint.py.151: total_norm: 47.473530 [I 2022-10-09 20:57:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 1050/2304 [I 2022-10-09 20:57:18] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:42 [I 2022-10-09 20:57:18] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00298 [I 2022-10-09 20:57:18] ...trap/engines/engine.py.230: loss: 0.18912 [I 2022-10-09 20:57:18] ...optimizers/trijoint.py.151: total_norm: 11.223531 [I 2022-10-09 20:57:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 1060/2304 [I 2022-10-09 20:57:26] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:35 [I 2022-10-09 20:57:26] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00308 [I 2022-10-09 20:57:26] ...trap/engines/engine.py.230: loss: 0.15890 [I 2022-10-09 20:57:26] ...optimizers/trijoint.py.151: total_norm: 13.800618 [I 2022-10-09 20:57:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 1070/2304 [I 2022-10-09 20:57:35] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:25 [I 2022-10-09 20:57:35] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00300 [I 2022-10-09 20:57:35] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-09 20:57:35] ...optimizers/trijoint.py.151: total_norm: 8.798777 [I 2022-10-09 20:57:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1080/2304 [I 2022-10-09 20:57:43] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:23 [I 2022-10-09 20:57:43] ...trap/engines/engine.py.229: process: 0.85525 | load: 0.00299 [I 2022-10-09 20:57:43] ...trap/engines/engine.py.230: loss: 0.15244 [I 2022-10-09 20:57:43] ...optimizers/trijoint.py.151: total_norm: 8.763248 [I 2022-10-09 20:57:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 1090/2304 [I 2022-10-09 20:57:52] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:09 [I 2022-10-09 20:57:52] ...trap/engines/engine.py.229: process: 0.85660 | load: 0.00306 [I 2022-10-09 20:57:52] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-09 20:57:52] ...optimizers/trijoint.py.151: total_norm: 5.955238 [I 2022-10-09 20:58:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1100/2304 [I 2022-10-09 20:58:00] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:07 [I 2022-10-09 20:58:00] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00306 [I 2022-10-09 20:58:00] ...trap/engines/engine.py.230: loss: 0.18876 [I 2022-10-09 20:58:00] ...optimizers/trijoint.py.151: total_norm: 39.651070 [I 2022-10-09 20:58:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 1110/2304 [I 2022-10-09 20:58:09] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:58 [I 2022-10-09 20:58:09] ...trap/engines/engine.py.229: process: 0.83035 | load: 0.00311 [I 2022-10-09 20:58:09] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-09 20:58:09] ...optimizers/trijoint.py.151: total_norm: 10.309167 [I 2022-10-09 20:58:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 1120/2304 [I 2022-10-09 20:58:18] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:49 [I 2022-10-09 20:58:18] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00310 [I 2022-10-09 20:58:18] ...trap/engines/engine.py.230: loss: 0.22249 [I 2022-10-09 20:58:18] ...optimizers/trijoint.py.151: total_norm: 34.574394 [I 2022-10-09 20:58:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 1130/2304 [I 2022-10-09 20:58:26] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:40 [I 2022-10-09 20:58:26] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00299 [I 2022-10-09 20:58:26] ...trap/engines/engine.py.230: loss: 0.11710 [I 2022-10-09 20:58:26] ...optimizers/trijoint.py.151: total_norm: 9.509802 [I 2022-10-09 20:58:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 1140/2304 [I 2022-10-09 20:58:35] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:22 [I 2022-10-09 20:58:35] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00308 [I 2022-10-09 20:58:35] ...trap/engines/engine.py.230: loss: 0.17222 [I 2022-10-09 20:58:35] ...optimizers/trijoint.py.151: total_norm: 19.306496 [I 2022-10-09 20:58:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1150/2304 [I 2022-10-09 20:58:43] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:20 [I 2022-10-09 20:58:43] ...trap/engines/engine.py.229: process: 0.84101 | load: 0.00306 [I 2022-10-09 20:58:43] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-09 20:58:43] ...optimizers/trijoint.py.151: total_norm: 12.884474 [I 2022-10-09 20:58:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 1160/2304 [I 2022-10-09 20:58:52] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:12 [I 2022-10-09 20:58:52] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00305 [I 2022-10-09 20:58:52] ...trap/engines/engine.py.230: loss: 0.21096 [I 2022-10-09 20:58:52] ...optimizers/trijoint.py.151: total_norm: 41.213921 [I 2022-10-09 20:59:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1170/2304 [I 2022-10-09 20:59:00] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:03 [I 2022-10-09 20:59:00] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00313 [I 2022-10-09 20:59:00] ...trap/engines/engine.py.230: loss: 0.15530 [I 2022-10-09 20:59:00] ...optimizers/trijoint.py.151: total_norm: 14.322824 [I 2022-10-09 20:59:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 1180/2304 [I 2022-10-09 20:59:09] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:55 [I 2022-10-09 20:59:09] ...trap/engines/engine.py.229: process: 0.83379 | load: 0.00297 [I 2022-10-09 20:59:09] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-09 20:59:09] ...optimizers/trijoint.py.151: total_norm: 12.573274 [I 2022-10-09 20:59:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 1190/2304 [I 2022-10-09 20:59:17] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:44 [I 2022-10-09 20:59:17] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00301 [I 2022-10-09 20:59:17] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-09 20:59:17] ...optimizers/trijoint.py.151: total_norm: 20.943279 [I 2022-10-09 20:59:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 1200/2304 [I 2022-10-09 20:59:26] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:43 [I 2022-10-09 20:59:26] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00304 [I 2022-10-09 20:59:26] ...trap/engines/engine.py.230: loss: 0.14988 [I 2022-10-09 20:59:26] ...optimizers/trijoint.py.151: total_norm: 17.155365 [I 2022-10-09 20:59:34] ...trap/engines/engine.py.224: train: epoch 54 | batch 1210/2304 [I 2022-10-09 20:59:34] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:19 [I 2022-10-09 20:59:34] ...trap/engines/engine.py.229: process: 0.83222 | load: 0.00310 [I 2022-10-09 20:59:34] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-09 20:59:34] ...optimizers/trijoint.py.151: total_norm: 23.173048 [I 2022-10-09 20:59:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1220/2304 [I 2022-10-09 20:59:43] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:15 [I 2022-10-09 20:59:43] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00294 [I 2022-10-09 20:59:43] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-09 20:59:43] ...optimizers/trijoint.py.151: total_norm: 13.418741 [I 2022-10-09 20:59:51] ...trap/engines/engine.py.224: train: epoch 54 | batch 1230/2304 [I 2022-10-09 20:59:51] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:15 [I 2022-10-09 20:59:51] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00305 [I 2022-10-09 20:59:51] ...trap/engines/engine.py.230: loss: 0.19746 [I 2022-10-09 20:59:51] ...optimizers/trijoint.py.151: total_norm: 10.412850 [I 2022-10-09 21:00:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1240/2304 [I 2022-10-09 21:00:00] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:59 [I 2022-10-09 21:00:00] ...trap/engines/engine.py.229: process: 0.83824 | load: 0.00311 [I 2022-10-09 21:00:00] ...trap/engines/engine.py.230: loss: 0.17278 [I 2022-10-09 21:00:00] ...optimizers/trijoint.py.151: total_norm: 20.209337 [I 2022-10-09 21:00:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 1250/2304 [I 2022-10-09 21:00:08] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:56 [I 2022-10-09 21:00:08] ...trap/engines/engine.py.229: process: 0.86641 | load: 0.00295 [I 2022-10-09 21:00:08] ...trap/engines/engine.py.230: loss: 0.16683 [I 2022-10-09 21:00:08] ...optimizers/trijoint.py.151: total_norm: 12.864027 [I 2022-10-09 21:00:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 1260/2304 [I 2022-10-09 21:00:17] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:46 [I 2022-10-09 21:00:17] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00313 [I 2022-10-09 21:00:17] ...trap/engines/engine.py.230: loss: 0.07940 [I 2022-10-09 21:00:17] ...optimizers/trijoint.py.151: total_norm: 10.313931 [I 2022-10-09 21:00:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1270/2304 [I 2022-10-09 21:00:25] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:31 [I 2022-10-09 21:00:25] ...trap/engines/engine.py.229: process: 0.83546 | load: 0.00296 [I 2022-10-09 21:00:25] ...trap/engines/engine.py.230: loss: 0.16926 [I 2022-10-09 21:00:25] ...optimizers/trijoint.py.151: total_norm: 22.695152 [I 2022-10-09 21:00:34] ...trap/engines/engine.py.224: train: epoch 54 | batch 1280/2304 [I 2022-10-09 21:00:34] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:27 [I 2022-10-09 21:00:34] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00298 [I 2022-10-09 21:00:34] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-09 21:00:34] ...optimizers/trijoint.py.151: total_norm: 13.674473 [I 2022-10-09 21:00:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1290/2304 [I 2022-10-09 21:00:42] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:23 [I 2022-10-09 21:00:42] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00308 [I 2022-10-09 21:00:42] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-09 21:00:42] ...optimizers/trijoint.py.151: total_norm: 17.717276 [I 2022-10-09 21:00:51] ...trap/engines/engine.py.224: train: epoch 54 | batch 1300/2304 [I 2022-10-09 21:00:51] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:14 [I 2022-10-09 21:00:51] ...trap/engines/engine.py.229: process: 0.84534 | load: 0.00308 [I 2022-10-09 21:00:51] ...trap/engines/engine.py.230: loss: 0.14308 [I 2022-10-09 21:00:51] ...optimizers/trijoint.py.151: total_norm: 24.696749 [I 2022-10-09 21:00:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1310/2304 [I 2022-10-09 21:00:59] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:04 [I 2022-10-09 21:00:59] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00313 [I 2022-10-09 21:00:59] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-09 21:00:59] ...optimizers/trijoint.py.151: total_norm: 10.709175 [I 2022-10-09 21:01:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 1320/2304 [I 2022-10-09 21:01:08] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:55 [I 2022-10-09 21:01:08] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00302 [I 2022-10-09 21:01:08] ...trap/engines/engine.py.230: loss: 0.10568 [I 2022-10-09 21:01:08] ...optimizers/trijoint.py.151: total_norm: 11.531985 [I 2022-10-09 21:01:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1330/2304 [I 2022-10-09 21:01:16] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:42 [I 2022-10-09 21:01:16] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00309 [I 2022-10-09 21:01:16] ...trap/engines/engine.py.230: loss: 0.22970 [I 2022-10-09 21:01:16] ...optimizers/trijoint.py.151: total_norm: 31.952530 [I 2022-10-09 21:01:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1340/2304 [I 2022-10-09 21:01:25] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:47 [I 2022-10-09 21:01:25] ...trap/engines/engine.py.229: process: 0.87990 | load: 0.00308 [I 2022-10-09 21:01:25] ...trap/engines/engine.py.230: loss: 0.17402 [I 2022-10-09 21:01:25] ...optimizers/trijoint.py.151: total_norm: 21.612885 [I 2022-10-09 21:01:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1350/2304 [I 2022-10-09 21:01:33] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:29 [I 2022-10-09 21:01:33] ...trap/engines/engine.py.229: process: 0.84651 | load: 0.00304 [I 2022-10-09 21:01:33] ...trap/engines/engine.py.230: loss: 0.15283 [I 2022-10-09 21:01:33] ...optimizers/trijoint.py.151: total_norm: 18.408409 [I 2022-10-09 21:01:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1360/2304 [I 2022-10-09 21:01:42] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:27 [I 2022-10-09 21:01:42] ...trap/engines/engine.py.229: process: 0.86232 | load: 0.00308 [I 2022-10-09 21:01:42] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-09 21:01:42] ...optimizers/trijoint.py.151: total_norm: 16.774479 [I 2022-10-09 21:01:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1370/2304 [I 2022-10-09 21:01:50] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:11 [I 2022-10-09 21:01:50] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00306 [I 2022-10-09 21:01:50] ...trap/engines/engine.py.230: loss: 0.18459 [I 2022-10-09 21:01:50] ...optimizers/trijoint.py.151: total_norm: 13.296588 [I 2022-10-09 21:01:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1380/2304 [I 2022-10-09 21:01:59] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:05 [I 2022-10-09 21:01:59] ...trap/engines/engine.py.229: process: 0.85623 | load: 0.00316 [I 2022-10-09 21:01:59] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-09 21:01:59] ...optimizers/trijoint.py.151: total_norm: 9.070780 [I 2022-10-09 21:02:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1390/2304 [I 2022-10-09 21:02:07] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:03 [I 2022-10-09 21:02:07] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00313 [I 2022-10-09 21:02:07] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-09 21:02:07] ...optimizers/trijoint.py.151: total_norm: 11.065019 [I 2022-10-09 21:02:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1400/2304 [I 2022-10-09 21:02:16] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:55 [I 2022-10-09 21:02:16] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00310 [I 2022-10-09 21:02:16] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-09 21:02:16] ...optimizers/trijoint.py.151: total_norm: 9.500286 [I 2022-10-09 21:02:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1410/2304 [I 2022-10-09 21:02:25] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:12:44 [I 2022-10-09 21:02:25] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00312 [I 2022-10-09 21:02:25] ...trap/engines/engine.py.230: loss: 0.10641 [I 2022-10-09 21:02:25] ...optimizers/trijoint.py.151: total_norm: 34.829689 [I 2022-10-09 21:02:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1420/2304 [I 2022-10-09 21:02:33] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:35 [I 2022-10-09 21:02:33] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00312 [I 2022-10-09 21:02:33] ...trap/engines/engine.py.230: loss: 0.13551 [I 2022-10-09 21:02:33] ...optimizers/trijoint.py.151: total_norm: 13.450830 [I 2022-10-09 21:02:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1430/2304 [I 2022-10-09 21:02:42] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:29 [I 2022-10-09 21:02:42] ...trap/engines/engine.py.229: process: 0.86509 | load: 0.00318 [I 2022-10-09 21:02:42] ...trap/engines/engine.py.230: loss: 0.16509 [I 2022-10-09 21:02:42] ...optimizers/trijoint.py.151: total_norm: 34.906345 [I 2022-10-09 21:02:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1440/2304 [I 2022-10-09 21:02:50] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:14 [I 2022-10-09 21:02:50] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00309 [I 2022-10-09 21:02:50] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-09 21:02:50] ...optimizers/trijoint.py.151: total_norm: 25.514006 [I 2022-10-09 21:02:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1450/2304 [I 2022-10-09 21:02:59] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:02 [I 2022-10-09 21:02:59] ...trap/engines/engine.py.229: process: 0.84603 | load: 0.00311 [I 2022-10-09 21:02:59] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-09 21:02:59] ...optimizers/trijoint.py.151: total_norm: 20.336800 [I 2022-10-09 21:03:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1460/2304 [I 2022-10-09 21:03:07] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:11:59 [I 2022-10-09 21:03:07] ...trap/engines/engine.py.229: process: 0.85158 | load: 0.00312 [I 2022-10-09 21:03:07] ...trap/engines/engine.py.230: loss: 0.14852 [I 2022-10-09 21:03:07] ...optimizers/trijoint.py.151: total_norm: 26.416027 [I 2022-10-09 21:03:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1470/2304 [I 2022-10-09 21:03:16] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:54 [I 2022-10-09 21:03:16] ...trap/engines/engine.py.229: process: 0.85898 | load: 0.00313 [I 2022-10-09 21:03:16] ...trap/engines/engine.py.230: loss: 0.16223 [I 2022-10-09 21:03:16] ...optimizers/trijoint.py.151: total_norm: 8.437238 [I 2022-10-09 21:03:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 1480/2304 [I 2022-10-09 21:03:24] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:45 [I 2022-10-09 21:03:24] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00314 [I 2022-10-09 21:03:24] ...trap/engines/engine.py.230: loss: 0.10892 [I 2022-10-09 21:03:24] ...optimizers/trijoint.py.151: total_norm: 15.614799 [I 2022-10-09 21:03:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1490/2304 [I 2022-10-09 21:03:33] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:37 [I 2022-10-09 21:03:33] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00314 [I 2022-10-09 21:03:33] ...trap/engines/engine.py.230: loss: 0.21632 [I 2022-10-09 21:03:33] ...optimizers/trijoint.py.151: total_norm: 18.270033 [I 2022-10-09 21:03:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1500/2304 [I 2022-10-09 21:03:42] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:28 [I 2022-10-09 21:03:42] ...trap/engines/engine.py.229: process: 0.85624 | load: 0.00317 [I 2022-10-09 21:03:42] ...trap/engines/engine.py.230: loss: 0.20407 [I 2022-10-09 21:03:42] ...optimizers/trijoint.py.151: total_norm: 22.591919 [I 2022-10-09 21:03:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1510/2304 [I 2022-10-09 21:03:50] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:19 [I 2022-10-09 21:03:50] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00312 [I 2022-10-09 21:03:50] ...trap/engines/engine.py.230: loss: 0.19473 [I 2022-10-09 21:03:50] ...optimizers/trijoint.py.151: total_norm: 10.350476 [I 2022-10-09 21:03:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1520/2304 [I 2022-10-09 21:03:59] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:08 [I 2022-10-09 21:03:59] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00308 [I 2022-10-09 21:03:59] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-09 21:03:59] ...optimizers/trijoint.py.151: total_norm: 11.893418 [I 2022-10-09 21:04:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1530/2304 [I 2022-10-09 21:04:07] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:01 [I 2022-10-09 21:04:07] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00320 [I 2022-10-09 21:04:07] ...trap/engines/engine.py.230: loss: 0.18550 [I 2022-10-09 21:04:07] ...optimizers/trijoint.py.151: total_norm: 23.913290 [I 2022-10-09 21:04:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1540/2304 [I 2022-10-09 21:04:16] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:53 [I 2022-10-09 21:04:16] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00311 [I 2022-10-09 21:04:16] ...trap/engines/engine.py.230: loss: 0.17066 [I 2022-10-09 21:04:16] ...optimizers/trijoint.py.151: total_norm: 17.685392 [I 2022-10-09 21:04:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1550/2304 [I 2022-10-09 21:04:25] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:48 [I 2022-10-09 21:04:25] ...trap/engines/engine.py.229: process: 0.88030 | load: 0.00309 [I 2022-10-09 21:04:25] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-09 21:04:25] ...optimizers/trijoint.py.151: total_norm: 14.164977 [I 2022-10-09 21:04:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1560/2304 [I 2022-10-09 21:04:33] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:34 [I 2022-10-09 21:04:33] ...trap/engines/engine.py.229: process: 0.85252 | load: 0.00316 [I 2022-10-09 21:04:33] ...trap/engines/engine.py.230: loss: 0.21795 [I 2022-10-09 21:04:33] ...optimizers/trijoint.py.151: total_norm: 43.150482 [I 2022-10-09 21:04:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1570/2304 [I 2022-10-09 21:04:42] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:30 [I 2022-10-09 21:04:42] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00312 [I 2022-10-09 21:04:42] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-09 21:04:42] ...optimizers/trijoint.py.151: total_norm: 19.838650 [I 2022-10-09 21:04:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1580/2304 [I 2022-10-09 21:04:50] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:17 [I 2022-10-09 21:04:50] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00312 [I 2022-10-09 21:04:50] ...trap/engines/engine.py.230: loss: 0.19952 [I 2022-10-09 21:04:50] ...optimizers/trijoint.py.151: total_norm: 8.373036 [I 2022-10-09 21:04:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1590/2304 [I 2022-10-09 21:04:59] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:13 [I 2022-10-09 21:04:59] ...trap/engines/engine.py.229: process: 0.86341 | load: 0.00312 [I 2022-10-09 21:04:59] ...trap/engines/engine.py.230: loss: 0.11755 [I 2022-10-09 21:04:59] ...optimizers/trijoint.py.151: total_norm: 41.573582 [I 2022-10-09 21:05:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1600/2304 [I 2022-10-09 21:05:07] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:03 [I 2022-10-09 21:05:07] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00312 [I 2022-10-09 21:05:07] ...trap/engines/engine.py.230: loss: 0.19358 [I 2022-10-09 21:05:07] ...optimizers/trijoint.py.151: total_norm: 13.977759 [I 2022-10-09 21:05:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1610/2304 [I 2022-10-09 21:05:16] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:55 [I 2022-10-09 21:05:16] ...trap/engines/engine.py.229: process: 0.86308 | load: 0.00310 [I 2022-10-09 21:05:16] ...trap/engines/engine.py.230: loss: 0.12240 [I 2022-10-09 21:05:16] ...optimizers/trijoint.py.151: total_norm: 9.263571 [I 2022-10-09 21:05:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1620/2304 [I 2022-10-09 21:05:25] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:09:45 [I 2022-10-09 21:05:25] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00315 [I 2022-10-09 21:05:25] ...trap/engines/engine.py.230: loss: 0.18866 [I 2022-10-09 21:05:25] ...optimizers/trijoint.py.151: total_norm: 14.886902 [I 2022-10-09 21:05:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1630/2304 [I 2022-10-09 21:05:33] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:35 [I 2022-10-09 21:05:33] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00314 [I 2022-10-09 21:05:33] ...trap/engines/engine.py.230: loss: 0.28304 [I 2022-10-09 21:05:33] ...optimizers/trijoint.py.151: total_norm: 29.680660 [I 2022-10-09 21:05:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1640/2304 [I 2022-10-09 21:05:42] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:27 [I 2022-10-09 21:05:42] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00311 [I 2022-10-09 21:05:42] ...trap/engines/engine.py.230: loss: 0.10333 [I 2022-10-09 21:05:42] ...optimizers/trijoint.py.151: total_norm: 7.820234 [I 2022-10-09 21:05:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1650/2304 [I 2022-10-09 21:05:50] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:15 [I 2022-10-09 21:05:50] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00311 [I 2022-10-09 21:05:50] ...trap/engines/engine.py.230: loss: 0.16337 [I 2022-10-09 21:05:50] ...optimizers/trijoint.py.151: total_norm: 24.866972 [I 2022-10-09 21:05:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1660/2304 [I 2022-10-09 21:05:59] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:10 [I 2022-10-09 21:05:59] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00325 [I 2022-10-09 21:05:59] ...trap/engines/engine.py.230: loss: 0.21533 [I 2022-10-09 21:05:59] ...optimizers/trijoint.py.151: total_norm: 19.072639 [I 2022-10-09 21:06:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1670/2304 [I 2022-10-09 21:06:07] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:03 [I 2022-10-09 21:06:07] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00319 [I 2022-10-09 21:06:07] ...trap/engines/engine.py.230: loss: 0.18779 [I 2022-10-09 21:06:07] ...optimizers/trijoint.py.151: total_norm: 14.874310 [I 2022-10-09 21:06:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1680/2304 [I 2022-10-09 21:06:16] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:51 [I 2022-10-09 21:06:16] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00310 [I 2022-10-09 21:06:16] ...trap/engines/engine.py.230: loss: 0.13422 [I 2022-10-09 21:06:16] ...optimizers/trijoint.py.151: total_norm: 19.679535 [I 2022-10-09 21:06:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1690/2304 [I 2022-10-09 21:06:25] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:48 [I 2022-10-09 21:06:25] ...trap/engines/engine.py.229: process: 0.87983 | load: 0.00341 [I 2022-10-09 21:06:25] ...trap/engines/engine.py.230: loss: 0.12518 [I 2022-10-09 21:06:25] ...optimizers/trijoint.py.151: total_norm: 13.312970 [I 2022-10-09 21:06:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1700/2304 [I 2022-10-09 21:06:33] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:33 [I 2022-10-09 21:06:33] ...trap/engines/engine.py.229: process: 0.85772 | load: 0.00318 [I 2022-10-09 21:06:33] ...trap/engines/engine.py.230: loss: 0.15029 [I 2022-10-09 21:06:33] ...optimizers/trijoint.py.151: total_norm: 8.431964 [I 2022-10-09 21:06:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1710/2304 [I 2022-10-09 21:06:42] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:25 [I 2022-10-09 21:06:42] ...trap/engines/engine.py.229: process: 0.86020 | load: 0.00312 [I 2022-10-09 21:06:42] ...trap/engines/engine.py.230: loss: 0.18614 [I 2022-10-09 21:06:42] ...optimizers/trijoint.py.151: total_norm: 18.549015 [I 2022-10-09 21:06:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1720/2304 [I 2022-10-09 21:06:50] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:20 [I 2022-10-09 21:06:50] ...trap/engines/engine.py.229: process: 0.86846 | load: 0.00313 [I 2022-10-09 21:06:50] ...trap/engines/engine.py.230: loss: 0.18293 [I 2022-10-09 21:06:50] ...optimizers/trijoint.py.151: total_norm: 13.743467 [I 2022-10-09 21:06:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1730/2304 [I 2022-10-09 21:06:59] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:09 [I 2022-10-09 21:06:59] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00311 [I 2022-10-09 21:06:59] ...trap/engines/engine.py.230: loss: 0.15022 [I 2022-10-09 21:06:59] ...optimizers/trijoint.py.151: total_norm: 21.995033 [I 2022-10-09 21:07:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1740/2304 [I 2022-10-09 21:07:07] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:06 [I 2022-10-09 21:07:07] ...trap/engines/engine.py.229: process: 0.87731 | load: 0.00312 [I 2022-10-09 21:07:07] ...trap/engines/engine.py.230: loss: 0.24522 [I 2022-10-09 21:07:07] ...optimizers/trijoint.py.151: total_norm: 30.613554 [I 2022-10-09 21:07:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1750/2304 [I 2022-10-09 21:07:16] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:07:56 [I 2022-10-09 21:07:16] ...trap/engines/engine.py.229: process: 0.86336 | load: 0.00313 [I 2022-10-09 21:07:16] ...trap/engines/engine.py.230: loss: 0.11703 [I 2022-10-09 21:07:16] ...optimizers/trijoint.py.151: total_norm: 13.561336 [I 2022-10-09 21:07:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1760/2304 [I 2022-10-09 21:07:25] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:46 [I 2022-10-09 21:07:25] ...trap/engines/engine.py.229: process: 0.86354 | load: 0.00315 [I 2022-10-09 21:07:25] ...trap/engines/engine.py.230: loss: 0.18655 [I 2022-10-09 21:07:25] ...optimizers/trijoint.py.151: total_norm: 16.523857 [I 2022-10-09 21:07:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1770/2304 [I 2022-10-09 21:07:33] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:34 [I 2022-10-09 21:07:33] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00302 [I 2022-10-09 21:07:33] ...trap/engines/engine.py.230: loss: 0.20060 [I 2022-10-09 21:07:33] ...optimizers/trijoint.py.151: total_norm: 10.929418 [I 2022-10-09 21:07:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1780/2304 [I 2022-10-09 21:07:42] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:24 [I 2022-10-09 21:07:42] ...trap/engines/engine.py.229: process: 0.83885 | load: 0.00308 [I 2022-10-09 21:07:42] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-09 21:07:42] ...optimizers/trijoint.py.151: total_norm: 33.168804 [I 2022-10-09 21:07:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1790/2304 [I 2022-10-09 21:07:50] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:15 [I 2022-10-09 21:07:50] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00313 [I 2022-10-09 21:07:50] ...trap/engines/engine.py.230: loss: 0.12918 [I 2022-10-09 21:07:50] ...optimizers/trijoint.py.151: total_norm: 11.966296 [I 2022-10-09 21:07:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 1800/2304 [I 2022-10-09 21:07:59] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:10 [I 2022-10-09 21:07:59] ...trap/engines/engine.py.229: process: 0.85679 | load: 0.00302 [I 2022-10-09 21:07:59] ...trap/engines/engine.py.230: loss: 0.15631 [I 2022-10-09 21:07:59] ...optimizers/trijoint.py.151: total_norm: 10.847885 [I 2022-10-09 21:08:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1810/2304 [I 2022-10-09 21:08:07] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:06:59 [I 2022-10-09 21:08:07] ...trap/engines/engine.py.229: process: 0.83255 | load: 0.00309 [I 2022-10-09 21:08:07] ...trap/engines/engine.py.230: loss: 0.25766 [I 2022-10-09 21:08:07] ...optimizers/trijoint.py.151: total_norm: 17.834053 [I 2022-10-09 21:08:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1820/2304 [I 2022-10-09 21:08:16] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:50 [I 2022-10-09 21:08:16] ...trap/engines/engine.py.229: process: 0.86512 | load: 0.00298 [I 2022-10-09 21:08:16] ...trap/engines/engine.py.230: loss: 0.18711 [I 2022-10-09 21:08:16] ...optimizers/trijoint.py.151: total_norm: 26.100626 [I 2022-10-09 21:08:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 1830/2304 [I 2022-10-09 21:08:24] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:41 [I 2022-10-09 21:08:24] ...trap/engines/engine.py.229: process: 0.85617 | load: 0.00300 [I 2022-10-09 21:08:24] ...trap/engines/engine.py.230: loss: 0.14415 [I 2022-10-09 21:08:24] ...optimizers/trijoint.py.151: total_norm: 10.486471 [I 2022-10-09 21:08:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1840/2304 [I 2022-10-09 21:08:33] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:33 [I 2022-10-09 21:08:33] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00298 [I 2022-10-09 21:08:33] ...trap/engines/engine.py.230: loss: 0.10846 [I 2022-10-09 21:08:33] ...optimizers/trijoint.py.151: total_norm: 18.128372 [I 2022-10-09 21:08:41] ...trap/engines/engine.py.224: train: epoch 54 | batch 1850/2304 [I 2022-10-09 21:08:41] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:27 [I 2022-10-09 21:08:41] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00369 [I 2022-10-09 21:08:41] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-09 21:08:41] ...optimizers/trijoint.py.151: total_norm: 17.433067 [I 2022-10-09 21:08:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1860/2304 [I 2022-10-09 21:08:50] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:18 [I 2022-10-09 21:08:50] ...trap/engines/engine.py.229: process: 0.86610 | load: 0.00302 [I 2022-10-09 21:08:50] ...trap/engines/engine.py.230: loss: 0.11206 [I 2022-10-09 21:08:50] ...optimizers/trijoint.py.151: total_norm: 8.975533 [I 2022-10-09 21:08:58] ...trap/engines/engine.py.224: train: epoch 54 | batch 1870/2304 [I 2022-10-09 21:08:58] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:07 [I 2022-10-09 21:08:58] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00303 [I 2022-10-09 21:08:58] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-09 21:08:58] ...optimizers/trijoint.py.151: total_norm: 14.087373 [I 2022-10-09 21:09:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1880/2304 [I 2022-10-09 21:09:07] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:05:57 [I 2022-10-09 21:09:07] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00311 [I 2022-10-09 21:09:07] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-09 21:09:07] ...optimizers/trijoint.py.151: total_norm: 11.711334 [I 2022-10-09 21:09:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 1890/2304 [I 2022-10-09 21:09:15] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:05:51 [I 2022-10-09 21:09:15] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00307 [I 2022-10-09 21:09:15] ...trap/engines/engine.py.230: loss: 0.13678 [I 2022-10-09 21:09:15] ...optimizers/trijoint.py.151: total_norm: 12.620133 [I 2022-10-09 21:09:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 1900/2304 [I 2022-10-09 21:09:24] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:41 [I 2022-10-09 21:09:24] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00304 [I 2022-10-09 21:09:24] ...trap/engines/engine.py.230: loss: 0.10712 [I 2022-10-09 21:09:24] ...optimizers/trijoint.py.151: total_norm: 9.455943 [I 2022-10-09 21:09:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 1910/2304 [I 2022-10-09 21:09:32] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:34 [I 2022-10-09 21:09:32] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00316 [I 2022-10-09 21:09:32] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-09 21:09:32] ...optimizers/trijoint.py.151: total_norm: 17.993967 [I 2022-10-09 21:09:41] ...trap/engines/engine.py.224: train: epoch 54 | batch 1920/2304 [I 2022-10-09 21:09:41] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:25 [I 2022-10-09 21:09:41] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00302 [I 2022-10-09 21:09:41] ...trap/engines/engine.py.230: loss: 0.21425 [I 2022-10-09 21:09:41] ...optimizers/trijoint.py.151: total_norm: 88.187927 [I 2022-10-09 21:09:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 1930/2304 [I 2022-10-09 21:09:49] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:17 [I 2022-10-09 21:09:49] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00309 [I 2022-10-09 21:09:49] ...trap/engines/engine.py.230: loss: 0.22495 [I 2022-10-09 21:09:49] ...optimizers/trijoint.py.151: total_norm: 29.668530 [I 2022-10-09 21:09:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 1940/2304 [I 2022-10-09 21:09:57] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:06 [I 2022-10-09 21:09:57] ...trap/engines/engine.py.229: process: 0.84207 | load: 0.00298 [I 2022-10-09 21:09:57] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-09 21:09:57] ...optimizers/trijoint.py.151: total_norm: 21.602427 [I 2022-10-09 21:10:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 1950/2304 [I 2022-10-09 21:10:06] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:05:01 [I 2022-10-09 21:10:06] ...trap/engines/engine.py.229: process: 0.85680 | load: 0.00306 [I 2022-10-09 21:10:06] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-09 21:10:06] ...optimizers/trijoint.py.151: total_norm: 12.583115 [I 2022-10-09 21:10:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 1960/2304 [I 2022-10-09 21:10:15] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:52 [I 2022-10-09 21:10:15] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00305 [I 2022-10-09 21:10:15] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-09 21:10:15] ...optimizers/trijoint.py.151: total_norm: 19.767651 [I 2022-10-09 21:10:23] ...trap/engines/engine.py.224: train: epoch 54 | batch 1970/2304 [I 2022-10-09 21:10:23] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:04:43 [I 2022-10-09 21:10:23] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00300 [I 2022-10-09 21:10:23] ...trap/engines/engine.py.230: loss: 0.15206 [I 2022-10-09 21:10:23] ...optimizers/trijoint.py.151: total_norm: 18.339052 [I 2022-10-09 21:10:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 1980/2304 [I 2022-10-09 21:10:32] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:34 [I 2022-10-09 21:10:32] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00301 [I 2022-10-09 21:10:32] ...trap/engines/engine.py.230: loss: 0.17092 [I 2022-10-09 21:10:32] ...optimizers/trijoint.py.151: total_norm: 11.491810 [I 2022-10-09 21:10:40] ...trap/engines/engine.py.224: train: epoch 54 | batch 1990/2304 [I 2022-10-09 21:10:40] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:27 [I 2022-10-09 21:10:40] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00302 [I 2022-10-09 21:10:40] ...trap/engines/engine.py.230: loss: 0.26773 [I 2022-10-09 21:10:40] ...optimizers/trijoint.py.151: total_norm: 20.728800 [I 2022-10-09 21:10:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 2000/2304 [I 2022-10-09 21:10:49] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:17 [I 2022-10-09 21:10:49] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00307 [I 2022-10-09 21:10:49] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-09 21:10:49] ...optimizers/trijoint.py.151: total_norm: 11.248590 [I 2022-10-09 21:10:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 2010/2304 [I 2022-10-09 21:10:57] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:09 [I 2022-10-09 21:10:57] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00309 [I 2022-10-09 21:10:57] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-09 21:10:57] ...optimizers/trijoint.py.151: total_norm: 16.296618 [I 2022-10-09 21:11:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 2020/2304 [I 2022-10-09 21:11:06] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:01 [I 2022-10-09 21:11:06] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00307 [I 2022-10-09 21:11:06] ...trap/engines/engine.py.230: loss: 0.14534 [I 2022-10-09 21:11:06] ...optimizers/trijoint.py.151: total_norm: 14.379549 [I 2022-10-09 21:11:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 2030/2304 [I 2022-10-09 21:11:14] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:03:51 [I 2022-10-09 21:11:14] ...trap/engines/engine.py.229: process: 0.85299 | load: 0.00310 [I 2022-10-09 21:11:14] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-09 21:11:14] ...optimizers/trijoint.py.151: total_norm: 12.399240 [I 2022-10-09 21:11:23] ...trap/engines/engine.py.224: train: epoch 54 | batch 2040/2304 [I 2022-10-09 21:11:23] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:44 [I 2022-10-09 21:11:23] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00309 [I 2022-10-09 21:11:23] ...trap/engines/engine.py.230: loss: 0.13151 [I 2022-10-09 21:11:23] ...optimizers/trijoint.py.151: total_norm: 15.013344 [I 2022-10-09 21:11:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 2050/2304 [I 2022-10-09 21:11:31] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:35 [I 2022-10-09 21:11:31] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00310 [I 2022-10-09 21:11:31] ...trap/engines/engine.py.230: loss: 0.17622 [I 2022-10-09 21:11:31] ...optimizers/trijoint.py.151: total_norm: 27.638309 [I 2022-10-09 21:11:40] ...trap/engines/engine.py.224: train: epoch 54 | batch 2060/2304 [I 2022-10-09 21:11:40] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:28 [I 2022-10-09 21:11:40] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00311 [I 2022-10-09 21:11:40] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-09 21:11:40] ...optimizers/trijoint.py.151: total_norm: 12.145597 [I 2022-10-09 21:11:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 2070/2304 [I 2022-10-09 21:11:48] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:17 [I 2022-10-09 21:11:48] ...trap/engines/engine.py.229: process: 0.83503 | load: 0.00306 [I 2022-10-09 21:11:48] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-09 21:11:48] ...optimizers/trijoint.py.151: total_norm: 7.950611 [I 2022-10-09 21:11:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 2080/2304 [I 2022-10-09 21:11:57] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:12 [I 2022-10-09 21:11:57] ...trap/engines/engine.py.229: process: 0.87017 | load: 0.00306 [I 2022-10-09 21:11:57] ...trap/engines/engine.py.230: loss: 0.25739 [I 2022-10-09 21:11:57] ...optimizers/trijoint.py.151: total_norm: 18.708010 [I 2022-10-09 21:12:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 2090/2304 [I 2022-10-09 21:12:05] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:02 [I 2022-10-09 21:12:05] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00311 [I 2022-10-09 21:12:05] ...trap/engines/engine.py.230: loss: 0.14990 [I 2022-10-09 21:12:05] ...optimizers/trijoint.py.151: total_norm: 16.935463 [I 2022-10-09 21:12:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 2100/2304 [I 2022-10-09 21:12:14] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:02:54 [I 2022-10-09 21:12:14] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00316 [I 2022-10-09 21:12:14] ...trap/engines/engine.py.230: loss: 0.12002 [I 2022-10-09 21:12:14] ...optimizers/trijoint.py.151: total_norm: 20.389954 [I 2022-10-09 21:12:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 2110/2304 [I 2022-10-09 21:12:22] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:45 [I 2022-10-09 21:12:22] ...trap/engines/engine.py.229: process: 0.84890 | load: 0.00310 [I 2022-10-09 21:12:22] ...trap/engines/engine.py.230: loss: 0.17077 [I 2022-10-09 21:12:22] ...optimizers/trijoint.py.151: total_norm: 30.364021 [I 2022-10-09 21:12:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 2120/2304 [I 2022-10-09 21:12:31] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:37 [I 2022-10-09 21:12:31] ...trap/engines/engine.py.229: process: 0.86695 | load: 0.00317 [I 2022-10-09 21:12:31] ...trap/engines/engine.py.230: loss: 0.14111 [I 2022-10-09 21:12:31] ...optimizers/trijoint.py.151: total_norm: 29.248737 [I 2022-10-09 21:12:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 2130/2304 [I 2022-10-09 21:12:39] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:28 [I 2022-10-09 21:12:39] ...trap/engines/engine.py.229: process: 0.86401 | load: 0.00308 [I 2022-10-09 21:12:39] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-09 21:12:39] ...optimizers/trijoint.py.151: total_norm: 14.424520 [I 2022-10-09 21:12:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 2140/2304 [I 2022-10-09 21:12:48] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:21 [I 2022-10-09 21:12:48] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00316 [I 2022-10-09 21:12:48] ...trap/engines/engine.py.230: loss: 0.27676 [I 2022-10-09 21:12:48] ...optimizers/trijoint.py.151: total_norm: 76.923828 [I 2022-10-09 21:12:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 2150/2304 [I 2022-10-09 21:12:57] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:10 [I 2022-10-09 21:12:57] ...trap/engines/engine.py.229: process: 0.83611 | load: 0.00313 [I 2022-10-09 21:12:57] ...trap/engines/engine.py.230: loss: 0.18697 [I 2022-10-09 21:12:57] ...optimizers/trijoint.py.151: total_norm: 23.165272 [I 2022-10-09 21:13:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 2160/2304 [I 2022-10-09 21:13:05] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:02 [I 2022-10-09 21:13:05] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00311 [I 2022-10-09 21:13:05] ...trap/engines/engine.py.230: loss: 0.16621 [I 2022-10-09 21:13:05] ...optimizers/trijoint.py.151: total_norm: 18.107595 [I 2022-10-09 21:13:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 2170/2304 [I 2022-10-09 21:13:14] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:01:53 [I 2022-10-09 21:13:14] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00313 [I 2022-10-09 21:13:14] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-09 21:13:14] ...optimizers/trijoint.py.151: total_norm: 10.899330 [I 2022-10-09 21:13:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 2180/2304 [I 2022-10-09 21:13:22] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:01:46 [I 2022-10-09 21:13:22] ...trap/engines/engine.py.229: process: 0.85899 | load: 0.00310 [I 2022-10-09 21:13:22] ...trap/engines/engine.py.230: loss: 0.10020 [I 2022-10-09 21:13:22] ...optimizers/trijoint.py.151: total_norm: 9.321245 [I 2022-10-09 21:13:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 2190/2304 [I 2022-10-09 21:13:31] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:37 [I 2022-10-09 21:13:31] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00311 [I 2022-10-09 21:13:31] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-09 21:13:31] ...optimizers/trijoint.py.151: total_norm: 60.494373 [I 2022-10-09 21:13:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 2200/2304 [I 2022-10-09 21:13:39] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:28 [I 2022-10-09 21:13:39] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00314 [I 2022-10-09 21:13:39] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-09 21:13:39] ...optimizers/trijoint.py.151: total_norm: 14.025975 [I 2022-10-09 21:13:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 2210/2304 [I 2022-10-09 21:13:48] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:20 [I 2022-10-09 21:13:48] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00311 [I 2022-10-09 21:13:48] ...trap/engines/engine.py.230: loss: 0.17051 [I 2022-10-09 21:13:48] ...optimizers/trijoint.py.151: total_norm: 35.774559 [I 2022-10-09 21:13:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 2220/2304 [I 2022-10-09 21:13:56] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:12 [I 2022-10-09 21:13:56] ...trap/engines/engine.py.229: process: 0.87586 | load: 0.00311 [I 2022-10-09 21:13:56] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-09 21:13:56] ...optimizers/trijoint.py.151: total_norm: 16.865253 [I 2022-10-09 21:14:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 2230/2304 [I 2022-10-09 21:14:05] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:02 [I 2022-10-09 21:14:05] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00340 [I 2022-10-09 21:14:05] ...trap/engines/engine.py.230: loss: 0.10572 [I 2022-10-09 21:14:05] ...optimizers/trijoint.py.151: total_norm: 12.979295 [I 2022-10-09 21:14:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 2240/2304 [I 2022-10-09 21:14:14] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:00:54 [I 2022-10-09 21:14:14] ...trap/engines/engine.py.229: process: 0.86338 | load: 0.00311 [I 2022-10-09 21:14:14] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-09 21:14:14] ...optimizers/trijoint.py.151: total_norm: 12.272464 [I 2022-10-09 21:14:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 2250/2304 [I 2022-10-09 21:14:22] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:46 [I 2022-10-09 21:14:22] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00310 [I 2022-10-09 21:14:22] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-09 21:14:22] ...optimizers/trijoint.py.151: total_norm: 40.538780 [I 2022-10-09 21:14:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 2260/2304 [I 2022-10-09 21:14:31] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:37 [I 2022-10-09 21:14:31] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00314 [I 2022-10-09 21:14:31] ...trap/engines/engine.py.230: loss: 0.17090 [I 2022-10-09 21:14:31] ...optimizers/trijoint.py.151: total_norm: 12.096423 [I 2022-10-09 21:14:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 2270/2304 [I 2022-10-09 21:14:39] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:29 [I 2022-10-09 21:14:39] ...trap/engines/engine.py.229: process: 0.86964 | load: 0.00311 [I 2022-10-09 21:14:39] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-09 21:14:39] ...optimizers/trijoint.py.151: total_norm: 20.721071 [I 2022-10-09 21:14:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 2280/2304 [I 2022-10-09 21:14:48] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:20 [I 2022-10-09 21:14:48] ...trap/engines/engine.py.229: process: 0.84106 | load: 0.00312 [I 2022-10-09 21:14:48] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-09 21:14:48] ...optimizers/trijoint.py.151: total_norm: 7.733586 [I 2022-10-09 21:14:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 2290/2304 [I 2022-10-09 21:14:56] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:11 [I 2022-10-09 21:14:56] ...trap/engines/engine.py.229: process: 0.83860 | load: 0.00304 [I 2022-10-09 21:14:56] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-09 21:14:56] ...optimizers/trijoint.py.151: total_norm: 17.251759 [I 2022-10-09 21:15:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 2300/2304 [I 2022-10-09 21:15:05] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:03 [I 2022-10-09 21:15:05] ...trap/engines/engine.py.229: process: 0.82949 | load: 0.00016 [I 2022-10-09 21:15:05] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-09 21:15:05] ...optimizers/trijoint.py.151: total_norm: 13.030449 [I 2022-10-09 21:15:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 2304/2304 [I 2022-10-09 21:15:08] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:00 [I 2022-10-09 21:15:08] ...trap/engines/engine.py.229: process: 0.82244 | load: 0.00015 [I 2022-10-09 21:15:08] ...trap/engines/engine.py.230: loss: 0.20107 [I 2022-10-09 21:15:08] ...optimizers/trijoint.py.151: total_norm: 13.904594 [S 2022-10-09 21:15:09] ...trap/engines/engine.py.129: train_epoch.epoch: 54 [S 2022-10-09 21:15:09] ...trap/engines/engine.py.129: train_epoch.loss: 0.15331612012456142 [I 2022-10-09 21:15:09] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 21:15:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 54 [I 2022-10-09 21:15:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 21:15:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 21:15:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 21:15:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 21:15:13] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 21:15:14] ...trap/engines/engine.py.321: eval: epoch 54 | batch 0/510 [I 2022-10-09 21:15:14] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:28 [I 2022-10-09 21:15:14] ...trap/engines/engine.py.326: process: 3.93803 | load: 3.43413 [I 2022-10-09 21:15:18] ...trap/engines/engine.py.321: eval: epoch 54 | batch 10/510 [I 2022-10-09 21:15:18] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:41 [I 2022-10-09 21:15:18] ...trap/engines/engine.py.326: process: 0.43597 | load: 0.00018 [I 2022-10-09 21:15:23] ...trap/engines/engine.py.321: eval: epoch 54 | batch 20/510 [I 2022-10-09 21:15:23] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:04:06 [I 2022-10-09 21:15:23] ...trap/engines/engine.py.326: process: 0.41745 | load: 0.00018 [I 2022-10-09 21:15:27] ...trap/engines/engine.py.321: eval: epoch 54 | batch 30/510 [I 2022-10-09 21:15:27] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:03:26 [I 2022-10-09 21:15:27] ...trap/engines/engine.py.326: process: 0.44182 | load: 0.00019 [I 2022-10-09 21:15:33] ...trap/engines/engine.py.321: eval: epoch 54 | batch 40/510 [I 2022-10-09 21:15:33] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:05 [I 2022-10-09 21:15:33] ...trap/engines/engine.py.326: process: 0.54477 | load: 0.00019 [I 2022-10-09 21:15:37] ...trap/engines/engine.py.321: eval: epoch 54 | batch 50/510 [I 2022-10-09 21:15:37] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:21 [I 2022-10-09 21:15:37] ...trap/engines/engine.py.326: process: 0.42377 | load: 0.00018 [I 2022-10-09 21:15:41] ...trap/engines/engine.py.321: eval: epoch 54 | batch 60/510 [I 2022-10-09 21:15:41] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:12 [I 2022-10-09 21:15:41] ...trap/engines/engine.py.326: process: 0.42079 | load: 0.00019 [I 2022-10-09 21:15:46] ...trap/engines/engine.py.321: eval: epoch 54 | batch 70/510 [I 2022-10-09 21:15:46] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:36 [I 2022-10-09 21:15:46] ...trap/engines/engine.py.326: process: 0.53527 | load: 0.00018 [I 2022-10-09 21:15:51] ...trap/engines/engine.py.321: eval: epoch 54 | batch 80/510 [I 2022-10-09 21:15:51] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:46 [I 2022-10-09 21:15:51] ...trap/engines/engine.py.326: process: 0.59815 | load: 0.00019 [I 2022-10-09 21:15:56] ...trap/engines/engine.py.321: eval: epoch 54 | batch 90/510 [I 2022-10-09 21:15:56] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:03 [I 2022-10-09 21:15:56] ...trap/engines/engine.py.326: process: 0.41731 | load: 0.00018 [I 2022-10-09 21:16:00] ...trap/engines/engine.py.321: eval: epoch 54 | batch 100/510 [I 2022-10-09 21:16:00] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:02:43 [I 2022-10-09 21:16:00] ...trap/engines/engine.py.326: process: 0.31133 | load: 0.00019 [I 2022-10-09 21:16:03] ...trap/engines/engine.py.321: eval: epoch 54 | batch 110/510 [I 2022-10-09 21:16:03] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:09 [I 2022-10-09 21:16:03] ...trap/engines/engine.py.326: process: 0.31416 | load: 0.00018 [I 2022-10-09 21:16:06] ...trap/engines/engine.py.321: eval: epoch 54 | batch 120/510 [I 2022-10-09 21:16:06] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:03 [I 2022-10-09 21:16:06] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00018 [I 2022-10-09 21:16:09] ...trap/engines/engine.py.321: eval: epoch 54 | batch 130/510 [I 2022-10-09 21:16:09] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:01 [I 2022-10-09 21:16:09] ...trap/engines/engine.py.326: process: 0.31494 | load: 0.00020 [I 2022-10-09 21:16:12] ...trap/engines/engine.py.321: eval: epoch 54 | batch 140/510 [I 2022-10-09 21:16:12] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:57 [I 2022-10-09 21:16:12] ...trap/engines/engine.py.326: process: 0.31235 | load: 0.00019 [I 2022-10-09 21:16:16] ...trap/engines/engine.py.321: eval: epoch 54 | batch 150/510 [I 2022-10-09 21:16:16] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:54 [I 2022-10-09 21:16:16] ...trap/engines/engine.py.326: process: 0.31972 | load: 0.00020 [I 2022-10-09 21:16:19] ...trap/engines/engine.py.321: eval: epoch 54 | batch 160/510 [I 2022-10-09 21:16:19] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:51 [I 2022-10-09 21:16:19] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00020 [I 2022-10-09 21:16:22] ...trap/engines/engine.py.321: eval: epoch 54 | batch 170/510 [I 2022-10-09 21:16:22] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:47 [I 2022-10-09 21:16:22] ...trap/engines/engine.py.326: process: 0.31720 | load: 0.00020 [I 2022-10-09 21:16:25] ...trap/engines/engine.py.321: eval: epoch 54 | batch 180/510 [I 2022-10-09 21:16:25] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:45 [I 2022-10-09 21:16:25] ...trap/engines/engine.py.326: process: 0.32004 | load: 0.00019 [I 2022-10-09 21:16:28] ...trap/engines/engine.py.321: eval: epoch 54 | batch 190/510 [I 2022-10-09 21:16:28] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:42 [I 2022-10-09 21:16:28] ...trap/engines/engine.py.326: process: 0.31635 | load: 0.00019 [I 2022-10-09 21:16:32] ...trap/engines/engine.py.321: eval: epoch 54 | batch 200/510 [I 2022-10-09 21:16:32] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:39 [I 2022-10-09 21:16:32] ...trap/engines/engine.py.326: process: 0.32488 | load: 0.00020 [I 2022-10-09 21:16:35] ...trap/engines/engine.py.321: eval: epoch 54 | batch 210/510 [I 2022-10-09 21:16:35] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:35 [I 2022-10-09 21:16:35] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00021 [I 2022-10-09 21:16:38] ...trap/engines/engine.py.321: eval: epoch 54 | batch 220/510 [I 2022-10-09 21:16:38] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:32 [I 2022-10-09 21:16:38] ...trap/engines/engine.py.326: process: 0.32159 | load: 0.00020 [I 2022-10-09 21:16:41] ...trap/engines/engine.py.321: eval: epoch 54 | batch 230/510 [I 2022-10-09 21:16:41] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:29 [I 2022-10-09 21:16:41] ...trap/engines/engine.py.326: process: 0.32215 | load: 0.00019 [I 2022-10-09 21:16:44] ...trap/engines/engine.py.321: eval: epoch 54 | batch 240/510 [I 2022-10-09 21:16:44] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:26 [I 2022-10-09 21:16:44] ...trap/engines/engine.py.326: process: 0.31835 | load: 0.00019 [I 2022-10-09 21:16:47] ...trap/engines/engine.py.321: eval: epoch 54 | batch 250/510 [I 2022-10-09 21:16:47] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:23 [I 2022-10-09 21:16:47] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00019 [I 2022-10-09 21:16:51] ...trap/engines/engine.py.321: eval: epoch 54 | batch 260/510 [I 2022-10-09 21:16:51] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:20 [I 2022-10-09 21:16:51] ...trap/engines/engine.py.326: process: 0.32343 | load: 0.00019 [I 2022-10-09 21:16:54] ...trap/engines/engine.py.321: eval: epoch 54 | batch 270/510 [I 2022-10-09 21:16:54] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:16 [I 2022-10-09 21:16:54] ...trap/engines/engine.py.326: process: 0.31753 | load: 0.00020 [I 2022-10-09 21:16:57] ...trap/engines/engine.py.321: eval: epoch 54 | batch 280/510 [I 2022-10-09 21:16:57] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:12 [I 2022-10-09 21:16:57] ...trap/engines/engine.py.326: process: 0.31209 | load: 0.00019 [I 2022-10-09 21:17:00] ...trap/engines/engine.py.321: eval: epoch 54 | batch 290/510 [I 2022-10-09 21:17:00] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:09 [I 2022-10-09 21:17:00] ...trap/engines/engine.py.326: process: 0.31853 | load: 0.00019 [I 2022-10-09 21:17:03] ...trap/engines/engine.py.321: eval: epoch 54 | batch 300/510 [I 2022-10-09 21:17:03] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:07 [I 2022-10-09 21:17:03] ...trap/engines/engine.py.326: process: 0.32166 | load: 0.00019 [I 2022-10-09 21:17:07] ...trap/engines/engine.py.321: eval: epoch 54 | batch 310/510 [I 2022-10-09 21:17:07] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:04 [I 2022-10-09 21:17:07] ...trap/engines/engine.py.326: process: 0.32685 | load: 0.00020 [I 2022-10-09 21:17:10] ...trap/engines/engine.py.321: eval: epoch 54 | batch 320/510 [I 2022-10-09 21:17:10] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:01 [I 2022-10-09 21:17:10] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00021 [I 2022-10-09 21:17:13] ...trap/engines/engine.py.321: eval: epoch 54 | batch 330/510 [I 2022-10-09 21:17:13] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:57 [I 2022-10-09 21:17:13] ...trap/engines/engine.py.326: process: 0.31947 | load: 0.00020 [I 2022-10-09 21:17:16] ...trap/engines/engine.py.321: eval: epoch 54 | batch 340/510 [I 2022-10-09 21:17:16] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:53 [I 2022-10-09 21:17:16] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00019 [I 2022-10-09 21:17:19] ...trap/engines/engine.py.321: eval: epoch 54 | batch 350/510 [I 2022-10-09 21:17:19] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:51 [I 2022-10-09 21:17:19] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00020 [I 2022-10-09 21:17:23] ...trap/engines/engine.py.321: eval: epoch 54 | batch 360/510 [I 2022-10-09 21:17:23] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:47 [I 2022-10-09 21:17:23] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00019 [I 2022-10-09 21:17:26] ...trap/engines/engine.py.321: eval: epoch 54 | batch 370/510 [I 2022-10-09 21:17:26] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:44 [I 2022-10-09 21:17:26] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00019 [I 2022-10-09 21:17:29] ...trap/engines/engine.py.321: eval: epoch 54 | batch 380/510 [I 2022-10-09 21:17:29] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:41 [I 2022-10-09 21:17:29] ...trap/engines/engine.py.326: process: 0.32338 | load: 0.00021 [I 2022-10-09 21:17:32] ...trap/engines/engine.py.321: eval: epoch 54 | batch 390/510 [I 2022-10-09 21:17:32] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:38 [I 2022-10-09 21:17:32] ...trap/engines/engine.py.326: process: 0.31747 | load: 0.00021 [I 2022-10-09 21:17:35] ...trap/engines/engine.py.321: eval: epoch 54 | batch 400/510 [I 2022-10-09 21:17:35] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:34 [I 2022-10-09 21:17:35] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00019 [I 2022-10-09 21:17:39] ...trap/engines/engine.py.321: eval: epoch 54 | batch 410/510 [I 2022-10-09 21:17:39] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:31 [I 2022-10-09 21:17:39] ...trap/engines/engine.py.326: process: 0.31967 | load: 0.00019 [I 2022-10-09 21:17:42] ...trap/engines/engine.py.321: eval: epoch 54 | batch 420/510 [I 2022-10-09 21:17:42] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:28 [I 2022-10-09 21:17:42] ...trap/engines/engine.py.326: process: 0.31999 | load: 0.00021 [I 2022-10-09 21:17:45] ...trap/engines/engine.py.321: eval: epoch 54 | batch 430/510 [I 2022-10-09 21:17:45] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:25 [I 2022-10-09 21:17:45] ...trap/engines/engine.py.326: process: 0.31613 | load: 0.00019 [I 2022-10-09 21:17:48] ...trap/engines/engine.py.321: eval: epoch 54 | batch 440/510 [I 2022-10-09 21:17:48] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:22 [I 2022-10-09 21:17:48] ...trap/engines/engine.py.326: process: 0.32610 | load: 0.00021 [I 2022-10-09 21:17:51] ...trap/engines/engine.py.321: eval: epoch 54 | batch 450/510 [I 2022-10-09 21:17:51] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:19 [I 2022-10-09 21:17:51] ...trap/engines/engine.py.326: process: 0.31528 | load: 0.00021 [I 2022-10-09 21:17:55] ...trap/engines/engine.py.321: eval: epoch 54 | batch 460/510 [I 2022-10-09 21:17:55] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:16 [I 2022-10-09 21:17:55] ...trap/engines/engine.py.326: process: 0.32242 | load: 0.00019 [I 2022-10-09 21:17:58] ...trap/engines/engine.py.321: eval: epoch 54 | batch 470/510 [I 2022-10-09 21:17:58] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:12 [I 2022-10-09 21:17:58] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00020 [I 2022-10-09 21:18:01] ...trap/engines/engine.py.321: eval: epoch 54 | batch 480/510 [I 2022-10-09 21:18:01] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:09 [I 2022-10-09 21:18:01] ...trap/engines/engine.py.326: process: 0.31713 | load: 0.00021 [I 2022-10-09 21:18:04] ...trap/engines/engine.py.321: eval: epoch 54 | batch 490/510 [I 2022-10-09 21:18:04] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:06 [I 2022-10-09 21:18:04] ...trap/engines/engine.py.326: process: 0.31224 | load: 0.00019 [I 2022-10-09 21:18:07] ...trap/engines/engine.py.321: eval: epoch 54 | batch 500/510 [I 2022-10-09 21:18:07] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:03 [I 2022-10-09 21:18:07] ...trap/engines/engine.py.326: process: 0.32325 | load: 0.00019 [I 2022-10-09 21:18:10] ...trap/engines/engine.py.321: eval: epoch 54 | batch 510/510 [I 2022-10-09 21:18:10] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:00 [I 2022-10-09 21:18:10] ...trap/engines/engine.py.326: process: 0.31540 | load: 0.00015 [S 2022-10-09 21:18:11] ...trap/engines/engine.py.132: eval_epoch.epoch: 54 [I 2022-10-09 21:18:11] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.723600 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904400 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936200 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.714300 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904200 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935600 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015749 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010111 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008681 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014478 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008998 [S 2022-10-09 21:18:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006974 [I 2022-10-09 21:18:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 21:18:48] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 21:18:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 21:18:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 21:18:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 21:18:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 21:18:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 21:18:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 21:18:58] ...trap/engines/engine.py.163: Training model on trainset for epoch 55 [I 2022-10-09 21:18:58] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 21:19:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 0/2304 [I 2022-10-09 21:19:01] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:55:00 [I 2022-10-09 21:19:01] ...trap/engines/engine.py.229: process: 2.99482 | load: 2.08105 [I 2022-10-09 21:19:01] ...trap/engines/engine.py.230: loss: 0.13997 [I 2022-10-09 21:19:01] ...optimizers/trijoint.py.151: total_norm: 10.866329 [I 2022-10-09 21:19:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 10/2304 [I 2022-10-09 21:19:10] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:17 [I 2022-10-09 21:19:10] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00772 [I 2022-10-09 21:19:10] ...trap/engines/engine.py.230: loss: 0.17356 [I 2022-10-09 21:19:10] ...optimizers/trijoint.py.151: total_norm: 16.741003 [I 2022-10-09 21:19:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 20/2304 [I 2022-10-09 21:19:18] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:20 [I 2022-10-09 21:19:18] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00482 [I 2022-10-09 21:19:18] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-09 21:19:18] ...optimizers/trijoint.py.151: total_norm: 9.150846 [I 2022-10-09 21:19:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 30/2304 [I 2022-10-09 21:19:27] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:28 [I 2022-10-09 21:19:27] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00413 [I 2022-10-09 21:19:27] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-09 21:19:27] ...optimizers/trijoint.py.151: total_norm: 10.093026 [I 2022-10-09 21:19:35] ...trap/engines/engine.py.224: train: epoch 55 | batch 40/2304 [I 2022-10-09 21:19:35] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:14 [I 2022-10-09 21:19:35] ...trap/engines/engine.py.229: process: 0.86959 | load: 0.00478 [I 2022-10-09 21:19:35] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-09 21:19:35] ...optimizers/trijoint.py.151: total_norm: 33.767643 [I 2022-10-09 21:19:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 50/2304 [I 2022-10-09 21:19:44] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:04 [I 2022-10-09 21:19:44] ...trap/engines/engine.py.229: process: 0.84379 | load: 0.00397 [I 2022-10-09 21:19:44] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-09 21:19:44] ...optimizers/trijoint.py.151: total_norm: 6.608190 [I 2022-10-09 21:19:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 60/2304 [I 2022-10-09 21:19:52] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:49 [I 2022-10-09 21:19:52] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00376 [I 2022-10-09 21:19:52] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-09 21:19:52] ...optimizers/trijoint.py.151: total_norm: 24.815687 [I 2022-10-09 21:20:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 70/2304 [I 2022-10-09 21:20:01] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:51 [I 2022-10-09 21:20:01] ...trap/engines/engine.py.229: process: 0.86682 | load: 0.00375 [I 2022-10-09 21:20:01] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-09 21:20:01] ...optimizers/trijoint.py.151: total_norm: 12.677645 [I 2022-10-09 21:20:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 80/2304 [I 2022-10-09 21:20:09] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:38 [I 2022-10-09 21:20:09] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00407 [I 2022-10-09 21:20:09] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-09 21:20:09] ...optimizers/trijoint.py.151: total_norm: 8.005421 [I 2022-10-09 21:20:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 90/2304 [I 2022-10-09 21:20:18] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:30 [I 2022-10-09 21:20:18] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00362 [I 2022-10-09 21:20:18] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-09 21:20:18] ...optimizers/trijoint.py.151: total_norm: 49.083958 [I 2022-10-09 21:20:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 100/2304 [I 2022-10-09 21:20:26] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:10 [I 2022-10-09 21:20:26] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00334 [I 2022-10-09 21:20:26] ...trap/engines/engine.py.230: loss: 0.10383 [I 2022-10-09 21:20:26] ...optimizers/trijoint.py.151: total_norm: 7.991970 [I 2022-10-09 21:20:35] ...trap/engines/engine.py.224: train: epoch 55 | batch 110/2304 [I 2022-10-09 21:20:35] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:12 [I 2022-10-09 21:20:35] ...trap/engines/engine.py.229: process: 0.85653 | load: 0.00299 [I 2022-10-09 21:20:35] ...trap/engines/engine.py.230: loss: 0.15555 [I 2022-10-09 21:20:35] ...optimizers/trijoint.py.151: total_norm: 23.963982 [I 2022-10-09 21:20:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 120/2304 [I 2022-10-09 21:20:43] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:04 [I 2022-10-09 21:20:43] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00359 [I 2022-10-09 21:20:43] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-09 21:20:43] ...optimizers/trijoint.py.151: total_norm: 14.716867 [I 2022-10-09 21:20:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 130/2304 [I 2022-10-09 21:20:52] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:57 [I 2022-10-09 21:20:52] ...trap/engines/engine.py.229: process: 0.86404 | load: 0.00345 [I 2022-10-09 21:20:52] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-09 21:20:52] ...optimizers/trijoint.py.151: total_norm: 11.099822 [I 2022-10-09 21:21:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 140/2304 [I 2022-10-09 21:21:00] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:49 [I 2022-10-09 21:21:00] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00341 [I 2022-10-09 21:21:00] ...trap/engines/engine.py.230: loss: 0.16111 [I 2022-10-09 21:21:00] ...optimizers/trijoint.py.151: total_norm: 18.592985 [I 2022-10-09 21:21:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 150/2304 [I 2022-10-09 21:21:09] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:34 [I 2022-10-09 21:21:09] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00311 [I 2022-10-09 21:21:09] ...trap/engines/engine.py.230: loss: 0.15796 [I 2022-10-09 21:21:09] ...optimizers/trijoint.py.151: total_norm: 7.907272 [I 2022-10-09 21:21:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 160/2304 [I 2022-10-09 21:21:18] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:26 [I 2022-10-09 21:21:18] ...trap/engines/engine.py.229: process: 0.85886 | load: 0.00333 [I 2022-10-09 21:21:18] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-09 21:21:18] ...optimizers/trijoint.py.151: total_norm: 19.762627 [I 2022-10-09 21:21:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 170/2304 [I 2022-10-09 21:21:26] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:15 [I 2022-10-09 21:21:26] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00360 [I 2022-10-09 21:21:26] ...trap/engines/engine.py.230: loss: 0.21742 [I 2022-10-09 21:21:26] ...optimizers/trijoint.py.151: total_norm: 15.873314 [I 2022-10-09 21:21:35] ...trap/engines/engine.py.224: train: epoch 55 | batch 180/2304 [I 2022-10-09 21:21:35] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:05 [I 2022-10-09 21:21:35] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00326 [I 2022-10-09 21:21:35] ...trap/engines/engine.py.230: loss: 0.16589 [I 2022-10-09 21:21:35] ...optimizers/trijoint.py.151: total_norm: 15.995690 [I 2022-10-09 21:21:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 190/2304 [I 2022-10-09 21:21:43] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:06 [I 2022-10-09 21:21:43] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00311 [I 2022-10-09 21:21:43] ...trap/engines/engine.py.230: loss: 0.17643 [I 2022-10-09 21:21:43] ...optimizers/trijoint.py.151: total_norm: 14.765318 [I 2022-10-09 21:21:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 200/2304 [I 2022-10-09 21:21:52] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:57 [I 2022-10-09 21:21:52] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00339 [I 2022-10-09 21:21:52] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-09 21:21:52] ...optimizers/trijoint.py.151: total_norm: 26.381996 [I 2022-10-09 21:22:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 210/2304 [I 2022-10-09 21:22:00] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:40 [I 2022-10-09 21:22:00] ...trap/engines/engine.py.229: process: 0.86420 | load: 0.00313 [I 2022-10-09 21:22:00] ...trap/engines/engine.py.230: loss: 0.19917 [I 2022-10-09 21:22:00] ...optimizers/trijoint.py.151: total_norm: 10.025179 [I 2022-10-09 21:22:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 220/2304 [I 2022-10-09 21:22:09] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:42 [I 2022-10-09 21:22:09] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00340 [I 2022-10-09 21:22:09] ...trap/engines/engine.py.230: loss: 0.18458 [I 2022-10-09 21:22:09] ...optimizers/trijoint.py.151: total_norm: 13.997413 [I 2022-10-09 21:22:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 230/2304 [I 2022-10-09 21:22:17] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:25 [I 2022-10-09 21:22:17] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00341 [I 2022-10-09 21:22:17] ...trap/engines/engine.py.230: loss: 0.19187 [I 2022-10-09 21:22:17] ...optimizers/trijoint.py.151: total_norm: 17.193617 [I 2022-10-09 21:22:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 240/2304 [I 2022-10-09 21:22:26] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:10 [I 2022-10-09 21:22:26] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00343 [I 2022-10-09 21:22:26] ...trap/engines/engine.py.230: loss: 0.19170 [I 2022-10-09 21:22:26] ...optimizers/trijoint.py.151: total_norm: 17.868078 [I 2022-10-09 21:22:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 250/2304 [I 2022-10-09 21:22:34] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:11 [I 2022-10-09 21:22:34] ...trap/engines/engine.py.229: process: 0.84854 | load: 0.00323 [I 2022-10-09 21:22:34] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-09 21:22:34] ...optimizers/trijoint.py.151: total_norm: 11.869192 [I 2022-10-09 21:22:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 260/2304 [I 2022-10-09 21:22:43] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:00 [I 2022-10-09 21:22:43] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00313 [I 2022-10-09 21:22:43] ...trap/engines/engine.py.230: loss: 0.18358 [I 2022-10-09 21:22:43] ...optimizers/trijoint.py.151: total_norm: 12.166508 [I 2022-10-09 21:22:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 270/2304 [I 2022-10-09 21:22:52] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:49 [I 2022-10-09 21:22:52] ...trap/engines/engine.py.229: process: 0.86236 | load: 0.00323 [I 2022-10-09 21:22:52] ...trap/engines/engine.py.230: loss: 0.19550 [I 2022-10-09 21:22:52] ...optimizers/trijoint.py.151: total_norm: 18.952036 [I 2022-10-09 21:23:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 280/2304 [I 2022-10-09 21:23:00] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:51 [I 2022-10-09 21:23:00] ...trap/engines/engine.py.229: process: 0.86558 | load: 0.00326 [I 2022-10-09 21:23:00] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-09 21:23:00] ...optimizers/trijoint.py.151: total_norm: 13.337330 [I 2022-10-09 21:23:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 290/2304 [I 2022-10-09 21:23:09] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:33 [I 2022-10-09 21:23:09] ...trap/engines/engine.py.229: process: 0.84226 | load: 0.00324 [I 2022-10-09 21:23:09] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-09 21:23:09] ...optimizers/trijoint.py.151: total_norm: 18.996233 [I 2022-10-09 21:23:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 300/2304 [I 2022-10-09 21:23:17] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:07 [I 2022-10-09 21:23:17] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00295 [I 2022-10-09 21:23:17] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-09 21:23:17] ...optimizers/trijoint.py.151: total_norm: 7.840909 [I 2022-10-09 21:23:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 310/2304 [I 2022-10-09 21:23:26] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:27:57 [I 2022-10-09 21:23:26] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00293 [I 2022-10-09 21:23:26] ...trap/engines/engine.py.230: loss: 0.21495 [I 2022-10-09 21:23:26] ...optimizers/trijoint.py.151: total_norm: 18.233940 [I 2022-10-09 21:23:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 320/2304 [I 2022-10-09 21:23:34] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:00 [I 2022-10-09 21:23:34] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00295 [I 2022-10-09 21:23:34] ...trap/engines/engine.py.230: loss: 0.12001 [I 2022-10-09 21:23:34] ...optimizers/trijoint.py.151: total_norm: 19.542715 [I 2022-10-09 21:23:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 330/2304 [I 2022-10-09 21:23:43] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:52 [I 2022-10-09 21:23:43] ...trap/engines/engine.py.229: process: 0.85817 | load: 0.00307 [I 2022-10-09 21:23:43] ...trap/engines/engine.py.230: loss: 0.21595 [I 2022-10-09 21:23:43] ...optimizers/trijoint.py.151: total_norm: 19.259602 [I 2022-10-09 21:23:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 340/2304 [I 2022-10-09 21:23:51] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:38 [I 2022-10-09 21:23:51] ...trap/engines/engine.py.229: process: 0.85393 | load: 0.00306 [I 2022-10-09 21:23:51] ...trap/engines/engine.py.230: loss: 0.17908 [I 2022-10-09 21:23:51] ...optimizers/trijoint.py.151: total_norm: 22.666433 [I 2022-10-09 21:23:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 350/2304 [I 2022-10-09 21:23:59] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:27 [I 2022-10-09 21:23:59] ...trap/engines/engine.py.229: process: 0.83818 | load: 0.00294 [I 2022-10-09 21:23:59] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-09 21:23:59] ...optimizers/trijoint.py.151: total_norm: 14.872794 [I 2022-10-09 21:24:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 360/2304 [I 2022-10-09 21:24:08] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:21 [I 2022-10-09 21:24:08] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00303 [I 2022-10-09 21:24:08] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-09 21:24:08] ...optimizers/trijoint.py.151: total_norm: 21.277178 [I 2022-10-09 21:24:16] ...trap/engines/engine.py.224: train: epoch 55 | batch 370/2304 [I 2022-10-09 21:24:16] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:16 [I 2022-10-09 21:24:16] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00294 [I 2022-10-09 21:24:16] ...trap/engines/engine.py.230: loss: 0.09991 [I 2022-10-09 21:24:16] ...optimizers/trijoint.py.151: total_norm: 10.118619 [I 2022-10-09 21:24:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 380/2304 [I 2022-10-09 21:24:25] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:00 [I 2022-10-09 21:24:25] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00303 [I 2022-10-09 21:24:25] ...trap/engines/engine.py.230: loss: 0.13893 [I 2022-10-09 21:24:25] ...optimizers/trijoint.py.151: total_norm: 18.800392 [I 2022-10-09 21:24:33] ...trap/engines/engine.py.224: train: epoch 55 | batch 390/2304 [I 2022-10-09 21:24:33] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:26:59 [I 2022-10-09 21:24:33] ...trap/engines/engine.py.229: process: 0.85705 | load: 0.00303 [I 2022-10-09 21:24:33] ...trap/engines/engine.py.230: loss: 0.13147 [I 2022-10-09 21:24:33] ...optimizers/trijoint.py.151: total_norm: 21.491276 [I 2022-10-09 21:24:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 400/2304 [I 2022-10-09 21:24:42] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:49 [I 2022-10-09 21:24:42] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00300 [I 2022-10-09 21:24:42] ...trap/engines/engine.py.230: loss: 0.18368 [I 2022-10-09 21:24:42] ...optimizers/trijoint.py.151: total_norm: 26.062265 [I 2022-10-09 21:24:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 410/2304 [I 2022-10-09 21:24:50] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:37 [I 2022-10-09 21:24:50] ...trap/engines/engine.py.229: process: 0.83877 | load: 0.00297 [I 2022-10-09 21:24:50] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-09 21:24:50] ...optimizers/trijoint.py.151: total_norm: 10.010886 [I 2022-10-09 21:24:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 420/2304 [I 2022-10-09 21:24:59] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:22 [I 2022-10-09 21:24:59] ...trap/engines/engine.py.229: process: 0.83294 | load: 0.00302 [I 2022-10-09 21:24:59] ...trap/engines/engine.py.230: loss: 0.16088 [I 2022-10-09 21:24:59] ...optimizers/trijoint.py.151: total_norm: 21.996065 [I 2022-10-09 21:25:07] ...trap/engines/engine.py.224: train: epoch 55 | batch 430/2304 [I 2022-10-09 21:25:07] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:24 [I 2022-10-09 21:25:07] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00295 [I 2022-10-09 21:25:07] ...trap/engines/engine.py.230: loss: 0.25287 [I 2022-10-09 21:25:07] ...optimizers/trijoint.py.151: total_norm: 17.230659 [I 2022-10-09 21:25:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 440/2304 [I 2022-10-09 21:25:15] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:04 [I 2022-10-09 21:25:15] ...trap/engines/engine.py.229: process: 0.84818 | load: 0.00297 [I 2022-10-09 21:25:15] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-09 21:25:15] ...optimizers/trijoint.py.151: total_norm: 13.444833 [I 2022-10-09 21:25:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 450/2304 [I 2022-10-09 21:25:24] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:05 [I 2022-10-09 21:25:24] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00295 [I 2022-10-09 21:25:24] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-09 21:25:24] ...optimizers/trijoint.py.151: total_norm: 15.369013 [I 2022-10-09 21:25:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 460/2304 [I 2022-10-09 21:25:32] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:25:49 [I 2022-10-09 21:25:32] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00306 [I 2022-10-09 21:25:32] ...trap/engines/engine.py.230: loss: 0.12479 [I 2022-10-09 21:25:32] ...optimizers/trijoint.py.151: total_norm: 7.977436 [I 2022-10-09 21:25:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 470/2304 [I 2022-10-09 21:25:41] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:25:48 [I 2022-10-09 21:25:41] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00298 [I 2022-10-09 21:25:41] ...trap/engines/engine.py.230: loss: 0.12460 [I 2022-10-09 21:25:41] ...optimizers/trijoint.py.151: total_norm: 12.821859 [I 2022-10-09 21:25:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 480/2304 [I 2022-10-09 21:25:49] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:25:38 [I 2022-10-09 21:25:49] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00299 [I 2022-10-09 21:25:49] ...trap/engines/engine.py.230: loss: 0.16751 [I 2022-10-09 21:25:49] ...optimizers/trijoint.py.151: total_norm: 13.560891 [I 2022-10-09 21:25:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 490/2304 [I 2022-10-09 21:25:58] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:25:29 [I 2022-10-09 21:25:58] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00298 [I 2022-10-09 21:25:58] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-09 21:25:58] ...optimizers/trijoint.py.151: total_norm: 18.939234 [I 2022-10-09 21:26:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 500/2304 [I 2022-10-09 21:26:06] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:17 [I 2022-10-09 21:26:06] ...trap/engines/engine.py.229: process: 0.84844 | load: 0.00295 [I 2022-10-09 21:26:06] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-09 21:26:06] ...optimizers/trijoint.py.151: total_norm: 16.533628 [I 2022-10-09 21:26:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 510/2304 [I 2022-10-09 21:26:15] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:25:20 [I 2022-10-09 21:26:15] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00301 [I 2022-10-09 21:26:15] ...trap/engines/engine.py.230: loss: 0.21915 [I 2022-10-09 21:26:15] ...optimizers/trijoint.py.151: total_norm: 22.008366 [I 2022-10-09 21:26:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 520/2304 [I 2022-10-09 21:26:23] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:01 [I 2022-10-09 21:26:23] ...trap/engines/engine.py.229: process: 0.83627 | load: 0.00298 [I 2022-10-09 21:26:23] ...trap/engines/engine.py.230: loss: 0.20942 [I 2022-10-09 21:26:23] ...optimizers/trijoint.py.151: total_norm: 21.841749 [I 2022-10-09 21:26:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 530/2304 [I 2022-10-09 21:26:32] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:06 [I 2022-10-09 21:26:32] ...trap/engines/engine.py.229: process: 0.86290 | load: 0.00300 [I 2022-10-09 21:26:32] ...trap/engines/engine.py.230: loss: 0.10578 [I 2022-10-09 21:26:32] ...optimizers/trijoint.py.151: total_norm: 25.557426 [I 2022-10-09 21:26:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 540/2304 [I 2022-10-09 21:26:40] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:24:46 [I 2022-10-09 21:26:40] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00299 [I 2022-10-09 21:26:40] ...trap/engines/engine.py.230: loss: 0.23298 [I 2022-10-09 21:26:40] ...optimizers/trijoint.py.151: total_norm: 22.118008 [I 2022-10-09 21:26:48] ...trap/engines/engine.py.224: train: epoch 55 | batch 550/2304 [I 2022-10-09 21:26:48] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:41 [I 2022-10-09 21:26:48] ...trap/engines/engine.py.229: process: 0.83424 | load: 0.00304 [I 2022-10-09 21:26:48] ...trap/engines/engine.py.230: loss: 0.16150 [I 2022-10-09 21:26:48] ...optimizers/trijoint.py.151: total_norm: 17.658850 [I 2022-10-09 21:26:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 560/2304 [I 2022-10-09 21:26:57] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:31 [I 2022-10-09 21:26:57] ...trap/engines/engine.py.229: process: 0.83645 | load: 0.00301 [I 2022-10-09 21:26:57] ...trap/engines/engine.py.230: loss: 0.16890 [I 2022-10-09 21:26:57] ...optimizers/trijoint.py.151: total_norm: 13.242863 [I 2022-10-09 21:27:05] ...trap/engines/engine.py.224: train: epoch 55 | batch 570/2304 [I 2022-10-09 21:27:05] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:30 [I 2022-10-09 21:27:05] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00297 [I 2022-10-09 21:27:05] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-09 21:27:05] ...optimizers/trijoint.py.151: total_norm: 16.696430 [I 2022-10-09 21:27:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 580/2304 [I 2022-10-09 21:27:14] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:16 [I 2022-10-09 21:27:14] ...trap/engines/engine.py.229: process: 0.85459 | load: 0.00298 [I 2022-10-09 21:27:14] ...trap/engines/engine.py.230: loss: 0.25722 [I 2022-10-09 21:27:14] ...optimizers/trijoint.py.151: total_norm: 22.933996 [I 2022-10-09 21:27:22] ...trap/engines/engine.py.224: train: epoch 55 | batch 590/2304 [I 2022-10-09 21:27:22] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:21 [I 2022-10-09 21:27:22] ...trap/engines/engine.py.229: process: 0.85605 | load: 0.00310 [I 2022-10-09 21:27:22] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-09 21:27:22] ...optimizers/trijoint.py.151: total_norm: 19.803787 [I 2022-10-09 21:27:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 600/2304 [I 2022-10-09 21:27:31] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:13 [I 2022-10-09 21:27:31] ...trap/engines/engine.py.229: process: 0.85846 | load: 0.00313 [I 2022-10-09 21:27:31] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-09 21:27:31] ...optimizers/trijoint.py.151: total_norm: 11.703965 [I 2022-10-09 21:27:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 610/2304 [I 2022-10-09 21:27:58] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:46:21 [I 2022-10-09 21:27:58] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00311 [I 2022-10-09 21:27:58] ...trap/engines/engine.py.230: loss: 0.16967 [I 2022-10-09 21:27:58] ...optimizers/trijoint.py.151: total_norm: 18.249773 [I 2022-10-09 21:28:07] ...trap/engines/engine.py.224: train: epoch 55 | batch 620/2304 [I 2022-10-09 21:28:07] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:26:25 [I 2022-10-09 21:28:07] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00310 [I 2022-10-09 21:28:07] ...trap/engines/engine.py.230: loss: 0.30717 [I 2022-10-09 21:28:07] ...optimizers/trijoint.py.151: total_norm: 27.747147 [I 2022-10-09 21:28:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 630/2304 [I 2022-10-09 21:28:15] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:24:02 [I 2022-10-09 21:28:15] ...trap/engines/engine.py.229: process: 0.85826 | load: 0.00313 [I 2022-10-09 21:28:15] ...trap/engines/engine.py.230: loss: 0.15760 [I 2022-10-09 21:28:15] ...optimizers/trijoint.py.151: total_norm: 8.773777 [I 2022-10-09 21:28:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 640/2304 [I 2022-10-09 21:28:24] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:31 [I 2022-10-09 21:28:24] ...trap/engines/engine.py.229: process: 0.83722 | load: 0.00312 [I 2022-10-09 21:28:24] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-09 21:28:24] ...optimizers/trijoint.py.151: total_norm: 11.685635 [I 2022-10-09 21:28:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 650/2304 [I 2022-10-09 21:28:32] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:29 [I 2022-10-09 21:28:32] ...trap/engines/engine.py.229: process: 0.86146 | load: 0.00314 [I 2022-10-09 21:28:32] ...trap/engines/engine.py.230: loss: 0.22836 [I 2022-10-09 21:28:32] ...optimizers/trijoint.py.151: total_norm: 20.259228 [I 2022-10-09 21:28:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 660/2304 [I 2022-10-09 21:28:41] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:31 [I 2022-10-09 21:28:41] ...trap/engines/engine.py.229: process: 0.86619 | load: 0.00313 [I 2022-10-09 21:28:41] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-09 21:28:41] ...optimizers/trijoint.py.151: total_norm: 10.931623 [I 2022-10-09 21:28:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 670/2304 [I 2022-10-09 21:28:50] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:23:21 [I 2022-10-09 21:28:50] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00313 [I 2022-10-09 21:28:50] ...trap/engines/engine.py.230: loss: 0.15030 [I 2022-10-09 21:28:50] ...optimizers/trijoint.py.151: total_norm: 16.250175 [I 2022-10-09 21:28:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 680/2304 [I 2022-10-09 21:28:58] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:23:05 [I 2022-10-09 21:28:58] ...trap/engines/engine.py.229: process: 0.85459 | load: 0.00311 [I 2022-10-09 21:28:58] ...trap/engines/engine.py.230: loss: 0.17924 [I 2022-10-09 21:28:58] ...optimizers/trijoint.py.151: total_norm: 44.129547 [I 2022-10-09 21:29:07] ...trap/engines/engine.py.224: train: epoch 55 | batch 690/2304 [I 2022-10-09 21:29:07] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:23:00 [I 2022-10-09 21:29:07] ...trap/engines/engine.py.229: process: 0.86480 | load: 0.00314 [I 2022-10-09 21:29:07] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-09 21:29:07] ...optimizers/trijoint.py.151: total_norm: 15.092114 [I 2022-10-09 21:29:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 700/2304 [I 2022-10-09 21:29:15] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:49 [I 2022-10-09 21:29:15] ...trap/engines/engine.py.229: process: 0.87234 | load: 0.00310 [I 2022-10-09 21:29:15] ...trap/engines/engine.py.230: loss: 0.09803 [I 2022-10-09 21:29:15] ...optimizers/trijoint.py.151: total_norm: 9.973962 [I 2022-10-09 21:29:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 710/2304 [I 2022-10-09 21:29:24] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:39 [I 2022-10-09 21:29:24] ...trap/engines/engine.py.229: process: 0.84004 | load: 0.00313 [I 2022-10-09 21:29:24] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-09 21:29:24] ...optimizers/trijoint.py.151: total_norm: 7.931117 [I 2022-10-09 21:29:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 720/2304 [I 2022-10-09 21:29:32] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:27 [I 2022-10-09 21:29:32] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00319 [I 2022-10-09 21:29:32] ...trap/engines/engine.py.230: loss: 0.14303 [I 2022-10-09 21:29:32] ...optimizers/trijoint.py.151: total_norm: 11.805572 [I 2022-10-09 21:29:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 730/2304 [I 2022-10-09 21:29:41] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:20 [I 2022-10-09 21:29:41] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00319 [I 2022-10-09 21:29:41] ...trap/engines/engine.py.230: loss: 0.10596 [I 2022-10-09 21:29:41] ...optimizers/trijoint.py.151: total_norm: 10.074204 [I 2022-10-09 21:29:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 740/2304 [I 2022-10-09 21:29:49] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:17 [I 2022-10-09 21:29:49] ...trap/engines/engine.py.229: process: 0.86412 | load: 0.00312 [I 2022-10-09 21:29:49] ...trap/engines/engine.py.230: loss: 0.19953 [I 2022-10-09 21:29:49] ...optimizers/trijoint.py.151: total_norm: 12.364790 [I 2022-10-09 21:29:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 750/2304 [I 2022-10-09 21:29:58] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:08 [I 2022-10-09 21:29:58] ...trap/engines/engine.py.229: process: 0.86469 | load: 0.00314 [I 2022-10-09 21:29:58] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-09 21:29:58] ...optimizers/trijoint.py.151: total_norm: 12.015206 [I 2022-10-09 21:30:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 760/2304 [I 2022-10-09 21:30:06] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:56 [I 2022-10-09 21:30:06] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00313 [I 2022-10-09 21:30:06] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-09 21:30:06] ...optimizers/trijoint.py.151: total_norm: 11.153780 [I 2022-10-09 21:30:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 770/2304 [I 2022-10-09 21:30:15] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:50 [I 2022-10-09 21:30:15] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00313 [I 2022-10-09 21:30:15] ...trap/engines/engine.py.230: loss: 0.13529 [I 2022-10-09 21:30:15] ...optimizers/trijoint.py.151: total_norm: 15.930508 [I 2022-10-09 21:30:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 780/2304 [I 2022-10-09 21:30:24] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:42 [I 2022-10-09 21:30:24] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00314 [I 2022-10-09 21:30:24] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-09 21:30:24] ...optimizers/trijoint.py.151: total_norm: 14.196869 [I 2022-10-09 21:30:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 790/2304 [I 2022-10-09 21:30:32] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:32 [I 2022-10-09 21:30:32] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00313 [I 2022-10-09 21:30:32] ...trap/engines/engine.py.230: loss: 0.22525 [I 2022-10-09 21:30:32] ...optimizers/trijoint.py.151: total_norm: 31.481499 [I 2022-10-09 21:30:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 800/2304 [I 2022-10-09 21:30:41] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:22 [I 2022-10-09 21:30:41] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00313 [I 2022-10-09 21:30:41] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-09 21:30:41] ...optimizers/trijoint.py.151: total_norm: 12.733047 [I 2022-10-09 21:30:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 810/2304 [I 2022-10-09 21:30:49] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:15 [I 2022-10-09 21:30:49] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00315 [I 2022-10-09 21:30:49] ...trap/engines/engine.py.230: loss: 0.19833 [I 2022-10-09 21:30:49] ...optimizers/trijoint.py.151: total_norm: 20.374420 [I 2022-10-09 21:30:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 820/2304 [I 2022-10-09 21:30:58] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:11 [I 2022-10-09 21:30:58] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00313 [I 2022-10-09 21:30:58] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-09 21:30:58] ...optimizers/trijoint.py.151: total_norm: 9.262323 [I 2022-10-09 21:31:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 830/2304 [I 2022-10-09 21:31:06] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:07 [I 2022-10-09 21:31:06] ...trap/engines/engine.py.229: process: 0.86380 | load: 0.00311 [I 2022-10-09 21:31:06] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-09 21:31:06] ...optimizers/trijoint.py.151: total_norm: 14.670395 [I 2022-10-09 21:31:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 840/2304 [I 2022-10-09 21:31:15] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:44 [I 2022-10-09 21:31:15] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00316 [I 2022-10-09 21:31:15] ...trap/engines/engine.py.230: loss: 0.12815 [I 2022-10-09 21:31:15] ...optimizers/trijoint.py.151: total_norm: 25.256645 [I 2022-10-09 21:31:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 850/2304 [I 2022-10-09 21:31:24] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:42 [I 2022-10-09 21:31:24] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00311 [I 2022-10-09 21:31:24] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-09 21:31:24] ...optimizers/trijoint.py.151: total_norm: 18.386713 [I 2022-10-09 21:31:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 860/2304 [I 2022-10-09 21:31:32] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:30 [I 2022-10-09 21:31:32] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00314 [I 2022-10-09 21:31:32] ...trap/engines/engine.py.230: loss: 0.16428 [I 2022-10-09 21:31:32] ...optimizers/trijoint.py.151: total_norm: 15.391250 [I 2022-10-09 21:31:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 870/2304 [I 2022-10-09 21:31:41] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:27 [I 2022-10-09 21:31:41] ...trap/engines/engine.py.229: process: 0.85880 | load: 0.00312 [I 2022-10-09 21:31:41] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-09 21:31:41] ...optimizers/trijoint.py.151: total_norm: 12.795497 [I 2022-10-09 21:31:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 880/2304 [I 2022-10-09 21:31:49] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:15 [I 2022-10-09 21:31:49] ...trap/engines/engine.py.229: process: 0.85718 | load: 0.00312 [I 2022-10-09 21:31:49] ...trap/engines/engine.py.230: loss: 0.14822 [I 2022-10-09 21:31:49] ...optimizers/trijoint.py.151: total_norm: 16.307333 [I 2022-10-09 21:31:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 890/2304 [I 2022-10-09 21:31:58] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:57 [I 2022-10-09 21:31:58] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00293 [I 2022-10-09 21:31:58] ...trap/engines/engine.py.230: loss: 0.13131 [I 2022-10-09 21:31:58] ...optimizers/trijoint.py.151: total_norm: 14.202113 [I 2022-10-09 21:32:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 900/2304 [I 2022-10-09 21:32:06] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:43 [I 2022-10-09 21:32:06] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00297 [I 2022-10-09 21:32:06] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-09 21:32:06] ...optimizers/trijoint.py.151: total_norm: 11.157195 [I 2022-10-09 21:32:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 910/2304 [I 2022-10-09 21:32:15] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:44 [I 2022-10-09 21:32:15] ...trap/engines/engine.py.229: process: 0.84761 | load: 0.00303 [I 2022-10-09 21:32:15] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-09 21:32:15] ...optimizers/trijoint.py.151: total_norm: 14.891735 [I 2022-10-09 21:32:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 920/2304 [I 2022-10-09 21:32:23] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:35 [I 2022-10-09 21:32:23] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00299 [I 2022-10-09 21:32:23] ...trap/engines/engine.py.230: loss: 0.19125 [I 2022-10-09 21:32:23] ...optimizers/trijoint.py.151: total_norm: 17.418119 [I 2022-10-09 21:32:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 930/2304 [I 2022-10-09 21:32:32] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:25 [I 2022-10-09 21:32:32] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00296 [I 2022-10-09 21:32:32] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-09 21:32:32] ...optimizers/trijoint.py.151: total_norm: 7.769942 [I 2022-10-09 21:32:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 940/2304 [I 2022-10-09 21:32:40] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:08 [I 2022-10-09 21:32:40] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00300 [I 2022-10-09 21:32:40] ...trap/engines/engine.py.230: loss: 0.17264 [I 2022-10-09 21:32:40] ...optimizers/trijoint.py.151: total_norm: 24.663202 [I 2022-10-09 21:32:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 950/2304 [I 2022-10-09 21:32:49] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:05 [I 2022-10-09 21:32:49] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00296 [I 2022-10-09 21:32:49] ...trap/engines/engine.py.230: loss: 0.11054 [I 2022-10-09 21:32:49] ...optimizers/trijoint.py.151: total_norm: 10.493008 [I 2022-10-09 21:32:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 960/2304 [I 2022-10-09 21:32:57] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:52 [I 2022-10-09 21:32:57] ...trap/engines/engine.py.229: process: 0.83024 | load: 0.00296 [I 2022-10-09 21:32:57] ...trap/engines/engine.py.230: loss: 0.18856 [I 2022-10-09 21:32:57] ...optimizers/trijoint.py.151: total_norm: 9.406671 [I 2022-10-09 21:33:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 970/2304 [I 2022-10-09 21:33:06] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:46 [I 2022-10-09 21:33:06] ...trap/engines/engine.py.229: process: 0.84999 | load: 0.00294 [I 2022-10-09 21:33:06] ...trap/engines/engine.py.230: loss: 0.15693 [I 2022-10-09 21:33:06] ...optimizers/trijoint.py.151: total_norm: 15.796437 [I 2022-10-09 21:33:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 980/2304 [I 2022-10-09 21:33:14] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:38 [I 2022-10-09 21:33:14] ...trap/engines/engine.py.229: process: 0.84715 | load: 0.00297 [I 2022-10-09 21:33:14] ...trap/engines/engine.py.230: loss: 0.23287 [I 2022-10-09 21:33:14] ...optimizers/trijoint.py.151: total_norm: 18.659668 [I 2022-10-09 21:33:22] ...trap/engines/engine.py.224: train: epoch 55 | batch 990/2304 [I 2022-10-09 21:33:22] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:28 [I 2022-10-09 21:33:22] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00298 [I 2022-10-09 21:33:22] ...trap/engines/engine.py.230: loss: 0.23729 [I 2022-10-09 21:33:22] ...optimizers/trijoint.py.151: total_norm: 25.856148 [I 2022-10-09 21:33:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 1000/2304 [I 2022-10-09 21:33:31] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:14 [I 2022-10-09 21:33:31] ...trap/engines/engine.py.229: process: 0.83011 | load: 0.00300 [I 2022-10-09 21:33:31] ...trap/engines/engine.py.230: loss: 0.20939 [I 2022-10-09 21:33:31] ...optimizers/trijoint.py.151: total_norm: 14.033856 [I 2022-10-09 21:33:39] ...trap/engines/engine.py.224: train: epoch 55 | batch 1010/2304 [I 2022-10-09 21:33:39] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:12 [I 2022-10-09 21:33:39] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00297 [I 2022-10-09 21:33:39] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-09 21:33:39] ...optimizers/trijoint.py.151: total_norm: 23.318739 [I 2022-10-09 21:33:48] ...trap/engines/engine.py.224: train: epoch 55 | batch 1020/2304 [I 2022-10-09 21:33:48] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:18:03 [I 2022-10-09 21:33:48] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00298 [I 2022-10-09 21:33:48] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-09 21:33:48] ...optimizers/trijoint.py.151: total_norm: 16.718052 [I 2022-10-09 21:33:56] ...trap/engines/engine.py.224: train: epoch 55 | batch 1030/2304 [I 2022-10-09 21:33:56] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:55 [I 2022-10-09 21:33:56] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00298 [I 2022-10-09 21:33:56] ...trap/engines/engine.py.230: loss: 0.15295 [I 2022-10-09 21:33:56] ...optimizers/trijoint.py.151: total_norm: 14.874743 [I 2022-10-09 21:34:05] ...trap/engines/engine.py.224: train: epoch 55 | batch 1040/2304 [I 2022-10-09 21:34:05] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:43 [I 2022-10-09 21:34:05] ...trap/engines/engine.py.229: process: 0.83260 | load: 0.00299 [I 2022-10-09 21:34:05] ...trap/engines/engine.py.230: loss: 0.20573 [I 2022-10-09 21:34:05] ...optimizers/trijoint.py.151: total_norm: 37.372269 [I 2022-10-09 21:34:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 1050/2304 [I 2022-10-09 21:34:13] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:39 [I 2022-10-09 21:34:13] ...trap/engines/engine.py.229: process: 0.82555 | load: 0.00298 [I 2022-10-09 21:34:13] ...trap/engines/engine.py.230: loss: 0.18802 [I 2022-10-09 21:34:13] ...optimizers/trijoint.py.151: total_norm: 37.771149 [I 2022-10-09 21:34:22] ...trap/engines/engine.py.224: train: epoch 55 | batch 1060/2304 [I 2022-10-09 21:34:22] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:27 [I 2022-10-09 21:34:22] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00298 [I 2022-10-09 21:34:22] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-09 21:34:22] ...optimizers/trijoint.py.151: total_norm: 22.752808 [I 2022-10-09 21:34:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 1070/2304 [I 2022-10-09 21:34:30] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:17 [I 2022-10-09 21:34:30] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00298 [I 2022-10-09 21:34:30] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-09 21:34:30] ...optimizers/trijoint.py.151: total_norm: 18.074028 [I 2022-10-09 21:34:39] ...trap/engines/engine.py.224: train: epoch 55 | batch 1080/2304 [I 2022-10-09 21:34:39] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:17 [I 2022-10-09 21:34:39] ...trap/engines/engine.py.229: process: 0.85970 | load: 0.00302 [I 2022-10-09 21:34:39] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-09 21:34:39] ...optimizers/trijoint.py.151: total_norm: 13.864597 [I 2022-10-09 21:34:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1090/2304 [I 2022-10-09 21:34:47] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:13 [I 2022-10-09 21:34:47] ...trap/engines/engine.py.229: process: 0.85716 | load: 0.00313 [I 2022-10-09 21:34:47] ...trap/engines/engine.py.230: loss: 0.14310 [I 2022-10-09 21:34:47] ...optimizers/trijoint.py.151: total_norm: 13.900916 [I 2022-10-09 21:34:56] ...trap/engines/engine.py.224: train: epoch 55 | batch 1100/2304 [I 2022-10-09 21:34:56] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:57 [I 2022-10-09 21:34:56] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00308 [I 2022-10-09 21:34:56] ...trap/engines/engine.py.230: loss: 0.12713 [I 2022-10-09 21:34:56] ...optimizers/trijoint.py.151: total_norm: 9.649020 [I 2022-10-09 21:35:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1110/2304 [I 2022-10-09 21:35:04] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:17:00 [I 2022-10-09 21:35:04] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00310 [I 2022-10-09 21:35:04] ...trap/engines/engine.py.230: loss: 0.14176 [I 2022-10-09 21:35:04] ...optimizers/trijoint.py.151: total_norm: 20.978018 [I 2022-10-09 21:35:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 1120/2304 [I 2022-10-09 21:35:13] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:44 [I 2022-10-09 21:35:13] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00313 [I 2022-10-09 21:35:13] ...trap/engines/engine.py.230: loss: 0.17075 [I 2022-10-09 21:35:13] ...optimizers/trijoint.py.151: total_norm: 14.263670 [I 2022-10-09 21:35:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1130/2304 [I 2022-10-09 21:35:21] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:37 [I 2022-10-09 21:35:21] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00308 [I 2022-10-09 21:35:21] ...trap/engines/engine.py.230: loss: 0.17878 [I 2022-10-09 21:35:21] ...optimizers/trijoint.py.151: total_norm: 15.874542 [I 2022-10-09 21:35:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 1140/2304 [I 2022-10-09 21:35:30] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:34 [I 2022-10-09 21:35:30] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00312 [I 2022-10-09 21:35:30] ...trap/engines/engine.py.230: loss: 0.12546 [I 2022-10-09 21:35:30] ...optimizers/trijoint.py.151: total_norm: 10.208200 [I 2022-10-09 21:35:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1150/2304 [I 2022-10-09 21:35:38] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:28 [I 2022-10-09 21:35:38] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00325 [I 2022-10-09 21:35:38] ...trap/engines/engine.py.230: loss: 0.16105 [I 2022-10-09 21:35:38] ...optimizers/trijoint.py.151: total_norm: 15.171100 [I 2022-10-09 21:35:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1160/2304 [I 2022-10-09 21:35:47] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:16:15 [I 2022-10-09 21:35:47] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00314 [I 2022-10-09 21:35:47] ...trap/engines/engine.py.230: loss: 0.15628 [I 2022-10-09 21:35:47] ...optimizers/trijoint.py.151: total_norm: 8.844976 [I 2022-10-09 21:35:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1170/2304 [I 2022-10-09 21:35:55] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:16:05 [I 2022-10-09 21:35:55] ...trap/engines/engine.py.229: process: 0.85836 | load: 0.00314 [I 2022-10-09 21:35:55] ...trap/engines/engine.py.230: loss: 0.16282 [I 2022-10-09 21:35:55] ...optimizers/trijoint.py.151: total_norm: 8.474792 [I 2022-10-09 21:36:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1180/2304 [I 2022-10-09 21:36:04] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:59 [I 2022-10-09 21:36:04] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00301 [I 2022-10-09 21:36:04] ...trap/engines/engine.py.230: loss: 0.15322 [I 2022-10-09 21:36:04] ...optimizers/trijoint.py.151: total_norm: 12.883121 [I 2022-10-09 21:36:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 1190/2304 [I 2022-10-09 21:36:13] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:45 [I 2022-10-09 21:36:13] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00309 [I 2022-10-09 21:36:13] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-09 21:36:13] ...optimizers/trijoint.py.151: total_norm: 19.143324 [I 2022-10-09 21:36:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1200/2304 [I 2022-10-09 21:36:21] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:39 [I 2022-10-09 21:36:21] ...trap/engines/engine.py.229: process: 0.83817 | load: 0.00314 [I 2022-10-09 21:36:21] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-09 21:36:21] ...optimizers/trijoint.py.151: total_norm: 17.566071 [I 2022-10-09 21:36:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 1210/2304 [I 2022-10-09 21:36:30] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:33 [I 2022-10-09 21:36:30] ...trap/engines/engine.py.229: process: 0.85868 | load: 0.00311 [I 2022-10-09 21:36:30] ...trap/engines/engine.py.230: loss: 0.17630 [I 2022-10-09 21:36:30] ...optimizers/trijoint.py.151: total_norm: 24.825167 [I 2022-10-09 21:36:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1220/2304 [I 2022-10-09 21:36:38] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:24 [I 2022-10-09 21:36:38] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00314 [I 2022-10-09 21:36:38] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-09 21:36:38] ...optimizers/trijoint.py.151: total_norm: 60.729748 [I 2022-10-09 21:36:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1230/2304 [I 2022-10-09 21:36:47] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:13 [I 2022-10-09 21:36:47] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00312 [I 2022-10-09 21:36:47] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-09 21:36:47] ...optimizers/trijoint.py.151: total_norm: 11.484560 [I 2022-10-09 21:36:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1240/2304 [I 2022-10-09 21:36:55] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:15:09 [I 2022-10-09 21:36:55] ...trap/engines/engine.py.229: process: 0.85624 | load: 0.00312 [I 2022-10-09 21:36:55] ...trap/engines/engine.py.230: loss: 0.18460 [I 2022-10-09 21:36:55] ...optimizers/trijoint.py.151: total_norm: 22.274794 [I 2022-10-09 21:37:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1250/2304 [I 2022-10-09 21:37:04] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:15:03 [I 2022-10-09 21:37:04] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00311 [I 2022-10-09 21:37:04] ...trap/engines/engine.py.230: loss: 0.14792 [I 2022-10-09 21:37:04] ...optimizers/trijoint.py.151: total_norm: 18.417078 [I 2022-10-09 21:37:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1260/2304 [I 2022-10-09 21:37:12] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:50 [I 2022-10-09 21:37:12] ...trap/engines/engine.py.229: process: 0.85880 | load: 0.00312 [I 2022-10-09 21:37:12] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-09 21:37:12] ...optimizers/trijoint.py.151: total_norm: 8.602145 [I 2022-10-09 21:37:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1270/2304 [I 2022-10-09 21:37:21] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:35 [I 2022-10-09 21:37:21] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00310 [I 2022-10-09 21:37:21] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-09 21:37:21] ...optimizers/trijoint.py.151: total_norm: 14.210666 [I 2022-10-09 21:37:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 1280/2304 [I 2022-10-09 21:37:29] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:27 [I 2022-10-09 21:37:29] ...trap/engines/engine.py.229: process: 0.84556 | load: 0.00313 [I 2022-10-09 21:37:29] ...trap/engines/engine.py.230: loss: 0.17203 [I 2022-10-09 21:37:29] ...optimizers/trijoint.py.151: total_norm: 14.004162 [I 2022-10-09 21:37:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1290/2304 [I 2022-10-09 21:37:38] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:25 [I 2022-10-09 21:37:38] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00312 [I 2022-10-09 21:37:38] ...trap/engines/engine.py.230: loss: 0.16277 [I 2022-10-09 21:37:38] ...optimizers/trijoint.py.151: total_norm: 31.265812 [I 2022-10-09 21:37:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 1300/2304 [I 2022-10-09 21:37:46] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:14 [I 2022-10-09 21:37:46] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00314 [I 2022-10-09 21:37:46] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-09 21:37:46] ...optimizers/trijoint.py.151: total_norm: 12.712486 [I 2022-10-09 21:37:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1310/2304 [I 2022-10-09 21:37:55] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:05 [I 2022-10-09 21:37:55] ...trap/engines/engine.py.229: process: 0.84827 | load: 0.00310 [I 2022-10-09 21:37:55] ...trap/engines/engine.py.230: loss: 0.16731 [I 2022-10-09 21:37:55] ...optimizers/trijoint.py.151: total_norm: 17.405788 [I 2022-10-09 21:38:03] ...trap/engines/engine.py.224: train: epoch 55 | batch 1320/2304 [I 2022-10-09 21:38:03] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:55 [I 2022-10-09 21:38:03] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00314 [I 2022-10-09 21:38:03] ...trap/engines/engine.py.230: loss: 0.16029 [I 2022-10-09 21:38:03] ...optimizers/trijoint.py.151: total_norm: 68.210793 [I 2022-10-09 21:38:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1330/2304 [I 2022-10-09 21:38:12] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:51 [I 2022-10-09 21:38:12] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00314 [I 2022-10-09 21:38:12] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-09 21:38:12] ...optimizers/trijoint.py.151: total_norm: 7.310119 [I 2022-10-09 21:38:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1340/2304 [I 2022-10-09 21:38:21] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:43 [I 2022-10-09 21:38:21] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00313 [I 2022-10-09 21:38:21] ...trap/engines/engine.py.230: loss: 0.12244 [I 2022-10-09 21:38:21] ...optimizers/trijoint.py.151: total_norm: 8.407758 [I 2022-10-09 21:38:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 1350/2304 [I 2022-10-09 21:38:29] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:28 [I 2022-10-09 21:38:29] ...trap/engines/engine.py.229: process: 0.83651 | load: 0.00313 [I 2022-10-09 21:38:29] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-09 21:38:29] ...optimizers/trijoint.py.151: total_norm: 22.263004 [I 2022-10-09 21:38:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1360/2304 [I 2022-10-09 21:38:38] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:26 [I 2022-10-09 21:38:38] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00314 [I 2022-10-09 21:38:38] ...trap/engines/engine.py.230: loss: 0.10813 [I 2022-10-09 21:38:38] ...optimizers/trijoint.py.151: total_norm: 15.605438 [I 2022-10-09 21:38:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 1370/2304 [I 2022-10-09 21:38:46] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:14 [I 2022-10-09 21:38:46] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00303 [I 2022-10-09 21:38:46] ...trap/engines/engine.py.230: loss: 0.14939 [I 2022-10-09 21:38:46] ...optimizers/trijoint.py.151: total_norm: 17.570478 [I 2022-10-09 21:38:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1380/2304 [I 2022-10-09 21:38:55] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:04 [I 2022-10-09 21:38:55] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00312 [I 2022-10-09 21:38:55] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-09 21:38:55] ...optimizers/trijoint.py.151: total_norm: 13.437121 [I 2022-10-09 21:39:03] ...trap/engines/engine.py.224: train: epoch 55 | batch 1390/2304 [I 2022-10-09 21:39:03] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:00 [I 2022-10-09 21:39:03] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00315 [I 2022-10-09 21:39:03] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-09 21:39:03] ...optimizers/trijoint.py.151: total_norm: 34.814167 [I 2022-10-09 21:39:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1400/2304 [I 2022-10-09 21:39:12] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:49 [I 2022-10-09 21:39:12] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00312 [I 2022-10-09 21:39:12] ...trap/engines/engine.py.230: loss: 0.21649 [I 2022-10-09 21:39:12] ...optimizers/trijoint.py.151: total_norm: 21.816689 [I 2022-10-09 21:39:20] ...trap/engines/engine.py.224: train: epoch 55 | batch 1410/2304 [I 2022-10-09 21:39:20] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:45 [I 2022-10-09 21:39:20] ...trap/engines/engine.py.229: process: 0.87024 | load: 0.00313 [I 2022-10-09 21:39:20] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-09 21:39:20] ...optimizers/trijoint.py.151: total_norm: 9.896461 [I 2022-10-09 21:39:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 1420/2304 [I 2022-10-09 21:39:29] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:33 [I 2022-10-09 21:39:29] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00314 [I 2022-10-09 21:39:29] ...trap/engines/engine.py.230: loss: 0.25073 [I 2022-10-09 21:39:29] ...optimizers/trijoint.py.151: total_norm: 27.745226 [I 2022-10-09 21:39:37] ...trap/engines/engine.py.224: train: epoch 55 | batch 1430/2304 [I 2022-10-09 21:39:37] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:19 [I 2022-10-09 21:39:37] ...trap/engines/engine.py.229: process: 0.82539 | load: 0.00301 [I 2022-10-09 21:39:37] ...trap/engines/engine.py.230: loss: 0.12344 [I 2022-10-09 21:39:37] ...optimizers/trijoint.py.151: total_norm: 14.462610 [I 2022-10-09 21:39:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 1440/2304 [I 2022-10-09 21:39:46] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:11 [I 2022-10-09 21:39:46] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00297 [I 2022-10-09 21:39:46] ...trap/engines/engine.py.230: loss: 0.12094 [I 2022-10-09 21:39:46] ...optimizers/trijoint.py.151: total_norm: 12.087502 [I 2022-10-09 21:39:54] ...trap/engines/engine.py.224: train: epoch 55 | batch 1450/2304 [I 2022-10-09 21:39:54] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:02 [I 2022-10-09 21:39:54] ...trap/engines/engine.py.229: process: 0.83811 | load: 0.00292 [I 2022-10-09 21:39:54] ...trap/engines/engine.py.230: loss: 0.10466 [I 2022-10-09 21:39:54] ...optimizers/trijoint.py.151: total_norm: 7.284762 [I 2022-10-09 21:40:03] ...trap/engines/engine.py.224: train: epoch 55 | batch 1460/2304 [I 2022-10-09 21:40:03] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:51 [I 2022-10-09 21:40:03] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00295 [I 2022-10-09 21:40:03] ...trap/engines/engine.py.230: loss: 0.12198 [I 2022-10-09 21:40:03] ...optimizers/trijoint.py.151: total_norm: 12.352600 [I 2022-10-09 21:40:11] ...trap/engines/engine.py.224: train: epoch 55 | batch 1470/2304 [I 2022-10-09 21:40:11] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:43 [I 2022-10-09 21:40:11] ...trap/engines/engine.py.229: process: 0.83688 | load: 0.00294 [I 2022-10-09 21:40:11] ...trap/engines/engine.py.230: loss: 0.14242 [I 2022-10-09 21:40:11] ...optimizers/trijoint.py.151: total_norm: 9.360404 [I 2022-10-09 21:40:20] ...trap/engines/engine.py.224: train: epoch 55 | batch 1480/2304 [I 2022-10-09 21:40:20] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:35 [I 2022-10-09 21:40:20] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00295 [I 2022-10-09 21:40:20] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-09 21:40:20] ...optimizers/trijoint.py.151: total_norm: 15.822109 [I 2022-10-09 21:40:28] ...trap/engines/engine.py.224: train: epoch 55 | batch 1490/2304 [I 2022-10-09 21:40:28] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:29 [I 2022-10-09 21:40:28] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00295 [I 2022-10-09 21:40:28] ...trap/engines/engine.py.230: loss: 0.17211 [I 2022-10-09 21:40:28] ...optimizers/trijoint.py.151: total_norm: 20.829762 [I 2022-10-09 21:40:37] ...trap/engines/engine.py.224: train: epoch 55 | batch 1500/2304 [I 2022-10-09 21:40:37] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:17 [I 2022-10-09 21:40:37] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00296 [I 2022-10-09 21:40:37] ...trap/engines/engine.py.230: loss: 0.14633 [I 2022-10-09 21:40:37] ...optimizers/trijoint.py.151: total_norm: 15.494185 [I 2022-10-09 21:40:45] ...trap/engines/engine.py.224: train: epoch 55 | batch 1510/2304 [I 2022-10-09 21:40:45] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:08 [I 2022-10-09 21:40:45] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00294 [I 2022-10-09 21:40:45] ...trap/engines/engine.py.230: loss: 0.18839 [I 2022-10-09 21:40:45] ...optimizers/trijoint.py.151: total_norm: 15.262959 [I 2022-10-09 21:40:54] ...trap/engines/engine.py.224: train: epoch 55 | batch 1520/2304 [I 2022-10-09 21:40:54] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:00 [I 2022-10-09 21:40:54] ...trap/engines/engine.py.229: process: 0.83067 | load: 0.00293 [I 2022-10-09 21:40:54] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-09 21:40:54] ...optimizers/trijoint.py.151: total_norm: 10.227709 [I 2022-10-09 21:41:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 1530/2304 [I 2022-10-09 21:41:02] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:53 [I 2022-10-09 21:41:02] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00293 [I 2022-10-09 21:41:02] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-09 21:41:02] ...optimizers/trijoint.py.151: total_norm: 16.957054 [I 2022-10-09 21:41:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 1540/2304 [I 2022-10-09 21:41:10] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:43 [I 2022-10-09 21:41:10] ...trap/engines/engine.py.229: process: 0.83125 | load: 0.00302 [I 2022-10-09 21:41:10] ...trap/engines/engine.py.230: loss: 0.11593 [I 2022-10-09 21:41:10] ...optimizers/trijoint.py.151: total_norm: 28.684370 [I 2022-10-09 21:41:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 1550/2304 [I 2022-10-09 21:41:19] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:34 [I 2022-10-09 21:41:19] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00296 [I 2022-10-09 21:41:19] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-09 21:41:19] ...optimizers/trijoint.py.151: total_norm: 18.139685 [I 2022-10-09 21:41:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 1560/2304 [I 2022-10-09 21:41:27] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:27 [I 2022-10-09 21:41:27] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00294 [I 2022-10-09 21:41:27] ...trap/engines/engine.py.230: loss: 0.19711 [I 2022-10-09 21:41:27] ...optimizers/trijoint.py.151: total_norm: 20.172367 [I 2022-10-09 21:41:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 1570/2304 [I 2022-10-09 21:41:36] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:18 [I 2022-10-09 21:41:36] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00290 [I 2022-10-09 21:41:36] ...trap/engines/engine.py.230: loss: 0.13187 [I 2022-10-09 21:41:36] ...optimizers/trijoint.py.151: total_norm: 12.150766 [I 2022-10-09 21:41:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 1580/2304 [I 2022-10-09 21:41:44] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:10 [I 2022-10-09 21:41:44] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00296 [I 2022-10-09 21:41:44] ...trap/engines/engine.py.230: loss: 0.13002 [I 2022-10-09 21:41:44] ...optimizers/trijoint.py.151: total_norm: 8.298244 [I 2022-10-09 21:41:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 1590/2304 [I 2022-10-09 21:41:53] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:01 [I 2022-10-09 21:41:53] ...trap/engines/engine.py.229: process: 0.85112 | load: 0.00293 [I 2022-10-09 21:41:53] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-09 21:41:53] ...optimizers/trijoint.py.151: total_norm: 10.768165 [I 2022-10-09 21:42:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 1600/2304 [I 2022-10-09 21:42:01] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:54 [I 2022-10-09 21:42:01] ...trap/engines/engine.py.229: process: 0.83959 | load: 0.00294 [I 2022-10-09 21:42:01] ...trap/engines/engine.py.230: loss: 0.17331 [I 2022-10-09 21:42:01] ...optimizers/trijoint.py.151: total_norm: 11.480570 [I 2022-10-09 21:42:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 1610/2304 [I 2022-10-09 21:42:10] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:44 [I 2022-10-09 21:42:10] ...trap/engines/engine.py.229: process: 0.82609 | load: 0.00296 [I 2022-10-09 21:42:10] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-09 21:42:10] ...optimizers/trijoint.py.151: total_norm: 10.343071 [I 2022-10-09 21:42:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 1620/2304 [I 2022-10-09 21:42:18] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:37 [I 2022-10-09 21:42:18] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00295 [I 2022-10-09 21:42:18] ...trap/engines/engine.py.230: loss: 0.09952 [I 2022-10-09 21:42:18] ...optimizers/trijoint.py.151: total_norm: 9.954457 [I 2022-10-09 21:42:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 1630/2304 [I 2022-10-09 21:42:26] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:27 [I 2022-10-09 21:42:26] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00297 [I 2022-10-09 21:42:26] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-09 21:42:26] ...optimizers/trijoint.py.151: total_norm: 8.085345 [I 2022-10-09 21:42:35] ...trap/engines/engine.py.224: train: epoch 55 | batch 1640/2304 [I 2022-10-09 21:42:35] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:20 [I 2022-10-09 21:42:35] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00296 [I 2022-10-09 21:42:35] ...trap/engines/engine.py.230: loss: 0.18289 [I 2022-10-09 21:42:35] ...optimizers/trijoint.py.151: total_norm: 22.195240 [I 2022-10-09 21:42:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 1650/2304 [I 2022-10-09 21:42:43] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:09 [I 2022-10-09 21:42:43] ...trap/engines/engine.py.229: process: 0.83374 | load: 0.00295 [I 2022-10-09 21:42:43] ...trap/engines/engine.py.230: loss: 0.15614 [I 2022-10-09 21:42:43] ...optimizers/trijoint.py.151: total_norm: 14.343225 [I 2022-10-09 21:42:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 1660/2304 [I 2022-10-09 21:42:52] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:04 [I 2022-10-09 21:42:52] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00295 [I 2022-10-09 21:42:52] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-09 21:42:52] ...optimizers/trijoint.py.151: total_norm: 16.488337 [I 2022-10-09 21:43:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 1670/2304 [I 2022-10-09 21:43:00] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:54 [I 2022-10-09 21:43:00] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00295 [I 2022-10-09 21:43:00] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-09 21:43:00] ...optimizers/trijoint.py.151: total_norm: 38.412281 [I 2022-10-09 21:43:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 1680/2304 [I 2022-10-09 21:43:09] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:47 [I 2022-10-09 21:43:09] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00297 [I 2022-10-09 21:43:09] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-09 21:43:09] ...optimizers/trijoint.py.151: total_norm: 9.558851 [I 2022-10-09 21:43:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1690/2304 [I 2022-10-09 21:43:17] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:41 [I 2022-10-09 21:43:17] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00295 [I 2022-10-09 21:43:17] ...trap/engines/engine.py.230: loss: 0.16885 [I 2022-10-09 21:43:17] ...optimizers/trijoint.py.151: total_norm: 15.384943 [I 2022-10-09 21:43:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 1700/2304 [I 2022-10-09 21:43:26] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:29 [I 2022-10-09 21:43:26] ...trap/engines/engine.py.229: process: 0.83287 | load: 0.00296 [I 2022-10-09 21:43:26] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-09 21:43:26] ...optimizers/trijoint.py.151: total_norm: 10.452481 [I 2022-10-09 21:43:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1710/2304 [I 2022-10-09 21:43:34] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:21 [I 2022-10-09 21:43:34] ...trap/engines/engine.py.229: process: 0.82787 | load: 0.00299 [I 2022-10-09 21:43:34] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-09 21:43:34] ...optimizers/trijoint.py.151: total_norm: 13.858077 [I 2022-10-09 21:43:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 1720/2304 [I 2022-10-09 21:43:43] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:12 [I 2022-10-09 21:43:43] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00319 [I 2022-10-09 21:43:43] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-09 21:43:43] ...optimizers/trijoint.py.151: total_norm: 27.707832 [I 2022-10-09 21:43:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1730/2304 [I 2022-10-09 21:43:51] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:09 [I 2022-10-09 21:43:51] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00312 [I 2022-10-09 21:43:51] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-09 21:43:51] ...optimizers/trijoint.py.151: total_norm: 13.159457 [I 2022-10-09 21:44:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 1740/2304 [I 2022-10-09 21:44:00] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:08:03 [I 2022-10-09 21:44:00] ...trap/engines/engine.py.229: process: 0.86199 | load: 0.00313 [I 2022-10-09 21:44:00] ...trap/engines/engine.py.230: loss: 0.17769 [I 2022-10-09 21:44:00] ...optimizers/trijoint.py.151: total_norm: 26.287317 [I 2022-10-09 21:44:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1750/2304 [I 2022-10-09 21:44:08] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:53 [I 2022-10-09 21:44:08] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00315 [I 2022-10-09 21:44:08] ...trap/engines/engine.py.230: loss: 0.17615 [I 2022-10-09 21:44:08] ...optimizers/trijoint.py.151: total_norm: 14.992702 [I 2022-10-09 21:44:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1760/2304 [I 2022-10-09 21:44:17] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:43 [I 2022-10-09 21:44:17] ...trap/engines/engine.py.229: process: 0.84711 | load: 0.00312 [I 2022-10-09 21:44:17] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-09 21:44:17] ...optimizers/trijoint.py.151: total_norm: 14.187603 [I 2022-10-09 21:44:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1770/2304 [I 2022-10-09 21:44:25] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:32 [I 2022-10-09 21:44:25] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00311 [I 2022-10-09 21:44:25] ...trap/engines/engine.py.230: loss: 0.10372 [I 2022-10-09 21:44:25] ...optimizers/trijoint.py.151: total_norm: 12.933865 [I 2022-10-09 21:44:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1780/2304 [I 2022-10-09 21:44:34] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:24 [I 2022-10-09 21:44:34] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00310 [I 2022-10-09 21:44:34] ...trap/engines/engine.py.230: loss: 0.23245 [I 2022-10-09 21:44:34] ...optimizers/trijoint.py.151: total_norm: 10.611854 [I 2022-10-09 21:44:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 1790/2304 [I 2022-10-09 21:44:43] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:18 [I 2022-10-09 21:44:43] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00310 [I 2022-10-09 21:44:43] ...trap/engines/engine.py.230: loss: 0.16863 [I 2022-10-09 21:44:43] ...optimizers/trijoint.py.151: total_norm: 16.168682 [I 2022-10-09 21:44:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1800/2304 [I 2022-10-09 21:44:51] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:07:08 [I 2022-10-09 21:44:51] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00312 [I 2022-10-09 21:44:51] ...trap/engines/engine.py.230: loss: 0.15869 [I 2022-10-09 21:44:51] ...optimizers/trijoint.py.151: total_norm: 32.129528 [I 2022-10-09 21:45:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 1810/2304 [I 2022-10-09 21:45:00] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:59 [I 2022-10-09 21:45:00] ...trap/engines/engine.py.229: process: 0.83979 | load: 0.00312 [I 2022-10-09 21:45:00] ...trap/engines/engine.py.230: loss: 0.15667 [I 2022-10-09 21:45:00] ...optimizers/trijoint.py.151: total_norm: 14.757802 [I 2022-10-09 21:45:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1820/2304 [I 2022-10-09 21:45:08] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:53 [I 2022-10-09 21:45:08] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00311 [I 2022-10-09 21:45:08] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-09 21:45:08] ...optimizers/trijoint.py.151: total_norm: 24.853659 [I 2022-10-09 21:45:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1830/2304 [I 2022-10-09 21:45:17] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:43 [I 2022-10-09 21:45:17] ...trap/engines/engine.py.229: process: 0.84701 | load: 0.00312 [I 2022-10-09 21:45:17] ...trap/engines/engine.py.230: loss: 0.09858 [I 2022-10-09 21:45:17] ...optimizers/trijoint.py.151: total_norm: 10.258294 [I 2022-10-09 21:45:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1840/2304 [I 2022-10-09 21:45:25] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:34 [I 2022-10-09 21:45:25] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00311 [I 2022-10-09 21:45:25] ...trap/engines/engine.py.230: loss: 0.16052 [I 2022-10-09 21:45:25] ...optimizers/trijoint.py.151: total_norm: 13.416087 [I 2022-10-09 21:45:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1850/2304 [I 2022-10-09 21:45:34] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:27 [I 2022-10-09 21:45:34] ...trap/engines/engine.py.229: process: 0.86018 | load: 0.00310 [I 2022-10-09 21:45:34] ...trap/engines/engine.py.230: loss: 0.11745 [I 2022-10-09 21:45:34] ...optimizers/trijoint.py.151: total_norm: 9.459952 [I 2022-10-09 21:45:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 1860/2304 [I 2022-10-09 21:45:42] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:19 [I 2022-10-09 21:45:42] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00311 [I 2022-10-09 21:45:42] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-09 21:45:42] ...optimizers/trijoint.py.151: total_norm: 12.852263 [I 2022-10-09 21:45:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1870/2304 [I 2022-10-09 21:45:51] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:10 [I 2022-10-09 21:45:51] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00311 [I 2022-10-09 21:45:51] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-09 21:45:51] ...optimizers/trijoint.py.151: total_norm: 19.288246 [I 2022-10-09 21:45:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 1880/2304 [I 2022-10-09 21:45:59] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:03 [I 2022-10-09 21:45:59] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00312 [I 2022-10-09 21:45:59] ...trap/engines/engine.py.230: loss: 0.13020 [I 2022-10-09 21:45:59] ...optimizers/trijoint.py.151: total_norm: 11.820396 [I 2022-10-09 21:46:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1890/2304 [I 2022-10-09 21:46:08] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:53 [I 2022-10-09 21:46:08] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00311 [I 2022-10-09 21:46:08] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-09 21:46:08] ...optimizers/trijoint.py.151: total_norm: 13.287471 [I 2022-10-09 21:46:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1900/2304 [I 2022-10-09 21:46:17] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:45 [I 2022-10-09 21:46:17] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00313 [I 2022-10-09 21:46:17] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-09 21:46:17] ...optimizers/trijoint.py.151: total_norm: 18.307245 [I 2022-10-09 21:46:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1910/2304 [I 2022-10-09 21:46:25] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:35 [I 2022-10-09 21:46:25] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00312 [I 2022-10-09 21:46:25] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-09 21:46:25] ...optimizers/trijoint.py.151: total_norm: 14.136921 [I 2022-10-09 21:46:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1920/2304 [I 2022-10-09 21:46:34] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:28 [I 2022-10-09 21:46:34] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00315 [I 2022-10-09 21:46:34] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-09 21:46:34] ...optimizers/trijoint.py.151: total_norm: 12.612301 [I 2022-10-09 21:46:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 1930/2304 [I 2022-10-09 21:46:42] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:20 [I 2022-10-09 21:46:42] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00313 [I 2022-10-09 21:46:42] ...trap/engines/engine.py.230: loss: 0.17842 [I 2022-10-09 21:46:42] ...optimizers/trijoint.py.151: total_norm: 19.578650 [I 2022-10-09 21:46:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1940/2304 [I 2022-10-09 21:46:51] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:12 [I 2022-10-09 21:46:51] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00314 [I 2022-10-09 21:46:51] ...trap/engines/engine.py.230: loss: 0.13495 [I 2022-10-09 21:46:51] ...optimizers/trijoint.py.151: total_norm: 11.317023 [I 2022-10-09 21:46:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 1950/2304 [I 2022-10-09 21:46:59] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:01 [I 2022-10-09 21:46:59] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00314 [I 2022-10-09 21:46:59] ...trap/engines/engine.py.230: loss: 0.13326 [I 2022-10-09 21:46:59] ...optimizers/trijoint.py.151: total_norm: 13.057361 [I 2022-10-09 21:47:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1960/2304 [I 2022-10-09 21:47:08] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:53 [I 2022-10-09 21:47:08] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00310 [I 2022-10-09 21:47:08] ...trap/engines/engine.py.230: loss: 0.12068 [I 2022-10-09 21:47:08] ...optimizers/trijoint.py.151: total_norm: 13.035425 [I 2022-10-09 21:47:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1970/2304 [I 2022-10-09 21:47:17] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:45 [I 2022-10-09 21:47:17] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00310 [I 2022-10-09 21:47:17] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-09 21:47:17] ...optimizers/trijoint.py.151: total_norm: 8.685065 [I 2022-10-09 21:47:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1980/2304 [I 2022-10-09 21:47:25] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:36 [I 2022-10-09 21:47:25] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00312 [I 2022-10-09 21:47:25] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-09 21:47:25] ...optimizers/trijoint.py.151: total_norm: 10.768884 [I 2022-10-09 21:47:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1990/2304 [I 2022-10-09 21:47:34] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:27 [I 2022-10-09 21:47:34] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00313 [I 2022-10-09 21:47:34] ...trap/engines/engine.py.230: loss: 0.10739 [I 2022-10-09 21:47:34] ...optimizers/trijoint.py.151: total_norm: 11.497980 [I 2022-10-09 21:47:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 2000/2304 [I 2022-10-09 21:47:42] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:19 [I 2022-10-09 21:47:42] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00309 [I 2022-10-09 21:47:42] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-09 21:47:42] ...optimizers/trijoint.py.151: total_norm: 14.852137 [I 2022-10-09 21:47:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 2010/2304 [I 2022-10-09 21:47:51] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:09 [I 2022-10-09 21:47:51] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00303 [I 2022-10-09 21:47:51] ...trap/engines/engine.py.230: loss: 0.17072 [I 2022-10-09 21:47:51] ...optimizers/trijoint.py.151: total_norm: 20.175873 [I 2022-10-09 21:47:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 2020/2304 [I 2022-10-09 21:47:59] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:58 [I 2022-10-09 21:47:59] ...trap/engines/engine.py.229: process: 0.83386 | load: 0.00298 [I 2022-10-09 21:47:59] ...trap/engines/engine.py.230: loss: 0.10712 [I 2022-10-09 21:47:59] ...optimizers/trijoint.py.151: total_norm: 15.087113 [I 2022-10-09 21:48:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 2030/2304 [I 2022-10-09 21:48:08] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:52 [I 2022-10-09 21:48:08] ...trap/engines/engine.py.229: process: 0.83746 | load: 0.00295 [I 2022-10-09 21:48:08] ...trap/engines/engine.py.230: loss: 0.10259 [I 2022-10-09 21:48:08] ...optimizers/trijoint.py.151: total_norm: 8.474656 [I 2022-10-09 21:48:16] ...trap/engines/engine.py.224: train: epoch 55 | batch 2040/2304 [I 2022-10-09 21:48:16] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:43 [I 2022-10-09 21:48:16] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00304 [I 2022-10-09 21:48:16] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-09 21:48:16] ...optimizers/trijoint.py.151: total_norm: 23.668232 [I 2022-10-09 21:48:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 2050/2304 [I 2022-10-09 21:48:25] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:35 [I 2022-10-09 21:48:25] ...trap/engines/engine.py.229: process: 0.84745 | load: 0.00301 [I 2022-10-09 21:48:25] ...trap/engines/engine.py.230: loss: 0.14343 [I 2022-10-09 21:48:25] ...optimizers/trijoint.py.151: total_norm: 15.970329 [I 2022-10-09 21:48:33] ...trap/engines/engine.py.224: train: epoch 55 | batch 2060/2304 [I 2022-10-09 21:48:33] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:25 [I 2022-10-09 21:48:33] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00308 [I 2022-10-09 21:48:33] ...trap/engines/engine.py.230: loss: 0.13683 [I 2022-10-09 21:48:33] ...optimizers/trijoint.py.151: total_norm: 11.118680 [I 2022-10-09 21:48:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 2070/2304 [I 2022-10-09 21:48:41] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:16 [I 2022-10-09 21:48:41] ...trap/engines/engine.py.229: process: 0.83147 | load: 0.00299 [I 2022-10-09 21:48:41] ...trap/engines/engine.py.230: loss: 0.11355 [I 2022-10-09 21:48:41] ...optimizers/trijoint.py.151: total_norm: 9.100017 [I 2022-10-09 21:48:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 2080/2304 [I 2022-10-09 21:48:50] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:08 [I 2022-10-09 21:48:50] ...trap/engines/engine.py.229: process: 0.82400 | load: 0.00301 [I 2022-10-09 21:48:50] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-09 21:48:50] ...optimizers/trijoint.py.151: total_norm: 25.668232 [I 2022-10-09 21:48:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 2090/2304 [I 2022-10-09 21:48:58] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:00 [I 2022-10-09 21:48:58] ...trap/engines/engine.py.229: process: 0.83823 | load: 0.00294 [I 2022-10-09 21:48:58] ...trap/engines/engine.py.230: loss: 0.16065 [I 2022-10-09 21:48:58] ...optimizers/trijoint.py.151: total_norm: 9.775441 [I 2022-10-09 21:49:07] ...trap/engines/engine.py.224: train: epoch 55 | batch 2100/2304 [I 2022-10-09 21:49:07] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:52 [I 2022-10-09 21:49:07] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00298 [I 2022-10-09 21:49:07] ...trap/engines/engine.py.230: loss: 0.19655 [I 2022-10-09 21:49:07] ...optimizers/trijoint.py.151: total_norm: 10.253791 [I 2022-10-09 21:49:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 2110/2304 [I 2022-10-09 21:49:15] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:44 [I 2022-10-09 21:49:15] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00299 [I 2022-10-09 21:49:15] ...trap/engines/engine.py.230: loss: 0.18319 [I 2022-10-09 21:49:15] ...optimizers/trijoint.py.151: total_norm: 20.644485 [I 2022-10-09 21:49:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 2120/2304 [I 2022-10-09 21:49:24] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:35 [I 2022-10-09 21:49:24] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00293 [I 2022-10-09 21:49:24] ...trap/engines/engine.py.230: loss: 0.20982 [I 2022-10-09 21:49:24] ...optimizers/trijoint.py.151: total_norm: 13.939368 [I 2022-10-09 21:49:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 2130/2304 [I 2022-10-09 21:49:32] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:27 [I 2022-10-09 21:49:32] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00301 [I 2022-10-09 21:49:32] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-09 21:49:32] ...optimizers/trijoint.py.151: total_norm: 9.207355 [I 2022-10-09 21:49:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 2140/2304 [I 2022-10-09 21:49:41] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:18 [I 2022-10-09 21:49:41] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00298 [I 2022-10-09 21:49:41] ...trap/engines/engine.py.230: loss: 0.13835 [I 2022-10-09 21:49:41] ...optimizers/trijoint.py.151: total_norm: 43.094841 [I 2022-10-09 21:49:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 2150/2304 [I 2022-10-09 21:49:49] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:11 [I 2022-10-09 21:49:49] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00302 [I 2022-10-09 21:49:49] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-09 21:49:49] ...optimizers/trijoint.py.151: total_norm: 11.890979 [I 2022-10-09 21:49:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 2160/2304 [I 2022-10-09 21:49:58] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:03 [I 2022-10-09 21:49:58] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00297 [I 2022-10-09 21:49:58] ...trap/engines/engine.py.230: loss: 0.18406 [I 2022-10-09 21:49:58] ...optimizers/trijoint.py.151: total_norm: 23.704430 [I 2022-10-09 21:50:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 2170/2304 [I 2022-10-09 21:50:06] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:53 [I 2022-10-09 21:50:06] ...trap/engines/engine.py.229: process: 0.82739 | load: 0.00301 [I 2022-10-09 21:50:06] ...trap/engines/engine.py.230: loss: 0.19200 [I 2022-10-09 21:50:07] ...optimizers/trijoint.py.151: total_norm: 11.089796 [I 2022-10-09 21:50:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 2180/2304 [I 2022-10-09 21:50:15] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:45 [I 2022-10-09 21:50:15] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00294 [I 2022-10-09 21:50:15] ...trap/engines/engine.py.230: loss: 0.21562 [I 2022-10-09 21:50:15] ...optimizers/trijoint.py.151: total_norm: 26.215717 [I 2022-10-09 21:50:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 2190/2304 [I 2022-10-09 21:50:23] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:35 [I 2022-10-09 21:50:23] ...trap/engines/engine.py.229: process: 0.83104 | load: 0.00299 [I 2022-10-09 21:50:23] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-09 21:50:23] ...optimizers/trijoint.py.151: total_norm: 7.849552 [I 2022-10-09 21:50:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 2200/2304 [I 2022-10-09 21:50:32] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:28 [I 2022-10-09 21:50:32] ...trap/engines/engine.py.229: process: 0.86071 | load: 0.00318 [I 2022-10-09 21:50:32] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-09 21:50:32] ...optimizers/trijoint.py.151: total_norm: 20.432459 [I 2022-10-09 21:50:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 2210/2304 [I 2022-10-09 21:50:41] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:20 [I 2022-10-09 21:50:41] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00316 [I 2022-10-09 21:50:41] ...trap/engines/engine.py.230: loss: 0.20774 [I 2022-10-09 21:50:41] ...optimizers/trijoint.py.151: total_norm: 32.135170 [I 2022-10-09 21:50:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 2220/2304 [I 2022-10-09 21:50:49] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:11 [I 2022-10-09 21:50:49] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00311 [I 2022-10-09 21:50:49] ...trap/engines/engine.py.230: loss: 0.17514 [I 2022-10-09 21:50:49] ...optimizers/trijoint.py.151: total_norm: 32.865116 [I 2022-10-09 21:50:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 2230/2304 [I 2022-10-09 21:50:58] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:02 [I 2022-10-09 21:50:58] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00314 [I 2022-10-09 21:50:58] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-09 21:50:58] ...optimizers/trijoint.py.151: total_norm: 11.926266 [I 2022-10-09 21:51:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 2240/2304 [I 2022-10-09 21:51:06] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:00:54 [I 2022-10-09 21:51:06] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00313 [I 2022-10-09 21:51:06] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-09 21:51:06] ...optimizers/trijoint.py.151: total_norm: 17.580799 [I 2022-10-09 21:51:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 2250/2304 [I 2022-10-09 21:51:15] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:46 [I 2022-10-09 21:51:15] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00311 [I 2022-10-09 21:51:15] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-09 21:51:15] ...optimizers/trijoint.py.151: total_norm: 14.290303 [I 2022-10-09 21:51:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 2260/2304 [I 2022-10-09 21:51:23] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:37 [I 2022-10-09 21:51:23] ...trap/engines/engine.py.229: process: 0.87752 | load: 0.00311 [I 2022-10-09 21:51:23] ...trap/engines/engine.py.230: loss: 0.17834 [I 2022-10-09 21:51:23] ...optimizers/trijoint.py.151: total_norm: 22.752954 [I 2022-10-09 21:51:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 2270/2304 [I 2022-10-09 21:51:32] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:29 [I 2022-10-09 21:51:32] ...trap/engines/engine.py.229: process: 0.86270 | load: 0.00313 [I 2022-10-09 21:51:32] ...trap/engines/engine.py.230: loss: 0.15841 [I 2022-10-09 21:51:32] ...optimizers/trijoint.py.151: total_norm: 28.482588 [I 2022-10-09 21:51:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 2280/2304 [I 2022-10-09 21:51:40] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:20 [I 2022-10-09 21:51:40] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00312 [I 2022-10-09 21:51:40] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-09 21:51:40] ...optimizers/trijoint.py.151: total_norm: 9.995177 [I 2022-10-09 21:51:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 2290/2304 [I 2022-10-09 21:51:49] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:12 [I 2022-10-09 21:51:49] ...trap/engines/engine.py.229: process: 0.86419 | load: 0.00311 [I 2022-10-09 21:51:49] ...trap/engines/engine.py.230: loss: 0.13360 [I 2022-10-09 21:51:49] ...optimizers/trijoint.py.151: total_norm: 13.269173 [I 2022-10-09 21:51:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 2300/2304 [I 2022-10-09 21:51:57] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:03 [I 2022-10-09 21:51:57] ...trap/engines/engine.py.229: process: 0.82724 | load: 0.00020 [I 2022-10-09 21:51:57] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-09 21:51:57] ...optimizers/trijoint.py.151: total_norm: 9.141413 [I 2022-10-09 21:52:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 2304/2304 [I 2022-10-09 21:52:01] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:00 [I 2022-10-09 21:52:01] ...trap/engines/engine.py.229: process: 0.83879 | load: 0.00019 [I 2022-10-09 21:52:01] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-09 21:52:01] ...optimizers/trijoint.py.151: total_norm: 9.704878 [S 2022-10-09 21:52:02] ...trap/engines/engine.py.129: train_epoch.epoch: 55 [S 2022-10-09 21:52:02] ...trap/engines/engine.py.129: train_epoch.loss: 0.15175978955762245 [I 2022-10-09 21:52:02] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 21:52:03] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 55 [I 2022-10-09 21:52:03] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 21:52:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 21:52:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 21:52:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 21:52:07] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 21:52:07] ...trap/engines/engine.py.321: eval: epoch 55 | batch 0/510 [I 2022-10-09 21:52:07] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:06 [I 2022-10-09 21:52:07] ...trap/engines/engine.py.326: process: 4.36642 | load: 3.38601 [I 2022-10-09 21:52:12] ...trap/engines/engine.py.321: eval: epoch 55 | batch 10/510 [I 2022-10-09 21:52:12] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:07 [I 2022-10-09 21:52:12] ...trap/engines/engine.py.326: process: 0.43679 | load: 0.00020 [I 2022-10-09 21:52:17] ...trap/engines/engine.py.321: eval: epoch 55 | batch 20/510 [I 2022-10-09 21:52:17] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:13 [I 2022-10-09 21:52:17] ...trap/engines/engine.py.326: process: 0.46725 | load: 0.00021 [I 2022-10-09 21:52:21] ...trap/engines/engine.py.321: eval: epoch 55 | batch 30/510 [I 2022-10-09 21:52:21] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:03:35 [I 2022-10-09 21:52:21] ...trap/engines/engine.py.326: process: 0.41832 | load: 0.00021 [I 2022-10-09 21:52:26] ...trap/engines/engine.py.321: eval: epoch 55 | batch 40/510 [I 2022-10-09 21:52:26] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:03:27 [I 2022-10-09 21:52:26] ...trap/engines/engine.py.326: process: 0.45254 | load: 0.00020 [I 2022-10-09 21:52:30] ...trap/engines/engine.py.321: eval: epoch 55 | batch 50/510 [I 2022-10-09 21:52:30] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:03:29 [I 2022-10-09 21:52:30] ...trap/engines/engine.py.326: process: 0.44219 | load: 0.00021 [I 2022-10-09 21:52:35] ...trap/engines/engine.py.321: eval: epoch 55 | batch 60/510 [I 2022-10-09 21:52:35] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:34 [I 2022-10-09 21:52:35] ...trap/engines/engine.py.326: process: 0.45249 | load: 0.00020 [I 2022-10-09 21:52:40] ...trap/engines/engine.py.321: eval: epoch 55 | batch 70/510 [I 2022-10-09 21:52:40] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:58 [I 2022-10-09 21:52:40] ...trap/engines/engine.py.326: process: 0.59260 | load: 0.00020 [I 2022-10-09 21:52:46] ...trap/engines/engine.py.321: eval: epoch 55 | batch 80/510 [I 2022-10-09 21:52:46] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:04:14 [I 2022-10-09 21:52:46] ...trap/engines/engine.py.326: process: 0.49753 | load: 0.00021 [I 2022-10-09 21:52:52] ...trap/engines/engine.py.321: eval: epoch 55 | batch 90/510 [I 2022-10-09 21:52:52] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:44 [I 2022-10-09 21:52:52] ...trap/engines/engine.py.326: process: 0.51736 | load: 0.00021 [I 2022-10-09 21:52:57] ...trap/engines/engine.py.321: eval: epoch 55 | batch 100/510 [I 2022-10-09 21:52:57] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:16 [I 2022-10-09 21:52:57] ...trap/engines/engine.py.326: process: 0.30841 | load: 0.00021 [I 2022-10-09 21:53:00] ...trap/engines/engine.py.321: eval: epoch 55 | batch 110/510 [I 2022-10-09 21:53:00] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:14 [I 2022-10-09 21:53:00] ...trap/engines/engine.py.326: process: 0.32409 | load: 0.00020 [I 2022-10-09 21:53:03] ...trap/engines/engine.py.321: eval: epoch 55 | batch 120/510 [I 2022-10-09 21:53:03] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:04 [I 2022-10-09 21:53:03] ...trap/engines/engine.py.326: process: 0.31384 | load: 0.00020 [I 2022-10-09 21:53:06] ...trap/engines/engine.py.321: eval: epoch 55 | batch 130/510 [I 2022-10-09 21:53:06] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:01 [I 2022-10-09 21:53:06] ...trap/engines/engine.py.326: process: 0.32187 | load: 0.00020 [I 2022-10-09 21:53:09] ...trap/engines/engine.py.321: eval: epoch 55 | batch 140/510 [I 2022-10-09 21:53:09] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:59 [I 2022-10-09 21:53:09] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00020 [I 2022-10-09 21:53:13] ...trap/engines/engine.py.321: eval: epoch 55 | batch 150/510 [I 2022-10-09 21:53:13] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:55 [I 2022-10-09 21:53:13] ...trap/engines/engine.py.326: process: 0.32163 | load: 0.00019 [I 2022-10-09 21:53:16] ...trap/engines/engine.py.321: eval: epoch 55 | batch 160/510 [I 2022-10-09 21:53:16] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:51 [I 2022-10-09 21:53:16] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00019 [I 2022-10-09 21:53:19] ...trap/engines/engine.py.321: eval: epoch 55 | batch 170/510 [I 2022-10-09 21:53:19] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:48 [I 2022-10-09 21:53:19] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00020 [I 2022-10-09 21:53:22] ...trap/engines/engine.py.321: eval: epoch 55 | batch 180/510 [I 2022-10-09 21:53:22] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-09 21:53:22] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00019 [I 2022-10-09 21:53:25] ...trap/engines/engine.py.321: eval: epoch 55 | batch 190/510 [I 2022-10-09 21:53:25] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:42 [I 2022-10-09 21:53:25] ...trap/engines/engine.py.326: process: 0.32194 | load: 0.00019 [I 2022-10-09 21:53:29] ...trap/engines/engine.py.321: eval: epoch 55 | batch 200/510 [I 2022-10-09 21:53:29] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:39 [I 2022-10-09 21:53:29] ...trap/engines/engine.py.326: process: 0.32247 | load: 0.00024 [I 2022-10-09 21:53:32] ...trap/engines/engine.py.321: eval: epoch 55 | batch 210/510 [I 2022-10-09 21:53:32] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:36 [I 2022-10-09 21:53:32] ...trap/engines/engine.py.326: process: 0.32401 | load: 0.00021 [I 2022-10-09 21:53:35] ...trap/engines/engine.py.321: eval: epoch 55 | batch 220/510 [I 2022-10-09 21:53:35] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:33 [I 2022-10-09 21:53:35] ...trap/engines/engine.py.326: process: 0.32471 | load: 0.00021 [I 2022-10-09 21:53:38] ...trap/engines/engine.py.321: eval: epoch 55 | batch 230/510 [I 2022-10-09 21:53:38] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:30 [I 2022-10-09 21:53:38] ...trap/engines/engine.py.326: process: 0.32403 | load: 0.00020 [I 2022-10-09 21:53:42] ...trap/engines/engine.py.321: eval: epoch 55 | batch 240/510 [I 2022-10-09 21:53:42] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-09 21:53:42] ...trap/engines/engine.py.326: process: 0.31946 | load: 0.00022 [I 2022-10-09 21:53:45] ...trap/engines/engine.py.321: eval: epoch 55 | batch 250/510 [I 2022-10-09 21:53:45] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:23 [I 2022-10-09 21:53:45] ...trap/engines/engine.py.326: process: 0.31746 | load: 0.00021 [I 2022-10-09 21:53:48] ...trap/engines/engine.py.321: eval: epoch 55 | batch 260/510 [I 2022-10-09 21:53:48] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:20 [I 2022-10-09 21:53:48] ...trap/engines/engine.py.326: process: 0.32324 | load: 0.00021 [I 2022-10-09 21:53:51] ...trap/engines/engine.py.321: eval: epoch 55 | batch 270/510 [I 2022-10-09 21:53:51] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:17 [I 2022-10-09 21:53:51] ...trap/engines/engine.py.326: process: 0.32326 | load: 0.00021 [I 2022-10-09 21:53:54] ...trap/engines/engine.py.321: eval: epoch 55 | batch 280/510 [I 2022-10-09 21:53:54] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-09 21:53:54] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00021 [I 2022-10-09 21:53:58] ...trap/engines/engine.py.321: eval: epoch 55 | batch 290/510 [I 2022-10-09 21:53:58] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:11 [I 2022-10-09 21:53:58] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00020 [I 2022-10-09 21:54:01] ...trap/engines/engine.py.321: eval: epoch 55 | batch 300/510 [I 2022-10-09 21:54:01] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:07 [I 2022-10-09 21:54:01] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00020 [I 2022-10-09 21:54:04] ...trap/engines/engine.py.321: eval: epoch 55 | batch 310/510 [I 2022-10-09 21:54:04] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:04 [I 2022-10-09 21:54:04] ...trap/engines/engine.py.326: process: 0.32661 | load: 0.00021 [I 2022-10-09 21:54:07] ...trap/engines/engine.py.321: eval: epoch 55 | batch 320/510 [I 2022-10-09 21:54:07] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 21:54:07] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00021 [I 2022-10-09 21:54:10] ...trap/engines/engine.py.321: eval: epoch 55 | batch 330/510 [I 2022-10-09 21:54:10] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:58 [I 2022-10-09 21:54:10] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00020 [I 2022-10-09 21:54:14] ...trap/engines/engine.py.321: eval: epoch 55 | batch 340/510 [I 2022-10-09 21:54:14] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-09 21:54:14] ...trap/engines/engine.py.326: process: 0.31576 | load: 0.00020 [I 2022-10-09 21:54:17] ...trap/engines/engine.py.321: eval: epoch 55 | batch 350/510 [I 2022-10-09 21:54:17] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-09 21:54:17] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00021 [I 2022-10-09 21:54:20] ...trap/engines/engine.py.321: eval: epoch 55 | batch 360/510 [I 2022-10-09 21:54:20] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-09 21:54:20] ...trap/engines/engine.py.326: process: 0.31734 | load: 0.00020 [I 2022-10-09 21:54:23] ...trap/engines/engine.py.321: eval: epoch 55 | batch 370/510 [I 2022-10-09 21:54:23] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-09 21:54:23] ...trap/engines/engine.py.326: process: 0.32453 | load: 0.00020 [I 2022-10-09 21:54:27] ...trap/engines/engine.py.321: eval: epoch 55 | batch 380/510 [I 2022-10-09 21:54:27] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-09 21:54:27] ...trap/engines/engine.py.326: process: 0.32463 | load: 0.00020 [I 2022-10-09 21:54:30] ...trap/engines/engine.py.321: eval: epoch 55 | batch 390/510 [I 2022-10-09 21:54:30] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-09 21:54:30] ...trap/engines/engine.py.326: process: 0.31135 | load: 0.00020 [I 2022-10-09 21:54:33] ...trap/engines/engine.py.321: eval: epoch 55 | batch 400/510 [I 2022-10-09 21:54:33] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-09 21:54:33] ...trap/engines/engine.py.326: process: 0.32161 | load: 0.00020 [I 2022-10-09 21:54:36] ...trap/engines/engine.py.321: eval: epoch 55 | batch 410/510 [I 2022-10-09 21:54:36] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-09 21:54:36] ...trap/engines/engine.py.326: process: 0.31614 | load: 0.00020 [I 2022-10-09 21:54:39] ...trap/engines/engine.py.321: eval: epoch 55 | batch 420/510 [I 2022-10-09 21:54:39] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 21:54:39] ...trap/engines/engine.py.326: process: 0.32094 | load: 0.00019 [I 2022-10-09 21:54:42] ...trap/engines/engine.py.321: eval: epoch 55 | batch 430/510 [I 2022-10-09 21:54:42] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-09 21:54:42] ...trap/engines/engine.py.326: process: 0.31597 | load: 0.00019 [I 2022-10-09 21:54:46] ...trap/engines/engine.py.321: eval: epoch 55 | batch 440/510 [I 2022-10-09 21:54:46] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-09 21:54:46] ...trap/engines/engine.py.326: process: 0.31851 | load: 0.00021 [I 2022-10-09 21:54:49] ...trap/engines/engine.py.321: eval: epoch 55 | batch 450/510 [I 2022-10-09 21:54:49] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-09 21:54:49] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00019 [I 2022-10-09 21:54:52] ...trap/engines/engine.py.321: eval: epoch 55 | batch 460/510 [I 2022-10-09 21:54:52] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:16 [I 2022-10-09 21:54:52] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00022 [I 2022-10-09 21:54:55] ...trap/engines/engine.py.321: eval: epoch 55 | batch 470/510 [I 2022-10-09 21:54:55] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 21:54:55] ...trap/engines/engine.py.326: process: 0.31367 | load: 0.00020 [I 2022-10-09 21:54:58] ...trap/engines/engine.py.321: eval: epoch 55 | batch 480/510 [I 2022-10-09 21:54:58] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-09 21:54:58] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00018 [I 2022-10-09 21:55:02] ...trap/engines/engine.py.321: eval: epoch 55 | batch 490/510 [I 2022-10-09 21:55:02] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-09 21:55:02] ...trap/engines/engine.py.326: process: 0.31801 | load: 0.00022 [I 2022-10-09 21:55:05] ...trap/engines/engine.py.321: eval: epoch 55 | batch 500/510 [I 2022-10-09 21:55:05] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-09 21:55:05] ...trap/engines/engine.py.326: process: 0.32226 | load: 0.00020 [I 2022-10-09 21:55:08] ...trap/engines/engine.py.321: eval: epoch 55 | batch 510/510 [I 2022-10-09 21:55:08] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-09 21:55:08] ...trap/engines/engine.py.326: process: 0.31505 | load: 0.00014 [S 2022-10-09 21:55:09] ...trap/engines/engine.py.132: eval_epoch.epoch: 55 [I 2022-10-09 21:55:09] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.727800 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908000 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939000 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.722700 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906700 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937600 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010638 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009747 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007720 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013282 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008764 [S 2022-10-09 21:55:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006666 [I 2022-10-09 21:55:44] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 21:55:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 21:55:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 21:55:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 21:55:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 21:55:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 21:55:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 21:55:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 21:55:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 21:55:54] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 21:55:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 21:56:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 21:56:03] ...trap/engines/engine.py.163: Training model on trainset for epoch 56 [I 2022-10-09 21:56:03] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 21:56:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 0/2304 [I 2022-10-09 21:56:06] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:56 [I 2022-10-09 21:56:06] ...trap/engines/engine.py.229: process: 3.04547 | load: 2.14964 [I 2022-10-09 21:56:06] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-09 21:56:06] ...optimizers/trijoint.py.151: total_norm: 17.529951 [I 2022-10-09 21:56:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 10/2304 [I 2022-10-09 21:56:14] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:09 [I 2022-10-09 21:56:14] ...trap/engines/engine.py.229: process: 0.82090 | load: 0.00393 [I 2022-10-09 21:56:14] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-09 21:56:14] ...optimizers/trijoint.py.151: total_norm: 7.822940 [I 2022-10-09 21:56:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 20/2304 [I 2022-10-09 21:56:23] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:57 [I 2022-10-09 21:56:23] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00529 [I 2022-10-09 21:56:23] ...trap/engines/engine.py.230: loss: 0.17168 [I 2022-10-09 21:56:23] ...optimizers/trijoint.py.151: total_norm: 12.269277 [I 2022-10-09 21:56:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 30/2304 [I 2022-10-09 21:56:31] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:09 [I 2022-10-09 21:56:31] ...trap/engines/engine.py.229: process: 0.86571 | load: 0.00427 [I 2022-10-09 21:56:31] ...trap/engines/engine.py.230: loss: 0.17615 [I 2022-10-09 21:56:31] ...optimizers/trijoint.py.151: total_norm: 20.150967 [I 2022-10-09 21:56:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 40/2304 [I 2022-10-09 21:56:40] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:44 [I 2022-10-09 21:56:40] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00330 [I 2022-10-09 21:56:40] ...trap/engines/engine.py.230: loss: 0.11922 [I 2022-10-09 21:56:40] ...optimizers/trijoint.py.151: total_norm: 9.624176 [I 2022-10-09 21:56:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 50/2304 [I 2022-10-09 21:56:48] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:37 [I 2022-10-09 21:56:48] ...trap/engines/engine.py.229: process: 0.83935 | load: 0.00328 [I 2022-10-09 21:56:48] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-09 21:56:48] ...optimizers/trijoint.py.151: total_norm: 13.541755 [I 2022-10-09 21:56:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 60/2304 [I 2022-10-09 21:56:57] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:27 [I 2022-10-09 21:56:57] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00321 [I 2022-10-09 21:56:57] ...trap/engines/engine.py.230: loss: 0.17515 [I 2022-10-09 21:56:57] ...optimizers/trijoint.py.151: total_norm: 18.278484 [I 2022-10-09 21:57:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 70/2304 [I 2022-10-09 21:57:05] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:21 [I 2022-10-09 21:57:05] ...trap/engines/engine.py.229: process: 0.83607 | load: 0.00350 [I 2022-10-09 21:57:05] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-09 21:57:05] ...optimizers/trijoint.py.151: total_norm: 14.443928 [I 2022-10-09 21:57:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 80/2304 [I 2022-10-09 21:57:13] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:02 [I 2022-10-09 21:57:13] ...trap/engines/engine.py.229: process: 0.84181 | load: 0.00337 [I 2022-10-09 21:57:13] ...trap/engines/engine.py.230: loss: 0.16231 [I 2022-10-09 21:57:13] ...optimizers/trijoint.py.151: total_norm: 16.218037 [I 2022-10-09 21:57:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 90/2304 [I 2022-10-09 21:57:22] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:07 [I 2022-10-09 21:57:22] ...trap/engines/engine.py.229: process: 0.84949 | load: 0.00317 [I 2022-10-09 21:57:22] ...trap/engines/engine.py.230: loss: 0.16732 [I 2022-10-09 21:57:22] ...optimizers/trijoint.py.151: total_norm: 13.404866 [I 2022-10-09 21:57:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 100/2304 [I 2022-10-09 21:57:30] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:03 [I 2022-10-09 21:57:30] ...trap/engines/engine.py.229: process: 0.84580 | load: 0.00341 [I 2022-10-09 21:57:30] ...trap/engines/engine.py.230: loss: 0.18516 [I 2022-10-09 21:57:30] ...optimizers/trijoint.py.151: total_norm: 35.028633 [I 2022-10-09 21:57:39] ...trap/engines/engine.py.224: train: epoch 56 | batch 110/2304 [I 2022-10-09 21:57:39] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:54 [I 2022-10-09 21:57:39] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00295 [I 2022-10-09 21:57:39] ...trap/engines/engine.py.230: loss: 0.14321 [I 2022-10-09 21:57:39] ...optimizers/trijoint.py.151: total_norm: 27.180460 [I 2022-10-09 21:57:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 120/2304 [I 2022-10-09 21:57:47] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:56 [I 2022-10-09 21:57:47] ...trap/engines/engine.py.229: process: 0.86929 | load: 0.00297 [I 2022-10-09 21:57:47] ...trap/engines/engine.py.230: loss: 0.19939 [I 2022-10-09 21:57:47] ...optimizers/trijoint.py.151: total_norm: 25.331608 [I 2022-10-09 21:57:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 130/2304 [I 2022-10-09 21:57:56] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:45 [I 2022-10-09 21:57:56] ...trap/engines/engine.py.229: process: 0.84563 | load: 0.00306 [I 2022-10-09 21:57:56] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-09 21:57:56] ...optimizers/trijoint.py.151: total_norm: 11.440838 [I 2022-10-09 21:58:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 140/2304 [I 2022-10-09 21:58:04] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:36 [I 2022-10-09 21:58:04] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00318 [I 2022-10-09 21:58:04] ...trap/engines/engine.py.230: loss: 0.14854 [I 2022-10-09 21:58:04] ...optimizers/trijoint.py.151: total_norm: 11.738021 [I 2022-10-09 21:58:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 150/2304 [I 2022-10-09 21:58:13] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:35 [I 2022-10-09 21:58:13] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00305 [I 2022-10-09 21:58:13] ...trap/engines/engine.py.230: loss: 0.17279 [I 2022-10-09 21:58:13] ...optimizers/trijoint.py.151: total_norm: 10.888655 [I 2022-10-09 21:58:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 160/2304 [I 2022-10-09 21:58:21] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:16 [I 2022-10-09 21:58:21] ...trap/engines/engine.py.229: process: 0.86829 | load: 0.00317 [I 2022-10-09 21:58:21] ...trap/engines/engine.py.230: loss: 0.10521 [I 2022-10-09 21:58:21] ...optimizers/trijoint.py.151: total_norm: 10.656529 [I 2022-10-09 21:58:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 170/2304 [I 2022-10-09 21:58:30] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:12 [I 2022-10-09 21:58:30] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00326 [I 2022-10-09 21:58:30] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-09 21:58:30] ...optimizers/trijoint.py.151: total_norm: 27.207802 [I 2022-10-09 21:58:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 180/2304 [I 2022-10-09 21:58:38] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:30:05 [I 2022-10-09 21:58:38] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00316 [I 2022-10-09 21:58:38] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-09 21:58:38] ...optimizers/trijoint.py.151: total_norm: 10.925562 [I 2022-10-09 21:58:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 190/2304 [I 2022-10-09 21:58:47] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:30:07 [I 2022-10-09 21:58:47] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00336 [I 2022-10-09 21:58:47] ...trap/engines/engine.py.230: loss: 0.17831 [I 2022-10-09 21:58:47] ...optimizers/trijoint.py.151: total_norm: 13.858266 [I 2022-10-09 21:58:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 200/2304 [I 2022-10-09 21:58:55] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:30:02 [I 2022-10-09 21:58:55] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00333 [I 2022-10-09 21:58:55] ...trap/engines/engine.py.230: loss: 0.17375 [I 2022-10-09 21:58:55] ...optimizers/trijoint.py.151: total_norm: 15.611877 [I 2022-10-09 21:59:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 210/2304 [I 2022-10-09 21:59:04] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:49 [I 2022-10-09 21:59:04] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00314 [I 2022-10-09 21:59:04] ...trap/engines/engine.py.230: loss: 0.09207 [I 2022-10-09 21:59:04] ...optimizers/trijoint.py.151: total_norm: 9.495531 [I 2022-10-09 21:59:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 220/2304 [I 2022-10-09 21:59:13] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:36 [I 2022-10-09 21:59:13] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00311 [I 2022-10-09 21:59:13] ...trap/engines/engine.py.230: loss: 0.17463 [I 2022-10-09 21:59:13] ...optimizers/trijoint.py.151: total_norm: 18.511885 [I 2022-10-09 21:59:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 230/2304 [I 2022-10-09 21:59:21] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:28 [I 2022-10-09 21:59:21] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00322 [I 2022-10-09 21:59:21] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-09 21:59:21] ...optimizers/trijoint.py.151: total_norm: 8.602576 [I 2022-10-09 21:59:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 240/2304 [I 2022-10-09 21:59:30] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:24 [I 2022-10-09 21:59:30] ...trap/engines/engine.py.229: process: 0.85706 | load: 0.00310 [I 2022-10-09 21:59:30] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-09 21:59:30] ...optimizers/trijoint.py.151: total_norm: 14.747905 [I 2022-10-09 21:59:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 250/2304 [I 2022-10-09 21:59:38] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:18 [I 2022-10-09 21:59:38] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00329 [I 2022-10-09 21:59:38] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-09 21:59:38] ...optimizers/trijoint.py.151: total_norm: 11.364614 [I 2022-10-09 21:59:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 260/2304 [I 2022-10-09 21:59:47] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:29:14 [I 2022-10-09 21:59:47] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00317 [I 2022-10-09 21:59:47] ...trap/engines/engine.py.230: loss: 0.10652 [I 2022-10-09 21:59:47] ...optimizers/trijoint.py.151: total_norm: 10.250059 [I 2022-10-09 21:59:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 270/2304 [I 2022-10-09 21:59:55] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:29:03 [I 2022-10-09 21:59:55] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00312 [I 2022-10-09 21:59:55] ...trap/engines/engine.py.230: loss: 0.13218 [I 2022-10-09 21:59:55] ...optimizers/trijoint.py.151: total_norm: 7.980000 [I 2022-10-09 22:00:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 280/2304 [I 2022-10-09 22:00:04] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:50 [I 2022-10-09 22:00:04] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00315 [I 2022-10-09 22:00:04] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-09 22:00:04] ...optimizers/trijoint.py.151: total_norm: 27.185883 [I 2022-10-09 22:00:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 290/2304 [I 2022-10-09 22:00:13] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:46 [I 2022-10-09 22:00:13] ...trap/engines/engine.py.229: process: 0.86183 | load: 0.00333 [I 2022-10-09 22:00:13] ...trap/engines/engine.py.230: loss: 0.12504 [I 2022-10-09 22:00:13] ...optimizers/trijoint.py.151: total_norm: 34.806309 [I 2022-10-09 22:00:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 300/2304 [I 2022-10-09 22:00:21] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:33 [I 2022-10-09 22:00:21] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00326 [I 2022-10-09 22:00:21] ...trap/engines/engine.py.230: loss: 0.08972 [I 2022-10-09 22:00:21] ...optimizers/trijoint.py.151: total_norm: 10.212181 [I 2022-10-09 22:00:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 310/2304 [I 2022-10-09 22:00:30] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:22 [I 2022-10-09 22:00:30] ...trap/engines/engine.py.229: process: 0.85210 | load: 0.00326 [I 2022-10-09 22:00:30] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-09 22:00:30] ...optimizers/trijoint.py.151: total_norm: 13.807867 [I 2022-10-09 22:00:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 320/2304 [I 2022-10-09 22:00:38] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:31 [I 2022-10-09 22:00:38] ...trap/engines/engine.py.229: process: 0.86810 | load: 0.00312 [I 2022-10-09 22:00:38] ...trap/engines/engine.py.230: loss: 0.11487 [I 2022-10-09 22:00:38] ...optimizers/trijoint.py.151: total_norm: 11.260118 [I 2022-10-09 22:00:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 330/2304 [I 2022-10-09 22:00:47] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:04 [I 2022-10-09 22:00:47] ...trap/engines/engine.py.229: process: 0.85796 | load: 0.00313 [I 2022-10-09 22:00:47] ...trap/engines/engine.py.230: loss: 0.19719 [I 2022-10-09 22:00:47] ...optimizers/trijoint.py.151: total_norm: 16.592733 [I 2022-10-09 22:00:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 340/2304 [I 2022-10-09 22:00:55] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:58 [I 2022-10-09 22:00:55] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00310 [I 2022-10-09 22:00:55] ...trap/engines/engine.py.230: loss: 0.13490 [I 2022-10-09 22:00:55] ...optimizers/trijoint.py.151: total_norm: 17.625824 [I 2022-10-09 22:01:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 350/2304 [I 2022-10-09 22:01:04] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:38 [I 2022-10-09 22:01:04] ...trap/engines/engine.py.229: process: 0.83179 | load: 0.00325 [I 2022-10-09 22:01:04] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-09 22:01:04] ...optimizers/trijoint.py.151: total_norm: 7.824864 [I 2022-10-09 22:01:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 360/2304 [I 2022-10-09 22:01:12] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:34 [I 2022-10-09 22:01:12] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00312 [I 2022-10-09 22:01:12] ...trap/engines/engine.py.230: loss: 0.14701 [I 2022-10-09 22:01:12] ...optimizers/trijoint.py.151: total_norm: 9.076676 [I 2022-10-09 22:01:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 370/2304 [I 2022-10-09 22:01:21] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:40 [I 2022-10-09 22:01:21] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00313 [I 2022-10-09 22:01:21] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-09 22:01:21] ...optimizers/trijoint.py.151: total_norm: 12.844025 [I 2022-10-09 22:01:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 380/2304 [I 2022-10-09 22:01:29] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:19 [I 2022-10-09 22:01:29] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00312 [I 2022-10-09 22:01:29] ...trap/engines/engine.py.230: loss: 0.11736 [I 2022-10-09 22:01:29] ...optimizers/trijoint.py.151: total_norm: 10.660778 [I 2022-10-09 22:01:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 390/2304 [I 2022-10-09 22:01:38] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:12 [I 2022-10-09 22:01:38] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00313 [I 2022-10-09 22:01:38] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-09 22:01:38] ...optimizers/trijoint.py.151: total_norm: 20.338848 [I 2022-10-09 22:01:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 400/2304 [I 2022-10-09 22:01:47] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:27:13 [I 2022-10-09 22:01:47] ...trap/engines/engine.py.229: process: 0.86388 | load: 0.00314 [I 2022-10-09 22:01:47] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-09 22:01:47] ...optimizers/trijoint.py.151: total_norm: 21.150608 [I 2022-10-09 22:01:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 410/2304 [I 2022-10-09 22:01:55] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:59 [I 2022-10-09 22:01:55] ...trap/engines/engine.py.229: process: 0.86855 | load: 0.00313 [I 2022-10-09 22:01:55] ...trap/engines/engine.py.230: loss: 0.15369 [I 2022-10-09 22:01:55] ...optimizers/trijoint.py.151: total_norm: 11.514684 [I 2022-10-09 22:02:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 420/2304 [I 2022-10-09 22:02:04] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:46 [I 2022-10-09 22:02:04] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00310 [I 2022-10-09 22:02:04] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-09 22:02:04] ...optimizers/trijoint.py.151: total_norm: 19.960365 [I 2022-10-09 22:02:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 430/2304 [I 2022-10-09 22:02:12] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:35 [I 2022-10-09 22:02:12] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00330 [I 2022-10-09 22:02:12] ...trap/engines/engine.py.230: loss: 0.19221 [I 2022-10-09 22:02:12] ...optimizers/trijoint.py.151: total_norm: 28.651403 [I 2022-10-09 22:02:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 440/2304 [I 2022-10-09 22:02:21] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:36 [I 2022-10-09 22:02:21] ...trap/engines/engine.py.229: process: 0.86375 | load: 0.00309 [I 2022-10-09 22:02:21] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-09 22:02:21] ...optimizers/trijoint.py.151: total_norm: 13.283850 [I 2022-10-09 22:02:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 450/2304 [I 2022-10-09 22:02:29] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:27 [I 2022-10-09 22:02:29] ...trap/engines/engine.py.229: process: 0.86245 | load: 0.00314 [I 2022-10-09 22:02:29] ...trap/engines/engine.py.230: loss: 0.11671 [I 2022-10-09 22:02:29] ...optimizers/trijoint.py.151: total_norm: 21.226269 [I 2022-10-09 22:02:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 460/2304 [I 2022-10-09 22:02:38] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:05 [I 2022-10-09 22:02:38] ...trap/engines/engine.py.229: process: 0.83891 | load: 0.00301 [I 2022-10-09 22:02:38] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-09 22:02:38] ...optimizers/trijoint.py.151: total_norm: 30.900005 [I 2022-10-09 22:02:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 470/2304 [I 2022-10-09 22:02:46] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:53 [I 2022-10-09 22:02:46] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00295 [I 2022-10-09 22:02:46] ...trap/engines/engine.py.230: loss: 0.12798 [I 2022-10-09 22:02:46] ...optimizers/trijoint.py.151: total_norm: 10.701336 [I 2022-10-09 22:02:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 480/2304 [I 2022-10-09 22:02:55] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:49 [I 2022-10-09 22:02:55] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00296 [I 2022-10-09 22:02:55] ...trap/engines/engine.py.230: loss: 0.16958 [I 2022-10-09 22:02:55] ...optimizers/trijoint.py.151: total_norm: 19.729795 [I 2022-10-09 22:03:03] ...trap/engines/engine.py.224: train: epoch 56 | batch 490/2304 [I 2022-10-09 22:03:03] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:33 [I 2022-10-09 22:03:03] ...trap/engines/engine.py.229: process: 0.82640 | load: 0.00300 [I 2022-10-09 22:03:03] ...trap/engines/engine.py.230: loss: 0.12735 [I 2022-10-09 22:03:03] ...optimizers/trijoint.py.151: total_norm: 17.107376 [I 2022-10-09 22:03:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 500/2304 [I 2022-10-09 22:03:12] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:34 [I 2022-10-09 22:03:12] ...trap/engines/engine.py.229: process: 0.86123 | load: 0.00306 [I 2022-10-09 22:03:12] ...trap/engines/engine.py.230: loss: 0.19360 [I 2022-10-09 22:03:12] ...optimizers/trijoint.py.151: total_norm: 12.508809 [I 2022-10-09 22:03:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 510/2304 [I 2022-10-09 22:03:20] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:19 [I 2022-10-09 22:03:20] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00293 [I 2022-10-09 22:03:20] ...trap/engines/engine.py.230: loss: 0.15114 [I 2022-10-09 22:03:20] ...optimizers/trijoint.py.151: total_norm: 17.070930 [I 2022-10-09 22:03:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 520/2304 [I 2022-10-09 22:03:29] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:06 [I 2022-10-09 22:03:29] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00299 [I 2022-10-09 22:03:29] ...trap/engines/engine.py.230: loss: 0.10534 [I 2022-10-09 22:03:29] ...optimizers/trijoint.py.151: total_norm: 11.946810 [I 2022-10-09 22:03:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 530/2304 [I 2022-10-09 22:03:37] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:56 [I 2022-10-09 22:03:37] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00293 [I 2022-10-09 22:03:37] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-09 22:03:37] ...optimizers/trijoint.py.151: total_norm: 17.062168 [I 2022-10-09 22:03:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 540/2304 [I 2022-10-09 22:03:46] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:52 [I 2022-10-09 22:03:46] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00295 [I 2022-10-09 22:03:46] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-09 22:03:46] ...optimizers/trijoint.py.151: total_norm: 10.599511 [I 2022-10-09 22:03:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 550/2304 [I 2022-10-09 22:03:54] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:50 [I 2022-10-09 22:03:54] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00293 [I 2022-10-09 22:03:54] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-09 22:03:54] ...optimizers/trijoint.py.151: total_norm: 10.782696 [I 2022-10-09 22:04:03] ...trap/engines/engine.py.224: train: epoch 56 | batch 560/2304 [I 2022-10-09 22:04:03] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:31 [I 2022-10-09 22:04:03] ...trap/engines/engine.py.229: process: 0.83752 | load: 0.00316 [I 2022-10-09 22:04:03] ...trap/engines/engine.py.230: loss: 0.21269 [I 2022-10-09 22:04:03] ...optimizers/trijoint.py.151: total_norm: 25.608362 [I 2022-10-09 22:04:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 570/2304 [I 2022-10-09 22:04:11] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:27 [I 2022-10-09 22:04:11] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00295 [I 2022-10-09 22:04:11] ...trap/engines/engine.py.230: loss: 0.17583 [I 2022-10-09 22:04:11] ...optimizers/trijoint.py.151: total_norm: 16.982006 [I 2022-10-09 22:04:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 580/2304 [I 2022-10-09 22:04:20] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:19 [I 2022-10-09 22:04:20] ...trap/engines/engine.py.229: process: 0.84145 | load: 0.00293 [I 2022-10-09 22:04:20] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-09 22:04:20] ...optimizers/trijoint.py.151: total_norm: 11.323915 [I 2022-10-09 22:04:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 590/2304 [I 2022-10-09 22:04:28] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:04 [I 2022-10-09 22:04:28] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00292 [I 2022-10-09 22:04:28] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-09 22:04:28] ...optimizers/trijoint.py.151: total_norm: 22.938831 [I 2022-10-09 22:04:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 600/2304 [I 2022-10-09 22:04:37] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:08 [I 2022-10-09 22:04:37] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00297 [I 2022-10-09 22:04:37] ...trap/engines/engine.py.230: loss: 0.13705 [I 2022-10-09 22:04:37] ...optimizers/trijoint.py.151: total_norm: 15.004405 [I 2022-10-09 22:04:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 610/2304 [I 2022-10-09 22:04:45] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:23:53 [I 2022-10-09 22:04:45] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00294 [I 2022-10-09 22:04:45] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-09 22:04:45] ...optimizers/trijoint.py.151: total_norm: 15.667025 [I 2022-10-09 22:04:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 620/2304 [I 2022-10-09 22:04:54] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:53 [I 2022-10-09 22:04:54] ...trap/engines/engine.py.229: process: 0.86739 | load: 0.00298 [I 2022-10-09 22:04:54] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-09 22:04:54] ...optimizers/trijoint.py.151: total_norm: 12.028352 [I 2022-10-09 22:05:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 630/2304 [I 2022-10-09 22:05:02] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:40 [I 2022-10-09 22:05:02] ...trap/engines/engine.py.229: process: 0.84649 | load: 0.00292 [I 2022-10-09 22:05:02] ...trap/engines/engine.py.230: loss: 0.16786 [I 2022-10-09 22:05:02] ...optimizers/trijoint.py.151: total_norm: 18.238255 [I 2022-10-09 22:05:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 640/2304 [I 2022-10-09 22:05:11] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:29 [I 2022-10-09 22:05:11] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00294 [I 2022-10-09 22:05:11] ...trap/engines/engine.py.230: loss: 0.14973 [I 2022-10-09 22:05:11] ...optimizers/trijoint.py.151: total_norm: 9.849767 [I 2022-10-09 22:05:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 650/2304 [I 2022-10-09 22:05:19] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:20 [I 2022-10-09 22:05:19] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00295 [I 2022-10-09 22:05:19] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-09 22:05:19] ...optimizers/trijoint.py.151: total_norm: 10.700181 [I 2022-10-09 22:05:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 660/2304 [I 2022-10-09 22:05:28] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:03 [I 2022-10-09 22:05:28] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00294 [I 2022-10-09 22:05:28] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-09 22:05:28] ...optimizers/trijoint.py.151: total_norm: 20.163532 [I 2022-10-09 22:05:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 670/2304 [I 2022-10-09 22:05:36] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:05 [I 2022-10-09 22:05:36] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00299 [I 2022-10-09 22:05:36] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-09 22:05:36] ...optimizers/trijoint.py.151: total_norm: 14.051577 [I 2022-10-09 22:05:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 680/2304 [I 2022-10-09 22:05:44] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:22:47 [I 2022-10-09 22:05:44] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00301 [I 2022-10-09 22:05:44] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-09 22:05:44] ...optimizers/trijoint.py.151: total_norm: 12.399679 [I 2022-10-09 22:05:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 690/2304 [I 2022-10-09 22:05:53] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:44 [I 2022-10-09 22:05:53] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00300 [I 2022-10-09 22:05:53] ...trap/engines/engine.py.230: loss: 0.22628 [I 2022-10-09 22:05:53] ...optimizers/trijoint.py.151: total_norm: 19.660187 [I 2022-10-09 22:06:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 700/2304 [I 2022-10-09 22:06:01] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:41 [I 2022-10-09 22:06:01] ...trap/engines/engine.py.229: process: 0.85925 | load: 0.00296 [I 2022-10-09 22:06:01] ...trap/engines/engine.py.230: loss: 0.20150 [I 2022-10-09 22:06:01] ...optimizers/trijoint.py.151: total_norm: 35.017250 [I 2022-10-09 22:06:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 710/2304 [I 2022-10-09 22:06:10] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:35 [I 2022-10-09 22:06:10] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00292 [I 2022-10-09 22:06:10] ...trap/engines/engine.py.230: loss: 0.14889 [I 2022-10-09 22:06:10] ...optimizers/trijoint.py.151: total_norm: 19.727291 [I 2022-10-09 22:06:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 720/2304 [I 2022-10-09 22:06:18] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:13 [I 2022-10-09 22:06:18] ...trap/engines/engine.py.229: process: 0.83851 | load: 0.00297 [I 2022-10-09 22:06:18] ...trap/engines/engine.py.230: loss: 0.09197 [I 2022-10-09 22:06:18] ...optimizers/trijoint.py.151: total_norm: 7.603527 [I 2022-10-09 22:06:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 730/2304 [I 2022-10-09 22:06:27] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:04 [I 2022-10-09 22:06:27] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00294 [I 2022-10-09 22:06:27] ...trap/engines/engine.py.230: loss: 0.16153 [I 2022-10-09 22:06:27] ...optimizers/trijoint.py.151: total_norm: 14.613087 [I 2022-10-09 22:06:35] ...trap/engines/engine.py.224: train: epoch 56 | batch 740/2304 [I 2022-10-09 22:06:35] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:02 [I 2022-10-09 22:06:35] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00303 [I 2022-10-09 22:06:35] ...trap/engines/engine.py.230: loss: 0.15143 [I 2022-10-09 22:06:35] ...optimizers/trijoint.py.151: total_norm: 10.378742 [I 2022-10-09 22:06:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 750/2304 [I 2022-10-09 22:06:44] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:21:56 [I 2022-10-09 22:06:44] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00296 [I 2022-10-09 22:06:44] ...trap/engines/engine.py.230: loss: 0.12492 [I 2022-10-09 22:06:44] ...optimizers/trijoint.py.151: total_norm: 13.042540 [I 2022-10-09 22:06:52] ...trap/engines/engine.py.224: train: epoch 56 | batch 760/2304 [I 2022-10-09 22:06:52] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:50 [I 2022-10-09 22:06:52] ...trap/engines/engine.py.229: process: 0.84256 | load: 0.00296 [I 2022-10-09 22:06:52] ...trap/engines/engine.py.230: loss: 0.21588 [I 2022-10-09 22:06:52] ...optimizers/trijoint.py.151: total_norm: 14.249697 [I 2022-10-09 22:07:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 770/2304 [I 2022-10-09 22:07:01] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:34 [I 2022-10-09 22:07:01] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00300 [I 2022-10-09 22:07:01] ...trap/engines/engine.py.230: loss: 0.11458 [I 2022-10-09 22:07:01] ...optimizers/trijoint.py.151: total_norm: 16.812178 [I 2022-10-09 22:07:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 780/2304 [I 2022-10-09 22:07:09] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:26 [I 2022-10-09 22:07:09] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00301 [I 2022-10-09 22:07:09] ...trap/engines/engine.py.230: loss: 0.19203 [I 2022-10-09 22:07:09] ...optimizers/trijoint.py.151: total_norm: 15.608187 [I 2022-10-09 22:07:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 790/2304 [I 2022-10-09 22:07:18] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:16 [I 2022-10-09 22:07:18] ...trap/engines/engine.py.229: process: 0.83649 | load: 0.00300 [I 2022-10-09 22:07:18] ...trap/engines/engine.py.230: loss: 0.11172 [I 2022-10-09 22:07:18] ...optimizers/trijoint.py.151: total_norm: 12.850037 [I 2022-10-09 22:07:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 800/2304 [I 2022-10-09 22:07:26] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:12 [I 2022-10-09 22:07:26] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00294 [I 2022-10-09 22:07:26] ...trap/engines/engine.py.230: loss: 0.15139 [I 2022-10-09 22:07:26] ...optimizers/trijoint.py.151: total_norm: 11.570470 [I 2022-10-09 22:07:35] ...trap/engines/engine.py.224: train: epoch 56 | batch 810/2304 [I 2022-10-09 22:07:35] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:03 [I 2022-10-09 22:07:35] ...trap/engines/engine.py.229: process: 0.84955 | load: 0.00299 [I 2022-10-09 22:07:35] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-09 22:07:35] ...optimizers/trijoint.py.151: total_norm: 12.698843 [I 2022-10-09 22:07:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 820/2304 [I 2022-10-09 22:07:43] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:20:46 [I 2022-10-09 22:07:43] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00293 [I 2022-10-09 22:07:43] ...trap/engines/engine.py.230: loss: 0.16467 [I 2022-10-09 22:07:43] ...optimizers/trijoint.py.151: total_norm: 63.220146 [I 2022-10-09 22:07:52] ...trap/engines/engine.py.224: train: epoch 56 | batch 830/2304 [I 2022-10-09 22:07:52] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:20:39 [I 2022-10-09 22:07:52] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00298 [I 2022-10-09 22:07:52] ...trap/engines/engine.py.230: loss: 0.15020 [I 2022-10-09 22:07:52] ...optimizers/trijoint.py.151: total_norm: 18.099688 [I 2022-10-09 22:08:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 840/2304 [I 2022-10-09 22:08:00] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:20:38 [I 2022-10-09 22:08:00] ...trap/engines/engine.py.229: process: 0.85825 | load: 0.00301 [I 2022-10-09 22:08:00] ...trap/engines/engine.py.230: loss: 0.12887 [I 2022-10-09 22:08:00] ...optimizers/trijoint.py.151: total_norm: 20.257526 [I 2022-10-09 22:08:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 850/2304 [I 2022-10-09 22:08:09] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:20:37 [I 2022-10-09 22:08:09] ...trap/engines/engine.py.229: process: 0.83566 | load: 0.00312 [I 2022-10-09 22:08:09] ...trap/engines/engine.py.230: loss: 0.12830 [I 2022-10-09 22:08:09] ...optimizers/trijoint.py.151: total_norm: 13.243234 [I 2022-10-09 22:08:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 860/2304 [I 2022-10-09 22:08:17] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:20:30 [I 2022-10-09 22:08:17] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00309 [I 2022-10-09 22:08:17] ...trap/engines/engine.py.230: loss: 0.11473 [I 2022-10-09 22:08:17] ...optimizers/trijoint.py.151: total_norm: 14.155482 [I 2022-10-09 22:08:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 870/2304 [I 2022-10-09 22:08:26] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:20:23 [I 2022-10-09 22:08:26] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00311 [I 2022-10-09 22:08:26] ...trap/engines/engine.py.230: loss: 0.14565 [I 2022-10-09 22:08:26] ...optimizers/trijoint.py.151: total_norm: 18.469191 [I 2022-10-09 22:08:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 880/2304 [I 2022-10-09 22:08:34] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:20:15 [I 2022-10-09 22:08:34] ...trap/engines/engine.py.229: process: 0.86634 | load: 0.00312 [I 2022-10-09 22:08:34] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-09 22:08:34] ...optimizers/trijoint.py.151: total_norm: 12.562408 [I 2022-10-09 22:08:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 890/2304 [I 2022-10-09 22:08:43] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:20:07 [I 2022-10-09 22:08:43] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00307 [I 2022-10-09 22:08:43] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-09 22:08:43] ...optimizers/trijoint.py.151: total_norm: 18.274036 [I 2022-10-09 22:08:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 900/2304 [I 2022-10-09 22:08:51] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:19:57 [I 2022-10-09 22:08:51] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00316 [I 2022-10-09 22:08:51] ...trap/engines/engine.py.230: loss: 0.14806 [I 2022-10-09 22:08:51] ...optimizers/trijoint.py.151: total_norm: 13.236518 [I 2022-10-09 22:09:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 910/2304 [I 2022-10-09 22:09:00] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:19:43 [I 2022-10-09 22:09:00] ...trap/engines/engine.py.229: process: 0.85577 | load: 0.00314 [I 2022-10-09 22:09:00] ...trap/engines/engine.py.230: loss: 0.13043 [I 2022-10-09 22:09:00] ...optimizers/trijoint.py.151: total_norm: 16.697460 [I 2022-10-09 22:09:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 920/2304 [I 2022-10-09 22:09:08] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:19:36 [I 2022-10-09 22:09:08] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00318 [I 2022-10-09 22:09:08] ...trap/engines/engine.py.230: loss: 0.24285 [I 2022-10-09 22:09:08] ...optimizers/trijoint.py.151: total_norm: 54.997692 [I 2022-10-09 22:09:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 930/2304 [I 2022-10-09 22:09:17] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:37 [I 2022-10-09 22:09:17] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00307 [I 2022-10-09 22:09:17] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-09 22:09:17] ...optimizers/trijoint.py.151: total_norm: 15.195158 [I 2022-10-09 22:09:25] ...trap/engines/engine.py.224: train: epoch 56 | batch 940/2304 [I 2022-10-09 22:09:25] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:19:20 [I 2022-10-09 22:09:25] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00313 [I 2022-10-09 22:09:25] ...trap/engines/engine.py.230: loss: 0.15648 [I 2022-10-09 22:09:25] ...optimizers/trijoint.py.151: total_norm: 17.371153 [I 2022-10-09 22:09:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 950/2304 [I 2022-10-09 22:09:34] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:19:13 [I 2022-10-09 22:09:34] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00310 [I 2022-10-09 22:09:34] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-09 22:09:34] ...optimizers/trijoint.py.151: total_norm: 16.867266 [I 2022-10-09 22:09:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 960/2304 [I 2022-10-09 22:09:43] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:19:04 [I 2022-10-09 22:09:43] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00312 [I 2022-10-09 22:09:43] ...trap/engines/engine.py.230: loss: 0.13651 [I 2022-10-09 22:09:43] ...optimizers/trijoint.py.151: total_norm: 11.998578 [I 2022-10-09 22:09:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 970/2304 [I 2022-10-09 22:09:51] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:19:01 [I 2022-10-09 22:09:51] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00323 [I 2022-10-09 22:09:51] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-09 22:09:51] ...optimizers/trijoint.py.151: total_norm: 16.624750 [I 2022-10-09 22:10:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 980/2304 [I 2022-10-09 22:10:00] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:56 [I 2022-10-09 22:10:00] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00320 [I 2022-10-09 22:10:00] ...trap/engines/engine.py.230: loss: 0.12204 [I 2022-10-09 22:10:00] ...optimizers/trijoint.py.151: total_norm: 13.991373 [I 2022-10-09 22:10:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 990/2304 [I 2022-10-09 22:10:08] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:43 [I 2022-10-09 22:10:08] ...trap/engines/engine.py.229: process: 0.85918 | load: 0.00311 [I 2022-10-09 22:10:08] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-09 22:10:08] ...optimizers/trijoint.py.151: total_norm: 29.347029 [I 2022-10-09 22:10:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 1000/2304 [I 2022-10-09 22:10:17] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:35 [I 2022-10-09 22:10:17] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00312 [I 2022-10-09 22:10:17] ...trap/engines/engine.py.230: loss: 0.14459 [I 2022-10-09 22:10:17] ...optimizers/trijoint.py.151: total_norm: 15.098485 [I 2022-10-09 22:10:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 1010/2304 [I 2022-10-09 22:10:44] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:35:45 [I 2022-10-09 22:10:44] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00317 [I 2022-10-09 22:10:44] ...trap/engines/engine.py.230: loss: 0.20321 [I 2022-10-09 22:10:44] ...optimizers/trijoint.py.151: total_norm: 13.400258 [I 2022-10-09 22:10:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 1020/2304 [I 2022-10-09 22:10:53] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:20:04 [I 2022-10-09 22:10:53] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00314 [I 2022-10-09 22:10:53] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-09 22:10:53] ...optimizers/trijoint.py.151: total_norm: 16.006859 [I 2022-10-09 22:11:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 1030/2304 [I 2022-10-09 22:11:01] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:10 [I 2022-10-09 22:11:01] ...trap/engines/engine.py.229: process: 0.84050 | load: 0.00313 [I 2022-10-09 22:11:01] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-09 22:11:01] ...optimizers/trijoint.py.151: total_norm: 49.987244 [I 2022-10-09 22:11:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 1040/2304 [I 2022-10-09 22:11:10] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:55 [I 2022-10-09 22:11:10] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00314 [I 2022-10-09 22:11:10] ...trap/engines/engine.py.230: loss: 0.11521 [I 2022-10-09 22:11:10] ...optimizers/trijoint.py.151: total_norm: 8.759523 [I 2022-10-09 22:11:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 1050/2304 [I 2022-10-09 22:11:19] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:50 [I 2022-10-09 22:11:19] ...trap/engines/engine.py.229: process: 0.85598 | load: 0.00314 [I 2022-10-09 22:11:19] ...trap/engines/engine.py.230: loss: 0.13268 [I 2022-10-09 22:11:19] ...optimizers/trijoint.py.151: total_norm: 8.405622 [I 2022-10-09 22:11:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 1060/2304 [I 2022-10-09 22:11:27] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:32 [I 2022-10-09 22:11:27] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00310 [I 2022-10-09 22:11:27] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-09 22:11:27] ...optimizers/trijoint.py.151: total_norm: 10.416394 [I 2022-10-09 22:11:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 1070/2304 [I 2022-10-09 22:11:36] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:34 [I 2022-10-09 22:11:36] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00312 [I 2022-10-09 22:11:36] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-09 22:11:36] ...optimizers/trijoint.py.151: total_norm: 14.559830 [I 2022-10-09 22:11:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 1080/2304 [I 2022-10-09 22:11:44] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:31 [I 2022-10-09 22:11:44] ...trap/engines/engine.py.229: process: 0.86869 | load: 0.00313 [I 2022-10-09 22:11:44] ...trap/engines/engine.py.230: loss: 0.18667 [I 2022-10-09 22:11:44] ...optimizers/trijoint.py.151: total_norm: 32.527416 [I 2022-10-09 22:11:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 1090/2304 [I 2022-10-09 22:11:53] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:12 [I 2022-10-09 22:11:53] ...trap/engines/engine.py.229: process: 0.85195 | load: 0.00310 [I 2022-10-09 22:11:53] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-09 22:11:53] ...optimizers/trijoint.py.151: total_norm: 13.230712 [I 2022-10-09 22:12:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 1100/2304 [I 2022-10-09 22:12:01] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:57 [I 2022-10-09 22:12:01] ...trap/engines/engine.py.229: process: 0.83412 | load: 0.00295 [I 2022-10-09 22:12:01] ...trap/engines/engine.py.230: loss: 0.09588 [I 2022-10-09 22:12:01] ...optimizers/trijoint.py.151: total_norm: 11.917560 [I 2022-10-09 22:12:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 1110/2304 [I 2022-10-09 22:12:10] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:41 [I 2022-10-09 22:12:10] ...trap/engines/engine.py.229: process: 0.83724 | load: 0.00298 [I 2022-10-09 22:12:10] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-09 22:12:10] ...optimizers/trijoint.py.151: total_norm: 10.729628 [I 2022-10-09 22:12:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 1120/2304 [I 2022-10-09 22:12:18] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:36 [I 2022-10-09 22:12:18] ...trap/engines/engine.py.229: process: 0.83691 | load: 0.00295 [I 2022-10-09 22:12:18] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-09 22:12:18] ...optimizers/trijoint.py.151: total_norm: 28.866041 [I 2022-10-09 22:12:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 1130/2304 [I 2022-10-09 22:12:27] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:32 [I 2022-10-09 22:12:27] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00298 [I 2022-10-09 22:12:27] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-09 22:12:27] ...optimizers/trijoint.py.151: total_norm: 12.079946 [I 2022-10-09 22:12:35] ...trap/engines/engine.py.224: train: epoch 56 | batch 1140/2304 [I 2022-10-09 22:12:35] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:20 [I 2022-10-09 22:12:35] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00295 [I 2022-10-09 22:12:35] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-09 22:12:35] ...optimizers/trijoint.py.151: total_norm: 19.498171 [I 2022-10-09 22:12:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 1150/2304 [I 2022-10-09 22:12:43] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:09 [I 2022-10-09 22:12:43] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00301 [I 2022-10-09 22:12:43] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-09 22:12:43] ...optimizers/trijoint.py.151: total_norm: 12.059242 [I 2022-10-09 22:12:52] ...trap/engines/engine.py.224: train: epoch 56 | batch 1160/2304 [I 2022-10-09 22:12:52] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:16:09 [I 2022-10-09 22:12:52] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00296 [I 2022-10-09 22:12:52] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-09 22:12:52] ...optimizers/trijoint.py.151: total_norm: 11.282381 [I 2022-10-09 22:13:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 1170/2304 [I 2022-10-09 22:13:00] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:51 [I 2022-10-09 22:13:00] ...trap/engines/engine.py.229: process: 0.83393 | load: 0.00300 [I 2022-10-09 22:13:00] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-09 22:13:00] ...optimizers/trijoint.py.151: total_norm: 21.219116 [I 2022-10-09 22:13:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 1180/2304 [I 2022-10-09 22:13:09] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:46 [I 2022-10-09 22:13:09] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00300 [I 2022-10-09 22:13:09] ...trap/engines/engine.py.230: loss: 0.12651 [I 2022-10-09 22:13:09] ...optimizers/trijoint.py.151: total_norm: 10.800570 [I 2022-10-09 22:13:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 1190/2304 [I 2022-10-09 22:13:17] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:38 [I 2022-10-09 22:13:17] ...trap/engines/engine.py.229: process: 0.85378 | load: 0.00294 [I 2022-10-09 22:13:17] ...trap/engines/engine.py.230: loss: 0.17522 [I 2022-10-09 22:13:17] ...optimizers/trijoint.py.151: total_norm: 14.703799 [I 2022-10-09 22:13:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 1200/2304 [I 2022-10-09 22:13:26] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:27 [I 2022-10-09 22:13:26] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00299 [I 2022-10-09 22:13:26] ...trap/engines/engine.py.230: loss: 0.15105 [I 2022-10-09 22:13:26] ...optimizers/trijoint.py.151: total_norm: 8.118993 [I 2022-10-09 22:13:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 1210/2304 [I 2022-10-09 22:13:34] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:20 [I 2022-10-09 22:13:34] ...trap/engines/engine.py.229: process: 0.83670 | load: 0.00293 [I 2022-10-09 22:13:34] ...trap/engines/engine.py.230: loss: 0.09889 [I 2022-10-09 22:13:34] ...optimizers/trijoint.py.151: total_norm: 20.177290 [I 2022-10-09 22:13:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 1220/2304 [I 2022-10-09 22:13:43] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:14 [I 2022-10-09 22:13:43] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00295 [I 2022-10-09 22:13:43] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-09 22:13:43] ...optimizers/trijoint.py.151: total_norm: 11.712142 [I 2022-10-09 22:13:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 1230/2304 [I 2022-10-09 22:13:51] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:07 [I 2022-10-09 22:13:51] ...trap/engines/engine.py.229: process: 0.84031 | load: 0.00296 [I 2022-10-09 22:13:51] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-09 22:13:51] ...optimizers/trijoint.py.151: total_norm: 12.275968 [I 2022-10-09 22:13:59] ...trap/engines/engine.py.224: train: epoch 56 | batch 1240/2304 [I 2022-10-09 22:13:59] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:15:01 [I 2022-10-09 22:13:59] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00302 [I 2022-10-09 22:13:59] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-09 22:13:59] ...optimizers/trijoint.py.151: total_norm: 12.938988 [I 2022-10-09 22:14:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 1250/2304 [I 2022-10-09 22:14:08] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:46 [I 2022-10-09 22:14:08] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00292 [I 2022-10-09 22:14:08] ...trap/engines/engine.py.230: loss: 0.16861 [I 2022-10-09 22:14:08] ...optimizers/trijoint.py.151: total_norm: 18.622793 [I 2022-10-09 22:14:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 1260/2304 [I 2022-10-09 22:14:16] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:43 [I 2022-10-09 22:14:16] ...trap/engines/engine.py.229: process: 0.85288 | load: 0.00298 [I 2022-10-09 22:14:16] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-09 22:14:16] ...optimizers/trijoint.py.151: total_norm: 26.486530 [I 2022-10-09 22:14:25] ...trap/engines/engine.py.224: train: epoch 56 | batch 1270/2304 [I 2022-10-09 22:14:25] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:32 [I 2022-10-09 22:14:25] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00292 [I 2022-10-09 22:14:25] ...trap/engines/engine.py.230: loss: 0.20529 [I 2022-10-09 22:14:25] ...optimizers/trijoint.py.151: total_norm: 26.128593 [I 2022-10-09 22:14:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 1280/2304 [I 2022-10-09 22:14:33] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:20 [I 2022-10-09 22:14:33] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00296 [I 2022-10-09 22:14:33] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-09 22:14:33] ...optimizers/trijoint.py.151: total_norm: 20.110258 [I 2022-10-09 22:14:42] ...trap/engines/engine.py.224: train: epoch 56 | batch 1290/2304 [I 2022-10-09 22:14:42] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:20 [I 2022-10-09 22:14:42] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00297 [I 2022-10-09 22:14:42] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-09 22:14:42] ...optimizers/trijoint.py.151: total_norm: 15.746582 [I 2022-10-09 22:14:50] ...trap/engines/engine.py.224: train: epoch 56 | batch 1300/2304 [I 2022-10-09 22:14:50] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:08 [I 2022-10-09 22:14:50] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00299 [I 2022-10-09 22:14:50] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-09 22:14:50] ...optimizers/trijoint.py.151: total_norm: 14.293521 [I 2022-10-09 22:14:59] ...trap/engines/engine.py.224: train: epoch 56 | batch 1310/2304 [I 2022-10-09 22:14:59] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:00 [I 2022-10-09 22:14:59] ...trap/engines/engine.py.229: process: 0.86042 | load: 0.00302 [I 2022-10-09 22:14:59] ...trap/engines/engine.py.230: loss: 0.17399 [I 2022-10-09 22:14:59] ...optimizers/trijoint.py.151: total_norm: 20.137423 [I 2022-10-09 22:15:07] ...trap/engines/engine.py.224: train: epoch 56 | batch 1320/2304 [I 2022-10-09 22:15:07] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:45 [I 2022-10-09 22:15:07] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00300 [I 2022-10-09 22:15:07] ...trap/engines/engine.py.230: loss: 0.16394 [I 2022-10-09 22:15:07] ...optimizers/trijoint.py.151: total_norm: 11.508037 [I 2022-10-09 22:15:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 1330/2304 [I 2022-10-09 22:15:16] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:39 [I 2022-10-09 22:15:16] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00296 [I 2022-10-09 22:15:16] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-09 22:15:16] ...optimizers/trijoint.py.151: total_norm: 15.170958 [I 2022-10-09 22:15:24] ...trap/engines/engine.py.224: train: epoch 56 | batch 1340/2304 [I 2022-10-09 22:15:24] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:32 [I 2022-10-09 22:15:24] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00300 [I 2022-10-09 22:15:24] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-09 22:15:24] ...optimizers/trijoint.py.151: total_norm: 19.843052 [I 2022-10-09 22:15:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 1350/2304 [I 2022-10-09 22:15:33] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:25 [I 2022-10-09 22:15:33] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00293 [I 2022-10-09 22:15:33] ...trap/engines/engine.py.230: loss: 0.18962 [I 2022-10-09 22:15:33] ...optimizers/trijoint.py.151: total_norm: 17.243988 [I 2022-10-09 22:15:41] ...trap/engines/engine.py.224: train: epoch 56 | batch 1360/2304 [I 2022-10-09 22:15:41] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:19 [I 2022-10-09 22:15:41] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00302 [I 2022-10-09 22:15:41] ...trap/engines/engine.py.230: loss: 0.16730 [I 2022-10-09 22:15:41] ...optimizers/trijoint.py.151: total_norm: 21.911755 [I 2022-10-09 22:15:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 1370/2304 [I 2022-10-09 22:15:49] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:08 [I 2022-10-09 22:15:49] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00298 [I 2022-10-09 22:15:49] ...trap/engines/engine.py.230: loss: 0.23693 [I 2022-10-09 22:15:49] ...optimizers/trijoint.py.151: total_norm: 15.903160 [I 2022-10-09 22:15:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 1380/2304 [I 2022-10-09 22:15:58] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:59 [I 2022-10-09 22:15:58] ...trap/engines/engine.py.229: process: 0.84138 | load: 0.00301 [I 2022-10-09 22:15:58] ...trap/engines/engine.py.230: loss: 0.10795 [I 2022-10-09 22:15:58] ...optimizers/trijoint.py.151: total_norm: 8.112088 [I 2022-10-09 22:16:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1390/2304 [I 2022-10-09 22:16:06] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:48 [I 2022-10-09 22:16:06] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00297 [I 2022-10-09 22:16:06] ...trap/engines/engine.py.230: loss: 0.13742 [I 2022-10-09 22:16:06] ...optimizers/trijoint.py.151: total_norm: 16.782309 [I 2022-10-09 22:16:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 1400/2304 [I 2022-10-09 22:16:15] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:49 [I 2022-10-09 22:16:15] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00315 [I 2022-10-09 22:16:15] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-09 22:16:15] ...optimizers/trijoint.py.151: total_norm: 17.098513 [I 2022-10-09 22:16:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1410/2304 [I 2022-10-09 22:16:23] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:41 [I 2022-10-09 22:16:23] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00313 [I 2022-10-09 22:16:23] ...trap/engines/engine.py.230: loss: 0.14358 [I 2022-10-09 22:16:23] ...optimizers/trijoint.py.151: total_norm: 7.009865 [I 2022-10-09 22:16:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 1420/2304 [I 2022-10-09 22:16:32] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:30 [I 2022-10-09 22:16:32] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00299 [I 2022-10-09 22:16:32] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-09 22:16:32] ...optimizers/trijoint.py.151: total_norm: 17.237471 [I 2022-10-09 22:16:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1430/2304 [I 2022-10-09 22:16:40] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:18 [I 2022-10-09 22:16:40] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00312 [I 2022-10-09 22:16:40] ...trap/engines/engine.py.230: loss: 0.18804 [I 2022-10-09 22:16:40] ...optimizers/trijoint.py.151: total_norm: 47.573399 [I 2022-10-09 22:16:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 1440/2304 [I 2022-10-09 22:16:49] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:14 [I 2022-10-09 22:16:49] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00313 [I 2022-10-09 22:16:49] ...trap/engines/engine.py.230: loss: 0.11022 [I 2022-10-09 22:16:49] ...optimizers/trijoint.py.151: total_norm: 7.220693 [I 2022-10-09 22:16:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1450/2304 [I 2022-10-09 22:16:57] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:08 [I 2022-10-09 22:16:57] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00309 [I 2022-10-09 22:16:57] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-09 22:16:57] ...optimizers/trijoint.py.151: total_norm: 13.430264 [I 2022-10-09 22:17:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1460/2304 [I 2022-10-09 22:17:06] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:58 [I 2022-10-09 22:17:06] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00301 [I 2022-10-09 22:17:06] ...trap/engines/engine.py.230: loss: 0.11597 [I 2022-10-09 22:17:06] ...optimizers/trijoint.py.151: total_norm: 17.103456 [I 2022-10-09 22:17:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 1470/2304 [I 2022-10-09 22:17:15] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:52 [I 2022-10-09 22:17:15] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00311 [I 2022-10-09 22:17:15] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-09 22:17:15] ...optimizers/trijoint.py.151: total_norm: 11.265321 [I 2022-10-09 22:17:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1480/2304 [I 2022-10-09 22:17:23] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:44 [I 2022-10-09 22:17:23] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00314 [I 2022-10-09 22:17:23] ...trap/engines/engine.py.230: loss: 0.16768 [I 2022-10-09 22:17:23] ...optimizers/trijoint.py.151: total_norm: 14.478399 [I 2022-10-09 22:17:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 1490/2304 [I 2022-10-09 22:17:32] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:31 [I 2022-10-09 22:17:32] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00297 [I 2022-10-09 22:17:32] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-09 22:17:32] ...optimizers/trijoint.py.151: total_norm: 18.042557 [I 2022-10-09 22:17:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1500/2304 [I 2022-10-09 22:17:40] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:26 [I 2022-10-09 22:17:40] ...trap/engines/engine.py.229: process: 0.86167 | load: 0.00312 [I 2022-10-09 22:17:40] ...trap/engines/engine.py.230: loss: 0.14821 [I 2022-10-09 22:17:40] ...optimizers/trijoint.py.151: total_norm: 14.471920 [I 2022-10-09 22:17:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 1510/2304 [I 2022-10-09 22:17:49] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:17 [I 2022-10-09 22:17:49] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00313 [I 2022-10-09 22:17:49] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-09 22:17:49] ...optimizers/trijoint.py.151: total_norm: 5.915529 [I 2022-10-09 22:17:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1520/2304 [I 2022-10-09 22:17:57] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:08 [I 2022-10-09 22:17:57] ...trap/engines/engine.py.229: process: 0.85158 | load: 0.00314 [I 2022-10-09 22:17:57] ...trap/engines/engine.py.230: loss: 0.18927 [I 2022-10-09 22:17:57] ...optimizers/trijoint.py.151: total_norm: 53.075951 [I 2022-10-09 22:18:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1530/2304 [I 2022-10-09 22:18:06] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:03 [I 2022-10-09 22:18:06] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00311 [I 2022-10-09 22:18:06] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-09 22:18:06] ...optimizers/trijoint.py.151: total_norm: 17.577229 [I 2022-10-09 22:18:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1540/2304 [I 2022-10-09 22:18:14] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:52 [I 2022-10-09 22:18:14] ...trap/engines/engine.py.229: process: 0.85622 | load: 0.00309 [I 2022-10-09 22:18:14] ...trap/engines/engine.py.230: loss: 0.09592 [I 2022-10-09 22:18:14] ...optimizers/trijoint.py.151: total_norm: 11.272321 [I 2022-10-09 22:18:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1550/2304 [I 2022-10-09 22:18:23] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:42 [I 2022-10-09 22:18:23] ...trap/engines/engine.py.229: process: 0.84539 | load: 0.00311 [I 2022-10-09 22:18:23] ...trap/engines/engine.py.230: loss: 0.12234 [I 2022-10-09 22:18:23] ...optimizers/trijoint.py.151: total_norm: 11.376410 [I 2022-10-09 22:18:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 1560/2304 [I 2022-10-09 22:18:32] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:38 [I 2022-10-09 22:18:32] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00312 [I 2022-10-09 22:18:32] ...trap/engines/engine.py.230: loss: 0.11891 [I 2022-10-09 22:18:32] ...optimizers/trijoint.py.151: total_norm: 21.293493 [I 2022-10-09 22:18:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1570/2304 [I 2022-10-09 22:18:40] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:26 [I 2022-10-09 22:18:40] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00314 [I 2022-10-09 22:18:40] ...trap/engines/engine.py.230: loss: 0.16484 [I 2022-10-09 22:18:40] ...optimizers/trijoint.py.151: total_norm: 12.591409 [I 2022-10-09 22:18:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 1580/2304 [I 2022-10-09 22:18:49] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:18 [I 2022-10-09 22:18:49] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00312 [I 2022-10-09 22:18:49] ...trap/engines/engine.py.230: loss: 0.17696 [I 2022-10-09 22:18:49] ...optimizers/trijoint.py.151: total_norm: 17.462555 [I 2022-10-09 22:18:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1590/2304 [I 2022-10-09 22:18:57] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:12 [I 2022-10-09 22:18:57] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00311 [I 2022-10-09 22:18:57] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-09 22:18:57] ...optimizers/trijoint.py.151: total_norm: 18.802824 [I 2022-10-09 22:19:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1600/2304 [I 2022-10-09 22:19:06] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:59 [I 2022-10-09 22:19:06] ...trap/engines/engine.py.229: process: 0.86143 | load: 0.00313 [I 2022-10-09 22:19:06] ...trap/engines/engine.py.230: loss: 0.12651 [I 2022-10-09 22:19:06] ...optimizers/trijoint.py.151: total_norm: 12.860070 [I 2022-10-09 22:19:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1610/2304 [I 2022-10-09 22:19:14] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:53 [I 2022-10-09 22:19:14] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00321 [I 2022-10-09 22:19:14] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-09 22:19:14] ...optimizers/trijoint.py.151: total_norm: 22.678205 [I 2022-10-09 22:19:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1620/2304 [I 2022-10-09 22:19:23] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:44 [I 2022-10-09 22:19:23] ...trap/engines/engine.py.229: process: 0.85102 | load: 0.00314 [I 2022-10-09 22:19:23] ...trap/engines/engine.py.230: loss: 0.22754 [I 2022-10-09 22:19:23] ...optimizers/trijoint.py.151: total_norm: 24.048409 [I 2022-10-09 22:19:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 1630/2304 [I 2022-10-09 22:19:31] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:36 [I 2022-10-09 22:19:31] ...trap/engines/engine.py.229: process: 0.85577 | load: 0.00314 [I 2022-10-09 22:19:31] ...trap/engines/engine.py.230: loss: 0.17094 [I 2022-10-09 22:19:31] ...optimizers/trijoint.py.151: total_norm: 15.682481 [I 2022-10-09 22:19:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1640/2304 [I 2022-10-09 22:19:40] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:27 [I 2022-10-09 22:19:40] ...trap/engines/engine.py.229: process: 0.86004 | load: 0.00313 [I 2022-10-09 22:19:40] ...trap/engines/engine.py.230: loss: 0.17012 [I 2022-10-09 22:19:40] ...optimizers/trijoint.py.151: total_norm: 63.983192 [I 2022-10-09 22:19:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 1650/2304 [I 2022-10-09 22:19:48] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:22 [I 2022-10-09 22:19:48] ...trap/engines/engine.py.229: process: 0.87183 | load: 0.00309 [I 2022-10-09 22:19:48] ...trap/engines/engine.py.230: loss: 0.15981 [I 2022-10-09 22:19:48] ...optimizers/trijoint.py.151: total_norm: 15.276254 [I 2022-10-09 22:19:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1660/2304 [I 2022-10-09 22:19:57] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:09:10 [I 2022-10-09 22:19:57] ...trap/engines/engine.py.229: process: 0.86891 | load: 0.00312 [I 2022-10-09 22:19:57] ...trap/engines/engine.py.230: loss: 0.10509 [I 2022-10-09 22:19:57] ...optimizers/trijoint.py.151: total_norm: 10.343469 [I 2022-10-09 22:20:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1670/2304 [I 2022-10-09 22:20:06] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:09:02 [I 2022-10-09 22:20:06] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00310 [I 2022-10-09 22:20:06] ...trap/engines/engine.py.230: loss: 0.20914 [I 2022-10-09 22:20:06] ...optimizers/trijoint.py.151: total_norm: 24.873354 [I 2022-10-09 22:20:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1680/2304 [I 2022-10-09 22:20:14] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:55 [I 2022-10-09 22:20:14] ...trap/engines/engine.py.229: process: 0.86654 | load: 0.00313 [I 2022-10-09 22:20:14] ...trap/engines/engine.py.230: loss: 0.15555 [I 2022-10-09 22:20:14] ...optimizers/trijoint.py.151: total_norm: 12.518441 [I 2022-10-09 22:20:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1690/2304 [I 2022-10-09 22:20:23] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:45 [I 2022-10-09 22:20:23] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00309 [I 2022-10-09 22:20:23] ...trap/engines/engine.py.230: loss: 0.11224 [I 2022-10-09 22:20:23] ...optimizers/trijoint.py.151: total_norm: 10.788079 [I 2022-10-09 22:20:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 1700/2304 [I 2022-10-09 22:20:31] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:34 [I 2022-10-09 22:20:31] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00313 [I 2022-10-09 22:20:31] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-09 22:20:31] ...optimizers/trijoint.py.151: total_norm: 24.182310 [I 2022-10-09 22:20:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1710/2304 [I 2022-10-09 22:20:40] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:25 [I 2022-10-09 22:20:40] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00312 [I 2022-10-09 22:20:40] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-09 22:20:40] ...optimizers/trijoint.py.151: total_norm: 17.687897 [I 2022-10-09 22:20:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 1720/2304 [I 2022-10-09 22:20:48] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:10 [I 2022-10-09 22:20:48] ...trap/engines/engine.py.229: process: 0.82729 | load: 0.00300 [I 2022-10-09 22:20:48] ...trap/engines/engine.py.230: loss: 0.17206 [I 2022-10-09 22:20:48] ...optimizers/trijoint.py.151: total_norm: 20.303696 [I 2022-10-09 22:20:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1730/2304 [I 2022-10-09 22:20:57] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:03 [I 2022-10-09 22:20:57] ...trap/engines/engine.py.229: process: 0.83160 | load: 0.00307 [I 2022-10-09 22:20:57] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-09 22:20:57] ...optimizers/trijoint.py.151: total_norm: 15.251473 [I 2022-10-09 22:21:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 1740/2304 [I 2022-10-09 22:21:05] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:55 [I 2022-10-09 22:21:05] ...trap/engines/engine.py.229: process: 0.83841 | load: 0.00299 [I 2022-10-09 22:21:05] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-09 22:21:05] ...optimizers/trijoint.py.151: total_norm: 20.149063 [I 2022-10-09 22:21:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1750/2304 [I 2022-10-09 22:21:14] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:47 [I 2022-10-09 22:21:14] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00295 [I 2022-10-09 22:21:14] ...trap/engines/engine.py.230: loss: 0.23222 [I 2022-10-09 22:21:14] ...optimizers/trijoint.py.151: total_norm: 50.082260 [I 2022-10-09 22:21:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 1760/2304 [I 2022-10-09 22:21:22] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:37 [I 2022-10-09 22:21:22] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00298 [I 2022-10-09 22:21:22] ...trap/engines/engine.py.230: loss: 0.19908 [I 2022-10-09 22:21:22] ...optimizers/trijoint.py.151: total_norm: 20.926949 [I 2022-10-09 22:21:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 1770/2304 [I 2022-10-09 22:21:31] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:30 [I 2022-10-09 22:21:31] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00295 [I 2022-10-09 22:21:31] ...trap/engines/engine.py.230: loss: 0.14932 [I 2022-10-09 22:21:31] ...optimizers/trijoint.py.151: total_norm: 20.016619 [I 2022-10-09 22:21:39] ...trap/engines/engine.py.224: train: epoch 56 | batch 1780/2304 [I 2022-10-09 22:21:39] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:24 [I 2022-10-09 22:21:39] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00294 [I 2022-10-09 22:21:39] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-09 22:21:39] ...optimizers/trijoint.py.151: total_norm: 11.664973 [I 2022-10-09 22:21:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 1790/2304 [I 2022-10-09 22:21:47] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:12 [I 2022-10-09 22:21:47] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00296 [I 2022-10-09 22:21:47] ...trap/engines/engine.py.230: loss: 0.20127 [I 2022-10-09 22:21:47] ...optimizers/trijoint.py.151: total_norm: 21.450178 [I 2022-10-09 22:21:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 1800/2304 [I 2022-10-09 22:21:56] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:07:03 [I 2022-10-09 22:21:56] ...trap/engines/engine.py.229: process: 0.83558 | load: 0.00296 [I 2022-10-09 22:21:56] ...trap/engines/engine.py.230: loss: 0.18594 [I 2022-10-09 22:21:56] ...optimizers/trijoint.py.151: total_norm: 10.561546 [I 2022-10-09 22:22:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 1810/2304 [I 2022-10-09 22:22:04] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:55 [I 2022-10-09 22:22:04] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00295 [I 2022-10-09 22:22:04] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-09 22:22:04] ...optimizers/trijoint.py.151: total_norm: 15.557108 [I 2022-10-09 22:22:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 1820/2304 [I 2022-10-09 22:22:13] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:50 [I 2022-10-09 22:22:13] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00291 [I 2022-10-09 22:22:13] ...trap/engines/engine.py.230: loss: 0.13146 [I 2022-10-09 22:22:13] ...optimizers/trijoint.py.151: total_norm: 18.111496 [I 2022-10-09 22:22:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 1830/2304 [I 2022-10-09 22:22:21] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:39 [I 2022-10-09 22:22:21] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00295 [I 2022-10-09 22:22:21] ...trap/engines/engine.py.230: loss: 0.11638 [I 2022-10-09 22:22:21] ...optimizers/trijoint.py.151: total_norm: 25.746811 [I 2022-10-09 22:22:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 1840/2304 [I 2022-10-09 22:22:30] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:34 [I 2022-10-09 22:22:30] ...trap/engines/engine.py.229: process: 0.87403 | load: 0.00294 [I 2022-10-09 22:22:30] ...trap/engines/engine.py.230: loss: 0.15109 [I 2022-10-09 22:22:30] ...optimizers/trijoint.py.151: total_norm: 8.450611 [I 2022-10-09 22:22:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 1850/2304 [I 2022-10-09 22:22:38] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:23 [I 2022-10-09 22:22:38] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00301 [I 2022-10-09 22:22:38] ...trap/engines/engine.py.230: loss: 0.22428 [I 2022-10-09 22:22:38] ...optimizers/trijoint.py.151: total_norm: 20.086363 [I 2022-10-09 22:22:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 1860/2304 [I 2022-10-09 22:22:47] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:15 [I 2022-10-09 22:22:47] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00293 [I 2022-10-09 22:22:47] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-09 22:22:47] ...optimizers/trijoint.py.151: total_norm: 10.836699 [I 2022-10-09 22:22:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 1870/2304 [I 2022-10-09 22:22:55] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:09 [I 2022-10-09 22:22:55] ...trap/engines/engine.py.229: process: 0.84020 | load: 0.00303 [I 2022-10-09 22:22:55] ...trap/engines/engine.py.230: loss: 0.15098 [I 2022-10-09 22:22:55] ...optimizers/trijoint.py.151: total_norm: 21.058172 [I 2022-10-09 22:23:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 1880/2304 [I 2022-10-09 22:23:04] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:59 [I 2022-10-09 22:23:04] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00296 [I 2022-10-09 22:23:04] ...trap/engines/engine.py.230: loss: 0.15415 [I 2022-10-09 22:23:04] ...optimizers/trijoint.py.151: total_norm: 11.300476 [I 2022-10-09 22:23:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 1890/2304 [I 2022-10-09 22:23:12] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:47 [I 2022-10-09 22:23:12] ...trap/engines/engine.py.229: process: 0.83533 | load: 0.00302 [I 2022-10-09 22:23:12] ...trap/engines/engine.py.230: loss: 0.11875 [I 2022-10-09 22:23:12] ...optimizers/trijoint.py.151: total_norm: 12.370687 [I 2022-10-09 22:23:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 1900/2304 [I 2022-10-09 22:23:21] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:40 [I 2022-10-09 22:23:21] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00296 [I 2022-10-09 22:23:21] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-09 22:23:21] ...optimizers/trijoint.py.151: total_norm: 21.617956 [I 2022-10-09 22:23:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 1910/2304 [I 2022-10-09 22:23:29] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:31 [I 2022-10-09 22:23:29] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00297 [I 2022-10-09 22:23:29] ...trap/engines/engine.py.230: loss: 0.15183 [I 2022-10-09 22:23:29] ...optimizers/trijoint.py.151: total_norm: 28.154129 [I 2022-10-09 22:23:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 1920/2304 [I 2022-10-09 22:23:38] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:24 [I 2022-10-09 22:23:38] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00295 [I 2022-10-09 22:23:38] ...trap/engines/engine.py.230: loss: 0.19418 [I 2022-10-09 22:23:38] ...optimizers/trijoint.py.151: total_norm: 15.640549 [I 2022-10-09 22:23:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 1930/2304 [I 2022-10-09 22:23:46] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:17 [I 2022-10-09 22:23:46] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00295 [I 2022-10-09 22:23:46] ...trap/engines/engine.py.230: loss: 0.11455 [I 2022-10-09 22:23:46] ...optimizers/trijoint.py.151: total_norm: 8.704901 [I 2022-10-09 22:23:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 1940/2304 [I 2022-10-09 22:23:55] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:10 [I 2022-10-09 22:23:55] ...trap/engines/engine.py.229: process: 0.85266 | load: 0.00304 [I 2022-10-09 22:23:55] ...trap/engines/engine.py.230: loss: 0.14211 [I 2022-10-09 22:23:55] ...optimizers/trijoint.py.151: total_norm: 10.136411 [I 2022-10-09 22:24:03] ...trap/engines/engine.py.224: train: epoch 56 | batch 1950/2304 [I 2022-10-09 22:24:03] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:01 [I 2022-10-09 22:24:03] ...trap/engines/engine.py.229: process: 0.86586 | load: 0.00295 [I 2022-10-09 22:24:03] ...trap/engines/engine.py.230: loss: 0.10754 [I 2022-10-09 22:24:03] ...optimizers/trijoint.py.151: total_norm: 14.716603 [I 2022-10-09 22:24:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 1960/2304 [I 2022-10-09 22:24:12] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:53 [I 2022-10-09 22:24:12] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00295 [I 2022-10-09 22:24:12] ...trap/engines/engine.py.230: loss: 0.12453 [I 2022-10-09 22:24:12] ...optimizers/trijoint.py.151: total_norm: 12.781581 [I 2022-10-09 22:24:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 1970/2304 [I 2022-10-09 22:24:20] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:43 [I 2022-10-09 22:24:20] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00294 [I 2022-10-09 22:24:20] ...trap/engines/engine.py.230: loss: 0.25475 [I 2022-10-09 22:24:20] ...optimizers/trijoint.py.151: total_norm: 13.907719 [I 2022-10-09 22:24:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 1980/2304 [I 2022-10-09 22:24:29] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:33 [I 2022-10-09 22:24:29] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00297 [I 2022-10-09 22:24:29] ...trap/engines/engine.py.230: loss: 0.14487 [I 2022-10-09 22:24:29] ...optimizers/trijoint.py.151: total_norm: 11.897168 [I 2022-10-09 22:24:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 1990/2304 [I 2022-10-09 22:24:37] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:26 [I 2022-10-09 22:24:37] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00303 [I 2022-10-09 22:24:37] ...trap/engines/engine.py.230: loss: 0.15790 [I 2022-10-09 22:24:37] ...optimizers/trijoint.py.151: total_norm: 18.473204 [I 2022-10-09 22:24:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 2000/2304 [I 2022-10-09 22:24:46] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:18 [I 2022-10-09 22:24:46] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00315 [I 2022-10-09 22:24:46] ...trap/engines/engine.py.230: loss: 0.16953 [I 2022-10-09 22:24:46] ...optimizers/trijoint.py.151: total_norm: 18.500450 [I 2022-10-09 22:24:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 2010/2304 [I 2022-10-09 22:24:54] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:10 [I 2022-10-09 22:24:54] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00316 [I 2022-10-09 22:24:54] ...trap/engines/engine.py.230: loss: 0.15254 [I 2022-10-09 22:24:54] ...optimizers/trijoint.py.151: total_norm: 18.728266 [I 2022-10-09 22:25:03] ...trap/engines/engine.py.224: train: epoch 56 | batch 2020/2304 [I 2022-10-09 22:25:03] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:04:02 [I 2022-10-09 22:25:03] ...trap/engines/engine.py.229: process: 0.84403 | load: 0.00318 [I 2022-10-09 22:25:03] ...trap/engines/engine.py.230: loss: 0.15600 [I 2022-10-09 22:25:03] ...optimizers/trijoint.py.151: total_norm: 10.924475 [I 2022-10-09 22:25:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 2030/2304 [I 2022-10-09 22:25:11] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:52 [I 2022-10-09 22:25:11] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00319 [I 2022-10-09 22:25:11] ...trap/engines/engine.py.230: loss: 0.10094 [I 2022-10-09 22:25:11] ...optimizers/trijoint.py.151: total_norm: 7.041679 [I 2022-10-09 22:25:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 2040/2304 [I 2022-10-09 22:25:20] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:44 [I 2022-10-09 22:25:20] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00315 [I 2022-10-09 22:25:20] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-09 22:25:20] ...optimizers/trijoint.py.151: total_norm: 10.567633 [I 2022-10-09 22:25:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 2050/2304 [I 2022-10-09 22:25:28] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:37 [I 2022-10-09 22:25:28] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00316 [I 2022-10-09 22:25:28] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-09 22:25:28] ...optimizers/trijoint.py.151: total_norm: 9.576008 [I 2022-10-09 22:25:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 2060/2304 [I 2022-10-09 22:25:37] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:27 [I 2022-10-09 22:25:37] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00313 [I 2022-10-09 22:25:37] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-09 22:25:37] ...optimizers/trijoint.py.151: total_norm: 18.740587 [I 2022-10-09 22:25:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 2070/2304 [I 2022-10-09 22:25:45] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:19 [I 2022-10-09 22:25:45] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00316 [I 2022-10-09 22:25:45] ...trap/engines/engine.py.230: loss: 0.19172 [I 2022-10-09 22:25:45] ...optimizers/trijoint.py.151: total_norm: 13.018919 [I 2022-10-09 22:25:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 2080/2304 [I 2022-10-09 22:25:54] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:03:10 [I 2022-10-09 22:25:54] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00315 [I 2022-10-09 22:25:54] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-09 22:25:54] ...optimizers/trijoint.py.151: total_norm: 9.095650 [I 2022-10-09 22:26:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 2090/2304 [I 2022-10-09 22:26:02] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:02 [I 2022-10-09 22:26:02] ...trap/engines/engine.py.229: process: 0.85335 | load: 0.00321 [I 2022-10-09 22:26:02] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-09 22:26:02] ...optimizers/trijoint.py.151: total_norm: 13.180538 [I 2022-10-09 22:26:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 2100/2304 [I 2022-10-09 22:26:11] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:54 [I 2022-10-09 22:26:11] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00314 [I 2022-10-09 22:26:11] ...trap/engines/engine.py.230: loss: 0.15466 [I 2022-10-09 22:26:11] ...optimizers/trijoint.py.151: total_norm: 25.871687 [I 2022-10-09 22:26:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 2110/2304 [I 2022-10-09 22:26:19] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:45 [I 2022-10-09 22:26:19] ...trap/engines/engine.py.229: process: 0.86075 | load: 0.00324 [I 2022-10-09 22:26:19] ...trap/engines/engine.py.230: loss: 0.09786 [I 2022-10-09 22:26:19] ...optimizers/trijoint.py.151: total_norm: 12.826847 [I 2022-10-09 22:26:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 2120/2304 [I 2022-10-09 22:26:28] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:35 [I 2022-10-09 22:26:28] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00310 [I 2022-10-09 22:26:28] ...trap/engines/engine.py.230: loss: 0.20387 [I 2022-10-09 22:26:28] ...optimizers/trijoint.py.151: total_norm: 22.310682 [I 2022-10-09 22:26:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 2130/2304 [I 2022-10-09 22:26:36] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:27 [I 2022-10-09 22:26:36] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00310 [I 2022-10-09 22:26:36] ...trap/engines/engine.py.230: loss: 0.13880 [I 2022-10-09 22:26:36] ...optimizers/trijoint.py.151: total_norm: 27.520346 [I 2022-10-09 22:26:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 2140/2304 [I 2022-10-09 22:26:45] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:19 [I 2022-10-09 22:26:45] ...trap/engines/engine.py.229: process: 0.88358 | load: 0.00311 [I 2022-10-09 22:26:45] ...trap/engines/engine.py.230: loss: 0.12837 [I 2022-10-09 22:26:45] ...optimizers/trijoint.py.151: total_norm: 19.044876 [I 2022-10-09 22:26:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 2150/2304 [I 2022-10-09 22:26:54] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:11 [I 2022-10-09 22:26:54] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00309 [I 2022-10-09 22:26:54] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-09 22:26:54] ...optimizers/trijoint.py.151: total_norm: 15.054087 [I 2022-10-09 22:27:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 2160/2304 [I 2022-10-09 22:27:02] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:03 [I 2022-10-09 22:27:02] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00319 [I 2022-10-09 22:27:02] ...trap/engines/engine.py.230: loss: 0.14247 [I 2022-10-09 22:27:02] ...optimizers/trijoint.py.151: total_norm: 34.643150 [I 2022-10-09 22:27:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 2170/2304 [I 2022-10-09 22:27:11] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:53 [I 2022-10-09 22:27:11] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00313 [I 2022-10-09 22:27:11] ...trap/engines/engine.py.230: loss: 0.22097 [I 2022-10-09 22:27:11] ...optimizers/trijoint.py.151: total_norm: 31.788704 [I 2022-10-09 22:27:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 2180/2304 [I 2022-10-09 22:27:19] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:45 [I 2022-10-09 22:27:19] ...trap/engines/engine.py.229: process: 0.86220 | load: 0.00313 [I 2022-10-09 22:27:19] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-09 22:27:19] ...optimizers/trijoint.py.151: total_norm: 34.684792 [I 2022-10-09 22:27:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 2190/2304 [I 2022-10-09 22:27:28] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:37 [I 2022-10-09 22:27:28] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00304 [I 2022-10-09 22:27:28] ...trap/engines/engine.py.230: loss: 0.19058 [I 2022-10-09 22:27:28] ...optimizers/trijoint.py.151: total_norm: 28.294603 [I 2022-10-09 22:27:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 2200/2304 [I 2022-10-09 22:27:36] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:28 [I 2022-10-09 22:27:36] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00309 [I 2022-10-09 22:27:36] ...trap/engines/engine.py.230: loss: 0.14077 [I 2022-10-09 22:27:36] ...optimizers/trijoint.py.151: total_norm: 12.592759 [I 2022-10-09 22:27:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 2210/2304 [I 2022-10-09 22:27:45] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:19 [I 2022-10-09 22:27:45] ...trap/engines/engine.py.229: process: 0.85048 | load: 0.00312 [I 2022-10-09 22:27:45] ...trap/engines/engine.py.230: loss: 0.11842 [I 2022-10-09 22:27:45] ...optimizers/trijoint.py.151: total_norm: 11.395887 [I 2022-10-09 22:27:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 2220/2304 [I 2022-10-09 22:27:53] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:11 [I 2022-10-09 22:27:53] ...trap/engines/engine.py.229: process: 0.86271 | load: 0.00309 [I 2022-10-09 22:27:53] ...trap/engines/engine.py.230: loss: 0.19716 [I 2022-10-09 22:27:53] ...optimizers/trijoint.py.151: total_norm: 15.521770 [I 2022-10-09 22:28:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 2230/2304 [I 2022-10-09 22:28:02] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:03 [I 2022-10-09 22:28:02] ...trap/engines/engine.py.229: process: 0.85835 | load: 0.00318 [I 2022-10-09 22:28:02] ...trap/engines/engine.py.230: loss: 0.13239 [I 2022-10-09 22:28:02] ...optimizers/trijoint.py.151: total_norm: 11.532490 [I 2022-10-09 22:28:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 2240/2304 [I 2022-10-09 22:28:10] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:54 [I 2022-10-09 22:28:10] ...trap/engines/engine.py.229: process: 0.86250 | load: 0.00311 [I 2022-10-09 22:28:10] ...trap/engines/engine.py.230: loss: 0.16412 [I 2022-10-09 22:28:10] ...optimizers/trijoint.py.151: total_norm: 20.827757 [I 2022-10-09 22:28:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 2250/2304 [I 2022-10-09 22:28:19] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:45 [I 2022-10-09 22:28:19] ...trap/engines/engine.py.229: process: 0.83199 | load: 0.00310 [I 2022-10-09 22:28:19] ...trap/engines/engine.py.230: loss: 0.14124 [I 2022-10-09 22:28:19] ...optimizers/trijoint.py.151: total_norm: 13.068305 [I 2022-10-09 22:28:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 2260/2304 [I 2022-10-09 22:28:27] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:37 [I 2022-10-09 22:28:27] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00314 [I 2022-10-09 22:28:27] ...trap/engines/engine.py.230: loss: 0.18502 [I 2022-10-09 22:28:27] ...optimizers/trijoint.py.151: total_norm: 15.742188 [I 2022-10-09 22:28:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 2270/2304 [I 2022-10-09 22:28:36] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:28 [I 2022-10-09 22:28:36] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00313 [I 2022-10-09 22:28:36] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-09 22:28:36] ...optimizers/trijoint.py.151: total_norm: 18.382122 [I 2022-10-09 22:28:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 2280/2304 [I 2022-10-09 22:28:44] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:20 [I 2022-10-09 22:28:44] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00312 [I 2022-10-09 22:28:44] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-09 22:28:44] ...optimizers/trijoint.py.151: total_norm: 8.885190 [I 2022-10-09 22:28:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 2290/2304 [I 2022-10-09 22:28:53] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:11 [I 2022-10-09 22:28:53] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00314 [I 2022-10-09 22:28:53] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-09 22:28:53] ...optimizers/trijoint.py.151: total_norm: 13.769030 [I 2022-10-09 22:29:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 2300/2304 [I 2022-10-09 22:29:02] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:03 [I 2022-10-09 22:29:02] ...trap/engines/engine.py.229: process: 0.83220 | load: 0.00019 [I 2022-10-09 22:29:02] ...trap/engines/engine.py.230: loss: 0.14367 [I 2022-10-09 22:29:02] ...optimizers/trijoint.py.151: total_norm: 12.112555 [I 2022-10-09 22:29:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 2304/2304 [I 2022-10-09 22:29:05] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:00 [I 2022-10-09 22:29:05] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00019 [I 2022-10-09 22:29:05] ...trap/engines/engine.py.230: loss: 0.09948 [I 2022-10-09 22:29:05] ...optimizers/trijoint.py.151: total_norm: 13.255162 [S 2022-10-09 22:29:06] ...trap/engines/engine.py.129: train_epoch.epoch: 56 [S 2022-10-09 22:29:06] ...trap/engines/engine.py.129: train_epoch.loss: 0.15192970754073654 [I 2022-10-09 22:29:06] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 22:29:07] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 56 [I 2022-10-09 22:29:07] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 22:29:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 22:29:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 22:29:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 22:29:11] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 22:29:11] ...trap/engines/engine.py.321: eval: epoch 56 | batch 0/510 [I 2022-10-09 22:29:11] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:19 [I 2022-10-09 22:29:11] ...trap/engines/engine.py.326: process: 4.39195 | load: 3.75865 [I 2022-10-09 22:29:17] ...trap/engines/engine.py.321: eval: epoch 56 | batch 10/510 [I 2022-10-09 22:29:17] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:54 [I 2022-10-09 22:29:17] ...trap/engines/engine.py.326: process: 0.44992 | load: 0.00019 [I 2022-10-09 22:29:23] ...trap/engines/engine.py.321: eval: epoch 56 | batch 20/510 [I 2022-10-09 22:29:23] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:10 [I 2022-10-09 22:29:23] ...trap/engines/engine.py.326: process: 0.58898 | load: 0.00020 [I 2022-10-09 22:29:29] ...trap/engines/engine.py.321: eval: epoch 56 | batch 30/510 [I 2022-10-09 22:29:29] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:16 [I 2022-10-09 22:29:29] ...trap/engines/engine.py.326: process: 0.62716 | load: 0.00020 [I 2022-10-09 22:29:33] ...trap/engines/engine.py.321: eval: epoch 56 | batch 40/510 [I 2022-10-09 22:29:33] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:36 [I 2022-10-09 22:29:33] ...trap/engines/engine.py.326: process: 0.44553 | load: 0.00020 [I 2022-10-09 22:29:38] ...trap/engines/engine.py.321: eval: epoch 56 | batch 50/510 [I 2022-10-09 22:29:38] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:43 [I 2022-10-09 22:29:38] ...trap/engines/engine.py.326: process: 0.54539 | load: 0.00020 [I 2022-10-09 22:29:43] ...trap/engines/engine.py.321: eval: epoch 56 | batch 60/510 [I 2022-10-09 22:29:43] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:33 [I 2022-10-09 22:29:43] ...trap/engines/engine.py.326: process: 0.55289 | load: 0.00019 [I 2022-10-09 22:29:48] ...trap/engines/engine.py.321: eval: epoch 56 | batch 70/510 [I 2022-10-09 22:29:48] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:07 [I 2022-10-09 22:29:48] ...trap/engines/engine.py.326: process: 0.63783 | load: 0.00020 [I 2022-10-09 22:29:53] ...trap/engines/engine.py.321: eval: epoch 56 | batch 80/510 [I 2022-10-09 22:29:53] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:28 [I 2022-10-09 22:29:53] ...trap/engines/engine.py.326: process: 0.42984 | load: 0.00020 [I 2022-10-09 22:29:58] ...trap/engines/engine.py.321: eval: epoch 56 | batch 90/510 [I 2022-10-09 22:29:58] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:04 [I 2022-10-09 22:29:58] ...trap/engines/engine.py.326: process: 0.42487 | load: 0.00019 [I 2022-10-09 22:30:02] ...trap/engines/engine.py.321: eval: epoch 56 | batch 100/510 [I 2022-10-09 22:30:02] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:56 [I 2022-10-09 22:30:02] ...trap/engines/engine.py.326: process: 0.30885 | load: 0.00018 [I 2022-10-09 22:30:05] ...trap/engines/engine.py.321: eval: epoch 56 | batch 110/510 [I 2022-10-09 22:30:05] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:10 [I 2022-10-09 22:30:05] ...trap/engines/engine.py.326: process: 0.30914 | load: 0.00019 [I 2022-10-09 22:30:08] ...trap/engines/engine.py.321: eval: epoch 56 | batch 120/510 [I 2022-10-09 22:30:08] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:03 [I 2022-10-09 22:30:08] ...trap/engines/engine.py.326: process: 0.31432 | load: 0.00018 [I 2022-10-09 22:30:12] ...trap/engines/engine.py.321: eval: epoch 56 | batch 130/510 [I 2022-10-09 22:30:12] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:59 [I 2022-10-09 22:30:12] ...trap/engines/engine.py.326: process: 0.31682 | load: 0.00020 [I 2022-10-09 22:30:15] ...trap/engines/engine.py.321: eval: epoch 56 | batch 140/510 [I 2022-10-09 22:30:15] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:56 [I 2022-10-09 22:30:15] ...trap/engines/engine.py.326: process: 0.31612 | load: 0.00018 [I 2022-10-09 22:30:18] ...trap/engines/engine.py.321: eval: epoch 56 | batch 150/510 [I 2022-10-09 22:30:18] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:53 [I 2022-10-09 22:30:18] ...trap/engines/engine.py.326: process: 0.31434 | load: 0.00019 [I 2022-10-09 22:30:21] ...trap/engines/engine.py.321: eval: epoch 56 | batch 160/510 [I 2022-10-09 22:30:21] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-09 22:30:21] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00019 [I 2022-10-09 22:30:24] ...trap/engines/engine.py.321: eval: epoch 56 | batch 170/510 [I 2022-10-09 22:30:24] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:46 [I 2022-10-09 22:30:24] ...trap/engines/engine.py.326: process: 0.30699 | load: 0.00020 [I 2022-10-09 22:30:27] ...trap/engines/engine.py.321: eval: epoch 56 | batch 180/510 [I 2022-10-09 22:30:27] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:43 [I 2022-10-09 22:30:27] ...trap/engines/engine.py.326: process: 0.31016 | load: 0.00019 [I 2022-10-09 22:30:30] ...trap/engines/engine.py.321: eval: epoch 56 | batch 190/510 [I 2022-10-09 22:30:30] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:40 [I 2022-10-09 22:30:30] ...trap/engines/engine.py.326: process: 0.31159 | load: 0.00019 [I 2022-10-09 22:30:34] ...trap/engines/engine.py.321: eval: epoch 56 | batch 200/510 [I 2022-10-09 22:30:34] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:37 [I 2022-10-09 22:30:34] ...trap/engines/engine.py.326: process: 0.31593 | load: 0.00019 [I 2022-10-09 22:30:37] ...trap/engines/engine.py.321: eval: epoch 56 | batch 210/510 [I 2022-10-09 22:30:37] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-09 22:30:37] ...trap/engines/engine.py.326: process: 0.31866 | load: 0.00020 [I 2022-10-09 22:30:40] ...trap/engines/engine.py.321: eval: epoch 56 | batch 220/510 [I 2022-10-09 22:30:40] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:31 [I 2022-10-09 22:30:40] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00020 [I 2022-10-09 22:30:43] ...trap/engines/engine.py.321: eval: epoch 56 | batch 230/510 [I 2022-10-09 22:30:43] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:28 [I 2022-10-09 22:30:43] ...trap/engines/engine.py.326: process: 0.31641 | load: 0.00020 [I 2022-10-09 22:30:46] ...trap/engines/engine.py.321: eval: epoch 56 | batch 240/510 [I 2022-10-09 22:30:46] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:25 [I 2022-10-09 22:30:46] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00020 [I 2022-10-09 22:30:49] ...trap/engines/engine.py.321: eval: epoch 56 | batch 250/510 [I 2022-10-09 22:30:49] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:22 [I 2022-10-09 22:30:49] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00019 [I 2022-10-09 22:30:53] ...trap/engines/engine.py.321: eval: epoch 56 | batch 260/510 [I 2022-10-09 22:30:53] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:19 [I 2022-10-09 22:30:53] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00021 [I 2022-10-09 22:30:56] ...trap/engines/engine.py.321: eval: epoch 56 | batch 270/510 [I 2022-10-09 22:30:56] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:16 [I 2022-10-09 22:30:56] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00021 [I 2022-10-09 22:30:59] ...trap/engines/engine.py.321: eval: epoch 56 | batch 280/510 [I 2022-10-09 22:30:59] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:12 [I 2022-10-09 22:30:59] ...trap/engines/engine.py.326: process: 0.30821 | load: 0.00021 [I 2022-10-09 22:31:02] ...trap/engines/engine.py.321: eval: epoch 56 | batch 290/510 [I 2022-10-09 22:31:02] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-09 22:31:02] ...trap/engines/engine.py.326: process: 0.31298 | load: 0.00021 [I 2022-10-09 22:31:05] ...trap/engines/engine.py.321: eval: epoch 56 | batch 300/510 [I 2022-10-09 22:31:05] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:07 [I 2022-10-09 22:31:05] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00021 [I 2022-10-09 22:31:09] ...trap/engines/engine.py.321: eval: epoch 56 | batch 310/510 [I 2022-10-09 22:31:09] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:03 [I 2022-10-09 22:31:09] ...trap/engines/engine.py.326: process: 0.31689 | load: 0.00021 [I 2022-10-09 22:31:12] ...trap/engines/engine.py.321: eval: epoch 56 | batch 320/510 [I 2022-10-09 22:31:12] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:00 [I 2022-10-09 22:31:12] ...trap/engines/engine.py.326: process: 0.32402 | load: 0.00019 [I 2022-10-09 22:31:15] ...trap/engines/engine.py.321: eval: epoch 56 | batch 330/510 [I 2022-10-09 22:31:15] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-09 22:31:15] ...trap/engines/engine.py.326: process: 0.31321 | load: 0.00021 [I 2022-10-09 22:31:18] ...trap/engines/engine.py.321: eval: epoch 56 | batch 340/510 [I 2022-10-09 22:31:18] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-09 22:31:18] ...trap/engines/engine.py.326: process: 0.31575 | load: 0.00021 [I 2022-10-09 22:31:21] ...trap/engines/engine.py.321: eval: epoch 56 | batch 350/510 [I 2022-10-09 22:31:21] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-09 22:31:21] ...trap/engines/engine.py.326: process: 0.31898 | load: 0.00020 [I 2022-10-09 22:31:25] ...trap/engines/engine.py.321: eval: epoch 56 | batch 360/510 [I 2022-10-09 22:31:25] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:47 [I 2022-10-09 22:31:25] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00021 [I 2022-10-09 22:31:28] ...trap/engines/engine.py.321: eval: epoch 56 | batch 370/510 [I 2022-10-09 22:31:28] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:44 [I 2022-10-09 22:31:28] ...trap/engines/engine.py.326: process: 0.31223 | load: 0.00020 [I 2022-10-09 22:31:31] ...trap/engines/engine.py.321: eval: epoch 56 | batch 380/510 [I 2022-10-09 22:31:31] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:41 [I 2022-10-09 22:31:31] ...trap/engines/engine.py.326: process: 0.31912 | load: 0.00020 [I 2022-10-09 22:31:34] ...trap/engines/engine.py.321: eval: epoch 56 | batch 390/510 [I 2022-10-09 22:31:34] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-09 22:31:34] ...trap/engines/engine.py.326: process: 0.32113 | load: 0.00020 [I 2022-10-09 22:31:37] ...trap/engines/engine.py.321: eval: epoch 56 | batch 400/510 [I 2022-10-09 22:31:37] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-09 22:31:37] ...trap/engines/engine.py.326: process: 0.31940 | load: 0.00020 [I 2022-10-09 22:31:40] ...trap/engines/engine.py.321: eval: epoch 56 | batch 410/510 [I 2022-10-09 22:31:40] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:31 [I 2022-10-09 22:31:40] ...trap/engines/engine.py.326: process: 0.31871 | load: 0.00021 [I 2022-10-09 22:31:44] ...trap/engines/engine.py.321: eval: epoch 56 | batch 420/510 [I 2022-10-09 22:31:44] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:28 [I 2022-10-09 22:31:44] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00021 [I 2022-10-09 22:31:47] ...trap/engines/engine.py.321: eval: epoch 56 | batch 430/510 [I 2022-10-09 22:31:47] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-09 22:31:47] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00021 [I 2022-10-09 22:31:50] ...trap/engines/engine.py.321: eval: epoch 56 | batch 440/510 [I 2022-10-09 22:31:50] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-09 22:31:50] ...trap/engines/engine.py.326: process: 0.32210 | load: 0.00019 [I 2022-10-09 22:31:53] ...trap/engines/engine.py.321: eval: epoch 56 | batch 450/510 [I 2022-10-09 22:31:53] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-09 22:31:53] ...trap/engines/engine.py.326: process: 0.32284 | load: 0.00021 [I 2022-10-09 22:31:56] ...trap/engines/engine.py.321: eval: epoch 56 | batch 460/510 [I 2022-10-09 22:31:56] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:15 [I 2022-10-09 22:31:56] ...trap/engines/engine.py.326: process: 0.31824 | load: 0.00020 [I 2022-10-09 22:32:00] ...trap/engines/engine.py.321: eval: epoch 56 | batch 470/510 [I 2022-10-09 22:32:00] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:12 [I 2022-10-09 22:32:00] ...trap/engines/engine.py.326: process: 0.31526 | load: 0.00020 [I 2022-10-09 22:32:03] ...trap/engines/engine.py.321: eval: epoch 56 | batch 480/510 [I 2022-10-09 22:32:03] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-09 22:32:03] ...trap/engines/engine.py.326: process: 0.31132 | load: 0.00019 [I 2022-10-09 22:32:06] ...trap/engines/engine.py.321: eval: epoch 56 | batch 490/510 [I 2022-10-09 22:32:06] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-09 22:32:06] ...trap/engines/engine.py.326: process: 0.31522 | load: 0.00022 [I 2022-10-09 22:32:09] ...trap/engines/engine.py.321: eval: epoch 56 | batch 500/510 [I 2022-10-09 22:32:09] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-09 22:32:09] ...trap/engines/engine.py.326: process: 0.33143 | load: 0.00020 [I 2022-10-09 22:32:12] ...trap/engines/engine.py.321: eval: epoch 56 | batch 510/510 [I 2022-10-09 22:32:12] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-09 22:32:12] ...trap/engines/engine.py.326: process: 0.31230 | load: 0.00013 [S 2022-10-09 22:32:13] ...trap/engines/engine.py.132: eval_epoch.epoch: 56 [I 2022-10-09 22:32:13] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.721300 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905100 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940200 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.711300 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903500 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936700 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015434 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007905 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007679 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009706 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007393 [S 2022-10-09 22:32:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006856 [I 2022-10-09 22:32:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 22:32:47] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 22:32:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 22:32:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 22:32:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 22:32:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 22:32:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 22:32:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 22:32:57] ...trap/engines/engine.py.163: Training model on trainset for epoch 57 [I 2022-10-09 22:32:57] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 22:33:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 0/2304 [I 2022-10-09 22:33:00] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:45 [I 2022-10-09 22:33:00] ...trap/engines/engine.py.229: process: 2.78000 | load: 1.88204 [I 2022-10-09 22:33:00] ...trap/engines/engine.py.230: loss: 0.16280 [I 2022-10-09 22:33:00] ...optimizers/trijoint.py.151: total_norm: 15.607998 [I 2022-10-09 22:33:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 10/2304 [I 2022-10-09 22:33:08] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:16 [I 2022-10-09 22:33:08] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00646 [I 2022-10-09 22:33:08] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-09 22:33:08] ...optimizers/trijoint.py.151: total_norm: 10.107537 [I 2022-10-09 22:33:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 20/2304 [I 2022-10-09 22:33:17] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:01 [I 2022-10-09 22:33:17] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00470 [I 2022-10-09 22:33:17] ...trap/engines/engine.py.230: loss: 0.16771 [I 2022-10-09 22:33:17] ...optimizers/trijoint.py.151: total_norm: 18.045450 [I 2022-10-09 22:33:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 30/2304 [I 2022-10-09 22:33:25] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:23 [I 2022-10-09 22:33:25] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00374 [I 2022-10-09 22:33:25] ...trap/engines/engine.py.230: loss: 0.11046 [I 2022-10-09 22:33:25] ...optimizers/trijoint.py.151: total_norm: 12.742967 [I 2022-10-09 22:33:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 40/2304 [I 2022-10-09 22:33:34] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:04 [I 2022-10-09 22:33:34] ...trap/engines/engine.py.229: process: 0.85872 | load: 0.00531 [I 2022-10-09 22:33:34] ...trap/engines/engine.py.230: loss: 0.11344 [I 2022-10-09 22:33:34] ...optimizers/trijoint.py.151: total_norm: 14.360522 [I 2022-10-09 22:33:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 50/2304 [I 2022-10-09 22:33:42] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:00 [I 2022-10-09 22:33:42] ...trap/engines/engine.py.229: process: 0.85977 | load: 0.00364 [I 2022-10-09 22:33:42] ...trap/engines/engine.py.230: loss: 0.19347 [I 2022-10-09 22:33:42] ...optimizers/trijoint.py.151: total_norm: 23.779766 [I 2022-10-09 22:33:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 60/2304 [I 2022-10-09 22:33:51] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:40 [I 2022-10-09 22:33:51] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00399 [I 2022-10-09 22:33:51] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-09 22:33:51] ...optimizers/trijoint.py.151: total_norm: 12.032183 [I 2022-10-09 22:33:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 70/2304 [I 2022-10-09 22:33:59] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:43 [I 2022-10-09 22:33:59] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00400 [I 2022-10-09 22:33:59] ...trap/engines/engine.py.230: loss: 0.17583 [I 2022-10-09 22:33:59] ...optimizers/trijoint.py.151: total_norm: 19.832264 [I 2022-10-09 22:34:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 80/2304 [I 2022-10-09 22:34:08] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:33 [I 2022-10-09 22:34:08] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00355 [I 2022-10-09 22:34:08] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-09 22:34:08] ...optimizers/trijoint.py.151: total_norm: 16.935013 [I 2022-10-09 22:34:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 90/2304 [I 2022-10-09 22:34:16] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:34 [I 2022-10-09 22:34:16] ...trap/engines/engine.py.229: process: 0.86592 | load: 0.00366 [I 2022-10-09 22:34:16] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-09 22:34:16] ...optimizers/trijoint.py.151: total_norm: 21.015017 [I 2022-10-09 22:34:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 100/2304 [I 2022-10-09 22:34:25] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:23 [I 2022-10-09 22:34:25] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00348 [I 2022-10-09 22:34:25] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-09 22:34:25] ...optimizers/trijoint.py.151: total_norm: 21.304842 [I 2022-10-09 22:34:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 110/2304 [I 2022-10-09 22:34:34] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:17 [I 2022-10-09 22:34:34] ...trap/engines/engine.py.229: process: 0.87312 | load: 0.00351 [I 2022-10-09 22:34:34] ...trap/engines/engine.py.230: loss: 0.16159 [I 2022-10-09 22:34:34] ...optimizers/trijoint.py.151: total_norm: 14.133766 [I 2022-10-09 22:34:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 120/2304 [I 2022-10-09 22:34:42] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:31:01 [I 2022-10-09 22:34:42] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00344 [I 2022-10-09 22:34:42] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-09 22:34:42] ...optimizers/trijoint.py.151: total_norm: 24.648676 [I 2022-10-09 22:34:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 130/2304 [I 2022-10-09 22:34:51] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:57 [I 2022-10-09 22:34:51] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00347 [I 2022-10-09 22:34:51] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-09 22:34:51] ...optimizers/trijoint.py.151: total_norm: 8.935205 [I 2022-10-09 22:34:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 140/2304 [I 2022-10-09 22:34:59] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:41 [I 2022-10-09 22:34:59] ...trap/engines/engine.py.229: process: 0.84887 | load: 0.00327 [I 2022-10-09 22:34:59] ...trap/engines/engine.py.230: loss: 0.17010 [I 2022-10-09 22:34:59] ...optimizers/trijoint.py.151: total_norm: 18.873678 [I 2022-10-09 22:35:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 150/2304 [I 2022-10-09 22:35:08] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:30 [I 2022-10-09 22:35:08] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00311 [I 2022-10-09 22:35:08] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-09 22:35:08] ...optimizers/trijoint.py.151: total_norm: 12.914160 [I 2022-10-09 22:35:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 160/2304 [I 2022-10-09 22:35:16] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:31 [I 2022-10-09 22:35:16] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00327 [I 2022-10-09 22:35:16] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-09 22:35:16] ...optimizers/trijoint.py.151: total_norm: 15.476828 [I 2022-10-09 22:35:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 170/2304 [I 2022-10-09 22:35:25] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:26 [I 2022-10-09 22:35:25] ...trap/engines/engine.py.229: process: 0.86273 | load: 0.00346 [I 2022-10-09 22:35:25] ...trap/engines/engine.py.230: loss: 0.13846 [I 2022-10-09 22:35:25] ...optimizers/trijoint.py.151: total_norm: 9.936810 [I 2022-10-09 22:35:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 180/2304 [I 2022-10-09 22:35:33] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:10 [I 2022-10-09 22:35:33] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00315 [I 2022-10-09 22:35:33] ...trap/engines/engine.py.230: loss: 0.17229 [I 2022-10-09 22:35:33] ...optimizers/trijoint.py.151: total_norm: 22.290636 [I 2022-10-09 22:35:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 190/2304 [I 2022-10-09 22:35:42] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:06 [I 2022-10-09 22:35:42] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00353 [I 2022-10-09 22:35:42] ...trap/engines/engine.py.230: loss: 0.21246 [I 2022-10-09 22:35:42] ...optimizers/trijoint.py.151: total_norm: 27.451792 [I 2022-10-09 22:35:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 200/2304 [I 2022-10-09 22:35:50] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:53 [I 2022-10-09 22:35:50] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00352 [I 2022-10-09 22:35:50] ...trap/engines/engine.py.230: loss: 0.14705 [I 2022-10-09 22:35:50] ...optimizers/trijoint.py.151: total_norm: 16.489742 [I 2022-10-09 22:35:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 210/2304 [I 2022-10-09 22:35:59] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:38 [I 2022-10-09 22:35:59] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00344 [I 2022-10-09 22:35:59] ...trap/engines/engine.py.230: loss: 0.20417 [I 2022-10-09 22:35:59] ...optimizers/trijoint.py.151: total_norm: 30.098789 [I 2022-10-09 22:36:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 220/2304 [I 2022-10-09 22:36:08] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:42 [I 2022-10-09 22:36:08] ...trap/engines/engine.py.229: process: 0.86598 | load: 0.00313 [I 2022-10-09 22:36:08] ...trap/engines/engine.py.230: loss: 0.11600 [I 2022-10-09 22:36:08] ...optimizers/trijoint.py.151: total_norm: 17.691481 [I 2022-10-09 22:36:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 230/2304 [I 2022-10-09 22:36:16] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:31 [I 2022-10-09 22:36:16] ...trap/engines/engine.py.229: process: 0.84747 | load: 0.00324 [I 2022-10-09 22:36:16] ...trap/engines/engine.py.230: loss: 0.10787 [I 2022-10-09 22:36:16] ...optimizers/trijoint.py.151: total_norm: 18.349693 [I 2022-10-09 22:36:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 240/2304 [I 2022-10-09 22:36:25] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:24 [I 2022-10-09 22:36:25] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00341 [I 2022-10-09 22:36:25] ...trap/engines/engine.py.230: loss: 0.17756 [I 2022-10-09 22:36:25] ...optimizers/trijoint.py.151: total_norm: 11.332756 [I 2022-10-09 22:36:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 250/2304 [I 2022-10-09 22:36:33] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:17 [I 2022-10-09 22:36:33] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00311 [I 2022-10-09 22:36:33] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-09 22:36:33] ...optimizers/trijoint.py.151: total_norm: 12.088337 [I 2022-10-09 22:36:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 260/2304 [I 2022-10-09 22:36:42] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:03 [I 2022-10-09 22:36:42] ...trap/engines/engine.py.229: process: 0.85525 | load: 0.00320 [I 2022-10-09 22:36:42] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-09 22:36:42] ...optimizers/trijoint.py.151: total_norm: 17.030626 [I 2022-10-09 22:36:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 270/2304 [I 2022-10-09 22:36:50] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:29:00 [I 2022-10-09 22:36:50] ...trap/engines/engine.py.229: process: 0.86838 | load: 0.00339 [I 2022-10-09 22:36:50] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-09 22:36:50] ...optimizers/trijoint.py.151: total_norm: 9.769814 [I 2022-10-09 22:36:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 280/2304 [I 2022-10-09 22:36:59] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:56 [I 2022-10-09 22:36:59] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00311 [I 2022-10-09 22:36:59] ...trap/engines/engine.py.230: loss: 0.14154 [I 2022-10-09 22:36:59] ...optimizers/trijoint.py.151: total_norm: 17.810680 [I 2022-10-09 22:37:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 290/2304 [I 2022-10-09 22:37:07] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:31 [I 2022-10-09 22:37:07] ...trap/engines/engine.py.229: process: 0.83968 | load: 0.00301 [I 2022-10-09 22:37:07] ...trap/engines/engine.py.230: loss: 0.21343 [I 2022-10-09 22:37:07] ...optimizers/trijoint.py.151: total_norm: 32.614292 [I 2022-10-09 22:37:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 300/2304 [I 2022-10-09 22:37:16] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:12 [I 2022-10-09 22:37:16] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00308 [I 2022-10-09 22:37:16] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-09 22:37:16] ...optimizers/trijoint.py.151: total_norm: 16.826233 [I 2022-10-09 22:37:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 310/2304 [I 2022-10-09 22:37:24] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:21 [I 2022-10-09 22:37:24] ...trap/engines/engine.py.229: process: 0.86768 | load: 0.00307 [I 2022-10-09 22:37:24] ...trap/engines/engine.py.230: loss: 0.24554 [I 2022-10-09 22:37:24] ...optimizers/trijoint.py.151: total_norm: 23.000538 [I 2022-10-09 22:37:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 320/2304 [I 2022-10-09 22:37:33] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:08 [I 2022-10-09 22:37:33] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00315 [I 2022-10-09 22:37:33] ...trap/engines/engine.py.230: loss: 0.14314 [I 2022-10-09 22:37:33] ...optimizers/trijoint.py.151: total_norm: 9.942286 [I 2022-10-09 22:37:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 330/2304 [I 2022-10-09 22:37:42] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:08 [I 2022-10-09 22:37:42] ...trap/engines/engine.py.229: process: 0.85909 | load: 0.00318 [I 2022-10-09 22:37:42] ...trap/engines/engine.py.230: loss: 0.15467 [I 2022-10-09 22:37:42] ...optimizers/trijoint.py.151: total_norm: 11.774875 [I 2022-10-09 22:37:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 340/2304 [I 2022-10-09 22:37:50] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:58 [I 2022-10-09 22:37:50] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00313 [I 2022-10-09 22:37:50] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-09 22:37:50] ...optimizers/trijoint.py.151: total_norm: 21.085699 [I 2022-10-09 22:37:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 350/2304 [I 2022-10-09 22:37:59] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:39 [I 2022-10-09 22:37:59] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00307 [I 2022-10-09 22:37:59] ...trap/engines/engine.py.230: loss: 0.10601 [I 2022-10-09 22:37:59] ...optimizers/trijoint.py.151: total_norm: 11.014057 [I 2022-10-09 22:38:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 360/2304 [I 2022-10-09 22:38:07] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:45 [I 2022-10-09 22:38:07] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00319 [I 2022-10-09 22:38:07] ...trap/engines/engine.py.230: loss: 0.09146 [I 2022-10-09 22:38:07] ...optimizers/trijoint.py.151: total_norm: 6.771845 [I 2022-10-09 22:38:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 370/2304 [I 2022-10-09 22:38:16] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:17 [I 2022-10-09 22:38:16] ...trap/engines/engine.py.229: process: 0.83884 | load: 0.00296 [I 2022-10-09 22:38:16] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-09 22:38:16] ...optimizers/trijoint.py.151: total_norm: 9.275854 [I 2022-10-09 22:38:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 380/2304 [I 2022-10-09 22:38:24] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:12 [I 2022-10-09 22:38:24] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00305 [I 2022-10-09 22:38:24] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-09 22:38:24] ...optimizers/trijoint.py.151: total_norm: 8.441619 [I 2022-10-09 22:38:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 390/2304 [I 2022-10-09 22:38:33] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:09 [I 2022-10-09 22:38:33] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00297 [I 2022-10-09 22:38:33] ...trap/engines/engine.py.230: loss: 0.17443 [I 2022-10-09 22:38:33] ...optimizers/trijoint.py.151: total_norm: 23.837034 [I 2022-10-09 22:38:41] ...trap/engines/engine.py.224: train: epoch 57 | batch 400/2304 [I 2022-10-09 22:38:41] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:44 [I 2022-10-09 22:38:41] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00300 [I 2022-10-09 22:38:41] ...trap/engines/engine.py.230: loss: 0.15179 [I 2022-10-09 22:38:41] ...optimizers/trijoint.py.151: total_norm: 16.998829 [I 2022-10-09 22:38:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 410/2304 [I 2022-10-09 22:38:49] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:32 [I 2022-10-09 22:38:49] ...trap/engines/engine.py.229: process: 0.83541 | load: 0.00293 [I 2022-10-09 22:38:49] ...trap/engines/engine.py.230: loss: 0.17589 [I 2022-10-09 22:38:49] ...optimizers/trijoint.py.151: total_norm: 9.460658 [I 2022-10-09 22:38:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 420/2304 [I 2022-10-09 22:38:58] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:33 [I 2022-10-09 22:38:58] ...trap/engines/engine.py.229: process: 0.84618 | load: 0.00294 [I 2022-10-09 22:38:58] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-09 22:38:58] ...optimizers/trijoint.py.151: total_norm: 15.201715 [I 2022-10-09 22:39:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 430/2304 [I 2022-10-09 22:39:06] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:16 [I 2022-10-09 22:39:06] ...trap/engines/engine.py.229: process: 0.83325 | load: 0.00293 [I 2022-10-09 22:39:06] ...trap/engines/engine.py.230: loss: 0.12416 [I 2022-10-09 22:39:06] ...optimizers/trijoint.py.151: total_norm: 11.491270 [I 2022-10-09 22:39:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 440/2304 [I 2022-10-09 22:39:15] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:15 [I 2022-10-09 22:39:15] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00295 [I 2022-10-09 22:39:15] ...trap/engines/engine.py.230: loss: 0.09290 [I 2022-10-09 22:39:15] ...optimizers/trijoint.py.151: total_norm: 9.430330 [I 2022-10-09 22:39:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 450/2304 [I 2022-10-09 22:39:23] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:07 [I 2022-10-09 22:39:23] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00300 [I 2022-10-09 22:39:23] ...trap/engines/engine.py.230: loss: 0.18308 [I 2022-10-09 22:39:23] ...optimizers/trijoint.py.151: total_norm: 16.599571 [I 2022-10-09 22:39:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 460/2304 [I 2022-10-09 22:39:32] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:01 [I 2022-10-09 22:39:32] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00295 [I 2022-10-09 22:39:32] ...trap/engines/engine.py.230: loss: 0.20337 [I 2022-10-09 22:39:32] ...optimizers/trijoint.py.151: total_norm: 16.657513 [I 2022-10-09 22:39:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 470/2304 [I 2022-10-09 22:39:40] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:00 [I 2022-10-09 22:39:40] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00290 [I 2022-10-09 22:39:40] ...trap/engines/engine.py.230: loss: 0.13442 [I 2022-10-09 22:39:40] ...optimizers/trijoint.py.151: total_norm: 10.240183 [I 2022-10-09 22:39:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 480/2304 [I 2022-10-09 22:39:49] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:25:44 [I 2022-10-09 22:39:49] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00294 [I 2022-10-09 22:39:49] ...trap/engines/engine.py.230: loss: 0.14600 [I 2022-10-09 22:39:49] ...optimizers/trijoint.py.151: total_norm: 21.875774 [I 2022-10-09 22:39:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 490/2304 [I 2022-10-09 22:39:57] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:19 [I 2022-10-09 22:39:57] ...trap/engines/engine.py.229: process: 0.83712 | load: 0.00293 [I 2022-10-09 22:39:57] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-09 22:39:57] ...optimizers/trijoint.py.151: total_norm: 12.801075 [I 2022-10-09 22:40:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 500/2304 [I 2022-10-09 22:40:06] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:22 [I 2022-10-09 22:40:06] ...trap/engines/engine.py.229: process: 0.83927 | load: 0.00292 [I 2022-10-09 22:40:06] ...trap/engines/engine.py.230: loss: 0.14071 [I 2022-10-09 22:40:06] ...optimizers/trijoint.py.151: total_norm: 16.297298 [I 2022-10-09 22:40:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 510/2304 [I 2022-10-09 22:40:14] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:09 [I 2022-10-09 22:40:14] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00302 [I 2022-10-09 22:40:14] ...trap/engines/engine.py.230: loss: 0.15517 [I 2022-10-09 22:40:14] ...optimizers/trijoint.py.151: total_norm: 21.279537 [I 2022-10-09 22:40:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 520/2304 [I 2022-10-09 22:40:23] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:02 [I 2022-10-09 22:40:23] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00296 [I 2022-10-09 22:40:23] ...trap/engines/engine.py.230: loss: 0.11306 [I 2022-10-09 22:40:23] ...optimizers/trijoint.py.151: total_norm: 13.212864 [I 2022-10-09 22:40:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 530/2304 [I 2022-10-09 22:40:31] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:24:50 [I 2022-10-09 22:40:31] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00294 [I 2022-10-09 22:40:31] ...trap/engines/engine.py.230: loss: 0.17981 [I 2022-10-09 22:40:31] ...optimizers/trijoint.py.151: total_norm: 17.018433 [I 2022-10-09 22:40:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 540/2304 [I 2022-10-09 22:40:40] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:47 [I 2022-10-09 22:40:40] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00295 [I 2022-10-09 22:40:40] ...trap/engines/engine.py.230: loss: 0.16563 [I 2022-10-09 22:40:40] ...optimizers/trijoint.py.151: total_norm: 38.212437 [I 2022-10-09 22:40:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 550/2304 [I 2022-10-09 22:40:48] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:35 [I 2022-10-09 22:40:48] ...trap/engines/engine.py.229: process: 0.83214 | load: 0.00298 [I 2022-10-09 22:40:48] ...trap/engines/engine.py.230: loss: 0.15065 [I 2022-10-09 22:40:48] ...optimizers/trijoint.py.151: total_norm: 34.973816 [I 2022-10-09 22:40:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 560/2304 [I 2022-10-09 22:40:56] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:28 [I 2022-10-09 22:40:56] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00300 [I 2022-10-09 22:40:56] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-09 22:40:56] ...optimizers/trijoint.py.151: total_norm: 8.503053 [I 2022-10-09 22:41:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 570/2304 [I 2022-10-09 22:41:05] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:19 [I 2022-10-09 22:41:05] ...trap/engines/engine.py.229: process: 0.82974 | load: 0.00298 [I 2022-10-09 22:41:05] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-09 22:41:05] ...optimizers/trijoint.py.151: total_norm: 17.149168 [I 2022-10-09 22:41:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 580/2304 [I 2022-10-09 22:41:13] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:24 [I 2022-10-09 22:41:13] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00297 [I 2022-10-09 22:41:13] ...trap/engines/engine.py.230: loss: 0.19743 [I 2022-10-09 22:41:13] ...optimizers/trijoint.py.151: total_norm: 15.404827 [I 2022-10-09 22:41:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 590/2304 [I 2022-10-09 22:41:22] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:13 [I 2022-10-09 22:41:22] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00296 [I 2022-10-09 22:41:22] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-09 22:41:22] ...optimizers/trijoint.py.151: total_norm: 13.749190 [I 2022-10-09 22:41:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 600/2304 [I 2022-10-09 22:41:30] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:05 [I 2022-10-09 22:41:30] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00300 [I 2022-10-09 22:41:30] ...trap/engines/engine.py.230: loss: 0.18358 [I 2022-10-09 22:41:30] ...optimizers/trijoint.py.151: total_norm: 43.933975 [I 2022-10-09 22:41:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 610/2304 [I 2022-10-09 22:41:39] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:23:52 [I 2022-10-09 22:41:39] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00296 [I 2022-10-09 22:41:39] ...trap/engines/engine.py.230: loss: 0.16620 [I 2022-10-09 22:41:39] ...optimizers/trijoint.py.151: total_norm: 22.058079 [I 2022-10-09 22:41:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 620/2304 [I 2022-10-09 22:41:47] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:49 [I 2022-10-09 22:41:47] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00308 [I 2022-10-09 22:41:47] ...trap/engines/engine.py.230: loss: 0.18811 [I 2022-10-09 22:41:47] ...optimizers/trijoint.py.151: total_norm: 15.571671 [I 2022-10-09 22:41:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 630/2304 [I 2022-10-09 22:41:56] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:36 [I 2022-10-09 22:41:56] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00299 [I 2022-10-09 22:41:56] ...trap/engines/engine.py.230: loss: 0.11634 [I 2022-10-09 22:41:56] ...optimizers/trijoint.py.151: total_norm: 10.810215 [I 2022-10-09 22:42:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 640/2304 [I 2022-10-09 22:42:04] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:31 [I 2022-10-09 22:42:04] ...trap/engines/engine.py.229: process: 0.85586 | load: 0.00295 [I 2022-10-09 22:42:04] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-09 22:42:04] ...optimizers/trijoint.py.151: total_norm: 6.815009 [I 2022-10-09 22:42:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 650/2304 [I 2022-10-09 22:42:13] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:27 [I 2022-10-09 22:42:13] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00310 [I 2022-10-09 22:42:13] ...trap/engines/engine.py.230: loss: 0.22338 [I 2022-10-09 22:42:13] ...optimizers/trijoint.py.151: total_norm: 33.793644 [I 2022-10-09 22:42:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 660/2304 [I 2022-10-09 22:42:21] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:26 [I 2022-10-09 22:42:21] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00310 [I 2022-10-09 22:42:21] ...trap/engines/engine.py.230: loss: 0.14187 [I 2022-10-09 22:42:21] ...optimizers/trijoint.py.151: total_norm: 13.545089 [I 2022-10-09 22:42:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 670/2304 [I 2022-10-09 22:42:30] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:13 [I 2022-10-09 22:42:30] ...trap/engines/engine.py.229: process: 0.84225 | load: 0.00313 [I 2022-10-09 22:42:30] ...trap/engines/engine.py.230: loss: 0.13540 [I 2022-10-09 22:42:30] ...optimizers/trijoint.py.151: total_norm: 21.683424 [I 2022-10-09 22:42:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 680/2304 [I 2022-10-09 22:42:39] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:06 [I 2022-10-09 22:42:39] ...trap/engines/engine.py.229: process: 0.86489 | load: 0.00317 [I 2022-10-09 22:42:39] ...trap/engines/engine.py.230: loss: 0.15793 [I 2022-10-09 22:42:39] ...optimizers/trijoint.py.151: total_norm: 25.205708 [I 2022-10-09 22:42:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 690/2304 [I 2022-10-09 22:42:47] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:57 [I 2022-10-09 22:42:47] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00308 [I 2022-10-09 22:42:47] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-09 22:42:47] ...optimizers/trijoint.py.151: total_norm: 32.217133 [I 2022-10-09 22:42:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 700/2304 [I 2022-10-09 22:42:56] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:57 [I 2022-10-09 22:42:56] ...trap/engines/engine.py.229: process: 0.86653 | load: 0.00297 [I 2022-10-09 22:42:56] ...trap/engines/engine.py.230: loss: 0.20474 [I 2022-10-09 22:42:56] ...optimizers/trijoint.py.151: total_norm: 18.474205 [I 2022-10-09 22:43:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 710/2304 [I 2022-10-09 22:43:04] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:34 [I 2022-10-09 22:43:04] ...trap/engines/engine.py.229: process: 0.83564 | load: 0.00326 [I 2022-10-09 22:43:04] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-09 22:43:04] ...optimizers/trijoint.py.151: total_norm: 13.455085 [I 2022-10-09 22:43:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 720/2304 [I 2022-10-09 22:43:13] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:36 [I 2022-10-09 22:43:13] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00310 [I 2022-10-09 22:43:13] ...trap/engines/engine.py.230: loss: 0.19222 [I 2022-10-09 22:43:13] ...optimizers/trijoint.py.151: total_norm: 25.928488 [I 2022-10-09 22:43:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 730/2304 [I 2022-10-09 22:43:21] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:40 [I 2022-10-09 22:43:21] ...trap/engines/engine.py.229: process: 0.89481 | load: 0.00312 [I 2022-10-09 22:43:21] ...trap/engines/engine.py.230: loss: 0.12580 [I 2022-10-09 22:43:21] ...optimizers/trijoint.py.151: total_norm: 30.231655 [I 2022-10-09 22:43:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 740/2304 [I 2022-10-09 22:43:30] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:20 [I 2022-10-09 22:43:30] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00317 [I 2022-10-09 22:43:30] ...trap/engines/engine.py.230: loss: 0.20720 [I 2022-10-09 22:43:30] ...optimizers/trijoint.py.151: total_norm: 39.665092 [I 2022-10-09 22:43:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 750/2304 [I 2022-10-09 22:43:39] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:06 [I 2022-10-09 22:43:39] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00313 [I 2022-10-09 22:43:39] ...trap/engines/engine.py.230: loss: 0.18753 [I 2022-10-09 22:43:39] ...optimizers/trijoint.py.151: total_norm: 22.946409 [I 2022-10-09 22:43:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 760/2304 [I 2022-10-09 22:43:47] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:55 [I 2022-10-09 22:43:47] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00314 [I 2022-10-09 22:43:47] ...trap/engines/engine.py.230: loss: 0.18166 [I 2022-10-09 22:43:47] ...optimizers/trijoint.py.151: total_norm: 17.884150 [I 2022-10-09 22:43:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 770/2304 [I 2022-10-09 22:43:56] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:48 [I 2022-10-09 22:43:56] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00311 [I 2022-10-09 22:43:56] ...trap/engines/engine.py.230: loss: 0.12896 [I 2022-10-09 22:43:56] ...optimizers/trijoint.py.151: total_norm: 21.778166 [I 2022-10-09 22:44:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 780/2304 [I 2022-10-09 22:44:04] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:47 [I 2022-10-09 22:44:04] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00325 [I 2022-10-09 22:44:04] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-09 22:44:04] ...optimizers/trijoint.py.151: total_norm: 14.224525 [I 2022-10-09 22:44:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 790/2304 [I 2022-10-09 22:44:13] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:31 [I 2022-10-09 22:44:13] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00308 [I 2022-10-09 22:44:13] ...trap/engines/engine.py.230: loss: 0.14493 [I 2022-10-09 22:44:13] ...optimizers/trijoint.py.151: total_norm: 17.786495 [I 2022-10-09 22:44:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 800/2304 [I 2022-10-09 22:44:21] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:18 [I 2022-10-09 22:44:21] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00312 [I 2022-10-09 22:44:21] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-09 22:44:21] ...optimizers/trijoint.py.151: total_norm: 16.955362 [I 2022-10-09 22:44:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 810/2304 [I 2022-10-09 22:44:30] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:06 [I 2022-10-09 22:44:30] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00314 [I 2022-10-09 22:44:30] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-09 22:44:30] ...optimizers/trijoint.py.151: total_norm: 26.387354 [I 2022-10-09 22:44:38] ...trap/engines/engine.py.224: train: epoch 57 | batch 820/2304 [I 2022-10-09 22:44:38] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:57 [I 2022-10-09 22:44:38] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00312 [I 2022-10-09 22:44:38] ...trap/engines/engine.py.230: loss: 0.16415 [I 2022-10-09 22:44:38] ...optimizers/trijoint.py.151: total_norm: 13.346041 [I 2022-10-09 22:44:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 830/2304 [I 2022-10-09 22:44:47] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:56 [I 2022-10-09 22:44:47] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00313 [I 2022-10-09 22:44:47] ...trap/engines/engine.py.230: loss: 0.12180 [I 2022-10-09 22:44:47] ...optimizers/trijoint.py.151: total_norm: 15.525343 [I 2022-10-09 22:44:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 840/2304 [I 2022-10-09 22:44:55] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:42 [I 2022-10-09 22:44:55] ...trap/engines/engine.py.229: process: 0.84563 | load: 0.00294 [I 2022-10-09 22:44:55] ...trap/engines/engine.py.230: loss: 0.19606 [I 2022-10-09 22:44:55] ...optimizers/trijoint.py.151: total_norm: 12.056903 [I 2022-10-09 22:45:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 850/2304 [I 2022-10-09 22:45:04] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:35 [I 2022-10-09 22:45:04] ...trap/engines/engine.py.229: process: 0.83417 | load: 0.00311 [I 2022-10-09 22:45:04] ...trap/engines/engine.py.230: loss: 0.18877 [I 2022-10-09 22:45:04] ...optimizers/trijoint.py.151: total_norm: 28.257135 [I 2022-10-09 22:45:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 860/2304 [I 2022-10-09 22:45:12] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:33 [I 2022-10-09 22:45:12] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00317 [I 2022-10-09 22:45:12] ...trap/engines/engine.py.230: loss: 0.15177 [I 2022-10-09 22:45:12] ...optimizers/trijoint.py.151: total_norm: 11.397820 [I 2022-10-09 22:45:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 870/2304 [I 2022-10-09 22:45:21] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:27 [I 2022-10-09 22:45:21] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00346 [I 2022-10-09 22:45:21] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-09 22:45:21] ...optimizers/trijoint.py.151: total_norm: 9.108774 [I 2022-10-09 22:45:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 880/2304 [I 2022-10-09 22:45:30] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:14 [I 2022-10-09 22:45:30] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00319 [I 2022-10-09 22:45:30] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-09 22:45:30] ...optimizers/trijoint.py.151: total_norm: 11.195954 [I 2022-10-09 22:45:38] ...trap/engines/engine.py.224: train: epoch 57 | batch 890/2304 [I 2022-10-09 22:45:38] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:04 [I 2022-10-09 22:45:38] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00314 [I 2022-10-09 22:45:38] ...trap/engines/engine.py.230: loss: 0.14364 [I 2022-10-09 22:45:38] ...optimizers/trijoint.py.151: total_norm: 10.586415 [I 2022-10-09 22:45:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 900/2304 [I 2022-10-09 22:45:47] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:53 [I 2022-10-09 22:45:47] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00311 [I 2022-10-09 22:45:47] ...trap/engines/engine.py.230: loss: 0.10874 [I 2022-10-09 22:45:47] ...optimizers/trijoint.py.151: total_norm: 16.977158 [I 2022-10-09 22:45:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 910/2304 [I 2022-10-09 22:45:55] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:49 [I 2022-10-09 22:45:55] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00312 [I 2022-10-09 22:45:55] ...trap/engines/engine.py.230: loss: 0.15462 [I 2022-10-09 22:45:55] ...optimizers/trijoint.py.151: total_norm: 17.097450 [I 2022-10-09 22:46:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 920/2304 [I 2022-10-09 22:46:04] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:41 [I 2022-10-09 22:46:04] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00311 [I 2022-10-09 22:46:04] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-09 22:46:04] ...optimizers/trijoint.py.151: total_norm: 33.526184 [I 2022-10-09 22:46:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 930/2304 [I 2022-10-09 22:46:12] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:36 [I 2022-10-09 22:46:12] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00303 [I 2022-10-09 22:46:12] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-09 22:46:12] ...optimizers/trijoint.py.151: total_norm: 20.472729 [I 2022-10-09 22:46:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 940/2304 [I 2022-10-09 22:46:21] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:17 [I 2022-10-09 22:46:21] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00312 [I 2022-10-09 22:46:21] ...trap/engines/engine.py.230: loss: 0.11927 [I 2022-10-09 22:46:21] ...optimizers/trijoint.py.151: total_norm: 14.242375 [I 2022-10-09 22:46:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 950/2304 [I 2022-10-09 22:46:29] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:13 [I 2022-10-09 22:46:29] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00308 [I 2022-10-09 22:46:29] ...trap/engines/engine.py.230: loss: 0.16593 [I 2022-10-09 22:46:29] ...optimizers/trijoint.py.151: total_norm: 20.772167 [I 2022-10-09 22:46:38] ...trap/engines/engine.py.224: train: epoch 57 | batch 960/2304 [I 2022-10-09 22:46:38] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:58 [I 2022-10-09 22:46:38] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00315 [I 2022-10-09 22:46:38] ...trap/engines/engine.py.230: loss: 0.13005 [I 2022-10-09 22:46:38] ...optimizers/trijoint.py.151: total_norm: 13.468096 [I 2022-10-09 22:46:46] ...trap/engines/engine.py.224: train: epoch 57 | batch 970/2304 [I 2022-10-09 22:46:46] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:51 [I 2022-10-09 22:46:46] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00310 [I 2022-10-09 22:46:46] ...trap/engines/engine.py.230: loss: 0.15171 [I 2022-10-09 22:46:46] ...optimizers/trijoint.py.151: total_norm: 15.469778 [I 2022-10-09 22:46:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 980/2304 [I 2022-10-09 22:46:55] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:38 [I 2022-10-09 22:46:55] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00297 [I 2022-10-09 22:46:55] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-09 22:46:55] ...optimizers/trijoint.py.151: total_norm: 19.186644 [I 2022-10-09 22:47:03] ...trap/engines/engine.py.224: train: epoch 57 | batch 990/2304 [I 2022-10-09 22:47:03] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:32 [I 2022-10-09 22:47:03] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00299 [I 2022-10-09 22:47:03] ...trap/engines/engine.py.230: loss: 0.17287 [I 2022-10-09 22:47:03] ...optimizers/trijoint.py.151: total_norm: 15.411805 [I 2022-10-09 22:47:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 1000/2304 [I 2022-10-09 22:47:12] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:12 [I 2022-10-09 22:47:12] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00297 [I 2022-10-09 22:47:12] ...trap/engines/engine.py.230: loss: 0.19674 [I 2022-10-09 22:47:12] ...optimizers/trijoint.py.151: total_norm: 19.337059 [I 2022-10-09 22:47:20] ...trap/engines/engine.py.224: train: epoch 57 | batch 1010/2304 [I 2022-10-09 22:47:20] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:19 [I 2022-10-09 22:47:20] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00305 [I 2022-10-09 22:47:20] ...trap/engines/engine.py.230: loss: 0.21273 [I 2022-10-09 22:47:20] ...optimizers/trijoint.py.151: total_norm: 28.190325 [I 2022-10-09 22:47:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 1020/2304 [I 2022-10-09 22:47:29] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:05 [I 2022-10-09 22:47:29] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00291 [I 2022-10-09 22:47:29] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-09 22:47:29] ...optimizers/trijoint.py.151: total_norm: 12.451595 [I 2022-10-09 22:47:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 1030/2304 [I 2022-10-09 22:47:37] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:46 [I 2022-10-09 22:47:37] ...trap/engines/engine.py.229: process: 0.82063 | load: 0.00294 [I 2022-10-09 22:47:37] ...trap/engines/engine.py.230: loss: 0.15244 [I 2022-10-09 22:47:37] ...optimizers/trijoint.py.151: total_norm: 16.676014 [I 2022-10-09 22:47:46] ...trap/engines/engine.py.224: train: epoch 57 | batch 1040/2304 [I 2022-10-09 22:47:46] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:45 [I 2022-10-09 22:47:46] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00293 [I 2022-10-09 22:47:46] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-09 22:47:46] ...optimizers/trijoint.py.151: total_norm: 12.942153 [I 2022-10-09 22:47:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 1050/2304 [I 2022-10-09 22:47:54] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:39 [I 2022-10-09 22:47:54] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00291 [I 2022-10-09 22:47:54] ...trap/engines/engine.py.230: loss: 0.15627 [I 2022-10-09 22:47:54] ...optimizers/trijoint.py.151: total_norm: 14.404678 [I 2022-10-09 22:48:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 1060/2304 [I 2022-10-09 22:48:02] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:30 [I 2022-10-09 22:48:02] ...trap/engines/engine.py.229: process: 0.83474 | load: 0.00291 [I 2022-10-09 22:48:02] ...trap/engines/engine.py.230: loss: 0.12965 [I 2022-10-09 22:48:02] ...optimizers/trijoint.py.151: total_norm: 10.921414 [I 2022-10-09 22:48:11] ...trap/engines/engine.py.224: train: epoch 57 | batch 1070/2304 [I 2022-10-09 22:48:11] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:23 [I 2022-10-09 22:48:11] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00291 [I 2022-10-09 22:48:11] ...trap/engines/engine.py.230: loss: 0.12421 [I 2022-10-09 22:48:11] ...optimizers/trijoint.py.151: total_norm: 19.874634 [I 2022-10-09 22:48:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 1080/2304 [I 2022-10-09 22:48:19] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:10 [I 2022-10-09 22:48:19] ...trap/engines/engine.py.229: process: 0.83498 | load: 0.00296 [I 2022-10-09 22:48:19] ...trap/engines/engine.py.230: loss: 0.10975 [I 2022-10-09 22:48:19] ...optimizers/trijoint.py.151: total_norm: 12.702015 [I 2022-10-09 22:48:28] ...trap/engines/engine.py.224: train: epoch 57 | batch 1090/2304 [I 2022-10-09 22:48:28] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:06 [I 2022-10-09 22:48:28] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00297 [I 2022-10-09 22:48:28] ...trap/engines/engine.py.230: loss: 0.13759 [I 2022-10-09 22:48:28] ...optimizers/trijoint.py.151: total_norm: 10.403885 [I 2022-10-09 22:48:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 1100/2304 [I 2022-10-09 22:48:36] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:56 [I 2022-10-09 22:48:36] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00293 [I 2022-10-09 22:48:36] ...trap/engines/engine.py.230: loss: 0.14832 [I 2022-10-09 22:48:36] ...optimizers/trijoint.py.151: total_norm: 16.221500 [I 2022-10-09 22:48:45] ...trap/engines/engine.py.224: train: epoch 57 | batch 1110/2304 [I 2022-10-09 22:48:45] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:50 [I 2022-10-09 22:48:45] ...trap/engines/engine.py.229: process: 0.85212 | load: 0.00292 [I 2022-10-09 22:48:45] ...trap/engines/engine.py.230: loss: 0.14733 [I 2022-10-09 22:48:45] ...optimizers/trijoint.py.151: total_norm: 10.334565 [I 2022-10-09 22:48:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 1120/2304 [I 2022-10-09 22:48:53] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:35 [I 2022-10-09 22:48:53] ...trap/engines/engine.py.229: process: 0.83520 | load: 0.00295 [I 2022-10-09 22:48:53] ...trap/engines/engine.py.230: loss: 0.11713 [I 2022-10-09 22:48:53] ...optimizers/trijoint.py.151: total_norm: 15.539504 [I 2022-10-09 22:49:20] ...trap/engines/engine.py.224: train: epoch 57 | batch 1130/2304 [I 2022-10-09 22:49:20] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 1:01:30 [I 2022-10-09 22:49:20] ...trap/engines/engine.py.229: process: 0.84164 | load: 0.00298 [I 2022-10-09 22:49:20] ...trap/engines/engine.py.230: loss: 0.12357 [I 2022-10-09 22:49:20] ...optimizers/trijoint.py.151: total_norm: 13.996085 [I 2022-10-09 22:49:28] ...trap/engines/engine.py.224: train: epoch 57 | batch 1140/2304 [I 2022-10-09 22:49:28] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:21:07 [I 2022-10-09 22:49:28] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00295 [I 2022-10-09 22:49:28] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-09 22:49:28] ...optimizers/trijoint.py.151: total_norm: 10.199728 [I 2022-10-09 22:49:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 1150/2304 [I 2022-10-09 22:49:37] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:49 [I 2022-10-09 22:49:37] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00294 [I 2022-10-09 22:49:37] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-09 22:49:37] ...optimizers/trijoint.py.151: total_norm: 14.506609 [I 2022-10-09 22:49:45] ...trap/engines/engine.py.224: train: epoch 57 | batch 1160/2304 [I 2022-10-09 22:49:45] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:13 [I 2022-10-09 22:49:45] ...trap/engines/engine.py.229: process: 0.84604 | load: 0.00295 [I 2022-10-09 22:49:45] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-09 22:49:45] ...optimizers/trijoint.py.151: total_norm: 9.878047 [I 2022-10-09 22:49:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 1170/2304 [I 2022-10-09 22:49:53] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:16:00 [I 2022-10-09 22:49:53] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00300 [I 2022-10-09 22:49:53] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-09 22:49:53] ...optimizers/trijoint.py.151: total_norm: 18.965975 [I 2022-10-09 22:50:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 1180/2304 [I 2022-10-09 22:50:02] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:47 [I 2022-10-09 22:50:02] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00299 [I 2022-10-09 22:50:02] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-09 22:50:02] ...optimizers/trijoint.py.151: total_norm: 18.856846 [I 2022-10-09 22:50:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 1190/2304 [I 2022-10-09 22:50:10] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:38 [I 2022-10-09 22:50:10] ...trap/engines/engine.py.229: process: 0.84558 | load: 0.00295 [I 2022-10-09 22:50:10] ...trap/engines/engine.py.230: loss: 0.18672 [I 2022-10-09 22:50:10] ...optimizers/trijoint.py.151: total_norm: 17.994202 [I 2022-10-09 22:50:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 1200/2304 [I 2022-10-09 22:50:19] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:30 [I 2022-10-09 22:50:19] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00295 [I 2022-10-09 22:50:19] ...trap/engines/engine.py.230: loss: 0.16690 [I 2022-10-09 22:50:19] ...optimizers/trijoint.py.151: total_norm: 15.000849 [I 2022-10-09 22:50:27] ...trap/engines/engine.py.224: train: epoch 57 | batch 1210/2304 [I 2022-10-09 22:50:27] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:30 [I 2022-10-09 22:50:27] ...trap/engines/engine.py.229: process: 0.86852 | load: 0.00297 [I 2022-10-09 22:50:27] ...trap/engines/engine.py.230: loss: 0.20019 [I 2022-10-09 22:50:27] ...optimizers/trijoint.py.151: total_norm: 18.162724 [I 2022-10-09 22:50:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 1220/2304 [I 2022-10-09 22:50:36] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:14 [I 2022-10-09 22:50:36] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00297 [I 2022-10-09 22:50:36] ...trap/engines/engine.py.230: loss: 0.17830 [I 2022-10-09 22:50:36] ...optimizers/trijoint.py.151: total_norm: 16.146198 [I 2022-10-09 22:50:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 1230/2304 [I 2022-10-09 22:50:44] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:15:09 [I 2022-10-09 22:50:44] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00322 [I 2022-10-09 22:50:44] ...trap/engines/engine.py.230: loss: 0.20805 [I 2022-10-09 22:50:44] ...optimizers/trijoint.py.151: total_norm: 11.686667 [I 2022-10-09 22:50:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 1240/2304 [I 2022-10-09 22:50:53] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:03 [I 2022-10-09 22:50:53] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00295 [I 2022-10-09 22:50:53] ...trap/engines/engine.py.230: loss: 0.14039 [I 2022-10-09 22:50:53] ...optimizers/trijoint.py.151: total_norm: 19.614359 [I 2022-10-09 22:51:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1250/2304 [I 2022-10-09 22:51:01] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:54 [I 2022-10-09 22:51:01] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00297 [I 2022-10-09 22:51:01] ...trap/engines/engine.py.230: loss: 0.14580 [I 2022-10-09 22:51:01] ...optimizers/trijoint.py.151: total_norm: 14.030389 [I 2022-10-09 22:51:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 1260/2304 [I 2022-10-09 22:51:10] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:43 [I 2022-10-09 22:51:10] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00299 [I 2022-10-09 22:51:10] ...trap/engines/engine.py.230: loss: 0.19602 [I 2022-10-09 22:51:10] ...optimizers/trijoint.py.151: total_norm: 50.575935 [I 2022-10-09 22:51:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 1270/2304 [I 2022-10-09 22:51:18] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:37 [I 2022-10-09 22:51:18] ...trap/engines/engine.py.229: process: 0.83645 | load: 0.00313 [I 2022-10-09 22:51:18] ...trap/engines/engine.py.230: loss: 0.14342 [I 2022-10-09 22:51:18] ...optimizers/trijoint.py.151: total_norm: 6.903158 [I 2022-10-09 22:51:27] ...trap/engines/engine.py.224: train: epoch 57 | batch 1280/2304 [I 2022-10-09 22:51:27] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:30 [I 2022-10-09 22:51:27] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00315 [I 2022-10-09 22:51:27] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-09 22:51:27] ...optimizers/trijoint.py.151: total_norm: 8.510706 [I 2022-10-09 22:51:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 1290/2304 [I 2022-10-09 22:51:35] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:26 [I 2022-10-09 22:51:35] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00311 [I 2022-10-09 22:51:35] ...trap/engines/engine.py.230: loss: 0.22323 [I 2022-10-09 22:51:35] ...optimizers/trijoint.py.151: total_norm: 24.166012 [I 2022-10-09 22:51:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 1300/2304 [I 2022-10-09 22:51:44] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:11 [I 2022-10-09 22:51:44] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00320 [I 2022-10-09 22:51:44] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-09 22:51:44] ...optimizers/trijoint.py.151: total_norm: 24.271832 [I 2022-10-09 22:51:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1310/2304 [I 2022-10-09 22:51:52] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:03 [I 2022-10-09 22:51:52] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00314 [I 2022-10-09 22:51:52] ...trap/engines/engine.py.230: loss: 0.14991 [I 2022-10-09 22:51:52] ...optimizers/trijoint.py.151: total_norm: 10.278470 [I 2022-10-09 22:52:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1320/2304 [I 2022-10-09 22:52:01] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:58 [I 2022-10-09 22:52:01] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00313 [I 2022-10-09 22:52:01] ...trap/engines/engine.py.230: loss: 0.20719 [I 2022-10-09 22:52:01] ...optimizers/trijoint.py.151: total_norm: 24.332262 [I 2022-10-09 22:52:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1330/2304 [I 2022-10-09 22:52:09] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:47 [I 2022-10-09 22:52:09] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00320 [I 2022-10-09 22:52:09] ...trap/engines/engine.py.230: loss: 0.21681 [I 2022-10-09 22:52:09] ...optimizers/trijoint.py.151: total_norm: 18.047188 [I 2022-10-09 22:52:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 1340/2304 [I 2022-10-09 22:52:18] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:43 [I 2022-10-09 22:52:18] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00327 [I 2022-10-09 22:52:18] ...trap/engines/engine.py.230: loss: 0.18136 [I 2022-10-09 22:52:18] ...optimizers/trijoint.py.151: total_norm: 17.990498 [I 2022-10-09 22:52:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1350/2304 [I 2022-10-09 22:52:26] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:32 [I 2022-10-09 22:52:26] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00328 [I 2022-10-09 22:52:26] ...trap/engines/engine.py.230: loss: 0.17464 [I 2022-10-09 22:52:26] ...optimizers/trijoint.py.151: total_norm: 21.438606 [I 2022-10-09 22:52:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 1360/2304 [I 2022-10-09 22:52:35] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:28 [I 2022-10-09 22:52:35] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00311 [I 2022-10-09 22:52:35] ...trap/engines/engine.py.230: loss: 0.11410 [I 2022-10-09 22:52:35] ...optimizers/trijoint.py.151: total_norm: 18.723532 [I 2022-10-09 22:52:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 1370/2304 [I 2022-10-09 22:52:44] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:22 [I 2022-10-09 22:52:44] ...trap/engines/engine.py.229: process: 0.86929 | load: 0.00375 [I 2022-10-09 22:52:44] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-09 22:52:44] ...optimizers/trijoint.py.151: total_norm: 15.221103 [I 2022-10-09 22:52:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1380/2304 [I 2022-10-09 22:52:52] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:13:10 [I 2022-10-09 22:52:52] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00311 [I 2022-10-09 22:52:52] ...trap/engines/engine.py.230: loss: 0.15388 [I 2022-10-09 22:52:52] ...optimizers/trijoint.py.151: total_norm: 14.100750 [I 2022-10-09 22:53:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1390/2304 [I 2022-10-09 22:53:01] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:59 [I 2022-10-09 22:53:01] ...trap/engines/engine.py.229: process: 0.85992 | load: 0.00304 [I 2022-10-09 22:53:01] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-09 22:53:01] ...optimizers/trijoint.py.151: total_norm: 17.585756 [I 2022-10-09 22:53:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1400/2304 [I 2022-10-09 22:53:09] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:52 [I 2022-10-09 22:53:09] ...trap/engines/engine.py.229: process: 0.86320 | load: 0.00311 [I 2022-10-09 22:53:09] ...trap/engines/engine.py.230: loss: 0.19351 [I 2022-10-09 22:53:09] ...optimizers/trijoint.py.151: total_norm: 44.903030 [I 2022-10-09 22:53:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 1410/2304 [I 2022-10-09 22:53:18] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:40 [I 2022-10-09 22:53:18] ...trap/engines/engine.py.229: process: 0.85802 | load: 0.00305 [I 2022-10-09 22:53:18] ...trap/engines/engine.py.230: loss: 0.21070 [I 2022-10-09 22:53:18] ...optimizers/trijoint.py.151: total_norm: 11.234257 [I 2022-10-09 22:53:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1420/2304 [I 2022-10-09 22:53:26] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:36 [I 2022-10-09 22:53:26] ...trap/engines/engine.py.229: process: 0.87509 | load: 0.00311 [I 2022-10-09 22:53:26] ...trap/engines/engine.py.230: loss: 0.17894 [I 2022-10-09 22:53:26] ...optimizers/trijoint.py.151: total_norm: 22.339531 [I 2022-10-09 22:53:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 1430/2304 [I 2022-10-09 22:53:35] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:21 [I 2022-10-09 22:53:35] ...trap/engines/engine.py.229: process: 0.83134 | load: 0.00319 [I 2022-10-09 22:53:35] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-09 22:53:35] ...optimizers/trijoint.py.151: total_norm: 6.163178 [I 2022-10-09 22:53:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1440/2304 [I 2022-10-09 22:53:43] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:15 [I 2022-10-09 22:53:43] ...trap/engines/engine.py.229: process: 0.83886 | load: 0.00312 [I 2022-10-09 22:53:43] ...trap/engines/engine.py.230: loss: 0.18169 [I 2022-10-09 22:53:43] ...optimizers/trijoint.py.151: total_norm: 23.731281 [I 2022-10-09 22:53:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1450/2304 [I 2022-10-09 22:53:52] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:07 [I 2022-10-09 22:53:52] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00319 [I 2022-10-09 22:53:52] ...trap/engines/engine.py.230: loss: 0.18548 [I 2022-10-09 22:53:52] ...optimizers/trijoint.py.151: total_norm: 18.851357 [I 2022-10-09 22:54:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1460/2304 [I 2022-10-09 22:54:01] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:03 [I 2022-10-09 22:54:01] ...trap/engines/engine.py.229: process: 0.86159 | load: 0.00315 [I 2022-10-09 22:54:01] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-09 22:54:01] ...optimizers/trijoint.py.151: total_norm: 19.440016 [I 2022-10-09 22:54:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1470/2304 [I 2022-10-09 22:54:09] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:50 [I 2022-10-09 22:54:09] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00313 [I 2022-10-09 22:54:09] ...trap/engines/engine.py.230: loss: 0.26650 [I 2022-10-09 22:54:09] ...optimizers/trijoint.py.151: total_norm: 30.474251 [I 2022-10-09 22:54:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 1480/2304 [I 2022-10-09 22:54:18] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:38 [I 2022-10-09 22:54:18] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00315 [I 2022-10-09 22:54:18] ...trap/engines/engine.py.230: loss: 0.12816 [I 2022-10-09 22:54:18] ...optimizers/trijoint.py.151: total_norm: 14.286481 [I 2022-10-09 22:54:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1490/2304 [I 2022-10-09 22:54:26] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:37 [I 2022-10-09 22:54:26] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00310 [I 2022-10-09 22:54:26] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-09 22:54:26] ...optimizers/trijoint.py.151: total_norm: 16.699791 [I 2022-10-09 22:54:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 1500/2304 [I 2022-10-09 22:54:35] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:24 [I 2022-10-09 22:54:35] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00313 [I 2022-10-09 22:54:35] ...trap/engines/engine.py.230: loss: 0.15410 [I 2022-10-09 22:54:35] ...optimizers/trijoint.py.151: total_norm: 28.550577 [I 2022-10-09 22:54:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1510/2304 [I 2022-10-09 22:54:43] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:15 [I 2022-10-09 22:54:43] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00296 [I 2022-10-09 22:54:43] ...trap/engines/engine.py.230: loss: 0.19920 [I 2022-10-09 22:54:43] ...optimizers/trijoint.py.151: total_norm: 15.405861 [I 2022-10-09 22:54:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1520/2304 [I 2022-10-09 22:54:52] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:06 [I 2022-10-09 22:54:52] ...trap/engines/engine.py.229: process: 0.83068 | load: 0.00309 [I 2022-10-09 22:54:52] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-09 22:54:52] ...optimizers/trijoint.py.151: total_norm: 16.543495 [I 2022-10-09 22:55:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 1530/2304 [I 2022-10-09 22:55:00] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:11:01 [I 2022-10-09 22:55:00] ...trap/engines/engine.py.229: process: 0.85434 | load: 0.00314 [I 2022-10-09 22:55:00] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-09 22:55:00] ...optimizers/trijoint.py.151: total_norm: 13.529409 [I 2022-10-09 22:55:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1540/2304 [I 2022-10-09 22:55:09] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:50 [I 2022-10-09 22:55:09] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00314 [I 2022-10-09 22:55:09] ...trap/engines/engine.py.230: loss: 0.09419 [I 2022-10-09 22:55:09] ...optimizers/trijoint.py.151: total_norm: 10.100356 [I 2022-10-09 22:55:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 1550/2304 [I 2022-10-09 22:55:17] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:42 [I 2022-10-09 22:55:17] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00308 [I 2022-10-09 22:55:17] ...trap/engines/engine.py.230: loss: 0.12453 [I 2022-10-09 22:55:17] ...optimizers/trijoint.py.151: total_norm: 15.340585 [I 2022-10-09 22:55:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1560/2304 [I 2022-10-09 22:55:26] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:39 [I 2022-10-09 22:55:26] ...trap/engines/engine.py.229: process: 0.86031 | load: 0.00324 [I 2022-10-09 22:55:26] ...trap/engines/engine.py.230: loss: 0.16915 [I 2022-10-09 22:55:26] ...optimizers/trijoint.py.151: total_norm: 57.145355 [I 2022-10-09 22:55:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 1570/2304 [I 2022-10-09 22:55:35] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:23 [I 2022-10-09 22:55:35] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00311 [I 2022-10-09 22:55:35] ...trap/engines/engine.py.230: loss: 0.14120 [I 2022-10-09 22:55:35] ...optimizers/trijoint.py.151: total_norm: 28.185410 [I 2022-10-09 22:55:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1580/2304 [I 2022-10-09 22:55:43] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:17 [I 2022-10-09 22:55:43] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00312 [I 2022-10-09 22:55:43] ...trap/engines/engine.py.230: loss: 0.18956 [I 2022-10-09 22:55:43] ...optimizers/trijoint.py.151: total_norm: 15.179904 [I 2022-10-09 22:55:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1590/2304 [I 2022-10-09 22:55:52] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:04 [I 2022-10-09 22:55:52] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00303 [I 2022-10-09 22:55:52] ...trap/engines/engine.py.230: loss: 0.12068 [I 2022-10-09 22:55:52] ...optimizers/trijoint.py.151: total_norm: 16.145363 [I 2022-10-09 22:56:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 1600/2304 [I 2022-10-09 22:56:00] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:10:01 [I 2022-10-09 22:56:00] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00314 [I 2022-10-09 22:56:00] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-09 22:56:00] ...optimizers/trijoint.py.151: total_norm: 14.921694 [I 2022-10-09 22:56:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1610/2304 [I 2022-10-09 22:56:09] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:51 [I 2022-10-09 22:56:09] ...trap/engines/engine.py.229: process: 0.86480 | load: 0.00300 [I 2022-10-09 22:56:09] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-09 22:56:09] ...optimizers/trijoint.py.151: total_norm: 15.308802 [I 2022-10-09 22:56:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 1620/2304 [I 2022-10-09 22:56:17] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:40 [I 2022-10-09 22:56:17] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00297 [I 2022-10-09 22:56:17] ...trap/engines/engine.py.230: loss: 0.10770 [I 2022-10-09 22:56:17] ...optimizers/trijoint.py.151: total_norm: 10.337320 [I 2022-10-09 22:56:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1630/2304 [I 2022-10-09 22:56:26] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:34 [I 2022-10-09 22:56:26] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00302 [I 2022-10-09 22:56:26] ...trap/engines/engine.py.230: loss: 0.15760 [I 2022-10-09 22:56:26] ...optimizers/trijoint.py.151: total_norm: 21.820641 [I 2022-10-09 22:56:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 1640/2304 [I 2022-10-09 22:56:34] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:21 [I 2022-10-09 22:56:34] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00293 [I 2022-10-09 22:56:34] ...trap/engines/engine.py.230: loss: 0.13030 [I 2022-10-09 22:56:34] ...optimizers/trijoint.py.151: total_norm: 15.536064 [I 2022-10-09 22:56:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1650/2304 [I 2022-10-09 22:56:43] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:12 [I 2022-10-09 22:56:43] ...trap/engines/engine.py.229: process: 0.85080 | load: 0.00294 [I 2022-10-09 22:56:43] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-09 22:56:43] ...optimizers/trijoint.py.151: total_norm: 15.954929 [I 2022-10-09 22:56:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 1660/2304 [I 2022-10-09 22:56:51] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:03 [I 2022-10-09 22:56:51] ...trap/engines/engine.py.229: process: 0.85148 | load: 0.00293 [I 2022-10-09 22:56:51] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-09 22:56:51] ...optimizers/trijoint.py.151: total_norm: 17.192417 [I 2022-10-09 22:57:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 1670/2304 [I 2022-10-09 22:57:00] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:54 [I 2022-10-09 22:57:00] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00295 [I 2022-10-09 22:57:00] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-09 22:57:00] ...optimizers/trijoint.py.151: total_norm: 18.636967 [I 2022-10-09 22:57:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 1680/2304 [I 2022-10-09 22:57:08] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:47 [I 2022-10-09 22:57:08] ...trap/engines/engine.py.229: process: 0.85939 | load: 0.00300 [I 2022-10-09 22:57:08] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-09 22:57:08] ...optimizers/trijoint.py.151: total_norm: 14.898710 [I 2022-10-09 22:57:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 1690/2304 [I 2022-10-09 22:57:16] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:40 [I 2022-10-09 22:57:16] ...trap/engines/engine.py.229: process: 0.84991 | load: 0.00296 [I 2022-10-09 22:57:16] ...trap/engines/engine.py.230: loss: 0.09518 [I 2022-10-09 22:57:16] ...optimizers/trijoint.py.151: total_norm: 14.525693 [I 2022-10-09 22:57:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 1700/2304 [I 2022-10-09 22:57:25] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:29 [I 2022-10-09 22:57:25] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00296 [I 2022-10-09 22:57:25] ...trap/engines/engine.py.230: loss: 0.17450 [I 2022-10-09 22:57:25] ...optimizers/trijoint.py.151: total_norm: 15.275113 [I 2022-10-09 22:57:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 1710/2304 [I 2022-10-09 22:57:33] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:23 [I 2022-10-09 22:57:33] ...trap/engines/engine.py.229: process: 0.84074 | load: 0.00295 [I 2022-10-09 22:57:33] ...trap/engines/engine.py.230: loss: 0.20320 [I 2022-10-09 22:57:33] ...optimizers/trijoint.py.151: total_norm: 27.004089 [I 2022-10-09 22:57:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 1720/2304 [I 2022-10-09 22:57:42] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:11 [I 2022-10-09 22:57:42] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00294 [I 2022-10-09 22:57:42] ...trap/engines/engine.py.230: loss: 0.15938 [I 2022-10-09 22:57:42] ...optimizers/trijoint.py.151: total_norm: 21.392790 [I 2022-10-09 22:57:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 1730/2304 [I 2022-10-09 22:57:50] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:03 [I 2022-10-09 22:57:50] ...trap/engines/engine.py.229: process: 0.83195 | load: 0.00304 [I 2022-10-09 22:57:50] ...trap/engines/engine.py.230: loss: 0.17897 [I 2022-10-09 22:57:50] ...optimizers/trijoint.py.151: total_norm: 10.279866 [I 2022-10-09 22:57:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 1740/2304 [I 2022-10-09 22:57:59] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:56 [I 2022-10-09 22:57:59] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00293 [I 2022-10-09 22:57:59] ...trap/engines/engine.py.230: loss: 0.13211 [I 2022-10-09 22:57:59] ...optimizers/trijoint.py.151: total_norm: 13.011196 [I 2022-10-09 22:58:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 1750/2304 [I 2022-10-09 22:58:07] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:48 [I 2022-10-09 22:58:07] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00294 [I 2022-10-09 22:58:07] ...trap/engines/engine.py.230: loss: 0.12204 [I 2022-10-09 22:58:07] ...optimizers/trijoint.py.151: total_norm: 10.068672 [I 2022-10-09 22:58:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 1760/2304 [I 2022-10-09 22:58:16] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:40 [I 2022-10-09 22:58:16] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00295 [I 2022-10-09 22:58:16] ...trap/engines/engine.py.230: loss: 0.14306 [I 2022-10-09 22:58:16] ...optimizers/trijoint.py.151: total_norm: 29.889725 [I 2022-10-09 22:58:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 1770/2304 [I 2022-10-09 22:58:24] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:32 [I 2022-10-09 22:58:24] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00294 [I 2022-10-09 22:58:24] ...trap/engines/engine.py.230: loss: 0.09798 [I 2022-10-09 22:58:24] ...optimizers/trijoint.py.151: total_norm: 10.247984 [I 2022-10-09 22:58:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 1780/2304 [I 2022-10-09 22:58:33] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:22 [I 2022-10-09 22:58:33] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00294 [I 2022-10-09 22:58:33] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-09 22:58:33] ...optimizers/trijoint.py.151: total_norm: 8.769497 [I 2022-10-09 22:58:41] ...trap/engines/engine.py.224: train: epoch 57 | batch 1790/2304 [I 2022-10-09 22:58:41] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:15 [I 2022-10-09 22:58:41] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00298 [I 2022-10-09 22:58:41] ...trap/engines/engine.py.230: loss: 0.24269 [I 2022-10-09 22:58:41] ...optimizers/trijoint.py.151: total_norm: 20.426292 [I 2022-10-09 22:58:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 1800/2304 [I 2022-10-09 22:58:50] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:04 [I 2022-10-09 22:58:50] ...trap/engines/engine.py.229: process: 0.83737 | load: 0.00296 [I 2022-10-09 22:58:50] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-09 22:58:50] ...optimizers/trijoint.py.151: total_norm: 9.255794 [I 2022-10-09 22:58:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 1810/2304 [I 2022-10-09 22:58:58] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:57 [I 2022-10-09 22:58:58] ...trap/engines/engine.py.229: process: 0.84101 | load: 0.00297 [I 2022-10-09 22:58:58] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-09 22:58:58] ...optimizers/trijoint.py.151: total_norm: 10.746458 [I 2022-10-09 22:59:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 1820/2304 [I 2022-10-09 22:59:07] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:49 [I 2022-10-09 22:59:07] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00306 [I 2022-10-09 22:59:07] ...trap/engines/engine.py.230: loss: 0.17684 [I 2022-10-09 22:59:07] ...optimizers/trijoint.py.151: total_norm: 14.483003 [I 2022-10-09 22:59:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 1830/2304 [I 2022-10-09 22:59:15] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:40 [I 2022-10-09 22:59:15] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00300 [I 2022-10-09 22:59:15] ...trap/engines/engine.py.230: loss: 0.18919 [I 2022-10-09 22:59:15] ...optimizers/trijoint.py.151: total_norm: 17.089882 [I 2022-10-09 22:59:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 1840/2304 [I 2022-10-09 22:59:23] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:32 [I 2022-10-09 22:59:23] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00299 [I 2022-10-09 22:59:23] ...trap/engines/engine.py.230: loss: 0.16378 [I 2022-10-09 22:59:23] ...optimizers/trijoint.py.151: total_norm: 25.970045 [I 2022-10-09 22:59:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 1850/2304 [I 2022-10-09 22:59:32] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:23 [I 2022-10-09 22:59:32] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00298 [I 2022-10-09 22:59:32] ...trap/engines/engine.py.230: loss: 0.19862 [I 2022-10-09 22:59:32] ...optimizers/trijoint.py.151: total_norm: 22.456942 [I 2022-10-09 22:59:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 1860/2304 [I 2022-10-09 22:59:40] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:15 [I 2022-10-09 22:59:40] ...trap/engines/engine.py.229: process: 0.86290 | load: 0.00296 [I 2022-10-09 22:59:40] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-09 22:59:40] ...optimizers/trijoint.py.151: total_norm: 26.280930 [I 2022-10-09 22:59:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 1870/2304 [I 2022-10-09 22:59:49] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:09 [I 2022-10-09 22:59:49] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00311 [I 2022-10-09 22:59:49] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-09 22:59:49] ...optimizers/trijoint.py.151: total_norm: 12.535342 [I 2022-10-09 22:59:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 1880/2304 [I 2022-10-09 22:59:57] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:00 [I 2022-10-09 22:59:57] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00315 [I 2022-10-09 22:59:57] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-09 22:59:57] ...optimizers/trijoint.py.151: total_norm: 10.703747 [I 2022-10-09 23:00:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 1890/2304 [I 2022-10-09 23:00:06] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:51 [I 2022-10-09 23:00:06] ...trap/engines/engine.py.229: process: 0.83754 | load: 0.00311 [I 2022-10-09 23:00:06] ...trap/engines/engine.py.230: loss: 0.15623 [I 2022-10-09 23:00:06] ...optimizers/trijoint.py.151: total_norm: 20.056835 [I 2022-10-09 23:00:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 1900/2304 [I 2022-10-09 23:00:14] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:44 [I 2022-10-09 23:00:14] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00312 [I 2022-10-09 23:00:14] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-09 23:00:14] ...optimizers/trijoint.py.151: total_norm: 46.296028 [I 2022-10-09 23:00:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 1910/2304 [I 2022-10-09 23:00:23] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:37 [I 2022-10-09 23:00:23] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00310 [I 2022-10-09 23:00:23] ...trap/engines/engine.py.230: loss: 0.16512 [I 2022-10-09 23:00:23] ...optimizers/trijoint.py.151: total_norm: 41.104305 [I 2022-10-09 23:00:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 1920/2304 [I 2022-10-09 23:00:32] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:28 [I 2022-10-09 23:00:32] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00315 [I 2022-10-09 23:00:32] ...trap/engines/engine.py.230: loss: 0.15921 [I 2022-10-09 23:00:32] ...optimizers/trijoint.py.151: total_norm: 15.564271 [I 2022-10-09 23:00:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 1930/2304 [I 2022-10-09 23:00:40] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:20 [I 2022-10-09 23:00:40] ...trap/engines/engine.py.229: process: 0.87119 | load: 0.00314 [I 2022-10-09 23:00:40] ...trap/engines/engine.py.230: loss: 0.21024 [I 2022-10-09 23:00:40] ...optimizers/trijoint.py.151: total_norm: 16.644915 [I 2022-10-09 23:00:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 1940/2304 [I 2022-10-09 23:00:49] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:10 [I 2022-10-09 23:00:49] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00316 [I 2022-10-09 23:00:49] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-09 23:00:49] ...optimizers/trijoint.py.151: total_norm: 39.717926 [I 2022-10-09 23:00:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 1950/2304 [I 2022-10-09 23:00:57] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:03 [I 2022-10-09 23:00:57] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00311 [I 2022-10-09 23:00:57] ...trap/engines/engine.py.230: loss: 0.12355 [I 2022-10-09 23:00:57] ...optimizers/trijoint.py.151: total_norm: 12.310843 [I 2022-10-09 23:01:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 1960/2304 [I 2022-10-09 23:01:06] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:52 [I 2022-10-09 23:01:06] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00320 [I 2022-10-09 23:01:06] ...trap/engines/engine.py.230: loss: 0.16589 [I 2022-10-09 23:01:06] ...optimizers/trijoint.py.151: total_norm: 23.866661 [I 2022-10-09 23:01:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 1970/2304 [I 2022-10-09 23:01:15] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:46 [I 2022-10-09 23:01:15] ...trap/engines/engine.py.229: process: 0.86208 | load: 0.00313 [I 2022-10-09 23:01:15] ...trap/engines/engine.py.230: loss: 0.24473 [I 2022-10-09 23:01:15] ...optimizers/trijoint.py.151: total_norm: 27.871555 [I 2022-10-09 23:01:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 1980/2304 [I 2022-10-09 23:01:23] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:36 [I 2022-10-09 23:01:23] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00311 [I 2022-10-09 23:01:23] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-09 23:01:23] ...optimizers/trijoint.py.151: total_norm: 10.980890 [I 2022-10-09 23:01:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 1990/2304 [I 2022-10-09 23:01:32] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:28 [I 2022-10-09 23:01:32] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00314 [I 2022-10-09 23:01:32] ...trap/engines/engine.py.230: loss: 0.14055 [I 2022-10-09 23:01:32] ...optimizers/trijoint.py.151: total_norm: 27.405062 [I 2022-10-09 23:01:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 2000/2304 [I 2022-10-09 23:01:40] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:18 [I 2022-10-09 23:01:40] ...trap/engines/engine.py.229: process: 0.85654 | load: 0.00314 [I 2022-10-09 23:01:40] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-09 23:01:40] ...optimizers/trijoint.py.151: total_norm: 18.441309 [I 2022-10-09 23:01:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 2010/2304 [I 2022-10-09 23:01:49] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:10 [I 2022-10-09 23:01:49] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00313 [I 2022-10-09 23:01:49] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-09 23:01:49] ...optimizers/trijoint.py.151: total_norm: 10.906733 [I 2022-10-09 23:01:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 2020/2304 [I 2022-10-09 23:01:57] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:02 [I 2022-10-09 23:01:57] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00312 [I 2022-10-09 23:01:57] ...trap/engines/engine.py.230: loss: 0.24433 [I 2022-10-09 23:01:57] ...optimizers/trijoint.py.151: total_norm: 19.740475 [I 2022-10-09 23:02:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 2030/2304 [I 2022-10-09 23:02:06] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:55 [I 2022-10-09 23:02:06] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00311 [I 2022-10-09 23:02:06] ...trap/engines/engine.py.230: loss: 0.23961 [I 2022-10-09 23:02:06] ...optimizers/trijoint.py.151: total_norm: 32.730015 [I 2022-10-09 23:02:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 2040/2304 [I 2022-10-09 23:02:14] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:45 [I 2022-10-09 23:02:14] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00314 [I 2022-10-09 23:02:14] ...trap/engines/engine.py.230: loss: 0.11172 [I 2022-10-09 23:02:14] ...optimizers/trijoint.py.151: total_norm: 10.903555 [I 2022-10-09 23:02:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 2050/2304 [I 2022-10-09 23:02:23] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:36 [I 2022-10-09 23:02:23] ...trap/engines/engine.py.229: process: 0.86381 | load: 0.00317 [I 2022-10-09 23:02:23] ...trap/engines/engine.py.230: loss: 0.12168 [I 2022-10-09 23:02:23] ...optimizers/trijoint.py.151: total_norm: 23.850014 [I 2022-10-09 23:02:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 2060/2304 [I 2022-10-09 23:02:31] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:28 [I 2022-10-09 23:02:31] ...trap/engines/engine.py.229: process: 0.85195 | load: 0.00313 [I 2022-10-09 23:02:31] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-09 23:02:31] ...optimizers/trijoint.py.151: total_norm: 9.810243 [I 2022-10-09 23:02:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 2070/2304 [I 2022-10-09 23:02:40] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:20 [I 2022-10-09 23:02:40] ...trap/engines/engine.py.229: process: 0.83914 | load: 0.00313 [I 2022-10-09 23:02:40] ...trap/engines/engine.py.230: loss: 0.18307 [I 2022-10-09 23:02:40] ...optimizers/trijoint.py.151: total_norm: 20.144947 [I 2022-10-09 23:02:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 2080/2304 [I 2022-10-09 23:02:49] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:10 [I 2022-10-09 23:02:49] ...trap/engines/engine.py.229: process: 0.85449 | load: 0.00317 [I 2022-10-09 23:02:49] ...trap/engines/engine.py.230: loss: 0.12562 [I 2022-10-09 23:02:49] ...optimizers/trijoint.py.151: total_norm: 20.211058 [I 2022-10-09 23:02:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 2090/2304 [I 2022-10-09 23:02:57] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:01 [I 2022-10-09 23:02:57] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00317 [I 2022-10-09 23:02:57] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-09 23:02:57] ...optimizers/trijoint.py.151: total_norm: 13.000711 [I 2022-10-09 23:03:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 2100/2304 [I 2022-10-09 23:03:06] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:02:54 [I 2022-10-09 23:03:06] ...trap/engines/engine.py.229: process: 0.86225 | load: 0.00312 [I 2022-10-09 23:03:06] ...trap/engines/engine.py.230: loss: 0.20789 [I 2022-10-09 23:03:06] ...optimizers/trijoint.py.151: total_norm: 32.358521 [I 2022-10-09 23:03:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 2110/2304 [I 2022-10-09 23:03:14] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:46 [I 2022-10-09 23:03:14] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00316 [I 2022-10-09 23:03:14] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-09 23:03:14] ...optimizers/trijoint.py.151: total_norm: 15.279114 [I 2022-10-09 23:03:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 2120/2304 [I 2022-10-09 23:03:23] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:37 [I 2022-10-09 23:03:23] ...trap/engines/engine.py.229: process: 0.85875 | load: 0.00314 [I 2022-10-09 23:03:23] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-09 23:03:23] ...optimizers/trijoint.py.151: total_norm: 18.267862 [I 2022-10-09 23:03:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 2130/2304 [I 2022-10-09 23:03:31] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:27 [I 2022-10-09 23:03:31] ...trap/engines/engine.py.229: process: 0.83711 | load: 0.00312 [I 2022-10-09 23:03:31] ...trap/engines/engine.py.230: loss: 0.20966 [I 2022-10-09 23:03:31] ...optimizers/trijoint.py.151: total_norm: 14.713394 [I 2022-10-09 23:03:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 2140/2304 [I 2022-10-09 23:03:40] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:18 [I 2022-10-09 23:03:40] ...trap/engines/engine.py.229: process: 0.84346 | load: 0.00314 [I 2022-10-09 23:03:40] ...trap/engines/engine.py.230: loss: 0.20128 [I 2022-10-09 23:03:40] ...optimizers/trijoint.py.151: total_norm: 15.734328 [I 2022-10-09 23:03:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 2150/2304 [I 2022-10-09 23:03:48] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:11 [I 2022-10-09 23:03:48] ...trap/engines/engine.py.229: process: 0.85117 | load: 0.00299 [I 2022-10-09 23:03:48] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-09 23:03:48] ...optimizers/trijoint.py.151: total_norm: 13.829643 [I 2022-10-09 23:03:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 2160/2304 [I 2022-10-09 23:03:57] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:02 [I 2022-10-09 23:03:57] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00316 [I 2022-10-09 23:03:57] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-09 23:03:57] ...optimizers/trijoint.py.151: total_norm: 25.497772 [I 2022-10-09 23:04:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 2170/2304 [I 2022-10-09 23:04:05] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:53 [I 2022-10-09 23:04:05] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00301 [I 2022-10-09 23:04:05] ...trap/engines/engine.py.230: loss: 0.20807 [I 2022-10-09 23:04:05] ...optimizers/trijoint.py.151: total_norm: 10.192192 [I 2022-10-09 23:04:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 2180/2304 [I 2022-10-09 23:04:14] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:44 [I 2022-10-09 23:04:14] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00307 [I 2022-10-09 23:04:14] ...trap/engines/engine.py.230: loss: 0.09736 [I 2022-10-09 23:04:14] ...optimizers/trijoint.py.151: total_norm: 9.617237 [I 2022-10-09 23:04:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 2190/2304 [I 2022-10-09 23:04:22] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:37 [I 2022-10-09 23:04:22] ...trap/engines/engine.py.229: process: 0.84594 | load: 0.00312 [I 2022-10-09 23:04:22] ...trap/engines/engine.py.230: loss: 0.19114 [I 2022-10-09 23:04:22] ...optimizers/trijoint.py.151: total_norm: 27.075840 [I 2022-10-09 23:04:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 2200/2304 [I 2022-10-09 23:04:31] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:28 [I 2022-10-09 23:04:31] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00308 [I 2022-10-09 23:04:31] ...trap/engines/engine.py.230: loss: 0.22848 [I 2022-10-09 23:04:31] ...optimizers/trijoint.py.151: total_norm: 15.815948 [I 2022-10-09 23:04:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 2210/2304 [I 2022-10-09 23:04:39] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:19 [I 2022-10-09 23:04:39] ...trap/engines/engine.py.229: process: 0.83314 | load: 0.00301 [I 2022-10-09 23:04:39] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-09 23:04:39] ...optimizers/trijoint.py.151: total_norm: 11.749989 [I 2022-10-09 23:04:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 2220/2304 [I 2022-10-09 23:04:48] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:10 [I 2022-10-09 23:04:48] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00297 [I 2022-10-09 23:04:48] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-09 23:04:48] ...optimizers/trijoint.py.151: total_norm: 8.684766 [I 2022-10-09 23:04:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 2230/2304 [I 2022-10-09 23:04:56] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:02 [I 2022-10-09 23:04:56] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00297 [I 2022-10-09 23:04:56] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-09 23:04:56] ...optimizers/trijoint.py.151: total_norm: 9.036870 [I 2022-10-09 23:05:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 2240/2304 [I 2022-10-09 23:05:05] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:53 [I 2022-10-09 23:05:05] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00293 [I 2022-10-09 23:05:05] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-09 23:05:05] ...optimizers/trijoint.py.151: total_norm: 6.774809 [I 2022-10-09 23:05:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 2250/2304 [I 2022-10-09 23:05:13] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:45 [I 2022-10-09 23:05:13] ...trap/engines/engine.py.229: process: 0.83722 | load: 0.00293 [I 2022-10-09 23:05:13] ...trap/engines/engine.py.230: loss: 0.20448 [I 2022-10-09 23:05:13] ...optimizers/trijoint.py.151: total_norm: 10.005015 [I 2022-10-09 23:05:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 2260/2304 [I 2022-10-09 23:05:22] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:36 [I 2022-10-09 23:05:22] ...trap/engines/engine.py.229: process: 0.83035 | load: 0.00300 [I 2022-10-09 23:05:22] ...trap/engines/engine.py.230: loss: 0.15469 [I 2022-10-09 23:05:22] ...optimizers/trijoint.py.151: total_norm: 16.675652 [I 2022-10-09 23:05:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 2270/2304 [I 2022-10-09 23:05:30] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:28 [I 2022-10-09 23:05:30] ...trap/engines/engine.py.229: process: 0.83988 | load: 0.00294 [I 2022-10-09 23:05:30] ...trap/engines/engine.py.230: loss: 0.15665 [I 2022-10-09 23:05:30] ...optimizers/trijoint.py.151: total_norm: 18.666431 [I 2022-10-09 23:05:38] ...trap/engines/engine.py.224: train: epoch 57 | batch 2280/2304 [I 2022-10-09 23:05:38] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:20 [I 2022-10-09 23:05:38] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00294 [I 2022-10-09 23:05:38] ...trap/engines/engine.py.230: loss: 0.19800 [I 2022-10-09 23:05:38] ...optimizers/trijoint.py.151: total_norm: 32.240780 [I 2022-10-09 23:05:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 2290/2304 [I 2022-10-09 23:05:47] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:11 [I 2022-10-09 23:05:47] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00293 [I 2022-10-09 23:05:47] ...trap/engines/engine.py.230: loss: 0.15658 [I 2022-10-09 23:05:47] ...optimizers/trijoint.py.151: total_norm: 27.050310 [I 2022-10-09 23:05:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 2300/2304 [I 2022-10-09 23:05:55] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:03 [I 2022-10-09 23:05:55] ...trap/engines/engine.py.229: process: 0.83220 | load: 0.00017 [I 2022-10-09 23:05:55] ...trap/engines/engine.py.230: loss: 0.14180 [I 2022-10-09 23:05:55] ...optimizers/trijoint.py.151: total_norm: 11.267200 [I 2022-10-09 23:05:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 2304/2304 [I 2022-10-09 23:05:59] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:00 [I 2022-10-09 23:05:59] ...trap/engines/engine.py.229: process: 0.82656 | load: 0.00016 [I 2022-10-09 23:05:59] ...trap/engines/engine.py.230: loss: 0.11949 [I 2022-10-09 23:05:59] ...optimizers/trijoint.py.151: total_norm: 10.694800 [S 2022-10-09 23:06:00] ...trap/engines/engine.py.129: train_epoch.epoch: 57 [S 2022-10-09 23:06:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.15008048415184022 [I 2022-10-09 23:06:00] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 23:06:01] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 57 [I 2022-10-09 23:06:01] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-09 23:06:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 23:06:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 23:06:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 23:06:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 23:06:05] ...trap/engines/engine.py.321: eval: epoch 57 | batch 0/510 [I 2022-10-09 23:06:05] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:18 [I 2022-10-09 23:06:05] ...trap/engines/engine.py.326: process: 4.15453 | load: 2.93772 [I 2022-10-09 23:06:10] ...trap/engines/engine.py.321: eval: epoch 57 | batch 10/510 [I 2022-10-09 23:06:10] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:52 [I 2022-10-09 23:06:10] ...trap/engines/engine.py.326: process: 0.52773 | load: 0.00018 [I 2022-10-09 23:06:16] ...trap/engines/engine.py.321: eval: epoch 57 | batch 20/510 [I 2022-10-09 23:06:16] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:04:57 [I 2022-10-09 23:06:16] ...trap/engines/engine.py.326: process: 0.61411 | load: 0.00018 [I 2022-10-09 23:06:20] ...trap/engines/engine.py.321: eval: epoch 57 | batch 30/510 [I 2022-10-09 23:06:20] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:03:54 [I 2022-10-09 23:06:20] ...trap/engines/engine.py.326: process: 0.64959 | load: 0.00018 [I 2022-10-09 23:06:25] ...trap/engines/engine.py.321: eval: epoch 57 | batch 40/510 [I 2022-10-09 23:06:25] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:49 [I 2022-10-09 23:06:25] ...trap/engines/engine.py.326: process: 0.43942 | load: 0.00018 [I 2022-10-09 23:06:30] ...trap/engines/engine.py.321: eval: epoch 57 | batch 50/510 [I 2022-10-09 23:06:30] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:23 [I 2022-10-09 23:06:30] ...trap/engines/engine.py.326: process: 0.45720 | load: 0.00018 [I 2022-10-09 23:06:34] ...trap/engines/engine.py.321: eval: epoch 57 | batch 60/510 [I 2022-10-09 23:06:34] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:25 [I 2022-10-09 23:06:34] ...trap/engines/engine.py.326: process: 0.42846 | load: 0.00018 [I 2022-10-09 23:06:39] ...trap/engines/engine.py.321: eval: epoch 57 | batch 70/510 [I 2022-10-09 23:06:39] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:24 [I 2022-10-09 23:06:39] ...trap/engines/engine.py.326: process: 0.46361 | load: 0.00018 [I 2022-10-09 23:06:45] ...trap/engines/engine.py.321: eval: epoch 57 | batch 80/510 [I 2022-10-09 23:06:45] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:08 [I 2022-10-09 23:06:45] ...trap/engines/engine.py.326: process: 0.56319 | load: 0.00019 [I 2022-10-09 23:06:49] ...trap/engines/engine.py.321: eval: epoch 57 | batch 90/510 [I 2022-10-09 23:06:49] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:09 [I 2022-10-09 23:06:49] ...trap/engines/engine.py.326: process: 0.44135 | load: 0.00019 [I 2022-10-09 23:06:53] ...trap/engines/engine.py.321: eval: epoch 57 | batch 100/510 [I 2022-10-09 23:06:53] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:47 [I 2022-10-09 23:06:53] ...trap/engines/engine.py.326: process: 0.30406 | load: 0.00018 [I 2022-10-09 23:06:57] ...trap/engines/engine.py.321: eval: epoch 57 | batch 110/510 [I 2022-10-09 23:06:57] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-09 23:06:57] ...trap/engines/engine.py.326: process: 0.31922 | load: 0.00020 [I 2022-10-09 23:07:00] ...trap/engines/engine.py.321: eval: epoch 57 | batch 120/510 [I 2022-10-09 23:07:00] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:03 [I 2022-10-09 23:07:00] ...trap/engines/engine.py.326: process: 0.31800 | load: 0.00018 [I 2022-10-09 23:07:03] ...trap/engines/engine.py.321: eval: epoch 57 | batch 130/510 [I 2022-10-09 23:07:03] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:00 [I 2022-10-09 23:07:03] ...trap/engines/engine.py.326: process: 0.31602 | load: 0.00019 [I 2022-10-09 23:07:06] ...trap/engines/engine.py.321: eval: epoch 57 | batch 140/510 [I 2022-10-09 23:07:06] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:57 [I 2022-10-09 23:07:06] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00019 [I 2022-10-09 23:07:09] ...trap/engines/engine.py.321: eval: epoch 57 | batch 150/510 [I 2022-10-09 23:07:09] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:53 [I 2022-10-09 23:07:09] ...trap/engines/engine.py.326: process: 0.31206 | load: 0.00020 [I 2022-10-09 23:07:12] ...trap/engines/engine.py.321: eval: epoch 57 | batch 160/510 [I 2022-10-09 23:07:12] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:51 [I 2022-10-09 23:07:12] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00019 [I 2022-10-09 23:07:16] ...trap/engines/engine.py.321: eval: epoch 57 | batch 170/510 [I 2022-10-09 23:07:16] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-09 23:07:16] ...trap/engines/engine.py.326: process: 0.31833 | load: 0.00020 [I 2022-10-09 23:07:19] ...trap/engines/engine.py.321: eval: epoch 57 | batch 180/510 [I 2022-10-09 23:07:19] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:45 [I 2022-10-09 23:07:19] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00020 [I 2022-10-09 23:07:22] ...trap/engines/engine.py.321: eval: epoch 57 | batch 190/510 [I 2022-10-09 23:07:22] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:41 [I 2022-10-09 23:07:22] ...trap/engines/engine.py.326: process: 0.31482 | load: 0.00019 [I 2022-10-09 23:07:25] ...trap/engines/engine.py.321: eval: epoch 57 | batch 200/510 [I 2022-10-09 23:07:25] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:37 [I 2022-10-09 23:07:25] ...trap/engines/engine.py.326: process: 0.32120 | load: 0.00019 [I 2022-10-09 23:07:28] ...trap/engines/engine.py.321: eval: epoch 57 | batch 210/510 [I 2022-10-09 23:07:28] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:34 [I 2022-10-09 23:07:28] ...trap/engines/engine.py.326: process: 0.31452 | load: 0.00019 [I 2022-10-09 23:07:31] ...trap/engines/engine.py.321: eval: epoch 57 | batch 220/510 [I 2022-10-09 23:07:31] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:31 [I 2022-10-09 23:07:31] ...trap/engines/engine.py.326: process: 0.32022 | load: 0.00019 [I 2022-10-09 23:07:35] ...trap/engines/engine.py.321: eval: epoch 57 | batch 230/510 [I 2022-10-09 23:07:35] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-09 23:07:35] ...trap/engines/engine.py.326: process: 0.31607 | load: 0.00020 [I 2022-10-09 23:07:38] ...trap/engines/engine.py.321: eval: epoch 57 | batch 240/510 [I 2022-10-09 23:07:38] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:25 [I 2022-10-09 23:07:38] ...trap/engines/engine.py.326: process: 0.31675 | load: 0.00019 [I 2022-10-09 23:07:41] ...trap/engines/engine.py.321: eval: epoch 57 | batch 250/510 [I 2022-10-09 23:07:41] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-09 23:07:41] ...trap/engines/engine.py.326: process: 0.31377 | load: 0.00021 [I 2022-10-09 23:07:44] ...trap/engines/engine.py.321: eval: epoch 57 | batch 260/510 [I 2022-10-09 23:07:44] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:19 [I 2022-10-09 23:07:44] ...trap/engines/engine.py.326: process: 0.31980 | load: 0.00019 [I 2022-10-09 23:07:47] ...trap/engines/engine.py.321: eval: epoch 57 | batch 270/510 [I 2022-10-09 23:07:47] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:16 [I 2022-10-09 23:07:47] ...trap/engines/engine.py.326: process: 0.31011 | load: 0.00021 [I 2022-10-09 23:07:50] ...trap/engines/engine.py.321: eval: epoch 57 | batch 280/510 [I 2022-10-09 23:07:50] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-09 23:07:50] ...trap/engines/engine.py.326: process: 0.31319 | load: 0.00020 [I 2022-10-09 23:07:54] ...trap/engines/engine.py.321: eval: epoch 57 | batch 290/510 [I 2022-10-09 23:07:54] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:10 [I 2022-10-09 23:07:54] ...trap/engines/engine.py.326: process: 0.32131 | load: 0.00019 [I 2022-10-09 23:07:57] ...trap/engines/engine.py.321: eval: epoch 57 | batch 300/510 [I 2022-10-09 23:07:57] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:07 [I 2022-10-09 23:07:57] ...trap/engines/engine.py.326: process: 0.32330 | load: 0.00020 [I 2022-10-09 23:08:00] ...trap/engines/engine.py.321: eval: epoch 57 | batch 310/510 [I 2022-10-09 23:08:00] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:03 [I 2022-10-09 23:08:00] ...trap/engines/engine.py.326: process: 0.31098 | load: 0.00021 [I 2022-10-09 23:08:03] ...trap/engines/engine.py.321: eval: epoch 57 | batch 320/510 [I 2022-10-09 23:08:03] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-09 23:08:03] ...trap/engines/engine.py.326: process: 0.31989 | load: 0.00024 [I 2022-10-09 23:08:06] ...trap/engines/engine.py.321: eval: epoch 57 | batch 330/510 [I 2022-10-09 23:08:06] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:56 [I 2022-10-09 23:08:06] ...trap/engines/engine.py.326: process: 0.31026 | load: 0.00019 [I 2022-10-09 23:08:10] ...trap/engines/engine.py.321: eval: epoch 57 | batch 340/510 [I 2022-10-09 23:08:10] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:53 [I 2022-10-09 23:08:10] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00019 [I 2022-10-09 23:08:13] ...trap/engines/engine.py.321: eval: epoch 57 | batch 350/510 [I 2022-10-09 23:08:13] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-09 23:08:13] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00021 [I 2022-10-09 23:08:16] ...trap/engines/engine.py.321: eval: epoch 57 | batch 360/510 [I 2022-10-09 23:08:16] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-09 23:08:16] ...trap/engines/engine.py.326: process: 0.31448 | load: 0.00022 [I 2022-10-09 23:08:19] ...trap/engines/engine.py.321: eval: epoch 57 | batch 370/510 [I 2022-10-09 23:08:19] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-09 23:08:19] ...trap/engines/engine.py.326: process: 0.31235 | load: 0.00019 [I 2022-10-09 23:08:22] ...trap/engines/engine.py.321: eval: epoch 57 | batch 380/510 [I 2022-10-09 23:08:22] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-09 23:08:22] ...trap/engines/engine.py.326: process: 0.32594 | load: 0.00021 [I 2022-10-09 23:08:25] ...trap/engines/engine.py.321: eval: epoch 57 | batch 390/510 [I 2022-10-09 23:08:25] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:37 [I 2022-10-09 23:08:25] ...trap/engines/engine.py.326: process: 0.31652 | load: 0.00019 [I 2022-10-09 23:08:29] ...trap/engines/engine.py.321: eval: epoch 57 | batch 400/510 [I 2022-10-09 23:08:29] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-09 23:08:29] ...trap/engines/engine.py.326: process: 0.30916 | load: 0.00021 [I 2022-10-09 23:08:32] ...trap/engines/engine.py.321: eval: epoch 57 | batch 410/510 [I 2022-10-09 23:08:32] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-09 23:08:32] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00021 [I 2022-10-09 23:08:35] ...trap/engines/engine.py.321: eval: epoch 57 | batch 420/510 [I 2022-10-09 23:08:35] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-09 23:08:35] ...trap/engines/engine.py.326: process: 0.32000 | load: 0.00022 [I 2022-10-09 23:08:38] ...trap/engines/engine.py.321: eval: epoch 57 | batch 430/510 [I 2022-10-09 23:08:38] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-09 23:08:38] ...trap/engines/engine.py.326: process: 0.32360 | load: 0.00019 [I 2022-10-09 23:08:41] ...trap/engines/engine.py.321: eval: epoch 57 | batch 440/510 [I 2022-10-09 23:08:41] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-09 23:08:41] ...trap/engines/engine.py.326: process: 0.31321 | load: 0.00019 [I 2022-10-09 23:08:45] ...trap/engines/engine.py.321: eval: epoch 57 | batch 450/510 [I 2022-10-09 23:08:45] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-09 23:08:45] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00019 [I 2022-10-09 23:08:48] ...trap/engines/engine.py.321: eval: epoch 57 | batch 460/510 [I 2022-10-09 23:08:48] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-09 23:08:48] ...trap/engines/engine.py.326: process: 0.31619 | load: 0.00019 [I 2022-10-09 23:08:51] ...trap/engines/engine.py.321: eval: epoch 57 | batch 470/510 [I 2022-10-09 23:08:51] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-09 23:08:51] ...trap/engines/engine.py.326: process: 0.32157 | load: 0.00020 [I 2022-10-09 23:08:54] ...trap/engines/engine.py.321: eval: epoch 57 | batch 480/510 [I 2022-10-09 23:08:54] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-09 23:08:54] ...trap/engines/engine.py.326: process: 0.32036 | load: 0.00019 [I 2022-10-09 23:08:57] ...trap/engines/engine.py.321: eval: epoch 57 | batch 490/510 [I 2022-10-09 23:08:57] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-09 23:08:57] ...trap/engines/engine.py.326: process: 0.31023 | load: 0.00019 [I 2022-10-09 23:09:00] ...trap/engines/engine.py.321: eval: epoch 57 | batch 500/510 [I 2022-10-09 23:09:00] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-09 23:09:00] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00020 [I 2022-10-09 23:09:04] ...trap/engines/engine.py.321: eval: epoch 57 | batch 510/510 [I 2022-10-09 23:09:04] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-09 23:09:04] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00015 [S 2022-10-09 23:09:04] ...trap/engines/engine.py.132: eval_epoch.epoch: 57 [I 2022-10-09 23:09:04] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.731300 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907100 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938400 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.719600 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903200 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935000 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.006133 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007077 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004454 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012290 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005546 [S 2022-10-09 23:09:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005814 [I 2022-10-09 23:09:38] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 23:09:38] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 23:09:38] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 23:09:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 23:09:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 23:09:41] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 23:09:42] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 23:09:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 23:09:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 23:09:48] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 23:09:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 23:09:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 23:09:57] ...trap/engines/engine.py.163: Training model on trainset for epoch 58 [I 2022-10-09 23:09:57] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 23:10:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 0/2304 [I 2022-10-09 23:10:00] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:16:26 [I 2022-10-09 23:10:00] ...trap/engines/engine.py.229: process: 3.55328 | load: 2.64220 [I 2022-10-09 23:10:00] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-09 23:10:00] ...optimizers/trijoint.py.151: total_norm: 17.199888 [I 2022-10-09 23:10:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 10/2304 [I 2022-10-09 23:10:09] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:37 [I 2022-10-09 23:10:09] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00538 [I 2022-10-09 23:10:09] ...trap/engines/engine.py.230: loss: 0.16583 [I 2022-10-09 23:10:09] ...optimizers/trijoint.py.151: total_norm: 15.214571 [I 2022-10-09 23:10:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 20/2304 [I 2022-10-09 23:10:17] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:34 [I 2022-10-09 23:10:17] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00439 [I 2022-10-09 23:10:17] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-09 23:10:17] ...optimizers/trijoint.py.151: total_norm: 17.752426 [I 2022-10-09 23:10:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 30/2304 [I 2022-10-09 23:10:26] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:37 [I 2022-10-09 23:10:26] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00548 [I 2022-10-09 23:10:26] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-09 23:10:26] ...optimizers/trijoint.py.151: total_norm: 12.678845 [I 2022-10-09 23:10:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 40/2304 [I 2022-10-09 23:10:35] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:06 [I 2022-10-09 23:10:35] ...trap/engines/engine.py.229: process: 0.84322 | load: 0.00496 [I 2022-10-09 23:10:35] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-09 23:10:35] ...optimizers/trijoint.py.151: total_norm: 14.321452 [I 2022-10-09 23:10:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 50/2304 [I 2022-10-09 23:10:43] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:31:56 [I 2022-10-09 23:10:43] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00390 [I 2022-10-09 23:10:43] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-09 23:10:43] ...optimizers/trijoint.py.151: total_norm: 28.107311 [I 2022-10-09 23:10:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 60/2304 [I 2022-10-09 23:10:52] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:42 [I 2022-10-09 23:10:52] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00355 [I 2022-10-09 23:10:52] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-09 23:10:52] ...optimizers/trijoint.py.151: total_norm: 13.693241 [I 2022-10-09 23:11:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 70/2304 [I 2022-10-09 23:11:00] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:53 [I 2022-10-09 23:11:00] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00326 [I 2022-10-09 23:11:00] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-09 23:11:00] ...optimizers/trijoint.py.151: total_norm: 30.140415 [I 2022-10-09 23:11:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 80/2304 [I 2022-10-09 23:11:09] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:45 [I 2022-10-09 23:11:09] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00371 [I 2022-10-09 23:11:09] ...trap/engines/engine.py.230: loss: 0.22661 [I 2022-10-09 23:11:09] ...optimizers/trijoint.py.151: total_norm: 17.624802 [I 2022-10-09 23:11:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 90/2304 [I 2022-10-09 23:11:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:23 [I 2022-10-09 23:11:17] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00359 [I 2022-10-09 23:11:17] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-09 23:11:17] ...optimizers/trijoint.py.151: total_norm: 14.257198 [I 2022-10-09 23:11:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 100/2304 [I 2022-10-09 23:11:26] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:17 [I 2022-10-09 23:11:26] ...trap/engines/engine.py.229: process: 0.84461 | load: 0.00367 [I 2022-10-09 23:11:26] ...trap/engines/engine.py.230: loss: 0.14591 [I 2022-10-09 23:11:26] ...optimizers/trijoint.py.151: total_norm: 10.579570 [I 2022-10-09 23:11:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 110/2304 [I 2022-10-09 23:11:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:06 [I 2022-10-09 23:11:34] ...trap/engines/engine.py.229: process: 0.83991 | load: 0.00346 [I 2022-10-09 23:11:34] ...trap/engines/engine.py.230: loss: 0.17232 [I 2022-10-09 23:11:34] ...optimizers/trijoint.py.151: total_norm: 17.816811 [I 2022-10-09 23:11:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 120/2304 [I 2022-10-09 23:11:43] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:02 [I 2022-10-09 23:11:43] ...trap/engines/engine.py.229: process: 0.85241 | load: 0.00349 [I 2022-10-09 23:11:43] ...trap/engines/engine.py.230: loss: 0.09408 [I 2022-10-09 23:11:43] ...optimizers/trijoint.py.151: total_norm: 9.109564 [I 2022-10-09 23:11:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 130/2304 [I 2022-10-09 23:11:51] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:57 [I 2022-10-09 23:11:51] ...trap/engines/engine.py.229: process: 0.86006 | load: 0.00345 [I 2022-10-09 23:11:51] ...trap/engines/engine.py.230: loss: 0.16768 [I 2022-10-09 23:11:51] ...optimizers/trijoint.py.151: total_norm: 15.607642 [I 2022-10-09 23:12:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 140/2304 [I 2022-10-09 23:12:00] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:54 [I 2022-10-09 23:12:00] ...trap/engines/engine.py.229: process: 0.86524 | load: 0.00331 [I 2022-10-09 23:12:00] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-09 23:12:00] ...optimizers/trijoint.py.151: total_norm: 31.891733 [I 2022-10-09 23:12:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 150/2304 [I 2022-10-09 23:12:09] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:37 [I 2022-10-09 23:12:09] ...trap/engines/engine.py.229: process: 0.85076 | load: 0.00345 [I 2022-10-09 23:12:09] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-09 23:12:09] ...optimizers/trijoint.py.151: total_norm: 9.583873 [I 2022-10-09 23:12:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 160/2304 [I 2022-10-09 23:12:17] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:38 [I 2022-10-09 23:12:17] ...trap/engines/engine.py.229: process: 0.86037 | load: 0.00334 [I 2022-10-09 23:12:17] ...trap/engines/engine.py.230: loss: 0.16057 [I 2022-10-09 23:12:17] ...optimizers/trijoint.py.151: total_norm: 14.266020 [I 2022-10-09 23:12:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 170/2304 [I 2022-10-09 23:12:26] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:22 [I 2022-10-09 23:12:26] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00337 [I 2022-10-09 23:12:26] ...trap/engines/engine.py.230: loss: 0.14652 [I 2022-10-09 23:12:26] ...optimizers/trijoint.py.151: total_norm: 19.223158 [I 2022-10-09 23:12:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 180/2304 [I 2022-10-09 23:12:34] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:10 [I 2022-10-09 23:12:34] ...trap/engines/engine.py.229: process: 0.85030 | load: 0.00326 [I 2022-10-09 23:12:34] ...trap/engines/engine.py.230: loss: 0.13606 [I 2022-10-09 23:12:34] ...optimizers/trijoint.py.151: total_norm: 28.948013 [I 2022-10-09 23:12:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 190/2304 [I 2022-10-09 23:12:43] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:57 [I 2022-10-09 23:12:43] ...trap/engines/engine.py.229: process: 0.83791 | load: 0.00301 [I 2022-10-09 23:12:43] ...trap/engines/engine.py.230: loss: 0.14737 [I 2022-10-09 23:12:43] ...optimizers/trijoint.py.151: total_norm: 34.508167 [I 2022-10-09 23:12:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 200/2304 [I 2022-10-09 23:12:51] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:53 [I 2022-10-09 23:12:51] ...trap/engines/engine.py.229: process: 0.85571 | load: 0.00331 [I 2022-10-09 23:12:51] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-09 23:12:51] ...optimizers/trijoint.py.151: total_norm: 11.172395 [I 2022-10-09 23:13:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 210/2304 [I 2022-10-09 23:13:00] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:45 [I 2022-10-09 23:13:00] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00325 [I 2022-10-09 23:13:00] ...trap/engines/engine.py.230: loss: 0.16478 [I 2022-10-09 23:13:00] ...optimizers/trijoint.py.151: total_norm: 23.157978 [I 2022-10-09 23:13:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 220/2304 [I 2022-10-09 23:13:09] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:47 [I 2022-10-09 23:13:09] ...trap/engines/engine.py.229: process: 0.85912 | load: 0.00339 [I 2022-10-09 23:13:09] ...trap/engines/engine.py.230: loss: 0.15697 [I 2022-10-09 23:13:09] ...optimizers/trijoint.py.151: total_norm: 22.511976 [I 2022-10-09 23:13:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 230/2304 [I 2022-10-09 23:13:17] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:29 [I 2022-10-09 23:13:17] ...trap/engines/engine.py.229: process: 0.84573 | load: 0.00324 [I 2022-10-09 23:13:17] ...trap/engines/engine.py.230: loss: 0.12652 [I 2022-10-09 23:13:17] ...optimizers/trijoint.py.151: total_norm: 8.361057 [I 2022-10-09 23:13:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 240/2304 [I 2022-10-09 23:13:26] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:21 [I 2022-10-09 23:13:26] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00302 [I 2022-10-09 23:13:26] ...trap/engines/engine.py.230: loss: 0.22546 [I 2022-10-09 23:13:26] ...optimizers/trijoint.py.151: total_norm: 27.223701 [I 2022-10-09 23:13:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 250/2304 [I 2022-10-09 23:13:34] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:17 [I 2022-10-09 23:13:34] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00312 [I 2022-10-09 23:13:34] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-09 23:13:34] ...optimizers/trijoint.py.151: total_norm: 11.899626 [I 2022-10-09 23:13:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 260/2304 [I 2022-10-09 23:13:43] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:51 [I 2022-10-09 23:13:43] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00311 [I 2022-10-09 23:13:43] ...trap/engines/engine.py.230: loss: 0.21746 [I 2022-10-09 23:13:43] ...optimizers/trijoint.py.151: total_norm: 18.430685 [I 2022-10-09 23:13:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 270/2304 [I 2022-10-09 23:13:51] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:51 [I 2022-10-09 23:13:51] ...trap/engines/engine.py.229: process: 0.85628 | load: 0.00329 [I 2022-10-09 23:13:51] ...trap/engines/engine.py.230: loss: 0.15032 [I 2022-10-09 23:13:51] ...optimizers/trijoint.py.151: total_norm: 9.441671 [I 2022-10-09 23:14:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 280/2304 [I 2022-10-09 23:14:00] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:50 [I 2022-10-09 23:14:00] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00327 [I 2022-10-09 23:14:00] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-09 23:14:00] ...optimizers/trijoint.py.151: total_norm: 17.026237 [I 2022-10-09 23:14:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 290/2304 [I 2022-10-09 23:14:08] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:41 [I 2022-10-09 23:14:08] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00300 [I 2022-10-09 23:14:08] ...trap/engines/engine.py.230: loss: 0.25944 [I 2022-10-09 23:14:08] ...optimizers/trijoint.py.151: total_norm: 39.123417 [I 2022-10-09 23:14:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 300/2304 [I 2022-10-09 23:14:17] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:23 [I 2022-10-09 23:14:17] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00314 [I 2022-10-09 23:14:17] ...trap/engines/engine.py.230: loss: 0.22516 [I 2022-10-09 23:14:17] ...optimizers/trijoint.py.151: total_norm: 17.570942 [I 2022-10-09 23:14:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 310/2304 [I 2022-10-09 23:14:25] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:24 [I 2022-10-09 23:14:25] ...trap/engines/engine.py.229: process: 0.86676 | load: 0.00316 [I 2022-10-09 23:14:25] ...trap/engines/engine.py.230: loss: 0.15835 [I 2022-10-09 23:14:25] ...optimizers/trijoint.py.151: total_norm: 11.610039 [I 2022-10-09 23:14:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 320/2304 [I 2022-10-09 23:14:34] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:08 [I 2022-10-09 23:14:34] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00308 [I 2022-10-09 23:14:34] ...trap/engines/engine.py.230: loss: 0.18483 [I 2022-10-09 23:14:34] ...optimizers/trijoint.py.151: total_norm: 15.579182 [I 2022-10-09 23:14:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 330/2304 [I 2022-10-09 23:14:42] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:27:46 [I 2022-10-09 23:14:42] ...trap/engines/engine.py.229: process: 0.82484 | load: 0.00298 [I 2022-10-09 23:14:42] ...trap/engines/engine.py.230: loss: 0.12105 [I 2022-10-09 23:14:42] ...optimizers/trijoint.py.151: total_norm: 7.414522 [I 2022-10-09 23:14:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 340/2304 [I 2022-10-09 23:14:51] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:27:42 [I 2022-10-09 23:14:51] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00298 [I 2022-10-09 23:14:51] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-09 23:14:51] ...optimizers/trijoint.py.151: total_norm: 8.758904 [I 2022-10-09 23:14:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 350/2304 [I 2022-10-09 23:14:59] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:36 [I 2022-10-09 23:14:59] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00302 [I 2022-10-09 23:14:59] ...trap/engines/engine.py.230: loss: 0.15132 [I 2022-10-09 23:14:59] ...optimizers/trijoint.py.151: total_norm: 11.260641 [I 2022-10-09 23:15:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 360/2304 [I 2022-10-09 23:15:08] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:19 [I 2022-10-09 23:15:08] ...trap/engines/engine.py.229: process: 0.84760 | load: 0.00303 [I 2022-10-09 23:15:08] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-09 23:15:08] ...optimizers/trijoint.py.151: total_norm: 25.609241 [I 2022-10-09 23:15:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 370/2304 [I 2022-10-09 23:15:16] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:10 [I 2022-10-09 23:15:16] ...trap/engines/engine.py.229: process: 0.85102 | load: 0.00294 [I 2022-10-09 23:15:16] ...trap/engines/engine.py.230: loss: 0.19581 [I 2022-10-09 23:15:16] ...optimizers/trijoint.py.151: total_norm: 22.152399 [I 2022-10-09 23:15:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 380/2304 [I 2022-10-09 23:15:25] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:19 [I 2022-10-09 23:15:25] ...trap/engines/engine.py.229: process: 0.86573 | load: 0.00303 [I 2022-10-09 23:15:25] ...trap/engines/engine.py.230: loss: 0.14373 [I 2022-10-09 23:15:25] ...optimizers/trijoint.py.151: total_norm: 23.313499 [I 2022-10-09 23:15:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 390/2304 [I 2022-10-09 23:15:33] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:26:53 [I 2022-10-09 23:15:33] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00292 [I 2022-10-09 23:15:33] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-09 23:15:33] ...optimizers/trijoint.py.151: total_norm: 17.415707 [I 2022-10-09 23:15:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 400/2304 [I 2022-10-09 23:15:42] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:41 [I 2022-10-09 23:15:42] ...trap/engines/engine.py.229: process: 0.84429 | load: 0.00290 [I 2022-10-09 23:15:42] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-09 23:15:42] ...optimizers/trijoint.py.151: total_norm: 15.197877 [I 2022-10-09 23:15:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 410/2304 [I 2022-10-09 23:15:50] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:31 [I 2022-10-09 23:15:50] ...trap/engines/engine.py.229: process: 0.83668 | load: 0.00297 [I 2022-10-09 23:15:50] ...trap/engines/engine.py.230: loss: 0.15041 [I 2022-10-09 23:15:50] ...optimizers/trijoint.py.151: total_norm: 17.974836 [I 2022-10-09 23:15:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 420/2304 [I 2022-10-09 23:15:59] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:39 [I 2022-10-09 23:15:59] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00304 [I 2022-10-09 23:15:59] ...trap/engines/engine.py.230: loss: 0.16674 [I 2022-10-09 23:15:59] ...optimizers/trijoint.py.151: total_norm: 32.040699 [I 2022-10-09 23:16:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 430/2304 [I 2022-10-09 23:16:07] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:27 [I 2022-10-09 23:16:07] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00291 [I 2022-10-09 23:16:07] ...trap/engines/engine.py.230: loss: 0.26874 [I 2022-10-09 23:16:07] ...optimizers/trijoint.py.151: total_norm: 22.836218 [I 2022-10-09 23:16:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 440/2304 [I 2022-10-09 23:16:16] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:07 [I 2022-10-09 23:16:16] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00292 [I 2022-10-09 23:16:16] ...trap/engines/engine.py.230: loss: 0.19619 [I 2022-10-09 23:16:16] ...optimizers/trijoint.py.151: total_norm: 26.089485 [I 2022-10-09 23:16:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 450/2304 [I 2022-10-09 23:16:24] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:25:56 [I 2022-10-09 23:16:24] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00296 [I 2022-10-09 23:16:24] ...trap/engines/engine.py.230: loss: 0.12399 [I 2022-10-09 23:16:24] ...optimizers/trijoint.py.151: total_norm: 14.459219 [I 2022-10-09 23:16:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 460/2304 [I 2022-10-09 23:16:33] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:53 [I 2022-10-09 23:16:33] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00293 [I 2022-10-09 23:16:33] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-09 23:16:33] ...optimizers/trijoint.py.151: total_norm: 12.976547 [I 2022-10-09 23:16:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 470/2304 [I 2022-10-09 23:16:41] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:42 [I 2022-10-09 23:16:41] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00292 [I 2022-10-09 23:16:41] ...trap/engines/engine.py.230: loss: 0.16936 [I 2022-10-09 23:16:41] ...optimizers/trijoint.py.151: total_norm: 21.026331 [I 2022-10-09 23:16:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 480/2304 [I 2022-10-09 23:16:49] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:37 [I 2022-10-09 23:16:49] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00292 [I 2022-10-09 23:16:49] ...trap/engines/engine.py.230: loss: 0.14223 [I 2022-10-09 23:16:49] ...optimizers/trijoint.py.151: total_norm: 14.141739 [I 2022-10-09 23:16:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 490/2304 [I 2022-10-09 23:16:58] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:34 [I 2022-10-09 23:16:58] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00294 [I 2022-10-09 23:16:58] ...trap/engines/engine.py.230: loss: 0.12365 [I 2022-10-09 23:16:58] ...optimizers/trijoint.py.151: total_norm: 13.823606 [I 2022-10-09 23:17:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 500/2304 [I 2022-10-09 23:17:06] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:26 [I 2022-10-09 23:17:06] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00292 [I 2022-10-09 23:17:06] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-09 23:17:06] ...optimizers/trijoint.py.151: total_norm: 13.835390 [I 2022-10-09 23:17:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 510/2304 [I 2022-10-09 23:17:15] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:21 [I 2022-10-09 23:17:15] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00301 [I 2022-10-09 23:17:15] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-09 23:17:15] ...optimizers/trijoint.py.151: total_norm: 18.509588 [I 2022-10-09 23:17:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 520/2304 [I 2022-10-09 23:17:23] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:15 [I 2022-10-09 23:17:23] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00296 [I 2022-10-09 23:17:23] ...trap/engines/engine.py.230: loss: 0.20404 [I 2022-10-09 23:17:23] ...optimizers/trijoint.py.151: total_norm: 20.591883 [I 2022-10-09 23:17:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 530/2304 [I 2022-10-09 23:17:32] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:51 [I 2022-10-09 23:17:32] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00298 [I 2022-10-09 23:17:32] ...trap/engines/engine.py.230: loss: 0.13604 [I 2022-10-09 23:17:32] ...optimizers/trijoint.py.151: total_norm: 18.856325 [I 2022-10-09 23:17:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 540/2304 [I 2022-10-09 23:17:40] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:42 [I 2022-10-09 23:17:40] ...trap/engines/engine.py.229: process: 0.84459 | load: 0.00294 [I 2022-10-09 23:17:40] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-09 23:17:40] ...optimizers/trijoint.py.151: total_norm: 11.864546 [I 2022-10-09 23:17:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 550/2304 [I 2022-10-09 23:17:49] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:43 [I 2022-10-09 23:17:49] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00294 [I 2022-10-09 23:17:49] ...trap/engines/engine.py.230: loss: 0.18408 [I 2022-10-09 23:17:49] ...optimizers/trijoint.py.151: total_norm: 20.073635 [I 2022-10-09 23:17:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 560/2304 [I 2022-10-09 23:17:57] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:28 [I 2022-10-09 23:17:57] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00298 [I 2022-10-09 23:17:57] ...trap/engines/engine.py.230: loss: 0.20464 [I 2022-10-09 23:17:57] ...optimizers/trijoint.py.151: total_norm: 26.032833 [I 2022-10-09 23:18:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 570/2304 [I 2022-10-09 23:18:06] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:20 [I 2022-10-09 23:18:06] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00309 [I 2022-10-09 23:18:06] ...trap/engines/engine.py.230: loss: 0.24038 [I 2022-10-09 23:18:06] ...optimizers/trijoint.py.151: total_norm: 11.318022 [I 2022-10-09 23:18:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 580/2304 [I 2022-10-09 23:18:14] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:14 [I 2022-10-09 23:18:14] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00297 [I 2022-10-09 23:18:14] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-09 23:18:14] ...optimizers/trijoint.py.151: total_norm: 23.422098 [I 2022-10-09 23:18:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 590/2304 [I 2022-10-09 23:18:22] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:05 [I 2022-10-09 23:18:22] ...trap/engines/engine.py.229: process: 0.83277 | load: 0.00300 [I 2022-10-09 23:18:22] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-09 23:18:22] ...optimizers/trijoint.py.151: total_norm: 20.851929 [I 2022-10-09 23:18:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 600/2304 [I 2022-10-09 23:18:31] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:23:53 [I 2022-10-09 23:18:31] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00296 [I 2022-10-09 23:18:31] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-09 23:18:31] ...optimizers/trijoint.py.151: total_norm: 11.263706 [I 2022-10-09 23:18:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 610/2304 [I 2022-10-09 23:18:39] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:23:50 [I 2022-10-09 23:18:39] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00300 [I 2022-10-09 23:18:39] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-09 23:18:39] ...optimizers/trijoint.py.151: total_norm: 17.048359 [I 2022-10-09 23:18:48] ...trap/engines/engine.py.224: train: epoch 58 | batch 620/2304 [I 2022-10-09 23:18:48] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:45 [I 2022-10-09 23:18:48] ...trap/engines/engine.py.229: process: 0.86036 | load: 0.00297 [I 2022-10-09 23:18:48] ...trap/engines/engine.py.230: loss: 0.16777 [I 2022-10-09 23:18:48] ...optimizers/trijoint.py.151: total_norm: 18.113459 [I 2022-10-09 23:18:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 630/2304 [I 2022-10-09 23:18:56] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:40 [I 2022-10-09 23:18:56] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00312 [I 2022-10-09 23:18:56] ...trap/engines/engine.py.230: loss: 0.20762 [I 2022-10-09 23:18:56] ...optimizers/trijoint.py.151: total_norm: 18.097731 [I 2022-10-09 23:19:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 640/2304 [I 2022-10-09 23:19:05] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:36 [I 2022-10-09 23:19:05] ...trap/engines/engine.py.229: process: 0.85929 | load: 0.00317 [I 2022-10-09 23:19:05] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-09 23:19:05] ...optimizers/trijoint.py.151: total_norm: 11.462270 [I 2022-10-09 23:19:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 650/2304 [I 2022-10-09 23:19:13] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:29 [I 2022-10-09 23:19:13] ...trap/engines/engine.py.229: process: 0.85706 | load: 0.00311 [I 2022-10-09 23:19:13] ...trap/engines/engine.py.230: loss: 0.16178 [I 2022-10-09 23:19:13] ...optimizers/trijoint.py.151: total_norm: 28.330379 [I 2022-10-09 23:19:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 660/2304 [I 2022-10-09 23:19:22] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:22 [I 2022-10-09 23:19:22] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00312 [I 2022-10-09 23:19:22] ...trap/engines/engine.py.230: loss: 0.12920 [I 2022-10-09 23:19:22] ...optimizers/trijoint.py.151: total_norm: 14.802175 [I 2022-10-09 23:19:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 670/2304 [I 2022-10-09 23:19:30] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:00 [I 2022-10-09 23:19:30] ...trap/engines/engine.py.229: process: 0.83611 | load: 0.00311 [I 2022-10-09 23:19:30] ...trap/engines/engine.py.230: loss: 0.23031 [I 2022-10-09 23:19:30] ...optimizers/trijoint.py.151: total_norm: 27.515732 [I 2022-10-09 23:19:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 680/2304 [I 2022-10-09 23:19:39] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:59 [I 2022-10-09 23:19:39] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00317 [I 2022-10-09 23:19:39] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-09 23:19:39] ...optimizers/trijoint.py.151: total_norm: 18.199224 [I 2022-10-09 23:19:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 690/2304 [I 2022-10-09 23:19:47] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:56 [I 2022-10-09 23:19:47] ...trap/engines/engine.py.229: process: 0.86071 | load: 0.00307 [I 2022-10-09 23:19:47] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-09 23:19:47] ...optimizers/trijoint.py.151: total_norm: 19.431231 [I 2022-10-09 23:19:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 700/2304 [I 2022-10-09 23:19:56] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:41 [I 2022-10-09 23:19:56] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00313 [I 2022-10-09 23:19:56] ...trap/engines/engine.py.230: loss: 0.11172 [I 2022-10-09 23:19:56] ...optimizers/trijoint.py.151: total_norm: 7.213025 [I 2022-10-09 23:20:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 710/2304 [I 2022-10-09 23:20:05] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:46 [I 2022-10-09 23:20:05] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00314 [I 2022-10-09 23:20:05] ...trap/engines/engine.py.230: loss: 0.11368 [I 2022-10-09 23:20:05] ...optimizers/trijoint.py.151: total_norm: 28.780209 [I 2022-10-09 23:20:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 720/2304 [I 2022-10-09 23:20:13] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:37 [I 2022-10-09 23:20:13] ...trap/engines/engine.py.229: process: 0.85789 | load: 0.00309 [I 2022-10-09 23:20:13] ...trap/engines/engine.py.230: loss: 0.20189 [I 2022-10-09 23:20:13] ...optimizers/trijoint.py.151: total_norm: 9.293656 [I 2022-10-09 23:20:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 730/2304 [I 2022-10-09 23:20:22] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:11 [I 2022-10-09 23:20:22] ...trap/engines/engine.py.229: process: 0.84846 | load: 0.00312 [I 2022-10-09 23:20:22] ...trap/engines/engine.py.230: loss: 0.14606 [I 2022-10-09 23:20:22] ...optimizers/trijoint.py.151: total_norm: 14.997499 [I 2022-10-09 23:20:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 740/2304 [I 2022-10-09 23:20:30] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:05 [I 2022-10-09 23:20:30] ...trap/engines/engine.py.229: process: 0.84995 | load: 0.00310 [I 2022-10-09 23:20:30] ...trap/engines/engine.py.230: loss: 0.10537 [I 2022-10-09 23:20:30] ...optimizers/trijoint.py.151: total_norm: 10.209139 [I 2022-10-09 23:20:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 750/2304 [I 2022-10-09 23:20:39] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:58 [I 2022-10-09 23:20:39] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00305 [I 2022-10-09 23:20:39] ...trap/engines/engine.py.230: loss: 0.09395 [I 2022-10-09 23:20:39] ...optimizers/trijoint.py.151: total_norm: 7.595927 [I 2022-10-09 23:20:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 760/2304 [I 2022-10-09 23:20:47] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:58 [I 2022-10-09 23:20:47] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00312 [I 2022-10-09 23:20:47] ...trap/engines/engine.py.230: loss: 0.19117 [I 2022-10-09 23:20:47] ...optimizers/trijoint.py.151: total_norm: 19.374249 [I 2022-10-09 23:20:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 770/2304 [I 2022-10-09 23:20:56] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:42 [I 2022-10-09 23:20:56] ...trap/engines/engine.py.229: process: 0.83545 | load: 0.00311 [I 2022-10-09 23:20:56] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-09 23:20:56] ...optimizers/trijoint.py.151: total_norm: 16.642700 [I 2022-10-09 23:21:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 780/2304 [I 2022-10-09 23:21:04] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:38 [I 2022-10-09 23:21:04] ...trap/engines/engine.py.229: process: 0.84731 | load: 0.00312 [I 2022-10-09 23:21:04] ...trap/engines/engine.py.230: loss: 0.15337 [I 2022-10-09 23:21:04] ...optimizers/trijoint.py.151: total_norm: 12.351302 [I 2022-10-09 23:21:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 790/2304 [I 2022-10-09 23:21:13] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:30 [I 2022-10-09 23:21:13] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00310 [I 2022-10-09 23:21:13] ...trap/engines/engine.py.230: loss: 0.11200 [I 2022-10-09 23:21:13] ...optimizers/trijoint.py.151: total_norm: 14.858147 [I 2022-10-09 23:21:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 800/2304 [I 2022-10-09 23:21:21] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:22 [I 2022-10-09 23:21:21] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00310 [I 2022-10-09 23:21:21] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-09 23:21:21] ...optimizers/trijoint.py.151: total_norm: 19.713871 [I 2022-10-09 23:21:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 810/2304 [I 2022-10-09 23:21:30] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:09 [I 2022-10-09 23:21:30] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00309 [I 2022-10-09 23:21:30] ...trap/engines/engine.py.230: loss: 0.14199 [I 2022-10-09 23:21:30] ...optimizers/trijoint.py.151: total_norm: 14.882237 [I 2022-10-09 23:21:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 820/2304 [I 2022-10-09 23:21:38] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:20:59 [I 2022-10-09 23:21:38] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00310 [I 2022-10-09 23:21:38] ...trap/engines/engine.py.230: loss: 0.17093 [I 2022-10-09 23:21:38] ...optimizers/trijoint.py.151: total_norm: 17.102547 [I 2022-10-09 23:21:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 830/2304 [I 2022-10-09 23:21:47] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:47 [I 2022-10-09 23:21:47] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00312 [I 2022-10-09 23:21:47] ...trap/engines/engine.py.230: loss: 0.15668 [I 2022-10-09 23:21:47] ...optimizers/trijoint.py.151: total_norm: 10.881850 [I 2022-10-09 23:21:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 840/2304 [I 2022-10-09 23:21:55] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:53 [I 2022-10-09 23:21:55] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00314 [I 2022-10-09 23:21:55] ...trap/engines/engine.py.230: loss: 0.15934 [I 2022-10-09 23:21:55] ...optimizers/trijoint.py.151: total_norm: 18.121845 [I 2022-10-09 23:22:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 850/2304 [I 2022-10-09 23:22:04] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:35 [I 2022-10-09 23:22:04] ...trap/engines/engine.py.229: process: 0.84999 | load: 0.00312 [I 2022-10-09 23:22:04] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-09 23:22:04] ...optimizers/trijoint.py.151: total_norm: 9.275909 [I 2022-10-09 23:22:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 860/2304 [I 2022-10-09 23:22:12] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:28 [I 2022-10-09 23:22:12] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00310 [I 2022-10-09 23:22:12] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-09 23:22:12] ...optimizers/trijoint.py.151: total_norm: 8.474308 [I 2022-10-09 23:22:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 870/2304 [I 2022-10-09 23:22:21] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:26 [I 2022-10-09 23:22:21] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00345 [I 2022-10-09 23:22:21] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-09 23:22:21] ...optimizers/trijoint.py.151: total_norm: 20.699068 [I 2022-10-09 23:22:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 880/2304 [I 2022-10-09 23:22:29] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:04 [I 2022-10-09 23:22:29] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00309 [I 2022-10-09 23:22:29] ...trap/engines/engine.py.230: loss: 0.16327 [I 2022-10-09 23:22:29] ...optimizers/trijoint.py.151: total_norm: 11.795063 [I 2022-10-09 23:22:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 890/2304 [I 2022-10-09 23:22:38] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:08 [I 2022-10-09 23:22:38] ...trap/engines/engine.py.229: process: 0.83711 | load: 0.00312 [I 2022-10-09 23:22:38] ...trap/engines/engine.py.230: loss: 0.20451 [I 2022-10-09 23:22:38] ...optimizers/trijoint.py.151: total_norm: 14.419220 [I 2022-10-09 23:22:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 900/2304 [I 2022-10-09 23:22:47] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:56 [I 2022-10-09 23:22:47] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00315 [I 2022-10-09 23:22:47] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-09 23:22:47] ...optimizers/trijoint.py.151: total_norm: 14.905766 [I 2022-10-09 23:22:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 910/2304 [I 2022-10-09 23:22:55] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:45 [I 2022-10-09 23:22:55] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00313 [I 2022-10-09 23:22:55] ...trap/engines/engine.py.230: loss: 0.16061 [I 2022-10-09 23:22:55] ...optimizers/trijoint.py.151: total_norm: 16.881359 [I 2022-10-09 23:23:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 920/2304 [I 2022-10-09 23:23:04] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:36 [I 2022-10-09 23:23:04] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00311 [I 2022-10-09 23:23:04] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-09 23:23:04] ...optimizers/trijoint.py.151: total_norm: 15.268697 [I 2022-10-09 23:23:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 930/2304 [I 2022-10-09 23:23:12] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:32 [I 2022-10-09 23:23:12] ...trap/engines/engine.py.229: process: 0.85981 | load: 0.00311 [I 2022-10-09 23:23:12] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-09 23:23:12] ...optimizers/trijoint.py.151: total_norm: 14.627507 [I 2022-10-09 23:23:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 940/2304 [I 2022-10-09 23:23:21] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:20 [I 2022-10-09 23:23:21] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00313 [I 2022-10-09 23:23:21] ...trap/engines/engine.py.230: loss: 0.16247 [I 2022-10-09 23:23:21] ...optimizers/trijoint.py.151: total_norm: 26.232483 [I 2022-10-09 23:23:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 950/2304 [I 2022-10-09 23:23:29] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:11 [I 2022-10-09 23:23:29] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00311 [I 2022-10-09 23:23:29] ...trap/engines/engine.py.230: loss: 0.15915 [I 2022-10-09 23:23:29] ...optimizers/trijoint.py.151: total_norm: 17.224846 [I 2022-10-09 23:23:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 960/2304 [I 2022-10-09 23:23:38] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:18:53 [I 2022-10-09 23:23:38] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00314 [I 2022-10-09 23:23:38] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-09 23:23:38] ...optimizers/trijoint.py.151: total_norm: 19.570164 [I 2022-10-09 23:23:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 970/2304 [I 2022-10-09 23:23:46] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:45 [I 2022-10-09 23:23:46] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00306 [I 2022-10-09 23:23:46] ...trap/engines/engine.py.230: loss: 0.16079 [I 2022-10-09 23:23:46] ...optimizers/trijoint.py.151: total_norm: 25.477856 [I 2022-10-09 23:23:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 980/2304 [I 2022-10-09 23:23:55] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:33 [I 2022-10-09 23:23:55] ...trap/engines/engine.py.229: process: 0.83654 | load: 0.00296 [I 2022-10-09 23:23:55] ...trap/engines/engine.py.230: loss: 0.23976 [I 2022-10-09 23:23:55] ...optimizers/trijoint.py.151: total_norm: 18.715908 [I 2022-10-09 23:24:03] ...trap/engines/engine.py.224: train: epoch 58 | batch 990/2304 [I 2022-10-09 23:24:03] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:25 [I 2022-10-09 23:24:03] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00291 [I 2022-10-09 23:24:03] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-09 23:24:03] ...optimizers/trijoint.py.151: total_norm: 19.679268 [I 2022-10-09 23:24:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 1000/2304 [I 2022-10-09 23:24:11] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:22 [I 2022-10-09 23:24:11] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00291 [I 2022-10-09 23:24:11] ...trap/engines/engine.py.230: loss: 0.12501 [I 2022-10-09 23:24:11] ...optimizers/trijoint.py.151: total_norm: 21.219418 [I 2022-10-09 23:24:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 1010/2304 [I 2022-10-09 23:24:20] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:11 [I 2022-10-09 23:24:20] ...trap/engines/engine.py.229: process: 0.85637 | load: 0.00297 [I 2022-10-09 23:24:20] ...trap/engines/engine.py.230: loss: 0.21015 [I 2022-10-09 23:24:20] ...optimizers/trijoint.py.151: total_norm: 15.210779 [I 2022-10-09 23:24:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 1020/2304 [I 2022-10-09 23:24:28] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:03 [I 2022-10-09 23:24:28] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00300 [I 2022-10-09 23:24:28] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-09 23:24:28] ...optimizers/trijoint.py.151: total_norm: 14.301875 [I 2022-10-09 23:24:37] ...trap/engines/engine.py.224: train: epoch 58 | batch 1030/2304 [I 2022-10-09 23:24:37] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:17:54 [I 2022-10-09 23:24:37] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00293 [I 2022-10-09 23:24:37] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-09 23:24:37] ...optimizers/trijoint.py.151: total_norm: 12.611049 [I 2022-10-09 23:24:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 1040/2304 [I 2022-10-09 23:24:45] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:45 [I 2022-10-09 23:24:45] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00296 [I 2022-10-09 23:24:45] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-09 23:24:45] ...optimizers/trijoint.py.151: total_norm: 18.630169 [I 2022-10-09 23:24:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 1050/2304 [I 2022-10-09 23:24:54] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:34 [I 2022-10-09 23:24:54] ...trap/engines/engine.py.229: process: 0.83413 | load: 0.00298 [I 2022-10-09 23:24:54] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-09 23:24:54] ...optimizers/trijoint.py.151: total_norm: 28.367920 [I 2022-10-09 23:25:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 1060/2304 [I 2022-10-09 23:25:02] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:26 [I 2022-10-09 23:25:02] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00294 [I 2022-10-09 23:25:02] ...trap/engines/engine.py.230: loss: 0.16069 [I 2022-10-09 23:25:02] ...optimizers/trijoint.py.151: total_norm: 11.824930 [I 2022-10-09 23:25:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 1070/2304 [I 2022-10-09 23:25:11] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:24 [I 2022-10-09 23:25:11] ...trap/engines/engine.py.229: process: 0.84176 | load: 0.00297 [I 2022-10-09 23:25:11] ...trap/engines/engine.py.230: loss: 0.10222 [I 2022-10-09 23:25:11] ...optimizers/trijoint.py.151: total_norm: 10.872814 [I 2022-10-09 23:25:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 1080/2304 [I 2022-10-09 23:25:19] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:14 [I 2022-10-09 23:25:19] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00297 [I 2022-10-09 23:25:19] ...trap/engines/engine.py.230: loss: 0.16391 [I 2022-10-09 23:25:19] ...optimizers/trijoint.py.151: total_norm: 28.822779 [I 2022-10-09 23:25:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 1090/2304 [I 2022-10-09 23:25:28] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:08 [I 2022-10-09 23:25:28] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00298 [I 2022-10-09 23:25:28] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-09 23:25:28] ...optimizers/trijoint.py.151: total_norm: 37.274925 [I 2022-10-09 23:25:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 1100/2304 [I 2022-10-09 23:25:36] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:01 [I 2022-10-09 23:25:36] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00298 [I 2022-10-09 23:25:36] ...trap/engines/engine.py.230: loss: 0.12721 [I 2022-10-09 23:25:36] ...optimizers/trijoint.py.151: total_norm: 11.189873 [I 2022-10-09 23:25:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 1110/2304 [I 2022-10-09 23:25:45] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:54 [I 2022-10-09 23:25:45] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00321 [I 2022-10-09 23:25:45] ...trap/engines/engine.py.230: loss: 0.09989 [I 2022-10-09 23:25:45] ...optimizers/trijoint.py.151: total_norm: 7.448305 [I 2022-10-09 23:25:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1120/2304 [I 2022-10-09 23:25:53] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:43 [I 2022-10-09 23:25:53] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00306 [I 2022-10-09 23:25:53] ...trap/engines/engine.py.230: loss: 0.10843 [I 2022-10-09 23:25:53] ...optimizers/trijoint.py.151: total_norm: 15.161410 [I 2022-10-09 23:26:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 1130/2304 [I 2022-10-09 23:26:02] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:31 [I 2022-10-09 23:26:02] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00297 [I 2022-10-09 23:26:02] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-09 23:26:02] ...optimizers/trijoint.py.151: total_norm: 10.243152 [I 2022-10-09 23:26:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1140/2304 [I 2022-10-09 23:26:10] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:08 [I 2022-10-09 23:26:10] ...trap/engines/engine.py.229: process: 0.82283 | load: 0.00296 [I 2022-10-09 23:26:10] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-09 23:26:10] ...optimizers/trijoint.py.151: total_norm: 13.180232 [I 2022-10-09 23:26:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1150/2304 [I 2022-10-09 23:26:18] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:08 [I 2022-10-09 23:26:18] ...trap/engines/engine.py.229: process: 0.83082 | load: 0.00296 [I 2022-10-09 23:26:18] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-09 23:26:18] ...optimizers/trijoint.py.151: total_norm: 11.564611 [I 2022-10-09 23:26:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1160/2304 [I 2022-10-09 23:26:27] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:08 [I 2022-10-09 23:26:27] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00297 [I 2022-10-09 23:26:27] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-09 23:26:27] ...optimizers/trijoint.py.151: total_norm: 13.832219 [I 2022-10-09 23:26:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 1170/2304 [I 2022-10-09 23:26:35] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:16:04 [I 2022-10-09 23:26:35] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00312 [I 2022-10-09 23:26:35] ...trap/engines/engine.py.230: loss: 0.12532 [I 2022-10-09 23:26:35] ...optimizers/trijoint.py.151: total_norm: 9.325289 [I 2022-10-09 23:26:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1180/2304 [I 2022-10-09 23:26:44] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:57 [I 2022-10-09 23:26:44] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00310 [I 2022-10-09 23:26:44] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-09 23:26:44] ...optimizers/trijoint.py.151: total_norm: 11.129932 [I 2022-10-09 23:26:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 1190/2304 [I 2022-10-09 23:26:52] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:51 [I 2022-10-09 23:26:52] ...trap/engines/engine.py.229: process: 0.86631 | load: 0.00312 [I 2022-10-09 23:26:52] ...trap/engines/engine.py.230: loss: 0.19093 [I 2022-10-09 23:26:52] ...optimizers/trijoint.py.151: total_norm: 27.199358 [I 2022-10-09 23:27:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1200/2304 [I 2022-10-09 23:27:01] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:41 [I 2022-10-09 23:27:01] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00301 [I 2022-10-09 23:27:01] ...trap/engines/engine.py.230: loss: 0.15826 [I 2022-10-09 23:27:01] ...optimizers/trijoint.py.151: total_norm: 20.872444 [I 2022-10-09 23:27:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1210/2304 [I 2022-10-09 23:27:10] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:38 [I 2022-10-09 23:27:10] ...trap/engines/engine.py.229: process: 0.87038 | load: 0.00309 [I 2022-10-09 23:27:10] ...trap/engines/engine.py.230: loss: 0.12026 [I 2022-10-09 23:27:10] ...optimizers/trijoint.py.151: total_norm: 13.378124 [I 2022-10-09 23:27:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1220/2304 [I 2022-10-09 23:27:18] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:22 [I 2022-10-09 23:27:18] ...trap/engines/engine.py.229: process: 0.84902 | load: 0.00322 [I 2022-10-09 23:27:18] ...trap/engines/engine.py.230: loss: 0.16117 [I 2022-10-09 23:27:18] ...optimizers/trijoint.py.151: total_norm: 20.216211 [I 2022-10-09 23:27:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1230/2304 [I 2022-10-09 23:27:27] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:15 [I 2022-10-09 23:27:27] ...trap/engines/engine.py.229: process: 0.85958 | load: 0.00312 [I 2022-10-09 23:27:27] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-09 23:27:27] ...optimizers/trijoint.py.151: total_norm: 17.818634 [I 2022-10-09 23:27:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 1240/2304 [I 2022-10-09 23:27:35] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:03 [I 2022-10-09 23:27:35] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00311 [I 2022-10-09 23:27:35] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-09 23:27:35] ...optimizers/trijoint.py.151: total_norm: 11.312756 [I 2022-10-09 23:27:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1250/2304 [I 2022-10-09 23:27:44] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:00 [I 2022-10-09 23:27:44] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00312 [I 2022-10-09 23:27:44] ...trap/engines/engine.py.230: loss: 0.14721 [I 2022-10-09 23:27:44] ...optimizers/trijoint.py.151: total_norm: 9.062098 [I 2022-10-09 23:27:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 1260/2304 [I 2022-10-09 23:27:52] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:50 [I 2022-10-09 23:27:52] ...trap/engines/engine.py.229: process: 0.85920 | load: 0.00314 [I 2022-10-09 23:27:52] ...trap/engines/engine.py.230: loss: 0.14682 [I 2022-10-09 23:27:52] ...optimizers/trijoint.py.151: total_norm: 8.632238 [I 2022-10-09 23:28:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1270/2304 [I 2022-10-09 23:28:01] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:14:35 [I 2022-10-09 23:28:01] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00315 [I 2022-10-09 23:28:01] ...trap/engines/engine.py.230: loss: 0.15270 [I 2022-10-09 23:28:01] ...optimizers/trijoint.py.151: total_norm: 29.519802 [I 2022-10-09 23:28:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 1280/2304 [I 2022-10-09 23:28:09] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:25 [I 2022-10-09 23:28:09] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00312 [I 2022-10-09 23:28:09] ...trap/engines/engine.py.230: loss: 0.18458 [I 2022-10-09 23:28:09] ...optimizers/trijoint.py.151: total_norm: 23.422802 [I 2022-10-09 23:28:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1290/2304 [I 2022-10-09 23:28:18] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:20 [I 2022-10-09 23:28:18] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00311 [I 2022-10-09 23:28:18] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-09 23:28:18] ...optimizers/trijoint.py.151: total_norm: 9.594966 [I 2022-10-09 23:28:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 1300/2304 [I 2022-10-09 23:28:45] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:46:17 [I 2022-10-09 23:28:45] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00313 [I 2022-10-09 23:28:45] ...trap/engines/engine.py.230: loss: 0.21213 [I 2022-10-09 23:28:45] ...optimizers/trijoint.py.151: total_norm: 16.553667 [I 2022-10-09 23:28:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1310/2304 [I 2022-10-09 23:28:53] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:17:29 [I 2022-10-09 23:28:53] ...trap/engines/engine.py.229: process: 0.85726 | load: 0.00313 [I 2022-10-09 23:28:53] ...trap/engines/engine.py.230: loss: 0.23588 [I 2022-10-09 23:28:53] ...optimizers/trijoint.py.151: total_norm: 18.855587 [I 2022-10-09 23:29:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 1320/2304 [I 2022-10-09 23:29:02] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:14:16 [I 2022-10-09 23:29:02] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00295 [I 2022-10-09 23:29:02] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-09 23:29:02] ...optimizers/trijoint.py.151: total_norm: 16.237507 [I 2022-10-09 23:29:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1330/2304 [I 2022-10-09 23:29:10] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:53 [I 2022-10-09 23:29:10] ...trap/engines/engine.py.229: process: 0.85184 | load: 0.00312 [I 2022-10-09 23:29:10] ...trap/engines/engine.py.230: loss: 0.20972 [I 2022-10-09 23:29:10] ...optimizers/trijoint.py.151: total_norm: 24.434410 [I 2022-10-09 23:29:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 1340/2304 [I 2022-10-09 23:29:19] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:38 [I 2022-10-09 23:29:19] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00308 [I 2022-10-09 23:29:19] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-09 23:29:19] ...optimizers/trijoint.py.151: total_norm: 22.439644 [I 2022-10-09 23:29:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1350/2304 [I 2022-10-09 23:29:27] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:29 [I 2022-10-09 23:29:27] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00308 [I 2022-10-09 23:29:27] ...trap/engines/engine.py.230: loss: 0.15042 [I 2022-10-09 23:29:27] ...optimizers/trijoint.py.151: total_norm: 15.092021 [I 2022-10-09 23:29:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 1360/2304 [I 2022-10-09 23:29:36] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:23 [I 2022-10-09 23:29:36] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00316 [I 2022-10-09 23:29:36] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-09 23:29:36] ...optimizers/trijoint.py.151: total_norm: 14.483235 [I 2022-10-09 23:29:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1370/2304 [I 2022-10-09 23:29:44] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:14 [I 2022-10-09 23:29:44] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00311 [I 2022-10-09 23:29:44] ...trap/engines/engine.py.230: loss: 0.16274 [I 2022-10-09 23:29:44] ...optimizers/trijoint.py.151: total_norm: 28.111879 [I 2022-10-09 23:29:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1380/2304 [I 2022-10-09 23:29:53] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:07 [I 2022-10-09 23:29:53] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00318 [I 2022-10-09 23:29:53] ...trap/engines/engine.py.230: loss: 0.18659 [I 2022-10-09 23:29:53] ...optimizers/trijoint.py.151: total_norm: 23.866968 [I 2022-10-09 23:30:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 1390/2304 [I 2022-10-09 23:30:02] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:00 [I 2022-10-09 23:30:02] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00312 [I 2022-10-09 23:30:02] ...trap/engines/engine.py.230: loss: 0.18021 [I 2022-10-09 23:30:02] ...optimizers/trijoint.py.151: total_norm: 24.547615 [I 2022-10-09 23:30:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1400/2304 [I 2022-10-09 23:30:10] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:46 [I 2022-10-09 23:30:10] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00314 [I 2022-10-09 23:30:10] ...trap/engines/engine.py.230: loss: 0.18973 [I 2022-10-09 23:30:10] ...optimizers/trijoint.py.151: total_norm: 48.577305 [I 2022-10-09 23:30:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 1410/2304 [I 2022-10-09 23:30:19] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:39 [I 2022-10-09 23:30:19] ...trap/engines/engine.py.229: process: 0.84949 | load: 0.00314 [I 2022-10-09 23:30:19] ...trap/engines/engine.py.230: loss: 0.11363 [I 2022-10-09 23:30:19] ...optimizers/trijoint.py.151: total_norm: 20.029705 [I 2022-10-09 23:30:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1420/2304 [I 2022-10-09 23:30:27] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:33 [I 2022-10-09 23:30:27] ...trap/engines/engine.py.229: process: 0.86226 | load: 0.00307 [I 2022-10-09 23:30:27] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-09 23:30:27] ...optimizers/trijoint.py.151: total_norm: 10.939922 [I 2022-10-09 23:30:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 1430/2304 [I 2022-10-09 23:30:36] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:27 [I 2022-10-09 23:30:36] ...trap/engines/engine.py.229: process: 0.85848 | load: 0.00312 [I 2022-10-09 23:30:36] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-09 23:30:36] ...optimizers/trijoint.py.151: total_norm: 9.447105 [I 2022-10-09 23:30:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1440/2304 [I 2022-10-09 23:30:44] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:21 [I 2022-10-09 23:30:44] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00308 [I 2022-10-09 23:30:44] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-09 23:30:44] ...optimizers/trijoint.py.151: total_norm: 15.172608 [I 2022-10-09 23:30:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1450/2304 [I 2022-10-09 23:30:53] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:12:08 [I 2022-10-09 23:30:53] ...trap/engines/engine.py.229: process: 0.85973 | load: 0.00315 [I 2022-10-09 23:30:53] ...trap/engines/engine.py.230: loss: 0.14443 [I 2022-10-09 23:30:53] ...optimizers/trijoint.py.151: total_norm: 11.390924 [I 2022-10-09 23:31:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1460/2304 [I 2022-10-09 23:31:01] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:12:00 [I 2022-10-09 23:31:01] ...trap/engines/engine.py.229: process: 0.85030 | load: 0.00313 [I 2022-10-09 23:31:01] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-09 23:31:01] ...optimizers/trijoint.py.151: total_norm: 9.179535 [I 2022-10-09 23:31:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1470/2304 [I 2022-10-09 23:31:10] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:45 [I 2022-10-09 23:31:10] ...trap/engines/engine.py.229: process: 0.83264 | load: 0.00313 [I 2022-10-09 23:31:10] ...trap/engines/engine.py.230: loss: 0.11126 [I 2022-10-09 23:31:10] ...optimizers/trijoint.py.151: total_norm: 7.511866 [I 2022-10-09 23:31:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1480/2304 [I 2022-10-09 23:31:18] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:39 [I 2022-10-09 23:31:18] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00307 [I 2022-10-09 23:31:18] ...trap/engines/engine.py.230: loss: 0.21540 [I 2022-10-09 23:31:18] ...optimizers/trijoint.py.151: total_norm: 18.318827 [I 2022-10-09 23:31:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1490/2304 [I 2022-10-09 23:31:27] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:27 [I 2022-10-09 23:31:27] ...trap/engines/engine.py.229: process: 0.85806 | load: 0.00308 [I 2022-10-09 23:31:27] ...trap/engines/engine.py.230: loss: 0.15788 [I 2022-10-09 23:31:27] ...optimizers/trijoint.py.151: total_norm: 18.050184 [I 2022-10-09 23:31:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 1500/2304 [I 2022-10-09 23:31:35] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:12 [I 2022-10-09 23:31:35] ...trap/engines/engine.py.229: process: 0.83163 | load: 0.00300 [I 2022-10-09 23:31:35] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-09 23:31:35] ...optimizers/trijoint.py.151: total_norm: 19.864283 [I 2022-10-09 23:31:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1510/2304 [I 2022-10-09 23:31:44] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:05 [I 2022-10-09 23:31:44] ...trap/engines/engine.py.229: process: 0.83108 | load: 0.00293 [I 2022-10-09 23:31:44] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-09 23:31:44] ...optimizers/trijoint.py.151: total_norm: 16.111029 [I 2022-10-09 23:31:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 1520/2304 [I 2022-10-09 23:31:52] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:57 [I 2022-10-09 23:31:52] ...trap/engines/engine.py.229: process: 0.82726 | load: 0.00292 [I 2022-10-09 23:31:52] ...trap/engines/engine.py.230: loss: 0.16549 [I 2022-10-09 23:31:52] ...optimizers/trijoint.py.151: total_norm: 107.263481 [I 2022-10-09 23:32:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1530/2304 [I 2022-10-09 23:32:01] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:54 [I 2022-10-09 23:32:01] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00293 [I 2022-10-09 23:32:01] ...trap/engines/engine.py.230: loss: 0.16797 [I 2022-10-09 23:32:01] ...optimizers/trijoint.py.151: total_norm: 21.229755 [I 2022-10-09 23:32:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 1540/2304 [I 2022-10-09 23:32:09] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:49 [I 2022-10-09 23:32:09] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00291 [I 2022-10-09 23:32:09] ...trap/engines/engine.py.230: loss: 0.10388 [I 2022-10-09 23:32:09] ...optimizers/trijoint.py.151: total_norm: 10.758785 [I 2022-10-09 23:32:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1550/2304 [I 2022-10-09 23:32:18] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:33 [I 2022-10-09 23:32:18] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00296 [I 2022-10-09 23:32:18] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-09 23:32:18] ...optimizers/trijoint.py.151: total_norm: 8.611012 [I 2022-10-09 23:32:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 1560/2304 [I 2022-10-09 23:32:26] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:26 [I 2022-10-09 23:32:26] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00305 [I 2022-10-09 23:32:26] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-09 23:32:26] ...optimizers/trijoint.py.151: total_norm: 16.167130 [I 2022-10-09 23:32:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 1570/2304 [I 2022-10-09 23:32:34] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:18 [I 2022-10-09 23:32:34] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00304 [I 2022-10-09 23:32:34] ...trap/engines/engine.py.230: loss: 0.16701 [I 2022-10-09 23:32:34] ...optimizers/trijoint.py.151: total_norm: 13.195008 [I 2022-10-09 23:32:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 1580/2304 [I 2022-10-09 23:32:43] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:09 [I 2022-10-09 23:32:43] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00297 [I 2022-10-09 23:32:43] ...trap/engines/engine.py.230: loss: 0.12989 [I 2022-10-09 23:32:43] ...optimizers/trijoint.py.151: total_norm: 17.436266 [I 2022-10-09 23:32:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 1590/2304 [I 2022-10-09 23:32:51] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:05 [I 2022-10-09 23:32:51] ...trap/engines/engine.py.229: process: 0.84448 | load: 0.00295 [I 2022-10-09 23:32:51] ...trap/engines/engine.py.230: loss: 0.25776 [I 2022-10-09 23:32:51] ...optimizers/trijoint.py.151: total_norm: 10.400966 [I 2022-10-09 23:33:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 1600/2304 [I 2022-10-09 23:33:00] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:52 [I 2022-10-09 23:33:00] ...trap/engines/engine.py.229: process: 0.84344 | load: 0.00296 [I 2022-10-09 23:33:00] ...trap/engines/engine.py.230: loss: 0.12331 [I 2022-10-09 23:33:00] ...optimizers/trijoint.py.151: total_norm: 14.564825 [I 2022-10-09 23:33:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 1610/2304 [I 2022-10-09 23:33:08] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:45 [I 2022-10-09 23:33:08] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00293 [I 2022-10-09 23:33:08] ...trap/engines/engine.py.230: loss: 0.20063 [I 2022-10-09 23:33:08] ...optimizers/trijoint.py.151: total_norm: 39.371536 [I 2022-10-09 23:33:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 1620/2304 [I 2022-10-09 23:33:17] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:38 [I 2022-10-09 23:33:17] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00296 [I 2022-10-09 23:33:17] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-09 23:33:17] ...optimizers/trijoint.py.151: total_norm: 13.186580 [I 2022-10-09 23:33:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 1630/2304 [I 2022-10-09 23:33:25] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:30 [I 2022-10-09 23:33:25] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00300 [I 2022-10-09 23:33:25] ...trap/engines/engine.py.230: loss: 0.13850 [I 2022-10-09 23:33:25] ...optimizers/trijoint.py.151: total_norm: 14.080535 [I 2022-10-09 23:33:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 1640/2304 [I 2022-10-09 23:33:34] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:20 [I 2022-10-09 23:33:34] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00297 [I 2022-10-09 23:33:34] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-09 23:33:34] ...optimizers/trijoint.py.151: total_norm: 12.851077 [I 2022-10-09 23:33:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 1650/2304 [I 2022-10-09 23:33:42] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:15 [I 2022-10-09 23:33:42] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00296 [I 2022-10-09 23:33:42] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-09 23:33:42] ...optimizers/trijoint.py.151: total_norm: 12.730948 [I 2022-10-09 23:33:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 1660/2304 [I 2022-10-09 23:33:51] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:05 [I 2022-10-09 23:33:51] ...trap/engines/engine.py.229: process: 0.84711 | load: 0.00301 [I 2022-10-09 23:33:51] ...trap/engines/engine.py.230: loss: 0.15992 [I 2022-10-09 23:33:51] ...optimizers/trijoint.py.151: total_norm: 16.467989 [I 2022-10-09 23:33:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 1670/2304 [I 2022-10-09 23:33:59] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:51 [I 2022-10-09 23:33:59] ...trap/engines/engine.py.229: process: 0.81520 | load: 0.00296 [I 2022-10-09 23:33:59] ...trap/engines/engine.py.230: loss: 0.19530 [I 2022-10-09 23:33:59] ...optimizers/trijoint.py.151: total_norm: 41.116516 [I 2022-10-09 23:34:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 1680/2304 [I 2022-10-09 23:34:07] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:49 [I 2022-10-09 23:34:07] ...trap/engines/engine.py.229: process: 0.86277 | load: 0.00297 [I 2022-10-09 23:34:07] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-09 23:34:07] ...optimizers/trijoint.py.151: total_norm: 12.767361 [I 2022-10-09 23:34:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 1690/2304 [I 2022-10-09 23:34:16] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:39 [I 2022-10-09 23:34:16] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00298 [I 2022-10-09 23:34:16] ...trap/engines/engine.py.230: loss: 0.12426 [I 2022-10-09 23:34:16] ...optimizers/trijoint.py.151: total_norm: 11.236615 [I 2022-10-09 23:34:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 1700/2304 [I 2022-10-09 23:34:24] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:31 [I 2022-10-09 23:34:24] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00313 [I 2022-10-09 23:34:24] ...trap/engines/engine.py.230: loss: 0.22395 [I 2022-10-09 23:34:24] ...optimizers/trijoint.py.151: total_norm: 69.590057 [I 2022-10-09 23:34:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 1710/2304 [I 2022-10-09 23:34:33] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:27 [I 2022-10-09 23:34:33] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00315 [I 2022-10-09 23:34:33] ...trap/engines/engine.py.230: loss: 0.20502 [I 2022-10-09 23:34:33] ...optimizers/trijoint.py.151: total_norm: 22.804197 [I 2022-10-09 23:34:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 1720/2304 [I 2022-10-09 23:34:42] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:16 [I 2022-10-09 23:34:42] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00314 [I 2022-10-09 23:34:42] ...trap/engines/engine.py.230: loss: 0.11753 [I 2022-10-09 23:34:42] ...optimizers/trijoint.py.151: total_norm: 9.294951 [I 2022-10-09 23:34:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 1730/2304 [I 2022-10-09 23:34:50] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:06 [I 2022-10-09 23:34:50] ...trap/engines/engine.py.229: process: 0.85112 | load: 0.00311 [I 2022-10-09 23:34:50] ...trap/engines/engine.py.230: loss: 0.11070 [I 2022-10-09 23:34:50] ...optimizers/trijoint.py.151: total_norm: 10.659572 [I 2022-10-09 23:34:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 1740/2304 [I 2022-10-09 23:34:59] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:59 [I 2022-10-09 23:34:59] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00314 [I 2022-10-09 23:34:59] ...trap/engines/engine.py.230: loss: 0.17597 [I 2022-10-09 23:34:59] ...optimizers/trijoint.py.151: total_norm: 18.623417 [I 2022-10-09 23:35:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 1750/2304 [I 2022-10-09 23:35:07] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:50 [I 2022-10-09 23:35:07] ...trap/engines/engine.py.229: process: 0.86680 | load: 0.00311 [I 2022-10-09 23:35:07] ...trap/engines/engine.py.230: loss: 0.14255 [I 2022-10-09 23:35:07] ...optimizers/trijoint.py.151: total_norm: 93.538025 [I 2022-10-09 23:35:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 1760/2304 [I 2022-10-09 23:35:16] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:44 [I 2022-10-09 23:35:16] ...trap/engines/engine.py.229: process: 0.86822 | load: 0.00313 [I 2022-10-09 23:35:16] ...trap/engines/engine.py.230: loss: 0.13974 [I 2022-10-09 23:35:16] ...optimizers/trijoint.py.151: total_norm: 8.806373 [I 2022-10-09 23:35:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 1770/2304 [I 2022-10-09 23:35:24] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:36 [I 2022-10-09 23:35:24] ...trap/engines/engine.py.229: process: 0.85812 | load: 0.00311 [I 2022-10-09 23:35:24] ...trap/engines/engine.py.230: loss: 0.18047 [I 2022-10-09 23:35:24] ...optimizers/trijoint.py.151: total_norm: 15.789343 [I 2022-10-09 23:35:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 1780/2304 [I 2022-10-09 23:35:33] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:25 [I 2022-10-09 23:35:33] ...trap/engines/engine.py.229: process: 0.85160 | load: 0.00312 [I 2022-10-09 23:35:33] ...trap/engines/engine.py.230: loss: 0.13820 [I 2022-10-09 23:35:33] ...optimizers/trijoint.py.151: total_norm: 11.048184 [I 2022-10-09 23:35:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 1790/2304 [I 2022-10-09 23:35:41] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:17 [I 2022-10-09 23:35:41] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00315 [I 2022-10-09 23:35:41] ...trap/engines/engine.py.230: loss: 0.18171 [I 2022-10-09 23:35:41] ...optimizers/trijoint.py.151: total_norm: 16.684071 [I 2022-10-09 23:35:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 1800/2304 [I 2022-10-09 23:35:50] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:09 [I 2022-10-09 23:35:50] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00313 [I 2022-10-09 23:35:50] ...trap/engines/engine.py.230: loss: 0.16486 [I 2022-10-09 23:35:50] ...optimizers/trijoint.py.151: total_norm: 14.671058 [I 2022-10-09 23:35:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 1810/2304 [I 2022-10-09 23:35:58] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:07:01 [I 2022-10-09 23:35:58] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00313 [I 2022-10-09 23:35:58] ...trap/engines/engine.py.230: loss: 0.10732 [I 2022-10-09 23:35:58] ...optimizers/trijoint.py.151: total_norm: 13.448950 [I 2022-10-09 23:36:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 1820/2304 [I 2022-10-09 23:36:07] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:51 [I 2022-10-09 23:36:07] ...trap/engines/engine.py.229: process: 0.83107 | load: 0.00316 [I 2022-10-09 23:36:07] ...trap/engines/engine.py.230: loss: 0.16886 [I 2022-10-09 23:36:07] ...optimizers/trijoint.py.151: total_norm: 18.424313 [I 2022-10-09 23:36:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 1830/2304 [I 2022-10-09 23:36:15] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:43 [I 2022-10-09 23:36:15] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00314 [I 2022-10-09 23:36:15] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-09 23:36:15] ...optimizers/trijoint.py.151: total_norm: 11.740672 [I 2022-10-09 23:36:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 1840/2304 [I 2022-10-09 23:36:24] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:32 [I 2022-10-09 23:36:24] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00314 [I 2022-10-09 23:36:24] ...trap/engines/engine.py.230: loss: 0.15899 [I 2022-10-09 23:36:24] ...optimizers/trijoint.py.151: total_norm: 17.260649 [I 2022-10-09 23:36:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 1850/2304 [I 2022-10-09 23:36:32] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:24 [I 2022-10-09 23:36:32] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00313 [I 2022-10-09 23:36:32] ...trap/engines/engine.py.230: loss: 0.17209 [I 2022-10-09 23:36:32] ...optimizers/trijoint.py.151: total_norm: 25.515036 [I 2022-10-09 23:36:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 1860/2304 [I 2022-10-09 23:36:41] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:18 [I 2022-10-09 23:36:41] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00311 [I 2022-10-09 23:36:41] ...trap/engines/engine.py.230: loss: 0.20330 [I 2022-10-09 23:36:41] ...optimizers/trijoint.py.151: total_norm: 12.424220 [I 2022-10-09 23:36:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 1870/2304 [I 2022-10-09 23:36:49] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:09 [I 2022-10-09 23:36:49] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00312 [I 2022-10-09 23:36:49] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-09 23:36:49] ...optimizers/trijoint.py.151: total_norm: 10.857299 [I 2022-10-09 23:36:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 1880/2304 [I 2022-10-09 23:36:58] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:01 [I 2022-10-09 23:36:58] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00320 [I 2022-10-09 23:36:58] ...trap/engines/engine.py.230: loss: 0.16459 [I 2022-10-09 23:36:58] ...optimizers/trijoint.py.151: total_norm: 8.964071 [I 2022-10-09 23:37:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 1890/2304 [I 2022-10-09 23:37:06] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:53 [I 2022-10-09 23:37:06] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00313 [I 2022-10-09 23:37:06] ...trap/engines/engine.py.230: loss: 0.21441 [I 2022-10-09 23:37:06] ...optimizers/trijoint.py.151: total_norm: 18.944336 [I 2022-10-09 23:37:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 1900/2304 [I 2022-10-09 23:37:15] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:43 [I 2022-10-09 23:37:15] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00312 [I 2022-10-09 23:37:15] ...trap/engines/engine.py.230: loss: 0.12623 [I 2022-10-09 23:37:15] ...optimizers/trijoint.py.151: total_norm: 12.509258 [I 2022-10-09 23:37:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 1910/2304 [I 2022-10-09 23:37:23] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:34 [I 2022-10-09 23:37:23] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00313 [I 2022-10-09 23:37:23] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-09 23:37:23] ...optimizers/trijoint.py.151: total_norm: 12.614024 [I 2022-10-09 23:37:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 1920/2304 [I 2022-10-09 23:37:32] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:26 [I 2022-10-09 23:37:32] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00312 [I 2022-10-09 23:37:32] ...trap/engines/engine.py.230: loss: 0.18824 [I 2022-10-09 23:37:32] ...optimizers/trijoint.py.151: total_norm: 17.653715 [I 2022-10-09 23:37:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 1930/2304 [I 2022-10-09 23:37:40] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:17 [I 2022-10-09 23:37:40] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00312 [I 2022-10-09 23:37:40] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-09 23:37:40] ...optimizers/trijoint.py.151: total_norm: 12.509755 [I 2022-10-09 23:37:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 1940/2304 [I 2022-10-09 23:37:49] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:11 [I 2022-10-09 23:37:49] ...trap/engines/engine.py.229: process: 0.87054 | load: 0.00312 [I 2022-10-09 23:37:49] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-09 23:37:49] ...optimizers/trijoint.py.151: total_norm: 18.692749 [I 2022-10-09 23:37:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 1950/2304 [I 2022-10-09 23:37:57] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:00 [I 2022-10-09 23:37:57] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00309 [I 2022-10-09 23:37:57] ...trap/engines/engine.py.230: loss: 0.09681 [I 2022-10-09 23:37:57] ...optimizers/trijoint.py.151: total_norm: 7.472105 [I 2022-10-09 23:38:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 1960/2304 [I 2022-10-09 23:38:06] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:53 [I 2022-10-09 23:38:06] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00311 [I 2022-10-09 23:38:06] ...trap/engines/engine.py.230: loss: 0.23445 [I 2022-10-09 23:38:06] ...optimizers/trijoint.py.151: total_norm: 13.323128 [I 2022-10-09 23:38:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 1970/2304 [I 2022-10-09 23:38:14] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:44 [I 2022-10-09 23:38:14] ...trap/engines/engine.py.229: process: 0.86502 | load: 0.00307 [I 2022-10-09 23:38:14] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-09 23:38:14] ...optimizers/trijoint.py.151: total_norm: 19.166903 [I 2022-10-09 23:38:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 1980/2304 [I 2022-10-09 23:38:23] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:37 [I 2022-10-09 23:38:23] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00314 [I 2022-10-09 23:38:23] ...trap/engines/engine.py.230: loss: 0.17500 [I 2022-10-09 23:38:23] ...optimizers/trijoint.py.151: total_norm: 14.214946 [I 2022-10-09 23:38:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 1990/2304 [I 2022-10-09 23:38:32] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:27 [I 2022-10-09 23:38:32] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00311 [I 2022-10-09 23:38:32] ...trap/engines/engine.py.230: loss: 0.16202 [I 2022-10-09 23:38:32] ...optimizers/trijoint.py.151: total_norm: 24.281553 [I 2022-10-09 23:38:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 2000/2304 [I 2022-10-09 23:38:40] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:17 [I 2022-10-09 23:38:40] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00314 [I 2022-10-09 23:38:40] ...trap/engines/engine.py.230: loss: 0.12751 [I 2022-10-09 23:38:40] ...optimizers/trijoint.py.151: total_norm: 17.249767 [I 2022-10-09 23:38:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 2010/2304 [I 2022-10-09 23:38:49] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:09 [I 2022-10-09 23:38:49] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00317 [I 2022-10-09 23:38:49] ...trap/engines/engine.py.230: loss: 0.14840 [I 2022-10-09 23:38:49] ...optimizers/trijoint.py.151: total_norm: 12.999411 [I 2022-10-09 23:38:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 2020/2304 [I 2022-10-09 23:38:57] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:01 [I 2022-10-09 23:38:57] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00311 [I 2022-10-09 23:38:57] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-09 23:38:57] ...optimizers/trijoint.py.151: total_norm: 14.209832 [I 2022-10-09 23:39:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 2030/2304 [I 2022-10-09 23:39:06] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:53 [I 2022-10-09 23:39:06] ...trap/engines/engine.py.229: process: 0.85987 | load: 0.00307 [I 2022-10-09 23:39:06] ...trap/engines/engine.py.230: loss: 0.18779 [I 2022-10-09 23:39:06] ...optimizers/trijoint.py.151: total_norm: 17.600214 [I 2022-10-09 23:39:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 2040/2304 [I 2022-10-09 23:39:14] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:42 [I 2022-10-09 23:39:14] ...trap/engines/engine.py.229: process: 0.83475 | load: 0.00297 [I 2022-10-09 23:39:14] ...trap/engines/engine.py.230: loss: 0.17360 [I 2022-10-09 23:39:14] ...optimizers/trijoint.py.151: total_norm: 23.388493 [I 2022-10-09 23:39:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 2050/2304 [I 2022-10-09 23:39:23] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:34 [I 2022-10-09 23:39:23] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00299 [I 2022-10-09 23:39:23] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-09 23:39:23] ...optimizers/trijoint.py.151: total_norm: 33.042351 [I 2022-10-09 23:39:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 2060/2304 [I 2022-10-09 23:39:31] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:26 [I 2022-10-09 23:39:31] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00300 [I 2022-10-09 23:39:31] ...trap/engines/engine.py.230: loss: 0.18750 [I 2022-10-09 23:39:31] ...optimizers/trijoint.py.151: total_norm: 11.402071 [I 2022-10-09 23:39:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 2070/2304 [I 2022-10-09 23:39:39] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:16 [I 2022-10-09 23:39:39] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00300 [I 2022-10-09 23:39:39] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-09 23:39:39] ...optimizers/trijoint.py.151: total_norm: 7.276027 [I 2022-10-09 23:39:48] ...trap/engines/engine.py.224: train: epoch 58 | batch 2080/2304 [I 2022-10-09 23:39:48] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:08 [I 2022-10-09 23:39:48] ...trap/engines/engine.py.229: process: 0.83184 | load: 0.00292 [I 2022-10-09 23:39:48] ...trap/engines/engine.py.230: loss: 0.16500 [I 2022-10-09 23:39:48] ...optimizers/trijoint.py.151: total_norm: 23.990265 [I 2022-10-09 23:39:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 2090/2304 [I 2022-10-09 23:39:56] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:00 [I 2022-10-09 23:39:56] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00293 [I 2022-10-09 23:39:56] ...trap/engines/engine.py.230: loss: 0.10955 [I 2022-10-09 23:39:56] ...optimizers/trijoint.py.151: total_norm: 19.285984 [I 2022-10-09 23:40:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 2100/2304 [I 2022-10-09 23:40:05] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:52 [I 2022-10-09 23:40:05] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00310 [I 2022-10-09 23:40:05] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-09 23:40:05] ...optimizers/trijoint.py.151: total_norm: 15.613137 [I 2022-10-09 23:40:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 2110/2304 [I 2022-10-09 23:40:13] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:43 [I 2022-10-09 23:40:13] ...trap/engines/engine.py.229: process: 0.82606 | load: 0.00298 [I 2022-10-09 23:40:13] ...trap/engines/engine.py.230: loss: 0.15117 [I 2022-10-09 23:40:13] ...optimizers/trijoint.py.151: total_norm: 27.919659 [I 2022-10-09 23:40:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 2120/2304 [I 2022-10-09 23:40:22] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:34 [I 2022-10-09 23:40:22] ...trap/engines/engine.py.229: process: 0.83643 | load: 0.00298 [I 2022-10-09 23:40:22] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-09 23:40:22] ...optimizers/trijoint.py.151: total_norm: 19.425594 [I 2022-10-09 23:40:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 2130/2304 [I 2022-10-09 23:40:30] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:26 [I 2022-10-09 23:40:30] ...trap/engines/engine.py.229: process: 0.83484 | load: 0.00299 [I 2022-10-09 23:40:30] ...trap/engines/engine.py.230: loss: 0.18653 [I 2022-10-09 23:40:30] ...optimizers/trijoint.py.151: total_norm: 27.538773 [I 2022-10-09 23:40:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 2140/2304 [I 2022-10-09 23:40:39] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:18 [I 2022-10-09 23:40:39] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00301 [I 2022-10-09 23:40:39] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-09 23:40:39] ...optimizers/trijoint.py.151: total_norm: 24.562223 [I 2022-10-09 23:40:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 2150/2304 [I 2022-10-09 23:40:47] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:09 [I 2022-10-09 23:40:47] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00299 [I 2022-10-09 23:40:47] ...trap/engines/engine.py.230: loss: 0.08517 [I 2022-10-09 23:40:47] ...optimizers/trijoint.py.151: total_norm: 7.182449 [I 2022-10-09 23:40:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 2160/2304 [I 2022-10-09 23:40:56] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:01 [I 2022-10-09 23:40:56] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00306 [I 2022-10-09 23:40:56] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-09 23:40:56] ...optimizers/trijoint.py.151: total_norm: 33.881596 [I 2022-10-09 23:41:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 2170/2304 [I 2022-10-09 23:41:04] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:53 [I 2022-10-09 23:41:04] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00297 [I 2022-10-09 23:41:04] ...trap/engines/engine.py.230: loss: 0.18282 [I 2022-10-09 23:41:04] ...optimizers/trijoint.py.151: total_norm: 26.564394 [I 2022-10-09 23:41:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 2180/2304 [I 2022-10-09 23:41:12] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:43 [I 2022-10-09 23:41:12] ...trap/engines/engine.py.229: process: 0.84157 | load: 0.00303 [I 2022-10-09 23:41:12] ...trap/engines/engine.py.230: loss: 0.17158 [I 2022-10-09 23:41:12] ...optimizers/trijoint.py.151: total_norm: 17.282274 [I 2022-10-09 23:41:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 2190/2304 [I 2022-10-09 23:41:21] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:36 [I 2022-10-09 23:41:21] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00295 [I 2022-10-09 23:41:21] ...trap/engines/engine.py.230: loss: 0.16086 [I 2022-10-09 23:41:21] ...optimizers/trijoint.py.151: total_norm: 22.478580 [I 2022-10-09 23:41:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 2200/2304 [I 2022-10-09 23:41:29] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:27 [I 2022-10-09 23:41:29] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00297 [I 2022-10-09 23:41:29] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-09 23:41:29] ...optimizers/trijoint.py.151: total_norm: 61.404381 [I 2022-10-09 23:41:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 2210/2304 [I 2022-10-09 23:41:38] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:19 [I 2022-10-09 23:41:38] ...trap/engines/engine.py.229: process: 0.83768 | load: 0.00312 [I 2022-10-09 23:41:38] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-09 23:41:38] ...optimizers/trijoint.py.151: total_norm: 12.167994 [I 2022-10-09 23:41:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 2220/2304 [I 2022-10-09 23:41:46] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:11 [I 2022-10-09 23:41:46] ...trap/engines/engine.py.229: process: 0.85401 | load: 0.00313 [I 2022-10-09 23:41:46] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-09 23:41:46] ...optimizers/trijoint.py.151: total_norm: 11.677933 [I 2022-10-09 23:41:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 2230/2304 [I 2022-10-09 23:41:55] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:03 [I 2022-10-09 23:41:55] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00309 [I 2022-10-09 23:41:55] ...trap/engines/engine.py.230: loss: 0.13609 [I 2022-10-09 23:41:55] ...optimizers/trijoint.py.151: total_norm: 11.206621 [I 2022-10-09 23:42:03] ...trap/engines/engine.py.224: train: epoch 58 | batch 2240/2304 [I 2022-10-09 23:42:03] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:00:54 [I 2022-10-09 23:42:03] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00315 [I 2022-10-09 23:42:03] ...trap/engines/engine.py.230: loss: 0.17948 [I 2022-10-09 23:42:03] ...optimizers/trijoint.py.151: total_norm: 20.393309 [I 2022-10-09 23:42:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 2250/2304 [I 2022-10-09 23:42:12] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:46 [I 2022-10-09 23:42:12] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00317 [I 2022-10-09 23:42:12] ...trap/engines/engine.py.230: loss: 0.26280 [I 2022-10-09 23:42:12] ...optimizers/trijoint.py.151: total_norm: 24.290592 [I 2022-10-09 23:42:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 2260/2304 [I 2022-10-09 23:42:20] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:37 [I 2022-10-09 23:42:20] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00312 [I 2022-10-09 23:42:20] ...trap/engines/engine.py.230: loss: 0.12746 [I 2022-10-09 23:42:20] ...optimizers/trijoint.py.151: total_norm: 6.560158 [I 2022-10-09 23:42:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 2270/2304 [I 2022-10-09 23:42:29] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:28 [I 2022-10-09 23:42:29] ...trap/engines/engine.py.229: process: 0.83579 | load: 0.00312 [I 2022-10-09 23:42:29] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-09 23:42:29] ...optimizers/trijoint.py.151: total_norm: 11.718303 [I 2022-10-09 23:42:37] ...trap/engines/engine.py.224: train: epoch 58 | batch 2280/2304 [I 2022-10-09 23:42:37] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:20 [I 2022-10-09 23:42:37] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00311 [I 2022-10-09 23:42:37] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-09 23:42:37] ...optimizers/trijoint.py.151: total_norm: 11.351737 [I 2022-10-09 23:42:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 2290/2304 [I 2022-10-09 23:42:46] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:11 [I 2022-10-09 23:42:46] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00310 [I 2022-10-09 23:42:46] ...trap/engines/engine.py.230: loss: 0.11334 [I 2022-10-09 23:42:46] ...optimizers/trijoint.py.151: total_norm: 10.838681 [I 2022-10-09 23:42:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 2300/2304 [I 2022-10-09 23:42:55] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:03 [I 2022-10-09 23:42:55] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00020 [I 2022-10-09 23:42:55] ...trap/engines/engine.py.230: loss: 0.12121 [I 2022-10-09 23:42:55] ...optimizers/trijoint.py.151: total_norm: 34.101044 [I 2022-10-09 23:42:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 2304/2304 [I 2022-10-09 23:42:58] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:00 [I 2022-10-09 23:42:58] ...trap/engines/engine.py.229: process: 0.83304 | load: 0.00020 [I 2022-10-09 23:42:58] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-09 23:42:58] ...optimizers/trijoint.py.151: total_norm: 9.001487 [S 2022-10-09 23:42:59] ...trap/engines/engine.py.129: train_epoch.epoch: 58 [S 2022-10-09 23:42:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.15026345377759665 [I 2022-10-09 23:42:59] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-09 23:43:01] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 58 [W 2022-10-09 23:43:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 23:43:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 23:43:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 23:43:01] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [I 2022-10-09 23:43:05] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 23:43:05] ...trap/engines/engine.py.321: eval: epoch 58 | batch 0/510 [I 2022-10-09 23:43:05] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:23 [I 2022-10-09 23:43:05] ...trap/engines/engine.py.326: process: 4.87039 | load: 3.89362 [I 2022-10-09 23:43:10] ...trap/engines/engine.py.321: eval: epoch 58 | batch 10/510 [I 2022-10-09 23:43:10] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:40 [I 2022-10-09 23:43:10] ...trap/engines/engine.py.326: process: 0.44710 | load: 0.00019 [I 2022-10-09 23:43:14] ...trap/engines/engine.py.321: eval: epoch 58 | batch 20/510 [I 2022-10-09 23:43:14] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:07 [I 2022-10-09 23:43:14] ...trap/engines/engine.py.326: process: 0.45928 | load: 0.00020 [I 2022-10-09 23:43:19] ...trap/engines/engine.py.321: eval: epoch 58 | batch 30/510 [I 2022-10-09 23:43:19] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:22 [I 2022-10-09 23:43:19] ...trap/engines/engine.py.326: process: 0.68146 | load: 0.00020 [I 2022-10-09 23:43:26] ...trap/engines/engine.py.321: eval: epoch 58 | batch 40/510 [I 2022-10-09 23:43:26] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:45 [I 2022-10-09 23:43:26] ...trap/engines/engine.py.326: process: 0.49112 | load: 0.00023 [I 2022-10-09 23:43:31] ...trap/engines/engine.py.321: eval: epoch 58 | batch 50/510 [I 2022-10-09 23:43:31] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:00 [I 2022-10-09 23:43:31] ...trap/engines/engine.py.326: process: 0.54669 | load: 0.00020 [I 2022-10-09 23:43:37] ...trap/engines/engine.py.321: eval: epoch 58 | batch 60/510 [I 2022-10-09 23:43:37] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:24 [I 2022-10-09 23:43:37] ...trap/engines/engine.py.326: process: 0.57083 | load: 0.00022 [I 2022-10-09 23:43:43] ...trap/engines/engine.py.321: eval: epoch 58 | batch 70/510 [I 2022-10-09 23:43:43] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:10 [I 2022-10-09 23:43:43] ...trap/engines/engine.py.326: process: 0.66088 | load: 0.00019 [I 2022-10-09 23:43:48] ...trap/engines/engine.py.321: eval: epoch 58 | batch 80/510 [I 2022-10-09 23:43:48] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:56 [I 2022-10-09 23:43:48] ...trap/engines/engine.py.326: process: 0.52329 | load: 0.00020 [I 2022-10-09 23:43:53] ...trap/engines/engine.py.321: eval: epoch 58 | batch 90/510 [I 2022-10-09 23:43:53] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:20 [I 2022-10-09 23:43:53] ...trap/engines/engine.py.326: process: 0.41621 | load: 0.00022 [I 2022-10-09 23:43:57] ...trap/engines/engine.py.321: eval: epoch 58 | batch 100/510 [I 2022-10-09 23:43:57] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:55 [I 2022-10-09 23:43:57] ...trap/engines/engine.py.326: process: 0.30598 | load: 0.00020 [I 2022-10-09 23:44:01] ...trap/engines/engine.py.321: eval: epoch 58 | batch 110/510 [I 2022-10-09 23:44:01] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:12 [I 2022-10-09 23:44:01] ...trap/engines/engine.py.326: process: 0.32329 | load: 0.00021 [I 2022-10-09 23:44:04] ...trap/engines/engine.py.321: eval: epoch 58 | batch 120/510 [I 2022-10-09 23:44:04] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:05 [I 2022-10-09 23:44:04] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00021 [I 2022-10-09 23:44:07] ...trap/engines/engine.py.321: eval: epoch 58 | batch 130/510 [I 2022-10-09 23:44:07] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:01 [I 2022-10-09 23:44:07] ...trap/engines/engine.py.326: process: 0.31238 | load: 0.00021 [I 2022-10-09 23:44:10] ...trap/engines/engine.py.321: eval: epoch 58 | batch 140/510 [I 2022-10-09 23:44:10] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:58 [I 2022-10-09 23:44:10] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00019 [I 2022-10-09 23:44:14] ...trap/engines/engine.py.321: eval: epoch 58 | batch 150/510 [I 2022-10-09 23:44:14] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:53 [I 2022-10-09 23:44:14] ...trap/engines/engine.py.326: process: 0.30910 | load: 0.00020 [I 2022-10-09 23:44:17] ...trap/engines/engine.py.321: eval: epoch 58 | batch 160/510 [I 2022-10-09 23:44:17] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:51 [I 2022-10-09 23:44:17] ...trap/engines/engine.py.326: process: 0.31519 | load: 0.00021 [I 2022-10-09 23:44:20] ...trap/engines/engine.py.321: eval: epoch 58 | batch 170/510 [I 2022-10-09 23:44:20] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-09 23:44:20] ...trap/engines/engine.py.326: process: 0.31891 | load: 0.00020 [I 2022-10-09 23:44:23] ...trap/engines/engine.py.321: eval: epoch 58 | batch 180/510 [I 2022-10-09 23:44:23] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:46 [I 2022-10-09 23:44:23] ...trap/engines/engine.py.326: process: 0.32786 | load: 0.00020 [I 2022-10-09 23:44:26] ...trap/engines/engine.py.321: eval: epoch 58 | batch 190/510 [I 2022-10-09 23:44:26] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:42 [I 2022-10-09 23:44:26] ...trap/engines/engine.py.326: process: 0.31435 | load: 0.00019 [I 2022-10-09 23:44:30] ...trap/engines/engine.py.321: eval: epoch 58 | batch 200/510 [I 2022-10-09 23:44:30] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-09 23:44:30] ...trap/engines/engine.py.326: process: 0.32134 | load: 0.00020 [I 2022-10-09 23:44:33] ...trap/engines/engine.py.321: eval: epoch 58 | batch 210/510 [I 2022-10-09 23:44:33] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:36 [I 2022-10-09 23:44:33] ...trap/engines/engine.py.326: process: 0.32421 | load: 0.00020 [I 2022-10-09 23:44:36] ...trap/engines/engine.py.321: eval: epoch 58 | batch 220/510 [I 2022-10-09 23:44:36] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:33 [I 2022-10-09 23:44:36] ...trap/engines/engine.py.326: process: 0.32210 | load: 0.00019 [I 2022-10-09 23:44:39] ...trap/engines/engine.py.321: eval: epoch 58 | batch 230/510 [I 2022-10-09 23:44:39] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:30 [I 2022-10-09 23:44:39] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00018 [I 2022-10-09 23:44:42] ...trap/engines/engine.py.321: eval: epoch 58 | batch 240/510 [I 2022-10-09 23:44:42] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-09 23:44:42] ...trap/engines/engine.py.326: process: 0.32349 | load: 0.00020 [I 2022-10-09 23:44:46] ...trap/engines/engine.py.321: eval: epoch 58 | batch 250/510 [I 2022-10-09 23:44:46] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:24 [I 2022-10-09 23:44:46] ...trap/engines/engine.py.326: process: 0.32587 | load: 0.00021 [I 2022-10-09 23:44:49] ...trap/engines/engine.py.321: eval: epoch 58 | batch 260/510 [I 2022-10-09 23:44:49] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:20 [I 2022-10-09 23:44:49] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00020 [I 2022-10-09 23:44:52] ...trap/engines/engine.py.321: eval: epoch 58 | batch 270/510 [I 2022-10-09 23:44:52] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:17 [I 2022-10-09 23:44:52] ...trap/engines/engine.py.326: process: 0.32263 | load: 0.00020 [I 2022-10-09 23:44:55] ...trap/engines/engine.py.321: eval: epoch 58 | batch 280/510 [I 2022-10-09 23:44:55] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-09 23:44:55] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00021 [I 2022-10-09 23:44:59] ...trap/engines/engine.py.321: eval: epoch 58 | batch 290/510 [I 2022-10-09 23:44:59] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:10 [I 2022-10-09 23:44:59] ...trap/engines/engine.py.326: process: 0.31526 | load: 0.00024 [I 2022-10-09 23:45:02] ...trap/engines/engine.py.321: eval: epoch 58 | batch 300/510 [I 2022-10-09 23:45:02] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:07 [I 2022-10-09 23:45:02] ...trap/engines/engine.py.326: process: 0.32356 | load: 0.00022 [I 2022-10-09 23:45:05] ...trap/engines/engine.py.321: eval: epoch 58 | batch 310/510 [I 2022-10-09 23:45:05] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:04 [I 2022-10-09 23:45:05] ...trap/engines/engine.py.326: process: 0.32436 | load: 0.00020 [I 2022-10-09 23:45:08] ...trap/engines/engine.py.321: eval: epoch 58 | batch 320/510 [I 2022-10-09 23:45:08] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-09 23:45:08] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00021 [I 2022-10-09 23:45:11] ...trap/engines/engine.py.321: eval: epoch 58 | batch 330/510 [I 2022-10-09 23:45:11] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-09 23:45:11] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00021 [I 2022-10-09 23:45:15] ...trap/engines/engine.py.321: eval: epoch 58 | batch 340/510 [I 2022-10-09 23:45:15] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:54 [I 2022-10-09 23:45:15] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00019 [I 2022-10-09 23:45:18] ...trap/engines/engine.py.321: eval: epoch 58 | batch 350/510 [I 2022-10-09 23:45:18] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-09 23:45:18] ...trap/engines/engine.py.326: process: 0.31573 | load: 0.00021 [I 2022-10-09 23:45:21] ...trap/engines/engine.py.321: eval: epoch 58 | batch 360/510 [I 2022-10-09 23:45:21] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:48 [I 2022-10-09 23:45:21] ...trap/engines/engine.py.326: process: 0.32387 | load: 0.00021 [I 2022-10-09 23:45:24] ...trap/engines/engine.py.321: eval: epoch 58 | batch 370/510 [I 2022-10-09 23:45:24] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:45 [I 2022-10-09 23:45:24] ...trap/engines/engine.py.326: process: 0.32560 | load: 0.00021 [I 2022-10-09 23:45:27] ...trap/engines/engine.py.321: eval: epoch 58 | batch 380/510 [I 2022-10-09 23:45:27] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:42 [I 2022-10-09 23:45:27] ...trap/engines/engine.py.326: process: 0.33023 | load: 0.00021 [I 2022-10-09 23:45:31] ...trap/engines/engine.py.321: eval: epoch 58 | batch 390/510 [I 2022-10-09 23:45:31] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:38 [I 2022-10-09 23:45:31] ...trap/engines/engine.py.326: process: 0.31517 | load: 0.00020 [I 2022-10-09 23:45:34] ...trap/engines/engine.py.321: eval: epoch 58 | batch 400/510 [I 2022-10-09 23:45:34] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:35 [I 2022-10-09 23:45:34] ...trap/engines/engine.py.326: process: 0.32322 | load: 0.00021 [I 2022-10-09 23:45:37] ...trap/engines/engine.py.321: eval: epoch 58 | batch 410/510 [I 2022-10-09 23:45:37] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:31 [I 2022-10-09 23:45:37] ...trap/engines/engine.py.326: process: 0.31965 | load: 0.00020 [I 2022-10-09 23:45:40] ...trap/engines/engine.py.321: eval: epoch 58 | batch 420/510 [I 2022-10-09 23:45:40] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:29 [I 2022-10-09 23:45:40] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00021 [I 2022-10-09 23:45:44] ...trap/engines/engine.py.321: eval: epoch 58 | batch 430/510 [I 2022-10-09 23:45:44] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:25 [I 2022-10-09 23:45:44] ...trap/engines/engine.py.326: process: 0.32259 | load: 0.00022 [I 2022-10-09 23:45:47] ...trap/engines/engine.py.321: eval: epoch 58 | batch 440/510 [I 2022-10-09 23:45:47] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:22 [I 2022-10-09 23:45:47] ...trap/engines/engine.py.326: process: 0.32315 | load: 0.00021 [I 2022-10-09 23:45:50] ...trap/engines/engine.py.321: eval: epoch 58 | batch 450/510 [I 2022-10-09 23:45:50] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-09 23:45:50] ...trap/engines/engine.py.326: process: 0.32543 | load: 0.00021 [I 2022-10-09 23:45:53] ...trap/engines/engine.py.321: eval: epoch 58 | batch 460/510 [I 2022-10-09 23:45:53] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:16 [I 2022-10-09 23:45:53] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00022 [I 2022-10-09 23:45:56] ...trap/engines/engine.py.321: eval: epoch 58 | batch 470/510 [I 2022-10-09 23:45:56] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-09 23:45:56] ...trap/engines/engine.py.326: process: 0.32229 | load: 0.00020 [I 2022-10-09 23:46:00] ...trap/engines/engine.py.321: eval: epoch 58 | batch 480/510 [I 2022-10-09 23:46:00] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:09 [I 2022-10-09 23:46:00] ...trap/engines/engine.py.326: process: 0.32553 | load: 0.00021 [I 2022-10-09 23:46:03] ...trap/engines/engine.py.321: eval: epoch 58 | batch 490/510 [I 2022-10-09 23:46:03] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:06 [I 2022-10-09 23:46:03] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00021 [I 2022-10-09 23:46:06] ...trap/engines/engine.py.321: eval: epoch 58 | batch 500/510 [I 2022-10-09 23:46:06] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:03 [I 2022-10-09 23:46:06] ...trap/engines/engine.py.326: process: 0.32471 | load: 0.00020 [I 2022-10-09 23:46:09] ...trap/engines/engine.py.321: eval: epoch 58 | batch 510/510 [I 2022-10-09 23:46:09] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-09 23:46:09] ...trap/engines/engine.py.326: process: 0.31433 | load: 0.00017 [S 2022-10-09 23:46:10] ...trap/engines/engine.py.132: eval_epoch.epoch: 58 [I 2022-10-09 23:46:10] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740300 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908500 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939700 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.728500 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906900 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936000 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008967 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007940 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007457 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.006087 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010044 [S 2022-10-09 23:46:32] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008087 [I 2022-10-09 23:46:46] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-09 23:46:46] ...trap/engines/engine.py.470: Saving model... [W 2022-10-09 23:46:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-09 23:46:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-09 23:46:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-09 23:46:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 23:46:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-09 23:46:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 23:46:57] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-09 23:46:57] ...trap/engines/engine.py.470: Saving model... [I 2022-10-09 23:46:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-09 23:47:05] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-09 23:47:05] ...trap/engines/engine.py.163: Training model on trainset for epoch 59 [I 2022-10-09 23:47:06] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-09 23:47:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 0/2304 [I 2022-10-09 23:47:08] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:51:53 [I 2022-10-09 23:47:08] ...trap/engines/engine.py.229: process: 2.91384 | load: 2.04805 [I 2022-10-09 23:47:08] ...trap/engines/engine.py.230: loss: 0.16325 [I 2022-10-09 23:47:08] ...optimizers/trijoint.py.151: total_norm: 11.454114 [I 2022-10-09 23:47:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 10/2304 [I 2022-10-09 23:47:17] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:46 [I 2022-10-09 23:47:17] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00626 [I 2022-10-09 23:47:17] ...trap/engines/engine.py.230: loss: 0.19171 [I 2022-10-09 23:47:17] ...optimizers/trijoint.py.151: total_norm: 40.809238 [I 2022-10-09 23:47:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 20/2304 [I 2022-10-09 23:47:25] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:52 [I 2022-10-09 23:47:25] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00424 [I 2022-10-09 23:47:25] ...trap/engines/engine.py.230: loss: 0.26089 [I 2022-10-09 23:47:25] ...optimizers/trijoint.py.151: total_norm: 28.138639 [I 2022-10-09 23:47:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 30/2304 [I 2022-10-09 23:47:34] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:49 [I 2022-10-09 23:47:34] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00458 [I 2022-10-09 23:47:34] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-09 23:47:34] ...optimizers/trijoint.py.151: total_norm: 19.398409 [I 2022-10-09 23:47:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 40/2304 [I 2022-10-09 23:47:42] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:41 [I 2022-10-09 23:47:42] ...trap/engines/engine.py.229: process: 0.83551 | load: 0.00393 [I 2022-10-09 23:47:42] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-09 23:47:42] ...optimizers/trijoint.py.151: total_norm: 14.858721 [I 2022-10-09 23:47:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 50/2304 [I 2022-10-09 23:47:50] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:35 [I 2022-10-09 23:47:50] ...trap/engines/engine.py.229: process: 0.83144 | load: 0.00366 [I 2022-10-09 23:47:50] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-09 23:47:50] ...optimizers/trijoint.py.151: total_norm: 49.042610 [I 2022-10-09 23:47:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 60/2304 [I 2022-10-09 23:47:59] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:44 [I 2022-10-09 23:47:59] ...trap/engines/engine.py.229: process: 0.85024 | load: 0.00358 [I 2022-10-09 23:47:59] ...trap/engines/engine.py.230: loss: 0.18833 [I 2022-10-09 23:47:59] ...optimizers/trijoint.py.151: total_norm: 16.283234 [I 2022-10-09 23:48:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 70/2304 [I 2022-10-09 23:48:07] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:23 [I 2022-10-09 23:48:07] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00324 [I 2022-10-09 23:48:07] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-09 23:48:07] ...optimizers/trijoint.py.151: total_norm: 19.319721 [I 2022-10-09 23:48:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 80/2304 [I 2022-10-09 23:48:16] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:26 [I 2022-10-09 23:48:16] ...trap/engines/engine.py.229: process: 0.85038 | load: 0.00343 [I 2022-10-09 23:48:16] ...trap/engines/engine.py.230: loss: 0.13298 [I 2022-10-09 23:48:16] ...optimizers/trijoint.py.151: total_norm: 17.052334 [I 2022-10-09 23:48:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 90/2304 [I 2022-10-09 23:48:25] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:29 [I 2022-10-09 23:48:25] ...trap/engines/engine.py.229: process: 0.87023 | load: 0.00315 [I 2022-10-09 23:48:25] ...trap/engines/engine.py.230: loss: 0.16638 [I 2022-10-09 23:48:25] ...optimizers/trijoint.py.151: total_norm: 16.389294 [I 2022-10-09 23:48:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 100/2304 [I 2022-10-09 23:48:33] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:13 [I 2022-10-09 23:48:33] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00353 [I 2022-10-09 23:48:33] ...trap/engines/engine.py.230: loss: 0.13845 [I 2022-10-09 23:48:33] ...optimizers/trijoint.py.151: total_norm: 8.883428 [I 2022-10-09 23:48:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 110/2304 [I 2022-10-09 23:48:42] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:57 [I 2022-10-09 23:48:42] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00321 [I 2022-10-09 23:48:42] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-09 23:48:42] ...optimizers/trijoint.py.151: total_norm: 14.378066 [I 2022-10-09 23:48:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 120/2304 [I 2022-10-09 23:48:50] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:40 [I 2022-10-09 23:48:50] ...trap/engines/engine.py.229: process: 0.83134 | load: 0.00329 [I 2022-10-09 23:48:50] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-09 23:48:50] ...optimizers/trijoint.py.151: total_norm: 7.001575 [I 2022-10-09 23:48:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 130/2304 [I 2022-10-09 23:48:58] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:37 [I 2022-10-09 23:48:58] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00315 [I 2022-10-09 23:48:58] ...trap/engines/engine.py.230: loss: 0.18265 [I 2022-10-09 23:48:58] ...optimizers/trijoint.py.151: total_norm: 21.233475 [I 2022-10-09 23:49:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 140/2304 [I 2022-10-09 23:49:07] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:20 [I 2022-10-09 23:49:07] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00338 [I 2022-10-09 23:49:07] ...trap/engines/engine.py.230: loss: 0.25173 [I 2022-10-09 23:49:07] ...optimizers/trijoint.py.151: total_norm: 28.971735 [I 2022-10-09 23:49:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 150/2304 [I 2022-10-09 23:49:15] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:16 [I 2022-10-09 23:49:15] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00325 [I 2022-10-09 23:49:15] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-09 23:49:15] ...optimizers/trijoint.py.151: total_norm: 9.327244 [I 2022-10-09 23:49:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 160/2304 [I 2022-10-09 23:49:24] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:07 [I 2022-10-09 23:49:24] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00353 [I 2022-10-09 23:49:24] ...trap/engines/engine.py.230: loss: 0.11903 [I 2022-10-09 23:49:24] ...optimizers/trijoint.py.151: total_norm: 8.301660 [I 2022-10-09 23:49:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 170/2304 [I 2022-10-09 23:49:32] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:25 [I 2022-10-09 23:49:32] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00337 [I 2022-10-09 23:49:32] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-09 23:49:32] ...optimizers/trijoint.py.151: total_norm: 13.018140 [I 2022-10-09 23:49:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 180/2304 [I 2022-10-09 23:49:41] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:30:07 [I 2022-10-09 23:49:41] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00330 [I 2022-10-09 23:49:41] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-09 23:49:41] ...optimizers/trijoint.py.151: total_norm: 22.306841 [I 2022-10-09 23:49:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 190/2304 [I 2022-10-09 23:49:49] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:53 [I 2022-10-09 23:49:49] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00313 [I 2022-10-09 23:49:49] ...trap/engines/engine.py.230: loss: 0.17433 [I 2022-10-09 23:49:49] ...optimizers/trijoint.py.151: total_norm: 18.930481 [I 2022-10-09 23:49:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 200/2304 [I 2022-10-09 23:49:58] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:54 [I 2022-10-09 23:49:58] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00340 [I 2022-10-09 23:49:58] ...trap/engines/engine.py.230: loss: 0.16974 [I 2022-10-09 23:49:58] ...optimizers/trijoint.py.151: total_norm: 9.155106 [I 2022-10-09 23:50:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 210/2304 [I 2022-10-09 23:50:07] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:37 [I 2022-10-09 23:50:07] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00325 [I 2022-10-09 23:50:07] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-09 23:50:07] ...optimizers/trijoint.py.151: total_norm: 9.335370 [I 2022-10-09 23:50:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 220/2304 [I 2022-10-09 23:50:15] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:32 [I 2022-10-09 23:50:15] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00353 [I 2022-10-09 23:50:15] ...trap/engines/engine.py.230: loss: 0.15501 [I 2022-10-09 23:50:15] ...optimizers/trijoint.py.151: total_norm: 17.182177 [I 2022-10-09 23:50:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 230/2304 [I 2022-10-09 23:50:24] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:22 [I 2022-10-09 23:50:24] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00320 [I 2022-10-09 23:50:24] ...trap/engines/engine.py.230: loss: 0.08984 [I 2022-10-09 23:50:24] ...optimizers/trijoint.py.151: total_norm: 14.132779 [I 2022-10-09 23:50:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 240/2304 [I 2022-10-09 23:50:32] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:13 [I 2022-10-09 23:50:32] ...trap/engines/engine.py.229: process: 0.84550 | load: 0.00313 [I 2022-10-09 23:50:32] ...trap/engines/engine.py.230: loss: 0.15902 [I 2022-10-09 23:50:32] ...optimizers/trijoint.py.151: total_norm: 36.056103 [I 2022-10-09 23:50:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 250/2304 [I 2022-10-09 23:50:41] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:19 [I 2022-10-09 23:50:41] ...trap/engines/engine.py.229: process: 0.86437 | load: 0.00314 [I 2022-10-09 23:50:41] ...trap/engines/engine.py.230: loss: 0.19030 [I 2022-10-09 23:50:41] ...optimizers/trijoint.py.151: total_norm: 20.336124 [I 2022-10-09 23:50:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 260/2304 [I 2022-10-09 23:50:49] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:51 [I 2022-10-09 23:50:49] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00309 [I 2022-10-09 23:50:49] ...trap/engines/engine.py.230: loss: 0.14716 [I 2022-10-09 23:50:49] ...optimizers/trijoint.py.151: total_norm: 15.705475 [I 2022-10-09 23:50:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 270/2304 [I 2022-10-09 23:50:58] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:57 [I 2022-10-09 23:50:58] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00315 [I 2022-10-09 23:50:58] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-09 23:50:58] ...optimizers/trijoint.py.151: total_norm: 17.174276 [I 2022-10-09 23:51:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 280/2304 [I 2022-10-09 23:51:06] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:50 [I 2022-10-09 23:51:06] ...trap/engines/engine.py.229: process: 0.84876 | load: 0.00320 [I 2022-10-09 23:51:06] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-09 23:51:06] ...optimizers/trijoint.py.151: total_norm: 16.680532 [I 2022-10-09 23:51:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 290/2304 [I 2022-10-09 23:51:15] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:41 [I 2022-10-09 23:51:15] ...trap/engines/engine.py.229: process: 0.85735 | load: 0.00326 [I 2022-10-09 23:51:15] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-09 23:51:15] ...optimizers/trijoint.py.151: total_norm: 21.507910 [I 2022-10-09 23:51:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 300/2304 [I 2022-10-09 23:51:23] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:32 [I 2022-10-09 23:51:23] ...trap/engines/engine.py.229: process: 0.84925 | load: 0.00314 [I 2022-10-09 23:51:23] ...trap/engines/engine.py.230: loss: 0.19607 [I 2022-10-09 23:51:23] ...optimizers/trijoint.py.151: total_norm: 14.341722 [I 2022-10-09 23:51:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 310/2304 [I 2022-10-09 23:51:32] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:22 [I 2022-10-09 23:51:32] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00311 [I 2022-10-09 23:51:32] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-09 23:51:32] ...optimizers/trijoint.py.151: total_norm: 22.943327 [I 2022-10-09 23:51:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 320/2304 [I 2022-10-09 23:51:41] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:14 [I 2022-10-09 23:51:41] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00323 [I 2022-10-09 23:51:41] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-09 23:51:41] ...optimizers/trijoint.py.151: total_norm: 13.055267 [I 2022-10-09 23:51:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 330/2304 [I 2022-10-09 23:51:49] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:03 [I 2022-10-09 23:51:49] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00315 [I 2022-10-09 23:51:49] ...trap/engines/engine.py.230: loss: 0.10742 [I 2022-10-09 23:51:49] ...optimizers/trijoint.py.151: total_norm: 8.138221 [I 2022-10-09 23:51:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 340/2304 [I 2022-10-09 23:51:58] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:54 [I 2022-10-09 23:51:58] ...trap/engines/engine.py.229: process: 0.83052 | load: 0.00330 [I 2022-10-09 23:51:58] ...trap/engines/engine.py.230: loss: 0.20442 [I 2022-10-09 23:51:58] ...optimizers/trijoint.py.151: total_norm: 14.855854 [I 2022-10-09 23:52:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 350/2304 [I 2022-10-09 23:52:06] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:49 [I 2022-10-09 23:52:06] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00313 [I 2022-10-09 23:52:06] ...trap/engines/engine.py.230: loss: 0.11399 [I 2022-10-09 23:52:06] ...optimizers/trijoint.py.151: total_norm: 10.627202 [I 2022-10-09 23:52:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 360/2304 [I 2022-10-09 23:52:15] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:35 [I 2022-10-09 23:52:15] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00313 [I 2022-10-09 23:52:15] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-09 23:52:15] ...optimizers/trijoint.py.151: total_norm: 21.262665 [I 2022-10-09 23:52:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 370/2304 [I 2022-10-09 23:52:23] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:28 [I 2022-10-09 23:52:23] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00319 [I 2022-10-09 23:52:23] ...trap/engines/engine.py.230: loss: 0.11620 [I 2022-10-09 23:52:23] ...optimizers/trijoint.py.151: total_norm: 21.190256 [I 2022-10-09 23:52:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 380/2304 [I 2022-10-09 23:52:32] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:23 [I 2022-10-09 23:52:32] ...trap/engines/engine.py.229: process: 0.84949 | load: 0.00315 [I 2022-10-09 23:52:32] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-09 23:52:32] ...optimizers/trijoint.py.151: total_norm: 19.263031 [I 2022-10-09 23:52:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 390/2304 [I 2022-10-09 23:52:40] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:13 [I 2022-10-09 23:52:40] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00314 [I 2022-10-09 23:52:40] ...trap/engines/engine.py.230: loss: 0.17522 [I 2022-10-09 23:52:40] ...optimizers/trijoint.py.151: total_norm: 22.328583 [I 2022-10-09 23:52:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 400/2304 [I 2022-10-09 23:52:49] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:27:02 [I 2022-10-09 23:52:49] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00324 [I 2022-10-09 23:52:49] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-09 23:52:49] ...optimizers/trijoint.py.151: total_norm: 10.303719 [I 2022-10-09 23:52:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 410/2304 [I 2022-10-09 23:52:58] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:58 [I 2022-10-09 23:52:58] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00315 [I 2022-10-09 23:52:58] ...trap/engines/engine.py.230: loss: 0.09912 [I 2022-10-09 23:52:58] ...optimizers/trijoint.py.151: total_norm: 9.874322 [I 2022-10-09 23:53:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 420/2304 [I 2022-10-09 23:53:06] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:46 [I 2022-10-09 23:53:06] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00323 [I 2022-10-09 23:53:06] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-09 23:53:06] ...optimizers/trijoint.py.151: total_norm: 16.767420 [I 2022-10-09 23:53:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 430/2304 [I 2022-10-09 23:53:15] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:46 [I 2022-10-09 23:53:15] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00315 [I 2022-10-09 23:53:15] ...trap/engines/engine.py.230: loss: 0.13564 [I 2022-10-09 23:53:15] ...optimizers/trijoint.py.151: total_norm: 14.921312 [I 2022-10-09 23:53:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 440/2304 [I 2022-10-09 23:53:23] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:29 [I 2022-10-09 23:53:23] ...trap/engines/engine.py.229: process: 0.84247 | load: 0.00316 [I 2022-10-09 23:53:23] ...trap/engines/engine.py.230: loss: 0.14976 [I 2022-10-09 23:53:23] ...optimizers/trijoint.py.151: total_norm: 32.898457 [I 2022-10-09 23:53:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 450/2304 [I 2022-10-09 23:53:32] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:18 [I 2022-10-09 23:53:32] ...trap/engines/engine.py.229: process: 0.85423 | load: 0.00318 [I 2022-10-09 23:53:32] ...trap/engines/engine.py.230: loss: 0.20137 [I 2022-10-09 23:53:32] ...optimizers/trijoint.py.151: total_norm: 12.023573 [I 2022-10-09 23:53:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 460/2304 [I 2022-10-09 23:53:40] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:14 [I 2022-10-09 23:53:40] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00312 [I 2022-10-09 23:53:40] ...trap/engines/engine.py.230: loss: 0.12068 [I 2022-10-09 23:53:40] ...optimizers/trijoint.py.151: total_norm: 10.884340 [I 2022-10-09 23:53:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 470/2304 [I 2022-10-09 23:53:49] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:12 [I 2022-10-09 23:53:49] ...trap/engines/engine.py.229: process: 0.85755 | load: 0.00302 [I 2022-10-09 23:53:49] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-09 23:53:49] ...optimizers/trijoint.py.151: total_norm: 15.574988 [I 2022-10-09 23:53:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 480/2304 [I 2022-10-09 23:53:57] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:25:57 [I 2022-10-09 23:53:57] ...trap/engines/engine.py.229: process: 0.85906 | load: 0.00310 [I 2022-10-09 23:53:57] ...trap/engines/engine.py.230: loss: 0.14459 [I 2022-10-09 23:53:57] ...optimizers/trijoint.py.151: total_norm: 36.589897 [I 2022-10-09 23:54:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 490/2304 [I 2022-10-09 23:54:06] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:37 [I 2022-10-09 23:54:06] ...trap/engines/engine.py.229: process: 0.85151 | load: 0.00322 [I 2022-10-09 23:54:06] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-09 23:54:06] ...optimizers/trijoint.py.151: total_norm: 15.719661 [I 2022-10-09 23:54:14] ...trap/engines/engine.py.224: train: epoch 59 | batch 500/2304 [I 2022-10-09 23:54:14] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:27 [I 2022-10-09 23:54:14] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00303 [I 2022-10-09 23:54:14] ...trap/engines/engine.py.230: loss: 0.17456 [I 2022-10-09 23:54:14] ...optimizers/trijoint.py.151: total_norm: 29.328512 [I 2022-10-09 23:54:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 510/2304 [I 2022-10-09 23:54:23] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:12 [I 2022-10-09 23:54:23] ...trap/engines/engine.py.229: process: 0.82578 | load: 0.00298 [I 2022-10-09 23:54:23] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-09 23:54:23] ...optimizers/trijoint.py.151: total_norm: 12.932700 [I 2022-10-09 23:54:31] ...trap/engines/engine.py.224: train: epoch 59 | batch 520/2304 [I 2022-10-09 23:54:31] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:12 [I 2022-10-09 23:54:31] ...trap/engines/engine.py.229: process: 0.84539 | load: 0.00295 [I 2022-10-09 23:54:31] ...trap/engines/engine.py.230: loss: 0.15524 [I 2022-10-09 23:54:31] ...optimizers/trijoint.py.151: total_norm: 10.571651 [I 2022-10-09 23:54:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 530/2304 [I 2022-10-09 23:54:40] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:54 [I 2022-10-09 23:54:40] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00292 [I 2022-10-09 23:54:40] ...trap/engines/engine.py.230: loss: 0.12179 [I 2022-10-09 23:54:40] ...optimizers/trijoint.py.151: total_norm: 24.485262 [I 2022-10-09 23:54:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 540/2304 [I 2022-10-09 23:54:48] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:46 [I 2022-10-09 23:54:48] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00292 [I 2022-10-09 23:54:48] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-09 23:54:48] ...optimizers/trijoint.py.151: total_norm: 14.138681 [I 2022-10-09 23:54:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 550/2304 [I 2022-10-09 23:54:57] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:42 [I 2022-10-09 23:54:57] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00293 [I 2022-10-09 23:54:57] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-09 23:54:57] ...optimizers/trijoint.py.151: total_norm: 10.368716 [I 2022-10-09 23:55:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 560/2304 [I 2022-10-09 23:55:05] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:31 [I 2022-10-09 23:55:05] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00295 [I 2022-10-09 23:55:05] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-09 23:55:05] ...optimizers/trijoint.py.151: total_norm: 25.647545 [I 2022-10-09 23:55:14] ...trap/engines/engine.py.224: train: epoch 59 | batch 570/2304 [I 2022-10-09 23:55:14] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:17 [I 2022-10-09 23:55:14] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00291 [I 2022-10-09 23:55:14] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-09 23:55:14] ...optimizers/trijoint.py.151: total_norm: 15.773665 [I 2022-10-09 23:55:22] ...trap/engines/engine.py.224: train: epoch 59 | batch 580/2304 [I 2022-10-09 23:55:22] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:08 [I 2022-10-09 23:55:22] ...trap/engines/engine.py.229: process: 0.83338 | load: 0.00316 [I 2022-10-09 23:55:22] ...trap/engines/engine.py.230: loss: 0.17048 [I 2022-10-09 23:55:22] ...optimizers/trijoint.py.151: total_norm: 9.413131 [I 2022-10-09 23:55:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 590/2304 [I 2022-10-09 23:55:30] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:05 [I 2022-10-09 23:55:30] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00304 [I 2022-10-09 23:55:30] ...trap/engines/engine.py.230: loss: 0.18071 [I 2022-10-09 23:55:30] ...optimizers/trijoint.py.151: total_norm: 20.035366 [I 2022-10-09 23:55:39] ...trap/engines/engine.py.224: train: epoch 59 | batch 600/2304 [I 2022-10-09 23:55:39] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:23:57 [I 2022-10-09 23:55:39] ...trap/engines/engine.py.229: process: 0.84323 | load: 0.00299 [I 2022-10-09 23:55:39] ...trap/engines/engine.py.230: loss: 0.09889 [I 2022-10-09 23:55:39] ...optimizers/trijoint.py.151: total_norm: 14.546535 [I 2022-10-09 23:55:47] ...trap/engines/engine.py.224: train: epoch 59 | batch 610/2304 [I 2022-10-09 23:55:47] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:00 [I 2022-10-09 23:55:47] ...trap/engines/engine.py.229: process: 0.85357 | load: 0.00295 [I 2022-10-09 23:55:47] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-09 23:55:47] ...optimizers/trijoint.py.151: total_norm: 12.486503 [I 2022-10-09 23:55:56] ...trap/engines/engine.py.224: train: epoch 59 | batch 620/2304 [I 2022-10-09 23:55:56] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:42 [I 2022-10-09 23:55:56] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00298 [I 2022-10-09 23:55:56] ...trap/engines/engine.py.230: loss: 0.21911 [I 2022-10-09 23:55:56] ...optimizers/trijoint.py.151: total_norm: 16.094904 [I 2022-10-09 23:56:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 630/2304 [I 2022-10-09 23:56:04] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:36 [I 2022-10-09 23:56:04] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00298 [I 2022-10-09 23:56:04] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-09 23:56:04] ...optimizers/trijoint.py.151: total_norm: 10.214134 [I 2022-10-09 23:56:13] ...trap/engines/engine.py.224: train: epoch 59 | batch 640/2304 [I 2022-10-09 23:56:13] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:33 [I 2022-10-09 23:56:13] ...trap/engines/engine.py.229: process: 0.83397 | load: 0.00297 [I 2022-10-09 23:56:13] ...trap/engines/engine.py.230: loss: 0.17903 [I 2022-10-09 23:56:13] ...optimizers/trijoint.py.151: total_norm: 22.327620 [I 2022-10-09 23:56:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 650/2304 [I 2022-10-09 23:56:21] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:23 [I 2022-10-09 23:56:21] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00295 [I 2022-10-09 23:56:21] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-09 23:56:21] ...optimizers/trijoint.py.151: total_norm: 16.521917 [I 2022-10-09 23:56:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 660/2304 [I 2022-10-09 23:56:30] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:12 [I 2022-10-09 23:56:30] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00294 [I 2022-10-09 23:56:30] ...trap/engines/engine.py.230: loss: 0.13606 [I 2022-10-09 23:56:30] ...optimizers/trijoint.py.151: total_norm: 21.834471 [I 2022-10-09 23:56:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 670/2304 [I 2022-10-09 23:56:38] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:22:53 [I 2022-10-09 23:56:38] ...trap/engines/engine.py.229: process: 0.82585 | load: 0.00304 [I 2022-10-09 23:56:38] ...trap/engines/engine.py.230: loss: 0.09971 [I 2022-10-09 23:56:38] ...optimizers/trijoint.py.151: total_norm: 10.855936 [I 2022-10-09 23:56:47] ...trap/engines/engine.py.224: train: epoch 59 | batch 680/2304 [I 2022-10-09 23:56:47] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:22:53 [I 2022-10-09 23:56:47] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00295 [I 2022-10-09 23:56:47] ...trap/engines/engine.py.230: loss: 0.21074 [I 2022-10-09 23:56:47] ...optimizers/trijoint.py.151: total_norm: 19.957966 [I 2022-10-09 23:56:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 690/2304 [I 2022-10-09 23:56:55] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:22:42 [I 2022-10-09 23:56:55] ...trap/engines/engine.py.229: process: 0.83982 | load: 0.00298 [I 2022-10-09 23:56:55] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-09 23:56:55] ...optimizers/trijoint.py.151: total_norm: 12.213267 [I 2022-10-09 23:57:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 700/2304 [I 2022-10-09 23:57:04] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:48 [I 2022-10-09 23:57:04] ...trap/engines/engine.py.229: process: 0.86926 | load: 0.00312 [I 2022-10-09 23:57:04] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-09 23:57:04] ...optimizers/trijoint.py.151: total_norm: 11.307867 [I 2022-10-09 23:57:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 710/2304 [I 2022-10-09 23:57:12] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:42 [I 2022-10-09 23:57:12] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00312 [I 2022-10-09 23:57:12] ...trap/engines/engine.py.230: loss: 0.17596 [I 2022-10-09 23:57:12] ...optimizers/trijoint.py.151: total_norm: 15.241189 [I 2022-10-09 23:57:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 720/2304 [I 2022-10-09 23:57:21] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:25 [I 2022-10-09 23:57:21] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00317 [I 2022-10-09 23:57:21] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-09 23:57:21] ...optimizers/trijoint.py.151: total_norm: 17.508757 [I 2022-10-09 23:57:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 730/2304 [I 2022-10-09 23:57:29] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:29 [I 2022-10-09 23:57:29] ...trap/engines/engine.py.229: process: 0.86687 | load: 0.00315 [I 2022-10-09 23:57:29] ...trap/engines/engine.py.230: loss: 0.11068 [I 2022-10-09 23:57:29] ...optimizers/trijoint.py.151: total_norm: 9.218452 [I 2022-10-09 23:57:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 740/2304 [I 2022-10-09 23:57:38] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:13 [I 2022-10-09 23:57:38] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00313 [I 2022-10-09 23:57:38] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-09 23:57:38] ...optimizers/trijoint.py.151: total_norm: 10.021961 [I 2022-10-09 23:57:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 750/2304 [I 2022-10-09 23:57:46] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:04 [I 2022-10-09 23:57:46] ...trap/engines/engine.py.229: process: 0.84719 | load: 0.00313 [I 2022-10-09 23:57:46] ...trap/engines/engine.py.230: loss: 0.17084 [I 2022-10-09 23:57:46] ...optimizers/trijoint.py.151: total_norm: 13.478944 [I 2022-10-09 23:57:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 760/2304 [I 2022-10-09 23:57:55] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:55 [I 2022-10-09 23:57:55] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00319 [I 2022-10-09 23:57:55] ...trap/engines/engine.py.230: loss: 0.10649 [I 2022-10-09 23:57:55] ...optimizers/trijoint.py.151: total_norm: 12.206655 [I 2022-10-09 23:58:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 770/2304 [I 2022-10-09 23:58:04] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:49 [I 2022-10-09 23:58:04] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00313 [I 2022-10-09 23:58:04] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-09 23:58:04] ...optimizers/trijoint.py.151: total_norm: 10.417008 [I 2022-10-09 23:58:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 780/2304 [I 2022-10-09 23:58:12] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:32 [I 2022-10-09 23:58:12] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00313 [I 2022-10-09 23:58:12] ...trap/engines/engine.py.230: loss: 0.15204 [I 2022-10-09 23:58:12] ...optimizers/trijoint.py.151: total_norm: 14.808122 [I 2022-10-09 23:58:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 790/2304 [I 2022-10-09 23:58:21] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:28 [I 2022-10-09 23:58:21] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00313 [I 2022-10-09 23:58:21] ...trap/engines/engine.py.230: loss: 0.18631 [I 2022-10-09 23:58:21] ...optimizers/trijoint.py.151: total_norm: 29.619017 [I 2022-10-09 23:58:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 800/2304 [I 2022-10-09 23:58:29] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:24 [I 2022-10-09 23:58:29] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00314 [I 2022-10-09 23:58:29] ...trap/engines/engine.py.230: loss: 0.10099 [I 2022-10-09 23:58:29] ...optimizers/trijoint.py.151: total_norm: 15.569325 [I 2022-10-09 23:58:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 810/2304 [I 2022-10-09 23:58:38] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:10 [I 2022-10-09 23:58:38] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00315 [I 2022-10-09 23:58:38] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-09 23:58:38] ...optimizers/trijoint.py.151: total_norm: 13.566488 [I 2022-10-09 23:58:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 820/2304 [I 2022-10-09 23:58:46] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:04 [I 2022-10-09 23:58:46] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00314 [I 2022-10-09 23:58:46] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-09 23:58:46] ...optimizers/trijoint.py.151: total_norm: 11.936057 [I 2022-10-09 23:58:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 830/2304 [I 2022-10-09 23:58:55] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:55 [I 2022-10-09 23:58:55] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00315 [I 2022-10-09 23:58:55] ...trap/engines/engine.py.230: loss: 0.17100 [I 2022-10-09 23:58:55] ...optimizers/trijoint.py.151: total_norm: 30.147232 [I 2022-10-09 23:59:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 840/2304 [I 2022-10-09 23:59:03] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:20:55 [I 2022-10-09 23:59:03] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00311 [I 2022-10-09 23:59:03] ...trap/engines/engine.py.230: loss: 0.10108 [I 2022-10-09 23:59:03] ...optimizers/trijoint.py.151: total_norm: 11.298528 [I 2022-10-09 23:59:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 850/2304 [I 2022-10-09 23:59:12] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:41 [I 2022-10-09 23:59:12] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00316 [I 2022-10-09 23:59:12] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-09 23:59:12] ...optimizers/trijoint.py.151: total_norm: 12.279552 [I 2022-10-09 23:59:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 860/2304 [I 2022-10-09 23:59:21] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:33 [I 2022-10-09 23:59:21] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00315 [I 2022-10-09 23:59:21] ...trap/engines/engine.py.230: loss: 0.12886 [I 2022-10-09 23:59:21] ...optimizers/trijoint.py.151: total_norm: 28.191565 [I 2022-10-09 23:59:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 870/2304 [I 2022-10-09 23:59:29] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:24 [I 2022-10-09 23:59:29] ...trap/engines/engine.py.229: process: 0.85327 | load: 0.00313 [I 2022-10-09 23:59:29] ...trap/engines/engine.py.230: loss: 0.13498 [I 2022-10-09 23:59:29] ...optimizers/trijoint.py.151: total_norm: 14.329320 [I 2022-10-09 23:59:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 880/2304 [I 2022-10-09 23:59:38] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:28 [I 2022-10-09 23:59:38] ...trap/engines/engine.py.229: process: 0.86968 | load: 0.00314 [I 2022-10-09 23:59:38] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-09 23:59:38] ...optimizers/trijoint.py.151: total_norm: 16.982002 [I 2022-10-09 23:59:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 890/2304 [I 2022-10-09 23:59:46] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:03 [I 2022-10-09 23:59:46] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00310 [I 2022-10-09 23:59:46] ...trap/engines/engine.py.230: loss: 0.19835 [I 2022-10-09 23:59:46] ...optimizers/trijoint.py.151: total_norm: 16.957634 [I 2022-10-09 23:59:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 900/2304 [I 2022-10-09 23:59:55] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:00 [I 2022-10-09 23:59:55] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00313 [I 2022-10-09 23:59:55] ...trap/engines/engine.py.230: loss: 0.14436 [I 2022-10-09 23:59:55] ...optimizers/trijoint.py.151: total_norm: 19.879116 [I 2022-10-10 00:00:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 910/2304 [I 2022-10-10 00:00:03] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:50 [I 2022-10-10 00:00:03] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00311 [I 2022-10-10 00:00:03] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-10 00:00:03] ...optimizers/trijoint.py.151: total_norm: 26.677475 [I 2022-10-10 00:00:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 920/2304 [I 2022-10-10 00:00:12] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:41 [I 2022-10-10 00:00:12] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00311 [I 2022-10-10 00:00:12] ...trap/engines/engine.py.230: loss: 0.12293 [I 2022-10-10 00:00:12] ...optimizers/trijoint.py.151: total_norm: 10.466795 [I 2022-10-10 00:00:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 930/2304 [I 2022-10-10 00:00:20] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:32 [I 2022-10-10 00:00:20] ...trap/engines/engine.py.229: process: 0.85724 | load: 0.00315 [I 2022-10-10 00:00:20] ...trap/engines/engine.py.230: loss: 0.16265 [I 2022-10-10 00:00:20] ...optimizers/trijoint.py.151: total_norm: 23.882772 [I 2022-10-10 00:00:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 940/2304 [I 2022-10-10 00:00:29] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:21 [I 2022-10-10 00:00:29] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00437 [I 2022-10-10 00:00:29] ...trap/engines/engine.py.230: loss: 0.18457 [I 2022-10-10 00:00:29] ...optimizers/trijoint.py.151: total_norm: 28.754642 [I 2022-10-10 00:00:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 950/2304 [I 2022-10-10 00:00:37] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:19:11 [I 2022-10-10 00:00:37] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00315 [I 2022-10-10 00:00:37] ...trap/engines/engine.py.230: loss: 0.17047 [I 2022-10-10 00:00:37] ...optimizers/trijoint.py.151: total_norm: 17.029270 [I 2022-10-10 00:00:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 960/2304 [I 2022-10-10 00:00:46] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:10 [I 2022-10-10 00:00:46] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00312 [I 2022-10-10 00:00:46] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-10 00:00:46] ...optimizers/trijoint.py.151: total_norm: 52.675423 [I 2022-10-10 00:00:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 970/2304 [I 2022-10-10 00:00:55] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:03 [I 2022-10-10 00:00:55] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00313 [I 2022-10-10 00:00:55] ...trap/engines/engine.py.230: loss: 0.14342 [I 2022-10-10 00:00:55] ...optimizers/trijoint.py.151: total_norm: 17.586832 [I 2022-10-10 00:01:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 980/2304 [I 2022-10-10 00:01:03] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:46 [I 2022-10-10 00:01:03] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00313 [I 2022-10-10 00:01:03] ...trap/engines/engine.py.230: loss: 0.14206 [I 2022-10-10 00:01:03] ...optimizers/trijoint.py.151: total_norm: 14.622828 [I 2022-10-10 00:01:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 990/2304 [I 2022-10-10 00:01:12] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:33 [I 2022-10-10 00:01:12] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00314 [I 2022-10-10 00:01:12] ...trap/engines/engine.py.230: loss: 0.11741 [I 2022-10-10 00:01:12] ...optimizers/trijoint.py.151: total_norm: 12.907410 [I 2022-10-10 00:01:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 1000/2304 [I 2022-10-10 00:01:20] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:32 [I 2022-10-10 00:01:20] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00314 [I 2022-10-10 00:01:20] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-10 00:01:20] ...optimizers/trijoint.py.151: total_norm: 20.326431 [I 2022-10-10 00:01:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 1010/2304 [I 2022-10-10 00:01:29] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:21 [I 2022-10-10 00:01:29] ...trap/engines/engine.py.229: process: 0.86740 | load: 0.00311 [I 2022-10-10 00:01:29] ...trap/engines/engine.py.230: loss: 0.19200 [I 2022-10-10 00:01:29] ...optimizers/trijoint.py.151: total_norm: 19.534021 [I 2022-10-10 00:01:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 1020/2304 [I 2022-10-10 00:01:37] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:11 [I 2022-10-10 00:01:37] ...trap/engines/engine.py.229: process: 0.84775 | load: 0.00312 [I 2022-10-10 00:01:37] ...trap/engines/engine.py.230: loss: 0.10378 [I 2022-10-10 00:01:37] ...optimizers/trijoint.py.151: total_norm: 8.201673 [I 2022-10-10 00:01:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 1030/2304 [I 2022-10-10 00:01:46] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:59 [I 2022-10-10 00:01:46] ...trap/engines/engine.py.229: process: 0.83536 | load: 0.00312 [I 2022-10-10 00:01:46] ...trap/engines/engine.py.230: loss: 0.10000 [I 2022-10-10 00:01:46] ...optimizers/trijoint.py.151: total_norm: 8.427961 [I 2022-10-10 00:01:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 1040/2304 [I 2022-10-10 00:01:54] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:17:49 [I 2022-10-10 00:01:54] ...trap/engines/engine.py.229: process: 0.83712 | load: 0.00301 [I 2022-10-10 00:01:54] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-10 00:01:54] ...optimizers/trijoint.py.151: total_norm: 10.445906 [I 2022-10-10 00:02:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 1050/2304 [I 2022-10-10 00:02:03] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:43 [I 2022-10-10 00:02:03] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00297 [I 2022-10-10 00:02:03] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-10 00:02:03] ...optimizers/trijoint.py.151: total_norm: 15.207909 [I 2022-10-10 00:02:11] ...trap/engines/engine.py.224: train: epoch 59 | batch 1060/2304 [I 2022-10-10 00:02:11] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:34 [I 2022-10-10 00:02:11] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00292 [I 2022-10-10 00:02:11] ...trap/engines/engine.py.230: loss: 0.13024 [I 2022-10-10 00:02:11] ...optimizers/trijoint.py.151: total_norm: 13.981499 [I 2022-10-10 00:02:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 1070/2304 [I 2022-10-10 00:02:20] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:25 [I 2022-10-10 00:02:20] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00296 [I 2022-10-10 00:02:20] ...trap/engines/engine.py.230: loss: 0.14556 [I 2022-10-10 00:02:20] ...optimizers/trijoint.py.151: total_norm: 42.363186 [I 2022-10-10 00:02:28] ...trap/engines/engine.py.224: train: epoch 59 | batch 1080/2304 [I 2022-10-10 00:02:28] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:15 [I 2022-10-10 00:02:28] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00295 [I 2022-10-10 00:02:28] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-10 00:02:28] ...optimizers/trijoint.py.151: total_norm: 10.878592 [I 2022-10-10 00:02:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 1090/2304 [I 2022-10-10 00:02:37] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:02 [I 2022-10-10 00:02:37] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00293 [I 2022-10-10 00:02:37] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-10 00:02:37] ...optimizers/trijoint.py.151: total_norm: 13.637624 [I 2022-10-10 00:02:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 1100/2304 [I 2022-10-10 00:02:45] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:59 [I 2022-10-10 00:02:45] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00304 [I 2022-10-10 00:02:45] ...trap/engines/engine.py.230: loss: 0.15752 [I 2022-10-10 00:02:45] ...optimizers/trijoint.py.151: total_norm: 15.170568 [I 2022-10-10 00:02:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 1110/2304 [I 2022-10-10 00:02:54] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:50 [I 2022-10-10 00:02:54] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00294 [I 2022-10-10 00:02:54] ...trap/engines/engine.py.230: loss: 0.11722 [I 2022-10-10 00:02:54] ...optimizers/trijoint.py.151: total_norm: 14.254139 [I 2022-10-10 00:03:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 1120/2304 [I 2022-10-10 00:03:02] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:35 [I 2022-10-10 00:03:02] ...trap/engines/engine.py.229: process: 0.83615 | load: 0.00302 [I 2022-10-10 00:03:02] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-10 00:03:02] ...optimizers/trijoint.py.151: total_norm: 12.658936 [I 2022-10-10 00:03:11] ...trap/engines/engine.py.224: train: epoch 59 | batch 1130/2304 [I 2022-10-10 00:03:11] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:33 [I 2022-10-10 00:03:11] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00298 [I 2022-10-10 00:03:11] ...trap/engines/engine.py.230: loss: 0.14582 [I 2022-10-10 00:03:11] ...optimizers/trijoint.py.151: total_norm: 13.822295 [I 2022-10-10 00:03:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 1140/2304 [I 2022-10-10 00:03:19] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:21 [I 2022-10-10 00:03:19] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00298 [I 2022-10-10 00:03:19] ...trap/engines/engine.py.230: loss: 0.10266 [I 2022-10-10 00:03:19] ...optimizers/trijoint.py.151: total_norm: 7.439941 [I 2022-10-10 00:03:28] ...trap/engines/engine.py.224: train: epoch 59 | batch 1150/2304 [I 2022-10-10 00:03:28] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:12 [I 2022-10-10 00:03:28] ...trap/engines/engine.py.229: process: 0.82959 | load: 0.00299 [I 2022-10-10 00:03:28] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-10 00:03:28] ...optimizers/trijoint.py.151: total_norm: 9.714261 [I 2022-10-10 00:03:36] ...trap/engines/engine.py.224: train: epoch 59 | batch 1160/2304 [I 2022-10-10 00:03:36] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:10 [I 2022-10-10 00:03:36] ...trap/engines/engine.py.229: process: 0.85374 | load: 0.00305 [I 2022-10-10 00:03:36] ...trap/engines/engine.py.230: loss: 0.27669 [I 2022-10-10 00:03:36] ...optimizers/trijoint.py.151: total_norm: 22.860334 [I 2022-10-10 00:03:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 1170/2304 [I 2022-10-10 00:03:44] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:15:54 [I 2022-10-10 00:03:44] ...trap/engines/engine.py.229: process: 0.84295 | load: 0.00300 [I 2022-10-10 00:03:44] ...trap/engines/engine.py.230: loss: 0.21999 [I 2022-10-10 00:03:44] ...optimizers/trijoint.py.151: total_norm: 20.624336 [I 2022-10-10 00:03:53] ...trap/engines/engine.py.224: train: epoch 59 | batch 1180/2304 [I 2022-10-10 00:03:53] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:44 [I 2022-10-10 00:03:53] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00299 [I 2022-10-10 00:03:53] ...trap/engines/engine.py.230: loss: 0.13009 [I 2022-10-10 00:03:53] ...optimizers/trijoint.py.151: total_norm: 14.677939 [I 2022-10-10 00:04:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 1190/2304 [I 2022-10-10 00:04:01] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:40 [I 2022-10-10 00:04:01] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00294 [I 2022-10-10 00:04:01] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-10 00:04:01] ...optimizers/trijoint.py.151: total_norm: 12.632408 [I 2022-10-10 00:04:10] ...trap/engines/engine.py.224: train: epoch 59 | batch 1200/2304 [I 2022-10-10 00:04:10] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:45 [I 2022-10-10 00:04:10] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00299 [I 2022-10-10 00:04:10] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-10 00:04:10] ...optimizers/trijoint.py.151: total_norm: 10.969526 [I 2022-10-10 00:04:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 1210/2304 [I 2022-10-10 00:04:18] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:29 [I 2022-10-10 00:04:18] ...trap/engines/engine.py.229: process: 0.85431 | load: 0.00298 [I 2022-10-10 00:04:18] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-10 00:04:18] ...optimizers/trijoint.py.151: total_norm: 27.989634 [I 2022-10-10 00:04:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 1220/2304 [I 2022-10-10 00:04:27] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:23 [I 2022-10-10 00:04:27] ...trap/engines/engine.py.229: process: 0.84067 | load: 0.00298 [I 2022-10-10 00:04:27] ...trap/engines/engine.py.230: loss: 0.16933 [I 2022-10-10 00:04:27] ...optimizers/trijoint.py.151: total_norm: 12.910697 [I 2022-10-10 00:04:35] ...trap/engines/engine.py.224: train: epoch 59 | batch 1230/2304 [I 2022-10-10 00:04:35] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:07 [I 2022-10-10 00:04:35] ...trap/engines/engine.py.229: process: 0.84447 | load: 0.00301 [I 2022-10-10 00:04:35] ...trap/engines/engine.py.230: loss: 0.16264 [I 2022-10-10 00:04:35] ...optimizers/trijoint.py.151: total_norm: 14.787194 [I 2022-10-10 00:04:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 1240/2304 [I 2022-10-10 00:04:44] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:05 [I 2022-10-10 00:04:44] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00301 [I 2022-10-10 00:04:44] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-10 00:04:44] ...optimizers/trijoint.py.151: total_norm: 8.646798 [I 2022-10-10 00:04:53] ...trap/engines/engine.py.224: train: epoch 59 | batch 1250/2304 [I 2022-10-10 00:04:53] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:54 [I 2022-10-10 00:04:53] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00301 [I 2022-10-10 00:04:53] ...trap/engines/engine.py.230: loss: 0.16648 [I 2022-10-10 00:04:53] ...optimizers/trijoint.py.151: total_norm: 18.499615 [I 2022-10-10 00:05:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 1260/2304 [I 2022-10-10 00:05:01] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:47 [I 2022-10-10 00:05:01] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00324 [I 2022-10-10 00:05:01] ...trap/engines/engine.py.230: loss: 0.18806 [I 2022-10-10 00:05:01] ...optimizers/trijoint.py.151: total_norm: 16.490046 [I 2022-10-10 00:05:10] ...trap/engines/engine.py.224: train: epoch 59 | batch 1270/2304 [I 2022-10-10 00:05:10] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:41 [I 2022-10-10 00:05:10] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00309 [I 2022-10-10 00:05:10] ...trap/engines/engine.py.230: loss: 0.15493 [I 2022-10-10 00:05:10] ...optimizers/trijoint.py.151: total_norm: 25.479288 [I 2022-10-10 00:05:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 1280/2304 [I 2022-10-10 00:05:18] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:27 [I 2022-10-10 00:05:18] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00301 [I 2022-10-10 00:05:18] ...trap/engines/engine.py.230: loss: 0.19717 [I 2022-10-10 00:05:18] ...optimizers/trijoint.py.151: total_norm: 38.639404 [I 2022-10-10 00:05:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 1290/2304 [I 2022-10-10 00:05:27] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:18 [I 2022-10-10 00:05:27] ...trap/engines/engine.py.229: process: 0.82919 | load: 0.00302 [I 2022-10-10 00:05:27] ...trap/engines/engine.py.230: loss: 0.16868 [I 2022-10-10 00:05:27] ...optimizers/trijoint.py.151: total_norm: 18.657845 [I 2022-10-10 00:05:35] ...trap/engines/engine.py.224: train: epoch 59 | batch 1300/2304 [I 2022-10-10 00:05:35] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:12 [I 2022-10-10 00:05:35] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00304 [I 2022-10-10 00:05:35] ...trap/engines/engine.py.230: loss: 0.13525 [I 2022-10-10 00:05:35] ...optimizers/trijoint.py.151: total_norm: 9.115510 [I 2022-10-10 00:05:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 1310/2304 [I 2022-10-10 00:05:44] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:04 [I 2022-10-10 00:05:44] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00312 [I 2022-10-10 00:05:44] ...trap/engines/engine.py.230: loss: 0.18667 [I 2022-10-10 00:05:44] ...optimizers/trijoint.py.151: total_norm: 18.719059 [I 2022-10-10 00:05:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 1320/2304 [I 2022-10-10 00:05:52] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:13:50 [I 2022-10-10 00:05:52] ...trap/engines/engine.py.229: process: 0.83163 | load: 0.00300 [I 2022-10-10 00:05:52] ...trap/engines/engine.py.230: loss: 0.13850 [I 2022-10-10 00:05:52] ...optimizers/trijoint.py.151: total_norm: 16.628925 [I 2022-10-10 00:06:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 1330/2304 [I 2022-10-10 00:06:01] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:43 [I 2022-10-10 00:06:01] ...trap/engines/engine.py.229: process: 0.83529 | load: 0.00304 [I 2022-10-10 00:06:01] ...trap/engines/engine.py.230: loss: 0.11241 [I 2022-10-10 00:06:01] ...optimizers/trijoint.py.151: total_norm: 8.241442 [I 2022-10-10 00:06:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 1340/2304 [I 2022-10-10 00:06:09] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:34 [I 2022-10-10 00:06:09] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00298 [I 2022-10-10 00:06:09] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-10 00:06:09] ...optimizers/trijoint.py.151: total_norm: 9.956258 [I 2022-10-10 00:06:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 1350/2304 [I 2022-10-10 00:06:18] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:32 [I 2022-10-10 00:06:18] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00312 [I 2022-10-10 00:06:18] ...trap/engines/engine.py.230: loss: 0.19631 [I 2022-10-10 00:06:18] ...optimizers/trijoint.py.151: total_norm: 22.245932 [I 2022-10-10 00:06:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 1360/2304 [I 2022-10-10 00:06:26] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:17 [I 2022-10-10 00:06:26] ...trap/engines/engine.py.229: process: 0.83557 | load: 0.00300 [I 2022-10-10 00:06:26] ...trap/engines/engine.py.230: loss: 0.19402 [I 2022-10-10 00:06:26] ...optimizers/trijoint.py.151: total_norm: 18.689537 [I 2022-10-10 00:06:35] ...trap/engines/engine.py.224: train: epoch 59 | batch 1370/2304 [I 2022-10-10 00:06:35] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:13 [I 2022-10-10 00:06:35] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00301 [I 2022-10-10 00:06:35] ...trap/engines/engine.py.230: loss: 0.16479 [I 2022-10-10 00:06:35] ...optimizers/trijoint.py.151: total_norm: 14.190946 [I 2022-10-10 00:06:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 1380/2304 [I 2022-10-10 00:06:43] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:06 [I 2022-10-10 00:06:43] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00299 [I 2022-10-10 00:06:43] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-10 00:06:43] ...optimizers/trijoint.py.151: total_norm: 10.872920 [I 2022-10-10 00:06:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 1390/2304 [I 2022-10-10 00:06:52] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:51 [I 2022-10-10 00:06:52] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00302 [I 2022-10-10 00:06:52] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-10 00:06:52] ...optimizers/trijoint.py.151: total_norm: 32.663147 [I 2022-10-10 00:07:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1400/2304 [I 2022-10-10 00:07:00] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:12:50 [I 2022-10-10 00:07:00] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00301 [I 2022-10-10 00:07:00] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-10 00:07:00] ...optimizers/trijoint.py.151: total_norm: 8.152904 [I 2022-10-10 00:07:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 1410/2304 [I 2022-10-10 00:07:09] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:38 [I 2022-10-10 00:07:09] ...trap/engines/engine.py.229: process: 0.86001 | load: 0.00337 [I 2022-10-10 00:07:09] ...trap/engines/engine.py.230: loss: 0.21099 [I 2022-10-10 00:07:09] ...optimizers/trijoint.py.151: total_norm: 12.975043 [I 2022-10-10 00:07:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1420/2304 [I 2022-10-10 00:07:17] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:30 [I 2022-10-10 00:07:17] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00318 [I 2022-10-10 00:07:17] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-10 00:07:17] ...optimizers/trijoint.py.151: total_norm: 21.190481 [I 2022-10-10 00:07:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 1430/2304 [I 2022-10-10 00:07:26] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:28 [I 2022-10-10 00:07:26] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00325 [I 2022-10-10 00:07:26] ...trap/engines/engine.py.230: loss: 0.15042 [I 2022-10-10 00:07:26] ...optimizers/trijoint.py.151: total_norm: 14.832324 [I 2022-10-10 00:07:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1440/2304 [I 2022-10-10 00:07:34] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:22 [I 2022-10-10 00:07:34] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00315 [I 2022-10-10 00:07:34] ...trap/engines/engine.py.230: loss: 0.18117 [I 2022-10-10 00:07:34] ...optimizers/trijoint.py.151: total_norm: 10.831987 [I 2022-10-10 00:07:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 1450/2304 [I 2022-10-10 00:07:43] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:07 [I 2022-10-10 00:07:43] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00300 [I 2022-10-10 00:07:43] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-10 00:07:43] ...optimizers/trijoint.py.151: total_norm: 11.549498 [I 2022-10-10 00:07:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1460/2304 [I 2022-10-10 00:07:51] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:11:59 [I 2022-10-10 00:07:51] ...trap/engines/engine.py.229: process: 0.84003 | load: 0.00302 [I 2022-10-10 00:07:51] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-10 00:07:51] ...optimizers/trijoint.py.151: total_norm: 23.198879 [I 2022-10-10 00:08:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1470/2304 [I 2022-10-10 00:08:00] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:46 [I 2022-10-10 00:08:00] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00309 [I 2022-10-10 00:08:00] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-10 00:08:00] ...optimizers/trijoint.py.151: total_norm: 10.817837 [I 2022-10-10 00:08:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1480/2304 [I 2022-10-10 00:08:08] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:42 [I 2022-10-10 00:08:08] ...trap/engines/engine.py.229: process: 0.84540 | load: 0.00301 [I 2022-10-10 00:08:08] ...trap/engines/engine.py.230: loss: 0.10992 [I 2022-10-10 00:08:08] ...optimizers/trijoint.py.151: total_norm: 13.846410 [I 2022-10-10 00:08:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1490/2304 [I 2022-10-10 00:08:17] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:33 [I 2022-10-10 00:08:17] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00308 [I 2022-10-10 00:08:17] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-10 00:08:17] ...optimizers/trijoint.py.151: total_norm: 23.325249 [I 2022-10-10 00:08:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 1500/2304 [I 2022-10-10 00:08:26] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:29 [I 2022-10-10 00:08:26] ...trap/engines/engine.py.229: process: 0.86960 | load: 0.00314 [I 2022-10-10 00:08:26] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-10 00:08:26] ...optimizers/trijoint.py.151: total_norm: 29.193008 [I 2022-10-10 00:08:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1510/2304 [I 2022-10-10 00:08:34] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:17 [I 2022-10-10 00:08:34] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00306 [I 2022-10-10 00:08:34] ...trap/engines/engine.py.230: loss: 0.17151 [I 2022-10-10 00:08:34] ...optimizers/trijoint.py.151: total_norm: 27.412096 [I 2022-10-10 00:08:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 1520/2304 [I 2022-10-10 00:08:43] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:07 [I 2022-10-10 00:08:43] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00309 [I 2022-10-10 00:08:43] ...trap/engines/engine.py.230: loss: 0.17336 [I 2022-10-10 00:08:43] ...optimizers/trijoint.py.151: total_norm: 20.241213 [I 2022-10-10 00:08:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1530/2304 [I 2022-10-10 00:08:51] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:00 [I 2022-10-10 00:08:51] ...trap/engines/engine.py.229: process: 0.86837 | load: 0.00296 [I 2022-10-10 00:08:51] ...trap/engines/engine.py.230: loss: 0.19638 [I 2022-10-10 00:08:51] ...optimizers/trijoint.py.151: total_norm: 10.213162 [I 2022-10-10 00:09:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1540/2304 [I 2022-10-10 00:09:00] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:51 [I 2022-10-10 00:09:00] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00301 [I 2022-10-10 00:09:00] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-10 00:09:00] ...optimizers/trijoint.py.151: total_norm: 8.476204 [I 2022-10-10 00:09:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1550/2304 [I 2022-10-10 00:09:08] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:36 [I 2022-10-10 00:09:08] ...trap/engines/engine.py.229: process: 0.83477 | load: 0.00310 [I 2022-10-10 00:09:08] ...trap/engines/engine.py.230: loss: 0.18222 [I 2022-10-10 00:09:08] ...optimizers/trijoint.py.151: total_norm: 29.161617 [I 2022-10-10 00:09:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1560/2304 [I 2022-10-10 00:09:17] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:33 [I 2022-10-10 00:09:17] ...trap/engines/engine.py.229: process: 0.86252 | load: 0.00296 [I 2022-10-10 00:09:17] ...trap/engines/engine.py.230: loss: 0.19897 [I 2022-10-10 00:09:17] ...optimizers/trijoint.py.151: total_norm: 14.661757 [I 2022-10-10 00:09:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1570/2304 [I 2022-10-10 00:09:25] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:28 [I 2022-10-10 00:09:25] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00311 [I 2022-10-10 00:09:25] ...trap/engines/engine.py.230: loss: 0.11365 [I 2022-10-10 00:09:25] ...optimizers/trijoint.py.151: total_norm: 10.053300 [I 2022-10-10 00:09:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1580/2304 [I 2022-10-10 00:09:34] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:21 [I 2022-10-10 00:09:34] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00312 [I 2022-10-10 00:09:34] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-10 00:09:34] ...optimizers/trijoint.py.151: total_norm: 12.388879 [I 2022-10-10 00:09:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1590/2304 [I 2022-10-10 00:09:42] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:10:08 [I 2022-10-10 00:09:42] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00309 [I 2022-10-10 00:09:42] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-10 00:09:42] ...optimizers/trijoint.py.151: total_norm: 18.657614 [I 2022-10-10 00:09:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1600/2304 [I 2022-10-10 00:09:51] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:04 [I 2022-10-10 00:09:51] ...trap/engines/engine.py.229: process: 0.87146 | load: 0.00311 [I 2022-10-10 00:09:51] ...trap/engines/engine.py.230: loss: 0.16270 [I 2022-10-10 00:09:51] ...optimizers/trijoint.py.151: total_norm: 23.354464 [I 2022-10-10 00:10:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1610/2304 [I 2022-10-10 00:10:00] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:51 [I 2022-10-10 00:10:00] ...trap/engines/engine.py.229: process: 0.85680 | load: 0.00310 [I 2022-10-10 00:10:00] ...trap/engines/engine.py.230: loss: 0.15605 [I 2022-10-10 00:10:00] ...optimizers/trijoint.py.151: total_norm: 13.037491 [I 2022-10-10 00:10:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1620/2304 [I 2022-10-10 00:10:08] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:46 [I 2022-10-10 00:10:08] ...trap/engines/engine.py.229: process: 0.87527 | load: 0.00312 [I 2022-10-10 00:10:08] ...trap/engines/engine.py.230: loss: 0.19765 [I 2022-10-10 00:10:08] ...optimizers/trijoint.py.151: total_norm: 21.445707 [I 2022-10-10 00:10:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1630/2304 [I 2022-10-10 00:10:17] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:36 [I 2022-10-10 00:10:17] ...trap/engines/engine.py.229: process: 0.86191 | load: 0.00314 [I 2022-10-10 00:10:17] ...trap/engines/engine.py.230: loss: 0.14972 [I 2022-10-10 00:10:17] ...optimizers/trijoint.py.151: total_norm: 23.366074 [I 2022-10-10 00:10:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1640/2304 [I 2022-10-10 00:10:25] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:25 [I 2022-10-10 00:10:25] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00311 [I 2022-10-10 00:10:25] ...trap/engines/engine.py.230: loss: 0.11759 [I 2022-10-10 00:10:25] ...optimizers/trijoint.py.151: total_norm: 13.627415 [I 2022-10-10 00:10:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1650/2304 [I 2022-10-10 00:10:34] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:17 [I 2022-10-10 00:10:34] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00312 [I 2022-10-10 00:10:34] ...trap/engines/engine.py.230: loss: 0.22146 [I 2022-10-10 00:10:34] ...optimizers/trijoint.py.151: total_norm: 16.353077 [I 2022-10-10 00:10:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1660/2304 [I 2022-10-10 00:10:42] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:07 [I 2022-10-10 00:10:42] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00313 [I 2022-10-10 00:10:42] ...trap/engines/engine.py.230: loss: 0.12868 [I 2022-10-10 00:10:42] ...optimizers/trijoint.py.151: total_norm: 44.168514 [I 2022-10-10 00:10:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1670/2304 [I 2022-10-10 00:10:51] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:01 [I 2022-10-10 00:10:51] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00312 [I 2022-10-10 00:10:51] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-10 00:10:51] ...optimizers/trijoint.py.151: total_norm: 12.766171 [I 2022-10-10 00:11:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1680/2304 [I 2022-10-10 00:11:00] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:53 [I 2022-10-10 00:11:00] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00311 [I 2022-10-10 00:11:00] ...trap/engines/engine.py.230: loss: 0.17833 [I 2022-10-10 00:11:00] ...optimizers/trijoint.py.151: total_norm: 24.472691 [I 2022-10-10 00:11:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1690/2304 [I 2022-10-10 00:11:08] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:45 [I 2022-10-10 00:11:08] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00312 [I 2022-10-10 00:11:08] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-10 00:11:08] ...optimizers/trijoint.py.151: total_norm: 17.014105 [I 2022-10-10 00:11:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1700/2304 [I 2022-10-10 00:11:17] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:39 [I 2022-10-10 00:11:17] ...trap/engines/engine.py.229: process: 0.87218 | load: 0.00310 [I 2022-10-10 00:11:17] ...trap/engines/engine.py.230: loss: 0.19934 [I 2022-10-10 00:11:17] ...optimizers/trijoint.py.151: total_norm: 8.310965 [I 2022-10-10 00:11:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1710/2304 [I 2022-10-10 00:11:25] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:27 [I 2022-10-10 00:11:25] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00310 [I 2022-10-10 00:11:25] ...trap/engines/engine.py.230: loss: 0.12012 [I 2022-10-10 00:11:25] ...optimizers/trijoint.py.151: total_norm: 16.886396 [I 2022-10-10 00:11:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1720/2304 [I 2022-10-10 00:11:34] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:15 [I 2022-10-10 00:11:34] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00311 [I 2022-10-10 00:11:34] ...trap/engines/engine.py.230: loss: 0.26100 [I 2022-10-10 00:11:34] ...optimizers/trijoint.py.151: total_norm: 32.124352 [I 2022-10-10 00:11:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1730/2304 [I 2022-10-10 00:11:42] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:10 [I 2022-10-10 00:11:42] ...trap/engines/engine.py.229: process: 0.85263 | load: 0.00315 [I 2022-10-10 00:11:42] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-10 00:11:42] ...optimizers/trijoint.py.151: total_norm: 37.454201 [I 2022-10-10 00:11:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1740/2304 [I 2022-10-10 00:11:51] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:59 [I 2022-10-10 00:11:51] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00308 [I 2022-10-10 00:11:51] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-10 00:11:51] ...optimizers/trijoint.py.151: total_norm: 16.327103 [I 2022-10-10 00:11:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 1750/2304 [I 2022-10-10 00:11:59] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:53 [I 2022-10-10 00:11:59] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00309 [I 2022-10-10 00:11:59] ...trap/engines/engine.py.230: loss: 0.17788 [I 2022-10-10 00:11:59] ...optimizers/trijoint.py.151: total_norm: 18.983595 [I 2022-10-10 00:12:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1760/2304 [I 2022-10-10 00:12:08] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:40 [I 2022-10-10 00:12:08] ...trap/engines/engine.py.229: process: 0.84217 | load: 0.00300 [I 2022-10-10 00:12:08] ...trap/engines/engine.py.230: loss: 0.13158 [I 2022-10-10 00:12:08] ...optimizers/trijoint.py.151: total_norm: 13.266509 [I 2022-10-10 00:12:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 1770/2304 [I 2022-10-10 00:12:16] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:33 [I 2022-10-10 00:12:16] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00301 [I 2022-10-10 00:12:16] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-10 00:12:16] ...optimizers/trijoint.py.151: total_norm: 24.117313 [I 2022-10-10 00:12:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1780/2304 [I 2022-10-10 00:12:25] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:22 [I 2022-10-10 00:12:25] ...trap/engines/engine.py.229: process: 0.82762 | load: 0.00301 [I 2022-10-10 00:12:25] ...trap/engines/engine.py.230: loss: 0.12624 [I 2022-10-10 00:12:25] ...optimizers/trijoint.py.151: total_norm: 12.036498 [I 2022-10-10 00:12:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 1790/2304 [I 2022-10-10 00:12:33] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:17 [I 2022-10-10 00:12:33] ...trap/engines/engine.py.229: process: 0.86239 | load: 0.00299 [I 2022-10-10 00:12:33] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-10 00:12:33] ...optimizers/trijoint.py.151: total_norm: 9.325352 [I 2022-10-10 00:12:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1800/2304 [I 2022-10-10 00:12:42] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:11 [I 2022-10-10 00:12:42] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00302 [I 2022-10-10 00:12:42] ...trap/engines/engine.py.230: loss: 0.16373 [I 2022-10-10 00:12:42] ...optimizers/trijoint.py.151: total_norm: 12.451556 [I 2022-10-10 00:12:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 1810/2304 [I 2022-10-10 00:12:50] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:06:56 [I 2022-10-10 00:12:50] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00299 [I 2022-10-10 00:12:50] ...trap/engines/engine.py.230: loss: 0.17710 [I 2022-10-10 00:12:50] ...optimizers/trijoint.py.151: total_norm: 18.130867 [I 2022-10-10 00:12:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 1820/2304 [I 2022-10-10 00:12:59] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:50 [I 2022-10-10 00:12:59] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00309 [I 2022-10-10 00:12:59] ...trap/engines/engine.py.230: loss: 0.12688 [I 2022-10-10 00:12:59] ...optimizers/trijoint.py.151: total_norm: 9.331994 [I 2022-10-10 00:13:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1830/2304 [I 2022-10-10 00:13:08] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:43 [I 2022-10-10 00:13:08] ...trap/engines/engine.py.229: process: 0.83739 | load: 0.00297 [I 2022-10-10 00:13:08] ...trap/engines/engine.py.230: loss: 0.17360 [I 2022-10-10 00:13:08] ...optimizers/trijoint.py.151: total_norm: 23.198593 [I 2022-10-10 00:13:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 1840/2304 [I 2022-10-10 00:13:16] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:33 [I 2022-10-10 00:13:16] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00295 [I 2022-10-10 00:13:16] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-10 00:13:16] ...optimizers/trijoint.py.151: total_norm: 8.564961 [I 2022-10-10 00:13:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1850/2304 [I 2022-10-10 00:13:25] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:25 [I 2022-10-10 00:13:25] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00314 [I 2022-10-10 00:13:25] ...trap/engines/engine.py.230: loss: 0.18924 [I 2022-10-10 00:13:25] ...optimizers/trijoint.py.151: total_norm: 12.279038 [I 2022-10-10 00:13:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 1860/2304 [I 2022-10-10 00:13:33] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:17 [I 2022-10-10 00:13:33] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00315 [I 2022-10-10 00:13:33] ...trap/engines/engine.py.230: loss: 0.14386 [I 2022-10-10 00:13:33] ...optimizers/trijoint.py.151: total_norm: 12.346327 [I 2022-10-10 00:13:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1870/2304 [I 2022-10-10 00:13:42] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:08 [I 2022-10-10 00:13:42] ...trap/engines/engine.py.229: process: 0.86262 | load: 0.00315 [I 2022-10-10 00:13:42] ...trap/engines/engine.py.230: loss: 0.17572 [I 2022-10-10 00:13:42] ...optimizers/trijoint.py.151: total_norm: 22.634556 [I 2022-10-10 00:13:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 1880/2304 [I 2022-10-10 00:13:50] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:02 [I 2022-10-10 00:13:50] ...trap/engines/engine.py.229: process: 0.86285 | load: 0.00313 [I 2022-10-10 00:13:50] ...trap/engines/engine.py.230: loss: 0.14151 [I 2022-10-10 00:13:50] ...optimizers/trijoint.py.151: total_norm: 9.310284 [I 2022-10-10 00:13:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 1890/2304 [I 2022-10-10 00:13:59] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:52 [I 2022-10-10 00:13:59] ...trap/engines/engine.py.229: process: 0.84800 | load: 0.00301 [I 2022-10-10 00:13:59] ...trap/engines/engine.py.230: loss: 0.17538 [I 2022-10-10 00:13:59] ...optimizers/trijoint.py.151: total_norm: 14.391701 [I 2022-10-10 00:14:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 1900/2304 [I 2022-10-10 00:14:07] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:43 [I 2022-10-10 00:14:07] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00300 [I 2022-10-10 00:14:07] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-10 00:14:07] ...optimizers/trijoint.py.151: total_norm: 13.499454 [I 2022-10-10 00:14:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 1910/2304 [I 2022-10-10 00:14:16] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:32 [I 2022-10-10 00:14:16] ...trap/engines/engine.py.229: process: 0.82505 | load: 0.00308 [I 2022-10-10 00:14:16] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-10 00:14:16] ...optimizers/trijoint.py.151: total_norm: 10.224574 [I 2022-10-10 00:14:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 1920/2304 [I 2022-10-10 00:14:24] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:26 [I 2022-10-10 00:14:24] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00310 [I 2022-10-10 00:14:24] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-10 00:14:24] ...optimizers/trijoint.py.151: total_norm: 9.481849 [I 2022-10-10 00:14:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 1930/2304 [I 2022-10-10 00:14:33] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:15 [I 2022-10-10 00:14:33] ...trap/engines/engine.py.229: process: 0.83157 | load: 0.00318 [I 2022-10-10 00:14:33] ...trap/engines/engine.py.230: loss: 0.27463 [I 2022-10-10 00:14:33] ...optimizers/trijoint.py.151: total_norm: 27.491070 [I 2022-10-10 00:14:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 1940/2304 [I 2022-10-10 00:14:41] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:10 [I 2022-10-10 00:14:41] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00297 [I 2022-10-10 00:14:41] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-10 00:14:41] ...optimizers/trijoint.py.151: total_norm: 20.771000 [I 2022-10-10 00:14:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 1950/2304 [I 2022-10-10 00:14:50] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:01 [I 2022-10-10 00:14:50] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00325 [I 2022-10-10 00:14:50] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-10 00:14:50] ...optimizers/trijoint.py.151: total_norm: 22.276350 [I 2022-10-10 00:14:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 1960/2304 [I 2022-10-10 00:14:58] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:52 [I 2022-10-10 00:14:58] ...trap/engines/engine.py.229: process: 0.83932 | load: 0.00308 [I 2022-10-10 00:14:58] ...trap/engines/engine.py.230: loss: 0.21149 [I 2022-10-10 00:14:58] ...optimizers/trijoint.py.151: total_norm: 12.029894 [I 2022-10-10 00:15:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 1970/2304 [I 2022-10-10 00:15:07] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:42 [I 2022-10-10 00:15:07] ...trap/engines/engine.py.229: process: 0.82626 | load: 0.00303 [I 2022-10-10 00:15:07] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-10 00:15:07] ...optimizers/trijoint.py.151: total_norm: 7.691416 [I 2022-10-10 00:15:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 1980/2304 [I 2022-10-10 00:15:15] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:34 [I 2022-10-10 00:15:15] ...trap/engines/engine.py.229: process: 0.85979 | load: 0.00293 [I 2022-10-10 00:15:15] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-10 00:15:15] ...optimizers/trijoint.py.151: total_norm: 24.141350 [I 2022-10-10 00:15:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 1990/2304 [I 2022-10-10 00:15:24] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:26 [I 2022-10-10 00:15:24] ...trap/engines/engine.py.229: process: 0.85472 | load: 0.00309 [I 2022-10-10 00:15:24] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-10 00:15:24] ...optimizers/trijoint.py.151: total_norm: 12.507560 [I 2022-10-10 00:15:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 2000/2304 [I 2022-10-10 00:15:32] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:18 [I 2022-10-10 00:15:32] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00292 [I 2022-10-10 00:15:32] ...trap/engines/engine.py.230: loss: 0.19486 [I 2022-10-10 00:15:32] ...optimizers/trijoint.py.151: total_norm: 24.203030 [I 2022-10-10 00:15:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 2010/2304 [I 2022-10-10 00:15:41] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:09 [I 2022-10-10 00:15:41] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00309 [I 2022-10-10 00:15:41] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-10 00:15:41] ...optimizers/trijoint.py.151: total_norm: 16.125851 [I 2022-10-10 00:15:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 2020/2304 [I 2022-10-10 00:15:49] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:01 [I 2022-10-10 00:15:49] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00310 [I 2022-10-10 00:15:49] ...trap/engines/engine.py.230: loss: 0.09764 [I 2022-10-10 00:15:49] ...optimizers/trijoint.py.151: total_norm: 12.285484 [I 2022-10-10 00:15:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 2030/2304 [I 2022-10-10 00:15:58] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:52 [I 2022-10-10 00:15:58] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00317 [I 2022-10-10 00:15:58] ...trap/engines/engine.py.230: loss: 0.20227 [I 2022-10-10 00:15:58] ...optimizers/trijoint.py.151: total_norm: 16.150654 [I 2022-10-10 00:16:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 2040/2304 [I 2022-10-10 00:16:06] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:43 [I 2022-10-10 00:16:06] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00303 [I 2022-10-10 00:16:06] ...trap/engines/engine.py.230: loss: 0.12755 [I 2022-10-10 00:16:06] ...optimizers/trijoint.py.151: total_norm: 25.193569 [I 2022-10-10 00:16:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 2050/2304 [I 2022-10-10 00:16:15] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:35 [I 2022-10-10 00:16:15] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00297 [I 2022-10-10 00:16:15] ...trap/engines/engine.py.230: loss: 0.13196 [I 2022-10-10 00:16:15] ...optimizers/trijoint.py.151: total_norm: 12.854156 [I 2022-10-10 00:16:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 2060/2304 [I 2022-10-10 00:16:23] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:27 [I 2022-10-10 00:16:23] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00312 [I 2022-10-10 00:16:23] ...trap/engines/engine.py.230: loss: 0.09890 [I 2022-10-10 00:16:23] ...optimizers/trijoint.py.151: total_norm: 9.201498 [I 2022-10-10 00:16:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 2070/2304 [I 2022-10-10 00:16:32] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:19 [I 2022-10-10 00:16:32] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00308 [I 2022-10-10 00:16:32] ...trap/engines/engine.py.230: loss: 0.17795 [I 2022-10-10 00:16:32] ...optimizers/trijoint.py.151: total_norm: 16.399353 [I 2022-10-10 00:16:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 2080/2304 [I 2022-10-10 00:16:40] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:12 [I 2022-10-10 00:16:40] ...trap/engines/engine.py.229: process: 0.87704 | load: 0.00316 [I 2022-10-10 00:16:40] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-10 00:16:40] ...optimizers/trijoint.py.151: total_norm: 19.317057 [I 2022-10-10 00:16:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 2090/2304 [I 2022-10-10 00:16:49] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:03 [I 2022-10-10 00:16:49] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00309 [I 2022-10-10 00:16:49] ...trap/engines/engine.py.230: loss: 0.16786 [I 2022-10-10 00:16:49] ...optimizers/trijoint.py.151: total_norm: 27.117899 [I 2022-10-10 00:16:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 2100/2304 [I 2022-10-10 00:16:57] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:02:54 [I 2022-10-10 00:16:57] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00315 [I 2022-10-10 00:16:57] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-10 00:16:57] ...optimizers/trijoint.py.151: total_norm: 9.040279 [I 2022-10-10 00:17:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 2110/2304 [I 2022-10-10 00:17:06] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:02:46 [I 2022-10-10 00:17:06] ...trap/engines/engine.py.229: process: 0.86070 | load: 0.00312 [I 2022-10-10 00:17:06] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-10 00:17:06] ...optimizers/trijoint.py.151: total_norm: 19.990578 [I 2022-10-10 00:17:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 2120/2304 [I 2022-10-10 00:17:15] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:37 [I 2022-10-10 00:17:15] ...trap/engines/engine.py.229: process: 0.86059 | load: 0.00313 [I 2022-10-10 00:17:15] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-10 00:17:15] ...optimizers/trijoint.py.151: total_norm: 12.416966 [I 2022-10-10 00:17:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 2130/2304 [I 2022-10-10 00:17:23] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:28 [I 2022-10-10 00:17:23] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00313 [I 2022-10-10 00:17:23] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-10 00:17:23] ...optimizers/trijoint.py.151: total_norm: 10.249702 [I 2022-10-10 00:17:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 2140/2304 [I 2022-10-10 00:17:32] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:20 [I 2022-10-10 00:17:32] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00312 [I 2022-10-10 00:17:32] ...trap/engines/engine.py.230: loss: 0.25619 [I 2022-10-10 00:17:32] ...optimizers/trijoint.py.151: total_norm: 32.600552 [I 2022-10-10 00:17:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 2150/2304 [I 2022-10-10 00:17:40] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:11 [I 2022-10-10 00:17:40] ...trap/engines/engine.py.229: process: 0.85653 | load: 0.00312 [I 2022-10-10 00:17:40] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-10 00:17:40] ...optimizers/trijoint.py.151: total_norm: 29.714191 [I 2022-10-10 00:17:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 2160/2304 [I 2022-10-10 00:17:49] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:03 [I 2022-10-10 00:17:49] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00316 [I 2022-10-10 00:17:49] ...trap/engines/engine.py.230: loss: 0.16355 [I 2022-10-10 00:17:49] ...optimizers/trijoint.py.151: total_norm: 21.038441 [I 2022-10-10 00:17:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 2170/2304 [I 2022-10-10 00:17:58] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:54 [I 2022-10-10 00:17:58] ...trap/engines/engine.py.229: process: 0.85352 | load: 0.00353 [I 2022-10-10 00:17:58] ...trap/engines/engine.py.230: loss: 0.16992 [I 2022-10-10 00:17:58] ...optimizers/trijoint.py.151: total_norm: 7.914412 [I 2022-10-10 00:18:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 2180/2304 [I 2022-10-10 00:18:06] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:01:45 [I 2022-10-10 00:18:06] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00312 [I 2022-10-10 00:18:06] ...trap/engines/engine.py.230: loss: 0.14993 [I 2022-10-10 00:18:06] ...optimizers/trijoint.py.151: total_norm: 7.877609 [I 2022-10-10 00:18:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 2190/2304 [I 2022-10-10 00:18:15] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:37 [I 2022-10-10 00:18:15] ...trap/engines/engine.py.229: process: 0.85950 | load: 0.00315 [I 2022-10-10 00:18:15] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-10 00:18:15] ...optimizers/trijoint.py.151: total_norm: 12.233975 [I 2022-10-10 00:18:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 2200/2304 [I 2022-10-10 00:18:23] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:29 [I 2022-10-10 00:18:23] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00311 [I 2022-10-10 00:18:23] ...trap/engines/engine.py.230: loss: 0.24614 [I 2022-10-10 00:18:23] ...optimizers/trijoint.py.151: total_norm: 22.786535 [I 2022-10-10 00:18:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 2210/2304 [I 2022-10-10 00:18:32] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:20 [I 2022-10-10 00:18:32] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00313 [I 2022-10-10 00:18:32] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-10 00:18:32] ...optimizers/trijoint.py.151: total_norm: 11.474031 [I 2022-10-10 00:18:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 2220/2304 [I 2022-10-10 00:18:40] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:11 [I 2022-10-10 00:18:40] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00312 [I 2022-10-10 00:18:40] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-10 00:18:40] ...optimizers/trijoint.py.151: total_norm: 33.220600 [I 2022-10-10 00:18:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 2230/2304 [I 2022-10-10 00:18:49] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:03 [I 2022-10-10 00:18:49] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00311 [I 2022-10-10 00:18:49] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-10 00:18:49] ...optimizers/trijoint.py.151: total_norm: 22.789459 [I 2022-10-10 00:18:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 2240/2304 [I 2022-10-10 00:18:58] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:54 [I 2022-10-10 00:18:58] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00313 [I 2022-10-10 00:18:58] ...trap/engines/engine.py.230: loss: 0.17600 [I 2022-10-10 00:18:58] ...optimizers/trijoint.py.151: total_norm: 14.224010 [I 2022-10-10 00:19:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 2250/2304 [I 2022-10-10 00:19:06] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:46 [I 2022-10-10 00:19:06] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00312 [I 2022-10-10 00:19:06] ...trap/engines/engine.py.230: loss: 0.18630 [I 2022-10-10 00:19:06] ...optimizers/trijoint.py.151: total_norm: 19.045576 [I 2022-10-10 00:19:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 2260/2304 [I 2022-10-10 00:19:15] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:37 [I 2022-10-10 00:19:15] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00315 [I 2022-10-10 00:19:15] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-10 00:19:15] ...optimizers/trijoint.py.151: total_norm: 18.924736 [I 2022-10-10 00:19:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 2270/2304 [I 2022-10-10 00:19:23] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:29 [I 2022-10-10 00:19:23] ...trap/engines/engine.py.229: process: 0.86621 | load: 0.00312 [I 2022-10-10 00:19:23] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-10 00:19:23] ...optimizers/trijoint.py.151: total_norm: 20.838470 [I 2022-10-10 00:19:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 2280/2304 [I 2022-10-10 00:19:32] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:20 [I 2022-10-10 00:19:32] ...trap/engines/engine.py.229: process: 0.84366 | load: 0.00296 [I 2022-10-10 00:19:32] ...trap/engines/engine.py.230: loss: 0.19409 [I 2022-10-10 00:19:32] ...optimizers/trijoint.py.151: total_norm: 22.810064 [I 2022-10-10 00:19:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 2290/2304 [I 2022-10-10 00:19:40] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:11 [I 2022-10-10 00:19:40] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00311 [I 2022-10-10 00:19:40] ...trap/engines/engine.py.230: loss: 0.18128 [I 2022-10-10 00:19:40] ...optimizers/trijoint.py.151: total_norm: 11.957684 [I 2022-10-10 00:19:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 2300/2304 [I 2022-10-10 00:19:49] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:03 [I 2022-10-10 00:19:49] ...trap/engines/engine.py.229: process: 0.83580 | load: 0.00017 [I 2022-10-10 00:19:49] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-10 00:19:49] ...optimizers/trijoint.py.151: total_norm: 14.245829 [I 2022-10-10 00:19:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 2304/2304 [I 2022-10-10 00:19:52] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:00 [I 2022-10-10 00:19:52] ...trap/engines/engine.py.229: process: 0.84033 | load: 0.00018 [I 2022-10-10 00:19:52] ...trap/engines/engine.py.230: loss: 0.17236 [I 2022-10-10 00:19:52] ...optimizers/trijoint.py.151: total_norm: 31.493513 [S 2022-10-10 00:19:53] ...trap/engines/engine.py.129: train_epoch.epoch: 59 [S 2022-10-10 00:19:53] ...trap/engines/engine.py.129: train_epoch.loss: 0.14920634811586517 [I 2022-10-10 00:19:53] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 00:19:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 59 [I 2022-10-10 00:19:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 00:19:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 00:19:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 00:19:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 00:19:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 00:19:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 0/510 [I 2022-10-10 00:19:59] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:13 [I 2022-10-10 00:19:59] ...trap/engines/engine.py.326: process: 4.49670 | load: 3.81812 [I 2022-10-10 00:20:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 10/510 [I 2022-10-10 00:20:06] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:11 [I 2022-10-10 00:20:06] ...trap/engines/engine.py.326: process: 0.74306 | load: 0.00020 [I 2022-10-10 00:20:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 20/510 [I 2022-10-10 00:20:12] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:51 [I 2022-10-10 00:20:12] ...trap/engines/engine.py.326: process: 0.82917 | load: 0.00022 [I 2022-10-10 00:20:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 30/510 [I 2022-10-10 00:20:17] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:08 [I 2022-10-10 00:20:17] ...trap/engines/engine.py.326: process: 0.47654 | load: 0.00019 [I 2022-10-10 00:20:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 40/510 [I 2022-10-10 00:20:23] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:49 [I 2022-10-10 00:20:23] ...trap/engines/engine.py.326: process: 0.57346 | load: 0.00020 [I 2022-10-10 00:20:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 50/510 [I 2022-10-10 00:20:29] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:04:18 [I 2022-10-10 00:20:29] ...trap/engines/engine.py.326: process: 0.55914 | load: 0.00018 [I 2022-10-10 00:20:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 60/510 [I 2022-10-10 00:20:35] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:03 [I 2022-10-10 00:20:35] ...trap/engines/engine.py.326: process: 0.58646 | load: 0.00019 [I 2022-10-10 00:20:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 70/510 [I 2022-10-10 00:20:40] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:33 [I 2022-10-10 00:20:40] ...trap/engines/engine.py.326: process: 0.42522 | load: 0.00025 [I 2022-10-10 00:20:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 80/510 [I 2022-10-10 00:20:44] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:37 [I 2022-10-10 00:20:44] ...trap/engines/engine.py.326: process: 0.49665 | load: 0.00021 [I 2022-10-10 00:20:49] ...trap/engines/engine.py.321: eval: epoch 59 | batch 90/510 [I 2022-10-10 00:20:49] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:24 [I 2022-10-10 00:20:49] ...trap/engines/engine.py.326: process: 0.47444 | load: 0.00024 [I 2022-10-10 00:20:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 100/510 [I 2022-10-10 00:20:54] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:03:04 [I 2022-10-10 00:20:54] ...trap/engines/engine.py.326: process: 0.30827 | load: 0.00020 [I 2022-10-10 00:20:57] ...trap/engines/engine.py.321: eval: epoch 59 | batch 110/510 [I 2022-10-10 00:20:57] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:12 [I 2022-10-10 00:20:57] ...trap/engines/engine.py.326: process: 0.31929 | load: 0.00019 [I 2022-10-10 00:21:00] ...trap/engines/engine.py.321: eval: epoch 59 | batch 120/510 [I 2022-10-10 00:21:00] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:04 [I 2022-10-10 00:21:00] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-10 00:21:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 130/510 [I 2022-10-10 00:21:03] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:00 [I 2022-10-10 00:21:03] ...trap/engines/engine.py.326: process: 0.31836 | load: 0.00020 [I 2022-10-10 00:21:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 140/510 [I 2022-10-10 00:21:07] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:57 [I 2022-10-10 00:21:07] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00019 [I 2022-10-10 00:21:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 150/510 [I 2022-10-10 00:21:10] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:54 [I 2022-10-10 00:21:10] ...trap/engines/engine.py.326: process: 0.31937 | load: 0.00018 [I 2022-10-10 00:21:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 160/510 [I 2022-10-10 00:21:13] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:51 [I 2022-10-10 00:21:13] ...trap/engines/engine.py.326: process: 0.31773 | load: 0.00018 [I 2022-10-10 00:21:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 170/510 [I 2022-10-10 00:21:16] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:47 [I 2022-10-10 00:21:16] ...trap/engines/engine.py.326: process: 0.31358 | load: 0.00019 [I 2022-10-10 00:21:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 180/510 [I 2022-10-10 00:21:19] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:45 [I 2022-10-10 00:21:19] ...trap/engines/engine.py.326: process: 0.32042 | load: 0.00020 [I 2022-10-10 00:21:22] ...trap/engines/engine.py.321: eval: epoch 59 | batch 190/510 [I 2022-10-10 00:21:22] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:41 [I 2022-10-10 00:21:22] ...trap/engines/engine.py.326: process: 0.31422 | load: 0.00019 [I 2022-10-10 00:21:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 200/510 [I 2022-10-10 00:21:26] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:38 [I 2022-10-10 00:21:26] ...trap/engines/engine.py.326: process: 0.31727 | load: 0.00022 [I 2022-10-10 00:21:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 210/510 [I 2022-10-10 00:21:29] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:34 [I 2022-10-10 00:21:29] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00020 [I 2022-10-10 00:21:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 220/510 [I 2022-10-10 00:21:32] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:32 [I 2022-10-10 00:21:32] ...trap/engines/engine.py.326: process: 0.31431 | load: 0.00021 [I 2022-10-10 00:21:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 230/510 [I 2022-10-10 00:21:35] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:29 [I 2022-10-10 00:21:35] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00020 [I 2022-10-10 00:21:38] ...trap/engines/engine.py.321: eval: epoch 59 | batch 240/510 [I 2022-10-10 00:21:38] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:25 [I 2022-10-10 00:21:38] ...trap/engines/engine.py.326: process: 0.31430 | load: 0.00022 [I 2022-10-10 00:21:42] ...trap/engines/engine.py.321: eval: epoch 59 | batch 250/510 [I 2022-10-10 00:21:42] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:22 [I 2022-10-10 00:21:42] ...trap/engines/engine.py.326: process: 0.31865 | load: 0.00020 [I 2022-10-10 00:21:45] ...trap/engines/engine.py.321: eval: epoch 59 | batch 260/510 [I 2022-10-10 00:21:45] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:19 [I 2022-10-10 00:21:45] ...trap/engines/engine.py.326: process: 0.31070 | load: 0.00020 [I 2022-10-10 00:21:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 270/510 [I 2022-10-10 00:21:48] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:16 [I 2022-10-10 00:21:48] ...trap/engines/engine.py.326: process: 0.31860 | load: 0.00019 [I 2022-10-10 00:21:51] ...trap/engines/engine.py.321: eval: epoch 59 | batch 280/510 [I 2022-10-10 00:21:51] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:13 [I 2022-10-10 00:21:51] ...trap/engines/engine.py.326: process: 0.32385 | load: 0.00021 [I 2022-10-10 00:21:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 290/510 [I 2022-10-10 00:21:54] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:10 [I 2022-10-10 00:21:54] ...trap/engines/engine.py.326: process: 0.31597 | load: 0.00021 [I 2022-10-10 00:21:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 300/510 [I 2022-10-10 00:21:58] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:07 [I 2022-10-10 00:21:58] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00020 [I 2022-10-10 00:22:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 310/510 [I 2022-10-10 00:22:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:03 [I 2022-10-10 00:22:01] ...trap/engines/engine.py.326: process: 0.31475 | load: 0.00021 [I 2022-10-10 00:22:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 320/510 [I 2022-10-10 00:22:04] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:00 [I 2022-10-10 00:22:04] ...trap/engines/engine.py.326: process: 0.32116 | load: 0.00021 [I 2022-10-10 00:22:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 330/510 [I 2022-10-10 00:22:07] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:57 [I 2022-10-10 00:22:07] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00021 [I 2022-10-10 00:22:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 340/510 [I 2022-10-10 00:22:10] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:54 [I 2022-10-10 00:22:10] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00021 [I 2022-10-10 00:22:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 350/510 [I 2022-10-10 00:22:14] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:51 [I 2022-10-10 00:22:14] ...trap/engines/engine.py.326: process: 0.32220 | load: 0.00020 [I 2022-10-10 00:22:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 360/510 [I 2022-10-10 00:22:17] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:47 [I 2022-10-10 00:22:17] ...trap/engines/engine.py.326: process: 0.31555 | load: 0.00020 [I 2022-10-10 00:22:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 370/510 [I 2022-10-10 00:22:20] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-10 00:22:20] ...trap/engines/engine.py.326: process: 0.32468 | load: 0.00020 [I 2022-10-10 00:22:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 380/510 [I 2022-10-10 00:22:23] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-10 00:22:23] ...trap/engines/engine.py.326: process: 0.31540 | load: 0.00022 [I 2022-10-10 00:22:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 390/510 [I 2022-10-10 00:22:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-10 00:22:26] ...trap/engines/engine.py.326: process: 0.32088 | load: 0.00020 [I 2022-10-10 00:22:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 400/510 [I 2022-10-10 00:22:30] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:35 [I 2022-10-10 00:22:30] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00019 [I 2022-10-10 00:22:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 410/510 [I 2022-10-10 00:22:33] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-10 00:22:33] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00024 [I 2022-10-10 00:22:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 420/510 [I 2022-10-10 00:22:36] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-10 00:22:36] ...trap/engines/engine.py.326: process: 0.32104 | load: 0.00020 [I 2022-10-10 00:22:39] ...trap/engines/engine.py.321: eval: epoch 59 | batch 430/510 [I 2022-10-10 00:22:39] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-10 00:22:39] ...trap/engines/engine.py.326: process: 0.32708 | load: 0.00020 [I 2022-10-10 00:22:42] ...trap/engines/engine.py.321: eval: epoch 59 | batch 440/510 [I 2022-10-10 00:22:42] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:22 [I 2022-10-10 00:22:42] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00021 [I 2022-10-10 00:22:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 450/510 [I 2022-10-10 00:22:46] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:19 [I 2022-10-10 00:22:46] ...trap/engines/engine.py.326: process: 0.33195 | load: 0.00023 [I 2022-10-10 00:22:49] ...trap/engines/engine.py.321: eval: epoch 59 | batch 460/510 [I 2022-10-10 00:22:49] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:16 [I 2022-10-10 00:22:49] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00020 [I 2022-10-10 00:22:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 470/510 [I 2022-10-10 00:22:52] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-10 00:22:52] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00023 [I 2022-10-10 00:22:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 480/510 [I 2022-10-10 00:22:55] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-10 00:22:55] ...trap/engines/engine.py.326: process: 0.32345 | load: 0.00020 [I 2022-10-10 00:22:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 490/510 [I 2022-10-10 00:22:58] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:06 [I 2022-10-10 00:22:58] ...trap/engines/engine.py.326: process: 0.32104 | load: 0.00025 [I 2022-10-10 00:23:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 500/510 [I 2022-10-10 00:23:02] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:03 [I 2022-10-10 00:23:02] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00020 [I 2022-10-10 00:23:05] ...trap/engines/engine.py.321: eval: epoch 59 | batch 510/510 [I 2022-10-10 00:23:05] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:00 [I 2022-10-10 00:23:05] ...trap/engines/engine.py.326: process: 0.31921 | load: 0.00015 [S 2022-10-10 00:23:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 59 [I 2022-10-10 00:23:05] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.725000 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904900 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936500 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724000 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906600 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935400 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015479 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007273 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005296 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013762 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007632 [S 2022-10-10 00:23:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007200 [I 2022-10-10 00:23:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 00:23:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 00:23:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 00:23:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 00:23:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 00:23:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 00:23:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 00:23:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 00:23:52] ...trap/engines/engine.py.163: Training model on trainset for epoch 60 [I 2022-10-10 00:23:52] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 00:23:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 0/2304 [I 2022-10-10 00:23:55] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:53 [I 2022-10-10 00:23:55] ...trap/engines/engine.py.229: process: 2.99205 | load: 2.08373 [I 2022-10-10 00:23:55] ...trap/engines/engine.py.230: loss: 0.13446 [I 2022-10-10 00:23:55] ...optimizers/trijoint.py.151: total_norm: 20.031466 [I 2022-10-10 00:24:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 10/2304 [I 2022-10-10 00:24:04] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:59 [I 2022-10-10 00:24:04] ...trap/engines/engine.py.229: process: 0.83029 | load: 0.00570 [I 2022-10-10 00:24:04] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-10 00:24:04] ...optimizers/trijoint.py.151: total_norm: 11.333263 [I 2022-10-10 00:24:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 20/2304 [I 2022-10-10 00:24:12] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:21 [I 2022-10-10 00:24:12] ...trap/engines/engine.py.229: process: 0.84583 | load: 0.00574 [I 2022-10-10 00:24:12] ...trap/engines/engine.py.230: loss: 0.15115 [I 2022-10-10 00:24:12] ...optimizers/trijoint.py.151: total_norm: 8.975731 [I 2022-10-10 00:24:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 30/2304 [I 2022-10-10 00:24:21] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:29 [I 2022-10-10 00:24:21] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00381 [I 2022-10-10 00:24:21] ...trap/engines/engine.py.230: loss: 0.16712 [I 2022-10-10 00:24:21] ...optimizers/trijoint.py.151: total_norm: 30.778925 [I 2022-10-10 00:24:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 40/2304 [I 2022-10-10 00:24:30] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:15 [I 2022-10-10 00:24:30] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00491 [I 2022-10-10 00:24:30] ...trap/engines/engine.py.230: loss: 0.13011 [I 2022-10-10 00:24:30] ...optimizers/trijoint.py.151: total_norm: 18.635677 [I 2022-10-10 00:24:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 50/2304 [I 2022-10-10 00:24:38] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:58 [I 2022-10-10 00:24:38] ...trap/engines/engine.py.229: process: 0.85782 | load: 0.00409 [I 2022-10-10 00:24:38] ...trap/engines/engine.py.230: loss: 0.15834 [I 2022-10-10 00:24:38] ...optimizers/trijoint.py.151: total_norm: 34.877987 [I 2022-10-10 00:24:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 60/2304 [I 2022-10-10 00:24:47] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:55 [I 2022-10-10 00:24:47] ...trap/engines/engine.py.229: process: 0.85537 | load: 0.00438 [I 2022-10-10 00:24:47] ...trap/engines/engine.py.230: loss: 0.18621 [I 2022-10-10 00:24:47] ...optimizers/trijoint.py.151: total_norm: 11.565681 [I 2022-10-10 00:24:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 70/2304 [I 2022-10-10 00:24:55] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:47 [I 2022-10-10 00:24:55] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00358 [I 2022-10-10 00:24:55] ...trap/engines/engine.py.230: loss: 0.11780 [I 2022-10-10 00:24:55] ...optimizers/trijoint.py.151: total_norm: 21.918011 [I 2022-10-10 00:25:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 80/2304 [I 2022-10-10 00:25:04] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:45 [I 2022-10-10 00:25:04] ...trap/engines/engine.py.229: process: 0.85756 | load: 0.00335 [I 2022-10-10 00:25:04] ...trap/engines/engine.py.230: loss: 0.22271 [I 2022-10-10 00:25:04] ...optimizers/trijoint.py.151: total_norm: 15.121075 [I 2022-10-10 00:25:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 90/2304 [I 2022-10-10 00:25:12] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:24 [I 2022-10-10 00:25:12] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00381 [I 2022-10-10 00:25:12] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-10 00:25:12] ...optimizers/trijoint.py.151: total_norm: 15.305320 [I 2022-10-10 00:25:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 100/2304 [I 2022-10-10 00:25:21] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:07 [I 2022-10-10 00:25:21] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00352 [I 2022-10-10 00:25:21] ...trap/engines/engine.py.230: loss: 0.20656 [I 2022-10-10 00:25:21] ...optimizers/trijoint.py.151: total_norm: 37.115467 [I 2022-10-10 00:25:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 110/2304 [I 2022-10-10 00:25:29] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:09 [I 2022-10-10 00:25:29] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00312 [I 2022-10-10 00:25:29] ...trap/engines/engine.py.230: loss: 0.18649 [I 2022-10-10 00:25:29] ...optimizers/trijoint.py.151: total_norm: 14.827333 [I 2022-10-10 00:25:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 120/2304 [I 2022-10-10 00:25:38] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:18 [I 2022-10-10 00:25:38] ...trap/engines/engine.py.229: process: 0.88206 | load: 0.00333 [I 2022-10-10 00:25:38] ...trap/engines/engine.py.230: loss: 0.14608 [I 2022-10-10 00:25:38] ...optimizers/trijoint.py.151: total_norm: 15.677365 [I 2022-10-10 00:25:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 130/2304 [I 2022-10-10 00:25:46] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:00 [I 2022-10-10 00:25:46] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00349 [I 2022-10-10 00:25:46] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-10 00:25:46] ...optimizers/trijoint.py.151: total_norm: 19.047266 [I 2022-10-10 00:25:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 140/2304 [I 2022-10-10 00:25:55] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:32 [I 2022-10-10 00:25:55] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00327 [I 2022-10-10 00:25:55] ...trap/engines/engine.py.230: loss: 0.10118 [I 2022-10-10 00:25:55] ...optimizers/trijoint.py.151: total_norm: 7.103997 [I 2022-10-10 00:26:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 150/2304 [I 2022-10-10 00:26:03] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:32 [I 2022-10-10 00:26:03] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00313 [I 2022-10-10 00:26:03] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-10 00:26:03] ...optimizers/trijoint.py.151: total_norm: 7.089751 [I 2022-10-10 00:26:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 160/2304 [I 2022-10-10 00:26:12] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:42 [I 2022-10-10 00:26:12] ...trap/engines/engine.py.229: process: 0.86201 | load: 0.00315 [I 2022-10-10 00:26:12] ...trap/engines/engine.py.230: loss: 0.12671 [I 2022-10-10 00:26:12] ...optimizers/trijoint.py.151: total_norm: 16.365463 [I 2022-10-10 00:26:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 170/2304 [I 2022-10-10 00:26:21] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:30 [I 2022-10-10 00:26:21] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00344 [I 2022-10-10 00:26:21] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-10 00:26:21] ...optimizers/trijoint.py.151: total_norm: 21.035679 [I 2022-10-10 00:26:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 180/2304 [I 2022-10-10 00:26:29] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:10 [I 2022-10-10 00:26:29] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00311 [I 2022-10-10 00:26:29] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-10 00:26:29] ...optimizers/trijoint.py.151: total_norm: 11.866838 [I 2022-10-10 00:26:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 190/2304 [I 2022-10-10 00:26:38] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:59 [I 2022-10-10 00:26:38] ...trap/engines/engine.py.229: process: 0.85732 | load: 0.00311 [I 2022-10-10 00:26:38] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-10 00:26:38] ...optimizers/trijoint.py.151: total_norm: 11.221481 [I 2022-10-10 00:26:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 200/2304 [I 2022-10-10 00:26:46] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:57 [I 2022-10-10 00:26:46] ...trap/engines/engine.py.229: process: 0.86088 | load: 0.00330 [I 2022-10-10 00:26:46] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-10 00:26:46] ...optimizers/trijoint.py.151: total_norm: 34.244930 [I 2022-10-10 00:26:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 210/2304 [I 2022-10-10 00:26:55] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:41 [I 2022-10-10 00:26:55] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00316 [I 2022-10-10 00:26:55] ...trap/engines/engine.py.230: loss: 0.18259 [I 2022-10-10 00:26:55] ...optimizers/trijoint.py.151: total_norm: 19.235132 [I 2022-10-10 00:27:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 220/2304 [I 2022-10-10 00:27:03] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:34 [I 2022-10-10 00:27:03] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00326 [I 2022-10-10 00:27:03] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-10 00:27:03] ...optimizers/trijoint.py.151: total_norm: 45.308029 [I 2022-10-10 00:27:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 230/2304 [I 2022-10-10 00:27:12] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:32 [I 2022-10-10 00:27:12] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00309 [I 2022-10-10 00:27:12] ...trap/engines/engine.py.230: loss: 0.22942 [I 2022-10-10 00:27:12] ...optimizers/trijoint.py.151: total_norm: 73.834129 [I 2022-10-10 00:27:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 240/2304 [I 2022-10-10 00:27:21] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:17 [I 2022-10-10 00:27:21] ...trap/engines/engine.py.229: process: 0.84137 | load: 0.00308 [I 2022-10-10 00:27:21] ...trap/engines/engine.py.230: loss: 0.16429 [I 2022-10-10 00:27:21] ...optimizers/trijoint.py.151: total_norm: 19.212358 [I 2022-10-10 00:27:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 250/2304 [I 2022-10-10 00:27:29] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:09 [I 2022-10-10 00:27:29] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00329 [I 2022-10-10 00:27:29] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-10 00:27:29] ...optimizers/trijoint.py.151: total_norm: 21.307014 [I 2022-10-10 00:27:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 260/2304 [I 2022-10-10 00:27:38] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:05 [I 2022-10-10 00:27:38] ...trap/engines/engine.py.229: process: 0.86434 | load: 0.00336 [I 2022-10-10 00:27:38] ...trap/engines/engine.py.230: loss: 0.10151 [I 2022-10-10 00:27:38] ...optimizers/trijoint.py.151: total_norm: 13.794814 [I 2022-10-10 00:27:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 270/2304 [I 2022-10-10 00:27:46] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:58 [I 2022-10-10 00:27:46] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00309 [I 2022-10-10 00:27:46] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-10 00:27:46] ...optimizers/trijoint.py.151: total_norm: 10.552332 [I 2022-10-10 00:27:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 280/2304 [I 2022-10-10 00:27:55] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:40 [I 2022-10-10 00:27:55] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00299 [I 2022-10-10 00:27:55] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-10 00:27:55] ...optimizers/trijoint.py.151: total_norm: 12.351268 [I 2022-10-10 00:28:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 290/2304 [I 2022-10-10 00:28:03] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:38 [I 2022-10-10 00:28:03] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00320 [I 2022-10-10 00:28:03] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-10 00:28:03] ...optimizers/trijoint.py.151: total_norm: 18.352961 [I 2022-10-10 00:28:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 300/2304 [I 2022-10-10 00:28:12] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:25 [I 2022-10-10 00:28:12] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00324 [I 2022-10-10 00:28:12] ...trap/engines/engine.py.230: loss: 0.22044 [I 2022-10-10 00:28:12] ...optimizers/trijoint.py.151: total_norm: 20.503725 [I 2022-10-10 00:28:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 310/2304 [I 2022-10-10 00:28:20] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:22 [I 2022-10-10 00:28:20] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00312 [I 2022-10-10 00:28:20] ...trap/engines/engine.py.230: loss: 0.16754 [I 2022-10-10 00:28:20] ...optimizers/trijoint.py.151: total_norm: 28.599255 [I 2022-10-10 00:28:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 320/2304 [I 2022-10-10 00:28:29] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:00 [I 2022-10-10 00:28:29] ...trap/engines/engine.py.229: process: 0.83822 | load: 0.00324 [I 2022-10-10 00:28:29] ...trap/engines/engine.py.230: loss: 0.13226 [I 2022-10-10 00:28:29] ...optimizers/trijoint.py.151: total_norm: 13.990830 [I 2022-10-10 00:28:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 330/2304 [I 2022-10-10 00:28:37] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:46 [I 2022-10-10 00:28:37] ...trap/engines/engine.py.229: process: 0.83189 | load: 0.00301 [I 2022-10-10 00:28:37] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-10 00:28:37] ...optimizers/trijoint.py.151: total_norm: 12.230608 [I 2022-10-10 00:28:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 340/2304 [I 2022-10-10 00:28:46] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:54 [I 2022-10-10 00:28:46] ...trap/engines/engine.py.229: process: 0.84433 | load: 0.00327 [I 2022-10-10 00:28:46] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-10 00:28:46] ...optimizers/trijoint.py.151: total_norm: 17.046656 [I 2022-10-10 00:28:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 350/2304 [I 2022-10-10 00:28:54] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:40 [I 2022-10-10 00:28:54] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00319 [I 2022-10-10 00:28:54] ...trap/engines/engine.py.230: loss: 0.13011 [I 2022-10-10 00:28:54] ...optimizers/trijoint.py.151: total_norm: 18.008465 [I 2022-10-10 00:29:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 360/2304 [I 2022-10-10 00:29:03] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:33 [I 2022-10-10 00:29:03] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00306 [I 2022-10-10 00:29:03] ...trap/engines/engine.py.230: loss: 0.10793 [I 2022-10-10 00:29:03] ...optimizers/trijoint.py.151: total_norm: 11.724635 [I 2022-10-10 00:29:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 370/2304 [I 2022-10-10 00:29:11] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:19 [I 2022-10-10 00:29:11] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00314 [I 2022-10-10 00:29:11] ...trap/engines/engine.py.230: loss: 0.19488 [I 2022-10-10 00:29:11] ...optimizers/trijoint.py.151: total_norm: 30.335686 [I 2022-10-10 00:29:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 380/2304 [I 2022-10-10 00:29:20] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:10 [I 2022-10-10 00:29:20] ...trap/engines/engine.py.229: process: 0.83309 | load: 0.00313 [I 2022-10-10 00:29:20] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-10 00:29:20] ...optimizers/trijoint.py.151: total_norm: 11.963810 [I 2022-10-10 00:29:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 390/2304 [I 2022-10-10 00:29:28] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:16 [I 2022-10-10 00:29:28] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00318 [I 2022-10-10 00:29:28] ...trap/engines/engine.py.230: loss: 0.14293 [I 2022-10-10 00:29:28] ...optimizers/trijoint.py.151: total_norm: 13.270144 [I 2022-10-10 00:29:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 400/2304 [I 2022-10-10 00:29:37] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:56 [I 2022-10-10 00:29:37] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00314 [I 2022-10-10 00:29:37] ...trap/engines/engine.py.230: loss: 0.17762 [I 2022-10-10 00:29:37] ...optimizers/trijoint.py.151: total_norm: 9.958359 [I 2022-10-10 00:29:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 410/2304 [I 2022-10-10 00:29:45] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:48 [I 2022-10-10 00:29:45] ...trap/engines/engine.py.229: process: 0.84081 | load: 0.00301 [I 2022-10-10 00:29:45] ...trap/engines/engine.py.230: loss: 0.18503 [I 2022-10-10 00:29:45] ...optimizers/trijoint.py.151: total_norm: 11.429013 [I 2022-10-10 00:29:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 420/2304 [I 2022-10-10 00:29:54] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:41 [I 2022-10-10 00:29:54] ...trap/engines/engine.py.229: process: 0.86589 | load: 0.00316 [I 2022-10-10 00:29:54] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-10 00:29:54] ...optimizers/trijoint.py.151: total_norm: 16.802536 [I 2022-10-10 00:30:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 430/2304 [I 2022-10-10 00:30:02] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:34 [I 2022-10-10 00:30:02] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00297 [I 2022-10-10 00:30:02] ...trap/engines/engine.py.230: loss: 0.14312 [I 2022-10-10 00:30:02] ...optimizers/trijoint.py.151: total_norm: 13.019489 [I 2022-10-10 00:30:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 440/2304 [I 2022-10-10 00:30:11] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:21 [I 2022-10-10 00:30:11] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00310 [I 2022-10-10 00:30:11] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-10 00:30:11] ...optimizers/trijoint.py.151: total_norm: 12.119395 [I 2022-10-10 00:30:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 450/2304 [I 2022-10-10 00:30:19] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:13 [I 2022-10-10 00:30:19] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00316 [I 2022-10-10 00:30:19] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-10 00:30:19] ...optimizers/trijoint.py.151: total_norm: 13.171749 [I 2022-10-10 00:30:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 460/2304 [I 2022-10-10 00:30:28] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:02 [I 2022-10-10 00:30:28] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00303 [I 2022-10-10 00:30:28] ...trap/engines/engine.py.230: loss: 0.18605 [I 2022-10-10 00:30:28] ...optimizers/trijoint.py.151: total_norm: 22.661308 [I 2022-10-10 00:30:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 470/2304 [I 2022-10-10 00:30:36] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:56 [I 2022-10-10 00:30:36] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00315 [I 2022-10-10 00:30:36] ...trap/engines/engine.py.230: loss: 0.13077 [I 2022-10-10 00:30:36] ...optimizers/trijoint.py.151: total_norm: 34.553616 [I 2022-10-10 00:30:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 480/2304 [I 2022-10-10 00:30:45] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:51 [I 2022-10-10 00:30:45] ...trap/engines/engine.py.229: process: 0.85744 | load: 0.00320 [I 2022-10-10 00:30:45] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-10 00:30:45] ...optimizers/trijoint.py.151: total_norm: 11.627829 [I 2022-10-10 00:30:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 490/2304 [I 2022-10-10 00:30:53] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:49 [I 2022-10-10 00:30:53] ...trap/engines/engine.py.229: process: 0.86652 | load: 0.00305 [I 2022-10-10 00:30:53] ...trap/engines/engine.py.230: loss: 0.17870 [I 2022-10-10 00:30:53] ...optimizers/trijoint.py.151: total_norm: 15.499310 [I 2022-10-10 00:31:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 500/2304 [I 2022-10-10 00:31:02] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:30 [I 2022-10-10 00:31:02] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00302 [I 2022-10-10 00:31:02] ...trap/engines/engine.py.230: loss: 0.19128 [I 2022-10-10 00:31:02] ...optimizers/trijoint.py.151: total_norm: 22.492571 [I 2022-10-10 00:31:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 510/2304 [I 2022-10-10 00:31:10] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:27 [I 2022-10-10 00:31:10] ...trap/engines/engine.py.229: process: 0.84995 | load: 0.00300 [I 2022-10-10 00:31:10] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-10 00:31:10] ...optimizers/trijoint.py.151: total_norm: 14.785828 [I 2022-10-10 00:31:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 520/2304 [I 2022-10-10 00:31:19] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:12 [I 2022-10-10 00:31:19] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00309 [I 2022-10-10 00:31:19] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-10 00:31:19] ...optimizers/trijoint.py.151: total_norm: 12.939360 [I 2022-10-10 00:31:27] ...trap/engines/engine.py.224: train: epoch 60 | batch 530/2304 [I 2022-10-10 00:31:27] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:03 [I 2022-10-10 00:31:27] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00309 [I 2022-10-10 00:31:27] ...trap/engines/engine.py.230: loss: 0.15949 [I 2022-10-10 00:31:27] ...optimizers/trijoint.py.151: total_norm: 11.539083 [I 2022-10-10 00:31:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 540/2304 [I 2022-10-10 00:31:36] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:00 [I 2022-10-10 00:31:36] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00311 [I 2022-10-10 00:31:36] ...trap/engines/engine.py.230: loss: 0.11067 [I 2022-10-10 00:31:36] ...optimizers/trijoint.py.151: total_norm: 15.465861 [I 2022-10-10 00:31:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 550/2304 [I 2022-10-10 00:31:45] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:24:56 [I 2022-10-10 00:31:45] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00313 [I 2022-10-10 00:31:45] ...trap/engines/engine.py.230: loss: 0.13114 [I 2022-10-10 00:31:45] ...optimizers/trijoint.py.151: total_norm: 14.767456 [I 2022-10-10 00:31:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 560/2304 [I 2022-10-10 00:31:53] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:45 [I 2022-10-10 00:31:53] ...trap/engines/engine.py.229: process: 0.85226 | load: 0.00318 [I 2022-10-10 00:31:53] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-10 00:31:53] ...optimizers/trijoint.py.151: total_norm: 19.765724 [I 2022-10-10 00:32:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 570/2304 [I 2022-10-10 00:32:02] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:46 [I 2022-10-10 00:32:02] ...trap/engines/engine.py.229: process: 0.85666 | load: 0.00310 [I 2022-10-10 00:32:02] ...trap/engines/engine.py.230: loss: 0.11946 [I 2022-10-10 00:32:02] ...optimizers/trijoint.py.151: total_norm: 45.276062 [I 2022-10-10 00:32:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 580/2304 [I 2022-10-10 00:32:10] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:33 [I 2022-10-10 00:32:10] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00314 [I 2022-10-10 00:32:10] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-10 00:32:10] ...optimizers/trijoint.py.151: total_norm: 17.315233 [I 2022-10-10 00:32:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 590/2304 [I 2022-10-10 00:32:19] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:27 [I 2022-10-10 00:32:19] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00317 [I 2022-10-10 00:32:19] ...trap/engines/engine.py.230: loss: 0.13238 [I 2022-10-10 00:32:19] ...optimizers/trijoint.py.151: total_norm: 19.368475 [I 2022-10-10 00:32:27] ...trap/engines/engine.py.224: train: epoch 60 | batch 600/2304 [I 2022-10-10 00:32:27] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:19 [I 2022-10-10 00:32:27] ...trap/engines/engine.py.229: process: 0.85966 | load: 0.00312 [I 2022-10-10 00:32:27] ...trap/engines/engine.py.230: loss: 0.18153 [I 2022-10-10 00:32:27] ...optimizers/trijoint.py.151: total_norm: 29.975950 [I 2022-10-10 00:32:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 610/2304 [I 2022-10-10 00:32:36] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:20 [I 2022-10-10 00:32:36] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00313 [I 2022-10-10 00:32:36] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-10 00:32:36] ...optimizers/trijoint.py.151: total_norm: 22.132387 [I 2022-10-10 00:32:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 620/2304 [I 2022-10-10 00:32:45] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:07 [I 2022-10-10 00:32:45] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00313 [I 2022-10-10 00:32:45] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-10 00:32:45] ...optimizers/trijoint.py.151: total_norm: 10.319145 [I 2022-10-10 00:32:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 630/2304 [I 2022-10-10 00:32:53] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:49 [I 2022-10-10 00:32:53] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00315 [I 2022-10-10 00:32:53] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-10 00:32:53] ...optimizers/trijoint.py.151: total_norm: 15.087130 [I 2022-10-10 00:33:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 640/2304 [I 2022-10-10 00:33:02] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:44 [I 2022-10-10 00:33:02] ...trap/engines/engine.py.229: process: 0.86041 | load: 0.00310 [I 2022-10-10 00:33:02] ...trap/engines/engine.py.230: loss: 0.20096 [I 2022-10-10 00:33:02] ...optimizers/trijoint.py.151: total_norm: 21.322229 [I 2022-10-10 00:33:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 650/2304 [I 2022-10-10 00:33:10] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:28 [I 2022-10-10 00:33:10] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00304 [I 2022-10-10 00:33:10] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-10 00:33:10] ...optimizers/trijoint.py.151: total_norm: 9.293641 [I 2022-10-10 00:33:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 660/2304 [I 2022-10-10 00:33:19] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:14 [I 2022-10-10 00:33:19] ...trap/engines/engine.py.229: process: 0.84370 | load: 0.00296 [I 2022-10-10 00:33:19] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-10 00:33:19] ...optimizers/trijoint.py.151: total_norm: 11.238080 [I 2022-10-10 00:33:27] ...trap/engines/engine.py.224: train: epoch 60 | batch 670/2304 [I 2022-10-10 00:33:27] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:22:57 [I 2022-10-10 00:33:27] ...trap/engines/engine.py.229: process: 0.83705 | load: 0.00297 [I 2022-10-10 00:33:27] ...trap/engines/engine.py.230: loss: 0.13775 [I 2022-10-10 00:33:27] ...optimizers/trijoint.py.151: total_norm: 19.759880 [I 2022-10-10 00:33:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 680/2304 [I 2022-10-10 00:33:36] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:22:51 [I 2022-10-10 00:33:36] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00296 [I 2022-10-10 00:33:36] ...trap/engines/engine.py.230: loss: 0.17683 [I 2022-10-10 00:33:36] ...optimizers/trijoint.py.151: total_norm: 31.544649 [I 2022-10-10 00:33:44] ...trap/engines/engine.py.224: train: epoch 60 | batch 690/2304 [I 2022-10-10 00:33:44] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:42 [I 2022-10-10 00:33:44] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00298 [I 2022-10-10 00:33:44] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-10 00:33:44] ...optimizers/trijoint.py.151: total_norm: 14.762739 [I 2022-10-10 00:33:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 700/2304 [I 2022-10-10 00:33:53] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:36 [I 2022-10-10 00:33:53] ...trap/engines/engine.py.229: process: 0.85138 | load: 0.00297 [I 2022-10-10 00:33:53] ...trap/engines/engine.py.230: loss: 0.10630 [I 2022-10-10 00:33:53] ...optimizers/trijoint.py.151: total_norm: 9.240371 [I 2022-10-10 00:34:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 710/2304 [I 2022-10-10 00:34:01] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:25 [I 2022-10-10 00:34:01] ...trap/engines/engine.py.229: process: 0.83608 | load: 0.00297 [I 2022-10-10 00:34:01] ...trap/engines/engine.py.230: loss: 0.16184 [I 2022-10-10 00:34:01] ...optimizers/trijoint.py.151: total_norm: 21.133104 [I 2022-10-10 00:34:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 720/2304 [I 2022-10-10 00:34:10] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:17 [I 2022-10-10 00:34:10] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00297 [I 2022-10-10 00:34:10] ...trap/engines/engine.py.230: loss: 0.20550 [I 2022-10-10 00:34:10] ...optimizers/trijoint.py.151: total_norm: 12.954335 [I 2022-10-10 00:34:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 730/2304 [I 2022-10-10 00:34:18] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:12 [I 2022-10-10 00:34:18] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00298 [I 2022-10-10 00:34:18] ...trap/engines/engine.py.230: loss: 0.13938 [I 2022-10-10 00:34:18] ...optimizers/trijoint.py.151: total_norm: 31.515110 [I 2022-10-10 00:34:27] ...trap/engines/engine.py.224: train: epoch 60 | batch 740/2304 [I 2022-10-10 00:34:27] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:21:58 [I 2022-10-10 00:34:27] ...trap/engines/engine.py.229: process: 0.83422 | load: 0.00299 [I 2022-10-10 00:34:27] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-10 00:34:27] ...optimizers/trijoint.py.151: total_norm: 12.148832 [I 2022-10-10 00:34:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 750/2304 [I 2022-10-10 00:34:35] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:55 [I 2022-10-10 00:34:35] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00304 [I 2022-10-10 00:34:35] ...trap/engines/engine.py.230: loss: 0.14429 [I 2022-10-10 00:34:35] ...optimizers/trijoint.py.151: total_norm: 9.761444 [I 2022-10-10 00:34:44] ...trap/engines/engine.py.224: train: epoch 60 | batch 760/2304 [I 2022-10-10 00:34:44] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:50 [I 2022-10-10 00:34:44] ...trap/engines/engine.py.229: process: 0.86094 | load: 0.00304 [I 2022-10-10 00:34:44] ...trap/engines/engine.py.230: loss: 0.26981 [I 2022-10-10 00:34:44] ...optimizers/trijoint.py.151: total_norm: 30.763491 [I 2022-10-10 00:34:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 770/2304 [I 2022-10-10 00:34:52] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:41 [I 2022-10-10 00:34:52] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00296 [I 2022-10-10 00:34:52] ...trap/engines/engine.py.230: loss: 0.09523 [I 2022-10-10 00:34:52] ...optimizers/trijoint.py.151: total_norm: 9.827775 [I 2022-10-10 00:35:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 780/2304 [I 2022-10-10 00:35:01] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:33 [I 2022-10-10 00:35:01] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00298 [I 2022-10-10 00:35:01] ...trap/engines/engine.py.230: loss: 0.17495 [I 2022-10-10 00:35:01] ...optimizers/trijoint.py.151: total_norm: 16.947186 [I 2022-10-10 00:35:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 790/2304 [I 2022-10-10 00:35:09] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:28 [I 2022-10-10 00:35:09] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00300 [I 2022-10-10 00:35:09] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-10 00:35:09] ...optimizers/trijoint.py.151: total_norm: 12.504806 [I 2022-10-10 00:35:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 800/2304 [I 2022-10-10 00:35:18] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:16 [I 2022-10-10 00:35:18] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00300 [I 2022-10-10 00:35:18] ...trap/engines/engine.py.230: loss: 0.15053 [I 2022-10-10 00:35:18] ...optimizers/trijoint.py.151: total_norm: 15.922396 [I 2022-10-10 00:35:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 810/2304 [I 2022-10-10 00:35:26] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:07 [I 2022-10-10 00:35:26] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00301 [I 2022-10-10 00:35:26] ...trap/engines/engine.py.230: loss: 0.14345 [I 2022-10-10 00:35:26] ...optimizers/trijoint.py.151: total_norm: 17.179558 [I 2022-10-10 00:35:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 820/2304 [I 2022-10-10 00:35:35] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:52 [I 2022-10-10 00:35:35] ...trap/engines/engine.py.229: process: 0.84774 | load: 0.00302 [I 2022-10-10 00:35:35] ...trap/engines/engine.py.230: loss: 0.18295 [I 2022-10-10 00:35:35] ...optimizers/trijoint.py.151: total_norm: 17.163294 [I 2022-10-10 00:35:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 830/2304 [I 2022-10-10 00:35:43] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:44 [I 2022-10-10 00:35:43] ...trap/engines/engine.py.229: process: 0.83207 | load: 0.00312 [I 2022-10-10 00:35:43] ...trap/engines/engine.py.230: loss: 0.15220 [I 2022-10-10 00:35:43] ...optimizers/trijoint.py.151: total_norm: 9.888163 [I 2022-10-10 00:35:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 840/2304 [I 2022-10-10 00:35:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:45 [I 2022-10-10 00:35:52] ...trap/engines/engine.py.229: process: 0.82668 | load: 0.00313 [I 2022-10-10 00:35:52] ...trap/engines/engine.py.230: loss: 0.16383 [I 2022-10-10 00:35:52] ...optimizers/trijoint.py.151: total_norm: 15.202727 [I 2022-10-10 00:36:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 850/2304 [I 2022-10-10 00:36:00] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:46 [I 2022-10-10 00:36:00] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00312 [I 2022-10-10 00:36:00] ...trap/engines/engine.py.230: loss: 0.15739 [I 2022-10-10 00:36:00] ...optimizers/trijoint.py.151: total_norm: 15.620693 [I 2022-10-10 00:36:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 860/2304 [I 2022-10-10 00:36:09] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:31 [I 2022-10-10 00:36:09] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00311 [I 2022-10-10 00:36:09] ...trap/engines/engine.py.230: loss: 0.16449 [I 2022-10-10 00:36:09] ...optimizers/trijoint.py.151: total_norm: 21.488201 [I 2022-10-10 00:36:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 870/2304 [I 2022-10-10 00:36:17] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:21 [I 2022-10-10 00:36:17] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00320 [I 2022-10-10 00:36:17] ...trap/engines/engine.py.230: loss: 0.14740 [I 2022-10-10 00:36:17] ...optimizers/trijoint.py.151: total_norm: 14.460212 [I 2022-10-10 00:36:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 880/2304 [I 2022-10-10 00:36:26] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:14 [I 2022-10-10 00:36:26] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00309 [I 2022-10-10 00:36:26] ...trap/engines/engine.py.230: loss: 0.18662 [I 2022-10-10 00:36:26] ...optimizers/trijoint.py.151: total_norm: 18.682734 [I 2022-10-10 00:36:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 890/2304 [I 2022-10-10 00:36:34] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:07 [I 2022-10-10 00:36:34] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00322 [I 2022-10-10 00:36:34] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-10 00:36:34] ...optimizers/trijoint.py.151: total_norm: 11.892076 [I 2022-10-10 00:36:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 900/2304 [I 2022-10-10 00:36:43] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:54 [I 2022-10-10 00:36:43] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00314 [I 2022-10-10 00:36:43] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-10 00:36:43] ...optimizers/trijoint.py.151: total_norm: 10.306247 [I 2022-10-10 00:36:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 910/2304 [I 2022-10-10 00:36:52] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:50 [I 2022-10-10 00:36:52] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00315 [I 2022-10-10 00:36:52] ...trap/engines/engine.py.230: loss: 0.13123 [I 2022-10-10 00:36:52] ...optimizers/trijoint.py.151: total_norm: 24.497898 [I 2022-10-10 00:37:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 920/2304 [I 2022-10-10 00:37:00] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:43 [I 2022-10-10 00:37:00] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00315 [I 2022-10-10 00:37:00] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-10 00:37:00] ...optimizers/trijoint.py.151: total_norm: 9.917286 [I 2022-10-10 00:37:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 930/2304 [I 2022-10-10 00:37:09] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:26 [I 2022-10-10 00:37:09] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00311 [I 2022-10-10 00:37:09] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-10 00:37:09] ...optimizers/trijoint.py.151: total_norm: 8.775983 [I 2022-10-10 00:37:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 940/2304 [I 2022-10-10 00:37:17] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:29 [I 2022-10-10 00:37:17] ...trap/engines/engine.py.229: process: 0.86937 | load: 0.00313 [I 2022-10-10 00:37:17] ...trap/engines/engine.py.230: loss: 0.22889 [I 2022-10-10 00:37:17] ...optimizers/trijoint.py.151: total_norm: 10.112261 [I 2022-10-10 00:37:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 950/2304 [I 2022-10-10 00:37:26] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:18 [I 2022-10-10 00:37:26] ...trap/engines/engine.py.229: process: 0.86446 | load: 0.00322 [I 2022-10-10 00:37:26] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-10 00:37:26] ...optimizers/trijoint.py.151: total_norm: 16.727209 [I 2022-10-10 00:37:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 960/2304 [I 2022-10-10 00:37:34] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:04 [I 2022-10-10 00:37:34] ...trap/engines/engine.py.229: process: 0.83995 | load: 0.00314 [I 2022-10-10 00:37:34] ...trap/engines/engine.py.230: loss: 0.18915 [I 2022-10-10 00:37:34] ...optimizers/trijoint.py.151: total_norm: 9.717933 [I 2022-10-10 00:37:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 970/2304 [I 2022-10-10 00:37:43] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:00 [I 2022-10-10 00:37:43] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00316 [I 2022-10-10 00:37:43] ...trap/engines/engine.py.230: loss: 0.21779 [I 2022-10-10 00:37:43] ...optimizers/trijoint.py.151: total_norm: 19.444881 [I 2022-10-10 00:37:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 980/2304 [I 2022-10-10 00:37:51] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:51 [I 2022-10-10 00:37:51] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00318 [I 2022-10-10 00:37:51] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-10 00:37:51] ...optimizers/trijoint.py.151: total_norm: 20.206762 [I 2022-10-10 00:38:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 990/2304 [I 2022-10-10 00:38:00] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:37 [I 2022-10-10 00:38:00] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00311 [I 2022-10-10 00:38:00] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-10 00:38:00] ...optimizers/trijoint.py.151: total_norm: 6.535022 [I 2022-10-10 00:38:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1000/2304 [I 2022-10-10 00:38:08] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:35 [I 2022-10-10 00:38:08] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00310 [I 2022-10-10 00:38:08] ...trap/engines/engine.py.230: loss: 0.11942 [I 2022-10-10 00:38:08] ...optimizers/trijoint.py.151: total_norm: 10.648365 [I 2022-10-10 00:38:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 1010/2304 [I 2022-10-10 00:38:17] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:26 [I 2022-10-10 00:38:17] ...trap/engines/engine.py.229: process: 0.86965 | load: 0.00312 [I 2022-10-10 00:38:17] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-10 00:38:17] ...optimizers/trijoint.py.151: total_norm: 25.839140 [I 2022-10-10 00:38:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 1020/2304 [I 2022-10-10 00:38:26] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:21 [I 2022-10-10 00:38:26] ...trap/engines/engine.py.229: process: 0.87366 | load: 0.00315 [I 2022-10-10 00:38:26] ...trap/engines/engine.py.230: loss: 0.13136 [I 2022-10-10 00:38:26] ...optimizers/trijoint.py.151: total_norm: 32.973385 [I 2022-10-10 00:38:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 1030/2304 [I 2022-10-10 00:38:34] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:11 [I 2022-10-10 00:38:34] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00313 [I 2022-10-10 00:38:34] ...trap/engines/engine.py.230: loss: 0.16391 [I 2022-10-10 00:38:34] ...optimizers/trijoint.py.151: total_norm: 12.004058 [I 2022-10-10 00:38:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 1040/2304 [I 2022-10-10 00:38:43] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:17:59 [I 2022-10-10 00:38:43] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00305 [I 2022-10-10 00:38:43] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-10 00:38:43] ...optimizers/trijoint.py.151: total_norm: 23.369410 [I 2022-10-10 00:38:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1050/2304 [I 2022-10-10 00:38:51] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:50 [I 2022-10-10 00:38:51] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00313 [I 2022-10-10 00:38:51] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-10 00:38:51] ...optimizers/trijoint.py.151: total_norm: 14.241933 [I 2022-10-10 00:39:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 1060/2304 [I 2022-10-10 00:39:00] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:40 [I 2022-10-10 00:39:00] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00319 [I 2022-10-10 00:39:00] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-10 00:39:00] ...optimizers/trijoint.py.151: total_norm: 21.226633 [I 2022-10-10 00:39:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1070/2304 [I 2022-10-10 00:39:08] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:30 [I 2022-10-10 00:39:08] ...trap/engines/engine.py.229: process: 0.84983 | load: 0.00316 [I 2022-10-10 00:39:08] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-10 00:39:08] ...optimizers/trijoint.py.151: total_norm: 11.563348 [I 2022-10-10 00:39:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 1080/2304 [I 2022-10-10 00:39:17] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:27 [I 2022-10-10 00:39:17] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00313 [I 2022-10-10 00:39:17] ...trap/engines/engine.py.230: loss: 0.17956 [I 2022-10-10 00:39:17] ...optimizers/trijoint.py.151: total_norm: 40.552345 [I 2022-10-10 00:39:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 1090/2304 [I 2022-10-10 00:39:26] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:21 [I 2022-10-10 00:39:26] ...trap/engines/engine.py.229: process: 0.86984 | load: 0.00312 [I 2022-10-10 00:39:26] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-10 00:39:26] ...optimizers/trijoint.py.151: total_norm: 13.234460 [I 2022-10-10 00:39:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 1100/2304 [I 2022-10-10 00:39:34] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:08 [I 2022-10-10 00:39:34] ...trap/engines/engine.py.229: process: 0.84731 | load: 0.00312 [I 2022-10-10 00:39:34] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-10 00:39:34] ...optimizers/trijoint.py.151: total_norm: 10.306456 [I 2022-10-10 00:39:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 1110/2304 [I 2022-10-10 00:39:43] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:58 [I 2022-10-10 00:39:43] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00311 [I 2022-10-10 00:39:43] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-10 00:39:43] ...optimizers/trijoint.py.151: total_norm: 11.233922 [I 2022-10-10 00:39:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1120/2304 [I 2022-10-10 00:39:51] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:48 [I 2022-10-10 00:39:51] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00319 [I 2022-10-10 00:39:51] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-10 00:39:51] ...optimizers/trijoint.py.151: total_norm: 14.344167 [I 2022-10-10 00:40:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 1130/2304 [I 2022-10-10 00:40:00] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:41 [I 2022-10-10 00:40:00] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00313 [I 2022-10-10 00:40:00] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-10 00:40:00] ...optimizers/trijoint.py.151: total_norm: 23.645885 [I 2022-10-10 00:40:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1140/2304 [I 2022-10-10 00:40:08] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:28 [I 2022-10-10 00:40:08] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00294 [I 2022-10-10 00:40:08] ...trap/engines/engine.py.230: loss: 0.14552 [I 2022-10-10 00:40:08] ...optimizers/trijoint.py.151: total_norm: 17.604513 [I 2022-10-10 00:40:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 1150/2304 [I 2022-10-10 00:40:17] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:14 [I 2022-10-10 00:40:17] ...trap/engines/engine.py.229: process: 0.83814 | load: 0.00293 [I 2022-10-10 00:40:17] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-10 00:40:17] ...optimizers/trijoint.py.151: total_norm: 21.732088 [I 2022-10-10 00:40:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 1160/2304 [I 2022-10-10 00:40:25] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:05 [I 2022-10-10 00:40:25] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00303 [I 2022-10-10 00:40:25] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-10 00:40:25] ...optimizers/trijoint.py.151: total_norm: 31.086950 [I 2022-10-10 00:40:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 1170/2304 [I 2022-10-10 00:40:34] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:58 [I 2022-10-10 00:40:34] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00305 [I 2022-10-10 00:40:34] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-10 00:40:34] ...optimizers/trijoint.py.151: total_norm: 11.174067 [I 2022-10-10 00:40:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 1180/2304 [I 2022-10-10 00:40:42] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:51 [I 2022-10-10 00:40:42] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00298 [I 2022-10-10 00:40:42] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-10 00:40:42] ...optimizers/trijoint.py.151: total_norm: 19.920797 [I 2022-10-10 00:40:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1190/2304 [I 2022-10-10 00:40:51] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:48 [I 2022-10-10 00:40:51] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00300 [I 2022-10-10 00:40:51] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-10 00:40:51] ...optimizers/trijoint.py.151: total_norm: 22.532785 [I 2022-10-10 00:40:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1200/2304 [I 2022-10-10 00:40:59] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:32 [I 2022-10-10 00:40:59] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00297 [I 2022-10-10 00:40:59] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-10 00:40:59] ...optimizers/trijoint.py.151: total_norm: 20.072294 [I 2022-10-10 00:41:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1210/2304 [I 2022-10-10 00:41:08] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:26 [I 2022-10-10 00:41:08] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00302 [I 2022-10-10 00:41:08] ...trap/engines/engine.py.230: loss: 0.19293 [I 2022-10-10 00:41:08] ...optimizers/trijoint.py.151: total_norm: 38.353966 [I 2022-10-10 00:41:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1220/2304 [I 2022-10-10 00:41:16] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:13 [I 2022-10-10 00:41:16] ...trap/engines/engine.py.229: process: 0.83841 | load: 0.00301 [I 2022-10-10 00:41:16] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-10 00:41:16] ...optimizers/trijoint.py.151: total_norm: 23.543985 [I 2022-10-10 00:41:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 1230/2304 [I 2022-10-10 00:41:42] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:23:43 [I 2022-10-10 00:41:42] ...trap/engines/engine.py.229: process: 0.84949 | load: 0.00302 [I 2022-10-10 00:41:43] ...trap/engines/engine.py.230: loss: 0.18050 [I 2022-10-10 00:41:43] ...optimizers/trijoint.py.151: total_norm: 15.072508 [I 2022-10-10 00:41:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1240/2304 [I 2022-10-10 00:41:51] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:15:49 [I 2022-10-10 00:41:51] ...trap/engines/engine.py.229: process: 0.83576 | load: 0.00300 [I 2022-10-10 00:41:51] ...trap/engines/engine.py.230: loss: 0.17074 [I 2022-10-10 00:41:51] ...optimizers/trijoint.py.151: total_norm: 20.778896 [I 2022-10-10 00:41:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1250/2304 [I 2022-10-10 00:41:59] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:53 [I 2022-10-10 00:41:59] ...trap/engines/engine.py.229: process: 0.84211 | load: 0.00302 [I 2022-10-10 00:41:59] ...trap/engines/engine.py.230: loss: 0.19044 [I 2022-10-10 00:41:59] ...optimizers/trijoint.py.151: total_norm: 18.325741 [I 2022-10-10 00:42:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1260/2304 [I 2022-10-10 00:42:08] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:48 [I 2022-10-10 00:42:08] ...trap/engines/engine.py.229: process: 0.86603 | load: 0.00322 [I 2022-10-10 00:42:08] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-10 00:42:08] ...optimizers/trijoint.py.151: total_norm: 9.844411 [I 2022-10-10 00:42:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1270/2304 [I 2022-10-10 00:42:16] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:40 [I 2022-10-10 00:42:16] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00325 [I 2022-10-10 00:42:16] ...trap/engines/engine.py.230: loss: 0.13995 [I 2022-10-10 00:42:16] ...optimizers/trijoint.py.151: total_norm: 16.303770 [I 2022-10-10 00:42:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 1280/2304 [I 2022-10-10 00:42:25] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:27 [I 2022-10-10 00:42:25] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00303 [I 2022-10-10 00:42:25] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-10 00:42:25] ...optimizers/trijoint.py.151: total_norm: 13.437778 [I 2022-10-10 00:42:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1290/2304 [I 2022-10-10 00:42:33] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:19 [I 2022-10-10 00:42:33] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00301 [I 2022-10-10 00:42:33] ...trap/engines/engine.py.230: loss: 0.15697 [I 2022-10-10 00:42:33] ...optimizers/trijoint.py.151: total_norm: 12.432680 [I 2022-10-10 00:42:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 1300/2304 [I 2022-10-10 00:42:42] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:12 [I 2022-10-10 00:42:42] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00321 [I 2022-10-10 00:42:42] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-10 00:42:42] ...optimizers/trijoint.py.151: total_norm: 25.429333 [I 2022-10-10 00:42:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1310/2304 [I 2022-10-10 00:42:50] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:14:03 [I 2022-10-10 00:42:50] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00314 [I 2022-10-10 00:42:50] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-10 00:42:50] ...optimizers/trijoint.py.151: total_norm: 16.859041 [I 2022-10-10 00:42:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1320/2304 [I 2022-10-10 00:42:59] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:56 [I 2022-10-10 00:42:59] ...trap/engines/engine.py.229: process: 0.85214 | load: 0.00309 [I 2022-10-10 00:42:59] ...trap/engines/engine.py.230: loss: 0.12463 [I 2022-10-10 00:42:59] ...optimizers/trijoint.py.151: total_norm: 14.721309 [I 2022-10-10 00:43:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1330/2304 [I 2022-10-10 00:43:08] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:53 [I 2022-10-10 00:43:08] ...trap/engines/engine.py.229: process: 0.86173 | load: 0.00310 [I 2022-10-10 00:43:08] ...trap/engines/engine.py.230: loss: 0.23579 [I 2022-10-10 00:43:08] ...optimizers/trijoint.py.151: total_norm: 26.384811 [I 2022-10-10 00:43:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1340/2304 [I 2022-10-10 00:43:16] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:39 [I 2022-10-10 00:43:16] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00316 [I 2022-10-10 00:43:16] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-10 00:43:16] ...optimizers/trijoint.py.151: total_norm: 9.871459 [I 2022-10-10 00:43:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 1350/2304 [I 2022-10-10 00:43:25] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:34 [I 2022-10-10 00:43:25] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00314 [I 2022-10-10 00:43:25] ...trap/engines/engine.py.230: loss: 0.14222 [I 2022-10-10 00:43:25] ...optimizers/trijoint.py.151: total_norm: 26.481501 [I 2022-10-10 00:43:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1360/2304 [I 2022-10-10 00:43:33] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:23 [I 2022-10-10 00:43:33] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00307 [I 2022-10-10 00:43:33] ...trap/engines/engine.py.230: loss: 0.14422 [I 2022-10-10 00:43:33] ...optimizers/trijoint.py.151: total_norm: 13.011748 [I 2022-10-10 00:43:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 1370/2304 [I 2022-10-10 00:43:42] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:17 [I 2022-10-10 00:43:42] ...trap/engines/engine.py.229: process: 0.85259 | load: 0.00312 [I 2022-10-10 00:43:42] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-10 00:43:42] ...optimizers/trijoint.py.151: total_norm: 19.166700 [I 2022-10-10 00:43:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1380/2304 [I 2022-10-10 00:43:50] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:13:10 [I 2022-10-10 00:43:50] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00313 [I 2022-10-10 00:43:50] ...trap/engines/engine.py.230: loss: 0.16425 [I 2022-10-10 00:43:50] ...optimizers/trijoint.py.151: total_norm: 35.223885 [I 2022-10-10 00:43:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1390/2304 [I 2022-10-10 00:43:59] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:13:00 [I 2022-10-10 00:43:59] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00312 [I 2022-10-10 00:43:59] ...trap/engines/engine.py.230: loss: 0.17352 [I 2022-10-10 00:43:59] ...optimizers/trijoint.py.151: total_norm: 19.262442 [I 2022-10-10 00:44:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1400/2304 [I 2022-10-10 00:44:07] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:53 [I 2022-10-10 00:44:07] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00312 [I 2022-10-10 00:44:07] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-10 00:44:07] ...optimizers/trijoint.py.151: total_norm: 21.776190 [I 2022-10-10 00:44:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1410/2304 [I 2022-10-10 00:44:16] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:43 [I 2022-10-10 00:44:16] ...trap/engines/engine.py.229: process: 0.86050 | load: 0.00318 [I 2022-10-10 00:44:16] ...trap/engines/engine.py.230: loss: 0.19092 [I 2022-10-10 00:44:16] ...optimizers/trijoint.py.151: total_norm: 20.479437 [I 2022-10-10 00:44:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 1420/2304 [I 2022-10-10 00:44:24] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:12:35 [I 2022-10-10 00:44:24] ...trap/engines/engine.py.229: process: 0.86152 | load: 0.00321 [I 2022-10-10 00:44:24] ...trap/engines/engine.py.230: loss: 0.19819 [I 2022-10-10 00:44:24] ...optimizers/trijoint.py.151: total_norm: 28.030478 [I 2022-10-10 00:44:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1430/2304 [I 2022-10-10 00:44:33] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:21 [I 2022-10-10 00:44:33] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00315 [I 2022-10-10 00:44:33] ...trap/engines/engine.py.230: loss: 0.25205 [I 2022-10-10 00:44:33] ...optimizers/trijoint.py.151: total_norm: 13.807778 [I 2022-10-10 00:44:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 1440/2304 [I 2022-10-10 00:44:41] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:18 [I 2022-10-10 00:44:41] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00317 [I 2022-10-10 00:44:41] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-10 00:44:41] ...optimizers/trijoint.py.151: total_norm: 12.567239 [I 2022-10-10 00:44:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1450/2304 [I 2022-10-10 00:44:50] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:07 [I 2022-10-10 00:44:50] ...trap/engines/engine.py.229: process: 0.85188 | load: 0.00307 [I 2022-10-10 00:44:50] ...trap/engines/engine.py.230: loss: 0.13652 [I 2022-10-10 00:44:50] ...optimizers/trijoint.py.151: total_norm: 10.705147 [I 2022-10-10 00:44:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1460/2304 [I 2022-10-10 00:44:59] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:58 [I 2022-10-10 00:44:59] ...trap/engines/engine.py.229: process: 0.84988 | load: 0.00312 [I 2022-10-10 00:44:59] ...trap/engines/engine.py.230: loss: 0.10368 [I 2022-10-10 00:44:59] ...optimizers/trijoint.py.151: total_norm: 8.432019 [I 2022-10-10 00:45:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1470/2304 [I 2022-10-10 00:45:07] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:50 [I 2022-10-10 00:45:07] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00313 [I 2022-10-10 00:45:07] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-10 00:45:07] ...optimizers/trijoint.py.151: total_norm: 28.978920 [I 2022-10-10 00:45:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1480/2304 [I 2022-10-10 00:45:16] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:48 [I 2022-10-10 00:45:16] ...trap/engines/engine.py.229: process: 0.86508 | load: 0.00313 [I 2022-10-10 00:45:16] ...trap/engines/engine.py.230: loss: 0.13046 [I 2022-10-10 00:45:16] ...optimizers/trijoint.py.151: total_norm: 21.762526 [I 2022-10-10 00:45:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 1490/2304 [I 2022-10-10 00:45:24] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:36 [I 2022-10-10 00:45:24] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00311 [I 2022-10-10 00:45:24] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-10 00:45:24] ...optimizers/trijoint.py.151: total_norm: 19.946903 [I 2022-10-10 00:45:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1500/2304 [I 2022-10-10 00:45:33] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:26 [I 2022-10-10 00:45:33] ...trap/engines/engine.py.229: process: 0.84266 | load: 0.00314 [I 2022-10-10 00:45:33] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-10 00:45:33] ...optimizers/trijoint.py.151: total_norm: 7.359463 [I 2022-10-10 00:45:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 1510/2304 [I 2022-10-10 00:45:41] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:11:16 [I 2022-10-10 00:45:41] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00319 [I 2022-10-10 00:45:41] ...trap/engines/engine.py.230: loss: 0.13691 [I 2022-10-10 00:45:41] ...optimizers/trijoint.py.151: total_norm: 21.034422 [I 2022-10-10 00:45:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1520/2304 [I 2022-10-10 00:45:50] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:10 [I 2022-10-10 00:45:50] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00312 [I 2022-10-10 00:45:50] ...trap/engines/engine.py.230: loss: 0.11558 [I 2022-10-10 00:45:50] ...optimizers/trijoint.py.151: total_norm: 8.055656 [I 2022-10-10 00:45:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 1530/2304 [I 2022-10-10 00:45:58] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:00 [I 2022-10-10 00:45:58] ...trap/engines/engine.py.229: process: 0.86109 | load: 0.00312 [I 2022-10-10 00:45:58] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-10 00:45:58] ...optimizers/trijoint.py.151: total_norm: 19.666115 [I 2022-10-10 00:46:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1540/2304 [I 2022-10-10 00:46:07] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:52 [I 2022-10-10 00:46:07] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00312 [I 2022-10-10 00:46:07] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-10 00:46:07] ...optimizers/trijoint.py.151: total_norm: 18.077803 [I 2022-10-10 00:46:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 1550/2304 [I 2022-10-10 00:46:15] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:37 [I 2022-10-10 00:46:15] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00313 [I 2022-10-10 00:46:15] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-10 00:46:15] ...optimizers/trijoint.py.151: total_norm: 23.505770 [I 2022-10-10 00:46:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 1560/2304 [I 2022-10-10 00:46:24] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:32 [I 2022-10-10 00:46:24] ...trap/engines/engine.py.229: process: 0.84014 | load: 0.00319 [I 2022-10-10 00:46:24] ...trap/engines/engine.py.230: loss: 0.14991 [I 2022-10-10 00:46:24] ...optimizers/trijoint.py.151: total_norm: 22.324974 [I 2022-10-10 00:46:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 1570/2304 [I 2022-10-10 00:46:32] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:24 [I 2022-10-10 00:46:33] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00320 [I 2022-10-10 00:46:33] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-10 00:46:33] ...optimizers/trijoint.py.151: total_norm: 18.068218 [I 2022-10-10 00:46:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 1580/2304 [I 2022-10-10 00:46:41] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:16 [I 2022-10-10 00:46:41] ...trap/engines/engine.py.229: process: 0.85276 | load: 0.00315 [I 2022-10-10 00:46:41] ...trap/engines/engine.py.230: loss: 0.17988 [I 2022-10-10 00:46:41] ...optimizers/trijoint.py.151: total_norm: 12.796279 [I 2022-10-10 00:46:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1590/2304 [I 2022-10-10 00:46:50] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:08 [I 2022-10-10 00:46:50] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00311 [I 2022-10-10 00:46:50] ...trap/engines/engine.py.230: loss: 0.17991 [I 2022-10-10 00:46:50] ...optimizers/trijoint.py.151: total_norm: 15.102230 [I 2022-10-10 00:46:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 1600/2304 [I 2022-10-10 00:46:58] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:10:00 [I 2022-10-10 00:46:58] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00306 [I 2022-10-10 00:46:58] ...trap/engines/engine.py.230: loss: 0.11468 [I 2022-10-10 00:46:58] ...optimizers/trijoint.py.151: total_norm: 15.303725 [I 2022-10-10 00:47:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1610/2304 [I 2022-10-10 00:47:07] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:48 [I 2022-10-10 00:47:07] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00299 [I 2022-10-10 00:47:07] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-10 00:47:07] ...optimizers/trijoint.py.151: total_norm: 16.726374 [I 2022-10-10 00:47:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 1620/2304 [I 2022-10-10 00:47:15] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:39 [I 2022-10-10 00:47:15] ...trap/engines/engine.py.229: process: 0.84967 | load: 0.00298 [I 2022-10-10 00:47:15] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-10 00:47:15] ...optimizers/trijoint.py.151: total_norm: 16.230043 [I 2022-10-10 00:47:23] ...trap/engines/engine.py.224: train: epoch 60 | batch 1630/2304 [I 2022-10-10 00:47:23] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:27 [I 2022-10-10 00:47:23] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00295 [I 2022-10-10 00:47:23] ...trap/engines/engine.py.230: loss: 0.14939 [I 2022-10-10 00:47:23] ...optimizers/trijoint.py.151: total_norm: 21.763943 [I 2022-10-10 00:47:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 1640/2304 [I 2022-10-10 00:47:32] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:17 [I 2022-10-10 00:47:32] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00294 [I 2022-10-10 00:47:32] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-10 00:47:32] ...optimizers/trijoint.py.151: total_norm: 26.417849 [I 2022-10-10 00:47:40] ...trap/engines/engine.py.224: train: epoch 60 | batch 1650/2304 [I 2022-10-10 00:47:40] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:10 [I 2022-10-10 00:47:40] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00295 [I 2022-10-10 00:47:40] ...trap/engines/engine.py.230: loss: 0.19003 [I 2022-10-10 00:47:40] ...optimizers/trijoint.py.151: total_norm: 17.780209 [I 2022-10-10 00:47:49] ...trap/engines/engine.py.224: train: epoch 60 | batch 1660/2304 [I 2022-10-10 00:47:49] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:59 [I 2022-10-10 00:47:49] ...trap/engines/engine.py.229: process: 0.83598 | load: 0.00298 [I 2022-10-10 00:47:49] ...trap/engines/engine.py.230: loss: 0.10121 [I 2022-10-10 00:47:49] ...optimizers/trijoint.py.151: total_norm: 8.292447 [I 2022-10-10 00:47:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 1670/2304 [I 2022-10-10 00:47:57] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:53 [I 2022-10-10 00:47:57] ...trap/engines/engine.py.229: process: 0.82893 | load: 0.00292 [I 2022-10-10 00:47:57] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-10 00:47:57] ...optimizers/trijoint.py.151: total_norm: 8.996557 [I 2022-10-10 00:48:06] ...trap/engines/engine.py.224: train: epoch 60 | batch 1680/2304 [I 2022-10-10 00:48:06] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:45 [I 2022-10-10 00:48:06] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00300 [I 2022-10-10 00:48:06] ...trap/engines/engine.py.230: loss: 0.11636 [I 2022-10-10 00:48:06] ...optimizers/trijoint.py.151: total_norm: 9.287631 [I 2022-10-10 00:48:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 1690/2304 [I 2022-10-10 00:48:14] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:37 [I 2022-10-10 00:48:14] ...trap/engines/engine.py.229: process: 0.83280 | load: 0.00299 [I 2022-10-10 00:48:14] ...trap/engines/engine.py.230: loss: 0.12165 [I 2022-10-10 00:48:14] ...optimizers/trijoint.py.151: total_norm: 10.292375 [I 2022-10-10 00:48:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 1700/2304 [I 2022-10-10 00:48:22] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:29 [I 2022-10-10 00:48:22] ...trap/engines/engine.py.229: process: 0.83686 | load: 0.00299 [I 2022-10-10 00:48:22] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-10 00:48:22] ...optimizers/trijoint.py.151: total_norm: 44.917225 [I 2022-10-10 00:48:31] ...trap/engines/engine.py.224: train: epoch 60 | batch 1710/2304 [I 2022-10-10 00:48:31] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:22 [I 2022-10-10 00:48:31] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00297 [I 2022-10-10 00:48:31] ...trap/engines/engine.py.230: loss: 0.15149 [I 2022-10-10 00:48:31] ...optimizers/trijoint.py.151: total_norm: 28.450542 [I 2022-10-10 00:48:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 1720/2304 [I 2022-10-10 00:48:39] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:14 [I 2022-10-10 00:48:39] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00312 [I 2022-10-10 00:48:39] ...trap/engines/engine.py.230: loss: 0.11009 [I 2022-10-10 00:48:39] ...optimizers/trijoint.py.151: total_norm: 9.265977 [I 2022-10-10 00:48:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 1730/2304 [I 2022-10-10 00:48:48] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:06 [I 2022-10-10 00:48:48] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00312 [I 2022-10-10 00:48:48] ...trap/engines/engine.py.230: loss: 0.17268 [I 2022-10-10 00:48:48] ...optimizers/trijoint.py.151: total_norm: 15.849090 [I 2022-10-10 00:48:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 1740/2304 [I 2022-10-10 00:48:56] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:55 [I 2022-10-10 00:48:56] ...trap/engines/engine.py.229: process: 0.83640 | load: 0.00298 [I 2022-10-10 00:48:56] ...trap/engines/engine.py.230: loss: 0.10827 [I 2022-10-10 00:48:56] ...optimizers/trijoint.py.151: total_norm: 11.962645 [I 2022-10-10 00:49:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 1750/2304 [I 2022-10-10 00:49:05] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:47 [I 2022-10-10 00:49:05] ...trap/engines/engine.py.229: process: 0.82133 | load: 0.00302 [I 2022-10-10 00:49:05] ...trap/engines/engine.py.230: loss: 0.09956 [I 2022-10-10 00:49:05] ...optimizers/trijoint.py.151: total_norm: 8.410769 [I 2022-10-10 00:49:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 1760/2304 [I 2022-10-10 00:49:13] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:41 [I 2022-10-10 00:49:13] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00298 [I 2022-10-10 00:49:13] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-10 00:49:13] ...optimizers/trijoint.py.151: total_norm: 16.994318 [I 2022-10-10 00:49:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 1770/2304 [I 2022-10-10 00:49:22] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:32 [I 2022-10-10 00:49:22] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00297 [I 2022-10-10 00:49:22] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-10 00:49:22] ...optimizers/trijoint.py.151: total_norm: 12.262318 [I 2022-10-10 00:49:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 1780/2304 [I 2022-10-10 00:49:30] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:20 [I 2022-10-10 00:49:30] ...trap/engines/engine.py.229: process: 0.82462 | load: 0.00299 [I 2022-10-10 00:49:30] ...trap/engines/engine.py.230: loss: 0.11864 [I 2022-10-10 00:49:30] ...optimizers/trijoint.py.151: total_norm: 7.610816 [I 2022-10-10 00:49:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 1790/2304 [I 2022-10-10 00:49:39] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:07:16 [I 2022-10-10 00:49:39] ...trap/engines/engine.py.229: process: 0.85483 | load: 0.00314 [I 2022-10-10 00:49:39] ...trap/engines/engine.py.230: loss: 0.17900 [I 2022-10-10 00:49:39] ...optimizers/trijoint.py.151: total_norm: 19.213915 [I 2022-10-10 00:49:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 1800/2304 [I 2022-10-10 00:49:47] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:11 [I 2022-10-10 00:49:47] ...trap/engines/engine.py.229: process: 0.86293 | load: 0.00316 [I 2022-10-10 00:49:47] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-10 00:49:47] ...optimizers/trijoint.py.151: total_norm: 13.055141 [I 2022-10-10 00:49:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 1810/2304 [I 2022-10-10 00:49:56] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:07:00 [I 2022-10-10 00:49:56] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00313 [I 2022-10-10 00:49:56] ...trap/engines/engine.py.230: loss: 0.10026 [I 2022-10-10 00:49:56] ...optimizers/trijoint.py.151: total_norm: 11.766088 [I 2022-10-10 00:50:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 1820/2304 [I 2022-10-10 00:50:04] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:53 [I 2022-10-10 00:50:04] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00315 [I 2022-10-10 00:50:04] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-10 00:50:04] ...optimizers/trijoint.py.151: total_norm: 12.821737 [I 2022-10-10 00:50:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 1830/2304 [I 2022-10-10 00:50:13] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:45 [I 2022-10-10 00:50:13] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00314 [I 2022-10-10 00:50:13] ...trap/engines/engine.py.230: loss: 0.17476 [I 2022-10-10 00:50:13] ...optimizers/trijoint.py.151: total_norm: 12.860737 [I 2022-10-10 00:50:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 1840/2304 [I 2022-10-10 00:50:21] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:36 [I 2022-10-10 00:50:21] ...trap/engines/engine.py.229: process: 0.85593 | load: 0.00315 [I 2022-10-10 00:50:21] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-10 00:50:21] ...optimizers/trijoint.py.151: total_norm: 13.791755 [I 2022-10-10 00:50:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 1850/2304 [I 2022-10-10 00:50:30] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:25 [I 2022-10-10 00:50:30] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00312 [I 2022-10-10 00:50:30] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-10 00:50:30] ...optimizers/trijoint.py.151: total_norm: 19.613104 [I 2022-10-10 00:50:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 1860/2304 [I 2022-10-10 00:50:39] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:21 [I 2022-10-10 00:50:39] ...trap/engines/engine.py.229: process: 0.86856 | load: 0.00314 [I 2022-10-10 00:50:39] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-10 00:50:39] ...optimizers/trijoint.py.151: total_norm: 32.064178 [I 2022-10-10 00:50:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 1870/2304 [I 2022-10-10 00:50:47] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:12 [I 2022-10-10 00:50:47] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00309 [I 2022-10-10 00:50:47] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-10 00:50:47] ...optimizers/trijoint.py.151: total_norm: 19.032776 [I 2022-10-10 00:50:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 1880/2304 [I 2022-10-10 00:50:56] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:06:02 [I 2022-10-10 00:50:56] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00304 [I 2022-10-10 00:50:56] ...trap/engines/engine.py.230: loss: 0.15001 [I 2022-10-10 00:50:56] ...optimizers/trijoint.py.151: total_norm: 17.489157 [I 2022-10-10 00:51:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 1890/2304 [I 2022-10-10 00:51:04] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:54 [I 2022-10-10 00:51:04] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00313 [I 2022-10-10 00:51:04] ...trap/engines/engine.py.230: loss: 0.11365 [I 2022-10-10 00:51:04] ...optimizers/trijoint.py.151: total_norm: 10.759583 [I 2022-10-10 00:51:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 1900/2304 [I 2022-10-10 00:51:13] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:47 [I 2022-10-10 00:51:13] ...trap/engines/engine.py.229: process: 0.87076 | load: 0.00313 [I 2022-10-10 00:51:13] ...trap/engines/engine.py.230: loss: 0.13547 [I 2022-10-10 00:51:13] ...optimizers/trijoint.py.151: total_norm: 17.220016 [I 2022-10-10 00:51:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 1910/2304 [I 2022-10-10 00:51:21] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:36 [I 2022-10-10 00:51:21] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00314 [I 2022-10-10 00:51:21] ...trap/engines/engine.py.230: loss: 0.10702 [I 2022-10-10 00:51:21] ...optimizers/trijoint.py.151: total_norm: 29.971828 [I 2022-10-10 00:51:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 1920/2304 [I 2022-10-10 00:51:30] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:26 [I 2022-10-10 00:51:30] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00312 [I 2022-10-10 00:51:30] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-10 00:51:30] ...optimizers/trijoint.py.151: total_norm: 11.630068 [I 2022-10-10 00:51:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 1930/2304 [I 2022-10-10 00:51:38] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:05:16 [I 2022-10-10 00:51:38] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00314 [I 2022-10-10 00:51:38] ...trap/engines/engine.py.230: loss: 0.26626 [I 2022-10-10 00:51:38] ...optimizers/trijoint.py.151: total_norm: 24.613485 [I 2022-10-10 00:51:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 1940/2304 [I 2022-10-10 00:51:47] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:05:08 [I 2022-10-10 00:51:47] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00313 [I 2022-10-10 00:51:47] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-10 00:51:47] ...optimizers/trijoint.py.151: total_norm: 20.332300 [I 2022-10-10 00:51:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 1950/2304 [I 2022-10-10 00:51:55] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:05:00 [I 2022-10-10 00:51:55] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00316 [I 2022-10-10 00:51:55] ...trap/engines/engine.py.230: loss: 0.12218 [I 2022-10-10 00:51:55] ...optimizers/trijoint.py.151: total_norm: 11.529617 [I 2022-10-10 00:52:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 1960/2304 [I 2022-10-10 00:52:04] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:52 [I 2022-10-10 00:52:04] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00316 [I 2022-10-10 00:52:04] ...trap/engines/engine.py.230: loss: 0.12544 [I 2022-10-10 00:52:04] ...optimizers/trijoint.py.151: total_norm: 13.357366 [I 2022-10-10 00:52:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 1970/2304 [I 2022-10-10 00:52:12] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:44 [I 2022-10-10 00:52:12] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00313 [I 2022-10-10 00:52:12] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-10 00:52:12] ...optimizers/trijoint.py.151: total_norm: 11.389480 [I 2022-10-10 00:52:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 1980/2304 [I 2022-10-10 00:52:21] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:36 [I 2022-10-10 00:52:21] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00312 [I 2022-10-10 00:52:21] ...trap/engines/engine.py.230: loss: 0.14325 [I 2022-10-10 00:52:21] ...optimizers/trijoint.py.151: total_norm: 30.762167 [I 2022-10-10 00:52:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 1990/2304 [I 2022-10-10 00:52:30] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:26 [I 2022-10-10 00:52:30] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00310 [I 2022-10-10 00:52:30] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-10 00:52:30] ...optimizers/trijoint.py.151: total_norm: 18.180065 [I 2022-10-10 00:52:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 2000/2304 [I 2022-10-10 00:52:38] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:19 [I 2022-10-10 00:52:38] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00311 [I 2022-10-10 00:52:38] ...trap/engines/engine.py.230: loss: 0.17207 [I 2022-10-10 00:52:38] ...optimizers/trijoint.py.151: total_norm: 17.455658 [I 2022-10-10 00:52:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 2010/2304 [I 2022-10-10 00:52:47] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:04:11 [I 2022-10-10 00:52:47] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00314 [I 2022-10-10 00:52:47] ...trap/engines/engine.py.230: loss: 0.09456 [I 2022-10-10 00:52:47] ...optimizers/trijoint.py.151: total_norm: 12.286177 [I 2022-10-10 00:52:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 2020/2304 [I 2022-10-10 00:52:55] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:03 [I 2022-10-10 00:52:55] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00313 [I 2022-10-10 00:52:55] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-10 00:52:55] ...optimizers/trijoint.py.151: total_norm: 9.237307 [I 2022-10-10 00:53:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 2030/2304 [I 2022-10-10 00:53:04] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:53 [I 2022-10-10 00:53:04] ...trap/engines/engine.py.229: process: 0.83806 | load: 0.00318 [I 2022-10-10 00:53:04] ...trap/engines/engine.py.230: loss: 0.18843 [I 2022-10-10 00:53:04] ...optimizers/trijoint.py.151: total_norm: 19.612228 [I 2022-10-10 00:53:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 2040/2304 [I 2022-10-10 00:53:12] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:45 [I 2022-10-10 00:53:12] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00304 [I 2022-10-10 00:53:12] ...trap/engines/engine.py.230: loss: 0.15524 [I 2022-10-10 00:53:12] ...optimizers/trijoint.py.151: total_norm: 18.819313 [I 2022-10-10 00:53:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 2050/2304 [I 2022-10-10 00:53:21] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:36 [I 2022-10-10 00:53:21] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00311 [I 2022-10-10 00:53:21] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-10 00:53:21] ...optimizers/trijoint.py.151: total_norm: 15.893561 [I 2022-10-10 00:53:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 2060/2304 [I 2022-10-10 00:53:29] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:28 [I 2022-10-10 00:53:29] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00313 [I 2022-10-10 00:53:29] ...trap/engines/engine.py.230: loss: 0.19544 [I 2022-10-10 00:53:29] ...optimizers/trijoint.py.151: total_norm: 11.264542 [I 2022-10-10 00:53:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 2070/2304 [I 2022-10-10 00:53:38] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:19 [I 2022-10-10 00:53:38] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00302 [I 2022-10-10 00:53:38] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-10 00:53:38] ...optimizers/trijoint.py.151: total_norm: 24.057819 [I 2022-10-10 00:53:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 2080/2304 [I 2022-10-10 00:53:47] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:12 [I 2022-10-10 00:53:47] ...trap/engines/engine.py.229: process: 0.89471 | load: 0.00311 [I 2022-10-10 00:53:47] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-10 00:53:47] ...optimizers/trijoint.py.151: total_norm: 15.675426 [I 2022-10-10 00:53:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 2090/2304 [I 2022-10-10 00:53:55] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:02 [I 2022-10-10 00:53:55] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00315 [I 2022-10-10 00:53:55] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-10 00:53:55] ...optimizers/trijoint.py.151: total_norm: 22.579624 [I 2022-10-10 00:54:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 2100/2304 [I 2022-10-10 00:54:04] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:54 [I 2022-10-10 00:54:04] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00311 [I 2022-10-10 00:54:04] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-10 00:54:04] ...optimizers/trijoint.py.151: total_norm: 11.338152 [I 2022-10-10 00:54:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 2110/2304 [I 2022-10-10 00:54:12] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:45 [I 2022-10-10 00:54:12] ...trap/engines/engine.py.229: process: 0.85536 | load: 0.00314 [I 2022-10-10 00:54:12] ...trap/engines/engine.py.230: loss: 0.13158 [I 2022-10-10 00:54:12] ...optimizers/trijoint.py.151: total_norm: 20.676229 [I 2022-10-10 00:54:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 2120/2304 [I 2022-10-10 00:54:21] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:36 [I 2022-10-10 00:54:21] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00311 [I 2022-10-10 00:54:21] ...trap/engines/engine.py.230: loss: 0.18841 [I 2022-10-10 00:54:21] ...optimizers/trijoint.py.151: total_norm: 21.214687 [I 2022-10-10 00:54:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 2130/2304 [I 2022-10-10 00:54:29] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:28 [I 2022-10-10 00:54:29] ...trap/engines/engine.py.229: process: 0.85832 | load: 0.00313 [I 2022-10-10 00:54:29] ...trap/engines/engine.py.230: loss: 0.16792 [I 2022-10-10 00:54:29] ...optimizers/trijoint.py.151: total_norm: 20.284424 [I 2022-10-10 00:54:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 2140/2304 [I 2022-10-10 00:54:38] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:18 [I 2022-10-10 00:54:38] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00297 [I 2022-10-10 00:54:38] ...trap/engines/engine.py.230: loss: 0.17026 [I 2022-10-10 00:54:38] ...optimizers/trijoint.py.151: total_norm: 14.094131 [I 2022-10-10 00:54:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 2150/2304 [I 2022-10-10 00:54:46] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:10 [I 2022-10-10 00:54:46] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00294 [I 2022-10-10 00:54:46] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-10 00:54:46] ...optimizers/trijoint.py.151: total_norm: 13.297616 [I 2022-10-10 00:54:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 2160/2304 [I 2022-10-10 00:54:55] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:01 [I 2022-10-10 00:54:55] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00297 [I 2022-10-10 00:54:55] ...trap/engines/engine.py.230: loss: 0.16495 [I 2022-10-10 00:54:55] ...optimizers/trijoint.py.151: total_norm: 17.141386 [I 2022-10-10 00:55:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 2170/2304 [I 2022-10-10 00:55:03] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:01:53 [I 2022-10-10 00:55:03] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00296 [I 2022-10-10 00:55:03] ...trap/engines/engine.py.230: loss: 0.11471 [I 2022-10-10 00:55:03] ...optimizers/trijoint.py.151: total_norm: 10.359327 [I 2022-10-10 00:55:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 2180/2304 [I 2022-10-10 00:55:12] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:44 [I 2022-10-10 00:55:12] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00295 [I 2022-10-10 00:55:12] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-10 00:55:12] ...optimizers/trijoint.py.151: total_norm: 10.202254 [I 2022-10-10 00:55:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 2190/2304 [I 2022-10-10 00:55:20] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:36 [I 2022-10-10 00:55:20] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00293 [I 2022-10-10 00:55:20] ...trap/engines/engine.py.230: loss: 0.11146 [I 2022-10-10 00:55:20] ...optimizers/trijoint.py.151: total_norm: 11.076900 [I 2022-10-10 00:55:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 2200/2304 [I 2022-10-10 00:55:28] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:27 [I 2022-10-10 00:55:28] ...trap/engines/engine.py.229: process: 0.83226 | load: 0.00294 [I 2022-10-10 00:55:28] ...trap/engines/engine.py.230: loss: 0.20626 [I 2022-10-10 00:55:28] ...optimizers/trijoint.py.151: total_norm: 13.064901 [I 2022-10-10 00:55:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 2210/2304 [I 2022-10-10 00:55:37] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:19 [I 2022-10-10 00:55:37] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00296 [I 2022-10-10 00:55:37] ...trap/engines/engine.py.230: loss: 0.14979 [I 2022-10-10 00:55:37] ...optimizers/trijoint.py.151: total_norm: 11.331690 [I 2022-10-10 00:55:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 2220/2304 [I 2022-10-10 00:55:45] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:10 [I 2022-10-10 00:55:45] ...trap/engines/engine.py.229: process: 0.84127 | load: 0.00302 [I 2022-10-10 00:55:45] ...trap/engines/engine.py.230: loss: 0.21341 [I 2022-10-10 00:55:45] ...optimizers/trijoint.py.151: total_norm: 12.553759 [I 2022-10-10 00:55:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 2230/2304 [I 2022-10-10 00:55:54] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:02 [I 2022-10-10 00:55:54] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00301 [I 2022-10-10 00:55:54] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-10 00:55:54] ...optimizers/trijoint.py.151: total_norm: 13.504221 [I 2022-10-10 00:56:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 2240/2304 [I 2022-10-10 00:56:02] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:53 [I 2022-10-10 00:56:02] ...trap/engines/engine.py.229: process: 0.83671 | load: 0.00298 [I 2022-10-10 00:56:02] ...trap/engines/engine.py.230: loss: 0.19100 [I 2022-10-10 00:56:02] ...optimizers/trijoint.py.151: total_norm: 18.165354 [I 2022-10-10 00:56:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 2250/2304 [I 2022-10-10 00:56:11] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:45 [I 2022-10-10 00:56:11] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00312 [I 2022-10-10 00:56:11] ...trap/engines/engine.py.230: loss: 0.18938 [I 2022-10-10 00:56:11] ...optimizers/trijoint.py.151: total_norm: 38.068527 [I 2022-10-10 00:56:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 2260/2304 [I 2022-10-10 00:56:19] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:37 [I 2022-10-10 00:56:19] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00302 [I 2022-10-10 00:56:19] ...trap/engines/engine.py.230: loss: 0.18670 [I 2022-10-10 00:56:19] ...optimizers/trijoint.py.151: total_norm: 16.473103 [I 2022-10-10 00:56:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 2270/2304 [I 2022-10-10 00:56:28] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:28 [I 2022-10-10 00:56:28] ...trap/engines/engine.py.229: process: 0.83560 | load: 0.00299 [I 2022-10-10 00:56:28] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-10 00:56:28] ...optimizers/trijoint.py.151: total_norm: 7.515091 [I 2022-10-10 00:56:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 2280/2304 [I 2022-10-10 00:56:36] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:20 [I 2022-10-10 00:56:36] ...trap/engines/engine.py.229: process: 0.83548 | load: 0.00299 [I 2022-10-10 00:56:36] ...trap/engines/engine.py.230: loss: 0.09886 [I 2022-10-10 00:56:36] ...optimizers/trijoint.py.151: total_norm: 10.203400 [I 2022-10-10 00:56:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 2290/2304 [I 2022-10-10 00:56:45] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:11 [I 2022-10-10 00:56:45] ...trap/engines/engine.py.229: process: 0.84960 | load: 0.00299 [I 2022-10-10 00:56:45] ...trap/engines/engine.py.230: loss: 0.12851 [I 2022-10-10 00:56:45] ...optimizers/trijoint.py.151: total_norm: 32.484478 [I 2022-10-10 00:56:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 2300/2304 [I 2022-10-10 00:56:53] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:03 [I 2022-10-10 00:56:53] ...trap/engines/engine.py.229: process: 0.83136 | load: 0.00017 [I 2022-10-10 00:56:53] ...trap/engines/engine.py.230: loss: 0.20121 [I 2022-10-10 00:56:53] ...optimizers/trijoint.py.151: total_norm: 30.074259 [I 2022-10-10 00:56:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 2304/2304 [I 2022-10-10 00:56:56] ...trap/engines/engine.py.228: elapsed: 0:33:04 | left: 0:00:00 [I 2022-10-10 00:56:56] ...trap/engines/engine.py.229: process: 0.82811 | load: 0.00017 [I 2022-10-10 00:56:56] ...trap/engines/engine.py.230: loss: 0.13942 [I 2022-10-10 00:56:56] ...optimizers/trijoint.py.151: total_norm: 19.017038 [S 2022-10-10 00:56:58] ...trap/engines/engine.py.129: train_epoch.epoch: 60 [S 2022-10-10 00:56:58] ...trap/engines/engine.py.129: train_epoch.loss: 0.14744403500942244 [I 2022-10-10 00:56:58] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 00:56:59] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 60 [I 2022-10-10 00:56:59] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 00:56:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 00:56:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 00:56:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 00:57:02] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 00:57:03] ...trap/engines/engine.py.321: eval: epoch 60 | batch 0/510 [I 2022-10-10 00:57:03] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:03 [I 2022-10-10 00:57:03] ...trap/engines/engine.py.326: process: 4.24272 | load: 3.17986 [I 2022-10-10 00:57:09] ...trap/engines/engine.py.321: eval: epoch 60 | batch 10/510 [I 2022-10-10 00:57:09] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:21 [I 2022-10-10 00:57:09] ...trap/engines/engine.py.326: process: 0.64044 | load: 0.00020 [I 2022-10-10 00:57:15] ...trap/engines/engine.py.321: eval: epoch 60 | batch 20/510 [I 2022-10-10 00:57:15] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:14 [I 2022-10-10 00:57:15] ...trap/engines/engine.py.326: process: 0.61163 | load: 0.00020 [I 2022-10-10 00:57:20] ...trap/engines/engine.py.321: eval: epoch 60 | batch 30/510 [I 2022-10-10 00:57:20] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:14 [I 2022-10-10 00:57:20] ...trap/engines/engine.py.326: process: 0.50167 | load: 0.00020 [I 2022-10-10 00:57:25] ...trap/engines/engine.py.321: eval: epoch 60 | batch 40/510 [I 2022-10-10 00:57:25] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:03:55 [I 2022-10-10 00:57:25] ...trap/engines/engine.py.326: process: 0.49686 | load: 0.00021 [I 2022-10-10 00:57:30] ...trap/engines/engine.py.321: eval: epoch 60 | batch 50/510 [I 2022-10-10 00:57:30] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:59 [I 2022-10-10 00:57:30] ...trap/engines/engine.py.326: process: 0.56421 | load: 0.00021 [I 2022-10-10 00:57:35] ...trap/engines/engine.py.321: eval: epoch 60 | batch 60/510 [I 2022-10-10 00:57:35] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:50 [I 2022-10-10 00:57:35] ...trap/engines/engine.py.326: process: 0.48008 | load: 0.00019 [I 2022-10-10 00:57:40] ...trap/engines/engine.py.321: eval: epoch 60 | batch 70/510 [I 2022-10-10 00:57:40] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:51 [I 2022-10-10 00:57:40] ...trap/engines/engine.py.326: process: 0.54410 | load: 0.00020 [I 2022-10-10 00:57:45] ...trap/engines/engine.py.321: eval: epoch 60 | batch 80/510 [I 2022-10-10 00:57:45] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:40 [I 2022-10-10 00:57:45] ...trap/engines/engine.py.326: process: 0.56915 | load: 0.00020 [I 2022-10-10 00:57:50] ...trap/engines/engine.py.321: eval: epoch 60 | batch 90/510 [I 2022-10-10 00:57:50] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:11 [I 2022-10-10 00:57:50] ...trap/engines/engine.py.326: process: 0.44882 | load: 0.00019 [I 2022-10-10 00:57:54] ...trap/engines/engine.py.321: eval: epoch 60 | batch 100/510 [I 2022-10-10 00:57:54] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:57 [I 2022-10-10 00:57:54] ...trap/engines/engine.py.326: process: 0.31058 | load: 0.00022 [I 2022-10-10 00:57:58] ...trap/engines/engine.py.321: eval: epoch 60 | batch 110/510 [I 2022-10-10 00:57:58] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:13 [I 2022-10-10 00:57:58] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00020 [I 2022-10-10 00:58:01] ...trap/engines/engine.py.321: eval: epoch 60 | batch 120/510 [I 2022-10-10 00:58:01] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:04 [I 2022-10-10 00:58:01] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00020 [I 2022-10-10 00:58:04] ...trap/engines/engine.py.321: eval: epoch 60 | batch 130/510 [I 2022-10-10 00:58:04] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:01 [I 2022-10-10 00:58:04] ...trap/engines/engine.py.326: process: 0.31998 | load: 0.00021 [I 2022-10-10 00:58:07] ...trap/engines/engine.py.321: eval: epoch 60 | batch 140/510 [I 2022-10-10 00:58:07] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:58 [I 2022-10-10 00:58:07] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00021 [I 2022-10-10 00:58:10] ...trap/engines/engine.py.321: eval: epoch 60 | batch 150/510 [I 2022-10-10 00:58:10] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:55 [I 2022-10-10 00:58:10] ...trap/engines/engine.py.326: process: 0.31681 | load: 0.00020 [I 2022-10-10 00:58:14] ...trap/engines/engine.py.321: eval: epoch 60 | batch 160/510 [I 2022-10-10 00:58:14] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:52 [I 2022-10-10 00:58:14] ...trap/engines/engine.py.326: process: 0.32111 | load: 0.00020 [I 2022-10-10 00:58:17] ...trap/engines/engine.py.321: eval: epoch 60 | batch 170/510 [I 2022-10-10 00:58:17] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:49 [I 2022-10-10 00:58:17] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00021 [I 2022-10-10 00:58:20] ...trap/engines/engine.py.321: eval: epoch 60 | batch 180/510 [I 2022-10-10 00:58:20] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:46 [I 2022-10-10 00:58:20] ...trap/engines/engine.py.326: process: 0.32437 | load: 0.00020 [I 2022-10-10 00:58:23] ...trap/engines/engine.py.321: eval: epoch 60 | batch 190/510 [I 2022-10-10 00:58:23] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:43 [I 2022-10-10 00:58:23] ...trap/engines/engine.py.326: process: 0.32599 | load: 0.00020 [I 2022-10-10 00:58:27] ...trap/engines/engine.py.321: eval: epoch 60 | batch 200/510 [I 2022-10-10 00:58:27] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-10 00:58:27] ...trap/engines/engine.py.326: process: 0.32151 | load: 0.00021 [I 2022-10-10 00:58:30] ...trap/engines/engine.py.321: eval: epoch 60 | batch 210/510 [I 2022-10-10 00:58:30] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:37 [I 2022-10-10 00:58:30] ...trap/engines/engine.py.326: process: 0.32067 | load: 0.00021 [I 2022-10-10 00:58:33] ...trap/engines/engine.py.321: eval: epoch 60 | batch 220/510 [I 2022-10-10 00:58:33] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:33 [I 2022-10-10 00:58:33] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00022 [I 2022-10-10 00:58:36] ...trap/engines/engine.py.321: eval: epoch 60 | batch 230/510 [I 2022-10-10 00:58:36] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:30 [I 2022-10-10 00:58:36] ...trap/engines/engine.py.326: process: 0.32509 | load: 0.00024 [I 2022-10-10 00:58:40] ...trap/engines/engine.py.321: eval: epoch 60 | batch 240/510 [I 2022-10-10 00:58:40] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:27 [I 2022-10-10 00:58:40] ...trap/engines/engine.py.326: process: 0.31849 | load: 0.00022 [I 2022-10-10 00:58:43] ...trap/engines/engine.py.321: eval: epoch 60 | batch 250/510 [I 2022-10-10 00:58:43] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:23 [I 2022-10-10 00:58:43] ...trap/engines/engine.py.326: process: 0.31354 | load: 0.00021 [I 2022-10-10 00:58:46] ...trap/engines/engine.py.321: eval: epoch 60 | batch 260/510 [I 2022-10-10 00:58:46] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:20 [I 2022-10-10 00:58:46] ...trap/engines/engine.py.326: process: 0.32201 | load: 0.00020 [I 2022-10-10 00:58:49] ...trap/engines/engine.py.321: eval: epoch 60 | batch 270/510 [I 2022-10-10 00:58:49] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:17 [I 2022-10-10 00:58:49] ...trap/engines/engine.py.326: process: 0.31328 | load: 0.00021 [I 2022-10-10 00:58:52] ...trap/engines/engine.py.321: eval: epoch 60 | batch 280/510 [I 2022-10-10 00:58:52] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-10 00:58:52] ...trap/engines/engine.py.326: process: 0.32074 | load: 0.00022 [I 2022-10-10 00:58:56] ...trap/engines/engine.py.321: eval: epoch 60 | batch 290/510 [I 2022-10-10 00:58:56] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-10 00:58:56] ...trap/engines/engine.py.326: process: 0.31623 | load: 0.00020 [I 2022-10-10 00:58:59] ...trap/engines/engine.py.321: eval: epoch 60 | batch 300/510 [I 2022-10-10 00:58:59] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:07 [I 2022-10-10 00:58:59] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00021 [I 2022-10-10 00:59:02] ...trap/engines/engine.py.321: eval: epoch 60 | batch 310/510 [I 2022-10-10 00:59:02] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:04 [I 2022-10-10 00:59:02] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00021 [I 2022-10-10 00:59:05] ...trap/engines/engine.py.321: eval: epoch 60 | batch 320/510 [I 2022-10-10 00:59:05] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:01 [I 2022-10-10 00:59:05] ...trap/engines/engine.py.326: process: 0.32224 | load: 0.00021 [I 2022-10-10 00:59:09] ...trap/engines/engine.py.321: eval: epoch 60 | batch 330/510 [I 2022-10-10 00:59:09] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:58 [I 2022-10-10 00:59:09] ...trap/engines/engine.py.326: process: 0.32642 | load: 0.00021 [I 2022-10-10 00:59:12] ...trap/engines/engine.py.321: eval: epoch 60 | batch 340/510 [I 2022-10-10 00:59:12] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-10 00:59:12] ...trap/engines/engine.py.326: process: 0.32519 | load: 0.00020 [I 2022-10-10 00:59:15] ...trap/engines/engine.py.321: eval: epoch 60 | batch 350/510 [I 2022-10-10 00:59:15] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-10 00:59:15] ...trap/engines/engine.py.326: process: 0.32415 | load: 0.00022 [I 2022-10-10 00:59:18] ...trap/engines/engine.py.321: eval: epoch 60 | batch 360/510 [I 2022-10-10 00:59:18] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:48 [I 2022-10-10 00:59:18] ...trap/engines/engine.py.326: process: 0.31677 | load: 0.00021 [I 2022-10-10 00:59:22] ...trap/engines/engine.py.321: eval: epoch 60 | batch 370/510 [I 2022-10-10 00:59:22] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:45 [I 2022-10-10 00:59:22] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00022 [I 2022-10-10 00:59:25] ...trap/engines/engine.py.321: eval: epoch 60 | batch 380/510 [I 2022-10-10 00:59:25] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-10 00:59:25] ...trap/engines/engine.py.326: process: 0.31759 | load: 0.00022 [I 2022-10-10 00:59:28] ...trap/engines/engine.py.321: eval: epoch 60 | batch 390/510 [I 2022-10-10 00:59:28] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:39 [I 2022-10-10 00:59:28] ...trap/engines/engine.py.326: process: 0.32817 | load: 0.00021 [I 2022-10-10 00:59:31] ...trap/engines/engine.py.321: eval: epoch 60 | batch 400/510 [I 2022-10-10 00:59:31] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-10 00:59:31] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00023 [I 2022-10-10 00:59:35] ...trap/engines/engine.py.321: eval: epoch 60 | batch 410/510 [I 2022-10-10 00:59:35] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:32 [I 2022-10-10 00:59:35] ...trap/engines/engine.py.326: process: 0.32544 | load: 0.00022 [I 2022-10-10 00:59:38] ...trap/engines/engine.py.321: eval: epoch 60 | batch 420/510 [I 2022-10-10 00:59:38] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:29 [I 2022-10-10 00:59:38] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00022 [I 2022-10-10 00:59:41] ...trap/engines/engine.py.321: eval: epoch 60 | batch 430/510 [I 2022-10-10 00:59:41] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-10 00:59:41] ...trap/engines/engine.py.326: process: 0.33072 | load: 0.00021 [I 2022-10-10 00:59:44] ...trap/engines/engine.py.321: eval: epoch 60 | batch 440/510 [I 2022-10-10 00:59:44] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-10 00:59:44] ...trap/engines/engine.py.326: process: 0.32201 | load: 0.00021 [I 2022-10-10 00:59:47] ...trap/engines/engine.py.321: eval: epoch 60 | batch 450/510 [I 2022-10-10 00:59:47] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-10 00:59:47] ...trap/engines/engine.py.326: process: 0.31614 | load: 0.00020 [I 2022-10-10 00:59:51] ...trap/engines/engine.py.321: eval: epoch 60 | batch 460/510 [I 2022-10-10 00:59:51] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:16 [I 2022-10-10 00:59:51] ...trap/engines/engine.py.326: process: 0.32721 | load: 0.00021 [I 2022-10-10 00:59:54] ...trap/engines/engine.py.321: eval: epoch 60 | batch 470/510 [I 2022-10-10 00:59:54] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-10 00:59:54] ...trap/engines/engine.py.326: process: 0.32466 | load: 0.00023 [I 2022-10-10 00:59:57] ...trap/engines/engine.py.321: eval: epoch 60 | batch 480/510 [I 2022-10-10 00:59:57] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-10 00:59:57] ...trap/engines/engine.py.326: process: 0.32517 | load: 0.00021 [I 2022-10-10 01:00:00] ...trap/engines/engine.py.321: eval: epoch 60 | batch 490/510 [I 2022-10-10 01:00:00] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-10 01:00:00] ...trap/engines/engine.py.326: process: 0.32770 | load: 0.00021 [I 2022-10-10 01:00:04] ...trap/engines/engine.py.321: eval: epoch 60 | batch 500/510 [I 2022-10-10 01:00:04] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-10 01:00:04] ...trap/engines/engine.py.326: process: 0.32454 | load: 0.00021 [I 2022-10-10 01:00:07] ...trap/engines/engine.py.321: eval: epoch 60 | batch 510/510 [I 2022-10-10 01:00:07] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 01:00:07] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00017 [S 2022-10-10 01:00:07] ...trap/engines/engine.py.132: eval_epoch.epoch: 60 [I 2022-10-10 01:00:07] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728100 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909500 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939500 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.716100 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905600 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937100 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012534 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006546 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005315 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013187 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010220 [S 2022-10-10 01:00:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008288 [I 2022-10-10 01:00:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 01:00:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 01:00:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 01:00:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 01:00:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 01:00:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 01:00:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 01:00:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 01:00:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 61 [I 2022-10-10 01:00:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 01:00:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 0/2304 [I 2022-10-10 01:00:57] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:14 [I 2022-10-10 01:00:57] ...trap/engines/engine.py.229: process: 3.02693 | load: 2.12197 [I 2022-10-10 01:00:57] ...trap/engines/engine.py.230: loss: 0.13270 [I 2022-10-10 01:00:57] ...optimizers/trijoint.py.151: total_norm: 12.973711 [I 2022-10-10 01:01:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 10/2304 [I 2022-10-10 01:01:06] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:31 [I 2022-10-10 01:01:06] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00670 [I 2022-10-10 01:01:06] ...trap/engines/engine.py.230: loss: 0.14768 [I 2022-10-10 01:01:06] ...optimizers/trijoint.py.151: total_norm: 12.775795 [I 2022-10-10 01:01:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 20/2304 [I 2022-10-10 01:01:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:04 [I 2022-10-10 01:01:14] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00362 [I 2022-10-10 01:01:14] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-10 01:01:14] ...optimizers/trijoint.py.151: total_norm: 17.000498 [I 2022-10-10 01:01:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 30/2304 [I 2022-10-10 01:01:23] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:05 [I 2022-10-10 01:01:23] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00464 [I 2022-10-10 01:01:23] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-10 01:01:23] ...optimizers/trijoint.py.151: total_norm: 18.225937 [I 2022-10-10 01:01:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 40/2304 [I 2022-10-10 01:01:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:50 [I 2022-10-10 01:01:31] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00421 [I 2022-10-10 01:01:31] ...trap/engines/engine.py.230: loss: 0.16476 [I 2022-10-10 01:01:31] ...optimizers/trijoint.py.151: total_norm: 14.729694 [I 2022-10-10 01:01:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 50/2304 [I 2022-10-10 01:01:40] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:46 [I 2022-10-10 01:01:40] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00322 [I 2022-10-10 01:01:40] ...trap/engines/engine.py.230: loss: 0.13877 [I 2022-10-10 01:01:40] ...optimizers/trijoint.py.151: total_norm: 17.629602 [I 2022-10-10 01:01:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 60/2304 [I 2022-10-10 01:01:48] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:44 [I 2022-10-10 01:01:48] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00347 [I 2022-10-10 01:01:48] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-10 01:01:48] ...optimizers/trijoint.py.151: total_norm: 31.655289 [I 2022-10-10 01:01:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 70/2304 [I 2022-10-10 01:01:57] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:25 [I 2022-10-10 01:01:57] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00382 [I 2022-10-10 01:01:57] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-10 01:01:57] ...optimizers/trijoint.py.151: total_norm: 12.225619 [I 2022-10-10 01:02:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 80/2304 [I 2022-10-10 01:02:05] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:20 [I 2022-10-10 01:02:05] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00345 [I 2022-10-10 01:02:05] ...trap/engines/engine.py.230: loss: 0.22511 [I 2022-10-10 01:02:05] ...optimizers/trijoint.py.151: total_norm: 13.607466 [I 2022-10-10 01:02:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 90/2304 [I 2022-10-10 01:02:14] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:21 [I 2022-10-10 01:02:14] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00338 [I 2022-10-10 01:02:14] ...trap/engines/engine.py.230: loss: 0.21903 [I 2022-10-10 01:02:14] ...optimizers/trijoint.py.151: total_norm: 12.481292 [I 2022-10-10 01:02:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 100/2304 [I 2022-10-10 01:02:22] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:49 [I 2022-10-10 01:02:22] ...trap/engines/engine.py.229: process: 0.83276 | load: 0.00332 [I 2022-10-10 01:02:22] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-10 01:02:22] ...optimizers/trijoint.py.151: total_norm: 14.381319 [I 2022-10-10 01:02:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 110/2304 [I 2022-10-10 01:02:30] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:57 [I 2022-10-10 01:02:30] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00323 [I 2022-10-10 01:02:30] ...trap/engines/engine.py.230: loss: 0.17424 [I 2022-10-10 01:02:30] ...optimizers/trijoint.py.151: total_norm: 24.133121 [I 2022-10-10 01:02:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 120/2304 [I 2022-10-10 01:02:39] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:45 [I 2022-10-10 01:02:39] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00319 [I 2022-10-10 01:02:39] ...trap/engines/engine.py.230: loss: 0.11257 [I 2022-10-10 01:02:39] ...optimizers/trijoint.py.151: total_norm: 9.177762 [I 2022-10-10 01:02:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 130/2304 [I 2022-10-10 01:02:47] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:43 [I 2022-10-10 01:02:47] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00300 [I 2022-10-10 01:02:47] ...trap/engines/engine.py.230: loss: 0.10877 [I 2022-10-10 01:02:47] ...optimizers/trijoint.py.151: total_norm: 18.064684 [I 2022-10-10 01:02:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 140/2304 [I 2022-10-10 01:02:56] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:26 [I 2022-10-10 01:02:56] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00337 [I 2022-10-10 01:02:56] ...trap/engines/engine.py.230: loss: 0.20814 [I 2022-10-10 01:02:56] ...optimizers/trijoint.py.151: total_norm: 14.524132 [I 2022-10-10 01:03:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 150/2304 [I 2022-10-10 01:03:04] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:17 [I 2022-10-10 01:03:04] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00319 [I 2022-10-10 01:03:04] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-10 01:03:04] ...optimizers/trijoint.py.151: total_norm: 16.578928 [I 2022-10-10 01:03:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 160/2304 [I 2022-10-10 01:03:13] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:00 [I 2022-10-10 01:03:13] ...trap/engines/engine.py.229: process: 0.83563 | load: 0.00340 [I 2022-10-10 01:03:13] ...trap/engines/engine.py.230: loss: 0.12669 [I 2022-10-10 01:03:13] ...optimizers/trijoint.py.151: total_norm: 22.945593 [I 2022-10-10 01:03:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 170/2304 [I 2022-10-10 01:03:21] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:57 [I 2022-10-10 01:03:21] ...trap/engines/engine.py.229: process: 0.83357 | load: 0.00326 [I 2022-10-10 01:03:21] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-10 01:03:21] ...optimizers/trijoint.py.151: total_norm: 7.750539 [I 2022-10-10 01:03:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 180/2304 [I 2022-10-10 01:03:30] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:57 [I 2022-10-10 01:03:30] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00325 [I 2022-10-10 01:03:30] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-10 01:03:30] ...optimizers/trijoint.py.151: total_norm: 18.701427 [I 2022-10-10 01:03:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 190/2304 [I 2022-10-10 01:03:38] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:59 [I 2022-10-10 01:03:38] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00312 [I 2022-10-10 01:03:38] ...trap/engines/engine.py.230: loss: 0.10000 [I 2022-10-10 01:03:38] ...optimizers/trijoint.py.151: total_norm: 13.090403 [I 2022-10-10 01:03:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 200/2304 [I 2022-10-10 01:03:47] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:43 [I 2022-10-10 01:03:47] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00326 [I 2022-10-10 01:03:47] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-10 01:03:47] ...optimizers/trijoint.py.151: total_norm: 34.472755 [I 2022-10-10 01:03:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 210/2304 [I 2022-10-10 01:03:55] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:42 [I 2022-10-10 01:03:55] ...trap/engines/engine.py.229: process: 0.85807 | load: 0.00338 [I 2022-10-10 01:03:55] ...trap/engines/engine.py.230: loss: 0.13596 [I 2022-10-10 01:03:55] ...optimizers/trijoint.py.151: total_norm: 14.064814 [I 2022-10-10 01:04:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 220/2304 [I 2022-10-10 01:04:04] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:26 [I 2022-10-10 01:04:04] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00306 [I 2022-10-10 01:04:04] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-10 01:04:04] ...optimizers/trijoint.py.151: total_norm: 18.443239 [I 2022-10-10 01:04:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 230/2304 [I 2022-10-10 01:04:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:31 [I 2022-10-10 01:04:12] ...trap/engines/engine.py.229: process: 0.86033 | load: 0.00326 [I 2022-10-10 01:04:12] ...trap/engines/engine.py.230: loss: 0.13655 [I 2022-10-10 01:04:12] ...optimizers/trijoint.py.151: total_norm: 22.599449 [I 2022-10-10 01:04:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 240/2304 [I 2022-10-10 01:04:21] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:22 [I 2022-10-10 01:04:21] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00312 [I 2022-10-10 01:04:21] ...trap/engines/engine.py.230: loss: 0.15839 [I 2022-10-10 01:04:21] ...optimizers/trijoint.py.151: total_norm: 10.831397 [I 2022-10-10 01:04:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 250/2304 [I 2022-10-10 01:04:29] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:16 [I 2022-10-10 01:04:29] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00327 [I 2022-10-10 01:04:29] ...trap/engines/engine.py.230: loss: 0.12797 [I 2022-10-10 01:04:29] ...optimizers/trijoint.py.151: total_norm: 15.458488 [I 2022-10-10 01:04:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 260/2304 [I 2022-10-10 01:04:38] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:29:05 [I 2022-10-10 01:04:38] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00327 [I 2022-10-10 01:04:38] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-10 01:04:38] ...optimizers/trijoint.py.151: total_norm: 18.078436 [I 2022-10-10 01:04:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 270/2304 [I 2022-10-10 01:04:47] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:29:08 [I 2022-10-10 01:04:47] ...trap/engines/engine.py.229: process: 0.86612 | load: 0.00310 [I 2022-10-10 01:04:47] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-10 01:04:47] ...optimizers/trijoint.py.151: total_norm: 13.336412 [I 2022-10-10 01:04:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 280/2304 [I 2022-10-10 01:04:55] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:52 [I 2022-10-10 01:04:55] ...trap/engines/engine.py.229: process: 0.86101 | load: 0.00329 [I 2022-10-10 01:04:55] ...trap/engines/engine.py.230: loss: 0.15802 [I 2022-10-10 01:04:55] ...optimizers/trijoint.py.151: total_norm: 18.279644 [I 2022-10-10 01:05:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 290/2304 [I 2022-10-10 01:05:04] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:41 [I 2022-10-10 01:05:04] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00326 [I 2022-10-10 01:05:04] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-10 01:05:04] ...optimizers/trijoint.py.151: total_norm: 16.744774 [I 2022-10-10 01:05:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 300/2304 [I 2022-10-10 01:05:12] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:40 [I 2022-10-10 01:05:12] ...trap/engines/engine.py.229: process: 0.85844 | load: 0.00328 [I 2022-10-10 01:05:12] ...trap/engines/engine.py.230: loss: 0.16558 [I 2022-10-10 01:05:12] ...optimizers/trijoint.py.151: total_norm: 15.343891 [I 2022-10-10 01:05:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 310/2304 [I 2022-10-10 01:05:21] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:24 [I 2022-10-10 01:05:21] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00314 [I 2022-10-10 01:05:21] ...trap/engines/engine.py.230: loss: 0.12166 [I 2022-10-10 01:05:21] ...optimizers/trijoint.py.151: total_norm: 51.376534 [I 2022-10-10 01:05:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 320/2304 [I 2022-10-10 01:05:29] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:19 [I 2022-10-10 01:05:29] ...trap/engines/engine.py.229: process: 0.86433 | load: 0.00315 [I 2022-10-10 01:05:29] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-10 01:05:29] ...optimizers/trijoint.py.151: total_norm: 15.683842 [I 2022-10-10 01:05:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 330/2304 [I 2022-10-10 01:05:38] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:11 [I 2022-10-10 01:05:38] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00312 [I 2022-10-10 01:05:38] ...trap/engines/engine.py.230: loss: 0.11621 [I 2022-10-10 01:05:38] ...optimizers/trijoint.py.151: total_norm: 11.570920 [I 2022-10-10 01:05:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 340/2304 [I 2022-10-10 01:05:47] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:57 [I 2022-10-10 01:05:47] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00331 [I 2022-10-10 01:05:47] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-10 01:05:47] ...optimizers/trijoint.py.151: total_norm: 19.944736 [I 2022-10-10 01:05:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 350/2304 [I 2022-10-10 01:05:55] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:42 [I 2022-10-10 01:05:55] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00313 [I 2022-10-10 01:05:55] ...trap/engines/engine.py.230: loss: 0.09972 [I 2022-10-10 01:05:55] ...optimizers/trijoint.py.151: total_norm: 11.893331 [I 2022-10-10 01:06:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 360/2304 [I 2022-10-10 01:06:04] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:26 [I 2022-10-10 01:06:04] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00318 [I 2022-10-10 01:06:04] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-10 01:06:04] ...optimizers/trijoint.py.151: total_norm: 12.300714 [I 2022-10-10 01:06:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 370/2304 [I 2022-10-10 01:06:12] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:39 [I 2022-10-10 01:06:12] ...trap/engines/engine.py.229: process: 0.87650 | load: 0.00314 [I 2022-10-10 01:06:12] ...trap/engines/engine.py.230: loss: 0.12972 [I 2022-10-10 01:06:12] ...optimizers/trijoint.py.151: total_norm: 21.616844 [I 2022-10-10 01:06:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 380/2304 [I 2022-10-10 01:06:21] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:20 [I 2022-10-10 01:06:21] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00312 [I 2022-10-10 01:06:21] ...trap/engines/engine.py.230: loss: 0.21985 [I 2022-10-10 01:06:21] ...optimizers/trijoint.py.151: total_norm: 13.013908 [I 2022-10-10 01:06:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 390/2304 [I 2022-10-10 01:06:29] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:12 [I 2022-10-10 01:06:29] ...trap/engines/engine.py.229: process: 0.86108 | load: 0.00316 [I 2022-10-10 01:06:29] ...trap/engines/engine.py.230: loss: 0.18094 [I 2022-10-10 01:06:29] ...optimizers/trijoint.py.151: total_norm: 17.101538 [I 2022-10-10 01:06:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 400/2304 [I 2022-10-10 01:06:38] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:54 [I 2022-10-10 01:06:38] ...trap/engines/engine.py.229: process: 0.83979 | load: 0.00314 [I 2022-10-10 01:06:38] ...trap/engines/engine.py.230: loss: 0.14626 [I 2022-10-10 01:06:38] ...optimizers/trijoint.py.151: total_norm: 26.622492 [I 2022-10-10 01:06:46] ...trap/engines/engine.py.224: train: epoch 61 | batch 410/2304 [I 2022-10-10 01:06:46] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:58 [I 2022-10-10 01:06:46] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00342 [I 2022-10-10 01:06:46] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-10 01:06:46] ...optimizers/trijoint.py.151: total_norm: 9.967947 [I 2022-10-10 01:06:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 420/2304 [I 2022-10-10 01:06:55] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:44 [I 2022-10-10 01:06:55] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00312 [I 2022-10-10 01:06:55] ...trap/engines/engine.py.230: loss: 0.08819 [I 2022-10-10 01:06:55] ...optimizers/trijoint.py.151: total_norm: 10.420723 [I 2022-10-10 01:07:03] ...trap/engines/engine.py.224: train: epoch 61 | batch 430/2304 [I 2022-10-10 01:07:03] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:32 [I 2022-10-10 01:07:03] ...trap/engines/engine.py.229: process: 0.84423 | load: 0.00311 [I 2022-10-10 01:07:03] ...trap/engines/engine.py.230: loss: 0.12190 [I 2022-10-10 01:07:03] ...optimizers/trijoint.py.151: total_norm: 14.824719 [I 2022-10-10 01:07:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 440/2304 [I 2022-10-10 01:07:12] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:33 [I 2022-10-10 01:07:12] ...trap/engines/engine.py.229: process: 0.84242 | load: 0.00326 [I 2022-10-10 01:07:12] ...trap/engines/engine.py.230: loss: 0.16464 [I 2022-10-10 01:07:12] ...optimizers/trijoint.py.151: total_norm: 18.015383 [I 2022-10-10 01:07:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 450/2304 [I 2022-10-10 01:07:21] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:29 [I 2022-10-10 01:07:21] ...trap/engines/engine.py.229: process: 0.86523 | load: 0.00312 [I 2022-10-10 01:07:21] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-10 01:07:21] ...optimizers/trijoint.py.151: total_norm: 12.214133 [I 2022-10-10 01:07:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 460/2304 [I 2022-10-10 01:07:29] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:18 [I 2022-10-10 01:07:29] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00311 [I 2022-10-10 01:07:29] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-10 01:07:29] ...optimizers/trijoint.py.151: total_norm: 13.999979 [I 2022-10-10 01:07:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 470/2304 [I 2022-10-10 01:07:38] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:10 [I 2022-10-10 01:07:38] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00310 [I 2022-10-10 01:07:38] ...trap/engines/engine.py.230: loss: 0.12114 [I 2022-10-10 01:07:38] ...optimizers/trijoint.py.151: total_norm: 24.203987 [I 2022-10-10 01:07:46] ...trap/engines/engine.py.224: train: epoch 61 | batch 480/2304 [I 2022-10-10 01:07:46] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:26:07 [I 2022-10-10 01:07:46] ...trap/engines/engine.py.229: process: 0.86109 | load: 0.00322 [I 2022-10-10 01:07:46] ...trap/engines/engine.py.230: loss: 0.13133 [I 2022-10-10 01:07:46] ...optimizers/trijoint.py.151: total_norm: 15.475306 [I 2022-10-10 01:07:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 490/2304 [I 2022-10-10 01:07:55] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:47 [I 2022-10-10 01:07:55] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00306 [I 2022-10-10 01:07:55] ...trap/engines/engine.py.230: loss: 0.19632 [I 2022-10-10 01:07:55] ...optimizers/trijoint.py.151: total_norm: 26.409004 [I 2022-10-10 01:08:03] ...trap/engines/engine.py.224: train: epoch 61 | batch 500/2304 [I 2022-10-10 01:08:03] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:23 [I 2022-10-10 01:08:03] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00302 [I 2022-10-10 01:08:03] ...trap/engines/engine.py.230: loss: 0.19032 [I 2022-10-10 01:08:03] ...optimizers/trijoint.py.151: total_norm: 18.669085 [I 2022-10-10 01:08:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 510/2304 [I 2022-10-10 01:08:12] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:05 [I 2022-10-10 01:08:12] ...trap/engines/engine.py.229: process: 0.82250 | load: 0.00294 [I 2022-10-10 01:08:12] ...trap/engines/engine.py.230: loss: 0.12785 [I 2022-10-10 01:08:12] ...optimizers/trijoint.py.151: total_norm: 15.559911 [I 2022-10-10 01:08:20] ...trap/engines/engine.py.224: train: epoch 61 | batch 520/2304 [I 2022-10-10 01:08:20] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:08 [I 2022-10-10 01:08:20] ...trap/engines/engine.py.229: process: 0.85671 | load: 0.00308 [I 2022-10-10 01:08:20] ...trap/engines/engine.py.230: loss: 0.16427 [I 2022-10-10 01:08:20] ...optimizers/trijoint.py.151: total_norm: 15.445918 [I 2022-10-10 01:08:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 530/2304 [I 2022-10-10 01:08:29] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:52 [I 2022-10-10 01:08:29] ...trap/engines/engine.py.229: process: 0.83646 | load: 0.00296 [I 2022-10-10 01:08:29] ...trap/engines/engine.py.230: loss: 0.14539 [I 2022-10-10 01:08:29] ...optimizers/trijoint.py.151: total_norm: 15.915889 [I 2022-10-10 01:08:37] ...trap/engines/engine.py.224: train: epoch 61 | batch 540/2304 [I 2022-10-10 01:08:37] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:49 [I 2022-10-10 01:08:37] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00297 [I 2022-10-10 01:08:37] ...trap/engines/engine.py.230: loss: 0.23303 [I 2022-10-10 01:08:37] ...optimizers/trijoint.py.151: total_norm: 12.613595 [I 2022-10-10 01:08:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 550/2304 [I 2022-10-10 01:08:45] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:43 [I 2022-10-10 01:08:45] ...trap/engines/engine.py.229: process: 0.86022 | load: 0.00299 [I 2022-10-10 01:08:45] ...trap/engines/engine.py.230: loss: 0.15456 [I 2022-10-10 01:08:45] ...optimizers/trijoint.py.151: total_norm: 14.167179 [I 2022-10-10 01:08:54] ...trap/engines/engine.py.224: train: epoch 61 | batch 560/2304 [I 2022-10-10 01:08:54] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:43 [I 2022-10-10 01:08:54] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00305 [I 2022-10-10 01:08:54] ...trap/engines/engine.py.230: loss: 0.17378 [I 2022-10-10 01:08:54] ...optimizers/trijoint.py.151: total_norm: 13.584435 [I 2022-10-10 01:09:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 570/2304 [I 2022-10-10 01:09:02] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:26 [I 2022-10-10 01:09:02] ...trap/engines/engine.py.229: process: 0.83457 | load: 0.00298 [I 2022-10-10 01:09:02] ...trap/engines/engine.py.230: loss: 0.11294 [I 2022-10-10 01:09:02] ...optimizers/trijoint.py.151: total_norm: 18.571594 [I 2022-10-10 01:09:11] ...trap/engines/engine.py.224: train: epoch 61 | batch 580/2304 [I 2022-10-10 01:09:11] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:22 [I 2022-10-10 01:09:11] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00298 [I 2022-10-10 01:09:11] ...trap/engines/engine.py.230: loss: 0.13100 [I 2022-10-10 01:09:11] ...optimizers/trijoint.py.151: total_norm: 11.180460 [I 2022-10-10 01:09:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 590/2304 [I 2022-10-10 01:09:19] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:07 [I 2022-10-10 01:09:19] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00299 [I 2022-10-10 01:09:19] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-10 01:09:19] ...optimizers/trijoint.py.151: total_norm: 8.757255 [I 2022-10-10 01:09:28] ...trap/engines/engine.py.224: train: epoch 61 | batch 600/2304 [I 2022-10-10 01:09:28] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:09 [I 2022-10-10 01:09:28] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00304 [I 2022-10-10 01:09:28] ...trap/engines/engine.py.230: loss: 0.14493 [I 2022-10-10 01:09:28] ...optimizers/trijoint.py.151: total_norm: 12.953087 [I 2022-10-10 01:09:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 610/2304 [I 2022-10-10 01:09:36] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:23:57 [I 2022-10-10 01:09:36] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00299 [I 2022-10-10 01:09:36] ...trap/engines/engine.py.230: loss: 0.12633 [I 2022-10-10 01:09:36] ...optimizers/trijoint.py.151: total_norm: 17.559156 [I 2022-10-10 01:09:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 620/2304 [I 2022-10-10 01:09:45] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:45 [I 2022-10-10 01:09:45] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00298 [I 2022-10-10 01:09:45] ...trap/engines/engine.py.230: loss: 0.23128 [I 2022-10-10 01:09:45] ...optimizers/trijoint.py.151: total_norm: 23.908035 [I 2022-10-10 01:09:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 630/2304 [I 2022-10-10 01:09:53] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:37 [I 2022-10-10 01:09:53] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00298 [I 2022-10-10 01:09:53] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-10 01:09:53] ...optimizers/trijoint.py.151: total_norm: 14.458490 [I 2022-10-10 01:10:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 640/2304 [I 2022-10-10 01:10:02] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:31 [I 2022-10-10 01:10:02] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00300 [I 2022-10-10 01:10:02] ...trap/engines/engine.py.230: loss: 0.15663 [I 2022-10-10 01:10:02] ...optimizers/trijoint.py.151: total_norm: 21.628502 [I 2022-10-10 01:10:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 650/2304 [I 2022-10-10 01:10:10] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:17 [I 2022-10-10 01:10:10] ...trap/engines/engine.py.229: process: 0.83657 | load: 0.00299 [I 2022-10-10 01:10:10] ...trap/engines/engine.py.230: loss: 0.11471 [I 2022-10-10 01:10:10] ...optimizers/trijoint.py.151: total_norm: 6.913403 [I 2022-10-10 01:10:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 660/2304 [I 2022-10-10 01:10:19] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:15 [I 2022-10-10 01:10:19] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00302 [I 2022-10-10 01:10:19] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-10 01:10:19] ...optimizers/trijoint.py.151: total_norm: 17.493311 [I 2022-10-10 01:10:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 670/2304 [I 2022-10-10 01:10:27] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:06 [I 2022-10-10 01:10:27] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00339 [I 2022-10-10 01:10:27] ...trap/engines/engine.py.230: loss: 0.13859 [I 2022-10-10 01:10:27] ...optimizers/trijoint.py.151: total_norm: 12.500727 [I 2022-10-10 01:10:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 680/2304 [I 2022-10-10 01:10:36] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:22:53 [I 2022-10-10 01:10:36] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00297 [I 2022-10-10 01:10:36] ...trap/engines/engine.py.230: loss: 0.12468 [I 2022-10-10 01:10:36] ...optimizers/trijoint.py.151: total_norm: 18.781307 [I 2022-10-10 01:10:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 690/2304 [I 2022-10-10 01:10:44] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:48 [I 2022-10-10 01:10:44] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00300 [I 2022-10-10 01:10:44] ...trap/engines/engine.py.230: loss: 0.19793 [I 2022-10-10 01:10:44] ...optimizers/trijoint.py.151: total_norm: 22.603329 [I 2022-10-10 01:10:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 700/2304 [I 2022-10-10 01:10:53] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:33 [I 2022-10-10 01:10:53] ...trap/engines/engine.py.229: process: 0.81873 | load: 0.00303 [I 2022-10-10 01:10:53] ...trap/engines/engine.py.230: loss: 0.10601 [I 2022-10-10 01:10:53] ...optimizers/trijoint.py.151: total_norm: 7.029199 [I 2022-10-10 01:11:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 710/2304 [I 2022-10-10 01:11:01] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:34 [I 2022-10-10 01:11:01] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00313 [I 2022-10-10 01:11:01] ...trap/engines/engine.py.230: loss: 0.10895 [I 2022-10-10 01:11:01] ...optimizers/trijoint.py.151: total_norm: 11.560383 [I 2022-10-10 01:11:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 720/2304 [I 2022-10-10 01:11:10] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:32 [I 2022-10-10 01:11:10] ...trap/engines/engine.py.229: process: 0.85776 | load: 0.00315 [I 2022-10-10 01:11:10] ...trap/engines/engine.py.230: loss: 0.13406 [I 2022-10-10 01:11:10] ...optimizers/trijoint.py.151: total_norm: 14.338157 [I 2022-10-10 01:11:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 730/2304 [I 2022-10-10 01:11:19] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:33 [I 2022-10-10 01:11:19] ...trap/engines/engine.py.229: process: 0.87030 | load: 0.00311 [I 2022-10-10 01:11:19] ...trap/engines/engine.py.230: loss: 0.24022 [I 2022-10-10 01:11:19] ...optimizers/trijoint.py.151: total_norm: 25.187716 [I 2022-10-10 01:11:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 740/2304 [I 2022-10-10 01:11:27] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:14 [I 2022-10-10 01:11:27] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00313 [I 2022-10-10 01:11:27] ...trap/engines/engine.py.230: loss: 0.11347 [I 2022-10-10 01:11:27] ...optimizers/trijoint.py.151: total_norm: 13.648335 [I 2022-10-10 01:11:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 750/2304 [I 2022-10-10 01:11:36] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:58 [I 2022-10-10 01:11:36] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00310 [I 2022-10-10 01:11:36] ...trap/engines/engine.py.230: loss: 0.15484 [I 2022-10-10 01:11:36] ...optimizers/trijoint.py.151: total_norm: 16.651840 [I 2022-10-10 01:11:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 760/2304 [I 2022-10-10 01:11:44] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:22:00 [I 2022-10-10 01:11:44] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00318 [I 2022-10-10 01:11:44] ...trap/engines/engine.py.230: loss: 0.25026 [I 2022-10-10 01:11:44] ...optimizers/trijoint.py.151: total_norm: 35.686432 [I 2022-10-10 01:11:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 770/2304 [I 2022-10-10 01:11:53] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:52 [I 2022-10-10 01:11:53] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00321 [I 2022-10-10 01:11:53] ...trap/engines/engine.py.230: loss: 0.11583 [I 2022-10-10 01:11:53] ...optimizers/trijoint.py.151: total_norm: 10.002855 [I 2022-10-10 01:12:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 780/2304 [I 2022-10-10 01:12:01] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:40 [I 2022-10-10 01:12:01] ...trap/engines/engine.py.229: process: 0.84303 | load: 0.00312 [I 2022-10-10 01:12:01] ...trap/engines/engine.py.230: loss: 0.15563 [I 2022-10-10 01:12:01] ...optimizers/trijoint.py.151: total_norm: 32.556511 [I 2022-10-10 01:12:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 790/2304 [I 2022-10-10 01:12:10] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:35 [I 2022-10-10 01:12:10] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00311 [I 2022-10-10 01:12:10] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-10 01:12:10] ...optimizers/trijoint.py.151: total_norm: 25.054848 [I 2022-10-10 01:12:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 800/2304 [I 2022-10-10 01:12:18] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:20 [I 2022-10-10 01:12:18] ...trap/engines/engine.py.229: process: 0.84280 | load: 0.00315 [I 2022-10-10 01:12:18] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-10 01:12:18] ...optimizers/trijoint.py.151: total_norm: 13.929332 [I 2022-10-10 01:12:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 810/2304 [I 2022-10-10 01:12:27] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:18 [I 2022-10-10 01:12:27] ...trap/engines/engine.py.229: process: 0.87081 | load: 0.00312 [I 2022-10-10 01:12:27] ...trap/engines/engine.py.230: loss: 0.14366 [I 2022-10-10 01:12:27] ...optimizers/trijoint.py.151: total_norm: 13.819475 [I 2022-10-10 01:12:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 820/2304 [I 2022-10-10 01:12:36] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:12 [I 2022-10-10 01:12:36] ...trap/engines/engine.py.229: process: 0.86628 | load: 0.00311 [I 2022-10-10 01:12:36] ...trap/engines/engine.py.230: loss: 0.13683 [I 2022-10-10 01:12:36] ...optimizers/trijoint.py.151: total_norm: 19.594439 [I 2022-10-10 01:12:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 830/2304 [I 2022-10-10 01:12:44] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:53 [I 2022-10-10 01:12:44] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00310 [I 2022-10-10 01:12:44] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-10 01:12:44] ...optimizers/trijoint.py.151: total_norm: 12.723476 [I 2022-10-10 01:12:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 840/2304 [I 2022-10-10 01:12:53] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:52 [I 2022-10-10 01:12:53] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00313 [I 2022-10-10 01:12:53] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-10 01:12:53] ...optimizers/trijoint.py.151: total_norm: 14.730489 [I 2022-10-10 01:13:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 850/2304 [I 2022-10-10 01:13:01] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:44 [I 2022-10-10 01:13:01] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00313 [I 2022-10-10 01:13:01] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-10 01:13:01] ...optimizers/trijoint.py.151: total_norm: 14.901691 [I 2022-10-10 01:13:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 860/2304 [I 2022-10-10 01:13:10] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:39 [I 2022-10-10 01:13:10] ...trap/engines/engine.py.229: process: 0.87117 | load: 0.00316 [I 2022-10-10 01:13:10] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-10 01:13:10] ...optimizers/trijoint.py.151: total_norm: 12.500381 [I 2022-10-10 01:13:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 870/2304 [I 2022-10-10 01:13:18] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:23 [I 2022-10-10 01:13:18] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00311 [I 2022-10-10 01:13:18] ...trap/engines/engine.py.230: loss: 0.14604 [I 2022-10-10 01:13:18] ...optimizers/trijoint.py.151: total_norm: 10.539243 [I 2022-10-10 01:13:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 880/2304 [I 2022-10-10 01:13:27] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:15 [I 2022-10-10 01:13:27] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00314 [I 2022-10-10 01:13:27] ...trap/engines/engine.py.230: loss: 0.16596 [I 2022-10-10 01:13:27] ...optimizers/trijoint.py.151: total_norm: 13.471108 [I 2022-10-10 01:13:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 890/2304 [I 2022-10-10 01:13:35] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:11 [I 2022-10-10 01:13:35] ...trap/engines/engine.py.229: process: 0.85592 | load: 0.00315 [I 2022-10-10 01:13:35] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-10 01:13:35] ...optimizers/trijoint.py.151: total_norm: 10.509565 [I 2022-10-10 01:13:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 900/2304 [I 2022-10-10 01:13:44] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:56 [I 2022-10-10 01:13:44] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00311 [I 2022-10-10 01:13:44] ...trap/engines/engine.py.230: loss: 0.11473 [I 2022-10-10 01:13:44] ...optimizers/trijoint.py.151: total_norm: 21.917616 [I 2022-10-10 01:13:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 910/2304 [I 2022-10-10 01:13:52] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:49 [I 2022-10-10 01:13:52] ...trap/engines/engine.py.229: process: 0.84805 | load: 0.00312 [I 2022-10-10 01:13:52] ...trap/engines/engine.py.230: loss: 0.12357 [I 2022-10-10 01:13:52] ...optimizers/trijoint.py.151: total_norm: 6.704275 [I 2022-10-10 01:14:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 920/2304 [I 2022-10-10 01:14:01] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:37 [I 2022-10-10 01:14:01] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00309 [I 2022-10-10 01:14:01] ...trap/engines/engine.py.230: loss: 0.19941 [I 2022-10-10 01:14:01] ...optimizers/trijoint.py.151: total_norm: 17.166336 [I 2022-10-10 01:14:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 930/2304 [I 2022-10-10 01:14:10] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:34 [I 2022-10-10 01:14:10] ...trap/engines/engine.py.229: process: 0.86382 | load: 0.00311 [I 2022-10-10 01:14:10] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-10 01:14:10] ...optimizers/trijoint.py.151: total_norm: 11.483963 [I 2022-10-10 01:14:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 940/2304 [I 2022-10-10 01:14:18] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:28 [I 2022-10-10 01:14:18] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00312 [I 2022-10-10 01:14:18] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-10 01:14:18] ...optimizers/trijoint.py.151: total_norm: 13.261146 [I 2022-10-10 01:14:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 950/2304 [I 2022-10-10 01:14:27] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:18 [I 2022-10-10 01:14:27] ...trap/engines/engine.py.229: process: 0.85554 | load: 0.00313 [I 2022-10-10 01:14:27] ...trap/engines/engine.py.230: loss: 0.11543 [I 2022-10-10 01:14:27] ...optimizers/trijoint.py.151: total_norm: 11.613225 [I 2022-10-10 01:14:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 960/2304 [I 2022-10-10 01:14:35] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:13 [I 2022-10-10 01:14:35] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00310 [I 2022-10-10 01:14:35] ...trap/engines/engine.py.230: loss: 0.11730 [I 2022-10-10 01:14:35] ...optimizers/trijoint.py.151: total_norm: 23.286818 [I 2022-10-10 01:14:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 970/2304 [I 2022-10-10 01:14:44] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:58 [I 2022-10-10 01:14:44] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00312 [I 2022-10-10 01:14:44] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-10 01:14:44] ...optimizers/trijoint.py.151: total_norm: 12.013155 [I 2022-10-10 01:14:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 980/2304 [I 2022-10-10 01:14:52] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:44 [I 2022-10-10 01:14:52] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00313 [I 2022-10-10 01:14:52] ...trap/engines/engine.py.230: loss: 0.18199 [I 2022-10-10 01:14:52] ...optimizers/trijoint.py.151: total_norm: 13.219626 [I 2022-10-10 01:15:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 990/2304 [I 2022-10-10 01:15:01] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:37 [I 2022-10-10 01:15:01] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00299 [I 2022-10-10 01:15:01] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-10 01:15:01] ...optimizers/trijoint.py.151: total_norm: 17.674002 [I 2022-10-10 01:15:09] ...trap/engines/engine.py.224: train: epoch 61 | batch 1000/2304 [I 2022-10-10 01:15:09] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:20 [I 2022-10-10 01:15:09] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00295 [I 2022-10-10 01:15:09] ...trap/engines/engine.py.230: loss: 0.13564 [I 2022-10-10 01:15:09] ...optimizers/trijoint.py.151: total_norm: 15.970713 [I 2022-10-10 01:15:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 1010/2304 [I 2022-10-10 01:15:18] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:30 [I 2022-10-10 01:15:18] ...trap/engines/engine.py.229: process: 0.88936 | load: 0.00316 [I 2022-10-10 01:15:18] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-10 01:15:18] ...optimizers/trijoint.py.151: total_norm: 10.838391 [I 2022-10-10 01:15:26] ...trap/engines/engine.py.224: train: epoch 61 | batch 1020/2304 [I 2022-10-10 01:15:26] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:03 [I 2022-10-10 01:15:26] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00292 [I 2022-10-10 01:15:26] ...trap/engines/engine.py.230: loss: 0.19497 [I 2022-10-10 01:15:26] ...optimizers/trijoint.py.151: total_norm: 12.135902 [I 2022-10-10 01:15:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 1030/2304 [I 2022-10-10 01:15:35] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:58 [I 2022-10-10 01:15:35] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00297 [I 2022-10-10 01:15:35] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-10 01:15:35] ...optimizers/trijoint.py.151: total_norm: 12.424643 [I 2022-10-10 01:15:43] ...trap/engines/engine.py.224: train: epoch 61 | batch 1040/2304 [I 2022-10-10 01:15:43] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:48 [I 2022-10-10 01:15:43] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00316 [I 2022-10-10 01:15:43] ...trap/engines/engine.py.230: loss: 0.17084 [I 2022-10-10 01:15:43] ...optimizers/trijoint.py.151: total_norm: 15.524192 [I 2022-10-10 01:15:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 1050/2304 [I 2022-10-10 01:15:52] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:43 [I 2022-10-10 01:15:52] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00295 [I 2022-10-10 01:15:52] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-10 01:15:52] ...optimizers/trijoint.py.151: total_norm: 19.778687 [I 2022-10-10 01:16:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 1060/2304 [I 2022-10-10 01:16:00] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:30 [I 2022-10-10 01:16:00] ...trap/engines/engine.py.229: process: 0.82881 | load: 0.00299 [I 2022-10-10 01:16:00] ...trap/engines/engine.py.230: loss: 0.24542 [I 2022-10-10 01:16:00] ...optimizers/trijoint.py.151: total_norm: 9.904330 [I 2022-10-10 01:16:09] ...trap/engines/engine.py.224: train: epoch 61 | batch 1070/2304 [I 2022-10-10 01:16:09] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:30 [I 2022-10-10 01:16:09] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00302 [I 2022-10-10 01:16:09] ...trap/engines/engine.py.230: loss: 0.15813 [I 2022-10-10 01:16:09] ...optimizers/trijoint.py.151: total_norm: 15.594382 [I 2022-10-10 01:16:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 1080/2304 [I 2022-10-10 01:16:17] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:16 [I 2022-10-10 01:16:17] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00297 [I 2022-10-10 01:16:17] ...trap/engines/engine.py.230: loss: 0.13282 [I 2022-10-10 01:16:17] ...optimizers/trijoint.py.151: total_norm: 13.280296 [I 2022-10-10 01:16:26] ...trap/engines/engine.py.224: train: epoch 61 | batch 1090/2304 [I 2022-10-10 01:16:26] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:08 [I 2022-10-10 01:16:26] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00292 [I 2022-10-10 01:16:26] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-10 01:16:26] ...optimizers/trijoint.py.151: total_norm: 17.866301 [I 2022-10-10 01:16:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 1100/2304 [I 2022-10-10 01:16:34] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:57 [I 2022-10-10 01:16:34] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00300 [I 2022-10-10 01:16:34] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-10 01:16:34] ...optimizers/trijoint.py.151: total_norm: 17.265413 [I 2022-10-10 01:16:43] ...trap/engines/engine.py.224: train: epoch 61 | batch 1110/2304 [I 2022-10-10 01:16:43] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:51 [I 2022-10-10 01:16:43] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00302 [I 2022-10-10 01:16:43] ...trap/engines/engine.py.230: loss: 0.22029 [I 2022-10-10 01:16:43] ...optimizers/trijoint.py.151: total_norm: 53.088627 [I 2022-10-10 01:16:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 1120/2304 [I 2022-10-10 01:16:51] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:41 [I 2022-10-10 01:16:51] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00300 [I 2022-10-10 01:16:51] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-10 01:16:51] ...optimizers/trijoint.py.151: total_norm: 11.491572 [I 2022-10-10 01:17:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 1130/2304 [I 2022-10-10 01:17:00] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:34 [I 2022-10-10 01:17:00] ...trap/engines/engine.py.229: process: 0.86154 | load: 0.00300 [I 2022-10-10 01:17:00] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-10 01:17:00] ...optimizers/trijoint.py.151: total_norm: 10.724297 [I 2022-10-10 01:17:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 1140/2304 [I 2022-10-10 01:17:08] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:30 [I 2022-10-10 01:17:08] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00299 [I 2022-10-10 01:17:08] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-10 01:17:08] ...optimizers/trijoint.py.151: total_norm: 76.868141 [I 2022-10-10 01:17:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 1150/2304 [I 2022-10-10 01:17:17] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:20 [I 2022-10-10 01:17:17] ...trap/engines/engine.py.229: process: 0.83510 | load: 0.00302 [I 2022-10-10 01:17:17] ...trap/engines/engine.py.230: loss: 0.13174 [I 2022-10-10 01:17:17] ...optimizers/trijoint.py.151: total_norm: 14.892040 [I 2022-10-10 01:17:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1160/2304 [I 2022-10-10 01:17:25] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:17 [I 2022-10-10 01:17:25] ...trap/engines/engine.py.229: process: 0.88124 | load: 0.00301 [I 2022-10-10 01:17:25] ...trap/engines/engine.py.230: loss: 0.16837 [I 2022-10-10 01:17:25] ...optimizers/trijoint.py.151: total_norm: 15.566933 [I 2022-10-10 01:17:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 1170/2304 [I 2022-10-10 01:17:34] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:15:56 [I 2022-10-10 01:17:34] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00302 [I 2022-10-10 01:17:34] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-10 01:17:34] ...optimizers/trijoint.py.151: total_norm: 14.794779 [I 2022-10-10 01:17:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1180/2304 [I 2022-10-10 01:17:42] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:55 [I 2022-10-10 01:17:42] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00300 [I 2022-10-10 01:17:42] ...trap/engines/engine.py.230: loss: 0.10405 [I 2022-10-10 01:17:42] ...optimizers/trijoint.py.151: total_norm: 8.461038 [I 2022-10-10 01:17:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 1190/2304 [I 2022-10-10 01:17:51] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:45 [I 2022-10-10 01:17:51] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00311 [I 2022-10-10 01:17:51] ...trap/engines/engine.py.230: loss: 0.23359 [I 2022-10-10 01:17:51] ...optimizers/trijoint.py.151: total_norm: 31.213663 [I 2022-10-10 01:17:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1200/2304 [I 2022-10-10 01:17:59] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:45 [I 2022-10-10 01:17:59] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00316 [I 2022-10-10 01:17:59] ...trap/engines/engine.py.230: loss: 0.14558 [I 2022-10-10 01:17:59] ...optimizers/trijoint.py.151: total_norm: 8.535688 [I 2022-10-10 01:18:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 1210/2304 [I 2022-10-10 01:18:08] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:31 [I 2022-10-10 01:18:08] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00313 [I 2022-10-10 01:18:08] ...trap/engines/engine.py.230: loss: 0.15855 [I 2022-10-10 01:18:08] ...optimizers/trijoint.py.151: total_norm: 22.265850 [I 2022-10-10 01:18:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1220/2304 [I 2022-10-10 01:18:16] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:30 [I 2022-10-10 01:18:16] ...trap/engines/engine.py.229: process: 0.88083 | load: 0.00312 [I 2022-10-10 01:18:16] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-10 01:18:16] ...optimizers/trijoint.py.151: total_norm: 16.809765 [I 2022-10-10 01:18:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1230/2304 [I 2022-10-10 01:18:25] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:14 [I 2022-10-10 01:18:25] ...trap/engines/engine.py.229: process: 0.85726 | load: 0.00312 [I 2022-10-10 01:18:25] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-10 01:18:25] ...optimizers/trijoint.py.151: total_norm: 13.182011 [I 2022-10-10 01:18:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1240/2304 [I 2022-10-10 01:18:33] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:05 [I 2022-10-10 01:18:33] ...trap/engines/engine.py.229: process: 0.84795 | load: 0.00311 [I 2022-10-10 01:18:33] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-10 01:18:33] ...optimizers/trijoint.py.151: total_norm: 15.930648 [I 2022-10-10 01:18:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1250/2304 [I 2022-10-10 01:18:42] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:55 [I 2022-10-10 01:18:42] ...trap/engines/engine.py.229: process: 0.84760 | load: 0.00313 [I 2022-10-10 01:18:42] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-10 01:18:42] ...optimizers/trijoint.py.151: total_norm: 14.240402 [I 2022-10-10 01:18:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 1260/2304 [I 2022-10-10 01:18:51] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:52 [I 2022-10-10 01:18:51] ...trap/engines/engine.py.229: process: 0.85876 | load: 0.00313 [I 2022-10-10 01:18:51] ...trap/engines/engine.py.230: loss: 0.14355 [I 2022-10-10 01:18:51] ...optimizers/trijoint.py.151: total_norm: 12.741790 [I 2022-10-10 01:18:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1270/2304 [I 2022-10-10 01:18:59] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:45 [I 2022-10-10 01:18:59] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00310 [I 2022-10-10 01:18:59] ...trap/engines/engine.py.230: loss: 0.24092 [I 2022-10-10 01:18:59] ...optimizers/trijoint.py.151: total_norm: 30.258684 [I 2022-10-10 01:19:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 1280/2304 [I 2022-10-10 01:19:08] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:34 [I 2022-10-10 01:19:08] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00313 [I 2022-10-10 01:19:08] ...trap/engines/engine.py.230: loss: 0.09149 [I 2022-10-10 01:19:08] ...optimizers/trijoint.py.151: total_norm: 8.005424 [I 2022-10-10 01:19:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1290/2304 [I 2022-10-10 01:19:16] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:27 [I 2022-10-10 01:19:16] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00313 [I 2022-10-10 01:19:16] ...trap/engines/engine.py.230: loss: 0.14442 [I 2022-10-10 01:19:16] ...optimizers/trijoint.py.151: total_norm: 26.415071 [I 2022-10-10 01:19:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1300/2304 [I 2022-10-10 01:19:25] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:21 [I 2022-10-10 01:19:25] ...trap/engines/engine.py.229: process: 0.86542 | load: 0.00310 [I 2022-10-10 01:19:25] ...trap/engines/engine.py.230: loss: 0.11375 [I 2022-10-10 01:19:25] ...optimizers/trijoint.py.151: total_norm: 10.695870 [I 2022-10-10 01:19:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1310/2304 [I 2022-10-10 01:19:33] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:07 [I 2022-10-10 01:19:33] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00313 [I 2022-10-10 01:19:33] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-10 01:19:33] ...optimizers/trijoint.py.151: total_norm: 19.321226 [I 2022-10-10 01:19:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1320/2304 [I 2022-10-10 01:19:42] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:59 [I 2022-10-10 01:19:42] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00311 [I 2022-10-10 01:19:42] ...trap/engines/engine.py.230: loss: 0.13235 [I 2022-10-10 01:19:42] ...optimizers/trijoint.py.151: total_norm: 15.785642 [I 2022-10-10 01:19:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 1330/2304 [I 2022-10-10 01:19:50] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:49 [I 2022-10-10 01:19:50] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00311 [I 2022-10-10 01:19:50] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-10 01:19:50] ...optimizers/trijoint.py.151: total_norm: 9.580159 [I 2022-10-10 01:19:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1340/2304 [I 2022-10-10 01:19:59] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:37 [I 2022-10-10 01:19:59] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00309 [I 2022-10-10 01:19:59] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-10 01:19:59] ...optimizers/trijoint.py.151: total_norm: 16.540600 [I 2022-10-10 01:20:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 1350/2304 [I 2022-10-10 01:20:08] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:36 [I 2022-10-10 01:20:08] ...trap/engines/engine.py.229: process: 0.85880 | load: 0.00312 [I 2022-10-10 01:20:08] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-10 01:20:08] ...optimizers/trijoint.py.151: total_norm: 21.910017 [I 2022-10-10 01:20:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1360/2304 [I 2022-10-10 01:20:16] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:24 [I 2022-10-10 01:20:16] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00328 [I 2022-10-10 01:20:16] ...trap/engines/engine.py.230: loss: 0.15914 [I 2022-10-10 01:20:16] ...optimizers/trijoint.py.151: total_norm: 15.690208 [I 2022-10-10 01:20:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1370/2304 [I 2022-10-10 01:20:25] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:17 [I 2022-10-10 01:20:25] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00312 [I 2022-10-10 01:20:25] ...trap/engines/engine.py.230: loss: 0.20070 [I 2022-10-10 01:20:25] ...optimizers/trijoint.py.151: total_norm: 32.434517 [I 2022-10-10 01:20:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1380/2304 [I 2022-10-10 01:20:33] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:12 [I 2022-10-10 01:20:33] ...trap/engines/engine.py.229: process: 0.87036 | load: 0.00311 [I 2022-10-10 01:20:33] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-10 01:20:33] ...optimizers/trijoint.py.151: total_norm: 11.307018 [I 2022-10-10 01:20:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1390/2304 [I 2022-10-10 01:20:42] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:57 [I 2022-10-10 01:20:42] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00315 [I 2022-10-10 01:20:42] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-10 01:20:42] ...optimizers/trijoint.py.151: total_norm: 12.985117 [I 2022-10-10 01:20:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 1400/2304 [I 2022-10-10 01:20:50] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:49 [I 2022-10-10 01:20:50] ...trap/engines/engine.py.229: process: 0.85960 | load: 0.00312 [I 2022-10-10 01:20:50] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-10 01:20:50] ...optimizers/trijoint.py.151: total_norm: 12.190963 [I 2022-10-10 01:20:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1410/2304 [I 2022-10-10 01:20:59] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:44 [I 2022-10-10 01:20:59] ...trap/engines/engine.py.229: process: 0.86036 | load: 0.00312 [I 2022-10-10 01:20:59] ...trap/engines/engine.py.230: loss: 0.18570 [I 2022-10-10 01:20:59] ...optimizers/trijoint.py.151: total_norm: 22.877506 [I 2022-10-10 01:21:07] ...trap/engines/engine.py.224: train: epoch 61 | batch 1420/2304 [I 2022-10-10 01:21:07] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:32 [I 2022-10-10 01:21:07] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00314 [I 2022-10-10 01:21:07] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-10 01:21:07] ...optimizers/trijoint.py.151: total_norm: 18.741104 [I 2022-10-10 01:21:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1430/2304 [I 2022-10-10 01:21:16] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:24 [I 2022-10-10 01:21:16] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00324 [I 2022-10-10 01:21:16] ...trap/engines/engine.py.230: loss: 0.20263 [I 2022-10-10 01:21:16] ...optimizers/trijoint.py.151: total_norm: 28.759449 [I 2022-10-10 01:21:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 1440/2304 [I 2022-10-10 01:21:24] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:15 [I 2022-10-10 01:21:24] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00313 [I 2022-10-10 01:21:24] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-10 01:21:24] ...optimizers/trijoint.py.151: total_norm: 11.498540 [I 2022-10-10 01:21:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1450/2304 [I 2022-10-10 01:21:33] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:08 [I 2022-10-10 01:21:33] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00314 [I 2022-10-10 01:21:33] ...trap/engines/engine.py.230: loss: 0.15684 [I 2022-10-10 01:21:33] ...optimizers/trijoint.py.151: total_norm: 19.424788 [I 2022-10-10 01:21:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1460/2304 [I 2022-10-10 01:21:42] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:01 [I 2022-10-10 01:21:42] ...trap/engines/engine.py.229: process: 0.86483 | load: 0.00311 [I 2022-10-10 01:21:42] ...trap/engines/engine.py.230: loss: 0.10695 [I 2022-10-10 01:21:42] ...optimizers/trijoint.py.151: total_norm: 11.165009 [I 2022-10-10 01:21:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 1470/2304 [I 2022-10-10 01:21:50] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:44 [I 2022-10-10 01:21:50] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00298 [I 2022-10-10 01:21:50] ...trap/engines/engine.py.230: loss: 0.13083 [I 2022-10-10 01:21:50] ...optimizers/trijoint.py.151: total_norm: 11.157837 [I 2022-10-10 01:21:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1480/2304 [I 2022-10-10 01:21:59] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:38 [I 2022-10-10 01:21:59] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00297 [I 2022-10-10 01:21:59] ...trap/engines/engine.py.230: loss: 0.13508 [I 2022-10-10 01:21:59] ...optimizers/trijoint.py.151: total_norm: 9.550168 [I 2022-10-10 01:22:07] ...trap/engines/engine.py.224: train: epoch 61 | batch 1490/2304 [I 2022-10-10 01:22:07] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:30 [I 2022-10-10 01:22:07] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00300 [I 2022-10-10 01:22:07] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-10 01:22:07] ...optimizers/trijoint.py.151: total_norm: 15.617375 [I 2022-10-10 01:22:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1500/2304 [I 2022-10-10 01:22:16] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:16 [I 2022-10-10 01:22:16] ...trap/engines/engine.py.229: process: 0.83159 | load: 0.00301 [I 2022-10-10 01:22:16] ...trap/engines/engine.py.230: loss: 0.10859 [I 2022-10-10 01:22:16] ...optimizers/trijoint.py.151: total_norm: 8.753661 [I 2022-10-10 01:22:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 1510/2304 [I 2022-10-10 01:22:24] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:11 [I 2022-10-10 01:22:24] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00297 [I 2022-10-10 01:22:24] ...trap/engines/engine.py.230: loss: 0.19202 [I 2022-10-10 01:22:24] ...optimizers/trijoint.py.151: total_norm: 13.462213 [I 2022-10-10 01:22:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 1520/2304 [I 2022-10-10 01:22:32] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:02 [I 2022-10-10 01:22:32] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00323 [I 2022-10-10 01:22:32] ...trap/engines/engine.py.230: loss: 0.14477 [I 2022-10-10 01:22:32] ...optimizers/trijoint.py.151: total_norm: 25.780737 [I 2022-10-10 01:22:41] ...trap/engines/engine.py.224: train: epoch 61 | batch 1530/2304 [I 2022-10-10 01:22:41] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:51 [I 2022-10-10 01:22:41] ...trap/engines/engine.py.229: process: 0.85300 | load: 0.00299 [I 2022-10-10 01:22:41] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-10 01:22:41] ...optimizers/trijoint.py.151: total_norm: 16.248734 [I 2022-10-10 01:22:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 1540/2304 [I 2022-10-10 01:22:49] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:44 [I 2022-10-10 01:22:49] ...trap/engines/engine.py.229: process: 0.84520 | load: 0.00301 [I 2022-10-10 01:22:49] ...trap/engines/engine.py.230: loss: 0.10746 [I 2022-10-10 01:22:49] ...optimizers/trijoint.py.151: total_norm: 14.808046 [I 2022-10-10 01:22:58] ...trap/engines/engine.py.224: train: epoch 61 | batch 1550/2304 [I 2022-10-10 01:22:58] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:36 [I 2022-10-10 01:22:58] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00305 [I 2022-10-10 01:22:58] ...trap/engines/engine.py.230: loss: 0.18227 [I 2022-10-10 01:22:58] ...optimizers/trijoint.py.151: total_norm: 26.493147 [I 2022-10-10 01:23:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 1560/2304 [I 2022-10-10 01:23:06] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:28 [I 2022-10-10 01:23:06] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00300 [I 2022-10-10 01:23:06] ...trap/engines/engine.py.230: loss: 0.25547 [I 2022-10-10 01:23:06] ...optimizers/trijoint.py.151: total_norm: 20.625877 [I 2022-10-10 01:23:15] ...trap/engines/engine.py.224: train: epoch 61 | batch 1570/2304 [I 2022-10-10 01:23:15] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:25 [I 2022-10-10 01:23:15] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00298 [I 2022-10-10 01:23:15] ...trap/engines/engine.py.230: loss: 0.15718 [I 2022-10-10 01:23:15] ...optimizers/trijoint.py.151: total_norm: 13.478248 [I 2022-10-10 01:23:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1580/2304 [I 2022-10-10 01:23:23] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:09 [I 2022-10-10 01:23:23] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00305 [I 2022-10-10 01:23:23] ...trap/engines/engine.py.230: loss: 0.25878 [I 2022-10-10 01:23:23] ...optimizers/trijoint.py.151: total_norm: 16.899057 [I 2022-10-10 01:23:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 1590/2304 [I 2022-10-10 01:23:32] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:08 [I 2022-10-10 01:23:32] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00304 [I 2022-10-10 01:23:32] ...trap/engines/engine.py.230: loss: 0.11290 [I 2022-10-10 01:23:32] ...optimizers/trijoint.py.151: total_norm: 16.365883 [I 2022-10-10 01:23:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 1600/2304 [I 2022-10-10 01:23:40] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:09:55 [I 2022-10-10 01:23:40] ...trap/engines/engine.py.229: process: 0.85146 | load: 0.00300 [I 2022-10-10 01:23:40] ...trap/engines/engine.py.230: loss: 0.10674 [I 2022-10-10 01:23:40] ...optimizers/trijoint.py.151: total_norm: 9.421001 [I 2022-10-10 01:23:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 1610/2304 [I 2022-10-10 01:23:49] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:45 [I 2022-10-10 01:23:49] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00298 [I 2022-10-10 01:23:49] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-10 01:23:49] ...optimizers/trijoint.py.151: total_norm: 22.059479 [I 2022-10-10 01:23:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 1620/2304 [I 2022-10-10 01:23:57] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:39 [I 2022-10-10 01:23:57] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00297 [I 2022-10-10 01:23:57] ...trap/engines/engine.py.230: loss: 0.17915 [I 2022-10-10 01:23:57] ...optimizers/trijoint.py.151: total_norm: 16.610607 [I 2022-10-10 01:24:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 1630/2304 [I 2022-10-10 01:24:06] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:29 [I 2022-10-10 01:24:06] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00304 [I 2022-10-10 01:24:06] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-10 01:24:06] ...optimizers/trijoint.py.151: total_norm: 15.948004 [I 2022-10-10 01:24:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1640/2304 [I 2022-10-10 01:24:14] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:23 [I 2022-10-10 01:24:14] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00310 [I 2022-10-10 01:24:14] ...trap/engines/engine.py.230: loss: 0.10577 [I 2022-10-10 01:24:14] ...optimizers/trijoint.py.151: total_norm: 10.355065 [I 2022-10-10 01:24:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1650/2304 [I 2022-10-10 01:24:23] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:18 [I 2022-10-10 01:24:23] ...trap/engines/engine.py.229: process: 0.86726 | load: 0.00311 [I 2022-10-10 01:24:23] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-10 01:24:23] ...optimizers/trijoint.py.151: total_norm: 13.872993 [I 2022-10-10 01:24:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1660/2304 [I 2022-10-10 01:24:31] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:09 [I 2022-10-10 01:24:31] ...trap/engines/engine.py.229: process: 0.85482 | load: 0.00320 [I 2022-10-10 01:24:31] ...trap/engines/engine.py.230: loss: 0.18978 [I 2022-10-10 01:24:31] ...optimizers/trijoint.py.151: total_norm: 10.283485 [I 2022-10-10 01:24:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 1670/2304 [I 2022-10-10 01:24:40] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:04 [I 2022-10-10 01:24:40] ...trap/engines/engine.py.229: process: 0.86045 | load: 0.00322 [I 2022-10-10 01:24:40] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-10 01:24:40] ...optimizers/trijoint.py.151: total_norm: 12.067476 [I 2022-10-10 01:24:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1680/2304 [I 2022-10-10 01:24:48] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:57 [I 2022-10-10 01:24:48] ...trap/engines/engine.py.229: process: 0.86815 | load: 0.00326 [I 2022-10-10 01:24:48] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-10 01:24:48] ...optimizers/trijoint.py.151: total_norm: 19.604906 [I 2022-10-10 01:24:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 1690/2304 [I 2022-10-10 01:24:57] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:43 [I 2022-10-10 01:24:57] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00311 [I 2022-10-10 01:24:57] ...trap/engines/engine.py.230: loss: 0.13270 [I 2022-10-10 01:24:57] ...optimizers/trijoint.py.151: total_norm: 31.481743 [I 2022-10-10 01:25:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 1700/2304 [I 2022-10-10 01:25:06] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:33 [I 2022-10-10 01:25:06] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00345 [I 2022-10-10 01:25:06] ...trap/engines/engine.py.230: loss: 0.11947 [I 2022-10-10 01:25:06] ...optimizers/trijoint.py.151: total_norm: 24.284286 [I 2022-10-10 01:25:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1710/2304 [I 2022-10-10 01:25:14] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:26 [I 2022-10-10 01:25:14] ...trap/engines/engine.py.229: process: 0.85801 | load: 0.00311 [I 2022-10-10 01:25:14] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-10 01:25:14] ...optimizers/trijoint.py.151: total_norm: 12.938637 [I 2022-10-10 01:25:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1720/2304 [I 2022-10-10 01:25:23] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:17 [I 2022-10-10 01:25:23] ...trap/engines/engine.py.229: process: 0.85383 | load: 0.00339 [I 2022-10-10 01:25:23] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-10 01:25:23] ...optimizers/trijoint.py.151: total_norm: 21.118240 [I 2022-10-10 01:25:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1730/2304 [I 2022-10-10 01:25:31] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:07 [I 2022-10-10 01:25:31] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00310 [I 2022-10-10 01:25:31] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-10 01:25:31] ...optimizers/trijoint.py.151: total_norm: 27.222692 [I 2022-10-10 01:25:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 1740/2304 [I 2022-10-10 01:25:40] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:00 [I 2022-10-10 01:25:40] ...trap/engines/engine.py.229: process: 0.84405 | load: 0.00325 [I 2022-10-10 01:25:40] ...trap/engines/engine.py.230: loss: 0.14892 [I 2022-10-10 01:25:40] ...optimizers/trijoint.py.151: total_norm: 12.404896 [I 2022-10-10 01:25:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1750/2304 [I 2022-10-10 01:25:48] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:53 [I 2022-10-10 01:25:48] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00319 [I 2022-10-10 01:25:48] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-10 01:25:48] ...optimizers/trijoint.py.151: total_norm: 14.901645 [I 2022-10-10 01:25:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 1760/2304 [I 2022-10-10 01:25:57] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:42 [I 2022-10-10 01:25:57] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00314 [I 2022-10-10 01:25:57] ...trap/engines/engine.py.230: loss: 0.15896 [I 2022-10-10 01:25:57] ...optimizers/trijoint.py.151: total_norm: 15.710808 [I 2022-10-10 01:26:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1770/2304 [I 2022-10-10 01:26:05] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:35 [I 2022-10-10 01:26:05] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00311 [I 2022-10-10 01:26:05] ...trap/engines/engine.py.230: loss: 0.14286 [I 2022-10-10 01:26:05] ...optimizers/trijoint.py.151: total_norm: 12.240061 [I 2022-10-10 01:26:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1780/2304 [I 2022-10-10 01:26:14] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:26 [I 2022-10-10 01:26:14] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00312 [I 2022-10-10 01:26:14] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-10 01:26:14] ...optimizers/trijoint.py.151: total_norm: 54.258759 [I 2022-10-10 01:26:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1790/2304 [I 2022-10-10 01:26:23] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:17 [I 2022-10-10 01:26:23] ...trap/engines/engine.py.229: process: 0.85623 | load: 0.00315 [I 2022-10-10 01:26:23] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-10 01:26:23] ...optimizers/trijoint.py.151: total_norm: 12.283289 [I 2022-10-10 01:26:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1800/2304 [I 2022-10-10 01:26:31] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:08 [I 2022-10-10 01:26:31] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00312 [I 2022-10-10 01:26:31] ...trap/engines/engine.py.230: loss: 0.13580 [I 2022-10-10 01:26:31] ...optimizers/trijoint.py.151: total_norm: 53.579212 [I 2022-10-10 01:26:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 1810/2304 [I 2022-10-10 01:26:40] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:03 [I 2022-10-10 01:26:40] ...trap/engines/engine.py.229: process: 0.86279 | load: 0.00309 [I 2022-10-10 01:26:40] ...trap/engines/engine.py.230: loss: 0.16962 [I 2022-10-10 01:26:40] ...optimizers/trijoint.py.151: total_norm: 31.262859 [I 2022-10-10 01:26:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1820/2304 [I 2022-10-10 01:26:48] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:55 [I 2022-10-10 01:26:48] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00306 [I 2022-10-10 01:26:48] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-10 01:26:48] ...optimizers/trijoint.py.151: total_norm: 21.090940 [I 2022-10-10 01:26:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 1830/2304 [I 2022-10-10 01:26:57] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:43 [I 2022-10-10 01:26:57] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00310 [I 2022-10-10 01:26:57] ...trap/engines/engine.py.230: loss: 0.14086 [I 2022-10-10 01:26:57] ...optimizers/trijoint.py.151: total_norm: 15.550422 [I 2022-10-10 01:27:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1840/2304 [I 2022-10-10 01:27:05] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:37 [I 2022-10-10 01:27:05] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00314 [I 2022-10-10 01:27:05] ...trap/engines/engine.py.230: loss: 0.09429 [I 2022-10-10 01:27:05] ...optimizers/trijoint.py.151: total_norm: 8.917856 [I 2022-10-10 01:27:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1850/2304 [I 2022-10-10 01:27:14] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:25 [I 2022-10-10 01:27:14] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00310 [I 2022-10-10 01:27:14] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-10 01:27:14] ...optimizers/trijoint.py.151: total_norm: 12.527777 [I 2022-10-10 01:27:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 1860/2304 [I 2022-10-10 01:27:22] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:18 [I 2022-10-10 01:27:22] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00315 [I 2022-10-10 01:27:22] ...trap/engines/engine.py.230: loss: 0.21929 [I 2022-10-10 01:27:22] ...optimizers/trijoint.py.151: total_norm: 24.479080 [I 2022-10-10 01:27:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1870/2304 [I 2022-10-10 01:27:31] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:09 [I 2022-10-10 01:27:31] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00313 [I 2022-10-10 01:27:31] ...trap/engines/engine.py.230: loss: 0.15926 [I 2022-10-10 01:27:31] ...optimizers/trijoint.py.151: total_norm: 21.239647 [I 2022-10-10 01:27:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 1880/2304 [I 2022-10-10 01:27:39] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:02 [I 2022-10-10 01:27:39] ...trap/engines/engine.py.229: process: 0.88072 | load: 0.00312 [I 2022-10-10 01:27:39] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-10 01:27:39] ...optimizers/trijoint.py.151: total_norm: 11.508950 [I 2022-10-10 01:27:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1890/2304 [I 2022-10-10 01:27:48] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:54 [I 2022-10-10 01:27:48] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00313 [I 2022-10-10 01:27:48] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-10 01:27:48] ...optimizers/trijoint.py.151: total_norm: 12.040183 [I 2022-10-10 01:27:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 1900/2304 [I 2022-10-10 01:27:56] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:44 [I 2022-10-10 01:27:56] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00306 [I 2022-10-10 01:27:56] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-10 01:27:56] ...optimizers/trijoint.py.151: total_norm: 21.177387 [I 2022-10-10 01:28:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1910/2304 [I 2022-10-10 01:28:05] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:37 [I 2022-10-10 01:28:05] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00307 [I 2022-10-10 01:28:05] ...trap/engines/engine.py.230: loss: 0.19180 [I 2022-10-10 01:28:05] ...optimizers/trijoint.py.151: total_norm: 22.569920 [I 2022-10-10 01:28:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1920/2304 [I 2022-10-10 01:28:14] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:27 [I 2022-10-10 01:28:14] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00303 [I 2022-10-10 01:28:14] ...trap/engines/engine.py.230: loss: 0.11094 [I 2022-10-10 01:28:14] ...optimizers/trijoint.py.151: total_norm: 17.427519 [I 2022-10-10 01:28:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 1930/2304 [I 2022-10-10 01:28:22] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:19 [I 2022-10-10 01:28:22] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00302 [I 2022-10-10 01:28:22] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-10 01:28:22] ...optimizers/trijoint.py.151: total_norm: 19.788225 [I 2022-10-10 01:28:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1940/2304 [I 2022-10-10 01:28:31] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:09 [I 2022-10-10 01:28:31] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00316 [I 2022-10-10 01:28:31] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-10 01:28:31] ...optimizers/trijoint.py.151: total_norm: 37.377445 [I 2022-10-10 01:28:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 1950/2304 [I 2022-10-10 01:28:39] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:02 [I 2022-10-10 01:28:39] ...trap/engines/engine.py.229: process: 0.87068 | load: 0.00307 [I 2022-10-10 01:28:39] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-10 01:28:39] ...optimizers/trijoint.py.151: total_norm: 11.832563 [I 2022-10-10 01:28:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1960/2304 [I 2022-10-10 01:28:48] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:04:54 [I 2022-10-10 01:28:48] ...trap/engines/engine.py.229: process: 0.86098 | load: 0.00309 [I 2022-10-10 01:28:48] ...trap/engines/engine.py.230: loss: 0.17046 [I 2022-10-10 01:28:48] ...optimizers/trijoint.py.151: total_norm: 51.486412 [I 2022-10-10 01:28:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 1970/2304 [I 2022-10-10 01:28:56] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:43 [I 2022-10-10 01:28:56] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00295 [I 2022-10-10 01:28:56] ...trap/engines/engine.py.230: loss: 0.19449 [I 2022-10-10 01:28:56] ...optimizers/trijoint.py.151: total_norm: 12.303959 [I 2022-10-10 01:29:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1980/2304 [I 2022-10-10 01:29:05] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:34 [I 2022-10-10 01:29:05] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00302 [I 2022-10-10 01:29:05] ...trap/engines/engine.py.230: loss: 0.11778 [I 2022-10-10 01:29:05] ...optimizers/trijoint.py.151: total_norm: 19.550291 [I 2022-10-10 01:29:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 1990/2304 [I 2022-10-10 01:29:13] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:26 [I 2022-10-10 01:29:13] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00305 [I 2022-10-10 01:29:13] ...trap/engines/engine.py.230: loss: 0.12238 [I 2022-10-10 01:29:13] ...optimizers/trijoint.py.151: total_norm: 19.298037 [I 2022-10-10 01:29:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 2000/2304 [I 2022-10-10 01:29:22] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:19 [I 2022-10-10 01:29:22] ...trap/engines/engine.py.229: process: 0.84846 | load: 0.00298 [I 2022-10-10 01:29:22] ...trap/engines/engine.py.230: loss: 0.23828 [I 2022-10-10 01:29:22] ...optimizers/trijoint.py.151: total_norm: 28.798222 [I 2022-10-10 01:29:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2010/2304 [I 2022-10-10 01:29:30] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:09 [I 2022-10-10 01:29:30] ...trap/engines/engine.py.229: process: 0.85441 | load: 0.00299 [I 2022-10-10 01:29:30] ...trap/engines/engine.py.230: loss: 0.16126 [I 2022-10-10 01:29:30] ...optimizers/trijoint.py.151: total_norm: 11.600076 [I 2022-10-10 01:29:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 2020/2304 [I 2022-10-10 01:29:39] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:00 [I 2022-10-10 01:29:39] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00299 [I 2022-10-10 01:29:39] ...trap/engines/engine.py.230: loss: 0.17058 [I 2022-10-10 01:29:39] ...optimizers/trijoint.py.151: total_norm: 26.400473 [I 2022-10-10 01:29:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 2030/2304 [I 2022-10-10 01:29:47] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:51 [I 2022-10-10 01:29:47] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00303 [I 2022-10-10 01:29:47] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-10 01:29:47] ...optimizers/trijoint.py.151: total_norm: 26.407396 [I 2022-10-10 01:29:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 2040/2304 [I 2022-10-10 01:29:56] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:43 [I 2022-10-10 01:29:56] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00330 [I 2022-10-10 01:29:56] ...trap/engines/engine.py.230: loss: 0.21553 [I 2022-10-10 01:29:56] ...optimizers/trijoint.py.151: total_norm: 25.385160 [I 2022-10-10 01:30:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 2050/2304 [I 2022-10-10 01:30:04] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:36 [I 2022-10-10 01:30:04] ...trap/engines/engine.py.229: process: 0.85740 | load: 0.00301 [I 2022-10-10 01:30:04] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-10 01:30:04] ...optimizers/trijoint.py.151: total_norm: 8.767900 [I 2022-10-10 01:30:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 2060/2304 [I 2022-10-10 01:30:13] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:27 [I 2022-10-10 01:30:13] ...trap/engines/engine.py.229: process: 0.86864 | load: 0.00304 [I 2022-10-10 01:30:13] ...trap/engines/engine.py.230: loss: 0.10095 [I 2022-10-10 01:30:13] ...optimizers/trijoint.py.151: total_norm: 8.451726 [I 2022-10-10 01:30:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 2070/2304 [I 2022-10-10 01:30:21] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:19 [I 2022-10-10 01:30:21] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00302 [I 2022-10-10 01:30:21] ...trap/engines/engine.py.230: loss: 0.13327 [I 2022-10-10 01:30:21] ...optimizers/trijoint.py.151: total_norm: 9.825709 [I 2022-10-10 01:30:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2080/2304 [I 2022-10-10 01:30:30] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:10 [I 2022-10-10 01:30:30] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00305 [I 2022-10-10 01:30:30] ...trap/engines/engine.py.230: loss: 0.14371 [I 2022-10-10 01:30:30] ...optimizers/trijoint.py.151: total_norm: 15.052646 [I 2022-10-10 01:30:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 2090/2304 [I 2022-10-10 01:30:38] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:02 [I 2022-10-10 01:30:38] ...trap/engines/engine.py.229: process: 0.85672 | load: 0.00304 [I 2022-10-10 01:30:38] ...trap/engines/engine.py.230: loss: 0.16866 [I 2022-10-10 01:30:38] ...optimizers/trijoint.py.151: total_norm: 16.891594 [I 2022-10-10 01:30:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 2100/2304 [I 2022-10-10 01:30:47] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:02:53 [I 2022-10-10 01:30:47] ...trap/engines/engine.py.229: process: 0.86477 | load: 0.00301 [I 2022-10-10 01:30:47] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-10 01:30:47] ...optimizers/trijoint.py.151: total_norm: 17.497866 [I 2022-10-10 01:30:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 2110/2304 [I 2022-10-10 01:30:55] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:44 [I 2022-10-10 01:30:55] ...trap/engines/engine.py.229: process: 0.84374 | load: 0.00297 [I 2022-10-10 01:30:55] ...trap/engines/engine.py.230: loss: 0.14678 [I 2022-10-10 01:30:55] ...optimizers/trijoint.py.151: total_norm: 17.430325 [I 2022-10-10 01:31:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 2120/2304 [I 2022-10-10 01:31:04] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:36 [I 2022-10-10 01:31:04] ...trap/engines/engine.py.229: process: 0.86500 | load: 0.00305 [I 2022-10-10 01:31:04] ...trap/engines/engine.py.230: loss: 0.21508 [I 2022-10-10 01:31:04] ...optimizers/trijoint.py.151: total_norm: 20.214396 [I 2022-10-10 01:31:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 2130/2304 [I 2022-10-10 01:31:12] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:28 [I 2022-10-10 01:31:12] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00304 [I 2022-10-10 01:31:12] ...trap/engines/engine.py.230: loss: 0.10522 [I 2022-10-10 01:31:12] ...optimizers/trijoint.py.151: total_norm: 8.153217 [I 2022-10-10 01:31:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 2140/2304 [I 2022-10-10 01:31:21] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:21 [I 2022-10-10 01:31:21] ...trap/engines/engine.py.229: process: 0.88376 | load: 0.00311 [I 2022-10-10 01:31:21] ...trap/engines/engine.py.230: loss: 0.13365 [I 2022-10-10 01:31:21] ...optimizers/trijoint.py.151: total_norm: 9.621058 [I 2022-10-10 01:31:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2150/2304 [I 2022-10-10 01:31:30] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:12 [I 2022-10-10 01:31:30] ...trap/engines/engine.py.229: process: 0.85447 | load: 0.00312 [I 2022-10-10 01:31:30] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-10 01:31:30] ...optimizers/trijoint.py.151: total_norm: 29.459450 [I 2022-10-10 01:31:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 2160/2304 [I 2022-10-10 01:31:38] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:02 [I 2022-10-10 01:31:38] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00342 [I 2022-10-10 01:31:38] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-10 01:31:38] ...optimizers/trijoint.py.151: total_norm: 9.184277 [I 2022-10-10 01:31:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 2170/2304 [I 2022-10-10 01:31:47] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:53 [I 2022-10-10 01:31:47] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00311 [I 2022-10-10 01:31:47] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-10 01:31:47] ...optimizers/trijoint.py.151: total_norm: 17.052969 [I 2022-10-10 01:31:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 2180/2304 [I 2022-10-10 01:31:55] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:45 [I 2022-10-10 01:31:55] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00313 [I 2022-10-10 01:31:55] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-10 01:31:55] ...optimizers/trijoint.py.151: total_norm: 17.125553 [I 2022-10-10 01:32:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 2190/2304 [I 2022-10-10 01:32:04] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:37 [I 2022-10-10 01:32:04] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00329 [I 2022-10-10 01:32:04] ...trap/engines/engine.py.230: loss: 0.13295 [I 2022-10-10 01:32:04] ...optimizers/trijoint.py.151: total_norm: 12.410806 [I 2022-10-10 01:32:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 2200/2304 [I 2022-10-10 01:32:12] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:29 [I 2022-10-10 01:32:12] ...trap/engines/engine.py.229: process: 0.87238 | load: 0.00310 [I 2022-10-10 01:32:12] ...trap/engines/engine.py.230: loss: 0.15003 [I 2022-10-10 01:32:12] ...optimizers/trijoint.py.151: total_norm: 12.709517 [I 2022-10-10 01:32:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 2210/2304 [I 2022-10-10 01:32:21] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:20 [I 2022-10-10 01:32:21] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00337 [I 2022-10-10 01:32:21] ...trap/engines/engine.py.230: loss: 0.13382 [I 2022-10-10 01:32:21] ...optimizers/trijoint.py.151: total_norm: 11.400321 [I 2022-10-10 01:32:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2220/2304 [I 2022-10-10 01:32:30] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:11 [I 2022-10-10 01:32:30] ...trap/engines/engine.py.229: process: 0.86185 | load: 0.00315 [I 2022-10-10 01:32:30] ...trap/engines/engine.py.230: loss: 0.17229 [I 2022-10-10 01:32:30] ...optimizers/trijoint.py.151: total_norm: 13.447071 [I 2022-10-10 01:32:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 2230/2304 [I 2022-10-10 01:32:38] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:03 [I 2022-10-10 01:32:38] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00313 [I 2022-10-10 01:32:38] ...trap/engines/engine.py.230: loss: 0.22029 [I 2022-10-10 01:32:38] ...optimizers/trijoint.py.151: total_norm: 17.665758 [I 2022-10-10 01:32:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 2240/2304 [I 2022-10-10 01:32:47] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:54 [I 2022-10-10 01:32:47] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00313 [I 2022-10-10 01:32:47] ...trap/engines/engine.py.230: loss: 0.08990 [I 2022-10-10 01:32:47] ...optimizers/trijoint.py.151: total_norm: 12.372028 [I 2022-10-10 01:32:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 2250/2304 [I 2022-10-10 01:32:55] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:00:46 [I 2022-10-10 01:32:55] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00311 [I 2022-10-10 01:32:55] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-10 01:32:55] ...optimizers/trijoint.py.151: total_norm: 14.966000 [I 2022-10-10 01:33:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 2260/2304 [I 2022-10-10 01:33:04] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:37 [I 2022-10-10 01:33:04] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00312 [I 2022-10-10 01:33:04] ...trap/engines/engine.py.230: loss: 0.14266 [I 2022-10-10 01:33:04] ...optimizers/trijoint.py.151: total_norm: 16.553879 [I 2022-10-10 01:33:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 2270/2304 [I 2022-10-10 01:33:13] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:28 [I 2022-10-10 01:33:13] ...trap/engines/engine.py.229: process: 0.87252 | load: 0.00318 [I 2022-10-10 01:33:13] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-10 01:33:13] ...optimizers/trijoint.py.151: total_norm: 29.270628 [I 2022-10-10 01:33:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 2280/2304 [I 2022-10-10 01:33:21] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:20 [I 2022-10-10 01:33:21] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00309 [I 2022-10-10 01:33:21] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-10 01:33:21] ...optimizers/trijoint.py.151: total_norm: 10.182532 [I 2022-10-10 01:33:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2290/2304 [I 2022-10-10 01:33:30] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:11 [I 2022-10-10 01:33:30] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00313 [I 2022-10-10 01:33:30] ...trap/engines/engine.py.230: loss: 0.16268 [I 2022-10-10 01:33:30] ...optimizers/trijoint.py.151: total_norm: 64.703346 [I 2022-10-10 01:33:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 2300/2304 [I 2022-10-10 01:33:38] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:03 [I 2022-10-10 01:33:38] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00019 [I 2022-10-10 01:33:38] ...trap/engines/engine.py.230: loss: 0.20396 [I 2022-10-10 01:33:38] ...optimizers/trijoint.py.151: total_norm: 46.592995 [I 2022-10-10 01:33:41] ...trap/engines/engine.py.224: train: epoch 61 | batch 2304/2304 [I 2022-10-10 01:33:42] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:00 [I 2022-10-10 01:33:42] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00020 [I 2022-10-10 01:33:42] ...trap/engines/engine.py.230: loss: 0.15575 [I 2022-10-10 01:33:42] ...optimizers/trijoint.py.151: total_norm: 13.865755 [S 2022-10-10 01:33:43] ...trap/engines/engine.py.129: train_epoch.epoch: 61 [S 2022-10-10 01:33:43] ...trap/engines/engine.py.129: train_epoch.loss: 0.14689418219369302 [I 2022-10-10 01:33:43] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 01:33:44] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 61 [I 2022-10-10 01:33:44] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 01:33:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 01:33:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 01:33:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 01:33:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 01:33:48] ...trap/engines/engine.py.321: eval: epoch 61 | batch 0/510 [I 2022-10-10 01:33:48] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:50 [I 2022-10-10 01:33:48] ...trap/engines/engine.py.326: process: 4.56960 | load: 3.48152 [I 2022-10-10 01:33:53] ...trap/engines/engine.py.321: eval: epoch 61 | batch 10/510 [I 2022-10-10 01:33:53] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:30 [I 2022-10-10 01:33:53] ...trap/engines/engine.py.326: process: 0.47541 | load: 0.00020 [I 2022-10-10 01:33:57] ...trap/engines/engine.py.321: eval: epoch 61 | batch 20/510 [I 2022-10-10 01:33:57] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:04:03 [I 2022-10-10 01:33:57] ...trap/engines/engine.py.326: process: 0.43991 | load: 0.00021 [I 2022-10-10 01:34:03] ...trap/engines/engine.py.321: eval: epoch 61 | batch 30/510 [I 2022-10-10 01:34:03] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:46 [I 2022-10-10 01:34:03] ...trap/engines/engine.py.326: process: 0.47960 | load: 0.00020 [I 2022-10-10 01:34:09] ...trap/engines/engine.py.321: eval: epoch 61 | batch 40/510 [I 2022-10-10 01:34:09] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:13 [I 2022-10-10 01:34:09] ...trap/engines/engine.py.326: process: 0.63210 | load: 0.00022 [I 2022-10-10 01:34:14] ...trap/engines/engine.py.321: eval: epoch 61 | batch 50/510 [I 2022-10-10 01:34:14] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:56 [I 2022-10-10 01:34:14] ...trap/engines/engine.py.326: process: 0.53073 | load: 0.00020 [I 2022-10-10 01:34:19] ...trap/engines/engine.py.321: eval: epoch 61 | batch 60/510 [I 2022-10-10 01:34:19] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:30 [I 2022-10-10 01:34:19] ...trap/engines/engine.py.326: process: 0.45037 | load: 0.00020 [I 2022-10-10 01:34:24] ...trap/engines/engine.py.321: eval: epoch 61 | batch 70/510 [I 2022-10-10 01:34:24] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:33 [I 2022-10-10 01:34:24] ...trap/engines/engine.py.326: process: 0.45752 | load: 0.00020 [I 2022-10-10 01:34:30] ...trap/engines/engine.py.321: eval: epoch 61 | batch 80/510 [I 2022-10-10 01:34:30] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:18 [I 2022-10-10 01:34:30] ...trap/engines/engine.py.326: process: 0.54941 | load: 0.00020 [I 2022-10-10 01:34:35] ...trap/engines/engine.py.321: eval: epoch 61 | batch 90/510 [I 2022-10-10 01:34:35] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:44 [I 2022-10-10 01:34:35] ...trap/engines/engine.py.326: process: 0.53314 | load: 0.00020 [I 2022-10-10 01:34:41] ...trap/engines/engine.py.321: eval: epoch 61 | batch 100/510 [I 2022-10-10 01:34:41] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:27 [I 2022-10-10 01:34:41] ...trap/engines/engine.py.326: process: 0.31370 | load: 0.00020 [I 2022-10-10 01:34:44] ...trap/engines/engine.py.321: eval: epoch 61 | batch 110/510 [I 2022-10-10 01:34:44] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:15 [I 2022-10-10 01:34:44] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00020 [I 2022-10-10 01:34:47] ...trap/engines/engine.py.321: eval: epoch 61 | batch 120/510 [I 2022-10-10 01:34:47] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-10 01:34:47] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00021 [I 2022-10-10 01:34:50] ...trap/engines/engine.py.321: eval: epoch 61 | batch 130/510 [I 2022-10-10 01:34:50] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:01 [I 2022-10-10 01:34:50] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00022 [I 2022-10-10 01:34:53] ...trap/engines/engine.py.321: eval: epoch 61 | batch 140/510 [I 2022-10-10 01:34:53] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:56 [I 2022-10-10 01:34:53] ...trap/engines/engine.py.326: process: 0.31142 | load: 0.00020 [I 2022-10-10 01:34:57] ...trap/engines/engine.py.321: eval: epoch 61 | batch 150/510 [I 2022-10-10 01:34:57] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-10 01:34:57] ...trap/engines/engine.py.326: process: 0.31699 | load: 0.00020 [I 2022-10-10 01:35:00] ...trap/engines/engine.py.321: eval: epoch 61 | batch 160/510 [I 2022-10-10 01:35:00] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:51 [I 2022-10-10 01:35:00] ...trap/engines/engine.py.326: process: 0.31438 | load: 0.00021 [I 2022-10-10 01:35:03] ...trap/engines/engine.py.321: eval: epoch 61 | batch 170/510 [I 2022-10-10 01:35:03] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-10 01:35:03] ...trap/engines/engine.py.326: process: 0.31553 | load: 0.00021 [I 2022-10-10 01:35:06] ...trap/engines/engine.py.321: eval: epoch 61 | batch 180/510 [I 2022-10-10 01:35:06] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:46 [I 2022-10-10 01:35:06] ...trap/engines/engine.py.326: process: 0.32398 | load: 0.00021 [I 2022-10-10 01:35:09] ...trap/engines/engine.py.321: eval: epoch 61 | batch 190/510 [I 2022-10-10 01:35:09] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:42 [I 2022-10-10 01:35:09] ...trap/engines/engine.py.326: process: 0.32192 | load: 0.00021 [I 2022-10-10 01:35:13] ...trap/engines/engine.py.321: eval: epoch 61 | batch 200/510 [I 2022-10-10 01:35:13] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-10 01:35:13] ...trap/engines/engine.py.326: process: 0.32036 | load: 0.00021 [I 2022-10-10 01:35:16] ...trap/engines/engine.py.321: eval: epoch 61 | batch 210/510 [I 2022-10-10 01:35:16] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:36 [I 2022-10-10 01:35:16] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00021 [I 2022-10-10 01:35:19] ...trap/engines/engine.py.321: eval: epoch 61 | batch 220/510 [I 2022-10-10 01:35:19] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:33 [I 2022-10-10 01:35:19] ...trap/engines/engine.py.326: process: 0.32654 | load: 0.00021 [I 2022-10-10 01:35:22] ...trap/engines/engine.py.321: eval: epoch 61 | batch 230/510 [I 2022-10-10 01:35:22] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:30 [I 2022-10-10 01:35:22] ...trap/engines/engine.py.326: process: 0.32417 | load: 0.00020 [I 2022-10-10 01:35:25] ...trap/engines/engine.py.321: eval: epoch 61 | batch 240/510 [I 2022-10-10 01:35:25] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-10 01:35:25] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00021 [I 2022-10-10 01:35:29] ...trap/engines/engine.py.321: eval: epoch 61 | batch 250/510 [I 2022-10-10 01:35:29] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:23 [I 2022-10-10 01:35:29] ...trap/engines/engine.py.326: process: 0.32065 | load: 0.00021 [I 2022-10-10 01:35:32] ...trap/engines/engine.py.321: eval: epoch 61 | batch 260/510 [I 2022-10-10 01:35:32] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:20 [I 2022-10-10 01:35:32] ...trap/engines/engine.py.326: process: 0.32445 | load: 0.00021 [I 2022-10-10 01:35:35] ...trap/engines/engine.py.321: eval: epoch 61 | batch 270/510 [I 2022-10-10 01:35:35] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-10 01:35:35] ...trap/engines/engine.py.326: process: 0.31959 | load: 0.00022 [I 2022-10-10 01:35:38] ...trap/engines/engine.py.321: eval: epoch 61 | batch 280/510 [I 2022-10-10 01:35:38] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-10 01:35:38] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00021 [I 2022-10-10 01:35:41] ...trap/engines/engine.py.321: eval: epoch 61 | batch 290/510 [I 2022-10-10 01:35:41] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:09 [I 2022-10-10 01:35:41] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00021 [I 2022-10-10 01:35:45] ...trap/engines/engine.py.321: eval: epoch 61 | batch 300/510 [I 2022-10-10 01:35:45] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:07 [I 2022-10-10 01:35:45] ...trap/engines/engine.py.326: process: 0.32723 | load: 0.00019 [I 2022-10-10 01:35:48] ...trap/engines/engine.py.321: eval: epoch 61 | batch 310/510 [I 2022-10-10 01:35:48] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-10 01:35:48] ...trap/engines/engine.py.326: process: 0.31612 | load: 0.00020 [I 2022-10-10 01:35:51] ...trap/engines/engine.py.321: eval: epoch 61 | batch 320/510 [I 2022-10-10 01:35:51] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-10 01:35:51] ...trap/engines/engine.py.326: process: 0.31629 | load: 0.00023 [I 2022-10-10 01:35:54] ...trap/engines/engine.py.321: eval: epoch 61 | batch 330/510 [I 2022-10-10 01:35:54] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-10 01:35:54] ...trap/engines/engine.py.326: process: 0.31690 | load: 0.00020 [I 2022-10-10 01:35:57] ...trap/engines/engine.py.321: eval: epoch 61 | batch 340/510 [I 2022-10-10 01:35:57] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-10 01:35:57] ...trap/engines/engine.py.326: process: 0.32498 | load: 0.00019 [I 2022-10-10 01:36:01] ...trap/engines/engine.py.321: eval: epoch 61 | batch 350/510 [I 2022-10-10 01:36:01] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-10 01:36:01] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00020 [I 2022-10-10 01:36:04] ...trap/engines/engine.py.321: eval: epoch 61 | batch 360/510 [I 2022-10-10 01:36:04] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:48 [I 2022-10-10 01:36:04] ...trap/engines/engine.py.326: process: 0.32216 | load: 0.00020 [I 2022-10-10 01:36:07] ...trap/engines/engine.py.321: eval: epoch 61 | batch 370/510 [I 2022-10-10 01:36:07] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-10 01:36:07] ...trap/engines/engine.py.326: process: 0.31519 | load: 0.00021 [I 2022-10-10 01:36:10] ...trap/engines/engine.py.321: eval: epoch 61 | batch 380/510 [I 2022-10-10 01:36:10] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:42 [I 2022-10-10 01:36:10] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00021 [I 2022-10-10 01:36:14] ...trap/engines/engine.py.321: eval: epoch 61 | batch 390/510 [I 2022-10-10 01:36:14] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-10 01:36:14] ...trap/engines/engine.py.326: process: 0.31478 | load: 0.00021 [I 2022-10-10 01:36:17] ...trap/engines/engine.py.321: eval: epoch 61 | batch 400/510 [I 2022-10-10 01:36:17] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:35 [I 2022-10-10 01:36:17] ...trap/engines/engine.py.326: process: 0.31703 | load: 0.00020 [I 2022-10-10 01:36:20] ...trap/engines/engine.py.321: eval: epoch 61 | batch 410/510 [I 2022-10-10 01:36:20] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:31 [I 2022-10-10 01:36:20] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00019 [I 2022-10-10 01:36:23] ...trap/engines/engine.py.321: eval: epoch 61 | batch 420/510 [I 2022-10-10 01:36:23] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:28 [I 2022-10-10 01:36:23] ...trap/engines/engine.py.326: process: 0.31072 | load: 0.00022 [I 2022-10-10 01:36:26] ...trap/engines/engine.py.321: eval: epoch 61 | batch 430/510 [I 2022-10-10 01:36:26] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-10 01:36:26] ...trap/engines/engine.py.326: process: 0.32145 | load: 0.00020 [I 2022-10-10 01:36:29] ...trap/engines/engine.py.321: eval: epoch 61 | batch 440/510 [I 2022-10-10 01:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-10 01:36:29] ...trap/engines/engine.py.326: process: 0.31012 | load: 0.00020 [I 2022-10-10 01:36:33] ...trap/engines/engine.py.321: eval: epoch 61 | batch 450/510 [I 2022-10-10 01:36:33] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-10 01:36:33] ...trap/engines/engine.py.326: process: 0.32107 | load: 0.00019 [I 2022-10-10 01:36:36] ...trap/engines/engine.py.321: eval: epoch 61 | batch 460/510 [I 2022-10-10 01:36:36] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:15 [I 2022-10-10 01:36:36] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00019 [I 2022-10-10 01:36:39] ...trap/engines/engine.py.321: eval: epoch 61 | batch 470/510 [I 2022-10-10 01:36:39] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-10 01:36:39] ...trap/engines/engine.py.326: process: 0.31161 | load: 0.00021 [I 2022-10-10 01:36:42] ...trap/engines/engine.py.321: eval: epoch 61 | batch 480/510 [I 2022-10-10 01:36:42] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-10 01:36:42] ...trap/engines/engine.py.326: process: 0.31970 | load: 0.00020 [I 2022-10-10 01:36:45] ...trap/engines/engine.py.321: eval: epoch 61 | batch 490/510 [I 2022-10-10 01:36:45] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-10 01:36:45] ...trap/engines/engine.py.326: process: 0.31367 | load: 0.00021 [I 2022-10-10 01:36:49] ...trap/engines/engine.py.321: eval: epoch 61 | batch 500/510 [I 2022-10-10 01:36:49] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-10 01:36:49] ...trap/engines/engine.py.326: process: 0.32076 | load: 0.00020 [I 2022-10-10 01:36:52] ...trap/engines/engine.py.321: eval: epoch 61 | batch 510/510 [I 2022-10-10 01:36:52] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 01:36:52] ...trap/engines/engine.py.326: process: 0.31563 | load: 0.00016 [S 2022-10-10 01:36:53] ...trap/engines/engine.py.132: eval_epoch.epoch: 61 [I 2022-10-10 01:36:53] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.726400 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905600 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934800 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.717800 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905700 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015041 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009340 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008022 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013273 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008174 [S 2022-10-10 01:37:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007308 [I 2022-10-10 01:37:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 01:37:29] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 01:37:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 01:37:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 01:37:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 01:37:33] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 01:37:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 01:37:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 01:37:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 62 [I 2022-10-10 01:37:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 01:37:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 0/2304 [I 2022-10-10 01:37:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:52 [I 2022-10-10 01:37:42] ...trap/engines/engine.py.229: process: 2.80944 | load: 1.90028 [I 2022-10-10 01:37:42] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-10 01:37:42] ...optimizers/trijoint.py.151: total_norm: 13.567464 [I 2022-10-10 01:37:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 10/2304 [I 2022-10-10 01:37:50] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:13 [I 2022-10-10 01:37:50] ...trap/engines/engine.py.229: process: 0.83507 | load: 0.00565 [I 2022-10-10 01:37:50] ...trap/engines/engine.py.230: loss: 0.16439 [I 2022-10-10 01:37:50] ...optimizers/trijoint.py.151: total_norm: 18.026787 [I 2022-10-10 01:37:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 20/2304 [I 2022-10-10 01:37:59] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:01 [I 2022-10-10 01:37:59] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00455 [I 2022-10-10 01:37:59] ...trap/engines/engine.py.230: loss: 0.19421 [I 2022-10-10 01:37:59] ...optimizers/trijoint.py.151: total_norm: 30.946280 [I 2022-10-10 01:38:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 30/2304 [I 2022-10-10 01:38:07] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:24 [I 2022-10-10 01:38:07] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00439 [I 2022-10-10 01:38:07] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-10 01:38:07] ...optimizers/trijoint.py.151: total_norm: 8.680791 [I 2022-10-10 01:38:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 40/2304 [I 2022-10-10 01:38:16] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:59 [I 2022-10-10 01:38:16] ...trap/engines/engine.py.229: process: 0.83073 | load: 0.00414 [I 2022-10-10 01:38:16] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-10 01:38:16] ...optimizers/trijoint.py.151: total_norm: 7.604307 [I 2022-10-10 01:38:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 50/2304 [I 2022-10-10 01:38:24] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:51 [I 2022-10-10 01:38:24] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00403 [I 2022-10-10 01:38:24] ...trap/engines/engine.py.230: loss: 0.18504 [I 2022-10-10 01:38:24] ...optimizers/trijoint.py.151: total_norm: 15.054240 [I 2022-10-10 01:38:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 60/2304 [I 2022-10-10 01:38:33] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:32:09 [I 2022-10-10 01:38:33] ...trap/engines/engine.py.229: process: 0.86892 | load: 0.00381 [I 2022-10-10 01:38:33] ...trap/engines/engine.py.230: loss: 0.14662 [I 2022-10-10 01:38:33] ...optimizers/trijoint.py.151: total_norm: 10.966358 [I 2022-10-10 01:38:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 70/2304 [I 2022-10-10 01:38:42] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:47 [I 2022-10-10 01:38:42] ...trap/engines/engine.py.229: process: 0.85768 | load: 0.00331 [I 2022-10-10 01:38:42] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-10 01:38:42] ...optimizers/trijoint.py.151: total_norm: 35.759373 [I 2022-10-10 01:38:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 80/2304 [I 2022-10-10 01:38:50] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:42 [I 2022-10-10 01:38:50] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00368 [I 2022-10-10 01:38:50] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-10 01:38:50] ...optimizers/trijoint.py.151: total_norm: 24.389469 [I 2022-10-10 01:38:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 90/2304 [I 2022-10-10 01:38:59] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:23 [I 2022-10-10 01:38:59] ...trap/engines/engine.py.229: process: 0.83807 | load: 0.00362 [I 2022-10-10 01:38:59] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-10 01:38:59] ...optimizers/trijoint.py.151: total_norm: 10.500822 [I 2022-10-10 01:39:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 100/2304 [I 2022-10-10 01:39:07] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:21 [I 2022-10-10 01:39:07] ...trap/engines/engine.py.229: process: 0.86089 | load: 0.00410 [I 2022-10-10 01:39:07] ...trap/engines/engine.py.230: loss: 0.17281 [I 2022-10-10 01:39:07] ...optimizers/trijoint.py.151: total_norm: 12.533121 [I 2022-10-10 01:39:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 110/2304 [I 2022-10-10 01:39:16] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:23 [I 2022-10-10 01:39:16] ...trap/engines/engine.py.229: process: 0.87703 | load: 0.00375 [I 2022-10-10 01:39:16] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-10 01:39:16] ...optimizers/trijoint.py.151: total_norm: 18.372038 [I 2022-10-10 01:39:25] ...trap/engines/engine.py.224: train: epoch 62 | batch 120/2304 [I 2022-10-10 01:39:25] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:21 [I 2022-10-10 01:39:25] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00310 [I 2022-10-10 01:39:25] ...trap/engines/engine.py.230: loss: 0.16220 [I 2022-10-10 01:39:25] ...optimizers/trijoint.py.151: total_norm: 11.651371 [I 2022-10-10 01:39:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 130/2304 [I 2022-10-10 01:39:33] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:56 [I 2022-10-10 01:39:33] ...trap/engines/engine.py.229: process: 0.85455 | load: 0.00369 [I 2022-10-10 01:39:33] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-10 01:39:33] ...optimizers/trijoint.py.151: total_norm: 11.139038 [I 2022-10-10 01:39:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 140/2304 [I 2022-10-10 01:39:42] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:44 [I 2022-10-10 01:39:42] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00325 [I 2022-10-10 01:39:42] ...trap/engines/engine.py.230: loss: 0.12195 [I 2022-10-10 01:39:42] ...optimizers/trijoint.py.151: total_norm: 17.369331 [I 2022-10-10 01:39:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 150/2304 [I 2022-10-10 01:39:50] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:32 [I 2022-10-10 01:39:50] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00344 [I 2022-10-10 01:39:50] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-10 01:39:50] ...optimizers/trijoint.py.151: total_norm: 10.107944 [I 2022-10-10 01:39:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 160/2304 [I 2022-10-10 01:39:59] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:41 [I 2022-10-10 01:39:59] ...trap/engines/engine.py.229: process: 0.86316 | load: 0.00339 [I 2022-10-10 01:39:59] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-10 01:39:59] ...optimizers/trijoint.py.151: total_norm: 18.555157 [I 2022-10-10 01:40:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 170/2304 [I 2022-10-10 01:40:07] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:31 [I 2022-10-10 01:40:07] ...trap/engines/engine.py.229: process: 0.86977 | load: 0.00323 [I 2022-10-10 01:40:07] ...trap/engines/engine.py.230: loss: 0.12292 [I 2022-10-10 01:40:07] ...optimizers/trijoint.py.151: total_norm: 11.431621 [I 2022-10-10 01:40:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 180/2304 [I 2022-10-10 01:40:16] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:23 [I 2022-10-10 01:40:16] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00357 [I 2022-10-10 01:40:16] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-10 01:40:16] ...optimizers/trijoint.py.151: total_norm: 38.038063 [I 2022-10-10 01:40:25] ...trap/engines/engine.py.224: train: epoch 62 | batch 190/2304 [I 2022-10-10 01:40:25] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:04 [I 2022-10-10 01:40:25] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00327 [I 2022-10-10 01:40:25] ...trap/engines/engine.py.230: loss: 0.18256 [I 2022-10-10 01:40:25] ...optimizers/trijoint.py.151: total_norm: 12.575530 [I 2022-10-10 01:40:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 200/2304 [I 2022-10-10 01:40:33] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:55 [I 2022-10-10 01:40:33] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00332 [I 2022-10-10 01:40:33] ...trap/engines/engine.py.230: loss: 0.22886 [I 2022-10-10 01:40:33] ...optimizers/trijoint.py.151: total_norm: 27.234877 [I 2022-10-10 01:40:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 210/2304 [I 2022-10-10 01:40:42] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:35 [I 2022-10-10 01:40:42] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00320 [I 2022-10-10 01:40:42] ...trap/engines/engine.py.230: loss: 0.10773 [I 2022-10-10 01:40:42] ...optimizers/trijoint.py.151: total_norm: 11.049678 [I 2022-10-10 01:40:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 220/2304 [I 2022-10-10 01:40:50] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:33 [I 2022-10-10 01:40:50] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00332 [I 2022-10-10 01:40:50] ...trap/engines/engine.py.230: loss: 0.13540 [I 2022-10-10 01:40:50] ...optimizers/trijoint.py.151: total_norm: 14.170238 [I 2022-10-10 01:40:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 230/2304 [I 2022-10-10 01:40:59] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:29 [I 2022-10-10 01:40:59] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00341 [I 2022-10-10 01:40:59] ...trap/engines/engine.py.230: loss: 0.14093 [I 2022-10-10 01:40:59] ...optimizers/trijoint.py.151: total_norm: 13.719513 [I 2022-10-10 01:41:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 240/2304 [I 2022-10-10 01:41:07] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:11 [I 2022-10-10 01:41:07] ...trap/engines/engine.py.229: process: 0.83656 | load: 0.00328 [I 2022-10-10 01:41:07] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-10 01:41:07] ...optimizers/trijoint.py.151: total_norm: 13.061975 [I 2022-10-10 01:41:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 250/2304 [I 2022-10-10 01:41:16] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:03 [I 2022-10-10 01:41:16] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00321 [I 2022-10-10 01:41:16] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-10 01:41:16] ...optimizers/trijoint.py.151: total_norm: 11.597853 [I 2022-10-10 01:41:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 260/2304 [I 2022-10-10 01:41:24] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:12 [I 2022-10-10 01:41:24] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00318 [I 2022-10-10 01:41:24] ...trap/engines/engine.py.230: loss: 0.09362 [I 2022-10-10 01:41:24] ...optimizers/trijoint.py.151: total_norm: 28.252312 [I 2022-10-10 01:41:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 270/2304 [I 2022-10-10 01:41:33] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:56 [I 2022-10-10 01:41:33] ...trap/engines/engine.py.229: process: 0.84732 | load: 0.00355 [I 2022-10-10 01:41:33] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-10 01:41:33] ...optimizers/trijoint.py.151: total_norm: 31.274157 [I 2022-10-10 01:41:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 280/2304 [I 2022-10-10 01:41:41] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:50 [I 2022-10-10 01:41:41] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00344 [I 2022-10-10 01:41:41] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-10 01:41:41] ...optimizers/trijoint.py.151: total_norm: 11.483319 [I 2022-10-10 01:41:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 290/2304 [I 2022-10-10 01:41:50] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:35 [I 2022-10-10 01:41:50] ...trap/engines/engine.py.229: process: 0.83015 | load: 0.00316 [I 2022-10-10 01:41:50] ...trap/engines/engine.py.230: loss: 0.17197 [I 2022-10-10 01:41:50] ...optimizers/trijoint.py.151: total_norm: 13.651472 [I 2022-10-10 01:41:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 300/2304 [I 2022-10-10 01:41:58] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:30 [I 2022-10-10 01:41:58] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00312 [I 2022-10-10 01:41:58] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-10 01:41:58] ...optimizers/trijoint.py.151: total_norm: 11.211179 [I 2022-10-10 01:42:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 310/2304 [I 2022-10-10 01:42:07] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:20 [I 2022-10-10 01:42:07] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00312 [I 2022-10-10 01:42:07] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-10 01:42:07] ...optimizers/trijoint.py.151: total_norm: 20.029428 [I 2022-10-10 01:42:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 320/2304 [I 2022-10-10 01:42:16] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:20 [I 2022-10-10 01:42:16] ...trap/engines/engine.py.229: process: 0.85979 | load: 0.00326 [I 2022-10-10 01:42:16] ...trap/engines/engine.py.230: loss: 0.12104 [I 2022-10-10 01:42:16] ...optimizers/trijoint.py.151: total_norm: 14.125294 [I 2022-10-10 01:42:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 330/2304 [I 2022-10-10 01:42:24] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:05 [I 2022-10-10 01:42:24] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00314 [I 2022-10-10 01:42:24] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-10 01:42:24] ...optimizers/trijoint.py.151: total_norm: 13.795651 [I 2022-10-10 01:42:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 340/2304 [I 2022-10-10 01:42:33] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:55 [I 2022-10-10 01:42:33] ...trap/engines/engine.py.229: process: 0.86378 | load: 0.00303 [I 2022-10-10 01:42:33] ...trap/engines/engine.py.230: loss: 0.11750 [I 2022-10-10 01:42:33] ...optimizers/trijoint.py.151: total_norm: 20.731800 [I 2022-10-10 01:42:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 350/2304 [I 2022-10-10 01:42:41] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:32 [I 2022-10-10 01:42:41] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00317 [I 2022-10-10 01:42:41] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-10 01:42:41] ...optimizers/trijoint.py.151: total_norm: 15.243796 [I 2022-10-10 01:42:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 360/2304 [I 2022-10-10 01:42:50] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:25 [I 2022-10-10 01:42:50] ...trap/engines/engine.py.229: process: 0.83522 | load: 0.00297 [I 2022-10-10 01:42:50] ...trap/engines/engine.py.230: loss: 0.08760 [I 2022-10-10 01:42:50] ...optimizers/trijoint.py.151: total_norm: 7.519567 [I 2022-10-10 01:42:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 370/2304 [I 2022-10-10 01:42:58] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:11 [I 2022-10-10 01:42:58] ...trap/engines/engine.py.229: process: 0.84496 | load: 0.00295 [I 2022-10-10 01:42:58] ...trap/engines/engine.py.230: loss: 0.15384 [I 2022-10-10 01:42:58] ...optimizers/trijoint.py.151: total_norm: 13.053775 [I 2022-10-10 01:43:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 380/2304 [I 2022-10-10 01:43:07] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:15 [I 2022-10-10 01:43:07] ...trap/engines/engine.py.229: process: 0.85337 | load: 0.00306 [I 2022-10-10 01:43:07] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-10 01:43:07] ...optimizers/trijoint.py.151: total_norm: 8.366053 [I 2022-10-10 01:43:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 390/2304 [I 2022-10-10 01:43:15] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:27:06 [I 2022-10-10 01:43:15] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00299 [I 2022-10-10 01:43:15] ...trap/engines/engine.py.230: loss: 0.11387 [I 2022-10-10 01:43:15] ...optimizers/trijoint.py.151: total_norm: 13.123799 [I 2022-10-10 01:43:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 400/2304 [I 2022-10-10 01:43:24] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:58 [I 2022-10-10 01:43:24] ...trap/engines/engine.py.229: process: 0.85485 | load: 0.00299 [I 2022-10-10 01:43:24] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-10 01:43:24] ...optimizers/trijoint.py.151: total_norm: 13.346076 [I 2022-10-10 01:43:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 410/2304 [I 2022-10-10 01:43:32] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:44 [I 2022-10-10 01:43:32] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00317 [I 2022-10-10 01:43:32] ...trap/engines/engine.py.230: loss: 0.14423 [I 2022-10-10 01:43:32] ...optimizers/trijoint.py.151: total_norm: 18.587538 [I 2022-10-10 01:43:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 420/2304 [I 2022-10-10 01:43:41] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:31 [I 2022-10-10 01:43:41] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00304 [I 2022-10-10 01:43:41] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-10 01:43:41] ...optimizers/trijoint.py.151: total_norm: 8.365499 [I 2022-10-10 01:43:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 430/2304 [I 2022-10-10 01:43:49] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:31 [I 2022-10-10 01:43:49] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00300 [I 2022-10-10 01:43:49] ...trap/engines/engine.py.230: loss: 0.11197 [I 2022-10-10 01:43:49] ...optimizers/trijoint.py.151: total_norm: 8.564939 [I 2022-10-10 01:43:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 440/2304 [I 2022-10-10 01:43:58] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:20 [I 2022-10-10 01:43:58] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00299 [I 2022-10-10 01:43:58] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-10 01:43:58] ...optimizers/trijoint.py.151: total_norm: 12.927775 [I 2022-10-10 01:44:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 450/2304 [I 2022-10-10 01:44:06] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:14 [I 2022-10-10 01:44:06] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00300 [I 2022-10-10 01:44:06] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-10 01:44:06] ...optimizers/trijoint.py.151: total_norm: 13.709710 [I 2022-10-10 01:44:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 460/2304 [I 2022-10-10 01:44:15] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:58 [I 2022-10-10 01:44:15] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00300 [I 2022-10-10 01:44:15] ...trap/engines/engine.py.230: loss: 0.11308 [I 2022-10-10 01:44:15] ...optimizers/trijoint.py.151: total_norm: 8.464009 [I 2022-10-10 01:44:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 470/2304 [I 2022-10-10 01:44:23] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:49 [I 2022-10-10 01:44:23] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00300 [I 2022-10-10 01:44:23] ...trap/engines/engine.py.230: loss: 0.17087 [I 2022-10-10 01:44:23] ...optimizers/trijoint.py.151: total_norm: 10.416280 [I 2022-10-10 01:44:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 480/2304 [I 2022-10-10 01:44:32] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:45 [I 2022-10-10 01:44:32] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00303 [I 2022-10-10 01:44:32] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-10 01:44:32] ...optimizers/trijoint.py.151: total_norm: 9.725722 [I 2022-10-10 01:44:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 490/2304 [I 2022-10-10 01:44:40] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:28 [I 2022-10-10 01:44:40] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00301 [I 2022-10-10 01:44:40] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-10 01:44:40] ...optimizers/trijoint.py.151: total_norm: 18.305227 [I 2022-10-10 01:44:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 500/2304 [I 2022-10-10 01:44:49] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:22 [I 2022-10-10 01:44:49] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00297 [I 2022-10-10 01:44:49] ...trap/engines/engine.py.230: loss: 0.16419 [I 2022-10-10 01:44:49] ...optimizers/trijoint.py.151: total_norm: 17.381420 [I 2022-10-10 01:44:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 510/2304 [I 2022-10-10 01:44:57] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:28 [I 2022-10-10 01:44:57] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00309 [I 2022-10-10 01:44:57] ...trap/engines/engine.py.230: loss: 0.15357 [I 2022-10-10 01:44:57] ...optimizers/trijoint.py.151: total_norm: 10.398806 [I 2022-10-10 01:45:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 520/2304 [I 2022-10-10 01:45:06] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:16 [I 2022-10-10 01:45:06] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00313 [I 2022-10-10 01:45:06] ...trap/engines/engine.py.230: loss: 0.11279 [I 2022-10-10 01:45:06] ...optimizers/trijoint.py.151: total_norm: 32.039478 [I 2022-10-10 01:45:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 530/2304 [I 2022-10-10 01:45:14] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:15 [I 2022-10-10 01:45:14] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00310 [I 2022-10-10 01:45:14] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-10 01:45:14] ...optimizers/trijoint.py.151: total_norm: 25.912495 [I 2022-10-10 01:45:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 540/2304 [I 2022-10-10 01:45:23] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:02 [I 2022-10-10 01:45:23] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00310 [I 2022-10-10 01:45:23] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-10 01:45:23] ...optimizers/trijoint.py.151: total_norm: 18.025658 [I 2022-10-10 01:45:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 550/2304 [I 2022-10-10 01:45:31] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:03 [I 2022-10-10 01:45:31] ...trap/engines/engine.py.229: process: 0.87076 | load: 0.00312 [I 2022-10-10 01:45:31] ...trap/engines/engine.py.230: loss: 0.13956 [I 2022-10-10 01:45:31] ...optimizers/trijoint.py.151: total_norm: 16.251726 [I 2022-10-10 01:45:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 560/2304 [I 2022-10-10 01:45:40] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:49 [I 2022-10-10 01:45:40] ...trap/engines/engine.py.229: process: 0.85888 | load: 0.00317 [I 2022-10-10 01:45:40] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-10 01:45:40] ...optimizers/trijoint.py.151: total_norm: 46.215233 [I 2022-10-10 01:45:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 570/2304 [I 2022-10-10 01:45:48] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:44 [I 2022-10-10 01:45:48] ...trap/engines/engine.py.229: process: 0.85777 | load: 0.00328 [I 2022-10-10 01:45:48] ...trap/engines/engine.py.230: loss: 0.09653 [I 2022-10-10 01:45:48] ...optimizers/trijoint.py.151: total_norm: 14.641618 [I 2022-10-10 01:45:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 580/2304 [I 2022-10-10 01:45:57] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:34 [I 2022-10-10 01:45:57] ...trap/engines/engine.py.229: process: 0.85812 | load: 0.00315 [I 2022-10-10 01:45:57] ...trap/engines/engine.py.230: loss: 0.11264 [I 2022-10-10 01:45:57] ...optimizers/trijoint.py.151: total_norm: 14.004528 [I 2022-10-10 01:46:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 590/2304 [I 2022-10-10 01:46:06] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:28 [I 2022-10-10 01:46:06] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00313 [I 2022-10-10 01:46:06] ...trap/engines/engine.py.230: loss: 0.19147 [I 2022-10-10 01:46:06] ...optimizers/trijoint.py.151: total_norm: 19.985535 [I 2022-10-10 01:46:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 600/2304 [I 2022-10-10 01:46:14] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:14 [I 2022-10-10 01:46:14] ...trap/engines/engine.py.229: process: 0.86543 | load: 0.00314 [I 2022-10-10 01:46:14] ...trap/engines/engine.py.230: loss: 0.19417 [I 2022-10-10 01:46:14] ...optimizers/trijoint.py.151: total_norm: 12.501957 [I 2022-10-10 01:46:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 610/2304 [I 2022-10-10 01:46:23] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:10 [I 2022-10-10 01:46:23] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00315 [I 2022-10-10 01:46:23] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-10 01:46:23] ...optimizers/trijoint.py.151: total_norm: 12.982033 [I 2022-10-10 01:46:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 620/2304 [I 2022-10-10 01:46:31] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:03 [I 2022-10-10 01:46:31] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00309 [I 2022-10-10 01:46:31] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-10 01:46:31] ...optimizers/trijoint.py.151: total_norm: 23.095152 [I 2022-10-10 01:46:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 630/2304 [I 2022-10-10 01:46:40] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:56 [I 2022-10-10 01:46:40] ...trap/engines/engine.py.229: process: 0.86857 | load: 0.00312 [I 2022-10-10 01:46:40] ...trap/engines/engine.py.230: loss: 0.19765 [I 2022-10-10 01:46:40] ...optimizers/trijoint.py.151: total_norm: 30.066549 [I 2022-10-10 01:46:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 640/2304 [I 2022-10-10 01:46:49] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:42 [I 2022-10-10 01:46:49] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00310 [I 2022-10-10 01:46:49] ...trap/engines/engine.py.230: loss: 0.10125 [I 2022-10-10 01:46:49] ...optimizers/trijoint.py.151: total_norm: 9.238205 [I 2022-10-10 01:46:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 650/2304 [I 2022-10-10 01:46:57] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:36 [I 2022-10-10 01:46:57] ...trap/engines/engine.py.229: process: 0.87437 | load: 0.00314 [I 2022-10-10 01:46:57] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-10 01:46:57] ...optimizers/trijoint.py.151: total_norm: 14.147719 [I 2022-10-10 01:47:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 660/2304 [I 2022-10-10 01:47:06] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:31 [I 2022-10-10 01:47:06] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00313 [I 2022-10-10 01:47:06] ...trap/engines/engine.py.230: loss: 0.12145 [I 2022-10-10 01:47:06] ...optimizers/trijoint.py.151: total_norm: 6.888098 [I 2022-10-10 01:47:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 670/2304 [I 2022-10-10 01:47:14] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:13 [I 2022-10-10 01:47:14] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00315 [I 2022-10-10 01:47:14] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-10 01:47:14] ...optimizers/trijoint.py.151: total_norm: 7.473198 [I 2022-10-10 01:47:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 680/2304 [I 2022-10-10 01:47:23] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:00 [I 2022-10-10 01:47:23] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00313 [I 2022-10-10 01:47:23] ...trap/engines/engine.py.230: loss: 0.26174 [I 2022-10-10 01:47:23] ...optimizers/trijoint.py.151: total_norm: 15.158197 [I 2022-10-10 01:47:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 690/2304 [I 2022-10-10 01:47:31] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:46 [I 2022-10-10 01:47:31] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00314 [I 2022-10-10 01:47:31] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-10 01:47:31] ...optimizers/trijoint.py.151: total_norm: 10.365180 [I 2022-10-10 01:47:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 700/2304 [I 2022-10-10 01:47:40] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:49 [I 2022-10-10 01:47:40] ...trap/engines/engine.py.229: process: 0.85967 | load: 0.00327 [I 2022-10-10 01:47:40] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-10 01:47:40] ...optimizers/trijoint.py.151: total_norm: 8.845871 [I 2022-10-10 01:47:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 710/2304 [I 2022-10-10 01:47:48] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:44 [I 2022-10-10 01:47:48] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00309 [I 2022-10-10 01:47:48] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-10 01:47:48] ...optimizers/trijoint.py.151: total_norm: 12.701784 [I 2022-10-10 01:47:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 720/2304 [I 2022-10-10 01:47:57] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:31 [I 2022-10-10 01:47:57] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00312 [I 2022-10-10 01:47:57] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-10 01:47:57] ...optimizers/trijoint.py.151: total_norm: 8.891536 [I 2022-10-10 01:48:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 730/2304 [I 2022-10-10 01:48:24] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:54:31 [I 2022-10-10 01:48:24] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00312 [I 2022-10-10 01:48:24] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-10 01:48:24] ...optimizers/trijoint.py.151: total_norm: 17.155657 [I 2022-10-10 01:48:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 740/2304 [I 2022-10-10 01:48:33] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:25:46 [I 2022-10-10 01:48:33] ...trap/engines/engine.py.229: process: 0.86578 | load: 0.00314 [I 2022-10-10 01:48:33] ...trap/engines/engine.py.230: loss: 0.12875 [I 2022-10-10 01:48:33] ...optimizers/trijoint.py.151: total_norm: 14.495334 [I 2022-10-10 01:48:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 750/2304 [I 2022-10-10 01:48:41] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:22:18 [I 2022-10-10 01:48:41] ...trap/engines/engine.py.229: process: 0.83175 | load: 0.00315 [I 2022-10-10 01:48:41] ...trap/engines/engine.py.230: loss: 0.18429 [I 2022-10-10 01:48:41] ...optimizers/trijoint.py.151: total_norm: 21.660120 [I 2022-10-10 01:48:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 760/2304 [I 2022-10-10 01:48:50] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:57 [I 2022-10-10 01:48:50] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00311 [I 2022-10-10 01:48:50] ...trap/engines/engine.py.230: loss: 0.21930 [I 2022-10-10 01:48:50] ...optimizers/trijoint.py.151: total_norm: 39.174389 [I 2022-10-10 01:48:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 770/2304 [I 2022-10-10 01:48:58] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:21:48 [I 2022-10-10 01:48:58] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00312 [I 2022-10-10 01:48:58] ...trap/engines/engine.py.230: loss: 0.17490 [I 2022-10-10 01:48:58] ...optimizers/trijoint.py.151: total_norm: 20.970047 [I 2022-10-10 01:49:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 780/2304 [I 2022-10-10 01:49:07] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:21:42 [I 2022-10-10 01:49:07] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00297 [I 2022-10-10 01:49:07] ...trap/engines/engine.py.230: loss: 0.13681 [I 2022-10-10 01:49:07] ...optimizers/trijoint.py.151: total_norm: 13.370393 [I 2022-10-10 01:49:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 790/2304 [I 2022-10-10 01:49:15] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:21:24 [I 2022-10-10 01:49:15] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00312 [I 2022-10-10 01:49:15] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-10 01:49:15] ...optimizers/trijoint.py.151: total_norm: 23.966036 [I 2022-10-10 01:49:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 800/2304 [I 2022-10-10 01:49:24] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:21:05 [I 2022-10-10 01:49:24] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00299 [I 2022-10-10 01:49:24] ...trap/engines/engine.py.230: loss: 0.13962 [I 2022-10-10 01:49:24] ...optimizers/trijoint.py.151: total_norm: 15.434788 [I 2022-10-10 01:49:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 810/2304 [I 2022-10-10 01:49:32] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:21:04 [I 2022-10-10 01:49:32] ...trap/engines/engine.py.229: process: 0.83689 | load: 0.00300 [I 2022-10-10 01:49:32] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-10 01:49:32] ...optimizers/trijoint.py.151: total_norm: 20.183317 [I 2022-10-10 01:49:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 820/2304 [I 2022-10-10 01:49:41] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:21:01 [I 2022-10-10 01:49:41] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00298 [I 2022-10-10 01:49:41] ...trap/engines/engine.py.230: loss: 0.13576 [I 2022-10-10 01:49:41] ...optimizers/trijoint.py.151: total_norm: 11.312199 [I 2022-10-10 01:49:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 830/2304 [I 2022-10-10 01:49:49] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:51 [I 2022-10-10 01:49:49] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00309 [I 2022-10-10 01:49:49] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-10 01:49:49] ...optimizers/trijoint.py.151: total_norm: 12.556378 [I 2022-10-10 01:49:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 840/2304 [I 2022-10-10 01:49:58] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:43 [I 2022-10-10 01:49:58] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00297 [I 2022-10-10 01:49:58] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-10 01:49:58] ...optimizers/trijoint.py.151: total_norm: 19.047487 [I 2022-10-10 01:50:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 850/2304 [I 2022-10-10 01:50:06] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:31 [I 2022-10-10 01:50:06] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00302 [I 2022-10-10 01:50:06] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-10 01:50:06] ...optimizers/trijoint.py.151: total_norm: 23.132584 [I 2022-10-10 01:50:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 860/2304 [I 2022-10-10 01:50:15] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:20 [I 2022-10-10 01:50:15] ...trap/engines/engine.py.229: process: 0.85861 | load: 0.00298 [I 2022-10-10 01:50:15] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-10 01:50:15] ...optimizers/trijoint.py.151: total_norm: 12.901731 [I 2022-10-10 01:50:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 870/2304 [I 2022-10-10 01:50:23] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:20:11 [I 2022-10-10 01:50:23] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00300 [I 2022-10-10 01:50:23] ...trap/engines/engine.py.230: loss: 0.09694 [I 2022-10-10 01:50:23] ...optimizers/trijoint.py.151: total_norm: 9.116194 [I 2022-10-10 01:50:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 880/2304 [I 2022-10-10 01:50:32] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:00 [I 2022-10-10 01:50:32] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00304 [I 2022-10-10 01:50:32] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-10 01:50:32] ...optimizers/trijoint.py.151: total_norm: 13.113516 [I 2022-10-10 01:50:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 890/2304 [I 2022-10-10 01:50:40] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:56 [I 2022-10-10 01:50:40] ...trap/engines/engine.py.229: process: 0.84003 | load: 0.00316 [I 2022-10-10 01:50:40] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-10 01:50:40] ...optimizers/trijoint.py.151: total_norm: 15.056029 [I 2022-10-10 01:50:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 900/2304 [I 2022-10-10 01:50:49] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:48 [I 2022-10-10 01:50:49] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00303 [I 2022-10-10 01:50:49] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-10 01:50:49] ...optimizers/trijoint.py.151: total_norm: 21.555557 [I 2022-10-10 01:50:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 910/2304 [I 2022-10-10 01:50:57] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:45 [I 2022-10-10 01:50:57] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00301 [I 2022-10-10 01:50:57] ...trap/engines/engine.py.230: loss: 0.15628 [I 2022-10-10 01:50:57] ...optimizers/trijoint.py.151: total_norm: 82.773804 [I 2022-10-10 01:51:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 920/2304 [I 2022-10-10 01:51:06] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:31 [I 2022-10-10 01:51:06] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00299 [I 2022-10-10 01:51:06] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-10 01:51:06] ...optimizers/trijoint.py.151: total_norm: 29.366451 [I 2022-10-10 01:51:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 930/2304 [I 2022-10-10 01:51:14] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:19 [I 2022-10-10 01:51:14] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00297 [I 2022-10-10 01:51:14] ...trap/engines/engine.py.230: loss: 0.14177 [I 2022-10-10 01:51:14] ...optimizers/trijoint.py.151: total_norm: 26.470148 [I 2022-10-10 01:51:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 940/2304 [I 2022-10-10 01:51:23] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:20 [I 2022-10-10 01:51:23] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00306 [I 2022-10-10 01:51:23] ...trap/engines/engine.py.230: loss: 0.14139 [I 2022-10-10 01:51:23] ...optimizers/trijoint.py.151: total_norm: 20.107285 [I 2022-10-10 01:51:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 950/2304 [I 2022-10-10 01:51:32] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:24 [I 2022-10-10 01:51:32] ...trap/engines/engine.py.229: process: 0.87220 | load: 0.00312 [I 2022-10-10 01:51:32] ...trap/engines/engine.py.230: loss: 0.14108 [I 2022-10-10 01:51:32] ...optimizers/trijoint.py.151: total_norm: 13.900963 [I 2022-10-10 01:51:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 960/2304 [I 2022-10-10 01:51:40] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:12 [I 2022-10-10 01:51:40] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00313 [I 2022-10-10 01:51:40] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-10 01:51:40] ...optimizers/trijoint.py.151: total_norm: 60.177048 [I 2022-10-10 01:51:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 970/2304 [I 2022-10-10 01:51:49] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:00 [I 2022-10-10 01:51:49] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00313 [I 2022-10-10 01:51:49] ...trap/engines/engine.py.230: loss: 0.15731 [I 2022-10-10 01:51:49] ...optimizers/trijoint.py.151: total_norm: 17.472496 [I 2022-10-10 01:51:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 980/2304 [I 2022-10-10 01:51:57] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:46 [I 2022-10-10 01:51:57] ...trap/engines/engine.py.229: process: 0.86186 | load: 0.00314 [I 2022-10-10 01:51:57] ...trap/engines/engine.py.230: loss: 0.15156 [I 2022-10-10 01:51:57] ...optimizers/trijoint.py.151: total_norm: 17.579638 [I 2022-10-10 01:52:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 990/2304 [I 2022-10-10 01:52:06] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:40 [I 2022-10-10 01:52:06] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00339 [I 2022-10-10 01:52:06] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-10 01:52:06] ...optimizers/trijoint.py.151: total_norm: 19.609426 [I 2022-10-10 01:52:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1000/2304 [I 2022-10-10 01:52:14] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:42 [I 2022-10-10 01:52:14] ...trap/engines/engine.py.229: process: 0.87648 | load: 0.00321 [I 2022-10-10 01:52:14] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-10 01:52:14] ...optimizers/trijoint.py.151: total_norm: 13.792444 [I 2022-10-10 01:52:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1010/2304 [I 2022-10-10 01:52:23] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:30 [I 2022-10-10 01:52:23] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00313 [I 2022-10-10 01:52:23] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-10 01:52:23] ...optimizers/trijoint.py.151: total_norm: 15.011597 [I 2022-10-10 01:52:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 1020/2304 [I 2022-10-10 01:52:32] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:16 [I 2022-10-10 01:52:32] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00315 [I 2022-10-10 01:52:32] ...trap/engines/engine.py.230: loss: 0.14124 [I 2022-10-10 01:52:32] ...optimizers/trijoint.py.151: total_norm: 17.554684 [I 2022-10-10 01:52:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1030/2304 [I 2022-10-10 01:52:40] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:18:06 [I 2022-10-10 01:52:40] ...trap/engines/engine.py.229: process: 0.86576 | load: 0.00308 [I 2022-10-10 01:52:40] ...trap/engines/engine.py.230: loss: 0.16907 [I 2022-10-10 01:52:40] ...optimizers/trijoint.py.151: total_norm: 47.116657 [I 2022-10-10 01:52:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 1040/2304 [I 2022-10-10 01:52:49] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:55 [I 2022-10-10 01:52:49] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00315 [I 2022-10-10 01:52:49] ...trap/engines/engine.py.230: loss: 0.16512 [I 2022-10-10 01:52:49] ...optimizers/trijoint.py.151: total_norm: 7.706488 [I 2022-10-10 01:52:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 1050/2304 [I 2022-10-10 01:52:57] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:53 [I 2022-10-10 01:52:57] ...trap/engines/engine.py.229: process: 0.86243 | load: 0.00312 [I 2022-10-10 01:52:57] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-10 01:52:57] ...optimizers/trijoint.py.151: total_norm: 8.049473 [I 2022-10-10 01:53:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 1060/2304 [I 2022-10-10 01:53:06] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:49 [I 2022-10-10 01:53:06] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00315 [I 2022-10-10 01:53:06] ...trap/engines/engine.py.230: loss: 0.16321 [I 2022-10-10 01:53:06] ...optimizers/trijoint.py.151: total_norm: 30.948803 [I 2022-10-10 01:53:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1070/2304 [I 2022-10-10 01:53:14] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:37 [I 2022-10-10 01:53:14] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00321 [I 2022-10-10 01:53:14] ...trap/engines/engine.py.230: loss: 0.13232 [I 2022-10-10 01:53:14] ...optimizers/trijoint.py.151: total_norm: 28.016731 [I 2022-10-10 01:53:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1080/2304 [I 2022-10-10 01:53:23] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:32 [I 2022-10-10 01:53:23] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00314 [I 2022-10-10 01:53:23] ...trap/engines/engine.py.230: loss: 0.23406 [I 2022-10-10 01:53:23] ...optimizers/trijoint.py.151: total_norm: 20.409000 [I 2022-10-10 01:53:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 1090/2304 [I 2022-10-10 01:53:32] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:19 [I 2022-10-10 01:53:32] ...trap/engines/engine.py.229: process: 0.85625 | load: 0.00311 [I 2022-10-10 01:53:32] ...trap/engines/engine.py.230: loss: 0.18712 [I 2022-10-10 01:53:32] ...optimizers/trijoint.py.151: total_norm: 24.671131 [I 2022-10-10 01:53:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1100/2304 [I 2022-10-10 01:53:40] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:17:10 [I 2022-10-10 01:53:40] ...trap/engines/engine.py.229: process: 0.86107 | load: 0.00313 [I 2022-10-10 01:53:40] ...trap/engines/engine.py.230: loss: 0.10371 [I 2022-10-10 01:53:40] ...optimizers/trijoint.py.151: total_norm: 12.877045 [I 2022-10-10 01:53:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 1110/2304 [I 2022-10-10 01:53:49] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:17:05 [I 2022-10-10 01:53:49] ...trap/engines/engine.py.229: process: 0.87000 | load: 0.00323 [I 2022-10-10 01:53:49] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-10 01:53:49] ...optimizers/trijoint.py.151: total_norm: 16.082911 [I 2022-10-10 01:53:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 1120/2304 [I 2022-10-10 01:53:57] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:52 [I 2022-10-10 01:53:57] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00317 [I 2022-10-10 01:53:57] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-10 01:53:57] ...optimizers/trijoint.py.151: total_norm: 21.951923 [I 2022-10-10 01:54:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 1130/2304 [I 2022-10-10 01:54:06] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:43 [I 2022-10-10 01:54:06] ...trap/engines/engine.py.229: process: 0.84660 | load: 0.00313 [I 2022-10-10 01:54:06] ...trap/engines/engine.py.230: loss: 0.14774 [I 2022-10-10 01:54:06] ...optimizers/trijoint.py.151: total_norm: 15.388126 [I 2022-10-10 01:54:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 1140/2304 [I 2022-10-10 01:54:15] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:31 [I 2022-10-10 01:54:15] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00313 [I 2022-10-10 01:54:15] ...trap/engines/engine.py.230: loss: 0.18918 [I 2022-10-10 01:54:15] ...optimizers/trijoint.py.151: total_norm: 17.036493 [I 2022-10-10 01:54:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1150/2304 [I 2022-10-10 01:54:23] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:21 [I 2022-10-10 01:54:23] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00305 [I 2022-10-10 01:54:23] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-10 01:54:23] ...optimizers/trijoint.py.151: total_norm: 11.941303 [I 2022-10-10 01:54:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 1160/2304 [I 2022-10-10 01:54:32] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:09 [I 2022-10-10 01:54:32] ...trap/engines/engine.py.229: process: 0.82797 | load: 0.00301 [I 2022-10-10 01:54:32] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-10 01:54:32] ...optimizers/trijoint.py.151: total_norm: 7.246331 [I 2022-10-10 01:54:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1170/2304 [I 2022-10-10 01:54:40] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:08 [I 2022-10-10 01:54:40] ...trap/engines/engine.py.229: process: 0.86532 | load: 0.00311 [I 2022-10-10 01:54:40] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-10 01:54:40] ...optimizers/trijoint.py.151: total_norm: 14.306185 [I 2022-10-10 01:54:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 1180/2304 [I 2022-10-10 01:54:49] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:56 [I 2022-10-10 01:54:49] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00312 [I 2022-10-10 01:54:49] ...trap/engines/engine.py.230: loss: 0.19495 [I 2022-10-10 01:54:49] ...optimizers/trijoint.py.151: total_norm: 16.768127 [I 2022-10-10 01:54:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 1190/2304 [I 2022-10-10 01:54:57] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:52 [I 2022-10-10 01:54:57] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00322 [I 2022-10-10 01:54:57] ...trap/engines/engine.py.230: loss: 0.15237 [I 2022-10-10 01:54:57] ...optimizers/trijoint.py.151: total_norm: 22.667648 [I 2022-10-10 01:55:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 1200/2304 [I 2022-10-10 01:55:06] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:35 [I 2022-10-10 01:55:06] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00313 [I 2022-10-10 01:55:06] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-10 01:55:06] ...optimizers/trijoint.py.151: total_norm: 12.010843 [I 2022-10-10 01:55:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1210/2304 [I 2022-10-10 01:55:14] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:35 [I 2022-10-10 01:55:14] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00315 [I 2022-10-10 01:55:14] ...trap/engines/engine.py.230: loss: 0.16380 [I 2022-10-10 01:55:14] ...optimizers/trijoint.py.151: total_norm: 11.457752 [I 2022-10-10 01:55:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1220/2304 [I 2022-10-10 01:55:23] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:23 [I 2022-10-10 01:55:23] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00310 [I 2022-10-10 01:55:23] ...trap/engines/engine.py.230: loss: 0.12149 [I 2022-10-10 01:55:23] ...optimizers/trijoint.py.151: total_norm: 17.362925 [I 2022-10-10 01:55:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 1230/2304 [I 2022-10-10 01:55:31] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:17 [I 2022-10-10 01:55:31] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00313 [I 2022-10-10 01:55:31] ...trap/engines/engine.py.230: loss: 0.11487 [I 2022-10-10 01:55:31] ...optimizers/trijoint.py.151: total_norm: 16.542686 [I 2022-10-10 01:55:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1240/2304 [I 2022-10-10 01:55:40] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:03 [I 2022-10-10 01:55:40] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00315 [I 2022-10-10 01:55:40] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-10 01:55:40] ...optimizers/trijoint.py.151: total_norm: 11.408267 [I 2022-10-10 01:55:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 1250/2304 [I 2022-10-10 01:55:48] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:14:55 [I 2022-10-10 01:55:48] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00293 [I 2022-10-10 01:55:48] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-10 01:55:48] ...optimizers/trijoint.py.151: total_norm: 15.476581 [I 2022-10-10 01:55:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 1260/2304 [I 2022-10-10 01:55:57] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:43 [I 2022-10-10 01:55:57] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00295 [I 2022-10-10 01:55:57] ...trap/engines/engine.py.230: loss: 0.19176 [I 2022-10-10 01:55:57] ...optimizers/trijoint.py.151: total_norm: 12.674475 [I 2022-10-10 01:56:05] ...trap/engines/engine.py.224: train: epoch 62 | batch 1270/2304 [I 2022-10-10 01:56:05] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:14:33 [I 2022-10-10 01:56:05] ...trap/engines/engine.py.229: process: 0.83064 | load: 0.00300 [I 2022-10-10 01:56:05] ...trap/engines/engine.py.230: loss: 0.17723 [I 2022-10-10 01:56:05] ...optimizers/trijoint.py.151: total_norm: 15.557487 [I 2022-10-10 01:56:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1280/2304 [I 2022-10-10 01:56:14] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:29 [I 2022-10-10 01:56:14] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00302 [I 2022-10-10 01:56:14] ...trap/engines/engine.py.230: loss: 0.09676 [I 2022-10-10 01:56:14] ...optimizers/trijoint.py.151: total_norm: 9.936075 [I 2022-10-10 01:56:22] ...trap/engines/engine.py.224: train: epoch 62 | batch 1290/2304 [I 2022-10-10 01:56:22] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:14:18 [I 2022-10-10 01:56:22] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00303 [I 2022-10-10 01:56:22] ...trap/engines/engine.py.230: loss: 0.21884 [I 2022-10-10 01:56:22] ...optimizers/trijoint.py.151: total_norm: 12.190079 [I 2022-10-10 01:56:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 1300/2304 [I 2022-10-10 01:56:31] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:14:10 [I 2022-10-10 01:56:31] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00302 [I 2022-10-10 01:56:31] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-10 01:56:31] ...optimizers/trijoint.py.151: total_norm: 17.852196 [I 2022-10-10 01:56:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 1310/2304 [I 2022-10-10 01:56:39] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:05 [I 2022-10-10 01:56:39] ...trap/engines/engine.py.229: process: 0.86486 | load: 0.00298 [I 2022-10-10 01:56:39] ...trap/engines/engine.py.230: loss: 0.14814 [I 2022-10-10 01:56:39] ...optimizers/trijoint.py.151: total_norm: 18.613277 [I 2022-10-10 01:56:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 1320/2304 [I 2022-10-10 01:56:48] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:13:51 [I 2022-10-10 01:56:48] ...trap/engines/engine.py.229: process: 0.83643 | load: 0.00301 [I 2022-10-10 01:56:48] ...trap/engines/engine.py.230: loss: 0.13949 [I 2022-10-10 01:56:48] ...optimizers/trijoint.py.151: total_norm: 10.662587 [I 2022-10-10 01:56:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1330/2304 [I 2022-10-10 01:56:56] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:43 [I 2022-10-10 01:56:56] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00299 [I 2022-10-10 01:56:56] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-10 01:56:56] ...optimizers/trijoint.py.151: total_norm: 17.325079 [I 2022-10-10 01:57:05] ...trap/engines/engine.py.224: train: epoch 62 | batch 1340/2304 [I 2022-10-10 01:57:05] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:36 [I 2022-10-10 01:57:05] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00300 [I 2022-10-10 01:57:05] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-10 01:57:05] ...optimizers/trijoint.py.151: total_norm: 14.350114 [I 2022-10-10 01:57:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1350/2304 [I 2022-10-10 01:57:13] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:26 [I 2022-10-10 01:57:13] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00298 [I 2022-10-10 01:57:13] ...trap/engines/engine.py.230: loss: 0.11702 [I 2022-10-10 01:57:13] ...optimizers/trijoint.py.151: total_norm: 7.899454 [I 2022-10-10 01:57:22] ...trap/engines/engine.py.224: train: epoch 62 | batch 1360/2304 [I 2022-10-10 01:57:22] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:18 [I 2022-10-10 01:57:22] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00303 [I 2022-10-10 01:57:22] ...trap/engines/engine.py.230: loss: 0.10569 [I 2022-10-10 01:57:22] ...optimizers/trijoint.py.151: total_norm: 18.075863 [I 2022-10-10 01:57:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1370/2304 [I 2022-10-10 01:57:30] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:13:06 [I 2022-10-10 01:57:30] ...trap/engines/engine.py.229: process: 0.84475 | load: 0.00301 [I 2022-10-10 01:57:30] ...trap/engines/engine.py.230: loss: 0.16486 [I 2022-10-10 01:57:30] ...optimizers/trijoint.py.151: total_norm: 73.361420 [I 2022-10-10 01:57:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 1380/2304 [I 2022-10-10 01:57:39] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:13:00 [I 2022-10-10 01:57:39] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00301 [I 2022-10-10 01:57:39] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-10 01:57:39] ...optimizers/trijoint.py.151: total_norm: 13.594875 [I 2022-10-10 01:57:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1390/2304 [I 2022-10-10 01:57:47] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:51 [I 2022-10-10 01:57:47] ...trap/engines/engine.py.229: process: 0.83565 | load: 0.00302 [I 2022-10-10 01:57:47] ...trap/engines/engine.py.230: loss: 0.14794 [I 2022-10-10 01:57:47] ...optimizers/trijoint.py.151: total_norm: 16.101431 [I 2022-10-10 01:57:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1400/2304 [I 2022-10-10 01:57:56] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:41 [I 2022-10-10 01:57:56] ...trap/engines/engine.py.229: process: 0.83498 | load: 0.00311 [I 2022-10-10 01:57:56] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-10 01:57:56] ...optimizers/trijoint.py.151: total_norm: 14.760754 [I 2022-10-10 01:58:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1410/2304 [I 2022-10-10 01:58:04] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:44 [I 2022-10-10 01:58:04] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00310 [I 2022-10-10 01:58:04] ...trap/engines/engine.py.230: loss: 0.17674 [I 2022-10-10 01:58:04] ...optimizers/trijoint.py.151: total_norm: 29.447277 [I 2022-10-10 01:58:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1420/2304 [I 2022-10-10 01:58:13] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:37 [I 2022-10-10 01:58:13] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00314 [I 2022-10-10 01:58:13] ...trap/engines/engine.py.230: loss: 0.15003 [I 2022-10-10 01:58:13] ...optimizers/trijoint.py.151: total_norm: 22.710615 [I 2022-10-10 01:58:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1430/2304 [I 2022-10-10 01:58:21] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:12:28 [I 2022-10-10 01:58:21] ...trap/engines/engine.py.229: process: 0.85832 | load: 0.00324 [I 2022-10-10 01:58:21] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-10 01:58:21] ...optimizers/trijoint.py.151: total_norm: 11.336395 [I 2022-10-10 01:58:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1440/2304 [I 2022-10-10 01:58:30] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:12:16 [I 2022-10-10 01:58:30] ...trap/engines/engine.py.229: process: 0.85095 | load: 0.00310 [I 2022-10-10 01:58:30] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-10 01:58:30] ...optimizers/trijoint.py.151: total_norm: 12.308084 [I 2022-10-10 01:58:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1450/2304 [I 2022-10-10 01:58:38] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:12:05 [I 2022-10-10 01:58:38] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00310 [I 2022-10-10 01:58:38] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-10 01:58:38] ...optimizers/trijoint.py.151: total_norm: 19.931625 [I 2022-10-10 01:58:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1460/2304 [I 2022-10-10 01:58:47] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:12:01 [I 2022-10-10 01:58:47] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00317 [I 2022-10-10 01:58:47] ...trap/engines/engine.py.230: loss: 0.11903 [I 2022-10-10 01:58:47] ...optimizers/trijoint.py.151: total_norm: 17.565166 [I 2022-10-10 01:58:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1470/2304 [I 2022-10-10 01:58:56] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:49 [I 2022-10-10 01:58:56] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00317 [I 2022-10-10 01:58:56] ...trap/engines/engine.py.230: loss: 0.18493 [I 2022-10-10 01:58:56] ...optimizers/trijoint.py.151: total_norm: 29.700951 [I 2022-10-10 01:59:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1480/2304 [I 2022-10-10 01:59:04] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:43 [I 2022-10-10 01:59:04] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00310 [I 2022-10-10 01:59:04] ...trap/engines/engine.py.230: loss: 0.11522 [I 2022-10-10 01:59:04] ...optimizers/trijoint.py.151: total_norm: 14.290804 [I 2022-10-10 01:59:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1490/2304 [I 2022-10-10 01:59:13] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:34 [I 2022-10-10 01:59:13] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00312 [I 2022-10-10 01:59:13] ...trap/engines/engine.py.230: loss: 0.13321 [I 2022-10-10 01:59:13] ...optimizers/trijoint.py.151: total_norm: 18.278963 [I 2022-10-10 01:59:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1500/2304 [I 2022-10-10 01:59:21] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:11:26 [I 2022-10-10 01:59:21] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00312 [I 2022-10-10 01:59:21] ...trap/engines/engine.py.230: loss: 0.15984 [I 2022-10-10 01:59:21] ...optimizers/trijoint.py.151: total_norm: 16.817984 [I 2022-10-10 01:59:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1510/2304 [I 2022-10-10 01:59:30] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:11:13 [I 2022-10-10 01:59:30] ...trap/engines/engine.py.229: process: 0.84135 | load: 0.00312 [I 2022-10-10 01:59:30] ...trap/engines/engine.py.230: loss: 0.17126 [I 2022-10-10 01:59:30] ...optimizers/trijoint.py.151: total_norm: 74.125687 [I 2022-10-10 01:59:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1520/2304 [I 2022-10-10 01:59:38] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:11:11 [I 2022-10-10 01:59:38] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00310 [I 2022-10-10 01:59:38] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-10 01:59:38] ...optimizers/trijoint.py.151: total_norm: 8.282476 [I 2022-10-10 01:59:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1530/2304 [I 2022-10-10 01:59:47] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:11:02 [I 2022-10-10 01:59:47] ...trap/engines/engine.py.229: process: 0.86420 | load: 0.00313 [I 2022-10-10 01:59:47] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-10 01:59:47] ...optimizers/trijoint.py.151: total_norm: 31.231268 [I 2022-10-10 01:59:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 1540/2304 [I 2022-10-10 01:59:55] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:50 [I 2022-10-10 01:59:55] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00314 [I 2022-10-10 01:59:55] ...trap/engines/engine.py.230: loss: 0.14771 [I 2022-10-10 01:59:55] ...optimizers/trijoint.py.151: total_norm: 11.770146 [I 2022-10-10 02:00:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1550/2304 [I 2022-10-10 02:00:04] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:44 [I 2022-10-10 02:00:04] ...trap/engines/engine.py.229: process: 0.84447 | load: 0.00309 [I 2022-10-10 02:00:04] ...trap/engines/engine.py.230: loss: 0.15079 [I 2022-10-10 02:00:04] ...optimizers/trijoint.py.151: total_norm: 16.226801 [I 2022-10-10 02:00:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1560/2304 [I 2022-10-10 02:00:13] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:37 [I 2022-10-10 02:00:13] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00321 [I 2022-10-10 02:00:13] ...trap/engines/engine.py.230: loss: 0.13110 [I 2022-10-10 02:00:13] ...optimizers/trijoint.py.151: total_norm: 10.059559 [I 2022-10-10 02:00:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1570/2304 [I 2022-10-10 02:00:21] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:10:28 [I 2022-10-10 02:00:21] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00311 [I 2022-10-10 02:00:21] ...trap/engines/engine.py.230: loss: 0.19508 [I 2022-10-10 02:00:21] ...optimizers/trijoint.py.151: total_norm: 51.527672 [I 2022-10-10 02:00:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1580/2304 [I 2022-10-10 02:00:30] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:10:22 [I 2022-10-10 02:00:30] ...trap/engines/engine.py.229: process: 0.86982 | load: 0.00310 [I 2022-10-10 02:00:30] ...trap/engines/engine.py.230: loss: 0.15731 [I 2022-10-10 02:00:30] ...optimizers/trijoint.py.151: total_norm: 17.805017 [I 2022-10-10 02:00:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1590/2304 [I 2022-10-10 02:00:38] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:10:09 [I 2022-10-10 02:00:38] ...trap/engines/engine.py.229: process: 0.84916 | load: 0.00311 [I 2022-10-10 02:00:38] ...trap/engines/engine.py.230: loss: 0.14740 [I 2022-10-10 02:00:38] ...optimizers/trijoint.py.151: total_norm: 32.965115 [I 2022-10-10 02:00:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1600/2304 [I 2022-10-10 02:00:47] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:10:04 [I 2022-10-10 02:00:47] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00311 [I 2022-10-10 02:00:47] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-10 02:00:47] ...optimizers/trijoint.py.151: total_norm: 8.597460 [I 2022-10-10 02:00:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1610/2304 [I 2022-10-10 02:00:56] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:54 [I 2022-10-10 02:00:56] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00314 [I 2022-10-10 02:00:56] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-10 02:00:56] ...optimizers/trijoint.py.151: total_norm: 9.880630 [I 2022-10-10 02:01:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1620/2304 [I 2022-10-10 02:01:04] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:09:45 [I 2022-10-10 02:01:04] ...trap/engines/engine.py.229: process: 0.85266 | load: 0.00312 [I 2022-10-10 02:01:04] ...trap/engines/engine.py.230: loss: 0.15566 [I 2022-10-10 02:01:04] ...optimizers/trijoint.py.151: total_norm: 9.650736 [I 2022-10-10 02:01:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1630/2304 [I 2022-10-10 02:01:13] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:09:34 [I 2022-10-10 02:01:13] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00314 [I 2022-10-10 02:01:13] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-10 02:01:13] ...optimizers/trijoint.py.151: total_norm: 19.662491 [I 2022-10-10 02:01:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1640/2304 [I 2022-10-10 02:01:21] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:09:28 [I 2022-10-10 02:01:21] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00313 [I 2022-10-10 02:01:21] ...trap/engines/engine.py.230: loss: 0.14606 [I 2022-10-10 02:01:21] ...optimizers/trijoint.py.151: total_norm: 18.045893 [I 2022-10-10 02:01:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1650/2304 [I 2022-10-10 02:01:30] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:22 [I 2022-10-10 02:01:30] ...trap/engines/engine.py.229: process: 0.86528 | load: 0.00311 [I 2022-10-10 02:01:30] ...trap/engines/engine.py.230: loss: 0.13766 [I 2022-10-10 02:01:30] ...optimizers/trijoint.py.151: total_norm: 10.628433 [I 2022-10-10 02:01:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 1660/2304 [I 2022-10-10 02:01:39] ...trap/engines/engine.py.228: elapsed: 0:23:59 | left: 0:09:06 [I 2022-10-10 02:01:39] ...trap/engines/engine.py.229: process: 0.83160 | load: 0.00313 [I 2022-10-10 02:01:39] ...trap/engines/engine.py.230: loss: 0.16295 [I 2022-10-10 02:01:39] ...optimizers/trijoint.py.151: total_norm: 16.799555 [I 2022-10-10 02:01:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1670/2304 [I 2022-10-10 02:01:47] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:09:03 [I 2022-10-10 02:01:47] ...trap/engines/engine.py.229: process: 0.85213 | load: 0.00312 [I 2022-10-10 02:01:47] ...trap/engines/engine.py.230: loss: 0.16859 [I 2022-10-10 02:01:47] ...optimizers/trijoint.py.151: total_norm: 23.991629 [I 2022-10-10 02:01:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1680/2304 [I 2022-10-10 02:01:56] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:08:48 [I 2022-10-10 02:01:56] ...trap/engines/engine.py.229: process: 0.84596 | load: 0.00308 [I 2022-10-10 02:01:56] ...trap/engines/engine.py.230: loss: 0.15804 [I 2022-10-10 02:01:56] ...optimizers/trijoint.py.151: total_norm: 19.543152 [I 2022-10-10 02:02:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1690/2304 [I 2022-10-10 02:02:04] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:08:41 [I 2022-10-10 02:02:04] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00312 [I 2022-10-10 02:02:04] ...trap/engines/engine.py.230: loss: 0.16359 [I 2022-10-10 02:02:04] ...optimizers/trijoint.py.151: total_norm: 29.278721 [I 2022-10-10 02:02:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1700/2304 [I 2022-10-10 02:02:13] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:31 [I 2022-10-10 02:02:13] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00305 [I 2022-10-10 02:02:13] ...trap/engines/engine.py.230: loss: 0.13704 [I 2022-10-10 02:02:13] ...optimizers/trijoint.py.151: total_norm: 8.644178 [I 2022-10-10 02:02:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1710/2304 [I 2022-10-10 02:02:21] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:23 [I 2022-10-10 02:02:21] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00300 [I 2022-10-10 02:02:21] ...trap/engines/engine.py.230: loss: 0.11272 [I 2022-10-10 02:02:21] ...optimizers/trijoint.py.151: total_norm: 8.204240 [I 2022-10-10 02:02:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1720/2304 [I 2022-10-10 02:02:30] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:10 [I 2022-10-10 02:02:30] ...trap/engines/engine.py.229: process: 0.84341 | load: 0.00298 [I 2022-10-10 02:02:30] ...trap/engines/engine.py.230: loss: 0.22466 [I 2022-10-10 02:02:30] ...optimizers/trijoint.py.151: total_norm: 28.394066 [I 2022-10-10 02:02:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1730/2304 [I 2022-10-10 02:02:38] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:06 [I 2022-10-10 02:02:38] ...trap/engines/engine.py.229: process: 0.84596 | load: 0.00302 [I 2022-10-10 02:02:38] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-10 02:02:38] ...optimizers/trijoint.py.151: total_norm: 7.759459 [I 2022-10-10 02:02:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1740/2304 [I 2022-10-10 02:02:47] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:07:56 [I 2022-10-10 02:02:47] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00297 [I 2022-10-10 02:02:47] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-10 02:02:47] ...optimizers/trijoint.py.151: total_norm: 19.906319 [I 2022-10-10 02:02:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 1750/2304 [I 2022-10-10 02:02:55] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:47 [I 2022-10-10 02:02:55] ...trap/engines/engine.py.229: process: 0.84120 | load: 0.00302 [I 2022-10-10 02:02:55] ...trap/engines/engine.py.230: loss: 0.22841 [I 2022-10-10 02:02:55] ...optimizers/trijoint.py.151: total_norm: 13.119482 [I 2022-10-10 02:03:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1760/2304 [I 2022-10-10 02:03:04] ...trap/engines/engine.py.228: elapsed: 0:25:24 | left: 0:07:43 [I 2022-10-10 02:03:04] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00299 [I 2022-10-10 02:03:04] ...trap/engines/engine.py.230: loss: 0.17003 [I 2022-10-10 02:03:04] ...optimizers/trijoint.py.151: total_norm: 28.215982 [I 2022-10-10 02:03:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1770/2304 [I 2022-10-10 02:03:12] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:30 [I 2022-10-10 02:03:12] ...trap/engines/engine.py.229: process: 0.83729 | load: 0.00299 [I 2022-10-10 02:03:12] ...trap/engines/engine.py.230: loss: 0.20974 [I 2022-10-10 02:03:12] ...optimizers/trijoint.py.151: total_norm: 35.783928 [I 2022-10-10 02:03:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1780/2304 [I 2022-10-10 02:03:21] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:23 [I 2022-10-10 02:03:21] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00305 [I 2022-10-10 02:03:21] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-10 02:03:21] ...optimizers/trijoint.py.151: total_norm: 13.361320 [I 2022-10-10 02:03:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 1790/2304 [I 2022-10-10 02:03:29] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:07:16 [I 2022-10-10 02:03:29] ...trap/engines/engine.py.229: process: 0.85357 | load: 0.00327 [I 2022-10-10 02:03:29] ...trap/engines/engine.py.230: loss: 0.19739 [I 2022-10-10 02:03:29] ...optimizers/trijoint.py.151: total_norm: 16.807903 [I 2022-10-10 02:03:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1800/2304 [I 2022-10-10 02:03:38] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:04 [I 2022-10-10 02:03:38] ...trap/engines/engine.py.229: process: 0.84601 | load: 0.00303 [I 2022-10-10 02:03:38] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-10 02:03:38] ...optimizers/trijoint.py.151: total_norm: 13.925920 [I 2022-10-10 02:03:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 1810/2304 [I 2022-10-10 02:03:46] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:56 [I 2022-10-10 02:03:46] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00316 [I 2022-10-10 02:03:46] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-10 02:03:46] ...optimizers/trijoint.py.151: total_norm: 19.216234 [I 2022-10-10 02:03:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 1820/2304 [I 2022-10-10 02:03:55] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:50 [I 2022-10-10 02:03:55] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00300 [I 2022-10-10 02:03:55] ...trap/engines/engine.py.230: loss: 0.14203 [I 2022-10-10 02:03:55] ...optimizers/trijoint.py.151: total_norm: 8.784108 [I 2022-10-10 02:04:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 1830/2304 [I 2022-10-10 02:04:03] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:06:43 [I 2022-10-10 02:04:03] ...trap/engines/engine.py.229: process: 0.84140 | load: 0.00319 [I 2022-10-10 02:04:03] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-10 02:04:03] ...optimizers/trijoint.py.151: total_norm: 13.556748 [I 2022-10-10 02:04:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1840/2304 [I 2022-10-10 02:04:12] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:36 [I 2022-10-10 02:04:12] ...trap/engines/engine.py.229: process: 0.86286 | load: 0.00300 [I 2022-10-10 02:04:12] ...trap/engines/engine.py.230: loss: 0.13634 [I 2022-10-10 02:04:12] ...optimizers/trijoint.py.151: total_norm: 18.746473 [I 2022-10-10 02:04:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 1850/2304 [I 2022-10-10 02:04:20] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:24 [I 2022-10-10 02:04:20] ...trap/engines/engine.py.229: process: 0.83033 | load: 0.00311 [I 2022-10-10 02:04:20] ...trap/engines/engine.py.230: loss: 0.09092 [I 2022-10-10 02:04:20] ...optimizers/trijoint.py.151: total_norm: 7.121252 [I 2022-10-10 02:04:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 1860/2304 [I 2022-10-10 02:04:29] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:21 [I 2022-10-10 02:04:29] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00315 [I 2022-10-10 02:04:29] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-10 02:04:29] ...optimizers/trijoint.py.151: total_norm: 10.053214 [I 2022-10-10 02:04:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 1870/2304 [I 2022-10-10 02:04:37] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:11 [I 2022-10-10 02:04:37] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00313 [I 2022-10-10 02:04:37] ...trap/engines/engine.py.230: loss: 0.13337 [I 2022-10-10 02:04:37] ...optimizers/trijoint.py.151: total_norm: 20.107721 [I 2022-10-10 02:04:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 1880/2304 [I 2022-10-10 02:04:46] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:00 [I 2022-10-10 02:04:46] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00312 [I 2022-10-10 02:04:46] ...trap/engines/engine.py.230: loss: 0.11794 [I 2022-10-10 02:04:46] ...optimizers/trijoint.py.151: total_norm: 25.793594 [I 2022-10-10 02:04:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 1890/2304 [I 2022-10-10 02:04:54] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:52 [I 2022-10-10 02:04:54] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00317 [I 2022-10-10 02:04:54] ...trap/engines/engine.py.230: loss: 0.08164 [I 2022-10-10 02:04:54] ...optimizers/trijoint.py.151: total_norm: 8.680971 [I 2022-10-10 02:05:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 1900/2304 [I 2022-10-10 02:05:03] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:47 [I 2022-10-10 02:05:03] ...trap/engines/engine.py.229: process: 0.86396 | load: 0.00300 [I 2022-10-10 02:05:03] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-10 02:05:03] ...optimizers/trijoint.py.151: total_norm: 16.674826 [I 2022-10-10 02:05:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1910/2304 [I 2022-10-10 02:05:12] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:36 [I 2022-10-10 02:05:12] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00311 [I 2022-10-10 02:05:12] ...trap/engines/engine.py.230: loss: 0.11307 [I 2022-10-10 02:05:12] ...optimizers/trijoint.py.151: total_norm: 9.210651 [I 2022-10-10 02:05:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 1920/2304 [I 2022-10-10 02:05:20] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:28 [I 2022-10-10 02:05:20] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00324 [I 2022-10-10 02:05:20] ...trap/engines/engine.py.230: loss: 0.22234 [I 2022-10-10 02:05:20] ...optimizers/trijoint.py.151: total_norm: 33.051388 [I 2022-10-10 02:05:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 1930/2304 [I 2022-10-10 02:05:29] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:19 [I 2022-10-10 02:05:29] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00312 [I 2022-10-10 02:05:29] ...trap/engines/engine.py.230: loss: 0.13121 [I 2022-10-10 02:05:29] ...optimizers/trijoint.py.151: total_norm: 34.621460 [I 2022-10-10 02:05:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 1940/2304 [I 2022-10-10 02:05:37] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:11 [I 2022-10-10 02:05:37] ...trap/engines/engine.py.229: process: 0.84519 | load: 0.00314 [I 2022-10-10 02:05:37] ...trap/engines/engine.py.230: loss: 0.12105 [I 2022-10-10 02:05:37] ...optimizers/trijoint.py.151: total_norm: 15.524964 [I 2022-10-10 02:05:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 1950/2304 [I 2022-10-10 02:05:46] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:03 [I 2022-10-10 02:05:46] ...trap/engines/engine.py.229: process: 0.85276 | load: 0.00313 [I 2022-10-10 02:05:46] ...trap/engines/engine.py.230: loss: 0.15054 [I 2022-10-10 02:05:46] ...optimizers/trijoint.py.151: total_norm: 17.372456 [I 2022-10-10 02:05:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 1960/2304 [I 2022-10-10 02:05:54] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:54 [I 2022-10-10 02:05:54] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00311 [I 2022-10-10 02:05:54] ...trap/engines/engine.py.230: loss: 0.09214 [I 2022-10-10 02:05:54] ...optimizers/trijoint.py.151: total_norm: 21.148409 [I 2022-10-10 02:06:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 1970/2304 [I 2022-10-10 02:06:03] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:45 [I 2022-10-10 02:06:03] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00318 [I 2022-10-10 02:06:03] ...trap/engines/engine.py.230: loss: 0.16492 [I 2022-10-10 02:06:03] ...optimizers/trijoint.py.151: total_norm: 36.579731 [I 2022-10-10 02:06:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1980/2304 [I 2022-10-10 02:06:12] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:36 [I 2022-10-10 02:06:12] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00313 [I 2022-10-10 02:06:12] ...trap/engines/engine.py.230: loss: 0.12722 [I 2022-10-10 02:06:12] ...optimizers/trijoint.py.151: total_norm: 11.171885 [I 2022-10-10 02:06:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 1990/2304 [I 2022-10-10 02:06:20] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:26 [I 2022-10-10 02:06:20] ...trap/engines/engine.py.229: process: 0.83448 | load: 0.00313 [I 2022-10-10 02:06:20] ...trap/engines/engine.py.230: loss: 0.14808 [I 2022-10-10 02:06:20] ...optimizers/trijoint.py.151: total_norm: 26.083797 [I 2022-10-10 02:06:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 2000/2304 [I 2022-10-10 02:06:29] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:17 [I 2022-10-10 02:06:29] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00315 [I 2022-10-10 02:06:29] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-10 02:06:29] ...optimizers/trijoint.py.151: total_norm: 16.282402 [I 2022-10-10 02:06:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 2010/2304 [I 2022-10-10 02:06:37] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:11 [I 2022-10-10 02:06:37] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00309 [I 2022-10-10 02:06:37] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-10 02:06:37] ...optimizers/trijoint.py.151: total_norm: 11.585913 [I 2022-10-10 02:06:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 2020/2304 [I 2022-10-10 02:06:46] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:04:02 [I 2022-10-10 02:06:46] ...trap/engines/engine.py.229: process: 0.86090 | load: 0.00321 [I 2022-10-10 02:06:46] ...trap/engines/engine.py.230: loss: 0.27896 [I 2022-10-10 02:06:46] ...optimizers/trijoint.py.151: total_norm: 18.205311 [I 2022-10-10 02:06:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 2030/2304 [I 2022-10-10 02:06:54] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:54 [I 2022-10-10 02:06:54] ...trap/engines/engine.py.229: process: 0.86062 | load: 0.00311 [I 2022-10-10 02:06:54] ...trap/engines/engine.py.230: loss: 0.11624 [I 2022-10-10 02:06:54] ...optimizers/trijoint.py.151: total_norm: 13.339283 [I 2022-10-10 02:07:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 2040/2304 [I 2022-10-10 02:07:03] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:47 [I 2022-10-10 02:07:03] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00312 [I 2022-10-10 02:07:03] ...trap/engines/engine.py.230: loss: 0.19246 [I 2022-10-10 02:07:03] ...optimizers/trijoint.py.151: total_norm: 17.697369 [I 2022-10-10 02:07:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 2050/2304 [I 2022-10-10 02:07:12] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:38 [I 2022-10-10 02:07:12] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00312 [I 2022-10-10 02:07:12] ...trap/engines/engine.py.230: loss: 0.12426 [I 2022-10-10 02:07:12] ...optimizers/trijoint.py.151: total_norm: 16.555161 [I 2022-10-10 02:07:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 2060/2304 [I 2022-10-10 02:07:20] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:28 [I 2022-10-10 02:07:20] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00314 [I 2022-10-10 02:07:20] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-10 02:07:20] ...optimizers/trijoint.py.151: total_norm: 21.948019 [I 2022-10-10 02:07:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 2070/2304 [I 2022-10-10 02:07:29] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:19 [I 2022-10-10 02:07:29] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00319 [I 2022-10-10 02:07:29] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-10 02:07:29] ...optimizers/trijoint.py.151: total_norm: 7.623554 [I 2022-10-10 02:07:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 2080/2304 [I 2022-10-10 02:07:37] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:10 [I 2022-10-10 02:07:37] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00311 [I 2022-10-10 02:07:37] ...trap/engines/engine.py.230: loss: 0.17173 [I 2022-10-10 02:07:37] ...optimizers/trijoint.py.151: total_norm: 14.529381 [I 2022-10-10 02:07:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 2090/2304 [I 2022-10-10 02:07:46] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:03 [I 2022-10-10 02:07:46] ...trap/engines/engine.py.229: process: 0.85597 | load: 0.00312 [I 2022-10-10 02:07:46] ...trap/engines/engine.py.230: loss: 0.12626 [I 2022-10-10 02:07:46] ...optimizers/trijoint.py.151: total_norm: 14.227533 [I 2022-10-10 02:07:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 2100/2304 [I 2022-10-10 02:07:55] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:55 [I 2022-10-10 02:07:55] ...trap/engines/engine.py.229: process: 0.85616 | load: 0.00316 [I 2022-10-10 02:07:55] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-10 02:07:55] ...optimizers/trijoint.py.151: total_norm: 9.619739 [I 2022-10-10 02:08:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 2110/2304 [I 2022-10-10 02:08:03] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:45 [I 2022-10-10 02:08:03] ...trap/engines/engine.py.229: process: 0.83830 | load: 0.00312 [I 2022-10-10 02:08:03] ...trap/engines/engine.py.230: loss: 0.11070 [I 2022-10-10 02:08:03] ...optimizers/trijoint.py.151: total_norm: 13.111670 [I 2022-10-10 02:08:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 2120/2304 [I 2022-10-10 02:08:12] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:38 [I 2022-10-10 02:08:12] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00327 [I 2022-10-10 02:08:12] ...trap/engines/engine.py.230: loss: 0.09824 [I 2022-10-10 02:08:12] ...optimizers/trijoint.py.151: total_norm: 9.968739 [I 2022-10-10 02:08:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 2130/2304 [I 2022-10-10 02:08:20] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:29 [I 2022-10-10 02:08:20] ...trap/engines/engine.py.229: process: 0.86003 | load: 0.00324 [I 2022-10-10 02:08:20] ...trap/engines/engine.py.230: loss: 0.19085 [I 2022-10-10 02:08:20] ...optimizers/trijoint.py.151: total_norm: 23.532413 [I 2022-10-10 02:08:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 2140/2304 [I 2022-10-10 02:08:29] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:19 [I 2022-10-10 02:08:29] ...trap/engines/engine.py.229: process: 0.83508 | load: 0.00345 [I 2022-10-10 02:08:29] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-10 02:08:29] ...optimizers/trijoint.py.151: total_norm: 7.298514 [I 2022-10-10 02:08:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 2150/2304 [I 2022-10-10 02:08:37] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:10 [I 2022-10-10 02:08:37] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00295 [I 2022-10-10 02:08:37] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-10 02:08:37] ...optimizers/trijoint.py.151: total_norm: 14.673152 [I 2022-10-10 02:08:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 2160/2304 [I 2022-10-10 02:08:46] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:02 [I 2022-10-10 02:08:46] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00302 [I 2022-10-10 02:08:46] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-10 02:08:46] ...optimizers/trijoint.py.151: total_norm: 81.181206 [I 2022-10-10 02:08:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 2170/2304 [I 2022-10-10 02:08:54] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:53 [I 2022-10-10 02:08:54] ...trap/engines/engine.py.229: process: 0.84864 | load: 0.00305 [I 2022-10-10 02:08:54] ...trap/engines/engine.py.230: loss: 0.17211 [I 2022-10-10 02:08:54] ...optimizers/trijoint.py.151: total_norm: 16.254322 [I 2022-10-10 02:09:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 2180/2304 [I 2022-10-10 02:09:03] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:44 [I 2022-10-10 02:09:03] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00303 [I 2022-10-10 02:09:03] ...trap/engines/engine.py.230: loss: 0.18472 [I 2022-10-10 02:09:03] ...optimizers/trijoint.py.151: total_norm: 25.765629 [I 2022-10-10 02:09:11] ...trap/engines/engine.py.224: train: epoch 62 | batch 2190/2304 [I 2022-10-10 02:09:11] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:36 [I 2022-10-10 02:09:11] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00299 [I 2022-10-10 02:09:11] ...trap/engines/engine.py.230: loss: 0.20133 [I 2022-10-10 02:09:11] ...optimizers/trijoint.py.151: total_norm: 26.459734 [I 2022-10-10 02:09:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 2200/2304 [I 2022-10-10 02:09:20] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:28 [I 2022-10-10 02:09:20] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00302 [I 2022-10-10 02:09:20] ...trap/engines/engine.py.230: loss: 0.21966 [I 2022-10-10 02:09:20] ...optimizers/trijoint.py.151: total_norm: 18.851912 [I 2022-10-10 02:09:28] ...trap/engines/engine.py.224: train: epoch 62 | batch 2210/2304 [I 2022-10-10 02:09:28] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:19 [I 2022-10-10 02:09:28] ...trap/engines/engine.py.229: process: 0.83153 | load: 0.00300 [I 2022-10-10 02:09:28] ...trap/engines/engine.py.230: loss: 0.10283 [I 2022-10-10 02:09:28] ...optimizers/trijoint.py.151: total_norm: 10.678028 [I 2022-10-10 02:09:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 2220/2304 [I 2022-10-10 02:09:37] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:10 [I 2022-10-10 02:09:37] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00331 [I 2022-10-10 02:09:37] ...trap/engines/engine.py.230: loss: 0.09808 [I 2022-10-10 02:09:37] ...optimizers/trijoint.py.151: total_norm: 9.493791 [I 2022-10-10 02:09:45] ...trap/engines/engine.py.224: train: epoch 62 | batch 2230/2304 [I 2022-10-10 02:09:45] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:02 [I 2022-10-10 02:09:45] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00299 [I 2022-10-10 02:09:45] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-10 02:09:45] ...optimizers/trijoint.py.151: total_norm: 90.569618 [I 2022-10-10 02:09:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 2240/2304 [I 2022-10-10 02:09:54] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:53 [I 2022-10-10 02:09:54] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00301 [I 2022-10-10 02:09:54] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-10 02:09:54] ...optimizers/trijoint.py.151: total_norm: 16.866936 [I 2022-10-10 02:10:02] ...trap/engines/engine.py.224: train: epoch 62 | batch 2250/2304 [I 2022-10-10 02:10:02] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:45 [I 2022-10-10 02:10:02] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00299 [I 2022-10-10 02:10:02] ...trap/engines/engine.py.230: loss: 0.23171 [I 2022-10-10 02:10:02] ...optimizers/trijoint.py.151: total_norm: 15.984975 [I 2022-10-10 02:10:11] ...trap/engines/engine.py.224: train: epoch 62 | batch 2260/2304 [I 2022-10-10 02:10:11] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:37 [I 2022-10-10 02:10:11] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00301 [I 2022-10-10 02:10:11] ...trap/engines/engine.py.230: loss: 0.24638 [I 2022-10-10 02:10:11] ...optimizers/trijoint.py.151: total_norm: 15.296517 [I 2022-10-10 02:10:19] ...trap/engines/engine.py.224: train: epoch 62 | batch 2270/2304 [I 2022-10-10 02:10:19] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:28 [I 2022-10-10 02:10:19] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00300 [I 2022-10-10 02:10:19] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-10 02:10:19] ...optimizers/trijoint.py.151: total_norm: 9.939149 [I 2022-10-10 02:10:28] ...trap/engines/engine.py.224: train: epoch 62 | batch 2280/2304 [I 2022-10-10 02:10:28] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:20 [I 2022-10-10 02:10:28] ...trap/engines/engine.py.229: process: 0.86884 | load: 0.00298 [I 2022-10-10 02:10:28] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-10 02:10:28] ...optimizers/trijoint.py.151: total_norm: 17.888680 [I 2022-10-10 02:10:36] ...trap/engines/engine.py.224: train: epoch 62 | batch 2290/2304 [I 2022-10-10 02:10:36] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:11 [I 2022-10-10 02:10:36] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00314 [I 2022-10-10 02:10:36] ...trap/engines/engine.py.230: loss: 0.14252 [I 2022-10-10 02:10:36] ...optimizers/trijoint.py.151: total_norm: 18.971706 [I 2022-10-10 02:10:45] ...trap/engines/engine.py.224: train: epoch 62 | batch 2300/2304 [I 2022-10-10 02:10:45] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:03 [I 2022-10-10 02:10:45] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00020 [I 2022-10-10 02:10:45] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-10 02:10:45] ...optimizers/trijoint.py.151: total_norm: 12.390327 [I 2022-10-10 02:10:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 2304/2304 [I 2022-10-10 02:10:48] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:00 [I 2022-10-10 02:10:48] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00020 [I 2022-10-10 02:10:48] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-10 02:10:48] ...optimizers/trijoint.py.151: total_norm: 9.688968 [S 2022-10-10 02:10:50] ...trap/engines/engine.py.129: train_epoch.epoch: 62 [S 2022-10-10 02:10:50] ...trap/engines/engine.py.129: train_epoch.loss: 0.14649438939810833 [I 2022-10-10 02:10:50] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 02:10:51] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 62 [I 2022-10-10 02:10:51] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 02:10:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 02:10:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 02:10:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 02:10:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 02:10:55] ...trap/engines/engine.py.321: eval: epoch 62 | batch 0/510 [I 2022-10-10 02:10:55] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:48 [I 2022-10-10 02:10:55] ...trap/engines/engine.py.326: process: 4.44745 | load: 3.93568 [I 2022-10-10 02:11:01] ...trap/engines/engine.py.321: eval: epoch 62 | batch 10/510 [I 2022-10-10 02:11:01] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:22 [I 2022-10-10 02:11:01] ...trap/engines/engine.py.326: process: 0.55936 | load: 0.00021 [I 2022-10-10 02:11:07] ...trap/engines/engine.py.321: eval: epoch 62 | batch 20/510 [I 2022-10-10 02:11:07] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:16 [I 2022-10-10 02:11:07] ...trap/engines/engine.py.326: process: 0.73937 | load: 0.00020 [I 2022-10-10 02:11:14] ...trap/engines/engine.py.321: eval: epoch 62 | batch 30/510 [I 2022-10-10 02:11:14] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:05:23 [I 2022-10-10 02:11:14] ...trap/engines/engine.py.326: process: 0.78624 | load: 0.00020 [I 2022-10-10 02:11:21] ...trap/engines/engine.py.321: eval: epoch 62 | batch 40/510 [I 2022-10-10 02:11:21] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:05:22 [I 2022-10-10 02:11:21] ...trap/engines/engine.py.326: process: 0.55591 | load: 0.00020 [I 2022-10-10 02:11:27] ...trap/engines/engine.py.321: eval: epoch 62 | batch 50/510 [I 2022-10-10 02:11:27] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:37 [I 2022-10-10 02:11:27] ...trap/engines/engine.py.326: process: 0.52272 | load: 0.00020 [I 2022-10-10 02:11:32] ...trap/engines/engine.py.321: eval: epoch 62 | batch 60/510 [I 2022-10-10 02:11:32] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:12 [I 2022-10-10 02:11:32] ...trap/engines/engine.py.326: process: 0.60476 | load: 0.00020 [I 2022-10-10 02:11:37] ...trap/engines/engine.py.321: eval: epoch 62 | batch 70/510 [I 2022-10-10 02:11:37] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:31 [I 2022-10-10 02:11:37] ...trap/engines/engine.py.326: process: 0.44327 | load: 0.00022 [I 2022-10-10 02:11:43] ...trap/engines/engine.py.321: eval: epoch 62 | batch 80/510 [I 2022-10-10 02:11:43] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:04:05 [I 2022-10-10 02:11:43] ...trap/engines/engine.py.326: process: 0.55737 | load: 0.00024 [I 2022-10-10 02:11:48] ...trap/engines/engine.py.321: eval: epoch 62 | batch 90/510 [I 2022-10-10 02:11:48] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:47 [I 2022-10-10 02:11:48] ...trap/engines/engine.py.326: process: 0.54609 | load: 0.00023 [I 2022-10-10 02:11:53] ...trap/engines/engine.py.321: eval: epoch 62 | batch 100/510 [I 2022-10-10 02:11:53] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:03:11 [I 2022-10-10 02:11:53] ...trap/engines/engine.py.326: process: 0.31183 | load: 0.00021 [I 2022-10-10 02:11:57] ...trap/engines/engine.py.321: eval: epoch 62 | batch 110/510 [I 2022-10-10 02:11:57] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:12 [I 2022-10-10 02:11:57] ...trap/engines/engine.py.326: process: 0.32063 | load: 0.00020 [I 2022-10-10 02:12:00] ...trap/engines/engine.py.321: eval: epoch 62 | batch 120/510 [I 2022-10-10 02:12:00] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:05 [I 2022-10-10 02:12:00] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00020 [I 2022-10-10 02:12:03] ...trap/engines/engine.py.321: eval: epoch 62 | batch 130/510 [I 2022-10-10 02:12:03] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:00 [I 2022-10-10 02:12:03] ...trap/engines/engine.py.326: process: 0.31236 | load: 0.00020 [I 2022-10-10 02:12:06] ...trap/engines/engine.py.321: eval: epoch 62 | batch 140/510 [I 2022-10-10 02:12:06] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:58 [I 2022-10-10 02:12:06] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00020 [I 2022-10-10 02:12:09] ...trap/engines/engine.py.321: eval: epoch 62 | batch 150/510 [I 2022-10-10 02:12:09] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:54 [I 2022-10-10 02:12:09] ...trap/engines/engine.py.326: process: 0.32131 | load: 0.00021 [I 2022-10-10 02:12:12] ...trap/engines/engine.py.321: eval: epoch 62 | batch 160/510 [I 2022-10-10 02:12:12] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:51 [I 2022-10-10 02:12:13] ...trap/engines/engine.py.326: process: 0.31529 | load: 0.00019 [I 2022-10-10 02:12:16] ...trap/engines/engine.py.321: eval: epoch 62 | batch 170/510 [I 2022-10-10 02:12:16] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:48 [I 2022-10-10 02:12:16] ...trap/engines/engine.py.326: process: 0.32098 | load: 0.00021 [I 2022-10-10 02:12:19] ...trap/engines/engine.py.321: eval: epoch 62 | batch 180/510 [I 2022-10-10 02:12:19] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:45 [I 2022-10-10 02:12:19] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00023 [I 2022-10-10 02:12:22] ...trap/engines/engine.py.321: eval: epoch 62 | batch 190/510 [I 2022-10-10 02:12:22] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:42 [I 2022-10-10 02:12:22] ...trap/engines/engine.py.326: process: 0.31845 | load: 0.00020 [I 2022-10-10 02:12:25] ...trap/engines/engine.py.321: eval: epoch 62 | batch 200/510 [I 2022-10-10 02:12:25] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:39 [I 2022-10-10 02:12:25] ...trap/engines/engine.py.326: process: 0.32423 | load: 0.00021 [I 2022-10-10 02:12:29] ...trap/engines/engine.py.321: eval: epoch 62 | batch 210/510 [I 2022-10-10 02:12:29] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:36 [I 2022-10-10 02:12:29] ...trap/engines/engine.py.326: process: 0.32477 | load: 0.00021 [I 2022-10-10 02:12:32] ...trap/engines/engine.py.321: eval: epoch 62 | batch 220/510 [I 2022-10-10 02:12:32] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:33 [I 2022-10-10 02:12:32] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00020 [I 2022-10-10 02:12:35] ...trap/engines/engine.py.321: eval: epoch 62 | batch 230/510 [I 2022-10-10 02:12:35] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:29 [I 2022-10-10 02:12:35] ...trap/engines/engine.py.326: process: 0.31675 | load: 0.00021 [I 2022-10-10 02:12:38] ...trap/engines/engine.py.321: eval: epoch 62 | batch 240/510 [I 2022-10-10 02:12:38] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:25 [I 2022-10-10 02:12:38] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00026 [I 2022-10-10 02:12:41] ...trap/engines/engine.py.321: eval: epoch 62 | batch 250/510 [I 2022-10-10 02:12:41] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:23 [I 2022-10-10 02:12:41] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00021 [I 2022-10-10 02:12:45] ...trap/engines/engine.py.321: eval: epoch 62 | batch 260/510 [I 2022-10-10 02:12:45] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:20 [I 2022-10-10 02:12:45] ...trap/engines/engine.py.326: process: 0.32178 | load: 0.00020 [I 2022-10-10 02:12:48] ...trap/engines/engine.py.321: eval: epoch 62 | batch 270/510 [I 2022-10-10 02:12:48] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:17 [I 2022-10-10 02:12:48] ...trap/engines/engine.py.326: process: 0.32608 | load: 0.00021 [I 2022-10-10 02:12:51] ...trap/engines/engine.py.321: eval: epoch 62 | batch 280/510 [I 2022-10-10 02:12:51] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:13 [I 2022-10-10 02:12:51] ...trap/engines/engine.py.326: process: 0.31681 | load: 0.00021 [I 2022-10-10 02:12:54] ...trap/engines/engine.py.321: eval: epoch 62 | batch 290/510 [I 2022-10-10 02:12:54] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:10 [I 2022-10-10 02:12:54] ...trap/engines/engine.py.326: process: 0.32075 | load: 0.00020 [I 2022-10-10 02:12:57] ...trap/engines/engine.py.321: eval: epoch 62 | batch 300/510 [I 2022-10-10 02:12:57] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:07 [I 2022-10-10 02:12:57] ...trap/engines/engine.py.326: process: 0.32068 | load: 0.00021 [I 2022-10-10 02:13:01] ...trap/engines/engine.py.321: eval: epoch 62 | batch 310/510 [I 2022-10-10 02:13:01] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:04 [I 2022-10-10 02:13:01] ...trap/engines/engine.py.326: process: 0.32374 | load: 0.00021 [I 2022-10-10 02:13:04] ...trap/engines/engine.py.321: eval: epoch 62 | batch 320/510 [I 2022-10-10 02:13:04] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:01:01 [I 2022-10-10 02:13:04] ...trap/engines/engine.py.326: process: 0.31550 | load: 0.00021 [I 2022-10-10 02:13:07] ...trap/engines/engine.py.321: eval: epoch 62 | batch 330/510 [I 2022-10-10 02:13:07] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:58 [I 2022-10-10 02:13:07] ...trap/engines/engine.py.326: process: 0.32086 | load: 0.00021 [I 2022-10-10 02:13:10] ...trap/engines/engine.py.321: eval: epoch 62 | batch 340/510 [I 2022-10-10 02:13:10] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:54 [I 2022-10-10 02:13:10] ...trap/engines/engine.py.326: process: 0.32585 | load: 0.00021 [I 2022-10-10 02:13:13] ...trap/engines/engine.py.321: eval: epoch 62 | batch 350/510 [I 2022-10-10 02:13:13] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:51 [I 2022-10-10 02:13:13] ...trap/engines/engine.py.326: process: 0.33254 | load: 0.00021 [I 2022-10-10 02:13:17] ...trap/engines/engine.py.321: eval: epoch 62 | batch 360/510 [I 2022-10-10 02:13:17] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:48 [I 2022-10-10 02:13:17] ...trap/engines/engine.py.326: process: 0.31962 | load: 0.00020 [I 2022-10-10 02:13:20] ...trap/engines/engine.py.321: eval: epoch 62 | batch 370/510 [I 2022-10-10 02:13:20] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:45 [I 2022-10-10 02:13:20] ...trap/engines/engine.py.326: process: 0.32424 | load: 0.00021 [I 2022-10-10 02:13:23] ...trap/engines/engine.py.321: eval: epoch 62 | batch 380/510 [I 2022-10-10 02:13:23] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:41 [I 2022-10-10 02:13:23] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00021 [I 2022-10-10 02:13:26] ...trap/engines/engine.py.321: eval: epoch 62 | batch 390/510 [I 2022-10-10 02:13:26] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:38 [I 2022-10-10 02:13:26] ...trap/engines/engine.py.326: process: 0.32122 | load: 0.00021 [I 2022-10-10 02:13:30] ...trap/engines/engine.py.321: eval: epoch 62 | batch 400/510 [I 2022-10-10 02:13:30] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:35 [I 2022-10-10 02:13:30] ...trap/engines/engine.py.326: process: 0.32290 | load: 0.00021 [I 2022-10-10 02:13:33] ...trap/engines/engine.py.321: eval: epoch 62 | batch 410/510 [I 2022-10-10 02:13:33] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:32 [I 2022-10-10 02:13:33] ...trap/engines/engine.py.326: process: 0.32553 | load: 0.00021 [I 2022-10-10 02:13:36] ...trap/engines/engine.py.321: eval: epoch 62 | batch 420/510 [I 2022-10-10 02:13:36] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:29 [I 2022-10-10 02:13:36] ...trap/engines/engine.py.326: process: 0.32216 | load: 0.00021 [I 2022-10-10 02:13:39] ...trap/engines/engine.py.321: eval: epoch 62 | batch 430/510 [I 2022-10-10 02:13:39] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:25 [I 2022-10-10 02:13:39] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00021 [I 2022-10-10 02:13:42] ...trap/engines/engine.py.321: eval: epoch 62 | batch 440/510 [I 2022-10-10 02:13:42] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:22 [I 2022-10-10 02:13:42] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00021 [I 2022-10-10 02:13:46] ...trap/engines/engine.py.321: eval: epoch 62 | batch 450/510 [I 2022-10-10 02:13:46] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:19 [I 2022-10-10 02:13:46] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00023 [I 2022-10-10 02:13:49] ...trap/engines/engine.py.321: eval: epoch 62 | batch 460/510 [I 2022-10-10 02:13:49] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:16 [I 2022-10-10 02:13:49] ...trap/engines/engine.py.326: process: 0.32383 | load: 0.00021 [I 2022-10-10 02:13:52] ...trap/engines/engine.py.321: eval: epoch 62 | batch 470/510 [I 2022-10-10 02:13:52] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:12 [I 2022-10-10 02:13:52] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00021 [I 2022-10-10 02:13:55] ...trap/engines/engine.py.321: eval: epoch 62 | batch 480/510 [I 2022-10-10 02:13:55] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:09 [I 2022-10-10 02:13:55] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00021 [I 2022-10-10 02:13:59] ...trap/engines/engine.py.321: eval: epoch 62 | batch 490/510 [I 2022-10-10 02:13:59] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:06 [I 2022-10-10 02:13:59] ...trap/engines/engine.py.326: process: 0.32612 | load: 0.00022 [I 2022-10-10 02:14:02] ...trap/engines/engine.py.321: eval: epoch 62 | batch 500/510 [I 2022-10-10 02:14:02] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:03 [I 2022-10-10 02:14:02] ...trap/engines/engine.py.326: process: 0.31700 | load: 0.00023 [I 2022-10-10 02:14:05] ...trap/engines/engine.py.321: eval: epoch 62 | batch 510/510 [I 2022-10-10 02:14:05] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:00 [I 2022-10-10 02:14:05] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00017 [S 2022-10-10 02:14:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 62 [I 2022-10-10 02:14:06] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.724300 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909800 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940000 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.719000 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905900 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937900 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012869 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008280 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007362 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011269 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009027 [S 2022-10-10 02:14:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007648 [I 2022-10-10 02:14:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 02:14:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 02:14:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 02:14:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 02:14:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 02:14:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 02:14:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 02:14:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 02:14:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 63 [I 2022-10-10 02:14:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 02:14:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 0/2304 [I 2022-10-10 02:14:56] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:48 [I 2022-10-10 02:14:56] ...trap/engines/engine.py.229: process: 2.80757 | load: 1.90135 [I 2022-10-10 02:14:56] ...trap/engines/engine.py.230: loss: 0.14176 [I 2022-10-10 02:14:56] ...optimizers/trijoint.py.151: total_norm: 13.953780 [I 2022-10-10 02:15:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 10/2304 [I 2022-10-10 02:15:05] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:35 [I 2022-10-10 02:15:05] ...trap/engines/engine.py.229: process: 0.87064 | load: 0.00526 [I 2022-10-10 02:15:05] ...trap/engines/engine.py.230: loss: 0.16039 [I 2022-10-10 02:15:05] ...optimizers/trijoint.py.151: total_norm: 40.446327 [I 2022-10-10 02:15:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 20/2304 [I 2022-10-10 02:15:13] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:06 [I 2022-10-10 02:15:13] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00472 [I 2022-10-10 02:15:13] ...trap/engines/engine.py.230: loss: 0.11597 [I 2022-10-10 02:15:13] ...optimizers/trijoint.py.151: total_norm: 12.961683 [I 2022-10-10 02:15:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 30/2304 [I 2022-10-10 02:15:22] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:03 [I 2022-10-10 02:15:22] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00435 [I 2022-10-10 02:15:22] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-10 02:15:22] ...optimizers/trijoint.py.151: total_norm: 9.831072 [I 2022-10-10 02:15:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 40/2304 [I 2022-10-10 02:15:30] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:00 [I 2022-10-10 02:15:30] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00358 [I 2022-10-10 02:15:30] ...trap/engines/engine.py.230: loss: 0.15395 [I 2022-10-10 02:15:30] ...optimizers/trijoint.py.151: total_norm: 14.734175 [I 2022-10-10 02:15:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 50/2304 [I 2022-10-10 02:15:39] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:42 [I 2022-10-10 02:15:39] ...trap/engines/engine.py.229: process: 0.83698 | load: 0.00325 [I 2022-10-10 02:15:39] ...trap/engines/engine.py.230: loss: 0.12016 [I 2022-10-10 02:15:39] ...optimizers/trijoint.py.151: total_norm: 12.766590 [I 2022-10-10 02:15:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 60/2304 [I 2022-10-10 02:15:47] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:37 [I 2022-10-10 02:15:47] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00370 [I 2022-10-10 02:15:47] ...trap/engines/engine.py.230: loss: 0.13882 [I 2022-10-10 02:15:47] ...optimizers/trijoint.py.151: total_norm: 21.990978 [I 2022-10-10 02:15:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 70/2304 [I 2022-10-10 02:15:56] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:35 [I 2022-10-10 02:15:56] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00334 [I 2022-10-10 02:15:56] ...trap/engines/engine.py.230: loss: 0.11228 [I 2022-10-10 02:15:56] ...optimizers/trijoint.py.151: total_norm: 9.825688 [I 2022-10-10 02:16:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 80/2304 [I 2022-10-10 02:16:04] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:20 [I 2022-10-10 02:16:04] ...trap/engines/engine.py.229: process: 0.83793 | load: 0.00352 [I 2022-10-10 02:16:04] ...trap/engines/engine.py.230: loss: 0.09556 [I 2022-10-10 02:16:04] ...optimizers/trijoint.py.151: total_norm: 9.773211 [I 2022-10-10 02:16:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 90/2304 [I 2022-10-10 02:16:13] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:04 [I 2022-10-10 02:16:13] ...trap/engines/engine.py.229: process: 0.84232 | load: 0.00312 [I 2022-10-10 02:16:13] ...trap/engines/engine.py.230: loss: 0.14596 [I 2022-10-10 02:16:13] ...optimizers/trijoint.py.151: total_norm: 30.660715 [I 2022-10-10 02:16:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 100/2304 [I 2022-10-10 02:16:21] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:03 [I 2022-10-10 02:16:21] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00331 [I 2022-10-10 02:16:21] ...trap/engines/engine.py.230: loss: 0.15037 [I 2022-10-10 02:16:21] ...optimizers/trijoint.py.151: total_norm: 15.101954 [I 2022-10-10 02:16:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 110/2304 [I 2022-10-10 02:16:30] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:58 [I 2022-10-10 02:16:30] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00319 [I 2022-10-10 02:16:30] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-10 02:16:30] ...optimizers/trijoint.py.151: total_norm: 17.315237 [I 2022-10-10 02:16:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 120/2304 [I 2022-10-10 02:16:38] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:38 [I 2022-10-10 02:16:38] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00298 [I 2022-10-10 02:16:38] ...trap/engines/engine.py.230: loss: 0.12227 [I 2022-10-10 02:16:38] ...optimizers/trijoint.py.151: total_norm: 22.228539 [I 2022-10-10 02:16:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 130/2304 [I 2022-10-10 02:16:47] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:45 [I 2022-10-10 02:16:47] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00327 [I 2022-10-10 02:16:47] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-10 02:16:47] ...optimizers/trijoint.py.151: total_norm: 10.976981 [I 2022-10-10 02:16:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 140/2304 [I 2022-10-10 02:16:55] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:50 [I 2022-10-10 02:16:55] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00336 [I 2022-10-10 02:16:55] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-10 02:16:55] ...optimizers/trijoint.py.151: total_norm: 24.836086 [I 2022-10-10 02:17:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 150/2304 [I 2022-10-10 02:17:04] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:41 [I 2022-10-10 02:17:04] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00364 [I 2022-10-10 02:17:04] ...trap/engines/engine.py.230: loss: 0.09554 [I 2022-10-10 02:17:04] ...optimizers/trijoint.py.151: total_norm: 10.660774 [I 2022-10-10 02:17:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 160/2304 [I 2022-10-10 02:17:13] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:40 [I 2022-10-10 02:17:13] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00335 [I 2022-10-10 02:17:13] ...trap/engines/engine.py.230: loss: 0.18405 [I 2022-10-10 02:17:13] ...optimizers/trijoint.py.151: total_norm: 9.610307 [I 2022-10-10 02:17:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 170/2304 [I 2022-10-10 02:17:21] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:18 [I 2022-10-10 02:17:21] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00343 [I 2022-10-10 02:17:21] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-10 02:17:21] ...optimizers/trijoint.py.151: total_norm: 22.391268 [I 2022-10-10 02:17:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 180/2304 [I 2022-10-10 02:17:30] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:12 [I 2022-10-10 02:17:30] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00343 [I 2022-10-10 02:17:30] ...trap/engines/engine.py.230: loss: 0.15782 [I 2022-10-10 02:17:30] ...optimizers/trijoint.py.151: total_norm: 31.503538 [I 2022-10-10 02:17:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 190/2304 [I 2022-10-10 02:17:38] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:09 [I 2022-10-10 02:17:38] ...trap/engines/engine.py.229: process: 0.86863 | load: 0.00313 [I 2022-10-10 02:17:38] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-10 02:17:38] ...optimizers/trijoint.py.151: total_norm: 27.491920 [I 2022-10-10 02:17:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 200/2304 [I 2022-10-10 02:17:47] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:30:00 [I 2022-10-10 02:17:47] ...trap/engines/engine.py.229: process: 0.86947 | load: 0.00329 [I 2022-10-10 02:17:47] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-10 02:17:47] ...optimizers/trijoint.py.151: total_norm: 10.834287 [I 2022-10-10 02:17:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 210/2304 [I 2022-10-10 02:17:55] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:45 [I 2022-10-10 02:17:55] ...trap/engines/engine.py.229: process: 0.86231 | load: 0.00324 [I 2022-10-10 02:17:55] ...trap/engines/engine.py.230: loss: 0.17581 [I 2022-10-10 02:17:55] ...optimizers/trijoint.py.151: total_norm: 15.741041 [I 2022-10-10 02:18:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 220/2304 [I 2022-10-10 02:18:04] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:41 [I 2022-10-10 02:18:04] ...trap/engines/engine.py.229: process: 0.85807 | load: 0.00311 [I 2022-10-10 02:18:04] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-10 02:18:04] ...optimizers/trijoint.py.151: total_norm: 19.422451 [I 2022-10-10 02:18:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 230/2304 [I 2022-10-10 02:18:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:40 [I 2022-10-10 02:18:12] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00321 [I 2022-10-10 02:18:12] ...trap/engines/engine.py.230: loss: 0.12600 [I 2022-10-10 02:18:12] ...optimizers/trijoint.py.151: total_norm: 13.852626 [I 2022-10-10 02:18:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 240/2304 [I 2022-10-10 02:18:21] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:23 [I 2022-10-10 02:18:21] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00338 [I 2022-10-10 02:18:21] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-10 02:18:21] ...optimizers/trijoint.py.151: total_norm: 23.784832 [I 2022-10-10 02:18:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 250/2304 [I 2022-10-10 02:18:30] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:08 [I 2022-10-10 02:18:30] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00313 [I 2022-10-10 02:18:30] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-10 02:18:30] ...optimizers/trijoint.py.151: total_norm: 13.549836 [I 2022-10-10 02:18:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 260/2304 [I 2022-10-10 02:18:38] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:03 [I 2022-10-10 02:18:38] ...trap/engines/engine.py.229: process: 0.85565 | load: 0.00317 [I 2022-10-10 02:18:38] ...trap/engines/engine.py.230: loss: 0.16394 [I 2022-10-10 02:18:38] ...optimizers/trijoint.py.151: total_norm: 15.680910 [I 2022-10-10 02:18:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 270/2304 [I 2022-10-10 02:18:47] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:49 [I 2022-10-10 02:18:47] ...trap/engines/engine.py.229: process: 0.84403 | load: 0.00314 [I 2022-10-10 02:18:47] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-10 02:18:47] ...optimizers/trijoint.py.151: total_norm: 24.642599 [I 2022-10-10 02:18:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 280/2304 [I 2022-10-10 02:18:55] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:45 [I 2022-10-10 02:18:55] ...trap/engines/engine.py.229: process: 0.86190 | load: 0.00328 [I 2022-10-10 02:18:55] ...trap/engines/engine.py.230: loss: 0.19879 [I 2022-10-10 02:18:55] ...optimizers/trijoint.py.151: total_norm: 25.261198 [I 2022-10-10 02:19:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 290/2304 [I 2022-10-10 02:19:04] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:37 [I 2022-10-10 02:19:04] ...trap/engines/engine.py.229: process: 0.86107 | load: 0.00324 [I 2022-10-10 02:19:04] ...trap/engines/engine.py.230: loss: 0.17822 [I 2022-10-10 02:19:04] ...optimizers/trijoint.py.151: total_norm: 19.135485 [I 2022-10-10 02:19:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 300/2304 [I 2022-10-10 02:19:12] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:24 [I 2022-10-10 02:19:12] ...trap/engines/engine.py.229: process: 0.83412 | load: 0.00309 [I 2022-10-10 02:19:12] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-10 02:19:12] ...optimizers/trijoint.py.151: total_norm: 11.072383 [I 2022-10-10 02:19:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 310/2304 [I 2022-10-10 02:19:21] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:28 [I 2022-10-10 02:19:21] ...trap/engines/engine.py.229: process: 0.86243 | load: 0.00341 [I 2022-10-10 02:19:21] ...trap/engines/engine.py.230: loss: 0.09020 [I 2022-10-10 02:19:21] ...optimizers/trijoint.py.151: total_norm: 8.782473 [I 2022-10-10 02:19:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 320/2304 [I 2022-10-10 02:19:29] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:14 [I 2022-10-10 02:19:29] ...trap/engines/engine.py.229: process: 0.86471 | load: 0.00318 [I 2022-10-10 02:19:29] ...trap/engines/engine.py.230: loss: 0.18991 [I 2022-10-10 02:19:29] ...optimizers/trijoint.py.151: total_norm: 23.225237 [I 2022-10-10 02:19:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 330/2304 [I 2022-10-10 02:19:38] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:11 [I 2022-10-10 02:19:38] ...trap/engines/engine.py.229: process: 0.85676 | load: 0.00320 [I 2022-10-10 02:19:38] ...trap/engines/engine.py.230: loss: 0.11218 [I 2022-10-10 02:19:38] ...optimizers/trijoint.py.151: total_norm: 14.492906 [I 2022-10-10 02:19:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 340/2304 [I 2022-10-10 02:19:46] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:28:01 [I 2022-10-10 02:19:46] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00310 [I 2022-10-10 02:19:46] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-10 02:19:46] ...optimizers/trijoint.py.151: total_norm: 15.636297 [I 2022-10-10 02:19:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 350/2304 [I 2022-10-10 02:19:55] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:56 [I 2022-10-10 02:19:55] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00314 [I 2022-10-10 02:19:55] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-10 02:19:55] ...optimizers/trijoint.py.151: total_norm: 21.110239 [I 2022-10-10 02:20:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 360/2304 [I 2022-10-10 02:20:03] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:41 [I 2022-10-10 02:20:04] ...trap/engines/engine.py.229: process: 0.86184 | load: 0.00316 [I 2022-10-10 02:20:04] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-10 02:20:04] ...optimizers/trijoint.py.151: total_norm: 11.656622 [I 2022-10-10 02:20:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 370/2304 [I 2022-10-10 02:20:12] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:30 [I 2022-10-10 02:20:12] ...trap/engines/engine.py.229: process: 0.85436 | load: 0.00312 [I 2022-10-10 02:20:12] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-10 02:20:12] ...optimizers/trijoint.py.151: total_norm: 16.425913 [I 2022-10-10 02:20:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 380/2304 [I 2022-10-10 02:20:21] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:23 [I 2022-10-10 02:20:21] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00315 [I 2022-10-10 02:20:21] ...trap/engines/engine.py.230: loss: 0.15571 [I 2022-10-10 02:20:21] ...optimizers/trijoint.py.151: total_norm: 33.731064 [I 2022-10-10 02:20:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 390/2304 [I 2022-10-10 02:20:29] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:19 [I 2022-10-10 02:20:29] ...trap/engines/engine.py.229: process: 0.85598 | load: 0.00312 [I 2022-10-10 02:20:29] ...trap/engines/engine.py.230: loss: 0.15144 [I 2022-10-10 02:20:29] ...optimizers/trijoint.py.151: total_norm: 10.939298 [I 2022-10-10 02:20:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 400/2304 [I 2022-10-10 02:20:38] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:04 [I 2022-10-10 02:20:38] ...trap/engines/engine.py.229: process: 0.85744 | load: 0.00311 [I 2022-10-10 02:20:38] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-10 02:20:38] ...optimizers/trijoint.py.151: total_norm: 11.098606 [I 2022-10-10 02:20:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 410/2304 [I 2022-10-10 02:20:46] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:53 [I 2022-10-10 02:20:46] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00316 [I 2022-10-10 02:20:46] ...trap/engines/engine.py.230: loss: 0.13237 [I 2022-10-10 02:20:46] ...optimizers/trijoint.py.151: total_norm: 18.712549 [I 2022-10-10 02:20:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 420/2304 [I 2022-10-10 02:20:55] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:45 [I 2022-10-10 02:20:55] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00310 [I 2022-10-10 02:20:55] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-10 02:20:55] ...optimizers/trijoint.py.151: total_norm: 34.435482 [I 2022-10-10 02:21:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 430/2304 [I 2022-10-10 02:21:03] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:51 [I 2022-10-10 02:21:03] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00322 [I 2022-10-10 02:21:03] ...trap/engines/engine.py.230: loss: 0.16452 [I 2022-10-10 02:21:03] ...optimizers/trijoint.py.151: total_norm: 18.624792 [I 2022-10-10 02:21:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 440/2304 [I 2022-10-10 02:21:12] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:41 [I 2022-10-10 02:21:12] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00313 [I 2022-10-10 02:21:12] ...trap/engines/engine.py.230: loss: 0.13877 [I 2022-10-10 02:21:12] ...optimizers/trijoint.py.151: total_norm: 18.156208 [I 2022-10-10 02:21:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 450/2304 [I 2022-10-10 02:21:21] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:23 [I 2022-10-10 02:21:21] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00311 [I 2022-10-10 02:21:21] ...trap/engines/engine.py.230: loss: 0.10388 [I 2022-10-10 02:21:21] ...optimizers/trijoint.py.151: total_norm: 10.037176 [I 2022-10-10 02:21:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 460/2304 [I 2022-10-10 02:21:29] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:10 [I 2022-10-10 02:21:29] ...trap/engines/engine.py.229: process: 0.85586 | load: 0.00293 [I 2022-10-10 02:21:29] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-10 02:21:29] ...optimizers/trijoint.py.151: total_norm: 12.166371 [I 2022-10-10 02:21:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 470/2304 [I 2022-10-10 02:21:38] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:53 [I 2022-10-10 02:21:38] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00322 [I 2022-10-10 02:21:38] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-10 02:21:38] ...optimizers/trijoint.py.151: total_norm: 13.209527 [I 2022-10-10 02:21:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 480/2304 [I 2022-10-10 02:21:46] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:26:10 [I 2022-10-10 02:21:46] ...trap/engines/engine.py.229: process: 0.88424 | load: 0.00297 [I 2022-10-10 02:21:46] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-10 02:21:46] ...optimizers/trijoint.py.151: total_norm: 16.574654 [I 2022-10-10 02:21:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 490/2304 [I 2022-10-10 02:21:55] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:41 [I 2022-10-10 02:21:55] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00299 [I 2022-10-10 02:21:55] ...trap/engines/engine.py.230: loss: 0.14603 [I 2022-10-10 02:21:55] ...optimizers/trijoint.py.151: total_norm: 23.628031 [I 2022-10-10 02:22:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 500/2304 [I 2022-10-10 02:22:03] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:30 [I 2022-10-10 02:22:03] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00296 [I 2022-10-10 02:22:03] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-10 02:22:03] ...optimizers/trijoint.py.151: total_norm: 9.872820 [I 2022-10-10 02:22:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 510/2304 [I 2022-10-10 02:22:12] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:24 [I 2022-10-10 02:22:12] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00302 [I 2022-10-10 02:22:12] ...trap/engines/engine.py.230: loss: 0.13584 [I 2022-10-10 02:22:12] ...optimizers/trijoint.py.151: total_norm: 15.053617 [I 2022-10-10 02:22:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 520/2304 [I 2022-10-10 02:22:20] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:08 [I 2022-10-10 02:22:20] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00300 [I 2022-10-10 02:22:20] ...trap/engines/engine.py.230: loss: 0.14968 [I 2022-10-10 02:22:20] ...optimizers/trijoint.py.151: total_norm: 9.552813 [I 2022-10-10 02:22:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 530/2304 [I 2022-10-10 02:22:29] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:24:54 [I 2022-10-10 02:22:29] ...trap/engines/engine.py.229: process: 0.83171 | load: 0.00299 [I 2022-10-10 02:22:29] ...trap/engines/engine.py.230: loss: 0.17342 [I 2022-10-10 02:22:29] ...optimizers/trijoint.py.151: total_norm: 11.479799 [I 2022-10-10 02:22:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 540/2304 [I 2022-10-10 02:22:37] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:24:46 [I 2022-10-10 02:22:37] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00299 [I 2022-10-10 02:22:37] ...trap/engines/engine.py.230: loss: 0.11648 [I 2022-10-10 02:22:37] ...optimizers/trijoint.py.151: total_norm: 12.510052 [I 2022-10-10 02:22:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 550/2304 [I 2022-10-10 02:22:46] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:24:47 [I 2022-10-10 02:22:46] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00299 [I 2022-10-10 02:22:46] ...trap/engines/engine.py.230: loss: 0.13043 [I 2022-10-10 02:22:46] ...optimizers/trijoint.py.151: total_norm: 18.345104 [I 2022-10-10 02:22:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 560/2304 [I 2022-10-10 02:22:54] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:39 [I 2022-10-10 02:22:54] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00300 [I 2022-10-10 02:22:54] ...trap/engines/engine.py.230: loss: 0.09683 [I 2022-10-10 02:22:54] ...optimizers/trijoint.py.151: total_norm: 10.174962 [I 2022-10-10 02:23:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 570/2304 [I 2022-10-10 02:23:03] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:27 [I 2022-10-10 02:23:03] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00304 [I 2022-10-10 02:23:03] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-10 02:23:03] ...optimizers/trijoint.py.151: total_norm: 16.272779 [I 2022-10-10 02:23:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 580/2304 [I 2022-10-10 02:23:11] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:24 [I 2022-10-10 02:23:11] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00323 [I 2022-10-10 02:23:11] ...trap/engines/engine.py.230: loss: 0.10505 [I 2022-10-10 02:23:11] ...optimizers/trijoint.py.151: total_norm: 8.254481 [I 2022-10-10 02:23:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 590/2304 [I 2022-10-10 02:23:20] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:21 [I 2022-10-10 02:23:20] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00311 [I 2022-10-10 02:23:20] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-10 02:23:20] ...optimizers/trijoint.py.151: total_norm: 31.665131 [I 2022-10-10 02:23:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 600/2304 [I 2022-10-10 02:23:28] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:10 [I 2022-10-10 02:23:28] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00313 [I 2022-10-10 02:23:28] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-10 02:23:28] ...optimizers/trijoint.py.151: total_norm: 14.689947 [I 2022-10-10 02:23:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 610/2304 [I 2022-10-10 02:23:37] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:08 [I 2022-10-10 02:23:37] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00316 [I 2022-10-10 02:23:37] ...trap/engines/engine.py.230: loss: 0.11680 [I 2022-10-10 02:23:37] ...optimizers/trijoint.py.151: total_norm: 10.132216 [I 2022-10-10 02:23:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 620/2304 [I 2022-10-10 02:23:45] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:03 [I 2022-10-10 02:23:45] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00326 [I 2022-10-10 02:23:45] ...trap/engines/engine.py.230: loss: 0.14074 [I 2022-10-10 02:23:45] ...optimizers/trijoint.py.151: total_norm: 15.346307 [I 2022-10-10 02:23:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 630/2304 [I 2022-10-10 02:23:54] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:59 [I 2022-10-10 02:23:54] ...trap/engines/engine.py.229: process: 0.87359 | load: 0.00311 [I 2022-10-10 02:23:54] ...trap/engines/engine.py.230: loss: 0.16990 [I 2022-10-10 02:23:54] ...optimizers/trijoint.py.151: total_norm: 21.044054 [I 2022-10-10 02:24:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 640/2304 [I 2022-10-10 02:24:03] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:47 [I 2022-10-10 02:24:03] ...trap/engines/engine.py.229: process: 0.86214 | load: 0.00311 [I 2022-10-10 02:24:03] ...trap/engines/engine.py.230: loss: 0.18547 [I 2022-10-10 02:24:03] ...optimizers/trijoint.py.151: total_norm: 26.158207 [I 2022-10-10 02:24:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 650/2304 [I 2022-10-10 02:24:11] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:25 [I 2022-10-10 02:24:11] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00303 [I 2022-10-10 02:24:11] ...trap/engines/engine.py.230: loss: 0.12781 [I 2022-10-10 02:24:11] ...optimizers/trijoint.py.151: total_norm: 14.686090 [I 2022-10-10 02:24:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 660/2304 [I 2022-10-10 02:24:20] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:27 [I 2022-10-10 02:24:20] ...trap/engines/engine.py.229: process: 0.85690 | load: 0.00313 [I 2022-10-10 02:24:20] ...trap/engines/engine.py.230: loss: 0.16914 [I 2022-10-10 02:24:20] ...optimizers/trijoint.py.151: total_norm: 18.972061 [I 2022-10-10 02:24:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 670/2304 [I 2022-10-10 02:24:28] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:14 [I 2022-10-10 02:24:28] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00324 [I 2022-10-10 02:24:28] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-10 02:24:28] ...optimizers/trijoint.py.151: total_norm: 15.459207 [I 2022-10-10 02:24:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 680/2304 [I 2022-10-10 02:24:37] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:03 [I 2022-10-10 02:24:37] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00302 [I 2022-10-10 02:24:37] ...trap/engines/engine.py.230: loss: 0.13841 [I 2022-10-10 02:24:37] ...optimizers/trijoint.py.151: total_norm: 23.859650 [I 2022-10-10 02:24:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 690/2304 [I 2022-10-10 02:24:45] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:53 [I 2022-10-10 02:24:45] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00315 [I 2022-10-10 02:24:45] ...trap/engines/engine.py.230: loss: 0.17939 [I 2022-10-10 02:24:45] ...optimizers/trijoint.py.151: total_norm: 15.406285 [I 2022-10-10 02:24:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 700/2304 [I 2022-10-10 02:24:54] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:45 [I 2022-10-10 02:24:54] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00311 [I 2022-10-10 02:24:54] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-10 02:24:54] ...optimizers/trijoint.py.151: total_norm: 12.002806 [I 2022-10-10 02:25:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 710/2304 [I 2022-10-10 02:25:02] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:33 [I 2022-10-10 02:25:02] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00311 [I 2022-10-10 02:25:02] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-10 02:25:02] ...optimizers/trijoint.py.151: total_norm: 9.674700 [I 2022-10-10 02:25:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 720/2304 [I 2022-10-10 02:25:11] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:24 [I 2022-10-10 02:25:11] ...trap/engines/engine.py.229: process: 0.84065 | load: 0.00322 [I 2022-10-10 02:25:11] ...trap/engines/engine.py.230: loss: 0.13612 [I 2022-10-10 02:25:11] ...optimizers/trijoint.py.151: total_norm: 14.793997 [I 2022-10-10 02:25:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 730/2304 [I 2022-10-10 02:25:19] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:26 [I 2022-10-10 02:25:19] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00315 [I 2022-10-10 02:25:19] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-10 02:25:19] ...optimizers/trijoint.py.151: total_norm: 11.245560 [I 2022-10-10 02:25:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 740/2304 [I 2022-10-10 02:25:28] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:12 [I 2022-10-10 02:25:28] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00313 [I 2022-10-10 02:25:28] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-10 02:25:28] ...optimizers/trijoint.py.151: total_norm: 14.579113 [I 2022-10-10 02:25:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 750/2304 [I 2022-10-10 02:25:36] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:02 [I 2022-10-10 02:25:36] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00313 [I 2022-10-10 02:25:36] ...trap/engines/engine.py.230: loss: 0.14310 [I 2022-10-10 02:25:36] ...optimizers/trijoint.py.151: total_norm: 11.122117 [I 2022-10-10 02:25:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 760/2304 [I 2022-10-10 02:25:45] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:54 [I 2022-10-10 02:25:45] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00313 [I 2022-10-10 02:25:45] ...trap/engines/engine.py.230: loss: 0.17270 [I 2022-10-10 02:25:45] ...optimizers/trijoint.py.151: total_norm: 17.654541 [I 2022-10-10 02:25:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 770/2304 [I 2022-10-10 02:25:54] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:47 [I 2022-10-10 02:25:54] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00316 [I 2022-10-10 02:25:54] ...trap/engines/engine.py.230: loss: 0.13779 [I 2022-10-10 02:25:54] ...optimizers/trijoint.py.151: total_norm: 27.772129 [I 2022-10-10 02:26:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 780/2304 [I 2022-10-10 02:26:02] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:31 [I 2022-10-10 02:26:02] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00315 [I 2022-10-10 02:26:02] ...trap/engines/engine.py.230: loss: 0.16895 [I 2022-10-10 02:26:02] ...optimizers/trijoint.py.151: total_norm: 26.814861 [I 2022-10-10 02:26:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 790/2304 [I 2022-10-10 02:26:11] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:31 [I 2022-10-10 02:26:11] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00321 [I 2022-10-10 02:26:11] ...trap/engines/engine.py.230: loss: 0.11110 [I 2022-10-10 02:26:11] ...optimizers/trijoint.py.151: total_norm: 16.065769 [I 2022-10-10 02:26:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 800/2304 [I 2022-10-10 02:26:19] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:21 [I 2022-10-10 02:26:19] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00312 [I 2022-10-10 02:26:19] ...trap/engines/engine.py.230: loss: 0.16142 [I 2022-10-10 02:26:19] ...optimizers/trijoint.py.151: total_norm: 18.113186 [I 2022-10-10 02:26:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 810/2304 [I 2022-10-10 02:26:28] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:21 [I 2022-10-10 02:26:28] ...trap/engines/engine.py.229: process: 0.86257 | load: 0.00312 [I 2022-10-10 02:26:28] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-10 02:26:28] ...optimizers/trijoint.py.151: total_norm: 17.155926 [I 2022-10-10 02:26:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 820/2304 [I 2022-10-10 02:26:36] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:07 [I 2022-10-10 02:26:36] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00314 [I 2022-10-10 02:26:36] ...trap/engines/engine.py.230: loss: 0.09311 [I 2022-10-10 02:26:36] ...optimizers/trijoint.py.151: total_norm: 8.242319 [I 2022-10-10 02:26:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 830/2304 [I 2022-10-10 02:26:45] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:59 [I 2022-10-10 02:26:45] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00311 [I 2022-10-10 02:26:45] ...trap/engines/engine.py.230: loss: 0.23815 [I 2022-10-10 02:26:45] ...optimizers/trijoint.py.151: total_norm: 16.190620 [I 2022-10-10 02:26:53] ...trap/engines/engine.py.224: train: epoch 63 | batch 840/2304 [I 2022-10-10 02:26:53] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:47 [I 2022-10-10 02:26:53] ...trap/engines/engine.py.229: process: 0.85327 | load: 0.00313 [I 2022-10-10 02:26:53] ...trap/engines/engine.py.230: loss: 0.09442 [I 2022-10-10 02:26:53] ...optimizers/trijoint.py.151: total_norm: 8.820788 [I 2022-10-10 02:27:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 850/2304 [I 2022-10-10 02:27:02] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:37 [I 2022-10-10 02:27:02] ...trap/engines/engine.py.229: process: 0.83762 | load: 0.00317 [I 2022-10-10 02:27:02] ...trap/engines/engine.py.230: loss: 0.10412 [I 2022-10-10 02:27:02] ...optimizers/trijoint.py.151: total_norm: 13.203398 [I 2022-10-10 02:27:10] ...trap/engines/engine.py.224: train: epoch 63 | batch 860/2304 [I 2022-10-10 02:27:10] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:26 [I 2022-10-10 02:27:10] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00313 [I 2022-10-10 02:27:10] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-10 02:27:10] ...optimizers/trijoint.py.151: total_norm: 8.296519 [I 2022-10-10 02:27:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 870/2304 [I 2022-10-10 02:27:19] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:17 [I 2022-10-10 02:27:19] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00303 [I 2022-10-10 02:27:19] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-10 02:27:19] ...optimizers/trijoint.py.151: total_norm: 18.319246 [I 2022-10-10 02:27:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 880/2304 [I 2022-10-10 02:27:28] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:19 [I 2022-10-10 02:27:28] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00308 [I 2022-10-10 02:27:28] ...trap/engines/engine.py.230: loss: 0.11904 [I 2022-10-10 02:27:28] ...optimizers/trijoint.py.151: total_norm: 11.743505 [I 2022-10-10 02:27:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 890/2304 [I 2022-10-10 02:27:36] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:08 [I 2022-10-10 02:27:36] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00317 [I 2022-10-10 02:27:36] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-10 02:27:36] ...optimizers/trijoint.py.151: total_norm: 17.001528 [I 2022-10-10 02:27:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 900/2304 [I 2022-10-10 02:27:45] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:54 [I 2022-10-10 02:27:45] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00312 [I 2022-10-10 02:27:45] ...trap/engines/engine.py.230: loss: 0.15730 [I 2022-10-10 02:27:45] ...optimizers/trijoint.py.151: total_norm: 15.407761 [I 2022-10-10 02:27:53] ...trap/engines/engine.py.224: train: epoch 63 | batch 910/2304 [I 2022-10-10 02:27:53] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:41 [I 2022-10-10 02:27:53] ...trap/engines/engine.py.229: process: 0.84804 | load: 0.00299 [I 2022-10-10 02:27:53] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-10 02:27:53] ...optimizers/trijoint.py.151: total_norm: 11.983535 [I 2022-10-10 02:28:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 920/2304 [I 2022-10-10 02:28:02] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:33 [I 2022-10-10 02:28:02] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00297 [I 2022-10-10 02:28:02] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-10 02:28:02] ...optimizers/trijoint.py.151: total_norm: 8.350655 [I 2022-10-10 02:28:10] ...trap/engines/engine.py.224: train: epoch 63 | batch 930/2304 [I 2022-10-10 02:28:10] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:24 [I 2022-10-10 02:28:10] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00300 [I 2022-10-10 02:28:10] ...trap/engines/engine.py.230: loss: 0.12041 [I 2022-10-10 02:28:10] ...optimizers/trijoint.py.151: total_norm: 11.098938 [I 2022-10-10 02:28:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 940/2304 [I 2022-10-10 02:28:19] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:09 [I 2022-10-10 02:28:19] ...trap/engines/engine.py.229: process: 0.83509 | load: 0.00298 [I 2022-10-10 02:28:19] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-10 02:28:19] ...optimizers/trijoint.py.151: total_norm: 24.403561 [I 2022-10-10 02:28:27] ...trap/engines/engine.py.224: train: epoch 63 | batch 950/2304 [I 2022-10-10 02:28:27] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:04 [I 2022-10-10 02:28:27] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00298 [I 2022-10-10 02:28:27] ...trap/engines/engine.py.230: loss: 0.10769 [I 2022-10-10 02:28:27] ...optimizers/trijoint.py.151: total_norm: 12.273179 [I 2022-10-10 02:28:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 960/2304 [I 2022-10-10 02:28:36] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:57 [I 2022-10-10 02:28:36] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00299 [I 2022-10-10 02:28:36] ...trap/engines/engine.py.230: loss: 0.13064 [I 2022-10-10 02:28:36] ...optimizers/trijoint.py.151: total_norm: 10.073290 [I 2022-10-10 02:28:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 970/2304 [I 2022-10-10 02:28:44] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:48 [I 2022-10-10 02:28:44] ...trap/engines/engine.py.229: process: 0.85991 | load: 0.00301 [I 2022-10-10 02:28:44] ...trap/engines/engine.py.230: loss: 0.11750 [I 2022-10-10 02:28:44] ...optimizers/trijoint.py.151: total_norm: 16.530416 [I 2022-10-10 02:28:52] ...trap/engines/engine.py.224: train: epoch 63 | batch 980/2304 [I 2022-10-10 02:28:52] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:34 [I 2022-10-10 02:28:52] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00297 [I 2022-10-10 02:28:52] ...trap/engines/engine.py.230: loss: 0.16009 [I 2022-10-10 02:28:52] ...optimizers/trijoint.py.151: total_norm: 33.734928 [I 2022-10-10 02:29:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 990/2304 [I 2022-10-10 02:29:01] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:31 [I 2022-10-10 02:29:01] ...trap/engines/engine.py.229: process: 0.84068 | load: 0.00298 [I 2022-10-10 02:29:01] ...trap/engines/engine.py.230: loss: 0.10219 [I 2022-10-10 02:29:01] ...optimizers/trijoint.py.151: total_norm: 12.752493 [I 2022-10-10 02:29:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 1000/2304 [I 2022-10-10 02:29:09] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:16 [I 2022-10-10 02:29:09] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00299 [I 2022-10-10 02:29:09] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-10 02:29:09] ...optimizers/trijoint.py.151: total_norm: 25.041374 [I 2022-10-10 02:29:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 1010/2304 [I 2022-10-10 02:29:18] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:07 [I 2022-10-10 02:29:18] ...trap/engines/engine.py.229: process: 0.83043 | load: 0.00300 [I 2022-10-10 02:29:18] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-10 02:29:18] ...optimizers/trijoint.py.151: total_norm: 11.411458 [I 2022-10-10 02:29:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 1020/2304 [I 2022-10-10 02:29:26] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:06 [I 2022-10-10 02:29:26] ...trap/engines/engine.py.229: process: 0.84176 | load: 0.00304 [I 2022-10-10 02:29:26] ...trap/engines/engine.py.230: loss: 0.12835 [I 2022-10-10 02:29:26] ...optimizers/trijoint.py.151: total_norm: 16.482073 [I 2022-10-10 02:29:35] ...trap/engines/engine.py.224: train: epoch 63 | batch 1030/2304 [I 2022-10-10 02:29:35] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:17:58 [I 2022-10-10 02:29:35] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00299 [I 2022-10-10 02:29:35] ...trap/engines/engine.py.230: loss: 0.10693 [I 2022-10-10 02:29:35] ...optimizers/trijoint.py.151: total_norm: 14.804797 [I 2022-10-10 02:29:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 1040/2304 [I 2022-10-10 02:29:43] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:47 [I 2022-10-10 02:29:43] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00303 [I 2022-10-10 02:29:43] ...trap/engines/engine.py.230: loss: 0.17411 [I 2022-10-10 02:29:43] ...optimizers/trijoint.py.151: total_norm: 14.227813 [I 2022-10-10 02:29:52] ...trap/engines/engine.py.224: train: epoch 63 | batch 1050/2304 [I 2022-10-10 02:29:52] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:41 [I 2022-10-10 02:29:52] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00299 [I 2022-10-10 02:29:52] ...trap/engines/engine.py.230: loss: 0.13631 [I 2022-10-10 02:29:52] ...optimizers/trijoint.py.151: total_norm: 16.722832 [I 2022-10-10 02:30:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 1060/2304 [I 2022-10-10 02:30:00] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:32 [I 2022-10-10 02:30:00] ...trap/engines/engine.py.229: process: 0.84395 | load: 0.00301 [I 2022-10-10 02:30:00] ...trap/engines/engine.py.230: loss: 0.10684 [I 2022-10-10 02:30:00] ...optimizers/trijoint.py.151: total_norm: 10.032313 [I 2022-10-10 02:30:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 1070/2304 [I 2022-10-10 02:30:09] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:24 [I 2022-10-10 02:30:09] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00298 [I 2022-10-10 02:30:09] ...trap/engines/engine.py.230: loss: 0.11862 [I 2022-10-10 02:30:09] ...optimizers/trijoint.py.151: total_norm: 9.233874 [I 2022-10-10 02:30:17] ...trap/engines/engine.py.224: train: epoch 63 | batch 1080/2304 [I 2022-10-10 02:30:17] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:14 [I 2022-10-10 02:30:17] ...trap/engines/engine.py.229: process: 0.83297 | load: 0.00312 [I 2022-10-10 02:30:17] ...trap/engines/engine.py.230: loss: 0.17973 [I 2022-10-10 02:30:17] ...optimizers/trijoint.py.151: total_norm: 21.942404 [I 2022-10-10 02:30:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 1090/2304 [I 2022-10-10 02:30:26] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:08 [I 2022-10-10 02:30:26] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00312 [I 2022-10-10 02:30:26] ...trap/engines/engine.py.230: loss: 0.12688 [I 2022-10-10 02:30:26] ...optimizers/trijoint.py.151: total_norm: 15.478366 [I 2022-10-10 02:30:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1100/2304 [I 2022-10-10 02:30:34] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:09 [I 2022-10-10 02:30:34] ...trap/engines/engine.py.229: process: 0.85565 | load: 0.00311 [I 2022-10-10 02:30:34] ...trap/engines/engine.py.230: loss: 0.10797 [I 2022-10-10 02:30:34] ...optimizers/trijoint.py.151: total_norm: 8.473579 [I 2022-10-10 02:30:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 1110/2304 [I 2022-10-10 02:30:43] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:54 [I 2022-10-10 02:30:43] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00310 [I 2022-10-10 02:30:43] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-10 02:30:43] ...optimizers/trijoint.py.151: total_norm: 17.025846 [I 2022-10-10 02:30:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1120/2304 [I 2022-10-10 02:30:51] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:50 [I 2022-10-10 02:30:51] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00312 [I 2022-10-10 02:30:51] ...trap/engines/engine.py.230: loss: 0.24574 [I 2022-10-10 02:30:51] ...optimizers/trijoint.py.151: total_norm: 22.202520 [I 2022-10-10 02:31:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 1130/2304 [I 2022-10-10 02:31:00] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:42 [I 2022-10-10 02:31:00] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00312 [I 2022-10-10 02:31:00] ...trap/engines/engine.py.230: loss: 0.16521 [I 2022-10-10 02:31:00] ...optimizers/trijoint.py.151: total_norm: 48.069767 [I 2022-10-10 02:31:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 1140/2304 [I 2022-10-10 02:31:09] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:34 [I 2022-10-10 02:31:09] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00312 [I 2022-10-10 02:31:09] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-10 02:31:09] ...optimizers/trijoint.py.151: total_norm: 12.004461 [I 2022-10-10 02:31:17] ...trap/engines/engine.py.224: train: epoch 63 | batch 1150/2304 [I 2022-10-10 02:31:17] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:24 [I 2022-10-10 02:31:17] ...trap/engines/engine.py.229: process: 0.86534 | load: 0.00315 [I 2022-10-10 02:31:17] ...trap/engines/engine.py.230: loss: 0.18932 [I 2022-10-10 02:31:17] ...optimizers/trijoint.py.151: total_norm: 31.804729 [I 2022-10-10 02:31:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 1160/2304 [I 2022-10-10 02:31:26] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:22 [I 2022-10-10 02:31:26] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00299 [I 2022-10-10 02:31:26] ...trap/engines/engine.py.230: loss: 0.13416 [I 2022-10-10 02:31:26] ...optimizers/trijoint.py.151: total_norm: 34.580418 [I 2022-10-10 02:31:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1170/2304 [I 2022-10-10 02:31:34] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:09 [I 2022-10-10 02:31:34] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00313 [I 2022-10-10 02:31:34] ...trap/engines/engine.py.230: loss: 0.17493 [I 2022-10-10 02:31:34] ...optimizers/trijoint.py.151: total_norm: 12.501925 [I 2022-10-10 02:31:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 1180/2304 [I 2022-10-10 02:31:43] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:59 [I 2022-10-10 02:31:43] ...trap/engines/engine.py.229: process: 0.85149 | load: 0.00320 [I 2022-10-10 02:31:43] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-10 02:31:43] ...optimizers/trijoint.py.151: total_norm: 12.235477 [I 2022-10-10 02:31:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1190/2304 [I 2022-10-10 02:31:51] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:44 [I 2022-10-10 02:31:51] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00314 [I 2022-10-10 02:31:51] ...trap/engines/engine.py.230: loss: 0.21892 [I 2022-10-10 02:31:51] ...optimizers/trijoint.py.151: total_norm: 13.280239 [I 2022-10-10 02:32:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 1200/2304 [I 2022-10-10 02:32:00] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:45 [I 2022-10-10 02:32:00] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00313 [I 2022-10-10 02:32:00] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-10 02:32:00] ...optimizers/trijoint.py.151: total_norm: 16.333513 [I 2022-10-10 02:32:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1210/2304 [I 2022-10-10 02:32:08] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:28 [I 2022-10-10 02:32:08] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00311 [I 2022-10-10 02:32:08] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-10 02:32:08] ...optimizers/trijoint.py.151: total_norm: 14.271097 [I 2022-10-10 02:32:17] ...trap/engines/engine.py.224: train: epoch 63 | batch 1220/2304 [I 2022-10-10 02:32:17] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:21 [I 2022-10-10 02:32:17] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00311 [I 2022-10-10 02:32:17] ...trap/engines/engine.py.230: loss: 0.16635 [I 2022-10-10 02:32:17] ...optimizers/trijoint.py.151: total_norm: 15.955283 [I 2022-10-10 02:32:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 1230/2304 [I 2022-10-10 02:32:25] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:11 [I 2022-10-10 02:32:25] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00316 [I 2022-10-10 02:32:25] ...trap/engines/engine.py.230: loss: 0.11946 [I 2022-10-10 02:32:25] ...optimizers/trijoint.py.151: total_norm: 27.882160 [I 2022-10-10 02:32:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1240/2304 [I 2022-10-10 02:32:34] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:08 [I 2022-10-10 02:32:34] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00312 [I 2022-10-10 02:32:34] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-10 02:32:34] ...optimizers/trijoint.py.151: total_norm: 17.571690 [I 2022-10-10 02:32:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 1250/2304 [I 2022-10-10 02:32:43] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:04 [I 2022-10-10 02:32:43] ...trap/engines/engine.py.229: process: 0.85880 | load: 0.00319 [I 2022-10-10 02:32:43] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-10 02:32:43] ...optimizers/trijoint.py.151: total_norm: 15.238085 [I 2022-10-10 02:32:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1260/2304 [I 2022-10-10 02:32:51] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:48 [I 2022-10-10 02:32:51] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00311 [I 2022-10-10 02:32:51] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-10 02:32:51] ...optimizers/trijoint.py.151: total_norm: 16.365471 [I 2022-10-10 02:33:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 1270/2304 [I 2022-10-10 02:33:00] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:43 [I 2022-10-10 02:33:00] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00311 [I 2022-10-10 02:33:00] ...trap/engines/engine.py.230: loss: 0.11804 [I 2022-10-10 02:33:00] ...optimizers/trijoint.py.151: total_norm: 11.020351 [I 2022-10-10 02:33:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1280/2304 [I 2022-10-10 02:33:08] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:30 [I 2022-10-10 02:33:08] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00311 [I 2022-10-10 02:33:08] ...trap/engines/engine.py.230: loss: 0.10170 [I 2022-10-10 02:33:08] ...optimizers/trijoint.py.151: total_norm: 7.029110 [I 2022-10-10 02:33:17] ...trap/engines/engine.py.224: train: epoch 63 | batch 1290/2304 [I 2022-10-10 02:33:17] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:20 [I 2022-10-10 02:33:17] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00312 [I 2022-10-10 02:33:17] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-10 02:33:17] ...optimizers/trijoint.py.151: total_norm: 11.117990 [I 2022-10-10 02:33:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 1300/2304 [I 2022-10-10 02:33:25] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:09 [I 2022-10-10 02:33:25] ...trap/engines/engine.py.229: process: 0.83915 | load: 0.00308 [I 2022-10-10 02:33:25] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-10 02:33:25] ...optimizers/trijoint.py.151: total_norm: 13.499894 [I 2022-10-10 02:33:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1310/2304 [I 2022-10-10 02:33:34] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:05 [I 2022-10-10 02:33:34] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00312 [I 2022-10-10 02:33:34] ...trap/engines/engine.py.230: loss: 0.17744 [I 2022-10-10 02:33:34] ...optimizers/trijoint.py.151: total_norm: 14.001501 [I 2022-10-10 02:33:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 1320/2304 [I 2022-10-10 02:33:42] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:00 [I 2022-10-10 02:33:42] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00321 [I 2022-10-10 02:33:42] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-10 02:33:42] ...optimizers/trijoint.py.151: total_norm: 10.307749 [I 2022-10-10 02:33:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1330/2304 [I 2022-10-10 02:33:51] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:58 [I 2022-10-10 02:33:51] ...trap/engines/engine.py.229: process: 0.86712 | load: 0.00315 [I 2022-10-10 02:33:51] ...trap/engines/engine.py.230: loss: 0.11240 [I 2022-10-10 02:33:51] ...optimizers/trijoint.py.151: total_norm: 7.598876 [I 2022-10-10 02:33:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 1340/2304 [I 2022-10-10 02:33:59] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:40 [I 2022-10-10 02:33:59] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00311 [I 2022-10-10 02:33:59] ...trap/engines/engine.py.230: loss: 0.12723 [I 2022-10-10 02:33:59] ...optimizers/trijoint.py.151: total_norm: 18.290653 [I 2022-10-10 02:34:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1350/2304 [I 2022-10-10 02:34:08] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:13:33 [I 2022-10-10 02:34:08] ...trap/engines/engine.py.229: process: 0.84785 | load: 0.00314 [I 2022-10-10 02:34:08] ...trap/engines/engine.py.230: loss: 0.14589 [I 2022-10-10 02:34:08] ...optimizers/trijoint.py.151: total_norm: 10.989474 [I 2022-10-10 02:34:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1360/2304 [I 2022-10-10 02:34:16] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:24 [I 2022-10-10 02:34:16] ...trap/engines/engine.py.229: process: 0.86543 | load: 0.00313 [I 2022-10-10 02:34:16] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-10 02:34:16] ...optimizers/trijoint.py.151: total_norm: 16.055614 [I 2022-10-10 02:34:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 1370/2304 [I 2022-10-10 02:34:25] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:17 [I 2022-10-10 02:34:25] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00305 [I 2022-10-10 02:34:25] ...trap/engines/engine.py.230: loss: 0.20379 [I 2022-10-10 02:34:25] ...optimizers/trijoint.py.151: total_norm: 15.714057 [I 2022-10-10 02:34:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1380/2304 [I 2022-10-10 02:34:34] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:06 [I 2022-10-10 02:34:34] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00314 [I 2022-10-10 02:34:34] ...trap/engines/engine.py.230: loss: 0.15490 [I 2022-10-10 02:34:34] ...optimizers/trijoint.py.151: total_norm: 12.794585 [I 2022-10-10 02:34:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 1390/2304 [I 2022-10-10 02:34:42] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:01 [I 2022-10-10 02:34:42] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00309 [I 2022-10-10 02:34:42] ...trap/engines/engine.py.230: loss: 0.20341 [I 2022-10-10 02:34:42] ...optimizers/trijoint.py.151: total_norm: 15.770559 [I 2022-10-10 02:34:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1400/2304 [I 2022-10-10 02:34:51] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:54 [I 2022-10-10 02:34:51] ...trap/engines/engine.py.229: process: 0.86595 | load: 0.00312 [I 2022-10-10 02:34:51] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-10 02:34:51] ...optimizers/trijoint.py.151: total_norm: 8.025444 [I 2022-10-10 02:34:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 1410/2304 [I 2022-10-10 02:34:59] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:12:40 [I 2022-10-10 02:34:59] ...trap/engines/engine.py.229: process: 0.83950 | load: 0.00296 [I 2022-10-10 02:34:59] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-10 02:34:59] ...optimizers/trijoint.py.151: total_norm: 14.194364 [I 2022-10-10 02:35:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1420/2304 [I 2022-10-10 02:35:08] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:28 [I 2022-10-10 02:35:08] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00296 [I 2022-10-10 02:35:08] ...trap/engines/engine.py.230: loss: 0.17086 [I 2022-10-10 02:35:08] ...optimizers/trijoint.py.151: total_norm: 15.575769 [I 2022-10-10 02:35:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1430/2304 [I 2022-10-10 02:35:16] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:20 [I 2022-10-10 02:35:16] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00294 [I 2022-10-10 02:35:16] ...trap/engines/engine.py.230: loss: 0.10821 [I 2022-10-10 02:35:16] ...optimizers/trijoint.py.151: total_norm: 7.552053 [I 2022-10-10 02:35:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 1440/2304 [I 2022-10-10 02:35:25] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:11 [I 2022-10-10 02:35:25] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00298 [I 2022-10-10 02:35:25] ...trap/engines/engine.py.230: loss: 0.10166 [I 2022-10-10 02:35:25] ...optimizers/trijoint.py.151: total_norm: 7.778648 [I 2022-10-10 02:35:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 1450/2304 [I 2022-10-10 02:35:33] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:03 [I 2022-10-10 02:35:33] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00296 [I 2022-10-10 02:35:33] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-10 02:35:33] ...optimizers/trijoint.py.151: total_norm: 10.852446 [I 2022-10-10 02:35:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 1460/2304 [I 2022-10-10 02:35:42] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:11:57 [I 2022-10-10 02:35:42] ...trap/engines/engine.py.229: process: 0.86001 | load: 0.00296 [I 2022-10-10 02:35:42] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-10 02:35:42] ...optimizers/trijoint.py.151: total_norm: 11.134580 [I 2022-10-10 02:35:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 1470/2304 [I 2022-10-10 02:35:50] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:46 [I 2022-10-10 02:35:50] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00298 [I 2022-10-10 02:35:50] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-10 02:35:50] ...optimizers/trijoint.py.151: total_norm: 14.669425 [I 2022-10-10 02:35:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 1480/2304 [I 2022-10-10 02:35:59] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:41 [I 2022-10-10 02:35:59] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00298 [I 2022-10-10 02:35:59] ...trap/engines/engine.py.230: loss: 0.16139 [I 2022-10-10 02:35:59] ...optimizers/trijoint.py.151: total_norm: 13.643023 [I 2022-10-10 02:36:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 1490/2304 [I 2022-10-10 02:36:07] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:31 [I 2022-10-10 02:36:07] ...trap/engines/engine.py.229: process: 0.84039 | load: 0.00296 [I 2022-10-10 02:36:07] ...trap/engines/engine.py.230: loss: 0.21056 [I 2022-10-10 02:36:07] ...optimizers/trijoint.py.151: total_norm: 38.110611 [I 2022-10-10 02:36:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1500/2304 [I 2022-10-10 02:36:16] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:19 [I 2022-10-10 02:36:16] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00298 [I 2022-10-10 02:36:16] ...trap/engines/engine.py.230: loss: 0.13512 [I 2022-10-10 02:36:16] ...optimizers/trijoint.py.151: total_norm: 12.267962 [I 2022-10-10 02:36:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 1510/2304 [I 2022-10-10 02:36:24] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:12 [I 2022-10-10 02:36:24] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00310 [I 2022-10-10 02:36:24] ...trap/engines/engine.py.230: loss: 0.14983 [I 2022-10-10 02:36:24] ...optimizers/trijoint.py.151: total_norm: 34.902271 [I 2022-10-10 02:36:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 1520/2304 [I 2022-10-10 02:36:33] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:01 [I 2022-10-10 02:36:33] ...trap/engines/engine.py.229: process: 0.85315 | load: 0.00299 [I 2022-10-10 02:36:33] ...trap/engines/engine.py.230: loss: 0.19431 [I 2022-10-10 02:36:33] ...optimizers/trijoint.py.151: total_norm: 13.660396 [I 2022-10-10 02:36:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1530/2304 [I 2022-10-10 02:36:41] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:10:54 [I 2022-10-10 02:36:41] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00298 [I 2022-10-10 02:36:41] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-10 02:36:41] ...optimizers/trijoint.py.151: total_norm: 14.860535 [I 2022-10-10 02:36:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 1540/2304 [I 2022-10-10 02:36:50] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:44 [I 2022-10-10 02:36:50] ...trap/engines/engine.py.229: process: 0.85323 | load: 0.00299 [I 2022-10-10 02:36:50] ...trap/engines/engine.py.230: loss: 0.15667 [I 2022-10-10 02:36:50] ...optimizers/trijoint.py.151: total_norm: 13.501799 [I 2022-10-10 02:36:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1550/2304 [I 2022-10-10 02:36:58] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:37 [I 2022-10-10 02:36:58] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00297 [I 2022-10-10 02:36:58] ...trap/engines/engine.py.230: loss: 0.09845 [I 2022-10-10 02:36:58] ...optimizers/trijoint.py.151: total_norm: 8.747252 [I 2022-10-10 02:37:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 1560/2304 [I 2022-10-10 02:37:07] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:31 [I 2022-10-10 02:37:07] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00295 [I 2022-10-10 02:37:07] ...trap/engines/engine.py.230: loss: 0.12011 [I 2022-10-10 02:37:07] ...optimizers/trijoint.py.151: total_norm: 16.559395 [I 2022-10-10 02:37:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1570/2304 [I 2022-10-10 02:37:15] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:18 [I 2022-10-10 02:37:15] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00301 [I 2022-10-10 02:37:15] ...trap/engines/engine.py.230: loss: 0.14384 [I 2022-10-10 02:37:15] ...optimizers/trijoint.py.151: total_norm: 13.981226 [I 2022-10-10 02:37:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1580/2304 [I 2022-10-10 02:37:23] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:11 [I 2022-10-10 02:37:23] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00303 [I 2022-10-10 02:37:23] ...trap/engines/engine.py.230: loss: 0.18241 [I 2022-10-10 02:37:23] ...optimizers/trijoint.py.151: total_norm: 13.584995 [I 2022-10-10 02:37:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1590/2304 [I 2022-10-10 02:37:32] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:07 [I 2022-10-10 02:37:32] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00317 [I 2022-10-10 02:37:32] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-10 02:37:32] ...optimizers/trijoint.py.151: total_norm: 36.102673 [I 2022-10-10 02:37:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1600/2304 [I 2022-10-10 02:37:41] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:02 [I 2022-10-10 02:37:41] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00319 [I 2022-10-10 02:37:41] ...trap/engines/engine.py.230: loss: 0.24407 [I 2022-10-10 02:37:41] ...optimizers/trijoint.py.151: total_norm: 19.062160 [I 2022-10-10 02:37:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 1610/2304 [I 2022-10-10 02:37:49] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:53 [I 2022-10-10 02:37:49] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00315 [I 2022-10-10 02:37:49] ...trap/engines/engine.py.230: loss: 0.16413 [I 2022-10-10 02:37:49] ...optimizers/trijoint.py.151: total_norm: 18.146967 [I 2022-10-10 02:37:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1620/2304 [I 2022-10-10 02:37:58] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:43 [I 2022-10-10 02:37:58] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00314 [I 2022-10-10 02:37:58] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-10 02:37:58] ...optimizers/trijoint.py.151: total_norm: 8.769213 [I 2022-10-10 02:38:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1630/2304 [I 2022-10-10 02:38:06] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:34 [I 2022-10-10 02:38:06] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00297 [I 2022-10-10 02:38:06] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-10 02:38:06] ...optimizers/trijoint.py.151: total_norm: 12.402960 [I 2022-10-10 02:38:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1640/2304 [I 2022-10-10 02:38:15] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:25 [I 2022-10-10 02:38:15] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00311 [I 2022-10-10 02:38:15] ...trap/engines/engine.py.230: loss: 0.14280 [I 2022-10-10 02:38:15] ...optimizers/trijoint.py.151: total_norm: 19.820848 [I 2022-10-10 02:38:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1650/2304 [I 2022-10-10 02:38:23] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:18 [I 2022-10-10 02:38:23] ...trap/engines/engine.py.229: process: 0.85444 | load: 0.00314 [I 2022-10-10 02:38:23] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-10 02:38:23] ...optimizers/trijoint.py.151: total_norm: 15.629719 [I 2022-10-10 02:38:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1660/2304 [I 2022-10-10 02:38:32] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:12 [I 2022-10-10 02:38:32] ...trap/engines/engine.py.229: process: 0.85190 | load: 0.00311 [I 2022-10-10 02:38:32] ...trap/engines/engine.py.230: loss: 0.23453 [I 2022-10-10 02:38:32] ...optimizers/trijoint.py.151: total_norm: 18.901726 [I 2022-10-10 02:38:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 1670/2304 [I 2022-10-10 02:38:40] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:01 [I 2022-10-10 02:38:40] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00310 [I 2022-10-10 02:38:40] ...trap/engines/engine.py.230: loss: 0.16361 [I 2022-10-10 02:38:40] ...optimizers/trijoint.py.151: total_norm: 14.129827 [I 2022-10-10 02:38:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 1680/2304 [I 2022-10-10 02:38:49] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:50 [I 2022-10-10 02:38:49] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00308 [I 2022-10-10 02:38:49] ...trap/engines/engine.py.230: loss: 0.13921 [I 2022-10-10 02:38:49] ...optimizers/trijoint.py.151: total_norm: 17.773521 [I 2022-10-10 02:38:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 1690/2304 [I 2022-10-10 02:38:57] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:42 [I 2022-10-10 02:38:57] ...trap/engines/engine.py.229: process: 0.84772 | load: 0.00312 [I 2022-10-10 02:38:57] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-10 02:38:57] ...optimizers/trijoint.py.151: total_norm: 13.632441 [I 2022-10-10 02:39:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1700/2304 [I 2022-10-10 02:39:06] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:34 [I 2022-10-10 02:39:06] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00311 [I 2022-10-10 02:39:06] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-10 02:39:06] ...optimizers/trijoint.py.151: total_norm: 13.886927 [I 2022-10-10 02:39:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1710/2304 [I 2022-10-10 02:39:15] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:28 [I 2022-10-10 02:39:15] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00306 [I 2022-10-10 02:39:15] ...trap/engines/engine.py.230: loss: 0.15573 [I 2022-10-10 02:39:15] ...optimizers/trijoint.py.151: total_norm: 17.584179 [I 2022-10-10 02:39:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1720/2304 [I 2022-10-10 02:39:23] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:15 [I 2022-10-10 02:39:23] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00313 [I 2022-10-10 02:39:23] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-10 02:39:23] ...optimizers/trijoint.py.151: total_norm: 17.803406 [I 2022-10-10 02:39:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1730/2304 [I 2022-10-10 02:39:32] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:06 [I 2022-10-10 02:39:32] ...trap/engines/engine.py.229: process: 0.83863 | load: 0.00310 [I 2022-10-10 02:39:32] ...trap/engines/engine.py.230: loss: 0.18496 [I 2022-10-10 02:39:32] ...optimizers/trijoint.py.151: total_norm: 24.849497 [I 2022-10-10 02:39:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 1740/2304 [I 2022-10-10 02:39:40] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:01 [I 2022-10-10 02:39:40] ...trap/engines/engine.py.229: process: 0.85465 | load: 0.00310 [I 2022-10-10 02:39:40] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-10 02:39:40] ...optimizers/trijoint.py.151: total_norm: 6.709634 [I 2022-10-10 02:39:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 1750/2304 [I 2022-10-10 02:39:49] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:53 [I 2022-10-10 02:39:49] ...trap/engines/engine.py.229: process: 0.86358 | load: 0.00312 [I 2022-10-10 02:39:49] ...trap/engines/engine.py.230: loss: 0.12628 [I 2022-10-10 02:39:49] ...optimizers/trijoint.py.151: total_norm: 12.902667 [I 2022-10-10 02:39:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 1760/2304 [I 2022-10-10 02:39:57] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:44 [I 2022-10-10 02:39:57] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00313 [I 2022-10-10 02:39:57] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-10 02:39:57] ...optimizers/trijoint.py.151: total_norm: 23.330322 [I 2022-10-10 02:40:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1770/2304 [I 2022-10-10 02:40:06] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:34 [I 2022-10-10 02:40:06] ...trap/engines/engine.py.229: process: 0.85635 | load: 0.00311 [I 2022-10-10 02:40:06] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-10 02:40:06] ...optimizers/trijoint.py.151: total_norm: 9.284774 [I 2022-10-10 02:40:14] ...trap/engines/engine.py.224: train: epoch 63 | batch 1780/2304 [I 2022-10-10 02:40:14] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:29 [I 2022-10-10 02:40:14] ...trap/engines/engine.py.229: process: 0.86720 | load: 0.00309 [I 2022-10-10 02:40:14] ...trap/engines/engine.py.230: loss: 0.15625 [I 2022-10-10 02:40:14] ...optimizers/trijoint.py.151: total_norm: 14.795397 [I 2022-10-10 02:40:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1790/2304 [I 2022-10-10 02:40:23] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:19 [I 2022-10-10 02:40:23] ...trap/engines/engine.py.229: process: 0.86131 | load: 0.00311 [I 2022-10-10 02:40:23] ...trap/engines/engine.py.230: loss: 0.14634 [I 2022-10-10 02:40:23] ...optimizers/trijoint.py.151: total_norm: 14.852710 [I 2022-10-10 02:40:31] ...trap/engines/engine.py.224: train: epoch 63 | batch 1800/2304 [I 2022-10-10 02:40:31] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:09 [I 2022-10-10 02:40:31] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00314 [I 2022-10-10 02:40:31] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-10 02:40:31] ...optimizers/trijoint.py.151: total_norm: 41.659546 [I 2022-10-10 02:40:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 1810/2304 [I 2022-10-10 02:40:59] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:22:42 [I 2022-10-10 02:40:59] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00315 [I 2022-10-10 02:40:59] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-10 02:40:59] ...optimizers/trijoint.py.151: total_norm: 11.961671 [I 2022-10-10 02:41:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 1820/2304 [I 2022-10-10 02:41:07] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:08:33 [I 2022-10-10 02:41:07] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00314 [I 2022-10-10 02:41:07] ...trap/engines/engine.py.230: loss: 0.12300 [I 2022-10-10 02:41:07] ...optimizers/trijoint.py.151: total_norm: 15.270268 [I 2022-10-10 02:41:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1830/2304 [I 2022-10-10 02:41:16] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:55 [I 2022-10-10 02:41:16] ...trap/engines/engine.py.229: process: 0.84836 | load: 0.00307 [I 2022-10-10 02:41:16] ...trap/engines/engine.py.230: loss: 0.29541 [I 2022-10-10 02:41:16] ...optimizers/trijoint.py.151: total_norm: 14.266824 [I 2022-10-10 02:41:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 1840/2304 [I 2022-10-10 02:41:24] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:35 [I 2022-10-10 02:41:24] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00315 [I 2022-10-10 02:41:24] ...trap/engines/engine.py.230: loss: 0.12686 [I 2022-10-10 02:41:24] ...optimizers/trijoint.py.151: total_norm: 9.377592 [I 2022-10-10 02:41:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 1850/2304 [I 2022-10-10 02:41:33] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:26 [I 2022-10-10 02:41:33] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00311 [I 2022-10-10 02:41:33] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-10 02:41:33] ...optimizers/trijoint.py.151: total_norm: 10.794941 [I 2022-10-10 02:41:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1860/2304 [I 2022-10-10 02:41:41] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:17 [I 2022-10-10 02:41:41] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00313 [I 2022-10-10 02:41:41] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-10 02:41:41] ...optimizers/trijoint.py.151: total_norm: 7.385959 [I 2022-10-10 02:41:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 1870/2304 [I 2022-10-10 02:41:50] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:10 [I 2022-10-10 02:41:50] ...trap/engines/engine.py.229: process: 0.85371 | load: 0.00312 [I 2022-10-10 02:41:50] ...trap/engines/engine.py.230: loss: 0.21422 [I 2022-10-10 02:41:50] ...optimizers/trijoint.py.151: total_norm: 12.404412 [I 2022-10-10 02:41:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1880/2304 [I 2022-10-10 02:41:58] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:06:00 [I 2022-10-10 02:41:58] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00298 [I 2022-10-10 02:41:58] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-10 02:41:58] ...optimizers/trijoint.py.151: total_norm: 16.490438 [I 2022-10-10 02:42:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 1890/2304 [I 2022-10-10 02:42:07] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:52 [I 2022-10-10 02:42:07] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00310 [I 2022-10-10 02:42:07] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-10 02:42:07] ...optimizers/trijoint.py.151: total_norm: 14.744444 [I 2022-10-10 02:42:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1900/2304 [I 2022-10-10 02:42:15] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:42 [I 2022-10-10 02:42:15] ...trap/engines/engine.py.229: process: 0.85157 | load: 0.00301 [I 2022-10-10 02:42:15] ...trap/engines/engine.py.230: loss: 0.16069 [I 2022-10-10 02:42:15] ...optimizers/trijoint.py.151: total_norm: 16.568678 [I 2022-10-10 02:42:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 1910/2304 [I 2022-10-10 02:42:24] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:33 [I 2022-10-10 02:42:24] ...trap/engines/engine.py.229: process: 0.85124 | load: 0.00306 [I 2022-10-10 02:42:24] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-10 02:42:24] ...optimizers/trijoint.py.151: total_norm: 25.696079 [I 2022-10-10 02:42:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1920/2304 [I 2022-10-10 02:42:32] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:25 [I 2022-10-10 02:42:32] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00338 [I 2022-10-10 02:42:32] ...trap/engines/engine.py.230: loss: 0.22284 [I 2022-10-10 02:42:32] ...optimizers/trijoint.py.151: total_norm: 33.103710 [I 2022-10-10 02:42:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1930/2304 [I 2022-10-10 02:42:41] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:16 [I 2022-10-10 02:42:41] ...trap/engines/engine.py.229: process: 0.83721 | load: 0.00297 [I 2022-10-10 02:42:41] ...trap/engines/engine.py.230: loss: 0.16780 [I 2022-10-10 02:42:41] ...optimizers/trijoint.py.151: total_norm: 16.748665 [I 2022-10-10 02:42:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 1940/2304 [I 2022-10-10 02:42:49] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:05:09 [I 2022-10-10 02:42:49] ...trap/engines/engine.py.229: process: 0.85206 | load: 0.00295 [I 2022-10-10 02:42:49] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-10 02:42:49] ...optimizers/trijoint.py.151: total_norm: 14.545869 [I 2022-10-10 02:42:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1950/2304 [I 2022-10-10 02:42:58] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:57 [I 2022-10-10 02:42:58] ...trap/engines/engine.py.229: process: 0.83009 | load: 0.00291 [I 2022-10-10 02:42:58] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-10 02:42:58] ...optimizers/trijoint.py.151: total_norm: 20.167356 [I 2022-10-10 02:43:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1960/2304 [I 2022-10-10 02:43:06] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:50 [I 2022-10-10 02:43:06] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00292 [I 2022-10-10 02:43:06] ...trap/engines/engine.py.230: loss: 0.11313 [I 2022-10-10 02:43:06] ...optimizers/trijoint.py.151: total_norm: 8.560839 [I 2022-10-10 02:43:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1970/2304 [I 2022-10-10 02:43:15] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:41 [I 2022-10-10 02:43:15] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00297 [I 2022-10-10 02:43:15] ...trap/engines/engine.py.230: loss: 0.16918 [I 2022-10-10 02:43:15] ...optimizers/trijoint.py.151: total_norm: 22.346891 [I 2022-10-10 02:43:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1980/2304 [I 2022-10-10 02:43:23] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:33 [I 2022-10-10 02:43:23] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00299 [I 2022-10-10 02:43:23] ...trap/engines/engine.py.230: loss: 0.16853 [I 2022-10-10 02:43:23] ...optimizers/trijoint.py.151: total_norm: 15.785803 [I 2022-10-10 02:43:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1990/2304 [I 2022-10-10 02:43:32] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:27 [I 2022-10-10 02:43:32] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00295 [I 2022-10-10 02:43:32] ...trap/engines/engine.py.230: loss: 0.19082 [I 2022-10-10 02:43:32] ...optimizers/trijoint.py.151: total_norm: 40.032101 [I 2022-10-10 02:43:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 2000/2304 [I 2022-10-10 02:43:40] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:17 [I 2022-10-10 02:43:40] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00297 [I 2022-10-10 02:43:40] ...trap/engines/engine.py.230: loss: 0.11877 [I 2022-10-10 02:43:40] ...optimizers/trijoint.py.151: total_norm: 14.655317 [I 2022-10-10 02:43:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 2010/2304 [I 2022-10-10 02:43:49] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:04:08 [I 2022-10-10 02:43:49] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00304 [I 2022-10-10 02:43:49] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-10 02:43:49] ...optimizers/trijoint.py.151: total_norm: 8.657229 [I 2022-10-10 02:43:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 2020/2304 [I 2022-10-10 02:43:57] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:59 [I 2022-10-10 02:43:57] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00297 [I 2022-10-10 02:43:57] ...trap/engines/engine.py.230: loss: 0.12676 [I 2022-10-10 02:43:57] ...optimizers/trijoint.py.151: total_norm: 20.389788 [I 2022-10-10 02:44:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 2030/2304 [I 2022-10-10 02:44:06] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:51 [I 2022-10-10 02:44:06] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00298 [I 2022-10-10 02:44:06] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-10 02:44:06] ...optimizers/trijoint.py.151: total_norm: 14.160415 [I 2022-10-10 02:44:14] ...trap/engines/engine.py.224: train: epoch 63 | batch 2040/2304 [I 2022-10-10 02:44:14] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:43 [I 2022-10-10 02:44:14] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00299 [I 2022-10-10 02:44:14] ...trap/engines/engine.py.230: loss: 0.09541 [I 2022-10-10 02:44:14] ...optimizers/trijoint.py.151: total_norm: 10.794239 [I 2022-10-10 02:44:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 2050/2304 [I 2022-10-10 02:44:23] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:35 [I 2022-10-10 02:44:23] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00298 [I 2022-10-10 02:44:23] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-10 02:44:23] ...optimizers/trijoint.py.151: total_norm: 49.926388 [I 2022-10-10 02:44:31] ...trap/engines/engine.py.224: train: epoch 63 | batch 2060/2304 [I 2022-10-10 02:44:31] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:24 [I 2022-10-10 02:44:31] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00297 [I 2022-10-10 02:44:31] ...trap/engines/engine.py.230: loss: 0.16947 [I 2022-10-10 02:44:31] ...optimizers/trijoint.py.151: total_norm: 16.645567 [I 2022-10-10 02:44:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 2070/2304 [I 2022-10-10 02:44:39] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:16 [I 2022-10-10 02:44:39] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00295 [I 2022-10-10 02:44:39] ...trap/engines/engine.py.230: loss: 0.18815 [I 2022-10-10 02:44:39] ...optimizers/trijoint.py.151: total_norm: 11.565910 [I 2022-10-10 02:44:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 2080/2304 [I 2022-10-10 02:44:48] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:08 [I 2022-10-10 02:44:48] ...trap/engines/engine.py.229: process: 0.84099 | load: 0.00298 [I 2022-10-10 02:44:48] ...trap/engines/engine.py.230: loss: 0.16405 [I 2022-10-10 02:44:48] ...optimizers/trijoint.py.151: total_norm: 19.580843 [I 2022-10-10 02:44:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 2090/2304 [I 2022-10-10 02:44:56] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:00 [I 2022-10-10 02:44:56] ...trap/engines/engine.py.229: process: 0.84687 | load: 0.00295 [I 2022-10-10 02:44:56] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-10 02:44:56] ...optimizers/trijoint.py.151: total_norm: 17.025330 [I 2022-10-10 02:45:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 2100/2304 [I 2022-10-10 02:45:05] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:52 [I 2022-10-10 02:45:05] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00309 [I 2022-10-10 02:45:05] ...trap/engines/engine.py.230: loss: 0.20575 [I 2022-10-10 02:45:05] ...optimizers/trijoint.py.151: total_norm: 19.827204 [I 2022-10-10 02:45:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 2110/2304 [I 2022-10-10 02:45:13] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:46 [I 2022-10-10 02:45:13] ...trap/engines/engine.py.229: process: 0.86133 | load: 0.00310 [I 2022-10-10 02:45:13] ...trap/engines/engine.py.230: loss: 0.24969 [I 2022-10-10 02:45:13] ...optimizers/trijoint.py.151: total_norm: 16.521736 [I 2022-10-10 02:45:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 2120/2304 [I 2022-10-10 02:45:22] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:36 [I 2022-10-10 02:45:22] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00298 [I 2022-10-10 02:45:22] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-10 02:45:22] ...optimizers/trijoint.py.151: total_norm: 18.486675 [I 2022-10-10 02:45:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 2130/2304 [I 2022-10-10 02:45:30] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:27 [I 2022-10-10 02:45:30] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00302 [I 2022-10-10 02:45:30] ...trap/engines/engine.py.230: loss: 0.18698 [I 2022-10-10 02:45:30] ...optimizers/trijoint.py.151: total_norm: 14.442002 [I 2022-10-10 02:45:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 2140/2304 [I 2022-10-10 02:45:39] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:19 [I 2022-10-10 02:45:39] ...trap/engines/engine.py.229: process: 0.85728 | load: 0.00312 [I 2022-10-10 02:45:39] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-10 02:45:39] ...optimizers/trijoint.py.151: total_norm: 10.705545 [I 2022-10-10 02:45:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 2150/2304 [I 2022-10-10 02:45:48] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:11 [I 2022-10-10 02:45:48] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00315 [I 2022-10-10 02:45:48] ...trap/engines/engine.py.230: loss: 0.13789 [I 2022-10-10 02:45:48] ...optimizers/trijoint.py.151: total_norm: 14.642581 [I 2022-10-10 02:45:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 2160/2304 [I 2022-10-10 02:45:56] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:02 [I 2022-10-10 02:45:56] ...trap/engines/engine.py.229: process: 0.85427 | load: 0.00312 [I 2022-10-10 02:45:56] ...trap/engines/engine.py.230: loss: 0.10932 [I 2022-10-10 02:45:56] ...optimizers/trijoint.py.151: total_norm: 9.169614 [I 2022-10-10 02:46:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 2170/2304 [I 2022-10-10 02:46:05] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:54 [I 2022-10-10 02:46:05] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00328 [I 2022-10-10 02:46:05] ...trap/engines/engine.py.230: loss: 0.32144 [I 2022-10-10 02:46:05] ...optimizers/trijoint.py.151: total_norm: 22.257790 [I 2022-10-10 02:46:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 2180/2304 [I 2022-10-10 02:46:13] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:46 [I 2022-10-10 02:46:13] ...trap/engines/engine.py.229: process: 0.86226 | load: 0.00312 [I 2022-10-10 02:46:13] ...trap/engines/engine.py.230: loss: 0.10777 [I 2022-10-10 02:46:13] ...optimizers/trijoint.py.151: total_norm: 12.801455 [I 2022-10-10 02:46:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 2190/2304 [I 2022-10-10 02:46:22] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:37 [I 2022-10-10 02:46:22] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00310 [I 2022-10-10 02:46:22] ...trap/engines/engine.py.230: loss: 0.15541 [I 2022-10-10 02:46:22] ...optimizers/trijoint.py.151: total_norm: 23.210224 [I 2022-10-10 02:46:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 2200/2304 [I 2022-10-10 02:46:30] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:28 [I 2022-10-10 02:46:30] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00314 [I 2022-10-10 02:46:30] ...trap/engines/engine.py.230: loss: 0.11716 [I 2022-10-10 02:46:30] ...optimizers/trijoint.py.151: total_norm: 12.664533 [I 2022-10-10 02:46:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 2210/2304 [I 2022-10-10 02:46:39] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:20 [I 2022-10-10 02:46:39] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00312 [I 2022-10-10 02:46:39] ...trap/engines/engine.py.230: loss: 0.16498 [I 2022-10-10 02:46:39] ...optimizers/trijoint.py.151: total_norm: 18.310045 [I 2022-10-10 02:46:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 2220/2304 [I 2022-10-10 02:46:47] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:11 [I 2022-10-10 02:46:47] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00312 [I 2022-10-10 02:46:47] ...trap/engines/engine.py.230: loss: 0.11142 [I 2022-10-10 02:46:47] ...optimizers/trijoint.py.151: total_norm: 15.002657 [I 2022-10-10 02:46:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 2230/2304 [I 2022-10-10 02:46:56] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:02 [I 2022-10-10 02:46:56] ...trap/engines/engine.py.229: process: 0.84978 | load: 0.00310 [I 2022-10-10 02:46:56] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-10 02:46:56] ...optimizers/trijoint.py.151: total_norm: 11.894615 [I 2022-10-10 02:47:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 2240/2304 [I 2022-10-10 02:47:04] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:00:54 [I 2022-10-10 02:47:04] ...trap/engines/engine.py.229: process: 0.85615 | load: 0.00312 [I 2022-10-10 02:47:04] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-10 02:47:04] ...optimizers/trijoint.py.151: total_norm: 9.211514 [I 2022-10-10 02:47:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 2250/2304 [I 2022-10-10 02:47:13] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:46 [I 2022-10-10 02:47:13] ...trap/engines/engine.py.229: process: 0.85108 | load: 0.00310 [I 2022-10-10 02:47:13] ...trap/engines/engine.py.230: loss: 0.14584 [I 2022-10-10 02:47:13] ...optimizers/trijoint.py.151: total_norm: 24.978243 [I 2022-10-10 02:47:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 2260/2304 [I 2022-10-10 02:47:22] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:37 [I 2022-10-10 02:47:22] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00314 [I 2022-10-10 02:47:22] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-10 02:47:22] ...optimizers/trijoint.py.151: total_norm: 18.261753 [I 2022-10-10 02:47:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 2270/2304 [I 2022-10-10 02:47:30] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:29 [I 2022-10-10 02:47:30] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00310 [I 2022-10-10 02:47:30] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-10 02:47:30] ...optimizers/trijoint.py.151: total_norm: 11.038869 [I 2022-10-10 02:47:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 2280/2304 [I 2022-10-10 02:47:39] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:20 [I 2022-10-10 02:47:39] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00311 [I 2022-10-10 02:47:39] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-10 02:47:39] ...optimizers/trijoint.py.151: total_norm: 14.111055 [I 2022-10-10 02:47:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 2290/2304 [I 2022-10-10 02:47:47] ...trap/engines/engine.py.228: elapsed: 0:32:53 | left: 0:00:11 [I 2022-10-10 02:47:47] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00308 [I 2022-10-10 02:47:47] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-10 02:47:47] ...optimizers/trijoint.py.151: total_norm: 13.243533 [I 2022-10-10 02:47:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 2300/2304 [I 2022-10-10 02:47:56] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:03 [I 2022-10-10 02:47:56] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00021 [I 2022-10-10 02:47:56] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-10 02:47:56] ...optimizers/trijoint.py.151: total_norm: 17.317827 [I 2022-10-10 02:47:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 2304/2304 [I 2022-10-10 02:47:59] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:00 [I 2022-10-10 02:47:59] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00020 [I 2022-10-10 02:47:59] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-10 02:47:59] ...optimizers/trijoint.py.151: total_norm: 10.849833 [S 2022-10-10 02:48:00] ...trap/engines/engine.py.129: train_epoch.epoch: 63 [S 2022-10-10 02:48:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.14543971759378263 [I 2022-10-10 02:48:00] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 02:48:01] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 63 [I 2022-10-10 02:48:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 02:48:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 02:48:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 02:48:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 02:48:06] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 02:48:06] ...trap/engines/engine.py.321: eval: epoch 63 | batch 0/510 [I 2022-10-10 02:48:06] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:16 [I 2022-10-10 02:48:06] ...trap/engines/engine.py.326: process: 4.73897 | load: 4.14122 [I 2022-10-10 02:48:12] ...trap/engines/engine.py.321: eval: epoch 63 | batch 10/510 [I 2022-10-10 02:48:12] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:46 [I 2022-10-10 02:48:12] ...trap/engines/engine.py.326: process: 0.57649 | load: 0.00020 [I 2022-10-10 02:48:17] ...trap/engines/engine.py.321: eval: epoch 63 | batch 20/510 [I 2022-10-10 02:48:17] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:32 [I 2022-10-10 02:48:17] ...trap/engines/engine.py.326: process: 0.42456 | load: 0.00020 [I 2022-10-10 02:48:22] ...trap/engines/engine.py.321: eval: epoch 63 | batch 30/510 [I 2022-10-10 02:48:22] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:37 [I 2022-10-10 02:48:22] ...trap/engines/engine.py.326: process: 0.42001 | load: 0.00020 [I 2022-10-10 02:48:26] ...trap/engines/engine.py.321: eval: epoch 63 | batch 40/510 [I 2022-10-10 02:48:26] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:25 [I 2022-10-10 02:48:26] ...trap/engines/engine.py.326: process: 0.42893 | load: 0.00020 [I 2022-10-10 02:48:31] ...trap/engines/engine.py.321: eval: epoch 63 | batch 50/510 [I 2022-10-10 02:48:31] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:26 [I 2022-10-10 02:48:31] ...trap/engines/engine.py.326: process: 0.43083 | load: 0.00021 [I 2022-10-10 02:48:36] ...trap/engines/engine.py.321: eval: epoch 63 | batch 60/510 [I 2022-10-10 02:48:36] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:30 [I 2022-10-10 02:48:36] ...trap/engines/engine.py.326: process: 0.43862 | load: 0.00020 [I 2022-10-10 02:48:40] ...trap/engines/engine.py.321: eval: epoch 63 | batch 70/510 [I 2022-10-10 02:48:40] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:34 [I 2022-10-10 02:48:40] ...trap/engines/engine.py.326: process: 0.50883 | load: 0.00020 [I 2022-10-10 02:48:46] ...trap/engines/engine.py.321: eval: epoch 63 | batch 80/510 [I 2022-10-10 02:48:46] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:09 [I 2022-10-10 02:48:46] ...trap/engines/engine.py.326: process: 0.52824 | load: 0.00020 [I 2022-10-10 02:48:52] ...trap/engines/engine.py.321: eval: epoch 63 | batch 90/510 [I 2022-10-10 02:48:52] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:48 [I 2022-10-10 02:48:52] ...trap/engines/engine.py.326: process: 0.53127 | load: 0.00020 [I 2022-10-10 02:48:57] ...trap/engines/engine.py.321: eval: epoch 63 | batch 100/510 [I 2022-10-10 02:48:57] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:18 [I 2022-10-10 02:48:57] ...trap/engines/engine.py.326: process: 0.31278 | load: 0.00021 [I 2022-10-10 02:49:00] ...trap/engines/engine.py.321: eval: epoch 63 | batch 110/510 [I 2022-10-10 02:49:00] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:15 [I 2022-10-10 02:49:00] ...trap/engines/engine.py.326: process: 0.31767 | load: 0.00020 [I 2022-10-10 02:49:03] ...trap/engines/engine.py.321: eval: epoch 63 | batch 120/510 [I 2022-10-10 02:49:03] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:04 [I 2022-10-10 02:49:03] ...trap/engines/engine.py.326: process: 0.31497 | load: 0.00018 [I 2022-10-10 02:49:06] ...trap/engines/engine.py.321: eval: epoch 63 | batch 130/510 [I 2022-10-10 02:49:06] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-10 02:49:06] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00019 [I 2022-10-10 02:49:10] ...trap/engines/engine.py.321: eval: epoch 63 | batch 140/510 [I 2022-10-10 02:49:10] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:58 [I 2022-10-10 02:49:10] ...trap/engines/engine.py.326: process: 0.33287 | load: 0.00024 [I 2022-10-10 02:49:13] ...trap/engines/engine.py.321: eval: epoch 63 | batch 150/510 [I 2022-10-10 02:49:13] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:55 [I 2022-10-10 02:49:13] ...trap/engines/engine.py.326: process: 0.31991 | load: 0.00021 [I 2022-10-10 02:49:16] ...trap/engines/engine.py.321: eval: epoch 63 | batch 160/510 [I 2022-10-10 02:49:16] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:51 [I 2022-10-10 02:49:16] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00021 [I 2022-10-10 02:49:19] ...trap/engines/engine.py.321: eval: epoch 63 | batch 170/510 [I 2022-10-10 02:49:19] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-10 02:49:19] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00020 [I 2022-10-10 02:49:22] ...trap/engines/engine.py.321: eval: epoch 63 | batch 180/510 [I 2022-10-10 02:49:22] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:46 [I 2022-10-10 02:49:22] ...trap/engines/engine.py.326: process: 0.32424 | load: 0.00021 [I 2022-10-10 02:49:26] ...trap/engines/engine.py.321: eval: epoch 63 | batch 190/510 [I 2022-10-10 02:49:26] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:42 [I 2022-10-10 02:49:26] ...trap/engines/engine.py.326: process: 0.31909 | load: 0.00021 [I 2022-10-10 02:49:29] ...trap/engines/engine.py.321: eval: epoch 63 | batch 200/510 [I 2022-10-10 02:49:29] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:39 [I 2022-10-10 02:49:29] ...trap/engines/engine.py.326: process: 0.32323 | load: 0.00019 [I 2022-10-10 02:49:32] ...trap/engines/engine.py.321: eval: epoch 63 | batch 210/510 [I 2022-10-10 02:49:32] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-10 02:49:32] ...trap/engines/engine.py.326: process: 0.32409 | load: 0.00020 [I 2022-10-10 02:49:35] ...trap/engines/engine.py.321: eval: epoch 63 | batch 220/510 [I 2022-10-10 02:49:35] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-10 02:49:35] ...trap/engines/engine.py.326: process: 0.32368 | load: 0.00020 [I 2022-10-10 02:49:38] ...trap/engines/engine.py.321: eval: epoch 63 | batch 230/510 [I 2022-10-10 02:49:38] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-10 02:49:38] ...trap/engines/engine.py.326: process: 0.31977 | load: 0.00021 [I 2022-10-10 02:49:42] ...trap/engines/engine.py.321: eval: epoch 63 | batch 240/510 [I 2022-10-10 02:49:42] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-10 02:49:42] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00021 [I 2022-10-10 02:49:45] ...trap/engines/engine.py.321: eval: epoch 63 | batch 250/510 [I 2022-10-10 02:49:45] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-10 02:49:45] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00020 [I 2022-10-10 02:49:48] ...trap/engines/engine.py.321: eval: epoch 63 | batch 260/510 [I 2022-10-10 02:49:48] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-10 02:49:48] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00020 [I 2022-10-10 02:49:51] ...trap/engines/engine.py.321: eval: epoch 63 | batch 270/510 [I 2022-10-10 02:49:51] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-10 02:49:51] ...trap/engines/engine.py.326: process: 0.32399 | load: 0.00022 [I 2022-10-10 02:49:54] ...trap/engines/engine.py.321: eval: epoch 63 | batch 280/510 [I 2022-10-10 02:49:54] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-10 02:49:54] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00020 [I 2022-10-10 02:49:58] ...trap/engines/engine.py.321: eval: epoch 63 | batch 290/510 [I 2022-10-10 02:49:58] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:10 [I 2022-10-10 02:49:58] ...trap/engines/engine.py.326: process: 0.31994 | load: 0.00021 [I 2022-10-10 02:50:01] ...trap/engines/engine.py.321: eval: epoch 63 | batch 300/510 [I 2022-10-10 02:50:01] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:07 [I 2022-10-10 02:50:01] ...trap/engines/engine.py.326: process: 0.31617 | load: 0.00021 [I 2022-10-10 02:50:04] ...trap/engines/engine.py.321: eval: epoch 63 | batch 310/510 [I 2022-10-10 02:50:04] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-10 02:50:04] ...trap/engines/engine.py.326: process: 0.31390 | load: 0.00020 [I 2022-10-10 02:50:07] ...trap/engines/engine.py.321: eval: epoch 63 | batch 320/510 [I 2022-10-10 02:50:07] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-10 02:50:07] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00021 [I 2022-10-10 02:50:10] ...trap/engines/engine.py.321: eval: epoch 63 | batch 330/510 [I 2022-10-10 02:50:10] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-10 02:50:10] ...trap/engines/engine.py.326: process: 0.31772 | load: 0.00020 [I 2022-10-10 02:50:14] ...trap/engines/engine.py.321: eval: epoch 63 | batch 340/510 [I 2022-10-10 02:50:14] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-10 02:50:14] ...trap/engines/engine.py.326: process: 0.32042 | load: 0.00022 [I 2022-10-10 02:50:17] ...trap/engines/engine.py.321: eval: epoch 63 | batch 350/510 [I 2022-10-10 02:50:17] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-10 02:50:17] ...trap/engines/engine.py.326: process: 0.32342 | load: 0.00020 [I 2022-10-10 02:50:20] ...trap/engines/engine.py.321: eval: epoch 63 | batch 360/510 [I 2022-10-10 02:50:20] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-10 02:50:20] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00020 [I 2022-10-10 02:50:23] ...trap/engines/engine.py.321: eval: epoch 63 | batch 370/510 [I 2022-10-10 02:50:23] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-10 02:50:23] ...trap/engines/engine.py.326: process: 0.31980 | load: 0.00021 [I 2022-10-10 02:50:26] ...trap/engines/engine.py.321: eval: epoch 63 | batch 380/510 [I 2022-10-10 02:50:26] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-10 02:50:26] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00020 [I 2022-10-10 02:50:30] ...trap/engines/engine.py.321: eval: epoch 63 | batch 390/510 [I 2022-10-10 02:50:30] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-10 02:50:30] ...trap/engines/engine.py.326: process: 0.32420 | load: 0.00020 [I 2022-10-10 02:50:33] ...trap/engines/engine.py.321: eval: epoch 63 | batch 400/510 [I 2022-10-10 02:50:33] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-10 02:50:33] ...trap/engines/engine.py.326: process: 0.32170 | load: 0.00021 [I 2022-10-10 02:50:36] ...trap/engines/engine.py.321: eval: epoch 63 | batch 410/510 [I 2022-10-10 02:50:36] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-10 02:50:36] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00021 [I 2022-10-10 02:50:39] ...trap/engines/engine.py.321: eval: epoch 63 | batch 420/510 [I 2022-10-10 02:50:39] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-10 02:50:39] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00020 [I 2022-10-10 02:50:42] ...trap/engines/engine.py.321: eval: epoch 63 | batch 430/510 [I 2022-10-10 02:50:42] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-10 02:50:42] ...trap/engines/engine.py.326: process: 0.32009 | load: 0.00020 [I 2022-10-10 02:50:45] ...trap/engines/engine.py.321: eval: epoch 63 | batch 440/510 [I 2022-10-10 02:50:45] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-10 02:50:45] ...trap/engines/engine.py.326: process: 0.31896 | load: 0.00019 [I 2022-10-10 02:50:49] ...trap/engines/engine.py.321: eval: epoch 63 | batch 450/510 [I 2022-10-10 02:50:49] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-10 02:50:49] ...trap/engines/engine.py.326: process: 0.31830 | load: 0.00019 [I 2022-10-10 02:50:52] ...trap/engines/engine.py.321: eval: epoch 63 | batch 460/510 [I 2022-10-10 02:50:52] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-10 02:50:52] ...trap/engines/engine.py.326: process: 0.32142 | load: 0.00020 [I 2022-10-10 02:50:55] ...trap/engines/engine.py.321: eval: epoch 63 | batch 470/510 [I 2022-10-10 02:50:55] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-10 02:50:55] ...trap/engines/engine.py.326: process: 0.32113 | load: 0.00020 [I 2022-10-10 02:50:58] ...trap/engines/engine.py.321: eval: epoch 63 | batch 480/510 [I 2022-10-10 02:50:58] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-10 02:50:58] ...trap/engines/engine.py.326: process: 0.31796 | load: 0.00020 [I 2022-10-10 02:51:01] ...trap/engines/engine.py.321: eval: epoch 63 | batch 490/510 [I 2022-10-10 02:51:01] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-10 02:51:01] ...trap/engines/engine.py.326: process: 0.31922 | load: 0.00021 [I 2022-10-10 02:51:05] ...trap/engines/engine.py.321: eval: epoch 63 | batch 500/510 [I 2022-10-10 02:51:05] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-10 02:51:05] ...trap/engines/engine.py.326: process: 0.32457 | load: 0.00020 [I 2022-10-10 02:51:08] ...trap/engines/engine.py.321: eval: epoch 63 | batch 510/510 [I 2022-10-10 02:51:08] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-10 02:51:08] ...trap/engines/engine.py.326: process: 0.31121 | load: 0.00017 [S 2022-10-10 02:51:08] ...trap/engines/engine.py.132: eval_epoch.epoch: 63 [I 2022-10-10 02:51:08] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.730600 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911600 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939900 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724300 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909700 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936400 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008823 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008823 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007854 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010621 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005832 [S 2022-10-10 02:51:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005953 [I 2022-10-10 02:51:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 02:51:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 02:51:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 02:51:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 02:51:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 02:51:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 02:51:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 02:51:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 02:51:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 64 [I 2022-10-10 02:51:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 02:51:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 0/2304 [I 2022-10-10 02:51:56] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:37 [I 2022-10-10 02:51:56] ...trap/engines/engine.py.229: process: 2.75045 | load: 1.84579 [I 2022-10-10 02:51:56] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-10 02:51:56] ...optimizers/trijoint.py.151: total_norm: 19.680155 [I 2022-10-10 02:52:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 10/2304 [I 2022-10-10 02:52:05] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:09 [I 2022-10-10 02:52:05] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00404 [I 2022-10-10 02:52:05] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-10 02:52:05] ...optimizers/trijoint.py.151: total_norm: 20.887951 [I 2022-10-10 02:52:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 20/2304 [I 2022-10-10 02:52:13] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-10 02:52:13] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00454 [I 2022-10-10 02:52:13] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-10 02:52:13] ...optimizers/trijoint.py.151: total_norm: 14.176836 [I 2022-10-10 02:52:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 30/2304 [I 2022-10-10 02:52:22] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:01 [I 2022-10-10 02:52:22] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00394 [I 2022-10-10 02:52:22] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-10 02:52:22] ...optimizers/trijoint.py.151: total_norm: 21.825287 [I 2022-10-10 02:52:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 40/2304 [I 2022-10-10 02:52:30] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:58 [I 2022-10-10 02:52:30] ...trap/engines/engine.py.229: process: 0.85565 | load: 0.00387 [I 2022-10-10 02:52:30] ...trap/engines/engine.py.230: loss: 0.13705 [I 2022-10-10 02:52:30] ...optimizers/trijoint.py.151: total_norm: 9.041774 [I 2022-10-10 02:52:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 50/2304 [I 2022-10-10 02:52:39] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:38 [I 2022-10-10 02:52:39] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00335 [I 2022-10-10 02:52:39] ...trap/engines/engine.py.230: loss: 0.11160 [I 2022-10-10 02:52:39] ...optimizers/trijoint.py.151: total_norm: 10.679451 [I 2022-10-10 02:52:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 60/2304 [I 2022-10-10 02:52:47] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:53 [I 2022-10-10 02:52:47] ...trap/engines/engine.py.229: process: 0.86774 | load: 0.00366 [I 2022-10-10 02:52:47] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-10 02:52:47] ...optimizers/trijoint.py.151: total_norm: 18.646782 [I 2022-10-10 02:52:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 70/2304 [I 2022-10-10 02:52:56] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:23 [I 2022-10-10 02:52:56] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00343 [I 2022-10-10 02:52:56] ...trap/engines/engine.py.230: loss: 0.14350 [I 2022-10-10 02:52:56] ...optimizers/trijoint.py.151: total_norm: 22.924932 [I 2022-10-10 02:53:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 80/2304 [I 2022-10-10 02:53:04] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:17 [I 2022-10-10 02:53:04] ...trap/engines/engine.py.229: process: 0.83295 | load: 0.00357 [I 2022-10-10 02:53:04] ...trap/engines/engine.py.230: loss: 0.10294 [I 2022-10-10 02:53:04] ...optimizers/trijoint.py.151: total_norm: 11.948050 [I 2022-10-10 02:53:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 90/2304 [I 2022-10-10 02:53:13] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:21 [I 2022-10-10 02:53:13] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00328 [I 2022-10-10 02:53:13] ...trap/engines/engine.py.230: loss: 0.12329 [I 2022-10-10 02:53:13] ...optimizers/trijoint.py.151: total_norm: 13.894544 [I 2022-10-10 02:53:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 100/2304 [I 2022-10-10 02:53:21] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:07 [I 2022-10-10 02:53:21] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00378 [I 2022-10-10 02:53:21] ...trap/engines/engine.py.230: loss: 0.15399 [I 2022-10-10 02:53:21] ...optimizers/trijoint.py.151: total_norm: 8.395380 [I 2022-10-10 02:53:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 110/2304 [I 2022-10-10 02:53:30] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:55 [I 2022-10-10 02:53:30] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00331 [I 2022-10-10 02:53:30] ...trap/engines/engine.py.230: loss: 0.11994 [I 2022-10-10 02:53:30] ...optimizers/trijoint.py.151: total_norm: 10.275694 [I 2022-10-10 02:53:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 120/2304 [I 2022-10-10 02:53:38] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:46 [I 2022-10-10 02:53:38] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00308 [I 2022-10-10 02:53:38] ...trap/engines/engine.py.230: loss: 0.12304 [I 2022-10-10 02:53:38] ...optimizers/trijoint.py.151: total_norm: 12.490012 [I 2022-10-10 02:53:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 130/2304 [I 2022-10-10 02:53:47] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:42 [I 2022-10-10 02:53:47] ...trap/engines/engine.py.229: process: 0.83965 | load: 0.00340 [I 2022-10-10 02:53:47] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-10 02:53:47] ...optimizers/trijoint.py.151: total_norm: 11.585669 [I 2022-10-10 02:53:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 140/2304 [I 2022-10-10 02:53:55] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:35 [I 2022-10-10 02:53:55] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00335 [I 2022-10-10 02:53:55] ...trap/engines/engine.py.230: loss: 0.16840 [I 2022-10-10 02:53:55] ...optimizers/trijoint.py.151: total_norm: 22.038202 [I 2022-10-10 02:54:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 150/2304 [I 2022-10-10 02:54:04] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:23 [I 2022-10-10 02:54:04] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00314 [I 2022-10-10 02:54:04] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-10 02:54:04] ...optimizers/trijoint.py.151: total_norm: 15.043723 [I 2022-10-10 02:54:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 160/2304 [I 2022-10-10 02:54:12] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:14 [I 2022-10-10 02:54:12] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00326 [I 2022-10-10 02:54:12] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-10 02:54:12] ...optimizers/trijoint.py.151: total_norm: 11.629506 [I 2022-10-10 02:54:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 170/2304 [I 2022-10-10 02:54:21] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:05 [I 2022-10-10 02:54:21] ...trap/engines/engine.py.229: process: 0.84954 | load: 0.00316 [I 2022-10-10 02:54:21] ...trap/engines/engine.py.230: loss: 0.16742 [I 2022-10-10 02:54:21] ...optimizers/trijoint.py.151: total_norm: 13.735727 [I 2022-10-10 02:54:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 180/2304 [I 2022-10-10 02:54:29] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:44 [I 2022-10-10 02:54:29] ...trap/engines/engine.py.229: process: 0.83656 | load: 0.00308 [I 2022-10-10 02:54:29] ...trap/engines/engine.py.230: loss: 0.16773 [I 2022-10-10 02:54:29] ...optimizers/trijoint.py.151: total_norm: 19.255985 [I 2022-10-10 02:54:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 190/2304 [I 2022-10-10 02:54:37] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:57 [I 2022-10-10 02:54:37] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00322 [I 2022-10-10 02:54:37] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-10 02:54:37] ...optimizers/trijoint.py.151: total_norm: 17.569403 [I 2022-10-10 02:54:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 200/2304 [I 2022-10-10 02:54:46] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:54 [I 2022-10-10 02:54:46] ...trap/engines/engine.py.229: process: 0.86858 | load: 0.00304 [I 2022-10-10 02:54:46] ...trap/engines/engine.py.230: loss: 0.09506 [I 2022-10-10 02:54:46] ...optimizers/trijoint.py.151: total_norm: 9.939746 [I 2022-10-10 02:54:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 210/2304 [I 2022-10-10 02:54:54] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:31 [I 2022-10-10 02:54:54] ...trap/engines/engine.py.229: process: 0.84400 | load: 0.00299 [I 2022-10-10 02:54:54] ...trap/engines/engine.py.230: loss: 0.12194 [I 2022-10-10 02:54:55] ...optimizers/trijoint.py.151: total_norm: 12.763671 [I 2022-10-10 02:55:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 220/2304 [I 2022-10-10 02:55:03] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:32 [I 2022-10-10 02:55:03] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00313 [I 2022-10-10 02:55:03] ...trap/engines/engine.py.230: loss: 0.17154 [I 2022-10-10 02:55:03] ...optimizers/trijoint.py.151: total_norm: 23.370180 [I 2022-10-10 02:55:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 230/2304 [I 2022-10-10 02:55:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:21 [I 2022-10-10 02:55:12] ...trap/engines/engine.py.229: process: 0.84471 | load: 0.00298 [I 2022-10-10 02:55:12] ...trap/engines/engine.py.230: loss: 0.17308 [I 2022-10-10 02:55:12] ...optimizers/trijoint.py.151: total_norm: 16.173740 [I 2022-10-10 02:55:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 240/2304 [I 2022-10-10 02:55:20] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:11 [I 2022-10-10 02:55:20] ...trap/engines/engine.py.229: process: 0.83954 | load: 0.00298 [I 2022-10-10 02:55:20] ...trap/engines/engine.py.230: loss: 0.10621 [I 2022-10-10 02:55:20] ...optimizers/trijoint.py.151: total_norm: 14.022464 [I 2022-10-10 02:55:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 250/2304 [I 2022-10-10 02:55:29] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:08 [I 2022-10-10 02:55:29] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00314 [I 2022-10-10 02:55:29] ...trap/engines/engine.py.230: loss: 0.16472 [I 2022-10-10 02:55:29] ...optimizers/trijoint.py.151: total_norm: 13.578150 [I 2022-10-10 02:55:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 260/2304 [I 2022-10-10 02:55:37] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:53 [I 2022-10-10 02:55:37] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00321 [I 2022-10-10 02:55:37] ...trap/engines/engine.py.230: loss: 0.10058 [I 2022-10-10 02:55:37] ...optimizers/trijoint.py.151: total_norm: 9.452870 [I 2022-10-10 02:55:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 270/2304 [I 2022-10-10 02:55:46] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:49 [I 2022-10-10 02:55:46] ...trap/engines/engine.py.229: process: 0.85092 | load: 0.00310 [I 2022-10-10 02:55:46] ...trap/engines/engine.py.230: loss: 0.13239 [I 2022-10-10 02:55:46] ...optimizers/trijoint.py.151: total_norm: 20.163040 [I 2022-10-10 02:55:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 280/2304 [I 2022-10-10 02:55:54] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:34 [I 2022-10-10 02:55:54] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00317 [I 2022-10-10 02:55:54] ...trap/engines/engine.py.230: loss: 0.22436 [I 2022-10-10 02:55:54] ...optimizers/trijoint.py.151: total_norm: 16.712223 [I 2022-10-10 02:56:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 290/2304 [I 2022-10-10 02:56:03] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:35 [I 2022-10-10 02:56:03] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00306 [I 2022-10-10 02:56:03] ...trap/engines/engine.py.230: loss: 0.20686 [I 2022-10-10 02:56:03] ...optimizers/trijoint.py.151: total_norm: 19.885458 [I 2022-10-10 02:56:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 300/2304 [I 2022-10-10 02:56:11] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:20 [I 2022-10-10 02:56:11] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00301 [I 2022-10-10 02:56:11] ...trap/engines/engine.py.230: loss: 0.12329 [I 2022-10-10 02:56:11] ...optimizers/trijoint.py.151: total_norm: 14.969374 [I 2022-10-10 02:56:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 310/2304 [I 2022-10-10 02:56:20] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:07 [I 2022-10-10 02:56:20] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00327 [I 2022-10-10 02:56:20] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-10 02:56:20] ...optimizers/trijoint.py.151: total_norm: 11.925471 [I 2022-10-10 02:56:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 320/2304 [I 2022-10-10 02:56:28] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:28:00 [I 2022-10-10 02:56:28] ...trap/engines/engine.py.229: process: 0.84680 | load: 0.00313 [I 2022-10-10 02:56:28] ...trap/engines/engine.py.230: loss: 0.16597 [I 2022-10-10 02:56:28] ...optimizers/trijoint.py.151: total_norm: 12.489845 [I 2022-10-10 02:56:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 330/2304 [I 2022-10-10 02:56:37] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:03 [I 2022-10-10 02:56:37] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00302 [I 2022-10-10 02:56:37] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-10 02:56:37] ...optimizers/trijoint.py.151: total_norm: 10.749044 [I 2022-10-10 02:56:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 340/2304 [I 2022-10-10 02:56:45] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:27:35 [I 2022-10-10 02:56:45] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00305 [I 2022-10-10 02:56:45] ...trap/engines/engine.py.230: loss: 0.15081 [I 2022-10-10 02:56:45] ...optimizers/trijoint.py.151: total_norm: 20.914951 [I 2022-10-10 02:56:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 350/2304 [I 2022-10-10 02:56:54] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:40 [I 2022-10-10 02:56:54] ...trap/engines/engine.py.229: process: 0.85967 | load: 0.00305 [I 2022-10-10 02:56:54] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-10 02:56:54] ...optimizers/trijoint.py.151: total_norm: 39.662926 [I 2022-10-10 02:57:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 360/2304 [I 2022-10-10 02:57:02] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:27:41 [I 2022-10-10 02:57:02] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00310 [I 2022-10-10 02:57:02] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-10 02:57:02] ...optimizers/trijoint.py.151: total_norm: 13.257182 [I 2022-10-10 02:57:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 370/2304 [I 2022-10-10 02:57:11] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:32 [I 2022-10-10 02:57:11] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00323 [I 2022-10-10 02:57:11] ...trap/engines/engine.py.230: loss: 0.15719 [I 2022-10-10 02:57:11] ...optimizers/trijoint.py.151: total_norm: 20.206167 [I 2022-10-10 02:57:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 380/2304 [I 2022-10-10 02:57:19] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:27:35 [I 2022-10-10 02:57:19] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00316 [I 2022-10-10 02:57:19] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-10 02:57:19] ...optimizers/trijoint.py.151: total_norm: 13.231581 [I 2022-10-10 02:57:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 390/2304 [I 2022-10-10 02:57:28] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:20 [I 2022-10-10 02:57:28] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00313 [I 2022-10-10 02:57:28] ...trap/engines/engine.py.230: loss: 0.20833 [I 2022-10-10 02:57:28] ...optimizers/trijoint.py.151: total_norm: 25.136566 [I 2022-10-10 02:57:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 400/2304 [I 2022-10-10 02:57:37] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:27:20 [I 2022-10-10 02:57:37] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00319 [I 2022-10-10 02:57:37] ...trap/engines/engine.py.230: loss: 0.19565 [I 2022-10-10 02:57:37] ...optimizers/trijoint.py.151: total_norm: 17.585068 [I 2022-10-10 02:57:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 410/2304 [I 2022-10-10 02:57:45] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:27:02 [I 2022-10-10 02:57:45] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00326 [I 2022-10-10 02:57:45] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-10 02:57:45] ...optimizers/trijoint.py.151: total_norm: 14.276471 [I 2022-10-10 02:57:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 420/2304 [I 2022-10-10 02:57:54] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:27:02 [I 2022-10-10 02:57:54] ...trap/engines/engine.py.229: process: 0.86342 | load: 0.00316 [I 2022-10-10 02:57:54] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-10 02:57:54] ...optimizers/trijoint.py.151: total_norm: 12.294384 [I 2022-10-10 02:58:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 430/2304 [I 2022-10-10 02:58:02] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:53 [I 2022-10-10 02:58:02] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00316 [I 2022-10-10 02:58:02] ...trap/engines/engine.py.230: loss: 0.13546 [I 2022-10-10 02:58:02] ...optimizers/trijoint.py.151: total_norm: 10.671727 [I 2022-10-10 02:58:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 440/2304 [I 2022-10-10 02:58:11] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:39 [I 2022-10-10 02:58:11] ...trap/engines/engine.py.229: process: 0.83826 | load: 0.00311 [I 2022-10-10 02:58:11] ...trap/engines/engine.py.230: loss: 0.17711 [I 2022-10-10 02:58:11] ...optimizers/trijoint.py.151: total_norm: 22.257069 [I 2022-10-10 02:58:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 450/2304 [I 2022-10-10 02:58:20] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:23 [I 2022-10-10 02:58:20] ...trap/engines/engine.py.229: process: 0.84467 | load: 0.00326 [I 2022-10-10 02:58:20] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-10 02:58:20] ...optimizers/trijoint.py.151: total_norm: 13.313762 [I 2022-10-10 02:58:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 460/2304 [I 2022-10-10 02:58:28] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:15 [I 2022-10-10 02:58:28] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00322 [I 2022-10-10 02:58:28] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-10 02:58:28] ...optimizers/trijoint.py.151: total_norm: 13.230062 [I 2022-10-10 02:58:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 470/2304 [I 2022-10-10 02:58:37] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:02 [I 2022-10-10 02:58:37] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00312 [I 2022-10-10 02:58:37] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-10 02:58:37] ...optimizers/trijoint.py.151: total_norm: 11.293960 [I 2022-10-10 02:58:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 480/2304 [I 2022-10-10 02:58:45] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:26:04 [I 2022-10-10 02:58:45] ...trap/engines/engine.py.229: process: 0.86089 | load: 0.00326 [I 2022-10-10 02:58:45] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-10 02:58:45] ...optimizers/trijoint.py.151: total_norm: 12.675699 [I 2022-10-10 02:58:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 490/2304 [I 2022-10-10 02:58:54] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:55 [I 2022-10-10 02:58:54] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00342 [I 2022-10-10 02:58:54] ...trap/engines/engine.py.230: loss: 0.22302 [I 2022-10-10 02:58:54] ...optimizers/trijoint.py.151: total_norm: 13.715763 [I 2022-10-10 02:59:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 500/2304 [I 2022-10-10 02:59:03] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:46 [I 2022-10-10 02:59:03] ...trap/engines/engine.py.229: process: 0.86635 | load: 0.00310 [I 2022-10-10 02:59:03] ...trap/engines/engine.py.230: loss: 0.25136 [I 2022-10-10 02:59:03] ...optimizers/trijoint.py.151: total_norm: 23.954929 [I 2022-10-10 02:59:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 510/2304 [I 2022-10-10 02:59:11] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:43 [I 2022-10-10 02:59:11] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00310 [I 2022-10-10 02:59:11] ...trap/engines/engine.py.230: loss: 0.09483 [I 2022-10-10 02:59:11] ...optimizers/trijoint.py.151: total_norm: 9.038924 [I 2022-10-10 02:59:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 520/2304 [I 2022-10-10 02:59:20] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:19 [I 2022-10-10 02:59:20] ...trap/engines/engine.py.229: process: 0.83921 | load: 0.00329 [I 2022-10-10 02:59:20] ...trap/engines/engine.py.230: loss: 0.11844 [I 2022-10-10 02:59:20] ...optimizers/trijoint.py.151: total_norm: 11.553919 [I 2022-10-10 02:59:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 530/2304 [I 2022-10-10 02:59:28] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:19 [I 2022-10-10 02:59:28] ...trap/engines/engine.py.229: process: 0.85482 | load: 0.00312 [I 2022-10-10 02:59:28] ...trap/engines/engine.py.230: loss: 0.17403 [I 2022-10-10 02:59:28] ...optimizers/trijoint.py.151: total_norm: 18.091362 [I 2022-10-10 02:59:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 540/2304 [I 2022-10-10 02:59:37] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:06 [I 2022-10-10 02:59:37] ...trap/engines/engine.py.229: process: 0.85980 | load: 0.00313 [I 2022-10-10 02:59:37] ...trap/engines/engine.py.230: loss: 0.11200 [I 2022-10-10 02:59:37] ...optimizers/trijoint.py.151: total_norm: 14.356420 [I 2022-10-10 02:59:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 550/2304 [I 2022-10-10 02:59:45] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:24:50 [I 2022-10-10 02:59:45] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00301 [I 2022-10-10 02:59:45] ...trap/engines/engine.py.230: loss: 0.16741 [I 2022-10-10 02:59:45] ...optimizers/trijoint.py.151: total_norm: 11.800429 [I 2022-10-10 02:59:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 560/2304 [I 2022-10-10 02:59:54] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:44 [I 2022-10-10 02:59:54] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00311 [I 2022-10-10 02:59:54] ...trap/engines/engine.py.230: loss: 0.17626 [I 2022-10-10 02:59:54] ...optimizers/trijoint.py.151: total_norm: 9.641143 [I 2022-10-10 03:00:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 570/2304 [I 2022-10-10 03:00:02] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:36 [I 2022-10-10 03:00:02] ...trap/engines/engine.py.229: process: 0.86309 | load: 0.00299 [I 2022-10-10 03:00:02] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-10 03:00:02] ...optimizers/trijoint.py.151: total_norm: 35.979027 [I 2022-10-10 03:00:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 580/2304 [I 2022-10-10 03:00:11] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:13 [I 2022-10-10 03:00:11] ...trap/engines/engine.py.229: process: 0.83456 | load: 0.00305 [I 2022-10-10 03:00:11] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-10 03:00:11] ...optimizers/trijoint.py.151: total_norm: 16.103333 [I 2022-10-10 03:00:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 590/2304 [I 2022-10-10 03:00:19] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:25 [I 2022-10-10 03:00:19] ...trap/engines/engine.py.229: process: 0.86655 | load: 0.00308 [I 2022-10-10 03:00:19] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-10 03:00:19] ...optimizers/trijoint.py.151: total_norm: 11.423008 [I 2022-10-10 03:00:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 600/2304 [I 2022-10-10 03:00:28] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:07 [I 2022-10-10 03:00:28] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00310 [I 2022-10-10 03:00:28] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-10 03:00:28] ...optimizers/trijoint.py.151: total_norm: 32.952713 [I 2022-10-10 03:00:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 610/2304 [I 2022-10-10 03:00:36] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:23:52 [I 2022-10-10 03:00:36] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00311 [I 2022-10-10 03:00:36] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-10 03:00:36] ...optimizers/trijoint.py.151: total_norm: 10.994576 [I 2022-10-10 03:00:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 620/2304 [I 2022-10-10 03:00:45] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:51 [I 2022-10-10 03:00:45] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00297 [I 2022-10-10 03:00:45] ...trap/engines/engine.py.230: loss: 0.13098 [I 2022-10-10 03:00:45] ...optimizers/trijoint.py.151: total_norm: 17.896067 [I 2022-10-10 03:00:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 630/2304 [I 2022-10-10 03:00:53] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:44 [I 2022-10-10 03:00:53] ...trap/engines/engine.py.229: process: 0.86382 | load: 0.00305 [I 2022-10-10 03:00:53] ...trap/engines/engine.py.230: loss: 0.20105 [I 2022-10-10 03:00:53] ...optimizers/trijoint.py.151: total_norm: 23.010405 [I 2022-10-10 03:01:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 640/2304 [I 2022-10-10 03:01:02] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:40 [I 2022-10-10 03:01:02] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00306 [I 2022-10-10 03:01:02] ...trap/engines/engine.py.230: loss: 0.16841 [I 2022-10-10 03:01:02] ...optimizers/trijoint.py.151: total_norm: 11.204294 [I 2022-10-10 03:01:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 650/2304 [I 2022-10-10 03:01:11] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:22 [I 2022-10-10 03:01:11] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00353 [I 2022-10-10 03:01:11] ...trap/engines/engine.py.230: loss: 0.17531 [I 2022-10-10 03:01:11] ...optimizers/trijoint.py.151: total_norm: 13.396955 [I 2022-10-10 03:01:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 660/2304 [I 2022-10-10 03:01:19] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:18 [I 2022-10-10 03:01:19] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00307 [I 2022-10-10 03:01:19] ...trap/engines/engine.py.230: loss: 0.12200 [I 2022-10-10 03:01:19] ...optimizers/trijoint.py.151: total_norm: 9.298364 [I 2022-10-10 03:01:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 670/2304 [I 2022-10-10 03:01:28] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:15 [I 2022-10-10 03:01:28] ...trap/engines/engine.py.229: process: 0.85647 | load: 0.00310 [I 2022-10-10 03:01:28] ...trap/engines/engine.py.230: loss: 0.11200 [I 2022-10-10 03:01:28] ...optimizers/trijoint.py.151: total_norm: 12.367870 [I 2022-10-10 03:01:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 680/2304 [I 2022-10-10 03:01:36] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:58 [I 2022-10-10 03:01:36] ...trap/engines/engine.py.229: process: 0.84080 | load: 0.00309 [I 2022-10-10 03:01:36] ...trap/engines/engine.py.230: loss: 0.17809 [I 2022-10-10 03:01:36] ...optimizers/trijoint.py.151: total_norm: 8.147322 [I 2022-10-10 03:01:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 690/2304 [I 2022-10-10 03:01:45] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:53 [I 2022-10-10 03:01:45] ...trap/engines/engine.py.229: process: 0.85883 | load: 0.00297 [I 2022-10-10 03:01:45] ...trap/engines/engine.py.230: loss: 0.17373 [I 2022-10-10 03:01:45] ...optimizers/trijoint.py.151: total_norm: 15.024055 [I 2022-10-10 03:01:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 700/2304 [I 2022-10-10 03:01:53] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:41 [I 2022-10-10 03:01:53] ...trap/engines/engine.py.229: process: 0.81966 | load: 0.00303 [I 2022-10-10 03:01:53] ...trap/engines/engine.py.230: loss: 0.15506 [I 2022-10-10 03:01:53] ...optimizers/trijoint.py.151: total_norm: 15.533945 [I 2022-10-10 03:02:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 710/2304 [I 2022-10-10 03:02:02] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:27 [I 2022-10-10 03:02:02] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00301 [I 2022-10-10 03:02:02] ...trap/engines/engine.py.230: loss: 0.10831 [I 2022-10-10 03:02:02] ...optimizers/trijoint.py.151: total_norm: 9.687894 [I 2022-10-10 03:02:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 720/2304 [I 2022-10-10 03:02:10] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:19 [I 2022-10-10 03:02:10] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00309 [I 2022-10-10 03:02:10] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-10 03:02:10] ...optimizers/trijoint.py.151: total_norm: 17.608286 [I 2022-10-10 03:02:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 730/2304 [I 2022-10-10 03:02:19] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:19 [I 2022-10-10 03:02:19] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00304 [I 2022-10-10 03:02:19] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-10 03:02:19] ...optimizers/trijoint.py.151: total_norm: 10.015816 [I 2022-10-10 03:02:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 740/2304 [I 2022-10-10 03:02:27] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:18 [I 2022-10-10 03:02:27] ...trap/engines/engine.py.229: process: 0.87557 | load: 0.00311 [I 2022-10-10 03:02:27] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-10 03:02:27] ...optimizers/trijoint.py.151: total_norm: 17.012466 [I 2022-10-10 03:02:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 750/2304 [I 2022-10-10 03:02:36] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:02 [I 2022-10-10 03:02:36] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00309 [I 2022-10-10 03:02:36] ...trap/engines/engine.py.230: loss: 0.16819 [I 2022-10-10 03:02:36] ...optimizers/trijoint.py.151: total_norm: 22.579458 [I 2022-10-10 03:02:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 760/2304 [I 2022-10-10 03:02:44] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:55 [I 2022-10-10 03:02:44] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00308 [I 2022-10-10 03:02:44] ...trap/engines/engine.py.230: loss: 0.17708 [I 2022-10-10 03:02:44] ...optimizers/trijoint.py.151: total_norm: 20.663927 [I 2022-10-10 03:02:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 770/2304 [I 2022-10-10 03:02:53] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:43 [I 2022-10-10 03:02:53] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00300 [I 2022-10-10 03:02:53] ...trap/engines/engine.py.230: loss: 0.11630 [I 2022-10-10 03:02:53] ...optimizers/trijoint.py.151: total_norm: 10.259826 [I 2022-10-10 03:03:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 780/2304 [I 2022-10-10 03:03:01] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:41 [I 2022-10-10 03:03:01] ...trap/engines/engine.py.229: process: 0.84485 | load: 0.00301 [I 2022-10-10 03:03:01] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-10 03:03:01] ...optimizers/trijoint.py.151: total_norm: 24.447067 [I 2022-10-10 03:03:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 790/2304 [I 2022-10-10 03:03:10] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:25 [I 2022-10-10 03:03:10] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00302 [I 2022-10-10 03:03:10] ...trap/engines/engine.py.230: loss: 0.14483 [I 2022-10-10 03:03:10] ...optimizers/trijoint.py.151: total_norm: 17.609993 [I 2022-10-10 03:03:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 800/2304 [I 2022-10-10 03:03:18] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:22 [I 2022-10-10 03:03:18] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00313 [I 2022-10-10 03:03:18] ...trap/engines/engine.py.230: loss: 0.15035 [I 2022-10-10 03:03:18] ...optimizers/trijoint.py.151: total_norm: 14.351130 [I 2022-10-10 03:03:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 810/2304 [I 2022-10-10 03:03:27] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:12 [I 2022-10-10 03:03:27] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00313 [I 2022-10-10 03:03:27] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-10 03:03:27] ...optimizers/trijoint.py.151: total_norm: 16.184614 [I 2022-10-10 03:03:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 820/2304 [I 2022-10-10 03:03:35] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:02 [I 2022-10-10 03:03:35] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00300 [I 2022-10-10 03:03:35] ...trap/engines/engine.py.230: loss: 0.23685 [I 2022-10-10 03:03:35] ...optimizers/trijoint.py.151: total_norm: 31.266462 [I 2022-10-10 03:03:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 830/2304 [I 2022-10-10 03:03:44] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:49 [I 2022-10-10 03:03:44] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00298 [I 2022-10-10 03:03:44] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-10 03:03:44] ...optimizers/trijoint.py.151: total_norm: 16.258833 [I 2022-10-10 03:03:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 840/2304 [I 2022-10-10 03:03:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:48 [I 2022-10-10 03:03:52] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00343 [I 2022-10-10 03:03:52] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-10 03:03:52] ...optimizers/trijoint.py.151: total_norm: 10.615368 [I 2022-10-10 03:04:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 850/2304 [I 2022-10-10 03:04:01] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:41 [I 2022-10-10 03:04:01] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00295 [I 2022-10-10 03:04:01] ...trap/engines/engine.py.230: loss: 0.12774 [I 2022-10-10 03:04:01] ...optimizers/trijoint.py.151: total_norm: 16.804810 [I 2022-10-10 03:04:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 860/2304 [I 2022-10-10 03:04:10] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:30 [I 2022-10-10 03:04:10] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00308 [I 2022-10-10 03:04:10] ...trap/engines/engine.py.230: loss: 0.18250 [I 2022-10-10 03:04:10] ...optimizers/trijoint.py.151: total_norm: 12.975348 [I 2022-10-10 03:04:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 870/2304 [I 2022-10-10 03:04:18] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:16 [I 2022-10-10 03:04:18] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00317 [I 2022-10-10 03:04:18] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-10 03:04:18] ...optimizers/trijoint.py.151: total_norm: 16.348993 [I 2022-10-10 03:04:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 880/2304 [I 2022-10-10 03:04:27] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:22 [I 2022-10-10 03:04:27] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00312 [I 2022-10-10 03:04:27] ...trap/engines/engine.py.230: loss: 0.11751 [I 2022-10-10 03:04:27] ...optimizers/trijoint.py.151: total_norm: 20.435444 [I 2022-10-10 03:04:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 890/2304 [I 2022-10-10 03:04:35] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:06 [I 2022-10-10 03:04:35] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00314 [I 2022-10-10 03:04:35] ...trap/engines/engine.py.230: loss: 0.23288 [I 2022-10-10 03:04:35] ...optimizers/trijoint.py.151: total_norm: 25.342119 [I 2022-10-10 03:04:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 900/2304 [I 2022-10-10 03:04:44] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:20:00 [I 2022-10-10 03:04:44] ...trap/engines/engine.py.229: process: 0.86286 | load: 0.00314 [I 2022-10-10 03:04:44] ...trap/engines/engine.py.230: loss: 0.14283 [I 2022-10-10 03:04:44] ...optimizers/trijoint.py.151: total_norm: 12.099163 [I 2022-10-10 03:04:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 910/2304 [I 2022-10-10 03:04:52] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:52 [I 2022-10-10 03:04:52] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00312 [I 2022-10-10 03:04:52] ...trap/engines/engine.py.230: loss: 0.22976 [I 2022-10-10 03:04:52] ...optimizers/trijoint.py.151: total_norm: 15.138578 [I 2022-10-10 03:05:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 920/2304 [I 2022-10-10 03:05:01] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:42 [I 2022-10-10 03:05:01] ...trap/engines/engine.py.229: process: 0.84433 | load: 0.00316 [I 2022-10-10 03:05:01] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-10 03:05:01] ...optimizers/trijoint.py.151: total_norm: 13.572860 [I 2022-10-10 03:05:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 930/2304 [I 2022-10-10 03:05:09] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:37 [I 2022-10-10 03:05:09] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00313 [I 2022-10-10 03:05:09] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-10 03:05:09] ...optimizers/trijoint.py.151: total_norm: 19.349819 [I 2022-10-10 03:05:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 940/2304 [I 2022-10-10 03:05:18] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:23 [I 2022-10-10 03:05:18] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00313 [I 2022-10-10 03:05:18] ...trap/engines/engine.py.230: loss: 0.15182 [I 2022-10-10 03:05:18] ...optimizers/trijoint.py.151: total_norm: 12.689403 [I 2022-10-10 03:05:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 950/2304 [I 2022-10-10 03:05:27] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:20 [I 2022-10-10 03:05:27] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00313 [I 2022-10-10 03:05:27] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-10 03:05:27] ...optimizers/trijoint.py.151: total_norm: 10.160706 [I 2022-10-10 03:05:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 960/2304 [I 2022-10-10 03:05:35] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:11 [I 2022-10-10 03:05:35] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00312 [I 2022-10-10 03:05:35] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-10 03:05:35] ...optimizers/trijoint.py.151: total_norm: 13.622158 [I 2022-10-10 03:05:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 970/2304 [I 2022-10-10 03:05:44] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:54 [I 2022-10-10 03:05:44] ...trap/engines/engine.py.229: process: 0.85991 | load: 0.00313 [I 2022-10-10 03:05:44] ...trap/engines/engine.py.230: loss: 0.17319 [I 2022-10-10 03:05:44] ...optimizers/trijoint.py.151: total_norm: 24.953203 [I 2022-10-10 03:05:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 980/2304 [I 2022-10-10 03:05:52] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:48 [I 2022-10-10 03:05:52] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00315 [I 2022-10-10 03:05:52] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-10 03:05:52] ...optimizers/trijoint.py.151: total_norm: 11.975790 [I 2022-10-10 03:06:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 990/2304 [I 2022-10-10 03:06:01] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:47 [I 2022-10-10 03:06:01] ...trap/engines/engine.py.229: process: 0.85911 | load: 0.00314 [I 2022-10-10 03:06:01] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-10 03:06:01] ...optimizers/trijoint.py.151: total_norm: 8.295258 [I 2022-10-10 03:06:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 1000/2304 [I 2022-10-10 03:06:10] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:33 [I 2022-10-10 03:06:10] ...trap/engines/engine.py.229: process: 0.86564 | load: 0.00312 [I 2022-10-10 03:06:10] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-10 03:06:10] ...optimizers/trijoint.py.151: total_norm: 15.379005 [I 2022-10-10 03:06:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 1010/2304 [I 2022-10-10 03:06:18] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:25 [I 2022-10-10 03:06:18] ...trap/engines/engine.py.229: process: 0.86029 | load: 0.00314 [I 2022-10-10 03:06:18] ...trap/engines/engine.py.230: loss: 0.13752 [I 2022-10-10 03:06:18] ...optimizers/trijoint.py.151: total_norm: 18.225630 [I 2022-10-10 03:06:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 1020/2304 [I 2022-10-10 03:06:27] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:11 [I 2022-10-10 03:06:27] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00311 [I 2022-10-10 03:06:27] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-10 03:06:27] ...optimizers/trijoint.py.151: total_norm: 20.364803 [I 2022-10-10 03:06:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 1030/2304 [I 2022-10-10 03:06:35] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:10 [I 2022-10-10 03:06:35] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00313 [I 2022-10-10 03:06:35] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-10 03:06:35] ...optimizers/trijoint.py.151: total_norm: 20.499018 [I 2022-10-10 03:06:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 1040/2304 [I 2022-10-10 03:06:44] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:17:56 [I 2022-10-10 03:06:44] ...trap/engines/engine.py.229: process: 0.84809 | load: 0.00313 [I 2022-10-10 03:06:44] ...trap/engines/engine.py.230: loss: 0.11310 [I 2022-10-10 03:06:44] ...optimizers/trijoint.py.151: total_norm: 7.984520 [I 2022-10-10 03:06:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 1050/2304 [I 2022-10-10 03:06:52] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:51 [I 2022-10-10 03:06:52] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00311 [I 2022-10-10 03:06:52] ...trap/engines/engine.py.230: loss: 0.13737 [I 2022-10-10 03:06:52] ...optimizers/trijoint.py.151: total_norm: 17.059465 [I 2022-10-10 03:07:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 1060/2304 [I 2022-10-10 03:07:01] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:38 [I 2022-10-10 03:07:01] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00311 [I 2022-10-10 03:07:01] ...trap/engines/engine.py.230: loss: 0.18089 [I 2022-10-10 03:07:01] ...optimizers/trijoint.py.151: total_norm: 50.388668 [I 2022-10-10 03:07:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 1070/2304 [I 2022-10-10 03:07:09] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:30 [I 2022-10-10 03:07:09] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00303 [I 2022-10-10 03:07:09] ...trap/engines/engine.py.230: loss: 0.21252 [I 2022-10-10 03:07:09] ...optimizers/trijoint.py.151: total_norm: 8.323661 [I 2022-10-10 03:07:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 1080/2304 [I 2022-10-10 03:07:18] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:19 [I 2022-10-10 03:07:18] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00300 [I 2022-10-10 03:07:18] ...trap/engines/engine.py.230: loss: 0.16424 [I 2022-10-10 03:07:18] ...optimizers/trijoint.py.151: total_norm: 20.773619 [I 2022-10-10 03:07:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 1090/2304 [I 2022-10-10 03:07:26] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:09 [I 2022-10-10 03:07:26] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00302 [I 2022-10-10 03:07:26] ...trap/engines/engine.py.230: loss: 0.19456 [I 2022-10-10 03:07:26] ...optimizers/trijoint.py.151: total_norm: 13.804880 [I 2022-10-10 03:07:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 1100/2304 [I 2022-10-10 03:07:35] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:01 [I 2022-10-10 03:07:35] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00306 [I 2022-10-10 03:07:35] ...trap/engines/engine.py.230: loss: 0.12218 [I 2022-10-10 03:07:35] ...optimizers/trijoint.py.151: total_norm: 14.109543 [I 2022-10-10 03:07:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 1110/2304 [I 2022-10-10 03:07:43] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:54 [I 2022-10-10 03:07:43] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00302 [I 2022-10-10 03:07:43] ...trap/engines/engine.py.230: loss: 0.13611 [I 2022-10-10 03:07:43] ...optimizers/trijoint.py.151: total_norm: 16.686289 [I 2022-10-10 03:07:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 1120/2304 [I 2022-10-10 03:07:52] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:44 [I 2022-10-10 03:07:52] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00308 [I 2022-10-10 03:07:52] ...trap/engines/engine.py.230: loss: 0.26122 [I 2022-10-10 03:07:52] ...optimizers/trijoint.py.151: total_norm: 12.997448 [I 2022-10-10 03:08:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 1130/2304 [I 2022-10-10 03:08:00] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:39 [I 2022-10-10 03:08:00] ...trap/engines/engine.py.229: process: 0.86316 | load: 0.00303 [I 2022-10-10 03:08:00] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-10 03:08:00] ...optimizers/trijoint.py.151: total_norm: 16.005209 [I 2022-10-10 03:08:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 1140/2304 [I 2022-10-10 03:08:09] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:29 [I 2022-10-10 03:08:09] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00304 [I 2022-10-10 03:08:09] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-10 03:08:09] ...optimizers/trijoint.py.151: total_norm: 10.613586 [I 2022-10-10 03:08:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 1150/2304 [I 2022-10-10 03:08:17] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:17 [I 2022-10-10 03:08:17] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00313 [I 2022-10-10 03:08:17] ...trap/engines/engine.py.230: loss: 0.13547 [I 2022-10-10 03:08:17] ...optimizers/trijoint.py.151: total_norm: 28.798700 [I 2022-10-10 03:08:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 1160/2304 [I 2022-10-10 03:08:26] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:11 [I 2022-10-10 03:08:26] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00313 [I 2022-10-10 03:08:26] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-10 03:08:26] ...optimizers/trijoint.py.151: total_norm: 12.274368 [I 2022-10-10 03:08:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 1170/2304 [I 2022-10-10 03:08:34] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:02 [I 2022-10-10 03:08:34] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00305 [I 2022-10-10 03:08:34] ...trap/engines/engine.py.230: loss: 0.21454 [I 2022-10-10 03:08:34] ...optimizers/trijoint.py.151: total_norm: 16.118193 [I 2022-10-10 03:08:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 1180/2304 [I 2022-10-10 03:08:43] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:50 [I 2022-10-10 03:08:43] ...trap/engines/engine.py.229: process: 0.84145 | load: 0.00309 [I 2022-10-10 03:08:43] ...trap/engines/engine.py.230: loss: 0.16904 [I 2022-10-10 03:08:43] ...optimizers/trijoint.py.151: total_norm: 15.523438 [I 2022-10-10 03:08:51] ...trap/engines/engine.py.224: train: epoch 64 | batch 1190/2304 [I 2022-10-10 03:08:51] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:39 [I 2022-10-10 03:08:51] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00303 [I 2022-10-10 03:08:51] ...trap/engines/engine.py.230: loss: 0.17428 [I 2022-10-10 03:08:51] ...optimizers/trijoint.py.151: total_norm: 17.210394 [I 2022-10-10 03:09:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 1200/2304 [I 2022-10-10 03:09:00] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:37 [I 2022-10-10 03:09:00] ...trap/engines/engine.py.229: process: 0.83342 | load: 0.00312 [I 2022-10-10 03:09:00] ...trap/engines/engine.py.230: loss: 0.11222 [I 2022-10-10 03:09:00] ...optimizers/trijoint.py.151: total_norm: 12.472355 [I 2022-10-10 03:09:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 1210/2304 [I 2022-10-10 03:09:08] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:30 [I 2022-10-10 03:09:08] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00302 [I 2022-10-10 03:09:08] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-10 03:09:08] ...optimizers/trijoint.py.151: total_norm: 16.892675 [I 2022-10-10 03:09:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 1220/2304 [I 2022-10-10 03:09:17] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:20 [I 2022-10-10 03:09:17] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00297 [I 2022-10-10 03:09:17] ...trap/engines/engine.py.230: loss: 0.12742 [I 2022-10-10 03:09:17] ...optimizers/trijoint.py.151: total_norm: 15.786815 [I 2022-10-10 03:09:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 1230/2304 [I 2022-10-10 03:09:25] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:08 [I 2022-10-10 03:09:25] ...trap/engines/engine.py.229: process: 0.82987 | load: 0.00305 [I 2022-10-10 03:09:25] ...trap/engines/engine.py.230: loss: 0.10154 [I 2022-10-10 03:09:25] ...optimizers/trijoint.py.151: total_norm: 15.013972 [I 2022-10-10 03:09:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 1240/2304 [I 2022-10-10 03:09:34] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:59 [I 2022-10-10 03:09:34] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00314 [I 2022-10-10 03:09:34] ...trap/engines/engine.py.230: loss: 0.14643 [I 2022-10-10 03:09:34] ...optimizers/trijoint.py.151: total_norm: 8.502974 [I 2022-10-10 03:09:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 1250/2304 [I 2022-10-10 03:09:42] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:00 [I 2022-10-10 03:09:42] ...trap/engines/engine.py.229: process: 0.86283 | load: 0.00312 [I 2022-10-10 03:09:42] ...trap/engines/engine.py.230: loss: 0.09920 [I 2022-10-10 03:09:42] ...optimizers/trijoint.py.151: total_norm: 13.542241 [I 2022-10-10 03:09:51] ...trap/engines/engine.py.224: train: epoch 64 | batch 1260/2304 [I 2022-10-10 03:09:51] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:45 [I 2022-10-10 03:09:51] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00296 [I 2022-10-10 03:09:51] ...trap/engines/engine.py.230: loss: 0.17243 [I 2022-10-10 03:09:51] ...optimizers/trijoint.py.151: total_norm: 15.048518 [I 2022-10-10 03:09:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 1270/2304 [I 2022-10-10 03:09:59] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:34 [I 2022-10-10 03:09:59] ...trap/engines/engine.py.229: process: 0.83779 | load: 0.00311 [I 2022-10-10 03:09:59] ...trap/engines/engine.py.230: loss: 0.13617 [I 2022-10-10 03:09:59] ...optimizers/trijoint.py.151: total_norm: 18.324823 [I 2022-10-10 03:10:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 1280/2304 [I 2022-10-10 03:10:08] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:29 [I 2022-10-10 03:10:08] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00307 [I 2022-10-10 03:10:08] ...trap/engines/engine.py.230: loss: 0.16162 [I 2022-10-10 03:10:08] ...optimizers/trijoint.py.151: total_norm: 33.345490 [I 2022-10-10 03:10:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 1290/2304 [I 2022-10-10 03:10:16] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:20 [I 2022-10-10 03:10:16] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00313 [I 2022-10-10 03:10:16] ...trap/engines/engine.py.230: loss: 0.12278 [I 2022-10-10 03:10:16] ...optimizers/trijoint.py.151: total_norm: 13.410346 [I 2022-10-10 03:10:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 1300/2304 [I 2022-10-10 03:10:25] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:12 [I 2022-10-10 03:10:25] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00314 [I 2022-10-10 03:10:25] ...trap/engines/engine.py.230: loss: 0.22756 [I 2022-10-10 03:10:25] ...optimizers/trijoint.py.151: total_norm: 19.390289 [I 2022-10-10 03:10:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1310/2304 [I 2022-10-10 03:10:33] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:05 [I 2022-10-10 03:10:33] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00323 [I 2022-10-10 03:10:33] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-10 03:10:33] ...optimizers/trijoint.py.151: total_norm: 14.824828 [I 2022-10-10 03:10:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 1320/2304 [I 2022-10-10 03:10:42] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:52 [I 2022-10-10 03:10:42] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00312 [I 2022-10-10 03:10:42] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-10 03:10:42] ...optimizers/trijoint.py.151: total_norm: 31.644440 [I 2022-10-10 03:10:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1330/2304 [I 2022-10-10 03:10:50] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:49 [I 2022-10-10 03:10:50] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00300 [I 2022-10-10 03:10:50] ...trap/engines/engine.py.230: loss: 0.10484 [I 2022-10-10 03:10:50] ...optimizers/trijoint.py.151: total_norm: 7.958651 [I 2022-10-10 03:10:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 1340/2304 [I 2022-10-10 03:10:59] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:39 [I 2022-10-10 03:10:59] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00308 [I 2022-10-10 03:10:59] ...trap/engines/engine.py.230: loss: 0.10306 [I 2022-10-10 03:10:59] ...optimizers/trijoint.py.151: total_norm: 11.857533 [I 2022-10-10 03:11:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1350/2304 [I 2022-10-10 03:11:07] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:24 [I 2022-10-10 03:11:07] ...trap/engines/engine.py.229: process: 0.83520 | load: 0.00308 [I 2022-10-10 03:11:07] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-10 03:11:07] ...optimizers/trijoint.py.151: total_norm: 11.503164 [I 2022-10-10 03:11:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 1360/2304 [I 2022-10-10 03:11:16] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:18 [I 2022-10-10 03:11:16] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00312 [I 2022-10-10 03:11:16] ...trap/engines/engine.py.230: loss: 0.12835 [I 2022-10-10 03:11:16] ...optimizers/trijoint.py.151: total_norm: 11.388781 [I 2022-10-10 03:11:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1370/2304 [I 2022-10-10 03:11:24] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:17 [I 2022-10-10 03:11:24] ...trap/engines/engine.py.229: process: 0.85680 | load: 0.00315 [I 2022-10-10 03:11:24] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-10 03:11:24] ...optimizers/trijoint.py.151: total_norm: 15.735968 [I 2022-10-10 03:11:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1380/2304 [I 2022-10-10 03:11:33] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:05 [I 2022-10-10 03:11:33] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00310 [I 2022-10-10 03:11:33] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-10 03:11:33] ...optimizers/trijoint.py.151: total_norm: 8.330203 [I 2022-10-10 03:11:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1390/2304 [I 2022-10-10 03:11:41] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:58 [I 2022-10-10 03:11:41] ...trap/engines/engine.py.229: process: 0.85786 | load: 0.00312 [I 2022-10-10 03:11:41] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-10 03:11:41] ...optimizers/trijoint.py.151: total_norm: 68.607628 [I 2022-10-10 03:11:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1400/2304 [I 2022-10-10 03:11:50] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:53 [I 2022-10-10 03:11:50] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00311 [I 2022-10-10 03:11:50] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-10 03:11:50] ...optimizers/trijoint.py.151: total_norm: 19.325718 [I 2022-10-10 03:11:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 1410/2304 [I 2022-10-10 03:11:58] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:43 [I 2022-10-10 03:11:58] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00311 [I 2022-10-10 03:11:58] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-10 03:11:58] ...optimizers/trijoint.py.151: total_norm: 20.715563 [I 2022-10-10 03:12:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1420/2304 [I 2022-10-10 03:12:07] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:34 [I 2022-10-10 03:12:07] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00312 [I 2022-10-10 03:12:07] ...trap/engines/engine.py.230: loss: 0.13849 [I 2022-10-10 03:12:07] ...optimizers/trijoint.py.151: total_norm: 14.006353 [I 2022-10-10 03:12:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 1430/2304 [I 2022-10-10 03:12:15] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:29 [I 2022-10-10 03:12:15] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00310 [I 2022-10-10 03:12:15] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-10 03:12:15] ...optimizers/trijoint.py.151: total_norm: 17.481474 [I 2022-10-10 03:12:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1440/2304 [I 2022-10-10 03:12:24] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:18 [I 2022-10-10 03:12:24] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00311 [I 2022-10-10 03:12:24] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-10 03:12:24] ...optimizers/trijoint.py.151: total_norm: 24.040977 [I 2022-10-10 03:12:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1450/2304 [I 2022-10-10 03:12:33] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:08 [I 2022-10-10 03:12:33] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00314 [I 2022-10-10 03:12:33] ...trap/engines/engine.py.230: loss: 0.12173 [I 2022-10-10 03:12:33] ...optimizers/trijoint.py.151: total_norm: 13.952142 [I 2022-10-10 03:12:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1460/2304 [I 2022-10-10 03:12:41] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:04 [I 2022-10-10 03:12:41] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00313 [I 2022-10-10 03:12:41] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-10 03:12:41] ...optimizers/trijoint.py.151: total_norm: 10.200110 [I 2022-10-10 03:12:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1470/2304 [I 2022-10-10 03:12:50] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:52 [I 2022-10-10 03:12:50] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00310 [I 2022-10-10 03:12:50] ...trap/engines/engine.py.230: loss: 0.10354 [I 2022-10-10 03:12:50] ...optimizers/trijoint.py.151: total_norm: 10.374949 [I 2022-10-10 03:12:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 1480/2304 [I 2022-10-10 03:12:58] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:45 [I 2022-10-10 03:12:58] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00317 [I 2022-10-10 03:12:58] ...trap/engines/engine.py.230: loss: 0.13630 [I 2022-10-10 03:12:58] ...optimizers/trijoint.py.151: total_norm: 13.293464 [I 2022-10-10 03:13:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1490/2304 [I 2022-10-10 03:13:07] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:34 [I 2022-10-10 03:13:07] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00316 [I 2022-10-10 03:13:07] ...trap/engines/engine.py.230: loss: 0.09014 [I 2022-10-10 03:13:07] ...optimizers/trijoint.py.151: total_norm: 10.292255 [I 2022-10-10 03:13:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 1500/2304 [I 2022-10-10 03:13:16] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:30 [I 2022-10-10 03:13:16] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00313 [I 2022-10-10 03:13:16] ...trap/engines/engine.py.230: loss: 0.19925 [I 2022-10-10 03:13:16] ...optimizers/trijoint.py.151: total_norm: 30.659742 [I 2022-10-10 03:13:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1510/2304 [I 2022-10-10 03:13:24] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:17 [I 2022-10-10 03:13:24] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00313 [I 2022-10-10 03:13:24] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-10 03:13:24] ...optimizers/trijoint.py.151: total_norm: 22.052063 [I 2022-10-10 03:13:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1520/2304 [I 2022-10-10 03:13:33] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:10 [I 2022-10-10 03:13:33] ...trap/engines/engine.py.229: process: 0.85977 | load: 0.00312 [I 2022-10-10 03:13:33] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-10 03:13:33] ...optimizers/trijoint.py.151: total_norm: 11.693412 [I 2022-10-10 03:13:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1530/2304 [I 2022-10-10 03:13:41] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:04 [I 2022-10-10 03:13:41] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00310 [I 2022-10-10 03:13:41] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-10 03:13:41] ...optimizers/trijoint.py.151: total_norm: 21.844639 [I 2022-10-10 03:13:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1540/2304 [I 2022-10-10 03:13:50] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:56 [I 2022-10-10 03:13:50] ...trap/engines/engine.py.229: process: 0.86043 | load: 0.00313 [I 2022-10-10 03:13:50] ...trap/engines/engine.py.230: loss: 0.09599 [I 2022-10-10 03:13:50] ...optimizers/trijoint.py.151: total_norm: 10.988854 [I 2022-10-10 03:13:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 1550/2304 [I 2022-10-10 03:13:59] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:45 [I 2022-10-10 03:13:59] ...trap/engines/engine.py.229: process: 0.85035 | load: 0.00311 [I 2022-10-10 03:13:59] ...trap/engines/engine.py.230: loss: 0.11936 [I 2022-10-10 03:13:59] ...optimizers/trijoint.py.151: total_norm: 11.019348 [I 2022-10-10 03:14:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1560/2304 [I 2022-10-10 03:14:07] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:41 [I 2022-10-10 03:14:07] ...trap/engines/engine.py.229: process: 0.87445 | load: 0.00311 [I 2022-10-10 03:14:07] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-10 03:14:07] ...optimizers/trijoint.py.151: total_norm: 30.176128 [I 2022-10-10 03:14:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 1570/2304 [I 2022-10-10 03:14:16] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:24 [I 2022-10-10 03:14:16] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00304 [I 2022-10-10 03:14:16] ...trap/engines/engine.py.230: loss: 0.12190 [I 2022-10-10 03:14:16] ...optimizers/trijoint.py.151: total_norm: 13.534369 [I 2022-10-10 03:14:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1580/2304 [I 2022-10-10 03:14:24] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:11 [I 2022-10-10 03:14:24] ...trap/engines/engine.py.229: process: 0.82217 | load: 0.00306 [I 2022-10-10 03:14:24] ...trap/engines/engine.py.230: loss: 0.10537 [I 2022-10-10 03:14:24] ...optimizers/trijoint.py.151: total_norm: 8.306022 [I 2022-10-10 03:14:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1590/2304 [I 2022-10-10 03:14:33] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:06 [I 2022-10-10 03:14:33] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00310 [I 2022-10-10 03:14:33] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-10 03:14:33] ...optimizers/trijoint.py.151: total_norm: 33.908676 [I 2022-10-10 03:14:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1600/2304 [I 2022-10-10 03:14:41] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:09:59 [I 2022-10-10 03:14:41] ...trap/engines/engine.py.229: process: 0.86875 | load: 0.00318 [I 2022-10-10 03:14:41] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-10 03:14:41] ...optimizers/trijoint.py.151: total_norm: 12.208574 [I 2022-10-10 03:14:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1610/2304 [I 2022-10-10 03:14:50] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:09:52 [I 2022-10-10 03:14:50] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00310 [I 2022-10-10 03:14:50] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-10 03:14:50] ...optimizers/trijoint.py.151: total_norm: 16.156479 [I 2022-10-10 03:14:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 1620/2304 [I 2022-10-10 03:14:58] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:38 [I 2022-10-10 03:14:58] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00301 [I 2022-10-10 03:14:58] ...trap/engines/engine.py.230: loss: 0.09679 [I 2022-10-10 03:14:58] ...optimizers/trijoint.py.151: total_norm: 17.733843 [I 2022-10-10 03:15:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1630/2304 [I 2022-10-10 03:15:07] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:29 [I 2022-10-10 03:15:07] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00308 [I 2022-10-10 03:15:07] ...trap/engines/engine.py.230: loss: 0.12300 [I 2022-10-10 03:15:07] ...optimizers/trijoint.py.151: total_norm: 24.619747 [I 2022-10-10 03:15:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 1640/2304 [I 2022-10-10 03:15:15] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:26 [I 2022-10-10 03:15:15] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00307 [I 2022-10-10 03:15:15] ...trap/engines/engine.py.230: loss: 0.14258 [I 2022-10-10 03:15:15] ...optimizers/trijoint.py.151: total_norm: 14.351905 [I 2022-10-10 03:15:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1650/2304 [I 2022-10-10 03:15:24] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:16 [I 2022-10-10 03:15:24] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00304 [I 2022-10-10 03:15:24] ...trap/engines/engine.py.230: loss: 0.19762 [I 2022-10-10 03:15:24] ...optimizers/trijoint.py.151: total_norm: 18.378836 [I 2022-10-10 03:15:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 1660/2304 [I 2022-10-10 03:15:32] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:07 [I 2022-10-10 03:15:32] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00298 [I 2022-10-10 03:15:32] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-10 03:15:32] ...optimizers/trijoint.py.151: total_norm: 19.562351 [I 2022-10-10 03:15:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1670/2304 [I 2022-10-10 03:15:41] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:08:57 [I 2022-10-10 03:15:41] ...trap/engines/engine.py.229: process: 0.84877 | load: 0.00309 [I 2022-10-10 03:15:41] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-10 03:15:41] ...optimizers/trijoint.py.151: total_norm: 14.806260 [I 2022-10-10 03:15:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 1680/2304 [I 2022-10-10 03:15:49] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:47 [I 2022-10-10 03:15:49] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00298 [I 2022-10-10 03:15:49] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-10 03:15:49] ...optimizers/trijoint.py.151: total_norm: 19.498613 [I 2022-10-10 03:15:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 1690/2304 [I 2022-10-10 03:15:58] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:37 [I 2022-10-10 03:15:58] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00295 [I 2022-10-10 03:15:58] ...trap/engines/engine.py.230: loss: 0.12292 [I 2022-10-10 03:15:58] ...optimizers/trijoint.py.151: total_norm: 10.476589 [I 2022-10-10 03:16:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 1700/2304 [I 2022-10-10 03:16:06] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:29 [I 2022-10-10 03:16:06] ...trap/engines/engine.py.229: process: 0.83663 | load: 0.00296 [I 2022-10-10 03:16:06] ...trap/engines/engine.py.230: loss: 0.16534 [I 2022-10-10 03:16:06] ...optimizers/trijoint.py.151: total_norm: 22.640654 [I 2022-10-10 03:16:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 1710/2304 [I 2022-10-10 03:16:15] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:23 [I 2022-10-10 03:16:15] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00304 [I 2022-10-10 03:16:15] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-10 03:16:15] ...optimizers/trijoint.py.151: total_norm: 18.130432 [I 2022-10-10 03:16:23] ...trap/engines/engine.py.224: train: epoch 64 | batch 1720/2304 [I 2022-10-10 03:16:23] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:14 [I 2022-10-10 03:16:23] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00313 [I 2022-10-10 03:16:23] ...trap/engines/engine.py.230: loss: 0.25005 [I 2022-10-10 03:16:23] ...optimizers/trijoint.py.151: total_norm: 15.721828 [I 2022-10-10 03:16:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 1730/2304 [I 2022-10-10 03:16:32] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:04 [I 2022-10-10 03:16:32] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00310 [I 2022-10-10 03:16:32] ...trap/engines/engine.py.230: loss: 0.14849 [I 2022-10-10 03:16:32] ...optimizers/trijoint.py.151: total_norm: 14.292843 [I 2022-10-10 03:16:40] ...trap/engines/engine.py.224: train: epoch 64 | batch 1740/2304 [I 2022-10-10 03:16:40] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:01 [I 2022-10-10 03:16:40] ...trap/engines/engine.py.229: process: 0.85986 | load: 0.00311 [I 2022-10-10 03:16:40] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-10 03:16:40] ...optimizers/trijoint.py.151: total_norm: 10.865958 [I 2022-10-10 03:16:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 1750/2304 [I 2022-10-10 03:16:49] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:51 [I 2022-10-10 03:16:49] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00305 [I 2022-10-10 03:16:49] ...trap/engines/engine.py.230: loss: 0.14493 [I 2022-10-10 03:16:49] ...optimizers/trijoint.py.151: total_norm: 11.495204 [I 2022-10-10 03:16:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 1760/2304 [I 2022-10-10 03:16:57] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:38 [I 2022-10-10 03:16:57] ...trap/engines/engine.py.229: process: 0.83166 | load: 0.00294 [I 2022-10-10 03:16:57] ...trap/engines/engine.py.230: loss: 0.15900 [I 2022-10-10 03:16:57] ...optimizers/trijoint.py.151: total_norm: 17.569275 [I 2022-10-10 03:17:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 1770/2304 [I 2022-10-10 03:17:06] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:33 [I 2022-10-10 03:17:06] ...trap/engines/engine.py.229: process: 0.83823 | load: 0.00302 [I 2022-10-10 03:17:06] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-10 03:17:06] ...optimizers/trijoint.py.151: total_norm: 6.447813 [I 2022-10-10 03:17:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 1780/2304 [I 2022-10-10 03:17:14] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:24 [I 2022-10-10 03:17:14] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00312 [I 2022-10-10 03:17:14] ...trap/engines/engine.py.230: loss: 0.12573 [I 2022-10-10 03:17:14] ...optimizers/trijoint.py.151: total_norm: 13.501027 [I 2022-10-10 03:17:23] ...trap/engines/engine.py.224: train: epoch 64 | batch 1790/2304 [I 2022-10-10 03:17:23] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:18 [I 2022-10-10 03:17:23] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00302 [I 2022-10-10 03:17:23] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-10 03:17:23] ...optimizers/trijoint.py.151: total_norm: 8.529794 [I 2022-10-10 03:17:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 1800/2304 [I 2022-10-10 03:17:31] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:07 [I 2022-10-10 03:17:31] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00309 [I 2022-10-10 03:17:31] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-10 03:17:31] ...optimizers/trijoint.py.151: total_norm: 29.403688 [I 2022-10-10 03:17:40] ...trap/engines/engine.py.224: train: epoch 64 | batch 1810/2304 [I 2022-10-10 03:17:40] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:06:59 [I 2022-10-10 03:17:40] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00312 [I 2022-10-10 03:17:40] ...trap/engines/engine.py.230: loss: 0.09492 [I 2022-10-10 03:17:40] ...optimizers/trijoint.py.151: total_norm: 14.521563 [I 2022-10-10 03:17:48] ...trap/engines/engine.py.224: train: epoch 64 | batch 1820/2304 [I 2022-10-10 03:17:48] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:49 [I 2022-10-10 03:17:48] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00302 [I 2022-10-10 03:17:48] ...trap/engines/engine.py.230: loss: 0.09433 [I 2022-10-10 03:17:48] ...optimizers/trijoint.py.151: total_norm: 7.631726 [I 2022-10-10 03:17:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 1830/2304 [I 2022-10-10 03:17:57] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:06:44 [I 2022-10-10 03:17:57] ...trap/engines/engine.py.229: process: 0.84580 | load: 0.00318 [I 2022-10-10 03:17:57] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-10 03:17:57] ...optimizers/trijoint.py.151: total_norm: 11.191237 [I 2022-10-10 03:18:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 1840/2304 [I 2022-10-10 03:18:05] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:34 [I 2022-10-10 03:18:05] ...trap/engines/engine.py.229: process: 0.85904 | load: 0.00312 [I 2022-10-10 03:18:05] ...trap/engines/engine.py.230: loss: 0.16351 [I 2022-10-10 03:18:05] ...optimizers/trijoint.py.151: total_norm: 31.912760 [I 2022-10-10 03:18:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 1850/2304 [I 2022-10-10 03:18:14] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:23 [I 2022-10-10 03:18:14] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00307 [I 2022-10-10 03:18:14] ...trap/engines/engine.py.230: loss: 0.16733 [I 2022-10-10 03:18:14] ...optimizers/trijoint.py.151: total_norm: 11.455492 [I 2022-10-10 03:18:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 1860/2304 [I 2022-10-10 03:18:22] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:15 [I 2022-10-10 03:18:22] ...trap/engines/engine.py.229: process: 0.84154 | load: 0.00307 [I 2022-10-10 03:18:22] ...trap/engines/engine.py.230: loss: 0.14559 [I 2022-10-10 03:18:22] ...optimizers/trijoint.py.151: total_norm: 23.273003 [I 2022-10-10 03:18:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 1870/2304 [I 2022-10-10 03:18:31] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:07 [I 2022-10-10 03:18:31] ...trap/engines/engine.py.229: process: 0.84235 | load: 0.00312 [I 2022-10-10 03:18:31] ...trap/engines/engine.py.230: loss: 0.12161 [I 2022-10-10 03:18:31] ...optimizers/trijoint.py.151: total_norm: 15.158701 [I 2022-10-10 03:18:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 1880/2304 [I 2022-10-10 03:18:39] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:05:58 [I 2022-10-10 03:18:39] ...trap/engines/engine.py.229: process: 0.86381 | load: 0.00309 [I 2022-10-10 03:18:39] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-10 03:18:39] ...optimizers/trijoint.py.151: total_norm: 9.151104 [I 2022-10-10 03:18:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 1890/2304 [I 2022-10-10 03:18:47] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:05:51 [I 2022-10-10 03:18:47] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00315 [I 2022-10-10 03:18:47] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-10 03:18:47] ...optimizers/trijoint.py.151: total_norm: 7.894257 [I 2022-10-10 03:18:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 1900/2304 [I 2022-10-10 03:18:56] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:43 [I 2022-10-10 03:18:56] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00315 [I 2022-10-10 03:18:56] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-10 03:18:56] ...optimizers/trijoint.py.151: total_norm: 13.858337 [I 2022-10-10 03:19:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 1910/2304 [I 2022-10-10 03:19:04] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:34 [I 2022-10-10 03:19:04] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00309 [I 2022-10-10 03:19:04] ...trap/engines/engine.py.230: loss: 0.09502 [I 2022-10-10 03:19:04] ...optimizers/trijoint.py.151: total_norm: 7.978694 [I 2022-10-10 03:19:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 1920/2304 [I 2022-10-10 03:19:13] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:26 [I 2022-10-10 03:19:13] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00313 [I 2022-10-10 03:19:13] ...trap/engines/engine.py.230: loss: 0.12279 [I 2022-10-10 03:19:13] ...optimizers/trijoint.py.151: total_norm: 21.536581 [I 2022-10-10 03:19:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 1930/2304 [I 2022-10-10 03:19:22] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:18 [I 2022-10-10 03:19:22] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00311 [I 2022-10-10 03:19:22] ...trap/engines/engine.py.230: loss: 0.18412 [I 2022-10-10 03:19:22] ...optimizers/trijoint.py.151: total_norm: 16.446629 [I 2022-10-10 03:19:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 1940/2304 [I 2022-10-10 03:19:30] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:11 [I 2022-10-10 03:19:30] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00310 [I 2022-10-10 03:19:30] ...trap/engines/engine.py.230: loss: 0.10621 [I 2022-10-10 03:19:30] ...optimizers/trijoint.py.151: total_norm: 13.646342 [I 2022-10-10 03:19:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 1950/2304 [I 2022-10-10 03:19:39] ...trap/engines/engine.py.228: elapsed: 0:27:45 | left: 0:05:01 [I 2022-10-10 03:19:39] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00310 [I 2022-10-10 03:19:39] ...trap/engines/engine.py.230: loss: 0.17347 [I 2022-10-10 03:19:39] ...optimizers/trijoint.py.151: total_norm: 15.581815 [I 2022-10-10 03:19:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 1960/2304 [I 2022-10-10 03:19:47] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:04:56 [I 2022-10-10 03:19:47] ...trap/engines/engine.py.229: process: 0.87456 | load: 0.00315 [I 2022-10-10 03:19:47] ...trap/engines/engine.py.230: loss: 0.12181 [I 2022-10-10 03:19:47] ...optimizers/trijoint.py.151: total_norm: 11.274491 [I 2022-10-10 03:19:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 1970/2304 [I 2022-10-10 03:19:56] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:04:45 [I 2022-10-10 03:19:56] ...trap/engines/engine.py.229: process: 0.86138 | load: 0.00312 [I 2022-10-10 03:19:56] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-10 03:19:56] ...optimizers/trijoint.py.151: total_norm: 10.304467 [I 2022-10-10 03:20:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 1980/2304 [I 2022-10-10 03:20:04] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:36 [I 2022-10-10 03:20:04] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00313 [I 2022-10-10 03:20:04] ...trap/engines/engine.py.230: loss: 0.20216 [I 2022-10-10 03:20:04] ...optimizers/trijoint.py.151: total_norm: 14.456282 [I 2022-10-10 03:20:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 1990/2304 [I 2022-10-10 03:20:13] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:27 [I 2022-10-10 03:20:13] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00313 [I 2022-10-10 03:20:13] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-10 03:20:13] ...optimizers/trijoint.py.151: total_norm: 13.479063 [I 2022-10-10 03:20:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 2000/2304 [I 2022-10-10 03:20:22] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:20 [I 2022-10-10 03:20:22] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00315 [I 2022-10-10 03:20:22] ...trap/engines/engine.py.230: loss: 0.15039 [I 2022-10-10 03:20:22] ...optimizers/trijoint.py.151: total_norm: 16.685427 [I 2022-10-10 03:20:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 2010/2304 [I 2022-10-10 03:20:30] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:10 [I 2022-10-10 03:20:30] ...trap/engines/engine.py.229: process: 0.85948 | load: 0.00313 [I 2022-10-10 03:20:30] ...trap/engines/engine.py.230: loss: 0.10459 [I 2022-10-10 03:20:30] ...optimizers/trijoint.py.151: total_norm: 10.041594 [I 2022-10-10 03:20:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 2020/2304 [I 2022-10-10 03:20:39] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:03 [I 2022-10-10 03:20:39] ...trap/engines/engine.py.229: process: 0.86691 | load: 0.00311 [I 2022-10-10 03:20:39] ...trap/engines/engine.py.230: loss: 0.20154 [I 2022-10-10 03:20:39] ...optimizers/trijoint.py.151: total_norm: 12.558640 [I 2022-10-10 03:20:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 2030/2304 [I 2022-10-10 03:20:47] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:03:55 [I 2022-10-10 03:20:47] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00313 [I 2022-10-10 03:20:47] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-10 03:20:47] ...optimizers/trijoint.py.151: total_norm: 22.384554 [I 2022-10-10 03:20:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 2040/2304 [I 2022-10-10 03:20:56] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:03:47 [I 2022-10-10 03:20:56] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00328 [I 2022-10-10 03:20:56] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-10 03:20:56] ...optimizers/trijoint.py.151: total_norm: 25.388800 [I 2022-10-10 03:21:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 2050/2304 [I 2022-10-10 03:21:05] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:37 [I 2022-10-10 03:21:05] ...trap/engines/engine.py.229: process: 0.85419 | load: 0.00313 [I 2022-10-10 03:21:05] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-10 03:21:05] ...optimizers/trijoint.py.151: total_norm: 17.507895 [I 2022-10-10 03:21:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 2060/2304 [I 2022-10-10 03:21:13] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:29 [I 2022-10-10 03:21:13] ...trap/engines/engine.py.229: process: 0.85946 | load: 0.00312 [I 2022-10-10 03:21:13] ...trap/engines/engine.py.230: loss: 0.19382 [I 2022-10-10 03:21:13] ...optimizers/trijoint.py.151: total_norm: 14.857357 [I 2022-10-10 03:21:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 2070/2304 [I 2022-10-10 03:21:22] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:16 [I 2022-10-10 03:21:22] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00300 [I 2022-10-10 03:21:22] ...trap/engines/engine.py.230: loss: 0.11329 [I 2022-10-10 03:21:22] ...optimizers/trijoint.py.151: total_norm: 17.930634 [I 2022-10-10 03:21:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 2080/2304 [I 2022-10-10 03:21:30] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:03:08 [I 2022-10-10 03:21:30] ...trap/engines/engine.py.229: process: 0.84251 | load: 0.00303 [I 2022-10-10 03:21:30] ...trap/engines/engine.py.230: loss: 0.12839 [I 2022-10-10 03:21:30] ...optimizers/trijoint.py.151: total_norm: 50.671055 [I 2022-10-10 03:21:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 2090/2304 [I 2022-10-10 03:21:39] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:02 [I 2022-10-10 03:21:39] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00299 [I 2022-10-10 03:21:39] ...trap/engines/engine.py.230: loss: 0.18172 [I 2022-10-10 03:21:39] ...optimizers/trijoint.py.151: total_norm: 14.588665 [I 2022-10-10 03:21:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 2100/2304 [I 2022-10-10 03:21:47] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:02:52 [I 2022-10-10 03:21:47] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00331 [I 2022-10-10 03:21:47] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-10 03:21:47] ...optimizers/trijoint.py.151: total_norm: 13.409413 [I 2022-10-10 03:21:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 2110/2304 [I 2022-10-10 03:21:56] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:44 [I 2022-10-10 03:21:56] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00296 [I 2022-10-10 03:21:56] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-10 03:21:56] ...optimizers/trijoint.py.151: total_norm: 11.540449 [I 2022-10-10 03:22:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 2120/2304 [I 2022-10-10 03:22:04] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:35 [I 2022-10-10 03:22:04] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00313 [I 2022-10-10 03:22:04] ...trap/engines/engine.py.230: loss: 0.10964 [I 2022-10-10 03:22:04] ...optimizers/trijoint.py.151: total_norm: 9.340765 [I 2022-10-10 03:22:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 2130/2304 [I 2022-10-10 03:22:13] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:27 [I 2022-10-10 03:22:13] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00300 [I 2022-10-10 03:22:13] ...trap/engines/engine.py.230: loss: 0.19340 [I 2022-10-10 03:22:13] ...optimizers/trijoint.py.151: total_norm: 23.853069 [I 2022-10-10 03:22:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 2140/2304 [I 2022-10-10 03:22:21] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:19 [I 2022-10-10 03:22:21] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00304 [I 2022-10-10 03:22:21] ...trap/engines/engine.py.230: loss: 0.21552 [I 2022-10-10 03:22:21] ...optimizers/trijoint.py.151: total_norm: 9.970098 [I 2022-10-10 03:22:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 2150/2304 [I 2022-10-10 03:22:30] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:11 [I 2022-10-10 03:22:30] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00304 [I 2022-10-10 03:22:30] ...trap/engines/engine.py.230: loss: 0.15495 [I 2022-10-10 03:22:30] ...optimizers/trijoint.py.151: total_norm: 13.885458 [I 2022-10-10 03:22:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 2160/2304 [I 2022-10-10 03:22:38] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:01 [I 2022-10-10 03:22:38] ...trap/engines/engine.py.229: process: 0.85156 | load: 0.00305 [I 2022-10-10 03:22:38] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-10 03:22:38] ...optimizers/trijoint.py.151: total_norm: 8.659241 [I 2022-10-10 03:22:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 2170/2304 [I 2022-10-10 03:22:47] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:01:53 [I 2022-10-10 03:22:47] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00303 [I 2022-10-10 03:22:47] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-10 03:22:47] ...optimizers/trijoint.py.151: total_norm: 10.494541 [I 2022-10-10 03:22:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 2180/2304 [I 2022-10-10 03:22:55] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:44 [I 2022-10-10 03:22:55] ...trap/engines/engine.py.229: process: 0.84807 | load: 0.00314 [I 2022-10-10 03:22:55] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-10 03:22:55] ...optimizers/trijoint.py.151: total_norm: 37.807999 [I 2022-10-10 03:23:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 2190/2304 [I 2022-10-10 03:23:04] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:01:37 [I 2022-10-10 03:23:04] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00314 [I 2022-10-10 03:23:04] ...trap/engines/engine.py.230: loss: 0.14500 [I 2022-10-10 03:23:04] ...optimizers/trijoint.py.151: total_norm: 15.407541 [I 2022-10-10 03:23:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 2200/2304 [I 2022-10-10 03:23:12] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:29 [I 2022-10-10 03:23:12] ...trap/engines/engine.py.229: process: 0.87094 | load: 0.00316 [I 2022-10-10 03:23:12] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-10 03:23:12] ...optimizers/trijoint.py.151: total_norm: 18.752081 [I 2022-10-10 03:23:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 2210/2304 [I 2022-10-10 03:23:21] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:20 [I 2022-10-10 03:23:21] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00313 [I 2022-10-10 03:23:21] ...trap/engines/engine.py.230: loss: 0.12086 [I 2022-10-10 03:23:21] ...optimizers/trijoint.py.151: total_norm: 12.079659 [I 2022-10-10 03:23:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 2220/2304 [I 2022-10-10 03:23:29] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:11 [I 2022-10-10 03:23:29] ...trap/engines/engine.py.229: process: 0.86059 | load: 0.00315 [I 2022-10-10 03:23:29] ...trap/engines/engine.py.230: loss: 0.17713 [I 2022-10-10 03:23:29] ...optimizers/trijoint.py.151: total_norm: 22.110289 [I 2022-10-10 03:23:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 2230/2304 [I 2022-10-10 03:23:38] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:03 [I 2022-10-10 03:23:38] ...trap/engines/engine.py.229: process: 0.85743 | load: 0.00311 [I 2022-10-10 03:23:38] ...trap/engines/engine.py.230: loss: 0.16419 [I 2022-10-10 03:23:38] ...optimizers/trijoint.py.151: total_norm: 16.532669 [I 2022-10-10 03:23:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 2240/2304 [I 2022-10-10 03:23:47] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:00:54 [I 2022-10-10 03:23:47] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00313 [I 2022-10-10 03:23:47] ...trap/engines/engine.py.230: loss: 0.20190 [I 2022-10-10 03:23:47] ...optimizers/trijoint.py.151: total_norm: 9.230955 [I 2022-10-10 03:23:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 2250/2304 [I 2022-10-10 03:23:55] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:00:46 [I 2022-10-10 03:23:55] ...trap/engines/engine.py.229: process: 0.86533 | load: 0.00317 [I 2022-10-10 03:23:55] ...trap/engines/engine.py.230: loss: 0.15897 [I 2022-10-10 03:23:55] ...optimizers/trijoint.py.151: total_norm: 15.066470 [I 2022-10-10 03:24:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 2260/2304 [I 2022-10-10 03:24:04] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:00:37 [I 2022-10-10 03:24:04] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00312 [I 2022-10-10 03:24:04] ...trap/engines/engine.py.230: loss: 0.17307 [I 2022-10-10 03:24:04] ...optimizers/trijoint.py.151: total_norm: 9.669933 [I 2022-10-10 03:24:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 2270/2304 [I 2022-10-10 03:24:12] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:29 [I 2022-10-10 03:24:12] ...trap/engines/engine.py.229: process: 0.87051 | load: 0.00312 [I 2022-10-10 03:24:12] ...trap/engines/engine.py.230: loss: 0.14300 [I 2022-10-10 03:24:12] ...optimizers/trijoint.py.151: total_norm: 15.532531 [I 2022-10-10 03:24:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 2280/2304 [I 2022-10-10 03:24:21] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:20 [I 2022-10-10 03:24:21] ...trap/engines/engine.py.229: process: 0.85852 | load: 0.00314 [I 2022-10-10 03:24:21] ...trap/engines/engine.py.230: loss: 0.24175 [I 2022-10-10 03:24:21] ...optimizers/trijoint.py.151: total_norm: 24.052332 [I 2022-10-10 03:24:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 2290/2304 [I 2022-10-10 03:24:29] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:11 [I 2022-10-10 03:24:29] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00311 [I 2022-10-10 03:24:29] ...trap/engines/engine.py.230: loss: 0.14543 [I 2022-10-10 03:24:29] ...optimizers/trijoint.py.151: total_norm: 31.548754 [I 2022-10-10 03:24:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 2300/2304 [I 2022-10-10 03:24:38] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:03 [I 2022-10-10 03:24:38] ...trap/engines/engine.py.229: process: 0.83083 | load: 0.00019 [I 2022-10-10 03:24:38] ...trap/engines/engine.py.230: loss: 0.17638 [I 2022-10-10 03:24:38] ...optimizers/trijoint.py.151: total_norm: 20.685001 [I 2022-10-10 03:24:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 2304/2304 [I 2022-10-10 03:24:41] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:00 [I 2022-10-10 03:24:41] ...trap/engines/engine.py.229: process: 0.83748 | load: 0.00019 [I 2022-10-10 03:24:41] ...trap/engines/engine.py.230: loss: 0.18829 [I 2022-10-10 03:24:41] ...optimizers/trijoint.py.151: total_norm: 25.799994 [S 2022-10-10 03:24:42] ...trap/engines/engine.py.129: train_epoch.epoch: 64 [S 2022-10-10 03:24:42] ...trap/engines/engine.py.129: train_epoch.loss: 0.14397753374630873 [I 2022-10-10 03:24:42] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 03:24:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 64 [I 2022-10-10 03:24:43] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 03:24:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 03:24:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 03:24:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 03:24:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 03:24:48] ...trap/engines/engine.py.321: eval: epoch 64 | batch 0/510 [I 2022-10-10 03:24:48] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:33 [I 2022-10-10 03:24:48] ...trap/engines/engine.py.326: process: 4.77140 | load: 4.03094 [I 2022-10-10 03:24:54] ...trap/engines/engine.py.321: eval: epoch 64 | batch 10/510 [I 2022-10-10 03:24:54] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:29 [I 2022-10-10 03:24:54] ...trap/engines/engine.py.326: process: 0.59524 | load: 0.00021 [I 2022-10-10 03:24:59] ...trap/engines/engine.py.321: eval: epoch 64 | batch 20/510 [I 2022-10-10 03:24:59] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:40 [I 2022-10-10 03:24:59] ...trap/engines/engine.py.326: process: 0.57170 | load: 0.00019 [I 2022-10-10 03:25:05] ...trap/engines/engine.py.321: eval: epoch 64 | batch 30/510 [I 2022-10-10 03:25:05] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:44 [I 2022-10-10 03:25:05] ...trap/engines/engine.py.326: process: 0.48001 | load: 0.00020 [I 2022-10-10 03:25:10] ...trap/engines/engine.py.321: eval: epoch 64 | batch 40/510 [I 2022-10-10 03:25:10] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:04 [I 2022-10-10 03:25:10] ...trap/engines/engine.py.326: process: 0.57114 | load: 0.00020 [I 2022-10-10 03:25:15] ...trap/engines/engine.py.321: eval: epoch 64 | batch 50/510 [I 2022-10-10 03:25:15] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:22 [I 2022-10-10 03:25:15] ...trap/engines/engine.py.326: process: 0.56824 | load: 0.00023 [I 2022-10-10 03:25:21] ...trap/engines/engine.py.321: eval: epoch 64 | batch 60/510 [I 2022-10-10 03:25:21] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:04:07 [I 2022-10-10 03:25:21] ...trap/engines/engine.py.326: process: 0.56596 | load: 0.00020 [I 2022-10-10 03:25:27] ...trap/engines/engine.py.321: eval: epoch 64 | batch 70/510 [I 2022-10-10 03:25:27] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:04:20 [I 2022-10-10 03:25:27] ...trap/engines/engine.py.326: process: 0.58087 | load: 0.00020 [I 2022-10-10 03:25:32] ...trap/engines/engine.py.321: eval: epoch 64 | batch 80/510 [I 2022-10-10 03:25:32] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:39 [I 2022-10-10 03:25:32] ...trap/engines/engine.py.326: process: 0.47100 | load: 0.00020 [I 2022-10-10 03:25:36] ...trap/engines/engine.py.321: eval: epoch 64 | batch 90/510 [I 2022-10-10 03:25:36] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:10 [I 2022-10-10 03:25:36] ...trap/engines/engine.py.326: process: 0.45562 | load: 0.00020 [I 2022-10-10 03:25:41] ...trap/engines/engine.py.321: eval: epoch 64 | batch 100/510 [I 2022-10-10 03:25:41] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:54 [I 2022-10-10 03:25:41] ...trap/engines/engine.py.326: process: 0.30859 | load: 0.00020 [I 2022-10-10 03:25:44] ...trap/engines/engine.py.321: eval: epoch 64 | batch 110/510 [I 2022-10-10 03:25:44] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:12 [I 2022-10-10 03:25:44] ...trap/engines/engine.py.326: process: 0.32103 | load: 0.00021 [I 2022-10-10 03:25:47] ...trap/engines/engine.py.321: eval: epoch 64 | batch 120/510 [I 2022-10-10 03:25:47] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-10 03:25:47] ...trap/engines/engine.py.326: process: 0.31627 | load: 0.00019 [I 2022-10-10 03:25:50] ...trap/engines/engine.py.321: eval: epoch 64 | batch 130/510 [I 2022-10-10 03:25:50] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:01 [I 2022-10-10 03:25:50] ...trap/engines/engine.py.326: process: 0.31953 | load: 0.00020 [I 2022-10-10 03:25:54] ...trap/engines/engine.py.321: eval: epoch 64 | batch 140/510 [I 2022-10-10 03:25:54] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:57 [I 2022-10-10 03:25:54] ...trap/engines/engine.py.326: process: 0.31622 | load: 0.00022 [I 2022-10-10 03:25:57] ...trap/engines/engine.py.321: eval: epoch 64 | batch 150/510 [I 2022-10-10 03:25:57] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:54 [I 2022-10-10 03:25:57] ...trap/engines/engine.py.326: process: 0.31721 | load: 0.00021 [I 2022-10-10 03:26:00] ...trap/engines/engine.py.321: eval: epoch 64 | batch 160/510 [I 2022-10-10 03:26:00] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:51 [I 2022-10-10 03:26:00] ...trap/engines/engine.py.326: process: 0.31539 | load: 0.00021 [I 2022-10-10 03:26:03] ...trap/engines/engine.py.321: eval: epoch 64 | batch 170/510 [I 2022-10-10 03:26:03] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-10 03:26:03] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00019 [I 2022-10-10 03:26:06] ...trap/engines/engine.py.321: eval: epoch 64 | batch 180/510 [I 2022-10-10 03:26:06] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:44 [I 2022-10-10 03:26:06] ...trap/engines/engine.py.326: process: 0.31406 | load: 0.00021 [I 2022-10-10 03:26:09] ...trap/engines/engine.py.321: eval: epoch 64 | batch 190/510 [I 2022-10-10 03:26:10] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:42 [I 2022-10-10 03:26:10] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00021 [I 2022-10-10 03:26:13] ...trap/engines/engine.py.321: eval: epoch 64 | batch 200/510 [I 2022-10-10 03:26:13] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:39 [I 2022-10-10 03:26:13] ...trap/engines/engine.py.326: process: 0.32275 | load: 0.00021 [I 2022-10-10 03:26:16] ...trap/engines/engine.py.321: eval: epoch 64 | batch 210/510 [I 2022-10-10 03:26:16] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:36 [I 2022-10-10 03:26:16] ...trap/engines/engine.py.326: process: 0.32241 | load: 0.00021 [I 2022-10-10 03:26:19] ...trap/engines/engine.py.321: eval: epoch 64 | batch 220/510 [I 2022-10-10 03:26:19] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:32 [I 2022-10-10 03:26:19] ...trap/engines/engine.py.326: process: 0.31740 | load: 0.00021 [I 2022-10-10 03:26:22] ...trap/engines/engine.py.321: eval: epoch 64 | batch 230/510 [I 2022-10-10 03:26:22] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-10 03:26:22] ...trap/engines/engine.py.326: process: 0.32309 | load: 0.00021 [I 2022-10-10 03:26:26] ...trap/engines/engine.py.321: eval: epoch 64 | batch 240/510 [I 2022-10-10 03:26:26] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:26 [I 2022-10-10 03:26:26] ...trap/engines/engine.py.326: process: 0.31469 | load: 0.00021 [I 2022-10-10 03:26:29] ...trap/engines/engine.py.321: eval: epoch 64 | batch 250/510 [I 2022-10-10 03:26:29] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:24 [I 2022-10-10 03:26:29] ...trap/engines/engine.py.326: process: 0.32774 | load: 0.00021 [I 2022-10-10 03:26:32] ...trap/engines/engine.py.321: eval: epoch 64 | batch 260/510 [I 2022-10-10 03:26:32] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:20 [I 2022-10-10 03:26:32] ...trap/engines/engine.py.326: process: 0.32355 | load: 0.00020 [I 2022-10-10 03:26:35] ...trap/engines/engine.py.321: eval: epoch 64 | batch 270/510 [I 2022-10-10 03:26:35] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:17 [I 2022-10-10 03:26:35] ...trap/engines/engine.py.326: process: 0.32499 | load: 0.00021 [I 2022-10-10 03:26:38] ...trap/engines/engine.py.321: eval: epoch 64 | batch 280/510 [I 2022-10-10 03:26:38] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:14 [I 2022-10-10 03:26:38] ...trap/engines/engine.py.326: process: 0.32351 | load: 0.00021 [I 2022-10-10 03:26:42] ...trap/engines/engine.py.321: eval: epoch 64 | batch 290/510 [I 2022-10-10 03:26:42] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:10 [I 2022-10-10 03:26:42] ...trap/engines/engine.py.326: process: 0.32254 | load: 0.00021 [I 2022-10-10 03:26:45] ...trap/engines/engine.py.321: eval: epoch 64 | batch 300/510 [I 2022-10-10 03:26:45] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:07 [I 2022-10-10 03:26:45] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00027 [I 2022-10-10 03:26:48] ...trap/engines/engine.py.321: eval: epoch 64 | batch 310/510 [I 2022-10-10 03:26:48] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-10 03:26:48] ...trap/engines/engine.py.326: process: 0.31376 | load: 0.00020 [I 2022-10-10 03:26:51] ...trap/engines/engine.py.321: eval: epoch 64 | batch 320/510 [I 2022-10-10 03:26:51] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-10 03:26:51] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00021 [I 2022-10-10 03:26:54] ...trap/engines/engine.py.321: eval: epoch 64 | batch 330/510 [I 2022-10-10 03:26:54] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:57 [I 2022-10-10 03:26:54] ...trap/engines/engine.py.326: process: 0.32484 | load: 0.00022 [I 2022-10-10 03:26:58] ...trap/engines/engine.py.321: eval: epoch 64 | batch 340/510 [I 2022-10-10 03:26:58] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:54 [I 2022-10-10 03:26:58] ...trap/engines/engine.py.326: process: 0.31664 | load: 0.00021 [I 2022-10-10 03:27:01] ...trap/engines/engine.py.321: eval: epoch 64 | batch 350/510 [I 2022-10-10 03:27:01] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-10 03:27:01] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00022 [I 2022-10-10 03:27:04] ...trap/engines/engine.py.321: eval: epoch 64 | batch 360/510 [I 2022-10-10 03:27:04] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:48 [I 2022-10-10 03:27:04] ...trap/engines/engine.py.326: process: 0.32023 | load: 0.00022 [I 2022-10-10 03:27:07] ...trap/engines/engine.py.321: eval: epoch 64 | batch 370/510 [I 2022-10-10 03:27:07] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:44 [I 2022-10-10 03:27:07] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00022 [I 2022-10-10 03:27:11] ...trap/engines/engine.py.321: eval: epoch 64 | batch 380/510 [I 2022-10-10 03:27:11] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:42 [I 2022-10-10 03:27:11] ...trap/engines/engine.py.326: process: 0.32665 | load: 0.00021 [I 2022-10-10 03:27:14] ...trap/engines/engine.py.321: eval: epoch 64 | batch 390/510 [I 2022-10-10 03:27:14] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:38 [I 2022-10-10 03:27:14] ...trap/engines/engine.py.326: process: 0.32036 | load: 0.00021 [I 2022-10-10 03:27:17] ...trap/engines/engine.py.321: eval: epoch 64 | batch 400/510 [I 2022-10-10 03:27:17] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:35 [I 2022-10-10 03:27:17] ...trap/engines/engine.py.326: process: 0.32170 | load: 0.00021 [I 2022-10-10 03:27:20] ...trap/engines/engine.py.321: eval: epoch 64 | batch 410/510 [I 2022-10-10 03:27:20] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:32 [I 2022-10-10 03:27:20] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00021 [I 2022-10-10 03:27:24] ...trap/engines/engine.py.321: eval: epoch 64 | batch 420/510 [I 2022-10-10 03:27:24] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:29 [I 2022-10-10 03:27:24] ...trap/engines/engine.py.326: process: 0.33071 | load: 0.00021 [I 2022-10-10 03:27:27] ...trap/engines/engine.py.321: eval: epoch 64 | batch 430/510 [I 2022-10-10 03:27:27] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:25 [I 2022-10-10 03:27:27] ...trap/engines/engine.py.326: process: 0.32792 | load: 0.00021 [I 2022-10-10 03:27:30] ...trap/engines/engine.py.321: eval: epoch 64 | batch 440/510 [I 2022-10-10 03:27:30] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:22 [I 2022-10-10 03:27:30] ...trap/engines/engine.py.326: process: 0.31142 | load: 0.00020 [I 2022-10-10 03:27:33] ...trap/engines/engine.py.321: eval: epoch 64 | batch 450/510 [I 2022-10-10 03:27:33] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-10 03:27:33] ...trap/engines/engine.py.326: process: 0.31874 | load: 0.00021 [I 2022-10-10 03:27:36] ...trap/engines/engine.py.321: eval: epoch 64 | batch 460/510 [I 2022-10-10 03:27:36] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:15 [I 2022-10-10 03:27:36] ...trap/engines/engine.py.326: process: 0.31394 | load: 0.00019 [I 2022-10-10 03:27:40] ...trap/engines/engine.py.321: eval: epoch 64 | batch 470/510 [I 2022-10-10 03:27:40] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:12 [I 2022-10-10 03:27:40] ...trap/engines/engine.py.326: process: 0.31591 | load: 0.00020 [I 2022-10-10 03:27:43] ...trap/engines/engine.py.321: eval: epoch 64 | batch 480/510 [I 2022-10-10 03:27:43] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:09 [I 2022-10-10 03:27:43] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00020 [I 2022-10-10 03:27:46] ...trap/engines/engine.py.321: eval: epoch 64 | batch 490/510 [I 2022-10-10 03:27:46] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:06 [I 2022-10-10 03:27:46] ...trap/engines/engine.py.326: process: 0.32000 | load: 0.00019 [I 2022-10-10 03:27:49] ...trap/engines/engine.py.321: eval: epoch 64 | batch 500/510 [I 2022-10-10 03:27:49] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:03 [I 2022-10-10 03:27:49] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00021 [I 2022-10-10 03:27:52] ...trap/engines/engine.py.321: eval: epoch 64 | batch 510/510 [I 2022-10-10 03:27:52] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 03:27:52] ...trap/engines/engine.py.326: process: 0.31441 | load: 0.00016 [S 2022-10-10 03:27:53] ...trap/engines/engine.py.132: eval_epoch.epoch: 64 [I 2022-10-10 03:27:53] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.730900 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907500 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936200 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.722300 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906900 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936000 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010339 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009222 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011187 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007643 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011853 [S 2022-10-10 03:28:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010918 [I 2022-10-10 03:28:28] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 03:28:28] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 03:28:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 03:28:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 03:28:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 03:28:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 03:28:32] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 03:28:38] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 03:28:38] ...trap/engines/engine.py.163: Training model on trainset for epoch 65 [I 2022-10-10 03:28:38] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 03:28:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 0/2304 [I 2022-10-10 03:28:41] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:33 [I 2022-10-10 03:28:41] ...trap/engines/engine.py.229: process: 2.77514 | load: 1.88161 [I 2022-10-10 03:28:41] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-10 03:28:41] ...optimizers/trijoint.py.151: total_norm: 20.859524 [I 2022-10-10 03:28:49] ...trap/engines/engine.py.224: train: epoch 65 | batch 10/2304 [I 2022-10-10 03:28:49] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:11 [I 2022-10-10 03:28:49] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00498 [I 2022-10-10 03:28:49] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-10 03:28:49] ...optimizers/trijoint.py.151: total_norm: 18.266434 [I 2022-10-10 03:28:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 20/2304 [I 2022-10-10 03:28:58] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:55 [I 2022-10-10 03:28:58] ...trap/engines/engine.py.229: process: 0.83496 | load: 0.00605 [I 2022-10-10 03:28:58] ...trap/engines/engine.py.230: loss: 0.15608 [I 2022-10-10 03:28:58] ...optimizers/trijoint.py.151: total_norm: 11.295643 [I 2022-10-10 03:29:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 30/2304 [I 2022-10-10 03:29:06] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:18 [I 2022-10-10 03:29:06] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00472 [I 2022-10-10 03:29:06] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-10 03:29:06] ...optimizers/trijoint.py.151: total_norm: 23.030935 [I 2022-10-10 03:29:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 40/2304 [I 2022-10-10 03:29:15] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:55 [I 2022-10-10 03:29:15] ...trap/engines/engine.py.229: process: 0.82946 | load: 0.00352 [I 2022-10-10 03:29:15] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-10 03:29:15] ...optimizers/trijoint.py.151: total_norm: 17.409559 [I 2022-10-10 03:29:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 50/2304 [I 2022-10-10 03:29:23] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:50 [I 2022-10-10 03:29:23] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00456 [I 2022-10-10 03:29:23] ...trap/engines/engine.py.230: loss: 0.15991 [I 2022-10-10 03:29:23] ...optimizers/trijoint.py.151: total_norm: 19.609087 [I 2022-10-10 03:29:32] ...trap/engines/engine.py.224: train: epoch 65 | batch 60/2304 [I 2022-10-10 03:29:32] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:31 [I 2022-10-10 03:29:32] ...trap/engines/engine.py.229: process: 0.83356 | load: 0.00378 [I 2022-10-10 03:29:32] ...trap/engines/engine.py.230: loss: 0.16926 [I 2022-10-10 03:29:32] ...optimizers/trijoint.py.151: total_norm: 23.783340 [I 2022-10-10 03:29:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 70/2304 [I 2022-10-10 03:29:40] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:30 [I 2022-10-10 03:29:40] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00321 [I 2022-10-10 03:29:40] ...trap/engines/engine.py.230: loss: 0.23454 [I 2022-10-10 03:29:40] ...optimizers/trijoint.py.151: total_norm: 13.668095 [I 2022-10-10 03:29:49] ...trap/engines/engine.py.224: train: epoch 65 | batch 80/2304 [I 2022-10-10 03:29:49] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:38 [I 2022-10-10 03:29:49] ...trap/engines/engine.py.229: process: 0.86642 | load: 0.00404 [I 2022-10-10 03:29:49] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-10 03:29:49] ...optimizers/trijoint.py.151: total_norm: 22.834366 [I 2022-10-10 03:29:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 90/2304 [I 2022-10-10 03:29:57] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:23 [I 2022-10-10 03:29:57] ...trap/engines/engine.py.229: process: 0.83513 | load: 0.00376 [I 2022-10-10 03:29:57] ...trap/engines/engine.py.230: loss: 0.11088 [I 2022-10-10 03:29:57] ...optimizers/trijoint.py.151: total_norm: 8.168564 [I 2022-10-10 03:30:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 100/2304 [I 2022-10-10 03:30:06] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:27 [I 2022-10-10 03:30:06] ...trap/engines/engine.py.229: process: 0.86102 | load: 0.00350 [I 2022-10-10 03:30:06] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-10 03:30:06] ...optimizers/trijoint.py.151: total_norm: 17.881788 [I 2022-10-10 03:30:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 110/2304 [I 2022-10-10 03:30:14] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:06 [I 2022-10-10 03:30:14] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00302 [I 2022-10-10 03:30:14] ...trap/engines/engine.py.230: loss: 0.09603 [I 2022-10-10 03:30:14] ...optimizers/trijoint.py.151: total_norm: 9.135681 [I 2022-10-10 03:30:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 120/2304 [I 2022-10-10 03:30:23] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:48 [I 2022-10-10 03:30:23] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00312 [I 2022-10-10 03:30:23] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-10 03:30:23] ...optimizers/trijoint.py.151: total_norm: 8.079484 [I 2022-10-10 03:30:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 130/2304 [I 2022-10-10 03:30:31] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:31:02 [I 2022-10-10 03:30:31] ...trap/engines/engine.py.229: process: 0.86267 | load: 0.00309 [I 2022-10-10 03:30:31] ...trap/engines/engine.py.230: loss: 0.12112 [I 2022-10-10 03:30:31] ...optimizers/trijoint.py.151: total_norm: 19.019403 [I 2022-10-10 03:30:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 140/2304 [I 2022-10-10 03:30:40] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:31 [I 2022-10-10 03:30:40] ...trap/engines/engine.py.229: process: 0.83631 | load: 0.00327 [I 2022-10-10 03:30:40] ...trap/engines/engine.py.230: loss: 0.20599 [I 2022-10-10 03:30:40] ...optimizers/trijoint.py.151: total_norm: 18.364208 [I 2022-10-10 03:30:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 150/2304 [I 2022-10-10 03:30:48] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:29 [I 2022-10-10 03:30:48] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00310 [I 2022-10-10 03:30:48] ...trap/engines/engine.py.230: loss: 0.09875 [I 2022-10-10 03:30:48] ...optimizers/trijoint.py.151: total_norm: 6.672348 [I 2022-10-10 03:30:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 160/2304 [I 2022-10-10 03:30:57] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:38 [I 2022-10-10 03:30:57] ...trap/engines/engine.py.229: process: 0.86139 | load: 0.00310 [I 2022-10-10 03:30:57] ...trap/engines/engine.py.230: loss: 0.11293 [I 2022-10-10 03:30:57] ...optimizers/trijoint.py.151: total_norm: 9.985484 [I 2022-10-10 03:31:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 170/2304 [I 2022-10-10 03:31:06] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:22 [I 2022-10-10 03:31:06] ...trap/engines/engine.py.229: process: 0.85355 | load: 0.00331 [I 2022-10-10 03:31:06] ...trap/engines/engine.py.230: loss: 0.15472 [I 2022-10-10 03:31:06] ...optimizers/trijoint.py.151: total_norm: 9.403101 [I 2022-10-10 03:31:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 180/2304 [I 2022-10-10 03:31:14] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:20 [I 2022-10-10 03:31:14] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00320 [I 2022-10-10 03:31:14] ...trap/engines/engine.py.230: loss: 0.14252 [I 2022-10-10 03:31:14] ...optimizers/trijoint.py.151: total_norm: 13.759802 [I 2022-10-10 03:31:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 190/2304 [I 2022-10-10 03:31:23] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:53 [I 2022-10-10 03:31:23] ...trap/engines/engine.py.229: process: 0.83420 | load: 0.00318 [I 2022-10-10 03:31:23] ...trap/engines/engine.py.230: loss: 0.16136 [I 2022-10-10 03:31:23] ...optimizers/trijoint.py.151: total_norm: 12.614141 [I 2022-10-10 03:31:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 200/2304 [I 2022-10-10 03:31:31] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:30:03 [I 2022-10-10 03:31:31] ...trap/engines/engine.py.229: process: 0.86524 | load: 0.00322 [I 2022-10-10 03:31:31] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-10 03:31:31] ...optimizers/trijoint.py.151: total_norm: 20.522728 [I 2022-10-10 03:31:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 210/2304 [I 2022-10-10 03:31:40] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:51 [I 2022-10-10 03:31:40] ...trap/engines/engine.py.229: process: 0.87242 | load: 0.00324 [I 2022-10-10 03:31:40] ...trap/engines/engine.py.230: loss: 0.13887 [I 2022-10-10 03:31:40] ...optimizers/trijoint.py.151: total_norm: 63.645245 [I 2022-10-10 03:31:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 220/2304 [I 2022-10-10 03:31:48] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:37 [I 2022-10-10 03:31:48] ...trap/engines/engine.py.229: process: 0.85030 | load: 0.00326 [I 2022-10-10 03:31:48] ...trap/engines/engine.py.230: loss: 0.21604 [I 2022-10-10 03:31:48] ...optimizers/trijoint.py.151: total_norm: 23.031269 [I 2022-10-10 03:31:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 230/2304 [I 2022-10-10 03:31:57] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:39 [I 2022-10-10 03:31:57] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00314 [I 2022-10-10 03:31:57] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-10 03:31:57] ...optimizers/trijoint.py.151: total_norm: 12.372098 [I 2022-10-10 03:32:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 240/2304 [I 2022-10-10 03:32:06] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:14 [I 2022-10-10 03:32:06] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00295 [I 2022-10-10 03:32:06] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-10 03:32:06] ...optimizers/trijoint.py.151: total_norm: 13.433724 [I 2022-10-10 03:32:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 250/2304 [I 2022-10-10 03:32:14] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:23 [I 2022-10-10 03:32:14] ...trap/engines/engine.py.229: process: 0.87165 | load: 0.00320 [I 2022-10-10 03:32:14] ...trap/engines/engine.py.230: loss: 0.14436 [I 2022-10-10 03:32:14] ...optimizers/trijoint.py.151: total_norm: 20.743803 [I 2022-10-10 03:32:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 260/2304 [I 2022-10-10 03:32:23] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:10 [I 2022-10-10 03:32:23] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00326 [I 2022-10-10 03:32:23] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-10 03:32:23] ...optimizers/trijoint.py.151: total_norm: 31.651060 [I 2022-10-10 03:32:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 270/2304 [I 2022-10-10 03:32:31] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:55 [I 2022-10-10 03:32:31] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00300 [I 2022-10-10 03:32:31] ...trap/engines/engine.py.230: loss: 0.13924 [I 2022-10-10 03:32:31] ...optimizers/trijoint.py.151: total_norm: 22.459171 [I 2022-10-10 03:32:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 280/2304 [I 2022-10-10 03:32:40] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:53 [I 2022-10-10 03:32:40] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00332 [I 2022-10-10 03:32:40] ...trap/engines/engine.py.230: loss: 0.18807 [I 2022-10-10 03:32:40] ...optimizers/trijoint.py.151: total_norm: 34.963982 [I 2022-10-10 03:32:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 290/2304 [I 2022-10-10 03:32:48] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:41 [I 2022-10-10 03:32:48] ...trap/engines/engine.py.229: process: 0.87283 | load: 0.00323 [I 2022-10-10 03:32:48] ...trap/engines/engine.py.230: loss: 0.09480 [I 2022-10-10 03:32:48] ...optimizers/trijoint.py.151: total_norm: 11.400373 [I 2022-10-10 03:32:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 300/2304 [I 2022-10-10 03:32:57] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:29 [I 2022-10-10 03:32:57] ...trap/engines/engine.py.229: process: 0.85402 | load: 0.00311 [I 2022-10-10 03:32:57] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-10 03:32:57] ...optimizers/trijoint.py.151: total_norm: 11.214746 [I 2022-10-10 03:33:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 310/2304 [I 2022-10-10 03:33:05] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:24 [I 2022-10-10 03:33:05] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00304 [I 2022-10-10 03:33:05] ...trap/engines/engine.py.230: loss: 0.13926 [I 2022-10-10 03:33:05] ...optimizers/trijoint.py.151: total_norm: 27.754438 [I 2022-10-10 03:33:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 320/2304 [I 2022-10-10 03:33:14] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:06 [I 2022-10-10 03:33:14] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00310 [I 2022-10-10 03:33:14] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-10 03:33:14] ...optimizers/trijoint.py.151: total_norm: 13.200473 [I 2022-10-10 03:33:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 330/2304 [I 2022-10-10 03:33:23] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:13 [I 2022-10-10 03:33:23] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00319 [I 2022-10-10 03:33:23] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-10 03:33:23] ...optimizers/trijoint.py.151: total_norm: 28.433651 [I 2022-10-10 03:33:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 340/2304 [I 2022-10-10 03:33:31] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:57 [I 2022-10-10 03:33:31] ...trap/engines/engine.py.229: process: 0.84649 | load: 0.00334 [I 2022-10-10 03:33:31] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-10 03:33:31] ...optimizers/trijoint.py.151: total_norm: 9.101843 [I 2022-10-10 03:33:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 350/2304 [I 2022-10-10 03:33:40] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:54 [I 2022-10-10 03:33:40] ...trap/engines/engine.py.229: process: 0.85872 | load: 0.00312 [I 2022-10-10 03:33:40] ...trap/engines/engine.py.230: loss: 0.20271 [I 2022-10-10 03:33:40] ...optimizers/trijoint.py.151: total_norm: 27.839891 [I 2022-10-10 03:33:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 360/2304 [I 2022-10-10 03:33:48] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:31 [I 2022-10-10 03:33:48] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00343 [I 2022-10-10 03:33:48] ...trap/engines/engine.py.230: loss: 0.16445 [I 2022-10-10 03:33:48] ...optimizers/trijoint.py.151: total_norm: 26.553331 [I 2022-10-10 03:33:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 370/2304 [I 2022-10-10 03:33:57] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:37 [I 2022-10-10 03:33:57] ...trap/engines/engine.py.229: process: 0.87231 | load: 0.00312 [I 2022-10-10 03:33:57] ...trap/engines/engine.py.230: loss: 0.19742 [I 2022-10-10 03:33:57] ...optimizers/trijoint.py.151: total_norm: 9.687760 [I 2022-10-10 03:34:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 380/2304 [I 2022-10-10 03:34:05] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:17 [I 2022-10-10 03:34:05] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00310 [I 2022-10-10 03:34:05] ...trap/engines/engine.py.230: loss: 0.16870 [I 2022-10-10 03:34:05] ...optimizers/trijoint.py.151: total_norm: 16.960880 [I 2022-10-10 03:34:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 390/2304 [I 2022-10-10 03:34:14] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:26:59 [I 2022-10-10 03:34:14] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00300 [I 2022-10-10 03:34:14] ...trap/engines/engine.py.230: loss: 0.17915 [I 2022-10-10 03:34:14] ...optimizers/trijoint.py.151: total_norm: 15.662726 [I 2022-10-10 03:34:22] ...trap/engines/engine.py.224: train: epoch 65 | batch 400/2304 [I 2022-10-10 03:34:22] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:00 [I 2022-10-10 03:34:22] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00309 [I 2022-10-10 03:34:22] ...trap/engines/engine.py.230: loss: 0.09928 [I 2022-10-10 03:34:22] ...optimizers/trijoint.py.151: total_norm: 12.419446 [I 2022-10-10 03:34:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 410/2304 [I 2022-10-10 03:34:31] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:54 [I 2022-10-10 03:34:31] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00304 [I 2022-10-10 03:34:31] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-10 03:34:31] ...optimizers/trijoint.py.151: total_norm: 19.636620 [I 2022-10-10 03:34:39] ...trap/engines/engine.py.224: train: epoch 65 | batch 420/2304 [I 2022-10-10 03:34:39] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:37 [I 2022-10-10 03:34:39] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00297 [I 2022-10-10 03:34:39] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-10 03:34:39] ...optimizers/trijoint.py.151: total_norm: 10.667965 [I 2022-10-10 03:34:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 430/2304 [I 2022-10-10 03:34:48] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:33 [I 2022-10-10 03:34:48] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00322 [I 2022-10-10 03:34:48] ...trap/engines/engine.py.230: loss: 0.12259 [I 2022-10-10 03:34:48] ...optimizers/trijoint.py.151: total_norm: 12.450532 [I 2022-10-10 03:34:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 440/2304 [I 2022-10-10 03:34:56] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:14 [I 2022-10-10 03:34:56] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00297 [I 2022-10-10 03:34:56] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-10 03:34:56] ...optimizers/trijoint.py.151: total_norm: 10.592793 [I 2022-10-10 03:35:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 450/2304 [I 2022-10-10 03:35:05] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:25:58 [I 2022-10-10 03:35:05] ...trap/engines/engine.py.229: process: 0.83890 | load: 0.00301 [I 2022-10-10 03:35:05] ...trap/engines/engine.py.230: loss: 0.13353 [I 2022-10-10 03:35:05] ...optimizers/trijoint.py.151: total_norm: 15.798861 [I 2022-10-10 03:35:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 460/2304 [I 2022-10-10 03:35:13] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:56 [I 2022-10-10 03:35:13] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00300 [I 2022-10-10 03:35:13] ...trap/engines/engine.py.230: loss: 0.15088 [I 2022-10-10 03:35:13] ...optimizers/trijoint.py.151: total_norm: 29.136383 [I 2022-10-10 03:35:22] ...trap/engines/engine.py.224: train: epoch 65 | batch 470/2304 [I 2022-10-10 03:35:22] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:50 [I 2022-10-10 03:35:22] ...trap/engines/engine.py.229: process: 0.84021 | load: 0.00297 [I 2022-10-10 03:35:22] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-10 03:35:22] ...optimizers/trijoint.py.151: total_norm: 9.457263 [I 2022-10-10 03:35:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 480/2304 [I 2022-10-10 03:35:30] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:43 [I 2022-10-10 03:35:30] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00304 [I 2022-10-10 03:35:30] ...trap/engines/engine.py.230: loss: 0.16773 [I 2022-10-10 03:35:30] ...optimizers/trijoint.py.151: total_norm: 46.092358 [I 2022-10-10 03:35:39] ...trap/engines/engine.py.224: train: epoch 65 | batch 490/2304 [I 2022-10-10 03:35:39] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:38 [I 2022-10-10 03:35:39] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00299 [I 2022-10-10 03:35:39] ...trap/engines/engine.py.230: loss: 0.09928 [I 2022-10-10 03:35:39] ...optimizers/trijoint.py.151: total_norm: 11.458428 [I 2022-10-10 03:35:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 500/2304 [I 2022-10-10 03:35:47] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:17 [I 2022-10-10 03:35:47] ...trap/engines/engine.py.229: process: 0.83339 | load: 0.00300 [I 2022-10-10 03:35:47] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-10 03:35:47] ...optimizers/trijoint.py.151: total_norm: 19.214520 [I 2022-10-10 03:35:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 510/2304 [I 2022-10-10 03:35:56] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:16 [I 2022-10-10 03:35:56] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00312 [I 2022-10-10 03:35:56] ...trap/engines/engine.py.230: loss: 0.15026 [I 2022-10-10 03:35:56] ...optimizers/trijoint.py.151: total_norm: 66.481918 [I 2022-10-10 03:36:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 520/2304 [I 2022-10-10 03:36:04] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:11 [I 2022-10-10 03:36:04] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00299 [I 2022-10-10 03:36:04] ...trap/engines/engine.py.230: loss: 0.11769 [I 2022-10-10 03:36:04] ...optimizers/trijoint.py.151: total_norm: 10.483860 [I 2022-10-10 03:36:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 530/2304 [I 2022-10-10 03:36:13] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:11 [I 2022-10-10 03:36:13] ...trap/engines/engine.py.229: process: 0.86273 | load: 0.00311 [I 2022-10-10 03:36:13] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-10 03:36:13] ...optimizers/trijoint.py.151: total_norm: 9.093091 [I 2022-10-10 03:36:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 540/2304 [I 2022-10-10 03:36:21] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:03 [I 2022-10-10 03:36:21] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00311 [I 2022-10-10 03:36:21] ...trap/engines/engine.py.230: loss: 0.15637 [I 2022-10-10 03:36:21] ...optimizers/trijoint.py.151: total_norm: 13.152177 [I 2022-10-10 03:36:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 550/2304 [I 2022-10-10 03:36:30] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:02 [I 2022-10-10 03:36:30] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00311 [I 2022-10-10 03:36:30] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-10 03:36:30] ...optimizers/trijoint.py.151: total_norm: 22.645805 [I 2022-10-10 03:36:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 560/2304 [I 2022-10-10 03:36:38] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:42 [I 2022-10-10 03:36:38] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00307 [I 2022-10-10 03:36:38] ...trap/engines/engine.py.230: loss: 0.13778 [I 2022-10-10 03:36:38] ...optimizers/trijoint.py.151: total_norm: 13.775047 [I 2022-10-10 03:36:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 570/2304 [I 2022-10-10 03:36:47] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:55 [I 2022-10-10 03:36:47] ...trap/engines/engine.py.229: process: 0.87384 | load: 0.00325 [I 2022-10-10 03:36:47] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-10 03:36:47] ...optimizers/trijoint.py.151: total_norm: 19.150574 [I 2022-10-10 03:36:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 580/2304 [I 2022-10-10 03:36:56] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:34 [I 2022-10-10 03:36:56] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00313 [I 2022-10-10 03:36:56] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-10 03:36:56] ...optimizers/trijoint.py.151: total_norm: 11.630204 [I 2022-10-10 03:37:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 590/2304 [I 2022-10-10 03:37:04] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:23 [I 2022-10-10 03:37:04] ...trap/engines/engine.py.229: process: 0.86455 | load: 0.00306 [I 2022-10-10 03:37:04] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-10 03:37:04] ...optimizers/trijoint.py.151: total_norm: 39.223824 [I 2022-10-10 03:37:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 600/2304 [I 2022-10-10 03:37:13] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:15 [I 2022-10-10 03:37:13] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00311 [I 2022-10-10 03:37:13] ...trap/engines/engine.py.230: loss: 0.10722 [I 2022-10-10 03:37:13] ...optimizers/trijoint.py.151: total_norm: 7.763679 [I 2022-10-10 03:37:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 610/2304 [I 2022-10-10 03:37:21] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:12 [I 2022-10-10 03:37:21] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00310 [I 2022-10-10 03:37:21] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-10 03:37:21] ...optimizers/trijoint.py.151: total_norm: 14.690002 [I 2022-10-10 03:37:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 620/2304 [I 2022-10-10 03:37:30] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:57 [I 2022-10-10 03:37:30] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00311 [I 2022-10-10 03:37:30] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-10 03:37:30] ...optimizers/trijoint.py.151: total_norm: 11.975413 [I 2022-10-10 03:37:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 630/2304 [I 2022-10-10 03:37:38] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:37 [I 2022-10-10 03:37:38] ...trap/engines/engine.py.229: process: 0.83334 | load: 0.00313 [I 2022-10-10 03:37:38] ...trap/engines/engine.py.230: loss: 0.18693 [I 2022-10-10 03:37:38] ...optimizers/trijoint.py.151: total_norm: 9.718096 [I 2022-10-10 03:37:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 640/2304 [I 2022-10-10 03:37:47] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:42 [I 2022-10-10 03:37:47] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00311 [I 2022-10-10 03:37:47] ...trap/engines/engine.py.230: loss: 0.16511 [I 2022-10-10 03:37:47] ...optimizers/trijoint.py.151: total_norm: 10.446800 [I 2022-10-10 03:37:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 650/2304 [I 2022-10-10 03:37:56] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:38 [I 2022-10-10 03:37:56] ...trap/engines/engine.py.229: process: 0.86366 | load: 0.00311 [I 2022-10-10 03:37:56] ...trap/engines/engine.py.230: loss: 0.20326 [I 2022-10-10 03:37:56] ...optimizers/trijoint.py.151: total_norm: 32.546082 [I 2022-10-10 03:38:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 660/2304 [I 2022-10-10 03:38:04] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:27 [I 2022-10-10 03:38:04] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00309 [I 2022-10-10 03:38:04] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-10 03:38:04] ...optimizers/trijoint.py.151: total_norm: 21.571655 [I 2022-10-10 03:38:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 670/2304 [I 2022-10-10 03:38:13] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:18 [I 2022-10-10 03:38:13] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00308 [I 2022-10-10 03:38:13] ...trap/engines/engine.py.230: loss: 0.10496 [I 2022-10-10 03:38:13] ...optimizers/trijoint.py.151: total_norm: 20.246569 [I 2022-10-10 03:38:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 680/2304 [I 2022-10-10 03:38:21] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:02 [I 2022-10-10 03:38:21] ...trap/engines/engine.py.229: process: 0.84988 | load: 0.00311 [I 2022-10-10 03:38:21] ...trap/engines/engine.py.230: loss: 0.15855 [I 2022-10-10 03:38:21] ...optimizers/trijoint.py.151: total_norm: 12.963302 [I 2022-10-10 03:38:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 690/2304 [I 2022-10-10 03:38:30] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:58 [I 2022-10-10 03:38:30] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00309 [I 2022-10-10 03:38:30] ...trap/engines/engine.py.230: loss: 0.12960 [I 2022-10-10 03:38:30] ...optimizers/trijoint.py.151: total_norm: 11.282651 [I 2022-10-10 03:38:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 700/2304 [I 2022-10-10 03:38:38] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:41 [I 2022-10-10 03:38:38] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00314 [I 2022-10-10 03:38:38] ...trap/engines/engine.py.230: loss: 0.21179 [I 2022-10-10 03:38:38] ...optimizers/trijoint.py.151: total_norm: 19.109758 [I 2022-10-10 03:38:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 710/2304 [I 2022-10-10 03:38:47] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:37 [I 2022-10-10 03:38:47] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00310 [I 2022-10-10 03:38:47] ...trap/engines/engine.py.230: loss: 0.18101 [I 2022-10-10 03:38:47] ...optimizers/trijoint.py.151: total_norm: 21.328640 [I 2022-10-10 03:38:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 720/2304 [I 2022-10-10 03:38:55] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:26 [I 2022-10-10 03:38:55] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00313 [I 2022-10-10 03:38:55] ...trap/engines/engine.py.230: loss: 0.15343 [I 2022-10-10 03:38:55] ...optimizers/trijoint.py.151: total_norm: 35.400635 [I 2022-10-10 03:39:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 730/2304 [I 2022-10-10 03:39:04] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:19 [I 2022-10-10 03:39:04] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00297 [I 2022-10-10 03:39:04] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-10 03:39:04] ...optimizers/trijoint.py.151: total_norm: 9.102957 [I 2022-10-10 03:39:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 740/2304 [I 2022-10-10 03:39:12] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:12 [I 2022-10-10 03:39:12] ...trap/engines/engine.py.229: process: 0.84226 | load: 0.00309 [I 2022-10-10 03:39:12] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-10 03:39:12] ...optimizers/trijoint.py.151: total_norm: 7.430476 [I 2022-10-10 03:39:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 750/2304 [I 2022-10-10 03:39:21] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:14 [I 2022-10-10 03:39:21] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00311 [I 2022-10-10 03:39:21] ...trap/engines/engine.py.230: loss: 0.11812 [I 2022-10-10 03:39:21] ...optimizers/trijoint.py.151: total_norm: 11.676474 [I 2022-10-10 03:39:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 760/2304 [I 2022-10-10 03:39:30] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:03 [I 2022-10-10 03:39:30] ...trap/engines/engine.py.229: process: 0.86231 | load: 0.00311 [I 2022-10-10 03:39:30] ...trap/engines/engine.py.230: loss: 0.17356 [I 2022-10-10 03:39:30] ...optimizers/trijoint.py.151: total_norm: 14.946339 [I 2022-10-10 03:39:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 770/2304 [I 2022-10-10 03:39:38] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:44 [I 2022-10-10 03:39:38] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00310 [I 2022-10-10 03:39:38] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-10 03:39:38] ...optimizers/trijoint.py.151: total_norm: 9.781369 [I 2022-10-10 03:39:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 780/2304 [I 2022-10-10 03:39:47] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:33 [I 2022-10-10 03:39:47] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00310 [I 2022-10-10 03:39:47] ...trap/engines/engine.py.230: loss: 0.09301 [I 2022-10-10 03:39:47] ...optimizers/trijoint.py.151: total_norm: 13.522421 [I 2022-10-10 03:39:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 790/2304 [I 2022-10-10 03:39:55] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:32 [I 2022-10-10 03:39:55] ...trap/engines/engine.py.229: process: 0.85315 | load: 0.00310 [I 2022-10-10 03:39:55] ...trap/engines/engine.py.230: loss: 0.15494 [I 2022-10-10 03:39:55] ...optimizers/trijoint.py.151: total_norm: 14.672808 [I 2022-10-10 03:40:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 800/2304 [I 2022-10-10 03:40:04] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:15 [I 2022-10-10 03:40:04] ...trap/engines/engine.py.229: process: 0.83912 | load: 0.00309 [I 2022-10-10 03:40:04] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-10 03:40:04] ...optimizers/trijoint.py.151: total_norm: 9.858438 [I 2022-10-10 03:40:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 810/2304 [I 2022-10-10 03:40:12] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:15 [I 2022-10-10 03:40:12] ...trap/engines/engine.py.229: process: 0.86202 | load: 0.00310 [I 2022-10-10 03:40:12] ...trap/engines/engine.py.230: loss: 0.17471 [I 2022-10-10 03:40:12] ...optimizers/trijoint.py.151: total_norm: 19.150457 [I 2022-10-10 03:40:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 820/2304 [I 2022-10-10 03:40:21] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:07 [I 2022-10-10 03:40:21] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00310 [I 2022-10-10 03:40:21] ...trap/engines/engine.py.230: loss: 0.16588 [I 2022-10-10 03:40:21] ...optimizers/trijoint.py.151: total_norm: 18.621672 [I 2022-10-10 03:40:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 830/2304 [I 2022-10-10 03:40:29] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:59 [I 2022-10-10 03:40:29] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00312 [I 2022-10-10 03:40:29] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-10 03:40:29] ...optimizers/trijoint.py.151: total_norm: 7.230514 [I 2022-10-10 03:40:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 840/2304 [I 2022-10-10 03:40:38] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:46 [I 2022-10-10 03:40:38] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00312 [I 2022-10-10 03:40:38] ...trap/engines/engine.py.230: loss: 0.14605 [I 2022-10-10 03:40:38] ...optimizers/trijoint.py.151: total_norm: 38.712898 [I 2022-10-10 03:40:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 850/2304 [I 2022-10-10 03:40:46] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:41 [I 2022-10-10 03:40:46] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00308 [I 2022-10-10 03:40:46] ...trap/engines/engine.py.230: loss: 0.11392 [I 2022-10-10 03:40:46] ...optimizers/trijoint.py.151: total_norm: 30.843170 [I 2022-10-10 03:40:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 860/2304 [I 2022-10-10 03:40:55] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:21 [I 2022-10-10 03:40:55] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00293 [I 2022-10-10 03:40:55] ...trap/engines/engine.py.230: loss: 0.09479 [I 2022-10-10 03:40:55] ...optimizers/trijoint.py.151: total_norm: 10.968411 [I 2022-10-10 03:41:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 870/2304 [I 2022-10-10 03:41:03] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:13 [I 2022-10-10 03:41:03] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00293 [I 2022-10-10 03:41:03] ...trap/engines/engine.py.230: loss: 0.14069 [I 2022-10-10 03:41:03] ...optimizers/trijoint.py.151: total_norm: 24.167112 [I 2022-10-10 03:41:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 880/2304 [I 2022-10-10 03:41:12] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:03 [I 2022-10-10 03:41:12] ...trap/engines/engine.py.229: process: 0.84778 | load: 0.00298 [I 2022-10-10 03:41:12] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-10 03:41:12] ...optimizers/trijoint.py.151: total_norm: 31.687666 [I 2022-10-10 03:41:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 890/2304 [I 2022-10-10 03:41:20] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:47 [I 2022-10-10 03:41:20] ...trap/engines/engine.py.229: process: 0.83062 | load: 0.00299 [I 2022-10-10 03:41:20] ...trap/engines/engine.py.230: loss: 0.10778 [I 2022-10-10 03:41:20] ...optimizers/trijoint.py.151: total_norm: 11.132120 [I 2022-10-10 03:41:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 900/2304 [I 2022-10-10 03:41:29] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:42 [I 2022-10-10 03:41:29] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00293 [I 2022-10-10 03:41:29] ...trap/engines/engine.py.230: loss: 0.22068 [I 2022-10-10 03:41:29] ...optimizers/trijoint.py.151: total_norm: 12.538890 [I 2022-10-10 03:41:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 910/2304 [I 2022-10-10 03:41:37] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:39 [I 2022-10-10 03:41:37] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00303 [I 2022-10-10 03:41:37] ...trap/engines/engine.py.230: loss: 0.09037 [I 2022-10-10 03:41:37] ...optimizers/trijoint.py.151: total_norm: 9.110438 [I 2022-10-10 03:41:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 920/2304 [I 2022-10-10 03:41:46] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:27 [I 2022-10-10 03:41:46] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00294 [I 2022-10-10 03:41:46] ...trap/engines/engine.py.230: loss: 0.10481 [I 2022-10-10 03:41:46] ...optimizers/trijoint.py.151: total_norm: 9.829426 [I 2022-10-10 03:41:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 930/2304 [I 2022-10-10 03:41:54] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:17 [I 2022-10-10 03:41:54] ...trap/engines/engine.py.229: process: 0.84310 | load: 0.00295 [I 2022-10-10 03:41:54] ...trap/engines/engine.py.230: loss: 0.21654 [I 2022-10-10 03:41:54] ...optimizers/trijoint.py.151: total_norm: 13.764144 [I 2022-10-10 03:42:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 940/2304 [I 2022-10-10 03:42:03] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:11 [I 2022-10-10 03:42:03] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00298 [I 2022-10-10 03:42:03] ...trap/engines/engine.py.230: loss: 0.12158 [I 2022-10-10 03:42:03] ...optimizers/trijoint.py.151: total_norm: 16.956026 [I 2022-10-10 03:42:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 950/2304 [I 2022-10-10 03:42:11] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:05 [I 2022-10-10 03:42:11] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00296 [I 2022-10-10 03:42:11] ...trap/engines/engine.py.230: loss: 0.12501 [I 2022-10-10 03:42:11] ...optimizers/trijoint.py.151: total_norm: 13.026566 [I 2022-10-10 03:42:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 960/2304 [I 2022-10-10 03:42:20] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:18:52 [I 2022-10-10 03:42:20] ...trap/engines/engine.py.229: process: 0.84429 | load: 0.00297 [I 2022-10-10 03:42:20] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-10 03:42:20] ...optimizers/trijoint.py.151: total_norm: 9.077728 [I 2022-10-10 03:42:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 970/2304 [I 2022-10-10 03:42:28] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:46 [I 2022-10-10 03:42:28] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00296 [I 2022-10-10 03:42:28] ...trap/engines/engine.py.230: loss: 0.16876 [I 2022-10-10 03:42:28] ...optimizers/trijoint.py.151: total_norm: 26.300776 [I 2022-10-10 03:42:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 980/2304 [I 2022-10-10 03:42:37] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:47 [I 2022-10-10 03:42:37] ...trap/engines/engine.py.229: process: 0.85401 | load: 0.00298 [I 2022-10-10 03:42:37] ...trap/engines/engine.py.230: loss: 0.16207 [I 2022-10-10 03:42:37] ...optimizers/trijoint.py.151: total_norm: 17.100431 [I 2022-10-10 03:42:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 990/2304 [I 2022-10-10 03:42:45] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:32 [I 2022-10-10 03:42:45] ...trap/engines/engine.py.229: process: 0.83042 | load: 0.00298 [I 2022-10-10 03:42:45] ...trap/engines/engine.py.230: loss: 0.19068 [I 2022-10-10 03:42:45] ...optimizers/trijoint.py.151: total_norm: 22.940079 [I 2022-10-10 03:42:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1000/2304 [I 2022-10-10 03:42:53] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:20 [I 2022-10-10 03:42:53] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00296 [I 2022-10-10 03:42:53] ...trap/engines/engine.py.230: loss: 0.09650 [I 2022-10-10 03:42:53] ...optimizers/trijoint.py.151: total_norm: 8.195930 [I 2022-10-10 03:43:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1010/2304 [I 2022-10-10 03:43:02] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:28 [I 2022-10-10 03:43:02] ...trap/engines/engine.py.229: process: 0.85888 | load: 0.00311 [I 2022-10-10 03:43:02] ...trap/engines/engine.py.230: loss: 0.25451 [I 2022-10-10 03:43:02] ...optimizers/trijoint.py.151: total_norm: 13.748367 [I 2022-10-10 03:43:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1020/2304 [I 2022-10-10 03:43:11] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:19 [I 2022-10-10 03:43:11] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00311 [I 2022-10-10 03:43:11] ...trap/engines/engine.py.230: loss: 0.10990 [I 2022-10-10 03:43:11] ...optimizers/trijoint.py.151: total_norm: 13.948255 [I 2022-10-10 03:43:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1030/2304 [I 2022-10-10 03:43:19] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:09 [I 2022-10-10 03:43:19] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00324 [I 2022-10-10 03:43:19] ...trap/engines/engine.py.230: loss: 0.13206 [I 2022-10-10 03:43:19] ...optimizers/trijoint.py.151: total_norm: 12.798088 [I 2022-10-10 03:43:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1040/2304 [I 2022-10-10 03:43:28] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:58 [I 2022-10-10 03:43:28] ...trap/engines/engine.py.229: process: 0.85180 | load: 0.00312 [I 2022-10-10 03:43:28] ...trap/engines/engine.py.230: loss: 0.12990 [I 2022-10-10 03:43:28] ...optimizers/trijoint.py.151: total_norm: 14.309308 [I 2022-10-10 03:43:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1050/2304 [I 2022-10-10 03:43:36] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:49 [I 2022-10-10 03:43:36] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00314 [I 2022-10-10 03:43:36] ...trap/engines/engine.py.230: loss: 0.12408 [I 2022-10-10 03:43:36] ...optimizers/trijoint.py.151: total_norm: 15.530465 [I 2022-10-10 03:43:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1060/2304 [I 2022-10-10 03:43:45] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:39 [I 2022-10-10 03:43:45] ...trap/engines/engine.py.229: process: 0.85081 | load: 0.00310 [I 2022-10-10 03:43:45] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-10 03:43:45] ...optimizers/trijoint.py.151: total_norm: 11.672416 [I 2022-10-10 03:43:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1070/2304 [I 2022-10-10 03:43:53] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:34 [I 2022-10-10 03:43:53] ...trap/engines/engine.py.229: process: 0.85770 | load: 0.00313 [I 2022-10-10 03:43:53] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-10 03:43:53] ...optimizers/trijoint.py.151: total_norm: 11.192205 [I 2022-10-10 03:44:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1080/2304 [I 2022-10-10 03:44:02] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:28 [I 2022-10-10 03:44:02] ...trap/engines/engine.py.229: process: 0.86229 | load: 0.00314 [I 2022-10-10 03:44:02] ...trap/engines/engine.py.230: loss: 0.09730 [I 2022-10-10 03:44:02] ...optimizers/trijoint.py.151: total_norm: 9.537628 [I 2022-10-10 03:44:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1090/2304 [I 2022-10-10 03:44:11] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:18 [I 2022-10-10 03:44:11] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00311 [I 2022-10-10 03:44:11] ...trap/engines/engine.py.230: loss: 0.09516 [I 2022-10-10 03:44:11] ...optimizers/trijoint.py.151: total_norm: 21.376822 [I 2022-10-10 03:44:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1100/2304 [I 2022-10-10 03:44:19] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:11 [I 2022-10-10 03:44:19] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00312 [I 2022-10-10 03:44:19] ...trap/engines/engine.py.230: loss: 0.15142 [I 2022-10-10 03:44:19] ...optimizers/trijoint.py.151: total_norm: 18.940498 [I 2022-10-10 03:44:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1110/2304 [I 2022-10-10 03:44:28] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:56 [I 2022-10-10 03:44:28] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00325 [I 2022-10-10 03:44:28] ...trap/engines/engine.py.230: loss: 0.21010 [I 2022-10-10 03:44:28] ...optimizers/trijoint.py.151: total_norm: 9.939249 [I 2022-10-10 03:44:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1120/2304 [I 2022-10-10 03:44:36] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:52 [I 2022-10-10 03:44:36] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00310 [I 2022-10-10 03:44:36] ...trap/engines/engine.py.230: loss: 0.16739 [I 2022-10-10 03:44:36] ...optimizers/trijoint.py.151: total_norm: 9.653991 [I 2022-10-10 03:44:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1130/2304 [I 2022-10-10 03:44:45] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:46 [I 2022-10-10 03:44:45] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00430 [I 2022-10-10 03:44:45] ...trap/engines/engine.py.230: loss: 0.17212 [I 2022-10-10 03:44:45] ...optimizers/trijoint.py.151: total_norm: 13.585322 [I 2022-10-10 03:44:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1140/2304 [I 2022-10-10 03:44:54] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:34 [I 2022-10-10 03:44:54] ...trap/engines/engine.py.229: process: 0.86582 | load: 0.00330 [I 2022-10-10 03:44:54] ...trap/engines/engine.py.230: loss: 0.24124 [I 2022-10-10 03:44:54] ...optimizers/trijoint.py.151: total_norm: 19.443878 [I 2022-10-10 03:45:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1150/2304 [I 2022-10-10 03:45:02] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:28 [I 2022-10-10 03:45:02] ...trap/engines/engine.py.229: process: 0.87088 | load: 0.00314 [I 2022-10-10 03:45:02] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-10 03:45:02] ...optimizers/trijoint.py.151: total_norm: 10.269894 [I 2022-10-10 03:45:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1160/2304 [I 2022-10-10 03:45:11] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:23 [I 2022-10-10 03:45:11] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00310 [I 2022-10-10 03:45:11] ...trap/engines/engine.py.230: loss: 0.10308 [I 2022-10-10 03:45:11] ...optimizers/trijoint.py.151: total_norm: 13.000532 [I 2022-10-10 03:45:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1170/2304 [I 2022-10-10 03:45:19] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:16 [I 2022-10-10 03:45:19] ...trap/engines/engine.py.229: process: 0.87685 | load: 0.00313 [I 2022-10-10 03:45:19] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-10 03:45:19] ...optimizers/trijoint.py.151: total_norm: 23.927227 [I 2022-10-10 03:45:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1180/2304 [I 2022-10-10 03:45:28] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:16:04 [I 2022-10-10 03:45:28] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00310 [I 2022-10-10 03:45:28] ...trap/engines/engine.py.230: loss: 0.12017 [I 2022-10-10 03:45:28] ...optimizers/trijoint.py.151: total_norm: 21.740498 [I 2022-10-10 03:45:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1190/2304 [I 2022-10-10 03:45:36] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:50 [I 2022-10-10 03:45:36] ...trap/engines/engine.py.229: process: 0.84832 | load: 0.00310 [I 2022-10-10 03:45:36] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-10 03:45:36] ...optimizers/trijoint.py.151: total_norm: 27.607630 [I 2022-10-10 03:45:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1200/2304 [I 2022-10-10 03:45:45] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:40 [I 2022-10-10 03:45:45] ...trap/engines/engine.py.229: process: 0.85030 | load: 0.00311 [I 2022-10-10 03:45:45] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-10 03:45:45] ...optimizers/trijoint.py.151: total_norm: 9.489721 [I 2022-10-10 03:45:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 1210/2304 [I 2022-10-10 03:45:54] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:39 [I 2022-10-10 03:45:54] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00312 [I 2022-10-10 03:45:54] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-10 03:45:54] ...optimizers/trijoint.py.151: total_norm: 13.564912 [I 2022-10-10 03:46:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1220/2304 [I 2022-10-10 03:46:02] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:23 [I 2022-10-10 03:46:02] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00317 [I 2022-10-10 03:46:02] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-10 03:46:02] ...optimizers/trijoint.py.151: total_norm: 12.249887 [I 2022-10-10 03:46:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1230/2304 [I 2022-10-10 03:46:11] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:15:14 [I 2022-10-10 03:46:11] ...trap/engines/engine.py.229: process: 0.84235 | load: 0.00314 [I 2022-10-10 03:46:11] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-10 03:46:11] ...optimizers/trijoint.py.151: total_norm: 14.375143 [I 2022-10-10 03:46:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1240/2304 [I 2022-10-10 03:46:19] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:15:05 [I 2022-10-10 03:46:19] ...trap/engines/engine.py.229: process: 0.83669 | load: 0.00313 [I 2022-10-10 03:46:19] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-10 03:46:19] ...optimizers/trijoint.py.151: total_norm: 7.506547 [I 2022-10-10 03:46:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1250/2304 [I 2022-10-10 03:46:28] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:01 [I 2022-10-10 03:46:28] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00311 [I 2022-10-10 03:46:28] ...trap/engines/engine.py.230: loss: 0.12552 [I 2022-10-10 03:46:28] ...optimizers/trijoint.py.151: total_norm: 16.050697 [I 2022-10-10 03:46:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1260/2304 [I 2022-10-10 03:46:36] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:45 [I 2022-10-10 03:46:36] ...trap/engines/engine.py.229: process: 0.85489 | load: 0.00311 [I 2022-10-10 03:46:36] ...trap/engines/engine.py.230: loss: 0.12860 [I 2022-10-10 03:46:36] ...optimizers/trijoint.py.151: total_norm: 8.712481 [I 2022-10-10 03:46:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1270/2304 [I 2022-10-10 03:46:45] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:43 [I 2022-10-10 03:46:45] ...trap/engines/engine.py.229: process: 0.85276 | load: 0.00309 [I 2022-10-10 03:46:45] ...trap/engines/engine.py.230: loss: 0.12102 [I 2022-10-10 03:46:45] ...optimizers/trijoint.py.151: total_norm: 20.215368 [I 2022-10-10 03:46:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1280/2304 [I 2022-10-10 03:46:53] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:29 [I 2022-10-10 03:46:53] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00309 [I 2022-10-10 03:46:53] ...trap/engines/engine.py.230: loss: 0.10564 [I 2022-10-10 03:46:53] ...optimizers/trijoint.py.151: total_norm: 9.711615 [I 2022-10-10 03:47:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1290/2304 [I 2022-10-10 03:47:02] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:24 [I 2022-10-10 03:47:02] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00308 [I 2022-10-10 03:47:02] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-10 03:47:02] ...optimizers/trijoint.py.151: total_norm: 21.334129 [I 2022-10-10 03:47:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1300/2304 [I 2022-10-10 03:47:10] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:07 [I 2022-10-10 03:47:10] ...trap/engines/engine.py.229: process: 0.83099 | load: 0.00300 [I 2022-10-10 03:47:10] ...trap/engines/engine.py.230: loss: 0.13843 [I 2022-10-10 03:47:10] ...optimizers/trijoint.py.151: total_norm: 18.314146 [I 2022-10-10 03:47:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1310/2304 [I 2022-10-10 03:47:19] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:13:56 [I 2022-10-10 03:47:19] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00293 [I 2022-10-10 03:47:19] ...trap/engines/engine.py.230: loss: 0.18227 [I 2022-10-10 03:47:19] ...optimizers/trijoint.py.151: total_norm: 17.314135 [I 2022-10-10 03:47:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1320/2304 [I 2022-10-10 03:47:27] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:13:52 [I 2022-10-10 03:47:27] ...trap/engines/engine.py.229: process: 0.83835 | load: 0.00303 [I 2022-10-10 03:47:27] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-10 03:47:27] ...optimizers/trijoint.py.151: total_norm: 15.838036 [I 2022-10-10 03:47:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1330/2304 [I 2022-10-10 03:47:36] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:45 [I 2022-10-10 03:47:36] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00297 [I 2022-10-10 03:47:36] ...trap/engines/engine.py.230: loss: 0.12405 [I 2022-10-10 03:47:36] ...optimizers/trijoint.py.151: total_norm: 20.790178 [I 2022-10-10 03:48:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 1340/2304 [I 2022-10-10 03:48:03] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:25:50 [I 2022-10-10 03:48:03] ...trap/engines/engine.py.229: process: 0.83508 | load: 0.00301 [I 2022-10-10 03:48:03] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-10 03:48:03] ...optimizers/trijoint.py.151: total_norm: 15.148127 [I 2022-10-10 03:48:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1350/2304 [I 2022-10-10 03:48:11] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:14:41 [I 2022-10-10 03:48:11] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00297 [I 2022-10-10 03:48:11] ...trap/engines/engine.py.230: loss: 0.09683 [I 2022-10-10 03:48:11] ...optimizers/trijoint.py.151: total_norm: 9.059282 [I 2022-10-10 03:48:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1360/2304 [I 2022-10-10 03:48:19] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:21 [I 2022-10-10 03:48:19] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00302 [I 2022-10-10 03:48:19] ...trap/engines/engine.py.230: loss: 0.09701 [I 2022-10-10 03:48:19] ...optimizers/trijoint.py.151: total_norm: 12.158354 [I 2022-10-10 03:48:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1370/2304 [I 2022-10-10 03:48:28] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:10 [I 2022-10-10 03:48:28] ...trap/engines/engine.py.229: process: 0.83568 | load: 0.00306 [I 2022-10-10 03:48:28] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-10 03:48:28] ...optimizers/trijoint.py.151: total_norm: 16.912991 [I 2022-10-10 03:48:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1380/2304 [I 2022-10-10 03:48:36] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:00 [I 2022-10-10 03:48:36] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00302 [I 2022-10-10 03:48:36] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-10 03:48:36] ...optimizers/trijoint.py.151: total_norm: 13.376773 [I 2022-10-10 03:48:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1390/2304 [I 2022-10-10 03:48:45] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:50 [I 2022-10-10 03:48:45] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00300 [I 2022-10-10 03:48:45] ...trap/engines/engine.py.230: loss: 0.12119 [I 2022-10-10 03:48:45] ...optimizers/trijoint.py.151: total_norm: 17.137749 [I 2022-10-10 03:48:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1400/2304 [I 2022-10-10 03:48:53] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:39 [I 2022-10-10 03:48:53] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00297 [I 2022-10-10 03:48:53] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-10 03:48:53] ...optimizers/trijoint.py.151: total_norm: 7.790540 [I 2022-10-10 03:49:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1410/2304 [I 2022-10-10 03:49:02] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:39 [I 2022-10-10 03:49:02] ...trap/engines/engine.py.229: process: 0.85981 | load: 0.00300 [I 2022-10-10 03:49:02] ...trap/engines/engine.py.230: loss: 0.18966 [I 2022-10-10 03:49:02] ...optimizers/trijoint.py.151: total_norm: 29.393402 [I 2022-10-10 03:49:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1420/2304 [I 2022-10-10 03:49:10] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:12:28 [I 2022-10-10 03:49:10] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00298 [I 2022-10-10 03:49:10] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-10 03:49:10] ...optimizers/trijoint.py.151: total_norm: 27.101088 [I 2022-10-10 03:49:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1430/2304 [I 2022-10-10 03:49:19] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:19 [I 2022-10-10 03:49:19] ...trap/engines/engine.py.229: process: 0.84718 | load: 0.00294 [I 2022-10-10 03:49:19] ...trap/engines/engine.py.230: loss: 0.11314 [I 2022-10-10 03:49:19] ...optimizers/trijoint.py.151: total_norm: 9.436483 [I 2022-10-10 03:49:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1440/2304 [I 2022-10-10 03:49:27] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:09 [I 2022-10-10 03:49:27] ...trap/engines/engine.py.229: process: 0.83471 | load: 0.00302 [I 2022-10-10 03:49:27] ...trap/engines/engine.py.230: loss: 0.20845 [I 2022-10-10 03:49:27] ...optimizers/trijoint.py.151: total_norm: 24.005573 [I 2022-10-10 03:49:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1450/2304 [I 2022-10-10 03:49:36] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:09 [I 2022-10-10 03:49:36] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00324 [I 2022-10-10 03:49:36] ...trap/engines/engine.py.230: loss: 0.10430 [I 2022-10-10 03:49:36] ...optimizers/trijoint.py.151: total_norm: 9.052210 [I 2022-10-10 03:49:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1460/2304 [I 2022-10-10 03:49:44] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:12:00 [I 2022-10-10 03:49:44] ...trap/engines/engine.py.229: process: 0.86948 | load: 0.00312 [I 2022-10-10 03:49:44] ...trap/engines/engine.py.230: loss: 0.12247 [I 2022-10-10 03:49:44] ...optimizers/trijoint.py.151: total_norm: 19.107433 [I 2022-10-10 03:49:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1470/2304 [I 2022-10-10 03:49:53] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:52 [I 2022-10-10 03:49:53] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00318 [I 2022-10-10 03:49:53] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-10 03:49:53] ...optimizers/trijoint.py.151: total_norm: 22.439283 [I 2022-10-10 03:50:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1480/2304 [I 2022-10-10 03:50:01] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:43 [I 2022-10-10 03:50:01] ...trap/engines/engine.py.229: process: 0.85775 | load: 0.00313 [I 2022-10-10 03:50:01] ...trap/engines/engine.py.230: loss: 0.13341 [I 2022-10-10 03:50:01] ...optimizers/trijoint.py.151: total_norm: 23.102991 [I 2022-10-10 03:50:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1490/2304 [I 2022-10-10 03:50:10] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:38 [I 2022-10-10 03:50:10] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00314 [I 2022-10-10 03:50:10] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-10 03:50:10] ...optimizers/trijoint.py.151: total_norm: 37.172230 [I 2022-10-10 03:50:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1500/2304 [I 2022-10-10 03:50:19] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:27 [I 2022-10-10 03:50:19] ...trap/engines/engine.py.229: process: 0.87202 | load: 0.00323 [I 2022-10-10 03:50:19] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-10 03:50:19] ...optimizers/trijoint.py.151: total_norm: 22.552235 [I 2022-10-10 03:50:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1510/2304 [I 2022-10-10 03:50:27] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:16 [I 2022-10-10 03:50:27] ...trap/engines/engine.py.229: process: 0.85080 | load: 0.00312 [I 2022-10-10 03:50:27] ...trap/engines/engine.py.230: loss: 0.14857 [I 2022-10-10 03:50:27] ...optimizers/trijoint.py.151: total_norm: 10.669283 [I 2022-10-10 03:50:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1520/2304 [I 2022-10-10 03:50:36] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:09 [I 2022-10-10 03:50:36] ...trap/engines/engine.py.229: process: 0.86456 | load: 0.00312 [I 2022-10-10 03:50:36] ...trap/engines/engine.py.230: loss: 0.09674 [I 2022-10-10 03:50:36] ...optimizers/trijoint.py.151: total_norm: 11.472157 [I 2022-10-10 03:50:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1530/2304 [I 2022-10-10 03:50:44] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:01 [I 2022-10-10 03:50:44] ...trap/engines/engine.py.229: process: 0.86167 | load: 0.00311 [I 2022-10-10 03:50:44] ...trap/engines/engine.py.230: loss: 0.13058 [I 2022-10-10 03:50:44] ...optimizers/trijoint.py.151: total_norm: 12.836134 [I 2022-10-10 03:50:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1540/2304 [I 2022-10-10 03:50:53] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:54 [I 2022-10-10 03:50:53] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00311 [I 2022-10-10 03:50:53] ...trap/engines/engine.py.230: loss: 0.15352 [I 2022-10-10 03:50:53] ...optimizers/trijoint.py.151: total_norm: 10.505537 [I 2022-10-10 03:51:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1550/2304 [I 2022-10-10 03:51:01] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:44 [I 2022-10-10 03:51:01] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00314 [I 2022-10-10 03:51:01] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-10 03:51:01] ...optimizers/trijoint.py.151: total_norm: 25.750790 [I 2022-10-10 03:51:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1560/2304 [I 2022-10-10 03:51:10] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:34 [I 2022-10-10 03:51:10] ...trap/engines/engine.py.229: process: 0.84577 | load: 0.00313 [I 2022-10-10 03:51:10] ...trap/engines/engine.py.230: loss: 0.09576 [I 2022-10-10 03:51:10] ...optimizers/trijoint.py.151: total_norm: 7.473754 [I 2022-10-10 03:51:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1570/2304 [I 2022-10-10 03:51:19] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:27 [I 2022-10-10 03:51:19] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00311 [I 2022-10-10 03:51:19] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-10 03:51:19] ...optimizers/trijoint.py.151: total_norm: 8.123512 [I 2022-10-10 03:51:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1580/2304 [I 2022-10-10 03:51:27] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:22 [I 2022-10-10 03:51:27] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00314 [I 2022-10-10 03:51:27] ...trap/engines/engine.py.230: loss: 0.10441 [I 2022-10-10 03:51:27] ...optimizers/trijoint.py.151: total_norm: 9.625235 [I 2022-10-10 03:51:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1590/2304 [I 2022-10-10 03:51:36] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:09 [I 2022-10-10 03:51:36] ...trap/engines/engine.py.229: process: 0.85672 | load: 0.00311 [I 2022-10-10 03:51:36] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-10 03:51:36] ...optimizers/trijoint.py.151: total_norm: 9.523354 [I 2022-10-10 03:51:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1600/2304 [I 2022-10-10 03:51:44] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:02 [I 2022-10-10 03:51:44] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00310 [I 2022-10-10 03:51:44] ...trap/engines/engine.py.230: loss: 0.11684 [I 2022-10-10 03:51:44] ...optimizers/trijoint.py.151: total_norm: 13.941116 [I 2022-10-10 03:51:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1610/2304 [I 2022-10-10 03:51:53] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:49 [I 2022-10-10 03:51:53] ...trap/engines/engine.py.229: process: 0.86050 | load: 0.00310 [I 2022-10-10 03:51:53] ...trap/engines/engine.py.230: loss: 0.12998 [I 2022-10-10 03:51:53] ...optimizers/trijoint.py.151: total_norm: 16.485756 [I 2022-10-10 03:52:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1620/2304 [I 2022-10-10 03:52:01] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:42 [I 2022-10-10 03:52:01] ...trap/engines/engine.py.229: process: 0.84366 | load: 0.00312 [I 2022-10-10 03:52:01] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-10 03:52:01] ...optimizers/trijoint.py.151: total_norm: 22.848860 [I 2022-10-10 03:52:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1630/2304 [I 2022-10-10 03:52:10] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:33 [I 2022-10-10 03:52:10] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00320 [I 2022-10-10 03:52:10] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-10 03:52:10] ...optimizers/trijoint.py.151: total_norm: 6.546916 [I 2022-10-10 03:52:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 1640/2304 [I 2022-10-10 03:52:18] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:26 [I 2022-10-10 03:52:18] ...trap/engines/engine.py.229: process: 0.84646 | load: 0.00314 [I 2022-10-10 03:52:18] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-10 03:52:18] ...optimizers/trijoint.py.151: total_norm: 11.807012 [I 2022-10-10 03:52:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1650/2304 [I 2022-10-10 03:52:27] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:15 [I 2022-10-10 03:52:27] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00314 [I 2022-10-10 03:52:27] ...trap/engines/engine.py.230: loss: 0.15873 [I 2022-10-10 03:52:27] ...optimizers/trijoint.py.151: total_norm: 13.548635 [I 2022-10-10 03:52:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1660/2304 [I 2022-10-10 03:52:36] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:09:10 [I 2022-10-10 03:52:36] ...trap/engines/engine.py.229: process: 0.85229 | load: 0.00313 [I 2022-10-10 03:52:36] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-10 03:52:36] ...optimizers/trijoint.py.151: total_norm: 23.234608 [I 2022-10-10 03:52:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1670/2304 [I 2022-10-10 03:52:44] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:09:01 [I 2022-10-10 03:52:44] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00315 [I 2022-10-10 03:52:44] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-10 03:52:44] ...optimizers/trijoint.py.151: total_norm: 8.417415 [I 2022-10-10 03:52:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1680/2304 [I 2022-10-10 03:52:53] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:54 [I 2022-10-10 03:52:53] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00311 [I 2022-10-10 03:52:53] ...trap/engines/engine.py.230: loss: 0.11345 [I 2022-10-10 03:52:53] ...optimizers/trijoint.py.151: total_norm: 16.438927 [I 2022-10-10 03:53:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1690/2304 [I 2022-10-10 03:53:01] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:44 [I 2022-10-10 03:53:01] ...trap/engines/engine.py.229: process: 0.86166 | load: 0.00312 [I 2022-10-10 03:53:01] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-10 03:53:01] ...optimizers/trijoint.py.151: total_norm: 8.588253 [I 2022-10-10 03:53:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1700/2304 [I 2022-10-10 03:53:10] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:34 [I 2022-10-10 03:53:10] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00314 [I 2022-10-10 03:53:10] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-10 03:53:10] ...optimizers/trijoint.py.151: total_norm: 27.849407 [I 2022-10-10 03:53:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 1710/2304 [I 2022-10-10 03:53:18] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:27 [I 2022-10-10 03:53:18] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00313 [I 2022-10-10 03:53:18] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-10 03:53:18] ...optimizers/trijoint.py.151: total_norm: 12.292996 [I 2022-10-10 03:53:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1720/2304 [I 2022-10-10 03:53:27] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:17 [I 2022-10-10 03:53:27] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00314 [I 2022-10-10 03:53:27] ...trap/engines/engine.py.230: loss: 0.15577 [I 2022-10-10 03:53:27] ...optimizers/trijoint.py.151: total_norm: 6.832802 [I 2022-10-10 03:53:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 1730/2304 [I 2022-10-10 03:53:35] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:05 [I 2022-10-10 03:53:35] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00305 [I 2022-10-10 03:53:35] ...trap/engines/engine.py.230: loss: 0.13882 [I 2022-10-10 03:53:35] ...optimizers/trijoint.py.151: total_norm: 12.067103 [I 2022-10-10 03:53:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1740/2304 [I 2022-10-10 03:53:44] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:57 [I 2022-10-10 03:53:44] ...trap/engines/engine.py.229: process: 0.85013 | load: 0.00298 [I 2022-10-10 03:53:44] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-10 03:53:44] ...optimizers/trijoint.py.151: total_norm: 18.611092 [I 2022-10-10 03:53:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 1750/2304 [I 2022-10-10 03:53:52] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:51 [I 2022-10-10 03:53:52] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00305 [I 2022-10-10 03:53:52] ...trap/engines/engine.py.230: loss: 0.14465 [I 2022-10-10 03:53:52] ...optimizers/trijoint.py.151: total_norm: 8.489674 [I 2022-10-10 03:54:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1760/2304 [I 2022-10-10 03:54:01] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:40 [I 2022-10-10 03:54:01] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00300 [I 2022-10-10 03:54:01] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-10 03:54:01] ...optimizers/trijoint.py.151: total_norm: 16.063164 [I 2022-10-10 03:54:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1770/2304 [I 2022-10-10 03:54:09] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:36 [I 2022-10-10 03:54:09] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00306 [I 2022-10-10 03:54:09] ...trap/engines/engine.py.230: loss: 0.19177 [I 2022-10-10 03:54:09] ...optimizers/trijoint.py.151: total_norm: 27.254103 [I 2022-10-10 03:54:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 1780/2304 [I 2022-10-10 03:54:18] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:26 [I 2022-10-10 03:54:18] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00303 [I 2022-10-10 03:54:18] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-10 03:54:18] ...optimizers/trijoint.py.151: total_norm: 13.049562 [I 2022-10-10 03:54:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1790/2304 [I 2022-10-10 03:54:26] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:15 [I 2022-10-10 03:54:26] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00298 [I 2022-10-10 03:54:26] ...trap/engines/engine.py.230: loss: 0.12777 [I 2022-10-10 03:54:26] ...optimizers/trijoint.py.151: total_norm: 14.602652 [I 2022-10-10 03:54:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 1800/2304 [I 2022-10-10 03:54:35] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:09 [I 2022-10-10 03:54:35] ...trap/engines/engine.py.229: process: 0.86755 | load: 0.00300 [I 2022-10-10 03:54:35] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-10 03:54:35] ...optimizers/trijoint.py.151: total_norm: 15.623128 [I 2022-10-10 03:54:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1810/2304 [I 2022-10-10 03:54:44] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:03 [I 2022-10-10 03:54:44] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00298 [I 2022-10-10 03:54:44] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-10 03:54:44] ...optimizers/trijoint.py.151: total_norm: 17.924414 [I 2022-10-10 03:54:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 1820/2304 [I 2022-10-10 03:54:52] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:54 [I 2022-10-10 03:54:52] ...trap/engines/engine.py.229: process: 0.88406 | load: 0.00299 [I 2022-10-10 03:54:52] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-10 03:54:52] ...optimizers/trijoint.py.151: total_norm: 15.187639 [I 2022-10-10 03:55:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1830/2304 [I 2022-10-10 03:55:01] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:43 [I 2022-10-10 03:55:01] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00299 [I 2022-10-10 03:55:01] ...trap/engines/engine.py.230: loss: 0.13778 [I 2022-10-10 03:55:01] ...optimizers/trijoint.py.151: total_norm: 14.655363 [I 2022-10-10 03:55:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1840/2304 [I 2022-10-10 03:55:09] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:35 [I 2022-10-10 03:55:09] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00305 [I 2022-10-10 03:55:09] ...trap/engines/engine.py.230: loss: 0.12209 [I 2022-10-10 03:55:09] ...optimizers/trijoint.py.151: total_norm: 9.167445 [I 2022-10-10 03:55:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 1850/2304 [I 2022-10-10 03:55:18] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:24 [I 2022-10-10 03:55:18] ...trap/engines/engine.py.229: process: 0.82977 | load: 0.00299 [I 2022-10-10 03:55:18] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-10 03:55:18] ...optimizers/trijoint.py.151: total_norm: 24.516237 [I 2022-10-10 03:55:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1860/2304 [I 2022-10-10 03:55:26] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:17 [I 2022-10-10 03:55:26] ...trap/engines/engine.py.229: process: 0.86413 | load: 0.00301 [I 2022-10-10 03:55:26] ...trap/engines/engine.py.230: loss: 0.16737 [I 2022-10-10 03:55:26] ...optimizers/trijoint.py.151: total_norm: 21.419300 [I 2022-10-10 03:55:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 1870/2304 [I 2022-10-10 03:55:35] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:08 [I 2022-10-10 03:55:35] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00301 [I 2022-10-10 03:55:35] ...trap/engines/engine.py.230: loss: 0.14852 [I 2022-10-10 03:55:35] ...optimizers/trijoint.py.151: total_norm: 13.048991 [I 2022-10-10 03:55:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 1880/2304 [I 2022-10-10 03:55:43] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:00 [I 2022-10-10 03:55:43] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00295 [I 2022-10-10 03:55:43] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-10 03:55:43] ...optimizers/trijoint.py.151: total_norm: 78.544617 [I 2022-10-10 03:55:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 1890/2304 [I 2022-10-10 03:55:52] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:51 [I 2022-10-10 03:55:52] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00300 [I 2022-10-10 03:55:52] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-10 03:55:52] ...optimizers/trijoint.py.151: total_norm: 12.992918 [I 2022-10-10 03:56:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 1900/2304 [I 2022-10-10 03:56:00] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:45 [I 2022-10-10 03:56:00] ...trap/engines/engine.py.229: process: 0.84806 | load: 0.00317 [I 2022-10-10 03:56:00] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-10 03:56:00] ...optimizers/trijoint.py.151: total_norm: 20.412611 [I 2022-10-10 03:56:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1910/2304 [I 2022-10-10 03:56:09] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:36 [I 2022-10-10 03:56:09] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00321 [I 2022-10-10 03:56:09] ...trap/engines/engine.py.230: loss: 0.11173 [I 2022-10-10 03:56:09] ...optimizers/trijoint.py.151: total_norm: 10.551094 [I 2022-10-10 03:56:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 1920/2304 [I 2022-10-10 03:56:17] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:25 [I 2022-10-10 03:56:17] ...trap/engines/engine.py.229: process: 0.83381 | load: 0.00313 [I 2022-10-10 03:56:17] ...trap/engines/engine.py.230: loss: 0.21394 [I 2022-10-10 03:56:17] ...optimizers/trijoint.py.151: total_norm: 30.320187 [I 2022-10-10 03:56:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1930/2304 [I 2022-10-10 03:56:26] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:17 [I 2022-10-10 03:56:26] ...trap/engines/engine.py.229: process: 0.83731 | load: 0.00316 [I 2022-10-10 03:56:26] ...trap/engines/engine.py.230: loss: 0.16077 [I 2022-10-10 03:56:26] ...optimizers/trijoint.py.151: total_norm: 44.893459 [I 2022-10-10 03:56:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 1940/2304 [I 2022-10-10 03:56:34] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:08 [I 2022-10-10 03:56:34] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00323 [I 2022-10-10 03:56:34] ...trap/engines/engine.py.230: loss: 0.17764 [I 2022-10-10 03:56:34] ...optimizers/trijoint.py.151: total_norm: 13.944393 [I 2022-10-10 03:56:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 1950/2304 [I 2022-10-10 03:56:43] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:05:01 [I 2022-10-10 03:56:43] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00311 [I 2022-10-10 03:56:43] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-10 03:56:43] ...optimizers/trijoint.py.151: total_norm: 9.652637 [I 2022-10-10 03:56:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 1960/2304 [I 2022-10-10 03:56:51] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:51 [I 2022-10-10 03:56:51] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00312 [I 2022-10-10 03:56:51] ...trap/engines/engine.py.230: loss: 0.14415 [I 2022-10-10 03:56:51] ...optimizers/trijoint.py.151: total_norm: 10.591706 [I 2022-10-10 03:57:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 1970/2304 [I 2022-10-10 03:57:00] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:44 [I 2022-10-10 03:57:00] ...trap/engines/engine.py.229: process: 0.84443 | load: 0.00313 [I 2022-10-10 03:57:00] ...trap/engines/engine.py.230: loss: 0.10541 [I 2022-10-10 03:57:00] ...optimizers/trijoint.py.151: total_norm: 16.842848 [I 2022-10-10 03:57:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1980/2304 [I 2022-10-10 03:57:09] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:35 [I 2022-10-10 03:57:09] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00314 [I 2022-10-10 03:57:09] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-10 03:57:09] ...optimizers/trijoint.py.151: total_norm: 17.486868 [I 2022-10-10 03:57:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 1990/2304 [I 2022-10-10 03:57:17] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:27 [I 2022-10-10 03:57:17] ...trap/engines/engine.py.229: process: 0.85539 | load: 0.00314 [I 2022-10-10 03:57:17] ...trap/engines/engine.py.230: loss: 0.15716 [I 2022-10-10 03:57:17] ...optimizers/trijoint.py.151: total_norm: 13.272789 [I 2022-10-10 03:57:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 2000/2304 [I 2022-10-10 03:57:26] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:19 [I 2022-10-10 03:57:26] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00315 [I 2022-10-10 03:57:26] ...trap/engines/engine.py.230: loss: 0.16914 [I 2022-10-10 03:57:26] ...optimizers/trijoint.py.151: total_norm: 18.286949 [I 2022-10-10 03:57:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 2010/2304 [I 2022-10-10 03:57:34] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:12 [I 2022-10-10 03:57:34] ...trap/engines/engine.py.229: process: 0.85893 | load: 0.00311 [I 2022-10-10 03:57:34] ...trap/engines/engine.py.230: loss: 0.17170 [I 2022-10-10 03:57:34] ...optimizers/trijoint.py.151: total_norm: 25.361071 [I 2022-10-10 03:57:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 2020/2304 [I 2022-10-10 03:57:43] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:04:02 [I 2022-10-10 03:57:43] ...trap/engines/engine.py.229: process: 0.86201 | load: 0.00317 [I 2022-10-10 03:57:43] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-10 03:57:43] ...optimizers/trijoint.py.151: total_norm: 10.677609 [I 2022-10-10 03:57:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 2030/2304 [I 2022-10-10 03:57:51] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:52 [I 2022-10-10 03:57:51] ...trap/engines/engine.py.229: process: 0.83874 | load: 0.00316 [I 2022-10-10 03:57:51] ...trap/engines/engine.py.230: loss: 0.10165 [I 2022-10-10 03:57:51] ...optimizers/trijoint.py.151: total_norm: 10.662502 [I 2022-10-10 03:58:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 2040/2304 [I 2022-10-10 03:58:00] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:45 [I 2022-10-10 03:58:00] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00314 [I 2022-10-10 03:58:00] ...trap/engines/engine.py.230: loss: 0.10651 [I 2022-10-10 03:58:00] ...optimizers/trijoint.py.151: total_norm: 13.416018 [I 2022-10-10 03:58:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 2050/2304 [I 2022-10-10 03:58:08] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:36 [I 2022-10-10 03:58:08] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00313 [I 2022-10-10 03:58:08] ...trap/engines/engine.py.230: loss: 0.13889 [I 2022-10-10 03:58:08] ...optimizers/trijoint.py.151: total_norm: 48.762157 [I 2022-10-10 03:58:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 2060/2304 [I 2022-10-10 03:58:17] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:29 [I 2022-10-10 03:58:17] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00319 [I 2022-10-10 03:58:17] ...trap/engines/engine.py.230: loss: 0.16753 [I 2022-10-10 03:58:17] ...optimizers/trijoint.py.151: total_norm: 9.319528 [I 2022-10-10 03:58:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 2070/2304 [I 2022-10-10 03:58:25] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:17 [I 2022-10-10 03:58:25] ...trap/engines/engine.py.229: process: 0.84697 | load: 0.00315 [I 2022-10-10 03:58:25] ...trap/engines/engine.py.230: loss: 0.19221 [I 2022-10-10 03:58:25] ...optimizers/trijoint.py.151: total_norm: 19.074463 [I 2022-10-10 03:58:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 2080/2304 [I 2022-10-10 03:58:34] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:03:10 [I 2022-10-10 03:58:34] ...trap/engines/engine.py.229: process: 0.85151 | load: 0.00313 [I 2022-10-10 03:58:34] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-10 03:58:34] ...optimizers/trijoint.py.151: total_norm: 16.945774 [I 2022-10-10 03:58:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 2090/2304 [I 2022-10-10 03:58:42] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:03:02 [I 2022-10-10 03:58:43] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00312 [I 2022-10-10 03:58:43] ...trap/engines/engine.py.230: loss: 0.14047 [I 2022-10-10 03:58:43] ...optimizers/trijoint.py.151: total_norm: 14.663776 [I 2022-10-10 03:58:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 2100/2304 [I 2022-10-10 03:58:51] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:53 [I 2022-10-10 03:58:51] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00317 [I 2022-10-10 03:58:51] ...trap/engines/engine.py.230: loss: 0.10853 [I 2022-10-10 03:58:51] ...optimizers/trijoint.py.151: total_norm: 13.291718 [I 2022-10-10 03:59:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 2110/2304 [I 2022-10-10 03:59:00] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:02:46 [I 2022-10-10 03:59:00] ...trap/engines/engine.py.229: process: 0.86644 | load: 0.00309 [I 2022-10-10 03:59:00] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-10 03:59:00] ...optimizers/trijoint.py.151: total_norm: 12.777431 [I 2022-10-10 03:59:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 2120/2304 [I 2022-10-10 03:59:08] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:37 [I 2022-10-10 03:59:08] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00313 [I 2022-10-10 03:59:08] ...trap/engines/engine.py.230: loss: 0.13671 [I 2022-10-10 03:59:08] ...optimizers/trijoint.py.151: total_norm: 20.757084 [I 2022-10-10 03:59:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 2130/2304 [I 2022-10-10 03:59:17] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:02:29 [I 2022-10-10 03:59:17] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00312 [I 2022-10-10 03:59:17] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-10 03:59:17] ...optimizers/trijoint.py.151: total_norm: 13.754233 [I 2022-10-10 03:59:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 2140/2304 [I 2022-10-10 03:59:25] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:02:20 [I 2022-10-10 03:59:25] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00315 [I 2022-10-10 03:59:25] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-10 03:59:25] ...optimizers/trijoint.py.151: total_norm: 21.137053 [I 2022-10-10 03:59:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 2150/2304 [I 2022-10-10 03:59:34] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:02:11 [I 2022-10-10 03:59:34] ...trap/engines/engine.py.229: process: 0.85634 | load: 0.00314 [I 2022-10-10 03:59:34] ...trap/engines/engine.py.230: loss: 0.18425 [I 2022-10-10 03:59:34] ...optimizers/trijoint.py.151: total_norm: 23.124296 [I 2022-10-10 03:59:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 2160/2304 [I 2022-10-10 03:59:42] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:02:03 [I 2022-10-10 03:59:42] ...trap/engines/engine.py.229: process: 0.86353 | load: 0.00312 [I 2022-10-10 03:59:42] ...trap/engines/engine.py.230: loss: 0.09631 [I 2022-10-10 03:59:42] ...optimizers/trijoint.py.151: total_norm: 7.003193 [I 2022-10-10 03:59:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 2170/2304 [I 2022-10-10 03:59:51] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:54 [I 2022-10-10 03:59:51] ...trap/engines/engine.py.229: process: 0.86393 | load: 0.00312 [I 2022-10-10 03:59:51] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-10 03:59:51] ...optimizers/trijoint.py.151: total_norm: 10.482704 [I 2022-10-10 03:59:59] ...trap/engines/engine.py.224: train: epoch 65 | batch 2180/2304 [I 2022-10-10 03:59:59] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:45 [I 2022-10-10 03:59:59] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00307 [I 2022-10-10 03:59:59] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-10 03:59:59] ...optimizers/trijoint.py.151: total_norm: 17.449450 [I 2022-10-10 04:00:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 2190/2304 [I 2022-10-10 04:00:08] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:36 [I 2022-10-10 04:00:08] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00295 [I 2022-10-10 04:00:08] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-10 04:00:08] ...optimizers/trijoint.py.151: total_norm: 12.457655 [I 2022-10-10 04:00:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 2200/2304 [I 2022-10-10 04:00:16] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:28 [I 2022-10-10 04:00:16] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00296 [I 2022-10-10 04:00:16] ...trap/engines/engine.py.230: loss: 0.13462 [I 2022-10-10 04:00:16] ...optimizers/trijoint.py.151: total_norm: 9.692245 [I 2022-10-10 04:00:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 2210/2304 [I 2022-10-10 04:00:25] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:19 [I 2022-10-10 04:00:25] ...trap/engines/engine.py.229: process: 0.84999 | load: 0.00298 [I 2022-10-10 04:00:25] ...trap/engines/engine.py.230: loss: 0.09223 [I 2022-10-10 04:00:25] ...optimizers/trijoint.py.151: total_norm: 11.867856 [I 2022-10-10 04:00:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 2220/2304 [I 2022-10-10 04:00:33] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:11 [I 2022-10-10 04:00:33] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00293 [I 2022-10-10 04:00:33] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-10 04:00:33] ...optimizers/trijoint.py.151: total_norm: 10.489590 [I 2022-10-10 04:00:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 2230/2304 [I 2022-10-10 04:00:42] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:01:02 [I 2022-10-10 04:00:42] ...trap/engines/engine.py.229: process: 0.83517 | load: 0.00291 [I 2022-10-10 04:00:42] ...trap/engines/engine.py.230: loss: 0.10849 [I 2022-10-10 04:00:42] ...optimizers/trijoint.py.151: total_norm: 9.384051 [I 2022-10-10 04:00:50] ...trap/engines/engine.py.224: train: epoch 65 | batch 2240/2304 [I 2022-10-10 04:00:50] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:54 [I 2022-10-10 04:00:50] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00295 [I 2022-10-10 04:00:50] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-10 04:00:50] ...optimizers/trijoint.py.151: total_norm: 12.227629 [I 2022-10-10 04:00:59] ...trap/engines/engine.py.224: train: epoch 65 | batch 2250/2304 [I 2022-10-10 04:00:59] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:45 [I 2022-10-10 04:00:59] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00296 [I 2022-10-10 04:00:59] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-10 04:00:59] ...optimizers/trijoint.py.151: total_norm: 13.082458 [I 2022-10-10 04:01:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 2260/2304 [I 2022-10-10 04:01:07] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:37 [I 2022-10-10 04:01:07] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00305 [I 2022-10-10 04:01:07] ...trap/engines/engine.py.230: loss: 0.14464 [I 2022-10-10 04:01:07] ...optimizers/trijoint.py.151: total_norm: 24.435093 [I 2022-10-10 04:01:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 2270/2304 [I 2022-10-10 04:01:16] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:29 [I 2022-10-10 04:01:16] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00301 [I 2022-10-10 04:01:16] ...trap/engines/engine.py.230: loss: 0.17471 [I 2022-10-10 04:01:16] ...optimizers/trijoint.py.151: total_norm: 19.125776 [I 2022-10-10 04:01:24] ...trap/engines/engine.py.224: train: epoch 65 | batch 2280/2304 [I 2022-10-10 04:01:24] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:20 [I 2022-10-10 04:01:24] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00294 [I 2022-10-10 04:01:24] ...trap/engines/engine.py.230: loss: 0.16485 [I 2022-10-10 04:01:24] ...optimizers/trijoint.py.151: total_norm: 11.725021 [I 2022-10-10 04:01:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 2290/2304 [I 2022-10-10 04:01:33] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:11 [I 2022-10-10 04:01:33] ...trap/engines/engine.py.229: process: 0.85188 | load: 0.00302 [I 2022-10-10 04:01:33] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-10 04:01:33] ...optimizers/trijoint.py.151: total_norm: 12.022729 [I 2022-10-10 04:01:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 2300/2304 [I 2022-10-10 04:01:41] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:03 [I 2022-10-10 04:01:41] ...trap/engines/engine.py.229: process: 0.83151 | load: 0.00017 [I 2022-10-10 04:01:41] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-10 04:01:41] ...optimizers/trijoint.py.151: total_norm: 11.340537 [I 2022-10-10 04:01:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 2304/2304 [I 2022-10-10 04:01:45] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:00 [I 2022-10-10 04:01:45] ...trap/engines/engine.py.229: process: 0.83611 | load: 0.00017 [I 2022-10-10 04:01:45] ...trap/engines/engine.py.230: loss: 0.10541 [I 2022-10-10 04:01:45] ...optimizers/trijoint.py.151: total_norm: 13.466634 [S 2022-10-10 04:01:46] ...trap/engines/engine.py.129: train_epoch.epoch: 65 [S 2022-10-10 04:01:46] ...trap/engines/engine.py.129: train_epoch.loss: 0.14346318065215088 [I 2022-10-10 04:01:46] ...s/criterions/triplet.py.84: margin = 0.3000000000000001 [I 2022-10-10 04:01:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 65 [I 2022-10-10 04:01:47] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 04:01:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 04:01:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 04:01:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 04:01:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 04:01:52] ...trap/engines/engine.py.321: eval: epoch 65 | batch 0/510 [I 2022-10-10 04:01:52] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:51 [I 2022-10-10 04:01:52] ...trap/engines/engine.py.326: process: 4.80747 | load: 3.38146 [I 2022-10-10 04:01:56] ...trap/engines/engine.py.321: eval: epoch 65 | batch 10/510 [I 2022-10-10 04:01:56] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:34 [I 2022-10-10 04:01:56] ...trap/engines/engine.py.326: process: 0.43183 | load: 0.00018 [I 2022-10-10 04:02:01] ...trap/engines/engine.py.321: eval: epoch 65 | batch 20/510 [I 2022-10-10 04:02:01] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:03:59 [I 2022-10-10 04:02:01] ...trap/engines/engine.py.326: process: 0.42460 | load: 0.00019 [I 2022-10-10 04:02:06] ...trap/engines/engine.py.321: eval: epoch 65 | batch 30/510 [I 2022-10-10 04:02:06] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:04:26 [I 2022-10-10 04:02:06] ...trap/engines/engine.py.326: process: 0.59533 | load: 0.00019 [I 2022-10-10 04:02:11] ...trap/engines/engine.py.321: eval: epoch 65 | batch 40/510 [I 2022-10-10 04:02:11] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:50 [I 2022-10-10 04:02:11] ...trap/engines/engine.py.326: process: 0.45679 | load: 0.00020 [I 2022-10-10 04:02:15] ...trap/engines/engine.py.321: eval: epoch 65 | batch 50/510 [I 2022-10-10 04:02:15] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:26 [I 2022-10-10 04:02:15] ...trap/engines/engine.py.326: process: 0.49266 | load: 0.00019 [I 2022-10-10 04:02:20] ...trap/engines/engine.py.321: eval: epoch 65 | batch 60/510 [I 2022-10-10 04:02:20] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:42 [I 2022-10-10 04:02:20] ...trap/engines/engine.py.326: process: 0.50248 | load: 0.00019 [I 2022-10-10 04:02:26] ...trap/engines/engine.py.321: eval: epoch 65 | batch 70/510 [I 2022-10-10 04:02:26] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:04:24 [I 2022-10-10 04:02:26] ...trap/engines/engine.py.326: process: 0.63799 | load: 0.00019 [I 2022-10-10 04:02:32] ...trap/engines/engine.py.321: eval: epoch 65 | batch 80/510 [I 2022-10-10 04:02:32] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:04:06 [I 2022-10-10 04:02:32] ...trap/engines/engine.py.326: process: 0.54374 | load: 0.00020 [I 2022-10-10 04:02:37] ...trap/engines/engine.py.321: eval: epoch 65 | batch 90/510 [I 2022-10-10 04:02:37] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:11 [I 2022-10-10 04:02:37] ...trap/engines/engine.py.326: process: 0.42336 | load: 0.00020 [I 2022-10-10 04:02:41] ...trap/engines/engine.py.321: eval: epoch 65 | batch 100/510 [I 2022-10-10 04:02:41] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:54 [I 2022-10-10 04:02:41] ...trap/engines/engine.py.326: process: 0.30300 | load: 0.00019 [I 2022-10-10 04:02:44] ...trap/engines/engine.py.321: eval: epoch 65 | batch 110/510 [I 2022-10-10 04:02:44] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:10 [I 2022-10-10 04:02:44] ...trap/engines/engine.py.326: process: 0.31259 | load: 0.00019 [I 2022-10-10 04:02:47] ...trap/engines/engine.py.321: eval: epoch 65 | batch 120/510 [I 2022-10-10 04:02:47] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-10 04:02:47] ...trap/engines/engine.py.326: process: 0.31781 | load: 0.00022 [I 2022-10-10 04:02:51] ...trap/engines/engine.py.321: eval: epoch 65 | batch 130/510 [I 2022-10-10 04:02:51] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-10 04:02:51] ...trap/engines/engine.py.326: process: 0.31611 | load: 0.00020 [I 2022-10-10 04:02:54] ...trap/engines/engine.py.321: eval: epoch 65 | batch 140/510 [I 2022-10-10 04:02:54] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:58 [I 2022-10-10 04:02:54] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00020 [I 2022-10-10 04:02:57] ...trap/engines/engine.py.321: eval: epoch 65 | batch 150/510 [I 2022-10-10 04:02:57] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:53 [I 2022-10-10 04:02:57] ...trap/engines/engine.py.326: process: 0.31759 | load: 0.00021 [I 2022-10-10 04:03:00] ...trap/engines/engine.py.321: eval: epoch 65 | batch 160/510 [I 2022-10-10 04:03:00] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:51 [I 2022-10-10 04:03:00] ...trap/engines/engine.py.326: process: 0.32300 | load: 0.00021 [I 2022-10-10 04:03:03] ...trap/engines/engine.py.321: eval: epoch 65 | batch 170/510 [I 2022-10-10 04:03:03] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:48 [I 2022-10-10 04:03:03] ...trap/engines/engine.py.326: process: 0.31868 | load: 0.00020 [I 2022-10-10 04:03:07] ...trap/engines/engine.py.321: eval: epoch 65 | batch 180/510 [I 2022-10-10 04:03:07] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:45 [I 2022-10-10 04:03:07] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00020 [I 2022-10-10 04:03:10] ...trap/engines/engine.py.321: eval: epoch 65 | batch 190/510 [I 2022-10-10 04:03:10] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:41 [I 2022-10-10 04:03:10] ...trap/engines/engine.py.326: process: 0.31831 | load: 0.00020 [I 2022-10-10 04:03:13] ...trap/engines/engine.py.321: eval: epoch 65 | batch 200/510 [I 2022-10-10 04:03:13] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-10 04:03:13] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00020 [I 2022-10-10 04:03:16] ...trap/engines/engine.py.321: eval: epoch 65 | batch 210/510 [I 2022-10-10 04:03:16] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-10 04:03:16] ...trap/engines/engine.py.326: process: 0.32252 | load: 0.00021 [I 2022-10-10 04:03:19] ...trap/engines/engine.py.321: eval: epoch 65 | batch 220/510 [I 2022-10-10 04:03:19] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:32 [I 2022-10-10 04:03:19] ...trap/engines/engine.py.326: process: 0.31763 | load: 0.00020 [I 2022-10-10 04:03:23] ...trap/engines/engine.py.321: eval: epoch 65 | batch 230/510 [I 2022-10-10 04:03:23] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:30 [I 2022-10-10 04:03:23] ...trap/engines/engine.py.326: process: 0.32379 | load: 0.00020 [I 2022-10-10 04:03:26] ...trap/engines/engine.py.321: eval: epoch 65 | batch 240/510 [I 2022-10-10 04:03:26] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-10 04:03:26] ...trap/engines/engine.py.326: process: 0.32095 | load: 0.00020 [I 2022-10-10 04:03:29] ...trap/engines/engine.py.321: eval: epoch 65 | batch 250/510 [I 2022-10-10 04:03:29] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-10 04:03:29] ...trap/engines/engine.py.326: process: 0.32208 | load: 0.00020 [I 2022-10-10 04:03:32] ...trap/engines/engine.py.321: eval: epoch 65 | batch 260/510 [I 2022-10-10 04:03:32] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:20 [I 2022-10-10 04:03:32] ...trap/engines/engine.py.326: process: 0.32399 | load: 0.00021 [I 2022-10-10 04:03:35] ...trap/engines/engine.py.321: eval: epoch 65 | batch 270/510 [I 2022-10-10 04:03:35] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:17 [I 2022-10-10 04:03:35] ...trap/engines/engine.py.326: process: 0.31893 | load: 0.00022 [I 2022-10-10 04:03:39] ...trap/engines/engine.py.321: eval: epoch 65 | batch 280/510 [I 2022-10-10 04:03:39] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-10 04:03:39] ...trap/engines/engine.py.326: process: 0.32547 | load: 0.00022 [I 2022-10-10 04:03:42] ...trap/engines/engine.py.321: eval: epoch 65 | batch 290/510 [I 2022-10-10 04:03:42] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-10 04:03:42] ...trap/engines/engine.py.326: process: 0.32617 | load: 0.00021 [I 2022-10-10 04:03:45] ...trap/engines/engine.py.321: eval: epoch 65 | batch 300/510 [I 2022-10-10 04:03:45] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:07 [I 2022-10-10 04:03:45] ...trap/engines/engine.py.326: process: 0.32489 | load: 0.00021 [I 2022-10-10 04:03:48] ...trap/engines/engine.py.321: eval: epoch 65 | batch 310/510 [I 2022-10-10 04:03:48] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:04 [I 2022-10-10 04:03:48] ...trap/engines/engine.py.326: process: 0.32085 | load: 0.00021 [I 2022-10-10 04:03:52] ...trap/engines/engine.py.321: eval: epoch 65 | batch 320/510 [I 2022-10-10 04:03:52] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:01 [I 2022-10-10 04:03:52] ...trap/engines/engine.py.326: process: 0.32325 | load: 0.00021 [I 2022-10-10 04:03:55] ...trap/engines/engine.py.321: eval: epoch 65 | batch 330/510 [I 2022-10-10 04:03:55] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:58 [I 2022-10-10 04:03:55] ...trap/engines/engine.py.326: process: 0.32384 | load: 0.00021 [I 2022-10-10 04:03:58] ...trap/engines/engine.py.321: eval: epoch 65 | batch 340/510 [I 2022-10-10 04:03:58] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-10 04:03:58] ...trap/engines/engine.py.326: process: 0.32555 | load: 0.00022 [I 2022-10-10 04:04:01] ...trap/engines/engine.py.321: eval: epoch 65 | batch 350/510 [I 2022-10-10 04:04:01] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-10 04:04:01] ...trap/engines/engine.py.326: process: 0.32609 | load: 0.00021 [I 2022-10-10 04:04:04] ...trap/engines/engine.py.321: eval: epoch 65 | batch 360/510 [I 2022-10-10 04:04:04] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:48 [I 2022-10-10 04:04:04] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00020 [I 2022-10-10 04:04:08] ...trap/engines/engine.py.321: eval: epoch 65 | batch 370/510 [I 2022-10-10 04:04:08] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:45 [I 2022-10-10 04:04:08] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00020 [I 2022-10-10 04:04:11] ...trap/engines/engine.py.321: eval: epoch 65 | batch 380/510 [I 2022-10-10 04:04:11] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-10 04:04:11] ...trap/engines/engine.py.326: process: 0.32563 | load: 0.00021 [I 2022-10-10 04:04:14] ...trap/engines/engine.py.321: eval: epoch 65 | batch 390/510 [I 2022-10-10 04:04:14] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-10 04:04:14] ...trap/engines/engine.py.326: process: 0.32358 | load: 0.00022 [I 2022-10-10 04:04:17] ...trap/engines/engine.py.321: eval: epoch 65 | batch 400/510 [I 2022-10-10 04:04:17] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-10 04:04:17] ...trap/engines/engine.py.326: process: 0.32365 | load: 0.00021 [I 2022-10-10 04:04:21] ...trap/engines/engine.py.321: eval: epoch 65 | batch 410/510 [I 2022-10-10 04:04:21] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:32 [I 2022-10-10 04:04:21] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00021 [I 2022-10-10 04:04:24] ...trap/engines/engine.py.321: eval: epoch 65 | batch 420/510 [I 2022-10-10 04:04:24] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:29 [I 2022-10-10 04:04:24] ...trap/engines/engine.py.326: process: 0.32282 | load: 0.00022 [I 2022-10-10 04:04:27] ...trap/engines/engine.py.321: eval: epoch 65 | batch 430/510 [I 2022-10-10 04:04:27] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-10 04:04:27] ...trap/engines/engine.py.326: process: 0.32510 | load: 0.00021 [I 2022-10-10 04:04:30] ...trap/engines/engine.py.321: eval: epoch 65 | batch 440/510 [I 2022-10-10 04:04:30] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-10 04:04:30] ...trap/engines/engine.py.326: process: 0.32209 | load: 0.00020 [I 2022-10-10 04:04:33] ...trap/engines/engine.py.321: eval: epoch 65 | batch 450/510 [I 2022-10-10 04:04:33] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-10 04:04:33] ...trap/engines/engine.py.326: process: 0.31959 | load: 0.00020 [I 2022-10-10 04:04:37] ...trap/engines/engine.py.321: eval: epoch 65 | batch 460/510 [I 2022-10-10 04:04:37] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:16 [I 2022-10-10 04:04:37] ...trap/engines/engine.py.326: process: 0.32206 | load: 0.00020 [I 2022-10-10 04:04:40] ...trap/engines/engine.py.321: eval: epoch 65 | batch 470/510 [I 2022-10-10 04:04:40] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-10 04:04:40] ...trap/engines/engine.py.326: process: 0.31964 | load: 0.00021 [I 2022-10-10 04:04:43] ...trap/engines/engine.py.321: eval: epoch 65 | batch 480/510 [I 2022-10-10 04:04:43] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-10 04:04:43] ...trap/engines/engine.py.326: process: 0.32224 | load: 0.00021 [I 2022-10-10 04:04:46] ...trap/engines/engine.py.321: eval: epoch 65 | batch 490/510 [I 2022-10-10 04:04:46] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 04:04:46] ...trap/engines/engine.py.326: process: 0.32371 | load: 0.00021 [I 2022-10-10 04:04:50] ...trap/engines/engine.py.321: eval: epoch 65 | batch 500/510 [I 2022-10-10 04:04:50] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-10 04:04:50] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00022 [I 2022-10-10 04:04:53] ...trap/engines/engine.py.321: eval: epoch 65 | batch 510/510 [I 2022-10-10 04:04:53] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-10 04:04:53] ...trap/engines/engine.py.326: process: 0.31290 | load: 0.00018 [S 2022-10-10 04:04:54] ...trap/engines/engine.py.132: eval_epoch.epoch: 65 [I 2022-10-10 04:04:54] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736100 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913600 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941600 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727900 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.913500 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.941400 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011954 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008913 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008369 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017857 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009667 [S 2022-10-10 04:05:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007800 [I 2022-10-10 04:05:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 04:05:29] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 04:05:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 04:05:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 04:05:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 04:05:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 04:05:35] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 04:05:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 04:05:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 66 [I 2022-10-10 04:05:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 04:05:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 0/2304 [I 2022-10-10 04:05:43] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:17 [I 2022-10-10 04:05:43] ...trap/engines/engine.py.229: process: 3.00244 | load: 2.07476 [I 2022-10-10 04:05:43] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-10 04:05:43] ...optimizers/trijoint.py.151: total_norm: 10.522077 [I 2022-10-10 04:05:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 10/2304 [I 2022-10-10 04:05:52] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:22 [I 2022-10-10 04:05:52] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00625 [I 2022-10-10 04:05:52] ...trap/engines/engine.py.230: loss: 0.14141 [I 2022-10-10 04:05:52] ...optimizers/trijoint.py.151: total_norm: 9.014701 [I 2022-10-10 04:06:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 20/2304 [I 2022-10-10 04:06:00] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:21 [I 2022-10-10 04:06:00] ...trap/engines/engine.py.229: process: 0.84103 | load: 0.00546 [I 2022-10-10 04:06:00] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-10 04:06:00] ...optimizers/trijoint.py.151: total_norm: 16.143616 [I 2022-10-10 04:06:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 30/2304 [I 2022-10-10 04:06:09] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:21 [I 2022-10-10 04:06:09] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00474 [I 2022-10-10 04:06:09] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-10 04:06:09] ...optimizers/trijoint.py.151: total_norm: 10.262050 [I 2022-10-10 04:06:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 40/2304 [I 2022-10-10 04:06:17] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:15 [I 2022-10-10 04:06:17] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00380 [I 2022-10-10 04:06:17] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-10 04:06:17] ...optimizers/trijoint.py.151: total_norm: 20.018753 [I 2022-10-10 04:06:26] ...trap/engines/engine.py.224: train: epoch 66 | batch 50/2304 [I 2022-10-10 04:06:26] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:51 [I 2022-10-10 04:06:26] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00387 [I 2022-10-10 04:06:26] ...trap/engines/engine.py.230: loss: 0.19617 [I 2022-10-10 04:06:26] ...optimizers/trijoint.py.151: total_norm: 10.655137 [I 2022-10-10 04:06:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 60/2304 [I 2022-10-10 04:06:34] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:48 [I 2022-10-10 04:06:34] ...trap/engines/engine.py.229: process: 0.85850 | load: 0.00407 [I 2022-10-10 04:06:34] ...trap/engines/engine.py.230: loss: 0.08824 [I 2022-10-10 04:06:34] ...optimizers/trijoint.py.151: total_norm: 12.033246 [I 2022-10-10 04:06:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 70/2304 [I 2022-10-10 04:06:43] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:59 [I 2022-10-10 04:06:43] ...trap/engines/engine.py.229: process: 0.86375 | load: 0.00349 [I 2022-10-10 04:06:43] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-10 04:06:43] ...optimizers/trijoint.py.151: total_norm: 11.707971 [I 2022-10-10 04:06:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 80/2304 [I 2022-10-10 04:06:52] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:34 [I 2022-10-10 04:06:52] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00346 [I 2022-10-10 04:06:52] ...trap/engines/engine.py.230: loss: 0.21284 [I 2022-10-10 04:06:52] ...optimizers/trijoint.py.151: total_norm: 25.090818 [I 2022-10-10 04:07:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 90/2304 [I 2022-10-10 04:07:00] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:15 [I 2022-10-10 04:07:00] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00339 [I 2022-10-10 04:07:00] ...trap/engines/engine.py.230: loss: 0.16844 [I 2022-10-10 04:07:00] ...optimizers/trijoint.py.151: total_norm: 39.680336 [I 2022-10-10 04:07:08] ...trap/engines/engine.py.224: train: epoch 66 | batch 100/2304 [I 2022-10-10 04:07:08] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:30:58 [I 2022-10-10 04:07:08] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00321 [I 2022-10-10 04:07:08] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-10 04:07:08] ...optimizers/trijoint.py.151: total_norm: 11.564252 [I 2022-10-10 04:07:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 110/2304 [I 2022-10-10 04:07:17] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:57 [I 2022-10-10 04:07:17] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00339 [I 2022-10-10 04:07:17] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-10 04:07:17] ...optimizers/trijoint.py.151: total_norm: 12.109223 [I 2022-10-10 04:07:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 120/2304 [I 2022-10-10 04:07:25] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:44 [I 2022-10-10 04:07:25] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00324 [I 2022-10-10 04:07:25] ...trap/engines/engine.py.230: loss: 0.16142 [I 2022-10-10 04:07:25] ...optimizers/trijoint.py.151: total_norm: 12.308116 [I 2022-10-10 04:07:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 130/2304 [I 2022-10-10 04:07:34] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:33 [I 2022-10-10 04:07:34] ...trap/engines/engine.py.229: process: 0.82612 | load: 0.00324 [I 2022-10-10 04:07:34] ...trap/engines/engine.py.230: loss: 0.17185 [I 2022-10-10 04:07:34] ...optimizers/trijoint.py.151: total_norm: 17.103415 [I 2022-10-10 04:07:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 140/2304 [I 2022-10-10 04:07:42] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:31 [I 2022-10-10 04:07:42] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00313 [I 2022-10-10 04:07:42] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-10 04:07:42] ...optimizers/trijoint.py.151: total_norm: 9.619197 [I 2022-10-10 04:07:51] ...trap/engines/engine.py.224: train: epoch 66 | batch 150/2304 [I 2022-10-10 04:07:51] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:13 [I 2022-10-10 04:07:51] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00312 [I 2022-10-10 04:07:51] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-10 04:07:51] ...optimizers/trijoint.py.151: total_norm: 18.796619 [I 2022-10-10 04:07:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 160/2304 [I 2022-10-10 04:07:59] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:15 [I 2022-10-10 04:07:59] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00317 [I 2022-10-10 04:07:59] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-10 04:07:59] ...optimizers/trijoint.py.151: total_norm: 9.995359 [I 2022-10-10 04:08:08] ...trap/engines/engine.py.224: train: epoch 66 | batch 170/2304 [I 2022-10-10 04:08:08] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:06 [I 2022-10-10 04:08:08] ...trap/engines/engine.py.229: process: 0.83872 | load: 0.00313 [I 2022-10-10 04:08:08] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-10 04:08:08] ...optimizers/trijoint.py.151: total_norm: 15.356792 [I 2022-10-10 04:08:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 180/2304 [I 2022-10-10 04:08:16] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:03 [I 2022-10-10 04:08:16] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00321 [I 2022-10-10 04:08:16] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-10 04:08:16] ...optimizers/trijoint.py.151: total_norm: 13.880870 [I 2022-10-10 04:08:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 190/2304 [I 2022-10-10 04:08:25] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:44 [I 2022-10-10 04:08:25] ...trap/engines/engine.py.229: process: 0.83335 | load: 0.00316 [I 2022-10-10 04:08:25] ...trap/engines/engine.py.230: loss: 0.11505 [I 2022-10-10 04:08:25] ...optimizers/trijoint.py.151: total_norm: 11.214733 [I 2022-10-10 04:08:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 200/2304 [I 2022-10-10 04:08:33] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:54 [I 2022-10-10 04:08:33] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00301 [I 2022-10-10 04:08:33] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-10 04:08:33] ...optimizers/trijoint.py.151: total_norm: 14.985641 [I 2022-10-10 04:08:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 210/2304 [I 2022-10-10 04:08:42] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:37 [I 2022-10-10 04:08:42] ...trap/engines/engine.py.229: process: 0.86169 | load: 0.00326 [I 2022-10-10 04:08:42] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-10 04:08:42] ...optimizers/trijoint.py.151: total_norm: 22.053085 [I 2022-10-10 04:08:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 220/2304 [I 2022-10-10 04:08:50] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:25 [I 2022-10-10 04:08:50] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00311 [I 2022-10-10 04:08:50] ...trap/engines/engine.py.230: loss: 0.14733 [I 2022-10-10 04:08:50] ...optimizers/trijoint.py.151: total_norm: 12.361162 [I 2022-10-10 04:08:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 230/2304 [I 2022-10-10 04:08:59] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:01 [I 2022-10-10 04:08:59] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00299 [I 2022-10-10 04:08:59] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-10 04:08:59] ...optimizers/trijoint.py.151: total_norm: 11.622730 [I 2022-10-10 04:09:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 240/2304 [I 2022-10-10 04:09:07] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:28:59 [I 2022-10-10 04:09:07] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00305 [I 2022-10-10 04:09:07] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-10 04:09:07] ...optimizers/trijoint.py.151: total_norm: 13.649437 [I 2022-10-10 04:09:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 250/2304 [I 2022-10-10 04:09:16] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:00 [I 2022-10-10 04:09:16] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00299 [I 2022-10-10 04:09:16] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-10 04:09:16] ...optimizers/trijoint.py.151: total_norm: 14.670430 [I 2022-10-10 04:09:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 260/2304 [I 2022-10-10 04:09:24] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:51 [I 2022-10-10 04:09:24] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00326 [I 2022-10-10 04:09:24] ...trap/engines/engine.py.230: loss: 0.10918 [I 2022-10-10 04:09:24] ...optimizers/trijoint.py.151: total_norm: 10.299865 [I 2022-10-10 04:09:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 270/2304 [I 2022-10-10 04:09:33] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:47 [I 2022-10-10 04:09:33] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00301 [I 2022-10-10 04:09:33] ...trap/engines/engine.py.230: loss: 0.21427 [I 2022-10-10 04:09:33] ...optimizers/trijoint.py.151: total_norm: 15.768858 [I 2022-10-10 04:09:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 280/2304 [I 2022-10-10 04:09:41] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:57 [I 2022-10-10 04:09:41] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00339 [I 2022-10-10 04:09:41] ...trap/engines/engine.py.230: loss: 0.19454 [I 2022-10-10 04:09:41] ...optimizers/trijoint.py.151: total_norm: 25.648838 [I 2022-10-10 04:09:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 290/2304 [I 2022-10-10 04:09:50] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:40 [I 2022-10-10 04:09:50] ...trap/engines/engine.py.229: process: 0.85523 | load: 0.00312 [I 2022-10-10 04:09:50] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-10 04:09:50] ...optimizers/trijoint.py.151: total_norm: 27.172359 [I 2022-10-10 04:09:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 300/2304 [I 2022-10-10 04:09:58] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:26 [I 2022-10-10 04:09:58] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00311 [I 2022-10-10 04:09:58] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-10 04:09:58] ...optimizers/trijoint.py.151: total_norm: 9.838494 [I 2022-10-10 04:10:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 310/2304 [I 2022-10-10 04:10:07] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:22 [I 2022-10-10 04:10:07] ...trap/engines/engine.py.229: process: 0.85213 | load: 0.00339 [I 2022-10-10 04:10:07] ...trap/engines/engine.py.230: loss: 0.11672 [I 2022-10-10 04:10:07] ...optimizers/trijoint.py.151: total_norm: 14.023548 [I 2022-10-10 04:10:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 320/2304 [I 2022-10-10 04:10:16] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:20 [I 2022-10-10 04:10:16] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00312 [I 2022-10-10 04:10:16] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-10 04:10:16] ...optimizers/trijoint.py.151: total_norm: 18.198704 [I 2022-10-10 04:10:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 330/2304 [I 2022-10-10 04:10:24] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:11 [I 2022-10-10 04:10:24] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00324 [I 2022-10-10 04:10:24] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-10 04:10:24] ...optimizers/trijoint.py.151: total_norm: 21.203180 [I 2022-10-10 04:10:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 340/2304 [I 2022-10-10 04:10:33] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:28:02 [I 2022-10-10 04:10:33] ...trap/engines/engine.py.229: process: 0.86756 | load: 0.00314 [I 2022-10-10 04:10:33] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-10 04:10:33] ...optimizers/trijoint.py.151: total_norm: 43.278526 [I 2022-10-10 04:10:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 350/2304 [I 2022-10-10 04:10:41] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:57 [I 2022-10-10 04:10:41] ...trap/engines/engine.py.229: process: 0.86476 | load: 0.00316 [I 2022-10-10 04:10:41] ...trap/engines/engine.py.230: loss: 0.18491 [I 2022-10-10 04:10:41] ...optimizers/trijoint.py.151: total_norm: 54.706165 [I 2022-10-10 04:10:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 360/2304 [I 2022-10-10 04:10:50] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:28:04 [I 2022-10-10 04:10:50] ...trap/engines/engine.py.229: process: 0.85709 | load: 0.00313 [I 2022-10-10 04:10:50] ...trap/engines/engine.py.230: loss: 0.11923 [I 2022-10-10 04:10:50] ...optimizers/trijoint.py.151: total_norm: 11.401057 [I 2022-10-10 04:10:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 370/2304 [I 2022-10-10 04:10:59] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:36 [I 2022-10-10 04:10:59] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00331 [I 2022-10-10 04:10:59] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-10 04:10:59] ...optimizers/trijoint.py.151: total_norm: 11.827181 [I 2022-10-10 04:11:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 380/2304 [I 2022-10-10 04:11:07] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:30 [I 2022-10-10 04:11:07] ...trap/engines/engine.py.229: process: 0.86124 | load: 0.00327 [I 2022-10-10 04:11:07] ...trap/engines/engine.py.230: loss: 0.14658 [I 2022-10-10 04:11:07] ...optimizers/trijoint.py.151: total_norm: 14.186409 [I 2022-10-10 04:11:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 390/2304 [I 2022-10-10 04:11:16] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:11 [I 2022-10-10 04:11:16] ...trap/engines/engine.py.229: process: 0.83549 | load: 0.00311 [I 2022-10-10 04:11:16] ...trap/engines/engine.py.230: loss: 0.09056 [I 2022-10-10 04:11:16] ...optimizers/trijoint.py.151: total_norm: 9.952862 [I 2022-10-10 04:11:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 400/2304 [I 2022-10-10 04:11:24] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:04 [I 2022-10-10 04:11:24] ...trap/engines/engine.py.229: process: 0.86311 | load: 0.00310 [I 2022-10-10 04:11:24] ...trap/engines/engine.py.230: loss: 0.15688 [I 2022-10-10 04:11:24] ...optimizers/trijoint.py.151: total_norm: 14.759273 [I 2022-10-10 04:11:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 410/2304 [I 2022-10-10 04:11:33] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:59 [I 2022-10-10 04:11:33] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00312 [I 2022-10-10 04:11:33] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-10 04:11:33] ...optimizers/trijoint.py.151: total_norm: 13.794606 [I 2022-10-10 04:11:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 420/2304 [I 2022-10-10 04:11:42] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:51 [I 2022-10-10 04:11:42] ...trap/engines/engine.py.229: process: 0.84184 | load: 0.00312 [I 2022-10-10 04:11:42] ...trap/engines/engine.py.230: loss: 0.09632 [I 2022-10-10 04:11:42] ...optimizers/trijoint.py.151: total_norm: 5.790615 [I 2022-10-10 04:11:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 430/2304 [I 2022-10-10 04:11:50] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:39 [I 2022-10-10 04:11:50] ...trap/engines/engine.py.229: process: 0.84938 | load: 0.00311 [I 2022-10-10 04:11:50] ...trap/engines/engine.py.230: loss: 0.16636 [I 2022-10-10 04:11:50] ...optimizers/trijoint.py.151: total_norm: 12.675626 [I 2022-10-10 04:11:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 440/2304 [I 2022-10-10 04:11:59] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:37 [I 2022-10-10 04:11:59] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00313 [I 2022-10-10 04:11:59] ...trap/engines/engine.py.230: loss: 0.11220 [I 2022-10-10 04:11:59] ...optimizers/trijoint.py.151: total_norm: 14.502343 [I 2022-10-10 04:12:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 450/2304 [I 2022-10-10 04:12:07] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:20 [I 2022-10-10 04:12:07] ...trap/engines/engine.py.229: process: 0.85044 | load: 0.00310 [I 2022-10-10 04:12:07] ...trap/engines/engine.py.230: loss: 0.21358 [I 2022-10-10 04:12:07] ...optimizers/trijoint.py.151: total_norm: 34.821518 [I 2022-10-10 04:12:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 460/2304 [I 2022-10-10 04:12:16] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:07 [I 2022-10-10 04:12:16] ...trap/engines/engine.py.229: process: 0.85203 | load: 0.00312 [I 2022-10-10 04:12:16] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-10 04:12:16] ...optimizers/trijoint.py.151: total_norm: 53.183006 [I 2022-10-10 04:12:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 470/2304 [I 2022-10-10 04:12:24] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:26:08 [I 2022-10-10 04:12:24] ...trap/engines/engine.py.229: process: 0.85771 | load: 0.00311 [I 2022-10-10 04:12:24] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-10 04:12:24] ...optimizers/trijoint.py.151: total_norm: 15.816180 [I 2022-10-10 04:12:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 480/2304 [I 2022-10-10 04:12:33] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:54 [I 2022-10-10 04:12:33] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00312 [I 2022-10-10 04:12:33] ...trap/engines/engine.py.230: loss: 0.14375 [I 2022-10-10 04:12:33] ...optimizers/trijoint.py.151: total_norm: 17.207220 [I 2022-10-10 04:12:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 490/2304 [I 2022-10-10 04:12:41] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:53 [I 2022-10-10 04:12:41] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00311 [I 2022-10-10 04:12:41] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-10 04:12:41] ...optimizers/trijoint.py.151: total_norm: 11.562132 [I 2022-10-10 04:12:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 500/2304 [I 2022-10-10 04:12:50] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:40 [I 2022-10-10 04:12:50] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00327 [I 2022-10-10 04:12:50] ...trap/engines/engine.py.230: loss: 0.17679 [I 2022-10-10 04:12:50] ...optimizers/trijoint.py.151: total_norm: 19.447929 [I 2022-10-10 04:12:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 510/2304 [I 2022-10-10 04:12:59] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:22 [I 2022-10-10 04:12:59] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00312 [I 2022-10-10 04:12:59] ...trap/engines/engine.py.230: loss: 0.17569 [I 2022-10-10 04:12:59] ...optimizers/trijoint.py.151: total_norm: 10.257289 [I 2022-10-10 04:13:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 520/2304 [I 2022-10-10 04:13:07] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:30 [I 2022-10-10 04:13:07] ...trap/engines/engine.py.229: process: 0.85875 | load: 0.00311 [I 2022-10-10 04:13:07] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-10 04:13:07] ...optimizers/trijoint.py.151: total_norm: 24.085823 [I 2022-10-10 04:13:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 530/2304 [I 2022-10-10 04:13:16] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:18 [I 2022-10-10 04:13:16] ...trap/engines/engine.py.229: process: 0.86681 | load: 0.00310 [I 2022-10-10 04:13:16] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-10 04:13:16] ...optimizers/trijoint.py.151: total_norm: 40.461189 [I 2022-10-10 04:13:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 540/2304 [I 2022-10-10 04:13:24] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:25:12 [I 2022-10-10 04:13:24] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00326 [I 2022-10-10 04:13:24] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-10 04:13:24] ...optimizers/trijoint.py.151: total_norm: 20.949724 [I 2022-10-10 04:13:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 550/2304 [I 2022-10-10 04:13:33] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:00 [I 2022-10-10 04:13:33] ...trap/engines/engine.py.229: process: 0.86347 | load: 0.00313 [I 2022-10-10 04:13:33] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-10 04:13:33] ...optimizers/trijoint.py.151: total_norm: 35.952946 [I 2022-10-10 04:13:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 560/2304 [I 2022-10-10 04:13:41] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:24:52 [I 2022-10-10 04:13:41] ...trap/engines/engine.py.229: process: 0.86146 | load: 0.00310 [I 2022-10-10 04:13:41] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-10 04:13:41] ...optimizers/trijoint.py.151: total_norm: 15.191462 [I 2022-10-10 04:13:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 570/2304 [I 2022-10-10 04:13:50] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:30 [I 2022-10-10 04:13:50] ...trap/engines/engine.py.229: process: 0.83956 | load: 0.00293 [I 2022-10-10 04:13:50] ...trap/engines/engine.py.230: loss: 0.15425 [I 2022-10-10 04:13:50] ...optimizers/trijoint.py.151: total_norm: 6.752390 [I 2022-10-10 04:13:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 580/2304 [I 2022-10-10 04:13:58] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:21 [I 2022-10-10 04:13:58] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00307 [I 2022-10-10 04:13:58] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-10 04:13:58] ...optimizers/trijoint.py.151: total_norm: 9.226686 [I 2022-10-10 04:14:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 590/2304 [I 2022-10-10 04:14:07] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:09 [I 2022-10-10 04:14:07] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00299 [I 2022-10-10 04:14:07] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-10 04:14:07] ...optimizers/trijoint.py.151: total_norm: 20.826302 [I 2022-10-10 04:14:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 600/2304 [I 2022-10-10 04:14:15] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:06 [I 2022-10-10 04:14:15] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00298 [I 2022-10-10 04:14:15] ...trap/engines/engine.py.230: loss: 0.11164 [I 2022-10-10 04:14:15] ...optimizers/trijoint.py.151: total_norm: 12.367562 [I 2022-10-10 04:14:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 610/2304 [I 2022-10-10 04:14:24] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:23:54 [I 2022-10-10 04:14:24] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00298 [I 2022-10-10 04:14:24] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-10 04:14:24] ...optimizers/trijoint.py.151: total_norm: 7.856725 [I 2022-10-10 04:14:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 620/2304 [I 2022-10-10 04:14:32] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:37 [I 2022-10-10 04:14:32] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00296 [I 2022-10-10 04:14:32] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-10 04:14:32] ...optimizers/trijoint.py.151: total_norm: 35.061462 [I 2022-10-10 04:14:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 630/2304 [I 2022-10-10 04:14:41] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:29 [I 2022-10-10 04:14:41] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00311 [I 2022-10-10 04:14:41] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-10 04:14:41] ...optimizers/trijoint.py.151: total_norm: 20.823006 [I 2022-10-10 04:14:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 640/2304 [I 2022-10-10 04:14:49] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:34 [I 2022-10-10 04:14:49] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00303 [I 2022-10-10 04:14:49] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-10 04:14:49] ...optimizers/trijoint.py.151: total_norm: 31.104067 [I 2022-10-10 04:14:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 650/2304 [I 2022-10-10 04:14:58] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:27 [I 2022-10-10 04:14:58] ...trap/engines/engine.py.229: process: 0.85488 | load: 0.00302 [I 2022-10-10 04:14:58] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-10 04:14:58] ...optimizers/trijoint.py.151: total_norm: 14.086955 [I 2022-10-10 04:15:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 660/2304 [I 2022-10-10 04:15:06] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:05 [I 2022-10-10 04:15:06] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00294 [I 2022-10-10 04:15:06] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-10 04:15:06] ...optimizers/trijoint.py.151: total_norm: 17.557606 [I 2022-10-10 04:15:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 670/2304 [I 2022-10-10 04:15:15] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:04 [I 2022-10-10 04:15:15] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00299 [I 2022-10-10 04:15:15] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-10 04:15:15] ...optimizers/trijoint.py.151: total_norm: 14.458340 [I 2022-10-10 04:15:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 680/2304 [I 2022-10-10 04:15:23] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:22:48 [I 2022-10-10 04:15:23] ...trap/engines/engine.py.229: process: 0.84284 | load: 0.00302 [I 2022-10-10 04:15:23] ...trap/engines/engine.py.230: loss: 0.11003 [I 2022-10-10 04:15:23] ...optimizers/trijoint.py.151: total_norm: 7.674346 [I 2022-10-10 04:15:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 690/2304 [I 2022-10-10 04:15:32] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:50 [I 2022-10-10 04:15:32] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00297 [I 2022-10-10 04:15:32] ...trap/engines/engine.py.230: loss: 0.15049 [I 2022-10-10 04:15:32] ...optimizers/trijoint.py.151: total_norm: 11.965436 [I 2022-10-10 04:15:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 700/2304 [I 2022-10-10 04:15:40] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:44 [I 2022-10-10 04:15:40] ...trap/engines/engine.py.229: process: 0.86496 | load: 0.00302 [I 2022-10-10 04:15:40] ...trap/engines/engine.py.230: loss: 0.25564 [I 2022-10-10 04:15:40] ...optimizers/trijoint.py.151: total_norm: 20.188961 [I 2022-10-10 04:15:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 710/2304 [I 2022-10-10 04:15:49] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:32 [I 2022-10-10 04:15:49] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00298 [I 2022-10-10 04:15:49] ...trap/engines/engine.py.230: loss: 0.16092 [I 2022-10-10 04:15:49] ...optimizers/trijoint.py.151: total_norm: 10.578565 [I 2022-10-10 04:15:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 720/2304 [I 2022-10-10 04:15:57] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:31 [I 2022-10-10 04:15:57] ...trap/engines/engine.py.229: process: 0.87050 | load: 0.00299 [I 2022-10-10 04:15:57] ...trap/engines/engine.py.230: loss: 0.12426 [I 2022-10-10 04:15:57] ...optimizers/trijoint.py.151: total_norm: 15.589300 [I 2022-10-10 04:16:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 730/2304 [I 2022-10-10 04:16:06] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:10 [I 2022-10-10 04:16:06] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00301 [I 2022-10-10 04:16:06] ...trap/engines/engine.py.230: loss: 0.14258 [I 2022-10-10 04:16:06] ...optimizers/trijoint.py.151: total_norm: 9.729703 [I 2022-10-10 04:16:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 740/2304 [I 2022-10-10 04:16:14] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:04 [I 2022-10-10 04:16:14] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00302 [I 2022-10-10 04:16:14] ...trap/engines/engine.py.230: loss: 0.15076 [I 2022-10-10 04:16:14] ...optimizers/trijoint.py.151: total_norm: 11.535390 [I 2022-10-10 04:16:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 750/2304 [I 2022-10-10 04:16:23] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:01 [I 2022-10-10 04:16:23] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00299 [I 2022-10-10 04:16:23] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-10 04:16:23] ...optimizers/trijoint.py.151: total_norm: 14.953660 [I 2022-10-10 04:16:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 760/2304 [I 2022-10-10 04:16:31] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:51 [I 2022-10-10 04:16:31] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00297 [I 2022-10-10 04:16:31] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-10 04:16:31] ...optimizers/trijoint.py.151: total_norm: 16.475452 [I 2022-10-10 04:16:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 770/2304 [I 2022-10-10 04:16:40] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:44 [I 2022-10-10 04:16:40] ...trap/engines/engine.py.229: process: 0.85720 | load: 0.00310 [I 2022-10-10 04:16:40] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-10 04:16:40] ...optimizers/trijoint.py.151: total_norm: 19.406357 [I 2022-10-10 04:16:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 780/2304 [I 2022-10-10 04:16:48] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:36 [I 2022-10-10 04:16:48] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00309 [I 2022-10-10 04:16:48] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-10 04:16:48] ...optimizers/trijoint.py.151: total_norm: 29.965712 [I 2022-10-10 04:16:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 790/2304 [I 2022-10-10 04:16:57] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:36 [I 2022-10-10 04:16:57] ...trap/engines/engine.py.229: process: 0.85755 | load: 0.00313 [I 2022-10-10 04:16:57] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-10 04:16:57] ...optimizers/trijoint.py.151: total_norm: 27.361212 [I 2022-10-10 04:17:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 800/2304 [I 2022-10-10 04:17:06] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:27 [I 2022-10-10 04:17:06] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00308 [I 2022-10-10 04:17:06] ...trap/engines/engine.py.230: loss: 0.17434 [I 2022-10-10 04:17:06] ...optimizers/trijoint.py.151: total_norm: 17.242176 [I 2022-10-10 04:17:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 810/2304 [I 2022-10-10 04:17:14] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:19 [I 2022-10-10 04:17:14] ...trap/engines/engine.py.229: process: 0.86402 | load: 0.00311 [I 2022-10-10 04:17:14] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-10 04:17:14] ...optimizers/trijoint.py.151: total_norm: 17.137766 [I 2022-10-10 04:17:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 820/2304 [I 2022-10-10 04:17:23] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:09 [I 2022-10-10 04:17:23] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00312 [I 2022-10-10 04:17:23] ...trap/engines/engine.py.230: loss: 0.12967 [I 2022-10-10 04:17:23] ...optimizers/trijoint.py.151: total_norm: 15.517758 [I 2022-10-10 04:17:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 830/2304 [I 2022-10-10 04:17:31] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:48 [I 2022-10-10 04:17:31] ...trap/engines/engine.py.229: process: 0.83244 | load: 0.00315 [I 2022-10-10 04:17:31] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-10 04:17:31] ...optimizers/trijoint.py.151: total_norm: 15.597681 [I 2022-10-10 04:17:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 840/2304 [I 2022-10-10 04:17:40] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:54 [I 2022-10-10 04:17:40] ...trap/engines/engine.py.229: process: 0.86514 | load: 0.00336 [I 2022-10-10 04:17:40] ...trap/engines/engine.py.230: loss: 0.20329 [I 2022-10-10 04:17:40] ...optimizers/trijoint.py.151: total_norm: 18.479172 [I 2022-10-10 04:17:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 850/2304 [I 2022-10-10 04:17:48] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:39 [I 2022-10-10 04:17:48] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00322 [I 2022-10-10 04:17:48] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-10 04:17:48] ...optimizers/trijoint.py.151: total_norm: 22.624979 [I 2022-10-10 04:17:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 860/2304 [I 2022-10-10 04:17:57] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:37 [I 2022-10-10 04:17:57] ...trap/engines/engine.py.229: process: 0.86089 | load: 0.00321 [I 2022-10-10 04:17:57] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-10 04:17:57] ...optimizers/trijoint.py.151: total_norm: 10.106064 [I 2022-10-10 04:18:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 870/2304 [I 2022-10-10 04:18:06] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:22 [I 2022-10-10 04:18:06] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00311 [I 2022-10-10 04:18:06] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-10 04:18:06] ...optimizers/trijoint.py.151: total_norm: 24.369825 [I 2022-10-10 04:18:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 880/2304 [I 2022-10-10 04:18:14] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:17 [I 2022-10-10 04:18:14] ...trap/engines/engine.py.229: process: 0.84060 | load: 0.00313 [I 2022-10-10 04:18:14] ...trap/engines/engine.py.230: loss: 0.12036 [I 2022-10-10 04:18:14] ...optimizers/trijoint.py.151: total_norm: 14.368630 [I 2022-10-10 04:18:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 890/2304 [I 2022-10-10 04:18:23] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:04 [I 2022-10-10 04:18:23] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00311 [I 2022-10-10 04:18:23] ...trap/engines/engine.py.230: loss: 0.21500 [I 2022-10-10 04:18:23] ...optimizers/trijoint.py.151: total_norm: 26.120211 [I 2022-10-10 04:18:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 900/2304 [I 2022-10-10 04:18:31] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:55 [I 2022-10-10 04:18:31] ...trap/engines/engine.py.229: process: 0.84781 | load: 0.00314 [I 2022-10-10 04:18:31] ...trap/engines/engine.py.230: loss: 0.12567 [I 2022-10-10 04:18:31] ...optimizers/trijoint.py.151: total_norm: 16.443871 [I 2022-10-10 04:18:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 910/2304 [I 2022-10-10 04:18:40] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:54 [I 2022-10-10 04:18:40] ...trap/engines/engine.py.229: process: 0.87045 | load: 0.00310 [I 2022-10-10 04:18:40] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-10 04:18:40] ...optimizers/trijoint.py.151: total_norm: 15.978523 [I 2022-10-10 04:18:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 920/2304 [I 2022-10-10 04:18:48] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:34 [I 2022-10-10 04:18:48] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00311 [I 2022-10-10 04:18:48] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-10 04:18:48] ...optimizers/trijoint.py.151: total_norm: 23.701035 [I 2022-10-10 04:18:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 930/2304 [I 2022-10-10 04:18:57] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:29 [I 2022-10-10 04:18:57] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00311 [I 2022-10-10 04:18:57] ...trap/engines/engine.py.230: loss: 0.19552 [I 2022-10-10 04:18:57] ...optimizers/trijoint.py.151: total_norm: 55.355602 [I 2022-10-10 04:19:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 940/2304 [I 2022-10-10 04:19:05] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:21 [I 2022-10-10 04:19:05] ...trap/engines/engine.py.229: process: 0.85776 | load: 0.00310 [I 2022-10-10 04:19:05] ...trap/engines/engine.py.230: loss: 0.15063 [I 2022-10-10 04:19:05] ...optimizers/trijoint.py.151: total_norm: 21.293207 [I 2022-10-10 04:19:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 950/2304 [I 2022-10-10 04:19:14] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:10 [I 2022-10-10 04:19:14] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00299 [I 2022-10-10 04:19:14] ...trap/engines/engine.py.230: loss: 0.21602 [I 2022-10-10 04:19:14] ...optimizers/trijoint.py.151: total_norm: 15.334747 [I 2022-10-10 04:19:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 960/2304 [I 2022-10-10 04:19:22] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:10 [I 2022-10-10 04:19:22] ...trap/engines/engine.py.229: process: 0.86229 | load: 0.00300 [I 2022-10-10 04:19:22] ...trap/engines/engine.py.230: loss: 0.10231 [I 2022-10-10 04:19:22] ...optimizers/trijoint.py.151: total_norm: 10.615806 [I 2022-10-10 04:19:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 970/2304 [I 2022-10-10 04:19:31] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:53 [I 2022-10-10 04:19:31] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00314 [I 2022-10-10 04:19:31] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-10 04:19:31] ...optimizers/trijoint.py.151: total_norm: 17.109463 [I 2022-10-10 04:19:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 980/2304 [I 2022-10-10 04:19:39] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:48 [I 2022-10-10 04:19:39] ...trap/engines/engine.py.229: process: 0.85219 | load: 0.00308 [I 2022-10-10 04:19:39] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-10 04:19:39] ...optimizers/trijoint.py.151: total_norm: 29.244492 [I 2022-10-10 04:19:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 990/2304 [I 2022-10-10 04:19:48] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:43 [I 2022-10-10 04:19:48] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00319 [I 2022-10-10 04:19:48] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-10 04:19:48] ...optimizers/trijoint.py.151: total_norm: 25.427372 [I 2022-10-10 04:19:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1000/2304 [I 2022-10-10 04:19:57] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:34 [I 2022-10-10 04:19:57] ...trap/engines/engine.py.229: process: 0.84370 | load: 0.00313 [I 2022-10-10 04:19:57] ...trap/engines/engine.py.230: loss: 0.09911 [I 2022-10-10 04:19:57] ...optimizers/trijoint.py.151: total_norm: 8.485934 [I 2022-10-10 04:20:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 1010/2304 [I 2022-10-10 04:20:05] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:30 [I 2022-10-10 04:20:05] ...trap/engines/engine.py.229: process: 0.86855 | load: 0.00311 [I 2022-10-10 04:20:05] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-10 04:20:05] ...optimizers/trijoint.py.151: total_norm: 10.863727 [I 2022-10-10 04:20:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1020/2304 [I 2022-10-10 04:20:14] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:28 [I 2022-10-10 04:20:14] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00315 [I 2022-10-10 04:20:14] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-10 04:20:14] ...optimizers/trijoint.py.151: total_norm: 30.100248 [I 2022-10-10 04:20:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1030/2304 [I 2022-10-10 04:20:22] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:07 [I 2022-10-10 04:20:22] ...trap/engines/engine.py.229: process: 0.85750 | load: 0.00315 [I 2022-10-10 04:20:22] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-10 04:20:22] ...optimizers/trijoint.py.151: total_norm: 14.120889 [I 2022-10-10 04:20:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1040/2304 [I 2022-10-10 04:20:31] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:17:57 [I 2022-10-10 04:20:31] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00298 [I 2022-10-10 04:20:31] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-10 04:20:31] ...optimizers/trijoint.py.151: total_norm: 12.922923 [I 2022-10-10 04:20:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1050/2304 [I 2022-10-10 04:20:39] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:46 [I 2022-10-10 04:20:39] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00325 [I 2022-10-10 04:20:39] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-10 04:20:39] ...optimizers/trijoint.py.151: total_norm: 11.489463 [I 2022-10-10 04:20:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 1060/2304 [I 2022-10-10 04:20:48] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:39 [I 2022-10-10 04:20:48] ...trap/engines/engine.py.229: process: 0.85229 | load: 0.00310 [I 2022-10-10 04:20:48] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-10 04:20:48] ...optimizers/trijoint.py.151: total_norm: 21.101362 [I 2022-10-10 04:20:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1070/2304 [I 2022-10-10 04:20:56] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:33 [I 2022-10-10 04:20:56] ...trap/engines/engine.py.229: process: 0.85100 | load: 0.00320 [I 2022-10-10 04:20:56] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-10 04:20:56] ...optimizers/trijoint.py.151: total_norm: 16.833187 [I 2022-10-10 04:21:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 1080/2304 [I 2022-10-10 04:21:05] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:25 [I 2022-10-10 04:21:05] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00312 [I 2022-10-10 04:21:05] ...trap/engines/engine.py.230: loss: 0.12174 [I 2022-10-10 04:21:05] ...optimizers/trijoint.py.151: total_norm: 15.365783 [I 2022-10-10 04:21:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1090/2304 [I 2022-10-10 04:21:14] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:09 [I 2022-10-10 04:21:14] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00298 [I 2022-10-10 04:21:14] ...trap/engines/engine.py.230: loss: 0.14134 [I 2022-10-10 04:21:14] ...optimizers/trijoint.py.151: total_norm: 38.151279 [I 2022-10-10 04:21:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1100/2304 [I 2022-10-10 04:21:22] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:00 [I 2022-10-10 04:21:22] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00302 [I 2022-10-10 04:21:22] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-10 04:21:22] ...optimizers/trijoint.py.151: total_norm: 14.512269 [I 2022-10-10 04:21:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1110/2304 [I 2022-10-10 04:21:31] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:45 [I 2022-10-10 04:21:31] ...trap/engines/engine.py.229: process: 0.81626 | load: 0.00304 [I 2022-10-10 04:21:31] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-10 04:21:31] ...optimizers/trijoint.py.151: total_norm: 14.565802 [I 2022-10-10 04:21:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1120/2304 [I 2022-10-10 04:21:39] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:38 [I 2022-10-10 04:21:39] ...trap/engines/engine.py.229: process: 0.85413 | load: 0.00298 [I 2022-10-10 04:21:39] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-10 04:21:39] ...optimizers/trijoint.py.151: total_norm: 10.436954 [I 2022-10-10 04:21:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1130/2304 [I 2022-10-10 04:21:47] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:33 [I 2022-10-10 04:21:47] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00298 [I 2022-10-10 04:21:47] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-10 04:21:47] ...optimizers/trijoint.py.151: total_norm: 11.982003 [I 2022-10-10 04:21:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1140/2304 [I 2022-10-10 04:21:56] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:26 [I 2022-10-10 04:21:56] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00302 [I 2022-10-10 04:21:56] ...trap/engines/engine.py.230: loss: 0.28567 [I 2022-10-10 04:21:56] ...optimizers/trijoint.py.151: total_norm: 30.468431 [I 2022-10-10 04:22:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1150/2304 [I 2022-10-10 04:22:04] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:21 [I 2022-10-10 04:22:04] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00302 [I 2022-10-10 04:22:04] ...trap/engines/engine.py.230: loss: 0.15583 [I 2022-10-10 04:22:04] ...optimizers/trijoint.py.151: total_norm: 14.071517 [I 2022-10-10 04:22:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 1160/2304 [I 2022-10-10 04:22:13] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:11 [I 2022-10-10 04:22:13] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00301 [I 2022-10-10 04:22:13] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-10 04:22:13] ...optimizers/trijoint.py.151: total_norm: 16.578007 [I 2022-10-10 04:22:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1170/2304 [I 2022-10-10 04:22:21] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:57 [I 2022-10-10 04:22:21] ...trap/engines/engine.py.229: process: 0.83692 | load: 0.00300 [I 2022-10-10 04:22:21] ...trap/engines/engine.py.230: loss: 0.12132 [I 2022-10-10 04:22:21] ...optimizers/trijoint.py.151: total_norm: 13.952599 [I 2022-10-10 04:22:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1180/2304 [I 2022-10-10 04:22:30] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:16:01 [I 2022-10-10 04:22:30] ...trap/engines/engine.py.229: process: 0.86615 | load: 0.00301 [I 2022-10-10 04:22:30] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-10 04:22:30] ...optimizers/trijoint.py.151: total_norm: 23.278547 [I 2022-10-10 04:22:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1190/2304 [I 2022-10-10 04:22:39] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:47 [I 2022-10-10 04:22:39] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00300 [I 2022-10-10 04:22:39] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-10 04:22:39] ...optimizers/trijoint.py.151: total_norm: 10.387527 [I 2022-10-10 04:22:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1200/2304 [I 2022-10-10 04:22:47] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:29 [I 2022-10-10 04:22:47] ...trap/engines/engine.py.229: process: 0.82676 | load: 0.00300 [I 2022-10-10 04:22:47] ...trap/engines/engine.py.230: loss: 0.10203 [I 2022-10-10 04:22:47] ...optimizers/trijoint.py.151: total_norm: 14.067368 [I 2022-10-10 04:22:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1210/2304 [I 2022-10-10 04:22:56] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:25 [I 2022-10-10 04:22:56] ...trap/engines/engine.py.229: process: 0.84863 | load: 0.00299 [I 2022-10-10 04:22:56] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-10 04:22:56] ...optimizers/trijoint.py.151: total_norm: 23.831097 [I 2022-10-10 04:23:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1220/2304 [I 2022-10-10 04:23:04] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:12 [I 2022-10-10 04:23:04] ...trap/engines/engine.py.229: process: 0.84017 | load: 0.00298 [I 2022-10-10 04:23:04] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-10 04:23:04] ...optimizers/trijoint.py.151: total_norm: 9.783978 [I 2022-10-10 04:23:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1230/2304 [I 2022-10-10 04:23:12] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:15:10 [I 2022-10-10 04:23:12] ...trap/engines/engine.py.229: process: 0.85481 | load: 0.00298 [I 2022-10-10 04:23:12] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-10 04:23:12] ...optimizers/trijoint.py.151: total_norm: 9.096892 [I 2022-10-10 04:23:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1240/2304 [I 2022-10-10 04:23:21] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:02 [I 2022-10-10 04:23:21] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00297 [I 2022-10-10 04:23:21] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-10 04:23:21] ...optimizers/trijoint.py.151: total_norm: 19.255949 [I 2022-10-10 04:23:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1250/2304 [I 2022-10-10 04:23:29] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:00 [I 2022-10-10 04:23:29] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00314 [I 2022-10-10 04:23:29] ...trap/engines/engine.py.230: loss: 0.16119 [I 2022-10-10 04:23:29] ...optimizers/trijoint.py.151: total_norm: 22.059484 [I 2022-10-10 04:23:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1260/2304 [I 2022-10-10 04:23:38] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:53 [I 2022-10-10 04:23:38] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00312 [I 2022-10-10 04:23:38] ...trap/engines/engine.py.230: loss: 0.18319 [I 2022-10-10 04:23:38] ...optimizers/trijoint.py.151: total_norm: 22.043985 [I 2022-10-10 04:23:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1270/2304 [I 2022-10-10 04:23:47] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:48 [I 2022-10-10 04:23:47] ...trap/engines/engine.py.229: process: 0.86210 | load: 0.00312 [I 2022-10-10 04:23:47] ...trap/engines/engine.py.230: loss: 0.13913 [I 2022-10-10 04:23:47] ...optimizers/trijoint.py.151: total_norm: 11.651860 [I 2022-10-10 04:23:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1280/2304 [I 2022-10-10 04:23:55] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:35 [I 2022-10-10 04:23:55] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00314 [I 2022-10-10 04:23:55] ...trap/engines/engine.py.230: loss: 0.15011 [I 2022-10-10 04:23:55] ...optimizers/trijoint.py.151: total_norm: 21.970896 [I 2022-10-10 04:24:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1290/2304 [I 2022-10-10 04:24:04] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:22 [I 2022-10-10 04:24:04] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00311 [I 2022-10-10 04:24:04] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-10 04:24:04] ...optimizers/trijoint.py.151: total_norm: 14.459058 [I 2022-10-10 08:44:00] ...ytorch/bootstrap/run.py.73: Saving environment info [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: options: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dataset: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: aux_kw_path: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: batch_sampler: triplet_classif [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: batch_size: 100 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: database: lmdb [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: eval_split: val [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: freq_mismatch: 0.0 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: image_from: database [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: import: recipe1m.datasets.factory [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: kw_path: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: recipe1m [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_classes: 1048 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_threads: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: randkw_p: 0.3 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: randkw_p_aux: 0.5 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: tokenized_raw_text: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: train_split: train [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: use_vcs: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: engine: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: logger [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_epochs: 120 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: print_freq: 10 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max'] [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: exp: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: resume: last [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: misc: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cuda: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: data_parrallel: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: logs_name: None [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: overrite: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: seed: 1338 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: model: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: criterion: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: tri_cross_encoder: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: itm_loss_weight: 1 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: keep_background: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: retrieval_strategy: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: aggregation: valid [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: margin: 0.05 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: margin_params: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: increment: 0.005 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: increment_margin: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: max_margin: 0.3 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: triplet [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_samples: 9999 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: sampling: max_negative [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: substrategy: ['IRR', 'RII', 'SIRR', 'SRII'] [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: substrategy_weights: [1.0, 1.0, 0.1, 0.1] [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: import: recipe1m.models.factory [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: metric: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: k_test: 10 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_bags: 10 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: nb_matchs_per_bag: 1000 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: trijoint: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: network: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: activations: ['tanh', 'normalize'] [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: aux_kwords: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: aux_kwords_encoder: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: avg_concat: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cat_pos: 0 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: class_attention_fus: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cls_token_fus: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_decoder_image: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_decoder_recipe: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_attention: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_decoder: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: double_decoder: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: get_tokens: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: load_backbones: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_layers: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: query_img: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: cross_transformer: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dim_emb: 1024 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dim_image_out: 2048 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dim_ingr_out: 300 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dim_instr_in: 1024 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: dim_instr_out: 1024 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: embedding_names: ['proj_image'] [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: freeze_im: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: freeze_rec: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: get_tokens_cross_decoder_recipe: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: hidden_size: 512 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: image_backbone_name: clip [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: image_res: 224 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: kwords_same_level: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: last_cat_aux: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_heads_cross: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_heads_cross_image: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_heads_single: 4 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_layers: 2 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_layers_cross: 2 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_layers_cross_image: 1 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: n_layers_single: 2 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: num_hidden_layers_kw: 2 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: old: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: recipe_encoder: tfoodvicha-transformer [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: text_encoder: bert-base-uncased [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: vision_width: 768 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: vit_all_tokens: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: with_ingrs: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: with_instrs: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: with_titles: True [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: with_classif: False [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: optimizer: [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: batch_size_factor: None [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: clip_grad: 8.0 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: import: recipe1m.optimizers.factory [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: lr: 1e-05 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: lr_img: 1e-06 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: name: trijoint_fixed_fine_tune [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: switch_epoch: 20 [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml [S 2022-10-10 08:44:08] ...ytorch/bootstrap/run.py.75: view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm'] [I 2022-10-10 08:44:09] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-a101', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64') [I 2022-10-10 08:44:09] ...ytorch/bootstrap/run.py.80: Available GPUs: [0, 1] [I 2022-10-10 08:44:09] ...strap/engines/factory.py.9: Creating engine... [I 2022-10-10 08:44:09] ...trap/datasets/factory.py.8: Creating dataset... [I 2022-10-10 08:44:26] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-10 08:44:30] ...m/datasets/recipe1m.py.165: randkw_p... 0.3 [I 2022-10-10 08:45:37] ...m/datasets/recipe1m.py.370: Calculate indices by class... [I 2022-10-10 08:45:39] ...m/datasets/recipe1m.py.375: Done! [I 2022-10-10 08:45:40] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-10 08:45:41] ...m/datasets/recipe1m.py.165: randkw_p... None [I 2022-10-10 08:46:52] ...rap/datasets/factory.py.18: Training will take place on trainset (238399 items) [I 2022-10-10 08:46:52] ...rap/datasets/factory.py.21: Evaluation will take place on valset (51119 items) [I 2022-10-10 08:46:52] ...strap/models/factory.py.21: Creating model... [I 2022-10-10 08:46:57] ...e_networks/networks.py.203: Loading CLIP Visual model... [I 2022-10-10 08:47:00] ...e_networks/networks.py.145: cuda [I 2022-10-10 08:47:01] ...networks/tfood_vicha.py.49: Load VCs encoder... [I 2022-10-10 08:47:03] .../criterions/trijoint.py.30: ignore_index=0 [I 2022-10-10 08:47:03] ...strap/models/factory.py.44: Enabling CUDA mode... [I 2022-10-10 08:47:09] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-10 08:47:09] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-10 08:47:09] ...otstrap/views/factory.py.8: Creating views... [I 2022-10-10 08:47:13] ...strap/engines/engine.py.98: Loading last checkpoint [I 2022-10-10 08:47:13] ...trap/engines/engine.py.440: Loading model... [I 2022-10-10 08:47:19] ...trap/engines/engine.py.446: Loading optimizer... [I 2022-10-10 08:47:24] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-10 08:47:24] ...trap/engines/engine.py.453: Loading engine... [I 2022-10-10 08:47:24] ...trap/engines/engine.py.125: Launching training procedures [I 2022-10-10 08:47:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 66 [I 2022-10-10 08:47:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 08:47:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 0/2304 [I 2022-10-10 08:47:32] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 5:09:08 [I 2022-10-10 08:47:32] ...trap/engines/engine.py.229: process: 8.05043 | load: 3.13636 [I 2022-10-10 08:47:32] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-10 08:47:32] ...optimizers/trijoint.py.151: total_norm: 10.522077 [I 2022-10-10 08:47:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 10/2304 [I 2022-10-10 08:47:41] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 1:03:00 [I 2022-10-10 08:47:41] ...trap/engines/engine.py.229: process: 0.87926 | load: 0.00615 [I 2022-10-10 08:47:41] ...trap/engines/engine.py.230: loss: 0.14141 [I 2022-10-10 08:47:41] ...optimizers/trijoint.py.151: total_norm: 9.014701 [I 2022-10-10 08:47:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 20/2304 [I 2022-10-10 08:47:49] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:35:17 [I 2022-10-10 08:47:49] ...trap/engines/engine.py.229: process: 0.82923 | load: 0.00512 [I 2022-10-10 08:47:49] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-10 08:47:49] ...optimizers/trijoint.py.151: total_norm: 16.143616 [I 2022-10-10 08:47:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 30/2304 [I 2022-10-10 08:47:58] ...trap/engines/engine.py.228: elapsed: 0:00:34 | left: 0:32:38 [I 2022-10-10 08:47:58] ...trap/engines/engine.py.229: process: 0.86836 | load: 0.00450 [I 2022-10-10 08:47:58] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-10 08:47:58] ...optimizers/trijoint.py.151: total_norm: 10.262050 [I 2022-10-10 08:48:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 40/2304 [I 2022-10-10 08:48:06] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:31:53 [I 2022-10-10 08:48:06] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00356 [I 2022-10-10 08:48:06] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-10 08:48:06] ...optimizers/trijoint.py.151: total_norm: 20.018753 [I 2022-10-10 08:48:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 50/2304 [I 2022-10-10 08:48:15] ...trap/engines/engine.py.228: elapsed: 0:00:51 | left: 0:31:41 [I 2022-10-10 08:48:15] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00381 [I 2022-10-10 08:48:15] ...trap/engines/engine.py.230: loss: 0.19617 [I 2022-10-10 08:48:15] ...optimizers/trijoint.py.151: total_norm: 10.655137 [I 2022-10-10 08:48:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 60/2304 [I 2022-10-10 08:48:23] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:31:34 [I 2022-10-10 08:48:23] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00383 [I 2022-10-10 08:48:23] ...trap/engines/engine.py.230: loss: 0.08824 [I 2022-10-10 08:48:23] ...optimizers/trijoint.py.151: total_norm: 12.033246 [I 2022-10-10 08:48:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 70/2304 [I 2022-10-10 08:48:32] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:31:36 [I 2022-10-10 08:48:32] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00326 [I 2022-10-10 08:48:32] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-10 08:48:32] ...optimizers/trijoint.py.151: total_norm: 11.707971 [I 2022-10-10 08:48:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 80/2304 [I 2022-10-10 08:48:40] ...trap/engines/engine.py.228: elapsed: 0:01:16 | left: 0:31:14 [I 2022-10-10 08:48:40] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00342 [I 2022-10-10 08:48:40] ...trap/engines/engine.py.230: loss: 0.21284 [I 2022-10-10 08:48:40] ...optimizers/trijoint.py.151: total_norm: 25.090818 [I 2022-10-10 08:48:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 90/2304 [I 2022-10-10 08:48:48] ...trap/engines/engine.py.228: elapsed: 0:01:24 | left: 0:31:16 [I 2022-10-10 08:48:48] ...trap/engines/engine.py.229: process: 0.84969 | load: 0.00350 [I 2022-10-10 08:48:48] ...trap/engines/engine.py.230: loss: 0.16844 [I 2022-10-10 08:48:48] ...optimizers/trijoint.py.151: total_norm: 39.680336 [I 2022-10-10 08:48:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 100/2304 [I 2022-10-10 08:48:57] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:30:54 [I 2022-10-10 08:48:57] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00326 [I 2022-10-10 08:48:57] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-10 08:48:57] ...optimizers/trijoint.py.151: total_norm: 11.564252 [I 2022-10-10 08:49:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 110/2304 [I 2022-10-10 08:49:05] ...trap/engines/engine.py.228: elapsed: 0:01:41 | left: 0:30:51 [I 2022-10-10 08:49:05] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00329 [I 2022-10-10 08:49:05] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-10 08:49:05] ...optimizers/trijoint.py.151: total_norm: 12.109223 [I 2022-10-10 08:49:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 120/2304 [I 2022-10-10 08:49:14] ...trap/engines/engine.py.228: elapsed: 0:01:50 | left: 0:30:48 [I 2022-10-10 08:49:14] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00313 [I 2022-10-10 08:49:14] ...trap/engines/engine.py.230: loss: 0.16142 [I 2022-10-10 08:49:14] ...optimizers/trijoint.py.151: total_norm: 12.308116 [I 2022-10-10 08:49:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 130/2304 [I 2022-10-10 08:49:22] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:30:30 [I 2022-10-10 08:49:22] ...trap/engines/engine.py.229: process: 0.83049 | load: 0.00319 [I 2022-10-10 08:49:22] ...trap/engines/engine.py.230: loss: 0.17185 [I 2022-10-10 08:49:22] ...optimizers/trijoint.py.151: total_norm: 17.103415 [I 2022-10-10 08:49:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 140/2304 [I 2022-10-10 08:49:31] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:30:22 [I 2022-10-10 08:49:31] ...trap/engines/engine.py.229: process: 0.84230 | load: 0.00310 [I 2022-10-10 08:49:31] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-10 08:49:31] ...optimizers/trijoint.py.151: total_norm: 9.619197 [I 2022-10-10 08:49:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 150/2304 [I 2022-10-10 08:49:39] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:30:07 [I 2022-10-10 08:49:39] ...trap/engines/engine.py.229: process: 0.84223 | load: 0.00317 [I 2022-10-10 08:49:39] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-10 08:49:39] ...optimizers/trijoint.py.151: total_norm: 18.796619 [I 2022-10-10 08:49:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 160/2304 [I 2022-10-10 08:49:48] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:30:12 [I 2022-10-10 08:49:48] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00316 [I 2022-10-10 08:49:48] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-10 08:49:48] ...optimizers/trijoint.py.151: total_norm: 9.995359 [I 2022-10-10 08:49:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 170/2304 [I 2022-10-10 08:49:56] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:30:03 [I 2022-10-10 08:49:56] ...trap/engines/engine.py.229: process: 0.83899 | load: 0.00313 [I 2022-10-10 08:49:56] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-10 08:49:56] ...optimizers/trijoint.py.151: total_norm: 15.356792 [I 2022-10-10 08:50:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 180/2304 [I 2022-10-10 08:50:05] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:30:01 [I 2022-10-10 08:50:05] ...trap/engines/engine.py.229: process: 0.85911 | load: 0.00324 [I 2022-10-10 08:50:05] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-10 08:50:05] ...optimizers/trijoint.py.151: total_norm: 13.880870 [I 2022-10-10 08:50:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 190/2304 [I 2022-10-10 08:50:13] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:29:42 [I 2022-10-10 08:50:13] ...trap/engines/engine.py.229: process: 0.83493 | load: 0.00300 [I 2022-10-10 08:50:13] ...trap/engines/engine.py.230: loss: 0.11505 [I 2022-10-10 08:50:13] ...optimizers/trijoint.py.151: total_norm: 11.214733 [I 2022-10-10 08:50:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 200/2304 [I 2022-10-10 08:50:22] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:29:44 [I 2022-10-10 08:50:22] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00306 [I 2022-10-10 08:50:22] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-10 08:50:22] ...optimizers/trijoint.py.151: total_norm: 14.985641 [I 2022-10-10 08:50:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 210/2304 [I 2022-10-10 08:50:30] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:29:36 [I 2022-10-10 08:50:30] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00321 [I 2022-10-10 08:50:30] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-10 08:50:30] ...optimizers/trijoint.py.151: total_norm: 22.053085 [I 2022-10-10 08:50:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 220/2304 [I 2022-10-10 08:50:38] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:29:18 [I 2022-10-10 08:50:38] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00315 [I 2022-10-10 08:50:38] ...trap/engines/engine.py.230: loss: 0.14733 [I 2022-10-10 08:50:38] ...optimizers/trijoint.py.151: total_norm: 12.361162 [I 2022-10-10 08:50:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 230/2304 [I 2022-10-10 08:50:47] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:29:05 [I 2022-10-10 08:50:47] ...trap/engines/engine.py.229: process: 0.84397 | load: 0.00303 [I 2022-10-10 08:50:47] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-10 08:50:47] ...optimizers/trijoint.py.151: total_norm: 11.622730 [I 2022-10-10 08:50:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 240/2304 [I 2022-10-10 08:50:55] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:28:51 [I 2022-10-10 08:50:55] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00293 [I 2022-10-10 08:50:55] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-10 08:50:55] ...optimizers/trijoint.py.151: total_norm: 13.649437 [I 2022-10-10 08:51:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 250/2304 [I 2022-10-10 08:51:04] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:29:02 [I 2022-10-10 08:51:04] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00298 [I 2022-10-10 08:51:04] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-10 08:51:04] ...optimizers/trijoint.py.151: total_norm: 14.670430 [I 2022-10-10 08:51:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 260/2304 [I 2022-10-10 08:51:12] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:28:54 [I 2022-10-10 08:51:12] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00300 [I 2022-10-10 08:51:12] ...trap/engines/engine.py.230: loss: 0.10918 [I 2022-10-10 08:51:12] ...optimizers/trijoint.py.151: total_norm: 10.299865 [I 2022-10-10 08:51:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 270/2304 [I 2022-10-10 08:51:21] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:28:48 [I 2022-10-10 08:51:21] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00292 [I 2022-10-10 08:51:21] ...trap/engines/engine.py.230: loss: 0.21427 [I 2022-10-10 08:51:21] ...optimizers/trijoint.py.151: total_norm: 15.768858 [I 2022-10-10 08:51:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 280/2304 [I 2022-10-10 08:51:29] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:28:39 [I 2022-10-10 08:51:29] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00302 [I 2022-10-10 08:51:29] ...trap/engines/engine.py.230: loss: 0.19454 [I 2022-10-10 08:51:29] ...optimizers/trijoint.py.151: total_norm: 25.648838 [I 2022-10-10 08:51:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 290/2304 [I 2022-10-10 08:51:38] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:28:15 [I 2022-10-10 08:51:38] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00291 [I 2022-10-10 08:51:38] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-10 08:51:38] ...optimizers/trijoint.py.151: total_norm: 27.172359 [I 2022-10-10 08:51:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 300/2304 [I 2022-10-10 08:51:46] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:11 [I 2022-10-10 08:51:46] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00298 [I 2022-10-10 08:51:46] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-10 08:51:46] ...optimizers/trijoint.py.151: total_norm: 9.838494 [I 2022-10-10 08:51:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 310/2304 [I 2022-10-10 08:51:55] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:27:54 [I 2022-10-10 08:51:55] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00307 [I 2022-10-10 08:51:55] ...trap/engines/engine.py.230: loss: 0.11672 [I 2022-10-10 08:51:55] ...optimizers/trijoint.py.151: total_norm: 14.023548 [I 2022-10-10 08:52:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 320/2304 [I 2022-10-10 08:52:03] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:03 [I 2022-10-10 08:52:03] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00292 [I 2022-10-10 08:52:03] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-10 08:52:03] ...optimizers/trijoint.py.151: total_norm: 18.198704 [I 2022-10-10 08:52:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 330/2304 [I 2022-10-10 08:52:11] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:27:42 [I 2022-10-10 08:52:11] ...trap/engines/engine.py.229: process: 0.83756 | load: 0.00299 [I 2022-10-10 08:52:11] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-10 08:52:11] ...optimizers/trijoint.py.151: total_norm: 21.203180 [I 2022-10-10 08:52:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 340/2304 [I 2022-10-10 08:52:20] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:27:40 [I 2022-10-10 08:52:20] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00292 [I 2022-10-10 08:52:20] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-10 08:52:20] ...optimizers/trijoint.py.151: total_norm: 43.278526 [I 2022-10-10 08:52:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 350/2304 [I 2022-10-10 08:52:28] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:30 [I 2022-10-10 08:52:28] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00297 [I 2022-10-10 08:52:28] ...trap/engines/engine.py.230: loss: 0.18491 [I 2022-10-10 08:52:28] ...optimizers/trijoint.py.151: total_norm: 54.706165 [I 2022-10-10 08:52:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 360/2304 [I 2022-10-10 08:52:37] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:27:25 [I 2022-10-10 08:52:37] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00298 [I 2022-10-10 08:52:37] ...trap/engines/engine.py.230: loss: 0.11923 [I 2022-10-10 08:52:37] ...optimizers/trijoint.py.151: total_norm: 11.401057 [I 2022-10-10 08:52:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 370/2304 [I 2022-10-10 08:52:45] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:10 [I 2022-10-10 08:52:45] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00305 [I 2022-10-10 08:52:45] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-10 08:52:45] ...optimizers/trijoint.py.151: total_norm: 11.827181 [I 2022-10-10 08:52:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 380/2304 [I 2022-10-10 08:52:54] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:10 [I 2022-10-10 08:52:54] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00302 [I 2022-10-10 08:52:54] ...trap/engines/engine.py.230: loss: 0.14658 [I 2022-10-10 08:52:54] ...optimizers/trijoint.py.151: total_norm: 14.186409 [I 2022-10-10 08:53:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 390/2304 [I 2022-10-10 08:53:02] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:01 [I 2022-10-10 08:53:02] ...trap/engines/engine.py.229: process: 0.83657 | load: 0.00289 [I 2022-10-10 08:53:02] ...trap/engines/engine.py.230: loss: 0.09056 [I 2022-10-10 08:53:02] ...optimizers/trijoint.py.151: total_norm: 9.952862 [I 2022-10-10 08:53:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 400/2304 [I 2022-10-10 08:53:11] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:26:49 [I 2022-10-10 08:53:11] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00290 [I 2022-10-10 08:53:11] ...trap/engines/engine.py.230: loss: 0.15688 [I 2022-10-10 08:53:11] ...optimizers/trijoint.py.151: total_norm: 14.759273 [I 2022-10-10 08:53:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 410/2304 [I 2022-10-10 08:53:19] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:26:48 [I 2022-10-10 08:53:19] ...trap/engines/engine.py.229: process: 0.85266 | load: 0.00294 [I 2022-10-10 08:53:19] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-10 08:53:19] ...optimizers/trijoint.py.151: total_norm: 13.794606 [I 2022-10-10 08:53:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 420/2304 [I 2022-10-10 08:53:28] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:32 [I 2022-10-10 08:53:28] ...trap/engines/engine.py.229: process: 0.83002 | load: 0.00290 [I 2022-10-10 08:53:28] ...trap/engines/engine.py.230: loss: 0.09632 [I 2022-10-10 08:53:28] ...optimizers/trijoint.py.151: total_norm: 5.790615 [I 2022-10-10 08:53:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 430/2304 [I 2022-10-10 08:53:36] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:26:25 [I 2022-10-10 08:53:36] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00293 [I 2022-10-10 08:53:36] ...trap/engines/engine.py.230: loss: 0.16636 [I 2022-10-10 08:53:36] ...optimizers/trijoint.py.151: total_norm: 12.675626 [I 2022-10-10 08:53:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 440/2304 [I 2022-10-10 08:53:45] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:17 [I 2022-10-10 08:53:45] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00293 [I 2022-10-10 08:53:45] ...trap/engines/engine.py.230: loss: 0.11220 [I 2022-10-10 08:53:45] ...optimizers/trijoint.py.151: total_norm: 14.502343 [I 2022-10-10 08:53:53] ...trap/engines/engine.py.224: train: epoch 66 | batch 450/2304 [I 2022-10-10 08:53:53] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:07 [I 2022-10-10 08:53:53] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00292 [I 2022-10-10 08:53:53] ...trap/engines/engine.py.230: loss: 0.21358 [I 2022-10-10 08:53:53] ...optimizers/trijoint.py.151: total_norm: 34.821518 [I 2022-10-10 08:54:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 460/2304 [I 2022-10-10 08:54:02] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:25:53 [I 2022-10-10 08:54:02] ...trap/engines/engine.py.229: process: 0.84183 | load: 0.00297 [I 2022-10-10 08:54:02] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-10 08:54:02] ...optimizers/trijoint.py.151: total_norm: 53.183006 [I 2022-10-10 08:54:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 470/2304 [I 2022-10-10 08:54:10] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:25:53 [I 2022-10-10 08:54:10] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00298 [I 2022-10-10 08:54:10] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-10 08:54:10] ...optimizers/trijoint.py.151: total_norm: 15.816180 [I 2022-10-10 08:54:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 480/2304 [I 2022-10-10 08:54:19] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:25:34 [I 2022-10-10 08:54:19] ...trap/engines/engine.py.229: process: 0.82924 | load: 0.00292 [I 2022-10-10 08:54:19] ...trap/engines/engine.py.230: loss: 0.14375 [I 2022-10-10 08:54:19] ...optimizers/trijoint.py.151: total_norm: 17.207220 [I 2022-10-10 08:54:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 490/2304 [I 2022-10-10 08:54:27] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:25:37 [I 2022-10-10 08:54:27] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00297 [I 2022-10-10 08:54:27] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-10 08:54:27] ...optimizers/trijoint.py.151: total_norm: 11.562132 [I 2022-10-10 08:54:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 500/2304 [I 2022-10-10 08:54:36] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:25:24 [I 2022-10-10 08:54:36] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00311 [I 2022-10-10 08:54:36] ...trap/engines/engine.py.230: loss: 0.17679 [I 2022-10-10 08:54:36] ...optimizers/trijoint.py.151: total_norm: 19.447929 [I 2022-10-10 08:54:44] ...trap/engines/engine.py.224: train: epoch 66 | batch 510/2304 [I 2022-10-10 08:54:44] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:25:13 [I 2022-10-10 08:54:44] ...trap/engines/engine.py.229: process: 0.83166 | load: 0.00293 [I 2022-10-10 08:54:44] ...trap/engines/engine.py.230: loss: 0.17569 [I 2022-10-10 08:54:44] ...optimizers/trijoint.py.151: total_norm: 10.257289 [I 2022-10-10 08:54:53] ...trap/engines/engine.py.224: train: epoch 66 | batch 520/2304 [I 2022-10-10 08:54:53] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:25:14 [I 2022-10-10 08:54:53] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00296 [I 2022-10-10 08:54:53] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-10 08:54:53] ...optimizers/trijoint.py.151: total_norm: 24.085823 [I 2022-10-10 08:55:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 530/2304 [I 2022-10-10 08:55:01] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:24:59 [I 2022-10-10 08:55:01] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00298 [I 2022-10-10 08:55:01] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-10 08:55:01] ...optimizers/trijoint.py.151: total_norm: 40.461189 [I 2022-10-10 08:55:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 540/2304 [I 2022-10-10 08:55:09] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:25:00 [I 2022-10-10 08:55:09] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00313 [I 2022-10-10 08:55:09] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-10 08:55:09] ...optimizers/trijoint.py.151: total_norm: 20.949724 [I 2022-10-10 08:55:18] ...trap/engines/engine.py.224: train: epoch 66 | batch 550/2304 [I 2022-10-10 08:55:18] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:24:59 [I 2022-10-10 08:55:18] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00299 [I 2022-10-10 08:55:18] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-10 08:55:18] ...optimizers/trijoint.py.151: total_norm: 35.952946 [I 2022-10-10 08:55:26] ...trap/engines/engine.py.224: train: epoch 66 | batch 560/2304 [I 2022-10-10 08:55:26] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:34 [I 2022-10-10 08:55:26] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00291 [I 2022-10-10 08:55:26] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-10 08:55:26] ...optimizers/trijoint.py.151: total_norm: 15.191462 [I 2022-10-10 08:55:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 570/2304 [I 2022-10-10 08:55:35] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:27 [I 2022-10-10 08:55:35] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00295 [I 2022-10-10 08:55:35] ...trap/engines/engine.py.230: loss: 0.15425 [I 2022-10-10 08:55:35] ...optimizers/trijoint.py.151: total_norm: 6.752390 [I 2022-10-10 08:55:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 580/2304 [I 2022-10-10 08:55:43] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:24:20 [I 2022-10-10 08:55:43] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00315 [I 2022-10-10 08:55:43] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-10 08:55:43] ...optimizers/trijoint.py.151: total_norm: 9.226686 [I 2022-10-10 08:55:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 590/2304 [I 2022-10-10 08:55:52] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:24:08 [I 2022-10-10 08:55:52] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00296 [I 2022-10-10 08:55:52] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-10 08:55:52] ...optimizers/trijoint.py.151: total_norm: 20.826302 [I 2022-10-10 08:56:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 600/2304 [I 2022-10-10 08:56:00] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:24:00 [I 2022-10-10 08:56:00] ...trap/engines/engine.py.229: process: 0.84081 | load: 0.00292 [I 2022-10-10 08:56:00] ...trap/engines/engine.py.230: loss: 0.11164 [I 2022-10-10 08:56:00] ...optimizers/trijoint.py.151: total_norm: 12.367562 [I 2022-10-10 08:56:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 610/2304 [I 2022-10-10 08:56:09] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:24:00 [I 2022-10-10 08:56:09] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00293 [I 2022-10-10 08:56:09] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-10 08:56:09] ...optimizers/trijoint.py.151: total_norm: 7.856725 [I 2022-10-10 08:56:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 620/2304 [I 2022-10-10 08:56:17] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:23:41 [I 2022-10-10 08:56:17] ...trap/engines/engine.py.229: process: 0.84841 | load: 0.00296 [I 2022-10-10 08:56:17] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-10 08:56:17] ...optimizers/trijoint.py.151: total_norm: 35.061462 [I 2022-10-10 08:56:26] ...trap/engines/engine.py.224: train: epoch 66 | batch 630/2304 [I 2022-10-10 08:56:26] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:25 [I 2022-10-10 08:56:26] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00310 [I 2022-10-10 08:56:26] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-10 08:56:26] ...optimizers/trijoint.py.151: total_norm: 20.823006 [I 2022-10-10 08:56:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 640/2304 [I 2022-10-10 08:56:34] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:23:30 [I 2022-10-10 08:56:34] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00290 [I 2022-10-10 08:56:34] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-10 08:56:34] ...optimizers/trijoint.py.151: total_norm: 31.104067 [I 2022-10-10 08:56:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 650/2304 [I 2022-10-10 08:56:43] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:16 [I 2022-10-10 08:56:43] ...trap/engines/engine.py.229: process: 0.84367 | load: 0.00294 [I 2022-10-10 08:56:43] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-10 08:56:43] ...optimizers/trijoint.py.151: total_norm: 14.086955 [I 2022-10-10 08:56:51] ...trap/engines/engine.py.224: train: epoch 66 | batch 660/2304 [I 2022-10-10 08:56:51] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:04 [I 2022-10-10 08:56:51] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00292 [I 2022-10-10 08:56:51] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-10 08:56:51] ...optimizers/trijoint.py.151: total_norm: 17.557606 [I 2022-10-10 08:57:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 670/2304 [I 2022-10-10 08:57:00] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:22:55 [I 2022-10-10 08:57:00] ...trap/engines/engine.py.229: process: 0.83701 | load: 0.00290 [I 2022-10-10 08:57:00] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-10 08:57:00] ...optimizers/trijoint.py.151: total_norm: 14.458340 [I 2022-10-10 08:57:08] ...trap/engines/engine.py.224: train: epoch 66 | batch 680/2304 [I 2022-10-10 08:57:08] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:38 [I 2022-10-10 08:57:08] ...trap/engines/engine.py.229: process: 0.82291 | load: 0.00291 [I 2022-10-10 08:57:08] ...trap/engines/engine.py.230: loss: 0.11003 [I 2022-10-10 08:57:08] ...optimizers/trijoint.py.151: total_norm: 7.674346 [I 2022-10-10 08:57:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 690/2304 [I 2022-10-10 08:57:16] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:42 [I 2022-10-10 08:57:16] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00290 [I 2022-10-10 08:57:16] ...trap/engines/engine.py.230: loss: 0.15049 [I 2022-10-10 08:57:16] ...optimizers/trijoint.py.151: total_norm: 11.965436 [I 2022-10-10 08:57:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 700/2304 [I 2022-10-10 08:57:25] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:36 [I 2022-10-10 08:57:25] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00296 [I 2022-10-10 08:57:25] ...trap/engines/engine.py.230: loss: 0.25564 [I 2022-10-10 08:57:25] ...optimizers/trijoint.py.151: total_norm: 20.188961 [I 2022-10-10 08:57:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 710/2304 [I 2022-10-10 08:57:33] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:23 [I 2022-10-10 08:57:33] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00301 [I 2022-10-10 08:57:33] ...trap/engines/engine.py.230: loss: 0.16092 [I 2022-10-10 08:57:33] ...optimizers/trijoint.py.151: total_norm: 10.578565 [I 2022-10-10 08:57:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 720/2304 [I 2022-10-10 08:57:42] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:18 [I 2022-10-10 08:57:42] ...trap/engines/engine.py.229: process: 0.85215 | load: 0.00296 [I 2022-10-10 08:57:42] ...trap/engines/engine.py.230: loss: 0.12426 [I 2022-10-10 08:57:42] ...optimizers/trijoint.py.151: total_norm: 15.589300 [I 2022-10-10 08:57:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 730/2304 [I 2022-10-10 08:57:50] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:06 [I 2022-10-10 08:57:50] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00304 [I 2022-10-10 08:57:50] ...trap/engines/engine.py.230: loss: 0.14258 [I 2022-10-10 08:57:50] ...optimizers/trijoint.py.151: total_norm: 9.729703 [I 2022-10-10 08:57:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 740/2304 [I 2022-10-10 08:57:59] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:09 [I 2022-10-10 08:57:59] ...trap/engines/engine.py.229: process: 0.85617 | load: 0.00297 [I 2022-10-10 08:57:59] ...trap/engines/engine.py.230: loss: 0.15076 [I 2022-10-10 08:57:59] ...optimizers/trijoint.py.151: total_norm: 11.535390 [I 2022-10-10 08:58:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 750/2304 [I 2022-10-10 08:58:07] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:01 [I 2022-10-10 08:58:07] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00292 [I 2022-10-10 08:58:07] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-10 08:58:07] ...optimizers/trijoint.py.151: total_norm: 14.953660 [I 2022-10-10 08:58:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 760/2304 [I 2022-10-10 08:58:16] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:49 [I 2022-10-10 08:58:16] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00296 [I 2022-10-10 08:58:16] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-10 08:58:16] ...optimizers/trijoint.py.151: total_norm: 16.475452 [I 2022-10-10 08:58:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 770/2304 [I 2022-10-10 08:58:24] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:38 [I 2022-10-10 08:58:24] ...trap/engines/engine.py.229: process: 0.85469 | load: 0.00301 [I 2022-10-10 08:58:24] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-10 08:58:24] ...optimizers/trijoint.py.151: total_norm: 19.406357 [I 2022-10-10 08:58:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 780/2304 [I 2022-10-10 08:58:33] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:28 [I 2022-10-10 08:58:33] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00302 [I 2022-10-10 08:58:33] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-10 08:58:33] ...optimizers/trijoint.py.151: total_norm: 29.965712 [I 2022-10-10 08:58:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 790/2304 [I 2022-10-10 08:58:41] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:27 [I 2022-10-10 08:58:41] ...trap/engines/engine.py.229: process: 0.86341 | load: 0.00318 [I 2022-10-10 08:58:41] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-10 08:58:41] ...optimizers/trijoint.py.151: total_norm: 27.361212 [I 2022-10-10 08:58:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 800/2304 [I 2022-10-10 08:58:50] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:17 [I 2022-10-10 08:58:50] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00306 [I 2022-10-10 08:58:50] ...trap/engines/engine.py.230: loss: 0.17434 [I 2022-10-10 08:58:50] ...optimizers/trijoint.py.151: total_norm: 17.242176 [I 2022-10-10 08:58:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 810/2304 [I 2022-10-10 08:58:58] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:11 [I 2022-10-10 08:58:58] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00295 [I 2022-10-10 08:58:58] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-10 08:58:58] ...optimizers/trijoint.py.151: total_norm: 17.137766 [I 2022-10-10 08:59:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 820/2304 [I 2022-10-10 08:59:07] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:57 [I 2022-10-10 08:59:07] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00299 [I 2022-10-10 08:59:07] ...trap/engines/engine.py.230: loss: 0.12967 [I 2022-10-10 08:59:07] ...optimizers/trijoint.py.151: total_norm: 15.517758 [I 2022-10-10 08:59:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 830/2304 [I 2022-10-10 08:59:15] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:41 [I 2022-10-10 08:59:15] ...trap/engines/engine.py.229: process: 0.82544 | load: 0.00302 [I 2022-10-10 08:59:15] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-10 08:59:15] ...optimizers/trijoint.py.151: total_norm: 15.597681 [I 2022-10-10 08:59:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 840/2304 [I 2022-10-10 08:59:24] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:48 [I 2022-10-10 08:59:24] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00297 [I 2022-10-10 08:59:24] ...trap/engines/engine.py.230: loss: 0.20329 [I 2022-10-10 08:59:24] ...optimizers/trijoint.py.151: total_norm: 18.479172 [I 2022-10-10 08:59:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 850/2304 [I 2022-10-10 08:59:32] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:39 [I 2022-10-10 08:59:32] ...trap/engines/engine.py.229: process: 0.85419 | load: 0.00301 [I 2022-10-10 08:59:32] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-10 08:59:32] ...optimizers/trijoint.py.151: total_norm: 22.624979 [I 2022-10-10 08:59:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 860/2304 [I 2022-10-10 08:59:41] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:28 [I 2022-10-10 08:59:41] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00296 [I 2022-10-10 08:59:41] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-10 08:59:41] ...optimizers/trijoint.py.151: total_norm: 10.106064 [I 2022-10-10 08:59:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 870/2304 [I 2022-10-10 08:59:49] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:18 [I 2022-10-10 08:59:49] ...trap/engines/engine.py.229: process: 0.83522 | load: 0.00296 [I 2022-10-10 08:59:49] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-10 08:59:49] ...optimizers/trijoint.py.151: total_norm: 24.369825 [I 2022-10-10 08:59:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 880/2304 [I 2022-10-10 08:59:58] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:17 [I 2022-10-10 08:59:58] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00509 [I 2022-10-10 08:59:58] ...trap/engines/engine.py.230: loss: 0.12036 [I 2022-10-10 08:59:58] ...optimizers/trijoint.py.151: total_norm: 14.368630 [I 2022-10-10 09:00:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 890/2304 [I 2022-10-10 09:00:06] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:10 [I 2022-10-10 09:00:06] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00307 [I 2022-10-10 09:00:06] ...trap/engines/engine.py.230: loss: 0.21500 [I 2022-10-10 09:00:06] ...optimizers/trijoint.py.151: total_norm: 26.120211 [I 2022-10-10 09:00:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 900/2304 [I 2022-10-10 09:00:15] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:59 [I 2022-10-10 09:00:15] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00304 [I 2022-10-10 09:00:15] ...trap/engines/engine.py.230: loss: 0.12567 [I 2022-10-10 09:00:15] ...optimizers/trijoint.py.151: total_norm: 16.443871 [I 2022-10-10 09:00:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 910/2304 [I 2022-10-10 09:00:23] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:53 [I 2022-10-10 09:00:23] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00311 [I 2022-10-10 09:00:23] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-10 09:00:23] ...optimizers/trijoint.py.151: total_norm: 15.978523 [I 2022-10-10 09:00:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 920/2304 [I 2022-10-10 09:00:32] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:43 [I 2022-10-10 09:00:32] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00308 [I 2022-10-10 09:00:32] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-10 09:00:32] ...optimizers/trijoint.py.151: total_norm: 23.701035 [I 2022-10-10 09:00:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 930/2304 [I 2022-10-10 09:00:41] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:30 [I 2022-10-10 09:00:41] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00298 [I 2022-10-10 09:00:41] ...trap/engines/engine.py.230: loss: 0.19552 [I 2022-10-10 09:00:41] ...optimizers/trijoint.py.151: total_norm: 55.355602 [I 2022-10-10 09:00:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 940/2304 [I 2022-10-10 09:00:49] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:18 [I 2022-10-10 09:00:49] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00301 [I 2022-10-10 09:00:49] ...trap/engines/engine.py.230: loss: 0.15063 [I 2022-10-10 09:00:49] ...optimizers/trijoint.py.151: total_norm: 21.293207 [I 2022-10-10 09:00:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 950/2304 [I 2022-10-10 09:00:58] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:16 [I 2022-10-10 09:00:58] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00295 [I 2022-10-10 09:00:58] ...trap/engines/engine.py.230: loss: 0.21602 [I 2022-10-10 09:00:58] ...optimizers/trijoint.py.151: total_norm: 15.334747 [I 2022-10-10 09:01:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 960/2304 [I 2022-10-10 09:01:06] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:11 [I 2022-10-10 09:01:06] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00310 [I 2022-10-10 09:01:06] ...trap/engines/engine.py.230: loss: 0.10231 [I 2022-10-10 09:01:06] ...optimizers/trijoint.py.151: total_norm: 10.615806 [I 2022-10-10 09:01:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 970/2304 [I 2022-10-10 09:01:15] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:18:53 [I 2022-10-10 09:01:15] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00310 [I 2022-10-10 09:01:15] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-10 09:01:15] ...optimizers/trijoint.py.151: total_norm: 17.109463 [I 2022-10-10 09:01:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 980/2304 [I 2022-10-10 09:01:23] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:51 [I 2022-10-10 09:01:23] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00301 [I 2022-10-10 09:01:23] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-10 09:01:23] ...optimizers/trijoint.py.151: total_norm: 29.244492 [I 2022-10-10 09:01:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 990/2304 [I 2022-10-10 09:01:32] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:39 [I 2022-10-10 09:01:32] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00307 [I 2022-10-10 09:01:32] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-10 09:01:32] ...optimizers/trijoint.py.151: total_norm: 25.427372 [I 2022-10-10 09:01:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 1000/2304 [I 2022-10-10 09:01:40] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:32 [I 2022-10-10 09:01:40] ...trap/engines/engine.py.229: process: 0.83896 | load: 0.00301 [I 2022-10-10 09:01:40] ...trap/engines/engine.py.230: loss: 0.09911 [I 2022-10-10 09:01:40] ...optimizers/trijoint.py.151: total_norm: 8.485934 [I 2022-10-10 09:01:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1010/2304 [I 2022-10-10 09:01:49] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:28 [I 2022-10-10 09:01:49] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00311 [I 2022-10-10 09:01:49] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-10 09:01:49] ...optimizers/trijoint.py.151: total_norm: 10.863727 [I 2022-10-10 09:01:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 1020/2304 [I 2022-10-10 09:01:58] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:27 [I 2022-10-10 09:01:58] ...trap/engines/engine.py.229: process: 0.86436 | load: 0.00306 [I 2022-10-10 09:01:58] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-10 09:01:58] ...optimizers/trijoint.py.151: total_norm: 30.100248 [I 2022-10-10 09:02:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1030/2304 [I 2022-10-10 09:02:06] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:04 [I 2022-10-10 09:02:06] ...trap/engines/engine.py.229: process: 0.86228 | load: 0.00298 [I 2022-10-10 09:02:06] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-10 09:02:06] ...optimizers/trijoint.py.151: total_norm: 14.120889 [I 2022-10-10 09:02:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 1040/2304 [I 2022-10-10 09:02:15] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:17:50 [I 2022-10-10 09:02:15] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00297 [I 2022-10-10 09:02:15] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-10 09:02:15] ...optimizers/trijoint.py.151: total_norm: 12.922923 [I 2022-10-10 09:02:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 1050/2304 [I 2022-10-10 09:02:23] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:41 [I 2022-10-10 09:02:23] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00306 [I 2022-10-10 09:02:23] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-10 09:02:23] ...optimizers/trijoint.py.151: total_norm: 11.489463 [I 2022-10-10 09:02:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 1060/2304 [I 2022-10-10 09:02:32] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:17:35 [I 2022-10-10 09:02:32] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00315 [I 2022-10-10 09:02:32] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-10 09:02:32] ...optimizers/trijoint.py.151: total_norm: 21.101362 [I 2022-10-10 09:02:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 1070/2304 [I 2022-10-10 09:02:40] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:28 [I 2022-10-10 09:02:40] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00295 [I 2022-10-10 09:02:40] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-10 09:02:40] ...optimizers/trijoint.py.151: total_norm: 16.833187 [I 2022-10-10 09:02:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1080/2304 [I 2022-10-10 09:02:49] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:18 [I 2022-10-10 09:02:49] ...trap/engines/engine.py.229: process: 0.85013 | load: 0.00298 [I 2022-10-10 09:02:49] ...trap/engines/engine.py.230: loss: 0.12174 [I 2022-10-10 09:02:49] ...optimizers/trijoint.py.151: total_norm: 15.365783 [I 2022-10-10 09:02:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1090/2304 [I 2022-10-10 09:02:57] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:09 [I 2022-10-10 09:02:57] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00300 [I 2022-10-10 09:02:57] ...trap/engines/engine.py.230: loss: 0.14134 [I 2022-10-10 09:02:57] ...optimizers/trijoint.py.151: total_norm: 38.151279 [I 2022-10-10 09:03:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1100/2304 [I 2022-10-10 09:03:06] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:05 [I 2022-10-10 09:03:06] ...trap/engines/engine.py.229: process: 0.85127 | load: 0.00299 [I 2022-10-10 09:03:06] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-10 09:03:06] ...optimizers/trijoint.py.151: total_norm: 14.512269 [I 2022-10-10 09:03:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1110/2304 [I 2022-10-10 09:03:14] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:46 [I 2022-10-10 09:03:14] ...trap/engines/engine.py.229: process: 0.83185 | load: 0.00299 [I 2022-10-10 09:03:14] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-10 09:03:14] ...optimizers/trijoint.py.151: total_norm: 14.565802 [I 2022-10-10 09:03:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 1120/2304 [I 2022-10-10 09:03:23] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:44 [I 2022-10-10 09:03:23] ...trap/engines/engine.py.229: process: 0.85474 | load: 0.00296 [I 2022-10-10 09:03:23] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-10 09:03:23] ...optimizers/trijoint.py.151: total_norm: 10.436954 [I 2022-10-10 09:03:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1130/2304 [I 2022-10-10 09:03:49] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 1:27:32 [I 2022-10-10 09:03:49] ...trap/engines/engine.py.229: process: 18.98136 | load: 0.00291 [I 2022-10-10 09:03:49] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-10 09:03:49] ...optimizers/trijoint.py.151: total_norm: 11.982003 [I 2022-10-10 09:03:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 1140/2304 [I 2022-10-10 09:03:58] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:24:01 [I 2022-10-10 09:03:58] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00294 [I 2022-10-10 09:03:58] ...trap/engines/engine.py.230: loss: 0.28567 [I 2022-10-10 09:03:58] ...optimizers/trijoint.py.151: total_norm: 30.468431 [I 2022-10-10 09:04:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1150/2304 [I 2022-10-10 09:04:06] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:17:11 [I 2022-10-10 09:04:06] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00308 [I 2022-10-10 09:04:06] ...trap/engines/engine.py.230: loss: 0.15583 [I 2022-10-10 09:04:06] ...optimizers/trijoint.py.151: total_norm: 14.071517 [I 2022-10-10 09:04:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 1160/2304 [I 2022-10-10 09:04:15] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:12 [I 2022-10-10 09:04:15] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00297 [I 2022-10-10 09:04:15] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-10 09:04:15] ...optimizers/trijoint.py.151: total_norm: 16.578007 [I 2022-10-10 09:04:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 1170/2304 [I 2022-10-10 09:04:23] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:53 [I 2022-10-10 09:04:23] ...trap/engines/engine.py.229: process: 0.83494 | load: 0.00295 [I 2022-10-10 09:04:23] ...trap/engines/engine.py.230: loss: 0.12132 [I 2022-10-10 09:04:23] ...optimizers/trijoint.py.151: total_norm: 13.952599 [I 2022-10-10 09:04:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 1180/2304 [I 2022-10-10 09:04:32] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:53 [I 2022-10-10 09:04:32] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00297 [I 2022-10-10 09:04:32] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-10 09:04:32] ...optimizers/trijoint.py.151: total_norm: 23.278547 [I 2022-10-10 09:04:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 1190/2304 [I 2022-10-10 09:04:40] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:44 [I 2022-10-10 09:04:40] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00294 [I 2022-10-10 09:04:40] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-10 09:04:40] ...optimizers/trijoint.py.151: total_norm: 10.387527 [I 2022-10-10 09:04:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1200/2304 [I 2022-10-10 09:04:49] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:30 [I 2022-10-10 09:04:49] ...trap/engines/engine.py.229: process: 0.82935 | load: 0.00302 [I 2022-10-10 09:04:49] ...trap/engines/engine.py.230: loss: 0.10203 [I 2022-10-10 09:04:49] ...optimizers/trijoint.py.151: total_norm: 14.067368 [I 2022-10-10 09:04:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1210/2304 [I 2022-10-10 09:04:57] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:22 [I 2022-10-10 09:04:57] ...trap/engines/engine.py.229: process: 0.84811 | load: 0.00299 [I 2022-10-10 09:04:57] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-10 09:04:57] ...optimizers/trijoint.py.151: total_norm: 23.831097 [I 2022-10-10 09:05:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1220/2304 [I 2022-10-10 09:05:06] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:16 [I 2022-10-10 09:05:06] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00296 [I 2022-10-10 09:05:06] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-10 09:05:06] ...optimizers/trijoint.py.151: total_norm: 9.783978 [I 2022-10-10 09:05:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1230/2304 [I 2022-10-10 09:05:14] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:11 [I 2022-10-10 09:05:14] ...trap/engines/engine.py.229: process: 0.85122 | load: 0.00294 [I 2022-10-10 09:05:14] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-10 09:05:14] ...optimizers/trijoint.py.151: total_norm: 9.096892 [I 2022-10-10 09:05:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 1240/2304 [I 2022-10-10 09:05:23] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:15:04 [I 2022-10-10 09:05:23] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00296 [I 2022-10-10 09:05:23] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-10 09:05:23] ...optimizers/trijoint.py.151: total_norm: 19.255949 [I 2022-10-10 09:05:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1250/2304 [I 2022-10-10 09:05:31] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:59 [I 2022-10-10 09:05:31] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00291 [I 2022-10-10 09:05:31] ...trap/engines/engine.py.230: loss: 0.16119 [I 2022-10-10 09:05:31] ...optimizers/trijoint.py.151: total_norm: 22.059484 [I 2022-10-10 09:05:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 1260/2304 [I 2022-10-10 09:05:40] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:46 [I 2022-10-10 09:05:40] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00301 [I 2022-10-10 09:05:40] ...trap/engines/engine.py.230: loss: 0.18319 [I 2022-10-10 09:05:40] ...optimizers/trijoint.py.151: total_norm: 22.043985 [I 2022-10-10 09:05:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 1270/2304 [I 2022-10-10 09:05:48] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:41 [I 2022-10-10 09:05:48] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00320 [I 2022-10-10 09:05:48] ...trap/engines/engine.py.230: loss: 0.13913 [I 2022-10-10 09:05:48] ...optimizers/trijoint.py.151: total_norm: 11.651860 [I 2022-10-10 09:05:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1280/2304 [I 2022-10-10 09:05:57] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:33 [I 2022-10-10 09:05:57] ...trap/engines/engine.py.229: process: 0.84660 | load: 0.00300 [I 2022-10-10 09:05:57] ...trap/engines/engine.py.230: loss: 0.15011 [I 2022-10-10 09:05:57] ...optimizers/trijoint.py.151: total_norm: 21.970896 [I 2022-10-10 09:06:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 1290/2304 [I 2022-10-10 09:06:05] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:16 [I 2022-10-10 09:06:05] ...trap/engines/engine.py.229: process: 0.83299 | load: 0.00295 [I 2022-10-10 09:06:05] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-10 09:06:05] ...optimizers/trijoint.py.151: total_norm: 14.459058 [I 2022-10-10 09:06:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1300/2304 [I 2022-10-10 09:06:14] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:14:12 [I 2022-10-10 09:06:14] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00296 [I 2022-10-10 09:06:14] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-10 09:06:14] ...optimizers/trijoint.py.151: total_norm: 45.264523 [I 2022-10-10 09:06:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1310/2304 [I 2022-10-10 09:06:22] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:14:01 [I 2022-10-10 09:06:22] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00293 [I 2022-10-10 09:06:22] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-10 09:06:22] ...optimizers/trijoint.py.151: total_norm: 9.549751 [I 2022-10-10 09:06:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1320/2304 [I 2022-10-10 09:06:31] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:13:57 [I 2022-10-10 09:06:31] ...trap/engines/engine.py.229: process: 0.86269 | load: 0.00302 [I 2022-10-10 09:06:31] ...trap/engines/engine.py.230: loss: 0.12674 [I 2022-10-10 09:06:31] ...optimizers/trijoint.py.151: total_norm: 11.245401 [I 2022-10-10 09:06:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1330/2304 [I 2022-10-10 09:06:39] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:47 [I 2022-10-10 09:06:39] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00303 [I 2022-10-10 09:06:39] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-10 09:06:39] ...optimizers/trijoint.py.151: total_norm: 20.186354 [I 2022-10-10 09:06:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 1340/2304 [I 2022-10-10 09:06:48] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:44 [I 2022-10-10 09:06:48] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00303 [I 2022-10-10 09:06:48] ...trap/engines/engine.py.230: loss: 0.12687 [I 2022-10-10 09:06:48] ...optimizers/trijoint.py.151: total_norm: 15.031819 [I 2022-10-10 09:06:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1350/2304 [I 2022-10-10 09:06:57] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:27 [I 2022-10-10 09:06:57] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00382 [I 2022-10-10 09:06:57] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-10 09:06:57] ...optimizers/trijoint.py.151: total_norm: 29.034353 [I 2022-10-10 09:07:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 1360/2304 [I 2022-10-10 09:07:05] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:18 [I 2022-10-10 09:07:05] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00311 [I 2022-10-10 09:07:05] ...trap/engines/engine.py.230: loss: 0.09216 [I 2022-10-10 09:07:05] ...optimizers/trijoint.py.151: total_norm: 9.124959 [I 2022-10-10 09:07:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1370/2304 [I 2022-10-10 09:07:14] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:12 [I 2022-10-10 09:07:14] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00296 [I 2022-10-10 09:07:14] ...trap/engines/engine.py.230: loss: 0.14713 [I 2022-10-10 09:07:14] ...optimizers/trijoint.py.151: total_norm: 16.948914 [I 2022-10-10 09:07:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1380/2304 [I 2022-10-10 09:07:22] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:01 [I 2022-10-10 09:07:22] ...trap/engines/engine.py.229: process: 0.83743 | load: 0.00294 [I 2022-10-10 09:07:22] ...trap/engines/engine.py.230: loss: 0.09314 [I 2022-10-10 09:07:22] ...optimizers/trijoint.py.151: total_norm: 7.777266 [I 2022-10-10 09:07:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1390/2304 [I 2022-10-10 09:07:31] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:12:57 [I 2022-10-10 09:07:31] ...trap/engines/engine.py.229: process: 0.85598 | load: 0.00295 [I 2022-10-10 09:07:31] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-10 09:07:31] ...optimizers/trijoint.py.151: total_norm: 24.899754 [I 2022-10-10 09:07:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1400/2304 [I 2022-10-10 09:07:39] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:46 [I 2022-10-10 09:07:39] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00294 [I 2022-10-10 09:07:39] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-10 09:07:39] ...optimizers/trijoint.py.151: total_norm: 11.848295 [I 2022-10-10 09:07:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 1410/2304 [I 2022-10-10 09:07:48] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:36 [I 2022-10-10 09:07:48] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00290 [I 2022-10-10 09:07:48] ...trap/engines/engine.py.230: loss: 0.17931 [I 2022-10-10 09:07:48] ...optimizers/trijoint.py.151: total_norm: 19.688738 [I 2022-10-10 09:07:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1420/2304 [I 2022-10-10 09:07:56] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:12:26 [I 2022-10-10 09:07:56] ...trap/engines/engine.py.229: process: 0.84145 | load: 0.00291 [I 2022-10-10 09:07:56] ...trap/engines/engine.py.230: loss: 0.18966 [I 2022-10-10 09:07:56] ...optimizers/trijoint.py.151: total_norm: 17.028538 [I 2022-10-10 09:08:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 1430/2304 [I 2022-10-10 09:08:05] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:16 [I 2022-10-10 09:08:05] ...trap/engines/engine.py.229: process: 0.82603 | load: 0.00294 [I 2022-10-10 09:08:05] ...trap/engines/engine.py.230: loss: 0.10219 [I 2022-10-10 09:08:05] ...optimizers/trijoint.py.151: total_norm: 10.058961 [I 2022-10-10 09:08:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 1440/2304 [I 2022-10-10 09:08:13] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:11 [I 2022-10-10 09:08:13] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00293 [I 2022-10-10 09:08:13] ...trap/engines/engine.py.230: loss: 0.16264 [I 2022-10-10 09:08:13] ...optimizers/trijoint.py.151: total_norm: 12.578856 [I 2022-10-10 09:08:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1450/2304 [I 2022-10-10 09:08:22] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:12:03 [I 2022-10-10 09:08:22] ...trap/engines/engine.py.229: process: 0.83065 | load: 0.00299 [I 2022-10-10 09:08:22] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-10 09:08:22] ...optimizers/trijoint.py.151: total_norm: 9.201765 [I 2022-10-10 09:08:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1460/2304 [I 2022-10-10 09:08:30] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:51 [I 2022-10-10 09:08:30] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00297 [I 2022-10-10 09:08:30] ...trap/engines/engine.py.230: loss: 0.16737 [I 2022-10-10 09:08:30] ...optimizers/trijoint.py.151: total_norm: 9.160087 [I 2022-10-10 09:08:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1470/2304 [I 2022-10-10 09:08:39] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:47 [I 2022-10-10 09:08:39] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00293 [I 2022-10-10 09:08:39] ...trap/engines/engine.py.230: loss: 0.12875 [I 2022-10-10 09:08:39] ...optimizers/trijoint.py.151: total_norm: 18.698837 [I 2022-10-10 09:08:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1480/2304 [I 2022-10-10 09:08:47] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:43 [I 2022-10-10 09:08:47] ...trap/engines/engine.py.229: process: 0.86416 | load: 0.00299 [I 2022-10-10 09:08:47] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-10 09:08:47] ...optimizers/trijoint.py.151: total_norm: 13.855875 [I 2022-10-10 09:08:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1490/2304 [I 2022-10-10 09:08:56] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:29 [I 2022-10-10 09:08:56] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00300 [I 2022-10-10 09:08:56] ...trap/engines/engine.py.230: loss: 0.12507 [I 2022-10-10 09:08:56] ...optimizers/trijoint.py.151: total_norm: 20.361500 [I 2022-10-10 09:09:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1500/2304 [I 2022-10-10 09:09:04] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:25 [I 2022-10-10 09:09:04] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00296 [I 2022-10-10 09:09:04] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-10 09:09:04] ...optimizers/trijoint.py.151: total_norm: 16.032373 [I 2022-10-10 09:09:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 1510/2304 [I 2022-10-10 09:09:13] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:14 [I 2022-10-10 09:09:13] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00298 [I 2022-10-10 09:09:13] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-10 09:09:13] ...optimizers/trijoint.py.151: total_norm: 13.647767 [I 2022-10-10 09:09:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1520/2304 [I 2022-10-10 09:09:21] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:03 [I 2022-10-10 09:09:21] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00296 [I 2022-10-10 09:09:21] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-10 09:09:21] ...optimizers/trijoint.py.151: total_norm: 14.277261 [I 2022-10-10 09:09:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1530/2304 [I 2022-10-10 09:09:30] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:10:58 [I 2022-10-10 09:09:30] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00303 [I 2022-10-10 09:09:30] ...trap/engines/engine.py.230: loss: 0.12337 [I 2022-10-10 09:09:30] ...optimizers/trijoint.py.151: total_norm: 22.344063 [I 2022-10-10 09:09:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1540/2304 [I 2022-10-10 09:09:38] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:51 [I 2022-10-10 09:09:38] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00302 [I 2022-10-10 09:09:38] ...trap/engines/engine.py.230: loss: 0.16631 [I 2022-10-10 09:09:38] ...optimizers/trijoint.py.151: total_norm: 10.208360 [I 2022-10-10 09:09:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1550/2304 [I 2022-10-10 09:09:47] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:43 [I 2022-10-10 09:09:47] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00294 [I 2022-10-10 09:09:47] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-10 09:09:47] ...optimizers/trijoint.py.151: total_norm: 15.769694 [I 2022-10-10 09:09:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1560/2304 [I 2022-10-10 09:09:55] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:28 [I 2022-10-10 09:09:55] ...trap/engines/engine.py.229: process: 0.83390 | load: 0.00301 [I 2022-10-10 09:09:55] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-10 09:09:55] ...optimizers/trijoint.py.151: total_norm: 11.362785 [I 2022-10-10 09:10:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1570/2304 [I 2022-10-10 09:10:04] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:19 [I 2022-10-10 09:10:04] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00301 [I 2022-10-10 09:10:04] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-10 09:10:04] ...optimizers/trijoint.py.151: total_norm: 27.763493 [I 2022-10-10 09:10:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1580/2304 [I 2022-10-10 09:10:12] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:10 [I 2022-10-10 09:10:12] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00295 [I 2022-10-10 09:10:12] ...trap/engines/engine.py.230: loss: 0.13776 [I 2022-10-10 09:10:12] ...optimizers/trijoint.py.151: total_norm: 19.438362 [I 2022-10-10 09:10:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1590/2304 [I 2022-10-10 09:10:21] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:12 [I 2022-10-10 09:10:21] ...trap/engines/engine.py.229: process: 0.87062 | load: 0.00296 [I 2022-10-10 09:10:21] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-10 09:10:21] ...optimizers/trijoint.py.151: total_norm: 20.330242 [I 2022-10-10 09:10:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1600/2304 [I 2022-10-10 09:10:30] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:10:03 [I 2022-10-10 09:10:30] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00297 [I 2022-10-10 09:10:30] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-10 09:10:30] ...optimizers/trijoint.py.151: total_norm: 8.353495 [I 2022-10-10 09:10:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1610/2304 [I 2022-10-10 09:10:38] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:50 [I 2022-10-10 09:10:38] ...trap/engines/engine.py.229: process: 0.83371 | load: 0.00295 [I 2022-10-10 09:10:38] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-10 09:10:38] ...optimizers/trijoint.py.151: total_norm: 6.699843 [I 2022-10-10 09:10:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1620/2304 [I 2022-10-10 09:10:47] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:45 [I 2022-10-10 09:10:47] ...trap/engines/engine.py.229: process: 0.86900 | load: 0.00302 [I 2022-10-10 09:10:47] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-10 09:10:47] ...optimizers/trijoint.py.151: total_norm: 15.828304 [I 2022-10-10 09:10:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1630/2304 [I 2022-10-10 09:10:55] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:30 [I 2022-10-10 09:10:55] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00295 [I 2022-10-10 09:10:55] ...trap/engines/engine.py.230: loss: 0.18196 [I 2022-10-10 09:10:55] ...optimizers/trijoint.py.151: total_norm: 30.150846 [I 2022-10-10 09:11:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1640/2304 [I 2022-10-10 09:11:04] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:23 [I 2022-10-10 09:11:04] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00297 [I 2022-10-10 09:11:04] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-10 09:11:04] ...optimizers/trijoint.py.151: total_norm: 8.109421 [I 2022-10-10 09:11:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1650/2304 [I 2022-10-10 09:11:12] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:15 [I 2022-10-10 09:11:12] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00292 [I 2022-10-10 09:11:12] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-10 09:11:12] ...optimizers/trijoint.py.151: total_norm: 8.037428 [I 2022-10-10 09:11:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1660/2304 [I 2022-10-10 09:11:21] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:09:06 [I 2022-10-10 09:11:21] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00294 [I 2022-10-10 09:11:21] ...trap/engines/engine.py.230: loss: 0.12652 [I 2022-10-10 09:11:21] ...optimizers/trijoint.py.151: total_norm: 14.153232 [I 2022-10-10 09:11:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1670/2304 [I 2022-10-10 09:11:29] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:58 [I 2022-10-10 09:11:29] ...trap/engines/engine.py.229: process: 0.84374 | load: 0.00300 [I 2022-10-10 09:11:29] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-10 09:11:29] ...optimizers/trijoint.py.151: total_norm: 9.064898 [I 2022-10-10 09:11:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1680/2304 [I 2022-10-10 09:11:38] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:50 [I 2022-10-10 09:11:38] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00311 [I 2022-10-10 09:11:38] ...trap/engines/engine.py.230: loss: 0.25140 [I 2022-10-10 09:11:38] ...optimizers/trijoint.py.151: total_norm: 34.312092 [I 2022-10-10 09:11:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1690/2304 [I 2022-10-10 09:11:46] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:44 [I 2022-10-10 09:11:46] ...trap/engines/engine.py.229: process: 0.86175 | load: 0.00301 [I 2022-10-10 09:11:46] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-10 09:11:46] ...optimizers/trijoint.py.151: total_norm: 16.248121 [I 2022-10-10 09:11:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1700/2304 [I 2022-10-10 09:11:55] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:31 [I 2022-10-10 09:11:55] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00296 [I 2022-10-10 09:11:55] ...trap/engines/engine.py.230: loss: 0.18678 [I 2022-10-10 09:11:55] ...optimizers/trijoint.py.151: total_norm: 17.931719 [I 2022-10-10 09:12:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1710/2304 [I 2022-10-10 09:12:03] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:26 [I 2022-10-10 09:12:03] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00333 [I 2022-10-10 09:12:03] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-10 09:12:03] ...optimizers/trijoint.py.151: total_norm: 13.038013 [I 2022-10-10 09:12:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1720/2304 [I 2022-10-10 09:12:12] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:17 [I 2022-10-10 09:12:12] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00297 [I 2022-10-10 09:12:12] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-10 09:12:12] ...optimizers/trijoint.py.151: total_norm: 13.844187 [I 2022-10-10 09:12:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1730/2304 [I 2022-10-10 09:12:21] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:10 [I 2022-10-10 09:12:21] ...trap/engines/engine.py.229: process: 0.86394 | load: 0.00302 [I 2022-10-10 09:12:21] ...trap/engines/engine.py.230: loss: 0.22224 [I 2022-10-10 09:12:21] ...optimizers/trijoint.py.151: total_norm: 21.691002 [I 2022-10-10 09:12:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1740/2304 [I 2022-10-10 09:12:29] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:59 [I 2022-10-10 09:12:29] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00295 [I 2022-10-10 09:12:29] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-10 09:12:29] ...optimizers/trijoint.py.151: total_norm: 14.979207 [I 2022-10-10 09:12:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1750/2304 [I 2022-10-10 09:12:38] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:52 [I 2022-10-10 09:12:38] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00300 [I 2022-10-10 09:12:38] ...trap/engines/engine.py.230: loss: 0.17763 [I 2022-10-10 09:12:38] ...optimizers/trijoint.py.151: total_norm: 18.680433 [I 2022-10-10 09:12:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1760/2304 [I 2022-10-10 09:12:46] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:46 [I 2022-10-10 09:12:46] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00298 [I 2022-10-10 09:12:46] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-10 09:12:46] ...optimizers/trijoint.py.151: total_norm: 13.001413 [I 2022-10-10 09:12:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1770/2304 [I 2022-10-10 09:12:55] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:36 [I 2022-10-10 09:12:55] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00297 [I 2022-10-10 09:12:55] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-10 09:12:55] ...optimizers/trijoint.py.151: total_norm: 17.713060 [I 2022-10-10 09:13:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1780/2304 [I 2022-10-10 09:13:03] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:22 [I 2022-10-10 09:13:03] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00296 [I 2022-10-10 09:13:03] ...trap/engines/engine.py.230: loss: 0.21745 [I 2022-10-10 09:13:03] ...optimizers/trijoint.py.151: total_norm: 64.115211 [I 2022-10-10 09:13:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1790/2304 [I 2022-10-10 09:13:12] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:15 [I 2022-10-10 09:13:12] ...trap/engines/engine.py.229: process: 0.85673 | load: 0.00300 [I 2022-10-10 09:13:12] ...trap/engines/engine.py.230: loss: 0.15275 [I 2022-10-10 09:13:12] ...optimizers/trijoint.py.151: total_norm: 20.553715 [I 2022-10-10 09:13:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 1800/2304 [I 2022-10-10 09:13:20] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:09 [I 2022-10-10 09:13:20] ...trap/engines/engine.py.229: process: 0.86612 | load: 0.00298 [I 2022-10-10 09:13:20] ...trap/engines/engine.py.230: loss: 0.22285 [I 2022-10-10 09:13:20] ...optimizers/trijoint.py.151: total_norm: 20.699705 [I 2022-10-10 09:13:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1810/2304 [I 2022-10-10 09:13:29] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:01 [I 2022-10-10 09:13:29] ...trap/engines/engine.py.229: process: 0.85403 | load: 0.00312 [I 2022-10-10 09:13:29] ...trap/engines/engine.py.230: loss: 0.18277 [I 2022-10-10 09:13:29] ...optimizers/trijoint.py.151: total_norm: 20.830278 [I 2022-10-10 09:13:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1820/2304 [I 2022-10-10 09:13:38] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:53 [I 2022-10-10 09:13:38] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00289 [I 2022-10-10 09:13:38] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-10 09:13:38] ...optimizers/trijoint.py.151: total_norm: 8.773241 [I 2022-10-10 09:13:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1830/2304 [I 2022-10-10 09:13:46] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:43 [I 2022-10-10 09:13:46] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00294 [I 2022-10-10 09:13:46] ...trap/engines/engine.py.230: loss: 0.18843 [I 2022-10-10 09:13:46] ...optimizers/trijoint.py.151: total_norm: 14.477799 [I 2022-10-10 09:13:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1840/2304 [I 2022-10-10 09:13:55] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:37 [I 2022-10-10 09:13:55] ...trap/engines/engine.py.229: process: 0.86783 | load: 0.00295 [I 2022-10-10 09:13:55] ...trap/engines/engine.py.230: loss: 0.16173 [I 2022-10-10 09:13:55] ...optimizers/trijoint.py.151: total_norm: 12.512453 [I 2022-10-10 09:14:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1850/2304 [I 2022-10-10 09:14:03] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:27 [I 2022-10-10 09:14:03] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00294 [I 2022-10-10 09:14:03] ...trap/engines/engine.py.230: loss: 0.12210 [I 2022-10-10 09:14:03] ...optimizers/trijoint.py.151: total_norm: 83.772133 [I 2022-10-10 09:14:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1860/2304 [I 2022-10-10 09:14:12] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:16 [I 2022-10-10 09:14:12] ...trap/engines/engine.py.229: process: 0.82642 | load: 0.00294 [I 2022-10-10 09:14:12] ...trap/engines/engine.py.230: loss: 0.10832 [I 2022-10-10 09:14:12] ...optimizers/trijoint.py.151: total_norm: 11.638496 [I 2022-10-10 09:14:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1870/2304 [I 2022-10-10 09:14:21] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:09 [I 2022-10-10 09:14:21] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00299 [I 2022-10-10 09:14:21] ...trap/engines/engine.py.230: loss: 0.10156 [I 2022-10-10 09:14:21] ...optimizers/trijoint.py.151: total_norm: 10.027867 [I 2022-10-10 09:14:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1880/2304 [I 2022-10-10 09:14:29] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:58 [I 2022-10-10 09:14:29] ...trap/engines/engine.py.229: process: 0.84164 | load: 0.00300 [I 2022-10-10 09:14:29] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-10 09:14:29] ...optimizers/trijoint.py.151: total_norm: 12.406498 [I 2022-10-10 09:14:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1890/2304 [I 2022-10-10 09:14:38] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:53 [I 2022-10-10 09:14:38] ...trap/engines/engine.py.229: process: 0.86907 | load: 0.00305 [I 2022-10-10 09:14:38] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-10 09:14:38] ...optimizers/trijoint.py.151: total_norm: 17.864214 [I 2022-10-10 09:14:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1900/2304 [I 2022-10-10 09:14:46] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:46 [I 2022-10-10 09:14:46] ...trap/engines/engine.py.229: process: 0.86165 | load: 0.00301 [I 2022-10-10 09:14:46] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-10 09:14:46] ...optimizers/trijoint.py.151: total_norm: 9.282716 [I 2022-10-10 09:14:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1910/2304 [I 2022-10-10 09:14:55] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:35 [I 2022-10-10 09:14:55] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00304 [I 2022-10-10 09:14:55] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-10 09:14:55] ...optimizers/trijoint.py.151: total_norm: 9.579584 [I 2022-10-10 09:15:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1920/2304 [I 2022-10-10 09:15:03] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:24 [I 2022-10-10 09:15:03] ...trap/engines/engine.py.229: process: 0.84391 | load: 0.00296 [I 2022-10-10 09:15:03] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-10 09:15:03] ...optimizers/trijoint.py.151: total_norm: 12.082612 [I 2022-10-10 09:15:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1930/2304 [I 2022-10-10 09:15:12] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:16 [I 2022-10-10 09:15:12] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00291 [I 2022-10-10 09:15:12] ...trap/engines/engine.py.230: loss: 0.18885 [I 2022-10-10 09:15:12] ...optimizers/trijoint.py.151: total_norm: 18.881186 [I 2022-10-10 09:15:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 1940/2304 [I 2022-10-10 09:15:20] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:08 [I 2022-10-10 09:15:20] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00300 [I 2022-10-10 09:15:20] ...trap/engines/engine.py.230: loss: 0.10969 [I 2022-10-10 09:15:20] ...optimizers/trijoint.py.151: total_norm: 13.986132 [I 2022-10-10 09:15:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1950/2304 [I 2022-10-10 09:15:29] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:04:59 [I 2022-10-10 09:15:29] ...trap/engines/engine.py.229: process: 0.85203 | load: 0.01049 [I 2022-10-10 09:15:29] ...trap/engines/engine.py.230: loss: 0.16341 [I 2022-10-10 09:15:29] ...optimizers/trijoint.py.151: total_norm: 8.394618 [I 2022-10-10 09:15:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 1960/2304 [I 2022-10-10 09:15:37] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:51 [I 2022-10-10 09:15:37] ...trap/engines/engine.py.229: process: 0.84603 | load: 0.00293 [I 2022-10-10 09:15:37] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-10 09:15:37] ...optimizers/trijoint.py.151: total_norm: 18.803822 [I 2022-10-10 09:15:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1970/2304 [I 2022-10-10 09:15:46] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:42 [I 2022-10-10 09:15:46] ...trap/engines/engine.py.229: process: 0.82847 | load: 0.00295 [I 2022-10-10 09:15:46] ...trap/engines/engine.py.230: loss: 0.18681 [I 2022-10-10 09:15:46] ...optimizers/trijoint.py.151: total_norm: 15.494184 [I 2022-10-10 09:15:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 1980/2304 [I 2022-10-10 09:15:54] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:35 [I 2022-10-10 09:15:54] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00301 [I 2022-10-10 09:15:54] ...trap/engines/engine.py.230: loss: 0.16825 [I 2022-10-10 09:15:54] ...optimizers/trijoint.py.151: total_norm: 16.900555 [I 2022-10-10 09:16:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1990/2304 [I 2022-10-10 09:16:03] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:28 [I 2022-10-10 09:16:03] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00294 [I 2022-10-10 09:16:03] ...trap/engines/engine.py.230: loss: 0.15256 [I 2022-10-10 09:16:03] ...optimizers/trijoint.py.151: total_norm: 16.853889 [I 2022-10-10 09:16:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 2000/2304 [I 2022-10-10 09:16:11] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:19 [I 2022-10-10 09:16:11] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00293 [I 2022-10-10 09:16:11] ...trap/engines/engine.py.230: loss: 0.22272 [I 2022-10-10 09:16:11] ...optimizers/trijoint.py.151: total_norm: 22.597479 [I 2022-10-10 09:16:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 2010/2304 [I 2022-10-10 09:16:20] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:10 [I 2022-10-10 09:16:20] ...trap/engines/engine.py.229: process: 0.85786 | load: 0.00308 [I 2022-10-10 09:16:20] ...trap/engines/engine.py.230: loss: 0.13188 [I 2022-10-10 09:16:20] ...optimizers/trijoint.py.151: total_norm: 8.673846 [I 2022-10-10 09:16:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 2020/2304 [I 2022-10-10 09:16:29] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:04:01 [I 2022-10-10 09:16:29] ...trap/engines/engine.py.229: process: 0.86346 | load: 0.00294 [I 2022-10-10 09:16:29] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-10 09:16:29] ...optimizers/trijoint.py.151: total_norm: 14.819733 [I 2022-10-10 09:16:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 2030/2304 [I 2022-10-10 09:16:37] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:53 [I 2022-10-10 09:16:37] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00305 [I 2022-10-10 09:16:37] ...trap/engines/engine.py.230: loss: 0.11491 [I 2022-10-10 09:16:37] ...optimizers/trijoint.py.151: total_norm: 15.732233 [I 2022-10-10 09:16:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 2040/2304 [I 2022-10-10 09:16:46] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:44 [I 2022-10-10 09:16:46] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00293 [I 2022-10-10 09:16:46] ...trap/engines/engine.py.230: loss: 0.16841 [I 2022-10-10 09:16:46] ...optimizers/trijoint.py.151: total_norm: 13.470971 [I 2022-10-10 09:16:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 2050/2304 [I 2022-10-10 09:16:54] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:34 [I 2022-10-10 09:16:54] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00295 [I 2022-10-10 09:16:54] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-10 09:16:54] ...optimizers/trijoint.py.151: total_norm: 22.450281 [I 2022-10-10 09:17:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 2060/2304 [I 2022-10-10 09:17:03] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:25 [I 2022-10-10 09:17:03] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00302 [I 2022-10-10 09:17:03] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-10 09:17:03] ...optimizers/trijoint.py.151: total_norm: 9.507142 [I 2022-10-10 09:17:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 2070/2304 [I 2022-10-10 09:17:11] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:18 [I 2022-10-10 09:17:11] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00301 [I 2022-10-10 09:17:11] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-10 09:17:11] ...optimizers/trijoint.py.151: total_norm: 10.672940 [I 2022-10-10 09:17:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 2080/2304 [I 2022-10-10 09:17:20] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:03:10 [I 2022-10-10 09:17:20] ...trap/engines/engine.py.229: process: 0.83882 | load: 0.00295 [I 2022-10-10 09:17:20] ...trap/engines/engine.py.230: loss: 0.13796 [I 2022-10-10 09:17:20] ...optimizers/trijoint.py.151: total_norm: 20.578022 [I 2022-10-10 09:17:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 2090/2304 [I 2022-10-10 09:17:28] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:03:01 [I 2022-10-10 09:17:28] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00292 [I 2022-10-10 09:17:28] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-10 09:17:28] ...optimizers/trijoint.py.151: total_norm: 15.600060 [I 2022-10-10 09:17:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 2100/2304 [I 2022-10-10 09:17:37] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:52 [I 2022-10-10 09:17:37] ...trap/engines/engine.py.229: process: 0.84217 | load: 0.00292 [I 2022-10-10 09:17:37] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-10 09:17:37] ...optimizers/trijoint.py.151: total_norm: 14.799171 [I 2022-10-10 09:17:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 2110/2304 [I 2022-10-10 09:17:45] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:02:43 [I 2022-10-10 09:17:45] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00291 [I 2022-10-10 09:17:45] ...trap/engines/engine.py.230: loss: 0.22121 [I 2022-10-10 09:17:45] ...optimizers/trijoint.py.151: total_norm: 21.671875 [I 2022-10-10 09:17:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 2120/2304 [I 2022-10-10 09:17:54] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:36 [I 2022-10-10 09:17:54] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00305 [I 2022-10-10 09:17:54] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-10 09:17:54] ...optimizers/trijoint.py.151: total_norm: 28.785810 [I 2022-10-10 09:18:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 2130/2304 [I 2022-10-10 09:18:02] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:02:28 [I 2022-10-10 09:18:02] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00301 [I 2022-10-10 09:18:02] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-10 09:18:02] ...optimizers/trijoint.py.151: total_norm: 8.080165 [I 2022-10-10 09:18:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 2140/2304 [I 2022-10-10 09:18:11] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:02:19 [I 2022-10-10 09:18:11] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00297 [I 2022-10-10 09:18:11] ...trap/engines/engine.py.230: loss: 0.12198 [I 2022-10-10 09:18:11] ...optimizers/trijoint.py.151: total_norm: 11.799054 [I 2022-10-10 09:18:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 2150/2304 [I 2022-10-10 09:18:19] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:02:10 [I 2022-10-10 09:18:19] ...trap/engines/engine.py.229: process: 0.83752 | load: 0.00297 [I 2022-10-10 09:18:19] ...trap/engines/engine.py.230: loss: 0.09925 [I 2022-10-10 09:18:19] ...optimizers/trijoint.py.151: total_norm: 14.914412 [I 2022-10-10 09:18:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 2160/2304 [I 2022-10-10 09:18:28] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:02:01 [I 2022-10-10 09:18:28] ...trap/engines/engine.py.229: process: 0.86361 | load: 0.00305 [I 2022-10-10 09:18:28] ...trap/engines/engine.py.230: loss: 0.13749 [I 2022-10-10 09:18:28] ...optimizers/trijoint.py.151: total_norm: 20.478611 [I 2022-10-10 09:18:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 2170/2304 [I 2022-10-10 09:18:37] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:53 [I 2022-10-10 09:18:37] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00301 [I 2022-10-10 09:18:37] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-10 09:18:37] ...optimizers/trijoint.py.151: total_norm: 10.688217 [I 2022-10-10 09:18:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 2180/2304 [I 2022-10-10 09:18:45] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:45 [I 2022-10-10 09:18:45] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00298 [I 2022-10-10 09:18:45] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-10 09:18:45] ...optimizers/trijoint.py.151: total_norm: 10.249726 [I 2022-10-10 09:18:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 2190/2304 [I 2022-10-10 09:18:54] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:36 [I 2022-10-10 09:18:54] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00296 [I 2022-10-10 09:18:54] ...trap/engines/engine.py.230: loss: 0.13779 [I 2022-10-10 09:18:54] ...optimizers/trijoint.py.151: total_norm: 14.274206 [I 2022-10-10 09:19:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 2200/2304 [I 2022-10-10 09:19:02] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:27 [I 2022-10-10 09:19:02] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00292 [I 2022-10-10 09:19:02] ...trap/engines/engine.py.230: loss: 0.15167 [I 2022-10-10 09:19:02] ...optimizers/trijoint.py.151: total_norm: 14.215787 [I 2022-10-10 09:19:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 2210/2304 [I 2022-10-10 09:19:11] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:19 [I 2022-10-10 09:19:11] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00309 [I 2022-10-10 09:19:11] ...trap/engines/engine.py.230: loss: 0.09044 [I 2022-10-10 09:19:11] ...optimizers/trijoint.py.151: total_norm: 16.842056 [I 2022-10-10 09:19:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 2220/2304 [I 2022-10-10 09:19:19] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:10 [I 2022-10-10 09:19:19] ...trap/engines/engine.py.229: process: 0.82694 | load: 0.00301 [I 2022-10-10 09:19:19] ...trap/engines/engine.py.230: loss: 0.10719 [I 2022-10-10 09:19:19] ...optimizers/trijoint.py.151: total_norm: 11.313153 [I 2022-10-10 09:19:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 2230/2304 [I 2022-10-10 09:19:28] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:01:02 [I 2022-10-10 09:19:28] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00294 [I 2022-10-10 09:19:28] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-10 09:19:28] ...optimizers/trijoint.py.151: total_norm: 13.858370 [I 2022-10-10 09:19:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 2240/2304 [I 2022-10-10 09:19:36] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:54 [I 2022-10-10 09:19:36] ...trap/engines/engine.py.229: process: 0.85115 | load: 0.00294 [I 2022-10-10 09:19:36] ...trap/engines/engine.py.230: loss: 0.14521 [I 2022-10-10 09:19:36] ...optimizers/trijoint.py.151: total_norm: 26.912384 [I 2022-10-10 09:19:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 2250/2304 [I 2022-10-10 09:19:45] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:46 [I 2022-10-10 09:19:45] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00290 [I 2022-10-10 09:19:45] ...trap/engines/engine.py.230: loss: 0.18575 [I 2022-10-10 09:19:45] ...optimizers/trijoint.py.151: total_norm: 13.605353 [I 2022-10-10 09:19:53] ...trap/engines/engine.py.224: train: epoch 66 | batch 2260/2304 [I 2022-10-10 09:19:53] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:37 [I 2022-10-10 09:19:53] ...trap/engines/engine.py.229: process: 0.83995 | load: 0.00297 [I 2022-10-10 09:19:53] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-10 09:19:53] ...optimizers/trijoint.py.151: total_norm: 16.697687 [I 2022-10-10 09:20:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 2270/2304 [I 2022-10-10 09:20:02] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:28 [I 2022-10-10 09:20:02] ...trap/engines/engine.py.229: process: 0.83130 | load: 0.00293 [I 2022-10-10 09:20:02] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-10 09:20:02] ...optimizers/trijoint.py.151: total_norm: 10.854917 [I 2022-10-10 09:20:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 2280/2304 [I 2022-10-10 09:20:10] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:20 [I 2022-10-10 09:20:10] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00299 [I 2022-10-10 09:20:10] ...trap/engines/engine.py.230: loss: 0.11151 [I 2022-10-10 09:20:10] ...optimizers/trijoint.py.151: total_norm: 15.555556 [I 2022-10-10 09:20:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 2290/2304 [I 2022-10-10 09:20:19] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:11 [I 2022-10-10 09:20:19] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00300 [I 2022-10-10 09:20:19] ...trap/engines/engine.py.230: loss: 0.10686 [I 2022-10-10 09:20:19] ...optimizers/trijoint.py.151: total_norm: 14.173812 [I 2022-10-10 09:20:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 2300/2304 [I 2022-10-10 09:20:27] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:03 [I 2022-10-10 09:20:27] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00016 [I 2022-10-10 09:20:27] ...trap/engines/engine.py.230: loss: 0.13525 [I 2022-10-10 09:20:27] ...optimizers/trijoint.py.151: total_norm: 16.066704 [I 2022-10-10 09:20:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 2304/2304 [I 2022-10-10 09:20:31] ...trap/engines/engine.py.228: elapsed: 0:33:07 | left: 0:00:00 [I 2022-10-10 09:20:31] ...trap/engines/engine.py.229: process: 0.84459 | load: 0.00016 [I 2022-10-10 09:20:31] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-10 09:20:31] ...optimizers/trijoint.py.151: total_norm: 20.193283 [S 2022-10-10 09:20:32] ...trap/engines/engine.py.129: train_epoch.epoch: 66 [S 2022-10-10 09:20:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.14199467420965886 [I 2022-10-10 09:20:32] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 09:20:33] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 66 [I 2022-10-10 09:20:33] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 09:20:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 09:20:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 09:20:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 09:20:38] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 09:20:39] ...trap/engines/engine.py.321: eval: epoch 66 | batch 0/510 [I 2022-10-10 09:20:39] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:42 [I 2022-10-10 09:20:39] ...trap/engines/engine.py.326: process: 5.73054 | load: 4.38345 [I 2022-10-10 09:20:44] ...trap/engines/engine.py.321: eval: epoch 66 | batch 10/510 [I 2022-10-10 09:20:44] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:09:01 [I 2022-10-10 09:20:44] ...trap/engines/engine.py.326: process: 0.48312 | load: 0.00017 [I 2022-10-10 09:20:49] ...trap/engines/engine.py.321: eval: epoch 66 | batch 20/510 [I 2022-10-10 09:20:49] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:25 [I 2022-10-10 09:20:49] ...trap/engines/engine.py.326: process: 0.51869 | load: 0.00019 [I 2022-10-10 09:20:55] ...trap/engines/engine.py.321: eval: epoch 66 | batch 30/510 [I 2022-10-10 09:20:55] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:48 [I 2022-10-10 09:20:55] ...trap/engines/engine.py.326: process: 0.67776 | load: 0.00018 [I 2022-10-10 09:21:01] ...trap/engines/engine.py.321: eval: epoch 66 | batch 40/510 [I 2022-10-10 09:21:01] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:19 [I 2022-10-10 09:21:01] ...trap/engines/engine.py.326: process: 0.56567 | load: 0.00019 [I 2022-10-10 09:21:06] ...trap/engines/engine.py.321: eval: epoch 66 | batch 50/510 [I 2022-10-10 09:21:06] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:04:02 [I 2022-10-10 09:21:06] ...trap/engines/engine.py.326: process: 0.56528 | load: 0.00018 [I 2022-10-10 09:21:11] ...trap/engines/engine.py.321: eval: epoch 66 | batch 60/510 [I 2022-10-10 09:21:11] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:11 [I 2022-10-10 09:21:11] ...trap/engines/engine.py.326: process: 0.60246 | load: 0.00018 [I 2022-10-10 09:21:17] ...trap/engines/engine.py.321: eval: epoch 66 | batch 70/510 [I 2022-10-10 09:21:17] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:08 [I 2022-10-10 09:21:17] ...trap/engines/engine.py.326: process: 0.56318 | load: 0.00018 [I 2022-10-10 09:21:22] ...trap/engines/engine.py.321: eval: epoch 66 | batch 80/510 [I 2022-10-10 09:21:22] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:48 [I 2022-10-10 09:21:22] ...trap/engines/engine.py.326: process: 0.58116 | load: 0.00018 [I 2022-10-10 09:21:26] ...trap/engines/engine.py.321: eval: epoch 66 | batch 90/510 [I 2022-10-10 09:21:26] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:05 [I 2022-10-10 09:21:26] ...trap/engines/engine.py.326: process: 0.43434 | load: 0.00019 [I 2022-10-10 09:21:31] ...trap/engines/engine.py.321: eval: epoch 66 | batch 100/510 [I 2022-10-10 09:21:31] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:51 [I 2022-10-10 09:21:31] ...trap/engines/engine.py.326: process: 0.31663 | load: 0.00019 [I 2022-10-10 09:21:34] ...trap/engines/engine.py.321: eval: epoch 66 | batch 110/510 [I 2022-10-10 09:21:34] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:11 [I 2022-10-10 09:21:34] ...trap/engines/engine.py.326: process: 0.30657 | load: 0.00018 [I 2022-10-10 09:21:37] ...trap/engines/engine.py.321: eval: epoch 66 | batch 120/510 [I 2022-10-10 09:21:37] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:05 [I 2022-10-10 09:21:37] ...trap/engines/engine.py.326: process: 0.31836 | load: 0.00019 [I 2022-10-10 09:21:40] ...trap/engines/engine.py.321: eval: epoch 66 | batch 130/510 [I 2022-10-10 09:21:40] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:01 [I 2022-10-10 09:21:40] ...trap/engines/engine.py.326: process: 0.32203 | load: 0.00019 [I 2022-10-10 09:21:44] ...trap/engines/engine.py.321: eval: epoch 66 | batch 140/510 [I 2022-10-10 09:21:44] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:58 [I 2022-10-10 09:21:44] ...trap/engines/engine.py.326: process: 0.32480 | load: 0.00020 [I 2022-10-10 09:21:47] ...trap/engines/engine.py.321: eval: epoch 66 | batch 150/510 [I 2022-10-10 09:21:47] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:55 [I 2022-10-10 09:21:47] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00017 [I 2022-10-10 09:21:50] ...trap/engines/engine.py.321: eval: epoch 66 | batch 160/510 [I 2022-10-10 09:21:50] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:52 [I 2022-10-10 09:21:50] ...trap/engines/engine.py.326: process: 0.32512 | load: 0.00017 [I 2022-10-10 09:21:53] ...trap/engines/engine.py.321: eval: epoch 66 | batch 170/510 [I 2022-10-10 09:21:53] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:48 [I 2022-10-10 09:21:53] ...trap/engines/engine.py.326: process: 0.31164 | load: 0.00018 [I 2022-10-10 09:21:56] ...trap/engines/engine.py.321: eval: epoch 66 | batch 180/510 [I 2022-10-10 09:21:56] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:45 [I 2022-10-10 09:21:56] ...trap/engines/engine.py.326: process: 0.32031 | load: 0.00019 [I 2022-10-10 09:22:00] ...trap/engines/engine.py.321: eval: epoch 66 | batch 190/510 [I 2022-10-10 09:22:00] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:42 [I 2022-10-10 09:22:00] ...trap/engines/engine.py.326: process: 0.31303 | load: 0.00017 [I 2022-10-10 09:22:03] ...trap/engines/engine.py.321: eval: epoch 66 | batch 200/510 [I 2022-10-10 09:22:03] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:39 [I 2022-10-10 09:22:03] ...trap/engines/engine.py.326: process: 0.32932 | load: 0.00019 [I 2022-10-10 09:22:06] ...trap/engines/engine.py.321: eval: epoch 66 | batch 210/510 [I 2022-10-10 09:22:06] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:37 [I 2022-10-10 09:22:06] ...trap/engines/engine.py.326: process: 0.32787 | load: 0.00019 [I 2022-10-10 09:22:09] ...trap/engines/engine.py.321: eval: epoch 66 | batch 220/510 [I 2022-10-10 09:22:09] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:33 [I 2022-10-10 09:22:09] ...trap/engines/engine.py.326: process: 0.32874 | load: 0.00019 [I 2022-10-10 09:22:13] ...trap/engines/engine.py.321: eval: epoch 66 | batch 230/510 [I 2022-10-10 09:22:13] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:31 [I 2022-10-10 09:22:13] ...trap/engines/engine.py.326: process: 0.33687 | load: 0.00020 [I 2022-10-10 09:22:16] ...trap/engines/engine.py.321: eval: epoch 66 | batch 240/510 [I 2022-10-10 09:22:16] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:27 [I 2022-10-10 09:22:16] ...trap/engines/engine.py.326: process: 0.32155 | load: 0.00019 [I 2022-10-10 09:22:19] ...trap/engines/engine.py.321: eval: epoch 66 | batch 250/510 [I 2022-10-10 09:22:19] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:23 [I 2022-10-10 09:22:19] ...trap/engines/engine.py.326: process: 0.32648 | load: 0.00020 [I 2022-10-10 09:22:22] ...trap/engines/engine.py.321: eval: epoch 66 | batch 260/510 [I 2022-10-10 09:22:22] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:19 [I 2022-10-10 09:22:22] ...trap/engines/engine.py.326: process: 0.30797 | load: 0.00019 [I 2022-10-10 09:22:25] ...trap/engines/engine.py.321: eval: epoch 66 | batch 270/510 [I 2022-10-10 09:22:25] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:17 [I 2022-10-10 09:22:25] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00019 [I 2022-10-10 09:22:29] ...trap/engines/engine.py.321: eval: epoch 66 | batch 280/510 [I 2022-10-10 09:22:29] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:13 [I 2022-10-10 09:22:29] ...trap/engines/engine.py.326: process: 0.31790 | load: 0.00019 [I 2022-10-10 09:22:32] ...trap/engines/engine.py.321: eval: epoch 66 | batch 290/510 [I 2022-10-10 09:22:32] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:10 [I 2022-10-10 09:22:32] ...trap/engines/engine.py.326: process: 0.31940 | load: 0.00019 [I 2022-10-10 09:22:35] ...trap/engines/engine.py.321: eval: epoch 66 | batch 300/510 [I 2022-10-10 09:22:35] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:07 [I 2022-10-10 09:22:35] ...trap/engines/engine.py.326: process: 0.31860 | load: 0.00019 [I 2022-10-10 09:22:38] ...trap/engines/engine.py.321: eval: epoch 66 | batch 310/510 [I 2022-10-10 09:22:38] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:04 [I 2022-10-10 09:22:38] ...trap/engines/engine.py.326: process: 0.32084 | load: 0.00019 [I 2022-10-10 09:22:41] ...trap/engines/engine.py.321: eval: epoch 66 | batch 320/510 [I 2022-10-10 09:22:41] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:01 [I 2022-10-10 09:22:41] ...trap/engines/engine.py.326: process: 0.32765 | load: 0.00019 [I 2022-10-10 09:22:45] ...trap/engines/engine.py.321: eval: epoch 66 | batch 330/510 [I 2022-10-10 09:22:45] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:57 [I 2022-10-10 09:22:45] ...trap/engines/engine.py.326: process: 0.32360 | load: 0.00019 [I 2022-10-10 09:22:48] ...trap/engines/engine.py.321: eval: epoch 66 | batch 340/510 [I 2022-10-10 09:22:48] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:54 [I 2022-10-10 09:22:48] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00019 [I 2022-10-10 09:22:51] ...trap/engines/engine.py.321: eval: epoch 66 | batch 350/510 [I 2022-10-10 09:22:51] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:51 [I 2022-10-10 09:22:51] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00019 [I 2022-10-10 09:22:54] ...trap/engines/engine.py.321: eval: epoch 66 | batch 360/510 [I 2022-10-10 09:22:54] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:47 [I 2022-10-10 09:22:54] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00022 [I 2022-10-10 09:22:57] ...trap/engines/engine.py.321: eval: epoch 66 | batch 370/510 [I 2022-10-10 09:22:57] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:44 [I 2022-10-10 09:22:57] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00017 [I 2022-10-10 09:23:01] ...trap/engines/engine.py.321: eval: epoch 66 | batch 380/510 [I 2022-10-10 09:23:01] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:41 [I 2022-10-10 09:23:01] ...trap/engines/engine.py.326: process: 0.31190 | load: 0.00018 [I 2022-10-10 09:23:04] ...trap/engines/engine.py.321: eval: epoch 66 | batch 390/510 [I 2022-10-10 09:23:04] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:38 [I 2022-10-10 09:23:04] ...trap/engines/engine.py.326: process: 0.31703 | load: 0.00018 [I 2022-10-10 09:23:07] ...trap/engines/engine.py.321: eval: epoch 66 | batch 400/510 [I 2022-10-10 09:23:07] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:35 [I 2022-10-10 09:23:07] ...trap/engines/engine.py.326: process: 0.33022 | load: 0.00018 [I 2022-10-10 09:23:10] ...trap/engines/engine.py.321: eval: epoch 66 | batch 410/510 [I 2022-10-10 09:23:10] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:32 [I 2022-10-10 09:23:10] ...trap/engines/engine.py.326: process: 0.32648 | load: 0.00019 [I 2022-10-10 09:23:14] ...trap/engines/engine.py.321: eval: epoch 66 | batch 420/510 [I 2022-10-10 09:23:14] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:28 [I 2022-10-10 09:23:14] ...trap/engines/engine.py.326: process: 0.33116 | load: 0.00017 [I 2022-10-10 09:23:17] ...trap/engines/engine.py.321: eval: epoch 66 | batch 430/510 [I 2022-10-10 09:23:17] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:26 [I 2022-10-10 09:23:17] ...trap/engines/engine.py.326: process: 0.32495 | load: 0.00018 [I 2022-10-10 09:23:20] ...trap/engines/engine.py.321: eval: epoch 66 | batch 440/510 [I 2022-10-10 09:23:20] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:22 [I 2022-10-10 09:23:20] ...trap/engines/engine.py.326: process: 0.32511 | load: 0.00018 [I 2022-10-10 09:23:23] ...trap/engines/engine.py.321: eval: epoch 66 | batch 450/510 [I 2022-10-10 09:23:23] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:19 [I 2022-10-10 09:23:23] ...trap/engines/engine.py.326: process: 0.32428 | load: 0.00018 [I 2022-10-10 09:23:27] ...trap/engines/engine.py.321: eval: epoch 66 | batch 460/510 [I 2022-10-10 09:23:27] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:16 [I 2022-10-10 09:23:27] ...trap/engines/engine.py.326: process: 0.31564 | load: 0.00019 [I 2022-10-10 09:23:30] ...trap/engines/engine.py.321: eval: epoch 66 | batch 470/510 [I 2022-10-10 09:23:30] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:12 [I 2022-10-10 09:23:30] ...trap/engines/engine.py.326: process: 0.33228 | load: 0.00019 [I 2022-10-10 09:23:33] ...trap/engines/engine.py.321: eval: epoch 66 | batch 480/510 [I 2022-10-10 09:23:33] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-10 09:23:33] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00018 [I 2022-10-10 09:23:36] ...trap/engines/engine.py.321: eval: epoch 66 | batch 490/510 [I 2022-10-10 09:23:36] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-10 09:23:36] ...trap/engines/engine.py.326: process: 0.33426 | load: 0.00019 [I 2022-10-10 09:23:39] ...trap/engines/engine.py.321: eval: epoch 66 | batch 500/510 [I 2022-10-10 09:23:39] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:03 [I 2022-10-10 09:23:39] ...trap/engines/engine.py.326: process: 0.30721 | load: 0.00018 [I 2022-10-10 09:23:43] ...trap/engines/engine.py.321: eval: epoch 66 | batch 510/510 [I 2022-10-10 09:23:43] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:00 [I 2022-10-10 09:23:43] ...trap/engines/engine.py.326: process: 0.31798 | load: 0.00013 [S 2022-10-10 09:23:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 66 [I 2022-10-10 09:23:44] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728200 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907300 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939800 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.726800 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908200 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938100 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008942 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006664 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006539 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014838 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008268 [S 2022-10-10 09:24:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005647 [I 2022-10-10 09:24:22] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 09:24:22] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 09:24:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 09:24:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 09:24:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 09:24:26] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 09:24:26] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 09:24:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 09:24:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 67 [I 2022-10-10 09:24:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 09:24:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 0/2304 [I 2022-10-10 09:24:36] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:09:33 [I 2022-10-10 09:24:36] ...trap/engines/engine.py.229: process: 3.37395 | load: 2.20691 [I 2022-10-10 09:24:36] ...trap/engines/engine.py.230: loss: 0.08357 [I 2022-10-10 09:24:36] ...optimizers/trijoint.py.151: total_norm: 9.315442 [I 2022-10-10 09:24:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 10/2304 [I 2022-10-10 09:24:45] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:38 [I 2022-10-10 09:24:45] ...trap/engines/engine.py.229: process: 0.84136 | load: 0.00578 [I 2022-10-10 09:24:45] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-10 09:24:45] ...optimizers/trijoint.py.151: total_norm: 7.784663 [I 2022-10-10 09:24:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 20/2304 [I 2022-10-10 09:24:53] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:18 [I 2022-10-10 09:24:53] ...trap/engines/engine.py.229: process: 0.85913 | load: 0.00499 [I 2022-10-10 09:24:53] ...trap/engines/engine.py.230: loss: 0.18059 [I 2022-10-10 09:24:53] ...optimizers/trijoint.py.151: total_norm: 15.809855 [I 2022-10-10 09:25:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 30/2304 [I 2022-10-10 09:25:01] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:03 [I 2022-10-10 09:25:01] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00387 [I 2022-10-10 09:25:01] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-10 09:25:01] ...optimizers/trijoint.py.151: total_norm: 14.209366 [I 2022-10-10 09:25:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 40/2304 [I 2022-10-10 09:25:10] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:41 [I 2022-10-10 09:25:10] ...trap/engines/engine.py.229: process: 0.82138 | load: 0.00379 [I 2022-10-10 09:25:10] ...trap/engines/engine.py.230: loss: 0.19935 [I 2022-10-10 09:25:10] ...optimizers/trijoint.py.151: total_norm: 29.233662 [I 2022-10-10 09:25:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 50/2304 [I 2022-10-10 09:25:18] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:32 [I 2022-10-10 09:25:18] ...trap/engines/engine.py.229: process: 0.84030 | load: 0.00318 [I 2022-10-10 09:25:18] ...trap/engines/engine.py.230: loss: 0.10234 [I 2022-10-10 09:25:18] ...optimizers/trijoint.py.151: total_norm: 9.339737 [I 2022-10-10 09:25:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 60/2304 [I 2022-10-10 09:25:27] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:25 [I 2022-10-10 09:25:27] ...trap/engines/engine.py.229: process: 0.82840 | load: 0.00291 [I 2022-10-10 09:25:27] ...trap/engines/engine.py.230: loss: 0.10529 [I 2022-10-10 09:25:27] ...optimizers/trijoint.py.151: total_norm: 22.854958 [I 2022-10-10 09:25:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 70/2304 [I 2022-10-10 09:25:35] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:19 [I 2022-10-10 09:25:35] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00342 [I 2022-10-10 09:25:35] ...trap/engines/engine.py.230: loss: 0.21569 [I 2022-10-10 09:25:35] ...optimizers/trijoint.py.151: total_norm: 18.077555 [I 2022-10-10 09:25:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 80/2304 [I 2022-10-10 09:25:44] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:06 [I 2022-10-10 09:25:44] ...trap/engines/engine.py.229: process: 0.82956 | load: 0.00352 [I 2022-10-10 09:25:44] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-10 09:25:44] ...optimizers/trijoint.py.151: total_norm: 13.412146 [I 2022-10-10 09:25:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 90/2304 [I 2022-10-10 09:25:53] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:03 [I 2022-10-10 09:25:53] ...trap/engines/engine.py.229: process: 0.83737 | load: 0.00329 [I 2022-10-10 09:25:53] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-10 09:25:53] ...optimizers/trijoint.py.151: total_norm: 15.804946 [I 2022-10-10 09:26:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 100/2304 [I 2022-10-10 09:26:01] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:30:52 [I 2022-10-10 09:26:01] ...trap/engines/engine.py.229: process: 0.83848 | load: 0.00327 [I 2022-10-10 09:26:01] ...trap/engines/engine.py.230: loss: 0.20525 [I 2022-10-10 09:26:01] ...optimizers/trijoint.py.151: total_norm: 12.735511 [I 2022-10-10 09:26:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 110/2304 [I 2022-10-10 09:26:09] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:45 [I 2022-10-10 09:26:09] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00306 [I 2022-10-10 09:26:09] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-10 09:26:09] ...optimizers/trijoint.py.151: total_norm: 19.420282 [I 2022-10-10 09:26:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 120/2304 [I 2022-10-10 09:26:18] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:43 [I 2022-10-10 09:26:18] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00340 [I 2022-10-10 09:26:18] ...trap/engines/engine.py.230: loss: 0.16010 [I 2022-10-10 09:26:18] ...optimizers/trijoint.py.151: total_norm: 23.124496 [I 2022-10-10 09:26:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 130/2304 [I 2022-10-10 09:26:26] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:30 [I 2022-10-10 09:26:26] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00307 [I 2022-10-10 09:26:26] ...trap/engines/engine.py.230: loss: 0.13200 [I 2022-10-10 09:26:26] ...optimizers/trijoint.py.151: total_norm: 20.373926 [I 2022-10-10 09:26:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 140/2304 [I 2022-10-10 09:26:35] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:05 [I 2022-10-10 09:26:35] ...trap/engines/engine.py.229: process: 0.83293 | load: 0.00303 [I 2022-10-10 09:26:35] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-10 09:26:35] ...optimizers/trijoint.py.151: total_norm: 10.099888 [I 2022-10-10 09:26:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 150/2304 [I 2022-10-10 09:26:43] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:17 [I 2022-10-10 09:26:43] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00307 [I 2022-10-10 09:26:43] ...trap/engines/engine.py.230: loss: 0.16955 [I 2022-10-10 09:26:43] ...optimizers/trijoint.py.151: total_norm: 47.479980 [I 2022-10-10 09:26:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 160/2304 [I 2022-10-10 09:26:52] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:12 [I 2022-10-10 09:26:52] ...trap/engines/engine.py.229: process: 0.83075 | load: 0.00301 [I 2022-10-10 09:26:52] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-10 09:26:52] ...optimizers/trijoint.py.151: total_norm: 12.570798 [I 2022-10-10 09:27:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 170/2304 [I 2022-10-10 09:27:00] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:30 [I 2022-10-10 09:27:00] ...trap/engines/engine.py.229: process: 0.86866 | load: 0.00309 [I 2022-10-10 09:27:00] ...trap/engines/engine.py.230: loss: 0.11476 [I 2022-10-10 09:27:00] ...optimizers/trijoint.py.151: total_norm: 9.966444 [I 2022-10-10 09:27:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 180/2304 [I 2022-10-10 09:27:09] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:29:50 [I 2022-10-10 09:27:09] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00312 [I 2022-10-10 09:27:09] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-10 09:27:09] ...optimizers/trijoint.py.151: total_norm: 9.452635 [I 2022-10-10 09:27:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 190/2304 [I 2022-10-10 09:27:17] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:55 [I 2022-10-10 09:27:17] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00289 [I 2022-10-10 09:27:17] ...trap/engines/engine.py.230: loss: 0.13283 [I 2022-10-10 09:27:17] ...optimizers/trijoint.py.151: total_norm: 30.910582 [I 2022-10-10 09:27:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 200/2304 [I 2022-10-10 09:27:26] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:38 [I 2022-10-10 09:27:26] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00300 [I 2022-10-10 09:27:26] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-10 09:27:26] ...optimizers/trijoint.py.151: total_norm: 13.453850 [I 2022-10-10 09:27:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 210/2304 [I 2022-10-10 09:27:34] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:30 [I 2022-10-10 09:27:34] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00324 [I 2022-10-10 09:27:34] ...trap/engines/engine.py.230: loss: 0.13951 [I 2022-10-10 09:27:34] ...optimizers/trijoint.py.151: total_norm: 18.445780 [I 2022-10-10 09:27:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 220/2304 [I 2022-10-10 09:27:43] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:31 [I 2022-10-10 09:27:43] ...trap/engines/engine.py.229: process: 0.85823 | load: 0.00310 [I 2022-10-10 09:27:43] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-10 09:27:43] ...optimizers/trijoint.py.151: total_norm: 13.510283 [I 2022-10-10 09:27:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 230/2304 [I 2022-10-10 09:27:51] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:17 [I 2022-10-10 09:27:51] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00288 [I 2022-10-10 09:27:51] ...trap/engines/engine.py.230: loss: 0.15225 [I 2022-10-10 09:27:51] ...optimizers/trijoint.py.151: total_norm: 25.533915 [I 2022-10-10 09:28:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 240/2304 [I 2022-10-10 09:28:00] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:28:56 [I 2022-10-10 09:28:00] ...trap/engines/engine.py.229: process: 0.82662 | load: 0.00297 [I 2022-10-10 09:28:00] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-10 09:28:00] ...optimizers/trijoint.py.151: total_norm: 11.165483 [I 2022-10-10 09:28:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 250/2304 [I 2022-10-10 09:28:08] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:28:55 [I 2022-10-10 09:28:08] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00289 [I 2022-10-10 09:28:08] ...trap/engines/engine.py.230: loss: 0.16740 [I 2022-10-10 09:28:08] ...optimizers/trijoint.py.151: total_norm: 16.832886 [I 2022-10-10 09:28:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 260/2304 [I 2022-10-10 09:28:17] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:51 [I 2022-10-10 09:28:17] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00289 [I 2022-10-10 09:28:17] ...trap/engines/engine.py.230: loss: 0.19108 [I 2022-10-10 09:28:17] ...optimizers/trijoint.py.151: total_norm: 29.966526 [I 2022-10-10 09:28:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 270/2304 [I 2022-10-10 09:28:25] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:34 [I 2022-10-10 09:28:25] ...trap/engines/engine.py.229: process: 0.83437 | load: 0.00289 [I 2022-10-10 09:28:25] ...trap/engines/engine.py.230: loss: 0.13506 [I 2022-10-10 09:28:25] ...optimizers/trijoint.py.151: total_norm: 18.434748 [I 2022-10-10 09:28:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 280/2304 [I 2022-10-10 09:28:34] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:35 [I 2022-10-10 09:28:34] ...trap/engines/engine.py.229: process: 0.83777 | load: 0.00300 [I 2022-10-10 09:28:34] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-10 09:28:34] ...optimizers/trijoint.py.151: total_norm: 17.664789 [I 2022-10-10 09:28:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 290/2304 [I 2022-10-10 09:28:42] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:13 [I 2022-10-10 09:28:42] ...trap/engines/engine.py.229: process: 0.85658 | load: 0.00305 [I 2022-10-10 09:28:42] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-10 09:28:42] ...optimizers/trijoint.py.151: total_norm: 13.435353 [I 2022-10-10 09:28:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 300/2304 [I 2022-10-10 09:28:51] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:11 [I 2022-10-10 09:28:51] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00290 [I 2022-10-10 09:28:51] ...trap/engines/engine.py.230: loss: 0.13452 [I 2022-10-10 09:28:51] ...optimizers/trijoint.py.151: total_norm: 15.959991 [I 2022-10-10 09:28:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 310/2304 [I 2022-10-10 09:28:59] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:01 [I 2022-10-10 09:28:59] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00303 [I 2022-10-10 09:28:59] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-10 09:28:59] ...optimizers/trijoint.py.151: total_norm: 11.773499 [I 2022-10-10 09:29:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 320/2304 [I 2022-10-10 09:29:07] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:27:57 [I 2022-10-10 09:29:07] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00293 [I 2022-10-10 09:29:07] ...trap/engines/engine.py.230: loss: 0.13260 [I 2022-10-10 09:29:07] ...optimizers/trijoint.py.151: total_norm: 21.372530 [I 2022-10-10 09:29:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 330/2304 [I 2022-10-10 09:29:16] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:27:47 [I 2022-10-10 09:29:16] ...trap/engines/engine.py.229: process: 0.84736 | load: 0.00290 [I 2022-10-10 09:29:16] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-10 09:29:16] ...optimizers/trijoint.py.151: total_norm: 8.032856 [I 2022-10-10 09:29:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 340/2304 [I 2022-10-10 09:29:24] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:27:40 [I 2022-10-10 09:29:24] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00294 [I 2022-10-10 09:29:24] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-10 09:29:24] ...optimizers/trijoint.py.151: total_norm: 11.234442 [I 2022-10-10 09:29:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 350/2304 [I 2022-10-10 09:29:33] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:36 [I 2022-10-10 09:29:33] ...trap/engines/engine.py.229: process: 0.84793 | load: 0.00289 [I 2022-10-10 09:29:33] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-10 09:29:33] ...optimizers/trijoint.py.151: total_norm: 18.667158 [I 2022-10-10 09:29:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 360/2304 [I 2022-10-10 09:29:41] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:27:11 [I 2022-10-10 09:29:41] ...trap/engines/engine.py.229: process: 0.82866 | load: 0.00295 [I 2022-10-10 09:29:41] ...trap/engines/engine.py.230: loss: 0.21164 [I 2022-10-10 09:29:41] ...optimizers/trijoint.py.151: total_norm: 20.822508 [I 2022-10-10 09:29:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 370/2304 [I 2022-10-10 09:29:50] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:10 [I 2022-10-10 09:29:50] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00289 [I 2022-10-10 09:29:50] ...trap/engines/engine.py.230: loss: 0.18780 [I 2022-10-10 09:29:50] ...optimizers/trijoint.py.151: total_norm: 27.263683 [I 2022-10-10 09:29:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 380/2304 [I 2022-10-10 09:29:58] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:26:59 [I 2022-10-10 09:29:58] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00289 [I 2022-10-10 09:29:58] ...trap/engines/engine.py.230: loss: 0.15587 [I 2022-10-10 09:29:58] ...optimizers/trijoint.py.151: total_norm: 16.485794 [I 2022-10-10 09:30:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 390/2304 [I 2022-10-10 09:30:07] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:26:50 [I 2022-10-10 09:30:07] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00298 [I 2022-10-10 09:30:07] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-10 09:30:07] ...optimizers/trijoint.py.151: total_norm: 12.338416 [I 2022-10-10 09:30:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 400/2304 [I 2022-10-10 09:30:15] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:26:49 [I 2022-10-10 09:30:15] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00293 [I 2022-10-10 09:30:15] ...trap/engines/engine.py.230: loss: 0.16927 [I 2022-10-10 09:30:15] ...optimizers/trijoint.py.151: total_norm: 21.585470 [I 2022-10-10 09:30:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 410/2304 [I 2022-10-10 09:30:24] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:26:35 [I 2022-10-10 09:30:24] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00288 [I 2022-10-10 09:30:24] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-10 09:30:24] ...optimizers/trijoint.py.151: total_norm: 19.321316 [I 2022-10-10 09:30:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 420/2304 [I 2022-10-10 09:30:32] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:26:32 [I 2022-10-10 09:30:32] ...trap/engines/engine.py.229: process: 0.84641 | load: 0.00288 [I 2022-10-10 09:30:32] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-10 09:30:32] ...optimizers/trijoint.py.151: total_norm: 11.868177 [I 2022-10-10 09:30:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 430/2304 [I 2022-10-10 09:30:41] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:16 [I 2022-10-10 09:30:41] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00290 [I 2022-10-10 09:30:41] ...trap/engines/engine.py.230: loss: 0.15627 [I 2022-10-10 09:30:41] ...optimizers/trijoint.py.151: total_norm: 18.304117 [I 2022-10-10 09:30:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 440/2304 [I 2022-10-10 09:30:49] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:14 [I 2022-10-10 09:30:49] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00304 [I 2022-10-10 09:30:49] ...trap/engines/engine.py.230: loss: 0.18323 [I 2022-10-10 09:30:49] ...optimizers/trijoint.py.151: total_norm: 13.342676 [I 2022-10-10 09:30:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 450/2304 [I 2022-10-10 09:30:58] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:11 [I 2022-10-10 09:30:58] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00293 [I 2022-10-10 09:30:58] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-10 09:30:58] ...optimizers/trijoint.py.151: total_norm: 18.994156 [I 2022-10-10 09:31:06] ...trap/engines/engine.py.224: train: epoch 67 | batch 460/2304 [I 2022-10-10 09:31:06] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:25:58 [I 2022-10-10 09:31:06] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00289 [I 2022-10-10 09:31:06] ...trap/engines/engine.py.230: loss: 0.11408 [I 2022-10-10 09:31:06] ...optimizers/trijoint.py.151: total_norm: 10.682493 [I 2022-10-10 09:31:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 470/2304 [I 2022-10-10 09:31:15] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:25:53 [I 2022-10-10 09:31:15] ...trap/engines/engine.py.229: process: 0.85842 | load: 0.00302 [I 2022-10-10 09:31:15] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-10 09:31:15] ...optimizers/trijoint.py.151: total_norm: 14.411637 [I 2022-10-10 09:31:23] ...trap/engines/engine.py.224: train: epoch 67 | batch 480/2304 [I 2022-10-10 09:31:23] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:55 [I 2022-10-10 09:31:23] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00292 [I 2022-10-10 09:31:23] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-10 09:31:23] ...optimizers/trijoint.py.151: total_norm: 27.013546 [I 2022-10-10 09:31:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 490/2304 [I 2022-10-10 09:31:31] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:25:21 [I 2022-10-10 09:31:31] ...trap/engines/engine.py.229: process: 0.83452 | load: 0.00288 [I 2022-10-10 09:31:31] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-10 09:31:31] ...optimizers/trijoint.py.151: total_norm: 8.065483 [I 2022-10-10 09:31:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 500/2304 [I 2022-10-10 09:31:40] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:18 [I 2022-10-10 09:31:40] ...trap/engines/engine.py.229: process: 0.84048 | load: 0.00287 [I 2022-10-10 09:31:40] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-10 09:31:40] ...optimizers/trijoint.py.151: total_norm: 15.896892 [I 2022-10-10 09:31:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 510/2304 [I 2022-10-10 09:31:48] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:17 [I 2022-10-10 09:31:48] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00289 [I 2022-10-10 09:31:48] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-10 09:31:48] ...optimizers/trijoint.py.151: total_norm: 19.417103 [I 2022-10-10 09:31:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 520/2304 [I 2022-10-10 09:31:57] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:11 [I 2022-10-10 09:31:57] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00287 [I 2022-10-10 09:31:57] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-10 09:31:57] ...optimizers/trijoint.py.151: total_norm: 23.275558 [I 2022-10-10 09:32:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 530/2304 [I 2022-10-10 09:32:05] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:24:59 [I 2022-10-10 09:32:05] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00290 [I 2022-10-10 09:32:05] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-10 09:32:05] ...optimizers/trijoint.py.151: total_norm: 19.169136 [I 2022-10-10 09:32:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 540/2304 [I 2022-10-10 09:32:14] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:24:48 [I 2022-10-10 09:32:14] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00293 [I 2022-10-10 09:32:14] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-10 09:32:14] ...optimizers/trijoint.py.151: total_norm: 18.496519 [I 2022-10-10 09:32:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 550/2304 [I 2022-10-10 09:32:22] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:24:36 [I 2022-10-10 09:32:22] ...trap/engines/engine.py.229: process: 0.83418 | load: 0.00290 [I 2022-10-10 09:32:22] ...trap/engines/engine.py.230: loss: 0.18427 [I 2022-10-10 09:32:22] ...optimizers/trijoint.py.151: total_norm: 29.879295 [I 2022-10-10 09:32:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 560/2304 [I 2022-10-10 09:32:31] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:35 [I 2022-10-10 09:32:31] ...trap/engines/engine.py.229: process: 0.84249 | load: 0.00291 [I 2022-10-10 09:32:31] ...trap/engines/engine.py.230: loss: 0.16593 [I 2022-10-10 09:32:31] ...optimizers/trijoint.py.151: total_norm: 15.535801 [I 2022-10-10 09:32:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 570/2304 [I 2022-10-10 09:32:39] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:24 [I 2022-10-10 09:32:39] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00290 [I 2022-10-10 09:32:39] ...trap/engines/engine.py.230: loss: 0.11633 [I 2022-10-10 09:32:39] ...optimizers/trijoint.py.151: total_norm: 15.143718 [I 2022-10-10 09:32:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 580/2304 [I 2022-10-10 09:32:48] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:19 [I 2022-10-10 09:32:48] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00293 [I 2022-10-10 09:32:48] ...trap/engines/engine.py.230: loss: 0.17408 [I 2022-10-10 09:32:48] ...optimizers/trijoint.py.151: total_norm: 18.219891 [I 2022-10-10 09:32:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 590/2304 [I 2022-10-10 09:32:56] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:05 [I 2022-10-10 09:32:56] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00297 [I 2022-10-10 09:32:56] ...trap/engines/engine.py.230: loss: 0.08491 [I 2022-10-10 09:32:56] ...optimizers/trijoint.py.151: total_norm: 8.344903 [I 2022-10-10 09:33:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 600/2304 [I 2022-10-10 09:33:05] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:23:53 [I 2022-10-10 09:33:05] ...trap/engines/engine.py.229: process: 0.83208 | load: 0.00307 [I 2022-10-10 09:33:05] ...trap/engines/engine.py.230: loss: 0.10865 [I 2022-10-10 09:33:05] ...optimizers/trijoint.py.151: total_norm: 22.208202 [I 2022-10-10 09:33:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 610/2304 [I 2022-10-10 09:33:13] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:23:47 [I 2022-10-10 09:33:13] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00292 [I 2022-10-10 09:33:13] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-10 09:33:13] ...optimizers/trijoint.py.151: total_norm: 12.288535 [I 2022-10-10 09:33:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 620/2304 [I 2022-10-10 09:33:22] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:23:49 [I 2022-10-10 09:33:22] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00292 [I 2022-10-10 09:33:22] ...trap/engines/engine.py.230: loss: 0.12302 [I 2022-10-10 09:33:22] ...optimizers/trijoint.py.151: total_norm: 18.001255 [I 2022-10-10 09:33:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 630/2304 [I 2022-10-10 09:33:30] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:46 [I 2022-10-10 09:33:30] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00294 [I 2022-10-10 09:33:30] ...trap/engines/engine.py.230: loss: 0.10664 [I 2022-10-10 09:33:30] ...optimizers/trijoint.py.151: total_norm: 8.786501 [I 2022-10-10 09:33:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 640/2304 [I 2022-10-10 09:33:39] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:19 [I 2022-10-10 09:33:39] ...trap/engines/engine.py.229: process: 0.83067 | load: 0.00291 [I 2022-10-10 09:33:39] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-10 09:33:39] ...optimizers/trijoint.py.151: total_norm: 27.871080 [I 2022-10-10 09:33:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 650/2304 [I 2022-10-10 09:33:47] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:08 [I 2022-10-10 09:33:47] ...trap/engines/engine.py.229: process: 0.82980 | load: 0.00289 [I 2022-10-10 09:33:47] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-10 09:33:47] ...optimizers/trijoint.py.151: total_norm: 16.813215 [I 2022-10-10 09:33:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 660/2304 [I 2022-10-10 09:33:56] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:02 [I 2022-10-10 09:33:56] ...trap/engines/engine.py.229: process: 0.83629 | load: 0.00290 [I 2022-10-10 09:33:56] ...trap/engines/engine.py.230: loss: 0.10007 [I 2022-10-10 09:33:56] ...optimizers/trijoint.py.151: total_norm: 9.388954 [I 2022-10-10 09:34:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 670/2304 [I 2022-10-10 09:34:04] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:22:56 [I 2022-10-10 09:34:04] ...trap/engines/engine.py.229: process: 0.82364 | load: 0.00291 [I 2022-10-10 09:34:04] ...trap/engines/engine.py.230: loss: 0.10761 [I 2022-10-10 09:34:04] ...optimizers/trijoint.py.151: total_norm: 7.465063 [I 2022-10-10 09:34:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 680/2304 [I 2022-10-10 09:34:13] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:22:57 [I 2022-10-10 09:34:13] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00291 [I 2022-10-10 09:34:13] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-10 09:34:13] ...optimizers/trijoint.py.151: total_norm: 11.584921 [I 2022-10-10 09:34:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 690/2304 [I 2022-10-10 09:34:21] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:22:43 [I 2022-10-10 09:34:21] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00299 [I 2022-10-10 09:34:21] ...trap/engines/engine.py.230: loss: 0.16953 [I 2022-10-10 09:34:21] ...optimizers/trijoint.py.151: total_norm: 24.523954 [I 2022-10-10 09:34:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 700/2304 [I 2022-10-10 09:34:30] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:22:32 [I 2022-10-10 09:34:30] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00298 [I 2022-10-10 09:34:30] ...trap/engines/engine.py.230: loss: 0.18157 [I 2022-10-10 09:34:30] ...optimizers/trijoint.py.151: total_norm: 34.551712 [I 2022-10-10 09:34:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 710/2304 [I 2022-10-10 09:34:38] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:31 [I 2022-10-10 09:34:38] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00291 [I 2022-10-10 09:34:38] ...trap/engines/engine.py.230: loss: 0.15847 [I 2022-10-10 09:34:38] ...optimizers/trijoint.py.151: total_norm: 19.426453 [I 2022-10-10 09:34:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 720/2304 [I 2022-10-10 09:34:47] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:22:24 [I 2022-10-10 09:34:47] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00294 [I 2022-10-10 09:34:47] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-10 09:34:47] ...optimizers/trijoint.py.151: total_norm: 9.748477 [I 2022-10-10 09:34:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 730/2304 [I 2022-10-10 09:34:55] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:08 [I 2022-10-10 09:34:55] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00289 [I 2022-10-10 09:34:55] ...trap/engines/engine.py.230: loss: 0.16059 [I 2022-10-10 09:34:55] ...optimizers/trijoint.py.151: total_norm: 24.176220 [I 2022-10-10 09:35:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 740/2304 [I 2022-10-10 09:35:03] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:21:59 [I 2022-10-10 09:35:03] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00290 [I 2022-10-10 09:35:03] ...trap/engines/engine.py.230: loss: 0.15243 [I 2022-10-10 09:35:03] ...optimizers/trijoint.py.151: total_norm: 11.079513 [I 2022-10-10 09:35:12] ...trap/engines/engine.py.224: train: epoch 67 | batch 750/2304 [I 2022-10-10 09:35:12] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:21:43 [I 2022-10-10 09:35:12] ...trap/engines/engine.py.229: process: 0.83598 | load: 0.00294 [I 2022-10-10 09:35:12] ...trap/engines/engine.py.230: loss: 0.14499 [I 2022-10-10 09:35:12] ...optimizers/trijoint.py.151: total_norm: 11.925108 [I 2022-10-10 09:35:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 760/2304 [I 2022-10-10 09:35:20] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:21:41 [I 2022-10-10 09:35:20] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00288 [I 2022-10-10 09:35:20] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-10 09:35:20] ...optimizers/trijoint.py.151: total_norm: 14.925106 [I 2022-10-10 09:35:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 770/2304 [I 2022-10-10 09:35:29] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:21:43 [I 2022-10-10 09:35:29] ...trap/engines/engine.py.229: process: 0.86014 | load: 0.00291 [I 2022-10-10 09:35:29] ...trap/engines/engine.py.230: loss: 0.12676 [I 2022-10-10 09:35:29] ...optimizers/trijoint.py.151: total_norm: 23.773462 [I 2022-10-10 09:35:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 780/2304 [I 2022-10-10 09:35:37] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:21:30 [I 2022-10-10 09:35:37] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00289 [I 2022-10-10 09:35:37] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-10 09:35:37] ...optimizers/trijoint.py.151: total_norm: 29.923979 [I 2022-10-10 09:35:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 790/2304 [I 2022-10-10 09:35:46] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:17 [I 2022-10-10 09:35:46] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00290 [I 2022-10-10 09:35:46] ...trap/engines/engine.py.230: loss: 0.14392 [I 2022-10-10 09:35:46] ...optimizers/trijoint.py.151: total_norm: 17.640911 [I 2022-10-10 09:35:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 800/2304 [I 2022-10-10 09:35:54] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:18 [I 2022-10-10 09:35:54] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00291 [I 2022-10-10 09:35:54] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-10 09:35:54] ...optimizers/trijoint.py.151: total_norm: 13.530674 [I 2022-10-10 09:36:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 810/2304 [I 2022-10-10 09:36:03] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:10 [I 2022-10-10 09:36:03] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00296 [I 2022-10-10 09:36:03] ...trap/engines/engine.py.230: loss: 0.18328 [I 2022-10-10 09:36:03] ...optimizers/trijoint.py.151: total_norm: 14.105363 [I 2022-10-10 09:36:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 820/2304 [I 2022-10-10 09:36:11] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:20:51 [I 2022-10-10 09:36:11] ...trap/engines/engine.py.229: process: 0.83328 | load: 0.00288 [I 2022-10-10 09:36:11] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-10 09:36:11] ...optimizers/trijoint.py.151: total_norm: 10.589252 [I 2022-10-10 09:36:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 830/2304 [I 2022-10-10 09:36:20] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:39 [I 2022-10-10 09:36:20] ...trap/engines/engine.py.229: process: 0.85347 | load: 0.00303 [I 2022-10-10 09:36:20] ...trap/engines/engine.py.230: loss: 0.11042 [I 2022-10-10 09:36:20] ...optimizers/trijoint.py.151: total_norm: 16.532982 [I 2022-10-10 09:36:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 840/2304 [I 2022-10-10 09:36:28] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:33 [I 2022-10-10 09:36:28] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00290 [I 2022-10-10 09:36:28] ...trap/engines/engine.py.230: loss: 0.13751 [I 2022-10-10 09:36:28] ...optimizers/trijoint.py.151: total_norm: 11.744106 [I 2022-10-10 09:36:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 850/2304 [I 2022-10-10 09:36:37] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:20:27 [I 2022-10-10 09:36:37] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00291 [I 2022-10-10 09:36:37] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-10 09:36:37] ...optimizers/trijoint.py.151: total_norm: 14.417558 [I 2022-10-10 09:36:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 860/2304 [I 2022-10-10 09:36:45] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:19 [I 2022-10-10 09:36:45] ...trap/engines/engine.py.229: process: 0.83142 | load: 0.00297 [I 2022-10-10 09:36:45] ...trap/engines/engine.py.230: loss: 0.09886 [I 2022-10-10 09:36:45] ...optimizers/trijoint.py.151: total_norm: 6.306279 [I 2022-10-10 09:36:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 870/2304 [I 2022-10-10 09:36:54] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:20:17 [I 2022-10-10 09:36:54] ...trap/engines/engine.py.229: process: 0.83639 | load: 0.00289 [I 2022-10-10 09:36:54] ...trap/engines/engine.py.230: loss: 0.17941 [I 2022-10-10 09:36:54] ...optimizers/trijoint.py.151: total_norm: 12.202839 [I 2022-10-10 09:37:02] ...trap/engines/engine.py.224: train: epoch 67 | batch 880/2304 [I 2022-10-10 09:37:02] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:19:57 [I 2022-10-10 09:37:02] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00290 [I 2022-10-10 09:37:02] ...trap/engines/engine.py.230: loss: 0.12393 [I 2022-10-10 09:37:02] ...optimizers/trijoint.py.151: total_norm: 16.009636 [I 2022-10-10 09:37:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 890/2304 [I 2022-10-10 09:37:11] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:19:56 [I 2022-10-10 09:37:11] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00301 [I 2022-10-10 09:37:11] ...trap/engines/engine.py.230: loss: 0.18677 [I 2022-10-10 09:37:11] ...optimizers/trijoint.py.151: total_norm: 19.559614 [I 2022-10-10 09:37:19] ...trap/engines/engine.py.224: train: epoch 67 | batch 900/2304 [I 2022-10-10 09:37:19] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:47 [I 2022-10-10 09:37:19] ...trap/engines/engine.py.229: process: 0.84839 | load: 0.00307 [I 2022-10-10 09:37:19] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-10 09:37:19] ...optimizers/trijoint.py.151: total_norm: 12.177794 [I 2022-10-10 09:37:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 910/2304 [I 2022-10-10 09:37:28] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:19:41 [I 2022-10-10 09:37:28] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00301 [I 2022-10-10 09:37:28] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-10 09:37:28] ...optimizers/trijoint.py.151: total_norm: 19.009838 [I 2022-10-10 09:37:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 920/2304 [I 2022-10-10 09:37:36] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:31 [I 2022-10-10 09:37:36] ...trap/engines/engine.py.229: process: 0.86082 | load: 0.00298 [I 2022-10-10 09:37:36] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-10 09:37:36] ...optimizers/trijoint.py.151: total_norm: 29.070480 [I 2022-10-10 09:37:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 930/2304 [I 2022-10-10 09:37:45] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:27 [I 2022-10-10 09:37:45] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00295 [I 2022-10-10 09:37:45] ...trap/engines/engine.py.230: loss: 0.13563 [I 2022-10-10 09:37:45] ...optimizers/trijoint.py.151: total_norm: 11.155309 [I 2022-10-10 09:37:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 940/2304 [I 2022-10-10 09:37:53] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:10 [I 2022-10-10 09:37:53] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00296 [I 2022-10-10 09:37:53] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-10 09:37:53] ...optimizers/trijoint.py.151: total_norm: 22.709311 [I 2022-10-10 09:38:02] ...trap/engines/engine.py.224: train: epoch 67 | batch 950/2304 [I 2022-10-10 09:38:02] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:07 [I 2022-10-10 09:38:02] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00295 [I 2022-10-10 09:38:02] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-10 09:38:02] ...optimizers/trijoint.py.151: total_norm: 10.373766 [I 2022-10-10 09:38:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 960/2304 [I 2022-10-10 09:38:10] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:18:52 [I 2022-10-10 09:38:10] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00299 [I 2022-10-10 09:38:10] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-10 09:38:10] ...optimizers/trijoint.py.151: total_norm: 17.861174 [I 2022-10-10 09:38:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 970/2304 [I 2022-10-10 09:38:18] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:18:46 [I 2022-10-10 09:38:18] ...trap/engines/engine.py.229: process: 0.83998 | load: 0.00290 [I 2022-10-10 09:38:18] ...trap/engines/engine.py.230: loss: 0.22034 [I 2022-10-10 09:38:18] ...optimizers/trijoint.py.151: total_norm: 76.729866 [I 2022-10-10 09:38:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 980/2304 [I 2022-10-10 09:38:27] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:34 [I 2022-10-10 09:38:27] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00299 [I 2022-10-10 09:38:27] ...trap/engines/engine.py.230: loss: 0.19130 [I 2022-10-10 09:38:27] ...optimizers/trijoint.py.151: total_norm: 21.719238 [I 2022-10-10 09:38:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 990/2304 [I 2022-10-10 09:38:35] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:38 [I 2022-10-10 09:38:35] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00296 [I 2022-10-10 09:38:35] ...trap/engines/engine.py.230: loss: 0.13091 [I 2022-10-10 09:38:35] ...optimizers/trijoint.py.151: total_norm: 14.298177 [I 2022-10-10 09:38:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1000/2304 [I 2022-10-10 09:38:44] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:27 [I 2022-10-10 09:38:44] ...trap/engines/engine.py.229: process: 0.83763 | load: 0.00296 [I 2022-10-10 09:38:44] ...trap/engines/engine.py.230: loss: 0.21800 [I 2022-10-10 09:38:44] ...optimizers/trijoint.py.151: total_norm: 36.055458 [I 2022-10-10 09:38:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 1010/2304 [I 2022-10-10 09:38:53] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:26 [I 2022-10-10 09:38:53] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00300 [I 2022-10-10 09:38:53] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-10 09:38:53] ...optimizers/trijoint.py.151: total_norm: 15.044110 [I 2022-10-10 09:39:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1020/2304 [I 2022-10-10 09:39:01] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:14 [I 2022-10-10 09:39:01] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00353 [I 2022-10-10 09:39:01] ...trap/engines/engine.py.230: loss: 0.10093 [I 2022-10-10 09:39:01] ...optimizers/trijoint.py.151: total_norm: 12.117398 [I 2022-10-10 09:39:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 1030/2304 [I 2022-10-10 09:39:10] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:17:58 [I 2022-10-10 09:39:10] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00303 [I 2022-10-10 09:39:10] ...trap/engines/engine.py.230: loss: 0.21871 [I 2022-10-10 09:39:10] ...optimizers/trijoint.py.151: total_norm: 16.343372 [I 2022-10-10 09:39:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1040/2304 [I 2022-10-10 09:39:18] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:17:51 [I 2022-10-10 09:39:18] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00300 [I 2022-10-10 09:39:18] ...trap/engines/engine.py.230: loss: 0.11979 [I 2022-10-10 09:39:18] ...optimizers/trijoint.py.151: total_norm: 22.873446 [I 2022-10-10 09:39:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 1050/2304 [I 2022-10-10 09:39:27] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:17:45 [I 2022-10-10 09:39:27] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00299 [I 2022-10-10 09:39:27] ...trap/engines/engine.py.230: loss: 0.12038 [I 2022-10-10 09:39:27] ...optimizers/trijoint.py.151: total_norm: 20.547092 [I 2022-10-10 09:39:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1060/2304 [I 2022-10-10 09:39:35] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:17:38 [I 2022-10-10 09:39:35] ...trap/engines/engine.py.229: process: 0.85569 | load: 0.00294 [I 2022-10-10 09:39:35] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-10 09:39:35] ...optimizers/trijoint.py.151: total_norm: 12.178786 [I 2022-10-10 09:39:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1070/2304 [I 2022-10-10 09:39:44] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:17:41 [I 2022-10-10 09:39:44] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00295 [I 2022-10-10 09:39:44] ...trap/engines/engine.py.230: loss: 0.14139 [I 2022-10-10 09:39:44] ...optimizers/trijoint.py.151: total_norm: 11.421594 [I 2022-10-10 09:39:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1080/2304 [I 2022-10-10 09:39:52] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:21 [I 2022-10-10 09:39:52] ...trap/engines/engine.py.229: process: 0.84594 | load: 0.00299 [I 2022-10-10 09:39:52] ...trap/engines/engine.py.230: loss: 0.09628 [I 2022-10-10 09:39:52] ...optimizers/trijoint.py.151: total_norm: 11.219724 [I 2022-10-10 09:40:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1090/2304 [I 2022-10-10 09:40:01] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:14 [I 2022-10-10 09:40:01] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00300 [I 2022-10-10 09:40:01] ...trap/engines/engine.py.230: loss: 0.13030 [I 2022-10-10 09:40:01] ...optimizers/trijoint.py.151: total_norm: 19.957607 [I 2022-10-10 09:40:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 1100/2304 [I 2022-10-10 09:40:10] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:04 [I 2022-10-10 09:40:10] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00295 [I 2022-10-10 09:40:10] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-10 09:40:10] ...optimizers/trijoint.py.151: total_norm: 15.833291 [I 2022-10-10 09:40:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1110/2304 [I 2022-10-10 09:40:18] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:16:57 [I 2022-10-10 09:40:18] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00294 [I 2022-10-10 09:40:18] ...trap/engines/engine.py.230: loss: 0.12435 [I 2022-10-10 09:40:18] ...optimizers/trijoint.py.151: total_norm: 12.255171 [I 2022-10-10 09:40:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 1120/2304 [I 2022-10-10 09:40:27] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:16:46 [I 2022-10-10 09:40:27] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00294 [I 2022-10-10 09:40:27] ...trap/engines/engine.py.230: loss: 0.18667 [I 2022-10-10 09:40:27] ...optimizers/trijoint.py.151: total_norm: 12.675363 [I 2022-10-10 09:40:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1130/2304 [I 2022-10-10 09:40:35] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:41 [I 2022-10-10 09:40:35] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00316 [I 2022-10-10 09:40:35] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-10 09:40:35] ...optimizers/trijoint.py.151: total_norm: 8.606168 [I 2022-10-10 09:40:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1140/2304 [I 2022-10-10 09:40:44] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:38 [I 2022-10-10 09:40:44] ...trap/engines/engine.py.229: process: 0.85720 | load: 0.00309 [I 2022-10-10 09:40:44] ...trap/engines/engine.py.230: loss: 0.19641 [I 2022-10-10 09:40:44] ...optimizers/trijoint.py.151: total_norm: 15.362233 [I 2022-10-10 09:40:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1150/2304 [I 2022-10-10 09:40:52] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:24 [I 2022-10-10 09:40:52] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00298 [I 2022-10-10 09:40:52] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-10 09:40:52] ...optimizers/trijoint.py.151: total_norm: 30.648338 [I 2022-10-10 09:41:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1160/2304 [I 2022-10-10 09:41:01] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:12 [I 2022-10-10 09:41:01] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00296 [I 2022-10-10 09:41:01] ...trap/engines/engine.py.230: loss: 0.10812 [I 2022-10-10 09:41:01] ...optimizers/trijoint.py.151: total_norm: 12.727253 [I 2022-10-10 09:41:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 1170/2304 [I 2022-10-10 09:41:09] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:12 [I 2022-10-10 09:41:09] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00298 [I 2022-10-10 09:41:09] ...trap/engines/engine.py.230: loss: 0.10389 [I 2022-10-10 09:41:09] ...optimizers/trijoint.py.151: total_norm: 19.069189 [I 2022-10-10 09:41:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1180/2304 [I 2022-10-10 09:41:18] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:15:52 [I 2022-10-10 09:41:18] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00305 [I 2022-10-10 09:41:18] ...trap/engines/engine.py.230: loss: 0.16090 [I 2022-10-10 09:41:18] ...optimizers/trijoint.py.151: total_norm: 19.016685 [I 2022-10-10 09:41:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 1190/2304 [I 2022-10-10 09:41:26] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:15:42 [I 2022-10-10 09:41:26] ...trap/engines/engine.py.229: process: 0.85852 | load: 0.00297 [I 2022-10-10 09:41:26] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-10 09:41:26] ...optimizers/trijoint.py.151: total_norm: 16.976984 [I 2022-10-10 09:41:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1200/2304 [I 2022-10-10 09:41:35] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:46 [I 2022-10-10 09:41:35] ...trap/engines/engine.py.229: process: 0.85909 | load: 0.00323 [I 2022-10-10 09:41:35] ...trap/engines/engine.py.230: loss: 0.12039 [I 2022-10-10 09:41:35] ...optimizers/trijoint.py.151: total_norm: 19.333776 [I 2022-10-10 09:41:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1210/2304 [I 2022-10-10 09:41:44] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:37 [I 2022-10-10 09:41:44] ...trap/engines/engine.py.229: process: 0.85106 | load: 0.00309 [I 2022-10-10 09:41:44] ...trap/engines/engine.py.230: loss: 0.15093 [I 2022-10-10 09:41:44] ...optimizers/trijoint.py.151: total_norm: 23.101698 [I 2022-10-10 09:41:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1220/2304 [I 2022-10-10 09:41:52] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:19 [I 2022-10-10 09:41:52] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00294 [I 2022-10-10 09:41:52] ...trap/engines/engine.py.230: loss: 0.13253 [I 2022-10-10 09:41:52] ...optimizers/trijoint.py.151: total_norm: 17.929169 [I 2022-10-10 09:42:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1230/2304 [I 2022-10-10 09:42:01] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:11 [I 2022-10-10 09:42:01] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00330 [I 2022-10-10 09:42:01] ...trap/engines/engine.py.230: loss: 0.11537 [I 2022-10-10 09:42:01] ...optimizers/trijoint.py.151: total_norm: 12.563242 [I 2022-10-10 09:42:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 1240/2304 [I 2022-10-10 09:42:09] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:06 [I 2022-10-10 09:42:09] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00294 [I 2022-10-10 09:42:09] ...trap/engines/engine.py.230: loss: 0.15466 [I 2022-10-10 09:42:09] ...optimizers/trijoint.py.151: total_norm: 15.492041 [I 2022-10-10 09:42:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1250/2304 [I 2022-10-10 09:42:18] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:14:54 [I 2022-10-10 09:42:18] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00304 [I 2022-10-10 09:42:18] ...trap/engines/engine.py.230: loss: 0.13812 [I 2022-10-10 09:42:18] ...optimizers/trijoint.py.151: total_norm: 9.124784 [I 2022-10-10 09:42:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 1260/2304 [I 2022-10-10 09:42:26] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:49 [I 2022-10-10 09:42:26] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00299 [I 2022-10-10 09:42:26] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-10 09:42:26] ...optimizers/trijoint.py.151: total_norm: 69.990547 [I 2022-10-10 09:42:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1270/2304 [I 2022-10-10 09:42:35] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:40 [I 2022-10-10 09:42:35] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00308 [I 2022-10-10 09:42:35] ...trap/engines/engine.py.230: loss: 0.14505 [I 2022-10-10 09:42:35] ...optimizers/trijoint.py.151: total_norm: 12.409424 [I 2022-10-10 09:42:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 1280/2304 [I 2022-10-10 09:42:43] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:27 [I 2022-10-10 09:42:43] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00298 [I 2022-10-10 09:42:43] ...trap/engines/engine.py.230: loss: 0.13820 [I 2022-10-10 09:42:43] ...optimizers/trijoint.py.151: total_norm: 16.157127 [I 2022-10-10 09:42:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1290/2304 [I 2022-10-10 09:42:52] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:14 [I 2022-10-10 09:42:52] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00289 [I 2022-10-10 09:42:52] ...trap/engines/engine.py.230: loss: 0.17074 [I 2022-10-10 09:42:52] ...optimizers/trijoint.py.151: total_norm: 20.572393 [I 2022-10-10 09:43:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 1300/2304 [I 2022-10-10 09:43:00] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:02 [I 2022-10-10 09:43:00] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00291 [I 2022-10-10 09:43:00] ...trap/engines/engine.py.230: loss: 0.10659 [I 2022-10-10 09:43:00] ...optimizers/trijoint.py.151: total_norm: 14.997368 [I 2022-10-10 09:43:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 1310/2304 [I 2022-10-10 09:43:09] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:13:57 [I 2022-10-10 09:43:09] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00292 [I 2022-10-10 09:43:09] ...trap/engines/engine.py.230: loss: 0.17412 [I 2022-10-10 09:43:09] ...optimizers/trijoint.py.151: total_norm: 32.634869 [I 2022-10-10 09:43:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 1320/2304 [I 2022-10-10 09:43:17] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:13:52 [I 2022-10-10 09:43:17] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00295 [I 2022-10-10 09:43:17] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-10 09:43:17] ...optimizers/trijoint.py.151: total_norm: 21.638723 [I 2022-10-10 09:43:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 1330/2304 [I 2022-10-10 09:43:25] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:39 [I 2022-10-10 09:43:25] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00293 [I 2022-10-10 09:43:25] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-10 09:43:25] ...optimizers/trijoint.py.151: total_norm: 12.340967 [I 2022-10-10 09:43:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 1340/2304 [I 2022-10-10 09:43:34] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:34 [I 2022-10-10 09:43:34] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00298 [I 2022-10-10 09:43:34] ...trap/engines/engine.py.230: loss: 0.09199 [I 2022-10-10 09:43:34] ...optimizers/trijoint.py.151: total_norm: 9.509484 [I 2022-10-10 09:43:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 1350/2304 [I 2022-10-10 09:43:42] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:23 [I 2022-10-10 09:43:42] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00298 [I 2022-10-10 09:43:42] ...trap/engines/engine.py.230: loss: 0.09401 [I 2022-10-10 09:43:42] ...optimizers/trijoint.py.151: total_norm: 13.057122 [I 2022-10-10 09:43:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 1360/2304 [I 2022-10-10 09:43:51] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:18 [I 2022-10-10 09:43:51] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00295 [I 2022-10-10 09:43:51] ...trap/engines/engine.py.230: loss: 0.14440 [I 2022-10-10 09:43:51] ...optimizers/trijoint.py.151: total_norm: 12.695375 [I 2022-10-10 09:43:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 1370/2304 [I 2022-10-10 09:43:59] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:07 [I 2022-10-10 09:43:59] ...trap/engines/engine.py.229: process: 0.83765 | load: 0.00296 [I 2022-10-10 09:43:59] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-10 09:43:59] ...optimizers/trijoint.py.151: total_norm: 30.849152 [I 2022-10-10 09:44:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 1380/2304 [I 2022-10-10 09:44:08] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:12:58 [I 2022-10-10 09:44:08] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00298 [I 2022-10-10 09:44:08] ...trap/engines/engine.py.230: loss: 0.14179 [I 2022-10-10 09:44:08] ...optimizers/trijoint.py.151: total_norm: 12.991139 [I 2022-10-10 09:44:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 1390/2304 [I 2022-10-10 09:44:16] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:12:53 [I 2022-10-10 09:44:16] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00290 [I 2022-10-10 09:44:16] ...trap/engines/engine.py.230: loss: 0.11428 [I 2022-10-10 09:44:16] ...optimizers/trijoint.py.151: total_norm: 22.833881 [I 2022-10-10 09:44:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 1400/2304 [I 2022-10-10 09:44:25] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:12:41 [I 2022-10-10 09:44:25] ...trap/engines/engine.py.229: process: 0.83140 | load: 0.00299 [I 2022-10-10 09:44:25] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-10 09:44:25] ...optimizers/trijoint.py.151: total_norm: 19.691847 [I 2022-10-10 09:44:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 1410/2304 [I 2022-10-10 09:44:33] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:31 [I 2022-10-10 09:44:33] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00292 [I 2022-10-10 09:44:33] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-10 09:44:33] ...optimizers/trijoint.py.151: total_norm: 11.564244 [I 2022-10-10 09:44:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 1420/2304 [I 2022-10-10 09:44:41] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:23 [I 2022-10-10 09:44:41] ...trap/engines/engine.py.229: process: 0.83033 | load: 0.00294 [I 2022-10-10 09:44:41] ...trap/engines/engine.py.230: loss: 0.11272 [I 2022-10-10 09:44:41] ...optimizers/trijoint.py.151: total_norm: 11.028593 [I 2022-10-10 09:44:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 1430/2304 [I 2022-10-10 09:44:50] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:14 [I 2022-10-10 09:44:50] ...trap/engines/engine.py.229: process: 0.83638 | load: 0.00291 [I 2022-10-10 09:44:50] ...trap/engines/engine.py.230: loss: 0.18293 [I 2022-10-10 09:44:50] ...optimizers/trijoint.py.151: total_norm: 52.324173 [I 2022-10-10 09:44:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 1440/2304 [I 2022-10-10 09:44:58] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:07 [I 2022-10-10 09:44:58] ...trap/engines/engine.py.229: process: 0.85124 | load: 0.00291 [I 2022-10-10 09:44:58] ...trap/engines/engine.py.230: loss: 0.19478 [I 2022-10-10 09:44:58] ...optimizers/trijoint.py.151: total_norm: 18.510815 [I 2022-10-10 09:45:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 1450/2304 [I 2022-10-10 09:45:07] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:11:59 [I 2022-10-10 09:45:07] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00292 [I 2022-10-10 09:45:07] ...trap/engines/engine.py.230: loss: 0.14149 [I 2022-10-10 09:45:07] ...optimizers/trijoint.py.151: total_norm: 13.808393 [I 2022-10-10 09:45:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 1460/2304 [I 2022-10-10 09:45:15] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:11:48 [I 2022-10-10 09:45:15] ...trap/engines/engine.py.229: process: 0.83789 | load: 0.00297 [I 2022-10-10 09:45:15] ...trap/engines/engine.py.230: loss: 0.18148 [I 2022-10-10 09:45:15] ...optimizers/trijoint.py.151: total_norm: 21.540058 [I 2022-10-10 09:45:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 1470/2304 [I 2022-10-10 09:45:24] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:11:45 [I 2022-10-10 09:45:24] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00288 [I 2022-10-10 09:45:24] ...trap/engines/engine.py.230: loss: 0.10716 [I 2022-10-10 09:45:24] ...optimizers/trijoint.py.151: total_norm: 25.858128 [I 2022-10-10 09:45:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 1480/2304 [I 2022-10-10 09:45:32] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:11:38 [I 2022-10-10 09:45:32] ...trap/engines/engine.py.229: process: 0.84890 | load: 0.00297 [I 2022-10-10 09:45:32] ...trap/engines/engine.py.230: loss: 0.11467 [I 2022-10-10 09:45:32] ...optimizers/trijoint.py.151: total_norm: 15.972452 [I 2022-10-10 09:45:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 1490/2304 [I 2022-10-10 09:45:41] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:11:28 [I 2022-10-10 09:45:41] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00293 [I 2022-10-10 09:45:41] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-10 09:45:41] ...optimizers/trijoint.py.151: total_norm: 20.610769 [I 2022-10-10 09:45:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 1500/2304 [I 2022-10-10 09:45:49] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:24 [I 2022-10-10 09:45:49] ...trap/engines/engine.py.229: process: 0.84805 | load: 0.00290 [I 2022-10-10 09:45:49] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-10 09:45:49] ...optimizers/trijoint.py.151: total_norm: 10.966855 [I 2022-10-10 09:45:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 1510/2304 [I 2022-10-10 09:45:57] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:02 [I 2022-10-10 09:45:57] ...trap/engines/engine.py.229: process: 0.83044 | load: 0.00289 [I 2022-10-10 09:45:57] ...trap/engines/engine.py.230: loss: 0.14506 [I 2022-10-10 09:45:57] ...optimizers/trijoint.py.151: total_norm: 27.057793 [I 2022-10-10 09:46:06] ...trap/engines/engine.py.224: train: epoch 67 | batch 1520/2304 [I 2022-10-10 09:46:06] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:10:59 [I 2022-10-10 09:46:06] ...trap/engines/engine.py.229: process: 0.83869 | load: 0.00295 [I 2022-10-10 09:46:06] ...trap/engines/engine.py.230: loss: 0.18252 [I 2022-10-10 09:46:06] ...optimizers/trijoint.py.151: total_norm: 16.628744 [I 2022-10-10 09:46:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 1530/2304 [I 2022-10-10 09:46:14] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:10:49 [I 2022-10-10 09:46:14] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00290 [I 2022-10-10 09:46:14] ...trap/engines/engine.py.230: loss: 0.17030 [I 2022-10-10 09:46:14] ...optimizers/trijoint.py.151: total_norm: 19.200268 [I 2022-10-10 09:46:23] ...trap/engines/engine.py.224: train: epoch 67 | batch 1540/2304 [I 2022-10-10 09:46:23] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:10:42 [I 2022-10-10 09:46:23] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00293 [I 2022-10-10 09:46:23] ...trap/engines/engine.py.230: loss: 0.11290 [I 2022-10-10 09:46:23] ...optimizers/trijoint.py.151: total_norm: 11.301885 [I 2022-10-10 09:46:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 1550/2304 [I 2022-10-10 09:46:31] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:10:31 [I 2022-10-10 09:46:31] ...trap/engines/engine.py.229: process: 0.82027 | load: 0.00296 [I 2022-10-10 09:46:31] ...trap/engines/engine.py.230: loss: 0.11877 [I 2022-10-10 09:46:31] ...optimizers/trijoint.py.151: total_norm: 7.361679 [I 2022-10-10 09:46:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 1560/2304 [I 2022-10-10 09:46:40] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:10:25 [I 2022-10-10 09:46:40] ...trap/engines/engine.py.229: process: 0.84315 | load: 0.00296 [I 2022-10-10 09:46:40] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-10 09:46:40] ...optimizers/trijoint.py.151: total_norm: 21.477221 [I 2022-10-10 09:46:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 1570/2304 [I 2022-10-10 09:46:48] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:15 [I 2022-10-10 09:46:48] ...trap/engines/engine.py.229: process: 0.81722 | load: 0.00291 [I 2022-10-10 09:46:48] ...trap/engines/engine.py.230: loss: 0.11282 [I 2022-10-10 09:46:48] ...optimizers/trijoint.py.151: total_norm: 6.946747 [I 2022-10-10 09:46:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 1580/2304 [I 2022-10-10 09:46:56] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:10 [I 2022-10-10 09:46:56] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00296 [I 2022-10-10 09:46:56] ...trap/engines/engine.py.230: loss: 0.13996 [I 2022-10-10 09:46:56] ...optimizers/trijoint.py.151: total_norm: 10.321405 [I 2022-10-10 09:47:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 1590/2304 [I 2022-10-10 09:47:05] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:01 [I 2022-10-10 09:47:05] ...trap/engines/engine.py.229: process: 0.84328 | load: 0.00290 [I 2022-10-10 09:47:05] ...trap/engines/engine.py.230: loss: 0.18031 [I 2022-10-10 09:47:05] ...optimizers/trijoint.py.151: total_norm: 18.094702 [I 2022-10-10 09:47:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 1600/2304 [I 2022-10-10 09:47:13] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:09:53 [I 2022-10-10 09:47:13] ...trap/engines/engine.py.229: process: 0.84540 | load: 0.00291 [I 2022-10-10 09:47:13] ...trap/engines/engine.py.230: loss: 0.12983 [I 2022-10-10 09:47:13] ...optimizers/trijoint.py.151: total_norm: 12.605296 [I 2022-10-10 09:47:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 1610/2304 [I 2022-10-10 09:47:22] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:09:47 [I 2022-10-10 09:47:22] ...trap/engines/engine.py.229: process: 0.84670 | load: 0.00298 [I 2022-10-10 09:47:22] ...trap/engines/engine.py.230: loss: 0.10874 [I 2022-10-10 09:47:22] ...optimizers/trijoint.py.151: total_norm: 15.705297 [I 2022-10-10 09:47:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 1620/2304 [I 2022-10-10 09:47:30] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:36 [I 2022-10-10 09:47:30] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00292 [I 2022-10-10 09:47:30] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-10 09:47:30] ...optimizers/trijoint.py.151: total_norm: 10.905557 [I 2022-10-10 09:47:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 1630/2304 [I 2022-10-10 09:47:39] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:09:27 [I 2022-10-10 09:47:39] ...trap/engines/engine.py.229: process: 0.84938 | load: 0.00293 [I 2022-10-10 09:47:39] ...trap/engines/engine.py.230: loss: 0.12170 [I 2022-10-10 09:47:39] ...optimizers/trijoint.py.151: total_norm: 15.562200 [I 2022-10-10 09:47:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 1640/2304 [I 2022-10-10 09:47:47] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:19 [I 2022-10-10 09:47:47] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00290 [I 2022-10-10 09:47:47] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-10 09:47:47] ...optimizers/trijoint.py.151: total_norm: 21.223146 [I 2022-10-10 09:47:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 1650/2304 [I 2022-10-10 09:47:55] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:07 [I 2022-10-10 09:47:55] ...trap/engines/engine.py.229: process: 0.83647 | load: 0.00289 [I 2022-10-10 09:47:55] ...trap/engines/engine.py.230: loss: 0.15290 [I 2022-10-10 09:47:55] ...optimizers/trijoint.py.151: total_norm: 17.222033 [I 2022-10-10 09:48:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 1660/2304 [I 2022-10-10 09:48:04] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:00 [I 2022-10-10 09:48:04] ...trap/engines/engine.py.229: process: 0.84015 | load: 0.00297 [I 2022-10-10 09:48:04] ...trap/engines/engine.py.230: loss: 0.14223 [I 2022-10-10 09:48:04] ...optimizers/trijoint.py.151: total_norm: 10.625870 [I 2022-10-10 09:48:12] ...trap/engines/engine.py.224: train: epoch 67 | batch 1670/2304 [I 2022-10-10 09:48:12] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:08:53 [I 2022-10-10 09:48:12] ...trap/engines/engine.py.229: process: 0.83419 | load: 0.00293 [I 2022-10-10 09:48:12] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-10 09:48:12] ...optimizers/trijoint.py.151: total_norm: 20.631683 [I 2022-10-10 09:48:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 1680/2304 [I 2022-10-10 09:48:21] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:45 [I 2022-10-10 09:48:21] ...trap/engines/engine.py.229: process: 0.83020 | load: 0.00291 [I 2022-10-10 09:48:21] ...trap/engines/engine.py.230: loss: 0.11997 [I 2022-10-10 09:48:21] ...optimizers/trijoint.py.151: total_norm: 36.433250 [I 2022-10-10 09:48:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 1690/2304 [I 2022-10-10 09:48:29] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:38 [I 2022-10-10 09:48:29] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00290 [I 2022-10-10 09:48:29] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-10 09:48:29] ...optimizers/trijoint.py.151: total_norm: 9.827035 [I 2022-10-10 09:48:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 1700/2304 [I 2022-10-10 09:48:38] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:30 [I 2022-10-10 09:48:38] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00289 [I 2022-10-10 09:48:38] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-10 09:48:38] ...optimizers/trijoint.py.151: total_norm: 24.575151 [I 2022-10-10 09:48:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 1710/2304 [I 2022-10-10 09:48:46] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:17 [I 2022-10-10 09:48:46] ...trap/engines/engine.py.229: process: 0.82775 | load: 0.00292 [I 2022-10-10 09:48:46] ...trap/engines/engine.py.230: loss: 0.17483 [I 2022-10-10 09:48:46] ...optimizers/trijoint.py.151: total_norm: 14.344096 [I 2022-10-10 09:48:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 1720/2304 [I 2022-10-10 09:48:54] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:11 [I 2022-10-10 09:48:54] ...trap/engines/engine.py.229: process: 0.83554 | load: 0.00290 [I 2022-10-10 09:48:54] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-10 09:48:54] ...optimizers/trijoint.py.151: total_norm: 9.723810 [I 2022-10-10 09:49:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 1730/2304 [I 2022-10-10 09:49:03] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:01 [I 2022-10-10 09:49:03] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00289 [I 2022-10-10 09:49:03] ...trap/engines/engine.py.230: loss: 0.11231 [I 2022-10-10 09:49:03] ...optimizers/trijoint.py.151: total_norm: 9.165856 [I 2022-10-10 09:49:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 1740/2304 [I 2022-10-10 09:49:11] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:07:52 [I 2022-10-10 09:49:11] ...trap/engines/engine.py.229: process: 0.83961 | load: 0.00294 [I 2022-10-10 09:49:11] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-10 09:49:11] ...optimizers/trijoint.py.151: total_norm: 18.526937 [I 2022-10-10 09:49:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 1750/2304 [I 2022-10-10 09:49:20] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:07:47 [I 2022-10-10 09:49:20] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00292 [I 2022-10-10 09:49:20] ...trap/engines/engine.py.230: loss: 0.20551 [I 2022-10-10 09:49:20] ...optimizers/trijoint.py.151: total_norm: 16.599827 [I 2022-10-10 09:49:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 1760/2304 [I 2022-10-10 09:49:28] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:37 [I 2022-10-10 09:49:28] ...trap/engines/engine.py.229: process: 0.83743 | load: 0.00292 [I 2022-10-10 09:49:28] ...trap/engines/engine.py.230: loss: 0.22785 [I 2022-10-10 09:49:28] ...optimizers/trijoint.py.151: total_norm: 17.365389 [I 2022-10-10 09:49:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 1770/2304 [I 2022-10-10 09:49:37] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:30 [I 2022-10-10 09:49:37] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00306 [I 2022-10-10 09:49:37] ...trap/engines/engine.py.230: loss: 0.18642 [I 2022-10-10 09:49:37] ...optimizers/trijoint.py.151: total_norm: 17.064468 [I 2022-10-10 09:49:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 1780/2304 [I 2022-10-10 09:49:45] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:20 [I 2022-10-10 09:49:45] ...trap/engines/engine.py.229: process: 0.83384 | load: 0.00288 [I 2022-10-10 09:49:45] ...trap/engines/engine.py.230: loss: 0.19359 [I 2022-10-10 09:49:45] ...optimizers/trijoint.py.151: total_norm: 34.276661 [I 2022-10-10 09:49:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 1790/2304 [I 2022-10-10 09:49:53] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:13 [I 2022-10-10 09:49:53] ...trap/engines/engine.py.229: process: 0.83684 | load: 0.00288 [I 2022-10-10 09:49:53] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-10 09:49:53] ...optimizers/trijoint.py.151: total_norm: 25.173292 [I 2022-10-10 09:50:02] ...trap/engines/engine.py.224: train: epoch 67 | batch 1800/2304 [I 2022-10-10 09:50:02] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:05 [I 2022-10-10 09:50:02] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00294 [I 2022-10-10 09:50:02] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-10 09:50:02] ...optimizers/trijoint.py.151: total_norm: 13.408970 [I 2022-10-10 09:50:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 1810/2304 [I 2022-10-10 09:50:10] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:06:56 [I 2022-10-10 09:50:10] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00290 [I 2022-10-10 09:50:10] ...trap/engines/engine.py.230: loss: 0.20852 [I 2022-10-10 09:50:10] ...optimizers/trijoint.py.151: total_norm: 20.825384 [I 2022-10-10 09:50:19] ...trap/engines/engine.py.224: train: epoch 67 | batch 1820/2304 [I 2022-10-10 09:50:19] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:06:47 [I 2022-10-10 09:50:19] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00292 [I 2022-10-10 09:50:19] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-10 09:50:19] ...optimizers/trijoint.py.151: total_norm: 22.207479 [I 2022-10-10 09:50:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 1830/2304 [I 2022-10-10 09:50:27] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:39 [I 2022-10-10 09:50:27] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00293 [I 2022-10-10 09:50:27] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-10 09:50:27] ...optimizers/trijoint.py.151: total_norm: 8.956321 [I 2022-10-10 09:50:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 1840/2304 [I 2022-10-10 09:50:36] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:29 [I 2022-10-10 09:50:36] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00289 [I 2022-10-10 09:50:36] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-10 09:50:36] ...optimizers/trijoint.py.151: total_norm: 10.823761 [I 2022-10-10 09:50:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1850/2304 [I 2022-10-10 09:50:44] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:20 [I 2022-10-10 09:50:44] ...trap/engines/engine.py.229: process: 0.83600 | load: 0.00291 [I 2022-10-10 09:50:44] ...trap/engines/engine.py.230: loss: 0.19936 [I 2022-10-10 09:50:44] ...optimizers/trijoint.py.151: total_norm: 20.303095 [I 2022-10-10 09:50:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 1860/2304 [I 2022-10-10 09:50:53] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:12 [I 2022-10-10 09:50:53] ...trap/engines/engine.py.229: process: 0.82717 | load: 0.00291 [I 2022-10-10 09:50:53] ...trap/engines/engine.py.230: loss: 0.14547 [I 2022-10-10 09:50:53] ...optimizers/trijoint.py.151: total_norm: 19.513809 [I 2022-10-10 09:51:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1870/2304 [I 2022-10-10 09:51:01] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:05 [I 2022-10-10 09:51:01] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00294 [I 2022-10-10 09:51:01] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-10 09:51:01] ...optimizers/trijoint.py.151: total_norm: 18.629103 [I 2022-10-10 09:51:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 1880/2304 [I 2022-10-10 09:51:09] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:05:56 [I 2022-10-10 09:51:09] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00293 [I 2022-10-10 09:51:09] ...trap/engines/engine.py.230: loss: 0.19113 [I 2022-10-10 09:51:09] ...optimizers/trijoint.py.151: total_norm: 22.109955 [I 2022-10-10 09:51:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1890/2304 [I 2022-10-10 09:51:18] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:05:49 [I 2022-10-10 09:51:18] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00289 [I 2022-10-10 09:51:18] ...trap/engines/engine.py.230: loss: 0.18858 [I 2022-10-10 09:51:18] ...optimizers/trijoint.py.151: total_norm: 21.006163 [I 2022-10-10 09:51:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 1900/2304 [I 2022-10-10 09:51:26] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:42 [I 2022-10-10 09:51:26] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00290 [I 2022-10-10 09:51:26] ...trap/engines/engine.py.230: loss: 0.14247 [I 2022-10-10 09:51:26] ...optimizers/trijoint.py.151: total_norm: 12.691798 [I 2022-10-10 09:51:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1910/2304 [I 2022-10-10 09:51:35] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:05:32 [I 2022-10-10 09:51:35] ...trap/engines/engine.py.229: process: 0.83702 | load: 0.00295 [I 2022-10-10 09:51:35] ...trap/engines/engine.py.230: loss: 0.15297 [I 2022-10-10 09:51:35] ...optimizers/trijoint.py.151: total_norm: 30.730658 [I 2022-10-10 09:51:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 1920/2304 [I 2022-10-10 09:51:43] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:22 [I 2022-10-10 09:51:43] ...trap/engines/engine.py.229: process: 0.83452 | load: 0.00295 [I 2022-10-10 09:51:43] ...trap/engines/engine.py.230: loss: 0.10234 [I 2022-10-10 09:51:43] ...optimizers/trijoint.py.151: total_norm: 12.116692 [I 2022-10-10 09:51:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1930/2304 [I 2022-10-10 09:51:52] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:15 [I 2022-10-10 09:51:52] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00300 [I 2022-10-10 09:51:52] ...trap/engines/engine.py.230: loss: 0.12511 [I 2022-10-10 09:51:52] ...optimizers/trijoint.py.151: total_norm: 17.603409 [I 2022-10-10 09:52:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 1940/2304 [I 2022-10-10 09:52:00] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:04 [I 2022-10-10 09:52:00] ...trap/engines/engine.py.229: process: 0.82284 | load: 0.00290 [I 2022-10-10 09:52:00] ...trap/engines/engine.py.230: loss: 0.20745 [I 2022-10-10 09:52:00] ...optimizers/trijoint.py.151: total_norm: 17.267155 [I 2022-10-10 09:52:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 1950/2304 [I 2022-10-10 09:52:08] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:04:56 [I 2022-10-10 09:52:08] ...trap/engines/engine.py.229: process: 0.83825 | load: 0.00290 [I 2022-10-10 09:52:08] ...trap/engines/engine.py.230: loss: 0.13378 [I 2022-10-10 09:52:08] ...optimizers/trijoint.py.151: total_norm: 11.020090 [I 2022-10-10 09:52:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 1960/2304 [I 2022-10-10 09:52:17] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:04:47 [I 2022-10-10 09:52:17] ...trap/engines/engine.py.229: process: 0.82233 | load: 0.00288 [I 2022-10-10 09:52:17] ...trap/engines/engine.py.230: loss: 0.20679 [I 2022-10-10 09:52:17] ...optimizers/trijoint.py.151: total_norm: 8.709792 [I 2022-10-10 09:52:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 1970/2304 [I 2022-10-10 09:52:25] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:40 [I 2022-10-10 09:52:25] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00293 [I 2022-10-10 09:52:25] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-10 09:52:25] ...optimizers/trijoint.py.151: total_norm: 8.066135 [I 2022-10-10 09:52:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 1980/2304 [I 2022-10-10 09:52:34] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:31 [I 2022-10-10 09:52:34] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00293 [I 2022-10-10 09:52:34] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-10 09:52:34] ...optimizers/trijoint.py.151: total_norm: 16.376556 [I 2022-10-10 09:52:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 1990/2304 [I 2022-10-10 09:52:42] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:22 [I 2022-10-10 09:52:42] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00290 [I 2022-10-10 09:52:42] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-10 09:52:42] ...optimizers/trijoint.py.151: total_norm: 7.361437 [I 2022-10-10 09:52:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 2000/2304 [I 2022-10-10 09:52:50] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:15 [I 2022-10-10 09:52:50] ...trap/engines/engine.py.229: process: 0.84071 | load: 0.00293 [I 2022-10-10 09:52:50] ...trap/engines/engine.py.230: loss: 0.19308 [I 2022-10-10 09:52:51] ...optimizers/trijoint.py.151: total_norm: 46.729172 [I 2022-10-10 09:52:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 2010/2304 [I 2022-10-10 09:52:59] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:06 [I 2022-10-10 09:52:59] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00291 [I 2022-10-10 09:52:59] ...trap/engines/engine.py.230: loss: 0.18015 [I 2022-10-10 09:52:59] ...optimizers/trijoint.py.151: total_norm: 16.617456 [I 2022-10-10 09:53:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 2020/2304 [I 2022-10-10 09:53:07] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:03:58 [I 2022-10-10 09:53:07] ...trap/engines/engine.py.229: process: 0.82497 | load: 0.00295 [I 2022-10-10 09:53:07] ...trap/engines/engine.py.230: loss: 0.10826 [I 2022-10-10 09:53:07] ...optimizers/trijoint.py.151: total_norm: 7.518206 [I 2022-10-10 09:53:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 2030/2304 [I 2022-10-10 09:53:16] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:49 [I 2022-10-10 09:53:16] ...trap/engines/engine.py.229: process: 0.83140 | load: 0.00294 [I 2022-10-10 09:53:16] ...trap/engines/engine.py.230: loss: 0.12011 [I 2022-10-10 09:53:16] ...optimizers/trijoint.py.151: total_norm: 15.387330 [I 2022-10-10 09:53:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 2040/2304 [I 2022-10-10 09:53:24] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:41 [I 2022-10-10 09:53:24] ...trap/engines/engine.py.229: process: 0.83439 | load: 0.00292 [I 2022-10-10 09:53:24] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-10 09:53:24] ...optimizers/trijoint.py.151: total_norm: 7.948385 [I 2022-10-10 09:53:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 2050/2304 [I 2022-10-10 09:53:33] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:33 [I 2022-10-10 09:53:33] ...trap/engines/engine.py.229: process: 0.83756 | load: 0.00289 [I 2022-10-10 09:53:33] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-10 09:53:33] ...optimizers/trijoint.py.151: total_norm: 15.822534 [I 2022-10-10 09:53:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 2060/2304 [I 2022-10-10 09:53:41] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:25 [I 2022-10-10 09:53:41] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00289 [I 2022-10-10 09:53:41] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-10 09:53:41] ...optimizers/trijoint.py.151: total_norm: 12.740890 [I 2022-10-10 09:53:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 2070/2304 [I 2022-10-10 09:53:49] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:16 [I 2022-10-10 09:53:49] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00289 [I 2022-10-10 09:53:49] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-10 09:53:49] ...optimizers/trijoint.py.151: total_norm: 19.371830 [I 2022-10-10 09:53:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 2080/2304 [I 2022-10-10 09:53:58] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:09 [I 2022-10-10 09:53:58] ...trap/engines/engine.py.229: process: 0.85032 | load: 0.00294 [I 2022-10-10 09:53:58] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-10 09:53:58] ...optimizers/trijoint.py.151: total_norm: 15.833866 [I 2022-10-10 09:54:06] ...trap/engines/engine.py.224: train: epoch 67 | batch 2090/2304 [I 2022-10-10 09:54:06] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:02:58 [I 2022-10-10 09:54:06] ...trap/engines/engine.py.229: process: 0.82438 | load: 0.00290 [I 2022-10-10 09:54:06] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-10 09:54:06] ...optimizers/trijoint.py.151: total_norm: 19.240250 [I 2022-10-10 09:54:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 2100/2304 [I 2022-10-10 09:54:15] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:02:50 [I 2022-10-10 09:54:15] ...trap/engines/engine.py.229: process: 0.83492 | load: 0.00289 [I 2022-10-10 09:54:15] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-10 09:54:15] ...optimizers/trijoint.py.151: total_norm: 11.059495 [I 2022-10-10 09:54:23] ...trap/engines/engine.py.224: train: epoch 67 | batch 2110/2304 [I 2022-10-10 09:54:23] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:44 [I 2022-10-10 09:54:23] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00290 [I 2022-10-10 09:54:23] ...trap/engines/engine.py.230: loss: 0.11336 [I 2022-10-10 09:54:23] ...optimizers/trijoint.py.151: total_norm: 10.317646 [I 2022-10-10 09:54:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 2120/2304 [I 2022-10-10 09:54:32] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:02:34 [I 2022-10-10 09:54:32] ...trap/engines/engine.py.229: process: 0.83217 | load: 0.00289 [I 2022-10-10 09:54:32] ...trap/engines/engine.py.230: loss: 0.14952 [I 2022-10-10 09:54:32] ...optimizers/trijoint.py.151: total_norm: 13.790532 [I 2022-10-10 09:54:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 2130/2304 [I 2022-10-10 09:54:40] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:25 [I 2022-10-10 09:54:40] ...trap/engines/engine.py.229: process: 0.82717 | load: 0.00298 [I 2022-10-10 09:54:40] ...trap/engines/engine.py.230: loss: 0.21074 [I 2022-10-10 09:54:40] ...optimizers/trijoint.py.151: total_norm: 29.129007 [I 2022-10-10 09:54:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 2140/2304 [I 2022-10-10 09:54:48] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:17 [I 2022-10-10 09:54:48] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00292 [I 2022-10-10 09:54:48] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-10 09:54:48] ...optimizers/trijoint.py.151: total_norm: 10.403344 [I 2022-10-10 09:54:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 2150/2304 [I 2022-10-10 09:54:57] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:09 [I 2022-10-10 09:54:57] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00290 [I 2022-10-10 09:54:57] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-10 09:54:57] ...optimizers/trijoint.py.151: total_norm: 13.815536 [I 2022-10-10 09:55:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 2160/2304 [I 2022-10-10 09:55:05] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:00 [I 2022-10-10 09:55:05] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00293 [I 2022-10-10 09:55:05] ...trap/engines/engine.py.230: loss: 0.10545 [I 2022-10-10 09:55:05] ...optimizers/trijoint.py.151: total_norm: 8.588158 [I 2022-10-10 09:55:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 2170/2304 [I 2022-10-10 09:55:14] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:01:52 [I 2022-10-10 09:55:14] ...trap/engines/engine.py.229: process: 0.83337 | load: 0.00291 [I 2022-10-10 09:55:14] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-10 09:55:14] ...optimizers/trijoint.py.151: total_norm: 26.266649 [I 2022-10-10 09:55:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 2180/2304 [I 2022-10-10 09:55:22] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:44 [I 2022-10-10 09:55:22] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00292 [I 2022-10-10 09:55:22] ...trap/engines/engine.py.230: loss: 0.09581 [I 2022-10-10 09:55:22] ...optimizers/trijoint.py.151: total_norm: 11.842710 [I 2022-10-10 09:55:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 2190/2304 [I 2022-10-10 09:55:30] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:35 [I 2022-10-10 09:55:30] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00291 [I 2022-10-10 09:55:30] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-10 09:55:30] ...optimizers/trijoint.py.151: total_norm: 55.299294 [I 2022-10-10 09:55:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 2200/2304 [I 2022-10-10 09:55:39] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:27 [I 2022-10-10 09:55:39] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00291 [I 2022-10-10 09:55:39] ...trap/engines/engine.py.230: loss: 0.18626 [I 2022-10-10 09:55:39] ...optimizers/trijoint.py.151: total_norm: 13.556806 [I 2022-10-10 09:55:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 2210/2304 [I 2022-10-10 09:55:47] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:19 [I 2022-10-10 09:55:47] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00290 [I 2022-10-10 09:55:47] ...trap/engines/engine.py.230: loss: 0.14473 [I 2022-10-10 09:55:47] ...optimizers/trijoint.py.151: total_norm: 18.754690 [I 2022-10-10 09:55:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 2220/2304 [I 2022-10-10 09:55:56] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:11 [I 2022-10-10 09:55:56] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00291 [I 2022-10-10 09:55:56] ...trap/engines/engine.py.230: loss: 0.15691 [I 2022-10-10 09:55:56] ...optimizers/trijoint.py.151: total_norm: 16.467171 [I 2022-10-10 09:56:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 2230/2304 [I 2022-10-10 09:56:04] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:02 [I 2022-10-10 09:56:04] ...trap/engines/engine.py.229: process: 0.82662 | load: 0.00289 [I 2022-10-10 09:56:04] ...trap/engines/engine.py.230: loss: 0.14546 [I 2022-10-10 09:56:04] ...optimizers/trijoint.py.151: total_norm: 26.524632 [I 2022-10-10 09:56:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 2240/2304 [I 2022-10-10 09:56:13] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:00:54 [I 2022-10-10 09:56:13] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00289 [I 2022-10-10 09:56:13] ...trap/engines/engine.py.230: loss: 0.19425 [I 2022-10-10 09:56:13] ...optimizers/trijoint.py.151: total_norm: 16.697395 [I 2022-10-10 09:56:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 2250/2304 [I 2022-10-10 09:56:21] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:45 [I 2022-10-10 09:56:21] ...trap/engines/engine.py.229: process: 0.82648 | load: 0.00289 [I 2022-10-10 09:56:21] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-10 09:56:21] ...optimizers/trijoint.py.151: total_norm: 21.694384 [I 2022-10-10 09:56:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 2260/2304 [I 2022-10-10 09:56:29] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:00:36 [I 2022-10-10 09:56:29] ...trap/engines/engine.py.229: process: 0.83447 | load: 0.00289 [I 2022-10-10 09:56:29] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-10 09:56:29] ...optimizers/trijoint.py.151: total_norm: 15.507795 [I 2022-10-10 09:56:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 2270/2304 [I 2022-10-10 09:56:38] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:28 [I 2022-10-10 09:56:38] ...trap/engines/engine.py.229: process: 0.83275 | load: 0.00290 [I 2022-10-10 09:56:38] ...trap/engines/engine.py.230: loss: 0.22137 [I 2022-10-10 09:56:38] ...optimizers/trijoint.py.151: total_norm: 11.584409 [I 2022-10-10 09:56:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 2280/2304 [I 2022-10-10 09:56:46] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:20 [I 2022-10-10 09:56:46] ...trap/engines/engine.py.229: process: 0.83230 | load: 0.00288 [I 2022-10-10 09:56:46] ...trap/engines/engine.py.230: loss: 0.16715 [I 2022-10-10 09:56:46] ...optimizers/trijoint.py.151: total_norm: 8.638569 [I 2022-10-10 09:56:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 2290/2304 [I 2022-10-10 09:56:55] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:11 [I 2022-10-10 09:56:55] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00291 [I 2022-10-10 09:56:55] ...trap/engines/engine.py.230: loss: 0.20042 [I 2022-10-10 09:56:55] ...optimizers/trijoint.py.151: total_norm: 34.501968 [I 2022-10-10 09:57:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 2300/2304 [I 2022-10-10 09:57:03] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:03 [I 2022-10-10 09:57:03] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00012 [I 2022-10-10 09:57:03] ...trap/engines/engine.py.230: loss: 0.19391 [I 2022-10-10 09:57:03] ...optimizers/trijoint.py.151: total_norm: 16.064188 [I 2022-10-10 09:57:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 2304/2304 [I 2022-10-10 09:57:07] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:00 [I 2022-10-10 09:57:07] ...trap/engines/engine.py.229: process: 0.82752 | load: 0.00012 [I 2022-10-10 09:57:07] ...trap/engines/engine.py.230: loss: 0.23171 [I 2022-10-10 09:57:07] ...optimizers/trijoint.py.151: total_norm: 30.054262 [S 2022-10-10 09:57:08] ...trap/engines/engine.py.129: train_epoch.epoch: 67 [S 2022-10-10 09:57:08] ...trap/engines/engine.py.129: train_epoch.loss: 0.14319094286629538 [I 2022-10-10 09:57:08] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 09:57:09] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 67 [I 2022-10-10 09:57:09] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 09:57:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 09:57:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 09:57:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 09:57:13] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 09:57:13] ...trap/engines/engine.py.321: eval: epoch 67 | batch 0/510 [I 2022-10-10 09:57:13] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:23 [I 2022-10-10 09:57:13] ...trap/engines/engine.py.326: process: 4.51703 | load: 3.99188 [I 2022-10-10 09:57:19] ...trap/engines/engine.py.321: eval: epoch 67 | batch 10/510 [I 2022-10-10 09:57:19] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:27 [I 2022-10-10 09:57:19] ...trap/engines/engine.py.326: process: 0.54997 | load: 0.00017 [I 2022-10-10 09:57:24] ...trap/engines/engine.py.321: eval: epoch 67 | batch 20/510 [I 2022-10-10 09:57:24] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:45 [I 2022-10-10 09:57:24] ...trap/engines/engine.py.326: process: 0.50146 | load: 0.00019 [I 2022-10-10 09:57:31] ...trap/engines/engine.py.321: eval: epoch 67 | batch 30/510 [I 2022-10-10 09:57:31] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:16 [I 2022-10-10 09:57:31] ...trap/engines/engine.py.326: process: 0.75040 | load: 0.00018 [I 2022-10-10 09:57:37] ...trap/engines/engine.py.321: eval: epoch 67 | batch 40/510 [I 2022-10-10 09:57:37] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:41 [I 2022-10-10 09:57:37] ...trap/engines/engine.py.326: process: 0.56971 | load: 0.00017 [I 2022-10-10 09:57:42] ...trap/engines/engine.py.321: eval: epoch 67 | batch 50/510 [I 2022-10-10 09:57:42] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:04:23 [I 2022-10-10 09:57:42] ...trap/engines/engine.py.326: process: 0.57791 | load: 0.00018 [I 2022-10-10 09:57:48] ...trap/engines/engine.py.321: eval: epoch 67 | batch 60/510 [I 2022-10-10 09:57:48] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:04 [I 2022-10-10 09:57:48] ...trap/engines/engine.py.326: process: 0.47998 | load: 0.00017 [I 2022-10-10 09:57:53] ...trap/engines/engine.py.321: eval: epoch 67 | batch 70/510 [I 2022-10-10 09:57:53] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:25 [I 2022-10-10 09:57:53] ...trap/engines/engine.py.326: process: 0.44827 | load: 0.00019 [I 2022-10-10 09:57:58] ...trap/engines/engine.py.321: eval: epoch 67 | batch 80/510 [I 2022-10-10 09:57:58] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:41 [I 2022-10-10 09:57:58] ...trap/engines/engine.py.326: process: 0.47958 | load: 0.00017 [I 2022-10-10 09:58:02] ...trap/engines/engine.py.321: eval: epoch 67 | batch 90/510 [I 2022-10-10 09:58:02] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:07 [I 2022-10-10 09:58:02] ...trap/engines/engine.py.326: process: 0.46351 | load: 0.00019 [I 2022-10-10 09:58:06] ...trap/engines/engine.py.321: eval: epoch 67 | batch 100/510 [I 2022-10-10 09:58:06] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:49 [I 2022-10-10 09:58:06] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00019 [I 2022-10-10 09:58:09] ...trap/engines/engine.py.321: eval: epoch 67 | batch 110/510 [I 2022-10-10 09:58:09] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:11 [I 2022-10-10 09:58:09] ...trap/engines/engine.py.326: process: 0.31314 | load: 0.00018 [I 2022-10-10 09:58:13] ...trap/engines/engine.py.321: eval: epoch 67 | batch 120/510 [I 2022-10-10 09:58:13] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:03 [I 2022-10-10 09:58:13] ...trap/engines/engine.py.326: process: 0.31331 | load: 0.00018 [I 2022-10-10 09:58:16] ...trap/engines/engine.py.321: eval: epoch 67 | batch 130/510 [I 2022-10-10 09:58:16] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:00 [I 2022-10-10 09:58:16] ...trap/engines/engine.py.326: process: 0.32303 | load: 0.00018 [I 2022-10-10 09:58:19] ...trap/engines/engine.py.321: eval: epoch 67 | batch 140/510 [I 2022-10-10 09:58:19] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:56 [I 2022-10-10 09:58:19] ...trap/engines/engine.py.326: process: 0.31056 | load: 0.00017 [I 2022-10-10 09:58:22] ...trap/engines/engine.py.321: eval: epoch 67 | batch 150/510 [I 2022-10-10 09:58:22] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:54 [I 2022-10-10 09:58:22] ...trap/engines/engine.py.326: process: 0.31150 | load: 0.00019 [I 2022-10-10 09:58:25] ...trap/engines/engine.py.321: eval: epoch 67 | batch 160/510 [I 2022-10-10 09:58:25] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:50 [I 2022-10-10 09:58:25] ...trap/engines/engine.py.326: process: 0.31561 | load: 0.00019 [I 2022-10-10 09:58:28] ...trap/engines/engine.py.321: eval: epoch 67 | batch 170/510 [I 2022-10-10 09:58:28] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:47 [I 2022-10-10 09:58:28] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00018 [I 2022-10-10 09:58:32] ...trap/engines/engine.py.321: eval: epoch 67 | batch 180/510 [I 2022-10-10 09:58:32] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:45 [I 2022-10-10 09:58:32] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00019 [I 2022-10-10 09:58:35] ...trap/engines/engine.py.321: eval: epoch 67 | batch 190/510 [I 2022-10-10 09:58:35] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:41 [I 2022-10-10 09:58:35] ...trap/engines/engine.py.326: process: 0.31506 | load: 0.00018 [I 2022-10-10 09:58:38] ...trap/engines/engine.py.321: eval: epoch 67 | batch 200/510 [I 2022-10-10 09:58:38] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:38 [I 2022-10-10 09:58:38] ...trap/engines/engine.py.326: process: 0.31577 | load: 0.00018 [I 2022-10-10 09:58:41] ...trap/engines/engine.py.321: eval: epoch 67 | batch 210/510 [I 2022-10-10 09:58:41] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:36 [I 2022-10-10 09:58:41] ...trap/engines/engine.py.326: process: 0.32508 | load: 0.00018 [I 2022-10-10 09:58:44] ...trap/engines/engine.py.321: eval: epoch 67 | batch 220/510 [I 2022-10-10 09:58:44] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:32 [I 2022-10-10 09:58:44] ...trap/engines/engine.py.326: process: 0.31536 | load: 0.00018 [I 2022-10-10 09:58:48] ...trap/engines/engine.py.321: eval: epoch 67 | batch 230/510 [I 2022-10-10 09:58:48] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-10 09:58:48] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00019 [I 2022-10-10 09:58:51] ...trap/engines/engine.py.321: eval: epoch 67 | batch 240/510 [I 2022-10-10 09:58:51] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:26 [I 2022-10-10 09:58:51] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00019 [I 2022-10-10 09:58:54] ...trap/engines/engine.py.321: eval: epoch 67 | batch 250/510 [I 2022-10-10 09:58:54] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:22 [I 2022-10-10 09:58:54] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00018 [I 2022-10-10 09:58:57] ...trap/engines/engine.py.321: eval: epoch 67 | batch 260/510 [I 2022-10-10 09:58:57] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:19 [I 2022-10-10 09:58:57] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00018 [I 2022-10-10 09:59:00] ...trap/engines/engine.py.321: eval: epoch 67 | batch 270/510 [I 2022-10-10 09:59:00] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-10 09:59:00] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00019 [I 2022-10-10 09:59:04] ...trap/engines/engine.py.321: eval: epoch 67 | batch 280/510 [I 2022-10-10 09:59:04] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-10 09:59:04] ...trap/engines/engine.py.326: process: 0.31954 | load: 0.00019 [I 2022-10-10 09:59:07] ...trap/engines/engine.py.321: eval: epoch 67 | batch 290/510 [I 2022-10-10 09:59:07] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-10 09:59:07] ...trap/engines/engine.py.326: process: 0.32479 | load: 0.00018 [I 2022-10-10 09:59:10] ...trap/engines/engine.py.321: eval: epoch 67 | batch 300/510 [I 2022-10-10 09:59:10] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:07 [I 2022-10-10 09:59:10] ...trap/engines/engine.py.326: process: 0.32511 | load: 0.00019 [I 2022-10-10 09:59:13] ...trap/engines/engine.py.321: eval: epoch 67 | batch 310/510 [I 2022-10-10 09:59:13] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:03 [I 2022-10-10 09:59:13] ...trap/engines/engine.py.326: process: 0.31496 | load: 0.00019 [I 2022-10-10 09:59:16] ...trap/engines/engine.py.321: eval: epoch 67 | batch 320/510 [I 2022-10-10 09:59:16] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:00 [I 2022-10-10 09:59:16] ...trap/engines/engine.py.326: process: 0.31599 | load: 0.00019 [I 2022-10-10 09:59:20] ...trap/engines/engine.py.321: eval: epoch 67 | batch 330/510 [I 2022-10-10 09:59:20] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:57 [I 2022-10-10 09:59:20] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00019 [I 2022-10-10 09:59:23] ...trap/engines/engine.py.321: eval: epoch 67 | batch 340/510 [I 2022-10-10 09:59:23] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-10 09:59:23] ...trap/engines/engine.py.326: process: 0.31885 | load: 0.00019 [I 2022-10-10 09:59:26] ...trap/engines/engine.py.321: eval: epoch 67 | batch 350/510 [I 2022-10-10 09:59:26] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-10 09:59:26] ...trap/engines/engine.py.326: process: 0.31358 | load: 0.00018 [I 2022-10-10 09:59:29] ...trap/engines/engine.py.321: eval: epoch 67 | batch 360/510 [I 2022-10-10 09:59:29] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:47 [I 2022-10-10 09:59:29] ...trap/engines/engine.py.326: process: 0.32427 | load: 0.00019 [I 2022-10-10 09:59:32] ...trap/engines/engine.py.321: eval: epoch 67 | batch 370/510 [I 2022-10-10 09:59:32] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-10 09:59:32] ...trap/engines/engine.py.326: process: 0.32391 | load: 0.00018 [I 2022-10-10 09:59:36] ...trap/engines/engine.py.321: eval: epoch 67 | batch 380/510 [I 2022-10-10 09:59:36] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-10 09:59:36] ...trap/engines/engine.py.326: process: 0.31622 | load: 0.00019 [I 2022-10-10 09:59:39] ...trap/engines/engine.py.321: eval: epoch 67 | batch 390/510 [I 2022-10-10 09:59:39] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-10 09:59:39] ...trap/engines/engine.py.326: process: 0.32617 | load: 0.00019 [I 2022-10-10 09:59:42] ...trap/engines/engine.py.321: eval: epoch 67 | batch 400/510 [I 2022-10-10 09:59:42] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:35 [I 2022-10-10 09:59:42] ...trap/engines/engine.py.326: process: 0.31774 | load: 0.00019 [I 2022-10-10 09:59:45] ...trap/engines/engine.py.321: eval: epoch 67 | batch 410/510 [I 2022-10-10 09:59:45] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:31 [I 2022-10-10 09:59:45] ...trap/engines/engine.py.326: process: 0.32449 | load: 0.00018 [I 2022-10-10 09:59:48] ...trap/engines/engine.py.321: eval: epoch 67 | batch 420/510 [I 2022-10-10 09:59:48] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:28 [I 2022-10-10 09:59:48] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00019 [I 2022-10-10 09:59:51] ...trap/engines/engine.py.321: eval: epoch 67 | batch 430/510 [I 2022-10-10 09:59:51] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-10 09:59:51] ...trap/engines/engine.py.326: process: 0.31168 | load: 0.00019 [I 2022-10-10 09:59:55] ...trap/engines/engine.py.321: eval: epoch 67 | batch 440/510 [I 2022-10-10 09:59:55] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-10 09:59:55] ...trap/engines/engine.py.326: process: 0.32338 | load: 0.00018 [I 2022-10-10 09:59:58] ...trap/engines/engine.py.321: eval: epoch 67 | batch 450/510 [I 2022-10-10 09:59:58] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-10 09:59:58] ...trap/engines/engine.py.326: process: 0.31636 | load: 0.00019 [I 2022-10-10 10:00:01] ...trap/engines/engine.py.321: eval: epoch 67 | batch 460/510 [I 2022-10-10 10:00:01] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:16 [I 2022-10-10 10:00:01] ...trap/engines/engine.py.326: process: 0.32447 | load: 0.00018 [I 2022-10-10 10:00:04] ...trap/engines/engine.py.321: eval: epoch 67 | batch 470/510 [I 2022-10-10 10:00:04] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-10 10:00:04] ...trap/engines/engine.py.326: process: 0.32713 | load: 0.00018 [I 2022-10-10 10:00:07] ...trap/engines/engine.py.321: eval: epoch 67 | batch 480/510 [I 2022-10-10 10:00:07] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-10 10:00:07] ...trap/engines/engine.py.326: process: 0.31322 | load: 0.00019 [I 2022-10-10 10:00:11] ...trap/engines/engine.py.321: eval: epoch 67 | batch 490/510 [I 2022-10-10 10:00:11] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-10 10:00:11] ...trap/engines/engine.py.326: process: 0.31446 | load: 0.00018 [I 2022-10-10 10:00:14] ...trap/engines/engine.py.321: eval: epoch 67 | batch 500/510 [I 2022-10-10 10:00:14] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:03 [I 2022-10-10 10:00:14] ...trap/engines/engine.py.326: process: 0.31952 | load: 0.00019 [I 2022-10-10 10:00:17] ...trap/engines/engine.py.321: eval: epoch 67 | batch 510/510 [I 2022-10-10 10:00:17] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 10:00:17] ...trap/engines/engine.py.326: process: 0.31454 | load: 0.00011 [S 2022-10-10 10:00:18] ...trap/engines/engine.py.132: eval_epoch.epoch: 67 [I 2022-10-10 10:00:18] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.732500 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909500 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940200 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724900 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906600 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936300 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012691 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006607 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007859 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012341 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010062 [S 2022-10-10 10:00:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007875 [I 2022-10-10 10:01:00] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 10:01:00] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 10:01:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 10:01:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 10:01:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 10:01:04] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 10:01:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 10:01:10] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 10:01:10] ...trap/engines/engine.py.163: Training model on trainset for epoch 68 [I 2022-10-10 10:01:10] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 10:01:13] ...trap/engines/engine.py.224: train: epoch 68 | batch 0/2304 [I 2022-10-10 10:01:13] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:20 [I 2022-10-10 10:01:13] ...trap/engines/engine.py.229: process: 3.00385 | load: 2.11468 [I 2022-10-10 10:01:13] ...trap/engines/engine.py.230: loss: 0.19072 [I 2022-10-10 10:01:13] ...optimizers/trijoint.py.151: total_norm: 10.825888 [I 2022-10-10 10:01:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 10/2304 [I 2022-10-10 10:01:21] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:15 [I 2022-10-10 10:01:21] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00483 [I 2022-10-10 10:01:21] ...trap/engines/engine.py.230: loss: 0.16477 [I 2022-10-10 10:01:21] ...optimizers/trijoint.py.151: total_norm: 15.405290 [I 2022-10-10 10:01:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 20/2304 [I 2022-10-10 10:01:30] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:30 [I 2022-10-10 10:01:30] ...trap/engines/engine.py.229: process: 0.86074 | load: 0.00529 [I 2022-10-10 10:01:30] ...trap/engines/engine.py.230: loss: 0.18173 [I 2022-10-10 10:01:30] ...optimizers/trijoint.py.151: total_norm: 21.895784 [I 2022-10-10 10:01:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 30/2304 [I 2022-10-10 10:01:38] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:13 [I 2022-10-10 10:01:38] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00377 [I 2022-10-10 10:01:38] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-10 10:01:38] ...optimizers/trijoint.py.151: total_norm: 18.138470 [I 2022-10-10 10:01:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 40/2304 [I 2022-10-10 10:01:47] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:59 [I 2022-10-10 10:01:47] ...trap/engines/engine.py.229: process: 0.84445 | load: 0.00384 [I 2022-10-10 10:01:47] ...trap/engines/engine.py.230: loss: 0.11166 [I 2022-10-10 10:01:47] ...optimizers/trijoint.py.151: total_norm: 9.915531 [I 2022-10-10 10:01:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 50/2304 [I 2022-10-10 10:01:55] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:45 [I 2022-10-10 10:01:55] ...trap/engines/engine.py.229: process: 0.84272 | load: 0.00379 [I 2022-10-10 10:01:55] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-10 10:01:55] ...optimizers/trijoint.py.151: total_norm: 20.289810 [I 2022-10-10 10:02:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 60/2304 [I 2022-10-10 10:02:04] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:40 [I 2022-10-10 10:02:04] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00348 [I 2022-10-10 10:02:04] ...trap/engines/engine.py.230: loss: 0.15832 [I 2022-10-10 10:02:04] ...optimizers/trijoint.py.151: total_norm: 15.751637 [I 2022-10-10 10:02:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 70/2304 [I 2022-10-10 10:02:12] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:31 [I 2022-10-10 10:02:12] ...trap/engines/engine.py.229: process: 0.85323 | load: 0.00314 [I 2022-10-10 10:02:12] ...trap/engines/engine.py.230: loss: 0.15277 [I 2022-10-10 10:02:12] ...optimizers/trijoint.py.151: total_norm: 14.416047 [I 2022-10-10 10:02:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 80/2304 [I 2022-10-10 10:02:21] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:23 [I 2022-10-10 10:02:21] ...trap/engines/engine.py.229: process: 0.85741 | load: 0.00375 [I 2022-10-10 10:02:21] ...trap/engines/engine.py.230: loss: 0.17042 [I 2022-10-10 10:02:21] ...optimizers/trijoint.py.151: total_norm: 19.463184 [I 2022-10-10 10:02:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 90/2304 [I 2022-10-10 10:02:29] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:38 [I 2022-10-10 10:02:29] ...trap/engines/engine.py.229: process: 0.86733 | load: 0.00343 [I 2022-10-10 10:02:29] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-10 10:02:29] ...optimizers/trijoint.py.151: total_norm: 15.506873 [I 2022-10-10 10:02:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 100/2304 [I 2022-10-10 10:02:38] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:17 [I 2022-10-10 10:02:38] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00300 [I 2022-10-10 10:02:38] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-10 10:02:38] ...optimizers/trijoint.py.151: total_norm: 26.099642 [I 2022-10-10 10:02:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 110/2304 [I 2022-10-10 10:02:46] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:21 [I 2022-10-10 10:02:46] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00336 [I 2022-10-10 10:02:46] ...trap/engines/engine.py.230: loss: 0.22571 [I 2022-10-10 10:02:46] ...optimizers/trijoint.py.151: total_norm: 17.932062 [I 2022-10-10 10:02:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 120/2304 [I 2022-10-10 10:02:55] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:47 [I 2022-10-10 10:02:55] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00343 [I 2022-10-10 10:02:55] ...trap/engines/engine.py.230: loss: 0.21348 [I 2022-10-10 10:02:55] ...optimizers/trijoint.py.151: total_norm: 9.350097 [I 2022-10-10 10:03:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 130/2304 [I 2022-10-10 10:03:03] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:46 [I 2022-10-10 10:03:03] ...trap/engines/engine.py.229: process: 0.85986 | load: 0.00300 [I 2022-10-10 10:03:03] ...trap/engines/engine.py.230: loss: 0.15133 [I 2022-10-10 10:03:04] ...optimizers/trijoint.py.151: total_norm: 8.096986 [I 2022-10-10 10:03:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 140/2304 [I 2022-10-10 10:03:12] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:26 [I 2022-10-10 10:03:12] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00314 [I 2022-10-10 10:03:12] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-10 10:03:12] ...optimizers/trijoint.py.151: total_norm: 24.730263 [I 2022-10-10 10:03:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 150/2304 [I 2022-10-10 10:03:20] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:19 [I 2022-10-10 10:03:20] ...trap/engines/engine.py.229: process: 0.83744 | load: 0.00295 [I 2022-10-10 10:03:20] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-10 10:03:20] ...optimizers/trijoint.py.151: total_norm: 12.339843 [I 2022-10-10 10:03:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 160/2304 [I 2022-10-10 10:03:29] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:03 [I 2022-10-10 10:03:29] ...trap/engines/engine.py.229: process: 0.83548 | load: 0.00304 [I 2022-10-10 10:03:29] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-10 10:03:29] ...optimizers/trijoint.py.151: total_norm: 19.179667 [I 2022-10-10 10:03:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 170/2304 [I 2022-10-10 10:03:37] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:01 [I 2022-10-10 10:03:37] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00323 [I 2022-10-10 10:03:37] ...trap/engines/engine.py.230: loss: 0.13974 [I 2022-10-10 10:03:37] ...optimizers/trijoint.py.151: total_norm: 9.681016 [I 2022-10-10 10:03:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 180/2304 [I 2022-10-10 10:03:46] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:29:45 [I 2022-10-10 10:03:46] ...trap/engines/engine.py.229: process: 0.83508 | load: 0.00313 [I 2022-10-10 10:03:46] ...trap/engines/engine.py.230: loss: 0.09246 [I 2022-10-10 10:03:46] ...optimizers/trijoint.py.151: total_norm: 11.442183 [I 2022-10-10 10:03:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 190/2304 [I 2022-10-10 10:03:54] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:44 [I 2022-10-10 10:03:54] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00301 [I 2022-10-10 10:03:54] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-10 10:03:54] ...optimizers/trijoint.py.151: total_norm: 14.400189 [I 2022-10-10 10:04:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 200/2304 [I 2022-10-10 10:04:03] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:52 [I 2022-10-10 10:04:03] ...trap/engines/engine.py.229: process: 0.86195 | load: 0.00303 [I 2022-10-10 10:04:03] ...trap/engines/engine.py.230: loss: 0.12329 [I 2022-10-10 10:04:03] ...optimizers/trijoint.py.151: total_norm: 16.299028 [I 2022-10-10 10:04:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 210/2304 [I 2022-10-10 10:04:11] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:40 [I 2022-10-10 10:04:11] ...trap/engines/engine.py.229: process: 0.84948 | load: 0.00317 [I 2022-10-10 10:04:11] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-10 10:04:11] ...optimizers/trijoint.py.151: total_norm: 8.574814 [I 2022-10-10 10:04:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 220/2304 [I 2022-10-10 10:04:20] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:23 [I 2022-10-10 10:04:20] ...trap/engines/engine.py.229: process: 0.83377 | load: 0.00300 [I 2022-10-10 10:04:20] ...trap/engines/engine.py.230: loss: 0.15144 [I 2022-10-10 10:04:20] ...optimizers/trijoint.py.151: total_norm: 17.562889 [I 2022-10-10 10:04:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 230/2304 [I 2022-10-10 10:04:28] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:20 [I 2022-10-10 10:04:28] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00295 [I 2022-10-10 10:04:28] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-10 10:04:28] ...optimizers/trijoint.py.151: total_norm: 11.544022 [I 2022-10-10 10:04:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 240/2304 [I 2022-10-10 10:04:37] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:05 [I 2022-10-10 10:04:37] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00299 [I 2022-10-10 10:04:37] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-10 10:04:37] ...optimizers/trijoint.py.151: total_norm: 22.484386 [I 2022-10-10 10:04:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 250/2304 [I 2022-10-10 10:04:45] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:28:52 [I 2022-10-10 10:04:45] ...trap/engines/engine.py.229: process: 0.82462 | load: 0.00288 [I 2022-10-10 10:04:45] ...trap/engines/engine.py.230: loss: 0.15285 [I 2022-10-10 10:04:45] ...optimizers/trijoint.py.151: total_norm: 15.826484 [I 2022-10-10 10:04:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 260/2304 [I 2022-10-10 10:04:54] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:44 [I 2022-10-10 10:04:54] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00305 [I 2022-10-10 10:04:54] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-10 10:04:54] ...optimizers/trijoint.py.151: total_norm: 15.979812 [I 2022-10-10 10:05:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 270/2304 [I 2022-10-10 10:05:02] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:48 [I 2022-10-10 10:05:02] ...trap/engines/engine.py.229: process: 0.85181 | load: 0.00302 [I 2022-10-10 10:05:02] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-10 10:05:02] ...optimizers/trijoint.py.151: total_norm: 6.209639 [I 2022-10-10 10:05:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 280/2304 [I 2022-10-10 10:05:11] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:34 [I 2022-10-10 10:05:11] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00288 [I 2022-10-10 10:05:11] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-10 10:05:11] ...optimizers/trijoint.py.151: total_norm: 25.567141 [I 2022-10-10 10:05:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 290/2304 [I 2022-10-10 10:05:19] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:37 [I 2022-10-10 10:05:19] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00299 [I 2022-10-10 10:05:19] ...trap/engines/engine.py.230: loss: 0.08087 [I 2022-10-10 10:05:19] ...optimizers/trijoint.py.151: total_norm: 8.657410 [I 2022-10-10 10:05:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 300/2304 [I 2022-10-10 10:05:28] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:28:10 [I 2022-10-10 10:05:28] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00293 [I 2022-10-10 10:05:28] ...trap/engines/engine.py.230: loss: 0.16308 [I 2022-10-10 10:05:28] ...optimizers/trijoint.py.151: total_norm: 14.225807 [I 2022-10-10 10:05:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 310/2304 [I 2022-10-10 10:05:36] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:08 [I 2022-10-10 10:05:36] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00296 [I 2022-10-10 10:05:36] ...trap/engines/engine.py.230: loss: 0.09449 [I 2022-10-10 10:05:36] ...optimizers/trijoint.py.151: total_norm: 9.372698 [I 2022-10-10 10:05:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 320/2304 [I 2022-10-10 10:05:45] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:28:00 [I 2022-10-10 10:05:45] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00287 [I 2022-10-10 10:05:45] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-10 10:05:45] ...optimizers/trijoint.py.151: total_norm: 12.706239 [I 2022-10-10 10:05:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 330/2304 [I 2022-10-10 10:05:53] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:27:52 [I 2022-10-10 10:05:53] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00287 [I 2022-10-10 10:05:53] ...trap/engines/engine.py.230: loss: 0.13013 [I 2022-10-10 10:05:53] ...optimizers/trijoint.py.151: total_norm: 21.364937 [I 2022-10-10 10:06:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 340/2304 [I 2022-10-10 10:06:02] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:27:47 [I 2022-10-10 10:06:02] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00287 [I 2022-10-10 10:06:02] ...trap/engines/engine.py.230: loss: 0.11264 [I 2022-10-10 10:06:02] ...optimizers/trijoint.py.151: total_norm: 6.966431 [I 2022-10-10 10:06:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 350/2304 [I 2022-10-10 10:06:10] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:42 [I 2022-10-10 10:06:10] ...trap/engines/engine.py.229: process: 0.84836 | load: 0.00300 [I 2022-10-10 10:06:10] ...trap/engines/engine.py.230: loss: 0.19730 [I 2022-10-10 10:06:10] ...optimizers/trijoint.py.151: total_norm: 14.677426 [I 2022-10-10 10:06:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 360/2304 [I 2022-10-10 10:06:19] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:27:22 [I 2022-10-10 10:06:19] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00303 [I 2022-10-10 10:06:19] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-10 10:06:19] ...optimizers/trijoint.py.151: total_norm: 28.791069 [I 2022-10-10 10:06:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 370/2304 [I 2022-10-10 10:06:27] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:09 [I 2022-10-10 10:06:27] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00294 [I 2022-10-10 10:06:27] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-10 10:06:27] ...optimizers/trijoint.py.151: total_norm: 14.387991 [I 2022-10-10 10:06:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 380/2304 [I 2022-10-10 10:06:36] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:27:14 [I 2022-10-10 10:06:36] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00309 [I 2022-10-10 10:06:36] ...trap/engines/engine.py.230: loss: 0.14630 [I 2022-10-10 10:06:36] ...optimizers/trijoint.py.151: total_norm: 15.579322 [I 2022-10-10 10:06:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 390/2304 [I 2022-10-10 10:06:44] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:03 [I 2022-10-10 10:06:44] ...trap/engines/engine.py.229: process: 0.86332 | load: 0.00295 [I 2022-10-10 10:06:44] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-10 10:06:44] ...optimizers/trijoint.py.151: total_norm: 19.406284 [I 2022-10-10 10:06:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 400/2304 [I 2022-10-10 10:06:53] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:27:06 [I 2022-10-10 10:06:53] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00295 [I 2022-10-10 10:06:53] ...trap/engines/engine.py.230: loss: 0.09880 [I 2022-10-10 10:06:53] ...optimizers/trijoint.py.151: total_norm: 11.616810 [I 2022-10-10 10:07:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 410/2304 [I 2022-10-10 10:07:01] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:45 [I 2022-10-10 10:07:01] ...trap/engines/engine.py.229: process: 0.84368 | load: 0.00297 [I 2022-10-10 10:07:01] ...trap/engines/engine.py.230: loss: 0.09958 [I 2022-10-10 10:07:01] ...optimizers/trijoint.py.151: total_norm: 9.025554 [I 2022-10-10 10:07:09] ...trap/engines/engine.py.224: train: epoch 68 | batch 420/2304 [I 2022-10-10 10:07:09] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:26:38 [I 2022-10-10 10:07:09] ...trap/engines/engine.py.229: process: 0.84190 | load: 0.00297 [I 2022-10-10 10:07:09] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-10 10:07:09] ...optimizers/trijoint.py.151: total_norm: 16.982841 [I 2022-10-10 10:07:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 430/2304 [I 2022-10-10 10:07:18] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:21 [I 2022-10-10 10:07:18] ...trap/engines/engine.py.229: process: 0.83858 | load: 0.00295 [I 2022-10-10 10:07:18] ...trap/engines/engine.py.230: loss: 0.12199 [I 2022-10-10 10:07:18] ...optimizers/trijoint.py.151: total_norm: 15.219481 [I 2022-10-10 10:07:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 440/2304 [I 2022-10-10 10:07:26] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:17 [I 2022-10-10 10:07:26] ...trap/engines/engine.py.229: process: 0.84983 | load: 0.00298 [I 2022-10-10 10:07:26] ...trap/engines/engine.py.230: loss: 0.13791 [I 2022-10-10 10:07:26] ...optimizers/trijoint.py.151: total_norm: 20.256855 [I 2022-10-10 10:07:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 450/2304 [I 2022-10-10 10:07:35] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:02 [I 2022-10-10 10:07:35] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00299 [I 2022-10-10 10:07:35] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-10 10:07:35] ...optimizers/trijoint.py.151: total_norm: 16.136742 [I 2022-10-10 10:07:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 460/2304 [I 2022-10-10 10:07:43] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:25:56 [I 2022-10-10 10:07:43] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00300 [I 2022-10-10 10:07:43] ...trap/engines/engine.py.230: loss: 0.14961 [I 2022-10-10 10:07:43] ...optimizers/trijoint.py.151: total_norm: 17.991774 [I 2022-10-10 10:07:52] ...trap/engines/engine.py.224: train: epoch 68 | batch 470/2304 [I 2022-10-10 10:07:52] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:25:53 [I 2022-10-10 10:07:52] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00314 [I 2022-10-10 10:07:52] ...trap/engines/engine.py.230: loss: 0.18014 [I 2022-10-10 10:07:52] ...optimizers/trijoint.py.151: total_norm: 33.064003 [I 2022-10-10 10:08:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 480/2304 [I 2022-10-10 10:08:00] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:48 [I 2022-10-10 10:08:00] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00289 [I 2022-10-10 10:08:00] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-10 10:08:00] ...optimizers/trijoint.py.151: total_norm: 9.651286 [I 2022-10-10 10:08:09] ...trap/engines/engine.py.224: train: epoch 68 | batch 490/2304 [I 2022-10-10 10:08:09] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:25:38 [I 2022-10-10 10:08:09] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00297 [I 2022-10-10 10:08:09] ...trap/engines/engine.py.230: loss: 0.11544 [I 2022-10-10 10:08:09] ...optimizers/trijoint.py.151: total_norm: 8.459147 [I 2022-10-10 10:08:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 500/2304 [I 2022-10-10 10:08:17] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:21 [I 2022-10-10 10:08:17] ...trap/engines/engine.py.229: process: 0.83513 | load: 0.00293 [I 2022-10-10 10:08:17] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-10 10:08:17] ...optimizers/trijoint.py.151: total_norm: 10.107203 [I 2022-10-10 10:08:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 510/2304 [I 2022-10-10 10:08:26] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:23 [I 2022-10-10 10:08:26] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00291 [I 2022-10-10 10:08:26] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-10 10:08:26] ...optimizers/trijoint.py.151: total_norm: 14.812753 [I 2022-10-10 10:08:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 520/2304 [I 2022-10-10 10:08:34] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:06 [I 2022-10-10 10:08:34] ...trap/engines/engine.py.229: process: 0.83738 | load: 0.00290 [I 2022-10-10 10:08:34] ...trap/engines/engine.py.230: loss: 0.17786 [I 2022-10-10 10:08:34] ...optimizers/trijoint.py.151: total_norm: 24.553205 [I 2022-10-10 10:08:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 530/2304 [I 2022-10-10 10:08:43] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:01 [I 2022-10-10 10:08:43] ...trap/engines/engine.py.229: process: 0.87443 | load: 0.00290 [I 2022-10-10 10:08:43] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-10 10:08:43] ...optimizers/trijoint.py.151: total_norm: 18.041687 [I 2022-10-10 10:08:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 540/2304 [I 2022-10-10 10:08:51] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:24:58 [I 2022-10-10 10:08:51] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00286 [I 2022-10-10 10:08:51] ...trap/engines/engine.py.230: loss: 0.10896 [I 2022-10-10 10:08:51] ...optimizers/trijoint.py.151: total_norm: 21.583029 [I 2022-10-10 10:09:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 550/2304 [I 2022-10-10 10:09:00] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:24:51 [I 2022-10-10 10:09:00] ...trap/engines/engine.py.229: process: 0.85265 | load: 0.00287 [I 2022-10-10 10:09:00] ...trap/engines/engine.py.230: loss: 0.14074 [I 2022-10-10 10:09:00] ...optimizers/trijoint.py.151: total_norm: 18.673241 [I 2022-10-10 10:09:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 560/2304 [I 2022-10-10 10:09:08] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:32 [I 2022-10-10 10:09:08] ...trap/engines/engine.py.229: process: 0.84546 | load: 0.00291 [I 2022-10-10 10:09:08] ...trap/engines/engine.py.230: loss: 0.12194 [I 2022-10-10 10:09:08] ...optimizers/trijoint.py.151: total_norm: 16.100002 [I 2022-10-10 10:09:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 570/2304 [I 2022-10-10 10:09:17] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:26 [I 2022-10-10 10:09:17] ...trap/engines/engine.py.229: process: 0.86859 | load: 0.00290 [I 2022-10-10 10:09:17] ...trap/engines/engine.py.230: loss: 0.17568 [I 2022-10-10 10:09:17] ...optimizers/trijoint.py.151: total_norm: 20.858299 [I 2022-10-10 10:09:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 580/2304 [I 2022-10-10 10:09:25] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:20 [I 2022-10-10 10:09:25] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00294 [I 2022-10-10 10:09:25] ...trap/engines/engine.py.230: loss: 0.10509 [I 2022-10-10 10:09:25] ...optimizers/trijoint.py.151: total_norm: 11.586513 [I 2022-10-10 10:09:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 590/2304 [I 2022-10-10 10:09:34] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:03 [I 2022-10-10 10:09:34] ...trap/engines/engine.py.229: process: 0.83249 | load: 0.00295 [I 2022-10-10 10:09:34] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-10 10:09:34] ...optimizers/trijoint.py.151: total_norm: 16.238726 [I 2022-10-10 10:09:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 600/2304 [I 2022-10-10 10:09:42] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:23:54 [I 2022-10-10 10:09:42] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00289 [I 2022-10-10 10:09:42] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-10 10:09:42] ...optimizers/trijoint.py.151: total_norm: 18.319752 [I 2022-10-10 10:09:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 610/2304 [I 2022-10-10 10:09:51] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:23:54 [I 2022-10-10 10:09:51] ...trap/engines/engine.py.229: process: 0.84991 | load: 0.00295 [I 2022-10-10 10:09:51] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-10 10:09:51] ...optimizers/trijoint.py.151: total_norm: 17.013510 [I 2022-10-10 10:09:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 620/2304 [I 2022-10-10 10:09:59] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:23:46 [I 2022-10-10 10:09:59] ...trap/engines/engine.py.229: process: 0.85881 | load: 0.00290 [I 2022-10-10 10:09:59] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-10 10:09:59] ...optimizers/trijoint.py.151: total_norm: 14.614915 [I 2022-10-10 10:10:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 630/2304 [I 2022-10-10 10:10:08] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:49 [I 2022-10-10 10:10:08] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00290 [I 2022-10-10 10:10:08] ...trap/engines/engine.py.230: loss: 0.09022 [I 2022-10-10 10:10:08] ...optimizers/trijoint.py.151: total_norm: 9.065473 [I 2022-10-10 10:10:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 640/2304 [I 2022-10-10 10:10:16] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:31 [I 2022-10-10 10:10:16] ...trap/engines/engine.py.229: process: 0.87492 | load: 0.00289 [I 2022-10-10 10:10:16] ...trap/engines/engine.py.230: loss: 0.17157 [I 2022-10-10 10:10:16] ...optimizers/trijoint.py.151: total_norm: 20.937445 [I 2022-10-10 10:10:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 650/2304 [I 2022-10-10 10:10:25] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:19 [I 2022-10-10 10:10:25] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00290 [I 2022-10-10 10:10:25] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-10 10:10:25] ...optimizers/trijoint.py.151: total_norm: 13.008055 [I 2022-10-10 10:10:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 660/2304 [I 2022-10-10 10:10:33] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:08 [I 2022-10-10 10:10:33] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00294 [I 2022-10-10 10:10:33] ...trap/engines/engine.py.230: loss: 0.09568 [I 2022-10-10 10:10:33] ...optimizers/trijoint.py.151: total_norm: 7.505010 [I 2022-10-10 10:10:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 670/2304 [I 2022-10-10 10:10:41] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:22:59 [I 2022-10-10 10:10:41] ...trap/engines/engine.py.229: process: 0.83829 | load: 0.00288 [I 2022-10-10 10:10:41] ...trap/engines/engine.py.230: loss: 0.12277 [I 2022-10-10 10:10:41] ...optimizers/trijoint.py.151: total_norm: 11.402094 [I 2022-10-10 10:10:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 680/2304 [I 2022-10-10 10:10:50] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:22:51 [I 2022-10-10 10:10:50] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00288 [I 2022-10-10 10:10:50] ...trap/engines/engine.py.230: loss: 0.13043 [I 2022-10-10 10:10:50] ...optimizers/trijoint.py.151: total_norm: 19.574934 [I 2022-10-10 10:10:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 690/2304 [I 2022-10-10 10:10:58] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:22:52 [I 2022-10-10 10:10:58] ...trap/engines/engine.py.229: process: 0.83739 | load: 0.00292 [I 2022-10-10 10:10:58] ...trap/engines/engine.py.230: loss: 0.10897 [I 2022-10-10 10:10:58] ...optimizers/trijoint.py.151: total_norm: 9.146683 [I 2022-10-10 10:11:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 700/2304 [I 2022-10-10 10:11:07] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:22:40 [I 2022-10-10 10:11:07] ...trap/engines/engine.py.229: process: 0.84275 | load: 0.00287 [I 2022-10-10 10:11:07] ...trap/engines/engine.py.230: loss: 0.18935 [I 2022-10-10 10:11:07] ...optimizers/trijoint.py.151: total_norm: 29.752090 [I 2022-10-10 10:11:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 710/2304 [I 2022-10-10 10:11:15] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:32 [I 2022-10-10 10:11:15] ...trap/engines/engine.py.229: process: 0.83604 | load: 0.00288 [I 2022-10-10 10:11:15] ...trap/engines/engine.py.230: loss: 0.12176 [I 2022-10-10 10:11:15] ...optimizers/trijoint.py.151: total_norm: 8.163785 [I 2022-10-10 10:11:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 720/2304 [I 2022-10-10 10:11:24] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:15 [I 2022-10-10 10:11:24] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00289 [I 2022-10-10 10:11:24] ...trap/engines/engine.py.230: loss: 0.15985 [I 2022-10-10 10:11:24] ...optimizers/trijoint.py.151: total_norm: 25.949261 [I 2022-10-10 10:11:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 730/2304 [I 2022-10-10 10:11:32] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:16 [I 2022-10-10 10:11:32] ...trap/engines/engine.py.229: process: 0.84798 | load: 0.00289 [I 2022-10-10 10:11:32] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-10 10:11:32] ...optimizers/trijoint.py.151: total_norm: 25.495699 [I 2022-10-10 10:11:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 740/2304 [I 2022-10-10 10:11:41] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:05 [I 2022-10-10 10:11:41] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00296 [I 2022-10-10 10:11:41] ...trap/engines/engine.py.230: loss: 0.15890 [I 2022-10-10 10:11:41] ...optimizers/trijoint.py.151: total_norm: 16.429340 [I 2022-10-10 10:11:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 750/2304 [I 2022-10-10 10:11:49] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:22:02 [I 2022-10-10 10:11:49] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00290 [I 2022-10-10 10:11:49] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-10 10:11:49] ...optimizers/trijoint.py.151: total_norm: 17.937357 [I 2022-10-10 10:11:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 760/2304 [I 2022-10-10 10:11:58] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:43 [I 2022-10-10 10:11:58] ...trap/engines/engine.py.229: process: 0.83438 | load: 0.00289 [I 2022-10-10 10:11:58] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-10 10:11:58] ...optimizers/trijoint.py.151: total_norm: 25.998497 [I 2022-10-10 10:12:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 770/2304 [I 2022-10-10 10:12:06] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:21:39 [I 2022-10-10 10:12:06] ...trap/engines/engine.py.229: process: 0.85654 | load: 0.00292 [I 2022-10-10 10:12:06] ...trap/engines/engine.py.230: loss: 0.10620 [I 2022-10-10 10:12:06] ...optimizers/trijoint.py.151: total_norm: 14.638814 [I 2022-10-10 10:12:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 780/2304 [I 2022-10-10 10:12:15] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:29 [I 2022-10-10 10:12:15] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00295 [I 2022-10-10 10:12:15] ...trap/engines/engine.py.230: loss: 0.15022 [I 2022-10-10 10:12:15] ...optimizers/trijoint.py.151: total_norm: 19.886898 [I 2022-10-10 10:12:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 790/2304 [I 2022-10-10 10:12:23] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:24 [I 2022-10-10 10:12:23] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00290 [I 2022-10-10 10:12:23] ...trap/engines/engine.py.230: loss: 0.18560 [I 2022-10-10 10:12:23] ...optimizers/trijoint.py.151: total_norm: 19.282991 [I 2022-10-10 10:12:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 800/2304 [I 2022-10-10 10:12:32] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:11 [I 2022-10-10 10:12:32] ...trap/engines/engine.py.229: process: 0.83099 | load: 0.00290 [I 2022-10-10 10:12:32] ...trap/engines/engine.py.230: loss: 0.16550 [I 2022-10-10 10:12:32] ...optimizers/trijoint.py.151: total_norm: 13.922840 [I 2022-10-10 10:12:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 810/2304 [I 2022-10-10 10:12:40] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:07 [I 2022-10-10 10:12:40] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00286 [I 2022-10-10 10:12:40] ...trap/engines/engine.py.230: loss: 0.16308 [I 2022-10-10 10:12:40] ...optimizers/trijoint.py.151: total_norm: 17.837814 [I 2022-10-10 10:12:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 820/2304 [I 2022-10-10 10:12:49] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:20:53 [I 2022-10-10 10:12:49] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00300 [I 2022-10-10 10:12:49] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-10 10:12:49] ...optimizers/trijoint.py.151: total_norm: 37.464855 [I 2022-10-10 10:12:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 830/2304 [I 2022-10-10 10:12:57] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:47 [I 2022-10-10 10:12:57] ...trap/engines/engine.py.229: process: 0.83241 | load: 0.00291 [I 2022-10-10 10:12:57] ...trap/engines/engine.py.230: loss: 0.13523 [I 2022-10-10 10:12:57] ...optimizers/trijoint.py.151: total_norm: 16.892080 [I 2022-10-10 10:13:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 840/2304 [I 2022-10-10 10:13:06] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:42 [I 2022-10-10 10:13:06] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00291 [I 2022-10-10 10:13:06] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-10 10:13:06] ...optimizers/trijoint.py.151: total_norm: 11.779737 [I 2022-10-10 10:13:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 850/2304 [I 2022-10-10 10:13:14] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:27 [I 2022-10-10 10:13:14] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00293 [I 2022-10-10 10:13:14] ...trap/engines/engine.py.230: loss: 0.14525 [I 2022-10-10 10:13:14] ...optimizers/trijoint.py.151: total_norm: 11.747777 [I 2022-10-10 10:13:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 860/2304 [I 2022-10-10 10:13:23] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:35 [I 2022-10-10 10:13:23] ...trap/engines/engine.py.229: process: 0.86920 | load: 0.00297 [I 2022-10-10 10:13:23] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-10 10:13:23] ...optimizers/trijoint.py.151: total_norm: 21.059862 [I 2022-10-10 10:13:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 870/2304 [I 2022-10-10 10:13:31] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:11 [I 2022-10-10 10:13:31] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00299 [I 2022-10-10 10:13:31] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-10 10:13:31] ...optimizers/trijoint.py.151: total_norm: 11.681823 [I 2022-10-10 10:13:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 880/2304 [I 2022-10-10 10:13:40] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:20:06 [I 2022-10-10 10:13:40] ...trap/engines/engine.py.229: process: 0.84289 | load: 0.00292 [I 2022-10-10 10:13:40] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-10 10:13:40] ...optimizers/trijoint.py.151: total_norm: 19.297031 [I 2022-10-10 10:13:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 890/2304 [I 2022-10-10 10:13:48] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:19:51 [I 2022-10-10 10:13:48] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00289 [I 2022-10-10 10:13:48] ...trap/engines/engine.py.230: loss: 0.16205 [I 2022-10-10 10:13:48] ...optimizers/trijoint.py.151: total_norm: 11.667297 [I 2022-10-10 10:13:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 900/2304 [I 2022-10-10 10:13:57] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:47 [I 2022-10-10 10:13:57] ...trap/engines/engine.py.229: process: 0.84288 | load: 0.00287 [I 2022-10-10 10:13:57] ...trap/engines/engine.py.230: loss: 0.16707 [I 2022-10-10 10:13:57] ...optimizers/trijoint.py.151: total_norm: 15.843079 [I 2022-10-10 10:14:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 910/2304 [I 2022-10-10 10:14:05] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:19:37 [I 2022-10-10 10:14:05] ...trap/engines/engine.py.229: process: 0.84048 | load: 0.00289 [I 2022-10-10 10:14:05] ...trap/engines/engine.py.230: loss: 0.20775 [I 2022-10-10 10:14:05] ...optimizers/trijoint.py.151: total_norm: 10.221281 [I 2022-10-10 10:14:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 920/2304 [I 2022-10-10 10:14:14] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:33 [I 2022-10-10 10:14:14] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00292 [I 2022-10-10 10:14:14] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-10 10:14:14] ...optimizers/trijoint.py.151: total_norm: 17.210970 [I 2022-10-10 10:14:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 930/2304 [I 2022-10-10 10:14:22] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:19:25 [I 2022-10-10 10:14:22] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00294 [I 2022-10-10 10:14:22] ...trap/engines/engine.py.230: loss: 0.11700 [I 2022-10-10 10:14:22] ...optimizers/trijoint.py.151: total_norm: 21.137600 [I 2022-10-10 10:14:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 940/2304 [I 2022-10-10 10:14:31] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:12 [I 2022-10-10 10:14:31] ...trap/engines/engine.py.229: process: 0.84731 | load: 0.00290 [I 2022-10-10 10:14:31] ...trap/engines/engine.py.230: loss: 0.12054 [I 2022-10-10 10:14:31] ...optimizers/trijoint.py.151: total_norm: 22.175615 [I 2022-10-10 10:14:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 950/2304 [I 2022-10-10 10:14:39] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:19:06 [I 2022-10-10 10:14:39] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00292 [I 2022-10-10 10:14:39] ...trap/engines/engine.py.230: loss: 0.10886 [I 2022-10-10 10:14:39] ...optimizers/trijoint.py.151: total_norm: 10.262364 [I 2022-10-10 10:14:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 960/2304 [I 2022-10-10 10:14:48] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:19:00 [I 2022-10-10 10:14:48] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00294 [I 2022-10-10 10:14:48] ...trap/engines/engine.py.230: loss: 0.21077 [I 2022-10-10 10:14:48] ...optimizers/trijoint.py.151: total_norm: 25.758568 [I 2022-10-10 10:14:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 970/2304 [I 2022-10-10 10:14:56] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:18:51 [I 2022-10-10 10:14:56] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00289 [I 2022-10-10 10:14:56] ...trap/engines/engine.py.230: loss: 0.20681 [I 2022-10-10 10:14:56] ...optimizers/trijoint.py.151: total_norm: 34.567429 [I 2022-10-10 10:15:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 980/2304 [I 2022-10-10 10:15:05] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:43 [I 2022-10-10 10:15:05] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00295 [I 2022-10-10 10:15:05] ...trap/engines/engine.py.230: loss: 0.16291 [I 2022-10-10 10:15:05] ...optimizers/trijoint.py.151: total_norm: 23.882822 [I 2022-10-10 10:15:13] ...trap/engines/engine.py.224: train: epoch 68 | batch 990/2304 [I 2022-10-10 10:15:13] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:18:33 [I 2022-10-10 10:15:13] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00292 [I 2022-10-10 10:15:13] ...trap/engines/engine.py.230: loss: 0.13365 [I 2022-10-10 10:15:13] ...optimizers/trijoint.py.151: total_norm: 11.243708 [I 2022-10-10 10:15:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 1000/2304 [I 2022-10-10 10:15:22] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:24 [I 2022-10-10 10:15:22] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00294 [I 2022-10-10 10:15:22] ...trap/engines/engine.py.230: loss: 0.15768 [I 2022-10-10 10:15:22] ...optimizers/trijoint.py.151: total_norm: 16.858501 [I 2022-10-10 10:15:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 1010/2304 [I 2022-10-10 10:15:30] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:18:14 [I 2022-10-10 10:15:30] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00289 [I 2022-10-10 10:15:30] ...trap/engines/engine.py.230: loss: 0.09269 [I 2022-10-10 10:15:30] ...optimizers/trijoint.py.151: total_norm: 14.983883 [I 2022-10-10 10:15:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 1020/2304 [I 2022-10-10 10:15:39] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:06 [I 2022-10-10 10:15:39] ...trap/engines/engine.py.229: process: 0.83230 | load: 0.00290 [I 2022-10-10 10:15:39] ...trap/engines/engine.py.230: loss: 0.09320 [I 2022-10-10 10:15:39] ...optimizers/trijoint.py.151: total_norm: 17.324469 [I 2022-10-10 10:15:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 1030/2304 [I 2022-10-10 10:15:47] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:17:58 [I 2022-10-10 10:15:47] ...trap/engines/engine.py.229: process: 0.84244 | load: 0.00298 [I 2022-10-10 10:15:47] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-10 10:15:47] ...optimizers/trijoint.py.151: total_norm: 13.738153 [I 2022-10-10 10:15:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 1040/2304 [I 2022-10-10 10:15:56] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:17:45 [I 2022-10-10 10:15:56] ...trap/engines/engine.py.229: process: 0.82859 | load: 0.00287 [I 2022-10-10 10:15:56] ...trap/engines/engine.py.230: loss: 0.11852 [I 2022-10-10 10:15:56] ...optimizers/trijoint.py.151: total_norm: 7.028232 [I 2022-10-10 10:16:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 1050/2304 [I 2022-10-10 10:16:04] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:17:38 [I 2022-10-10 10:16:04] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00292 [I 2022-10-10 10:16:04] ...trap/engines/engine.py.230: loss: 0.15834 [I 2022-10-10 10:16:04] ...optimizers/trijoint.py.151: total_norm: 12.908594 [I 2022-10-10 10:16:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1060/2304 [I 2022-10-10 10:16:12] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:17:30 [I 2022-10-10 10:16:12] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00294 [I 2022-10-10 10:16:12] ...trap/engines/engine.py.230: loss: 0.28479 [I 2022-10-10 10:16:12] ...optimizers/trijoint.py.151: total_norm: 23.780397 [I 2022-10-10 10:16:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 1070/2304 [I 2022-10-10 10:16:21] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:17:18 [I 2022-10-10 10:16:21] ...trap/engines/engine.py.229: process: 0.83729 | load: 0.00292 [I 2022-10-10 10:16:21] ...trap/engines/engine.py.230: loss: 0.10545 [I 2022-10-10 10:16:21] ...optimizers/trijoint.py.151: total_norm: 11.162314 [I 2022-10-10 10:16:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 1080/2304 [I 2022-10-10 10:16:29] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:12 [I 2022-10-10 10:16:29] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00290 [I 2022-10-10 10:16:29] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-10 10:16:29] ...optimizers/trijoint.py.151: total_norm: 19.773287 [I 2022-10-10 10:16:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 1090/2304 [I 2022-10-10 10:16:38] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:04 [I 2022-10-10 10:16:38] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00287 [I 2022-10-10 10:16:38] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-10 10:16:38] ...optimizers/trijoint.py.151: total_norm: 10.808572 [I 2022-10-10 10:16:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 1100/2304 [I 2022-10-10 10:16:46] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:04 [I 2022-10-10 10:16:46] ...trap/engines/engine.py.229: process: 0.86384 | load: 0.00294 [I 2022-10-10 10:16:46] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-10 10:16:46] ...optimizers/trijoint.py.151: total_norm: 14.558429 [I 2022-10-10 10:16:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 1110/2304 [I 2022-10-10 10:16:55] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:16:54 [I 2022-10-10 10:16:55] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00290 [I 2022-10-10 10:16:55] ...trap/engines/engine.py.230: loss: 0.12486 [I 2022-10-10 10:16:55] ...optimizers/trijoint.py.151: total_norm: 10.748873 [I 2022-10-10 10:17:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 1120/2304 [I 2022-10-10 10:17:03] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:16:46 [I 2022-10-10 10:17:03] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00291 [I 2022-10-10 10:17:03] ...trap/engines/engine.py.230: loss: 0.10080 [I 2022-10-10 10:17:03] ...optimizers/trijoint.py.151: total_norm: 23.240349 [I 2022-10-10 10:17:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1130/2304 [I 2022-10-10 10:17:12] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:38 [I 2022-10-10 10:17:12] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00289 [I 2022-10-10 10:17:12] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-10 10:17:12] ...optimizers/trijoint.py.151: total_norm: 10.322812 [I 2022-10-10 10:17:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 1140/2304 [I 2022-10-10 10:17:38] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:28:09 [I 2022-10-10 10:17:38] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00291 [I 2022-10-10 10:17:38] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-10 10:17:38] ...optimizers/trijoint.py.151: total_norm: 30.080343 [I 2022-10-10 10:17:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 1150/2304 [I 2022-10-10 10:17:47] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:17:28 [I 2022-10-10 10:17:47] ...trap/engines/engine.py.229: process: 0.83262 | load: 0.00289 [I 2022-10-10 10:17:47] ...trap/engines/engine.py.230: loss: 0.11852 [I 2022-10-10 10:17:47] ...optimizers/trijoint.py.151: total_norm: 11.554087 [I 2022-10-10 10:17:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 1160/2304 [I 2022-10-10 10:17:55] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:16:15 [I 2022-10-10 10:17:55] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00294 [I 2022-10-10 10:17:55] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-10 10:17:55] ...optimizers/trijoint.py.151: total_norm: 23.020754 [I 2022-10-10 10:18:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 1170/2304 [I 2022-10-10 10:18:04] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:15:52 [I 2022-10-10 10:18:04] ...trap/engines/engine.py.229: process: 0.83707 | load: 0.00289 [I 2022-10-10 10:18:04] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-10 10:18:04] ...optimizers/trijoint.py.151: total_norm: 13.177662 [I 2022-10-10 10:18:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1180/2304 [I 2022-10-10 10:18:12] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:54 [I 2022-10-10 10:18:12] ...trap/engines/engine.py.229: process: 0.85706 | load: 0.00288 [I 2022-10-10 10:18:12] ...trap/engines/engine.py.230: loss: 0.19097 [I 2022-10-10 10:18:12] ...optimizers/trijoint.py.151: total_norm: 16.434984 [I 2022-10-10 10:18:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 1190/2304 [I 2022-10-10 10:18:21] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:40 [I 2022-10-10 10:18:21] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00296 [I 2022-10-10 10:18:21] ...trap/engines/engine.py.230: loss: 0.13476 [I 2022-10-10 10:18:21] ...optimizers/trijoint.py.151: total_norm: 40.892136 [I 2022-10-10 10:18:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 1200/2304 [I 2022-10-10 10:18:29] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:33 [I 2022-10-10 10:18:29] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00289 [I 2022-10-10 10:18:29] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-10 10:18:29] ...optimizers/trijoint.py.151: total_norm: 22.299015 [I 2022-10-10 10:18:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 1210/2304 [I 2022-10-10 10:18:38] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:28 [I 2022-10-10 10:18:38] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00288 [I 2022-10-10 10:18:38] ...trap/engines/engine.py.230: loss: 0.15511 [I 2022-10-10 10:18:38] ...optimizers/trijoint.py.151: total_norm: 10.247538 [I 2022-10-10 10:18:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 1220/2304 [I 2022-10-10 10:18:46] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:19 [I 2022-10-10 10:18:46] ...trap/engines/engine.py.229: process: 0.85100 | load: 0.00286 [I 2022-10-10 10:18:46] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-10 10:18:46] ...optimizers/trijoint.py.151: total_norm: 23.618563 [I 2022-10-10 10:18:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 1230/2304 [I 2022-10-10 10:18:55] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:05 [I 2022-10-10 10:18:55] ...trap/engines/engine.py.229: process: 0.84403 | load: 0.00289 [I 2022-10-10 10:18:55] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-10 10:18:55] ...optimizers/trijoint.py.151: total_norm: 20.425602 [I 2022-10-10 10:19:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 1240/2304 [I 2022-10-10 10:19:03] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:02 [I 2022-10-10 10:19:03] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00291 [I 2022-10-10 10:19:03] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-10 10:19:03] ...optimizers/trijoint.py.151: total_norm: 9.938508 [I 2022-10-10 10:19:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1250/2304 [I 2022-10-10 10:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:52 [I 2022-10-10 10:19:12] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00288 [I 2022-10-10 10:19:12] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-10 10:19:12] ...optimizers/trijoint.py.151: total_norm: 11.153474 [I 2022-10-10 10:19:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 1260/2304 [I 2022-10-10 10:19:20] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:39 [I 2022-10-10 10:19:20] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00287 [I 2022-10-10 10:19:20] ...trap/engines/engine.py.230: loss: 0.17989 [I 2022-10-10 10:19:20] ...optimizers/trijoint.py.151: total_norm: 64.216385 [I 2022-10-10 10:19:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 1270/2304 [I 2022-10-10 10:19:28] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:37 [I 2022-10-10 10:19:28] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00290 [I 2022-10-10 10:19:28] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-10 10:19:28] ...optimizers/trijoint.py.151: total_norm: 12.532016 [I 2022-10-10 10:19:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 1280/2304 [I 2022-10-10 10:19:37] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:25 [I 2022-10-10 10:19:37] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00288 [I 2022-10-10 10:19:37] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-10 10:19:37] ...optimizers/trijoint.py.151: total_norm: 26.166883 [I 2022-10-10 10:19:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 1290/2304 [I 2022-10-10 10:19:45] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:21 [I 2022-10-10 10:19:45] ...trap/engines/engine.py.229: process: 0.85416 | load: 0.00295 [I 2022-10-10 10:19:45] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-10 10:19:45] ...optimizers/trijoint.py.151: total_norm: 15.660889 [I 2022-10-10 10:19:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 1300/2304 [I 2022-10-10 10:19:54] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:14 [I 2022-10-10 10:19:54] ...trap/engines/engine.py.229: process: 0.83671 | load: 0.00289 [I 2022-10-10 10:19:54] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-10 10:19:54] ...optimizers/trijoint.py.151: total_norm: 14.031827 [I 2022-10-10 10:20:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1310/2304 [I 2022-10-10 10:20:02] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:13:58 [I 2022-10-10 10:20:02] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00295 [I 2022-10-10 10:20:02] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-10 10:20:02] ...optimizers/trijoint.py.151: total_norm: 20.882570 [I 2022-10-10 10:20:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 1320/2304 [I 2022-10-10 10:20:11] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:55 [I 2022-10-10 10:20:11] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00290 [I 2022-10-10 10:20:11] ...trap/engines/engine.py.230: loss: 0.10402 [I 2022-10-10 10:20:11] ...optimizers/trijoint.py.151: total_norm: 10.843062 [I 2022-10-10 10:20:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1330/2304 [I 2022-10-10 10:20:19] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:47 [I 2022-10-10 10:20:19] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00294 [I 2022-10-10 10:20:19] ...trap/engines/engine.py.230: loss: 0.20369 [I 2022-10-10 10:20:19] ...optimizers/trijoint.py.151: total_norm: 29.888083 [I 2022-10-10 10:20:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 1340/2304 [I 2022-10-10 10:20:28] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:38 [I 2022-10-10 10:20:28] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00296 [I 2022-10-10 10:20:28] ...trap/engines/engine.py.230: loss: 0.10676 [I 2022-10-10 10:20:28] ...optimizers/trijoint.py.151: total_norm: 11.015412 [I 2022-10-10 10:20:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 1350/2304 [I 2022-10-10 10:20:36] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:29 [I 2022-10-10 10:20:36] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00293 [I 2022-10-10 10:20:36] ...trap/engines/engine.py.230: loss: 0.18978 [I 2022-10-10 10:20:36] ...optimizers/trijoint.py.151: total_norm: 17.253313 [I 2022-10-10 10:20:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 1360/2304 [I 2022-10-10 10:20:45] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:21 [I 2022-10-10 10:20:45] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00292 [I 2022-10-10 10:20:45] ...trap/engines/engine.py.230: loss: 0.21919 [I 2022-10-10 10:20:45] ...optimizers/trijoint.py.151: total_norm: 19.634165 [I 2022-10-10 10:20:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 1370/2304 [I 2022-10-10 10:20:53] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:13:11 [I 2022-10-10 10:20:53] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00297 [I 2022-10-10 10:20:53] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-10 10:20:53] ...optimizers/trijoint.py.151: total_norm: 9.058576 [I 2022-10-10 10:21:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1380/2304 [I 2022-10-10 10:21:02] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:05 [I 2022-10-10 10:21:02] ...trap/engines/engine.py.229: process: 0.85004 | load: 0.00300 [I 2022-10-10 10:21:02] ...trap/engines/engine.py.230: loss: 0.13850 [I 2022-10-10 10:21:02] ...optimizers/trijoint.py.151: total_norm: 12.366882 [I 2022-10-10 10:21:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 1390/2304 [I 2022-10-10 10:21:10] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:54 [I 2022-10-10 10:21:10] ...trap/engines/engine.py.229: process: 0.84948 | load: 0.00303 [I 2022-10-10 10:21:10] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-10 10:21:10] ...optimizers/trijoint.py.151: total_norm: 12.615442 [I 2022-10-10 10:21:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1400/2304 [I 2022-10-10 10:21:19] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:46 [I 2022-10-10 10:21:19] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00300 [I 2022-10-10 10:21:19] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-10 10:21:19] ...optimizers/trijoint.py.151: total_norm: 23.243362 [I 2022-10-10 10:21:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 1410/2304 [I 2022-10-10 10:21:27] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:38 [I 2022-10-10 10:21:27] ...trap/engines/engine.py.229: process: 0.84275 | load: 0.00294 [I 2022-10-10 10:21:27] ...trap/engines/engine.py.230: loss: 0.11522 [I 2022-10-10 10:21:27] ...optimizers/trijoint.py.151: total_norm: 16.511171 [I 2022-10-10 10:21:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 1420/2304 [I 2022-10-10 10:21:36] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:35 [I 2022-10-10 10:21:36] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00295 [I 2022-10-10 10:21:36] ...trap/engines/engine.py.230: loss: 0.22147 [I 2022-10-10 10:21:36] ...optimizers/trijoint.py.151: total_norm: 18.183479 [I 2022-10-10 10:21:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 1430/2304 [I 2022-10-10 10:21:44] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:19 [I 2022-10-10 10:21:44] ...trap/engines/engine.py.229: process: 0.84811 | load: 0.00299 [I 2022-10-10 10:21:44] ...trap/engines/engine.py.230: loss: 0.10205 [I 2022-10-10 10:21:44] ...optimizers/trijoint.py.151: total_norm: 8.870193 [I 2022-10-10 10:21:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 1440/2304 [I 2022-10-10 10:21:53] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:11 [I 2022-10-10 10:21:53] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00294 [I 2022-10-10 10:21:53] ...trap/engines/engine.py.230: loss: 0.09641 [I 2022-10-10 10:21:53] ...optimizers/trijoint.py.151: total_norm: 9.947123 [I 2022-10-10 10:22:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1450/2304 [I 2022-10-10 10:22:02] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:12:06 [I 2022-10-10 10:22:02] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00288 [I 2022-10-10 10:22:02] ...trap/engines/engine.py.230: loss: 0.12513 [I 2022-10-10 10:22:02] ...optimizers/trijoint.py.151: total_norm: 11.704183 [I 2022-10-10 10:22:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 1460/2304 [I 2022-10-10 10:22:10] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:58 [I 2022-10-10 10:22:10] ...trap/engines/engine.py.229: process: 0.85485 | load: 0.00288 [I 2022-10-10 10:22:10] ...trap/engines/engine.py.230: loss: 0.09128 [I 2022-10-10 10:22:10] ...optimizers/trijoint.py.151: total_norm: 9.938089 [I 2022-10-10 10:22:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1470/2304 [I 2022-10-10 10:22:19] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:47 [I 2022-10-10 10:22:19] ...trap/engines/engine.py.229: process: 0.84296 | load: 0.00288 [I 2022-10-10 10:22:19] ...trap/engines/engine.py.230: loss: 0.11124 [I 2022-10-10 10:22:19] ...optimizers/trijoint.py.151: total_norm: 7.277035 [I 2022-10-10 10:22:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 1480/2304 [I 2022-10-10 10:22:27] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:42 [I 2022-10-10 10:22:27] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00296 [I 2022-10-10 10:22:27] ...trap/engines/engine.py.230: loss: 0.20511 [I 2022-10-10 10:22:27] ...optimizers/trijoint.py.151: total_norm: 20.417114 [I 2022-10-10 10:22:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 1490/2304 [I 2022-10-10 10:22:36] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:35 [I 2022-10-10 10:22:36] ...trap/engines/engine.py.229: process: 0.87130 | load: 0.00292 [I 2022-10-10 10:22:36] ...trap/engines/engine.py.230: loss: 0.22836 [I 2022-10-10 10:22:36] ...optimizers/trijoint.py.151: total_norm: 19.725609 [I 2022-10-10 10:22:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 1500/2304 [I 2022-10-10 10:22:44] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:11:25 [I 2022-10-10 10:22:44] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00292 [I 2022-10-10 10:22:44] ...trap/engines/engine.py.230: loss: 0.11315 [I 2022-10-10 10:22:44] ...optimizers/trijoint.py.151: total_norm: 13.241380 [I 2022-10-10 10:22:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 1510/2304 [I 2022-10-10 10:22:53] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:11:13 [I 2022-10-10 10:22:53] ...trap/engines/engine.py.229: process: 0.84601 | load: 0.00288 [I 2022-10-10 10:22:53] ...trap/engines/engine.py.230: loss: 0.13885 [I 2022-10-10 10:22:53] ...optimizers/trijoint.py.151: total_norm: 36.052799 [I 2022-10-10 10:23:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 1520/2304 [I 2022-10-10 10:23:01] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:11:05 [I 2022-10-10 10:23:01] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00291 [I 2022-10-10 10:23:01] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-10 10:23:01] ...optimizers/trijoint.py.151: total_norm: 31.319405 [I 2022-10-10 10:23:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 1530/2304 [I 2022-10-10 10:23:10] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:10:56 [I 2022-10-10 10:23:10] ...trap/engines/engine.py.229: process: 0.85671 | load: 0.00295 [I 2022-10-10 10:23:10] ...trap/engines/engine.py.230: loss: 0.17619 [I 2022-10-10 10:23:10] ...optimizers/trijoint.py.151: total_norm: 20.918392 [I 2022-10-10 10:23:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 1540/2304 [I 2022-10-10 10:23:18] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:10:49 [I 2022-10-10 10:23:18] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00288 [I 2022-10-10 10:23:18] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-10 10:23:18] ...optimizers/trijoint.py.151: total_norm: 17.149658 [I 2022-10-10 10:23:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 1550/2304 [I 2022-10-10 10:23:27] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:42 [I 2022-10-10 10:23:27] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00286 [I 2022-10-10 10:23:27] ...trap/engines/engine.py.230: loss: 0.14781 [I 2022-10-10 10:23:27] ...optimizers/trijoint.py.151: total_norm: 19.217768 [I 2022-10-10 10:23:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 1560/2304 [I 2022-10-10 10:23:35] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:10:30 [I 2022-10-10 10:23:35] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00287 [I 2022-10-10 10:23:35] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-10 10:23:35] ...optimizers/trijoint.py.151: total_norm: 14.313567 [I 2022-10-10 10:23:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 1570/2304 [I 2022-10-10 10:23:44] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:19 [I 2022-10-10 10:23:44] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00292 [I 2022-10-10 10:23:44] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-10 10:23:44] ...optimizers/trijoint.py.151: total_norm: 16.680645 [I 2022-10-10 10:23:52] ...trap/engines/engine.py.224: train: epoch 68 | batch 1580/2304 [I 2022-10-10 10:23:52] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:10:12 [I 2022-10-10 10:23:52] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00288 [I 2022-10-10 10:23:52] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-10 10:23:52] ...optimizers/trijoint.py.151: total_norm: 24.233358 [I 2022-10-10 10:24:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 1590/2304 [I 2022-10-10 10:24:01] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:10:03 [I 2022-10-10 10:24:01] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00288 [I 2022-10-10 10:24:01] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-10 10:24:01] ...optimizers/trijoint.py.151: total_norm: 47.554901 [I 2022-10-10 10:24:09] ...trap/engines/engine.py.224: train: epoch 68 | batch 1600/2304 [I 2022-10-10 10:24:09] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:09:54 [I 2022-10-10 10:24:09] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00303 [I 2022-10-10 10:24:09] ...trap/engines/engine.py.230: loss: 0.09316 [I 2022-10-10 10:24:09] ...optimizers/trijoint.py.151: total_norm: 10.684344 [I 2022-10-10 10:24:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 1610/2304 [I 2022-10-10 10:24:18] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:09:47 [I 2022-10-10 10:24:18] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00288 [I 2022-10-10 10:24:18] ...trap/engines/engine.py.230: loss: 0.13583 [I 2022-10-10 10:24:18] ...optimizers/trijoint.py.151: total_norm: 53.452797 [I 2022-10-10 10:24:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 1620/2304 [I 2022-10-10 10:24:26] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:40 [I 2022-10-10 10:24:26] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00289 [I 2022-10-10 10:24:26] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-10 10:24:26] ...optimizers/trijoint.py.151: total_norm: 12.869069 [I 2022-10-10 10:24:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 1630/2304 [I 2022-10-10 10:24:35] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:31 [I 2022-10-10 10:24:35] ...trap/engines/engine.py.229: process: 0.85422 | load: 0.00291 [I 2022-10-10 10:24:35] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-10 10:24:35] ...optimizers/trijoint.py.151: total_norm: 19.111979 [I 2022-10-10 10:24:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 1640/2304 [I 2022-10-10 10:24:43] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:09:20 [I 2022-10-10 10:24:43] ...trap/engines/engine.py.229: process: 0.84549 | load: 0.00290 [I 2022-10-10 10:24:43] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-10 10:24:43] ...optimizers/trijoint.py.151: total_norm: 17.757891 [I 2022-10-10 10:24:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 1650/2304 [I 2022-10-10 10:24:51] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:12 [I 2022-10-10 10:24:51] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00287 [I 2022-10-10 10:24:51] ...trap/engines/engine.py.230: loss: 0.13206 [I 2022-10-10 10:24:51] ...optimizers/trijoint.py.151: total_norm: 16.903828 [I 2022-10-10 10:25:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 1660/2304 [I 2022-10-10 10:25:00] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:05 [I 2022-10-10 10:25:00] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00290 [I 2022-10-10 10:25:00] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-10 10:25:00] ...optimizers/trijoint.py.151: total_norm: 13.139982 [I 2022-10-10 10:25:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 1670/2304 [I 2022-10-10 10:25:08] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:08:57 [I 2022-10-10 10:25:08] ...trap/engines/engine.py.229: process: 0.84106 | load: 0.00295 [I 2022-10-10 10:25:08] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-10 10:25:08] ...optimizers/trijoint.py.151: total_norm: 55.881618 [I 2022-10-10 10:25:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 1680/2304 [I 2022-10-10 10:25:17] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:48 [I 2022-10-10 10:25:17] ...trap/engines/engine.py.229: process: 0.84322 | load: 0.00289 [I 2022-10-10 10:25:17] ...trap/engines/engine.py.230: loss: 0.16037 [I 2022-10-10 10:25:17] ...optimizers/trijoint.py.151: total_norm: 21.003639 [I 2022-10-10 10:25:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 1690/2304 [I 2022-10-10 10:25:25] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:39 [I 2022-10-10 10:25:25] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00300 [I 2022-10-10 10:25:25] ...trap/engines/engine.py.230: loss: 0.11542 [I 2022-10-10 10:25:25] ...optimizers/trijoint.py.151: total_norm: 16.374786 [I 2022-10-10 10:25:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 1700/2304 [I 2022-10-10 10:25:34] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:32 [I 2022-10-10 10:25:34] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00290 [I 2022-10-10 10:25:34] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-10 10:25:34] ...optimizers/trijoint.py.151: total_norm: 17.189535 [I 2022-10-10 10:25:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 1710/2304 [I 2022-10-10 10:25:42] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:24 [I 2022-10-10 10:25:42] ...trap/engines/engine.py.229: process: 0.83146 | load: 0.00294 [I 2022-10-10 10:25:42] ...trap/engines/engine.py.230: loss: 0.10140 [I 2022-10-10 10:25:42] ...optimizers/trijoint.py.151: total_norm: 10.452741 [I 2022-10-10 10:25:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 1720/2304 [I 2022-10-10 10:25:51] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:16 [I 2022-10-10 10:25:51] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00295 [I 2022-10-10 10:25:51] ...trap/engines/engine.py.230: loss: 0.13238 [I 2022-10-10 10:25:51] ...optimizers/trijoint.py.151: total_norm: 28.894772 [I 2022-10-10 10:25:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 1730/2304 [I 2022-10-10 10:25:59] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:07 [I 2022-10-10 10:25:59] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00298 [I 2022-10-10 10:25:59] ...trap/engines/engine.py.230: loss: 0.10666 [I 2022-10-10 10:25:59] ...optimizers/trijoint.py.151: total_norm: 11.181841 [I 2022-10-10 10:26:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 1740/2304 [I 2022-10-10 10:26:08] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:08:00 [I 2022-10-10 10:26:08] ...trap/engines/engine.py.229: process: 0.85844 | load: 0.00297 [I 2022-10-10 10:26:08] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-10 10:26:08] ...optimizers/trijoint.py.151: total_norm: 19.398479 [I 2022-10-10 10:26:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 1750/2304 [I 2022-10-10 10:26:16] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:52 [I 2022-10-10 10:26:16] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00295 [I 2022-10-10 10:26:16] ...trap/engines/engine.py.230: loss: 0.13796 [I 2022-10-10 10:26:16] ...optimizers/trijoint.py.151: total_norm: 17.630682 [I 2022-10-10 10:26:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 1760/2304 [I 2022-10-10 10:26:25] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:39 [I 2022-10-10 10:26:25] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00288 [I 2022-10-10 10:26:25] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-10 10:26:25] ...optimizers/trijoint.py.151: total_norm: 15.226960 [I 2022-10-10 10:26:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 1770/2304 [I 2022-10-10 10:26:33] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:30 [I 2022-10-10 10:26:33] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00292 [I 2022-10-10 10:26:33] ...trap/engines/engine.py.230: loss: 0.14483 [I 2022-10-10 10:26:33] ...optimizers/trijoint.py.151: total_norm: 14.835834 [I 2022-10-10 10:26:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 1780/2304 [I 2022-10-10 10:26:42] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:23 [I 2022-10-10 10:26:42] ...trap/engines/engine.py.229: process: 0.84467 | load: 0.00289 [I 2022-10-10 10:26:42] ...trap/engines/engine.py.230: loss: 0.13129 [I 2022-10-10 10:26:42] ...optimizers/trijoint.py.151: total_norm: 22.877848 [I 2022-10-10 10:26:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 1790/2304 [I 2022-10-10 10:26:50] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:14 [I 2022-10-10 10:26:50] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00292 [I 2022-10-10 10:26:50] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-10 10:26:50] ...optimizers/trijoint.py.151: total_norm: 21.492693 [I 2022-10-10 10:26:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 1800/2304 [I 2022-10-10 10:26:59] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:04 [I 2022-10-10 10:26:59] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00290 [I 2022-10-10 10:26:59] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-10 10:26:59] ...optimizers/trijoint.py.151: total_norm: 16.234943 [I 2022-10-10 10:27:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 1810/2304 [I 2022-10-10 10:27:07] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:06:59 [I 2022-10-10 10:27:07] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00286 [I 2022-10-10 10:27:07] ...trap/engines/engine.py.230: loss: 0.13404 [I 2022-10-10 10:27:07] ...optimizers/trijoint.py.151: total_norm: 16.542597 [I 2022-10-10 10:27:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 1820/2304 [I 2022-10-10 10:27:16] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:49 [I 2022-10-10 10:27:16] ...trap/engines/engine.py.229: process: 0.84772 | load: 0.00296 [I 2022-10-10 10:27:16] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-10 10:27:16] ...optimizers/trijoint.py.151: total_norm: 18.612322 [I 2022-10-10 10:27:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 1830/2304 [I 2022-10-10 10:27:24] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:38 [I 2022-10-10 10:27:24] ...trap/engines/engine.py.229: process: 0.84118 | load: 0.00292 [I 2022-10-10 10:27:24] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-10 10:27:24] ...optimizers/trijoint.py.151: total_norm: 20.482735 [I 2022-10-10 10:27:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 1840/2304 [I 2022-10-10 10:27:33] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:34 [I 2022-10-10 10:27:33] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00291 [I 2022-10-10 10:27:33] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-10 10:27:33] ...optimizers/trijoint.py.151: total_norm: 16.902414 [I 2022-10-10 10:27:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 1850/2304 [I 2022-10-10 10:27:41] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:25 [I 2022-10-10 10:27:41] ...trap/engines/engine.py.229: process: 0.84611 | load: 0.00292 [I 2022-10-10 10:27:41] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-10 10:27:41] ...optimizers/trijoint.py.151: total_norm: 29.821594 [I 2022-10-10 10:27:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 1860/2304 [I 2022-10-10 10:27:50] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:15 [I 2022-10-10 10:27:50] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00291 [I 2022-10-10 10:27:50] ...trap/engines/engine.py.230: loss: 0.22792 [I 2022-10-10 10:27:50] ...optimizers/trijoint.py.151: total_norm: 16.423155 [I 2022-10-10 10:27:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 1870/2304 [I 2022-10-10 10:27:58] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:09 [I 2022-10-10 10:27:58] ...trap/engines/engine.py.229: process: 0.85368 | load: 0.00293 [I 2022-10-10 10:27:58] ...trap/engines/engine.py.230: loss: 0.11152 [I 2022-10-10 10:27:58] ...optimizers/trijoint.py.151: total_norm: 10.696248 [I 2022-10-10 10:28:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 1880/2304 [I 2022-10-10 10:28:07] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:00 [I 2022-10-10 10:28:07] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00296 [I 2022-10-10 10:28:07] ...trap/engines/engine.py.230: loss: 0.20708 [I 2022-10-10 10:28:07] ...optimizers/trijoint.py.151: total_norm: 46.069996 [I 2022-10-10 10:28:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 1890/2304 [I 2022-10-10 10:28:15] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:50 [I 2022-10-10 10:28:15] ...trap/engines/engine.py.229: process: 0.84853 | load: 0.00289 [I 2022-10-10 10:28:15] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-10 10:28:15] ...optimizers/trijoint.py.151: total_norm: 17.850637 [I 2022-10-10 10:28:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 1900/2304 [I 2022-10-10 10:28:24] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:43 [I 2022-10-10 10:28:24] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00287 [I 2022-10-10 10:28:24] ...trap/engines/engine.py.230: loss: 0.15111 [I 2022-10-10 10:28:24] ...optimizers/trijoint.py.151: total_norm: 16.852425 [I 2022-10-10 10:28:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 1910/2304 [I 2022-10-10 10:28:32] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:33 [I 2022-10-10 10:28:32] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00287 [I 2022-10-10 10:28:32] ...trap/engines/engine.py.230: loss: 0.19172 [I 2022-10-10 10:28:32] ...optimizers/trijoint.py.151: total_norm: 25.214987 [I 2022-10-10 10:28:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 1920/2304 [I 2022-10-10 10:28:41] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:25 [I 2022-10-10 10:28:41] ...trap/engines/engine.py.229: process: 0.84939 | load: 0.00290 [I 2022-10-10 10:28:41] ...trap/engines/engine.py.230: loss: 0.19957 [I 2022-10-10 10:28:41] ...optimizers/trijoint.py.151: total_norm: 19.228020 [I 2022-10-10 10:28:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 1930/2304 [I 2022-10-10 10:28:49] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:17 [I 2022-10-10 10:28:49] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00290 [I 2022-10-10 10:28:49] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-10 10:28:49] ...optimizers/trijoint.py.151: total_norm: 11.361243 [I 2022-10-10 10:28:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 1940/2304 [I 2022-10-10 10:28:58] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:09 [I 2022-10-10 10:28:58] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00295 [I 2022-10-10 10:28:58] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-10 10:28:58] ...optimizers/trijoint.py.151: total_norm: 21.208391 [I 2022-10-10 10:29:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 1950/2304 [I 2022-10-10 10:29:06] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:00 [I 2022-10-10 10:29:06] ...trap/engines/engine.py.229: process: 0.84457 | load: 0.00295 [I 2022-10-10 10:29:06] ...trap/engines/engine.py.230: loss: 0.20672 [I 2022-10-10 10:29:06] ...optimizers/trijoint.py.151: total_norm: 11.680515 [I 2022-10-10 10:29:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 1960/2304 [I 2022-10-10 10:29:15] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:51 [I 2022-10-10 10:29:15] ...trap/engines/engine.py.229: process: 0.83421 | load: 0.00287 [I 2022-10-10 10:29:15] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-10 10:29:15] ...optimizers/trijoint.py.151: total_norm: 10.588697 [I 2022-10-10 10:29:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 1970/2304 [I 2022-10-10 10:29:23] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:43 [I 2022-10-10 10:29:23] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00292 [I 2022-10-10 10:29:23] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-10 10:29:23] ...optimizers/trijoint.py.151: total_norm: 19.425598 [I 2022-10-10 10:29:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 1980/2304 [I 2022-10-10 10:29:32] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:32 [I 2022-10-10 10:29:32] ...trap/engines/engine.py.229: process: 0.82924 | load: 0.00289 [I 2022-10-10 10:29:32] ...trap/engines/engine.py.230: loss: 0.09253 [I 2022-10-10 10:29:32] ...optimizers/trijoint.py.151: total_norm: 13.252258 [I 2022-10-10 10:29:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 1990/2304 [I 2022-10-10 10:29:40] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:27 [I 2022-10-10 10:29:40] ...trap/engines/engine.py.229: process: 0.86562 | load: 0.00292 [I 2022-10-10 10:29:40] ...trap/engines/engine.py.230: loss: 0.10533 [I 2022-10-10 10:29:40] ...optimizers/trijoint.py.151: total_norm: 8.589906 [I 2022-10-10 10:29:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 2000/2304 [I 2022-10-10 10:29:49] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:17 [I 2022-10-10 10:29:49] ...trap/engines/engine.py.229: process: 0.83860 | load: 0.00288 [I 2022-10-10 10:29:49] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-10 10:29:49] ...optimizers/trijoint.py.151: total_norm: 12.960893 [I 2022-10-10 10:29:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 2010/2304 [I 2022-10-10 10:29:57] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:10 [I 2022-10-10 10:29:57] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00295 [I 2022-10-10 10:29:57] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-10 10:29:57] ...optimizers/trijoint.py.151: total_norm: 8.728769 [I 2022-10-10 10:30:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 2020/2304 [I 2022-10-10 10:30:06] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:04:02 [I 2022-10-10 10:30:06] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00290 [I 2022-10-10 10:30:06] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-10 10:30:06] ...optimizers/trijoint.py.151: total_norm: 14.737557 [I 2022-10-10 10:30:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 2030/2304 [I 2022-10-10 10:30:14] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:52 [I 2022-10-10 10:30:14] ...trap/engines/engine.py.229: process: 0.83666 | load: 0.00292 [I 2022-10-10 10:30:14] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-10 10:30:14] ...optimizers/trijoint.py.151: total_norm: 14.925093 [I 2022-10-10 10:30:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 2040/2304 [I 2022-10-10 10:30:23] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:44 [I 2022-10-10 10:30:23] ...trap/engines/engine.py.229: process: 0.84851 | load: 0.00302 [I 2022-10-10 10:30:23] ...trap/engines/engine.py.230: loss: 0.19121 [I 2022-10-10 10:30:23] ...optimizers/trijoint.py.151: total_norm: 26.702539 [I 2022-10-10 10:30:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 2050/2304 [I 2022-10-10 10:30:31] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:35 [I 2022-10-10 10:30:31] ...trap/engines/engine.py.229: process: 0.83970 | load: 0.00289 [I 2022-10-10 10:30:31] ...trap/engines/engine.py.230: loss: 0.12219 [I 2022-10-10 10:30:31] ...optimizers/trijoint.py.151: total_norm: 11.610028 [I 2022-10-10 10:30:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 2060/2304 [I 2022-10-10 10:30:40] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:27 [I 2022-10-10 10:30:40] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00287 [I 2022-10-10 10:30:40] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-10 10:30:40] ...optimizers/trijoint.py.151: total_norm: 26.351883 [I 2022-10-10 10:30:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 2070/2304 [I 2022-10-10 10:30:48] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:19 [I 2022-10-10 10:30:48] ...trap/engines/engine.py.229: process: 0.85658 | load: 0.00289 [I 2022-10-10 10:30:48] ...trap/engines/engine.py.230: loss: 0.11839 [I 2022-10-10 10:30:48] ...optimizers/trijoint.py.151: total_norm: 8.860538 [I 2022-10-10 10:30:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 2080/2304 [I 2022-10-10 10:30:57] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:10 [I 2022-10-10 10:30:57] ...trap/engines/engine.py.229: process: 0.84475 | load: 0.00289 [I 2022-10-10 10:30:57] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-10 10:30:57] ...optimizers/trijoint.py.151: total_norm: 17.926102 [I 2022-10-10 10:31:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 2090/2304 [I 2022-10-10 10:31:05] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:03:01 [I 2022-10-10 10:31:05] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00288 [I 2022-10-10 10:31:05] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-10 10:31:05] ...optimizers/trijoint.py.151: total_norm: 8.235589 [I 2022-10-10 10:31:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 2100/2304 [I 2022-10-10 10:31:14] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:02:54 [I 2022-10-10 10:31:14] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00288 [I 2022-10-10 10:31:14] ...trap/engines/engine.py.230: loss: 0.14791 [I 2022-10-10 10:31:14] ...optimizers/trijoint.py.151: total_norm: 12.049976 [I 2022-10-10 10:31:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 2110/2304 [I 2022-10-10 10:31:22] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:45 [I 2022-10-10 10:31:22] ...trap/engines/engine.py.229: process: 0.85567 | load: 0.00299 [I 2022-10-10 10:31:22] ...trap/engines/engine.py.230: loss: 0.19706 [I 2022-10-10 10:31:22] ...optimizers/trijoint.py.151: total_norm: 13.833377 [I 2022-10-10 10:31:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 2120/2304 [I 2022-10-10 10:31:31] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:02:36 [I 2022-10-10 10:31:31] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00289 [I 2022-10-10 10:31:31] ...trap/engines/engine.py.230: loss: 0.13639 [I 2022-10-10 10:31:31] ...optimizers/trijoint.py.151: total_norm: 16.339951 [I 2022-10-10 10:31:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 2130/2304 [I 2022-10-10 10:31:39] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:27 [I 2022-10-10 10:31:39] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00288 [I 2022-10-10 10:31:39] ...trap/engines/engine.py.230: loss: 0.09986 [I 2022-10-10 10:31:39] ...optimizers/trijoint.py.151: total_norm: 12.802507 [I 2022-10-10 10:31:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 2140/2304 [I 2022-10-10 10:31:48] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:02:19 [I 2022-10-10 10:31:48] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00288 [I 2022-10-10 10:31:48] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-10 10:31:48] ...optimizers/trijoint.py.151: total_norm: 29.583544 [I 2022-10-10 10:31:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 2150/2304 [I 2022-10-10 10:31:57] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:02:11 [I 2022-10-10 10:31:57] ...trap/engines/engine.py.229: process: 0.85882 | load: 0.00293 [I 2022-10-10 10:31:57] ...trap/engines/engine.py.230: loss: 0.10525 [I 2022-10-10 10:31:57] ...optimizers/trijoint.py.151: total_norm: 16.012381 [I 2022-10-10 10:32:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 2160/2304 [I 2022-10-10 10:32:05] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:02:02 [I 2022-10-10 10:32:05] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00290 [I 2022-10-10 10:32:05] ...trap/engines/engine.py.230: loss: 0.12010 [I 2022-10-10 10:32:05] ...optimizers/trijoint.py.151: total_norm: 22.840712 [I 2022-10-10 10:32:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 2170/2304 [I 2022-10-10 10:32:14] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:01:53 [I 2022-10-10 10:32:14] ...trap/engines/engine.py.229: process: 0.83197 | load: 0.00292 [I 2022-10-10 10:32:14] ...trap/engines/engine.py.230: loss: 0.17611 [I 2022-10-10 10:32:14] ...optimizers/trijoint.py.151: total_norm: 22.403728 [I 2022-10-10 10:32:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 2180/2304 [I 2022-10-10 10:32:22] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:44 [I 2022-10-10 10:32:22] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00289 [I 2022-10-10 10:32:22] ...trap/engines/engine.py.230: loss: 0.13178 [I 2022-10-10 10:32:22] ...optimizers/trijoint.py.151: total_norm: 20.657751 [I 2022-10-10 10:32:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 2190/2304 [I 2022-10-10 10:32:30] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:36 [I 2022-10-10 10:32:30] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00288 [I 2022-10-10 10:32:30] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-10 10:32:30] ...optimizers/trijoint.py.151: total_norm: 12.416260 [I 2022-10-10 10:32:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 2200/2304 [I 2022-10-10 10:32:39] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:28 [I 2022-10-10 10:32:39] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00292 [I 2022-10-10 10:32:39] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-10 10:32:39] ...optimizers/trijoint.py.151: total_norm: 18.611105 [I 2022-10-10 10:32:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 2210/2304 [I 2022-10-10 10:32:47] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:19 [I 2022-10-10 10:32:47] ...trap/engines/engine.py.229: process: 0.85776 | load: 0.00292 [I 2022-10-10 10:32:47] ...trap/engines/engine.py.230: loss: 0.18000 [I 2022-10-10 10:32:47] ...optimizers/trijoint.py.151: total_norm: 11.266319 [I 2022-10-10 10:32:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 2220/2304 [I 2022-10-10 10:32:56] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:11 [I 2022-10-10 10:32:56] ...trap/engines/engine.py.229: process: 0.86817 | load: 0.00288 [I 2022-10-10 10:32:56] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-10 10:32:56] ...optimizers/trijoint.py.151: total_norm: 11.867635 [I 2022-10-10 10:33:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 2230/2304 [I 2022-10-10 10:33:04] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:01:02 [I 2022-10-10 10:33:04] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00290 [I 2022-10-10 10:33:04] ...trap/engines/engine.py.230: loss: 0.13900 [I 2022-10-10 10:33:04] ...optimizers/trijoint.py.151: total_norm: 27.905710 [I 2022-10-10 10:33:13] ...trap/engines/engine.py.224: train: epoch 68 | batch 2240/2304 [I 2022-10-10 10:33:13] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:54 [I 2022-10-10 10:33:13] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00297 [I 2022-10-10 10:33:13] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-10 10:33:13] ...optimizers/trijoint.py.151: total_norm: 12.542539 [I 2022-10-10 10:33:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 2250/2304 [I 2022-10-10 10:33:21] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:45 [I 2022-10-10 10:33:21] ...trap/engines/engine.py.229: process: 0.83813 | load: 0.00292 [I 2022-10-10 10:33:21] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-10 10:33:21] ...optimizers/trijoint.py.151: total_norm: 15.847239 [I 2022-10-10 10:33:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 2260/2304 [I 2022-10-10 10:33:30] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:37 [I 2022-10-10 10:33:30] ...trap/engines/engine.py.229: process: 0.85430 | load: 0.00294 [I 2022-10-10 10:33:30] ...trap/engines/engine.py.230: loss: 0.11435 [I 2022-10-10 10:33:30] ...optimizers/trijoint.py.151: total_norm: 12.611839 [I 2022-10-10 10:33:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 2270/2304 [I 2022-10-10 10:33:38] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:28 [I 2022-10-10 10:33:38] ...trap/engines/engine.py.229: process: 0.85266 | load: 0.00288 [I 2022-10-10 10:33:38] ...trap/engines/engine.py.230: loss: 0.15190 [I 2022-10-10 10:33:38] ...optimizers/trijoint.py.151: total_norm: 17.451550 [I 2022-10-10 10:33:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 2280/2304 [I 2022-10-10 10:33:47] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:20 [I 2022-10-10 10:33:47] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00290 [I 2022-10-10 10:33:47] ...trap/engines/engine.py.230: loss: 0.13652 [I 2022-10-10 10:33:47] ...optimizers/trijoint.py.151: total_norm: 8.856698 [I 2022-10-10 10:33:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 2290/2304 [I 2022-10-10 10:33:55] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:11 [I 2022-10-10 10:33:55] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00292 [I 2022-10-10 10:33:55] ...trap/engines/engine.py.230: loss: 0.16878 [I 2022-10-10 10:33:55] ...optimizers/trijoint.py.151: total_norm: 17.022326 [I 2022-10-10 10:34:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 2300/2304 [I 2022-10-10 10:34:04] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:03 [I 2022-10-10 10:34:04] ...trap/engines/engine.py.229: process: 0.83543 | load: 0.00013 [I 2022-10-10 10:34:04] ...trap/engines/engine.py.230: loss: 0.10093 [I 2022-10-10 10:34:04] ...optimizers/trijoint.py.151: total_norm: 9.375031 [I 2022-10-10 10:34:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 2304/2304 [I 2022-10-10 10:34:07] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:00 [I 2022-10-10 10:34:07] ...trap/engines/engine.py.229: process: 0.84189 | load: 0.00013 [I 2022-10-10 10:34:07] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-10 10:34:07] ...optimizers/trijoint.py.151: total_norm: 17.708136 [S 2022-10-10 10:34:08] ...trap/engines/engine.py.129: train_epoch.epoch: 68 [S 2022-10-10 10:34:08] ...trap/engines/engine.py.129: train_epoch.loss: 0.14030477404271186 [I 2022-10-10 10:34:09] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 10:34:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 68 [I 2022-10-10 10:34:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 10:34:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 10:34:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 10:34:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 10:34:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 10:34:14] ...trap/engines/engine.py.321: eval: epoch 68 | batch 0/510 [I 2022-10-10 10:34:14] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:01 [I 2022-10-10 10:34:14] ...trap/engines/engine.py.326: process: 4.70965 | load: 4.15035 [I 2022-10-10 10:34:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 10/510 [I 2022-10-10 10:34:20] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:35 [I 2022-10-10 10:34:20] ...trap/engines/engine.py.326: process: 0.58944 | load: 0.00017 [I 2022-10-10 10:34:26] ...trap/engines/engine.py.321: eval: epoch 68 | batch 20/510 [I 2022-10-10 10:34:26] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:17 [I 2022-10-10 10:34:26] ...trap/engines/engine.py.326: process: 0.62843 | load: 0.00017 [I 2022-10-10 10:34:32] ...trap/engines/engine.py.321: eval: epoch 68 | batch 30/510 [I 2022-10-10 10:34:32] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:58 [I 2022-10-10 10:34:32] ...trap/engines/engine.py.326: process: 0.50311 | load: 0.00017 [I 2022-10-10 10:34:39] ...trap/engines/engine.py.321: eval: epoch 68 | batch 40/510 [I 2022-10-10 10:34:39] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:05:08 [I 2022-10-10 10:34:39] ...trap/engines/engine.py.326: process: 0.58891 | load: 0.00017 [I 2022-10-10 10:34:45] ...trap/engines/engine.py.321: eval: epoch 68 | batch 50/510 [I 2022-10-10 10:34:45] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:04:15 [I 2022-10-10 10:34:45] ...trap/engines/engine.py.326: process: 0.52181 | load: 0.00017 [I 2022-10-10 10:34:50] ...trap/engines/engine.py.321: eval: epoch 68 | batch 60/510 [I 2022-10-10 10:34:50] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:16 [I 2022-10-10 10:34:50] ...trap/engines/engine.py.326: process: 0.57622 | load: 0.00018 [I 2022-10-10 10:34:56] ...trap/engines/engine.py.321: eval: epoch 68 | batch 70/510 [I 2022-10-10 10:34:56] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:04 [I 2022-10-10 10:34:56] ...trap/engines/engine.py.326: process: 0.60602 | load: 0.00018 [I 2022-10-10 10:35:01] ...trap/engines/engine.py.321: eval: epoch 68 | batch 80/510 [I 2022-10-10 10:35:01] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:53 [I 2022-10-10 10:35:01] ...trap/engines/engine.py.326: process: 0.50879 | load: 0.00018 [I 2022-10-10 10:35:06] ...trap/engines/engine.py.321: eval: epoch 68 | batch 90/510 [I 2022-10-10 10:35:06] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:17 [I 2022-10-10 10:35:06] ...trap/engines/engine.py.326: process: 0.49207 | load: 0.00019 [I 2022-10-10 10:35:10] ...trap/engines/engine.py.321: eval: epoch 68 | batch 100/510 [I 2022-10-10 10:35:10] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:52 [I 2022-10-10 10:35:10] ...trap/engines/engine.py.326: process: 0.31113 | load: 0.00017 [I 2022-10-10 10:35:13] ...trap/engines/engine.py.321: eval: epoch 68 | batch 110/510 [I 2022-10-10 10:35:13] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:11 [I 2022-10-10 10:35:13] ...trap/engines/engine.py.326: process: 0.31643 | load: 0.00018 [I 2022-10-10 10:35:16] ...trap/engines/engine.py.321: eval: epoch 68 | batch 120/510 [I 2022-10-10 10:35:16] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:04 [I 2022-10-10 10:35:16] ...trap/engines/engine.py.326: process: 0.31517 | load: 0.00018 [I 2022-10-10 10:35:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 130/510 [I 2022-10-10 10:35:20] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:00 [I 2022-10-10 10:35:20] ...trap/engines/engine.py.326: process: 0.31377 | load: 0.00018 [I 2022-10-10 10:35:23] ...trap/engines/engine.py.321: eval: epoch 68 | batch 140/510 [I 2022-10-10 10:35:23] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:56 [I 2022-10-10 10:35:23] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00018 [I 2022-10-10 10:35:26] ...trap/engines/engine.py.321: eval: epoch 68 | batch 150/510 [I 2022-10-10 10:35:26] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:53 [I 2022-10-10 10:35:26] ...trap/engines/engine.py.326: process: 0.30848 | load: 0.00018 [I 2022-10-10 10:35:29] ...trap/engines/engine.py.321: eval: epoch 68 | batch 160/510 [I 2022-10-10 10:35:29] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:50 [I 2022-10-10 10:35:29] ...trap/engines/engine.py.326: process: 0.31637 | load: 0.00019 [I 2022-10-10 10:35:32] ...trap/engines/engine.py.321: eval: epoch 68 | batch 170/510 [I 2022-10-10 10:35:32] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:47 [I 2022-10-10 10:35:32] ...trap/engines/engine.py.326: process: 0.31890 | load: 0.00018 [I 2022-10-10 10:35:35] ...trap/engines/engine.py.321: eval: epoch 68 | batch 180/510 [I 2022-10-10 10:35:35] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:44 [I 2022-10-10 10:35:35] ...trap/engines/engine.py.326: process: 0.31792 | load: 0.00018 [I 2022-10-10 10:35:38] ...trap/engines/engine.py.321: eval: epoch 68 | batch 190/510 [I 2022-10-10 10:35:38] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:41 [I 2022-10-10 10:35:38] ...trap/engines/engine.py.326: process: 0.31348 | load: 0.00017 [I 2022-10-10 10:35:42] ...trap/engines/engine.py.321: eval: epoch 68 | batch 200/510 [I 2022-10-10 10:35:42] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:38 [I 2022-10-10 10:35:42] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00018 [I 2022-10-10 10:35:45] ...trap/engines/engine.py.321: eval: epoch 68 | batch 210/510 [I 2022-10-10 10:35:45] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:34 [I 2022-10-10 10:35:45] ...trap/engines/engine.py.326: process: 0.31376 | load: 0.00019 [I 2022-10-10 10:35:48] ...trap/engines/engine.py.321: eval: epoch 68 | batch 220/510 [I 2022-10-10 10:35:48] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:32 [I 2022-10-10 10:35:48] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00018 [I 2022-10-10 10:35:51] ...trap/engines/engine.py.321: eval: epoch 68 | batch 230/510 [I 2022-10-10 10:35:51] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:28 [I 2022-10-10 10:35:51] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00019 [I 2022-10-10 10:35:54] ...trap/engines/engine.py.321: eval: epoch 68 | batch 240/510 [I 2022-10-10 10:35:54] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:25 [I 2022-10-10 10:35:54] ...trap/engines/engine.py.326: process: 0.31943 | load: 0.00019 [I 2022-10-10 10:35:58] ...trap/engines/engine.py.321: eval: epoch 68 | batch 250/510 [I 2022-10-10 10:35:58] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:22 [I 2022-10-10 10:35:58] ...trap/engines/engine.py.326: process: 0.31220 | load: 0.00019 [I 2022-10-10 10:36:01] ...trap/engines/engine.py.321: eval: epoch 68 | batch 260/510 [I 2022-10-10 10:36:01] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:18 [I 2022-10-10 10:36:01] ...trap/engines/engine.py.326: process: 0.31108 | load: 0.00018 [I 2022-10-10 10:36:04] ...trap/engines/engine.py.321: eval: epoch 68 | batch 270/510 [I 2022-10-10 10:36:04] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:16 [I 2022-10-10 10:36:04] ...trap/engines/engine.py.326: process: 0.31427 | load: 0.00019 [I 2022-10-10 10:36:07] ...trap/engines/engine.py.321: eval: epoch 68 | batch 280/510 [I 2022-10-10 10:36:07] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:12 [I 2022-10-10 10:36:07] ...trap/engines/engine.py.326: process: 0.31204 | load: 0.00018 [I 2022-10-10 10:36:10] ...trap/engines/engine.py.321: eval: epoch 68 | batch 290/510 [I 2022-10-10 10:36:10] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:09 [I 2022-10-10 10:36:10] ...trap/engines/engine.py.326: process: 0.31070 | load: 0.00019 [I 2022-10-10 10:36:13] ...trap/engines/engine.py.321: eval: epoch 68 | batch 300/510 [I 2022-10-10 10:36:13] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:06 [I 2022-10-10 10:36:13] ...trap/engines/engine.py.326: process: 0.32424 | load: 0.00019 [I 2022-10-10 10:36:17] ...trap/engines/engine.py.321: eval: epoch 68 | batch 310/510 [I 2022-10-10 10:36:17] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:03 [I 2022-10-10 10:36:17] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00019 [I 2022-10-10 10:36:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 320/510 [I 2022-10-10 10:36:20] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:00 [I 2022-10-10 10:36:20] ...trap/engines/engine.py.326: process: 0.31751 | load: 0.00019 [I 2022-10-10 10:36:23] ...trap/engines/engine.py.321: eval: epoch 68 | batch 330/510 [I 2022-10-10 10:36:23] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:56 [I 2022-10-10 10:36:23] ...trap/engines/engine.py.326: process: 0.31681 | load: 0.00019 [I 2022-10-10 10:36:26] ...trap/engines/engine.py.321: eval: epoch 68 | batch 340/510 [I 2022-10-10 10:36:26] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:53 [I 2022-10-10 10:36:26] ...trap/engines/engine.py.326: process: 0.31640 | load: 0.00019 [I 2022-10-10 10:36:29] ...trap/engines/engine.py.321: eval: epoch 68 | batch 350/510 [I 2022-10-10 10:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:50 [I 2022-10-10 10:36:29] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00019 [I 2022-10-10 10:36:32] ...trap/engines/engine.py.321: eval: epoch 68 | batch 360/510 [I 2022-10-10 10:36:32] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:47 [I 2022-10-10 10:36:32] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00018 [I 2022-10-10 10:36:36] ...trap/engines/engine.py.321: eval: epoch 68 | batch 370/510 [I 2022-10-10 10:36:36] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-10 10:36:36] ...trap/engines/engine.py.326: process: 0.31446 | load: 0.00019 [I 2022-10-10 10:36:39] ...trap/engines/engine.py.321: eval: epoch 68 | batch 380/510 [I 2022-10-10 10:36:39] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:41 [I 2022-10-10 10:36:39] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00019 [I 2022-10-10 10:36:42] ...trap/engines/engine.py.321: eval: epoch 68 | batch 390/510 [I 2022-10-10 10:36:42] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-10 10:36:42] ...trap/engines/engine.py.326: process: 0.31427 | load: 0.00019 [I 2022-10-10 10:36:45] ...trap/engines/engine.py.321: eval: epoch 68 | batch 400/510 [I 2022-10-10 10:36:45] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:34 [I 2022-10-10 10:36:45] ...trap/engines/engine.py.326: process: 0.31186 | load: 0.00018 [I 2022-10-10 10:36:48] ...trap/engines/engine.py.321: eval: epoch 68 | batch 410/510 [I 2022-10-10 10:36:48] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-10 10:36:48] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00019 [I 2022-10-10 10:36:51] ...trap/engines/engine.py.321: eval: epoch 68 | batch 420/510 [I 2022-10-10 10:36:51] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-10 10:36:51] ...trap/engines/engine.py.326: process: 0.31189 | load: 0.00019 [I 2022-10-10 10:36:55] ...trap/engines/engine.py.321: eval: epoch 68 | batch 430/510 [I 2022-10-10 10:36:55] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-10 10:36:55] ...trap/engines/engine.py.326: process: 0.31243 | load: 0.00018 [I 2022-10-10 10:36:58] ...trap/engines/engine.py.321: eval: epoch 68 | batch 440/510 [I 2022-10-10 10:36:58] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:22 [I 2022-10-10 10:36:58] ...trap/engines/engine.py.326: process: 0.32265 | load: 0.00018 [I 2022-10-10 10:37:01] ...trap/engines/engine.py.321: eval: epoch 68 | batch 450/510 [I 2022-10-10 10:37:01] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:19 [I 2022-10-10 10:37:01] ...trap/engines/engine.py.326: process: 0.31708 | load: 0.00018 [I 2022-10-10 10:37:04] ...trap/engines/engine.py.321: eval: epoch 68 | batch 460/510 [I 2022-10-10 10:37:04] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:15 [I 2022-10-10 10:37:04] ...trap/engines/engine.py.326: process: 0.31819 | load: 0.00018 [I 2022-10-10 10:37:07] ...trap/engines/engine.py.321: eval: epoch 68 | batch 470/510 [I 2022-10-10 10:37:07] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-10 10:37:07] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00019 [I 2022-10-10 10:37:10] ...trap/engines/engine.py.321: eval: epoch 68 | batch 480/510 [I 2022-10-10 10:37:10] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-10 10:37:10] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00019 [I 2022-10-10 10:37:14] ...trap/engines/engine.py.321: eval: epoch 68 | batch 490/510 [I 2022-10-10 10:37:14] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:06 [I 2022-10-10 10:37:14] ...trap/engines/engine.py.326: process: 0.30987 | load: 0.00019 [I 2022-10-10 10:37:17] ...trap/engines/engine.py.321: eval: epoch 68 | batch 500/510 [I 2022-10-10 10:37:17] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:03 [I 2022-10-10 10:37:17] ...trap/engines/engine.py.326: process: 0.31059 | load: 0.00019 [I 2022-10-10 10:37:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 510/510 [I 2022-10-10 10:37:20] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:00 [I 2022-10-10 10:37:20] ...trap/engines/engine.py.326: process: 0.31767 | load: 0.00011 [S 2022-10-10 10:37:21] ...trap/engines/engine.py.132: eval_epoch.epoch: 68 [I 2022-10-10 10:37:21] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.726400 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910900 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939900 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.719000 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908400 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936900 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010938 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013450 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.013164 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013755 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012651 [S 2022-10-10 10:37:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011397 [I 2022-10-10 10:38:00] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 10:38:01] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 10:38:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 10:38:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 10:38:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 10:38:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 10:38:05] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 10:38:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 10:38:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 69 [I 2022-10-10 10:38:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 10:38:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 0/2304 [I 2022-10-10 10:38:14] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:59:13 [I 2022-10-10 10:38:14] ...trap/engines/engine.py.229: process: 3.10461 | load: 2.21325 [I 2022-10-10 10:38:14] ...trap/engines/engine.py.230: loss: 0.11583 [I 2022-10-10 10:38:14] ...optimizers/trijoint.py.151: total_norm: 9.845568 [I 2022-10-10 10:38:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 10/2304 [I 2022-10-10 10:38:22] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:41 [I 2022-10-10 10:38:22] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00540 [I 2022-10-10 10:38:22] ...trap/engines/engine.py.230: loss: 0.08997 [I 2022-10-10 10:38:22] ...optimizers/trijoint.py.151: total_norm: 10.489580 [I 2022-10-10 10:38:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 20/2304 [I 2022-10-10 10:38:31] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:20 [I 2022-10-10 10:38:31] ...trap/engines/engine.py.229: process: 0.84148 | load: 0.00362 [I 2022-10-10 10:38:31] ...trap/engines/engine.py.230: loss: 0.12860 [I 2022-10-10 10:38:31] ...optimizers/trijoint.py.151: total_norm: 19.618269 [I 2022-10-10 10:38:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 30/2304 [I 2022-10-10 10:38:39] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:13 [I 2022-10-10 10:38:39] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00358 [I 2022-10-10 10:38:39] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-10 10:38:39] ...optimizers/trijoint.py.151: total_norm: 12.052444 [I 2022-10-10 10:38:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 40/2304 [I 2022-10-10 10:38:48] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:57 [I 2022-10-10 10:38:48] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00394 [I 2022-10-10 10:38:48] ...trap/engines/engine.py.230: loss: 0.18214 [I 2022-10-10 10:38:48] ...optimizers/trijoint.py.151: total_norm: 14.450272 [I 2022-10-10 10:38:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 50/2304 [I 2022-10-10 10:38:56] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:52 [I 2022-10-10 10:38:56] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00422 [I 2022-10-10 10:38:56] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-10 10:38:56] ...optimizers/trijoint.py.151: total_norm: 18.147123 [I 2022-10-10 10:39:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 60/2304 [I 2022-10-10 10:39:05] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:48 [I 2022-10-10 10:39:05] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00309 [I 2022-10-10 10:39:05] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-10 10:39:05] ...optimizers/trijoint.py.151: total_norm: 19.961163 [I 2022-10-10 10:39:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 70/2304 [I 2022-10-10 10:39:13] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:34 [I 2022-10-10 10:39:13] ...trap/engines/engine.py.229: process: 0.84191 | load: 0.00339 [I 2022-10-10 10:39:13] ...trap/engines/engine.py.230: loss: 0.11401 [I 2022-10-10 10:39:13] ...optimizers/trijoint.py.151: total_norm: 30.317694 [I 2022-10-10 10:39:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 80/2304 [I 2022-10-10 10:39:22] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:24 [I 2022-10-10 10:39:22] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00341 [I 2022-10-10 10:39:22] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-10 10:39:22] ...optimizers/trijoint.py.151: total_norm: 27.101908 [I 2022-10-10 10:39:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 90/2304 [I 2022-10-10 10:39:30] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:21 [I 2022-10-10 10:39:30] ...trap/engines/engine.py.229: process: 0.84118 | load: 0.00320 [I 2022-10-10 10:39:30] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-10 10:39:30] ...optimizers/trijoint.py.151: total_norm: 11.513972 [I 2022-10-10 10:39:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 100/2304 [I 2022-10-10 10:39:39] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:04 [I 2022-10-10 10:39:39] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00331 [I 2022-10-10 10:39:39] ...trap/engines/engine.py.230: loss: 0.16506 [I 2022-10-10 10:39:39] ...optimizers/trijoint.py.151: total_norm: 17.306801 [I 2022-10-10 10:39:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 110/2304 [I 2022-10-10 10:39:47] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:12 [I 2022-10-10 10:39:47] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00308 [I 2022-10-10 10:39:47] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-10 10:39:47] ...optimizers/trijoint.py.151: total_norm: 9.317657 [I 2022-10-10 10:39:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 120/2304 [I 2022-10-10 10:39:56] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:52 [I 2022-10-10 10:39:56] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00311 [I 2022-10-10 10:39:56] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-10 10:39:56] ...optimizers/trijoint.py.151: total_norm: 17.066740 [I 2022-10-10 10:40:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 130/2304 [I 2022-10-10 10:40:04] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:40 [I 2022-10-10 10:40:04] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00309 [I 2022-10-10 10:40:04] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-10 10:40:04] ...optimizers/trijoint.py.151: total_norm: 11.869408 [I 2022-10-10 10:40:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 140/2304 [I 2022-10-10 10:40:13] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:40 [I 2022-10-10 10:40:13] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00325 [I 2022-10-10 10:40:13] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-10 10:40:13] ...optimizers/trijoint.py.151: total_norm: 14.986297 [I 2022-10-10 10:40:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 150/2304 [I 2022-10-10 10:40:21] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:26 [I 2022-10-10 10:40:21] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00333 [I 2022-10-10 10:40:21] ...trap/engines/engine.py.230: loss: 0.13330 [I 2022-10-10 10:40:21] ...optimizers/trijoint.py.151: total_norm: 11.913799 [I 2022-10-10 10:40:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 160/2304 [I 2022-10-10 10:40:30] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:14 [I 2022-10-10 10:40:30] ...trap/engines/engine.py.229: process: 0.86793 | load: 0.00300 [I 2022-10-10 10:40:30] ...trap/engines/engine.py.230: loss: 0.18395 [I 2022-10-10 10:40:30] ...optimizers/trijoint.py.151: total_norm: 30.139917 [I 2022-10-10 10:40:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 170/2304 [I 2022-10-10 10:40:38] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:15 [I 2022-10-10 10:40:38] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00313 [I 2022-10-10 10:40:38] ...trap/engines/engine.py.230: loss: 0.10489 [I 2022-10-10 10:40:38] ...optimizers/trijoint.py.151: total_norm: 11.336448 [I 2022-10-10 10:40:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 180/2304 [I 2022-10-10 10:40:47] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:02 [I 2022-10-10 10:40:47] ...trap/engines/engine.py.229: process: 0.85034 | load: 0.00293 [I 2022-10-10 10:40:47] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-10 10:40:47] ...optimizers/trijoint.py.151: total_norm: 9.581012 [I 2022-10-10 10:40:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 190/2304 [I 2022-10-10 10:40:55] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:54 [I 2022-10-10 10:40:55] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00294 [I 2022-10-10 10:40:55] ...trap/engines/engine.py.230: loss: 0.11902 [I 2022-10-10 10:40:55] ...optimizers/trijoint.py.151: total_norm: 15.342014 [I 2022-10-10 10:41:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 200/2304 [I 2022-10-10 10:41:04] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:44 [I 2022-10-10 10:41:04] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00301 [I 2022-10-10 10:41:04] ...trap/engines/engine.py.230: loss: 0.19227 [I 2022-10-10 10:41:04] ...optimizers/trijoint.py.151: total_norm: 23.737396 [I 2022-10-10 10:41:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 210/2304 [I 2022-10-10 10:41:12] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:22 [I 2022-10-10 10:41:12] ...trap/engines/engine.py.229: process: 0.84536 | load: 0.00290 [I 2022-10-10 10:41:12] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-10 10:41:12] ...optimizers/trijoint.py.151: total_norm: 11.321803 [I 2022-10-10 10:41:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 220/2304 [I 2022-10-10 10:41:21] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:29 [I 2022-10-10 10:41:21] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00298 [I 2022-10-10 10:41:21] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-10 10:41:21] ...optimizers/trijoint.py.151: total_norm: 13.694571 [I 2022-10-10 10:41:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 230/2304 [I 2022-10-10 10:41:29] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:11 [I 2022-10-10 10:41:29] ...trap/engines/engine.py.229: process: 0.83564 | load: 0.00294 [I 2022-10-10 10:41:29] ...trap/engines/engine.py.230: loss: 0.26238 [I 2022-10-10 10:41:29] ...optimizers/trijoint.py.151: total_norm: 19.742968 [I 2022-10-10 10:41:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 240/2304 [I 2022-10-10 10:41:38] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:19 [I 2022-10-10 10:41:38] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00292 [I 2022-10-10 10:41:38] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-10 10:41:38] ...optimizers/trijoint.py.151: total_norm: 11.044802 [I 2022-10-10 10:41:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 250/2304 [I 2022-10-10 10:41:46] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:28:47 [I 2022-10-10 10:41:46] ...trap/engines/engine.py.229: process: 0.83174 | load: 0.00299 [I 2022-10-10 10:41:46] ...trap/engines/engine.py.230: loss: 0.18625 [I 2022-10-10 10:41:46] ...optimizers/trijoint.py.151: total_norm: 18.285126 [I 2022-10-10 10:41:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 260/2304 [I 2022-10-10 10:41:55] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:03 [I 2022-10-10 10:41:55] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00290 [I 2022-10-10 10:41:55] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-10 10:41:55] ...optimizers/trijoint.py.151: total_norm: 16.155924 [I 2022-10-10 10:42:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 270/2304 [I 2022-10-10 10:42:03] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:44 [I 2022-10-10 10:42:03] ...trap/engines/engine.py.229: process: 0.83528 | load: 0.00289 [I 2022-10-10 10:42:03] ...trap/engines/engine.py.230: loss: 0.13397 [I 2022-10-10 10:42:03] ...optimizers/trijoint.py.151: total_norm: 10.638830 [I 2022-10-10 10:42:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 280/2304 [I 2022-10-10 10:42:12] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:38 [I 2022-10-10 10:42:12] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00287 [I 2022-10-10 10:42:12] ...trap/engines/engine.py.230: loss: 0.10018 [I 2022-10-10 10:42:12] ...optimizers/trijoint.py.151: total_norm: 7.203151 [I 2022-10-10 10:42:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 290/2304 [I 2022-10-10 10:42:20] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:35 [I 2022-10-10 10:42:20] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00314 [I 2022-10-10 10:42:20] ...trap/engines/engine.py.230: loss: 0.08582 [I 2022-10-10 10:42:20] ...optimizers/trijoint.py.151: total_norm: 36.661419 [I 2022-10-10 10:42:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 300/2304 [I 2022-10-10 10:42:29] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:16 [I 2022-10-10 10:42:29] ...trap/engines/engine.py.229: process: 0.84745 | load: 0.00290 [I 2022-10-10 10:42:29] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-10 10:42:29] ...optimizers/trijoint.py.151: total_norm: 9.745719 [I 2022-10-10 10:42:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 310/2304 [I 2022-10-10 10:42:37] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:12 [I 2022-10-10 10:42:37] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00308 [I 2022-10-10 10:42:37] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-10 10:42:37] ...optimizers/trijoint.py.151: total_norm: 47.610764 [I 2022-10-10 10:42:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 320/2304 [I 2022-10-10 10:42:46] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:27:59 [I 2022-10-10 10:42:46] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00300 [I 2022-10-10 10:42:46] ...trap/engines/engine.py.230: loss: 0.20937 [I 2022-10-10 10:42:46] ...optimizers/trijoint.py.151: total_norm: 8.235366 [I 2022-10-10 10:42:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 330/2304 [I 2022-10-10 10:42:54] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:27:58 [I 2022-10-10 10:42:54] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00292 [I 2022-10-10 10:42:54] ...trap/engines/engine.py.230: loss: 0.10782 [I 2022-10-10 10:42:54] ...optimizers/trijoint.py.151: total_norm: 10.481751 [I 2022-10-10 10:43:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 340/2304 [I 2022-10-10 10:43:03] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:45 [I 2022-10-10 10:43:03] ...trap/engines/engine.py.229: process: 0.84108 | load: 0.00288 [I 2022-10-10 10:43:03] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-10 10:43:03] ...optimizers/trijoint.py.151: total_norm: 25.775873 [I 2022-10-10 10:43:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 350/2304 [I 2022-10-10 10:43:11] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:27:33 [I 2022-10-10 10:43:11] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00290 [I 2022-10-10 10:43:11] ...trap/engines/engine.py.230: loss: 0.23054 [I 2022-10-10 10:43:11] ...optimizers/trijoint.py.151: total_norm: 20.520639 [I 2022-10-10 10:43:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 360/2304 [I 2022-10-10 10:43:20] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:24 [I 2022-10-10 10:43:20] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00290 [I 2022-10-10 10:43:20] ...trap/engines/engine.py.230: loss: 0.11117 [I 2022-10-10 10:43:20] ...optimizers/trijoint.py.151: total_norm: 17.738884 [I 2022-10-10 10:43:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 370/2304 [I 2022-10-10 10:43:28] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:27:13 [I 2022-10-10 10:43:28] ...trap/engines/engine.py.229: process: 0.84398 | load: 0.00294 [I 2022-10-10 10:43:28] ...trap/engines/engine.py.230: loss: 0.12001 [I 2022-10-10 10:43:28] ...optimizers/trijoint.py.151: total_norm: 20.540461 [I 2022-10-10 10:43:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 380/2304 [I 2022-10-10 10:43:37] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:23 [I 2022-10-10 10:43:37] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00292 [I 2022-10-10 10:43:37] ...trap/engines/engine.py.230: loss: 0.20035 [I 2022-10-10 10:43:37] ...optimizers/trijoint.py.151: total_norm: 18.703814 [I 2022-10-10 10:43:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 390/2304 [I 2022-10-10 10:43:45] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:18 [I 2022-10-10 10:43:45] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00298 [I 2022-10-10 10:43:45] ...trap/engines/engine.py.230: loss: 0.22182 [I 2022-10-10 10:43:45] ...optimizers/trijoint.py.151: total_norm: 19.772478 [I 2022-10-10 10:43:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 400/2304 [I 2022-10-10 10:43:54] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:27:03 [I 2022-10-10 10:43:54] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00294 [I 2022-10-10 10:43:54] ...trap/engines/engine.py.230: loss: 0.11531 [I 2022-10-10 10:43:54] ...optimizers/trijoint.py.151: total_norm: 12.367000 [I 2022-10-10 10:44:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 410/2304 [I 2022-10-10 10:44:02] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:27:02 [I 2022-10-10 10:44:02] ...trap/engines/engine.py.229: process: 0.86609 | load: 0.00289 [I 2022-10-10 10:44:02] ...trap/engines/engine.py.230: loss: 0.11501 [I 2022-10-10 10:44:02] ...optimizers/trijoint.py.151: total_norm: 20.673149 [I 2022-10-10 10:44:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 420/2304 [I 2022-10-10 10:44:11] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:39 [I 2022-10-10 10:44:11] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00298 [I 2022-10-10 10:44:11] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-10 10:44:11] ...optimizers/trijoint.py.151: total_norm: 11.629446 [I 2022-10-10 10:44:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 430/2304 [I 2022-10-10 10:44:19] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:29 [I 2022-10-10 10:44:19] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00292 [I 2022-10-10 10:44:19] ...trap/engines/engine.py.230: loss: 0.20202 [I 2022-10-10 10:44:19] ...optimizers/trijoint.py.151: total_norm: 27.594105 [I 2022-10-10 10:44:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 440/2304 [I 2022-10-10 10:44:28] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:26:34 [I 2022-10-10 10:44:28] ...trap/engines/engine.py.229: process: 0.86310 | load: 0.00289 [I 2022-10-10 10:44:28] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-10 10:44:28] ...optimizers/trijoint.py.151: total_norm: 9.002067 [I 2022-10-10 10:44:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 450/2304 [I 2022-10-10 10:44:36] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:20 [I 2022-10-10 10:44:36] ...trap/engines/engine.py.229: process: 0.86756 | load: 0.00301 [I 2022-10-10 10:44:36] ...trap/engines/engine.py.230: loss: 0.17614 [I 2022-10-10 10:44:36] ...optimizers/trijoint.py.151: total_norm: 16.032701 [I 2022-10-10 10:44:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 460/2304 [I 2022-10-10 10:44:45] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:00 [I 2022-10-10 10:44:45] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00294 [I 2022-10-10 10:44:45] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-10 10:44:45] ...optimizers/trijoint.py.151: total_norm: 14.620548 [I 2022-10-10 10:44:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 470/2304 [I 2022-10-10 10:44:53] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:25:48 [I 2022-10-10 10:44:53] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00288 [I 2022-10-10 10:44:53] ...trap/engines/engine.py.230: loss: 0.13803 [I 2022-10-10 10:44:53] ...optimizers/trijoint.py.151: total_norm: 17.962172 [I 2022-10-10 10:45:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 480/2304 [I 2022-10-10 10:45:02] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:25:50 [I 2022-10-10 10:45:02] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00289 [I 2022-10-10 10:45:02] ...trap/engines/engine.py.230: loss: 0.13623 [I 2022-10-10 10:45:02] ...optimizers/trijoint.py.151: total_norm: 9.647967 [I 2022-10-10 10:45:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 490/2304 [I 2022-10-10 10:45:10] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:25:44 [I 2022-10-10 10:45:10] ...trap/engines/engine.py.229: process: 0.85633 | load: 0.00289 [I 2022-10-10 10:45:10] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-10 10:45:10] ...optimizers/trijoint.py.151: total_norm: 14.523314 [I 2022-10-10 10:45:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 500/2304 [I 2022-10-10 10:45:19] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:25:32 [I 2022-10-10 10:45:19] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00287 [I 2022-10-10 10:45:19] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-10 10:45:19] ...optimizers/trijoint.py.151: total_norm: 9.142069 [I 2022-10-10 10:45:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 510/2304 [I 2022-10-10 10:45:27] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:25:21 [I 2022-10-10 10:45:27] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00290 [I 2022-10-10 10:45:27] ...trap/engines/engine.py.230: loss: 0.20501 [I 2022-10-10 10:45:27] ...optimizers/trijoint.py.151: total_norm: 14.098172 [I 2022-10-10 10:45:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 520/2304 [I 2022-10-10 10:45:36] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:12 [I 2022-10-10 10:45:36] ...trap/engines/engine.py.229: process: 0.85802 | load: 0.00289 [I 2022-10-10 10:45:36] ...trap/engines/engine.py.230: loss: 0.08283 [I 2022-10-10 10:45:36] ...optimizers/trijoint.py.151: total_norm: 8.821566 [I 2022-10-10 10:45:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 530/2304 [I 2022-10-10 10:45:44] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:12 [I 2022-10-10 10:45:44] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00297 [I 2022-10-10 10:45:44] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-10 10:45:44] ...optimizers/trijoint.py.151: total_norm: 17.532812 [I 2022-10-10 10:45:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 540/2304 [I 2022-10-10 10:45:53] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:49 [I 2022-10-10 10:45:53] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00301 [I 2022-10-10 10:45:53] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-10 10:45:53] ...optimizers/trijoint.py.151: total_norm: 22.839542 [I 2022-10-10 10:46:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 550/2304 [I 2022-10-10 10:46:01] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:52 [I 2022-10-10 10:46:01] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00290 [I 2022-10-10 10:46:01] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-10 10:46:01] ...optimizers/trijoint.py.151: total_norm: 13.445333 [I 2022-10-10 10:46:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 560/2304 [I 2022-10-10 10:46:10] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:46 [I 2022-10-10 10:46:10] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00292 [I 2022-10-10 10:46:10] ...trap/engines/engine.py.230: loss: 0.30620 [I 2022-10-10 10:46:10] ...optimizers/trijoint.py.151: total_norm: 12.455726 [I 2022-10-10 10:46:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 570/2304 [I 2022-10-10 10:46:19] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:37 [I 2022-10-10 10:46:19] ...trap/engines/engine.py.229: process: 0.86859 | load: 0.00288 [I 2022-10-10 10:46:19] ...trap/engines/engine.py.230: loss: 0.19793 [I 2022-10-10 10:46:19] ...optimizers/trijoint.py.151: total_norm: 12.866926 [I 2022-10-10 10:46:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 580/2304 [I 2022-10-10 10:46:27] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:37 [I 2022-10-10 10:46:27] ...trap/engines/engine.py.229: process: 0.86890 | load: 0.00299 [I 2022-10-10 10:46:27] ...trap/engines/engine.py.230: loss: 0.12107 [I 2022-10-10 10:46:27] ...optimizers/trijoint.py.151: total_norm: 15.538383 [I 2022-10-10 10:46:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 590/2304 [I 2022-10-10 10:46:36] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:25 [I 2022-10-10 10:46:36] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00298 [I 2022-10-10 10:46:36] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-10 10:46:36] ...optimizers/trijoint.py.151: total_norm: 15.193061 [I 2022-10-10 10:46:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 600/2304 [I 2022-10-10 10:46:44] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:17 [I 2022-10-10 10:46:44] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00295 [I 2022-10-10 10:46:44] ...trap/engines/engine.py.230: loss: 0.12018 [I 2022-10-10 10:46:44] ...optimizers/trijoint.py.151: total_norm: 9.049483 [I 2022-10-10 10:46:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 610/2304 [I 2022-10-10 10:46:53] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:07 [I 2022-10-10 10:46:53] ...trap/engines/engine.py.229: process: 0.84844 | load: 0.00291 [I 2022-10-10 10:46:53] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-10 10:46:53] ...optimizers/trijoint.py.151: total_norm: 11.270029 [I 2022-10-10 10:47:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 620/2304 [I 2022-10-10 10:47:01] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:23:49 [I 2022-10-10 10:47:01] ...trap/engines/engine.py.229: process: 0.84295 | load: 0.00296 [I 2022-10-10 10:47:01] ...trap/engines/engine.py.230: loss: 0.12835 [I 2022-10-10 10:47:01] ...optimizers/trijoint.py.151: total_norm: 8.914773 [I 2022-10-10 10:47:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 630/2304 [I 2022-10-10 10:47:10] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:47 [I 2022-10-10 10:47:10] ...trap/engines/engine.py.229: process: 0.85811 | load: 0.00293 [I 2022-10-10 10:47:10] ...trap/engines/engine.py.230: loss: 0.15259 [I 2022-10-10 10:47:10] ...optimizers/trijoint.py.151: total_norm: 44.659985 [I 2022-10-10 10:47:18] ...trap/engines/engine.py.224: train: epoch 69 | batch 640/2304 [I 2022-10-10 10:47:18] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:38 [I 2022-10-10 10:47:18] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00300 [I 2022-10-10 10:47:18] ...trap/engines/engine.py.230: loss: 0.13627 [I 2022-10-10 10:47:18] ...optimizers/trijoint.py.151: total_norm: 12.050379 [I 2022-10-10 10:47:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 650/2304 [I 2022-10-10 10:47:27] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:25 [I 2022-10-10 10:47:27] ...trap/engines/engine.py.229: process: 0.83329 | load: 0.00299 [I 2022-10-10 10:47:27] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-10 10:47:27] ...optimizers/trijoint.py.151: total_norm: 12.817724 [I 2022-10-10 10:47:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 660/2304 [I 2022-10-10 10:47:35] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:28 [I 2022-10-10 10:47:35] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00292 [I 2022-10-10 10:47:35] ...trap/engines/engine.py.230: loss: 0.13940 [I 2022-10-10 10:47:35] ...optimizers/trijoint.py.151: total_norm: 13.612251 [I 2022-10-10 10:47:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 670/2304 [I 2022-10-10 10:47:44] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:12 [I 2022-10-10 10:47:44] ...trap/engines/engine.py.229: process: 0.84976 | load: 0.00292 [I 2022-10-10 10:47:44] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-10 10:47:44] ...optimizers/trijoint.py.151: total_norm: 11.196972 [I 2022-10-10 10:47:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 680/2304 [I 2022-10-10 10:47:53] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:05 [I 2022-10-10 10:47:53] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00290 [I 2022-10-10 10:47:53] ...trap/engines/engine.py.230: loss: 0.11713 [I 2022-10-10 10:47:53] ...optimizers/trijoint.py.151: total_norm: 15.508103 [I 2022-10-10 10:48:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 690/2304 [I 2022-10-10 10:48:01] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:53 [I 2022-10-10 10:48:01] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00294 [I 2022-10-10 10:48:01] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-10 10:48:01] ...optimizers/trijoint.py.151: total_norm: 23.951061 [I 2022-10-10 10:48:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 700/2304 [I 2022-10-10 10:48:10] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:52 [I 2022-10-10 10:48:10] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00290 [I 2022-10-10 10:48:10] ...trap/engines/engine.py.230: loss: 0.20095 [I 2022-10-10 10:48:10] ...optimizers/trijoint.py.151: total_norm: 16.728106 [I 2022-10-10 10:48:18] ...trap/engines/engine.py.224: train: epoch 69 | batch 710/2304 [I 2022-10-10 10:48:18] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:34 [I 2022-10-10 10:48:18] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00292 [I 2022-10-10 10:48:18] ...trap/engines/engine.py.230: loss: 0.19001 [I 2022-10-10 10:48:18] ...optimizers/trijoint.py.151: total_norm: 11.746685 [I 2022-10-10 10:48:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 720/2304 [I 2022-10-10 10:48:27] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:28 [I 2022-10-10 10:48:27] ...trap/engines/engine.py.229: process: 0.85567 | load: 0.00290 [I 2022-10-10 10:48:27] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-10 10:48:27] ...optimizers/trijoint.py.151: total_norm: 11.784448 [I 2022-10-10 10:48:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 730/2304 [I 2022-10-10 10:48:35] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:10 [I 2022-10-10 10:48:35] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00291 [I 2022-10-10 10:48:35] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-10 10:48:35] ...optimizers/trijoint.py.151: total_norm: 9.061866 [I 2022-10-10 10:48:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 740/2304 [I 2022-10-10 10:48:44] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:03 [I 2022-10-10 10:48:44] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00294 [I 2022-10-10 10:48:44] ...trap/engines/engine.py.230: loss: 0.17673 [I 2022-10-10 10:48:44] ...optimizers/trijoint.py.151: total_norm: 25.653664 [I 2022-10-10 10:48:52] ...trap/engines/engine.py.224: train: epoch 69 | batch 750/2304 [I 2022-10-10 10:48:52] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:21:58 [I 2022-10-10 10:48:52] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00296 [I 2022-10-10 10:48:52] ...trap/engines/engine.py.230: loss: 0.11039 [I 2022-10-10 10:48:52] ...optimizers/trijoint.py.151: total_norm: 12.240107 [I 2022-10-10 10:49:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 760/2304 [I 2022-10-10 10:49:01] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:48 [I 2022-10-10 10:49:01] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00290 [I 2022-10-10 10:49:01] ...trap/engines/engine.py.230: loss: 0.28287 [I 2022-10-10 10:49:01] ...optimizers/trijoint.py.151: total_norm: 30.019632 [I 2022-10-10 10:49:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 770/2304 [I 2022-10-10 10:49:09] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:43 [I 2022-10-10 10:49:09] ...trap/engines/engine.py.229: process: 0.84983 | load: 0.00290 [I 2022-10-10 10:49:09] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-10 10:49:09] ...optimizers/trijoint.py.151: total_norm: 23.167986 [I 2022-10-10 10:49:18] ...trap/engines/engine.py.224: train: epoch 69 | batch 780/2304 [I 2022-10-10 10:49:18] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:34 [I 2022-10-10 10:49:18] ...trap/engines/engine.py.229: process: 0.83959 | load: 0.00295 [I 2022-10-10 10:49:18] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-10 10:49:18] ...optimizers/trijoint.py.151: total_norm: 10.587652 [I 2022-10-10 10:49:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 790/2304 [I 2022-10-10 10:49:26] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:23 [I 2022-10-10 10:49:26] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00291 [I 2022-10-10 10:49:26] ...trap/engines/engine.py.230: loss: 0.14643 [I 2022-10-10 10:49:26] ...optimizers/trijoint.py.151: total_norm: 17.849552 [I 2022-10-10 10:49:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 800/2304 [I 2022-10-10 10:49:35] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:15 [I 2022-10-10 10:49:35] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00295 [I 2022-10-10 10:49:35] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-10 10:49:35] ...optimizers/trijoint.py.151: total_norm: 7.709886 [I 2022-10-10 10:49:43] ...trap/engines/engine.py.224: train: epoch 69 | batch 810/2304 [I 2022-10-10 10:49:43] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:16 [I 2022-10-10 10:49:43] ...trap/engines/engine.py.229: process: 0.87102 | load: 0.00297 [I 2022-10-10 10:49:43] ...trap/engines/engine.py.230: loss: 0.14200 [I 2022-10-10 10:49:43] ...optimizers/trijoint.py.151: total_norm: 14.927893 [I 2022-10-10 10:49:52] ...trap/engines/engine.py.224: train: epoch 69 | batch 820/2304 [I 2022-10-10 10:49:52] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:20:56 [I 2022-10-10 10:49:52] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00290 [I 2022-10-10 10:49:52] ...trap/engines/engine.py.230: loss: 0.10318 [I 2022-10-10 10:49:52] ...optimizers/trijoint.py.151: total_norm: 12.273006 [I 2022-10-10 10:50:00] ...trap/engines/engine.py.224: train: epoch 69 | batch 830/2304 [I 2022-10-10 10:50:00] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:20:50 [I 2022-10-10 10:50:00] ...trap/engines/engine.py.229: process: 0.84651 | load: 0.00289 [I 2022-10-10 10:50:00] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-10 10:50:00] ...optimizers/trijoint.py.151: total_norm: 13.798380 [I 2022-10-10 10:50:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 840/2304 [I 2022-10-10 10:50:09] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:20:45 [I 2022-10-10 10:50:09] ...trap/engines/engine.py.229: process: 0.85468 | load: 0.00303 [I 2022-10-10 10:50:09] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-10 10:50:09] ...optimizers/trijoint.py.151: total_norm: 24.610138 [I 2022-10-10 10:50:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 850/2304 [I 2022-10-10 10:50:17] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:39 [I 2022-10-10 10:50:17] ...trap/engines/engine.py.229: process: 0.85391 | load: 0.00297 [I 2022-10-10 10:50:17] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-10 10:50:17] ...optimizers/trijoint.py.151: total_norm: 9.581378 [I 2022-10-10 10:50:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 860/2304 [I 2022-10-10 10:50:26] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:17 [I 2022-10-10 10:50:26] ...trap/engines/engine.py.229: process: 0.83083 | load: 0.00300 [I 2022-10-10 10:50:26] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-10 10:50:26] ...optimizers/trijoint.py.151: total_norm: 29.157280 [I 2022-10-10 10:50:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 870/2304 [I 2022-10-10 10:50:34] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:20 [I 2022-10-10 10:50:34] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00300 [I 2022-10-10 10:50:34] ...trap/engines/engine.py.230: loss: 0.19213 [I 2022-10-10 10:50:34] ...optimizers/trijoint.py.151: total_norm: 11.179845 [I 2022-10-10 10:50:43] ...trap/engines/engine.py.224: train: epoch 69 | batch 880/2304 [I 2022-10-10 10:50:43] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:10 [I 2022-10-10 10:50:43] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00302 [I 2022-10-10 10:50:43] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-10 10:50:43] ...optimizers/trijoint.py.151: total_norm: 20.667137 [I 2022-10-10 10:50:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 890/2304 [I 2022-10-10 10:50:51] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:19:59 [I 2022-10-10 10:50:51] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00300 [I 2022-10-10 10:50:51] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-10 10:50:51] ...optimizers/trijoint.py.151: total_norm: 24.137747 [I 2022-10-10 10:51:00] ...trap/engines/engine.py.224: train: epoch 69 | batch 900/2304 [I 2022-10-10 10:51:00] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:55 [I 2022-10-10 10:51:00] ...trap/engines/engine.py.229: process: 0.85846 | load: 0.00308 [I 2022-10-10 10:51:00] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-10 10:51:00] ...optimizers/trijoint.py.151: total_norm: 22.932028 [I 2022-10-10 10:51:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 910/2304 [I 2022-10-10 10:51:08] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:19:44 [I 2022-10-10 10:51:08] ...trap/engines/engine.py.229: process: 0.85553 | load: 0.00306 [I 2022-10-10 10:51:08] ...trap/engines/engine.py.230: loss: 0.12258 [I 2022-10-10 10:51:08] ...optimizers/trijoint.py.151: total_norm: 12.131517 [I 2022-10-10 10:51:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 920/2304 [I 2022-10-10 10:51:17] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:36 [I 2022-10-10 10:51:17] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00305 [I 2022-10-10 10:51:17] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-10 10:51:17] ...optimizers/trijoint.py.151: total_norm: 27.683243 [I 2022-10-10 10:51:25] ...trap/engines/engine.py.224: train: epoch 69 | batch 930/2304 [I 2022-10-10 10:51:25] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:36 [I 2022-10-10 10:51:25] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00302 [I 2022-10-10 10:51:25] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-10 10:51:25] ...optimizers/trijoint.py.151: total_norm: 14.827564 [I 2022-10-10 10:51:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 940/2304 [I 2022-10-10 10:51:34] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:19 [I 2022-10-10 10:51:34] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00301 [I 2022-10-10 10:51:34] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-10 10:51:34] ...optimizers/trijoint.py.151: total_norm: 10.295342 [I 2022-10-10 10:51:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 950/2304 [I 2022-10-10 10:51:42] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:19:14 [I 2022-10-10 10:51:42] ...trap/engines/engine.py.229: process: 0.85587 | load: 0.00311 [I 2022-10-10 10:51:42] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-10 10:51:42] ...optimizers/trijoint.py.151: total_norm: 10.971715 [I 2022-10-10 10:51:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 960/2304 [I 2022-10-10 10:51:51] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:00 [I 2022-10-10 10:51:51] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00302 [I 2022-10-10 10:51:51] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-10 10:51:51] ...optimizers/trijoint.py.151: total_norm: 12.629299 [I 2022-10-10 10:51:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 970/2304 [I 2022-10-10 10:51:59] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:18:53 [I 2022-10-10 10:51:59] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00296 [I 2022-10-10 10:51:59] ...trap/engines/engine.py.230: loss: 0.14743 [I 2022-10-10 10:51:59] ...optimizers/trijoint.py.151: total_norm: 17.934814 [I 2022-10-10 10:52:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 980/2304 [I 2022-10-10 10:52:08] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:18:41 [I 2022-10-10 10:52:08] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00297 [I 2022-10-10 10:52:08] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-10 10:52:08] ...optimizers/trijoint.py.151: total_norm: 25.060822 [I 2022-10-10 10:52:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 990/2304 [I 2022-10-10 10:52:16] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:27 [I 2022-10-10 10:52:16] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00293 [I 2022-10-10 10:52:16] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-10 10:52:16] ...optimizers/trijoint.py.151: total_norm: 17.107908 [I 2022-10-10 10:52:25] ...trap/engines/engine.py.224: train: epoch 69 | batch 1000/2304 [I 2022-10-10 10:52:25] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:24 [I 2022-10-10 10:52:25] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00296 [I 2022-10-10 10:52:25] ...trap/engines/engine.py.230: loss: 0.10803 [I 2022-10-10 10:52:25] ...optimizers/trijoint.py.151: total_norm: 26.744059 [I 2022-10-10 10:52:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 1010/2304 [I 2022-10-10 10:52:33] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:21 [I 2022-10-10 10:52:33] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00301 [I 2022-10-10 10:52:33] ...trap/engines/engine.py.230: loss: 0.18632 [I 2022-10-10 10:52:33] ...optimizers/trijoint.py.151: total_norm: 17.984781 [I 2022-10-10 10:52:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 1020/2304 [I 2022-10-10 10:52:42] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:09 [I 2022-10-10 10:52:42] ...trap/engines/engine.py.229: process: 0.86104 | load: 0.00292 [I 2022-10-10 10:52:42] ...trap/engines/engine.py.230: loss: 0.14557 [I 2022-10-10 10:52:42] ...optimizers/trijoint.py.151: total_norm: 11.171907 [I 2022-10-10 10:52:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 1030/2304 [I 2022-10-10 10:52:50] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:04 [I 2022-10-10 10:52:50] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00293 [I 2022-10-10 10:52:50] ...trap/engines/engine.py.230: loss: 0.13336 [I 2022-10-10 10:52:50] ...optimizers/trijoint.py.151: total_norm: 10.600905 [I 2022-10-10 10:52:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 1040/2304 [I 2022-10-10 10:52:59] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:18:00 [I 2022-10-10 10:52:59] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00293 [I 2022-10-10 10:52:59] ...trap/engines/engine.py.230: loss: 0.18673 [I 2022-10-10 10:52:59] ...optimizers/trijoint.py.151: total_norm: 22.585560 [I 2022-10-10 10:53:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1050/2304 [I 2022-10-10 10:53:07] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:17:44 [I 2022-10-10 10:53:07] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00295 [I 2022-10-10 10:53:07] ...trap/engines/engine.py.230: loss: 0.27481 [I 2022-10-10 10:53:07] ...optimizers/trijoint.py.151: total_norm: 18.401213 [I 2022-10-10 10:53:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 1060/2304 [I 2022-10-10 10:53:16] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:17:35 [I 2022-10-10 10:53:16] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00298 [I 2022-10-10 10:53:16] ...trap/engines/engine.py.230: loss: 0.20426 [I 2022-10-10 10:53:16] ...optimizers/trijoint.py.151: total_norm: 20.244850 [I 2022-10-10 10:53:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 1070/2304 [I 2022-10-10 10:53:24] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:23 [I 2022-10-10 10:53:24] ...trap/engines/engine.py.229: process: 0.83821 | load: 0.00299 [I 2022-10-10 10:53:24] ...trap/engines/engine.py.230: loss: 0.12015 [I 2022-10-10 10:53:24] ...optimizers/trijoint.py.151: total_norm: 12.209211 [I 2022-10-10 10:53:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 1080/2304 [I 2022-10-10 10:53:33] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:32 [I 2022-10-10 10:53:33] ...trap/engines/engine.py.229: process: 0.86316 | load: 0.00303 [I 2022-10-10 10:53:33] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-10 10:53:33] ...optimizers/trijoint.py.151: total_norm: 15.076802 [I 2022-10-10 10:53:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 1090/2304 [I 2022-10-10 10:53:42] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:15 [I 2022-10-10 10:53:42] ...trap/engines/engine.py.229: process: 0.84990 | load: 0.00297 [I 2022-10-10 10:53:42] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-10 10:53:42] ...optimizers/trijoint.py.151: total_norm: 9.627768 [I 2022-10-10 10:53:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 1100/2304 [I 2022-10-10 10:53:50] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:02 [I 2022-10-10 10:53:50] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00296 [I 2022-10-10 10:53:50] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-10 10:53:50] ...optimizers/trijoint.py.151: total_norm: 22.758488 [I 2022-10-10 10:53:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 1110/2304 [I 2022-10-10 10:53:59] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:16:55 [I 2022-10-10 10:53:59] ...trap/engines/engine.py.229: process: 0.86356 | load: 0.00295 [I 2022-10-10 10:53:59] ...trap/engines/engine.py.230: loss: 0.13467 [I 2022-10-10 10:53:59] ...optimizers/trijoint.py.151: total_norm: 14.171889 [I 2022-10-10 10:54:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1120/2304 [I 2022-10-10 10:54:07] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:46 [I 2022-10-10 10:54:07] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00294 [I 2022-10-10 10:54:07] ...trap/engines/engine.py.230: loss: 0.21999 [I 2022-10-10 10:54:07] ...optimizers/trijoint.py.151: total_norm: 31.759834 [I 2022-10-10 10:54:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 1130/2304 [I 2022-10-10 10:54:16] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:36 [I 2022-10-10 10:54:16] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00288 [I 2022-10-10 10:54:16] ...trap/engines/engine.py.230: loss: 0.18054 [I 2022-10-10 10:54:16] ...optimizers/trijoint.py.151: total_norm: 30.507504 [I 2022-10-10 10:54:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 1140/2304 [I 2022-10-10 10:54:24] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:27 [I 2022-10-10 10:54:24] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00302 [I 2022-10-10 10:54:24] ...trap/engines/engine.py.230: loss: 0.11156 [I 2022-10-10 10:54:24] ...optimizers/trijoint.py.151: total_norm: 8.300591 [I 2022-10-10 10:54:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 1150/2304 [I 2022-10-10 10:54:33] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:15 [I 2022-10-10 10:54:33] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00290 [I 2022-10-10 10:54:33] ...trap/engines/engine.py.230: loss: 0.12639 [I 2022-10-10 10:54:33] ...optimizers/trijoint.py.151: total_norm: 20.440229 [I 2022-10-10 10:54:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 1160/2304 [I 2022-10-10 10:54:41] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:13 [I 2022-10-10 10:54:41] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00294 [I 2022-10-10 10:54:41] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-10 10:54:41] ...optimizers/trijoint.py.151: total_norm: 22.315517 [I 2022-10-10 10:54:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 1170/2304 [I 2022-10-10 10:54:50] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:05 [I 2022-10-10 10:54:50] ...trap/engines/engine.py.229: process: 0.85940 | load: 0.00290 [I 2022-10-10 10:54:50] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-10 10:54:50] ...optimizers/trijoint.py.151: total_norm: 20.309153 [I 2022-10-10 10:54:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 1180/2304 [I 2022-10-10 10:54:58] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:58 [I 2022-10-10 10:54:58] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00288 [I 2022-10-10 10:54:58] ...trap/engines/engine.py.230: loss: 0.18711 [I 2022-10-10 10:54:58] ...optimizers/trijoint.py.151: total_norm: 15.433517 [I 2022-10-10 10:55:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1190/2304 [I 2022-10-10 10:55:07] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:53 [I 2022-10-10 10:55:07] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00314 [I 2022-10-10 10:55:07] ...trap/engines/engine.py.230: loss: 0.07534 [I 2022-10-10 10:55:07] ...optimizers/trijoint.py.151: total_norm: 7.534515 [I 2022-10-10 10:55:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 1200/2304 [I 2022-10-10 10:55:15] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:38 [I 2022-10-10 10:55:15] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00294 [I 2022-10-10 10:55:15] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-10 10:55:15] ...optimizers/trijoint.py.151: total_norm: 6.703937 [I 2022-10-10 10:55:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 1210/2304 [I 2022-10-10 10:55:24] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:32 [I 2022-10-10 10:55:24] ...trap/engines/engine.py.229: process: 0.86215 | load: 0.00295 [I 2022-10-10 10:55:24] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-10 10:55:24] ...optimizers/trijoint.py.151: total_norm: 16.188700 [I 2022-10-10 10:55:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 1220/2304 [I 2022-10-10 10:55:32] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:18 [I 2022-10-10 10:55:32] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00294 [I 2022-10-10 10:55:32] ...trap/engines/engine.py.230: loss: 0.15967 [I 2022-10-10 10:55:32] ...optimizers/trijoint.py.151: total_norm: 14.429587 [I 2022-10-10 10:55:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 1230/2304 [I 2022-10-10 10:55:41] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:14 [I 2022-10-10 10:55:41] ...trap/engines/engine.py.229: process: 0.84054 | load: 0.00295 [I 2022-10-10 10:55:41] ...trap/engines/engine.py.230: loss: 0.11792 [I 2022-10-10 10:55:41] ...optimizers/trijoint.py.151: total_norm: 12.560487 [I 2022-10-10 10:55:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 1240/2304 [I 2022-10-10 10:55:50] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:07 [I 2022-10-10 10:55:50] ...trap/engines/engine.py.229: process: 0.84665 | load: 0.00295 [I 2022-10-10 10:55:50] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-10 10:55:50] ...optimizers/trijoint.py.151: total_norm: 23.288210 [I 2022-10-10 10:55:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 1250/2304 [I 2022-10-10 10:55:58] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:57 [I 2022-10-10 10:55:58] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00297 [I 2022-10-10 10:55:58] ...trap/engines/engine.py.230: loss: 0.11163 [I 2022-10-10 10:55:58] ...optimizers/trijoint.py.151: total_norm: 9.107489 [I 2022-10-10 10:56:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1260/2304 [I 2022-10-10 10:56:07] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:14:54 [I 2022-10-10 10:56:07] ...trap/engines/engine.py.229: process: 0.88096 | load: 0.00293 [I 2022-10-10 10:56:07] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-10 10:56:07] ...optimizers/trijoint.py.151: total_norm: 13.492744 [I 2022-10-10 10:56:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 1270/2304 [I 2022-10-10 10:56:15] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:40 [I 2022-10-10 10:56:15] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00291 [I 2022-10-10 10:56:15] ...trap/engines/engine.py.230: loss: 0.16902 [I 2022-10-10 10:56:15] ...optimizers/trijoint.py.151: total_norm: 8.374707 [I 2022-10-10 10:56:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 1280/2304 [I 2022-10-10 10:56:24] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:32 [I 2022-10-10 10:56:24] ...trap/engines/engine.py.229: process: 0.82923 | load: 0.00297 [I 2022-10-10 10:56:24] ...trap/engines/engine.py.230: loss: 0.15928 [I 2022-10-10 10:56:24] ...optimizers/trijoint.py.151: total_norm: 11.989586 [I 2022-10-10 10:56:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 1290/2304 [I 2022-10-10 10:56:32] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:27 [I 2022-10-10 10:56:32] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00293 [I 2022-10-10 10:56:32] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-10 10:56:32] ...optimizers/trijoint.py.151: total_norm: 15.074875 [I 2022-10-10 10:56:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 1300/2304 [I 2022-10-10 10:56:41] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:09 [I 2022-10-10 10:56:41] ...trap/engines/engine.py.229: process: 0.83816 | load: 0.00294 [I 2022-10-10 10:56:41] ...trap/engines/engine.py.230: loss: 0.11278 [I 2022-10-10 10:56:41] ...optimizers/trijoint.py.151: total_norm: 18.707485 [I 2022-10-10 10:56:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 1310/2304 [I 2022-10-10 10:56:49] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:04 [I 2022-10-10 10:56:49] ...trap/engines/engine.py.229: process: 0.85604 | load: 0.00309 [I 2022-10-10 10:56:49] ...trap/engines/engine.py.230: loss: 0.14168 [I 2022-10-10 10:56:49] ...optimizers/trijoint.py.151: total_norm: 12.111837 [I 2022-10-10 10:56:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 1320/2304 [I 2022-10-10 10:56:58] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:56 [I 2022-10-10 10:56:58] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00294 [I 2022-10-10 10:56:58] ...trap/engines/engine.py.230: loss: 0.09595 [I 2022-10-10 10:56:58] ...optimizers/trijoint.py.151: total_norm: 7.578088 [I 2022-10-10 10:57:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 1330/2304 [I 2022-10-10 10:57:06] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:49 [I 2022-10-10 10:57:06] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00292 [I 2022-10-10 10:57:06] ...trap/engines/engine.py.230: loss: 0.21855 [I 2022-10-10 10:57:06] ...optimizers/trijoint.py.151: total_norm: 25.631281 [I 2022-10-10 10:57:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 1340/2304 [I 2022-10-10 10:57:15] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:33 [I 2022-10-10 10:57:15] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00291 [I 2022-10-10 10:57:15] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-10 10:57:15] ...optimizers/trijoint.py.151: total_norm: 24.076668 [I 2022-10-10 10:57:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1350/2304 [I 2022-10-10 10:57:23] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:32 [I 2022-10-10 10:57:23] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00290 [I 2022-10-10 10:57:23] ...trap/engines/engine.py.230: loss: 0.14771 [I 2022-10-10 10:57:23] ...optimizers/trijoint.py.151: total_norm: 18.971247 [I 2022-10-10 10:57:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 1360/2304 [I 2022-10-10 10:57:32] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:20 [I 2022-10-10 10:57:32] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00293 [I 2022-10-10 10:57:32] ...trap/engines/engine.py.230: loss: 0.09921 [I 2022-10-10 10:57:32] ...optimizers/trijoint.py.151: total_norm: 12.980360 [I 2022-10-10 10:57:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1370/2304 [I 2022-10-10 10:57:40] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:14 [I 2022-10-10 10:57:40] ...trap/engines/engine.py.229: process: 0.83851 | load: 0.00302 [I 2022-10-10 10:57:40] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-10 10:57:40] ...optimizers/trijoint.py.151: total_norm: 10.668600 [I 2022-10-10 10:57:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 1380/2304 [I 2022-10-10 10:57:49] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:07 [I 2022-10-10 10:57:49] ...trap/engines/engine.py.229: process: 0.85912 | load: 0.00299 [I 2022-10-10 10:57:49] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-10 10:57:49] ...optimizers/trijoint.py.151: total_norm: 13.411167 [I 2022-10-10 10:57:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 1390/2304 [I 2022-10-10 10:57:57] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:56 [I 2022-10-10 10:57:57] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00296 [I 2022-10-10 10:57:57] ...trap/engines/engine.py.230: loss: 0.17317 [I 2022-10-10 10:57:57] ...optimizers/trijoint.py.151: total_norm: 25.542879 [I 2022-10-10 10:58:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 1400/2304 [I 2022-10-10 10:58:06] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:46 [I 2022-10-10 10:58:06] ...trap/engines/engine.py.229: process: 0.84260 | load: 0.00296 [I 2022-10-10 10:58:06] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-10 10:58:06] ...optimizers/trijoint.py.151: total_norm: 25.819664 [I 2022-10-10 10:58:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 1410/2304 [I 2022-10-10 10:58:14] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:43 [I 2022-10-10 10:58:14] ...trap/engines/engine.py.229: process: 0.85674 | load: 0.00292 [I 2022-10-10 10:58:14] ...trap/engines/engine.py.230: loss: 0.08768 [I 2022-10-10 10:58:14] ...optimizers/trijoint.py.151: total_norm: 9.888702 [I 2022-10-10 10:58:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1420/2304 [I 2022-10-10 10:58:23] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:33 [I 2022-10-10 10:58:23] ...trap/engines/engine.py.229: process: 0.85803 | load: 0.00299 [I 2022-10-10 10:58:23] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-10 10:58:23] ...optimizers/trijoint.py.151: total_norm: 14.557362 [I 2022-10-10 10:58:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 1430/2304 [I 2022-10-10 10:58:31] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:22 [I 2022-10-10 10:58:31] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00290 [I 2022-10-10 10:58:31] ...trap/engines/engine.py.230: loss: 0.09674 [I 2022-10-10 10:58:31] ...optimizers/trijoint.py.151: total_norm: 10.783640 [I 2022-10-10 10:58:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1440/2304 [I 2022-10-10 10:58:40] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:16 [I 2022-10-10 10:58:40] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00292 [I 2022-10-10 10:58:40] ...trap/engines/engine.py.230: loss: 0.16080 [I 2022-10-10 10:58:40] ...optimizers/trijoint.py.151: total_norm: 15.338548 [I 2022-10-10 10:58:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 1450/2304 [I 2022-10-10 10:58:49] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:08 [I 2022-10-10 10:58:49] ...trap/engines/engine.py.229: process: 0.86313 | load: 0.00291 [I 2022-10-10 10:58:49] ...trap/engines/engine.py.230: loss: 0.20570 [I 2022-10-10 10:58:49] ...optimizers/trijoint.py.151: total_norm: 56.774666 [I 2022-10-10 10:58:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 1460/2304 [I 2022-10-10 10:58:57] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:56 [I 2022-10-10 10:58:57] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00296 [I 2022-10-10 10:58:57] ...trap/engines/engine.py.230: loss: 0.16736 [I 2022-10-10 10:58:57] ...optimizers/trijoint.py.151: total_norm: 11.127079 [I 2022-10-10 10:59:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 1470/2304 [I 2022-10-10 10:59:06] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:46 [I 2022-10-10 10:59:06] ...trap/engines/engine.py.229: process: 0.83994 | load: 0.00293 [I 2022-10-10 10:59:06] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-10 10:59:06] ...optimizers/trijoint.py.151: total_norm: 8.995265 [I 2022-10-10 10:59:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 1480/2304 [I 2022-10-10 10:59:14] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:39 [I 2022-10-10 10:59:14] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00292 [I 2022-10-10 10:59:14] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-10 10:59:14] ...optimizers/trijoint.py.151: total_norm: 11.591388 [I 2022-10-10 10:59:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1490/2304 [I 2022-10-10 10:59:23] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:36 [I 2022-10-10 10:59:23] ...trap/engines/engine.py.229: process: 0.87004 | load: 0.00293 [I 2022-10-10 10:59:23] ...trap/engines/engine.py.230: loss: 0.15774 [I 2022-10-10 10:59:23] ...optimizers/trijoint.py.151: total_norm: 22.734310 [I 2022-10-10 10:59:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 1500/2304 [I 2022-10-10 10:59:31] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:25 [I 2022-10-10 10:59:31] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00300 [I 2022-10-10 10:59:31] ...trap/engines/engine.py.230: loss: 0.14269 [I 2022-10-10 10:59:31] ...optimizers/trijoint.py.151: total_norm: 18.332895 [I 2022-10-10 10:59:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1510/2304 [I 2022-10-10 10:59:40] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:20 [I 2022-10-10 10:59:40] ...trap/engines/engine.py.229: process: 0.85748 | load: 0.00296 [I 2022-10-10 10:59:40] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-10 10:59:40] ...optimizers/trijoint.py.151: total_norm: 17.225393 [I 2022-10-10 10:59:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 1520/2304 [I 2022-10-10 10:59:48] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:06 [I 2022-10-10 10:59:48] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00298 [I 2022-10-10 10:59:48] ...trap/engines/engine.py.230: loss: 0.08995 [I 2022-10-10 10:59:48] ...optimizers/trijoint.py.151: total_norm: 9.976080 [I 2022-10-10 10:59:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 1530/2304 [I 2022-10-10 10:59:57] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:56 [I 2022-10-10 10:59:57] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00296 [I 2022-10-10 10:59:57] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-10 10:59:57] ...optimizers/trijoint.py.151: total_norm: 20.656141 [I 2022-10-10 11:00:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 1540/2304 [I 2022-10-10 11:00:05] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:47 [I 2022-10-10 11:00:05] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00299 [I 2022-10-10 11:00:05] ...trap/engines/engine.py.230: loss: 0.18490 [I 2022-10-10 11:00:05] ...optimizers/trijoint.py.151: total_norm: 14.529082 [I 2022-10-10 11:00:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 1550/2304 [I 2022-10-10 11:00:14] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:42 [I 2022-10-10 11:00:14] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00292 [I 2022-10-10 11:00:14] ...trap/engines/engine.py.230: loss: 0.15594 [I 2022-10-10 11:00:14] ...optimizers/trijoint.py.151: total_norm: 16.141781 [I 2022-10-10 11:00:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 1560/2304 [I 2022-10-10 11:00:22] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:26 [I 2022-10-10 11:00:22] ...trap/engines/engine.py.229: process: 0.83821 | load: 0.00291 [I 2022-10-10 11:00:22] ...trap/engines/engine.py.230: loss: 0.10241 [I 2022-10-10 11:00:22] ...optimizers/trijoint.py.151: total_norm: 11.643788 [I 2022-10-10 11:00:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 1570/2304 [I 2022-10-10 11:00:31] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:23 [I 2022-10-10 11:00:31] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00297 [I 2022-10-10 11:00:31] ...trap/engines/engine.py.230: loss: 0.09805 [I 2022-10-10 11:00:31] ...optimizers/trijoint.py.151: total_norm: 12.976391 [I 2022-10-10 11:00:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 1580/2304 [I 2022-10-10 11:00:39] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:16 [I 2022-10-10 11:00:39] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00293 [I 2022-10-10 11:00:39] ...trap/engines/engine.py.230: loss: 0.11652 [I 2022-10-10 11:00:39] ...optimizers/trijoint.py.151: total_norm: 31.446213 [I 2022-10-10 11:00:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 1590/2304 [I 2022-10-10 11:00:48] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:05 [I 2022-10-10 11:00:48] ...trap/engines/engine.py.229: process: 0.84356 | load: 0.00297 [I 2022-10-10 11:00:48] ...trap/engines/engine.py.230: loss: 0.19853 [I 2022-10-10 11:00:48] ...optimizers/trijoint.py.151: total_norm: 55.175220 [I 2022-10-10 11:00:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1600/2304 [I 2022-10-10 11:00:56] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:09:59 [I 2022-10-10 11:00:56] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00291 [I 2022-10-10 11:00:56] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-10 11:00:56] ...optimizers/trijoint.py.151: total_norm: 20.209261 [I 2022-10-10 11:01:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 1610/2304 [I 2022-10-10 11:01:05] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:47 [I 2022-10-10 11:01:05] ...trap/engines/engine.py.229: process: 0.84777 | load: 0.00293 [I 2022-10-10 11:01:05] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-10 11:01:05] ...optimizers/trijoint.py.151: total_norm: 9.626570 [I 2022-10-10 11:01:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 1620/2304 [I 2022-10-10 11:01:13] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:40 [I 2022-10-10 11:01:13] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00296 [I 2022-10-10 11:01:13] ...trap/engines/engine.py.230: loss: 0.23806 [I 2022-10-10 11:01:13] ...optimizers/trijoint.py.151: total_norm: 30.646414 [I 2022-10-10 11:01:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 1630/2304 [I 2022-10-10 11:01:22] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:34 [I 2022-10-10 11:01:22] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00292 [I 2022-10-10 11:01:22] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-10 11:01:22] ...optimizers/trijoint.py.151: total_norm: 35.716045 [I 2022-10-10 11:01:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 1640/2304 [I 2022-10-10 11:01:31] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:27 [I 2022-10-10 11:01:31] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00304 [I 2022-10-10 11:01:31] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-10 11:01:31] ...optimizers/trijoint.py.151: total_norm: 7.535804 [I 2022-10-10 11:01:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 1650/2304 [I 2022-10-10 11:01:39] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:15 [I 2022-10-10 11:01:39] ...trap/engines/engine.py.229: process: 0.84603 | load: 0.00306 [I 2022-10-10 11:01:39] ...trap/engines/engine.py.230: loss: 0.18833 [I 2022-10-10 11:01:39] ...optimizers/trijoint.py.151: total_norm: 15.733627 [I 2022-10-10 11:01:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 1660/2304 [I 2022-10-10 11:01:48] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:10 [I 2022-10-10 11:01:48] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00303 [I 2022-10-10 11:01:48] ...trap/engines/engine.py.230: loss: 0.12767 [I 2022-10-10 11:01:48] ...optimizers/trijoint.py.151: total_norm: 47.287254 [I 2022-10-10 11:01:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1670/2304 [I 2022-10-10 11:01:56] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:00 [I 2022-10-10 11:01:56] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00291 [I 2022-10-10 11:01:56] ...trap/engines/engine.py.230: loss: 0.12837 [I 2022-10-10 11:01:56] ...optimizers/trijoint.py.151: total_norm: 14.681522 [I 2022-10-10 11:02:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 1680/2304 [I 2022-10-10 11:02:05] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:50 [I 2022-10-10 11:02:05] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00290 [I 2022-10-10 11:02:05] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-10 11:02:05] ...optimizers/trijoint.py.151: total_norm: 9.590583 [I 2022-10-10 11:02:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 1690/2304 [I 2022-10-10 11:02:13] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:46 [I 2022-10-10 11:02:13] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00290 [I 2022-10-10 11:02:13] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-10 11:02:13] ...optimizers/trijoint.py.151: total_norm: 11.350162 [I 2022-10-10 11:02:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 1700/2304 [I 2022-10-10 11:02:22] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:34 [I 2022-10-10 11:02:22] ...trap/engines/engine.py.229: process: 0.84898 | load: 0.00292 [I 2022-10-10 11:02:22] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-10 11:02:22] ...optimizers/trijoint.py.151: total_norm: 12.375250 [I 2022-10-10 11:02:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1710/2304 [I 2022-10-10 11:02:30] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:25 [I 2022-10-10 11:02:30] ...trap/engines/engine.py.229: process: 0.84649 | load: 0.00289 [I 2022-10-10 11:02:30] ...trap/engines/engine.py.230: loss: 0.14956 [I 2022-10-10 11:02:30] ...optimizers/trijoint.py.151: total_norm: 26.677753 [I 2022-10-10 11:02:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 1720/2304 [I 2022-10-10 11:02:39] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:14 [I 2022-10-10 11:02:39] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00298 [I 2022-10-10 11:02:39] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-10 11:02:39] ...optimizers/trijoint.py.151: total_norm: 20.805801 [I 2022-10-10 11:02:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1730/2304 [I 2022-10-10 11:02:47] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:04 [I 2022-10-10 11:02:47] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00297 [I 2022-10-10 11:02:47] ...trap/engines/engine.py.230: loss: 0.09011 [I 2022-10-10 11:02:47] ...optimizers/trijoint.py.151: total_norm: 7.293029 [I 2022-10-10 11:02:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1740/2304 [I 2022-10-10 11:02:56] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:07:58 [I 2022-10-10 11:02:56] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00297 [I 2022-10-10 11:02:56] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-10 11:02:56] ...optimizers/trijoint.py.151: total_norm: 19.304365 [I 2022-10-10 11:03:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1750/2304 [I 2022-10-10 11:03:04] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:54 [I 2022-10-10 11:03:04] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00294 [I 2022-10-10 11:03:04] ...trap/engines/engine.py.230: loss: 0.23314 [I 2022-10-10 11:03:04] ...optimizers/trijoint.py.151: total_norm: 24.681881 [I 2022-10-10 11:03:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 1760/2304 [I 2022-10-10 11:03:13] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:40 [I 2022-10-10 11:03:13] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00291 [I 2022-10-10 11:03:13] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-10 11:03:13] ...optimizers/trijoint.py.151: total_norm: 26.123079 [I 2022-10-10 11:03:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 1770/2304 [I 2022-10-10 11:03:22] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:34 [I 2022-10-10 11:03:22] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00293 [I 2022-10-10 11:03:22] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-10 11:03:22] ...optimizers/trijoint.py.151: total_norm: 15.868600 [I 2022-10-10 11:03:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1780/2304 [I 2022-10-10 11:03:30] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:25 [I 2022-10-10 11:03:30] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00292 [I 2022-10-10 11:03:30] ...trap/engines/engine.py.230: loss: 0.09816 [I 2022-10-10 11:03:30] ...optimizers/trijoint.py.151: total_norm: 8.869982 [I 2022-10-10 11:03:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 1790/2304 [I 2022-10-10 11:03:39] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:15 [I 2022-10-10 11:03:39] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00296 [I 2022-10-10 11:03:39] ...trap/engines/engine.py.230: loss: 0.10334 [I 2022-10-10 11:03:39] ...optimizers/trijoint.py.151: total_norm: 9.861905 [I 2022-10-10 11:03:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1800/2304 [I 2022-10-10 11:03:47] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:12 [I 2022-10-10 11:03:47] ...trap/engines/engine.py.229: process: 0.87122 | load: 0.00297 [I 2022-10-10 11:03:47] ...trap/engines/engine.py.230: loss: 0.32068 [I 2022-10-10 11:03:47] ...optimizers/trijoint.py.151: total_norm: 15.428367 [I 2022-10-10 11:03:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1810/2304 [I 2022-10-10 11:03:56] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:02 [I 2022-10-10 11:03:56] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00296 [I 2022-10-10 11:03:56] ...trap/engines/engine.py.230: loss: 0.13747 [I 2022-10-10 11:03:56] ...optimizers/trijoint.py.151: total_norm: 13.451148 [I 2022-10-10 11:04:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1820/2304 [I 2022-10-10 11:04:04] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:49 [I 2022-10-10 11:04:04] ...trap/engines/engine.py.229: process: 0.83469 | load: 0.00292 [I 2022-10-10 11:04:04] ...trap/engines/engine.py.230: loss: 0.18439 [I 2022-10-10 11:04:04] ...optimizers/trijoint.py.151: total_norm: 11.617965 [I 2022-10-10 11:04:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 1830/2304 [I 2022-10-10 11:04:13] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:06:42 [I 2022-10-10 11:04:13] ...trap/engines/engine.py.229: process: 0.86286 | load: 0.00296 [I 2022-10-10 11:04:13] ...trap/engines/engine.py.230: loss: 0.09063 [I 2022-10-10 11:04:13] ...optimizers/trijoint.py.151: total_norm: 10.690852 [I 2022-10-10 11:04:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 1840/2304 [I 2022-10-10 11:04:21] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:34 [I 2022-10-10 11:04:21] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00294 [I 2022-10-10 11:04:21] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-10 11:04:21] ...optimizers/trijoint.py.151: total_norm: 16.916065 [I 2022-10-10 11:04:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1850/2304 [I 2022-10-10 11:04:30] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:25 [I 2022-10-10 11:04:30] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00295 [I 2022-10-10 11:04:30] ...trap/engines/engine.py.230: loss: 0.11552 [I 2022-10-10 11:04:30] ...optimizers/trijoint.py.151: total_norm: 26.432056 [I 2022-10-10 11:04:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 1860/2304 [I 2022-10-10 11:04:38] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:16 [I 2022-10-10 11:04:38] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00294 [I 2022-10-10 11:04:38] ...trap/engines/engine.py.230: loss: 0.13573 [I 2022-10-10 11:04:38] ...optimizers/trijoint.py.151: total_norm: 42.903042 [I 2022-10-10 11:04:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1870/2304 [I 2022-10-10 11:04:47] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:09 [I 2022-10-10 11:04:47] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00299 [I 2022-10-10 11:04:47] ...trap/engines/engine.py.230: loss: 0.10715 [I 2022-10-10 11:04:47] ...optimizers/trijoint.py.151: total_norm: 8.525006 [I 2022-10-10 11:04:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 1880/2304 [I 2022-10-10 11:04:55] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:01 [I 2022-10-10 11:04:55] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00291 [I 2022-10-10 11:04:55] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-10 11:04:55] ...optimizers/trijoint.py.151: total_norm: 16.102947 [I 2022-10-10 11:05:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1890/2304 [I 2022-10-10 11:05:04] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:55 [I 2022-10-10 11:05:04] ...trap/engines/engine.py.229: process: 0.86874 | load: 0.00303 [I 2022-10-10 11:05:04] ...trap/engines/engine.py.230: loss: 0.12562 [I 2022-10-10 11:05:04] ...optimizers/trijoint.py.151: total_norm: 11.323071 [I 2022-10-10 11:05:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 1900/2304 [I 2022-10-10 11:05:12] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:47 [I 2022-10-10 11:05:12] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00298 [I 2022-10-10 11:05:12] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-10 11:05:12] ...optimizers/trijoint.py.151: total_norm: 12.724057 [I 2022-10-10 11:05:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 1910/2304 [I 2022-10-10 11:05:21] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:38 [I 2022-10-10 11:05:21] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00297 [I 2022-10-10 11:05:21] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-10 11:05:21] ...optimizers/trijoint.py.151: total_norm: 8.967948 [I 2022-10-10 11:05:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1920/2304 [I 2022-10-10 11:05:30] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:28 [I 2022-10-10 11:05:30] ...trap/engines/engine.py.229: process: 0.85822 | load: 0.00294 [I 2022-10-10 11:05:30] ...trap/engines/engine.py.230: loss: 0.15327 [I 2022-10-10 11:05:30] ...optimizers/trijoint.py.151: total_norm: 16.520775 [I 2022-10-10 11:05:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 1930/2304 [I 2022-10-10 11:05:38] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:19 [I 2022-10-10 11:05:38] ...trap/engines/engine.py.229: process: 0.86223 | load: 0.00295 [I 2022-10-10 11:05:38] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-10 11:05:38] ...optimizers/trijoint.py.151: total_norm: 14.575561 [I 2022-10-10 11:05:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1940/2304 [I 2022-10-10 11:05:47] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:13 [I 2022-10-10 11:05:47] ...trap/engines/engine.py.229: process: 0.85085 | load: 0.00299 [I 2022-10-10 11:05:47] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-10 11:05:47] ...optimizers/trijoint.py.151: total_norm: 22.895824 [I 2022-10-10 11:05:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 1950/2304 [I 2022-10-10 11:05:55] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:02 [I 2022-10-10 11:05:55] ...trap/engines/engine.py.229: process: 0.85810 | load: 0.00293 [I 2022-10-10 11:05:55] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-10 11:05:55] ...optimizers/trijoint.py.151: total_norm: 16.432999 [I 2022-10-10 11:06:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1960/2304 [I 2022-10-10 11:06:04] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:04:52 [I 2022-10-10 11:06:04] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00295 [I 2022-10-10 11:06:04] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-10 11:06:04] ...optimizers/trijoint.py.151: total_norm: 30.503815 [I 2022-10-10 11:06:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 1970/2304 [I 2022-10-10 11:06:12] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:45 [I 2022-10-10 11:06:12] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00293 [I 2022-10-10 11:06:12] ...trap/engines/engine.py.230: loss: 0.15804 [I 2022-10-10 11:06:12] ...optimizers/trijoint.py.151: total_norm: 21.255484 [I 2022-10-10 11:06:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 1980/2304 [I 2022-10-10 11:06:21] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:35 [I 2022-10-10 11:06:21] ...trap/engines/engine.py.229: process: 0.84661 | load: 0.00294 [I 2022-10-10 11:06:21] ...trap/engines/engine.py.230: loss: 0.10409 [I 2022-10-10 11:06:21] ...optimizers/trijoint.py.151: total_norm: 13.520083 [I 2022-10-10 11:06:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 1990/2304 [I 2022-10-10 11:06:29] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:27 [I 2022-10-10 11:06:29] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00298 [I 2022-10-10 11:06:29] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-10 11:06:29] ...optimizers/trijoint.py.151: total_norm: 36.179726 [I 2022-10-10 11:06:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 2000/2304 [I 2022-10-10 11:06:38] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:20 [I 2022-10-10 11:06:38] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00290 [I 2022-10-10 11:06:38] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-10 11:06:38] ...optimizers/trijoint.py.151: total_norm: 56.067226 [I 2022-10-10 11:06:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 2010/2304 [I 2022-10-10 11:06:47] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:11 [I 2022-10-10 11:06:47] ...trap/engines/engine.py.229: process: 0.86799 | load: 0.00315 [I 2022-10-10 11:06:47] ...trap/engines/engine.py.230: loss: 0.12776 [I 2022-10-10 11:06:47] ...optimizers/trijoint.py.151: total_norm: 10.783274 [I 2022-10-10 11:06:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 2020/2304 [I 2022-10-10 11:06:55] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:00 [I 2022-10-10 11:06:55] ...trap/engines/engine.py.229: process: 0.84752 | load: 0.00303 [I 2022-10-10 11:06:55] ...trap/engines/engine.py.230: loss: 0.22171 [I 2022-10-10 11:06:55] ...optimizers/trijoint.py.151: total_norm: 15.950101 [I 2022-10-10 11:07:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 2030/2304 [I 2022-10-10 11:07:04] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:53 [I 2022-10-10 11:07:04] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00294 [I 2022-10-10 11:07:04] ...trap/engines/engine.py.230: loss: 0.19116 [I 2022-10-10 11:07:04] ...optimizers/trijoint.py.151: total_norm: 32.809273 [I 2022-10-10 11:07:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 2040/2304 [I 2022-10-10 11:07:12] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:03:44 [I 2022-10-10 11:07:12] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00291 [I 2022-10-10 11:07:12] ...trap/engines/engine.py.230: loss: 0.19347 [I 2022-10-10 11:07:12] ...optimizers/trijoint.py.151: total_norm: 21.756178 [I 2022-10-10 11:07:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 2050/2304 [I 2022-10-10 11:07:21] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:03:36 [I 2022-10-10 11:07:21] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00298 [I 2022-10-10 11:07:21] ...trap/engines/engine.py.230: loss: 0.13310 [I 2022-10-10 11:07:21] ...optimizers/trijoint.py.151: total_norm: 19.171001 [I 2022-10-10 11:07:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 2060/2304 [I 2022-10-10 11:07:29] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:28 [I 2022-10-10 11:07:29] ...trap/engines/engine.py.229: process: 0.85929 | load: 0.00295 [I 2022-10-10 11:07:29] ...trap/engines/engine.py.230: loss: 0.19780 [I 2022-10-10 11:07:29] ...optimizers/trijoint.py.151: total_norm: 15.900194 [I 2022-10-10 11:07:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 2070/2304 [I 2022-10-10 11:07:38] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:18 [I 2022-10-10 11:07:38] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00296 [I 2022-10-10 11:07:38] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-10 11:07:38] ...optimizers/trijoint.py.151: total_norm: 14.683317 [I 2022-10-10 11:07:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 2080/2304 [I 2022-10-10 11:07:46] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:11 [I 2022-10-10 11:07:46] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00295 [I 2022-10-10 11:07:46] ...trap/engines/engine.py.230: loss: 0.10891 [I 2022-10-10 11:07:46] ...optimizers/trijoint.py.151: total_norm: 10.887139 [I 2022-10-10 11:07:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 2090/2304 [I 2022-10-10 11:07:55] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:01 [I 2022-10-10 11:07:55] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00292 [I 2022-10-10 11:07:55] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-10 11:07:55] ...optimizers/trijoint.py.151: total_norm: 10.053771 [I 2022-10-10 11:08:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 2100/2304 [I 2022-10-10 11:08:03] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:02:52 [I 2022-10-10 11:08:03] ...trap/engines/engine.py.229: process: 0.83193 | load: 0.00295 [I 2022-10-10 11:08:03] ...trap/engines/engine.py.230: loss: 0.08928 [I 2022-10-10 11:08:03] ...optimizers/trijoint.py.151: total_norm: 6.371507 [I 2022-10-10 11:08:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 2110/2304 [I 2022-10-10 11:08:12] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:42 [I 2022-10-10 11:08:12] ...trap/engines/engine.py.229: process: 0.83849 | load: 0.00296 [I 2022-10-10 11:08:12] ...trap/engines/engine.py.230: loss: 0.14300 [I 2022-10-10 11:08:12] ...optimizers/trijoint.py.151: total_norm: 17.067236 [I 2022-10-10 11:08:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 2120/2304 [I 2022-10-10 11:08:20] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:02:35 [I 2022-10-10 11:08:20] ...trap/engines/engine.py.229: process: 0.84527 | load: 0.00297 [I 2022-10-10 11:08:20] ...trap/engines/engine.py.230: loss: 0.13920 [I 2022-10-10 11:08:20] ...optimizers/trijoint.py.151: total_norm: 10.630326 [I 2022-10-10 11:08:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 2130/2304 [I 2022-10-10 11:08:29] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:28 [I 2022-10-10 11:08:29] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00299 [I 2022-10-10 11:08:29] ...trap/engines/engine.py.230: loss: 0.18537 [I 2022-10-10 11:08:29] ...optimizers/trijoint.py.151: total_norm: 33.100384 [I 2022-10-10 11:08:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 2140/2304 [I 2022-10-10 11:08:37] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:19 [I 2022-10-10 11:08:37] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00294 [I 2022-10-10 11:08:37] ...trap/engines/engine.py.230: loss: 0.12272 [I 2022-10-10 11:08:37] ...optimizers/trijoint.py.151: total_norm: 15.342810 [I 2022-10-10 11:08:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 2150/2304 [I 2022-10-10 11:08:46] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:11 [I 2022-10-10 11:08:46] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00289 [I 2022-10-10 11:08:46] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-10 11:08:46] ...optimizers/trijoint.py.151: total_norm: 14.801403 [I 2022-10-10 11:08:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 2160/2304 [I 2022-10-10 11:08:54] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:02 [I 2022-10-10 11:08:54] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00300 [I 2022-10-10 11:08:54] ...trap/engines/engine.py.230: loss: 0.09500 [I 2022-10-10 11:08:54] ...optimizers/trijoint.py.151: total_norm: 15.354181 [I 2022-10-10 11:09:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 2170/2304 [I 2022-10-10 11:09:03] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:01:54 [I 2022-10-10 11:09:03] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00290 [I 2022-10-10 11:09:03] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-10 11:09:03] ...optimizers/trijoint.py.151: total_norm: 19.618380 [I 2022-10-10 11:09:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 2180/2304 [I 2022-10-10 11:09:11] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:01:45 [I 2022-10-10 11:09:11] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00288 [I 2022-10-10 11:09:11] ...trap/engines/engine.py.230: loss: 0.14773 [I 2022-10-10 11:09:11] ...optimizers/trijoint.py.151: total_norm: 9.489395 [I 2022-10-10 11:09:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 2190/2304 [I 2022-10-10 11:09:20] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:37 [I 2022-10-10 11:09:20] ...trap/engines/engine.py.229: process: 0.85770 | load: 0.00295 [I 2022-10-10 11:09:20] ...trap/engines/engine.py.230: loss: 0.13227 [I 2022-10-10 11:09:20] ...optimizers/trijoint.py.151: total_norm: 17.040802 [I 2022-10-10 11:09:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 2200/2304 [I 2022-10-10 11:09:29] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:28 [I 2022-10-10 11:09:29] ...trap/engines/engine.py.229: process: 0.84978 | load: 0.00292 [I 2022-10-10 11:09:29] ...trap/engines/engine.py.230: loss: 0.09873 [I 2022-10-10 11:09:29] ...optimizers/trijoint.py.151: total_norm: 7.243341 [I 2022-10-10 11:09:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 2210/2304 [I 2022-10-10 11:09:37] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:20 [I 2022-10-10 11:09:37] ...trap/engines/engine.py.229: process: 0.87332 | load: 0.00293 [I 2022-10-10 11:09:37] ...trap/engines/engine.py.230: loss: 0.13282 [I 2022-10-10 11:09:37] ...optimizers/trijoint.py.151: total_norm: 23.925474 [I 2022-10-10 11:09:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 2220/2304 [I 2022-10-10 11:09:46] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:11 [I 2022-10-10 11:09:46] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00298 [I 2022-10-10 11:09:46] ...trap/engines/engine.py.230: loss: 0.15246 [I 2022-10-10 11:09:46] ...optimizers/trijoint.py.151: total_norm: 38.830341 [I 2022-10-10 11:09:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 2230/2304 [I 2022-10-10 11:09:54] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:02 [I 2022-10-10 11:09:54] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00291 [I 2022-10-10 11:09:54] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-10 11:09:54] ...optimizers/trijoint.py.151: total_norm: 13.896142 [I 2022-10-10 11:10:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 2240/2304 [I 2022-10-10 11:10:03] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:54 [I 2022-10-10 11:10:03] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00290 [I 2022-10-10 11:10:03] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-10 11:10:03] ...optimizers/trijoint.py.151: total_norm: 20.330490 [I 2022-10-10 11:10:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 2250/2304 [I 2022-10-10 11:10:11] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:45 [I 2022-10-10 11:10:11] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00292 [I 2022-10-10 11:10:11] ...trap/engines/engine.py.230: loss: 0.20703 [I 2022-10-10 11:10:11] ...optimizers/trijoint.py.151: total_norm: 37.124268 [I 2022-10-10 11:10:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 2260/2304 [I 2022-10-10 11:10:20] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:37 [I 2022-10-10 11:10:20] ...trap/engines/engine.py.229: process: 0.86437 | load: 0.00297 [I 2022-10-10 11:10:20] ...trap/engines/engine.py.230: loss: 0.20394 [I 2022-10-10 11:10:20] ...optimizers/trijoint.py.151: total_norm: 28.415287 [I 2022-10-10 11:10:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 2270/2304 [I 2022-10-10 11:10:28] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:28 [I 2022-10-10 11:10:28] ...trap/engines/engine.py.229: process: 0.86228 | load: 0.00299 [I 2022-10-10 11:10:28] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-10 11:10:28] ...optimizers/trijoint.py.151: total_norm: 33.527721 [I 2022-10-10 11:10:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 2280/2304 [I 2022-10-10 11:10:37] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:20 [I 2022-10-10 11:10:37] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00294 [I 2022-10-10 11:10:37] ...trap/engines/engine.py.230: loss: 0.16840 [I 2022-10-10 11:10:37] ...optimizers/trijoint.py.151: total_norm: 13.034774 [I 2022-10-10 11:10:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 2290/2304 [I 2022-10-10 11:10:45] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:11 [I 2022-10-10 11:10:45] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00291 [I 2022-10-10 11:10:45] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-10 11:10:45] ...optimizers/trijoint.py.151: total_norm: 17.471155 [I 2022-10-10 11:10:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 2300/2304 [I 2022-10-10 11:10:54] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:03 [I 2022-10-10 11:10:54] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00014 [I 2022-10-10 11:10:54] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-10 11:10:54] ...optimizers/trijoint.py.151: total_norm: 15.791256 [I 2022-10-10 11:10:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 2304/2304 [I 2022-10-10 11:10:57] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:00 [I 2022-10-10 11:10:57] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00015 [I 2022-10-10 11:10:57] ...trap/engines/engine.py.230: loss: 0.20413 [I 2022-10-10 11:10:57] ...optimizers/trijoint.py.151: total_norm: 17.380939 [S 2022-10-10 11:10:58] ...trap/engines/engine.py.129: train_epoch.epoch: 69 [S 2022-10-10 11:10:58] ...trap/engines/engine.py.129: train_epoch.loss: 0.14208255894401328 [I 2022-10-10 11:10:59] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 11:11:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 69 [I 2022-10-10 11:11:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 11:11:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 11:11:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 11:11:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 11:11:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 11:11:04] ...trap/engines/engine.py.321: eval: epoch 69 | batch 0/510 [I 2022-10-10 11:11:04] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:12 [I 2022-10-10 11:11:04] ...trap/engines/engine.py.326: process: 4.72993 | load: 4.21010 [I 2022-10-10 11:11:10] ...trap/engines/engine.py.321: eval: epoch 69 | batch 10/510 [I 2022-10-10 11:11:10] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:33 [I 2022-10-10 11:11:10] ...trap/engines/engine.py.326: process: 0.50927 | load: 0.00018 [I 2022-10-10 11:11:17] ...trap/engines/engine.py.321: eval: epoch 69 | batch 20/510 [I 2022-10-10 11:11:17] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:55 [I 2022-10-10 11:11:17] ...trap/engines/engine.py.326: process: 0.74304 | load: 0.00019 [I 2022-10-10 11:11:24] ...trap/engines/engine.py.321: eval: epoch 69 | batch 30/510 [I 2022-10-10 11:11:24] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:26 [I 2022-10-10 11:11:24] ...trap/engines/engine.py.326: process: 0.75805 | load: 0.00018 [I 2022-10-10 11:11:31] ...trap/engines/engine.py.321: eval: epoch 69 | batch 40/510 [I 2022-10-10 11:11:31] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:05:34 [I 2022-10-10 11:11:31] ...trap/engines/engine.py.326: process: 0.68355 | load: 0.00018 [I 2022-10-10 11:11:36] ...trap/engines/engine.py.321: eval: epoch 69 | batch 50/510 [I 2022-10-10 11:11:36] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:15 [I 2022-10-10 11:11:36] ...trap/engines/engine.py.326: process: 0.53227 | load: 0.00019 [I 2022-10-10 11:11:42] ...trap/engines/engine.py.321: eval: epoch 69 | batch 60/510 [I 2022-10-10 11:11:42] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:02 [I 2022-10-10 11:11:42] ...trap/engines/engine.py.326: process: 0.53215 | load: 0.00018 [I 2022-10-10 11:11:48] ...trap/engines/engine.py.321: eval: epoch 69 | batch 70/510 [I 2022-10-10 11:11:48] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:18 [I 2022-10-10 11:11:48] ...trap/engines/engine.py.326: process: 0.57162 | load: 0.00018 [I 2022-10-10 11:11:53] ...trap/engines/engine.py.321: eval: epoch 69 | batch 80/510 [I 2022-10-10 11:11:53] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:11 [I 2022-10-10 11:11:53] ...trap/engines/engine.py.326: process: 0.59196 | load: 0.00019 [I 2022-10-10 11:11:58] ...trap/engines/engine.py.321: eval: epoch 69 | batch 90/510 [I 2022-10-10 11:11:58] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:24 [I 2022-10-10 11:11:58] ...trap/engines/engine.py.326: process: 0.45764 | load: 0.00018 [I 2022-10-10 11:12:03] ...trap/engines/engine.py.321: eval: epoch 69 | batch 100/510 [I 2022-10-10 11:12:03] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:59 [I 2022-10-10 11:12:03] ...trap/engines/engine.py.326: process: 0.30946 | load: 0.00018 [I 2022-10-10 11:12:06] ...trap/engines/engine.py.321: eval: epoch 69 | batch 110/510 [I 2022-10-10 11:12:06] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:13 [I 2022-10-10 11:12:06] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00018 [I 2022-10-10 11:12:09] ...trap/engines/engine.py.321: eval: epoch 69 | batch 120/510 [I 2022-10-10 11:12:09] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:04 [I 2022-10-10 11:12:09] ...trap/engines/engine.py.326: process: 0.31631 | load: 0.00019 [I 2022-10-10 11:12:12] ...trap/engines/engine.py.321: eval: epoch 69 | batch 130/510 [I 2022-10-10 11:12:12] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:01 [I 2022-10-10 11:12:12] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00020 [I 2022-10-10 11:12:16] ...trap/engines/engine.py.321: eval: epoch 69 | batch 140/510 [I 2022-10-10 11:12:16] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:57 [I 2022-10-10 11:12:16] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00020 [I 2022-10-10 11:12:19] ...trap/engines/engine.py.321: eval: epoch 69 | batch 150/510 [I 2022-10-10 11:12:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:54 [I 2022-10-10 11:12:19] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00018 [I 2022-10-10 11:12:22] ...trap/engines/engine.py.321: eval: epoch 69 | batch 160/510 [I 2022-10-10 11:12:22] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:51 [I 2022-10-10 11:12:22] ...trap/engines/engine.py.326: process: 0.32082 | load: 0.00019 [I 2022-10-10 11:12:25] ...trap/engines/engine.py.321: eval: epoch 69 | batch 170/510 [I 2022-10-10 11:12:25] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:47 [I 2022-10-10 11:12:25] ...trap/engines/engine.py.326: process: 0.30967 | load: 0.00019 [I 2022-10-10 11:12:28] ...trap/engines/engine.py.321: eval: epoch 69 | batch 180/510 [I 2022-10-10 11:12:28] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:45 [I 2022-10-10 11:12:28] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00019 [I 2022-10-10 11:12:31] ...trap/engines/engine.py.321: eval: epoch 69 | batch 190/510 [I 2022-10-10 11:12:31] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:41 [I 2022-10-10 11:12:31] ...trap/engines/engine.py.326: process: 0.31466 | load: 0.00018 [I 2022-10-10 11:12:35] ...trap/engines/engine.py.321: eval: epoch 69 | batch 200/510 [I 2022-10-10 11:12:35] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:38 [I 2022-10-10 11:12:35] ...trap/engines/engine.py.326: process: 0.31616 | load: 0.00018 [I 2022-10-10 11:12:38] ...trap/engines/engine.py.321: eval: epoch 69 | batch 210/510 [I 2022-10-10 11:12:38] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:35 [I 2022-10-10 11:12:38] ...trap/engines/engine.py.326: process: 0.32168 | load: 0.00019 [I 2022-10-10 11:12:41] ...trap/engines/engine.py.321: eval: epoch 69 | batch 220/510 [I 2022-10-10 11:12:41] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:32 [I 2022-10-10 11:12:41] ...trap/engines/engine.py.326: process: 0.31623 | load: 0.00019 [I 2022-10-10 11:12:44] ...trap/engines/engine.py.321: eval: epoch 69 | batch 230/510 [I 2022-10-10 11:12:44] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:29 [I 2022-10-10 11:12:44] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00019 [I 2022-10-10 11:12:47] ...trap/engines/engine.py.321: eval: epoch 69 | batch 240/510 [I 2022-10-10 11:12:47] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:25 [I 2022-10-10 11:12:47] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00019 [I 2022-10-10 11:12:51] ...trap/engines/engine.py.321: eval: epoch 69 | batch 250/510 [I 2022-10-10 11:12:51] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:22 [I 2022-10-10 11:12:51] ...trap/engines/engine.py.326: process: 0.31551 | load: 0.00023 [I 2022-10-10 11:12:54] ...trap/engines/engine.py.321: eval: epoch 69 | batch 260/510 [I 2022-10-10 11:12:54] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:19 [I 2022-10-10 11:12:54] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00019 [I 2022-10-10 11:12:57] ...trap/engines/engine.py.321: eval: epoch 69 | batch 270/510 [I 2022-10-10 11:12:57] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:16 [I 2022-10-10 11:12:57] ...trap/engines/engine.py.326: process: 0.31749 | load: 0.00023 [I 2022-10-10 11:13:00] ...trap/engines/engine.py.321: eval: epoch 69 | batch 280/510 [I 2022-10-10 11:13:00] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:13 [I 2022-10-10 11:13:00] ...trap/engines/engine.py.326: process: 0.32442 | load: 0.00029 [I 2022-10-10 11:13:03] ...trap/engines/engine.py.321: eval: epoch 69 | batch 290/510 [I 2022-10-10 11:13:03] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:09 [I 2022-10-10 11:13:03] ...trap/engines/engine.py.326: process: 0.30870 | load: 0.00020 [I 2022-10-10 11:13:06] ...trap/engines/engine.py.321: eval: epoch 69 | batch 300/510 [I 2022-10-10 11:13:06] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:06 [I 2022-10-10 11:13:06] ...trap/engines/engine.py.326: process: 0.31392 | load: 0.00019 [I 2022-10-10 11:13:10] ...trap/engines/engine.py.321: eval: epoch 69 | batch 310/510 [I 2022-10-10 11:13:10] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:03 [I 2022-10-10 11:13:10] ...trap/engines/engine.py.326: process: 0.31314 | load: 0.00023 [I 2022-10-10 11:13:13] ...trap/engines/engine.py.321: eval: epoch 69 | batch 320/510 [I 2022-10-10 11:13:13] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:01:00 [I 2022-10-10 11:13:13] ...trap/engines/engine.py.326: process: 0.31808 | load: 0.00019 [I 2022-10-10 11:13:16] ...trap/engines/engine.py.321: eval: epoch 69 | batch 330/510 [I 2022-10-10 11:13:16] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:57 [I 2022-10-10 11:13:16] ...trap/engines/engine.py.326: process: 0.32007 | load: 0.00019 [I 2022-10-10 11:13:19] ...trap/engines/engine.py.321: eval: epoch 69 | batch 340/510 [I 2022-10-10 11:13:19] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:54 [I 2022-10-10 11:13:19] ...trap/engines/engine.py.326: process: 0.32039 | load: 0.00018 [I 2022-10-10 11:13:22] ...trap/engines/engine.py.321: eval: epoch 69 | batch 350/510 [I 2022-10-10 11:13:22] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:51 [I 2022-10-10 11:13:22] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00019 [I 2022-10-10 11:13:26] ...trap/engines/engine.py.321: eval: epoch 69 | batch 360/510 [I 2022-10-10 11:13:26] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:47 [I 2022-10-10 11:13:26] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00019 [I 2022-10-10 11:13:29] ...trap/engines/engine.py.321: eval: epoch 69 | batch 370/510 [I 2022-10-10 11:13:29] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:44 [I 2022-10-10 11:13:29] ...trap/engines/engine.py.326: process: 0.31991 | load: 0.00019 [I 2022-10-10 11:13:32] ...trap/engines/engine.py.321: eval: epoch 69 | batch 380/510 [I 2022-10-10 11:13:32] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:41 [I 2022-10-10 11:13:32] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00019 [I 2022-10-10 11:13:35] ...trap/engines/engine.py.321: eval: epoch 69 | batch 390/510 [I 2022-10-10 11:13:35] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:38 [I 2022-10-10 11:13:35] ...trap/engines/engine.py.326: process: 0.32462 | load: 0.00019 [I 2022-10-10 11:13:38] ...trap/engines/engine.py.321: eval: epoch 69 | batch 400/510 [I 2022-10-10 11:13:38] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:35 [I 2022-10-10 11:13:38] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00020 [I 2022-10-10 11:13:42] ...trap/engines/engine.py.321: eval: epoch 69 | batch 410/510 [I 2022-10-10 11:13:42] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:31 [I 2022-10-10 11:13:42] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00019 [I 2022-10-10 11:13:45] ...trap/engines/engine.py.321: eval: epoch 69 | batch 420/510 [I 2022-10-10 11:13:45] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:28 [I 2022-10-10 11:13:45] ...trap/engines/engine.py.326: process: 0.31779 | load: 0.00019 [I 2022-10-10 11:13:48] ...trap/engines/engine.py.321: eval: epoch 69 | batch 430/510 [I 2022-10-10 11:13:48] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:25 [I 2022-10-10 11:13:48] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00019 [I 2022-10-10 11:13:51] ...trap/engines/engine.py.321: eval: epoch 69 | batch 440/510 [I 2022-10-10 11:13:51] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:22 [I 2022-10-10 11:13:51] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00019 [I 2022-10-10 11:13:54] ...trap/engines/engine.py.321: eval: epoch 69 | batch 450/510 [I 2022-10-10 11:13:54] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:19 [I 2022-10-10 11:13:54] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00020 [I 2022-10-10 11:13:57] ...trap/engines/engine.py.321: eval: epoch 69 | batch 460/510 [I 2022-10-10 11:13:57] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:15 [I 2022-10-10 11:13:57] ...trap/engines/engine.py.326: process: 0.31608 | load: 0.00020 [I 2022-10-10 11:14:01] ...trap/engines/engine.py.321: eval: epoch 69 | batch 470/510 [I 2022-10-10 11:14:01] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:12 [I 2022-10-10 11:14:01] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00019 [I 2022-10-10 11:14:04] ...trap/engines/engine.py.321: eval: epoch 69 | batch 480/510 [I 2022-10-10 11:14:04] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:09 [I 2022-10-10 11:14:04] ...trap/engines/engine.py.326: process: 0.31512 | load: 0.00020 [I 2022-10-10 11:14:07] ...trap/engines/engine.py.321: eval: epoch 69 | batch 490/510 [I 2022-10-10 11:14:07] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:06 [I 2022-10-10 11:14:07] ...trap/engines/engine.py.326: process: 0.32106 | load: 0.00019 [I 2022-10-10 11:14:10] ...trap/engines/engine.py.321: eval: epoch 69 | batch 500/510 [I 2022-10-10 11:14:10] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:03 [I 2022-10-10 11:14:10] ...trap/engines/engine.py.326: process: 0.31568 | load: 0.00019 [I 2022-10-10 11:14:13] ...trap/engines/engine.py.321: eval: epoch 69 | batch 510/510 [I 2022-10-10 11:14:13] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:00 [I 2022-10-10 11:14:13] ...trap/engines/engine.py.326: process: 0.31772 | load: 0.00012 [S 2022-10-10 11:14:14] ...trap/engines/engine.py.132: eval_epoch.epoch: 69 [I 2022-10-10 11:14:14] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736700 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909400 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938800 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.728400 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906500 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937800 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 11:14:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016739 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007255 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005075 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014479 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006376 [S 2022-10-10 11:14:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006765 [I 2022-10-10 11:14:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 11:14:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 11:14:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 11:14:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 11:14:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 11:14:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 11:15:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 11:15:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 11:15:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 70 [I 2022-10-10 11:15:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 11:15:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 0/2304 [I 2022-10-10 11:15:26] ...trap/engines/engine.py.228: elapsed: 0:00:21 | left: 13:39:28 [I 2022-10-10 11:15:26] ...trap/engines/engine.py.229: process: 21.34030 | load: 20.43624 [I 2022-10-10 11:15:26] ...trap/engines/engine.py.230: loss: 0.14214 [I 2022-10-10 11:15:26] ...optimizers/trijoint.py.151: total_norm: 12.457950 [I 2022-10-10 11:15:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 10/2304 [I 2022-10-10 11:15:34] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 1:56:30 [I 2022-10-10 11:15:34] ...trap/engines/engine.py.229: process: 0.84496 | load: 0.00506 [I 2022-10-10 11:15:34] ...trap/engines/engine.py.230: loss: 0.12076 [I 2022-10-10 11:15:34] ...optimizers/trijoint.py.151: total_norm: 19.349609 [I 2022-10-10 11:15:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 20/2304 [I 2022-10-10 11:15:43] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:41:18 [I 2022-10-10 11:15:43] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00529 [I 2022-10-10 11:15:43] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-10 11:15:43] ...optimizers/trijoint.py.151: total_norm: 12.389102 [I 2022-10-10 11:15:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 30/2304 [I 2022-10-10 11:15:51] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:33:00 [I 2022-10-10 11:15:51] ...trap/engines/engine.py.229: process: 0.85059 | load: 0.00392 [I 2022-10-10 11:15:51] ...trap/engines/engine.py.230: loss: 0.14729 [I 2022-10-10 11:15:51] ...optimizers/trijoint.py.151: total_norm: 9.700049 [I 2022-10-10 11:16:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 40/2304 [I 2022-10-10 11:16:00] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:12 [I 2022-10-10 11:16:00] ...trap/engines/engine.py.229: process: 0.87135 | load: 0.00436 [I 2022-10-10 11:16:00] ...trap/engines/engine.py.230: loss: 0.12486 [I 2022-10-10 11:16:00] ...optimizers/trijoint.py.151: total_norm: 12.944107 [I 2022-10-10 11:16:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 50/2304 [I 2022-10-10 11:16:08] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:50 [I 2022-10-10 11:16:08] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00345 [I 2022-10-10 11:16:08] ...trap/engines/engine.py.230: loss: 0.13249 [I 2022-10-10 11:16:08] ...optimizers/trijoint.py.151: total_norm: 8.542024 [I 2022-10-10 11:16:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 60/2304 [I 2022-10-10 11:16:17] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:39 [I 2022-10-10 11:16:17] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00328 [I 2022-10-10 11:16:17] ...trap/engines/engine.py.230: loss: 0.13419 [I 2022-10-10 11:16:17] ...optimizers/trijoint.py.151: total_norm: 13.689306 [I 2022-10-10 11:16:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 70/2304 [I 2022-10-10 11:16:25] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:28 [I 2022-10-10 11:16:25] ...trap/engines/engine.py.229: process: 0.83642 | load: 0.00326 [I 2022-10-10 11:16:25] ...trap/engines/engine.py.230: loss: 0.17830 [I 2022-10-10 11:16:25] ...optimizers/trijoint.py.151: total_norm: 27.452169 [I 2022-10-10 11:16:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 80/2304 [I 2022-10-10 11:16:34] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:22 [I 2022-10-10 11:16:34] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00292 [I 2022-10-10 11:16:34] ...trap/engines/engine.py.230: loss: 0.15655 [I 2022-10-10 11:16:34] ...optimizers/trijoint.py.151: total_norm: 15.668175 [I 2022-10-10 11:16:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 90/2304 [I 2022-10-10 11:16:42] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:13 [I 2022-10-10 11:16:42] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00366 [I 2022-10-10 11:16:42] ...trap/engines/engine.py.230: loss: 0.17110 [I 2022-10-10 11:16:42] ...optimizers/trijoint.py.151: total_norm: 13.487508 [I 2022-10-10 11:16:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 100/2304 [I 2022-10-10 11:16:51] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:14 [I 2022-10-10 11:16:51] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00324 [I 2022-10-10 11:16:51] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-10 11:16:51] ...optimizers/trijoint.py.151: total_norm: 11.898302 [I 2022-10-10 11:16:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 110/2304 [I 2022-10-10 11:16:59] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:13 [I 2022-10-10 11:16:59] ...trap/engines/engine.py.229: process: 0.86272 | load: 0.00320 [I 2022-10-10 11:16:59] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-10 11:16:59] ...optimizers/trijoint.py.151: total_norm: 15.064304 [I 2022-10-10 11:17:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 120/2304 [I 2022-10-10 11:17:08] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:54 [I 2022-10-10 11:17:08] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00335 [I 2022-10-10 11:17:08] ...trap/engines/engine.py.230: loss: 0.08728 [I 2022-10-10 11:17:08] ...optimizers/trijoint.py.151: total_norm: 11.426317 [I 2022-10-10 11:17:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 130/2304 [I 2022-10-10 11:17:16] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:44 [I 2022-10-10 11:17:16] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00323 [I 2022-10-10 11:17:16] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-10 11:17:16] ...optimizers/trijoint.py.151: total_norm: 16.737631 [I 2022-10-10 11:17:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 140/2304 [I 2022-10-10 11:17:25] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:32 [I 2022-10-10 11:17:25] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00304 [I 2022-10-10 11:17:25] ...trap/engines/engine.py.230: loss: 0.08868 [I 2022-10-10 11:17:25] ...optimizers/trijoint.py.151: total_norm: 8.136822 [I 2022-10-10 11:17:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 150/2304 [I 2022-10-10 11:17:33] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:41 [I 2022-10-10 11:17:33] ...trap/engines/engine.py.229: process: 0.86209 | load: 0.00309 [I 2022-10-10 11:17:33] ...trap/engines/engine.py.230: loss: 0.11994 [I 2022-10-10 11:17:33] ...optimizers/trijoint.py.151: total_norm: 13.596278 [I 2022-10-10 11:17:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 160/2304 [I 2022-10-10 11:17:42] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:43 [I 2022-10-10 11:17:42] ...trap/engines/engine.py.229: process: 0.85625 | load: 0.00312 [I 2022-10-10 11:17:42] ...trap/engines/engine.py.230: loss: 0.10578 [I 2022-10-10 11:17:42] ...optimizers/trijoint.py.151: total_norm: 14.204465 [I 2022-10-10 11:17:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 170/2304 [I 2022-10-10 11:17:50] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:12 [I 2022-10-10 11:17:50] ...trap/engines/engine.py.229: process: 0.85357 | load: 0.00297 [I 2022-10-10 11:17:50] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-10 11:17:50] ...optimizers/trijoint.py.151: total_norm: 21.355303 [I 2022-10-10 11:17:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 180/2304 [I 2022-10-10 11:17:59] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:07 [I 2022-10-10 11:17:59] ...trap/engines/engine.py.229: process: 0.85637 | load: 0.00301 [I 2022-10-10 11:17:59] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-10 11:17:59] ...optimizers/trijoint.py.151: total_norm: 14.904199 [I 2022-10-10 11:18:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 190/2304 [I 2022-10-10 11:18:08] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:30:05 [I 2022-10-10 11:18:08] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00295 [I 2022-10-10 11:18:08] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-10 11:18:08] ...optimizers/trijoint.py.151: total_norm: 14.919586 [I 2022-10-10 11:18:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 200/2304 [I 2022-10-10 11:18:16] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:59 [I 2022-10-10 11:18:16] ...trap/engines/engine.py.229: process: 0.85088 | load: 0.00316 [I 2022-10-10 11:18:16] ...trap/engines/engine.py.230: loss: 0.11967 [I 2022-10-10 11:18:16] ...optimizers/trijoint.py.151: total_norm: 10.033175 [I 2022-10-10 11:18:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 210/2304 [I 2022-10-10 11:18:25] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:42 [I 2022-10-10 11:18:25] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00300 [I 2022-10-10 11:18:25] ...trap/engines/engine.py.230: loss: 0.13931 [I 2022-10-10 11:18:25] ...optimizers/trijoint.py.151: total_norm: 16.974792 [I 2022-10-10 11:18:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 220/2304 [I 2022-10-10 11:18:33] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:31 [I 2022-10-10 11:18:33] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00292 [I 2022-10-10 11:18:33] ...trap/engines/engine.py.230: loss: 0.16499 [I 2022-10-10 11:18:33] ...optimizers/trijoint.py.151: total_norm: 10.481469 [I 2022-10-10 11:18:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 230/2304 [I 2022-10-10 11:18:42] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:22 [I 2022-10-10 11:18:42] ...trap/engines/engine.py.229: process: 0.86309 | load: 0.00304 [I 2022-10-10 11:18:42] ...trap/engines/engine.py.230: loss: 0.12953 [I 2022-10-10 11:18:42] ...optimizers/trijoint.py.151: total_norm: 9.541848 [I 2022-10-10 11:18:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 240/2304 [I 2022-10-10 11:18:50] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:21 [I 2022-10-10 11:18:50] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00302 [I 2022-10-10 11:18:50] ...trap/engines/engine.py.230: loss: 0.11786 [I 2022-10-10 11:18:50] ...optimizers/trijoint.py.151: total_norm: 14.739566 [I 2022-10-10 11:18:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 250/2304 [I 2022-10-10 11:18:59] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:10 [I 2022-10-10 11:18:59] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00307 [I 2022-10-10 11:18:59] ...trap/engines/engine.py.230: loss: 0.19279 [I 2022-10-10 11:18:59] ...optimizers/trijoint.py.151: total_norm: 15.441714 [I 2022-10-10 11:19:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 260/2304 [I 2022-10-10 11:19:07] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:29:03 [I 2022-10-10 11:19:07] ...trap/engines/engine.py.229: process: 0.83951 | load: 0.00304 [I 2022-10-10 11:19:07] ...trap/engines/engine.py.230: loss: 0.10648 [I 2022-10-10 11:19:07] ...optimizers/trijoint.py.151: total_norm: 8.068846 [I 2022-10-10 11:19:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 270/2304 [I 2022-10-10 11:19:16] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:29:02 [I 2022-10-10 11:19:16] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00346 [I 2022-10-10 11:19:16] ...trap/engines/engine.py.230: loss: 0.17180 [I 2022-10-10 11:19:16] ...optimizers/trijoint.py.151: total_norm: 10.386469 [I 2022-10-10 11:19:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 280/2304 [I 2022-10-10 11:19:24] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:49 [I 2022-10-10 11:19:24] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00295 [I 2022-10-10 11:19:24] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-10 11:19:24] ...optimizers/trijoint.py.151: total_norm: 13.310102 [I 2022-10-10 11:19:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 290/2304 [I 2022-10-10 11:19:33] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:33 [I 2022-10-10 11:19:33] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00307 [I 2022-10-10 11:19:33] ...trap/engines/engine.py.230: loss: 0.20459 [I 2022-10-10 11:19:33] ...optimizers/trijoint.py.151: total_norm: 21.800461 [I 2022-10-10 11:19:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 300/2304 [I 2022-10-10 11:19:42] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:26 [I 2022-10-10 11:19:42] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00323 [I 2022-10-10 11:19:42] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-10 11:19:42] ...optimizers/trijoint.py.151: total_norm: 28.702047 [I 2022-10-10 11:19:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 310/2304 [I 2022-10-10 11:19:50] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:21 [I 2022-10-10 11:19:50] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00296 [I 2022-10-10 11:19:50] ...trap/engines/engine.py.230: loss: 0.09795 [I 2022-10-10 11:19:50] ...optimizers/trijoint.py.151: total_norm: 8.446739 [I 2022-10-10 11:19:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 320/2304 [I 2022-10-10 11:19:59] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:13 [I 2022-10-10 11:19:59] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00296 [I 2022-10-10 11:19:59] ...trap/engines/engine.py.230: loss: 0.15855 [I 2022-10-10 11:19:59] ...optimizers/trijoint.py.151: total_norm: 36.952930 [I 2022-10-10 11:20:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 330/2304 [I 2022-10-10 11:20:07] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:59 [I 2022-10-10 11:20:07] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00301 [I 2022-10-10 11:20:07] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-10 11:20:07] ...optimizers/trijoint.py.151: total_norm: 8.974886 [I 2022-10-10 11:20:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 340/2304 [I 2022-10-10 11:20:16] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:51 [I 2022-10-10 11:20:16] ...trap/engines/engine.py.229: process: 0.85374 | load: 0.00319 [I 2022-10-10 11:20:16] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-10 11:20:16] ...optimizers/trijoint.py.151: total_norm: 9.837341 [I 2022-10-10 11:20:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 350/2304 [I 2022-10-10 11:20:24] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:39 [I 2022-10-10 11:20:24] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00310 [I 2022-10-10 11:20:24] ...trap/engines/engine.py.230: loss: 0.09845 [I 2022-10-10 11:20:24] ...optimizers/trijoint.py.151: total_norm: 9.593665 [I 2022-10-10 11:20:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 360/2304 [I 2022-10-10 11:20:33] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:33 [I 2022-10-10 11:20:33] ...trap/engines/engine.py.229: process: 0.85665 | load: 0.00316 [I 2022-10-10 11:20:33] ...trap/engines/engine.py.230: loss: 0.15880 [I 2022-10-10 11:20:33] ...optimizers/trijoint.py.151: total_norm: 30.661919 [I 2022-10-10 11:20:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 370/2304 [I 2022-10-10 11:20:41] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:27:36 [I 2022-10-10 11:20:41] ...trap/engines/engine.py.229: process: 0.87544 | load: 0.00302 [I 2022-10-10 11:20:41] ...trap/engines/engine.py.230: loss: 0.13467 [I 2022-10-10 11:20:41] ...optimizers/trijoint.py.151: total_norm: 25.690918 [I 2022-10-10 11:20:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 380/2304 [I 2022-10-10 11:20:50] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:03 [I 2022-10-10 11:20:50] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00316 [I 2022-10-10 11:20:50] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-10 11:20:50] ...optimizers/trijoint.py.151: total_norm: 12.471951 [I 2022-10-10 11:20:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 390/2304 [I 2022-10-10 11:20:58] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:59 [I 2022-10-10 11:20:58] ...trap/engines/engine.py.229: process: 0.84242 | load: 0.00296 [I 2022-10-10 11:20:58] ...trap/engines/engine.py.230: loss: 0.17279 [I 2022-10-10 11:20:58] ...optimizers/trijoint.py.151: total_norm: 11.103683 [I 2022-10-10 11:21:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 400/2304 [I 2022-10-10 11:21:07] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:27:15 [I 2022-10-10 11:21:07] ...trap/engines/engine.py.229: process: 0.86856 | load: 0.00294 [I 2022-10-10 11:21:07] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-10 11:21:07] ...optimizers/trijoint.py.151: total_norm: 16.609745 [I 2022-10-10 11:21:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 410/2304 [I 2022-10-10 11:21:15] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:27:01 [I 2022-10-10 11:21:15] ...trap/engines/engine.py.229: process: 0.87015 | load: 0.00301 [I 2022-10-10 11:21:15] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-10 11:21:15] ...optimizers/trijoint.py.151: total_norm: 31.990385 [I 2022-10-10 11:21:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 420/2304 [I 2022-10-10 11:21:24] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:26:58 [I 2022-10-10 11:21:24] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00291 [I 2022-10-10 11:21:24] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-10 11:21:24] ...optimizers/trijoint.py.151: total_norm: 11.003548 [I 2022-10-10 11:21:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 430/2304 [I 2022-10-10 11:21:33] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:26:25 [I 2022-10-10 11:21:33] ...trap/engines/engine.py.229: process: 0.83460 | load: 0.00306 [I 2022-10-10 11:21:33] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-10 11:21:33] ...optimizers/trijoint.py.151: total_norm: 14.444775 [I 2022-10-10 11:21:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 440/2304 [I 2022-10-10 11:21:41] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:26:26 [I 2022-10-10 11:21:41] ...trap/engines/engine.py.229: process: 0.84925 | load: 0.00307 [I 2022-10-10 11:21:41] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-10 11:21:41] ...optimizers/trijoint.py.151: total_norm: 17.480326 [I 2022-10-10 11:21:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 450/2304 [I 2022-10-10 11:21:50] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:26:16 [I 2022-10-10 11:21:50] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00290 [I 2022-10-10 11:21:50] ...trap/engines/engine.py.230: loss: 0.16111 [I 2022-10-10 11:21:50] ...optimizers/trijoint.py.151: total_norm: 19.936230 [I 2022-10-10 11:21:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 460/2304 [I 2022-10-10 11:21:58] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:26:14 [I 2022-10-10 11:21:58] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00292 [I 2022-10-10 11:21:58] ...trap/engines/engine.py.230: loss: 0.19848 [I 2022-10-10 11:21:58] ...optimizers/trijoint.py.151: total_norm: 13.527468 [I 2022-10-10 11:22:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 470/2304 [I 2022-10-10 11:22:07] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:26:06 [I 2022-10-10 11:22:07] ...trap/engines/engine.py.229: process: 0.86171 | load: 0.00292 [I 2022-10-10 11:22:07] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-10 11:22:07] ...optimizers/trijoint.py.151: total_norm: 16.526907 [I 2022-10-10 11:22:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 480/2304 [I 2022-10-10 11:22:15] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:54 [I 2022-10-10 11:22:15] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00292 [I 2022-10-10 11:22:15] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-10 11:22:15] ...optimizers/trijoint.py.151: total_norm: 16.643965 [I 2022-10-10 11:22:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 490/2304 [I 2022-10-10 11:22:24] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:25:43 [I 2022-10-10 11:22:24] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00289 [I 2022-10-10 11:22:24] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-10 11:22:24] ...optimizers/trijoint.py.151: total_norm: 33.816521 [I 2022-10-10 11:22:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 500/2304 [I 2022-10-10 11:22:32] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:25:37 [I 2022-10-10 11:22:32] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00295 [I 2022-10-10 11:22:32] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-10 11:22:32] ...optimizers/trijoint.py.151: total_norm: 14.128406 [I 2022-10-10 11:22:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 510/2304 [I 2022-10-10 11:22:41] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:25:18 [I 2022-10-10 11:22:41] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00301 [I 2022-10-10 11:22:41] ...trap/engines/engine.py.230: loss: 0.14148 [I 2022-10-10 11:22:41] ...optimizers/trijoint.py.151: total_norm: 11.100056 [I 2022-10-10 11:22:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 520/2304 [I 2022-10-10 11:22:49] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:25:27 [I 2022-10-10 11:22:49] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00294 [I 2022-10-10 11:22:49] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-10 11:22:49] ...optimizers/trijoint.py.151: total_norm: 22.270206 [I 2022-10-10 11:22:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 530/2304 [I 2022-10-10 11:22:58] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:25:15 [I 2022-10-10 11:22:58] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00293 [I 2022-10-10 11:22:58] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-10 11:22:58] ...optimizers/trijoint.py.151: total_norm: 10.510376 [I 2022-10-10 11:23:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 540/2304 [I 2022-10-10 11:23:06] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:24:56 [I 2022-10-10 11:23:06] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00294 [I 2022-10-10 11:23:06] ...trap/engines/engine.py.230: loss: 0.14674 [I 2022-10-10 11:23:06] ...optimizers/trijoint.py.151: total_norm: 19.196024 [I 2022-10-10 11:23:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 550/2304 [I 2022-10-10 11:23:15] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:24:45 [I 2022-10-10 11:23:15] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00291 [I 2022-10-10 11:23:15] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-10 11:23:15] ...optimizers/trijoint.py.151: total_norm: 9.360842 [I 2022-10-10 11:23:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 560/2304 [I 2022-10-10 11:23:23] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:39 [I 2022-10-10 11:23:23] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00291 [I 2022-10-10 11:23:23] ...trap/engines/engine.py.230: loss: 0.12800 [I 2022-10-10 11:23:23] ...optimizers/trijoint.py.151: total_norm: 21.377478 [I 2022-10-10 11:23:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 570/2304 [I 2022-10-10 11:23:32] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:24:25 [I 2022-10-10 11:23:32] ...trap/engines/engine.py.229: process: 0.85489 | load: 0.00290 [I 2022-10-10 11:23:32] ...trap/engines/engine.py.230: loss: 0.16024 [I 2022-10-10 11:23:32] ...optimizers/trijoint.py.151: total_norm: 21.129461 [I 2022-10-10 11:23:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 580/2304 [I 2022-10-10 11:23:40] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:34 [I 2022-10-10 11:23:40] ...trap/engines/engine.py.229: process: 0.87016 | load: 0.00290 [I 2022-10-10 11:23:40] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-10 11:23:40] ...optimizers/trijoint.py.151: total_norm: 20.571554 [I 2022-10-10 11:23:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 590/2304 [I 2022-10-10 11:23:49] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:24:21 [I 2022-10-10 11:23:49] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00304 [I 2022-10-10 11:23:49] ...trap/engines/engine.py.230: loss: 0.12272 [I 2022-10-10 11:23:49] ...optimizers/trijoint.py.151: total_norm: 11.402950 [I 2022-10-10 11:23:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 600/2304 [I 2022-10-10 11:23:57] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:24:12 [I 2022-10-10 11:23:57] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00299 [I 2022-10-10 11:23:57] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-10 11:23:57] ...optimizers/trijoint.py.151: total_norm: 8.146506 [I 2022-10-10 11:24:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 610/2304 [I 2022-10-10 11:24:06] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:48 [I 2022-10-10 11:24:06] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00290 [I 2022-10-10 11:24:06] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-10 11:24:06] ...optimizers/trijoint.py.151: total_norm: 64.552589 [I 2022-10-10 11:24:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 620/2304 [I 2022-10-10 11:24:14] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:50 [I 2022-10-10 11:24:14] ...trap/engines/engine.py.229: process: 0.85802 | load: 0.00337 [I 2022-10-10 11:24:14] ...trap/engines/engine.py.230: loss: 0.13544 [I 2022-10-10 11:24:14] ...optimizers/trijoint.py.151: total_norm: 78.371559 [I 2022-10-10 11:24:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 630/2304 [I 2022-10-10 11:24:23] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:47 [I 2022-10-10 11:24:23] ...trap/engines/engine.py.229: process: 0.85184 | load: 0.00290 [I 2022-10-10 11:24:23] ...trap/engines/engine.py.230: loss: 0.16204 [I 2022-10-10 11:24:23] ...optimizers/trijoint.py.151: total_norm: 13.047925 [I 2022-10-10 11:24:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 640/2304 [I 2022-10-10 11:24:31] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:38 [I 2022-10-10 11:24:31] ...trap/engines/engine.py.229: process: 0.85252 | load: 0.00291 [I 2022-10-10 11:24:31] ...trap/engines/engine.py.230: loss: 0.09462 [I 2022-10-10 11:24:31] ...optimizers/trijoint.py.151: total_norm: 8.304019 [I 2022-10-10 11:24:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 650/2304 [I 2022-10-10 11:24:40] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:25 [I 2022-10-10 11:24:40] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00292 [I 2022-10-10 11:24:40] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-10 11:24:40] ...optimizers/trijoint.py.151: total_norm: 11.644441 [I 2022-10-10 11:24:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 660/2304 [I 2022-10-10 11:24:49] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:15 [I 2022-10-10 11:24:49] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00294 [I 2022-10-10 11:24:49] ...trap/engines/engine.py.230: loss: 0.15223 [I 2022-10-10 11:24:49] ...optimizers/trijoint.py.151: total_norm: 10.355112 [I 2022-10-10 11:24:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 670/2304 [I 2022-10-10 11:24:57] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:23:02 [I 2022-10-10 11:24:57] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00292 [I 2022-10-10 11:24:57] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-10 11:24:57] ...optimizers/trijoint.py.151: total_norm: 12.959632 [I 2022-10-10 11:25:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 680/2304 [I 2022-10-10 11:25:06] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:23:03 [I 2022-10-10 11:25:06] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00296 [I 2022-10-10 11:25:06] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-10 11:25:06] ...optimizers/trijoint.py.151: total_norm: 17.245808 [I 2022-10-10 11:25:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 690/2304 [I 2022-10-10 11:25:14] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:49 [I 2022-10-10 11:25:14] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00311 [I 2022-10-10 11:25:14] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-10 11:25:14] ...optimizers/trijoint.py.151: total_norm: 21.245039 [I 2022-10-10 11:25:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 700/2304 [I 2022-10-10 11:25:23] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:47 [I 2022-10-10 11:25:23] ...trap/engines/engine.py.229: process: 0.86184 | load: 0.00296 [I 2022-10-10 11:25:23] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-10 11:25:23] ...optimizers/trijoint.py.151: total_norm: 31.586872 [I 2022-10-10 11:25:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 710/2304 [I 2022-10-10 11:25:31] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:30 [I 2022-10-10 11:25:31] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00297 [I 2022-10-10 11:25:31] ...trap/engines/engine.py.230: loss: 0.18646 [I 2022-10-10 11:25:31] ...optimizers/trijoint.py.151: total_norm: 29.813572 [I 2022-10-10 11:25:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 720/2304 [I 2022-10-10 11:25:40] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:25 [I 2022-10-10 11:25:40] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00300 [I 2022-10-10 11:25:40] ...trap/engines/engine.py.230: loss: 0.15130 [I 2022-10-10 11:25:40] ...optimizers/trijoint.py.151: total_norm: 10.204299 [I 2022-10-10 11:25:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 730/2304 [I 2022-10-10 11:25:48] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:25 [I 2022-10-10 11:25:48] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00295 [I 2022-10-10 11:25:48] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-10 11:25:48] ...optimizers/trijoint.py.151: total_norm: 13.628931 [I 2022-10-10 11:25:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 740/2304 [I 2022-10-10 11:25:57] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:22:06 [I 2022-10-10 11:25:57] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00294 [I 2022-10-10 11:25:57] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-10 11:25:57] ...optimizers/trijoint.py.151: total_norm: 17.863951 [I 2022-10-10 11:26:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 750/2304 [I 2022-10-10 11:26:05] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:59 [I 2022-10-10 11:26:05] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00305 [I 2022-10-10 11:26:05] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-10 11:26:05] ...optimizers/trijoint.py.151: total_norm: 9.862348 [I 2022-10-10 11:26:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 760/2304 [I 2022-10-10 11:26:14] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:51 [I 2022-10-10 11:26:14] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00293 [I 2022-10-10 11:26:14] ...trap/engines/engine.py.230: loss: 0.13070 [I 2022-10-10 11:26:14] ...optimizers/trijoint.py.151: total_norm: 15.652401 [I 2022-10-10 11:26:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 770/2304 [I 2022-10-10 11:26:22] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:37 [I 2022-10-10 11:26:22] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00290 [I 2022-10-10 11:26:22] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-10 11:26:22] ...optimizers/trijoint.py.151: total_norm: 15.872950 [I 2022-10-10 11:26:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 780/2304 [I 2022-10-10 11:26:31] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:32 [I 2022-10-10 11:26:31] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00290 [I 2022-10-10 11:26:31] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-10 11:26:31] ...optimizers/trijoint.py.151: total_norm: 15.097367 [I 2022-10-10 11:26:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 790/2304 [I 2022-10-10 11:26:39] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:28 [I 2022-10-10 11:26:39] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00289 [I 2022-10-10 11:26:39] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-10 11:26:39] ...optimizers/trijoint.py.151: total_norm: 12.714216 [I 2022-10-10 11:26:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 800/2304 [I 2022-10-10 11:26:48] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:16 [I 2022-10-10 11:26:48] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00294 [I 2022-10-10 11:26:48] ...trap/engines/engine.py.230: loss: 0.20401 [I 2022-10-10 11:26:48] ...optimizers/trijoint.py.151: total_norm: 22.996471 [I 2022-10-10 11:26:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 810/2304 [I 2022-10-10 11:26:56] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:19 [I 2022-10-10 11:26:56] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00299 [I 2022-10-10 11:26:56] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-10 11:26:56] ...optimizers/trijoint.py.151: total_norm: 24.088808 [I 2022-10-10 11:27:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 820/2304 [I 2022-10-10 11:27:05] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:06 [I 2022-10-10 11:27:05] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00298 [I 2022-10-10 11:27:05] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-10 11:27:05] ...optimizers/trijoint.py.151: total_norm: 18.112398 [I 2022-10-10 11:27:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 830/2304 [I 2022-10-10 11:27:13] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:56 [I 2022-10-10 11:27:13] ...trap/engines/engine.py.229: process: 0.85044 | load: 0.00303 [I 2022-10-10 11:27:13] ...trap/engines/engine.py.230: loss: 0.15443 [I 2022-10-10 11:27:13] ...optimizers/trijoint.py.151: total_norm: 15.391661 [I 2022-10-10 11:27:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 840/2304 [I 2022-10-10 11:27:22] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:42 [I 2022-10-10 11:27:22] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00294 [I 2022-10-10 11:27:22] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-10 11:27:22] ...optimizers/trijoint.py.151: total_norm: 14.142585 [I 2022-10-10 11:27:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 850/2304 [I 2022-10-10 11:27:30] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:31 [I 2022-10-10 11:27:30] ...trap/engines/engine.py.229: process: 0.84462 | load: 0.00301 [I 2022-10-10 11:27:30] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-10 11:27:30] ...optimizers/trijoint.py.151: total_norm: 7.414821 [I 2022-10-10 11:27:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 860/2304 [I 2022-10-10 11:27:39] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:33 [I 2022-10-10 11:27:39] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00302 [I 2022-10-10 11:27:39] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-10 11:27:39] ...optimizers/trijoint.py.151: total_norm: 16.639814 [I 2022-10-10 11:27:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 870/2304 [I 2022-10-10 11:27:47] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:25 [I 2022-10-10 11:27:47] ...trap/engines/engine.py.229: process: 0.86300 | load: 0.00312 [I 2022-10-10 11:27:47] ...trap/engines/engine.py.230: loss: 0.11675 [I 2022-10-10 11:27:47] ...optimizers/trijoint.py.151: total_norm: 13.000075 [I 2022-10-10 11:27:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 880/2304 [I 2022-10-10 11:27:56] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:19 [I 2022-10-10 11:27:56] ...trap/engines/engine.py.229: process: 0.85676 | load: 0.00300 [I 2022-10-10 11:27:56] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-10 11:27:56] ...optimizers/trijoint.py.151: total_norm: 13.828708 [I 2022-10-10 11:28:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 890/2304 [I 2022-10-10 11:28:05] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:09 [I 2022-10-10 11:28:05] ...trap/engines/engine.py.229: process: 0.83523 | load: 0.00312 [I 2022-10-10 11:28:05] ...trap/engines/engine.py.230: loss: 0.17444 [I 2022-10-10 11:28:05] ...optimizers/trijoint.py.151: total_norm: 27.683695 [I 2022-10-10 11:28:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 900/2304 [I 2022-10-10 11:28:13] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:20:00 [I 2022-10-10 11:28:13] ...trap/engines/engine.py.229: process: 0.86995 | load: 0.00292 [I 2022-10-10 11:28:13] ...trap/engines/engine.py.230: loss: 0.11244 [I 2022-10-10 11:28:13] ...optimizers/trijoint.py.151: total_norm: 13.653802 [I 2022-10-10 11:28:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 910/2304 [I 2022-10-10 11:28:22] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:55 [I 2022-10-10 11:28:22] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00304 [I 2022-10-10 11:28:22] ...trap/engines/engine.py.230: loss: 0.12013 [I 2022-10-10 11:28:22] ...optimizers/trijoint.py.151: total_norm: 21.160351 [I 2022-10-10 11:28:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 920/2304 [I 2022-10-10 11:28:30] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:29 [I 2022-10-10 11:28:30] ...trap/engines/engine.py.229: process: 0.83265 | load: 0.00305 [I 2022-10-10 11:28:30] ...trap/engines/engine.py.230: loss: 0.10791 [I 2022-10-10 11:28:30] ...optimizers/trijoint.py.151: total_norm: 8.664148 [I 2022-10-10 11:28:53] ...trap/engines/engine.py.224: train: epoch 70 | batch 930/2304 [I 2022-10-10 11:28:53] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:36:03 [I 2022-10-10 11:28:53] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00301 [I 2022-10-10 11:28:53] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-10 11:28:53] ...optimizers/trijoint.py.151: total_norm: 22.675257 [I 2022-10-10 11:29:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 940/2304 [I 2022-10-10 11:29:01] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:21:02 [I 2022-10-10 11:29:01] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00295 [I 2022-10-10 11:29:01] ...trap/engines/engine.py.230: loss: 0.19795 [I 2022-10-10 11:29:01] ...optimizers/trijoint.py.151: total_norm: 8.573977 [I 2022-10-10 11:29:10] ...trap/engines/engine.py.224: train: epoch 70 | batch 950/2304 [I 2022-10-10 11:29:10] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:19:18 [I 2022-10-10 11:29:10] ...trap/engines/engine.py.229: process: 0.84775 | load: 0.00296 [I 2022-10-10 11:29:10] ...trap/engines/engine.py.230: loss: 0.12036 [I 2022-10-10 11:29:10] ...optimizers/trijoint.py.151: total_norm: 27.687002 [I 2022-10-10 11:29:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 960/2304 [I 2022-10-10 11:29:18] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:19:04 [I 2022-10-10 11:29:18] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00292 [I 2022-10-10 11:29:18] ...trap/engines/engine.py.230: loss: 0.12200 [I 2022-10-10 11:29:18] ...optimizers/trijoint.py.151: total_norm: 10.650639 [I 2022-10-10 11:29:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 970/2304 [I 2022-10-10 11:29:27] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:53 [I 2022-10-10 11:29:27] ...trap/engines/engine.py.229: process: 0.84241 | load: 0.00292 [I 2022-10-10 11:29:27] ...trap/engines/engine.py.230: loss: 0.12829 [I 2022-10-10 11:29:27] ...optimizers/trijoint.py.151: total_norm: 16.343052 [I 2022-10-10 11:29:35] ...trap/engines/engine.py.224: train: epoch 70 | batch 980/2304 [I 2022-10-10 11:29:35] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:18:43 [I 2022-10-10 11:29:35] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00298 [I 2022-10-10 11:29:35] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-10 11:29:35] ...optimizers/trijoint.py.151: total_norm: 10.437270 [I 2022-10-10 11:29:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 990/2304 [I 2022-10-10 11:29:44] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:32 [I 2022-10-10 11:29:44] ...trap/engines/engine.py.229: process: 0.83626 | load: 0.00294 [I 2022-10-10 11:29:44] ...trap/engines/engine.py.230: loss: 0.11940 [I 2022-10-10 11:29:44] ...optimizers/trijoint.py.151: total_norm: 13.175750 [I 2022-10-10 11:29:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 1000/2304 [I 2022-10-10 11:29:52] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:18:25 [I 2022-10-10 11:29:52] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00297 [I 2022-10-10 11:29:52] ...trap/engines/engine.py.230: loss: 0.21684 [I 2022-10-10 11:29:52] ...optimizers/trijoint.py.151: total_norm: 34.947231 [I 2022-10-10 11:30:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 1010/2304 [I 2022-10-10 11:30:01] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:18:20 [I 2022-10-10 11:30:01] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00299 [I 2022-10-10 11:30:01] ...trap/engines/engine.py.230: loss: 0.16787 [I 2022-10-10 11:30:01] ...optimizers/trijoint.py.151: total_norm: 14.791262 [I 2022-10-10 11:30:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 1020/2304 [I 2022-10-10 11:30:09] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:18:12 [I 2022-10-10 11:30:09] ...trap/engines/engine.py.229: process: 0.86024 | load: 0.00299 [I 2022-10-10 11:30:09] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-10 11:30:09] ...optimizers/trijoint.py.151: total_norm: 12.308980 [I 2022-10-10 11:30:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 1030/2304 [I 2022-10-10 11:30:18] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:18:01 [I 2022-10-10 11:30:18] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00296 [I 2022-10-10 11:30:18] ...trap/engines/engine.py.230: loss: 0.12815 [I 2022-10-10 11:30:18] ...optimizers/trijoint.py.151: total_norm: 14.173986 [I 2022-10-10 11:30:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 1040/2304 [I 2022-10-10 11:30:26] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:17:46 [I 2022-10-10 11:30:26] ...trap/engines/engine.py.229: process: 0.83742 | load: 0.00304 [I 2022-10-10 11:30:26] ...trap/engines/engine.py.230: loss: 0.10987 [I 2022-10-10 11:30:26] ...optimizers/trijoint.py.151: total_norm: 10.386449 [I 2022-10-10 11:30:35] ...trap/engines/engine.py.224: train: epoch 70 | batch 1050/2304 [I 2022-10-10 11:30:35] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:46 [I 2022-10-10 11:30:35] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00300 [I 2022-10-10 11:30:35] ...trap/engines/engine.py.230: loss: 0.11193 [I 2022-10-10 11:30:35] ...optimizers/trijoint.py.151: total_norm: 10.901296 [I 2022-10-10 11:30:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 1060/2304 [I 2022-10-10 11:30:43] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:17:34 [I 2022-10-10 11:30:43] ...trap/engines/engine.py.229: process: 0.84798 | load: 0.00292 [I 2022-10-10 11:30:43] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-10 11:30:43] ...optimizers/trijoint.py.151: total_norm: 13.471217 [I 2022-10-10 11:30:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 1070/2304 [I 2022-10-10 11:30:52] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:17:29 [I 2022-10-10 11:30:52] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00293 [I 2022-10-10 11:30:52] ...trap/engines/engine.py.230: loss: 0.19501 [I 2022-10-10 11:30:52] ...optimizers/trijoint.py.151: total_norm: 29.236118 [I 2022-10-10 11:31:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 1080/2304 [I 2022-10-10 11:31:00] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:17:24 [I 2022-10-10 11:31:00] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00293 [I 2022-10-10 11:31:00] ...trap/engines/engine.py.230: loss: 0.14537 [I 2022-10-10 11:31:00] ...optimizers/trijoint.py.151: total_norm: 10.368611 [I 2022-10-10 11:31:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 1090/2304 [I 2022-10-10 11:31:09] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:17:12 [I 2022-10-10 11:31:09] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00294 [I 2022-10-10 11:31:09] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-10 11:31:09] ...optimizers/trijoint.py.151: total_norm: 17.169430 [I 2022-10-10 11:31:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 1100/2304 [I 2022-10-10 11:31:17] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:17:00 [I 2022-10-10 11:31:17] ...trap/engines/engine.py.229: process: 0.82849 | load: 0.00292 [I 2022-10-10 11:31:17] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-10 11:31:17] ...optimizers/trijoint.py.151: total_norm: 18.663157 [I 2022-10-10 11:31:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 1110/2304 [I 2022-10-10 11:31:26] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:48 [I 2022-10-10 11:31:26] ...trap/engines/engine.py.229: process: 0.83013 | load: 0.00293 [I 2022-10-10 11:31:26] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-10 11:31:26] ...optimizers/trijoint.py.151: total_norm: 14.815477 [I 2022-10-10 11:31:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 1120/2304 [I 2022-10-10 11:31:34] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:16:51 [I 2022-10-10 11:31:34] ...trap/engines/engine.py.229: process: 0.87002 | load: 0.00290 [I 2022-10-10 11:31:34] ...trap/engines/engine.py.230: loss: 0.15293 [I 2022-10-10 11:31:34] ...optimizers/trijoint.py.151: total_norm: 11.951591 [I 2022-10-10 11:31:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 1130/2304 [I 2022-10-10 11:31:43] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:16:37 [I 2022-10-10 11:31:43] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00290 [I 2022-10-10 11:31:43] ...trap/engines/engine.py.230: loss: 0.17104 [I 2022-10-10 11:31:43] ...optimizers/trijoint.py.151: total_norm: 18.671701 [I 2022-10-10 11:31:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1140/2304 [I 2022-10-10 11:31:51] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:31 [I 2022-10-10 11:31:51] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00294 [I 2022-10-10 11:31:51] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-10 11:31:51] ...optimizers/trijoint.py.151: total_norm: 26.188318 [I 2022-10-10 11:32:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 1150/2304 [I 2022-10-10 11:32:00] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:16:22 [I 2022-10-10 11:32:00] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00294 [I 2022-10-10 11:32:00] ...trap/engines/engine.py.230: loss: 0.12293 [I 2022-10-10 11:32:00] ...optimizers/trijoint.py.151: total_norm: 18.828838 [I 2022-10-10 11:32:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1160/2304 [I 2022-10-10 11:32:08] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:16:15 [I 2022-10-10 11:32:08] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00295 [I 2022-10-10 11:32:08] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-10 11:32:08] ...optimizers/trijoint.py.151: total_norm: 8.210861 [I 2022-10-10 11:32:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 1170/2304 [I 2022-10-10 11:32:17] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:16:03 [I 2022-10-10 11:32:17] ...trap/engines/engine.py.229: process: 0.85776 | load: 0.00296 [I 2022-10-10 11:32:17] ...trap/engines/engine.py.230: loss: 0.13699 [I 2022-10-10 11:32:17] ...optimizers/trijoint.py.151: total_norm: 28.624226 [I 2022-10-10 11:32:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 1180/2304 [I 2022-10-10 11:32:26] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:59 [I 2022-10-10 11:32:26] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00293 [I 2022-10-10 11:32:26] ...trap/engines/engine.py.230: loss: 0.09631 [I 2022-10-10 11:32:26] ...optimizers/trijoint.py.151: total_norm: 9.993597 [I 2022-10-10 11:32:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 1190/2304 [I 2022-10-10 11:32:34] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:47 [I 2022-10-10 11:32:34] ...trap/engines/engine.py.229: process: 0.86522 | load: 0.00303 [I 2022-10-10 11:32:34] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-10 11:32:34] ...optimizers/trijoint.py.151: total_norm: 8.227165 [I 2022-10-10 11:32:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 1200/2304 [I 2022-10-10 11:32:43] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:38 [I 2022-10-10 11:32:43] ...trap/engines/engine.py.229: process: 0.83680 | load: 0.00290 [I 2022-10-10 11:32:43] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-10 11:32:43] ...optimizers/trijoint.py.151: total_norm: 16.204447 [I 2022-10-10 11:32:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1210/2304 [I 2022-10-10 11:32:51] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:28 [I 2022-10-10 11:32:51] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00292 [I 2022-10-10 11:32:51] ...trap/engines/engine.py.230: loss: 0.15615 [I 2022-10-10 11:32:51] ...optimizers/trijoint.py.151: total_norm: 15.065078 [I 2022-10-10 11:33:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 1220/2304 [I 2022-10-10 11:33:00] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:19 [I 2022-10-10 11:33:00] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00292 [I 2022-10-10 11:33:00] ...trap/engines/engine.py.230: loss: 0.14098 [I 2022-10-10 11:33:00] ...optimizers/trijoint.py.151: total_norm: 12.147581 [I 2022-10-10 11:33:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1230/2304 [I 2022-10-10 11:33:08] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:15 [I 2022-10-10 11:33:08] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00291 [I 2022-10-10 11:33:08] ...trap/engines/engine.py.230: loss: 0.09576 [I 2022-10-10 11:33:08] ...optimizers/trijoint.py.151: total_norm: 7.460824 [I 2022-10-10 11:33:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 1240/2304 [I 2022-10-10 11:33:17] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:02 [I 2022-10-10 11:33:17] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00299 [I 2022-10-10 11:33:17] ...trap/engines/engine.py.230: loss: 0.11794 [I 2022-10-10 11:33:17] ...optimizers/trijoint.py.151: total_norm: 8.529622 [I 2022-10-10 11:33:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1250/2304 [I 2022-10-10 11:33:25] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:57 [I 2022-10-10 11:33:25] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00294 [I 2022-10-10 11:33:25] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-10 11:33:25] ...optimizers/trijoint.py.151: total_norm: 10.616366 [I 2022-10-10 11:33:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 1260/2304 [I 2022-10-10 11:33:34] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:47 [I 2022-10-10 11:33:34] ...trap/engines/engine.py.229: process: 0.85017 | load: 0.00292 [I 2022-10-10 11:33:34] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-10 11:33:34] ...optimizers/trijoint.py.151: total_norm: 35.712337 [I 2022-10-10 11:33:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1270/2304 [I 2022-10-10 11:33:42] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:14:37 [I 2022-10-10 11:33:42] ...trap/engines/engine.py.229: process: 0.85881 | load: 0.00305 [I 2022-10-10 11:33:42] ...trap/engines/engine.py.230: loss: 0.19731 [I 2022-10-10 11:33:42] ...optimizers/trijoint.py.151: total_norm: 16.875793 [I 2022-10-10 11:33:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1280/2304 [I 2022-10-10 11:33:51] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:27 [I 2022-10-10 11:33:51] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00300 [I 2022-10-10 11:33:51] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-10 11:33:51] ...optimizers/trijoint.py.151: total_norm: 39.216171 [I 2022-10-10 11:33:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1290/2304 [I 2022-10-10 11:33:59] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:20 [I 2022-10-10 11:33:59] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00300 [I 2022-10-10 11:33:59] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-10 11:33:59] ...optimizers/trijoint.py.151: total_norm: 51.104057 [I 2022-10-10 11:34:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1300/2304 [I 2022-10-10 11:34:08] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:14:13 [I 2022-10-10 11:34:08] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00299 [I 2022-10-10 11:34:08] ...trap/engines/engine.py.230: loss: 0.12106 [I 2022-10-10 11:34:08] ...optimizers/trijoint.py.151: total_norm: 20.340799 [I 2022-10-10 11:34:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1310/2304 [I 2022-10-10 11:34:16] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:14:05 [I 2022-10-10 11:34:16] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00301 [I 2022-10-10 11:34:16] ...trap/engines/engine.py.230: loss: 0.14322 [I 2022-10-10 11:34:16] ...optimizers/trijoint.py.151: total_norm: 16.076048 [I 2022-10-10 11:34:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1320/2304 [I 2022-10-10 11:34:25] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:14:03 [I 2022-10-10 11:34:25] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00330 [I 2022-10-10 11:34:25] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-10 11:34:25] ...optimizers/trijoint.py.151: total_norm: 35.259686 [I 2022-10-10 11:34:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1330/2304 [I 2022-10-10 11:34:33] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:50 [I 2022-10-10 11:34:33] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00295 [I 2022-10-10 11:34:33] ...trap/engines/engine.py.230: loss: 0.12742 [I 2022-10-10 11:34:33] ...optimizers/trijoint.py.151: total_norm: 14.841015 [I 2022-10-10 11:34:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1340/2304 [I 2022-10-10 11:34:42] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:43 [I 2022-10-10 11:34:42] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00289 [I 2022-10-10 11:34:42] ...trap/engines/engine.py.230: loss: 0.24475 [I 2022-10-10 11:34:42] ...optimizers/trijoint.py.151: total_norm: 21.821400 [I 2022-10-10 11:34:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1350/2304 [I 2022-10-10 11:34:51] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:34 [I 2022-10-10 11:34:51] ...trap/engines/engine.py.229: process: 0.87121 | load: 0.00292 [I 2022-10-10 11:34:51] ...trap/engines/engine.py.230: loss: 0.13024 [I 2022-10-10 11:34:51] ...optimizers/trijoint.py.151: total_norm: 114.472168 [I 2022-10-10 11:34:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1360/2304 [I 2022-10-10 11:34:59] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:21 [I 2022-10-10 11:34:59] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00291 [I 2022-10-10 11:34:59] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-10 11:34:59] ...optimizers/trijoint.py.151: total_norm: 7.779120 [I 2022-10-10 11:35:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1370/2304 [I 2022-10-10 11:35:08] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:21 [I 2022-10-10 11:35:08] ...trap/engines/engine.py.229: process: 0.88147 | load: 0.00292 [I 2022-10-10 11:35:08] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-10 11:35:08] ...optimizers/trijoint.py.151: total_norm: 13.524652 [I 2022-10-10 11:35:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1380/2304 [I 2022-10-10 11:35:16] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:13:12 [I 2022-10-10 11:35:16] ...trap/engines/engine.py.229: process: 0.84876 | load: 0.00302 [I 2022-10-10 11:35:16] ...trap/engines/engine.py.230: loss: 0.25463 [I 2022-10-10 11:35:16] ...optimizers/trijoint.py.151: total_norm: 21.687462 [I 2022-10-10 11:35:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1390/2304 [I 2022-10-10 11:35:25] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:13:03 [I 2022-10-10 11:35:25] ...trap/engines/engine.py.229: process: 0.84051 | load: 0.00318 [I 2022-10-10 11:35:25] ...trap/engines/engine.py.230: loss: 0.14028 [I 2022-10-10 11:35:25] ...optimizers/trijoint.py.151: total_norm: 14.621841 [I 2022-10-10 11:35:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1400/2304 [I 2022-10-10 11:35:33] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:57 [I 2022-10-10 11:35:33] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00307 [I 2022-10-10 11:35:33] ...trap/engines/engine.py.230: loss: 0.08482 [I 2022-10-10 11:35:33] ...optimizers/trijoint.py.151: total_norm: 12.420054 [I 2022-10-10 11:35:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1410/2304 [I 2022-10-10 11:35:42] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:40 [I 2022-10-10 11:35:42] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00292 [I 2022-10-10 11:35:42] ...trap/engines/engine.py.230: loss: 0.17529 [I 2022-10-10 11:35:42] ...optimizers/trijoint.py.151: total_norm: 16.048798 [I 2022-10-10 11:35:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1420/2304 [I 2022-10-10 11:35:51] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:30 [I 2022-10-10 11:35:51] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00294 [I 2022-10-10 11:35:51] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-10 11:35:51] ...optimizers/trijoint.py.151: total_norm: 14.605422 [I 2022-10-10 11:35:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1430/2304 [I 2022-10-10 11:35:59] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:21 [I 2022-10-10 11:35:59] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00292 [I 2022-10-10 11:35:59] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-10 11:35:59] ...optimizers/trijoint.py.151: total_norm: 21.998133 [I 2022-10-10 11:36:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1440/2304 [I 2022-10-10 11:36:08] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:15 [I 2022-10-10 11:36:08] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00297 [I 2022-10-10 11:36:08] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-10 11:36:08] ...optimizers/trijoint.py.151: total_norm: 11.502071 [I 2022-10-10 11:36:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1450/2304 [I 2022-10-10 11:36:16] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:12:08 [I 2022-10-10 11:36:16] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00290 [I 2022-10-10 11:36:16] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-10 11:36:16] ...optimizers/trijoint.py.151: total_norm: 12.548606 [I 2022-10-10 11:36:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1460/2304 [I 2022-10-10 11:36:25] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:58 [I 2022-10-10 11:36:25] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00295 [I 2022-10-10 11:36:25] ...trap/engines/engine.py.230: loss: 0.10296 [I 2022-10-10 11:36:25] ...optimizers/trijoint.py.151: total_norm: 9.762397 [I 2022-10-10 11:36:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1470/2304 [I 2022-10-10 11:36:33] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:49 [I 2022-10-10 11:36:33] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00297 [I 2022-10-10 11:36:33] ...trap/engines/engine.py.230: loss: 0.15959 [I 2022-10-10 11:36:33] ...optimizers/trijoint.py.151: total_norm: 20.563152 [I 2022-10-10 11:36:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1480/2304 [I 2022-10-10 11:36:42] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:40 [I 2022-10-10 11:36:42] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00293 [I 2022-10-10 11:36:42] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-10 11:36:42] ...optimizers/trijoint.py.151: total_norm: 17.181934 [I 2022-10-10 11:36:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 1490/2304 [I 2022-10-10 11:36:50] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:36 [I 2022-10-10 11:36:50] ...trap/engines/engine.py.229: process: 0.87025 | load: 0.00290 [I 2022-10-10 11:36:50] ...trap/engines/engine.py.230: loss: 0.09549 [I 2022-10-10 11:36:50] ...optimizers/trijoint.py.151: total_norm: 9.753160 [I 2022-10-10 11:36:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1500/2304 [I 2022-10-10 11:36:59] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:23 [I 2022-10-10 11:36:59] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00295 [I 2022-10-10 11:36:59] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-10 11:36:59] ...optimizers/trijoint.py.151: total_norm: 7.714453 [I 2022-10-10 11:37:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 1510/2304 [I 2022-10-10 11:37:07] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:16 [I 2022-10-10 11:37:07] ...trap/engines/engine.py.229: process: 0.83599 | load: 0.00299 [I 2022-10-10 11:37:07] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-10 11:37:07] ...optimizers/trijoint.py.151: total_norm: 16.005455 [I 2022-10-10 11:37:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1520/2304 [I 2022-10-10 11:37:16] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:11:10 [I 2022-10-10 11:37:16] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00303 [I 2022-10-10 11:37:16] ...trap/engines/engine.py.230: loss: 0.24216 [I 2022-10-10 11:37:16] ...optimizers/trijoint.py.151: total_norm: 31.070045 [I 2022-10-10 11:37:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 1530/2304 [I 2022-10-10 11:37:24] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:11:01 [I 2022-10-10 11:37:24] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00304 [I 2022-10-10 11:37:24] ...trap/engines/engine.py.230: loss: 0.17977 [I 2022-10-10 11:37:24] ...optimizers/trijoint.py.151: total_norm: 21.382715 [I 2022-10-10 11:37:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1540/2304 [I 2022-10-10 11:37:33] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:52 [I 2022-10-10 11:37:33] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00304 [I 2022-10-10 11:37:33] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-10 11:37:33] ...optimizers/trijoint.py.151: total_norm: 9.928332 [I 2022-10-10 11:37:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 1550/2304 [I 2022-10-10 11:37:41] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:38 [I 2022-10-10 11:37:41] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00297 [I 2022-10-10 11:37:41] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-10 11:37:41] ...optimizers/trijoint.py.151: total_norm: 9.132348 [I 2022-10-10 11:37:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 1560/2304 [I 2022-10-10 11:37:50] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:33 [I 2022-10-10 11:37:50] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00294 [I 2022-10-10 11:37:50] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-10 11:37:50] ...optimizers/trijoint.py.151: total_norm: 17.793512 [I 2022-10-10 11:37:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1570/2304 [I 2022-10-10 11:37:59] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:26 [I 2022-10-10 11:37:59] ...trap/engines/engine.py.229: process: 0.86294 | load: 0.00292 [I 2022-10-10 11:37:59] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-10 11:37:59] ...optimizers/trijoint.py.151: total_norm: 15.692591 [I 2022-10-10 11:38:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 1580/2304 [I 2022-10-10 11:38:07] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:10:14 [I 2022-10-10 11:38:07] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00299 [I 2022-10-10 11:38:07] ...trap/engines/engine.py.230: loss: 0.09933 [I 2022-10-10 11:38:07] ...optimizers/trijoint.py.151: total_norm: 8.918642 [I 2022-10-10 11:38:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1590/2304 [I 2022-10-10 11:38:16] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:10:10 [I 2022-10-10 11:38:16] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00294 [I 2022-10-10 11:38:16] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-10 11:38:16] ...optimizers/trijoint.py.151: total_norm: 18.239330 [I 2022-10-10 11:38:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 1600/2304 [I 2022-10-10 11:38:24] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:53 [I 2022-10-10 11:38:24] ...trap/engines/engine.py.229: process: 0.84456 | load: 0.00294 [I 2022-10-10 11:38:24] ...trap/engines/engine.py.230: loss: 0.18934 [I 2022-10-10 11:38:24] ...optimizers/trijoint.py.151: total_norm: 25.829433 [I 2022-10-10 11:38:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1610/2304 [I 2022-10-10 11:38:33] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:47 [I 2022-10-10 11:38:33] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00296 [I 2022-10-10 11:38:33] ...trap/engines/engine.py.230: loss: 0.12000 [I 2022-10-10 11:38:33] ...optimizers/trijoint.py.151: total_norm: 9.265345 [I 2022-10-10 11:38:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 1620/2304 [I 2022-10-10 11:38:41] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:42 [I 2022-10-10 11:38:41] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00292 [I 2022-10-10 11:38:41] ...trap/engines/engine.py.230: loss: 0.20807 [I 2022-10-10 11:38:41] ...optimizers/trijoint.py.151: total_norm: 10.731815 [I 2022-10-10 11:38:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 1630/2304 [I 2022-10-10 11:38:50] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:33 [I 2022-10-10 11:38:50] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00293 [I 2022-10-10 11:38:50] ...trap/engines/engine.py.230: loss: 0.09326 [I 2022-10-10 11:38:50] ...optimizers/trijoint.py.151: total_norm: 18.504894 [I 2022-10-10 11:38:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 1640/2304 [I 2022-10-10 11:38:58] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:24 [I 2022-10-10 11:38:58] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00300 [I 2022-10-10 11:38:58] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-10 11:38:58] ...optimizers/trijoint.py.151: total_norm: 11.292058 [I 2022-10-10 11:39:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 1650/2304 [I 2022-10-10 11:39:07] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:09:16 [I 2022-10-10 11:39:07] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00294 [I 2022-10-10 11:39:07] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-10 11:39:07] ...optimizers/trijoint.py.151: total_norm: 15.567398 [I 2022-10-10 11:39:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 1660/2304 [I 2022-10-10 11:39:15] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:09:07 [I 2022-10-10 11:39:15] ...trap/engines/engine.py.229: process: 0.86505 | load: 0.00292 [I 2022-10-10 11:39:15] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-10 11:39:15] ...optimizers/trijoint.py.151: total_norm: 27.251383 [I 2022-10-10 11:39:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 1670/2304 [I 2022-10-10 11:39:24] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:58 [I 2022-10-10 11:39:24] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00292 [I 2022-10-10 11:39:24] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-10 11:39:24] ...optimizers/trijoint.py.151: total_norm: 20.046709 [I 2022-10-10 11:39:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 1680/2304 [I 2022-10-10 11:39:32] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:48 [I 2022-10-10 11:39:32] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00289 [I 2022-10-10 11:39:32] ...trap/engines/engine.py.230: loss: 0.10420 [I 2022-10-10 11:39:32] ...optimizers/trijoint.py.151: total_norm: 8.476399 [I 2022-10-10 11:39:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 1690/2304 [I 2022-10-10 11:39:41] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:42 [I 2022-10-10 11:39:41] ...trap/engines/engine.py.229: process: 0.83936 | load: 0.00296 [I 2022-10-10 11:39:41] ...trap/engines/engine.py.230: loss: 0.10386 [I 2022-10-10 11:39:41] ...optimizers/trijoint.py.151: total_norm: 8.978713 [I 2022-10-10 11:39:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 1700/2304 [I 2022-10-10 11:39:49] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:30 [I 2022-10-10 11:39:49] ...trap/engines/engine.py.229: process: 0.83580 | load: 0.00292 [I 2022-10-10 11:39:49] ...trap/engines/engine.py.230: loss: 0.15020 [I 2022-10-10 11:39:49] ...optimizers/trijoint.py.151: total_norm: 9.614551 [I 2022-10-10 11:39:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 1710/2304 [I 2022-10-10 11:39:58] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:20 [I 2022-10-10 11:39:58] ...trap/engines/engine.py.229: process: 0.83906 | load: 0.00292 [I 2022-10-10 11:39:58] ...trap/engines/engine.py.230: loss: 0.09289 [I 2022-10-10 11:39:58] ...optimizers/trijoint.py.151: total_norm: 9.497607 [I 2022-10-10 11:40:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 1720/2304 [I 2022-10-10 11:40:06] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:08:14 [I 2022-10-10 11:40:06] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00292 [I 2022-10-10 11:40:06] ...trap/engines/engine.py.230: loss: 0.17388 [I 2022-10-10 11:40:06] ...optimizers/trijoint.py.151: total_norm: 12.883080 [I 2022-10-10 11:40:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 1730/2304 [I 2022-10-10 11:40:15] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:08:06 [I 2022-10-10 11:40:15] ...trap/engines/engine.py.229: process: 0.85928 | load: 0.00290 [I 2022-10-10 11:40:15] ...trap/engines/engine.py.230: loss: 0.14050 [I 2022-10-10 11:40:15] ...optimizers/trijoint.py.151: total_norm: 9.829483 [I 2022-10-10 11:40:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 1740/2304 [I 2022-10-10 11:40:23] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:57 [I 2022-10-10 11:40:23] ...trap/engines/engine.py.229: process: 0.84051 | load: 0.00295 [I 2022-10-10 11:40:23] ...trap/engines/engine.py.230: loss: 0.27453 [I 2022-10-10 11:40:23] ...optimizers/trijoint.py.151: total_norm: 22.986645 [I 2022-10-10 11:40:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 1750/2304 [I 2022-10-10 11:40:32] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:50 [I 2022-10-10 11:40:32] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00296 [I 2022-10-10 11:40:32] ...trap/engines/engine.py.230: loss: 0.25234 [I 2022-10-10 11:40:32] ...optimizers/trijoint.py.151: total_norm: 92.527359 [I 2022-10-10 11:40:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1760/2304 [I 2022-10-10 11:40:40] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:41 [I 2022-10-10 11:40:40] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00293 [I 2022-10-10 11:40:40] ...trap/engines/engine.py.230: loss: 0.16544 [I 2022-10-10 11:40:40] ...optimizers/trijoint.py.151: total_norm: 28.167921 [I 2022-10-10 11:40:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 1770/2304 [I 2022-10-10 11:40:49] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:31 [I 2022-10-10 11:40:49] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00291 [I 2022-10-10 11:40:49] ...trap/engines/engine.py.230: loss: 0.10771 [I 2022-10-10 11:40:49] ...optimizers/trijoint.py.151: total_norm: 11.903861 [I 2022-10-10 11:40:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1780/2304 [I 2022-10-10 11:40:57] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:25 [I 2022-10-10 11:40:57] ...trap/engines/engine.py.229: process: 0.83357 | load: 0.00293 [I 2022-10-10 11:40:57] ...trap/engines/engine.py.230: loss: 0.16853 [I 2022-10-10 11:40:57] ...optimizers/trijoint.py.151: total_norm: 11.520087 [I 2022-10-10 11:41:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 1790/2304 [I 2022-10-10 11:41:06] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:07:17 [I 2022-10-10 11:41:06] ...trap/engines/engine.py.229: process: 0.84743 | load: 0.00291 [I 2022-10-10 11:41:06] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-10 11:41:06] ...optimizers/trijoint.py.151: total_norm: 12.413317 [I 2022-10-10 11:41:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1800/2304 [I 2022-10-10 11:41:14] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:05 [I 2022-10-10 11:41:14] ...trap/engines/engine.py.229: process: 0.84969 | load: 0.00289 [I 2022-10-10 11:41:14] ...trap/engines/engine.py.230: loss: 0.11458 [I 2022-10-10 11:41:14] ...optimizers/trijoint.py.151: total_norm: 12.514390 [I 2022-10-10 11:41:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 1810/2304 [I 2022-10-10 11:41:23] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:58 [I 2022-10-10 11:41:23] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00299 [I 2022-10-10 11:41:23] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-10 11:41:23] ...optimizers/trijoint.py.151: total_norm: 14.911017 [I 2022-10-10 11:41:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1820/2304 [I 2022-10-10 11:41:31] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:51 [I 2022-10-10 11:41:31] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00291 [I 2022-10-10 11:41:31] ...trap/engines/engine.py.230: loss: 0.23734 [I 2022-10-10 11:41:31] ...optimizers/trijoint.py.151: total_norm: 17.063862 [I 2022-10-10 11:41:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1830/2304 [I 2022-10-10 11:41:40] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:42 [I 2022-10-10 11:41:40] ...trap/engines/engine.py.229: process: 0.83551 | load: 0.00294 [I 2022-10-10 11:41:40] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-10 11:41:40] ...optimizers/trijoint.py.151: total_norm: 11.921994 [I 2022-10-10 11:41:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 1840/2304 [I 2022-10-10 11:41:48] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:34 [I 2022-10-10 11:41:48] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00291 [I 2022-10-10 11:41:48] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-10 11:41:48] ...optimizers/trijoint.py.151: total_norm: 23.277872 [I 2022-10-10 11:41:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1850/2304 [I 2022-10-10 11:41:57] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:25 [I 2022-10-10 11:41:57] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00292 [I 2022-10-10 11:41:57] ...trap/engines/engine.py.230: loss: 0.17025 [I 2022-10-10 11:41:57] ...optimizers/trijoint.py.151: total_norm: 16.343876 [I 2022-10-10 11:42:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 1860/2304 [I 2022-10-10 11:42:05] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:18 [I 2022-10-10 11:42:05] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00292 [I 2022-10-10 11:42:05] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-10 11:42:05] ...optimizers/trijoint.py.151: total_norm: 9.660499 [I 2022-10-10 11:42:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1870/2304 [I 2022-10-10 11:42:14] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:07 [I 2022-10-10 11:42:14] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00291 [I 2022-10-10 11:42:14] ...trap/engines/engine.py.230: loss: 0.09079 [I 2022-10-10 11:42:14] ...optimizers/trijoint.py.151: total_norm: 7.804233 [I 2022-10-10 11:42:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 1880/2304 [I 2022-10-10 11:42:22] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:58 [I 2022-10-10 11:42:22] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00292 [I 2022-10-10 11:42:22] ...trap/engines/engine.py.230: loss: 0.10862 [I 2022-10-10 11:42:22] ...optimizers/trijoint.py.151: total_norm: 9.041753 [I 2022-10-10 11:42:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1890/2304 [I 2022-10-10 11:42:31] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:54 [I 2022-10-10 11:42:31] ...trap/engines/engine.py.229: process: 0.85172 | load: 0.00301 [I 2022-10-10 11:42:31] ...trap/engines/engine.py.230: loss: 0.13178 [I 2022-10-10 11:42:31] ...optimizers/trijoint.py.151: total_norm: 10.804498 [I 2022-10-10 11:42:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 1900/2304 [I 2022-10-10 11:42:39] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:42 [I 2022-10-10 11:42:39] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00299 [I 2022-10-10 11:42:39] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-10 11:42:39] ...optimizers/trijoint.py.151: total_norm: 37.890648 [I 2022-10-10 11:42:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 1910/2304 [I 2022-10-10 11:42:48] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:34 [I 2022-10-10 11:42:48] ...trap/engines/engine.py.229: process: 0.84310 | load: 0.00295 [I 2022-10-10 11:42:48] ...trap/engines/engine.py.230: loss: 0.16373 [I 2022-10-10 11:42:48] ...optimizers/trijoint.py.151: total_norm: 36.822151 [I 2022-10-10 11:42:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1920/2304 [I 2022-10-10 11:42:57] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:28 [I 2022-10-10 11:42:57] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00319 [I 2022-10-10 11:42:57] ...trap/engines/engine.py.230: loss: 0.16770 [I 2022-10-10 11:42:57] ...optimizers/trijoint.py.151: total_norm: 16.126966 [I 2022-10-10 11:43:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 1930/2304 [I 2022-10-10 11:43:05] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:19 [I 2022-10-10 11:43:05] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00316 [I 2022-10-10 11:43:05] ...trap/engines/engine.py.230: loss: 0.11258 [I 2022-10-10 11:43:05] ...optimizers/trijoint.py.151: total_norm: 9.448228 [I 2022-10-10 11:43:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1940/2304 [I 2022-10-10 11:43:14] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:11 [I 2022-10-10 11:43:14] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00326 [I 2022-10-10 11:43:14] ...trap/engines/engine.py.230: loss: 0.19224 [I 2022-10-10 11:43:14] ...optimizers/trijoint.py.151: total_norm: 13.490568 [I 2022-10-10 11:43:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 1950/2304 [I 2022-10-10 11:43:22] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:05:00 [I 2022-10-10 11:43:22] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00299 [I 2022-10-10 11:43:22] ...trap/engines/engine.py.230: loss: 0.21797 [I 2022-10-10 11:43:22] ...optimizers/trijoint.py.151: total_norm: 24.061440 [I 2022-10-10 11:43:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1960/2304 [I 2022-10-10 11:43:31] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:52 [I 2022-10-10 11:43:31] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00293 [I 2022-10-10 11:43:31] ...trap/engines/engine.py.230: loss: 0.10730 [I 2022-10-10 11:43:31] ...optimizers/trijoint.py.151: total_norm: 9.791522 [I 2022-10-10 11:43:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 1970/2304 [I 2022-10-10 11:43:39] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:42 [I 2022-10-10 11:43:39] ...trap/engines/engine.py.229: process: 0.83454 | load: 0.00293 [I 2022-10-10 11:43:39] ...trap/engines/engine.py.230: loss: 0.16648 [I 2022-10-10 11:43:39] ...optimizers/trijoint.py.151: total_norm: 7.448329 [I 2022-10-10 11:43:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 1980/2304 [I 2022-10-10 11:43:48] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:35 [I 2022-10-10 11:43:48] ...trap/engines/engine.py.229: process: 0.86601 | load: 0.00296 [I 2022-10-10 11:43:48] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-10 11:43:48] ...optimizers/trijoint.py.151: total_norm: 14.952209 [I 2022-10-10 11:43:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 1990/2304 [I 2022-10-10 11:43:56] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:28 [I 2022-10-10 11:43:56] ...trap/engines/engine.py.229: process: 0.87008 | load: 0.00302 [I 2022-10-10 11:43:56] ...trap/engines/engine.py.230: loss: 0.17990 [I 2022-10-10 11:43:56] ...optimizers/trijoint.py.151: total_norm: 39.061939 [I 2022-10-10 11:44:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 2000/2304 [I 2022-10-10 11:44:05] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:21 [I 2022-10-10 11:44:05] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00297 [I 2022-10-10 11:44:05] ...trap/engines/engine.py.230: loss: 0.14095 [I 2022-10-10 11:44:05] ...optimizers/trijoint.py.151: total_norm: 14.271951 [I 2022-10-10 11:44:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2010/2304 [I 2022-10-10 11:44:13] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:04:08 [I 2022-10-10 11:44:13] ...trap/engines/engine.py.229: process: 0.83008 | load: 0.00296 [I 2022-10-10 11:44:13] ...trap/engines/engine.py.230: loss: 0.10889 [I 2022-10-10 11:44:13] ...optimizers/trijoint.py.151: total_norm: 11.549610 [I 2022-10-10 11:44:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 2020/2304 [I 2022-10-10 11:44:22] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:02 [I 2022-10-10 11:44:22] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00298 [I 2022-10-10 11:44:22] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-10 11:44:22] ...optimizers/trijoint.py.151: total_norm: 22.149258 [I 2022-10-10 11:44:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 2030/2304 [I 2022-10-10 11:44:30] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:53 [I 2022-10-10 11:44:30] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00296 [I 2022-10-10 11:44:30] ...trap/engines/engine.py.230: loss: 0.11585 [I 2022-10-10 11:44:30] ...optimizers/trijoint.py.151: total_norm: 17.997168 [I 2022-10-10 11:44:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 2040/2304 [I 2022-10-10 11:44:39] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:44 [I 2022-10-10 11:44:39] ...trap/engines/engine.py.229: process: 0.83818 | load: 0.00296 [I 2022-10-10 11:44:39] ...trap/engines/engine.py.230: loss: 0.11549 [I 2022-10-10 11:44:39] ...optimizers/trijoint.py.151: total_norm: 12.466887 [I 2022-10-10 11:44:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 2050/2304 [I 2022-10-10 11:44:48] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:35 [I 2022-10-10 11:44:48] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00298 [I 2022-10-10 11:44:48] ...trap/engines/engine.py.230: loss: 0.19452 [I 2022-10-10 11:44:48] ...optimizers/trijoint.py.151: total_norm: 13.689995 [I 2022-10-10 11:44:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 2060/2304 [I 2022-10-10 11:44:56] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:27 [I 2022-10-10 11:44:56] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00315 [I 2022-10-10 11:44:56] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-10 11:44:56] ...optimizers/trijoint.py.151: total_norm: 14.650270 [I 2022-10-10 11:45:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 2070/2304 [I 2022-10-10 11:45:05] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:19 [I 2022-10-10 11:45:05] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00312 [I 2022-10-10 11:45:05] ...trap/engines/engine.py.230: loss: 0.10313 [I 2022-10-10 11:45:05] ...optimizers/trijoint.py.151: total_norm: 8.828470 [I 2022-10-10 11:45:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2080/2304 [I 2022-10-10 11:45:13] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:11 [I 2022-10-10 11:45:13] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00308 [I 2022-10-10 11:45:13] ...trap/engines/engine.py.230: loss: 0.18235 [I 2022-10-10 11:45:13] ...optimizers/trijoint.py.151: total_norm: 33.804970 [I 2022-10-10 11:45:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 2090/2304 [I 2022-10-10 11:45:22] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:03:02 [I 2022-10-10 11:45:22] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00309 [I 2022-10-10 11:45:22] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-10 11:45:22] ...optimizers/trijoint.py.151: total_norm: 17.307604 [I 2022-10-10 11:45:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 2100/2304 [I 2022-10-10 11:45:30] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:53 [I 2022-10-10 11:45:30] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00295 [I 2022-10-10 11:45:30] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-10 11:45:30] ...optimizers/trijoint.py.151: total_norm: 46.949299 [I 2022-10-10 11:45:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 2110/2304 [I 2022-10-10 11:45:39] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:45 [I 2022-10-10 11:45:39] ...trap/engines/engine.py.229: process: 0.85310 | load: 0.00298 [I 2022-10-10 11:45:39] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-10 11:45:39] ...optimizers/trijoint.py.151: total_norm: 13.473639 [I 2022-10-10 11:45:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 2120/2304 [I 2022-10-10 11:45:47] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:34 [I 2022-10-10 11:45:47] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00302 [I 2022-10-10 11:45:47] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-10 11:45:47] ...optimizers/trijoint.py.151: total_norm: 18.585482 [I 2022-10-10 11:45:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 2130/2304 [I 2022-10-10 11:45:56] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:28 [I 2022-10-10 11:45:56] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00311 [I 2022-10-10 11:45:56] ...trap/engines/engine.py.230: loss: 0.17739 [I 2022-10-10 11:45:56] ...optimizers/trijoint.py.151: total_norm: 13.985657 [I 2022-10-10 11:46:04] ...trap/engines/engine.py.224: train: epoch 70 | batch 2140/2304 [I 2022-10-10 11:46:04] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:20 [I 2022-10-10 11:46:04] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00310 [I 2022-10-10 11:46:04] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-10 11:46:04] ...optimizers/trijoint.py.151: total_norm: 31.790237 [I 2022-10-10 11:46:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2150/2304 [I 2022-10-10 11:46:13] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:02:11 [I 2022-10-10 11:46:13] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00310 [I 2022-10-10 11:46:13] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-10 11:46:13] ...optimizers/trijoint.py.151: total_norm: 11.179519 [I 2022-10-10 11:46:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 2160/2304 [I 2022-10-10 11:46:22] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:02:03 [I 2022-10-10 11:46:22] ...trap/engines/engine.py.229: process: 0.86187 | load: 0.00312 [I 2022-10-10 11:46:22] ...trap/engines/engine.py.230: loss: 0.17703 [I 2022-10-10 11:46:22] ...optimizers/trijoint.py.151: total_norm: 16.593386 [I 2022-10-10 11:46:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 2170/2304 [I 2022-10-10 11:46:30] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:54 [I 2022-10-10 11:46:30] ...trap/engines/engine.py.229: process: 0.85530 | load: 0.00307 [I 2022-10-10 11:46:30] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-10 11:46:30] ...optimizers/trijoint.py.151: total_norm: 9.617899 [I 2022-10-10 11:46:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 2180/2304 [I 2022-10-10 11:46:39] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:46 [I 2022-10-10 11:46:39] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00334 [I 2022-10-10 11:46:39] ...trap/engines/engine.py.230: loss: 0.12796 [I 2022-10-10 11:46:39] ...optimizers/trijoint.py.151: total_norm: 16.951269 [I 2022-10-10 11:46:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 2190/2304 [I 2022-10-10 11:46:47] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:37 [I 2022-10-10 11:46:47] ...trap/engines/engine.py.229: process: 0.85241 | load: 0.00307 [I 2022-10-10 11:46:47] ...trap/engines/engine.py.230: loss: 0.14068 [I 2022-10-10 11:46:47] ...optimizers/trijoint.py.151: total_norm: 19.143324 [I 2022-10-10 11:46:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 2200/2304 [I 2022-10-10 11:46:56] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:29 [I 2022-10-10 11:46:56] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00307 [I 2022-10-10 11:46:56] ...trap/engines/engine.py.230: loss: 0.15526 [I 2022-10-10 11:46:56] ...optimizers/trijoint.py.151: total_norm: 14.816532 [I 2022-10-10 11:47:04] ...trap/engines/engine.py.224: train: epoch 70 | batch 2210/2304 [I 2022-10-10 11:47:04] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:20 [I 2022-10-10 11:47:04] ...trap/engines/engine.py.229: process: 0.85770 | load: 0.00320 [I 2022-10-10 11:47:04] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-10 11:47:04] ...optimizers/trijoint.py.151: total_norm: 13.135633 [I 2022-10-10 11:47:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2220/2304 [I 2022-10-10 11:47:13] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:12 [I 2022-10-10 11:47:13] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00316 [I 2022-10-10 11:47:13] ...trap/engines/engine.py.230: loss: 0.08410 [I 2022-10-10 11:47:13] ...optimizers/trijoint.py.151: total_norm: 7.270968 [I 2022-10-10 11:47:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 2230/2304 [I 2022-10-10 11:47:22] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:01:03 [I 2022-10-10 11:47:22] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00312 [I 2022-10-10 11:47:22] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-10 11:47:22] ...optimizers/trijoint.py.151: total_norm: 16.534378 [I 2022-10-10 11:47:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 2240/2304 [I 2022-10-10 11:47:30] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:54 [I 2022-10-10 11:47:30] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00321 [I 2022-10-10 11:47:30] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-10 11:47:30] ...optimizers/trijoint.py.151: total_norm: 11.074173 [I 2022-10-10 11:47:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 2250/2304 [I 2022-10-10 11:47:39] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:46 [I 2022-10-10 11:47:39] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00311 [I 2022-10-10 11:47:39] ...trap/engines/engine.py.230: loss: 0.16893 [I 2022-10-10 11:47:39] ...optimizers/trijoint.py.151: total_norm: 12.331381 [I 2022-10-10 11:47:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 2260/2304 [I 2022-10-10 11:47:47] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:37 [I 2022-10-10 11:47:47] ...trap/engines/engine.py.229: process: 0.85392 | load: 0.00309 [I 2022-10-10 11:47:47] ...trap/engines/engine.py.230: loss: 0.13488 [I 2022-10-10 11:47:47] ...optimizers/trijoint.py.151: total_norm: 33.200367 [I 2022-10-10 11:47:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 2270/2304 [I 2022-10-10 11:47:56] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:29 [I 2022-10-10 11:47:56] ...trap/engines/engine.py.229: process: 0.86162 | load: 0.00320 [I 2022-10-10 11:47:56] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-10 11:47:56] ...optimizers/trijoint.py.151: total_norm: 9.812969 [I 2022-10-10 11:48:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 2280/2304 [I 2022-10-10 11:48:05] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:20 [I 2022-10-10 11:48:05] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00318 [I 2022-10-10 11:48:05] ...trap/engines/engine.py.230: loss: 0.13495 [I 2022-10-10 11:48:05] ...optimizers/trijoint.py.151: total_norm: 17.465958 [I 2022-10-10 11:48:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2290/2304 [I 2022-10-10 11:48:13] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:12 [I 2022-10-10 11:48:13] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00318 [I 2022-10-10 11:48:13] ...trap/engines/engine.py.230: loss: 0.13650 [I 2022-10-10 11:48:13] ...optimizers/trijoint.py.151: total_norm: 10.501687 [I 2022-10-10 11:48:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 2300/2304 [I 2022-10-10 11:48:22] ...trap/engines/engine.py.228: elapsed: 0:33:17 | left: 0:00:03 [I 2022-10-10 11:48:22] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00018 [I 2022-10-10 11:48:22] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-10 11:48:22] ...optimizers/trijoint.py.151: total_norm: 14.345221 [I 2022-10-10 11:48:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 2304/2304 [I 2022-10-10 11:48:25] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:00 [I 2022-10-10 11:48:25] ...trap/engines/engine.py.229: process: 0.83282 | load: 0.00018 [I 2022-10-10 11:48:25] ...trap/engines/engine.py.230: loss: 0.09488 [I 2022-10-10 11:48:25] ...optimizers/trijoint.py.151: total_norm: 9.081957 [S 2022-10-10 11:48:27] ...trap/engines/engine.py.129: train_epoch.epoch: 70 [S 2022-10-10 11:48:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.14072585976046234 [I 2022-10-10 11:48:27] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 11:48:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 70 [I 2022-10-10 11:48:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 11:48:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 11:48:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 11:48:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 11:48:33] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 11:48:33] ...trap/engines/engine.py.321: eval: epoch 70 | batch 0/510 [I 2022-10-10 11:48:33] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:42:35 [I 2022-10-10 11:48:33] ...trap/engines/engine.py.326: process: 5.01104 | load: 3.47659 [I 2022-10-10 11:48:38] ...trap/engines/engine.py.321: eval: epoch 70 | batch 10/510 [I 2022-10-10 11:48:38] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:24 [I 2022-10-10 11:48:38] ...trap/engines/engine.py.326: process: 0.49277 | load: 0.00020 [I 2022-10-10 11:48:43] ...trap/engines/engine.py.321: eval: epoch 70 | batch 20/510 [I 2022-10-10 11:48:43] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:22 [I 2022-10-10 11:48:43] ...trap/engines/engine.py.326: process: 0.43481 | load: 0.00025 [I 2022-10-10 11:48:49] ...trap/engines/engine.py.321: eval: epoch 70 | batch 30/510 [I 2022-10-10 11:48:49] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:44 [I 2022-10-10 11:48:49] ...trap/engines/engine.py.326: process: 0.67732 | load: 0.00019 [I 2022-10-10 11:48:55] ...trap/engines/engine.py.321: eval: epoch 70 | batch 40/510 [I 2022-10-10 11:48:55] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:34 [I 2022-10-10 11:48:55] ...trap/engines/engine.py.326: process: 0.54247 | load: 0.00020 [I 2022-10-10 11:49:00] ...trap/engines/engine.py.321: eval: epoch 70 | batch 50/510 [I 2022-10-10 11:49:00] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:44 [I 2022-10-10 11:49:00] ...trap/engines/engine.py.326: process: 0.50928 | load: 0.00020 [I 2022-10-10 11:49:05] ...trap/engines/engine.py.321: eval: epoch 70 | batch 60/510 [I 2022-10-10 11:49:05] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:39 [I 2022-10-10 11:49:05] ...trap/engines/engine.py.326: process: 0.52296 | load: 0.00019 [I 2022-10-10 11:49:09] ...trap/engines/engine.py.321: eval: epoch 70 | batch 70/510 [I 2022-10-10 11:49:09] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:25 [I 2022-10-10 11:49:09] ...trap/engines/engine.py.326: process: 0.45180 | load: 0.00020 [I 2022-10-10 11:49:14] ...trap/engines/engine.py.321: eval: epoch 70 | batch 80/510 [I 2022-10-10 11:49:14] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:10 [I 2022-10-10 11:49:14] ...trap/engines/engine.py.326: process: 0.45500 | load: 0.00021 [I 2022-10-10 11:49:18] ...trap/engines/engine.py.321: eval: epoch 70 | batch 90/510 [I 2022-10-10 11:49:18] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:11 [I 2022-10-10 11:49:18] ...trap/engines/engine.py.326: process: 0.44831 | load: 0.00019 [I 2022-10-10 11:49:23] ...trap/engines/engine.py.321: eval: epoch 70 | batch 100/510 [I 2022-10-10 11:49:23] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:57 [I 2022-10-10 11:49:23] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00020 [I 2022-10-10 11:49:26] ...trap/engines/engine.py.321: eval: epoch 70 | batch 110/510 [I 2022-10-10 11:49:26] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:13 [I 2022-10-10 11:49:26] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00020 [I 2022-10-10 11:49:29] ...trap/engines/engine.py.321: eval: epoch 70 | batch 120/510 [I 2022-10-10 11:49:29] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:05 [I 2022-10-10 11:49:29] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00020 [I 2022-10-10 11:49:33] ...trap/engines/engine.py.321: eval: epoch 70 | batch 130/510 [I 2022-10-10 11:49:33] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:03 [I 2022-10-10 11:49:33] ...trap/engines/engine.py.326: process: 0.33343 | load: 0.00020 [I 2022-10-10 11:49:36] ...trap/engines/engine.py.321: eval: epoch 70 | batch 140/510 [I 2022-10-10 11:49:36] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:59 [I 2022-10-10 11:49:36] ...trap/engines/engine.py.326: process: 0.31344 | load: 0.00020 [I 2022-10-10 11:49:39] ...trap/engines/engine.py.321: eval: epoch 70 | batch 150/510 [I 2022-10-10 11:49:39] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:55 [I 2022-10-10 11:49:39] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00021 [I 2022-10-10 11:49:42] ...trap/engines/engine.py.321: eval: epoch 70 | batch 160/510 [I 2022-10-10 11:49:42] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:52 [I 2022-10-10 11:49:42] ...trap/engines/engine.py.326: process: 0.31477 | load: 0.00021 [I 2022-10-10 11:49:46] ...trap/engines/engine.py.321: eval: epoch 70 | batch 170/510 [I 2022-10-10 11:49:46] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:50 [I 2022-10-10 11:49:46] ...trap/engines/engine.py.326: process: 0.32925 | load: 0.00022 [I 2022-10-10 11:49:49] ...trap/engines/engine.py.321: eval: epoch 70 | batch 180/510 [I 2022-10-10 11:49:49] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-10 11:49:49] ...trap/engines/engine.py.326: process: 0.31677 | load: 0.00020 [I 2022-10-10 11:49:52] ...trap/engines/engine.py.321: eval: epoch 70 | batch 190/510 [I 2022-10-10 11:49:52] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:41 [I 2022-10-10 11:49:52] ...trap/engines/engine.py.326: process: 0.31740 | load: 0.00019 [I 2022-10-10 11:49:55] ...trap/engines/engine.py.321: eval: epoch 70 | batch 200/510 [I 2022-10-10 11:49:55] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:37 [I 2022-10-10 11:49:55] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00019 [I 2022-10-10 11:49:58] ...trap/engines/engine.py.321: eval: epoch 70 | batch 210/510 [I 2022-10-10 11:49:58] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-10 11:49:58] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00018 [I 2022-10-10 11:50:02] ...trap/engines/engine.py.321: eval: epoch 70 | batch 220/510 [I 2022-10-10 11:50:02] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:33 [I 2022-10-10 11:50:02] ...trap/engines/engine.py.326: process: 0.32967 | load: 0.00021 [I 2022-10-10 11:50:05] ...trap/engines/engine.py.321: eval: epoch 70 | batch 230/510 [I 2022-10-10 11:50:05] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-10 11:50:05] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00021 [I 2022-10-10 11:50:08] ...trap/engines/engine.py.321: eval: epoch 70 | batch 240/510 [I 2022-10-10 11:50:08] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-10 11:50:08] ...trap/engines/engine.py.326: process: 0.32208 | load: 0.00019 [I 2022-10-10 11:50:11] ...trap/engines/engine.py.321: eval: epoch 70 | batch 250/510 [I 2022-10-10 11:50:11] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:22 [I 2022-10-10 11:50:11] ...trap/engines/engine.py.326: process: 0.31608 | load: 0.00019 [I 2022-10-10 11:50:14] ...trap/engines/engine.py.321: eval: epoch 70 | batch 260/510 [I 2022-10-10 11:50:14] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:19 [I 2022-10-10 11:50:14] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00021 [I 2022-10-10 11:50:18] ...trap/engines/engine.py.321: eval: epoch 70 | batch 270/510 [I 2022-10-10 11:50:18] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:15 [I 2022-10-10 11:50:18] ...trap/engines/engine.py.326: process: 0.31413 | load: 0.00020 [I 2022-10-10 11:50:21] ...trap/engines/engine.py.321: eval: epoch 70 | batch 280/510 [I 2022-10-10 11:50:21] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:12 [I 2022-10-10 11:50:21] ...trap/engines/engine.py.326: process: 0.31857 | load: 0.00019 [I 2022-10-10 11:50:24] ...trap/engines/engine.py.321: eval: epoch 70 | batch 290/510 [I 2022-10-10 11:50:24] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:09 [I 2022-10-10 11:50:24] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00019 [I 2022-10-10 11:50:27] ...trap/engines/engine.py.321: eval: epoch 70 | batch 300/510 [I 2022-10-10 11:50:27] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:06 [I 2022-10-10 11:50:27] ...trap/engines/engine.py.326: process: 0.31171 | load: 0.00020 [I 2022-10-10 11:50:30] ...trap/engines/engine.py.321: eval: epoch 70 | batch 310/510 [I 2022-10-10 11:50:30] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:03 [I 2022-10-10 11:50:30] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00019 [I 2022-10-10 11:50:33] ...trap/engines/engine.py.321: eval: epoch 70 | batch 320/510 [I 2022-10-10 11:50:33] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-10 11:50:33] ...trap/engines/engine.py.326: process: 0.32391 | load: 0.00019 [I 2022-10-10 11:50:37] ...trap/engines/engine.py.321: eval: epoch 70 | batch 330/510 [I 2022-10-10 11:50:37] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-10 11:50:37] ...trap/engines/engine.py.326: process: 0.31797 | load: 0.00021 [I 2022-10-10 11:50:40] ...trap/engines/engine.py.321: eval: epoch 70 | batch 340/510 [I 2022-10-10 11:50:40] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-10 11:50:40] ...trap/engines/engine.py.326: process: 0.31461 | load: 0.00019 [I 2022-10-10 11:50:43] ...trap/engines/engine.py.321: eval: epoch 70 | batch 350/510 [I 2022-10-10 11:50:43] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:50 [I 2022-10-10 11:50:43] ...trap/engines/engine.py.326: process: 0.31998 | load: 0.00019 [I 2022-10-10 11:50:46] ...trap/engines/engine.py.321: eval: epoch 70 | batch 360/510 [I 2022-10-10 11:50:46] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-10 11:50:46] ...trap/engines/engine.py.326: process: 0.31680 | load: 0.00019 [I 2022-10-10 11:50:49] ...trap/engines/engine.py.321: eval: epoch 70 | batch 370/510 [I 2022-10-10 11:50:49] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-10 11:50:49] ...trap/engines/engine.py.326: process: 0.31019 | load: 0.00020 [I 2022-10-10 11:50:53] ...trap/engines/engine.py.321: eval: epoch 70 | batch 380/510 [I 2022-10-10 11:50:53] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-10 11:50:53] ...trap/engines/engine.py.326: process: 0.31846 | load: 0.00019 [I 2022-10-10 11:50:56] ...trap/engines/engine.py.321: eval: epoch 70 | batch 390/510 [I 2022-10-10 11:50:56] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-10 11:50:56] ...trap/engines/engine.py.326: process: 0.31338 | load: 0.00020 [I 2022-10-10 11:50:59] ...trap/engines/engine.py.321: eval: epoch 70 | batch 400/510 [I 2022-10-10 11:50:59] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:34 [I 2022-10-10 11:50:59] ...trap/engines/engine.py.326: process: 0.30833 | load: 0.00020 [I 2022-10-10 11:51:02] ...trap/engines/engine.py.321: eval: epoch 70 | batch 410/510 [I 2022-10-10 11:51:02] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-10 11:51:02] ...trap/engines/engine.py.326: process: 0.31866 | load: 0.00019 [I 2022-10-10 11:51:05] ...trap/engines/engine.py.321: eval: epoch 70 | batch 420/510 [I 2022-10-10 11:51:05] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-10 11:51:05] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00019 [I 2022-10-10 11:51:09] ...trap/engines/engine.py.321: eval: epoch 70 | batch 430/510 [I 2022-10-10 11:51:09] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-10 11:51:09] ...trap/engines/engine.py.326: process: 0.31878 | load: 0.00019 [I 2022-10-10 11:51:12] ...trap/engines/engine.py.321: eval: epoch 70 | batch 440/510 [I 2022-10-10 11:51:12] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-10 11:51:12] ...trap/engines/engine.py.326: process: 0.31616 | load: 0.00019 [I 2022-10-10 11:51:15] ...trap/engines/engine.py.321: eval: epoch 70 | batch 450/510 [I 2022-10-10 11:51:15] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-10 11:51:15] ...trap/engines/engine.py.326: process: 0.31550 | load: 0.00020 [I 2022-10-10 11:51:18] ...trap/engines/engine.py.321: eval: epoch 70 | batch 460/510 [I 2022-10-10 11:51:18] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-10 11:51:18] ...trap/engines/engine.py.326: process: 0.31939 | load: 0.00030 [I 2022-10-10 11:51:21] ...trap/engines/engine.py.321: eval: epoch 70 | batch 470/510 [I 2022-10-10 11:51:21] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-10 11:51:21] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00020 [I 2022-10-10 11:51:24] ...trap/engines/engine.py.321: eval: epoch 70 | batch 480/510 [I 2022-10-10 11:51:24] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-10 11:51:24] ...trap/engines/engine.py.326: process: 0.31283 | load: 0.00020 [I 2022-10-10 11:51:28] ...trap/engines/engine.py.321: eval: epoch 70 | batch 490/510 [I 2022-10-10 11:51:28] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 11:51:28] ...trap/engines/engine.py.326: process: 0.31761 | load: 0.00019 [I 2022-10-10 11:51:31] ...trap/engines/engine.py.321: eval: epoch 70 | batch 500/510 [I 2022-10-10 11:51:31] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-10 11:51:31] ...trap/engines/engine.py.326: process: 0.31435 | load: 0.00020 [I 2022-10-10 11:51:34] ...trap/engines/engine.py.321: eval: epoch 70 | batch 510/510 [I 2022-10-10 11:51:34] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-10 11:51:34] ...trap/engines/engine.py.326: process: 0.31599 | load: 0.00016 [S 2022-10-10 11:51:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 70 [I 2022-10-10 11:51:35] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728200 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905100 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934200 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724700 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.902800 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933000 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009847 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009071 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006675 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009696 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006838 [S 2022-10-10 11:52:03] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006033 [I 2022-10-10 11:52:20] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 11:52:20] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 11:52:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 11:52:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 11:52:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 11:52:25] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 11:52:25] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 11:52:31] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 11:52:31] ...trap/engines/engine.py.163: Training model on trainset for epoch 71 [I 2022-10-10 11:52:31] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 11:52:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 0/2304 [I 2022-10-10 11:52:35] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:07:15 [I 2022-10-10 11:52:35] ...trap/engines/engine.py.229: process: 3.31408 | load: 2.39999 [I 2022-10-10 11:52:35] ...trap/engines/engine.py.230: loss: 0.16007 [I 2022-10-10 11:52:35] ...optimizers/trijoint.py.151: total_norm: 16.655479 [I 2022-10-10 11:52:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 10/2304 [I 2022-10-10 11:52:43] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:43:05 [I 2022-10-10 11:52:43] ...trap/engines/engine.py.229: process: 0.86523 | load: 0.00610 [I 2022-10-10 11:52:43] ...trap/engines/engine.py.230: loss: 0.13941 [I 2022-10-10 11:52:43] ...optimizers/trijoint.py.151: total_norm: 24.645535 [I 2022-10-10 11:52:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 20/2304 [I 2022-10-10 11:52:52] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:37 [I 2022-10-10 11:52:52] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00436 [I 2022-10-10 11:52:52] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-10 11:52:52] ...optimizers/trijoint.py.151: total_norm: 13.845592 [I 2022-10-10 11:53:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 30/2304 [I 2022-10-10 11:53:01] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:50 [I 2022-10-10 11:53:01] ...trap/engines/engine.py.229: process: 0.87770 | load: 0.00467 [I 2022-10-10 11:53:01] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-10 11:53:01] ...optimizers/trijoint.py.151: total_norm: 13.306827 [I 2022-10-10 11:53:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 40/2304 [I 2022-10-10 11:53:09] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:39 [I 2022-10-10 11:53:09] ...trap/engines/engine.py.229: process: 0.87201 | load: 0.00471 [I 2022-10-10 11:53:09] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-10 11:53:09] ...optimizers/trijoint.py.151: total_norm: 14.741296 [I 2022-10-10 11:53:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 50/2304 [I 2022-10-10 11:53:18] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:19 [I 2022-10-10 11:53:18] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00437 [I 2022-10-10 11:53:18] ...trap/engines/engine.py.230: loss: 0.09344 [I 2022-10-10 11:53:18] ...optimizers/trijoint.py.151: total_norm: 9.761564 [I 2022-10-10 11:53:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 60/2304 [I 2022-10-10 11:53:26] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:05 [I 2022-10-10 11:53:26] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00350 [I 2022-10-10 11:53:26] ...trap/engines/engine.py.230: loss: 0.28135 [I 2022-10-10 11:53:26] ...optimizers/trijoint.py.151: total_norm: 34.007191 [I 2022-10-10 11:53:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 70/2304 [I 2022-10-10 11:53:35] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:03 [I 2022-10-10 11:53:35] ...trap/engines/engine.py.229: process: 0.86688 | load: 0.00370 [I 2022-10-10 11:53:35] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-10 11:53:35] ...optimizers/trijoint.py.151: total_norm: 13.373830 [I 2022-10-10 11:53:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 80/2304 [I 2022-10-10 11:53:44] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:53 [I 2022-10-10 11:53:44] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00337 [I 2022-10-10 11:53:44] ...trap/engines/engine.py.230: loss: 0.11506 [I 2022-10-10 11:53:44] ...optimizers/trijoint.py.151: total_norm: 16.233395 [I 2022-10-10 11:53:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 90/2304 [I 2022-10-10 11:53:52] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:43 [I 2022-10-10 11:53:52] ...trap/engines/engine.py.229: process: 0.85079 | load: 0.00358 [I 2022-10-10 11:53:52] ...trap/engines/engine.py.230: loss: 0.10255 [I 2022-10-10 11:53:52] ...optimizers/trijoint.py.151: total_norm: 16.602356 [I 2022-10-10 11:54:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 100/2304 [I 2022-10-10 11:54:01] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:22 [I 2022-10-10 11:54:01] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00342 [I 2022-10-10 11:54:01] ...trap/engines/engine.py.230: loss: 0.11117 [I 2022-10-10 11:54:01] ...optimizers/trijoint.py.151: total_norm: 10.072445 [I 2022-10-10 11:54:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 110/2304 [I 2022-10-10 11:54:09] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:19 [I 2022-10-10 11:54:09] ...trap/engines/engine.py.229: process: 0.84923 | load: 0.00355 [I 2022-10-10 11:54:09] ...trap/engines/engine.py.230: loss: 0.15275 [I 2022-10-10 11:54:09] ...optimizers/trijoint.py.151: total_norm: 64.543030 [I 2022-10-10 11:54:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 120/2304 [I 2022-10-10 11:54:18] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:14 [I 2022-10-10 11:54:18] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00377 [I 2022-10-10 11:54:18] ...trap/engines/engine.py.230: loss: 0.17977 [I 2022-10-10 11:54:18] ...optimizers/trijoint.py.151: total_norm: 12.027959 [I 2022-10-10 11:54:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 130/2304 [I 2022-10-10 11:54:27] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:03 [I 2022-10-10 11:54:27] ...trap/engines/engine.py.229: process: 0.85802 | load: 0.00381 [I 2022-10-10 11:54:27] ...trap/engines/engine.py.230: loss: 0.09937 [I 2022-10-10 11:54:27] ...optimizers/trijoint.py.151: total_norm: 11.464215 [I 2022-10-10 11:54:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 140/2304 [I 2022-10-10 11:54:35] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:31:14 [I 2022-10-10 11:54:35] ...trap/engines/engine.py.229: process: 0.87474 | load: 0.00371 [I 2022-10-10 11:54:35] ...trap/engines/engine.py.230: loss: 0.17788 [I 2022-10-10 11:54:35] ...optimizers/trijoint.py.151: total_norm: 38.747330 [I 2022-10-10 11:54:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 150/2304 [I 2022-10-10 11:54:44] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:55 [I 2022-10-10 11:54:44] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00343 [I 2022-10-10 11:54:44] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-10 11:54:44] ...optimizers/trijoint.py.151: total_norm: 7.882437 [I 2022-10-10 11:54:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 160/2304 [I 2022-10-10 11:54:53] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:31:00 [I 2022-10-10 11:54:53] ...trap/engines/engine.py.229: process: 0.87280 | load: 0.00338 [I 2022-10-10 11:54:53] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-10 11:54:53] ...optimizers/trijoint.py.151: total_norm: 7.658854 [I 2022-10-10 11:55:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 170/2304 [I 2022-10-10 11:55:01] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:26 [I 2022-10-10 11:55:01] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00325 [I 2022-10-10 11:55:01] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-10 11:55:01] ...optimizers/trijoint.py.151: total_norm: 15.444310 [I 2022-10-10 11:55:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 180/2304 [I 2022-10-10 11:55:10] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:17 [I 2022-10-10 11:55:10] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00328 [I 2022-10-10 11:55:10] ...trap/engines/engine.py.230: loss: 0.11437 [I 2022-10-10 11:55:10] ...optimizers/trijoint.py.151: total_norm: 8.298449 [I 2022-10-10 11:55:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 190/2304 [I 2022-10-10 11:55:18] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:14 [I 2022-10-10 11:55:18] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00319 [I 2022-10-10 11:55:18] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-10 11:55:18] ...optimizers/trijoint.py.151: total_norm: 8.525990 [I 2022-10-10 11:55:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 200/2304 [I 2022-10-10 11:55:27] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:50 [I 2022-10-10 11:55:27] ...trap/engines/engine.py.229: process: 0.84583 | load: 0.00323 [I 2022-10-10 11:55:27] ...trap/engines/engine.py.230: loss: 0.09748 [I 2022-10-10 11:55:27] ...optimizers/trijoint.py.151: total_norm: 7.458818 [I 2022-10-10 11:55:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 210/2304 [I 2022-10-10 11:55:36] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:40 [I 2022-10-10 11:55:36] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00294 [I 2022-10-10 11:55:36] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-10 11:55:36] ...optimizers/trijoint.py.151: total_norm: 18.741901 [I 2022-10-10 11:55:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 220/2304 [I 2022-10-10 11:55:44] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:29 [I 2022-10-10 11:55:44] ...trap/engines/engine.py.229: process: 0.85967 | load: 0.00321 [I 2022-10-10 11:55:44] ...trap/engines/engine.py.230: loss: 0.10436 [I 2022-10-10 11:55:44] ...optimizers/trijoint.py.151: total_norm: 10.990036 [I 2022-10-10 11:55:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 230/2304 [I 2022-10-10 11:55:53] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:30 [I 2022-10-10 11:55:53] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00306 [I 2022-10-10 11:55:53] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-10 11:55:53] ...optimizers/trijoint.py.151: total_norm: 15.373581 [I 2022-10-10 11:56:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 240/2304 [I 2022-10-10 11:56:01] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:17 [I 2022-10-10 11:56:01] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00297 [I 2022-10-10 11:56:01] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-10 11:56:01] ...optimizers/trijoint.py.151: total_norm: 12.094866 [I 2022-10-10 11:56:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 250/2304 [I 2022-10-10 11:56:10] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:03 [I 2022-10-10 11:56:10] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00307 [I 2022-10-10 11:56:10] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-10 11:56:10] ...optimizers/trijoint.py.151: total_norm: 20.389120 [I 2022-10-10 11:56:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 260/2304 [I 2022-10-10 11:56:18] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:28:59 [I 2022-10-10 11:56:18] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00323 [I 2022-10-10 11:56:18] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-10 11:56:18] ...optimizers/trijoint.py.151: total_norm: 12.260910 [I 2022-10-10 11:56:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 270/2304 [I 2022-10-10 11:56:27] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:28:51 [I 2022-10-10 11:56:27] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00307 [I 2022-10-10 11:56:27] ...trap/engines/engine.py.230: loss: 0.13376 [I 2022-10-10 11:56:27] ...optimizers/trijoint.py.151: total_norm: 15.109674 [I 2022-10-10 11:56:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 280/2304 [I 2022-10-10 11:56:35] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:38 [I 2022-10-10 11:56:35] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00310 [I 2022-10-10 11:56:35] ...trap/engines/engine.py.230: loss: 0.10857 [I 2022-10-10 11:56:35] ...optimizers/trijoint.py.151: total_norm: 15.061659 [I 2022-10-10 11:56:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 290/2304 [I 2022-10-10 11:56:44] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:33 [I 2022-10-10 11:56:44] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00294 [I 2022-10-10 11:56:44] ...trap/engines/engine.py.230: loss: 0.11638 [I 2022-10-10 11:56:44] ...optimizers/trijoint.py.151: total_norm: 22.835159 [I 2022-10-10 11:56:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 300/2304 [I 2022-10-10 11:56:52] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:22 [I 2022-10-10 11:56:52] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00296 [I 2022-10-10 11:56:52] ...trap/engines/engine.py.230: loss: 0.16367 [I 2022-10-10 11:56:52] ...optimizers/trijoint.py.151: total_norm: 15.092576 [I 2022-10-10 11:57:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 310/2304 [I 2022-10-10 11:57:01] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:19 [I 2022-10-10 11:57:01] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00298 [I 2022-10-10 11:57:01] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-10 11:57:01] ...optimizers/trijoint.py.151: total_norm: 18.443171 [I 2022-10-10 11:57:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 320/2304 [I 2022-10-10 11:57:09] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:02 [I 2022-10-10 11:57:09] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00295 [I 2022-10-10 11:57:09] ...trap/engines/engine.py.230: loss: 0.10594 [I 2022-10-10 11:57:09] ...optimizers/trijoint.py.151: total_norm: 21.094383 [I 2022-10-10 11:57:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 330/2304 [I 2022-10-10 11:57:18] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:27:57 [I 2022-10-10 11:57:18] ...trap/engines/engine.py.229: process: 0.86161 | load: 0.00300 [I 2022-10-10 11:57:18] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-10 11:57:18] ...optimizers/trijoint.py.151: total_norm: 17.002417 [I 2022-10-10 11:57:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 340/2304 [I 2022-10-10 11:57:26] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:04 [I 2022-10-10 11:57:26] ...trap/engines/engine.py.229: process: 0.87651 | load: 0.00302 [I 2022-10-10 11:57:26] ...trap/engines/engine.py.230: loss: 0.15660 [I 2022-10-10 11:57:26] ...optimizers/trijoint.py.151: total_norm: 11.088085 [I 2022-10-10 11:57:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 350/2304 [I 2022-10-10 11:57:35] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:27:44 [I 2022-10-10 11:57:35] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00314 [I 2022-10-10 11:57:35] ...trap/engines/engine.py.230: loss: 0.15159 [I 2022-10-10 11:57:35] ...optimizers/trijoint.py.151: total_norm: 16.216099 [I 2022-10-10 11:57:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 360/2304 [I 2022-10-10 11:57:44] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:52 [I 2022-10-10 11:57:44] ...trap/engines/engine.py.229: process: 0.87355 | load: 0.00320 [I 2022-10-10 11:57:44] ...trap/engines/engine.py.230: loss: 0.20186 [I 2022-10-10 11:57:44] ...optimizers/trijoint.py.151: total_norm: 33.043491 [I 2022-10-10 11:57:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 370/2304 [I 2022-10-10 11:57:52] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:29 [I 2022-10-10 11:57:52] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00311 [I 2022-10-10 11:57:52] ...trap/engines/engine.py.230: loss: 0.08876 [I 2022-10-10 11:57:52] ...optimizers/trijoint.py.151: total_norm: 11.539583 [I 2022-10-10 11:58:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 380/2304 [I 2022-10-10 11:58:01] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:27:41 [I 2022-10-10 11:58:01] ...trap/engines/engine.py.229: process: 0.88164 | load: 0.00322 [I 2022-10-10 11:58:01] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-10 11:58:01] ...optimizers/trijoint.py.151: total_norm: 10.622894 [I 2022-10-10 11:58:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 390/2304 [I 2022-10-10 11:58:09] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:27 [I 2022-10-10 11:58:09] ...trap/engines/engine.py.229: process: 0.85516 | load: 0.00311 [I 2022-10-10 11:58:09] ...trap/engines/engine.py.230: loss: 0.18458 [I 2022-10-10 11:58:09] ...optimizers/trijoint.py.151: total_norm: 13.750893 [I 2022-10-10 11:58:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 400/2304 [I 2022-10-10 11:58:18] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:27:30 [I 2022-10-10 11:58:18] ...trap/engines/engine.py.229: process: 0.86737 | load: 0.00315 [I 2022-10-10 11:58:18] ...trap/engines/engine.py.230: loss: 0.14443 [I 2022-10-10 11:58:18] ...optimizers/trijoint.py.151: total_norm: 12.412663 [I 2022-10-10 11:58:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 410/2304 [I 2022-10-10 11:58:27] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:27:11 [I 2022-10-10 11:58:27] ...trap/engines/engine.py.229: process: 0.87644 | load: 0.00314 [I 2022-10-10 11:58:27] ...trap/engines/engine.py.230: loss: 0.15319 [I 2022-10-10 11:58:27] ...optimizers/trijoint.py.151: total_norm: 13.004379 [I 2022-10-10 11:58:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 420/2304 [I 2022-10-10 11:58:35] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:27:05 [I 2022-10-10 11:58:35] ...trap/engines/engine.py.229: process: 0.86252 | load: 0.00312 [I 2022-10-10 11:58:35] ...trap/engines/engine.py.230: loss: 0.17138 [I 2022-10-10 11:58:35] ...optimizers/trijoint.py.151: total_norm: 11.900429 [I 2022-10-10 11:58:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 430/2304 [I 2022-10-10 11:58:44] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:26:55 [I 2022-10-10 11:58:44] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00314 [I 2022-10-10 11:58:44] ...trap/engines/engine.py.230: loss: 0.16087 [I 2022-10-10 11:58:44] ...optimizers/trijoint.py.151: total_norm: 11.460157 [I 2022-10-10 11:58:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 440/2304 [I 2022-10-10 11:58:53] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:48 [I 2022-10-10 11:58:53] ...trap/engines/engine.py.229: process: 0.86511 | load: 0.00309 [I 2022-10-10 11:58:53] ...trap/engines/engine.py.230: loss: 0.20010 [I 2022-10-10 11:58:53] ...optimizers/trijoint.py.151: total_norm: 8.276689 [I 2022-10-10 11:59:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 450/2304 [I 2022-10-10 11:59:01] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:37 [I 2022-10-10 11:59:01] ...trap/engines/engine.py.229: process: 0.86387 | load: 0.00320 [I 2022-10-10 11:59:01] ...trap/engines/engine.py.230: loss: 0.16693 [I 2022-10-10 11:59:01] ...optimizers/trijoint.py.151: total_norm: 25.367388 [I 2022-10-10 11:59:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 460/2304 [I 2022-10-10 11:59:10] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:26:17 [I 2022-10-10 11:59:10] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00304 [I 2022-10-10 11:59:10] ...trap/engines/engine.py.230: loss: 0.12920 [I 2022-10-10 11:59:10] ...optimizers/trijoint.py.151: total_norm: 17.755480 [I 2022-10-10 11:59:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 470/2304 [I 2022-10-10 11:59:18] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:30 [I 2022-10-10 11:59:18] ...trap/engines/engine.py.229: process: 0.87865 | load: 0.00306 [I 2022-10-10 11:59:18] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-10 11:59:18] ...optimizers/trijoint.py.151: total_norm: 45.926189 [I 2022-10-10 11:59:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 480/2304 [I 2022-10-10 11:59:27] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:26:08 [I 2022-10-10 11:59:27] ...trap/engines/engine.py.229: process: 0.86792 | load: 0.00316 [I 2022-10-10 11:59:27] ...trap/engines/engine.py.230: loss: 0.12852 [I 2022-10-10 11:59:27] ...optimizers/trijoint.py.151: total_norm: 16.625893 [I 2022-10-10 11:59:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 490/2304 [I 2022-10-10 11:59:36] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:59 [I 2022-10-10 11:59:36] ...trap/engines/engine.py.229: process: 0.88240 | load: 0.00306 [I 2022-10-10 11:59:36] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-10 11:59:36] ...optimizers/trijoint.py.151: total_norm: 23.341261 [I 2022-10-10 11:59:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 500/2304 [I 2022-10-10 11:59:44] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:25:56 [I 2022-10-10 11:59:44] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00309 [I 2022-10-10 11:59:44] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-10 11:59:44] ...optimizers/trijoint.py.151: total_norm: 9.354417 [I 2022-10-10 11:59:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 510/2304 [I 2022-10-10 11:59:53] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:33 [I 2022-10-10 11:59:53] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00307 [I 2022-10-10 11:59:53] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-10 11:59:53] ...optimizers/trijoint.py.151: total_norm: 10.704142 [I 2022-10-10 12:00:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 520/2304 [I 2022-10-10 12:00:01] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:28 [I 2022-10-10 12:00:01] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00307 [I 2022-10-10 12:00:01] ...trap/engines/engine.py.230: loss: 0.13697 [I 2022-10-10 12:00:02] ...optimizers/trijoint.py.151: total_norm: 50.098236 [I 2022-10-10 12:00:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 530/2304 [I 2022-10-10 12:00:10] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:25:29 [I 2022-10-10 12:00:10] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00315 [I 2022-10-10 12:00:10] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-10 12:00:10] ...optimizers/trijoint.py.151: total_norm: 11.967395 [I 2022-10-10 12:00:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 540/2304 [I 2022-10-10 12:00:19] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:25:24 [I 2022-10-10 12:00:19] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00324 [I 2022-10-10 12:00:19] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-10 12:00:19] ...optimizers/trijoint.py.151: total_norm: 12.957705 [I 2022-10-10 12:00:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 550/2304 [I 2022-10-10 12:00:27] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:25:10 [I 2022-10-10 12:00:27] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00310 [I 2022-10-10 12:00:27] ...trap/engines/engine.py.230: loss: 0.17297 [I 2022-10-10 12:00:27] ...optimizers/trijoint.py.151: total_norm: 13.454811 [I 2022-10-10 12:00:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 560/2304 [I 2022-10-10 12:00:36] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:25:04 [I 2022-10-10 12:00:36] ...trap/engines/engine.py.229: process: 0.87036 | load: 0.00308 [I 2022-10-10 12:00:36] ...trap/engines/engine.py.230: loss: 0.10439 [I 2022-10-10 12:00:36] ...optimizers/trijoint.py.151: total_norm: 13.393600 [I 2022-10-10 12:00:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 570/2304 [I 2022-10-10 12:00:45] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:51 [I 2022-10-10 12:00:45] ...trap/engines/engine.py.229: process: 0.86761 | load: 0.00309 [I 2022-10-10 12:00:45] ...trap/engines/engine.py.230: loss: 0.12979 [I 2022-10-10 12:00:45] ...optimizers/trijoint.py.151: total_norm: 16.460384 [I 2022-10-10 12:00:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 580/2304 [I 2022-10-10 12:00:53] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:43 [I 2022-10-10 12:00:53] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00314 [I 2022-10-10 12:00:53] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-10 12:00:53] ...optimizers/trijoint.py.151: total_norm: 26.993118 [I 2022-10-10 12:01:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 590/2304 [I 2022-10-10 12:01:02] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:35 [I 2022-10-10 12:01:02] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00311 [I 2022-10-10 12:01:02] ...trap/engines/engine.py.230: loss: 0.11407 [I 2022-10-10 12:01:02] ...optimizers/trijoint.py.151: total_norm: 11.542286 [I 2022-10-10 12:01:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 600/2304 [I 2022-10-10 12:01:11] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:24:25 [I 2022-10-10 12:01:11] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00315 [I 2022-10-10 12:01:11] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-10 12:01:11] ...optimizers/trijoint.py.151: total_norm: 18.542532 [I 2022-10-10 12:01:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 610/2304 [I 2022-10-10 12:01:19] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:24:11 [I 2022-10-10 12:01:19] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00310 [I 2022-10-10 12:01:19] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-10 12:01:19] ...optimizers/trijoint.py.151: total_norm: 25.591623 [I 2022-10-10 12:01:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 620/2304 [I 2022-10-10 12:01:28] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:24:05 [I 2022-10-10 12:01:28] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00314 [I 2022-10-10 12:01:28] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-10 12:01:28] ...optimizers/trijoint.py.151: total_norm: 11.183365 [I 2022-10-10 12:01:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 630/2304 [I 2022-10-10 12:01:36] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:47 [I 2022-10-10 12:01:36] ...trap/engines/engine.py.229: process: 0.85335 | load: 0.00295 [I 2022-10-10 12:01:36] ...trap/engines/engine.py.230: loss: 0.22145 [I 2022-10-10 12:01:36] ...optimizers/trijoint.py.151: total_norm: 25.760773 [I 2022-10-10 12:01:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 640/2304 [I 2022-10-10 12:01:45] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:34 [I 2022-10-10 12:01:45] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00297 [I 2022-10-10 12:01:45] ...trap/engines/engine.py.230: loss: 0.13514 [I 2022-10-10 12:01:45] ...optimizers/trijoint.py.151: total_norm: 16.548912 [I 2022-10-10 12:01:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 650/2304 [I 2022-10-10 12:01:53] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:36 [I 2022-10-10 12:01:53] ...trap/engines/engine.py.229: process: 0.87114 | load: 0.00298 [I 2022-10-10 12:01:53] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-10 12:01:53] ...optimizers/trijoint.py.151: total_norm: 8.972509 [I 2022-10-10 12:02:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 660/2304 [I 2022-10-10 12:02:02] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:19 [I 2022-10-10 12:02:02] ...trap/engines/engine.py.229: process: 0.84367 | load: 0.00301 [I 2022-10-10 12:02:02] ...trap/engines/engine.py.230: loss: 0.13922 [I 2022-10-10 12:02:02] ...optimizers/trijoint.py.151: total_norm: 13.024508 [I 2022-10-10 12:02:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 670/2304 [I 2022-10-10 12:02:10] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:23:14 [I 2022-10-10 12:02:10] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00294 [I 2022-10-10 12:02:10] ...trap/engines/engine.py.230: loss: 0.19412 [I 2022-10-10 12:02:10] ...optimizers/trijoint.py.151: total_norm: 9.603111 [I 2022-10-10 12:02:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 680/2304 [I 2022-10-10 12:02:19] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:22:57 [I 2022-10-10 12:02:19] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00299 [I 2022-10-10 12:02:19] ...trap/engines/engine.py.230: loss: 0.09868 [I 2022-10-10 12:02:19] ...optimizers/trijoint.py.151: total_norm: 12.222644 [I 2022-10-10 12:02:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 690/2304 [I 2022-10-10 12:02:27] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:22:57 [I 2022-10-10 12:02:27] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00298 [I 2022-10-10 12:02:27] ...trap/engines/engine.py.230: loss: 0.17822 [I 2022-10-10 12:02:27] ...optimizers/trijoint.py.151: total_norm: 16.628347 [I 2022-10-10 12:02:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 700/2304 [I 2022-10-10 12:02:36] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:22:41 [I 2022-10-10 12:02:36] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00307 [I 2022-10-10 12:02:36] ...trap/engines/engine.py.230: loss: 0.15898 [I 2022-10-10 12:02:36] ...optimizers/trijoint.py.151: total_norm: 26.688040 [I 2022-10-10 12:02:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 710/2304 [I 2022-10-10 12:02:45] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:22:41 [I 2022-10-10 12:02:45] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00301 [I 2022-10-10 12:02:45] ...trap/engines/engine.py.230: loss: 0.11231 [I 2022-10-10 12:02:45] ...optimizers/trijoint.py.151: total_norm: 19.086769 [I 2022-10-10 12:02:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 720/2304 [I 2022-10-10 12:02:53] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:22:42 [I 2022-10-10 12:02:53] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00294 [I 2022-10-10 12:02:53] ...trap/engines/engine.py.230: loss: 0.20677 [I 2022-10-10 12:02:53] ...optimizers/trijoint.py.151: total_norm: 12.332336 [I 2022-10-10 12:03:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 730/2304 [I 2022-10-10 12:03:02] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:22:20 [I 2022-10-10 12:03:02] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00296 [I 2022-10-10 12:03:02] ...trap/engines/engine.py.230: loss: 0.19727 [I 2022-10-10 12:03:02] ...optimizers/trijoint.py.151: total_norm: 23.477692 [I 2022-10-10 12:03:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 740/2304 [I 2022-10-10 12:03:10] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:22:00 [I 2022-10-10 12:03:10] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00293 [I 2022-10-10 12:03:10] ...trap/engines/engine.py.230: loss: 0.14900 [I 2022-10-10 12:03:10] ...optimizers/trijoint.py.151: total_norm: 14.883648 [I 2022-10-10 12:03:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 750/2304 [I 2022-10-10 12:03:19] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:21:56 [I 2022-10-10 12:03:19] ...trap/engines/engine.py.229: process: 0.86488 | load: 0.00303 [I 2022-10-10 12:03:19] ...trap/engines/engine.py.230: loss: 0.15473 [I 2022-10-10 12:03:19] ...optimizers/trijoint.py.151: total_norm: 9.284309 [I 2022-10-10 12:03:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 760/2304 [I 2022-10-10 12:03:27] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:21:50 [I 2022-10-10 12:03:27] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00294 [I 2022-10-10 12:03:27] ...trap/engines/engine.py.230: loss: 0.18476 [I 2022-10-10 12:03:27] ...optimizers/trijoint.py.151: total_norm: 13.413321 [I 2022-10-10 12:03:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 770/2304 [I 2022-10-10 12:03:36] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:21:43 [I 2022-10-10 12:03:36] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00305 [I 2022-10-10 12:03:36] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-10 12:03:36] ...optimizers/trijoint.py.151: total_norm: 29.943556 [I 2022-10-10 12:03:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 780/2304 [I 2022-10-10 12:03:44] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:21:34 [I 2022-10-10 12:03:44] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00297 [I 2022-10-10 12:03:44] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-10 12:03:44] ...optimizers/trijoint.py.151: total_norm: 7.168659 [I 2022-10-10 12:03:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 790/2304 [I 2022-10-10 12:03:53] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:35 [I 2022-10-10 12:03:53] ...trap/engines/engine.py.229: process: 0.86567 | load: 0.00314 [I 2022-10-10 12:03:53] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-10 12:03:53] ...optimizers/trijoint.py.151: total_norm: 16.851927 [I 2022-10-10 12:04:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 800/2304 [I 2022-10-10 12:04:01] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:21:21 [I 2022-10-10 12:04:01] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00308 [I 2022-10-10 12:04:01] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-10 12:04:01] ...optimizers/trijoint.py.151: total_norm: 13.391709 [I 2022-10-10 12:04:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 810/2304 [I 2022-10-10 12:04:10] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:21:18 [I 2022-10-10 12:04:10] ...trap/engines/engine.py.229: process: 0.85094 | load: 0.00310 [I 2022-10-10 12:04:10] ...trap/engines/engine.py.230: loss: 0.11045 [I 2022-10-10 12:04:10] ...optimizers/trijoint.py.151: total_norm: 13.951903 [I 2022-10-10 12:04:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 820/2304 [I 2022-10-10 12:04:19] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:21:20 [I 2022-10-10 12:04:19] ...trap/engines/engine.py.229: process: 0.85587 | load: 0.00315 [I 2022-10-10 12:04:19] ...trap/engines/engine.py.230: loss: 0.12464 [I 2022-10-10 12:04:19] ...optimizers/trijoint.py.151: total_norm: 15.950974 [I 2022-10-10 12:04:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 830/2304 [I 2022-10-10 12:04:27] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:21:02 [I 2022-10-10 12:04:27] ...trap/engines/engine.py.229: process: 0.85842 | load: 0.00373 [I 2022-10-10 12:04:27] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-10 12:04:27] ...optimizers/trijoint.py.151: total_norm: 25.161163 [I 2022-10-10 12:04:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 840/2304 [I 2022-10-10 12:04:36] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:59 [I 2022-10-10 12:04:36] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00312 [I 2022-10-10 12:04:36] ...trap/engines/engine.py.230: loss: 0.13114 [I 2022-10-10 12:04:36] ...optimizers/trijoint.py.151: total_norm: 27.506596 [I 2022-10-10 12:04:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 850/2304 [I 2022-10-10 12:04:44] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:46 [I 2022-10-10 12:04:44] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00309 [I 2022-10-10 12:04:44] ...trap/engines/engine.py.230: loss: 0.20619 [I 2022-10-10 12:04:44] ...optimizers/trijoint.py.151: total_norm: 20.314997 [I 2022-10-10 12:04:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 860/2304 [I 2022-10-10 12:04:53] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:41 [I 2022-10-10 12:04:53] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00307 [I 2022-10-10 12:04:53] ...trap/engines/engine.py.230: loss: 0.14168 [I 2022-10-10 12:04:53] ...optimizers/trijoint.py.151: total_norm: 20.863882 [I 2022-10-10 12:05:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 870/2304 [I 2022-10-10 12:05:02] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:31 [I 2022-10-10 12:05:02] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00315 [I 2022-10-10 12:05:02] ...trap/engines/engine.py.230: loss: 0.14513 [I 2022-10-10 12:05:02] ...optimizers/trijoint.py.151: total_norm: 29.201735 [I 2022-10-10 12:05:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 880/2304 [I 2022-10-10 12:05:10] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:20:24 [I 2022-10-10 12:05:10] ...trap/engines/engine.py.229: process: 0.86979 | load: 0.00311 [I 2022-10-10 12:05:10] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-10 12:05:10] ...optimizers/trijoint.py.151: total_norm: 21.459572 [I 2022-10-10 12:05:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 890/2304 [I 2022-10-10 12:05:19] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:19 [I 2022-10-10 12:05:19] ...trap/engines/engine.py.229: process: 0.86630 | load: 0.00308 [I 2022-10-10 12:05:19] ...trap/engines/engine.py.230: loss: 0.09915 [I 2022-10-10 12:05:19] ...optimizers/trijoint.py.151: total_norm: 8.378286 [I 2022-10-10 12:05:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 900/2304 [I 2022-10-10 12:05:27] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:20:01 [I 2022-10-10 12:05:27] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00311 [I 2022-10-10 12:05:27] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-10 12:05:27] ...optimizers/trijoint.py.151: total_norm: 17.151136 [I 2022-10-10 12:05:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 910/2304 [I 2022-10-10 12:05:36] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:53 [I 2022-10-10 12:05:36] ...trap/engines/engine.py.229: process: 0.86480 | load: 0.00306 [I 2022-10-10 12:05:36] ...trap/engines/engine.py.230: loss: 0.09933 [I 2022-10-10 12:05:36] ...optimizers/trijoint.py.151: total_norm: 9.268806 [I 2022-10-10 12:05:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 920/2304 [I 2022-10-10 12:05:45] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:19:45 [I 2022-10-10 12:05:45] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00313 [I 2022-10-10 12:05:45] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-10 12:05:45] ...optimizers/trijoint.py.151: total_norm: 14.846439 [I 2022-10-10 12:05:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 930/2304 [I 2022-10-10 12:05:53] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:49 [I 2022-10-10 12:05:53] ...trap/engines/engine.py.229: process: 0.86475 | load: 0.00313 [I 2022-10-10 12:05:53] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-10 12:05:53] ...optimizers/trijoint.py.151: total_norm: 13.837601 [I 2022-10-10 12:06:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 940/2304 [I 2022-10-10 12:06:02] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:19:24 [I 2022-10-10 12:06:02] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00315 [I 2022-10-10 12:06:02] ...trap/engines/engine.py.230: loss: 0.15056 [I 2022-10-10 12:06:02] ...optimizers/trijoint.py.151: total_norm: 11.749343 [I 2022-10-10 12:06:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 950/2304 [I 2022-10-10 12:06:10] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:19:22 [I 2022-10-10 12:06:10] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00310 [I 2022-10-10 12:06:10] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-10 12:06:10] ...optimizers/trijoint.py.151: total_norm: 39.547783 [I 2022-10-10 12:06:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 960/2304 [I 2022-10-10 12:06:19] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:19:13 [I 2022-10-10 12:06:19] ...trap/engines/engine.py.229: process: 0.87548 | load: 0.00315 [I 2022-10-10 12:06:19] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-10 12:06:19] ...optimizers/trijoint.py.151: total_norm: 11.795322 [I 2022-10-10 12:06:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 970/2304 [I 2022-10-10 12:06:28] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:19:05 [I 2022-10-10 12:06:28] ...trap/engines/engine.py.229: process: 0.87023 | load: 0.00315 [I 2022-10-10 12:06:28] ...trap/engines/engine.py.230: loss: 0.10014 [I 2022-10-10 12:06:28] ...optimizers/trijoint.py.151: total_norm: 9.820743 [I 2022-10-10 12:06:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 980/2304 [I 2022-10-10 12:06:36] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:53 [I 2022-10-10 12:06:36] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00314 [I 2022-10-10 12:06:36] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-10 12:06:36] ...optimizers/trijoint.py.151: total_norm: 9.860073 [I 2022-10-10 12:06:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 990/2304 [I 2022-10-10 12:06:45] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:50 [I 2022-10-10 12:06:45] ...trap/engines/engine.py.229: process: 0.86752 | load: 0.00311 [I 2022-10-10 12:06:45] ...trap/engines/engine.py.230: loss: 0.17628 [I 2022-10-10 12:06:45] ...optimizers/trijoint.py.151: total_norm: 32.022877 [I 2022-10-10 12:06:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 1000/2304 [I 2022-10-10 12:06:53] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:39 [I 2022-10-10 12:06:53] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00313 [I 2022-10-10 12:06:53] ...trap/engines/engine.py.230: loss: 0.21876 [I 2022-10-10 12:06:53] ...optimizers/trijoint.py.151: total_norm: 84.248901 [I 2022-10-10 12:07:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 1010/2304 [I 2022-10-10 12:07:02] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:18:31 [I 2022-10-10 12:07:02] ...trap/engines/engine.py.229: process: 0.85679 | load: 0.00313 [I 2022-10-10 12:07:02] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-10 12:07:02] ...optimizers/trijoint.py.151: total_norm: 32.281490 [I 2022-10-10 12:07:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 1020/2304 [I 2022-10-10 12:07:11] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:29 [I 2022-10-10 12:07:11] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00315 [I 2022-10-10 12:07:11] ...trap/engines/engine.py.230: loss: 0.12639 [I 2022-10-10 12:07:11] ...optimizers/trijoint.py.151: total_norm: 8.798335 [I 2022-10-10 12:07:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 1030/2304 [I 2022-10-10 12:07:19] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:18:13 [I 2022-10-10 12:07:19] ...trap/engines/engine.py.229: process: 0.85548 | load: 0.00310 [I 2022-10-10 12:07:19] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-10 12:07:19] ...optimizers/trijoint.py.151: total_norm: 33.107304 [I 2022-10-10 12:07:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1040/2304 [I 2022-10-10 12:07:28] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:18:03 [I 2022-10-10 12:07:28] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00319 [I 2022-10-10 12:07:28] ...trap/engines/engine.py.230: loss: 0.20308 [I 2022-10-10 12:07:28] ...optimizers/trijoint.py.151: total_norm: 15.185564 [I 2022-10-10 12:07:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 1050/2304 [I 2022-10-10 12:07:37] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:18:07 [I 2022-10-10 12:07:37] ...trap/engines/engine.py.229: process: 0.88167 | load: 0.00311 [I 2022-10-10 12:07:37] ...trap/engines/engine.py.230: loss: 0.11223 [I 2022-10-10 12:07:37] ...optimizers/trijoint.py.151: total_norm: 13.909536 [I 2022-10-10 12:07:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 1060/2304 [I 2022-10-10 12:07:45] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:17:49 [I 2022-10-10 12:07:45] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00310 [I 2022-10-10 12:07:45] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-10 12:07:45] ...optimizers/trijoint.py.151: total_norm: 11.364885 [I 2022-10-10 12:07:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 1070/2304 [I 2022-10-10 12:07:54] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:28 [I 2022-10-10 12:07:54] ...trap/engines/engine.py.229: process: 0.84649 | load: 0.00297 [I 2022-10-10 12:07:54] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-10 12:07:54] ...optimizers/trijoint.py.151: total_norm: 14.636951 [I 2022-10-10 12:08:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 1080/2304 [I 2022-10-10 12:08:02] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:25 [I 2022-10-10 12:08:02] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00309 [I 2022-10-10 12:08:02] ...trap/engines/engine.py.230: loss: 0.14039 [I 2022-10-10 12:08:02] ...optimizers/trijoint.py.151: total_norm: 14.045931 [I 2022-10-10 12:08:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 1090/2304 [I 2022-10-10 12:08:11] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:09 [I 2022-10-10 12:08:11] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00304 [I 2022-10-10 12:08:11] ...trap/engines/engine.py.230: loss: 0.22149 [I 2022-10-10 12:08:11] ...optimizers/trijoint.py.151: total_norm: 15.279405 [I 2022-10-10 12:08:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 1100/2304 [I 2022-10-10 12:08:19] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:17:01 [I 2022-10-10 12:08:19] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00308 [I 2022-10-10 12:08:19] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-10 12:08:19] ...optimizers/trijoint.py.151: total_norm: 12.639114 [I 2022-10-10 12:08:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1110/2304 [I 2022-10-10 12:08:28] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:57 [I 2022-10-10 12:08:28] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00304 [I 2022-10-10 12:08:28] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-10 12:08:28] ...optimizers/trijoint.py.151: total_norm: 14.520491 [I 2022-10-10 12:08:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 1120/2304 [I 2022-10-10 12:08:37] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:53 [I 2022-10-10 12:08:37] ...trap/engines/engine.py.229: process: 0.85578 | load: 0.00296 [I 2022-10-10 12:08:37] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-10 12:08:37] ...optimizers/trijoint.py.151: total_norm: 13.577090 [I 2022-10-10 12:08:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 1130/2304 [I 2022-10-10 12:08:45] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:41 [I 2022-10-10 12:08:45] ...trap/engines/engine.py.229: process: 0.86121 | load: 0.00301 [I 2022-10-10 12:08:45] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-10 12:08:45] ...optimizers/trijoint.py.151: total_norm: 12.432199 [I 2022-10-10 12:08:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 1140/2304 [I 2022-10-10 12:08:54] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:24 [I 2022-10-10 12:08:54] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00299 [I 2022-10-10 12:08:54] ...trap/engines/engine.py.230: loss: 0.14315 [I 2022-10-10 12:08:54] ...optimizers/trijoint.py.151: total_norm: 17.816753 [I 2022-10-10 12:09:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 1150/2304 [I 2022-10-10 12:09:02] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:20 [I 2022-10-10 12:09:02] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00293 [I 2022-10-10 12:09:02] ...trap/engines/engine.py.230: loss: 0.17617 [I 2022-10-10 12:09:02] ...optimizers/trijoint.py.151: total_norm: 26.352747 [I 2022-10-10 12:09:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 1160/2304 [I 2022-10-10 12:09:11] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:15 [I 2022-10-10 12:09:11] ...trap/engines/engine.py.229: process: 0.84982 | load: 0.00296 [I 2022-10-10 12:09:11] ...trap/engines/engine.py.230: loss: 0.12274 [I 2022-10-10 12:09:11] ...optimizers/trijoint.py.151: total_norm: 21.389208 [I 2022-10-10 12:09:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 1170/2304 [I 2022-10-10 12:09:19] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:03 [I 2022-10-10 12:09:19] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00297 [I 2022-10-10 12:09:19] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-10 12:09:19] ...optimizers/trijoint.py.151: total_norm: 14.468377 [I 2022-10-10 12:09:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1180/2304 [I 2022-10-10 12:09:28] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:51 [I 2022-10-10 12:09:28] ...trap/engines/engine.py.229: process: 0.84081 | load: 0.00297 [I 2022-10-10 12:09:28] ...trap/engines/engine.py.230: loss: 0.07991 [I 2022-10-10 12:09:28] ...optimizers/trijoint.py.151: total_norm: 7.997409 [I 2022-10-10 12:09:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 1190/2304 [I 2022-10-10 12:09:36] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:49 [I 2022-10-10 12:09:36] ...trap/engines/engine.py.229: process: 0.85889 | load: 0.00304 [I 2022-10-10 12:09:36] ...trap/engines/engine.py.230: loss: 0.13352 [I 2022-10-10 12:09:36] ...optimizers/trijoint.py.151: total_norm: 19.823410 [I 2022-10-10 12:09:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 1200/2304 [I 2022-10-10 12:09:45] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:52 [I 2022-10-10 12:09:45] ...trap/engines/engine.py.229: process: 0.88522 | load: 0.00321 [I 2022-10-10 12:09:45] ...trap/engines/engine.py.230: loss: 0.16276 [I 2022-10-10 12:09:45] ...optimizers/trijoint.py.151: total_norm: 20.870913 [I 2022-10-10 12:10:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1210/2304 [I 2022-10-10 12:10:13] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:44:10 [I 2022-10-10 12:10:13] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00306 [I 2022-10-10 12:10:13] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-10 12:10:13] ...optimizers/trijoint.py.151: total_norm: 13.623248 [I 2022-10-10 12:10:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1220/2304 [I 2022-10-10 12:10:21] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:18:27 [I 2022-10-10 12:10:21] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00314 [I 2022-10-10 12:10:21] ...trap/engines/engine.py.230: loss: 0.14828 [I 2022-10-10 12:10:21] ...optimizers/trijoint.py.151: total_norm: 68.949524 [I 2022-10-10 12:10:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1230/2304 [I 2022-10-10 12:10:30] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:15:43 [I 2022-10-10 12:10:30] ...trap/engines/engine.py.229: process: 0.86949 | load: 0.00311 [I 2022-10-10 12:10:30] ...trap/engines/engine.py.230: loss: 0.19445 [I 2022-10-10 12:10:30] ...optimizers/trijoint.py.151: total_norm: 21.664047 [I 2022-10-10 12:10:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1240/2304 [I 2022-10-10 12:10:38] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:15:10 [I 2022-10-10 12:10:38] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00341 [I 2022-10-10 12:10:38] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-10 12:10:38] ...optimizers/trijoint.py.151: total_norm: 59.368839 [I 2022-10-10 12:10:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1250/2304 [I 2022-10-10 12:10:47] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:15:01 [I 2022-10-10 12:10:47] ...trap/engines/engine.py.229: process: 0.85571 | load: 0.00310 [I 2022-10-10 12:10:47] ...trap/engines/engine.py.230: loss: 0.14375 [I 2022-10-10 12:10:47] ...optimizers/trijoint.py.151: total_norm: 17.424467 [I 2022-10-10 12:10:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1260/2304 [I 2022-10-10 12:10:55] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:53 [I 2022-10-10 12:10:55] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00309 [I 2022-10-10 12:10:55] ...trap/engines/engine.py.230: loss: 0.17511 [I 2022-10-10 12:10:55] ...optimizers/trijoint.py.151: total_norm: 13.473859 [I 2022-10-10 12:11:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1270/2304 [I 2022-10-10 12:11:04] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:48 [I 2022-10-10 12:11:04] ...trap/engines/engine.py.229: process: 0.85620 | load: 0.00309 [I 2022-10-10 12:11:04] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-10 12:11:04] ...optimizers/trijoint.py.151: total_norm: 11.832090 [I 2022-10-10 12:11:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1280/2304 [I 2022-10-10 12:11:13] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:36 [I 2022-10-10 12:11:13] ...trap/engines/engine.py.229: process: 0.85606 | load: 0.00311 [I 2022-10-10 12:11:13] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-10 12:11:13] ...optimizers/trijoint.py.151: total_norm: 19.211231 [I 2022-10-10 12:11:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1290/2304 [I 2022-10-10 12:11:21] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:29 [I 2022-10-10 12:11:21] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00311 [I 2022-10-10 12:11:21] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-10 12:11:21] ...optimizers/trijoint.py.151: total_norm: 7.144125 [I 2022-10-10 12:11:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1300/2304 [I 2022-10-10 12:11:30] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:14:27 [I 2022-10-10 12:11:30] ...trap/engines/engine.py.229: process: 0.86551 | load: 0.00311 [I 2022-10-10 12:11:30] ...trap/engines/engine.py.230: loss: 0.15586 [I 2022-10-10 12:11:30] ...optimizers/trijoint.py.151: total_norm: 10.713785 [I 2022-10-10 12:11:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1310/2304 [I 2022-10-10 12:11:38] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:14:12 [I 2022-10-10 12:11:38] ...trap/engines/engine.py.229: process: 0.85960 | load: 0.00306 [I 2022-10-10 12:11:38] ...trap/engines/engine.py.230: loss: 0.13255 [I 2022-10-10 12:11:38] ...optimizers/trijoint.py.151: total_norm: 15.336325 [I 2022-10-10 12:11:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1320/2304 [I 2022-10-10 12:11:47] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:14:06 [I 2022-10-10 12:11:47] ...trap/engines/engine.py.229: process: 0.86735 | load: 0.00312 [I 2022-10-10 12:11:47] ...trap/engines/engine.py.230: loss: 0.11811 [I 2022-10-10 12:11:47] ...optimizers/trijoint.py.151: total_norm: 13.021024 [I 2022-10-10 12:11:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1330/2304 [I 2022-10-10 12:11:56] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:52 [I 2022-10-10 12:11:56] ...trap/engines/engine.py.229: process: 0.84435 | load: 0.00312 [I 2022-10-10 12:11:56] ...trap/engines/engine.py.230: loss: 0.18411 [I 2022-10-10 12:11:56] ...optimizers/trijoint.py.151: total_norm: 11.634433 [I 2022-10-10 12:12:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1340/2304 [I 2022-10-10 12:12:04] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:49 [I 2022-10-10 12:12:04] ...trap/engines/engine.py.229: process: 0.87781 | load: 0.00312 [I 2022-10-10 12:12:04] ...trap/engines/engine.py.230: loss: 0.11264 [I 2022-10-10 12:12:04] ...optimizers/trijoint.py.151: total_norm: 24.461887 [I 2022-10-10 12:12:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1350/2304 [I 2022-10-10 12:12:13] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:44 [I 2022-10-10 12:12:13] ...trap/engines/engine.py.229: process: 0.86966 | load: 0.00315 [I 2022-10-10 12:12:13] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-10 12:12:13] ...optimizers/trijoint.py.151: total_norm: 25.399736 [I 2022-10-10 12:12:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1360/2304 [I 2022-10-10 12:12:21] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:13:27 [I 2022-10-10 12:12:21] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00309 [I 2022-10-10 12:12:21] ...trap/engines/engine.py.230: loss: 0.13941 [I 2022-10-10 12:12:21] ...optimizers/trijoint.py.151: total_norm: 11.661023 [I 2022-10-10 12:12:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1370/2304 [I 2022-10-10 12:12:30] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:38 [I 2022-10-10 12:12:30] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00317 [I 2022-10-10 12:12:30] ...trap/engines/engine.py.230: loss: 0.26141 [I 2022-10-10 12:12:30] ...optimizers/trijoint.py.151: total_norm: 33.938385 [I 2022-10-10 12:12:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 1380/2304 [I 2022-10-10 12:12:39] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:13:18 [I 2022-10-10 12:12:39] ...trap/engines/engine.py.229: process: 0.87369 | load: 0.00307 [I 2022-10-10 12:12:39] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-10 12:12:39] ...optimizers/trijoint.py.151: total_norm: 15.572337 [I 2022-10-10 12:12:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 1390/2304 [I 2022-10-10 12:12:48] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:13:02 [I 2022-10-10 12:12:48] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00310 [I 2022-10-10 12:12:48] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-10 12:12:48] ...optimizers/trijoint.py.151: total_norm: 50.419834 [I 2022-10-10 12:12:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1400/2304 [I 2022-10-10 12:12:56] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:54 [I 2022-10-10 12:12:56] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00315 [I 2022-10-10 12:12:56] ...trap/engines/engine.py.230: loss: 0.13872 [I 2022-10-10 12:12:56] ...optimizers/trijoint.py.151: total_norm: 17.531271 [I 2022-10-10 12:13:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 1410/2304 [I 2022-10-10 12:13:05] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:49 [I 2022-10-10 12:13:05] ...trap/engines/engine.py.229: process: 0.88024 | load: 0.00324 [I 2022-10-10 12:13:05] ...trap/engines/engine.py.230: loss: 0.13292 [I 2022-10-10 12:13:05] ...optimizers/trijoint.py.151: total_norm: 19.143330 [I 2022-10-10 12:13:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1420/2304 [I 2022-10-10 12:13:13] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:40 [I 2022-10-10 12:13:13] ...trap/engines/engine.py.229: process: 0.86219 | load: 0.00322 [I 2022-10-10 12:13:13] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-10 12:13:13] ...optimizers/trijoint.py.151: total_norm: 39.035011 [I 2022-10-10 12:13:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 1430/2304 [I 2022-10-10 12:13:22] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:12:32 [I 2022-10-10 12:13:22] ...trap/engines/engine.py.229: process: 0.85848 | load: 0.00307 [I 2022-10-10 12:13:22] ...trap/engines/engine.py.230: loss: 0.11104 [I 2022-10-10 12:13:22] ...optimizers/trijoint.py.151: total_norm: 11.642567 [I 2022-10-10 12:13:31] ...trap/engines/engine.py.224: train: epoch 71 | batch 1440/2304 [I 2022-10-10 12:13:31] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:12:22 [I 2022-10-10 12:13:31] ...trap/engines/engine.py.229: process: 0.86361 | load: 0.00305 [I 2022-10-10 12:13:31] ...trap/engines/engine.py.230: loss: 0.13569 [I 2022-10-10 12:13:31] ...optimizers/trijoint.py.151: total_norm: 21.966131 [I 2022-10-10 12:13:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 1450/2304 [I 2022-10-10 12:13:39] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:12:13 [I 2022-10-10 12:13:39] ...trap/engines/engine.py.229: process: 0.86878 | load: 0.00297 [I 2022-10-10 12:13:39] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-10 12:13:39] ...optimizers/trijoint.py.151: total_norm: 11.267826 [I 2022-10-10 12:13:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 1460/2304 [I 2022-10-10 12:13:48] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:12:03 [I 2022-10-10 12:13:48] ...trap/engines/engine.py.229: process: 0.86431 | load: 0.00300 [I 2022-10-10 12:13:48] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-10 12:13:48] ...optimizers/trijoint.py.151: total_norm: 10.360936 [I 2022-10-10 12:13:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1470/2304 [I 2022-10-10 12:13:56] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:44 [I 2022-10-10 12:13:56] ...trap/engines/engine.py.229: process: 0.83933 | load: 0.00301 [I 2022-10-10 12:13:56] ...trap/engines/engine.py.230: loss: 0.13578 [I 2022-10-10 12:13:56] ...optimizers/trijoint.py.151: total_norm: 18.424488 [I 2022-10-10 12:14:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 1480/2304 [I 2022-10-10 12:14:05] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:41 [I 2022-10-10 12:14:05] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00294 [I 2022-10-10 12:14:05] ...trap/engines/engine.py.230: loss: 0.09703 [I 2022-10-10 12:14:05] ...optimizers/trijoint.py.151: total_norm: 14.647911 [I 2022-10-10 12:14:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1490/2304 [I 2022-10-10 12:14:13] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:33 [I 2022-10-10 12:14:13] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00298 [I 2022-10-10 12:14:13] ...trap/engines/engine.py.230: loss: 0.10752 [I 2022-10-10 12:14:13] ...optimizers/trijoint.py.151: total_norm: 6.239885 [I 2022-10-10 12:14:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 1500/2304 [I 2022-10-10 12:14:22] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:11:20 [I 2022-10-10 12:14:22] ...trap/engines/engine.py.229: process: 0.83719 | load: 0.00293 [I 2022-10-10 12:14:22] ...trap/engines/engine.py.230: loss: 0.16992 [I 2022-10-10 12:14:22] ...optimizers/trijoint.py.151: total_norm: 17.842268 [I 2022-10-10 12:14:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1510/2304 [I 2022-10-10 12:14:30] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:11:13 [I 2022-10-10 12:14:30] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00292 [I 2022-10-10 12:14:30] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-10 12:14:30] ...optimizers/trijoint.py.151: total_norm: 14.868122 [I 2022-10-10 12:14:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 1520/2304 [I 2022-10-10 12:14:39] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:11:06 [I 2022-10-10 12:14:39] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00290 [I 2022-10-10 12:14:39] ...trap/engines/engine.py.230: loss: 0.17746 [I 2022-10-10 12:14:39] ...optimizers/trijoint.py.151: total_norm: 15.707349 [I 2022-10-10 12:14:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1530/2304 [I 2022-10-10 12:14:47] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:53 [I 2022-10-10 12:14:47] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00294 [I 2022-10-10 12:14:47] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-10 12:14:47] ...optimizers/trijoint.py.151: total_norm: 14.834721 [I 2022-10-10 12:14:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1540/2304 [I 2022-10-10 12:14:56] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:41 [I 2022-10-10 12:14:56] ...trap/engines/engine.py.229: process: 0.82958 | load: 0.00293 [I 2022-10-10 12:14:56] ...trap/engines/engine.py.230: loss: 0.16589 [I 2022-10-10 12:14:56] ...optimizers/trijoint.py.151: total_norm: 25.616285 [I 2022-10-10 12:15:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1550/2304 [I 2022-10-10 12:15:04] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:34 [I 2022-10-10 12:15:04] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00294 [I 2022-10-10 12:15:04] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-10 12:15:04] ...optimizers/trijoint.py.151: total_norm: 7.797618 [I 2022-10-10 12:15:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1560/2304 [I 2022-10-10 12:15:13] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:32 [I 2022-10-10 12:15:13] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00293 [I 2022-10-10 12:15:13] ...trap/engines/engine.py.230: loss: 0.18738 [I 2022-10-10 12:15:13] ...optimizers/trijoint.py.151: total_norm: 11.953975 [I 2022-10-10 12:15:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1570/2304 [I 2022-10-10 12:15:21] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:25 [I 2022-10-10 12:15:21] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00293 [I 2022-10-10 12:15:21] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-10 12:15:21] ...optimizers/trijoint.py.151: total_norm: 15.531122 [I 2022-10-10 12:15:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1580/2304 [I 2022-10-10 12:15:30] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:14 [I 2022-10-10 12:15:30] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00294 [I 2022-10-10 12:15:30] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-10 12:15:30] ...optimizers/trijoint.py.151: total_norm: 13.191573 [I 2022-10-10 12:15:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1590/2304 [I 2022-10-10 12:15:38] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:08 [I 2022-10-10 12:15:38] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00292 [I 2022-10-10 12:15:38] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-10 12:15:38] ...optimizers/trijoint.py.151: total_norm: 20.277147 [I 2022-10-10 12:15:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1600/2304 [I 2022-10-10 12:15:47] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:58 [I 2022-10-10 12:15:47] ...trap/engines/engine.py.229: process: 0.84014 | load: 0.00301 [I 2022-10-10 12:15:47] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-10 12:15:47] ...optimizers/trijoint.py.151: total_norm: 35.935532 [I 2022-10-10 12:15:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1610/2304 [I 2022-10-10 12:15:55] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:49 [I 2022-10-10 12:15:55] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00297 [I 2022-10-10 12:15:55] ...trap/engines/engine.py.230: loss: 0.18347 [I 2022-10-10 12:15:55] ...optimizers/trijoint.py.151: total_norm: 18.969572 [I 2022-10-10 12:16:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1620/2304 [I 2022-10-10 12:16:04] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:43 [I 2022-10-10 12:16:04] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00294 [I 2022-10-10 12:16:04] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-10 12:16:04] ...optimizers/trijoint.py.151: total_norm: 10.167696 [I 2022-10-10 12:16:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 1630/2304 [I 2022-10-10 12:16:12] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:33 [I 2022-10-10 12:16:12] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00296 [I 2022-10-10 12:16:12] ...trap/engines/engine.py.230: loss: 0.09645 [I 2022-10-10 12:16:12] ...optimizers/trijoint.py.151: total_norm: 9.330535 [I 2022-10-10 12:16:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1640/2304 [I 2022-10-10 12:16:21] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:19 [I 2022-10-10 12:16:21] ...trap/engines/engine.py.229: process: 0.83870 | load: 0.00300 [I 2022-10-10 12:16:21] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-10 12:16:21] ...optimizers/trijoint.py.151: total_norm: 11.098426 [I 2022-10-10 12:16:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 1650/2304 [I 2022-10-10 12:16:29] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:15 [I 2022-10-10 12:16:29] ...trap/engines/engine.py.229: process: 0.84975 | load: 0.00308 [I 2022-10-10 12:16:29] ...trap/engines/engine.py.230: loss: 0.10329 [I 2022-10-10 12:16:29] ...optimizers/trijoint.py.151: total_norm: 9.110579 [I 2022-10-10 12:16:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1660/2304 [I 2022-10-10 12:16:38] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:09:04 [I 2022-10-10 12:16:38] ...trap/engines/engine.py.229: process: 0.84786 | load: 0.00299 [I 2022-10-10 12:16:38] ...trap/engines/engine.py.230: loss: 0.17316 [I 2022-10-10 12:16:38] ...optimizers/trijoint.py.151: total_norm: 14.143929 [I 2022-10-10 12:16:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 1670/2304 [I 2022-10-10 12:16:46] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:55 [I 2022-10-10 12:16:46] ...trap/engines/engine.py.229: process: 0.83291 | load: 0.00295 [I 2022-10-10 12:16:46] ...trap/engines/engine.py.230: loss: 0.09847 [I 2022-10-10 12:16:46] ...optimizers/trijoint.py.151: total_norm: 9.454592 [I 2022-10-10 12:16:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1680/2304 [I 2022-10-10 12:16:55] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:46 [I 2022-10-10 12:16:55] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00302 [I 2022-10-10 12:16:55] ...trap/engines/engine.py.230: loss: 0.17100 [I 2022-10-10 12:16:55] ...optimizers/trijoint.py.151: total_norm: 18.049728 [I 2022-10-10 12:17:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 1690/2304 [I 2022-10-10 12:17:03] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:37 [I 2022-10-10 12:17:03] ...trap/engines/engine.py.229: process: 0.83561 | load: 0.00301 [I 2022-10-10 12:17:03] ...trap/engines/engine.py.230: loss: 0.18289 [I 2022-10-10 12:17:03] ...optimizers/trijoint.py.151: total_norm: 18.133909 [I 2022-10-10 12:17:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 1700/2304 [I 2022-10-10 12:17:12] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:30 [I 2022-10-10 12:17:12] ...trap/engines/engine.py.229: process: 0.83654 | load: 0.00302 [I 2022-10-10 12:17:12] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-10 12:17:12] ...optimizers/trijoint.py.151: total_norm: 44.113789 [I 2022-10-10 12:17:20] ...trap/engines/engine.py.224: train: epoch 71 | batch 1710/2304 [I 2022-10-10 12:17:20] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:28 [I 2022-10-10 12:17:20] ...trap/engines/engine.py.229: process: 0.86682 | load: 0.00309 [I 2022-10-10 12:17:20] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-10 12:17:20] ...optimizers/trijoint.py.151: total_norm: 22.133898 [I 2022-10-10 12:17:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 1720/2304 [I 2022-10-10 12:17:29] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:19 [I 2022-10-10 12:17:29] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00332 [I 2022-10-10 12:17:29] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-10 12:17:29] ...optimizers/trijoint.py.151: total_norm: 11.207777 [I 2022-10-10 12:17:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1730/2304 [I 2022-10-10 12:17:38] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:16 [I 2022-10-10 12:17:38] ...trap/engines/engine.py.229: process: 0.86608 | load: 0.00322 [I 2022-10-10 12:17:38] ...trap/engines/engine.py.230: loss: 0.11329 [I 2022-10-10 12:17:38] ...optimizers/trijoint.py.151: total_norm: 11.036170 [I 2022-10-10 12:17:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 1740/2304 [I 2022-10-10 12:17:46] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:08:07 [I 2022-10-10 12:17:46] ...trap/engines/engine.py.229: process: 0.86244 | load: 0.00331 [I 2022-10-10 12:17:46] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-10 12:17:46] ...optimizers/trijoint.py.151: total_norm: 8.027411 [I 2022-10-10 12:17:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1750/2304 [I 2022-10-10 12:17:55] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:58 [I 2022-10-10 12:17:55] ...trap/engines/engine.py.229: process: 0.86616 | load: 0.00310 [I 2022-10-10 12:17:55] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-10 12:17:55] ...optimizers/trijoint.py.151: total_norm: 26.618885 [I 2022-10-10 12:18:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1760/2304 [I 2022-10-10 12:18:04] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:47 [I 2022-10-10 12:18:04] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00329 [I 2022-10-10 12:18:04] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-10 12:18:04] ...optimizers/trijoint.py.151: total_norm: 37.237301 [I 2022-10-10 12:18:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 1770/2304 [I 2022-10-10 12:18:12] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:38 [I 2022-10-10 12:18:12] ...trap/engines/engine.py.229: process: 0.86982 | load: 0.00312 [I 2022-10-10 12:18:12] ...trap/engines/engine.py.230: loss: 0.15847 [I 2022-10-10 12:18:12] ...optimizers/trijoint.py.151: total_norm: 15.218010 [I 2022-10-10 12:18:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1780/2304 [I 2022-10-10 12:18:21] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:31 [I 2022-10-10 12:18:21] ...trap/engines/engine.py.229: process: 0.86743 | load: 0.00312 [I 2022-10-10 12:18:21] ...trap/engines/engine.py.230: loss: 0.11401 [I 2022-10-10 12:18:21] ...optimizers/trijoint.py.151: total_norm: 27.846313 [I 2022-10-10 12:18:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 1790/2304 [I 2022-10-10 12:18:29] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:22 [I 2022-10-10 12:18:29] ...trap/engines/engine.py.229: process: 0.87807 | load: 0.00315 [I 2022-10-10 12:18:29] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-10 12:18:29] ...optimizers/trijoint.py.151: total_norm: 12.903389 [I 2022-10-10 12:18:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1800/2304 [I 2022-10-10 12:18:38] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:13 [I 2022-10-10 12:18:38] ...trap/engines/engine.py.229: process: 0.86169 | load: 0.00310 [I 2022-10-10 12:18:38] ...trap/engines/engine.py.230: loss: 0.18685 [I 2022-10-10 12:18:38] ...optimizers/trijoint.py.151: total_norm: 14.338317 [I 2022-10-10 12:18:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1810/2304 [I 2022-10-10 12:18:47] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:07:02 [I 2022-10-10 12:18:47] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00310 [I 2022-10-10 12:18:47] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-10 12:18:47] ...optimizers/trijoint.py.151: total_norm: 17.492722 [I 2022-10-10 12:18:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1820/2304 [I 2022-10-10 12:18:55] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:55 [I 2022-10-10 12:18:55] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00395 [I 2022-10-10 12:18:55] ...trap/engines/engine.py.230: loss: 0.10711 [I 2022-10-10 12:18:55] ...optimizers/trijoint.py.151: total_norm: 14.570672 [I 2022-10-10 12:19:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1830/2304 [I 2022-10-10 12:19:04] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:44 [I 2022-10-10 12:19:04] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00316 [I 2022-10-10 12:19:04] ...trap/engines/engine.py.230: loss: 0.18892 [I 2022-10-10 12:19:04] ...optimizers/trijoint.py.151: total_norm: 35.978371 [I 2022-10-10 12:19:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 1840/2304 [I 2022-10-10 12:19:12] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:36 [I 2022-10-10 12:19:12] ...trap/engines/engine.py.229: process: 0.85865 | load: 0.00311 [I 2022-10-10 12:19:12] ...trap/engines/engine.py.230: loss: 0.26184 [I 2022-10-10 12:19:12] ...optimizers/trijoint.py.151: total_norm: 16.988358 [I 2022-10-10 12:19:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1850/2304 [I 2022-10-10 12:19:21] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:30 [I 2022-10-10 12:19:21] ...trap/engines/engine.py.229: process: 0.86472 | load: 0.00309 [I 2022-10-10 12:19:21] ...trap/engines/engine.py.230: loss: 0.14177 [I 2022-10-10 12:19:21] ...optimizers/trijoint.py.151: total_norm: 13.823316 [I 2022-10-10 12:19:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1860/2304 [I 2022-10-10 12:19:30] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:21 [I 2022-10-10 12:19:30] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00315 [I 2022-10-10 12:19:30] ...trap/engines/engine.py.230: loss: 0.13726 [I 2022-10-10 12:19:30] ...optimizers/trijoint.py.151: total_norm: 9.371553 [I 2022-10-10 12:19:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1870/2304 [I 2022-10-10 12:19:38] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:13 [I 2022-10-10 12:19:38] ...trap/engines/engine.py.229: process: 0.85352 | load: 0.00311 [I 2022-10-10 12:19:38] ...trap/engines/engine.py.230: loss: 0.12489 [I 2022-10-10 12:19:38] ...optimizers/trijoint.py.151: total_norm: 16.085983 [I 2022-10-10 12:19:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1880/2304 [I 2022-10-10 12:19:47] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:06:04 [I 2022-10-10 12:19:47] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00316 [I 2022-10-10 12:19:47] ...trap/engines/engine.py.230: loss: 0.12819 [I 2022-10-10 12:19:47] ...optimizers/trijoint.py.151: total_norm: 21.363132 [I 2022-10-10 12:19:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1890/2304 [I 2022-10-10 12:19:55] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:56 [I 2022-10-10 12:19:55] ...trap/engines/engine.py.229: process: 0.87117 | load: 0.00307 [I 2022-10-10 12:19:55] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-10 12:19:55] ...optimizers/trijoint.py.151: total_norm: 9.615849 [I 2022-10-10 12:20:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1900/2304 [I 2022-10-10 12:20:04] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:45 [I 2022-10-10 12:20:04] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00315 [I 2022-10-10 12:20:04] ...trap/engines/engine.py.230: loss: 0.09607 [I 2022-10-10 12:20:04] ...optimizers/trijoint.py.151: total_norm: 9.088011 [I 2022-10-10 12:20:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1910/2304 [I 2022-10-10 12:20:13] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:38 [I 2022-10-10 12:20:13] ...trap/engines/engine.py.229: process: 0.85597 | load: 0.00315 [I 2022-10-10 12:20:13] ...trap/engines/engine.py.230: loss: 0.17003 [I 2022-10-10 12:20:13] ...optimizers/trijoint.py.151: total_norm: 27.291584 [I 2022-10-10 12:20:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1920/2304 [I 2022-10-10 12:20:21] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:30 [I 2022-10-10 12:20:21] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00320 [I 2022-10-10 12:20:21] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-10 12:20:21] ...optimizers/trijoint.py.151: total_norm: 13.577154 [I 2022-10-10 12:20:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1930/2304 [I 2022-10-10 12:20:30] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:24 [I 2022-10-10 12:20:30] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00316 [I 2022-10-10 12:20:30] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-10 12:20:30] ...optimizers/trijoint.py.151: total_norm: 11.323821 [I 2022-10-10 12:20:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1940/2304 [I 2022-10-10 12:20:38] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:05:13 [I 2022-10-10 12:20:38] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00314 [I 2022-10-10 12:20:38] ...trap/engines/engine.py.230: loss: 0.14678 [I 2022-10-10 12:20:38] ...optimizers/trijoint.py.151: total_norm: 21.732592 [I 2022-10-10 12:20:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1950/2304 [I 2022-10-10 12:20:47] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:05:04 [I 2022-10-10 12:20:47] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00306 [I 2022-10-10 12:20:47] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-10 12:20:47] ...optimizers/trijoint.py.151: total_norm: 34.459602 [I 2022-10-10 12:20:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1960/2304 [I 2022-10-10 12:20:56] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:53 [I 2022-10-10 12:20:56] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00324 [I 2022-10-10 12:20:56] ...trap/engines/engine.py.230: loss: 0.12633 [I 2022-10-10 12:20:56] ...optimizers/trijoint.py.151: total_norm: 20.393833 [I 2022-10-10 12:21:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1970/2304 [I 2022-10-10 12:21:04] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:44 [I 2022-10-10 12:21:04] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00307 [I 2022-10-10 12:21:04] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-10 12:21:04] ...optimizers/trijoint.py.151: total_norm: 21.761745 [I 2022-10-10 12:21:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1980/2304 [I 2022-10-10 12:21:13] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:38 [I 2022-10-10 12:21:13] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00309 [I 2022-10-10 12:21:13] ...trap/engines/engine.py.230: loss: 0.09703 [I 2022-10-10 12:21:13] ...optimizers/trijoint.py.151: total_norm: 10.701837 [I 2022-10-10 12:21:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1990/2304 [I 2022-10-10 12:21:21] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:04:28 [I 2022-10-10 12:21:21] ...trap/engines/engine.py.229: process: 0.85024 | load: 0.00312 [I 2022-10-10 12:21:21] ...trap/engines/engine.py.230: loss: 0.13871 [I 2022-10-10 12:21:21] ...optimizers/trijoint.py.151: total_norm: 13.178573 [I 2022-10-10 12:21:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 2000/2304 [I 2022-10-10 12:21:30] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:19 [I 2022-10-10 12:21:30] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00300 [I 2022-10-10 12:21:30] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-10 12:21:30] ...optimizers/trijoint.py.151: total_norm: 14.133053 [I 2022-10-10 12:21:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 2010/2304 [I 2022-10-10 12:21:39] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:09 [I 2022-10-10 12:21:39] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00301 [I 2022-10-10 12:21:39] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-10 12:21:39] ...optimizers/trijoint.py.151: total_norm: 23.266788 [I 2022-10-10 12:21:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 2020/2304 [I 2022-10-10 12:21:47] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:04:02 [I 2022-10-10 12:21:47] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00296 [I 2022-10-10 12:21:47] ...trap/engines/engine.py.230: loss: 0.14713 [I 2022-10-10 12:21:47] ...optimizers/trijoint.py.151: total_norm: 24.823284 [I 2022-10-10 12:21:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 2030/2304 [I 2022-10-10 12:21:56] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:52 [I 2022-10-10 12:21:56] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00298 [I 2022-10-10 12:21:56] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-10 12:21:56] ...optimizers/trijoint.py.151: total_norm: 14.816369 [I 2022-10-10 12:22:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 2040/2304 [I 2022-10-10 12:22:04] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:43 [I 2022-10-10 12:22:04] ...trap/engines/engine.py.229: process: 0.86090 | load: 0.00305 [I 2022-10-10 12:22:04] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-10 12:22:04] ...optimizers/trijoint.py.151: total_norm: 11.714991 [I 2022-10-10 12:22:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 2050/2304 [I 2022-10-10 12:22:13] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:36 [I 2022-10-10 12:22:13] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00303 [I 2022-10-10 12:22:13] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-10 12:22:13] ...optimizers/trijoint.py.151: total_norm: 16.232891 [I 2022-10-10 12:22:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 2060/2304 [I 2022-10-10 12:22:21] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:27 [I 2022-10-10 12:22:21] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00306 [I 2022-10-10 12:22:21] ...trap/engines/engine.py.230: loss: 0.09777 [I 2022-10-10 12:22:21] ...optimizers/trijoint.py.151: total_norm: 10.775611 [I 2022-10-10 12:22:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 2070/2304 [I 2022-10-10 12:22:30] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:18 [I 2022-10-10 12:22:30] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00303 [I 2022-10-10 12:22:30] ...trap/engines/engine.py.230: loss: 0.12019 [I 2022-10-10 12:22:30] ...optimizers/trijoint.py.151: total_norm: 11.278690 [I 2022-10-10 12:22:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 2080/2304 [I 2022-10-10 12:22:38] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:10 [I 2022-10-10 12:22:38] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00299 [I 2022-10-10 12:22:38] ...trap/engines/engine.py.230: loss: 0.15818 [I 2022-10-10 12:22:38] ...optimizers/trijoint.py.151: total_norm: 42.708969 [I 2022-10-10 12:22:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 2090/2304 [I 2022-10-10 12:22:47] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:03:01 [I 2022-10-10 12:22:47] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00292 [I 2022-10-10 12:22:47] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-10 12:22:47] ...optimizers/trijoint.py.151: total_norm: 17.835764 [I 2022-10-10 12:22:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 2100/2304 [I 2022-10-10 12:22:55] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:54 [I 2022-10-10 12:22:55] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00299 [I 2022-10-10 12:22:55] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-10 12:22:55] ...optimizers/trijoint.py.151: total_norm: 14.530132 [I 2022-10-10 12:23:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 2110/2304 [I 2022-10-10 12:23:04] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:45 [I 2022-10-10 12:23:04] ...trap/engines/engine.py.229: process: 0.86493 | load: 0.00297 [I 2022-10-10 12:23:04] ...trap/engines/engine.py.230: loss: 0.12268 [I 2022-10-10 12:23:04] ...optimizers/trijoint.py.151: total_norm: 25.994141 [I 2022-10-10 12:23:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 2120/2304 [I 2022-10-10 12:23:12] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:38 [I 2022-10-10 12:23:12] ...trap/engines/engine.py.229: process: 0.86840 | load: 0.00301 [I 2022-10-10 12:23:12] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-10 12:23:12] ...optimizers/trijoint.py.151: total_norm: 11.792967 [I 2022-10-10 12:23:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 2130/2304 [I 2022-10-10 12:23:21] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:29 [I 2022-10-10 12:23:21] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00296 [I 2022-10-10 12:23:21] ...trap/engines/engine.py.230: loss: 0.10457 [I 2022-10-10 12:23:21] ...optimizers/trijoint.py.151: total_norm: 19.119417 [I 2022-10-10 12:23:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 2140/2304 [I 2022-10-10 12:23:30] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:20 [I 2022-10-10 12:23:30] ...trap/engines/engine.py.229: process: 0.85784 | load: 0.00302 [I 2022-10-10 12:23:30] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-10 12:23:30] ...optimizers/trijoint.py.151: total_norm: 45.430462 [I 2022-10-10 12:23:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 2150/2304 [I 2022-10-10 12:23:38] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:10 [I 2022-10-10 12:23:38] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00301 [I 2022-10-10 12:23:38] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-10 12:23:38] ...optimizers/trijoint.py.151: total_norm: 9.673267 [I 2022-10-10 12:23:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 2160/2304 [I 2022-10-10 12:23:47] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:02:02 [I 2022-10-10 12:23:47] ...trap/engines/engine.py.229: process: 0.87589 | load: 0.00296 [I 2022-10-10 12:23:47] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-10 12:23:47] ...optimizers/trijoint.py.151: total_norm: 13.580855 [I 2022-10-10 12:23:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 2170/2304 [I 2022-10-10 12:23:55] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:53 [I 2022-10-10 12:23:55] ...trap/engines/engine.py.229: process: 0.83475 | load: 0.00306 [I 2022-10-10 12:23:55] ...trap/engines/engine.py.230: loss: 0.20180 [I 2022-10-10 12:23:55] ...optimizers/trijoint.py.151: total_norm: 25.837635 [I 2022-10-10 12:24:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 2180/2304 [I 2022-10-10 12:24:04] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:45 [I 2022-10-10 12:24:04] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00304 [I 2022-10-10 12:24:04] ...trap/engines/engine.py.230: loss: 0.22582 [I 2022-10-10 12:24:04] ...optimizers/trijoint.py.151: total_norm: 45.688126 [I 2022-10-10 12:24:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 2190/2304 [I 2022-10-10 12:24:12] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:37 [I 2022-10-10 12:24:12] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00308 [I 2022-10-10 12:24:12] ...trap/engines/engine.py.230: loss: 0.09657 [I 2022-10-10 12:24:12] ...optimizers/trijoint.py.151: total_norm: 9.239965 [I 2022-10-10 12:24:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 2200/2304 [I 2022-10-10 12:24:21] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:28 [I 2022-10-10 12:24:21] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00308 [I 2022-10-10 12:24:21] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-10 12:24:21] ...optimizers/trijoint.py.151: total_norm: 12.758791 [I 2022-10-10 12:24:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 2210/2304 [I 2022-10-10 12:24:29] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:20 [I 2022-10-10 12:24:29] ...trap/engines/engine.py.229: process: 0.85072 | load: 0.00307 [I 2022-10-10 12:24:29] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-10 12:24:29] ...optimizers/trijoint.py.151: total_norm: 12.534511 [I 2022-10-10 12:24:40] ...trap/engines/engine.py.224: train: epoch 71 | batch 2220/2304 [I 2022-10-10 12:24:40] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:24 [I 2022-10-10 12:24:40] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00316 [I 2022-10-10 12:24:40] ...trap/engines/engine.py.230: loss: 0.23754 [I 2022-10-10 12:24:40] ...optimizers/trijoint.py.151: total_norm: 19.623678 [I 2022-10-10 12:24:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 2230/2304 [I 2022-10-10 12:24:48] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:01:04 [I 2022-10-10 12:24:48] ...trap/engines/engine.py.229: process: 0.87210 | load: 0.00310 [I 2022-10-10 12:24:48] ...trap/engines/engine.py.230: loss: 0.13936 [I 2022-10-10 12:24:48] ...optimizers/trijoint.py.151: total_norm: 19.649027 [I 2022-10-10 12:24:57] ...trap/engines/engine.py.224: train: epoch 71 | batch 2240/2304 [I 2022-10-10 12:24:57] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:54 [I 2022-10-10 12:24:57] ...trap/engines/engine.py.229: process: 0.85149 | load: 0.00308 [I 2022-10-10 12:24:57] ...trap/engines/engine.py.230: loss: 0.18330 [I 2022-10-10 12:24:57] ...optimizers/trijoint.py.151: total_norm: 23.622593 [I 2022-10-10 12:25:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 2250/2304 [I 2022-10-10 12:25:05] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:46 [I 2022-10-10 12:25:05] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00307 [I 2022-10-10 12:25:05] ...trap/engines/engine.py.230: loss: 0.09460 [I 2022-10-10 12:25:05] ...optimizers/trijoint.py.151: total_norm: 16.347414 [I 2022-10-10 12:25:14] ...trap/engines/engine.py.224: train: epoch 71 | batch 2260/2304 [I 2022-10-10 12:25:14] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:37 [I 2022-10-10 12:25:14] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00310 [I 2022-10-10 12:25:14] ...trap/engines/engine.py.230: loss: 0.08572 [I 2022-10-10 12:25:14] ...optimizers/trijoint.py.151: total_norm: 9.043281 [I 2022-10-10 12:25:23] ...trap/engines/engine.py.224: train: epoch 71 | batch 2270/2304 [I 2022-10-10 12:25:23] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:29 [I 2022-10-10 12:25:23] ...trap/engines/engine.py.229: process: 0.86486 | load: 0.00307 [I 2022-10-10 12:25:23] ...trap/engines/engine.py.230: loss: 0.12608 [I 2022-10-10 12:25:23] ...optimizers/trijoint.py.151: total_norm: 22.134613 [I 2022-10-10 12:25:31] ...trap/engines/engine.py.224: train: epoch 71 | batch 2280/2304 [I 2022-10-10 12:25:31] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:20 [I 2022-10-10 12:25:31] ...trap/engines/engine.py.229: process: 0.86886 | load: 0.00306 [I 2022-10-10 12:25:31] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-10 12:25:31] ...optimizers/trijoint.py.151: total_norm: 9.996307 [I 2022-10-10 12:25:40] ...trap/engines/engine.py.224: train: epoch 71 | batch 2290/2304 [I 2022-10-10 12:25:40] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:11 [I 2022-10-10 12:25:40] ...trap/engines/engine.py.229: process: 0.85250 | load: 0.00311 [I 2022-10-10 12:25:40] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-10 12:25:40] ...optimizers/trijoint.py.151: total_norm: 26.350758 [I 2022-10-10 12:25:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 2300/2304 [I 2022-10-10 12:25:48] ...trap/engines/engine.py.228: elapsed: 0:33:16 | left: 0:00:03 [I 2022-10-10 12:25:48] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00018 [I 2022-10-10 12:25:48] ...trap/engines/engine.py.230: loss: 0.13561 [I 2022-10-10 12:25:48] ...optimizers/trijoint.py.151: total_norm: 9.807516 [I 2022-10-10 12:25:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 2304/2304 [I 2022-10-10 12:25:52] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:00 [I 2022-10-10 12:25:52] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00020 [I 2022-10-10 12:25:52] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-10 12:25:52] ...optimizers/trijoint.py.151: total_norm: 73.064026 [S 2022-10-10 12:25:53] ...trap/engines/engine.py.129: train_epoch.epoch: 71 [S 2022-10-10 12:25:53] ...trap/engines/engine.py.129: train_epoch.loss: 0.13792460197158593 [I 2022-10-10 12:25:53] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 12:25:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 71 [I 2022-10-10 12:25:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 12:25:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 12:25:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 12:25:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 12:25:59] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 12:25:59] ...trap/engines/engine.py.321: eval: epoch 71 | batch 0/510 [I 2022-10-10 12:25:59] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:42:24 [I 2022-10-10 12:25:59] ...trap/engines/engine.py.326: process: 4.98980 | load: 3.45896 [I 2022-10-10 12:26:05] ...trap/engines/engine.py.321: eval: epoch 71 | batch 10/510 [I 2022-10-10 12:26:05] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:40 [I 2022-10-10 12:26:05] ...trap/engines/engine.py.326: process: 0.56526 | load: 0.00020 [I 2022-10-10 12:26:11] ...trap/engines/engine.py.321: eval: epoch 71 | batch 20/510 [I 2022-10-10 12:26:11] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:08 [I 2022-10-10 12:26:11] ...trap/engines/engine.py.326: process: 0.68173 | load: 0.00022 [I 2022-10-10 12:26:17] ...trap/engines/engine.py.321: eval: epoch 71 | batch 30/510 [I 2022-10-10 12:26:17] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:05:19 [I 2022-10-10 12:26:17] ...trap/engines/engine.py.326: process: 0.62216 | load: 0.00020 [I 2022-10-10 12:26:23] ...trap/engines/engine.py.321: eval: epoch 71 | batch 40/510 [I 2022-10-10 12:26:23] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:04:37 [I 2022-10-10 12:26:23] ...trap/engines/engine.py.326: process: 0.54682 | load: 0.00020 [I 2022-10-10 12:26:28] ...trap/engines/engine.py.321: eval: epoch 71 | batch 50/510 [I 2022-10-10 12:26:28] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:55 [I 2022-10-10 12:26:28] ...trap/engines/engine.py.326: process: 0.50712 | load: 0.00020 [I 2022-10-10 12:26:33] ...trap/engines/engine.py.321: eval: epoch 71 | batch 60/510 [I 2022-10-10 12:26:33] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:48 [I 2022-10-10 12:26:33] ...trap/engines/engine.py.326: process: 0.47532 | load: 0.00020 [I 2022-10-10 12:26:38] ...trap/engines/engine.py.321: eval: epoch 71 | batch 70/510 [I 2022-10-10 12:26:38] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:53 [I 2022-10-10 12:26:38] ...trap/engines/engine.py.326: process: 0.52385 | load: 0.00020 [I 2022-10-10 12:26:43] ...trap/engines/engine.py.321: eval: epoch 71 | batch 80/510 [I 2022-10-10 12:26:43] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:33 [I 2022-10-10 12:26:43] ...trap/engines/engine.py.326: process: 0.45776 | load: 0.00019 [I 2022-10-10 12:26:48] ...trap/engines/engine.py.321: eval: epoch 71 | batch 90/510 [I 2022-10-10 12:26:48] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:16 [I 2022-10-10 12:26:48] ...trap/engines/engine.py.326: process: 0.47640 | load: 0.00021 [I 2022-10-10 12:26:52] ...trap/engines/engine.py.321: eval: epoch 71 | batch 100/510 [I 2022-10-10 12:26:52] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:02 [I 2022-10-10 12:26:52] ...trap/engines/engine.py.326: process: 0.31610 | load: 0.00020 [I 2022-10-10 12:26:56] ...trap/engines/engine.py.321: eval: epoch 71 | batch 110/510 [I 2022-10-10 12:26:56] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:14 [I 2022-10-10 12:26:56] ...trap/engines/engine.py.326: process: 0.31819 | load: 0.00022 [I 2022-10-10 12:26:59] ...trap/engines/engine.py.321: eval: epoch 71 | batch 120/510 [I 2022-10-10 12:26:59] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:05 [I 2022-10-10 12:26:59] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00020 [I 2022-10-10 12:27:02] ...trap/engines/engine.py.321: eval: epoch 71 | batch 130/510 [I 2022-10-10 12:27:02] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:02 [I 2022-10-10 12:27:02] ...trap/engines/engine.py.326: process: 0.32907 | load: 0.00027 [I 2022-10-10 12:27:05] ...trap/engines/engine.py.321: eval: epoch 71 | batch 140/510 [I 2022-10-10 12:27:05] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:58 [I 2022-10-10 12:27:05] ...trap/engines/engine.py.326: process: 0.32390 | load: 0.00021 [I 2022-10-10 12:27:09] ...trap/engines/engine.py.321: eval: epoch 71 | batch 150/510 [I 2022-10-10 12:27:09] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:55 [I 2022-10-10 12:27:09] ...trap/engines/engine.py.326: process: 0.32281 | load: 0.00021 [I 2022-10-10 12:27:12] ...trap/engines/engine.py.321: eval: epoch 71 | batch 160/510 [I 2022-10-10 12:27:12] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:52 [I 2022-10-10 12:27:12] ...trap/engines/engine.py.326: process: 0.31896 | load: 0.00022 [I 2022-10-10 12:27:15] ...trap/engines/engine.py.321: eval: epoch 71 | batch 170/510 [I 2022-10-10 12:27:15] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:50 [I 2022-10-10 12:27:15] ...trap/engines/engine.py.326: process: 0.32343 | load: 0.00022 [I 2022-10-10 12:27:18] ...trap/engines/engine.py.321: eval: epoch 71 | batch 180/510 [I 2022-10-10 12:27:18] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:47 [I 2022-10-10 12:27:18] ...trap/engines/engine.py.326: process: 0.32062 | load: 0.00021 [I 2022-10-10 12:27:22] ...trap/engines/engine.py.321: eval: epoch 71 | batch 190/510 [I 2022-10-10 12:27:22] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:43 [I 2022-10-10 12:27:22] ...trap/engines/engine.py.326: process: 0.32416 | load: 0.00021 [I 2022-10-10 12:27:25] ...trap/engines/engine.py.321: eval: epoch 71 | batch 200/510 [I 2022-10-10 12:27:25] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:40 [I 2022-10-10 12:27:25] ...trap/engines/engine.py.326: process: 0.32356 | load: 0.00021 [I 2022-10-10 12:27:28] ...trap/engines/engine.py.321: eval: epoch 71 | batch 210/510 [I 2022-10-10 12:27:28] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:37 [I 2022-10-10 12:27:28] ...trap/engines/engine.py.326: process: 0.32302 | load: 0.00022 [I 2022-10-10 12:27:31] ...trap/engines/engine.py.321: eval: epoch 71 | batch 220/510 [I 2022-10-10 12:27:31] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:34 [I 2022-10-10 12:27:31] ...trap/engines/engine.py.326: process: 0.32896 | load: 0.00021 [I 2022-10-10 12:27:35] ...trap/engines/engine.py.321: eval: epoch 71 | batch 230/510 [I 2022-10-10 12:27:35] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:31 [I 2022-10-10 12:27:35] ...trap/engines/engine.py.326: process: 0.32092 | load: 0.00022 [I 2022-10-10 12:27:38] ...trap/engines/engine.py.321: eval: epoch 71 | batch 240/510 [I 2022-10-10 12:27:38] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:28 [I 2022-10-10 12:27:38] ...trap/engines/engine.py.326: process: 0.32708 | load: 0.00022 [I 2022-10-10 12:27:41] ...trap/engines/engine.py.321: eval: epoch 71 | batch 250/510 [I 2022-10-10 12:27:41] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:25 [I 2022-10-10 12:27:41] ...trap/engines/engine.py.326: process: 0.32963 | load: 0.00022 [I 2022-10-10 12:27:44] ...trap/engines/engine.py.321: eval: epoch 71 | batch 260/510 [I 2022-10-10 12:27:44] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:21 [I 2022-10-10 12:27:44] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00022 [I 2022-10-10 12:27:48] ...trap/engines/engine.py.321: eval: epoch 71 | batch 270/510 [I 2022-10-10 12:27:48] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:18 [I 2022-10-10 12:27:48] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00022 [I 2022-10-10 12:27:51] ...trap/engines/engine.py.321: eval: epoch 71 | batch 280/510 [I 2022-10-10 12:27:51] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:14 [I 2022-10-10 12:27:51] ...trap/engines/engine.py.326: process: 0.32601 | load: 0.00021 [I 2022-10-10 12:27:54] ...trap/engines/engine.py.321: eval: epoch 71 | batch 290/510 [I 2022-10-10 12:27:54] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:11 [I 2022-10-10 12:27:54] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00022 [I 2022-10-10 12:27:57] ...trap/engines/engine.py.321: eval: epoch 71 | batch 300/510 [I 2022-10-10 12:27:57] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:08 [I 2022-10-10 12:27:57] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00027 [I 2022-10-10 12:28:01] ...trap/engines/engine.py.321: eval: epoch 71 | batch 310/510 [I 2022-10-10 12:28:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:05 [I 2022-10-10 12:28:01] ...trap/engines/engine.py.326: process: 0.32335 | load: 0.00020 [I 2022-10-10 12:28:04] ...trap/engines/engine.py.321: eval: epoch 71 | batch 320/510 [I 2022-10-10 12:28:04] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:01 [I 2022-10-10 12:28:04] ...trap/engines/engine.py.326: process: 0.32229 | load: 0.00023 [I 2022-10-10 12:28:07] ...trap/engines/engine.py.321: eval: epoch 71 | batch 330/510 [I 2022-10-10 12:28:07] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:57 [I 2022-10-10 12:28:07] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00021 [I 2022-10-10 12:28:10] ...trap/engines/engine.py.321: eval: epoch 71 | batch 340/510 [I 2022-10-10 12:28:10] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:54 [I 2022-10-10 12:28:10] ...trap/engines/engine.py.326: process: 0.32281 | load: 0.00020 [I 2022-10-10 12:28:14] ...trap/engines/engine.py.321: eval: epoch 71 | batch 350/510 [I 2022-10-10 12:28:14] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:51 [I 2022-10-10 12:28:14] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00021 [I 2022-10-10 12:28:17] ...trap/engines/engine.py.321: eval: epoch 71 | batch 360/510 [I 2022-10-10 12:28:17] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:48 [I 2022-10-10 12:28:17] ...trap/engines/engine.py.326: process: 0.32189 | load: 0.00019 [I 2022-10-10 12:28:20] ...trap/engines/engine.py.321: eval: epoch 71 | batch 370/510 [I 2022-10-10 12:28:20] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-10 12:28:20] ...trap/engines/engine.py.326: process: 0.31813 | load: 0.00019 [I 2022-10-10 12:28:23] ...trap/engines/engine.py.321: eval: epoch 71 | batch 380/510 [I 2022-10-10 12:28:23] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-10 12:28:23] ...trap/engines/engine.py.326: process: 0.32557 | load: 0.00022 [I 2022-10-10 12:28:26] ...trap/engines/engine.py.321: eval: epoch 71 | batch 390/510 [I 2022-10-10 12:28:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-10 12:28:26] ...trap/engines/engine.py.326: process: 0.31782 | load: 0.00020 [I 2022-10-10 12:28:30] ...trap/engines/engine.py.321: eval: epoch 71 | batch 400/510 [I 2022-10-10 12:28:30] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:35 [I 2022-10-10 12:28:30] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00019 [I 2022-10-10 12:28:33] ...trap/engines/engine.py.321: eval: epoch 71 | batch 410/510 [I 2022-10-10 12:28:33] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-10 12:28:33] ...trap/engines/engine.py.326: process: 0.31684 | load: 0.00021 [I 2022-10-10 12:28:36] ...trap/engines/engine.py.321: eval: epoch 71 | batch 420/510 [I 2022-10-10 12:28:36] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-10 12:28:36] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00019 [I 2022-10-10 12:28:39] ...trap/engines/engine.py.321: eval: epoch 71 | batch 430/510 [I 2022-10-10 12:28:39] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-10 12:28:39] ...trap/engines/engine.py.326: process: 0.31696 | load: 0.00020 [I 2022-10-10 12:28:42] ...trap/engines/engine.py.321: eval: epoch 71 | batch 440/510 [I 2022-10-10 12:28:42] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:22 [I 2022-10-10 12:28:42] ...trap/engines/engine.py.326: process: 0.32255 | load: 0.00019 [I 2022-10-10 12:28:46] ...trap/engines/engine.py.321: eval: epoch 71 | batch 450/510 [I 2022-10-10 12:28:46] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:19 [I 2022-10-10 12:28:46] ...trap/engines/engine.py.326: process: 0.32872 | load: 0.00019 [I 2022-10-10 12:28:52] ...trap/engines/engine.py.321: eval: epoch 71 | batch 460/510 [I 2022-10-10 12:28:52] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:32 [I 2022-10-10 12:28:52] ...trap/engines/engine.py.326: process: 0.32310 | load: 0.00017 [I 2022-10-10 12:28:57] ...trap/engines/engine.py.321: eval: epoch 71 | batch 470/510 [I 2022-10-10 12:28:57] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:15 [I 2022-10-10 12:28:57] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00017 [I 2022-10-10 12:29:00] ...trap/engines/engine.py.321: eval: epoch 71 | batch 480/510 [I 2022-10-10 12:29:00] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:09 [I 2022-10-10 12:29:00] ...trap/engines/engine.py.326: process: 0.31843 | load: 0.00019 [I 2022-10-10 12:29:03] ...trap/engines/engine.py.321: eval: epoch 71 | batch 490/510 [I 2022-10-10 12:29:03] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:06 [I 2022-10-10 12:29:03] ...trap/engines/engine.py.326: process: 0.32267 | load: 0.00019 [I 2022-10-10 12:29:06] ...trap/engines/engine.py.321: eval: epoch 71 | batch 500/510 [I 2022-10-10 12:29:06] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:03 [I 2022-10-10 12:29:06] ...trap/engines/engine.py.326: process: 0.32418 | load: 0.00019 [I 2022-10-10 12:29:09] ...trap/engines/engine.py.321: eval: epoch 71 | batch 510/510 [I 2022-10-10 12:29:09] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:00 [I 2022-10-10 12:29:09] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00017 [S 2022-10-10 12:29:10] ...trap/engines/engine.py.132: eval_epoch.epoch: 71 [I 2022-10-10 12:29:10] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736500 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907400 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935900 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.728900 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903200 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934700 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007270 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007877 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008203 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014604 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008292 [S 2022-10-10 12:29:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006325 [I 2022-10-10 12:29:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 12:29:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 12:29:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 12:29:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 12:29:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 12:29:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 12:29:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 12:29:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 12:29:55] ...trap/engines/engine.py.163: Training model on trainset for epoch 72 [I 2022-10-10 12:29:55] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 12:29:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 0/2304 [I 2022-10-10 12:29:58] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:18:46 [I 2022-10-10 12:29:58] ...trap/engines/engine.py.229: process: 3.61384 | load: 2.71545 [I 2022-10-10 12:29:58] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-10 12:29:58] ...optimizers/trijoint.py.151: total_norm: 18.565872 [I 2022-10-10 12:30:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 10/2304 [I 2022-10-10 12:30:07] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:44:07 [I 2022-10-10 12:30:07] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00518 [I 2022-10-10 12:30:07] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-10 12:30:07] ...optimizers/trijoint.py.151: total_norm: 17.347481 [I 2022-10-10 12:30:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 20/2304 [I 2022-10-10 12:30:16] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:59 [I 2022-10-10 12:30:16] ...trap/engines/engine.py.229: process: 0.86930 | load: 0.00580 [I 2022-10-10 12:30:16] ...trap/engines/engine.py.230: loss: 0.11012 [I 2022-10-10 12:30:16] ...optimizers/trijoint.py.151: total_norm: 11.791987 [I 2022-10-10 12:30:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 30/2304 [I 2022-10-10 12:30:24] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:38 [I 2022-10-10 12:30:24] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00418 [I 2022-10-10 12:30:24] ...trap/engines/engine.py.230: loss: 0.09732 [I 2022-10-10 12:30:24] ...optimizers/trijoint.py.151: total_norm: 7.077137 [I 2022-10-10 12:30:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 40/2304 [I 2022-10-10 12:30:33] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:18 [I 2022-10-10 12:30:33] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00404 [I 2022-10-10 12:30:33] ...trap/engines/engine.py.230: loss: 0.09853 [I 2022-10-10 12:30:33] ...optimizers/trijoint.py.151: total_norm: 13.724595 [I 2022-10-10 12:30:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 50/2304 [I 2022-10-10 12:30:42] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:14 [I 2022-10-10 12:30:42] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00382 [I 2022-10-10 12:30:42] ...trap/engines/engine.py.230: loss: 0.11555 [I 2022-10-10 12:30:42] ...optimizers/trijoint.py.151: total_norm: 15.587996 [I 2022-10-10 12:30:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 60/2304 [I 2022-10-10 12:30:50] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:04 [I 2022-10-10 12:30:50] ...trap/engines/engine.py.229: process: 0.86728 | load: 0.00423 [I 2022-10-10 12:30:50] ...trap/engines/engine.py.230: loss: 0.13961 [I 2022-10-10 12:30:50] ...optimizers/trijoint.py.151: total_norm: 15.158900 [I 2022-10-10 12:30:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 70/2304 [I 2022-10-10 12:30:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:13 [I 2022-10-10 12:30:59] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00360 [I 2022-10-10 12:30:59] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-10 12:30:59] ...optimizers/trijoint.py.151: total_norm: 13.218696 [I 2022-10-10 12:31:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 80/2304 [I 2022-10-10 12:31:07] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:01 [I 2022-10-10 12:31:07] ...trap/engines/engine.py.229: process: 0.87154 | load: 0.00384 [I 2022-10-10 12:31:07] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-10 12:31:07] ...optimizers/trijoint.py.151: total_norm: 14.417384 [I 2022-10-10 12:31:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 90/2304 [I 2022-10-10 12:31:16] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:32:01 [I 2022-10-10 12:31:16] ...trap/engines/engine.py.229: process: 0.87180 | load: 0.00326 [I 2022-10-10 12:31:16] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-10 12:31:16] ...optimizers/trijoint.py.151: total_norm: 35.052898 [I 2022-10-10 12:31:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 100/2304 [I 2022-10-10 12:31:25] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:32 [I 2022-10-10 12:31:25] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00391 [I 2022-10-10 12:31:25] ...trap/engines/engine.py.230: loss: 0.15602 [I 2022-10-10 12:31:25] ...optimizers/trijoint.py.151: total_norm: 10.980604 [I 2022-10-10 12:31:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 110/2304 [I 2022-10-10 12:31:33] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:51 [I 2022-10-10 12:31:33] ...trap/engines/engine.py.229: process: 0.86647 | load: 0.00304 [I 2022-10-10 12:31:33] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-10 12:31:33] ...optimizers/trijoint.py.151: total_norm: 44.365162 [I 2022-10-10 12:31:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 120/2304 [I 2022-10-10 12:31:42] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:34 [I 2022-10-10 12:31:42] ...trap/engines/engine.py.229: process: 0.86272 | load: 0.00332 [I 2022-10-10 12:31:42] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-10 12:31:42] ...optimizers/trijoint.py.151: total_norm: 11.495930 [I 2022-10-10 12:31:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 130/2304 [I 2022-10-10 12:31:51] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:30 [I 2022-10-10 12:31:51] ...trap/engines/engine.py.229: process: 0.87068 | load: 0.00332 [I 2022-10-10 12:31:51] ...trap/engines/engine.py.230: loss: 0.11055 [I 2022-10-10 12:31:51] ...optimizers/trijoint.py.151: total_norm: 12.169106 [I 2022-10-10 12:31:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 140/2304 [I 2022-10-10 12:31:59] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:30:53 [I 2022-10-10 12:31:59] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00320 [I 2022-10-10 12:31:59] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-10 12:31:59] ...optimizers/trijoint.py.151: total_norm: 10.135363 [I 2022-10-10 12:32:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 150/2304 [I 2022-10-10 12:32:08] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:50 [I 2022-10-10 12:32:08] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00339 [I 2022-10-10 12:32:08] ...trap/engines/engine.py.230: loss: 0.08900 [I 2022-10-10 12:32:08] ...optimizers/trijoint.py.151: total_norm: 16.551163 [I 2022-10-10 12:32:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 160/2304 [I 2022-10-10 12:32:17] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:35 [I 2022-10-10 12:32:17] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00384 [I 2022-10-10 12:32:17] ...trap/engines/engine.py.230: loss: 0.11458 [I 2022-10-10 12:32:17] ...optimizers/trijoint.py.151: total_norm: 12.230457 [I 2022-10-10 12:32:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 170/2304 [I 2022-10-10 12:32:25] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:43 [I 2022-10-10 12:32:25] ...trap/engines/engine.py.229: process: 0.86669 | load: 0.00310 [I 2022-10-10 12:32:25] ...trap/engines/engine.py.230: loss: 0.14000 [I 2022-10-10 12:32:25] ...optimizers/trijoint.py.151: total_norm: 18.118462 [I 2022-10-10 12:32:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 180/2304 [I 2022-10-10 12:32:34] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:24 [I 2022-10-10 12:32:34] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00335 [I 2022-10-10 12:32:34] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-10 12:32:34] ...optimizers/trijoint.py.151: total_norm: 10.121901 [I 2022-10-10 12:32:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 190/2304 [I 2022-10-10 12:32:42] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:04 [I 2022-10-10 12:32:42] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00340 [I 2022-10-10 12:32:42] ...trap/engines/engine.py.230: loss: 0.11209 [I 2022-10-10 12:32:42] ...optimizers/trijoint.py.151: total_norm: 10.808892 [I 2022-10-10 12:32:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 200/2304 [I 2022-10-10 12:32:51] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:29:56 [I 2022-10-10 12:32:51] ...trap/engines/engine.py.229: process: 0.85910 | load: 0.00310 [I 2022-10-10 12:32:51] ...trap/engines/engine.py.230: loss: 0.16499 [I 2022-10-10 12:32:51] ...optimizers/trijoint.py.151: total_norm: 16.095945 [I 2022-10-10 12:33:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 210/2304 [I 2022-10-10 12:33:00] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:30:01 [I 2022-10-10 12:33:00] ...trap/engines/engine.py.229: process: 0.85451 | load: 0.00335 [I 2022-10-10 12:33:00] ...trap/engines/engine.py.230: loss: 0.10515 [I 2022-10-10 12:33:00] ...optimizers/trijoint.py.151: total_norm: 25.149780 [I 2022-10-10 12:33:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 220/2304 [I 2022-10-10 12:33:08] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:40 [I 2022-10-10 12:33:08] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00333 [I 2022-10-10 12:33:08] ...trap/engines/engine.py.230: loss: 0.17707 [I 2022-10-10 12:33:08] ...optimizers/trijoint.py.151: total_norm: 16.550432 [I 2022-10-10 12:33:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 230/2304 [I 2022-10-10 12:33:17] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:51 [I 2022-10-10 12:33:17] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00307 [I 2022-10-10 12:33:17] ...trap/engines/engine.py.230: loss: 0.10950 [I 2022-10-10 12:33:17] ...optimizers/trijoint.py.151: total_norm: 13.929292 [I 2022-10-10 12:33:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 240/2304 [I 2022-10-10 12:33:25] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:40 [I 2022-10-10 12:33:25] ...trap/engines/engine.py.229: process: 0.85918 | load: 0.00332 [I 2022-10-10 12:33:25] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-10 12:33:25] ...optimizers/trijoint.py.151: total_norm: 26.283533 [I 2022-10-10 12:33:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 250/2304 [I 2022-10-10 12:33:34] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:38 [I 2022-10-10 12:33:34] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00317 [I 2022-10-10 12:33:34] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-10 12:33:34] ...optimizers/trijoint.py.151: total_norm: 12.185066 [I 2022-10-10 12:33:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 260/2304 [I 2022-10-10 12:33:43] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:27 [I 2022-10-10 12:33:43] ...trap/engines/engine.py.229: process: 0.87388 | load: 0.00327 [I 2022-10-10 12:33:43] ...trap/engines/engine.py.230: loss: 0.17613 [I 2022-10-10 12:33:43] ...optimizers/trijoint.py.151: total_norm: 22.808889 [I 2022-10-10 12:33:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 270/2304 [I 2022-10-10 12:33:51] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:28:50 [I 2022-10-10 12:33:51] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00334 [I 2022-10-10 12:33:51] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-10 12:33:51] ...optimizers/trijoint.py.151: total_norm: 19.093243 [I 2022-10-10 12:34:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 280/2304 [I 2022-10-10 12:34:00] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:29:05 [I 2022-10-10 12:34:00] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00307 [I 2022-10-10 12:34:00] ...trap/engines/engine.py.230: loss: 0.13985 [I 2022-10-10 12:34:00] ...optimizers/trijoint.py.151: total_norm: 16.005051 [I 2022-10-10 12:34:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 290/2304 [I 2022-10-10 12:34:09] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:52 [I 2022-10-10 12:34:09] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00323 [I 2022-10-10 12:34:09] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-10 12:34:09] ...optimizers/trijoint.py.151: total_norm: 9.665819 [I 2022-10-10 12:34:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 300/2304 [I 2022-10-10 12:34:17] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:35 [I 2022-10-10 12:34:17] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00306 [I 2022-10-10 12:34:17] ...trap/engines/engine.py.230: loss: 0.22183 [I 2022-10-10 12:34:17] ...optimizers/trijoint.py.151: total_norm: 19.511625 [I 2022-10-10 12:34:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 310/2304 [I 2022-10-10 12:34:26] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:23 [I 2022-10-10 12:34:26] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00297 [I 2022-10-10 12:34:26] ...trap/engines/engine.py.230: loss: 0.17018 [I 2022-10-10 12:34:26] ...optimizers/trijoint.py.151: total_norm: 13.189653 [I 2022-10-10 12:34:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 320/2304 [I 2022-10-10 12:34:34] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:19 [I 2022-10-10 12:34:34] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00295 [I 2022-10-10 12:34:34] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-10 12:34:34] ...optimizers/trijoint.py.151: total_norm: 8.801448 [I 2022-10-10 12:34:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 330/2304 [I 2022-10-10 12:34:43] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:28:08 [I 2022-10-10 12:34:43] ...trap/engines/engine.py.229: process: 0.86082 | load: 0.00307 [I 2022-10-10 12:34:43] ...trap/engines/engine.py.230: loss: 0.10014 [I 2022-10-10 12:34:43] ...optimizers/trijoint.py.151: total_norm: 9.347314 [I 2022-10-10 12:34:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 340/2304 [I 2022-10-10 12:34:51] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:27:53 [I 2022-10-10 12:34:51] ...trap/engines/engine.py.229: process: 0.84824 | load: 0.00296 [I 2022-10-10 12:34:51] ...trap/engines/engine.py.230: loss: 0.09233 [I 2022-10-10 12:34:51] ...optimizers/trijoint.py.151: total_norm: 14.369879 [I 2022-10-10 12:35:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 350/2304 [I 2022-10-10 12:35:00] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:41 [I 2022-10-10 12:35:00] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00300 [I 2022-10-10 12:35:00] ...trap/engines/engine.py.230: loss: 0.09045 [I 2022-10-10 12:35:00] ...optimizers/trijoint.py.151: total_norm: 17.069500 [I 2022-10-10 12:35:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 360/2304 [I 2022-10-10 12:35:08] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:27:31 [I 2022-10-10 12:35:08] ...trap/engines/engine.py.229: process: 0.83723 | load: 0.00297 [I 2022-10-10 12:35:08] ...trap/engines/engine.py.230: loss: 0.15407 [I 2022-10-10 12:35:08] ...optimizers/trijoint.py.151: total_norm: 18.412743 [I 2022-10-10 12:35:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 370/2304 [I 2022-10-10 12:35:17] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:22 [I 2022-10-10 12:35:17] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00334 [I 2022-10-10 12:35:17] ...trap/engines/engine.py.230: loss: 0.11124 [I 2022-10-10 12:35:17] ...optimizers/trijoint.py.151: total_norm: 8.449452 [I 2022-10-10 12:35:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 380/2304 [I 2022-10-10 12:35:25] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:20 [I 2022-10-10 12:35:25] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00297 [I 2022-10-10 12:35:25] ...trap/engines/engine.py.230: loss: 0.21304 [I 2022-10-10 12:35:25] ...optimizers/trijoint.py.151: total_norm: 30.024397 [I 2022-10-10 12:35:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 390/2304 [I 2022-10-10 12:35:34] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:08 [I 2022-10-10 12:35:34] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00313 [I 2022-10-10 12:35:34] ...trap/engines/engine.py.230: loss: 0.20508 [I 2022-10-10 12:35:34] ...optimizers/trijoint.py.151: total_norm: 22.508808 [I 2022-10-10 12:35:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 400/2304 [I 2022-10-10 12:35:43] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:02 [I 2022-10-10 12:35:43] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00295 [I 2022-10-10 12:35:43] ...trap/engines/engine.py.230: loss: 0.08584 [I 2022-10-10 12:35:43] ...optimizers/trijoint.py.151: total_norm: 12.776581 [I 2022-10-10 12:35:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 410/2304 [I 2022-10-10 12:35:51] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:00 [I 2022-10-10 12:35:51] ...trap/engines/engine.py.229: process: 0.87008 | load: 0.00302 [I 2022-10-10 12:35:51] ...trap/engines/engine.py.230: loss: 0.16417 [I 2022-10-10 12:35:51] ...optimizers/trijoint.py.151: total_norm: 8.920327 [I 2022-10-10 12:36:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 420/2304 [I 2022-10-10 12:36:00] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:48 [I 2022-10-10 12:36:00] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00341 [I 2022-10-10 12:36:00] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-10 12:36:00] ...optimizers/trijoint.py.151: total_norm: 16.702366 [I 2022-10-10 12:36:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 430/2304 [I 2022-10-10 12:36:08] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:33 [I 2022-10-10 12:36:08] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00293 [I 2022-10-10 12:36:08] ...trap/engines/engine.py.230: loss: 0.16731 [I 2022-10-10 12:36:08] ...optimizers/trijoint.py.151: total_norm: 17.603735 [I 2022-10-10 12:36:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 440/2304 [I 2022-10-10 12:36:17] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:34 [I 2022-10-10 12:36:17] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00292 [I 2022-10-10 12:36:17] ...trap/engines/engine.py.230: loss: 0.08852 [I 2022-10-10 12:36:17] ...optimizers/trijoint.py.151: total_norm: 10.749547 [I 2022-10-10 12:36:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 450/2304 [I 2022-10-10 12:36:25] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:21 [I 2022-10-10 12:36:25] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00303 [I 2022-10-10 12:36:25] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-10 12:36:25] ...optimizers/trijoint.py.151: total_norm: 23.743176 [I 2022-10-10 12:36:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 460/2304 [I 2022-10-10 12:36:34] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:14 [I 2022-10-10 12:36:34] ...trap/engines/engine.py.229: process: 0.86188 | load: 0.00301 [I 2022-10-10 12:36:34] ...trap/engines/engine.py.230: loss: 0.10025 [I 2022-10-10 12:36:34] ...optimizers/trijoint.py.151: total_norm: 11.722301 [I 2022-10-10 12:36:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 470/2304 [I 2022-10-10 12:36:42] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:01 [I 2022-10-10 12:36:42] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00291 [I 2022-10-10 12:36:42] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-10 12:36:42] ...optimizers/trijoint.py.151: total_norm: 20.563372 [I 2022-10-10 12:36:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 480/2304 [I 2022-10-10 12:36:51] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:06 [I 2022-10-10 12:36:51] ...trap/engines/engine.py.229: process: 0.87537 | load: 0.00296 [I 2022-10-10 12:36:51] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-10 12:36:51] ...optimizers/trijoint.py.151: total_norm: 17.898287 [I 2022-10-10 12:37:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 490/2304 [I 2022-10-10 12:37:00] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:26:07 [I 2022-10-10 12:37:00] ...trap/engines/engine.py.229: process: 0.86161 | load: 0.00307 [I 2022-10-10 12:37:00] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-10 12:37:00] ...optimizers/trijoint.py.151: total_norm: 13.140463 [I 2022-10-10 12:37:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 500/2304 [I 2022-10-10 12:37:08] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:26:03 [I 2022-10-10 12:37:08] ...trap/engines/engine.py.229: process: 0.86455 | load: 0.00305 [I 2022-10-10 12:37:08] ...trap/engines/engine.py.230: loss: 0.22144 [I 2022-10-10 12:37:08] ...optimizers/trijoint.py.151: total_norm: 12.816569 [I 2022-10-10 12:37:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 510/2304 [I 2022-10-10 12:37:17] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:52 [I 2022-10-10 12:37:17] ...trap/engines/engine.py.229: process: 0.87480 | load: 0.00318 [I 2022-10-10 12:37:17] ...trap/engines/engine.py.230: loss: 0.15168 [I 2022-10-10 12:37:17] ...optimizers/trijoint.py.151: total_norm: 27.917139 [I 2022-10-10 12:37:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 520/2304 [I 2022-10-10 12:37:26] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:47 [I 2022-10-10 12:37:26] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00311 [I 2022-10-10 12:37:26] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-10 12:37:26] ...optimizers/trijoint.py.151: total_norm: 16.844282 [I 2022-10-10 12:37:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 530/2304 [I 2022-10-10 12:37:34] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:32 [I 2022-10-10 12:37:34] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00308 [I 2022-10-10 12:37:34] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-10 12:37:34] ...optimizers/trijoint.py.151: total_norm: 12.204631 [I 2022-10-10 12:37:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 540/2304 [I 2022-10-10 12:37:43] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:30 [I 2022-10-10 12:37:43] ...trap/engines/engine.py.229: process: 0.87206 | load: 0.00307 [I 2022-10-10 12:37:43] ...trap/engines/engine.py.230: loss: 0.20877 [I 2022-10-10 12:37:43] ...optimizers/trijoint.py.151: total_norm: 13.167110 [I 2022-10-10 12:37:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 550/2304 [I 2022-10-10 12:37:52] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:25:13 [I 2022-10-10 12:37:52] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00307 [I 2022-10-10 12:37:52] ...trap/engines/engine.py.230: loss: 0.14713 [I 2022-10-10 12:37:52] ...optimizers/trijoint.py.151: total_norm: 12.056710 [I 2022-10-10 12:38:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 560/2304 [I 2022-10-10 12:38:00] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:09 [I 2022-10-10 12:38:00] ...trap/engines/engine.py.229: process: 0.85786 | load: 0.00310 [I 2022-10-10 12:38:00] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-10 12:38:00] ...optimizers/trijoint.py.151: total_norm: 11.632443 [I 2022-10-10 12:38:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 570/2304 [I 2022-10-10 12:38:09] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:59 [I 2022-10-10 12:38:09] ...trap/engines/engine.py.229: process: 0.86494 | load: 0.00310 [I 2022-10-10 12:38:09] ...trap/engines/engine.py.230: loss: 0.11742 [I 2022-10-10 12:38:09] ...optimizers/trijoint.py.151: total_norm: 15.578450 [I 2022-10-10 12:38:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 580/2304 [I 2022-10-10 12:38:18] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:55 [I 2022-10-10 12:38:18] ...trap/engines/engine.py.229: process: 0.86918 | load: 0.00311 [I 2022-10-10 12:38:18] ...trap/engines/engine.py.230: loss: 0.11395 [I 2022-10-10 12:38:18] ...optimizers/trijoint.py.151: total_norm: 10.889267 [I 2022-10-10 12:38:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 590/2304 [I 2022-10-10 12:38:26] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:40 [I 2022-10-10 12:38:26] ...trap/engines/engine.py.229: process: 0.86178 | load: 0.00324 [I 2022-10-10 12:38:26] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-10 12:38:26] ...optimizers/trijoint.py.151: total_norm: 20.230717 [I 2022-10-10 12:38:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 600/2304 [I 2022-10-10 12:38:35] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:30 [I 2022-10-10 12:38:35] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00317 [I 2022-10-10 12:38:35] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-10 12:38:35] ...optimizers/trijoint.py.151: total_norm: 13.246621 [I 2022-10-10 12:38:44] ...trap/engines/engine.py.224: train: epoch 72 | batch 610/2304 [I 2022-10-10 12:38:44] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:33 [I 2022-10-10 12:38:44] ...trap/engines/engine.py.229: process: 0.85957 | load: 0.00323 [I 2022-10-10 12:38:44] ...trap/engines/engine.py.230: loss: 0.17839 [I 2022-10-10 12:38:44] ...optimizers/trijoint.py.151: total_norm: 9.605349 [I 2022-10-10 12:38:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 620/2304 [I 2022-10-10 12:38:53] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:24:18 [I 2022-10-10 12:38:53] ...trap/engines/engine.py.229: process: 0.85474 | load: 0.00308 [I 2022-10-10 12:38:53] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-10 12:38:53] ...optimizers/trijoint.py.151: total_norm: 7.268278 [I 2022-10-10 12:39:01] ...trap/engines/engine.py.224: train: epoch 72 | batch 630/2304 [I 2022-10-10 12:39:01] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:24:08 [I 2022-10-10 12:39:01] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00328 [I 2022-10-10 12:39:01] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-10 12:39:01] ...optimizers/trijoint.py.151: total_norm: 14.135994 [I 2022-10-10 12:39:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 640/2304 [I 2022-10-10 12:39:10] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:57 [I 2022-10-10 12:39:10] ...trap/engines/engine.py.229: process: 0.87288 | load: 0.00315 [I 2022-10-10 12:39:10] ...trap/engines/engine.py.230: loss: 0.18228 [I 2022-10-10 12:39:10] ...optimizers/trijoint.py.151: total_norm: 20.156055 [I 2022-10-10 12:39:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 650/2304 [I 2022-10-10 12:39:19] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:52 [I 2022-10-10 12:39:19] ...trap/engines/engine.py.229: process: 0.86835 | load: 0.00303 [I 2022-10-10 12:39:19] ...trap/engines/engine.py.230: loss: 0.10616 [I 2022-10-10 12:39:19] ...optimizers/trijoint.py.151: total_norm: 9.805718 [I 2022-10-10 12:39:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 660/2304 [I 2022-10-10 12:39:27] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:50 [I 2022-10-10 12:39:27] ...trap/engines/engine.py.229: process: 0.86674 | load: 0.00310 [I 2022-10-10 12:39:27] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-10 12:39:27] ...optimizers/trijoint.py.151: total_norm: 18.041323 [I 2022-10-10 12:39:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 670/2304 [I 2022-10-10 12:39:36] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:34 [I 2022-10-10 12:39:36] ...trap/engines/engine.py.229: process: 0.85068 | load: 0.00319 [I 2022-10-10 12:39:36] ...trap/engines/engine.py.230: loss: 0.19353 [I 2022-10-10 12:39:36] ...optimizers/trijoint.py.151: total_norm: 21.479971 [I 2022-10-10 12:39:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 680/2304 [I 2022-10-10 12:39:45] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:22 [I 2022-10-10 12:39:45] ...trap/engines/engine.py.229: process: 0.86092 | load: 0.00315 [I 2022-10-10 12:39:45] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-10 12:39:45] ...optimizers/trijoint.py.151: total_norm: 22.090937 [I 2022-10-10 12:39:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 690/2304 [I 2022-10-10 12:39:53] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:17 [I 2022-10-10 12:39:53] ...trap/engines/engine.py.229: process: 0.86349 | load: 0.00309 [I 2022-10-10 12:39:53] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-10 12:39:53] ...optimizers/trijoint.py.151: total_norm: 17.030214 [I 2022-10-10 12:40:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 700/2304 [I 2022-10-10 12:40:02] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:23:00 [I 2022-10-10 12:40:02] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00307 [I 2022-10-10 12:40:02] ...trap/engines/engine.py.230: loss: 0.17359 [I 2022-10-10 12:40:02] ...optimizers/trijoint.py.151: total_norm: 12.019424 [I 2022-10-10 12:40:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 710/2304 [I 2022-10-10 12:40:11] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:55 [I 2022-10-10 12:40:11] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00315 [I 2022-10-10 12:40:11] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-10 12:40:11] ...optimizers/trijoint.py.151: total_norm: 13.809100 [I 2022-10-10 12:40:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 720/2304 [I 2022-10-10 12:40:19] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:46 [I 2022-10-10 12:40:19] ...trap/engines/engine.py.229: process: 0.85904 | load: 0.00309 [I 2022-10-10 12:40:19] ...trap/engines/engine.py.230: loss: 0.16090 [I 2022-10-10 12:40:19] ...optimizers/trijoint.py.151: total_norm: 22.255993 [I 2022-10-10 12:40:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 730/2304 [I 2022-10-10 12:40:28] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:31 [I 2022-10-10 12:40:28] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00308 [I 2022-10-10 12:40:28] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-10 12:40:28] ...optimizers/trijoint.py.151: total_norm: 65.058662 [I 2022-10-10 12:40:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 740/2304 [I 2022-10-10 12:40:37] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:20 [I 2022-10-10 12:40:37] ...trap/engines/engine.py.229: process: 0.86256 | load: 0.00305 [I 2022-10-10 12:40:37] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-10 12:40:37] ...optimizers/trijoint.py.151: total_norm: 30.679373 [I 2022-10-10 12:40:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 750/2304 [I 2022-10-10 12:40:45] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:23 [I 2022-10-10 12:40:45] ...trap/engines/engine.py.229: process: 0.87169 | load: 0.00312 [I 2022-10-10 12:40:45] ...trap/engines/engine.py.230: loss: 0.19001 [I 2022-10-10 12:40:45] ...optimizers/trijoint.py.151: total_norm: 17.333973 [I 2022-10-10 12:40:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 760/2304 [I 2022-10-10 12:40:54] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:12 [I 2022-10-10 12:40:54] ...trap/engines/engine.py.229: process: 0.86755 | load: 0.00296 [I 2022-10-10 12:40:54] ...trap/engines/engine.py.230: loss: 0.13008 [I 2022-10-10 12:40:54] ...optimizers/trijoint.py.151: total_norm: 17.948399 [I 2022-10-10 12:41:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 770/2304 [I 2022-10-10 12:41:02] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:01 [I 2022-10-10 12:41:02] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00292 [I 2022-10-10 12:41:02] ...trap/engines/engine.py.230: loss: 0.13680 [I 2022-10-10 12:41:02] ...optimizers/trijoint.py.151: total_norm: 25.137260 [I 2022-10-10 12:41:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 780/2304 [I 2022-10-10 12:41:11] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:45 [I 2022-10-10 12:41:11] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00299 [I 2022-10-10 12:41:11] ...trap/engines/engine.py.230: loss: 0.11374 [I 2022-10-10 12:41:11] ...optimizers/trijoint.py.151: total_norm: 12.237890 [I 2022-10-10 12:41:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 790/2304 [I 2022-10-10 12:41:20] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:46 [I 2022-10-10 12:41:20] ...trap/engines/engine.py.229: process: 0.88000 | load: 0.00292 [I 2022-10-10 12:41:20] ...trap/engines/engine.py.230: loss: 0.15724 [I 2022-10-10 12:41:20] ...optimizers/trijoint.py.151: total_norm: 17.749737 [I 2022-10-10 12:41:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 800/2304 [I 2022-10-10 12:41:28] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:26 [I 2022-10-10 12:41:28] ...trap/engines/engine.py.229: process: 0.86427 | load: 0.00306 [I 2022-10-10 12:41:28] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-10 12:41:28] ...optimizers/trijoint.py.151: total_norm: 25.124807 [I 2022-10-10 12:41:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 810/2304 [I 2022-10-10 12:41:37] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:14 [I 2022-10-10 12:41:37] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00299 [I 2022-10-10 12:41:37] ...trap/engines/engine.py.230: loss: 0.15578 [I 2022-10-10 12:41:37] ...optimizers/trijoint.py.151: total_norm: 27.058424 [I 2022-10-10 12:41:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 820/2304 [I 2022-10-10 12:41:45] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:12 [I 2022-10-10 12:41:45] ...trap/engines/engine.py.229: process: 0.86495 | load: 0.00302 [I 2022-10-10 12:41:45] ...trap/engines/engine.py.230: loss: 0.16525 [I 2022-10-10 12:41:45] ...optimizers/trijoint.py.151: total_norm: 18.049728 [I 2022-10-10 12:41:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 830/2304 [I 2022-10-10 12:41:54] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:56 [I 2022-10-10 12:41:54] ...trap/engines/engine.py.229: process: 0.85162 | load: 0.00296 [I 2022-10-10 12:41:54] ...trap/engines/engine.py.230: loss: 0.17644 [I 2022-10-10 12:41:54] ...optimizers/trijoint.py.151: total_norm: 23.449759 [I 2022-10-10 12:42:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 840/2304 [I 2022-10-10 12:42:03] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:54 [I 2022-10-10 12:42:03] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00300 [I 2022-10-10 12:42:03] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-10 12:42:03] ...optimizers/trijoint.py.151: total_norm: 11.270555 [I 2022-10-10 12:42:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 850/2304 [I 2022-10-10 12:42:11] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:46 [I 2022-10-10 12:42:11] ...trap/engines/engine.py.229: process: 0.86697 | load: 0.00444 [I 2022-10-10 12:42:11] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-10 12:42:11] ...optimizers/trijoint.py.151: total_norm: 38.481808 [I 2022-10-10 12:42:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 860/2304 [I 2022-10-10 12:42:20] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:34 [I 2022-10-10 12:42:20] ...trap/engines/engine.py.229: process: 0.85102 | load: 0.00296 [I 2022-10-10 12:42:20] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-10 12:42:20] ...optimizers/trijoint.py.151: total_norm: 13.584391 [I 2022-10-10 12:42:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 870/2304 [I 2022-10-10 12:42:28] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:18 [I 2022-10-10 12:42:28] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00295 [I 2022-10-10 12:42:28] ...trap/engines/engine.py.230: loss: 0.12146 [I 2022-10-10 12:42:28] ...optimizers/trijoint.py.151: total_norm: 15.915662 [I 2022-10-10 12:42:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 880/2304 [I 2022-10-10 12:42:37] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:19 [I 2022-10-10 12:42:37] ...trap/engines/engine.py.229: process: 0.86253 | load: 0.00298 [I 2022-10-10 12:42:37] ...trap/engines/engine.py.230: loss: 0.12784 [I 2022-10-10 12:42:37] ...optimizers/trijoint.py.151: total_norm: 12.542906 [I 2022-10-10 12:42:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 890/2304 [I 2022-10-10 12:42:45] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:20:01 [I 2022-10-10 12:42:45] ...trap/engines/engine.py.229: process: 0.82857 | load: 0.00295 [I 2022-10-10 12:42:45] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-10 12:42:45] ...optimizers/trijoint.py.151: total_norm: 18.754120 [I 2022-10-10 12:42:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 900/2304 [I 2022-10-10 12:42:54] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:20:04 [I 2022-10-10 12:42:54] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00295 [I 2022-10-10 12:42:54] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-10 12:42:54] ...optimizers/trijoint.py.151: total_norm: 14.406389 [I 2022-10-10 12:43:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 910/2304 [I 2022-10-10 12:43:03] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:40 [I 2022-10-10 12:43:03] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00297 [I 2022-10-10 12:43:03] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-10 12:43:03] ...optimizers/trijoint.py.151: total_norm: 10.029487 [I 2022-10-10 12:43:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 920/2304 [I 2022-10-10 12:43:11] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:44 [I 2022-10-10 12:43:11] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00295 [I 2022-10-10 12:43:11] ...trap/engines/engine.py.230: loss: 0.15822 [I 2022-10-10 12:43:11] ...optimizers/trijoint.py.151: total_norm: 19.566423 [I 2022-10-10 12:43:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 930/2304 [I 2022-10-10 12:43:20] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:29 [I 2022-10-10 12:43:20] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00308 [I 2022-10-10 12:43:20] ...trap/engines/engine.py.230: loss: 0.10632 [I 2022-10-10 12:43:20] ...optimizers/trijoint.py.151: total_norm: 9.790610 [I 2022-10-10 12:43:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 940/2304 [I 2022-10-10 12:43:28] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:21 [I 2022-10-10 12:43:28] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00302 [I 2022-10-10 12:43:28] ...trap/engines/engine.py.230: loss: 0.15914 [I 2022-10-10 12:43:28] ...optimizers/trijoint.py.151: total_norm: 23.145573 [I 2022-10-10 12:43:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 950/2304 [I 2022-10-10 12:43:37] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:16 [I 2022-10-10 12:43:37] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00303 [I 2022-10-10 12:43:37] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-10 12:43:37] ...optimizers/trijoint.py.151: total_norm: 13.230933 [I 2022-10-10 12:43:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 960/2304 [I 2022-10-10 12:43:45] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:10 [I 2022-10-10 12:43:45] ...trap/engines/engine.py.229: process: 0.84416 | load: 0.00297 [I 2022-10-10 12:43:45] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-10 12:43:45] ...optimizers/trijoint.py.151: total_norm: 21.516735 [I 2022-10-10 12:43:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 970/2304 [I 2022-10-10 12:43:54] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:19:00 [I 2022-10-10 12:43:54] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00299 [I 2022-10-10 12:43:54] ...trap/engines/engine.py.230: loss: 0.18748 [I 2022-10-10 12:43:54] ...optimizers/trijoint.py.151: total_norm: 16.923716 [I 2022-10-10 12:44:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 980/2304 [I 2022-10-10 12:44:02] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:50 [I 2022-10-10 12:44:02] ...trap/engines/engine.py.229: process: 0.85723 | load: 0.00295 [I 2022-10-10 12:44:02] ...trap/engines/engine.py.230: loss: 0.13434 [I 2022-10-10 12:44:02] ...optimizers/trijoint.py.151: total_norm: 18.048967 [I 2022-10-10 12:44:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 990/2304 [I 2022-10-10 12:44:11] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:45 [I 2022-10-10 12:44:11] ...trap/engines/engine.py.229: process: 0.87475 | load: 0.00312 [I 2022-10-10 12:44:11] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-10 12:44:11] ...optimizers/trijoint.py.151: total_norm: 12.568689 [I 2022-10-10 12:44:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 1000/2304 [I 2022-10-10 12:44:20] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:31 [I 2022-10-10 12:44:20] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00292 [I 2022-10-10 12:44:20] ...trap/engines/engine.py.230: loss: 0.14807 [I 2022-10-10 12:44:20] ...optimizers/trijoint.py.151: total_norm: 15.673971 [I 2022-10-10 12:44:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 1010/2304 [I 2022-10-10 12:44:28] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:23 [I 2022-10-10 12:44:28] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00303 [I 2022-10-10 12:44:28] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-10 12:44:28] ...optimizers/trijoint.py.151: total_norm: 13.969906 [I 2022-10-10 12:44:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 1020/2304 [I 2022-10-10 12:44:37] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:15 [I 2022-10-10 12:44:37] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00296 [I 2022-10-10 12:44:37] ...trap/engines/engine.py.230: loss: 0.13216 [I 2022-10-10 12:44:37] ...optimizers/trijoint.py.151: total_norm: 13.185613 [I 2022-10-10 12:44:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 1030/2304 [I 2022-10-10 12:44:45] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:06 [I 2022-10-10 12:44:45] ...trap/engines/engine.py.229: process: 0.86114 | load: 0.00303 [I 2022-10-10 12:44:45] ...trap/engines/engine.py.230: loss: 0.14864 [I 2022-10-10 12:44:45] ...optimizers/trijoint.py.151: total_norm: 17.631535 [I 2022-10-10 12:44:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 1040/2304 [I 2022-10-10 12:44:54] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:57 [I 2022-10-10 12:44:54] ...trap/engines/engine.py.229: process: 0.86077 | load: 0.00294 [I 2022-10-10 12:44:54] ...trap/engines/engine.py.230: loss: 0.19333 [I 2022-10-10 12:44:54] ...optimizers/trijoint.py.151: total_norm: 11.699814 [I 2022-10-10 12:45:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 1050/2304 [I 2022-10-10 12:45:02] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:56 [I 2022-10-10 12:45:02] ...trap/engines/engine.py.229: process: 0.86603 | load: 0.00294 [I 2022-10-10 12:45:02] ...trap/engines/engine.py.230: loss: 0.24398 [I 2022-10-10 12:45:02] ...optimizers/trijoint.py.151: total_norm: 32.855412 [I 2022-10-10 12:45:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 1060/2304 [I 2022-10-10 12:45:11] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:46 [I 2022-10-10 12:45:11] ...trap/engines/engine.py.229: process: 0.86654 | load: 0.00296 [I 2022-10-10 12:45:11] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-10 12:45:11] ...optimizers/trijoint.py.151: total_norm: 25.810734 [I 2022-10-10 12:45:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 1070/2304 [I 2022-10-10 12:45:20] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:31 [I 2022-10-10 12:45:20] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00297 [I 2022-10-10 12:45:20] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-10 12:45:20] ...optimizers/trijoint.py.151: total_norm: 55.285484 [I 2022-10-10 12:45:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 1080/2304 [I 2022-10-10 12:45:28] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:29 [I 2022-10-10 12:45:28] ...trap/engines/engine.py.229: process: 0.86029 | load: 0.00306 [I 2022-10-10 12:45:28] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-10 12:45:28] ...optimizers/trijoint.py.151: total_norm: 9.955816 [I 2022-10-10 12:45:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 1090/2304 [I 2022-10-10 12:45:37] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:27 [I 2022-10-10 12:45:37] ...trap/engines/engine.py.229: process: 0.87341 | load: 0.00305 [I 2022-10-10 12:45:37] ...trap/engines/engine.py.230: loss: 0.11850 [I 2022-10-10 12:45:37] ...optimizers/trijoint.py.151: total_norm: 45.096279 [I 2022-10-10 12:45:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 1100/2304 [I 2022-10-10 12:45:45] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:17:20 [I 2022-10-10 12:45:45] ...trap/engines/engine.py.229: process: 0.87221 | load: 0.00308 [I 2022-10-10 12:45:45] ...trap/engines/engine.py.230: loss: 0.15841 [I 2022-10-10 12:45:45] ...optimizers/trijoint.py.151: total_norm: 10.368982 [I 2022-10-10 12:45:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 1110/2304 [I 2022-10-10 12:45:54] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:17:12 [I 2022-10-10 12:45:54] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00308 [I 2022-10-10 12:45:54] ...trap/engines/engine.py.230: loss: 0.11737 [I 2022-10-10 12:45:54] ...optimizers/trijoint.py.151: total_norm: 14.057825 [I 2022-10-10 12:46:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 1120/2304 [I 2022-10-10 12:46:03] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:17:03 [I 2022-10-10 12:46:03] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00308 [I 2022-10-10 12:46:03] ...trap/engines/engine.py.230: loss: 0.15870 [I 2022-10-10 12:46:03] ...optimizers/trijoint.py.151: total_norm: 10.461111 [I 2022-10-10 12:46:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 1130/2304 [I 2022-10-10 12:46:11] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:47 [I 2022-10-10 12:46:11] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00316 [I 2022-10-10 12:46:11] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-10 12:46:11] ...optimizers/trijoint.py.151: total_norm: 9.656764 [I 2022-10-10 12:46:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 1140/2304 [I 2022-10-10 12:46:20] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:47 [I 2022-10-10 12:46:20] ...trap/engines/engine.py.229: process: 0.86327 | load: 0.00308 [I 2022-10-10 12:46:20] ...trap/engines/engine.py.230: loss: 0.21628 [I 2022-10-10 12:46:20] ...optimizers/trijoint.py.151: total_norm: 14.841059 [I 2022-10-10 12:46:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 1150/2304 [I 2022-10-10 12:46:29] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:31 [I 2022-10-10 12:46:29] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00315 [I 2022-10-10 12:46:29] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-10 12:46:29] ...optimizers/trijoint.py.151: total_norm: 26.678520 [I 2022-10-10 12:46:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 1160/2304 [I 2022-10-10 12:46:37] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:32 [I 2022-10-10 12:46:37] ...trap/engines/engine.py.229: process: 0.87697 | load: 0.00307 [I 2022-10-10 12:46:37] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-10 12:46:37] ...optimizers/trijoint.py.151: total_norm: 10.334656 [I 2022-10-10 12:46:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 1170/2304 [I 2022-10-10 12:46:46] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:19 [I 2022-10-10 12:46:46] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00308 [I 2022-10-10 12:46:46] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-10 12:46:46] ...optimizers/trijoint.py.151: total_norm: 16.295321 [I 2022-10-10 12:46:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 1180/2304 [I 2022-10-10 12:46:55] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:16:07 [I 2022-10-10 12:46:55] ...trap/engines/engine.py.229: process: 0.86453 | load: 0.00308 [I 2022-10-10 12:46:55] ...trap/engines/engine.py.230: loss: 0.11363 [I 2022-10-10 12:46:55] ...optimizers/trijoint.py.151: total_norm: 10.394967 [I 2022-10-10 12:47:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 1190/2304 [I 2022-10-10 12:47:03] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:16:03 [I 2022-10-10 12:47:03] ...trap/engines/engine.py.229: process: 0.87294 | load: 0.00308 [I 2022-10-10 12:47:03] ...trap/engines/engine.py.230: loss: 0.10868 [I 2022-10-10 12:47:03] ...optimizers/trijoint.py.151: total_norm: 36.575550 [I 2022-10-10 12:47:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 1200/2304 [I 2022-10-10 12:47:12] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:51 [I 2022-10-10 12:47:12] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00307 [I 2022-10-10 12:47:12] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-10 12:47:12] ...optimizers/trijoint.py.151: total_norm: 10.251278 [I 2022-10-10 12:47:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 1210/2304 [I 2022-10-10 12:47:21] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:47 [I 2022-10-10 12:47:21] ...trap/engines/engine.py.229: process: 0.86895 | load: 0.00307 [I 2022-10-10 12:47:21] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-10 12:47:21] ...optimizers/trijoint.py.151: total_norm: 32.832512 [I 2022-10-10 12:47:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 1220/2304 [I 2022-10-10 12:47:29] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:38 [I 2022-10-10 12:47:29] ...trap/engines/engine.py.229: process: 0.87560 | load: 0.00312 [I 2022-10-10 12:47:29] ...trap/engines/engine.py.230: loss: 0.15830 [I 2022-10-10 12:47:29] ...optimizers/trijoint.py.151: total_norm: 26.925421 [I 2022-10-10 12:47:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 1230/2304 [I 2022-10-10 12:47:38] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:28 [I 2022-10-10 12:47:38] ...trap/engines/engine.py.229: process: 0.86101 | load: 0.00309 [I 2022-10-10 12:47:38] ...trap/engines/engine.py.230: loss: 0.15035 [I 2022-10-10 12:47:38] ...optimizers/trijoint.py.151: total_norm: 21.408506 [I 2022-10-10 12:47:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 1240/2304 [I 2022-10-10 12:47:47] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:17 [I 2022-10-10 12:47:47] ...trap/engines/engine.py.229: process: 0.86759 | load: 0.00317 [I 2022-10-10 12:47:47] ...trap/engines/engine.py.230: loss: 0.20722 [I 2022-10-10 12:47:47] ...optimizers/trijoint.py.151: total_norm: 17.185217 [I 2022-10-10 12:47:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 1250/2304 [I 2022-10-10 12:47:55] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:06 [I 2022-10-10 12:47:55] ...trap/engines/engine.py.229: process: 0.85690 | load: 0.00313 [I 2022-10-10 12:47:55] ...trap/engines/engine.py.230: loss: 0.24489 [I 2022-10-10 12:47:55] ...optimizers/trijoint.py.151: total_norm: 24.671665 [I 2022-10-10 12:48:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 1260/2304 [I 2022-10-10 12:48:04] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:15:00 [I 2022-10-10 12:48:04] ...trap/engines/engine.py.229: process: 0.85674 | load: 0.00309 [I 2022-10-10 12:48:04] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-10 12:48:04] ...optimizers/trijoint.py.151: total_norm: 17.364048 [I 2022-10-10 12:48:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 1270/2304 [I 2022-10-10 12:48:12] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:53 [I 2022-10-10 12:48:12] ...trap/engines/engine.py.229: process: 0.87567 | load: 0.00316 [I 2022-10-10 12:48:12] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-10 12:48:12] ...optimizers/trijoint.py.151: total_norm: 15.625524 [I 2022-10-10 12:48:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 1280/2304 [I 2022-10-10 12:48:21] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:14:47 [I 2022-10-10 12:48:21] ...trap/engines/engine.py.229: process: 0.86928 | load: 0.00308 [I 2022-10-10 12:48:21] ...trap/engines/engine.py.230: loss: 0.20912 [I 2022-10-10 12:48:21] ...optimizers/trijoint.py.151: total_norm: 21.946369 [I 2022-10-10 12:48:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 1290/2304 [I 2022-10-10 12:48:30] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:36 [I 2022-10-10 12:48:30] ...trap/engines/engine.py.229: process: 0.88040 | load: 0.00302 [I 2022-10-10 12:48:30] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-10 12:48:30] ...optimizers/trijoint.py.151: total_norm: 28.871986 [I 2022-10-10 12:48:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 1300/2304 [I 2022-10-10 12:48:38] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:14:25 [I 2022-10-10 12:48:38] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00308 [I 2022-10-10 12:48:38] ...trap/engines/engine.py.230: loss: 0.09245 [I 2022-10-10 12:48:38] ...optimizers/trijoint.py.151: total_norm: 16.868563 [I 2022-10-10 12:48:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 1310/2304 [I 2022-10-10 12:48:47] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:16 [I 2022-10-10 12:48:47] ...trap/engines/engine.py.229: process: 0.85536 | load: 0.00316 [I 2022-10-10 12:48:47] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-10 12:48:47] ...optimizers/trijoint.py.151: total_norm: 16.858316 [I 2022-10-10 12:48:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 1320/2304 [I 2022-10-10 12:48:56] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:10 [I 2022-10-10 12:48:56] ...trap/engines/engine.py.229: process: 0.86414 | load: 0.00317 [I 2022-10-10 12:48:56] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-10 12:48:56] ...optimizers/trijoint.py.151: total_norm: 30.396828 [I 2022-10-10 12:49:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 1330/2304 [I 2022-10-10 12:49:04] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:14:00 [I 2022-10-10 12:49:04] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00319 [I 2022-10-10 12:49:04] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-10 12:49:04] ...optimizers/trijoint.py.151: total_norm: 12.359262 [I 2022-10-10 12:49:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 1340/2304 [I 2022-10-10 12:49:13] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:56 [I 2022-10-10 12:49:13] ...trap/engines/engine.py.229: process: 0.87978 | load: 0.00319 [I 2022-10-10 12:49:13] ...trap/engines/engine.py.230: loss: 0.16439 [I 2022-10-10 12:49:13] ...optimizers/trijoint.py.151: total_norm: 16.274918 [I 2022-10-10 12:49:22] ...trap/engines/engine.py.224: train: epoch 72 | batch 1350/2304 [I 2022-10-10 12:49:22] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:47 [I 2022-10-10 12:49:22] ...trap/engines/engine.py.229: process: 0.87048 | load: 0.00323 [I 2022-10-10 12:49:22] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-10 12:49:22] ...optimizers/trijoint.py.151: total_norm: 26.216331 [I 2022-10-10 12:49:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 1360/2304 [I 2022-10-10 12:49:30] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:40 [I 2022-10-10 12:49:30] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00318 [I 2022-10-10 12:49:30] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-10 12:49:30] ...optimizers/trijoint.py.151: total_norm: 42.586246 [I 2022-10-10 12:49:39] ...trap/engines/engine.py.224: train: epoch 72 | batch 1370/2304 [I 2022-10-10 12:49:39] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:23 [I 2022-10-10 12:49:39] ...trap/engines/engine.py.229: process: 0.85816 | load: 0.00316 [I 2022-10-10 12:49:39] ...trap/engines/engine.py.230: loss: 0.13069 [I 2022-10-10 12:49:39] ...optimizers/trijoint.py.151: total_norm: 13.210334 [I 2022-10-10 12:49:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 1380/2304 [I 2022-10-10 12:49:48] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:19 [I 2022-10-10 12:49:48] ...trap/engines/engine.py.229: process: 0.86435 | load: 0.00317 [I 2022-10-10 12:49:48] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-10 12:49:48] ...optimizers/trijoint.py.151: total_norm: 25.788017 [I 2022-10-10 12:49:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 1390/2304 [I 2022-10-10 12:49:56] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:10 [I 2022-10-10 12:49:56] ...trap/engines/engine.py.229: process: 0.86388 | load: 0.00312 [I 2022-10-10 12:49:56] ...trap/engines/engine.py.230: loss: 0.15478 [I 2022-10-10 12:49:56] ...optimizers/trijoint.py.151: total_norm: 24.562189 [I 2022-10-10 12:50:05] ...trap/engines/engine.py.224: train: epoch 72 | batch 1400/2304 [I 2022-10-10 12:50:05] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:58 [I 2022-10-10 12:50:05] ...trap/engines/engine.py.229: process: 0.86273 | load: 0.00353 [I 2022-10-10 12:50:05] ...trap/engines/engine.py.230: loss: 0.15447 [I 2022-10-10 12:50:05] ...optimizers/trijoint.py.151: total_norm: 34.011696 [I 2022-10-10 12:50:14] ...trap/engines/engine.py.224: train: epoch 72 | batch 1410/2304 [I 2022-10-10 12:50:14] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:56 [I 2022-10-10 12:50:14] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00315 [I 2022-10-10 12:50:14] ...trap/engines/engine.py.230: loss: 0.10104 [I 2022-10-10 12:50:14] ...optimizers/trijoint.py.151: total_norm: 8.382367 [I 2022-10-10 12:50:22] ...trap/engines/engine.py.224: train: epoch 72 | batch 1420/2304 [I 2022-10-10 12:50:22] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:45 [I 2022-10-10 12:50:22] ...trap/engines/engine.py.229: process: 0.86636 | load: 0.00312 [I 2022-10-10 12:50:22] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-10 12:50:22] ...optimizers/trijoint.py.151: total_norm: 13.863809 [I 2022-10-10 12:50:31] ...trap/engines/engine.py.224: train: epoch 72 | batch 1430/2304 [I 2022-10-10 12:50:31] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:35 [I 2022-10-10 12:50:31] ...trap/engines/engine.py.229: process: 0.86876 | load: 0.00306 [I 2022-10-10 12:50:31] ...trap/engines/engine.py.230: loss: 0.10016 [I 2022-10-10 12:50:31] ...optimizers/trijoint.py.151: total_norm: 10.505349 [I 2022-10-10 12:50:40] ...trap/engines/engine.py.224: train: epoch 72 | batch 1440/2304 [I 2022-10-10 12:50:40] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:28 [I 2022-10-10 12:50:40] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00308 [I 2022-10-10 12:50:40] ...trap/engines/engine.py.230: loss: 0.13220 [I 2022-10-10 12:50:40] ...optimizers/trijoint.py.151: total_norm: 21.731533 [I 2022-10-10 12:50:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 1450/2304 [I 2022-10-10 12:50:48] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:22 [I 2022-10-10 12:50:48] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00313 [I 2022-10-10 12:50:48] ...trap/engines/engine.py.230: loss: 0.19554 [I 2022-10-10 12:50:49] ...optimizers/trijoint.py.151: total_norm: 156.248077 [I 2022-10-10 12:50:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 1460/2304 [I 2022-10-10 12:50:57] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:12:13 [I 2022-10-10 12:50:57] ...trap/engines/engine.py.229: process: 0.85917 | load: 0.00310 [I 2022-10-10 12:50:57] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-10 12:50:57] ...optimizers/trijoint.py.151: total_norm: 8.681473 [I 2022-10-10 12:51:06] ...trap/engines/engine.py.224: train: epoch 72 | batch 1470/2304 [I 2022-10-10 12:51:06] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:12:01 [I 2022-10-10 12:51:06] ...trap/engines/engine.py.229: process: 0.86812 | load: 0.00308 [I 2022-10-10 12:51:06] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-10 12:51:06] ...optimizers/trijoint.py.151: total_norm: 12.376761 [I 2022-10-10 12:51:15] ...trap/engines/engine.py.224: train: epoch 72 | batch 1480/2304 [I 2022-10-10 12:51:15] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:50 [I 2022-10-10 12:51:15] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00306 [I 2022-10-10 12:51:15] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-10 12:51:15] ...optimizers/trijoint.py.151: total_norm: 16.092480 [I 2022-10-10 12:51:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 1490/2304 [I 2022-10-10 12:51:23] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:42 [I 2022-10-10 12:51:23] ...trap/engines/engine.py.229: process: 0.86157 | load: 0.00311 [I 2022-10-10 12:51:23] ...trap/engines/engine.py.230: loss: 0.11180 [I 2022-10-10 12:51:23] ...optimizers/trijoint.py.151: total_norm: 7.084299 [I 2022-10-10 12:51:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 1500/2304 [I 2022-10-10 12:51:32] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:36 [I 2022-10-10 12:51:32] ...trap/engines/engine.py.229: process: 0.87618 | load: 0.00312 [I 2022-10-10 12:51:32] ...trap/engines/engine.py.230: loss: 0.09718 [I 2022-10-10 12:51:32] ...optimizers/trijoint.py.151: total_norm: 27.107286 [I 2022-10-10 12:51:41] ...trap/engines/engine.py.224: train: epoch 72 | batch 1510/2304 [I 2022-10-10 12:51:41] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:25 [I 2022-10-10 12:51:41] ...trap/engines/engine.py.229: process: 0.85639 | load: 0.00308 [I 2022-10-10 12:51:41] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-10 12:51:41] ...optimizers/trijoint.py.151: total_norm: 18.316133 [I 2022-10-10 12:51:49] ...trap/engines/engine.py.224: train: epoch 72 | batch 1520/2304 [I 2022-10-10 12:51:49] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:18 [I 2022-10-10 12:51:49] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00308 [I 2022-10-10 12:51:49] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-10 12:51:49] ...optimizers/trijoint.py.151: total_norm: 8.453441 [I 2022-10-10 12:51:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 1530/2304 [I 2022-10-10 12:51:58] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:11:07 [I 2022-10-10 12:51:58] ...trap/engines/engine.py.229: process: 0.86244 | load: 0.00303 [I 2022-10-10 12:51:58] ...trap/engines/engine.py.230: loss: 0.10289 [I 2022-10-10 12:51:58] ...optimizers/trijoint.py.151: total_norm: 9.638931 [I 2022-10-10 12:52:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 1540/2304 [I 2022-10-10 12:52:07] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:59 [I 2022-10-10 12:52:07] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00306 [I 2022-10-10 12:52:07] ...trap/engines/engine.py.230: loss: 0.11410 [I 2022-10-10 12:52:07] ...optimizers/trijoint.py.151: total_norm: 40.130417 [I 2022-10-10 12:52:15] ...trap/engines/engine.py.224: train: epoch 72 | batch 1550/2304 [I 2022-10-10 12:52:15] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:49 [I 2022-10-10 12:52:15] ...trap/engines/engine.py.229: process: 0.86633 | load: 0.00314 [I 2022-10-10 12:52:15] ...trap/engines/engine.py.230: loss: 0.09893 [I 2022-10-10 12:52:15] ...optimizers/trijoint.py.151: total_norm: 13.765360 [I 2022-10-10 12:52:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 1560/2304 [I 2022-10-10 12:52:24] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:43 [I 2022-10-10 12:52:24] ...trap/engines/engine.py.229: process: 0.87503 | load: 0.00310 [I 2022-10-10 12:52:24] ...trap/engines/engine.py.230: loss: 0.15810 [I 2022-10-10 12:52:24] ...optimizers/trijoint.py.151: total_norm: 21.975843 [I 2022-10-10 12:52:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 1570/2304 [I 2022-10-10 12:52:32] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:33 [I 2022-10-10 12:52:32] ...trap/engines/engine.py.229: process: 0.84598 | load: 0.00317 [I 2022-10-10 12:52:32] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-10 12:52:32] ...optimizers/trijoint.py.151: total_norm: 18.559652 [I 2022-10-10 12:52:41] ...trap/engines/engine.py.224: train: epoch 72 | batch 1580/2304 [I 2022-10-10 12:52:41] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:24 [I 2022-10-10 12:52:41] ...trap/engines/engine.py.229: process: 0.86573 | load: 0.00310 [I 2022-10-10 12:52:41] ...trap/engines/engine.py.230: loss: 0.09337 [I 2022-10-10 12:52:41] ...optimizers/trijoint.py.151: total_norm: 13.481301 [I 2022-10-10 12:52:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 1590/2304 [I 2022-10-10 12:52:50] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:19 [I 2022-10-10 12:52:50] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00307 [I 2022-10-10 12:52:50] ...trap/engines/engine.py.230: loss: 0.13870 [I 2022-10-10 12:52:50] ...optimizers/trijoint.py.151: total_norm: 10.288991 [I 2022-10-10 12:52:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 1600/2304 [I 2022-10-10 12:52:58] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:10:06 [I 2022-10-10 12:52:58] ...trap/engines/engine.py.229: process: 0.85944 | load: 0.00311 [I 2022-10-10 12:52:58] ...trap/engines/engine.py.230: loss: 0.15200 [I 2022-10-10 12:52:58] ...optimizers/trijoint.py.151: total_norm: 10.238528 [I 2022-10-10 12:53:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 1610/2304 [I 2022-10-10 12:53:07] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:58 [I 2022-10-10 12:53:07] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00305 [I 2022-10-10 12:53:07] ...trap/engines/engine.py.230: loss: 0.13213 [I 2022-10-10 12:53:07] ...optimizers/trijoint.py.151: total_norm: 45.511562 [I 2022-10-10 12:53:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 1620/2304 [I 2022-10-10 12:53:16] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:51 [I 2022-10-10 12:53:16] ...trap/engines/engine.py.229: process: 0.86074 | load: 0.00308 [I 2022-10-10 12:53:16] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-10 12:53:16] ...optimizers/trijoint.py.151: total_norm: 14.315336 [I 2022-10-10 12:53:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 1630/2304 [I 2022-10-10 12:53:24] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:42 [I 2022-10-10 12:53:24] ...trap/engines/engine.py.229: process: 0.86679 | load: 0.00309 [I 2022-10-10 12:53:24] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-10 12:53:24] ...optimizers/trijoint.py.151: total_norm: 18.728704 [I 2022-10-10 12:53:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 1640/2304 [I 2022-10-10 12:53:33] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:35 [I 2022-10-10 12:53:33] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00324 [I 2022-10-10 12:53:33] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-10 12:53:33] ...optimizers/trijoint.py.151: total_norm: 12.713171 [I 2022-10-10 12:53:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 1650/2304 [I 2022-10-10 12:53:42] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:27 [I 2022-10-10 12:53:42] ...trap/engines/engine.py.229: process: 0.87099 | load: 0.00298 [I 2022-10-10 12:53:42] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-10 12:53:42] ...optimizers/trijoint.py.151: total_norm: 17.790968 [I 2022-10-10 12:53:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 1660/2304 [I 2022-10-10 12:53:50] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:19 [I 2022-10-10 12:53:50] ...trap/engines/engine.py.229: process: 0.86699 | load: 0.00316 [I 2022-10-10 12:53:50] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-10 12:53:50] ...optimizers/trijoint.py.151: total_norm: 16.128565 [I 2022-10-10 12:53:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 1670/2304 [I 2022-10-10 12:53:59] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:09:08 [I 2022-10-10 12:53:59] ...trap/engines/engine.py.229: process: 0.87403 | load: 0.00313 [I 2022-10-10 12:53:59] ...trap/engines/engine.py.230: loss: 0.11514 [I 2022-10-10 12:53:59] ...optimizers/trijoint.py.151: total_norm: 11.444892 [I 2022-10-10 12:54:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 1680/2304 [I 2022-10-10 12:54:08] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:59 [I 2022-10-10 12:54:08] ...trap/engines/engine.py.229: process: 0.87783 | load: 0.00306 [I 2022-10-10 12:54:08] ...trap/engines/engine.py.230: loss: 0.08894 [I 2022-10-10 12:54:08] ...optimizers/trijoint.py.151: total_norm: 9.394135 [I 2022-10-10 12:54:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 1690/2304 [I 2022-10-10 12:54:16] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:47 [I 2022-10-10 12:54:16] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00311 [I 2022-10-10 12:54:16] ...trap/engines/engine.py.230: loss: 0.11935 [I 2022-10-10 12:54:16] ...optimizers/trijoint.py.151: total_norm: 9.928962 [I 2022-10-10 12:54:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 1700/2304 [I 2022-10-10 12:54:25] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:42 [I 2022-10-10 12:54:25] ...trap/engines/engine.py.229: process: 0.86841 | load: 0.00313 [I 2022-10-10 12:54:25] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-10 12:54:25] ...optimizers/trijoint.py.151: total_norm: 18.810740 [I 2022-10-10 12:54:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 1710/2304 [I 2022-10-10 12:54:34] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:29 [I 2022-10-10 12:54:34] ...trap/engines/engine.py.229: process: 0.87104 | load: 0.00309 [I 2022-10-10 12:54:34] ...trap/engines/engine.py.230: loss: 0.10514 [I 2022-10-10 12:54:34] ...optimizers/trijoint.py.151: total_norm: 15.447801 [I 2022-10-10 12:54:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 1720/2304 [I 2022-10-10 12:54:42] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:26 [I 2022-10-10 12:54:42] ...trap/engines/engine.py.229: process: 0.88181 | load: 0.00311 [I 2022-10-10 12:54:42] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-10 12:54:42] ...optimizers/trijoint.py.151: total_norm: 20.359489 [I 2022-10-10 12:54:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 1730/2304 [I 2022-10-10 12:54:51] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:13 [I 2022-10-10 12:54:51] ...trap/engines/engine.py.229: process: 0.85713 | load: 0.00308 [I 2022-10-10 12:54:51] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-10 12:54:51] ...optimizers/trijoint.py.151: total_norm: 9.140841 [I 2022-10-10 12:55:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 1740/2304 [I 2022-10-10 12:55:00] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:08:01 [I 2022-10-10 12:55:00] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00309 [I 2022-10-10 12:55:00] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-10 12:55:00] ...optimizers/trijoint.py.151: total_norm: 61.101215 [I 2022-10-10 12:55:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 1750/2304 [I 2022-10-10 12:55:08] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:58 [I 2022-10-10 12:55:08] ...trap/engines/engine.py.229: process: 0.86740 | load: 0.00305 [I 2022-10-10 12:55:08] ...trap/engines/engine.py.230: loss: 0.11897 [I 2022-10-10 12:55:08] ...optimizers/trijoint.py.151: total_norm: 32.810719 [I 2022-10-10 12:55:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 1760/2304 [I 2022-10-10 12:55:17] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:07:48 [I 2022-10-10 12:55:17] ...trap/engines/engine.py.229: process: 0.85692 | load: 0.00310 [I 2022-10-10 12:55:17] ...trap/engines/engine.py.230: loss: 0.16056 [I 2022-10-10 12:55:17] ...optimizers/trijoint.py.151: total_norm: 13.461583 [I 2022-10-10 12:55:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 1770/2304 [I 2022-10-10 12:55:25] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:40 [I 2022-10-10 12:55:25] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00312 [I 2022-10-10 12:55:25] ...trap/engines/engine.py.230: loss: 0.17282 [I 2022-10-10 12:55:25] ...optimizers/trijoint.py.151: total_norm: 30.742523 [I 2022-10-10 12:55:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 1780/2304 [I 2022-10-10 12:55:34] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:29 [I 2022-10-10 12:55:34] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00307 [I 2022-10-10 12:55:34] ...trap/engines/engine.py.230: loss: 0.15132 [I 2022-10-10 12:55:34] ...optimizers/trijoint.py.151: total_norm: 10.746316 [I 2022-10-10 12:55:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 1790/2304 [I 2022-10-10 12:55:43] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:23 [I 2022-10-10 12:55:43] ...trap/engines/engine.py.229: process: 0.87789 | load: 0.00303 [I 2022-10-10 12:55:43] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-10 12:55:43] ...optimizers/trijoint.py.151: total_norm: 18.929077 [I 2022-10-10 12:55:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 1800/2304 [I 2022-10-10 12:55:51] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:17 [I 2022-10-10 12:55:51] ...trap/engines/engine.py.229: process: 0.87462 | load: 0.00319 [I 2022-10-10 12:55:51] ...trap/engines/engine.py.230: loss: 0.15110 [I 2022-10-10 12:55:51] ...optimizers/trijoint.py.151: total_norm: 13.022213 [I 2022-10-10 12:56:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 1810/2304 [I 2022-10-10 12:56:00] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:03 [I 2022-10-10 12:56:00] ...trap/engines/engine.py.229: process: 0.85768 | load: 0.00310 [I 2022-10-10 12:56:00] ...trap/engines/engine.py.230: loss: 0.15816 [I 2022-10-10 12:56:00] ...optimizers/trijoint.py.151: total_norm: 8.112581 [I 2022-10-10 12:56:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 1820/2304 [I 2022-10-10 12:56:09] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:58 [I 2022-10-10 12:56:09] ...trap/engines/engine.py.229: process: 0.86696 | load: 0.00307 [I 2022-10-10 12:56:09] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-10 12:56:09] ...optimizers/trijoint.py.151: total_norm: 15.617814 [I 2022-10-10 12:56:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 1830/2304 [I 2022-10-10 12:56:17] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:50 [I 2022-10-10 12:56:17] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00306 [I 2022-10-10 12:56:17] ...trap/engines/engine.py.230: loss: 0.21022 [I 2022-10-10 12:56:17] ...optimizers/trijoint.py.151: total_norm: 17.055525 [I 2022-10-10 12:56:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 1840/2304 [I 2022-10-10 12:56:26] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:39 [I 2022-10-10 12:56:26] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00307 [I 2022-10-10 12:56:26] ...trap/engines/engine.py.230: loss: 0.15909 [I 2022-10-10 12:56:26] ...optimizers/trijoint.py.151: total_norm: 27.304861 [I 2022-10-10 12:56:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 1850/2304 [I 2022-10-10 12:56:35] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:30 [I 2022-10-10 12:56:35] ...trap/engines/engine.py.229: process: 0.86621 | load: 0.00307 [I 2022-10-10 12:56:35] ...trap/engines/engine.py.230: loss: 0.12678 [I 2022-10-10 12:56:35] ...optimizers/trijoint.py.151: total_norm: 9.957688 [I 2022-10-10 12:56:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 1860/2304 [I 2022-10-10 12:56:43] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:24 [I 2022-10-10 12:56:43] ...trap/engines/engine.py.229: process: 0.86364 | load: 0.00317 [I 2022-10-10 12:56:43] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-10 12:56:43] ...optimizers/trijoint.py.151: total_norm: 16.201057 [I 2022-10-10 12:56:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 1870/2304 [I 2022-10-10 12:56:52] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:11 [I 2022-10-10 12:56:52] ...trap/engines/engine.py.229: process: 0.85453 | load: 0.00307 [I 2022-10-10 12:56:52] ...trap/engines/engine.py.230: loss: 0.16603 [I 2022-10-10 12:56:52] ...optimizers/trijoint.py.151: total_norm: 27.546364 [I 2022-10-10 12:57:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 1880/2304 [I 2022-10-10 12:57:00] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:06 [I 2022-10-10 12:57:00] ...trap/engines/engine.py.229: process: 0.87346 | load: 0.00308 [I 2022-10-10 12:57:00] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-10 12:57:00] ...optimizers/trijoint.py.151: total_norm: 14.960396 [I 2022-10-10 12:57:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 1890/2304 [I 2022-10-10 12:57:09] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:05:58 [I 2022-10-10 12:57:09] ...trap/engines/engine.py.229: process: 0.86836 | load: 0.00341 [I 2022-10-10 12:57:09] ...trap/engines/engine.py.230: loss: 0.11468 [I 2022-10-10 12:57:09] ...optimizers/trijoint.py.151: total_norm: 14.536870 [I 2022-10-10 12:57:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 1900/2304 [I 2022-10-10 12:57:18] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:50 [I 2022-10-10 12:57:18] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00305 [I 2022-10-10 12:57:18] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-10 12:57:18] ...optimizers/trijoint.py.151: total_norm: 19.792408 [I 2022-10-10 12:57:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 1910/2304 [I 2022-10-10 12:57:27] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:40 [I 2022-10-10 12:57:27] ...trap/engines/engine.py.229: process: 0.86951 | load: 0.00313 [I 2022-10-10 12:57:27] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-10 12:57:27] ...optimizers/trijoint.py.151: total_norm: 11.695714 [I 2022-10-10 12:57:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 1920/2304 [I 2022-10-10 12:57:35] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:05:33 [I 2022-10-10 12:57:35] ...trap/engines/engine.py.229: process: 0.86250 | load: 0.00316 [I 2022-10-10 12:57:35] ...trap/engines/engine.py.230: loss: 0.09079 [I 2022-10-10 12:57:35] ...optimizers/trijoint.py.151: total_norm: 8.783618 [I 2022-10-10 12:57:44] ...trap/engines/engine.py.224: train: epoch 72 | batch 1930/2304 [I 2022-10-10 12:57:44] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:24 [I 2022-10-10 12:57:44] ...trap/engines/engine.py.229: process: 0.86410 | load: 0.00431 [I 2022-10-10 12:57:44] ...trap/engines/engine.py.230: loss: 0.12198 [I 2022-10-10 12:57:44] ...optimizers/trijoint.py.151: total_norm: 12.282623 [I 2022-10-10 12:57:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 1940/2304 [I 2022-10-10 12:57:53] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:05:16 [I 2022-10-10 12:57:53] ...trap/engines/engine.py.229: process: 0.87694 | load: 0.00311 [I 2022-10-10 12:57:53] ...trap/engines/engine.py.230: loss: 0.14546 [I 2022-10-10 12:57:53] ...optimizers/trijoint.py.151: total_norm: 36.666473 [I 2022-10-10 12:58:01] ...trap/engines/engine.py.224: train: epoch 72 | batch 1950/2304 [I 2022-10-10 12:58:01] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:06 [I 2022-10-10 12:58:01] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00308 [I 2022-10-10 12:58:01] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-10 12:58:01] ...optimizers/trijoint.py.151: total_norm: 12.677475 [I 2022-10-10 12:58:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 1960/2304 [I 2022-10-10 12:58:10] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:57 [I 2022-10-10 12:58:10] ...trap/engines/engine.py.229: process: 0.86387 | load: 0.00336 [I 2022-10-10 12:58:10] ...trap/engines/engine.py.230: loss: 0.15016 [I 2022-10-10 12:58:10] ...optimizers/trijoint.py.151: total_norm: 22.539104 [I 2022-10-10 12:58:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 1970/2304 [I 2022-10-10 12:58:19] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:48 [I 2022-10-10 12:58:19] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00310 [I 2022-10-10 12:58:19] ...trap/engines/engine.py.230: loss: 0.18010 [I 2022-10-10 12:58:19] ...optimizers/trijoint.py.151: total_norm: 20.571951 [I 2022-10-10 12:58:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 1980/2304 [I 2022-10-10 12:58:27] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:40 [I 2022-10-10 12:58:27] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00309 [I 2022-10-10 12:58:27] ...trap/engines/engine.py.230: loss: 0.10914 [I 2022-10-10 12:58:27] ...optimizers/trijoint.py.151: total_norm: 19.401533 [I 2022-10-10 12:58:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 1990/2304 [I 2022-10-10 12:58:36] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:31 [I 2022-10-10 12:58:36] ...trap/engines/engine.py.229: process: 0.87628 | load: 0.00305 [I 2022-10-10 12:58:36] ...trap/engines/engine.py.230: loss: 0.09860 [I 2022-10-10 12:58:36] ...optimizers/trijoint.py.151: total_norm: 12.547709 [I 2022-10-10 12:58:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 2000/2304 [I 2022-10-10 12:58:45] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:21 [I 2022-10-10 12:58:45] ...trap/engines/engine.py.229: process: 0.86473 | load: 0.00313 [I 2022-10-10 12:58:45] ...trap/engines/engine.py.230: loss: 0.10996 [I 2022-10-10 12:58:45] ...optimizers/trijoint.py.151: total_norm: 11.147860 [I 2022-10-10 12:58:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 2010/2304 [I 2022-10-10 12:58:53] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:14 [I 2022-10-10 12:58:53] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00330 [I 2022-10-10 12:58:53] ...trap/engines/engine.py.230: loss: 0.15555 [I 2022-10-10 12:58:53] ...optimizers/trijoint.py.151: total_norm: 16.494831 [I 2022-10-10 12:59:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 2020/2304 [I 2022-10-10 12:59:02] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:06 [I 2022-10-10 12:59:02] ...trap/engines/engine.py.229: process: 0.86331 | load: 0.00317 [I 2022-10-10 12:59:02] ...trap/engines/engine.py.230: loss: 0.10436 [I 2022-10-10 12:59:02] ...optimizers/trijoint.py.151: total_norm: 8.539681 [I 2022-10-10 12:59:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 2030/2304 [I 2022-10-10 12:59:11] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:55 [I 2022-10-10 12:59:11] ...trap/engines/engine.py.229: process: 0.87821 | load: 0.00305 [I 2022-10-10 12:59:11] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-10 12:59:11] ...optimizers/trijoint.py.151: total_norm: 14.012622 [I 2022-10-10 12:59:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 2040/2304 [I 2022-10-10 12:59:19] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:48 [I 2022-10-10 12:59:19] ...trap/engines/engine.py.229: process: 0.86577 | load: 0.00310 [I 2022-10-10 12:59:19] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-10 12:59:19] ...optimizers/trijoint.py.151: total_norm: 14.863670 [I 2022-10-10 12:59:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 2050/2304 [I 2022-10-10 12:59:28] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:39 [I 2022-10-10 12:59:28] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00313 [I 2022-10-10 12:59:28] ...trap/engines/engine.py.230: loss: 0.19323 [I 2022-10-10 12:59:28] ...optimizers/trijoint.py.151: total_norm: 17.813593 [I 2022-10-10 12:59:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 2060/2304 [I 2022-10-10 12:59:37] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:32 [I 2022-10-10 12:59:37] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00315 [I 2022-10-10 12:59:37] ...trap/engines/engine.py.230: loss: 0.12712 [I 2022-10-10 12:59:37] ...optimizers/trijoint.py.151: total_norm: 21.791441 [I 2022-10-10 12:59:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 2070/2304 [I 2022-10-10 12:59:45] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:03:22 [I 2022-10-10 12:59:45] ...trap/engines/engine.py.229: process: 0.85268 | load: 0.00314 [I 2022-10-10 12:59:45] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-10 12:59:45] ...optimizers/trijoint.py.151: total_norm: 16.633112 [I 2022-10-10 12:59:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 2080/2304 [I 2022-10-10 12:59:54] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:13 [I 2022-10-10 12:59:54] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00311 [I 2022-10-10 12:59:54] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-10 12:59:54] ...optimizers/trijoint.py.151: total_norm: 27.688341 [I 2022-10-10 13:00:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 2090/2304 [I 2022-10-10 13:00:03] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:03:05 [I 2022-10-10 13:00:03] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00300 [I 2022-10-10 13:00:03] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-10 13:00:03] ...optimizers/trijoint.py.151: total_norm: 17.220625 [I 2022-10-10 13:00:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 2100/2304 [I 2022-10-10 13:00:11] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:56 [I 2022-10-10 13:00:11] ...trap/engines/engine.py.229: process: 0.87412 | load: 0.00309 [I 2022-10-10 13:00:11] ...trap/engines/engine.py.230: loss: 0.13255 [I 2022-10-10 13:00:11] ...optimizers/trijoint.py.151: total_norm: 17.482056 [I 2022-10-10 13:00:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 2110/2304 [I 2022-10-10 13:00:20] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:48 [I 2022-10-10 13:00:20] ...trap/engines/engine.py.229: process: 0.88612 | load: 0.00310 [I 2022-10-10 13:00:20] ...trap/engines/engine.py.230: loss: 0.20883 [I 2022-10-10 13:00:20] ...optimizers/trijoint.py.151: total_norm: 26.443663 [I 2022-10-10 13:00:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 2120/2304 [I 2022-10-10 13:00:29] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:39 [I 2022-10-10 13:00:29] ...trap/engines/engine.py.229: process: 0.87074 | load: 0.00308 [I 2022-10-10 13:00:29] ...trap/engines/engine.py.230: loss: 0.15352 [I 2022-10-10 13:00:29] ...optimizers/trijoint.py.151: total_norm: 20.239595 [I 2022-10-10 13:00:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 2130/2304 [I 2022-10-10 13:00:37] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:30 [I 2022-10-10 13:00:37] ...trap/engines/engine.py.229: process: 0.86052 | load: 0.00307 [I 2022-10-10 13:00:37] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-10 13:00:37] ...optimizers/trijoint.py.151: total_norm: 14.666350 [I 2022-10-10 13:00:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 2140/2304 [I 2022-10-10 13:00:46] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:21 [I 2022-10-10 13:00:46] ...trap/engines/engine.py.229: process: 0.86260 | load: 0.00306 [I 2022-10-10 13:00:46] ...trap/engines/engine.py.230: loss: 0.14269 [I 2022-10-10 13:00:46] ...optimizers/trijoint.py.151: total_norm: 15.634508 [I 2022-10-10 13:00:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 2150/2304 [I 2022-10-10 13:00:55] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:12 [I 2022-10-10 13:00:55] ...trap/engines/engine.py.229: process: 0.86425 | load: 0.00309 [I 2022-10-10 13:00:55] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-10 13:00:55] ...optimizers/trijoint.py.151: total_norm: 10.868414 [I 2022-10-10 13:01:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 2160/2304 [I 2022-10-10 13:01:03] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:02:04 [I 2022-10-10 13:01:03] ...trap/engines/engine.py.229: process: 0.87904 | load: 0.00324 [I 2022-10-10 13:01:03] ...trap/engines/engine.py.230: loss: 0.21435 [I 2022-10-10 13:01:03] ...optimizers/trijoint.py.151: total_norm: 33.240608 [I 2022-10-10 13:01:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 2170/2304 [I 2022-10-10 13:01:12] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:56 [I 2022-10-10 13:01:12] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00312 [I 2022-10-10 13:01:12] ...trap/engines/engine.py.230: loss: 0.10021 [I 2022-10-10 13:01:12] ...optimizers/trijoint.py.151: total_norm: 12.362745 [I 2022-10-10 13:01:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 2180/2304 [I 2022-10-10 13:01:20] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:47 [I 2022-10-10 13:01:20] ...trap/engines/engine.py.229: process: 0.86839 | load: 0.00307 [I 2022-10-10 13:01:20] ...trap/engines/engine.py.230: loss: 0.18438 [I 2022-10-10 13:01:20] ...optimizers/trijoint.py.151: total_norm: 25.210009 [I 2022-10-10 13:01:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 2190/2304 [I 2022-10-10 13:01:29] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:39 [I 2022-10-10 13:01:29] ...trap/engines/engine.py.229: process: 0.87443 | load: 0.00310 [I 2022-10-10 13:01:29] ...trap/engines/engine.py.230: loss: 0.11807 [I 2022-10-10 13:01:29] ...optimizers/trijoint.py.151: total_norm: 18.466490 [I 2022-10-10 13:01:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 2200/2304 [I 2022-10-10 13:01:38] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:29 [I 2022-10-10 13:01:38] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00311 [I 2022-10-10 13:01:38] ...trap/engines/engine.py.230: loss: 0.11700 [I 2022-10-10 13:01:38] ...optimizers/trijoint.py.151: total_norm: 13.204010 [I 2022-10-10 13:01:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 2210/2304 [I 2022-10-10 13:01:47] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:20 [I 2022-10-10 13:01:47] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00291 [I 2022-10-10 13:01:47] ...trap/engines/engine.py.230: loss: 0.11088 [I 2022-10-10 13:01:47] ...optimizers/trijoint.py.151: total_norm: 8.333492 [I 2022-10-10 13:01:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 2220/2304 [I 2022-10-10 13:01:55] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:01:11 [I 2022-10-10 13:01:55] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00290 [I 2022-10-10 13:01:55] ...trap/engines/engine.py.230: loss: 0.15679 [I 2022-10-10 13:01:55] ...optimizers/trijoint.py.151: total_norm: 24.454454 [I 2022-10-10 13:02:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 2230/2304 [I 2022-10-10 13:02:04] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:01:02 [I 2022-10-10 13:02:04] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00295 [I 2022-10-10 13:02:04] ...trap/engines/engine.py.230: loss: 0.15969 [I 2022-10-10 13:02:04] ...optimizers/trijoint.py.151: total_norm: 34.976898 [I 2022-10-10 13:02:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 2240/2304 [I 2022-10-10 13:02:12] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:54 [I 2022-10-10 13:02:12] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00299 [I 2022-10-10 13:02:12] ...trap/engines/engine.py.230: loss: 0.09600 [I 2022-10-10 13:02:12] ...optimizers/trijoint.py.151: total_norm: 8.946204 [I 2022-10-10 13:02:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 2250/2304 [I 2022-10-10 13:02:21] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:46 [I 2022-10-10 13:02:21] ...trap/engines/engine.py.229: process: 0.86411 | load: 0.00294 [I 2022-10-10 13:02:21] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-10 13:02:21] ...optimizers/trijoint.py.151: total_norm: 13.345163 [I 2022-10-10 13:02:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 2260/2304 [I 2022-10-10 13:02:30] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:37 [I 2022-10-10 13:02:30] ...trap/engines/engine.py.229: process: 0.84272 | load: 0.00299 [I 2022-10-10 13:02:30] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-10 13:02:30] ...optimizers/trijoint.py.151: total_norm: 12.522291 [I 2022-10-10 13:02:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 2270/2304 [I 2022-10-10 13:02:38] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:28 [I 2022-10-10 13:02:38] ...trap/engines/engine.py.229: process: 0.84041 | load: 0.00294 [I 2022-10-10 13:02:38] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-10 13:02:38] ...optimizers/trijoint.py.151: total_norm: 23.032951 [I 2022-10-10 13:02:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 2280/2304 [I 2022-10-10 13:02:47] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:20 [I 2022-10-10 13:02:47] ...trap/engines/engine.py.229: process: 0.86107 | load: 0.00301 [I 2022-10-10 13:02:47] ...trap/engines/engine.py.230: loss: 0.10246 [I 2022-10-10 13:02:47] ...optimizers/trijoint.py.151: total_norm: 18.925259 [I 2022-10-10 13:02:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 2290/2304 [I 2022-10-10 13:02:55] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:11 [I 2022-10-10 13:02:55] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00299 [I 2022-10-10 13:02:55] ...trap/engines/engine.py.230: loss: 0.09496 [I 2022-10-10 13:02:55] ...optimizers/trijoint.py.151: total_norm: 9.405744 [I 2022-10-10 13:03:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 2300/2304 [I 2022-10-10 13:03:04] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:03 [I 2022-10-10 13:03:04] ...trap/engines/engine.py.229: process: 0.83250 | load: 0.00016 [I 2022-10-10 13:03:04] ...trap/engines/engine.py.230: loss: 0.16307 [I 2022-10-10 13:03:04] ...optimizers/trijoint.py.151: total_norm: 15.518725 [I 2022-10-10 13:03:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 2304/2304 [I 2022-10-10 13:03:07] ...trap/engines/engine.py.228: elapsed: 0:33:12 | left: 0:00:00 [I 2022-10-10 13:03:07] ...trap/engines/engine.py.229: process: 0.85349 | load: 0.00016 [I 2022-10-10 13:03:07] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-10 13:03:07] ...optimizers/trijoint.py.151: total_norm: 18.805880 [S 2022-10-10 13:03:08] ...trap/engines/engine.py.129: train_epoch.epoch: 72 [S 2022-10-10 13:03:08] ...trap/engines/engine.py.129: train_epoch.loss: 0.13862047589458767 [I 2022-10-10 13:03:08] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 13:03:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 72 [I 2022-10-10 13:03:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 13:03:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 13:03:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 13:03:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 13:03:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 13:03:15] ...trap/engines/engine.py.321: eval: epoch 72 | batch 0/510 [I 2022-10-10 13:03:15] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:42:08 [I 2022-10-10 13:03:15] ...trap/engines/engine.py.326: process: 4.95760 | load: 4.41322 [I 2022-10-10 13:03:19] ...trap/engines/engine.py.321: eval: epoch 72 | batch 10/510 [I 2022-10-10 13:03:19] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:06 [I 2022-10-10 13:03:19] ...trap/engines/engine.py.326: process: 0.50676 | load: 0.00019 [I 2022-10-10 13:03:25] ...trap/engines/engine.py.321: eval: epoch 72 | batch 20/510 [I 2022-10-10 13:03:25] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:22 [I 2022-10-10 13:03:25] ...trap/engines/engine.py.326: process: 0.43852 | load: 0.00022 [I 2022-10-10 13:03:31] ...trap/engines/engine.py.321: eval: epoch 72 | batch 30/510 [I 2022-10-10 13:03:31] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:05:09 [I 2022-10-10 13:03:31] ...trap/engines/engine.py.326: process: 0.68791 | load: 0.00019 [I 2022-10-10 13:03:37] ...trap/engines/engine.py.321: eval: epoch 72 | batch 40/510 [I 2022-10-10 13:03:37] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:32 [I 2022-10-10 13:03:37] ...trap/engines/engine.py.326: process: 0.50483 | load: 0.00019 [I 2022-10-10 13:03:42] ...trap/engines/engine.py.321: eval: epoch 72 | batch 50/510 [I 2022-10-10 13:03:42] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:42 [I 2022-10-10 13:03:42] ...trap/engines/engine.py.326: process: 0.45973 | load: 0.00018 [I 2022-10-10 13:03:47] ...trap/engines/engine.py.321: eval: epoch 72 | batch 60/510 [I 2022-10-10 13:03:47] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:40 [I 2022-10-10 13:03:47] ...trap/engines/engine.py.326: process: 0.49966 | load: 0.00018 [I 2022-10-10 13:03:52] ...trap/engines/engine.py.321: eval: epoch 72 | batch 70/510 [I 2022-10-10 13:03:52] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:50 [I 2022-10-10 13:03:52] ...trap/engines/engine.py.326: process: 0.56910 | load: 0.00019 [I 2022-10-10 13:03:57] ...trap/engines/engine.py.321: eval: epoch 72 | batch 80/510 [I 2022-10-10 13:03:57] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:40 [I 2022-10-10 13:03:57] ...trap/engines/engine.py.326: process: 0.45768 | load: 0.00019 [I 2022-10-10 13:04:02] ...trap/engines/engine.py.321: eval: epoch 72 | batch 90/510 [I 2022-10-10 13:04:02] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:06 [I 2022-10-10 13:04:02] ...trap/engines/engine.py.326: process: 0.44368 | load: 0.00018 [I 2022-10-10 13:04:06] ...trap/engines/engine.py.321: eval: epoch 72 | batch 100/510 [I 2022-10-10 13:04:06] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:47 [I 2022-10-10 13:04:06] ...trap/engines/engine.py.326: process: 0.31181 | load: 0.00019 [I 2022-10-10 13:04:09] ...trap/engines/engine.py.321: eval: epoch 72 | batch 110/510 [I 2022-10-10 13:04:09] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:10 [I 2022-10-10 13:04:09] ...trap/engines/engine.py.326: process: 0.31127 | load: 0.00019 [I 2022-10-10 13:04:12] ...trap/engines/engine.py.321: eval: epoch 72 | batch 120/510 [I 2022-10-10 13:04:12] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:04 [I 2022-10-10 13:04:12] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00018 [I 2022-10-10 13:04:15] ...trap/engines/engine.py.321: eval: epoch 72 | batch 130/510 [I 2022-10-10 13:04:15] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:00 [I 2022-10-10 13:04:15] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00019 [I 2022-10-10 13:04:19] ...trap/engines/engine.py.321: eval: epoch 72 | batch 140/510 [I 2022-10-10 13:04:19] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:57 [I 2022-10-10 13:04:19] ...trap/engines/engine.py.326: process: 0.32169 | load: 0.00019 [I 2022-10-10 13:04:22] ...trap/engines/engine.py.321: eval: epoch 72 | batch 150/510 [I 2022-10-10 13:04:22] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:54 [I 2022-10-10 13:04:22] ...trap/engines/engine.py.326: process: 0.31387 | load: 0.00021 [I 2022-10-10 13:04:25] ...trap/engines/engine.py.321: eval: epoch 72 | batch 160/510 [I 2022-10-10 13:04:25] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:52 [I 2022-10-10 13:04:25] ...trap/engines/engine.py.326: process: 0.32243 | load: 0.00021 [I 2022-10-10 13:04:28] ...trap/engines/engine.py.321: eval: epoch 72 | batch 170/510 [I 2022-10-10 13:04:28] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:49 [I 2022-10-10 13:04:28] ...trap/engines/engine.py.326: process: 0.31472 | load: 0.00020 [I 2022-10-10 13:04:31] ...trap/engines/engine.py.321: eval: epoch 72 | batch 180/510 [I 2022-10-10 13:04:31] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:45 [I 2022-10-10 13:04:31] ...trap/engines/engine.py.326: process: 0.31991 | load: 0.00021 [I 2022-10-10 13:04:35] ...trap/engines/engine.py.321: eval: epoch 72 | batch 190/510 [I 2022-10-10 13:04:35] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:42 [I 2022-10-10 13:04:35] ...trap/engines/engine.py.326: process: 0.32241 | load: 0.00021 [I 2022-10-10 13:04:38] ...trap/engines/engine.py.321: eval: epoch 72 | batch 200/510 [I 2022-10-10 13:04:38] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:39 [I 2022-10-10 13:04:38] ...trap/engines/engine.py.326: process: 0.32226 | load: 0.00020 [I 2022-10-10 13:04:41] ...trap/engines/engine.py.321: eval: epoch 72 | batch 210/510 [I 2022-10-10 13:04:41] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:36 [I 2022-10-10 13:04:41] ...trap/engines/engine.py.326: process: 0.32809 | load: 0.00020 [I 2022-10-10 13:04:44] ...trap/engines/engine.py.321: eval: epoch 72 | batch 220/510 [I 2022-10-10 13:04:44] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:33 [I 2022-10-10 13:04:44] ...trap/engines/engine.py.326: process: 0.32822 | load: 0.00020 [I 2022-10-10 13:04:47] ...trap/engines/engine.py.321: eval: epoch 72 | batch 230/510 [I 2022-10-10 13:04:47] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:30 [I 2022-10-10 13:04:47] ...trap/engines/engine.py.326: process: 0.32873 | load: 0.00022 [I 2022-10-10 13:04:51] ...trap/engines/engine.py.321: eval: epoch 72 | batch 240/510 [I 2022-10-10 13:04:51] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-10 13:04:51] ...trap/engines/engine.py.326: process: 0.32106 | load: 0.00020 [I 2022-10-10 13:04:54] ...trap/engines/engine.py.321: eval: epoch 72 | batch 250/510 [I 2022-10-10 13:04:54] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:24 [I 2022-10-10 13:04:54] ...trap/engines/engine.py.326: process: 0.32280 | load: 0.00021 [I 2022-10-10 13:04:57] ...trap/engines/engine.py.321: eval: epoch 72 | batch 260/510 [I 2022-10-10 13:04:57] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:20 [I 2022-10-10 13:04:57] ...trap/engines/engine.py.326: process: 0.32113 | load: 0.00022 [I 2022-10-10 13:05:00] ...trap/engines/engine.py.321: eval: epoch 72 | batch 270/510 [I 2022-10-10 13:05:00] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:17 [I 2022-10-10 13:05:00] ...trap/engines/engine.py.326: process: 0.32299 | load: 0.00021 [I 2022-10-10 13:05:04] ...trap/engines/engine.py.321: eval: epoch 72 | batch 280/510 [I 2022-10-10 13:05:04] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:13 [I 2022-10-10 13:05:04] ...trap/engines/engine.py.326: process: 0.32347 | load: 0.00022 [I 2022-10-10 13:05:07] ...trap/engines/engine.py.321: eval: epoch 72 | batch 290/510 [I 2022-10-10 13:05:07] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:11 [I 2022-10-10 13:05:07] ...trap/engines/engine.py.326: process: 0.32486 | load: 0.00021 [I 2022-10-10 13:05:10] ...trap/engines/engine.py.321: eval: epoch 72 | batch 300/510 [I 2022-10-10 13:05:10] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:08 [I 2022-10-10 13:05:10] ...trap/engines/engine.py.326: process: 0.32343 | load: 0.00022 [I 2022-10-10 13:05:13] ...trap/engines/engine.py.321: eval: epoch 72 | batch 310/510 [I 2022-10-10 13:05:13] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:04 [I 2022-10-10 13:05:13] ...trap/engines/engine.py.326: process: 0.32518 | load: 0.00021 [I 2022-10-10 13:05:17] ...trap/engines/engine.py.321: eval: epoch 72 | batch 320/510 [I 2022-10-10 13:05:17] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:01 [I 2022-10-10 13:05:17] ...trap/engines/engine.py.326: process: 0.32292 | load: 0.00021 [I 2022-10-10 13:05:20] ...trap/engines/engine.py.321: eval: epoch 72 | batch 330/510 [I 2022-10-10 13:05:20] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:58 [I 2022-10-10 13:05:20] ...trap/engines/engine.py.326: process: 0.32867 | load: 0.00022 [I 2022-10-10 13:05:23] ...trap/engines/engine.py.321: eval: epoch 72 | batch 340/510 [I 2022-10-10 13:05:23] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-10 13:05:23] ...trap/engines/engine.py.326: process: 0.31660 | load: 0.00021 [I 2022-10-10 13:05:26] ...trap/engines/engine.py.321: eval: epoch 72 | batch 350/510 [I 2022-10-10 13:05:26] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:52 [I 2022-10-10 13:05:26] ...trap/engines/engine.py.326: process: 0.32676 | load: 0.00021 [I 2022-10-10 13:05:30] ...trap/engines/engine.py.321: eval: epoch 72 | batch 360/510 [I 2022-10-10 13:05:30] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:48 [I 2022-10-10 13:05:30] ...trap/engines/engine.py.326: process: 0.32503 | load: 0.00022 [I 2022-10-10 13:05:33] ...trap/engines/engine.py.321: eval: epoch 72 | batch 370/510 [I 2022-10-10 13:05:33] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:45 [I 2022-10-10 13:05:33] ...trap/engines/engine.py.326: process: 0.32477 | load: 0.00021 [I 2022-10-10 13:05:36] ...trap/engines/engine.py.321: eval: epoch 72 | batch 380/510 [I 2022-10-10 13:05:36] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:42 [I 2022-10-10 13:05:36] ...trap/engines/engine.py.326: process: 0.32774 | load: 0.00021 [I 2022-10-10 13:05:39] ...trap/engines/engine.py.321: eval: epoch 72 | batch 390/510 [I 2022-10-10 13:05:39] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-10 13:05:39] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00021 [I 2022-10-10 13:05:42] ...trap/engines/engine.py.321: eval: epoch 72 | batch 400/510 [I 2022-10-10 13:05:42] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-10 13:05:42] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00022 [I 2022-10-10 13:05:46] ...trap/engines/engine.py.321: eval: epoch 72 | batch 410/510 [I 2022-10-10 13:05:46] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:32 [I 2022-10-10 13:05:46] ...trap/engines/engine.py.326: process: 0.32845 | load: 0.00021 [I 2022-10-10 13:05:49] ...trap/engines/engine.py.321: eval: epoch 72 | batch 420/510 [I 2022-10-10 13:05:49] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:29 [I 2022-10-10 13:05:49] ...trap/engines/engine.py.326: process: 0.32146 | load: 0.00021 [I 2022-10-10 13:05:52] ...trap/engines/engine.py.321: eval: epoch 72 | batch 430/510 [I 2022-10-10 13:05:52] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-10 13:05:52] ...trap/engines/engine.py.326: process: 0.31805 | load: 0.00021 [I 2022-10-10 13:05:55] ...trap/engines/engine.py.321: eval: epoch 72 | batch 440/510 [I 2022-10-10 13:05:55] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-10 13:05:55] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00021 [I 2022-10-10 13:05:59] ...trap/engines/engine.py.321: eval: epoch 72 | batch 450/510 [I 2022-10-10 13:05:59] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-10 13:05:59] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00021 [I 2022-10-10 13:06:02] ...trap/engines/engine.py.321: eval: epoch 72 | batch 460/510 [I 2022-10-10 13:06:02] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:16 [I 2022-10-10 13:06:02] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00021 [I 2022-10-10 13:06:05] ...trap/engines/engine.py.321: eval: epoch 72 | batch 470/510 [I 2022-10-10 13:06:05] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-10 13:06:05] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00027 [I 2022-10-10 13:06:08] ...trap/engines/engine.py.321: eval: epoch 72 | batch 480/510 [I 2022-10-10 13:06:08] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-10 13:06:08] ...trap/engines/engine.py.326: process: 0.31904 | load: 0.00021 [I 2022-10-10 13:06:12] ...trap/engines/engine.py.321: eval: epoch 72 | batch 490/510 [I 2022-10-10 13:06:12] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:06 [I 2022-10-10 13:06:12] ...trap/engines/engine.py.326: process: 0.31940 | load: 0.00021 [I 2022-10-10 13:06:15] ...trap/engines/engine.py.321: eval: epoch 72 | batch 500/510 [I 2022-10-10 13:06:15] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:03 [I 2022-10-10 13:06:15] ...trap/engines/engine.py.326: process: 0.32615 | load: 0.00021 [I 2022-10-10 13:06:18] ...trap/engines/engine.py.321: eval: epoch 72 | batch 510/510 [I 2022-10-10 13:06:18] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 13:06:18] ...trap/engines/engine.py.326: process: 0.31751 | load: 0.00016 [S 2022-10-10 13:06:19] ...trap/engines/engine.py.132: eval_epoch.epoch: 72 [I 2022-10-10 13:06:19] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.733400 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.912800 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939600 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.728500 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911500 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937300 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012682 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011754 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009861 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014486 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011307 [S 2022-10-10 13:06:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010218 [I 2022-10-10 13:06:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 13:06:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 13:06:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 13:06:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 13:06:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 13:06:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 13:07:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 13:07:06] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 13:07:06] ...trap/engines/engine.py.163: Training model on trainset for epoch 73 [I 2022-10-10 13:07:06] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 13:07:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 0/2304 [I 2022-10-10 13:07:09] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:03:09 [I 2022-10-10 13:07:09] ...trap/engines/engine.py.229: process: 3.20744 | load: 2.24507 [I 2022-10-10 13:07:09] ...trap/engines/engine.py.230: loss: 0.15708 [I 2022-10-10 13:07:09] ...optimizers/trijoint.py.151: total_norm: 25.981384 [I 2022-10-10 13:07:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 10/2304 [I 2022-10-10 13:07:18] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:41 [I 2022-10-10 13:07:18] ...trap/engines/engine.py.229: process: 0.87484 | load: 0.00545 [I 2022-10-10 13:07:18] ...trap/engines/engine.py.230: loss: 0.11699 [I 2022-10-10 13:07:18] ...optimizers/trijoint.py.151: total_norm: 13.215481 [I 2022-10-10 13:07:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 20/2304 [I 2022-10-10 13:07:26] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:59 [I 2022-10-10 13:07:26] ...trap/engines/engine.py.229: process: 0.87598 | load: 0.00529 [I 2022-10-10 13:07:26] ...trap/engines/engine.py.230: loss: 0.10198 [I 2022-10-10 13:07:26] ...optimizers/trijoint.py.151: total_norm: 14.570185 [I 2022-10-10 13:07:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 30/2304 [I 2022-10-10 13:07:35] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:56 [I 2022-10-10 13:07:35] ...trap/engines/engine.py.229: process: 0.86318 | load: 0.00506 [I 2022-10-10 13:07:35] ...trap/engines/engine.py.230: loss: 0.22917 [I 2022-10-10 13:07:35] ...optimizers/trijoint.py.151: total_norm: 35.855759 [I 2022-10-10 13:07:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 40/2304 [I 2022-10-10 13:07:44] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:45 [I 2022-10-10 13:07:44] ...trap/engines/engine.py.229: process: 0.88220 | load: 0.00392 [I 2022-10-10 13:07:44] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-10 13:07:44] ...optimizers/trijoint.py.151: total_norm: 20.165386 [I 2022-10-10 13:07:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 50/2304 [I 2022-10-10 13:07:52] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:32 [I 2022-10-10 13:07:52] ...trap/engines/engine.py.229: process: 0.87198 | load: 0.00438 [I 2022-10-10 13:07:52] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-10 13:07:52] ...optimizers/trijoint.py.151: total_norm: 26.558947 [I 2022-10-10 13:08:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 60/2304 [I 2022-10-10 13:08:01] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:15 [I 2022-10-10 13:08:01] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00386 [I 2022-10-10 13:08:01] ...trap/engines/engine.py.230: loss: 0.13017 [I 2022-10-10 13:08:01] ...optimizers/trijoint.py.151: total_norm: 14.642104 [I 2022-10-10 13:08:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 70/2304 [I 2022-10-10 13:08:10] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:03 [I 2022-10-10 13:08:10] ...trap/engines/engine.py.229: process: 0.87786 | load: 0.00375 [I 2022-10-10 13:08:10] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-10 13:08:10] ...optimizers/trijoint.py.151: total_norm: 13.697480 [I 2022-10-10 13:08:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 80/2304 [I 2022-10-10 13:08:18] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:52 [I 2022-10-10 13:08:18] ...trap/engines/engine.py.229: process: 0.86131 | load: 0.00363 [I 2022-10-10 13:08:18] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-10 13:08:18] ...optimizers/trijoint.py.151: total_norm: 18.440905 [I 2022-10-10 13:08:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 90/2304 [I 2022-10-10 13:08:27] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:41 [I 2022-10-10 13:08:27] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00320 [I 2022-10-10 13:08:27] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-10 13:08:27] ...optimizers/trijoint.py.151: total_norm: 15.040120 [I 2022-10-10 13:08:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 100/2304 [I 2022-10-10 13:08:35] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:23 [I 2022-10-10 13:08:35] ...trap/engines/engine.py.229: process: 0.85602 | load: 0.00323 [I 2022-10-10 13:08:35] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-10 13:08:35] ...optimizers/trijoint.py.151: total_norm: 19.020924 [I 2022-10-10 13:08:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 110/2304 [I 2022-10-10 13:08:44] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:17 [I 2022-10-10 13:08:44] ...trap/engines/engine.py.229: process: 0.87238 | load: 0.00301 [I 2022-10-10 13:08:44] ...trap/engines/engine.py.230: loss: 0.11336 [I 2022-10-10 13:08:44] ...optimizers/trijoint.py.151: total_norm: 12.199342 [I 2022-10-10 13:08:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 120/2304 [I 2022-10-10 13:08:52] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:00 [I 2022-10-10 13:08:52] ...trap/engines/engine.py.229: process: 0.85362 | load: 0.00320 [I 2022-10-10 13:08:52] ...trap/engines/engine.py.230: loss: 0.13598 [I 2022-10-10 13:08:52] ...optimizers/trijoint.py.151: total_norm: 12.020640 [I 2022-10-10 13:09:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 130/2304 [I 2022-10-10 13:09:01] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:30:52 [I 2022-10-10 13:09:01] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00309 [I 2022-10-10 13:09:01] ...trap/engines/engine.py.230: loss: 0.11351 [I 2022-10-10 13:09:01] ...optimizers/trijoint.py.151: total_norm: 13.703284 [I 2022-10-10 13:09:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 140/2304 [I 2022-10-10 13:09:09] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:38 [I 2022-10-10 13:09:09] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00306 [I 2022-10-10 13:09:09] ...trap/engines/engine.py.230: loss: 0.19833 [I 2022-10-10 13:09:09] ...optimizers/trijoint.py.151: total_norm: 24.593271 [I 2022-10-10 13:09:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 150/2304 [I 2022-10-10 13:09:18] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:25 [I 2022-10-10 13:09:18] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00290 [I 2022-10-10 13:09:18] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-10 13:09:18] ...optimizers/trijoint.py.151: total_norm: 13.497373 [I 2022-10-10 13:09:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 160/2304 [I 2022-10-10 13:09:26] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:18 [I 2022-10-10 13:09:26] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00306 [I 2022-10-10 13:09:26] ...trap/engines/engine.py.230: loss: 0.18070 [I 2022-10-10 13:09:26] ...optimizers/trijoint.py.151: total_norm: 25.837931 [I 2022-10-10 13:09:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 170/2304 [I 2022-10-10 13:09:35] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:22 [I 2022-10-10 13:09:35] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00326 [I 2022-10-10 13:09:35] ...trap/engines/engine.py.230: loss: 0.10041 [I 2022-10-10 13:09:35] ...optimizers/trijoint.py.151: total_norm: 11.330565 [I 2022-10-10 13:09:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 180/2304 [I 2022-10-10 13:09:43] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:12 [I 2022-10-10 13:09:43] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00306 [I 2022-10-10 13:09:43] ...trap/engines/engine.py.230: loss: 0.11176 [I 2022-10-10 13:09:43] ...optimizers/trijoint.py.151: total_norm: 16.293325 [I 2022-10-10 13:09:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 190/2304 [I 2022-10-10 13:09:52] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:10 [I 2022-10-10 13:09:52] ...trap/engines/engine.py.229: process: 0.86484 | load: 0.00307 [I 2022-10-10 13:09:52] ...trap/engines/engine.py.230: loss: 0.14565 [I 2022-10-10 13:09:52] ...optimizers/trijoint.py.151: total_norm: 10.876943 [I 2022-10-10 13:10:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 200/2304 [I 2022-10-10 13:10:01] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:05 [I 2022-10-10 13:10:01] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00292 [I 2022-10-10 13:10:01] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-10 13:10:01] ...optimizers/trijoint.py.151: total_norm: 10.443806 [I 2022-10-10 13:10:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 210/2304 [I 2022-10-10 13:10:09] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:52 [I 2022-10-10 13:10:09] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00310 [I 2022-10-10 13:10:09] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-10 13:10:09] ...optimizers/trijoint.py.151: total_norm: 10.807457 [I 2022-10-10 13:10:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 220/2304 [I 2022-10-10 13:10:18] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:54 [I 2022-10-10 13:10:18] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00310 [I 2022-10-10 13:10:18] ...trap/engines/engine.py.230: loss: 0.16842 [I 2022-10-10 13:10:18] ...optimizers/trijoint.py.151: total_norm: 12.889299 [I 2022-10-10 13:10:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 230/2304 [I 2022-10-10 13:10:26] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:29 [I 2022-10-10 13:10:26] ...trap/engines/engine.py.229: process: 0.85156 | load: 0.00299 [I 2022-10-10 13:10:26] ...trap/engines/engine.py.230: loss: 0.13097 [I 2022-10-10 13:10:26] ...optimizers/trijoint.py.151: total_norm: 15.936534 [I 2022-10-10 13:10:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 240/2304 [I 2022-10-10 13:10:35] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:21 [I 2022-10-10 13:10:35] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00293 [I 2022-10-10 13:10:35] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-10 13:10:35] ...optimizers/trijoint.py.151: total_norm: 9.740490 [I 2022-10-10 13:10:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 250/2304 [I 2022-10-10 13:10:44] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:23 [I 2022-10-10 13:10:44] ...trap/engines/engine.py.229: process: 0.86462 | load: 0.00305 [I 2022-10-10 13:10:44] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-10 13:10:44] ...optimizers/trijoint.py.151: total_norm: 17.680965 [I 2022-10-10 13:10:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 260/2304 [I 2022-10-10 13:10:52] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:12 [I 2022-10-10 13:10:52] ...trap/engines/engine.py.229: process: 0.85276 | load: 0.00306 [I 2022-10-10 13:10:52] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-10 13:10:52] ...optimizers/trijoint.py.151: total_norm: 16.982279 [I 2022-10-10 13:11:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 270/2304 [I 2022-10-10 13:11:01] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:28:53 [I 2022-10-10 13:11:01] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00297 [I 2022-10-10 13:11:01] ...trap/engines/engine.py.230: loss: 0.19502 [I 2022-10-10 13:11:01] ...optimizers/trijoint.py.151: total_norm: 13.158257 [I 2022-10-10 13:11:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 280/2304 [I 2022-10-10 13:11:09] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:47 [I 2022-10-10 13:11:09] ...trap/engines/engine.py.229: process: 0.86272 | load: 0.00298 [I 2022-10-10 13:11:09] ...trap/engines/engine.py.230: loss: 0.18578 [I 2022-10-10 13:11:09] ...optimizers/trijoint.py.151: total_norm: 10.964541 [I 2022-10-10 13:11:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 290/2304 [I 2022-10-10 13:11:18] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:37 [I 2022-10-10 13:11:18] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00313 [I 2022-10-10 13:11:18] ...trap/engines/engine.py.230: loss: 0.15162 [I 2022-10-10 13:11:18] ...optimizers/trijoint.py.151: total_norm: 19.929234 [I 2022-10-10 13:11:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 300/2304 [I 2022-10-10 13:11:27] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:57 [I 2022-10-10 13:11:27] ...trap/engines/engine.py.229: process: 0.87050 | load: 0.00319 [I 2022-10-10 13:11:27] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-10 13:11:27] ...optimizers/trijoint.py.151: total_norm: 12.656026 [I 2022-10-10 13:11:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 310/2304 [I 2022-10-10 13:11:35] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:37 [I 2022-10-10 13:11:35] ...trap/engines/engine.py.229: process: 0.86938 | load: 0.00324 [I 2022-10-10 13:11:35] ...trap/engines/engine.py.230: loss: 0.15966 [I 2022-10-10 13:11:35] ...optimizers/trijoint.py.151: total_norm: 19.037689 [I 2022-10-10 13:11:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 320/2304 [I 2022-10-10 13:11:44] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:47 [I 2022-10-10 13:11:44] ...trap/engines/engine.py.229: process: 0.86439 | load: 0.00319 [I 2022-10-10 13:11:44] ...trap/engines/engine.py.230: loss: 0.22069 [I 2022-10-10 13:11:44] ...optimizers/trijoint.py.151: total_norm: 17.200539 [I 2022-10-10 13:11:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 330/2304 [I 2022-10-10 13:11:53] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:33 [I 2022-10-10 13:11:53] ...trap/engines/engine.py.229: process: 0.88231 | load: 0.00306 [I 2022-10-10 13:11:53] ...trap/engines/engine.py.230: loss: 0.14615 [I 2022-10-10 13:11:53] ...optimizers/trijoint.py.151: total_norm: 17.770218 [I 2022-10-10 13:12:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 340/2304 [I 2022-10-10 13:12:01] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:11 [I 2022-10-10 13:12:01] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00305 [I 2022-10-10 13:12:01] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-10 13:12:01] ...optimizers/trijoint.py.151: total_norm: 15.131796 [I 2022-10-10 13:12:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 350/2304 [I 2022-10-10 13:12:10] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:01 [I 2022-10-10 13:12:10] ...trap/engines/engine.py.229: process: 0.85539 | load: 0.00308 [I 2022-10-10 13:12:10] ...trap/engines/engine.py.230: loss: 0.14770 [I 2022-10-10 13:12:10] ...optimizers/trijoint.py.151: total_norm: 20.386745 [I 2022-10-10 13:12:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 360/2304 [I 2022-10-10 13:12:19] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:51 [I 2022-10-10 13:12:19] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00326 [I 2022-10-10 13:12:19] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-10 13:12:19] ...optimizers/trijoint.py.151: total_norm: 23.046116 [I 2022-10-10 13:12:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 370/2304 [I 2022-10-10 13:12:27] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:56 [I 2022-10-10 13:12:27] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00321 [I 2022-10-10 13:12:27] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-10 13:12:27] ...optimizers/trijoint.py.151: total_norm: 14.036777 [I 2022-10-10 13:12:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 380/2304 [I 2022-10-10 13:12:36] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:50 [I 2022-10-10 13:12:36] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00316 [I 2022-10-10 13:12:36] ...trap/engines/engine.py.230: loss: 0.10156 [I 2022-10-10 13:12:36] ...optimizers/trijoint.py.151: total_norm: 6.940722 [I 2022-10-10 13:12:45] ...trap/engines/engine.py.224: train: epoch 73 | batch 390/2304 [I 2022-10-10 13:12:45] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:35 [I 2022-10-10 13:12:45] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00310 [I 2022-10-10 13:12:45] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-10 13:12:45] ...optimizers/trijoint.py.151: total_norm: 14.589020 [I 2022-10-10 13:12:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 400/2304 [I 2022-10-10 13:12:53] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:22 [I 2022-10-10 13:12:53] ...trap/engines/engine.py.229: process: 0.86477 | load: 0.00307 [I 2022-10-10 13:12:53] ...trap/engines/engine.py.230: loss: 0.18356 [I 2022-10-10 13:12:53] ...optimizers/trijoint.py.151: total_norm: 22.009645 [I 2022-10-10 13:13:02] ...trap/engines/engine.py.224: train: epoch 73 | batch 410/2304 [I 2022-10-10 13:13:02] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:23 [I 2022-10-10 13:13:02] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00307 [I 2022-10-10 13:13:02] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-10 13:13:02] ...optimizers/trijoint.py.151: total_norm: 15.398487 [I 2022-10-10 13:13:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 420/2304 [I 2022-10-10 13:13:11] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:27:10 [I 2022-10-10 13:13:11] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00305 [I 2022-10-10 13:13:11] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-10 13:13:11] ...optimizers/trijoint.py.151: total_norm: 12.175533 [I 2022-10-10 13:13:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 430/2304 [I 2022-10-10 13:13:19] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:03 [I 2022-10-10 13:13:19] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00318 [I 2022-10-10 13:13:19] ...trap/engines/engine.py.230: loss: 0.22931 [I 2022-10-10 13:13:19] ...optimizers/trijoint.py.151: total_norm: 12.957261 [I 2022-10-10 13:13:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 440/2304 [I 2022-10-10 13:13:28] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:27:01 [I 2022-10-10 13:13:28] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00308 [I 2022-10-10 13:13:28] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-10 13:13:28] ...optimizers/trijoint.py.151: total_norm: 7.594048 [I 2022-10-10 13:13:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 450/2304 [I 2022-10-10 13:13:37] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:44 [I 2022-10-10 13:13:37] ...trap/engines/engine.py.229: process: 0.86874 | load: 0.00326 [I 2022-10-10 13:13:37] ...trap/engines/engine.py.230: loss: 0.13125 [I 2022-10-10 13:13:37] ...optimizers/trijoint.py.151: total_norm: 11.098078 [I 2022-10-10 13:13:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 460/2304 [I 2022-10-10 13:13:46] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:42 [I 2022-10-10 13:13:46] ...trap/engines/engine.py.229: process: 0.86687 | load: 0.00311 [I 2022-10-10 13:13:46] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-10 13:13:46] ...optimizers/trijoint.py.151: total_norm: 15.562267 [I 2022-10-10 13:13:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 470/2304 [I 2022-10-10 13:13:54] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:34 [I 2022-10-10 13:13:54] ...trap/engines/engine.py.229: process: 0.87079 | load: 0.00307 [I 2022-10-10 13:13:54] ...trap/engines/engine.py.230: loss: 0.13471 [I 2022-10-10 13:13:54] ...optimizers/trijoint.py.151: total_norm: 12.521534 [I 2022-10-10 13:14:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 480/2304 [I 2022-10-10 13:14:03] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:20 [I 2022-10-10 13:14:03] ...trap/engines/engine.py.229: process: 0.86076 | load: 0.00310 [I 2022-10-10 13:14:03] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-10 13:14:03] ...optimizers/trijoint.py.151: total_norm: 11.922822 [I 2022-10-10 13:14:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 490/2304 [I 2022-10-10 13:14:12] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:08 [I 2022-10-10 13:14:12] ...trap/engines/engine.py.229: process: 0.87885 | load: 0.00313 [I 2022-10-10 13:14:12] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-10 13:14:12] ...optimizers/trijoint.py.151: total_norm: 11.715596 [I 2022-10-10 13:14:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 500/2304 [I 2022-10-10 13:14:20] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:26:06 [I 2022-10-10 13:14:20] ...trap/engines/engine.py.229: process: 0.86852 | load: 0.00346 [I 2022-10-10 13:14:20] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-10 13:14:20] ...optimizers/trijoint.py.151: total_norm: 27.947084 [I 2022-10-10 13:14:29] ...trap/engines/engine.py.224: train: epoch 73 | batch 510/2304 [I 2022-10-10 13:14:29] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:26:02 [I 2022-10-10 13:14:29] ...trap/engines/engine.py.229: process: 0.87609 | load: 0.00317 [I 2022-10-10 13:14:29] ...trap/engines/engine.py.230: loss: 0.14712 [I 2022-10-10 13:14:29] ...optimizers/trijoint.py.151: total_norm: 17.808725 [I 2022-10-10 13:14:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 520/2304 [I 2022-10-10 13:14:38] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:50 [I 2022-10-10 13:14:38] ...trap/engines/engine.py.229: process: 0.87805 | load: 0.00310 [I 2022-10-10 13:14:38] ...trap/engines/engine.py.230: loss: 0.10932 [I 2022-10-10 13:14:38] ...optimizers/trijoint.py.151: total_norm: 9.638744 [I 2022-10-10 13:14:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 530/2304 [I 2022-10-10 13:14:46] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:38 [I 2022-10-10 13:14:46] ...trap/engines/engine.py.229: process: 0.86324 | load: 0.00320 [I 2022-10-10 13:14:46] ...trap/engines/engine.py.230: loss: 0.16293 [I 2022-10-10 13:14:46] ...optimizers/trijoint.py.151: total_norm: 24.313042 [I 2022-10-10 13:14:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 540/2304 [I 2022-10-10 13:14:55] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:31 [I 2022-10-10 13:14:55] ...trap/engines/engine.py.229: process: 0.87789 | load: 0.00305 [I 2022-10-10 13:14:55] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-10 13:14:55] ...optimizers/trijoint.py.151: total_norm: 13.948922 [I 2022-10-10 13:15:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 550/2304 [I 2022-10-10 13:15:04] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:20 [I 2022-10-10 13:15:04] ...trap/engines/engine.py.229: process: 0.87167 | load: 0.00359 [I 2022-10-10 13:15:04] ...trap/engines/engine.py.230: loss: 0.10378 [I 2022-10-10 13:15:04] ...optimizers/trijoint.py.151: total_norm: 26.726145 [I 2022-10-10 13:15:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 560/2304 [I 2022-10-10 13:15:12] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:17 [I 2022-10-10 13:15:12] ...trap/engines/engine.py.229: process: 0.88175 | load: 0.00310 [I 2022-10-10 13:15:12] ...trap/engines/engine.py.230: loss: 0.13904 [I 2022-10-10 13:15:12] ...optimizers/trijoint.py.151: total_norm: 16.743786 [I 2022-10-10 13:15:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 570/2304 [I 2022-10-10 13:15:21] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:39 [I 2022-10-10 13:15:21] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00300 [I 2022-10-10 13:15:21] ...trap/engines/engine.py.230: loss: 0.11997 [I 2022-10-10 13:15:21] ...optimizers/trijoint.py.151: total_norm: 23.371202 [I 2022-10-10 13:15:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 580/2304 [I 2022-10-10 13:15:30] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:35 [I 2022-10-10 13:15:30] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00295 [I 2022-10-10 13:15:30] ...trap/engines/engine.py.230: loss: 0.22591 [I 2022-10-10 13:15:30] ...optimizers/trijoint.py.151: total_norm: 27.414724 [I 2022-10-10 13:15:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 590/2304 [I 2022-10-10 13:15:38] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:18 [I 2022-10-10 13:15:38] ...trap/engines/engine.py.229: process: 0.84231 | load: 0.00297 [I 2022-10-10 13:15:38] ...trap/engines/engine.py.230: loss: 0.09445 [I 2022-10-10 13:15:38] ...optimizers/trijoint.py.151: total_norm: 11.134667 [I 2022-10-10 13:15:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 600/2304 [I 2022-10-10 13:15:47] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:16 [I 2022-10-10 13:15:47] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00331 [I 2022-10-10 13:15:47] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-10 13:15:47] ...optimizers/trijoint.py.151: total_norm: 14.903623 [I 2022-10-10 13:15:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 610/2304 [I 2022-10-10 13:15:55] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:20 [I 2022-10-10 13:15:55] ...trap/engines/engine.py.229: process: 0.86073 | load: 0.00299 [I 2022-10-10 13:15:55] ...trap/engines/engine.py.230: loss: 0.19970 [I 2022-10-10 13:15:55] ...optimizers/trijoint.py.151: total_norm: 19.493677 [I 2022-10-10 13:16:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 620/2304 [I 2022-10-10 13:16:04] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:07 [I 2022-10-10 13:16:04] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00298 [I 2022-10-10 13:16:04] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-10 13:16:04] ...optimizers/trijoint.py.151: total_norm: 10.060465 [I 2022-10-10 13:16:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 630/2304 [I 2022-10-10 13:16:13] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:24:03 [I 2022-10-10 13:16:13] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00296 [I 2022-10-10 13:16:13] ...trap/engines/engine.py.230: loss: 0.09849 [I 2022-10-10 13:16:13] ...optimizers/trijoint.py.151: total_norm: 8.078382 [I 2022-10-10 13:16:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 640/2304 [I 2022-10-10 13:16:21] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:42 [I 2022-10-10 13:16:21] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00297 [I 2022-10-10 13:16:21] ...trap/engines/engine.py.230: loss: 0.16087 [I 2022-10-10 13:16:21] ...optimizers/trijoint.py.151: total_norm: 17.790291 [I 2022-10-10 13:16:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 650/2304 [I 2022-10-10 13:16:30] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:31 [I 2022-10-10 13:16:30] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00298 [I 2022-10-10 13:16:30] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-10 13:16:30] ...optimizers/trijoint.py.151: total_norm: 14.840535 [I 2022-10-10 13:16:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 660/2304 [I 2022-10-10 13:16:38] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:18 [I 2022-10-10 13:16:38] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00296 [I 2022-10-10 13:16:38] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-10 13:16:38] ...optimizers/trijoint.py.151: total_norm: 43.350769 [I 2022-10-10 13:16:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 670/2304 [I 2022-10-10 13:16:47] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:11 [I 2022-10-10 13:16:47] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00294 [I 2022-10-10 13:16:47] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-10 13:16:47] ...optimizers/trijoint.py.151: total_norm: 19.210512 [I 2022-10-10 13:16:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 680/2304 [I 2022-10-10 13:16:55] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:04 [I 2022-10-10 13:16:55] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00318 [I 2022-10-10 13:16:55] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-10 13:16:55] ...optimizers/trijoint.py.151: total_norm: 13.070850 [I 2022-10-10 13:17:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 690/2304 [I 2022-10-10 13:17:04] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:54 [I 2022-10-10 13:17:04] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00294 [I 2022-10-10 13:17:04] ...trap/engines/engine.py.230: loss: 0.13747 [I 2022-10-10 13:17:04] ...optimizers/trijoint.py.151: total_norm: 36.391285 [I 2022-10-10 13:17:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 700/2304 [I 2022-10-10 13:17:13] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:50 [I 2022-10-10 13:17:13] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00294 [I 2022-10-10 13:17:13] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-10 13:17:13] ...optimizers/trijoint.py.151: total_norm: 25.497599 [I 2022-10-10 13:17:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 710/2304 [I 2022-10-10 13:17:21] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:52 [I 2022-10-10 13:17:21] ...trap/engines/engine.py.229: process: 0.86404 | load: 0.00315 [I 2022-10-10 13:17:21] ...trap/engines/engine.py.230: loss: 0.20960 [I 2022-10-10 13:17:21] ...optimizers/trijoint.py.151: total_norm: 17.191210 [I 2022-10-10 13:17:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 720/2304 [I 2022-10-10 13:17:30] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:50 [I 2022-10-10 13:17:30] ...trap/engines/engine.py.229: process: 0.87227 | load: 0.00327 [I 2022-10-10 13:17:30] ...trap/engines/engine.py.230: loss: 0.13566 [I 2022-10-10 13:17:30] ...optimizers/trijoint.py.151: total_norm: 39.619854 [I 2022-10-10 13:17:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 730/2304 [I 2022-10-10 13:17:38] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:34 [I 2022-10-10 13:17:38] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00310 [I 2022-10-10 13:17:39] ...trap/engines/engine.py.230: loss: 0.13915 [I 2022-10-10 13:17:39] ...optimizers/trijoint.py.151: total_norm: 35.149731 [I 2022-10-10 13:17:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 740/2304 [I 2022-10-10 13:17:47] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:31 [I 2022-10-10 13:17:47] ...trap/engines/engine.py.229: process: 0.86423 | load: 0.00313 [I 2022-10-10 13:17:47] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-10 13:17:47] ...optimizers/trijoint.py.151: total_norm: 7.546607 [I 2022-10-10 13:17:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 750/2304 [I 2022-10-10 13:17:56] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:35 [I 2022-10-10 13:17:56] ...trap/engines/engine.py.229: process: 0.87845 | load: 0.00312 [I 2022-10-10 13:17:56] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-10 13:17:56] ...optimizers/trijoint.py.151: total_norm: 20.386761 [I 2022-10-10 13:18:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 760/2304 [I 2022-10-10 13:18:05] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:11 [I 2022-10-10 13:18:05] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00321 [I 2022-10-10 13:18:05] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-10 13:18:05] ...optimizers/trijoint.py.151: total_norm: 11.342070 [I 2022-10-10 13:18:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 770/2304 [I 2022-10-10 13:18:13] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:03 [I 2022-10-10 13:18:13] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00315 [I 2022-10-10 13:18:13] ...trap/engines/engine.py.230: loss: 0.14848 [I 2022-10-10 13:18:13] ...optimizers/trijoint.py.151: total_norm: 9.870460 [I 2022-10-10 13:18:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 780/2304 [I 2022-10-10 13:18:22] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:22:01 [I 2022-10-10 13:18:22] ...trap/engines/engine.py.229: process: 0.85991 | load: 0.00340 [I 2022-10-10 13:18:22] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-10 13:18:22] ...optimizers/trijoint.py.151: total_norm: 16.826193 [I 2022-10-10 13:18:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 790/2304 [I 2022-10-10 13:18:31] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:52 [I 2022-10-10 13:18:31] ...trap/engines/engine.py.229: process: 0.86984 | load: 0.00309 [I 2022-10-10 13:18:31] ...trap/engines/engine.py.230: loss: 0.09162 [I 2022-10-10 13:18:31] ...optimizers/trijoint.py.151: total_norm: 8.889977 [I 2022-10-10 13:18:39] ...trap/engines/engine.py.224: train: epoch 73 | batch 800/2304 [I 2022-10-10 13:18:39] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:44 [I 2022-10-10 13:18:39] ...trap/engines/engine.py.229: process: 0.86598 | load: 0.00308 [I 2022-10-10 13:18:39] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-10 13:18:39] ...optimizers/trijoint.py.151: total_norm: 10.313293 [I 2022-10-10 13:18:48] ...trap/engines/engine.py.224: train: epoch 73 | batch 810/2304 [I 2022-10-10 13:18:48] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:35 [I 2022-10-10 13:18:48] ...trap/engines/engine.py.229: process: 0.87651 | load: 0.00318 [I 2022-10-10 13:18:48] ...trap/engines/engine.py.230: loss: 0.19260 [I 2022-10-10 13:18:48] ...optimizers/trijoint.py.151: total_norm: 14.600534 [I 2022-10-10 13:18:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 820/2304 [I 2022-10-10 13:18:57] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:27 [I 2022-10-10 13:18:57] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00314 [I 2022-10-10 13:18:57] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-10 13:18:57] ...optimizers/trijoint.py.151: total_norm: 8.565756 [I 2022-10-10 13:19:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 830/2304 [I 2022-10-10 13:19:05] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:13 [I 2022-10-10 13:19:05] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00315 [I 2022-10-10 13:19:05] ...trap/engines/engine.py.230: loss: 0.14673 [I 2022-10-10 13:19:05] ...optimizers/trijoint.py.151: total_norm: 17.657070 [I 2022-10-10 13:19:14] ...trap/engines/engine.py.224: train: epoch 73 | batch 840/2304 [I 2022-10-10 13:19:14] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:03 [I 2022-10-10 13:19:14] ...trap/engines/engine.py.229: process: 0.85909 | load: 0.00314 [I 2022-10-10 13:19:14] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-10 13:19:14] ...optimizers/trijoint.py.151: total_norm: 38.862713 [I 2022-10-10 13:19:23] ...trap/engines/engine.py.224: train: epoch 73 | batch 850/2304 [I 2022-10-10 13:19:23] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:57 [I 2022-10-10 13:19:23] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00307 [I 2022-10-10 13:19:23] ...trap/engines/engine.py.230: loss: 0.18715 [I 2022-10-10 13:19:23] ...optimizers/trijoint.py.151: total_norm: 26.988842 [I 2022-10-10 13:19:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 860/2304 [I 2022-10-10 13:19:31] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:47 [I 2022-10-10 13:19:31] ...trap/engines/engine.py.229: process: 0.86985 | load: 0.00314 [I 2022-10-10 13:19:31] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-10 13:19:31] ...optimizers/trijoint.py.151: total_norm: 36.960674 [I 2022-10-10 13:19:40] ...trap/engines/engine.py.224: train: epoch 73 | batch 870/2304 [I 2022-10-10 13:19:40] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:46 [I 2022-10-10 13:19:40] ...trap/engines/engine.py.229: process: 0.86751 | load: 0.00305 [I 2022-10-10 13:19:40] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-10 13:19:40] ...optimizers/trijoint.py.151: total_norm: 23.349266 [I 2022-10-10 13:19:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 880/2304 [I 2022-10-10 13:19:49] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:34 [I 2022-10-10 13:19:49] ...trap/engines/engine.py.229: process: 0.87056 | load: 0.00308 [I 2022-10-10 13:19:49] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-10 13:19:49] ...optimizers/trijoint.py.151: total_norm: 15.465294 [I 2022-10-10 13:19:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 890/2304 [I 2022-10-10 13:19:57] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:24 [I 2022-10-10 13:19:57] ...trap/engines/engine.py.229: process: 0.86348 | load: 0.00307 [I 2022-10-10 13:19:57] ...trap/engines/engine.py.230: loss: 0.15841 [I 2022-10-10 13:19:57] ...optimizers/trijoint.py.151: total_norm: 17.439411 [I 2022-10-10 13:20:06] ...trap/engines/engine.py.224: train: epoch 73 | batch 900/2304 [I 2022-10-10 13:20:06] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:11 [I 2022-10-10 13:20:06] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00309 [I 2022-10-10 13:20:06] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-10 13:20:06] ...optimizers/trijoint.py.151: total_norm: 11.279216 [I 2022-10-10 13:20:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 910/2304 [I 2022-10-10 13:20:15] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:08 [I 2022-10-10 13:20:15] ...trap/engines/engine.py.229: process: 0.86380 | load: 0.00310 [I 2022-10-10 13:20:15] ...trap/engines/engine.py.230: loss: 0.14988 [I 2022-10-10 13:20:15] ...optimizers/trijoint.py.151: total_norm: 13.543661 [I 2022-10-10 13:20:23] ...trap/engines/engine.py.224: train: epoch 73 | batch 920/2304 [I 2022-10-10 13:20:23] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:20:02 [I 2022-10-10 13:20:23] ...trap/engines/engine.py.229: process: 0.86458 | load: 0.00308 [I 2022-10-10 13:20:23] ...trap/engines/engine.py.230: loss: 0.13192 [I 2022-10-10 13:20:23] ...optimizers/trijoint.py.151: total_norm: 12.613272 [I 2022-10-10 13:20:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 930/2304 [I 2022-10-10 13:20:32] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:48 [I 2022-10-10 13:20:32] ...trap/engines/engine.py.229: process: 0.86860 | load: 0.00310 [I 2022-10-10 13:20:32] ...trap/engines/engine.py.230: loss: 0.14570 [I 2022-10-10 13:20:32] ...optimizers/trijoint.py.151: total_norm: 13.468025 [I 2022-10-10 13:20:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 940/2304 [I 2022-10-10 13:20:41] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:55 [I 2022-10-10 13:20:41] ...trap/engines/engine.py.229: process: 0.87288 | load: 0.00316 [I 2022-10-10 13:20:41] ...trap/engines/engine.py.230: loss: 0.15585 [I 2022-10-10 13:20:41] ...optimizers/trijoint.py.151: total_norm: 10.784444 [I 2022-10-10 13:20:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 950/2304 [I 2022-10-10 13:20:50] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:34 [I 2022-10-10 13:20:50] ...trap/engines/engine.py.229: process: 0.87055 | load: 0.00315 [I 2022-10-10 13:20:50] ...trap/engines/engine.py.230: loss: 0.09334 [I 2022-10-10 13:20:50] ...optimizers/trijoint.py.151: total_norm: 11.431089 [I 2022-10-10 13:20:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 960/2304 [I 2022-10-10 13:20:58] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:22 [I 2022-10-10 13:20:58] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00309 [I 2022-10-10 13:20:58] ...trap/engines/engine.py.230: loss: 0.20964 [I 2022-10-10 13:20:58] ...optimizers/trijoint.py.151: total_norm: 14.604670 [I 2022-10-10 13:21:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 970/2304 [I 2022-10-10 13:21:07] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:14 [I 2022-10-10 13:21:07] ...trap/engines/engine.py.229: process: 0.87275 | load: 0.00306 [I 2022-10-10 13:21:07] ...trap/engines/engine.py.230: loss: 0.17763 [I 2022-10-10 13:21:07] ...optimizers/trijoint.py.151: total_norm: 13.774872 [I 2022-10-10 13:21:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 980/2304 [I 2022-10-10 13:21:16] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:07 [I 2022-10-10 13:21:16] ...trap/engines/engine.py.229: process: 0.86353 | load: 0.00303 [I 2022-10-10 13:21:16] ...trap/engines/engine.py.230: loss: 0.10877 [I 2022-10-10 13:21:16] ...optimizers/trijoint.py.151: total_norm: 13.791308 [I 2022-10-10 13:21:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 990/2304 [I 2022-10-10 13:21:24] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:45 [I 2022-10-10 13:21:24] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00295 [I 2022-10-10 13:21:24] ...trap/engines/engine.py.230: loss: 0.09451 [I 2022-10-10 13:21:24] ...optimizers/trijoint.py.151: total_norm: 9.637965 [I 2022-10-10 13:21:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 1000/2304 [I 2022-10-10 13:21:33] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:32 [I 2022-10-10 13:21:33] ...trap/engines/engine.py.229: process: 0.83849 | load: 0.00295 [I 2022-10-10 13:21:33] ...trap/engines/engine.py.230: loss: 0.14010 [I 2022-10-10 13:21:33] ...optimizers/trijoint.py.151: total_norm: 8.901380 [I 2022-10-10 13:21:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 1010/2304 [I 2022-10-10 13:21:41] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:26 [I 2022-10-10 13:21:41] ...trap/engines/engine.py.229: process: 0.85882 | load: 0.00295 [I 2022-10-10 13:21:41] ...trap/engines/engine.py.230: loss: 0.10749 [I 2022-10-10 13:21:41] ...optimizers/trijoint.py.151: total_norm: 16.894684 [I 2022-10-10 13:21:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 1020/2304 [I 2022-10-10 13:21:50] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:14 [I 2022-10-10 13:21:50] ...trap/engines/engine.py.229: process: 0.84158 | load: 0.00297 [I 2022-10-10 13:21:50] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-10 13:21:50] ...optimizers/trijoint.py.151: total_norm: 16.459599 [I 2022-10-10 13:21:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 1030/2304 [I 2022-10-10 13:21:58] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:04 [I 2022-10-10 13:21:58] ...trap/engines/engine.py.229: process: 0.84626 | load: 0.00293 [I 2022-10-10 13:21:58] ...trap/engines/engine.py.230: loss: 0.24281 [I 2022-10-10 13:21:58] ...optimizers/trijoint.py.151: total_norm: 31.499241 [I 2022-10-10 13:22:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 1040/2304 [I 2022-10-10 13:22:07] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:00 [I 2022-10-10 13:22:07] ...trap/engines/engine.py.229: process: 0.85853 | load: 0.00302 [I 2022-10-10 13:22:07] ...trap/engines/engine.py.230: loss: 0.12979 [I 2022-10-10 13:22:07] ...optimizers/trijoint.py.151: total_norm: 42.179943 [I 2022-10-10 13:22:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 1050/2304 [I 2022-10-10 13:22:16] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:52 [I 2022-10-10 13:22:16] ...trap/engines/engine.py.229: process: 0.85453 | load: 0.00294 [I 2022-10-10 13:22:16] ...trap/engines/engine.py.230: loss: 0.11061 [I 2022-10-10 13:22:16] ...optimizers/trijoint.py.151: total_norm: 10.285731 [I 2022-10-10 13:22:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 1060/2304 [I 2022-10-10 13:22:24] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:41 [I 2022-10-10 13:22:24] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00305 [I 2022-10-10 13:22:24] ...trap/engines/engine.py.230: loss: 0.10804 [I 2022-10-10 13:22:24] ...optimizers/trijoint.py.151: total_norm: 13.403151 [I 2022-10-10 13:22:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 1070/2304 [I 2022-10-10 13:22:33] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:29 [I 2022-10-10 13:22:33] ...trap/engines/engine.py.229: process: 0.86829 | load: 0.00292 [I 2022-10-10 13:22:33] ...trap/engines/engine.py.230: loss: 0.15096 [I 2022-10-10 13:22:33] ...optimizers/trijoint.py.151: total_norm: 15.637036 [I 2022-10-10 13:22:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 1080/2304 [I 2022-10-10 13:22:41] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:20 [I 2022-10-10 13:22:41] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00306 [I 2022-10-10 13:22:41] ...trap/engines/engine.py.230: loss: 0.08773 [I 2022-10-10 13:22:41] ...optimizers/trijoint.py.151: total_norm: 17.946878 [I 2022-10-10 13:22:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 1090/2304 [I 2022-10-10 13:22:50] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:09 [I 2022-10-10 13:22:50] ...trap/engines/engine.py.229: process: 0.83353 | load: 0.00305 [I 2022-10-10 13:22:50] ...trap/engines/engine.py.230: loss: 0.18617 [I 2022-10-10 13:22:50] ...optimizers/trijoint.py.151: total_norm: 19.731220 [I 2022-10-10 13:22:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 1100/2304 [I 2022-10-10 13:22:58] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:05 [I 2022-10-10 13:22:58] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00302 [I 2022-10-10 13:22:58] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-10 13:22:58] ...optimizers/trijoint.py.151: total_norm: 16.833895 [I 2022-10-10 13:23:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 1110/2304 [I 2022-10-10 13:23:07] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:58 [I 2022-10-10 13:23:07] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00302 [I 2022-10-10 13:23:07] ...trap/engines/engine.py.230: loss: 0.15729 [I 2022-10-10 13:23:07] ...optimizers/trijoint.py.151: total_norm: 34.472893 [I 2022-10-10 13:23:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 1120/2304 [I 2022-10-10 13:23:15] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:46 [I 2022-10-10 13:23:15] ...trap/engines/engine.py.229: process: 0.86545 | load: 0.00304 [I 2022-10-10 13:23:15] ...trap/engines/engine.py.230: loss: 0.13953 [I 2022-10-10 13:23:15] ...optimizers/trijoint.py.151: total_norm: 15.069747 [I 2022-10-10 13:23:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 1130/2304 [I 2022-10-10 13:23:24] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:37 [I 2022-10-10 13:23:24] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00305 [I 2022-10-10 13:23:24] ...trap/engines/engine.py.230: loss: 0.15865 [I 2022-10-10 13:23:24] ...optimizers/trijoint.py.151: total_norm: 11.283518 [I 2022-10-10 13:23:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 1140/2304 [I 2022-10-10 13:23:32] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:35 [I 2022-10-10 13:23:32] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00303 [I 2022-10-10 13:23:32] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-10 13:23:32] ...optimizers/trijoint.py.151: total_norm: 19.384865 [I 2022-10-10 13:23:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 1150/2304 [I 2022-10-10 13:23:41] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:27 [I 2022-10-10 13:23:41] ...trap/engines/engine.py.229: process: 0.85741 | load: 0.00298 [I 2022-10-10 13:23:41] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-10 13:23:41] ...optimizers/trijoint.py.151: total_norm: 18.372993 [I 2022-10-10 13:23:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 1160/2304 [I 2022-10-10 13:23:50] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:21 [I 2022-10-10 13:23:50] ...trap/engines/engine.py.229: process: 0.85679 | load: 0.00294 [I 2022-10-10 13:23:50] ...trap/engines/engine.py.230: loss: 0.15854 [I 2022-10-10 13:23:50] ...optimizers/trijoint.py.151: total_norm: 16.582285 [I 2022-10-10 13:23:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 1170/2304 [I 2022-10-10 13:23:58] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:11 [I 2022-10-10 13:23:58] ...trap/engines/engine.py.229: process: 0.86204 | load: 0.00295 [I 2022-10-10 13:23:58] ...trap/engines/engine.py.230: loss: 0.22963 [I 2022-10-10 13:23:58] ...optimizers/trijoint.py.151: total_norm: 24.320915 [I 2022-10-10 13:24:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 1180/2304 [I 2022-10-10 13:24:07] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:03 [I 2022-10-10 13:24:07] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00295 [I 2022-10-10 13:24:07] ...trap/engines/engine.py.230: loss: 0.14696 [I 2022-10-10 13:24:07] ...optimizers/trijoint.py.151: total_norm: 17.600088 [I 2022-10-10 13:24:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 1190/2304 [I 2022-10-10 13:24:15] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:54 [I 2022-10-10 13:24:15] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00294 [I 2022-10-10 13:24:15] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-10 13:24:15] ...optimizers/trijoint.py.151: total_norm: 12.756515 [I 2022-10-10 13:24:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 1200/2304 [I 2022-10-10 13:24:24] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:48 [I 2022-10-10 13:24:24] ...trap/engines/engine.py.229: process: 0.85635 | load: 0.00309 [I 2022-10-10 13:24:24] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-10 13:24:24] ...optimizers/trijoint.py.151: total_norm: 22.124260 [I 2022-10-10 13:24:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 1210/2304 [I 2022-10-10 13:24:33] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:46 [I 2022-10-10 13:24:33] ...trap/engines/engine.py.229: process: 0.86933 | load: 0.00314 [I 2022-10-10 13:24:33] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-10 13:24:33] ...optimizers/trijoint.py.151: total_norm: 16.614836 [I 2022-10-10 13:24:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 1220/2304 [I 2022-10-10 13:24:41] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:34 [I 2022-10-10 13:24:41] ...trap/engines/engine.py.229: process: 0.86050 | load: 0.00308 [I 2022-10-10 13:24:41] ...trap/engines/engine.py.230: loss: 0.10979 [I 2022-10-10 13:24:41] ...optimizers/trijoint.py.151: total_norm: 19.974974 [I 2022-10-10 13:24:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 1230/2304 [I 2022-10-10 13:24:50] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:36 [I 2022-10-10 13:24:50] ...trap/engines/engine.py.229: process: 0.87388 | load: 0.00308 [I 2022-10-10 13:24:50] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-10 13:24:50] ...optimizers/trijoint.py.151: total_norm: 19.373928 [I 2022-10-10 13:24:59] ...trap/engines/engine.py.224: train: epoch 73 | batch 1240/2304 [I 2022-10-10 13:24:59] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:20 [I 2022-10-10 13:24:59] ...trap/engines/engine.py.229: process: 0.85369 | load: 0.00314 [I 2022-10-10 13:24:59] ...trap/engines/engine.py.230: loss: 0.15059 [I 2022-10-10 13:24:59] ...optimizers/trijoint.py.151: total_norm: 15.781128 [I 2022-10-10 13:25:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 1250/2304 [I 2022-10-10 13:25:07] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:07 [I 2022-10-10 13:25:07] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00307 [I 2022-10-10 13:25:07] ...trap/engines/engine.py.230: loss: 0.11117 [I 2022-10-10 13:25:07] ...optimizers/trijoint.py.151: total_norm: 12.129191 [I 2022-10-10 13:25:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 1260/2304 [I 2022-10-10 13:25:16] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:15:00 [I 2022-10-10 13:25:16] ...trap/engines/engine.py.229: process: 0.86451 | load: 0.00308 [I 2022-10-10 13:25:16] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-10 13:25:16] ...optimizers/trijoint.py.151: total_norm: 19.641891 [I 2022-10-10 13:25:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 1270/2304 [I 2022-10-10 13:25:25] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:50 [I 2022-10-10 13:25:25] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00320 [I 2022-10-10 13:25:25] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-10 13:25:25] ...optimizers/trijoint.py.151: total_norm: 17.040480 [I 2022-10-10 13:25:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 1280/2304 [I 2022-10-10 13:25:33] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:44 [I 2022-10-10 13:25:33] ...trap/engines/engine.py.229: process: 0.88040 | load: 0.00306 [I 2022-10-10 13:25:33] ...trap/engines/engine.py.230: loss: 0.11037 [I 2022-10-10 13:25:33] ...optimizers/trijoint.py.151: total_norm: 9.196342 [I 2022-10-10 13:25:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 1290/2304 [I 2022-10-10 13:25:42] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:32 [I 2022-10-10 13:25:42] ...trap/engines/engine.py.229: process: 0.86205 | load: 0.00307 [I 2022-10-10 13:25:42] ...trap/engines/engine.py.230: loss: 0.09786 [I 2022-10-10 13:25:42] ...optimizers/trijoint.py.151: total_norm: 13.188951 [I 2022-10-10 13:25:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 1300/2304 [I 2022-10-10 13:25:51] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:34 [I 2022-10-10 13:25:51] ...trap/engines/engine.py.229: process: 0.89099 | load: 0.00320 [I 2022-10-10 13:25:51] ...trap/engines/engine.py.230: loss: 0.10523 [I 2022-10-10 13:25:51] ...optimizers/trijoint.py.151: total_norm: 14.601213 [I 2022-10-10 13:25:59] ...trap/engines/engine.py.224: train: epoch 73 | batch 1310/2304 [I 2022-10-10 13:25:59] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:14:17 [I 2022-10-10 13:25:59] ...trap/engines/engine.py.229: process: 0.85590 | load: 0.00312 [I 2022-10-10 13:25:59] ...trap/engines/engine.py.230: loss: 0.11313 [I 2022-10-10 13:25:59] ...optimizers/trijoint.py.151: total_norm: 7.092532 [I 2022-10-10 13:26:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 1320/2304 [I 2022-10-10 13:26:08] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:11 [I 2022-10-10 13:26:08] ...trap/engines/engine.py.229: process: 0.86170 | load: 0.00311 [I 2022-10-10 13:26:08] ...trap/engines/engine.py.230: loss: 0.17586 [I 2022-10-10 13:26:08] ...optimizers/trijoint.py.151: total_norm: 23.626438 [I 2022-10-10 13:26:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 1330/2304 [I 2022-10-10 13:26:17] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:14:00 [I 2022-10-10 13:26:17] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00307 [I 2022-10-10 13:26:17] ...trap/engines/engine.py.230: loss: 0.18487 [I 2022-10-10 13:26:17] ...optimizers/trijoint.py.151: total_norm: 20.166178 [I 2022-10-10 13:26:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 1340/2304 [I 2022-10-10 13:26:25] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:56 [I 2022-10-10 13:26:25] ...trap/engines/engine.py.229: process: 0.86637 | load: 0.00316 [I 2022-10-10 13:26:25] ...trap/engines/engine.py.230: loss: 0.15643 [I 2022-10-10 13:26:25] ...optimizers/trijoint.py.151: total_norm: 15.489716 [I 2022-10-10 13:26:34] ...trap/engines/engine.py.224: train: epoch 73 | batch 1350/2304 [I 2022-10-10 13:26:34] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:50 [I 2022-10-10 13:26:34] ...trap/engines/engine.py.229: process: 0.86862 | load: 0.00311 [I 2022-10-10 13:26:34] ...trap/engines/engine.py.230: loss: 0.21823 [I 2022-10-10 13:26:34] ...optimizers/trijoint.py.151: total_norm: 27.872383 [I 2022-10-10 13:26:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 1360/2304 [I 2022-10-10 13:26:43] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:39 [I 2022-10-10 13:26:43] ...trap/engines/engine.py.229: process: 0.86270 | load: 0.00307 [I 2022-10-10 13:26:43] ...trap/engines/engine.py.230: loss: 0.11257 [I 2022-10-10 13:26:43] ...optimizers/trijoint.py.151: total_norm: 12.773956 [I 2022-10-10 13:26:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 1370/2304 [I 2022-10-10 13:26:51] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:28 [I 2022-10-10 13:26:51] ...trap/engines/engine.py.229: process: 0.86812 | load: 0.00313 [I 2022-10-10 13:26:51] ...trap/engines/engine.py.230: loss: 0.09508 [I 2022-10-10 13:26:51] ...optimizers/trijoint.py.151: total_norm: 10.155050 [I 2022-10-10 13:27:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 1380/2304 [I 2022-10-10 13:27:00] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:15 [I 2022-10-10 13:27:00] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00308 [I 2022-10-10 13:27:00] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-10 13:27:00] ...optimizers/trijoint.py.151: total_norm: 9.011206 [I 2022-10-10 13:27:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 1390/2304 [I 2022-10-10 13:27:09] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:09 [I 2022-10-10 13:27:09] ...trap/engines/engine.py.229: process: 0.86800 | load: 0.00309 [I 2022-10-10 13:27:09] ...trap/engines/engine.py.230: loss: 0.19253 [I 2022-10-10 13:27:09] ...optimizers/trijoint.py.151: total_norm: 24.189905 [I 2022-10-10 13:27:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 1400/2304 [I 2022-10-10 13:27:17] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:13:02 [I 2022-10-10 13:27:17] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00316 [I 2022-10-10 13:27:17] ...trap/engines/engine.py.230: loss: 0.11267 [I 2022-10-10 13:27:17] ...optimizers/trijoint.py.151: total_norm: 15.992961 [I 2022-10-10 13:27:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 1410/2304 [I 2022-10-10 13:27:26] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:13:00 [I 2022-10-10 13:27:26] ...trap/engines/engine.py.229: process: 0.88150 | load: 0.00311 [I 2022-10-10 13:27:26] ...trap/engines/engine.py.230: loss: 0.14141 [I 2022-10-10 13:27:26] ...optimizers/trijoint.py.151: total_norm: 13.373541 [I 2022-10-10 13:27:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 1420/2304 [I 2022-10-10 13:27:35] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:50 [I 2022-10-10 13:27:35] ...trap/engines/engine.py.229: process: 0.86869 | load: 0.00309 [I 2022-10-10 13:27:35] ...trap/engines/engine.py.230: loss: 0.21153 [I 2022-10-10 13:27:35] ...optimizers/trijoint.py.151: total_norm: 17.516926 [I 2022-10-10 13:27:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1430/2304 [I 2022-10-10 13:27:44] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:35 [I 2022-10-10 13:27:44] ...trap/engines/engine.py.229: process: 0.85680 | load: 0.00309 [I 2022-10-10 13:27:44] ...trap/engines/engine.py.230: loss: 0.11411 [I 2022-10-10 13:27:44] ...optimizers/trijoint.py.151: total_norm: 6.981874 [I 2022-10-10 13:27:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 1440/2304 [I 2022-10-10 13:27:52] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:28 [I 2022-10-10 13:27:52] ...trap/engines/engine.py.229: process: 0.86859 | load: 0.00308 [I 2022-10-10 13:27:52] ...trap/engines/engine.py.230: loss: 0.13154 [I 2022-10-10 13:27:52] ...optimizers/trijoint.py.151: total_norm: 11.994437 [I 2022-10-10 13:28:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 1450/2304 [I 2022-10-10 13:28:01] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:12:19 [I 2022-10-10 13:28:01] ...trap/engines/engine.py.229: process: 0.88138 | load: 0.00310 [I 2022-10-10 13:28:01] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-10 13:28:01] ...optimizers/trijoint.py.151: total_norm: 11.620296 [I 2022-10-10 13:28:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1460/2304 [I 2022-10-10 13:28:10] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:13 [I 2022-10-10 13:28:10] ...trap/engines/engine.py.229: process: 0.86566 | load: 0.00304 [I 2022-10-10 13:28:10] ...trap/engines/engine.py.230: loss: 0.11486 [I 2022-10-10 13:28:10] ...optimizers/trijoint.py.151: total_norm: 11.135647 [I 2022-10-10 13:28:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 1470/2304 [I 2022-10-10 13:28:18] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:57 [I 2022-10-10 13:28:18] ...trap/engines/engine.py.229: process: 0.85740 | load: 0.00307 [I 2022-10-10 13:28:18] ...trap/engines/engine.py.230: loss: 0.15638 [I 2022-10-10 13:28:18] ...optimizers/trijoint.py.151: total_norm: 9.784919 [I 2022-10-10 13:28:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 1480/2304 [I 2022-10-10 13:28:27] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:48 [I 2022-10-10 13:28:27] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00292 [I 2022-10-10 13:28:27] ...trap/engines/engine.py.230: loss: 0.17279 [I 2022-10-10 13:28:27] ...optimizers/trijoint.py.151: total_norm: 20.367619 [I 2022-10-10 13:28:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 1490/2304 [I 2022-10-10 13:28:35] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:36 [I 2022-10-10 13:28:35] ...trap/engines/engine.py.229: process: 0.83365 | load: 0.00302 [I 2022-10-10 13:28:35] ...trap/engines/engine.py.230: loss: 0.18195 [I 2022-10-10 13:28:35] ...optimizers/trijoint.py.151: total_norm: 15.222461 [I 2022-10-10 13:28:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1500/2304 [I 2022-10-10 13:28:44] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:25 [I 2022-10-10 13:28:44] ...trap/engines/engine.py.229: process: 0.83964 | load: 0.00292 [I 2022-10-10 13:28:44] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-10 13:28:44] ...optimizers/trijoint.py.151: total_norm: 17.259657 [I 2022-10-10 13:28:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1510/2304 [I 2022-10-10 13:28:53] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:14 [I 2022-10-10 13:28:53] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00292 [I 2022-10-10 13:28:53] ...trap/engines/engine.py.230: loss: 0.13695 [I 2022-10-10 13:28:53] ...optimizers/trijoint.py.151: total_norm: 43.715786 [I 2022-10-10 13:29:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 1520/2304 [I 2022-10-10 13:29:01] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:07 [I 2022-10-10 13:29:01] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00291 [I 2022-10-10 13:29:01] ...trap/engines/engine.py.230: loss: 0.12687 [I 2022-10-10 13:29:01] ...optimizers/trijoint.py.151: total_norm: 19.100424 [I 2022-10-10 13:29:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1530/2304 [I 2022-10-10 13:29:10] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:11:00 [I 2022-10-10 13:29:10] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00293 [I 2022-10-10 13:29:10] ...trap/engines/engine.py.230: loss: 0.14235 [I 2022-10-10 13:29:10] ...optimizers/trijoint.py.151: total_norm: 29.290701 [I 2022-10-10 13:29:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 1540/2304 [I 2022-10-10 13:29:18] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:52 [I 2022-10-10 13:29:18] ...trap/engines/engine.py.229: process: 0.86283 | load: 0.00298 [I 2022-10-10 13:29:18] ...trap/engines/engine.py.230: loss: 0.17830 [I 2022-10-10 13:29:18] ...optimizers/trijoint.py.151: total_norm: 10.970311 [I 2022-10-10 13:29:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 1550/2304 [I 2022-10-10 13:29:27] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:47 [I 2022-10-10 13:29:27] ...trap/engines/engine.py.229: process: 0.85122 | load: 0.00297 [I 2022-10-10 13:29:27] ...trap/engines/engine.py.230: loss: 0.16767 [I 2022-10-10 13:29:27] ...optimizers/trijoint.py.151: total_norm: 28.064423 [I 2022-10-10 13:29:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 1560/2304 [I 2022-10-10 13:29:35] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:34 [I 2022-10-10 13:29:35] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00294 [I 2022-10-10 13:29:35] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-10 13:29:35] ...optimizers/trijoint.py.151: total_norm: 14.775325 [I 2022-10-10 13:29:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1570/2304 [I 2022-10-10 13:29:44] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:25 [I 2022-10-10 13:29:44] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00298 [I 2022-10-10 13:29:44] ...trap/engines/engine.py.230: loss: 0.09141 [I 2022-10-10 13:29:44] ...optimizers/trijoint.py.151: total_norm: 11.164474 [I 2022-10-10 13:29:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1580/2304 [I 2022-10-10 13:29:53] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:18 [I 2022-10-10 13:29:53] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00295 [I 2022-10-10 13:29:53] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-10 13:29:53] ...optimizers/trijoint.py.151: total_norm: 33.292831 [I 2022-10-10 13:30:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 1590/2304 [I 2022-10-10 13:30:01] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:12 [I 2022-10-10 13:30:01] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00297 [I 2022-10-10 13:30:01] ...trap/engines/engine.py.230: loss: 0.13516 [I 2022-10-10 13:30:01] ...optimizers/trijoint.py.151: total_norm: 14.253709 [I 2022-10-10 13:30:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1600/2304 [I 2022-10-10 13:30:10] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:10:04 [I 2022-10-10 13:30:10] ...trap/engines/engine.py.229: process: 0.85861 | load: 0.00298 [I 2022-10-10 13:30:10] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-10 13:30:10] ...optimizers/trijoint.py.151: total_norm: 13.430174 [I 2022-10-10 13:30:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 1610/2304 [I 2022-10-10 13:30:18] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:55 [I 2022-10-10 13:30:18] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00295 [I 2022-10-10 13:30:18] ...trap/engines/engine.py.230: loss: 0.10483 [I 2022-10-10 13:30:18] ...optimizers/trijoint.py.151: total_norm: 15.091947 [I 2022-10-10 13:30:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 1620/2304 [I 2022-10-10 13:30:27] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:47 [I 2022-10-10 13:30:27] ...trap/engines/engine.py.229: process: 0.87836 | load: 0.00300 [I 2022-10-10 13:30:27] ...trap/engines/engine.py.230: loss: 0.11338 [I 2022-10-10 13:30:27] ...optimizers/trijoint.py.151: total_norm: 12.736531 [I 2022-10-10 13:30:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 1630/2304 [I 2022-10-10 13:30:36] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:37 [I 2022-10-10 13:30:36] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00298 [I 2022-10-10 13:30:36] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-10 13:30:36] ...optimizers/trijoint.py.151: total_norm: 11.532617 [I 2022-10-10 13:30:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1640/2304 [I 2022-10-10 13:30:44] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:24 [I 2022-10-10 13:30:44] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00300 [I 2022-10-10 13:30:44] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-10 13:30:44] ...optimizers/trijoint.py.151: total_norm: 10.468667 [I 2022-10-10 13:30:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1650/2304 [I 2022-10-10 13:30:53] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:21 [I 2022-10-10 13:30:53] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00306 [I 2022-10-10 13:30:53] ...trap/engines/engine.py.230: loss: 0.20703 [I 2022-10-10 13:30:53] ...optimizers/trijoint.py.151: total_norm: 13.933654 [I 2022-10-10 13:31:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 1660/2304 [I 2022-10-10 13:31:01] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:19 [I 2022-10-10 13:31:01] ...trap/engines/engine.py.229: process: 0.87669 | load: 0.00306 [I 2022-10-10 13:31:01] ...trap/engines/engine.py.230: loss: 0.13278 [I 2022-10-10 13:31:01] ...optimizers/trijoint.py.151: total_norm: 16.355671 [I 2022-10-10 13:31:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1670/2304 [I 2022-10-10 13:31:10] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:09:06 [I 2022-10-10 13:31:10] ...trap/engines/engine.py.229: process: 0.86474 | load: 0.00316 [I 2022-10-10 13:31:10] ...trap/engines/engine.py.230: loss: 0.14471 [I 2022-10-10 13:31:10] ...optimizers/trijoint.py.151: total_norm: 45.013771 [I 2022-10-10 13:31:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 1680/2304 [I 2022-10-10 13:31:19] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:09:02 [I 2022-10-10 13:31:19] ...trap/engines/engine.py.229: process: 0.87233 | load: 0.00306 [I 2022-10-10 13:31:19] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-10 13:31:19] ...optimizers/trijoint.py.151: total_norm: 14.929325 [I 2022-10-10 13:31:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 1690/2304 [I 2022-10-10 13:31:27] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:52 [I 2022-10-10 13:31:27] ...trap/engines/engine.py.229: process: 0.87124 | load: 0.00323 [I 2022-10-10 13:31:27] ...trap/engines/engine.py.230: loss: 0.08886 [I 2022-10-10 13:31:27] ...optimizers/trijoint.py.151: total_norm: 8.169837 [I 2022-10-10 13:31:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 1700/2304 [I 2022-10-10 13:31:36] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:42 [I 2022-10-10 13:31:36] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00326 [I 2022-10-10 13:31:36] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-10 13:31:36] ...optimizers/trijoint.py.151: total_norm: 6.882671 [I 2022-10-10 13:31:45] ...trap/engines/engine.py.224: train: epoch 73 | batch 1710/2304 [I 2022-10-10 13:31:45] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:35 [I 2022-10-10 13:31:45] ...trap/engines/engine.py.229: process: 0.87071 | load: 0.00314 [I 2022-10-10 13:31:45] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-10 13:31:45] ...optimizers/trijoint.py.151: total_norm: 11.099369 [I 2022-10-10 13:31:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 1720/2304 [I 2022-10-10 13:31:54] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:22 [I 2022-10-10 13:31:54] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00309 [I 2022-10-10 13:31:54] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-10 13:31:54] ...optimizers/trijoint.py.151: total_norm: 32.423466 [I 2022-10-10 13:32:02] ...trap/engines/engine.py.224: train: epoch 73 | batch 1730/2304 [I 2022-10-10 13:32:02] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:16 [I 2022-10-10 13:32:02] ...trap/engines/engine.py.229: process: 0.87745 | load: 0.00312 [I 2022-10-10 13:32:02] ...trap/engines/engine.py.230: loss: 0.10398 [I 2022-10-10 13:32:02] ...optimizers/trijoint.py.151: total_norm: 6.637685 [I 2022-10-10 13:32:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 1740/2304 [I 2022-10-10 13:32:11] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:08:09 [I 2022-10-10 13:32:11] ...trap/engines/engine.py.229: process: 0.86912 | load: 0.00311 [I 2022-10-10 13:32:11] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-10 13:32:11] ...optimizers/trijoint.py.151: total_norm: 12.791390 [I 2022-10-10 13:32:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 1750/2304 [I 2022-10-10 13:32:20] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:08:00 [I 2022-10-10 13:32:20] ...trap/engines/engine.py.229: process: 0.87019 | load: 0.00310 [I 2022-10-10 13:32:20] ...trap/engines/engine.py.230: loss: 0.14187 [I 2022-10-10 13:32:20] ...optimizers/trijoint.py.151: total_norm: 9.735641 [I 2022-10-10 13:32:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 1760/2304 [I 2022-10-10 13:32:28] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:52 [I 2022-10-10 13:32:28] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00308 [I 2022-10-10 13:32:28] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-10 13:32:28] ...optimizers/trijoint.py.151: total_norm: 26.069164 [I 2022-10-10 13:32:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 1770/2304 [I 2022-10-10 13:32:37] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:42 [I 2022-10-10 13:32:37] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00306 [I 2022-10-10 13:32:37] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-10 13:32:37] ...optimizers/trijoint.py.151: total_norm: 13.717997 [I 2022-10-10 13:32:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 1780/2304 [I 2022-10-10 13:32:46] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:32 [I 2022-10-10 13:32:46] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00315 [I 2022-10-10 13:32:46] ...trap/engines/engine.py.230: loss: 0.14656 [I 2022-10-10 13:32:46] ...optimizers/trijoint.py.151: total_norm: 15.790393 [I 2022-10-10 13:32:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 1790/2304 [I 2022-10-10 13:32:54] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:19 [I 2022-10-10 13:32:54] ...trap/engines/engine.py.229: process: 0.83927 | load: 0.00307 [I 2022-10-10 13:32:54] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-10 13:32:54] ...optimizers/trijoint.py.151: total_norm: 17.012611 [I 2022-10-10 13:33:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 1800/2304 [I 2022-10-10 13:33:03] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:17 [I 2022-10-10 13:33:03] ...trap/engines/engine.py.229: process: 0.87253 | load: 0.00308 [I 2022-10-10 13:33:03] ...trap/engines/engine.py.230: loss: 0.15841 [I 2022-10-10 13:33:03] ...optimizers/trijoint.py.151: total_norm: 13.006378 [I 2022-10-10 13:33:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 1810/2304 [I 2022-10-10 13:33:12] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:05 [I 2022-10-10 13:33:12] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00305 [I 2022-10-10 13:33:12] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-10 13:33:12] ...optimizers/trijoint.py.151: total_norm: 20.710016 [I 2022-10-10 13:33:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 1820/2304 [I 2022-10-10 13:33:20] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:59 [I 2022-10-10 13:33:20] ...trap/engines/engine.py.229: process: 0.86565 | load: 0.00313 [I 2022-10-10 13:33:20] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-10 13:33:20] ...optimizers/trijoint.py.151: total_norm: 9.319544 [I 2022-10-10 13:33:29] ...trap/engines/engine.py.224: train: epoch 73 | batch 1830/2304 [I 2022-10-10 13:33:29] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:49 [I 2022-10-10 13:33:29] ...trap/engines/engine.py.229: process: 0.89178 | load: 0.00326 [I 2022-10-10 13:33:29] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-10 13:33:29] ...optimizers/trijoint.py.151: total_norm: 15.346218 [I 2022-10-10 13:33:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 1840/2304 [I 2022-10-10 13:33:38] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:40 [I 2022-10-10 13:33:38] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00313 [I 2022-10-10 13:33:38] ...trap/engines/engine.py.230: loss: 0.11423 [I 2022-10-10 13:33:38] ...optimizers/trijoint.py.151: total_norm: 29.698551 [I 2022-10-10 13:33:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 1850/2304 [I 2022-10-10 13:33:46] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:33 [I 2022-10-10 13:33:46] ...trap/engines/engine.py.229: process: 0.87378 | load: 0.00312 [I 2022-10-10 13:33:46] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-10 13:33:46] ...optimizers/trijoint.py.151: total_norm: 14.723140 [I 2022-10-10 13:33:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 1860/2304 [I 2022-10-10 13:33:55] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:24 [I 2022-10-10 13:33:55] ...trap/engines/engine.py.229: process: 0.86585 | load: 0.00309 [I 2022-10-10 13:33:55] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-10 13:33:55] ...optimizers/trijoint.py.151: total_norm: 15.150373 [I 2022-10-10 13:34:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 1870/2304 [I 2022-10-10 13:34:04] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:13 [I 2022-10-10 13:34:04] ...trap/engines/engine.py.229: process: 0.85506 | load: 0.00317 [I 2022-10-10 13:34:04] ...trap/engines/engine.py.230: loss: 0.12375 [I 2022-10-10 13:34:04] ...optimizers/trijoint.py.151: total_norm: 11.588847 [I 2022-10-10 13:34:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 1880/2304 [I 2022-10-10 13:34:12] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:06 [I 2022-10-10 13:34:12] ...trap/engines/engine.py.229: process: 0.87277 | load: 0.00307 [I 2022-10-10 13:34:12] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-10 13:34:12] ...optimizers/trijoint.py.151: total_norm: 16.696770 [I 2022-10-10 13:34:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 1890/2304 [I 2022-10-10 13:34:21] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:57 [I 2022-10-10 13:34:21] ...trap/engines/engine.py.229: process: 0.86619 | load: 0.00311 [I 2022-10-10 13:34:21] ...trap/engines/engine.py.230: loss: 0.12711 [I 2022-10-10 13:34:21] ...optimizers/trijoint.py.151: total_norm: 22.847179 [I 2022-10-10 13:34:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 1900/2304 [I 2022-10-10 13:34:30] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:51 [I 2022-10-10 13:34:30] ...trap/engines/engine.py.229: process: 0.88595 | load: 0.00309 [I 2022-10-10 13:34:30] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-10 13:34:30] ...optimizers/trijoint.py.151: total_norm: 13.280295 [I 2022-10-10 13:34:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 1910/2304 [I 2022-10-10 13:34:38] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:43 [I 2022-10-10 13:34:38] ...trap/engines/engine.py.229: process: 0.88016 | load: 0.00312 [I 2022-10-10 13:34:38] ...trap/engines/engine.py.230: loss: 0.19649 [I 2022-10-10 13:34:38] ...optimizers/trijoint.py.151: total_norm: 25.791340 [I 2022-10-10 13:34:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 1920/2304 [I 2022-10-10 13:34:47] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:29 [I 2022-10-10 13:34:47] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00295 [I 2022-10-10 13:34:47] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-10 13:34:47] ...optimizers/trijoint.py.151: total_norm: 31.572338 [I 2022-10-10 13:34:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 1930/2304 [I 2022-10-10 13:34:55] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:19 [I 2022-10-10 13:34:55] ...trap/engines/engine.py.229: process: 0.84914 | load: 0.00300 [I 2022-10-10 13:34:55] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-10 13:34:55] ...optimizers/trijoint.py.151: total_norm: 23.132795 [I 2022-10-10 13:35:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 1940/2304 [I 2022-10-10 13:35:04] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:12 [I 2022-10-10 13:35:04] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00292 [I 2022-10-10 13:35:04] ...trap/engines/engine.py.230: loss: 0.09575 [I 2022-10-10 13:35:04] ...optimizers/trijoint.py.151: total_norm: 40.092297 [I 2022-10-10 13:35:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 1950/2304 [I 2022-10-10 13:35:13] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:00 [I 2022-10-10 13:35:13] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00294 [I 2022-10-10 13:35:13] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-10 13:35:13] ...optimizers/trijoint.py.151: total_norm: 15.092151 [I 2022-10-10 13:35:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 1960/2304 [I 2022-10-10 13:35:21] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:54 [I 2022-10-10 13:35:21] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00290 [I 2022-10-10 13:35:21] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-10 13:35:21] ...optimizers/trijoint.py.151: total_norm: 12.672462 [I 2022-10-10 13:35:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 1970/2304 [I 2022-10-10 13:35:30] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:47 [I 2022-10-10 13:35:30] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00291 [I 2022-10-10 13:35:30] ...trap/engines/engine.py.230: loss: 0.10121 [I 2022-10-10 13:35:30] ...optimizers/trijoint.py.151: total_norm: 13.859296 [I 2022-10-10 13:35:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 1980/2304 [I 2022-10-10 13:35:38] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:35 [I 2022-10-10 13:35:38] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00292 [I 2022-10-10 13:35:38] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-10 13:35:38] ...optimizers/trijoint.py.151: total_norm: 22.464861 [I 2022-10-10 13:35:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 1990/2304 [I 2022-10-10 13:35:47] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:29 [I 2022-10-10 13:35:47] ...trap/engines/engine.py.229: process: 0.85917 | load: 0.00294 [I 2022-10-10 13:35:47] ...trap/engines/engine.py.230: loss: 0.18965 [I 2022-10-10 13:35:47] ...optimizers/trijoint.py.151: total_norm: 21.256798 [I 2022-10-10 13:35:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 2000/2304 [I 2022-10-10 13:35:56] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:19 [I 2022-10-10 13:35:56] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00294 [I 2022-10-10 13:35:56] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-10 13:35:56] ...optimizers/trijoint.py.151: total_norm: 20.600523 [I 2022-10-10 13:36:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 2010/2304 [I 2022-10-10 13:36:04] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:11 [I 2022-10-10 13:36:04] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00325 [I 2022-10-10 13:36:04] ...trap/engines/engine.py.230: loss: 0.08408 [I 2022-10-10 13:36:04] ...optimizers/trijoint.py.151: total_norm: 10.392299 [I 2022-10-10 13:36:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 2020/2304 [I 2022-10-10 13:36:13] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:02 [I 2022-10-10 13:36:13] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00296 [I 2022-10-10 13:36:13] ...trap/engines/engine.py.230: loss: 0.18396 [I 2022-10-10 13:36:13] ...optimizers/trijoint.py.151: total_norm: 13.983631 [I 2022-10-10 13:36:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 2030/2304 [I 2022-10-10 13:36:21] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:54 [I 2022-10-10 13:36:21] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00302 [I 2022-10-10 13:36:21] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-10 13:36:21] ...optimizers/trijoint.py.151: total_norm: 32.542938 [I 2022-10-10 13:36:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 2040/2304 [I 2022-10-10 13:36:30] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:45 [I 2022-10-10 13:36:30] ...trap/engines/engine.py.229: process: 0.87116 | load: 0.00332 [I 2022-10-10 13:36:30] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-10 13:36:30] ...optimizers/trijoint.py.151: total_norm: 19.593346 [I 2022-10-10 13:36:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 2050/2304 [I 2022-10-10 13:36:38] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:37 [I 2022-10-10 13:36:38] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00295 [I 2022-10-10 13:36:38] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-10 13:36:38] ...optimizers/trijoint.py.151: total_norm: 13.026519 [I 2022-10-10 13:36:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 2060/2304 [I 2022-10-10 13:36:47] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:27 [I 2022-10-10 13:36:47] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00302 [I 2022-10-10 13:36:47] ...trap/engines/engine.py.230: loss: 0.22656 [I 2022-10-10 13:36:47] ...optimizers/trijoint.py.151: total_norm: 31.497690 [I 2022-10-10 13:36:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 2070/2304 [I 2022-10-10 13:36:56] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:19 [I 2022-10-10 13:36:56] ...trap/engines/engine.py.229: process: 0.86122 | load: 0.00294 [I 2022-10-10 13:36:56] ...trap/engines/engine.py.230: loss: 0.08766 [I 2022-10-10 13:36:56] ...optimizers/trijoint.py.151: total_norm: 12.856943 [I 2022-10-10 13:37:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 2080/2304 [I 2022-10-10 13:37:04] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:11 [I 2022-10-10 13:37:04] ...trap/engines/engine.py.229: process: 0.86284 | load: 0.00298 [I 2022-10-10 13:37:04] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-10 13:37:04] ...optimizers/trijoint.py.151: total_norm: 15.811000 [I 2022-10-10 13:37:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 2090/2304 [I 2022-10-10 13:37:13] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:02 [I 2022-10-10 13:37:13] ...trap/engines/engine.py.229: process: 0.84793 | load: 0.00308 [I 2022-10-10 13:37:13] ...trap/engines/engine.py.230: loss: 0.17057 [I 2022-10-10 13:37:13] ...optimizers/trijoint.py.151: total_norm: 14.765319 [I 2022-10-10 13:37:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 2100/2304 [I 2022-10-10 13:37:21] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:58 [I 2022-10-10 13:37:21] ...trap/engines/engine.py.229: process: 0.88866 | load: 0.00311 [I 2022-10-10 13:37:21] ...trap/engines/engine.py.230: loss: 0.10778 [I 2022-10-10 13:37:21] ...optimizers/trijoint.py.151: total_norm: 12.376851 [I 2022-10-10 13:37:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 2110/2304 [I 2022-10-10 13:37:49] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:07:49 [I 2022-10-10 13:37:49] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00318 [I 2022-10-10 13:37:49] ...trap/engines/engine.py.230: loss: 0.15163 [I 2022-10-10 13:37:49] ...optimizers/trijoint.py.151: total_norm: 9.992249 [I 2022-10-10 13:37:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 2120/2304 [I 2022-10-10 13:37:58] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:03:09 [I 2022-10-10 13:37:58] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00310 [I 2022-10-10 13:37:58] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-10 13:37:58] ...optimizers/trijoint.py.151: total_norm: 15.203589 [I 2022-10-10 13:38:06] ...trap/engines/engine.py.224: train: epoch 73 | batch 2130/2304 [I 2022-10-10 13:38:06] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:33 [I 2022-10-10 13:38:06] ...trap/engines/engine.py.229: process: 0.85660 | load: 0.00307 [I 2022-10-10 13:38:06] ...trap/engines/engine.py.230: loss: 0.14076 [I 2022-10-10 13:38:06] ...optimizers/trijoint.py.151: total_norm: 13.870987 [I 2022-10-10 13:38:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 2140/2304 [I 2022-10-10 13:38:15] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:02:22 [I 2022-10-10 13:38:15] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00304 [I 2022-10-10 13:38:15] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-10 13:38:15] ...optimizers/trijoint.py.151: total_norm: 16.208202 [I 2022-10-10 13:38:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 2150/2304 [I 2022-10-10 13:38:24] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:02:13 [I 2022-10-10 13:38:24] ...trap/engines/engine.py.229: process: 0.86209 | load: 0.00316 [I 2022-10-10 13:38:24] ...trap/engines/engine.py.230: loss: 0.15203 [I 2022-10-10 13:38:24] ...optimizers/trijoint.py.151: total_norm: 16.813021 [I 2022-10-10 13:38:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 2160/2304 [I 2022-10-10 13:38:32] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:02:04 [I 2022-10-10 13:38:32] ...trap/engines/engine.py.229: process: 0.87155 | load: 0.00308 [I 2022-10-10 13:38:32] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-10 13:38:32] ...optimizers/trijoint.py.151: total_norm: 10.487470 [I 2022-10-10 13:38:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 2170/2304 [I 2022-10-10 13:38:41] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:56 [I 2022-10-10 13:38:41] ...trap/engines/engine.py.229: process: 0.86806 | load: 0.00313 [I 2022-10-10 13:38:41] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-10 13:38:41] ...optimizers/trijoint.py.151: total_norm: 28.836479 [I 2022-10-10 13:38:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 2180/2304 [I 2022-10-10 13:38:50] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:47 [I 2022-10-10 13:38:50] ...trap/engines/engine.py.229: process: 0.85750 | load: 0.00309 [I 2022-10-10 13:38:50] ...trap/engines/engine.py.230: loss: 0.11030 [I 2022-10-10 13:38:50] ...optimizers/trijoint.py.151: total_norm: 28.284143 [I 2022-10-10 13:38:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 2190/2304 [I 2022-10-10 13:38:58] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:38 [I 2022-10-10 13:38:58] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00324 [I 2022-10-10 13:38:58] ...trap/engines/engine.py.230: loss: 0.13404 [I 2022-10-10 13:38:58] ...optimizers/trijoint.py.151: total_norm: 13.135806 [I 2022-10-10 13:39:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 2200/2304 [I 2022-10-10 13:39:07] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:30 [I 2022-10-10 13:39:07] ...trap/engines/engine.py.229: process: 0.87485 | load: 0.00319 [I 2022-10-10 13:39:07] ...trap/engines/engine.py.230: loss: 0.14611 [I 2022-10-10 13:39:07] ...optimizers/trijoint.py.151: total_norm: 9.366115 [I 2022-10-10 13:39:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 2210/2304 [I 2022-10-10 13:39:16] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:21 [I 2022-10-10 13:39:16] ...trap/engines/engine.py.229: process: 0.86774 | load: 0.00317 [I 2022-10-10 13:39:16] ...trap/engines/engine.py.230: loss: 0.16344 [I 2022-10-10 13:39:16] ...optimizers/trijoint.py.151: total_norm: 17.629488 [I 2022-10-10 13:39:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 2220/2304 [I 2022-10-10 13:39:24] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:12 [I 2022-10-10 13:39:24] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00310 [I 2022-10-10 13:39:24] ...trap/engines/engine.py.230: loss: 0.23225 [I 2022-10-10 13:39:24] ...optimizers/trijoint.py.151: total_norm: 24.360588 [I 2022-10-10 13:39:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 2230/2304 [I 2022-10-10 13:39:33] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:01:03 [I 2022-10-10 13:39:33] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00320 [I 2022-10-10 13:39:33] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-10 13:39:33] ...optimizers/trijoint.py.151: total_norm: 15.098445 [I 2022-10-10 13:39:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 2240/2304 [I 2022-10-10 13:39:42] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:55 [I 2022-10-10 13:39:42] ...trap/engines/engine.py.229: process: 0.86989 | load: 0.00311 [I 2022-10-10 13:39:42] ...trap/engines/engine.py.230: loss: 0.17221 [I 2022-10-10 13:39:42] ...optimizers/trijoint.py.151: total_norm: 16.733980 [I 2022-10-10 13:39:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 2250/2304 [I 2022-10-10 13:39:50] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:46 [I 2022-10-10 13:39:50] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00308 [I 2022-10-10 13:39:50] ...trap/engines/engine.py.230: loss: 0.18554 [I 2022-10-10 13:39:50] ...optimizers/trijoint.py.151: total_norm: 16.489754 [I 2022-10-10 13:39:59] ...trap/engines/engine.py.224: train: epoch 73 | batch 2260/2304 [I 2022-10-10 13:39:59] ...trap/engines/engine.py.228: elapsed: 0:32:53 | left: 0:00:38 [I 2022-10-10 13:39:59] ...trap/engines/engine.py.229: process: 0.87191 | load: 0.00310 [I 2022-10-10 13:39:59] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-10 13:39:59] ...optimizers/trijoint.py.151: total_norm: 14.537983 [I 2022-10-10 13:40:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 2270/2304 [I 2022-10-10 13:40:08] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:29 [I 2022-10-10 13:40:08] ...trap/engines/engine.py.229: process: 0.87107 | load: 0.00307 [I 2022-10-10 13:40:08] ...trap/engines/engine.py.230: loss: 0.16142 [I 2022-10-10 13:40:08] ...optimizers/trijoint.py.151: total_norm: 15.263309 [I 2022-10-10 13:40:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 2280/2304 [I 2022-10-10 13:40:16] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:20 [I 2022-10-10 13:40:16] ...trap/engines/engine.py.229: process: 0.85791 | load: 0.00315 [I 2022-10-10 13:40:16] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-10 13:40:16] ...optimizers/trijoint.py.151: total_norm: 15.821743 [I 2022-10-10 13:40:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 2290/2304 [I 2022-10-10 13:40:25] ...trap/engines/engine.py.228: elapsed: 0:33:19 | left: 0:00:12 [I 2022-10-10 13:40:25] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00312 [I 2022-10-10 13:40:25] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-10 13:40:25] ...optimizers/trijoint.py.151: total_norm: 9.318835 [I 2022-10-10 13:40:34] ...trap/engines/engine.py.224: train: epoch 73 | batch 2300/2304 [I 2022-10-10 13:40:34] ...trap/engines/engine.py.228: elapsed: 0:33:27 | left: 0:00:03 [I 2022-10-10 13:40:34] ...trap/engines/engine.py.229: process: 0.85299 | load: 0.00023 [I 2022-10-10 13:40:34] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-10 13:40:34] ...optimizers/trijoint.py.151: total_norm: 15.837778 [I 2022-10-10 13:40:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 2304/2304 [I 2022-10-10 13:40:37] ...trap/engines/engine.py.228: elapsed: 0:33:31 | left: 0:00:00 [I 2022-10-10 13:40:37] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00018 [I 2022-10-10 13:40:37] ...trap/engines/engine.py.230: loss: 0.21287 [I 2022-10-10 13:40:37] ...optimizers/trijoint.py.151: total_norm: 10.442898 [S 2022-10-10 13:40:39] ...trap/engines/engine.py.129: train_epoch.epoch: 73 [S 2022-10-10 13:40:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.13753787309513174 [I 2022-10-10 13:40:39] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 13:40:40] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 73 [I 2022-10-10 13:40:40] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 13:40:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 13:40:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 13:40:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 13:40:44] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 13:40:45] ...trap/engines/engine.py.321: eval: epoch 73 | batch 0/510 [I 2022-10-10 13:40:45] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:43:08 [I 2022-10-10 13:40:45] ...trap/engines/engine.py.326: process: 5.07628 | load: 4.26113 [I 2022-10-10 13:40:49] ...trap/engines/engine.py.321: eval: epoch 73 | batch 10/510 [I 2022-10-10 13:40:49] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:50 [I 2022-10-10 13:40:49] ...trap/engines/engine.py.326: process: 0.42928 | load: 0.00020 [I 2022-10-10 13:40:55] ...trap/engines/engine.py.321: eval: epoch 73 | batch 20/510 [I 2022-10-10 13:40:55] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:01 [I 2022-10-10 13:40:55] ...trap/engines/engine.py.326: process: 0.61831 | load: 0.00021 [I 2022-10-10 13:41:01] ...trap/engines/engine.py.321: eval: epoch 73 | batch 30/510 [I 2022-10-10 13:41:01] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:51 [I 2022-10-10 13:41:01] ...trap/engines/engine.py.326: process: 0.58686 | load: 0.00020 [I 2022-10-10 13:41:07] ...trap/engines/engine.py.321: eval: epoch 73 | batch 40/510 [I 2022-10-10 13:41:07] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:23 [I 2022-10-10 13:41:07] ...trap/engines/engine.py.326: process: 0.58214 | load: 0.00019 [I 2022-10-10 13:41:11] ...trap/engines/engine.py.321: eval: epoch 73 | batch 50/510 [I 2022-10-10 13:41:11] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:36 [I 2022-10-10 13:41:11] ...trap/engines/engine.py.326: process: 0.44795 | load: 0.00019 [I 2022-10-10 13:41:16] ...trap/engines/engine.py.321: eval: epoch 73 | batch 60/510 [I 2022-10-10 13:41:16] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:19 [I 2022-10-10 13:41:16] ...trap/engines/engine.py.326: process: 0.43541 | load: 0.00018 [I 2022-10-10 13:41:21] ...trap/engines/engine.py.321: eval: epoch 73 | batch 70/510 [I 2022-10-10 13:41:21] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:26 [I 2022-10-10 13:41:21] ...trap/engines/engine.py.326: process: 0.51897 | load: 0.00018 [I 2022-10-10 13:41:26] ...trap/engines/engine.py.321: eval: epoch 73 | batch 80/510 [I 2022-10-10 13:41:26] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:42 [I 2022-10-10 13:41:26] ...trap/engines/engine.py.326: process: 0.52564 | load: 0.00018 [I 2022-10-10 13:41:30] ...trap/engines/engine.py.321: eval: epoch 73 | batch 90/510 [I 2022-10-10 13:41:30] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:06 [I 2022-10-10 13:41:30] ...trap/engines/engine.py.326: process: 0.45211 | load: 0.00018 [I 2022-10-10 13:41:34] ...trap/engines/engine.py.321: eval: epoch 73 | batch 100/510 [I 2022-10-10 13:41:34] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:48 [I 2022-10-10 13:41:34] ...trap/engines/engine.py.326: process: 0.31074 | load: 0.00018 [I 2022-10-10 13:41:38] ...trap/engines/engine.py.321: eval: epoch 73 | batch 110/510 [I 2022-10-10 13:41:38] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:11 [I 2022-10-10 13:41:38] ...trap/engines/engine.py.326: process: 0.32143 | load: 0.00021 [I 2022-10-10 13:41:41] ...trap/engines/engine.py.321: eval: epoch 73 | batch 120/510 [I 2022-10-10 13:41:41] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:03 [I 2022-10-10 13:41:41] ...trap/engines/engine.py.326: process: 0.31503 | load: 0.00021 [I 2022-10-10 13:41:44] ...trap/engines/engine.py.321: eval: epoch 73 | batch 130/510 [I 2022-10-10 13:41:44] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-10 13:41:44] ...trap/engines/engine.py.326: process: 0.31751 | load: 0.00019 [I 2022-10-10 13:41:47] ...trap/engines/engine.py.321: eval: epoch 73 | batch 140/510 [I 2022-10-10 13:41:47] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:58 [I 2022-10-10 13:41:47] ...trap/engines/engine.py.326: process: 0.31923 | load: 0.00019 [I 2022-10-10 13:41:50] ...trap/engines/engine.py.321: eval: epoch 73 | batch 150/510 [I 2022-10-10 13:41:50] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:55 [I 2022-10-10 13:41:50] ...trap/engines/engine.py.326: process: 0.31795 | load: 0.00020 [I 2022-10-10 13:41:54] ...trap/engines/engine.py.321: eval: epoch 73 | batch 160/510 [I 2022-10-10 13:41:54] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:50 [I 2022-10-10 13:41:54] ...trap/engines/engine.py.326: process: 0.31160 | load: 0.00023 [I 2022-10-10 13:41:57] ...trap/engines/engine.py.321: eval: epoch 73 | batch 170/510 [I 2022-10-10 13:41:57] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-10 13:41:57] ...trap/engines/engine.py.326: process: 0.32083 | load: 0.00019 [I 2022-10-10 13:42:00] ...trap/engines/engine.py.321: eval: epoch 73 | batch 180/510 [I 2022-10-10 13:42:00] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-10 13:42:00] ...trap/engines/engine.py.326: process: 0.31849 | load: 0.00020 [I 2022-10-10 13:42:03] ...trap/engines/engine.py.321: eval: epoch 73 | batch 190/510 [I 2022-10-10 13:42:03] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:42 [I 2022-10-10 13:42:03] ...trap/engines/engine.py.326: process: 0.31933 | load: 0.00019 [I 2022-10-10 13:42:06] ...trap/engines/engine.py.321: eval: epoch 73 | batch 200/510 [I 2022-10-10 13:42:06] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-10 13:42:06] ...trap/engines/engine.py.326: process: 0.32403 | load: 0.00019 [I 2022-10-10 13:42:10] ...trap/engines/engine.py.321: eval: epoch 73 | batch 210/510 [I 2022-10-10 13:42:10] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:35 [I 2022-10-10 13:42:10] ...trap/engines/engine.py.326: process: 0.31759 | load: 0.00019 [I 2022-10-10 13:42:13] ...trap/engines/engine.py.321: eval: epoch 73 | batch 220/510 [I 2022-10-10 13:42:13] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:33 [I 2022-10-10 13:42:13] ...trap/engines/engine.py.326: process: 0.32621 | load: 0.00019 [I 2022-10-10 13:42:16] ...trap/engines/engine.py.321: eval: epoch 73 | batch 230/510 [I 2022-10-10 13:42:16] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-10 13:42:16] ...trap/engines/engine.py.326: process: 0.31304 | load: 0.00019 [I 2022-10-10 13:42:19] ...trap/engines/engine.py.321: eval: epoch 73 | batch 240/510 [I 2022-10-10 13:42:19] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-10 13:42:19] ...trap/engines/engine.py.326: process: 0.31685 | load: 0.00019 [I 2022-10-10 13:42:22] ...trap/engines/engine.py.321: eval: epoch 73 | batch 250/510 [I 2022-10-10 13:42:22] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-10 13:42:22] ...trap/engines/engine.py.326: process: 0.31822 | load: 0.00019 [I 2022-10-10 13:42:26] ...trap/engines/engine.py.321: eval: epoch 73 | batch 260/510 [I 2022-10-10 13:42:26] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:20 [I 2022-10-10 13:42:26] ...trap/engines/engine.py.326: process: 0.32620 | load: 0.00019 [I 2022-10-10 13:42:29] ...trap/engines/engine.py.321: eval: epoch 73 | batch 270/510 [I 2022-10-10 13:42:29] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:17 [I 2022-10-10 13:42:29] ...trap/engines/engine.py.326: process: 0.32050 | load: 0.00023 [I 2022-10-10 13:42:32] ...trap/engines/engine.py.321: eval: epoch 73 | batch 280/510 [I 2022-10-10 13:42:32] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-10 13:42:32] ...trap/engines/engine.py.326: process: 0.31560 | load: 0.00019 [I 2022-10-10 13:42:35] ...trap/engines/engine.py.321: eval: epoch 73 | batch 290/510 [I 2022-10-10 13:42:35] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-10 13:42:35] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00020 [I 2022-10-10 13:42:39] ...trap/engines/engine.py.321: eval: epoch 73 | batch 300/510 [I 2022-10-10 13:42:39] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:06 [I 2022-10-10 13:42:39] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00018 [I 2022-10-10 13:42:42] ...trap/engines/engine.py.321: eval: epoch 73 | batch 310/510 [I 2022-10-10 13:42:42] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-10 13:42:42] ...trap/engines/engine.py.326: process: 0.31797 | load: 0.00020 [I 2022-10-10 13:42:45] ...trap/engines/engine.py.321: eval: epoch 73 | batch 320/510 [I 2022-10-10 13:42:45] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-10 13:42:45] ...trap/engines/engine.py.326: process: 0.31746 | load: 0.00020 [I 2022-10-10 13:42:48] ...trap/engines/engine.py.321: eval: epoch 73 | batch 330/510 [I 2022-10-10 13:42:48] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-10 13:42:48] ...trap/engines/engine.py.326: process: 0.32213 | load: 0.00020 [I 2022-10-10 13:42:51] ...trap/engines/engine.py.321: eval: epoch 73 | batch 340/510 [I 2022-10-10 13:42:51] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-10 13:42:51] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00020 [I 2022-10-10 13:42:55] ...trap/engines/engine.py.321: eval: epoch 73 | batch 350/510 [I 2022-10-10 13:42:55] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-10 13:42:55] ...trap/engines/engine.py.326: process: 0.32291 | load: 0.00019 [I 2022-10-10 13:42:58] ...trap/engines/engine.py.321: eval: epoch 73 | batch 360/510 [I 2022-10-10 13:42:58] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-10 13:42:58] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00021 [I 2022-10-10 13:43:01] ...trap/engines/engine.py.321: eval: epoch 73 | batch 370/510 [I 2022-10-10 13:43:01] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-10 13:43:01] ...trap/engines/engine.py.326: process: 0.32278 | load: 0.00019 [I 2022-10-10 13:43:04] ...trap/engines/engine.py.321: eval: epoch 73 | batch 380/510 [I 2022-10-10 13:43:04] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-10 13:43:04] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00020 [I 2022-10-10 13:43:07] ...trap/engines/engine.py.321: eval: epoch 73 | batch 390/510 [I 2022-10-10 13:43:07] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-10 13:43:07] ...trap/engines/engine.py.326: process: 0.31360 | load: 0.00020 [I 2022-10-10 13:43:11] ...trap/engines/engine.py.321: eval: epoch 73 | batch 400/510 [I 2022-10-10 13:43:11] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-10 13:43:11] ...trap/engines/engine.py.326: process: 0.32372 | load: 0.00019 [I 2022-10-10 13:43:14] ...trap/engines/engine.py.321: eval: epoch 73 | batch 410/510 [I 2022-10-10 13:43:14] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-10 13:43:14] ...trap/engines/engine.py.326: process: 0.32699 | load: 0.00019 [I 2022-10-10 13:43:17] ...trap/engines/engine.py.321: eval: epoch 73 | batch 420/510 [I 2022-10-10 13:43:17] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-10 13:43:17] ...trap/engines/engine.py.326: process: 0.32106 | load: 0.00019 [I 2022-10-10 13:43:20] ...trap/engines/engine.py.321: eval: epoch 73 | batch 430/510 [I 2022-10-10 13:43:20] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-10 13:43:20] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00021 [I 2022-10-10 13:43:24] ...trap/engines/engine.py.321: eval: epoch 73 | batch 440/510 [I 2022-10-10 13:43:24] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-10 13:43:24] ...trap/engines/engine.py.326: process: 0.32920 | load: 0.00021 [I 2022-10-10 13:43:27] ...trap/engines/engine.py.321: eval: epoch 73 | batch 450/510 [I 2022-10-10 13:43:27] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-10 13:43:27] ...trap/engines/engine.py.326: process: 0.33106 | load: 0.00019 [I 2022-10-10 13:43:30] ...trap/engines/engine.py.321: eval: epoch 73 | batch 460/510 [I 2022-10-10 13:43:30] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-10 13:43:30] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00019 [I 2022-10-10 13:43:33] ...trap/engines/engine.py.321: eval: epoch 73 | batch 470/510 [I 2022-10-10 13:43:33] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-10 13:43:33] ...trap/engines/engine.py.326: process: 0.31932 | load: 0.00019 [I 2022-10-10 13:43:36] ...trap/engines/engine.py.321: eval: epoch 73 | batch 480/510 [I 2022-10-10 13:43:36] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-10 13:43:36] ...trap/engines/engine.py.326: process: 0.32565 | load: 0.00020 [I 2022-10-10 13:43:40] ...trap/engines/engine.py.321: eval: epoch 73 | batch 490/510 [I 2022-10-10 13:43:40] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 13:43:40] ...trap/engines/engine.py.326: process: 0.31736 | load: 0.00018 [I 2022-10-10 13:43:43] ...trap/engines/engine.py.321: eval: epoch 73 | batch 500/510 [I 2022-10-10 13:43:43] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-10 13:43:43] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00019 [I 2022-10-10 13:43:46] ...trap/engines/engine.py.321: eval: epoch 73 | batch 510/510 [I 2022-10-10 13:43:46] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-10 13:43:46] ...trap/engines/engine.py.326: process: 0.31658 | load: 0.00015 [S 2022-10-10 13:43:47] ...trap/engines/engine.py.132: eval_epoch.epoch: 73 [I 2022-10-10 13:43:47] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746100 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.914000 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940300 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732800 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911700 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.939200 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010416 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006768 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007281 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017110 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008556 [S 2022-10-10 13:44:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006161 [I 2022-10-10 13:44:30] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-10 13:44:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 13:44:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 13:44:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 13:44:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 13:44:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 13:44:35] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 13:44:41] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 13:44:41] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 13:44:41] ...trap/engines/engine.py.470: Saving model... [I 2022-10-10 13:44:44] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 13:44:50] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 13:44:50] ...trap/engines/engine.py.163: Training model on trainset for epoch 74 [I 2022-10-10 13:44:50] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 13:44:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 0/2304 [I 2022-10-10 13:44:54] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:00:07 [I 2022-10-10 13:44:54] ...trap/engines/engine.py.229: process: 3.12814 | load: 2.19244 [I 2022-10-10 13:44:54] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-10 13:44:54] ...optimizers/trijoint.py.151: total_norm: 7.572218 [I 2022-10-10 13:45:02] ...trap/engines/engine.py.224: train: epoch 74 | batch 10/2304 [I 2022-10-10 13:45:02] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:21 [I 2022-10-10 13:45:02] ...trap/engines/engine.py.229: process: 0.85787 | load: 0.00736 [I 2022-10-10 13:45:02] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-10 13:45:02] ...optimizers/trijoint.py.151: total_norm: 21.872869 [I 2022-10-10 13:45:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 20/2304 [I 2022-10-10 13:45:11] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:55 [I 2022-10-10 13:45:11] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00481 [I 2022-10-10 13:45:11] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-10 13:45:11] ...optimizers/trijoint.py.151: total_norm: 18.506058 [I 2022-10-10 13:45:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 30/2304 [I 2022-10-10 13:45:20] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:59 [I 2022-10-10 13:45:20] ...trap/engines/engine.py.229: process: 0.86672 | load: 0.00574 [I 2022-10-10 13:45:20] ...trap/engines/engine.py.230: loss: 0.12106 [I 2022-10-10 13:45:20] ...optimizers/trijoint.py.151: total_norm: 11.538970 [I 2022-10-10 13:45:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 40/2304 [I 2022-10-10 13:45:28] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:39 [I 2022-10-10 13:45:28] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00445 [I 2022-10-10 13:45:28] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-10 13:45:28] ...optimizers/trijoint.py.151: total_norm: 14.094319 [I 2022-10-10 13:45:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 50/2304 [I 2022-10-10 13:45:37] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:38 [I 2022-10-10 13:45:37] ...trap/engines/engine.py.229: process: 0.87337 | load: 0.00456 [I 2022-10-10 13:45:37] ...trap/engines/engine.py.230: loss: 0.14537 [I 2022-10-10 13:45:37] ...optimizers/trijoint.py.151: total_norm: 14.502896 [I 2022-10-10 13:45:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 60/2304 [I 2022-10-10 13:45:46] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:21 [I 2022-10-10 13:45:46] ...trap/engines/engine.py.229: process: 0.86410 | load: 0.00384 [I 2022-10-10 13:45:46] ...trap/engines/engine.py.230: loss: 0.18478 [I 2022-10-10 13:45:46] ...optimizers/trijoint.py.151: total_norm: 15.801023 [I 2022-10-10 13:45:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 70/2304 [I 2022-10-10 13:45:54] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:25 [I 2022-10-10 13:45:54] ...trap/engines/engine.py.229: process: 0.88303 | load: 0.00412 [I 2022-10-10 13:45:54] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-10 13:45:54] ...optimizers/trijoint.py.151: total_norm: 20.799301 [I 2022-10-10 13:46:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 80/2304 [I 2022-10-10 13:46:03] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:13 [I 2022-10-10 13:46:03] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00353 [I 2022-10-10 13:46:03] ...trap/engines/engine.py.230: loss: 0.12620 [I 2022-10-10 13:46:03] ...optimizers/trijoint.py.151: total_norm: 23.553530 [I 2022-10-10 13:46:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 90/2304 [I 2022-10-10 13:46:12] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:32:01 [I 2022-10-10 13:46:12] ...trap/engines/engine.py.229: process: 0.88747 | load: 0.00341 [I 2022-10-10 13:46:12] ...trap/engines/engine.py.230: loss: 0.10607 [I 2022-10-10 13:46:12] ...optimizers/trijoint.py.151: total_norm: 11.273048 [I 2022-10-10 13:46:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 100/2304 [I 2022-10-10 13:46:20] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:36 [I 2022-10-10 13:46:20] ...trap/engines/engine.py.229: process: 0.84969 | load: 0.00355 [I 2022-10-10 13:46:20] ...trap/engines/engine.py.230: loss: 0.17073 [I 2022-10-10 13:46:20] ...optimizers/trijoint.py.151: total_norm: 13.344046 [I 2022-10-10 13:46:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 110/2304 [I 2022-10-10 13:46:29] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:24 [I 2022-10-10 13:46:29] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00337 [I 2022-10-10 13:46:29] ...trap/engines/engine.py.230: loss: 0.10448 [I 2022-10-10 13:46:29] ...optimizers/trijoint.py.151: total_norm: 10.384721 [I 2022-10-10 13:46:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 120/2304 [I 2022-10-10 13:46:38] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:34 [I 2022-10-10 13:46:38] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00365 [I 2022-10-10 13:46:38] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-10 13:46:38] ...optimizers/trijoint.py.151: total_norm: 12.635610 [I 2022-10-10 13:46:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 130/2304 [I 2022-10-10 13:46:46] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:21 [I 2022-10-10 13:46:46] ...trap/engines/engine.py.229: process: 0.86678 | load: 0.00313 [I 2022-10-10 13:46:46] ...trap/engines/engine.py.230: loss: 0.12169 [I 2022-10-10 13:46:46] ...optimizers/trijoint.py.151: total_norm: 10.189567 [I 2022-10-10 13:46:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 140/2304 [I 2022-10-10 13:46:55] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:14 [I 2022-10-10 13:46:55] ...trap/engines/engine.py.229: process: 0.86384 | load: 0.00337 [I 2022-10-10 13:46:55] ...trap/engines/engine.py.230: loss: 0.13047 [I 2022-10-10 13:46:55] ...optimizers/trijoint.py.151: total_norm: 12.739064 [I 2022-10-10 13:47:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 150/2304 [I 2022-10-10 13:47:04] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:50 [I 2022-10-10 13:47:04] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00344 [I 2022-10-10 13:47:04] ...trap/engines/engine.py.230: loss: 0.13154 [I 2022-10-10 13:47:04] ...optimizers/trijoint.py.151: total_norm: 13.061251 [I 2022-10-10 13:47:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 160/2304 [I 2022-10-10 13:47:12] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:31:06 [I 2022-10-10 13:47:12] ...trap/engines/engine.py.229: process: 0.86946 | load: 0.00343 [I 2022-10-10 13:47:12] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-10 13:47:12] ...optimizers/trijoint.py.151: total_norm: 13.047645 [I 2022-10-10 13:47:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 170/2304 [I 2022-10-10 13:47:21] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:49 [I 2022-10-10 13:47:21] ...trap/engines/engine.py.229: process: 0.87272 | load: 0.00339 [I 2022-10-10 13:47:21] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-10 13:47:21] ...optimizers/trijoint.py.151: total_norm: 69.931709 [I 2022-10-10 13:47:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 180/2304 [I 2022-10-10 13:47:30] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:34 [I 2022-10-10 13:47:30] ...trap/engines/engine.py.229: process: 0.87735 | load: 0.00334 [I 2022-10-10 13:47:30] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-10 13:47:30] ...optimizers/trijoint.py.151: total_norm: 16.867285 [I 2022-10-10 13:47:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 190/2304 [I 2022-10-10 13:47:38] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:16 [I 2022-10-10 13:47:38] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00350 [I 2022-10-10 13:47:38] ...trap/engines/engine.py.230: loss: 0.15295 [I 2022-10-10 13:47:38] ...optimizers/trijoint.py.151: total_norm: 19.143833 [I 2022-10-10 13:47:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 200/2304 [I 2022-10-10 13:47:47] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:23 [I 2022-10-10 13:47:47] ...trap/engines/engine.py.229: process: 0.87363 | load: 0.00312 [I 2022-10-10 13:47:47] ...trap/engines/engine.py.230: loss: 0.12616 [I 2022-10-10 13:47:47] ...optimizers/trijoint.py.151: total_norm: 14.928016 [I 2022-10-10 13:47:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 210/2304 [I 2022-10-10 13:47:56] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:30:02 [I 2022-10-10 13:47:56] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00342 [I 2022-10-10 13:47:56] ...trap/engines/engine.py.230: loss: 0.13530 [I 2022-10-10 13:47:56] ...optimizers/trijoint.py.151: total_norm: 10.090578 [I 2022-10-10 13:48:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 220/2304 [I 2022-10-10 13:48:04] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:49 [I 2022-10-10 13:48:04] ...trap/engines/engine.py.229: process: 0.85307 | load: 0.00301 [I 2022-10-10 13:48:04] ...trap/engines/engine.py.230: loss: 0.16600 [I 2022-10-10 13:48:04] ...optimizers/trijoint.py.151: total_norm: 27.346394 [I 2022-10-10 13:48:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 230/2304 [I 2022-10-10 13:48:13] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:27 [I 2022-10-10 13:48:13] ...trap/engines/engine.py.229: process: 0.85957 | load: 0.00311 [I 2022-10-10 13:48:13] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-10 13:48:13] ...optimizers/trijoint.py.151: total_norm: 16.157469 [I 2022-10-10 13:48:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 240/2304 [I 2022-10-10 13:48:21] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:26 [I 2022-10-10 13:48:21] ...trap/engines/engine.py.229: process: 0.85996 | load: 0.00321 [I 2022-10-10 13:48:21] ...trap/engines/engine.py.230: loss: 0.15774 [I 2022-10-10 13:48:21] ...optimizers/trijoint.py.151: total_norm: 16.973986 [I 2022-10-10 13:48:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 250/2304 [I 2022-10-10 13:48:30] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:08 [I 2022-10-10 13:48:30] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00296 [I 2022-10-10 13:48:30] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-10 13:48:30] ...optimizers/trijoint.py.151: total_norm: 9.120267 [I 2022-10-10 13:48:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 260/2304 [I 2022-10-10 13:48:38] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:02 [I 2022-10-10 13:48:38] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00309 [I 2022-10-10 13:48:38] ...trap/engines/engine.py.230: loss: 0.19401 [I 2022-10-10 13:48:38] ...optimizers/trijoint.py.151: total_norm: 27.881817 [I 2022-10-10 13:48:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 270/2304 [I 2022-10-10 13:48:47] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:28:43 [I 2022-10-10 13:48:47] ...trap/engines/engine.py.229: process: 0.84923 | load: 0.00295 [I 2022-10-10 13:48:47] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-10 13:48:47] ...optimizers/trijoint.py.151: total_norm: 9.805900 [I 2022-10-10 13:48:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 280/2304 [I 2022-10-10 13:48:55] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:56 [I 2022-10-10 13:48:55] ...trap/engines/engine.py.229: process: 0.86698 | load: 0.00290 [I 2022-10-10 13:48:55] ...trap/engines/engine.py.230: loss: 0.16098 [I 2022-10-10 13:48:55] ...optimizers/trijoint.py.151: total_norm: 13.480024 [I 2022-10-10 13:49:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 290/2304 [I 2022-10-10 13:49:04] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:42 [I 2022-10-10 13:49:04] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00292 [I 2022-10-10 13:49:04] ...trap/engines/engine.py.230: loss: 0.15208 [I 2022-10-10 13:49:04] ...optimizers/trijoint.py.151: total_norm: 14.562844 [I 2022-10-10 13:49:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 300/2304 [I 2022-10-10 13:49:13] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:31 [I 2022-10-10 13:49:13] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00293 [I 2022-10-10 13:49:13] ...trap/engines/engine.py.230: loss: 0.13696 [I 2022-10-10 13:49:13] ...optimizers/trijoint.py.151: total_norm: 16.643808 [I 2022-10-10 13:49:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 310/2304 [I 2022-10-10 13:49:21] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:28 [I 2022-10-10 13:49:21] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00297 [I 2022-10-10 13:49:21] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-10 13:49:21] ...optimizers/trijoint.py.151: total_norm: 15.300025 [I 2022-10-10 13:49:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 320/2304 [I 2022-10-10 13:49:30] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:00 [I 2022-10-10 13:49:30] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00296 [I 2022-10-10 13:49:30] ...trap/engines/engine.py.230: loss: 0.13330 [I 2022-10-10 13:49:30] ...optimizers/trijoint.py.151: total_norm: 15.036716 [I 2022-10-10 13:49:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 330/2304 [I 2022-10-10 13:49:38] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:27:56 [I 2022-10-10 13:49:38] ...trap/engines/engine.py.229: process: 0.83738 | load: 0.00322 [I 2022-10-10 13:49:38] ...trap/engines/engine.py.230: loss: 0.21668 [I 2022-10-10 13:49:38] ...optimizers/trijoint.py.151: total_norm: 21.093561 [I 2022-10-10 13:49:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 340/2304 [I 2022-10-10 13:49:47] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:27:53 [I 2022-10-10 13:49:47] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00299 [I 2022-10-10 13:49:47] ...trap/engines/engine.py.230: loss: 0.15122 [I 2022-10-10 13:49:47] ...optimizers/trijoint.py.151: total_norm: 9.430688 [I 2022-10-10 13:49:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 350/2304 [I 2022-10-10 13:49:55] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:36 [I 2022-10-10 13:49:55] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00303 [I 2022-10-10 13:49:55] ...trap/engines/engine.py.230: loss: 0.14497 [I 2022-10-10 13:49:55] ...optimizers/trijoint.py.151: total_norm: 16.948933 [I 2022-10-10 13:50:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 360/2304 [I 2022-10-10 13:50:04] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:27:39 [I 2022-10-10 13:50:04] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00293 [I 2022-10-10 13:50:04] ...trap/engines/engine.py.230: loss: 0.19129 [I 2022-10-10 13:50:04] ...optimizers/trijoint.py.151: total_norm: 30.223299 [I 2022-10-10 13:50:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 370/2304 [I 2022-10-10 13:50:12] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:29 [I 2022-10-10 13:50:12] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00295 [I 2022-10-10 13:50:12] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-10 13:50:12] ...optimizers/trijoint.py.151: total_norm: 10.340892 [I 2022-10-10 13:50:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 380/2304 [I 2022-10-10 13:50:21] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:32 [I 2022-10-10 13:50:21] ...trap/engines/engine.py.229: process: 0.86810 | load: 0.00306 [I 2022-10-10 13:50:21] ...trap/engines/engine.py.230: loss: 0.09122 [I 2022-10-10 13:50:21] ...optimizers/trijoint.py.151: total_norm: 9.044140 [I 2022-10-10 13:50:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 390/2304 [I 2022-10-10 13:50:30] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:17 [I 2022-10-10 13:50:30] ...trap/engines/engine.py.229: process: 0.86287 | load: 0.00302 [I 2022-10-10 13:50:30] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-10 13:50:30] ...optimizers/trijoint.py.151: total_norm: 11.638629 [I 2022-10-10 13:50:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 400/2304 [I 2022-10-10 13:50:38] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:03 [I 2022-10-10 13:50:38] ...trap/engines/engine.py.229: process: 0.85229 | load: 0.00295 [I 2022-10-10 13:50:38] ...trap/engines/engine.py.230: loss: 0.11206 [I 2022-10-10 13:50:38] ...optimizers/trijoint.py.151: total_norm: 18.472931 [I 2022-10-10 13:50:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 410/2304 [I 2022-10-10 13:50:47] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:57 [I 2022-10-10 13:50:47] ...trap/engines/engine.py.229: process: 0.85922 | load: 0.00297 [I 2022-10-10 13:50:47] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-10 13:50:47] ...optimizers/trijoint.py.151: total_norm: 16.858326 [I 2022-10-10 13:50:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 420/2304 [I 2022-10-10 13:50:55] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:45 [I 2022-10-10 13:50:55] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00292 [I 2022-10-10 13:50:55] ...trap/engines/engine.py.230: loss: 0.13462 [I 2022-10-10 13:50:55] ...optimizers/trijoint.py.151: total_norm: 32.222435 [I 2022-10-10 13:51:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 430/2304 [I 2022-10-10 13:51:04] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:33 [I 2022-10-10 13:51:04] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00294 [I 2022-10-10 13:51:04] ...trap/engines/engine.py.230: loss: 0.09258 [I 2022-10-10 13:51:04] ...optimizers/trijoint.py.151: total_norm: 9.136481 [I 2022-10-10 13:51:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 440/2304 [I 2022-10-10 13:51:12] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:32 [I 2022-10-10 13:51:12] ...trap/engines/engine.py.229: process: 0.86423 | load: 0.00292 [I 2022-10-10 13:51:12] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-10 13:51:12] ...optimizers/trijoint.py.151: total_norm: 20.836811 [I 2022-10-10 13:51:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 450/2304 [I 2022-10-10 13:51:21] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:14 [I 2022-10-10 13:51:21] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00296 [I 2022-10-10 13:51:21] ...trap/engines/engine.py.230: loss: 0.11284 [I 2022-10-10 13:51:21] ...optimizers/trijoint.py.151: total_norm: 12.248033 [I 2022-10-10 13:51:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 460/2304 [I 2022-10-10 13:51:30] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:03 [I 2022-10-10 13:51:30] ...trap/engines/engine.py.229: process: 0.84263 | load: 0.00294 [I 2022-10-10 13:51:30] ...trap/engines/engine.py.230: loss: 0.15260 [I 2022-10-10 13:51:30] ...optimizers/trijoint.py.151: total_norm: 26.683842 [I 2022-10-10 13:51:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 470/2304 [I 2022-10-10 13:51:38] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:05 [I 2022-10-10 13:51:38] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00299 [I 2022-10-10 13:51:38] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-10 13:51:38] ...optimizers/trijoint.py.151: total_norm: 11.482906 [I 2022-10-10 13:51:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 480/2304 [I 2022-10-10 13:51:47] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:56 [I 2022-10-10 13:51:47] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00291 [I 2022-10-10 13:51:47] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-10 13:51:47] ...optimizers/trijoint.py.151: total_norm: 13.975355 [I 2022-10-10 13:51:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 490/2304 [I 2022-10-10 13:51:55] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:52 [I 2022-10-10 13:51:55] ...trap/engines/engine.py.229: process: 0.86289 | load: 0.00296 [I 2022-10-10 13:51:55] ...trap/engines/engine.py.230: loss: 0.15660 [I 2022-10-10 13:51:55] ...optimizers/trijoint.py.151: total_norm: 13.358474 [I 2022-10-10 13:52:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 500/2304 [I 2022-10-10 13:52:04] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:38 [I 2022-10-10 13:52:04] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00298 [I 2022-10-10 13:52:04] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-10 13:52:04] ...optimizers/trijoint.py.151: total_norm: 20.846148 [I 2022-10-10 13:52:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 510/2304 [I 2022-10-10 13:52:12] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:26 [I 2022-10-10 13:52:12] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00291 [I 2022-10-10 13:52:12] ...trap/engines/engine.py.230: loss: 0.13749 [I 2022-10-10 13:52:12] ...optimizers/trijoint.py.151: total_norm: 22.487547 [I 2022-10-10 13:52:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 520/2304 [I 2022-10-10 13:52:21] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:26 [I 2022-10-10 13:52:21] ...trap/engines/engine.py.229: process: 0.85620 | load: 0.00293 [I 2022-10-10 13:52:21] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-10 13:52:21] ...optimizers/trijoint.py.151: total_norm: 33.332825 [I 2022-10-10 13:52:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 530/2304 [I 2022-10-10 13:52:29] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:35 [I 2022-10-10 13:52:29] ...trap/engines/engine.py.229: process: 0.88333 | load: 0.00324 [I 2022-10-10 13:52:29] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-10 13:52:29] ...optimizers/trijoint.py.151: total_norm: 9.621521 [I 2022-10-10 13:52:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 540/2304 [I 2022-10-10 13:52:38] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:25:33 [I 2022-10-10 13:52:38] ...trap/engines/engine.py.229: process: 0.86600 | load: 0.00318 [I 2022-10-10 13:52:38] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-10 13:52:38] ...optimizers/trijoint.py.151: total_norm: 10.626557 [I 2022-10-10 13:52:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 550/2304 [I 2022-10-10 13:52:47] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:25:23 [I 2022-10-10 13:52:47] ...trap/engines/engine.py.229: process: 0.86981 | load: 0.00308 [I 2022-10-10 13:52:47] ...trap/engines/engine.py.230: loss: 0.17607 [I 2022-10-10 13:52:47] ...optimizers/trijoint.py.151: total_norm: 40.314903 [I 2022-10-10 13:52:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 560/2304 [I 2022-10-10 13:52:56] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:03 [I 2022-10-10 13:52:56] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00305 [I 2022-10-10 13:52:56] ...trap/engines/engine.py.230: loss: 0.13178 [I 2022-10-10 13:52:56] ...optimizers/trijoint.py.151: total_norm: 37.027622 [I 2022-10-10 13:53:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 570/2304 [I 2022-10-10 13:53:04] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:56 [I 2022-10-10 13:53:04] ...trap/engines/engine.py.229: process: 0.86299 | load: 0.00316 [I 2022-10-10 13:53:04] ...trap/engines/engine.py.230: loss: 0.14730 [I 2022-10-10 13:53:04] ...optimizers/trijoint.py.151: total_norm: 13.361426 [I 2022-10-10 13:53:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 580/2304 [I 2022-10-10 13:53:13] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:44 [I 2022-10-10 13:53:13] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00308 [I 2022-10-10 13:53:13] ...trap/engines/engine.py.230: loss: 0.20865 [I 2022-10-10 13:53:13] ...optimizers/trijoint.py.151: total_norm: 12.009117 [I 2022-10-10 13:53:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 590/2304 [I 2022-10-10 13:53:22] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:45 [I 2022-10-10 13:53:22] ...trap/engines/engine.py.229: process: 0.86483 | load: 0.00308 [I 2022-10-10 13:53:22] ...trap/engines/engine.py.230: loss: 0.11376 [I 2022-10-10 13:53:22] ...optimizers/trijoint.py.151: total_norm: 19.106735 [I 2022-10-10 13:53:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 600/2304 [I 2022-10-10 13:53:30] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:24:46 [I 2022-10-10 13:53:30] ...trap/engines/engine.py.229: process: 0.87742 | load: 0.00313 [I 2022-10-10 13:53:30] ...trap/engines/engine.py.230: loss: 0.14832 [I 2022-10-10 13:53:30] ...optimizers/trijoint.py.151: total_norm: 19.094509 [I 2022-10-10 13:53:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 610/2304 [I 2022-10-10 13:53:39] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:29 [I 2022-10-10 13:53:39] ...trap/engines/engine.py.229: process: 0.86894 | load: 0.00307 [I 2022-10-10 13:53:39] ...trap/engines/engine.py.230: loss: 0.10102 [I 2022-10-10 13:53:39] ...optimizers/trijoint.py.151: total_norm: 8.141644 [I 2022-10-10 13:53:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 620/2304 [I 2022-10-10 13:53:48] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:24:27 [I 2022-10-10 13:53:48] ...trap/engines/engine.py.229: process: 0.88133 | load: 0.00312 [I 2022-10-10 13:53:48] ...trap/engines/engine.py.230: loss: 0.08963 [I 2022-10-10 13:53:48] ...optimizers/trijoint.py.151: total_norm: 10.537722 [I 2022-10-10 13:53:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 630/2304 [I 2022-10-10 13:53:56] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:24:12 [I 2022-10-10 13:53:56] ...trap/engines/engine.py.229: process: 0.86428 | load: 0.00322 [I 2022-10-10 13:53:56] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-10 13:53:56] ...optimizers/trijoint.py.151: total_norm: 17.014002 [I 2022-10-10 13:54:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 640/2304 [I 2022-10-10 13:54:05] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:54 [I 2022-10-10 13:54:05] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00313 [I 2022-10-10 13:54:05] ...trap/engines/engine.py.230: loss: 0.14386 [I 2022-10-10 13:54:05] ...optimizers/trijoint.py.151: total_norm: 18.154203 [I 2022-10-10 13:54:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 650/2304 [I 2022-10-10 13:54:14] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:24:01 [I 2022-10-10 13:54:14] ...trap/engines/engine.py.229: process: 0.87467 | load: 0.00311 [I 2022-10-10 13:54:14] ...trap/engines/engine.py.230: loss: 0.12473 [I 2022-10-10 13:54:14] ...optimizers/trijoint.py.151: total_norm: 8.669466 [I 2022-10-10 13:54:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 660/2304 [I 2022-10-10 13:54:23] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:50 [I 2022-10-10 13:54:23] ...trap/engines/engine.py.229: process: 0.87809 | load: 0.00309 [I 2022-10-10 13:54:23] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-10 13:54:23] ...optimizers/trijoint.py.151: total_norm: 13.409334 [I 2022-10-10 13:54:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 670/2304 [I 2022-10-10 13:54:31] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:40 [I 2022-10-10 13:54:31] ...trap/engines/engine.py.229: process: 0.86167 | load: 0.00342 [I 2022-10-10 13:54:31] ...trap/engines/engine.py.230: loss: 0.12688 [I 2022-10-10 13:54:31] ...optimizers/trijoint.py.151: total_norm: 8.432630 [I 2022-10-10 13:54:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 680/2304 [I 2022-10-10 13:54:40] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:31 [I 2022-10-10 13:54:40] ...trap/engines/engine.py.229: process: 0.87069 | load: 0.00308 [I 2022-10-10 13:54:40] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-10 13:54:40] ...optimizers/trijoint.py.151: total_norm: 8.308247 [I 2022-10-10 13:54:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 690/2304 [I 2022-10-10 13:54:49] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:15 [I 2022-10-10 13:54:49] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00318 [I 2022-10-10 13:54:49] ...trap/engines/engine.py.230: loss: 0.12516 [I 2022-10-10 13:54:49] ...optimizers/trijoint.py.151: total_norm: 13.479323 [I 2022-10-10 13:54:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 700/2304 [I 2022-10-10 13:54:57] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:23:03 [I 2022-10-10 13:54:57] ...trap/engines/engine.py.229: process: 0.86234 | load: 0.00316 [I 2022-10-10 13:54:57] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-10 13:54:57] ...optimizers/trijoint.py.151: total_norm: 8.974919 [I 2022-10-10 13:55:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 710/2304 [I 2022-10-10 13:55:06] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:23:04 [I 2022-10-10 13:55:06] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00311 [I 2022-10-10 13:55:06] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-10 13:55:06] ...optimizers/trijoint.py.151: total_norm: 15.380636 [I 2022-10-10 13:55:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 720/2304 [I 2022-10-10 13:55:15] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:23:00 [I 2022-10-10 13:55:15] ...trap/engines/engine.py.229: process: 0.87611 | load: 0.00315 [I 2022-10-10 13:55:15] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-10 13:55:15] ...optimizers/trijoint.py.151: total_norm: 19.923885 [I 2022-10-10 13:55:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 730/2304 [I 2022-10-10 13:55:23] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:34 [I 2022-10-10 13:55:23] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00309 [I 2022-10-10 13:55:23] ...trap/engines/engine.py.230: loss: 0.22225 [I 2022-10-10 13:55:23] ...optimizers/trijoint.py.151: total_norm: 19.056778 [I 2022-10-10 13:55:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 740/2304 [I 2022-10-10 13:55:32] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:30 [I 2022-10-10 13:55:32] ...trap/engines/engine.py.229: process: 0.87447 | load: 0.00310 [I 2022-10-10 13:55:32] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-10 13:55:32] ...optimizers/trijoint.py.151: total_norm: 15.131252 [I 2022-10-10 13:55:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 750/2304 [I 2022-10-10 13:55:41] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:27 [I 2022-10-10 13:55:41] ...trap/engines/engine.py.229: process: 0.88797 | load: 0.00321 [I 2022-10-10 13:55:41] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-10 13:55:41] ...optimizers/trijoint.py.151: total_norm: 12.891419 [I 2022-10-10 13:55:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 760/2304 [I 2022-10-10 13:55:49] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:21 [I 2022-10-10 13:55:49] ...trap/engines/engine.py.229: process: 0.88304 | load: 0.00310 [I 2022-10-10 13:55:49] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-10 13:55:49] ...optimizers/trijoint.py.151: total_norm: 10.707722 [I 2022-10-10 13:55:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 770/2304 [I 2022-10-10 13:55:58] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:05 [I 2022-10-10 13:55:58] ...trap/engines/engine.py.229: process: 0.85665 | load: 0.00317 [I 2022-10-10 13:55:58] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-10 13:55:58] ...optimizers/trijoint.py.151: total_norm: 18.089132 [I 2022-10-10 13:56:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 780/2304 [I 2022-10-10 13:56:07] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:22:06 [I 2022-10-10 13:56:07] ...trap/engines/engine.py.229: process: 0.87509 | load: 0.00316 [I 2022-10-10 13:56:07] ...trap/engines/engine.py.230: loss: 0.10364 [I 2022-10-10 13:56:07] ...optimizers/trijoint.py.151: total_norm: 8.784722 [I 2022-10-10 13:56:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 790/2304 [I 2022-10-10 13:56:15] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:50 [I 2022-10-10 13:56:15] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00306 [I 2022-10-10 13:56:15] ...trap/engines/engine.py.230: loss: 0.17899 [I 2022-10-10 13:56:15] ...optimizers/trijoint.py.151: total_norm: 18.388723 [I 2022-10-10 13:56:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 800/2304 [I 2022-10-10 13:56:24] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:31 [I 2022-10-10 13:56:24] ...trap/engines/engine.py.229: process: 0.85149 | load: 0.00297 [I 2022-10-10 13:56:24] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-10 13:56:24] ...optimizers/trijoint.py.151: total_norm: 14.565192 [I 2022-10-10 13:56:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 810/2304 [I 2022-10-10 13:56:33] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:23 [I 2022-10-10 13:56:33] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00298 [I 2022-10-10 13:56:33] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-10 13:56:33] ...optimizers/trijoint.py.151: total_norm: 15.188046 [I 2022-10-10 13:56:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 820/2304 [I 2022-10-10 13:56:41] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:07 [I 2022-10-10 13:56:41] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00290 [I 2022-10-10 13:56:41] ...trap/engines/engine.py.230: loss: 0.10792 [I 2022-10-10 13:56:41] ...optimizers/trijoint.py.151: total_norm: 16.643793 [I 2022-10-10 13:56:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 830/2304 [I 2022-10-10 13:56:50] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:02 [I 2022-10-10 13:56:50] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00289 [I 2022-10-10 13:56:50] ...trap/engines/engine.py.230: loss: 0.10362 [I 2022-10-10 13:56:50] ...optimizers/trijoint.py.151: total_norm: 26.813046 [I 2022-10-10 13:56:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 840/2304 [I 2022-10-10 13:56:58] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:48 [I 2022-10-10 13:56:58] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00293 [I 2022-10-10 13:56:58] ...trap/engines/engine.py.230: loss: 0.18712 [I 2022-10-10 13:56:58] ...optimizers/trijoint.py.151: total_norm: 15.542098 [I 2022-10-10 13:57:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 850/2304 [I 2022-10-10 13:57:07] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:38 [I 2022-10-10 13:57:07] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00300 [I 2022-10-10 13:57:07] ...trap/engines/engine.py.230: loss: 0.14933 [I 2022-10-10 13:57:07] ...optimizers/trijoint.py.151: total_norm: 13.102200 [I 2022-10-10 13:57:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 860/2304 [I 2022-10-10 13:57:15] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:33 [I 2022-10-10 13:57:15] ...trap/engines/engine.py.229: process: 0.85372 | load: 0.00302 [I 2022-10-10 13:57:15] ...trap/engines/engine.py.230: loss: 0.13727 [I 2022-10-10 13:57:15] ...optimizers/trijoint.py.151: total_norm: 41.940289 [I 2022-10-10 13:57:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 870/2304 [I 2022-10-10 13:57:24] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:33 [I 2022-10-10 13:57:24] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00296 [I 2022-10-10 13:57:24] ...trap/engines/engine.py.230: loss: 0.09050 [I 2022-10-10 13:57:24] ...optimizers/trijoint.py.151: total_norm: 7.416917 [I 2022-10-10 13:57:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 880/2304 [I 2022-10-10 13:57:33] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:24 [I 2022-10-10 13:57:33] ...trap/engines/engine.py.229: process: 0.86187 | load: 0.00295 [I 2022-10-10 13:57:33] ...trap/engines/engine.py.230: loss: 0.10970 [I 2022-10-10 13:57:33] ...optimizers/trijoint.py.151: total_norm: 17.495510 [I 2022-10-10 13:57:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 890/2304 [I 2022-10-10 13:57:41] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:20:11 [I 2022-10-10 13:57:41] ...trap/engines/engine.py.229: process: 0.84288 | load: 0.00297 [I 2022-10-10 13:57:41] ...trap/engines/engine.py.230: loss: 0.13163 [I 2022-10-10 13:57:41] ...optimizers/trijoint.py.151: total_norm: 15.600011 [I 2022-10-10 13:57:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 900/2304 [I 2022-10-10 13:57:50] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:59 [I 2022-10-10 13:57:50] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00299 [I 2022-10-10 13:57:50] ...trap/engines/engine.py.230: loss: 0.13478 [I 2022-10-10 13:57:50] ...optimizers/trijoint.py.151: total_norm: 18.629974 [I 2022-10-10 13:57:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 910/2304 [I 2022-10-10 13:57:58] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:49 [I 2022-10-10 13:57:58] ...trap/engines/engine.py.229: process: 0.82978 | load: 0.00294 [I 2022-10-10 13:57:58] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-10 13:57:58] ...optimizers/trijoint.py.151: total_norm: 13.093368 [I 2022-10-10 13:58:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 920/2304 [I 2022-10-10 13:58:07] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:46 [I 2022-10-10 13:58:07] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00295 [I 2022-10-10 13:58:07] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-10 13:58:07] ...optimizers/trijoint.py.151: total_norm: 11.405122 [I 2022-10-10 13:58:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 930/2304 [I 2022-10-10 13:58:16] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:28 [I 2022-10-10 13:58:16] ...trap/engines/engine.py.229: process: 0.86045 | load: 0.00309 [I 2022-10-10 13:58:16] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-10 13:58:16] ...optimizers/trijoint.py.151: total_norm: 10.033645 [I 2022-10-10 13:58:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 940/2304 [I 2022-10-10 13:58:24] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:29 [I 2022-10-10 13:58:24] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00299 [I 2022-10-10 13:58:24] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-10 13:58:24] ...optimizers/trijoint.py.151: total_norm: 23.043568 [I 2022-10-10 13:58:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 950/2304 [I 2022-10-10 13:58:33] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:16 [I 2022-10-10 13:58:33] ...trap/engines/engine.py.229: process: 0.85964 | load: 0.00296 [I 2022-10-10 13:58:33] ...trap/engines/engine.py.230: loss: 0.12261 [I 2022-10-10 13:58:33] ...optimizers/trijoint.py.151: total_norm: 10.161861 [I 2022-10-10 13:58:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 960/2304 [I 2022-10-10 13:58:41] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:08 [I 2022-10-10 13:58:41] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00299 [I 2022-10-10 13:58:41] ...trap/engines/engine.py.230: loss: 0.18617 [I 2022-10-10 13:58:41] ...optimizers/trijoint.py.151: total_norm: 28.165377 [I 2022-10-10 13:58:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 970/2304 [I 2022-10-10 13:58:50] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:19:10 [I 2022-10-10 13:58:50] ...trap/engines/engine.py.229: process: 0.87378 | load: 0.00294 [I 2022-10-10 13:58:50] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-10 13:58:50] ...optimizers/trijoint.py.151: total_norm: 15.357195 [I 2022-10-10 13:58:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 980/2304 [I 2022-10-10 13:58:58] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:51 [I 2022-10-10 13:58:58] ...trap/engines/engine.py.229: process: 0.86470 | load: 0.00293 [I 2022-10-10 13:58:58] ...trap/engines/engine.py.230: loss: 0.13294 [I 2022-10-10 13:58:58] ...optimizers/trijoint.py.151: total_norm: 8.485168 [I 2022-10-10 13:59:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 990/2304 [I 2022-10-10 13:59:07] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:42 [I 2022-10-10 13:59:07] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00311 [I 2022-10-10 13:59:07] ...trap/engines/engine.py.230: loss: 0.13538 [I 2022-10-10 13:59:07] ...optimizers/trijoint.py.151: total_norm: 10.274741 [I 2022-10-10 13:59:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 1000/2304 [I 2022-10-10 13:59:16] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:36 [I 2022-10-10 13:59:16] ...trap/engines/engine.py.229: process: 0.85184 | load: 0.00309 [I 2022-10-10 13:59:16] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-10 13:59:16] ...optimizers/trijoint.py.151: total_norm: 21.050638 [I 2022-10-10 13:59:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 1010/2304 [I 2022-10-10 13:59:24] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:33 [I 2022-10-10 13:59:24] ...trap/engines/engine.py.229: process: 0.86533 | load: 0.00325 [I 2022-10-10 13:59:24] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-10 13:59:24] ...optimizers/trijoint.py.151: total_norm: 14.051876 [I 2022-10-10 13:59:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 1020/2304 [I 2022-10-10 13:59:33] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:28 [I 2022-10-10 13:59:33] ...trap/engines/engine.py.229: process: 0.87616 | load: 0.00313 [I 2022-10-10 13:59:33] ...trap/engines/engine.py.230: loss: 0.16224 [I 2022-10-10 13:59:33] ...optimizers/trijoint.py.151: total_norm: 14.020921 [I 2022-10-10 13:59:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 1030/2304 [I 2022-10-10 13:59:42] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:18:25 [I 2022-10-10 13:59:42] ...trap/engines/engine.py.229: process: 0.88057 | load: 0.00330 [I 2022-10-10 13:59:42] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-10 13:59:42] ...optimizers/trijoint.py.151: total_norm: 16.535236 [I 2022-10-10 13:59:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 1040/2304 [I 2022-10-10 13:59:50] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:18:09 [I 2022-10-10 13:59:50] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00324 [I 2022-10-10 13:59:50] ...trap/engines/engine.py.230: loss: 0.15340 [I 2022-10-10 13:59:50] ...optimizers/trijoint.py.151: total_norm: 11.935811 [I 2022-10-10 14:00:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 1050/2304 [I 2022-10-10 14:00:00] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:19:20 [I 2022-10-10 14:00:00] ...trap/engines/engine.py.229: process: 0.86944 | load: 0.00302 [I 2022-10-10 14:00:00] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-10 14:00:00] ...optimizers/trijoint.py.151: total_norm: 12.087052 [I 2022-10-10 14:00:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 1060/2304 [I 2022-10-10 14:00:09] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:55 [I 2022-10-10 14:00:09] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00317 [I 2022-10-10 14:00:09] ...trap/engines/engine.py.230: loss: 0.08801 [I 2022-10-10 14:00:09] ...optimizers/trijoint.py.151: total_norm: 10.831313 [I 2022-10-10 14:00:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 1070/2304 [I 2022-10-10 14:00:18] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:49 [I 2022-10-10 14:00:18] ...trap/engines/engine.py.229: process: 0.87033 | load: 0.00356 [I 2022-10-10 14:00:18] ...trap/engines/engine.py.230: loss: 0.13431 [I 2022-10-10 14:00:18] ...optimizers/trijoint.py.151: total_norm: 14.661701 [I 2022-10-10 14:00:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 1080/2304 [I 2022-10-10 14:00:27] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:38 [I 2022-10-10 14:00:27] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00307 [I 2022-10-10 14:00:27] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-10 14:00:27] ...optimizers/trijoint.py.151: total_norm: 10.199279 [I 2022-10-10 14:00:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 1090/2304 [I 2022-10-10 14:00:35] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:23 [I 2022-10-10 14:00:35] ...trap/engines/engine.py.229: process: 0.84357 | load: 0.00354 [I 2022-10-10 14:00:35] ...trap/engines/engine.py.230: loss: 0.22185 [I 2022-10-10 14:00:35] ...optimizers/trijoint.py.151: total_norm: 32.020306 [I 2022-10-10 14:00:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 1100/2304 [I 2022-10-10 14:00:44] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:24 [I 2022-10-10 14:00:44] ...trap/engines/engine.py.229: process: 0.85948 | load: 0.00317 [I 2022-10-10 14:00:44] ...trap/engines/engine.py.230: loss: 0.10710 [I 2022-10-10 14:00:44] ...optimizers/trijoint.py.151: total_norm: 10.659621 [I 2022-10-10 14:00:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 1110/2304 [I 2022-10-10 14:00:53] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:18 [I 2022-10-10 14:00:53] ...trap/engines/engine.py.229: process: 0.86750 | load: 0.00305 [I 2022-10-10 14:00:53] ...trap/engines/engine.py.230: loss: 0.18071 [I 2022-10-10 14:00:53] ...optimizers/trijoint.py.151: total_norm: 22.192667 [I 2022-10-10 14:01:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 1120/2304 [I 2022-10-10 14:01:01] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:17:07 [I 2022-10-10 14:01:01] ...trap/engines/engine.py.229: process: 0.87724 | load: 0.00310 [I 2022-10-10 14:01:01] ...trap/engines/engine.py.230: loss: 0.14343 [I 2022-10-10 14:01:01] ...optimizers/trijoint.py.151: total_norm: 28.271769 [I 2022-10-10 14:01:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 1130/2304 [I 2022-10-10 14:01:10] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:17:02 [I 2022-10-10 14:01:10] ...trap/engines/engine.py.229: process: 0.87419 | load: 0.00311 [I 2022-10-10 14:01:10] ...trap/engines/engine.py.230: loss: 0.12793 [I 2022-10-10 14:01:10] ...optimizers/trijoint.py.151: total_norm: 12.055919 [I 2022-10-10 14:01:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 1140/2304 [I 2022-10-10 14:01:19] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:48 [I 2022-10-10 14:01:19] ...trap/engines/engine.py.229: process: 0.87183 | load: 0.00325 [I 2022-10-10 14:01:19] ...trap/engines/engine.py.230: loss: 0.17762 [I 2022-10-10 14:01:19] ...optimizers/trijoint.py.151: total_norm: 16.600286 [I 2022-10-10 14:01:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 1150/2304 [I 2022-10-10 14:01:27] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:35 [I 2022-10-10 14:01:27] ...trap/engines/engine.py.229: process: 0.85735 | load: 0.00311 [I 2022-10-10 14:01:27] ...trap/engines/engine.py.230: loss: 0.20926 [I 2022-10-10 14:01:27] ...optimizers/trijoint.py.151: total_norm: 20.174477 [I 2022-10-10 14:01:36] ...trap/engines/engine.py.224: train: epoch 74 | batch 1160/2304 [I 2022-10-10 14:01:36] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:16:31 [I 2022-10-10 14:01:36] ...trap/engines/engine.py.229: process: 0.86630 | load: 0.00313 [I 2022-10-10 14:01:36] ...trap/engines/engine.py.230: loss: 0.13100 [I 2022-10-10 14:01:36] ...optimizers/trijoint.py.151: total_norm: 26.714184 [I 2022-10-10 14:01:45] ...trap/engines/engine.py.224: train: epoch 74 | batch 1170/2304 [I 2022-10-10 14:01:45] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:17 [I 2022-10-10 14:01:45] ...trap/engines/engine.py.229: process: 0.88052 | load: 0.00311 [I 2022-10-10 14:01:45] ...trap/engines/engine.py.230: loss: 0.14059 [I 2022-10-10 14:01:45] ...optimizers/trijoint.py.151: total_norm: 15.756166 [I 2022-10-10 14:01:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 1180/2304 [I 2022-10-10 14:01:53] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:16:11 [I 2022-10-10 14:01:53] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00324 [I 2022-10-10 14:01:53] ...trap/engines/engine.py.230: loss: 0.08325 [I 2022-10-10 14:01:53] ...optimizers/trijoint.py.151: total_norm: 9.662951 [I 2022-10-10 14:02:02] ...trap/engines/engine.py.224: train: epoch 74 | batch 1190/2304 [I 2022-10-10 14:02:02] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:59 [I 2022-10-10 14:02:02] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00324 [I 2022-10-10 14:02:02] ...trap/engines/engine.py.230: loss: 0.20082 [I 2022-10-10 14:02:02] ...optimizers/trijoint.py.151: total_norm: 22.376518 [I 2022-10-10 14:02:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1200/2304 [I 2022-10-10 14:02:11] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:56 [I 2022-10-10 14:02:11] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00321 [I 2022-10-10 14:02:11] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-10 14:02:11] ...optimizers/trijoint.py.151: total_norm: 11.276569 [I 2022-10-10 14:02:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 1210/2304 [I 2022-10-10 14:02:19] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:45 [I 2022-10-10 14:02:19] ...trap/engines/engine.py.229: process: 0.85481 | load: 0.00319 [I 2022-10-10 14:02:19] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-10 14:02:19] ...optimizers/trijoint.py.151: total_norm: 14.022999 [I 2022-10-10 14:02:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 1220/2304 [I 2022-10-10 14:02:28] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:15:42 [I 2022-10-10 14:02:28] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00326 [I 2022-10-10 14:02:28] ...trap/engines/engine.py.230: loss: 0.25484 [I 2022-10-10 14:02:28] ...optimizers/trijoint.py.151: total_norm: 20.457186 [I 2022-10-10 14:02:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1230/2304 [I 2022-10-10 14:02:37] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:31 [I 2022-10-10 14:02:37] ...trap/engines/engine.py.229: process: 0.86343 | load: 0.00327 [I 2022-10-10 14:02:37] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-10 14:02:37] ...optimizers/trijoint.py.151: total_norm: 10.455360 [I 2022-10-10 14:02:45] ...trap/engines/engine.py.224: train: epoch 74 | batch 1240/2304 [I 2022-10-10 14:02:45] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:22 [I 2022-10-10 14:02:45] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00321 [I 2022-10-10 14:02:45] ...trap/engines/engine.py.230: loss: 0.10542 [I 2022-10-10 14:02:45] ...optimizers/trijoint.py.151: total_norm: 18.588791 [I 2022-10-10 14:02:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 1250/2304 [I 2022-10-10 14:02:54] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:09 [I 2022-10-10 14:02:54] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00325 [I 2022-10-10 14:02:54] ...trap/engines/engine.py.230: loss: 0.09533 [I 2022-10-10 14:02:54] ...optimizers/trijoint.py.151: total_norm: 10.661259 [I 2022-10-10 14:03:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1260/2304 [I 2022-10-10 14:03:03] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:59 [I 2022-10-10 14:03:03] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00298 [I 2022-10-10 14:03:03] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-10 14:03:03] ...optimizers/trijoint.py.151: total_norm: 9.530861 [I 2022-10-10 14:03:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1270/2304 [I 2022-10-10 14:03:11] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:43 [I 2022-10-10 14:03:11] ...trap/engines/engine.py.229: process: 0.84515 | load: 0.00296 [I 2022-10-10 14:03:11] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-10 14:03:11] ...optimizers/trijoint.py.151: total_norm: 26.525053 [I 2022-10-10 14:03:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 1280/2304 [I 2022-10-10 14:03:20] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:37 [I 2022-10-10 14:03:20] ...trap/engines/engine.py.229: process: 0.84528 | load: 0.00299 [I 2022-10-10 14:03:20] ...trap/engines/engine.py.230: loss: 0.09927 [I 2022-10-10 14:03:20] ...optimizers/trijoint.py.151: total_norm: 12.331609 [I 2022-10-10 14:03:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 1290/2304 [I 2022-10-10 14:03:28] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:20 [I 2022-10-10 14:03:28] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00299 [I 2022-10-10 14:03:28] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-10 14:03:28] ...optimizers/trijoint.py.151: total_norm: 25.229650 [I 2022-10-10 14:03:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1300/2304 [I 2022-10-10 14:03:37] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:21 [I 2022-10-10 14:03:37] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00297 [I 2022-10-10 14:03:37] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-10 14:03:37] ...optimizers/trijoint.py.151: total_norm: 18.446285 [I 2022-10-10 14:03:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1310/2304 [I 2022-10-10 14:03:46] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:08 [I 2022-10-10 14:03:46] ...trap/engines/engine.py.229: process: 0.87152 | load: 0.00293 [I 2022-10-10 14:03:46] ...trap/engines/engine.py.230: loss: 0.16865 [I 2022-10-10 14:03:46] ...optimizers/trijoint.py.151: total_norm: 40.213982 [I 2022-10-10 14:03:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 1320/2304 [I 2022-10-10 14:03:54] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:57 [I 2022-10-10 14:03:54] ...trap/engines/engine.py.229: process: 0.86605 | load: 0.00296 [I 2022-10-10 14:03:54] ...trap/engines/engine.py.230: loss: 0.11665 [I 2022-10-10 14:03:54] ...optimizers/trijoint.py.151: total_norm: 10.723505 [I 2022-10-10 14:04:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1330/2304 [I 2022-10-10 14:04:03] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:50 [I 2022-10-10 14:04:03] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00299 [I 2022-10-10 14:04:03] ...trap/engines/engine.py.230: loss: 0.20358 [I 2022-10-10 14:04:03] ...optimizers/trijoint.py.151: total_norm: 16.747547 [I 2022-10-10 14:04:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1340/2304 [I 2022-10-10 14:04:11] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:41 [I 2022-10-10 14:04:11] ...trap/engines/engine.py.229: process: 0.85980 | load: 0.00295 [I 2022-10-10 14:04:11] ...trap/engines/engine.py.230: loss: 0.15134 [I 2022-10-10 14:04:11] ...optimizers/trijoint.py.151: total_norm: 15.627459 [I 2022-10-10 14:04:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 1350/2304 [I 2022-10-10 14:04:20] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:34 [I 2022-10-10 14:04:20] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00297 [I 2022-10-10 14:04:20] ...trap/engines/engine.py.230: loss: 0.11979 [I 2022-10-10 14:04:20] ...optimizers/trijoint.py.151: total_norm: 21.404627 [I 2022-10-10 14:04:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 1360/2304 [I 2022-10-10 14:04:28] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:25 [I 2022-10-10 14:04:28] ...trap/engines/engine.py.229: process: 0.85472 | load: 0.00302 [I 2022-10-10 14:04:28] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-10 14:04:29] ...optimizers/trijoint.py.151: total_norm: 26.711887 [I 2022-10-10 14:04:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1370/2304 [I 2022-10-10 14:04:37] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:18 [I 2022-10-10 14:04:37] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00306 [I 2022-10-10 14:04:37] ...trap/engines/engine.py.230: loss: 0.15333 [I 2022-10-10 14:04:37] ...optimizers/trijoint.py.151: total_norm: 11.788326 [I 2022-10-10 14:04:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1380/2304 [I 2022-10-10 14:04:46] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:13:11 [I 2022-10-10 14:04:46] ...trap/engines/engine.py.229: process: 0.86192 | load: 0.00297 [I 2022-10-10 14:04:46] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-10 14:04:46] ...optimizers/trijoint.py.151: total_norm: 17.985956 [I 2022-10-10 14:04:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 1390/2304 [I 2022-10-10 14:04:54] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:02 [I 2022-10-10 14:04:54] ...trap/engines/engine.py.229: process: 0.85966 | load: 0.00297 [I 2022-10-10 14:04:54] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-10 14:04:54] ...optimizers/trijoint.py.151: total_norm: 35.243649 [I 2022-10-10 14:05:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1400/2304 [I 2022-10-10 14:05:03] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:52 [I 2022-10-10 14:05:03] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00302 [I 2022-10-10 14:05:03] ...trap/engines/engine.py.230: loss: 0.10754 [I 2022-10-10 14:05:03] ...optimizers/trijoint.py.151: total_norm: 14.656361 [I 2022-10-10 14:05:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1410/2304 [I 2022-10-10 14:05:11] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:40 [I 2022-10-10 14:05:11] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00299 [I 2022-10-10 14:05:11] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-10 14:05:11] ...optimizers/trijoint.py.151: total_norm: 15.738333 [I 2022-10-10 14:05:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 1420/2304 [I 2022-10-10 14:05:20] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:39 [I 2022-10-10 14:05:20] ...trap/engines/engine.py.229: process: 0.86196 | load: 0.00313 [I 2022-10-10 14:05:20] ...trap/engines/engine.py.230: loss: 0.18948 [I 2022-10-10 14:05:20] ...optimizers/trijoint.py.151: total_norm: 20.738050 [I 2022-10-10 14:05:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 1430/2304 [I 2022-10-10 14:05:29] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:37 [I 2022-10-10 14:05:29] ...trap/engines/engine.py.229: process: 0.86464 | load: 0.00308 [I 2022-10-10 14:05:29] ...trap/engines/engine.py.230: loss: 0.11974 [I 2022-10-10 14:05:29] ...optimizers/trijoint.py.151: total_norm: 9.023993 [I 2022-10-10 14:05:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1440/2304 [I 2022-10-10 14:05:37] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:23 [I 2022-10-10 14:05:37] ...trap/engines/engine.py.229: process: 0.85786 | load: 0.00308 [I 2022-10-10 14:05:37] ...trap/engines/engine.py.230: loss: 0.10170 [I 2022-10-10 14:05:37] ...optimizers/trijoint.py.151: total_norm: 9.595394 [I 2022-10-10 14:05:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1450/2304 [I 2022-10-10 14:05:46] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:12:17 [I 2022-10-10 14:05:46] ...trap/engines/engine.py.229: process: 0.86196 | load: 0.00307 [I 2022-10-10 14:05:46] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-10 14:05:46] ...optimizers/trijoint.py.151: total_norm: 17.608494 [I 2022-10-10 14:05:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 1460/2304 [I 2022-10-10 14:05:55] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:12:11 [I 2022-10-10 14:05:55] ...trap/engines/engine.py.229: process: 0.86966 | load: 0.00311 [I 2022-10-10 14:05:55] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-10 14:05:55] ...optimizers/trijoint.py.151: total_norm: 13.355384 [I 2022-10-10 14:06:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1470/2304 [I 2022-10-10 14:06:03] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:58 [I 2022-10-10 14:06:03] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00305 [I 2022-10-10 14:06:03] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-10 14:06:03] ...optimizers/trijoint.py.151: total_norm: 16.066059 [I 2022-10-10 14:06:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 1480/2304 [I 2022-10-10 14:06:12] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:53 [I 2022-10-10 14:06:12] ...trap/engines/engine.py.229: process: 0.86498 | load: 0.00304 [I 2022-10-10 14:06:12] ...trap/engines/engine.py.230: loss: 0.13573 [I 2022-10-10 14:06:12] ...optimizers/trijoint.py.151: total_norm: 19.669724 [I 2022-10-10 14:06:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 1490/2304 [I 2022-10-10 14:06:21] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:44 [I 2022-10-10 14:06:21] ...trap/engines/engine.py.229: process: 0.86839 | load: 0.00314 [I 2022-10-10 14:06:21] ...trap/engines/engine.py.230: loss: 0.14131 [I 2022-10-10 14:06:21] ...optimizers/trijoint.py.151: total_norm: 17.556807 [I 2022-10-10 14:06:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 1500/2304 [I 2022-10-10 14:06:29] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:39 [I 2022-10-10 14:06:29] ...trap/engines/engine.py.229: process: 0.87356 | load: 0.00315 [I 2022-10-10 14:06:29] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-10 14:06:29] ...optimizers/trijoint.py.151: total_norm: 19.204521 [I 2022-10-10 14:06:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 1510/2304 [I 2022-10-10 14:06:38] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:31 [I 2022-10-10 14:06:38] ...trap/engines/engine.py.229: process: 0.87857 | load: 0.00311 [I 2022-10-10 14:06:38] ...trap/engines/engine.py.230: loss: 0.08559 [I 2022-10-10 14:06:38] ...optimizers/trijoint.py.151: total_norm: 13.891341 [I 2022-10-10 14:06:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 1520/2304 [I 2022-10-10 14:06:47] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:19 [I 2022-10-10 14:06:47] ...trap/engines/engine.py.229: process: 0.86854 | load: 0.00311 [I 2022-10-10 14:06:47] ...trap/engines/engine.py.230: loss: 0.20727 [I 2022-10-10 14:06:47] ...optimizers/trijoint.py.151: total_norm: 23.911196 [I 2022-10-10 14:06:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 1530/2304 [I 2022-10-10 14:06:55] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:11:10 [I 2022-10-10 14:06:55] ...trap/engines/engine.py.229: process: 0.86690 | load: 0.00339 [I 2022-10-10 14:06:55] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-10 14:06:55] ...optimizers/trijoint.py.151: total_norm: 9.584981 [I 2022-10-10 14:07:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 1540/2304 [I 2022-10-10 14:07:04] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:11:02 [I 2022-10-10 14:07:04] ...trap/engines/engine.py.229: process: 0.86612 | load: 0.00314 [I 2022-10-10 14:07:04] ...trap/engines/engine.py.230: loss: 0.13975 [I 2022-10-10 14:07:04] ...optimizers/trijoint.py.151: total_norm: 14.082666 [I 2022-10-10 14:07:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 1550/2304 [I 2022-10-10 14:07:13] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:50 [I 2022-10-10 14:07:13] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00306 [I 2022-10-10 14:07:13] ...trap/engines/engine.py.230: loss: 0.10730 [I 2022-10-10 14:07:13] ...optimizers/trijoint.py.151: total_norm: 11.882847 [I 2022-10-10 14:07:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 1560/2304 [I 2022-10-10 14:07:21] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:42 [I 2022-10-10 14:07:21] ...trap/engines/engine.py.229: process: 0.86973 | load: 0.00313 [I 2022-10-10 14:07:21] ...trap/engines/engine.py.230: loss: 0.11687 [I 2022-10-10 14:07:21] ...optimizers/trijoint.py.151: total_norm: 36.665546 [I 2022-10-10 14:07:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 1570/2304 [I 2022-10-10 14:07:30] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:38 [I 2022-10-10 14:07:30] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00308 [I 2022-10-10 14:07:30] ...trap/engines/engine.py.230: loss: 0.12325 [I 2022-10-10 14:07:30] ...optimizers/trijoint.py.151: total_norm: 13.414742 [I 2022-10-10 14:07:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 1580/2304 [I 2022-10-10 14:07:39] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:28 [I 2022-10-10 14:07:39] ...trap/engines/engine.py.229: process: 0.86742 | load: 0.00318 [I 2022-10-10 14:07:39] ...trap/engines/engine.py.230: loss: 0.08108 [I 2022-10-10 14:07:39] ...optimizers/trijoint.py.151: total_norm: 13.602720 [I 2022-10-10 14:07:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 1590/2304 [I 2022-10-10 14:07:48] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:18 [I 2022-10-10 14:07:48] ...trap/engines/engine.py.229: process: 0.86845 | load: 0.00309 [I 2022-10-10 14:07:48] ...trap/engines/engine.py.230: loss: 0.16416 [I 2022-10-10 14:07:48] ...optimizers/trijoint.py.151: total_norm: 19.568371 [I 2022-10-10 14:07:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 1600/2304 [I 2022-10-10 14:07:56] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:10:08 [I 2022-10-10 14:07:56] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00313 [I 2022-10-10 14:07:56] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-10 14:07:56] ...optimizers/trijoint.py.151: total_norm: 19.155298 [I 2022-10-10 14:08:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 1610/2304 [I 2022-10-10 14:08:05] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:10:00 [I 2022-10-10 14:08:05] ...trap/engines/engine.py.229: process: 0.86911 | load: 0.00312 [I 2022-10-10 14:08:05] ...trap/engines/engine.py.230: loss: 0.13327 [I 2022-10-10 14:08:05] ...optimizers/trijoint.py.151: total_norm: 12.060296 [I 2022-10-10 14:08:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1620/2304 [I 2022-10-10 14:08:14] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:53 [I 2022-10-10 14:08:14] ...trap/engines/engine.py.229: process: 0.88260 | load: 0.00317 [I 2022-10-10 14:08:14] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-10 14:08:14] ...optimizers/trijoint.py.151: total_norm: 14.120258 [I 2022-10-10 14:08:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 1630/2304 [I 2022-10-10 14:08:22] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:43 [I 2022-10-10 14:08:22] ...trap/engines/engine.py.229: process: 0.88171 | load: 0.00310 [I 2022-10-10 14:08:22] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-10 14:08:22] ...optimizers/trijoint.py.151: total_norm: 29.030518 [I 2022-10-10 14:08:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1640/2304 [I 2022-10-10 14:08:31] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:35 [I 2022-10-10 14:08:31] ...trap/engines/engine.py.229: process: 0.88055 | load: 0.00309 [I 2022-10-10 14:08:31] ...trap/engines/engine.py.230: loss: 0.17068 [I 2022-10-10 14:08:31] ...optimizers/trijoint.py.151: total_norm: 15.114391 [I 2022-10-10 14:08:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 1650/2304 [I 2022-10-10 14:08:40] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:31 [I 2022-10-10 14:08:40] ...trap/engines/engine.py.229: process: 0.87062 | load: 0.00311 [I 2022-10-10 14:08:40] ...trap/engines/engine.py.230: loss: 0.15186 [I 2022-10-10 14:08:40] ...optimizers/trijoint.py.151: total_norm: 11.006934 [I 2022-10-10 14:08:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 1660/2304 [I 2022-10-10 14:08:48] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:09:16 [I 2022-10-10 14:08:48] ...trap/engines/engine.py.229: process: 0.86192 | load: 0.00315 [I 2022-10-10 14:08:48] ...trap/engines/engine.py.230: loss: 0.08522 [I 2022-10-10 14:08:48] ...optimizers/trijoint.py.151: total_norm: 8.573521 [I 2022-10-10 14:08:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1670/2304 [I 2022-10-10 14:08:57] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:09:14 [I 2022-10-10 14:08:57] ...trap/engines/engine.py.229: process: 0.89583 | load: 0.00311 [I 2022-10-10 14:08:57] ...trap/engines/engine.py.230: loss: 0.11512 [I 2022-10-10 14:08:57] ...optimizers/trijoint.py.151: total_norm: 18.359400 [I 2022-10-10 14:09:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 1680/2304 [I 2022-10-10 14:09:06] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:00 [I 2022-10-10 14:09:06] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00312 [I 2022-10-10 14:09:06] ...trap/engines/engine.py.230: loss: 0.13938 [I 2022-10-10 14:09:06] ...optimizers/trijoint.py.151: total_norm: 17.120913 [I 2022-10-10 14:09:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1690/2304 [I 2022-10-10 14:09:14] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:48 [I 2022-10-10 14:09:14] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00309 [I 2022-10-10 14:09:14] ...trap/engines/engine.py.230: loss: 0.16083 [I 2022-10-10 14:09:14] ...optimizers/trijoint.py.151: total_norm: 10.406157 [I 2022-10-10 14:09:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 1700/2304 [I 2022-10-10 14:09:23] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:34 [I 2022-10-10 14:09:23] ...trap/engines/engine.py.229: process: 0.84413 | load: 0.00298 [I 2022-10-10 14:09:23] ...trap/engines/engine.py.230: loss: 0.14008 [I 2022-10-10 14:09:23] ...optimizers/trijoint.py.151: total_norm: 16.272238 [I 2022-10-10 14:09:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 1710/2304 [I 2022-10-10 14:09:32] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:29 [I 2022-10-10 14:09:32] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00388 [I 2022-10-10 14:09:32] ...trap/engines/engine.py.230: loss: 0.18251 [I 2022-10-10 14:09:32] ...optimizers/trijoint.py.151: total_norm: 22.614408 [I 2022-10-10 14:09:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 1720/2304 [I 2022-10-10 14:09:40] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:17 [I 2022-10-10 14:09:40] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00295 [I 2022-10-10 14:09:40] ...trap/engines/engine.py.230: loss: 0.11532 [I 2022-10-10 14:09:40] ...optimizers/trijoint.py.151: total_norm: 18.590956 [I 2022-10-10 14:09:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 1730/2304 [I 2022-10-10 14:09:49] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:08:09 [I 2022-10-10 14:09:49] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00299 [I 2022-10-10 14:09:49] ...trap/engines/engine.py.230: loss: 0.15355 [I 2022-10-10 14:09:49] ...optimizers/trijoint.py.151: total_norm: 9.741467 [I 2022-10-10 14:09:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1740/2304 [I 2022-10-10 14:09:57] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:02 [I 2022-10-10 14:09:57] ...trap/engines/engine.py.229: process: 0.85488 | load: 0.00292 [I 2022-10-10 14:09:57] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-10 14:09:57] ...optimizers/trijoint.py.151: total_norm: 20.557962 [I 2022-10-10 14:10:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 1750/2304 [I 2022-10-10 14:10:06] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:51 [I 2022-10-10 14:10:06] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00302 [I 2022-10-10 14:10:06] ...trap/engines/engine.py.230: loss: 0.11073 [I 2022-10-10 14:10:06] ...optimizers/trijoint.py.151: total_norm: 15.724219 [I 2022-10-10 14:10:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1760/2304 [I 2022-10-10 14:10:14] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:41 [I 2022-10-10 14:10:14] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00292 [I 2022-10-10 14:10:14] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-10 14:10:14] ...optimizers/trijoint.py.151: total_norm: 18.136543 [I 2022-10-10 14:10:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 1770/2304 [I 2022-10-10 14:10:23] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:33 [I 2022-10-10 14:10:23] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00298 [I 2022-10-10 14:10:23] ...trap/engines/engine.py.230: loss: 0.17021 [I 2022-10-10 14:10:23] ...optimizers/trijoint.py.151: total_norm: 23.236776 [I 2022-10-10 14:10:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1780/2304 [I 2022-10-10 14:10:31] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:28 [I 2022-10-10 14:10:31] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00296 [I 2022-10-10 14:10:31] ...trap/engines/engine.py.230: loss: 0.12181 [I 2022-10-10 14:10:31] ...optimizers/trijoint.py.151: total_norm: 13.793399 [I 2022-10-10 14:10:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 1790/2304 [I 2022-10-10 14:10:40] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:21 [I 2022-10-10 14:10:40] ...trap/engines/engine.py.229: process: 0.87145 | load: 0.00305 [I 2022-10-10 14:10:40] ...trap/engines/engine.py.230: loss: 0.13193 [I 2022-10-10 14:10:40] ...optimizers/trijoint.py.151: total_norm: 10.468888 [I 2022-10-10 14:10:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 1800/2304 [I 2022-10-10 14:10:49] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:11 [I 2022-10-10 14:10:49] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00299 [I 2022-10-10 14:10:49] ...trap/engines/engine.py.230: loss: 0.07860 [I 2022-10-10 14:10:49] ...optimizers/trijoint.py.151: total_norm: 6.973846 [I 2022-10-10 14:10:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1810/2304 [I 2022-10-10 14:10:57] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:04 [I 2022-10-10 14:10:57] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00293 [I 2022-10-10 14:10:57] ...trap/engines/engine.py.230: loss: 0.14174 [I 2022-10-10 14:10:57] ...optimizers/trijoint.py.151: total_norm: 24.006069 [I 2022-10-10 14:11:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 1820/2304 [I 2022-10-10 14:11:06] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:54 [I 2022-10-10 14:11:06] ...trap/engines/engine.py.229: process: 0.87308 | load: 0.00301 [I 2022-10-10 14:11:06] ...trap/engines/engine.py.230: loss: 0.13602 [I 2022-10-10 14:11:06] ...optimizers/trijoint.py.151: total_norm: 15.110668 [I 2022-10-10 14:11:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1830/2304 [I 2022-10-10 14:11:14] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:43 [I 2022-10-10 14:11:14] ...trap/engines/engine.py.229: process: 0.86533 | load: 0.00294 [I 2022-10-10 14:11:14] ...trap/engines/engine.py.230: loss: 0.16819 [I 2022-10-10 14:11:14] ...optimizers/trijoint.py.151: total_norm: 14.959316 [I 2022-10-10 14:11:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 1840/2304 [I 2022-10-10 14:11:23] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:39 [I 2022-10-10 14:11:23] ...trap/engines/engine.py.229: process: 0.85665 | load: 0.00300 [I 2022-10-10 14:11:23] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-10 14:11:23] ...optimizers/trijoint.py.151: total_norm: 20.840366 [I 2022-10-10 14:11:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1850/2304 [I 2022-10-10 14:11:31] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:26 [I 2022-10-10 14:11:31] ...trap/engines/engine.py.229: process: 0.87136 | load: 0.00299 [I 2022-10-10 14:11:31] ...trap/engines/engine.py.230: loss: 0.17244 [I 2022-10-10 14:11:31] ...optimizers/trijoint.py.151: total_norm: 21.504650 [I 2022-10-10 14:11:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 1860/2304 [I 2022-10-10 14:11:40] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:20 [I 2022-10-10 14:11:40] ...trap/engines/engine.py.229: process: 0.85402 | load: 0.00297 [I 2022-10-10 14:11:40] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-10 14:11:40] ...optimizers/trijoint.py.151: total_norm: 22.990822 [I 2022-10-10 14:11:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 1870/2304 [I 2022-10-10 14:11:49] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:09 [I 2022-10-10 14:11:49] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00297 [I 2022-10-10 14:11:49] ...trap/engines/engine.py.230: loss: 0.20365 [I 2022-10-10 14:11:49] ...optimizers/trijoint.py.151: total_norm: 18.322868 [I 2022-10-10 14:11:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1880/2304 [I 2022-10-10 14:11:57] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:02 [I 2022-10-10 14:11:57] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00310 [I 2022-10-10 14:11:57] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-10 14:11:57] ...optimizers/trijoint.py.151: total_norm: 17.950289 [I 2022-10-10 14:12:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 1890/2304 [I 2022-10-10 14:12:06] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:52 [I 2022-10-10 14:12:06] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00293 [I 2022-10-10 14:12:06] ...trap/engines/engine.py.230: loss: 0.12559 [I 2022-10-10 14:12:06] ...optimizers/trijoint.py.151: total_norm: 12.331845 [I 2022-10-10 14:12:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1900/2304 [I 2022-10-10 14:12:14] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:44 [I 2022-10-10 14:12:14] ...trap/engines/engine.py.229: process: 0.86126 | load: 0.00291 [I 2022-10-10 14:12:14] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-10 14:12:14] ...optimizers/trijoint.py.151: total_norm: 15.415214 [I 2022-10-10 14:12:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 1910/2304 [I 2022-10-10 14:12:23] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:33 [I 2022-10-10 14:12:23] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00292 [I 2022-10-10 14:12:23] ...trap/engines/engine.py.230: loss: 0.12472 [I 2022-10-10 14:12:23] ...optimizers/trijoint.py.151: total_norm: 12.629285 [I 2022-10-10 14:12:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 1920/2304 [I 2022-10-10 14:12:50] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:21:03 [I 2022-10-10 14:12:50] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00314 [I 2022-10-10 14:12:50] ...trap/engines/engine.py.230: loss: 0.07613 [I 2022-10-10 14:12:50] ...optimizers/trijoint.py.151: total_norm: 10.653617 [I 2022-10-10 14:12:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 1930/2304 [I 2022-10-10 14:12:59] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:07:02 [I 2022-10-10 14:12:59] ...trap/engines/engine.py.229: process: 0.87192 | load: 0.00308 [I 2022-10-10 14:12:59] ...trap/engines/engine.py.230: loss: 0.16907 [I 2022-10-10 14:12:59] ...optimizers/trijoint.py.151: total_norm: 18.151907 [I 2022-10-10 14:13:08] ...trap/engines/engine.py.224: train: epoch 74 | batch 1940/2304 [I 2022-10-10 14:13:08] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:05:24 [I 2022-10-10 14:13:08] ...trap/engines/engine.py.229: process: 0.86144 | load: 0.00307 [I 2022-10-10 14:13:08] ...trap/engines/engine.py.230: loss: 0.11022 [I 2022-10-10 14:13:08] ...optimizers/trijoint.py.151: total_norm: 11.709787 [I 2022-10-10 14:13:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 1950/2304 [I 2022-10-10 14:13:16] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:05:06 [I 2022-10-10 14:13:16] ...trap/engines/engine.py.229: process: 0.87165 | load: 0.00309 [I 2022-10-10 14:13:16] ...trap/engines/engine.py.230: loss: 0.17555 [I 2022-10-10 14:13:16] ...optimizers/trijoint.py.151: total_norm: 16.273758 [I 2022-10-10 14:13:25] ...trap/engines/engine.py.224: train: epoch 74 | batch 1960/2304 [I 2022-10-10 14:13:25] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:57 [I 2022-10-10 14:13:25] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00320 [I 2022-10-10 14:13:25] ...trap/engines/engine.py.230: loss: 0.20102 [I 2022-10-10 14:13:25] ...optimizers/trijoint.py.151: total_norm: 24.311213 [I 2022-10-10 14:13:34] ...trap/engines/engine.py.224: train: epoch 74 | batch 1970/2304 [I 2022-10-10 14:13:34] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:48 [I 2022-10-10 14:13:34] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00308 [I 2022-10-10 14:13:34] ...trap/engines/engine.py.230: loss: 0.11004 [I 2022-10-10 14:13:34] ...optimizers/trijoint.py.151: total_norm: 11.180624 [I 2022-10-10 14:13:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 1980/2304 [I 2022-10-10 14:13:42] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:38 [I 2022-10-10 14:13:42] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00324 [I 2022-10-10 14:13:42] ...trap/engines/engine.py.230: loss: 0.10785 [I 2022-10-10 14:13:42] ...optimizers/trijoint.py.151: total_norm: 22.503490 [I 2022-10-10 14:13:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 1990/2304 [I 2022-10-10 14:13:51] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:29 [I 2022-10-10 14:13:51] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00321 [I 2022-10-10 14:13:51] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-10 14:13:51] ...optimizers/trijoint.py.151: total_norm: 20.466774 [I 2022-10-10 14:14:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 2000/2304 [I 2022-10-10 14:14:00] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:22 [I 2022-10-10 14:14:00] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00310 [I 2022-10-10 14:14:00] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-10 14:14:00] ...optimizers/trijoint.py.151: total_norm: 11.352883 [I 2022-10-10 14:14:08] ...trap/engines/engine.py.224: train: epoch 74 | batch 2010/2304 [I 2022-10-10 14:14:08] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:13 [I 2022-10-10 14:14:08] ...trap/engines/engine.py.229: process: 0.86373 | load: 0.00309 [I 2022-10-10 14:14:08] ...trap/engines/engine.py.230: loss: 0.13330 [I 2022-10-10 14:14:08] ...optimizers/trijoint.py.151: total_norm: 11.213224 [I 2022-10-10 14:14:17] ...trap/engines/engine.py.224: train: epoch 74 | batch 2020/2304 [I 2022-10-10 14:14:17] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:04:06 [I 2022-10-10 14:14:17] ...trap/engines/engine.py.229: process: 0.87010 | load: 0.00316 [I 2022-10-10 14:14:17] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-10 14:14:17] ...optimizers/trijoint.py.151: total_norm: 17.609962 [I 2022-10-10 14:14:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 2030/2304 [I 2022-10-10 14:14:26] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:55 [I 2022-10-10 14:14:26] ...trap/engines/engine.py.229: process: 0.84279 | load: 0.00310 [I 2022-10-10 14:14:26] ...trap/engines/engine.py.230: loss: 0.09630 [I 2022-10-10 14:14:26] ...optimizers/trijoint.py.151: total_norm: 11.237514 [I 2022-10-10 14:14:34] ...trap/engines/engine.py.224: train: epoch 74 | batch 2040/2304 [I 2022-10-10 14:14:34] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:48 [I 2022-10-10 14:14:34] ...trap/engines/engine.py.229: process: 0.86544 | load: 0.00309 [I 2022-10-10 14:14:34] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-10 14:14:34] ...optimizers/trijoint.py.151: total_norm: 13.787149 [I 2022-10-10 14:14:43] ...trap/engines/engine.py.224: train: epoch 74 | batch 2050/2304 [I 2022-10-10 14:14:43] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:39 [I 2022-10-10 14:14:43] ...trap/engines/engine.py.229: process: 0.85925 | load: 0.00306 [I 2022-10-10 14:14:43] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-10 14:14:43] ...optimizers/trijoint.py.151: total_norm: 11.767520 [I 2022-10-10 14:14:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 2060/2304 [I 2022-10-10 14:14:52] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:30 [I 2022-10-10 14:14:52] ...trap/engines/engine.py.229: process: 0.86956 | load: 0.00310 [I 2022-10-10 14:14:52] ...trap/engines/engine.py.230: loss: 0.16247 [I 2022-10-10 14:14:52] ...optimizers/trijoint.py.151: total_norm: 21.081928 [I 2022-10-10 14:15:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 2070/2304 [I 2022-10-10 14:15:00] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:03:23 [I 2022-10-10 14:15:00] ...trap/engines/engine.py.229: process: 0.87432 | load: 0.00310 [I 2022-10-10 14:15:00] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-10 14:15:00] ...optimizers/trijoint.py.151: total_norm: 32.935303 [I 2022-10-10 14:15:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 2080/2304 [I 2022-10-10 14:15:09] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:03:13 [I 2022-10-10 14:15:09] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00313 [I 2022-10-10 14:15:09] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-10 14:15:09] ...optimizers/trijoint.py.151: total_norm: 21.193264 [I 2022-10-10 14:15:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 2090/2304 [I 2022-10-10 14:15:18] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:03:04 [I 2022-10-10 14:15:18] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00307 [I 2022-10-10 14:15:18] ...trap/engines/engine.py.230: loss: 0.16847 [I 2022-10-10 14:15:18] ...optimizers/trijoint.py.151: total_norm: 11.231217 [I 2022-10-10 14:15:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 2100/2304 [I 2022-10-10 14:15:26] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:56 [I 2022-10-10 14:15:26] ...trap/engines/engine.py.229: process: 0.88151 | load: 0.00320 [I 2022-10-10 14:15:26] ...trap/engines/engine.py.230: loss: 0.12761 [I 2022-10-10 14:15:26] ...optimizers/trijoint.py.151: total_norm: 14.826743 [I 2022-10-10 14:15:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 2110/2304 [I 2022-10-10 14:15:35] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:47 [I 2022-10-10 14:15:35] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00312 [I 2022-10-10 14:15:35] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-10 14:15:35] ...optimizers/trijoint.py.151: total_norm: 10.557036 [I 2022-10-10 14:15:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 2120/2304 [I 2022-10-10 14:15:44] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:40 [I 2022-10-10 14:15:44] ...trap/engines/engine.py.229: process: 0.87181 | load: 0.00317 [I 2022-10-10 14:15:44] ...trap/engines/engine.py.230: loss: 0.20658 [I 2022-10-10 14:15:44] ...optimizers/trijoint.py.151: total_norm: 61.439667 [I 2022-10-10 14:15:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 2130/2304 [I 2022-10-10 14:15:52] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:30 [I 2022-10-10 14:15:52] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00307 [I 2022-10-10 14:15:52] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-10 14:15:52] ...optimizers/trijoint.py.151: total_norm: 29.499769 [I 2022-10-10 14:16:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 2140/2304 [I 2022-10-10 14:16:01] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:22 [I 2022-10-10 14:16:01] ...trap/engines/engine.py.229: process: 0.85257 | load: 0.00309 [I 2022-10-10 14:16:01] ...trap/engines/engine.py.230: loss: 0.17351 [I 2022-10-10 14:16:01] ...optimizers/trijoint.py.151: total_norm: 38.020447 [I 2022-10-10 14:16:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 2150/2304 [I 2022-10-10 14:16:10] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:02:12 [I 2022-10-10 14:16:10] ...trap/engines/engine.py.229: process: 0.86682 | load: 0.00339 [I 2022-10-10 14:16:10] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-10 14:16:10] ...optimizers/trijoint.py.151: total_norm: 17.278357 [I 2022-10-10 14:16:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 2160/2304 [I 2022-10-10 14:16:18] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:02:04 [I 2022-10-10 14:16:18] ...trap/engines/engine.py.229: process: 0.87218 | load: 0.00306 [I 2022-10-10 14:16:18] ...trap/engines/engine.py.230: loss: 0.14252 [I 2022-10-10 14:16:18] ...optimizers/trijoint.py.151: total_norm: 24.183683 [I 2022-10-10 14:16:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 2170/2304 [I 2022-10-10 14:16:27] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:55 [I 2022-10-10 14:16:27] ...trap/engines/engine.py.229: process: 0.86269 | load: 0.00296 [I 2022-10-10 14:16:27] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-10 14:16:27] ...optimizers/trijoint.py.151: total_norm: 22.456902 [I 2022-10-10 14:16:36] ...trap/engines/engine.py.224: train: epoch 74 | batch 2180/2304 [I 2022-10-10 14:16:36] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:46 [I 2022-10-10 14:16:36] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00305 [I 2022-10-10 14:16:36] ...trap/engines/engine.py.230: loss: 0.15295 [I 2022-10-10 14:16:36] ...optimizers/trijoint.py.151: total_norm: 26.811100 [I 2022-10-10 14:16:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 2190/2304 [I 2022-10-10 14:16:44] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:37 [I 2022-10-10 14:16:44] ...trap/engines/engine.py.229: process: 0.83756 | load: 0.00297 [I 2022-10-10 14:16:44] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-10 14:16:44] ...optimizers/trijoint.py.151: total_norm: 15.218435 [I 2022-10-10 14:16:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 2200/2304 [I 2022-10-10 14:16:53] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:29 [I 2022-10-10 14:16:53] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00298 [I 2022-10-10 14:16:53] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-10 14:16:53] ...optimizers/trijoint.py.151: total_norm: 15.396936 [I 2022-10-10 14:17:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 2210/2304 [I 2022-10-10 14:17:01] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:20 [I 2022-10-10 14:17:01] ...trap/engines/engine.py.229: process: 0.86269 | load: 0.00297 [I 2022-10-10 14:17:01] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-10 14:17:01] ...optimizers/trijoint.py.151: total_norm: 21.279461 [I 2022-10-10 14:17:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 2220/2304 [I 2022-10-10 14:17:10] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:01:11 [I 2022-10-10 14:17:10] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00298 [I 2022-10-10 14:17:10] ...trap/engines/engine.py.230: loss: 0.09180 [I 2022-10-10 14:17:10] ...optimizers/trijoint.py.151: total_norm: 7.918490 [I 2022-10-10 14:17:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 2230/2304 [I 2022-10-10 14:17:18] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:01:02 [I 2022-10-10 14:17:18] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00293 [I 2022-10-10 14:17:18] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-10 14:17:18] ...optimizers/trijoint.py.151: total_norm: 26.535969 [I 2022-10-10 14:17:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 2240/2304 [I 2022-10-10 14:17:27] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:54 [I 2022-10-10 14:17:27] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00294 [I 2022-10-10 14:17:27] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-10 14:17:27] ...optimizers/trijoint.py.151: total_norm: 11.488780 [I 2022-10-10 14:17:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 2250/2304 [I 2022-10-10 14:17:35] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:45 [I 2022-10-10 14:17:35] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00293 [I 2022-10-10 14:17:35] ...trap/engines/engine.py.230: loss: 0.17342 [I 2022-10-10 14:17:35] ...optimizers/trijoint.py.151: total_norm: 14.225878 [I 2022-10-10 14:17:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 2260/2304 [I 2022-10-10 14:17:44] ...trap/engines/engine.py.228: elapsed: 0:32:53 | left: 0:00:37 [I 2022-10-10 14:17:44] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00294 [I 2022-10-10 14:17:44] ...trap/engines/engine.py.230: loss: 0.14778 [I 2022-10-10 14:17:44] ...optimizers/trijoint.py.151: total_norm: 10.291525 [I 2022-10-10 14:17:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 2270/2304 [I 2022-10-10 14:17:53] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:29 [I 2022-10-10 14:17:53] ...trap/engines/engine.py.229: process: 0.86864 | load: 0.00303 [I 2022-10-10 14:17:53] ...trap/engines/engine.py.230: loss: 0.18038 [I 2022-10-10 14:17:53] ...optimizers/trijoint.py.151: total_norm: 24.128090 [I 2022-10-10 14:18:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 2280/2304 [I 2022-10-10 14:18:01] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:20 [I 2022-10-10 14:18:01] ...trap/engines/engine.py.229: process: 0.84664 | load: 0.00294 [I 2022-10-10 14:18:01] ...trap/engines/engine.py.230: loss: 0.16850 [I 2022-10-10 14:18:01] ...optimizers/trijoint.py.151: total_norm: 22.184818 [I 2022-10-10 14:18:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 2290/2304 [I 2022-10-10 14:18:10] ...trap/engines/engine.py.228: elapsed: 0:33:19 | left: 0:00:11 [I 2022-10-10 14:18:10] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00292 [I 2022-10-10 14:18:10] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-10 14:18:10] ...optimizers/trijoint.py.151: total_norm: 10.439876 [I 2022-10-10 14:18:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 2300/2304 [I 2022-10-10 14:18:18] ...trap/engines/engine.py.228: elapsed: 0:33:27 | left: 0:00:03 [I 2022-10-10 14:18:18] ...trap/engines/engine.py.229: process: 0.83909 | load: 0.00017 [I 2022-10-10 14:18:18] ...trap/engines/engine.py.230: loss: 0.19570 [I 2022-10-10 14:18:18] ...optimizers/trijoint.py.151: total_norm: 24.898020 [I 2022-10-10 14:18:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 2304/2304 [I 2022-10-10 14:18:22] ...trap/engines/engine.py.228: elapsed: 0:33:31 | left: 0:00:00 [I 2022-10-10 14:18:22] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00017 [I 2022-10-10 14:18:22] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-10 14:18:22] ...optimizers/trijoint.py.151: total_norm: 17.073196 [S 2022-10-10 14:18:23] ...trap/engines/engine.py.129: train_epoch.epoch: 74 [S 2022-10-10 14:18:23] ...trap/engines/engine.py.129: train_epoch.loss: 0.1369745093351331 [I 2022-10-10 14:18:23] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 14:18:24] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 74 [I 2022-10-10 14:18:25] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 14:18:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 14:18:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 14:18:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 14:18:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 14:18:29] ...trap/engines/engine.py.321: eval: epoch 74 | batch 0/510 [I 2022-10-10 14:18:29] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:46 [I 2022-10-10 14:18:29] ...trap/engines/engine.py.326: process: 4.91457 | load: 4.37403 [I 2022-10-10 14:18:34] ...trap/engines/engine.py.321: eval: epoch 74 | batch 10/510 [I 2022-10-10 14:18:34] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:09 [I 2022-10-10 14:18:34] ...trap/engines/engine.py.326: process: 0.50158 | load: 0.00018 [I 2022-10-10 14:18:40] ...trap/engines/engine.py.321: eval: epoch 74 | batch 20/510 [I 2022-10-10 14:18:40] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:17 [I 2022-10-10 14:18:40] ...trap/engines/engine.py.326: process: 0.66460 | load: 0.00019 [I 2022-10-10 14:18:46] ...trap/engines/engine.py.321: eval: epoch 74 | batch 30/510 [I 2022-10-10 14:18:46] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:52 [I 2022-10-10 14:18:46] ...trap/engines/engine.py.326: process: 0.66622 | load: 0.00018 [I 2022-10-10 14:18:53] ...trap/engines/engine.py.321: eval: epoch 74 | batch 40/510 [I 2022-10-10 14:18:53] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:05:14 [I 2022-10-10 14:18:53] ...trap/engines/engine.py.326: process: 0.61847 | load: 0.00020 [I 2022-10-10 14:18:58] ...trap/engines/engine.py.321: eval: epoch 74 | batch 50/510 [I 2022-10-10 14:18:58] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:04:02 [I 2022-10-10 14:18:58] ...trap/engines/engine.py.326: process: 0.52484 | load: 0.00018 [I 2022-10-10 14:19:03] ...trap/engines/engine.py.321: eval: epoch 74 | batch 60/510 [I 2022-10-10 14:19:03] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:47 [I 2022-10-10 14:19:03] ...trap/engines/engine.py.326: process: 0.48402 | load: 0.00019 [I 2022-10-10 14:19:08] ...trap/engines/engine.py.321: eval: epoch 74 | batch 70/510 [I 2022-10-10 14:19:08] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:48 [I 2022-10-10 14:19:08] ...trap/engines/engine.py.326: process: 0.54797 | load: 0.00019 [I 2022-10-10 14:19:13] ...trap/engines/engine.py.321: eval: epoch 74 | batch 80/510 [I 2022-10-10 14:19:13] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:36 [I 2022-10-10 14:19:13] ...trap/engines/engine.py.326: process: 0.51676 | load: 0.00018 [I 2022-10-10 14:19:19] ...trap/engines/engine.py.321: eval: epoch 74 | batch 90/510 [I 2022-10-10 14:19:19] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:05:13 [I 2022-10-10 14:19:19] ...trap/engines/engine.py.326: process: 0.48294 | load: 0.00020 [I 2022-10-10 14:19:26] ...trap/engines/engine.py.321: eval: epoch 74 | batch 100/510 [I 2022-10-10 14:19:26] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:05:26 [I 2022-10-10 14:19:26] ...trap/engines/engine.py.326: process: 0.35096 | load: 0.00018 [I 2022-10-10 14:19:31] ...trap/engines/engine.py.321: eval: epoch 74 | batch 110/510 [I 2022-10-10 14:19:31] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:47 [I 2022-10-10 14:19:31] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00020 [I 2022-10-10 14:19:34] ...trap/engines/engine.py.321: eval: epoch 74 | batch 120/510 [I 2022-10-10 14:19:34] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:09 [I 2022-10-10 14:19:34] ...trap/engines/engine.py.326: process: 0.31885 | load: 0.00021 [I 2022-10-10 14:19:37] ...trap/engines/engine.py.321: eval: epoch 74 | batch 130/510 [I 2022-10-10 14:19:37] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:02 [I 2022-10-10 14:19:37] ...trap/engines/engine.py.326: process: 0.31637 | load: 0.00020 [I 2022-10-10 14:19:40] ...trap/engines/engine.py.321: eval: epoch 74 | batch 140/510 [I 2022-10-10 14:19:40] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:58 [I 2022-10-10 14:19:40] ...trap/engines/engine.py.326: process: 0.31683 | load: 0.00020 [I 2022-10-10 14:19:43] ...trap/engines/engine.py.321: eval: epoch 74 | batch 150/510 [I 2022-10-10 14:19:43] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-10 14:19:43] ...trap/engines/engine.py.326: process: 0.32348 | load: 0.00020 [I 2022-10-10 14:19:47] ...trap/engines/engine.py.321: eval: epoch 74 | batch 160/510 [I 2022-10-10 14:19:47] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:52 [I 2022-10-10 14:19:47] ...trap/engines/engine.py.326: process: 0.32704 | load: 0.00020 [I 2022-10-10 14:19:50] ...trap/engines/engine.py.321: eval: epoch 74 | batch 170/510 [I 2022-10-10 14:19:50] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:50 [I 2022-10-10 14:19:50] ...trap/engines/engine.py.326: process: 0.32472 | load: 0.00021 [I 2022-10-10 14:19:53] ...trap/engines/engine.py.321: eval: epoch 74 | batch 180/510 [I 2022-10-10 14:19:53] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:46 [I 2022-10-10 14:19:53] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00021 [I 2022-10-10 14:19:56] ...trap/engines/engine.py.321: eval: epoch 74 | batch 190/510 [I 2022-10-10 14:19:56] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:43 [I 2022-10-10 14:19:56] ...trap/engines/engine.py.326: process: 0.32454 | load: 0.00021 [I 2022-10-10 14:20:00] ...trap/engines/engine.py.321: eval: epoch 74 | batch 200/510 [I 2022-10-10 14:20:00] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:40 [I 2022-10-10 14:20:00] ...trap/engines/engine.py.326: process: 0.33302 | load: 0.00020 [I 2022-10-10 14:20:03] ...trap/engines/engine.py.321: eval: epoch 74 | batch 210/510 [I 2022-10-10 14:20:03] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:37 [I 2022-10-10 14:20:03] ...trap/engines/engine.py.326: process: 0.33807 | load: 0.00023 [I 2022-10-10 14:20:06] ...trap/engines/engine.py.321: eval: epoch 74 | batch 220/510 [I 2022-10-10 14:20:06] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:33 [I 2022-10-10 14:20:06] ...trap/engines/engine.py.326: process: 0.32174 | load: 0.00023 [I 2022-10-10 14:20:09] ...trap/engines/engine.py.321: eval: epoch 74 | batch 230/510 [I 2022-10-10 14:20:09] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:30 [I 2022-10-10 14:20:09] ...trap/engines/engine.py.326: process: 0.32064 | load: 0.00023 [I 2022-10-10 14:20:13] ...trap/engines/engine.py.321: eval: epoch 74 | batch 240/510 [I 2022-10-10 14:20:13] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:27 [I 2022-10-10 14:20:13] ...trap/engines/engine.py.326: process: 0.32301 | load: 0.00021 [I 2022-10-10 14:20:16] ...trap/engines/engine.py.321: eval: epoch 74 | batch 250/510 [I 2022-10-10 14:20:16] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:24 [I 2022-10-10 14:20:16] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00025 [I 2022-10-10 14:20:19] ...trap/engines/engine.py.321: eval: epoch 74 | batch 260/510 [I 2022-10-10 14:20:19] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:21 [I 2022-10-10 14:20:19] ...trap/engines/engine.py.326: process: 0.32457 | load: 0.00021 [I 2022-10-10 14:20:22] ...trap/engines/engine.py.321: eval: epoch 74 | batch 270/510 [I 2022-10-10 14:20:22] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:17 [I 2022-10-10 14:20:22] ...trap/engines/engine.py.326: process: 0.32771 | load: 0.00021 [I 2022-10-10 14:20:26] ...trap/engines/engine.py.321: eval: epoch 74 | batch 280/510 [I 2022-10-10 14:20:26] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:15 [I 2022-10-10 14:20:26] ...trap/engines/engine.py.326: process: 0.33576 | load: 0.00022 [I 2022-10-10 14:20:29] ...trap/engines/engine.py.321: eval: epoch 74 | batch 290/510 [I 2022-10-10 14:20:29] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:11 [I 2022-10-10 14:20:29] ...trap/engines/engine.py.326: process: 0.32082 | load: 0.00021 [I 2022-10-10 14:20:32] ...trap/engines/engine.py.321: eval: epoch 74 | batch 300/510 [I 2022-10-10 14:20:32] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:08 [I 2022-10-10 14:20:32] ...trap/engines/engine.py.326: process: 0.32845 | load: 0.00022 [I 2022-10-10 14:20:35] ...trap/engines/engine.py.321: eval: epoch 74 | batch 310/510 [I 2022-10-10 14:20:35] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-10 14:20:35] ...trap/engines/engine.py.326: process: 0.32536 | load: 0.00023 [I 2022-10-10 14:20:38] ...trap/engines/engine.py.321: eval: epoch 74 | batch 320/510 [I 2022-10-10 14:20:38] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:01 [I 2022-10-10 14:20:38] ...trap/engines/engine.py.326: process: 0.32914 | load: 0.00021 [I 2022-10-10 14:20:42] ...trap/engines/engine.py.321: eval: epoch 74 | batch 330/510 [I 2022-10-10 14:20:42] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:58 [I 2022-10-10 14:20:42] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00021 [I 2022-10-10 14:20:45] ...trap/engines/engine.py.321: eval: epoch 74 | batch 340/510 [I 2022-10-10 14:20:45] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:55 [I 2022-10-10 14:20:45] ...trap/engines/engine.py.326: process: 0.32876 | load: 0.00021 [I 2022-10-10 14:20:48] ...trap/engines/engine.py.321: eval: epoch 74 | batch 350/510 [I 2022-10-10 14:20:48] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:51 [I 2022-10-10 14:20:48] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00021 [I 2022-10-10 14:20:52] ...trap/engines/engine.py.321: eval: epoch 74 | batch 360/510 [I 2022-10-10 14:20:52] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-10 14:20:52] ...trap/engines/engine.py.326: process: 0.31801 | load: 0.00021 [I 2022-10-10 14:20:55] ...trap/engines/engine.py.321: eval: epoch 74 | batch 370/510 [I 2022-10-10 14:20:55] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:45 [I 2022-10-10 14:20:55] ...trap/engines/engine.py.326: process: 0.32110 | load: 0.00022 [I 2022-10-10 14:20:58] ...trap/engines/engine.py.321: eval: epoch 74 | batch 380/510 [I 2022-10-10 14:20:58] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:42 [I 2022-10-10 14:20:58] ...trap/engines/engine.py.326: process: 0.32496 | load: 0.00022 [I 2022-10-10 14:21:01] ...trap/engines/engine.py.321: eval: epoch 74 | batch 390/510 [I 2022-10-10 14:21:01] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:38 [I 2022-10-10 14:21:01] ...trap/engines/engine.py.326: process: 0.32398 | load: 0.00022 [I 2022-10-10 14:21:04] ...trap/engines/engine.py.321: eval: epoch 74 | batch 400/510 [I 2022-10-10 14:21:04] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-10 14:21:04] ...trap/engines/engine.py.326: process: 0.32606 | load: 0.00023 [I 2022-10-10 14:21:08] ...trap/engines/engine.py.321: eval: epoch 74 | batch 410/510 [I 2022-10-10 14:21:08] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-10 14:21:08] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00022 [I 2022-10-10 14:21:11] ...trap/engines/engine.py.321: eval: epoch 74 | batch 420/510 [I 2022-10-10 14:21:11] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:29 [I 2022-10-10 14:21:11] ...trap/engines/engine.py.326: process: 0.32272 | load: 0.00023 [I 2022-10-10 14:21:14] ...trap/engines/engine.py.321: eval: epoch 74 | batch 430/510 [I 2022-10-10 14:21:14] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:25 [I 2022-10-10 14:21:14] ...trap/engines/engine.py.326: process: 0.32253 | load: 0.00021 [I 2022-10-10 14:21:17] ...trap/engines/engine.py.321: eval: epoch 74 | batch 440/510 [I 2022-10-10 14:21:17] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-10 14:21:17] ...trap/engines/engine.py.326: process: 0.32460 | load: 0.00021 [I 2022-10-10 14:21:21] ...trap/engines/engine.py.321: eval: epoch 74 | batch 450/510 [I 2022-10-10 14:21:21] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-10 14:21:21] ...trap/engines/engine.py.326: process: 0.32399 | load: 0.00023 [I 2022-10-10 14:21:24] ...trap/engines/engine.py.321: eval: epoch 74 | batch 460/510 [I 2022-10-10 14:21:24] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:16 [I 2022-10-10 14:21:24] ...trap/engines/engine.py.326: process: 0.32755 | load: 0.00021 [I 2022-10-10 14:21:27] ...trap/engines/engine.py.321: eval: epoch 74 | batch 470/510 [I 2022-10-10 14:21:27] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-10 14:21:27] ...trap/engines/engine.py.326: process: 0.31884 | load: 0.00022 [I 2022-10-10 14:21:30] ...trap/engines/engine.py.321: eval: epoch 74 | batch 480/510 [I 2022-10-10 14:21:30] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-10 14:21:30] ...trap/engines/engine.py.326: process: 0.31523 | load: 0.00021 [I 2022-10-10 14:21:34] ...trap/engines/engine.py.321: eval: epoch 74 | batch 490/510 [I 2022-10-10 14:21:34] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-10 14:21:34] ...trap/engines/engine.py.326: process: 0.33327 | load: 0.00021 [I 2022-10-10 14:21:37] ...trap/engines/engine.py.321: eval: epoch 74 | batch 500/510 [I 2022-10-10 14:21:37] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-10 14:21:37] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00023 [I 2022-10-10 14:21:40] ...trap/engines/engine.py.321: eval: epoch 74 | batch 510/510 [I 2022-10-10 14:21:40] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:00 [I 2022-10-10 14:21:40] ...trap/engines/engine.py.326: process: 0.33392 | load: 0.00017 [S 2022-10-10 14:21:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 74 [I 2022-10-10 14:21:41] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737200 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913500 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939800 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727400 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910500 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937200 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017221 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008090 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006882 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017229 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008405 [S 2022-10-10 14:22:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006046 [I 2022-10-10 14:22:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 14:22:16] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 14:22:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 14:22:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 14:22:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 14:22:21] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 14:22:22] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 14:22:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 14:22:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 75 [I 2022-10-10 14:22:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 14:22:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 0/2304 [I 2022-10-10 14:22:31] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:00:07 [I 2022-10-10 14:22:31] ...trap/engines/engine.py.229: process: 3.12841 | load: 2.25783 [I 2022-10-10 14:22:31] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-10 14:22:31] ...optimizers/trijoint.py.151: total_norm: 15.544672 [I 2022-10-10 14:22:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 10/2304 [I 2022-10-10 14:22:39] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:21 [I 2022-10-10 14:22:39] ...trap/engines/engine.py.229: process: 0.87136 | load: 0.00632 [I 2022-10-10 14:22:39] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-10 14:22:39] ...optimizers/trijoint.py.151: total_norm: 12.216428 [I 2022-10-10 14:22:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 20/2304 [I 2022-10-10 14:22:48] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:01 [I 2022-10-10 14:22:48] ...trap/engines/engine.py.229: process: 0.89011 | load: 0.00469 [I 2022-10-10 14:22:48] ...trap/engines/engine.py.230: loss: 0.12417 [I 2022-10-10 14:22:48] ...optimizers/trijoint.py.151: total_norm: 34.981720 [I 2022-10-10 14:22:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 30/2304 [I 2022-10-10 14:22:57] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:56 [I 2022-10-10 14:22:57] ...trap/engines/engine.py.229: process: 0.86806 | load: 0.00580 [I 2022-10-10 14:22:57] ...trap/engines/engine.py.230: loss: 0.20168 [I 2022-10-10 14:22:57] ...optimizers/trijoint.py.151: total_norm: 21.396959 [I 2022-10-10 14:23:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 40/2304 [I 2022-10-10 14:23:05] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:42 [I 2022-10-10 14:23:05] ...trap/engines/engine.py.229: process: 0.87271 | load: 0.00395 [I 2022-10-10 14:23:05] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-10 14:23:05] ...optimizers/trijoint.py.151: total_norm: 15.329379 [I 2022-10-10 14:23:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 50/2304 [I 2022-10-10 14:23:14] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:20 [I 2022-10-10 14:23:14] ...trap/engines/engine.py.229: process: 0.86135 | load: 0.00483 [I 2022-10-10 14:23:14] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-10 14:23:14] ...optimizers/trijoint.py.151: total_norm: 10.991323 [I 2022-10-10 14:23:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 60/2304 [I 2022-10-10 14:23:23] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:10 [I 2022-10-10 14:23:23] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00352 [I 2022-10-10 14:23:23] ...trap/engines/engine.py.230: loss: 0.11085 [I 2022-10-10 14:23:23] ...optimizers/trijoint.py.151: total_norm: 15.524986 [I 2022-10-10 14:23:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 70/2304 [I 2022-10-10 14:23:31] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:47 [I 2022-10-10 14:23:31] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00335 [I 2022-10-10 14:23:31] ...trap/engines/engine.py.230: loss: 0.16537 [I 2022-10-10 14:23:31] ...optimizers/trijoint.py.151: total_norm: 14.629344 [I 2022-10-10 14:23:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 80/2304 [I 2022-10-10 14:23:40] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:33 [I 2022-10-10 14:23:40] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00337 [I 2022-10-10 14:23:40] ...trap/engines/engine.py.230: loss: 0.19158 [I 2022-10-10 14:23:40] ...optimizers/trijoint.py.151: total_norm: 26.748730 [I 2022-10-10 14:23:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 90/2304 [I 2022-10-10 14:23:48] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:28 [I 2022-10-10 14:23:48] ...trap/engines/engine.py.229: process: 0.83496 | load: 0.00356 [I 2022-10-10 14:23:48] ...trap/engines/engine.py.230: loss: 0.14159 [I 2022-10-10 14:23:48] ...optimizers/trijoint.py.151: total_norm: 21.290993 [I 2022-10-10 14:23:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 100/2304 [I 2022-10-10 14:23:57] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:09 [I 2022-10-10 14:23:57] ...trap/engines/engine.py.229: process: 0.84583 | load: 0.00319 [I 2022-10-10 14:23:57] ...trap/engines/engine.py.230: loss: 0.12179 [I 2022-10-10 14:23:57] ...optimizers/trijoint.py.151: total_norm: 10.309599 [I 2022-10-10 14:24:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 110/2304 [I 2022-10-10 14:24:06] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:15 [I 2022-10-10 14:24:06] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00355 [I 2022-10-10 14:24:06] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-10 14:24:06] ...optimizers/trijoint.py.151: total_norm: 14.922485 [I 2022-10-10 14:24:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 120/2304 [I 2022-10-10 14:24:14] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:03 [I 2022-10-10 14:24:14] ...trap/engines/engine.py.229: process: 0.83804 | load: 0.00316 [I 2022-10-10 14:24:14] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-10 14:24:14] ...optimizers/trijoint.py.151: total_norm: 16.969357 [I 2022-10-10 14:24:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 130/2304 [I 2022-10-10 14:24:23] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:35 [I 2022-10-10 14:24:23] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00310 [I 2022-10-10 14:24:23] ...trap/engines/engine.py.230: loss: 0.09197 [I 2022-10-10 14:24:23] ...optimizers/trijoint.py.151: total_norm: 7.508277 [I 2022-10-10 14:24:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 140/2304 [I 2022-10-10 14:24:31] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:36 [I 2022-10-10 14:24:31] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00340 [I 2022-10-10 14:24:31] ...trap/engines/engine.py.230: loss: 0.08899 [I 2022-10-10 14:24:31] ...optimizers/trijoint.py.151: total_norm: 8.931117 [I 2022-10-10 14:24:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 150/2304 [I 2022-10-10 14:24:40] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:36 [I 2022-10-10 14:24:40] ...trap/engines/engine.py.229: process: 0.85991 | load: 0.00306 [I 2022-10-10 14:24:40] ...trap/engines/engine.py.230: loss: 0.20772 [I 2022-10-10 14:24:40] ...optimizers/trijoint.py.151: total_norm: 17.511217 [I 2022-10-10 14:24:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 160/2304 [I 2022-10-10 14:24:48] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:22 [I 2022-10-10 14:24:48] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00298 [I 2022-10-10 14:24:48] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-10 14:24:48] ...optimizers/trijoint.py.151: total_norm: 13.145828 [I 2022-10-10 14:24:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 170/2304 [I 2022-10-10 14:24:57] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:11 [I 2022-10-10 14:24:57] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00297 [I 2022-10-10 14:24:57] ...trap/engines/engine.py.230: loss: 0.16924 [I 2022-10-10 14:24:57] ...optimizers/trijoint.py.151: total_norm: 12.428819 [I 2022-10-10 14:25:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 180/2304 [I 2022-10-10 14:25:05] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:15 [I 2022-10-10 14:25:05] ...trap/engines/engine.py.229: process: 0.86014 | load: 0.00295 [I 2022-10-10 14:25:05] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-10 14:25:05] ...optimizers/trijoint.py.151: total_norm: 14.477296 [I 2022-10-10 14:25:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 190/2304 [I 2022-10-10 14:25:14] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:29:59 [I 2022-10-10 14:25:14] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00307 [I 2022-10-10 14:25:14] ...trap/engines/engine.py.230: loss: 0.09111 [I 2022-10-10 14:25:14] ...optimizers/trijoint.py.151: total_norm: 11.300865 [I 2022-10-10 14:25:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 200/2304 [I 2022-10-10 14:25:22] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:43 [I 2022-10-10 14:25:22] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00302 [I 2022-10-10 14:25:22] ...trap/engines/engine.py.230: loss: 0.15338 [I 2022-10-10 14:25:22] ...optimizers/trijoint.py.151: total_norm: 29.222807 [I 2022-10-10 14:25:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 210/2304 [I 2022-10-10 14:25:31] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:35 [I 2022-10-10 14:25:31] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00297 [I 2022-10-10 14:25:31] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-10 14:25:31] ...optimizers/trijoint.py.151: total_norm: 20.690540 [I 2022-10-10 14:25:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 220/2304 [I 2022-10-10 14:25:39] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:32 [I 2022-10-10 14:25:39] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00295 [I 2022-10-10 14:25:39] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-10 14:25:39] ...optimizers/trijoint.py.151: total_norm: 27.737814 [I 2022-10-10 14:25:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 230/2304 [I 2022-10-10 14:25:48] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:21 [I 2022-10-10 14:25:48] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00306 [I 2022-10-10 14:25:48] ...trap/engines/engine.py.230: loss: 0.12099 [I 2022-10-10 14:25:48] ...optimizers/trijoint.py.151: total_norm: 18.797049 [I 2022-10-10 14:25:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 240/2304 [I 2022-10-10 14:25:56] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:05 [I 2022-10-10 14:25:56] ...trap/engines/engine.py.229: process: 0.82752 | load: 0.00294 [I 2022-10-10 14:25:56] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-10 14:25:56] ...optimizers/trijoint.py.151: total_norm: 7.107487 [I 2022-10-10 14:26:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 250/2304 [I 2022-10-10 14:26:05] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:04 [I 2022-10-10 14:26:05] ...trap/engines/engine.py.229: process: 0.85024 | load: 0.00306 [I 2022-10-10 14:26:05] ...trap/engines/engine.py.230: loss: 0.12746 [I 2022-10-10 14:26:05] ...optimizers/trijoint.py.151: total_norm: 15.239426 [I 2022-10-10 14:26:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 260/2304 [I 2022-10-10 14:26:14] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:03 [I 2022-10-10 14:26:14] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00307 [I 2022-10-10 14:26:14] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-10 14:26:14] ...optimizers/trijoint.py.151: total_norm: 38.007168 [I 2022-10-10 14:26:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 270/2304 [I 2022-10-10 14:26:22] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:47 [I 2022-10-10 14:26:22] ...trap/engines/engine.py.229: process: 0.83421 | load: 0.00298 [I 2022-10-10 14:26:22] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-10 14:26:22] ...optimizers/trijoint.py.151: total_norm: 8.200220 [I 2022-10-10 14:26:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 280/2304 [I 2022-10-10 14:26:31] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:39 [I 2022-10-10 14:26:31] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00292 [I 2022-10-10 14:26:31] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-10 14:26:31] ...optimizers/trijoint.py.151: total_norm: 49.126087 [I 2022-10-10 14:26:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 290/2304 [I 2022-10-10 14:26:39] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:39 [I 2022-10-10 14:26:39] ...trap/engines/engine.py.229: process: 0.85721 | load: 0.00305 [I 2022-10-10 14:26:39] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-10 14:26:39] ...optimizers/trijoint.py.151: total_norm: 14.912209 [I 2022-10-10 14:26:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 300/2304 [I 2022-10-10 14:26:48] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:29 [I 2022-10-10 14:26:48] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00293 [I 2022-10-10 14:26:48] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-10 14:26:48] ...optimizers/trijoint.py.151: total_norm: 16.278902 [I 2022-10-10 14:26:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 310/2304 [I 2022-10-10 14:26:56] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:31 [I 2022-10-10 14:26:56] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00312 [I 2022-10-10 14:26:56] ...trap/engines/engine.py.230: loss: 0.15644 [I 2022-10-10 14:26:56] ...optimizers/trijoint.py.151: total_norm: 14.440609 [I 2022-10-10 14:27:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 320/2304 [I 2022-10-10 14:27:05] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:38 [I 2022-10-10 14:27:05] ...trap/engines/engine.py.229: process: 0.86833 | load: 0.00305 [I 2022-10-10 14:27:05] ...trap/engines/engine.py.230: loss: 0.15383 [I 2022-10-10 14:27:05] ...optimizers/trijoint.py.151: total_norm: 19.928396 [I 2022-10-10 14:27:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 330/2304 [I 2022-10-10 14:27:14] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:35 [I 2022-10-10 14:27:14] ...trap/engines/engine.py.229: process: 0.86653 | load: 0.00328 [I 2022-10-10 14:27:14] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-10 14:27:14] ...optimizers/trijoint.py.151: total_norm: 42.248268 [I 2022-10-10 14:27:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 340/2304 [I 2022-10-10 14:27:22] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:24 [I 2022-10-10 14:27:22] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00328 [I 2022-10-10 14:27:22] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-10 14:27:22] ...optimizers/trijoint.py.151: total_norm: 15.628671 [I 2022-10-10 14:27:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 350/2304 [I 2022-10-10 14:27:31] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:27:53 [I 2022-10-10 14:27:31] ...trap/engines/engine.py.229: process: 0.83800 | load: 0.00309 [I 2022-10-10 14:27:31] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-10 14:27:31] ...optimizers/trijoint.py.151: total_norm: 18.714371 [I 2022-10-10 14:27:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 360/2304 [I 2022-10-10 14:27:40] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:28:03 [I 2022-10-10 14:27:40] ...trap/engines/engine.py.229: process: 0.87127 | load: 0.00309 [I 2022-10-10 14:27:40] ...trap/engines/engine.py.230: loss: 0.11286 [I 2022-10-10 14:27:40] ...optimizers/trijoint.py.151: total_norm: 8.770493 [I 2022-10-10 14:27:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 370/2304 [I 2022-10-10 14:27:48] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:50 [I 2022-10-10 14:27:48] ...trap/engines/engine.py.229: process: 0.87390 | load: 0.00311 [I 2022-10-10 14:27:48] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-10 14:27:48] ...optimizers/trijoint.py.151: total_norm: 12.812894 [I 2022-10-10 14:27:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 380/2304 [I 2022-10-10 14:27:57] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:27:56 [I 2022-10-10 14:27:57] ...trap/engines/engine.py.229: process: 0.88828 | load: 0.00317 [I 2022-10-10 14:27:57] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-10 14:27:57] ...optimizers/trijoint.py.151: total_norm: 11.329653 [I 2022-10-10 14:28:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 390/2304 [I 2022-10-10 14:28:06] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:54 [I 2022-10-10 14:28:06] ...trap/engines/engine.py.229: process: 0.87907 | load: 0.00317 [I 2022-10-10 14:28:06] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-10 14:28:06] ...optimizers/trijoint.py.151: total_norm: 9.197408 [I 2022-10-10 14:28:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 400/2304 [I 2022-10-10 14:28:15] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:27:32 [I 2022-10-10 14:28:15] ...trap/engines/engine.py.229: process: 0.87717 | load: 0.00321 [I 2022-10-10 14:28:15] ...trap/engines/engine.py.230: loss: 0.11875 [I 2022-10-10 14:28:15] ...optimizers/trijoint.py.151: total_norm: 13.836976 [I 2022-10-10 14:28:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 410/2304 [I 2022-10-10 14:28:23] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:27:19 [I 2022-10-10 14:28:23] ...trap/engines/engine.py.229: process: 0.86844 | load: 0.00316 [I 2022-10-10 14:28:23] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-10 14:28:23] ...optimizers/trijoint.py.151: total_norm: 14.968422 [I 2022-10-10 14:28:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 420/2304 [I 2022-10-10 14:28:32] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:27:17 [I 2022-10-10 14:28:32] ...trap/engines/engine.py.229: process: 0.87094 | load: 0.00320 [I 2022-10-10 14:28:32] ...trap/engines/engine.py.230: loss: 0.12380 [I 2022-10-10 14:28:32] ...optimizers/trijoint.py.151: total_norm: 16.797405 [I 2022-10-10 14:28:41] ...trap/engines/engine.py.224: train: epoch 75 | batch 430/2304 [I 2022-10-10 14:28:41] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:15 [I 2022-10-10 14:28:41] ...trap/engines/engine.py.229: process: 0.86647 | load: 0.00306 [I 2022-10-10 14:28:41] ...trap/engines/engine.py.230: loss: 0.10467 [I 2022-10-10 14:28:41] ...optimizers/trijoint.py.151: total_norm: 10.819976 [I 2022-10-10 14:28:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 440/2304 [I 2022-10-10 14:28:49] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:57 [I 2022-10-10 14:28:49] ...trap/engines/engine.py.229: process: 0.87212 | load: 0.00308 [I 2022-10-10 14:28:49] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-10 14:28:49] ...optimizers/trijoint.py.151: total_norm: 10.694776 [I 2022-10-10 14:28:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 450/2304 [I 2022-10-10 14:28:58] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:53 [I 2022-10-10 14:28:58] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00325 [I 2022-10-10 14:28:58] ...trap/engines/engine.py.230: loss: 0.21117 [I 2022-10-10 14:28:58] ...optimizers/trijoint.py.151: total_norm: 13.794650 [I 2022-10-10 14:29:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 460/2304 [I 2022-10-10 14:29:07] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:39 [I 2022-10-10 14:29:07] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00308 [I 2022-10-10 14:29:07] ...trap/engines/engine.py.230: loss: 0.09566 [I 2022-10-10 14:29:07] ...optimizers/trijoint.py.151: total_norm: 13.740101 [I 2022-10-10 14:29:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 470/2304 [I 2022-10-10 14:29:15] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:24 [I 2022-10-10 14:29:15] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00313 [I 2022-10-10 14:29:15] ...trap/engines/engine.py.230: loss: 0.16421 [I 2022-10-10 14:29:15] ...optimizers/trijoint.py.151: total_norm: 10.720681 [I 2022-10-10 14:29:24] ...trap/engines/engine.py.224: train: epoch 75 | batch 480/2304 [I 2022-10-10 14:29:24] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:16 [I 2022-10-10 14:29:24] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00328 [I 2022-10-10 14:29:24] ...trap/engines/engine.py.230: loss: 0.16234 [I 2022-10-10 14:29:24] ...optimizers/trijoint.py.151: total_norm: 18.839121 [I 2022-10-10 14:29:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 490/2304 [I 2022-10-10 14:29:33] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:12 [I 2022-10-10 14:29:33] ...trap/engines/engine.py.229: process: 0.88141 | load: 0.00316 [I 2022-10-10 14:29:33] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-10 14:29:33] ...optimizers/trijoint.py.151: total_norm: 11.279298 [I 2022-10-10 14:29:41] ...trap/engines/engine.py.224: train: epoch 75 | batch 500/2304 [I 2022-10-10 14:29:41] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:56 [I 2022-10-10 14:29:41] ...trap/engines/engine.py.229: process: 0.86757 | load: 0.00312 [I 2022-10-10 14:29:41] ...trap/engines/engine.py.230: loss: 0.08355 [I 2022-10-10 14:29:41] ...optimizers/trijoint.py.151: total_norm: 10.105869 [I 2022-10-10 14:29:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 510/2304 [I 2022-10-10 14:29:50] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:40 [I 2022-10-10 14:29:50] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00313 [I 2022-10-10 14:29:50] ...trap/engines/engine.py.230: loss: 0.21231 [I 2022-10-10 14:29:50] ...optimizers/trijoint.py.151: total_norm: 67.015503 [I 2022-10-10 14:29:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 520/2304 [I 2022-10-10 14:29:59] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:38 [I 2022-10-10 14:29:59] ...trap/engines/engine.py.229: process: 0.86500 | load: 0.00310 [I 2022-10-10 14:29:59] ...trap/engines/engine.py.230: loss: 0.14255 [I 2022-10-10 14:29:59] ...optimizers/trijoint.py.151: total_norm: 17.317211 [I 2022-10-10 14:30:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 530/2304 [I 2022-10-10 14:30:07] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:38 [I 2022-10-10 14:30:07] ...trap/engines/engine.py.229: process: 0.86985 | load: 0.00308 [I 2022-10-10 14:30:07] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-10 14:30:07] ...optimizers/trijoint.py.151: total_norm: 19.312428 [I 2022-10-10 14:30:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 540/2304 [I 2022-10-10 14:30:16] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:30 [I 2022-10-10 14:30:16] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00310 [I 2022-10-10 14:30:16] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-10 14:30:16] ...optimizers/trijoint.py.151: total_norm: 8.160173 [I 2022-10-10 14:30:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 550/2304 [I 2022-10-10 14:30:25] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:13 [I 2022-10-10 14:30:25] ...trap/engines/engine.py.229: process: 0.85961 | load: 0.00307 [I 2022-10-10 14:30:25] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-10 14:30:25] ...optimizers/trijoint.py.151: total_norm: 17.657099 [I 2022-10-10 14:30:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 560/2304 [I 2022-10-10 14:30:33] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:03 [I 2022-10-10 14:30:33] ...trap/engines/engine.py.229: process: 0.86943 | load: 0.00308 [I 2022-10-10 14:30:33] ...trap/engines/engine.py.230: loss: 0.17644 [I 2022-10-10 14:30:33] ...optimizers/trijoint.py.151: total_norm: 25.734165 [I 2022-10-10 14:30:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 570/2304 [I 2022-10-10 14:30:42] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:51 [I 2022-10-10 14:30:42] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00304 [I 2022-10-10 14:30:42] ...trap/engines/engine.py.230: loss: 0.16618 [I 2022-10-10 14:30:42] ...optimizers/trijoint.py.151: total_norm: 19.529919 [I 2022-10-10 14:30:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 580/2304 [I 2022-10-10 14:30:51] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:44 [I 2022-10-10 14:30:51] ...trap/engines/engine.py.229: process: 0.86658 | load: 0.00296 [I 2022-10-10 14:30:51] ...trap/engines/engine.py.230: loss: 0.12303 [I 2022-10-10 14:30:51] ...optimizers/trijoint.py.151: total_norm: 9.184450 [I 2022-10-10 14:30:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 590/2304 [I 2022-10-10 14:30:59] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:42 [I 2022-10-10 14:30:59] ...trap/engines/engine.py.229: process: 0.89795 | load: 0.00317 [I 2022-10-10 14:30:59] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-10 14:30:59] ...optimizers/trijoint.py.151: total_norm: 11.888121 [I 2022-10-10 14:31:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 600/2304 [I 2022-10-10 14:31:08] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:14 [I 2022-10-10 14:31:08] ...trap/engines/engine.py.229: process: 0.85863 | load: 0.00326 [I 2022-10-10 14:31:08] ...trap/engines/engine.py.230: loss: 0.08547 [I 2022-10-10 14:31:08] ...optimizers/trijoint.py.151: total_norm: 13.871232 [I 2022-10-10 14:31:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 610/2304 [I 2022-10-10 14:31:16] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:19 [I 2022-10-10 14:31:16] ...trap/engines/engine.py.229: process: 0.86600 | load: 0.00296 [I 2022-10-10 14:31:16] ...trap/engines/engine.py.230: loss: 0.15653 [I 2022-10-10 14:31:16] ...optimizers/trijoint.py.151: total_norm: 11.446429 [I 2022-10-10 14:31:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 620/2304 [I 2022-10-10 14:31:25] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:24:04 [I 2022-10-10 14:31:25] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00290 [I 2022-10-10 14:31:25] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-10 14:31:25] ...optimizers/trijoint.py.151: total_norm: 12.135264 [I 2022-10-10 14:31:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 630/2304 [I 2022-10-10 14:31:34] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:56 [I 2022-10-10 14:31:34] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00292 [I 2022-10-10 14:31:34] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-10 14:31:34] ...optimizers/trijoint.py.151: total_norm: 20.159088 [I 2022-10-10 14:31:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 640/2304 [I 2022-10-10 14:31:42] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:41 [I 2022-10-10 14:31:42] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00292 [I 2022-10-10 14:31:42] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-10 14:31:42] ...optimizers/trijoint.py.151: total_norm: 14.694655 [I 2022-10-10 14:31:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 650/2304 [I 2022-10-10 14:31:51] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:33 [I 2022-10-10 14:31:51] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00296 [I 2022-10-10 14:31:51] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-10 14:31:51] ...optimizers/trijoint.py.151: total_norm: 16.543383 [I 2022-10-10 14:31:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 660/2304 [I 2022-10-10 14:31:59] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:21 [I 2022-10-10 14:31:59] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00295 [I 2022-10-10 14:31:59] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-10 14:31:59] ...optimizers/trijoint.py.151: total_norm: 9.521990 [I 2022-10-10 14:32:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 670/2304 [I 2022-10-10 14:32:08] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:15 [I 2022-10-10 14:32:08] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00301 [I 2022-10-10 14:32:08] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-10 14:32:08] ...optimizers/trijoint.py.151: total_norm: 16.366581 [I 2022-10-10 14:32:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 680/2304 [I 2022-10-10 14:32:17] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:23:07 [I 2022-10-10 14:32:17] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00298 [I 2022-10-10 14:32:17] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-10 14:32:17] ...optimizers/trijoint.py.151: total_norm: 8.724962 [I 2022-10-10 14:32:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 690/2304 [I 2022-10-10 14:32:25] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:22:59 [I 2022-10-10 14:32:25] ...trap/engines/engine.py.229: process: 0.84851 | load: 0.00299 [I 2022-10-10 14:32:25] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-10 14:32:25] ...optimizers/trijoint.py.151: total_norm: 11.930915 [I 2022-10-10 14:32:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 700/2304 [I 2022-10-10 14:32:34] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:53 [I 2022-10-10 14:32:34] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00298 [I 2022-10-10 14:32:34] ...trap/engines/engine.py.230: loss: 0.11646 [I 2022-10-10 14:32:34] ...optimizers/trijoint.py.151: total_norm: 14.445730 [I 2022-10-10 14:32:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 710/2304 [I 2022-10-10 14:32:42] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:35 [I 2022-10-10 14:32:42] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00298 [I 2022-10-10 14:32:42] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-10 14:32:42] ...optimizers/trijoint.py.151: total_norm: 36.734283 [I 2022-10-10 14:32:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 720/2304 [I 2022-10-10 14:32:51] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:34 [I 2022-10-10 14:32:51] ...trap/engines/engine.py.229: process: 0.85401 | load: 0.00297 [I 2022-10-10 14:32:51] ...trap/engines/engine.py.230: loss: 0.13709 [I 2022-10-10 14:32:51] ...optimizers/trijoint.py.151: total_norm: 26.899601 [I 2022-10-10 14:32:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 730/2304 [I 2022-10-10 14:32:59] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:26 [I 2022-10-10 14:32:59] ...trap/engines/engine.py.229: process: 0.86281 | load: 0.00295 [I 2022-10-10 14:32:59] ...trap/engines/engine.py.230: loss: 0.08870 [I 2022-10-10 14:32:59] ...optimizers/trijoint.py.151: total_norm: 12.689259 [I 2022-10-10 14:33:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 740/2304 [I 2022-10-10 14:33:08] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:22:02 [I 2022-10-10 14:33:08] ...trap/engines/engine.py.229: process: 0.83176 | load: 0.00296 [I 2022-10-10 14:33:08] ...trap/engines/engine.py.230: loss: 0.07979 [I 2022-10-10 14:33:08] ...optimizers/trijoint.py.151: total_norm: 14.586455 [I 2022-10-10 14:33:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 750/2304 [I 2022-10-10 14:33:16] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:22:03 [I 2022-10-10 14:33:16] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00295 [I 2022-10-10 14:33:16] ...trap/engines/engine.py.230: loss: 0.10584 [I 2022-10-10 14:33:16] ...optimizers/trijoint.py.151: total_norm: 27.729097 [I 2022-10-10 14:33:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 760/2304 [I 2022-10-10 14:33:25] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:53 [I 2022-10-10 14:33:25] ...trap/engines/engine.py.229: process: 0.83828 | load: 0.00295 [I 2022-10-10 14:33:25] ...trap/engines/engine.py.230: loss: 0.09188 [I 2022-10-10 14:33:25] ...optimizers/trijoint.py.151: total_norm: 7.266632 [I 2022-10-10 14:33:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 770/2304 [I 2022-10-10 14:33:34] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:46 [I 2022-10-10 14:33:34] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00298 [I 2022-10-10 14:33:34] ...trap/engines/engine.py.230: loss: 0.14078 [I 2022-10-10 14:33:34] ...optimizers/trijoint.py.151: total_norm: 16.627884 [I 2022-10-10 14:33:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 780/2304 [I 2022-10-10 14:33:42] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:57 [I 2022-10-10 14:33:42] ...trap/engines/engine.py.229: process: 0.87385 | load: 0.00312 [I 2022-10-10 14:33:42] ...trap/engines/engine.py.230: loss: 0.17689 [I 2022-10-10 14:33:42] ...optimizers/trijoint.py.151: total_norm: 57.645424 [I 2022-10-10 14:33:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 790/2304 [I 2022-10-10 14:33:51] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:45 [I 2022-10-10 14:33:51] ...trap/engines/engine.py.229: process: 0.86553 | load: 0.00312 [I 2022-10-10 14:33:51] ...trap/engines/engine.py.230: loss: 0.17744 [I 2022-10-10 14:33:51] ...optimizers/trijoint.py.151: total_norm: 18.172693 [I 2022-10-10 14:34:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 800/2304 [I 2022-10-10 14:34:00] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:44 [I 2022-10-10 14:34:00] ...trap/engines/engine.py.229: process: 0.86454 | load: 0.00308 [I 2022-10-10 14:34:00] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-10 14:34:00] ...optimizers/trijoint.py.151: total_norm: 14.362203 [I 2022-10-10 14:34:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 810/2304 [I 2022-10-10 14:34:08] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:28 [I 2022-10-10 14:34:08] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00311 [I 2022-10-10 14:34:08] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-10 14:34:08] ...optimizers/trijoint.py.151: total_norm: 10.717253 [I 2022-10-10 14:34:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 820/2304 [I 2022-10-10 14:34:17] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:21:16 [I 2022-10-10 14:34:17] ...trap/engines/engine.py.229: process: 0.85886 | load: 0.00314 [I 2022-10-10 14:34:17] ...trap/engines/engine.py.230: loss: 0.13003 [I 2022-10-10 14:34:17] ...optimizers/trijoint.py.151: total_norm: 24.427902 [I 2022-10-10 14:34:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 830/2304 [I 2022-10-10 14:34:26] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:21:14 [I 2022-10-10 14:34:26] ...trap/engines/engine.py.229: process: 0.84606 | load: 0.00312 [I 2022-10-10 14:34:26] ...trap/engines/engine.py.230: loss: 0.15809 [I 2022-10-10 14:34:26] ...optimizers/trijoint.py.151: total_norm: 25.329100 [I 2022-10-10 14:34:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 840/2304 [I 2022-10-10 14:34:34] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:21:04 [I 2022-10-10 14:34:34] ...trap/engines/engine.py.229: process: 0.86305 | load: 0.00307 [I 2022-10-10 14:34:34] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-10 14:34:34] ...optimizers/trijoint.py.151: total_norm: 20.411224 [I 2022-10-10 14:34:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 850/2304 [I 2022-10-10 14:34:43] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:57 [I 2022-10-10 14:34:43] ...trap/engines/engine.py.229: process: 0.87528 | load: 0.00315 [I 2022-10-10 14:34:43] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-10 14:34:43] ...optimizers/trijoint.py.151: total_norm: 12.866565 [I 2022-10-10 14:34:52] ...trap/engines/engine.py.224: train: epoch 75 | batch 860/2304 [I 2022-10-10 14:34:52] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:51 [I 2022-10-10 14:34:52] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00311 [I 2022-10-10 14:34:52] ...trap/engines/engine.py.230: loss: 0.13837 [I 2022-10-10 14:34:52] ...optimizers/trijoint.py.151: total_norm: 8.786135 [I 2022-10-10 14:35:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 870/2304 [I 2022-10-10 14:35:00] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:42 [I 2022-10-10 14:35:00] ...trap/engines/engine.py.229: process: 0.87333 | load: 0.00320 [I 2022-10-10 14:35:00] ...trap/engines/engine.py.230: loss: 0.19482 [I 2022-10-10 14:35:00] ...optimizers/trijoint.py.151: total_norm: 16.750683 [I 2022-10-10 14:35:09] ...trap/engines/engine.py.224: train: epoch 75 | batch 880/2304 [I 2022-10-10 14:35:09] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:40 [I 2022-10-10 14:35:09] ...trap/engines/engine.py.229: process: 0.88138 | load: 0.00308 [I 2022-10-10 14:35:09] ...trap/engines/engine.py.230: loss: 0.12315 [I 2022-10-10 14:35:09] ...optimizers/trijoint.py.151: total_norm: 16.530720 [I 2022-10-10 14:35:18] ...trap/engines/engine.py.224: train: epoch 75 | batch 890/2304 [I 2022-10-10 14:35:18] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:24 [I 2022-10-10 14:35:18] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00315 [I 2022-10-10 14:35:18] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-10 14:35:18] ...optimizers/trijoint.py.151: total_norm: 16.039127 [I 2022-10-10 14:35:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 900/2304 [I 2022-10-10 14:35:26] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:20:24 [I 2022-10-10 14:35:26] ...trap/engines/engine.py.229: process: 0.89284 | load: 0.00313 [I 2022-10-10 14:35:26] ...trap/engines/engine.py.230: loss: 0.17273 [I 2022-10-10 14:35:26] ...optimizers/trijoint.py.151: total_norm: 16.440409 [I 2022-10-10 14:35:35] ...trap/engines/engine.py.224: train: epoch 75 | batch 910/2304 [I 2022-10-10 14:35:35] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:20:09 [I 2022-10-10 14:35:35] ...trap/engines/engine.py.229: process: 0.87085 | load: 0.00308 [I 2022-10-10 14:35:35] ...trap/engines/engine.py.230: loss: 0.22253 [I 2022-10-10 14:35:35] ...optimizers/trijoint.py.151: total_norm: 47.398781 [I 2022-10-10 14:35:44] ...trap/engines/engine.py.224: train: epoch 75 | batch 920/2304 [I 2022-10-10 14:35:44] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:20:01 [I 2022-10-10 14:35:44] ...trap/engines/engine.py.229: process: 0.86817 | load: 0.00317 [I 2022-10-10 14:35:44] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-10 14:35:44] ...optimizers/trijoint.py.151: total_norm: 9.839964 [I 2022-10-10 14:35:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 930/2304 [I 2022-10-10 14:35:53] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:54 [I 2022-10-10 14:35:53] ...trap/engines/engine.py.229: process: 0.87269 | load: 0.00315 [I 2022-10-10 14:35:53] ...trap/engines/engine.py.230: loss: 0.12358 [I 2022-10-10 14:35:53] ...optimizers/trijoint.py.151: total_norm: 9.467815 [I 2022-10-10 14:36:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 940/2304 [I 2022-10-10 14:36:01] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:52 [I 2022-10-10 14:36:01] ...trap/engines/engine.py.229: process: 0.90396 | load: 0.00310 [I 2022-10-10 14:36:01] ...trap/engines/engine.py.230: loss: 0.15656 [I 2022-10-10 14:36:01] ...optimizers/trijoint.py.151: total_norm: 13.049192 [I 2022-10-10 14:36:10] ...trap/engines/engine.py.224: train: epoch 75 | batch 950/2304 [I 2022-10-10 14:36:10] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:38 [I 2022-10-10 14:36:10] ...trap/engines/engine.py.229: process: 0.86469 | load: 0.00323 [I 2022-10-10 14:36:10] ...trap/engines/engine.py.230: loss: 0.14984 [I 2022-10-10 14:36:10] ...optimizers/trijoint.py.151: total_norm: 27.185915 [I 2022-10-10 14:36:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 960/2304 [I 2022-10-10 14:36:19] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:19:24 [I 2022-10-10 14:36:19] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00317 [I 2022-10-10 14:36:19] ...trap/engines/engine.py.230: loss: 0.10444 [I 2022-10-10 14:36:19] ...optimizers/trijoint.py.151: total_norm: 9.367907 [I 2022-10-10 14:36:27] ...trap/engines/engine.py.224: train: epoch 75 | batch 970/2304 [I 2022-10-10 14:36:27] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:19:17 [I 2022-10-10 14:36:27] ...trap/engines/engine.py.229: process: 0.87032 | load: 0.00320 [I 2022-10-10 14:36:27] ...trap/engines/engine.py.230: loss: 0.16273 [I 2022-10-10 14:36:27] ...optimizers/trijoint.py.151: total_norm: 11.564344 [I 2022-10-10 14:36:36] ...trap/engines/engine.py.224: train: epoch 75 | batch 980/2304 [I 2022-10-10 14:36:36] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:19:08 [I 2022-10-10 14:36:36] ...trap/engines/engine.py.229: process: 0.87217 | load: 0.00326 [I 2022-10-10 14:36:36] ...trap/engines/engine.py.230: loss: 0.16416 [I 2022-10-10 14:36:36] ...optimizers/trijoint.py.151: total_norm: 9.532125 [I 2022-10-10 14:36:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 990/2304 [I 2022-10-10 14:36:45] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:56 [I 2022-10-10 14:36:45] ...trap/engines/engine.py.229: process: 0.85849 | load: 0.00318 [I 2022-10-10 14:36:45] ...trap/engines/engine.py.230: loss: 0.14392 [I 2022-10-10 14:36:45] ...optimizers/trijoint.py.151: total_norm: 28.534359 [I 2022-10-10 14:36:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 1000/2304 [I 2022-10-10 14:36:53] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:53 [I 2022-10-10 14:36:53] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00329 [I 2022-10-10 14:36:53] ...trap/engines/engine.py.230: loss: 0.12966 [I 2022-10-10 14:36:53] ...optimizers/trijoint.py.151: total_norm: 98.003212 [I 2022-10-10 14:37:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 1010/2304 [I 2022-10-10 14:37:02] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:46 [I 2022-10-10 14:37:02] ...trap/engines/engine.py.229: process: 0.87921 | load: 0.00326 [I 2022-10-10 14:37:02] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-10 14:37:02] ...optimizers/trijoint.py.151: total_norm: 18.954937 [I 2022-10-10 14:37:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1020/2304 [I 2022-10-10 14:37:11] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:37 [I 2022-10-10 14:37:11] ...trap/engines/engine.py.229: process: 0.86982 | load: 0.00317 [I 2022-10-10 14:37:11] ...trap/engines/engine.py.230: loss: 0.12593 [I 2022-10-10 14:37:11] ...optimizers/trijoint.py.151: total_norm: 29.505669 [I 2022-10-10 14:37:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1030/2304 [I 2022-10-10 14:37:20] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:18:29 [I 2022-10-10 14:37:20] ...trap/engines/engine.py.229: process: 0.87781 | load: 0.00330 [I 2022-10-10 14:37:20] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-10 14:37:20] ...optimizers/trijoint.py.151: total_norm: 9.858829 [I 2022-10-10 14:37:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1040/2304 [I 2022-10-10 14:37:28] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:18:18 [I 2022-10-10 14:37:28] ...trap/engines/engine.py.229: process: 0.87170 | load: 0.00321 [I 2022-10-10 14:37:28] ...trap/engines/engine.py.230: loss: 0.15327 [I 2022-10-10 14:37:28] ...optimizers/trijoint.py.151: total_norm: 11.765712 [I 2022-10-10 14:37:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 1050/2304 [I 2022-10-10 14:37:37] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:55 [I 2022-10-10 14:37:37] ...trap/engines/engine.py.229: process: 0.86786 | load: 0.00305 [I 2022-10-10 14:37:37] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-10 14:37:37] ...optimizers/trijoint.py.151: total_norm: 27.808319 [I 2022-10-10 14:37:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 1060/2304 [I 2022-10-10 14:37:46] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:43 [I 2022-10-10 14:37:46] ...trap/engines/engine.py.229: process: 0.84228 | load: 0.00301 [I 2022-10-10 14:37:46] ...trap/engines/engine.py.230: loss: 0.19192 [I 2022-10-10 14:37:46] ...optimizers/trijoint.py.151: total_norm: 23.809862 [I 2022-10-10 14:37:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1070/2304 [I 2022-10-10 14:37:54] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:33 [I 2022-10-10 14:37:54] ...trap/engines/engine.py.229: process: 0.84750 | load: 0.00328 [I 2022-10-10 14:37:54] ...trap/engines/engine.py.230: loss: 0.13013 [I 2022-10-10 14:37:54] ...optimizers/trijoint.py.151: total_norm: 11.025431 [I 2022-10-10 14:38:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 1080/2304 [I 2022-10-10 14:38:03] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:25 [I 2022-10-10 14:38:03] ...trap/engines/engine.py.229: process: 0.86865 | load: 0.00302 [I 2022-10-10 14:38:03] ...trap/engines/engine.py.230: loss: 0.09849 [I 2022-10-10 14:38:03] ...optimizers/trijoint.py.151: total_norm: 12.628779 [I 2022-10-10 14:38:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1090/2304 [I 2022-10-10 14:38:11] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:15 [I 2022-10-10 14:38:11] ...trap/engines/engine.py.229: process: 0.84554 | load: 0.00304 [I 2022-10-10 14:38:11] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-10 14:38:11] ...optimizers/trijoint.py.151: total_norm: 22.129007 [I 2022-10-10 14:38:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1100/2304 [I 2022-10-10 14:38:20] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:08 [I 2022-10-10 14:38:20] ...trap/engines/engine.py.229: process: 0.86042 | load: 0.00304 [I 2022-10-10 14:38:20] ...trap/engines/engine.py.230: loss: 0.10739 [I 2022-10-10 14:38:20] ...optimizers/trijoint.py.151: total_norm: 9.912454 [I 2022-10-10 14:38:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1110/2304 [I 2022-10-10 14:38:28] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:57 [I 2022-10-10 14:38:28] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00293 [I 2022-10-10 14:38:28] ...trap/engines/engine.py.230: loss: 0.17832 [I 2022-10-10 14:38:28] ...optimizers/trijoint.py.151: total_norm: 10.532168 [I 2022-10-10 14:38:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 1120/2304 [I 2022-10-10 14:38:37] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:50 [I 2022-10-10 14:38:37] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00294 [I 2022-10-10 14:38:37] ...trap/engines/engine.py.230: loss: 0.09970 [I 2022-10-10 14:38:37] ...optimizers/trijoint.py.151: total_norm: 25.631283 [I 2022-10-10 14:38:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 1130/2304 [I 2022-10-10 14:38:45] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:35 [I 2022-10-10 14:38:45] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00294 [I 2022-10-10 14:38:45] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-10 14:38:45] ...optimizers/trijoint.py.151: total_norm: 16.748508 [I 2022-10-10 14:38:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1140/2304 [I 2022-10-10 14:38:54] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:34 [I 2022-10-10 14:38:54] ...trap/engines/engine.py.229: process: 0.86821 | load: 0.00295 [I 2022-10-10 14:38:54] ...trap/engines/engine.py.230: loss: 0.14856 [I 2022-10-10 14:38:54] ...optimizers/trijoint.py.151: total_norm: 10.464789 [I 2022-10-10 14:39:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 1150/2304 [I 2022-10-10 14:39:02] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:22 [I 2022-10-10 14:39:02] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00296 [I 2022-10-10 14:39:02] ...trap/engines/engine.py.230: loss: 0.14797 [I 2022-10-10 14:39:02] ...optimizers/trijoint.py.151: total_norm: 32.092117 [I 2022-10-10 14:39:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1160/2304 [I 2022-10-10 14:39:11] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:14 [I 2022-10-10 14:39:11] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00295 [I 2022-10-10 14:39:11] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-10 14:39:11] ...optimizers/trijoint.py.151: total_norm: 26.190516 [I 2022-10-10 14:39:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1170/2304 [I 2022-10-10 14:39:20] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:05 [I 2022-10-10 14:39:20] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00295 [I 2022-10-10 14:39:20] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-10 14:39:20] ...optimizers/trijoint.py.151: total_norm: 21.410028 [I 2022-10-10 14:39:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1180/2304 [I 2022-10-10 14:39:28] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:16:00 [I 2022-10-10 14:39:28] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00295 [I 2022-10-10 14:39:28] ...trap/engines/engine.py.230: loss: 0.08939 [I 2022-10-10 14:39:28] ...optimizers/trijoint.py.151: total_norm: 8.333488 [I 2022-10-10 14:39:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 1190/2304 [I 2022-10-10 14:39:37] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:52 [I 2022-10-10 14:39:37] ...trap/engines/engine.py.229: process: 0.85718 | load: 0.00294 [I 2022-10-10 14:39:37] ...trap/engines/engine.py.230: loss: 0.15835 [I 2022-10-10 14:39:37] ...optimizers/trijoint.py.151: total_norm: 15.032005 [I 2022-10-10 14:39:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 1200/2304 [I 2022-10-10 14:39:45] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:45 [I 2022-10-10 14:39:45] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00297 [I 2022-10-10 14:39:45] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-10 14:39:45] ...optimizers/trijoint.py.151: total_norm: 13.173809 [I 2022-10-10 14:39:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1210/2304 [I 2022-10-10 14:39:54] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:28 [I 2022-10-10 14:39:54] ...trap/engines/engine.py.229: process: 0.82603 | load: 0.00298 [I 2022-10-10 14:39:54] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-10 14:39:54] ...optimizers/trijoint.py.151: total_norm: 16.324017 [I 2022-10-10 14:40:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 1220/2304 [I 2022-10-10 14:40:02] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:22 [I 2022-10-10 14:40:02] ...trap/engines/engine.py.229: process: 0.85081 | load: 0.00298 [I 2022-10-10 14:40:02] ...trap/engines/engine.py.230: loss: 0.17165 [I 2022-10-10 14:40:02] ...optimizers/trijoint.py.151: total_norm: 21.610933 [I 2022-10-10 14:40:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1230/2304 [I 2022-10-10 14:40:11] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:16 [I 2022-10-10 14:40:11] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00292 [I 2022-10-10 14:40:11] ...trap/engines/engine.py.230: loss: 0.25081 [I 2022-10-10 14:40:11] ...optimizers/trijoint.py.151: total_norm: 12.976804 [I 2022-10-10 14:40:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1240/2304 [I 2022-10-10 14:40:20] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:22 [I 2022-10-10 14:40:20] ...trap/engines/engine.py.229: process: 0.87234 | load: 0.00318 [I 2022-10-10 14:40:20] ...trap/engines/engine.py.230: loss: 0.20808 [I 2022-10-10 14:40:20] ...optimizers/trijoint.py.151: total_norm: 26.319105 [I 2022-10-10 14:40:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1250/2304 [I 2022-10-10 14:40:28] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:10 [I 2022-10-10 14:40:28] ...trap/engines/engine.py.229: process: 0.85337 | load: 0.00316 [I 2022-10-10 14:40:28] ...trap/engines/engine.py.230: loss: 0.12627 [I 2022-10-10 14:40:28] ...optimizers/trijoint.py.151: total_norm: 11.085975 [I 2022-10-10 14:40:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 1260/2304 [I 2022-10-10 14:40:37] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:15:08 [I 2022-10-10 14:40:37] ...trap/engines/engine.py.229: process: 0.88745 | load: 0.00321 [I 2022-10-10 14:40:37] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-10 14:40:37] ...optimizers/trijoint.py.151: total_norm: 17.716583 [I 2022-10-10 14:40:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 1270/2304 [I 2022-10-10 14:40:46] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:15:02 [I 2022-10-10 14:40:46] ...trap/engines/engine.py.229: process: 0.87473 | load: 0.00309 [I 2022-10-10 14:40:46] ...trap/engines/engine.py.230: loss: 0.23396 [I 2022-10-10 14:40:46] ...optimizers/trijoint.py.151: total_norm: 24.844124 [I 2022-10-10 14:40:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1280/2304 [I 2022-10-10 14:40:54] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:14:47 [I 2022-10-10 14:40:54] ...trap/engines/engine.py.229: process: 0.86749 | load: 0.00321 [I 2022-10-10 14:40:54] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-10 14:40:54] ...optimizers/trijoint.py.151: total_norm: 10.316315 [I 2022-10-10 14:41:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 1290/2304 [I 2022-10-10 14:41:03] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:42 [I 2022-10-10 14:41:03] ...trap/engines/engine.py.229: process: 0.86268 | load: 0.00315 [I 2022-10-10 14:41:03] ...trap/engines/engine.py.230: loss: 0.08550 [I 2022-10-10 14:41:03] ...optimizers/trijoint.py.151: total_norm: 8.040550 [I 2022-10-10 14:41:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 1300/2304 [I 2022-10-10 14:41:12] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:32 [I 2022-10-10 14:41:12] ...trap/engines/engine.py.229: process: 0.87641 | load: 0.00314 [I 2022-10-10 14:41:12] ...trap/engines/engine.py.230: loss: 0.13959 [I 2022-10-10 14:41:12] ...optimizers/trijoint.py.151: total_norm: 62.347618 [I 2022-10-10 14:41:21] ...trap/engines/engine.py.224: train: epoch 75 | batch 1310/2304 [I 2022-10-10 14:41:21] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:26 [I 2022-10-10 14:41:21] ...trap/engines/engine.py.229: process: 0.88074 | load: 0.00310 [I 2022-10-10 14:41:21] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-10 14:41:21] ...optimizers/trijoint.py.151: total_norm: 10.063745 [I 2022-10-10 14:41:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 1320/2304 [I 2022-10-10 14:41:29] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:14:16 [I 2022-10-10 14:41:29] ...trap/engines/engine.py.229: process: 0.88280 | load: 0.00344 [I 2022-10-10 14:41:29] ...trap/engines/engine.py.230: loss: 0.08748 [I 2022-10-10 14:41:29] ...optimizers/trijoint.py.151: total_norm: 9.472239 [I 2022-10-10 14:41:38] ...trap/engines/engine.py.224: train: epoch 75 | batch 1330/2304 [I 2022-10-10 14:41:38] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:14:10 [I 2022-10-10 14:41:38] ...trap/engines/engine.py.229: process: 0.86542 | load: 0.00323 [I 2022-10-10 14:41:38] ...trap/engines/engine.py.230: loss: 0.14408 [I 2022-10-10 14:41:38] ...optimizers/trijoint.py.151: total_norm: 11.447716 [I 2022-10-10 14:41:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 1340/2304 [I 2022-10-10 14:41:47] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:51 [I 2022-10-10 14:41:47] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00307 [I 2022-10-10 14:41:47] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-10 14:41:47] ...optimizers/trijoint.py.151: total_norm: 10.397328 [I 2022-10-10 14:41:55] ...trap/engines/engine.py.224: train: epoch 75 | batch 1350/2304 [I 2022-10-10 14:41:55] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:45 [I 2022-10-10 14:41:55] ...trap/engines/engine.py.229: process: 0.86229 | load: 0.00321 [I 2022-10-10 14:41:55] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-10 14:41:55] ...optimizers/trijoint.py.151: total_norm: 12.569075 [I 2022-10-10 14:42:04] ...trap/engines/engine.py.224: train: epoch 75 | batch 1360/2304 [I 2022-10-10 14:42:04] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:39 [I 2022-10-10 14:42:04] ...trap/engines/engine.py.229: process: 0.86490 | load: 0.00310 [I 2022-10-10 14:42:04] ...trap/engines/engine.py.230: loss: 0.14373 [I 2022-10-10 14:42:04] ...optimizers/trijoint.py.151: total_norm: 9.362264 [I 2022-10-10 14:42:13] ...trap/engines/engine.py.224: train: epoch 75 | batch 1370/2304 [I 2022-10-10 14:42:13] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:32 [I 2022-10-10 14:42:13] ...trap/engines/engine.py.229: process: 0.87537 | load: 0.00315 [I 2022-10-10 14:42:13] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-10 14:42:13] ...optimizers/trijoint.py.151: total_norm: 12.251259 [I 2022-10-10 14:42:21] ...trap/engines/engine.py.224: train: epoch 75 | batch 1380/2304 [I 2022-10-10 14:42:21] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:13:21 [I 2022-10-10 14:42:21] ...trap/engines/engine.py.229: process: 0.86366 | load: 0.00308 [I 2022-10-10 14:42:21] ...trap/engines/engine.py.230: loss: 0.13047 [I 2022-10-10 14:42:21] ...optimizers/trijoint.py.151: total_norm: 9.770145 [I 2022-10-10 14:42:30] ...trap/engines/engine.py.224: train: epoch 75 | batch 1390/2304 [I 2022-10-10 14:42:30] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:13:06 [I 2022-10-10 14:42:30] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00312 [I 2022-10-10 14:42:30] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-10 14:42:30] ...optimizers/trijoint.py.151: total_norm: 7.790254 [I 2022-10-10 14:42:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 1400/2304 [I 2022-10-10 14:42:39] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:59 [I 2022-10-10 14:42:39] ...trap/engines/engine.py.229: process: 0.86170 | load: 0.00310 [I 2022-10-10 14:42:39] ...trap/engines/engine.py.230: loss: 0.14043 [I 2022-10-10 14:42:39] ...optimizers/trijoint.py.151: total_norm: 33.616936 [I 2022-10-10 14:42:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 1410/2304 [I 2022-10-10 14:42:47] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:12:56 [I 2022-10-10 14:42:47] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00311 [I 2022-10-10 14:42:47] ...trap/engines/engine.py.230: loss: 0.08103 [I 2022-10-10 14:42:47] ...optimizers/trijoint.py.151: total_norm: 9.674614 [I 2022-10-10 14:42:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 1420/2304 [I 2022-10-10 14:42:56] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:44 [I 2022-10-10 14:42:56] ...trap/engines/engine.py.229: process: 0.86543 | load: 0.00318 [I 2022-10-10 14:42:56] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-10 14:42:56] ...optimizers/trijoint.py.151: total_norm: 10.671403 [I 2022-10-10 14:43:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 1430/2304 [I 2022-10-10 14:43:05] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:35 [I 2022-10-10 14:43:05] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00315 [I 2022-10-10 14:43:05] ...trap/engines/engine.py.230: loss: 0.18414 [I 2022-10-10 14:43:05] ...optimizers/trijoint.py.151: total_norm: 85.309624 [I 2022-10-10 14:43:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1440/2304 [I 2022-10-10 14:43:14] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:12:30 [I 2022-10-10 14:43:14] ...trap/engines/engine.py.229: process: 0.85565 | load: 0.00312 [I 2022-10-10 14:43:14] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-10 14:43:14] ...optimizers/trijoint.py.151: total_norm: 31.609762 [I 2022-10-10 14:43:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 1450/2304 [I 2022-10-10 14:43:22] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:23 [I 2022-10-10 14:43:22] ...trap/engines/engine.py.229: process: 0.86183 | load: 0.00314 [I 2022-10-10 14:43:22] ...trap/engines/engine.py.230: loss: 0.11058 [I 2022-10-10 14:43:22] ...optimizers/trijoint.py.151: total_norm: 9.919840 [I 2022-10-10 14:43:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1460/2304 [I 2022-10-10 14:43:31] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:07 [I 2022-10-10 14:43:31] ...trap/engines/engine.py.229: process: 0.86719 | load: 0.00311 [I 2022-10-10 14:43:31] ...trap/engines/engine.py.230: loss: 0.17436 [I 2022-10-10 14:43:31] ...optimizers/trijoint.py.151: total_norm: 9.405354 [I 2022-10-10 14:43:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1470/2304 [I 2022-10-10 14:43:40] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:12:05 [I 2022-10-10 14:43:40] ...trap/engines/engine.py.229: process: 0.88426 | load: 0.00313 [I 2022-10-10 14:43:40] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-10 14:43:40] ...optimizers/trijoint.py.151: total_norm: 18.490849 [I 2022-10-10 14:43:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 1480/2304 [I 2022-10-10 14:43:48] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:56 [I 2022-10-10 14:43:48] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00313 [I 2022-10-10 14:43:48] ...trap/engines/engine.py.230: loss: 0.14869 [I 2022-10-10 14:43:48] ...optimizers/trijoint.py.151: total_norm: 11.830767 [I 2022-10-10 14:43:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1490/2304 [I 2022-10-10 14:43:57] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:43 [I 2022-10-10 14:43:57] ...trap/engines/engine.py.229: process: 0.86173 | load: 0.00312 [I 2022-10-10 14:43:57] ...trap/engines/engine.py.230: loss: 0.08965 [I 2022-10-10 14:43:57] ...optimizers/trijoint.py.151: total_norm: 10.350821 [I 2022-10-10 14:44:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1500/2304 [I 2022-10-10 14:44:06] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:33 [I 2022-10-10 14:44:06] ...trap/engines/engine.py.229: process: 0.85589 | load: 0.00312 [I 2022-10-10 14:44:06] ...trap/engines/engine.py.230: loss: 0.17667 [I 2022-10-10 14:44:06] ...optimizers/trijoint.py.151: total_norm: 13.637951 [I 2022-10-10 14:44:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1510/2304 [I 2022-10-10 14:44:14] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:22 [I 2022-10-10 14:44:14] ...trap/engines/engine.py.229: process: 0.87069 | load: 0.00301 [I 2022-10-10 14:44:14] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-10 14:44:14] ...optimizers/trijoint.py.151: total_norm: 20.199530 [I 2022-10-10 14:44:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1520/2304 [I 2022-10-10 14:44:23] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:09 [I 2022-10-10 14:44:23] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00301 [I 2022-10-10 14:44:23] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-10 14:44:23] ...optimizers/trijoint.py.151: total_norm: 9.674673 [I 2022-10-10 14:44:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1530/2304 [I 2022-10-10 14:44:31] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:58 [I 2022-10-10 14:44:31] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00301 [I 2022-10-10 14:44:31] ...trap/engines/engine.py.230: loss: 0.11361 [I 2022-10-10 14:44:31] ...optimizers/trijoint.py.151: total_norm: 11.575475 [I 2022-10-10 14:44:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1540/2304 [I 2022-10-10 14:44:40] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:52 [I 2022-10-10 14:44:40] ...trap/engines/engine.py.229: process: 0.85434 | load: 0.00294 [I 2022-10-10 14:44:40] ...trap/engines/engine.py.230: loss: 0.15107 [I 2022-10-10 14:44:40] ...optimizers/trijoint.py.151: total_norm: 17.663483 [I 2022-10-10 14:44:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1550/2304 [I 2022-10-10 14:44:49] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:46 [I 2022-10-10 14:44:49] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00293 [I 2022-10-10 14:44:49] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-10 14:44:49] ...optimizers/trijoint.py.151: total_norm: 15.160460 [I 2022-10-10 14:44:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1560/2304 [I 2022-10-10 14:44:57] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:37 [I 2022-10-10 14:44:57] ...trap/engines/engine.py.229: process: 0.86679 | load: 0.00296 [I 2022-10-10 14:44:57] ...trap/engines/engine.py.230: loss: 0.12641 [I 2022-10-10 14:44:57] ...optimizers/trijoint.py.151: total_norm: 20.389791 [I 2022-10-10 14:45:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1570/2304 [I 2022-10-10 14:45:06] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:30 [I 2022-10-10 14:45:06] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00296 [I 2022-10-10 14:45:06] ...trap/engines/engine.py.230: loss: 0.13971 [I 2022-10-10 14:45:06] ...optimizers/trijoint.py.151: total_norm: 23.056381 [I 2022-10-10 14:45:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1580/2304 [I 2022-10-10 14:45:14] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:19 [I 2022-10-10 14:45:14] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00297 [I 2022-10-10 14:45:14] ...trap/engines/engine.py.230: loss: 0.16359 [I 2022-10-10 14:45:14] ...optimizers/trijoint.py.151: total_norm: 51.468063 [I 2022-10-10 14:45:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1590/2304 [I 2022-10-10 14:45:23] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:06 [I 2022-10-10 14:45:23] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00292 [I 2022-10-10 14:45:23] ...trap/engines/engine.py.230: loss: 0.14248 [I 2022-10-10 14:45:23] ...optimizers/trijoint.py.151: total_norm: 10.483533 [I 2022-10-10 14:45:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 1600/2304 [I 2022-10-10 14:45:32] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:57 [I 2022-10-10 14:45:32] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00293 [I 2022-10-10 14:45:32] ...trap/engines/engine.py.230: loss: 0.15597 [I 2022-10-10 14:45:32] ...optimizers/trijoint.py.151: total_norm: 14.334997 [I 2022-10-10 14:45:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1610/2304 [I 2022-10-10 14:45:40] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:51 [I 2022-10-10 14:45:40] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00299 [I 2022-10-10 14:45:40] ...trap/engines/engine.py.230: loss: 0.13024 [I 2022-10-10 14:45:40] ...optimizers/trijoint.py.151: total_norm: 15.132518 [I 2022-10-10 14:45:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1620/2304 [I 2022-10-10 14:45:49] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:44 [I 2022-10-10 14:45:49] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00294 [I 2022-10-10 14:45:49] ...trap/engines/engine.py.230: loss: 0.12733 [I 2022-10-10 14:45:49] ...optimizers/trijoint.py.151: total_norm: 23.282974 [I 2022-10-10 14:45:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1630/2304 [I 2022-10-10 14:45:57] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:35 [I 2022-10-10 14:45:57] ...trap/engines/engine.py.229: process: 0.83732 | load: 0.00294 [I 2022-10-10 14:45:57] ...trap/engines/engine.py.230: loss: 0.16672 [I 2022-10-10 14:45:57] ...optimizers/trijoint.py.151: total_norm: 18.084959 [I 2022-10-10 14:46:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1640/2304 [I 2022-10-10 14:46:06] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:21 [I 2022-10-10 14:46:06] ...trap/engines/engine.py.229: process: 0.83719 | load: 0.00294 [I 2022-10-10 14:46:06] ...trap/engines/engine.py.230: loss: 0.19075 [I 2022-10-10 14:46:06] ...optimizers/trijoint.py.151: total_norm: 23.273390 [I 2022-10-10 14:46:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1650/2304 [I 2022-10-10 14:46:14] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:15 [I 2022-10-10 14:46:14] ...trap/engines/engine.py.229: process: 0.84177 | load: 0.00290 [I 2022-10-10 14:46:14] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-10 14:46:14] ...optimizers/trijoint.py.151: total_norm: 10.375444 [I 2022-10-10 14:46:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1660/2304 [I 2022-10-10 14:46:23] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:10 [I 2022-10-10 14:46:23] ...trap/engines/engine.py.229: process: 0.85620 | load: 0.00290 [I 2022-10-10 14:46:23] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-10 14:46:23] ...optimizers/trijoint.py.151: total_norm: 25.103815 [I 2022-10-10 14:46:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 1670/2304 [I 2022-10-10 14:46:32] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:58 [I 2022-10-10 14:46:32] ...trap/engines/engine.py.229: process: 0.85116 | load: 0.00295 [I 2022-10-10 14:46:32] ...trap/engines/engine.py.230: loss: 0.09339 [I 2022-10-10 14:46:32] ...optimizers/trijoint.py.151: total_norm: 9.191946 [I 2022-10-10 14:46:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1680/2304 [I 2022-10-10 14:46:40] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:51 [I 2022-10-10 14:46:40] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00293 [I 2022-10-10 14:46:40] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-10 14:46:40] ...optimizers/trijoint.py.151: total_norm: 10.583515 [I 2022-10-10 14:46:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1690/2304 [I 2022-10-10 14:46:49] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:43 [I 2022-10-10 14:46:49] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00295 [I 2022-10-10 14:46:49] ...trap/engines/engine.py.230: loss: 0.15379 [I 2022-10-10 14:46:49] ...optimizers/trijoint.py.151: total_norm: 14.193320 [I 2022-10-10 14:46:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1700/2304 [I 2022-10-10 14:46:57] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:33 [I 2022-10-10 14:46:57] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00292 [I 2022-10-10 14:46:57] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-10 14:46:57] ...optimizers/trijoint.py.151: total_norm: 16.902117 [I 2022-10-10 14:47:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1710/2304 [I 2022-10-10 14:47:06] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:28 [I 2022-10-10 14:47:06] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00301 [I 2022-10-10 14:47:06] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-10 14:47:06] ...optimizers/trijoint.py.151: total_norm: 11.012670 [I 2022-10-10 14:47:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1720/2304 [I 2022-10-10 14:47:14] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:18 [I 2022-10-10 14:47:14] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00295 [I 2022-10-10 14:47:14] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-10 14:47:14] ...optimizers/trijoint.py.151: total_norm: 11.767246 [I 2022-10-10 14:47:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1730/2304 [I 2022-10-10 14:47:23] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:10 [I 2022-10-10 14:47:23] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00291 [I 2022-10-10 14:47:23] ...trap/engines/engine.py.230: loss: 0.16829 [I 2022-10-10 14:47:23] ...optimizers/trijoint.py.151: total_norm: 22.978388 [I 2022-10-10 14:47:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1740/2304 [I 2022-10-10 14:47:31] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:59 [I 2022-10-10 14:47:31] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00289 [I 2022-10-10 14:47:31] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-10 14:47:31] ...optimizers/trijoint.py.151: total_norm: 23.298681 [I 2022-10-10 14:47:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1750/2304 [I 2022-10-10 14:47:40] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:51 [I 2022-10-10 14:47:40] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00295 [I 2022-10-10 14:47:40] ...trap/engines/engine.py.230: loss: 0.11039 [I 2022-10-10 14:47:40] ...optimizers/trijoint.py.151: total_norm: 23.349468 [I 2022-10-10 14:47:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 1760/2304 [I 2022-10-10 14:47:48] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:41 [I 2022-10-10 14:47:48] ...trap/engines/engine.py.229: process: 0.84334 | load: 0.00292 [I 2022-10-10 14:47:48] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-10 14:47:48] ...optimizers/trijoint.py.151: total_norm: 50.002304 [I 2022-10-10 14:47:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1770/2304 [I 2022-10-10 14:47:57] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:39 [I 2022-10-10 14:47:57] ...trap/engines/engine.py.229: process: 0.90331 | load: 0.00293 [I 2022-10-10 14:47:57] ...trap/engines/engine.py.230: loss: 0.17587 [I 2022-10-10 14:47:57] ...optimizers/trijoint.py.151: total_norm: 23.263733 [I 2022-10-10 14:48:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1780/2304 [I 2022-10-10 14:48:06] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:23 [I 2022-10-10 14:48:06] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00293 [I 2022-10-10 14:48:06] ...trap/engines/engine.py.230: loss: 0.19573 [I 2022-10-10 14:48:06] ...optimizers/trijoint.py.151: total_norm: 32.837986 [I 2022-10-10 14:48:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1790/2304 [I 2022-10-10 14:48:14] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:07:16 [I 2022-10-10 14:48:14] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00296 [I 2022-10-10 14:48:14] ...trap/engines/engine.py.230: loss: 0.09761 [I 2022-10-10 14:48:14] ...optimizers/trijoint.py.151: total_norm: 11.737011 [I 2022-10-10 14:48:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1800/2304 [I 2022-10-10 14:48:23] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:07:10 [I 2022-10-10 14:48:23] ...trap/engines/engine.py.229: process: 0.86681 | load: 0.00293 [I 2022-10-10 14:48:23] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-10 14:48:23] ...optimizers/trijoint.py.151: total_norm: 26.013828 [I 2022-10-10 14:48:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1810/2304 [I 2022-10-10 14:48:31] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:07:01 [I 2022-10-10 14:48:31] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00298 [I 2022-10-10 14:48:31] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-10 14:48:31] ...optimizers/trijoint.py.151: total_norm: 35.681511 [I 2022-10-10 14:48:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1820/2304 [I 2022-10-10 14:48:40] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:52 [I 2022-10-10 14:48:40] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00294 [I 2022-10-10 14:48:40] ...trap/engines/engine.py.230: loss: 0.11953 [I 2022-10-10 14:48:40] ...optimizers/trijoint.py.151: total_norm: 8.192401 [I 2022-10-10 14:48:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 1830/2304 [I 2022-10-10 14:48:48] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:42 [I 2022-10-10 14:48:48] ...trap/engines/engine.py.229: process: 0.85832 | load: 0.00299 [I 2022-10-10 14:48:48] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-10 14:48:48] ...optimizers/trijoint.py.151: total_norm: 9.952765 [I 2022-10-10 14:48:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1840/2304 [I 2022-10-10 14:48:57] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:36 [I 2022-10-10 14:48:57] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00295 [I 2022-10-10 14:48:57] ...trap/engines/engine.py.230: loss: 0.18030 [I 2022-10-10 14:48:57] ...optimizers/trijoint.py.151: total_norm: 24.656683 [I 2022-10-10 14:49:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 1850/2304 [I 2022-10-10 14:49:05] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:27 [I 2022-10-10 14:49:05] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00298 [I 2022-10-10 14:49:05] ...trap/engines/engine.py.230: loss: 0.18425 [I 2022-10-10 14:49:05] ...optimizers/trijoint.py.151: total_norm: 19.194975 [I 2022-10-10 14:49:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1860/2304 [I 2022-10-10 14:49:14] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:18 [I 2022-10-10 14:49:14] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00311 [I 2022-10-10 14:49:14] ...trap/engines/engine.py.230: loss: 0.13175 [I 2022-10-10 14:49:14] ...optimizers/trijoint.py.151: total_norm: 14.595609 [I 2022-10-10 14:49:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1870/2304 [I 2022-10-10 14:49:23] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:13 [I 2022-10-10 14:49:23] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00312 [I 2022-10-10 14:49:23] ...trap/engines/engine.py.230: loss: 0.10008 [I 2022-10-10 14:49:23] ...optimizers/trijoint.py.151: total_norm: 14.626872 [I 2022-10-10 14:49:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1880/2304 [I 2022-10-10 14:49:31] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:06:05 [I 2022-10-10 14:49:31] ...trap/engines/engine.py.229: process: 0.86986 | load: 0.00314 [I 2022-10-10 14:49:31] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-10 14:49:31] ...optimizers/trijoint.py.151: total_norm: 32.524899 [I 2022-10-10 14:49:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1890/2304 [I 2022-10-10 14:49:40] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:58 [I 2022-10-10 14:49:40] ...trap/engines/engine.py.229: process: 0.85812 | load: 0.00319 [I 2022-10-10 14:49:40] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-10 14:49:40] ...optimizers/trijoint.py.151: total_norm: 28.036640 [I 2022-10-10 14:49:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1900/2304 [I 2022-10-10 14:49:49] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:49 [I 2022-10-10 14:49:49] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00308 [I 2022-10-10 14:49:49] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-10 14:49:49] ...optimizers/trijoint.py.151: total_norm: 17.680044 [I 2022-10-10 14:49:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1910/2304 [I 2022-10-10 14:49:57] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:41 [I 2022-10-10 14:49:57] ...trap/engines/engine.py.229: process: 0.87395 | load: 0.00315 [I 2022-10-10 14:49:57] ...trap/engines/engine.py.230: loss: 0.19935 [I 2022-10-10 14:49:57] ...optimizers/trijoint.py.151: total_norm: 18.994684 [I 2022-10-10 14:50:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1920/2304 [I 2022-10-10 14:50:06] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:31 [I 2022-10-10 14:50:06] ...trap/engines/engine.py.229: process: 0.86224 | load: 0.00326 [I 2022-10-10 14:50:06] ...trap/engines/engine.py.230: loss: 0.11897 [I 2022-10-10 14:50:06] ...optimizers/trijoint.py.151: total_norm: 14.879331 [I 2022-10-10 14:50:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 1930/2304 [I 2022-10-10 14:50:15] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:26 [I 2022-10-10 14:50:15] ...trap/engines/engine.py.229: process: 0.88327 | load: 0.00325 [I 2022-10-10 14:50:15] ...trap/engines/engine.py.230: loss: 0.19504 [I 2022-10-10 14:50:15] ...optimizers/trijoint.py.151: total_norm: 19.683691 [I 2022-10-10 14:50:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1940/2304 [I 2022-10-10 14:50:23] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:05:15 [I 2022-10-10 14:50:23] ...trap/engines/engine.py.229: process: 0.87073 | load: 0.00314 [I 2022-10-10 14:50:23] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-10 14:50:23] ...optimizers/trijoint.py.151: total_norm: 33.200844 [I 2022-10-10 14:50:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 1950/2304 [I 2022-10-10 14:50:32] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:05:07 [I 2022-10-10 14:50:32] ...trap/engines/engine.py.229: process: 0.87249 | load: 0.00314 [I 2022-10-10 14:50:32] ...trap/engines/engine.py.230: loss: 0.13691 [I 2022-10-10 14:50:32] ...optimizers/trijoint.py.151: total_norm: 10.525373 [I 2022-10-10 14:50:41] ...trap/engines/engine.py.224: train: epoch 75 | batch 1960/2304 [I 2022-10-10 14:50:41] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:58 [I 2022-10-10 14:50:41] ...trap/engines/engine.py.229: process: 0.85789 | load: 0.00317 [I 2022-10-10 14:50:41] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-10 14:50:41] ...optimizers/trijoint.py.151: total_norm: 15.369795 [I 2022-10-10 14:50:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 1970/2304 [I 2022-10-10 14:50:50] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:50 [I 2022-10-10 14:50:50] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00316 [I 2022-10-10 14:50:50] ...trap/engines/engine.py.230: loss: 0.16448 [I 2022-10-10 14:50:50] ...optimizers/trijoint.py.151: total_norm: 21.362352 [I 2022-10-10 14:50:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 1980/2304 [I 2022-10-10 14:50:58] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:43 [I 2022-10-10 14:50:58] ...trap/engines/engine.py.229: process: 0.87218 | load: 0.00320 [I 2022-10-10 14:50:58] ...trap/engines/engine.py.230: loss: 0.18066 [I 2022-10-10 14:50:58] ...optimizers/trijoint.py.151: total_norm: 20.760595 [I 2022-10-10 14:51:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 1990/2304 [I 2022-10-10 14:51:07] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:32 [I 2022-10-10 14:51:07] ...trap/engines/engine.py.229: process: 0.86368 | load: 0.00315 [I 2022-10-10 14:51:07] ...trap/engines/engine.py.230: loss: 0.18353 [I 2022-10-10 14:51:07] ...optimizers/trijoint.py.151: total_norm: 18.551912 [I 2022-10-10 14:51:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 2000/2304 [I 2022-10-10 14:51:16] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:21 [I 2022-10-10 14:51:16] ...trap/engines/engine.py.229: process: 0.85436 | load: 0.00313 [I 2022-10-10 14:51:16] ...trap/engines/engine.py.230: loss: 0.12366 [I 2022-10-10 14:51:16] ...optimizers/trijoint.py.151: total_norm: 24.779177 [I 2022-10-10 14:51:24] ...trap/engines/engine.py.224: train: epoch 75 | batch 2010/2304 [I 2022-10-10 14:51:24] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:15 [I 2022-10-10 14:51:24] ...trap/engines/engine.py.229: process: 0.87535 | load: 0.00330 [I 2022-10-10 14:51:24] ...trap/engines/engine.py.230: loss: 0.14142 [I 2022-10-10 14:51:24] ...optimizers/trijoint.py.151: total_norm: 10.557102 [I 2022-10-10 14:51:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 2020/2304 [I 2022-10-10 14:51:33] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:04:05 [I 2022-10-10 14:51:33] ...trap/engines/engine.py.229: process: 0.85728 | load: 0.00320 [I 2022-10-10 14:51:33] ...trap/engines/engine.py.230: loss: 0.10470 [I 2022-10-10 14:51:33] ...optimizers/trijoint.py.151: total_norm: 14.486618 [I 2022-10-10 14:51:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 2030/2304 [I 2022-10-10 14:51:42] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:57 [I 2022-10-10 14:51:42] ...trap/engines/engine.py.229: process: 0.87000 | load: 0.00306 [I 2022-10-10 14:51:42] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-10 14:51:42] ...optimizers/trijoint.py.151: total_norm: 17.077578 [I 2022-10-10 14:51:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 2040/2304 [I 2022-10-10 14:51:50] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:47 [I 2022-10-10 14:51:50] ...trap/engines/engine.py.229: process: 0.86177 | load: 0.00312 [I 2022-10-10 14:51:50] ...trap/engines/engine.py.230: loss: 0.13891 [I 2022-10-10 14:51:50] ...optimizers/trijoint.py.151: total_norm: 10.729129 [I 2022-10-10 14:51:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 2050/2304 [I 2022-10-10 14:51:59] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:39 [I 2022-10-10 14:51:59] ...trap/engines/engine.py.229: process: 0.86120 | load: 0.00310 [I 2022-10-10 14:51:59] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-10 14:51:59] ...optimizers/trijoint.py.151: total_norm: 9.349369 [I 2022-10-10 14:52:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 2060/2304 [I 2022-10-10 14:52:08] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:03:32 [I 2022-10-10 14:52:08] ...trap/engines/engine.py.229: process: 0.88370 | load: 0.00313 [I 2022-10-10 14:52:08] ...trap/engines/engine.py.230: loss: 0.09822 [I 2022-10-10 14:52:08] ...optimizers/trijoint.py.151: total_norm: 12.268804 [I 2022-10-10 14:52:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 2070/2304 [I 2022-10-10 14:52:16] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:03:22 [I 2022-10-10 14:52:16] ...trap/engines/engine.py.229: process: 0.87601 | load: 0.00317 [I 2022-10-10 14:52:16] ...trap/engines/engine.py.230: loss: 0.17628 [I 2022-10-10 14:52:16] ...optimizers/trijoint.py.151: total_norm: 15.787281 [I 2022-10-10 14:52:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 2080/2304 [I 2022-10-10 14:52:25] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:14 [I 2022-10-10 14:52:25] ...trap/engines/engine.py.229: process: 0.88260 | load: 0.00321 [I 2022-10-10 14:52:25] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-10 14:52:25] ...optimizers/trijoint.py.151: total_norm: 24.392862 [I 2022-10-10 14:52:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 2090/2304 [I 2022-10-10 14:52:34] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:03 [I 2022-10-10 14:52:34] ...trap/engines/engine.py.229: process: 0.85033 | load: 0.00309 [I 2022-10-10 14:52:34] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-10 14:52:34] ...optimizers/trijoint.py.151: total_norm: 11.066512 [I 2022-10-10 14:52:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 2100/2304 [I 2022-10-10 14:52:42] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:55 [I 2022-10-10 14:52:42] ...trap/engines/engine.py.229: process: 0.86330 | load: 0.00314 [I 2022-10-10 14:52:42] ...trap/engines/engine.py.230: loss: 0.11040 [I 2022-10-10 14:52:42] ...optimizers/trijoint.py.151: total_norm: 14.280444 [I 2022-10-10 14:52:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 2110/2304 [I 2022-10-10 14:52:51] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:48 [I 2022-10-10 14:52:51] ...trap/engines/engine.py.229: process: 0.88208 | load: 0.00309 [I 2022-10-10 14:52:51] ...trap/engines/engine.py.230: loss: 0.14783 [I 2022-10-10 14:52:51] ...optimizers/trijoint.py.151: total_norm: 12.877100 [I 2022-10-10 14:53:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 2120/2304 [I 2022-10-10 14:53:00] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:38 [I 2022-10-10 14:53:00] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00316 [I 2022-10-10 14:53:00] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-10 14:53:00] ...optimizers/trijoint.py.151: total_norm: 14.840048 [I 2022-10-10 14:53:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 2130/2304 [I 2022-10-10 14:53:08] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:30 [I 2022-10-10 14:53:08] ...trap/engines/engine.py.229: process: 0.86429 | load: 0.00310 [I 2022-10-10 14:53:08] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-10 14:53:08] ...optimizers/trijoint.py.151: total_norm: 9.923275 [I 2022-10-10 14:53:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 2140/2304 [I 2022-10-10 14:53:17] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:20 [I 2022-10-10 14:53:17] ...trap/engines/engine.py.229: process: 0.84101 | load: 0.00296 [I 2022-10-10 14:53:17] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-10 14:53:17] ...optimizers/trijoint.py.151: total_norm: 16.926292 [I 2022-10-10 14:53:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 2150/2304 [I 2022-10-10 14:53:26] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:11 [I 2022-10-10 14:53:26] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00294 [I 2022-10-10 14:53:26] ...trap/engines/engine.py.230: loss: 0.22878 [I 2022-10-10 14:53:26] ...optimizers/trijoint.py.151: total_norm: 14.206275 [I 2022-10-10 14:53:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 2160/2304 [I 2022-10-10 14:53:34] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:03 [I 2022-10-10 14:53:34] ...trap/engines/engine.py.229: process: 0.86060 | load: 0.00300 [I 2022-10-10 14:53:34] ...trap/engines/engine.py.230: loss: 0.11388 [I 2022-10-10 14:53:34] ...optimizers/trijoint.py.151: total_norm: 12.385022 [I 2022-10-10 14:53:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 2170/2304 [I 2022-10-10 14:53:43] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:54 [I 2022-10-10 14:53:43] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00297 [I 2022-10-10 14:53:43] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-10 14:53:43] ...optimizers/trijoint.py.151: total_norm: 12.923423 [I 2022-10-10 14:53:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 2180/2304 [I 2022-10-10 14:53:51] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:45 [I 2022-10-10 14:53:51] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00304 [I 2022-10-10 14:53:51] ...trap/engines/engine.py.230: loss: 0.11426 [I 2022-10-10 14:53:51] ...optimizers/trijoint.py.151: total_norm: 13.407125 [I 2022-10-10 14:54:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 2190/2304 [I 2022-10-10 14:54:00] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:36 [I 2022-10-10 14:54:00] ...trap/engines/engine.py.229: process: 0.84103 | load: 0.00300 [I 2022-10-10 14:54:00] ...trap/engines/engine.py.230: loss: 0.08179 [I 2022-10-10 14:54:00] ...optimizers/trijoint.py.151: total_norm: 14.717360 [I 2022-10-10 14:54:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 2200/2304 [I 2022-10-10 14:54:08] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:28 [I 2022-10-10 14:54:08] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00294 [I 2022-10-10 14:54:08] ...trap/engines/engine.py.230: loss: 0.19193 [I 2022-10-10 14:54:08] ...optimizers/trijoint.py.151: total_norm: 9.346976 [I 2022-10-10 14:54:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 2210/2304 [I 2022-10-10 14:54:17] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:20 [I 2022-10-10 14:54:17] ...trap/engines/engine.py.229: process: 0.85778 | load: 0.00306 [I 2022-10-10 14:54:17] ...trap/engines/engine.py.230: loss: 0.15307 [I 2022-10-10 14:54:17] ...optimizers/trijoint.py.151: total_norm: 9.217041 [I 2022-10-10 14:54:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 2220/2304 [I 2022-10-10 14:54:26] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:12 [I 2022-10-10 14:54:26] ...trap/engines/engine.py.229: process: 0.86297 | load: 0.00309 [I 2022-10-10 14:54:26] ...trap/engines/engine.py.230: loss: 0.11600 [I 2022-10-10 14:54:26] ...optimizers/trijoint.py.151: total_norm: 10.380142 [I 2022-10-10 14:54:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 2230/2304 [I 2022-10-10 14:54:34] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:03 [I 2022-10-10 14:54:34] ...trap/engines/engine.py.229: process: 0.86508 | load: 0.00295 [I 2022-10-10 14:54:34] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-10 14:54:34] ...optimizers/trijoint.py.151: total_norm: 17.783665 [I 2022-10-10 14:54:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 2240/2304 [I 2022-10-10 14:54:43] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:54 [I 2022-10-10 14:54:43] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00298 [I 2022-10-10 14:54:43] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-10 14:54:43] ...optimizers/trijoint.py.151: total_norm: 16.877903 [I 2022-10-10 14:54:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 2250/2304 [I 2022-10-10 14:54:51] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:46 [I 2022-10-10 14:54:51] ...trap/engines/engine.py.229: process: 0.86321 | load: 0.00309 [I 2022-10-10 14:54:51] ...trap/engines/engine.py.230: loss: 0.09406 [I 2022-10-10 14:54:51] ...optimizers/trijoint.py.151: total_norm: 8.964853 [I 2022-10-10 14:55:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 2260/2304 [I 2022-10-10 14:55:00] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:37 [I 2022-10-10 14:55:00] ...trap/engines/engine.py.229: process: 0.84949 | load: 0.00296 [I 2022-10-10 14:55:00] ...trap/engines/engine.py.230: loss: 0.10691 [I 2022-10-10 14:55:00] ...optimizers/trijoint.py.151: total_norm: 10.584211 [I 2022-10-10 14:55:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 2270/2304 [I 2022-10-10 14:55:08] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:29 [I 2022-10-10 14:55:08] ...trap/engines/engine.py.229: process: 0.86328 | load: 0.00296 [I 2022-10-10 14:55:08] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-10 14:55:08] ...optimizers/trijoint.py.151: total_norm: 13.060147 [I 2022-10-10 14:55:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 2280/2304 [I 2022-10-10 14:55:17] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:20 [I 2022-10-10 14:55:17] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00296 [I 2022-10-10 14:55:17] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-10 14:55:17] ...optimizers/trijoint.py.151: total_norm: 13.667766 [I 2022-10-10 14:55:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 2290/2304 [I 2022-10-10 14:55:25] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:12 [I 2022-10-10 14:55:25] ...trap/engines/engine.py.229: process: 0.86858 | load: 0.00306 [I 2022-10-10 14:55:25] ...trap/engines/engine.py.230: loss: 0.13480 [I 2022-10-10 14:55:25] ...optimizers/trijoint.py.151: total_norm: 29.762829 [I 2022-10-10 14:55:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 2300/2304 [I 2022-10-10 14:55:34] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:03 [I 2022-10-10 14:55:34] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00018 [I 2022-10-10 14:55:34] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-10 14:55:34] ...optimizers/trijoint.py.151: total_norm: 12.892756 [I 2022-10-10 14:55:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 2304/2304 [I 2022-10-10 14:55:37] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:00 [I 2022-10-10 14:55:37] ...trap/engines/engine.py.229: process: 0.83811 | load: 0.00018 [I 2022-10-10 14:55:37] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-10 14:55:37] ...optimizers/trijoint.py.151: total_norm: 7.230934 [S 2022-10-10 14:55:39] ...trap/engines/engine.py.129: train_epoch.epoch: 75 [S 2022-10-10 14:55:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.13674576099226635 [I 2022-10-10 14:55:39] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 14:55:40] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 75 [I 2022-10-10 14:55:40] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 14:55:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 14:55:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 14:55:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 14:55:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 14:55:45] ...trap/engines/engine.py.321: eval: epoch 75 | batch 0/510 [I 2022-10-10 14:55:45] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:43 [I 2022-10-10 14:55:45] ...trap/engines/engine.py.326: process: 5.26261 | load: 4.68291 [I 2022-10-10 14:55:50] ...trap/engines/engine.py.321: eval: epoch 75 | batch 10/510 [I 2022-10-10 14:55:50] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:22 [I 2022-10-10 14:55:50] ...trap/engines/engine.py.326: process: 0.53057 | load: 0.00019 [I 2022-10-10 14:55:55] ...trap/engines/engine.py.321: eval: epoch 75 | batch 20/510 [I 2022-10-10 14:55:55] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:33 [I 2022-10-10 14:55:55] ...trap/engines/engine.py.326: process: 0.46570 | load: 0.00023 [I 2022-10-10 14:56:01] ...trap/engines/engine.py.321: eval: epoch 75 | batch 30/510 [I 2022-10-10 14:56:01] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:33 [I 2022-10-10 14:56:01] ...trap/engines/engine.py.326: process: 0.59125 | load: 0.00020 [I 2022-10-10 14:56:07] ...trap/engines/engine.py.321: eval: epoch 75 | batch 40/510 [I 2022-10-10 14:56:07] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:05:10 [I 2022-10-10 14:56:07] ...trap/engines/engine.py.326: process: 0.58798 | load: 0.00020 [I 2022-10-10 14:56:12] ...trap/engines/engine.py.321: eval: epoch 75 | batch 50/510 [I 2022-10-10 14:56:12] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:04:12 [I 2022-10-10 14:56:12] ...trap/engines/engine.py.326: process: 0.54400 | load: 0.00020 [I 2022-10-10 14:56:18] ...trap/engines/engine.py.321: eval: epoch 75 | batch 60/510 [I 2022-10-10 14:56:18] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:48 [I 2022-10-10 14:56:18] ...trap/engines/engine.py.326: process: 0.43004 | load: 0.00021 [I 2022-10-10 14:56:23] ...trap/engines/engine.py.321: eval: epoch 75 | batch 70/510 [I 2022-10-10 14:56:23] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:48 [I 2022-10-10 14:56:23] ...trap/engines/engine.py.326: process: 0.47126 | load: 0.00024 [I 2022-10-10 14:56:28] ...trap/engines/engine.py.321: eval: epoch 75 | batch 80/510 [I 2022-10-10 14:56:28] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:28 [I 2022-10-10 14:56:28] ...trap/engines/engine.py.326: process: 0.49715 | load: 0.00025 [I 2022-10-10 14:56:32] ...trap/engines/engine.py.321: eval: epoch 75 | batch 90/510 [I 2022-10-10 14:56:32] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:12 [I 2022-10-10 14:56:32] ...trap/engines/engine.py.326: process: 0.43544 | load: 0.00022 [I 2022-10-10 14:56:37] ...trap/engines/engine.py.321: eval: epoch 75 | batch 100/510 [I 2022-10-10 14:56:37] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:53 [I 2022-10-10 14:56:37] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00030 [I 2022-10-10 14:56:40] ...trap/engines/engine.py.321: eval: epoch 75 | batch 110/510 [I 2022-10-10 14:56:40] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:12 [I 2022-10-10 14:56:40] ...trap/engines/engine.py.326: process: 0.32184 | load: 0.00020 [I 2022-10-10 14:56:43] ...trap/engines/engine.py.321: eval: epoch 75 | batch 120/510 [I 2022-10-10 14:56:43] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:06 [I 2022-10-10 14:56:43] ...trap/engines/engine.py.326: process: 0.32468 | load: 0.00020 [I 2022-10-10 14:56:47] ...trap/engines/engine.py.321: eval: epoch 75 | batch 130/510 [I 2022-10-10 14:56:47] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:02 [I 2022-10-10 14:56:47] ...trap/engines/engine.py.326: process: 0.32846 | load: 0.00020 [I 2022-10-10 14:56:50] ...trap/engines/engine.py.321: eval: epoch 75 | batch 140/510 [I 2022-10-10 14:56:50] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:59 [I 2022-10-10 14:56:50] ...trap/engines/engine.py.326: process: 0.33082 | load: 0.00021 [I 2022-10-10 14:56:53] ...trap/engines/engine.py.321: eval: epoch 75 | batch 150/510 [I 2022-10-10 14:56:53] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:56 [I 2022-10-10 14:56:53] ...trap/engines/engine.py.326: process: 0.31804 | load: 0.00020 [I 2022-10-10 14:56:56] ...trap/engines/engine.py.321: eval: epoch 75 | batch 160/510 [I 2022-10-10 14:56:56] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:53 [I 2022-10-10 14:56:56] ...trap/engines/engine.py.326: process: 0.32707 | load: 0.00021 [I 2022-10-10 14:56:59] ...trap/engines/engine.py.321: eval: epoch 75 | batch 170/510 [I 2022-10-10 14:56:59] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:50 [I 2022-10-10 14:56:59] ...trap/engines/engine.py.326: process: 0.33192 | load: 0.00021 [I 2022-10-10 14:57:03] ...trap/engines/engine.py.321: eval: epoch 75 | batch 180/510 [I 2022-10-10 14:57:03] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:46 [I 2022-10-10 14:57:03] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00021 [I 2022-10-10 14:57:06] ...trap/engines/engine.py.321: eval: epoch 75 | batch 190/510 [I 2022-10-10 14:57:06] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:43 [I 2022-10-10 14:57:06] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00021 [I 2022-10-10 14:57:09] ...trap/engines/engine.py.321: eval: epoch 75 | batch 200/510 [I 2022-10-10 14:57:09] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:40 [I 2022-10-10 14:57:09] ...trap/engines/engine.py.326: process: 0.32471 | load: 0.00021 [I 2022-10-10 14:57:13] ...trap/engines/engine.py.321: eval: epoch 75 | batch 210/510 [I 2022-10-10 14:57:13] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:37 [I 2022-10-10 14:57:13] ...trap/engines/engine.py.326: process: 0.32354 | load: 0.00021 [I 2022-10-10 14:57:16] ...trap/engines/engine.py.321: eval: epoch 75 | batch 220/510 [I 2022-10-10 14:57:16] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:33 [I 2022-10-10 14:57:16] ...trap/engines/engine.py.326: process: 0.32471 | load: 0.00021 [I 2022-10-10 14:57:19] ...trap/engines/engine.py.321: eval: epoch 75 | batch 230/510 [I 2022-10-10 14:57:19] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:31 [I 2022-10-10 14:57:19] ...trap/engines/engine.py.326: process: 0.32790 | load: 0.00022 [I 2022-10-10 14:57:22] ...trap/engines/engine.py.321: eval: epoch 75 | batch 240/510 [I 2022-10-10 14:57:22] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:27 [I 2022-10-10 14:57:22] ...trap/engines/engine.py.326: process: 0.32891 | load: 0.00022 [I 2022-10-10 14:57:25] ...trap/engines/engine.py.321: eval: epoch 75 | batch 250/510 [I 2022-10-10 14:57:25] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:24 [I 2022-10-10 14:57:25] ...trap/engines/engine.py.326: process: 0.32109 | load: 0.00021 [I 2022-10-10 14:57:29] ...trap/engines/engine.py.321: eval: epoch 75 | batch 260/510 [I 2022-10-10 14:57:29] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:20 [I 2022-10-10 14:57:29] ...trap/engines/engine.py.326: process: 0.32221 | load: 0.00021 [I 2022-10-10 14:57:32] ...trap/engines/engine.py.321: eval: epoch 75 | batch 270/510 [I 2022-10-10 14:57:32] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:17 [I 2022-10-10 14:57:32] ...trap/engines/engine.py.326: process: 0.32314 | load: 0.00024 [I 2022-10-10 14:57:35] ...trap/engines/engine.py.321: eval: epoch 75 | batch 280/510 [I 2022-10-10 14:57:35] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:14 [I 2022-10-10 14:57:35] ...trap/engines/engine.py.326: process: 0.32596 | load: 0.00021 [I 2022-10-10 14:57:38] ...trap/engines/engine.py.321: eval: epoch 75 | batch 290/510 [I 2022-10-10 14:57:38] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:11 [I 2022-10-10 14:57:38] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00024 [I 2022-10-10 14:57:42] ...trap/engines/engine.py.321: eval: epoch 75 | batch 300/510 [I 2022-10-10 14:57:42] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:07 [I 2022-10-10 14:57:42] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00021 [I 2022-10-10 14:57:45] ...trap/engines/engine.py.321: eval: epoch 75 | batch 310/510 [I 2022-10-10 14:57:45] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:04 [I 2022-10-10 14:57:45] ...trap/engines/engine.py.326: process: 0.31699 | load: 0.00027 [I 2022-10-10 14:57:48] ...trap/engines/engine.py.321: eval: epoch 75 | batch 320/510 [I 2022-10-10 14:57:48] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:01 [I 2022-10-10 14:57:48] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00021 [I 2022-10-10 14:57:51] ...trap/engines/engine.py.321: eval: epoch 75 | batch 330/510 [I 2022-10-10 14:57:51] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:57 [I 2022-10-10 14:57:51] ...trap/engines/engine.py.326: process: 0.32018 | load: 0.00021 [I 2022-10-10 14:57:55] ...trap/engines/engine.py.321: eval: epoch 75 | batch 340/510 [I 2022-10-10 14:57:55] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:54 [I 2022-10-10 14:57:55] ...trap/engines/engine.py.326: process: 0.32511 | load: 0.00022 [I 2022-10-10 14:57:58] ...trap/engines/engine.py.321: eval: epoch 75 | batch 350/510 [I 2022-10-10 14:57:58] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-10 14:57:58] ...trap/engines/engine.py.326: process: 0.32281 | load: 0.00020 [I 2022-10-10 14:58:01] ...trap/engines/engine.py.321: eval: epoch 75 | batch 360/510 [I 2022-10-10 14:58:01] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:48 [I 2022-10-10 14:58:01] ...trap/engines/engine.py.326: process: 0.32418 | load: 0.00021 [I 2022-10-10 14:58:04] ...trap/engines/engine.py.321: eval: epoch 75 | batch 370/510 [I 2022-10-10 14:58:04] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:44 [I 2022-10-10 14:58:04] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00022 [I 2022-10-10 14:58:08] ...trap/engines/engine.py.321: eval: epoch 75 | batch 380/510 [I 2022-10-10 14:58:08] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:42 [I 2022-10-10 14:58:08] ...trap/engines/engine.py.326: process: 0.32444 | load: 0.00022 [I 2022-10-10 14:58:11] ...trap/engines/engine.py.321: eval: epoch 75 | batch 390/510 [I 2022-10-10 14:58:11] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:39 [I 2022-10-10 14:58:11] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00021 [I 2022-10-10 14:58:14] ...trap/engines/engine.py.321: eval: epoch 75 | batch 400/510 [I 2022-10-10 14:58:14] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:35 [I 2022-10-10 14:58:14] ...trap/engines/engine.py.326: process: 0.32224 | load: 0.00021 [I 2022-10-10 14:58:17] ...trap/engines/engine.py.321: eval: epoch 75 | batch 410/510 [I 2022-10-10 14:58:17] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:32 [I 2022-10-10 14:58:17] ...trap/engines/engine.py.326: process: 0.32303 | load: 0.00021 [I 2022-10-10 14:58:21] ...trap/engines/engine.py.321: eval: epoch 75 | batch 420/510 [I 2022-10-10 14:58:21] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:29 [I 2022-10-10 14:58:21] ...trap/engines/engine.py.326: process: 0.32899 | load: 0.00029 [I 2022-10-10 14:58:24] ...trap/engines/engine.py.321: eval: epoch 75 | batch 430/510 [I 2022-10-10 14:58:24] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:26 [I 2022-10-10 14:58:24] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00021 [I 2022-10-10 14:58:27] ...trap/engines/engine.py.321: eval: epoch 75 | batch 440/510 [I 2022-10-10 14:58:27] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:22 [I 2022-10-10 14:58:27] ...trap/engines/engine.py.326: process: 0.33002 | load: 0.00022 [I 2022-10-10 14:58:30] ...trap/engines/engine.py.321: eval: epoch 75 | batch 450/510 [I 2022-10-10 14:58:30] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:19 [I 2022-10-10 14:58:30] ...trap/engines/engine.py.326: process: 0.32911 | load: 0.00021 [I 2022-10-10 14:58:34] ...trap/engines/engine.py.321: eval: epoch 75 | batch 460/510 [I 2022-10-10 14:58:34] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:16 [I 2022-10-10 14:58:34] ...trap/engines/engine.py.326: process: 0.32242 | load: 0.00021 [I 2022-10-10 14:58:37] ...trap/engines/engine.py.321: eval: epoch 75 | batch 470/510 [I 2022-10-10 14:58:37] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:12 [I 2022-10-10 14:58:37] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00021 [I 2022-10-10 14:58:40] ...trap/engines/engine.py.321: eval: epoch 75 | batch 480/510 [I 2022-10-10 14:58:40] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-10 14:58:40] ...trap/engines/engine.py.326: process: 0.32746 | load: 0.00021 [I 2022-10-10 14:58:43] ...trap/engines/engine.py.321: eval: epoch 75 | batch 490/510 [I 2022-10-10 14:58:43] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-10 14:58:43] ...trap/engines/engine.py.326: process: 0.32828 | load: 0.00021 [I 2022-10-10 14:58:47] ...trap/engines/engine.py.321: eval: epoch 75 | batch 500/510 [I 2022-10-10 14:58:47] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:03 [I 2022-10-10 14:58:47] ...trap/engines/engine.py.326: process: 0.32383 | load: 0.00021 [I 2022-10-10 14:58:50] ...trap/engines/engine.py.321: eval: epoch 75 | batch 510/510 [I 2022-10-10 14:58:50] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:00 [I 2022-10-10 14:58:50] ...trap/engines/engine.py.326: process: 0.31954 | load: 0.00017 [S 2022-10-10 14:58:51] ...trap/engines/engine.py.132: eval_epoch.epoch: 75 [I 2022-10-10 14:58:51] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737400 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.914200 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940900 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.736200 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911900 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.940400 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009404 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005212 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004061 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012352 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010894 [S 2022-10-10 14:59:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005238 [I 2022-10-10 15:00:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 15:00:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 15:00:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 15:00:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 15:00:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 15:00:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 15:00:08] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 15:00:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 15:00:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 76 [I 2022-10-10 15:00:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 15:00:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 0/2304 [I 2022-10-10 15:00:15] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:59:01 [I 2022-10-10 15:00:15] ...trap/engines/engine.py.229: process: 3.09959 | load: 2.20316 [I 2022-10-10 15:00:15] ...trap/engines/engine.py.230: loss: 0.10105 [I 2022-10-10 15:00:15] ...optimizers/trijoint.py.151: total_norm: 11.667402 [I 2022-10-10 15:00:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 10/2304 [I 2022-10-10 15:00:24] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:31 [I 2022-10-10 15:00:24] ...trap/engines/engine.py.229: process: 0.84096 | load: 0.00475 [I 2022-10-10 15:00:24] ...trap/engines/engine.py.230: loss: 0.24671 [I 2022-10-10 15:00:24] ...optimizers/trijoint.py.151: total_norm: 14.210961 [I 2022-10-10 15:00:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 20/2304 [I 2022-10-10 15:00:32] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:20 [I 2022-10-10 15:00:32] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00516 [I 2022-10-10 15:00:32] ...trap/engines/engine.py.230: loss: 0.12917 [I 2022-10-10 15:00:32] ...optimizers/trijoint.py.151: total_norm: 8.249568 [I 2022-10-10 15:00:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 30/2304 [I 2022-10-10 15:00:41] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:18 [I 2022-10-10 15:00:41] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00362 [I 2022-10-10 15:00:41] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-10 15:00:41] ...optimizers/trijoint.py.151: total_norm: 15.500891 [I 2022-10-10 15:00:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 40/2304 [I 2022-10-10 15:00:49] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:15 [I 2022-10-10 15:00:49] ...trap/engines/engine.py.229: process: 0.85864 | load: 0.00345 [I 2022-10-10 15:00:49] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-10 15:00:49] ...optimizers/trijoint.py.151: total_norm: 20.008680 [I 2022-10-10 15:00:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 50/2304 [I 2022-10-10 15:00:58] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:09 [I 2022-10-10 15:00:58] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00389 [I 2022-10-10 15:00:58] ...trap/engines/engine.py.230: loss: 0.10582 [I 2022-10-10 15:00:58] ...optimizers/trijoint.py.151: total_norm: 11.866110 [I 2022-10-10 15:01:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 60/2304 [I 2022-10-10 15:01:06] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:54 [I 2022-10-10 15:01:06] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00345 [I 2022-10-10 15:01:06] ...trap/engines/engine.py.230: loss: 0.08606 [I 2022-10-10 15:01:06] ...optimizers/trijoint.py.151: total_norm: 9.986719 [I 2022-10-10 15:01:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 70/2304 [I 2022-10-10 15:01:15] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:58 [I 2022-10-10 15:01:15] ...trap/engines/engine.py.229: process: 0.85782 | load: 0.00369 [I 2022-10-10 15:01:15] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-10 15:01:15] ...optimizers/trijoint.py.151: total_norm: 22.516768 [I 2022-10-10 15:01:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 80/2304 [I 2022-10-10 15:01:24] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:20 [I 2022-10-10 15:01:24] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00334 [I 2022-10-10 15:01:24] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-10 15:01:24] ...optimizers/trijoint.py.151: total_norm: 21.758455 [I 2022-10-10 15:01:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 90/2304 [I 2022-10-10 15:01:32] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:17 [I 2022-10-10 15:01:32] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00329 [I 2022-10-10 15:01:32] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-10 15:01:32] ...optimizers/trijoint.py.151: total_norm: 6.272881 [I 2022-10-10 15:01:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 100/2304 [I 2022-10-10 15:01:41] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:15 [I 2022-10-10 15:01:41] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00340 [I 2022-10-10 15:01:41] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-10 15:01:41] ...optimizers/trijoint.py.151: total_norm: 16.913347 [I 2022-10-10 15:01:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 110/2304 [I 2022-10-10 15:01:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:11 [I 2022-10-10 15:01:49] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00332 [I 2022-10-10 15:01:49] ...trap/engines/engine.py.230: loss: 0.16159 [I 2022-10-10 15:01:49] ...optimizers/trijoint.py.151: total_norm: 52.873661 [I 2022-10-10 15:01:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 120/2304 [I 2022-10-10 15:01:58] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:57 [I 2022-10-10 15:01:58] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00324 [I 2022-10-10 15:01:58] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-10 15:01:58] ...optimizers/trijoint.py.151: total_norm: 11.890630 [I 2022-10-10 15:02:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 130/2304 [I 2022-10-10 15:02:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:50 [I 2022-10-10 15:02:06] ...trap/engines/engine.py.229: process: 0.85649 | load: 0.00306 [I 2022-10-10 15:02:06] ...trap/engines/engine.py.230: loss: 0.12489 [I 2022-10-10 15:02:06] ...optimizers/trijoint.py.151: total_norm: 22.576128 [I 2022-10-10 15:02:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 140/2304 [I 2022-10-10 15:02:15] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:50 [I 2022-10-10 15:02:15] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00330 [I 2022-10-10 15:02:15] ...trap/engines/engine.py.230: loss: 0.13488 [I 2022-10-10 15:02:15] ...optimizers/trijoint.py.151: total_norm: 12.783250 [I 2022-10-10 15:02:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 150/2304 [I 2022-10-10 15:02:23] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:45 [I 2022-10-10 15:02:23] ...trap/engines/engine.py.229: process: 0.86746 | load: 0.00312 [I 2022-10-10 15:02:23] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-10 15:02:23] ...optimizers/trijoint.py.151: total_norm: 12.023890 [I 2022-10-10 15:02:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 160/2304 [I 2022-10-10 15:02:32] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:35 [I 2022-10-10 15:02:32] ...trap/engines/engine.py.229: process: 0.85241 | load: 0.00309 [I 2022-10-10 15:02:32] ...trap/engines/engine.py.230: loss: 0.11418 [I 2022-10-10 15:02:32] ...optimizers/trijoint.py.151: total_norm: 24.760725 [I 2022-10-10 15:02:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 170/2304 [I 2022-10-10 15:02:41] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:28 [I 2022-10-10 15:02:41] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00304 [I 2022-10-10 15:02:41] ...trap/engines/engine.py.230: loss: 0.13950 [I 2022-10-10 15:02:41] ...optimizers/trijoint.py.151: total_norm: 16.140795 [I 2022-10-10 15:02:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 180/2304 [I 2022-10-10 15:02:49] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:10 [I 2022-10-10 15:02:49] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00317 [I 2022-10-10 15:02:49] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-10 15:02:49] ...optimizers/trijoint.py.151: total_norm: 12.355671 [I 2022-10-10 15:02:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 190/2304 [I 2022-10-10 15:02:58] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:10 [I 2022-10-10 15:02:58] ...trap/engines/engine.py.229: process: 0.86195 | load: 0.00296 [I 2022-10-10 15:02:58] ...trap/engines/engine.py.230: loss: 0.14155 [I 2022-10-10 15:02:58] ...optimizers/trijoint.py.151: total_norm: 20.679855 [I 2022-10-10 15:03:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 200/2304 [I 2022-10-10 15:03:06] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:51 [I 2022-10-10 15:03:06] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00317 [I 2022-10-10 15:03:06] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-10 15:03:06] ...optimizers/trijoint.py.151: total_norm: 17.274750 [I 2022-10-10 15:03:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 210/2304 [I 2022-10-10 15:03:15] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:47 [I 2022-10-10 15:03:15] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00295 [I 2022-10-10 15:03:15] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-10 15:03:15] ...optimizers/trijoint.py.151: total_norm: 8.597439 [I 2022-10-10 15:03:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 220/2304 [I 2022-10-10 15:03:23] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:43 [I 2022-10-10 15:03:23] ...trap/engines/engine.py.229: process: 0.86513 | load: 0.00313 [I 2022-10-10 15:03:23] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-10 15:03:23] ...optimizers/trijoint.py.151: total_norm: 8.850862 [I 2022-10-10 15:03:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 230/2304 [I 2022-10-10 15:03:32] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:51 [I 2022-10-10 15:03:32] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00326 [I 2022-10-10 15:03:32] ...trap/engines/engine.py.230: loss: 0.13046 [I 2022-10-10 15:03:32] ...optimizers/trijoint.py.151: total_norm: 18.716198 [I 2022-10-10 15:03:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 240/2304 [I 2022-10-10 15:03:41] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:42 [I 2022-10-10 15:03:41] ...trap/engines/engine.py.229: process: 0.85550 | load: 0.00320 [I 2022-10-10 15:03:41] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-10 15:03:41] ...optimizers/trijoint.py.151: total_norm: 9.038519 [I 2022-10-10 15:03:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 250/2304 [I 2022-10-10 15:03:49] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:32 [I 2022-10-10 15:03:49] ...trap/engines/engine.py.229: process: 0.86075 | load: 0.00325 [I 2022-10-10 15:03:49] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-10 15:03:49] ...optimizers/trijoint.py.151: total_norm: 14.024637 [I 2022-10-10 15:03:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 260/2304 [I 2022-10-10 15:03:58] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:30 [I 2022-10-10 15:03:58] ...trap/engines/engine.py.229: process: 0.86100 | load: 0.00332 [I 2022-10-10 15:03:58] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-10 15:03:58] ...optimizers/trijoint.py.151: total_norm: 14.370499 [I 2022-10-10 15:04:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 270/2304 [I 2022-10-10 15:04:07] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:22 [I 2022-10-10 15:04:07] ...trap/engines/engine.py.229: process: 0.85852 | load: 0.00347 [I 2022-10-10 15:04:07] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-10 15:04:07] ...optimizers/trijoint.py.151: total_norm: 18.334747 [I 2022-10-10 15:04:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 280/2304 [I 2022-10-10 15:04:16] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:29:24 [I 2022-10-10 15:04:16] ...trap/engines/engine.py.229: process: 0.87440 | load: 0.00324 [I 2022-10-10 15:04:16] ...trap/engines/engine.py.230: loss: 0.13141 [I 2022-10-10 15:04:16] ...optimizers/trijoint.py.151: total_norm: 14.873719 [I 2022-10-10 15:04:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 290/2304 [I 2022-10-10 15:04:24] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:29:01 [I 2022-10-10 15:04:24] ...trap/engines/engine.py.229: process: 0.87411 | load: 0.00309 [I 2022-10-10 15:04:24] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-10 15:04:24] ...optimizers/trijoint.py.151: total_norm: 20.063625 [I 2022-10-10 15:04:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 300/2304 [I 2022-10-10 15:04:33] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:49 [I 2022-10-10 15:04:33] ...trap/engines/engine.py.229: process: 0.87349 | load: 0.00310 [I 2022-10-10 15:04:33] ...trap/engines/engine.py.230: loss: 0.13476 [I 2022-10-10 15:04:33] ...optimizers/trijoint.py.151: total_norm: 25.853601 [I 2022-10-10 15:04:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 310/2304 [I 2022-10-10 15:04:42] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:52 [I 2022-10-10 15:04:42] ...trap/engines/engine.py.229: process: 0.87009 | load: 0.00313 [I 2022-10-10 15:04:42] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-10 15:04:42] ...optimizers/trijoint.py.151: total_norm: 14.775714 [I 2022-10-10 15:04:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 320/2304 [I 2022-10-10 15:04:50] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:41 [I 2022-10-10 15:04:50] ...trap/engines/engine.py.229: process: 0.86742 | load: 0.00310 [I 2022-10-10 15:04:50] ...trap/engines/engine.py.230: loss: 0.20904 [I 2022-10-10 15:04:50] ...optimizers/trijoint.py.151: total_norm: 18.594137 [I 2022-10-10 15:04:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 330/2304 [I 2022-10-10 15:04:59] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:24 [I 2022-10-10 15:04:59] ...trap/engines/engine.py.229: process: 0.86244 | load: 0.00307 [I 2022-10-10 15:04:59] ...trap/engines/engine.py.230: loss: 0.13452 [I 2022-10-10 15:04:59] ...optimizers/trijoint.py.151: total_norm: 14.565325 [I 2022-10-10 15:05:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 340/2304 [I 2022-10-10 15:05:08] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:18 [I 2022-10-10 15:05:08] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00338 [I 2022-10-10 15:05:08] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-10 15:05:08] ...optimizers/trijoint.py.151: total_norm: 15.922054 [I 2022-10-10 15:05:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 350/2304 [I 2022-10-10 15:05:16] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:06 [I 2022-10-10 15:05:16] ...trap/engines/engine.py.229: process: 0.86334 | load: 0.00314 [I 2022-10-10 15:05:16] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-10 15:05:16] ...optimizers/trijoint.py.151: total_norm: 11.343630 [I 2022-10-10 15:05:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 360/2304 [I 2022-10-10 15:05:25] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:28:06 [I 2022-10-10 15:05:25] ...trap/engines/engine.py.229: process: 0.86925 | load: 0.00306 [I 2022-10-10 15:05:25] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-10 15:05:25] ...optimizers/trijoint.py.151: total_norm: 13.624972 [I 2022-10-10 15:05:34] ...trap/engines/engine.py.224: train: epoch 76 | batch 370/2304 [I 2022-10-10 15:05:34] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:52 [I 2022-10-10 15:05:34] ...trap/engines/engine.py.229: process: 0.87762 | load: 0.00307 [I 2022-10-10 15:05:34] ...trap/engines/engine.py.230: loss: 0.14089 [I 2022-10-10 15:05:34] ...optimizers/trijoint.py.151: total_norm: 14.995148 [I 2022-10-10 15:05:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 380/2304 [I 2022-10-10 15:05:42] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:50 [I 2022-10-10 15:05:42] ...trap/engines/engine.py.229: process: 0.87245 | load: 0.00320 [I 2022-10-10 15:05:42] ...trap/engines/engine.py.230: loss: 0.12942 [I 2022-10-10 15:05:42] ...optimizers/trijoint.py.151: total_norm: 11.065233 [I 2022-10-10 15:05:51] ...trap/engines/engine.py.224: train: epoch 76 | batch 390/2304 [I 2022-10-10 15:05:51] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:34 [I 2022-10-10 15:05:51] ...trap/engines/engine.py.229: process: 0.86471 | load: 0.00325 [I 2022-10-10 15:05:51] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-10 15:05:51] ...optimizers/trijoint.py.151: total_norm: 14.759077 [I 2022-10-10 15:06:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 400/2304 [I 2022-10-10 15:06:00] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:21 [I 2022-10-10 15:06:00] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00322 [I 2022-10-10 15:06:00] ...trap/engines/engine.py.230: loss: 0.17073 [I 2022-10-10 15:06:00] ...optimizers/trijoint.py.151: total_norm: 11.845069 [I 2022-10-10 15:06:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 410/2304 [I 2022-10-10 15:06:08] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:17 [I 2022-10-10 15:06:08] ...trap/engines/engine.py.229: process: 0.86962 | load: 0.00315 [I 2022-10-10 15:06:08] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-10 15:06:08] ...optimizers/trijoint.py.151: total_norm: 17.469879 [I 2022-10-10 15:06:17] ...trap/engines/engine.py.224: train: epoch 76 | batch 420/2304 [I 2022-10-10 15:06:17] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:27:00 [I 2022-10-10 15:06:17] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00307 [I 2022-10-10 15:06:17] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-10 15:06:17] ...optimizers/trijoint.py.151: total_norm: 10.494035 [I 2022-10-10 15:06:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 430/2304 [I 2022-10-10 15:06:26] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:52 [I 2022-10-10 15:06:26] ...trap/engines/engine.py.229: process: 0.86427 | load: 0.00312 [I 2022-10-10 15:06:26] ...trap/engines/engine.py.230: loss: 0.11622 [I 2022-10-10 15:06:26] ...optimizers/trijoint.py.151: total_norm: 11.700245 [I 2022-10-10 15:06:34] ...trap/engines/engine.py.224: train: epoch 76 | batch 440/2304 [I 2022-10-10 15:06:34] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:46 [I 2022-10-10 15:06:34] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00308 [I 2022-10-10 15:06:34] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-10 15:06:34] ...optimizers/trijoint.py.151: total_norm: 20.153881 [I 2022-10-10 15:06:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 450/2304 [I 2022-10-10 15:06:43] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:36 [I 2022-10-10 15:06:43] ...trap/engines/engine.py.229: process: 0.86603 | load: 0.00327 [I 2022-10-10 15:06:43] ...trap/engines/engine.py.230: loss: 0.13295 [I 2022-10-10 15:06:43] ...optimizers/trijoint.py.151: total_norm: 10.547745 [I 2022-10-10 15:06:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 460/2304 [I 2022-10-10 15:06:52] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:27 [I 2022-10-10 15:06:52] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00311 [I 2022-10-10 15:06:52] ...trap/engines/engine.py.230: loss: 0.11476 [I 2022-10-10 15:06:52] ...optimizers/trijoint.py.151: total_norm: 10.284007 [I 2022-10-10 15:07:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 470/2304 [I 2022-10-10 15:07:00] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:28 [I 2022-10-10 15:07:00] ...trap/engines/engine.py.229: process: 0.87456 | load: 0.00311 [I 2022-10-10 15:07:00] ...trap/engines/engine.py.230: loss: 0.12957 [I 2022-10-10 15:07:00] ...optimizers/trijoint.py.151: total_norm: 15.285944 [I 2022-10-10 15:07:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 480/2304 [I 2022-10-10 15:07:09] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:18 [I 2022-10-10 15:07:09] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00309 [I 2022-10-10 15:07:09] ...trap/engines/engine.py.230: loss: 0.11922 [I 2022-10-10 15:07:09] ...optimizers/trijoint.py.151: total_norm: 11.855312 [I 2022-10-10 15:07:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 490/2304 [I 2022-10-10 15:07:18] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:03 [I 2022-10-10 15:07:18] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00301 [I 2022-10-10 15:07:18] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-10 15:07:18] ...optimizers/trijoint.py.151: total_norm: 15.460550 [I 2022-10-10 15:07:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 500/2304 [I 2022-10-10 15:07:26] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:30 [I 2022-10-10 15:07:26] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00294 [I 2022-10-10 15:07:26] ...trap/engines/engine.py.230: loss: 0.11694 [I 2022-10-10 15:07:26] ...optimizers/trijoint.py.151: total_norm: 8.020146 [I 2022-10-10 15:07:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 510/2304 [I 2022-10-10 15:07:35] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:33 [I 2022-10-10 15:07:35] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00295 [I 2022-10-10 15:07:35] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-10 15:07:35] ...optimizers/trijoint.py.151: total_norm: 21.588284 [I 2022-10-10 15:07:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 520/2304 [I 2022-10-10 15:07:43] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:18 [I 2022-10-10 15:07:43] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00293 [I 2022-10-10 15:07:43] ...trap/engines/engine.py.230: loss: 0.10090 [I 2022-10-10 15:07:43] ...optimizers/trijoint.py.151: total_norm: 8.249249 [I 2022-10-10 15:07:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 530/2304 [I 2022-10-10 15:07:52] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:19 [I 2022-10-10 15:07:52] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00299 [I 2022-10-10 15:07:52] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-10 15:07:52] ...optimizers/trijoint.py.151: total_norm: 18.343843 [I 2022-10-10 15:08:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 540/2304 [I 2022-10-10 15:08:00] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:06 [I 2022-10-10 15:08:00] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00293 [I 2022-10-10 15:08:00] ...trap/engines/engine.py.230: loss: 0.10704 [I 2022-10-10 15:08:00] ...optimizers/trijoint.py.151: total_norm: 7.734504 [I 2022-10-10 15:08:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 550/2304 [I 2022-10-10 15:08:09] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:55 [I 2022-10-10 15:08:09] ...trap/engines/engine.py.229: process: 0.84795 | load: 0.00297 [I 2022-10-10 15:08:09] ...trap/engines/engine.py.230: loss: 0.19604 [I 2022-10-10 15:08:09] ...optimizers/trijoint.py.151: total_norm: 17.534723 [I 2022-10-10 15:08:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 560/2304 [I 2022-10-10 15:08:18] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:00 [I 2022-10-10 15:08:18] ...trap/engines/engine.py.229: process: 0.86778 | load: 0.00297 [I 2022-10-10 15:08:18] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-10 15:08:18] ...optimizers/trijoint.py.151: total_norm: 14.524424 [I 2022-10-10 15:08:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 570/2304 [I 2022-10-10 15:08:26] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:51 [I 2022-10-10 15:08:26] ...trap/engines/engine.py.229: process: 0.86788 | load: 0.00302 [I 2022-10-10 15:08:26] ...trap/engines/engine.py.230: loss: 0.17955 [I 2022-10-10 15:08:26] ...optimizers/trijoint.py.151: total_norm: 27.172880 [I 2022-10-10 15:08:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 580/2304 [I 2022-10-10 15:08:35] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:35 [I 2022-10-10 15:08:35] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00292 [I 2022-10-10 15:08:35] ...trap/engines/engine.py.230: loss: 0.09699 [I 2022-10-10 15:08:35] ...optimizers/trijoint.py.151: total_norm: 18.944498 [I 2022-10-10 15:08:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 590/2304 [I 2022-10-10 15:08:43] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:29 [I 2022-10-10 15:08:43] ...trap/engines/engine.py.229: process: 0.85261 | load: 0.00298 [I 2022-10-10 15:08:43] ...trap/engines/engine.py.230: loss: 0.12670 [I 2022-10-10 15:08:43] ...optimizers/trijoint.py.151: total_norm: 11.371539 [I 2022-10-10 15:08:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 600/2304 [I 2022-10-10 15:08:52] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:24:26 [I 2022-10-10 15:08:52] ...trap/engines/engine.py.229: process: 0.86127 | load: 0.00297 [I 2022-10-10 15:08:52] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-10 15:08:52] ...optimizers/trijoint.py.151: total_norm: 71.587196 [I 2022-10-10 15:09:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 610/2304 [I 2022-10-10 15:09:01] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:06 [I 2022-10-10 15:09:01] ...trap/engines/engine.py.229: process: 0.86068 | load: 0.00297 [I 2022-10-10 15:09:01] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-10 15:09:01] ...optimizers/trijoint.py.151: total_norm: 32.281101 [I 2022-10-10 15:09:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 620/2304 [I 2022-10-10 15:09:09] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:57 [I 2022-10-10 15:09:09] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00297 [I 2022-10-10 15:09:09] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-10 15:09:09] ...optimizers/trijoint.py.151: total_norm: 19.722116 [I 2022-10-10 15:09:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 630/2304 [I 2022-10-10 15:09:18] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:54 [I 2022-10-10 15:09:18] ...trap/engines/engine.py.229: process: 0.86208 | load: 0.00295 [I 2022-10-10 15:09:18] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-10 15:09:18] ...optimizers/trijoint.py.151: total_norm: 9.744406 [I 2022-10-10 15:09:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 640/2304 [I 2022-10-10 15:09:26] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:44 [I 2022-10-10 15:09:26] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00296 [I 2022-10-10 15:09:26] ...trap/engines/engine.py.230: loss: 0.20169 [I 2022-10-10 15:09:26] ...optimizers/trijoint.py.151: total_norm: 10.426619 [I 2022-10-10 15:09:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 650/2304 [I 2022-10-10 15:09:35] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:37 [I 2022-10-10 15:09:35] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00291 [I 2022-10-10 15:09:35] ...trap/engines/engine.py.230: loss: 0.13860 [I 2022-10-10 15:09:35] ...optimizers/trijoint.py.151: total_norm: 12.031184 [I 2022-10-10 15:09:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 660/2304 [I 2022-10-10 15:09:44] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:34 [I 2022-10-10 15:09:44] ...trap/engines/engine.py.229: process: 0.87536 | load: 0.00294 [I 2022-10-10 15:09:44] ...trap/engines/engine.py.230: loss: 0.20171 [I 2022-10-10 15:09:44] ...optimizers/trijoint.py.151: total_norm: 17.291660 [I 2022-10-10 15:09:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 670/2304 [I 2022-10-10 15:09:52] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:23:11 [I 2022-10-10 15:09:52] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00294 [I 2022-10-10 15:09:52] ...trap/engines/engine.py.230: loss: 0.15996 [I 2022-10-10 15:09:52] ...optimizers/trijoint.py.151: total_norm: 31.102142 [I 2022-10-10 15:10:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 680/2304 [I 2022-10-10 15:10:01] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:23:06 [I 2022-10-10 15:10:01] ...trap/engines/engine.py.229: process: 0.84894 | load: 0.00295 [I 2022-10-10 15:10:01] ...trap/engines/engine.py.230: loss: 0.11294 [I 2022-10-10 15:10:01] ...optimizers/trijoint.py.151: total_norm: 33.938705 [I 2022-10-10 15:10:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 690/2304 [I 2022-10-10 15:10:09] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:23:03 [I 2022-10-10 15:10:09] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00294 [I 2022-10-10 15:10:09] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-10 15:10:09] ...optimizers/trijoint.py.151: total_norm: 16.584127 [I 2022-10-10 15:10:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 700/2304 [I 2022-10-10 15:10:18] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:59 [I 2022-10-10 15:10:18] ...trap/engines/engine.py.229: process: 0.87584 | load: 0.00295 [I 2022-10-10 15:10:18] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-10 15:10:18] ...optimizers/trijoint.py.151: total_norm: 15.113524 [I 2022-10-10 15:10:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 710/2304 [I 2022-10-10 15:10:26] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:46 [I 2022-10-10 15:10:26] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00297 [I 2022-10-10 15:10:26] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-10 15:10:26] ...optimizers/trijoint.py.151: total_norm: 10.023556 [I 2022-10-10 15:10:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 720/2304 [I 2022-10-10 15:10:35] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:36 [I 2022-10-10 15:10:35] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00297 [I 2022-10-10 15:10:35] ...trap/engines/engine.py.230: loss: 0.13515 [I 2022-10-10 15:10:35] ...optimizers/trijoint.py.151: total_norm: 20.226450 [I 2022-10-10 15:10:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 730/2304 [I 2022-10-10 15:10:44] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:21 [I 2022-10-10 15:10:44] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00296 [I 2022-10-10 15:10:44] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-10 15:10:44] ...optimizers/trijoint.py.151: total_norm: 18.413092 [I 2022-10-10 15:10:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 740/2304 [I 2022-10-10 15:10:52] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:22:15 [I 2022-10-10 15:10:52] ...trap/engines/engine.py.229: process: 0.83920 | load: 0.00292 [I 2022-10-10 15:10:52] ...trap/engines/engine.py.230: loss: 0.08943 [I 2022-10-10 15:10:52] ...optimizers/trijoint.py.151: total_norm: 10.483475 [I 2022-10-10 15:11:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 750/2304 [I 2022-10-10 15:11:01] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:58 [I 2022-10-10 15:11:01] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00295 [I 2022-10-10 15:11:01] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-10 15:11:01] ...optimizers/trijoint.py.151: total_norm: 9.134416 [I 2022-10-10 15:11:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 760/2304 [I 2022-10-10 15:11:09] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:22:01 [I 2022-10-10 15:11:09] ...trap/engines/engine.py.229: process: 0.85691 | load: 0.00310 [I 2022-10-10 15:11:09] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-10 15:11:09] ...optimizers/trijoint.py.151: total_norm: 9.206613 [I 2022-10-10 15:11:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 770/2304 [I 2022-10-10 15:11:18] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:54 [I 2022-10-10 15:11:18] ...trap/engines/engine.py.229: process: 0.86195 | load: 0.00296 [I 2022-10-10 15:11:18] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-10 15:11:18] ...optimizers/trijoint.py.151: total_norm: 10.112166 [I 2022-10-10 15:11:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 780/2304 [I 2022-10-10 15:11:26] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:51 [I 2022-10-10 15:11:26] ...trap/engines/engine.py.229: process: 0.87004 | load: 0.00298 [I 2022-10-10 15:11:26] ...trap/engines/engine.py.230: loss: 0.18528 [I 2022-10-10 15:11:26] ...optimizers/trijoint.py.151: total_norm: 27.389889 [I 2022-10-10 15:11:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 790/2304 [I 2022-10-10 15:11:35] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:31 [I 2022-10-10 15:11:35] ...trap/engines/engine.py.229: process: 0.85108 | load: 0.00294 [I 2022-10-10 15:11:35] ...trap/engines/engine.py.230: loss: 0.13920 [I 2022-10-10 15:11:35] ...optimizers/trijoint.py.151: total_norm: 12.963988 [I 2022-10-10 15:11:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 800/2304 [I 2022-10-10 15:11:44] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:28 [I 2022-10-10 15:11:44] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00335 [I 2022-10-10 15:11:44] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-10 15:11:44] ...optimizers/trijoint.py.151: total_norm: 28.536795 [I 2022-10-10 15:11:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 810/2304 [I 2022-10-10 15:11:52] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:34 [I 2022-10-10 15:11:52] ...trap/engines/engine.py.229: process: 0.87989 | load: 0.00318 [I 2022-10-10 15:11:52] ...trap/engines/engine.py.230: loss: 0.11796 [I 2022-10-10 15:11:52] ...optimizers/trijoint.py.151: total_norm: 21.272051 [I 2022-10-10 15:12:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 820/2304 [I 2022-10-10 15:12:01] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:21:21 [I 2022-10-10 15:12:01] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00326 [I 2022-10-10 15:12:01] ...trap/engines/engine.py.230: loss: 0.10226 [I 2022-10-10 15:12:01] ...optimizers/trijoint.py.151: total_norm: 9.860731 [I 2022-10-10 15:12:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 830/2304 [I 2022-10-10 15:12:10] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:21:16 [I 2022-10-10 15:12:10] ...trap/engines/engine.py.229: process: 0.85658 | load: 0.00311 [I 2022-10-10 15:12:10] ...trap/engines/engine.py.230: loss: 0.14582 [I 2022-10-10 15:12:10] ...optimizers/trijoint.py.151: total_norm: 14.497635 [I 2022-10-10 15:12:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 840/2304 [I 2022-10-10 15:12:18] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:21:07 [I 2022-10-10 15:12:18] ...trap/engines/engine.py.229: process: 0.86883 | load: 0.00305 [I 2022-10-10 15:12:18] ...trap/engines/engine.py.230: loss: 0.10618 [I 2022-10-10 15:12:18] ...optimizers/trijoint.py.151: total_norm: 9.584892 [I 2022-10-10 15:12:27] ...trap/engines/engine.py.224: train: epoch 76 | batch 850/2304 [I 2022-10-10 15:12:27] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:21:03 [I 2022-10-10 15:12:27] ...trap/engines/engine.py.229: process: 0.86928 | load: 0.00305 [I 2022-10-10 15:12:27] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-10 15:12:27] ...optimizers/trijoint.py.151: total_norm: 14.054235 [I 2022-10-10 15:12:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 860/2304 [I 2022-10-10 15:12:36] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:48 [I 2022-10-10 15:12:36] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00315 [I 2022-10-10 15:12:36] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-10 15:12:36] ...optimizers/trijoint.py.151: total_norm: 63.250145 [I 2022-10-10 15:12:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 870/2304 [I 2022-10-10 15:12:44] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:41 [I 2022-10-10 15:12:44] ...trap/engines/engine.py.229: process: 0.87209 | load: 0.00305 [I 2022-10-10 15:12:44] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-10 15:12:44] ...optimizers/trijoint.py.151: total_norm: 17.022867 [I 2022-10-10 15:12:53] ...trap/engines/engine.py.224: train: epoch 76 | batch 880/2304 [I 2022-10-10 15:12:53] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:37 [I 2022-10-10 15:12:53] ...trap/engines/engine.py.229: process: 0.88136 | load: 0.00306 [I 2022-10-10 15:12:53] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-10 15:12:53] ...optimizers/trijoint.py.151: total_norm: 10.844660 [I 2022-10-10 15:13:02] ...trap/engines/engine.py.224: train: epoch 76 | batch 890/2304 [I 2022-10-10 15:13:02] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:14 [I 2022-10-10 15:13:02] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00315 [I 2022-10-10 15:13:02] ...trap/engines/engine.py.230: loss: 0.13105 [I 2022-10-10 15:13:02] ...optimizers/trijoint.py.151: total_norm: 23.042767 [I 2022-10-10 15:13:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 900/2304 [I 2022-10-10 15:13:10] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:20:20 [I 2022-10-10 15:13:10] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00312 [I 2022-10-10 15:13:10] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-10 15:13:10] ...optimizers/trijoint.py.151: total_norm: 15.464508 [I 2022-10-10 15:13:19] ...trap/engines/engine.py.224: train: epoch 76 | batch 910/2304 [I 2022-10-10 15:13:19] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:20:01 [I 2022-10-10 15:13:19] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00310 [I 2022-10-10 15:13:19] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-10 15:13:19] ...optimizers/trijoint.py.151: total_norm: 18.037685 [I 2022-10-10 15:13:28] ...trap/engines/engine.py.224: train: epoch 76 | batch 920/2304 [I 2022-10-10 15:13:28] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:57 [I 2022-10-10 15:13:28] ...trap/engines/engine.py.229: process: 0.85879 | load: 0.00309 [I 2022-10-10 15:13:28] ...trap/engines/engine.py.230: loss: 0.20652 [I 2022-10-10 15:13:28] ...optimizers/trijoint.py.151: total_norm: 11.000278 [I 2022-10-10 15:13:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 930/2304 [I 2022-10-10 15:13:36] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:49 [I 2022-10-10 15:13:36] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00319 [I 2022-10-10 15:13:36] ...trap/engines/engine.py.230: loss: 0.08689 [I 2022-10-10 15:13:36] ...optimizers/trijoint.py.151: total_norm: 8.124690 [I 2022-10-10 15:13:45] ...trap/engines/engine.py.224: train: epoch 76 | batch 940/2304 [I 2022-10-10 15:13:45] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:50 [I 2022-10-10 15:13:45] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00321 [I 2022-10-10 15:13:45] ...trap/engines/engine.py.230: loss: 0.21762 [I 2022-10-10 15:13:45] ...optimizers/trijoint.py.151: total_norm: 27.573143 [I 2022-10-10 15:13:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 950/2304 [I 2022-10-10 15:13:54] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:29 [I 2022-10-10 15:13:54] ...trap/engines/engine.py.229: process: 0.86824 | load: 0.00318 [I 2022-10-10 15:13:54] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-10 15:13:54] ...optimizers/trijoint.py.151: total_norm: 16.106144 [I 2022-10-10 15:14:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 960/2304 [I 2022-10-10 15:14:03] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:21 [I 2022-10-10 15:14:03] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00319 [I 2022-10-10 15:14:03] ...trap/engines/engine.py.230: loss: 0.14763 [I 2022-10-10 15:14:03] ...optimizers/trijoint.py.151: total_norm: 16.235840 [I 2022-10-10 15:14:11] ...trap/engines/engine.py.224: train: epoch 76 | batch 970/2304 [I 2022-10-10 15:14:11] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:19:18 [I 2022-10-10 15:14:11] ...trap/engines/engine.py.229: process: 0.86634 | load: 0.00311 [I 2022-10-10 15:14:11] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-10 15:14:11] ...optimizers/trijoint.py.151: total_norm: 12.984608 [I 2022-10-10 15:14:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 980/2304 [I 2022-10-10 15:14:20] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:19:06 [I 2022-10-10 15:14:20] ...trap/engines/engine.py.229: process: 0.86972 | load: 0.00307 [I 2022-10-10 15:14:20] ...trap/engines/engine.py.230: loss: 0.18414 [I 2022-10-10 15:14:20] ...optimizers/trijoint.py.151: total_norm: 41.862881 [I 2022-10-10 15:14:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 990/2304 [I 2022-10-10 15:14:29] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:57 [I 2022-10-10 15:14:29] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00316 [I 2022-10-10 15:14:29] ...trap/engines/engine.py.230: loss: 0.18701 [I 2022-10-10 15:14:29] ...optimizers/trijoint.py.151: total_norm: 15.165231 [I 2022-10-10 15:14:37] ...trap/engines/engine.py.224: train: epoch 76 | batch 1000/2304 [I 2022-10-10 15:14:37] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:46 [I 2022-10-10 15:14:37] ...trap/engines/engine.py.229: process: 0.86644 | load: 0.00312 [I 2022-10-10 15:14:37] ...trap/engines/engine.py.230: loss: 0.10306 [I 2022-10-10 15:14:37] ...optimizers/trijoint.py.151: total_norm: 17.543751 [I 2022-10-10 15:14:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1010/2304 [I 2022-10-10 15:14:46] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:40 [I 2022-10-10 15:14:46] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00305 [I 2022-10-10 15:14:46] ...trap/engines/engine.py.230: loss: 0.17703 [I 2022-10-10 15:14:46] ...optimizers/trijoint.py.151: total_norm: 62.215050 [I 2022-10-10 15:14:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1020/2304 [I 2022-10-10 15:14:55] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:31 [I 2022-10-10 15:14:55] ...trap/engines/engine.py.229: process: 0.86611 | load: 0.00314 [I 2022-10-10 15:14:55] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-10 15:14:55] ...optimizers/trijoint.py.151: total_norm: 8.753977 [I 2022-10-10 15:15:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1030/2304 [I 2022-10-10 15:15:03] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:18:24 [I 2022-10-10 15:15:03] ...trap/engines/engine.py.229: process: 0.86208 | load: 0.00321 [I 2022-10-10 15:15:03] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-10 15:15:03] ...optimizers/trijoint.py.151: total_norm: 9.381280 [I 2022-10-10 15:15:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1040/2304 [I 2022-10-10 15:15:12] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:18:10 [I 2022-10-10 15:15:12] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00312 [I 2022-10-10 15:15:12] ...trap/engines/engine.py.230: loss: 0.14060 [I 2022-10-10 15:15:12] ...optimizers/trijoint.py.151: total_norm: 31.042500 [I 2022-10-10 15:15:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1050/2304 [I 2022-10-10 15:15:21] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:18:01 [I 2022-10-10 15:15:21] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00313 [I 2022-10-10 15:15:21] ...trap/engines/engine.py.230: loss: 0.11963 [I 2022-10-10 15:15:21] ...optimizers/trijoint.py.151: total_norm: 14.664134 [I 2022-10-10 15:15:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1060/2304 [I 2022-10-10 15:15:29] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:52 [I 2022-10-10 15:15:29] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00315 [I 2022-10-10 15:15:29] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-10 15:15:29] ...optimizers/trijoint.py.151: total_norm: 11.585772 [I 2022-10-10 15:15:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1070/2304 [I 2022-10-10 15:15:38] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:39 [I 2022-10-10 15:15:38] ...trap/engines/engine.py.229: process: 0.87100 | load: 0.00312 [I 2022-10-10 15:15:38] ...trap/engines/engine.py.230: loss: 0.12165 [I 2022-10-10 15:15:38] ...optimizers/trijoint.py.151: total_norm: 30.091980 [I 2022-10-10 15:15:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1080/2304 [I 2022-10-10 15:15:46] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:36 [I 2022-10-10 15:15:46] ...trap/engines/engine.py.229: process: 0.87155 | load: 0.00309 [I 2022-10-10 15:15:46] ...trap/engines/engine.py.230: loss: 0.13530 [I 2022-10-10 15:15:46] ...optimizers/trijoint.py.151: total_norm: 9.254565 [I 2022-10-10 15:15:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1090/2304 [I 2022-10-10 15:15:55] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:11 [I 2022-10-10 15:15:55] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00292 [I 2022-10-10 15:15:55] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-10 15:15:55] ...optimizers/trijoint.py.151: total_norm: 19.269323 [I 2022-10-10 15:16:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1100/2304 [I 2022-10-10 15:16:03] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:17:08 [I 2022-10-10 15:16:03] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00303 [I 2022-10-10 15:16:03] ...trap/engines/engine.py.230: loss: 0.10260 [I 2022-10-10 15:16:03] ...optimizers/trijoint.py.151: total_norm: 12.203279 [I 2022-10-10 15:16:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1110/2304 [I 2022-10-10 15:16:12] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:58 [I 2022-10-10 15:16:12] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00293 [I 2022-10-10 15:16:12] ...trap/engines/engine.py.230: loss: 0.19226 [I 2022-10-10 15:16:12] ...optimizers/trijoint.py.151: total_norm: 19.249823 [I 2022-10-10 15:16:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1120/2304 [I 2022-10-10 15:16:21] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:51 [I 2022-10-10 15:16:21] ...trap/engines/engine.py.229: process: 0.84570 | load: 0.00296 [I 2022-10-10 15:16:21] ...trap/engines/engine.py.230: loss: 0.11099 [I 2022-10-10 15:16:21] ...optimizers/trijoint.py.151: total_norm: 20.867329 [I 2022-10-10 15:16:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1130/2304 [I 2022-10-10 15:16:29] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:38 [I 2022-10-10 15:16:29] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00292 [I 2022-10-10 15:16:29] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-10 15:16:29] ...optimizers/trijoint.py.151: total_norm: 17.183668 [I 2022-10-10 15:16:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1140/2304 [I 2022-10-10 15:16:38] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:33 [I 2022-10-10 15:16:38] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00294 [I 2022-10-10 15:16:38] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-10 15:16:38] ...optimizers/trijoint.py.151: total_norm: 7.222251 [I 2022-10-10 15:16:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1150/2304 [I 2022-10-10 15:16:46] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:29 [I 2022-10-10 15:16:46] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00292 [I 2022-10-10 15:16:46] ...trap/engines/engine.py.230: loss: 0.17388 [I 2022-10-10 15:16:46] ...optimizers/trijoint.py.151: total_norm: 23.393589 [I 2022-10-10 15:16:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1160/2304 [I 2022-10-10 15:16:55] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:11 [I 2022-10-10 15:16:55] ...trap/engines/engine.py.229: process: 0.83968 | load: 0.00293 [I 2022-10-10 15:16:55] ...trap/engines/engine.py.230: loss: 0.11173 [I 2022-10-10 15:16:55] ...optimizers/trijoint.py.151: total_norm: 15.647147 [I 2022-10-10 15:17:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1170/2304 [I 2022-10-10 15:17:03] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:12 [I 2022-10-10 15:17:03] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00298 [I 2022-10-10 15:17:03] ...trap/engines/engine.py.230: loss: 0.14640 [I 2022-10-10 15:17:03] ...optimizers/trijoint.py.151: total_norm: 19.257305 [I 2022-10-10 15:17:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1180/2304 [I 2022-10-10 15:17:12] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:16:00 [I 2022-10-10 15:17:12] ...trap/engines/engine.py.229: process: 0.85941 | load: 0.00291 [I 2022-10-10 15:17:12] ...trap/engines/engine.py.230: loss: 0.13584 [I 2022-10-10 15:17:12] ...optimizers/trijoint.py.151: total_norm: 45.564171 [I 2022-10-10 15:17:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1190/2304 [I 2022-10-10 15:17:21] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:55 [I 2022-10-10 15:17:21] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00292 [I 2022-10-10 15:17:21] ...trap/engines/engine.py.230: loss: 0.20163 [I 2022-10-10 15:17:21] ...optimizers/trijoint.py.151: total_norm: 11.375509 [I 2022-10-10 15:17:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1200/2304 [I 2022-10-10 15:17:29] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:49 [I 2022-10-10 15:17:29] ...trap/engines/engine.py.229: process: 0.86358 | load: 0.00293 [I 2022-10-10 15:17:29] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-10 15:17:29] ...optimizers/trijoint.py.151: total_norm: 22.609636 [I 2022-10-10 15:17:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1210/2304 [I 2022-10-10 15:17:38] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:41 [I 2022-10-10 15:17:38] ...trap/engines/engine.py.229: process: 0.85966 | load: 0.00299 [I 2022-10-10 15:17:38] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-10 15:17:38] ...optimizers/trijoint.py.151: total_norm: 8.487473 [I 2022-10-10 15:17:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 1220/2304 [I 2022-10-10 15:17:47] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:39 [I 2022-10-10 15:17:47] ...trap/engines/engine.py.229: process: 0.87796 | load: 0.00295 [I 2022-10-10 15:17:47] ...trap/engines/engine.py.230: loss: 0.20354 [I 2022-10-10 15:17:47] ...optimizers/trijoint.py.151: total_norm: 26.888262 [I 2022-10-10 15:17:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1230/2304 [I 2022-10-10 15:17:55] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:22 [I 2022-10-10 15:17:55] ...trap/engines/engine.py.229: process: 0.87198 | load: 0.00291 [I 2022-10-10 15:17:55] ...trap/engines/engine.py.230: loss: 0.15877 [I 2022-10-10 15:17:55] ...optimizers/trijoint.py.151: total_norm: 17.725828 [I 2022-10-10 15:18:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1240/2304 [I 2022-10-10 15:18:04] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:02 [I 2022-10-10 15:18:04] ...trap/engines/engine.py.229: process: 0.83386 | load: 0.00296 [I 2022-10-10 15:18:04] ...trap/engines/engine.py.230: loss: 0.12652 [I 2022-10-10 15:18:04] ...optimizers/trijoint.py.151: total_norm: 12.749286 [I 2022-10-10 15:18:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1250/2304 [I 2022-10-10 15:18:12] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:14:57 [I 2022-10-10 15:18:12] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00294 [I 2022-10-10 15:18:12] ...trap/engines/engine.py.230: loss: 0.13174 [I 2022-10-10 15:18:12] ...optimizers/trijoint.py.151: total_norm: 12.470317 [I 2022-10-10 15:18:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1260/2304 [I 2022-10-10 15:18:21] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:47 [I 2022-10-10 15:18:21] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00296 [I 2022-10-10 15:18:21] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-10 15:18:21] ...optimizers/trijoint.py.151: total_norm: 16.954165 [I 2022-10-10 15:18:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1270/2304 [I 2022-10-10 15:18:29] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:45 [I 2022-10-10 15:18:29] ...trap/engines/engine.py.229: process: 0.86712 | load: 0.00292 [I 2022-10-10 15:18:29] ...trap/engines/engine.py.230: loss: 0.09624 [I 2022-10-10 15:18:29] ...optimizers/trijoint.py.151: total_norm: 13.265080 [I 2022-10-10 15:18:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1280/2304 [I 2022-10-10 15:18:38] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:35 [I 2022-10-10 15:18:38] ...trap/engines/engine.py.229: process: 0.84465 | load: 0.00296 [I 2022-10-10 15:18:38] ...trap/engines/engine.py.230: loss: 0.13062 [I 2022-10-10 15:18:38] ...optimizers/trijoint.py.151: total_norm: 20.725151 [I 2022-10-10 15:18:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1290/2304 [I 2022-10-10 15:18:46] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:30 [I 2022-10-10 15:18:46] ...trap/engines/engine.py.229: process: 0.88136 | load: 0.00300 [I 2022-10-10 15:18:46] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-10 15:18:46] ...optimizers/trijoint.py.151: total_norm: 41.939583 [I 2022-10-10 15:18:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1300/2304 [I 2022-10-10 15:18:55] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:14:16 [I 2022-10-10 15:18:55] ...trap/engines/engine.py.229: process: 0.84737 | load: 0.00300 [I 2022-10-10 15:18:55] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-10 15:18:55] ...optimizers/trijoint.py.151: total_norm: 9.914320 [I 2022-10-10 15:19:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1310/2304 [I 2022-10-10 15:19:04] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:14:08 [I 2022-10-10 15:19:04] ...trap/engines/engine.py.229: process: 0.86801 | load: 0.00294 [I 2022-10-10 15:19:04] ...trap/engines/engine.py.230: loss: 0.12665 [I 2022-10-10 15:19:04] ...optimizers/trijoint.py.151: total_norm: 12.020070 [I 2022-10-10 15:19:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1320/2304 [I 2022-10-10 15:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:13:59 [I 2022-10-10 15:19:12] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00300 [I 2022-10-10 15:19:12] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-10 15:19:12] ...optimizers/trijoint.py.151: total_norm: 11.636505 [I 2022-10-10 15:19:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1330/2304 [I 2022-10-10 15:19:21] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:13:50 [I 2022-10-10 15:19:21] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00294 [I 2022-10-10 15:19:21] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-10 15:19:21] ...optimizers/trijoint.py.151: total_norm: 18.839888 [I 2022-10-10 15:19:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1340/2304 [I 2022-10-10 15:19:29] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:40 [I 2022-10-10 15:19:29] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00295 [I 2022-10-10 15:19:29] ...trap/engines/engine.py.230: loss: 0.21133 [I 2022-10-10 15:19:29] ...optimizers/trijoint.py.151: total_norm: 13.832459 [I 2022-10-10 15:19:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1350/2304 [I 2022-10-10 15:19:38] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:34 [I 2022-10-10 15:19:38] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00290 [I 2022-10-10 15:19:38] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-10 15:19:38] ...optimizers/trijoint.py.151: total_norm: 29.642042 [I 2022-10-10 15:19:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1360/2304 [I 2022-10-10 15:19:46] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:23 [I 2022-10-10 15:19:46] ...trap/engines/engine.py.229: process: 0.83384 | load: 0.00294 [I 2022-10-10 15:19:46] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-10 15:19:46] ...optimizers/trijoint.py.151: total_norm: 6.815992 [I 2022-10-10 15:19:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1370/2304 [I 2022-10-10 15:19:55] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:20 [I 2022-10-10 15:19:55] ...trap/engines/engine.py.229: process: 0.86197 | load: 0.00307 [I 2022-10-10 15:19:55] ...trap/engines/engine.py.230: loss: 0.23521 [I 2022-10-10 15:19:55] ...optimizers/trijoint.py.151: total_norm: 26.266176 [I 2022-10-10 15:20:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1380/2304 [I 2022-10-10 15:20:04] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:13:12 [I 2022-10-10 15:20:04] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00304 [I 2022-10-10 15:20:04] ...trap/engines/engine.py.230: loss: 0.14294 [I 2022-10-10 15:20:04] ...optimizers/trijoint.py.151: total_norm: 24.687557 [I 2022-10-10 15:20:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1390/2304 [I 2022-10-10 15:20:12] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:13:05 [I 2022-10-10 15:20:12] ...trap/engines/engine.py.229: process: 0.85714 | load: 0.00309 [I 2022-10-10 15:20:12] ...trap/engines/engine.py.230: loss: 0.11113 [I 2022-10-10 15:20:12] ...optimizers/trijoint.py.151: total_norm: 10.276166 [I 2022-10-10 15:20:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1400/2304 [I 2022-10-10 15:20:21] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:54 [I 2022-10-10 15:20:21] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00317 [I 2022-10-10 15:20:21] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-10 15:20:21] ...optimizers/trijoint.py.151: total_norm: 13.387587 [I 2022-10-10 15:20:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1410/2304 [I 2022-10-10 15:20:29] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:44 [I 2022-10-10 15:20:29] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00310 [I 2022-10-10 15:20:29] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-10 15:20:29] ...optimizers/trijoint.py.151: total_norm: 15.105926 [I 2022-10-10 15:20:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1420/2304 [I 2022-10-10 15:20:38] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:34 [I 2022-10-10 15:20:38] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00314 [I 2022-10-10 15:20:38] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-10 15:20:38] ...optimizers/trijoint.py.151: total_norm: 15.402016 [I 2022-10-10 15:20:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 1430/2304 [I 2022-10-10 15:20:47] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:26 [I 2022-10-10 15:20:47] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00296 [I 2022-10-10 15:20:47] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-10 15:20:47] ...optimizers/trijoint.py.151: total_norm: 27.650925 [I 2022-10-10 15:20:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1440/2304 [I 2022-10-10 15:20:55] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:20 [I 2022-10-10 15:20:55] ...trap/engines/engine.py.229: process: 0.86007 | load: 0.00297 [I 2022-10-10 15:20:55] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-10 15:20:55] ...optimizers/trijoint.py.151: total_norm: 16.692572 [I 2022-10-10 15:21:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1450/2304 [I 2022-10-10 15:21:04] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:12:18 [I 2022-10-10 15:21:04] ...trap/engines/engine.py.229: process: 0.87074 | load: 0.00308 [I 2022-10-10 15:21:04] ...trap/engines/engine.py.230: loss: 0.08791 [I 2022-10-10 15:21:04] ...optimizers/trijoint.py.151: total_norm: 14.452808 [I 2022-10-10 15:21:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1460/2304 [I 2022-10-10 15:21:12] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:12:11 [I 2022-10-10 15:21:12] ...trap/engines/engine.py.229: process: 0.87157 | load: 0.00307 [I 2022-10-10 15:21:12] ...trap/engines/engine.py.230: loss: 0.10338 [I 2022-10-10 15:21:12] ...optimizers/trijoint.py.151: total_norm: 13.840746 [I 2022-10-10 15:21:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1470/2304 [I 2022-10-10 15:21:21] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:11:59 [I 2022-10-10 15:21:21] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00308 [I 2022-10-10 15:21:21] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-10 15:21:21] ...optimizers/trijoint.py.151: total_norm: 22.700790 [I 2022-10-10 15:21:30] ...trap/engines/engine.py.224: train: epoch 76 | batch 1480/2304 [I 2022-10-10 15:21:30] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:51 [I 2022-10-10 15:21:30] ...trap/engines/engine.py.229: process: 0.86202 | load: 0.00317 [I 2022-10-10 15:21:30] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-10 15:21:30] ...optimizers/trijoint.py.151: total_norm: 13.305623 [I 2022-10-10 15:21:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1490/2304 [I 2022-10-10 15:21:38] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:41 [I 2022-10-10 15:21:38] ...trap/engines/engine.py.229: process: 0.86166 | load: 0.00315 [I 2022-10-10 15:21:38] ...trap/engines/engine.py.230: loss: 0.10131 [I 2022-10-10 15:21:38] ...optimizers/trijoint.py.151: total_norm: 15.420431 [I 2022-10-10 15:21:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 1500/2304 [I 2022-10-10 15:21:47] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:11:30 [I 2022-10-10 15:21:47] ...trap/engines/engine.py.229: process: 0.85649 | load: 0.00312 [I 2022-10-10 15:21:47] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-10 15:21:47] ...optimizers/trijoint.py.151: total_norm: 17.993401 [I 2022-10-10 15:21:56] ...trap/engines/engine.py.224: train: epoch 76 | batch 1510/2304 [I 2022-10-10 15:21:56] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:11:23 [I 2022-10-10 15:21:56] ...trap/engines/engine.py.229: process: 0.86157 | load: 0.00313 [I 2022-10-10 15:21:56] ...trap/engines/engine.py.230: loss: 0.12662 [I 2022-10-10 15:21:56] ...optimizers/trijoint.py.151: total_norm: 14.460679 [I 2022-10-10 15:22:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1520/2304 [I 2022-10-10 15:22:04] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:11 [I 2022-10-10 15:22:04] ...trap/engines/engine.py.229: process: 0.86218 | load: 0.00310 [I 2022-10-10 15:22:04] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-10 15:22:04] ...optimizers/trijoint.py.151: total_norm: 13.389004 [I 2022-10-10 15:22:13] ...trap/engines/engine.py.224: train: epoch 76 | batch 1530/2304 [I 2022-10-10 15:22:13] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:11:08 [I 2022-10-10 15:22:13] ...trap/engines/engine.py.229: process: 0.86651 | load: 0.00312 [I 2022-10-10 15:22:13] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-10 15:22:13] ...optimizers/trijoint.py.151: total_norm: 13.430522 [I 2022-10-10 15:22:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 1540/2304 [I 2022-10-10 15:22:22] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:59 [I 2022-10-10 15:22:22] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00305 [I 2022-10-10 15:22:22] ...trap/engines/engine.py.230: loss: 0.15678 [I 2022-10-10 15:22:22] ...optimizers/trijoint.py.151: total_norm: 16.928926 [I 2022-10-10 15:22:30] ...trap/engines/engine.py.224: train: epoch 76 | batch 1550/2304 [I 2022-10-10 15:22:30] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:50 [I 2022-10-10 15:22:30] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00311 [I 2022-10-10 15:22:30] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-10 15:22:30] ...optimizers/trijoint.py.151: total_norm: 9.658214 [I 2022-10-10 15:22:39] ...trap/engines/engine.py.224: train: epoch 76 | batch 1560/2304 [I 2022-10-10 15:22:39] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:42 [I 2022-10-10 15:22:39] ...trap/engines/engine.py.229: process: 0.86414 | load: 0.00309 [I 2022-10-10 15:22:39] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-10 15:22:39] ...optimizers/trijoint.py.151: total_norm: 16.059477 [I 2022-10-10 15:22:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 1570/2304 [I 2022-10-10 15:22:48] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:37 [I 2022-10-10 15:22:48] ...trap/engines/engine.py.229: process: 0.87091 | load: 0.00313 [I 2022-10-10 15:22:48] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-10 15:22:48] ...optimizers/trijoint.py.151: total_norm: 15.348948 [I 2022-10-10 15:22:56] ...trap/engines/engine.py.224: train: epoch 76 | batch 1580/2304 [I 2022-10-10 15:22:56] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:29 [I 2022-10-10 15:22:56] ...trap/engines/engine.py.229: process: 0.86821 | load: 0.00314 [I 2022-10-10 15:22:56] ...trap/engines/engine.py.230: loss: 0.10793 [I 2022-10-10 15:22:56] ...optimizers/trijoint.py.151: total_norm: 12.235444 [I 2022-10-10 15:23:05] ...trap/engines/engine.py.224: train: epoch 76 | batch 1590/2304 [I 2022-10-10 15:23:05] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:10:17 [I 2022-10-10 15:23:05] ...trap/engines/engine.py.229: process: 0.86702 | load: 0.00313 [I 2022-10-10 15:23:05] ...trap/engines/engine.py.230: loss: 0.11328 [I 2022-10-10 15:23:05] ...optimizers/trijoint.py.151: total_norm: 9.766440 [I 2022-10-10 15:23:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 1600/2304 [I 2022-10-10 15:23:14] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:10 [I 2022-10-10 15:23:14] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00306 [I 2022-10-10 15:23:14] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-10 15:23:14] ...optimizers/trijoint.py.151: total_norm: 13.510795 [I 2022-10-10 15:23:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 1610/2304 [I 2022-10-10 15:23:22] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:59 [I 2022-10-10 15:23:22] ...trap/engines/engine.py.229: process: 0.87522 | load: 0.00347 [I 2022-10-10 15:23:22] ...trap/engines/engine.py.230: loss: 0.16239 [I 2022-10-10 15:23:22] ...optimizers/trijoint.py.151: total_norm: 19.768768 [I 2022-10-10 15:23:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 1620/2304 [I 2022-10-10 15:23:31] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:49 [I 2022-10-10 15:23:31] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00308 [I 2022-10-10 15:23:31] ...trap/engines/engine.py.230: loss: 0.16206 [I 2022-10-10 15:23:31] ...optimizers/trijoint.py.151: total_norm: 16.893204 [I 2022-10-10 15:23:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1630/2304 [I 2022-10-10 15:23:40] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:45 [I 2022-10-10 15:23:40] ...trap/engines/engine.py.229: process: 0.87260 | load: 0.00306 [I 2022-10-10 15:23:40] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-10 15:23:40] ...optimizers/trijoint.py.151: total_norm: 28.221325 [I 2022-10-10 15:23:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 1640/2304 [I 2022-10-10 15:23:48] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:32 [I 2022-10-10 15:23:48] ...trap/engines/engine.py.229: process: 0.87126 | load: 0.00304 [I 2022-10-10 15:23:48] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-10 15:23:48] ...optimizers/trijoint.py.151: total_norm: 15.317014 [I 2022-10-10 15:23:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 1650/2304 [I 2022-10-10 15:23:57] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:25 [I 2022-10-10 15:23:57] ...trap/engines/engine.py.229: process: 0.87341 | load: 0.00313 [I 2022-10-10 15:23:57] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-10 15:23:57] ...optimizers/trijoint.py.151: total_norm: 11.632190 [I 2022-10-10 15:24:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 1660/2304 [I 2022-10-10 15:24:06] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:17 [I 2022-10-10 15:24:06] ...trap/engines/engine.py.229: process: 0.86699 | load: 0.00310 [I 2022-10-10 15:24:06] ...trap/engines/engine.py.230: loss: 0.11745 [I 2022-10-10 15:24:06] ...optimizers/trijoint.py.151: total_norm: 22.452568 [I 2022-10-10 15:24:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 1670/2304 [I 2022-10-10 15:24:14] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:09:07 [I 2022-10-10 15:24:14] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00306 [I 2022-10-10 15:24:14] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-10 15:24:14] ...optimizers/trijoint.py.151: total_norm: 24.958014 [I 2022-10-10 15:24:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 1680/2304 [I 2022-10-10 15:24:23] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:55 [I 2022-10-10 15:24:23] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00310 [I 2022-10-10 15:24:23] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-10 15:24:23] ...optimizers/trijoint.py.151: total_norm: 10.220576 [I 2022-10-10 15:24:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 1690/2304 [I 2022-10-10 15:24:32] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:49 [I 2022-10-10 15:24:32] ...trap/engines/engine.py.229: process: 0.85926 | load: 0.00310 [I 2022-10-10 15:24:32] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-10 15:24:32] ...optimizers/trijoint.py.151: total_norm: 16.668045 [I 2022-10-10 15:24:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1700/2304 [I 2022-10-10 15:24:40] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:43 [I 2022-10-10 15:24:40] ...trap/engines/engine.py.229: process: 0.87079 | load: 0.00308 [I 2022-10-10 15:24:40] ...trap/engines/engine.py.230: loss: 0.09286 [I 2022-10-10 15:24:40] ...optimizers/trijoint.py.151: total_norm: 10.823814 [I 2022-10-10 15:24:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 1710/2304 [I 2022-10-10 15:24:49] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:29 [I 2022-10-10 15:24:49] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00307 [I 2022-10-10 15:24:49] ...trap/engines/engine.py.230: loss: 0.14616 [I 2022-10-10 15:24:49] ...optimizers/trijoint.py.151: total_norm: 13.481106 [I 2022-10-10 15:24:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 1720/2304 [I 2022-10-10 15:24:57] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:23 [I 2022-10-10 15:24:57] ...trap/engines/engine.py.229: process: 0.86041 | load: 0.00309 [I 2022-10-10 15:24:57] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-10 15:24:57] ...optimizers/trijoint.py.151: total_norm: 17.388952 [I 2022-10-10 15:25:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 1730/2304 [I 2022-10-10 15:25:06] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:08 [I 2022-10-10 15:25:06] ...trap/engines/engine.py.229: process: 0.85168 | load: 0.00291 [I 2022-10-10 15:25:06] ...trap/engines/engine.py.230: loss: 0.16664 [I 2022-10-10 15:25:06] ...optimizers/trijoint.py.151: total_norm: 15.203963 [I 2022-10-10 15:25:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 1740/2304 [I 2022-10-10 15:25:15] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:08:03 [I 2022-10-10 15:25:15] ...trap/engines/engine.py.229: process: 0.86732 | load: 0.00298 [I 2022-10-10 15:25:15] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-10 15:25:15] ...optimizers/trijoint.py.151: total_norm: 28.158094 [I 2022-10-10 15:25:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 1750/2304 [I 2022-10-10 15:25:23] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:56 [I 2022-10-10 15:25:23] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00295 [I 2022-10-10 15:25:23] ...trap/engines/engine.py.230: loss: 0.13498 [I 2022-10-10 15:25:23] ...optimizers/trijoint.py.151: total_norm: 17.055956 [I 2022-10-10 15:25:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 1760/2304 [I 2022-10-10 15:25:32] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:46 [I 2022-10-10 15:25:32] ...trap/engines/engine.py.229: process: 0.86800 | load: 0.00303 [I 2022-10-10 15:25:32] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-10 15:25:32] ...optimizers/trijoint.py.151: total_norm: 12.342196 [I 2022-10-10 15:25:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1770/2304 [I 2022-10-10 15:25:40] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:36 [I 2022-10-10 15:25:40] ...trap/engines/engine.py.229: process: 0.85488 | load: 0.00296 [I 2022-10-10 15:25:40] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-10 15:25:40] ...optimizers/trijoint.py.151: total_norm: 11.230867 [I 2022-10-10 15:25:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 1780/2304 [I 2022-10-10 15:25:49] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:29 [I 2022-10-10 15:25:49] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00300 [I 2022-10-10 15:25:49] ...trap/engines/engine.py.230: loss: 0.10340 [I 2022-10-10 15:25:49] ...optimizers/trijoint.py.151: total_norm: 9.160380 [I 2022-10-10 15:25:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 1790/2304 [I 2022-10-10 15:25:58] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:20 [I 2022-10-10 15:25:58] ...trap/engines/engine.py.229: process: 0.87104 | load: 0.00299 [I 2022-10-10 15:25:58] ...trap/engines/engine.py.230: loss: 0.14373 [I 2022-10-10 15:25:58] ...optimizers/trijoint.py.151: total_norm: 12.408916 [I 2022-10-10 15:26:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 1800/2304 [I 2022-10-10 15:26:06] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:10 [I 2022-10-10 15:26:06] ...trap/engines/engine.py.229: process: 0.85745 | load: 0.00310 [I 2022-10-10 15:26:06] ...trap/engines/engine.py.230: loss: 0.18427 [I 2022-10-10 15:26:06] ...optimizers/trijoint.py.151: total_norm: 22.233923 [I 2022-10-10 15:26:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 1810/2304 [I 2022-10-10 15:26:15] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:07:01 [I 2022-10-10 15:26:15] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00298 [I 2022-10-10 15:26:15] ...trap/engines/engine.py.230: loss: 0.10252 [I 2022-10-10 15:26:15] ...optimizers/trijoint.py.151: total_norm: 15.064488 [I 2022-10-10 15:26:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 1820/2304 [I 2022-10-10 15:26:23] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:55 [I 2022-10-10 15:26:23] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00307 [I 2022-10-10 15:26:23] ...trap/engines/engine.py.230: loss: 0.22131 [I 2022-10-10 15:26:23] ...optimizers/trijoint.py.151: total_norm: 21.262241 [I 2022-10-10 15:26:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 1830/2304 [I 2022-10-10 15:26:32] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:45 [I 2022-10-10 15:26:32] ...trap/engines/engine.py.229: process: 0.86520 | load: 0.00308 [I 2022-10-10 15:26:32] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-10 15:26:32] ...optimizers/trijoint.py.151: total_norm: 15.177986 [I 2022-10-10 15:26:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1840/2304 [I 2022-10-10 15:26:40] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:36 [I 2022-10-10 15:26:40] ...trap/engines/engine.py.229: process: 0.86317 | load: 0.00297 [I 2022-10-10 15:26:40] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-10 15:26:40] ...optimizers/trijoint.py.151: total_norm: 18.062542 [I 2022-10-10 15:26:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 1850/2304 [I 2022-10-10 15:26:49] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:28 [I 2022-10-10 15:26:49] ...trap/engines/engine.py.229: process: 0.86728 | load: 0.00308 [I 2022-10-10 15:26:49] ...trap/engines/engine.py.230: loss: 0.15349 [I 2022-10-10 15:26:49] ...optimizers/trijoint.py.151: total_norm: 18.740440 [I 2022-10-10 15:26:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 1860/2304 [I 2022-10-10 15:26:58] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:21 [I 2022-10-10 15:26:58] ...trap/engines/engine.py.229: process: 0.85928 | load: 0.00297 [I 2022-10-10 15:26:58] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-10 15:26:58] ...optimizers/trijoint.py.151: total_norm: 10.032928 [I 2022-10-10 15:27:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 1870/2304 [I 2022-10-10 15:27:06] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:15 [I 2022-10-10 15:27:06] ...trap/engines/engine.py.229: process: 0.87238 | load: 0.00303 [I 2022-10-10 15:27:06] ...trap/engines/engine.py.230: loss: 0.19464 [I 2022-10-10 15:27:06] ...optimizers/trijoint.py.151: total_norm: 12.887001 [I 2022-10-10 15:27:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 1880/2304 [I 2022-10-10 15:27:15] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:06:05 [I 2022-10-10 15:27:15] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00293 [I 2022-10-10 15:27:15] ...trap/engines/engine.py.230: loss: 0.11856 [I 2022-10-10 15:27:15] ...optimizers/trijoint.py.151: total_norm: 36.438801 [I 2022-10-10 15:27:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 1890/2304 [I 2022-10-10 15:27:24] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:56 [I 2022-10-10 15:27:24] ...trap/engines/engine.py.229: process: 0.86367 | load: 0.00322 [I 2022-10-10 15:27:24] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-10 15:27:24] ...optimizers/trijoint.py.151: total_norm: 15.342987 [I 2022-10-10 15:27:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 1900/2304 [I 2022-10-10 15:27:32] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:45 [I 2022-10-10 15:27:32] ...trap/engines/engine.py.229: process: 0.85523 | load: 0.00296 [I 2022-10-10 15:27:32] ...trap/engines/engine.py.230: loss: 0.16795 [I 2022-10-10 15:27:32] ...optimizers/trijoint.py.151: total_norm: 16.351877 [I 2022-10-10 15:27:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 1910/2304 [I 2022-10-10 15:27:41] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:39 [I 2022-10-10 15:27:41] ...trap/engines/engine.py.229: process: 0.86224 | load: 0.00295 [I 2022-10-10 15:27:41] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-10 15:27:41] ...optimizers/trijoint.py.151: total_norm: 8.481373 [I 2022-10-10 15:27:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 1920/2304 [I 2022-10-10 15:27:50] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:30 [I 2022-10-10 15:27:50] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00300 [I 2022-10-10 15:27:50] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-10 15:27:50] ...optimizers/trijoint.py.151: total_norm: 21.496101 [I 2022-10-10 15:27:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 1930/2304 [I 2022-10-10 15:27:58] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:05:21 [I 2022-10-10 15:27:58] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00296 [I 2022-10-10 15:27:58] ...trap/engines/engine.py.230: loss: 0.09704 [I 2022-10-10 15:27:58] ...optimizers/trijoint.py.151: total_norm: 14.519650 [I 2022-10-10 15:28:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 1940/2304 [I 2022-10-10 15:28:07] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:05:10 [I 2022-10-10 15:28:07] ...trap/engines/engine.py.229: process: 0.86551 | load: 0.00293 [I 2022-10-10 15:28:07] ...trap/engines/engine.py.230: loss: 0.15383 [I 2022-10-10 15:28:07] ...optimizers/trijoint.py.151: total_norm: 12.581343 [I 2022-10-10 15:28:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 1950/2304 [I 2022-10-10 15:28:15] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:05:01 [I 2022-10-10 15:28:15] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00294 [I 2022-10-10 15:28:15] ...trap/engines/engine.py.230: loss: 0.10060 [I 2022-10-10 15:28:15] ...optimizers/trijoint.py.151: total_norm: 13.282126 [I 2022-10-10 15:28:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 1960/2304 [I 2022-10-10 15:28:24] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:04:53 [I 2022-10-10 15:28:24] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00298 [I 2022-10-10 15:28:24] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-10 15:28:24] ...optimizers/trijoint.py.151: total_norm: 10.189597 [I 2022-10-10 15:28:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 1970/2304 [I 2022-10-10 15:28:32] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:46 [I 2022-10-10 15:28:32] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00292 [I 2022-10-10 15:28:32] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-10 15:28:32] ...optimizers/trijoint.py.151: total_norm: 15.060491 [I 2022-10-10 15:28:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 1980/2304 [I 2022-10-10 15:28:41] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:38 [I 2022-10-10 15:28:41] ...trap/engines/engine.py.229: process: 0.86344 | load: 0.00298 [I 2022-10-10 15:28:41] ...trap/engines/engine.py.230: loss: 0.10599 [I 2022-10-10 15:28:41] ...optimizers/trijoint.py.151: total_norm: 10.387714 [I 2022-10-10 15:28:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 1990/2304 [I 2022-10-10 15:28:50] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:27 [I 2022-10-10 15:28:50] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00299 [I 2022-10-10 15:28:50] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-10 15:28:50] ...optimizers/trijoint.py.151: total_norm: 8.740237 [I 2022-10-10 15:28:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 2000/2304 [I 2022-10-10 15:28:58] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:19 [I 2022-10-10 15:28:58] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00297 [I 2022-10-10 15:28:58] ...trap/engines/engine.py.230: loss: 0.19359 [I 2022-10-10 15:28:58] ...optimizers/trijoint.py.151: total_norm: 32.106056 [I 2022-10-10 15:29:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 2010/2304 [I 2022-10-10 15:29:07] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:04:12 [I 2022-10-10 15:29:07] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00294 [I 2022-10-10 15:29:07] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-10 15:29:07] ...optimizers/trijoint.py.151: total_norm: 9.001163 [I 2022-10-10 15:29:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 2020/2304 [I 2022-10-10 15:29:15] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:04:04 [I 2022-10-10 15:29:15] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00306 [I 2022-10-10 15:29:15] ...trap/engines/engine.py.230: loss: 0.18277 [I 2022-10-10 15:29:15] ...optimizers/trijoint.py.151: total_norm: 18.069410 [I 2022-10-10 15:29:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 2030/2304 [I 2022-10-10 15:29:24] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:55 [I 2022-10-10 15:29:24] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00302 [I 2022-10-10 15:29:24] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-10 15:29:24] ...optimizers/trijoint.py.151: total_norm: 17.657179 [I 2022-10-10 15:29:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2040/2304 [I 2022-10-10 15:29:33] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:46 [I 2022-10-10 15:29:33] ...trap/engines/engine.py.229: process: 0.86337 | load: 0.00293 [I 2022-10-10 15:29:33] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-10 15:29:33] ...optimizers/trijoint.py.151: total_norm: 11.190468 [I 2022-10-10 15:29:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 2050/2304 [I 2022-10-10 15:29:41] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:36 [I 2022-10-10 15:29:41] ...trap/engines/engine.py.229: process: 0.85670 | load: 0.00301 [I 2022-10-10 15:29:41] ...trap/engines/engine.py.230: loss: 0.08885 [I 2022-10-10 15:29:41] ...optimizers/trijoint.py.151: total_norm: 9.685661 [I 2022-10-10 15:29:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 2060/2304 [I 2022-10-10 15:29:50] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:28 [I 2022-10-10 15:29:50] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00301 [I 2022-10-10 15:29:50] ...trap/engines/engine.py.230: loss: 0.08753 [I 2022-10-10 15:29:50] ...optimizers/trijoint.py.151: total_norm: 14.004465 [I 2022-10-10 15:29:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 2070/2304 [I 2022-10-10 15:29:58] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:03:20 [I 2022-10-10 15:29:58] ...trap/engines/engine.py.229: process: 0.87153 | load: 0.00300 [I 2022-10-10 15:29:58] ...trap/engines/engine.py.230: loss: 0.11150 [I 2022-10-10 15:29:58] ...optimizers/trijoint.py.151: total_norm: 29.929705 [I 2022-10-10 15:30:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 2080/2304 [I 2022-10-10 15:30:07] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:11 [I 2022-10-10 15:30:07] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00293 [I 2022-10-10 15:30:07] ...trap/engines/engine.py.230: loss: 0.16788 [I 2022-10-10 15:30:07] ...optimizers/trijoint.py.151: total_norm: 13.760734 [I 2022-10-10 15:30:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 2090/2304 [I 2022-10-10 15:30:16] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:03:03 [I 2022-10-10 15:30:16] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00301 [I 2022-10-10 15:30:16] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-10 15:30:16] ...optimizers/trijoint.py.151: total_norm: 19.328588 [I 2022-10-10 15:30:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 2100/2304 [I 2022-10-10 15:30:24] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:54 [I 2022-10-10 15:30:24] ...trap/engines/engine.py.229: process: 0.85826 | load: 0.00298 [I 2022-10-10 15:30:24] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-10 15:30:24] ...optimizers/trijoint.py.151: total_norm: 20.257854 [I 2022-10-10 15:30:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2110/2304 [I 2022-10-10 15:30:33] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:46 [I 2022-10-10 15:30:33] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00299 [I 2022-10-10 15:30:33] ...trap/engines/engine.py.230: loss: 0.13180 [I 2022-10-10 15:30:33] ...optimizers/trijoint.py.151: total_norm: 16.820284 [I 2022-10-10 15:30:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 2120/2304 [I 2022-10-10 15:30:41] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:37 [I 2022-10-10 15:30:41] ...trap/engines/engine.py.229: process: 0.85553 | load: 0.00298 [I 2022-10-10 15:30:41] ...trap/engines/engine.py.230: loss: 0.07675 [I 2022-10-10 15:30:41] ...optimizers/trijoint.py.151: total_norm: 8.273196 [I 2022-10-10 15:30:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 2130/2304 [I 2022-10-10 15:30:50] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:28 [I 2022-10-10 15:30:50] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00306 [I 2022-10-10 15:30:50] ...trap/engines/engine.py.230: loss: 0.11114 [I 2022-10-10 15:30:50] ...optimizers/trijoint.py.151: total_norm: 22.992956 [I 2022-10-10 15:30:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 2140/2304 [I 2022-10-10 15:30:59] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:02:20 [I 2022-10-10 15:30:59] ...trap/engines/engine.py.229: process: 0.86109 | load: 0.00297 [I 2022-10-10 15:30:59] ...trap/engines/engine.py.230: loss: 0.15967 [I 2022-10-10 15:30:59] ...optimizers/trijoint.py.151: total_norm: 18.534519 [I 2022-10-10 15:31:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 2150/2304 [I 2022-10-10 15:31:07] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:11 [I 2022-10-10 15:31:07] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00300 [I 2022-10-10 15:31:07] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-10 15:31:07] ...optimizers/trijoint.py.151: total_norm: 18.817940 [I 2022-10-10 15:31:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 2160/2304 [I 2022-10-10 15:31:16] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:02:03 [I 2022-10-10 15:31:16] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00307 [I 2022-10-10 15:31:16] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-10 15:31:16] ...optimizers/trijoint.py.151: total_norm: 28.058586 [I 2022-10-10 15:31:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 2170/2304 [I 2022-10-10 15:31:24] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:54 [I 2022-10-10 15:31:24] ...trap/engines/engine.py.229: process: 0.86037 | load: 0.00303 [I 2022-10-10 15:31:24] ...trap/engines/engine.py.230: loss: 0.17332 [I 2022-10-10 15:31:24] ...optimizers/trijoint.py.151: total_norm: 30.681997 [I 2022-10-10 15:31:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2180/2304 [I 2022-10-10 15:31:33] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:46 [I 2022-10-10 15:31:33] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00328 [I 2022-10-10 15:31:33] ...trap/engines/engine.py.230: loss: 0.19378 [I 2022-10-10 15:31:33] ...optimizers/trijoint.py.151: total_norm: 23.301947 [I 2022-10-10 15:31:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 2190/2304 [I 2022-10-10 15:31:42] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:37 [I 2022-10-10 15:31:42] ...trap/engines/engine.py.229: process: 0.85980 | load: 0.00301 [I 2022-10-10 15:31:42] ...trap/engines/engine.py.230: loss: 0.20036 [I 2022-10-10 15:31:42] ...optimizers/trijoint.py.151: total_norm: 9.521710 [I 2022-10-10 15:31:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 2200/2304 [I 2022-10-10 15:31:50] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:28 [I 2022-10-10 15:31:50] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00295 [I 2022-10-10 15:31:50] ...trap/engines/engine.py.230: loss: 0.08725 [I 2022-10-10 15:31:50] ...optimizers/trijoint.py.151: total_norm: 26.904411 [I 2022-10-10 15:31:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 2210/2304 [I 2022-10-10 15:31:59] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:20 [I 2022-10-10 15:31:59] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00290 [I 2022-10-10 15:31:59] ...trap/engines/engine.py.230: loss: 0.19656 [I 2022-10-10 15:31:59] ...optimizers/trijoint.py.151: total_norm: 22.046066 [I 2022-10-10 15:32:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 2220/2304 [I 2022-10-10 15:32:07] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:11 [I 2022-10-10 15:32:07] ...trap/engines/engine.py.229: process: 0.83465 | load: 0.00298 [I 2022-10-10 15:32:07] ...trap/engines/engine.py.230: loss: 0.09934 [I 2022-10-10 15:32:07] ...optimizers/trijoint.py.151: total_norm: 6.455420 [I 2022-10-10 15:32:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 2230/2304 [I 2022-10-10 15:32:16] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:01:03 [I 2022-10-10 15:32:16] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00307 [I 2022-10-10 15:32:16] ...trap/engines/engine.py.230: loss: 0.18372 [I 2022-10-10 15:32:16] ...optimizers/trijoint.py.151: total_norm: 16.119646 [I 2022-10-10 15:32:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 2240/2304 [I 2022-10-10 15:32:24] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:54 [I 2022-10-10 15:32:24] ...trap/engines/engine.py.229: process: 0.86883 | load: 0.00311 [I 2022-10-10 15:32:24] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-10 15:32:24] ...optimizers/trijoint.py.151: total_norm: 15.519865 [I 2022-10-10 15:32:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2250/2304 [I 2022-10-10 15:32:33] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:46 [I 2022-10-10 15:32:33] ...trap/engines/engine.py.229: process: 0.86896 | load: 0.00314 [I 2022-10-10 15:32:33] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-10 15:32:33] ...optimizers/trijoint.py.151: total_norm: 14.458708 [I 2022-10-10 15:32:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 2260/2304 [I 2022-10-10 15:32:42] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:37 [I 2022-10-10 15:32:42] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00312 [I 2022-10-10 15:32:42] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-10 15:32:42] ...optimizers/trijoint.py.151: total_norm: 10.514182 [I 2022-10-10 15:32:51] ...trap/engines/engine.py.224: train: epoch 76 | batch 2270/2304 [I 2022-10-10 15:32:51] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:29 [I 2022-10-10 15:32:51] ...trap/engines/engine.py.229: process: 0.87322 | load: 0.00307 [I 2022-10-10 15:32:51] ...trap/engines/engine.py.230: loss: 0.11045 [I 2022-10-10 15:32:51] ...optimizers/trijoint.py.151: total_norm: 10.632638 [I 2022-10-10 15:32:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 2280/2304 [I 2022-10-10 15:32:59] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:20 [I 2022-10-10 15:32:59] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00308 [I 2022-10-10 15:32:59] ...trap/engines/engine.py.230: loss: 0.10447 [I 2022-10-10 15:32:59] ...optimizers/trijoint.py.151: total_norm: 15.690467 [I 2022-10-10 15:33:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 2290/2304 [I 2022-10-10 15:33:08] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:12 [I 2022-10-10 15:33:08] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00307 [I 2022-10-10 15:33:08] ...trap/engines/engine.py.230: loss: 0.14985 [I 2022-10-10 15:33:08] ...optimizers/trijoint.py.151: total_norm: 20.058899 [I 2022-10-10 15:33:17] ...trap/engines/engine.py.224: train: epoch 76 | batch 2300/2304 [I 2022-10-10 15:33:17] ...trap/engines/engine.py.228: elapsed: 0:33:04 | left: 0:00:03 [I 2022-10-10 15:33:17] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00019 [I 2022-10-10 15:33:17] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-10 15:33:17] ...optimizers/trijoint.py.151: total_norm: 14.903801 [I 2022-10-10 15:33:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 2304/2304 [I 2022-10-10 15:33:20] ...trap/engines/engine.py.228: elapsed: 0:33:07 | left: 0:00:00 [I 2022-10-10 15:33:20] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00019 [I 2022-10-10 15:33:20] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-10 15:33:20] ...optimizers/trijoint.py.151: total_norm: 15.484114 [S 2022-10-10 15:33:21] ...trap/engines/engine.py.129: train_epoch.epoch: 76 [S 2022-10-10 15:33:21] ...trap/engines/engine.py.129: train_epoch.loss: 0.13504711277055637 [I 2022-10-10 15:33:21] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 15:33:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 76 [I 2022-10-10 15:33:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 15:33:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 15:33:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 15:33:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 15:33:27] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 15:33:28] ...trap/engines/engine.py.321: eval: epoch 76 | batch 0/510 [I 2022-10-10 15:33:28] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:19 [I 2022-10-10 15:33:28] ...trap/engines/engine.py.326: process: 5.21395 | load: 3.54557 [I 2022-10-10 15:33:33] ...trap/engines/engine.py.321: eval: epoch 76 | batch 10/510 [I 2022-10-10 15:33:33] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:31 [I 2022-10-10 15:33:33] ...trap/engines/engine.py.326: process: 0.49516 | load: 0.00019 [I 2022-10-10 15:33:38] ...trap/engines/engine.py.321: eval: epoch 76 | batch 20/510 [I 2022-10-10 15:33:38] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:11 [I 2022-10-10 15:33:38] ...trap/engines/engine.py.326: process: 0.43468 | load: 0.00019 [I 2022-10-10 15:33:42] ...trap/engines/engine.py.321: eval: epoch 76 | batch 30/510 [I 2022-10-10 15:33:42] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:03:38 [I 2022-10-10 15:33:42] ...trap/engines/engine.py.326: process: 0.43048 | load: 0.00019 [I 2022-10-10 15:33:47] ...trap/engines/engine.py.321: eval: epoch 76 | batch 40/510 [I 2022-10-10 15:33:47] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:25 [I 2022-10-10 15:33:47] ...trap/engines/engine.py.326: process: 0.42065 | load: 0.00019 [I 2022-10-10 15:33:51] ...trap/engines/engine.py.321: eval: epoch 76 | batch 50/510 [I 2022-10-10 15:33:51] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:17 [I 2022-10-10 15:33:51] ...trap/engines/engine.py.326: process: 0.42298 | load: 0.00019 [I 2022-10-10 15:33:56] ...trap/engines/engine.py.321: eval: epoch 76 | batch 60/510 [I 2022-10-10 15:33:56] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:38 [I 2022-10-10 15:33:56] ...trap/engines/engine.py.326: process: 0.52600 | load: 0.00019 [I 2022-10-10 15:34:01] ...trap/engines/engine.py.321: eval: epoch 76 | batch 70/510 [I 2022-10-10 15:34:01] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:05 [I 2022-10-10 15:34:01] ...trap/engines/engine.py.326: process: 0.59218 | load: 0.00020 [I 2022-10-10 15:34:07] ...trap/engines/engine.py.321: eval: epoch 76 | batch 80/510 [I 2022-10-10 15:34:07] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:10 [I 2022-10-10 15:34:07] ...trap/engines/engine.py.326: process: 0.56523 | load: 0.00024 [I 2022-10-10 15:34:12] ...trap/engines/engine.py.321: eval: epoch 76 | batch 90/510 [I 2022-10-10 15:34:12] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:37 [I 2022-10-10 15:34:12] ...trap/engines/engine.py.326: process: 0.49791 | load: 0.00019 [I 2022-10-10 15:34:17] ...trap/engines/engine.py.321: eval: epoch 76 | batch 100/510 [I 2022-10-10 15:34:17] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:02 [I 2022-10-10 15:34:17] ...trap/engines/engine.py.326: process: 0.31849 | load: 0.00020 [I 2022-10-10 15:34:20] ...trap/engines/engine.py.321: eval: epoch 76 | batch 110/510 [I 2022-10-10 15:34:20] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:13 [I 2022-10-10 15:34:20] ...trap/engines/engine.py.326: process: 0.32462 | load: 0.00019 [I 2022-10-10 15:34:23] ...trap/engines/engine.py.321: eval: epoch 76 | batch 120/510 [I 2022-10-10 15:34:23] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:04 [I 2022-10-10 15:34:23] ...trap/engines/engine.py.326: process: 0.32429 | load: 0.00020 [I 2022-10-10 15:34:27] ...trap/engines/engine.py.321: eval: epoch 76 | batch 130/510 [I 2022-10-10 15:34:27] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:02 [I 2022-10-10 15:34:27] ...trap/engines/engine.py.326: process: 0.32658 | load: 0.00020 [I 2022-10-10 15:34:30] ...trap/engines/engine.py.321: eval: epoch 76 | batch 140/510 [I 2022-10-10 15:34:30] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:58 [I 2022-10-10 15:34:30] ...trap/engines/engine.py.326: process: 0.32129 | load: 0.00020 [I 2022-10-10 15:34:33] ...trap/engines/engine.py.321: eval: epoch 76 | batch 150/510 [I 2022-10-10 15:34:33] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:56 [I 2022-10-10 15:34:33] ...trap/engines/engine.py.326: process: 0.32803 | load: 0.00024 [I 2022-10-10 15:34:36] ...trap/engines/engine.py.321: eval: epoch 76 | batch 160/510 [I 2022-10-10 15:34:36] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:52 [I 2022-10-10 15:34:36] ...trap/engines/engine.py.326: process: 0.31360 | load: 0.00021 [I 2022-10-10 15:34:39] ...trap/engines/engine.py.321: eval: epoch 76 | batch 170/510 [I 2022-10-10 15:34:39] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:50 [I 2022-10-10 15:34:39] ...trap/engines/engine.py.326: process: 0.32563 | load: 0.00020 [I 2022-10-10 15:34:43] ...trap/engines/engine.py.321: eval: epoch 76 | batch 180/510 [I 2022-10-10 15:34:43] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:46 [I 2022-10-10 15:34:43] ...trap/engines/engine.py.326: process: 0.32403 | load: 0.00020 [I 2022-10-10 15:34:46] ...trap/engines/engine.py.321: eval: epoch 76 | batch 190/510 [I 2022-10-10 15:34:46] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:43 [I 2022-10-10 15:34:46] ...trap/engines/engine.py.326: process: 0.32471 | load: 0.00021 [I 2022-10-10 15:34:49] ...trap/engines/engine.py.321: eval: epoch 76 | batch 200/510 [I 2022-10-10 15:34:49] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:40 [I 2022-10-10 15:34:49] ...trap/engines/engine.py.326: process: 0.31850 | load: 0.00021 [I 2022-10-10 15:34:52] ...trap/engines/engine.py.321: eval: epoch 76 | batch 210/510 [I 2022-10-10 15:34:52] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:36 [I 2022-10-10 15:34:52] ...trap/engines/engine.py.326: process: 0.32421 | load: 0.00020 [I 2022-10-10 15:34:56] ...trap/engines/engine.py.321: eval: epoch 76 | batch 220/510 [I 2022-10-10 15:34:56] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:32 [I 2022-10-10 15:34:56] ...trap/engines/engine.py.326: process: 0.31651 | load: 0.00021 [I 2022-10-10 15:34:59] ...trap/engines/engine.py.321: eval: epoch 76 | batch 230/510 [I 2022-10-10 15:34:59] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-10 15:34:59] ...trap/engines/engine.py.326: process: 0.32056 | load: 0.00020 [I 2022-10-10 15:35:02] ...trap/engines/engine.py.321: eval: epoch 76 | batch 240/510 [I 2022-10-10 15:35:02] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-10 15:35:02] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00021 [I 2022-10-10 15:35:05] ...trap/engines/engine.py.321: eval: epoch 76 | batch 250/510 [I 2022-10-10 15:35:05] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-10 15:35:05] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00020 [I 2022-10-10 15:35:08] ...trap/engines/engine.py.321: eval: epoch 76 | batch 260/510 [I 2022-10-10 15:35:08] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:20 [I 2022-10-10 15:35:08] ...trap/engines/engine.py.326: process: 0.32523 | load: 0.00021 [I 2022-10-10 15:35:12] ...trap/engines/engine.py.321: eval: epoch 76 | batch 270/510 [I 2022-10-10 15:35:12] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:17 [I 2022-10-10 15:35:12] ...trap/engines/engine.py.326: process: 0.32391 | load: 0.00026 [I 2022-10-10 15:35:15] ...trap/engines/engine.py.321: eval: epoch 76 | batch 280/510 [I 2022-10-10 15:35:15] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:14 [I 2022-10-10 15:35:15] ...trap/engines/engine.py.326: process: 0.32088 | load: 0.00020 [I 2022-10-10 15:35:18] ...trap/engines/engine.py.321: eval: epoch 76 | batch 290/510 [I 2022-10-10 15:35:18] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:11 [I 2022-10-10 15:35:18] ...trap/engines/engine.py.326: process: 0.32433 | load: 0.00021 [I 2022-10-10 15:35:21] ...trap/engines/engine.py.321: eval: epoch 76 | batch 300/510 [I 2022-10-10 15:35:21] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:08 [I 2022-10-10 15:35:21] ...trap/engines/engine.py.326: process: 0.32594 | load: 0.00020 [I 2022-10-10 15:35:25] ...trap/engines/engine.py.321: eval: epoch 76 | batch 310/510 [I 2022-10-10 15:35:25] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-10 15:35:25] ...trap/engines/engine.py.326: process: 0.32420 | load: 0.00021 [I 2022-10-10 15:35:28] ...trap/engines/engine.py.321: eval: epoch 76 | batch 320/510 [I 2022-10-10 15:35:28] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:01 [I 2022-10-10 15:35:28] ...trap/engines/engine.py.326: process: 0.32259 | load: 0.00021 [I 2022-10-10 15:35:31] ...trap/engines/engine.py.321: eval: epoch 76 | batch 330/510 [I 2022-10-10 15:35:31] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:58 [I 2022-10-10 15:35:31] ...trap/engines/engine.py.326: process: 0.32568 | load: 0.00021 [I 2022-10-10 15:35:34] ...trap/engines/engine.py.321: eval: epoch 76 | batch 340/510 [I 2022-10-10 15:35:34] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-10 15:35:34] ...trap/engines/engine.py.326: process: 0.32315 | load: 0.00020 [I 2022-10-10 15:35:38] ...trap/engines/engine.py.321: eval: epoch 76 | batch 350/510 [I 2022-10-10 15:35:38] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:51 [I 2022-10-10 15:35:38] ...trap/engines/engine.py.326: process: 0.32476 | load: 0.00021 [I 2022-10-10 15:35:41] ...trap/engines/engine.py.321: eval: epoch 76 | batch 360/510 [I 2022-10-10 15:35:41] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-10 15:35:41] ...trap/engines/engine.py.326: process: 0.32239 | load: 0.00020 [I 2022-10-10 15:35:44] ...trap/engines/engine.py.321: eval: epoch 76 | batch 370/510 [I 2022-10-10 15:35:44] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:45 [I 2022-10-10 15:35:44] ...trap/engines/engine.py.326: process: 0.32568 | load: 0.00022 [I 2022-10-10 15:35:47] ...trap/engines/engine.py.321: eval: epoch 76 | batch 380/510 [I 2022-10-10 15:35:47] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-10 15:35:47] ...trap/engines/engine.py.326: process: 0.31667 | load: 0.00019 [I 2022-10-10 15:35:51] ...trap/engines/engine.py.321: eval: epoch 76 | batch 390/510 [I 2022-10-10 15:35:51] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-10 15:35:51] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00021 [I 2022-10-10 15:35:54] ...trap/engines/engine.py.321: eval: epoch 76 | batch 400/510 [I 2022-10-10 15:35:54] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-10 15:35:54] ...trap/engines/engine.py.326: process: 0.32275 | load: 0.00021 [I 2022-10-10 15:35:57] ...trap/engines/engine.py.321: eval: epoch 76 | batch 410/510 [I 2022-10-10 15:35:57] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-10 15:35:57] ...trap/engines/engine.py.326: process: 0.32529 | load: 0.00021 [I 2022-10-10 15:36:00] ...trap/engines/engine.py.321: eval: epoch 76 | batch 420/510 [I 2022-10-10 15:36:00] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:29 [I 2022-10-10 15:36:00] ...trap/engines/engine.py.326: process: 0.32355 | load: 0.00020 [I 2022-10-10 15:36:03] ...trap/engines/engine.py.321: eval: epoch 76 | batch 430/510 [I 2022-10-10 15:36:03] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:26 [I 2022-10-10 15:36:03] ...trap/engines/engine.py.326: process: 0.32738 | load: 0.00020 [I 2022-10-10 15:36:07] ...trap/engines/engine.py.321: eval: epoch 76 | batch 440/510 [I 2022-10-10 15:36:07] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-10 15:36:07] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00021 [I 2022-10-10 15:36:10] ...trap/engines/engine.py.321: eval: epoch 76 | batch 450/510 [I 2022-10-10 15:36:10] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-10 15:36:10] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00021 [I 2022-10-10 15:36:13] ...trap/engines/engine.py.321: eval: epoch 76 | batch 460/510 [I 2022-10-10 15:36:13] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-10 15:36:13] ...trap/engines/engine.py.326: process: 0.32552 | load: 0.00021 [I 2022-10-10 15:36:16] ...trap/engines/engine.py.321: eval: epoch 76 | batch 470/510 [I 2022-10-10 15:36:16] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-10 15:36:16] ...trap/engines/engine.py.326: process: 0.32291 | load: 0.00020 [I 2022-10-10 15:36:20] ...trap/engines/engine.py.321: eval: epoch 76 | batch 480/510 [I 2022-10-10 15:36:20] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-10 15:36:20] ...trap/engines/engine.py.326: process: 0.32466 | load: 0.00020 [I 2022-10-10 15:36:23] ...trap/engines/engine.py.321: eval: epoch 76 | batch 490/510 [I 2022-10-10 15:36:23] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-10 15:36:23] ...trap/engines/engine.py.326: process: 0.32617 | load: 0.00020 [I 2022-10-10 15:36:26] ...trap/engines/engine.py.321: eval: epoch 76 | batch 500/510 [I 2022-10-10 15:36:26] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-10 15:36:26] ...trap/engines/engine.py.326: process: 0.32412 | load: 0.00021 [I 2022-10-10 15:36:29] ...trap/engines/engine.py.321: eval: epoch 76 | batch 510/510 [I 2022-10-10 15:36:29] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-10 15:36:29] ...trap/engines/engine.py.326: process: 0.31430 | load: 0.00017 [S 2022-10-10 15:36:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 76 [I 2022-10-10 15:36:30] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.742000 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.915600 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.943000 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.736300 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.913200 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.942300 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011883 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009058 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008000 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015192 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009867 [S 2022-10-10 15:36:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009122 [I 2022-10-10 15:37:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 15:37:16] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 15:37:17] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 15:37:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 15:37:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 15:37:21] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 15:37:23] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 15:37:27] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 15:37:27] ...trap/engines/engine.py.163: Training model on trainset for epoch 77 [I 2022-10-10 15:37:27] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 15:37:31] ...trap/engines/engine.py.224: train: epoch 77 | batch 0/2304 [I 2022-10-10 15:37:31] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:11:55 [I 2022-10-10 15:37:31] ...trap/engines/engine.py.229: process: 3.43566 | load: 2.51174 [I 2022-10-10 15:37:31] ...trap/engines/engine.py.230: loss: 0.10934 [I 2022-10-10 15:37:31] ...optimizers/trijoint.py.151: total_norm: 10.074446 [I 2022-10-10 15:37:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 10/2304 [I 2022-10-10 15:37:59] ...trap/engines/engine.py.228: elapsed: 0:00:31 | left: 3:10:43 [I 2022-10-10 15:37:59] ...trap/engines/engine.py.229: process: 20.11058 | load: 0.00595 [I 2022-10-10 15:37:59] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-10 15:37:59] ...optimizers/trijoint.py.151: total_norm: 9.242669 [I 2022-10-10 15:38:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 20/2304 [I 2022-10-10 15:38:07] ...trap/engines/engine.py.228: elapsed: 0:00:39 | left: 0:49:34 [I 2022-10-10 15:38:07] ...trap/engines/engine.py.229: process: 0.87740 | load: 0.00590 [I 2022-10-10 15:38:07] ...trap/engines/engine.py.230: loss: 0.10913 [I 2022-10-10 15:38:07] ...optimizers/trijoint.py.151: total_norm: 26.680822 [I 2022-10-10 15:38:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 30/2304 [I 2022-10-10 15:38:16] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:34:25 [I 2022-10-10 15:38:16] ...trap/engines/engine.py.229: process: 0.86832 | load: 0.00476 [I 2022-10-10 15:38:16] ...trap/engines/engine.py.230: loss: 0.23478 [I 2022-10-10 15:38:16] ...optimizers/trijoint.py.151: total_norm: 25.141588 [I 2022-10-10 15:38:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 40/2304 [I 2022-10-10 15:38:24] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:32:43 [I 2022-10-10 15:38:24] ...trap/engines/engine.py.229: process: 0.86741 | load: 0.00460 [I 2022-10-10 15:38:24] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-10 15:38:24] ...optimizers/trijoint.py.151: total_norm: 28.155619 [I 2022-10-10 15:38:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 50/2304 [I 2022-10-10 15:38:33] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:32:32 [I 2022-10-10 15:38:33] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00365 [I 2022-10-10 15:38:33] ...trap/engines/engine.py.230: loss: 0.09939 [I 2022-10-10 15:38:33] ...optimizers/trijoint.py.151: total_norm: 15.924586 [I 2022-10-10 15:38:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 60/2304 [I 2022-10-10 15:38:42] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:32:19 [I 2022-10-10 15:38:42] ...trap/engines/engine.py.229: process: 0.86886 | load: 0.00464 [I 2022-10-10 15:38:42] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-10 15:38:42] ...optimizers/trijoint.py.151: total_norm: 15.972762 [I 2022-10-10 15:38:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 70/2304 [I 2022-10-10 15:38:50] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:32:03 [I 2022-10-10 15:38:50] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00365 [I 2022-10-10 15:38:50] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-10 15:38:50] ...optimizers/trijoint.py.151: total_norm: 18.851923 [I 2022-10-10 15:38:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 80/2304 [I 2022-10-10 15:38:59] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:31:56 [I 2022-10-10 15:38:59] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00377 [I 2022-10-10 15:38:59] ...trap/engines/engine.py.230: loss: 0.14638 [I 2022-10-10 15:38:59] ...optimizers/trijoint.py.151: total_norm: 27.027531 [I 2022-10-10 15:39:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 90/2304 [I 2022-10-10 15:39:08] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:31:44 [I 2022-10-10 15:39:08] ...trap/engines/engine.py.229: process: 0.86166 | load: 0.00376 [I 2022-10-10 15:39:08] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-10 15:39:08] ...optimizers/trijoint.py.151: total_norm: 24.472244 [I 2022-10-10 15:39:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 100/2304 [I 2022-10-10 15:39:16] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:31:50 [I 2022-10-10 15:39:16] ...trap/engines/engine.py.229: process: 0.87139 | load: 0.00393 [I 2022-10-10 15:39:16] ...trap/engines/engine.py.230: loss: 0.10098 [I 2022-10-10 15:39:16] ...optimizers/trijoint.py.151: total_norm: 11.283087 [I 2022-10-10 15:39:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 110/2304 [I 2022-10-10 15:39:25] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:31:35 [I 2022-10-10 15:39:25] ...trap/engines/engine.py.229: process: 0.86665 | load: 0.00346 [I 2022-10-10 15:39:25] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-10 15:39:25] ...optimizers/trijoint.py.151: total_norm: 13.837190 [I 2022-10-10 15:39:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 120/2304 [I 2022-10-10 15:39:34] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:31:25 [I 2022-10-10 15:39:34] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00363 [I 2022-10-10 15:39:34] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-10 15:39:34] ...optimizers/trijoint.py.151: total_norm: 20.506674 [I 2022-10-10 15:39:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 130/2304 [I 2022-10-10 15:39:42] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:31:19 [I 2022-10-10 15:39:42] ...trap/engines/engine.py.229: process: 0.86346 | load: 0.00385 [I 2022-10-10 15:39:42] ...trap/engines/engine.py.230: loss: 0.12297 [I 2022-10-10 15:39:42] ...optimizers/trijoint.py.151: total_norm: 11.021118 [I 2022-10-10 15:39:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 140/2304 [I 2022-10-10 15:39:51] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:31:05 [I 2022-10-10 15:39:51] ...trap/engines/engine.py.229: process: 0.86535 | load: 0.00326 [I 2022-10-10 15:39:51] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-10 15:39:51] ...optimizers/trijoint.py.151: total_norm: 9.102170 [I 2022-10-10 15:40:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 150/2304 [I 2022-10-10 15:40:00] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:31:14 [I 2022-10-10 15:40:00] ...trap/engines/engine.py.229: process: 0.86300 | load: 0.00342 [I 2022-10-10 15:40:00] ...trap/engines/engine.py.230: loss: 0.09702 [I 2022-10-10 15:40:00] ...optimizers/trijoint.py.151: total_norm: 8.552132 [I 2022-10-10 15:40:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 160/2304 [I 2022-10-10 15:40:08] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:30:58 [I 2022-10-10 15:40:08] ...trap/engines/engine.py.229: process: 0.87208 | load: 0.00367 [I 2022-10-10 15:40:08] ...trap/engines/engine.py.230: loss: 0.10620 [I 2022-10-10 15:40:08] ...optimizers/trijoint.py.151: total_norm: 12.473867 [I 2022-10-10 15:40:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 170/2304 [I 2022-10-10 15:40:17] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:30:44 [I 2022-10-10 15:40:17] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00310 [I 2022-10-10 15:40:17] ...trap/engines/engine.py.230: loss: 0.14733 [I 2022-10-10 15:40:17] ...optimizers/trijoint.py.151: total_norm: 18.442533 [I 2022-10-10 15:40:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 180/2304 [I 2022-10-10 15:40:26] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:30:30 [I 2022-10-10 15:40:26] ...trap/engines/engine.py.229: process: 0.85617 | load: 0.00334 [I 2022-10-10 15:40:26] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-10 15:40:26] ...optimizers/trijoint.py.151: total_norm: 18.742989 [I 2022-10-10 15:40:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 190/2304 [I 2022-10-10 15:40:34] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:30:13 [I 2022-10-10 15:40:34] ...trap/engines/engine.py.229: process: 0.86189 | load: 0.00307 [I 2022-10-10 15:40:34] ...trap/engines/engine.py.230: loss: 0.11698 [I 2022-10-10 15:40:34] ...optimizers/trijoint.py.151: total_norm: 15.570800 [I 2022-10-10 15:40:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 200/2304 [I 2022-10-10 15:40:43] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:30:22 [I 2022-10-10 15:40:43] ...trap/engines/engine.py.229: process: 0.86998 | load: 0.00321 [I 2022-10-10 15:40:43] ...trap/engines/engine.py.230: loss: 0.14424 [I 2022-10-10 15:40:43] ...optimizers/trijoint.py.151: total_norm: 19.884640 [I 2022-10-10 15:40:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 210/2304 [I 2022-10-10 15:40:52] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:30:05 [I 2022-10-10 15:40:52] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00310 [I 2022-10-10 15:40:52] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-10 15:40:52] ...optimizers/trijoint.py.151: total_norm: 12.198771 [I 2022-10-10 15:41:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 220/2304 [I 2022-10-10 15:41:00] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:29:48 [I 2022-10-10 15:41:00] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00331 [I 2022-10-10 15:41:00] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-10 15:41:00] ...optimizers/trijoint.py.151: total_norm: 7.570823 [I 2022-10-10 15:41:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 230/2304 [I 2022-10-10 15:41:09] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:29:52 [I 2022-10-10 15:41:09] ...trap/engines/engine.py.229: process: 0.87459 | load: 0.00325 [I 2022-10-10 15:41:09] ...trap/engines/engine.py.230: loss: 0.12918 [I 2022-10-10 15:41:09] ...optimizers/trijoint.py.151: total_norm: 19.156164 [I 2022-10-10 15:41:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 240/2304 [I 2022-10-10 15:41:18] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:29:32 [I 2022-10-10 15:41:18] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00312 [I 2022-10-10 15:41:18] ...trap/engines/engine.py.230: loss: 0.11730 [I 2022-10-10 15:41:18] ...optimizers/trijoint.py.151: total_norm: 17.406815 [I 2022-10-10 15:41:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 250/2304 [I 2022-10-10 15:41:26] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:29:17 [I 2022-10-10 15:41:26] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00328 [I 2022-10-10 15:41:26] ...trap/engines/engine.py.230: loss: 0.15399 [I 2022-10-10 15:41:26] ...optimizers/trijoint.py.151: total_norm: 8.352369 [I 2022-10-10 15:41:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 260/2304 [I 2022-10-10 15:41:35] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:29:25 [I 2022-10-10 15:41:35] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00321 [I 2022-10-10 15:41:35] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-10 15:41:35] ...optimizers/trijoint.py.151: total_norm: 16.496292 [I 2022-10-10 15:41:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 270/2304 [I 2022-10-10 15:41:43] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:29:21 [I 2022-10-10 15:41:43] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00325 [I 2022-10-10 15:41:43] ...trap/engines/engine.py.230: loss: 0.11378 [I 2022-10-10 15:41:43] ...optimizers/trijoint.py.151: total_norm: 27.074018 [I 2022-10-10 15:41:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 280/2304 [I 2022-10-10 15:41:52] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:29:01 [I 2022-10-10 15:41:52] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00397 [I 2022-10-10 15:41:52] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-10 15:41:52] ...optimizers/trijoint.py.151: total_norm: 11.192479 [I 2022-10-10 15:42:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 290/2304 [I 2022-10-10 15:42:01] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:59 [I 2022-10-10 15:42:01] ...trap/engines/engine.py.229: process: 0.87358 | load: 0.00349 [I 2022-10-10 15:42:01] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-10 15:42:01] ...optimizers/trijoint.py.151: total_norm: 12.567000 [I 2022-10-10 15:42:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 300/2304 [I 2022-10-10 15:42:09] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:28:56 [I 2022-10-10 15:42:09] ...trap/engines/engine.py.229: process: 0.87708 | load: 0.00339 [I 2022-10-10 15:42:09] ...trap/engines/engine.py.230: loss: 0.17693 [I 2022-10-10 15:42:09] ...optimizers/trijoint.py.151: total_norm: 18.924028 [I 2022-10-10 15:42:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 310/2304 [I 2022-10-10 15:42:18] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:28:27 [I 2022-10-10 15:42:18] ...trap/engines/engine.py.229: process: 0.86061 | load: 0.00333 [I 2022-10-10 15:42:18] ...trap/engines/engine.py.230: loss: 0.11799 [I 2022-10-10 15:42:18] ...optimizers/trijoint.py.151: total_norm: 23.322302 [I 2022-10-10 15:42:27] ...trap/engines/engine.py.224: train: epoch 77 | batch 320/2304 [I 2022-10-10 15:42:27] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:28:30 [I 2022-10-10 15:42:27] ...trap/engines/engine.py.229: process: 0.87248 | load: 0.00317 [I 2022-10-10 15:42:27] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-10 15:42:27] ...optimizers/trijoint.py.151: total_norm: 21.634352 [I 2022-10-10 15:42:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 330/2304 [I 2022-10-10 15:42:35] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:28:25 [I 2022-10-10 15:42:35] ...trap/engines/engine.py.229: process: 0.86323 | load: 0.00312 [I 2022-10-10 15:42:35] ...trap/engines/engine.py.230: loss: 0.09527 [I 2022-10-10 15:42:35] ...optimizers/trijoint.py.151: total_norm: 17.055586 [I 2022-10-10 15:42:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 340/2304 [I 2022-10-10 15:42:44] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:28:19 [I 2022-10-10 15:42:44] ...trap/engines/engine.py.229: process: 0.86899 | load: 0.00326 [I 2022-10-10 15:42:44] ...trap/engines/engine.py.230: loss: 0.13348 [I 2022-10-10 15:42:44] ...optimizers/trijoint.py.151: total_norm: 10.045413 [I 2022-10-10 15:42:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 350/2304 [I 2022-10-10 15:42:53] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:28:11 [I 2022-10-10 15:42:53] ...trap/engines/engine.py.229: process: 0.87259 | load: 0.00312 [I 2022-10-10 15:42:53] ...trap/engines/engine.py.230: loss: 0.15953 [I 2022-10-10 15:42:53] ...optimizers/trijoint.py.151: total_norm: 19.575142 [I 2022-10-10 15:43:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 360/2304 [I 2022-10-10 15:43:01] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:54 [I 2022-10-10 15:43:01] ...trap/engines/engine.py.229: process: 0.85203 | load: 0.00317 [I 2022-10-10 15:43:01] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-10 15:43:01] ...optimizers/trijoint.py.151: total_norm: 17.283718 [I 2022-10-10 15:43:10] ...trap/engines/engine.py.224: train: epoch 77 | batch 370/2304 [I 2022-10-10 15:43:10] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:27:50 [I 2022-10-10 15:43:10] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00306 [I 2022-10-10 15:43:10] ...trap/engines/engine.py.230: loss: 0.13729 [I 2022-10-10 15:43:10] ...optimizers/trijoint.py.151: total_norm: 24.436417 [I 2022-10-10 15:43:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 380/2304 [I 2022-10-10 15:43:19] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:27:42 [I 2022-10-10 15:43:19] ...trap/engines/engine.py.229: process: 0.86504 | load: 0.00311 [I 2022-10-10 15:43:19] ...trap/engines/engine.py.230: loss: 0.18216 [I 2022-10-10 15:43:19] ...optimizers/trijoint.py.151: total_norm: 18.423061 [I 2022-10-10 15:43:27] ...trap/engines/engine.py.224: train: epoch 77 | batch 390/2304 [I 2022-10-10 15:43:27] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:27:32 [I 2022-10-10 15:43:27] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00308 [I 2022-10-10 15:43:27] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-10 15:43:27] ...optimizers/trijoint.py.151: total_norm: 14.158023 [I 2022-10-10 15:43:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 400/2304 [I 2022-10-10 15:43:36] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:27:18 [I 2022-10-10 15:43:36] ...trap/engines/engine.py.229: process: 0.86330 | load: 0.00313 [I 2022-10-10 15:43:36] ...trap/engines/engine.py.230: loss: 0.12657 [I 2022-10-10 15:43:36] ...optimizers/trijoint.py.151: total_norm: 13.888672 [I 2022-10-10 15:43:45] ...trap/engines/engine.py.224: train: epoch 77 | batch 410/2304 [I 2022-10-10 15:43:45] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:27:09 [I 2022-10-10 15:43:45] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00307 [I 2022-10-10 15:43:45] ...trap/engines/engine.py.230: loss: 0.21391 [I 2022-10-10 15:43:45] ...optimizers/trijoint.py.151: total_norm: 25.628677 [I 2022-10-10 15:43:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 420/2304 [I 2022-10-10 15:43:53] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:27:09 [I 2022-10-10 15:43:53] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00313 [I 2022-10-10 15:43:53] ...trap/engines/engine.py.230: loss: 0.10099 [I 2022-10-10 15:43:53] ...optimizers/trijoint.py.151: total_norm: 11.728991 [I 2022-10-10 15:44:02] ...trap/engines/engine.py.224: train: epoch 77 | batch 430/2304 [I 2022-10-10 15:44:02] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:26:52 [I 2022-10-10 15:44:02] ...trap/engines/engine.py.229: process: 0.86094 | load: 0.00309 [I 2022-10-10 15:44:02] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-10 15:44:02] ...optimizers/trijoint.py.151: total_norm: 11.673417 [I 2022-10-10 15:44:10] ...trap/engines/engine.py.224: train: epoch 77 | batch 440/2304 [I 2022-10-10 15:44:10] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:26:55 [I 2022-10-10 15:44:10] ...trap/engines/engine.py.229: process: 0.85817 | load: 0.00307 [I 2022-10-10 15:44:10] ...trap/engines/engine.py.230: loss: 0.10155 [I 2022-10-10 15:44:10] ...optimizers/trijoint.py.151: total_norm: 15.490299 [I 2022-10-10 15:44:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 450/2304 [I 2022-10-10 15:44:19] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:26:28 [I 2022-10-10 15:44:19] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00317 [I 2022-10-10 15:44:19] ...trap/engines/engine.py.230: loss: 0.13162 [I 2022-10-10 15:44:19] ...optimizers/trijoint.py.151: total_norm: 8.073353 [I 2022-10-10 15:44:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 460/2304 [I 2022-10-10 15:44:28] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:26:32 [I 2022-10-10 15:44:28] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00318 [I 2022-10-10 15:44:28] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-10 15:44:28] ...optimizers/trijoint.py.151: total_norm: 17.015438 [I 2022-10-10 15:44:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 470/2304 [I 2022-10-10 15:44:36] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:26:20 [I 2022-10-10 15:44:36] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00312 [I 2022-10-10 15:44:36] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-10 15:44:36] ...optimizers/trijoint.py.151: total_norm: 9.706407 [I 2022-10-10 15:44:45] ...trap/engines/engine.py.224: train: epoch 77 | batch 480/2304 [I 2022-10-10 15:44:45] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:26:12 [I 2022-10-10 15:44:45] ...trap/engines/engine.py.229: process: 0.86983 | load: 0.00309 [I 2022-10-10 15:44:45] ...trap/engines/engine.py.230: loss: 0.15864 [I 2022-10-10 15:44:45] ...optimizers/trijoint.py.151: total_norm: 17.446020 [I 2022-10-10 15:44:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 490/2304 [I 2022-10-10 15:44:54] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:26:05 [I 2022-10-10 15:44:54] ...trap/engines/engine.py.229: process: 0.86840 | load: 0.00319 [I 2022-10-10 15:44:54] ...trap/engines/engine.py.230: loss: 0.08521 [I 2022-10-10 15:44:54] ...optimizers/trijoint.py.151: total_norm: 9.183621 [I 2022-10-10 15:45:02] ...trap/engines/engine.py.224: train: epoch 77 | batch 500/2304 [I 2022-10-10 15:45:02] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:48 [I 2022-10-10 15:45:02] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00309 [I 2022-10-10 15:45:02] ...trap/engines/engine.py.230: loss: 0.09568 [I 2022-10-10 15:45:02] ...optimizers/trijoint.py.151: total_norm: 10.614695 [I 2022-10-10 15:45:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 510/2304 [I 2022-10-10 15:45:11] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:32 [I 2022-10-10 15:45:11] ...trap/engines/engine.py.229: process: 0.84118 | load: 0.00316 [I 2022-10-10 15:45:11] ...trap/engines/engine.py.230: loss: 0.18297 [I 2022-10-10 15:45:11] ...optimizers/trijoint.py.151: total_norm: 13.534815 [I 2022-10-10 15:45:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 520/2304 [I 2022-10-10 15:45:19] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:38 [I 2022-10-10 15:45:19] ...trap/engines/engine.py.229: process: 0.88152 | load: 0.00320 [I 2022-10-10 15:45:19] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-10 15:45:19] ...optimizers/trijoint.py.151: total_norm: 14.779583 [I 2022-10-10 15:45:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 530/2304 [I 2022-10-10 15:45:28] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:25:24 [I 2022-10-10 15:45:28] ...trap/engines/engine.py.229: process: 0.86760 | load: 0.00310 [I 2022-10-10 15:45:28] ...trap/engines/engine.py.230: loss: 0.09847 [I 2022-10-10 15:45:28] ...optimizers/trijoint.py.151: total_norm: 14.625134 [I 2022-10-10 15:45:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 540/2304 [I 2022-10-10 15:45:37] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:25:24 [I 2022-10-10 15:45:37] ...trap/engines/engine.py.229: process: 0.86903 | load: 0.00308 [I 2022-10-10 15:45:37] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-10 15:45:37] ...optimizers/trijoint.py.151: total_norm: 41.165672 [I 2022-10-10 15:45:45] ...trap/engines/engine.py.224: train: epoch 77 | batch 550/2304 [I 2022-10-10 15:45:45] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:25:08 [I 2022-10-10 15:45:45] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00321 [I 2022-10-10 15:45:45] ...trap/engines/engine.py.230: loss: 0.15171 [I 2022-10-10 15:45:45] ...optimizers/trijoint.py.151: total_norm: 20.904194 [I 2022-10-10 15:45:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 560/2304 [I 2022-10-10 15:45:54] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:25:06 [I 2022-10-10 15:45:54] ...trap/engines/engine.py.229: process: 0.86203 | load: 0.00319 [I 2022-10-10 15:45:54] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-10 15:45:54] ...optimizers/trijoint.py.151: total_norm: 13.008863 [I 2022-10-10 15:46:03] ...trap/engines/engine.py.224: train: epoch 77 | batch 570/2304 [I 2022-10-10 15:46:03] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:25:00 [I 2022-10-10 15:46:03] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00308 [I 2022-10-10 15:46:03] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-10 15:46:03] ...optimizers/trijoint.py.151: total_norm: 10.029137 [I 2022-10-10 15:46:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 580/2304 [I 2022-10-10 15:46:11] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:24:52 [I 2022-10-10 15:46:11] ...trap/engines/engine.py.229: process: 0.86758 | load: 0.00318 [I 2022-10-10 15:46:11] ...trap/engines/engine.py.230: loss: 0.09228 [I 2022-10-10 15:46:11] ...optimizers/trijoint.py.151: total_norm: 9.593515 [I 2022-10-10 15:46:20] ...trap/engines/engine.py.224: train: epoch 77 | batch 590/2304 [I 2022-10-10 15:46:20] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:38 [I 2022-10-10 15:46:20] ...trap/engines/engine.py.229: process: 0.85446 | load: 0.00316 [I 2022-10-10 15:46:20] ...trap/engines/engine.py.230: loss: 0.15242 [I 2022-10-10 15:46:20] ...optimizers/trijoint.py.151: total_norm: 18.986010 [I 2022-10-10 15:46:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 600/2304 [I 2022-10-10 15:46:29] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:24:36 [I 2022-10-10 15:46:29] ...trap/engines/engine.py.229: process: 0.86612 | load: 0.00315 [I 2022-10-10 15:46:29] ...trap/engines/engine.py.230: loss: 0.11447 [I 2022-10-10 15:46:29] ...optimizers/trijoint.py.151: total_norm: 23.944162 [I 2022-10-10 15:46:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 610/2304 [I 2022-10-10 15:46:37] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:24:18 [I 2022-10-10 15:46:37] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00312 [I 2022-10-10 15:46:37] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-10 15:46:37] ...optimizers/trijoint.py.151: total_norm: 57.682003 [I 2022-10-10 15:46:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 620/2304 [I 2022-10-10 15:46:46] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:24:08 [I 2022-10-10 15:46:46] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00310 [I 2022-10-10 15:46:46] ...trap/engines/engine.py.230: loss: 0.13260 [I 2022-10-10 15:46:46] ...optimizers/trijoint.py.151: total_norm: 19.609344 [I 2022-10-10 15:46:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 630/2304 [I 2022-10-10 15:46:55] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:24:08 [I 2022-10-10 15:46:55] ...trap/engines/engine.py.229: process: 0.88189 | load: 0.00316 [I 2022-10-10 15:46:55] ...trap/engines/engine.py.230: loss: 0.11829 [I 2022-10-10 15:46:55] ...optimizers/trijoint.py.151: total_norm: 63.475407 [I 2022-10-10 15:47:03] ...trap/engines/engine.py.224: train: epoch 77 | batch 640/2304 [I 2022-10-10 15:47:03] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:24:01 [I 2022-10-10 15:47:03] ...trap/engines/engine.py.229: process: 0.87181 | load: 0.00308 [I 2022-10-10 15:47:03] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-10 15:47:03] ...optimizers/trijoint.py.151: total_norm: 10.425817 [I 2022-10-10 15:47:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 650/2304 [I 2022-10-10 15:47:12] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:46 [I 2022-10-10 15:47:12] ...trap/engines/engine.py.229: process: 0.86248 | load: 0.00309 [I 2022-10-10 15:47:12] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-10 15:47:12] ...optimizers/trijoint.py.151: total_norm: 22.200996 [I 2022-10-10 15:47:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 660/2304 [I 2022-10-10 15:47:21] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:23:34 [I 2022-10-10 15:47:21] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00309 [I 2022-10-10 15:47:21] ...trap/engines/engine.py.230: loss: 0.19242 [I 2022-10-10 15:47:21] ...optimizers/trijoint.py.151: total_norm: 38.433571 [I 2022-10-10 15:47:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 670/2304 [I 2022-10-10 15:47:29] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:23:32 [I 2022-10-10 15:47:29] ...trap/engines/engine.py.229: process: 0.88255 | load: 0.00311 [I 2022-10-10 15:47:29] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-10 15:47:29] ...optimizers/trijoint.py.151: total_norm: 19.145559 [I 2022-10-10 15:47:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 680/2304 [I 2022-10-10 15:47:38] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:23:31 [I 2022-10-10 15:47:38] ...trap/engines/engine.py.229: process: 0.87794 | load: 0.00309 [I 2022-10-10 15:47:38] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-10 15:47:38] ...optimizers/trijoint.py.151: total_norm: 37.859501 [I 2022-10-10 15:47:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 690/2304 [I 2022-10-10 15:47:47] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:23:25 [I 2022-10-10 15:47:47] ...trap/engines/engine.py.229: process: 0.87426 | load: 0.00316 [I 2022-10-10 15:47:47] ...trap/engines/engine.py.230: loss: 0.13513 [I 2022-10-10 15:47:47] ...optimizers/trijoint.py.151: total_norm: 19.289537 [I 2022-10-10 15:47:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 700/2304 [I 2022-10-10 15:47:55] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:23:12 [I 2022-10-10 15:47:55] ...trap/engines/engine.py.229: process: 0.85735 | load: 0.00309 [I 2022-10-10 15:47:55] ...trap/engines/engine.py.230: loss: 0.17472 [I 2022-10-10 15:47:55] ...optimizers/trijoint.py.151: total_norm: 27.801586 [I 2022-10-10 15:48:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 710/2304 [I 2022-10-10 15:48:04] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:23:06 [I 2022-10-10 15:48:04] ...trap/engines/engine.py.229: process: 0.87279 | load: 0.00315 [I 2022-10-10 15:48:04] ...trap/engines/engine.py.230: loss: 0.09005 [I 2022-10-10 15:48:04] ...optimizers/trijoint.py.151: total_norm: 17.100086 [I 2022-10-10 15:48:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 720/2304 [I 2022-10-10 15:48:13] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:22:48 [I 2022-10-10 15:48:13] ...trap/engines/engine.py.229: process: 0.87652 | load: 0.00313 [I 2022-10-10 15:48:13] ...trap/engines/engine.py.230: loss: 0.11327 [I 2022-10-10 15:48:13] ...optimizers/trijoint.py.151: total_norm: 13.363852 [I 2022-10-10 15:48:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 730/2304 [I 2022-10-10 15:48:21] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:22:23 [I 2022-10-10 15:48:21] ...trap/engines/engine.py.229: process: 0.86521 | load: 0.00293 [I 2022-10-10 15:48:21] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-10 15:48:21] ...optimizers/trijoint.py.151: total_norm: 12.836246 [I 2022-10-10 15:48:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 740/2304 [I 2022-10-10 15:48:30] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:22:14 [I 2022-10-10 15:48:30] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00313 [I 2022-10-10 15:48:30] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-10 15:48:30] ...optimizers/trijoint.py.151: total_norm: 21.570038 [I 2022-10-10 15:48:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 750/2304 [I 2022-10-10 15:48:38] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:22:09 [I 2022-10-10 15:48:38] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00295 [I 2022-10-10 15:48:38] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-10 15:48:38] ...optimizers/trijoint.py.151: total_norm: 18.986517 [I 2022-10-10 15:48:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 760/2304 [I 2022-10-10 15:48:47] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:21:50 [I 2022-10-10 15:48:47] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00295 [I 2022-10-10 15:48:47] ...trap/engines/engine.py.230: loss: 0.08294 [I 2022-10-10 15:48:47] ...optimizers/trijoint.py.151: total_norm: 10.995146 [I 2022-10-10 15:48:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 770/2304 [I 2022-10-10 15:48:56] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:21:54 [I 2022-10-10 15:48:56] ...trap/engines/engine.py.229: process: 0.86391 | load: 0.00298 [I 2022-10-10 15:48:56] ...trap/engines/engine.py.230: loss: 0.18689 [I 2022-10-10 15:48:56] ...optimizers/trijoint.py.151: total_norm: 12.676001 [I 2022-10-10 15:49:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 780/2304 [I 2022-10-10 15:49:04] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:21:47 [I 2022-10-10 15:49:04] ...trap/engines/engine.py.229: process: 0.87568 | load: 0.00297 [I 2022-10-10 15:49:04] ...trap/engines/engine.py.230: loss: 0.20685 [I 2022-10-10 15:49:04] ...optimizers/trijoint.py.151: total_norm: 25.353598 [I 2022-10-10 15:49:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 790/2304 [I 2022-10-10 15:49:13] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:21:36 [I 2022-10-10 15:49:13] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00301 [I 2022-10-10 15:49:13] ...trap/engines/engine.py.230: loss: 0.10233 [I 2022-10-10 15:49:13] ...optimizers/trijoint.py.151: total_norm: 10.629139 [I 2022-10-10 15:49:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 800/2304 [I 2022-10-10 15:49:21] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:21:31 [I 2022-10-10 15:49:21] ...trap/engines/engine.py.229: process: 0.86120 | load: 0.00299 [I 2022-10-10 15:49:21] ...trap/engines/engine.py.230: loss: 0.14339 [I 2022-10-10 15:49:21] ...optimizers/trijoint.py.151: total_norm: 10.202980 [I 2022-10-10 15:49:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 810/2304 [I 2022-10-10 15:49:30] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:21:19 [I 2022-10-10 15:49:30] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00297 [I 2022-10-10 15:49:30] ...trap/engines/engine.py.230: loss: 0.16119 [I 2022-10-10 15:49:30] ...optimizers/trijoint.py.151: total_norm: 12.469908 [I 2022-10-10 15:49:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 820/2304 [I 2022-10-10 15:49:39] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:21:21 [I 2022-10-10 15:49:39] ...trap/engines/engine.py.229: process: 0.89004 | load: 0.00311 [I 2022-10-10 15:49:39] ...trap/engines/engine.py.230: loss: 0.12486 [I 2022-10-10 15:49:39] ...optimizers/trijoint.py.151: total_norm: 24.831738 [I 2022-10-10 15:49:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 830/2304 [I 2022-10-10 15:49:47] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:56 [I 2022-10-10 15:49:47] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00303 [I 2022-10-10 15:49:47] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-10 15:49:47] ...optimizers/trijoint.py.151: total_norm: 10.866719 [I 2022-10-10 15:49:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 840/2304 [I 2022-10-10 15:49:56] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:20:45 [I 2022-10-10 15:49:56] ...trap/engines/engine.py.229: process: 0.85803 | load: 0.00300 [I 2022-10-10 15:49:56] ...trap/engines/engine.py.230: loss: 0.17592 [I 2022-10-10 15:49:56] ...optimizers/trijoint.py.151: total_norm: 13.985003 [I 2022-10-10 15:50:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 850/2304 [I 2022-10-10 15:50:04] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:20:36 [I 2022-10-10 15:50:04] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00297 [I 2022-10-10 15:50:04] ...trap/engines/engine.py.230: loss: 0.09097 [I 2022-10-10 15:50:04] ...optimizers/trijoint.py.151: total_norm: 7.220573 [I 2022-10-10 15:50:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 860/2304 [I 2022-10-10 15:50:13] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:20:29 [I 2022-10-10 15:50:13] ...trap/engines/engine.py.229: process: 0.84098 | load: 0.00299 [I 2022-10-10 15:50:13] ...trap/engines/engine.py.230: loss: 0.10865 [I 2022-10-10 15:50:13] ...optimizers/trijoint.py.151: total_norm: 13.031441 [I 2022-10-10 15:50:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 870/2304 [I 2022-10-10 15:50:21] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:20:15 [I 2022-10-10 15:50:21] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00292 [I 2022-10-10 15:50:21] ...trap/engines/engine.py.230: loss: 0.16273 [I 2022-10-10 15:50:21] ...optimizers/trijoint.py.151: total_norm: 24.430138 [I 2022-10-10 15:50:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 880/2304 [I 2022-10-10 15:50:30] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:20:12 [I 2022-10-10 15:50:30] ...trap/engines/engine.py.229: process: 0.84942 | load: 0.00296 [I 2022-10-10 15:50:30] ...trap/engines/engine.py.230: loss: 0.09631 [I 2022-10-10 15:50:30] ...optimizers/trijoint.py.151: total_norm: 10.675767 [I 2022-10-10 15:50:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 890/2304 [I 2022-10-10 15:50:38] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:59 [I 2022-10-10 15:50:38] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00301 [I 2022-10-10 15:50:38] ...trap/engines/engine.py.230: loss: 0.13301 [I 2022-10-10 15:50:38] ...optimizers/trijoint.py.151: total_norm: 17.387657 [I 2022-10-10 15:50:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 900/2304 [I 2022-10-10 15:50:47] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:19:48 [I 2022-10-10 15:50:47] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00296 [I 2022-10-10 15:50:47] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-10 15:50:47] ...optimizers/trijoint.py.151: total_norm: 16.015640 [I 2022-10-10 15:50:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 910/2304 [I 2022-10-10 15:50:55] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:45 [I 2022-10-10 15:50:55] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00294 [I 2022-10-10 15:50:55] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-10 15:50:55] ...optimizers/trijoint.py.151: total_norm: 16.394354 [I 2022-10-10 15:51:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 920/2304 [I 2022-10-10 15:51:04] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:19:42 [I 2022-10-10 15:51:04] ...trap/engines/engine.py.229: process: 0.86200 | load: 0.00300 [I 2022-10-10 15:51:04] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-10 15:51:04] ...optimizers/trijoint.py.151: total_norm: 37.780918 [I 2022-10-10 15:51:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 930/2304 [I 2022-10-10 15:51:13] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:19:30 [I 2022-10-10 15:51:13] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00294 [I 2022-10-10 15:51:13] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-10 15:51:13] ...optimizers/trijoint.py.151: total_norm: 15.997307 [I 2022-10-10 15:51:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 940/2304 [I 2022-10-10 15:51:21] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:20 [I 2022-10-10 15:51:21] ...trap/engines/engine.py.229: process: 0.85925 | load: 0.00296 [I 2022-10-10 15:51:21] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-10 15:51:21] ...optimizers/trijoint.py.151: total_norm: 8.567352 [I 2022-10-10 15:51:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 950/2304 [I 2022-10-10 15:51:30] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:19:16 [I 2022-10-10 15:51:30] ...trap/engines/engine.py.229: process: 0.83647 | load: 0.00291 [I 2022-10-10 15:51:30] ...trap/engines/engine.py.230: loss: 0.18050 [I 2022-10-10 15:51:30] ...optimizers/trijoint.py.151: total_norm: 49.594406 [I 2022-10-10 15:51:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 960/2304 [I 2022-10-10 15:51:38] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:01 [I 2022-10-10 15:51:38] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00291 [I 2022-10-10 15:51:38] ...trap/engines/engine.py.230: loss: 0.21759 [I 2022-10-10 15:51:38] ...optimizers/trijoint.py.151: total_norm: 11.814271 [I 2022-10-10 15:51:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 970/2304 [I 2022-10-10 15:51:47] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:51 [I 2022-10-10 15:51:47] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00292 [I 2022-10-10 15:51:47] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-10 15:51:47] ...optimizers/trijoint.py.151: total_norm: 15.891785 [I 2022-10-10 15:51:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 980/2304 [I 2022-10-10 15:51:55] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:38 [I 2022-10-10 15:51:55] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00297 [I 2022-10-10 15:51:55] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-10 15:51:55] ...optimizers/trijoint.py.151: total_norm: 13.149618 [I 2022-10-10 15:52:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 990/2304 [I 2022-10-10 15:52:04] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:43 [I 2022-10-10 15:52:04] ...trap/engines/engine.py.229: process: 0.84604 | load: 0.00293 [I 2022-10-10 15:52:04] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-10 15:52:04] ...optimizers/trijoint.py.151: total_norm: 9.456345 [I 2022-10-10 15:52:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 1000/2304 [I 2022-10-10 15:52:12] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:18:28 [I 2022-10-10 15:52:12] ...trap/engines/engine.py.229: process: 0.83474 | load: 0.00300 [I 2022-10-10 15:52:12] ...trap/engines/engine.py.230: loss: 0.20134 [I 2022-10-10 15:52:12] ...optimizers/trijoint.py.151: total_norm: 12.982249 [I 2022-10-10 15:52:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 1010/2304 [I 2022-10-10 15:52:21] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:26 [I 2022-10-10 15:52:21] ...trap/engines/engine.py.229: process: 0.85695 | load: 0.00292 [I 2022-10-10 15:52:21] ...trap/engines/engine.py.230: loss: 0.10309 [I 2022-10-10 15:52:21] ...optimizers/trijoint.py.151: total_norm: 15.484035 [I 2022-10-10 15:52:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 1020/2304 [I 2022-10-10 15:52:30] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:18:11 [I 2022-10-10 15:52:30] ...trap/engines/engine.py.229: process: 0.85092 | load: 0.00303 [I 2022-10-10 15:52:30] ...trap/engines/engine.py.230: loss: 0.07809 [I 2022-10-10 15:52:30] ...optimizers/trijoint.py.151: total_norm: 9.390754 [I 2022-10-10 15:52:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 1030/2304 [I 2022-10-10 15:52:38] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:08 [I 2022-10-10 15:52:38] ...trap/engines/engine.py.229: process: 0.86337 | load: 0.00300 [I 2022-10-10 15:52:38] ...trap/engines/engine.py.230: loss: 0.10235 [I 2022-10-10 15:52:38] ...optimizers/trijoint.py.151: total_norm: 16.474451 [I 2022-10-10 15:52:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 1040/2304 [I 2022-10-10 15:52:47] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:52 [I 2022-10-10 15:52:47] ...trap/engines/engine.py.229: process: 0.84626 | load: 0.00297 [I 2022-10-10 15:52:47] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-10 15:52:47] ...optimizers/trijoint.py.151: total_norm: 16.626007 [I 2022-10-10 15:52:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 1050/2304 [I 2022-10-10 15:52:55] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:55 [I 2022-10-10 15:52:55] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00312 [I 2022-10-10 15:52:55] ...trap/engines/engine.py.230: loss: 0.20992 [I 2022-10-10 15:52:55] ...optimizers/trijoint.py.151: total_norm: 17.084291 [I 2022-10-10 15:53:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 1060/2304 [I 2022-10-10 15:53:04] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:47 [I 2022-10-10 15:53:04] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00306 [I 2022-10-10 15:53:04] ...trap/engines/engine.py.230: loss: 0.09596 [I 2022-10-10 15:53:04] ...optimizers/trijoint.py.151: total_norm: 9.751637 [I 2022-10-10 15:53:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 1070/2304 [I 2022-10-10 15:53:12] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:43 [I 2022-10-10 15:53:12] ...trap/engines/engine.py.229: process: 0.86507 | load: 0.00325 [I 2022-10-10 15:53:12] ...trap/engines/engine.py.230: loss: 0.09596 [I 2022-10-10 15:53:12] ...optimizers/trijoint.py.151: total_norm: 8.858171 [I 2022-10-10 15:53:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 1080/2304 [I 2022-10-10 15:53:21] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:33 [I 2022-10-10 15:53:21] ...trap/engines/engine.py.229: process: 0.86005 | load: 0.00308 [I 2022-10-10 15:53:21] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-10 15:53:21] ...optimizers/trijoint.py.151: total_norm: 15.787355 [I 2022-10-10 15:53:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 1090/2304 [I 2022-10-10 15:53:30] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:29 [I 2022-10-10 15:53:30] ...trap/engines/engine.py.229: process: 0.87122 | load: 0.00309 [I 2022-10-10 15:53:30] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-10 15:53:30] ...optimizers/trijoint.py.151: total_norm: 7.473252 [I 2022-10-10 15:53:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 1100/2304 [I 2022-10-10 15:53:38] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:23 [I 2022-10-10 15:53:38] ...trap/engines/engine.py.229: process: 0.86603 | load: 0.00312 [I 2022-10-10 15:53:38] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-10 15:53:38] ...optimizers/trijoint.py.151: total_norm: 11.798041 [I 2022-10-10 15:53:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 1110/2304 [I 2022-10-10 15:53:47] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:17:06 [I 2022-10-10 15:53:47] ...trap/engines/engine.py.229: process: 0.86144 | load: 0.00308 [I 2022-10-10 15:53:47] ...trap/engines/engine.py.230: loss: 0.13779 [I 2022-10-10 15:53:47] ...optimizers/trijoint.py.151: total_norm: 10.063440 [I 2022-10-10 15:53:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 1120/2304 [I 2022-10-10 15:53:56] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:17:05 [I 2022-10-10 15:53:56] ...trap/engines/engine.py.229: process: 0.86231 | load: 0.00306 [I 2022-10-10 15:53:56] ...trap/engines/engine.py.230: loss: 0.21398 [I 2022-10-10 15:53:56] ...optimizers/trijoint.py.151: total_norm: 13.829051 [I 2022-10-10 15:54:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 1130/2304 [I 2022-10-10 15:54:04] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:51 [I 2022-10-10 15:54:04] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00308 [I 2022-10-10 15:54:04] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-10 15:54:04] ...optimizers/trijoint.py.151: total_norm: 22.568459 [I 2022-10-10 15:54:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 1140/2304 [I 2022-10-10 15:54:13] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:16:39 [I 2022-10-10 15:54:13] ...trap/engines/engine.py.229: process: 0.86422 | load: 0.00312 [I 2022-10-10 15:54:13] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-10 15:54:13] ...optimizers/trijoint.py.151: total_norm: 13.720270 [I 2022-10-10 15:54:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 1150/2304 [I 2022-10-10 15:54:22] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:32 [I 2022-10-10 15:54:22] ...trap/engines/engine.py.229: process: 0.85855 | load: 0.00309 [I 2022-10-10 15:54:22] ...trap/engines/engine.py.230: loss: 0.12000 [I 2022-10-10 15:54:22] ...optimizers/trijoint.py.151: total_norm: 27.112541 [I 2022-10-10 15:54:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 1160/2304 [I 2022-10-10 15:54:30] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:16:26 [I 2022-10-10 15:54:30] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00320 [I 2022-10-10 15:54:30] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-10 15:54:30] ...optimizers/trijoint.py.151: total_norm: 23.954788 [I 2022-10-10 15:54:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 1170/2304 [I 2022-10-10 15:54:39] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:16:15 [I 2022-10-10 15:54:39] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00310 [I 2022-10-10 15:54:39] ...trap/engines/engine.py.230: loss: 0.11587 [I 2022-10-10 15:54:39] ...optimizers/trijoint.py.151: total_norm: 19.848700 [I 2022-10-10 15:54:48] ...trap/engines/engine.py.224: train: epoch 77 | batch 1180/2304 [I 2022-10-10 15:54:48] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:16:12 [I 2022-10-10 15:54:48] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00314 [I 2022-10-10 15:54:48] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-10 15:54:48] ...optimizers/trijoint.py.151: total_norm: 14.739349 [I 2022-10-10 15:54:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 1190/2304 [I 2022-10-10 15:54:56] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:16:00 [I 2022-10-10 15:54:56] ...trap/engines/engine.py.229: process: 0.85840 | load: 0.00310 [I 2022-10-10 15:54:56] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-10 15:54:56] ...optimizers/trijoint.py.151: total_norm: 8.397594 [I 2022-10-10 15:55:05] ...trap/engines/engine.py.224: train: epoch 77 | batch 1200/2304 [I 2022-10-10 15:55:05] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:15:53 [I 2022-10-10 15:55:05] ...trap/engines/engine.py.229: process: 0.86013 | load: 0.00317 [I 2022-10-10 15:55:05] ...trap/engines/engine.py.230: loss: 0.09750 [I 2022-10-10 15:55:05] ...optimizers/trijoint.py.151: total_norm: 8.257546 [I 2022-10-10 15:55:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1210/2304 [I 2022-10-10 15:55:14] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:45 [I 2022-10-10 15:55:14] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00319 [I 2022-10-10 15:55:14] ...trap/engines/engine.py.230: loss: 0.09065 [I 2022-10-10 15:55:14] ...optimizers/trijoint.py.151: total_norm: 7.782199 [I 2022-10-10 15:55:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 1220/2304 [I 2022-10-10 15:55:22] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:15:34 [I 2022-10-10 15:55:22] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00311 [I 2022-10-10 15:55:22] ...trap/engines/engine.py.230: loss: 0.10081 [I 2022-10-10 15:55:22] ...optimizers/trijoint.py.151: total_norm: 24.517412 [I 2022-10-10 15:55:31] ...trap/engines/engine.py.224: train: epoch 77 | batch 1230/2304 [I 2022-10-10 15:55:31] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:27 [I 2022-10-10 15:55:31] ...trap/engines/engine.py.229: process: 0.88094 | load: 0.00323 [I 2022-10-10 15:55:31] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-10 15:55:31] ...optimizers/trijoint.py.151: total_norm: 24.240923 [I 2022-10-10 15:55:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1240/2304 [I 2022-10-10 15:55:40] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:23 [I 2022-10-10 15:55:40] ...trap/engines/engine.py.229: process: 0.88510 | load: 0.00310 [I 2022-10-10 15:55:40] ...trap/engines/engine.py.230: loss: 0.09353 [I 2022-10-10 15:55:40] ...optimizers/trijoint.py.151: total_norm: 9.018141 [I 2022-10-10 15:55:48] ...trap/engines/engine.py.224: train: epoch 77 | batch 1250/2304 [I 2022-10-10 15:55:48] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:15:07 [I 2022-10-10 15:55:48] ...trap/engines/engine.py.229: process: 0.86318 | load: 0.00314 [I 2022-10-10 15:55:48] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-10 15:55:48] ...optimizers/trijoint.py.151: total_norm: 10.627523 [I 2022-10-10 15:55:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1260/2304 [I 2022-10-10 15:55:57] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:15:03 [I 2022-10-10 15:55:57] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00316 [I 2022-10-10 15:55:57] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-10 15:55:57] ...optimizers/trijoint.py.151: total_norm: 31.809029 [I 2022-10-10 15:56:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1270/2304 [I 2022-10-10 15:56:06] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:56 [I 2022-10-10 15:56:06] ...trap/engines/engine.py.229: process: 0.86875 | load: 0.00307 [I 2022-10-10 15:56:06] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-10 15:56:06] ...optimizers/trijoint.py.151: total_norm: 22.909473 [I 2022-10-10 15:56:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1280/2304 [I 2022-10-10 15:56:14] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:45 [I 2022-10-10 15:56:14] ...trap/engines/engine.py.229: process: 0.86579 | load: 0.00312 [I 2022-10-10 15:56:14] ...trap/engines/engine.py.230: loss: 0.10792 [I 2022-10-10 15:56:14] ...optimizers/trijoint.py.151: total_norm: 12.887002 [I 2022-10-10 15:56:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1290/2304 [I 2022-10-10 15:56:23] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:30 [I 2022-10-10 15:56:23] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00318 [I 2022-10-10 15:56:23] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-10 15:56:23] ...optimizers/trijoint.py.151: total_norm: 38.180222 [I 2022-10-10 15:56:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1300/2304 [I 2022-10-10 15:56:32] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:14:35 [I 2022-10-10 15:56:32] ...trap/engines/engine.py.229: process: 0.88295 | load: 0.00310 [I 2022-10-10 15:56:32] ...trap/engines/engine.py.230: loss: 0.10322 [I 2022-10-10 15:56:32] ...optimizers/trijoint.py.151: total_norm: 16.357210 [I 2022-10-10 15:56:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1310/2304 [I 2022-10-10 15:56:40] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:14:21 [I 2022-10-10 15:56:40] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00307 [I 2022-10-10 15:56:40] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-10 15:56:40] ...optimizers/trijoint.py.151: total_norm: 25.408293 [I 2022-10-10 15:56:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1320/2304 [I 2022-10-10 15:56:49] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:14:00 [I 2022-10-10 15:56:49] ...trap/engines/engine.py.229: process: 0.87221 | load: 0.00306 [I 2022-10-10 15:56:49] ...trap/engines/engine.py.230: loss: 0.16056 [I 2022-10-10 15:56:49] ...optimizers/trijoint.py.151: total_norm: 8.033210 [I 2022-10-10 15:56:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1330/2304 [I 2022-10-10 15:56:57] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:53 [I 2022-10-10 15:56:57] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00303 [I 2022-10-10 15:56:57] ...trap/engines/engine.py.230: loss: 0.12099 [I 2022-10-10 15:56:57] ...optimizers/trijoint.py.151: total_norm: 17.286804 [I 2022-10-10 15:57:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1340/2304 [I 2022-10-10 15:57:06] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:45 [I 2022-10-10 15:57:06] ...trap/engines/engine.py.229: process: 0.85872 | load: 0.00300 [I 2022-10-10 15:57:06] ...trap/engines/engine.py.230: loss: 0.10244 [I 2022-10-10 15:57:06] ...optimizers/trijoint.py.151: total_norm: 9.953705 [I 2022-10-10 15:57:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 1350/2304 [I 2022-10-10 15:57:15] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:29 [I 2022-10-10 15:57:15] ...trap/engines/engine.py.229: process: 0.83732 | load: 0.00301 [I 2022-10-10 15:57:15] ...trap/engines/engine.py.230: loss: 0.16991 [I 2022-10-10 15:57:15] ...optimizers/trijoint.py.151: total_norm: 23.229815 [I 2022-10-10 15:57:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1360/2304 [I 2022-10-10 15:57:23] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:13:24 [I 2022-10-10 15:57:23] ...trap/engines/engine.py.229: process: 0.84715 | load: 0.00295 [I 2022-10-10 15:57:23] ...trap/engines/engine.py.230: loss: 0.13251 [I 2022-10-10 15:57:23] ...optimizers/trijoint.py.151: total_norm: 12.555050 [I 2022-10-10 15:57:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1370/2304 [I 2022-10-10 15:57:32] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:15 [I 2022-10-10 15:57:32] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00305 [I 2022-10-10 15:57:32] ...trap/engines/engine.py.230: loss: 0.18961 [I 2022-10-10 15:57:32] ...optimizers/trijoint.py.151: total_norm: 19.463669 [I 2022-10-10 15:57:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1380/2304 [I 2022-10-10 15:57:40] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:13:08 [I 2022-10-10 15:57:40] ...trap/engines/engine.py.229: process: 0.86800 | load: 0.00307 [I 2022-10-10 15:57:40] ...trap/engines/engine.py.230: loss: 0.22816 [I 2022-10-10 15:57:40] ...optimizers/trijoint.py.151: total_norm: 26.369778 [I 2022-10-10 15:57:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1390/2304 [I 2022-10-10 15:57:49] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:13:04 [I 2022-10-10 15:57:49] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00312 [I 2022-10-10 15:57:49] ...trap/engines/engine.py.230: loss: 0.12621 [I 2022-10-10 15:57:49] ...optimizers/trijoint.py.151: total_norm: 15.885464 [I 2022-10-10 15:57:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1400/2304 [I 2022-10-10 15:57:57] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:53 [I 2022-10-10 15:57:57] ...trap/engines/engine.py.229: process: 0.86422 | load: 0.00297 [I 2022-10-10 15:57:57] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-10 15:57:57] ...optimizers/trijoint.py.151: total_norm: 13.433892 [I 2022-10-10 15:58:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1410/2304 [I 2022-10-10 15:58:06] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:43 [I 2022-10-10 15:58:06] ...trap/engines/engine.py.229: process: 0.84798 | load: 0.00295 [I 2022-10-10 15:58:06] ...trap/engines/engine.py.230: loss: 0.11016 [I 2022-10-10 15:58:06] ...optimizers/trijoint.py.151: total_norm: 9.420741 [I 2022-10-10 15:58:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1420/2304 [I 2022-10-10 15:58:14] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:33 [I 2022-10-10 15:58:14] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00298 [I 2022-10-10 15:58:14] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-10 15:58:14] ...optimizers/trijoint.py.151: total_norm: 10.034601 [I 2022-10-10 15:58:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1430/2304 [I 2022-10-10 15:58:23] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:12:25 [I 2022-10-10 15:58:23] ...trap/engines/engine.py.229: process: 0.84926 | load: 0.00296 [I 2022-10-10 15:58:23] ...trap/engines/engine.py.230: loss: 0.14639 [I 2022-10-10 15:58:23] ...optimizers/trijoint.py.151: total_norm: 24.347916 [I 2022-10-10 15:58:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1440/2304 [I 2022-10-10 15:58:32] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:12:14 [I 2022-10-10 15:58:32] ...trap/engines/engine.py.229: process: 0.84099 | load: 0.00318 [I 2022-10-10 15:58:32] ...trap/engines/engine.py.230: loss: 0.20100 [I 2022-10-10 15:58:32] ...optimizers/trijoint.py.151: total_norm: 48.844090 [I 2022-10-10 15:58:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1450/2304 [I 2022-10-10 15:58:40] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:12:05 [I 2022-10-10 15:58:40] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00294 [I 2022-10-10 15:58:40] ...trap/engines/engine.py.230: loss: 0.11251 [I 2022-10-10 15:58:40] ...optimizers/trijoint.py.151: total_norm: 13.335838 [I 2022-10-10 15:58:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1460/2304 [I 2022-10-10 15:58:49] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:12:00 [I 2022-10-10 15:58:49] ...trap/engines/engine.py.229: process: 0.85449 | load: 0.00296 [I 2022-10-10 15:58:49] ...trap/engines/engine.py.230: loss: 0.14553 [I 2022-10-10 15:58:49] ...optimizers/trijoint.py.151: total_norm: 10.034607 [I 2022-10-10 15:58:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1470/2304 [I 2022-10-10 15:58:57] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:53 [I 2022-10-10 15:58:57] ...trap/engines/engine.py.229: process: 0.85801 | load: 0.00294 [I 2022-10-10 15:58:57] ...trap/engines/engine.py.230: loss: 0.11127 [I 2022-10-10 15:58:57] ...optimizers/trijoint.py.151: total_norm: 35.986401 [I 2022-10-10 15:59:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1480/2304 [I 2022-10-10 15:59:06] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:35 [I 2022-10-10 15:59:06] ...trap/engines/engine.py.229: process: 0.83523 | load: 0.00298 [I 2022-10-10 15:59:06] ...trap/engines/engine.py.230: loss: 0.10019 [I 2022-10-10 15:59:06] ...optimizers/trijoint.py.151: total_norm: 11.288460 [I 2022-10-10 15:59:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1490/2304 [I 2022-10-10 15:59:14] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:32 [I 2022-10-10 15:59:14] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00302 [I 2022-10-10 15:59:14] ...trap/engines/engine.py.230: loss: 0.12139 [I 2022-10-10 15:59:14] ...optimizers/trijoint.py.151: total_norm: 12.992468 [I 2022-10-10 15:59:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1500/2304 [I 2022-10-10 15:59:23] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:22 [I 2022-10-10 15:59:23] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00299 [I 2022-10-10 15:59:23] ...trap/engines/engine.py.230: loss: 0.17005 [I 2022-10-10 15:59:23] ...optimizers/trijoint.py.151: total_norm: 11.248616 [I 2022-10-10 15:59:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1510/2304 [I 2022-10-10 15:59:32] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:11:21 [I 2022-10-10 15:59:32] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00306 [I 2022-10-10 15:59:32] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-10 15:59:32] ...optimizers/trijoint.py.151: total_norm: 17.112226 [I 2022-10-10 15:59:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1520/2304 [I 2022-10-10 15:59:40] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:11:16 [I 2022-10-10 15:59:40] ...trap/engines/engine.py.229: process: 0.87832 | load: 0.00332 [I 2022-10-10 15:59:40] ...trap/engines/engine.py.230: loss: 0.14408 [I 2022-10-10 15:59:40] ...optimizers/trijoint.py.151: total_norm: 11.979111 [I 2022-10-10 15:59:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1530/2304 [I 2022-10-10 15:59:49] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:11:05 [I 2022-10-10 15:59:49] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00308 [I 2022-10-10 15:59:49] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-10 15:59:49] ...optimizers/trijoint.py.151: total_norm: 22.390345 [I 2022-10-10 15:59:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1540/2304 [I 2022-10-10 15:59:57] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:11:00 [I 2022-10-10 15:59:57] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00307 [I 2022-10-10 15:59:57] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-10 15:59:57] ...optimizers/trijoint.py.151: total_norm: 11.321509 [I 2022-10-10 16:00:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1550/2304 [I 2022-10-10 16:00:06] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:47 [I 2022-10-10 16:00:06] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00306 [I 2022-10-10 16:00:06] ...trap/engines/engine.py.230: loss: 0.10355 [I 2022-10-10 16:00:06] ...optimizers/trijoint.py.151: total_norm: 10.365216 [I 2022-10-10 16:00:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 1560/2304 [I 2022-10-10 16:00:15] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:38 [I 2022-10-10 16:00:15] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00308 [I 2022-10-10 16:00:15] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-10 16:00:15] ...optimizers/trijoint.py.151: total_norm: 10.751528 [I 2022-10-10 16:00:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1570/2304 [I 2022-10-10 16:00:23] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:10:29 [I 2022-10-10 16:00:23] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00311 [I 2022-10-10 16:00:23] ...trap/engines/engine.py.230: loss: 0.11224 [I 2022-10-10 16:00:23] ...optimizers/trijoint.py.151: total_norm: 14.752654 [I 2022-10-10 16:00:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1580/2304 [I 2022-10-10 16:00:32] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:10:22 [I 2022-10-10 16:00:32] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00309 [I 2022-10-10 16:00:32] ...trap/engines/engine.py.230: loss: 0.10446 [I 2022-10-10 16:00:32] ...optimizers/trijoint.py.151: total_norm: 9.206407 [I 2022-10-10 16:00:41] ...trap/engines/engine.py.224: train: epoch 77 | batch 1590/2304 [I 2022-10-10 16:00:41] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:10:14 [I 2022-10-10 16:00:41] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00318 [I 2022-10-10 16:00:41] ...trap/engines/engine.py.230: loss: 0.10074 [I 2022-10-10 16:00:41] ...optimizers/trijoint.py.151: total_norm: 31.124853 [I 2022-10-10 16:00:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1600/2304 [I 2022-10-10 16:00:49] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:10:08 [I 2022-10-10 16:00:49] ...trap/engines/engine.py.229: process: 0.85960 | load: 0.00338 [I 2022-10-10 16:00:49] ...trap/engines/engine.py.230: loss: 0.19258 [I 2022-10-10 16:00:49] ...optimizers/trijoint.py.151: total_norm: 11.324082 [I 2022-10-10 16:00:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 1610/2304 [I 2022-10-10 16:00:58] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:10:02 [I 2022-10-10 16:00:58] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00319 [I 2022-10-10 16:00:58] ...trap/engines/engine.py.230: loss: 0.10217 [I 2022-10-10 16:00:58] ...optimizers/trijoint.py.151: total_norm: 10.189888 [I 2022-10-10 16:01:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 1620/2304 [I 2022-10-10 16:01:07] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:47 [I 2022-10-10 16:01:07] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00314 [I 2022-10-10 16:01:07] ...trap/engines/engine.py.230: loss: 0.16728 [I 2022-10-10 16:01:07] ...optimizers/trijoint.py.151: total_norm: 12.749674 [I 2022-10-10 16:01:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 1630/2304 [I 2022-10-10 16:01:15] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:41 [I 2022-10-10 16:01:15] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00313 [I 2022-10-10 16:01:15] ...trap/engines/engine.py.230: loss: 0.11137 [I 2022-10-10 16:01:15] ...optimizers/trijoint.py.151: total_norm: 27.285843 [I 2022-10-10 16:01:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 1640/2304 [I 2022-10-10 16:01:24] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:09:32 [I 2022-10-10 16:01:24] ...trap/engines/engine.py.229: process: 0.85446 | load: 0.00312 [I 2022-10-10 16:01:24] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-10 16:01:24] ...optimizers/trijoint.py.151: total_norm: 11.534844 [I 2022-10-10 16:01:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 1650/2304 [I 2022-10-10 16:01:33] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:09:23 [I 2022-10-10 16:01:33] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00318 [I 2022-10-10 16:01:33] ...trap/engines/engine.py.230: loss: 0.11534 [I 2022-10-10 16:01:33] ...optimizers/trijoint.py.151: total_norm: 13.809120 [I 2022-10-10 16:01:41] ...trap/engines/engine.py.224: train: epoch 77 | batch 1660/2304 [I 2022-10-10 16:01:41] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:09:13 [I 2022-10-10 16:01:41] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00312 [I 2022-10-10 16:01:41] ...trap/engines/engine.py.230: loss: 0.12469 [I 2022-10-10 16:01:41] ...optimizers/trijoint.py.151: total_norm: 19.631159 [I 2022-10-10 16:01:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 1670/2304 [I 2022-10-10 16:01:50] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:09:04 [I 2022-10-10 16:01:50] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00320 [I 2022-10-10 16:01:50] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-10 16:01:50] ...optimizers/trijoint.py.151: total_norm: 30.269674 [I 2022-10-10 16:01:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 1680/2304 [I 2022-10-10 16:01:58] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:56 [I 2022-10-10 16:01:58] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00318 [I 2022-10-10 16:01:58] ...trap/engines/engine.py.230: loss: 0.17153 [I 2022-10-10 16:01:58] ...optimizers/trijoint.py.151: total_norm: 10.800138 [I 2022-10-10 16:02:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 1690/2304 [I 2022-10-10 16:02:07] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:49 [I 2022-10-10 16:02:07] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00318 [I 2022-10-10 16:02:07] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-10 16:02:07] ...optimizers/trijoint.py.151: total_norm: 8.717313 [I 2022-10-10 16:02:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1700/2304 [I 2022-10-10 16:02:16] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:42 [I 2022-10-10 16:02:16] ...trap/engines/engine.py.229: process: 0.87116 | load: 0.00313 [I 2022-10-10 16:02:16] ...trap/engines/engine.py.230: loss: 0.16789 [I 2022-10-10 16:02:16] ...optimizers/trijoint.py.151: total_norm: 15.320284 [I 2022-10-10 16:02:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 1710/2304 [I 2022-10-10 16:02:24] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:30 [I 2022-10-10 16:02:24] ...trap/engines/engine.py.229: process: 0.83650 | load: 0.00318 [I 2022-10-10 16:02:24] ...trap/engines/engine.py.230: loss: 0.10300 [I 2022-10-10 16:02:24] ...optimizers/trijoint.py.151: total_norm: 6.443663 [I 2022-10-10 16:02:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 1720/2304 [I 2022-10-10 16:02:33] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:08:24 [I 2022-10-10 16:02:33] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00336 [I 2022-10-10 16:02:33] ...trap/engines/engine.py.230: loss: 0.13161 [I 2022-10-10 16:02:33] ...optimizers/trijoint.py.151: total_norm: 22.011316 [I 2022-10-10 16:02:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 1730/2304 [I 2022-10-10 16:02:42] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:08:17 [I 2022-10-10 16:02:42] ...trap/engines/engine.py.229: process: 0.87047 | load: 0.00329 [I 2022-10-10 16:02:42] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-10 16:02:42] ...optimizers/trijoint.py.151: total_norm: 25.809910 [I 2022-10-10 16:02:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 1740/2304 [I 2022-10-10 16:02:50] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:08:09 [I 2022-10-10 16:02:50] ...trap/engines/engine.py.229: process: 0.86232 | load: 0.00329 [I 2022-10-10 16:02:50] ...trap/engines/engine.py.230: loss: 0.14693 [I 2022-10-10 16:02:50] ...optimizers/trijoint.py.151: total_norm: 10.640924 [I 2022-10-10 16:02:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 1750/2304 [I 2022-10-10 16:02:59] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:08:01 [I 2022-10-10 16:02:59] ...trap/engines/engine.py.229: process: 0.88864 | load: 0.00328 [I 2022-10-10 16:02:59] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-10 16:02:59] ...optimizers/trijoint.py.151: total_norm: 19.583685 [I 2022-10-10 16:03:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1760/2304 [I 2022-10-10 16:03:08] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:49 [I 2022-10-10 16:03:08] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00311 [I 2022-10-10 16:03:08] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-10 16:03:08] ...optimizers/trijoint.py.151: total_norm: 33.059917 [I 2022-10-10 16:03:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1770/2304 [I 2022-10-10 16:03:16] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:42 [I 2022-10-10 16:03:16] ...trap/engines/engine.py.229: process: 0.84823 | load: 0.00317 [I 2022-10-10 16:03:16] ...trap/engines/engine.py.230: loss: 0.10662 [I 2022-10-10 16:03:16] ...optimizers/trijoint.py.151: total_norm: 7.870842 [I 2022-10-10 16:03:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1780/2304 [I 2022-10-10 16:03:25] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:32 [I 2022-10-10 16:03:25] ...trap/engines/engine.py.229: process: 0.85489 | load: 0.00299 [I 2022-10-10 16:03:25] ...trap/engines/engine.py.230: loss: 0.12209 [I 2022-10-10 16:03:25] ...optimizers/trijoint.py.151: total_norm: 20.593008 [I 2022-10-10 16:03:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 1790/2304 [I 2022-10-10 16:03:34] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:17 [I 2022-10-10 16:03:34] ...trap/engines/engine.py.229: process: 0.83726 | load: 0.00320 [I 2022-10-10 16:03:34] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-10 16:03:34] ...optimizers/trijoint.py.151: total_norm: 11.169322 [I 2022-10-10 16:03:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 1800/2304 [I 2022-10-10 16:03:42] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:07:09 [I 2022-10-10 16:03:42] ...trap/engines/engine.py.229: process: 0.85061 | load: 0.00311 [I 2022-10-10 16:03:42] ...trap/engines/engine.py.230: loss: 0.16832 [I 2022-10-10 16:03:42] ...optimizers/trijoint.py.151: total_norm: 25.213932 [I 2022-10-10 16:03:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 1810/2304 [I 2022-10-10 16:03:51] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:07:02 [I 2022-10-10 16:03:51] ...trap/engines/engine.py.229: process: 0.86668 | load: 0.00309 [I 2022-10-10 16:03:51] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-10 16:03:51] ...optimizers/trijoint.py.151: total_norm: 19.708643 [I 2022-10-10 16:03:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 1820/2304 [I 2022-10-10 16:03:59] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:51 [I 2022-10-10 16:03:59] ...trap/engines/engine.py.229: process: 0.84094 | load: 0.00307 [I 2022-10-10 16:03:59] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-10 16:03:59] ...optimizers/trijoint.py.151: total_norm: 31.657164 [I 2022-10-10 16:04:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1830/2304 [I 2022-10-10 16:04:08] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:42 [I 2022-10-10 16:04:08] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00306 [I 2022-10-10 16:04:08] ...trap/engines/engine.py.230: loss: 0.14436 [I 2022-10-10 16:04:08] ...optimizers/trijoint.py.151: total_norm: 12.161286 [I 2022-10-10 16:04:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1840/2304 [I 2022-10-10 16:04:16] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:36 [I 2022-10-10 16:04:16] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00305 [I 2022-10-10 16:04:16] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-10 16:04:16] ...optimizers/trijoint.py.151: total_norm: 9.275668 [I 2022-10-10 16:04:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1850/2304 [I 2022-10-10 16:04:25] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:27 [I 2022-10-10 16:04:25] ...trap/engines/engine.py.229: process: 0.85995 | load: 0.00306 [I 2022-10-10 16:04:25] ...trap/engines/engine.py.230: loss: 0.13793 [I 2022-10-10 16:04:25] ...optimizers/trijoint.py.151: total_norm: 16.868082 [I 2022-10-10 16:04:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 1860/2304 [I 2022-10-10 16:04:34] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:20 [I 2022-10-10 16:04:34] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00338 [I 2022-10-10 16:04:34] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-10 16:04:34] ...optimizers/trijoint.py.151: total_norm: 13.572981 [I 2022-10-10 16:04:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 1870/2304 [I 2022-10-10 16:04:42] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:06:08 [I 2022-10-10 16:04:42] ...trap/engines/engine.py.229: process: 0.84325 | load: 0.00319 [I 2022-10-10 16:04:42] ...trap/engines/engine.py.230: loss: 0.11891 [I 2022-10-10 16:04:42] ...optimizers/trijoint.py.151: total_norm: 10.506946 [I 2022-10-10 16:04:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 1880/2304 [I 2022-10-10 16:04:51] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:06:04 [I 2022-10-10 16:04:51] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00309 [I 2022-10-10 16:04:51] ...trap/engines/engine.py.230: loss: 0.08817 [I 2022-10-10 16:04:51] ...optimizers/trijoint.py.151: total_norm: 9.831538 [I 2022-10-10 16:04:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 1890/2304 [I 2022-10-10 16:04:59] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:53 [I 2022-10-10 16:04:59] ...trap/engines/engine.py.229: process: 0.84264 | load: 0.00307 [I 2022-10-10 16:04:59] ...trap/engines/engine.py.230: loss: 0.17135 [I 2022-10-10 16:04:59] ...optimizers/trijoint.py.151: total_norm: 11.682083 [I 2022-10-10 16:05:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1900/2304 [I 2022-10-10 16:05:08] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:05:44 [I 2022-10-10 16:05:08] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00294 [I 2022-10-10 16:05:08] ...trap/engines/engine.py.230: loss: 0.08504 [I 2022-10-10 16:05:08] ...optimizers/trijoint.py.151: total_norm: 10.578226 [I 2022-10-10 16:05:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1910/2304 [I 2022-10-10 16:05:16] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:35 [I 2022-10-10 16:05:16] ...trap/engines/engine.py.229: process: 0.84908 | load: 0.00297 [I 2022-10-10 16:05:16] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-10 16:05:16] ...optimizers/trijoint.py.151: total_norm: 13.419065 [I 2022-10-10 16:05:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1920/2304 [I 2022-10-10 16:05:25] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:05:27 [I 2022-10-10 16:05:25] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00301 [I 2022-10-10 16:05:25] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-10 16:05:25] ...optimizers/trijoint.py.151: total_norm: 17.560993 [I 2022-10-10 16:05:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 1930/2304 [I 2022-10-10 16:05:33] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:18 [I 2022-10-10 16:05:33] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00295 [I 2022-10-10 16:05:33] ...trap/engines/engine.py.230: loss: 0.12274 [I 2022-10-10 16:05:33] ...optimizers/trijoint.py.151: total_norm: 11.269519 [I 2022-10-10 16:05:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 1940/2304 [I 2022-10-10 16:05:42] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:05:11 [I 2022-10-10 16:05:42] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00300 [I 2022-10-10 16:05:42] ...trap/engines/engine.py.230: loss: 0.14122 [I 2022-10-10 16:05:42] ...optimizers/trijoint.py.151: total_norm: 12.888747 [I 2022-10-10 16:05:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 1950/2304 [I 2022-10-10 16:05:51] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:05:02 [I 2022-10-10 16:05:51] ...trap/engines/engine.py.229: process: 0.86210 | load: 0.00309 [I 2022-10-10 16:05:51] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-10 16:05:51] ...optimizers/trijoint.py.151: total_norm: 10.417070 [I 2022-10-10 16:05:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 1960/2304 [I 2022-10-10 16:05:59] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:53 [I 2022-10-10 16:05:59] ...trap/engines/engine.py.229: process: 0.88389 | load: 0.00301 [I 2022-10-10 16:05:59] ...trap/engines/engine.py.230: loss: 0.11510 [I 2022-10-10 16:05:59] ...optimizers/trijoint.py.151: total_norm: 15.505087 [I 2022-10-10 16:06:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1970/2304 [I 2022-10-10 16:06:08] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:04:47 [I 2022-10-10 16:06:08] ...trap/engines/engine.py.229: process: 0.85957 | load: 0.00309 [I 2022-10-10 16:06:08] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-10 16:06:08] ...optimizers/trijoint.py.151: total_norm: 9.091178 [I 2022-10-10 16:06:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1980/2304 [I 2022-10-10 16:06:16] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:37 [I 2022-10-10 16:06:16] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00311 [I 2022-10-10 16:06:16] ...trap/engines/engine.py.230: loss: 0.08556 [I 2022-10-10 16:06:16] ...optimizers/trijoint.py.151: total_norm: 7.705365 [I 2022-10-10 16:06:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1990/2304 [I 2022-10-10 16:06:25] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:04:28 [I 2022-10-10 16:06:25] ...trap/engines/engine.py.229: process: 0.85539 | load: 0.00310 [I 2022-10-10 16:06:25] ...trap/engines/engine.py.230: loss: 0.20649 [I 2022-10-10 16:06:25] ...optimizers/trijoint.py.151: total_norm: 25.662745 [I 2022-10-10 16:06:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 2000/2304 [I 2022-10-10 16:06:34] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:04:21 [I 2022-10-10 16:06:34] ...trap/engines/engine.py.229: process: 0.86033 | load: 0.00312 [I 2022-10-10 16:06:34] ...trap/engines/engine.py.230: loss: 0.09038 [I 2022-10-10 16:06:34] ...optimizers/trijoint.py.151: total_norm: 8.792213 [I 2022-10-10 16:06:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 2010/2304 [I 2022-10-10 16:06:42] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:04:13 [I 2022-10-10 16:06:42] ...trap/engines/engine.py.229: process: 0.86023 | load: 0.00342 [I 2022-10-10 16:06:42] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-10 16:06:42] ...optimizers/trijoint.py.151: total_norm: 16.663246 [I 2022-10-10 16:06:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 2020/2304 [I 2022-10-10 16:06:51] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:04:04 [I 2022-10-10 16:06:51] ...trap/engines/engine.py.229: process: 0.87206 | load: 0.00311 [I 2022-10-10 16:06:51] ...trap/engines/engine.py.230: loss: 0.24018 [I 2022-10-10 16:06:51] ...optimizers/trijoint.py.151: total_norm: 17.382795 [I 2022-10-10 16:07:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2030/2304 [I 2022-10-10 16:07:00] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:56 [I 2022-10-10 16:07:00] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00332 [I 2022-10-10 16:07:00] ...trap/engines/engine.py.230: loss: 0.13297 [I 2022-10-10 16:07:00] ...optimizers/trijoint.py.151: total_norm: 22.149385 [I 2022-10-10 16:07:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 2040/2304 [I 2022-10-10 16:07:26] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:06:21 [I 2022-10-10 16:07:26] ...trap/engines/engine.py.229: process: 0.85326 | load: 0.00313 [I 2022-10-10 16:07:26] ...trap/engines/engine.py.230: loss: 0.14029 [I 2022-10-10 16:07:26] ...optimizers/trijoint.py.151: total_norm: 19.586998 [I 2022-10-10 16:07:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 2050/2304 [I 2022-10-10 16:07:34] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:03:53 [I 2022-10-10 16:07:34] ...trap/engines/engine.py.229: process: 0.86922 | load: 0.00319 [I 2022-10-10 16:07:34] ...trap/engines/engine.py.230: loss: 0.12318 [I 2022-10-10 16:07:34] ...optimizers/trijoint.py.151: total_norm: 13.721040 [I 2022-10-10 16:07:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 2060/2304 [I 2022-10-10 16:07:43] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:03:30 [I 2022-10-10 16:07:43] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00320 [I 2022-10-10 16:07:43] ...trap/engines/engine.py.230: loss: 0.12117 [I 2022-10-10 16:07:43] ...optimizers/trijoint.py.151: total_norm: 13.129942 [I 2022-10-10 16:07:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 2070/2304 [I 2022-10-10 16:07:52] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:03:20 [I 2022-10-10 16:07:52] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00327 [I 2022-10-10 16:07:52] ...trap/engines/engine.py.230: loss: 0.12430 [I 2022-10-10 16:07:52] ...optimizers/trijoint.py.151: total_norm: 11.687720 [I 2022-10-10 16:08:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2080/2304 [I 2022-10-10 16:08:00] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:03:10 [I 2022-10-10 16:08:00] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00321 [I 2022-10-10 16:08:00] ...trap/engines/engine.py.230: loss: 0.17546 [I 2022-10-10 16:08:00] ...optimizers/trijoint.py.151: total_norm: 26.000813 [I 2022-10-10 16:08:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 2090/2304 [I 2022-10-10 16:08:09] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:03:02 [I 2022-10-10 16:08:09] ...trap/engines/engine.py.229: process: 0.84221 | load: 0.00322 [I 2022-10-10 16:08:09] ...trap/engines/engine.py.230: loss: 0.16252 [I 2022-10-10 16:08:09] ...optimizers/trijoint.py.151: total_norm: 11.649639 [I 2022-10-10 16:08:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 2100/2304 [I 2022-10-10 16:08:17] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:54 [I 2022-10-10 16:08:17] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00331 [I 2022-10-10 16:08:17] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-10 16:08:17] ...optimizers/trijoint.py.151: total_norm: 13.493507 [I 2022-10-10 16:08:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 2110/2304 [I 2022-10-10 16:08:26] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:45 [I 2022-10-10 16:08:26] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00308 [I 2022-10-10 16:08:26] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-10 16:08:26] ...optimizers/trijoint.py.151: total_norm: 17.590322 [I 2022-10-10 16:08:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 2120/2304 [I 2022-10-10 16:08:34] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:37 [I 2022-10-10 16:08:34] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00311 [I 2022-10-10 16:08:34] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-10 16:08:34] ...optimizers/trijoint.py.151: total_norm: 7.835559 [I 2022-10-10 16:08:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 2130/2304 [I 2022-10-10 16:08:43] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:02:29 [I 2022-10-10 16:08:43] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00309 [I 2022-10-10 16:08:43] ...trap/engines/engine.py.230: loss: 0.13990 [I 2022-10-10 16:08:43] ...optimizers/trijoint.py.151: total_norm: 16.457783 [I 2022-10-10 16:08:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 2140/2304 [I 2022-10-10 16:08:51] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:02:21 [I 2022-10-10 16:08:51] ...trap/engines/engine.py.229: process: 0.85325 | load: 0.00310 [I 2022-10-10 16:08:51] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-10 16:08:51] ...optimizers/trijoint.py.151: total_norm: 9.150418 [I 2022-10-10 16:09:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2150/2304 [I 2022-10-10 16:09:00] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:02:12 [I 2022-10-10 16:09:00] ...trap/engines/engine.py.229: process: 0.84594 | load: 0.00310 [I 2022-10-10 16:09:00] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-10 16:09:00] ...optimizers/trijoint.py.151: total_norm: 8.948049 [I 2022-10-10 16:09:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 2160/2304 [I 2022-10-10 16:09:09] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:02:02 [I 2022-10-10 16:09:09] ...trap/engines/engine.py.229: process: 0.84898 | load: 0.00315 [I 2022-10-10 16:09:09] ...trap/engines/engine.py.230: loss: 0.14277 [I 2022-10-10 16:09:09] ...optimizers/trijoint.py.151: total_norm: 17.901779 [I 2022-10-10 16:09:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 2170/2304 [I 2022-10-10 16:09:17] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:54 [I 2022-10-10 16:09:17] ...trap/engines/engine.py.229: process: 0.85811 | load: 0.00317 [I 2022-10-10 16:09:17] ...trap/engines/engine.py.230: loss: 0.16477 [I 2022-10-10 16:09:17] ...optimizers/trijoint.py.151: total_norm: 23.045193 [I 2022-10-10 16:09:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 2180/2304 [I 2022-10-10 16:09:26] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:45 [I 2022-10-10 16:09:26] ...trap/engines/engine.py.229: process: 0.85248 | load: 0.00313 [I 2022-10-10 16:09:26] ...trap/engines/engine.py.230: loss: 0.10881 [I 2022-10-10 16:09:26] ...optimizers/trijoint.py.151: total_norm: 12.497153 [I 2022-10-10 16:09:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 2190/2304 [I 2022-10-10 16:09:34] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:37 [I 2022-10-10 16:09:34] ...trap/engines/engine.py.229: process: 0.86032 | load: 0.00311 [I 2022-10-10 16:09:34] ...trap/engines/engine.py.230: loss: 0.09448 [I 2022-10-10 16:09:34] ...optimizers/trijoint.py.151: total_norm: 15.509130 [I 2022-10-10 16:09:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 2200/2304 [I 2022-10-10 16:09:43] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:01:28 [I 2022-10-10 16:09:43] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00313 [I 2022-10-10 16:09:43] ...trap/engines/engine.py.230: loss: 0.11754 [I 2022-10-10 16:09:43] ...optimizers/trijoint.py.151: total_norm: 14.424788 [I 2022-10-10 16:09:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 2210/2304 [I 2022-10-10 16:09:51] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:01:20 [I 2022-10-10 16:09:51] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00322 [I 2022-10-10 16:09:51] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-10 16:09:51] ...optimizers/trijoint.py.151: total_norm: 16.808403 [I 2022-10-10 16:10:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2220/2304 [I 2022-10-10 16:10:00] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:01:11 [I 2022-10-10 16:10:00] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00304 [I 2022-10-10 16:10:00] ...trap/engines/engine.py.230: loss: 0.12788 [I 2022-10-10 16:10:00] ...optimizers/trijoint.py.151: total_norm: 31.523935 [I 2022-10-10 16:10:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 2230/2304 [I 2022-10-10 16:10:08] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:01:02 [I 2022-10-10 16:10:08] ...trap/engines/engine.py.229: process: 0.83834 | load: 0.00295 [I 2022-10-10 16:10:08] ...trap/engines/engine.py.230: loss: 0.08770 [I 2022-10-10 16:10:08] ...optimizers/trijoint.py.151: total_norm: 12.977467 [I 2022-10-10 16:10:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 2240/2304 [I 2022-10-10 16:10:17] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:53 [I 2022-10-10 16:10:17] ...trap/engines/engine.py.229: process: 0.83007 | load: 0.00296 [I 2022-10-10 16:10:17] ...trap/engines/engine.py.230: loss: 0.14357 [I 2022-10-10 16:10:17] ...optimizers/trijoint.py.151: total_norm: 26.430397 [I 2022-10-10 16:10:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 2250/2304 [I 2022-10-10 16:10:25] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:45 [I 2022-10-10 16:10:25] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00300 [I 2022-10-10 16:10:25] ...trap/engines/engine.py.230: loss: 0.16045 [I 2022-10-10 16:10:25] ...optimizers/trijoint.py.151: total_norm: 21.059563 [I 2022-10-10 16:10:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 2260/2304 [I 2022-10-10 16:10:34] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:37 [I 2022-10-10 16:10:34] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00294 [I 2022-10-10 16:10:34] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-10 16:10:34] ...optimizers/trijoint.py.151: total_norm: 37.694683 [I 2022-10-10 16:10:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 2270/2304 [I 2022-10-10 16:10:42] ...trap/engines/engine.py.228: elapsed: 0:33:15 | left: 0:00:28 [I 2022-10-10 16:10:42] ...trap/engines/engine.py.229: process: 0.83752 | load: 0.00298 [I 2022-10-10 16:10:42] ...trap/engines/engine.py.230: loss: 0.09115 [I 2022-10-10 16:10:42] ...optimizers/trijoint.py.151: total_norm: 18.147448 [I 2022-10-10 16:10:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 2280/2304 [I 2022-10-10 16:10:51] ...trap/engines/engine.py.228: elapsed: 0:33:23 | left: 0:00:20 [I 2022-10-10 16:10:51] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00294 [I 2022-10-10 16:10:51] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-10 16:10:51] ...optimizers/trijoint.py.151: total_norm: 10.563679 [I 2022-10-10 16:10:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 2290/2304 [I 2022-10-10 16:10:59] ...trap/engines/engine.py.228: elapsed: 0:33:31 | left: 0:00:11 [I 2022-10-10 16:10:59] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00297 [I 2022-10-10 16:10:59] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-10 16:10:59] ...optimizers/trijoint.py.151: total_norm: 18.061008 [I 2022-10-10 16:11:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 2300/2304 [I 2022-10-10 16:11:08] ...trap/engines/engine.py.228: elapsed: 0:33:40 | left: 0:00:03 [I 2022-10-10 16:11:08] ...trap/engines/engine.py.229: process: 0.82118 | load: 0.00014 [I 2022-10-10 16:11:08] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-10 16:11:08] ...optimizers/trijoint.py.151: total_norm: 7.339614 [I 2022-10-10 16:11:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 2304/2304 [I 2022-10-10 16:11:11] ...trap/engines/engine.py.228: elapsed: 0:33:43 | left: 0:00:00 [I 2022-10-10 16:11:11] ...trap/engines/engine.py.229: process: 0.85002 | load: 0.00015 [I 2022-10-10 16:11:11] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-10 16:11:11] ...optimizers/trijoint.py.151: total_norm: 18.481785 [S 2022-10-10 16:11:13] ...trap/engines/engine.py.129: train_epoch.epoch: 77 [S 2022-10-10 16:11:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.13484651779921608 [I 2022-10-10 16:11:13] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 16:11:14] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 77 [I 2022-10-10 16:11:14] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 16:11:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 16:11:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 16:11:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 16:11:19] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 16:11:19] ...trap/engines/engine.py.321: eval: epoch 77 | batch 0/510 [I 2022-10-10 16:11:19] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:02 [I 2022-10-10 16:11:19] ...trap/engines/engine.py.326: process: 5.18123 | load: 4.07725 [I 2022-10-10 16:11:23] ...trap/engines/engine.py.321: eval: epoch 77 | batch 10/510 [I 2022-10-10 16:11:23] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:45 [I 2022-10-10 16:11:23] ...trap/engines/engine.py.326: process: 0.42669 | load: 0.00019 [I 2022-10-10 16:11:28] ...trap/engines/engine.py.321: eval: epoch 77 | batch 20/510 [I 2022-10-10 16:11:28] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:03:56 [I 2022-10-10 16:11:28] ...trap/engines/engine.py.326: process: 0.42961 | load: 0.00018 [I 2022-10-10 16:11:32] ...trap/engines/engine.py.321: eval: epoch 77 | batch 30/510 [I 2022-10-10 16:11:32] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:03:46 [I 2022-10-10 16:11:32] ...trap/engines/engine.py.326: process: 0.42693 | load: 0.00019 [I 2022-10-10 16:11:37] ...trap/engines/engine.py.321: eval: epoch 77 | batch 40/510 [I 2022-10-10 16:11:37] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:03:51 [I 2022-10-10 16:11:37] ...trap/engines/engine.py.326: process: 0.41969 | load: 0.00020 [I 2022-10-10 16:11:42] ...trap/engines/engine.py.321: eval: epoch 77 | batch 50/510 [I 2022-10-10 16:11:42] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:03:20 [I 2022-10-10 16:11:42] ...trap/engines/engine.py.326: process: 0.42924 | load: 0.00018 [I 2022-10-10 16:11:46] ...trap/engines/engine.py.321: eval: epoch 77 | batch 60/510 [I 2022-10-10 16:11:46] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:12 [I 2022-10-10 16:11:46] ...trap/engines/engine.py.326: process: 0.42193 | load: 0.00023 [I 2022-10-10 16:11:50] ...trap/engines/engine.py.321: eval: epoch 77 | batch 70/510 [I 2022-10-10 16:11:50] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:06 [I 2022-10-10 16:11:50] ...trap/engines/engine.py.326: process: 0.45167 | load: 0.00019 [I 2022-10-10 16:11:55] ...trap/engines/engine.py.321: eval: epoch 77 | batch 80/510 [I 2022-10-10 16:11:55] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:04 [I 2022-10-10 16:11:55] ...trap/engines/engine.py.326: process: 0.42338 | load: 0.00018 [I 2022-10-10 16:11:59] ...trap/engines/engine.py.321: eval: epoch 77 | batch 90/510 [I 2022-10-10 16:11:59] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:12 [I 2022-10-10 16:11:59] ...trap/engines/engine.py.326: process: 0.43582 | load: 0.00021 [I 2022-10-10 16:12:04] ...trap/engines/engine.py.321: eval: epoch 77 | batch 100/510 [I 2022-10-10 16:12:04] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:02:52 [I 2022-10-10 16:12:04] ...trap/engines/engine.py.326: process: 0.30784 | load: 0.00019 [I 2022-10-10 16:12:07] ...trap/engines/engine.py.321: eval: epoch 77 | batch 110/510 [I 2022-10-10 16:12:07] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:11 [I 2022-10-10 16:12:07] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00020 [I 2022-10-10 16:12:10] ...trap/engines/engine.py.321: eval: epoch 77 | batch 120/510 [I 2022-10-10 16:12:10] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:04 [I 2022-10-10 16:12:10] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00020 [I 2022-10-10 16:12:13] ...trap/engines/engine.py.321: eval: epoch 77 | batch 130/510 [I 2022-10-10 16:12:13] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:00 [I 2022-10-10 16:12:13] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00020 [I 2022-10-10 16:12:16] ...trap/engines/engine.py.321: eval: epoch 77 | batch 140/510 [I 2022-10-10 16:12:16] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:57 [I 2022-10-10 16:12:16] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00021 [I 2022-10-10 16:12:19] ...trap/engines/engine.py.321: eval: epoch 77 | batch 150/510 [I 2022-10-10 16:12:19] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:54 [I 2022-10-10 16:12:19] ...trap/engines/engine.py.326: process: 0.31862 | load: 0.00020 [I 2022-10-10 16:12:23] ...trap/engines/engine.py.321: eval: epoch 77 | batch 160/510 [I 2022-10-10 16:12:23] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:51 [I 2022-10-10 16:12:23] ...trap/engines/engine.py.326: process: 0.30909 | load: 0.00020 [I 2022-10-10 16:12:26] ...trap/engines/engine.py.321: eval: epoch 77 | batch 170/510 [I 2022-10-10 16:12:26] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:48 [I 2022-10-10 16:12:26] ...trap/engines/engine.py.326: process: 0.32206 | load: 0.00020 [I 2022-10-10 16:12:29] ...trap/engines/engine.py.321: eval: epoch 77 | batch 180/510 [I 2022-10-10 16:12:29] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:45 [I 2022-10-10 16:12:29] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00019 [I 2022-10-10 16:12:32] ...trap/engines/engine.py.321: eval: epoch 77 | batch 190/510 [I 2022-10-10 16:12:32] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:41 [I 2022-10-10 16:12:32] ...trap/engines/engine.py.326: process: 0.31996 | load: 0.00019 [I 2022-10-10 16:12:35] ...trap/engines/engine.py.321: eval: epoch 77 | batch 200/510 [I 2022-10-10 16:12:35] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:38 [I 2022-10-10 16:12:35] ...trap/engines/engine.py.326: process: 0.31943 | load: 0.00019 [I 2022-10-10 16:12:39] ...trap/engines/engine.py.321: eval: epoch 77 | batch 210/510 [I 2022-10-10 16:12:39] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:36 [I 2022-10-10 16:12:39] ...trap/engines/engine.py.326: process: 0.32533 | load: 0.00020 [I 2022-10-10 16:12:42] ...trap/engines/engine.py.321: eval: epoch 77 | batch 220/510 [I 2022-10-10 16:12:42] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:32 [I 2022-10-10 16:12:42] ...trap/engines/engine.py.326: process: 0.31323 | load: 0.00020 [I 2022-10-10 16:12:45] ...trap/engines/engine.py.321: eval: epoch 77 | batch 230/510 [I 2022-10-10 16:12:45] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:29 [I 2022-10-10 16:12:45] ...trap/engines/engine.py.326: process: 0.32215 | load: 0.00022 [I 2022-10-10 16:12:48] ...trap/engines/engine.py.321: eval: epoch 77 | batch 240/510 [I 2022-10-10 16:12:48] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:25 [I 2022-10-10 16:12:48] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00019 [I 2022-10-10 16:12:51] ...trap/engines/engine.py.321: eval: epoch 77 | batch 250/510 [I 2022-10-10 16:12:51] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:22 [I 2022-10-10 16:12:51] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00020 [I 2022-10-10 16:12:55] ...trap/engines/engine.py.321: eval: epoch 77 | batch 260/510 [I 2022-10-10 16:12:55] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:19 [I 2022-10-10 16:12:55] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00020 [I 2022-10-10 16:12:58] ...trap/engines/engine.py.321: eval: epoch 77 | batch 270/510 [I 2022-10-10 16:12:58] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:16 [I 2022-10-10 16:12:58] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00019 [I 2022-10-10 16:13:01] ...trap/engines/engine.py.321: eval: epoch 77 | batch 280/510 [I 2022-10-10 16:13:01] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:13 [I 2022-10-10 16:13:01] ...trap/engines/engine.py.326: process: 0.31479 | load: 0.00020 [I 2022-10-10 16:13:04] ...trap/engines/engine.py.321: eval: epoch 77 | batch 290/510 [I 2022-10-10 16:13:04] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:10 [I 2022-10-10 16:13:04] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00020 [I 2022-10-10 16:13:07] ...trap/engines/engine.py.321: eval: epoch 77 | batch 300/510 [I 2022-10-10 16:13:07] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:06 [I 2022-10-10 16:13:07] ...trap/engines/engine.py.326: process: 0.31629 | load: 0.00020 [I 2022-10-10 16:13:11] ...trap/engines/engine.py.321: eval: epoch 77 | batch 310/510 [I 2022-10-10 16:13:11] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:03 [I 2022-10-10 16:13:11] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00020 [I 2022-10-10 16:13:14] ...trap/engines/engine.py.321: eval: epoch 77 | batch 320/510 [I 2022-10-10 16:13:14] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:00 [I 2022-10-10 16:13:14] ...trap/engines/engine.py.326: process: 0.32078 | load: 0.00020 [I 2022-10-10 16:13:17] ...trap/engines/engine.py.321: eval: epoch 77 | batch 330/510 [I 2022-10-10 16:13:17] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:57 [I 2022-10-10 16:13:17] ...trap/engines/engine.py.326: process: 0.32483 | load: 0.00021 [I 2022-10-10 16:13:20] ...trap/engines/engine.py.321: eval: epoch 77 | batch 340/510 [I 2022-10-10 16:13:20] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:54 [I 2022-10-10 16:13:20] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00019 [I 2022-10-10 16:13:23] ...trap/engines/engine.py.321: eval: epoch 77 | batch 350/510 [I 2022-10-10 16:13:23] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:50 [I 2022-10-10 16:13:23] ...trap/engines/engine.py.326: process: 0.31643 | load: 0.00019 [I 2022-10-10 16:13:27] ...trap/engines/engine.py.321: eval: epoch 77 | batch 360/510 [I 2022-10-10 16:13:27] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:47 [I 2022-10-10 16:13:27] ...trap/engines/engine.py.326: process: 0.31830 | load: 0.00019 [I 2022-10-10 16:13:30] ...trap/engines/engine.py.321: eval: epoch 77 | batch 370/510 [I 2022-10-10 16:13:30] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:44 [I 2022-10-10 16:13:30] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00021 [I 2022-10-10 16:13:33] ...trap/engines/engine.py.321: eval: epoch 77 | batch 380/510 [I 2022-10-10 16:13:33] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:41 [I 2022-10-10 16:13:33] ...trap/engines/engine.py.326: process: 0.31706 | load: 0.00021 [I 2022-10-10 16:13:36] ...trap/engines/engine.py.321: eval: epoch 77 | batch 390/510 [I 2022-10-10 16:13:36] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:38 [I 2022-10-10 16:13:36] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00019 [I 2022-10-10 16:13:39] ...trap/engines/engine.py.321: eval: epoch 77 | batch 400/510 [I 2022-10-10 16:13:39] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:35 [I 2022-10-10 16:13:39] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00019 [I 2022-10-10 16:13:42] ...trap/engines/engine.py.321: eval: epoch 77 | batch 410/510 [I 2022-10-10 16:13:42] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:31 [I 2022-10-10 16:13:42] ...trap/engines/engine.py.326: process: 0.31591 | load: 0.00019 [I 2022-10-10 16:13:46] ...trap/engines/engine.py.321: eval: epoch 77 | batch 420/510 [I 2022-10-10 16:13:46] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:28 [I 2022-10-10 16:13:46] ...trap/engines/engine.py.326: process: 0.31841 | load: 0.00019 [I 2022-10-10 16:13:49] ...trap/engines/engine.py.321: eval: epoch 77 | batch 430/510 [I 2022-10-10 16:13:49] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:25 [I 2022-10-10 16:13:49] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00019 [I 2022-10-10 16:13:52] ...trap/engines/engine.py.321: eval: epoch 77 | batch 440/510 [I 2022-10-10 16:13:52] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:22 [I 2022-10-10 16:13:52] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00019 [I 2022-10-10 16:13:55] ...trap/engines/engine.py.321: eval: epoch 77 | batch 450/510 [I 2022-10-10 16:13:55] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:18 [I 2022-10-10 16:13:55] ...trap/engines/engine.py.326: process: 0.31345 | load: 0.00019 [I 2022-10-10 16:13:58] ...trap/engines/engine.py.321: eval: epoch 77 | batch 460/510 [I 2022-10-10 16:13:58] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:15 [I 2022-10-10 16:13:58] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00020 [I 2022-10-10 16:14:02] ...trap/engines/engine.py.321: eval: epoch 77 | batch 470/510 [I 2022-10-10 16:14:02] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:12 [I 2022-10-10 16:14:02] ...trap/engines/engine.py.326: process: 0.32422 | load: 0.00021 [I 2022-10-10 16:14:05] ...trap/engines/engine.py.321: eval: epoch 77 | batch 480/510 [I 2022-10-10 16:14:05] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:09 [I 2022-10-10 16:14:05] ...trap/engines/engine.py.326: process: 0.33112 | load: 0.00025 [I 2022-10-10 16:14:08] ...trap/engines/engine.py.321: eval: epoch 77 | batch 490/510 [I 2022-10-10 16:14:08] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:06 [I 2022-10-10 16:14:08] ...trap/engines/engine.py.326: process: 0.32805 | load: 0.00025 [I 2022-10-10 16:14:11] ...trap/engines/engine.py.321: eval: epoch 77 | batch 500/510 [I 2022-10-10 16:14:11] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:03 [I 2022-10-10 16:14:11] ...trap/engines/engine.py.326: process: 0.32479 | load: 0.00021 [I 2022-10-10 16:14:15] ...trap/engines/engine.py.321: eval: epoch 77 | batch 510/510 [I 2022-10-10 16:14:15] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:00 [I 2022-10-10 16:14:15] ...trap/engines/engine.py.326: process: 0.31903 | load: 0.00017 [S 2022-10-10 16:14:15] ...trap/engines/engine.py.132: eval_epoch.epoch: 77 [I 2022-10-10 16:14:15] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737600 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913900 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941400 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.734300 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.912000 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938200 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011655 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008336 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007310 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010789 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008331 [S 2022-10-10 16:14:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007264 [I 2022-10-10 16:15:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 16:15:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 16:15:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 16:15:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 16:15:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 16:15:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 16:15:09] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 16:15:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 16:15:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 78 [I 2022-10-10 16:15:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 16:15:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 0/2304 [I 2022-10-10 16:15:17] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:14:10 [I 2022-10-10 16:15:17] ...trap/engines/engine.py.229: process: 3.49435 | load: 2.57316 [I 2022-10-10 16:15:17] ...trap/engines/engine.py.230: loss: 0.13873 [I 2022-10-10 16:15:17] ...optimizers/trijoint.py.151: total_norm: 15.801500 [I 2022-10-10 16:15:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 10/2304 [I 2022-10-10 16:15:26] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:38 [I 2022-10-10 16:15:26] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00513 [I 2022-10-10 16:15:26] ...trap/engines/engine.py.230: loss: 0.15966 [I 2022-10-10 16:15:26] ...optimizers/trijoint.py.151: total_norm: 40.721500 [I 2022-10-10 16:15:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 20/2304 [I 2022-10-10 16:15:34] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:51 [I 2022-10-10 16:15:34] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00501 [I 2022-10-10 16:15:34] ...trap/engines/engine.py.230: loss: 0.09654 [I 2022-10-10 16:15:34] ...optimizers/trijoint.py.151: total_norm: 7.495986 [I 2022-10-10 16:15:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 30/2304 [I 2022-10-10 16:15:43] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:42 [I 2022-10-10 16:15:43] ...trap/engines/engine.py.229: process: 0.86403 | load: 0.00485 [I 2022-10-10 16:15:43] ...trap/engines/engine.py.230: loss: 0.13625 [I 2022-10-10 16:15:43] ...optimizers/trijoint.py.151: total_norm: 29.330576 [I 2022-10-10 16:15:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 40/2304 [I 2022-10-10 16:15:51] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:02 [I 2022-10-10 16:15:51] ...trap/engines/engine.py.229: process: 0.84786 | load: 0.00403 [I 2022-10-10 16:15:51] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-10 16:15:51] ...optimizers/trijoint.py.151: total_norm: 20.691137 [I 2022-10-10 16:16:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 50/2304 [I 2022-10-10 16:16:00] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:31:55 [I 2022-10-10 16:16:00] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00349 [I 2022-10-10 16:16:00] ...trap/engines/engine.py.230: loss: 0.16883 [I 2022-10-10 16:16:00] ...optimizers/trijoint.py.151: total_norm: 19.965919 [I 2022-10-10 16:16:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 60/2304 [I 2022-10-10 16:16:08] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:47 [I 2022-10-10 16:16:08] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00403 [I 2022-10-10 16:16:08] ...trap/engines/engine.py.230: loss: 0.09005 [I 2022-10-10 16:16:08] ...optimizers/trijoint.py.151: total_norm: 8.406512 [I 2022-10-10 16:16:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 70/2304 [I 2022-10-10 16:16:17] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:51 [I 2022-10-10 16:16:17] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00366 [I 2022-10-10 16:16:17] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-10 16:16:17] ...optimizers/trijoint.py.151: total_norm: 15.947043 [I 2022-10-10 16:16:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 80/2304 [I 2022-10-10 16:16:26] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:41 [I 2022-10-10 16:16:26] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00360 [I 2022-10-10 16:16:26] ...trap/engines/engine.py.230: loss: 0.12663 [I 2022-10-10 16:16:26] ...optimizers/trijoint.py.151: total_norm: 19.211222 [I 2022-10-10 16:16:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 90/2304 [I 2022-10-10 16:16:34] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:43 [I 2022-10-10 16:16:34] ...trap/engines/engine.py.229: process: 0.87566 | load: 0.00367 [I 2022-10-10 16:16:34] ...trap/engines/engine.py.230: loss: 0.18598 [I 2022-10-10 16:16:34] ...optimizers/trijoint.py.151: total_norm: 20.566402 [I 2022-10-10 16:16:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 100/2304 [I 2022-10-10 16:16:43] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:12 [I 2022-10-10 16:16:43] ...trap/engines/engine.py.229: process: 0.84140 | load: 0.00347 [I 2022-10-10 16:16:43] ...trap/engines/engine.py.230: loss: 0.15670 [I 2022-10-10 16:16:43] ...optimizers/trijoint.py.151: total_norm: 19.997374 [I 2022-10-10 16:16:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 110/2304 [I 2022-10-10 16:16:51] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:29 [I 2022-10-10 16:16:51] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00369 [I 2022-10-10 16:16:51] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-10 16:16:51] ...optimizers/trijoint.py.151: total_norm: 15.066740 [I 2022-10-10 16:17:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 120/2304 [I 2022-10-10 16:17:00] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:08 [I 2022-10-10 16:17:00] ...trap/engines/engine.py.229: process: 0.84919 | load: 0.00348 [I 2022-10-10 16:17:00] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-10 16:17:00] ...optimizers/trijoint.py.151: total_norm: 22.979067 [I 2022-10-10 16:17:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 130/2304 [I 2022-10-10 16:17:08] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:59 [I 2022-10-10 16:17:08] ...trap/engines/engine.py.229: process: 0.83952 | load: 0.00325 [I 2022-10-10 16:17:08] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-10 16:17:08] ...optimizers/trijoint.py.151: total_norm: 23.245699 [I 2022-10-10 16:17:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 140/2304 [I 2022-10-10 16:17:17] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:52 [I 2022-10-10 16:17:17] ...trap/engines/engine.py.229: process: 0.85978 | load: 0.00353 [I 2022-10-10 16:17:17] ...trap/engines/engine.py.230: loss: 0.14149 [I 2022-10-10 16:17:17] ...optimizers/trijoint.py.151: total_norm: 10.831410 [I 2022-10-10 16:17:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 150/2304 [I 2022-10-10 16:17:26] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:39 [I 2022-10-10 16:17:26] ...trap/engines/engine.py.229: process: 0.86442 | load: 0.00351 [I 2022-10-10 16:17:26] ...trap/engines/engine.py.230: loss: 0.18473 [I 2022-10-10 16:17:26] ...optimizers/trijoint.py.151: total_norm: 15.976430 [I 2022-10-10 16:17:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 160/2304 [I 2022-10-10 16:17:34] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:36 [I 2022-10-10 16:17:34] ...trap/engines/engine.py.229: process: 0.86665 | load: 0.00329 [I 2022-10-10 16:17:34] ...trap/engines/engine.py.230: loss: 0.17233 [I 2022-10-10 16:17:34] ...optimizers/trijoint.py.151: total_norm: 19.448700 [I 2022-10-10 16:17:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 170/2304 [I 2022-10-10 16:17:43] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:28 [I 2022-10-10 16:17:43] ...trap/engines/engine.py.229: process: 0.85667 | load: 0.00321 [I 2022-10-10 16:17:43] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-10 16:17:43] ...optimizers/trijoint.py.151: total_norm: 10.447995 [I 2022-10-10 16:17:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 180/2304 [I 2022-10-10 16:17:51] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:20 [I 2022-10-10 16:17:51] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00331 [I 2022-10-10 16:17:51] ...trap/engines/engine.py.230: loss: 0.10681 [I 2022-10-10 16:17:51] ...optimizers/trijoint.py.151: total_norm: 10.249394 [I 2022-10-10 16:18:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 190/2304 [I 2022-10-10 16:18:00] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:09 [I 2022-10-10 16:18:00] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00331 [I 2022-10-10 16:18:00] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-10 16:18:00] ...optimizers/trijoint.py.151: total_norm: 17.310972 [I 2022-10-10 16:18:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 200/2304 [I 2022-10-10 16:18:09] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:57 [I 2022-10-10 16:18:09] ...trap/engines/engine.py.229: process: 0.86023 | load: 0.00295 [I 2022-10-10 16:18:09] ...trap/engines/engine.py.230: loss: 0.13837 [I 2022-10-10 16:18:09] ...optimizers/trijoint.py.151: total_norm: 14.753241 [I 2022-10-10 16:18:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 210/2304 [I 2022-10-10 16:18:17] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:43 [I 2022-10-10 16:18:17] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00319 [I 2022-10-10 16:18:17] ...trap/engines/engine.py.230: loss: 0.13809 [I 2022-10-10 16:18:17] ...optimizers/trijoint.py.151: total_norm: 16.161215 [I 2022-10-10 16:18:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 220/2304 [I 2022-10-10 16:18:25] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:15 [I 2022-10-10 16:18:25] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00317 [I 2022-10-10 16:18:25] ...trap/engines/engine.py.230: loss: 0.12065 [I 2022-10-10 16:18:25] ...optimizers/trijoint.py.151: total_norm: 17.478500 [I 2022-10-10 16:18:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 230/2304 [I 2022-10-10 16:18:34] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:14 [I 2022-10-10 16:18:34] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00319 [I 2022-10-10 16:18:34] ...trap/engines/engine.py.230: loss: 0.17365 [I 2022-10-10 16:18:34] ...optimizers/trijoint.py.151: total_norm: 13.885905 [I 2022-10-10 16:18:42] ...trap/engines/engine.py.224: train: epoch 78 | batch 240/2304 [I 2022-10-10 16:18:42] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:05 [I 2022-10-10 16:18:42] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00291 [I 2022-10-10 16:18:42] ...trap/engines/engine.py.230: loss: 0.25857 [I 2022-10-10 16:18:42] ...optimizers/trijoint.py.151: total_norm: 14.130980 [I 2022-10-10 16:18:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 250/2304 [I 2022-10-10 16:18:51] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:28:54 [I 2022-10-10 16:18:51] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00290 [I 2022-10-10 16:18:51] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-10 16:18:51] ...optimizers/trijoint.py.151: total_norm: 18.585867 [I 2022-10-10 16:18:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 260/2304 [I 2022-10-10 16:18:59] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:34 [I 2022-10-10 16:18:59] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00292 [I 2022-10-10 16:18:59] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-10 16:18:59] ...optimizers/trijoint.py.151: total_norm: 25.890404 [I 2022-10-10 16:19:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 270/2304 [I 2022-10-10 16:19:08] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:38 [I 2022-10-10 16:19:08] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00292 [I 2022-10-10 16:19:08] ...trap/engines/engine.py.230: loss: 0.12616 [I 2022-10-10 16:19:08] ...optimizers/trijoint.py.151: total_norm: 13.741990 [I 2022-10-10 16:19:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 280/2304 [I 2022-10-10 16:19:16] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:24 [I 2022-10-10 16:19:16] ...trap/engines/engine.py.229: process: 0.84133 | load: 0.00296 [I 2022-10-10 16:19:16] ...trap/engines/engine.py.230: loss: 0.16006 [I 2022-10-10 16:19:16] ...optimizers/trijoint.py.151: total_norm: 14.076575 [I 2022-10-10 16:19:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 290/2304 [I 2022-10-10 16:19:25] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:27 [I 2022-10-10 16:19:25] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00312 [I 2022-10-10 16:19:25] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-10 16:19:25] ...optimizers/trijoint.py.151: total_norm: 12.674196 [I 2022-10-10 16:19:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 300/2304 [I 2022-10-10 16:19:33] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:12 [I 2022-10-10 16:19:33] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00297 [I 2022-10-10 16:19:33] ...trap/engines/engine.py.230: loss: 0.10368 [I 2022-10-10 16:19:33] ...optimizers/trijoint.py.151: total_norm: 14.230397 [I 2022-10-10 16:19:42] ...trap/engines/engine.py.224: train: epoch 78 | batch 310/2304 [I 2022-10-10 16:19:42] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:02 [I 2022-10-10 16:19:42] ...trap/engines/engine.py.229: process: 0.84849 | load: 0.00296 [I 2022-10-10 16:19:42] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-10 16:19:42] ...optimizers/trijoint.py.151: total_norm: 11.844326 [I 2022-10-10 16:19:50] ...trap/engines/engine.py.224: train: epoch 78 | batch 320/2304 [I 2022-10-10 16:19:50] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:06 [I 2022-10-10 16:19:50] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00289 [I 2022-10-10 16:19:50] ...trap/engines/engine.py.230: loss: 0.14049 [I 2022-10-10 16:19:50] ...optimizers/trijoint.py.151: total_norm: 18.073978 [I 2022-10-10 16:19:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 330/2304 [I 2022-10-10 16:19:59] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:27:48 [I 2022-10-10 16:19:59] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00303 [I 2022-10-10 16:19:59] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-10 16:19:59] ...optimizers/trijoint.py.151: total_norm: 11.154960 [I 2022-10-10 16:20:07] ...trap/engines/engine.py.224: train: epoch 78 | batch 340/2304 [I 2022-10-10 16:20:07] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:36 [I 2022-10-10 16:20:07] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00306 [I 2022-10-10 16:20:07] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-10 16:20:07] ...optimizers/trijoint.py.151: total_norm: 43.976093 [I 2022-10-10 16:20:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 350/2304 [I 2022-10-10 16:20:15] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:27 [I 2022-10-10 16:20:15] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00298 [I 2022-10-10 16:20:15] ...trap/engines/engine.py.230: loss: 0.23180 [I 2022-10-10 16:20:15] ...optimizers/trijoint.py.151: total_norm: 16.862143 [I 2022-10-10 16:20:24] ...trap/engines/engine.py.224: train: epoch 78 | batch 360/2304 [I 2022-10-10 16:20:24] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:27 [I 2022-10-10 16:20:24] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00292 [I 2022-10-10 16:20:24] ...trap/engines/engine.py.230: loss: 0.31369 [I 2022-10-10 16:20:24] ...optimizers/trijoint.py.151: total_norm: 19.202536 [I 2022-10-10 16:20:32] ...trap/engines/engine.py.224: train: epoch 78 | batch 370/2304 [I 2022-10-10 16:20:32] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:17 [I 2022-10-10 16:20:32] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00292 [I 2022-10-10 16:20:32] ...trap/engines/engine.py.230: loss: 0.09767 [I 2022-10-10 16:20:32] ...optimizers/trijoint.py.151: total_norm: 17.675783 [I 2022-10-10 16:20:41] ...trap/engines/engine.py.224: train: epoch 78 | batch 380/2304 [I 2022-10-10 16:20:41] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:03 [I 2022-10-10 16:20:41] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00298 [I 2022-10-10 16:20:41] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-10 16:20:41] ...optimizers/trijoint.py.151: total_norm: 14.799192 [I 2022-10-10 16:20:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 390/2304 [I 2022-10-10 16:20:49] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:26:55 [I 2022-10-10 16:20:49] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00293 [I 2022-10-10 16:20:49] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-10 16:20:49] ...optimizers/trijoint.py.151: total_norm: 23.245106 [I 2022-10-10 16:20:58] ...trap/engines/engine.py.224: train: epoch 78 | batch 400/2304 [I 2022-10-10 16:20:58] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:49 [I 2022-10-10 16:20:58] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00292 [I 2022-10-10 16:20:58] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-10 16:20:58] ...optimizers/trijoint.py.151: total_norm: 13.592641 [I 2022-10-10 16:21:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 410/2304 [I 2022-10-10 16:21:06] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:36 [I 2022-10-10 16:21:06] ...trap/engines/engine.py.229: process: 0.84325 | load: 0.00310 [I 2022-10-10 16:21:06] ...trap/engines/engine.py.230: loss: 0.12039 [I 2022-10-10 16:21:06] ...optimizers/trijoint.py.151: total_norm: 9.917987 [I 2022-10-10 16:21:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 420/2304 [I 2022-10-10 16:21:15] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:34 [I 2022-10-10 16:21:15] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00297 [I 2022-10-10 16:21:15] ...trap/engines/engine.py.230: loss: 0.12799 [I 2022-10-10 16:21:15] ...optimizers/trijoint.py.151: total_norm: 28.596582 [I 2022-10-10 16:21:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 430/2304 [I 2022-10-10 16:21:23] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:27 [I 2022-10-10 16:21:23] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00295 [I 2022-10-10 16:21:23] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-10 16:21:23] ...optimizers/trijoint.py.151: total_norm: 12.841199 [I 2022-10-10 16:21:32] ...trap/engines/engine.py.224: train: epoch 78 | batch 440/2304 [I 2022-10-10 16:21:32] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:14 [I 2022-10-10 16:21:32] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00295 [I 2022-10-10 16:21:32] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-10 16:21:32] ...optimizers/trijoint.py.151: total_norm: 9.039435 [I 2022-10-10 16:21:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 450/2304 [I 2022-10-10 16:21:40] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:06 [I 2022-10-10 16:21:40] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00294 [I 2022-10-10 16:21:40] ...trap/engines/engine.py.230: loss: 0.12847 [I 2022-10-10 16:21:40] ...optimizers/trijoint.py.151: total_norm: 17.488142 [I 2022-10-10 16:21:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 460/2304 [I 2022-10-10 16:21:49] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:55 [I 2022-10-10 16:21:49] ...trap/engines/engine.py.229: process: 0.84289 | load: 0.00297 [I 2022-10-10 16:21:49] ...trap/engines/engine.py.230: loss: 0.23480 [I 2022-10-10 16:21:49] ...optimizers/trijoint.py.151: total_norm: 15.731020 [I 2022-10-10 16:21:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 470/2304 [I 2022-10-10 16:21:57] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:25:45 [I 2022-10-10 16:21:57] ...trap/engines/engine.py.229: process: 0.83795 | load: 0.00307 [I 2022-10-10 16:21:57] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-10 16:21:57] ...optimizers/trijoint.py.151: total_norm: 9.549201 [I 2022-10-10 16:22:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 480/2304 [I 2022-10-10 16:22:06] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:32 [I 2022-10-10 16:22:06] ...trap/engines/engine.py.229: process: 0.84206 | load: 0.00305 [I 2022-10-10 16:22:06] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-10 16:22:06] ...optimizers/trijoint.py.151: total_norm: 24.709597 [I 2022-10-10 16:22:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 490/2304 [I 2022-10-10 16:22:14] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:25:38 [I 2022-10-10 16:22:14] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00305 [I 2022-10-10 16:22:14] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-10 16:22:14] ...optimizers/trijoint.py.151: total_norm: 9.182136 [I 2022-10-10 16:22:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 500/2304 [I 2022-10-10 16:22:23] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:33 [I 2022-10-10 16:22:23] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00305 [I 2022-10-10 16:22:23] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-10 16:22:23] ...optimizers/trijoint.py.151: total_norm: 17.006798 [I 2022-10-10 16:22:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 510/2304 [I 2022-10-10 16:22:31] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:20 [I 2022-10-10 16:22:31] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00297 [I 2022-10-10 16:22:31] ...trap/engines/engine.py.230: loss: 0.09592 [I 2022-10-10 16:22:31] ...optimizers/trijoint.py.151: total_norm: 9.977572 [I 2022-10-10 16:22:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 520/2304 [I 2022-10-10 16:22:40] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:26 [I 2022-10-10 16:22:40] ...trap/engines/engine.py.229: process: 0.86332 | load: 0.00332 [I 2022-10-10 16:22:40] ...trap/engines/engine.py.230: loss: 0.16116 [I 2022-10-10 16:22:40] ...optimizers/trijoint.py.151: total_norm: 17.731247 [I 2022-10-10 16:22:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 530/2304 [I 2022-10-10 16:22:48] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:16 [I 2022-10-10 16:22:48] ...trap/engines/engine.py.229: process: 0.85080 | load: 0.00308 [I 2022-10-10 16:22:48] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-10 16:22:48] ...optimizers/trijoint.py.151: total_norm: 13.322405 [I 2022-10-10 16:22:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 540/2304 [I 2022-10-10 16:22:57] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:12 [I 2022-10-10 16:22:57] ...trap/engines/engine.py.229: process: 0.85926 | load: 0.00310 [I 2022-10-10 16:22:57] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-10 16:22:57] ...optimizers/trijoint.py.151: total_norm: 12.770791 [I 2022-10-10 16:23:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 550/2304 [I 2022-10-10 16:23:06] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:08 [I 2022-10-10 16:23:06] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00317 [I 2022-10-10 16:23:06] ...trap/engines/engine.py.230: loss: 0.11488 [I 2022-10-10 16:23:06] ...optimizers/trijoint.py.151: total_norm: 18.276838 [I 2022-10-10 16:23:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 560/2304 [I 2022-10-10 16:23:14] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:48 [I 2022-10-10 16:23:14] ...trap/engines/engine.py.229: process: 0.85212 | load: 0.00311 [I 2022-10-10 16:23:14] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-10 16:23:14] ...optimizers/trijoint.py.151: total_norm: 17.186319 [I 2022-10-10 16:23:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 570/2304 [I 2022-10-10 16:23:23] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:40 [I 2022-10-10 16:23:23] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00317 [I 2022-10-10 16:23:23] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-10 16:23:23] ...optimizers/trijoint.py.151: total_norm: 15.210235 [I 2022-10-10 16:23:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 580/2304 [I 2022-10-10 16:23:31] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:28 [I 2022-10-10 16:23:31] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00313 [I 2022-10-10 16:23:31] ...trap/engines/engine.py.230: loss: 0.15511 [I 2022-10-10 16:23:31] ...optimizers/trijoint.py.151: total_norm: 20.559181 [I 2022-10-10 16:23:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 590/2304 [I 2022-10-10 16:23:40] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:25 [I 2022-10-10 16:23:40] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00315 [I 2022-10-10 16:23:40] ...trap/engines/engine.py.230: loss: 0.16273 [I 2022-10-10 16:23:40] ...optimizers/trijoint.py.151: total_norm: 9.070017 [I 2022-10-10 16:23:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 600/2304 [I 2022-10-10 16:23:48] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:13 [I 2022-10-10 16:23:48] ...trap/engines/engine.py.229: process: 0.84849 | load: 0.00315 [I 2022-10-10 16:23:48] ...trap/engines/engine.py.230: loss: 0.17827 [I 2022-10-10 16:23:48] ...optimizers/trijoint.py.151: total_norm: 23.781408 [I 2022-10-10 16:23:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 610/2304 [I 2022-10-10 16:23:57] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:21 [I 2022-10-10 16:23:57] ...trap/engines/engine.py.229: process: 0.87258 | load: 0.00310 [I 2022-10-10 16:23:57] ...trap/engines/engine.py.230: loss: 0.13260 [I 2022-10-10 16:23:57] ...optimizers/trijoint.py.151: total_norm: 13.510382 [I 2022-10-10 16:24:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 620/2304 [I 2022-10-10 16:24:06] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:08 [I 2022-10-10 16:24:06] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00314 [I 2022-10-10 16:24:06] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-10 16:24:06] ...optimizers/trijoint.py.151: total_norm: 12.008364 [I 2022-10-10 16:24:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 630/2304 [I 2022-10-10 16:24:14] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:55 [I 2022-10-10 16:24:14] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00310 [I 2022-10-10 16:24:14] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-10 16:24:14] ...optimizers/trijoint.py.151: total_norm: 17.912319 [I 2022-10-10 16:24:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 640/2304 [I 2022-10-10 16:24:23] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:44 [I 2022-10-10 16:24:23] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00311 [I 2022-10-10 16:24:23] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-10 16:24:23] ...optimizers/trijoint.py.151: total_norm: 16.706648 [I 2022-10-10 16:24:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 650/2304 [I 2022-10-10 16:24:31] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:32 [I 2022-10-10 16:24:31] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00311 [I 2022-10-10 16:24:31] ...trap/engines/engine.py.230: loss: 0.13181 [I 2022-10-10 16:24:31] ...optimizers/trijoint.py.151: total_norm: 25.153585 [I 2022-10-10 16:24:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 660/2304 [I 2022-10-10 16:24:40] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:30 [I 2022-10-10 16:24:40] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00317 [I 2022-10-10 16:24:40] ...trap/engines/engine.py.230: loss: 0.12064 [I 2022-10-10 16:24:40] ...optimizers/trijoint.py.151: total_norm: 14.431438 [I 2022-10-10 16:24:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 670/2304 [I 2022-10-10 16:24:48] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:15 [I 2022-10-10 16:24:48] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00309 [I 2022-10-10 16:24:48] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-10 16:24:48] ...optimizers/trijoint.py.151: total_norm: 15.149762 [I 2022-10-10 16:24:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 680/2304 [I 2022-10-10 16:24:57] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:04 [I 2022-10-10 16:24:57] ...trap/engines/engine.py.229: process: 0.84967 | load: 0.00317 [I 2022-10-10 16:24:57] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-10 16:24:57] ...optimizers/trijoint.py.151: total_norm: 35.464451 [I 2022-10-10 16:25:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 690/2304 [I 2022-10-10 16:25:06] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:23:02 [I 2022-10-10 16:25:06] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00306 [I 2022-10-10 16:25:06] ...trap/engines/engine.py.230: loss: 0.11035 [I 2022-10-10 16:25:06] ...optimizers/trijoint.py.151: total_norm: 8.064565 [I 2022-10-10 16:25:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 700/2304 [I 2022-10-10 16:25:14] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:22:49 [I 2022-10-10 16:25:14] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00315 [I 2022-10-10 16:25:14] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-10 16:25:14] ...optimizers/trijoint.py.151: total_norm: 16.674316 [I 2022-10-10 16:25:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 710/2304 [I 2022-10-10 16:25:23] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:41 [I 2022-10-10 16:25:23] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00308 [I 2022-10-10 16:25:23] ...trap/engines/engine.py.230: loss: 0.15064 [I 2022-10-10 16:25:23] ...optimizers/trijoint.py.151: total_norm: 26.031029 [I 2022-10-10 16:25:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 720/2304 [I 2022-10-10 16:25:31] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:39 [I 2022-10-10 16:25:31] ...trap/engines/engine.py.229: process: 0.85453 | load: 0.00341 [I 2022-10-10 16:25:31] ...trap/engines/engine.py.230: loss: 0.21342 [I 2022-10-10 16:25:31] ...optimizers/trijoint.py.151: total_norm: 23.760391 [I 2022-10-10 16:25:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 730/2304 [I 2022-10-10 16:25:40] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:36 [I 2022-10-10 16:25:40] ...trap/engines/engine.py.229: process: 0.86716 | load: 0.00309 [I 2022-10-10 16:25:40] ...trap/engines/engine.py.230: loss: 0.19434 [I 2022-10-10 16:25:40] ...optimizers/trijoint.py.151: total_norm: 34.082272 [I 2022-10-10 16:25:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 740/2304 [I 2022-10-10 16:25:49] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:31 [I 2022-10-10 16:25:49] ...trap/engines/engine.py.229: process: 0.85464 | load: 0.00318 [I 2022-10-10 16:25:49] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-10 16:25:49] ...optimizers/trijoint.py.151: total_norm: 11.108635 [I 2022-10-10 16:25:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 750/2304 [I 2022-10-10 16:25:57] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:11 [I 2022-10-10 16:25:57] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00307 [I 2022-10-10 16:25:57] ...trap/engines/engine.py.230: loss: 0.17980 [I 2022-10-10 16:25:57] ...optimizers/trijoint.py.151: total_norm: 15.441882 [I 2022-10-10 16:26:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 760/2304 [I 2022-10-10 16:26:06] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:57 [I 2022-10-10 16:26:06] ...trap/engines/engine.py.229: process: 0.86537 | load: 0.00310 [I 2022-10-10 16:26:06] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-10 16:26:06] ...optimizers/trijoint.py.151: total_norm: 19.489014 [I 2022-10-10 16:26:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 770/2304 [I 2022-10-10 16:26:14] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:51 [I 2022-10-10 16:26:14] ...trap/engines/engine.py.229: process: 0.85898 | load: 0.00315 [I 2022-10-10 16:26:14] ...trap/engines/engine.py.230: loss: 0.09350 [I 2022-10-10 16:26:14] ...optimizers/trijoint.py.151: total_norm: 14.736376 [I 2022-10-10 16:26:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 780/2304 [I 2022-10-10 16:26:23] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:48 [I 2022-10-10 16:26:23] ...trap/engines/engine.py.229: process: 0.86362 | load: 0.00307 [I 2022-10-10 16:26:23] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-10 16:26:23] ...optimizers/trijoint.py.151: total_norm: 17.537878 [I 2022-10-10 16:26:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 790/2304 [I 2022-10-10 16:26:31] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:23 [I 2022-10-10 16:26:31] ...trap/engines/engine.py.229: process: 0.83115 | load: 0.00308 [I 2022-10-10 16:26:31] ...trap/engines/engine.py.230: loss: 0.11922 [I 2022-10-10 16:26:31] ...optimizers/trijoint.py.151: total_norm: 19.042978 [I 2022-10-10 16:26:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 800/2304 [I 2022-10-10 16:26:40] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:17 [I 2022-10-10 16:26:40] ...trap/engines/engine.py.229: process: 0.85274 | load: 0.00317 [I 2022-10-10 16:26:40] ...trap/engines/engine.py.230: loss: 0.13813 [I 2022-10-10 16:26:40] ...optimizers/trijoint.py.151: total_norm: 9.987949 [I 2022-10-10 16:26:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 810/2304 [I 2022-10-10 16:26:48] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:00 [I 2022-10-10 16:26:48] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00294 [I 2022-10-10 16:26:48] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-10 16:26:48] ...optimizers/trijoint.py.151: total_norm: 16.123951 [I 2022-10-10 16:26:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 820/2304 [I 2022-10-10 16:26:57] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:20:54 [I 2022-10-10 16:26:57] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00293 [I 2022-10-10 16:26:57] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-10 16:26:57] ...optimizers/trijoint.py.151: total_norm: 19.350214 [I 2022-10-10 16:27:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 830/2304 [I 2022-10-10 16:27:05] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:20:37 [I 2022-10-10 16:27:05] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00296 [I 2022-10-10 16:27:05] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-10 16:27:05] ...optimizers/trijoint.py.151: total_norm: 28.847977 [I 2022-10-10 16:27:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 840/2304 [I 2022-10-10 16:27:14] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:33 [I 2022-10-10 16:27:14] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00295 [I 2022-10-10 16:27:14] ...trap/engines/engine.py.230: loss: 0.10742 [I 2022-10-10 16:27:14] ...optimizers/trijoint.py.151: total_norm: 13.894692 [I 2022-10-10 16:27:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 850/2304 [I 2022-10-10 16:27:22] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:20:25 [I 2022-10-10 16:27:22] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00297 [I 2022-10-10 16:27:22] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-10 16:27:22] ...optimizers/trijoint.py.151: total_norm: 8.010038 [I 2022-10-10 16:27:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 860/2304 [I 2022-10-10 16:27:31] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:18 [I 2022-10-10 16:27:31] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00289 [I 2022-10-10 16:27:31] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-10 16:27:31] ...optimizers/trijoint.py.151: total_norm: 22.001661 [I 2022-10-10 16:27:39] ...trap/engines/engine.py.224: train: epoch 78 | batch 870/2304 [I 2022-10-10 16:27:39] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:15 [I 2022-10-10 16:27:39] ...trap/engines/engine.py.229: process: 0.85861 | load: 0.00292 [I 2022-10-10 16:27:39] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-10 16:27:39] ...optimizers/trijoint.py.151: total_norm: 16.304106 [I 2022-10-10 16:27:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 880/2304 [I 2022-10-10 16:27:48] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:05 [I 2022-10-10 16:27:48] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00337 [I 2022-10-10 16:27:48] ...trap/engines/engine.py.230: loss: 0.15904 [I 2022-10-10 16:27:48] ...optimizers/trijoint.py.151: total_norm: 21.393385 [I 2022-10-10 16:27:56] ...trap/engines/engine.py.224: train: epoch 78 | batch 890/2304 [I 2022-10-10 16:27:56] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:07 [I 2022-10-10 16:27:56] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00297 [I 2022-10-10 16:27:56] ...trap/engines/engine.py.230: loss: 0.22699 [I 2022-10-10 16:27:56] ...optimizers/trijoint.py.151: total_norm: 12.598925 [I 2022-10-10 16:28:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 900/2304 [I 2022-10-10 16:28:05] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:52 [I 2022-10-10 16:28:05] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00297 [I 2022-10-10 16:28:05] ...trap/engines/engine.py.230: loss: 0.15262 [I 2022-10-10 16:28:05] ...optimizers/trijoint.py.151: total_norm: 27.512363 [I 2022-10-10 16:28:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 910/2304 [I 2022-10-10 16:28:13] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:49 [I 2022-10-10 16:28:13] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00299 [I 2022-10-10 16:28:13] ...trap/engines/engine.py.230: loss: 0.14323 [I 2022-10-10 16:28:13] ...optimizers/trijoint.py.151: total_norm: 13.487864 [I 2022-10-10 16:28:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 920/2304 [I 2022-10-10 16:28:22] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:36 [I 2022-10-10 16:28:22] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00300 [I 2022-10-10 16:28:22] ...trap/engines/engine.py.230: loss: 0.10734 [I 2022-10-10 16:28:22] ...optimizers/trijoint.py.151: total_norm: 29.963634 [I 2022-10-10 16:28:30] ...trap/engines/engine.py.224: train: epoch 78 | batch 930/2304 [I 2022-10-10 16:28:30] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:14 [I 2022-10-10 16:28:30] ...trap/engines/engine.py.229: process: 0.83913 | load: 0.00294 [I 2022-10-10 16:28:30] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-10 16:28:30] ...optimizers/trijoint.py.151: total_norm: 18.432039 [I 2022-10-10 16:28:39] ...trap/engines/engine.py.224: train: epoch 78 | batch 940/2304 [I 2022-10-10 16:28:39] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:15 [I 2022-10-10 16:28:39] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00293 [I 2022-10-10 16:28:39] ...trap/engines/engine.py.230: loss: 0.17348 [I 2022-10-10 16:28:39] ...optimizers/trijoint.py.151: total_norm: 40.488544 [I 2022-10-10 16:28:47] ...trap/engines/engine.py.224: train: epoch 78 | batch 950/2304 [I 2022-10-10 16:28:47] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:03 [I 2022-10-10 16:28:47] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00305 [I 2022-10-10 16:28:47] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-10 16:28:47] ...optimizers/trijoint.py.151: total_norm: 21.169781 [I 2022-10-10 16:28:56] ...trap/engines/engine.py.224: train: epoch 78 | batch 960/2304 [I 2022-10-10 16:28:56] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:18:58 [I 2022-10-10 16:28:56] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00297 [I 2022-10-10 16:28:56] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-10 16:28:56] ...optimizers/trijoint.py.151: total_norm: 18.771660 [I 2022-10-10 16:29:04] ...trap/engines/engine.py.224: train: epoch 78 | batch 970/2304 [I 2022-10-10 16:29:04] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:52 [I 2022-10-10 16:29:04] ...trap/engines/engine.py.229: process: 0.84082 | load: 0.00293 [I 2022-10-10 16:29:04] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-10 16:29:04] ...optimizers/trijoint.py.151: total_norm: 14.108865 [I 2022-10-10 16:29:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 980/2304 [I 2022-10-10 16:29:13] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:40 [I 2022-10-10 16:29:13] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00289 [I 2022-10-10 16:29:13] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-10 16:29:13] ...optimizers/trijoint.py.151: total_norm: 8.527864 [I 2022-10-10 16:29:21] ...trap/engines/engine.py.224: train: epoch 78 | batch 990/2304 [I 2022-10-10 16:29:21] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:32 [I 2022-10-10 16:29:21] ...trap/engines/engine.py.229: process: 0.84753 | load: 0.00294 [I 2022-10-10 16:29:21] ...trap/engines/engine.py.230: loss: 0.12969 [I 2022-10-10 16:29:21] ...optimizers/trijoint.py.151: total_norm: 47.057217 [I 2022-10-10 16:29:30] ...trap/engines/engine.py.224: train: epoch 78 | batch 1000/2304 [I 2022-10-10 16:29:30] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:20 [I 2022-10-10 16:29:30] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00292 [I 2022-10-10 16:29:30] ...trap/engines/engine.py.230: loss: 0.11844 [I 2022-10-10 16:29:30] ...optimizers/trijoint.py.151: total_norm: 19.039083 [I 2022-10-10 16:29:38] ...trap/engines/engine.py.224: train: epoch 78 | batch 1010/2304 [I 2022-10-10 16:29:38] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:11 [I 2022-10-10 16:29:38] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00292 [I 2022-10-10 16:29:38] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-10 16:29:38] ...optimizers/trijoint.py.151: total_norm: 12.582773 [I 2022-10-10 16:29:47] ...trap/engines/engine.py.224: train: epoch 78 | batch 1020/2304 [I 2022-10-10 16:29:47] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:01 [I 2022-10-10 16:29:47] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00291 [I 2022-10-10 16:29:47] ...trap/engines/engine.py.230: loss: 0.10886 [I 2022-10-10 16:29:47] ...optimizers/trijoint.py.151: total_norm: 25.041365 [I 2022-10-10 16:29:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 1030/2304 [I 2022-10-10 16:29:55] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:17:50 [I 2022-10-10 16:29:55] ...trap/engines/engine.py.229: process: 0.84079 | load: 0.00290 [I 2022-10-10 16:29:55] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-10 16:29:55] ...optimizers/trijoint.py.151: total_norm: 10.313203 [I 2022-10-10 16:30:03] ...trap/engines/engine.py.224: train: epoch 78 | batch 1040/2304 [I 2022-10-10 16:30:03] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:43 [I 2022-10-10 16:30:03] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00291 [I 2022-10-10 16:30:03] ...trap/engines/engine.py.230: loss: 0.15494 [I 2022-10-10 16:30:03] ...optimizers/trijoint.py.151: total_norm: 8.464573 [I 2022-10-10 16:30:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 1050/2304 [I 2022-10-10 16:30:12] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:37 [I 2022-10-10 16:30:12] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00289 [I 2022-10-10 16:30:12] ...trap/engines/engine.py.230: loss: 0.16941 [I 2022-10-10 16:30:12] ...optimizers/trijoint.py.151: total_norm: 23.060255 [I 2022-10-10 16:30:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 1060/2304 [I 2022-10-10 16:30:20] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:32 [I 2022-10-10 16:30:20] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00289 [I 2022-10-10 16:30:20] ...trap/engines/engine.py.230: loss: 0.11759 [I 2022-10-10 16:30:20] ...optimizers/trijoint.py.151: total_norm: 17.483686 [I 2022-10-10 16:30:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 1070/2304 [I 2022-10-10 16:30:29] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:12 [I 2022-10-10 16:30:29] ...trap/engines/engine.py.229: process: 0.82876 | load: 0.00292 [I 2022-10-10 16:30:29] ...trap/engines/engine.py.230: loss: 0.12817 [I 2022-10-10 16:30:29] ...optimizers/trijoint.py.151: total_norm: 21.151855 [I 2022-10-10 16:30:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 1080/2304 [I 2022-10-10 16:30:37] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:14 [I 2022-10-10 16:30:37] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00294 [I 2022-10-10 16:30:37] ...trap/engines/engine.py.230: loss: 0.15085 [I 2022-10-10 16:30:37] ...optimizers/trijoint.py.151: total_norm: 14.944253 [I 2022-10-10 16:30:46] ...trap/engines/engine.py.224: train: epoch 78 | batch 1090/2304 [I 2022-10-10 16:30:46] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:16:57 [I 2022-10-10 16:30:46] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00291 [I 2022-10-10 16:30:46] ...trap/engines/engine.py.230: loss: 0.20941 [I 2022-10-10 16:30:46] ...optimizers/trijoint.py.151: total_norm: 15.792089 [I 2022-10-10 16:30:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 1100/2304 [I 2022-10-10 16:30:54] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:01 [I 2022-10-10 16:30:54] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00299 [I 2022-10-10 16:30:54] ...trap/engines/engine.py.230: loss: 0.10803 [I 2022-10-10 16:30:54] ...optimizers/trijoint.py.151: total_norm: 14.536382 [I 2022-10-10 16:31:03] ...trap/engines/engine.py.224: train: epoch 78 | batch 1110/2304 [I 2022-10-10 16:31:03] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:51 [I 2022-10-10 16:31:03] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00294 [I 2022-10-10 16:31:03] ...trap/engines/engine.py.230: loss: 0.16263 [I 2022-10-10 16:31:03] ...optimizers/trijoint.py.151: total_norm: 15.336818 [I 2022-10-10 16:31:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 1120/2304 [I 2022-10-10 16:31:11] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:40 [I 2022-10-10 16:31:11] ...trap/engines/engine.py.229: process: 0.84744 | load: 0.00292 [I 2022-10-10 16:31:11] ...trap/engines/engine.py.230: loss: 0.12865 [I 2022-10-10 16:31:11] ...optimizers/trijoint.py.151: total_norm: 12.334569 [I 2022-10-10 16:31:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 1130/2304 [I 2022-10-10 16:31:20] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:29 [I 2022-10-10 16:31:20] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00292 [I 2022-10-10 16:31:20] ...trap/engines/engine.py.230: loss: 0.09893 [I 2022-10-10 16:31:20] ...optimizers/trijoint.py.151: total_norm: 13.635378 [I 2022-10-10 16:31:28] ...trap/engines/engine.py.224: train: epoch 78 | batch 1140/2304 [I 2022-10-10 16:31:28] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:22 [I 2022-10-10 16:31:28] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00291 [I 2022-10-10 16:31:28] ...trap/engines/engine.py.230: loss: 0.11668 [I 2022-10-10 16:31:28] ...optimizers/trijoint.py.151: total_norm: 25.500696 [I 2022-10-10 16:31:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 1150/2304 [I 2022-10-10 16:31:37] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:09 [I 2022-10-10 16:31:37] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00292 [I 2022-10-10 16:31:37] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-10 16:31:37] ...optimizers/trijoint.py.151: total_norm: 14.541244 [I 2022-10-10 16:31:45] ...trap/engines/engine.py.224: train: epoch 78 | batch 1160/2304 [I 2022-10-10 16:31:45] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:07 [I 2022-10-10 16:31:45] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00296 [I 2022-10-10 16:31:45] ...trap/engines/engine.py.230: loss: 0.20670 [I 2022-10-10 16:31:45] ...optimizers/trijoint.py.151: total_norm: 11.874236 [I 2022-10-10 16:31:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 1170/2304 [I 2022-10-10 16:31:54] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:15:57 [I 2022-10-10 16:31:54] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00297 [I 2022-10-10 16:31:54] ...trap/engines/engine.py.230: loss: 0.11130 [I 2022-10-10 16:31:54] ...optimizers/trijoint.py.151: total_norm: 12.161516 [I 2022-10-10 16:32:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1180/2304 [I 2022-10-10 16:32:02] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:45 [I 2022-10-10 16:32:02] ...trap/engines/engine.py.229: process: 0.83057 | load: 0.00293 [I 2022-10-10 16:32:02] ...trap/engines/engine.py.230: loss: 0.16622 [I 2022-10-10 16:32:02] ...optimizers/trijoint.py.151: total_norm: 14.006019 [I 2022-10-10 16:32:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 1190/2304 [I 2022-10-10 16:32:11] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:43 [I 2022-10-10 16:32:11] ...trap/engines/engine.py.229: process: 0.84718 | load: 0.00293 [I 2022-10-10 16:32:11] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-10 16:32:11] ...optimizers/trijoint.py.151: total_norm: 17.637501 [I 2022-10-10 16:32:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1200/2304 [I 2022-10-10 16:32:19] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:28 [I 2022-10-10 16:32:19] ...trap/engines/engine.py.229: process: 0.84456 | load: 0.00295 [I 2022-10-10 16:32:19] ...trap/engines/engine.py.230: loss: 0.11254 [I 2022-10-10 16:32:19] ...optimizers/trijoint.py.151: total_norm: 12.444687 [I 2022-10-10 16:32:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1210/2304 [I 2022-10-10 16:32:27] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:21 [I 2022-10-10 16:32:27] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00294 [I 2022-10-10 16:32:27] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-10 16:32:27] ...optimizers/trijoint.py.151: total_norm: 14.976739 [I 2022-10-10 16:32:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1220/2304 [I 2022-10-10 16:32:36] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:14 [I 2022-10-10 16:32:36] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00296 [I 2022-10-10 16:32:36] ...trap/engines/engine.py.230: loss: 0.11951 [I 2022-10-10 16:32:36] ...optimizers/trijoint.py.151: total_norm: 18.429462 [I 2022-10-10 16:32:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1230/2304 [I 2022-10-10 16:32:44] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:07 [I 2022-10-10 16:32:45] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00293 [I 2022-10-10 16:32:45] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-10 16:32:45] ...optimizers/trijoint.py.151: total_norm: 13.404909 [I 2022-10-10 16:32:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 1240/2304 [I 2022-10-10 16:32:53] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:07 [I 2022-10-10 16:32:53] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00291 [I 2022-10-10 16:32:53] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-10 16:32:53] ...optimizers/trijoint.py.151: total_norm: 17.069197 [I 2022-10-10 16:33:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1250/2304 [I 2022-10-10 16:33:02] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:52 [I 2022-10-10 16:33:02] ...trap/engines/engine.py.229: process: 0.84916 | load: 0.00294 [I 2022-10-10 16:33:02] ...trap/engines/engine.py.230: loss: 0.16224 [I 2022-10-10 16:33:02] ...optimizers/trijoint.py.151: total_norm: 17.309967 [I 2022-10-10 16:33:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 1260/2304 [I 2022-10-10 16:33:10] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:41 [I 2022-10-10 16:33:10] ...trap/engines/engine.py.229: process: 0.84103 | load: 0.00300 [I 2022-10-10 16:33:10] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-10 16:33:10] ...optimizers/trijoint.py.151: total_norm: 24.151129 [I 2022-10-10 16:33:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1270/2304 [I 2022-10-10 16:33:19] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:32 [I 2022-10-10 16:33:19] ...trap/engines/engine.py.229: process: 0.84995 | load: 0.00293 [I 2022-10-10 16:33:19] ...trap/engines/engine.py.230: loss: 0.15944 [I 2022-10-10 16:33:19] ...optimizers/trijoint.py.151: total_norm: 17.651306 [I 2022-10-10 16:33:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1280/2304 [I 2022-10-10 16:33:27] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:22 [I 2022-10-10 16:33:27] ...trap/engines/engine.py.229: process: 0.84636 | load: 0.00292 [I 2022-10-10 16:33:27] ...trap/engines/engine.py.230: loss: 0.15304 [I 2022-10-10 16:33:27] ...optimizers/trijoint.py.151: total_norm: 25.033022 [I 2022-10-10 16:33:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1290/2304 [I 2022-10-10 16:33:36] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:14 [I 2022-10-10 16:33:36] ...trap/engines/engine.py.229: process: 0.83867 | load: 0.00293 [I 2022-10-10 16:33:36] ...trap/engines/engine.py.230: loss: 0.12165 [I 2022-10-10 16:33:36] ...optimizers/trijoint.py.151: total_norm: 16.221851 [I 2022-10-10 16:33:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1300/2304 [I 2022-10-10 16:33:44] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:01 [I 2022-10-10 16:33:44] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00299 [I 2022-10-10 16:33:44] ...trap/engines/engine.py.230: loss: 0.10587 [I 2022-10-10 16:33:44] ...optimizers/trijoint.py.151: total_norm: 16.794029 [I 2022-10-10 16:33:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1310/2304 [I 2022-10-10 16:33:52] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:03 [I 2022-10-10 16:33:52] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00307 [I 2022-10-10 16:33:52] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-10 16:33:52] ...optimizers/trijoint.py.151: total_norm: 9.156145 [I 2022-10-10 16:34:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1320/2304 [I 2022-10-10 16:34:01] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:51 [I 2022-10-10 16:34:01] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00303 [I 2022-10-10 16:34:01] ...trap/engines/engine.py.230: loss: 0.21856 [I 2022-10-10 16:34:01] ...optimizers/trijoint.py.151: total_norm: 17.400581 [I 2022-10-10 16:34:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1330/2304 [I 2022-10-10 16:34:09] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:44 [I 2022-10-10 16:34:09] ...trap/engines/engine.py.229: process: 0.83671 | load: 0.00301 [I 2022-10-10 16:34:09] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-10 16:34:09] ...optimizers/trijoint.py.151: total_norm: 14.937578 [I 2022-10-10 16:34:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1340/2304 [I 2022-10-10 16:34:18] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:36 [I 2022-10-10 16:34:18] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00304 [I 2022-10-10 16:34:18] ...trap/engines/engine.py.230: loss: 0.14380 [I 2022-10-10 16:34:18] ...optimizers/trijoint.py.151: total_norm: 55.262684 [I 2022-10-10 16:34:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1350/2304 [I 2022-10-10 16:34:26] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:28 [I 2022-10-10 16:34:26] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00304 [I 2022-10-10 16:34:26] ...trap/engines/engine.py.230: loss: 0.10484 [I 2022-10-10 16:34:26] ...optimizers/trijoint.py.151: total_norm: 10.740406 [I 2022-10-10 16:34:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1360/2304 [I 2022-10-10 16:34:35] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:17 [I 2022-10-10 16:34:35] ...trap/engines/engine.py.229: process: 0.84700 | load: 0.00303 [I 2022-10-10 16:34:35] ...trap/engines/engine.py.230: loss: 0.21221 [I 2022-10-10 16:34:35] ...optimizers/trijoint.py.151: total_norm: 24.175680 [I 2022-10-10 16:34:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1370/2304 [I 2022-10-10 16:34:43] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:14 [I 2022-10-10 16:34:43] ...trap/engines/engine.py.229: process: 0.84591 | load: 0.00311 [I 2022-10-10 16:34:43] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-10 16:34:43] ...optimizers/trijoint.py.151: total_norm: 18.222574 [I 2022-10-10 16:34:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1380/2304 [I 2022-10-10 16:34:52] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:08 [I 2022-10-10 16:34:52] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00311 [I 2022-10-10 16:34:52] ...trap/engines/engine.py.230: loss: 0.17255 [I 2022-10-10 16:34:52] ...optimizers/trijoint.py.151: total_norm: 18.826273 [I 2022-10-10 16:35:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1390/2304 [I 2022-10-10 16:35:01] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:06 [I 2022-10-10 16:35:01] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00318 [I 2022-10-10 16:35:01] ...trap/engines/engine.py.230: loss: 0.21862 [I 2022-10-10 16:35:01] ...optimizers/trijoint.py.151: total_norm: 13.197166 [I 2022-10-10 16:35:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1400/2304 [I 2022-10-10 16:35:09] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:53 [I 2022-10-10 16:35:09] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00311 [I 2022-10-10 16:35:09] ...trap/engines/engine.py.230: loss: 0.14825 [I 2022-10-10 16:35:09] ...optimizers/trijoint.py.151: total_norm: 18.841722 [I 2022-10-10 16:35:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1410/2304 [I 2022-10-10 16:35:18] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:48 [I 2022-10-10 16:35:18] ...trap/engines/engine.py.229: process: 0.85861 | load: 0.00313 [I 2022-10-10 16:35:18] ...trap/engines/engine.py.230: loss: 0.08978 [I 2022-10-10 16:35:18] ...optimizers/trijoint.py.151: total_norm: 7.683826 [I 2022-10-10 16:35:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1420/2304 [I 2022-10-10 16:35:26] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:40 [I 2022-10-10 16:35:26] ...trap/engines/engine.py.229: process: 0.87184 | load: 0.00339 [I 2022-10-10 16:35:26] ...trap/engines/engine.py.230: loss: 0.12981 [I 2022-10-10 16:35:26] ...optimizers/trijoint.py.151: total_norm: 16.801624 [I 2022-10-10 16:35:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1430/2304 [I 2022-10-10 16:35:35] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:26 [I 2022-10-10 16:35:35] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00310 [I 2022-10-10 16:35:35] ...trap/engines/engine.py.230: loss: 0.12304 [I 2022-10-10 16:35:35] ...optimizers/trijoint.py.151: total_norm: 12.206317 [I 2022-10-10 16:35:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1440/2304 [I 2022-10-10 16:35:43] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:21 [I 2022-10-10 16:35:43] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00310 [I 2022-10-10 16:35:43] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-10 16:35:43] ...optimizers/trijoint.py.151: total_norm: 8.956851 [I 2022-10-10 16:35:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1450/2304 [I 2022-10-10 16:35:52] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:14 [I 2022-10-10 16:35:52] ...trap/engines/engine.py.229: process: 0.88164 | load: 0.00310 [I 2022-10-10 16:35:52] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-10 16:35:52] ...optimizers/trijoint.py.151: total_norm: 11.774358 [I 2022-10-10 16:36:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1460/2304 [I 2022-10-10 16:36:01] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:03 [I 2022-10-10 16:36:01] ...trap/engines/engine.py.229: process: 0.87302 | load: 0.00314 [I 2022-10-10 16:36:01] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-10 16:36:01] ...optimizers/trijoint.py.151: total_norm: 44.889801 [I 2022-10-10 16:36:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 1470/2304 [I 2022-10-10 16:36:17] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:14:48 [I 2022-10-10 16:36:17] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00306 [I 2022-10-10 16:36:17] ...trap/engines/engine.py.230: loss: 0.13053 [I 2022-10-10 16:36:17] ...optimizers/trijoint.py.151: total_norm: 12.584500 [I 2022-10-10 16:36:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1480/2304 [I 2022-10-10 16:36:26] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:12:08 [I 2022-10-10 16:36:26] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00334 [I 2022-10-10 16:36:26] ...trap/engines/engine.py.230: loss: 0.18203 [I 2022-10-10 16:36:26] ...optimizers/trijoint.py.151: total_norm: 18.851896 [I 2022-10-10 16:36:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 1490/2304 [I 2022-10-10 16:36:34] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:39 [I 2022-10-10 16:36:34] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00309 [I 2022-10-10 16:36:34] ...trap/engines/engine.py.230: loss: 0.09737 [I 2022-10-10 16:36:34] ...optimizers/trijoint.py.151: total_norm: 15.648339 [I 2022-10-10 16:36:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1500/2304 [I 2022-10-10 16:36:43] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:30 [I 2022-10-10 16:36:43] ...trap/engines/engine.py.229: process: 0.87086 | load: 0.00308 [I 2022-10-10 16:36:43] ...trap/engines/engine.py.230: loss: 0.14971 [I 2022-10-10 16:36:43] ...optimizers/trijoint.py.151: total_norm: 12.740612 [I 2022-10-10 16:36:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 1510/2304 [I 2022-10-10 16:36:51] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:19 [I 2022-10-10 16:36:51] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00307 [I 2022-10-10 16:36:51] ...trap/engines/engine.py.230: loss: 0.11943 [I 2022-10-10 16:36:51] ...optimizers/trijoint.py.151: total_norm: 9.605862 [I 2022-10-10 16:37:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 1520/2304 [I 2022-10-10 16:37:00] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:14 [I 2022-10-10 16:37:00] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00308 [I 2022-10-10 16:37:00] ...trap/engines/engine.py.230: loss: 0.11986 [I 2022-10-10 16:37:00] ...optimizers/trijoint.py.151: total_norm: 17.858511 [I 2022-10-10 16:37:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1530/2304 [I 2022-10-10 16:37:09] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:03 [I 2022-10-10 16:37:09] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00312 [I 2022-10-10 16:37:09] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-10 16:37:09] ...optimizers/trijoint.py.151: total_norm: 17.857655 [I 2022-10-10 16:37:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 1540/2304 [I 2022-10-10 16:37:17] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:56 [I 2022-10-10 16:37:17] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00316 [I 2022-10-10 16:37:17] ...trap/engines/engine.py.230: loss: 0.15671 [I 2022-10-10 16:37:17] ...optimizers/trijoint.py.151: total_norm: 9.202610 [I 2022-10-10 16:37:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1550/2304 [I 2022-10-10 16:37:26] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:46 [I 2022-10-10 16:37:26] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00317 [I 2022-10-10 16:37:26] ...trap/engines/engine.py.230: loss: 0.10005 [I 2022-10-10 16:37:26] ...optimizers/trijoint.py.151: total_norm: 11.480621 [I 2022-10-10 16:37:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 1560/2304 [I 2022-10-10 16:37:34] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:41 [I 2022-10-10 16:37:34] ...trap/engines/engine.py.229: process: 0.87645 | load: 0.00320 [I 2022-10-10 16:37:35] ...trap/engines/engine.py.230: loss: 0.12489 [I 2022-10-10 16:37:35] ...optimizers/trijoint.py.151: total_norm: 16.415241 [I 2022-10-10 16:37:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1570/2304 [I 2022-10-10 16:37:43] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:30 [I 2022-10-10 16:37:43] ...trap/engines/engine.py.229: process: 0.85703 | load: 0.00331 [I 2022-10-10 16:37:43] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-10 16:37:43] ...optimizers/trijoint.py.151: total_norm: 15.986165 [I 2022-10-10 16:37:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1580/2304 [I 2022-10-10 16:37:52] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:19 [I 2022-10-10 16:37:52] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00325 [I 2022-10-10 16:37:52] ...trap/engines/engine.py.230: loss: 0.14366 [I 2022-10-10 16:37:52] ...optimizers/trijoint.py.151: total_norm: 23.562891 [I 2022-10-10 16:38:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 1590/2304 [I 2022-10-10 16:38:00] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:14 [I 2022-10-10 16:38:00] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00309 [I 2022-10-10 16:38:00] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-10 16:38:00] ...optimizers/trijoint.py.151: total_norm: 21.081371 [I 2022-10-10 16:38:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1600/2304 [I 2022-10-10 16:38:09] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:02 [I 2022-10-10 16:38:09] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00312 [I 2022-10-10 16:38:09] ...trap/engines/engine.py.230: loss: 0.10743 [I 2022-10-10 16:38:09] ...optimizers/trijoint.py.151: total_norm: 10.816860 [I 2022-10-10 16:38:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1610/2304 [I 2022-10-10 16:38:18] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:55 [I 2022-10-10 16:38:18] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00314 [I 2022-10-10 16:38:18] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-10 16:38:18] ...optimizers/trijoint.py.151: total_norm: 16.747368 [I 2022-10-10 16:38:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1620/2304 [I 2022-10-10 16:38:26] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:46 [I 2022-10-10 16:38:26] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00310 [I 2022-10-10 16:38:26] ...trap/engines/engine.py.230: loss: 0.16875 [I 2022-10-10 16:38:26] ...optimizers/trijoint.py.151: total_norm: 31.348497 [I 2022-10-10 16:38:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1630/2304 [I 2022-10-10 16:38:35] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:34 [I 2022-10-10 16:38:35] ...trap/engines/engine.py.229: process: 0.83869 | load: 0.00295 [I 2022-10-10 16:38:35] ...trap/engines/engine.py.230: loss: 0.12816 [I 2022-10-10 16:38:35] ...optimizers/trijoint.py.151: total_norm: 39.301651 [I 2022-10-10 16:38:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1640/2304 [I 2022-10-10 16:38:43] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:21 [I 2022-10-10 16:38:43] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00304 [I 2022-10-10 16:38:43] ...trap/engines/engine.py.230: loss: 0.11664 [I 2022-10-10 16:38:43] ...optimizers/trijoint.py.151: total_norm: 34.867489 [I 2022-10-10 16:38:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 1650/2304 [I 2022-10-10 16:38:54] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:10:51 [I 2022-10-10 16:38:54] ...trap/engines/engine.py.229: process: 0.85992 | load: 0.00294 [I 2022-10-10 16:38:54] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-10 16:38:54] ...optimizers/trijoint.py.151: total_norm: 14.928207 [I 2022-10-10 16:39:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1660/2304 [I 2022-10-10 16:39:02] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:15 [I 2022-10-10 16:39:02] ...trap/engines/engine.py.229: process: 0.84786 | load: 0.00290 [I 2022-10-10 16:39:02] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-10 16:39:02] ...optimizers/trijoint.py.151: total_norm: 19.824188 [I 2022-10-10 16:39:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 1670/2304 [I 2022-10-10 16:39:11] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:08:54 [I 2022-10-10 16:39:11] ...trap/engines/engine.py.229: process: 0.83609 | load: 0.00297 [I 2022-10-10 16:39:11] ...trap/engines/engine.py.230: loss: 0.16475 [I 2022-10-10 16:39:11] ...optimizers/trijoint.py.151: total_norm: 26.536825 [I 2022-10-10 16:39:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1680/2304 [I 2022-10-10 16:39:19] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:48 [I 2022-10-10 16:39:19] ...trap/engines/engine.py.229: process: 0.84217 | load: 0.00297 [I 2022-10-10 16:39:19] ...trap/engines/engine.py.230: loss: 0.13740 [I 2022-10-10 16:39:19] ...optimizers/trijoint.py.151: total_norm: 14.886456 [I 2022-10-10 16:39:28] ...trap/engines/engine.py.224: train: epoch 78 | batch 1690/2304 [I 2022-10-10 16:39:28] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:38 [I 2022-10-10 16:39:28] ...trap/engines/engine.py.229: process: 0.84494 | load: 0.00298 [I 2022-10-10 16:39:28] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-10 16:39:28] ...optimizers/trijoint.py.151: total_norm: 11.628874 [I 2022-10-10 16:39:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1700/2304 [I 2022-10-10 16:39:36] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:33 [I 2022-10-10 16:39:36] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00297 [I 2022-10-10 16:39:36] ...trap/engines/engine.py.230: loss: 0.08554 [I 2022-10-10 16:39:36] ...optimizers/trijoint.py.151: total_norm: 9.304358 [I 2022-10-10 16:39:45] ...trap/engines/engine.py.224: train: epoch 78 | batch 1710/2304 [I 2022-10-10 16:39:45] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:22 [I 2022-10-10 16:39:45] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00295 [I 2022-10-10 16:39:45] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-10 16:39:45] ...optimizers/trijoint.py.151: total_norm: 19.258711 [I 2022-10-10 16:39:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 1720/2304 [I 2022-10-10 16:39:53] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:14 [I 2022-10-10 16:39:53] ...trap/engines/engine.py.229: process: 0.84986 | load: 0.00294 [I 2022-10-10 16:39:53] ...trap/engines/engine.py.230: loss: 0.14049 [I 2022-10-10 16:39:53] ...optimizers/trijoint.py.151: total_norm: 12.204792 [I 2022-10-10 16:40:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1730/2304 [I 2022-10-10 16:40:02] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:03 [I 2022-10-10 16:40:02] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00294 [I 2022-10-10 16:40:02] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-10 16:40:02] ...optimizers/trijoint.py.151: total_norm: 17.941898 [I 2022-10-10 16:40:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 1740/2304 [I 2022-10-10 16:40:10] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:07:57 [I 2022-10-10 16:40:10] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00298 [I 2022-10-10 16:40:10] ...trap/engines/engine.py.230: loss: 0.12562 [I 2022-10-10 16:40:10] ...optimizers/trijoint.py.151: total_norm: 12.547560 [I 2022-10-10 16:40:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1750/2304 [I 2022-10-10 16:40:19] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:53 [I 2022-10-10 16:40:19] ...trap/engines/engine.py.229: process: 0.86412 | load: 0.00294 [I 2022-10-10 16:40:19] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-10 16:40:19] ...optimizers/trijoint.py.151: total_norm: 18.468912 [I 2022-10-10 16:40:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1760/2304 [I 2022-10-10 16:40:27] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:44 [I 2022-10-10 16:40:27] ...trap/engines/engine.py.229: process: 0.86826 | load: 0.00294 [I 2022-10-10 16:40:27] ...trap/engines/engine.py.230: loss: 0.11747 [I 2022-10-10 16:40:27] ...optimizers/trijoint.py.151: total_norm: 10.792197 [I 2022-10-10 16:40:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1770/2304 [I 2022-10-10 16:40:36] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:32 [I 2022-10-10 16:40:36] ...trap/engines/engine.py.229: process: 0.83480 | load: 0.00293 [I 2022-10-10 16:40:36] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-10 16:40:36] ...optimizers/trijoint.py.151: total_norm: 17.438206 [I 2022-10-10 16:40:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1780/2304 [I 2022-10-10 16:40:44] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:20 [I 2022-10-10 16:40:44] ...trap/engines/engine.py.229: process: 0.83769 | load: 0.00294 [I 2022-10-10 16:40:44] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-10 16:40:44] ...optimizers/trijoint.py.151: total_norm: 14.299523 [I 2022-10-10 16:40:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 1790/2304 [I 2022-10-10 16:40:53] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:11 [I 2022-10-10 16:40:53] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00292 [I 2022-10-10 16:40:53] ...trap/engines/engine.py.230: loss: 0.13115 [I 2022-10-10 16:40:53] ...optimizers/trijoint.py.151: total_norm: 18.177462 [I 2022-10-10 16:41:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1800/2304 [I 2022-10-10 16:41:01] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:04 [I 2022-10-10 16:41:01] ...trap/engines/engine.py.229: process: 0.83404 | load: 0.00291 [I 2022-10-10 16:41:01] ...trap/engines/engine.py.230: loss: 0.17094 [I 2022-10-10 16:41:01] ...optimizers/trijoint.py.151: total_norm: 21.959677 [I 2022-10-10 16:41:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 1810/2304 [I 2022-10-10 16:41:10] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:06:56 [I 2022-10-10 16:41:10] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00295 [I 2022-10-10 16:41:10] ...trap/engines/engine.py.230: loss: 0.21990 [I 2022-10-10 16:41:10] ...optimizers/trijoint.py.151: total_norm: 13.455380 [I 2022-10-10 16:41:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1820/2304 [I 2022-10-10 16:41:18] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:49 [I 2022-10-10 16:41:18] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00298 [I 2022-10-10 16:41:18] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-10 16:41:18] ...optimizers/trijoint.py.151: total_norm: 18.641684 [I 2022-10-10 16:41:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1830/2304 [I 2022-10-10 16:41:27] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:39 [I 2022-10-10 16:41:27] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00295 [I 2022-10-10 16:41:27] ...trap/engines/engine.py.230: loss: 0.20922 [I 2022-10-10 16:41:27] ...optimizers/trijoint.py.151: total_norm: 25.519508 [I 2022-10-10 16:41:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1840/2304 [I 2022-10-10 16:41:35] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:30 [I 2022-10-10 16:41:35] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00295 [I 2022-10-10 16:41:35] ...trap/engines/engine.py.230: loss: 0.17149 [I 2022-10-10 16:41:35] ...optimizers/trijoint.py.151: total_norm: 11.594790 [I 2022-10-10 16:41:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1850/2304 [I 2022-10-10 16:41:43] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:21 [I 2022-10-10 16:41:43] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00291 [I 2022-10-10 16:41:43] ...trap/engines/engine.py.230: loss: 0.08977 [I 2022-10-10 16:41:43] ...optimizers/trijoint.py.151: total_norm: 11.812819 [I 2022-10-10 16:41:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1860/2304 [I 2022-10-10 16:41:52] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:17 [I 2022-10-10 16:41:52] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00308 [I 2022-10-10 16:41:52] ...trap/engines/engine.py.230: loss: 0.13773 [I 2022-10-10 16:41:52] ...optimizers/trijoint.py.151: total_norm: 16.226942 [I 2022-10-10 16:42:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1870/2304 [I 2022-10-10 16:42:01] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:11 [I 2022-10-10 16:42:01] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00310 [I 2022-10-10 16:42:01] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-10 16:42:01] ...optimizers/trijoint.py.151: total_norm: 13.180397 [I 2022-10-10 16:42:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1880/2304 [I 2022-10-10 16:42:09] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:06:03 [I 2022-10-10 16:42:09] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00309 [I 2022-10-10 16:42:09] ...trap/engines/engine.py.230: loss: 0.20893 [I 2022-10-10 16:42:09] ...optimizers/trijoint.py.151: total_norm: 13.403125 [I 2022-10-10 16:42:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1890/2304 [I 2022-10-10 16:42:18] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:55 [I 2022-10-10 16:42:18] ...trap/engines/engine.py.229: process: 0.85910 | load: 0.00341 [I 2022-10-10 16:42:18] ...trap/engines/engine.py.230: loss: 0.16930 [I 2022-10-10 16:42:18] ...optimizers/trijoint.py.151: total_norm: 18.380293 [I 2022-10-10 16:42:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1900/2304 [I 2022-10-10 16:42:26] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:47 [I 2022-10-10 16:42:26] ...trap/engines/engine.py.229: process: 0.87060 | load: 0.00311 [I 2022-10-10 16:42:26] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-10 16:42:26] ...optimizers/trijoint.py.151: total_norm: 22.460146 [I 2022-10-10 16:42:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1910/2304 [I 2022-10-10 16:42:35] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:39 [I 2022-10-10 16:42:35] ...trap/engines/engine.py.229: process: 0.87059 | load: 0.00326 [I 2022-10-10 16:42:35] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-10 16:42:35] ...optimizers/trijoint.py.151: total_norm: 8.227976 [I 2022-10-10 16:42:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1920/2304 [I 2022-10-10 16:42:44] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:29 [I 2022-10-10 16:42:44] ...trap/engines/engine.py.229: process: 0.85890 | load: 0.00319 [I 2022-10-10 16:42:44] ...trap/engines/engine.py.230: loss: 0.17828 [I 2022-10-10 16:42:44] ...optimizers/trijoint.py.151: total_norm: 18.402016 [I 2022-10-10 16:42:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1930/2304 [I 2022-10-10 16:42:52] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:21 [I 2022-10-10 16:42:52] ...trap/engines/engine.py.229: process: 0.87963 | load: 0.00307 [I 2022-10-10 16:42:52] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-10 16:42:52] ...optimizers/trijoint.py.151: total_norm: 43.175789 [I 2022-10-10 16:43:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1940/2304 [I 2022-10-10 16:43:01] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:12 [I 2022-10-10 16:43:01] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00331 [I 2022-10-10 16:43:01] ...trap/engines/engine.py.230: loss: 0.09458 [I 2022-10-10 16:43:01] ...optimizers/trijoint.py.151: total_norm: 14.910707 [I 2022-10-10 16:43:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1950/2304 [I 2022-10-10 16:43:09] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:05:05 [I 2022-10-10 16:43:09] ...trap/engines/engine.py.229: process: 0.87197 | load: 0.00323 [I 2022-10-10 16:43:09] ...trap/engines/engine.py.230: loss: 0.19834 [I 2022-10-10 16:43:09] ...optimizers/trijoint.py.151: total_norm: 11.583923 [I 2022-10-10 16:43:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1960/2304 [I 2022-10-10 16:43:18] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:55 [I 2022-10-10 16:43:18] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00325 [I 2022-10-10 16:43:18] ...trap/engines/engine.py.230: loss: 0.14787 [I 2022-10-10 16:43:18] ...optimizers/trijoint.py.151: total_norm: 11.614473 [I 2022-10-10 16:43:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1970/2304 [I 2022-10-10 16:43:27] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:47 [I 2022-10-10 16:43:27] ...trap/engines/engine.py.229: process: 0.86708 | load: 0.00328 [I 2022-10-10 16:43:27] ...trap/engines/engine.py.230: loss: 0.15732 [I 2022-10-10 16:43:27] ...optimizers/trijoint.py.151: total_norm: 20.353256 [I 2022-10-10 16:43:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1980/2304 [I 2022-10-10 16:43:35] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:38 [I 2022-10-10 16:43:35] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00327 [I 2022-10-10 16:43:35] ...trap/engines/engine.py.230: loss: 0.13324 [I 2022-10-10 16:43:35] ...optimizers/trijoint.py.151: total_norm: 24.576710 [I 2022-10-10 16:43:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1990/2304 [I 2022-10-10 16:43:44] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:30 [I 2022-10-10 16:43:44] ...trap/engines/engine.py.229: process: 0.85598 | load: 0.00312 [I 2022-10-10 16:43:44] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-10 16:43:44] ...optimizers/trijoint.py.151: total_norm: 11.028667 [I 2022-10-10 16:43:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 2000/2304 [I 2022-10-10 16:43:53] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:20 [I 2022-10-10 16:43:53] ...trap/engines/engine.py.229: process: 0.87608 | load: 0.00312 [I 2022-10-10 16:43:53] ...trap/engines/engine.py.230: loss: 0.11868 [I 2022-10-10 16:43:53] ...optimizers/trijoint.py.151: total_norm: 18.221773 [I 2022-10-10 16:44:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 2010/2304 [I 2022-10-10 16:44:01] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:10 [I 2022-10-10 16:44:01] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00316 [I 2022-10-10 16:44:01] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-10 16:44:01] ...optimizers/trijoint.py.151: total_norm: 14.393812 [I 2022-10-10 16:44:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 2020/2304 [I 2022-10-10 16:44:10] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:02 [I 2022-10-10 16:44:10] ...trap/engines/engine.py.229: process: 0.85530 | load: 0.00314 [I 2022-10-10 16:44:10] ...trap/engines/engine.py.230: loss: 0.17322 [I 2022-10-10 16:44:10] ...optimizers/trijoint.py.151: total_norm: 20.536329 [I 2022-10-10 16:44:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 2030/2304 [I 2022-10-10 16:44:18] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:55 [I 2022-10-10 16:44:18] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00332 [I 2022-10-10 16:44:18] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-10 16:44:18] ...optimizers/trijoint.py.151: total_norm: 21.916300 [I 2022-10-10 16:44:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 2040/2304 [I 2022-10-10 16:44:27] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:46 [I 2022-10-10 16:44:27] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00308 [I 2022-10-10 16:44:27] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-10 16:44:27] ...optimizers/trijoint.py.151: total_norm: 11.974017 [I 2022-10-10 16:44:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 2050/2304 [I 2022-10-10 16:44:35] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:37 [I 2022-10-10 16:44:35] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00311 [I 2022-10-10 16:44:35] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-10 16:44:35] ...optimizers/trijoint.py.151: total_norm: 23.437181 [I 2022-10-10 16:44:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 2060/2304 [I 2022-10-10 16:44:44] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:29 [I 2022-10-10 16:44:44] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00327 [I 2022-10-10 16:44:44] ...trap/engines/engine.py.230: loss: 0.11205 [I 2022-10-10 16:44:44] ...optimizers/trijoint.py.151: total_norm: 17.703754 [I 2022-10-10 16:44:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 2070/2304 [I 2022-10-10 16:44:53] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:20 [I 2022-10-10 16:44:53] ...trap/engines/engine.py.229: process: 0.86751 | load: 0.00321 [I 2022-10-10 16:44:53] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-10 16:44:53] ...optimizers/trijoint.py.151: total_norm: 19.973066 [I 2022-10-10 16:45:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 2080/2304 [I 2022-10-10 16:45:01] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:12 [I 2022-10-10 16:45:01] ...trap/engines/engine.py.229: process: 0.87105 | load: 0.00317 [I 2022-10-10 16:45:01] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-10 16:45:01] ...optimizers/trijoint.py.151: total_norm: 17.580048 [I 2022-10-10 16:45:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 2090/2304 [I 2022-10-10 16:45:10] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:03:04 [I 2022-10-10 16:45:10] ...trap/engines/engine.py.229: process: 0.86727 | load: 0.00325 [I 2022-10-10 16:45:10] ...trap/engines/engine.py.230: loss: 0.11594 [I 2022-10-10 16:45:10] ...optimizers/trijoint.py.151: total_norm: 15.587346 [I 2022-10-10 16:45:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 2100/2304 [I 2022-10-10 16:45:19] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:02:55 [I 2022-10-10 16:45:19] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00320 [I 2022-10-10 16:45:19] ...trap/engines/engine.py.230: loss: 0.10370 [I 2022-10-10 16:45:19] ...optimizers/trijoint.py.151: total_norm: 20.258566 [I 2022-10-10 16:45:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 2110/2304 [I 2022-10-10 16:45:27] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:46 [I 2022-10-10 16:45:27] ...trap/engines/engine.py.229: process: 0.86116 | load: 0.00322 [I 2022-10-10 16:45:27] ...trap/engines/engine.py.230: loss: 0.18314 [I 2022-10-10 16:45:27] ...optimizers/trijoint.py.151: total_norm: 22.752335 [I 2022-10-10 16:45:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 2120/2304 [I 2022-10-10 16:45:36] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:38 [I 2022-10-10 16:45:36] ...trap/engines/engine.py.229: process: 0.87012 | load: 0.00313 [I 2022-10-10 16:45:36] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-10 16:45:36] ...optimizers/trijoint.py.151: total_norm: 24.302746 [I 2022-10-10 16:45:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 2130/2304 [I 2022-10-10 16:45:44] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:29 [I 2022-10-10 16:45:44] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00304 [I 2022-10-10 16:45:44] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-10 16:45:44] ...optimizers/trijoint.py.151: total_norm: 11.077557 [I 2022-10-10 16:45:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 2140/2304 [I 2022-10-10 16:45:53] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:18 [I 2022-10-10 16:45:53] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00302 [I 2022-10-10 16:45:53] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-10 16:45:53] ...optimizers/trijoint.py.151: total_norm: 13.440649 [I 2022-10-10 16:46:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 2150/2304 [I 2022-10-10 16:46:01] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:02:09 [I 2022-10-10 16:46:01] ...trap/engines/engine.py.229: process: 0.83258 | load: 0.00297 [I 2022-10-10 16:46:01] ...trap/engines/engine.py.230: loss: 0.12632 [I 2022-10-10 16:46:01] ...optimizers/trijoint.py.151: total_norm: 11.442355 [I 2022-10-10 16:46:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 2160/2304 [I 2022-10-10 16:46:10] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:02:02 [I 2022-10-10 16:46:10] ...trap/engines/engine.py.229: process: 0.84706 | load: 0.00295 [I 2022-10-10 16:46:10] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-10 16:46:10] ...optimizers/trijoint.py.151: total_norm: 14.357264 [I 2022-10-10 16:46:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 2170/2304 [I 2022-10-10 16:46:18] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:52 [I 2022-10-10 16:46:18] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00299 [I 2022-10-10 16:46:18] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-10 16:46:18] ...optimizers/trijoint.py.151: total_norm: 15.213696 [I 2022-10-10 16:46:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 2180/2304 [I 2022-10-10 16:46:27] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:45 [I 2022-10-10 16:46:27] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00296 [I 2022-10-10 16:46:27] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-10 16:46:27] ...optimizers/trijoint.py.151: total_norm: 8.614244 [I 2022-10-10 16:46:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 2190/2304 [I 2022-10-10 16:46:35] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:36 [I 2022-10-10 16:46:35] ...trap/engines/engine.py.229: process: 0.84050 | load: 0.00299 [I 2022-10-10 16:46:35] ...trap/engines/engine.py.230: loss: 0.17889 [I 2022-10-10 16:46:35] ...optimizers/trijoint.py.151: total_norm: 45.130596 [I 2022-10-10 16:46:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 2200/2304 [I 2022-10-10 16:46:44] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:28 [I 2022-10-10 16:46:44] ...trap/engines/engine.py.229: process: 0.86239 | load: 0.00296 [I 2022-10-10 16:46:44] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-10 16:46:44] ...optimizers/trijoint.py.151: total_norm: 21.628202 [I 2022-10-10 16:46:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 2210/2304 [I 2022-10-10 16:46:52] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:19 [I 2022-10-10 16:46:52] ...trap/engines/engine.py.229: process: 0.85175 | load: 0.00301 [I 2022-10-10 16:46:52] ...trap/engines/engine.py.230: loss: 0.19935 [I 2022-10-10 16:46:52] ...optimizers/trijoint.py.151: total_norm: 17.774982 [I 2022-10-10 16:47:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 2220/2304 [I 2022-10-10 16:47:01] ...trap/engines/engine.py.228: elapsed: 0:31:47 | left: 0:01:11 [I 2022-10-10 16:47:01] ...trap/engines/engine.py.229: process: 0.84719 | load: 0.00307 [I 2022-10-10 16:47:01] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-10 16:47:01] ...optimizers/trijoint.py.151: total_norm: 19.743221 [I 2022-10-10 16:47:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 2230/2304 [I 2022-10-10 16:47:09] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:02 [I 2022-10-10 16:47:09] ...trap/engines/engine.py.229: process: 0.83614 | load: 0.00295 [I 2022-10-10 16:47:09] ...trap/engines/engine.py.230: loss: 0.10629 [I 2022-10-10 16:47:09] ...optimizers/trijoint.py.151: total_norm: 20.626413 [I 2022-10-10 16:47:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 2240/2304 [I 2022-10-10 16:47:18] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:54 [I 2022-10-10 16:47:18] ...trap/engines/engine.py.229: process: 0.83516 | load: 0.00294 [I 2022-10-10 16:47:18] ...trap/engines/engine.py.230: loss: 0.11518 [I 2022-10-10 16:47:18] ...optimizers/trijoint.py.151: total_norm: 14.510542 [I 2022-10-10 16:47:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 2250/2304 [I 2022-10-10 16:47:26] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:45 [I 2022-10-10 16:47:26] ...trap/engines/engine.py.229: process: 0.83550 | load: 0.00295 [I 2022-10-10 16:47:26] ...trap/engines/engine.py.230: loss: 0.16940 [I 2022-10-10 16:47:26] ...optimizers/trijoint.py.151: total_norm: 21.739431 [I 2022-10-10 16:47:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 2260/2304 [I 2022-10-10 16:47:35] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:37 [I 2022-10-10 16:47:35] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00295 [I 2022-10-10 16:47:35] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-10 16:47:35] ...optimizers/trijoint.py.151: total_norm: 15.720329 [I 2022-10-10 16:47:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 2270/2304 [I 2022-10-10 16:47:43] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:28 [I 2022-10-10 16:47:43] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00294 [I 2022-10-10 16:47:43] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-10 16:47:43] ...optimizers/trijoint.py.151: total_norm: 11.222466 [I 2022-10-10 16:47:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 2280/2304 [I 2022-10-10 16:47:52] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:20 [I 2022-10-10 16:47:52] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00293 [I 2022-10-10 16:47:52] ...trap/engines/engine.py.230: loss: 0.21089 [I 2022-10-10 16:47:52] ...optimizers/trijoint.py.151: total_norm: 35.995514 [I 2022-10-10 16:48:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 2290/2304 [I 2022-10-10 16:48:00] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:11 [I 2022-10-10 16:48:00] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00294 [I 2022-10-10 16:48:00] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-10 16:48:00] ...optimizers/trijoint.py.151: total_norm: 27.440983 [I 2022-10-10 16:48:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 2300/2304 [I 2022-10-10 16:48:09] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:03 [I 2022-10-10 16:48:09] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00015 [I 2022-10-10 16:48:09] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-10 16:48:09] ...optimizers/trijoint.py.151: total_norm: 54.988049 [I 2022-10-10 16:48:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 2304/2304 [I 2022-10-10 16:48:12] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:00 [I 2022-10-10 16:48:12] ...trap/engines/engine.py.229: process: 0.83181 | load: 0.00015 [I 2022-10-10 16:48:12] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-10 16:48:12] ...optimizers/trijoint.py.151: total_norm: 11.984799 [S 2022-10-10 16:48:13] ...trap/engines/engine.py.129: train_epoch.epoch: 78 [S 2022-10-10 16:48:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.1346623649307549 [I 2022-10-10 16:48:13] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 16:48:15] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 78 [I 2022-10-10 16:48:15] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 16:48:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 16:48:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 16:48:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 16:48:19] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 16:48:20] ...trap/engines/engine.py.321: eval: epoch 78 | batch 0/510 [I 2022-10-10 16:48:20] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:56 [I 2022-10-10 16:48:20] ...trap/engines/engine.py.326: process: 5.28669 | load: 3.26805 [I 2022-10-10 16:48:26] ...trap/engines/engine.py.321: eval: epoch 78 | batch 10/510 [I 2022-10-10 16:48:26] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:31 [I 2022-10-10 16:48:26] ...trap/engines/engine.py.326: process: 0.72670 | load: 0.00018 [I 2022-10-10 16:48:31] ...trap/engines/engine.py.321: eval: epoch 78 | batch 20/510 [I 2022-10-10 16:48:31] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:20 [I 2022-10-10 16:48:31] ...trap/engines/engine.py.326: process: 0.39553 | load: 0.00018 [I 2022-10-10 16:48:35] ...trap/engines/engine.py.321: eval: epoch 78 | batch 30/510 [I 2022-10-10 16:48:35] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:26 [I 2022-10-10 16:48:35] ...trap/engines/engine.py.326: process: 0.40527 | load: 0.00018 [I 2022-10-10 16:48:40] ...trap/engines/engine.py.321: eval: epoch 78 | batch 40/510 [I 2022-10-10 16:48:40] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:37 [I 2022-10-10 16:48:40] ...trap/engines/engine.py.326: process: 0.42588 | load: 0.00018 [I 2022-10-10 16:48:44] ...trap/engines/engine.py.321: eval: epoch 78 | batch 50/510 [I 2022-10-10 16:48:44] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:15 [I 2022-10-10 16:48:44] ...trap/engines/engine.py.326: process: 0.42667 | load: 0.00018 [I 2022-10-10 16:48:49] ...trap/engines/engine.py.321: eval: epoch 78 | batch 60/510 [I 2022-10-10 16:48:49] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:37 [I 2022-10-10 16:48:49] ...trap/engines/engine.py.326: process: 0.47357 | load: 0.00018 [I 2022-10-10 16:48:54] ...trap/engines/engine.py.321: eval: epoch 78 | batch 70/510 [I 2022-10-10 16:48:54] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:36 [I 2022-10-10 16:48:54] ...trap/engines/engine.py.326: process: 0.41951 | load: 0.00019 [I 2022-10-10 16:48:58] ...trap/engines/engine.py.321: eval: epoch 78 | batch 80/510 [I 2022-10-10 16:48:58] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:21 [I 2022-10-10 16:48:58] ...trap/engines/engine.py.326: process: 0.46543 | load: 0.00018 [I 2022-10-10 16:49:03] ...trap/engines/engine.py.321: eval: epoch 78 | batch 90/510 [I 2022-10-10 16:49:03] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:16 [I 2022-10-10 16:49:03] ...trap/engines/engine.py.326: process: 0.45616 | load: 0.00026 [I 2022-10-10 16:49:08] ...trap/engines/engine.py.321: eval: epoch 78 | batch 100/510 [I 2022-10-10 16:49:08] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:59 [I 2022-10-10 16:49:08] ...trap/engines/engine.py.326: process: 0.31477 | load: 0.00020 [I 2022-10-10 16:49:11] ...trap/engines/engine.py.321: eval: epoch 78 | batch 110/510 [I 2022-10-10 16:49:11] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:13 [I 2022-10-10 16:49:11] ...trap/engines/engine.py.326: process: 0.32363 | load: 0.00020 [I 2022-10-10 16:49:14] ...trap/engines/engine.py.321: eval: epoch 78 | batch 120/510 [I 2022-10-10 16:49:14] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:05 [I 2022-10-10 16:49:14] ...trap/engines/engine.py.326: process: 0.31466 | load: 0.00020 [I 2022-10-10 16:49:17] ...trap/engines/engine.py.321: eval: epoch 78 | batch 130/510 [I 2022-10-10 16:49:17] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:02 [I 2022-10-10 16:49:17] ...trap/engines/engine.py.326: process: 0.32411 | load: 0.00021 [I 2022-10-10 16:49:20] ...trap/engines/engine.py.321: eval: epoch 78 | batch 140/510 [I 2022-10-10 16:49:20] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:58 [I 2022-10-10 16:49:20] ...trap/engines/engine.py.326: process: 0.32391 | load: 0.00020 [I 2022-10-10 16:49:24] ...trap/engines/engine.py.321: eval: epoch 78 | batch 150/510 [I 2022-10-10 16:49:24] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:55 [I 2022-10-10 16:49:24] ...trap/engines/engine.py.326: process: 0.32130 | load: 0.00022 [I 2022-10-10 16:49:27] ...trap/engines/engine.py.321: eval: epoch 78 | batch 160/510 [I 2022-10-10 16:49:27] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:53 [I 2022-10-10 16:49:27] ...trap/engines/engine.py.326: process: 0.32467 | load: 0.00021 [I 2022-10-10 16:49:30] ...trap/engines/engine.py.321: eval: epoch 78 | batch 170/510 [I 2022-10-10 16:49:30] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:49 [I 2022-10-10 16:49:30] ...trap/engines/engine.py.326: process: 0.32275 | load: 0.00020 [I 2022-10-10 16:49:33] ...trap/engines/engine.py.321: eval: epoch 78 | batch 180/510 [I 2022-10-10 16:49:33] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:46 [I 2022-10-10 16:49:33] ...trap/engines/engine.py.326: process: 0.32283 | load: 0.00020 [I 2022-10-10 16:49:37] ...trap/engines/engine.py.321: eval: epoch 78 | batch 190/510 [I 2022-10-10 16:49:37] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:43 [I 2022-10-10 16:49:37] ...trap/engines/engine.py.326: process: 0.31567 | load: 0.00020 [I 2022-10-10 16:49:40] ...trap/engines/engine.py.321: eval: epoch 78 | batch 200/510 [I 2022-10-10 16:49:40] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:40 [I 2022-10-10 16:49:40] ...trap/engines/engine.py.326: process: 0.32379 | load: 0.00022 [I 2022-10-10 16:49:43] ...trap/engines/engine.py.321: eval: epoch 78 | batch 210/510 [I 2022-10-10 16:49:43] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:37 [I 2022-10-10 16:49:43] ...trap/engines/engine.py.326: process: 0.32458 | load: 0.00021 [I 2022-10-10 16:49:46] ...trap/engines/engine.py.321: eval: epoch 78 | batch 220/510 [I 2022-10-10 16:49:46] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:33 [I 2022-10-10 16:49:46] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00021 [I 2022-10-10 16:49:50] ...trap/engines/engine.py.321: eval: epoch 78 | batch 230/510 [I 2022-10-10 16:49:50] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:31 [I 2022-10-10 16:49:50] ...trap/engines/engine.py.326: process: 0.32978 | load: 0.00021 [I 2022-10-10 16:49:53] ...trap/engines/engine.py.321: eval: epoch 78 | batch 240/510 [I 2022-10-10 16:49:53] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:27 [I 2022-10-10 16:49:53] ...trap/engines/engine.py.326: process: 0.32926 | load: 0.00024 [I 2022-10-10 16:49:56] ...trap/engines/engine.py.321: eval: epoch 78 | batch 250/510 [I 2022-10-10 16:49:56] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:24 [I 2022-10-10 16:49:56] ...trap/engines/engine.py.326: process: 0.32261 | load: 0.00021 [I 2022-10-10 16:49:59] ...trap/engines/engine.py.321: eval: epoch 78 | batch 260/510 [I 2022-10-10 16:49:59] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:21 [I 2022-10-10 16:49:59] ...trap/engines/engine.py.326: process: 0.32822 | load: 0.00021 [I 2022-10-10 16:50:03] ...trap/engines/engine.py.321: eval: epoch 78 | batch 270/510 [I 2022-10-10 16:50:03] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:18 [I 2022-10-10 16:50:03] ...trap/engines/engine.py.326: process: 0.32742 | load: 0.00022 [I 2022-10-10 16:50:06] ...trap/engines/engine.py.321: eval: epoch 78 | batch 280/510 [I 2022-10-10 16:50:06] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:14 [I 2022-10-10 16:50:06] ...trap/engines/engine.py.326: process: 0.32491 | load: 0.00021 [I 2022-10-10 16:50:09] ...trap/engines/engine.py.321: eval: epoch 78 | batch 290/510 [I 2022-10-10 16:50:09] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-10 16:50:09] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00023 [I 2022-10-10 16:50:12] ...trap/engines/engine.py.321: eval: epoch 78 | batch 300/510 [I 2022-10-10 16:50:12] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:08 [I 2022-10-10 16:50:12] ...trap/engines/engine.py.326: process: 0.32852 | load: 0.00021 [I 2022-10-10 16:50:16] ...trap/engines/engine.py.321: eval: epoch 78 | batch 310/510 [I 2022-10-10 16:50:16] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:04 [I 2022-10-10 16:50:16] ...trap/engines/engine.py.326: process: 0.32441 | load: 0.00022 [I 2022-10-10 16:50:19] ...trap/engines/engine.py.321: eval: epoch 78 | batch 320/510 [I 2022-10-10 16:50:19] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:01 [I 2022-10-10 16:50:19] ...trap/engines/engine.py.326: process: 0.32773 | load: 0.00022 [I 2022-10-10 16:50:22] ...trap/engines/engine.py.321: eval: epoch 78 | batch 330/510 [I 2022-10-10 16:50:22] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:58 [I 2022-10-10 16:50:22] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00021 [I 2022-10-10 16:50:25] ...trap/engines/engine.py.321: eval: epoch 78 | batch 340/510 [I 2022-10-10 16:50:25] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:55 [I 2022-10-10 16:50:25] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00021 [I 2022-10-10 16:50:29] ...trap/engines/engine.py.321: eval: epoch 78 | batch 350/510 [I 2022-10-10 16:50:29] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:52 [I 2022-10-10 16:50:29] ...trap/engines/engine.py.326: process: 0.32665 | load: 0.00024 [I 2022-10-10 16:50:32] ...trap/engines/engine.py.321: eval: epoch 78 | batch 360/510 [I 2022-10-10 16:50:32] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:48 [I 2022-10-10 16:50:32] ...trap/engines/engine.py.326: process: 0.32555 | load: 0.00021 [I 2022-10-10 16:50:35] ...trap/engines/engine.py.321: eval: epoch 78 | batch 370/510 [I 2022-10-10 16:50:35] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:45 [I 2022-10-10 16:50:35] ...trap/engines/engine.py.326: process: 0.32293 | load: 0.00024 [I 2022-10-10 16:50:38] ...trap/engines/engine.py.321: eval: epoch 78 | batch 380/510 [I 2022-10-10 16:50:38] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:42 [I 2022-10-10 16:50:38] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00022 [I 2022-10-10 16:50:42] ...trap/engines/engine.py.321: eval: epoch 78 | batch 390/510 [I 2022-10-10 16:50:42] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:39 [I 2022-10-10 16:50:42] ...trap/engines/engine.py.326: process: 0.32704 | load: 0.00021 [I 2022-10-10 16:50:45] ...trap/engines/engine.py.321: eval: epoch 78 | batch 400/510 [I 2022-10-10 16:50:45] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:35 [I 2022-10-10 16:50:45] ...trap/engines/engine.py.326: process: 0.32359 | load: 0.00022 [I 2022-10-10 16:50:48] ...trap/engines/engine.py.321: eval: epoch 78 | batch 410/510 [I 2022-10-10 16:50:48] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:32 [I 2022-10-10 16:50:48] ...trap/engines/engine.py.326: process: 0.32629 | load: 0.00021 [I 2022-10-10 16:50:51] ...trap/engines/engine.py.321: eval: epoch 78 | batch 420/510 [I 2022-10-10 16:50:51] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:29 [I 2022-10-10 16:50:51] ...trap/engines/engine.py.326: process: 0.32853 | load: 0.00026 [I 2022-10-10 16:50:55] ...trap/engines/engine.py.321: eval: epoch 78 | batch 430/510 [I 2022-10-10 16:50:55] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-10 16:50:55] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00021 [I 2022-10-10 16:50:58] ...trap/engines/engine.py.321: eval: epoch 78 | batch 440/510 [I 2022-10-10 16:50:58] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-10 16:50:58] ...trap/engines/engine.py.326: process: 0.31507 | load: 0.00022 [I 2022-10-10 16:51:01] ...trap/engines/engine.py.321: eval: epoch 78 | batch 450/510 [I 2022-10-10 16:51:01] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-10 16:51:01] ...trap/engines/engine.py.326: process: 0.32564 | load: 0.00022 [I 2022-10-10 16:51:04] ...trap/engines/engine.py.321: eval: epoch 78 | batch 460/510 [I 2022-10-10 16:51:04] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:16 [I 2022-10-10 16:51:04] ...trap/engines/engine.py.326: process: 0.32283 | load: 0.00021 [I 2022-10-10 16:51:08] ...trap/engines/engine.py.321: eval: epoch 78 | batch 470/510 [I 2022-10-10 16:51:08] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:13 [I 2022-10-10 16:51:08] ...trap/engines/engine.py.326: process: 0.32639 | load: 0.00021 [I 2022-10-10 16:51:11] ...trap/engines/engine.py.321: eval: epoch 78 | batch 480/510 [I 2022-10-10 16:51:11] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-10 16:51:11] ...trap/engines/engine.py.326: process: 0.33328 | load: 0.00026 [I 2022-10-10 16:51:14] ...trap/engines/engine.py.321: eval: epoch 78 | batch 490/510 [I 2022-10-10 16:51:14] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 16:51:14] ...trap/engines/engine.py.326: process: 0.32178 | load: 0.00023 [I 2022-10-10 16:51:17] ...trap/engines/engine.py.321: eval: epoch 78 | batch 500/510 [I 2022-10-10 16:51:17] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-10 16:51:17] ...trap/engines/engine.py.326: process: 0.32207 | load: 0.00021 [I 2022-10-10 16:51:21] ...trap/engines/engine.py.321: eval: epoch 78 | batch 510/510 [I 2022-10-10 16:51:21] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-10 16:51:21] ...trap/engines/engine.py.326: process: 0.31102 | load: 0.00017 [S 2022-10-10 16:51:21] ...trap/engines/engine.py.132: eval_epoch.epoch: 78 [I 2022-10-10 16:51:21] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740200 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908000 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934900 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724800 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906400 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933400 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014211 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006450 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003961 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010028 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.004587 [S 2022-10-10 16:51:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.003720 [I 2022-10-10 16:52:07] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 16:52:07] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 16:52:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 16:52:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 16:52:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 16:52:11] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 16:52:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 16:52:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 16:52:18] ...trap/engines/engine.py.163: Training model on trainset for epoch 79 [I 2022-10-10 16:52:18] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 16:52:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 0/2304 [I 2022-10-10 16:52:22] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:13:11 [I 2022-10-10 16:52:22] ...trap/engines/engine.py.229: process: 3.46834 | load: 2.57864 [I 2022-10-10 16:52:22] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-10 16:52:22] ...optimizers/trijoint.py.151: total_norm: 16.146635 [I 2022-10-10 16:52:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 10/2304 [I 2022-10-10 16:52:30] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:38 [I 2022-10-10 16:52:30] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00561 [I 2022-10-10 16:52:30] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-10 16:52:30] ...optimizers/trijoint.py.151: total_norm: 26.046238 [I 2022-10-10 16:52:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 20/2304 [I 2022-10-10 16:52:39] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:56 [I 2022-10-10 16:52:39] ...trap/engines/engine.py.229: process: 0.86949 | load: 0.00444 [I 2022-10-10 16:52:39] ...trap/engines/engine.py.230: loss: 0.10657 [I 2022-10-10 16:52:39] ...optimizers/trijoint.py.151: total_norm: 14.952903 [I 2022-10-10 16:52:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 30/2304 [I 2022-10-10 16:52:48] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:51 [I 2022-10-10 16:52:48] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00468 [I 2022-10-10 16:52:48] ...trap/engines/engine.py.230: loss: 0.19202 [I 2022-10-10 16:52:48] ...optimizers/trijoint.py.151: total_norm: 7.549118 [I 2022-10-10 16:52:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 40/2304 [I 2022-10-10 16:52:56] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:29 [I 2022-10-10 16:52:56] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00399 [I 2022-10-10 16:52:56] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-10 16:52:56] ...optimizers/trijoint.py.151: total_norm: 17.065207 [I 2022-10-10 16:53:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 50/2304 [I 2022-10-10 16:53:05] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:02 [I 2022-10-10 16:53:05] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00361 [I 2022-10-10 16:53:05] ...trap/engines/engine.py.230: loss: 0.15155 [I 2022-10-10 16:53:05] ...optimizers/trijoint.py.151: total_norm: 24.911982 [I 2022-10-10 16:53:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 60/2304 [I 2022-10-10 16:53:13] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:31:46 [I 2022-10-10 16:53:13] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00353 [I 2022-10-10 16:53:13] ...trap/engines/engine.py.230: loss: 0.15415 [I 2022-10-10 16:53:13] ...optimizers/trijoint.py.151: total_norm: 8.713064 [I 2022-10-10 16:53:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 70/2304 [I 2022-10-10 16:53:22] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:53 [I 2022-10-10 16:53:22] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00363 [I 2022-10-10 16:53:22] ...trap/engines/engine.py.230: loss: 0.21858 [I 2022-10-10 16:53:22] ...optimizers/trijoint.py.151: total_norm: 19.109133 [I 2022-10-10 16:53:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 80/2304 [I 2022-10-10 16:53:31] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:25 [I 2022-10-10 16:53:31] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00371 [I 2022-10-10 16:53:31] ...trap/engines/engine.py.230: loss: 0.11677 [I 2022-10-10 16:53:31] ...optimizers/trijoint.py.151: total_norm: 13.631449 [I 2022-10-10 16:53:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 90/2304 [I 2022-10-10 16:53:39] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:28 [I 2022-10-10 16:53:39] ...trap/engines/engine.py.229: process: 0.85664 | load: 0.00331 [I 2022-10-10 16:53:39] ...trap/engines/engine.py.230: loss: 0.13120 [I 2022-10-10 16:53:39] ...optimizers/trijoint.py.151: total_norm: 25.457146 [I 2022-10-10 16:53:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 100/2304 [I 2022-10-10 16:53:48] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:20 [I 2022-10-10 16:53:48] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00317 [I 2022-10-10 16:53:48] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-10 16:53:48] ...optimizers/trijoint.py.151: total_norm: 16.114214 [I 2022-10-10 16:53:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 110/2304 [I 2022-10-10 16:53:56] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:10 [I 2022-10-10 16:53:56] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00339 [I 2022-10-10 16:53:56] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-10 16:53:56] ...optimizers/trijoint.py.151: total_norm: 16.025404 [I 2022-10-10 16:54:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 120/2304 [I 2022-10-10 16:54:05] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:30:55 [I 2022-10-10 16:54:05] ...trap/engines/engine.py.229: process: 0.83201 | load: 0.00290 [I 2022-10-10 16:54:05] ...trap/engines/engine.py.230: loss: 0.18291 [I 2022-10-10 16:54:05] ...optimizers/trijoint.py.151: total_norm: 13.141909 [I 2022-10-10 16:54:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 130/2304 [I 2022-10-10 16:54:13] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:30:42 [I 2022-10-10 16:54:13] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00307 [I 2022-10-10 16:54:13] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-10 16:54:13] ...optimizers/trijoint.py.151: total_norm: 15.800689 [I 2022-10-10 16:54:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 140/2304 [I 2022-10-10 16:54:22] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:44 [I 2022-10-10 16:54:22] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00329 [I 2022-10-10 16:54:22] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-10 16:54:22] ...optimizers/trijoint.py.151: total_norm: 14.236268 [I 2022-10-10 16:54:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 150/2304 [I 2022-10-10 16:54:30] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:45 [I 2022-10-10 16:54:30] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00314 [I 2022-10-10 16:54:30] ...trap/engines/engine.py.230: loss: 0.20314 [I 2022-10-10 16:54:30] ...optimizers/trijoint.py.151: total_norm: 18.287777 [I 2022-10-10 16:54:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 160/2304 [I 2022-10-10 16:54:39] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:24 [I 2022-10-10 16:54:39] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00291 [I 2022-10-10 16:54:39] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-10 16:54:39] ...optimizers/trijoint.py.151: total_norm: 25.809381 [I 2022-10-10 16:54:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 170/2304 [I 2022-10-10 16:54:48] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:29 [I 2022-10-10 16:54:48] ...trap/engines/engine.py.229: process: 0.85949 | load: 0.00297 [I 2022-10-10 16:54:48] ...trap/engines/engine.py.230: loss: 0.08137 [I 2022-10-10 16:54:48] ...optimizers/trijoint.py.151: total_norm: 10.333730 [I 2022-10-10 16:54:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 180/2304 [I 2022-10-10 16:54:56] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:05 [I 2022-10-10 16:54:56] ...trap/engines/engine.py.229: process: 0.84860 | load: 0.00295 [I 2022-10-10 16:54:56] ...trap/engines/engine.py.230: loss: 0.12326 [I 2022-10-10 16:54:56] ...optimizers/trijoint.py.151: total_norm: 17.937355 [I 2022-10-10 16:55:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 190/2304 [I 2022-10-10 16:55:05] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:29:52 [I 2022-10-10 16:55:05] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00306 [I 2022-10-10 16:55:05] ...trap/engines/engine.py.230: loss: 0.10153 [I 2022-10-10 16:55:05] ...optimizers/trijoint.py.151: total_norm: 10.960163 [I 2022-10-10 16:55:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 200/2304 [I 2022-10-10 16:55:13] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:44 [I 2022-10-10 16:55:13] ...trap/engines/engine.py.229: process: 0.84116 | load: 0.00297 [I 2022-10-10 16:55:13] ...trap/engines/engine.py.230: loss: 0.12721 [I 2022-10-10 16:55:13] ...optimizers/trijoint.py.151: total_norm: 13.723357 [I 2022-10-10 16:55:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 210/2304 [I 2022-10-10 16:55:22] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:47 [I 2022-10-10 16:55:22] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00327 [I 2022-10-10 16:55:22] ...trap/engines/engine.py.230: loss: 0.09280 [I 2022-10-10 16:55:22] ...optimizers/trijoint.py.151: total_norm: 13.353682 [I 2022-10-10 16:55:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 220/2304 [I 2022-10-10 16:55:30] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:29 [I 2022-10-10 16:55:30] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00302 [I 2022-10-10 16:55:30] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-10 16:55:30] ...optimizers/trijoint.py.151: total_norm: 14.359418 [I 2022-10-10 16:55:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 230/2304 [I 2022-10-10 16:55:39] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:30 [I 2022-10-10 16:55:39] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00299 [I 2022-10-10 16:55:39] ...trap/engines/engine.py.230: loss: 0.11012 [I 2022-10-10 16:55:39] ...optimizers/trijoint.py.151: total_norm: 10.853946 [I 2022-10-10 16:55:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 240/2304 [I 2022-10-10 16:55:47] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:19 [I 2022-10-10 16:55:47] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00297 [I 2022-10-10 16:55:47] ...trap/engines/engine.py.230: loss: 0.12977 [I 2022-10-10 16:55:47] ...optimizers/trijoint.py.151: total_norm: 10.230668 [I 2022-10-10 16:55:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 250/2304 [I 2022-10-10 16:55:56] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:15 [I 2022-10-10 16:55:56] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00313 [I 2022-10-10 16:55:56] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-10 16:55:56] ...optimizers/trijoint.py.151: total_norm: 7.259639 [I 2022-10-10 16:56:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 260/2304 [I 2022-10-10 16:56:04] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:02 [I 2022-10-10 16:56:04] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00298 [I 2022-10-10 16:56:04] ...trap/engines/engine.py.230: loss: 0.09292 [I 2022-10-10 16:56:04] ...optimizers/trijoint.py.151: total_norm: 13.477341 [I 2022-10-10 16:56:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 270/2304 [I 2022-10-10 16:56:13] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:57 [I 2022-10-10 16:56:13] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00300 [I 2022-10-10 16:56:13] ...trap/engines/engine.py.230: loss: 0.12227 [I 2022-10-10 16:56:13] ...optimizers/trijoint.py.151: total_norm: 24.109610 [I 2022-10-10 16:56:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 280/2304 [I 2022-10-10 16:56:22] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:58 [I 2022-10-10 16:56:22] ...trap/engines/engine.py.229: process: 0.85853 | load: 0.00299 [I 2022-10-10 16:56:22] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-10 16:56:22] ...optimizers/trijoint.py.151: total_norm: 9.656073 [I 2022-10-10 16:56:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 290/2304 [I 2022-10-10 16:56:30] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:45 [I 2022-10-10 16:56:30] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00307 [I 2022-10-10 16:56:30] ...trap/engines/engine.py.230: loss: 0.12116 [I 2022-10-10 16:56:30] ...optimizers/trijoint.py.151: total_norm: 15.955653 [I 2022-10-10 16:56:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 300/2304 [I 2022-10-10 16:56:39] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:53 [I 2022-10-10 16:56:39] ...trap/engines/engine.py.229: process: 0.86402 | load: 0.00310 [I 2022-10-10 16:56:39] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-10 16:56:39] ...optimizers/trijoint.py.151: total_norm: 8.192979 [I 2022-10-10 16:56:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 310/2304 [I 2022-10-10 16:56:48] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:36 [I 2022-10-10 16:56:48] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00307 [I 2022-10-10 16:56:48] ...trap/engines/engine.py.230: loss: 0.13660 [I 2022-10-10 16:56:48] ...optimizers/trijoint.py.151: total_norm: 13.681876 [I 2022-10-10 16:56:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 320/2304 [I 2022-10-10 16:56:56] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:45 [I 2022-10-10 16:56:56] ...trap/engines/engine.py.229: process: 0.87847 | load: 0.00333 [I 2022-10-10 16:56:56] ...trap/engines/engine.py.230: loss: 0.14481 [I 2022-10-10 16:56:56] ...optimizers/trijoint.py.151: total_norm: 19.457132 [I 2022-10-10 16:57:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 330/2304 [I 2022-10-10 16:57:05] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:25 [I 2022-10-10 16:57:05] ...trap/engines/engine.py.229: process: 0.85858 | load: 0.00324 [I 2022-10-10 16:57:05] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-10 16:57:05] ...optimizers/trijoint.py.151: total_norm: 16.172249 [I 2022-10-10 16:57:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 340/2304 [I 2022-10-10 16:57:14] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:13 [I 2022-10-10 16:57:14] ...trap/engines/engine.py.229: process: 0.86610 | load: 0.00308 [I 2022-10-10 16:57:14] ...trap/engines/engine.py.230: loss: 0.11452 [I 2022-10-10 16:57:14] ...optimizers/trijoint.py.151: total_norm: 16.321131 [I 2022-10-10 16:57:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 350/2304 [I 2022-10-10 16:57:22] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:10 [I 2022-10-10 16:57:22] ...trap/engines/engine.py.229: process: 0.87082 | load: 0.00315 [I 2022-10-10 16:57:22] ...trap/engines/engine.py.230: loss: 0.11172 [I 2022-10-10 16:57:22] ...optimizers/trijoint.py.151: total_norm: 11.812820 [I 2022-10-10 16:57:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 360/2304 [I 2022-10-10 16:57:31] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:53 [I 2022-10-10 16:57:31] ...trap/engines/engine.py.229: process: 0.86609 | load: 0.00329 [I 2022-10-10 16:57:31] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-10 16:57:31] ...optimizers/trijoint.py.151: total_norm: 14.786190 [I 2022-10-10 16:57:40] ...trap/engines/engine.py.224: train: epoch 79 | batch 370/2304 [I 2022-10-10 16:57:40] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:48 [I 2022-10-10 16:57:40] ...trap/engines/engine.py.229: process: 0.86821 | load: 0.00312 [I 2022-10-10 16:57:40] ...trap/engines/engine.py.230: loss: 0.08625 [I 2022-10-10 16:57:40] ...optimizers/trijoint.py.151: total_norm: 13.901593 [I 2022-10-10 16:57:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 380/2304 [I 2022-10-10 16:57:48] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:45 [I 2022-10-10 16:57:48] ...trap/engines/engine.py.229: process: 0.86356 | load: 0.00310 [I 2022-10-10 16:57:48] ...trap/engines/engine.py.230: loss: 0.16580 [I 2022-10-10 16:57:48] ...optimizers/trijoint.py.151: total_norm: 17.993940 [I 2022-10-10 16:57:57] ...trap/engines/engine.py.224: train: epoch 79 | batch 390/2304 [I 2022-10-10 16:57:57] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:45 [I 2022-10-10 16:57:57] ...trap/engines/engine.py.229: process: 0.87514 | load: 0.00326 [I 2022-10-10 16:57:57] ...trap/engines/engine.py.230: loss: 0.09795 [I 2022-10-10 16:57:57] ...optimizers/trijoint.py.151: total_norm: 9.066553 [I 2022-10-10 16:58:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 400/2304 [I 2022-10-10 16:58:06] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:18 [I 2022-10-10 16:58:06] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00328 [I 2022-10-10 16:58:06] ...trap/engines/engine.py.230: loss: 0.12376 [I 2022-10-10 16:58:06] ...optimizers/trijoint.py.151: total_norm: 14.498377 [I 2022-10-10 16:58:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 410/2304 [I 2022-10-10 16:58:14] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:17 [I 2022-10-10 16:58:14] ...trap/engines/engine.py.229: process: 0.87983 | load: 0.00310 [I 2022-10-10 16:58:14] ...trap/engines/engine.py.230: loss: 0.18214 [I 2022-10-10 16:58:14] ...optimizers/trijoint.py.151: total_norm: 13.558777 [I 2022-10-10 16:58:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 420/2304 [I 2022-10-10 16:58:23] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:27:03 [I 2022-10-10 16:58:23] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00331 [I 2022-10-10 16:58:23] ...trap/engines/engine.py.230: loss: 0.14906 [I 2022-10-10 16:58:23] ...optimizers/trijoint.py.151: total_norm: 13.959642 [I 2022-10-10 16:58:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 430/2304 [I 2022-10-10 16:58:32] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:07 [I 2022-10-10 16:58:32] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00314 [I 2022-10-10 16:58:32] ...trap/engines/engine.py.230: loss: 0.10065 [I 2022-10-10 16:58:32] ...optimizers/trijoint.py.151: total_norm: 11.642232 [I 2022-10-10 16:58:40] ...trap/engines/engine.py.224: train: epoch 79 | batch 440/2304 [I 2022-10-10 16:58:40] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:45 [I 2022-10-10 16:58:40] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00315 [I 2022-10-10 16:58:40] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-10 16:58:40] ...optimizers/trijoint.py.151: total_norm: 7.512469 [I 2022-10-10 16:58:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 450/2304 [I 2022-10-10 16:58:49] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:39 [I 2022-10-10 16:58:49] ...trap/engines/engine.py.229: process: 0.86228 | load: 0.00307 [I 2022-10-10 16:58:49] ...trap/engines/engine.py.230: loss: 0.15053 [I 2022-10-10 16:58:49] ...optimizers/trijoint.py.151: total_norm: 11.178772 [I 2022-10-10 16:58:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 460/2304 [I 2022-10-10 16:58:58] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:35 [I 2022-10-10 16:58:58] ...trap/engines/engine.py.229: process: 0.87044 | load: 0.00323 [I 2022-10-10 16:58:58] ...trap/engines/engine.py.230: loss: 0.10383 [I 2022-10-10 16:58:58] ...optimizers/trijoint.py.151: total_norm: 12.946026 [I 2022-10-10 16:59:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 470/2304 [I 2022-10-10 16:59:06] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:26 [I 2022-10-10 16:59:06] ...trap/engines/engine.py.229: process: 0.87291 | load: 0.00316 [I 2022-10-10 16:59:06] ...trap/engines/engine.py.230: loss: 0.18693 [I 2022-10-10 16:59:06] ...optimizers/trijoint.py.151: total_norm: 14.616907 [I 2022-10-10 16:59:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 480/2304 [I 2022-10-10 16:59:15] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:22 [I 2022-10-10 16:59:15] ...trap/engines/engine.py.229: process: 0.88088 | load: 0.00341 [I 2022-10-10 16:59:15] ...trap/engines/engine.py.230: loss: 0.12718 [I 2022-10-10 16:59:15] ...optimizers/trijoint.py.151: total_norm: 21.667704 [I 2022-10-10 16:59:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 490/2304 [I 2022-10-10 16:59:24] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:08 [I 2022-10-10 16:59:24] ...trap/engines/engine.py.229: process: 0.87201 | load: 0.00315 [I 2022-10-10 16:59:24] ...trap/engines/engine.py.230: loss: 0.21003 [I 2022-10-10 16:59:24] ...optimizers/trijoint.py.151: total_norm: 22.191254 [I 2022-10-10 16:59:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 500/2304 [I 2022-10-10 16:59:32] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:26:01 [I 2022-10-10 16:59:32] ...trap/engines/engine.py.229: process: 0.87756 | load: 0.00315 [I 2022-10-10 16:59:32] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-10 16:59:32] ...optimizers/trijoint.py.151: total_norm: 13.556678 [I 2022-10-10 16:59:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 510/2304 [I 2022-10-10 16:59:41] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:49 [I 2022-10-10 16:59:41] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00325 [I 2022-10-10 16:59:41] ...trap/engines/engine.py.230: loss: 0.23019 [I 2022-10-10 16:59:41] ...optimizers/trijoint.py.151: total_norm: 28.241947 [I 2022-10-10 16:59:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 520/2304 [I 2022-10-10 16:59:50] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:43 [I 2022-10-10 16:59:50] ...trap/engines/engine.py.229: process: 0.87068 | load: 0.00308 [I 2022-10-10 16:59:50] ...trap/engines/engine.py.230: loss: 0.14973 [I 2022-10-10 16:59:50] ...optimizers/trijoint.py.151: total_norm: 17.480251 [I 2022-10-10 16:59:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 530/2304 [I 2022-10-10 16:59:58] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:32 [I 2022-10-10 16:59:58] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00311 [I 2022-10-10 16:59:58] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-10 16:59:58] ...optimizers/trijoint.py.151: total_norm: 23.171881 [I 2022-10-10 17:00:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 540/2304 [I 2022-10-10 17:00:07] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:23 [I 2022-10-10 17:00:07] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00312 [I 2022-10-10 17:00:07] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-10 17:00:07] ...optimizers/trijoint.py.151: total_norm: 18.681776 [I 2022-10-10 17:00:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 550/2304 [I 2022-10-10 17:00:16] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:09 [I 2022-10-10 17:00:16] ...trap/engines/engine.py.229: process: 0.86383 | load: 0.00310 [I 2022-10-10 17:00:16] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-10 17:00:16] ...optimizers/trijoint.py.151: total_norm: 10.830472 [I 2022-10-10 17:00:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 560/2304 [I 2022-10-10 17:00:24] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:00 [I 2022-10-10 17:00:24] ...trap/engines/engine.py.229: process: 0.84092 | load: 0.00306 [I 2022-10-10 17:00:24] ...trap/engines/engine.py.230: loss: 0.21937 [I 2022-10-10 17:00:24] ...optimizers/trijoint.py.151: total_norm: 14.867703 [I 2022-10-10 17:00:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 570/2304 [I 2022-10-10 17:00:33] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:49 [I 2022-10-10 17:00:33] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00298 [I 2022-10-10 17:00:33] ...trap/engines/engine.py.230: loss: 0.13305 [I 2022-10-10 17:00:33] ...optimizers/trijoint.py.151: total_norm: 12.612146 [I 2022-10-10 17:00:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 580/2304 [I 2022-10-10 17:00:41] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:37 [I 2022-10-10 17:00:41] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00301 [I 2022-10-10 17:00:41] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-10 17:00:41] ...optimizers/trijoint.py.151: total_norm: 13.774930 [I 2022-10-10 17:00:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 590/2304 [I 2022-10-10 17:00:50] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:25 [I 2022-10-10 17:00:50] ...trap/engines/engine.py.229: process: 0.86287 | load: 0.00310 [I 2022-10-10 17:00:50] ...trap/engines/engine.py.230: loss: 0.12463 [I 2022-10-10 17:00:50] ...optimizers/trijoint.py.151: total_norm: 13.483785 [I 2022-10-10 17:00:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 600/2304 [I 2022-10-10 17:00:58] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:12 [I 2022-10-10 17:00:58] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00295 [I 2022-10-10 17:00:58] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-10 17:00:58] ...optimizers/trijoint.py.151: total_norm: 8.591374 [I 2022-10-10 17:01:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 610/2304 [I 2022-10-10 17:01:07] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:03 [I 2022-10-10 17:01:07] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00294 [I 2022-10-10 17:01:07] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-10 17:01:07] ...optimizers/trijoint.py.151: total_norm: 36.530140 [I 2022-10-10 17:01:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 620/2304 [I 2022-10-10 17:01:16] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:55 [I 2022-10-10 17:01:16] ...trap/engines/engine.py.229: process: 0.84364 | load: 0.00298 [I 2022-10-10 17:01:16] ...trap/engines/engine.py.230: loss: 0.11033 [I 2022-10-10 17:01:16] ...optimizers/trijoint.py.151: total_norm: 13.515251 [I 2022-10-10 17:01:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 630/2304 [I 2022-10-10 17:01:24] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:41 [I 2022-10-10 17:01:24] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00294 [I 2022-10-10 17:01:24] ...trap/engines/engine.py.230: loss: 0.11109 [I 2022-10-10 17:01:24] ...optimizers/trijoint.py.151: total_norm: 9.880425 [I 2022-10-10 17:01:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 640/2304 [I 2022-10-10 17:01:33] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:37 [I 2022-10-10 17:01:33] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00298 [I 2022-10-10 17:01:33] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-10 17:01:33] ...optimizers/trijoint.py.151: total_norm: 12.699963 [I 2022-10-10 17:01:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 650/2304 [I 2022-10-10 17:01:41] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:29 [I 2022-10-10 17:01:41] ...trap/engines/engine.py.229: process: 0.86418 | load: 0.00294 [I 2022-10-10 17:01:41] ...trap/engines/engine.py.230: loss: 0.16066 [I 2022-10-10 17:01:41] ...optimizers/trijoint.py.151: total_norm: 12.165000 [I 2022-10-10 17:01:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 660/2304 [I 2022-10-10 17:01:50] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:17 [I 2022-10-10 17:01:50] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00294 [I 2022-10-10 17:01:50] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-10 17:01:50] ...optimizers/trijoint.py.151: total_norm: 17.898333 [I 2022-10-10 17:01:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 670/2304 [I 2022-10-10 17:01:58] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:11 [I 2022-10-10 17:01:58] ...trap/engines/engine.py.229: process: 0.85666 | load: 0.00297 [I 2022-10-10 17:01:58] ...trap/engines/engine.py.230: loss: 0.13950 [I 2022-10-10 17:01:58] ...optimizers/trijoint.py.151: total_norm: 16.771282 [I 2022-10-10 17:02:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 680/2304 [I 2022-10-10 17:02:07] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:23:12 [I 2022-10-10 17:02:07] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00315 [I 2022-10-10 17:02:07] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-10 17:02:07] ...optimizers/trijoint.py.151: total_norm: 28.792088 [I 2022-10-10 17:02:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 690/2304 [I 2022-10-10 17:02:15] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:23:00 [I 2022-10-10 17:02:15] ...trap/engines/engine.py.229: process: 0.85740 | load: 0.00299 [I 2022-10-10 17:02:15] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-10 17:02:15] ...optimizers/trijoint.py.151: total_norm: 26.975233 [I 2022-10-10 17:02:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 700/2304 [I 2022-10-10 17:02:24] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:52 [I 2022-10-10 17:02:24] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00294 [I 2022-10-10 17:02:24] ...trap/engines/engine.py.230: loss: 0.17105 [I 2022-10-10 17:02:24] ...optimizers/trijoint.py.151: total_norm: 13.131723 [I 2022-10-10 17:02:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 710/2304 [I 2022-10-10 17:02:33] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:41 [I 2022-10-10 17:02:33] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00298 [I 2022-10-10 17:02:33] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-10 17:02:33] ...optimizers/trijoint.py.151: total_norm: 16.786568 [I 2022-10-10 17:02:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 720/2304 [I 2022-10-10 17:02:41] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:37 [I 2022-10-10 17:02:41] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00295 [I 2022-10-10 17:02:41] ...trap/engines/engine.py.230: loss: 0.08660 [I 2022-10-10 17:02:41] ...optimizers/trijoint.py.151: total_norm: 7.928107 [I 2022-10-10 17:02:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 730/2304 [I 2022-10-10 17:02:50] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:20 [I 2022-10-10 17:02:50] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00292 [I 2022-10-10 17:02:50] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-10 17:02:50] ...optimizers/trijoint.py.151: total_norm: 9.305930 [I 2022-10-10 17:02:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 740/2304 [I 2022-10-10 17:02:58] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:22:14 [I 2022-10-10 17:02:58] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00294 [I 2022-10-10 17:02:58] ...trap/engines/engine.py.230: loss: 0.16305 [I 2022-10-10 17:02:58] ...optimizers/trijoint.py.151: total_norm: 43.301140 [I 2022-10-10 17:03:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 750/2304 [I 2022-10-10 17:03:07] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:22:03 [I 2022-10-10 17:03:07] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00296 [I 2022-10-10 17:03:07] ...trap/engines/engine.py.230: loss: 0.20469 [I 2022-10-10 17:03:07] ...optimizers/trijoint.py.151: total_norm: 26.175219 [I 2022-10-10 17:03:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 760/2304 [I 2022-10-10 17:03:15] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:22:02 [I 2022-10-10 17:03:15] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00293 [I 2022-10-10 17:03:15] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-10 17:03:15] ...optimizers/trijoint.py.151: total_norm: 16.709204 [I 2022-10-10 17:03:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 770/2304 [I 2022-10-10 17:03:24] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:45 [I 2022-10-10 17:03:24] ...trap/engines/engine.py.229: process: 0.83876 | load: 0.00291 [I 2022-10-10 17:03:24] ...trap/engines/engine.py.230: loss: 0.10108 [I 2022-10-10 17:03:24] ...optimizers/trijoint.py.151: total_norm: 12.829208 [I 2022-10-10 17:03:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 780/2304 [I 2022-10-10 17:03:33] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:44 [I 2022-10-10 17:03:33] ...trap/engines/engine.py.229: process: 0.85036 | load: 0.00292 [I 2022-10-10 17:03:33] ...trap/engines/engine.py.230: loss: 0.09755 [I 2022-10-10 17:03:33] ...optimizers/trijoint.py.151: total_norm: 11.541609 [I 2022-10-10 17:03:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 790/2304 [I 2022-10-10 17:03:41] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:26 [I 2022-10-10 17:03:41] ...trap/engines/engine.py.229: process: 0.84258 | load: 0.00292 [I 2022-10-10 17:03:41] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-10 17:03:41] ...optimizers/trijoint.py.151: total_norm: 12.071623 [I 2022-10-10 17:03:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 800/2304 [I 2022-10-10 17:03:50] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:16 [I 2022-10-10 17:03:50] ...trap/engines/engine.py.229: process: 0.84849 | load: 0.00307 [I 2022-10-10 17:03:50] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-10 17:03:50] ...optimizers/trijoint.py.151: total_norm: 21.813761 [I 2022-10-10 17:03:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 810/2304 [I 2022-10-10 17:03:58] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:19 [I 2022-10-10 17:03:58] ...trap/engines/engine.py.229: process: 0.88274 | load: 0.00295 [I 2022-10-10 17:03:58] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-10 17:03:58] ...optimizers/trijoint.py.151: total_norm: 12.169538 [I 2022-10-10 17:04:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 820/2304 [I 2022-10-10 17:04:07] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:21:07 [I 2022-10-10 17:04:07] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00299 [I 2022-10-10 17:04:07] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-10 17:04:07] ...optimizers/trijoint.py.151: total_norm: 12.787868 [I 2022-10-10 17:04:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 830/2304 [I 2022-10-10 17:04:15] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:21:09 [I 2022-10-10 17:04:15] ...trap/engines/engine.py.229: process: 0.87443 | load: 0.00307 [I 2022-10-10 17:04:15] ...trap/engines/engine.py.230: loss: 0.14280 [I 2022-10-10 17:04:15] ...optimizers/trijoint.py.151: total_norm: 18.891581 [I 2022-10-10 17:04:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 840/2304 [I 2022-10-10 17:04:24] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:21:01 [I 2022-10-10 17:04:24] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00310 [I 2022-10-10 17:04:24] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-10 17:04:24] ...optimizers/trijoint.py.151: total_norm: 14.727285 [I 2022-10-10 17:04:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 850/2304 [I 2022-10-10 17:04:33] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:21:05 [I 2022-10-10 17:04:33] ...trap/engines/engine.py.229: process: 0.88341 | load: 0.00314 [I 2022-10-10 17:04:33] ...trap/engines/engine.py.230: loss: 0.19694 [I 2022-10-10 17:04:33] ...optimizers/trijoint.py.151: total_norm: 18.693293 [I 2022-10-10 17:04:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 860/2304 [I 2022-10-10 17:04:41] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:50 [I 2022-10-10 17:04:41] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00307 [I 2022-10-10 17:04:41] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-10 17:04:41] ...optimizers/trijoint.py.151: total_norm: 16.149351 [I 2022-10-10 17:04:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 870/2304 [I 2022-10-10 17:04:50] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:20:43 [I 2022-10-10 17:04:50] ...trap/engines/engine.py.229: process: 0.86342 | load: 0.00312 [I 2022-10-10 17:04:50] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-10 17:04:50] ...optimizers/trijoint.py.151: total_norm: 13.264208 [I 2022-10-10 17:04:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 880/2304 [I 2022-10-10 17:04:59] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:24 [I 2022-10-10 17:04:59] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00316 [I 2022-10-10 17:04:59] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-10 17:04:59] ...optimizers/trijoint.py.151: total_norm: 11.368091 [I 2022-10-10 17:05:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 890/2304 [I 2022-10-10 17:05:07] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:27 [I 2022-10-10 17:05:07] ...trap/engines/engine.py.229: process: 0.88401 | load: 0.00309 [I 2022-10-10 17:05:07] ...trap/engines/engine.py.230: loss: 0.13211 [I 2022-10-10 17:05:07] ...optimizers/trijoint.py.151: total_norm: 13.583028 [I 2022-10-10 17:05:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 900/2304 [I 2022-10-10 17:05:16] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:20:01 [I 2022-10-10 17:05:16] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00313 [I 2022-10-10 17:05:16] ...trap/engines/engine.py.230: loss: 0.17805 [I 2022-10-10 17:05:16] ...optimizers/trijoint.py.151: total_norm: 21.497999 [I 2022-10-10 17:05:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 910/2304 [I 2022-10-10 17:05:24] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:51 [I 2022-10-10 17:05:24] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00307 [I 2022-10-10 17:05:24] ...trap/engines/engine.py.230: loss: 0.15122 [I 2022-10-10 17:05:24] ...optimizers/trijoint.py.151: total_norm: 16.985294 [I 2022-10-10 17:05:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 920/2304 [I 2022-10-10 17:05:33] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:47 [I 2022-10-10 17:05:33] ...trap/engines/engine.py.229: process: 0.85188 | load: 0.00313 [I 2022-10-10 17:05:33] ...trap/engines/engine.py.230: loss: 0.08832 [I 2022-10-10 17:05:33] ...optimizers/trijoint.py.151: total_norm: 9.212120 [I 2022-10-10 17:05:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 930/2304 [I 2022-10-10 17:05:42] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:45 [I 2022-10-10 17:05:42] ...trap/engines/engine.py.229: process: 0.87370 | load: 0.00310 [I 2022-10-10 17:05:42] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-10 17:05:42] ...optimizers/trijoint.py.151: total_norm: 15.803470 [I 2022-10-10 17:05:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 940/2304 [I 2022-10-10 17:05:50] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:41 [I 2022-10-10 17:05:50] ...trap/engines/engine.py.229: process: 0.86833 | load: 0.00312 [I 2022-10-10 17:05:50] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-10 17:05:50] ...optimizers/trijoint.py.151: total_norm: 14.002515 [I 2022-10-10 17:05:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 950/2304 [I 2022-10-10 17:05:59] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:26 [I 2022-10-10 17:05:59] ...trap/engines/engine.py.229: process: 0.86747 | load: 0.00313 [I 2022-10-10 17:05:59] ...trap/engines/engine.py.230: loss: 0.12090 [I 2022-10-10 17:05:59] ...optimizers/trijoint.py.151: total_norm: 13.712906 [I 2022-10-10 17:06:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 960/2304 [I 2022-10-10 17:06:08] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:22 [I 2022-10-10 17:06:08] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00315 [I 2022-10-10 17:06:08] ...trap/engines/engine.py.230: loss: 0.10955 [I 2022-10-10 17:06:08] ...optimizers/trijoint.py.151: total_norm: 16.314529 [I 2022-10-10 17:06:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 970/2304 [I 2022-10-10 17:06:16] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:12 [I 2022-10-10 17:06:16] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00312 [I 2022-10-10 17:06:16] ...trap/engines/engine.py.230: loss: 0.14539 [I 2022-10-10 17:06:16] ...optimizers/trijoint.py.151: total_norm: 11.127911 [I 2022-10-10 17:06:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 980/2304 [I 2022-10-10 17:06:25] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:19:09 [I 2022-10-10 17:06:25] ...trap/engines/engine.py.229: process: 0.87501 | load: 0.00321 [I 2022-10-10 17:06:25] ...trap/engines/engine.py.230: loss: 0.14148 [I 2022-10-10 17:06:25] ...optimizers/trijoint.py.151: total_norm: 22.795954 [I 2022-10-10 17:06:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 990/2304 [I 2022-10-10 17:06:34] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:53 [I 2022-10-10 17:06:34] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00310 [I 2022-10-10 17:06:34] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-10 17:06:34] ...optimizers/trijoint.py.151: total_norm: 15.060800 [I 2022-10-10 17:06:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 1000/2304 [I 2022-10-10 17:06:42] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:36 [I 2022-10-10 17:06:42] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00308 [I 2022-10-10 17:06:42] ...trap/engines/engine.py.230: loss: 0.12687 [I 2022-10-10 17:06:42] ...optimizers/trijoint.py.151: total_norm: 27.870684 [I 2022-10-10 17:06:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1010/2304 [I 2022-10-10 17:06:51] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:42 [I 2022-10-10 17:06:51] ...trap/engines/engine.py.229: process: 0.87229 | load: 0.00313 [I 2022-10-10 17:06:51] ...trap/engines/engine.py.230: loss: 0.09237 [I 2022-10-10 17:06:51] ...optimizers/trijoint.py.151: total_norm: 8.884964 [I 2022-10-10 17:07:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1020/2304 [I 2022-10-10 17:07:00] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:33 [I 2022-10-10 17:07:00] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00316 [I 2022-10-10 17:07:00] ...trap/engines/engine.py.230: loss: 0.10201 [I 2022-10-10 17:07:00] ...optimizers/trijoint.py.151: total_norm: 9.329103 [I 2022-10-10 17:07:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 1030/2304 [I 2022-10-10 17:07:08] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:15 [I 2022-10-10 17:07:08] ...trap/engines/engine.py.229: process: 0.85839 | load: 0.00313 [I 2022-10-10 17:07:08] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-10 17:07:08] ...optimizers/trijoint.py.151: total_norm: 13.204186 [I 2022-10-10 17:07:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1040/2304 [I 2022-10-10 17:07:17] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:09 [I 2022-10-10 17:07:17] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00316 [I 2022-10-10 17:07:17] ...trap/engines/engine.py.230: loss: 0.14578 [I 2022-10-10 17:07:17] ...optimizers/trijoint.py.151: total_norm: 24.917440 [I 2022-10-10 17:07:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 1050/2304 [I 2022-10-10 17:07:26] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:55 [I 2022-10-10 17:07:26] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00310 [I 2022-10-10 17:07:26] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-10 17:07:26] ...optimizers/trijoint.py.151: total_norm: 11.995885 [I 2022-10-10 17:07:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1060/2304 [I 2022-10-10 17:07:34] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:45 [I 2022-10-10 17:07:34] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00314 [I 2022-10-10 17:07:34] ...trap/engines/engine.py.230: loss: 0.13745 [I 2022-10-10 17:07:34] ...optimizers/trijoint.py.151: total_norm: 17.652246 [I 2022-10-10 17:07:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 1070/2304 [I 2022-10-10 17:07:43] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:44 [I 2022-10-10 17:07:43] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00310 [I 2022-10-10 17:07:43] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-10 17:07:43] ...optimizers/trijoint.py.151: total_norm: 21.102894 [I 2022-10-10 17:07:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1080/2304 [I 2022-10-10 17:07:51] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:38 [I 2022-10-10 17:07:51] ...trap/engines/engine.py.229: process: 0.87149 | load: 0.00313 [I 2022-10-10 17:07:51] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-10 17:07:51] ...optimizers/trijoint.py.151: total_norm: 12.695404 [I 2022-10-10 17:08:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1090/2304 [I 2022-10-10 17:08:00] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:22 [I 2022-10-10 17:08:00] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00396 [I 2022-10-10 17:08:00] ...trap/engines/engine.py.230: loss: 0.16240 [I 2022-10-10 17:08:00] ...optimizers/trijoint.py.151: total_norm: 16.134911 [I 2022-10-10 17:08:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1100/2304 [I 2022-10-10 17:08:09] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:17:17 [I 2022-10-10 17:08:09] ...trap/engines/engine.py.229: process: 0.85944 | load: 0.00310 [I 2022-10-10 17:08:09] ...trap/engines/engine.py.230: loss: 0.11482 [I 2022-10-10 17:08:09] ...optimizers/trijoint.py.151: total_norm: 14.113270 [I 2022-10-10 17:08:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1110/2304 [I 2022-10-10 17:08:17] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:59 [I 2022-10-10 17:08:17] ...trap/engines/engine.py.229: process: 0.83666 | load: 0.00295 [I 2022-10-10 17:08:17] ...trap/engines/engine.py.230: loss: 0.15242 [I 2022-10-10 17:08:17] ...optimizers/trijoint.py.151: total_norm: 18.908636 [I 2022-10-10 17:08:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 1120/2304 [I 2022-10-10 17:08:26] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:56 [I 2022-10-10 17:08:26] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00294 [I 2022-10-10 17:08:26] ...trap/engines/engine.py.230: loss: 0.18379 [I 2022-10-10 17:08:26] ...optimizers/trijoint.py.151: total_norm: 39.370121 [I 2022-10-10 17:08:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1130/2304 [I 2022-10-10 17:08:34] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:35 [I 2022-10-10 17:08:34] ...trap/engines/engine.py.229: process: 0.84301 | load: 0.00293 [I 2022-10-10 17:08:34] ...trap/engines/engine.py.230: loss: 0.11705 [I 2022-10-10 17:08:34] ...optimizers/trijoint.py.151: total_norm: 21.248631 [I 2022-10-10 17:08:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 1140/2304 [I 2022-10-10 17:08:43] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:33 [I 2022-10-10 17:08:43] ...trap/engines/engine.py.229: process: 0.85485 | load: 0.00296 [I 2022-10-10 17:08:43] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-10 17:08:43] ...optimizers/trijoint.py.151: total_norm: 9.440046 [I 2022-10-10 17:08:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1150/2304 [I 2022-10-10 17:08:51] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:22 [I 2022-10-10 17:08:51] ...trap/engines/engine.py.229: process: 0.85300 | load: 0.00298 [I 2022-10-10 17:08:51] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-10 17:08:51] ...optimizers/trijoint.py.151: total_norm: 47.807415 [I 2022-10-10 17:09:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1160/2304 [I 2022-10-10 17:09:00] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:14 [I 2022-10-10 17:09:00] ...trap/engines/engine.py.229: process: 0.85094 | load: 0.00298 [I 2022-10-10 17:09:00] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-10 17:09:00] ...optimizers/trijoint.py.151: total_norm: 11.344549 [I 2022-10-10 17:09:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1170/2304 [I 2022-10-10 17:09:09] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:10 [I 2022-10-10 17:09:09] ...trap/engines/engine.py.229: process: 0.86108 | load: 0.00294 [I 2022-10-10 17:09:09] ...trap/engines/engine.py.230: loss: 0.10191 [I 2022-10-10 17:09:09] ...optimizers/trijoint.py.151: total_norm: 15.439237 [I 2022-10-10 17:09:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1180/2304 [I 2022-10-10 17:09:17] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:53 [I 2022-10-10 17:09:17] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00298 [I 2022-10-10 17:09:17] ...trap/engines/engine.py.230: loss: 0.17742 [I 2022-10-10 17:09:17] ...optimizers/trijoint.py.151: total_norm: 30.213730 [I 2022-10-10 17:09:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 1190/2304 [I 2022-10-10 17:09:26] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:52 [I 2022-10-10 17:09:26] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00296 [I 2022-10-10 17:09:26] ...trap/engines/engine.py.230: loss: 0.12535 [I 2022-10-10 17:09:26] ...optimizers/trijoint.py.151: total_norm: 18.718994 [I 2022-10-10 17:09:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1200/2304 [I 2022-10-10 17:09:34] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:49 [I 2022-10-10 17:09:34] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00306 [I 2022-10-10 17:09:34] ...trap/engines/engine.py.230: loss: 0.16476 [I 2022-10-10 17:09:34] ...optimizers/trijoint.py.151: total_norm: 12.439179 [I 2022-10-10 17:09:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 1210/2304 [I 2022-10-10 17:09:43] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:35 [I 2022-10-10 17:09:43] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00300 [I 2022-10-10 17:09:43] ...trap/engines/engine.py.230: loss: 0.19333 [I 2022-10-10 17:09:43] ...optimizers/trijoint.py.151: total_norm: 25.654245 [I 2022-10-10 17:09:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1220/2304 [I 2022-10-10 17:09:51] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:25 [I 2022-10-10 17:09:51] ...trap/engines/engine.py.229: process: 0.86550 | load: 0.00307 [I 2022-10-10 17:09:51] ...trap/engines/engine.py.230: loss: 0.12802 [I 2022-10-10 17:09:51] ...optimizers/trijoint.py.151: total_norm: 17.546799 [I 2022-10-10 17:10:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1230/2304 [I 2022-10-10 17:10:00] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:15:12 [I 2022-10-10 17:10:00] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00297 [I 2022-10-10 17:10:00] ...trap/engines/engine.py.230: loss: 0.21702 [I 2022-10-10 17:10:00] ...optimizers/trijoint.py.151: total_norm: 6.954984 [I 2022-10-10 17:10:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1240/2304 [I 2022-10-10 17:10:09] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:03 [I 2022-10-10 17:10:09] ...trap/engines/engine.py.229: process: 0.85133 | load: 0.00294 [I 2022-10-10 17:10:09] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-10 17:10:09] ...optimizers/trijoint.py.151: total_norm: 12.176481 [I 2022-10-10 17:10:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1250/2304 [I 2022-10-10 17:10:17] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:58 [I 2022-10-10 17:10:17] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00299 [I 2022-10-10 17:10:17] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-10 17:10:17] ...optimizers/trijoint.py.151: total_norm: 19.741604 [I 2022-10-10 17:10:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 1260/2304 [I 2022-10-10 17:10:26] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:51 [I 2022-10-10 17:10:26] ...trap/engines/engine.py.229: process: 0.86414 | load: 0.00295 [I 2022-10-10 17:10:26] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-10 17:10:26] ...optimizers/trijoint.py.151: total_norm: 17.764145 [I 2022-10-10 17:10:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1270/2304 [I 2022-10-10 17:10:34] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:41 [I 2022-10-10 17:10:34] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00298 [I 2022-10-10 17:10:34] ...trap/engines/engine.py.230: loss: 0.19021 [I 2022-10-10 17:10:34] ...optimizers/trijoint.py.151: total_norm: 71.440125 [I 2022-10-10 17:10:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 1280/2304 [I 2022-10-10 17:10:43] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:36 [I 2022-10-10 17:10:43] ...trap/engines/engine.py.229: process: 0.87118 | load: 0.00297 [I 2022-10-10 17:10:43] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-10 17:10:43] ...optimizers/trijoint.py.151: total_norm: 18.913729 [I 2022-10-10 17:10:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1290/2304 [I 2022-10-10 17:10:51] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:27 [I 2022-10-10 17:10:51] ...trap/engines/engine.py.229: process: 0.86404 | load: 0.00300 [I 2022-10-10 17:10:51] ...trap/engines/engine.py.230: loss: 0.15440 [I 2022-10-10 17:10:51] ...optimizers/trijoint.py.151: total_norm: 11.518633 [I 2022-10-10 17:11:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1300/2304 [I 2022-10-10 17:11:00] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:25 [I 2022-10-10 17:11:00] ...trap/engines/engine.py.229: process: 0.86595 | load: 0.00320 [I 2022-10-10 17:11:00] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-10 17:11:00] ...optimizers/trijoint.py.151: total_norm: 20.524088 [I 2022-10-10 17:11:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1310/2304 [I 2022-10-10 17:11:09] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:14:22 [I 2022-10-10 17:11:09] ...trap/engines/engine.py.229: process: 0.87886 | load: 0.00318 [I 2022-10-10 17:11:09] ...trap/engines/engine.py.230: loss: 0.09153 [I 2022-10-10 17:11:09] ...optimizers/trijoint.py.151: total_norm: 8.611831 [I 2022-10-10 17:11:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1320/2304 [I 2022-10-10 17:11:17] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:14:09 [I 2022-10-10 17:11:17] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00317 [I 2022-10-10 17:11:17] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-10 17:11:17] ...optimizers/trijoint.py.151: total_norm: 13.226629 [I 2022-10-10 17:11:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 1330/2304 [I 2022-10-10 17:11:26] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:14:07 [I 2022-10-10 17:11:26] ...trap/engines/engine.py.229: process: 0.87056 | load: 0.00310 [I 2022-10-10 17:11:26] ...trap/engines/engine.py.230: loss: 0.15783 [I 2022-10-10 17:11:26] ...optimizers/trijoint.py.151: total_norm: 12.996414 [I 2022-10-10 17:11:35] ...trap/engines/engine.py.224: train: epoch 79 | batch 1340/2304 [I 2022-10-10 17:11:35] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:50 [I 2022-10-10 17:11:35] ...trap/engines/engine.py.229: process: 0.86154 | load: 0.00316 [I 2022-10-10 17:11:35] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-10 17:11:35] ...optimizers/trijoint.py.151: total_norm: 13.352132 [I 2022-10-10 17:11:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 1350/2304 [I 2022-10-10 17:11:43] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:13:46 [I 2022-10-10 17:11:43] ...trap/engines/engine.py.229: process: 0.87819 | load: 0.00308 [I 2022-10-10 17:11:43] ...trap/engines/engine.py.230: loss: 0.19745 [I 2022-10-10 17:11:43] ...optimizers/trijoint.py.151: total_norm: 22.390579 [I 2022-10-10 17:11:52] ...trap/engines/engine.py.224: train: epoch 79 | batch 1360/2304 [I 2022-10-10 17:11:52] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:13:32 [I 2022-10-10 17:11:52] ...trap/engines/engine.py.229: process: 0.85624 | load: 0.00310 [I 2022-10-10 17:11:52] ...trap/engines/engine.py.230: loss: 0.11555 [I 2022-10-10 17:11:52] ...optimizers/trijoint.py.151: total_norm: 18.248560 [I 2022-10-10 17:12:01] ...trap/engines/engine.py.224: train: epoch 79 | batch 1370/2304 [I 2022-10-10 17:12:01] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:31 [I 2022-10-10 17:12:01] ...trap/engines/engine.py.229: process: 0.87568 | load: 0.00309 [I 2022-10-10 17:12:01] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-10 17:12:01] ...optimizers/trijoint.py.151: total_norm: 31.948362 [I 2022-10-10 17:12:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1380/2304 [I 2022-10-10 17:12:09] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:13:18 [I 2022-10-10 17:12:09] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00324 [I 2022-10-10 17:12:09] ...trap/engines/engine.py.230: loss: 0.14551 [I 2022-10-10 17:12:09] ...optimizers/trijoint.py.151: total_norm: 17.686764 [I 2022-10-10 17:12:18] ...trap/engines/engine.py.224: train: epoch 79 | batch 1390/2304 [I 2022-10-10 17:12:18] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:13:16 [I 2022-10-10 17:12:18] ...trap/engines/engine.py.229: process: 0.87236 | load: 0.00324 [I 2022-10-10 17:12:18] ...trap/engines/engine.py.230: loss: 0.18314 [I 2022-10-10 17:12:18] ...optimizers/trijoint.py.151: total_norm: 17.806799 [I 2022-10-10 17:12:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 1400/2304 [I 2022-10-10 17:12:27] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:13:05 [I 2022-10-10 17:12:27] ...trap/engines/engine.py.229: process: 0.87210 | load: 0.00326 [I 2022-10-10 17:12:27] ...trap/engines/engine.py.230: loss: 0.10446 [I 2022-10-10 17:12:27] ...optimizers/trijoint.py.151: total_norm: 10.781105 [I 2022-10-10 17:12:35] ...trap/engines/engine.py.224: train: epoch 79 | batch 1410/2304 [I 2022-10-10 17:12:35] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:54 [I 2022-10-10 17:12:35] ...trap/engines/engine.py.229: process: 0.87302 | load: 0.00315 [I 2022-10-10 17:12:35] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-10 17:12:35] ...optimizers/trijoint.py.151: total_norm: 10.420785 [I 2022-10-10 17:12:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 1420/2304 [I 2022-10-10 17:12:44] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:48 [I 2022-10-10 17:12:44] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00323 [I 2022-10-10 17:12:44] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-10 17:12:44] ...optimizers/trijoint.py.151: total_norm: 19.743826 [I 2022-10-10 17:12:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 1430/2304 [I 2022-10-10 17:12:53] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:37 [I 2022-10-10 17:12:53] ...trap/engines/engine.py.229: process: 0.86403 | load: 0.00311 [I 2022-10-10 17:12:53] ...trap/engines/engine.py.230: loss: 0.18759 [I 2022-10-10 17:12:53] ...optimizers/trijoint.py.151: total_norm: 20.283640 [I 2022-10-10 17:13:01] ...trap/engines/engine.py.224: train: epoch 79 | batch 1440/2304 [I 2022-10-10 17:13:01] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:26 [I 2022-10-10 17:13:01] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00308 [I 2022-10-10 17:13:01] ...trap/engines/engine.py.230: loss: 0.12041 [I 2022-10-10 17:13:01] ...optimizers/trijoint.py.151: total_norm: 22.812489 [I 2022-10-10 17:13:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 1450/2304 [I 2022-10-10 17:13:10] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:12:17 [I 2022-10-10 17:13:10] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00310 [I 2022-10-10 17:13:10] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-10 17:13:10] ...optimizers/trijoint.py.151: total_norm: 69.883682 [I 2022-10-10 17:13:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 1460/2304 [I 2022-10-10 17:13:19] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:12:07 [I 2022-10-10 17:13:19] ...trap/engines/engine.py.229: process: 0.86469 | load: 0.00321 [I 2022-10-10 17:13:19] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-10 17:13:19] ...optimizers/trijoint.py.151: total_norm: 19.974136 [I 2022-10-10 17:13:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 1470/2304 [I 2022-10-10 17:13:27] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:12:06 [I 2022-10-10 17:13:27] ...trap/engines/engine.py.229: process: 0.88324 | load: 0.00315 [I 2022-10-10 17:13:27] ...trap/engines/engine.py.230: loss: 0.16513 [I 2022-10-10 17:13:27] ...optimizers/trijoint.py.151: total_norm: 14.642426 [I 2022-10-10 17:13:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 1480/2304 [I 2022-10-10 17:13:36] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:51 [I 2022-10-10 17:13:36] ...trap/engines/engine.py.229: process: 0.86212 | load: 0.00319 [I 2022-10-10 17:13:36] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-10 17:13:36] ...optimizers/trijoint.py.151: total_norm: 32.759762 [I 2022-10-10 17:13:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1490/2304 [I 2022-10-10 17:13:45] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:31 [I 2022-10-10 17:13:45] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00312 [I 2022-10-10 17:13:45] ...trap/engines/engine.py.230: loss: 0.12139 [I 2022-10-10 17:13:45] ...optimizers/trijoint.py.151: total_norm: 15.579736 [I 2022-10-10 17:13:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 1500/2304 [I 2022-10-10 17:13:54] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:12:09 [I 2022-10-10 17:13:54] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00306 [I 2022-10-10 17:13:54] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-10 17:13:54] ...optimizers/trijoint.py.151: total_norm: 11.906374 [I 2022-10-10 17:14:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 1510/2304 [I 2022-10-10 17:14:02] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:28 [I 2022-10-10 17:14:02] ...trap/engines/engine.py.229: process: 0.86851 | load: 0.00312 [I 2022-10-10 17:14:02] ...trap/engines/engine.py.230: loss: 0.07900 [I 2022-10-10 17:14:02] ...optimizers/trijoint.py.151: total_norm: 8.353057 [I 2022-10-10 17:14:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 1520/2304 [I 2022-10-10 17:14:11] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:19 [I 2022-10-10 17:14:11] ...trap/engines/engine.py.229: process: 0.87649 | load: 0.00311 [I 2022-10-10 17:14:11] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-10 17:14:11] ...optimizers/trijoint.py.151: total_norm: 11.277594 [I 2022-10-10 17:14:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 1530/2304 [I 2022-10-10 17:14:20] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:11:07 [I 2022-10-10 17:14:20] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00310 [I 2022-10-10 17:14:20] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-10 17:14:20] ...optimizers/trijoint.py.151: total_norm: 11.657539 [I 2022-10-10 17:14:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 1540/2304 [I 2022-10-10 17:14:28] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:11:00 [I 2022-10-10 17:14:28] ...trap/engines/engine.py.229: process: 0.86957 | load: 0.00324 [I 2022-10-10 17:14:28] ...trap/engines/engine.py.230: loss: 0.13979 [I 2022-10-10 17:14:28] ...optimizers/trijoint.py.151: total_norm: 37.108772 [I 2022-10-10 17:14:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1550/2304 [I 2022-10-10 17:14:37] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:47 [I 2022-10-10 17:14:37] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00318 [I 2022-10-10 17:14:37] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-10 17:14:37] ...optimizers/trijoint.py.151: total_norm: 12.072637 [I 2022-10-10 17:14:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1560/2304 [I 2022-10-10 17:14:45] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:38 [I 2022-10-10 17:14:45] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00314 [I 2022-10-10 17:14:45] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-10 17:14:45] ...optimizers/trijoint.py.151: total_norm: 16.026619 [I 2022-10-10 17:14:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 1570/2304 [I 2022-10-10 17:14:54] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:28 [I 2022-10-10 17:14:54] ...trap/engines/engine.py.229: process: 0.83643 | load: 0.00296 [I 2022-10-10 17:14:54] ...trap/engines/engine.py.230: loss: 0.16735 [I 2022-10-10 17:14:54] ...optimizers/trijoint.py.151: total_norm: 23.345882 [I 2022-10-10 17:15:03] ...trap/engines/engine.py.224: train: epoch 79 | batch 1580/2304 [I 2022-10-10 17:15:03] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:17 [I 2022-10-10 17:15:03] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00317 [I 2022-10-10 17:15:03] ...trap/engines/engine.py.230: loss: 0.08445 [I 2022-10-10 17:15:03] ...optimizers/trijoint.py.151: total_norm: 8.210324 [I 2022-10-10 17:15:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 1590/2304 [I 2022-10-10 17:15:11] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:10:09 [I 2022-10-10 17:15:11] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00290 [I 2022-10-10 17:15:11] ...trap/engines/engine.py.230: loss: 0.18246 [I 2022-10-10 17:15:11] ...optimizers/trijoint.py.151: total_norm: 30.823595 [I 2022-10-10 17:15:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 1600/2304 [I 2022-10-10 17:15:20] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:02 [I 2022-10-10 17:15:20] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00292 [I 2022-10-10 17:15:20] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-10 17:15:20] ...optimizers/trijoint.py.151: total_norm: 12.199810 [I 2022-10-10 17:15:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 1610/2304 [I 2022-10-10 17:15:28] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:50 [I 2022-10-10 17:15:28] ...trap/engines/engine.py.229: process: 0.83792 | load: 0.00293 [I 2022-10-10 17:15:28] ...trap/engines/engine.py.230: loss: 0.11691 [I 2022-10-10 17:15:28] ...optimizers/trijoint.py.151: total_norm: 6.524148 [I 2022-10-10 17:15:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1620/2304 [I 2022-10-10 17:15:37] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:39 [I 2022-10-10 17:15:37] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00298 [I 2022-10-10 17:15:37] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-10 17:15:37] ...optimizers/trijoint.py.151: total_norm: 18.112129 [I 2022-10-10 17:15:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1630/2304 [I 2022-10-10 17:15:45] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:34 [I 2022-10-10 17:15:45] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00294 [I 2022-10-10 17:15:45] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-10 17:15:45] ...optimizers/trijoint.py.151: total_norm: 21.976665 [I 2022-10-10 17:15:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 1640/2304 [I 2022-10-10 17:15:54] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:30 [I 2022-10-10 17:15:54] ...trap/engines/engine.py.229: process: 0.86455 | load: 0.00331 [I 2022-10-10 17:15:54] ...trap/engines/engine.py.230: loss: 0.14493 [I 2022-10-10 17:15:54] ...optimizers/trijoint.py.151: total_norm: 19.221090 [I 2022-10-10 17:16:03] ...trap/engines/engine.py.224: train: epoch 79 | batch 1650/2304 [I 2022-10-10 17:16:03] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:22 [I 2022-10-10 17:16:03] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00299 [I 2022-10-10 17:16:03] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-10 17:16:03] ...optimizers/trijoint.py.151: total_norm: 11.993522 [I 2022-10-10 17:16:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 1660/2304 [I 2022-10-10 17:16:11] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:09:10 [I 2022-10-10 17:16:11] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00297 [I 2022-10-10 17:16:11] ...trap/engines/engine.py.230: loss: 0.09840 [I 2022-10-10 17:16:11] ...optimizers/trijoint.py.151: total_norm: 24.093466 [I 2022-10-10 17:16:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 1670/2304 [I 2022-10-10 17:16:20] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:09:00 [I 2022-10-10 17:16:20] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00294 [I 2022-10-10 17:16:20] ...trap/engines/engine.py.230: loss: 0.23130 [I 2022-10-10 17:16:20] ...optimizers/trijoint.py.151: total_norm: 15.944434 [I 2022-10-10 17:16:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 1680/2304 [I 2022-10-10 17:16:28] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:08:50 [I 2022-10-10 17:16:28] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00300 [I 2022-10-10 17:16:28] ...trap/engines/engine.py.230: loss: 0.11138 [I 2022-10-10 17:16:28] ...optimizers/trijoint.py.151: total_norm: 14.611149 [I 2022-10-10 17:16:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1690/2304 [I 2022-10-10 17:16:37] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:42 [I 2022-10-10 17:16:37] ...trap/engines/engine.py.229: process: 0.84382 | load: 0.00295 [I 2022-10-10 17:16:37] ...trap/engines/engine.py.230: loss: 0.24190 [I 2022-10-10 17:16:37] ...optimizers/trijoint.py.151: total_norm: 38.091000 [I 2022-10-10 17:16:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1700/2304 [I 2022-10-10 17:16:45] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:32 [I 2022-10-10 17:16:45] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00304 [I 2022-10-10 17:16:45] ...trap/engines/engine.py.230: loss: 0.13797 [I 2022-10-10 17:16:45] ...optimizers/trijoint.py.151: total_norm: 31.654934 [I 2022-10-10 17:16:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 1710/2304 [I 2022-10-10 17:16:54] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:25 [I 2022-10-10 17:16:54] ...trap/engines/engine.py.229: process: 0.83589 | load: 0.00296 [I 2022-10-10 17:16:54] ...trap/engines/engine.py.230: loss: 0.12662 [I 2022-10-10 17:16:54] ...optimizers/trijoint.py.151: total_norm: 20.659834 [I 2022-10-10 17:17:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 1720/2304 [I 2022-10-10 17:17:02] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:15 [I 2022-10-10 17:17:02] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00297 [I 2022-10-10 17:17:02] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-10 17:17:02] ...optimizers/trijoint.py.151: total_norm: 11.954146 [I 2022-10-10 17:17:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 1730/2304 [I 2022-10-10 17:17:11] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:08:11 [I 2022-10-10 17:17:11] ...trap/engines/engine.py.229: process: 0.86160 | load: 0.00304 [I 2022-10-10 17:17:11] ...trap/engines/engine.py.230: loss: 0.17165 [I 2022-10-10 17:17:11] ...optimizers/trijoint.py.151: total_norm: 10.762572 [I 2022-10-10 17:17:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 1740/2304 [I 2022-10-10 17:17:19] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:59 [I 2022-10-10 17:17:19] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00300 [I 2022-10-10 17:17:19] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-10 17:17:19] ...optimizers/trijoint.py.151: total_norm: 12.945810 [I 2022-10-10 17:17:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 1750/2304 [I 2022-10-10 17:17:28] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:54 [I 2022-10-10 17:17:28] ...trap/engines/engine.py.229: process: 0.86332 | load: 0.00304 [I 2022-10-10 17:17:28] ...trap/engines/engine.py.230: loss: 0.13105 [I 2022-10-10 17:17:28] ...optimizers/trijoint.py.151: total_norm: 21.975382 [I 2022-10-10 17:17:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1760/2304 [I 2022-10-10 17:17:37] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:45 [I 2022-10-10 17:17:37] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00308 [I 2022-10-10 17:17:37] ...trap/engines/engine.py.230: loss: 0.21605 [I 2022-10-10 17:17:37] ...optimizers/trijoint.py.151: total_norm: 18.279337 [I 2022-10-10 17:17:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1770/2304 [I 2022-10-10 17:17:45] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:43 [I 2022-10-10 17:17:45] ...trap/engines/engine.py.229: process: 0.89486 | load: 0.00320 [I 2022-10-10 17:17:45] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-10 17:17:45] ...optimizers/trijoint.py.151: total_norm: 11.417753 [I 2022-10-10 17:17:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 1780/2304 [I 2022-10-10 17:17:54] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:37 [I 2022-10-10 17:17:54] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00322 [I 2022-10-10 17:17:54] ...trap/engines/engine.py.230: loss: 0.18043 [I 2022-10-10 17:17:54] ...optimizers/trijoint.py.151: total_norm: 15.799268 [I 2022-10-10 17:18:03] ...trap/engines/engine.py.224: train: epoch 79 | batch 1790/2304 [I 2022-10-10 17:18:03] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:21 [I 2022-10-10 17:18:03] ...trap/engines/engine.py.229: process: 0.85148 | load: 0.00314 [I 2022-10-10 17:18:03] ...trap/engines/engine.py.230: loss: 0.08007 [I 2022-10-10 17:18:03] ...optimizers/trijoint.py.151: total_norm: 17.665785 [I 2022-10-10 17:18:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 1800/2304 [I 2022-10-10 17:18:11] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:07:18 [I 2022-10-10 17:18:11] ...trap/engines/engine.py.229: process: 0.87285 | load: 0.00325 [I 2022-10-10 17:18:11] ...trap/engines/engine.py.230: loss: 0.11657 [I 2022-10-10 17:18:11] ...optimizers/trijoint.py.151: total_norm: 16.828041 [I 2022-10-10 17:18:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 1810/2304 [I 2022-10-10 17:18:20] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:07:07 [I 2022-10-10 17:18:20] ...trap/engines/engine.py.229: process: 0.87190 | load: 0.00317 [I 2022-10-10 17:18:20] ...trap/engines/engine.py.230: loss: 0.10727 [I 2022-10-10 17:18:20] ...optimizers/trijoint.py.151: total_norm: 12.467317 [I 2022-10-10 17:18:29] ...trap/engines/engine.py.224: train: epoch 79 | batch 1820/2304 [I 2022-10-10 17:18:29] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:59 [I 2022-10-10 17:18:29] ...trap/engines/engine.py.229: process: 0.87267 | load: 0.00326 [I 2022-10-10 17:18:29] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-10 17:18:29] ...optimizers/trijoint.py.151: total_norm: 12.801197 [I 2022-10-10 17:18:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1830/2304 [I 2022-10-10 17:18:37] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:51 [I 2022-10-10 17:18:37] ...trap/engines/engine.py.229: process: 0.87658 | load: 0.00318 [I 2022-10-10 17:18:37] ...trap/engines/engine.py.230: loss: 0.09726 [I 2022-10-10 17:18:37] ...optimizers/trijoint.py.151: total_norm: 14.975381 [I 2022-10-10 17:18:46] ...trap/engines/engine.py.224: train: epoch 79 | batch 1840/2304 [I 2022-10-10 17:18:46] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:44 [I 2022-10-10 17:18:46] ...trap/engines/engine.py.229: process: 0.86959 | load: 0.00324 [I 2022-10-10 17:18:46] ...trap/engines/engine.py.230: loss: 0.14499 [I 2022-10-10 17:18:46] ...optimizers/trijoint.py.151: total_norm: 16.924864 [I 2022-10-10 17:18:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 1850/2304 [I 2022-10-10 17:18:55] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:33 [I 2022-10-10 17:18:55] ...trap/engines/engine.py.229: process: 0.86742 | load: 0.00325 [I 2022-10-10 17:18:55] ...trap/engines/engine.py.230: loss: 0.16638 [I 2022-10-10 17:18:55] ...optimizers/trijoint.py.151: total_norm: 16.692411 [I 2022-10-10 17:19:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 1860/2304 [I 2022-10-10 17:19:04] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:27 [I 2022-10-10 17:19:04] ...trap/engines/engine.py.229: process: 0.87959 | load: 0.00324 [I 2022-10-10 17:19:04] ...trap/engines/engine.py.230: loss: 0.12411 [I 2022-10-10 17:19:04] ...optimizers/trijoint.py.151: total_norm: 31.271704 [I 2022-10-10 17:19:12] ...trap/engines/engine.py.224: train: epoch 79 | batch 1870/2304 [I 2022-10-10 17:19:12] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:18 [I 2022-10-10 17:19:12] ...trap/engines/engine.py.229: process: 0.88726 | load: 0.00320 [I 2022-10-10 17:19:12] ...trap/engines/engine.py.230: loss: 0.10238 [I 2022-10-10 17:19:12] ...optimizers/trijoint.py.151: total_norm: 11.720554 [I 2022-10-10 17:19:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 1880/2304 [I 2022-10-10 17:19:21] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:06:05 [I 2022-10-10 17:19:21] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00324 [I 2022-10-10 17:19:21] ...trap/engines/engine.py.230: loss: 0.13915 [I 2022-10-10 17:19:21] ...optimizers/trijoint.py.151: total_norm: 28.706995 [I 2022-10-10 17:19:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 1890/2304 [I 2022-10-10 17:19:30] ...trap/engines/engine.py.228: elapsed: 0:27:11 | left: 0:05:58 [I 2022-10-10 17:19:30] ...trap/engines/engine.py.229: process: 0.86775 | load: 0.00317 [I 2022-10-10 17:19:30] ...trap/engines/engine.py.230: loss: 0.10542 [I 2022-10-10 17:19:30] ...optimizers/trijoint.py.151: total_norm: 11.140821 [I 2022-10-10 17:19:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 1900/2304 [I 2022-10-10 17:19:38] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:47 [I 2022-10-10 17:19:38] ...trap/engines/engine.py.229: process: 0.85215 | load: 0.00312 [I 2022-10-10 17:19:38] ...trap/engines/engine.py.230: loss: 0.11608 [I 2022-10-10 17:19:38] ...optimizers/trijoint.py.151: total_norm: 7.040861 [I 2022-10-10 17:19:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 1910/2304 [I 2022-10-10 17:19:47] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:38 [I 2022-10-10 17:19:47] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00318 [I 2022-10-10 17:19:47] ...trap/engines/engine.py.230: loss: 0.12030 [I 2022-10-10 17:19:47] ...optimizers/trijoint.py.151: total_norm: 15.458323 [I 2022-10-10 17:19:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 1920/2304 [I 2022-10-10 17:19:56] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:33 [I 2022-10-10 17:19:56] ...trap/engines/engine.py.229: process: 0.87525 | load: 0.00311 [I 2022-10-10 17:19:56] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-10 17:19:56] ...optimizers/trijoint.py.151: total_norm: 21.302906 [I 2022-10-10 17:20:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 1930/2304 [I 2022-10-10 17:20:04] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:05:24 [I 2022-10-10 17:20:04] ...trap/engines/engine.py.229: process: 0.86837 | load: 0.00322 [I 2022-10-10 17:20:04] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-10 17:20:04] ...optimizers/trijoint.py.151: total_norm: 52.174618 [I 2022-10-10 17:20:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 1940/2304 [I 2022-10-10 17:20:13] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:05:13 [I 2022-10-10 17:20:13] ...trap/engines/engine.py.229: process: 0.86507 | load: 0.00314 [I 2022-10-10 17:20:13] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-10 17:20:13] ...optimizers/trijoint.py.151: total_norm: 9.245671 [I 2022-10-10 17:20:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 1950/2304 [I 2022-10-10 17:20:22] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:05:07 [I 2022-10-10 17:20:22] ...trap/engines/engine.py.229: process: 0.87186 | load: 0.00314 [I 2022-10-10 17:20:22] ...trap/engines/engine.py.230: loss: 0.12990 [I 2022-10-10 17:20:22] ...optimizers/trijoint.py.151: total_norm: 17.050131 [I 2022-10-10 17:20:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 1960/2304 [I 2022-10-10 17:20:30] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:56 [I 2022-10-10 17:20:30] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00319 [I 2022-10-10 17:20:30] ...trap/engines/engine.py.230: loss: 0.08974 [I 2022-10-10 17:20:30] ...optimizers/trijoint.py.151: total_norm: 12.648809 [I 2022-10-10 17:20:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 1970/2304 [I 2022-10-10 17:20:39] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:44 [I 2022-10-10 17:20:39] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00316 [I 2022-10-10 17:20:39] ...trap/engines/engine.py.230: loss: 0.09046 [I 2022-10-10 17:20:39] ...optimizers/trijoint.py.151: total_norm: 8.941805 [I 2022-10-10 17:20:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 1980/2304 [I 2022-10-10 17:20:47] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:40 [I 2022-10-10 17:20:47] ...trap/engines/engine.py.229: process: 0.88206 | load: 0.00317 [I 2022-10-10 17:20:47] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-10 17:20:47] ...optimizers/trijoint.py.151: total_norm: 13.365983 [I 2022-10-10 17:20:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 1990/2304 [I 2022-10-10 17:20:56] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:30 [I 2022-10-10 17:20:56] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00318 [I 2022-10-10 17:20:56] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-10 17:20:56] ...optimizers/trijoint.py.151: total_norm: 12.132651 [I 2022-10-10 17:21:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 2000/2304 [I 2022-10-10 17:21:05] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:23 [I 2022-10-10 17:21:05] ...trap/engines/engine.py.229: process: 0.86734 | load: 0.00327 [I 2022-10-10 17:21:05] ...trap/engines/engine.py.230: loss: 0.18329 [I 2022-10-10 17:21:05] ...optimizers/trijoint.py.151: total_norm: 31.756655 [I 2022-10-10 17:21:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 2010/2304 [I 2022-10-10 17:21:14] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:04:14 [I 2022-10-10 17:21:14] ...trap/engines/engine.py.229: process: 0.86062 | load: 0.00329 [I 2022-10-10 17:21:14] ...trap/engines/engine.py.230: loss: 0.12719 [I 2022-10-10 17:21:14] ...optimizers/trijoint.py.151: total_norm: 18.497887 [I 2022-10-10 17:21:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 2020/2304 [I 2022-10-10 17:21:22] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:04:04 [I 2022-10-10 17:21:22] ...trap/engines/engine.py.229: process: 0.87092 | load: 0.00322 [I 2022-10-10 17:21:22] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-10 17:21:22] ...optimizers/trijoint.py.151: total_norm: 23.930809 [I 2022-10-10 17:21:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 2030/2304 [I 2022-10-10 17:21:31] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:58 [I 2022-10-10 17:21:31] ...trap/engines/engine.py.229: process: 0.88070 | load: 0.00321 [I 2022-10-10 17:21:31] ...trap/engines/engine.py.230: loss: 0.21127 [I 2022-10-10 17:21:31] ...optimizers/trijoint.py.151: total_norm: 18.803596 [I 2022-10-10 17:21:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 2040/2304 [I 2022-10-10 17:21:39] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:44 [I 2022-10-10 17:21:39] ...trap/engines/engine.py.229: process: 0.82841 | load: 0.00305 [I 2022-10-10 17:21:39] ...trap/engines/engine.py.230: loss: 0.11544 [I 2022-10-10 17:21:39] ...optimizers/trijoint.py.151: total_norm: 15.968928 [I 2022-10-10 17:21:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 2050/2304 [I 2022-10-10 17:21:48] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:37 [I 2022-10-10 17:21:48] ...trap/engines/engine.py.229: process: 0.85710 | load: 0.00313 [I 2022-10-10 17:21:48] ...trap/engines/engine.py.230: loss: 0.11963 [I 2022-10-10 17:21:48] ...optimizers/trijoint.py.151: total_norm: 41.117229 [I 2022-10-10 17:21:57] ...trap/engines/engine.py.224: train: epoch 79 | batch 2060/2304 [I 2022-10-10 17:21:57] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:27 [I 2022-10-10 17:21:57] ...trap/engines/engine.py.229: process: 0.82953 | load: 0.00303 [I 2022-10-10 17:21:57] ...trap/engines/engine.py.230: loss: 0.18040 [I 2022-10-10 17:21:57] ...optimizers/trijoint.py.151: total_norm: 17.114420 [I 2022-10-10 17:22:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 2070/2304 [I 2022-10-10 17:22:24] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:05:18 [I 2022-10-10 17:22:24] ...trap/engines/engine.py.229: process: 0.85832 | load: 0.00293 [I 2022-10-10 17:22:24] ...trap/engines/engine.py.230: loss: 0.21331 [I 2022-10-10 17:22:24] ...optimizers/trijoint.py.151: total_norm: 16.318825 [I 2022-10-10 17:22:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 2080/2304 [I 2022-10-10 17:22:33] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:03:22 [I 2022-10-10 17:22:33] ...trap/engines/engine.py.229: process: 0.85435 | load: 0.00292 [I 2022-10-10 17:22:33] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-10 17:22:33] ...optimizers/trijoint.py.151: total_norm: 21.487083 [I 2022-10-10 17:22:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 2090/2304 [I 2022-10-10 17:22:41] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:03:02 [I 2022-10-10 17:22:41] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00291 [I 2022-10-10 17:22:41] ...trap/engines/engine.py.230: loss: 0.10662 [I 2022-10-10 17:22:41] ...optimizers/trijoint.py.151: total_norm: 11.460229 [I 2022-10-10 17:22:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 2100/2304 [I 2022-10-10 17:22:50] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:54 [I 2022-10-10 17:22:50] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00293 [I 2022-10-10 17:22:50] ...trap/engines/engine.py.230: loss: 0.11934 [I 2022-10-10 17:22:50] ...optimizers/trijoint.py.151: total_norm: 20.886538 [I 2022-10-10 17:22:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 2110/2304 [I 2022-10-10 17:22:58] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:45 [I 2022-10-10 17:22:58] ...trap/engines/engine.py.229: process: 0.85250 | load: 0.00290 [I 2022-10-10 17:22:58] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-10 17:22:58] ...optimizers/trijoint.py.151: total_norm: 23.217289 [I 2022-10-10 17:23:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 2120/2304 [I 2022-10-10 17:23:07] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:35 [I 2022-10-10 17:23:07] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00291 [I 2022-10-10 17:23:07] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-10 17:23:07] ...optimizers/trijoint.py.151: total_norm: 9.788410 [I 2022-10-10 17:23:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 2130/2304 [I 2022-10-10 17:23:15] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:26 [I 2022-10-10 17:23:15] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00296 [I 2022-10-10 17:23:15] ...trap/engines/engine.py.230: loss: 0.16819 [I 2022-10-10 17:23:15] ...optimizers/trijoint.py.151: total_norm: 7.419736 [I 2022-10-10 17:23:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 2140/2304 [I 2022-10-10 17:23:24] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:19 [I 2022-10-10 17:23:24] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00293 [I 2022-10-10 17:23:24] ...trap/engines/engine.py.230: loss: 0.13012 [I 2022-10-10 17:23:24] ...optimizers/trijoint.py.151: total_norm: 8.980910 [I 2022-10-10 17:23:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 2150/2304 [I 2022-10-10 17:23:32] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:02:10 [I 2022-10-10 17:23:32] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00308 [I 2022-10-10 17:23:32] ...trap/engines/engine.py.230: loss: 0.16303 [I 2022-10-10 17:23:32] ...optimizers/trijoint.py.151: total_norm: 22.110743 [I 2022-10-10 17:23:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 2160/2304 [I 2022-10-10 17:23:41] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:02:03 [I 2022-10-10 17:23:41] ...trap/engines/engine.py.229: process: 0.85355 | load: 0.00305 [I 2022-10-10 17:23:41] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-10 17:23:41] ...optimizers/trijoint.py.151: total_norm: 14.895559 [I 2022-10-10 17:23:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 2170/2304 [I 2022-10-10 17:23:49] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:53 [I 2022-10-10 17:23:49] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00303 [I 2022-10-10 17:23:49] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-10 17:23:49] ...optimizers/trijoint.py.151: total_norm: 15.188154 [I 2022-10-10 17:23:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 2180/2304 [I 2022-10-10 17:23:58] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:45 [I 2022-10-10 17:23:58] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00297 [I 2022-10-10 17:23:58] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-10 17:23:58] ...optimizers/trijoint.py.151: total_norm: 12.585566 [I 2022-10-10 17:24:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 2190/2304 [I 2022-10-10 17:24:07] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:37 [I 2022-10-10 17:24:07] ...trap/engines/engine.py.229: process: 0.86302 | load: 0.00295 [I 2022-10-10 17:24:07] ...trap/engines/engine.py.230: loss: 0.09438 [I 2022-10-10 17:24:07] ...optimizers/trijoint.py.151: total_norm: 16.269468 [I 2022-10-10 17:24:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 2200/2304 [I 2022-10-10 17:24:15] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:01:28 [I 2022-10-10 17:24:15] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00293 [I 2022-10-10 17:24:15] ...trap/engines/engine.py.230: loss: 0.11318 [I 2022-10-10 17:24:15] ...optimizers/trijoint.py.151: total_norm: 14.154038 [I 2022-10-10 17:24:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 2210/2304 [I 2022-10-10 17:24:24] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:19 [I 2022-10-10 17:24:24] ...trap/engines/engine.py.229: process: 0.85187 | load: 0.00295 [I 2022-10-10 17:24:24] ...trap/engines/engine.py.230: loss: 0.11455 [I 2022-10-10 17:24:24] ...optimizers/trijoint.py.151: total_norm: 13.251595 [I 2022-10-10 17:24:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 2220/2304 [I 2022-10-10 17:24:32] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:01:11 [I 2022-10-10 17:24:32] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00293 [I 2022-10-10 17:24:32] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-10 17:24:32] ...optimizers/trijoint.py.151: total_norm: 21.444355 [I 2022-10-10 17:24:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 2230/2304 [I 2022-10-10 17:24:41] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:01:02 [I 2022-10-10 17:24:41] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00296 [I 2022-10-10 17:24:41] ...trap/engines/engine.py.230: loss: 0.13750 [I 2022-10-10 17:24:41] ...optimizers/trijoint.py.151: total_norm: 26.658253 [I 2022-10-10 17:24:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 2240/2304 [I 2022-10-10 17:24:49] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:54 [I 2022-10-10 17:24:49] ...trap/engines/engine.py.229: process: 0.85628 | load: 0.00292 [I 2022-10-10 17:24:49] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-10 17:24:49] ...optimizers/trijoint.py.151: total_norm: 12.574707 [I 2022-10-10 17:24:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 2250/2304 [I 2022-10-10 17:24:58] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:45 [I 2022-10-10 17:24:58] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00293 [I 2022-10-10 17:24:58] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-10 17:24:58] ...optimizers/trijoint.py.151: total_norm: 24.079882 [I 2022-10-10 17:25:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 2260/2304 [I 2022-10-10 17:25:06] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:37 [I 2022-10-10 17:25:06] ...trap/engines/engine.py.229: process: 0.86419 | load: 0.00302 [I 2022-10-10 17:25:06] ...trap/engines/engine.py.230: loss: 0.17767 [I 2022-10-10 17:25:06] ...optimizers/trijoint.py.151: total_norm: 26.868517 [I 2022-10-10 17:25:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 2270/2304 [I 2022-10-10 17:25:15] ...trap/engines/engine.py.228: elapsed: 0:32:56 | left: 0:00:29 [I 2022-10-10 17:25:15] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00295 [I 2022-10-10 17:25:15] ...trap/engines/engine.py.230: loss: 0.16225 [I 2022-10-10 17:25:15] ...optimizers/trijoint.py.151: total_norm: 22.305775 [I 2022-10-10 17:25:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 2280/2304 [I 2022-10-10 17:25:23] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:20 [I 2022-10-10 17:25:23] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00296 [I 2022-10-10 17:25:23] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-10 17:25:23] ...optimizers/trijoint.py.151: total_norm: 16.505680 [I 2022-10-10 17:25:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 2290/2304 [I 2022-10-10 17:25:32] ...trap/engines/engine.py.228: elapsed: 0:33:13 | left: 0:00:11 [I 2022-10-10 17:25:32] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00309 [I 2022-10-10 17:25:32] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-10 17:25:32] ...optimizers/trijoint.py.151: total_norm: 14.965670 [I 2022-10-10 17:25:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 2300/2304 [I 2022-10-10 17:25:41] ...trap/engines/engine.py.228: elapsed: 0:33:22 | left: 0:00:03 [I 2022-10-10 17:25:41] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00019 [I 2022-10-10 17:25:41] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-10 17:25:41] ...optimizers/trijoint.py.151: total_norm: 13.869478 [I 2022-10-10 17:25:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 2304/2304 [I 2022-10-10 17:25:44] ...trap/engines/engine.py.228: elapsed: 0:33:25 | left: 0:00:00 [I 2022-10-10 17:25:44] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00020 [I 2022-10-10 17:25:44] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-10 17:25:44] ...optimizers/trijoint.py.151: total_norm: 14.592651 [S 2022-10-10 17:25:45] ...trap/engines/engine.py.129: train_epoch.epoch: 79 [S 2022-10-10 17:25:45] ...trap/engines/engine.py.129: train_epoch.loss: 0.13352678407169474 [I 2022-10-10 17:25:46] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 17:25:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 79 [I 2022-10-10 17:25:47] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 17:25:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 17:25:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 17:25:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 17:25:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 17:25:52] ...trap/engines/engine.py.321: eval: epoch 79 | batch 0/510 [I 2022-10-10 17:25:52] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:13 [I 2022-10-10 17:25:52] ...trap/engines/engine.py.326: process: 5.43766 | load: 4.90398 [I 2022-10-10 17:25:58] ...trap/engines/engine.py.321: eval: epoch 79 | batch 10/510 [I 2022-10-10 17:25:58] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:25 [I 2022-10-10 17:25:58] ...trap/engines/engine.py.326: process: 0.64213 | load: 0.00022 [I 2022-10-10 17:26:05] ...trap/engines/engine.py.321: eval: epoch 79 | batch 20/510 [I 2022-10-10 17:26:05] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:40 [I 2022-10-10 17:26:05] ...trap/engines/engine.py.326: process: 0.78387 | load: 0.00021 [I 2022-10-10 17:26:11] ...trap/engines/engine.py.321: eval: epoch 79 | batch 30/510 [I 2022-10-10 17:26:11] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:31 [I 2022-10-10 17:26:11] ...trap/engines/engine.py.326: process: 0.78703 | load: 0.00022 [I 2022-10-10 17:26:18] ...trap/engines/engine.py.321: eval: epoch 79 | batch 40/510 [I 2022-10-10 17:26:18] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:05:10 [I 2022-10-10 17:26:18] ...trap/engines/engine.py.326: process: 0.44019 | load: 0.00020 [I 2022-10-10 17:26:23] ...trap/engines/engine.py.321: eval: epoch 79 | batch 50/510 [I 2022-10-10 17:26:23] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:33 [I 2022-10-10 17:26:23] ...trap/engines/engine.py.326: process: 0.45218 | load: 0.00020 [I 2022-10-10 17:26:28] ...trap/engines/engine.py.321: eval: epoch 79 | batch 60/510 [I 2022-10-10 17:26:28] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:58 [I 2022-10-10 17:26:28] ...trap/engines/engine.py.326: process: 0.57711 | load: 0.00020 [I 2022-10-10 17:26:34] ...trap/engines/engine.py.321: eval: epoch 79 | batch 70/510 [I 2022-10-10 17:26:34] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:15 [I 2022-10-10 17:26:34] ...trap/engines/engine.py.326: process: 0.59508 | load: 0.00021 [I 2022-10-10 17:26:40] ...trap/engines/engine.py.321: eval: epoch 79 | batch 80/510 [I 2022-10-10 17:26:40] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:15 [I 2022-10-10 17:26:40] ...trap/engines/engine.py.326: process: 0.53332 | load: 0.00020 [I 2022-10-10 17:26:45] ...trap/engines/engine.py.321: eval: epoch 79 | batch 90/510 [I 2022-10-10 17:26:45] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:35 [I 2022-10-10 17:26:45] ...trap/engines/engine.py.326: process: 0.52488 | load: 0.00019 [I 2022-10-10 17:26:50] ...trap/engines/engine.py.321: eval: epoch 79 | batch 100/510 [I 2022-10-10 17:26:50] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:03:05 [I 2022-10-10 17:26:50] ...trap/engines/engine.py.326: process: 0.29863 | load: 0.00020 [I 2022-10-10 17:26:53] ...trap/engines/engine.py.321: eval: epoch 79 | batch 110/510 [I 2022-10-10 17:26:53] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:14 [I 2022-10-10 17:26:53] ...trap/engines/engine.py.326: process: 0.32732 | load: 0.00021 [I 2022-10-10 17:26:56] ...trap/engines/engine.py.321: eval: epoch 79 | batch 120/510 [I 2022-10-10 17:26:56] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:07 [I 2022-10-10 17:26:56] ...trap/engines/engine.py.326: process: 0.32402 | load: 0.00020 [I 2022-10-10 17:26:59] ...trap/engines/engine.py.321: eval: epoch 79 | batch 130/510 [I 2022-10-10 17:26:59] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:02 [I 2022-10-10 17:26:59] ...trap/engines/engine.py.326: process: 0.32149 | load: 0.00021 [I 2022-10-10 17:27:03] ...trap/engines/engine.py.321: eval: epoch 79 | batch 140/510 [I 2022-10-10 17:27:03] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:58 [I 2022-10-10 17:27:03] ...trap/engines/engine.py.326: process: 0.31756 | load: 0.00021 [I 2022-10-10 17:27:06] ...trap/engines/engine.py.321: eval: epoch 79 | batch 150/510 [I 2022-10-10 17:27:06] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-10 17:27:06] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00021 [I 2022-10-10 17:27:09] ...trap/engines/engine.py.321: eval: epoch 79 | batch 160/510 [I 2022-10-10 17:27:09] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:52 [I 2022-10-10 17:27:09] ...trap/engines/engine.py.326: process: 0.31969 | load: 0.00022 [I 2022-10-10 17:27:12] ...trap/engines/engine.py.321: eval: epoch 79 | batch 170/510 [I 2022-10-10 17:27:12] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:50 [I 2022-10-10 17:27:12] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00024 [I 2022-10-10 17:27:16] ...trap/engines/engine.py.321: eval: epoch 79 | batch 180/510 [I 2022-10-10 17:27:16] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:46 [I 2022-10-10 17:27:16] ...trap/engines/engine.py.326: process: 0.31467 | load: 0.00021 [I 2022-10-10 17:27:19] ...trap/engines/engine.py.321: eval: epoch 79 | batch 190/510 [I 2022-10-10 17:27:19] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:43 [I 2022-10-10 17:27:19] ...trap/engines/engine.py.326: process: 0.32628 | load: 0.00021 [I 2022-10-10 17:27:22] ...trap/engines/engine.py.321: eval: epoch 79 | batch 200/510 [I 2022-10-10 17:27:22] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:41 [I 2022-10-10 17:27:22] ...trap/engines/engine.py.326: process: 0.34083 | load: 0.00021 [I 2022-10-10 17:27:25] ...trap/engines/engine.py.321: eval: epoch 79 | batch 210/510 [I 2022-10-10 17:27:25] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:37 [I 2022-10-10 17:27:25] ...trap/engines/engine.py.326: process: 0.32693 | load: 0.00021 [I 2022-10-10 17:27:29] ...trap/engines/engine.py.321: eval: epoch 79 | batch 220/510 [I 2022-10-10 17:27:29] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:34 [I 2022-10-10 17:27:29] ...trap/engines/engine.py.326: process: 0.33270 | load: 0.00023 [I 2022-10-10 17:27:32] ...trap/engines/engine.py.321: eval: epoch 79 | batch 230/510 [I 2022-10-10 17:27:32] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:30 [I 2022-10-10 17:27:32] ...trap/engines/engine.py.326: process: 0.32295 | load: 0.00022 [I 2022-10-10 17:27:35] ...trap/engines/engine.py.321: eval: epoch 79 | batch 240/510 [I 2022-10-10 17:27:35] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:27 [I 2022-10-10 17:27:35] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00025 [I 2022-10-10 17:27:38] ...trap/engines/engine.py.321: eval: epoch 79 | batch 250/510 [I 2022-10-10 17:27:38] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:23 [I 2022-10-10 17:27:38] ...trap/engines/engine.py.326: process: 0.32613 | load: 0.00021 [I 2022-10-10 17:27:42] ...trap/engines/engine.py.321: eval: epoch 79 | batch 260/510 [I 2022-10-10 17:27:42] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:20 [I 2022-10-10 17:27:42] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00021 [I 2022-10-10 17:27:45] ...trap/engines/engine.py.321: eval: epoch 79 | batch 270/510 [I 2022-10-10 17:27:45] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:17 [I 2022-10-10 17:27:45] ...trap/engines/engine.py.326: process: 0.32415 | load: 0.00021 [I 2022-10-10 17:27:48] ...trap/engines/engine.py.321: eval: epoch 79 | batch 280/510 [I 2022-10-10 17:27:48] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:13 [I 2022-10-10 17:27:48] ...trap/engines/engine.py.326: process: 0.32328 | load: 0.00021 [I 2022-10-10 17:27:51] ...trap/engines/engine.py.321: eval: epoch 79 | batch 290/510 [I 2022-10-10 17:27:51] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:10 [I 2022-10-10 17:27:51] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00021 [I 2022-10-10 17:27:54] ...trap/engines/engine.py.321: eval: epoch 79 | batch 300/510 [I 2022-10-10 17:27:54] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:07 [I 2022-10-10 17:27:54] ...trap/engines/engine.py.326: process: 0.31846 | load: 0.00021 [I 2022-10-10 17:27:58] ...trap/engines/engine.py.321: eval: epoch 79 | batch 310/510 [I 2022-10-10 17:27:58] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:04 [I 2022-10-10 17:27:58] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00021 [I 2022-10-10 17:28:01] ...trap/engines/engine.py.321: eval: epoch 79 | batch 320/510 [I 2022-10-10 17:28:01] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:01 [I 2022-10-10 17:28:01] ...trap/engines/engine.py.326: process: 0.32107 | load: 0.00022 [I 2022-10-10 17:28:04] ...trap/engines/engine.py.321: eval: epoch 79 | batch 330/510 [I 2022-10-10 17:28:04] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:58 [I 2022-10-10 17:28:04] ...trap/engines/engine.py.326: process: 0.32303 | load: 0.00022 [I 2022-10-10 17:28:07] ...trap/engines/engine.py.321: eval: epoch 79 | batch 340/510 [I 2022-10-10 17:28:07] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:54 [I 2022-10-10 17:28:07] ...trap/engines/engine.py.326: process: 0.31904 | load: 0.00021 [I 2022-10-10 17:28:11] ...trap/engines/engine.py.321: eval: epoch 79 | batch 350/510 [I 2022-10-10 17:28:11] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:51 [I 2022-10-10 17:28:11] ...trap/engines/engine.py.326: process: 0.32231 | load: 0.00021 [I 2022-10-10 17:28:14] ...trap/engines/engine.py.321: eval: epoch 79 | batch 360/510 [I 2022-10-10 17:28:14] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-10 17:28:14] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00021 [I 2022-10-10 17:28:17] ...trap/engines/engine.py.321: eval: epoch 79 | batch 370/510 [I 2022-10-10 17:28:17] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:45 [I 2022-10-10 17:28:17] ...trap/engines/engine.py.326: process: 0.33262 | load: 0.00022 [I 2022-10-10 17:28:20] ...trap/engines/engine.py.321: eval: epoch 79 | batch 380/510 [I 2022-10-10 17:28:20] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:42 [I 2022-10-10 17:28:20] ...trap/engines/engine.py.326: process: 0.32425 | load: 0.00021 [I 2022-10-10 17:28:24] ...trap/engines/engine.py.321: eval: epoch 79 | batch 390/510 [I 2022-10-10 17:28:24] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:38 [I 2022-10-10 17:28:24] ...trap/engines/engine.py.326: process: 0.32456 | load: 0.00021 [I 2022-10-10 17:28:27] ...trap/engines/engine.py.321: eval: epoch 79 | batch 400/510 [I 2022-10-10 17:28:27] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-10 17:28:27] ...trap/engines/engine.py.326: process: 0.32714 | load: 0.00021 [I 2022-10-10 17:28:30] ...trap/engines/engine.py.321: eval: epoch 79 | batch 410/510 [I 2022-10-10 17:28:30] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-10 17:28:30] ...trap/engines/engine.py.326: process: 0.32553 | load: 0.00026 [I 2022-10-10 17:28:33] ...trap/engines/engine.py.321: eval: epoch 79 | batch 420/510 [I 2022-10-10 17:28:33] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:29 [I 2022-10-10 17:28:33] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00022 [I 2022-10-10 17:28:37] ...trap/engines/engine.py.321: eval: epoch 79 | batch 430/510 [I 2022-10-10 17:28:37] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:25 [I 2022-10-10 17:28:37] ...trap/engines/engine.py.326: process: 0.32809 | load: 0.00022 [I 2022-10-10 17:28:40] ...trap/engines/engine.py.321: eval: epoch 79 | batch 440/510 [I 2022-10-10 17:28:40] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:22 [I 2022-10-10 17:28:40] ...trap/engines/engine.py.326: process: 0.31586 | load: 0.00022 [I 2022-10-10 17:28:43] ...trap/engines/engine.py.321: eval: epoch 79 | batch 450/510 [I 2022-10-10 17:28:43] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-10 17:28:43] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00022 [I 2022-10-10 17:28:46] ...trap/engines/engine.py.321: eval: epoch 79 | batch 460/510 [I 2022-10-10 17:28:46] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:16 [I 2022-10-10 17:28:46] ...trap/engines/engine.py.326: process: 0.32598 | load: 0.00021 [I 2022-10-10 17:28:50] ...trap/engines/engine.py.321: eval: epoch 79 | batch 470/510 [I 2022-10-10 17:28:50] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-10 17:28:50] ...trap/engines/engine.py.326: process: 0.32842 | load: 0.00022 [I 2022-10-10 17:28:53] ...trap/engines/engine.py.321: eval: epoch 79 | batch 480/510 [I 2022-10-10 17:28:53] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-10 17:28:53] ...trap/engines/engine.py.326: process: 0.32486 | load: 0.00025 [I 2022-10-10 17:28:56] ...trap/engines/engine.py.321: eval: epoch 79 | batch 490/510 [I 2022-10-10 17:28:56] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-10 17:28:56] ...trap/engines/engine.py.326: process: 0.32047 | load: 0.00022 [I 2022-10-10 17:28:59] ...trap/engines/engine.py.321: eval: epoch 79 | batch 500/510 [I 2022-10-10 17:28:59] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-10 17:28:59] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00022 [I 2022-10-10 17:29:03] ...trap/engines/engine.py.321: eval: epoch 79 | batch 510/510 [I 2022-10-10 17:29:03] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:00 [I 2022-10-10 17:29:03] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00016 [S 2022-10-10 17:29:03] ...trap/engines/engine.py.132: eval_epoch.epoch: 79 [I 2022-10-10 17:29:03] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.733700 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906400 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935700 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.726600 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907100 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017401 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008754 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006051 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013418 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008848 [S 2022-10-10 17:29:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007203 [I 2022-10-10 17:29:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 17:29:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 17:29:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 17:29:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 17:29:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 17:29:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 17:29:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 17:29:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 17:29:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 80 [I 2022-10-10 17:29:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 17:29:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 0/2304 [I 2022-10-10 17:29:56] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:44 [I 2022-10-10 17:29:56] ...trap/engines/engine.py.229: process: 3.09223 | load: 2.18112 [I 2022-10-10 17:29:56] ...trap/engines/engine.py.230: loss: 0.10335 [I 2022-10-10 17:29:56] ...optimizers/trijoint.py.151: total_norm: 15.588238 [I 2022-10-10 17:30:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 10/2304 [I 2022-10-10 17:30:04] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:51 [I 2022-10-10 17:30:04] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00519 [I 2022-10-10 17:30:04] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-10 17:30:04] ...optimizers/trijoint.py.151: total_norm: 10.491539 [I 2022-10-10 17:30:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 20/2304 [I 2022-10-10 17:30:13] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:17 [I 2022-10-10 17:30:13] ...trap/engines/engine.py.229: process: 0.85423 | load: 0.00487 [I 2022-10-10 17:30:13] ...trap/engines/engine.py.230: loss: 0.12959 [I 2022-10-10 17:30:13] ...optimizers/trijoint.py.151: total_norm: 23.340662 [I 2022-10-10 17:30:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 30/2304 [I 2022-10-10 17:30:21] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:30 [I 2022-10-10 17:30:21] ...trap/engines/engine.py.229: process: 0.86711 | load: 0.00378 [I 2022-10-10 17:30:21] ...trap/engines/engine.py.230: loss: 0.15383 [I 2022-10-10 17:30:21] ...optimizers/trijoint.py.151: total_norm: 11.937964 [I 2022-10-10 17:30:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 40/2304 [I 2022-10-10 17:30:30] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:08 [I 2022-10-10 17:30:30] ...trap/engines/engine.py.229: process: 0.85122 | load: 0.00333 [I 2022-10-10 17:30:30] ...trap/engines/engine.py.230: loss: 0.12188 [I 2022-10-10 17:30:30] ...optimizers/trijoint.py.151: total_norm: 18.419132 [I 2022-10-10 17:30:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 50/2304 [I 2022-10-10 17:30:39] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:13 [I 2022-10-10 17:30:39] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00358 [I 2022-10-10 17:30:39] ...trap/engines/engine.py.230: loss: 0.15426 [I 2022-10-10 17:30:39] ...optimizers/trijoint.py.151: total_norm: 10.559859 [I 2022-10-10 17:30:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 60/2304 [I 2022-10-10 17:30:47] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:54 [I 2022-10-10 17:30:47] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00357 [I 2022-10-10 17:30:47] ...trap/engines/engine.py.230: loss: 0.09381 [I 2022-10-10 17:30:47] ...optimizers/trijoint.py.151: total_norm: 12.018852 [I 2022-10-10 17:30:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 70/2304 [I 2022-10-10 17:30:56] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:36 [I 2022-10-10 17:30:56] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00360 [I 2022-10-10 17:30:56] ...trap/engines/engine.py.230: loss: 0.09901 [I 2022-10-10 17:30:56] ...optimizers/trijoint.py.151: total_norm: 16.425116 [I 2022-10-10 17:31:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 80/2304 [I 2022-10-10 17:31:04] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:22 [I 2022-10-10 17:31:04] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00389 [I 2022-10-10 17:31:04] ...trap/engines/engine.py.230: loss: 0.08516 [I 2022-10-10 17:31:04] ...optimizers/trijoint.py.151: total_norm: 11.088097 [I 2022-10-10 17:31:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 90/2304 [I 2022-10-10 17:31:13] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:13 [I 2022-10-10 17:31:13] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00316 [I 2022-10-10 17:31:13] ...trap/engines/engine.py.230: loss: 0.21744 [I 2022-10-10 17:31:13] ...optimizers/trijoint.py.151: total_norm: 19.744312 [I 2022-10-10 17:31:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 100/2304 [I 2022-10-10 17:31:21] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:32 [I 2022-10-10 17:31:21] ...trap/engines/engine.py.229: process: 0.85824 | load: 0.00338 [I 2022-10-10 17:31:21] ...trap/engines/engine.py.230: loss: 0.16611 [I 2022-10-10 17:31:21] ...optimizers/trijoint.py.151: total_norm: 21.824505 [I 2022-10-10 17:31:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 110/2304 [I 2022-10-10 17:31:30] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:02 [I 2022-10-10 17:31:30] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00347 [I 2022-10-10 17:31:30] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-10 17:31:30] ...optimizers/trijoint.py.151: total_norm: 15.907987 [I 2022-10-10 17:31:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 120/2304 [I 2022-10-10 17:31:38] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:55 [I 2022-10-10 17:31:38] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00313 [I 2022-10-10 17:31:38] ...trap/engines/engine.py.230: loss: 0.14270 [I 2022-10-10 17:31:38] ...optimizers/trijoint.py.151: total_norm: 9.863184 [I 2022-10-10 17:31:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 130/2304 [I 2022-10-10 17:31:47] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:54 [I 2022-10-10 17:31:47] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00302 [I 2022-10-10 17:31:47] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-10 17:31:47] ...optimizers/trijoint.py.151: total_norm: 21.534540 [I 2022-10-10 17:31:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 140/2304 [I 2022-10-10 17:31:55] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:45 [I 2022-10-10 17:31:55] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00300 [I 2022-10-10 17:31:55] ...trap/engines/engine.py.230: loss: 0.12355 [I 2022-10-10 17:31:55] ...optimizers/trijoint.py.151: total_norm: 27.684580 [I 2022-10-10 17:32:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 150/2304 [I 2022-10-10 17:32:04] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:44 [I 2022-10-10 17:32:04] ...trap/engines/engine.py.229: process: 0.87292 | load: 0.00313 [I 2022-10-10 17:32:04] ...trap/engines/engine.py.230: loss: 0.18687 [I 2022-10-10 17:32:04] ...optimizers/trijoint.py.151: total_norm: 13.552176 [I 2022-10-10 17:32:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 160/2304 [I 2022-10-10 17:32:12] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:34 [I 2022-10-10 17:32:12] ...trap/engines/engine.py.229: process: 0.86212 | load: 0.00295 [I 2022-10-10 17:32:12] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-10 17:32:12] ...optimizers/trijoint.py.151: total_norm: 13.967775 [I 2022-10-10 17:32:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 170/2304 [I 2022-10-10 17:32:21] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:25 [I 2022-10-10 17:32:21] ...trap/engines/engine.py.229: process: 0.84704 | load: 0.00312 [I 2022-10-10 17:32:21] ...trap/engines/engine.py.230: loss: 0.20477 [I 2022-10-10 17:32:21] ...optimizers/trijoint.py.151: total_norm: 23.544079 [I 2022-10-10 17:32:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 180/2304 [I 2022-10-10 17:32:30] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:12 [I 2022-10-10 17:32:30] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00312 [I 2022-10-10 17:32:30] ...trap/engines/engine.py.230: loss: 0.16808 [I 2022-10-10 17:32:30] ...optimizers/trijoint.py.151: total_norm: 23.406889 [I 2022-10-10 17:32:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 190/2304 [I 2022-10-10 17:32:38] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:59 [I 2022-10-10 17:32:38] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00311 [I 2022-10-10 17:32:38] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-10 17:32:38] ...optimizers/trijoint.py.151: total_norm: 16.078808 [I 2022-10-10 17:32:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 200/2304 [I 2022-10-10 17:32:47] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:54 [I 2022-10-10 17:32:47] ...trap/engines/engine.py.229: process: 0.85222 | load: 0.00292 [I 2022-10-10 17:32:47] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-10 17:32:47] ...optimizers/trijoint.py.151: total_norm: 25.938437 [I 2022-10-10 17:32:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 210/2304 [I 2022-10-10 17:32:55] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:53 [I 2022-10-10 17:32:55] ...trap/engines/engine.py.229: process: 0.86797 | load: 0.00308 [I 2022-10-10 17:32:55] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-10 17:32:55] ...optimizers/trijoint.py.151: total_norm: 22.831028 [I 2022-10-10 17:33:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 220/2304 [I 2022-10-10 17:33:04] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:29 [I 2022-10-10 17:33:04] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00307 [I 2022-10-10 17:33:04] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-10 17:33:04] ...optimizers/trijoint.py.151: total_norm: 10.867298 [I 2022-10-10 17:33:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 230/2304 [I 2022-10-10 17:33:12] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:25 [I 2022-10-10 17:33:12] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00306 [I 2022-10-10 17:33:12] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-10 17:33:12] ...optimizers/trijoint.py.151: total_norm: 12.575445 [I 2022-10-10 17:33:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 240/2304 [I 2022-10-10 17:33:21] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:14 [I 2022-10-10 17:33:21] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00291 [I 2022-10-10 17:33:21] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-10 17:33:21] ...optimizers/trijoint.py.151: total_norm: 26.664064 [I 2022-10-10 17:33:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 250/2304 [I 2022-10-10 17:33:29] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:00 [I 2022-10-10 17:33:29] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00295 [I 2022-10-10 17:33:29] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-10 17:33:29] ...optimizers/trijoint.py.151: total_norm: 11.252099 [I 2022-10-10 17:33:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 260/2304 [I 2022-10-10 17:33:38] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:05 [I 2022-10-10 17:33:38] ...trap/engines/engine.py.229: process: 0.85359 | load: 0.00296 [I 2022-10-10 17:33:38] ...trap/engines/engine.py.230: loss: 0.09383 [I 2022-10-10 17:33:38] ...optimizers/trijoint.py.151: total_norm: 26.056740 [I 2022-10-10 17:33:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 270/2304 [I 2022-10-10 17:33:46] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:52 [I 2022-10-10 17:33:46] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00294 [I 2022-10-10 17:33:46] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-10 17:33:46] ...optimizers/trijoint.py.151: total_norm: 10.286933 [I 2022-10-10 17:33:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 280/2304 [I 2022-10-10 17:33:55] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:32 [I 2022-10-10 17:33:55] ...trap/engines/engine.py.229: process: 0.84462 | load: 0.00293 [I 2022-10-10 17:33:55] ...trap/engines/engine.py.230: loss: 0.16015 [I 2022-10-10 17:33:55] ...optimizers/trijoint.py.151: total_norm: 12.415856 [I 2022-10-10 17:34:03] ...trap/engines/engine.py.224: train: epoch 80 | batch 290/2304 [I 2022-10-10 17:34:03] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:38 [I 2022-10-10 17:34:03] ...trap/engines/engine.py.229: process: 0.85617 | load: 0.00296 [I 2022-10-10 17:34:03] ...trap/engines/engine.py.230: loss: 0.09106 [I 2022-10-10 17:34:03] ...optimizers/trijoint.py.151: total_norm: 9.263068 [I 2022-10-10 17:34:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 300/2304 [I 2022-10-10 17:34:12] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:37 [I 2022-10-10 17:34:12] ...trap/engines/engine.py.229: process: 0.85043 | load: 0.00316 [I 2022-10-10 17:34:12] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-10 17:34:12] ...optimizers/trijoint.py.151: total_norm: 15.966163 [I 2022-10-10 17:34:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 310/2304 [I 2022-10-10 17:34:21] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:24 [I 2022-10-10 17:34:21] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00294 [I 2022-10-10 17:34:21] ...trap/engines/engine.py.230: loss: 0.14574 [I 2022-10-10 17:34:21] ...optimizers/trijoint.py.151: total_norm: 16.885130 [I 2022-10-10 17:34:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 320/2304 [I 2022-10-10 17:34:29] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:21 [I 2022-10-10 17:34:29] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00296 [I 2022-10-10 17:34:29] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-10 17:34:29] ...optimizers/trijoint.py.151: total_norm: 14.462526 [I 2022-10-10 17:34:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 330/2304 [I 2022-10-10 17:34:38] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:14 [I 2022-10-10 17:34:38] ...trap/engines/engine.py.229: process: 0.86579 | load: 0.00305 [I 2022-10-10 17:34:38] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-10 17:34:38] ...optimizers/trijoint.py.151: total_norm: 28.955374 [I 2022-10-10 17:34:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 340/2304 [I 2022-10-10 17:34:46] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:28:15 [I 2022-10-10 17:34:46] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00309 [I 2022-10-10 17:34:46] ...trap/engines/engine.py.230: loss: 0.15600 [I 2022-10-10 17:34:46] ...optimizers/trijoint.py.151: total_norm: 24.520895 [I 2022-10-10 17:34:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 350/2304 [I 2022-10-10 17:34:55] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:57 [I 2022-10-10 17:34:55] ...trap/engines/engine.py.229: process: 0.85005 | load: 0.00328 [I 2022-10-10 17:34:55] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-10 17:34:55] ...optimizers/trijoint.py.151: total_norm: 15.255549 [I 2022-10-10 17:35:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 360/2304 [I 2022-10-10 17:35:04] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:58 [I 2022-10-10 17:35:04] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00308 [I 2022-10-10 17:35:04] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-10 17:35:04] ...optimizers/trijoint.py.151: total_norm: 17.196638 [I 2022-10-10 17:35:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 370/2304 [I 2022-10-10 17:35:12] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:28:08 [I 2022-10-10 17:35:12] ...trap/engines/engine.py.229: process: 0.90374 | load: 0.00320 [I 2022-10-10 17:35:12] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-10 17:35:12] ...optimizers/trijoint.py.151: total_norm: 10.717415 [I 2022-10-10 17:35:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 380/2304 [I 2022-10-10 17:35:21] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:51 [I 2022-10-10 17:35:21] ...trap/engines/engine.py.229: process: 0.89104 | load: 0.00319 [I 2022-10-10 17:35:21] ...trap/engines/engine.py.230: loss: 0.14870 [I 2022-10-10 17:35:21] ...optimizers/trijoint.py.151: total_norm: 19.704018 [I 2022-10-10 17:35:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 390/2304 [I 2022-10-10 17:35:30] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:26 [I 2022-10-10 17:35:30] ...trap/engines/engine.py.229: process: 0.85842 | load: 0.00322 [I 2022-10-10 17:35:30] ...trap/engines/engine.py.230: loss: 0.18501 [I 2022-10-10 17:35:30] ...optimizers/trijoint.py.151: total_norm: 14.089355 [I 2022-10-10 17:35:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 400/2304 [I 2022-10-10 17:35:38] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:12 [I 2022-10-10 17:35:38] ...trap/engines/engine.py.229: process: 0.86336 | load: 0.00308 [I 2022-10-10 17:35:38] ...trap/engines/engine.py.230: loss: 0.10343 [I 2022-10-10 17:35:38] ...optimizers/trijoint.py.151: total_norm: 9.809216 [I 2022-10-10 17:35:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 410/2304 [I 2022-10-10 17:35:47] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:27:15 [I 2022-10-10 17:35:47] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00320 [I 2022-10-10 17:35:47] ...trap/engines/engine.py.230: loss: 0.11903 [I 2022-10-10 17:35:47] ...optimizers/trijoint.py.151: total_norm: 11.618709 [I 2022-10-10 17:35:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 420/2304 [I 2022-10-10 17:35:56] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:27:01 [I 2022-10-10 17:35:56] ...trap/engines/engine.py.229: process: 0.87561 | load: 0.00306 [I 2022-10-10 17:35:56] ...trap/engines/engine.py.230: loss: 0.11223 [I 2022-10-10 17:35:56] ...optimizers/trijoint.py.151: total_norm: 18.013332 [I 2022-10-10 17:36:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 430/2304 [I 2022-10-10 17:36:04] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:56 [I 2022-10-10 17:36:04] ...trap/engines/engine.py.229: process: 0.86881 | load: 0.00351 [I 2022-10-10 17:36:04] ...trap/engines/engine.py.230: loss: 0.13172 [I 2022-10-10 17:36:04] ...optimizers/trijoint.py.151: total_norm: 7.016600 [I 2022-10-10 17:36:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 440/2304 [I 2022-10-10 17:36:13] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:48 [I 2022-10-10 17:36:13] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00313 [I 2022-10-10 17:36:13] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-10 17:36:13] ...optimizers/trijoint.py.151: total_norm: 16.650688 [I 2022-10-10 17:36:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 450/2304 [I 2022-10-10 17:36:22] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:49 [I 2022-10-10 17:36:22] ...trap/engines/engine.py.229: process: 0.86746 | load: 0.00316 [I 2022-10-10 17:36:22] ...trap/engines/engine.py.230: loss: 0.11175 [I 2022-10-10 17:36:22] ...optimizers/trijoint.py.151: total_norm: 8.610671 [I 2022-10-10 17:36:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 460/2304 [I 2022-10-10 17:36:30] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:26:39 [I 2022-10-10 17:36:30] ...trap/engines/engine.py.229: process: 0.87856 | load: 0.00319 [I 2022-10-10 17:36:30] ...trap/engines/engine.py.230: loss: 0.09863 [I 2022-10-10 17:36:30] ...optimizers/trijoint.py.151: total_norm: 13.507539 [I 2022-10-10 17:36:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 470/2304 [I 2022-10-10 17:36:39] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:26:16 [I 2022-10-10 17:36:39] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00310 [I 2022-10-10 17:36:39] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-10 17:36:39] ...optimizers/trijoint.py.151: total_norm: 13.413018 [I 2022-10-10 17:36:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 480/2304 [I 2022-10-10 17:36:48] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:26:25 [I 2022-10-10 17:36:48] ...trap/engines/engine.py.229: process: 0.86299 | load: 0.00309 [I 2022-10-10 17:36:48] ...trap/engines/engine.py.230: loss: 0.10523 [I 2022-10-10 17:36:48] ...optimizers/trijoint.py.151: total_norm: 31.671864 [I 2022-10-10 17:36:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 490/2304 [I 2022-10-10 17:36:56] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:26:03 [I 2022-10-10 17:36:56] ...trap/engines/engine.py.229: process: 0.86199 | load: 0.00310 [I 2022-10-10 17:36:56] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-10 17:36:56] ...optimizers/trijoint.py.151: total_norm: 9.140073 [I 2022-10-10 17:37:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 500/2304 [I 2022-10-10 17:37:05] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:26:01 [I 2022-10-10 17:37:05] ...trap/engines/engine.py.229: process: 0.86128 | load: 0.00315 [I 2022-10-10 17:37:05] ...trap/engines/engine.py.230: loss: 0.10582 [I 2022-10-10 17:37:05] ...optimizers/trijoint.py.151: total_norm: 9.781493 [I 2022-10-10 17:37:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 510/2304 [I 2022-10-10 17:37:14] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:53 [I 2022-10-10 17:37:14] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00309 [I 2022-10-10 17:37:14] ...trap/engines/engine.py.230: loss: 0.11267 [I 2022-10-10 17:37:14] ...optimizers/trijoint.py.151: total_norm: 11.317707 [I 2022-10-10 17:37:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 520/2304 [I 2022-10-10 17:37:22] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:25:31 [I 2022-10-10 17:37:22] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00312 [I 2022-10-10 17:37:22] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-10 17:37:22] ...optimizers/trijoint.py.151: total_norm: 15.334439 [I 2022-10-10 17:37:31] ...trap/engines/engine.py.224: train: epoch 80 | batch 530/2304 [I 2022-10-10 17:37:31] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:25:34 [I 2022-10-10 17:37:31] ...trap/engines/engine.py.229: process: 0.88350 | load: 0.00319 [I 2022-10-10 17:37:31] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-10 17:37:31] ...optimizers/trijoint.py.151: total_norm: 51.904797 [I 2022-10-10 17:37:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 540/2304 [I 2022-10-10 17:37:40] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:25:28 [I 2022-10-10 17:37:40] ...trap/engines/engine.py.229: process: 0.87231 | load: 0.00321 [I 2022-10-10 17:37:40] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-10 17:37:40] ...optimizers/trijoint.py.151: total_norm: 26.001181 [I 2022-10-10 17:37:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 550/2304 [I 2022-10-10 17:37:48] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:11 [I 2022-10-10 17:37:48] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00317 [I 2022-10-10 17:37:48] ...trap/engines/engine.py.230: loss: 0.09420 [I 2022-10-10 17:37:48] ...optimizers/trijoint.py.151: total_norm: 7.430538 [I 2022-10-10 17:37:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 560/2304 [I 2022-10-10 17:37:57] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:25:07 [I 2022-10-10 17:37:57] ...trap/engines/engine.py.229: process: 0.86695 | load: 0.00312 [I 2022-10-10 17:37:57] ...trap/engines/engine.py.230: loss: 0.17248 [I 2022-10-10 17:37:57] ...optimizers/trijoint.py.151: total_norm: 13.996088 [I 2022-10-10 17:38:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 570/2304 [I 2022-10-10 17:38:06] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:57 [I 2022-10-10 17:38:06] ...trap/engines/engine.py.229: process: 0.85928 | load: 0.00314 [I 2022-10-10 17:38:06] ...trap/engines/engine.py.230: loss: 0.17405 [I 2022-10-10 17:38:06] ...optimizers/trijoint.py.151: total_norm: 24.736526 [I 2022-10-10 17:38:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 580/2304 [I 2022-10-10 17:38:14] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:52 [I 2022-10-10 17:38:14] ...trap/engines/engine.py.229: process: 0.87691 | load: 0.00312 [I 2022-10-10 17:38:14] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-10 17:38:14] ...optimizers/trijoint.py.151: total_norm: 12.799395 [I 2022-10-10 17:38:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 590/2304 [I 2022-10-10 17:38:23] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:34 [I 2022-10-10 17:38:23] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00315 [I 2022-10-10 17:38:23] ...trap/engines/engine.py.230: loss: 0.16094 [I 2022-10-10 17:38:23] ...optimizers/trijoint.py.151: total_norm: 46.043331 [I 2022-10-10 17:38:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 600/2304 [I 2022-10-10 17:38:32] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:24:33 [I 2022-10-10 17:38:32] ...trap/engines/engine.py.229: process: 0.87216 | load: 0.00303 [I 2022-10-10 17:38:32] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-10 17:38:32] ...optimizers/trijoint.py.151: total_norm: 11.108809 [I 2022-10-10 17:38:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 610/2304 [I 2022-10-10 17:38:40] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:24:06 [I 2022-10-10 17:38:40] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00295 [I 2022-10-10 17:38:40] ...trap/engines/engine.py.230: loss: 0.11032 [I 2022-10-10 17:38:40] ...optimizers/trijoint.py.151: total_norm: 15.285495 [I 2022-10-10 17:38:49] ...trap/engines/engine.py.224: train: epoch 80 | batch 620/2304 [I 2022-10-10 17:38:49] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:24:05 [I 2022-10-10 17:38:49] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00306 [I 2022-10-10 17:38:49] ...trap/engines/engine.py.230: loss: 0.08208 [I 2022-10-10 17:38:49] ...optimizers/trijoint.py.151: total_norm: 8.068751 [I 2022-10-10 17:38:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 630/2304 [I 2022-10-10 17:38:57] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:41 [I 2022-10-10 17:38:57] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00299 [I 2022-10-10 17:38:57] ...trap/engines/engine.py.230: loss: 0.18343 [I 2022-10-10 17:38:57] ...optimizers/trijoint.py.151: total_norm: 15.736667 [I 2022-10-10 17:39:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 640/2304 [I 2022-10-10 17:39:06] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:30 [I 2022-10-10 17:39:06] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00301 [I 2022-10-10 17:39:06] ...trap/engines/engine.py.230: loss: 0.15324 [I 2022-10-10 17:39:06] ...optimizers/trijoint.py.151: total_norm: 19.391659 [I 2022-10-10 17:39:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 650/2304 [I 2022-10-10 17:39:14] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:20 [I 2022-10-10 17:39:14] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00290 [I 2022-10-10 17:39:14] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-10 17:39:14] ...optimizers/trijoint.py.151: total_norm: 11.436362 [I 2022-10-10 17:39:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 660/2304 [I 2022-10-10 17:39:23] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:15 [I 2022-10-10 17:39:23] ...trap/engines/engine.py.229: process: 0.84646 | load: 0.00290 [I 2022-10-10 17:39:23] ...trap/engines/engine.py.230: loss: 0.11442 [I 2022-10-10 17:39:23] ...optimizers/trijoint.py.151: total_norm: 33.664898 [I 2022-10-10 17:39:31] ...trap/engines/engine.py.224: train: epoch 80 | batch 670/2304 [I 2022-10-10 17:39:31] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:23:14 [I 2022-10-10 17:39:31] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00295 [I 2022-10-10 17:39:31] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-10 17:39:31] ...optimizers/trijoint.py.151: total_norm: 11.831220 [I 2022-10-10 17:39:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 680/2304 [I 2022-10-10 17:39:40] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:23:06 [I 2022-10-10 17:39:40] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00292 [I 2022-10-10 17:39:40] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-10 17:39:40] ...optimizers/trijoint.py.151: total_norm: 23.874865 [I 2022-10-10 17:39:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 690/2304 [I 2022-10-10 17:39:48] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:23:04 [I 2022-10-10 17:39:48] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00294 [I 2022-10-10 17:39:48] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-10 17:39:48] ...optimizers/trijoint.py.151: total_norm: 10.835551 [I 2022-10-10 17:39:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 700/2304 [I 2022-10-10 17:39:57] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:22:48 [I 2022-10-10 17:39:57] ...trap/engines/engine.py.229: process: 0.84409 | load: 0.00295 [I 2022-10-10 17:39:57] ...trap/engines/engine.py.230: loss: 0.09064 [I 2022-10-10 17:39:57] ...optimizers/trijoint.py.151: total_norm: 9.808409 [I 2022-10-10 17:40:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 710/2304 [I 2022-10-10 17:40:06] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:22:41 [I 2022-10-10 17:40:06] ...trap/engines/engine.py.229: process: 0.85489 | load: 0.00294 [I 2022-10-10 17:40:06] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-10 17:40:06] ...optimizers/trijoint.py.151: total_norm: 10.632953 [I 2022-10-10 17:40:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 720/2304 [I 2022-10-10 17:40:14] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:22:43 [I 2022-10-10 17:40:14] ...trap/engines/engine.py.229: process: 0.86560 | load: 0.00293 [I 2022-10-10 17:40:14] ...trap/engines/engine.py.230: loss: 0.15403 [I 2022-10-10 17:40:14] ...optimizers/trijoint.py.151: total_norm: 22.753431 [I 2022-10-10 17:40:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 730/2304 [I 2022-10-10 17:40:23] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:22:25 [I 2022-10-10 17:40:23] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00295 [I 2022-10-10 17:40:23] ...trap/engines/engine.py.230: loss: 0.12463 [I 2022-10-10 17:40:23] ...optimizers/trijoint.py.151: total_norm: 8.449017 [I 2022-10-10 17:40:31] ...trap/engines/engine.py.224: train: epoch 80 | batch 740/2304 [I 2022-10-10 17:40:31] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:22:19 [I 2022-10-10 17:40:31] ...trap/engines/engine.py.229: process: 0.86349 | load: 0.00299 [I 2022-10-10 17:40:31] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-10 17:40:31] ...optimizers/trijoint.py.151: total_norm: 31.844076 [I 2022-10-10 17:40:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 750/2304 [I 2022-10-10 17:40:40] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:22:07 [I 2022-10-10 17:40:40] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00293 [I 2022-10-10 17:40:40] ...trap/engines/engine.py.230: loss: 0.12509 [I 2022-10-10 17:40:40] ...optimizers/trijoint.py.151: total_norm: 16.672651 [I 2022-10-10 17:40:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 760/2304 [I 2022-10-10 17:40:48] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:21:43 [I 2022-10-10 17:40:48] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00296 [I 2022-10-10 17:40:48] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-10 17:40:48] ...optimizers/trijoint.py.151: total_norm: 18.995575 [I 2022-10-10 17:40:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 770/2304 [I 2022-10-10 17:40:57] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:21:45 [I 2022-10-10 17:40:57] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00291 [I 2022-10-10 17:40:57] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-10 17:40:57] ...optimizers/trijoint.py.151: total_norm: 16.325636 [I 2022-10-10 17:41:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 780/2304 [I 2022-10-10 17:41:05] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:21:37 [I 2022-10-10 17:41:05] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00292 [I 2022-10-10 17:41:05] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-10 17:41:05] ...optimizers/trijoint.py.151: total_norm: 21.172689 [I 2022-10-10 17:41:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 790/2304 [I 2022-10-10 17:41:14] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:29 [I 2022-10-10 17:41:14] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00307 [I 2022-10-10 17:41:14] ...trap/engines/engine.py.230: loss: 0.08403 [I 2022-10-10 17:41:14] ...optimizers/trijoint.py.151: total_norm: 8.970049 [I 2022-10-10 17:41:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 800/2304 [I 2022-10-10 17:41:23] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:21:25 [I 2022-10-10 17:41:23] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00299 [I 2022-10-10 17:41:23] ...trap/engines/engine.py.230: loss: 0.12776 [I 2022-10-10 17:41:23] ...optimizers/trijoint.py.151: total_norm: 13.139633 [I 2022-10-10 17:41:31] ...trap/engines/engine.py.224: train: epoch 80 | batch 810/2304 [I 2022-10-10 17:41:31] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:21:11 [I 2022-10-10 17:41:31] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00315 [I 2022-10-10 17:41:31] ...trap/engines/engine.py.230: loss: 0.09151 [I 2022-10-10 17:41:31] ...optimizers/trijoint.py.151: total_norm: 14.822145 [I 2022-10-10 17:41:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 820/2304 [I 2022-10-10 17:41:40] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:21:04 [I 2022-10-10 17:41:40] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00296 [I 2022-10-10 17:41:40] ...trap/engines/engine.py.230: loss: 0.17307 [I 2022-10-10 17:41:40] ...optimizers/trijoint.py.151: total_norm: 8.437973 [I 2022-10-10 17:41:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 830/2304 [I 2022-10-10 17:41:48] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:59 [I 2022-10-10 17:41:48] ...trap/engines/engine.py.229: process: 0.86580 | load: 0.00297 [I 2022-10-10 17:41:48] ...trap/engines/engine.py.230: loss: 0.14039 [I 2022-10-10 17:41:48] ...optimizers/trijoint.py.151: total_norm: 11.809729 [I 2022-10-10 17:41:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 840/2304 [I 2022-10-10 17:41:57] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:21:08 [I 2022-10-10 17:41:57] ...trap/engines/engine.py.229: process: 0.87049 | load: 0.00307 [I 2022-10-10 17:41:57] ...trap/engines/engine.py.230: loss: 0.13382 [I 2022-10-10 17:41:57] ...optimizers/trijoint.py.151: total_norm: 21.749624 [I 2022-10-10 17:42:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 850/2304 [I 2022-10-10 17:42:05] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:53 [I 2022-10-10 17:42:05] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00327 [I 2022-10-10 17:42:05] ...trap/engines/engine.py.230: loss: 0.12608 [I 2022-10-10 17:42:05] ...optimizers/trijoint.py.151: total_norm: 11.400142 [I 2022-10-10 17:42:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 860/2304 [I 2022-10-10 17:42:14] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:57 [I 2022-10-10 17:42:14] ...trap/engines/engine.py.229: process: 0.88064 | load: 0.00306 [I 2022-10-10 17:42:14] ...trap/engines/engine.py.230: loss: 0.09524 [I 2022-10-10 17:42:14] ...optimizers/trijoint.py.151: total_norm: 21.702253 [I 2022-10-10 17:42:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 870/2304 [I 2022-10-10 17:42:23] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:41 [I 2022-10-10 17:42:23] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00307 [I 2022-10-10 17:42:23] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-10 17:42:23] ...optimizers/trijoint.py.151: total_norm: 23.549654 [I 2022-10-10 17:42:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 880/2304 [I 2022-10-10 17:42:32] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:20:25 [I 2022-10-10 17:42:32] ...trap/engines/engine.py.229: process: 0.86478 | load: 0.00324 [I 2022-10-10 17:42:32] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-10 17:42:32] ...optimizers/trijoint.py.151: total_norm: 25.887836 [I 2022-10-10 17:42:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 890/2304 [I 2022-10-10 17:42:40] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:24 [I 2022-10-10 17:42:40] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00313 [I 2022-10-10 17:42:40] ...trap/engines/engine.py.230: loss: 0.10158 [I 2022-10-10 17:42:40] ...optimizers/trijoint.py.151: total_norm: 13.219910 [I 2022-10-10 17:42:49] ...trap/engines/engine.py.224: train: epoch 80 | batch 900/2304 [I 2022-10-10 17:42:49] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:20:16 [I 2022-10-10 17:42:49] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00315 [I 2022-10-10 17:42:49] ...trap/engines/engine.py.230: loss: 0.15015 [I 2022-10-10 17:42:49] ...optimizers/trijoint.py.151: total_norm: 18.292585 [I 2022-10-10 17:42:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 910/2304 [I 2022-10-10 17:42:58] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:20:10 [I 2022-10-10 17:42:58] ...trap/engines/engine.py.229: process: 0.86824 | load: 0.00311 [I 2022-10-10 17:42:58] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-10 17:42:58] ...optimizers/trijoint.py.151: total_norm: 17.109638 [I 2022-10-10 17:43:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 920/2304 [I 2022-10-10 17:43:06] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:20:01 [I 2022-10-10 17:43:06] ...trap/engines/engine.py.229: process: 0.88339 | load: 0.00315 [I 2022-10-10 17:43:06] ...trap/engines/engine.py.230: loss: 0.18802 [I 2022-10-10 17:43:06] ...optimizers/trijoint.py.151: total_norm: 25.559858 [I 2022-10-10 17:43:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 930/2304 [I 2022-10-10 17:43:15] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:20:00 [I 2022-10-10 17:43:15] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00309 [I 2022-10-10 17:43:15] ...trap/engines/engine.py.230: loss: 0.10795 [I 2022-10-10 17:43:15] ...optimizers/trijoint.py.151: total_norm: 21.292938 [I 2022-10-10 17:43:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 940/2304 [I 2022-10-10 17:43:24] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:19:44 [I 2022-10-10 17:43:24] ...trap/engines/engine.py.229: process: 0.86713 | load: 0.00319 [I 2022-10-10 17:43:24] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-10 17:43:24] ...optimizers/trijoint.py.151: total_norm: 13.326380 [I 2022-10-10 17:43:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 950/2304 [I 2022-10-10 17:43:32] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:19:30 [I 2022-10-10 17:43:32] ...trap/engines/engine.py.229: process: 0.86957 | load: 0.00326 [I 2022-10-10 17:43:32] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-10 17:43:32] ...optimizers/trijoint.py.151: total_norm: 19.583124 [I 2022-10-10 17:43:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 960/2304 [I 2022-10-10 17:43:41] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:19:27 [I 2022-10-10 17:43:41] ...trap/engines/engine.py.229: process: 0.89342 | load: 0.00314 [I 2022-10-10 17:43:41] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-10 17:43:41] ...optimizers/trijoint.py.151: total_norm: 37.576702 [I 2022-10-10 17:43:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 970/2304 [I 2022-10-10 17:43:50] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:19:10 [I 2022-10-10 17:43:50] ...trap/engines/engine.py.229: process: 0.87028 | load: 0.00313 [I 2022-10-10 17:43:50] ...trap/engines/engine.py.230: loss: 0.09168 [I 2022-10-10 17:43:50] ...optimizers/trijoint.py.151: total_norm: 12.495246 [I 2022-10-10 17:43:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 980/2304 [I 2022-10-10 17:43:58] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:19:05 [I 2022-10-10 17:43:58] ...trap/engines/engine.py.229: process: 0.86154 | load: 0.00315 [I 2022-10-10 17:43:58] ...trap/engines/engine.py.230: loss: 0.14925 [I 2022-10-10 17:43:58] ...optimizers/trijoint.py.151: total_norm: 19.623417 [I 2022-10-10 17:44:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 990/2304 [I 2022-10-10 17:44:07] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:57 [I 2022-10-10 17:44:07] ...trap/engines/engine.py.229: process: 0.85592 | load: 0.00307 [I 2022-10-10 17:44:07] ...trap/engines/engine.py.230: loss: 0.10715 [I 2022-10-10 17:44:07] ...optimizers/trijoint.py.151: total_norm: 13.447844 [I 2022-10-10 17:44:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1000/2304 [I 2022-10-10 17:44:16] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:41 [I 2022-10-10 17:44:16] ...trap/engines/engine.py.229: process: 0.85369 | load: 0.00309 [I 2022-10-10 17:44:16] ...trap/engines/engine.py.230: loss: 0.13242 [I 2022-10-10 17:44:16] ...optimizers/trijoint.py.151: total_norm: 20.789976 [I 2022-10-10 17:44:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 1010/2304 [I 2022-10-10 17:44:24] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:40 [I 2022-10-10 17:44:24] ...trap/engines/engine.py.229: process: 0.86794 | load: 0.00311 [I 2022-10-10 17:44:24] ...trap/engines/engine.py.230: loss: 0.19317 [I 2022-10-10 17:44:24] ...optimizers/trijoint.py.151: total_norm: 9.742749 [I 2022-10-10 17:44:33] ...trap/engines/engine.py.224: train: epoch 80 | batch 1020/2304 [I 2022-10-10 17:44:33] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:26 [I 2022-10-10 17:44:33] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00310 [I 2022-10-10 17:44:33] ...trap/engines/engine.py.230: loss: 0.16909 [I 2022-10-10 17:44:33] ...optimizers/trijoint.py.151: total_norm: 30.101667 [I 2022-10-10 17:44:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 1030/2304 [I 2022-10-10 17:44:42] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:18:23 [I 2022-10-10 17:44:42] ...trap/engines/engine.py.229: process: 0.85718 | load: 0.00306 [I 2022-10-10 17:44:42] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-10 17:44:42] ...optimizers/trijoint.py.151: total_norm: 22.775755 [I 2022-10-10 17:44:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 1040/2304 [I 2022-10-10 17:44:50] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:18:18 [I 2022-10-10 17:44:50] ...trap/engines/engine.py.229: process: 0.88194 | load: 0.00314 [I 2022-10-10 17:44:50] ...trap/engines/engine.py.230: loss: 0.10990 [I 2022-10-10 17:44:50] ...optimizers/trijoint.py.151: total_norm: 31.341150 [I 2022-10-10 17:44:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 1050/2304 [I 2022-10-10 17:44:59] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:18:08 [I 2022-10-10 17:44:59] ...trap/engines/engine.py.229: process: 0.86619 | load: 0.00313 [I 2022-10-10 17:44:59] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-10 17:44:59] ...optimizers/trijoint.py.151: total_norm: 22.845770 [I 2022-10-10 17:45:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1060/2304 [I 2022-10-10 17:45:08] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:50 [I 2022-10-10 17:45:08] ...trap/engines/engine.py.229: process: 0.87031 | load: 0.00307 [I 2022-10-10 17:45:08] ...trap/engines/engine.py.230: loss: 0.16160 [I 2022-10-10 17:45:08] ...optimizers/trijoint.py.151: total_norm: 20.499086 [I 2022-10-10 17:45:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1070/2304 [I 2022-10-10 17:45:16] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:45 [I 2022-10-10 17:45:16] ...trap/engines/engine.py.229: process: 0.88348 | load: 0.00341 [I 2022-10-10 17:45:16] ...trap/engines/engine.py.230: loss: 0.14111 [I 2022-10-10 17:45:16] ...optimizers/trijoint.py.151: total_norm: 13.377572 [I 2022-10-10 17:45:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1080/2304 [I 2022-10-10 17:45:25] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:37 [I 2022-10-10 17:45:25] ...trap/engines/engine.py.229: process: 0.84895 | load: 0.00306 [I 2022-10-10 17:45:25] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-10 17:45:25] ...optimizers/trijoint.py.151: total_norm: 12.282320 [I 2022-10-10 17:45:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1090/2304 [I 2022-10-10 17:45:34] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:35 [I 2022-10-10 17:45:34] ...trap/engines/engine.py.229: process: 0.87033 | load: 0.00307 [I 2022-10-10 17:45:34] ...trap/engines/engine.py.230: loss: 0.14211 [I 2022-10-10 17:45:34] ...optimizers/trijoint.py.151: total_norm: 16.161016 [I 2022-10-10 17:45:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 1100/2304 [I 2022-10-10 17:45:42] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:17 [I 2022-10-10 17:45:42] ...trap/engines/engine.py.229: process: 0.84497 | load: 0.00305 [I 2022-10-10 17:45:42] ...trap/engines/engine.py.230: loss: 0.10278 [I 2022-10-10 17:45:42] ...optimizers/trijoint.py.151: total_norm: 12.219005 [I 2022-10-10 17:45:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1110/2304 [I 2022-10-10 17:45:51] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:17:06 [I 2022-10-10 17:45:51] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00295 [I 2022-10-10 17:45:51] ...trap/engines/engine.py.230: loss: 0.13666 [I 2022-10-10 17:45:51] ...optimizers/trijoint.py.151: total_norm: 12.831676 [I 2022-10-10 17:45:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 1120/2304 [I 2022-10-10 17:45:59] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:58 [I 2022-10-10 17:45:59] ...trap/engines/engine.py.229: process: 0.85772 | load: 0.00311 [I 2022-10-10 17:45:59] ...trap/engines/engine.py.230: loss: 0.16779 [I 2022-10-10 17:45:59] ...optimizers/trijoint.py.151: total_norm: 49.624325 [I 2022-10-10 17:46:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1130/2304 [I 2022-10-10 17:46:08] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:44 [I 2022-10-10 17:46:08] ...trap/engines/engine.py.229: process: 0.86672 | load: 0.00292 [I 2022-10-10 17:46:08] ...trap/engines/engine.py.230: loss: 0.15327 [I 2022-10-10 17:46:08] ...optimizers/trijoint.py.151: total_norm: 16.816069 [I 2022-10-10 17:46:17] ...trap/engines/engine.py.224: train: epoch 80 | batch 1140/2304 [I 2022-10-10 17:46:17] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:37 [I 2022-10-10 17:46:17] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00293 [I 2022-10-10 17:46:17] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-10 17:46:17] ...optimizers/trijoint.py.151: total_norm: 10.331411 [I 2022-10-10 17:46:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1150/2304 [I 2022-10-10 17:46:25] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:33 [I 2022-10-10 17:46:25] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00288 [I 2022-10-10 17:46:25] ...trap/engines/engine.py.230: loss: 0.08826 [I 2022-10-10 17:46:25] ...optimizers/trijoint.py.151: total_norm: 13.601111 [I 2022-10-10 17:46:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1160/2304 [I 2022-10-10 17:46:34] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:16 [I 2022-10-10 17:46:34] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00294 [I 2022-10-10 17:46:34] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-10 17:46:34] ...optimizers/trijoint.py.151: total_norm: 11.119497 [I 2022-10-10 17:46:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 1170/2304 [I 2022-10-10 17:46:42] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:16:06 [I 2022-10-10 17:46:42] ...trap/engines/engine.py.229: process: 0.85076 | load: 0.00296 [I 2022-10-10 17:46:42] ...trap/engines/engine.py.230: loss: 0.10716 [I 2022-10-10 17:46:42] ...optimizers/trijoint.py.151: total_norm: 10.291492 [I 2022-10-10 17:46:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1180/2304 [I 2022-10-10 17:46:51] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:57 [I 2022-10-10 17:46:51] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00291 [I 2022-10-10 17:46:51] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-10 17:46:51] ...optimizers/trijoint.py.151: total_norm: 22.247379 [I 2022-10-10 17:46:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 1190/2304 [I 2022-10-10 17:46:59] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:45 [I 2022-10-10 17:46:59] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00294 [I 2022-10-10 17:46:59] ...trap/engines/engine.py.230: loss: 0.17016 [I 2022-10-10 17:46:59] ...optimizers/trijoint.py.151: total_norm: 10.630357 [I 2022-10-10 17:47:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1200/2304 [I 2022-10-10 17:47:08] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:34 [I 2022-10-10 17:47:08] ...trap/engines/engine.py.229: process: 0.84198 | load: 0.00292 [I 2022-10-10 17:47:08] ...trap/engines/engine.py.230: loss: 0.13962 [I 2022-10-10 17:47:08] ...optimizers/trijoint.py.151: total_norm: 88.208794 [I 2022-10-10 17:47:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1210/2304 [I 2022-10-10 17:47:16] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:35 [I 2022-10-10 17:47:16] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00290 [I 2022-10-10 17:47:16] ...trap/engines/engine.py.230: loss: 0.18985 [I 2022-10-10 17:47:16] ...optimizers/trijoint.py.151: total_norm: 31.405022 [I 2022-10-10 17:47:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1220/2304 [I 2022-10-10 17:47:25] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:15:23 [I 2022-10-10 17:47:25] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00293 [I 2022-10-10 17:47:25] ...trap/engines/engine.py.230: loss: 0.22598 [I 2022-10-10 17:47:25] ...optimizers/trijoint.py.151: total_norm: 26.658867 [I 2022-10-10 17:47:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1230/2304 [I 2022-10-10 17:47:34] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:19 [I 2022-10-10 17:47:34] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00294 [I 2022-10-10 17:47:34] ...trap/engines/engine.py.230: loss: 0.11722 [I 2022-10-10 17:47:34] ...optimizers/trijoint.py.151: total_norm: 10.675739 [I 2022-10-10 17:47:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 1240/2304 [I 2022-10-10 17:47:42] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:11 [I 2022-10-10 17:47:42] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00294 [I 2022-10-10 17:47:42] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-10 17:47:42] ...optimizers/trijoint.py.151: total_norm: 22.470642 [I 2022-10-10 17:47:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1250/2304 [I 2022-10-10 17:47:51] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:58 [I 2022-10-10 17:47:51] ...trap/engines/engine.py.229: process: 0.85667 | load: 0.00292 [I 2022-10-10 17:47:51] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-10 17:47:51] ...optimizers/trijoint.py.151: total_norm: 11.552087 [I 2022-10-10 17:47:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 1260/2304 [I 2022-10-10 17:47:59] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:53 [I 2022-10-10 17:47:59] ...trap/engines/engine.py.229: process: 0.85844 | load: 0.00293 [I 2022-10-10 17:47:59] ...trap/engines/engine.py.230: loss: 0.10989 [I 2022-10-10 17:47:59] ...optimizers/trijoint.py.151: total_norm: 12.573971 [I 2022-10-10 17:48:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1270/2304 [I 2022-10-10 17:48:08] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:41 [I 2022-10-10 17:48:08] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00297 [I 2022-10-10 17:48:08] ...trap/engines/engine.py.230: loss: 0.13013 [I 2022-10-10 17:48:08] ...optimizers/trijoint.py.151: total_norm: 15.453747 [I 2022-10-10 17:48:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1280/2304 [I 2022-10-10 17:48:16] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:37 [I 2022-10-10 17:48:16] ...trap/engines/engine.py.229: process: 0.86297 | load: 0.00294 [I 2022-10-10 17:48:16] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-10 17:48:16] ...optimizers/trijoint.py.151: total_norm: 13.600259 [I 2022-10-10 17:48:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1290/2304 [I 2022-10-10 17:48:25] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:26 [I 2022-10-10 17:48:25] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00296 [I 2022-10-10 17:48:25] ...trap/engines/engine.py.230: loss: 0.16470 [I 2022-10-10 17:48:25] ...optimizers/trijoint.py.151: total_norm: 27.946264 [I 2022-10-10 17:48:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1300/2304 [I 2022-10-10 17:48:34] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:14 [I 2022-10-10 17:48:34] ...trap/engines/engine.py.229: process: 0.84931 | load: 0.00298 [I 2022-10-10 17:48:34] ...trap/engines/engine.py.230: loss: 0.11234 [I 2022-10-10 17:48:34] ...optimizers/trijoint.py.151: total_norm: 11.503673 [I 2022-10-10 17:48:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 1310/2304 [I 2022-10-10 17:48:42] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:04 [I 2022-10-10 17:48:42] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00295 [I 2022-10-10 17:48:42] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-10 17:48:42] ...optimizers/trijoint.py.151: total_norm: 19.924078 [I 2022-10-10 17:48:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1320/2304 [I 2022-10-10 17:48:51] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:58 [I 2022-10-10 17:48:51] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00305 [I 2022-10-10 17:48:51] ...trap/engines/engine.py.230: loss: 0.14217 [I 2022-10-10 17:48:51] ...optimizers/trijoint.py.151: total_norm: 25.476822 [I 2022-10-10 17:48:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 1330/2304 [I 2022-10-10 17:48:59] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:52 [I 2022-10-10 17:48:59] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00311 [I 2022-10-10 17:48:59] ...trap/engines/engine.py.230: loss: 0.10684 [I 2022-10-10 17:48:59] ...optimizers/trijoint.py.151: total_norm: 19.695816 [I 2022-10-10 17:49:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1340/2304 [I 2022-10-10 17:49:08] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:51 [I 2022-10-10 17:49:08] ...trap/engines/engine.py.229: process: 0.86803 | load: 0.00342 [I 2022-10-10 17:49:08] ...trap/engines/engine.py.230: loss: 0.08772 [I 2022-10-10 17:49:08] ...optimizers/trijoint.py.151: total_norm: 8.240866 [I 2022-10-10 17:49:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1350/2304 [I 2022-10-10 17:49:16] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:43 [I 2022-10-10 17:49:16] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00325 [I 2022-10-10 17:49:16] ...trap/engines/engine.py.230: loss: 0.14287 [I 2022-10-10 17:49:16] ...optimizers/trijoint.py.151: total_norm: 14.494108 [I 2022-10-10 17:49:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1360/2304 [I 2022-10-10 17:49:25] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:34 [I 2022-10-10 17:49:25] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00309 [I 2022-10-10 17:49:25] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-10 17:49:25] ...optimizers/trijoint.py.151: total_norm: 49.372440 [I 2022-10-10 17:49:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1370/2304 [I 2022-10-10 17:49:34] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:28 [I 2022-10-10 17:49:34] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00308 [I 2022-10-10 17:49:34] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-10 17:49:34] ...optimizers/trijoint.py.151: total_norm: 21.428970 [I 2022-10-10 17:49:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 1380/2304 [I 2022-10-10 17:49:43] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:20 [I 2022-10-10 17:49:43] ...trap/engines/engine.py.229: process: 0.86738 | load: 0.00310 [I 2022-10-10 17:49:43] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-10 17:49:43] ...optimizers/trijoint.py.151: total_norm: 13.090124 [I 2022-10-10 17:49:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1390/2304 [I 2022-10-10 17:49:51] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:11 [I 2022-10-10 17:49:51] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00317 [I 2022-10-10 17:49:51] ...trap/engines/engine.py.230: loss: 0.08583 [I 2022-10-10 17:49:51] ...optimizers/trijoint.py.151: total_norm: 10.183543 [I 2022-10-10 17:50:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 1400/2304 [I 2022-10-10 17:50:00] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:13:02 [I 2022-10-10 17:50:00] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00318 [I 2022-10-10 17:50:00] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-10 17:50:00] ...optimizers/trijoint.py.151: total_norm: 14.375456 [I 2022-10-10 17:50:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 1410/2304 [I 2022-10-10 17:50:09] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:49 [I 2022-10-10 17:50:09] ...trap/engines/engine.py.229: process: 0.87084 | load: 0.00344 [I 2022-10-10 17:50:09] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-10 17:50:09] ...optimizers/trijoint.py.151: total_norm: 11.758658 [I 2022-10-10 17:50:17] ...trap/engines/engine.py.224: train: epoch 80 | batch 1420/2304 [I 2022-10-10 17:50:17] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:12:38 [I 2022-10-10 17:50:17] ...trap/engines/engine.py.229: process: 0.84991 | load: 0.00311 [I 2022-10-10 17:50:17] ...trap/engines/engine.py.230: loss: 0.10246 [I 2022-10-10 17:50:17] ...optimizers/trijoint.py.151: total_norm: 19.999346 [I 2022-10-10 17:50:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 1430/2304 [I 2022-10-10 17:50:26] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:36 [I 2022-10-10 17:50:26] ...trap/engines/engine.py.229: process: 0.86872 | load: 0.00309 [I 2022-10-10 17:50:26] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-10 17:50:26] ...optimizers/trijoint.py.151: total_norm: 35.496231 [I 2022-10-10 17:50:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1440/2304 [I 2022-10-10 17:50:34] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:28 [I 2022-10-10 17:50:34] ...trap/engines/engine.py.229: process: 0.86806 | load: 0.00307 [I 2022-10-10 17:50:35] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-10 17:50:35] ...optimizers/trijoint.py.151: total_norm: 55.145912 [I 2022-10-10 17:50:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 1450/2304 [I 2022-10-10 17:50:43] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:12:18 [I 2022-10-10 17:50:43] ...trap/engines/engine.py.229: process: 0.85835 | load: 0.00311 [I 2022-10-10 17:50:43] ...trap/engines/engine.py.230: loss: 0.08723 [I 2022-10-10 17:50:43] ...optimizers/trijoint.py.151: total_norm: 11.353310 [I 2022-10-10 17:50:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 1460/2304 [I 2022-10-10 17:50:52] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:12:11 [I 2022-10-10 17:50:52] ...trap/engines/engine.py.229: process: 0.88793 | load: 0.00311 [I 2022-10-10 17:50:52] ...trap/engines/engine.py.230: loss: 0.19837 [I 2022-10-10 17:50:52] ...optimizers/trijoint.py.151: total_norm: 29.588766 [I 2022-10-10 17:51:01] ...trap/engines/engine.py.224: train: epoch 80 | batch 1470/2304 [I 2022-10-10 17:51:01] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:12:07 [I 2022-10-10 17:51:01] ...trap/engines/engine.py.229: process: 0.89559 | load: 0.00312 [I 2022-10-10 17:51:01] ...trap/engines/engine.py.230: loss: 0.14664 [I 2022-10-10 17:51:01] ...optimizers/trijoint.py.151: total_norm: 40.880478 [I 2022-10-10 17:51:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 1480/2304 [I 2022-10-10 17:51:09] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:50 [I 2022-10-10 17:51:09] ...trap/engines/engine.py.229: process: 0.86056 | load: 0.00309 [I 2022-10-10 17:51:09] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-10 17:51:09] ...optimizers/trijoint.py.151: total_norm: 14.874269 [I 2022-10-10 17:51:18] ...trap/engines/engine.py.224: train: epoch 80 | batch 1490/2304 [I 2022-10-10 17:51:18] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:44 [I 2022-10-10 17:51:18] ...trap/engines/engine.py.229: process: 0.88132 | load: 0.00310 [I 2022-10-10 17:51:18] ...trap/engines/engine.py.230: loss: 0.18000 [I 2022-10-10 17:51:18] ...optimizers/trijoint.py.151: total_norm: 22.214684 [I 2022-10-10 17:51:27] ...trap/engines/engine.py.224: train: epoch 80 | batch 1500/2304 [I 2022-10-10 17:51:27] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:35 [I 2022-10-10 17:51:27] ...trap/engines/engine.py.229: process: 0.86772 | load: 0.00313 [I 2022-10-10 17:51:27] ...trap/engines/engine.py.230: loss: 0.16096 [I 2022-10-10 17:51:27] ...optimizers/trijoint.py.151: total_norm: 20.770168 [I 2022-10-10 17:51:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 1510/2304 [I 2022-10-10 17:51:35] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:11:24 [I 2022-10-10 17:51:35] ...trap/engines/engine.py.229: process: 0.87387 | load: 0.00312 [I 2022-10-10 17:51:35] ...trap/engines/engine.py.230: loss: 0.20830 [I 2022-10-10 17:51:35] ...optimizers/trijoint.py.151: total_norm: 29.531136 [I 2022-10-10 17:51:44] ...trap/engines/engine.py.224: train: epoch 80 | batch 1520/2304 [I 2022-10-10 17:51:44] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:11:17 [I 2022-10-10 17:51:44] ...trap/engines/engine.py.229: process: 0.86661 | load: 0.00308 [I 2022-10-10 17:51:44] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-10 17:51:44] ...optimizers/trijoint.py.151: total_norm: 9.829537 [I 2022-10-10 17:51:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 1530/2304 [I 2022-10-10 17:51:52] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:11:03 [I 2022-10-10 17:51:52] ...trap/engines/engine.py.229: process: 0.84701 | load: 0.00314 [I 2022-10-10 17:51:52] ...trap/engines/engine.py.230: loss: 0.21429 [I 2022-10-10 17:51:52] ...optimizers/trijoint.py.151: total_norm: 19.236042 [I 2022-10-10 17:52:01] ...trap/engines/engine.py.224: train: epoch 80 | batch 1540/2304 [I 2022-10-10 17:52:01] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:10:59 [I 2022-10-10 17:52:01] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00308 [I 2022-10-10 17:52:01] ...trap/engines/engine.py.230: loss: 0.10634 [I 2022-10-10 17:52:01] ...optimizers/trijoint.py.151: total_norm: 17.099096 [I 2022-10-10 17:52:10] ...trap/engines/engine.py.224: train: epoch 80 | batch 1550/2304 [I 2022-10-10 17:52:10] ...trap/engines/engine.py.228: elapsed: 0:22:17 | left: 0:10:52 [I 2022-10-10 17:52:10] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00314 [I 2022-10-10 17:52:10] ...trap/engines/engine.py.230: loss: 0.12696 [I 2022-10-10 17:52:10] ...optimizers/trijoint.py.151: total_norm: 20.206585 [I 2022-10-10 17:52:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1560/2304 [I 2022-10-10 17:52:19] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:10:45 [I 2022-10-10 17:52:19] ...trap/engines/engine.py.229: process: 0.86740 | load: 0.00314 [I 2022-10-10 17:52:19] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-10 17:52:19] ...optimizers/trijoint.py.151: total_norm: 18.314121 [I 2022-10-10 17:52:27] ...trap/engines/engine.py.224: train: epoch 80 | batch 1570/2304 [I 2022-10-10 17:52:27] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:10:36 [I 2022-10-10 17:52:27] ...trap/engines/engine.py.229: process: 0.87795 | load: 0.00310 [I 2022-10-10 17:52:27] ...trap/engines/engine.py.230: loss: 0.13406 [I 2022-10-10 17:52:27] ...optimizers/trijoint.py.151: total_norm: 15.797045 [I 2022-10-10 17:52:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 1580/2304 [I 2022-10-10 17:52:36] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:10:24 [I 2022-10-10 17:52:36] ...trap/engines/engine.py.229: process: 0.87301 | load: 0.00318 [I 2022-10-10 17:52:36] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-10 17:52:36] ...optimizers/trijoint.py.151: total_norm: 21.984837 [I 2022-10-10 17:52:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1590/2304 [I 2022-10-10 17:52:45] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:10:20 [I 2022-10-10 17:52:45] ...trap/engines/engine.py.229: process: 0.87516 | load: 0.00316 [I 2022-10-10 17:52:45] ...trap/engines/engine.py.230: loss: 0.10967 [I 2022-10-10 17:52:45] ...optimizers/trijoint.py.151: total_norm: 14.299242 [I 2022-10-10 17:52:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 1600/2304 [I 2022-10-10 17:52:53] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:10:04 [I 2022-10-10 17:52:53] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00327 [I 2022-10-10 17:52:53] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-10 17:52:53] ...optimizers/trijoint.py.151: total_norm: 14.809729 [I 2022-10-10 17:53:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1610/2304 [I 2022-10-10 17:53:02] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:58 [I 2022-10-10 17:53:02] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00324 [I 2022-10-10 17:53:02] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-10 17:53:02] ...optimizers/trijoint.py.151: total_norm: 24.115631 [I 2022-10-10 17:53:10] ...trap/engines/engine.py.224: train: epoch 80 | batch 1620/2304 [I 2022-10-10 17:53:10] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:44 [I 2022-10-10 17:53:10] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00302 [I 2022-10-10 17:53:10] ...trap/engines/engine.py.230: loss: 0.14447 [I 2022-10-10 17:53:10] ...optimizers/trijoint.py.151: total_norm: 13.279430 [I 2022-10-10 17:53:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1630/2304 [I 2022-10-10 17:53:19] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:35 [I 2022-10-10 17:53:19] ...trap/engines/engine.py.229: process: 0.85615 | load: 0.00312 [I 2022-10-10 17:53:19] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-10 17:53:19] ...optimizers/trijoint.py.151: total_norm: 23.931273 [I 2022-10-10 17:53:27] ...trap/engines/engine.py.224: train: epoch 80 | batch 1640/2304 [I 2022-10-10 17:53:27] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:27 [I 2022-10-10 17:53:27] ...trap/engines/engine.py.229: process: 0.86476 | load: 0.00302 [I 2022-10-10 17:53:27] ...trap/engines/engine.py.230: loss: 0.21258 [I 2022-10-10 17:53:27] ...optimizers/trijoint.py.151: total_norm: 24.594656 [I 2022-10-10 17:53:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 1650/2304 [I 2022-10-10 17:53:36] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:09:22 [I 2022-10-10 17:53:36] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00311 [I 2022-10-10 17:53:36] ...trap/engines/engine.py.230: loss: 0.10889 [I 2022-10-10 17:53:36] ...optimizers/trijoint.py.151: total_norm: 11.441361 [I 2022-10-10 17:53:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1660/2304 [I 2022-10-10 17:53:45] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:09:16 [I 2022-10-10 17:53:45] ...trap/engines/engine.py.229: process: 0.87376 | load: 0.00307 [I 2022-10-10 17:53:45] ...trap/engines/engine.py.230: loss: 0.09463 [I 2022-10-10 17:53:45] ...optimizers/trijoint.py.151: total_norm: 13.045417 [I 2022-10-10 17:53:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 1670/2304 [I 2022-10-10 17:53:53] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:09:04 [I 2022-10-10 17:53:53] ...trap/engines/engine.py.229: process: 0.87016 | load: 0.00334 [I 2022-10-10 17:53:53] ...trap/engines/engine.py.230: loss: 0.09856 [I 2022-10-10 17:53:53] ...optimizers/trijoint.py.151: total_norm: 11.310320 [I 2022-10-10 17:54:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1680/2304 [I 2022-10-10 17:54:02] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:55 [I 2022-10-10 17:54:02] ...trap/engines/engine.py.229: process: 0.85615 | load: 0.00303 [I 2022-10-10 17:54:02] ...trap/engines/engine.py.230: loss: 0.23835 [I 2022-10-10 17:54:02] ...optimizers/trijoint.py.151: total_norm: 22.274422 [I 2022-10-10 17:54:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1690/2304 [I 2022-10-10 17:54:11] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:43 [I 2022-10-10 17:54:11] ...trap/engines/engine.py.229: process: 0.84330 | load: 0.00296 [I 2022-10-10 17:54:11] ...trap/engines/engine.py.230: loss: 0.16818 [I 2022-10-10 17:54:11] ...optimizers/trijoint.py.151: total_norm: 15.229011 [I 2022-10-10 17:54:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1700/2304 [I 2022-10-10 17:54:19] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:37 [I 2022-10-10 17:54:19] ...trap/engines/engine.py.229: process: 0.84102 | load: 0.00304 [I 2022-10-10 17:54:19] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-10 17:54:19] ...optimizers/trijoint.py.151: total_norm: 21.681908 [I 2022-10-10 17:54:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1710/2304 [I 2022-10-10 17:54:28] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:26 [I 2022-10-10 17:54:28] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00297 [I 2022-10-10 17:54:28] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-10 17:54:28] ...optimizers/trijoint.py.151: total_norm: 18.296669 [I 2022-10-10 17:54:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 1720/2304 [I 2022-10-10 17:54:36] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:18 [I 2022-10-10 17:54:36] ...trap/engines/engine.py.229: process: 0.86257 | load: 0.00299 [I 2022-10-10 17:54:36] ...trap/engines/engine.py.230: loss: 0.11450 [I 2022-10-10 17:54:36] ...optimizers/trijoint.py.151: total_norm: 7.196390 [I 2022-10-10 17:54:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1730/2304 [I 2022-10-10 17:54:45] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:08:12 [I 2022-10-10 17:54:45] ...trap/engines/engine.py.229: process: 0.86579 | load: 0.00292 [I 2022-10-10 17:54:45] ...trap/engines/engine.py.230: loss: 0.11041 [I 2022-10-10 17:54:45] ...optimizers/trijoint.py.151: total_norm: 16.970993 [I 2022-10-10 17:54:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 1740/2304 [I 2022-10-10 17:54:53] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:08:01 [I 2022-10-10 17:54:53] ...trap/engines/engine.py.229: process: 0.85911 | load: 0.00291 [I 2022-10-10 17:54:53] ...trap/engines/engine.py.230: loss: 0.09141 [I 2022-10-10 17:54:53] ...optimizers/trijoint.py.151: total_norm: 11.495072 [I 2022-10-10 17:55:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1750/2304 [I 2022-10-10 17:55:02] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:07:52 [I 2022-10-10 17:55:02] ...trap/engines/engine.py.229: process: 0.84140 | load: 0.00292 [I 2022-10-10 17:55:02] ...trap/engines/engine.py.230: loss: 0.10603 [I 2022-10-10 17:55:02] ...optimizers/trijoint.py.151: total_norm: 11.427101 [I 2022-10-10 17:55:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1760/2304 [I 2022-10-10 17:55:11] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:44 [I 2022-10-10 17:55:11] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00293 [I 2022-10-10 17:55:11] ...trap/engines/engine.py.230: loss: 0.08392 [I 2022-10-10 17:55:11] ...optimizers/trijoint.py.151: total_norm: 10.786891 [I 2022-10-10 17:55:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1770/2304 [I 2022-10-10 17:55:19] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:36 [I 2022-10-10 17:55:19] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00291 [I 2022-10-10 17:55:19] ...trap/engines/engine.py.230: loss: 0.13728 [I 2022-10-10 17:55:19] ...optimizers/trijoint.py.151: total_norm: 19.719385 [I 2022-10-10 17:55:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1780/2304 [I 2022-10-10 17:55:28] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:23 [I 2022-10-10 17:55:28] ...trap/engines/engine.py.229: process: 0.83381 | load: 0.00294 [I 2022-10-10 17:55:28] ...trap/engines/engine.py.230: loss: 0.12304 [I 2022-10-10 17:55:28] ...optimizers/trijoint.py.151: total_norm: 23.137716 [I 2022-10-10 17:55:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 1790/2304 [I 2022-10-10 17:55:36] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:15 [I 2022-10-10 17:55:36] ...trap/engines/engine.py.229: process: 0.83548 | load: 0.00293 [I 2022-10-10 17:55:36] ...trap/engines/engine.py.230: loss: 0.18168 [I 2022-10-10 17:55:36] ...optimizers/trijoint.py.151: total_norm: 17.011551 [I 2022-10-10 17:55:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1800/2304 [I 2022-10-10 17:55:45] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:11 [I 2022-10-10 17:55:45] ...trap/engines/engine.py.229: process: 0.86176 | load: 0.00300 [I 2022-10-10 17:55:45] ...trap/engines/engine.py.230: loss: 0.09852 [I 2022-10-10 17:55:45] ...optimizers/trijoint.py.151: total_norm: 12.140336 [I 2022-10-10 17:55:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 1810/2304 [I 2022-10-10 17:55:53] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:01 [I 2022-10-10 17:55:53] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00305 [I 2022-10-10 17:55:53] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-10 17:55:53] ...optimizers/trijoint.py.151: total_norm: 17.474981 [I 2022-10-10 17:56:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1820/2304 [I 2022-10-10 17:56:02] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:54 [I 2022-10-10 17:56:02] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00306 [I 2022-10-10 17:56:02] ...trap/engines/engine.py.230: loss: 0.16855 [I 2022-10-10 17:56:02] ...optimizers/trijoint.py.151: total_norm: 13.320745 [I 2022-10-10 17:56:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1830/2304 [I 2022-10-10 17:56:11] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:46 [I 2022-10-10 17:56:11] ...trap/engines/engine.py.229: process: 0.87083 | load: 0.00298 [I 2022-10-10 17:56:11] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-10 17:56:11] ...optimizers/trijoint.py.151: total_norm: 13.386714 [I 2022-10-10 17:56:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1840/2304 [I 2022-10-10 17:56:19] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:37 [I 2022-10-10 17:56:19] ...trap/engines/engine.py.229: process: 0.86055 | load: 0.00300 [I 2022-10-10 17:56:19] ...trap/engines/engine.py.230: loss: 0.18609 [I 2022-10-10 17:56:19] ...optimizers/trijoint.py.151: total_norm: 23.606791 [I 2022-10-10 17:56:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1850/2304 [I 2022-10-10 17:56:28] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:32 [I 2022-10-10 17:56:28] ...trap/engines/engine.py.229: process: 0.86420 | load: 0.00308 [I 2022-10-10 17:56:28] ...trap/engines/engine.py.230: loss: 0.09798 [I 2022-10-10 17:56:28] ...optimizers/trijoint.py.151: total_norm: 9.972218 [I 2022-10-10 17:56:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 1860/2304 [I 2022-10-10 17:56:37] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:25 [I 2022-10-10 17:56:37] ...trap/engines/engine.py.229: process: 0.86627 | load: 0.00310 [I 2022-10-10 17:56:37] ...trap/engines/engine.py.230: loss: 0.18099 [I 2022-10-10 17:56:37] ...optimizers/trijoint.py.151: total_norm: 21.943388 [I 2022-10-10 17:56:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1870/2304 [I 2022-10-10 17:56:45] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:13 [I 2022-10-10 17:56:45] ...trap/engines/engine.py.229: process: 0.86400 | load: 0.00308 [I 2022-10-10 17:56:45] ...trap/engines/engine.py.230: loss: 0.10644 [I 2022-10-10 17:56:45] ...optimizers/trijoint.py.151: total_norm: 11.754050 [I 2022-10-10 17:56:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 1880/2304 [I 2022-10-10 17:56:54] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:08 [I 2022-10-10 17:56:54] ...trap/engines/engine.py.229: process: 0.87107 | load: 0.00312 [I 2022-10-10 17:56:54] ...trap/engines/engine.py.230: loss: 0.10182 [I 2022-10-10 17:56:54] ...optimizers/trijoint.py.151: total_norm: 9.991746 [I 2022-10-10 17:57:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1890/2304 [I 2022-10-10 17:57:02] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:56 [I 2022-10-10 17:57:02] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00307 [I 2022-10-10 17:57:02] ...trap/engines/engine.py.230: loss: 0.08848 [I 2022-10-10 17:57:02] ...optimizers/trijoint.py.151: total_norm: 10.311819 [I 2022-10-10 17:57:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1900/2304 [I 2022-10-10 17:57:11] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:49 [I 2022-10-10 17:57:11] ...trap/engines/engine.py.229: process: 0.87007 | load: 0.00321 [I 2022-10-10 17:57:11] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-10 17:57:11] ...optimizers/trijoint.py.151: total_norm: 18.793322 [I 2022-10-10 17:57:20] ...trap/engines/engine.py.224: train: epoch 80 | batch 1910/2304 [I 2022-10-10 17:57:20] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:38 [I 2022-10-10 17:57:20] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00313 [I 2022-10-10 17:57:20] ...trap/engines/engine.py.230: loss: 0.13034 [I 2022-10-10 17:57:20] ...optimizers/trijoint.py.151: total_norm: 25.272144 [I 2022-10-10 17:57:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1920/2304 [I 2022-10-10 17:57:28] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:31 [I 2022-10-10 17:57:28] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00312 [I 2022-10-10 17:57:28] ...trap/engines/engine.py.230: loss: 0.11420 [I 2022-10-10 17:57:28] ...optimizers/trijoint.py.151: total_norm: 12.393560 [I 2022-10-10 17:57:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 1930/2304 [I 2022-10-10 17:57:37] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:25 [I 2022-10-10 17:57:37] ...trap/engines/engine.py.229: process: 0.87208 | load: 0.00311 [I 2022-10-10 17:57:37] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-10 17:57:37] ...optimizers/trijoint.py.151: total_norm: 15.737042 [I 2022-10-10 17:57:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 1940/2304 [I 2022-10-10 17:57:46] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:05:12 [I 2022-10-10 17:57:46] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00310 [I 2022-10-10 17:57:46] ...trap/engines/engine.py.230: loss: 0.09299 [I 2022-10-10 17:57:46] ...optimizers/trijoint.py.151: total_norm: 9.569847 [I 2022-10-10 17:57:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 1950/2304 [I 2022-10-10 17:57:54] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:05 [I 2022-10-10 17:57:54] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00309 [I 2022-10-10 17:57:54] ...trap/engines/engine.py.230: loss: 0.10712 [I 2022-10-10 17:57:54] ...optimizers/trijoint.py.151: total_norm: 12.885884 [I 2022-10-10 17:58:03] ...trap/engines/engine.py.224: train: epoch 80 | batch 1960/2304 [I 2022-10-10 17:58:03] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:58 [I 2022-10-10 17:58:03] ...trap/engines/engine.py.229: process: 0.88524 | load: 0.00310 [I 2022-10-10 17:58:03] ...trap/engines/engine.py.230: loss: 0.13505 [I 2022-10-10 17:58:03] ...optimizers/trijoint.py.151: total_norm: 25.051180 [I 2022-10-10 17:58:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 1970/2304 [I 2022-10-10 17:58:12] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:47 [I 2022-10-10 17:58:12] ...trap/engines/engine.py.229: process: 0.85300 | load: 0.00311 [I 2022-10-10 17:58:12] ...trap/engines/engine.py.230: loss: 0.21285 [I 2022-10-10 17:58:12] ...optimizers/trijoint.py.151: total_norm: 16.678314 [I 2022-10-10 17:58:20] ...trap/engines/engine.py.224: train: epoch 80 | batch 1980/2304 [I 2022-10-10 17:58:20] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:41 [I 2022-10-10 17:58:20] ...trap/engines/engine.py.229: process: 0.88686 | load: 0.00306 [I 2022-10-10 17:58:20] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-10 17:58:20] ...optimizers/trijoint.py.151: total_norm: 36.723751 [I 2022-10-10 17:58:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 1990/2304 [I 2022-10-10 17:58:29] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:33 [I 2022-10-10 17:58:29] ...trap/engines/engine.py.229: process: 0.88026 | load: 0.00310 [I 2022-10-10 17:58:29] ...trap/engines/engine.py.230: loss: 0.15380 [I 2022-10-10 17:58:29] ...optimizers/trijoint.py.151: total_norm: 21.819809 [I 2022-10-10 17:58:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 2000/2304 [I 2022-10-10 17:58:38] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:20 [I 2022-10-10 17:58:38] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00314 [I 2022-10-10 17:58:38] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-10 17:58:38] ...optimizers/trijoint.py.151: total_norm: 31.781416 [I 2022-10-10 17:58:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 2010/2304 [I 2022-10-10 17:58:46] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:04:13 [I 2022-10-10 17:58:46] ...trap/engines/engine.py.229: process: 0.85757 | load: 0.00329 [I 2022-10-10 17:58:46] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-10 17:58:46] ...optimizers/trijoint.py.151: total_norm: 14.218001 [I 2022-10-10 17:58:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 2020/2304 [I 2022-10-10 17:58:55] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:03 [I 2022-10-10 17:58:55] ...trap/engines/engine.py.229: process: 0.86350 | load: 0.00312 [I 2022-10-10 17:58:55] ...trap/engines/engine.py.230: loss: 0.14581 [I 2022-10-10 17:58:55] ...optimizers/trijoint.py.151: total_norm: 24.102322 [I 2022-10-10 17:59:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 2030/2304 [I 2022-10-10 17:59:04] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:53 [I 2022-10-10 17:59:04] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00329 [I 2022-10-10 17:59:04] ...trap/engines/engine.py.230: loss: 0.09945 [I 2022-10-10 17:59:04] ...optimizers/trijoint.py.151: total_norm: 12.739801 [I 2022-10-10 17:59:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 2040/2304 [I 2022-10-10 17:59:12] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:47 [I 2022-10-10 17:59:12] ...trap/engines/engine.py.229: process: 0.84466 | load: 0.00311 [I 2022-10-10 17:59:12] ...trap/engines/engine.py.230: loss: 0.14687 [I 2022-10-10 17:59:12] ...optimizers/trijoint.py.151: total_norm: 16.472012 [I 2022-10-10 17:59:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 2050/2304 [I 2022-10-10 17:59:21] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:38 [I 2022-10-10 17:59:21] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00315 [I 2022-10-10 17:59:21] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-10 17:59:21] ...optimizers/trijoint.py.151: total_norm: 16.379599 [I 2022-10-10 17:59:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 2060/2304 [I 2022-10-10 17:59:30] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:03:30 [I 2022-10-10 17:59:30] ...trap/engines/engine.py.229: process: 0.88222 | load: 0.00305 [I 2022-10-10 17:59:30] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-10 17:59:30] ...optimizers/trijoint.py.151: total_norm: 21.327108 [I 2022-10-10 17:59:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 2070/2304 [I 2022-10-10 17:59:38] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:21 [I 2022-10-10 17:59:38] ...trap/engines/engine.py.229: process: 0.86563 | load: 0.00312 [I 2022-10-10 17:59:38] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-10 17:59:38] ...optimizers/trijoint.py.151: total_norm: 14.192248 [I 2022-10-10 17:59:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 2080/2304 [I 2022-10-10 17:59:47] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:14 [I 2022-10-10 17:59:47] ...trap/engines/engine.py.229: process: 0.87530 | load: 0.00315 [I 2022-10-10 17:59:47] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-10 17:59:47] ...optimizers/trijoint.py.151: total_norm: 9.706873 [I 2022-10-10 17:59:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 2090/2304 [I 2022-10-10 17:59:56] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:03:05 [I 2022-10-10 17:59:56] ...trap/engines/engine.py.229: process: 0.87222 | load: 0.00306 [I 2022-10-10 17:59:56] ...trap/engines/engine.py.230: loss: 0.21276 [I 2022-10-10 17:59:56] ...optimizers/trijoint.py.151: total_norm: 24.359303 [I 2022-10-10 18:00:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 2100/2304 [I 2022-10-10 18:00:04] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:55 [I 2022-10-10 18:00:04] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00307 [I 2022-10-10 18:00:04] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-10 18:00:04] ...optimizers/trijoint.py.151: total_norm: 11.308335 [I 2022-10-10 18:00:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 2110/2304 [I 2022-10-10 18:00:13] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:47 [I 2022-10-10 18:00:13] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00305 [I 2022-10-10 18:00:13] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-10 18:00:13] ...optimizers/trijoint.py.151: total_norm: 8.796268 [I 2022-10-10 18:00:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 2120/2304 [I 2022-10-10 18:00:22] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:36 [I 2022-10-10 18:00:22] ...trap/engines/engine.py.229: process: 0.84788 | load: 0.00295 [I 2022-10-10 18:00:22] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-10 18:00:22] ...optimizers/trijoint.py.151: total_norm: 26.548460 [I 2022-10-10 18:00:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 2130/2304 [I 2022-10-10 18:00:30] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:28 [I 2022-10-10 18:00:30] ...trap/engines/engine.py.229: process: 0.84463 | load: 0.00304 [I 2022-10-10 18:00:30] ...trap/engines/engine.py.230: loss: 0.11649 [I 2022-10-10 18:00:30] ...optimizers/trijoint.py.151: total_norm: 11.371380 [I 2022-10-10 18:00:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 2140/2304 [I 2022-10-10 18:00:39] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:19 [I 2022-10-10 18:00:39] ...trap/engines/engine.py.229: process: 0.86167 | load: 0.00293 [I 2022-10-10 18:00:39] ...trap/engines/engine.py.230: loss: 0.10585 [I 2022-10-10 18:00:39] ...optimizers/trijoint.py.151: total_norm: 16.446117 [I 2022-10-10 18:00:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 2150/2304 [I 2022-10-10 18:00:47] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:11 [I 2022-10-10 18:00:47] ...trap/engines/engine.py.229: process: 0.86683 | load: 0.00289 [I 2022-10-10 18:00:47] ...trap/engines/engine.py.230: loss: 0.13754 [I 2022-10-10 18:00:47] ...optimizers/trijoint.py.151: total_norm: 30.014383 [I 2022-10-10 18:00:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 2160/2304 [I 2022-10-10 18:00:56] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:02:03 [I 2022-10-10 18:00:56] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00293 [I 2022-10-10 18:00:56] ...trap/engines/engine.py.230: loss: 0.18419 [I 2022-10-10 18:00:56] ...optimizers/trijoint.py.151: total_norm: 21.940025 [I 2022-10-10 18:01:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 2170/2304 [I 2022-10-10 18:01:04] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:54 [I 2022-10-10 18:01:04] ...trap/engines/engine.py.229: process: 0.85946 | load: 0.00294 [I 2022-10-10 18:01:04] ...trap/engines/engine.py.230: loss: 0.13440 [I 2022-10-10 18:01:04] ...optimizers/trijoint.py.151: total_norm: 11.771132 [I 2022-10-10 18:01:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 2180/2304 [I 2022-10-10 18:01:13] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:46 [I 2022-10-10 18:01:13] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00292 [I 2022-10-10 18:01:13] ...trap/engines/engine.py.230: loss: 0.15637 [I 2022-10-10 18:01:13] ...optimizers/trijoint.py.151: total_norm: 40.327892 [I 2022-10-10 18:01:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 2190/2304 [I 2022-10-10 18:01:21] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:36 [I 2022-10-10 18:01:21] ...trap/engines/engine.py.229: process: 0.84938 | load: 0.00302 [I 2022-10-10 18:01:21] ...trap/engines/engine.py.230: loss: 0.18827 [I 2022-10-10 18:01:21] ...optimizers/trijoint.py.151: total_norm: 17.063103 [I 2022-10-10 18:01:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 2200/2304 [I 2022-10-10 18:01:30] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:28 [I 2022-10-10 18:01:30] ...trap/engines/engine.py.229: process: 0.84470 | load: 0.00306 [I 2022-10-10 18:01:30] ...trap/engines/engine.py.230: loss: 0.17519 [I 2022-10-10 18:01:30] ...optimizers/trijoint.py.151: total_norm: 9.966304 [I 2022-10-10 18:01:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 2210/2304 [I 2022-10-10 18:01:38] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:20 [I 2022-10-10 18:01:38] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00298 [I 2022-10-10 18:01:38] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-10 18:01:38] ...optimizers/trijoint.py.151: total_norm: 11.372647 [I 2022-10-10 18:01:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 2220/2304 [I 2022-10-10 18:01:47] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:01:11 [I 2022-10-10 18:01:47] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00297 [I 2022-10-10 18:01:47] ...trap/engines/engine.py.230: loss: 0.09801 [I 2022-10-10 18:01:47] ...optimizers/trijoint.py.151: total_norm: 12.463742 [I 2022-10-10 18:01:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 2230/2304 [I 2022-10-10 18:01:56] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:01:03 [I 2022-10-10 18:01:56] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00300 [I 2022-10-10 18:01:56] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-10 18:01:56] ...optimizers/trijoint.py.151: total_norm: 14.146764 [I 2022-10-10 18:02:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 2240/2304 [I 2022-10-10 18:02:04] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:54 [I 2022-10-10 18:02:04] ...trap/engines/engine.py.229: process: 0.84127 | load: 0.00298 [I 2022-10-10 18:02:04] ...trap/engines/engine.py.230: loss: 0.09432 [I 2022-10-10 18:02:04] ...optimizers/trijoint.py.151: total_norm: 10.692144 [I 2022-10-10 18:02:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 2250/2304 [I 2022-10-10 18:02:13] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:46 [I 2022-10-10 18:02:13] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00302 [I 2022-10-10 18:02:13] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-10 18:02:13] ...optimizers/trijoint.py.151: total_norm: 11.806220 [I 2022-10-10 18:02:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 2260/2304 [I 2022-10-10 18:02:21] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:37 [I 2022-10-10 18:02:21] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00318 [I 2022-10-10 18:02:21] ...trap/engines/engine.py.230: loss: 0.08764 [I 2022-10-10 18:02:21] ...optimizers/trijoint.py.151: total_norm: 9.417925 [I 2022-10-10 18:02:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 2270/2304 [I 2022-10-10 18:02:30] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:28 [I 2022-10-10 18:02:30] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00312 [I 2022-10-10 18:02:30] ...trap/engines/engine.py.230: loss: 0.23287 [I 2022-10-10 18:02:30] ...optimizers/trijoint.py.151: total_norm: 22.480379 [I 2022-10-10 18:02:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 2280/2304 [I 2022-10-10 18:02:38] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:20 [I 2022-10-10 18:02:38] ...trap/engines/engine.py.229: process: 0.85138 | load: 0.00296 [I 2022-10-10 18:02:38] ...trap/engines/engine.py.230: loss: 0.13235 [I 2022-10-10 18:02:38] ...optimizers/trijoint.py.151: total_norm: 17.015821 [I 2022-10-10 18:02:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 2290/2304 [I 2022-10-10 18:02:47] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:11 [I 2022-10-10 18:02:47] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00299 [I 2022-10-10 18:02:47] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-10 18:02:47] ...optimizers/trijoint.py.151: total_norm: 24.238398 [I 2022-10-10 18:02:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 2300/2304 [I 2022-10-10 18:02:56] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:03 [I 2022-10-10 18:02:56] ...trap/engines/engine.py.229: process: 0.85037 | load: 0.00016 [I 2022-10-10 18:02:56] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-10 18:02:56] ...optimizers/trijoint.py.151: total_norm: 10.706854 [I 2022-10-10 18:02:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 2304/2304 [I 2022-10-10 18:02:59] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:00 [I 2022-10-10 18:02:59] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00019 [I 2022-10-10 18:02:59] ...trap/engines/engine.py.230: loss: 0.13093 [I 2022-10-10 18:02:59] ...optimizers/trijoint.py.151: total_norm: 11.653668 [S 2022-10-10 18:03:00] ...trap/engines/engine.py.129: train_epoch.epoch: 80 [S 2022-10-10 18:03:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.13277208317644942 [I 2022-10-10 18:03:00] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 18:03:02] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 80 [I 2022-10-10 18:03:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 18:03:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 18:03:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 18:03:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 18:03:07] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 18:03:08] ...trap/engines/engine.py.321: eval: epoch 80 | batch 0/510 [I 2022-10-10 18:03:08] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:36 [I 2022-10-10 18:03:08] ...trap/engines/engine.py.326: process: 5.83686 | load: 3.98101 [I 2022-10-10 18:03:13] ...trap/engines/engine.py.321: eval: epoch 80 | batch 10/510 [I 2022-10-10 18:03:13] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:06 [I 2022-10-10 18:03:13] ...trap/engines/engine.py.326: process: 0.56019 | load: 0.00020 [I 2022-10-10 18:03:19] ...trap/engines/engine.py.321: eval: epoch 80 | batch 20/510 [I 2022-10-10 18:03:19] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:14 [I 2022-10-10 18:03:19] ...trap/engines/engine.py.326: process: 0.59338 | load: 0.00020 [I 2022-10-10 18:03:25] ...trap/engines/engine.py.321: eval: epoch 80 | batch 30/510 [I 2022-10-10 18:03:25] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:04:59 [I 2022-10-10 18:03:25] ...trap/engines/engine.py.326: process: 0.64918 | load: 0.00019 [I 2022-10-10 18:03:32] ...trap/engines/engine.py.321: eval: epoch 80 | batch 40/510 [I 2022-10-10 18:03:32] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:47 [I 2022-10-10 18:03:32] ...trap/engines/engine.py.326: process: 0.53562 | load: 0.00021 [I 2022-10-10 18:03:37] ...trap/engines/engine.py.321: eval: epoch 80 | batch 50/510 [I 2022-10-10 18:03:37] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:02 [I 2022-10-10 18:03:37] ...trap/engines/engine.py.326: process: 0.46967 | load: 0.00022 [I 2022-10-10 18:03:43] ...trap/engines/engine.py.321: eval: epoch 80 | batch 60/510 [I 2022-10-10 18:03:43] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:29 [I 2022-10-10 18:03:43] ...trap/engines/engine.py.326: process: 0.59351 | load: 0.00019 [I 2022-10-10 18:03:49] ...trap/engines/engine.py.321: eval: epoch 80 | batch 70/510 [I 2022-10-10 18:03:49] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:25 [I 2022-10-10 18:03:49] ...trap/engines/engine.py.326: process: 0.63039 | load: 0.00020 [I 2022-10-10 18:03:55] ...trap/engines/engine.py.321: eval: epoch 80 | batch 80/510 [I 2022-10-10 18:03:55] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:34 [I 2022-10-10 18:03:55] ...trap/engines/engine.py.326: process: 0.70901 | load: 0.00020 [I 2022-10-10 18:04:00] ...trap/engines/engine.py.321: eval: epoch 80 | batch 90/510 [I 2022-10-10 18:04:00] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:43 [I 2022-10-10 18:04:00] ...trap/engines/engine.py.326: process: 0.55041 | load: 0.00023 [I 2022-10-10 18:04:06] ...trap/engines/engine.py.321: eval: epoch 80 | batch 100/510 [I 2022-10-10 18:04:06] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:22 [I 2022-10-10 18:04:06] ...trap/engines/engine.py.326: process: 0.31100 | load: 0.00020 [I 2022-10-10 18:04:09] ...trap/engines/engine.py.321: eval: epoch 80 | batch 110/510 [I 2022-10-10 18:04:09] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:16 [I 2022-10-10 18:04:09] ...trap/engines/engine.py.326: process: 0.31686 | load: 0.00020 [I 2022-10-10 18:04:12] ...trap/engines/engine.py.321: eval: epoch 80 | batch 120/510 [I 2022-10-10 18:04:12] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:06 [I 2022-10-10 18:04:12] ...trap/engines/engine.py.326: process: 0.32099 | load: 0.00021 [I 2022-10-10 18:04:15] ...trap/engines/engine.py.321: eval: epoch 80 | batch 130/510 [I 2022-10-10 18:04:15] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:02 [I 2022-10-10 18:04:15] ...trap/engines/engine.py.326: process: 0.31885 | load: 0.00021 [I 2022-10-10 18:04:19] ...trap/engines/engine.py.321: eval: epoch 80 | batch 140/510 [I 2022-10-10 18:04:19] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:58 [I 2022-10-10 18:04:19] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00021 [I 2022-10-10 18:04:22] ...trap/engines/engine.py.321: eval: epoch 80 | batch 150/510 [I 2022-10-10 18:04:22] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:55 [I 2022-10-10 18:04:22] ...trap/engines/engine.py.326: process: 0.31792 | load: 0.00022 [I 2022-10-10 18:04:25] ...trap/engines/engine.py.321: eval: epoch 80 | batch 160/510 [I 2022-10-10 18:04:25] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:52 [I 2022-10-10 18:04:25] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00021 [I 2022-10-10 18:04:28] ...trap/engines/engine.py.321: eval: epoch 80 | batch 170/510 [I 2022-10-10 18:04:28] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:50 [I 2022-10-10 18:04:28] ...trap/engines/engine.py.326: process: 0.32685 | load: 0.00023 [I 2022-10-10 18:04:31] ...trap/engines/engine.py.321: eval: epoch 80 | batch 180/510 [I 2022-10-10 18:04:31] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:46 [I 2022-10-10 18:04:31] ...trap/engines/engine.py.326: process: 0.32329 | load: 0.00020 [I 2022-10-10 18:04:35] ...trap/engines/engine.py.321: eval: epoch 80 | batch 190/510 [I 2022-10-10 18:04:35] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:44 [I 2022-10-10 18:04:35] ...trap/engines/engine.py.326: process: 0.33007 | load: 0.00024 [I 2022-10-10 18:04:38] ...trap/engines/engine.py.321: eval: epoch 80 | batch 200/510 [I 2022-10-10 18:04:38] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:39 [I 2022-10-10 18:04:38] ...trap/engines/engine.py.326: process: 0.31533 | load: 0.00020 [I 2022-10-10 18:04:41] ...trap/engines/engine.py.321: eval: epoch 80 | batch 210/510 [I 2022-10-10 18:04:41] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:37 [I 2022-10-10 18:04:41] ...trap/engines/engine.py.326: process: 0.32529 | load: 0.00023 [I 2022-10-10 18:04:44] ...trap/engines/engine.py.321: eval: epoch 80 | batch 220/510 [I 2022-10-10 18:04:44] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:33 [I 2022-10-10 18:04:44] ...trap/engines/engine.py.326: process: 0.32324 | load: 0.00021 [I 2022-10-10 18:04:48] ...trap/engines/engine.py.321: eval: epoch 80 | batch 230/510 [I 2022-10-10 18:04:48] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:30 [I 2022-10-10 18:04:48] ...trap/engines/engine.py.326: process: 0.32394 | load: 0.00021 [I 2022-10-10 18:04:51] ...trap/engines/engine.py.321: eval: epoch 80 | batch 240/510 [I 2022-10-10 18:04:51] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:27 [I 2022-10-10 18:04:51] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00021 [I 2022-10-10 18:04:54] ...trap/engines/engine.py.321: eval: epoch 80 | batch 250/510 [I 2022-10-10 18:04:54] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:24 [I 2022-10-10 18:04:54] ...trap/engines/engine.py.326: process: 0.32560 | load: 0.00022 [I 2022-10-10 18:04:57] ...trap/engines/engine.py.321: eval: epoch 80 | batch 260/510 [I 2022-10-10 18:04:57] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:20 [I 2022-10-10 18:04:57] ...trap/engines/engine.py.326: process: 0.32318 | load: 0.00021 [I 2022-10-10 18:05:01] ...trap/engines/engine.py.321: eval: epoch 80 | batch 270/510 [I 2022-10-10 18:05:01] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:17 [I 2022-10-10 18:05:01] ...trap/engines/engine.py.326: process: 0.32581 | load: 0.00021 [I 2022-10-10 18:05:04] ...trap/engines/engine.py.321: eval: epoch 80 | batch 280/510 [I 2022-10-10 18:05:04] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:14 [I 2022-10-10 18:05:04] ...trap/engines/engine.py.326: process: 0.33105 | load: 0.00030 [I 2022-10-10 18:05:07] ...trap/engines/engine.py.321: eval: epoch 80 | batch 290/510 [I 2022-10-10 18:05:07] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:11 [I 2022-10-10 18:05:07] ...trap/engines/engine.py.326: process: 0.32904 | load: 0.00021 [I 2022-10-10 18:05:10] ...trap/engines/engine.py.321: eval: epoch 80 | batch 300/510 [I 2022-10-10 18:05:10] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:08 [I 2022-10-10 18:05:10] ...trap/engines/engine.py.326: process: 0.33061 | load: 0.00021 [I 2022-10-10 18:05:14] ...trap/engines/engine.py.321: eval: epoch 80 | batch 310/510 [I 2022-10-10 18:05:14] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-10 18:05:14] ...trap/engines/engine.py.326: process: 0.31954 | load: 0.00021 [I 2022-10-10 18:05:17] ...trap/engines/engine.py.321: eval: epoch 80 | batch 320/510 [I 2022-10-10 18:05:17] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:01 [I 2022-10-10 18:05:17] ...trap/engines/engine.py.326: process: 0.32352 | load: 0.00022 [I 2022-10-10 18:05:20] ...trap/engines/engine.py.321: eval: epoch 80 | batch 330/510 [I 2022-10-10 18:05:20] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:58 [I 2022-10-10 18:05:20] ...trap/engines/engine.py.326: process: 0.32634 | load: 0.00021 [I 2022-10-10 18:05:23] ...trap/engines/engine.py.321: eval: epoch 80 | batch 340/510 [I 2022-10-10 18:05:23] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:54 [I 2022-10-10 18:05:23] ...trap/engines/engine.py.326: process: 0.32385 | load: 0.00021 [I 2022-10-10 18:05:27] ...trap/engines/engine.py.321: eval: epoch 80 | batch 350/510 [I 2022-10-10 18:05:27] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:51 [I 2022-10-10 18:05:27] ...trap/engines/engine.py.326: process: 0.32099 | load: 0.00022 [I 2022-10-10 18:05:30] ...trap/engines/engine.py.321: eval: epoch 80 | batch 360/510 [I 2022-10-10 18:05:30] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:48 [I 2022-10-10 18:05:30] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00021 [I 2022-10-10 18:05:33] ...trap/engines/engine.py.321: eval: epoch 80 | batch 370/510 [I 2022-10-10 18:05:33] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:45 [I 2022-10-10 18:05:33] ...trap/engines/engine.py.326: process: 0.32858 | load: 0.00021 [I 2022-10-10 18:05:36] ...trap/engines/engine.py.321: eval: epoch 80 | batch 380/510 [I 2022-10-10 18:05:36] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:42 [I 2022-10-10 18:05:36] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00021 [I 2022-10-10 18:05:39] ...trap/engines/engine.py.321: eval: epoch 80 | batch 390/510 [I 2022-10-10 18:05:39] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:38 [I 2022-10-10 18:05:39] ...trap/engines/engine.py.326: process: 0.31474 | load: 0.00022 [I 2022-10-10 18:05:43] ...trap/engines/engine.py.321: eval: epoch 80 | batch 400/510 [I 2022-10-10 18:05:43] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:35 [I 2022-10-10 18:05:43] ...trap/engines/engine.py.326: process: 0.32262 | load: 0.00021 [I 2022-10-10 18:05:46] ...trap/engines/engine.py.321: eval: epoch 80 | batch 410/510 [I 2022-10-10 18:05:46] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:32 [I 2022-10-10 18:05:46] ...trap/engines/engine.py.326: process: 0.32613 | load: 0.00021 [I 2022-10-10 18:05:49] ...trap/engines/engine.py.321: eval: epoch 80 | batch 420/510 [I 2022-10-10 18:05:49] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:29 [I 2022-10-10 18:05:49] ...trap/engines/engine.py.326: process: 0.32596 | load: 0.00021 [I 2022-10-10 18:05:52] ...trap/engines/engine.py.321: eval: epoch 80 | batch 430/510 [I 2022-10-10 18:05:52] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:25 [I 2022-10-10 18:05:52] ...trap/engines/engine.py.326: process: 0.32664 | load: 0.00021 [I 2022-10-10 18:05:56] ...trap/engines/engine.py.321: eval: epoch 80 | batch 440/510 [I 2022-10-10 18:05:56] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-10 18:05:56] ...trap/engines/engine.py.326: process: 0.32713 | load: 0.00022 [I 2022-10-10 18:05:59] ...trap/engines/engine.py.321: eval: epoch 80 | batch 450/510 [I 2022-10-10 18:05:59] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:19 [I 2022-10-10 18:05:59] ...trap/engines/engine.py.326: process: 0.32586 | load: 0.00021 [I 2022-10-10 18:06:02] ...trap/engines/engine.py.321: eval: epoch 80 | batch 460/510 [I 2022-10-10 18:06:02] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:16 [I 2022-10-10 18:06:02] ...trap/engines/engine.py.326: process: 0.32662 | load: 0.00020 [I 2022-10-10 18:06:05] ...trap/engines/engine.py.321: eval: epoch 80 | batch 470/510 [I 2022-10-10 18:06:05] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:12 [I 2022-10-10 18:06:05] ...trap/engines/engine.py.326: process: 0.32470 | load: 0.00021 [I 2022-10-10 18:06:09] ...trap/engines/engine.py.321: eval: epoch 80 | batch 480/510 [I 2022-10-10 18:06:09] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-10 18:06:09] ...trap/engines/engine.py.326: process: 0.33396 | load: 0.00021 [I 2022-10-10 18:06:12] ...trap/engines/engine.py.321: eval: epoch 80 | batch 490/510 [I 2022-10-10 18:06:12] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:06 [I 2022-10-10 18:06:12] ...trap/engines/engine.py.326: process: 0.32904 | load: 0.00022 [I 2022-10-10 18:06:15] ...trap/engines/engine.py.321: eval: epoch 80 | batch 500/510 [I 2022-10-10 18:06:15] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:03 [I 2022-10-10 18:06:15] ...trap/engines/engine.py.326: process: 0.32990 | load: 0.00021 [I 2022-10-10 18:06:18] ...trap/engines/engine.py.321: eval: epoch 80 | batch 510/510 [I 2022-10-10 18:06:18] ...trap/engines/engine.py.325: elapsed: 0:03:16 | left: 0:00:00 [I 2022-10-10 18:06:18] ...trap/engines/engine.py.326: process: 0.31757 | load: 0.00017 [S 2022-10-10 18:06:19] ...trap/engines/engine.py.132: eval_epoch.epoch: 80 [I 2022-10-10 18:06:19] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.739000 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913200 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940100 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732000 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.912100 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938800 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014429 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012278 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010634 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017861 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011502 [S 2022-10-10 18:06:42] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010028 [I 2022-10-10 18:06:57] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 18:06:57] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 18:06:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 18:06:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 18:06:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 18:07:01] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 18:07:04] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 18:07:07] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 18:07:07] ...trap/engines/engine.py.163: Training model on trainset for epoch 81 [I 2022-10-10 18:07:07] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 18:07:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 0/2304 [I 2022-10-10 18:07:11] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:11 [I 2022-10-10 18:07:11] ...trap/engines/engine.py.229: process: 3.23420 | load: 2.32283 [I 2022-10-10 18:07:11] ...trap/engines/engine.py.230: loss: 0.17223 [I 2022-10-10 18:07:11] ...optimizers/trijoint.py.151: total_norm: 24.605419 [I 2022-10-10 18:07:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 10/2304 [I 2022-10-10 18:07:19] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:15 [I 2022-10-10 18:07:19] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00472 [I 2022-10-10 18:07:19] ...trap/engines/engine.py.230: loss: 0.10525 [I 2022-10-10 18:07:19] ...optimizers/trijoint.py.151: total_norm: 7.800156 [I 2022-10-10 18:07:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 20/2304 [I 2022-10-10 18:07:28] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:44 [I 2022-10-10 18:07:28] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00500 [I 2022-10-10 18:07:28] ...trap/engines/engine.py.230: loss: 0.24501 [I 2022-10-10 18:07:28] ...optimizers/trijoint.py.151: total_norm: 16.847958 [I 2022-10-10 18:07:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 30/2304 [I 2022-10-10 18:07:36] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:41 [I 2022-10-10 18:07:36] ...trap/engines/engine.py.229: process: 0.86127 | load: 0.00450 [I 2022-10-10 18:07:36] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-10 18:07:36] ...optimizers/trijoint.py.151: total_norm: 20.141859 [I 2022-10-10 18:07:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 40/2304 [I 2022-10-10 18:07:45] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:18 [I 2022-10-10 18:07:45] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00447 [I 2022-10-10 18:07:45] ...trap/engines/engine.py.230: loss: 0.14479 [I 2022-10-10 18:07:45] ...optimizers/trijoint.py.151: total_norm: 23.507065 [I 2022-10-10 18:07:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 50/2304 [I 2022-10-10 18:07:54] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:01 [I 2022-10-10 18:07:54] ...trap/engines/engine.py.229: process: 0.84978 | load: 0.00383 [I 2022-10-10 18:07:54] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-10 18:07:54] ...optimizers/trijoint.py.151: total_norm: 14.488159 [I 2022-10-10 18:08:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 60/2304 [I 2022-10-10 18:08:02] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:48 [I 2022-10-10 18:08:02] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00401 [I 2022-10-10 18:08:02] ...trap/engines/engine.py.230: loss: 0.13754 [I 2022-10-10 18:08:02] ...optimizers/trijoint.py.151: total_norm: 12.624531 [I 2022-10-10 18:08:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 70/2304 [I 2022-10-10 18:08:11] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:50 [I 2022-10-10 18:08:11] ...trap/engines/engine.py.229: process: 0.85757 | load: 0.00397 [I 2022-10-10 18:08:11] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-10 18:08:11] ...optimizers/trijoint.py.151: total_norm: 15.638503 [I 2022-10-10 18:08:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 80/2304 [I 2022-10-10 18:08:19] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:42 [I 2022-10-10 18:08:19] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00310 [I 2022-10-10 18:08:19] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-10 18:08:19] ...optimizers/trijoint.py.151: total_norm: 9.034277 [I 2022-10-10 18:08:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 90/2304 [I 2022-10-10 18:08:28] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:14 [I 2022-10-10 18:08:28] ...trap/engines/engine.py.229: process: 0.82565 | load: 0.00329 [I 2022-10-10 18:08:28] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-10 18:08:28] ...optimizers/trijoint.py.151: total_norm: 13.461732 [I 2022-10-10 18:08:37] ...trap/engines/engine.py.224: train: epoch 81 | batch 100/2304 [I 2022-10-10 18:08:37] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:26 [I 2022-10-10 18:08:37] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00319 [I 2022-10-10 18:08:37] ...trap/engines/engine.py.230: loss: 0.12927 [I 2022-10-10 18:08:37] ...optimizers/trijoint.py.151: total_norm: 12.241177 [I 2022-10-10 18:08:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 110/2304 [I 2022-10-10 18:08:45] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:07 [I 2022-10-10 18:08:45] ...trap/engines/engine.py.229: process: 0.83471 | load: 0.00325 [I 2022-10-10 18:08:45] ...trap/engines/engine.py.230: loss: 0.12205 [I 2022-10-10 18:08:45] ...optimizers/trijoint.py.151: total_norm: 22.583511 [I 2022-10-10 18:08:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 120/2304 [I 2022-10-10 18:08:54] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:30:57 [I 2022-10-10 18:08:54] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00363 [I 2022-10-10 18:08:54] ...trap/engines/engine.py.230: loss: 0.09914 [I 2022-10-10 18:08:54] ...optimizers/trijoint.py.151: total_norm: 9.520536 [I 2022-10-10 18:09:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 130/2304 [I 2022-10-10 18:09:02] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:00 [I 2022-10-10 18:09:02] ...trap/engines/engine.py.229: process: 0.87099 | load: 0.00338 [I 2022-10-10 18:09:02] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-10 18:09:02] ...optimizers/trijoint.py.151: total_norm: 27.384258 [I 2022-10-10 18:09:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 140/2304 [I 2022-10-10 18:09:11] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:46 [I 2022-10-10 18:09:11] ...trap/engines/engine.py.229: process: 0.85447 | load: 0.00345 [I 2022-10-10 18:09:11] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-10 18:09:11] ...optimizers/trijoint.py.151: total_norm: 8.363400 [I 2022-10-10 18:09:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 150/2304 [I 2022-10-10 18:09:19] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:51 [I 2022-10-10 18:09:19] ...trap/engines/engine.py.229: process: 0.87815 | load: 0.00328 [I 2022-10-10 18:09:19] ...trap/engines/engine.py.230: loss: 0.24080 [I 2022-10-10 18:09:19] ...optimizers/trijoint.py.151: total_norm: 13.456109 [I 2022-10-10 18:09:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 160/2304 [I 2022-10-10 18:09:28] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:41 [I 2022-10-10 18:09:28] ...trap/engines/engine.py.229: process: 0.86257 | load: 0.00300 [I 2022-10-10 18:09:28] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-10 18:09:28] ...optimizers/trijoint.py.151: total_norm: 15.502253 [I 2022-10-10 18:09:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 170/2304 [I 2022-10-10 18:09:36] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:25 [I 2022-10-10 18:09:36] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00327 [I 2022-10-10 18:09:36] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-10 18:09:36] ...optimizers/trijoint.py.151: total_norm: 12.886358 [I 2022-10-10 18:09:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 180/2304 [I 2022-10-10 18:09:45] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:26 [I 2022-10-10 18:09:45] ...trap/engines/engine.py.229: process: 0.88109 | load: 0.00395 [I 2022-10-10 18:09:45] ...trap/engines/engine.py.230: loss: 0.14428 [I 2022-10-10 18:09:45] ...optimizers/trijoint.py.151: total_norm: 19.045712 [I 2022-10-10 18:09:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 190/2304 [I 2022-10-10 18:09:54] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:27 [I 2022-10-10 18:09:54] ...trap/engines/engine.py.229: process: 0.86751 | load: 0.00349 [I 2022-10-10 18:09:54] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-10 18:09:54] ...optimizers/trijoint.py.151: total_norm: 15.196774 [I 2022-10-10 18:10:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 200/2304 [I 2022-10-10 18:10:02] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:16 [I 2022-10-10 18:10:02] ...trap/engines/engine.py.229: process: 0.87038 | load: 0.00364 [I 2022-10-10 18:10:02] ...trap/engines/engine.py.230: loss: 0.10964 [I 2022-10-10 18:10:02] ...optimizers/trijoint.py.151: total_norm: 36.400379 [I 2022-10-10 18:10:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 210/2304 [I 2022-10-10 18:10:11] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:30:23 [I 2022-10-10 18:10:11] ...trap/engines/engine.py.229: process: 0.88656 | load: 0.00314 [I 2022-10-10 18:10:11] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-10 18:10:11] ...optimizers/trijoint.py.151: total_norm: 13.521344 [I 2022-10-10 18:10:20] ...trap/engines/engine.py.224: train: epoch 81 | batch 220/2304 [I 2022-10-10 18:10:20] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:30:03 [I 2022-10-10 18:10:20] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00309 [I 2022-10-10 18:10:20] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-10 18:10:20] ...optimizers/trijoint.py.151: total_norm: 23.656511 [I 2022-10-10 18:10:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 230/2304 [I 2022-10-10 18:10:28] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:30:02 [I 2022-10-10 18:10:28] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00334 [I 2022-10-10 18:10:28] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-10 18:10:28] ...optimizers/trijoint.py.151: total_norm: 17.809290 [I 2022-10-10 18:10:37] ...trap/engines/engine.py.224: train: epoch 81 | batch 240/2304 [I 2022-10-10 18:10:37] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:48 [I 2022-10-10 18:10:37] ...trap/engines/engine.py.229: process: 0.85372 | load: 0.00316 [I 2022-10-10 18:10:37] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-10 18:10:37] ...optimizers/trijoint.py.151: total_norm: 8.869408 [I 2022-10-10 18:10:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 250/2304 [I 2022-10-10 18:10:46] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:46 [I 2022-10-10 18:10:46] ...trap/engines/engine.py.229: process: 0.87187 | load: 0.00325 [I 2022-10-10 18:10:46] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-10 18:10:46] ...optimizers/trijoint.py.151: total_norm: 10.834464 [I 2022-10-10 18:10:55] ...trap/engines/engine.py.224: train: epoch 81 | batch 260/2304 [I 2022-10-10 18:10:55] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:21 [I 2022-10-10 18:10:55] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00312 [I 2022-10-10 18:10:55] ...trap/engines/engine.py.230: loss: 0.14100 [I 2022-10-10 18:10:55] ...optimizers/trijoint.py.151: total_norm: 30.289890 [I 2022-10-10 18:11:03] ...trap/engines/engine.py.224: train: epoch 81 | batch 270/2304 [I 2022-10-10 18:11:03] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:29:10 [I 2022-10-10 18:11:03] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00320 [I 2022-10-10 18:11:03] ...trap/engines/engine.py.230: loss: 0.15114 [I 2022-10-10 18:11:03] ...optimizers/trijoint.py.151: total_norm: 20.323750 [I 2022-10-10 18:11:12] ...trap/engines/engine.py.224: train: epoch 81 | batch 280/2304 [I 2022-10-10 18:11:12] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:29:11 [I 2022-10-10 18:11:12] ...trap/engines/engine.py.229: process: 0.85956 | load: 0.00325 [I 2022-10-10 18:11:12] ...trap/engines/engine.py.230: loss: 0.15700 [I 2022-10-10 18:11:12] ...optimizers/trijoint.py.151: total_norm: 19.757416 [I 2022-10-10 18:11:20] ...trap/engines/engine.py.224: train: epoch 81 | batch 290/2304 [I 2022-10-10 18:11:20] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:59 [I 2022-10-10 18:11:20] ...trap/engines/engine.py.229: process: 0.86214 | load: 0.00306 [I 2022-10-10 18:11:20] ...trap/engines/engine.py.230: loss: 0.16800 [I 2022-10-10 18:11:20] ...optimizers/trijoint.py.151: total_norm: 12.481196 [I 2022-10-10 18:11:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 300/2304 [I 2022-10-10 18:11:29] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:50 [I 2022-10-10 18:11:29] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00308 [I 2022-10-10 18:11:29] ...trap/engines/engine.py.230: loss: 0.12168 [I 2022-10-10 18:11:29] ...optimizers/trijoint.py.151: total_norm: 7.593953 [I 2022-10-10 18:11:38] ...trap/engines/engine.py.224: train: epoch 81 | batch 310/2304 [I 2022-10-10 18:11:38] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:39 [I 2022-10-10 18:11:38] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00316 [I 2022-10-10 18:11:38] ...trap/engines/engine.py.230: loss: 0.12220 [I 2022-10-10 18:11:38] ...optimizers/trijoint.py.151: total_norm: 13.349006 [I 2022-10-10 18:11:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 320/2304 [I 2022-10-10 18:11:46] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:29 [I 2022-10-10 18:11:46] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00326 [I 2022-10-10 18:11:46] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-10 18:11:46] ...optimizers/trijoint.py.151: total_norm: 22.553320 [I 2022-10-10 18:11:55] ...trap/engines/engine.py.224: train: epoch 81 | batch 330/2304 [I 2022-10-10 18:11:55] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:28:16 [I 2022-10-10 18:11:55] ...trap/engines/engine.py.229: process: 0.85298 | load: 0.00308 [I 2022-10-10 18:11:55] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-10 18:11:55] ...optimizers/trijoint.py.151: total_norm: 15.165208 [I 2022-10-10 18:12:04] ...trap/engines/engine.py.224: train: epoch 81 | batch 340/2304 [I 2022-10-10 18:12:04] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:28:23 [I 2022-10-10 18:12:04] ...trap/engines/engine.py.229: process: 0.88318 | load: 0.00306 [I 2022-10-10 18:12:04] ...trap/engines/engine.py.230: loss: 0.10125 [I 2022-10-10 18:12:04] ...optimizers/trijoint.py.151: total_norm: 30.901093 [I 2022-10-10 18:12:13] ...trap/engines/engine.py.224: train: epoch 81 | batch 350/2304 [I 2022-10-10 18:12:13] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:18 [I 2022-10-10 18:12:13] ...trap/engines/engine.py.229: process: 0.87364 | load: 0.00306 [I 2022-10-10 18:12:13] ...trap/engines/engine.py.230: loss: 0.13382 [I 2022-10-10 18:12:13] ...optimizers/trijoint.py.151: total_norm: 32.725349 [I 2022-10-10 18:12:21] ...trap/engines/engine.py.224: train: epoch 81 | batch 360/2304 [I 2022-10-10 18:12:21] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:27:58 [I 2022-10-10 18:12:21] ...trap/engines/engine.py.229: process: 0.87111 | load: 0.00306 [I 2022-10-10 18:12:21] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-10 18:12:21] ...optimizers/trijoint.py.151: total_norm: 16.767693 [I 2022-10-10 18:12:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 370/2304 [I 2022-10-10 18:12:30] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:56 [I 2022-10-10 18:12:30] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00310 [I 2022-10-10 18:12:30] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-10 18:12:30] ...optimizers/trijoint.py.151: total_norm: 16.436939 [I 2022-10-10 18:12:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 380/2304 [I 2022-10-10 18:12:39] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:42 [I 2022-10-10 18:12:39] ...trap/engines/engine.py.229: process: 0.85158 | load: 0.00308 [I 2022-10-10 18:12:39] ...trap/engines/engine.py.230: loss: 0.18397 [I 2022-10-10 18:12:39] ...optimizers/trijoint.py.151: total_norm: 20.842255 [I 2022-10-10 18:12:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 390/2304 [I 2022-10-10 18:12:47] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:42 [I 2022-10-10 18:12:47] ...trap/engines/engine.py.229: process: 0.85721 | load: 0.00322 [I 2022-10-10 18:12:47] ...trap/engines/engine.py.230: loss: 0.09497 [I 2022-10-10 18:12:47] ...optimizers/trijoint.py.151: total_norm: 9.049196 [I 2022-10-10 18:12:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 400/2304 [I 2022-10-10 18:12:56] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:25 [I 2022-10-10 18:12:56] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00312 [I 2022-10-10 18:12:56] ...trap/engines/engine.py.230: loss: 0.11110 [I 2022-10-10 18:12:56] ...optimizers/trijoint.py.151: total_norm: 15.363326 [I 2022-10-10 18:13:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 410/2304 [I 2022-10-10 18:13:05] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:18 [I 2022-10-10 18:13:05] ...trap/engines/engine.py.229: process: 0.87247 | load: 0.00305 [I 2022-10-10 18:13:05] ...trap/engines/engine.py.230: loss: 0.15374 [I 2022-10-10 18:13:05] ...optimizers/trijoint.py.151: total_norm: 12.180273 [I 2022-10-10 18:13:13] ...trap/engines/engine.py.224: train: epoch 81 | batch 420/2304 [I 2022-10-10 18:13:13] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:27:05 [I 2022-10-10 18:13:13] ...trap/engines/engine.py.229: process: 0.86744 | load: 0.00307 [I 2022-10-10 18:13:13] ...trap/engines/engine.py.230: loss: 0.12545 [I 2022-10-10 18:13:13] ...optimizers/trijoint.py.151: total_norm: 13.158419 [I 2022-10-10 18:13:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 430/2304 [I 2022-10-10 18:13:22] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:27:07 [I 2022-10-10 18:13:22] ...trap/engines/engine.py.229: process: 0.86748 | load: 0.00311 [I 2022-10-10 18:13:22] ...trap/engines/engine.py.230: loss: 0.16156 [I 2022-10-10 18:13:22] ...optimizers/trijoint.py.151: total_norm: 23.936050 [I 2022-10-10 18:13:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 440/2304 [I 2022-10-10 18:13:31] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:55 [I 2022-10-10 18:13:31] ...trap/engines/engine.py.229: process: 0.86828 | load: 0.00324 [I 2022-10-10 18:13:31] ...trap/engines/engine.py.230: loss: 0.21611 [I 2022-10-10 18:13:31] ...optimizers/trijoint.py.151: total_norm: 11.594454 [I 2022-10-10 18:13:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 450/2304 [I 2022-10-10 18:13:39] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:49 [I 2022-10-10 18:13:39] ...trap/engines/engine.py.229: process: 0.88215 | load: 0.00324 [I 2022-10-10 18:13:39] ...trap/engines/engine.py.230: loss: 0.10745 [I 2022-10-10 18:13:39] ...optimizers/trijoint.py.151: total_norm: 17.608677 [I 2022-10-10 18:13:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 460/2304 [I 2022-10-10 18:13:48] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:25 [I 2022-10-10 18:13:48] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00301 [I 2022-10-10 18:13:48] ...trap/engines/engine.py.230: loss: 0.11502 [I 2022-10-10 18:13:48] ...optimizers/trijoint.py.151: total_norm: 22.421577 [I 2022-10-10 18:13:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 470/2304 [I 2022-10-10 18:13:56] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:09 [I 2022-10-10 18:13:56] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00317 [I 2022-10-10 18:13:56] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-10 18:13:56] ...optimizers/trijoint.py.151: total_norm: 17.173473 [I 2022-10-10 18:14:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 480/2304 [I 2022-10-10 18:14:05] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:14 [I 2022-10-10 18:14:05] ...trap/engines/engine.py.229: process: 0.86715 | load: 0.00299 [I 2022-10-10 18:14:05] ...trap/engines/engine.py.230: loss: 0.07567 [I 2022-10-10 18:14:05] ...optimizers/trijoint.py.151: total_norm: 10.289091 [I 2022-10-10 18:14:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 490/2304 [I 2022-10-10 18:14:14] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:57 [I 2022-10-10 18:14:14] ...trap/engines/engine.py.229: process: 0.87176 | load: 0.00295 [I 2022-10-10 18:14:14] ...trap/engines/engine.py.230: loss: 0.13680 [I 2022-10-10 18:14:14] ...optimizers/trijoint.py.151: total_norm: 22.411076 [I 2022-10-10 18:14:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 500/2304 [I 2022-10-10 18:14:22] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:41 [I 2022-10-10 18:14:22] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00304 [I 2022-10-10 18:14:22] ...trap/engines/engine.py.230: loss: 0.10037 [I 2022-10-10 18:14:22] ...optimizers/trijoint.py.151: total_norm: 24.341368 [I 2022-10-10 18:14:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 510/2304 [I 2022-10-10 18:14:31] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:24 [I 2022-10-10 18:14:31] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00296 [I 2022-10-10 18:14:31] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-10 18:14:31] ...optimizers/trijoint.py.151: total_norm: 11.663455 [I 2022-10-10 18:14:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 520/2304 [I 2022-10-10 18:14:39] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:17 [I 2022-10-10 18:14:39] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00295 [I 2022-10-10 18:14:39] ...trap/engines/engine.py.230: loss: 0.10641 [I 2022-10-10 18:14:39] ...optimizers/trijoint.py.151: total_norm: 16.462786 [I 2022-10-10 18:14:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 530/2304 [I 2022-10-10 18:14:48] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:12 [I 2022-10-10 18:14:48] ...trap/engines/engine.py.229: process: 0.86103 | load: 0.00296 [I 2022-10-10 18:14:48] ...trap/engines/engine.py.230: loss: 0.17372 [I 2022-10-10 18:14:48] ...optimizers/trijoint.py.151: total_norm: 34.990444 [I 2022-10-10 18:14:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 540/2304 [I 2022-10-10 18:14:56] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:13 [I 2022-10-10 18:14:56] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00298 [I 2022-10-10 18:14:56] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-10 18:14:56] ...optimizers/trijoint.py.151: total_norm: 10.373589 [I 2022-10-10 18:15:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 550/2304 [I 2022-10-10 18:15:05] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:00 [I 2022-10-10 18:15:05] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00306 [I 2022-10-10 18:15:05] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-10 18:15:05] ...optimizers/trijoint.py.151: total_norm: 83.375160 [I 2022-10-10 18:15:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 560/2304 [I 2022-10-10 18:15:14] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:51 [I 2022-10-10 18:15:14] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00308 [I 2022-10-10 18:15:14] ...trap/engines/engine.py.230: loss: 0.10868 [I 2022-10-10 18:15:14] ...optimizers/trijoint.py.151: total_norm: 9.248825 [I 2022-10-10 18:15:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 570/2304 [I 2022-10-10 18:15:22] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:41 [I 2022-10-10 18:15:22] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00298 [I 2022-10-10 18:15:22] ...trap/engines/engine.py.230: loss: 0.13466 [I 2022-10-10 18:15:22] ...optimizers/trijoint.py.151: total_norm: 15.037498 [I 2022-10-10 18:15:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 580/2304 [I 2022-10-10 18:15:31] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:39 [I 2022-10-10 18:15:31] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00292 [I 2022-10-10 18:15:31] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-10 18:15:31] ...optimizers/trijoint.py.151: total_norm: 11.228707 [I 2022-10-10 18:15:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 590/2304 [I 2022-10-10 18:15:39] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:31 [I 2022-10-10 18:15:39] ...trap/engines/engine.py.229: process: 0.86196 | load: 0.00295 [I 2022-10-10 18:15:39] ...trap/engines/engine.py.230: loss: 0.10837 [I 2022-10-10 18:15:39] ...optimizers/trijoint.py.151: total_norm: 10.124964 [I 2022-10-10 18:15:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 600/2304 [I 2022-10-10 18:15:48] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:26 [I 2022-10-10 18:15:48] ...trap/engines/engine.py.229: process: 0.84761 | load: 0.00295 [I 2022-10-10 18:15:48] ...trap/engines/engine.py.230: loss: 0.20504 [I 2022-10-10 18:15:48] ...optimizers/trijoint.py.151: total_norm: 33.032108 [I 2022-10-10 18:15:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 610/2304 [I 2022-10-10 18:15:57] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:10 [I 2022-10-10 18:15:57] ...trap/engines/engine.py.229: process: 0.84761 | load: 0.00295 [I 2022-10-10 18:15:57] ...trap/engines/engine.py.230: loss: 0.24464 [I 2022-10-10 18:15:57] ...optimizers/trijoint.py.151: total_norm: 13.184570 [I 2022-10-10 18:16:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 620/2304 [I 2022-10-10 18:16:05] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:24:15 [I 2022-10-10 18:16:05] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00309 [I 2022-10-10 18:16:05] ...trap/engines/engine.py.230: loss: 0.13253 [I 2022-10-10 18:16:05] ...optimizers/trijoint.py.151: total_norm: 12.125692 [I 2022-10-10 18:16:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 630/2304 [I 2022-10-10 18:16:14] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:24:06 [I 2022-10-10 18:16:14] ...trap/engines/engine.py.229: process: 0.87463 | load: 0.00402 [I 2022-10-10 18:16:14] ...trap/engines/engine.py.230: loss: 0.08550 [I 2022-10-10 18:16:14] ...optimizers/trijoint.py.151: total_norm: 16.528141 [I 2022-10-10 18:16:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 640/2304 [I 2022-10-10 18:16:23] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:24:08 [I 2022-10-10 18:16:23] ...trap/engines/engine.py.229: process: 0.86467 | load: 0.00315 [I 2022-10-10 18:16:23] ...trap/engines/engine.py.230: loss: 0.15067 [I 2022-10-10 18:16:23] ...optimizers/trijoint.py.151: total_norm: 10.119830 [I 2022-10-10 18:16:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 650/2304 [I 2022-10-10 18:16:31] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:45 [I 2022-10-10 18:16:31] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00309 [I 2022-10-10 18:16:31] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-10 18:16:31] ...optimizers/trijoint.py.151: total_norm: 25.511889 [I 2022-10-10 18:16:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 660/2304 [I 2022-10-10 18:16:40] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:29 [I 2022-10-10 18:16:40] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00313 [I 2022-10-10 18:16:40] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-10 18:16:40] ...optimizers/trijoint.py.151: total_norm: 15.961513 [I 2022-10-10 18:16:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 670/2304 [I 2022-10-10 18:16:48] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:26 [I 2022-10-10 18:16:48] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00309 [I 2022-10-10 18:16:48] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-10 18:16:48] ...optimizers/trijoint.py.151: total_norm: 16.987511 [I 2022-10-10 18:16:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 680/2304 [I 2022-10-10 18:16:57] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:15 [I 2022-10-10 18:16:57] ...trap/engines/engine.py.229: process: 0.85295 | load: 0.00308 [I 2022-10-10 18:16:57] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-10 18:16:57] ...optimizers/trijoint.py.151: total_norm: 17.804760 [I 2022-10-10 18:17:06] ...trap/engines/engine.py.224: train: epoch 81 | batch 690/2304 [I 2022-10-10 18:17:06] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:19 [I 2022-10-10 18:17:06] ...trap/engines/engine.py.229: process: 0.87136 | load: 0.00325 [I 2022-10-10 18:17:06] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-10 18:17:06] ...optimizers/trijoint.py.151: total_norm: 30.875542 [I 2022-10-10 18:17:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 700/2304 [I 2022-10-10 18:17:14] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:23:07 [I 2022-10-10 18:17:14] ...trap/engines/engine.py.229: process: 0.86259 | load: 0.00309 [I 2022-10-10 18:17:14] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-10 18:17:14] ...optimizers/trijoint.py.151: total_norm: 13.381364 [I 2022-10-10 18:17:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 710/2304 [I 2022-10-10 18:17:23] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:23:00 [I 2022-10-10 18:17:23] ...trap/engines/engine.py.229: process: 0.86960 | load: 0.00310 [I 2022-10-10 18:17:23] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-10 18:17:23] ...optimizers/trijoint.py.151: total_norm: 12.613599 [I 2022-10-10 18:17:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 720/2304 [I 2022-10-10 18:17:32] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:53 [I 2022-10-10 18:17:32] ...trap/engines/engine.py.229: process: 0.86819 | load: 0.00315 [I 2022-10-10 18:17:32] ...trap/engines/engine.py.230: loss: 0.13722 [I 2022-10-10 18:17:32] ...optimizers/trijoint.py.151: total_norm: 20.385967 [I 2022-10-10 18:17:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 730/2304 [I 2022-10-10 18:17:40] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:42 [I 2022-10-10 18:17:40] ...trap/engines/engine.py.229: process: 0.87380 | load: 0.00311 [I 2022-10-10 18:17:40] ...trap/engines/engine.py.230: loss: 0.10484 [I 2022-10-10 18:17:40] ...optimizers/trijoint.py.151: total_norm: 25.732887 [I 2022-10-10 18:17:49] ...trap/engines/engine.py.224: train: epoch 81 | batch 740/2304 [I 2022-10-10 18:17:49] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:41 [I 2022-10-10 18:17:49] ...trap/engines/engine.py.229: process: 0.86528 | load: 0.00309 [I 2022-10-10 18:17:49] ...trap/engines/engine.py.230: loss: 0.13268 [I 2022-10-10 18:17:49] ...optimizers/trijoint.py.151: total_norm: 22.615322 [I 2022-10-10 18:17:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 750/2304 [I 2022-10-10 18:17:58] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:21 [I 2022-10-10 18:17:58] ...trap/engines/engine.py.229: process: 0.86697 | load: 0.00314 [I 2022-10-10 18:17:58] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-10 18:17:58] ...optimizers/trijoint.py.151: total_norm: 50.939365 [I 2022-10-10 18:18:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 760/2304 [I 2022-10-10 18:18:07] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:22 [I 2022-10-10 18:18:07] ...trap/engines/engine.py.229: process: 0.86975 | load: 0.00312 [I 2022-10-10 18:18:07] ...trap/engines/engine.py.230: loss: 0.15399 [I 2022-10-10 18:18:07] ...optimizers/trijoint.py.151: total_norm: 21.507872 [I 2022-10-10 18:18:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 770/2304 [I 2022-10-10 18:18:15] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:04 [I 2022-10-10 18:18:15] ...trap/engines/engine.py.229: process: 0.87121 | load: 0.00314 [I 2022-10-10 18:18:15] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-10 18:18:15] ...optimizers/trijoint.py.151: total_norm: 12.586014 [I 2022-10-10 18:18:24] ...trap/engines/engine.py.224: train: epoch 81 | batch 780/2304 [I 2022-10-10 18:18:24] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:22:03 [I 2022-10-10 18:18:24] ...trap/engines/engine.py.229: process: 0.86323 | load: 0.00309 [I 2022-10-10 18:18:24] ...trap/engines/engine.py.230: loss: 0.09829 [I 2022-10-10 18:18:24] ...optimizers/trijoint.py.151: total_norm: 15.286544 [I 2022-10-10 18:18:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 790/2304 [I 2022-10-10 18:18:33] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:57 [I 2022-10-10 18:18:33] ...trap/engines/engine.py.229: process: 0.86911 | load: 0.00308 [I 2022-10-10 18:18:33] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-10 18:18:33] ...optimizers/trijoint.py.151: total_norm: 11.312434 [I 2022-10-10 18:18:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 800/2304 [I 2022-10-10 18:18:41] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:39 [I 2022-10-10 18:18:41] ...trap/engines/engine.py.229: process: 0.87287 | load: 0.00307 [I 2022-10-10 18:18:41] ...trap/engines/engine.py.230: loss: 0.09651 [I 2022-10-10 18:18:41] ...optimizers/trijoint.py.151: total_norm: 16.335022 [I 2022-10-10 18:18:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 810/2304 [I 2022-10-10 18:18:50] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:28 [I 2022-10-10 18:18:50] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00307 [I 2022-10-10 18:18:50] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-10 18:18:50] ...optimizers/trijoint.py.151: total_norm: 20.831928 [I 2022-10-10 18:18:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 820/2304 [I 2022-10-10 18:18:59] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:19 [I 2022-10-10 18:18:59] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00308 [I 2022-10-10 18:18:59] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-10 18:18:59] ...optimizers/trijoint.py.151: total_norm: 12.496736 [I 2022-10-10 18:19:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 830/2304 [I 2022-10-10 18:19:07] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:15 [I 2022-10-10 18:19:07] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00305 [I 2022-10-10 18:19:07] ...trap/engines/engine.py.230: loss: 0.15965 [I 2022-10-10 18:19:07] ...optimizers/trijoint.py.151: total_norm: 52.106720 [I 2022-10-10 18:19:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 840/2304 [I 2022-10-10 18:19:16] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:06 [I 2022-10-10 18:19:16] ...trap/engines/engine.py.229: process: 0.86285 | load: 0.00345 [I 2022-10-10 18:19:16] ...trap/engines/engine.py.230: loss: 0.12216 [I 2022-10-10 18:19:16] ...optimizers/trijoint.py.151: total_norm: 19.315372 [I 2022-10-10 18:19:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 850/2304 [I 2022-10-10 18:19:25] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:21:07 [I 2022-10-10 18:19:25] ...trap/engines/engine.py.229: process: 0.87871 | load: 0.00315 [I 2022-10-10 18:19:25] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-10 18:19:25] ...optimizers/trijoint.py.151: total_norm: 12.195346 [I 2022-10-10 18:19:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 860/2304 [I 2022-10-10 18:19:33] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:55 [I 2022-10-10 18:19:33] ...trap/engines/engine.py.229: process: 0.87979 | load: 0.00371 [I 2022-10-10 18:19:33] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-10 18:19:33] ...optimizers/trijoint.py.151: total_norm: 12.873476 [I 2022-10-10 18:19:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 870/2304 [I 2022-10-10 18:19:42] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:50 [I 2022-10-10 18:19:42] ...trap/engines/engine.py.229: process: 0.88411 | load: 0.00312 [I 2022-10-10 18:19:42] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-10 18:19:42] ...optimizers/trijoint.py.151: total_norm: 15.999100 [I 2022-10-10 18:19:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 880/2304 [I 2022-10-10 18:19:51] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:25 [I 2022-10-10 18:19:51] ...trap/engines/engine.py.229: process: 0.84602 | load: 0.00310 [I 2022-10-10 18:19:51] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-10 18:19:51] ...optimizers/trijoint.py.151: total_norm: 9.368940 [I 2022-10-10 18:19:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 890/2304 [I 2022-10-10 18:19:59] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:25 [I 2022-10-10 18:19:59] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00317 [I 2022-10-10 18:19:59] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-10 18:19:59] ...optimizers/trijoint.py.151: total_norm: 22.092012 [I 2022-10-10 18:20:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 900/2304 [I 2022-10-10 18:20:08] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:04 [I 2022-10-10 18:20:08] ...trap/engines/engine.py.229: process: 0.85312 | load: 0.00291 [I 2022-10-10 18:20:08] ...trap/engines/engine.py.230: loss: 0.18897 [I 2022-10-10 18:20:08] ...optimizers/trijoint.py.151: total_norm: 42.755302 [I 2022-10-10 18:20:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 910/2304 [I 2022-10-10 18:20:17] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:00 [I 2022-10-10 18:20:17] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00301 [I 2022-10-10 18:20:17] ...trap/engines/engine.py.230: loss: 0.08788 [I 2022-10-10 18:20:17] ...optimizers/trijoint.py.151: total_norm: 10.679580 [I 2022-10-10 18:20:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 920/2304 [I 2022-10-10 18:20:25] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:46 [I 2022-10-10 18:20:25] ...trap/engines/engine.py.229: process: 0.86495 | load: 0.00295 [I 2022-10-10 18:20:25] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-10 18:20:25] ...optimizers/trijoint.py.151: total_norm: 11.924979 [I 2022-10-10 18:20:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 930/2304 [I 2022-10-10 18:20:34] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:38 [I 2022-10-10 18:20:34] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00298 [I 2022-10-10 18:20:34] ...trap/engines/engine.py.230: loss: 0.09416 [I 2022-10-10 18:20:34] ...optimizers/trijoint.py.151: total_norm: 8.940023 [I 2022-10-10 18:20:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 940/2304 [I 2022-10-10 18:20:42] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:23 [I 2022-10-10 18:20:42] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00297 [I 2022-10-10 18:20:42] ...trap/engines/engine.py.230: loss: 0.14979 [I 2022-10-10 18:20:42] ...optimizers/trijoint.py.151: total_norm: 9.722649 [I 2022-10-10 18:20:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 950/2304 [I 2022-10-10 18:20:51] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:25 [I 2022-10-10 18:20:51] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00298 [I 2022-10-10 18:20:51] ...trap/engines/engine.py.230: loss: 0.16644 [I 2022-10-10 18:20:51] ...optimizers/trijoint.py.151: total_norm: 12.347804 [I 2022-10-10 18:21:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 960/2304 [I 2022-10-10 18:21:00] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:04 [I 2022-10-10 18:21:00] ...trap/engines/engine.py.229: process: 0.84392 | load: 0.00298 [I 2022-10-10 18:21:00] ...trap/engines/engine.py.230: loss: 0.08784 [I 2022-10-10 18:21:00] ...optimizers/trijoint.py.151: total_norm: 12.746683 [I 2022-10-10 18:21:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 970/2304 [I 2022-10-10 18:21:08] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:19:06 [I 2022-10-10 18:21:08] ...trap/engines/engine.py.229: process: 0.86556 | load: 0.00298 [I 2022-10-10 18:21:08] ...trap/engines/engine.py.230: loss: 0.12784 [I 2022-10-10 18:21:08] ...optimizers/trijoint.py.151: total_norm: 17.744904 [I 2022-10-10 18:21:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 980/2304 [I 2022-10-10 18:21:17] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:48 [I 2022-10-10 18:21:17] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00306 [I 2022-10-10 18:21:17] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-10 18:21:17] ...optimizers/trijoint.py.151: total_norm: 12.000076 [I 2022-10-10 18:21:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 990/2304 [I 2022-10-10 18:21:25] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:39 [I 2022-10-10 18:21:25] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00300 [I 2022-10-10 18:21:25] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-10 18:21:25] ...optimizers/trijoint.py.151: total_norm: 10.946915 [I 2022-10-10 18:21:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1000/2304 [I 2022-10-10 18:21:34] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:41 [I 2022-10-10 18:21:34] ...trap/engines/engine.py.229: process: 0.87050 | load: 0.00318 [I 2022-10-10 18:21:34] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-10 18:21:34] ...optimizers/trijoint.py.151: total_norm: 14.061222 [I 2022-10-10 18:21:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 1010/2304 [I 2022-10-10 18:21:42] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:29 [I 2022-10-10 18:21:42] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00308 [I 2022-10-10 18:21:42] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-10 18:21:42] ...optimizers/trijoint.py.151: total_norm: 21.063242 [I 2022-10-10 18:21:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1020/2304 [I 2022-10-10 18:21:51] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:32 [I 2022-10-10 18:21:51] ...trap/engines/engine.py.229: process: 0.87414 | load: 0.00299 [I 2022-10-10 18:21:51] ...trap/engines/engine.py.230: loss: 0.17241 [I 2022-10-10 18:21:51] ...optimizers/trijoint.py.151: total_norm: 23.775547 [I 2022-10-10 18:22:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1030/2304 [I 2022-10-10 18:22:00] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:11 [I 2022-10-10 18:22:00] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00294 [I 2022-10-10 18:22:00] ...trap/engines/engine.py.230: loss: 0.14784 [I 2022-10-10 18:22:00] ...optimizers/trijoint.py.151: total_norm: 8.673869 [I 2022-10-10 18:22:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1040/2304 [I 2022-10-10 18:22:08] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:59 [I 2022-10-10 18:22:08] ...trap/engines/engine.py.229: process: 0.85825 | load: 0.00306 [I 2022-10-10 18:22:08] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-10 18:22:08] ...optimizers/trijoint.py.151: total_norm: 23.411627 [I 2022-10-10 18:22:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 1050/2304 [I 2022-10-10 18:22:17] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:52 [I 2022-10-10 18:22:17] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00306 [I 2022-10-10 18:22:17] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-10 18:22:17] ...optimizers/trijoint.py.151: total_norm: 17.215822 [I 2022-10-10 18:22:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1060/2304 [I 2022-10-10 18:22:26] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:46 [I 2022-10-10 18:22:26] ...trap/engines/engine.py.229: process: 0.86439 | load: 0.00303 [I 2022-10-10 18:22:26] ...trap/engines/engine.py.230: loss: 0.15160 [I 2022-10-10 18:22:26] ...optimizers/trijoint.py.151: total_norm: 23.478947 [I 2022-10-10 18:22:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1070/2304 [I 2022-10-10 18:22:34] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:33 [I 2022-10-10 18:22:34] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00311 [I 2022-10-10 18:22:34] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-10 18:22:34] ...optimizers/trijoint.py.151: total_norm: 9.232668 [I 2022-10-10 18:22:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 1080/2304 [I 2022-10-10 18:22:43] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:31 [I 2022-10-10 18:22:43] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00295 [I 2022-10-10 18:22:43] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-10 18:22:43] ...optimizers/trijoint.py.151: total_norm: 16.808624 [I 2022-10-10 18:22:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1090/2304 [I 2022-10-10 18:22:51] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:16 [I 2022-10-10 18:22:51] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00297 [I 2022-10-10 18:22:51] ...trap/engines/engine.py.230: loss: 0.10253 [I 2022-10-10 18:22:51] ...optimizers/trijoint.py.151: total_norm: 8.630566 [I 2022-10-10 18:23:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1100/2304 [I 2022-10-10 18:23:00] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:08 [I 2022-10-10 18:23:00] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00308 [I 2022-10-10 18:23:00] ...trap/engines/engine.py.230: loss: 0.10183 [I 2022-10-10 18:23:00] ...optimizers/trijoint.py.151: total_norm: 15.240829 [I 2022-10-10 18:23:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1110/2304 [I 2022-10-10 18:23:08] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:59 [I 2022-10-10 18:23:08] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00296 [I 2022-10-10 18:23:08] ...trap/engines/engine.py.230: loss: 0.10329 [I 2022-10-10 18:23:08] ...optimizers/trijoint.py.151: total_norm: 15.675576 [I 2022-10-10 18:23:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 1120/2304 [I 2022-10-10 18:23:17] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:55 [I 2022-10-10 18:23:17] ...trap/engines/engine.py.229: process: 0.85735 | load: 0.00303 [I 2022-10-10 18:23:17] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-10 18:23:17] ...optimizers/trijoint.py.151: total_norm: 32.180210 [I 2022-10-10 18:23:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1130/2304 [I 2022-10-10 18:23:26] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:48 [I 2022-10-10 18:23:26] ...trap/engines/engine.py.229: process: 0.85649 | load: 0.00312 [I 2022-10-10 18:23:26] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-10 18:23:26] ...optimizers/trijoint.py.151: total_norm: 23.564585 [I 2022-10-10 18:23:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1140/2304 [I 2022-10-10 18:23:34] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:36 [I 2022-10-10 18:23:34] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00301 [I 2022-10-10 18:23:34] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-10 18:23:34] ...optimizers/trijoint.py.151: total_norm: 27.320793 [I 2022-10-10 18:23:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 1150/2304 [I 2022-10-10 18:23:43] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:28 [I 2022-10-10 18:23:43] ...trap/engines/engine.py.229: process: 0.86425 | load: 0.00298 [I 2022-10-10 18:23:43] ...trap/engines/engine.py.230: loss: 0.14865 [I 2022-10-10 18:23:43] ...optimizers/trijoint.py.151: total_norm: 8.775460 [I 2022-10-10 18:23:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1160/2304 [I 2022-10-10 18:23:51] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:18 [I 2022-10-10 18:23:51] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00296 [I 2022-10-10 18:23:51] ...trap/engines/engine.py.230: loss: 0.11114 [I 2022-10-10 18:23:51] ...optimizers/trijoint.py.151: total_norm: 36.370453 [I 2022-10-10 18:24:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1170/2304 [I 2022-10-10 18:24:00] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:17 [I 2022-10-10 18:24:00] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00314 [I 2022-10-10 18:24:00] ...trap/engines/engine.py.230: loss: 0.19489 [I 2022-10-10 18:24:00] ...optimizers/trijoint.py.151: total_norm: 37.460991 [I 2022-10-10 18:24:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1180/2304 [I 2022-10-10 18:24:09] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:01 [I 2022-10-10 18:24:09] ...trap/engines/engine.py.229: process: 0.86749 | load: 0.00311 [I 2022-10-10 18:24:09] ...trap/engines/engine.py.230: loss: 0.10995 [I 2022-10-10 18:24:09] ...optimizers/trijoint.py.151: total_norm: 20.560928 [I 2022-10-10 18:24:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 1190/2304 [I 2022-10-10 18:24:17] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:52 [I 2022-10-10 18:24:17] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00309 [I 2022-10-10 18:24:17] ...trap/engines/engine.py.230: loss: 0.19232 [I 2022-10-10 18:24:17] ...optimizers/trijoint.py.151: total_norm: 24.494656 [I 2022-10-10 18:24:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1200/2304 [I 2022-10-10 18:24:26] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:45 [I 2022-10-10 18:24:26] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00311 [I 2022-10-10 18:24:26] ...trap/engines/engine.py.230: loss: 0.17864 [I 2022-10-10 18:24:26] ...optimizers/trijoint.py.151: total_norm: 30.104248 [I 2022-10-10 18:24:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1210/2304 [I 2022-10-10 18:24:34] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:35 [I 2022-10-10 18:24:34] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00315 [I 2022-10-10 18:24:34] ...trap/engines/engine.py.230: loss: 0.19759 [I 2022-10-10 18:24:34] ...optimizers/trijoint.py.151: total_norm: 24.248533 [I 2022-10-10 18:24:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 1220/2304 [I 2022-10-10 18:24:43] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:27 [I 2022-10-10 18:24:43] ...trap/engines/engine.py.229: process: 0.84976 | load: 0.00311 [I 2022-10-10 18:24:43] ...trap/engines/engine.py.230: loss: 0.15715 [I 2022-10-10 18:24:43] ...optimizers/trijoint.py.151: total_norm: 69.065445 [I 2022-10-10 18:24:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 1230/2304 [I 2022-10-10 18:24:52] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:26 [I 2022-10-10 18:24:52] ...trap/engines/engine.py.229: process: 0.86928 | load: 0.00320 [I 2022-10-10 18:24:52] ...trap/engines/engine.py.230: loss: 0.16151 [I 2022-10-10 18:24:52] ...optimizers/trijoint.py.151: total_norm: 37.559433 [I 2022-10-10 18:25:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1240/2304 [I 2022-10-10 18:25:00] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:20 [I 2022-10-10 18:25:00] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00321 [I 2022-10-10 18:25:00] ...trap/engines/engine.py.230: loss: 0.16916 [I 2022-10-10 18:25:00] ...optimizers/trijoint.py.151: total_norm: 19.653316 [I 2022-10-10 18:25:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1250/2304 [I 2022-10-10 18:25:09] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:12 [I 2022-10-10 18:25:09] ...trap/engines/engine.py.229: process: 0.87551 | load: 0.00311 [I 2022-10-10 18:25:09] ...trap/engines/engine.py.230: loss: 0.15473 [I 2022-10-10 18:25:09] ...optimizers/trijoint.py.151: total_norm: 16.743242 [I 2022-10-10 18:25:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 1260/2304 [I 2022-10-10 18:25:18] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:58 [I 2022-10-10 18:25:18] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00312 [I 2022-10-10 18:25:18] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-10 18:25:18] ...optimizers/trijoint.py.151: total_norm: 17.525146 [I 2022-10-10 18:25:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1270/2304 [I 2022-10-10 18:25:26] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:55 [I 2022-10-10 18:25:26] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00308 [I 2022-10-10 18:25:26] ...trap/engines/engine.py.230: loss: 0.08742 [I 2022-10-10 18:25:26] ...optimizers/trijoint.py.151: total_norm: 12.863633 [I 2022-10-10 18:25:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 1280/2304 [I 2022-10-10 18:25:35] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:50 [I 2022-10-10 18:25:35] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00308 [I 2022-10-10 18:25:35] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-10 18:25:35] ...optimizers/trijoint.py.151: total_norm: 12.811432 [I 2022-10-10 18:25:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 1290/2304 [I 2022-10-10 18:25:44] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:36 [I 2022-10-10 18:25:44] ...trap/engines/engine.py.229: process: 0.85241 | load: 0.00309 [I 2022-10-10 18:25:44] ...trap/engines/engine.py.230: loss: 0.17955 [I 2022-10-10 18:25:44] ...optimizers/trijoint.py.151: total_norm: 11.005479 [I 2022-10-10 18:25:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 1300/2304 [I 2022-10-10 18:25:52] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:28 [I 2022-10-10 18:25:52] ...trap/engines/engine.py.229: process: 0.86564 | load: 0.00308 [I 2022-10-10 18:25:52] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-10 18:25:52] ...optimizers/trijoint.py.151: total_norm: 46.588348 [I 2022-10-10 18:26:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 1310/2304 [I 2022-10-10 18:26:01] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:14:19 [I 2022-10-10 18:26:01] ...trap/engines/engine.py.229: process: 0.87300 | load: 0.00309 [I 2022-10-10 18:26:01] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-10 18:26:01] ...optimizers/trijoint.py.151: total_norm: 23.325903 [I 2022-10-10 18:26:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 1320/2304 [I 2022-10-10 18:26:10] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:08 [I 2022-10-10 18:26:10] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00308 [I 2022-10-10 18:26:10] ...trap/engines/engine.py.230: loss: 0.15066 [I 2022-10-10 18:26:10] ...optimizers/trijoint.py.151: total_norm: 9.592100 [I 2022-10-10 18:26:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 1330/2304 [I 2022-10-10 18:26:18] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:14:00 [I 2022-10-10 18:26:18] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00311 [I 2022-10-10 18:26:18] ...trap/engines/engine.py.230: loss: 0.13536 [I 2022-10-10 18:26:18] ...optimizers/trijoint.py.151: total_norm: 17.059423 [I 2022-10-10 18:26:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 1340/2304 [I 2022-10-10 18:26:27] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:52 [I 2022-10-10 18:26:27] ...trap/engines/engine.py.229: process: 0.85981 | load: 0.00315 [I 2022-10-10 18:26:27] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-10 18:26:27] ...optimizers/trijoint.py.151: total_norm: 9.743630 [I 2022-10-10 18:26:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 1350/2304 [I 2022-10-10 18:26:36] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:46 [I 2022-10-10 18:26:36] ...trap/engines/engine.py.229: process: 0.87475 | load: 0.00312 [I 2022-10-10 18:26:36] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-10 18:26:36] ...optimizers/trijoint.py.151: total_norm: 14.868672 [I 2022-10-10 18:26:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 1360/2304 [I 2022-10-10 18:26:44] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:38 [I 2022-10-10 18:26:44] ...trap/engines/engine.py.229: process: 0.87832 | load: 0.00319 [I 2022-10-10 18:26:44] ...trap/engines/engine.py.230: loss: 0.16983 [I 2022-10-10 18:26:44] ...optimizers/trijoint.py.151: total_norm: 33.493233 [I 2022-10-10 18:26:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 1370/2304 [I 2022-10-10 18:26:53] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:34 [I 2022-10-10 18:26:53] ...trap/engines/engine.py.229: process: 0.86845 | load: 0.00313 [I 2022-10-10 18:26:53] ...trap/engines/engine.py.230: loss: 0.12513 [I 2022-10-10 18:26:53] ...optimizers/trijoint.py.151: total_norm: 18.099972 [I 2022-10-10 18:27:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 1380/2304 [I 2022-10-10 18:27:02] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:17 [I 2022-10-10 18:27:02] ...trap/engines/engine.py.229: process: 0.86461 | load: 0.00312 [I 2022-10-10 18:27:02] ...trap/engines/engine.py.230: loss: 0.15285 [I 2022-10-10 18:27:02] ...optimizers/trijoint.py.151: total_norm: 18.032909 [I 2022-10-10 18:27:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 1390/2304 [I 2022-10-10 18:27:10] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:13 [I 2022-10-10 18:27:10] ...trap/engines/engine.py.229: process: 0.87573 | load: 0.00310 [I 2022-10-10 18:27:10] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-10 18:27:10] ...optimizers/trijoint.py.151: total_norm: 15.898921 [I 2022-10-10 18:27:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 1400/2304 [I 2022-10-10 18:27:19] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:13:00 [I 2022-10-10 18:27:19] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00311 [I 2022-10-10 18:27:19] ...trap/engines/engine.py.230: loss: 0.11182 [I 2022-10-10 18:27:19] ...optimizers/trijoint.py.151: total_norm: 14.271337 [I 2022-10-10 18:27:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 1410/2304 [I 2022-10-10 18:27:28] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:48 [I 2022-10-10 18:27:28] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00313 [I 2022-10-10 18:27:28] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-10 18:27:28] ...optimizers/trijoint.py.151: total_norm: 14.822560 [I 2022-10-10 18:27:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 1420/2304 [I 2022-10-10 18:27:36] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:42 [I 2022-10-10 18:27:36] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00331 [I 2022-10-10 18:27:36] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-10 18:27:36] ...optimizers/trijoint.py.151: total_norm: 12.038743 [I 2022-10-10 18:27:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 1430/2304 [I 2022-10-10 18:27:45] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:37 [I 2022-10-10 18:27:45] ...trap/engines/engine.py.229: process: 0.86467 | load: 0.00309 [I 2022-10-10 18:27:45] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-10 18:27:45] ...optimizers/trijoint.py.151: total_norm: 12.258135 [I 2022-10-10 18:27:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 1440/2304 [I 2022-10-10 18:27:54] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:26 [I 2022-10-10 18:27:54] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00304 [I 2022-10-10 18:27:54] ...trap/engines/engine.py.230: loss: 0.09676 [I 2022-10-10 18:27:54] ...optimizers/trijoint.py.151: total_norm: 11.671837 [I 2022-10-10 18:28:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 1450/2304 [I 2022-10-10 18:28:22] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:30:17 [I 2022-10-10 18:28:22] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00312 [I 2022-10-10 18:28:22] ...trap/engines/engine.py.230: loss: 0.10062 [I 2022-10-10 18:28:22] ...optimizers/trijoint.py.151: total_norm: 9.255057 [I 2022-10-10 18:28:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 1460/2304 [I 2022-10-10 18:28:30] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:14:02 [I 2022-10-10 18:28:30] ...trap/engines/engine.py.229: process: 0.86186 | load: 0.00310 [I 2022-10-10 18:28:30] ...trap/engines/engine.py.230: loss: 0.16523 [I 2022-10-10 18:28:30] ...optimizers/trijoint.py.151: total_norm: 16.249632 [I 2022-10-10 18:28:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 1470/2304 [I 2022-10-10 18:28:39] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:12:13 [I 2022-10-10 18:28:39] ...trap/engines/engine.py.229: process: 0.87949 | load: 0.00350 [I 2022-10-10 18:28:39] ...trap/engines/engine.py.230: loss: 0.10278 [I 2022-10-10 18:28:39] ...optimizers/trijoint.py.151: total_norm: 25.274860 [I 2022-10-10 18:28:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 1480/2304 [I 2022-10-10 18:28:48] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:51 [I 2022-10-10 18:28:48] ...trap/engines/engine.py.229: process: 0.86160 | load: 0.00317 [I 2022-10-10 18:28:48] ...trap/engines/engine.py.230: loss: 0.12776 [I 2022-10-10 18:28:48] ...optimizers/trijoint.py.151: total_norm: 17.275213 [I 2022-10-10 18:28:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 1490/2304 [I 2022-10-10 18:28:56] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:11:41 [I 2022-10-10 18:28:56] ...trap/engines/engine.py.229: process: 0.86577 | load: 0.00300 [I 2022-10-10 18:28:56] ...trap/engines/engine.py.230: loss: 0.12628 [I 2022-10-10 18:28:56] ...optimizers/trijoint.py.151: total_norm: 14.115270 [I 2022-10-10 18:29:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 1500/2304 [I 2022-10-10 18:29:05] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:35 [I 2022-10-10 18:29:05] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00318 [I 2022-10-10 18:29:05] ...trap/engines/engine.py.230: loss: 0.14023 [I 2022-10-10 18:29:05] ...optimizers/trijoint.py.151: total_norm: 16.098400 [I 2022-10-10 18:29:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 1510/2304 [I 2022-10-10 18:29:14] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:27 [I 2022-10-10 18:29:14] ...trap/engines/engine.py.229: process: 0.87692 | load: 0.00312 [I 2022-10-10 18:29:14] ...trap/engines/engine.py.230: loss: 0.10451 [I 2022-10-10 18:29:14] ...optimizers/trijoint.py.151: total_norm: 18.109219 [I 2022-10-10 18:29:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 1520/2304 [I 2022-10-10 18:29:22] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:11:18 [I 2022-10-10 18:29:22] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00309 [I 2022-10-10 18:29:22] ...trap/engines/engine.py.230: loss: 0.13921 [I 2022-10-10 18:29:22] ...optimizers/trijoint.py.151: total_norm: 26.041002 [I 2022-10-10 18:29:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 1530/2304 [I 2022-10-10 18:29:31] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:11:08 [I 2022-10-10 18:29:31] ...trap/engines/engine.py.229: process: 0.87332 | load: 0.00405 [I 2022-10-10 18:29:31] ...trap/engines/engine.py.230: loss: 0.16191 [I 2022-10-10 18:29:31] ...optimizers/trijoint.py.151: total_norm: 24.636919 [I 2022-10-10 18:29:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 1540/2304 [I 2022-10-10 18:29:40] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:54 [I 2022-10-10 18:29:40] ...trap/engines/engine.py.229: process: 0.86669 | load: 0.00305 [I 2022-10-10 18:29:40] ...trap/engines/engine.py.230: loss: 0.13987 [I 2022-10-10 18:29:40] ...optimizers/trijoint.py.151: total_norm: 15.966758 [I 2022-10-10 18:29:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 1550/2304 [I 2022-10-10 18:29:48] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:49 [I 2022-10-10 18:29:48] ...trap/engines/engine.py.229: process: 0.87110 | load: 0.00313 [I 2022-10-10 18:29:48] ...trap/engines/engine.py.230: loss: 0.19849 [I 2022-10-10 18:29:48] ...optimizers/trijoint.py.151: total_norm: 15.189593 [I 2022-10-10 18:29:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 1560/2304 [I 2022-10-10 18:29:57] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:40 [I 2022-10-10 18:29:57] ...trap/engines/engine.py.229: process: 0.86276 | load: 0.00312 [I 2022-10-10 18:29:57] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-10 18:29:57] ...optimizers/trijoint.py.151: total_norm: 13.417451 [I 2022-10-10 18:30:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 1570/2304 [I 2022-10-10 18:30:05] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:32 [I 2022-10-10 18:30:05] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00327 [I 2022-10-10 18:30:05] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-10 18:30:05] ...optimizers/trijoint.py.151: total_norm: 12.311168 [I 2022-10-10 18:30:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 1580/2304 [I 2022-10-10 18:30:14] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:28 [I 2022-10-10 18:30:14] ...trap/engines/engine.py.229: process: 0.88708 | load: 0.00309 [I 2022-10-10 18:30:14] ...trap/engines/engine.py.230: loss: 0.07821 [I 2022-10-10 18:30:14] ...optimizers/trijoint.py.151: total_norm: 8.927112 [I 2022-10-10 18:30:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 1590/2304 [I 2022-10-10 18:30:23] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:10:12 [I 2022-10-10 18:30:23] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00317 [I 2022-10-10 18:30:23] ...trap/engines/engine.py.230: loss: 0.13407 [I 2022-10-10 18:30:23] ...optimizers/trijoint.py.151: total_norm: 29.836773 [I 2022-10-10 18:30:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 1600/2304 [I 2022-10-10 18:30:31] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:10:05 [I 2022-10-10 18:30:31] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00311 [I 2022-10-10 18:30:31] ...trap/engines/engine.py.230: loss: 0.11481 [I 2022-10-10 18:30:31] ...optimizers/trijoint.py.151: total_norm: 10.474250 [I 2022-10-10 18:30:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 1610/2304 [I 2022-10-10 18:30:40] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:58 [I 2022-10-10 18:30:40] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00294 [I 2022-10-10 18:30:40] ...trap/engines/engine.py.230: loss: 0.14499 [I 2022-10-10 18:30:40] ...optimizers/trijoint.py.151: total_norm: 17.986063 [I 2022-10-10 18:30:49] ...trap/engines/engine.py.224: train: epoch 81 | batch 1620/2304 [I 2022-10-10 18:30:49] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:52 [I 2022-10-10 18:30:49] ...trap/engines/engine.py.229: process: 0.86484 | load: 0.00316 [I 2022-10-10 18:30:49] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-10 18:30:49] ...optimizers/trijoint.py.151: total_norm: 13.722707 [I 2022-10-10 18:30:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 1630/2304 [I 2022-10-10 18:30:57] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:44 [I 2022-10-10 18:30:57] ...trap/engines/engine.py.229: process: 0.85306 | load: 0.00312 [I 2022-10-10 18:30:57] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-10 18:30:57] ...optimizers/trijoint.py.151: total_norm: 19.484526 [I 2022-10-10 18:31:06] ...trap/engines/engine.py.224: train: epoch 81 | batch 1640/2304 [I 2022-10-10 18:31:06] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:35 [I 2022-10-10 18:31:06] ...trap/engines/engine.py.229: process: 0.85806 | load: 0.00311 [I 2022-10-10 18:31:06] ...trap/engines/engine.py.230: loss: 0.15043 [I 2022-10-10 18:31:06] ...optimizers/trijoint.py.151: total_norm: 11.416776 [I 2022-10-10 18:31:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 1650/2304 [I 2022-10-10 18:31:15] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:24 [I 2022-10-10 18:31:15] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00311 [I 2022-10-10 18:31:15] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-10 18:31:15] ...optimizers/trijoint.py.151: total_norm: 12.434794 [I 2022-10-10 18:31:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 1660/2304 [I 2022-10-10 18:31:23] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:21 [I 2022-10-10 18:31:23] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00313 [I 2022-10-10 18:31:23] ...trap/engines/engine.py.230: loss: 0.16043 [I 2022-10-10 18:31:23] ...optimizers/trijoint.py.151: total_norm: 8.969999 [I 2022-10-10 18:31:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 1670/2304 [I 2022-10-10 18:31:32] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:09:10 [I 2022-10-10 18:31:32] ...trap/engines/engine.py.229: process: 0.87105 | load: 0.00312 [I 2022-10-10 18:31:32] ...trap/engines/engine.py.230: loss: 0.10642 [I 2022-10-10 18:31:32] ...optimizers/trijoint.py.151: total_norm: 27.785818 [I 2022-10-10 18:31:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 1680/2304 [I 2022-10-10 18:31:41] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:58 [I 2022-10-10 18:31:41] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00311 [I 2022-10-10 18:31:41] ...trap/engines/engine.py.230: loss: 0.09629 [I 2022-10-10 18:31:41] ...optimizers/trijoint.py.151: total_norm: 12.144571 [I 2022-10-10 18:31:49] ...trap/engines/engine.py.224: train: epoch 81 | batch 1690/2304 [I 2022-10-10 18:31:49] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:49 [I 2022-10-10 18:31:49] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00312 [I 2022-10-10 18:31:49] ...trap/engines/engine.py.230: loss: 0.18247 [I 2022-10-10 18:31:49] ...optimizers/trijoint.py.151: total_norm: 33.589916 [I 2022-10-10 18:31:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 1700/2304 [I 2022-10-10 18:31:58] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:44 [I 2022-10-10 18:31:58] ...trap/engines/engine.py.229: process: 0.87986 | load: 0.00318 [I 2022-10-10 18:31:58] ...trap/engines/engine.py.230: loss: 0.13356 [I 2022-10-10 18:31:58] ...optimizers/trijoint.py.151: total_norm: 12.212324 [I 2022-10-10 18:32:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 1710/2304 [I 2022-10-10 18:32:07] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:33 [I 2022-10-10 18:32:07] ...trap/engines/engine.py.229: process: 0.86657 | load: 0.00306 [I 2022-10-10 18:32:07] ...trap/engines/engine.py.230: loss: 0.17520 [I 2022-10-10 18:32:07] ...optimizers/trijoint.py.151: total_norm: 9.948256 [I 2022-10-10 18:32:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 1720/2304 [I 2022-10-10 18:32:15] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:08:22 [I 2022-10-10 18:32:15] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00310 [I 2022-10-10 18:32:15] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-10 18:32:15] ...optimizers/trijoint.py.151: total_norm: 19.666901 [I 2022-10-10 18:32:24] ...trap/engines/engine.py.224: train: epoch 81 | batch 1730/2304 [I 2022-10-10 18:32:24] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:08:15 [I 2022-10-10 18:32:24] ...trap/engines/engine.py.229: process: 0.86364 | load: 0.00315 [I 2022-10-10 18:32:24] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-10 18:32:24] ...optimizers/trijoint.py.151: total_norm: 18.588934 [I 2022-10-10 18:32:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 1740/2304 [I 2022-10-10 18:32:33] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:08:07 [I 2022-10-10 18:32:33] ...trap/engines/engine.py.229: process: 0.86956 | load: 0.00306 [I 2022-10-10 18:32:33] ...trap/engines/engine.py.230: loss: 0.13170 [I 2022-10-10 18:32:33] ...optimizers/trijoint.py.151: total_norm: 16.584463 [I 2022-10-10 18:32:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 1750/2304 [I 2022-10-10 18:32:41] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:57 [I 2022-10-10 18:32:41] ...trap/engines/engine.py.229: process: 0.86436 | load: 0.00307 [I 2022-10-10 18:32:41] ...trap/engines/engine.py.230: loss: 0.10650 [I 2022-10-10 18:32:41] ...optimizers/trijoint.py.151: total_norm: 21.015762 [I 2022-10-10 18:32:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 1760/2304 [I 2022-10-10 18:32:50] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:49 [I 2022-10-10 18:32:50] ...trap/engines/engine.py.229: process: 0.88221 | load: 0.00312 [I 2022-10-10 18:32:50] ...trap/engines/engine.py.230: loss: 0.09652 [I 2022-10-10 18:32:50] ...optimizers/trijoint.py.151: total_norm: 8.669446 [I 2022-10-10 18:32:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 1770/2304 [I 2022-10-10 18:32:59] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:39 [I 2022-10-10 18:32:59] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00309 [I 2022-10-10 18:32:59] ...trap/engines/engine.py.230: loss: 0.13633 [I 2022-10-10 18:32:59] ...optimizers/trijoint.py.151: total_norm: 10.759622 [I 2022-10-10 18:33:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 1780/2304 [I 2022-10-10 18:33:07] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:32 [I 2022-10-10 18:33:07] ...trap/engines/engine.py.229: process: 0.86591 | load: 0.00313 [I 2022-10-10 18:33:07] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-10 18:33:07] ...optimizers/trijoint.py.151: total_norm: 9.659513 [I 2022-10-10 18:33:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 1790/2304 [I 2022-10-10 18:33:16] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:07:23 [I 2022-10-10 18:33:16] ...trap/engines/engine.py.229: process: 0.85656 | load: 0.00323 [I 2022-10-10 18:33:16] ...trap/engines/engine.py.230: loss: 0.10155 [I 2022-10-10 18:33:16] ...optimizers/trijoint.py.151: total_norm: 8.001519 [I 2022-10-10 18:33:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 1800/2304 [I 2022-10-10 18:33:25] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:07:14 [I 2022-10-10 18:33:25] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00310 [I 2022-10-10 18:33:25] ...trap/engines/engine.py.230: loss: 0.08584 [I 2022-10-10 18:33:25] ...optimizers/trijoint.py.151: total_norm: 14.074704 [I 2022-10-10 18:33:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 1810/2304 [I 2022-10-10 18:33:33] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:07:05 [I 2022-10-10 18:33:33] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00313 [I 2022-10-10 18:33:33] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-10 18:33:33] ...optimizers/trijoint.py.151: total_norm: 11.668322 [I 2022-10-10 18:33:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 1820/2304 [I 2022-10-10 18:33:42] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:06:58 [I 2022-10-10 18:33:42] ...trap/engines/engine.py.229: process: 0.87563 | load: 0.00317 [I 2022-10-10 18:33:42] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-10 18:33:42] ...optimizers/trijoint.py.151: total_norm: 14.821187 [I 2022-10-10 18:33:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1830/2304 [I 2022-10-10 18:33:51] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:49 [I 2022-10-10 18:33:51] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00315 [I 2022-10-10 18:33:51] ...trap/engines/engine.py.230: loss: 0.10085 [I 2022-10-10 18:33:51] ...optimizers/trijoint.py.151: total_norm: 11.045936 [I 2022-10-10 18:33:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 1840/2304 [I 2022-10-10 18:33:59] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:40 [I 2022-10-10 18:33:59] ...trap/engines/engine.py.229: process: 0.86699 | load: 0.00314 [I 2022-10-10 18:33:59] ...trap/engines/engine.py.230: loss: 0.10504 [I 2022-10-10 18:33:59] ...optimizers/trijoint.py.151: total_norm: 10.357952 [I 2022-10-10 18:34:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1850/2304 [I 2022-10-10 18:34:08] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:31 [I 2022-10-10 18:34:08] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00308 [I 2022-10-10 18:34:08] ...trap/engines/engine.py.230: loss: 0.09594 [I 2022-10-10 18:34:08] ...optimizers/trijoint.py.151: total_norm: 10.686807 [I 2022-10-10 18:34:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 1860/2304 [I 2022-10-10 18:34:17] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:21 [I 2022-10-10 18:34:17] ...trap/engines/engine.py.229: process: 0.85210 | load: 0.00312 [I 2022-10-10 18:34:17] ...trap/engines/engine.py.230: loss: 0.10713 [I 2022-10-10 18:34:17] ...optimizers/trijoint.py.151: total_norm: 15.720586 [I 2022-10-10 18:34:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 1870/2304 [I 2022-10-10 18:34:25] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:06:12 [I 2022-10-10 18:34:25] ...trap/engines/engine.py.229: process: 0.86032 | load: 0.00313 [I 2022-10-10 18:34:25] ...trap/engines/engine.py.230: loss: 0.09070 [I 2022-10-10 18:34:25] ...optimizers/trijoint.py.151: total_norm: 10.566784 [I 2022-10-10 18:34:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1880/2304 [I 2022-10-10 18:34:34] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:06:05 [I 2022-10-10 18:34:34] ...trap/engines/engine.py.229: process: 0.86375 | load: 0.00308 [I 2022-10-10 18:34:34] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-10 18:34:34] ...optimizers/trijoint.py.151: total_norm: 26.333263 [I 2022-10-10 18:34:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 1890/2304 [I 2022-10-10 18:34:42] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:56 [I 2022-10-10 18:34:42] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00311 [I 2022-10-10 18:34:42] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-10 18:34:42] ...optimizers/trijoint.py.151: total_norm: 18.769192 [I 2022-10-10 18:34:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1900/2304 [I 2022-10-10 18:34:51] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:49 [I 2022-10-10 18:34:51] ...trap/engines/engine.py.229: process: 0.86777 | load: 0.00311 [I 2022-10-10 18:34:51] ...trap/engines/engine.py.230: loss: 0.17916 [I 2022-10-10 18:34:51] ...optimizers/trijoint.py.151: total_norm: 16.781963 [I 2022-10-10 18:35:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1910/2304 [I 2022-10-10 18:35:00] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:39 [I 2022-10-10 18:35:00] ...trap/engines/engine.py.229: process: 0.86170 | load: 0.00307 [I 2022-10-10 18:35:00] ...trap/engines/engine.py.230: loss: 0.13613 [I 2022-10-10 18:35:00] ...optimizers/trijoint.py.151: total_norm: 23.550798 [I 2022-10-10 18:35:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1920/2304 [I 2022-10-10 18:35:08] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:30 [I 2022-10-10 18:35:08] ...trap/engines/engine.py.229: process: 0.87362 | load: 0.00327 [I 2022-10-10 18:35:08] ...trap/engines/engine.py.230: loss: 0.14639 [I 2022-10-10 18:35:08] ...optimizers/trijoint.py.151: total_norm: 14.243917 [I 2022-10-10 18:35:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 1930/2304 [I 2022-10-10 18:35:17] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:22 [I 2022-10-10 18:35:17] ...trap/engines/engine.py.229: process: 0.86321 | load: 0.00325 [I 2022-10-10 18:35:17] ...trap/engines/engine.py.230: loss: 0.23127 [I 2022-10-10 18:35:17] ...optimizers/trijoint.py.151: total_norm: 16.680428 [I 2022-10-10 18:35:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1940/2304 [I 2022-10-10 18:35:26] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:05:13 [I 2022-10-10 18:35:26] ...trap/engines/engine.py.229: process: 0.86808 | load: 0.00301 [I 2022-10-10 18:35:26] ...trap/engines/engine.py.230: loss: 0.07341 [I 2022-10-10 18:35:26] ...optimizers/trijoint.py.151: total_norm: 10.018978 [I 2022-10-10 18:35:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1950/2304 [I 2022-10-10 18:35:34] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:05:02 [I 2022-10-10 18:35:34] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00319 [I 2022-10-10 18:35:34] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-10 18:35:34] ...optimizers/trijoint.py.151: total_norm: 12.462839 [I 2022-10-10 18:35:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 1960/2304 [I 2022-10-10 18:35:43] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:55 [I 2022-10-10 18:35:43] ...trap/engines/engine.py.229: process: 0.87495 | load: 0.00344 [I 2022-10-10 18:35:43] ...trap/engines/engine.py.230: loss: 0.15933 [I 2022-10-10 18:35:43] ...optimizers/trijoint.py.151: total_norm: 14.097451 [I 2022-10-10 18:35:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1970/2304 [I 2022-10-10 18:35:51] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:48 [I 2022-10-10 18:35:51] ...trap/engines/engine.py.229: process: 0.87502 | load: 0.00315 [I 2022-10-10 18:35:51] ...trap/engines/engine.py.230: loss: 0.08096 [I 2022-10-10 18:35:51] ...optimizers/trijoint.py.151: total_norm: 8.072803 [I 2022-10-10 18:36:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 1980/2304 [I 2022-10-10 18:36:00] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:39 [I 2022-10-10 18:36:00] ...trap/engines/engine.py.229: process: 0.86656 | load: 0.00309 [I 2022-10-10 18:36:00] ...trap/engines/engine.py.230: loss: 0.12038 [I 2022-10-10 18:36:00] ...optimizers/trijoint.py.151: total_norm: 12.111679 [I 2022-10-10 18:36:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1990/2304 [I 2022-10-10 18:36:09] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:04:33 [I 2022-10-10 18:36:09] ...trap/engines/engine.py.229: process: 0.88286 | load: 0.00320 [I 2022-10-10 18:36:09] ...trap/engines/engine.py.230: loss: 0.09563 [I 2022-10-10 18:36:09] ...optimizers/trijoint.py.151: total_norm: 10.557794 [I 2022-10-10 18:36:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 2000/2304 [I 2022-10-10 18:36:17] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:21 [I 2022-10-10 18:36:17] ...trap/engines/engine.py.229: process: 0.85720 | load: 0.00648 [I 2022-10-10 18:36:17] ...trap/engines/engine.py.230: loss: 0.13376 [I 2022-10-10 18:36:17] ...optimizers/trijoint.py.151: total_norm: 14.970934 [I 2022-10-10 18:36:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 2010/2304 [I 2022-10-10 18:36:26] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:04:12 [I 2022-10-10 18:36:26] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00313 [I 2022-10-10 18:36:26] ...trap/engines/engine.py.230: loss: 0.16295 [I 2022-10-10 18:36:26] ...optimizers/trijoint.py.151: total_norm: 25.787197 [I 2022-10-10 18:36:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 2020/2304 [I 2022-10-10 18:36:35] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:04:05 [I 2022-10-10 18:36:35] ...trap/engines/engine.py.229: process: 0.86801 | load: 0.00313 [I 2022-10-10 18:36:35] ...trap/engines/engine.py.230: loss: 0.23007 [I 2022-10-10 18:36:35] ...optimizers/trijoint.py.151: total_norm: 29.370213 [I 2022-10-10 18:36:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 2030/2304 [I 2022-10-10 18:36:43] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:55 [I 2022-10-10 18:36:43] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00312 [I 2022-10-10 18:36:43] ...trap/engines/engine.py.230: loss: 0.13404 [I 2022-10-10 18:36:43] ...optimizers/trijoint.py.151: total_norm: 16.658901 [I 2022-10-10 18:36:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 2040/2304 [I 2022-10-10 18:36:52] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:46 [I 2022-10-10 18:36:52] ...trap/engines/engine.py.229: process: 0.86935 | load: 0.00314 [I 2022-10-10 18:36:52] ...trap/engines/engine.py.230: loss: 0.14037 [I 2022-10-10 18:36:52] ...optimizers/trijoint.py.151: total_norm: 11.104547 [I 2022-10-10 18:37:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 2050/2304 [I 2022-10-10 18:37:00] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:37 [I 2022-10-10 18:37:00] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00312 [I 2022-10-10 18:37:00] ...trap/engines/engine.py.230: loss: 0.14334 [I 2022-10-10 18:37:00] ...optimizers/trijoint.py.151: total_norm: 32.781479 [I 2022-10-10 18:37:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 2060/2304 [I 2022-10-10 18:37:09] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:28 [I 2022-10-10 18:37:09] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00309 [I 2022-10-10 18:37:09] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-10 18:37:09] ...optimizers/trijoint.py.151: total_norm: 13.360885 [I 2022-10-10 18:37:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 2070/2304 [I 2022-10-10 18:37:18] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:03:23 [I 2022-10-10 18:37:18] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00309 [I 2022-10-10 18:37:18] ...trap/engines/engine.py.230: loss: 0.09856 [I 2022-10-10 18:37:18] ...optimizers/trijoint.py.151: total_norm: 15.961284 [I 2022-10-10 18:37:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 2080/2304 [I 2022-10-10 18:37:26] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:03:13 [I 2022-10-10 18:37:26] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00317 [I 2022-10-10 18:37:26] ...trap/engines/engine.py.230: loss: 0.11689 [I 2022-10-10 18:37:26] ...optimizers/trijoint.py.151: total_norm: 21.763580 [I 2022-10-10 18:37:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 2090/2304 [I 2022-10-10 18:37:35] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:03:04 [I 2022-10-10 18:37:35] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00311 [I 2022-10-10 18:37:35] ...trap/engines/engine.py.230: loss: 0.12486 [I 2022-10-10 18:37:35] ...optimizers/trijoint.py.151: total_norm: 30.494667 [I 2022-10-10 18:37:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 2100/2304 [I 2022-10-10 18:37:44] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:55 [I 2022-10-10 18:37:44] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00310 [I 2022-10-10 18:37:44] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-10 18:37:44] ...optimizers/trijoint.py.151: total_norm: 15.329466 [I 2022-10-10 18:37:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 2110/2304 [I 2022-10-10 18:37:52] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:46 [I 2022-10-10 18:37:52] ...trap/engines/engine.py.229: process: 0.86201 | load: 0.00310 [I 2022-10-10 18:37:52] ...trap/engines/engine.py.230: loss: 0.18376 [I 2022-10-10 18:37:52] ...optimizers/trijoint.py.151: total_norm: 15.637255 [I 2022-10-10 18:38:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 2120/2304 [I 2022-10-10 18:38:01] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:38 [I 2022-10-10 18:38:01] ...trap/engines/engine.py.229: process: 0.87990 | load: 0.00318 [I 2022-10-10 18:38:01] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-10 18:38:01] ...optimizers/trijoint.py.151: total_norm: 15.713640 [I 2022-10-10 18:38:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 2130/2304 [I 2022-10-10 18:38:09] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:29 [I 2022-10-10 18:38:09] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00318 [I 2022-10-10 18:38:09] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-10 18:38:09] ...optimizers/trijoint.py.151: total_norm: 12.410199 [I 2022-10-10 18:38:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 2140/2304 [I 2022-10-10 18:38:18] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:20 [I 2022-10-10 18:38:18] ...trap/engines/engine.py.229: process: 0.86224 | load: 0.00313 [I 2022-10-10 18:38:18] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-10 18:38:18] ...optimizers/trijoint.py.151: total_norm: 17.303699 [I 2022-10-10 18:38:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 2150/2304 [I 2022-10-10 18:38:27] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:02:12 [I 2022-10-10 18:38:27] ...trap/engines/engine.py.229: process: 0.87235 | load: 0.00310 [I 2022-10-10 18:38:27] ...trap/engines/engine.py.230: loss: 0.09330 [I 2022-10-10 18:38:27] ...optimizers/trijoint.py.151: total_norm: 9.660577 [I 2022-10-10 18:38:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 2160/2304 [I 2022-10-10 18:38:35] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:02:03 [I 2022-10-10 18:38:35] ...trap/engines/engine.py.229: process: 0.86574 | load: 0.00317 [I 2022-10-10 18:38:35] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-10 18:38:35] ...optimizers/trijoint.py.151: total_norm: 11.433571 [I 2022-10-10 18:38:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 2170/2304 [I 2022-10-10 18:38:44] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:54 [I 2022-10-10 18:38:44] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00311 [I 2022-10-10 18:38:44] ...trap/engines/engine.py.230: loss: 0.19021 [I 2022-10-10 18:38:44] ...optimizers/trijoint.py.151: total_norm: 25.546503 [I 2022-10-10 18:38:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 2180/2304 [I 2022-10-10 18:38:53] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:46 [I 2022-10-10 18:38:53] ...trap/engines/engine.py.229: process: 0.85147 | load: 0.00312 [I 2022-10-10 18:38:53] ...trap/engines/engine.py.230: loss: 0.10529 [I 2022-10-10 18:38:53] ...optimizers/trijoint.py.151: total_norm: 12.517808 [I 2022-10-10 18:39:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 2190/2304 [I 2022-10-10 18:39:01] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:37 [I 2022-10-10 18:39:01] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00313 [I 2022-10-10 18:39:01] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-10 18:39:01] ...optimizers/trijoint.py.151: total_norm: 14.526825 [I 2022-10-10 18:39:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 2200/2304 [I 2022-10-10 18:39:10] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:28 [I 2022-10-10 18:39:10] ...trap/engines/engine.py.229: process: 0.86344 | load: 0.00312 [I 2022-10-10 18:39:10] ...trap/engines/engine.py.230: loss: 0.11978 [I 2022-10-10 18:39:10] ...optimizers/trijoint.py.151: total_norm: 15.330367 [I 2022-10-10 18:39:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 2210/2304 [I 2022-10-10 18:39:18] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:20 [I 2022-10-10 18:39:18] ...trap/engines/engine.py.229: process: 0.86214 | load: 0.00311 [I 2022-10-10 18:39:18] ...trap/engines/engine.py.230: loss: 0.21983 [I 2022-10-10 18:39:18] ...optimizers/trijoint.py.151: total_norm: 16.500679 [I 2022-10-10 18:39:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 2220/2304 [I 2022-10-10 18:39:27] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:01:11 [I 2022-10-10 18:39:27] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00319 [I 2022-10-10 18:39:27] ...trap/engines/engine.py.230: loss: 0.21445 [I 2022-10-10 18:39:27] ...optimizers/trijoint.py.151: total_norm: 19.031717 [I 2022-10-10 18:39:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 2230/2304 [I 2022-10-10 18:39:35] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:01:03 [I 2022-10-10 18:39:35] ...trap/engines/engine.py.229: process: 0.86332 | load: 0.00310 [I 2022-10-10 18:39:35] ...trap/engines/engine.py.230: loss: 0.16330 [I 2022-10-10 18:39:35] ...optimizers/trijoint.py.151: total_norm: 17.913731 [I 2022-10-10 18:39:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 2240/2304 [I 2022-10-10 18:39:44] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:55 [I 2022-10-10 18:39:44] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00322 [I 2022-10-10 18:39:44] ...trap/engines/engine.py.230: loss: 0.15209 [I 2022-10-10 18:39:44] ...optimizers/trijoint.py.151: total_norm: 20.340534 [I 2022-10-10 18:39:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 2250/2304 [I 2022-10-10 18:39:53] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:46 [I 2022-10-10 18:39:53] ...trap/engines/engine.py.229: process: 0.87774 | load: 0.00308 [I 2022-10-10 18:39:53] ...trap/engines/engine.py.230: loss: 0.19625 [I 2022-10-10 18:39:53] ...optimizers/trijoint.py.151: total_norm: 27.752243 [I 2022-10-10 18:40:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 2260/2304 [I 2022-10-10 18:40:01] ...trap/engines/engine.py.228: elapsed: 0:32:53 | left: 0:00:37 [I 2022-10-10 18:40:01] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00321 [I 2022-10-10 18:40:01] ...trap/engines/engine.py.230: loss: 0.12527 [I 2022-10-10 18:40:01] ...optimizers/trijoint.py.151: total_norm: 9.868083 [I 2022-10-10 18:40:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 2270/2304 [I 2022-10-10 18:40:10] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:29 [I 2022-10-10 18:40:10] ...trap/engines/engine.py.229: process: 0.85420 | load: 0.00319 [I 2022-10-10 18:40:10] ...trap/engines/engine.py.230: loss: 0.09834 [I 2022-10-10 18:40:10] ...optimizers/trijoint.py.151: total_norm: 16.589054 [I 2022-10-10 18:40:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 2280/2304 [I 2022-10-10 18:40:18] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:20 [I 2022-10-10 18:40:18] ...trap/engines/engine.py.229: process: 0.86442 | load: 0.00315 [I 2022-10-10 18:40:18] ...trap/engines/engine.py.230: loss: 0.11407 [I 2022-10-10 18:40:18] ...optimizers/trijoint.py.151: total_norm: 16.059996 [I 2022-10-10 18:40:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 2290/2304 [I 2022-10-10 18:40:27] ...trap/engines/engine.py.228: elapsed: 0:33:19 | left: 0:00:12 [I 2022-10-10 18:40:27] ...trap/engines/engine.py.229: process: 0.86724 | load: 0.00310 [I 2022-10-10 18:40:27] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-10 18:40:27] ...optimizers/trijoint.py.151: total_norm: 18.525728 [I 2022-10-10 18:40:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 2300/2304 [I 2022-10-10 18:40:36] ...trap/engines/engine.py.228: elapsed: 0:33:28 | left: 0:00:03 [I 2022-10-10 18:40:36] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00018 [I 2022-10-10 18:40:36] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-10 18:40:36] ...optimizers/trijoint.py.151: total_norm: 40.782352 [I 2022-10-10 18:40:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 2304/2304 [I 2022-10-10 18:40:39] ...trap/engines/engine.py.228: elapsed: 0:33:31 | left: 0:00:00 [I 2022-10-10 18:40:39] ...trap/engines/engine.py.229: process: 0.85447 | load: 0.00019 [I 2022-10-10 18:40:39] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-10 18:40:39] ...optimizers/trijoint.py.151: total_norm: 8.882700 [S 2022-10-10 18:40:41] ...trap/engines/engine.py.129: train_epoch.epoch: 81 [S 2022-10-10 18:40:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.13287422769498927 [I 2022-10-10 18:40:41] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 18:40:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 81 [I 2022-10-10 18:40:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 18:40:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 18:40:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 18:40:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 18:40:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 18:40:48] ...trap/engines/engine.py.321: eval: epoch 81 | batch 0/510 [I 2022-10-10 18:40:48] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:26 [I 2022-10-10 18:40:48] ...trap/engines/engine.py.326: process: 5.46370 | load: 4.93141 [I 2022-10-10 18:40:54] ...trap/engines/engine.py.321: eval: epoch 81 | batch 10/510 [I 2022-10-10 18:40:54] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:26 [I 2022-10-10 18:40:54] ...trap/engines/engine.py.326: process: 0.52356 | load: 0.00019 [I 2022-10-10 18:41:00] ...trap/engines/engine.py.321: eval: epoch 81 | batch 20/510 [I 2022-10-10 18:41:00] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:05:36 [I 2022-10-10 18:41:00] ...trap/engines/engine.py.326: process: 0.52351 | load: 0.00018 [I 2022-10-10 18:41:07] ...trap/engines/engine.py.321: eval: epoch 81 | batch 30/510 [I 2022-10-10 18:41:07] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:21 [I 2022-10-10 18:41:07] ...trap/engines/engine.py.326: process: 0.73058 | load: 0.00018 [I 2022-10-10 18:41:13] ...trap/engines/engine.py.321: eval: epoch 81 | batch 40/510 [I 2022-10-10 18:41:13] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:58 [I 2022-10-10 18:41:13] ...trap/engines/engine.py.326: process: 0.49704 | load: 0.00019 [I 2022-10-10 18:41:18] ...trap/engines/engine.py.321: eval: epoch 81 | batch 50/510 [I 2022-10-10 18:41:18] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:54 [I 2022-10-10 18:41:18] ...trap/engines/engine.py.326: process: 0.48778 | load: 0.00018 [I 2022-10-10 18:41:23] ...trap/engines/engine.py.321: eval: epoch 81 | batch 60/510 [I 2022-10-10 18:41:23] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:48 [I 2022-10-10 18:41:23] ...trap/engines/engine.py.326: process: 0.46897 | load: 0.00019 [I 2022-10-10 18:41:28] ...trap/engines/engine.py.321: eval: epoch 81 | batch 70/510 [I 2022-10-10 18:41:28] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:22 [I 2022-10-10 18:41:28] ...trap/engines/engine.py.326: process: 0.46398 | load: 0.00023 [I 2022-10-10 18:41:32] ...trap/engines/engine.py.321: eval: epoch 81 | batch 80/510 [I 2022-10-10 18:41:32] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:14 [I 2022-10-10 18:41:32] ...trap/engines/engine.py.326: process: 0.53478 | load: 0.00020 [I 2022-10-10 18:41:42] ...trap/engines/engine.py.321: eval: epoch 81 | batch 90/510 [I 2022-10-10 18:41:42] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:04:09 [I 2022-10-10 18:41:42] ...trap/engines/engine.py.326: process: 0.47784 | load: 0.00060 [I 2022-10-10 18:41:46] ...trap/engines/engine.py.321: eval: epoch 81 | batch 100/510 [I 2022-10-10 18:41:46] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:56 [I 2022-10-10 18:41:46] ...trap/engines/engine.py.326: process: 0.31187 | load: 0.00018 [I 2022-10-10 18:41:50] ...trap/engines/engine.py.321: eval: epoch 81 | batch 110/510 [I 2022-10-10 18:41:50] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:12 [I 2022-10-10 18:41:50] ...trap/engines/engine.py.326: process: 0.32414 | load: 0.00019 [I 2022-10-10 18:41:53] ...trap/engines/engine.py.321: eval: epoch 81 | batch 120/510 [I 2022-10-10 18:41:53] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:05 [I 2022-10-10 18:41:53] ...trap/engines/engine.py.326: process: 0.32006 | load: 0.00019 [I 2022-10-10 18:41:56] ...trap/engines/engine.py.321: eval: epoch 81 | batch 130/510 [I 2022-10-10 18:41:56] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:01 [I 2022-10-10 18:41:56] ...trap/engines/engine.py.326: process: 0.31558 | load: 0.00022 [I 2022-10-10 18:41:59] ...trap/engines/engine.py.321: eval: epoch 81 | batch 140/510 [I 2022-10-10 18:41:59] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:58 [I 2022-10-10 18:41:59] ...trap/engines/engine.py.326: process: 0.31464 | load: 0.00019 [I 2022-10-10 18:42:02] ...trap/engines/engine.py.321: eval: epoch 81 | batch 150/510 [I 2022-10-10 18:42:02] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:55 [I 2022-10-10 18:42:02] ...trap/engines/engine.py.326: process: 0.32143 | load: 0.00019 [I 2022-10-10 18:42:06] ...trap/engines/engine.py.321: eval: epoch 81 | batch 160/510 [I 2022-10-10 18:42:06] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:52 [I 2022-10-10 18:42:06] ...trap/engines/engine.py.326: process: 0.32627 | load: 0.00019 [I 2022-10-10 18:42:09] ...trap/engines/engine.py.321: eval: epoch 81 | batch 170/510 [I 2022-10-10 18:42:09] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:47 [I 2022-10-10 18:42:09] ...trap/engines/engine.py.326: process: 0.30948 | load: 0.00019 [I 2022-10-10 18:42:12] ...trap/engines/engine.py.321: eval: epoch 81 | batch 180/510 [I 2022-10-10 18:42:12] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:45 [I 2022-10-10 18:42:12] ...trap/engines/engine.py.326: process: 0.32210 | load: 0.00020 [I 2022-10-10 18:42:15] ...trap/engines/engine.py.321: eval: epoch 81 | batch 190/510 [I 2022-10-10 18:42:15] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:41 [I 2022-10-10 18:42:15] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00019 [I 2022-10-10 18:42:18] ...trap/engines/engine.py.321: eval: epoch 81 | batch 200/510 [I 2022-10-10 18:42:18] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:38 [I 2022-10-10 18:42:18] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00019 [I 2022-10-10 18:42:22] ...trap/engines/engine.py.321: eval: epoch 81 | batch 210/510 [I 2022-10-10 18:42:22] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:35 [I 2022-10-10 18:42:22] ...trap/engines/engine.py.326: process: 0.31273 | load: 0.00017 [I 2022-10-10 18:42:25] ...trap/engines/engine.py.321: eval: epoch 81 | batch 220/510 [I 2022-10-10 18:42:25] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:32 [I 2022-10-10 18:42:25] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00019 [I 2022-10-10 18:42:28] ...trap/engines/engine.py.321: eval: epoch 81 | batch 230/510 [I 2022-10-10 18:42:28] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:30 [I 2022-10-10 18:42:28] ...trap/engines/engine.py.326: process: 0.32148 | load: 0.00023 [I 2022-10-10 18:42:31] ...trap/engines/engine.py.321: eval: epoch 81 | batch 240/510 [I 2022-10-10 18:42:31] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:26 [I 2022-10-10 18:42:31] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00020 [I 2022-10-10 18:42:34] ...trap/engines/engine.py.321: eval: epoch 81 | batch 250/510 [I 2022-10-10 18:42:34] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:23 [I 2022-10-10 18:42:34] ...trap/engines/engine.py.326: process: 0.32625 | load: 0.00020 [I 2022-10-10 18:42:38] ...trap/engines/engine.py.321: eval: epoch 81 | batch 260/510 [I 2022-10-10 18:42:38] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:19 [I 2022-10-10 18:42:38] ...trap/engines/engine.py.326: process: 0.31878 | load: 0.00021 [I 2022-10-10 18:42:41] ...trap/engines/engine.py.321: eval: epoch 81 | batch 270/510 [I 2022-10-10 18:42:41] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:15 [I 2022-10-10 18:42:41] ...trap/engines/engine.py.326: process: 0.31205 | load: 0.00019 [I 2022-10-10 18:42:44] ...trap/engines/engine.py.321: eval: epoch 81 | batch 280/510 [I 2022-10-10 18:42:44] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:13 [I 2022-10-10 18:42:44] ...trap/engines/engine.py.326: process: 0.32099 | load: 0.00020 [I 2022-10-10 18:42:47] ...trap/engines/engine.py.321: eval: epoch 81 | batch 290/510 [I 2022-10-10 18:42:47] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:10 [I 2022-10-10 18:42:47] ...trap/engines/engine.py.326: process: 0.31950 | load: 0.00020 [I 2022-10-10 18:42:50] ...trap/engines/engine.py.321: eval: epoch 81 | batch 300/510 [I 2022-10-10 18:42:50] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:07 [I 2022-10-10 18:42:50] ...trap/engines/engine.py.326: process: 0.31464 | load: 0.00019 [I 2022-10-10 18:42:54] ...trap/engines/engine.py.321: eval: epoch 81 | batch 310/510 [I 2022-10-10 18:42:54] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:03 [I 2022-10-10 18:42:54] ...trap/engines/engine.py.326: process: 0.31893 | load: 0.00019 [I 2022-10-10 18:42:57] ...trap/engines/engine.py.321: eval: epoch 81 | batch 320/510 [I 2022-10-10 18:42:57] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:00 [I 2022-10-10 18:42:57] ...trap/engines/engine.py.326: process: 0.32244 | load: 0.00019 [I 2022-10-10 18:43:00] ...trap/engines/engine.py.321: eval: epoch 81 | batch 330/510 [I 2022-10-10 18:43:00] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:57 [I 2022-10-10 18:43:00] ...trap/engines/engine.py.326: process: 0.31923 | load: 0.00021 [I 2022-10-10 18:43:03] ...trap/engines/engine.py.321: eval: epoch 81 | batch 340/510 [I 2022-10-10 18:43:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:54 [I 2022-10-10 18:43:03] ...trap/engines/engine.py.326: process: 0.31158 | load: 0.00022 [I 2022-10-10 18:43:07] ...trap/engines/engine.py.321: eval: epoch 81 | batch 350/510 [I 2022-10-10 18:43:07] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:51 [I 2022-10-10 18:43:07] ...trap/engines/engine.py.326: process: 0.32152 | load: 0.00021 [I 2022-10-10 18:43:10] ...trap/engines/engine.py.321: eval: epoch 81 | batch 360/510 [I 2022-10-10 18:43:10] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-10 18:43:10] ...trap/engines/engine.py.326: process: 0.31900 | load: 0.00021 [I 2022-10-10 18:43:13] ...trap/engines/engine.py.321: eval: epoch 81 | batch 370/510 [I 2022-10-10 18:43:13] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:45 [I 2022-10-10 18:43:13] ...trap/engines/engine.py.326: process: 0.32363 | load: 0.00021 [I 2022-10-10 18:43:16] ...trap/engines/engine.py.321: eval: epoch 81 | batch 380/510 [I 2022-10-10 18:43:16] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:42 [I 2022-10-10 18:43:16] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00020 [I 2022-10-10 18:43:19] ...trap/engines/engine.py.321: eval: epoch 81 | batch 390/510 [I 2022-10-10 18:43:19] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:38 [I 2022-10-10 18:43:19] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00021 [I 2022-10-10 18:43:23] ...trap/engines/engine.py.321: eval: epoch 81 | batch 400/510 [I 2022-10-10 18:43:23] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-10 18:43:23] ...trap/engines/engine.py.326: process: 0.31533 | load: 0.00022 [I 2022-10-10 18:43:26] ...trap/engines/engine.py.321: eval: epoch 81 | batch 410/510 [I 2022-10-10 18:43:26] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-10 18:43:26] ...trap/engines/engine.py.326: process: 0.32526 | load: 0.00021 [I 2022-10-10 18:43:29] ...trap/engines/engine.py.321: eval: epoch 81 | batch 420/510 [I 2022-10-10 18:43:29] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:28 [I 2022-10-10 18:43:29] ...trap/engines/engine.py.326: process: 0.32342 | load: 0.00020 [I 2022-10-10 18:43:32] ...trap/engines/engine.py.321: eval: epoch 81 | batch 430/510 [I 2022-10-10 18:43:32] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:26 [I 2022-10-10 18:43:32] ...trap/engines/engine.py.326: process: 0.33410 | load: 0.00022 [I 2022-10-10 18:43:36] ...trap/engines/engine.py.321: eval: epoch 81 | batch 440/510 [I 2022-10-10 18:43:36] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-10 18:43:36] ...trap/engines/engine.py.326: process: 0.32857 | load: 0.00021 [I 2022-10-10 18:43:39] ...trap/engines/engine.py.321: eval: epoch 81 | batch 450/510 [I 2022-10-10 18:43:39] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-10 18:43:39] ...trap/engines/engine.py.326: process: 0.32007 | load: 0.00021 [I 2022-10-10 18:43:42] ...trap/engines/engine.py.321: eval: epoch 81 | batch 460/510 [I 2022-10-10 18:43:42] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:16 [I 2022-10-10 18:43:42] ...trap/engines/engine.py.326: process: 0.32147 | load: 0.00021 [I 2022-10-10 18:43:45] ...trap/engines/engine.py.321: eval: epoch 81 | batch 470/510 [I 2022-10-10 18:43:45] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:13 [I 2022-10-10 18:43:45] ...trap/engines/engine.py.326: process: 0.33502 | load: 0.00021 [I 2022-10-10 18:43:49] ...trap/engines/engine.py.321: eval: epoch 81 | batch 480/510 [I 2022-10-10 18:43:49] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-10 18:43:49] ...trap/engines/engine.py.326: process: 0.32854 | load: 0.00020 [I 2022-10-10 18:43:52] ...trap/engines/engine.py.321: eval: epoch 81 | batch 490/510 [I 2022-10-10 18:43:52] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-10 18:43:52] ...trap/engines/engine.py.326: process: 0.32635 | load: 0.00021 [I 2022-10-10 18:43:55] ...trap/engines/engine.py.321: eval: epoch 81 | batch 500/510 [I 2022-10-10 18:43:55] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:03 [I 2022-10-10 18:43:55] ...trap/engines/engine.py.326: process: 0.32469 | load: 0.00020 [I 2022-10-10 18:43:58] ...trap/engines/engine.py.321: eval: epoch 81 | batch 510/510 [I 2022-10-10 18:43:58] ...trap/engines/engine.py.325: elapsed: 0:03:16 | left: 0:00:00 [I 2022-10-10 18:43:58] ...trap/engines/engine.py.326: process: 0.31349 | load: 0.00017 [S 2022-10-10 18:43:59] ...trap/engines/engine.py.132: eval_epoch.epoch: 81 [I 2022-10-10 18:43:59] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736600 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906400 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935200 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727900 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908600 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935700 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007351 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006785 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005980 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008396 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007214 [S 2022-10-10 18:44:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006664 [I 2022-10-10 18:44:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 18:44:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 18:44:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 18:44:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 18:44:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 18:44:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 18:44:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 18:44:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 18:44:57] ...trap/engines/engine.py.163: Training model on trainset for epoch 82 [I 2022-10-10 18:44:57] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 18:45:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 0/2304 [I 2022-10-10 18:45:00] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:06:11 [I 2022-10-10 18:45:00] ...trap/engines/engine.py.229: process: 3.28637 | load: 2.32059 [I 2022-10-10 18:45:00] ...trap/engines/engine.py.230: loss: 0.14200 [I 2022-10-10 18:45:00] ...optimizers/trijoint.py.151: total_norm: 11.853851 [I 2022-10-10 18:45:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 10/2304 [I 2022-10-10 18:45:09] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:50 [I 2022-10-10 18:45:09] ...trap/engines/engine.py.229: process: 0.86678 | load: 0.00567 [I 2022-10-10 18:45:09] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-10 18:45:09] ...optimizers/trijoint.py.151: total_norm: 12.406396 [I 2022-10-10 18:45:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 20/2304 [I 2022-10-10 18:45:18] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:50 [I 2022-10-10 18:45:18] ...trap/engines/engine.py.229: process: 0.86864 | load: 0.00553 [I 2022-10-10 18:45:18] ...trap/engines/engine.py.230: loss: 0.10477 [I 2022-10-10 18:45:18] ...optimizers/trijoint.py.151: total_norm: 9.208390 [I 2022-10-10 18:45:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 30/2304 [I 2022-10-10 18:45:26] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:56 [I 2022-10-10 18:45:26] ...trap/engines/engine.py.229: process: 0.89182 | load: 0.00531 [I 2022-10-10 18:45:26] ...trap/engines/engine.py.230: loss: 0.15362 [I 2022-10-10 18:45:26] ...optimizers/trijoint.py.151: total_norm: 10.696132 [I 2022-10-10 18:45:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 40/2304 [I 2022-10-10 18:45:35] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:42 [I 2022-10-10 18:45:35] ...trap/engines/engine.py.229: process: 0.87803 | load: 0.00393 [I 2022-10-10 18:45:35] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-10 18:45:35] ...optimizers/trijoint.py.151: total_norm: 23.318531 [I 2022-10-10 18:45:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 50/2304 [I 2022-10-10 18:45:43] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:30 [I 2022-10-10 18:45:43] ...trap/engines/engine.py.229: process: 0.87957 | load: 0.00488 [I 2022-10-10 18:45:43] ...trap/engines/engine.py.230: loss: 0.23129 [I 2022-10-10 18:45:43] ...optimizers/trijoint.py.151: total_norm: 21.039715 [I 2022-10-10 18:45:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 60/2304 [I 2022-10-10 18:45:52] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:14 [I 2022-10-10 18:45:52] ...trap/engines/engine.py.229: process: 0.87618 | load: 0.00395 [I 2022-10-10 18:45:52] ...trap/engines/engine.py.230: loss: 0.16920 [I 2022-10-10 18:45:52] ...optimizers/trijoint.py.151: total_norm: 16.321507 [I 2022-10-10 18:46:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 70/2304 [I 2022-10-10 18:46:01] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:20 [I 2022-10-10 18:46:01] ...trap/engines/engine.py.229: process: 0.88164 | load: 0.00347 [I 2022-10-10 18:46:01] ...trap/engines/engine.py.230: loss: 0.15856 [I 2022-10-10 18:46:01] ...optimizers/trijoint.py.151: total_norm: 24.940657 [I 2022-10-10 18:46:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 80/2304 [I 2022-10-10 18:46:10] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:05 [I 2022-10-10 18:46:10] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00360 [I 2022-10-10 18:46:10] ...trap/engines/engine.py.230: loss: 0.17642 [I 2022-10-10 18:46:10] ...optimizers/trijoint.py.151: total_norm: 57.961395 [I 2022-10-10 18:46:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 90/2304 [I 2022-10-10 18:46:18] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:59 [I 2022-10-10 18:46:18] ...trap/engines/engine.py.229: process: 0.87566 | load: 0.00358 [I 2022-10-10 18:46:18] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-10 18:46:18] ...optimizers/trijoint.py.151: total_norm: 14.450778 [I 2022-10-10 18:46:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 100/2304 [I 2022-10-10 18:46:27] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:46 [I 2022-10-10 18:46:27] ...trap/engines/engine.py.229: process: 0.86100 | load: 0.00355 [I 2022-10-10 18:46:27] ...trap/engines/engine.py.230: loss: 0.21368 [I 2022-10-10 18:46:27] ...optimizers/trijoint.py.151: total_norm: 22.843685 [I 2022-10-10 18:46:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 110/2304 [I 2022-10-10 18:46:36] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:41 [I 2022-10-10 18:46:36] ...trap/engines/engine.py.229: process: 0.87000 | load: 0.00351 [I 2022-10-10 18:46:36] ...trap/engines/engine.py.230: loss: 0.13643 [I 2022-10-10 18:46:36] ...optimizers/trijoint.py.151: total_norm: 39.191750 [I 2022-10-10 18:46:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 120/2304 [I 2022-10-10 18:46:44] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:29 [I 2022-10-10 18:46:44] ...trap/engines/engine.py.229: process: 0.86888 | load: 0.00357 [I 2022-10-10 18:46:44] ...trap/engines/engine.py.230: loss: 0.12847 [I 2022-10-10 18:46:44] ...optimizers/trijoint.py.151: total_norm: 23.398113 [I 2022-10-10 18:46:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 130/2304 [I 2022-10-10 18:46:53] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:02 [I 2022-10-10 18:46:53] ...trap/engines/engine.py.229: process: 0.85607 | load: 0.00349 [I 2022-10-10 18:46:53] ...trap/engines/engine.py.230: loss: 0.12683 [I 2022-10-10 18:46:53] ...optimizers/trijoint.py.151: total_norm: 8.955113 [I 2022-10-10 18:47:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 140/2304 [I 2022-10-10 18:47:01] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:30:55 [I 2022-10-10 18:47:01] ...trap/engines/engine.py.229: process: 0.85136 | load: 0.00344 [I 2022-10-10 18:47:01] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-10 18:47:01] ...optimizers/trijoint.py.151: total_norm: 13.866392 [I 2022-10-10 18:47:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 150/2304 [I 2022-10-10 18:47:10] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:38 [I 2022-10-10 18:47:10] ...trap/engines/engine.py.229: process: 0.86316 | load: 0.00326 [I 2022-10-10 18:47:10] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-10 18:47:10] ...optimizers/trijoint.py.151: total_norm: 22.777290 [I 2022-10-10 18:47:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 160/2304 [I 2022-10-10 18:47:19] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:34 [I 2022-10-10 18:47:19] ...trap/engines/engine.py.229: process: 0.86058 | load: 0.00312 [I 2022-10-10 18:47:19] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-10 18:47:19] ...optimizers/trijoint.py.151: total_norm: 10.542445 [I 2022-10-10 18:47:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 170/2304 [I 2022-10-10 18:47:27] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:18 [I 2022-10-10 18:47:27] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00294 [I 2022-10-10 18:47:27] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-10 18:47:27] ...optimizers/trijoint.py.151: total_norm: 9.789345 [I 2022-10-10 18:47:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 180/2304 [I 2022-10-10 18:47:36] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:11 [I 2022-10-10 18:47:36] ...trap/engines/engine.py.229: process: 0.86507 | load: 0.00317 [I 2022-10-10 18:47:36] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-10 18:47:36] ...optimizers/trijoint.py.151: total_norm: 14.040748 [I 2022-10-10 18:47:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 190/2304 [I 2022-10-10 18:47:44] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:09 [I 2022-10-10 18:47:44] ...trap/engines/engine.py.229: process: 0.85850 | load: 0.00305 [I 2022-10-10 18:47:44] ...trap/engines/engine.py.230: loss: 0.12139 [I 2022-10-10 18:47:44] ...optimizers/trijoint.py.151: total_norm: 15.506666 [I 2022-10-10 18:47:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 200/2304 [I 2022-10-10 18:47:53] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:51 [I 2022-10-10 18:47:53] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00297 [I 2022-10-10 18:47:53] ...trap/engines/engine.py.230: loss: 0.13828 [I 2022-10-10 18:47:53] ...optimizers/trijoint.py.151: total_norm: 13.213223 [I 2022-10-10 18:48:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 210/2304 [I 2022-10-10 18:48:01] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:46 [I 2022-10-10 18:48:01] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00309 [I 2022-10-10 18:48:01] ...trap/engines/engine.py.230: loss: 0.12380 [I 2022-10-10 18:48:01] ...optimizers/trijoint.py.151: total_norm: 24.340672 [I 2022-10-10 18:48:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 220/2304 [I 2022-10-10 18:48:10] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:45 [I 2022-10-10 18:48:10] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00326 [I 2022-10-10 18:48:10] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-10 18:48:10] ...optimizers/trijoint.py.151: total_norm: 15.928332 [I 2022-10-10 18:48:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 230/2304 [I 2022-10-10 18:48:18] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:51 [I 2022-10-10 18:48:18] ...trap/engines/engine.py.229: process: 0.88052 | load: 0.00297 [I 2022-10-10 18:48:18] ...trap/engines/engine.py.230: loss: 0.11942 [I 2022-10-10 18:48:18] ...optimizers/trijoint.py.151: total_norm: 10.516574 [I 2022-10-10 18:48:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 240/2304 [I 2022-10-10 18:48:27] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:31 [I 2022-10-10 18:48:27] ...trap/engines/engine.py.229: process: 0.84446 | load: 0.00296 [I 2022-10-10 18:48:27] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-10 18:48:27] ...optimizers/trijoint.py.151: total_norm: 19.422426 [I 2022-10-10 18:48:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 250/2304 [I 2022-10-10 18:48:36] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:24 [I 2022-10-10 18:48:36] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00297 [I 2022-10-10 18:48:36] ...trap/engines/engine.py.230: loss: 0.28132 [I 2022-10-10 18:48:36] ...optimizers/trijoint.py.151: total_norm: 26.609488 [I 2022-10-10 18:48:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 260/2304 [I 2022-10-10 18:48:44] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:17 [I 2022-10-10 18:48:44] ...trap/engines/engine.py.229: process: 0.88075 | load: 0.00301 [I 2022-10-10 18:48:44] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-10 18:48:44] ...optimizers/trijoint.py.151: total_norm: 33.696285 [I 2022-10-10 18:48:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 270/2304 [I 2022-10-10 18:48:53] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:28:55 [I 2022-10-10 18:48:53] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00296 [I 2022-10-10 18:48:53] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-10 18:48:53] ...optimizers/trijoint.py.151: total_norm: 24.387539 [I 2022-10-10 18:49:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 280/2304 [I 2022-10-10 18:49:01] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:29:02 [I 2022-10-10 18:49:01] ...trap/engines/engine.py.229: process: 0.86367 | load: 0.00301 [I 2022-10-10 18:49:01] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-10 18:49:01] ...optimizers/trijoint.py.151: total_norm: 11.392152 [I 2022-10-10 18:49:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 290/2304 [I 2022-10-10 18:49:10] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:37 [I 2022-10-10 18:49:10] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00309 [I 2022-10-10 18:49:10] ...trap/engines/engine.py.230: loss: 0.14179 [I 2022-10-10 18:49:10] ...optimizers/trijoint.py.151: total_norm: 68.805801 [I 2022-10-10 18:49:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 300/2304 [I 2022-10-10 18:49:19] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:54 [I 2022-10-10 18:49:19] ...trap/engines/engine.py.229: process: 0.86959 | load: 0.00306 [I 2022-10-10 18:49:19] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-10 18:49:19] ...optimizers/trijoint.py.151: total_norm: 14.774156 [I 2022-10-10 18:49:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 310/2304 [I 2022-10-10 18:49:27] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:46 [I 2022-10-10 18:49:27] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00312 [I 2022-10-10 18:49:27] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-10 18:49:27] ...optimizers/trijoint.py.151: total_norm: 16.695477 [I 2022-10-10 18:49:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 320/2304 [I 2022-10-10 18:49:36] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:35 [I 2022-10-10 18:49:36] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00336 [I 2022-10-10 18:49:36] ...trap/engines/engine.py.230: loss: 0.09053 [I 2022-10-10 18:49:36] ...optimizers/trijoint.py.151: total_norm: 12.564405 [I 2022-10-10 18:49:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 330/2304 [I 2022-10-10 18:49:45] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:28:22 [I 2022-10-10 18:49:45] ...trap/engines/engine.py.229: process: 0.86252 | load: 0.00314 [I 2022-10-10 18:49:45] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-10 18:49:45] ...optimizers/trijoint.py.151: total_norm: 13.036872 [I 2022-10-10 18:49:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 340/2304 [I 2022-10-10 18:49:53] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:28:22 [I 2022-10-10 18:49:53] ...trap/engines/engine.py.229: process: 0.86532 | load: 0.00308 [I 2022-10-10 18:49:53] ...trap/engines/engine.py.230: loss: 0.14226 [I 2022-10-10 18:49:53] ...optimizers/trijoint.py.151: total_norm: 12.271883 [I 2022-10-10 18:50:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 350/2304 [I 2022-10-10 18:50:02] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:59 [I 2022-10-10 18:50:02] ...trap/engines/engine.py.229: process: 0.85307 | load: 0.00313 [I 2022-10-10 18:50:02] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-10 18:50:02] ...optimizers/trijoint.py.151: total_norm: 15.173265 [I 2022-10-10 18:50:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 360/2304 [I 2022-10-10 18:50:11] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:28:03 [I 2022-10-10 18:50:11] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00332 [I 2022-10-10 18:50:11] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-10 18:50:11] ...optimizers/trijoint.py.151: total_norm: 12.772491 [I 2022-10-10 18:50:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 370/2304 [I 2022-10-10 18:50:19] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:50 [I 2022-10-10 18:50:19] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00331 [I 2022-10-10 18:50:19] ...trap/engines/engine.py.230: loss: 0.13963 [I 2022-10-10 18:50:19] ...optimizers/trijoint.py.151: total_norm: 10.559493 [I 2022-10-10 18:50:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 380/2304 [I 2022-10-10 18:50:28] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:43 [I 2022-10-10 18:50:28] ...trap/engines/engine.py.229: process: 0.85587 | load: 0.00315 [I 2022-10-10 18:50:28] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-10 18:50:28] ...optimizers/trijoint.py.151: total_norm: 20.406845 [I 2022-10-10 18:50:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 390/2304 [I 2022-10-10 18:50:37] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:37 [I 2022-10-10 18:50:37] ...trap/engines/engine.py.229: process: 0.86809 | load: 0.00306 [I 2022-10-10 18:50:37] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-10 18:50:37] ...optimizers/trijoint.py.151: total_norm: 13.421401 [I 2022-10-10 18:50:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 400/2304 [I 2022-10-10 18:50:45] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:25 [I 2022-10-10 18:50:45] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00312 [I 2022-10-10 18:50:45] ...trap/engines/engine.py.230: loss: 0.12275 [I 2022-10-10 18:50:45] ...optimizers/trijoint.py.151: total_norm: 16.401602 [I 2022-10-10 18:50:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 410/2304 [I 2022-10-10 18:50:54] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:19 [I 2022-10-10 18:50:54] ...trap/engines/engine.py.229: process: 0.87425 | load: 0.00319 [I 2022-10-10 18:50:54] ...trap/engines/engine.py.230: loss: 0.08995 [I 2022-10-10 18:50:54] ...optimizers/trijoint.py.151: total_norm: 16.000519 [I 2022-10-10 18:51:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 420/2304 [I 2022-10-10 18:51:03] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:27:05 [I 2022-10-10 18:51:03] ...trap/engines/engine.py.229: process: 0.86992 | load: 0.00319 [I 2022-10-10 18:51:03] ...trap/engines/engine.py.230: loss: 0.22249 [I 2022-10-10 18:51:03] ...optimizers/trijoint.py.151: total_norm: 13.409086 [I 2022-10-10 18:51:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 430/2304 [I 2022-10-10 18:51:11] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:27:08 [I 2022-10-10 18:51:11] ...trap/engines/engine.py.229: process: 0.86916 | load: 0.00309 [I 2022-10-10 18:51:11] ...trap/engines/engine.py.230: loss: 0.11985 [I 2022-10-10 18:51:11] ...optimizers/trijoint.py.151: total_norm: 10.976532 [I 2022-10-10 18:51:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 440/2304 [I 2022-10-10 18:51:20] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:53 [I 2022-10-10 18:51:20] ...trap/engines/engine.py.229: process: 0.86905 | load: 0.00445 [I 2022-10-10 18:51:20] ...trap/engines/engine.py.230: loss: 0.10301 [I 2022-10-10 18:51:20] ...optimizers/trijoint.py.151: total_norm: 16.391413 [I 2022-10-10 18:51:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 450/2304 [I 2022-10-10 18:51:29] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:38 [I 2022-10-10 18:51:29] ...trap/engines/engine.py.229: process: 0.86920 | load: 0.00308 [I 2022-10-10 18:51:29] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-10 18:51:29] ...optimizers/trijoint.py.151: total_norm: 17.226135 [I 2022-10-10 18:51:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 460/2304 [I 2022-10-10 18:51:37] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:31 [I 2022-10-10 18:51:37] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00312 [I 2022-10-10 18:51:37] ...trap/engines/engine.py.230: loss: 0.22851 [I 2022-10-10 18:51:37] ...optimizers/trijoint.py.151: total_norm: 21.935600 [I 2022-10-10 18:51:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 470/2304 [I 2022-10-10 18:51:46] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:26 [I 2022-10-10 18:51:46] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00317 [I 2022-10-10 18:51:46] ...trap/engines/engine.py.230: loss: 0.18859 [I 2022-10-10 18:51:46] ...optimizers/trijoint.py.151: total_norm: 22.511396 [I 2022-10-10 18:51:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 480/2304 [I 2022-10-10 18:51:55] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:19 [I 2022-10-10 18:51:55] ...trap/engines/engine.py.229: process: 0.86839 | load: 0.00308 [I 2022-10-10 18:51:55] ...trap/engines/engine.py.230: loss: 0.24732 [I 2022-10-10 18:51:55] ...optimizers/trijoint.py.151: total_norm: 18.308554 [I 2022-10-10 18:52:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 490/2304 [I 2022-10-10 18:52:03] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:51 [I 2022-10-10 18:52:03] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00320 [I 2022-10-10 18:52:03] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-10 18:52:03] ...optimizers/trijoint.py.151: total_norm: 12.755316 [I 2022-10-10 18:52:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 500/2304 [I 2022-10-10 18:52:12] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:45 [I 2022-10-10 18:52:12] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00310 [I 2022-10-10 18:52:12] ...trap/engines/engine.py.230: loss: 0.17701 [I 2022-10-10 18:52:12] ...optimizers/trijoint.py.151: total_norm: 19.231081 [I 2022-10-10 18:52:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 510/2304 [I 2022-10-10 18:52:21] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:45 [I 2022-10-10 18:52:21] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00308 [I 2022-10-10 18:52:21] ...trap/engines/engine.py.230: loss: 0.11252 [I 2022-10-10 18:52:21] ...optimizers/trijoint.py.151: total_norm: 14.219220 [I 2022-10-10 18:52:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 520/2304 [I 2022-10-10 18:52:29] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:46 [I 2022-10-10 18:52:29] ...trap/engines/engine.py.229: process: 0.85743 | load: 0.00339 [I 2022-10-10 18:52:29] ...trap/engines/engine.py.230: loss: 0.19338 [I 2022-10-10 18:52:29] ...optimizers/trijoint.py.151: total_norm: 29.752481 [I 2022-10-10 18:52:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 530/2304 [I 2022-10-10 18:52:38] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:25:35 [I 2022-10-10 18:52:38] ...trap/engines/engine.py.229: process: 0.86314 | load: 0.00316 [I 2022-10-10 18:52:38] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-10 18:52:38] ...optimizers/trijoint.py.151: total_norm: 15.561728 [I 2022-10-10 18:52:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 540/2304 [I 2022-10-10 18:52:47] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:34 [I 2022-10-10 18:52:47] ...trap/engines/engine.py.229: process: 0.85882 | load: 0.00324 [I 2022-10-10 18:52:47] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-10 18:52:47] ...optimizers/trijoint.py.151: total_norm: 23.869118 [I 2022-10-10 18:52:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 550/2304 [I 2022-10-10 18:52:55] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:22 [I 2022-10-10 18:52:55] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00421 [I 2022-10-10 18:52:55] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-10 18:52:55] ...optimizers/trijoint.py.151: total_norm: 11.221769 [I 2022-10-10 18:53:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 560/2304 [I 2022-10-10 18:53:04] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:25:09 [I 2022-10-10 18:53:04] ...trap/engines/engine.py.229: process: 0.86392 | load: 0.00308 [I 2022-10-10 18:53:04] ...trap/engines/engine.py.230: loss: 0.14613 [I 2022-10-10 18:53:04] ...optimizers/trijoint.py.151: total_norm: 21.575909 [I 2022-10-10 18:53:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 570/2304 [I 2022-10-10 18:53:13] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:41 [I 2022-10-10 18:53:13] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00295 [I 2022-10-10 18:53:13] ...trap/engines/engine.py.230: loss: 0.12967 [I 2022-10-10 18:53:13] ...optimizers/trijoint.py.151: total_norm: 8.834225 [I 2022-10-10 18:53:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 580/2304 [I 2022-10-10 18:53:21] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:35 [I 2022-10-10 18:53:21] ...trap/engines/engine.py.229: process: 0.85710 | load: 0.00312 [I 2022-10-10 18:53:21] ...trap/engines/engine.py.230: loss: 0.11894 [I 2022-10-10 18:53:21] ...optimizers/trijoint.py.151: total_norm: 16.148962 [I 2022-10-10 18:53:30] ...trap/engines/engine.py.224: train: epoch 82 | batch 590/2304 [I 2022-10-10 18:53:30] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:32 [I 2022-10-10 18:53:30] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00306 [I 2022-10-10 18:53:30] ...trap/engines/engine.py.230: loss: 0.13123 [I 2022-10-10 18:53:30] ...optimizers/trijoint.py.151: total_norm: 13.719178 [I 2022-10-10 18:53:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 600/2304 [I 2022-10-10 18:53:39] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:22 [I 2022-10-10 18:53:39] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00300 [I 2022-10-10 18:53:39] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-10 18:53:39] ...optimizers/trijoint.py.151: total_norm: 13.784712 [I 2022-10-10 18:53:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 610/2304 [I 2022-10-10 18:53:47] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:02 [I 2022-10-10 18:53:47] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00304 [I 2022-10-10 18:53:47] ...trap/engines/engine.py.230: loss: 0.18587 [I 2022-10-10 18:53:47] ...optimizers/trijoint.py.151: total_norm: 12.459014 [I 2022-10-10 18:53:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 620/2304 [I 2022-10-10 18:53:56] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:02 [I 2022-10-10 18:53:56] ...trap/engines/engine.py.229: process: 0.87797 | load: 0.00293 [I 2022-10-10 18:53:56] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-10 18:53:56] ...optimizers/trijoint.py.151: total_norm: 17.943384 [I 2022-10-10 18:54:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 630/2304 [I 2022-10-10 18:54:04] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:46 [I 2022-10-10 18:54:04] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00300 [I 2022-10-10 18:54:04] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-10 18:54:04] ...optimizers/trijoint.py.151: total_norm: 14.063251 [I 2022-10-10 18:54:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 640/2304 [I 2022-10-10 18:54:13] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:41 [I 2022-10-10 18:54:13] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00297 [I 2022-10-10 18:54:13] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-10 18:54:13] ...optimizers/trijoint.py.151: total_norm: 11.386379 [I 2022-10-10 18:54:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 650/2304 [I 2022-10-10 18:54:21] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:35 [I 2022-10-10 18:54:21] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00298 [I 2022-10-10 18:54:21] ...trap/engines/engine.py.230: loss: 0.10116 [I 2022-10-10 18:54:21] ...optimizers/trijoint.py.151: total_norm: 16.614828 [I 2022-10-10 18:54:30] ...trap/engines/engine.py.224: train: epoch 82 | batch 660/2304 [I 2022-10-10 18:54:30] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:15 [I 2022-10-10 18:54:30] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00298 [I 2022-10-10 18:54:30] ...trap/engines/engine.py.230: loss: 0.18427 [I 2022-10-10 18:54:30] ...optimizers/trijoint.py.151: total_norm: 11.290164 [I 2022-10-10 18:54:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 670/2304 [I 2022-10-10 18:54:38] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:14 [I 2022-10-10 18:54:38] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00300 [I 2022-10-10 18:54:38] ...trap/engines/engine.py.230: loss: 0.11663 [I 2022-10-10 18:54:38] ...optimizers/trijoint.py.151: total_norm: 16.189604 [I 2022-10-10 18:54:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 680/2304 [I 2022-10-10 18:54:47] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:05 [I 2022-10-10 18:54:47] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00297 [I 2022-10-10 18:54:47] ...trap/engines/engine.py.230: loss: 0.11278 [I 2022-10-10 18:54:47] ...optimizers/trijoint.py.151: total_norm: 12.565929 [I 2022-10-10 18:54:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 690/2304 [I 2022-10-10 18:54:56] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:57 [I 2022-10-10 18:54:56] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00297 [I 2022-10-10 18:54:56] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-10 18:54:56] ...optimizers/trijoint.py.151: total_norm: 8.576555 [I 2022-10-10 18:55:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 700/2304 [I 2022-10-10 18:55:04] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:55 [I 2022-10-10 18:55:04] ...trap/engines/engine.py.229: process: 0.86100 | load: 0.00292 [I 2022-10-10 18:55:04] ...trap/engines/engine.py.230: loss: 0.14419 [I 2022-10-10 18:55:04] ...optimizers/trijoint.py.151: total_norm: 11.953054 [I 2022-10-10 18:55:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 710/2304 [I 2022-10-10 18:55:13] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:49 [I 2022-10-10 18:55:13] ...trap/engines/engine.py.229: process: 0.86368 | load: 0.00310 [I 2022-10-10 18:55:13] ...trap/engines/engine.py.230: loss: 0.12866 [I 2022-10-10 18:55:13] ...optimizers/trijoint.py.151: total_norm: 14.265685 [I 2022-10-10 18:55:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 720/2304 [I 2022-10-10 18:55:21] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:49 [I 2022-10-10 18:55:21] ...trap/engines/engine.py.229: process: 0.86700 | load: 0.00308 [I 2022-10-10 18:55:21] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-10 18:55:21] ...optimizers/trijoint.py.151: total_norm: 15.128920 [I 2022-10-10 18:55:30] ...trap/engines/engine.py.224: train: epoch 82 | batch 730/2304 [I 2022-10-10 18:55:30] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:41 [I 2022-10-10 18:55:30] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00306 [I 2022-10-10 18:55:30] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-10 18:55:30] ...optimizers/trijoint.py.151: total_norm: 12.420727 [I 2022-10-10 18:55:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 740/2304 [I 2022-10-10 18:55:39] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:34 [I 2022-10-10 18:55:39] ...trap/engines/engine.py.229: process: 0.87034 | load: 0.00311 [I 2022-10-10 18:55:39] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-10 18:55:39] ...optimizers/trijoint.py.151: total_norm: 19.203081 [I 2022-10-10 18:55:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 750/2304 [I 2022-10-10 18:55:47] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:23 [I 2022-10-10 18:55:47] ...trap/engines/engine.py.229: process: 0.86731 | load: 0.00319 [I 2022-10-10 18:55:47] ...trap/engines/engine.py.230: loss: 0.17659 [I 2022-10-10 18:55:47] ...optimizers/trijoint.py.151: total_norm: 18.848528 [I 2022-10-10 18:55:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 760/2304 [I 2022-10-10 18:55:56] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:17 [I 2022-10-10 18:55:56] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00318 [I 2022-10-10 18:55:56] ...trap/engines/engine.py.230: loss: 0.11282 [I 2022-10-10 18:55:56] ...optimizers/trijoint.py.151: total_norm: 15.522021 [I 2022-10-10 18:56:05] ...trap/engines/engine.py.224: train: epoch 82 | batch 770/2304 [I 2022-10-10 18:56:05] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:11 [I 2022-10-10 18:56:05] ...trap/engines/engine.py.229: process: 0.86951 | load: 0.00308 [I 2022-10-10 18:56:05] ...trap/engines/engine.py.230: loss: 0.14764 [I 2022-10-10 18:56:05] ...optimizers/trijoint.py.151: total_norm: 16.773922 [I 2022-10-10 18:56:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 780/2304 [I 2022-10-10 18:56:13] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:56 [I 2022-10-10 18:56:13] ...trap/engines/engine.py.229: process: 0.86002 | load: 0.00321 [I 2022-10-10 18:56:13] ...trap/engines/engine.py.230: loss: 0.10858 [I 2022-10-10 18:56:13] ...optimizers/trijoint.py.151: total_norm: 8.807974 [I 2022-10-10 18:56:22] ...trap/engines/engine.py.224: train: epoch 82 | batch 790/2304 [I 2022-10-10 18:56:22] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:53 [I 2022-10-10 18:56:22] ...trap/engines/engine.py.229: process: 0.87601 | load: 0.00308 [I 2022-10-10 18:56:22] ...trap/engines/engine.py.230: loss: 0.09475 [I 2022-10-10 18:56:22] ...optimizers/trijoint.py.151: total_norm: 8.641372 [I 2022-10-10 18:56:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 800/2304 [I 2022-10-10 18:56:31] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:50 [I 2022-10-10 18:56:31] ...trap/engines/engine.py.229: process: 0.86829 | load: 0.00323 [I 2022-10-10 18:56:31] ...trap/engines/engine.py.230: loss: 0.12299 [I 2022-10-10 18:56:31] ...optimizers/trijoint.py.151: total_norm: 11.724709 [I 2022-10-10 18:56:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 810/2304 [I 2022-10-10 18:56:39] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:33 [I 2022-10-10 18:56:39] ...trap/engines/engine.py.229: process: 0.86510 | load: 0.00310 [I 2022-10-10 18:56:39] ...trap/engines/engine.py.230: loss: 0.17061 [I 2022-10-10 18:56:39] ...optimizers/trijoint.py.151: total_norm: 11.328568 [I 2022-10-10 18:56:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 820/2304 [I 2022-10-10 18:56:48] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:30 [I 2022-10-10 18:56:48] ...trap/engines/engine.py.229: process: 0.86582 | load: 0.00307 [I 2022-10-10 18:56:48] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-10 18:56:48] ...optimizers/trijoint.py.151: total_norm: 21.606581 [I 2022-10-10 18:56:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 830/2304 [I 2022-10-10 18:56:57] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:13 [I 2022-10-10 18:56:57] ...trap/engines/engine.py.229: process: 0.86182 | load: 0.00308 [I 2022-10-10 18:56:57] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-10 18:56:57] ...optimizers/trijoint.py.151: total_norm: 27.058149 [I 2022-10-10 18:57:05] ...trap/engines/engine.py.224: train: epoch 82 | batch 840/2304 [I 2022-10-10 18:57:05] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:01 [I 2022-10-10 18:57:05] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00306 [I 2022-10-10 18:57:05] ...trap/engines/engine.py.230: loss: 0.16513 [I 2022-10-10 18:57:05] ...optimizers/trijoint.py.151: total_norm: 31.941475 [I 2022-10-10 18:57:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 850/2304 [I 2022-10-10 18:57:14] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:46 [I 2022-10-10 18:57:14] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00307 [I 2022-10-10 18:57:14] ...trap/engines/engine.py.230: loss: 0.17893 [I 2022-10-10 18:57:14] ...optimizers/trijoint.py.151: total_norm: 17.458698 [I 2022-10-10 18:57:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 860/2304 [I 2022-10-10 18:57:23] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:35 [I 2022-10-10 18:57:23] ...trap/engines/engine.py.229: process: 0.84345 | load: 0.00307 [I 2022-10-10 18:57:23] ...trap/engines/engine.py.230: loss: 0.13689 [I 2022-10-10 18:57:23] ...optimizers/trijoint.py.151: total_norm: 8.788153 [I 2022-10-10 18:57:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 870/2304 [I 2022-10-10 18:57:31] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:36 [I 2022-10-10 18:57:31] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00309 [I 2022-10-10 18:57:31] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-10 18:57:31] ...optimizers/trijoint.py.151: total_norm: 11.967911 [I 2022-10-10 18:57:40] ...trap/engines/engine.py.224: train: epoch 82 | batch 880/2304 [I 2022-10-10 18:57:40] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:28 [I 2022-10-10 18:57:40] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00314 [I 2022-10-10 18:57:40] ...trap/engines/engine.py.230: loss: 0.19116 [I 2022-10-10 18:57:40] ...optimizers/trijoint.py.151: total_norm: 34.960011 [I 2022-10-10 18:57:49] ...trap/engines/engine.py.224: train: epoch 82 | batch 890/2304 [I 2022-10-10 18:57:49] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:17 [I 2022-10-10 18:57:49] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00313 [I 2022-10-10 18:57:49] ...trap/engines/engine.py.230: loss: 0.15388 [I 2022-10-10 18:57:49] ...optimizers/trijoint.py.151: total_norm: 13.371115 [I 2022-10-10 18:57:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 900/2304 [I 2022-10-10 18:57:57] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:10 [I 2022-10-10 18:57:57] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00314 [I 2022-10-10 18:57:57] ...trap/engines/engine.py.230: loss: 0.14946 [I 2022-10-10 18:57:57] ...optimizers/trijoint.py.151: total_norm: 17.606562 [I 2022-10-10 18:58:06] ...trap/engines/engine.py.224: train: epoch 82 | batch 910/2304 [I 2022-10-10 18:58:06] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:05 [I 2022-10-10 18:58:06] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00318 [I 2022-10-10 18:58:06] ...trap/engines/engine.py.230: loss: 0.17660 [I 2022-10-10 18:58:06] ...optimizers/trijoint.py.151: total_norm: 11.325788 [I 2022-10-10 18:58:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 920/2304 [I 2022-10-10 18:58:15] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:51 [I 2022-10-10 18:58:15] ...trap/engines/engine.py.229: process: 0.85870 | load: 0.00309 [I 2022-10-10 18:58:15] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-10 18:58:15] ...optimizers/trijoint.py.151: total_norm: 44.892136 [I 2022-10-10 18:58:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 930/2304 [I 2022-10-10 18:58:23] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:48 [I 2022-10-10 18:58:23] ...trap/engines/engine.py.229: process: 0.87134 | load: 0.00310 [I 2022-10-10 18:58:23] ...trap/engines/engine.py.230: loss: 0.19399 [I 2022-10-10 18:58:23] ...optimizers/trijoint.py.151: total_norm: 47.072552 [I 2022-10-10 18:58:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 940/2304 [I 2022-10-10 18:58:32] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:38 [I 2022-10-10 18:58:32] ...trap/engines/engine.py.229: process: 0.86898 | load: 0.00334 [I 2022-10-10 18:58:32] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-10 18:58:32] ...optimizers/trijoint.py.151: total_norm: 15.340158 [I 2022-10-10 18:58:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 950/2304 [I 2022-10-10 18:58:41] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:27 [I 2022-10-10 18:58:41] ...trap/engines/engine.py.229: process: 0.86945 | load: 0.00307 [I 2022-10-10 18:58:41] ...trap/engines/engine.py.230: loss: 0.11093 [I 2022-10-10 18:58:41] ...optimizers/trijoint.py.151: total_norm: 26.838577 [I 2022-10-10 18:58:49] ...trap/engines/engine.py.224: train: epoch 82 | batch 960/2304 [I 2022-10-10 18:58:49] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:21 [I 2022-10-10 18:58:49] ...trap/engines/engine.py.229: process: 0.87659 | load: 0.00330 [I 2022-10-10 18:58:49] ...trap/engines/engine.py.230: loss: 0.10314 [I 2022-10-10 18:58:49] ...optimizers/trijoint.py.151: total_norm: 18.654400 [I 2022-10-10 18:58:58] ...trap/engines/engine.py.224: train: epoch 82 | batch 970/2304 [I 2022-10-10 18:58:58] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:15 [I 2022-10-10 18:58:58] ...trap/engines/engine.py.229: process: 0.86452 | load: 0.00305 [I 2022-10-10 18:58:58] ...trap/engines/engine.py.230: loss: 0.12045 [I 2022-10-10 18:58:58] ...optimizers/trijoint.py.151: total_norm: 11.311797 [I 2022-10-10 18:59:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 980/2304 [I 2022-10-10 18:59:07] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:54 [I 2022-10-10 18:59:07] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00300 [I 2022-10-10 18:59:07] ...trap/engines/engine.py.230: loss: 0.14581 [I 2022-10-10 18:59:07] ...optimizers/trijoint.py.151: total_norm: 19.819740 [I 2022-10-10 18:59:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 990/2304 [I 2022-10-10 18:59:15] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:47 [I 2022-10-10 18:59:15] ...trap/engines/engine.py.229: process: 0.86307 | load: 0.00294 [I 2022-10-10 18:59:15] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-10 18:59:15] ...optimizers/trijoint.py.151: total_norm: 14.883242 [I 2022-10-10 18:59:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 1000/2304 [I 2022-10-10 18:59:24] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:31 [I 2022-10-10 18:59:24] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00299 [I 2022-10-10 18:59:24] ...trap/engines/engine.py.230: loss: 0.16086 [I 2022-10-10 18:59:24] ...optimizers/trijoint.py.151: total_norm: 14.001413 [I 2022-10-10 18:59:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 1010/2304 [I 2022-10-10 18:59:32] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:30 [I 2022-10-10 18:59:32] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00296 [I 2022-10-10 18:59:32] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-10 18:59:32] ...optimizers/trijoint.py.151: total_norm: 28.939297 [I 2022-10-10 18:59:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 1020/2304 [I 2022-10-10 18:59:41] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:24 [I 2022-10-10 18:59:41] ...trap/engines/engine.py.229: process: 0.85995 | load: 0.00304 [I 2022-10-10 18:59:41] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-10 18:59:41] ...optimizers/trijoint.py.151: total_norm: 12.092447 [I 2022-10-10 18:59:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 1030/2304 [I 2022-10-10 18:59:50] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:04 [I 2022-10-10 18:59:50] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00298 [I 2022-10-10 18:59:50] ...trap/engines/engine.py.230: loss: 0.11985 [I 2022-10-10 18:59:50] ...optimizers/trijoint.py.151: total_norm: 11.237901 [I 2022-10-10 18:59:58] ...trap/engines/engine.py.224: train: epoch 82 | batch 1040/2304 [I 2022-10-10 18:59:58] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:59 [I 2022-10-10 18:59:58] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00299 [I 2022-10-10 18:59:58] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-10 18:59:58] ...optimizers/trijoint.py.151: total_norm: 16.202377 [I 2022-10-10 19:00:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 1050/2304 [I 2022-10-10 19:00:07] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:51 [I 2022-10-10 19:00:07] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00299 [I 2022-10-10 19:00:07] ...trap/engines/engine.py.230: loss: 0.13797 [I 2022-10-10 19:00:07] ...optimizers/trijoint.py.151: total_norm: 32.198025 [I 2022-10-10 19:00:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 1060/2304 [I 2022-10-10 19:00:15] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:41 [I 2022-10-10 19:00:15] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00312 [I 2022-10-10 19:00:15] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-10 19:00:15] ...optimizers/trijoint.py.151: total_norm: 21.912334 [I 2022-10-10 19:00:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 1070/2304 [I 2022-10-10 19:00:24] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:34 [I 2022-10-10 19:00:24] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00302 [I 2022-10-10 19:00:24] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-10 19:00:24] ...optimizers/trijoint.py.151: total_norm: 11.240335 [I 2022-10-10 19:00:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 1080/2304 [I 2022-10-10 19:00:32] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:25 [I 2022-10-10 19:00:32] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00303 [I 2022-10-10 19:00:32] ...trap/engines/engine.py.230: loss: 0.13271 [I 2022-10-10 19:00:32] ...optimizers/trijoint.py.151: total_norm: 13.552488 [I 2022-10-10 19:00:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 1090/2304 [I 2022-10-10 19:00:41] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:19 [I 2022-10-10 19:00:41] ...trap/engines/engine.py.229: process: 0.85963 | load: 0.00299 [I 2022-10-10 19:00:41] ...trap/engines/engine.py.230: loss: 0.17365 [I 2022-10-10 19:00:41] ...optimizers/trijoint.py.151: total_norm: 60.277508 [I 2022-10-10 19:00:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 1100/2304 [I 2022-10-10 19:00:50] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:09 [I 2022-10-10 19:00:50] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00301 [I 2022-10-10 19:00:50] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-10 19:00:50] ...optimizers/trijoint.py.151: total_norm: 23.415815 [I 2022-10-10 19:00:58] ...trap/engines/engine.py.224: train: epoch 82 | batch 1110/2304 [I 2022-10-10 19:00:58] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:59 [I 2022-10-10 19:00:58] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00303 [I 2022-10-10 19:00:58] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-10 19:00:58] ...optimizers/trijoint.py.151: total_norm: 10.720366 [I 2022-10-10 19:01:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 1120/2304 [I 2022-10-10 19:01:07] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:54 [I 2022-10-10 19:01:07] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00305 [I 2022-10-10 19:01:07] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-10 19:01:07] ...optimizers/trijoint.py.151: total_norm: 14.857162 [I 2022-10-10 19:01:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 1130/2304 [I 2022-10-10 19:01:15] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:58 [I 2022-10-10 19:01:15] ...trap/engines/engine.py.229: process: 0.88197 | load: 0.00312 [I 2022-10-10 19:01:15] ...trap/engines/engine.py.230: loss: 0.11662 [I 2022-10-10 19:01:15] ...optimizers/trijoint.py.151: total_norm: 23.642950 [I 2022-10-10 19:01:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 1140/2304 [I 2022-10-10 19:01:24] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:47 [I 2022-10-10 19:01:24] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00308 [I 2022-10-10 19:01:24] ...trap/engines/engine.py.230: loss: 0.09542 [I 2022-10-10 19:01:24] ...optimizers/trijoint.py.151: total_norm: 8.881046 [I 2022-10-10 19:01:33] ...trap/engines/engine.py.224: train: epoch 82 | batch 1150/2304 [I 2022-10-10 19:01:33] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:28 [I 2022-10-10 19:01:33] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00311 [I 2022-10-10 19:01:33] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-10 19:01:33] ...optimizers/trijoint.py.151: total_norm: 8.642297 [I 2022-10-10 19:01:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 1160/2304 [I 2022-10-10 19:01:41] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:26 [I 2022-10-10 19:01:41] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00310 [I 2022-10-10 19:01:41] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-10 19:01:41] ...optimizers/trijoint.py.151: total_norm: 49.981194 [I 2022-10-10 19:01:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 1170/2304 [I 2022-10-10 19:01:50] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:15 [I 2022-10-10 19:01:50] ...trap/engines/engine.py.229: process: 0.85720 | load: 0.00309 [I 2022-10-10 19:01:50] ...trap/engines/engine.py.230: loss: 0.11662 [I 2022-10-10 19:01:50] ...optimizers/trijoint.py.151: total_norm: 9.756654 [I 2022-10-10 19:01:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 1180/2304 [I 2022-10-10 19:01:59] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:09 [I 2022-10-10 19:01:59] ...trap/engines/engine.py.229: process: 0.86301 | load: 0.00310 [I 2022-10-10 19:01:59] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-10 19:01:59] ...optimizers/trijoint.py.151: total_norm: 20.448877 [I 2022-10-10 19:02:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 1190/2304 [I 2022-10-10 19:02:07] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:16:11 [I 2022-10-10 19:02:07] ...trap/engines/engine.py.229: process: 0.87590 | load: 0.00315 [I 2022-10-10 19:02:07] ...trap/engines/engine.py.230: loss: 0.08761 [I 2022-10-10 19:02:07] ...optimizers/trijoint.py.151: total_norm: 8.612308 [I 2022-10-10 19:02:16] ...trap/engines/engine.py.224: train: epoch 82 | batch 1200/2304 [I 2022-10-10 19:02:16] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:57 [I 2022-10-10 19:02:16] ...trap/engines/engine.py.229: process: 0.86910 | load: 0.00325 [I 2022-10-10 19:02:16] ...trap/engines/engine.py.230: loss: 0.15058 [I 2022-10-10 19:02:16] ...optimizers/trijoint.py.151: total_norm: 38.370094 [I 2022-10-10 19:02:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1210/2304 [I 2022-10-10 19:02:25] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:49 [I 2022-10-10 19:02:25] ...trap/engines/engine.py.229: process: 0.87002 | load: 0.00314 [I 2022-10-10 19:02:25] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-10 19:02:25] ...optimizers/trijoint.py.151: total_norm: 13.161546 [I 2022-10-10 19:02:33] ...trap/engines/engine.py.224: train: epoch 82 | batch 1220/2304 [I 2022-10-10 19:02:33] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:35 [I 2022-10-10 19:02:33] ...trap/engines/engine.py.229: process: 0.86459 | load: 0.00310 [I 2022-10-10 19:02:33] ...trap/engines/engine.py.230: loss: 0.09721 [I 2022-10-10 19:02:33] ...optimizers/trijoint.py.151: total_norm: 13.293001 [I 2022-10-10 19:02:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1230/2304 [I 2022-10-10 19:02:42] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:15:25 [I 2022-10-10 19:02:42] ...trap/engines/engine.py.229: process: 0.84991 | load: 0.00303 [I 2022-10-10 19:02:42] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-10 19:02:42] ...optimizers/trijoint.py.151: total_norm: 19.093090 [I 2022-10-10 19:02:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1240/2304 [I 2022-10-10 19:02:51] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:13 [I 2022-10-10 19:02:51] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00328 [I 2022-10-10 19:02:51] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-10 19:02:51] ...optimizers/trijoint.py.151: total_norm: 11.040249 [I 2022-10-10 19:02:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 1250/2304 [I 2022-10-10 19:02:59] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:15:05 [I 2022-10-10 19:02:59] ...trap/engines/engine.py.229: process: 0.88259 | load: 0.00314 [I 2022-10-10 19:02:59] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-10 19:02:59] ...optimizers/trijoint.py.151: total_norm: 10.636934 [I 2022-10-10 19:03:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 1260/2304 [I 2022-10-10 19:03:08] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:56 [I 2022-10-10 19:03:08] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00311 [I 2022-10-10 19:03:08] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-10 19:03:08] ...optimizers/trijoint.py.151: total_norm: 16.501095 [I 2022-10-10 19:03:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1270/2304 [I 2022-10-10 19:03:17] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:46 [I 2022-10-10 19:03:17] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00344 [I 2022-10-10 19:03:17] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-10 19:03:17] ...optimizers/trijoint.py.151: total_norm: 13.964343 [I 2022-10-10 19:03:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1280/2304 [I 2022-10-10 19:03:25] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:39 [I 2022-10-10 19:03:25] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00649 [I 2022-10-10 19:03:25] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-10 19:03:25] ...optimizers/trijoint.py.151: total_norm: 9.858413 [I 2022-10-10 19:03:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1290/2304 [I 2022-10-10 19:03:34] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:35 [I 2022-10-10 19:03:34] ...trap/engines/engine.py.229: process: 0.86024 | load: 0.00312 [I 2022-10-10 19:03:34] ...trap/engines/engine.py.230: loss: 0.10060 [I 2022-10-10 19:03:34] ...optimizers/trijoint.py.151: total_norm: 24.381031 [I 2022-10-10 19:03:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1300/2304 [I 2022-10-10 19:03:42] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:22 [I 2022-10-10 19:03:42] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00312 [I 2022-10-10 19:03:42] ...trap/engines/engine.py.230: loss: 0.17685 [I 2022-10-10 19:03:42] ...optimizers/trijoint.py.151: total_norm: 20.525175 [I 2022-10-10 19:03:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1310/2304 [I 2022-10-10 19:03:51] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:22 [I 2022-10-10 19:03:51] ...trap/engines/engine.py.229: process: 0.86228 | load: 0.00309 [I 2022-10-10 19:03:51] ...trap/engines/engine.py.230: loss: 0.09641 [I 2022-10-10 19:03:51] ...optimizers/trijoint.py.151: total_norm: 11.091546 [I 2022-10-10 19:04:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1320/2304 [I 2022-10-10 19:04:00] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:14 [I 2022-10-10 19:04:00] ...trap/engines/engine.py.229: process: 0.86884 | load: 0.00308 [I 2022-10-10 19:04:00] ...trap/engines/engine.py.230: loss: 0.13899 [I 2022-10-10 19:04:00] ...optimizers/trijoint.py.151: total_norm: 10.200323 [I 2022-10-10 19:04:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1330/2304 [I 2022-10-10 19:04:09] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:14:07 [I 2022-10-10 19:04:09] ...trap/engines/engine.py.229: process: 0.87702 | load: 0.00312 [I 2022-10-10 19:04:09] ...trap/engines/engine.py.230: loss: 0.16971 [I 2022-10-10 19:04:09] ...optimizers/trijoint.py.151: total_norm: 17.158566 [I 2022-10-10 19:04:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1340/2304 [I 2022-10-10 19:04:17] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:47 [I 2022-10-10 19:04:17] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00307 [I 2022-10-10 19:04:17] ...trap/engines/engine.py.230: loss: 0.11007 [I 2022-10-10 19:04:17] ...optimizers/trijoint.py.151: total_norm: 9.644479 [I 2022-10-10 19:04:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1350/2304 [I 2022-10-10 19:04:26] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:40 [I 2022-10-10 19:04:26] ...trap/engines/engine.py.229: process: 0.86151 | load: 0.00313 [I 2022-10-10 19:04:26] ...trap/engines/engine.py.230: loss: 0.09827 [I 2022-10-10 19:04:26] ...optimizers/trijoint.py.151: total_norm: 13.176446 [I 2022-10-10 19:04:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1360/2304 [I 2022-10-10 19:04:34] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:31 [I 2022-10-10 19:04:34] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00314 [I 2022-10-10 19:04:34] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-10 19:04:34] ...optimizers/trijoint.py.151: total_norm: 30.205006 [I 2022-10-10 19:04:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1370/2304 [I 2022-10-10 19:04:43] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:21 [I 2022-10-10 19:04:43] ...trap/engines/engine.py.229: process: 0.85550 | load: 0.00317 [I 2022-10-10 19:04:43] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-10 19:04:43] ...optimizers/trijoint.py.151: total_norm: 13.512606 [I 2022-10-10 19:04:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1380/2304 [I 2022-10-10 19:04:52] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:14 [I 2022-10-10 19:04:52] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00313 [I 2022-10-10 19:04:52] ...trap/engines/engine.py.230: loss: 0.13096 [I 2022-10-10 19:04:52] ...optimizers/trijoint.py.151: total_norm: 14.626410 [I 2022-10-10 19:05:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1390/2304 [I 2022-10-10 19:05:00] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:01 [I 2022-10-10 19:05:00] ...trap/engines/engine.py.229: process: 0.85184 | load: 0.00308 [I 2022-10-10 19:05:00] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-10 19:05:00] ...optimizers/trijoint.py.151: total_norm: 33.874992 [I 2022-10-10 19:05:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1400/2304 [I 2022-10-10 19:05:09] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:52 [I 2022-10-10 19:05:09] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00293 [I 2022-10-10 19:05:09] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-10 19:05:09] ...optimizers/trijoint.py.151: total_norm: 18.142069 [I 2022-10-10 19:05:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1410/2304 [I 2022-10-10 19:05:17] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:42 [I 2022-10-10 19:05:17] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00298 [I 2022-10-10 19:05:17] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-10 19:05:17] ...optimizers/trijoint.py.151: total_norm: 15.970435 [I 2022-10-10 19:05:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1420/2304 [I 2022-10-10 19:05:26] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:33 [I 2022-10-10 19:05:26] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00298 [I 2022-10-10 19:05:26] ...trap/engines/engine.py.230: loss: 0.14577 [I 2022-10-10 19:05:26] ...optimizers/trijoint.py.151: total_norm: 16.796299 [I 2022-10-10 19:05:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1430/2304 [I 2022-10-10 19:05:34] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:16 [I 2022-10-10 19:05:34] ...trap/engines/engine.py.229: process: 0.83511 | load: 0.00296 [I 2022-10-10 19:05:34] ...trap/engines/engine.py.230: loss: 0.12258 [I 2022-10-10 19:05:34] ...optimizers/trijoint.py.151: total_norm: 12.031993 [I 2022-10-10 19:05:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1440/2304 [I 2022-10-10 19:05:43] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:12:12 [I 2022-10-10 19:05:43] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00292 [I 2022-10-10 19:05:43] ...trap/engines/engine.py.230: loss: 0.11856 [I 2022-10-10 19:05:43] ...optimizers/trijoint.py.151: total_norm: 16.892138 [I 2022-10-10 19:05:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1450/2304 [I 2022-10-10 19:05:51] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:04 [I 2022-10-10 19:05:51] ...trap/engines/engine.py.229: process: 0.85981 | load: 0.00295 [I 2022-10-10 19:05:51] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-10 19:05:51] ...optimizers/trijoint.py.151: total_norm: 14.398053 [I 2022-10-10 19:06:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1460/2304 [I 2022-10-10 19:06:00] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:03 [I 2022-10-10 19:06:00] ...trap/engines/engine.py.229: process: 0.87666 | load: 0.00291 [I 2022-10-10 19:06:00] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-10 19:06:00] ...optimizers/trijoint.py.151: total_norm: 9.760857 [I 2022-10-10 19:06:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1470/2304 [I 2022-10-10 19:06:09] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:48 [I 2022-10-10 19:06:09] ...trap/engines/engine.py.229: process: 0.85343 | load: 0.00290 [I 2022-10-10 19:06:09] ...trap/engines/engine.py.230: loss: 0.19424 [I 2022-10-10 19:06:09] ...optimizers/trijoint.py.151: total_norm: 17.851099 [I 2022-10-10 19:06:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1480/2304 [I 2022-10-10 19:06:17] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:41 [I 2022-10-10 19:06:17] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00294 [I 2022-10-10 19:06:17] ...trap/engines/engine.py.230: loss: 0.18718 [I 2022-10-10 19:06:17] ...optimizers/trijoint.py.151: total_norm: 18.516537 [I 2022-10-10 19:06:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1490/2304 [I 2022-10-10 19:06:26] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:34 [I 2022-10-10 19:06:26] ...trap/engines/engine.py.229: process: 0.85865 | load: 0.00290 [I 2022-10-10 19:06:26] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-10 19:06:26] ...optimizers/trijoint.py.151: total_norm: 38.924225 [I 2022-10-10 19:06:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1500/2304 [I 2022-10-10 19:06:34] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:22 [I 2022-10-10 19:06:34] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00292 [I 2022-10-10 19:06:34] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-10 19:06:34] ...optimizers/trijoint.py.151: total_norm: 30.237360 [I 2022-10-10 19:06:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1510/2304 [I 2022-10-10 19:06:43] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:14 [I 2022-10-10 19:06:43] ...trap/engines/engine.py.229: process: 0.85402 | load: 0.00292 [I 2022-10-10 19:06:43] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-10 19:06:43] ...optimizers/trijoint.py.151: total_norm: 15.935729 [I 2022-10-10 19:06:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1520/2304 [I 2022-10-10 19:06:51] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:08 [I 2022-10-10 19:06:51] ...trap/engines/engine.py.229: process: 0.85732 | load: 0.00293 [I 2022-10-10 19:06:51] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-10 19:06:51] ...optimizers/trijoint.py.151: total_norm: 13.091431 [I 2022-10-10 19:07:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1530/2304 [I 2022-10-10 19:07:00] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:59 [I 2022-10-10 19:07:00] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00296 [I 2022-10-10 19:07:00] ...trap/engines/engine.py.230: loss: 0.08595 [I 2022-10-10 19:07:00] ...optimizers/trijoint.py.151: total_norm: 13.758485 [I 2022-10-10 19:07:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 1540/2304 [I 2022-10-10 19:07:08] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:54 [I 2022-10-10 19:07:08] ...trap/engines/engine.py.229: process: 0.86004 | load: 0.00295 [I 2022-10-10 19:07:08] ...trap/engines/engine.py.230: loss: 0.09327 [I 2022-10-10 19:07:08] ...optimizers/trijoint.py.151: total_norm: 7.872740 [I 2022-10-10 19:07:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1550/2304 [I 2022-10-10 19:07:17] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:43 [I 2022-10-10 19:07:17] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00290 [I 2022-10-10 19:07:17] ...trap/engines/engine.py.230: loss: 0.12517 [I 2022-10-10 19:07:17] ...optimizers/trijoint.py.151: total_norm: 10.570665 [I 2022-10-10 19:07:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1560/2304 [I 2022-10-10 19:07:25] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:35 [I 2022-10-10 19:07:25] ...trap/engines/engine.py.229: process: 0.86620 | load: 0.00292 [I 2022-10-10 19:07:25] ...trap/engines/engine.py.230: loss: 0.09225 [I 2022-10-10 19:07:25] ...optimizers/trijoint.py.151: total_norm: 12.416140 [I 2022-10-10 19:07:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1570/2304 [I 2022-10-10 19:07:34] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:10:26 [I 2022-10-10 19:07:34] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00317 [I 2022-10-10 19:07:34] ...trap/engines/engine.py.230: loss: 0.10924 [I 2022-10-10 19:07:34] ...optimizers/trijoint.py.151: total_norm: 9.529827 [I 2022-10-10 19:07:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1580/2304 [I 2022-10-10 19:07:42] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:22 [I 2022-10-10 19:07:42] ...trap/engines/engine.py.229: process: 0.86992 | load: 0.00293 [I 2022-10-10 19:07:42] ...trap/engines/engine.py.230: loss: 0.09238 [I 2022-10-10 19:07:42] ...optimizers/trijoint.py.151: total_norm: 15.179893 [I 2022-10-10 19:07:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1590/2304 [I 2022-10-10 19:07:51] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:08 [I 2022-10-10 19:07:51] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00297 [I 2022-10-10 19:07:51] ...trap/engines/engine.py.230: loss: 0.18841 [I 2022-10-10 19:07:51] ...optimizers/trijoint.py.151: total_norm: 12.436078 [I 2022-10-10 19:08:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1600/2304 [I 2022-10-10 19:08:00] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:10:00 [I 2022-10-10 19:08:00] ...trap/engines/engine.py.229: process: 0.85090 | load: 0.00310 [I 2022-10-10 19:08:00] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-10 19:08:00] ...optimizers/trijoint.py.151: total_norm: 10.429226 [I 2022-10-10 19:08:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 1610/2304 [I 2022-10-10 19:08:08] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:53 [I 2022-10-10 19:08:08] ...trap/engines/engine.py.229: process: 0.85468 | load: 0.00311 [I 2022-10-10 19:08:08] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-10 19:08:08] ...optimizers/trijoint.py.151: total_norm: 8.256795 [I 2022-10-10 19:08:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1620/2304 [I 2022-10-10 19:08:17] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:48 [I 2022-10-10 19:08:17] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00314 [I 2022-10-10 19:08:17] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-10 19:08:17] ...optimizers/trijoint.py.151: total_norm: 10.225866 [I 2022-10-10 19:08:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1630/2304 [I 2022-10-10 19:08:25] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:40 [I 2022-10-10 19:08:25] ...trap/engines/engine.py.229: process: 0.86361 | load: 0.00310 [I 2022-10-10 19:08:25] ...trap/engines/engine.py.230: loss: 0.20005 [I 2022-10-10 19:08:25] ...optimizers/trijoint.py.151: total_norm: 10.051388 [I 2022-10-10 19:08:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1640/2304 [I 2022-10-10 19:08:34] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:29 [I 2022-10-10 19:08:34] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00301 [I 2022-10-10 19:08:34] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-10 19:08:34] ...optimizers/trijoint.py.151: total_norm: 13.250046 [I 2022-10-10 19:08:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1650/2304 [I 2022-10-10 19:08:43] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:23 [I 2022-10-10 19:08:43] ...trap/engines/engine.py.229: process: 0.86842 | load: 0.00307 [I 2022-10-10 19:08:43] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-10 19:08:43] ...optimizers/trijoint.py.151: total_norm: 8.920474 [I 2022-10-10 19:08:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1660/2304 [I 2022-10-10 19:08:51] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:09:16 [I 2022-10-10 19:08:51] ...trap/engines/engine.py.229: process: 0.86633 | load: 0.00308 [I 2022-10-10 19:08:51] ...trap/engines/engine.py.230: loss: 0.12730 [I 2022-10-10 19:08:51] ...optimizers/trijoint.py.151: total_norm: 12.626667 [I 2022-10-10 19:09:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1670/2304 [I 2022-10-10 19:09:00] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:09:06 [I 2022-10-10 19:09:00] ...trap/engines/engine.py.229: process: 0.86527 | load: 0.00310 [I 2022-10-10 19:09:00] ...trap/engines/engine.py.230: loss: 0.14290 [I 2022-10-10 19:09:00] ...optimizers/trijoint.py.151: total_norm: 9.063479 [I 2022-10-10 19:09:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1680/2304 [I 2022-10-10 19:09:09] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:56 [I 2022-10-10 19:09:09] ...trap/engines/engine.py.229: process: 0.86626 | load: 0.00307 [I 2022-10-10 19:09:09] ...trap/engines/engine.py.230: loss: 0.08862 [I 2022-10-10 19:09:09] ...optimizers/trijoint.py.151: total_norm: 8.141169 [I 2022-10-10 19:09:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1690/2304 [I 2022-10-10 19:09:17] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:47 [I 2022-10-10 19:09:17] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00317 [I 2022-10-10 19:09:17] ...trap/engines/engine.py.230: loss: 0.14753 [I 2022-10-10 19:09:17] ...optimizers/trijoint.py.151: total_norm: 11.965081 [I 2022-10-10 19:09:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1700/2304 [I 2022-10-10 19:09:26] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:39 [I 2022-10-10 19:09:26] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00318 [I 2022-10-10 19:09:26] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-10 19:09:26] ...optimizers/trijoint.py.151: total_norm: 25.187761 [I 2022-10-10 19:09:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 1710/2304 [I 2022-10-10 19:09:35] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:31 [I 2022-10-10 19:09:35] ...trap/engines/engine.py.229: process: 0.86035 | load: 0.00308 [I 2022-10-10 19:09:35] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-10 19:09:35] ...optimizers/trijoint.py.151: total_norm: 10.215976 [I 2022-10-10 19:09:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1720/2304 [I 2022-10-10 19:09:43] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:24 [I 2022-10-10 19:09:43] ...trap/engines/engine.py.229: process: 0.87506 | load: 0.00309 [I 2022-10-10 19:09:43] ...trap/engines/engine.py.230: loss: 0.30084 [I 2022-10-10 19:09:43] ...optimizers/trijoint.py.151: total_norm: 19.631569 [I 2022-10-10 19:09:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1730/2304 [I 2022-10-10 19:09:52] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:08:14 [I 2022-10-10 19:09:52] ...trap/engines/engine.py.229: process: 0.85393 | load: 0.00309 [I 2022-10-10 19:09:52] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-10 19:09:52] ...optimizers/trijoint.py.151: total_norm: 13.737703 [I 2022-10-10 19:10:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 1740/2304 [I 2022-10-10 19:10:01] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:08:06 [I 2022-10-10 19:10:01] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00310 [I 2022-10-10 19:10:01] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-10 19:10:01] ...optimizers/trijoint.py.151: total_norm: 18.637995 [I 2022-10-10 19:10:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1750/2304 [I 2022-10-10 19:10:09] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:59 [I 2022-10-10 19:10:09] ...trap/engines/engine.py.229: process: 0.86784 | load: 0.00312 [I 2022-10-10 19:10:09] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-10 19:10:09] ...optimizers/trijoint.py.151: total_norm: 18.644623 [I 2022-10-10 19:10:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 1760/2304 [I 2022-10-10 19:10:18] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:46 [I 2022-10-10 19:10:18] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00318 [I 2022-10-10 19:10:18] ...trap/engines/engine.py.230: loss: 0.17707 [I 2022-10-10 19:10:18] ...optimizers/trijoint.py.151: total_norm: 12.151921 [I 2022-10-10 19:10:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1770/2304 [I 2022-10-10 19:10:26] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:42 [I 2022-10-10 19:10:26] ...trap/engines/engine.py.229: process: 0.85893 | load: 0.00321 [I 2022-10-10 19:10:26] ...trap/engines/engine.py.230: loss: 0.09659 [I 2022-10-10 19:10:26] ...optimizers/trijoint.py.151: total_norm: 7.600942 [I 2022-10-10 19:10:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 1780/2304 [I 2022-10-10 19:10:35] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:32 [I 2022-10-10 19:10:35] ...trap/engines/engine.py.229: process: 0.87859 | load: 0.00308 [I 2022-10-10 19:10:35] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-10 19:10:35] ...optimizers/trijoint.py.151: total_norm: 27.658962 [I 2022-10-10 19:10:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 1790/2304 [I 2022-10-10 19:10:44] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:07:24 [I 2022-10-10 19:10:44] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00308 [I 2022-10-10 19:10:44] ...trap/engines/engine.py.230: loss: 0.09997 [I 2022-10-10 19:10:44] ...optimizers/trijoint.py.151: total_norm: 12.581403 [I 2022-10-10 19:10:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1800/2304 [I 2022-10-10 19:10:52] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:07:16 [I 2022-10-10 19:10:52] ...trap/engines/engine.py.229: process: 0.86796 | load: 0.00307 [I 2022-10-10 19:10:52] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-10 19:10:52] ...optimizers/trijoint.py.151: total_norm: 11.454866 [I 2022-10-10 19:11:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 1810/2304 [I 2022-10-10 19:11:01] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:07:09 [I 2022-10-10 19:11:01] ...trap/engines/engine.py.229: process: 0.86624 | load: 0.00333 [I 2022-10-10 19:11:01] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-10 19:11:01] ...optimizers/trijoint.py.151: total_norm: 14.463206 [I 2022-10-10 19:11:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 1820/2304 [I 2022-10-10 19:11:10] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:57 [I 2022-10-10 19:11:10] ...trap/engines/engine.py.229: process: 0.86428 | load: 0.00305 [I 2022-10-10 19:11:10] ...trap/engines/engine.py.230: loss: 0.14065 [I 2022-10-10 19:11:10] ...optimizers/trijoint.py.151: total_norm: 16.361694 [I 2022-10-10 19:11:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 1830/2304 [I 2022-10-10 19:11:18] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:49 [I 2022-10-10 19:11:18] ...trap/engines/engine.py.229: process: 0.86735 | load: 0.00309 [I 2022-10-10 19:11:18] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-10 19:11:18] ...optimizers/trijoint.py.151: total_norm: 12.911654 [I 2022-10-10 19:11:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 1840/2304 [I 2022-10-10 19:11:27] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:40 [I 2022-10-10 19:11:27] ...trap/engines/engine.py.229: process: 0.86578 | load: 0.00308 [I 2022-10-10 19:11:27] ...trap/engines/engine.py.230: loss: 0.10493 [I 2022-10-10 19:11:27] ...optimizers/trijoint.py.151: total_norm: 11.129708 [I 2022-10-10 19:11:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 1850/2304 [I 2022-10-10 19:11:36] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:33 [I 2022-10-10 19:11:36] ...trap/engines/engine.py.229: process: 0.86410 | load: 0.00307 [I 2022-10-10 19:11:36] ...trap/engines/engine.py.230: loss: 0.14154 [I 2022-10-10 19:11:36] ...optimizers/trijoint.py.151: total_norm: 13.923023 [I 2022-10-10 19:11:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 1860/2304 [I 2022-10-10 19:11:45] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:24 [I 2022-10-10 19:11:45] ...trap/engines/engine.py.229: process: 0.86672 | load: 0.00310 [I 2022-10-10 19:11:45] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-10 19:11:45] ...optimizers/trijoint.py.151: total_norm: 12.919023 [I 2022-10-10 19:11:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 1870/2304 [I 2022-10-10 19:11:53] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:13 [I 2022-10-10 19:11:53] ...trap/engines/engine.py.229: process: 0.87119 | load: 0.00326 [I 2022-10-10 19:11:53] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-10 19:11:53] ...optimizers/trijoint.py.151: total_norm: 39.419941 [I 2022-10-10 19:12:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 1880/2304 [I 2022-10-10 19:12:02] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:06:05 [I 2022-10-10 19:12:02] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00308 [I 2022-10-10 19:12:02] ...trap/engines/engine.py.230: loss: 0.10000 [I 2022-10-10 19:12:02] ...optimizers/trijoint.py.151: total_norm: 18.787245 [I 2022-10-10 19:12:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 1890/2304 [I 2022-10-10 19:12:10] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:58 [I 2022-10-10 19:12:10] ...trap/engines/engine.py.229: process: 0.87986 | load: 0.00306 [I 2022-10-10 19:12:10] ...trap/engines/engine.py.230: loss: 0.13068 [I 2022-10-10 19:12:10] ...optimizers/trijoint.py.151: total_norm: 17.903723 [I 2022-10-10 19:12:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 1900/2304 [I 2022-10-10 19:12:19] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:47 [I 2022-10-10 19:12:19] ...trap/engines/engine.py.229: process: 0.85723 | load: 0.00309 [I 2022-10-10 19:12:19] ...trap/engines/engine.py.230: loss: 0.16333 [I 2022-10-10 19:12:19] ...optimizers/trijoint.py.151: total_norm: 13.170272 [I 2022-10-10 19:12:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 1910/2304 [I 2022-10-10 19:12:28] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:40 [I 2022-10-10 19:12:28] ...trap/engines/engine.py.229: process: 0.88052 | load: 0.00309 [I 2022-10-10 19:12:28] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-10 19:12:28] ...optimizers/trijoint.py.151: total_norm: 25.239834 [I 2022-10-10 19:12:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 1920/2304 [I 2022-10-10 19:12:36] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:28 [I 2022-10-10 19:12:36] ...trap/engines/engine.py.229: process: 0.86041 | load: 0.00291 [I 2022-10-10 19:12:36] ...trap/engines/engine.py.230: loss: 0.10506 [I 2022-10-10 19:12:36] ...optimizers/trijoint.py.151: total_norm: 15.966700 [I 2022-10-10 19:12:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 1930/2304 [I 2022-10-10 19:12:45] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:21 [I 2022-10-10 19:12:45] ...trap/engines/engine.py.229: process: 0.86449 | load: 0.00329 [I 2022-10-10 19:12:45] ...trap/engines/engine.py.230: loss: 0.08953 [I 2022-10-10 19:12:45] ...optimizers/trijoint.py.151: total_norm: 17.627903 [I 2022-10-10 19:12:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 1940/2304 [I 2022-10-10 19:12:53] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:11 [I 2022-10-10 19:12:53] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00297 [I 2022-10-10 19:12:53] ...trap/engines/engine.py.230: loss: 0.12661 [I 2022-10-10 19:12:53] ...optimizers/trijoint.py.151: total_norm: 20.974390 [I 2022-10-10 19:13:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 1950/2304 [I 2022-10-10 19:13:02] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:05:02 [I 2022-10-10 19:13:02] ...trap/engines/engine.py.229: process: 0.86234 | load: 0.00292 [I 2022-10-10 19:13:02] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-10 19:13:02] ...optimizers/trijoint.py.151: total_norm: 20.659592 [I 2022-10-10 19:13:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 1960/2304 [I 2022-10-10 19:13:11] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:53 [I 2022-10-10 19:13:11] ...trap/engines/engine.py.229: process: 0.85312 | load: 0.00296 [I 2022-10-10 19:13:11] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-10 19:13:11] ...optimizers/trijoint.py.151: total_norm: 15.442699 [I 2022-10-10 19:13:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 1970/2304 [I 2022-10-10 19:13:19] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:44 [I 2022-10-10 19:13:19] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00298 [I 2022-10-10 19:13:19] ...trap/engines/engine.py.230: loss: 0.12313 [I 2022-10-10 19:13:19] ...optimizers/trijoint.py.151: total_norm: 14.075611 [I 2022-10-10 19:13:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 1980/2304 [I 2022-10-10 19:13:28] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:36 [I 2022-10-10 19:13:28] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00296 [I 2022-10-10 19:13:28] ...trap/engines/engine.py.230: loss: 0.14570 [I 2022-10-10 19:13:28] ...optimizers/trijoint.py.151: total_norm: 19.055191 [I 2022-10-10 19:13:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 1990/2304 [I 2022-10-10 19:13:36] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:26 [I 2022-10-10 19:13:36] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00297 [I 2022-10-10 19:13:36] ...trap/engines/engine.py.230: loss: 0.07298 [I 2022-10-10 19:13:36] ...optimizers/trijoint.py.151: total_norm: 10.609059 [I 2022-10-10 19:13:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 2000/2304 [I 2022-10-10 19:13:45] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:18 [I 2022-10-10 19:13:45] ...trap/engines/engine.py.229: process: 0.84456 | load: 0.00297 [I 2022-10-10 19:13:45] ...trap/engines/engine.py.230: loss: 0.15759 [I 2022-10-10 19:13:45] ...optimizers/trijoint.py.151: total_norm: 12.429914 [I 2022-10-10 19:13:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 2010/2304 [I 2022-10-10 19:13:53] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:12 [I 2022-10-10 19:13:53] ...trap/engines/engine.py.229: process: 0.85791 | load: 0.00297 [I 2022-10-10 19:13:53] ...trap/engines/engine.py.230: loss: 0.14623 [I 2022-10-10 19:13:53] ...optimizers/trijoint.py.151: total_norm: 13.958617 [I 2022-10-10 19:14:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 2020/2304 [I 2022-10-10 19:14:02] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:04:04 [I 2022-10-10 19:14:02] ...trap/engines/engine.py.229: process: 0.86232 | load: 0.00296 [I 2022-10-10 19:14:02] ...trap/engines/engine.py.230: loss: 0.08822 [I 2022-10-10 19:14:02] ...optimizers/trijoint.py.151: total_norm: 12.532805 [I 2022-10-10 19:14:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 2030/2304 [I 2022-10-10 19:14:11] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:53 [I 2022-10-10 19:14:11] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00298 [I 2022-10-10 19:14:11] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-10 19:14:11] ...optimizers/trijoint.py.151: total_norm: 12.577125 [I 2022-10-10 19:14:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 2040/2304 [I 2022-10-10 19:14:19] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:46 [I 2022-10-10 19:14:19] ...trap/engines/engine.py.229: process: 0.87806 | load: 0.00307 [I 2022-10-10 19:14:19] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-10 19:14:19] ...optimizers/trijoint.py.151: total_norm: 12.649011 [I 2022-10-10 19:14:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 2050/2304 [I 2022-10-10 19:14:28] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:37 [I 2022-10-10 19:14:28] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00316 [I 2022-10-10 19:14:28] ...trap/engines/engine.py.230: loss: 0.17366 [I 2022-10-10 19:14:28] ...optimizers/trijoint.py.151: total_norm: 14.841256 [I 2022-10-10 19:14:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 2060/2304 [I 2022-10-10 19:14:36] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:31 [I 2022-10-10 19:14:36] ...trap/engines/engine.py.229: process: 0.87227 | load: 0.00312 [I 2022-10-10 19:14:36] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-10 19:14:36] ...optimizers/trijoint.py.151: total_norm: 15.196445 [I 2022-10-10 19:14:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 2070/2304 [I 2022-10-10 19:14:45] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:03:22 [I 2022-10-10 19:14:45] ...trap/engines/engine.py.229: process: 0.87074 | load: 0.00324 [I 2022-10-10 19:14:45] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-10 19:14:45] ...optimizers/trijoint.py.151: total_norm: 14.020120 [I 2022-10-10 19:14:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 2080/2304 [I 2022-10-10 19:14:54] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:03:13 [I 2022-10-10 19:14:54] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00323 [I 2022-10-10 19:14:54] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-10 19:14:54] ...optimizers/trijoint.py.151: total_norm: 18.138002 [I 2022-10-10 19:15:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 2090/2304 [I 2022-10-10 19:15:02] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:03:04 [I 2022-10-10 19:15:02] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00318 [I 2022-10-10 19:15:02] ...trap/engines/engine.py.230: loss: 0.17950 [I 2022-10-10 19:15:02] ...optimizers/trijoint.py.151: total_norm: 30.483004 [I 2022-10-10 19:15:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 2100/2304 [I 2022-10-10 19:15:11] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:56 [I 2022-10-10 19:15:11] ...trap/engines/engine.py.229: process: 0.86530 | load: 0.00331 [I 2022-10-10 19:15:11] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-10 19:15:11] ...optimizers/trijoint.py.151: total_norm: 12.742635 [I 2022-10-10 19:15:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 2110/2304 [I 2022-10-10 19:15:20] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:48 [I 2022-10-10 19:15:20] ...trap/engines/engine.py.229: process: 0.87373 | load: 0.00317 [I 2022-10-10 19:15:20] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-10 19:15:20] ...optimizers/trijoint.py.151: total_norm: 12.908370 [I 2022-10-10 19:15:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 2120/2304 [I 2022-10-10 19:15:28] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:39 [I 2022-10-10 19:15:28] ...trap/engines/engine.py.229: process: 0.86755 | load: 0.00330 [I 2022-10-10 19:15:28] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-10 19:15:28] ...optimizers/trijoint.py.151: total_norm: 9.995041 [I 2022-10-10 19:15:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 2130/2304 [I 2022-10-10 19:15:37] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:29 [I 2022-10-10 19:15:37] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00311 [I 2022-10-10 19:15:37] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-10 19:15:37] ...optimizers/trijoint.py.151: total_norm: 11.852022 [I 2022-10-10 19:15:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 2140/2304 [I 2022-10-10 19:15:46] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:21 [I 2022-10-10 19:15:46] ...trap/engines/engine.py.229: process: 0.87021 | load: 0.00308 [I 2022-10-10 19:15:46] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-10 19:15:46] ...optimizers/trijoint.py.151: total_norm: 17.357258 [I 2022-10-10 19:15:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 2150/2304 [I 2022-10-10 19:15:54] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:12 [I 2022-10-10 19:15:54] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00312 [I 2022-10-10 19:15:54] ...trap/engines/engine.py.230: loss: 0.13724 [I 2022-10-10 19:15:54] ...optimizers/trijoint.py.151: total_norm: 14.427837 [I 2022-10-10 19:16:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 2160/2304 [I 2022-10-10 19:16:03] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:03 [I 2022-10-10 19:16:03] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00309 [I 2022-10-10 19:16:03] ...trap/engines/engine.py.230: loss: 0.14286 [I 2022-10-10 19:16:03] ...optimizers/trijoint.py.151: total_norm: 29.355080 [I 2022-10-10 19:16:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 2170/2304 [I 2022-10-10 19:16:12] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:55 [I 2022-10-10 19:16:12] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00309 [I 2022-10-10 19:16:12] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-10 19:16:12] ...optimizers/trijoint.py.151: total_norm: 24.432810 [I 2022-10-10 19:16:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 2180/2304 [I 2022-10-10 19:16:20] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:46 [I 2022-10-10 19:16:20] ...trap/engines/engine.py.229: process: 0.85848 | load: 0.00307 [I 2022-10-10 19:16:20] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-10 19:16:20] ...optimizers/trijoint.py.151: total_norm: 8.621886 [I 2022-10-10 19:16:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 2190/2304 [I 2022-10-10 19:16:29] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:38 [I 2022-10-10 19:16:29] ...trap/engines/engine.py.229: process: 0.86763 | load: 0.00317 [I 2022-10-10 19:16:29] ...trap/engines/engine.py.230: loss: 0.10497 [I 2022-10-10 19:16:29] ...optimizers/trijoint.py.151: total_norm: 11.751163 [I 2022-10-10 19:16:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 2200/2304 [I 2022-10-10 19:16:37] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:29 [I 2022-10-10 19:16:37] ...trap/engines/engine.py.229: process: 0.87645 | load: 0.00310 [I 2022-10-10 19:16:37] ...trap/engines/engine.py.230: loss: 0.10116 [I 2022-10-10 19:16:37] ...optimizers/trijoint.py.151: total_norm: 21.836260 [I 2022-10-10 19:16:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 2210/2304 [I 2022-10-10 19:16:46] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:20 [I 2022-10-10 19:16:46] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00310 [I 2022-10-10 19:16:46] ...trap/engines/engine.py.230: loss: 0.09641 [I 2022-10-10 19:16:46] ...optimizers/trijoint.py.151: total_norm: 9.089823 [I 2022-10-10 19:16:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 2220/2304 [I 2022-10-10 19:16:55] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:12 [I 2022-10-10 19:16:55] ...trap/engines/engine.py.229: process: 0.86574 | load: 0.00308 [I 2022-10-10 19:16:55] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-10 19:16:55] ...optimizers/trijoint.py.151: total_norm: 12.006522 [I 2022-10-10 19:17:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 2230/2304 [I 2022-10-10 19:17:03] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:03 [I 2022-10-10 19:17:03] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00306 [I 2022-10-10 19:17:03] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-10 19:17:03] ...optimizers/trijoint.py.151: total_norm: 14.285438 [I 2022-10-10 19:17:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 2240/2304 [I 2022-10-10 19:17:12] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:55 [I 2022-10-10 19:17:12] ...trap/engines/engine.py.229: process: 0.87078 | load: 0.00317 [I 2022-10-10 19:17:12] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-10 19:17:12] ...optimizers/trijoint.py.151: total_norm: 11.156937 [I 2022-10-10 19:17:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 2250/2304 [I 2022-10-10 19:17:21] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:46 [I 2022-10-10 19:17:21] ...trap/engines/engine.py.229: process: 0.86867 | load: 0.00309 [I 2022-10-10 19:17:21] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-10 19:17:21] ...optimizers/trijoint.py.151: total_norm: 16.491526 [I 2022-10-10 19:17:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 2260/2304 [I 2022-10-10 19:17:29] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:37 [I 2022-10-10 19:17:29] ...trap/engines/engine.py.229: process: 0.86743 | load: 0.00311 [I 2022-10-10 19:17:29] ...trap/engines/engine.py.230: loss: 0.17813 [I 2022-10-10 19:17:29] ...optimizers/trijoint.py.151: total_norm: 18.453817 [I 2022-10-10 19:17:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 2270/2304 [I 2022-10-10 19:17:38] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:29 [I 2022-10-10 19:17:38] ...trap/engines/engine.py.229: process: 0.86714 | load: 0.00310 [I 2022-10-10 19:17:38] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-10 19:17:38] ...optimizers/trijoint.py.151: total_norm: 10.003352 [I 2022-10-10 19:17:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 2280/2304 [I 2022-10-10 19:17:47] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:20 [I 2022-10-10 19:17:47] ...trap/engines/engine.py.229: process: 0.86641 | load: 0.00318 [I 2022-10-10 19:17:47] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-10 19:17:47] ...optimizers/trijoint.py.151: total_norm: 16.152744 [I 2022-10-10 19:17:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 2290/2304 [I 2022-10-10 19:17:55] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:12 [I 2022-10-10 19:17:55] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00333 [I 2022-10-10 19:17:55] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-10 19:17:55] ...optimizers/trijoint.py.151: total_norm: 10.913810 [I 2022-10-10 19:18:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 2300/2304 [I 2022-10-10 19:18:04] ...trap/engines/engine.py.228: elapsed: 0:33:07 | left: 0:00:03 [I 2022-10-10 19:18:04] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00018 [I 2022-10-10 19:18:04] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-10 19:18:04] ...optimizers/trijoint.py.151: total_norm: 39.472767 [I 2022-10-10 19:18:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 2304/2304 [I 2022-10-10 19:18:07] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:00 [I 2022-10-10 19:18:07] ...trap/engines/engine.py.229: process: 0.84618 | load: 0.00018 [I 2022-10-10 19:18:07] ...trap/engines/engine.py.230: loss: 0.16122 [I 2022-10-10 19:18:07] ...optimizers/trijoint.py.151: total_norm: 10.701489 [S 2022-10-10 19:18:09] ...trap/engines/engine.py.129: train_epoch.epoch: 82 [S 2022-10-10 19:18:09] ...trap/engines/engine.py.129: train_epoch.loss: 0.13241186291949092 [I 2022-10-10 19:18:09] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 19:18:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 82 [I 2022-10-10 19:18:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 19:18:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 19:18:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 19:18:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 19:18:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 19:18:16] ...trap/engines/engine.py.321: eval: epoch 82 | batch 0/510 [I 2022-10-10 19:18:16] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:49 [I 2022-10-10 19:18:16] ...trap/engines/engine.py.326: process: 5.50949 | load: 3.12995 [I 2022-10-10 19:18:22] ...trap/engines/engine.py.321: eval: epoch 82 | batch 10/510 [I 2022-10-10 19:18:22] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:35 [I 2022-10-10 19:18:22] ...trap/engines/engine.py.326: process: 0.62109 | load: 0.00020 [I 2022-10-10 19:18:28] ...trap/engines/engine.py.321: eval: epoch 82 | batch 20/510 [I 2022-10-10 19:18:28] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:15 [I 2022-10-10 19:18:28] ...trap/engines/engine.py.326: process: 0.55421 | load: 0.00020 [I 2022-10-10 19:18:35] ...trap/engines/engine.py.321: eval: epoch 82 | batch 30/510 [I 2022-10-10 19:18:35] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:15 [I 2022-10-10 19:18:35] ...trap/engines/engine.py.326: process: 0.48947 | load: 0.00021 [I 2022-10-10 19:18:41] ...trap/engines/engine.py.321: eval: epoch 82 | batch 40/510 [I 2022-10-10 19:18:41] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:31 [I 2022-10-10 19:18:41] ...trap/engines/engine.py.326: process: 0.53763 | load: 0.00019 [I 2022-10-10 19:18:45] ...trap/engines/engine.py.321: eval: epoch 82 | batch 50/510 [I 2022-10-10 19:18:45] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:34 [I 2022-10-10 19:18:45] ...trap/engines/engine.py.326: process: 0.42319 | load: 0.00017 [I 2022-10-10 19:18:50] ...trap/engines/engine.py.321: eval: epoch 82 | batch 60/510 [I 2022-10-10 19:18:50] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:25 [I 2022-10-10 19:18:50] ...trap/engines/engine.py.326: process: 0.45234 | load: 0.00018 [I 2022-10-10 19:18:55] ...trap/engines/engine.py.321: eval: epoch 82 | batch 70/510 [I 2022-10-10 19:18:55] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:40 [I 2022-10-10 19:18:55] ...trap/engines/engine.py.326: process: 0.42505 | load: 0.00018 [I 2022-10-10 19:19:00] ...trap/engines/engine.py.321: eval: epoch 82 | batch 80/510 [I 2022-10-10 19:19:00] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:55 [I 2022-10-10 19:19:00] ...trap/engines/engine.py.326: process: 0.51344 | load: 0.00021 [I 2022-10-10 19:19:05] ...trap/engines/engine.py.321: eval: epoch 82 | batch 90/510 [I 2022-10-10 19:19:05] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:03:14 [I 2022-10-10 19:19:05] ...trap/engines/engine.py.326: process: 0.44676 | load: 0.00019 [I 2022-10-10 19:19:14] ...trap/engines/engine.py.321: eval: epoch 82 | batch 100/510 [I 2022-10-10 19:19:14] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:04:00 [I 2022-10-10 19:19:14] ...trap/engines/engine.py.326: process: 0.30621 | load: 0.00018 [I 2022-10-10 19:19:17] ...trap/engines/engine.py.321: eval: epoch 82 | batch 110/510 [I 2022-10-10 19:19:17] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:18 [I 2022-10-10 19:19:17] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00020 [I 2022-10-10 19:19:21] ...trap/engines/engine.py.321: eval: epoch 82 | batch 120/510 [I 2022-10-10 19:19:21] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:04 [I 2022-10-10 19:19:21] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00019 [I 2022-10-10 19:19:24] ...trap/engines/engine.py.321: eval: epoch 82 | batch 130/510 [I 2022-10-10 19:19:24] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:00 [I 2022-10-10 19:19:24] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00019 [I 2022-10-10 19:19:27] ...trap/engines/engine.py.321: eval: epoch 82 | batch 140/510 [I 2022-10-10 19:19:27] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:57 [I 2022-10-10 19:19:27] ...trap/engines/engine.py.326: process: 0.31635 | load: 0.00020 [I 2022-10-10 19:19:30] ...trap/engines/engine.py.321: eval: epoch 82 | batch 150/510 [I 2022-10-10 19:19:30] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:53 [I 2022-10-10 19:19:30] ...trap/engines/engine.py.326: process: 0.31724 | load: 0.00019 [I 2022-10-10 19:19:33] ...trap/engines/engine.py.321: eval: epoch 82 | batch 160/510 [I 2022-10-10 19:19:33] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:50 [I 2022-10-10 19:19:33] ...trap/engines/engine.py.326: process: 0.31571 | load: 0.00018 [I 2022-10-10 19:19:37] ...trap/engines/engine.py.321: eval: epoch 82 | batch 170/510 [I 2022-10-10 19:19:37] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:47 [I 2022-10-10 19:19:37] ...trap/engines/engine.py.326: process: 0.31758 | load: 0.00018 [I 2022-10-10 19:19:40] ...trap/engines/engine.py.321: eval: epoch 82 | batch 180/510 [I 2022-10-10 19:19:40] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:44 [I 2022-10-10 19:19:40] ...trap/engines/engine.py.326: process: 0.31801 | load: 0.00019 [I 2022-10-10 19:19:43] ...trap/engines/engine.py.321: eval: epoch 82 | batch 190/510 [I 2022-10-10 19:19:43] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:41 [I 2022-10-10 19:19:43] ...trap/engines/engine.py.326: process: 0.31761 | load: 0.00019 [I 2022-10-10 19:19:46] ...trap/engines/engine.py.321: eval: epoch 82 | batch 200/510 [I 2022-10-10 19:19:46] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:38 [I 2022-10-10 19:19:46] ...trap/engines/engine.py.326: process: 0.31769 | load: 0.00019 [I 2022-10-10 19:19:49] ...trap/engines/engine.py.321: eval: epoch 82 | batch 210/510 [I 2022-10-10 19:19:49] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:34 [I 2022-10-10 19:19:49] ...trap/engines/engine.py.326: process: 0.30776 | load: 0.00019 [I 2022-10-10 19:19:52] ...trap/engines/engine.py.321: eval: epoch 82 | batch 220/510 [I 2022-10-10 19:19:52] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:32 [I 2022-10-10 19:19:52] ...trap/engines/engine.py.326: process: 0.31837 | load: 0.00018 [I 2022-10-10 19:19:56] ...trap/engines/engine.py.321: eval: epoch 82 | batch 230/510 [I 2022-10-10 19:19:56] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:28 [I 2022-10-10 19:19:56] ...trap/engines/engine.py.326: process: 0.31948 | load: 0.00019 [I 2022-10-10 19:19:59] ...trap/engines/engine.py.321: eval: epoch 82 | batch 240/510 [I 2022-10-10 19:19:59] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:26 [I 2022-10-10 19:19:59] ...trap/engines/engine.py.326: process: 0.32173 | load: 0.00019 [I 2022-10-10 19:20:02] ...trap/engines/engine.py.321: eval: epoch 82 | batch 250/510 [I 2022-10-10 19:20:02] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:23 [I 2022-10-10 19:20:02] ...trap/engines/engine.py.326: process: 0.32508 | load: 0.00019 [I 2022-10-10 19:20:05] ...trap/engines/engine.py.321: eval: epoch 82 | batch 260/510 [I 2022-10-10 19:20:05] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:19 [I 2022-10-10 19:20:05] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00020 [I 2022-10-10 19:20:08] ...trap/engines/engine.py.321: eval: epoch 82 | batch 270/510 [I 2022-10-10 19:20:08] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:16 [I 2022-10-10 19:20:08] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00019 [I 2022-10-10 19:20:12] ...trap/engines/engine.py.321: eval: epoch 82 | batch 280/510 [I 2022-10-10 19:20:12] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:13 [I 2022-10-10 19:20:12] ...trap/engines/engine.py.326: process: 0.31417 | load: 0.00020 [I 2022-10-10 19:20:15] ...trap/engines/engine.py.321: eval: epoch 82 | batch 290/510 [I 2022-10-10 19:20:15] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:10 [I 2022-10-10 19:20:15] ...trap/engines/engine.py.326: process: 0.32236 | load: 0.00019 [I 2022-10-10 19:20:18] ...trap/engines/engine.py.321: eval: epoch 82 | batch 300/510 [I 2022-10-10 19:20:18] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:07 [I 2022-10-10 19:20:18] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00021 [I 2022-10-10 19:20:21] ...trap/engines/engine.py.321: eval: epoch 82 | batch 310/510 [I 2022-10-10 19:20:21] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-10 19:20:21] ...trap/engines/engine.py.326: process: 0.31879 | load: 0.00019 [I 2022-10-10 19:20:24] ...trap/engines/engine.py.321: eval: epoch 82 | batch 320/510 [I 2022-10-10 19:20:24] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:00 [I 2022-10-10 19:20:24] ...trap/engines/engine.py.326: process: 0.31238 | load: 0.00021 [I 2022-10-10 19:20:28] ...trap/engines/engine.py.321: eval: epoch 82 | batch 330/510 [I 2022-10-10 19:20:28] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:57 [I 2022-10-10 19:20:28] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00019 [I 2022-10-10 19:20:31] ...trap/engines/engine.py.321: eval: epoch 82 | batch 340/510 [I 2022-10-10 19:20:31] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:54 [I 2022-10-10 19:20:31] ...trap/engines/engine.py.326: process: 0.32264 | load: 0.00020 [I 2022-10-10 19:20:34] ...trap/engines/engine.py.321: eval: epoch 82 | batch 350/510 [I 2022-10-10 19:20:34] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:51 [I 2022-10-10 19:20:34] ...trap/engines/engine.py.326: process: 0.32251 | load: 0.00027 [I 2022-10-10 19:20:37] ...trap/engines/engine.py.321: eval: epoch 82 | batch 360/510 [I 2022-10-10 19:20:37] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-10 19:20:37] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00021 [I 2022-10-10 19:20:41] ...trap/engines/engine.py.321: eval: epoch 82 | batch 370/510 [I 2022-10-10 19:20:41] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:44 [I 2022-10-10 19:20:41] ...trap/engines/engine.py.326: process: 0.32277 | load: 0.00021 [I 2022-10-10 19:20:44] ...trap/engines/engine.py.321: eval: epoch 82 | batch 380/510 [I 2022-10-10 19:20:44] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:41 [I 2022-10-10 19:20:44] ...trap/engines/engine.py.326: process: 0.32546 | load: 0.00021 [I 2022-10-10 19:20:47] ...trap/engines/engine.py.321: eval: epoch 82 | batch 390/510 [I 2022-10-10 19:20:47] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:38 [I 2022-10-10 19:20:47] ...trap/engines/engine.py.326: process: 0.32357 | load: 0.00020 [I 2022-10-10 19:20:50] ...trap/engines/engine.py.321: eval: epoch 82 | batch 400/510 [I 2022-10-10 19:20:50] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-10 19:20:50] ...trap/engines/engine.py.326: process: 0.32214 | load: 0.00021 [I 2022-10-10 19:20:53] ...trap/engines/engine.py.321: eval: epoch 82 | batch 410/510 [I 2022-10-10 19:20:53] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-10 19:20:53] ...trap/engines/engine.py.326: process: 0.32041 | load: 0.00021 [I 2022-10-10 19:20:57] ...trap/engines/engine.py.321: eval: epoch 82 | batch 420/510 [I 2022-10-10 19:20:57] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:28 [I 2022-10-10 19:20:57] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00024 [I 2022-10-10 19:21:00] ...trap/engines/engine.py.321: eval: epoch 82 | batch 430/510 [I 2022-10-10 19:21:00] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:25 [I 2022-10-10 19:21:00] ...trap/engines/engine.py.326: process: 0.32143 | load: 0.00024 [I 2022-10-10 19:21:03] ...trap/engines/engine.py.321: eval: epoch 82 | batch 440/510 [I 2022-10-10 19:21:03] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:22 [I 2022-10-10 19:21:03] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00021 [I 2022-10-10 19:21:06] ...trap/engines/engine.py.321: eval: epoch 82 | batch 450/510 [I 2022-10-10 19:21:06] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-10 19:21:06] ...trap/engines/engine.py.326: process: 0.32871 | load: 0.00023 [I 2022-10-10 19:21:10] ...trap/engines/engine.py.321: eval: epoch 82 | batch 460/510 [I 2022-10-10 19:21:10] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:16 [I 2022-10-10 19:21:10] ...trap/engines/engine.py.326: process: 0.32241 | load: 0.00021 [I 2022-10-10 19:21:13] ...trap/engines/engine.py.321: eval: epoch 82 | batch 470/510 [I 2022-10-10 19:21:13] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-10 19:21:13] ...trap/engines/engine.py.326: process: 0.32453 | load: 0.00020 [I 2022-10-10 19:21:16] ...trap/engines/engine.py.321: eval: epoch 82 | batch 480/510 [I 2022-10-10 19:21:16] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:09 [I 2022-10-10 19:21:16] ...trap/engines/engine.py.326: process: 0.32798 | load: 0.00021 [I 2022-10-10 19:21:19] ...trap/engines/engine.py.321: eval: epoch 82 | batch 490/510 [I 2022-10-10 19:21:19] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-10 19:21:19] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00021 [I 2022-10-10 19:21:23] ...trap/engines/engine.py.321: eval: epoch 82 | batch 500/510 [I 2022-10-10 19:21:23] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-10 19:21:23] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00022 [I 2022-10-10 19:21:26] ...trap/engines/engine.py.321: eval: epoch 82 | batch 510/510 [I 2022-10-10 19:21:26] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:00 [I 2022-10-10 19:21:26] ...trap/engines/engine.py.326: process: 0.31967 | load: 0.00016 [S 2022-10-10 19:21:27] ...trap/engines/engine.py.132: eval_epoch.epoch: 82 [I 2022-10-10 19:21:27] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736500 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905600 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934600 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727000 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904600 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934200 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014678 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008273 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006974 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015310 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006468 [S 2022-10-10 19:21:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006384 [I 2022-10-10 19:22:08] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 19:22:08] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 19:22:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 19:22:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 19:22:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 19:22:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 19:22:16] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 19:22:20] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 19:22:20] ...trap/engines/engine.py.163: Training model on trainset for epoch 83 [I 2022-10-10 19:22:20] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 19:22:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 0/2304 [I 2022-10-10 19:22:23] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:07:34 [I 2022-10-10 19:22:23] ...trap/engines/engine.py.229: process: 3.32238 | load: 2.38564 [I 2022-10-10 19:22:23] ...trap/engines/engine.py.230: loss: 0.17047 [I 2022-10-10 19:22:23] ...optimizers/trijoint.py.151: total_norm: 12.440160 [I 2022-10-10 19:22:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 10/2304 [I 2022-10-10 19:22:32] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:43:14 [I 2022-10-10 19:22:32] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00602 [I 2022-10-10 19:22:32] ...trap/engines/engine.py.230: loss: 0.09888 [I 2022-10-10 19:22:32] ...optimizers/trijoint.py.151: total_norm: 9.176371 [I 2022-10-10 19:22:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 20/2304 [I 2022-10-10 19:22:41] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:04 [I 2022-10-10 19:22:41] ...trap/engines/engine.py.229: process: 0.87110 | load: 0.00456 [I 2022-10-10 19:22:41] ...trap/engines/engine.py.230: loss: 0.23852 [I 2022-10-10 19:22:41] ...optimizers/trijoint.py.151: total_norm: 25.856297 [I 2022-10-10 19:22:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 30/2304 [I 2022-10-10 19:22:49] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:44 [I 2022-10-10 19:22:49] ...trap/engines/engine.py.229: process: 0.86459 | load: 0.00470 [I 2022-10-10 19:22:49] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-10 19:22:49] ...optimizers/trijoint.py.151: total_norm: 10.639938 [I 2022-10-10 19:22:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 40/2304 [I 2022-10-10 19:22:58] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:36 [I 2022-10-10 19:22:58] ...trap/engines/engine.py.229: process: 0.85160 | load: 0.00423 [I 2022-10-10 19:22:58] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-10 19:22:58] ...optimizers/trijoint.py.151: total_norm: 23.340309 [I 2022-10-10 19:23:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 50/2304 [I 2022-10-10 19:23:07] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:31 [I 2022-10-10 19:23:07] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00425 [I 2022-10-10 19:23:07] ...trap/engines/engine.py.230: loss: 0.11268 [I 2022-10-10 19:23:07] ...optimizers/trijoint.py.151: total_norm: 11.729144 [I 2022-10-10 19:23:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 60/2304 [I 2022-10-10 19:23:15] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:19 [I 2022-10-10 19:23:15] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00359 [I 2022-10-10 19:23:15] ...trap/engines/engine.py.230: loss: 0.11445 [I 2022-10-10 19:23:15] ...optimizers/trijoint.py.151: total_norm: 10.884630 [I 2022-10-10 19:23:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 70/2304 [I 2022-10-10 19:23:24] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:10 [I 2022-10-10 19:23:24] ...trap/engines/engine.py.229: process: 0.87048 | load: 0.00441 [I 2022-10-10 19:23:24] ...trap/engines/engine.py.230: loss: 0.15328 [I 2022-10-10 19:23:24] ...optimizers/trijoint.py.151: total_norm: 83.031044 [I 2022-10-10 19:23:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 80/2304 [I 2022-10-10 19:23:33] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:11 [I 2022-10-10 19:23:33] ...trap/engines/engine.py.229: process: 0.87544 | load: 0.00392 [I 2022-10-10 19:23:33] ...trap/engines/engine.py.230: loss: 0.13856 [I 2022-10-10 19:23:33] ...optimizers/trijoint.py.151: total_norm: 21.480068 [I 2022-10-10 19:23:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 90/2304 [I 2022-10-10 19:23:42] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:58 [I 2022-10-10 19:23:42] ...trap/engines/engine.py.229: process: 0.86986 | load: 0.00354 [I 2022-10-10 19:23:42] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-10 19:23:42] ...optimizers/trijoint.py.151: total_norm: 8.056572 [I 2022-10-10 19:23:50] ...trap/engines/engine.py.224: train: epoch 83 | batch 100/2304 [I 2022-10-10 19:23:50] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:32:07 [I 2022-10-10 19:23:50] ...trap/engines/engine.py.229: process: 0.87969 | load: 0.00390 [I 2022-10-10 19:23:50] ...trap/engines/engine.py.230: loss: 0.08961 [I 2022-10-10 19:23:50] ...optimizers/trijoint.py.151: total_norm: 76.943100 [I 2022-10-10 19:23:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 110/2304 [I 2022-10-10 19:23:59] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:57 [I 2022-10-10 19:23:59] ...trap/engines/engine.py.229: process: 0.88002 | load: 0.00343 [I 2022-10-10 19:23:59] ...trap/engines/engine.py.230: loss: 0.08905 [I 2022-10-10 19:23:59] ...optimizers/trijoint.py.151: total_norm: 12.469656 [I 2022-10-10 19:24:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 120/2304 [I 2022-10-10 19:24:08] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:35 [I 2022-10-10 19:24:08] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00323 [I 2022-10-10 19:24:08] ...trap/engines/engine.py.230: loss: 0.08583 [I 2022-10-10 19:24:08] ...optimizers/trijoint.py.151: total_norm: 7.191412 [I 2022-10-10 19:24:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 130/2304 [I 2022-10-10 19:24:16] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:32 [I 2022-10-10 19:24:16] ...trap/engines/engine.py.229: process: 0.86281 | load: 0.00308 [I 2022-10-10 19:24:16] ...trap/engines/engine.py.230: loss: 0.11420 [I 2022-10-10 19:24:16] ...optimizers/trijoint.py.151: total_norm: 18.494253 [I 2022-10-10 19:24:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 140/2304 [I 2022-10-10 19:24:25] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:12 [I 2022-10-10 19:24:25] ...trap/engines/engine.py.229: process: 0.87526 | load: 0.00318 [I 2022-10-10 19:24:25] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-10 19:24:25] ...optimizers/trijoint.py.151: total_norm: 14.702381 [I 2022-10-10 19:24:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 150/2304 [I 2022-10-10 19:24:34] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:59 [I 2022-10-10 19:24:34] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00305 [I 2022-10-10 19:24:34] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-10 19:24:34] ...optimizers/trijoint.py.151: total_norm: 10.410996 [I 2022-10-10 19:24:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 160/2304 [I 2022-10-10 19:24:42] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:30:31 [I 2022-10-10 19:24:42] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00339 [I 2022-10-10 19:24:42] ...trap/engines/engine.py.230: loss: 0.12690 [I 2022-10-10 19:24:42] ...optimizers/trijoint.py.151: total_norm: 10.482541 [I 2022-10-10 19:24:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 170/2304 [I 2022-10-10 19:24:51] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:24 [I 2022-10-10 19:24:51] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00304 [I 2022-10-10 19:24:51] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-10 19:24:51] ...optimizers/trijoint.py.151: total_norm: 12.891546 [I 2022-10-10 19:24:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 180/2304 [I 2022-10-10 19:24:59] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:12 [I 2022-10-10 19:24:59] ...trap/engines/engine.py.229: process: 0.86157 | load: 0.00295 [I 2022-10-10 19:24:59] ...trap/engines/engine.py.230: loss: 0.20599 [I 2022-10-10 19:24:59] ...optimizers/trijoint.py.151: total_norm: 14.881699 [I 2022-10-10 19:25:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 190/2304 [I 2022-10-10 19:25:08] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:05 [I 2022-10-10 19:25:08] ...trap/engines/engine.py.229: process: 0.85639 | load: 0.00309 [I 2022-10-10 19:25:08] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-10 19:25:08] ...optimizers/trijoint.py.151: total_norm: 18.416805 [I 2022-10-10 19:25:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 200/2304 [I 2022-10-10 19:25:17] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:07 [I 2022-10-10 19:25:17] ...trap/engines/engine.py.229: process: 0.85959 | load: 0.00316 [I 2022-10-10 19:25:17] ...trap/engines/engine.py.230: loss: 0.13886 [I 2022-10-10 19:25:17] ...optimizers/trijoint.py.151: total_norm: 11.620163 [I 2022-10-10 19:25:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 210/2304 [I 2022-10-10 19:25:25] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:29:47 [I 2022-10-10 19:25:25] ...trap/engines/engine.py.229: process: 0.85055 | load: 0.00304 [I 2022-10-10 19:25:25] ...trap/engines/engine.py.230: loss: 0.18745 [I 2022-10-10 19:25:25] ...optimizers/trijoint.py.151: total_norm: 16.489239 [I 2022-10-10 19:25:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 220/2304 [I 2022-10-10 19:25:34] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:44 [I 2022-10-10 19:25:34] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00318 [I 2022-10-10 19:25:34] ...trap/engines/engine.py.230: loss: 0.20570 [I 2022-10-10 19:25:34] ...optimizers/trijoint.py.151: total_norm: 9.793192 [I 2022-10-10 19:25:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 230/2304 [I 2022-10-10 19:25:42] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:24 [I 2022-10-10 19:25:42] ...trap/engines/engine.py.229: process: 0.84574 | load: 0.00303 [I 2022-10-10 19:25:42] ...trap/engines/engine.py.230: loss: 0.08953 [I 2022-10-10 19:25:42] ...optimizers/trijoint.py.151: total_norm: 7.515073 [I 2022-10-10 19:25:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 240/2304 [I 2022-10-10 19:25:51] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:12 [I 2022-10-10 19:25:51] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00308 [I 2022-10-10 19:25:51] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-10 19:25:51] ...optimizers/trijoint.py.151: total_norm: 31.253683 [I 2022-10-10 19:25:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 250/2304 [I 2022-10-10 19:25:59] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:00 [I 2022-10-10 19:25:59] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00293 [I 2022-10-10 19:25:59] ...trap/engines/engine.py.230: loss: 0.16134 [I 2022-10-10 19:25:59] ...optimizers/trijoint.py.151: total_norm: 13.812986 [I 2022-10-10 19:26:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 260/2304 [I 2022-10-10 19:26:08] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:28:57 [I 2022-10-10 19:26:08] ...trap/engines/engine.py.229: process: 0.83549 | load: 0.00297 [I 2022-10-10 19:26:08] ...trap/engines/engine.py.230: loss: 0.19580 [I 2022-10-10 19:26:08] ...optimizers/trijoint.py.151: total_norm: 12.587137 [I 2022-10-10 19:26:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 270/2304 [I 2022-10-10 19:26:16] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:28:51 [I 2022-10-10 19:26:16] ...trap/engines/engine.py.229: process: 0.85481 | load: 0.00299 [I 2022-10-10 19:26:16] ...trap/engines/engine.py.230: loss: 0.11782 [I 2022-10-10 19:26:16] ...optimizers/trijoint.py.151: total_norm: 12.480798 [I 2022-10-10 19:26:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 280/2304 [I 2022-10-10 19:26:25] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:42 [I 2022-10-10 19:26:25] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00367 [I 2022-10-10 19:26:25] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-10 19:26:25] ...optimizers/trijoint.py.151: total_norm: 26.845442 [I 2022-10-10 19:26:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 290/2304 [I 2022-10-10 19:26:34] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:29:01 [I 2022-10-10 19:26:34] ...trap/engines/engine.py.229: process: 0.87295 | load: 0.00320 [I 2022-10-10 19:26:34] ...trap/engines/engine.py.230: loss: 0.15630 [I 2022-10-10 19:26:34] ...optimizers/trijoint.py.151: total_norm: 28.825331 [I 2022-10-10 19:26:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 300/2304 [I 2022-10-10 19:26:42] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:47 [I 2022-10-10 19:26:42] ...trap/engines/engine.py.229: process: 0.86676 | load: 0.00310 [I 2022-10-10 19:26:42] ...trap/engines/engine.py.230: loss: 0.14937 [I 2022-10-10 19:26:42] ...optimizers/trijoint.py.151: total_norm: 8.550836 [I 2022-10-10 19:26:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 310/2304 [I 2022-10-10 19:26:51] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:38 [I 2022-10-10 19:26:51] ...trap/engines/engine.py.229: process: 0.87092 | load: 0.00304 [I 2022-10-10 19:26:51] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-10 19:26:51] ...optimizers/trijoint.py.151: total_norm: 19.233778 [I 2022-10-10 19:27:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 320/2304 [I 2022-10-10 19:27:00] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:38 [I 2022-10-10 19:27:00] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00308 [I 2022-10-10 19:27:00] ...trap/engines/engine.py.230: loss: 0.16344 [I 2022-10-10 19:27:00] ...optimizers/trijoint.py.151: total_norm: 65.430336 [I 2022-10-10 19:27:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 330/2304 [I 2022-10-10 19:27:08] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:28 [I 2022-10-10 19:27:08] ...trap/engines/engine.py.229: process: 0.86680 | load: 0.00320 [I 2022-10-10 19:27:08] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-10 19:27:08] ...optimizers/trijoint.py.151: total_norm: 20.582367 [I 2022-10-10 19:27:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 340/2304 [I 2022-10-10 19:27:17] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:28:14 [I 2022-10-10 19:27:17] ...trap/engines/engine.py.229: process: 0.85827 | load: 0.00326 [I 2022-10-10 19:27:17] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-10 19:27:17] ...optimizers/trijoint.py.151: total_norm: 17.409578 [I 2022-10-10 19:27:26] ...trap/engines/engine.py.224: train: epoch 83 | batch 350/2304 [I 2022-10-10 19:27:26] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:10 [I 2022-10-10 19:27:26] ...trap/engines/engine.py.229: process: 0.87312 | load: 0.00328 [I 2022-10-10 19:27:26] ...trap/engines/engine.py.230: loss: 0.10719 [I 2022-10-10 19:27:26] ...optimizers/trijoint.py.151: total_norm: 9.723527 [I 2022-10-10 19:27:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 360/2304 [I 2022-10-10 19:27:34] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:57 [I 2022-10-10 19:27:34] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00328 [I 2022-10-10 19:27:34] ...trap/engines/engine.py.230: loss: 0.15087 [I 2022-10-10 19:27:34] ...optimizers/trijoint.py.151: total_norm: 85.780937 [I 2022-10-10 19:27:43] ...trap/engines/engine.py.224: train: epoch 83 | batch 370/2304 [I 2022-10-10 19:27:43] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:53 [I 2022-10-10 19:27:43] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00311 [I 2022-10-10 19:27:43] ...trap/engines/engine.py.230: loss: 0.11712 [I 2022-10-10 19:27:43] ...optimizers/trijoint.py.151: total_norm: 12.720482 [I 2022-10-10 19:27:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 380/2304 [I 2022-10-10 19:27:52] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:50 [I 2022-10-10 19:27:52] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00315 [I 2022-10-10 19:27:52] ...trap/engines/engine.py.230: loss: 0.15162 [I 2022-10-10 19:27:52] ...optimizers/trijoint.py.151: total_norm: 24.218861 [I 2022-10-10 19:28:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 390/2304 [I 2022-10-10 19:28:00] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:27 [I 2022-10-10 19:28:00] ...trap/engines/engine.py.229: process: 0.86022 | load: 0.00310 [I 2022-10-10 19:28:00] ...trap/engines/engine.py.230: loss: 0.10253 [I 2022-10-10 19:28:00] ...optimizers/trijoint.py.151: total_norm: 16.788420 [I 2022-10-10 19:28:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 400/2304 [I 2022-10-10 19:28:09] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:17 [I 2022-10-10 19:28:09] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00300 [I 2022-10-10 19:28:09] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-10 19:28:09] ...optimizers/trijoint.py.151: total_norm: 20.575701 [I 2022-10-10 19:28:18] ...trap/engines/engine.py.224: train: epoch 83 | batch 410/2304 [I 2022-10-10 19:28:18] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:59 [I 2022-10-10 19:28:18] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00325 [I 2022-10-10 19:28:18] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-10 19:28:18] ...optimizers/trijoint.py.151: total_norm: 27.832237 [I 2022-10-10 19:28:26] ...trap/engines/engine.py.224: train: epoch 83 | batch 420/2304 [I 2022-10-10 19:28:26] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:27:03 [I 2022-10-10 19:28:26] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00307 [I 2022-10-10 19:28:26] ...trap/engines/engine.py.230: loss: 0.15793 [I 2022-10-10 19:28:26] ...optimizers/trijoint.py.151: total_norm: 21.390543 [I 2022-10-10 19:28:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 430/2304 [I 2022-10-10 19:28:35] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:27:05 [I 2022-10-10 19:28:35] ...trap/engines/engine.py.229: process: 0.86935 | load: 0.00311 [I 2022-10-10 19:28:35] ...trap/engines/engine.py.230: loss: 0.11593 [I 2022-10-10 19:28:35] ...optimizers/trijoint.py.151: total_norm: 10.784368 [I 2022-10-10 19:28:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 440/2304 [I 2022-10-10 19:28:44] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:55 [I 2022-10-10 19:28:44] ...trap/engines/engine.py.229: process: 0.86618 | load: 0.00320 [I 2022-10-10 19:28:44] ...trap/engines/engine.py.230: loss: 0.09082 [I 2022-10-10 19:28:44] ...optimizers/trijoint.py.151: total_norm: 10.456194 [I 2022-10-10 19:28:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 450/2304 [I 2022-10-10 19:28:52] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:32 [I 2022-10-10 19:28:52] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00327 [I 2022-10-10 19:28:52] ...trap/engines/engine.py.230: loss: 0.08954 [I 2022-10-10 19:28:52] ...optimizers/trijoint.py.151: total_norm: 15.089790 [I 2022-10-10 19:29:01] ...trap/engines/engine.py.224: train: epoch 83 | batch 460/2304 [I 2022-10-10 19:29:01] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:37 [I 2022-10-10 19:29:01] ...trap/engines/engine.py.229: process: 0.87935 | load: 0.00307 [I 2022-10-10 19:29:01] ...trap/engines/engine.py.230: loss: 0.11280 [I 2022-10-10 19:29:01] ...optimizers/trijoint.py.151: total_norm: 15.768482 [I 2022-10-10 19:29:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 470/2304 [I 2022-10-10 19:29:09] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:21 [I 2022-10-10 19:29:09] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00312 [I 2022-10-10 19:29:09] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-10 19:29:09] ...optimizers/trijoint.py.151: total_norm: 10.609485 [I 2022-10-10 19:29:18] ...trap/engines/engine.py.224: train: epoch 83 | batch 480/2304 [I 2022-10-10 19:29:18] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:20 [I 2022-10-10 19:29:18] ...trap/engines/engine.py.229: process: 0.86849 | load: 0.00308 [I 2022-10-10 19:29:18] ...trap/engines/engine.py.230: loss: 0.08448 [I 2022-10-10 19:29:18] ...optimizers/trijoint.py.151: total_norm: 13.851784 [I 2022-10-10 19:29:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 490/2304 [I 2022-10-10 19:29:27] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:26:14 [I 2022-10-10 19:29:27] ...trap/engines/engine.py.229: process: 0.88257 | load: 0.00312 [I 2022-10-10 19:29:27] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-10 19:29:27] ...optimizers/trijoint.py.151: total_norm: 9.522303 [I 2022-10-10 19:29:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 500/2304 [I 2022-10-10 19:29:36] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:26:07 [I 2022-10-10 19:29:36] ...trap/engines/engine.py.229: process: 0.86906 | load: 0.00334 [I 2022-10-10 19:29:36] ...trap/engines/engine.py.230: loss: 0.15992 [I 2022-10-10 19:29:36] ...optimizers/trijoint.py.151: total_norm: 21.050093 [I 2022-10-10 19:29:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 510/2304 [I 2022-10-10 19:29:44] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:54 [I 2022-10-10 19:29:44] ...trap/engines/engine.py.229: process: 0.87991 | load: 0.00310 [I 2022-10-10 19:29:44] ...trap/engines/engine.py.230: loss: 0.19929 [I 2022-10-10 19:29:44] ...optimizers/trijoint.py.151: total_norm: 21.946018 [I 2022-10-10 19:29:53] ...trap/engines/engine.py.224: train: epoch 83 | batch 520/2304 [I 2022-10-10 19:29:53] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:44 [I 2022-10-10 19:29:53] ...trap/engines/engine.py.229: process: 0.85973 | load: 0.00310 [I 2022-10-10 19:29:53] ...trap/engines/engine.py.230: loss: 0.20612 [I 2022-10-10 19:29:53] ...optimizers/trijoint.py.151: total_norm: 64.516922 [I 2022-10-10 19:30:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 530/2304 [I 2022-10-10 19:30:02] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:25:43 [I 2022-10-10 19:30:02] ...trap/engines/engine.py.229: process: 0.88705 | load: 0.00324 [I 2022-10-10 19:30:02] ...trap/engines/engine.py.230: loss: 0.09993 [I 2022-10-10 19:30:02] ...optimizers/trijoint.py.151: total_norm: 10.050312 [I 2022-10-10 19:30:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 540/2304 [I 2022-10-10 19:30:10] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:25:32 [I 2022-10-10 19:30:10] ...trap/engines/engine.py.229: process: 0.87295 | load: 0.00321 [I 2022-10-10 19:30:10] ...trap/engines/engine.py.230: loss: 0.11357 [I 2022-10-10 19:30:10] ...optimizers/trijoint.py.151: total_norm: 18.940645 [I 2022-10-10 19:30:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 550/2304 [I 2022-10-10 19:30:19] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:19 [I 2022-10-10 19:30:19] ...trap/engines/engine.py.229: process: 0.86511 | load: 0.00312 [I 2022-10-10 19:30:19] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-10 19:30:19] ...optimizers/trijoint.py.151: total_norm: 11.929358 [I 2022-10-10 19:30:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 560/2304 [I 2022-10-10 19:30:28] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:59 [I 2022-10-10 19:30:28] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00301 [I 2022-10-10 19:30:28] ...trap/engines/engine.py.230: loss: 0.17959 [I 2022-10-10 19:30:28] ...optimizers/trijoint.py.151: total_norm: 10.735601 [I 2022-10-10 19:30:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 570/2304 [I 2022-10-10 19:30:36] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:41 [I 2022-10-10 19:30:36] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00301 [I 2022-10-10 19:30:36] ...trap/engines/engine.py.230: loss: 0.18510 [I 2022-10-10 19:30:36] ...optimizers/trijoint.py.151: total_norm: 12.378113 [I 2022-10-10 19:30:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 580/2304 [I 2022-10-10 19:30:45] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:44 [I 2022-10-10 19:30:45] ...trap/engines/engine.py.229: process: 0.85076 | load: 0.00296 [I 2022-10-10 19:30:45] ...trap/engines/engine.py.230: loss: 0.10153 [I 2022-10-10 19:30:45] ...optimizers/trijoint.py.151: total_norm: 10.876023 [I 2022-10-10 19:30:53] ...trap/engines/engine.py.224: train: epoch 83 | batch 590/2304 [I 2022-10-10 19:30:53] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:27 [I 2022-10-10 19:30:53] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00315 [I 2022-10-10 19:30:53] ...trap/engines/engine.py.230: loss: 0.10163 [I 2022-10-10 19:30:53] ...optimizers/trijoint.py.151: total_norm: 14.677883 [I 2022-10-10 19:31:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 600/2304 [I 2022-10-10 19:31:02] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:20 [I 2022-10-10 19:31:02] ...trap/engines/engine.py.229: process: 0.86631 | load: 0.00304 [I 2022-10-10 19:31:02] ...trap/engines/engine.py.230: loss: 0.13292 [I 2022-10-10 19:31:02] ...optimizers/trijoint.py.151: total_norm: 13.282740 [I 2022-10-10 19:31:11] ...trap/engines/engine.py.224: train: epoch 83 | batch 610/2304 [I 2022-10-10 19:31:11] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:06 [I 2022-10-10 19:31:11] ...trap/engines/engine.py.229: process: 0.85873 | load: 0.00304 [I 2022-10-10 19:31:11] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-10 19:31:11] ...optimizers/trijoint.py.151: total_norm: 8.742577 [I 2022-10-10 19:31:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 620/2304 [I 2022-10-10 19:31:19] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:00 [I 2022-10-10 19:31:19] ...trap/engines/engine.py.229: process: 0.85207 | load: 0.00294 [I 2022-10-10 19:31:19] ...trap/engines/engine.py.230: loss: 0.09374 [I 2022-10-10 19:31:19] ...optimizers/trijoint.py.151: total_norm: 11.444997 [I 2022-10-10 19:31:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 630/2304 [I 2022-10-10 19:31:28] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:46 [I 2022-10-10 19:31:28] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00297 [I 2022-10-10 19:31:28] ...trap/engines/engine.py.230: loss: 0.16766 [I 2022-10-10 19:31:28] ...optimizers/trijoint.py.151: total_norm: 10.226500 [I 2022-10-10 19:31:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 640/2304 [I 2022-10-10 19:31:36] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:45 [I 2022-10-10 19:31:36] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00306 [I 2022-10-10 19:31:36] ...trap/engines/engine.py.230: loss: 0.13981 [I 2022-10-10 19:31:36] ...optimizers/trijoint.py.151: total_norm: 15.980213 [I 2022-10-10 19:31:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 650/2304 [I 2022-10-10 19:31:45] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:39 [I 2022-10-10 19:31:45] ...trap/engines/engine.py.229: process: 0.85428 | load: 0.00306 [I 2022-10-10 19:31:45] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-10 19:31:45] ...optimizers/trijoint.py.151: total_norm: 6.745142 [I 2022-10-10 19:31:53] ...trap/engines/engine.py.224: train: epoch 83 | batch 660/2304 [I 2022-10-10 19:31:53] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:26 [I 2022-10-10 19:31:53] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00312 [I 2022-10-10 19:31:53] ...trap/engines/engine.py.230: loss: 0.10225 [I 2022-10-10 19:31:53] ...optimizers/trijoint.py.151: total_norm: 13.600456 [I 2022-10-10 19:32:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 670/2304 [I 2022-10-10 19:32:02] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:19 [I 2022-10-10 19:32:02] ...trap/engines/engine.py.229: process: 0.85967 | load: 0.00295 [I 2022-10-10 19:32:02] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-10 19:32:02] ...optimizers/trijoint.py.151: total_norm: 12.018746 [I 2022-10-10 19:32:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 680/2304 [I 2022-10-10 19:32:10] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:00 [I 2022-10-10 19:32:10] ...trap/engines/engine.py.229: process: 0.85672 | load: 0.00296 [I 2022-10-10 19:32:10] ...trap/engines/engine.py.230: loss: 0.10974 [I 2022-10-10 19:32:11] ...optimizers/trijoint.py.151: total_norm: 8.751505 [I 2022-10-10 19:32:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 690/2304 [I 2022-10-10 19:32:19] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:54 [I 2022-10-10 19:32:19] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00297 [I 2022-10-10 19:32:19] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-10 19:32:19] ...optimizers/trijoint.py.151: total_norm: 17.421387 [I 2022-10-10 19:32:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 700/2304 [I 2022-10-10 19:32:28] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:52 [I 2022-10-10 19:32:28] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00294 [I 2022-10-10 19:32:28] ...trap/engines/engine.py.230: loss: 0.09511 [I 2022-10-10 19:32:28] ...optimizers/trijoint.py.151: total_norm: 10.410115 [I 2022-10-10 19:32:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 710/2304 [I 2022-10-10 19:32:36] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:50 [I 2022-10-10 19:32:36] ...trap/engines/engine.py.229: process: 0.85977 | load: 0.00311 [I 2022-10-10 19:32:36] ...trap/engines/engine.py.230: loss: 0.08418 [I 2022-10-10 19:32:36] ...optimizers/trijoint.py.151: total_norm: 12.244354 [I 2022-10-10 19:32:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 720/2304 [I 2022-10-10 19:32:45] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:54 [I 2022-10-10 19:32:45] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00313 [I 2022-10-10 19:32:45] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-10 19:32:45] ...optimizers/trijoint.py.151: total_norm: 13.057689 [I 2022-10-10 19:32:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 730/2304 [I 2022-10-10 19:32:54] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:36 [I 2022-10-10 19:32:54] ...trap/engines/engine.py.229: process: 0.86047 | load: 0.00320 [I 2022-10-10 19:32:54] ...trap/engines/engine.py.230: loss: 0.21305 [I 2022-10-10 19:32:54] ...optimizers/trijoint.py.151: total_norm: 19.662174 [I 2022-10-10 19:33:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 740/2304 [I 2022-10-10 19:33:02] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:40 [I 2022-10-10 19:33:02] ...trap/engines/engine.py.229: process: 0.88379 | load: 0.00312 [I 2022-10-10 19:33:02] ...trap/engines/engine.py.230: loss: 0.13939 [I 2022-10-10 19:33:02] ...optimizers/trijoint.py.151: total_norm: 8.865578 [I 2022-10-10 19:33:11] ...trap/engines/engine.py.224: train: epoch 83 | batch 750/2304 [I 2022-10-10 19:33:11] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:22 [I 2022-10-10 19:33:11] ...trap/engines/engine.py.229: process: 0.86436 | load: 0.00347 [I 2022-10-10 19:33:11] ...trap/engines/engine.py.230: loss: 0.09858 [I 2022-10-10 19:33:11] ...optimizers/trijoint.py.151: total_norm: 11.551431 [I 2022-10-10 19:33:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 760/2304 [I 2022-10-10 19:33:20] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:20 [I 2022-10-10 19:33:20] ...trap/engines/engine.py.229: process: 0.86036 | load: 0.00312 [I 2022-10-10 19:33:20] ...trap/engines/engine.py.230: loss: 0.12391 [I 2022-10-10 19:33:20] ...optimizers/trijoint.py.151: total_norm: 28.928711 [I 2022-10-10 19:33:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 770/2304 [I 2022-10-10 19:33:28] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:22:11 [I 2022-10-10 19:33:28] ...trap/engines/engine.py.229: process: 0.87683 | load: 0.00320 [I 2022-10-10 19:33:28] ...trap/engines/engine.py.230: loss: 0.10941 [I 2022-10-10 19:33:28] ...optimizers/trijoint.py.151: total_norm: 11.751441 [I 2022-10-10 19:33:37] ...trap/engines/engine.py.224: train: epoch 83 | batch 780/2304 [I 2022-10-10 19:33:37] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:55 [I 2022-10-10 19:33:37] ...trap/engines/engine.py.229: process: 0.86245 | load: 0.00311 [I 2022-10-10 19:33:37] ...trap/engines/engine.py.230: loss: 0.10571 [I 2022-10-10 19:33:37] ...optimizers/trijoint.py.151: total_norm: 13.748711 [I 2022-10-10 19:33:46] ...trap/engines/engine.py.224: train: epoch 83 | batch 790/2304 [I 2022-10-10 19:33:46] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:46 [I 2022-10-10 19:33:46] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00309 [I 2022-10-10 19:33:46] ...trap/engines/engine.py.230: loss: 0.11636 [I 2022-10-10 19:33:46] ...optimizers/trijoint.py.151: total_norm: 10.874015 [I 2022-10-10 19:33:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 800/2304 [I 2022-10-10 19:33:54] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:40 [I 2022-10-10 19:33:54] ...trap/engines/engine.py.229: process: 0.86777 | load: 0.00318 [I 2022-10-10 19:33:54] ...trap/engines/engine.py.230: loss: 0.21551 [I 2022-10-10 19:33:54] ...optimizers/trijoint.py.151: total_norm: 11.558682 [I 2022-10-10 19:34:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 810/2304 [I 2022-10-10 19:34:03] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:25 [I 2022-10-10 19:34:03] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00315 [I 2022-10-10 19:34:03] ...trap/engines/engine.py.230: loss: 0.16121 [I 2022-10-10 19:34:03] ...optimizers/trijoint.py.151: total_norm: 19.802128 [I 2022-10-10 19:34:12] ...trap/engines/engine.py.224: train: epoch 83 | batch 820/2304 [I 2022-10-10 19:34:12] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:23 [I 2022-10-10 19:34:12] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00309 [I 2022-10-10 19:34:12] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-10 19:34:12] ...optimizers/trijoint.py.151: total_norm: 14.703375 [I 2022-10-10 19:34:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 830/2304 [I 2022-10-10 19:34:20] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:15 [I 2022-10-10 19:34:20] ...trap/engines/engine.py.229: process: 0.85989 | load: 0.00309 [I 2022-10-10 19:34:20] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-10 19:34:20] ...optimizers/trijoint.py.151: total_norm: 14.456182 [I 2022-10-10 19:34:29] ...trap/engines/engine.py.224: train: epoch 83 | batch 840/2304 [I 2022-10-10 19:34:29] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:15 [I 2022-10-10 19:34:29] ...trap/engines/engine.py.229: process: 0.87802 | load: 0.00321 [I 2022-10-10 19:34:29] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-10 19:34:29] ...optimizers/trijoint.py.151: total_norm: 13.271902 [I 2022-10-10 19:34:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 850/2304 [I 2022-10-10 19:34:38] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:21:00 [I 2022-10-10 19:34:38] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00319 [I 2022-10-10 19:34:38] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-10 19:34:38] ...optimizers/trijoint.py.151: total_norm: 12.133130 [I 2022-10-10 19:34:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 860/2304 [I 2022-10-10 19:34:47] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:50 [I 2022-10-10 19:34:47] ...trap/engines/engine.py.229: process: 0.86480 | load: 0.00319 [I 2022-10-10 19:34:47] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-10 19:34:47] ...optimizers/trijoint.py.151: total_norm: 89.339798 [I 2022-10-10 19:34:55] ...trap/engines/engine.py.224: train: epoch 83 | batch 870/2304 [I 2022-10-10 19:34:55] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:39 [I 2022-10-10 19:34:55] ...trap/engines/engine.py.229: process: 0.87143 | load: 0.00316 [I 2022-10-10 19:34:55] ...trap/engines/engine.py.230: loss: 0.10360 [I 2022-10-10 19:34:55] ...optimizers/trijoint.py.151: total_norm: 11.149823 [I 2022-10-10 19:35:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 880/2304 [I 2022-10-10 19:35:04] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:35 [I 2022-10-10 19:35:04] ...trap/engines/engine.py.229: process: 0.87334 | load: 0.00313 [I 2022-10-10 19:35:04] ...trap/engines/engine.py.230: loss: 0.19758 [I 2022-10-10 19:35:04] ...optimizers/trijoint.py.151: total_norm: 23.591230 [I 2022-10-10 19:35:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 890/2304 [I 2022-10-10 19:35:13] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:24 [I 2022-10-10 19:35:13] ...trap/engines/engine.py.229: process: 0.87467 | load: 0.00320 [I 2022-10-10 19:35:13] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-10 19:35:13] ...optimizers/trijoint.py.151: total_norm: 11.293218 [I 2022-10-10 19:35:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 900/2304 [I 2022-10-10 19:35:21] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:08 [I 2022-10-10 19:35:21] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00317 [I 2022-10-10 19:35:21] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-10 19:35:21] ...optimizers/trijoint.py.151: total_norm: 28.541622 [I 2022-10-10 19:35:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 910/2304 [I 2022-10-10 19:35:30] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:05 [I 2022-10-10 19:35:30] ...trap/engines/engine.py.229: process: 0.86323 | load: 0.00311 [I 2022-10-10 19:35:30] ...trap/engines/engine.py.230: loss: 0.09851 [I 2022-10-10 19:35:30] ...optimizers/trijoint.py.151: total_norm: 12.973368 [I 2022-10-10 19:35:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 920/2304 [I 2022-10-10 19:35:38] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:54 [I 2022-10-10 19:35:38] ...trap/engines/engine.py.229: process: 0.86100 | load: 0.00310 [I 2022-10-10 19:35:38] ...trap/engines/engine.py.230: loss: 0.12639 [I 2022-10-10 19:35:38] ...optimizers/trijoint.py.151: total_norm: 20.542957 [I 2022-10-10 19:35:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 930/2304 [I 2022-10-10 19:35:47] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:41 [I 2022-10-10 19:35:47] ...trap/engines/engine.py.229: process: 0.86229 | load: 0.00316 [I 2022-10-10 19:35:47] ...trap/engines/engine.py.230: loss: 0.14567 [I 2022-10-10 19:35:47] ...optimizers/trijoint.py.151: total_norm: 19.747234 [I 2022-10-10 19:35:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 940/2304 [I 2022-10-10 19:35:56] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:34 [I 2022-10-10 19:35:56] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00316 [I 2022-10-10 19:35:56] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-10 19:35:56] ...optimizers/trijoint.py.151: total_norm: 45.369087 [I 2022-10-10 19:36:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 950/2304 [I 2022-10-10 19:36:04] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:34 [I 2022-10-10 19:36:04] ...trap/engines/engine.py.229: process: 0.91085 | load: 0.00310 [I 2022-10-10 19:36:04] ...trap/engines/engine.py.230: loss: 0.14197 [I 2022-10-10 19:36:04] ...optimizers/trijoint.py.151: total_norm: 15.161285 [I 2022-10-10 19:36:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 960/2304 [I 2022-10-10 19:36:13] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:27 [I 2022-10-10 19:36:13] ...trap/engines/engine.py.229: process: 0.86982 | load: 0.00311 [I 2022-10-10 19:36:13] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-10 19:36:13] ...optimizers/trijoint.py.151: total_norm: 17.960989 [I 2022-10-10 19:36:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 970/2304 [I 2022-10-10 19:36:22] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:15 [I 2022-10-10 19:36:22] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00306 [I 2022-10-10 19:36:22] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-10 19:36:22] ...optimizers/trijoint.py.151: total_norm: 16.205650 [I 2022-10-10 19:36:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 980/2304 [I 2022-10-10 19:36:30] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:05 [I 2022-10-10 19:36:30] ...trap/engines/engine.py.229: process: 0.85963 | load: 0.00307 [I 2022-10-10 19:36:30] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-10 19:36:30] ...optimizers/trijoint.py.151: total_norm: 25.617275 [I 2022-10-10 19:36:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 990/2304 [I 2022-10-10 19:36:39] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:46 [I 2022-10-10 19:36:39] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00294 [I 2022-10-10 19:36:39] ...trap/engines/engine.py.230: loss: 0.11220 [I 2022-10-10 19:36:39] ...optimizers/trijoint.py.151: total_norm: 12.315672 [I 2022-10-10 19:36:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1000/2304 [I 2022-10-10 19:36:48] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:42 [I 2022-10-10 19:36:48] ...trap/engines/engine.py.229: process: 0.85198 | load: 0.00315 [I 2022-10-10 19:36:48] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-10 19:36:48] ...optimizers/trijoint.py.151: total_norm: 15.175850 [I 2022-10-10 19:36:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1010/2304 [I 2022-10-10 19:36:56] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:44 [I 2022-10-10 19:36:56] ...trap/engines/engine.py.229: process: 0.86274 | load: 0.00312 [I 2022-10-10 19:36:56] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-10 19:36:56] ...optimizers/trijoint.py.151: total_norm: 14.077810 [I 2022-10-10 19:37:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1020/2304 [I 2022-10-10 19:37:05] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:33 [I 2022-10-10 19:37:05] ...trap/engines/engine.py.229: process: 0.87126 | load: 0.00304 [I 2022-10-10 19:37:05] ...trap/engines/engine.py.230: loss: 0.12323 [I 2022-10-10 19:37:05] ...optimizers/trijoint.py.151: total_norm: 23.510006 [I 2022-10-10 19:37:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 1030/2304 [I 2022-10-10 19:37:14] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:13 [I 2022-10-10 19:37:14] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00297 [I 2022-10-10 19:37:14] ...trap/engines/engine.py.230: loss: 0.18386 [I 2022-10-10 19:37:14] ...optimizers/trijoint.py.151: total_norm: 11.285336 [I 2022-10-10 19:37:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1040/2304 [I 2022-10-10 19:37:22] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:18:01 [I 2022-10-10 19:37:22] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00302 [I 2022-10-10 19:37:22] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-10 19:37:22] ...optimizers/trijoint.py.151: total_norm: 30.143133 [I 2022-10-10 19:37:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1050/2304 [I 2022-10-10 19:37:31] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:17:52 [I 2022-10-10 19:37:31] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00291 [I 2022-10-10 19:37:31] ...trap/engines/engine.py.230: loss: 0.11353 [I 2022-10-10 19:37:31] ...optimizers/trijoint.py.151: total_norm: 13.423355 [I 2022-10-10 19:37:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1060/2304 [I 2022-10-10 19:37:39] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:40 [I 2022-10-10 19:37:39] ...trap/engines/engine.py.229: process: 0.86014 | load: 0.00291 [I 2022-10-10 19:37:39] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-10 19:37:39] ...optimizers/trijoint.py.151: total_norm: 23.259611 [I 2022-10-10 19:37:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1070/2304 [I 2022-10-10 19:37:48] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:34 [I 2022-10-10 19:37:48] ...trap/engines/engine.py.229: process: 0.86055 | load: 0.00297 [I 2022-10-10 19:37:48] ...trap/engines/engine.py.230: loss: 0.08787 [I 2022-10-10 19:37:48] ...optimizers/trijoint.py.151: total_norm: 8.723739 [I 2022-10-10 19:37:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1080/2304 [I 2022-10-10 19:37:56] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:27 [I 2022-10-10 19:37:56] ...trap/engines/engine.py.229: process: 0.85967 | load: 0.00302 [I 2022-10-10 19:37:56] ...trap/engines/engine.py.230: loss: 0.22370 [I 2022-10-10 19:37:56] ...optimizers/trijoint.py.151: total_norm: 45.679623 [I 2022-10-10 19:38:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1090/2304 [I 2022-10-10 19:38:05] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:19 [I 2022-10-10 19:38:05] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00313 [I 2022-10-10 19:38:05] ...trap/engines/engine.py.230: loss: 0.10201 [I 2022-10-10 19:38:05] ...optimizers/trijoint.py.151: total_norm: 14.300676 [I 2022-10-10 19:38:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1100/2304 [I 2022-10-10 19:38:13] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:05 [I 2022-10-10 19:38:13] ...trap/engines/engine.py.229: process: 0.84926 | load: 0.00292 [I 2022-10-10 19:38:13] ...trap/engines/engine.py.230: loss: 0.13344 [I 2022-10-10 19:38:14] ...optimizers/trijoint.py.151: total_norm: 14.315771 [I 2022-10-10 19:38:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1110/2304 [I 2022-10-10 19:38:22] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:59 [I 2022-10-10 19:38:22] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00298 [I 2022-10-10 19:38:22] ...trap/engines/engine.py.230: loss: 0.11819 [I 2022-10-10 19:38:22] ...optimizers/trijoint.py.151: total_norm: 10.316150 [I 2022-10-10 19:38:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1120/2304 [I 2022-10-10 19:38:31] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:16:48 [I 2022-10-10 19:38:31] ...trap/engines/engine.py.229: process: 0.86448 | load: 0.00293 [I 2022-10-10 19:38:31] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-10 19:38:31] ...optimizers/trijoint.py.151: total_norm: 20.506857 [I 2022-10-10 19:38:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1130/2304 [I 2022-10-10 19:38:39] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:37 [I 2022-10-10 19:38:39] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00296 [I 2022-10-10 19:38:39] ...trap/engines/engine.py.230: loss: 0.11888 [I 2022-10-10 19:38:39] ...optimizers/trijoint.py.151: total_norm: 8.320762 [I 2022-10-10 19:38:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1140/2304 [I 2022-10-10 19:38:48] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:31 [I 2022-10-10 19:38:48] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00302 [I 2022-10-10 19:38:48] ...trap/engines/engine.py.230: loss: 0.13676 [I 2022-10-10 19:38:48] ...optimizers/trijoint.py.151: total_norm: 8.800933 [I 2022-10-10 19:38:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1150/2304 [I 2022-10-10 19:38:56] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:24 [I 2022-10-10 19:38:56] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00293 [I 2022-10-10 19:38:56] ...trap/engines/engine.py.230: loss: 0.12104 [I 2022-10-10 19:38:56] ...optimizers/trijoint.py.151: total_norm: 17.775255 [I 2022-10-10 19:39:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1160/2304 [I 2022-10-10 19:39:05] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:16 [I 2022-10-10 19:39:05] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00300 [I 2022-10-10 19:39:05] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-10 19:39:05] ...optimizers/trijoint.py.151: total_norm: 14.496297 [I 2022-10-10 19:39:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1170/2304 [I 2022-10-10 19:39:13] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:05 [I 2022-10-10 19:39:13] ...trap/engines/engine.py.229: process: 0.84858 | load: 0.00292 [I 2022-10-10 19:39:13] ...trap/engines/engine.py.230: loss: 0.08668 [I 2022-10-10 19:39:13] ...optimizers/trijoint.py.151: total_norm: 10.399806 [I 2022-10-10 19:39:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1180/2304 [I 2022-10-10 19:39:22] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:15:54 [I 2022-10-10 19:39:22] ...trap/engines/engine.py.229: process: 0.84953 | load: 0.00293 [I 2022-10-10 19:39:22] ...trap/engines/engine.py.230: loss: 0.17214 [I 2022-10-10 19:39:22] ...optimizers/trijoint.py.151: total_norm: 40.375263 [I 2022-10-10 19:39:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1190/2304 [I 2022-10-10 19:39:30] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:44 [I 2022-10-10 19:39:30] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00291 [I 2022-10-10 19:39:30] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-10 19:39:30] ...optimizers/trijoint.py.151: total_norm: 12.272663 [I 2022-10-10 19:39:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1200/2304 [I 2022-10-10 19:39:39] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:35 [I 2022-10-10 19:39:39] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00293 [I 2022-10-10 19:39:39] ...trap/engines/engine.py.230: loss: 0.19384 [I 2022-10-10 19:39:39] ...optimizers/trijoint.py.151: total_norm: 32.699593 [I 2022-10-10 19:39:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 1210/2304 [I 2022-10-10 19:39:47] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:29 [I 2022-10-10 19:39:47] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00294 [I 2022-10-10 19:39:47] ...trap/engines/engine.py.230: loss: 0.10568 [I 2022-10-10 19:39:47] ...optimizers/trijoint.py.151: total_norm: 20.548677 [I 2022-10-10 19:39:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1220/2304 [I 2022-10-10 19:39:56] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:27 [I 2022-10-10 19:39:56] ...trap/engines/engine.py.229: process: 0.86663 | load: 0.00302 [I 2022-10-10 19:39:56] ...trap/engines/engine.py.230: loss: 0.12087 [I 2022-10-10 19:39:56] ...optimizers/trijoint.py.151: total_norm: 13.391655 [I 2022-10-10 19:40:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 1230/2304 [I 2022-10-10 19:40:04] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:09 [I 2022-10-10 19:40:04] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00297 [I 2022-10-10 19:40:04] ...trap/engines/engine.py.230: loss: 0.10034 [I 2022-10-10 19:40:04] ...optimizers/trijoint.py.151: total_norm: 23.938820 [I 2022-10-10 19:40:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1240/2304 [I 2022-10-10 19:40:13] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:06 [I 2022-10-10 19:40:13] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00296 [I 2022-10-10 19:40:13] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-10 19:40:13] ...optimizers/trijoint.py.151: total_norm: 12.414685 [I 2022-10-10 19:40:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1250/2304 [I 2022-10-10 19:40:22] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:14:56 [I 2022-10-10 19:40:22] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00291 [I 2022-10-10 19:40:22] ...trap/engines/engine.py.230: loss: 0.09278 [I 2022-10-10 19:40:22] ...optimizers/trijoint.py.151: total_norm: 7.868534 [I 2022-10-10 19:40:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1260/2304 [I 2022-10-10 19:40:30] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:14:47 [I 2022-10-10 19:40:30] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00293 [I 2022-10-10 19:40:30] ...trap/engines/engine.py.230: loss: 0.17399 [I 2022-10-10 19:40:30] ...optimizers/trijoint.py.151: total_norm: 13.774800 [I 2022-10-10 19:40:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1270/2304 [I 2022-10-10 19:40:39] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:38 [I 2022-10-10 19:40:39] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00292 [I 2022-10-10 19:40:39] ...trap/engines/engine.py.230: loss: 0.09933 [I 2022-10-10 19:40:39] ...optimizers/trijoint.py.151: total_norm: 6.949719 [I 2022-10-10 19:40:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 1280/2304 [I 2022-10-10 19:40:47] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:36 [I 2022-10-10 19:40:47] ...trap/engines/engine.py.229: process: 0.86268 | load: 0.00293 [I 2022-10-10 19:40:47] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-10 19:40:47] ...optimizers/trijoint.py.151: total_norm: 9.979724 [I 2022-10-10 19:40:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1290/2304 [I 2022-10-10 19:40:56] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:21 [I 2022-10-10 19:40:56] ...trap/engines/engine.py.229: process: 0.85667 | load: 0.00294 [I 2022-10-10 19:40:56] ...trap/engines/engine.py.230: loss: 0.14424 [I 2022-10-10 19:40:56] ...optimizers/trijoint.py.151: total_norm: 40.954456 [I 2022-10-10 19:41:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 1300/2304 [I 2022-10-10 19:41:04] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:13 [I 2022-10-10 19:41:04] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00296 [I 2022-10-10 19:41:04] ...trap/engines/engine.py.230: loss: 0.09377 [I 2022-10-10 19:41:04] ...optimizers/trijoint.py.151: total_norm: 13.549553 [I 2022-10-10 19:41:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1310/2304 [I 2022-10-10 19:41:13] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:03 [I 2022-10-10 19:41:13] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00295 [I 2022-10-10 19:41:13] ...trap/engines/engine.py.230: loss: 0.20178 [I 2022-10-10 19:41:13] ...optimizers/trijoint.py.151: total_norm: 37.414772 [I 2022-10-10 19:41:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 1320/2304 [I 2022-10-10 19:41:21] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:56 [I 2022-10-10 19:41:21] ...trap/engines/engine.py.229: process: 0.83637 | load: 0.00300 [I 2022-10-10 19:41:21] ...trap/engines/engine.py.230: loss: 0.09304 [I 2022-10-10 19:41:21] ...optimizers/trijoint.py.151: total_norm: 13.966894 [I 2022-10-10 19:41:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1330/2304 [I 2022-10-10 19:41:30] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:49 [I 2022-10-10 19:41:30] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00306 [I 2022-10-10 19:41:30] ...trap/engines/engine.py.230: loss: 0.12474 [I 2022-10-10 19:41:30] ...optimizers/trijoint.py.151: total_norm: 19.190670 [I 2022-10-10 19:41:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 1340/2304 [I 2022-10-10 19:41:38] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:47 [I 2022-10-10 19:41:38] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00307 [I 2022-10-10 19:41:38] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-10 19:41:38] ...optimizers/trijoint.py.151: total_norm: 12.301091 [I 2022-10-10 19:41:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 1350/2304 [I 2022-10-10 19:41:47] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:40 [I 2022-10-10 19:41:47] ...trap/engines/engine.py.229: process: 0.86406 | load: 0.00314 [I 2022-10-10 19:41:47] ...trap/engines/engine.py.230: loss: 0.12721 [I 2022-10-10 19:41:47] ...optimizers/trijoint.py.151: total_norm: 12.179718 [I 2022-10-10 19:41:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1360/2304 [I 2022-10-10 19:41:56] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:26 [I 2022-10-10 19:41:56] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00305 [I 2022-10-10 19:41:56] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-10 19:41:56] ...optimizers/trijoint.py.151: total_norm: 19.026192 [I 2022-10-10 19:42:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 1370/2304 [I 2022-10-10 19:42:04] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:21 [I 2022-10-10 19:42:04] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00313 [I 2022-10-10 19:42:04] ...trap/engines/engine.py.230: loss: 0.14746 [I 2022-10-10 19:42:04] ...optimizers/trijoint.py.151: total_norm: 10.609718 [I 2022-10-10 19:42:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1380/2304 [I 2022-10-10 19:42:13] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:13 [I 2022-10-10 19:42:13] ...trap/engines/engine.py.229: process: 0.86310 | load: 0.00312 [I 2022-10-10 19:42:13] ...trap/engines/engine.py.230: loss: 0.10577 [I 2022-10-10 19:42:13] ...optimizers/trijoint.py.151: total_norm: 17.104429 [I 2022-10-10 19:42:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 1390/2304 [I 2022-10-10 19:42:21] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:05 [I 2022-10-10 19:42:21] ...trap/engines/engine.py.229: process: 0.85212 | load: 0.00310 [I 2022-10-10 19:42:21] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-10 19:42:21] ...optimizers/trijoint.py.151: total_norm: 16.026060 [I 2022-10-10 19:42:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1400/2304 [I 2022-10-10 19:42:30] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:58 [I 2022-10-10 19:42:30] ...trap/engines/engine.py.229: process: 0.85870 | load: 0.00308 [I 2022-10-10 19:42:30] ...trap/engines/engine.py.230: loss: 0.14721 [I 2022-10-10 19:42:30] ...optimizers/trijoint.py.151: total_norm: 19.798006 [I 2022-10-10 19:42:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1410/2304 [I 2022-10-10 19:42:39] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:53 [I 2022-10-10 19:42:39] ...trap/engines/engine.py.229: process: 0.86739 | load: 0.00311 [I 2022-10-10 19:42:39] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-10 19:42:39] ...optimizers/trijoint.py.151: total_norm: 23.550117 [I 2022-10-10 19:42:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 1420/2304 [I 2022-10-10 19:42:47] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:45 [I 2022-10-10 19:42:47] ...trap/engines/engine.py.229: process: 0.87308 | load: 0.00311 [I 2022-10-10 19:42:47] ...trap/engines/engine.py.230: loss: 0.08943 [I 2022-10-10 19:42:47] ...optimizers/trijoint.py.151: total_norm: 10.134752 [I 2022-10-10 19:42:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1430/2304 [I 2022-10-10 19:42:56] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:34 [I 2022-10-10 19:42:56] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00320 [I 2022-10-10 19:42:56] ...trap/engines/engine.py.230: loss: 0.10068 [I 2022-10-10 19:42:56] ...optimizers/trijoint.py.151: total_norm: 17.804848 [I 2022-10-10 19:43:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1440/2304 [I 2022-10-10 19:43:05] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:23 [I 2022-10-10 19:43:05] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00309 [I 2022-10-10 19:43:05] ...trap/engines/engine.py.230: loss: 0.18602 [I 2022-10-10 19:43:05] ...optimizers/trijoint.py.151: total_norm: 25.521837 [I 2022-10-10 19:43:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1450/2304 [I 2022-10-10 19:43:13] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:17 [I 2022-10-10 19:43:13] ...trap/engines/engine.py.229: process: 0.85886 | load: 0.00313 [I 2022-10-10 19:43:13] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-10 19:43:13] ...optimizers/trijoint.py.151: total_norm: 50.860237 [I 2022-10-10 19:43:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1460/2304 [I 2022-10-10 19:43:22] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:12:08 [I 2022-10-10 19:43:22] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00311 [I 2022-10-10 19:43:22] ...trap/engines/engine.py.230: loss: 0.17503 [I 2022-10-10 19:43:22] ...optimizers/trijoint.py.151: total_norm: 15.298296 [I 2022-10-10 19:43:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1470/2304 [I 2022-10-10 19:43:31] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:12:02 [I 2022-10-10 19:43:31] ...trap/engines/engine.py.229: process: 0.87898 | load: 0.00311 [I 2022-10-10 19:43:31] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-10 19:43:31] ...optimizers/trijoint.py.151: total_norm: 17.413561 [I 2022-10-10 19:43:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 1480/2304 [I 2022-10-10 19:43:39] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:49 [I 2022-10-10 19:43:39] ...trap/engines/engine.py.229: process: 0.86186 | load: 0.00319 [I 2022-10-10 19:43:39] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-10 19:43:39] ...optimizers/trijoint.py.151: total_norm: 13.519126 [I 2022-10-10 19:43:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1490/2304 [I 2022-10-10 19:43:48] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:43 [I 2022-10-10 19:43:48] ...trap/engines/engine.py.229: process: 0.85592 | load: 0.00308 [I 2022-10-10 19:43:48] ...trap/engines/engine.py.230: loss: 0.10858 [I 2022-10-10 19:43:48] ...optimizers/trijoint.py.151: total_norm: 11.317433 [I 2022-10-10 19:43:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1500/2304 [I 2022-10-10 19:43:57] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:11:34 [I 2022-10-10 19:43:57] ...trap/engines/engine.py.229: process: 0.85598 | load: 0.00312 [I 2022-10-10 19:43:57] ...trap/engines/engine.py.230: loss: 0.11991 [I 2022-10-10 19:43:57] ...optimizers/trijoint.py.151: total_norm: 11.402280 [I 2022-10-10 19:44:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1510/2304 [I 2022-10-10 19:44:05] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:27 [I 2022-10-10 19:44:05] ...trap/engines/engine.py.229: process: 0.86660 | load: 0.00310 [I 2022-10-10 19:44:05] ...trap/engines/engine.py.230: loss: 0.09863 [I 2022-10-10 19:44:05] ...optimizers/trijoint.py.151: total_norm: 9.022793 [I 2022-10-10 19:44:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 1520/2304 [I 2022-10-10 19:44:14] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:11:15 [I 2022-10-10 19:44:14] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00314 [I 2022-10-10 19:44:14] ...trap/engines/engine.py.230: loss: 0.11891 [I 2022-10-10 19:44:14] ...optimizers/trijoint.py.151: total_norm: 13.326303 [I 2022-10-10 19:44:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1530/2304 [I 2022-10-10 19:44:23] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:08 [I 2022-10-10 19:44:23] ...trap/engines/engine.py.229: process: 0.85746 | load: 0.00310 [I 2022-10-10 19:44:23] ...trap/engines/engine.py.230: loss: 0.09004 [I 2022-10-10 19:44:23] ...optimizers/trijoint.py.151: total_norm: 10.589177 [I 2022-10-10 19:44:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1540/2304 [I 2022-10-10 19:44:31] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:57 [I 2022-10-10 19:44:31] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00311 [I 2022-10-10 19:44:31] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-10 19:44:31] ...optimizers/trijoint.py.151: total_norm: 10.724949 [I 2022-10-10 19:44:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1550/2304 [I 2022-10-10 19:44:40] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:48 [I 2022-10-10 19:44:40] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00306 [I 2022-10-10 19:44:40] ...trap/engines/engine.py.230: loss: 0.08663 [I 2022-10-10 19:44:40] ...optimizers/trijoint.py.151: total_norm: 17.177641 [I 2022-10-10 19:44:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1560/2304 [I 2022-10-10 19:44:49] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:41 [I 2022-10-10 19:44:49] ...trap/engines/engine.py.229: process: 0.86246 | load: 0.00307 [I 2022-10-10 19:44:49] ...trap/engines/engine.py.230: loss: 0.20542 [I 2022-10-10 19:44:49] ...optimizers/trijoint.py.151: total_norm: 22.320253 [I 2022-10-10 19:44:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1570/2304 [I 2022-10-10 19:44:57] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:29 [I 2022-10-10 19:44:57] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00314 [I 2022-10-10 19:44:57] ...trap/engines/engine.py.230: loss: 0.08370 [I 2022-10-10 19:44:57] ...optimizers/trijoint.py.151: total_norm: 8.030726 [I 2022-10-10 19:45:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1580/2304 [I 2022-10-10 19:45:06] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:23 [I 2022-10-10 19:45:06] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00311 [I 2022-10-10 19:45:06] ...trap/engines/engine.py.230: loss: 0.11488 [I 2022-10-10 19:45:06] ...optimizers/trijoint.py.151: total_norm: 10.690892 [I 2022-10-10 19:45:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1590/2304 [I 2022-10-10 19:45:15] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:14 [I 2022-10-10 19:45:15] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00310 [I 2022-10-10 19:45:15] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-10 19:45:15] ...optimizers/trijoint.py.151: total_norm: 13.220408 [I 2022-10-10 19:45:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1600/2304 [I 2022-10-10 19:45:23] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:10:06 [I 2022-10-10 19:45:23] ...trap/engines/engine.py.229: process: 0.85449 | load: 0.00310 [I 2022-10-10 19:45:23] ...trap/engines/engine.py.230: loss: 0.15096 [I 2022-10-10 19:45:23] ...optimizers/trijoint.py.151: total_norm: 12.609877 [I 2022-10-10 19:45:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 1610/2304 [I 2022-10-10 19:45:32] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:52 [I 2022-10-10 19:45:32] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00295 [I 2022-10-10 19:45:32] ...trap/engines/engine.py.230: loss: 0.09932 [I 2022-10-10 19:45:32] ...optimizers/trijoint.py.151: total_norm: 8.980103 [I 2022-10-10 19:45:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1620/2304 [I 2022-10-10 19:45:40] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:50 [I 2022-10-10 19:45:40] ...trap/engines/engine.py.229: process: 0.87247 | load: 0.00292 [I 2022-10-10 19:45:40] ...trap/engines/engine.py.230: loss: 0.10952 [I 2022-10-10 19:45:40] ...optimizers/trijoint.py.151: total_norm: 8.200559 [I 2022-10-10 19:45:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1630/2304 [I 2022-10-10 19:45:49] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:33 [I 2022-10-10 19:45:49] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00296 [I 2022-10-10 19:45:49] ...trap/engines/engine.py.230: loss: 0.22276 [I 2022-10-10 19:45:49] ...optimizers/trijoint.py.151: total_norm: 21.241846 [I 2022-10-10 19:45:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 1640/2304 [I 2022-10-10 19:45:58] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:27 [I 2022-10-10 19:45:58] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00306 [I 2022-10-10 19:45:58] ...trap/engines/engine.py.230: loss: 0.16166 [I 2022-10-10 19:45:58] ...optimizers/trijoint.py.151: total_norm: 19.966919 [I 2022-10-10 19:46:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1650/2304 [I 2022-10-10 19:46:06] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:18 [I 2022-10-10 19:46:06] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00315 [I 2022-10-10 19:46:06] ...trap/engines/engine.py.230: loss: 0.17535 [I 2022-10-10 19:46:06] ...optimizers/trijoint.py.151: total_norm: 14.305799 [I 2022-10-10 19:46:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1660/2304 [I 2022-10-10 19:46:15] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:09:08 [I 2022-10-10 19:46:15] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00293 [I 2022-10-10 19:46:15] ...trap/engines/engine.py.230: loss: 0.10566 [I 2022-10-10 19:46:15] ...optimizers/trijoint.py.151: total_norm: 19.015955 [I 2022-10-10 19:46:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1670/2304 [I 2022-10-10 19:46:23] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:09:00 [I 2022-10-10 19:46:23] ...trap/engines/engine.py.229: process: 0.84757 | load: 0.00297 [I 2022-10-10 19:46:23] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-10 19:46:23] ...optimizers/trijoint.py.151: total_norm: 9.313311 [I 2022-10-10 19:46:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 1680/2304 [I 2022-10-10 19:46:32] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:54 [I 2022-10-10 19:46:32] ...trap/engines/engine.py.229: process: 0.86153 | load: 0.00295 [I 2022-10-10 19:46:32] ...trap/engines/engine.py.230: loss: 0.16166 [I 2022-10-10 19:46:32] ...optimizers/trijoint.py.151: total_norm: 15.029592 [I 2022-10-10 19:46:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1690/2304 [I 2022-10-10 19:46:40] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:45 [I 2022-10-10 19:46:40] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00294 [I 2022-10-10 19:46:40] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-10 19:46:40] ...optimizers/trijoint.py.151: total_norm: 18.956919 [I 2022-10-10 19:46:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1700/2304 [I 2022-10-10 19:46:49] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:31 [I 2022-10-10 19:46:49] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00306 [I 2022-10-10 19:46:49] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-10 19:46:49] ...optimizers/trijoint.py.151: total_norm: 11.660330 [I 2022-10-10 19:46:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1710/2304 [I 2022-10-10 19:46:57] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:26 [I 2022-10-10 19:46:57] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00296 [I 2022-10-10 19:46:57] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-10 19:46:57] ...optimizers/trijoint.py.151: total_norm: 12.851059 [I 2022-10-10 19:47:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1720/2304 [I 2022-10-10 19:47:06] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:17 [I 2022-10-10 19:47:06] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00299 [I 2022-10-10 19:47:06] ...trap/engines/engine.py.230: loss: 0.09625 [I 2022-10-10 19:47:06] ...optimizers/trijoint.py.151: total_norm: 17.129776 [I 2022-10-10 19:47:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 1730/2304 [I 2022-10-10 19:47:14] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:08:06 [I 2022-10-10 19:47:14] ...trap/engines/engine.py.229: process: 0.83551 | load: 0.00296 [I 2022-10-10 19:47:14] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-10 19:47:14] ...optimizers/trijoint.py.151: total_norm: 16.563261 [I 2022-10-10 19:47:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1740/2304 [I 2022-10-10 19:47:23] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:08:00 [I 2022-10-10 19:47:23] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00299 [I 2022-10-10 19:47:23] ...trap/engines/engine.py.230: loss: 0.16923 [I 2022-10-10 19:47:23] ...optimizers/trijoint.py.151: total_norm: 9.759822 [I 2022-10-10 19:47:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 1750/2304 [I 2022-10-10 19:47:32] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:52 [I 2022-10-10 19:47:32] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00298 [I 2022-10-10 19:47:32] ...trap/engines/engine.py.230: loss: 0.19195 [I 2022-10-10 19:47:32] ...optimizers/trijoint.py.151: total_norm: 18.882929 [I 2022-10-10 19:47:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1760/2304 [I 2022-10-10 19:47:40] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:43 [I 2022-10-10 19:47:40] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00301 [I 2022-10-10 19:47:40] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-10 19:47:40] ...optimizers/trijoint.py.151: total_norm: 18.589918 [I 2022-10-10 19:47:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1770/2304 [I 2022-10-10 19:47:49] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:33 [I 2022-10-10 19:47:49] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00298 [I 2022-10-10 19:47:49] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-10 19:47:49] ...optimizers/trijoint.py.151: total_norm: 31.021460 [I 2022-10-10 19:47:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1780/2304 [I 2022-10-10 19:47:57] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:27 [I 2022-10-10 19:47:57] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00295 [I 2022-10-10 19:47:57] ...trap/engines/engine.py.230: loss: 0.10755 [I 2022-10-10 19:47:57] ...optimizers/trijoint.py.151: total_norm: 10.943407 [I 2022-10-10 19:48:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1790/2304 [I 2022-10-10 19:48:06] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:22 [I 2022-10-10 19:48:06] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00307 [I 2022-10-10 19:48:06] ...trap/engines/engine.py.230: loss: 0.11587 [I 2022-10-10 19:48:06] ...optimizers/trijoint.py.151: total_norm: 11.043735 [I 2022-10-10 19:48:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1800/2304 [I 2022-10-10 19:48:15] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:15 [I 2022-10-10 19:48:15] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00312 [I 2022-10-10 19:48:15] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-10 19:48:15] ...optimizers/trijoint.py.151: total_norm: 15.215226 [I 2022-10-10 19:48:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1810/2304 [I 2022-10-10 19:48:23] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:07:05 [I 2022-10-10 19:48:23] ...trap/engines/engine.py.229: process: 0.86868 | load: 0.00308 [I 2022-10-10 19:48:23] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-10 19:48:23] ...optimizers/trijoint.py.151: total_norm: 22.609682 [I 2022-10-10 19:48:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 1820/2304 [I 2022-10-10 19:48:32] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:57 [I 2022-10-10 19:48:32] ...trap/engines/engine.py.229: process: 0.88783 | load: 0.00307 [I 2022-10-10 19:48:32] ...trap/engines/engine.py.230: loss: 0.11718 [I 2022-10-10 19:48:32] ...optimizers/trijoint.py.151: total_norm: 9.235371 [I 2022-10-10 19:48:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1830/2304 [I 2022-10-10 19:48:40] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:49 [I 2022-10-10 19:48:40] ...trap/engines/engine.py.229: process: 0.86346 | load: 0.00315 [I 2022-10-10 19:48:40] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-10 19:48:40] ...optimizers/trijoint.py.151: total_norm: 15.357493 [I 2022-10-10 19:48:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1840/2304 [I 2022-10-10 19:48:49] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:41 [I 2022-10-10 19:48:49] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00308 [I 2022-10-10 19:48:49] ...trap/engines/engine.py.230: loss: 0.10913 [I 2022-10-10 19:48:49] ...optimizers/trijoint.py.151: total_norm: 7.989695 [I 2022-10-10 19:48:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 1850/2304 [I 2022-10-10 19:48:58] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:32 [I 2022-10-10 19:48:58] ...trap/engines/engine.py.229: process: 0.86151 | load: 0.00317 [I 2022-10-10 19:48:58] ...trap/engines/engine.py.230: loss: 0.14687 [I 2022-10-10 19:48:58] ...optimizers/trijoint.py.151: total_norm: 22.516716 [I 2022-10-10 19:49:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 1860/2304 [I 2022-10-10 19:49:07] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:25 [I 2022-10-10 19:49:07] ...trap/engines/engine.py.229: process: 0.87199 | load: 0.00308 [I 2022-10-10 19:49:07] ...trap/engines/engine.py.230: loss: 0.08970 [I 2022-10-10 19:49:07] ...optimizers/trijoint.py.151: total_norm: 15.487402 [I 2022-10-10 19:49:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1870/2304 [I 2022-10-10 19:49:15] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:06:15 [I 2022-10-10 19:49:15] ...trap/engines/engine.py.229: process: 0.87796 | load: 0.00309 [I 2022-10-10 19:49:15] ...trap/engines/engine.py.230: loss: 0.13576 [I 2022-10-10 19:49:15] ...optimizers/trijoint.py.151: total_norm: 19.550152 [I 2022-10-10 19:49:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 1880/2304 [I 2022-10-10 19:49:24] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:06:07 [I 2022-10-10 19:49:24] ...trap/engines/engine.py.229: process: 0.86377 | load: 0.00322 [I 2022-10-10 19:49:24] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-10 19:49:24] ...optimizers/trijoint.py.151: total_norm: 35.361740 [I 2022-10-10 19:49:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 1890/2304 [I 2022-10-10 19:49:33] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:58 [I 2022-10-10 19:49:33] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00312 [I 2022-10-10 19:49:33] ...trap/engines/engine.py.230: loss: 0.10857 [I 2022-10-10 19:49:33] ...optimizers/trijoint.py.151: total_norm: 19.911474 [I 2022-10-10 19:49:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 1900/2304 [I 2022-10-10 19:49:41] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:49 [I 2022-10-10 19:49:41] ...trap/engines/engine.py.229: process: 0.86977 | load: 0.00336 [I 2022-10-10 19:49:41] ...trap/engines/engine.py.230: loss: 0.15973 [I 2022-10-10 19:49:41] ...optimizers/trijoint.py.151: total_norm: 12.341081 [I 2022-10-10 19:49:50] ...trap/engines/engine.py.224: train: epoch 83 | batch 1910/2304 [I 2022-10-10 19:49:50] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:38 [I 2022-10-10 19:49:50] ...trap/engines/engine.py.229: process: 0.86675 | load: 0.00320 [I 2022-10-10 19:49:50] ...trap/engines/engine.py.230: loss: 0.11344 [I 2022-10-10 19:49:50] ...optimizers/trijoint.py.151: total_norm: 16.892513 [I 2022-10-10 19:49:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 1920/2304 [I 2022-10-10 19:49:59] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:31 [I 2022-10-10 19:49:59] ...trap/engines/engine.py.229: process: 0.86876 | load: 0.00308 [I 2022-10-10 19:49:59] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-10 19:49:59] ...optimizers/trijoint.py.151: total_norm: 57.974239 [I 2022-10-10 19:50:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 1930/2304 [I 2022-10-10 19:50:07] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:23 [I 2022-10-10 19:50:07] ...trap/engines/engine.py.229: process: 0.87003 | load: 0.00309 [I 2022-10-10 19:50:07] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-10 19:50:07] ...optimizers/trijoint.py.151: total_norm: 18.135275 [I 2022-10-10 19:50:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 1940/2304 [I 2022-10-10 19:50:16] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:05:13 [I 2022-10-10 19:50:16] ...trap/engines/engine.py.229: process: 0.86123 | load: 0.00311 [I 2022-10-10 19:50:16] ...trap/engines/engine.py.230: loss: 0.14971 [I 2022-10-10 19:50:16] ...optimizers/trijoint.py.151: total_norm: 12.321373 [I 2022-10-10 19:50:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 1950/2304 [I 2022-10-10 19:50:25] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:05:07 [I 2022-10-10 19:50:25] ...trap/engines/engine.py.229: process: 0.87034 | load: 0.00317 [I 2022-10-10 19:50:25] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-10 19:50:25] ...optimizers/trijoint.py.151: total_norm: 16.208292 [I 2022-10-10 19:50:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 1960/2304 [I 2022-10-10 19:50:33] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:57 [I 2022-10-10 19:50:33] ...trap/engines/engine.py.229: process: 0.87434 | load: 0.00309 [I 2022-10-10 19:50:33] ...trap/engines/engine.py.230: loss: 0.09975 [I 2022-10-10 19:50:33] ...optimizers/trijoint.py.151: total_norm: 12.764361 [I 2022-10-10 19:50:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 1970/2304 [I 2022-10-10 19:50:42] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:50 [I 2022-10-10 19:50:42] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00317 [I 2022-10-10 19:50:42] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-10 19:50:42] ...optimizers/trijoint.py.151: total_norm: 14.424963 [I 2022-10-10 19:50:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 1980/2304 [I 2022-10-10 19:50:51] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:40 [I 2022-10-10 19:50:51] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00315 [I 2022-10-10 19:50:51] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-10 19:50:51] ...optimizers/trijoint.py.151: total_norm: 13.005010 [I 2022-10-10 19:50:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 1990/2304 [I 2022-10-10 19:50:59] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:31 [I 2022-10-10 19:50:59] ...trap/engines/engine.py.229: process: 0.88176 | load: 0.00315 [I 2022-10-10 19:50:59] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-10 19:50:59] ...optimizers/trijoint.py.151: total_norm: 9.053483 [I 2022-10-10 19:51:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 2000/2304 [I 2022-10-10 19:51:08] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:23 [I 2022-10-10 19:51:08] ...trap/engines/engine.py.229: process: 0.87394 | load: 0.00314 [I 2022-10-10 19:51:08] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-10 19:51:08] ...optimizers/trijoint.py.151: total_norm: 14.110915 [I 2022-10-10 19:51:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2010/2304 [I 2022-10-10 19:51:17] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:13 [I 2022-10-10 19:51:17] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00310 [I 2022-10-10 19:51:17] ...trap/engines/engine.py.230: loss: 0.09730 [I 2022-10-10 19:51:17] ...optimizers/trijoint.py.151: total_norm: 13.965656 [I 2022-10-10 19:51:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 2020/2304 [I 2022-10-10 19:51:25] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:04:03 [I 2022-10-10 19:51:25] ...trap/engines/engine.py.229: process: 0.84681 | load: 0.00311 [I 2022-10-10 19:51:25] ...trap/engines/engine.py.230: loss: 0.12219 [I 2022-10-10 19:51:25] ...optimizers/trijoint.py.151: total_norm: 9.249604 [I 2022-10-10 19:51:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 2030/2304 [I 2022-10-10 19:51:34] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:55 [I 2022-10-10 19:51:34] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00312 [I 2022-10-10 19:51:34] ...trap/engines/engine.py.230: loss: 0.19628 [I 2022-10-10 19:51:34] ...optimizers/trijoint.py.151: total_norm: 22.219851 [I 2022-10-10 19:51:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 2040/2304 [I 2022-10-10 19:51:42] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:48 [I 2022-10-10 19:51:42] ...trap/engines/engine.py.229: process: 0.87245 | load: 0.00314 [I 2022-10-10 19:51:42] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-10 19:51:42] ...optimizers/trijoint.py.151: total_norm: 15.272898 [I 2022-10-10 19:51:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 2050/2304 [I 2022-10-10 19:51:51] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:38 [I 2022-10-10 19:51:51] ...trap/engines/engine.py.229: process: 0.86707 | load: 0.00309 [I 2022-10-10 19:51:51] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-10 19:51:51] ...optimizers/trijoint.py.151: total_norm: 19.032961 [I 2022-10-10 19:52:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 2060/2304 [I 2022-10-10 19:52:00] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:29 [I 2022-10-10 19:52:00] ...trap/engines/engine.py.229: process: 0.86072 | load: 0.00293 [I 2022-10-10 19:52:00] ...trap/engines/engine.py.230: loss: 0.07333 [I 2022-10-10 19:52:00] ...optimizers/trijoint.py.151: total_norm: 11.368638 [I 2022-10-10 19:52:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 2070/2304 [I 2022-10-10 19:52:08] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:03:21 [I 2022-10-10 19:52:08] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00303 [I 2022-10-10 19:52:08] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-10 19:52:08] ...optimizers/trijoint.py.151: total_norm: 18.901031 [I 2022-10-10 19:52:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2080/2304 [I 2022-10-10 19:52:17] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:03:12 [I 2022-10-10 19:52:17] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00302 [I 2022-10-10 19:52:17] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-10 19:52:17] ...optimizers/trijoint.py.151: total_norm: 8.337268 [I 2022-10-10 19:52:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 2090/2304 [I 2022-10-10 19:52:25] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:03:02 [I 2022-10-10 19:52:25] ...trap/engines/engine.py.229: process: 0.85481 | load: 0.00295 [I 2022-10-10 19:52:25] ...trap/engines/engine.py.230: loss: 0.10944 [I 2022-10-10 19:52:25] ...optimizers/trijoint.py.151: total_norm: 14.022735 [I 2022-10-10 19:52:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 2100/2304 [I 2022-10-10 19:52:34] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:55 [I 2022-10-10 19:52:34] ...trap/engines/engine.py.229: process: 0.85737 | load: 0.00298 [I 2022-10-10 19:52:34] ...trap/engines/engine.py.230: loss: 0.11905 [I 2022-10-10 19:52:34] ...optimizers/trijoint.py.151: total_norm: 9.547085 [I 2022-10-10 19:52:43] ...trap/engines/engine.py.224: train: epoch 83 | batch 2110/2304 [I 2022-10-10 19:52:43] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:46 [I 2022-10-10 19:52:43] ...trap/engines/engine.py.229: process: 0.86624 | load: 0.00300 [I 2022-10-10 19:52:43] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-10 19:52:43] ...optimizers/trijoint.py.151: total_norm: 16.161161 [I 2022-10-10 19:52:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 2120/2304 [I 2022-10-10 19:52:51] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:37 [I 2022-10-10 19:52:51] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00295 [I 2022-10-10 19:52:51] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-10 19:52:51] ...optimizers/trijoint.py.151: total_norm: 40.747269 [I 2022-10-10 19:53:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 2130/2304 [I 2022-10-10 19:53:00] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:28 [I 2022-10-10 19:53:00] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00296 [I 2022-10-10 19:53:00] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-10 19:53:00] ...optimizers/trijoint.py.151: total_norm: 9.501811 [I 2022-10-10 19:53:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 2140/2304 [I 2022-10-10 19:53:08] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:19 [I 2022-10-10 19:53:08] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00297 [I 2022-10-10 19:53:08] ...trap/engines/engine.py.230: loss: 0.10246 [I 2022-10-10 19:53:08] ...optimizers/trijoint.py.151: total_norm: 15.493364 [I 2022-10-10 19:53:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2150/2304 [I 2022-10-10 19:53:17] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:02:11 [I 2022-10-10 19:53:17] ...trap/engines/engine.py.229: process: 0.84661 | load: 0.00302 [I 2022-10-10 19:53:17] ...trap/engines/engine.py.230: loss: 0.14058 [I 2022-10-10 19:53:17] ...optimizers/trijoint.py.151: total_norm: 17.087809 [I 2022-10-10 19:53:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 2160/2304 [I 2022-10-10 19:53:25] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:02 [I 2022-10-10 19:53:25] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00299 [I 2022-10-10 19:53:25] ...trap/engines/engine.py.230: loss: 0.13592 [I 2022-10-10 19:53:25] ...optimizers/trijoint.py.151: total_norm: 17.411657 [I 2022-10-10 19:53:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 2170/2304 [I 2022-10-10 19:53:34] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:53 [I 2022-10-10 19:53:34] ...trap/engines/engine.py.229: process: 0.83674 | load: 0.00303 [I 2022-10-10 19:53:34] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-10 19:53:34] ...optimizers/trijoint.py.151: total_norm: 8.550771 [I 2022-10-10 19:53:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 2180/2304 [I 2022-10-10 19:53:42] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:45 [I 2022-10-10 19:53:42] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00301 [I 2022-10-10 19:53:42] ...trap/engines/engine.py.230: loss: 0.11353 [I 2022-10-10 19:53:42] ...optimizers/trijoint.py.151: total_norm: 16.943989 [I 2022-10-10 19:53:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 2190/2304 [I 2022-10-10 19:53:51] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:36 [I 2022-10-10 19:53:51] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00296 [I 2022-10-10 19:53:51] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-10 19:53:51] ...optimizers/trijoint.py.151: total_norm: 13.950204 [I 2022-10-10 19:54:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 2200/2304 [I 2022-10-10 19:54:00] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:29 [I 2022-10-10 19:54:00] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00313 [I 2022-10-10 19:54:00] ...trap/engines/engine.py.230: loss: 0.16090 [I 2022-10-10 19:54:00] ...optimizers/trijoint.py.151: total_norm: 15.941019 [I 2022-10-10 19:54:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 2210/2304 [I 2022-10-10 19:54:08] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:20 [I 2022-10-10 19:54:08] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00311 [I 2022-10-10 19:54:08] ...trap/engines/engine.py.230: loss: 0.14569 [I 2022-10-10 19:54:08] ...optimizers/trijoint.py.151: total_norm: 26.251282 [I 2022-10-10 19:54:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2220/2304 [I 2022-10-10 19:54:17] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:01:12 [I 2022-10-10 19:54:17] ...trap/engines/engine.py.229: process: 0.86147 | load: 0.00306 [I 2022-10-10 19:54:17] ...trap/engines/engine.py.230: loss: 0.14900 [I 2022-10-10 19:54:17] ...optimizers/trijoint.py.151: total_norm: 11.502207 [I 2022-10-10 19:54:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 2230/2304 [I 2022-10-10 19:54:25] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:03 [I 2022-10-10 19:54:25] ...trap/engines/engine.py.229: process: 0.87606 | load: 0.00311 [I 2022-10-10 19:54:26] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-10 19:54:26] ...optimizers/trijoint.py.151: total_norm: 8.244176 [I 2022-10-10 19:54:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 2240/2304 [I 2022-10-10 19:54:34] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:55 [I 2022-10-10 19:54:34] ...trap/engines/engine.py.229: process: 0.88091 | load: 0.00315 [I 2022-10-10 19:54:34] ...trap/engines/engine.py.230: loss: 0.16147 [I 2022-10-10 19:54:34] ...optimizers/trijoint.py.151: total_norm: 20.813477 [I 2022-10-10 19:54:43] ...trap/engines/engine.py.224: train: epoch 83 | batch 2250/2304 [I 2022-10-10 19:54:43] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:46 [I 2022-10-10 19:54:43] ...trap/engines/engine.py.229: process: 0.86104 | load: 0.00305 [I 2022-10-10 19:54:43] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-10 19:54:43] ...optimizers/trijoint.py.151: total_norm: 14.015753 [I 2022-10-10 19:54:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 2260/2304 [I 2022-10-10 19:54:51] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:38 [I 2022-10-10 19:54:51] ...trap/engines/engine.py.229: process: 0.88214 | load: 0.00305 [I 2022-10-10 19:54:51] ...trap/engines/engine.py.230: loss: 0.13532 [I 2022-10-10 19:54:51] ...optimizers/trijoint.py.151: total_norm: 21.073790 [I 2022-10-10 19:55:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 2270/2304 [I 2022-10-10 19:55:00] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:29 [I 2022-10-10 19:55:00] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00306 [I 2022-10-10 19:55:00] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-10 19:55:00] ...optimizers/trijoint.py.151: total_norm: 10.263946 [I 2022-10-10 19:55:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 2280/2304 [I 2022-10-10 19:55:09] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:20 [I 2022-10-10 19:55:09] ...trap/engines/engine.py.229: process: 0.86752 | load: 0.00314 [I 2022-10-10 19:55:09] ...trap/engines/engine.py.230: loss: 0.10721 [I 2022-10-10 19:55:09] ...optimizers/trijoint.py.151: total_norm: 49.625141 [I 2022-10-10 19:55:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2290/2304 [I 2022-10-10 19:55:17] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:12 [I 2022-10-10 19:55:17] ...trap/engines/engine.py.229: process: 0.86693 | load: 0.00312 [I 2022-10-10 19:55:17] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-10 19:55:17] ...optimizers/trijoint.py.151: total_norm: 10.120447 [I 2022-10-10 19:55:26] ...trap/engines/engine.py.224: train: epoch 83 | batch 2300/2304 [I 2022-10-10 19:55:26] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:03 [I 2022-10-10 19:55:26] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00019 [I 2022-10-10 19:55:26] ...trap/engines/engine.py.230: loss: 0.12468 [I 2022-10-10 19:55:26] ...optimizers/trijoint.py.151: total_norm: 30.105778 [I 2022-10-10 19:55:29] ...trap/engines/engine.py.224: train: epoch 83 | batch 2304/2304 [I 2022-10-10 19:55:29] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:00 [I 2022-10-10 19:55:29] ...trap/engines/engine.py.229: process: 0.83779 | load: 0.00019 [I 2022-10-10 19:55:29] ...trap/engines/engine.py.230: loss: 0.10507 [I 2022-10-10 19:55:29] ...optimizers/trijoint.py.151: total_norm: 8.211506 [S 2022-10-10 19:55:31] ...trap/engines/engine.py.129: train_epoch.epoch: 83 [S 2022-10-10 19:55:31] ...trap/engines/engine.py.129: train_epoch.loss: 0.13120160250239674 [I 2022-10-10 19:55:31] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 19:55:32] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 83 [I 2022-10-10 19:55:32] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 19:55:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 19:55:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 19:55:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 19:55:37] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 19:55:38] ...trap/engines/engine.py.321: eval: epoch 83 | batch 0/510 [I 2022-10-10 19:55:38] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:38 [I 2022-10-10 19:55:38] ...trap/engines/engine.py.326: process: 5.60518 | load: 3.40367 [I 2022-10-10 19:55:43] ...trap/engines/engine.py.321: eval: epoch 83 | batch 10/510 [I 2022-10-10 19:55:43] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:08 [I 2022-10-10 19:55:43] ...trap/engines/engine.py.326: process: 0.55177 | load: 0.00021 [I 2022-10-10 19:55:49] ...trap/engines/engine.py.321: eval: epoch 83 | batch 20/510 [I 2022-10-10 19:55:49] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:24 [I 2022-10-10 19:55:49] ...trap/engines/engine.py.326: process: 0.62404 | load: 0.00021 [I 2022-10-10 19:55:55] ...trap/engines/engine.py.321: eval: epoch 83 | batch 30/510 [I 2022-10-10 19:55:55] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:52 [I 2022-10-10 19:55:55] ...trap/engines/engine.py.326: process: 0.75274 | load: 0.00020 [I 2022-10-10 19:56:02] ...trap/engines/engine.py.321: eval: epoch 83 | batch 40/510 [I 2022-10-10 19:56:02] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:05:09 [I 2022-10-10 19:56:02] ...trap/engines/engine.py.326: process: 0.43391 | load: 0.00022 [I 2022-10-10 19:56:07] ...trap/engines/engine.py.321: eval: epoch 83 | batch 50/510 [I 2022-10-10 19:56:07] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:08 [I 2022-10-10 19:56:07] ...trap/engines/engine.py.326: process: 0.53868 | load: 0.00020 [I 2022-10-10 19:56:13] ...trap/engines/engine.py.321: eval: epoch 83 | batch 60/510 [I 2022-10-10 19:56:13] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:10 [I 2022-10-10 19:56:13] ...trap/engines/engine.py.326: process: 0.56946 | load: 0.00020 [I 2022-10-10 19:56:18] ...trap/engines/engine.py.321: eval: epoch 83 | batch 70/510 [I 2022-10-10 19:56:18] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:03 [I 2022-10-10 19:56:18] ...trap/engines/engine.py.326: process: 0.57199 | load: 0.00022 [I 2022-10-10 19:56:24] ...trap/engines/engine.py.321: eval: epoch 83 | batch 80/510 [I 2022-10-10 19:56:24] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:04:27 [I 2022-10-10 19:56:24] ...trap/engines/engine.py.326: process: 0.72993 | load: 0.00021 [I 2022-10-10 19:56:30] ...trap/engines/engine.py.321: eval: epoch 83 | batch 90/510 [I 2022-10-10 19:56:30] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:54 [I 2022-10-10 19:56:30] ...trap/engines/engine.py.326: process: 0.66393 | load: 0.00023 [I 2022-10-10 19:56:34] ...trap/engines/engine.py.321: eval: epoch 83 | batch 100/510 [I 2022-10-10 19:56:34] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:03:11 [I 2022-10-10 19:56:34] ...trap/engines/engine.py.326: process: 0.31414 | load: 0.00020 [I 2022-10-10 19:56:38] ...trap/engines/engine.py.321: eval: epoch 83 | batch 110/510 [I 2022-10-10 19:56:38] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:15 [I 2022-10-10 19:56:38] ...trap/engines/engine.py.326: process: 0.32194 | load: 0.00020 [I 2022-10-10 19:56:41] ...trap/engines/engine.py.321: eval: epoch 83 | batch 120/510 [I 2022-10-10 19:56:41] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:06 [I 2022-10-10 19:56:41] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00021 [I 2022-10-10 19:56:44] ...trap/engines/engine.py.321: eval: epoch 83 | batch 130/510 [I 2022-10-10 19:56:44] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:02 [I 2022-10-10 19:56:44] ...trap/engines/engine.py.326: process: 0.32542 | load: 0.00021 [I 2022-10-10 19:56:47] ...trap/engines/engine.py.321: eval: epoch 83 | batch 140/510 [I 2022-10-10 19:56:47] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:02:00 [I 2022-10-10 19:56:47] ...trap/engines/engine.py.326: process: 0.32417 | load: 0.00021 [I 2022-10-10 19:56:51] ...trap/engines/engine.py.321: eval: epoch 83 | batch 150/510 [I 2022-10-10 19:56:51] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:55 [I 2022-10-10 19:56:51] ...trap/engines/engine.py.326: process: 0.33022 | load: 0.00020 [I 2022-10-10 19:56:54] ...trap/engines/engine.py.321: eval: epoch 83 | batch 160/510 [I 2022-10-10 19:56:54] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:53 [I 2022-10-10 19:56:54] ...trap/engines/engine.py.326: process: 0.32031 | load: 0.00020 [I 2022-10-10 19:56:57] ...trap/engines/engine.py.321: eval: epoch 83 | batch 170/510 [I 2022-10-10 19:56:57] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:49 [I 2022-10-10 19:56:57] ...trap/engines/engine.py.326: process: 0.32168 | load: 0.00021 [I 2022-10-10 19:57:00] ...trap/engines/engine.py.321: eval: epoch 83 | batch 180/510 [I 2022-10-10 19:57:00] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:46 [I 2022-10-10 19:57:00] ...trap/engines/engine.py.326: process: 0.32961 | load: 0.00021 [I 2022-10-10 19:57:04] ...trap/engines/engine.py.321: eval: epoch 83 | batch 190/510 [I 2022-10-10 19:57:04] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:43 [I 2022-10-10 19:57:04] ...trap/engines/engine.py.326: process: 0.32562 | load: 0.00020 [I 2022-10-10 19:57:07] ...trap/engines/engine.py.321: eval: epoch 83 | batch 200/510 [I 2022-10-10 19:57:07] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:40 [I 2022-10-10 19:57:07] ...trap/engines/engine.py.326: process: 0.32312 | load: 0.00020 [I 2022-10-10 19:57:10] ...trap/engines/engine.py.321: eval: epoch 83 | batch 210/510 [I 2022-10-10 19:57:10] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:36 [I 2022-10-10 19:57:10] ...trap/engines/engine.py.326: process: 0.32301 | load: 0.00021 [I 2022-10-10 19:57:13] ...trap/engines/engine.py.321: eval: epoch 83 | batch 220/510 [I 2022-10-10 19:57:13] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:33 [I 2022-10-10 19:57:13] ...trap/engines/engine.py.326: process: 0.32480 | load: 0.00020 [I 2022-10-10 19:57:16] ...trap/engines/engine.py.321: eval: epoch 83 | batch 230/510 [I 2022-10-10 19:57:16] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:30 [I 2022-10-10 19:57:16] ...trap/engines/engine.py.326: process: 0.32520 | load: 0.00020 [I 2022-10-10 19:57:20] ...trap/engines/engine.py.321: eval: epoch 83 | batch 240/510 [I 2022-10-10 19:57:20] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:27 [I 2022-10-10 19:57:20] ...trap/engines/engine.py.326: process: 0.32464 | load: 0.00022 [I 2022-10-10 19:57:23] ...trap/engines/engine.py.321: eval: epoch 83 | batch 250/510 [I 2022-10-10 19:57:23] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:24 [I 2022-10-10 19:57:23] ...trap/engines/engine.py.326: process: 0.34303 | load: 0.00022 [I 2022-10-10 19:57:26] ...trap/engines/engine.py.321: eval: epoch 83 | batch 260/510 [I 2022-10-10 19:57:26] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:21 [I 2022-10-10 19:57:26] ...trap/engines/engine.py.326: process: 0.32550 | load: 0.00021 [I 2022-10-10 19:57:29] ...trap/engines/engine.py.321: eval: epoch 83 | batch 270/510 [I 2022-10-10 19:57:29] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:17 [I 2022-10-10 19:57:29] ...trap/engines/engine.py.326: process: 0.32144 | load: 0.00022 [I 2022-10-10 19:57:33] ...trap/engines/engine.py.321: eval: epoch 83 | batch 280/510 [I 2022-10-10 19:57:33] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:14 [I 2022-10-10 19:57:33] ...trap/engines/engine.py.326: process: 0.32442 | load: 0.00022 [I 2022-10-10 19:57:36] ...trap/engines/engine.py.321: eval: epoch 83 | batch 290/510 [I 2022-10-10 19:57:36] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:11 [I 2022-10-10 19:57:36] ...trap/engines/engine.py.326: process: 0.31701 | load: 0.00022 [I 2022-10-10 19:57:39] ...trap/engines/engine.py.321: eval: epoch 83 | batch 300/510 [I 2022-10-10 19:57:39] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:07 [I 2022-10-10 19:57:39] ...trap/engines/engine.py.326: process: 0.32520 | load: 0.00022 [I 2022-10-10 19:57:42] ...trap/engines/engine.py.321: eval: epoch 83 | batch 310/510 [I 2022-10-10 19:57:42] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:04 [I 2022-10-10 19:57:42] ...trap/engines/engine.py.326: process: 0.32685 | load: 0.00022 [I 2022-10-10 19:57:46] ...trap/engines/engine.py.321: eval: epoch 83 | batch 320/510 [I 2022-10-10 19:57:46] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:01:01 [I 2022-10-10 19:57:46] ...trap/engines/engine.py.326: process: 0.31533 | load: 0.00020 [I 2022-10-10 19:57:49] ...trap/engines/engine.py.321: eval: epoch 83 | batch 330/510 [I 2022-10-10 19:57:49] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:57 [I 2022-10-10 19:57:49] ...trap/engines/engine.py.326: process: 0.32014 | load: 0.00020 [I 2022-10-10 19:57:52] ...trap/engines/engine.py.321: eval: epoch 83 | batch 340/510 [I 2022-10-10 19:57:52] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:54 [I 2022-10-10 19:57:52] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00020 [I 2022-10-10 19:57:55] ...trap/engines/engine.py.321: eval: epoch 83 | batch 350/510 [I 2022-10-10 19:57:55] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:51 [I 2022-10-10 19:57:55] ...trap/engines/engine.py.326: process: 0.32330 | load: 0.00019 [I 2022-10-10 19:57:58] ...trap/engines/engine.py.321: eval: epoch 83 | batch 360/510 [I 2022-10-10 19:57:58] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:48 [I 2022-10-10 19:57:58] ...trap/engines/engine.py.326: process: 0.32140 | load: 0.00020 [I 2022-10-10 19:58:02] ...trap/engines/engine.py.321: eval: epoch 83 | batch 370/510 [I 2022-10-10 19:58:02] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:44 [I 2022-10-10 19:58:02] ...trap/engines/engine.py.326: process: 0.31674 | load: 0.00021 [I 2022-10-10 19:58:05] ...trap/engines/engine.py.321: eval: epoch 83 | batch 380/510 [I 2022-10-10 19:58:05] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:41 [I 2022-10-10 19:58:05] ...trap/engines/engine.py.326: process: 0.31712 | load: 0.00020 [I 2022-10-10 19:58:08] ...trap/engines/engine.py.321: eval: epoch 83 | batch 390/510 [I 2022-10-10 19:58:08] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:38 [I 2022-10-10 19:58:08] ...trap/engines/engine.py.326: process: 0.31665 | load: 0.00020 [I 2022-10-10 19:58:11] ...trap/engines/engine.py.321: eval: epoch 83 | batch 400/510 [I 2022-10-10 19:58:11] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:35 [I 2022-10-10 19:58:11] ...trap/engines/engine.py.326: process: 0.32080 | load: 0.00020 [I 2022-10-10 19:58:14] ...trap/engines/engine.py.321: eval: epoch 83 | batch 410/510 [I 2022-10-10 19:58:14] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:31 [I 2022-10-10 19:58:14] ...trap/engines/engine.py.326: process: 0.32144 | load: 0.00021 [I 2022-10-10 19:58:18] ...trap/engines/engine.py.321: eval: epoch 83 | batch 420/510 [I 2022-10-10 19:58:18] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:28 [I 2022-10-10 19:58:18] ...trap/engines/engine.py.326: process: 0.31904 | load: 0.00020 [I 2022-10-10 19:58:21] ...trap/engines/engine.py.321: eval: epoch 83 | batch 430/510 [I 2022-10-10 19:58:21] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:25 [I 2022-10-10 19:58:21] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00021 [I 2022-10-10 19:58:24] ...trap/engines/engine.py.321: eval: epoch 83 | batch 440/510 [I 2022-10-10 19:58:24] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:22 [I 2022-10-10 19:58:24] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00019 [I 2022-10-10 19:58:27] ...trap/engines/engine.py.321: eval: epoch 83 | batch 450/510 [I 2022-10-10 19:58:27] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:19 [I 2022-10-10 19:58:27] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00021 [I 2022-10-10 19:58:30] ...trap/engines/engine.py.321: eval: epoch 83 | batch 460/510 [I 2022-10-10 19:58:30] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:16 [I 2022-10-10 19:58:30] ...trap/engines/engine.py.326: process: 0.32862 | load: 0.00021 [I 2022-10-10 19:58:34] ...trap/engines/engine.py.321: eval: epoch 83 | batch 470/510 [I 2022-10-10 19:58:34] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:12 [I 2022-10-10 19:58:34] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00020 [I 2022-10-10 19:58:37] ...trap/engines/engine.py.321: eval: epoch 83 | batch 480/510 [I 2022-10-10 19:58:37] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:09 [I 2022-10-10 19:58:37] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00021 [I 2022-10-10 19:58:40] ...trap/engines/engine.py.321: eval: epoch 83 | batch 490/510 [I 2022-10-10 19:58:40] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:06 [I 2022-10-10 19:58:40] ...trap/engines/engine.py.326: process: 0.30986 | load: 0.00019 [I 2022-10-10 19:58:43] ...trap/engines/engine.py.321: eval: epoch 83 | batch 500/510 [I 2022-10-10 19:58:43] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:03 [I 2022-10-10 19:58:43] ...trap/engines/engine.py.326: process: 0.31536 | load: 0.00020 [I 2022-10-10 19:58:46] ...trap/engines/engine.py.321: eval: epoch 83 | batch 510/510 [I 2022-10-10 19:58:46] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:00 [I 2022-10-10 19:58:46] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00016 [S 2022-10-10 19:58:47] ...trap/engines/engine.py.132: eval_epoch.epoch: 83 [I 2022-10-10 19:58:47] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.745900 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909600 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936300 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.739600 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906200 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016232 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012306 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009089 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015108 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013159 [S 2022-10-10 19:59:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008124 [I 2022-10-10 19:59:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 19:59:29] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 19:59:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 19:59:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 19:59:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 19:59:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 19:59:37] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 19:59:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 19:59:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 84 [I 2022-10-10 19:59:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 19:59:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 0/2304 [I 2022-10-10 19:59:44] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:43 [I 2022-10-10 19:59:44] ...trap/engines/engine.py.229: process: 3.24812 | load: 2.31580 [I 2022-10-10 19:59:44] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-10 19:59:44] ...optimizers/trijoint.py.151: total_norm: 19.774570 [I 2022-10-10 19:59:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 10/2304 [I 2022-10-10 19:59:52] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:44 [I 2022-10-10 19:59:52] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00650 [I 2022-10-10 19:59:52] ...trap/engines/engine.py.230: loss: 0.10217 [I 2022-10-10 19:59:52] ...optimizers/trijoint.py.151: total_norm: 20.237911 [I 2022-10-10 20:00:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 20/2304 [I 2022-10-10 20:00:01] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:52 [I 2022-10-10 20:00:01] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00526 [I 2022-10-10 20:00:01] ...trap/engines/engine.py.230: loss: 0.08103 [I 2022-10-10 20:00:01] ...optimizers/trijoint.py.151: total_norm: 9.557718 [I 2022-10-10 20:00:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 30/2304 [I 2022-10-10 20:00:10] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:56 [I 2022-10-10 20:00:10] ...trap/engines/engine.py.229: process: 0.87035 | load: 0.00434 [I 2022-10-10 20:00:10] ...trap/engines/engine.py.230: loss: 0.10177 [I 2022-10-10 20:00:10] ...optimizers/trijoint.py.151: total_norm: 17.812296 [I 2022-10-10 20:00:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 40/2304 [I 2022-10-10 20:00:18] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:34 [I 2022-10-10 20:00:18] ...trap/engines/engine.py.229: process: 0.87431 | load: 0.00365 [I 2022-10-10 20:00:18] ...trap/engines/engine.py.230: loss: 0.14985 [I 2022-10-10 20:00:18] ...optimizers/trijoint.py.151: total_norm: 22.427355 [I 2022-10-10 20:00:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 50/2304 [I 2022-10-10 20:00:46] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 1:19:40 [I 2022-10-10 20:00:46] ...trap/engines/engine.py.229: process: 0.86651 | load: 0.00370 [I 2022-10-10 20:00:46] ...trap/engines/engine.py.230: loss: 0.10989 [I 2022-10-10 20:00:46] ...optimizers/trijoint.py.151: total_norm: 13.045717 [I 2022-10-10 20:00:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 60/2304 [I 2022-10-10 20:00:55] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:37:11 [I 2022-10-10 20:00:55] ...trap/engines/engine.py.229: process: 0.85762 | load: 0.00396 [I 2022-10-10 20:00:55] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-10 20:00:55] ...optimizers/trijoint.py.151: total_norm: 9.030338 [I 2022-10-10 20:01:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 70/2304 [I 2022-10-10 20:01:03] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:32:27 [I 2022-10-10 20:01:03] ...trap/engines/engine.py.229: process: 0.87197 | load: 0.00325 [I 2022-10-10 20:01:03] ...trap/engines/engine.py.230: loss: 0.12418 [I 2022-10-10 20:01:03] ...optimizers/trijoint.py.151: total_norm: 16.073809 [I 2022-10-10 20:01:12] ...trap/engines/engine.py.224: train: epoch 84 | batch 80/2304 [I 2022-10-10 20:01:12] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:32:04 [I 2022-10-10 20:01:12] ...trap/engines/engine.py.229: process: 0.86690 | load: 0.00389 [I 2022-10-10 20:01:12] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-10 20:01:12] ...optimizers/trijoint.py.151: total_norm: 9.608467 [I 2022-10-10 20:01:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 90/2304 [I 2022-10-10 20:01:21] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:31:49 [I 2022-10-10 20:01:21] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00351 [I 2022-10-10 20:01:21] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-10 20:01:21] ...optimizers/trijoint.py.151: total_norm: 20.724195 [I 2022-10-10 20:01:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 100/2304 [I 2022-10-10 20:01:29] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:31:41 [I 2022-10-10 20:01:29] ...trap/engines/engine.py.229: process: 0.86609 | load: 0.00340 [I 2022-10-10 20:01:29] ...trap/engines/engine.py.230: loss: 0.14472 [I 2022-10-10 20:01:29] ...optimizers/trijoint.py.151: total_norm: 12.035501 [I 2022-10-10 20:01:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 110/2304 [I 2022-10-10 20:01:38] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:31:37 [I 2022-10-10 20:01:38] ...trap/engines/engine.py.229: process: 0.85965 | load: 0.00312 [I 2022-10-10 20:01:38] ...trap/engines/engine.py.230: loss: 0.18774 [I 2022-10-10 20:01:38] ...optimizers/trijoint.py.151: total_norm: 21.615770 [I 2022-10-10 20:01:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 120/2304 [I 2022-10-10 20:01:47] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:31:21 [I 2022-10-10 20:01:47] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00369 [I 2022-10-10 20:01:47] ...trap/engines/engine.py.230: loss: 0.17020 [I 2022-10-10 20:01:47] ...optimizers/trijoint.py.151: total_norm: 23.452753 [I 2022-10-10 20:01:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 130/2304 [I 2022-10-10 20:01:55] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:31:09 [I 2022-10-10 20:01:55] ...trap/engines/engine.py.229: process: 0.84939 | load: 0.00309 [I 2022-10-10 20:01:55] ...trap/engines/engine.py.230: loss: 0.15798 [I 2022-10-10 20:01:55] ...optimizers/trijoint.py.151: total_norm: 14.053709 [I 2022-10-10 20:02:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 140/2304 [I 2022-10-10 20:02:04] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:31:01 [I 2022-10-10 20:02:04] ...trap/engines/engine.py.229: process: 0.86503 | load: 0.00325 [I 2022-10-10 20:02:04] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-10 20:02:04] ...optimizers/trijoint.py.151: total_norm: 43.073051 [I 2022-10-10 20:02:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 150/2304 [I 2022-10-10 20:02:13] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:31:23 [I 2022-10-10 20:02:13] ...trap/engines/engine.py.229: process: 0.88468 | load: 0.00306 [I 2022-10-10 20:02:13] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-10 20:02:13] ...optimizers/trijoint.py.151: total_norm: 15.881549 [I 2022-10-10 20:02:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 160/2304 [I 2022-10-10 20:02:21] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:30:50 [I 2022-10-10 20:02:21] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00342 [I 2022-10-10 20:02:21] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-10 20:02:21] ...optimizers/trijoint.py.151: total_norm: 38.917019 [I 2022-10-10 20:02:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 170/2304 [I 2022-10-10 20:02:30] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:30:46 [I 2022-10-10 20:02:30] ...trap/engines/engine.py.229: process: 0.86745 | load: 0.00310 [I 2022-10-10 20:02:30] ...trap/engines/engine.py.230: loss: 0.14270 [I 2022-10-10 20:02:30] ...optimizers/trijoint.py.151: total_norm: 29.496536 [I 2022-10-10 20:02:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 180/2304 [I 2022-10-10 20:02:39] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:30:34 [I 2022-10-10 20:02:39] ...trap/engines/engine.py.229: process: 0.86121 | load: 0.00324 [I 2022-10-10 20:02:39] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-10 20:02:39] ...optimizers/trijoint.py.151: total_norm: 10.339775 [I 2022-10-10 20:02:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 190/2304 [I 2022-10-10 20:02:47] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:30:21 [I 2022-10-10 20:02:47] ...trap/engines/engine.py.229: process: 0.86010 | load: 0.00314 [I 2022-10-10 20:02:47] ...trap/engines/engine.py.230: loss: 0.13390 [I 2022-10-10 20:02:47] ...optimizers/trijoint.py.151: total_norm: 11.167531 [I 2022-10-10 20:02:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 200/2304 [I 2022-10-10 20:02:56] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:30:23 [I 2022-10-10 20:02:56] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00337 [I 2022-10-10 20:02:56] ...trap/engines/engine.py.230: loss: 0.12794 [I 2022-10-10 20:02:56] ...optimizers/trijoint.py.151: total_norm: 19.646168 [I 2022-10-10 20:03:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 210/2304 [I 2022-10-10 20:03:05] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:30:24 [I 2022-10-10 20:03:05] ...trap/engines/engine.py.229: process: 0.88411 | load: 0.00310 [I 2022-10-10 20:03:05] ...trap/engines/engine.py.230: loss: 0.14990 [I 2022-10-10 20:03:05] ...optimizers/trijoint.py.151: total_norm: 12.391153 [I 2022-10-10 20:03:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 220/2304 [I 2022-10-10 20:03:13] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:29:58 [I 2022-10-10 20:03:13] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00341 [I 2022-10-10 20:03:13] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-10 20:03:13] ...optimizers/trijoint.py.151: total_norm: 35.475868 [I 2022-10-10 20:03:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 230/2304 [I 2022-10-10 20:03:22] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:29:43 [I 2022-10-10 20:03:22] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00312 [I 2022-10-10 20:03:22] ...trap/engines/engine.py.230: loss: 0.08755 [I 2022-10-10 20:03:22] ...optimizers/trijoint.py.151: total_norm: 7.281729 [I 2022-10-10 20:03:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 240/2304 [I 2022-10-10 20:03:31] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:29:37 [I 2022-10-10 20:03:31] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00321 [I 2022-10-10 20:03:31] ...trap/engines/engine.py.230: loss: 0.16278 [I 2022-10-10 20:03:31] ...optimizers/trijoint.py.151: total_norm: 19.526842 [I 2022-10-10 20:03:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 250/2304 [I 2022-10-10 20:03:39] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:29:16 [I 2022-10-10 20:03:39] ...trap/engines/engine.py.229: process: 0.83961 | load: 0.00293 [I 2022-10-10 20:03:39] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-10 20:03:39] ...optimizers/trijoint.py.151: total_norm: 8.319984 [I 2022-10-10 20:03:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 260/2304 [I 2022-10-10 20:03:48] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:29:20 [I 2022-10-10 20:03:48] ...trap/engines/engine.py.229: process: 0.86949 | load: 0.00307 [I 2022-10-10 20:03:48] ...trap/engines/engine.py.230: loss: 0.08392 [I 2022-10-10 20:03:48] ...optimizers/trijoint.py.151: total_norm: 10.254792 [I 2022-10-10 20:03:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 270/2304 [I 2022-10-10 20:03:56] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:58 [I 2022-10-10 20:03:56] ...trap/engines/engine.py.229: process: 0.85523 | load: 0.00307 [I 2022-10-10 20:03:56] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-10 20:03:56] ...optimizers/trijoint.py.151: total_norm: 8.717777 [I 2022-10-10 20:04:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 280/2304 [I 2022-10-10 20:04:05] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:52 [I 2022-10-10 20:04:05] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00308 [I 2022-10-10 20:04:05] ...trap/engines/engine.py.230: loss: 0.10092 [I 2022-10-10 20:04:05] ...optimizers/trijoint.py.151: total_norm: 10.507729 [I 2022-10-10 20:04:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 290/2304 [I 2022-10-10 20:04:14] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:53 [I 2022-10-10 20:04:14] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00302 [I 2022-10-10 20:04:14] ...trap/engines/engine.py.230: loss: 0.14931 [I 2022-10-10 20:04:14] ...optimizers/trijoint.py.151: total_norm: 24.880255 [I 2022-10-10 20:04:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 300/2304 [I 2022-10-10 20:04:22] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:28:36 [I 2022-10-10 20:04:22] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00316 [I 2022-10-10 20:04:22] ...trap/engines/engine.py.230: loss: 0.13120 [I 2022-10-10 20:04:22] ...optimizers/trijoint.py.151: total_norm: 14.524091 [I 2022-10-10 20:04:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 310/2304 [I 2022-10-10 20:04:31] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:28:27 [I 2022-10-10 20:04:31] ...trap/engines/engine.py.229: process: 0.85571 | load: 0.00307 [I 2022-10-10 20:04:31] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-10 20:04:31] ...optimizers/trijoint.py.151: total_norm: 13.268947 [I 2022-10-10 20:04:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 320/2304 [I 2022-10-10 20:04:40] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:28:23 [I 2022-10-10 20:04:40] ...trap/engines/engine.py.229: process: 0.86771 | load: 0.00299 [I 2022-10-10 20:04:40] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-10 20:04:40] ...optimizers/trijoint.py.151: total_norm: 11.485244 [I 2022-10-10 20:04:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 330/2304 [I 2022-10-10 20:04:48] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:28:04 [I 2022-10-10 20:04:48] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00303 [I 2022-10-10 20:04:48] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-10 20:04:48] ...optimizers/trijoint.py.151: total_norm: 13.307822 [I 2022-10-10 20:04:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 340/2304 [I 2022-10-10 20:04:57] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:28:07 [I 2022-10-10 20:04:57] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00318 [I 2022-10-10 20:04:57] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-10 20:04:57] ...optimizers/trijoint.py.151: total_norm: 14.879920 [I 2022-10-10 20:05:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 350/2304 [I 2022-10-10 20:05:05] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:27:53 [I 2022-10-10 20:05:05] ...trap/engines/engine.py.229: process: 0.85840 | load: 0.00301 [I 2022-10-10 20:05:05] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-10 20:05:05] ...optimizers/trijoint.py.151: total_norm: 19.636511 [I 2022-10-10 20:05:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 360/2304 [I 2022-10-10 20:05:14] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:50 [I 2022-10-10 20:05:14] ...trap/engines/engine.py.229: process: 0.86279 | load: 0.00299 [I 2022-10-10 20:05:14] ...trap/engines/engine.py.230: loss: 0.09231 [I 2022-10-10 20:05:14] ...optimizers/trijoint.py.151: total_norm: 17.355740 [I 2022-10-10 20:05:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 370/2304 [I 2022-10-10 20:05:23] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:27:32 [I 2022-10-10 20:05:23] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00298 [I 2022-10-10 20:05:23] ...trap/engines/engine.py.230: loss: 0.14995 [I 2022-10-10 20:05:23] ...optimizers/trijoint.py.151: total_norm: 12.331074 [I 2022-10-10 20:05:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 380/2304 [I 2022-10-10 20:05:31] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:27:48 [I 2022-10-10 20:05:31] ...trap/engines/engine.py.229: process: 0.88307 | load: 0.00313 [I 2022-10-10 20:05:31] ...trap/engines/engine.py.230: loss: 0.10225 [I 2022-10-10 20:05:31] ...optimizers/trijoint.py.151: total_norm: 14.189363 [I 2022-10-10 20:05:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 390/2304 [I 2022-10-10 20:05:40] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:27:32 [I 2022-10-10 20:05:40] ...trap/engines/engine.py.229: process: 0.86424 | load: 0.00308 [I 2022-10-10 20:05:40] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-10 20:05:40] ...optimizers/trijoint.py.151: total_norm: 19.152655 [I 2022-10-10 20:05:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 400/2304 [I 2022-10-10 20:05:48] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:27:19 [I 2022-10-10 20:05:48] ...trap/engines/engine.py.229: process: 0.85087 | load: 0.00322 [I 2022-10-10 20:05:48] ...trap/engines/engine.py.230: loss: 0.11582 [I 2022-10-10 20:05:48] ...optimizers/trijoint.py.151: total_norm: 7.984203 [I 2022-10-10 20:05:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 410/2304 [I 2022-10-10 20:05:57] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:27:20 [I 2022-10-10 20:05:57] ...trap/engines/engine.py.229: process: 0.86608 | load: 0.00314 [I 2022-10-10 20:05:57] ...trap/engines/engine.py.230: loss: 0.08940 [I 2022-10-10 20:05:57] ...optimizers/trijoint.py.151: total_norm: 9.952362 [I 2022-10-10 20:06:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 420/2304 [I 2022-10-10 20:06:06] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:27:12 [I 2022-10-10 20:06:06] ...trap/engines/engine.py.229: process: 0.87831 | load: 0.00314 [I 2022-10-10 20:06:06] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-10 20:06:06] ...optimizers/trijoint.py.151: total_norm: 33.712914 [I 2022-10-10 20:06:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 430/2304 [I 2022-10-10 20:06:15] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:27:02 [I 2022-10-10 20:06:15] ...trap/engines/engine.py.229: process: 0.87168 | load: 0.00330 [I 2022-10-10 20:06:15] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-10 20:06:15] ...optimizers/trijoint.py.151: total_norm: 17.672888 [I 2022-10-10 20:06:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 440/2304 [I 2022-10-10 20:06:23] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:26:51 [I 2022-10-10 20:06:23] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00315 [I 2022-10-10 20:06:23] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-10 20:06:23] ...optimizers/trijoint.py.151: total_norm: 13.822083 [I 2022-10-10 20:06:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 450/2304 [I 2022-10-10 20:06:32] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:26:53 [I 2022-10-10 20:06:32] ...trap/engines/engine.py.229: process: 0.87832 | load: 0.00308 [I 2022-10-10 20:06:32] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-10 20:06:32] ...optimizers/trijoint.py.151: total_norm: 18.218920 [I 2022-10-10 20:06:41] ...trap/engines/engine.py.224: train: epoch 84 | batch 460/2304 [I 2022-10-10 20:06:41] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:26:32 [I 2022-10-10 20:06:41] ...trap/engines/engine.py.229: process: 0.86476 | load: 0.00341 [I 2022-10-10 20:06:41] ...trap/engines/engine.py.230: loss: 0.13226 [I 2022-10-10 20:06:41] ...optimizers/trijoint.py.151: total_norm: 9.440448 [I 2022-10-10 20:06:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 470/2304 [I 2022-10-10 20:06:49] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:26:23 [I 2022-10-10 20:06:49] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00310 [I 2022-10-10 20:06:49] ...trap/engines/engine.py.230: loss: 0.09950 [I 2022-10-10 20:06:49] ...optimizers/trijoint.py.151: total_norm: 13.844796 [I 2022-10-10 20:06:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 480/2304 [I 2022-10-10 20:06:58] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:26:15 [I 2022-10-10 20:06:58] ...trap/engines/engine.py.229: process: 0.86006 | load: 0.00313 [I 2022-10-10 20:06:58] ...trap/engines/engine.py.230: loss: 0.19904 [I 2022-10-10 20:06:58] ...optimizers/trijoint.py.151: total_norm: 14.231315 [I 2022-10-10 20:07:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 490/2304 [I 2022-10-10 20:07:06] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:59 [I 2022-10-10 20:07:06] ...trap/engines/engine.py.229: process: 0.87249 | load: 0.00321 [I 2022-10-10 20:07:06] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-10 20:07:06] ...optimizers/trijoint.py.151: total_norm: 13.598265 [I 2022-10-10 20:07:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 500/2304 [I 2022-10-10 20:07:15] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:50 [I 2022-10-10 20:07:15] ...trap/engines/engine.py.229: process: 0.86822 | load: 0.00310 [I 2022-10-10 20:07:15] ...trap/engines/engine.py.230: loss: 0.12195 [I 2022-10-10 20:07:15] ...optimizers/trijoint.py.151: total_norm: 17.695324 [I 2022-10-10 20:07:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 510/2304 [I 2022-10-10 20:07:24] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:49 [I 2022-10-10 20:07:24] ...trap/engines/engine.py.229: process: 0.86598 | load: 0.00314 [I 2022-10-10 20:07:24] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-10 20:07:24] ...optimizers/trijoint.py.151: total_norm: 27.880362 [I 2022-10-10 20:07:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 520/2304 [I 2022-10-10 20:07:32] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:44 [I 2022-10-10 20:07:32] ...trap/engines/engine.py.229: process: 0.86737 | load: 0.00320 [I 2022-10-10 20:07:32] ...trap/engines/engine.py.230: loss: 0.10040 [I 2022-10-10 20:07:32] ...optimizers/trijoint.py.151: total_norm: 9.512460 [I 2022-10-10 20:07:41] ...trap/engines/engine.py.224: train: epoch 84 | batch 530/2304 [I 2022-10-10 20:07:41] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:25:37 [I 2022-10-10 20:07:41] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00310 [I 2022-10-10 20:07:41] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-10 20:07:41] ...optimizers/trijoint.py.151: total_norm: 19.518354 [I 2022-10-10 20:07:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 540/2304 [I 2022-10-10 20:07:50] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:25:31 [I 2022-10-10 20:07:50] ...trap/engines/engine.py.229: process: 0.88367 | load: 0.00309 [I 2022-10-10 20:07:50] ...trap/engines/engine.py.230: loss: 0.23252 [I 2022-10-10 20:07:50] ...optimizers/trijoint.py.151: total_norm: 15.374736 [I 2022-10-10 20:07:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 550/2304 [I 2022-10-10 20:07:59] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:25:16 [I 2022-10-10 20:07:59] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00314 [I 2022-10-10 20:07:59] ...trap/engines/engine.py.230: loss: 0.10616 [I 2022-10-10 20:07:59] ...optimizers/trijoint.py.151: total_norm: 12.599692 [I 2022-10-10 20:08:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 560/2304 [I 2022-10-10 20:08:07] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:25:12 [I 2022-10-10 20:08:07] ...trap/engines/engine.py.229: process: 0.87471 | load: 0.00323 [I 2022-10-10 20:08:07] ...trap/engines/engine.py.230: loss: 0.17107 [I 2022-10-10 20:08:07] ...optimizers/trijoint.py.151: total_norm: 9.761729 [I 2022-10-10 20:08:16] ...trap/engines/engine.py.224: train: epoch 84 | batch 570/2304 [I 2022-10-10 20:08:16] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:25:06 [I 2022-10-10 20:08:16] ...trap/engines/engine.py.229: process: 0.88260 | load: 0.00307 [I 2022-10-10 20:08:16] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-10 20:08:16] ...optimizers/trijoint.py.151: total_norm: 13.298591 [I 2022-10-10 20:08:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 580/2304 [I 2022-10-10 20:08:25] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:24:52 [I 2022-10-10 20:08:25] ...trap/engines/engine.py.229: process: 0.87462 | load: 0.00321 [I 2022-10-10 20:08:25] ...trap/engines/engine.py.230: loss: 0.10050 [I 2022-10-10 20:08:25] ...optimizers/trijoint.py.151: total_norm: 12.983718 [I 2022-10-10 20:08:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 590/2304 [I 2022-10-10 20:08:33] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:34 [I 2022-10-10 20:08:33] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00310 [I 2022-10-10 20:08:33] ...trap/engines/engine.py.230: loss: 0.09463 [I 2022-10-10 20:08:33] ...optimizers/trijoint.py.151: total_norm: 15.634156 [I 2022-10-10 20:08:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 600/2304 [I 2022-10-10 20:08:42] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:24:37 [I 2022-10-10 20:08:42] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00310 [I 2022-10-10 20:08:42] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-10 20:08:42] ...optimizers/trijoint.py.151: total_norm: 16.032856 [I 2022-10-10 20:08:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 610/2304 [I 2022-10-10 20:08:51] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:24:27 [I 2022-10-10 20:08:51] ...trap/engines/engine.py.229: process: 0.87525 | load: 0.00315 [I 2022-10-10 20:08:51] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-10 20:08:51] ...optimizers/trijoint.py.151: total_norm: 10.871266 [I 2022-10-10 20:08:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 620/2304 [I 2022-10-10 20:08:59] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:24:22 [I 2022-10-10 20:08:59] ...trap/engines/engine.py.229: process: 0.87838 | load: 0.00319 [I 2022-10-10 20:08:59] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-10 20:08:59] ...optimizers/trijoint.py.151: total_norm: 31.063993 [I 2022-10-10 20:09:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 630/2304 [I 2022-10-10 20:09:08] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:24:10 [I 2022-10-10 20:09:08] ...trap/engines/engine.py.229: process: 0.87447 | load: 0.00312 [I 2022-10-10 20:09:08] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-10 20:09:08] ...optimizers/trijoint.py.151: total_norm: 15.537148 [I 2022-10-10 20:09:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 640/2304 [I 2022-10-10 20:09:17] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:24:00 [I 2022-10-10 20:09:17] ...trap/engines/engine.py.229: process: 0.87026 | load: 0.00318 [I 2022-10-10 20:09:17] ...trap/engines/engine.py.230: loss: 0.15048 [I 2022-10-10 20:09:17] ...optimizers/trijoint.py.151: total_norm: 26.479136 [I 2022-10-10 20:09:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 650/2304 [I 2022-10-10 20:09:25] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:38 [I 2022-10-10 20:09:25] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00294 [I 2022-10-10 20:09:25] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-10 20:09:25] ...optimizers/trijoint.py.151: total_norm: 19.600168 [I 2022-10-10 20:09:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 660/2304 [I 2022-10-10 20:09:34] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:23:28 [I 2022-10-10 20:09:34] ...trap/engines/engine.py.229: process: 0.85605 | load: 0.00308 [I 2022-10-10 20:09:34] ...trap/engines/engine.py.230: loss: 0.13662 [I 2022-10-10 20:09:34] ...optimizers/trijoint.py.151: total_norm: 16.429535 [I 2022-10-10 20:09:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 670/2304 [I 2022-10-10 20:09:42] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:23:26 [I 2022-10-10 20:09:42] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00298 [I 2022-10-10 20:09:42] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-10 20:09:42] ...optimizers/trijoint.py.151: total_norm: 15.381038 [I 2022-10-10 20:09:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 680/2304 [I 2022-10-10 20:09:51] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:23:10 [I 2022-10-10 20:09:51] ...trap/engines/engine.py.229: process: 0.83527 | load: 0.00303 [I 2022-10-10 20:09:51] ...trap/engines/engine.py.230: loss: 0.09449 [I 2022-10-10 20:09:51] ...optimizers/trijoint.py.151: total_norm: 16.623867 [I 2022-10-10 20:10:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 690/2304 [I 2022-10-10 20:10:00] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:23:05 [I 2022-10-10 20:10:00] ...trap/engines/engine.py.229: process: 0.86543 | load: 0.00300 [I 2022-10-10 20:10:00] ...trap/engines/engine.py.230: loss: 0.15689 [I 2022-10-10 20:10:00] ...optimizers/trijoint.py.151: total_norm: 12.699036 [I 2022-10-10 20:10:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 700/2304 [I 2022-10-10 20:10:08] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:58 [I 2022-10-10 20:10:08] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00300 [I 2022-10-10 20:10:08] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-10 20:10:08] ...optimizers/trijoint.py.151: total_norm: 18.422501 [I 2022-10-10 20:10:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 710/2304 [I 2022-10-10 20:10:17] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:22:39 [I 2022-10-10 20:10:17] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00297 [I 2022-10-10 20:10:17] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-10 20:10:17] ...optimizers/trijoint.py.151: total_norm: 12.299964 [I 2022-10-10 20:10:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 720/2304 [I 2022-10-10 20:10:25] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:22:42 [I 2022-10-10 20:10:25] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00296 [I 2022-10-10 20:10:25] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-10 20:10:25] ...optimizers/trijoint.py.151: total_norm: 23.116894 [I 2022-10-10 20:10:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 730/2304 [I 2022-10-10 20:10:34] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:22:27 [I 2022-10-10 20:10:34] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00311 [I 2022-10-10 20:10:34] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-10 20:10:34] ...optimizers/trijoint.py.151: total_norm: 9.124196 [I 2022-10-10 20:10:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 740/2304 [I 2022-10-10 20:10:43] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:22:26 [I 2022-10-10 20:10:43] ...trap/engines/engine.py.229: process: 0.86299 | load: 0.00303 [I 2022-10-10 20:10:43] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-10 20:10:43] ...optimizers/trijoint.py.151: total_norm: 27.091200 [I 2022-10-10 20:10:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 750/2304 [I 2022-10-10 20:10:51] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:22:08 [I 2022-10-10 20:10:51] ...trap/engines/engine.py.229: process: 0.86730 | load: 0.00297 [I 2022-10-10 20:10:51] ...trap/engines/engine.py.230: loss: 0.18119 [I 2022-10-10 20:10:51] ...optimizers/trijoint.py.151: total_norm: 23.133490 [I 2022-10-10 20:11:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 760/2304 [I 2022-10-10 20:11:00] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:22:07 [I 2022-10-10 20:11:00] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00297 [I 2022-10-10 20:11:00] ...trap/engines/engine.py.230: loss: 0.10828 [I 2022-10-10 20:11:00] ...optimizers/trijoint.py.151: total_norm: 16.419462 [I 2022-10-10 20:11:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 770/2304 [I 2022-10-10 20:11:08] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:21:59 [I 2022-10-10 20:11:08] ...trap/engines/engine.py.229: process: 0.86379 | load: 0.00300 [I 2022-10-10 20:11:08] ...trap/engines/engine.py.230: loss: 0.17952 [I 2022-10-10 20:11:08] ...optimizers/trijoint.py.151: total_norm: 15.812387 [I 2022-10-10 20:11:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 780/2304 [I 2022-10-10 20:11:17] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:21:56 [I 2022-10-10 20:11:17] ...trap/engines/engine.py.229: process: 0.87342 | load: 0.00308 [I 2022-10-10 20:11:17] ...trap/engines/engine.py.230: loss: 0.21418 [I 2022-10-10 20:11:17] ...optimizers/trijoint.py.151: total_norm: 37.297359 [I 2022-10-10 20:11:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 790/2304 [I 2022-10-10 20:11:26] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:21:41 [I 2022-10-10 20:11:26] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00328 [I 2022-10-10 20:11:26] ...trap/engines/engine.py.230: loss: 0.11754 [I 2022-10-10 20:11:26] ...optimizers/trijoint.py.151: total_norm: 25.578268 [I 2022-10-10 20:11:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 800/2304 [I 2022-10-10 20:11:34] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:21:44 [I 2022-10-10 20:11:34] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00309 [I 2022-10-10 20:11:34] ...trap/engines/engine.py.230: loss: 0.09403 [I 2022-10-10 20:11:34] ...optimizers/trijoint.py.151: total_norm: 20.864990 [I 2022-10-10 20:11:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 810/2304 [I 2022-10-10 20:11:43] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:21:26 [I 2022-10-10 20:11:43] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00308 [I 2022-10-10 20:11:43] ...trap/engines/engine.py.230: loss: 0.11070 [I 2022-10-10 20:11:43] ...optimizers/trijoint.py.151: total_norm: 17.212259 [I 2022-10-10 20:11:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 820/2304 [I 2022-10-10 20:11:52] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:21:22 [I 2022-10-10 20:11:52] ...trap/engines/engine.py.229: process: 0.85836 | load: 0.00307 [I 2022-10-10 20:11:52] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-10 20:11:52] ...optimizers/trijoint.py.151: total_norm: 10.514809 [I 2022-10-10 20:12:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 830/2304 [I 2022-10-10 20:12:00] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:21:11 [I 2022-10-10 20:12:00] ...trap/engines/engine.py.229: process: 0.86308 | load: 0.00316 [I 2022-10-10 20:12:00] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-10 20:12:00] ...optimizers/trijoint.py.151: total_norm: 19.436668 [I 2022-10-10 20:12:09] ...trap/engines/engine.py.224: train: epoch 84 | batch 840/2304 [I 2022-10-10 20:12:09] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:21:04 [I 2022-10-10 20:12:09] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00317 [I 2022-10-10 20:12:09] ...trap/engines/engine.py.230: loss: 0.09849 [I 2022-10-10 20:12:09] ...optimizers/trijoint.py.151: total_norm: 15.072657 [I 2022-10-10 20:12:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 850/2304 [I 2022-10-10 20:12:18] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:20:56 [I 2022-10-10 20:12:18] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00323 [I 2022-10-10 20:12:18] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-10 20:12:18] ...optimizers/trijoint.py.151: total_norm: 19.592159 [I 2022-10-10 20:12:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 860/2304 [I 2022-10-10 20:12:26] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:20:42 [I 2022-10-10 20:12:26] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00314 [I 2022-10-10 20:12:26] ...trap/engines/engine.py.230: loss: 0.17072 [I 2022-10-10 20:12:26] ...optimizers/trijoint.py.151: total_norm: 13.513611 [I 2022-10-10 20:12:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 870/2304 [I 2022-10-10 20:12:35] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:20:46 [I 2022-10-10 20:12:35] ...trap/engines/engine.py.229: process: 0.87092 | load: 0.00315 [I 2022-10-10 20:12:35] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-10 20:12:35] ...optimizers/trijoint.py.151: total_norm: 20.912992 [I 2022-10-10 20:12:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 880/2304 [I 2022-10-10 20:12:44] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:20:33 [I 2022-10-10 20:12:44] ...trap/engines/engine.py.229: process: 0.86513 | load: 0.00321 [I 2022-10-10 20:12:44] ...trap/engines/engine.py.230: loss: 0.11994 [I 2022-10-10 20:12:44] ...optimizers/trijoint.py.151: total_norm: 17.589657 [I 2022-10-10 20:12:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 890/2304 [I 2022-10-10 20:12:52] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:20:28 [I 2022-10-10 20:12:52] ...trap/engines/engine.py.229: process: 0.87071 | load: 0.00323 [I 2022-10-10 20:12:52] ...trap/engines/engine.py.230: loss: 0.11890 [I 2022-10-10 20:12:52] ...optimizers/trijoint.py.151: total_norm: 19.206839 [I 2022-10-10 20:13:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 900/2304 [I 2022-10-10 20:13:01] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:20:18 [I 2022-10-10 20:13:01] ...trap/engines/engine.py.229: process: 0.87599 | load: 0.00308 [I 2022-10-10 20:13:01] ...trap/engines/engine.py.230: loss: 0.16467 [I 2022-10-10 20:13:01] ...optimizers/trijoint.py.151: total_norm: 15.237606 [I 2022-10-10 20:13:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 910/2304 [I 2022-10-10 20:13:10] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:20:05 [I 2022-10-10 20:13:10] ...trap/engines/engine.py.229: process: 0.87262 | load: 0.00331 [I 2022-10-10 20:13:10] ...trap/engines/engine.py.230: loss: 0.12344 [I 2022-10-10 20:13:10] ...optimizers/trijoint.py.151: total_norm: 25.383286 [I 2022-10-10 20:13:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 920/2304 [I 2022-10-10 20:13:18] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:19:55 [I 2022-10-10 20:13:18] ...trap/engines/engine.py.229: process: 0.87252 | load: 0.00312 [I 2022-10-10 20:13:18] ...trap/engines/engine.py.230: loss: 0.12730 [I 2022-10-10 20:13:18] ...optimizers/trijoint.py.151: total_norm: 14.852855 [I 2022-10-10 20:13:27] ...trap/engines/engine.py.224: train: epoch 84 | batch 930/2304 [I 2022-10-10 20:13:27] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:19:46 [I 2022-10-10 20:13:27] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00307 [I 2022-10-10 20:13:27] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-10 20:13:27] ...optimizers/trijoint.py.151: total_norm: 13.910647 [I 2022-10-10 20:13:36] ...trap/engines/engine.py.224: train: epoch 84 | batch 940/2304 [I 2022-10-10 20:13:36] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:19:33 [I 2022-10-10 20:13:36] ...trap/engines/engine.py.229: process: 0.85626 | load: 0.00308 [I 2022-10-10 20:13:36] ...trap/engines/engine.py.230: loss: 0.10778 [I 2022-10-10 20:13:36] ...optimizers/trijoint.py.151: total_norm: 40.920288 [I 2022-10-10 20:13:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 950/2304 [I 2022-10-10 20:13:44] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:19:31 [I 2022-10-10 20:13:44] ...trap/engines/engine.py.229: process: 0.85823 | load: 0.00315 [I 2022-10-10 20:13:44] ...trap/engines/engine.py.230: loss: 0.09230 [I 2022-10-10 20:13:44] ...optimizers/trijoint.py.151: total_norm: 7.880668 [I 2022-10-10 20:13:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 960/2304 [I 2022-10-10 20:13:53] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:19:26 [I 2022-10-10 20:13:53] ...trap/engines/engine.py.229: process: 0.86803 | load: 0.00306 [I 2022-10-10 20:13:53] ...trap/engines/engine.py.230: loss: 0.10370 [I 2022-10-10 20:13:53] ...optimizers/trijoint.py.151: total_norm: 8.615988 [I 2022-10-10 20:14:02] ...trap/engines/engine.py.224: train: epoch 84 | batch 970/2304 [I 2022-10-10 20:14:02] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:19:17 [I 2022-10-10 20:14:02] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00312 [I 2022-10-10 20:14:02] ...trap/engines/engine.py.230: loss: 0.11581 [I 2022-10-10 20:14:02] ...optimizers/trijoint.py.151: total_norm: 11.247701 [I 2022-10-10 20:14:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 980/2304 [I 2022-10-10 20:14:11] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:19:12 [I 2022-10-10 20:14:11] ...trap/engines/engine.py.229: process: 0.86634 | load: 0.00314 [I 2022-10-10 20:14:11] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-10 20:14:11] ...optimizers/trijoint.py.151: total_norm: 17.402729 [I 2022-10-10 20:14:19] ...trap/engines/engine.py.224: train: epoch 84 | batch 990/2304 [I 2022-10-10 20:14:19] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:18:59 [I 2022-10-10 20:14:19] ...trap/engines/engine.py.229: process: 0.87004 | load: 0.00316 [I 2022-10-10 20:14:19] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-10 20:14:19] ...optimizers/trijoint.py.151: total_norm: 12.279551 [I 2022-10-10 20:14:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 1000/2304 [I 2022-10-10 20:14:28] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:18:53 [I 2022-10-10 20:14:28] ...trap/engines/engine.py.229: process: 0.87851 | load: 0.00311 [I 2022-10-10 20:14:28] ...trap/engines/engine.py.230: loss: 0.13654 [I 2022-10-10 20:14:28] ...optimizers/trijoint.py.151: total_norm: 27.724331 [I 2022-10-10 20:14:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 1010/2304 [I 2022-10-10 20:14:37] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:18:42 [I 2022-10-10 20:14:37] ...trap/engines/engine.py.229: process: 0.86349 | load: 0.00316 [I 2022-10-10 20:14:37] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-10 20:14:37] ...optimizers/trijoint.py.151: total_norm: 9.388271 [I 2022-10-10 20:14:45] ...trap/engines/engine.py.224: train: epoch 84 | batch 1020/2304 [I 2022-10-10 20:14:45] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:18:36 [I 2022-10-10 20:14:45] ...trap/engines/engine.py.229: process: 0.86328 | load: 0.00318 [I 2022-10-10 20:14:45] ...trap/engines/engine.py.230: loss: 0.12573 [I 2022-10-10 20:14:45] ...optimizers/trijoint.py.151: total_norm: 12.007594 [I 2022-10-10 20:14:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 1030/2304 [I 2022-10-10 20:14:54] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:18:23 [I 2022-10-10 20:14:54] ...trap/engines/engine.py.229: process: 0.86763 | load: 0.00309 [I 2022-10-10 20:14:54] ...trap/engines/engine.py.230: loss: 0.19309 [I 2022-10-10 20:14:54] ...optimizers/trijoint.py.151: total_norm: 19.729006 [I 2022-10-10 20:15:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 1040/2304 [I 2022-10-10 20:15:03] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:18:13 [I 2022-10-10 20:15:03] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00327 [I 2022-10-10 20:15:03] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-10 20:15:03] ...optimizers/trijoint.py.151: total_norm: 23.177900 [I 2022-10-10 20:15:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 1050/2304 [I 2022-10-10 20:15:11] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:17:49 [I 2022-10-10 20:15:11] ...trap/engines/engine.py.229: process: 0.83713 | load: 0.00301 [I 2022-10-10 20:15:11] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-10 20:15:11] ...optimizers/trijoint.py.151: total_norm: 32.406128 [I 2022-10-10 20:15:20] ...trap/engines/engine.py.224: train: epoch 84 | batch 1060/2304 [I 2022-10-10 20:15:20] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:44 [I 2022-10-10 20:15:20] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00305 [I 2022-10-10 20:15:20] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-10 20:15:20] ...optimizers/trijoint.py.151: total_norm: 14.072784 [I 2022-10-10 20:15:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 1070/2304 [I 2022-10-10 20:15:28] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:17:37 [I 2022-10-10 20:15:28] ...trap/engines/engine.py.229: process: 0.85983 | load: 0.00301 [I 2022-10-10 20:15:28] ...trap/engines/engine.py.230: loss: 0.15772 [I 2022-10-10 20:15:28] ...optimizers/trijoint.py.151: total_norm: 13.877773 [I 2022-10-10 20:15:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 1080/2304 [I 2022-10-10 20:15:37] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:17:30 [I 2022-10-10 20:15:37] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00300 [I 2022-10-10 20:15:37] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-10 20:15:37] ...optimizers/trijoint.py.151: total_norm: 16.381630 [I 2022-10-10 20:15:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1090/2304 [I 2022-10-10 20:15:46] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:17:27 [I 2022-10-10 20:15:46] ...trap/engines/engine.py.229: process: 0.88201 | load: 0.00302 [I 2022-10-10 20:15:46] ...trap/engines/engine.py.230: loss: 0.15030 [I 2022-10-10 20:15:46] ...optimizers/trijoint.py.151: total_norm: 22.350939 [I 2022-10-10 20:15:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 1100/2304 [I 2022-10-10 20:15:54] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:17:15 [I 2022-10-10 20:15:54] ...trap/engines/engine.py.229: process: 0.85249 | load: 0.00316 [I 2022-10-10 20:15:54] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-10 20:15:54] ...optimizers/trijoint.py.151: total_norm: 11.936569 [I 2022-10-10 20:16:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 1110/2304 [I 2022-10-10 20:16:03] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:17:04 [I 2022-10-10 20:16:03] ...trap/engines/engine.py.229: process: 0.85713 | load: 0.00315 [I 2022-10-10 20:16:03] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-10 20:16:03] ...optimizers/trijoint.py.151: total_norm: 10.504696 [I 2022-10-10 20:16:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 1120/2304 [I 2022-10-10 20:16:11] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:54 [I 2022-10-10 20:16:11] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00309 [I 2022-10-10 20:16:11] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-10 20:16:11] ...optimizers/trijoint.py.151: total_norm: 12.150642 [I 2022-10-10 20:16:20] ...trap/engines/engine.py.224: train: epoch 84 | batch 1130/2304 [I 2022-10-10 20:16:20] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:16:41 [I 2022-10-10 20:16:20] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00302 [I 2022-10-10 20:16:20] ...trap/engines/engine.py.230: loss: 0.17716 [I 2022-10-10 20:16:20] ...optimizers/trijoint.py.151: total_norm: 11.034378 [I 2022-10-10 20:16:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 1140/2304 [I 2022-10-10 20:16:29] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:16:35 [I 2022-10-10 20:16:29] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00301 [I 2022-10-10 20:16:29] ...trap/engines/engine.py.230: loss: 0.17600 [I 2022-10-10 20:16:29] ...optimizers/trijoint.py.151: total_norm: 11.071196 [I 2022-10-10 20:16:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 1150/2304 [I 2022-10-10 20:16:37] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:16:29 [I 2022-10-10 20:16:37] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00297 [I 2022-10-10 20:16:37] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-10 20:16:37] ...optimizers/trijoint.py.151: total_norm: 14.870860 [I 2022-10-10 20:16:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1160/2304 [I 2022-10-10 20:16:46] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:16:21 [I 2022-10-10 20:16:46] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00299 [I 2022-10-10 20:16:46] ...trap/engines/engine.py.230: loss: 0.11835 [I 2022-10-10 20:16:46] ...optimizers/trijoint.py.151: total_norm: 20.452606 [I 2022-10-10 20:16:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 1170/2304 [I 2022-10-10 20:16:54] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:16:12 [I 2022-10-10 20:16:54] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00307 [I 2022-10-10 20:16:54] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-10 20:16:54] ...optimizers/trijoint.py.151: total_norm: 34.480385 [I 2022-10-10 20:17:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 1180/2304 [I 2022-10-10 20:17:03] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:16:10 [I 2022-10-10 20:17:03] ...trap/engines/engine.py.229: process: 0.87071 | load: 0.00311 [I 2022-10-10 20:17:03] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-10 20:17:03] ...optimizers/trijoint.py.151: total_norm: 30.280783 [I 2022-10-10 20:17:12] ...trap/engines/engine.py.224: train: epoch 84 | batch 1190/2304 [I 2022-10-10 20:17:12] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:58 [I 2022-10-10 20:17:12] ...trap/engines/engine.py.229: process: 0.86439 | load: 0.00309 [I 2022-10-10 20:17:12] ...trap/engines/engine.py.230: loss: 0.14764 [I 2022-10-10 20:17:12] ...optimizers/trijoint.py.151: total_norm: 21.378059 [I 2022-10-10 20:17:20] ...trap/engines/engine.py.224: train: epoch 84 | batch 1200/2304 [I 2022-10-10 20:17:20] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:53 [I 2022-10-10 20:17:20] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00310 [I 2022-10-10 20:17:20] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-10 20:17:20] ...optimizers/trijoint.py.151: total_norm: 18.272982 [I 2022-10-10 20:17:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 1210/2304 [I 2022-10-10 20:17:29] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:41 [I 2022-10-10 20:17:29] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00317 [I 2022-10-10 20:17:29] ...trap/engines/engine.py.230: loss: 0.13766 [I 2022-10-10 20:17:29] ...optimizers/trijoint.py.151: total_norm: 13.549602 [I 2022-10-10 20:17:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 1220/2304 [I 2022-10-10 20:17:38] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:15:40 [I 2022-10-10 20:17:38] ...trap/engines/engine.py.229: process: 0.86605 | load: 0.00312 [I 2022-10-10 20:17:38] ...trap/engines/engine.py.230: loss: 0.08610 [I 2022-10-10 20:17:38] ...optimizers/trijoint.py.151: total_norm: 15.798982 [I 2022-10-10 20:17:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1230/2304 [I 2022-10-10 20:17:46] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:15:28 [I 2022-10-10 20:17:46] ...trap/engines/engine.py.229: process: 0.86348 | load: 0.00311 [I 2022-10-10 20:17:46] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-10 20:17:46] ...optimizers/trijoint.py.151: total_norm: 9.237964 [I 2022-10-10 20:17:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 1240/2304 [I 2022-10-10 20:17:55] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:15:11 [I 2022-10-10 20:17:55] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00313 [I 2022-10-10 20:17:55] ...trap/engines/engine.py.230: loss: 0.08551 [I 2022-10-10 20:17:55] ...optimizers/trijoint.py.151: total_norm: 8.291252 [I 2022-10-10 20:18:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 1250/2304 [I 2022-10-10 20:18:04] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:15:08 [I 2022-10-10 20:18:04] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00308 [I 2022-10-10 20:18:04] ...trap/engines/engine.py.230: loss: 0.13434 [I 2022-10-10 20:18:04] ...optimizers/trijoint.py.151: total_norm: 15.824509 [I 2022-10-10 20:18:12] ...trap/engines/engine.py.224: train: epoch 84 | batch 1260/2304 [I 2022-10-10 20:18:12] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:15:01 [I 2022-10-10 20:18:12] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00307 [I 2022-10-10 20:18:12] ...trap/engines/engine.py.230: loss: 0.20557 [I 2022-10-10 20:18:12] ...optimizers/trijoint.py.151: total_norm: 15.998453 [I 2022-10-10 20:18:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 1270/2304 [I 2022-10-10 20:18:21] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:54 [I 2022-10-10 20:18:21] ...trap/engines/engine.py.229: process: 0.87117 | load: 0.00308 [I 2022-10-10 20:18:21] ...trap/engines/engine.py.230: loss: 0.16088 [I 2022-10-10 20:18:21] ...optimizers/trijoint.py.151: total_norm: 21.090117 [I 2022-10-10 20:18:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 1280/2304 [I 2022-10-10 20:18:30] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:47 [I 2022-10-10 20:18:30] ...trap/engines/engine.py.229: process: 0.86448 | load: 0.00329 [I 2022-10-10 20:18:30] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-10 20:18:30] ...optimizers/trijoint.py.151: total_norm: 16.461184 [I 2022-10-10 20:18:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 1290/2304 [I 2022-10-10 20:18:38] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:14:40 [I 2022-10-10 20:18:38] ...trap/engines/engine.py.229: process: 0.86590 | load: 0.00310 [I 2022-10-10 20:18:38] ...trap/engines/engine.py.230: loss: 0.14687 [I 2022-10-10 20:18:38] ...optimizers/trijoint.py.151: total_norm: 19.424896 [I 2022-10-10 20:18:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 1300/2304 [I 2022-10-10 20:18:47] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:14:24 [I 2022-10-10 20:18:47] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00312 [I 2022-10-10 20:18:47] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-10 20:18:47] ...optimizers/trijoint.py.151: total_norm: 12.768139 [I 2022-10-10 20:18:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 1310/2304 [I 2022-10-10 20:18:56] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:14:21 [I 2022-10-10 20:18:56] ...trap/engines/engine.py.229: process: 0.87593 | load: 0.00312 [I 2022-10-10 20:18:56] ...trap/engines/engine.py.230: loss: 0.14421 [I 2022-10-10 20:18:56] ...optimizers/trijoint.py.151: total_norm: 18.193966 [I 2022-10-10 20:19:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 1320/2304 [I 2022-10-10 20:19:04] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:14:11 [I 2022-10-10 20:19:04] ...trap/engines/engine.py.229: process: 0.87393 | load: 0.00310 [I 2022-10-10 20:19:04] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-10 20:19:04] ...optimizers/trijoint.py.151: total_norm: 10.403552 [I 2022-10-10 20:19:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 1330/2304 [I 2022-10-10 20:19:13] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:14:03 [I 2022-10-10 20:19:13] ...trap/engines/engine.py.229: process: 0.87796 | load: 0.00308 [I 2022-10-10 20:19:13] ...trap/engines/engine.py.230: loss: 0.16017 [I 2022-10-10 20:19:13] ...optimizers/trijoint.py.151: total_norm: 101.722511 [I 2022-10-10 20:19:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 1340/2304 [I 2022-10-10 20:19:22] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:52 [I 2022-10-10 20:19:22] ...trap/engines/engine.py.229: process: 0.85744 | load: 0.00310 [I 2022-10-10 20:19:22] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-10 20:19:22] ...optimizers/trijoint.py.151: total_norm: 23.104834 [I 2022-10-10 20:19:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 1350/2304 [I 2022-10-10 20:19:30] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:13:40 [I 2022-10-10 20:19:30] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00317 [I 2022-10-10 20:19:30] ...trap/engines/engine.py.230: loss: 0.09197 [I 2022-10-10 20:19:30] ...optimizers/trijoint.py.151: total_norm: 12.646207 [I 2022-10-10 20:19:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 1360/2304 [I 2022-10-10 20:19:39] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:43 [I 2022-10-10 20:19:39] ...trap/engines/engine.py.229: process: 0.86770 | load: 0.00318 [I 2022-10-10 20:19:39] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-10 20:19:39] ...optimizers/trijoint.py.151: total_norm: 15.391111 [I 2022-10-10 20:19:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 1370/2304 [I 2022-10-10 20:19:48] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:13:28 [I 2022-10-10 20:19:48] ...trap/engines/engine.py.229: process: 0.86861 | load: 0.00319 [I 2022-10-10 20:19:48] ...trap/engines/engine.py.230: loss: 0.10573 [I 2022-10-10 20:19:48] ...optimizers/trijoint.py.151: total_norm: 18.353552 [I 2022-10-10 20:19:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1380/2304 [I 2022-10-10 20:19:57] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:13:18 [I 2022-10-10 20:19:57] ...trap/engines/engine.py.229: process: 0.88129 | load: 0.00324 [I 2022-10-10 20:19:57] ...trap/engines/engine.py.230: loss: 0.12397 [I 2022-10-10 20:19:57] ...optimizers/trijoint.py.151: total_norm: 26.111568 [I 2022-10-10 20:20:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 1390/2304 [I 2022-10-10 20:20:05] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:13:05 [I 2022-10-10 20:20:05] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00317 [I 2022-10-10 20:20:05] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-10 20:20:05] ...optimizers/trijoint.py.151: total_norm: 7.557496 [I 2022-10-10 20:20:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 1400/2304 [I 2022-10-10 20:20:14] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:13:02 [I 2022-10-10 20:20:14] ...trap/engines/engine.py.229: process: 0.86339 | load: 0.00313 [I 2022-10-10 20:20:14] ...trap/engines/engine.py.230: loss: 0.16944 [I 2022-10-10 20:20:14] ...optimizers/trijoint.py.151: total_norm: 17.460106 [I 2022-10-10 20:20:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1410/2304 [I 2022-10-10 20:20:23] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:12:50 [I 2022-10-10 20:20:23] ...trap/engines/engine.py.229: process: 0.86073 | load: 0.00309 [I 2022-10-10 20:20:23] ...trap/engines/engine.py.230: loss: 0.11180 [I 2022-10-10 20:20:23] ...optimizers/trijoint.py.151: total_norm: 14.544508 [I 2022-10-10 20:20:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 1420/2304 [I 2022-10-10 20:20:31] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:12:48 [I 2022-10-10 20:20:31] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00311 [I 2022-10-10 20:20:31] ...trap/engines/engine.py.230: loss: 0.14039 [I 2022-10-10 20:20:31] ...optimizers/trijoint.py.151: total_norm: 13.347262 [I 2022-10-10 20:20:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1430/2304 [I 2022-10-10 20:20:40] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:12:36 [I 2022-10-10 20:20:40] ...trap/engines/engine.py.229: process: 0.86787 | load: 0.00323 [I 2022-10-10 20:20:40] ...trap/engines/engine.py.230: loss: 0.14447 [I 2022-10-10 20:20:40] ...optimizers/trijoint.py.151: total_norm: 13.056523 [I 2022-10-10 20:20:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1440/2304 [I 2022-10-10 20:20:49] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:12:27 [I 2022-10-10 20:20:49] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00312 [I 2022-10-10 20:20:49] ...trap/engines/engine.py.230: loss: 0.12068 [I 2022-10-10 20:20:49] ...optimizers/trijoint.py.151: total_norm: 9.663805 [I 2022-10-10 20:20:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1450/2304 [I 2022-10-10 20:20:57] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:12:11 [I 2022-10-10 20:20:57] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00306 [I 2022-10-10 20:20:57] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-10 20:20:57] ...optimizers/trijoint.py.151: total_norm: 10.348581 [I 2022-10-10 20:21:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 1460/2304 [I 2022-10-10 20:21:06] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:12:09 [I 2022-10-10 20:21:06] ...trap/engines/engine.py.229: process: 0.86313 | load: 0.00309 [I 2022-10-10 20:21:06] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-10 20:21:06] ...optimizers/trijoint.py.151: total_norm: 13.098188 [I 2022-10-10 20:21:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 1470/2304 [I 2022-10-10 20:21:14] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:54 [I 2022-10-10 20:21:14] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00302 [I 2022-10-10 20:21:14] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-10 20:21:14] ...optimizers/trijoint.py.151: total_norm: 12.036702 [I 2022-10-10 20:21:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1480/2304 [I 2022-10-10 20:21:23] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:11:44 [I 2022-10-10 20:21:23] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00312 [I 2022-10-10 20:21:23] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-10 20:21:23] ...optimizers/trijoint.py.151: total_norm: 28.379232 [I 2022-10-10 20:21:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1490/2304 [I 2022-10-10 20:21:32] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:11:40 [I 2022-10-10 20:21:32] ...trap/engines/engine.py.229: process: 0.86296 | load: 0.00306 [I 2022-10-10 20:21:32] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-10 20:21:32] ...optimizers/trijoint.py.151: total_norm: 30.190891 [I 2022-10-10 20:21:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1500/2304 [I 2022-10-10 20:21:40] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:11:28 [I 2022-10-10 20:21:40] ...trap/engines/engine.py.229: process: 0.83379 | load: 0.00313 [I 2022-10-10 20:21:40] ...trap/engines/engine.py.230: loss: 0.10212 [I 2022-10-10 20:21:40] ...optimizers/trijoint.py.151: total_norm: 10.704312 [I 2022-10-10 20:21:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1510/2304 [I 2022-10-10 20:21:49] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:11:23 [I 2022-10-10 20:21:49] ...trap/engines/engine.py.229: process: 0.88165 | load: 0.00302 [I 2022-10-10 20:21:49] ...trap/engines/engine.py.230: loss: 0.14023 [I 2022-10-10 20:21:49] ...optimizers/trijoint.py.151: total_norm: 18.166351 [I 2022-10-10 20:21:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1520/2304 [I 2022-10-10 20:21:57] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:11:09 [I 2022-10-10 20:21:57] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00302 [I 2022-10-10 20:21:57] ...trap/engines/engine.py.230: loss: 0.10319 [I 2022-10-10 20:21:57] ...optimizers/trijoint.py.151: total_norm: 14.795725 [I 2022-10-10 20:22:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 1530/2304 [I 2022-10-10 20:22:06] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:11:03 [I 2022-10-10 20:22:06] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00304 [I 2022-10-10 20:22:06] ...trap/engines/engine.py.230: loss: 0.15436 [I 2022-10-10 20:22:06] ...optimizers/trijoint.py.151: total_norm: 12.978605 [I 2022-10-10 20:22:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1540/2304 [I 2022-10-10 20:22:15] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:10:58 [I 2022-10-10 20:22:15] ...trap/engines/engine.py.229: process: 0.86001 | load: 0.00308 [I 2022-10-10 20:22:15] ...trap/engines/engine.py.230: loss: 0.10981 [I 2022-10-10 20:22:15] ...optimizers/trijoint.py.151: total_norm: 37.382992 [I 2022-10-10 20:22:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1550/2304 [I 2022-10-10 20:22:23] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:10:44 [I 2022-10-10 20:22:23] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00313 [I 2022-10-10 20:22:23] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-10 20:22:23] ...optimizers/trijoint.py.151: total_norm: 16.417700 [I 2022-10-10 20:22:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1560/2304 [I 2022-10-10 20:22:32] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:10:35 [I 2022-10-10 20:22:32] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00314 [I 2022-10-10 20:22:32] ...trap/engines/engine.py.230: loss: 0.14777 [I 2022-10-10 20:22:32] ...optimizers/trijoint.py.151: total_norm: 8.889457 [I 2022-10-10 20:22:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1570/2304 [I 2022-10-10 20:22:40] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:10:28 [I 2022-10-10 20:22:40] ...trap/engines/engine.py.229: process: 0.87123 | load: 0.00305 [I 2022-10-10 20:22:40] ...trap/engines/engine.py.230: loss: 0.11039 [I 2022-10-10 20:22:40] ...optimizers/trijoint.py.151: total_norm: 17.785456 [I 2022-10-10 20:22:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1580/2304 [I 2022-10-10 20:22:49] ...trap/engines/engine.py.228: elapsed: 0:23:08 | left: 0:10:21 [I 2022-10-10 20:22:49] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00314 [I 2022-10-10 20:22:49] ...trap/engines/engine.py.230: loss: 0.14615 [I 2022-10-10 20:22:49] ...optimizers/trijoint.py.151: total_norm: 33.511696 [I 2022-10-10 20:22:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1590/2304 [I 2022-10-10 20:22:57] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:10:12 [I 2022-10-10 20:22:57] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00307 [I 2022-10-10 20:22:57] ...trap/engines/engine.py.230: loss: 0.11339 [I 2022-10-10 20:22:57] ...optimizers/trijoint.py.151: total_norm: 27.055052 [I 2022-10-10 20:23:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 1600/2304 [I 2022-10-10 20:23:06] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:10:04 [I 2022-10-10 20:23:06] ...trap/engines/engine.py.229: process: 0.86367 | load: 0.00295 [I 2022-10-10 20:23:06] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-10 20:23:06] ...optimizers/trijoint.py.151: total_norm: 16.800095 [I 2022-10-10 20:23:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1610/2304 [I 2022-10-10 20:23:15] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:53 [I 2022-10-10 20:23:15] ...trap/engines/engine.py.229: process: 0.84778 | load: 0.00297 [I 2022-10-10 20:23:15] ...trap/engines/engine.py.230: loss: 0.09974 [I 2022-10-10 20:23:15] ...optimizers/trijoint.py.151: total_norm: 9.534862 [I 2022-10-10 20:23:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1620/2304 [I 2022-10-10 20:23:23] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:09:46 [I 2022-10-10 20:23:23] ...trap/engines/engine.py.229: process: 0.85478 | load: 0.00298 [I 2022-10-10 20:23:23] ...trap/engines/engine.py.230: loss: 0.12082 [I 2022-10-10 20:23:23] ...optimizers/trijoint.py.151: total_norm: 12.402233 [I 2022-10-10 20:23:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1630/2304 [I 2022-10-10 20:23:32] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:09:37 [I 2022-10-10 20:23:32] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00325 [I 2022-10-10 20:23:32] ...trap/engines/engine.py.230: loss: 0.09026 [I 2022-10-10 20:23:32] ...optimizers/trijoint.py.151: total_norm: 27.699436 [I 2022-10-10 20:23:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1640/2304 [I 2022-10-10 20:23:40] ...trap/engines/engine.py.228: elapsed: 0:23:59 | left: 0:09:27 [I 2022-10-10 20:23:40] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00298 [I 2022-10-10 20:23:40] ...trap/engines/engine.py.230: loss: 0.11697 [I 2022-10-10 20:23:40] ...optimizers/trijoint.py.151: total_norm: 14.767913 [I 2022-10-10 20:23:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1650/2304 [I 2022-10-10 20:23:49] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:09:19 [I 2022-10-10 20:23:49] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00303 [I 2022-10-10 20:23:49] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-10 20:23:49] ...optimizers/trijoint.py.151: total_norm: 66.384659 [I 2022-10-10 20:23:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 1660/2304 [I 2022-10-10 20:23:58] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:09:07 [I 2022-10-10 20:23:58] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00297 [I 2022-10-10 20:23:58] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-10 20:23:58] ...optimizers/trijoint.py.151: total_norm: 16.811621 [I 2022-10-10 20:24:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 1670/2304 [I 2022-10-10 20:24:06] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:09:07 [I 2022-10-10 20:24:06] ...trap/engines/engine.py.229: process: 0.87621 | load: 0.00324 [I 2022-10-10 20:24:06] ...trap/engines/engine.py.230: loss: 0.22274 [I 2022-10-10 20:24:06] ...optimizers/trijoint.py.151: total_norm: 21.150032 [I 2022-10-10 20:24:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1680/2304 [I 2022-10-10 20:24:15] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:09:00 [I 2022-10-10 20:24:15] ...trap/engines/engine.py.229: process: 0.86741 | load: 0.00309 [I 2022-10-10 20:24:15] ...trap/engines/engine.py.230: loss: 0.10873 [I 2022-10-10 20:24:15] ...optimizers/trijoint.py.151: total_norm: 10.256261 [I 2022-10-10 20:24:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1690/2304 [I 2022-10-10 20:24:23] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:50 [I 2022-10-10 20:24:23] ...trap/engines/engine.py.229: process: 0.87140 | load: 0.00306 [I 2022-10-10 20:24:23] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-10 20:24:23] ...optimizers/trijoint.py.151: total_norm: 10.617009 [I 2022-10-10 20:24:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1700/2304 [I 2022-10-10 20:24:32] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:08:37 [I 2022-10-10 20:24:32] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00309 [I 2022-10-10 20:24:32] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-10 20:24:32] ...optimizers/trijoint.py.151: total_norm: 15.390270 [I 2022-10-10 20:24:41] ...trap/engines/engine.py.224: train: epoch 84 | batch 1710/2304 [I 2022-10-10 20:24:41] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:08:33 [I 2022-10-10 20:24:41] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00307 [I 2022-10-10 20:24:41] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-10 20:24:41] ...optimizers/trijoint.py.151: total_norm: 15.692494 [I 2022-10-10 20:24:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1720/2304 [I 2022-10-10 20:24:49] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:08:26 [I 2022-10-10 20:24:49] ...trap/engines/engine.py.229: process: 0.86230 | load: 0.00307 [I 2022-10-10 20:24:49] ...trap/engines/engine.py.230: loss: 0.09421 [I 2022-10-10 20:24:49] ...optimizers/trijoint.py.151: total_norm: 13.460346 [I 2022-10-10 20:24:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 1730/2304 [I 2022-10-10 20:24:58] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:08:16 [I 2022-10-10 20:24:58] ...trap/engines/engine.py.229: process: 0.86736 | load: 0.00307 [I 2022-10-10 20:24:58] ...trap/engines/engine.py.230: loss: 0.08794 [I 2022-10-10 20:24:58] ...optimizers/trijoint.py.151: total_norm: 14.192909 [I 2022-10-10 20:25:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 1740/2304 [I 2022-10-10 20:25:07] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:08:05 [I 2022-10-10 20:25:07] ...trap/engines/engine.py.229: process: 0.86564 | load: 0.00305 [I 2022-10-10 20:25:07] ...trap/engines/engine.py.230: loss: 0.10699 [I 2022-10-10 20:25:07] ...optimizers/trijoint.py.151: total_norm: 26.122728 [I 2022-10-10 20:25:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1750/2304 [I 2022-10-10 20:25:15] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:58 [I 2022-10-10 20:25:15] ...trap/engines/engine.py.229: process: 0.87238 | load: 0.00313 [I 2022-10-10 20:25:15] ...trap/engines/engine.py.230: loss: 0.12018 [I 2022-10-10 20:25:15] ...optimizers/trijoint.py.151: total_norm: 16.795971 [I 2022-10-10 20:25:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 1760/2304 [I 2022-10-10 20:25:24] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:49 [I 2022-10-10 20:25:24] ...trap/engines/engine.py.229: process: 0.87564 | load: 0.00312 [I 2022-10-10 20:25:24] ...trap/engines/engine.py.230: loss: 0.10055 [I 2022-10-10 20:25:24] ...optimizers/trijoint.py.151: total_norm: 23.299603 [I 2022-10-10 20:25:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 1770/2304 [I 2022-10-10 20:25:33] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:37 [I 2022-10-10 20:25:33] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00322 [I 2022-10-10 20:25:33] ...trap/engines/engine.py.230: loss: 0.17562 [I 2022-10-10 20:25:33] ...optimizers/trijoint.py.151: total_norm: 21.177706 [I 2022-10-10 20:25:41] ...trap/engines/engine.py.224: train: epoch 84 | batch 1780/2304 [I 2022-10-10 20:25:41] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:29 [I 2022-10-10 20:25:41] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00309 [I 2022-10-10 20:25:41] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-10 20:25:41] ...optimizers/trijoint.py.151: total_norm: 16.059446 [I 2022-10-10 20:25:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 1790/2304 [I 2022-10-10 20:25:50] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:21 [I 2022-10-10 20:25:50] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00311 [I 2022-10-10 20:25:50] ...trap/engines/engine.py.230: loss: 0.16832 [I 2022-10-10 20:25:50] ...optimizers/trijoint.py.151: total_norm: 20.316168 [I 2022-10-10 20:25:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 1800/2304 [I 2022-10-10 20:25:59] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:07:13 [I 2022-10-10 20:25:59] ...trap/engines/engine.py.229: process: 0.86357 | load: 0.00309 [I 2022-10-10 20:25:59] ...trap/engines/engine.py.230: loss: 0.14060 [I 2022-10-10 20:25:59] ...optimizers/trijoint.py.151: total_norm: 8.982525 [I 2022-10-10 20:26:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 1810/2304 [I 2022-10-10 20:26:07] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:07:08 [I 2022-10-10 20:26:07] ...trap/engines/engine.py.229: process: 0.86094 | load: 0.00308 [I 2022-10-10 20:26:07] ...trap/engines/engine.py.230: loss: 0.12678 [I 2022-10-10 20:26:07] ...optimizers/trijoint.py.151: total_norm: 24.668507 [I 2022-10-10 20:26:16] ...trap/engines/engine.py.224: train: epoch 84 | batch 1820/2304 [I 2022-10-10 20:26:16] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:57 [I 2022-10-10 20:26:16] ...trap/engines/engine.py.229: process: 0.86711 | load: 0.00316 [I 2022-10-10 20:26:16] ...trap/engines/engine.py.230: loss: 0.20063 [I 2022-10-10 20:26:16] ...optimizers/trijoint.py.151: total_norm: 37.909840 [I 2022-10-10 20:26:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 1830/2304 [I 2022-10-10 20:26:25] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:50 [I 2022-10-10 20:26:25] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00309 [I 2022-10-10 20:26:25] ...trap/engines/engine.py.230: loss: 0.16031 [I 2022-10-10 20:26:25] ...optimizers/trijoint.py.151: total_norm: 14.990477 [I 2022-10-10 20:26:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 1840/2304 [I 2022-10-10 20:26:33] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:41 [I 2022-10-10 20:26:33] ...trap/engines/engine.py.229: process: 0.87618 | load: 0.00312 [I 2022-10-10 20:26:33] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-10 20:26:33] ...optimizers/trijoint.py.151: total_norm: 9.423889 [I 2022-10-10 20:26:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 1850/2304 [I 2022-10-10 20:26:42] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:30 [I 2022-10-10 20:26:42] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00309 [I 2022-10-10 20:26:42] ...trap/engines/engine.py.230: loss: 0.10893 [I 2022-10-10 20:26:42] ...optimizers/trijoint.py.151: total_norm: 34.495407 [I 2022-10-10 20:26:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 1860/2304 [I 2022-10-10 20:26:51] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:06:22 [I 2022-10-10 20:26:51] ...trap/engines/engine.py.229: process: 0.85649 | load: 0.00323 [I 2022-10-10 20:26:51] ...trap/engines/engine.py.230: loss: 0.17531 [I 2022-10-10 20:26:51] ...optimizers/trijoint.py.151: total_norm: 24.991871 [I 2022-10-10 20:27:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 1870/2304 [I 2022-10-10 20:27:17] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:16:32 [I 2022-10-10 20:27:17] ...trap/engines/engine.py.229: process: 0.86135 | load: 0.00313 [I 2022-10-10 20:27:17] ...trap/engines/engine.py.230: loss: 0.12409 [I 2022-10-10 20:27:17] ...optimizers/trijoint.py.151: total_norm: 18.606239 [I 2022-10-10 20:27:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 1880/2304 [I 2022-10-10 20:27:25] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:07:10 [I 2022-10-10 20:27:25] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00314 [I 2022-10-10 20:27:25] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-10 20:27:25] ...optimizers/trijoint.py.151: total_norm: 14.415236 [I 2022-10-10 20:27:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 1890/2304 [I 2022-10-10 20:27:34] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:06:05 [I 2022-10-10 20:27:34] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00311 [I 2022-10-10 20:27:34] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-10 20:27:34] ...optimizers/trijoint.py.151: total_norm: 23.193985 [I 2022-10-10 20:27:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 1900/2304 [I 2022-10-10 20:27:43] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:05:49 [I 2022-10-10 20:27:43] ...trap/engines/engine.py.229: process: 0.86381 | load: 0.00310 [I 2022-10-10 20:27:43] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-10 20:27:43] ...optimizers/trijoint.py.151: total_norm: 21.158192 [I 2022-10-10 20:27:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 1910/2304 [I 2022-10-10 20:27:51] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:05:42 [I 2022-10-10 20:27:51] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00308 [I 2022-10-10 20:27:51] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-10 20:27:51] ...optimizers/trijoint.py.151: total_norm: 11.302231 [I 2022-10-10 20:28:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 1920/2304 [I 2022-10-10 20:28:00] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:05:30 [I 2022-10-10 20:28:00] ...trap/engines/engine.py.229: process: 0.86017 | load: 0.00303 [I 2022-10-10 20:28:00] ...trap/engines/engine.py.230: loss: 0.17247 [I 2022-10-10 20:28:00] ...optimizers/trijoint.py.151: total_norm: 31.710821 [I 2022-10-10 20:28:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 1930/2304 [I 2022-10-10 20:28:08] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:05:18 [I 2022-10-10 20:28:08] ...trap/engines/engine.py.229: process: 0.86902 | load: 0.00322 [I 2022-10-10 20:28:08] ...trap/engines/engine.py.230: loss: 0.17797 [I 2022-10-10 20:28:08] ...optimizers/trijoint.py.151: total_norm: 24.164368 [I 2022-10-10 20:28:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 1940/2304 [I 2022-10-10 20:28:17] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:05:11 [I 2022-10-10 20:28:17] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00293 [I 2022-10-10 20:28:17] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-10 20:28:17] ...optimizers/trijoint.py.151: total_norm: 25.363295 [I 2022-10-10 20:28:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 1950/2304 [I 2022-10-10 20:28:26] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:05:03 [I 2022-10-10 20:28:26] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00295 [I 2022-10-10 20:28:26] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-10 20:28:26] ...optimizers/trijoint.py.151: total_norm: 14.092492 [I 2022-10-10 20:28:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 1960/2304 [I 2022-10-10 20:28:34] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:04:53 [I 2022-10-10 20:28:34] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00298 [I 2022-10-10 20:28:34] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-10 20:28:34] ...optimizers/trijoint.py.151: total_norm: 11.473002 [I 2022-10-10 20:28:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 1970/2304 [I 2022-10-10 20:28:43] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:45 [I 2022-10-10 20:28:43] ...trap/engines/engine.py.229: process: 0.87021 | load: 0.00299 [I 2022-10-10 20:28:43] ...trap/engines/engine.py.230: loss: 0.11775 [I 2022-10-10 20:28:43] ...optimizers/trijoint.py.151: total_norm: 14.432501 [I 2022-10-10 20:28:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 1980/2304 [I 2022-10-10 20:28:51] ...trap/engines/engine.py.228: elapsed: 0:29:10 | left: 0:04:36 [I 2022-10-10 20:28:51] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00311 [I 2022-10-10 20:28:51] ...trap/engines/engine.py.230: loss: 0.10052 [I 2022-10-10 20:28:51] ...optimizers/trijoint.py.151: total_norm: 13.358187 [I 2022-10-10 20:29:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 1990/2304 [I 2022-10-10 20:29:00] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:04:28 [I 2022-10-10 20:29:00] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00298 [I 2022-10-10 20:29:00] ...trap/engines/engine.py.230: loss: 0.17509 [I 2022-10-10 20:29:00] ...optimizers/trijoint.py.151: total_norm: 14.965364 [I 2022-10-10 20:29:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 2000/2304 [I 2022-10-10 20:29:08] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:04:17 [I 2022-10-10 20:29:08] ...trap/engines/engine.py.229: process: 0.86074 | load: 0.00298 [I 2022-10-10 20:29:08] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-10 20:29:08] ...optimizers/trijoint.py.151: total_norm: 9.388783 [I 2022-10-10 20:29:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 2010/2304 [I 2022-10-10 20:29:17] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:04:12 [I 2022-10-10 20:29:17] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00296 [I 2022-10-10 20:29:17] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-10 20:29:17] ...optimizers/trijoint.py.151: total_norm: 15.739361 [I 2022-10-10 20:29:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 2020/2304 [I 2022-10-10 20:29:25] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:04:02 [I 2022-10-10 20:29:25] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00298 [I 2022-10-10 20:29:25] ...trap/engines/engine.py.230: loss: 0.10201 [I 2022-10-10 20:29:25] ...optimizers/trijoint.py.151: total_norm: 8.598679 [I 2022-10-10 20:29:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 2030/2304 [I 2022-10-10 20:29:34] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:03:54 [I 2022-10-10 20:29:34] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00302 [I 2022-10-10 20:29:34] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-10 20:29:34] ...optimizers/trijoint.py.151: total_norm: 21.533012 [I 2022-10-10 20:29:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 2040/2304 [I 2022-10-10 20:29:42] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:45 [I 2022-10-10 20:29:42] ...trap/engines/engine.py.229: process: 0.85162 | load: 0.00299 [I 2022-10-10 20:29:42] ...trap/engines/engine.py.230: loss: 0.23098 [I 2022-10-10 20:29:43] ...optimizers/trijoint.py.151: total_norm: 27.017221 [I 2022-10-10 20:29:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 2050/2304 [I 2022-10-10 20:29:51] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:03:37 [I 2022-10-10 20:29:51] ...trap/engines/engine.py.229: process: 0.86369 | load: 0.00300 [I 2022-10-10 20:29:51] ...trap/engines/engine.py.230: loss: 0.12030 [I 2022-10-10 20:29:51] ...optimizers/trijoint.py.151: total_norm: 15.888584 [I 2022-10-10 20:30:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 2060/2304 [I 2022-10-10 20:30:00] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:03:29 [I 2022-10-10 20:30:00] ...trap/engines/engine.py.229: process: 0.86025 | load: 0.00308 [I 2022-10-10 20:30:00] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-10 20:30:00] ...optimizers/trijoint.py.151: total_norm: 13.952991 [I 2022-10-10 20:30:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 2070/2304 [I 2022-10-10 20:30:08] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:03:19 [I 2022-10-10 20:30:08] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00318 [I 2022-10-10 20:30:08] ...trap/engines/engine.py.230: loss: 0.11374 [I 2022-10-10 20:30:08] ...optimizers/trijoint.py.151: total_norm: 8.140925 [I 2022-10-10 20:30:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 2080/2304 [I 2022-10-10 20:30:17] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:03:13 [I 2022-10-10 20:30:17] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00309 [I 2022-10-10 20:30:17] ...trap/engines/engine.py.230: loss: 0.08623 [I 2022-10-10 20:30:17] ...optimizers/trijoint.py.151: total_norm: 7.827379 [I 2022-10-10 20:30:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 2090/2304 [I 2022-10-10 20:30:26] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:03:05 [I 2022-10-10 20:30:26] ...trap/engines/engine.py.229: process: 0.84926 | load: 0.00307 [I 2022-10-10 20:30:26] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-10 20:30:26] ...optimizers/trijoint.py.151: total_norm: 21.068846 [I 2022-10-10 20:30:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 2100/2304 [I 2022-10-10 20:30:34] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:55 [I 2022-10-10 20:30:34] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00307 [I 2022-10-10 20:30:34] ...trap/engines/engine.py.230: loss: 0.09611 [I 2022-10-10 20:30:34] ...optimizers/trijoint.py.151: total_norm: 26.086615 [I 2022-10-10 20:30:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 2110/2304 [I 2022-10-10 20:30:43] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:47 [I 2022-10-10 20:30:43] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00306 [I 2022-10-10 20:30:43] ...trap/engines/engine.py.230: loss: 0.12711 [I 2022-10-10 20:30:43] ...optimizers/trijoint.py.151: total_norm: 12.384923 [I 2022-10-10 20:30:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 2120/2304 [I 2022-10-10 20:30:51] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:02:39 [I 2022-10-10 20:30:51] ...trap/engines/engine.py.229: process: 0.86089 | load: 0.00308 [I 2022-10-10 20:30:51] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-10 20:30:51] ...optimizers/trijoint.py.151: total_norm: 11.591807 [I 2022-10-10 20:31:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 2130/2304 [I 2022-10-10 20:31:00] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:02:31 [I 2022-10-10 20:31:00] ...trap/engines/engine.py.229: process: 0.86409 | load: 0.00316 [I 2022-10-10 20:31:00] ...trap/engines/engine.py.230: loss: 0.14211 [I 2022-10-10 20:31:00] ...optimizers/trijoint.py.151: total_norm: 14.853357 [I 2022-10-10 20:31:09] ...trap/engines/engine.py.224: train: epoch 84 | batch 2140/2304 [I 2022-10-10 20:31:09] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:02:21 [I 2022-10-10 20:31:09] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00312 [I 2022-10-10 20:31:09] ...trap/engines/engine.py.230: loss: 0.08603 [I 2022-10-10 20:31:09] ...optimizers/trijoint.py.151: total_norm: 10.260575 [I 2022-10-10 20:31:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 2150/2304 [I 2022-10-10 20:31:17] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:02:12 [I 2022-10-10 20:31:17] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00310 [I 2022-10-10 20:31:17] ...trap/engines/engine.py.230: loss: 0.13630 [I 2022-10-10 20:31:17] ...optimizers/trijoint.py.151: total_norm: 16.871773 [I 2022-10-10 20:31:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 2160/2304 [I 2022-10-10 20:31:26] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:02:03 [I 2022-10-10 20:31:26] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00310 [I 2022-10-10 20:31:26] ...trap/engines/engine.py.230: loss: 0.16319 [I 2022-10-10 20:31:26] ...optimizers/trijoint.py.151: total_norm: 13.970786 [I 2022-10-10 20:31:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 2170/2304 [I 2022-10-10 20:31:35] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:01:55 [I 2022-10-10 20:31:35] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00312 [I 2022-10-10 20:31:35] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-10 20:31:35] ...optimizers/trijoint.py.151: total_norm: 23.135843 [I 2022-10-10 20:31:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 2180/2304 [I 2022-10-10 20:31:43] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:47 [I 2022-10-10 20:31:43] ...trap/engines/engine.py.229: process: 0.86277 | load: 0.00313 [I 2022-10-10 20:31:43] ...trap/engines/engine.py.230: loss: 0.08868 [I 2022-10-10 20:31:43] ...optimizers/trijoint.py.151: total_norm: 21.176296 [I 2022-10-10 20:31:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 2190/2304 [I 2022-10-10 20:31:52] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:01:38 [I 2022-10-10 20:31:52] ...trap/engines/engine.py.229: process: 0.86712 | load: 0.00309 [I 2022-10-10 20:31:52] ...trap/engines/engine.py.230: loss: 0.11193 [I 2022-10-10 20:31:52] ...optimizers/trijoint.py.151: total_norm: 11.394901 [I 2022-10-10 20:32:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 2200/2304 [I 2022-10-10 20:32:01] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:01:29 [I 2022-10-10 20:32:01] ...trap/engines/engine.py.229: process: 0.87574 | load: 0.00310 [I 2022-10-10 20:32:01] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-10 20:32:01] ...optimizers/trijoint.py.151: total_norm: 13.035935 [I 2022-10-10 20:32:09] ...trap/engines/engine.py.224: train: epoch 84 | batch 2210/2304 [I 2022-10-10 20:32:09] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:01:20 [I 2022-10-10 20:32:09] ...trap/engines/engine.py.229: process: 0.85817 | load: 0.00309 [I 2022-10-10 20:32:09] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-10 20:32:09] ...optimizers/trijoint.py.151: total_norm: 14.682260 [I 2022-10-10 20:32:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 2220/2304 [I 2022-10-10 20:32:18] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:01:12 [I 2022-10-10 20:32:18] ...trap/engines/engine.py.229: process: 0.87024 | load: 0.00309 [I 2022-10-10 20:32:18] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-10 20:32:18] ...optimizers/trijoint.py.151: total_norm: 12.342117 [I 2022-10-10 20:32:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 2230/2304 [I 2022-10-10 20:32:26] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:01:03 [I 2022-10-10 20:32:26] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00318 [I 2022-10-10 20:32:26] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-10 20:32:26] ...optimizers/trijoint.py.151: total_norm: 17.199507 [I 2022-10-10 20:32:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 2240/2304 [I 2022-10-10 20:32:35] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:55 [I 2022-10-10 20:32:35] ...trap/engines/engine.py.229: process: 0.86729 | load: 0.00312 [I 2022-10-10 20:32:35] ...trap/engines/engine.py.230: loss: 0.17194 [I 2022-10-10 20:32:35] ...optimizers/trijoint.py.151: total_norm: 28.227461 [I 2022-10-10 20:32:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 2250/2304 [I 2022-10-10 20:32:44] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:46 [I 2022-10-10 20:32:44] ...trap/engines/engine.py.229: process: 0.85501 | load: 0.00320 [I 2022-10-10 20:32:44] ...trap/engines/engine.py.230: loss: 0.12833 [I 2022-10-10 20:32:44] ...optimizers/trijoint.py.151: total_norm: 14.571665 [I 2022-10-10 20:32:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 2260/2304 [I 2022-10-10 20:32:52] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:37 [I 2022-10-10 20:32:52] ...trap/engines/engine.py.229: process: 0.85877 | load: 0.00321 [I 2022-10-10 20:32:52] ...trap/engines/engine.py.230: loss: 0.13241 [I 2022-10-10 20:32:52] ...optimizers/trijoint.py.151: total_norm: 13.729975 [I 2022-10-10 20:33:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 2270/2304 [I 2022-10-10 20:33:01] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:29 [I 2022-10-10 20:33:01] ...trap/engines/engine.py.229: process: 0.86294 | load: 0.00308 [I 2022-10-10 20:33:01] ...trap/engines/engine.py.230: loss: 0.12468 [I 2022-10-10 20:33:01] ...optimizers/trijoint.py.151: total_norm: 21.471109 [I 2022-10-10 20:33:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 2280/2304 [I 2022-10-10 20:33:10] ...trap/engines/engine.py.228: elapsed: 0:33:29 | left: 0:00:20 [I 2022-10-10 20:33:10] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00318 [I 2022-10-10 20:33:10] ...trap/engines/engine.py.230: loss: 0.16683 [I 2022-10-10 20:33:10] ...optimizers/trijoint.py.151: total_norm: 14.590920 [I 2022-10-10 20:33:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 2290/2304 [I 2022-10-10 20:33:18] ...trap/engines/engine.py.228: elapsed: 0:33:37 | left: 0:00:12 [I 2022-10-10 20:33:18] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00308 [I 2022-10-10 20:33:18] ...trap/engines/engine.py.230: loss: 0.11598 [I 2022-10-10 20:33:18] ...optimizers/trijoint.py.151: total_norm: 19.374271 [I 2022-10-10 20:33:27] ...trap/engines/engine.py.224: train: epoch 84 | batch 2300/2304 [I 2022-10-10 20:33:27] ...trap/engines/engine.py.228: elapsed: 0:33:46 | left: 0:00:03 [I 2022-10-10 20:33:27] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00019 [I 2022-10-10 20:33:27] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-10 20:33:27] ...optimizers/trijoint.py.151: total_norm: 6.995662 [I 2022-10-10 20:33:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 2304/2304 [I 2022-10-10 20:33:30] ...trap/engines/engine.py.228: elapsed: 0:33:49 | left: 0:00:00 [I 2022-10-10 20:33:30] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00019 [I 2022-10-10 20:33:30] ...trap/engines/engine.py.230: loss: 0.09566 [I 2022-10-10 20:33:30] ...optimizers/trijoint.py.151: total_norm: 11.647339 [S 2022-10-10 20:33:32] ...trap/engines/engine.py.129: train_epoch.epoch: 84 [S 2022-10-10 20:33:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.13094631648761848 [I 2022-10-10 20:33:32] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 20:33:33] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 84 [I 2022-10-10 20:33:33] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 20:33:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 20:33:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 20:33:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 20:33:39] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 20:33:39] ...trap/engines/engine.py.321: eval: epoch 84 | batch 0/510 [I 2022-10-10 20:33:39] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:14 [I 2022-10-10 20:33:39] ...trap/engines/engine.py.326: process: 5.67592 | load: 4.43260 [I 2022-10-10 20:33:44] ...trap/engines/engine.py.321: eval: epoch 84 | batch 10/510 [I 2022-10-10 20:33:44] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:36 [I 2022-10-10 20:33:44] ...trap/engines/engine.py.326: process: 0.46140 | load: 0.00020 [I 2022-10-10 20:33:49] ...trap/engines/engine.py.321: eval: epoch 84 | batch 20/510 [I 2022-10-10 20:33:49] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:05:00 [I 2022-10-10 20:33:49] ...trap/engines/engine.py.326: process: 0.57558 | load: 0.00019 [I 2022-10-10 20:33:54] ...trap/engines/engine.py.321: eval: epoch 84 | batch 30/510 [I 2022-10-10 20:33:54] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:02 [I 2022-10-10 20:33:54] ...trap/engines/engine.py.326: process: 0.61254 | load: 0.00020 [I 2022-10-10 20:33:59] ...trap/engines/engine.py.321: eval: epoch 84 | batch 40/510 [I 2022-10-10 20:33:59] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:04:22 [I 2022-10-10 20:33:59] ...trap/engines/engine.py.326: process: 0.44335 | load: 0.00019 [I 2022-10-10 20:34:04] ...trap/engines/engine.py.321: eval: epoch 84 | batch 50/510 [I 2022-10-10 20:34:04] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:47 [I 2022-10-10 20:34:04] ...trap/engines/engine.py.326: process: 0.54100 | load: 0.00019 [I 2022-10-10 20:34:09] ...trap/engines/engine.py.321: eval: epoch 84 | batch 60/510 [I 2022-10-10 20:34:09] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:03:22 [I 2022-10-10 20:34:09] ...trap/engines/engine.py.326: process: 0.47880 | load: 0.00019 [I 2022-10-10 20:34:13] ...trap/engines/engine.py.321: eval: epoch 84 | batch 70/510 [I 2022-10-10 20:34:13] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:41 [I 2022-10-10 20:34:13] ...trap/engines/engine.py.326: process: 0.57853 | load: 0.00021 [I 2022-10-10 20:34:18] ...trap/engines/engine.py.321: eval: epoch 84 | batch 80/510 [I 2022-10-10 20:34:18] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:34 [I 2022-10-10 20:34:18] ...trap/engines/engine.py.326: process: 0.48557 | load: 0.00019 [I 2022-10-10 20:34:23] ...trap/engines/engine.py.321: eval: epoch 84 | batch 90/510 [I 2022-10-10 20:34:23] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:03:00 [I 2022-10-10 20:34:23] ...trap/engines/engine.py.326: process: 0.41691 | load: 0.00019 [I 2022-10-10 20:34:27] ...trap/engines/engine.py.321: eval: epoch 84 | batch 100/510 [I 2022-10-10 20:34:27] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:47 [I 2022-10-10 20:34:27] ...trap/engines/engine.py.326: process: 0.30894 | load: 0.00018 [I 2022-10-10 20:34:30] ...trap/engines/engine.py.321: eval: epoch 84 | batch 110/510 [I 2022-10-10 20:34:30] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:10 [I 2022-10-10 20:34:30] ...trap/engines/engine.py.326: process: 0.31710 | load: 0.00020 [I 2022-10-10 20:34:33] ...trap/engines/engine.py.321: eval: epoch 84 | batch 120/510 [I 2022-10-10 20:34:33] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:03 [I 2022-10-10 20:34:33] ...trap/engines/engine.py.326: process: 0.31818 | load: 0.00019 [I 2022-10-10 20:34:36] ...trap/engines/engine.py.321: eval: epoch 84 | batch 130/510 [I 2022-10-10 20:34:36] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:00 [I 2022-10-10 20:34:36] ...trap/engines/engine.py.326: process: 0.32015 | load: 0.00019 [I 2022-10-10 20:34:40] ...trap/engines/engine.py.321: eval: epoch 84 | batch 140/510 [I 2022-10-10 20:34:40] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:56 [I 2022-10-10 20:34:40] ...trap/engines/engine.py.326: process: 0.30651 | load: 0.00020 [I 2022-10-10 20:34:43] ...trap/engines/engine.py.321: eval: epoch 84 | batch 150/510 [I 2022-10-10 20:34:43] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:54 [I 2022-10-10 20:34:43] ...trap/engines/engine.py.326: process: 0.31581 | load: 0.00019 [I 2022-10-10 20:34:46] ...trap/engines/engine.py.321: eval: epoch 84 | batch 160/510 [I 2022-10-10 20:34:46] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:51 [I 2022-10-10 20:34:46] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00019 [I 2022-10-10 20:34:49] ...trap/engines/engine.py.321: eval: epoch 84 | batch 170/510 [I 2022-10-10 20:34:49] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:48 [I 2022-10-10 20:34:49] ...trap/engines/engine.py.326: process: 0.31832 | load: 0.00019 [I 2022-10-10 20:34:52] ...trap/engines/engine.py.321: eval: epoch 84 | batch 180/510 [I 2022-10-10 20:34:52] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:44 [I 2022-10-10 20:34:52] ...trap/engines/engine.py.326: process: 0.31330 | load: 0.00020 [I 2022-10-10 20:34:56] ...trap/engines/engine.py.321: eval: epoch 84 | batch 190/510 [I 2022-10-10 20:34:56] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:41 [I 2022-10-10 20:34:56] ...trap/engines/engine.py.326: process: 0.31303 | load: 0.00019 [I 2022-10-10 20:34:59] ...trap/engines/engine.py.321: eval: epoch 84 | batch 200/510 [I 2022-10-10 20:34:59] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:38 [I 2022-10-10 20:34:59] ...trap/engines/engine.py.326: process: 0.31760 | load: 0.00019 [I 2022-10-10 20:35:02] ...trap/engines/engine.py.321: eval: epoch 84 | batch 210/510 [I 2022-10-10 20:35:02] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:36 [I 2022-10-10 20:35:02] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00020 [I 2022-10-10 20:35:05] ...trap/engines/engine.py.321: eval: epoch 84 | batch 220/510 [I 2022-10-10 20:35:05] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:31 [I 2022-10-10 20:35:05] ...trap/engines/engine.py.326: process: 0.30836 | load: 0.00020 [I 2022-10-10 20:35:08] ...trap/engines/engine.py.321: eval: epoch 84 | batch 230/510 [I 2022-10-10 20:35:08] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:28 [I 2022-10-10 20:35:08] ...trap/engines/engine.py.326: process: 0.31654 | load: 0.00019 [I 2022-10-10 20:35:11] ...trap/engines/engine.py.321: eval: epoch 84 | batch 240/510 [I 2022-10-10 20:35:11] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:26 [I 2022-10-10 20:35:11] ...trap/engines/engine.py.326: process: 0.32139 | load: 0.00019 [I 2022-10-10 20:35:15] ...trap/engines/engine.py.321: eval: epoch 84 | batch 250/510 [I 2022-10-10 20:35:15] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:22 [I 2022-10-10 20:35:15] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00020 [I 2022-10-10 20:35:18] ...trap/engines/engine.py.321: eval: epoch 84 | batch 260/510 [I 2022-10-10 20:35:18] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:19 [I 2022-10-10 20:35:18] ...trap/engines/engine.py.326: process: 0.31653 | load: 0.00019 [I 2022-10-10 20:35:21] ...trap/engines/engine.py.321: eval: epoch 84 | batch 270/510 [I 2022-10-10 20:35:21] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:16 [I 2022-10-10 20:35:21] ...trap/engines/engine.py.326: process: 0.31624 | load: 0.00019 [I 2022-10-10 20:35:24] ...trap/engines/engine.py.321: eval: epoch 84 | batch 280/510 [I 2022-10-10 20:35:24] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:13 [I 2022-10-10 20:35:24] ...trap/engines/engine.py.326: process: 0.32261 | load: 0.00019 [I 2022-10-10 20:35:27] ...trap/engines/engine.py.321: eval: epoch 84 | batch 290/510 [I 2022-10-10 20:35:27] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:10 [I 2022-10-10 20:35:27] ...trap/engines/engine.py.326: process: 0.32271 | load: 0.00019 [I 2022-10-10 20:35:31] ...trap/engines/engine.py.321: eval: epoch 84 | batch 300/510 [I 2022-10-10 20:35:31] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:07 [I 2022-10-10 20:35:31] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00020 [I 2022-10-10 20:35:34] ...trap/engines/engine.py.321: eval: epoch 84 | batch 310/510 [I 2022-10-10 20:35:34] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:03 [I 2022-10-10 20:35:34] ...trap/engines/engine.py.326: process: 0.31547 | load: 0.00019 [I 2022-10-10 20:35:37] ...trap/engines/engine.py.321: eval: epoch 84 | batch 320/510 [I 2022-10-10 20:35:37] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-10 20:35:37] ...trap/engines/engine.py.326: process: 0.31786 | load: 0.00020 [I 2022-10-10 20:35:40] ...trap/engines/engine.py.321: eval: epoch 84 | batch 330/510 [I 2022-10-10 20:35:40] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:57 [I 2022-10-10 20:35:40] ...trap/engines/engine.py.326: process: 0.32184 | load: 0.00022 [I 2022-10-10 20:35:44] ...trap/engines/engine.py.321: eval: epoch 84 | batch 340/510 [I 2022-10-10 20:35:44] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:54 [I 2022-10-10 20:35:44] ...trap/engines/engine.py.326: process: 0.31492 | load: 0.00021 [I 2022-10-10 20:35:47] ...trap/engines/engine.py.321: eval: epoch 84 | batch 350/510 [I 2022-10-10 20:35:47] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:51 [I 2022-10-10 20:35:47] ...trap/engines/engine.py.326: process: 0.31997 | load: 0.00022 [I 2022-10-10 20:35:50] ...trap/engines/engine.py.321: eval: epoch 84 | batch 360/510 [I 2022-10-10 20:35:50] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:48 [I 2022-10-10 20:35:50] ...trap/engines/engine.py.326: process: 0.32702 | load: 0.00024 [I 2022-10-10 20:35:53] ...trap/engines/engine.py.321: eval: epoch 84 | batch 370/510 [I 2022-10-10 20:35:53] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:45 [I 2022-10-10 20:35:53] ...trap/engines/engine.py.326: process: 0.32994 | load: 0.00021 [I 2022-10-10 20:35:57] ...trap/engines/engine.py.321: eval: epoch 84 | batch 380/510 [I 2022-10-10 20:35:57] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:42 [I 2022-10-10 20:35:57] ...trap/engines/engine.py.326: process: 0.32556 | load: 0.00025 [I 2022-10-10 20:36:00] ...trap/engines/engine.py.321: eval: epoch 84 | batch 390/510 [I 2022-10-10 20:36:00] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:38 [I 2022-10-10 20:36:00] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00020 [I 2022-10-10 20:36:03] ...trap/engines/engine.py.321: eval: epoch 84 | batch 400/510 [I 2022-10-10 20:36:03] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:35 [I 2022-10-10 20:36:03] ...trap/engines/engine.py.326: process: 0.32693 | load: 0.00023 [I 2022-10-10 20:36:06] ...trap/engines/engine.py.321: eval: epoch 84 | batch 410/510 [I 2022-10-10 20:36:06] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:32 [I 2022-10-10 20:36:06] ...trap/engines/engine.py.326: process: 0.32369 | load: 0.00020 [I 2022-10-10 20:36:10] ...trap/engines/engine.py.321: eval: epoch 84 | batch 420/510 [I 2022-10-10 20:36:10] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:29 [I 2022-10-10 20:36:10] ...trap/engines/engine.py.326: process: 0.32647 | load: 0.00021 [I 2022-10-10 20:36:13] ...trap/engines/engine.py.321: eval: epoch 84 | batch 430/510 [I 2022-10-10 20:36:13] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:25 [I 2022-10-10 20:36:13] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00021 [I 2022-10-10 20:36:16] ...trap/engines/engine.py.321: eval: epoch 84 | batch 440/510 [I 2022-10-10 20:36:16] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:22 [I 2022-10-10 20:36:16] ...trap/engines/engine.py.326: process: 0.32230 | load: 0.00021 [I 2022-10-10 20:36:19] ...trap/engines/engine.py.321: eval: epoch 84 | batch 450/510 [I 2022-10-10 20:36:19] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:19 [I 2022-10-10 20:36:19] ...trap/engines/engine.py.326: process: 0.33180 | load: 0.00021 [I 2022-10-10 20:36:23] ...trap/engines/engine.py.321: eval: epoch 84 | batch 460/510 [I 2022-10-10 20:36:23] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:16 [I 2022-10-10 20:36:23] ...trap/engines/engine.py.326: process: 0.32657 | load: 0.00022 [I 2022-10-10 20:36:26] ...trap/engines/engine.py.321: eval: epoch 84 | batch 470/510 [I 2022-10-10 20:36:26] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:13 [I 2022-10-10 20:36:26] ...trap/engines/engine.py.326: process: 0.33010 | load: 0.00022 [I 2022-10-10 20:36:29] ...trap/engines/engine.py.321: eval: epoch 84 | batch 480/510 [I 2022-10-10 20:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:09 [I 2022-10-10 20:36:29] ...trap/engines/engine.py.326: process: 0.32657 | load: 0.00023 [I 2022-10-10 20:36:32] ...trap/engines/engine.py.321: eval: epoch 84 | batch 490/510 [I 2022-10-10 20:36:32] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 20:36:32] ...trap/engines/engine.py.326: process: 0.32594 | load: 0.00022 [I 2022-10-10 20:36:36] ...trap/engines/engine.py.321: eval: epoch 84 | batch 500/510 [I 2022-10-10 20:36:36] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-10 20:36:36] ...trap/engines/engine.py.326: process: 0.32254 | load: 0.00022 [I 2022-10-10 20:36:39] ...trap/engines/engine.py.321: eval: epoch 84 | batch 510/510 [I 2022-10-10 20:36:39] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-10 20:36:39] ...trap/engines/engine.py.326: process: 0.32082 | load: 0.00017 [S 2022-10-10 20:36:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 84 [I 2022-10-10 20:36:40] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.743400 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.915100 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941100 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.740000 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.916200 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.939600 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009489 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007765 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006057 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011009 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007454 [S 2022-10-10 20:37:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007446 [I 2022-10-10 20:37:26] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 20:37:26] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 20:37:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 20:37:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 20:37:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 20:37:30] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 20:37:33] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 20:37:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 20:37:37] ...trap/engines/engine.py.163: Training model on trainset for epoch 85 [I 2022-10-10 20:37:37] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 20:37:41] ...trap/engines/engine.py.224: train: epoch 85 | batch 0/2304 [I 2022-10-10 20:37:41] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:16:50 [I 2022-10-10 20:37:41] ...trap/engines/engine.py.229: process: 3.56349 | load: 2.62055 [I 2022-10-10 20:37:41] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-10 20:37:41] ...optimizers/trijoint.py.151: total_norm: 11.809501 [I 2022-10-10 20:37:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 10/2304 [I 2022-10-10 20:37:49] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:44:02 [I 2022-10-10 20:37:49] ...trap/engines/engine.py.229: process: 0.87234 | load: 0.00771 [I 2022-10-10 20:37:49] ...trap/engines/engine.py.230: loss: 0.08577 [I 2022-10-10 20:37:49] ...optimizers/trijoint.py.151: total_norm: 14.958423 [I 2022-10-10 20:37:58] ...trap/engines/engine.py.224: train: epoch 85 | batch 20/2304 [I 2022-10-10 20:37:58] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:06 [I 2022-10-10 20:37:58] ...trap/engines/engine.py.229: process: 0.87721 | load: 0.00673 [I 2022-10-10 20:37:58] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-10 20:37:58] ...optimizers/trijoint.py.151: total_norm: 8.797515 [I 2022-10-10 20:38:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 30/2304 [I 2022-10-10 20:38:06] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:00 [I 2022-10-10 20:38:06] ...trap/engines/engine.py.229: process: 0.86450 | load: 0.00461 [I 2022-10-10 20:38:06] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-10 20:38:06] ...optimizers/trijoint.py.151: total_norm: 17.073437 [I 2022-10-10 20:38:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 40/2304 [I 2022-10-10 20:38:15] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:29 [I 2022-10-10 20:38:15] ...trap/engines/engine.py.229: process: 0.86059 | load: 0.00506 [I 2022-10-10 20:38:15] ...trap/engines/engine.py.230: loss: 0.09890 [I 2022-10-10 20:38:15] ...optimizers/trijoint.py.151: total_norm: 11.810760 [I 2022-10-10 20:38:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 50/2304 [I 2022-10-10 20:38:24] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:36 [I 2022-10-10 20:38:24] ...trap/engines/engine.py.229: process: 0.86512 | load: 0.00444 [I 2022-10-10 20:38:24] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-10 20:38:24] ...optimizers/trijoint.py.151: total_norm: 10.895283 [I 2022-10-10 20:38:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 60/2304 [I 2022-10-10 20:38:32] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:27 [I 2022-10-10 20:38:32] ...trap/engines/engine.py.229: process: 0.86929 | load: 0.00376 [I 2022-10-10 20:38:32] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-10 20:38:32] ...optimizers/trijoint.py.151: total_norm: 18.824413 [I 2022-10-10 20:38:41] ...trap/engines/engine.py.224: train: epoch 85 | batch 70/2304 [I 2022-10-10 20:38:41] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:03 [I 2022-10-10 20:38:41] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00335 [I 2022-10-10 20:38:41] ...trap/engines/engine.py.230: loss: 0.16983 [I 2022-10-10 20:38:41] ...optimizers/trijoint.py.151: total_norm: 18.535963 [I 2022-10-10 20:38:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 80/2304 [I 2022-10-10 20:38:50] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:02 [I 2022-10-10 20:38:50] ...trap/engines/engine.py.229: process: 0.85614 | load: 0.00926 [I 2022-10-10 20:38:50] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-10 20:38:50] ...optimizers/trijoint.py.151: total_norm: 37.849846 [I 2022-10-10 20:38:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 90/2304 [I 2022-10-10 20:38:59] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:32:00 [I 2022-10-10 20:38:59] ...trap/engines/engine.py.229: process: 0.86562 | load: 0.00324 [I 2022-10-10 20:38:59] ...trap/engines/engine.py.230: loss: 0.08507 [I 2022-10-10 20:38:59] ...optimizers/trijoint.py.151: total_norm: 23.259689 [I 2022-10-10 20:39:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 100/2304 [I 2022-10-10 20:39:07] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:31:45 [I 2022-10-10 20:39:07] ...trap/engines/engine.py.229: process: 0.85925 | load: 0.00350 [I 2022-10-10 20:39:07] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-10 20:39:07] ...optimizers/trijoint.py.151: total_norm: 12.660705 [I 2022-10-10 20:39:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 110/2304 [I 2022-10-10 20:39:16] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:43 [I 2022-10-10 20:39:16] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00356 [I 2022-10-10 20:39:16] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-10 20:39:16] ...optimizers/trijoint.py.151: total_norm: 18.367130 [I 2022-10-10 20:39:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 120/2304 [I 2022-10-10 20:39:25] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:30 [I 2022-10-10 20:39:25] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00344 [I 2022-10-10 20:39:25] ...trap/engines/engine.py.230: loss: 0.17245 [I 2022-10-10 20:39:25] ...optimizers/trijoint.py.151: total_norm: 14.474428 [I 2022-10-10 20:39:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 130/2304 [I 2022-10-10 20:39:33] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:12 [I 2022-10-10 20:39:33] ...trap/engines/engine.py.229: process: 0.85569 | load: 0.00325 [I 2022-10-10 20:39:33] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-10 20:39:33] ...optimizers/trijoint.py.151: total_norm: 8.980177 [I 2022-10-10 20:39:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 140/2304 [I 2022-10-10 20:39:42] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:30:57 [I 2022-10-10 20:39:42] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00325 [I 2022-10-10 20:39:42] ...trap/engines/engine.py.230: loss: 0.13218 [I 2022-10-10 20:39:42] ...optimizers/trijoint.py.151: total_norm: 19.217203 [I 2022-10-10 20:39:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 150/2304 [I 2022-10-10 20:39:50] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:40 [I 2022-10-10 20:39:50] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00305 [I 2022-10-10 20:39:50] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-10 20:39:50] ...optimizers/trijoint.py.151: total_norm: 13.061961 [I 2022-10-10 20:39:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 160/2304 [I 2022-10-10 20:39:59] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:30 [I 2022-10-10 20:39:59] ...trap/engines/engine.py.229: process: 0.85857 | load: 0.00308 [I 2022-10-10 20:39:59] ...trap/engines/engine.py.230: loss: 0.18076 [I 2022-10-10 20:39:59] ...optimizers/trijoint.py.151: total_norm: 23.568190 [I 2022-10-10 20:40:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 170/2304 [I 2022-10-10 20:40:08] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:49 [I 2022-10-10 20:40:08] ...trap/engines/engine.py.229: process: 0.86719 | load: 0.00389 [I 2022-10-10 20:40:08] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-10 20:40:08] ...optimizers/trijoint.py.151: total_norm: 24.788429 [I 2022-10-10 20:40:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 180/2304 [I 2022-10-10 20:40:16] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:19 [I 2022-10-10 20:40:16] ...trap/engines/engine.py.229: process: 0.85654 | load: 0.00316 [I 2022-10-10 20:40:16] ...trap/engines/engine.py.230: loss: 0.09829 [I 2022-10-10 20:40:16] ...optimizers/trijoint.py.151: total_norm: 14.000154 [I 2022-10-10 20:40:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 190/2304 [I 2022-10-10 20:40:25] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:06 [I 2022-10-10 20:40:25] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00303 [I 2022-10-10 20:40:25] ...trap/engines/engine.py.230: loss: 0.14800 [I 2022-10-10 20:40:25] ...optimizers/trijoint.py.151: total_norm: 24.087601 [I 2022-10-10 20:40:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 200/2304 [I 2022-10-10 20:40:33] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:29:55 [I 2022-10-10 20:40:33] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00313 [I 2022-10-10 20:40:33] ...trap/engines/engine.py.230: loss: 0.20191 [I 2022-10-10 20:40:33] ...optimizers/trijoint.py.151: total_norm: 21.091831 [I 2022-10-10 20:40:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 210/2304 [I 2022-10-10 20:40:42] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:39 [I 2022-10-10 20:40:42] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00294 [I 2022-10-10 20:40:42] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-10 20:40:42] ...optimizers/trijoint.py.151: total_norm: 22.590570 [I 2022-10-10 20:40:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 220/2304 [I 2022-10-10 20:40:50] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:37 [I 2022-10-10 20:40:50] ...trap/engines/engine.py.229: process: 0.85237 | load: 0.00301 [I 2022-10-10 20:40:50] ...trap/engines/engine.py.230: loss: 0.12161 [I 2022-10-10 20:40:50] ...optimizers/trijoint.py.151: total_norm: 13.395988 [I 2022-10-10 20:40:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 230/2304 [I 2022-10-10 20:40:59] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:30 [I 2022-10-10 20:40:59] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00299 [I 2022-10-10 20:40:59] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-10 20:40:59] ...optimizers/trijoint.py.151: total_norm: 30.723291 [I 2022-10-10 20:41:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 240/2304 [I 2022-10-10 20:41:08] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:21 [I 2022-10-10 20:41:08] ...trap/engines/engine.py.229: process: 0.86259 | load: 0.00298 [I 2022-10-10 20:41:08] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-10 20:41:08] ...optimizers/trijoint.py.151: total_norm: 8.455139 [I 2022-10-10 20:41:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 250/2304 [I 2022-10-10 20:41:16] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:14 [I 2022-10-10 20:41:16] ...trap/engines/engine.py.229: process: 0.86171 | load: 0.00429 [I 2022-10-10 20:41:16] ...trap/engines/engine.py.230: loss: 0.11531 [I 2022-10-10 20:41:16] ...optimizers/trijoint.py.151: total_norm: 14.679977 [I 2022-10-10 20:41:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 260/2304 [I 2022-10-10 20:41:25] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:04 [I 2022-10-10 20:41:25] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00299 [I 2022-10-10 20:41:25] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-10 20:41:25] ...optimizers/trijoint.py.151: total_norm: 9.900405 [I 2022-10-10 20:41:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 270/2304 [I 2022-10-10 20:41:33] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:28:49 [I 2022-10-10 20:41:33] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00293 [I 2022-10-10 20:41:33] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-10 20:41:33] ...optimizers/trijoint.py.151: total_norm: 19.171103 [I 2022-10-10 20:41:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 280/2304 [I 2022-10-10 20:41:42] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:40 [I 2022-10-10 20:41:42] ...trap/engines/engine.py.229: process: 0.84994 | load: 0.00295 [I 2022-10-10 20:41:42] ...trap/engines/engine.py.230: loss: 0.20455 [I 2022-10-10 20:41:42] ...optimizers/trijoint.py.151: total_norm: 23.244022 [I 2022-10-10 20:41:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 290/2304 [I 2022-10-10 20:41:50] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:36 [I 2022-10-10 20:41:50] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00292 [I 2022-10-10 20:41:50] ...trap/engines/engine.py.230: loss: 0.09051 [I 2022-10-10 20:41:50] ...optimizers/trijoint.py.151: total_norm: 10.381528 [I 2022-10-10 20:41:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 300/2304 [I 2022-10-10 20:41:59] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:28 [I 2022-10-10 20:41:59] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00296 [I 2022-10-10 20:41:59] ...trap/engines/engine.py.230: loss: 0.17178 [I 2022-10-10 20:41:59] ...optimizers/trijoint.py.151: total_norm: 18.578987 [I 2022-10-10 20:42:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 310/2304 [I 2022-10-10 20:42:07] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:20 [I 2022-10-10 20:42:07] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00293 [I 2022-10-10 20:42:07] ...trap/engines/engine.py.230: loss: 0.12508 [I 2022-10-10 20:42:07] ...optimizers/trijoint.py.151: total_norm: 12.785200 [I 2022-10-10 20:42:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 320/2304 [I 2022-10-10 20:42:16] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:29 [I 2022-10-10 20:42:16] ...trap/engines/engine.py.229: process: 0.88245 | load: 0.00315 [I 2022-10-10 20:42:16] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-10 20:42:16] ...optimizers/trijoint.py.151: total_norm: 11.649451 [I 2022-10-10 20:42:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 330/2304 [I 2022-10-10 20:42:24] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:27:55 [I 2022-10-10 20:42:24] ...trap/engines/engine.py.229: process: 0.83565 | load: 0.00310 [I 2022-10-10 20:42:24] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-10 20:42:24] ...optimizers/trijoint.py.151: total_norm: 12.523524 [I 2022-10-10 20:42:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 340/2304 [I 2022-10-10 20:42:33] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:27:51 [I 2022-10-10 20:42:33] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00296 [I 2022-10-10 20:42:33] ...trap/engines/engine.py.230: loss: 0.19892 [I 2022-10-10 20:42:33] ...optimizers/trijoint.py.151: total_norm: 22.344362 [I 2022-10-10 20:42:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 350/2304 [I 2022-10-10 20:42:42] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:58 [I 2022-10-10 20:42:42] ...trap/engines/engine.py.229: process: 0.86877 | load: 0.00298 [I 2022-10-10 20:42:42] ...trap/engines/engine.py.230: loss: 0.11855 [I 2022-10-10 20:42:42] ...optimizers/trijoint.py.151: total_norm: 13.648964 [I 2022-10-10 20:42:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 360/2304 [I 2022-10-10 20:42:50] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:27:49 [I 2022-10-10 20:42:50] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00309 [I 2022-10-10 20:42:50] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-10 20:42:50] ...optimizers/trijoint.py.151: total_norm: 41.596317 [I 2022-10-10 20:42:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 370/2304 [I 2022-10-10 20:42:59] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:54 [I 2022-10-10 20:42:59] ...trap/engines/engine.py.229: process: 0.86827 | load: 0.00314 [I 2022-10-10 20:42:59] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-10 20:42:59] ...optimizers/trijoint.py.151: total_norm: 55.021770 [I 2022-10-10 20:43:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 380/2304 [I 2022-10-10 20:43:07] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:32 [I 2022-10-10 20:43:07] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00307 [I 2022-10-10 20:43:07] ...trap/engines/engine.py.230: loss: 0.10738 [I 2022-10-10 20:43:07] ...optimizers/trijoint.py.151: total_norm: 9.250776 [I 2022-10-10 20:43:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 390/2304 [I 2022-10-10 20:43:16] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:37 [I 2022-10-10 20:43:16] ...trap/engines/engine.py.229: process: 0.86314 | load: 0.00312 [I 2022-10-10 20:43:16] ...trap/engines/engine.py.230: loss: 0.19097 [I 2022-10-10 20:43:16] ...optimizers/trijoint.py.151: total_norm: 48.878216 [I 2022-10-10 20:43:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 400/2304 [I 2022-10-10 20:43:25] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:15 [I 2022-10-10 20:43:25] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00313 [I 2022-10-10 20:43:25] ...trap/engines/engine.py.230: loss: 0.19264 [I 2022-10-10 20:43:25] ...optimizers/trijoint.py.151: total_norm: 15.172274 [I 2022-10-10 20:43:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 410/2304 [I 2022-10-10 20:43:33] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:27 [I 2022-10-10 20:43:33] ...trap/engines/engine.py.229: process: 0.87420 | load: 0.00328 [I 2022-10-10 20:43:33] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-10 20:43:33] ...optimizers/trijoint.py.151: total_norm: 8.058531 [I 2022-10-10 20:43:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 420/2304 [I 2022-10-10 20:43:42] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:56 [I 2022-10-10 20:43:42] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00309 [I 2022-10-10 20:43:42] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-10 20:43:42] ...optimizers/trijoint.py.151: total_norm: 13.129573 [I 2022-10-10 20:43:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 430/2304 [I 2022-10-10 20:43:51] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:00 [I 2022-10-10 20:43:51] ...trap/engines/engine.py.229: process: 0.87067 | load: 0.00311 [I 2022-10-10 20:43:51] ...trap/engines/engine.py.230: loss: 0.15984 [I 2022-10-10 20:43:51] ...optimizers/trijoint.py.151: total_norm: 13.752082 [I 2022-10-10 20:43:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 440/2304 [I 2022-10-10 20:43:59] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:54 [I 2022-10-10 20:43:59] ...trap/engines/engine.py.229: process: 0.88338 | load: 0.00322 [I 2022-10-10 20:43:59] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-10 20:43:59] ...optimizers/trijoint.py.151: total_norm: 13.265518 [I 2022-10-10 20:44:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 450/2304 [I 2022-10-10 20:44:08] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:46 [I 2022-10-10 20:44:08] ...trap/engines/engine.py.229: process: 0.86646 | load: 0.00308 [I 2022-10-10 20:44:08] ...trap/engines/engine.py.230: loss: 0.11506 [I 2022-10-10 20:44:08] ...optimizers/trijoint.py.151: total_norm: 15.053174 [I 2022-10-10 20:44:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 460/2304 [I 2022-10-10 20:44:17] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:34 [I 2022-10-10 20:44:17] ...trap/engines/engine.py.229: process: 0.86943 | load: 0.00309 [I 2022-10-10 20:44:17] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-10 20:44:17] ...optimizers/trijoint.py.151: total_norm: 12.502406 [I 2022-10-10 20:44:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 470/2304 [I 2022-10-10 20:44:25] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:17 [I 2022-10-10 20:44:25] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00317 [I 2022-10-10 20:44:25] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-10 20:44:25] ...optimizers/trijoint.py.151: total_norm: 22.276037 [I 2022-10-10 20:44:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 480/2304 [I 2022-10-10 20:44:34] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:07 [I 2022-10-10 20:44:34] ...trap/engines/engine.py.229: process: 0.85775 | load: 0.00318 [I 2022-10-10 20:44:34] ...trap/engines/engine.py.230: loss: 0.12801 [I 2022-10-10 20:44:34] ...optimizers/trijoint.py.151: total_norm: 27.048174 [I 2022-10-10 20:44:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 490/2304 [I 2022-10-10 20:44:43] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:08 [I 2022-10-10 20:44:43] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00313 [I 2022-10-10 20:44:43] ...trap/engines/engine.py.230: loss: 0.10039 [I 2022-10-10 20:44:43] ...optimizers/trijoint.py.151: total_norm: 9.337438 [I 2022-10-10 20:44:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 500/2304 [I 2022-10-10 20:44:51] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:26:02 [I 2022-10-10 20:44:51] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00308 [I 2022-10-10 20:44:51] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-10 20:44:51] ...optimizers/trijoint.py.151: total_norm: 20.209587 [I 2022-10-10 20:45:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 510/2304 [I 2022-10-10 20:45:00] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:26:02 [I 2022-10-10 20:45:00] ...trap/engines/engine.py.229: process: 0.86771 | load: 0.00314 [I 2022-10-10 20:45:00] ...trap/engines/engine.py.230: loss: 0.11070 [I 2022-10-10 20:45:00] ...optimizers/trijoint.py.151: total_norm: 17.938107 [I 2022-10-10 20:45:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 520/2304 [I 2022-10-10 20:45:09] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:56 [I 2022-10-10 20:45:09] ...trap/engines/engine.py.229: process: 0.87447 | load: 0.00307 [I 2022-10-10 20:45:09] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-10 20:45:09] ...optimizers/trijoint.py.151: total_norm: 18.072979 [I 2022-10-10 20:45:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 530/2304 [I 2022-10-10 20:45:17] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:36 [I 2022-10-10 20:45:17] ...trap/engines/engine.py.229: process: 0.87310 | load: 0.00309 [I 2022-10-10 20:45:17] ...trap/engines/engine.py.230: loss: 0.16107 [I 2022-10-10 20:45:17] ...optimizers/trijoint.py.151: total_norm: 15.825205 [I 2022-10-10 20:45:26] ...trap/engines/engine.py.224: train: epoch 85 | batch 540/2304 [I 2022-10-10 20:45:26] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:26 [I 2022-10-10 20:45:26] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00310 [I 2022-10-10 20:45:26] ...trap/engines/engine.py.230: loss: 0.15727 [I 2022-10-10 20:45:26] ...optimizers/trijoint.py.151: total_norm: 36.186588 [I 2022-10-10 20:45:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 550/2304 [I 2022-10-10 20:45:35] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:16 [I 2022-10-10 20:45:35] ...trap/engines/engine.py.229: process: 0.87478 | load: 0.00308 [I 2022-10-10 20:45:35] ...trap/engines/engine.py.230: loss: 0.18618 [I 2022-10-10 20:45:35] ...optimizers/trijoint.py.151: total_norm: 22.603926 [I 2022-10-10 20:45:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 560/2304 [I 2022-10-10 20:45:43] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:10 [I 2022-10-10 20:45:43] ...trap/engines/engine.py.229: process: 0.86449 | load: 0.00309 [I 2022-10-10 20:45:43] ...trap/engines/engine.py.230: loss: 0.07507 [I 2022-10-10 20:45:43] ...optimizers/trijoint.py.151: total_norm: 11.285728 [I 2022-10-10 20:45:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 570/2304 [I 2022-10-10 20:45:52] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:53 [I 2022-10-10 20:45:52] ...trap/engines/engine.py.229: process: 0.86431 | load: 0.00310 [I 2022-10-10 20:45:52] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-10 20:45:52] ...optimizers/trijoint.py.151: total_norm: 16.359516 [I 2022-10-10 20:46:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 580/2304 [I 2022-10-10 20:46:01] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:40 [I 2022-10-10 20:46:01] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00312 [I 2022-10-10 20:46:01] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-10 20:46:01] ...optimizers/trijoint.py.151: total_norm: 23.781570 [I 2022-10-10 20:46:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 590/2304 [I 2022-10-10 20:46:09] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:37 [I 2022-10-10 20:46:09] ...trap/engines/engine.py.229: process: 0.86999 | load: 0.00514 [I 2022-10-10 20:46:09] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-10 20:46:09] ...optimizers/trijoint.py.151: total_norm: 13.079664 [I 2022-10-10 20:46:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 600/2304 [I 2022-10-10 20:46:18] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:43 [I 2022-10-10 20:46:18] ...trap/engines/engine.py.229: process: 0.87670 | load: 0.00322 [I 2022-10-10 20:46:18] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-10 20:46:18] ...optimizers/trijoint.py.151: total_norm: 9.895108 [I 2022-10-10 20:46:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 610/2304 [I 2022-10-10 20:46:27] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:24 [I 2022-10-10 20:46:27] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00312 [I 2022-10-10 20:46:27] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-10 20:46:27] ...optimizers/trijoint.py.151: total_norm: 20.538097 [I 2022-10-10 20:46:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 620/2304 [I 2022-10-10 20:46:35] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:26 [I 2022-10-10 20:46:35] ...trap/engines/engine.py.229: process: 0.88013 | load: 0.00368 [I 2022-10-10 20:46:35] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-10 20:46:35] ...optimizers/trijoint.py.151: total_norm: 19.617117 [I 2022-10-10 20:46:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 630/2304 [I 2022-10-10 20:46:44] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:24:05 [I 2022-10-10 20:46:44] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00306 [I 2022-10-10 20:46:44] ...trap/engines/engine.py.230: loss: 0.12852 [I 2022-10-10 20:46:44] ...optimizers/trijoint.py.151: total_norm: 14.298629 [I 2022-10-10 20:46:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 640/2304 [I 2022-10-10 20:46:53] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:44 [I 2022-10-10 20:46:53] ...trap/engines/engine.py.229: process: 0.86201 | load: 0.00303 [I 2022-10-10 20:46:53] ...trap/engines/engine.py.230: loss: 0.15082 [I 2022-10-10 20:46:53] ...optimizers/trijoint.py.151: total_norm: 69.988167 [I 2022-10-10 20:47:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 650/2304 [I 2022-10-10 20:47:01] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:39 [I 2022-10-10 20:47:01] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00305 [I 2022-10-10 20:47:01] ...trap/engines/engine.py.230: loss: 0.10641 [I 2022-10-10 20:47:01] ...optimizers/trijoint.py.151: total_norm: 12.320765 [I 2022-10-10 20:47:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 660/2304 [I 2022-10-10 20:47:10] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:20 [I 2022-10-10 20:47:10] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00298 [I 2022-10-10 20:47:10] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-10 20:47:10] ...optimizers/trijoint.py.151: total_norm: 14.862371 [I 2022-10-10 20:47:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 670/2304 [I 2022-10-10 20:47:18] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:16 [I 2022-10-10 20:47:18] ...trap/engines/engine.py.229: process: 0.86127 | load: 0.00300 [I 2022-10-10 20:47:18] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-10 20:47:18] ...optimizers/trijoint.py.151: total_norm: 17.614134 [I 2022-10-10 20:47:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 680/2304 [I 2022-10-10 20:47:27] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:22:58 [I 2022-10-10 20:47:27] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00295 [I 2022-10-10 20:47:27] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-10 20:47:27] ...optimizers/trijoint.py.151: total_norm: 14.632465 [I 2022-10-10 20:47:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 690/2304 [I 2022-10-10 20:47:35] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:06 [I 2022-10-10 20:47:35] ...trap/engines/engine.py.229: process: 0.86155 | load: 0.00305 [I 2022-10-10 20:47:35] ...trap/engines/engine.py.230: loss: 0.08550 [I 2022-10-10 20:47:35] ...optimizers/trijoint.py.151: total_norm: 16.968550 [I 2022-10-10 20:47:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 700/2304 [I 2022-10-10 20:47:44] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:53 [I 2022-10-10 20:47:44] ...trap/engines/engine.py.229: process: 0.85864 | load: 0.00294 [I 2022-10-10 20:47:44] ...trap/engines/engine.py.230: loss: 0.16702 [I 2022-10-10 20:47:44] ...optimizers/trijoint.py.151: total_norm: 15.750553 [I 2022-10-10 20:47:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 710/2304 [I 2022-10-10 20:47:53] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:40 [I 2022-10-10 20:47:53] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00303 [I 2022-10-10 20:47:53] ...trap/engines/engine.py.230: loss: 0.08346 [I 2022-10-10 20:47:53] ...optimizers/trijoint.py.151: total_norm: 16.633001 [I 2022-10-10 20:48:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 720/2304 [I 2022-10-10 20:48:01] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:42 [I 2022-10-10 20:48:01] ...trap/engines/engine.py.229: process: 0.86737 | load: 0.00305 [I 2022-10-10 20:48:01] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-10 20:48:01] ...optimizers/trijoint.py.151: total_norm: 23.098734 [I 2022-10-10 20:48:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 730/2304 [I 2022-10-10 20:48:10] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:24 [I 2022-10-10 20:48:10] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00294 [I 2022-10-10 20:48:10] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-10 20:48:10] ...optimizers/trijoint.py.151: total_norm: 9.786961 [I 2022-10-10 20:48:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 740/2304 [I 2022-10-10 20:48:18] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:17 [I 2022-10-10 20:48:18] ...trap/engines/engine.py.229: process: 0.85876 | load: 0.00298 [I 2022-10-10 20:48:18] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-10 20:48:18] ...optimizers/trijoint.py.151: total_norm: 9.848207 [I 2022-10-10 20:48:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 750/2304 [I 2022-10-10 20:48:27] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:22:06 [I 2022-10-10 20:48:27] ...trap/engines/engine.py.229: process: 0.87207 | load: 0.00301 [I 2022-10-10 20:48:27] ...trap/engines/engine.py.230: loss: 0.11871 [I 2022-10-10 20:48:27] ...optimizers/trijoint.py.151: total_norm: 14.188869 [I 2022-10-10 20:48:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 760/2304 [I 2022-10-10 20:48:35] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:09 [I 2022-10-10 20:48:35] ...trap/engines/engine.py.229: process: 0.86379 | load: 0.00307 [I 2022-10-10 20:48:35] ...trap/engines/engine.py.230: loss: 0.22423 [I 2022-10-10 20:48:35] ...optimizers/trijoint.py.151: total_norm: 22.925114 [I 2022-10-10 20:48:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 770/2304 [I 2022-10-10 20:48:44] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:01 [I 2022-10-10 20:48:44] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00308 [I 2022-10-10 20:48:44] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-10 20:48:44] ...optimizers/trijoint.py.151: total_norm: 9.884533 [I 2022-10-10 20:48:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 780/2304 [I 2022-10-10 20:48:53] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:44 [I 2022-10-10 20:48:53] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00302 [I 2022-10-10 20:48:53] ...trap/engines/engine.py.230: loss: 0.12390 [I 2022-10-10 20:48:53] ...optimizers/trijoint.py.151: total_norm: 16.526520 [I 2022-10-10 20:49:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 790/2304 [I 2022-10-10 20:49:01] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:34 [I 2022-10-10 20:49:01] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00298 [I 2022-10-10 20:49:01] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-10 20:49:01] ...optimizers/trijoint.py.151: total_norm: 19.685696 [I 2022-10-10 20:49:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 800/2304 [I 2022-10-10 20:49:10] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:28 [I 2022-10-10 20:49:10] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00301 [I 2022-10-10 20:49:10] ...trap/engines/engine.py.230: loss: 0.13008 [I 2022-10-10 20:49:10] ...optimizers/trijoint.py.151: total_norm: 17.236855 [I 2022-10-10 20:49:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 810/2304 [I 2022-10-10 20:49:18] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:20 [I 2022-10-10 20:49:18] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00297 [I 2022-10-10 20:49:18] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-10 20:49:18] ...optimizers/trijoint.py.151: total_norm: 15.867500 [I 2022-10-10 20:49:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 820/2304 [I 2022-10-10 20:49:27] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:13 [I 2022-10-10 20:49:27] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00314 [I 2022-10-10 20:49:27] ...trap/engines/engine.py.230: loss: 0.10086 [I 2022-10-10 20:49:27] ...optimizers/trijoint.py.151: total_norm: 19.410437 [I 2022-10-10 20:49:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 830/2304 [I 2022-10-10 20:49:36] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:59 [I 2022-10-10 20:49:36] ...trap/engines/engine.py.229: process: 0.85606 | load: 0.00297 [I 2022-10-10 20:49:36] ...trap/engines/engine.py.230: loss: 0.12446 [I 2022-10-10 20:49:36] ...optimizers/trijoint.py.151: total_norm: 14.117202 [I 2022-10-10 20:49:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 840/2304 [I 2022-10-10 20:49:44] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:55 [I 2022-10-10 20:49:44] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00305 [I 2022-10-10 20:49:44] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-10 20:49:44] ...optimizers/trijoint.py.151: total_norm: 16.627691 [I 2022-10-10 20:49:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 850/2304 [I 2022-10-10 20:49:53] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:45 [I 2022-10-10 20:49:53] ...trap/engines/engine.py.229: process: 0.87619 | load: 0.00299 [I 2022-10-10 20:49:53] ...trap/engines/engine.py.230: loss: 0.19860 [I 2022-10-10 20:49:53] ...optimizers/trijoint.py.151: total_norm: 18.386971 [I 2022-10-10 20:50:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 860/2304 [I 2022-10-10 20:50:01] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:28 [I 2022-10-10 20:50:01] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00298 [I 2022-10-10 20:50:01] ...trap/engines/engine.py.230: loss: 0.11941 [I 2022-10-10 20:50:01] ...optimizers/trijoint.py.151: total_norm: 18.471687 [I 2022-10-10 20:50:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 870/2304 [I 2022-10-10 20:50:10] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:30 [I 2022-10-10 20:50:10] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00296 [I 2022-10-10 20:50:10] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-10 20:50:10] ...optimizers/trijoint.py.151: total_norm: 11.915741 [I 2022-10-10 20:50:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 880/2304 [I 2022-10-10 20:50:18] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:10 [I 2022-10-10 20:50:18] ...trap/engines/engine.py.229: process: 0.82408 | load: 0.00298 [I 2022-10-10 20:50:18] ...trap/engines/engine.py.230: loss: 0.21595 [I 2022-10-10 20:50:18] ...optimizers/trijoint.py.151: total_norm: 14.995500 [I 2022-10-10 20:50:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 890/2304 [I 2022-10-10 20:50:27] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:20:07 [I 2022-10-10 20:50:27] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00301 [I 2022-10-10 20:50:27] ...trap/engines/engine.py.230: loss: 0.12532 [I 2022-10-10 20:50:27] ...optimizers/trijoint.py.151: total_norm: 19.886595 [I 2022-10-10 20:50:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 900/2304 [I 2022-10-10 20:50:36] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:20:00 [I 2022-10-10 20:50:36] ...trap/engines/engine.py.229: process: 0.85838 | load: 0.00303 [I 2022-10-10 20:50:36] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-10 20:50:36] ...optimizers/trijoint.py.151: total_norm: 16.163126 [I 2022-10-10 20:50:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 910/2304 [I 2022-10-10 20:50:44] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:55 [I 2022-10-10 20:50:44] ...trap/engines/engine.py.229: process: 0.86454 | load: 0.00300 [I 2022-10-10 20:50:44] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-10 20:50:44] ...optimizers/trijoint.py.151: total_norm: 22.105070 [I 2022-10-10 20:50:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 920/2304 [I 2022-10-10 20:50:53] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:43 [I 2022-10-10 20:50:53] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00299 [I 2022-10-10 20:50:53] ...trap/engines/engine.py.230: loss: 0.15013 [I 2022-10-10 20:50:53] ...optimizers/trijoint.py.151: total_norm: 17.444014 [I 2022-10-10 20:51:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 930/2304 [I 2022-10-10 20:51:01] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:33 [I 2022-10-10 20:51:01] ...trap/engines/engine.py.229: process: 0.85623 | load: 0.00306 [I 2022-10-10 20:51:01] ...trap/engines/engine.py.230: loss: 0.17915 [I 2022-10-10 20:51:01] ...optimizers/trijoint.py.151: total_norm: 11.283895 [I 2022-10-10 20:51:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 940/2304 [I 2022-10-10 20:51:10] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:24 [I 2022-10-10 20:51:10] ...trap/engines/engine.py.229: process: 0.86061 | load: 0.00320 [I 2022-10-10 20:51:10] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-10 20:51:10] ...optimizers/trijoint.py.151: total_norm: 11.289876 [I 2022-10-10 20:51:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 950/2304 [I 2022-10-10 20:51:19] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:26 [I 2022-10-10 20:51:19] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00317 [I 2022-10-10 20:51:19] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-10 20:51:19] ...optimizers/trijoint.py.151: total_norm: 20.763845 [I 2022-10-10 20:51:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 960/2304 [I 2022-10-10 20:51:27] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:21 [I 2022-10-10 20:51:27] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00329 [I 2022-10-10 20:51:27] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-10 20:51:27] ...optimizers/trijoint.py.151: total_norm: 31.700270 [I 2022-10-10 20:51:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 970/2304 [I 2022-10-10 20:51:36] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:14 [I 2022-10-10 20:51:36] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00316 [I 2022-10-10 20:51:36] ...trap/engines/engine.py.230: loss: 0.12030 [I 2022-10-10 20:51:36] ...optimizers/trijoint.py.151: total_norm: 19.134985 [I 2022-10-10 20:51:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 980/2304 [I 2022-10-10 20:51:44] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:19:00 [I 2022-10-10 20:51:44] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00308 [I 2022-10-10 20:51:44] ...trap/engines/engine.py.230: loss: 0.16527 [I 2022-10-10 20:51:44] ...optimizers/trijoint.py.151: total_norm: 18.465313 [I 2022-10-10 20:51:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 990/2304 [I 2022-10-10 20:51:53] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:52 [I 2022-10-10 20:51:53] ...trap/engines/engine.py.229: process: 0.85695 | load: 0.00310 [I 2022-10-10 20:51:53] ...trap/engines/engine.py.230: loss: 0.10463 [I 2022-10-10 20:51:53] ...optimizers/trijoint.py.151: total_norm: 8.141145 [I 2022-10-10 20:52:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 1000/2304 [I 2022-10-10 20:52:02] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:46 [I 2022-10-10 20:52:02] ...trap/engines/engine.py.229: process: 0.87370 | load: 0.00345 [I 2022-10-10 20:52:02] ...trap/engines/engine.py.230: loss: 0.15155 [I 2022-10-10 20:52:02] ...optimizers/trijoint.py.151: total_norm: 27.573519 [I 2022-10-10 20:52:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 1010/2304 [I 2022-10-10 20:52:11] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:41 [I 2022-10-10 20:52:11] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00313 [I 2022-10-10 20:52:11] ...trap/engines/engine.py.230: loss: 0.19363 [I 2022-10-10 20:52:11] ...optimizers/trijoint.py.151: total_norm: 10.709509 [I 2022-10-10 20:52:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 1020/2304 [I 2022-10-10 20:52:19] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:33 [I 2022-10-10 20:52:19] ...trap/engines/engine.py.229: process: 0.87205 | load: 0.00313 [I 2022-10-10 20:52:19] ...trap/engines/engine.py.230: loss: 0.10761 [I 2022-10-10 20:52:19] ...optimizers/trijoint.py.151: total_norm: 12.265218 [I 2022-10-10 20:52:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 1030/2304 [I 2022-10-10 20:52:28] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:22 [I 2022-10-10 20:52:28] ...trap/engines/engine.py.229: process: 0.86226 | load: 0.00307 [I 2022-10-10 20:52:28] ...trap/engines/engine.py.230: loss: 0.17371 [I 2022-10-10 20:52:28] ...optimizers/trijoint.py.151: total_norm: 25.467514 [I 2022-10-10 20:52:37] ...trap/engines/engine.py.224: train: epoch 85 | batch 1040/2304 [I 2022-10-10 20:52:37] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:18:20 [I 2022-10-10 20:52:37] ...trap/engines/engine.py.229: process: 0.87358 | load: 0.00313 [I 2022-10-10 20:52:37] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-10 20:52:37] ...optimizers/trijoint.py.151: total_norm: 16.375977 [I 2022-10-10 20:52:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 1050/2304 [I 2022-10-10 20:52:45] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:18:08 [I 2022-10-10 20:52:45] ...trap/engines/engine.py.229: process: 0.88455 | load: 0.00312 [I 2022-10-10 20:52:45] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-10 20:52:45] ...optimizers/trijoint.py.151: total_norm: 13.948330 [I 2022-10-10 20:52:54] ...trap/engines/engine.py.224: train: epoch 85 | batch 1060/2304 [I 2022-10-10 20:52:54] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:54 [I 2022-10-10 20:52:54] ...trap/engines/engine.py.229: process: 0.86443 | load: 0.00317 [I 2022-10-10 20:52:54] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-10 20:52:54] ...optimizers/trijoint.py.151: total_norm: 9.325204 [I 2022-10-10 20:53:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 1070/2304 [I 2022-10-10 20:53:03] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:43 [I 2022-10-10 20:53:03] ...trap/engines/engine.py.229: process: 0.86218 | load: 0.00311 [I 2022-10-10 20:53:03] ...trap/engines/engine.py.230: loss: 0.25605 [I 2022-10-10 20:53:03] ...optimizers/trijoint.py.151: total_norm: 19.152317 [I 2022-10-10 20:53:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 1080/2304 [I 2022-10-10 20:53:11] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:33 [I 2022-10-10 20:53:11] ...trap/engines/engine.py.229: process: 0.86831 | load: 0.00308 [I 2022-10-10 20:53:11] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-10 20:53:11] ...optimizers/trijoint.py.151: total_norm: 14.178759 [I 2022-10-10 20:53:20] ...trap/engines/engine.py.224: train: epoch 85 | batch 1090/2304 [I 2022-10-10 20:53:20] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:32 [I 2022-10-10 20:53:20] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00307 [I 2022-10-10 20:53:20] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-10 20:53:20] ...optimizers/trijoint.py.151: total_norm: 15.817678 [I 2022-10-10 20:53:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 1100/2304 [I 2022-10-10 20:53:29] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:17:24 [I 2022-10-10 20:53:29] ...trap/engines/engine.py.229: process: 0.86484 | load: 0.00309 [I 2022-10-10 20:53:29] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-10 20:53:29] ...optimizers/trijoint.py.151: total_norm: 20.821281 [I 2022-10-10 20:53:37] ...trap/engines/engine.py.224: train: epoch 85 | batch 1110/2304 [I 2022-10-10 20:53:37] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:17:07 [I 2022-10-10 20:53:37] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00399 [I 2022-10-10 20:53:37] ...trap/engines/engine.py.230: loss: 0.16546 [I 2022-10-10 20:53:37] ...optimizers/trijoint.py.151: total_norm: 20.519056 [I 2022-10-10 20:53:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 1120/2304 [I 2022-10-10 20:53:46] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:17:05 [I 2022-10-10 20:53:46] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00309 [I 2022-10-10 20:53:46] ...trap/engines/engine.py.230: loss: 0.09404 [I 2022-10-10 20:53:46] ...optimizers/trijoint.py.151: total_norm: 8.258401 [I 2022-10-10 20:53:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 1130/2304 [I 2022-10-10 20:53:55] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:52 [I 2022-10-10 20:53:55] ...trap/engines/engine.py.229: process: 0.86852 | load: 0.00306 [I 2022-10-10 20:53:55] ...trap/engines/engine.py.230: loss: 0.08880 [I 2022-10-10 20:53:55] ...optimizers/trijoint.py.151: total_norm: 11.590731 [I 2022-10-10 20:54:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 1140/2304 [I 2022-10-10 20:54:03] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:41 [I 2022-10-10 20:54:03] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00314 [I 2022-10-10 20:54:03] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-10 20:54:03] ...optimizers/trijoint.py.151: total_norm: 22.904480 [I 2022-10-10 20:54:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 1150/2304 [I 2022-10-10 20:54:12] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:35 [I 2022-10-10 20:54:12] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00314 [I 2022-10-10 20:54:12] ...trap/engines/engine.py.230: loss: 0.09308 [I 2022-10-10 20:54:12] ...optimizers/trijoint.py.151: total_norm: 13.246217 [I 2022-10-10 20:54:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 1160/2304 [I 2022-10-10 20:54:21] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:27 [I 2022-10-10 20:54:21] ...trap/engines/engine.py.229: process: 0.86073 | load: 0.00306 [I 2022-10-10 20:54:21] ...trap/engines/engine.py.230: loss: 0.09687 [I 2022-10-10 20:54:21] ...optimizers/trijoint.py.151: total_norm: 9.677169 [I 2022-10-10 20:54:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 1170/2304 [I 2022-10-10 20:54:29] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:18 [I 2022-10-10 20:54:29] ...trap/engines/engine.py.229: process: 0.86921 | load: 0.00310 [I 2022-10-10 20:54:29] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-10 20:54:29] ...optimizers/trijoint.py.151: total_norm: 15.799238 [I 2022-10-10 20:54:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 1180/2304 [I 2022-10-10 20:54:38] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:16:13 [I 2022-10-10 20:54:38] ...trap/engines/engine.py.229: process: 0.87254 | load: 0.00311 [I 2022-10-10 20:54:38] ...trap/engines/engine.py.230: loss: 0.10874 [I 2022-10-10 20:54:38] ...optimizers/trijoint.py.151: total_norm: 12.061542 [I 2022-10-10 20:54:47] ...trap/engines/engine.py.224: train: epoch 85 | batch 1190/2304 [I 2022-10-10 20:54:47] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:16:04 [I 2022-10-10 20:54:47] ...trap/engines/engine.py.229: process: 0.87473 | load: 0.00310 [I 2022-10-10 20:54:47] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-10 20:54:47] ...optimizers/trijoint.py.151: total_norm: 12.890435 [I 2022-10-10 20:54:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 1200/2304 [I 2022-10-10 20:54:55] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:52 [I 2022-10-10 20:54:55] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00307 [I 2022-10-10 20:54:55] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-10 20:54:55] ...optimizers/trijoint.py.151: total_norm: 26.607975 [I 2022-10-10 20:55:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 1210/2304 [I 2022-10-10 20:55:04] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:47 [I 2022-10-10 20:55:04] ...trap/engines/engine.py.229: process: 0.87335 | load: 0.00308 [I 2022-10-10 20:55:04] ...trap/engines/engine.py.230: loss: 0.08765 [I 2022-10-10 20:55:04] ...optimizers/trijoint.py.151: total_norm: 9.535519 [I 2022-10-10 20:55:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 1220/2304 [I 2022-10-10 20:55:12] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:36 [I 2022-10-10 20:55:12] ...trap/engines/engine.py.229: process: 0.87131 | load: 0.00303 [I 2022-10-10 20:55:12] ...trap/engines/engine.py.230: loss: 0.10989 [I 2022-10-10 20:55:12] ...optimizers/trijoint.py.151: total_norm: 11.512721 [I 2022-10-10 20:55:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 1230/2304 [I 2022-10-10 20:55:21] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:30 [I 2022-10-10 20:55:21] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00309 [I 2022-10-10 20:55:21] ...trap/engines/engine.py.230: loss: 0.18187 [I 2022-10-10 20:55:21] ...optimizers/trijoint.py.151: total_norm: 34.169010 [I 2022-10-10 20:55:30] ...trap/engines/engine.py.224: train: epoch 85 | batch 1240/2304 [I 2022-10-10 20:55:30] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:15 [I 2022-10-10 20:55:30] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00310 [I 2022-10-10 20:55:30] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-10 20:55:30] ...optimizers/trijoint.py.151: total_norm: 25.423660 [I 2022-10-10 20:55:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 1250/2304 [I 2022-10-10 20:55:38] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:10 [I 2022-10-10 20:55:38] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00315 [I 2022-10-10 20:55:38] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-10 20:55:38] ...optimizers/trijoint.py.151: total_norm: 19.266962 [I 2022-10-10 20:55:47] ...trap/engines/engine.py.224: train: epoch 85 | batch 1260/2304 [I 2022-10-10 20:55:47] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:59 [I 2022-10-10 20:55:47] ...trap/engines/engine.py.229: process: 0.87640 | load: 0.00311 [I 2022-10-10 20:55:47] ...trap/engines/engine.py.230: loss: 0.13623 [I 2022-10-10 20:55:47] ...optimizers/trijoint.py.151: total_norm: 9.333074 [I 2022-10-10 20:55:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 1270/2304 [I 2022-10-10 20:55:56] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:14:48 [I 2022-10-10 20:55:56] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00325 [I 2022-10-10 20:55:56] ...trap/engines/engine.py.230: loss: 0.12500 [I 2022-10-10 20:55:56] ...optimizers/trijoint.py.151: total_norm: 7.109568 [I 2022-10-10 20:56:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 1280/2304 [I 2022-10-10 20:56:04] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:39 [I 2022-10-10 20:56:04] ...trap/engines/engine.py.229: process: 0.83981 | load: 0.00321 [I 2022-10-10 20:56:04] ...trap/engines/engine.py.230: loss: 0.09993 [I 2022-10-10 20:56:04] ...optimizers/trijoint.py.151: total_norm: 8.034678 [I 2022-10-10 20:56:13] ...trap/engines/engine.py.224: train: epoch 85 | batch 1290/2304 [I 2022-10-10 20:56:13] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:43 [I 2022-10-10 20:56:13] ...trap/engines/engine.py.229: process: 0.89150 | load: 0.00330 [I 2022-10-10 20:56:13] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-10 20:56:13] ...optimizers/trijoint.py.151: total_norm: 16.962980 [I 2022-10-10 20:56:22] ...trap/engines/engine.py.224: train: epoch 85 | batch 1300/2304 [I 2022-10-10 20:56:22] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:31 [I 2022-10-10 20:56:22] ...trap/engines/engine.py.229: process: 0.86921 | load: 0.00336 [I 2022-10-10 20:56:22] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-10 20:56:22] ...optimizers/trijoint.py.151: total_norm: 11.466660 [I 2022-10-10 20:56:30] ...trap/engines/engine.py.224: train: epoch 85 | batch 1310/2304 [I 2022-10-10 20:56:30] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:14:25 [I 2022-10-10 20:56:30] ...trap/engines/engine.py.229: process: 0.88135 | load: 0.00308 [I 2022-10-10 20:56:30] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-10 20:56:30] ...optimizers/trijoint.py.151: total_norm: 17.380262 [I 2022-10-10 20:56:39] ...trap/engines/engine.py.224: train: epoch 85 | batch 1320/2304 [I 2022-10-10 20:56:39] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:09 [I 2022-10-10 20:56:39] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00307 [I 2022-10-10 20:56:39] ...trap/engines/engine.py.230: loss: 0.10996 [I 2022-10-10 20:56:39] ...optimizers/trijoint.py.151: total_norm: 21.487276 [I 2022-10-10 20:56:48] ...trap/engines/engine.py.224: train: epoch 85 | batch 1330/2304 [I 2022-10-10 20:56:48] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:59 [I 2022-10-10 20:56:48] ...trap/engines/engine.py.229: process: 0.85648 | load: 0.00308 [I 2022-10-10 20:56:48] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-10 20:56:48] ...optimizers/trijoint.py.151: total_norm: 15.176473 [I 2022-10-10 20:56:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 1340/2304 [I 2022-10-10 20:56:56] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:52 [I 2022-10-10 20:56:56] ...trap/engines/engine.py.229: process: 0.87295 | load: 0.00308 [I 2022-10-10 20:56:56] ...trap/engines/engine.py.230: loss: 0.09589 [I 2022-10-10 20:56:56] ...optimizers/trijoint.py.151: total_norm: 20.439741 [I 2022-10-10 20:57:05] ...trap/engines/engine.py.224: train: epoch 85 | batch 1350/2304 [I 2022-10-10 20:57:05] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:44 [I 2022-10-10 20:57:05] ...trap/engines/engine.py.229: process: 0.87066 | load: 0.00307 [I 2022-10-10 20:57:05] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-10 20:57:05] ...optimizers/trijoint.py.151: total_norm: 12.815997 [I 2022-10-10 20:57:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 1360/2304 [I 2022-10-10 20:57:14] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:35 [I 2022-10-10 20:57:14] ...trap/engines/engine.py.229: process: 0.85403 | load: 0.00312 [I 2022-10-10 20:57:14] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-10 20:57:14] ...optimizers/trijoint.py.151: total_norm: 8.787123 [I 2022-10-10 20:57:22] ...trap/engines/engine.py.224: train: epoch 85 | batch 1370/2304 [I 2022-10-10 20:57:22] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:27 [I 2022-10-10 20:57:22] ...trap/engines/engine.py.229: process: 0.86933 | load: 0.00313 [I 2022-10-10 20:57:22] ...trap/engines/engine.py.230: loss: 0.10584 [I 2022-10-10 20:57:22] ...optimizers/trijoint.py.151: total_norm: 12.548886 [I 2022-10-10 20:57:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 1380/2304 [I 2022-10-10 20:57:31] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:22 [I 2022-10-10 20:57:31] ...trap/engines/engine.py.229: process: 0.86223 | load: 0.00316 [I 2022-10-10 20:57:31] ...trap/engines/engine.py.230: loss: 0.10269 [I 2022-10-10 20:57:31] ...optimizers/trijoint.py.151: total_norm: 11.792135 [I 2022-10-10 20:57:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 1390/2304 [I 2022-10-10 20:57:40] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:13:05 [I 2022-10-10 20:57:40] ...trap/engines/engine.py.229: process: 0.86128 | load: 0.00312 [I 2022-10-10 20:57:40] ...trap/engines/engine.py.230: loss: 0.14303 [I 2022-10-10 20:57:40] ...optimizers/trijoint.py.151: total_norm: 30.369011 [I 2022-10-10 20:57:48] ...trap/engines/engine.py.224: train: epoch 85 | batch 1400/2304 [I 2022-10-10 20:57:48] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:13:03 [I 2022-10-10 20:57:48] ...trap/engines/engine.py.229: process: 0.86627 | load: 0.00314 [I 2022-10-10 20:57:48] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-10 20:57:48] ...optimizers/trijoint.py.151: total_norm: 12.560524 [I 2022-10-10 20:57:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 1410/2304 [I 2022-10-10 20:57:57] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:55 [I 2022-10-10 20:57:57] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00311 [I 2022-10-10 20:57:57] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-10 20:57:57] ...optimizers/trijoint.py.151: total_norm: 20.234402 [I 2022-10-10 20:58:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 1420/2304 [I 2022-10-10 20:58:06] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:40 [I 2022-10-10 20:58:06] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00311 [I 2022-10-10 20:58:06] ...trap/engines/engine.py.230: loss: 0.15434 [I 2022-10-10 20:58:06] ...optimizers/trijoint.py.151: total_norm: 24.782406 [I 2022-10-10 20:58:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 1430/2304 [I 2022-10-10 20:58:14] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:28 [I 2022-10-10 20:58:14] ...trap/engines/engine.py.229: process: 0.87255 | load: 0.00310 [I 2022-10-10 20:58:14] ...trap/engines/engine.py.230: loss: 0.10017 [I 2022-10-10 20:58:14] ...optimizers/trijoint.py.151: total_norm: 11.409493 [I 2022-10-10 20:58:23] ...trap/engines/engine.py.224: train: epoch 85 | batch 1440/2304 [I 2022-10-10 20:58:23] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:12:22 [I 2022-10-10 20:58:23] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00312 [I 2022-10-10 20:58:23] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-10 20:58:23] ...optimizers/trijoint.py.151: total_norm: 19.747511 [I 2022-10-10 20:58:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 1450/2304 [I 2022-10-10 20:58:31] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:11 [I 2022-10-10 20:58:31] ...trap/engines/engine.py.229: process: 0.86055 | load: 0.00316 [I 2022-10-10 20:58:31] ...trap/engines/engine.py.230: loss: 0.09368 [I 2022-10-10 20:58:31] ...optimizers/trijoint.py.151: total_norm: 10.778896 [I 2022-10-10 20:58:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 1460/2304 [I 2022-10-10 20:58:40] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:04 [I 2022-10-10 20:58:40] ...trap/engines/engine.py.229: process: 0.86953 | load: 0.00311 [I 2022-10-10 20:58:40] ...trap/engines/engine.py.230: loss: 0.10632 [I 2022-10-10 20:58:40] ...optimizers/trijoint.py.151: total_norm: 17.612190 [I 2022-10-10 20:58:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 1470/2304 [I 2022-10-10 20:58:49] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:12:00 [I 2022-10-10 20:58:49] ...trap/engines/engine.py.229: process: 0.86530 | load: 0.00314 [I 2022-10-10 20:58:49] ...trap/engines/engine.py.230: loss: 0.16145 [I 2022-10-10 20:58:49] ...optimizers/trijoint.py.151: total_norm: 20.887436 [I 2022-10-10 20:58:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 1480/2304 [I 2022-10-10 20:58:57] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:54 [I 2022-10-10 20:58:57] ...trap/engines/engine.py.229: process: 0.87929 | load: 0.00310 [I 2022-10-10 20:58:57] ...trap/engines/engine.py.230: loss: 0.14142 [I 2022-10-10 20:58:57] ...optimizers/trijoint.py.151: total_norm: 37.082611 [I 2022-10-10 20:59:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 1490/2304 [I 2022-10-10 20:59:06] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:46 [I 2022-10-10 20:59:06] ...trap/engines/engine.py.229: process: 0.86381 | load: 0.00318 [I 2022-10-10 20:59:06] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-10 20:59:06] ...optimizers/trijoint.py.151: total_norm: 23.821445 [I 2022-10-10 20:59:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 1500/2304 [I 2022-10-10 20:59:15] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:31 [I 2022-10-10 20:59:15] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00308 [I 2022-10-10 20:59:15] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-10 20:59:15] ...optimizers/trijoint.py.151: total_norm: 14.437805 [I 2022-10-10 20:59:23] ...trap/engines/engine.py.224: train: epoch 85 | batch 1510/2304 [I 2022-10-10 20:59:23] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:26 [I 2022-10-10 20:59:23] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00313 [I 2022-10-10 20:59:23] ...trap/engines/engine.py.230: loss: 0.13568 [I 2022-10-10 20:59:23] ...optimizers/trijoint.py.151: total_norm: 19.957651 [I 2022-10-10 20:59:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 1520/2304 [I 2022-10-10 20:59:32] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:12 [I 2022-10-10 20:59:32] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00311 [I 2022-10-10 20:59:32] ...trap/engines/engine.py.230: loss: 0.16666 [I 2022-10-10 20:59:32] ...optimizers/trijoint.py.151: total_norm: 20.694857 [I 2022-10-10 20:59:41] ...trap/engines/engine.py.224: train: epoch 85 | batch 1530/2304 [I 2022-10-10 20:59:41] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:11:09 [I 2022-10-10 20:59:41] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00311 [I 2022-10-10 20:59:41] ...trap/engines/engine.py.230: loss: 0.20436 [I 2022-10-10 20:59:41] ...optimizers/trijoint.py.151: total_norm: 16.478838 [I 2022-10-10 20:59:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 1540/2304 [I 2022-10-10 20:59:49] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:58 [I 2022-10-10 20:59:49] ...trap/engines/engine.py.229: process: 0.86791 | load: 0.00303 [I 2022-10-10 20:59:49] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-10 20:59:49] ...optimizers/trijoint.py.151: total_norm: 9.435243 [I 2022-10-10 20:59:58] ...trap/engines/engine.py.224: train: epoch 85 | batch 1550/2304 [I 2022-10-10 20:59:58] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:55 [I 2022-10-10 20:59:58] ...trap/engines/engine.py.229: process: 0.87434 | load: 0.00319 [I 2022-10-10 20:59:58] ...trap/engines/engine.py.230: loss: 0.08573 [I 2022-10-10 20:59:58] ...optimizers/trijoint.py.151: total_norm: 9.936750 [I 2022-10-10 21:00:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 1560/2304 [I 2022-10-10 21:00:07] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:43 [I 2022-10-10 21:00:07] ...trap/engines/engine.py.229: process: 0.86884 | load: 0.00309 [I 2022-10-10 21:00:07] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-10 21:00:07] ...optimizers/trijoint.py.151: total_norm: 7.123456 [I 2022-10-10 21:00:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 1570/2304 [I 2022-10-10 21:00:15] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:38 [I 2022-10-10 21:00:15] ...trap/engines/engine.py.229: process: 0.86466 | load: 0.00307 [I 2022-10-10 21:00:15] ...trap/engines/engine.py.230: loss: 0.13629 [I 2022-10-10 21:00:15] ...optimizers/trijoint.py.151: total_norm: 14.059896 [I 2022-10-10 21:00:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 1580/2304 [I 2022-10-10 21:00:24] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:25 [I 2022-10-10 21:00:24] ...trap/engines/engine.py.229: process: 0.86128 | load: 0.00314 [I 2022-10-10 21:00:24] ...trap/engines/engine.py.230: loss: 0.10765 [I 2022-10-10 21:00:24] ...optimizers/trijoint.py.151: total_norm: 9.805941 [I 2022-10-10 21:00:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 1590/2304 [I 2022-10-10 21:00:33] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:20 [I 2022-10-10 21:00:33] ...trap/engines/engine.py.229: process: 0.88011 | load: 0.00315 [I 2022-10-10 21:00:33] ...trap/engines/engine.py.230: loss: 0.16096 [I 2022-10-10 21:00:33] ...optimizers/trijoint.py.151: total_norm: 32.939957 [I 2022-10-10 21:00:41] ...trap/engines/engine.py.224: train: epoch 85 | batch 1600/2304 [I 2022-10-10 21:00:41] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:10:07 [I 2022-10-10 21:00:41] ...trap/engines/engine.py.229: process: 0.86086 | load: 0.00327 [I 2022-10-10 21:00:41] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-10 21:00:41] ...optimizers/trijoint.py.151: total_norm: 16.922695 [I 2022-10-10 21:00:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 1610/2304 [I 2022-10-10 21:00:50] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:10:05 [I 2022-10-10 21:00:50] ...trap/engines/engine.py.229: process: 0.87598 | load: 0.00306 [I 2022-10-10 21:00:50] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-10 21:00:50] ...optimizers/trijoint.py.151: total_norm: 11.515107 [I 2022-10-10 21:00:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 1620/2304 [I 2022-10-10 21:00:59] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:50 [I 2022-10-10 21:00:59] ...trap/engines/engine.py.229: process: 0.86302 | load: 0.00317 [I 2022-10-10 21:00:59] ...trap/engines/engine.py.230: loss: 0.09296 [I 2022-10-10 21:00:59] ...optimizers/trijoint.py.151: total_norm: 22.250790 [I 2022-10-10 21:01:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 1630/2304 [I 2022-10-10 21:01:07] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:38 [I 2022-10-10 21:01:07] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00318 [I 2022-10-10 21:01:07] ...trap/engines/engine.py.230: loss: 0.11395 [I 2022-10-10 21:01:07] ...optimizers/trijoint.py.151: total_norm: 11.622925 [I 2022-10-10 21:01:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 1640/2304 [I 2022-10-10 21:01:16] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:31 [I 2022-10-10 21:01:16] ...trap/engines/engine.py.229: process: 0.86003 | load: 0.00305 [I 2022-10-10 21:01:16] ...trap/engines/engine.py.230: loss: 0.18716 [I 2022-10-10 21:01:16] ...optimizers/trijoint.py.151: total_norm: 15.845591 [I 2022-10-10 21:01:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 1650/2304 [I 2022-10-10 21:01:25] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:25 [I 2022-10-10 21:01:25] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00311 [I 2022-10-10 21:01:25] ...trap/engines/engine.py.230: loss: 0.09544 [I 2022-10-10 21:01:25] ...optimizers/trijoint.py.151: total_norm: 9.317654 [I 2022-10-10 21:01:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 1660/2304 [I 2022-10-10 21:01:33] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:09:13 [I 2022-10-10 21:01:33] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00318 [I 2022-10-10 21:01:33] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-10 21:01:33] ...optimizers/trijoint.py.151: total_norm: 12.091840 [I 2022-10-10 21:01:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 1670/2304 [I 2022-10-10 21:01:42] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:09:06 [I 2022-10-10 21:01:42] ...trap/engines/engine.py.229: process: 0.87261 | load: 0.00314 [I 2022-10-10 21:01:42] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-10 21:01:42] ...optimizers/trijoint.py.151: total_norm: 10.779186 [I 2022-10-10 21:01:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 1680/2304 [I 2022-10-10 21:01:51] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:54 [I 2022-10-10 21:01:51] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00306 [I 2022-10-10 21:01:51] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-10 21:01:51] ...optimizers/trijoint.py.151: total_norm: 17.055304 [I 2022-10-10 21:01:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 1690/2304 [I 2022-10-10 21:01:59] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:48 [I 2022-10-10 21:01:59] ...trap/engines/engine.py.229: process: 0.86060 | load: 0.00321 [I 2022-10-10 21:01:59] ...trap/engines/engine.py.230: loss: 0.12498 [I 2022-10-10 21:01:59] ...optimizers/trijoint.py.151: total_norm: 12.775046 [I 2022-10-10 21:02:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 1700/2304 [I 2022-10-10 21:02:08] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:41 [I 2022-10-10 21:02:08] ...trap/engines/engine.py.229: process: 0.86204 | load: 0.00328 [I 2022-10-10 21:02:08] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-10 21:02:08] ...optimizers/trijoint.py.151: total_norm: 27.714153 [I 2022-10-10 21:02:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 1710/2304 [I 2022-10-10 21:02:17] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:34 [I 2022-10-10 21:02:17] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00314 [I 2022-10-10 21:02:17] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-10 21:02:17] ...optimizers/trijoint.py.151: total_norm: 12.575161 [I 2022-10-10 21:02:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 1720/2304 [I 2022-10-10 21:02:25] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:20 [I 2022-10-10 21:02:25] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00313 [I 2022-10-10 21:02:25] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-10 21:02:25] ...optimizers/trijoint.py.151: total_norm: 13.465240 [I 2022-10-10 21:02:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 1730/2304 [I 2022-10-10 21:02:34] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:12 [I 2022-10-10 21:02:34] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00308 [I 2022-10-10 21:02:34] ...trap/engines/engine.py.230: loss: 0.10424 [I 2022-10-10 21:02:34] ...optimizers/trijoint.py.151: total_norm: 15.956095 [I 2022-10-10 21:02:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 1740/2304 [I 2022-10-10 21:02:42] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:08:08 [I 2022-10-10 21:02:42] ...trap/engines/engine.py.229: process: 0.86813 | load: 0.00311 [I 2022-10-10 21:02:42] ...trap/engines/engine.py.230: loss: 0.09871 [I 2022-10-10 21:02:42] ...optimizers/trijoint.py.151: total_norm: 8.701135 [I 2022-10-10 21:02:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 1750/2304 [I 2022-10-10 21:02:51] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:57 [I 2022-10-10 21:02:51] ...trap/engines/engine.py.229: process: 0.86591 | load: 0.00312 [I 2022-10-10 21:02:51] ...trap/engines/engine.py.230: loss: 0.17033 [I 2022-10-10 21:02:51] ...optimizers/trijoint.py.151: total_norm: 17.878841 [I 2022-10-10 21:03:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 1760/2304 [I 2022-10-10 21:03:00] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:48 [I 2022-10-10 21:03:00] ...trap/engines/engine.py.229: process: 0.87434 | load: 0.00310 [I 2022-10-10 21:03:00] ...trap/engines/engine.py.230: loss: 0.11355 [I 2022-10-10 21:03:00] ...optimizers/trijoint.py.151: total_norm: 13.313697 [I 2022-10-10 21:03:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 1770/2304 [I 2022-10-10 21:03:08] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:41 [I 2022-10-10 21:03:08] ...trap/engines/engine.py.229: process: 0.86383 | load: 0.00312 [I 2022-10-10 21:03:08] ...trap/engines/engine.py.230: loss: 0.17283 [I 2022-10-10 21:03:08] ...optimizers/trijoint.py.151: total_norm: 17.031174 [I 2022-10-10 21:03:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 1780/2304 [I 2022-10-10 21:03:17] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:31 [I 2022-10-10 21:03:17] ...trap/engines/engine.py.229: process: 0.86658 | load: 0.00360 [I 2022-10-10 21:03:17] ...trap/engines/engine.py.230: loss: 0.08655 [I 2022-10-10 21:03:17] ...optimizers/trijoint.py.151: total_norm: 9.075583 [I 2022-10-10 21:03:26] ...trap/engines/engine.py.224: train: epoch 85 | batch 1790/2304 [I 2022-10-10 21:03:26] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:24 [I 2022-10-10 21:03:26] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00315 [I 2022-10-10 21:03:26] ...trap/engines/engine.py.230: loss: 0.09662 [I 2022-10-10 21:03:26] ...optimizers/trijoint.py.151: total_norm: 10.101530 [I 2022-10-10 21:03:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 1800/2304 [I 2022-10-10 21:03:34] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:14 [I 2022-10-10 21:03:34] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00309 [I 2022-10-10 21:03:34] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-10 21:03:34] ...optimizers/trijoint.py.151: total_norm: 14.553475 [I 2022-10-10 21:03:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 1810/2304 [I 2022-10-10 21:03:43] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:03 [I 2022-10-10 21:03:43] ...trap/engines/engine.py.229: process: 0.86122 | load: 0.00311 [I 2022-10-10 21:03:43] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-10 21:03:43] ...optimizers/trijoint.py.151: total_norm: 12.848662 [I 2022-10-10 21:03:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 1820/2304 [I 2022-10-10 21:03:52] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:53 [I 2022-10-10 21:03:52] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00308 [I 2022-10-10 21:03:52] ...trap/engines/engine.py.230: loss: 0.17980 [I 2022-10-10 21:03:52] ...optimizers/trijoint.py.151: total_norm: 13.103629 [I 2022-10-10 21:04:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 1830/2304 [I 2022-10-10 21:04:00] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:46 [I 2022-10-10 21:04:00] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00307 [I 2022-10-10 21:04:00] ...trap/engines/engine.py.230: loss: 0.09882 [I 2022-10-10 21:04:00] ...optimizers/trijoint.py.151: total_norm: 14.362892 [I 2022-10-10 21:04:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 1840/2304 [I 2022-10-10 21:04:09] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:38 [I 2022-10-10 21:04:09] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00313 [I 2022-10-10 21:04:09] ...trap/engines/engine.py.230: loss: 0.10591 [I 2022-10-10 21:04:09] ...optimizers/trijoint.py.151: total_norm: 13.466016 [I 2022-10-10 21:04:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 1850/2304 [I 2022-10-10 21:04:17] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:31 [I 2022-10-10 21:04:17] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00307 [I 2022-10-10 21:04:17] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-10 21:04:17] ...optimizers/trijoint.py.151: total_norm: 9.367546 [I 2022-10-10 21:04:26] ...trap/engines/engine.py.224: train: epoch 85 | batch 1860/2304 [I 2022-10-10 21:04:26] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:24 [I 2022-10-10 21:04:26] ...trap/engines/engine.py.229: process: 0.86580 | load: 0.00311 [I 2022-10-10 21:04:26] ...trap/engines/engine.py.230: loss: 0.11109 [I 2022-10-10 21:04:26] ...optimizers/trijoint.py.151: total_norm: 47.036655 [I 2022-10-10 21:04:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 1870/2304 [I 2022-10-10 21:04:35] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:16 [I 2022-10-10 21:04:35] ...trap/engines/engine.py.229: process: 0.87498 | load: 0.00314 [I 2022-10-10 21:04:35] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-10 21:04:35] ...optimizers/trijoint.py.151: total_norm: 17.329491 [I 2022-10-10 21:04:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 1880/2304 [I 2022-10-10 21:04:43] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:05 [I 2022-10-10 21:04:43] ...trap/engines/engine.py.229: process: 0.85822 | load: 0.00308 [I 2022-10-10 21:04:43] ...trap/engines/engine.py.230: loss: 0.14493 [I 2022-10-10 21:04:43] ...optimizers/trijoint.py.151: total_norm: 14.438898 [I 2022-10-10 21:04:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 1890/2304 [I 2022-10-10 21:04:52] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:05:54 [I 2022-10-10 21:04:52] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00306 [I 2022-10-10 21:04:52] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-10 21:04:52] ...optimizers/trijoint.py.151: total_norm: 12.451799 [I 2022-10-10 21:05:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 1900/2304 [I 2022-10-10 21:05:01] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:50 [I 2022-10-10 21:05:01] ...trap/engines/engine.py.229: process: 0.88092 | load: 0.00307 [I 2022-10-10 21:05:01] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-10 21:05:01] ...optimizers/trijoint.py.151: total_norm: 36.868725 [I 2022-10-10 21:05:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 1910/2304 [I 2022-10-10 21:05:09] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:37 [I 2022-10-10 21:05:09] ...trap/engines/engine.py.229: process: 0.84706 | load: 0.00311 [I 2022-10-10 21:05:09] ...trap/engines/engine.py.230: loss: 0.15907 [I 2022-10-10 21:05:09] ...optimizers/trijoint.py.151: total_norm: 36.909016 [I 2022-10-10 21:05:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 1920/2304 [I 2022-10-10 21:05:18] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:05:31 [I 2022-10-10 21:05:18] ...trap/engines/engine.py.229: process: 0.87234 | load: 0.00316 [I 2022-10-10 21:05:18] ...trap/engines/engine.py.230: loss: 0.29120 [I 2022-10-10 21:05:18] ...optimizers/trijoint.py.151: total_norm: 38.512863 [I 2022-10-10 21:05:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 1930/2304 [I 2022-10-10 21:05:27] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:23 [I 2022-10-10 21:05:27] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00310 [I 2022-10-10 21:05:27] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-10 21:05:27] ...optimizers/trijoint.py.151: total_norm: 14.137853 [I 2022-10-10 21:05:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 1940/2304 [I 2022-10-10 21:05:35] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:14 [I 2022-10-10 21:05:35] ...trap/engines/engine.py.229: process: 0.86421 | load: 0.00315 [I 2022-10-10 21:05:35] ...trap/engines/engine.py.230: loss: 0.19672 [I 2022-10-10 21:05:35] ...optimizers/trijoint.py.151: total_norm: 35.623226 [I 2022-10-10 21:05:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 1950/2304 [I 2022-10-10 21:05:44] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:02 [I 2022-10-10 21:05:44] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00309 [I 2022-10-10 21:05:44] ...trap/engines/engine.py.230: loss: 0.14677 [I 2022-10-10 21:05:44] ...optimizers/trijoint.py.151: total_norm: 15.557188 [I 2022-10-10 21:05:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 1960/2304 [I 2022-10-10 21:05:52] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:56 [I 2022-10-10 21:05:52] ...trap/engines/engine.py.229: process: 0.86144 | load: 0.00324 [I 2022-10-10 21:05:52] ...trap/engines/engine.py.230: loss: 0.10158 [I 2022-10-10 21:05:52] ...optimizers/trijoint.py.151: total_norm: 12.524848 [I 2022-10-10 21:06:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 1970/2304 [I 2022-10-10 21:06:01] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:46 [I 2022-10-10 21:06:01] ...trap/engines/engine.py.229: process: 0.87207 | load: 0.00319 [I 2022-10-10 21:06:01] ...trap/engines/engine.py.230: loss: 0.13154 [I 2022-10-10 21:06:01] ...optimizers/trijoint.py.151: total_norm: 14.699949 [I 2022-10-10 21:06:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 1980/2304 [I 2022-10-10 21:06:10] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:37 [I 2022-10-10 21:06:10] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00308 [I 2022-10-10 21:06:10] ...trap/engines/engine.py.230: loss: 0.19080 [I 2022-10-10 21:06:10] ...optimizers/trijoint.py.151: total_norm: 11.580763 [I 2022-10-10 21:06:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 1990/2304 [I 2022-10-10 21:06:18] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:31 [I 2022-10-10 21:06:18] ...trap/engines/engine.py.229: process: 0.87161 | load: 0.00306 [I 2022-10-10 21:06:18] ...trap/engines/engine.py.230: loss: 0.12171 [I 2022-10-10 21:06:18] ...optimizers/trijoint.py.151: total_norm: 15.265203 [I 2022-10-10 21:06:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 2000/2304 [I 2022-10-10 21:06:27] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:20 [I 2022-10-10 21:06:27] ...trap/engines/engine.py.229: process: 0.84563 | load: 0.00312 [I 2022-10-10 21:06:27] ...trap/engines/engine.py.230: loss: 0.13954 [I 2022-10-10 21:06:27] ...optimizers/trijoint.py.151: total_norm: 29.775206 [I 2022-10-10 21:06:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 2010/2304 [I 2022-10-10 21:06:36] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:13 [I 2022-10-10 21:06:36] ...trap/engines/engine.py.229: process: 0.86691 | load: 0.00314 [I 2022-10-10 21:06:36] ...trap/engines/engine.py.230: loss: 0.11819 [I 2022-10-10 21:06:36] ...optimizers/trijoint.py.151: total_norm: 13.769895 [I 2022-10-10 21:06:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 2020/2304 [I 2022-10-10 21:06:44] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:05 [I 2022-10-10 21:06:44] ...trap/engines/engine.py.229: process: 0.86792 | load: 0.00306 [I 2022-10-10 21:06:44] ...trap/engines/engine.py.230: loss: 0.19518 [I 2022-10-10 21:06:44] ...optimizers/trijoint.py.151: total_norm: 38.873318 [I 2022-10-10 21:06:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 2030/2304 [I 2022-10-10 21:06:53] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:57 [I 2022-10-10 21:06:53] ...trap/engines/engine.py.229: process: 0.87336 | load: 0.00307 [I 2022-10-10 21:06:53] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-10 21:06:53] ...optimizers/trijoint.py.151: total_norm: 24.559782 [I 2022-10-10 21:07:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 2040/2304 [I 2022-10-10 21:07:02] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:48 [I 2022-10-10 21:07:02] ...trap/engines/engine.py.229: process: 0.86322 | load: 0.00315 [I 2022-10-10 21:07:02] ...trap/engines/engine.py.230: loss: 0.20314 [I 2022-10-10 21:07:02] ...optimizers/trijoint.py.151: total_norm: 13.844507 [I 2022-10-10 21:07:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 2050/2304 [I 2022-10-10 21:07:10] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:39 [I 2022-10-10 21:07:10] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00307 [I 2022-10-10 21:07:10] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-10 21:07:10] ...optimizers/trijoint.py.151: total_norm: 14.950889 [I 2022-10-10 21:07:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 2060/2304 [I 2022-10-10 21:07:19] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:31 [I 2022-10-10 21:07:19] ...trap/engines/engine.py.229: process: 0.87260 | load: 0.00305 [I 2022-10-10 21:07:19] ...trap/engines/engine.py.230: loss: 0.13642 [I 2022-10-10 21:07:19] ...optimizers/trijoint.py.151: total_norm: 25.808819 [I 2022-10-10 21:07:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 2070/2304 [I 2022-10-10 21:07:28] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:03:20 [I 2022-10-10 21:07:28] ...trap/engines/engine.py.229: process: 0.85937 | load: 0.00300 [I 2022-10-10 21:07:28] ...trap/engines/engine.py.230: loss: 0.09211 [I 2022-10-10 21:07:28] ...optimizers/trijoint.py.151: total_norm: 14.563730 [I 2022-10-10 21:07:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 2080/2304 [I 2022-10-10 21:07:36] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:11 [I 2022-10-10 21:07:36] ...trap/engines/engine.py.229: process: 0.85451 | load: 0.00306 [I 2022-10-10 21:07:36] ...trap/engines/engine.py.230: loss: 0.15384 [I 2022-10-10 21:07:36] ...optimizers/trijoint.py.151: total_norm: 20.852951 [I 2022-10-10 21:07:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 2090/2304 [I 2022-10-10 21:07:45] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:03:04 [I 2022-10-10 21:07:45] ...trap/engines/engine.py.229: process: 0.86216 | load: 0.00309 [I 2022-10-10 21:07:45] ...trap/engines/engine.py.230: loss: 0.18650 [I 2022-10-10 21:07:45] ...optimizers/trijoint.py.151: total_norm: 17.979990 [I 2022-10-10 21:07:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 2100/2304 [I 2022-10-10 21:07:53] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:54 [I 2022-10-10 21:07:53] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00296 [I 2022-10-10 21:07:53] ...trap/engines/engine.py.230: loss: 0.10496 [I 2022-10-10 21:07:53] ...optimizers/trijoint.py.151: total_norm: 11.160415 [I 2022-10-10 21:08:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 2110/2304 [I 2022-10-10 21:08:02] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:46 [I 2022-10-10 21:08:02] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00296 [I 2022-10-10 21:08:02] ...trap/engines/engine.py.230: loss: 0.17617 [I 2022-10-10 21:08:02] ...optimizers/trijoint.py.151: total_norm: 8.159240 [I 2022-10-10 21:08:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 2120/2304 [I 2022-10-10 21:08:11] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:37 [I 2022-10-10 21:08:11] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00298 [I 2022-10-10 21:08:11] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-10 21:08:11] ...optimizers/trijoint.py.151: total_norm: 14.456135 [I 2022-10-10 21:08:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 2130/2304 [I 2022-10-10 21:08:19] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:29 [I 2022-10-10 21:08:19] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00304 [I 2022-10-10 21:08:19] ...trap/engines/engine.py.230: loss: 0.11392 [I 2022-10-10 21:08:19] ...optimizers/trijoint.py.151: total_norm: 31.563120 [I 2022-10-10 21:08:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 2140/2304 [I 2022-10-10 21:08:28] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:19 [I 2022-10-10 21:08:28] ...trap/engines/engine.py.229: process: 0.84665 | load: 0.00298 [I 2022-10-10 21:08:28] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-10 21:08:28] ...optimizers/trijoint.py.151: total_norm: 8.385283 [I 2022-10-10 21:08:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 2150/2304 [I 2022-10-10 21:08:36] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:11 [I 2022-10-10 21:08:36] ...trap/engines/engine.py.229: process: 0.85167 | load: 0.00294 [I 2022-10-10 21:08:36] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-10 21:08:36] ...optimizers/trijoint.py.151: total_norm: 7.908016 [I 2022-10-10 21:08:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 2160/2304 [I 2022-10-10 21:08:45] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:02:02 [I 2022-10-10 21:08:45] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00300 [I 2022-10-10 21:08:45] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-10 21:08:45] ...optimizers/trijoint.py.151: total_norm: 16.785309 [I 2022-10-10 21:08:54] ...trap/engines/engine.py.224: train: epoch 85 | batch 2170/2304 [I 2022-10-10 21:08:54] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:55 [I 2022-10-10 21:08:54] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00295 [I 2022-10-10 21:08:54] ...trap/engines/engine.py.230: loss: 0.24803 [I 2022-10-10 21:08:54] ...optimizers/trijoint.py.151: total_norm: 14.266056 [I 2022-10-10 21:09:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 2180/2304 [I 2022-10-10 21:09:02] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:46 [I 2022-10-10 21:09:02] ...trap/engines/engine.py.229: process: 0.85420 | load: 0.00296 [I 2022-10-10 21:09:02] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-10 21:09:02] ...optimizers/trijoint.py.151: total_norm: 18.058744 [I 2022-10-10 21:09:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 2190/2304 [I 2022-10-10 21:09:11] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:38 [I 2022-10-10 21:09:11] ...trap/engines/engine.py.229: process: 0.86420 | load: 0.00312 [I 2022-10-10 21:09:11] ...trap/engines/engine.py.230: loss: 0.09664 [I 2022-10-10 21:09:11] ...optimizers/trijoint.py.151: total_norm: 10.296199 [I 2022-10-10 21:09:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 2200/2304 [I 2022-10-10 21:09:19] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:30 [I 2022-10-10 21:09:19] ...trap/engines/engine.py.229: process: 0.86794 | load: 0.00312 [I 2022-10-10 21:09:19] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-10 21:09:19] ...optimizers/trijoint.py.151: total_norm: 16.526983 [I 2022-10-10 21:09:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 2210/2304 [I 2022-10-10 21:09:28] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:21 [I 2022-10-10 21:09:28] ...trap/engines/engine.py.229: process: 0.86015 | load: 0.00311 [I 2022-10-10 21:09:28] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-10 21:09:28] ...optimizers/trijoint.py.151: total_norm: 29.554716 [I 2022-10-10 21:09:37] ...trap/engines/engine.py.224: train: epoch 85 | batch 2220/2304 [I 2022-10-10 21:09:37] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:12 [I 2022-10-10 21:09:37] ...trap/engines/engine.py.229: process: 0.86958 | load: 0.00308 [I 2022-10-10 21:09:37] ...trap/engines/engine.py.230: loss: 0.09023 [I 2022-10-10 21:09:37] ...optimizers/trijoint.py.151: total_norm: 9.263107 [I 2022-10-10 21:09:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 2230/2304 [I 2022-10-10 21:09:45] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:03 [I 2022-10-10 21:09:45] ...trap/engines/engine.py.229: process: 0.86893 | load: 0.00345 [I 2022-10-10 21:09:45] ...trap/engines/engine.py.230: loss: 0.11227 [I 2022-10-10 21:09:45] ...optimizers/trijoint.py.151: total_norm: 14.527996 [I 2022-10-10 21:09:54] ...trap/engines/engine.py.224: train: epoch 85 | batch 2240/2304 [I 2022-10-10 21:09:54] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:55 [I 2022-10-10 21:09:54] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00306 [I 2022-10-10 21:09:54] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-10 21:09:54] ...optimizers/trijoint.py.151: total_norm: 25.124136 [I 2022-10-10 21:10:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 2250/2304 [I 2022-10-10 21:10:03] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:46 [I 2022-10-10 21:10:03] ...trap/engines/engine.py.229: process: 0.86802 | load: 0.00309 [I 2022-10-10 21:10:03] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-10 21:10:03] ...optimizers/trijoint.py.151: total_norm: 10.774071 [I 2022-10-10 21:10:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 2260/2304 [I 2022-10-10 21:10:11] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:37 [I 2022-10-10 21:10:11] ...trap/engines/engine.py.229: process: 0.86191 | load: 0.00333 [I 2022-10-10 21:10:11] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-10 21:10:11] ...optimizers/trijoint.py.151: total_norm: 16.788162 [I 2022-10-10 21:10:20] ...trap/engines/engine.py.224: train: epoch 85 | batch 2270/2304 [I 2022-10-10 21:10:20] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:29 [I 2022-10-10 21:10:20] ...trap/engines/engine.py.229: process: 0.86827 | load: 0.00320 [I 2022-10-10 21:10:20] ...trap/engines/engine.py.230: loss: 0.12867 [I 2022-10-10 21:10:20] ...optimizers/trijoint.py.151: total_norm: 9.864340 [I 2022-10-10 21:10:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 2280/2304 [I 2022-10-10 21:10:29] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:20 [I 2022-10-10 21:10:29] ...trap/engines/engine.py.229: process: 0.85516 | load: 0.00309 [I 2022-10-10 21:10:29] ...trap/engines/engine.py.230: loss: 0.18111 [I 2022-10-10 21:10:29] ...optimizers/trijoint.py.151: total_norm: 12.272835 [I 2022-10-10 21:10:37] ...trap/engines/engine.py.224: train: epoch 85 | batch 2290/2304 [I 2022-10-10 21:10:37] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:12 [I 2022-10-10 21:10:37] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00309 [I 2022-10-10 21:10:37] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-10 21:10:37] ...optimizers/trijoint.py.151: total_norm: 13.492266 [I 2022-10-10 21:10:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 2300/2304 [I 2022-10-10 21:10:46] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:03 [I 2022-10-10 21:10:46] ...trap/engines/engine.py.229: process: 0.85566 | load: 0.00019 [I 2022-10-10 21:10:46] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-10 21:10:46] ...optimizers/trijoint.py.151: total_norm: 70.325905 [I 2022-10-10 21:10:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 2304/2304 [I 2022-10-10 21:10:49] ...trap/engines/engine.py.228: elapsed: 0:33:12 | left: 0:00:00 [I 2022-10-10 21:10:49] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00018 [I 2022-10-10 21:10:49] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-10 21:10:49] ...optimizers/trijoint.py.151: total_norm: 19.893185 [S 2022-10-10 21:10:51] ...trap/engines/engine.py.129: train_epoch.epoch: 85 [S 2022-10-10 21:10:51] ...trap/engines/engine.py.129: train_epoch.loss: 0.13008115471637688 [I 2022-10-10 21:10:51] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 21:10:52] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 85 [I 2022-10-10 21:10:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 21:10:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 21:10:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 21:10:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 21:10:57] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 21:10:58] ...trap/engines/engine.py.321: eval: epoch 85 | batch 0/510 [I 2022-10-10 21:10:58] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:34 [I 2022-10-10 21:10:58] ...trap/engines/engine.py.326: process: 5.83325 | load: 5.25923 [I 2022-10-10 21:11:04] ...trap/engines/engine.py.321: eval: epoch 85 | batch 10/510 [I 2022-10-10 21:11:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:56 [I 2022-10-10 21:11:04] ...trap/engines/engine.py.326: process: 0.69082 | load: 0.00020 [I 2022-10-10 21:11:10] ...trap/engines/engine.py.321: eval: epoch 85 | batch 20/510 [I 2022-10-10 21:11:10] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:05:37 [I 2022-10-10 21:11:10] ...trap/engines/engine.py.326: process: 0.71832 | load: 0.00020 [I 2022-10-10 21:11:17] ...trap/engines/engine.py.321: eval: epoch 85 | batch 30/510 [I 2022-10-10 21:11:17] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:05:31 [I 2022-10-10 21:11:17] ...trap/engines/engine.py.326: process: 0.68123 | load: 0.00020 [I 2022-10-10 21:11:25] ...trap/engines/engine.py.321: eval: epoch 85 | batch 40/510 [I 2022-10-10 21:11:25] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:05:45 [I 2022-10-10 21:11:25] ...trap/engines/engine.py.326: process: 0.65518 | load: 0.00021 [I 2022-10-10 21:11:30] ...trap/engines/engine.py.321: eval: epoch 85 | batch 50/510 [I 2022-10-10 21:11:30] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:19 [I 2022-10-10 21:11:30] ...trap/engines/engine.py.326: process: 0.54729 | load: 0.00021 [I 2022-10-10 21:11:36] ...trap/engines/engine.py.321: eval: epoch 85 | batch 60/510 [I 2022-10-10 21:11:36] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:04:00 [I 2022-10-10 21:11:36] ...trap/engines/engine.py.326: process: 0.54091 | load: 0.00024 [I 2022-10-10 21:11:41] ...trap/engines/engine.py.321: eval: epoch 85 | batch 70/510 [I 2022-10-10 21:11:41] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:04:09 [I 2022-10-10 21:11:41] ...trap/engines/engine.py.326: process: 0.61813 | load: 0.00021 [I 2022-10-10 21:11:47] ...trap/engines/engine.py.321: eval: epoch 85 | batch 80/510 [I 2022-10-10 21:11:47] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:04:09 [I 2022-10-10 21:11:47] ...trap/engines/engine.py.326: process: 0.59524 | load: 0.00020 [I 2022-10-10 21:11:52] ...trap/engines/engine.py.321: eval: epoch 85 | batch 90/510 [I 2022-10-10 21:11:52] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:03:12 [I 2022-10-10 21:11:52] ...trap/engines/engine.py.326: process: 0.42638 | load: 0.00020 [I 2022-10-10 21:11:56] ...trap/engines/engine.py.321: eval: epoch 85 | batch 100/510 [I 2022-10-10 21:11:56] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:47 [I 2022-10-10 21:11:56] ...trap/engines/engine.py.326: process: 0.30992 | load: 0.00019 [I 2022-10-10 21:11:59] ...trap/engines/engine.py.321: eval: epoch 85 | batch 110/510 [I 2022-10-10 21:11:59] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:12 [I 2022-10-10 21:11:59] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00020 [I 2022-10-10 21:12:02] ...trap/engines/engine.py.321: eval: epoch 85 | batch 120/510 [I 2022-10-10 21:12:02] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:05 [I 2022-10-10 21:12:02] ...trap/engines/engine.py.326: process: 0.32393 | load: 0.00020 [I 2022-10-10 21:12:06] ...trap/engines/engine.py.321: eval: epoch 85 | batch 130/510 [I 2022-10-10 21:12:06] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:02 [I 2022-10-10 21:12:06] ...trap/engines/engine.py.326: process: 0.32387 | load: 0.00022 [I 2022-10-10 21:12:09] ...trap/engines/engine.py.321: eval: epoch 85 | batch 140/510 [I 2022-10-10 21:12:09] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:58 [I 2022-10-10 21:12:09] ...trap/engines/engine.py.326: process: 0.31470 | load: 0.00020 [I 2022-10-10 21:12:12] ...trap/engines/engine.py.321: eval: epoch 85 | batch 150/510 [I 2022-10-10 21:12:12] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-10 21:12:12] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00021 [I 2022-10-10 21:12:15] ...trap/engines/engine.py.321: eval: epoch 85 | batch 160/510 [I 2022-10-10 21:12:15] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:52 [I 2022-10-10 21:12:15] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00022 [I 2022-10-10 21:12:18] ...trap/engines/engine.py.321: eval: epoch 85 | batch 170/510 [I 2022-10-10 21:12:18] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:48 [I 2022-10-10 21:12:18] ...trap/engines/engine.py.326: process: 0.31269 | load: 0.00021 [I 2022-10-10 21:12:22] ...trap/engines/engine.py.321: eval: epoch 85 | batch 180/510 [I 2022-10-10 21:12:22] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:45 [I 2022-10-10 21:12:22] ...trap/engines/engine.py.326: process: 0.32096 | load: 0.00021 [I 2022-10-10 21:12:25] ...trap/engines/engine.py.321: eval: epoch 85 | batch 190/510 [I 2022-10-10 21:12:25] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:42 [I 2022-10-10 21:12:25] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00020 [I 2022-10-10 21:12:28] ...trap/engines/engine.py.321: eval: epoch 85 | batch 200/510 [I 2022-10-10 21:12:28] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:40 [I 2022-10-10 21:12:28] ...trap/engines/engine.py.326: process: 0.32386 | load: 0.00021 [I 2022-10-10 21:12:31] ...trap/engines/engine.py.321: eval: epoch 85 | batch 210/510 [I 2022-10-10 21:12:31] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:36 [I 2022-10-10 21:12:31] ...trap/engines/engine.py.326: process: 0.31905 | load: 0.00021 [I 2022-10-10 21:12:35] ...trap/engines/engine.py.321: eval: epoch 85 | batch 220/510 [I 2022-10-10 21:12:35] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:32 [I 2022-10-10 21:12:35] ...trap/engines/engine.py.326: process: 0.31223 | load: 0.00021 [I 2022-10-10 21:12:38] ...trap/engines/engine.py.321: eval: epoch 85 | batch 230/510 [I 2022-10-10 21:12:38] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:30 [I 2022-10-10 21:12:38] ...trap/engines/engine.py.326: process: 0.32900 | load: 0.00022 [I 2022-10-10 21:12:41] ...trap/engines/engine.py.321: eval: epoch 85 | batch 240/510 [I 2022-10-10 21:12:41] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:27 [I 2022-10-10 21:12:41] ...trap/engines/engine.py.326: process: 0.32622 | load: 0.00021 [I 2022-10-10 21:12:44] ...trap/engines/engine.py.321: eval: epoch 85 | batch 250/510 [I 2022-10-10 21:12:44] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:24 [I 2022-10-10 21:12:44] ...trap/engines/engine.py.326: process: 0.33356 | load: 0.00021 [I 2022-10-10 21:12:48] ...trap/engines/engine.py.321: eval: epoch 85 | batch 260/510 [I 2022-10-10 21:12:48] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:20 [I 2022-10-10 21:12:48] ...trap/engines/engine.py.326: process: 0.32331 | load: 0.00021 [I 2022-10-10 21:12:51] ...trap/engines/engine.py.321: eval: epoch 85 | batch 270/510 [I 2022-10-10 21:12:51] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:17 [I 2022-10-10 21:12:51] ...trap/engines/engine.py.326: process: 0.32892 | load: 0.00021 [I 2022-10-10 21:12:54] ...trap/engines/engine.py.321: eval: epoch 85 | batch 280/510 [I 2022-10-10 21:12:54] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:13 [I 2022-10-10 21:12:54] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00021 [I 2022-10-10 21:12:57] ...trap/engines/engine.py.321: eval: epoch 85 | batch 290/510 [I 2022-10-10 21:12:57] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:11 [I 2022-10-10 21:12:57] ...trap/engines/engine.py.326: process: 0.32141 | load: 0.00022 [I 2022-10-10 21:13:01] ...trap/engines/engine.py.321: eval: epoch 85 | batch 300/510 [I 2022-10-10 21:13:01] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:07 [I 2022-10-10 21:13:01] ...trap/engines/engine.py.326: process: 0.32435 | load: 0.00022 [I 2022-10-10 21:13:04] ...trap/engines/engine.py.321: eval: epoch 85 | batch 310/510 [I 2022-10-10 21:13:04] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-10 21:13:04] ...trap/engines/engine.py.326: process: 0.32339 | load: 0.00019 [I 2022-10-10 21:13:07] ...trap/engines/engine.py.321: eval: epoch 85 | batch 320/510 [I 2022-10-10 21:13:07] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:00 [I 2022-10-10 21:13:07] ...trap/engines/engine.py.326: process: 0.31616 | load: 0.00020 [I 2022-10-10 21:13:10] ...trap/engines/engine.py.321: eval: epoch 85 | batch 330/510 [I 2022-10-10 21:13:10] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:57 [I 2022-10-10 21:13:10] ...trap/engines/engine.py.326: process: 0.31882 | load: 0.00020 [I 2022-10-10 21:13:13] ...trap/engines/engine.py.321: eval: epoch 85 | batch 340/510 [I 2022-10-10 21:13:13] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:54 [I 2022-10-10 21:13:13] ...trap/engines/engine.py.326: process: 0.32320 | load: 0.00020 [I 2022-10-10 21:13:16] ...trap/engines/engine.py.321: eval: epoch 85 | batch 350/510 [I 2022-10-10 21:13:16] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:51 [I 2022-10-10 21:13:16] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00019 [I 2022-10-10 21:13:20] ...trap/engines/engine.py.321: eval: epoch 85 | batch 360/510 [I 2022-10-10 21:13:20] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:47 [I 2022-10-10 21:13:20] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00021 [I 2022-10-10 21:13:23] ...trap/engines/engine.py.321: eval: epoch 85 | batch 370/510 [I 2022-10-10 21:13:23] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:44 [I 2022-10-10 21:13:23] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00021 [I 2022-10-10 21:13:26] ...trap/engines/engine.py.321: eval: epoch 85 | batch 380/510 [I 2022-10-10 21:13:26] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:41 [I 2022-10-10 21:13:26] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00021 [I 2022-10-10 21:13:29] ...trap/engines/engine.py.321: eval: epoch 85 | batch 390/510 [I 2022-10-10 21:13:29] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:38 [I 2022-10-10 21:13:29] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00019 [I 2022-10-10 21:13:33] ...trap/engines/engine.py.321: eval: epoch 85 | batch 400/510 [I 2022-10-10 21:13:33] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-10 21:13:33] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00022 [I 2022-10-10 21:13:36] ...trap/engines/engine.py.321: eval: epoch 85 | batch 410/510 [I 2022-10-10 21:13:36] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-10 21:13:36] ...trap/engines/engine.py.326: process: 0.31755 | load: 0.00021 [I 2022-10-10 21:13:39] ...trap/engines/engine.py.321: eval: epoch 85 | batch 420/510 [I 2022-10-10 21:13:39] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:28 [I 2022-10-10 21:13:39] ...trap/engines/engine.py.326: process: 0.31929 | load: 0.00020 [I 2022-10-10 21:13:42] ...trap/engines/engine.py.321: eval: epoch 85 | batch 430/510 [I 2022-10-10 21:13:42] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:25 [I 2022-10-10 21:13:42] ...trap/engines/engine.py.326: process: 0.32370 | load: 0.00020 [I 2022-10-10 21:13:45] ...trap/engines/engine.py.321: eval: epoch 85 | batch 440/510 [I 2022-10-10 21:13:45] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-10 21:13:45] ...trap/engines/engine.py.326: process: 0.32318 | load: 0.00019 [I 2022-10-10 21:13:49] ...trap/engines/engine.py.321: eval: epoch 85 | batch 450/510 [I 2022-10-10 21:13:49] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-10 21:13:49] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-10 21:13:52] ...trap/engines/engine.py.321: eval: epoch 85 | batch 460/510 [I 2022-10-10 21:13:52] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:15 [I 2022-10-10 21:13:52] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00020 [I 2022-10-10 21:13:55] ...trap/engines/engine.py.321: eval: epoch 85 | batch 470/510 [I 2022-10-10 21:13:55] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-10 21:13:55] ...trap/engines/engine.py.326: process: 0.32162 | load: 0.00020 [I 2022-10-10 21:13:58] ...trap/engines/engine.py.321: eval: epoch 85 | batch 480/510 [I 2022-10-10 21:13:58] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-10 21:13:58] ...trap/engines/engine.py.326: process: 0.31631 | load: 0.00028 [I 2022-10-10 21:14:01] ...trap/engines/engine.py.321: eval: epoch 85 | batch 490/510 [I 2022-10-10 21:14:01] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-10 21:14:01] ...trap/engines/engine.py.326: process: 0.32055 | load: 0.00020 [I 2022-10-10 21:14:04] ...trap/engines/engine.py.321: eval: epoch 85 | batch 500/510 [I 2022-10-10 21:14:04] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-10 21:14:04] ...trap/engines/engine.py.326: process: 0.32165 | load: 0.00020 [I 2022-10-10 21:14:08] ...trap/engines/engine.py.321: eval: epoch 85 | batch 510/510 [I 2022-10-10 21:14:08] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:00 [I 2022-10-10 21:14:08] ...trap/engines/engine.py.326: process: 0.31222 | load: 0.00016 [S 2022-10-10 21:14:08] ...trap/engines/engine.py.132: eval_epoch.epoch: 85 [I 2022-10-10 21:14:08] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740800 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910700 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935600 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.737100 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908700 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936200 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.006911 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006573 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006902 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011614 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007643 [S 2022-10-10 21:14:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007096 [I 2022-10-10 21:14:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 21:14:47] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 21:14:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 21:14:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 21:14:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 21:14:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 21:14:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 21:14:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 21:14:59] ...trap/engines/engine.py.163: Training model on trainset for epoch 86 [I 2022-10-10 21:14:59] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 21:15:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 0/2304 [I 2022-10-10 21:15:02] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:10:17 [I 2022-10-10 21:15:02] ...trap/engines/engine.py.229: process: 3.39293 | load: 2.47432 [I 2022-10-10 21:15:02] ...trap/engines/engine.py.230: loss: 0.10322 [I 2022-10-10 21:15:02] ...optimizers/trijoint.py.151: total_norm: 15.615080 [I 2022-10-10 21:15:11] ...trap/engines/engine.py.224: train: epoch 86 | batch 10/2304 [I 2022-10-10 21:15:11] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:10 [I 2022-10-10 21:15:11] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00490 [I 2022-10-10 21:15:11] ...trap/engines/engine.py.230: loss: 0.07967 [I 2022-10-10 21:15:11] ...optimizers/trijoint.py.151: total_norm: 9.024495 [I 2022-10-10 21:15:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 20/2304 [I 2022-10-10 21:15:20] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:15 [I 2022-10-10 21:15:20] ...trap/engines/engine.py.229: process: 0.87634 | load: 0.00543 [I 2022-10-10 21:15:20] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-10 21:15:20] ...optimizers/trijoint.py.151: total_norm: 25.123392 [I 2022-10-10 21:15:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 30/2304 [I 2022-10-10 21:15:28] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:55 [I 2022-10-10 21:15:28] ...trap/engines/engine.py.229: process: 0.86850 | load: 0.00439 [I 2022-10-10 21:15:28] ...trap/engines/engine.py.230: loss: 0.12301 [I 2022-10-10 21:15:28] ...optimizers/trijoint.py.151: total_norm: 10.133224 [I 2022-10-10 21:15:37] ...trap/engines/engine.py.224: train: epoch 86 | batch 40/2304 [I 2022-10-10 21:15:37] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:42 [I 2022-10-10 21:15:37] ...trap/engines/engine.py.229: process: 0.85946 | load: 0.00428 [I 2022-10-10 21:15:37] ...trap/engines/engine.py.230: loss: 0.13944 [I 2022-10-10 21:15:37] ...optimizers/trijoint.py.151: total_norm: 9.173960 [I 2022-10-10 21:15:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 50/2304 [I 2022-10-10 21:15:46] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:29 [I 2022-10-10 21:15:46] ...trap/engines/engine.py.229: process: 0.87456 | load: 0.00377 [I 2022-10-10 21:15:46] ...trap/engines/engine.py.230: loss: 0.18503 [I 2022-10-10 21:15:46] ...optimizers/trijoint.py.151: total_norm: 11.547632 [I 2022-10-10 21:15:54] ...trap/engines/engine.py.224: train: epoch 86 | batch 60/2304 [I 2022-10-10 21:15:54] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:27 [I 2022-10-10 21:15:54] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00394 [I 2022-10-10 21:15:54] ...trap/engines/engine.py.230: loss: 0.18385 [I 2022-10-10 21:15:54] ...optimizers/trijoint.py.151: total_norm: 13.543306 [I 2022-10-10 21:16:22] ...trap/engines/engine.py.224: train: epoch 86 | batch 70/2304 [I 2022-10-10 21:16:22] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 2:56:52 [I 2022-10-10 21:16:22] ...trap/engines/engine.py.229: process: 20.28659 | load: 0.00332 [I 2022-10-10 21:16:22] ...trap/engines/engine.py.230: loss: 0.14600 [I 2022-10-10 21:16:22] ...optimizers/trijoint.py.151: total_norm: 9.989964 [I 2022-10-10 21:16:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 80/2304 [I 2022-10-10 21:16:31] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:47:32 [I 2022-10-10 21:16:31] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00333 [I 2022-10-10 21:16:31] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-10 21:16:31] ...optimizers/trijoint.py.151: total_norm: 14.088620 [I 2022-10-10 21:16:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 90/2304 [I 2022-10-10 21:16:40] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:33:31 [I 2022-10-10 21:16:40] ...trap/engines/engine.py.229: process: 0.86036 | load: 0.00345 [I 2022-10-10 21:16:40] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-10 21:16:40] ...optimizers/trijoint.py.151: total_norm: 8.218651 [I 2022-10-10 21:16:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 100/2304 [I 2022-10-10 21:16:48] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:32:01 [I 2022-10-10 21:16:48] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00352 [I 2022-10-10 21:16:48] ...trap/engines/engine.py.230: loss: 0.12016 [I 2022-10-10 21:16:48] ...optimizers/trijoint.py.151: total_norm: 18.587069 [I 2022-10-10 21:16:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 110/2304 [I 2022-10-10 21:16:57] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:31:43 [I 2022-10-10 21:16:57] ...trap/engines/engine.py.229: process: 0.86894 | load: 0.00361 [I 2022-10-10 21:16:57] ...trap/engines/engine.py.230: loss: 0.08574 [I 2022-10-10 21:16:57] ...optimizers/trijoint.py.151: total_norm: 10.952065 [I 2022-10-10 21:17:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 120/2304 [I 2022-10-10 21:17:06] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:31:22 [I 2022-10-10 21:17:06] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00333 [I 2022-10-10 21:17:06] ...trap/engines/engine.py.230: loss: 0.15182 [I 2022-10-10 21:17:06] ...optimizers/trijoint.py.151: total_norm: 11.883910 [I 2022-10-10 21:17:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 130/2304 [I 2022-10-10 21:17:14] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:31:18 [I 2022-10-10 21:17:14] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00312 [I 2022-10-10 21:17:14] ...trap/engines/engine.py.230: loss: 0.21874 [I 2022-10-10 21:17:14] ...optimizers/trijoint.py.151: total_norm: 12.778133 [I 2022-10-10 21:17:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 140/2304 [I 2022-10-10 21:17:23] ...trap/engines/engine.py.228: elapsed: 0:02:24 | left: 0:31:14 [I 2022-10-10 21:17:23] ...trap/engines/engine.py.229: process: 0.87141 | load: 0.00328 [I 2022-10-10 21:17:23] ...trap/engines/engine.py.230: loss: 0.21233 [I 2022-10-10 21:17:23] ...optimizers/trijoint.py.151: total_norm: 32.828541 [I 2022-10-10 21:17:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 150/2304 [I 2022-10-10 21:17:32] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:30:57 [I 2022-10-10 21:17:32] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00343 [I 2022-10-10 21:17:32] ...trap/engines/engine.py.230: loss: 0.09839 [I 2022-10-10 21:17:32] ...optimizers/trijoint.py.151: total_norm: 8.746998 [I 2022-10-10 21:17:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 160/2304 [I 2022-10-10 21:17:40] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:30:52 [I 2022-10-10 21:17:40] ...trap/engines/engine.py.229: process: 0.85495 | load: 0.00336 [I 2022-10-10 21:17:40] ...trap/engines/engine.py.230: loss: 0.10608 [I 2022-10-10 21:17:40] ...optimizers/trijoint.py.151: total_norm: 8.317140 [I 2022-10-10 21:17:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 170/2304 [I 2022-10-10 21:17:49] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:30:50 [I 2022-10-10 21:17:49] ...trap/engines/engine.py.229: process: 0.87196 | load: 0.00346 [I 2022-10-10 21:17:49] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-10 21:17:49] ...optimizers/trijoint.py.151: total_norm: 9.562666 [I 2022-10-10 21:17:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 180/2304 [I 2022-10-10 21:17:58] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:30:54 [I 2022-10-10 21:17:58] ...trap/engines/engine.py.229: process: 0.87066 | load: 0.00332 [I 2022-10-10 21:17:58] ...trap/engines/engine.py.230: loss: 0.12796 [I 2022-10-10 21:17:58] ...optimizers/trijoint.py.151: total_norm: 25.327602 [I 2022-10-10 21:18:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 190/2304 [I 2022-10-10 21:18:07] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:30:46 [I 2022-10-10 21:18:07] ...trap/engines/engine.py.229: process: 0.89183 | load: 0.00319 [I 2022-10-10 21:18:07] ...trap/engines/engine.py.230: loss: 0.08660 [I 2022-10-10 21:18:07] ...optimizers/trijoint.py.151: total_norm: 10.413988 [I 2022-10-10 21:18:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 200/2304 [I 2022-10-10 21:18:15] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:30:10 [I 2022-10-10 21:18:15] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00366 [I 2022-10-10 21:18:15] ...trap/engines/engine.py.230: loss: 0.18895 [I 2022-10-10 21:18:15] ...optimizers/trijoint.py.151: total_norm: 23.509993 [I 2022-10-10 21:18:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 210/2304 [I 2022-10-10 21:18:24] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:30:11 [I 2022-10-10 21:18:24] ...trap/engines/engine.py.229: process: 0.86073 | load: 0.00349 [I 2022-10-10 21:18:24] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-10 21:18:24] ...optimizers/trijoint.py.151: total_norm: 9.533458 [I 2022-10-10 21:18:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 220/2304 [I 2022-10-10 21:18:32] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:29:59 [I 2022-10-10 21:18:32] ...trap/engines/engine.py.229: process: 0.86220 | load: 0.00321 [I 2022-10-10 21:18:32] ...trap/engines/engine.py.230: loss: 0.13481 [I 2022-10-10 21:18:32] ...optimizers/trijoint.py.151: total_norm: 10.706470 [I 2022-10-10 21:18:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 230/2304 [I 2022-10-10 21:18:41] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:29:58 [I 2022-10-10 21:18:41] ...trap/engines/engine.py.229: process: 0.87785 | load: 0.00318 [I 2022-10-10 21:18:41] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-10 21:18:41] ...optimizers/trijoint.py.151: total_norm: 17.649246 [I 2022-10-10 21:18:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 240/2304 [I 2022-10-10 21:18:50] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:29:33 [I 2022-10-10 21:18:50] ...trap/engines/engine.py.229: process: 0.85516 | load: 0.00309 [I 2022-10-10 21:18:50] ...trap/engines/engine.py.230: loss: 0.14004 [I 2022-10-10 21:18:50] ...optimizers/trijoint.py.151: total_norm: 28.631655 [I 2022-10-10 21:18:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 250/2304 [I 2022-10-10 21:18:58] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:29:19 [I 2022-10-10 21:18:58] ...trap/engines/engine.py.229: process: 0.87296 | load: 0.00315 [I 2022-10-10 21:18:58] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-10 21:18:58] ...optimizers/trijoint.py.151: total_norm: 17.098827 [I 2022-10-10 21:19:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 260/2304 [I 2022-10-10 21:19:07] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:29:11 [I 2022-10-10 21:19:07] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00306 [I 2022-10-10 21:19:07] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-10 21:19:07] ...optimizers/trijoint.py.151: total_norm: 13.902447 [I 2022-10-10 21:19:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 270/2304 [I 2022-10-10 21:19:16] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:59 [I 2022-10-10 21:19:16] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00299 [I 2022-10-10 21:19:16] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-10 21:19:16] ...optimizers/trijoint.py.151: total_norm: 11.526408 [I 2022-10-10 21:19:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 280/2304 [I 2022-10-10 21:19:24] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:48 [I 2022-10-10 21:19:24] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00309 [I 2022-10-10 21:19:24] ...trap/engines/engine.py.230: loss: 0.09288 [I 2022-10-10 21:19:24] ...optimizers/trijoint.py.151: total_norm: 12.484005 [I 2022-10-10 21:19:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 290/2304 [I 2022-10-10 21:19:33] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:43 [I 2022-10-10 21:19:33] ...trap/engines/engine.py.229: process: 0.86893 | load: 0.00314 [I 2022-10-10 21:19:33] ...trap/engines/engine.py.230: loss: 0.17718 [I 2022-10-10 21:19:33] ...optimizers/trijoint.py.151: total_norm: 21.731857 [I 2022-10-10 21:19:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 300/2304 [I 2022-10-10 21:19:41] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:28:31 [I 2022-10-10 21:19:41] ...trap/engines/engine.py.229: process: 0.86203 | load: 0.00307 [I 2022-10-10 21:19:41] ...trap/engines/engine.py.230: loss: 0.11296 [I 2022-10-10 21:19:41] ...optimizers/trijoint.py.151: total_norm: 12.490026 [I 2022-10-10 21:19:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 310/2304 [I 2022-10-10 21:19:50] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:28:30 [I 2022-10-10 21:19:50] ...trap/engines/engine.py.229: process: 0.87188 | load: 0.00305 [I 2022-10-10 21:19:50] ...trap/engines/engine.py.230: loss: 0.07604 [I 2022-10-10 21:19:50] ...optimizers/trijoint.py.151: total_norm: 9.212119 [I 2022-10-10 21:19:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 320/2304 [I 2022-10-10 21:19:58] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:28:24 [I 2022-10-10 21:19:58] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00305 [I 2022-10-10 21:19:58] ...trap/engines/engine.py.230: loss: 0.18560 [I 2022-10-10 21:19:58] ...optimizers/trijoint.py.151: total_norm: 29.494820 [I 2022-10-10 21:20:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 330/2304 [I 2022-10-10 21:20:07] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:28:08 [I 2022-10-10 21:20:07] ...trap/engines/engine.py.229: process: 0.84469 | load: 0.00305 [I 2022-10-10 21:20:07] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-10 21:20:07] ...optimizers/trijoint.py.151: total_norm: 13.628762 [I 2022-10-10 21:20:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 340/2304 [I 2022-10-10 21:20:16] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:28:00 [I 2022-10-10 21:20:16] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00301 [I 2022-10-10 21:20:16] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-10 21:20:16] ...optimizers/trijoint.py.151: total_norm: 8.071822 [I 2022-10-10 21:20:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 350/2304 [I 2022-10-10 21:20:24] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:27:49 [I 2022-10-10 21:20:24] ...trap/engines/engine.py.229: process: 0.84423 | load: 0.00329 [I 2022-10-10 21:20:24] ...trap/engines/engine.py.230: loss: 0.17626 [I 2022-10-10 21:20:24] ...optimizers/trijoint.py.151: total_norm: 19.039181 [I 2022-10-10 21:20:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 360/2304 [I 2022-10-10 21:20:33] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:43 [I 2022-10-10 21:20:33] ...trap/engines/engine.py.229: process: 0.85959 | load: 0.00293 [I 2022-10-10 21:20:33] ...trap/engines/engine.py.230: loss: 0.11641 [I 2022-10-10 21:20:33] ...optimizers/trijoint.py.151: total_norm: 10.407809 [I 2022-10-10 21:20:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 370/2304 [I 2022-10-10 21:20:41] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:27:31 [I 2022-10-10 21:20:41] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00299 [I 2022-10-10 21:20:41] ...trap/engines/engine.py.230: loss: 0.07671 [I 2022-10-10 21:20:41] ...optimizers/trijoint.py.151: total_norm: 11.362304 [I 2022-10-10 21:20:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 380/2304 [I 2022-10-10 21:20:50] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:27:25 [I 2022-10-10 21:20:50] ...trap/engines/engine.py.229: process: 0.85589 | load: 0.00301 [I 2022-10-10 21:20:50] ...trap/engines/engine.py.230: loss: 0.10443 [I 2022-10-10 21:20:50] ...optimizers/trijoint.py.151: total_norm: 7.547639 [I 2022-10-10 21:20:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 390/2304 [I 2022-10-10 21:20:58] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:27:09 [I 2022-10-10 21:20:58] ...trap/engines/engine.py.229: process: 0.85115 | load: 0.00291 [I 2022-10-10 21:20:58] ...trap/engines/engine.py.230: loss: 0.11549 [I 2022-10-10 21:20:58] ...optimizers/trijoint.py.151: total_norm: 15.968575 [I 2022-10-10 21:21:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 400/2304 [I 2022-10-10 21:21:07] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:52 [I 2022-10-10 21:21:07] ...trap/engines/engine.py.229: process: 0.82944 | load: 0.00302 [I 2022-10-10 21:21:07] ...trap/engines/engine.py.230: loss: 0.15852 [I 2022-10-10 21:21:07] ...optimizers/trijoint.py.151: total_norm: 19.665100 [I 2022-10-10 21:21:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 410/2304 [I 2022-10-10 21:21:15] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:47 [I 2022-10-10 21:21:15] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00301 [I 2022-10-10 21:21:15] ...trap/engines/engine.py.230: loss: 0.08371 [I 2022-10-10 21:21:15] ...optimizers/trijoint.py.151: total_norm: 15.638324 [I 2022-10-10 21:21:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 420/2304 [I 2022-10-10 21:21:24] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:26:50 [I 2022-10-10 21:21:24] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00292 [I 2022-10-10 21:21:24] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-10 21:21:24] ...optimizers/trijoint.py.151: total_norm: 11.752847 [I 2022-10-10 21:21:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 430/2304 [I 2022-10-10 21:21:33] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:26:36 [I 2022-10-10 21:21:33] ...trap/engines/engine.py.229: process: 0.85939 | load: 0.00293 [I 2022-10-10 21:21:33] ...trap/engines/engine.py.230: loss: 0.13417 [I 2022-10-10 21:21:33] ...optimizers/trijoint.py.151: total_norm: 15.674729 [I 2022-10-10 21:21:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 440/2304 [I 2022-10-10 21:21:41] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:26:32 [I 2022-10-10 21:21:41] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00299 [I 2022-10-10 21:21:41] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-10 21:21:41] ...optimizers/trijoint.py.151: total_norm: 16.665928 [I 2022-10-10 21:21:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 450/2304 [I 2022-10-10 21:21:50] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:26:28 [I 2022-10-10 21:21:50] ...trap/engines/engine.py.229: process: 0.87079 | load: 0.00322 [I 2022-10-10 21:21:50] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-10 21:21:50] ...optimizers/trijoint.py.151: total_norm: 20.132923 [I 2022-10-10 21:21:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 460/2304 [I 2022-10-10 21:21:58] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:26:29 [I 2022-10-10 21:21:58] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00309 [I 2022-10-10 21:21:58] ...trap/engines/engine.py.230: loss: 0.11510 [I 2022-10-10 21:21:58] ...optimizers/trijoint.py.151: total_norm: 15.952540 [I 2022-10-10 21:22:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 470/2304 [I 2022-10-10 21:22:07] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:26:27 [I 2022-10-10 21:22:07] ...trap/engines/engine.py.229: process: 0.86534 | load: 0.00311 [I 2022-10-10 21:22:07] ...trap/engines/engine.py.230: loss: 0.12091 [I 2022-10-10 21:22:07] ...optimizers/trijoint.py.151: total_norm: 38.654720 [I 2022-10-10 21:22:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 480/2304 [I 2022-10-10 21:22:16] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:26:17 [I 2022-10-10 21:22:16] ...trap/engines/engine.py.229: process: 0.85939 | load: 0.00312 [I 2022-10-10 21:22:16] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-10 21:22:16] ...optimizers/trijoint.py.151: total_norm: 44.021107 [I 2022-10-10 21:22:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 490/2304 [I 2022-10-10 21:22:24] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:26:13 [I 2022-10-10 21:22:24] ...trap/engines/engine.py.229: process: 0.87364 | load: 0.00324 [I 2022-10-10 21:22:24] ...trap/engines/engine.py.230: loss: 0.11735 [I 2022-10-10 21:22:24] ...optimizers/trijoint.py.151: total_norm: 12.126472 [I 2022-10-10 21:22:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 500/2304 [I 2022-10-10 21:22:33] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:25:56 [I 2022-10-10 21:22:33] ...trap/engines/engine.py.229: process: 0.87347 | load: 0.00313 [I 2022-10-10 21:22:33] ...trap/engines/engine.py.230: loss: 0.08446 [I 2022-10-10 21:22:33] ...optimizers/trijoint.py.151: total_norm: 8.381661 [I 2022-10-10 21:22:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 510/2304 [I 2022-10-10 21:22:42] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:25:54 [I 2022-10-10 21:22:42] ...trap/engines/engine.py.229: process: 0.86367 | load: 0.00311 [I 2022-10-10 21:22:42] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-10 21:22:42] ...optimizers/trijoint.py.151: total_norm: 33.969387 [I 2022-10-10 21:22:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 520/2304 [I 2022-10-10 21:22:50] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:44 [I 2022-10-10 21:22:50] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00310 [I 2022-10-10 21:22:50] ...trap/engines/engine.py.230: loss: 0.21454 [I 2022-10-10 21:22:50] ...optimizers/trijoint.py.151: total_norm: 23.771370 [I 2022-10-10 21:22:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 530/2304 [I 2022-10-10 21:22:59] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:25:41 [I 2022-10-10 21:22:59] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00314 [I 2022-10-10 21:22:59] ...trap/engines/engine.py.230: loss: 0.19445 [I 2022-10-10 21:22:59] ...optimizers/trijoint.py.151: total_norm: 10.535633 [I 2022-10-10 21:23:08] ...trap/engines/engine.py.224: train: epoch 86 | batch 540/2304 [I 2022-10-10 21:23:08] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:25:21 [I 2022-10-10 21:23:08] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00333 [I 2022-10-10 21:23:08] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-10 21:23:08] ...optimizers/trijoint.py.151: total_norm: 20.112947 [I 2022-10-10 21:23:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 550/2304 [I 2022-10-10 21:23:16] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:25:19 [I 2022-10-10 21:23:16] ...trap/engines/engine.py.229: process: 0.87378 | load: 0.00343 [I 2022-10-10 21:23:16] ...trap/engines/engine.py.230: loss: 0.21322 [I 2022-10-10 21:23:16] ...optimizers/trijoint.py.151: total_norm: 17.931124 [I 2022-10-10 21:23:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 560/2304 [I 2022-10-10 21:23:25] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:25:22 [I 2022-10-10 21:23:25] ...trap/engines/engine.py.229: process: 0.87525 | load: 0.00316 [I 2022-10-10 21:23:25] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-10 21:23:25] ...optimizers/trijoint.py.151: total_norm: 16.635475 [I 2022-10-10 21:23:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 570/2304 [I 2022-10-10 21:23:34] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:25:09 [I 2022-10-10 21:23:34] ...trap/engines/engine.py.229: process: 0.86429 | load: 0.00322 [I 2022-10-10 21:23:34] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-10 21:23:34] ...optimizers/trijoint.py.151: total_norm: 18.790483 [I 2022-10-10 21:23:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 580/2304 [I 2022-10-10 21:23:43] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:24:54 [I 2022-10-10 21:23:43] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00313 [I 2022-10-10 21:23:43] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-10 21:23:43] ...optimizers/trijoint.py.151: total_norm: 8.342349 [I 2022-10-10 21:23:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 590/2304 [I 2022-10-10 21:23:51] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:43 [I 2022-10-10 21:23:51] ...trap/engines/engine.py.229: process: 0.87102 | load: 0.00311 [I 2022-10-10 21:23:51] ...trap/engines/engine.py.230: loss: 0.18597 [I 2022-10-10 21:23:51] ...optimizers/trijoint.py.151: total_norm: 33.723240 [I 2022-10-10 21:24:00] ...trap/engines/engine.py.224: train: epoch 86 | batch 600/2304 [I 2022-10-10 21:24:00] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:24:38 [I 2022-10-10 21:24:00] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00312 [I 2022-10-10 21:24:00] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-10 21:24:00] ...optimizers/trijoint.py.151: total_norm: 10.011202 [I 2022-10-10 21:24:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 610/2304 [I 2022-10-10 21:24:09] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:24:30 [I 2022-10-10 21:24:09] ...trap/engines/engine.py.229: process: 0.88212 | load: 0.00312 [I 2022-10-10 21:24:09] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-10 21:24:09] ...optimizers/trijoint.py.151: total_norm: 21.670189 [I 2022-10-10 21:24:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 620/2304 [I 2022-10-10 21:24:17] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:24:13 [I 2022-10-10 21:24:17] ...trap/engines/engine.py.229: process: 0.85371 | load: 0.00312 [I 2022-10-10 21:24:17] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-10 21:24:17] ...optimizers/trijoint.py.151: total_norm: 48.247265 [I 2022-10-10 21:24:26] ...trap/engines/engine.py.224: train: epoch 86 | batch 630/2304 [I 2022-10-10 21:24:26] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:24:07 [I 2022-10-10 21:24:26] ...trap/engines/engine.py.229: process: 0.87812 | load: 0.00322 [I 2022-10-10 21:24:26] ...trap/engines/engine.py.230: loss: 0.12118 [I 2022-10-10 21:24:26] ...optimizers/trijoint.py.151: total_norm: 8.644650 [I 2022-10-10 21:24:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 640/2304 [I 2022-10-10 21:24:35] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:54 [I 2022-10-10 21:24:35] ...trap/engines/engine.py.229: process: 0.85622 | load: 0.00312 [I 2022-10-10 21:24:35] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-10 21:24:35] ...optimizers/trijoint.py.151: total_norm: 24.005655 [I 2022-10-10 21:24:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 650/2304 [I 2022-10-10 21:24:43] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:53 [I 2022-10-10 21:24:43] ...trap/engines/engine.py.229: process: 0.85920 | load: 0.00315 [I 2022-10-10 21:24:43] ...trap/engines/engine.py.230: loss: 0.09749 [I 2022-10-10 21:24:43] ...optimizers/trijoint.py.151: total_norm: 11.154187 [I 2022-10-10 21:24:52] ...trap/engines/engine.py.224: train: epoch 86 | batch 660/2304 [I 2022-10-10 21:24:52] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:23:35 [I 2022-10-10 21:24:52] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00313 [I 2022-10-10 21:24:52] ...trap/engines/engine.py.230: loss: 0.10267 [I 2022-10-10 21:24:52] ...optimizers/trijoint.py.151: total_norm: 8.805285 [I 2022-10-10 21:25:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 670/2304 [I 2022-10-10 21:25:01] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:23:35 [I 2022-10-10 21:25:01] ...trap/engines/engine.py.229: process: 0.87766 | load: 0.00335 [I 2022-10-10 21:25:01] ...trap/engines/engine.py.230: loss: 0.10633 [I 2022-10-10 21:25:01] ...optimizers/trijoint.py.151: total_norm: 14.320678 [I 2022-10-10 21:25:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 680/2304 [I 2022-10-10 21:25:09] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:23:22 [I 2022-10-10 21:25:09] ...trap/engines/engine.py.229: process: 0.86625 | load: 0.00310 [I 2022-10-10 21:25:09] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-10 21:25:09] ...optimizers/trijoint.py.151: total_norm: 15.301656 [I 2022-10-10 21:25:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 690/2304 [I 2022-10-10 21:25:18] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:23:26 [I 2022-10-10 21:25:18] ...trap/engines/engine.py.229: process: 0.87194 | load: 0.00321 [I 2022-10-10 21:25:18] ...trap/engines/engine.py.230: loss: 0.17823 [I 2022-10-10 21:25:18] ...optimizers/trijoint.py.151: total_norm: 26.713263 [I 2022-10-10 21:25:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 700/2304 [I 2022-10-10 21:25:27] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:23:08 [I 2022-10-10 21:25:27] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00320 [I 2022-10-10 21:25:27] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-10 21:25:27] ...optimizers/trijoint.py.151: total_norm: 18.486052 [I 2022-10-10 21:25:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 710/2304 [I 2022-10-10 21:25:36] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:22:57 [I 2022-10-10 21:25:36] ...trap/engines/engine.py.229: process: 0.86435 | load: 0.00315 [I 2022-10-10 21:25:36] ...trap/engines/engine.py.230: loss: 0.11621 [I 2022-10-10 21:25:36] ...optimizers/trijoint.py.151: total_norm: 15.609719 [I 2022-10-10 21:25:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 720/2304 [I 2022-10-10 21:25:44] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:22:49 [I 2022-10-10 21:25:44] ...trap/engines/engine.py.229: process: 0.86177 | load: 0.00307 [I 2022-10-10 21:25:44] ...trap/engines/engine.py.230: loss: 0.14258 [I 2022-10-10 21:25:44] ...optimizers/trijoint.py.151: total_norm: 14.188653 [I 2022-10-10 21:25:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 730/2304 [I 2022-10-10 21:25:53] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:22:24 [I 2022-10-10 21:25:53] ...trap/engines/engine.py.229: process: 0.86061 | load: 0.00310 [I 2022-10-10 21:25:53] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-10 21:25:53] ...optimizers/trijoint.py.151: total_norm: 21.091667 [I 2022-10-10 21:26:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 740/2304 [I 2022-10-10 21:26:01] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:22:23 [I 2022-10-10 21:26:01] ...trap/engines/engine.py.229: process: 0.85937 | load: 0.00305 [I 2022-10-10 21:26:01] ...trap/engines/engine.py.230: loss: 0.12696 [I 2022-10-10 21:26:01] ...optimizers/trijoint.py.151: total_norm: 21.095873 [I 2022-10-10 21:26:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 750/2304 [I 2022-10-10 21:26:10] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:22:08 [I 2022-10-10 21:26:10] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00292 [I 2022-10-10 21:26:10] ...trap/engines/engine.py.230: loss: 0.08490 [I 2022-10-10 21:26:10] ...optimizers/trijoint.py.151: total_norm: 9.024136 [I 2022-10-10 21:26:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 760/2304 [I 2022-10-10 21:26:19] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:21:57 [I 2022-10-10 21:26:19] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00293 [I 2022-10-10 21:26:19] ...trap/engines/engine.py.230: loss: 0.21527 [I 2022-10-10 21:26:19] ...optimizers/trijoint.py.151: total_norm: 24.544895 [I 2022-10-10 21:26:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 770/2304 [I 2022-10-10 21:26:27] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:21:55 [I 2022-10-10 21:26:27] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00311 [I 2022-10-10 21:26:27] ...trap/engines/engine.py.230: loss: 0.13995 [I 2022-10-10 21:26:27] ...optimizers/trijoint.py.151: total_norm: 17.695295 [I 2022-10-10 21:26:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 780/2304 [I 2022-10-10 21:26:36] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:21:40 [I 2022-10-10 21:26:36] ...trap/engines/engine.py.229: process: 0.84895 | load: 0.00293 [I 2022-10-10 21:26:36] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-10 21:26:36] ...optimizers/trijoint.py.151: total_norm: 15.289563 [I 2022-10-10 21:26:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 790/2304 [I 2022-10-10 21:26:44] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:21:34 [I 2022-10-10 21:26:44] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00295 [I 2022-10-10 21:26:44] ...trap/engines/engine.py.230: loss: 0.09559 [I 2022-10-10 21:26:44] ...optimizers/trijoint.py.151: total_norm: 20.002502 [I 2022-10-10 21:26:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 800/2304 [I 2022-10-10 21:26:53] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:21:22 [I 2022-10-10 21:26:53] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00297 [I 2022-10-10 21:26:53] ...trap/engines/engine.py.230: loss: 0.09956 [I 2022-10-10 21:26:53] ...optimizers/trijoint.py.151: total_norm: 14.423265 [I 2022-10-10 21:27:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 810/2304 [I 2022-10-10 21:27:01] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:21:14 [I 2022-10-10 21:27:01] ...trap/engines/engine.py.229: process: 0.85758 | load: 0.00297 [I 2022-10-10 21:27:01] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-10 21:27:01] ...optimizers/trijoint.py.151: total_norm: 19.939743 [I 2022-10-10 21:27:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 820/2304 [I 2022-10-10 21:27:10] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:21:10 [I 2022-10-10 21:27:10] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00301 [I 2022-10-10 21:27:10] ...trap/engines/engine.py.230: loss: 0.12843 [I 2022-10-10 21:27:10] ...optimizers/trijoint.py.151: total_norm: 19.503590 [I 2022-10-10 21:27:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 830/2304 [I 2022-10-10 21:27:19] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:56 [I 2022-10-10 21:27:19] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00303 [I 2022-10-10 21:27:19] ...trap/engines/engine.py.230: loss: 0.14124 [I 2022-10-10 21:27:19] ...optimizers/trijoint.py.151: total_norm: 12.918632 [I 2022-10-10 21:27:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 840/2304 [I 2022-10-10 21:27:27] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:20:56 [I 2022-10-10 21:27:27] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00308 [I 2022-10-10 21:27:27] ...trap/engines/engine.py.230: loss: 0.11351 [I 2022-10-10 21:27:27] ...optimizers/trijoint.py.151: total_norm: 19.973801 [I 2022-10-10 21:27:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 850/2304 [I 2022-10-10 21:27:36] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:20:42 [I 2022-10-10 21:27:36] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00307 [I 2022-10-10 21:27:36] ...trap/engines/engine.py.230: loss: 0.10834 [I 2022-10-10 21:27:36] ...optimizers/trijoint.py.151: total_norm: 8.386885 [I 2022-10-10 21:27:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 860/2304 [I 2022-10-10 21:27:44] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:20:32 [I 2022-10-10 21:27:44] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00297 [I 2022-10-10 21:27:44] ...trap/engines/engine.py.230: loss: 0.14935 [I 2022-10-10 21:27:44] ...optimizers/trijoint.py.151: total_norm: 12.547444 [I 2022-10-10 21:27:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 870/2304 [I 2022-10-10 21:27:53] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:20:24 [I 2022-10-10 21:27:53] ...trap/engines/engine.py.229: process: 0.86410 | load: 0.00295 [I 2022-10-10 21:27:53] ...trap/engines/engine.py.230: loss: 0.10423 [I 2022-10-10 21:27:53] ...optimizers/trijoint.py.151: total_norm: 10.254509 [I 2022-10-10 21:28:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 880/2304 [I 2022-10-10 21:28:01] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:20:14 [I 2022-10-10 21:28:01] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00297 [I 2022-10-10 21:28:01] ...trap/engines/engine.py.230: loss: 0.11743 [I 2022-10-10 21:28:01] ...optimizers/trijoint.py.151: total_norm: 18.354635 [I 2022-10-10 21:28:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 890/2304 [I 2022-10-10 21:28:10] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:20:06 [I 2022-10-10 21:28:10] ...trap/engines/engine.py.229: process: 0.85993 | load: 0.00293 [I 2022-10-10 21:28:10] ...trap/engines/engine.py.230: loss: 0.07690 [I 2022-10-10 21:28:10] ...optimizers/trijoint.py.151: total_norm: 11.676326 [I 2022-10-10 21:28:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 900/2304 [I 2022-10-10 21:28:18] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:19:58 [I 2022-10-10 21:28:18] ...trap/engines/engine.py.229: process: 0.85821 | load: 0.00300 [I 2022-10-10 21:28:18] ...trap/engines/engine.py.230: loss: 0.10531 [I 2022-10-10 21:28:18] ...optimizers/trijoint.py.151: total_norm: 9.903945 [I 2022-10-10 21:28:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 910/2304 [I 2022-10-10 21:28:27] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:48 [I 2022-10-10 21:28:27] ...trap/engines/engine.py.229: process: 0.84437 | load: 0.00294 [I 2022-10-10 21:28:27] ...trap/engines/engine.py.230: loss: 0.15452 [I 2022-10-10 21:28:27] ...optimizers/trijoint.py.151: total_norm: 11.846714 [I 2022-10-10 21:28:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 920/2304 [I 2022-10-10 21:28:36] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:19:46 [I 2022-10-10 21:28:36] ...trap/engines/engine.py.229: process: 0.85257 | load: 0.00297 [I 2022-10-10 21:28:36] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-10 21:28:36] ...optimizers/trijoint.py.151: total_norm: 10.508121 [I 2022-10-10 21:28:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 930/2304 [I 2022-10-10 21:28:44] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:19:32 [I 2022-10-10 21:28:44] ...trap/engines/engine.py.229: process: 0.86020 | load: 0.00296 [I 2022-10-10 21:28:44] ...trap/engines/engine.py.230: loss: 0.08614 [I 2022-10-10 21:28:44] ...optimizers/trijoint.py.151: total_norm: 9.473329 [I 2022-10-10 21:28:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 940/2304 [I 2022-10-10 21:28:53] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:23 [I 2022-10-10 21:28:53] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00295 [I 2022-10-10 21:28:53] ...trap/engines/engine.py.230: loss: 0.13508 [I 2022-10-10 21:28:53] ...optimizers/trijoint.py.151: total_norm: 20.590334 [I 2022-10-10 21:29:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 950/2304 [I 2022-10-10 21:29:01] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:19:23 [I 2022-10-10 21:29:01] ...trap/engines/engine.py.229: process: 0.86159 | load: 0.00310 [I 2022-10-10 21:29:01] ...trap/engines/engine.py.230: loss: 0.15507 [I 2022-10-10 21:29:01] ...optimizers/trijoint.py.151: total_norm: 24.234627 [I 2022-10-10 21:29:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 960/2304 [I 2022-10-10 21:29:10] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:19:27 [I 2022-10-10 21:29:10] ...trap/engines/engine.py.229: process: 0.88752 | load: 0.00312 [I 2022-10-10 21:29:10] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-10 21:29:10] ...optimizers/trijoint.py.151: total_norm: 8.995434 [I 2022-10-10 21:29:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 970/2304 [I 2022-10-10 21:29:19] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:19:21 [I 2022-10-10 21:29:19] ...trap/engines/engine.py.229: process: 0.87037 | load: 0.00310 [I 2022-10-10 21:29:19] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-10 21:29:19] ...optimizers/trijoint.py.151: total_norm: 10.577121 [I 2022-10-10 21:29:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 980/2304 [I 2022-10-10 21:29:28] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:19:04 [I 2022-10-10 21:29:28] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00310 [I 2022-10-10 21:29:28] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-10 21:29:28] ...optimizers/trijoint.py.151: total_norm: 25.215891 [I 2022-10-10 21:29:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 990/2304 [I 2022-10-10 21:29:36] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:19:00 [I 2022-10-10 21:29:36] ...trap/engines/engine.py.229: process: 0.88505 | load: 0.00309 [I 2022-10-10 21:29:36] ...trap/engines/engine.py.230: loss: 0.09090 [I 2022-10-10 21:29:36] ...optimizers/trijoint.py.151: total_norm: 9.909170 [I 2022-10-10 21:29:45] ...trap/engines/engine.py.224: train: epoch 86 | batch 1000/2304 [I 2022-10-10 21:29:45] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:18:53 [I 2022-10-10 21:29:45] ...trap/engines/engine.py.229: process: 0.84611 | load: 0.00311 [I 2022-10-10 21:29:45] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-10 21:29:45] ...optimizers/trijoint.py.151: total_norm: 13.603314 [I 2022-10-10 21:29:54] ...trap/engines/engine.py.224: train: epoch 86 | batch 1010/2304 [I 2022-10-10 21:29:54] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:18:43 [I 2022-10-10 21:29:54] ...trap/engines/engine.py.229: process: 0.86991 | load: 0.00306 [I 2022-10-10 21:29:54] ...trap/engines/engine.py.230: loss: 0.09512 [I 2022-10-10 21:29:54] ...optimizers/trijoint.py.151: total_norm: 12.446075 [I 2022-10-10 21:30:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 1020/2304 [I 2022-10-10 21:30:02] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:18:29 [I 2022-10-10 21:30:02] ...trap/engines/engine.py.229: process: 0.86790 | load: 0.00312 [I 2022-10-10 21:30:02] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-10 21:30:02] ...optimizers/trijoint.py.151: total_norm: 21.907797 [I 2022-10-10 21:30:11] ...trap/engines/engine.py.224: train: epoch 86 | batch 1030/2304 [I 2022-10-10 21:30:11] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:18:22 [I 2022-10-10 21:30:11] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00322 [I 2022-10-10 21:30:11] ...trap/engines/engine.py.230: loss: 0.07910 [I 2022-10-10 21:30:11] ...optimizers/trijoint.py.151: total_norm: 11.967033 [I 2022-10-10 21:30:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 1040/2304 [I 2022-10-10 21:30:20] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:18:14 [I 2022-10-10 21:30:20] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00307 [I 2022-10-10 21:30:20] ...trap/engines/engine.py.230: loss: 0.19987 [I 2022-10-10 21:30:20] ...optimizers/trijoint.py.151: total_norm: 14.256677 [I 2022-10-10 21:30:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 1050/2304 [I 2022-10-10 21:30:28] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:17:57 [I 2022-10-10 21:30:28] ...trap/engines/engine.py.229: process: 0.85763 | load: 0.00315 [I 2022-10-10 21:30:28] ...trap/engines/engine.py.230: loss: 0.08785 [I 2022-10-10 21:30:28] ...optimizers/trijoint.py.151: total_norm: 10.138146 [I 2022-10-10 21:30:37] ...trap/engines/engine.py.224: train: epoch 86 | batch 1060/2304 [I 2022-10-10 21:30:37] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:17:54 [I 2022-10-10 21:30:37] ...trap/engines/engine.py.229: process: 0.86137 | load: 0.00308 [I 2022-10-10 21:30:37] ...trap/engines/engine.py.230: loss: 0.16349 [I 2022-10-10 21:30:37] ...optimizers/trijoint.py.151: total_norm: 14.345167 [I 2022-10-10 21:30:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 1070/2304 [I 2022-10-10 21:30:46] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:17:45 [I 2022-10-10 21:30:46] ...trap/engines/engine.py.229: process: 0.87181 | load: 0.00324 [I 2022-10-10 21:30:46] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-10 21:30:46] ...optimizers/trijoint.py.151: total_norm: 11.771349 [I 2022-10-10 21:30:54] ...trap/engines/engine.py.224: train: epoch 86 | batch 1080/2304 [I 2022-10-10 21:30:54] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:17:36 [I 2022-10-10 21:30:54] ...trap/engines/engine.py.229: process: 0.87357 | load: 0.00321 [I 2022-10-10 21:30:54] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-10 21:30:54] ...optimizers/trijoint.py.151: total_norm: 24.480667 [I 2022-10-10 21:31:03] ...trap/engines/engine.py.224: train: epoch 86 | batch 1090/2304 [I 2022-10-10 21:31:03] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:17:27 [I 2022-10-10 21:31:03] ...trap/engines/engine.py.229: process: 0.86437 | load: 0.00318 [I 2022-10-10 21:31:03] ...trap/engines/engine.py.230: loss: 0.10518 [I 2022-10-10 21:31:03] ...optimizers/trijoint.py.151: total_norm: 19.444513 [I 2022-10-10 21:31:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 1100/2304 [I 2022-10-10 21:31:12] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:17:18 [I 2022-10-10 21:31:12] ...trap/engines/engine.py.229: process: 0.86768 | load: 0.00315 [I 2022-10-10 21:31:12] ...trap/engines/engine.py.230: loss: 0.09328 [I 2022-10-10 21:31:12] ...optimizers/trijoint.py.151: total_norm: 10.848076 [I 2022-10-10 21:31:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 1110/2304 [I 2022-10-10 21:31:20] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:17:06 [I 2022-10-10 21:31:20] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00313 [I 2022-10-10 21:31:20] ...trap/engines/engine.py.230: loss: 0.08023 [I 2022-10-10 21:31:20] ...optimizers/trijoint.py.151: total_norm: 10.509048 [I 2022-10-10 21:31:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 1120/2304 [I 2022-10-10 21:31:29] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:17:07 [I 2022-10-10 21:31:29] ...trap/engines/engine.py.229: process: 0.88501 | load: 0.00311 [I 2022-10-10 21:31:29] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-10 21:31:29] ...optimizers/trijoint.py.151: total_norm: 10.659718 [I 2022-10-10 21:31:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1130/2304 [I 2022-10-10 21:31:38] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:16:52 [I 2022-10-10 21:31:38] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00320 [I 2022-10-10 21:31:38] ...trap/engines/engine.py.230: loss: 0.15408 [I 2022-10-10 21:31:38] ...optimizers/trijoint.py.151: total_norm: 18.758793 [I 2022-10-10 21:31:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 1140/2304 [I 2022-10-10 21:31:46] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:48 [I 2022-10-10 21:31:46] ...trap/engines/engine.py.229: process: 0.86852 | load: 0.00308 [I 2022-10-10 21:31:46] ...trap/engines/engine.py.230: loss: 0.16017 [I 2022-10-10 21:31:46] ...optimizers/trijoint.py.151: total_norm: 25.934658 [I 2022-10-10 21:31:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1150/2304 [I 2022-10-10 21:31:55] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:16:37 [I 2022-10-10 21:31:55] ...trap/engines/engine.py.229: process: 0.86559 | load: 0.00310 [I 2022-10-10 21:31:55] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-10 21:31:55] ...optimizers/trijoint.py.151: total_norm: 33.787952 [I 2022-10-10 21:32:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1160/2304 [I 2022-10-10 21:32:04] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:16:26 [I 2022-10-10 21:32:04] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00332 [I 2022-10-10 21:32:04] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-10 21:32:04] ...optimizers/trijoint.py.151: total_norm: 15.639102 [I 2022-10-10 21:32:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 1170/2304 [I 2022-10-10 21:32:12] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:16:21 [I 2022-10-10 21:32:12] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00313 [I 2022-10-10 21:32:12] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-10 21:32:12] ...optimizers/trijoint.py.151: total_norm: 26.112326 [I 2022-10-10 21:32:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1180/2304 [I 2022-10-10 21:32:21] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:16:11 [I 2022-10-10 21:32:21] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00310 [I 2022-10-10 21:32:21] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-10 21:32:21] ...optimizers/trijoint.py.151: total_norm: 17.968588 [I 2022-10-10 21:32:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1190/2304 [I 2022-10-10 21:32:30] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:16:04 [I 2022-10-10 21:32:30] ...trap/engines/engine.py.229: process: 0.88099 | load: 0.00315 [I 2022-10-10 21:32:30] ...trap/engines/engine.py.230: loss: 0.09481 [I 2022-10-10 21:32:30] ...optimizers/trijoint.py.151: total_norm: 22.205091 [I 2022-10-10 21:32:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1200/2304 [I 2022-10-10 21:32:38] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:16:05 [I 2022-10-10 21:32:38] ...trap/engines/engine.py.229: process: 0.86183 | load: 0.00331 [I 2022-10-10 21:32:38] ...trap/engines/engine.py.230: loss: 0.14227 [I 2022-10-10 21:32:38] ...optimizers/trijoint.py.151: total_norm: 13.722479 [I 2022-10-10 21:32:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1210/2304 [I 2022-10-10 21:32:47] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:49 [I 2022-10-10 21:32:47] ...trap/engines/engine.py.229: process: 0.87374 | load: 0.00313 [I 2022-10-10 21:32:47] ...trap/engines/engine.py.230: loss: 0.15525 [I 2022-10-10 21:32:47] ...optimizers/trijoint.py.151: total_norm: 20.544382 [I 2022-10-10 21:32:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1220/2304 [I 2022-10-10 21:32:56] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:15:34 [I 2022-10-10 21:32:56] ...trap/engines/engine.py.229: process: 0.86595 | load: 0.00296 [I 2022-10-10 21:32:56] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-10 21:32:56] ...optimizers/trijoint.py.151: total_norm: 16.714291 [I 2022-10-10 21:33:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1230/2304 [I 2022-10-10 21:33:04] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:15:17 [I 2022-10-10 21:33:04] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00292 [I 2022-10-10 21:33:04] ...trap/engines/engine.py.230: loss: 0.17893 [I 2022-10-10 21:33:04] ...optimizers/trijoint.py.151: total_norm: 12.764664 [I 2022-10-10 21:33:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1240/2304 [I 2022-10-10 21:33:13] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:15:11 [I 2022-10-10 21:33:13] ...trap/engines/engine.py.229: process: 0.86385 | load: 0.00295 [I 2022-10-10 21:33:13] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-10 21:33:13] ...optimizers/trijoint.py.151: total_norm: 14.088550 [I 2022-10-10 21:33:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1250/2304 [I 2022-10-10 21:33:21] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:59 [I 2022-10-10 21:33:21] ...trap/engines/engine.py.229: process: 0.85881 | load: 0.00294 [I 2022-10-10 21:33:21] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-10 21:33:21] ...optimizers/trijoint.py.151: total_norm: 34.047646 [I 2022-10-10 21:33:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1260/2304 [I 2022-10-10 21:33:30] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:52 [I 2022-10-10 21:33:30] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00295 [I 2022-10-10 21:33:30] ...trap/engines/engine.py.230: loss: 0.09039 [I 2022-10-10 21:33:30] ...optimizers/trijoint.py.151: total_norm: 10.553827 [I 2022-10-10 21:33:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1270/2304 [I 2022-10-10 21:33:38] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:46 [I 2022-10-10 21:33:38] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00293 [I 2022-10-10 21:33:38] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-10 21:33:38] ...optimizers/trijoint.py.151: total_norm: 23.473400 [I 2022-10-10 21:33:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1280/2304 [I 2022-10-10 21:33:47] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:33 [I 2022-10-10 21:33:47] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00293 [I 2022-10-10 21:33:47] ...trap/engines/engine.py.230: loss: 0.08332 [I 2022-10-10 21:33:47] ...optimizers/trijoint.py.151: total_norm: 9.272182 [I 2022-10-10 21:33:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1290/2304 [I 2022-10-10 21:33:56] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:21 [I 2022-10-10 21:33:56] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00295 [I 2022-10-10 21:33:56] ...trap/engines/engine.py.230: loss: 0.09139 [I 2022-10-10 21:33:56] ...optimizers/trijoint.py.151: total_norm: 12.868453 [I 2022-10-10 21:34:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1300/2304 [I 2022-10-10 21:34:04] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:14:11 [I 2022-10-10 21:34:04] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00296 [I 2022-10-10 21:34:04] ...trap/engines/engine.py.230: loss: 0.14574 [I 2022-10-10 21:34:04] ...optimizers/trijoint.py.151: total_norm: 14.987750 [I 2022-10-10 21:34:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1310/2304 [I 2022-10-10 21:34:13] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:14:07 [I 2022-10-10 21:34:13] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00293 [I 2022-10-10 21:34:13] ...trap/engines/engine.py.230: loss: 0.12366 [I 2022-10-10 21:34:13] ...optimizers/trijoint.py.151: total_norm: 19.621323 [I 2022-10-10 21:34:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1320/2304 [I 2022-10-10 21:34:21] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:57 [I 2022-10-10 21:34:21] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00293 [I 2022-10-10 21:34:21] ...trap/engines/engine.py.230: loss: 0.08313 [I 2022-10-10 21:34:21] ...optimizers/trijoint.py.151: total_norm: 15.381804 [I 2022-10-10 21:34:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1330/2304 [I 2022-10-10 21:34:30] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:56 [I 2022-10-10 21:34:30] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00296 [I 2022-10-10 21:34:30] ...trap/engines/engine.py.230: loss: 0.15605 [I 2022-10-10 21:34:30] ...optimizers/trijoint.py.151: total_norm: 19.612232 [I 2022-10-10 21:34:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1340/2304 [I 2022-10-10 21:34:38] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:46 [I 2022-10-10 21:34:38] ...trap/engines/engine.py.229: process: 0.85556 | load: 0.00295 [I 2022-10-10 21:34:38] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-10 21:34:38] ...optimizers/trijoint.py.151: total_norm: 12.813135 [I 2022-10-10 21:34:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1350/2304 [I 2022-10-10 21:34:47] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:37 [I 2022-10-10 21:34:47] ...trap/engines/engine.py.229: process: 0.86591 | load: 0.00297 [I 2022-10-10 21:34:47] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-10 21:34:47] ...optimizers/trijoint.py.151: total_norm: 35.290195 [I 2022-10-10 21:34:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1360/2304 [I 2022-10-10 21:34:55] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:28 [I 2022-10-10 21:34:55] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00293 [I 2022-10-10 21:34:55] ...trap/engines/engine.py.230: loss: 0.16021 [I 2022-10-10 21:34:55] ...optimizers/trijoint.py.151: total_norm: 14.707066 [I 2022-10-10 21:35:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1370/2304 [I 2022-10-10 21:35:04] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:20 [I 2022-10-10 21:35:04] ...trap/engines/engine.py.229: process: 0.85641 | load: 0.00294 [I 2022-10-10 21:35:04] ...trap/engines/engine.py.230: loss: 0.17102 [I 2022-10-10 21:35:04] ...optimizers/trijoint.py.151: total_norm: 20.770048 [I 2022-10-10 21:35:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1380/2304 [I 2022-10-10 21:35:13] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:13:06 [I 2022-10-10 21:35:13] ...trap/engines/engine.py.229: process: 0.85485 | load: 0.00296 [I 2022-10-10 21:35:13] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-10 21:35:13] ...optimizers/trijoint.py.151: total_norm: 18.165743 [I 2022-10-10 21:35:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1390/2304 [I 2022-10-10 21:35:21] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:13:00 [I 2022-10-10 21:35:21] ...trap/engines/engine.py.229: process: 0.86137 | load: 0.00295 [I 2022-10-10 21:35:21] ...trap/engines/engine.py.230: loss: 0.15852 [I 2022-10-10 21:35:21] ...optimizers/trijoint.py.151: total_norm: 10.779962 [I 2022-10-10 21:35:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1400/2304 [I 2022-10-10 21:35:30] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:55 [I 2022-10-10 21:35:30] ...trap/engines/engine.py.229: process: 0.86103 | load: 0.00293 [I 2022-10-10 21:35:30] ...trap/engines/engine.py.230: loss: 0.15304 [I 2022-10-10 21:35:30] ...optimizers/trijoint.py.151: total_norm: 13.820777 [I 2022-10-10 21:35:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1410/2304 [I 2022-10-10 21:35:38] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:45 [I 2022-10-10 21:35:38] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00292 [I 2022-10-10 21:35:38] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-10 21:35:38] ...optimizers/trijoint.py.151: total_norm: 28.498137 [I 2022-10-10 21:35:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1420/2304 [I 2022-10-10 21:35:47] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:35 [I 2022-10-10 21:35:47] ...trap/engines/engine.py.229: process: 0.86011 | load: 0.00297 [I 2022-10-10 21:35:47] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-10 21:35:47] ...optimizers/trijoint.py.151: total_norm: 16.301928 [I 2022-10-10 21:35:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1430/2304 [I 2022-10-10 21:35:55] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:31 [I 2022-10-10 21:35:55] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00294 [I 2022-10-10 21:35:55] ...trap/engines/engine.py.230: loss: 0.13676 [I 2022-10-10 21:35:55] ...optimizers/trijoint.py.151: total_norm: 20.652271 [I 2022-10-10 21:36:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1440/2304 [I 2022-10-10 21:36:04] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:12:14 [I 2022-10-10 21:36:04] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00314 [I 2022-10-10 21:36:04] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-10 21:36:04] ...optimizers/trijoint.py.151: total_norm: 12.737961 [I 2022-10-10 21:36:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1450/2304 [I 2022-10-10 21:36:13] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:12:08 [I 2022-10-10 21:36:13] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00293 [I 2022-10-10 21:36:13] ...trap/engines/engine.py.230: loss: 0.14163 [I 2022-10-10 21:36:13] ...optimizers/trijoint.py.151: total_norm: 20.306116 [I 2022-10-10 21:36:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1460/2304 [I 2022-10-10 21:36:21] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:12:00 [I 2022-10-10 21:36:21] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00296 [I 2022-10-10 21:36:21] ...trap/engines/engine.py.230: loss: 0.15261 [I 2022-10-10 21:36:21] ...optimizers/trijoint.py.151: total_norm: 15.321102 [I 2022-10-10 21:36:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1470/2304 [I 2022-10-10 21:36:30] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:51 [I 2022-10-10 21:36:30] ...trap/engines/engine.py.229: process: 0.84731 | load: 0.00302 [I 2022-10-10 21:36:30] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-10 21:36:30] ...optimizers/trijoint.py.151: total_norm: 16.523046 [I 2022-10-10 21:36:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1480/2304 [I 2022-10-10 21:36:38] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:52 [I 2022-10-10 21:36:38] ...trap/engines/engine.py.229: process: 0.86318 | load: 0.00315 [I 2022-10-10 21:36:38] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-10 21:36:38] ...optimizers/trijoint.py.151: total_norm: 13.129215 [I 2022-10-10 21:36:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1490/2304 [I 2022-10-10 21:36:47] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:11:42 [I 2022-10-10 21:36:47] ...trap/engines/engine.py.229: process: 0.86266 | load: 0.00362 [I 2022-10-10 21:36:47] ...trap/engines/engine.py.230: loss: 0.10184 [I 2022-10-10 21:36:47] ...optimizers/trijoint.py.151: total_norm: 11.841464 [I 2022-10-10 21:36:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1500/2304 [I 2022-10-10 21:36:56] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:39 [I 2022-10-10 21:36:56] ...trap/engines/engine.py.229: process: 0.87846 | load: 0.00309 [I 2022-10-10 21:36:56] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-10 21:36:56] ...optimizers/trijoint.py.151: total_norm: 15.298990 [I 2022-10-10 21:37:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1510/2304 [I 2022-10-10 21:37:04] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:11:31 [I 2022-10-10 21:37:04] ...trap/engines/engine.py.229: process: 0.87574 | load: 0.00312 [I 2022-10-10 21:37:04] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-10 21:37:04] ...optimizers/trijoint.py.151: total_norm: 18.039011 [I 2022-10-10 21:37:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1520/2304 [I 2022-10-10 21:37:13] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:11:23 [I 2022-10-10 21:37:13] ...trap/engines/engine.py.229: process: 0.88279 | load: 0.00313 [I 2022-10-10 21:37:13] ...trap/engines/engine.py.230: loss: 0.12089 [I 2022-10-10 21:37:13] ...optimizers/trijoint.py.151: total_norm: 18.239601 [I 2022-10-10 21:37:22] ...trap/engines/engine.py.224: train: epoch 86 | batch 1530/2304 [I 2022-10-10 21:37:22] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:11:09 [I 2022-10-10 21:37:22] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00311 [I 2022-10-10 21:37:22] ...trap/engines/engine.py.230: loss: 0.14293 [I 2022-10-10 21:37:22] ...optimizers/trijoint.py.151: total_norm: 19.734520 [I 2022-10-10 21:37:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 1540/2304 [I 2022-10-10 21:37:30] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:59 [I 2022-10-10 21:37:30] ...trap/engines/engine.py.229: process: 0.87021 | load: 0.00320 [I 2022-10-10 21:37:30] ...trap/engines/engine.py.230: loss: 0.09043 [I 2022-10-10 21:37:30] ...optimizers/trijoint.py.151: total_norm: 10.775033 [I 2022-10-10 21:37:39] ...trap/engines/engine.py.224: train: epoch 86 | batch 1550/2304 [I 2022-10-10 21:37:39] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:50 [I 2022-10-10 21:37:39] ...trap/engines/engine.py.229: process: 0.85405 | load: 0.00314 [I 2022-10-10 21:37:39] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-10 21:37:39] ...optimizers/trijoint.py.151: total_norm: 21.661766 [I 2022-10-10 21:37:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 1560/2304 [I 2022-10-10 21:37:48] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:42 [I 2022-10-10 21:37:48] ...trap/engines/engine.py.229: process: 0.86245 | load: 0.00307 [I 2022-10-10 21:37:48] ...trap/engines/engine.py.230: loss: 0.11642 [I 2022-10-10 21:37:48] ...optimizers/trijoint.py.151: total_norm: 20.365101 [I 2022-10-10 21:37:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1570/2304 [I 2022-10-10 21:37:56] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:33 [I 2022-10-10 21:37:56] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00315 [I 2022-10-10 21:37:56] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-10 21:37:56] ...optimizers/trijoint.py.151: total_norm: 53.302845 [I 2022-10-10 21:38:05] ...trap/engines/engine.py.224: train: epoch 86 | batch 1580/2304 [I 2022-10-10 21:38:05] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:28 [I 2022-10-10 21:38:05] ...trap/engines/engine.py.229: process: 0.86845 | load: 0.00336 [I 2022-10-10 21:38:05] ...trap/engines/engine.py.230: loss: 0.12869 [I 2022-10-10 21:38:05] ...optimizers/trijoint.py.151: total_norm: 23.463806 [I 2022-10-10 21:38:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 1590/2304 [I 2022-10-10 21:38:14] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:10:20 [I 2022-10-10 21:38:14] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00331 [I 2022-10-10 21:38:14] ...trap/engines/engine.py.230: loss: 0.09432 [I 2022-10-10 21:38:14] ...optimizers/trijoint.py.151: total_norm: 11.131569 [I 2022-10-10 21:38:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 1600/2304 [I 2022-10-10 21:38:23] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:10:08 [I 2022-10-10 21:38:23] ...trap/engines/engine.py.229: process: 0.86204 | load: 0.00315 [I 2022-10-10 21:38:23] ...trap/engines/engine.py.230: loss: 0.14191 [I 2022-10-10 21:38:23] ...optimizers/trijoint.py.151: total_norm: 15.514199 [I 2022-10-10 21:38:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 1610/2304 [I 2022-10-10 21:38:31] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:10:02 [I 2022-10-10 21:38:31] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00311 [I 2022-10-10 21:38:31] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-10 21:38:31] ...optimizers/trijoint.py.151: total_norm: 10.208053 [I 2022-10-10 21:38:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 1620/2304 [I 2022-10-10 21:38:40] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:53 [I 2022-10-10 21:38:40] ...trap/engines/engine.py.229: process: 0.87407 | load: 0.00306 [I 2022-10-10 21:38:40] ...trap/engines/engine.py.230: loss: 0.13030 [I 2022-10-10 21:38:40] ...optimizers/trijoint.py.151: total_norm: 9.604378 [I 2022-10-10 21:38:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 1630/2304 [I 2022-10-10 21:38:49] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:42 [I 2022-10-10 21:38:49] ...trap/engines/engine.py.229: process: 0.86575 | load: 0.00311 [I 2022-10-10 21:38:49] ...trap/engines/engine.py.230: loss: 0.12677 [I 2022-10-10 21:38:49] ...optimizers/trijoint.py.151: total_norm: 14.250600 [I 2022-10-10 21:38:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 1640/2304 [I 2022-10-10 21:38:57] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:36 [I 2022-10-10 21:38:57] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00312 [I 2022-10-10 21:38:57] ...trap/engines/engine.py.230: loss: 0.16829 [I 2022-10-10 21:38:57] ...optimizers/trijoint.py.151: total_norm: 11.234639 [I 2022-10-10 21:39:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 1650/2304 [I 2022-10-10 21:39:06] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:26 [I 2022-10-10 21:39:06] ...trap/engines/engine.py.229: process: 0.87243 | load: 0.00313 [I 2022-10-10 21:39:06] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-10 21:39:06] ...optimizers/trijoint.py.151: total_norm: 16.497175 [I 2022-10-10 21:39:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 1660/2304 [I 2022-10-10 21:39:15] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:17 [I 2022-10-10 21:39:15] ...trap/engines/engine.py.229: process: 0.85726 | load: 0.00317 [I 2022-10-10 21:39:15] ...trap/engines/engine.py.230: loss: 0.19625 [I 2022-10-10 21:39:15] ...optimizers/trijoint.py.151: total_norm: 18.642557 [I 2022-10-10 21:39:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 1670/2304 [I 2022-10-10 21:39:23] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:09:06 [I 2022-10-10 21:39:23] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00313 [I 2022-10-10 21:39:23] ...trap/engines/engine.py.230: loss: 0.15911 [I 2022-10-10 21:39:23] ...optimizers/trijoint.py.151: total_norm: 9.348750 [I 2022-10-10 21:39:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 1680/2304 [I 2022-10-10 21:39:32] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:54 [I 2022-10-10 21:39:32] ...trap/engines/engine.py.229: process: 0.83289 | load: 0.00312 [I 2022-10-10 21:39:32] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-10 21:39:32] ...optimizers/trijoint.py.151: total_norm: 12.656245 [I 2022-10-10 21:39:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1690/2304 [I 2022-10-10 21:39:41] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:52 [I 2022-10-10 21:39:41] ...trap/engines/engine.py.229: process: 0.88454 | load: 0.00307 [I 2022-10-10 21:39:41] ...trap/engines/engine.py.230: loss: 0.10467 [I 2022-10-10 21:39:41] ...optimizers/trijoint.py.151: total_norm: 12.534942 [I 2022-10-10 21:39:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 1700/2304 [I 2022-10-10 21:39:49] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:41 [I 2022-10-10 21:39:49] ...trap/engines/engine.py.229: process: 0.84960 | load: 0.00312 [I 2022-10-10 21:39:49] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-10 21:39:49] ...optimizers/trijoint.py.151: total_norm: 14.573715 [I 2022-10-10 21:39:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 1710/2304 [I 2022-10-10 21:39:58] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:35 [I 2022-10-10 21:39:58] ...trap/engines/engine.py.229: process: 0.87164 | load: 0.00313 [I 2022-10-10 21:39:58] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-10 21:39:58] ...optimizers/trijoint.py.151: total_norm: 14.824834 [I 2022-10-10 21:40:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1720/2304 [I 2022-10-10 21:40:07] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:08:26 [I 2022-10-10 21:40:07] ...trap/engines/engine.py.229: process: 0.87835 | load: 0.00312 [I 2022-10-10 21:40:07] ...trap/engines/engine.py.230: loss: 0.10778 [I 2022-10-10 21:40:07] ...optimizers/trijoint.py.151: total_norm: 41.638927 [I 2022-10-10 21:40:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 1730/2304 [I 2022-10-10 21:40:15] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:08:19 [I 2022-10-10 21:40:15] ...trap/engines/engine.py.229: process: 0.88309 | load: 0.00314 [I 2022-10-10 21:40:15] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-10 21:40:15] ...optimizers/trijoint.py.151: total_norm: 16.289062 [I 2022-10-10 21:40:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1740/2304 [I 2022-10-10 21:40:24] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:08:10 [I 2022-10-10 21:40:24] ...trap/engines/engine.py.229: process: 0.87019 | load: 0.00311 [I 2022-10-10 21:40:24] ...trap/engines/engine.py.230: loss: 0.12868 [I 2022-10-10 21:40:24] ...optimizers/trijoint.py.151: total_norm: 16.471525 [I 2022-10-10 21:40:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 1750/2304 [I 2022-10-10 21:40:33] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:56 [I 2022-10-10 21:40:33] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00291 [I 2022-10-10 21:40:33] ...trap/engines/engine.py.230: loss: 0.09577 [I 2022-10-10 21:40:33] ...optimizers/trijoint.py.151: total_norm: 10.134412 [I 2022-10-10 21:40:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1760/2304 [I 2022-10-10 21:40:41] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:46 [I 2022-10-10 21:40:41] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00296 [I 2022-10-10 21:40:41] ...trap/engines/engine.py.230: loss: 0.10834 [I 2022-10-10 21:40:41] ...optimizers/trijoint.py.151: total_norm: 17.045519 [I 2022-10-10 21:40:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1770/2304 [I 2022-10-10 21:40:50] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:34 [I 2022-10-10 21:40:50] ...trap/engines/engine.py.229: process: 0.85755 | load: 0.00290 [I 2022-10-10 21:40:50] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-10 21:40:50] ...optimizers/trijoint.py.151: total_norm: 17.762802 [I 2022-10-10 21:40:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 1780/2304 [I 2022-10-10 21:40:58] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:26 [I 2022-10-10 21:40:58] ...trap/engines/engine.py.229: process: 0.83597 | load: 0.00294 [I 2022-10-10 21:40:58] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-10 21:40:58] ...optimizers/trijoint.py.151: total_norm: 16.647573 [I 2022-10-10 21:41:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1790/2304 [I 2022-10-10 21:41:07] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:07:19 [I 2022-10-10 21:41:07] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00297 [I 2022-10-10 21:41:07] ...trap/engines/engine.py.230: loss: 0.18406 [I 2022-10-10 21:41:07] ...optimizers/trijoint.py.151: total_norm: 7.563582 [I 2022-10-10 21:41:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 1800/2304 [I 2022-10-10 21:41:16] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:07:09 [I 2022-10-10 21:41:16] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00295 [I 2022-10-10 21:41:16] ...trap/engines/engine.py.230: loss: 0.12025 [I 2022-10-10 21:41:16] ...optimizers/trijoint.py.151: total_norm: 13.947790 [I 2022-10-10 21:41:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1810/2304 [I 2022-10-10 21:41:24] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:07:01 [I 2022-10-10 21:41:24] ...trap/engines/engine.py.229: process: 0.85954 | load: 0.00292 [I 2022-10-10 21:41:24] ...trap/engines/engine.py.230: loss: 0.10171 [I 2022-10-10 21:41:24] ...optimizers/trijoint.py.151: total_norm: 8.630598 [I 2022-10-10 21:41:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 1820/2304 [I 2022-10-10 21:41:33] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:51 [I 2022-10-10 21:41:33] ...trap/engines/engine.py.229: process: 0.84618 | load: 0.00297 [I 2022-10-10 21:41:33] ...trap/engines/engine.py.230: loss: 0.11018 [I 2022-10-10 21:41:33] ...optimizers/trijoint.py.151: total_norm: 12.420140 [I 2022-10-10 21:41:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1830/2304 [I 2022-10-10 21:41:41] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:46 [I 2022-10-10 21:41:41] ...trap/engines/engine.py.229: process: 0.87505 | load: 0.00292 [I 2022-10-10 21:41:41] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-10 21:41:41] ...optimizers/trijoint.py.151: total_norm: 34.318451 [I 2022-10-10 21:41:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1840/2304 [I 2022-10-10 21:41:50] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:36 [I 2022-10-10 21:41:50] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00296 [I 2022-10-10 21:41:50] ...trap/engines/engine.py.230: loss: 0.09474 [I 2022-10-10 21:41:50] ...optimizers/trijoint.py.151: total_norm: 11.233903 [I 2022-10-10 21:41:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 1850/2304 [I 2022-10-10 21:41:58] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:06:28 [I 2022-10-10 21:41:58] ...trap/engines/engine.py.229: process: 0.86418 | load: 0.00303 [I 2022-10-10 21:41:58] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-10 21:41:58] ...optimizers/trijoint.py.151: total_norm: 16.473867 [I 2022-10-10 21:42:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1860/2304 [I 2022-10-10 21:42:07] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:06:17 [I 2022-10-10 21:42:07] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00296 [I 2022-10-10 21:42:07] ...trap/engines/engine.py.230: loss: 0.12433 [I 2022-10-10 21:42:07] ...optimizers/trijoint.py.151: total_norm: 17.628366 [I 2022-10-10 21:42:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 1870/2304 [I 2022-10-10 21:42:16] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:06:10 [I 2022-10-10 21:42:16] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00297 [I 2022-10-10 21:42:16] ...trap/engines/engine.py.230: loss: 0.13030 [I 2022-10-10 21:42:16] ...optimizers/trijoint.py.151: total_norm: 24.671865 [I 2022-10-10 21:42:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1880/2304 [I 2022-10-10 21:42:24] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:06:02 [I 2022-10-10 21:42:24] ...trap/engines/engine.py.229: process: 0.86507 | load: 0.00300 [I 2022-10-10 21:42:24] ...trap/engines/engine.py.230: loss: 0.22513 [I 2022-10-10 21:42:24] ...optimizers/trijoint.py.151: total_norm: 18.676376 [I 2022-10-10 21:42:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 1890/2304 [I 2022-10-10 21:42:33] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:53 [I 2022-10-10 21:42:33] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00301 [I 2022-10-10 21:42:33] ...trap/engines/engine.py.230: loss: 0.11650 [I 2022-10-10 21:42:33] ...optimizers/trijoint.py.151: total_norm: 18.132767 [I 2022-10-10 21:42:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1900/2304 [I 2022-10-10 21:42:41] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:46 [I 2022-10-10 21:42:41] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00301 [I 2022-10-10 21:42:41] ...trap/engines/engine.py.230: loss: 0.07897 [I 2022-10-10 21:42:41] ...optimizers/trijoint.py.151: total_norm: 13.968600 [I 2022-10-10 21:42:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1910/2304 [I 2022-10-10 21:42:50] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:37 [I 2022-10-10 21:42:50] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00298 [I 2022-10-10 21:42:50] ...trap/engines/engine.py.230: loss: 0.18078 [I 2022-10-10 21:42:50] ...optimizers/trijoint.py.151: total_norm: 15.420181 [I 2022-10-10 21:42:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 1920/2304 [I 2022-10-10 21:42:58] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:05:30 [I 2022-10-10 21:42:58] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00315 [I 2022-10-10 21:42:58] ...trap/engines/engine.py.230: loss: 0.11131 [I 2022-10-10 21:42:58] ...optimizers/trijoint.py.151: total_norm: 14.460684 [I 2022-10-10 21:43:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1930/2304 [I 2022-10-10 21:43:07] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:05:19 [I 2022-10-10 21:43:07] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00293 [I 2022-10-10 21:43:07] ...trap/engines/engine.py.230: loss: 0.12935 [I 2022-10-10 21:43:07] ...optimizers/trijoint.py.151: total_norm: 10.700979 [I 2022-10-10 21:43:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 1940/2304 [I 2022-10-10 21:43:16] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:05:10 [I 2022-10-10 21:43:16] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00298 [I 2022-10-10 21:43:16] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-10 21:43:16] ...optimizers/trijoint.py.151: total_norm: 15.911580 [I 2022-10-10 21:43:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1950/2304 [I 2022-10-10 21:43:24] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:05:02 [I 2022-10-10 21:43:24] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00299 [I 2022-10-10 21:43:24] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-10 21:43:24] ...optimizers/trijoint.py.151: total_norm: 22.215065 [I 2022-10-10 21:43:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 1960/2304 [I 2022-10-10 21:43:33] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:52 [I 2022-10-10 21:43:33] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00303 [I 2022-10-10 21:43:33] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-10 21:43:33] ...optimizers/trijoint.py.151: total_norm: 9.597420 [I 2022-10-10 21:43:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1970/2304 [I 2022-10-10 21:43:41] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:44 [I 2022-10-10 21:43:41] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00295 [I 2022-10-10 21:43:41] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-10 21:43:41] ...optimizers/trijoint.py.151: total_norm: 21.689646 [I 2022-10-10 21:43:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1980/2304 [I 2022-10-10 21:43:50] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:37 [I 2022-10-10 21:43:50] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00322 [I 2022-10-10 21:43:50] ...trap/engines/engine.py.230: loss: 0.11681 [I 2022-10-10 21:43:50] ...optimizers/trijoint.py.151: total_norm: 8.648072 [I 2022-10-10 21:43:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 1990/2304 [I 2022-10-10 21:43:59] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:04:31 [I 2022-10-10 21:43:59] ...trap/engines/engine.py.229: process: 0.88235 | load: 0.00310 [I 2022-10-10 21:43:59] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-10 21:43:59] ...optimizers/trijoint.py.151: total_norm: 13.893205 [I 2022-10-10 21:44:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 2000/2304 [I 2022-10-10 21:44:07] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:04:23 [I 2022-10-10 21:44:07] ...trap/engines/engine.py.229: process: 0.87371 | load: 0.00313 [I 2022-10-10 21:44:07] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-10 21:44:07] ...optimizers/trijoint.py.151: total_norm: 30.514729 [I 2022-10-10 21:44:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 2010/2304 [I 2022-10-10 21:44:16] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:13 [I 2022-10-10 21:44:16] ...trap/engines/engine.py.229: process: 0.85873 | load: 0.00311 [I 2022-10-10 21:44:16] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-10 21:44:16] ...optimizers/trijoint.py.151: total_norm: 16.277481 [I 2022-10-10 21:44:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 2020/2304 [I 2022-10-10 21:44:25] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:04:05 [I 2022-10-10 21:44:25] ...trap/engines/engine.py.229: process: 0.86200 | load: 0.00312 [I 2022-10-10 21:44:25] ...trap/engines/engine.py.230: loss: 0.10022 [I 2022-10-10 21:44:25] ...optimizers/trijoint.py.151: total_norm: 16.002443 [I 2022-10-10 21:44:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 2030/2304 [I 2022-10-10 21:44:33] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:59 [I 2022-10-10 21:44:33] ...trap/engines/engine.py.229: process: 0.88001 | load: 0.00312 [I 2022-10-10 21:44:33] ...trap/engines/engine.py.230: loss: 0.13199 [I 2022-10-10 21:44:33] ...optimizers/trijoint.py.151: total_norm: 36.700462 [I 2022-10-10 21:44:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 2040/2304 [I 2022-10-10 21:44:42] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:48 [I 2022-10-10 21:44:42] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00313 [I 2022-10-10 21:44:42] ...trap/engines/engine.py.230: loss: 0.12453 [I 2022-10-10 21:44:42] ...optimizers/trijoint.py.151: total_norm: 12.974682 [I 2022-10-10 21:44:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 2050/2304 [I 2022-10-10 21:44:51] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:40 [I 2022-10-10 21:44:51] ...trap/engines/engine.py.229: process: 0.87005 | load: 0.00313 [I 2022-10-10 21:44:51] ...trap/engines/engine.py.230: loss: 0.09590 [I 2022-10-10 21:44:51] ...optimizers/trijoint.py.151: total_norm: 9.099214 [I 2022-10-10 21:44:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 2060/2304 [I 2022-10-10 21:44:59] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:30 [I 2022-10-10 21:44:59] ...trap/engines/engine.py.229: process: 0.87214 | load: 0.00312 [I 2022-10-10 21:44:59] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-10 21:44:59] ...optimizers/trijoint.py.151: total_norm: 27.994577 [I 2022-10-10 21:45:08] ...trap/engines/engine.py.224: train: epoch 86 | batch 2070/2304 [I 2022-10-10 21:45:08] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:03:23 [I 2022-10-10 21:45:08] ...trap/engines/engine.py.229: process: 0.88335 | load: 0.00308 [I 2022-10-10 21:45:08] ...trap/engines/engine.py.230: loss: 0.08721 [I 2022-10-10 21:45:08] ...optimizers/trijoint.py.151: total_norm: 9.247709 [I 2022-10-10 21:45:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 2080/2304 [I 2022-10-10 21:45:17] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:03:15 [I 2022-10-10 21:45:17] ...trap/engines/engine.py.229: process: 0.87030 | load: 0.00323 [I 2022-10-10 21:45:17] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-10 21:45:17] ...optimizers/trijoint.py.151: total_norm: 27.152252 [I 2022-10-10 21:45:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 2090/2304 [I 2022-10-10 21:45:26] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:03:06 [I 2022-10-10 21:45:26] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00307 [I 2022-10-10 21:45:26] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-10 21:45:26] ...optimizers/trijoint.py.151: total_norm: 12.462955 [I 2022-10-10 21:45:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 2100/2304 [I 2022-10-10 21:45:34] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:56 [I 2022-10-10 21:45:34] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00323 [I 2022-10-10 21:45:34] ...trap/engines/engine.py.230: loss: 0.13896 [I 2022-10-10 21:45:34] ...optimizers/trijoint.py.151: total_norm: 17.975101 [I 2022-10-10 21:45:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 2110/2304 [I 2022-10-10 21:45:43] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:48 [I 2022-10-10 21:45:43] ...trap/engines/engine.py.229: process: 0.86804 | load: 0.00311 [I 2022-10-10 21:45:43] ...trap/engines/engine.py.230: loss: 0.10269 [I 2022-10-10 21:45:43] ...optimizers/trijoint.py.151: total_norm: 9.010993 [I 2022-10-10 21:45:52] ...trap/engines/engine.py.224: train: epoch 86 | batch 2120/2304 [I 2022-10-10 21:45:52] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:38 [I 2022-10-10 21:45:52] ...trap/engines/engine.py.229: process: 0.86784 | load: 0.00310 [I 2022-10-10 21:45:52] ...trap/engines/engine.py.230: loss: 0.13891 [I 2022-10-10 21:45:52] ...optimizers/trijoint.py.151: total_norm: 10.723607 [I 2022-10-10 21:46:00] ...trap/engines/engine.py.224: train: epoch 86 | batch 2130/2304 [I 2022-10-10 21:46:00] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:31 [I 2022-10-10 21:46:00] ...trap/engines/engine.py.229: process: 0.88526 | load: 0.00331 [I 2022-10-10 21:46:00] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-10 21:46:00] ...optimizers/trijoint.py.151: total_norm: 22.561474 [I 2022-10-10 21:46:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 2140/2304 [I 2022-10-10 21:46:09] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:21 [I 2022-10-10 21:46:09] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00311 [I 2022-10-10 21:46:09] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-10 21:46:09] ...optimizers/trijoint.py.151: total_norm: 29.310074 [I 2022-10-10 21:46:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 2150/2304 [I 2022-10-10 21:46:18] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:02:13 [I 2022-10-10 21:46:18] ...trap/engines/engine.py.229: process: 0.87299 | load: 0.00313 [I 2022-10-10 21:46:18] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-10 21:46:18] ...optimizers/trijoint.py.151: total_norm: 27.118172 [I 2022-10-10 21:46:26] ...trap/engines/engine.py.224: train: epoch 86 | batch 2160/2304 [I 2022-10-10 21:46:27] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:02:04 [I 2022-10-10 21:46:27] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00315 [I 2022-10-10 21:46:27] ...trap/engines/engine.py.230: loss: 0.20202 [I 2022-10-10 21:46:27] ...optimizers/trijoint.py.151: total_norm: 23.014275 [I 2022-10-10 21:46:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 2170/2304 [I 2022-10-10 21:46:35] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:55 [I 2022-10-10 21:46:35] ...trap/engines/engine.py.229: process: 0.87783 | load: 0.00316 [I 2022-10-10 21:46:35] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-10 21:46:35] ...optimizers/trijoint.py.151: total_norm: 19.355570 [I 2022-10-10 21:46:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 2180/2304 [I 2022-10-10 21:46:44] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:47 [I 2022-10-10 21:46:44] ...trap/engines/engine.py.229: process: 0.86315 | load: 0.00310 [I 2022-10-10 21:46:44] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-10 21:46:44] ...optimizers/trijoint.py.151: total_norm: 41.273010 [I 2022-10-10 21:46:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 2190/2304 [I 2022-10-10 21:46:53] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:38 [I 2022-10-10 21:46:53] ...trap/engines/engine.py.229: process: 0.85757 | load: 0.00314 [I 2022-10-10 21:46:53] ...trap/engines/engine.py.230: loss: 0.19437 [I 2022-10-10 21:46:53] ...optimizers/trijoint.py.151: total_norm: 15.533900 [I 2022-10-10 21:47:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 2200/2304 [I 2022-10-10 21:47:01] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:30 [I 2022-10-10 21:47:01] ...trap/engines/engine.py.229: process: 0.86415 | load: 0.00309 [I 2022-10-10 21:47:01] ...trap/engines/engine.py.230: loss: 0.17269 [I 2022-10-10 21:47:01] ...optimizers/trijoint.py.151: total_norm: 21.681826 [I 2022-10-10 21:47:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 2210/2304 [I 2022-10-10 21:47:10] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:01:21 [I 2022-10-10 21:47:10] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00308 [I 2022-10-10 21:47:10] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-10 21:47:10] ...optimizers/trijoint.py.151: total_norm: 12.893578 [I 2022-10-10 21:47:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 2220/2304 [I 2022-10-10 21:47:19] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:01:13 [I 2022-10-10 21:47:19] ...trap/engines/engine.py.229: process: 0.87175 | load: 0.00348 [I 2022-10-10 21:47:19] ...trap/engines/engine.py.230: loss: 0.11778 [I 2022-10-10 21:47:19] ...optimizers/trijoint.py.151: total_norm: 10.935107 [I 2022-10-10 21:47:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 2230/2304 [I 2022-10-10 21:47:27] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:01:04 [I 2022-10-10 21:47:27] ...trap/engines/engine.py.229: process: 0.87448 | load: 0.00310 [I 2022-10-10 21:47:27] ...trap/engines/engine.py.230: loss: 0.15175 [I 2022-10-10 21:47:27] ...optimizers/trijoint.py.151: total_norm: 14.532099 [I 2022-10-10 21:47:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 2240/2304 [I 2022-10-10 21:47:36] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:55 [I 2022-10-10 21:47:36] ...trap/engines/engine.py.229: process: 0.86708 | load: 0.00309 [I 2022-10-10 21:47:36] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-10 21:47:36] ...optimizers/trijoint.py.151: total_norm: 72.922668 [I 2022-10-10 21:47:45] ...trap/engines/engine.py.224: train: epoch 86 | batch 2250/2304 [I 2022-10-10 21:47:45] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:46 [I 2022-10-10 21:47:45] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00293 [I 2022-10-10 21:47:45] ...trap/engines/engine.py.230: loss: 0.10743 [I 2022-10-10 21:47:45] ...optimizers/trijoint.py.151: total_norm: 18.932613 [I 2022-10-10 21:47:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 2260/2304 [I 2022-10-10 21:47:53] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:37 [I 2022-10-10 21:47:53] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00300 [I 2022-10-10 21:47:53] ...trap/engines/engine.py.230: loss: 0.09887 [I 2022-10-10 21:47:53] ...optimizers/trijoint.py.151: total_norm: 14.335967 [I 2022-10-10 21:48:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 2270/2304 [I 2022-10-10 21:48:02] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:28 [I 2022-10-10 21:48:02] ...trap/engines/engine.py.229: process: 0.83442 | load: 0.00294 [I 2022-10-10 21:48:02] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-10 21:48:02] ...optimizers/trijoint.py.151: total_norm: 14.922386 [I 2022-10-10 21:48:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 2280/2304 [I 2022-10-10 21:48:10] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:20 [I 2022-10-10 21:48:10] ...trap/engines/engine.py.229: process: 0.85909 | load: 0.00316 [I 2022-10-10 21:48:10] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-10 21:48:10] ...optimizers/trijoint.py.151: total_norm: 18.307547 [I 2022-10-10 21:48:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 2290/2304 [I 2022-10-10 21:48:19] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:11 [I 2022-10-10 21:48:19] ...trap/engines/engine.py.229: process: 0.85416 | load: 0.00295 [I 2022-10-10 21:48:19] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-10 21:48:19] ...optimizers/trijoint.py.151: total_norm: 15.917342 [I 2022-10-10 21:48:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 2300/2304 [I 2022-10-10 21:48:28] ...trap/engines/engine.py.228: elapsed: 0:33:28 | left: 0:00:03 [I 2022-10-10 21:48:28] ...trap/engines/engine.py.229: process: 0.83791 | load: 0.00016 [I 2022-10-10 21:48:28] ...trap/engines/engine.py.230: loss: 0.16217 [I 2022-10-10 21:48:28] ...optimizers/trijoint.py.151: total_norm: 17.617867 [I 2022-10-10 21:48:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 2304/2304 [I 2022-10-10 21:48:31] ...trap/engines/engine.py.228: elapsed: 0:33:32 | left: 0:00:00 [I 2022-10-10 21:48:31] ...trap/engines/engine.py.229: process: 0.83870 | load: 0.00018 [I 2022-10-10 21:48:31] ...trap/engines/engine.py.230: loss: 0.10030 [I 2022-10-10 21:48:31] ...optimizers/trijoint.py.151: total_norm: 6.609535 [S 2022-10-10 21:48:32] ...trap/engines/engine.py.129: train_epoch.epoch: 86 [S 2022-10-10 21:48:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.12994342047454477 [I 2022-10-10 21:48:32] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 21:48:34] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 86 [I 2022-10-10 21:48:34] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 21:48:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 21:48:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 21:48:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 21:48:39] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 21:48:39] ...trap/engines/engine.py.321: eval: epoch 86 | batch 0/510 [I 2022-10-10 21:48:39] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:27 [I 2022-10-10 21:48:39] ...trap/engines/engine.py.326: process: 5.46639 | load: 4.94200 [I 2022-10-10 21:48:43] ...trap/engines/engine.py.321: eval: epoch 86 | batch 10/510 [I 2022-10-10 21:48:43] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:59 [I 2022-10-10 21:48:43] ...trap/engines/engine.py.326: process: 0.41370 | load: 0.00019 [I 2022-10-10 21:48:48] ...trap/engines/engine.py.321: eval: epoch 86 | batch 20/510 [I 2022-10-10 21:48:48] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:03:53 [I 2022-10-10 21:48:48] ...trap/engines/engine.py.326: process: 0.42116 | load: 0.00019 [I 2022-10-10 21:48:52] ...trap/engines/engine.py.321: eval: epoch 86 | batch 30/510 [I 2022-10-10 21:48:52] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:03:29 [I 2022-10-10 21:48:52] ...trap/engines/engine.py.326: process: 0.48111 | load: 0.00021 [I 2022-10-10 21:48:58] ...trap/engines/engine.py.321: eval: epoch 86 | batch 40/510 [I 2022-10-10 21:48:58] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:04:14 [I 2022-10-10 21:48:58] ...trap/engines/engine.py.326: process: 0.50283 | load: 0.00018 [I 2022-10-10 21:49:03] ...trap/engines/engine.py.321: eval: epoch 86 | batch 50/510 [I 2022-10-10 21:49:03] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:45 [I 2022-10-10 21:49:03] ...trap/engines/engine.py.326: process: 0.45965 | load: 0.00019 [I 2022-10-10 21:49:07] ...trap/engines/engine.py.321: eval: epoch 86 | batch 60/510 [I 2022-10-10 21:49:07] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:11 [I 2022-10-10 21:49:07] ...trap/engines/engine.py.326: process: 0.42086 | load: 0.00019 [I 2022-10-10 21:49:11] ...trap/engines/engine.py.321: eval: epoch 86 | batch 70/510 [I 2022-10-10 21:49:11] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:04 [I 2022-10-10 21:49:11] ...trap/engines/engine.py.326: process: 0.43948 | load: 0.00019 [I 2022-10-10 21:49:16] ...trap/engines/engine.py.321: eval: epoch 86 | batch 80/510 [I 2022-10-10 21:49:16] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:21 [I 2022-10-10 21:49:16] ...trap/engines/engine.py.326: process: 0.47847 | load: 0.00019 [I 2022-10-10 21:49:20] ...trap/engines/engine.py.321: eval: epoch 86 | batch 90/510 [I 2022-10-10 21:49:20] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:07 [I 2022-10-10 21:49:20] ...trap/engines/engine.py.326: process: 0.43909 | load: 0.00018 [I 2022-10-10 21:49:24] ...trap/engines/engine.py.321: eval: epoch 86 | batch 100/510 [I 2022-10-10 21:49:24] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:02:48 [I 2022-10-10 21:49:24] ...trap/engines/engine.py.326: process: 0.31766 | load: 0.00020 [I 2022-10-10 21:49:27] ...trap/engines/engine.py.321: eval: epoch 86 | batch 110/510 [I 2022-10-10 21:49:27] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:02:11 [I 2022-10-10 21:49:27] ...trap/engines/engine.py.326: process: 0.32338 | load: 0.00019 [I 2022-10-10 21:49:31] ...trap/engines/engine.py.321: eval: epoch 86 | batch 120/510 [I 2022-10-10 21:49:31] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:05 [I 2022-10-10 21:49:31] ...trap/engines/engine.py.326: process: 0.31871 | load: 0.00018 [I 2022-10-10 21:49:34] ...trap/engines/engine.py.321: eval: epoch 86 | batch 130/510 [I 2022-10-10 21:49:34] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:02 [I 2022-10-10 21:49:34] ...trap/engines/engine.py.326: process: 0.32243 | load: 0.00019 [I 2022-10-10 21:49:37] ...trap/engines/engine.py.321: eval: epoch 86 | batch 140/510 [I 2022-10-10 21:49:37] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:58 [I 2022-10-10 21:49:37] ...trap/engines/engine.py.326: process: 0.31810 | load: 0.00019 [I 2022-10-10 21:49:40] ...trap/engines/engine.py.321: eval: epoch 86 | batch 150/510 [I 2022-10-10 21:49:40] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:55 [I 2022-10-10 21:49:40] ...trap/engines/engine.py.326: process: 0.31256 | load: 0.00019 [I 2022-10-10 21:49:44] ...trap/engines/engine.py.321: eval: epoch 86 | batch 160/510 [I 2022-10-10 21:49:44] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:51 [I 2022-10-10 21:49:44] ...trap/engines/engine.py.326: process: 0.32282 | load: 0.00021 [I 2022-10-10 21:49:47] ...trap/engines/engine.py.321: eval: epoch 86 | batch 170/510 [I 2022-10-10 21:49:47] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:48 [I 2022-10-10 21:49:47] ...trap/engines/engine.py.326: process: 0.32172 | load: 0.00018 [I 2022-10-10 21:49:50] ...trap/engines/engine.py.321: eval: epoch 86 | batch 180/510 [I 2022-10-10 21:49:50] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:45 [I 2022-10-10 21:49:50] ...trap/engines/engine.py.326: process: 0.33147 | load: 0.00022 [I 2022-10-10 21:49:53] ...trap/engines/engine.py.321: eval: epoch 86 | batch 190/510 [I 2022-10-10 21:49:53] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:42 [I 2022-10-10 21:49:53] ...trap/engines/engine.py.326: process: 0.31850 | load: 0.00020 [I 2022-10-10 21:49:56] ...trap/engines/engine.py.321: eval: epoch 86 | batch 200/510 [I 2022-10-10 21:49:56] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:38 [I 2022-10-10 21:49:56] ...trap/engines/engine.py.326: process: 0.30837 | load: 0.00018 [I 2022-10-10 21:50:00] ...trap/engines/engine.py.321: eval: epoch 86 | batch 210/510 [I 2022-10-10 21:50:00] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:35 [I 2022-10-10 21:50:00] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00019 [I 2022-10-10 21:50:03] ...trap/engines/engine.py.321: eval: epoch 86 | batch 220/510 [I 2022-10-10 21:50:03] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:31 [I 2022-10-10 21:50:03] ...trap/engines/engine.py.326: process: 0.31471 | load: 0.00020 [I 2022-10-10 21:50:06] ...trap/engines/engine.py.321: eval: epoch 86 | batch 230/510 [I 2022-10-10 21:50:06] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:30 [I 2022-10-10 21:50:06] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00019 [I 2022-10-10 21:50:09] ...trap/engines/engine.py.321: eval: epoch 86 | batch 240/510 [I 2022-10-10 21:50:09] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:26 [I 2022-10-10 21:50:09] ...trap/engines/engine.py.326: process: 0.32873 | load: 0.00020 [I 2022-10-10 21:50:12] ...trap/engines/engine.py.321: eval: epoch 86 | batch 250/510 [I 2022-10-10 21:50:12] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:23 [I 2022-10-10 21:50:12] ...trap/engines/engine.py.326: process: 0.32014 | load: 0.00017 [I 2022-10-10 21:50:16] ...trap/engines/engine.py.321: eval: epoch 86 | batch 260/510 [I 2022-10-10 21:50:16] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:20 [I 2022-10-10 21:50:16] ...trap/engines/engine.py.326: process: 0.32348 | load: 0.00020 [I 2022-10-10 21:50:19] ...trap/engines/engine.py.321: eval: epoch 86 | batch 270/510 [I 2022-10-10 21:50:19] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:17 [I 2022-10-10 21:50:19] ...trap/engines/engine.py.326: process: 0.32359 | load: 0.00020 [I 2022-10-10 21:50:22] ...trap/engines/engine.py.321: eval: epoch 86 | batch 280/510 [I 2022-10-10 21:50:22] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:13 [I 2022-10-10 21:50:22] ...trap/engines/engine.py.326: process: 0.32333 | load: 0.00021 [I 2022-10-10 21:50:26] ...trap/engines/engine.py.321: eval: epoch 86 | batch 290/510 [I 2022-10-10 21:50:26] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:16 [I 2022-10-10 21:50:26] ...trap/engines/engine.py.326: process: 0.31872 | load: 0.00021 [I 2022-10-10 21:50:29] ...trap/engines/engine.py.321: eval: epoch 86 | batch 300/510 [I 2022-10-10 21:50:29] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:07 [I 2022-10-10 21:50:29] ...trap/engines/engine.py.326: process: 0.31467 | load: 0.00022 [I 2022-10-10 21:50:33] ...trap/engines/engine.py.321: eval: epoch 86 | batch 310/510 [I 2022-10-10 21:50:33] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-10 21:50:33] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00017 [I 2022-10-10 21:50:36] ...trap/engines/engine.py.321: eval: epoch 86 | batch 320/510 [I 2022-10-10 21:50:36] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-10 21:50:36] ...trap/engines/engine.py.326: process: 0.32242 | load: 0.00022 [I 2022-10-10 21:50:39] ...trap/engines/engine.py.321: eval: epoch 86 | batch 330/510 [I 2022-10-10 21:50:39] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-10 21:50:39] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00017 [I 2022-10-10 21:50:42] ...trap/engines/engine.py.321: eval: epoch 86 | batch 340/510 [I 2022-10-10 21:50:42] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:55 [I 2022-10-10 21:50:42] ...trap/engines/engine.py.326: process: 0.34334 | load: 0.00024 [I 2022-10-10 21:50:45] ...trap/engines/engine.py.321: eval: epoch 86 | batch 350/510 [I 2022-10-10 21:50:45] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:51 [I 2022-10-10 21:50:45] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00019 [I 2022-10-10 21:50:49] ...trap/engines/engine.py.321: eval: epoch 86 | batch 360/510 [I 2022-10-10 21:50:49] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:48 [I 2022-10-10 21:50:49] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00020 [I 2022-10-10 21:50:52] ...trap/engines/engine.py.321: eval: epoch 86 | batch 370/510 [I 2022-10-10 21:50:52] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:45 [I 2022-10-10 21:50:52] ...trap/engines/engine.py.326: process: 0.31967 | load: 0.00025 [I 2022-10-10 21:50:55] ...trap/engines/engine.py.321: eval: epoch 86 | batch 380/510 [I 2022-10-10 21:50:55] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:42 [I 2022-10-10 21:50:55] ...trap/engines/engine.py.326: process: 0.32749 | load: 0.00021 [I 2022-10-10 21:50:58] ...trap/engines/engine.py.321: eval: epoch 86 | batch 390/510 [I 2022-10-10 21:50:58] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-10 21:50:58] ...trap/engines/engine.py.326: process: 0.31528 | load: 0.00021 [I 2022-10-10 21:51:02] ...trap/engines/engine.py.321: eval: epoch 86 | batch 400/510 [I 2022-10-10 21:51:02] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:36 [I 2022-10-10 21:51:02] ...trap/engines/engine.py.326: process: 0.32990 | load: 0.00018 [I 2022-10-10 21:51:05] ...trap/engines/engine.py.321: eval: epoch 86 | batch 410/510 [I 2022-10-10 21:51:05] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:32 [I 2022-10-10 21:51:05] ...trap/engines/engine.py.326: process: 0.32396 | load: 0.00021 [I 2022-10-10 21:51:08] ...trap/engines/engine.py.321: eval: epoch 86 | batch 420/510 [I 2022-10-10 21:51:08] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:29 [I 2022-10-10 21:51:08] ...trap/engines/engine.py.326: process: 0.32826 | load: 0.00023 [I 2022-10-10 21:51:12] ...trap/engines/engine.py.321: eval: epoch 86 | batch 430/510 [I 2022-10-10 21:51:12] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-10 21:51:12] ...trap/engines/engine.py.326: process: 0.31839 | load: 0.00021 [I 2022-10-10 21:51:15] ...trap/engines/engine.py.321: eval: epoch 86 | batch 440/510 [I 2022-10-10 21:51:15] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-10 21:51:15] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00023 [I 2022-10-10 21:51:18] ...trap/engines/engine.py.321: eval: epoch 86 | batch 450/510 [I 2022-10-10 21:51:18] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:19 [I 2022-10-10 21:51:18] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00019 [I 2022-10-10 21:51:22] ...trap/engines/engine.py.321: eval: epoch 86 | batch 460/510 [I 2022-10-10 21:51:22] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:21 [I 2022-10-10 21:51:22] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00021 [I 2022-10-10 21:51:26] ...trap/engines/engine.py.321: eval: epoch 86 | batch 470/510 [I 2022-10-10 21:51:26] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:18 [I 2022-10-10 21:51:26] ...trap/engines/engine.py.326: process: 0.83375 | load: 0.51519 [I 2022-10-10 21:51:30] ...trap/engines/engine.py.321: eval: epoch 86 | batch 480/510 [I 2022-10-10 21:51:30] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:10 [I 2022-10-10 21:51:30] ...trap/engines/engine.py.326: process: 0.32587 | load: 0.00022 [I 2022-10-10 21:51:33] ...trap/engines/engine.py.321: eval: epoch 86 | batch 490/510 [I 2022-10-10 21:51:33] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-10 21:51:33] ...trap/engines/engine.py.326: process: 0.32221 | load: 0.00019 [I 2022-10-10 21:51:36] ...trap/engines/engine.py.321: eval: epoch 86 | batch 500/510 [I 2022-10-10 21:51:36] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:03 [I 2022-10-10 21:51:36] ...trap/engines/engine.py.326: process: 0.33635 | load: 0.00020 [I 2022-10-10 21:51:39] ...trap/engines/engine.py.321: eval: epoch 86 | batch 510/510 [I 2022-10-10 21:51:39] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-10 21:51:39] ...trap/engines/engine.py.326: process: 0.31962 | load: 0.00017 [S 2022-10-10 21:51:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 86 [I 2022-10-10 21:51:40] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.744600 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907800 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936300 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.736800 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908500 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935900 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017850 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009304 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009920 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.020702 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008346 [S 2022-10-10 21:52:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007190 [I 2022-10-10 21:52:25] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 21:52:25] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 21:52:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 21:52:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 21:52:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 21:52:30] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 21:52:33] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 21:52:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 21:52:37] ...trap/engines/engine.py.163: Training model on trainset for epoch 87 [I 2022-10-10 21:52:37] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 21:52:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 0/2304 [I 2022-10-10 21:52:40] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:09:14 [I 2022-10-10 21:52:40] ...trap/engines/engine.py.229: process: 3.36582 | load: 2.40759 [I 2022-10-10 21:52:40] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-10 21:52:40] ...optimizers/trijoint.py.151: total_norm: 11.135331 [I 2022-10-10 21:52:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 10/2304 [I 2022-10-10 21:52:49] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:24 [I 2022-10-10 21:52:49] ...trap/engines/engine.py.229: process: 0.87180 | load: 0.00693 [I 2022-10-10 21:52:49] ...trap/engines/engine.py.230: loss: 0.11669 [I 2022-10-10 21:52:49] ...optimizers/trijoint.py.151: total_norm: 26.684204 [I 2022-10-10 21:52:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 20/2304 [I 2022-10-10 21:52:57] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:53 [I 2022-10-10 21:52:57] ...trap/engines/engine.py.229: process: 0.87035 | load: 0.00468 [I 2022-10-10 21:52:57] ...trap/engines/engine.py.230: loss: 0.12611 [I 2022-10-10 21:52:57] ...optimizers/trijoint.py.151: total_norm: 14.035880 [I 2022-10-10 21:53:06] ...trap/engines/engine.py.224: train: epoch 87 | batch 30/2304 [I 2022-10-10 21:53:06] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:01 [I 2022-10-10 21:53:06] ...trap/engines/engine.py.229: process: 0.85857 | load: 0.00527 [I 2022-10-10 21:53:06] ...trap/engines/engine.py.230: loss: 0.08081 [I 2022-10-10 21:53:06] ...optimizers/trijoint.py.151: total_norm: 12.108988 [I 2022-10-10 21:53:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 40/2304 [I 2022-10-10 21:53:15] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:29 [I 2022-10-10 21:53:15] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00414 [I 2022-10-10 21:53:15] ...trap/engines/engine.py.230: loss: 0.13191 [I 2022-10-10 21:53:15] ...optimizers/trijoint.py.151: total_norm: 28.636253 [I 2022-10-10 21:53:23] ...trap/engines/engine.py.224: train: epoch 87 | batch 50/2304 [I 2022-10-10 21:53:23] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:39 [I 2022-10-10 21:53:23] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00394 [I 2022-10-10 21:53:23] ...trap/engines/engine.py.230: loss: 0.08770 [I 2022-10-10 21:53:24] ...optimizers/trijoint.py.151: total_norm: 9.060302 [I 2022-10-10 21:53:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 60/2304 [I 2022-10-10 21:53:32] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:25 [I 2022-10-10 21:53:32] ...trap/engines/engine.py.229: process: 0.86502 | load: 0.00351 [I 2022-10-10 21:53:32] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-10 21:53:32] ...optimizers/trijoint.py.151: total_norm: 15.723015 [I 2022-10-10 21:53:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 70/2304 [I 2022-10-10 21:53:41] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:18 [I 2022-10-10 21:53:41] ...trap/engines/engine.py.229: process: 0.86925 | load: 0.00392 [I 2022-10-10 21:53:41] ...trap/engines/engine.py.230: loss: 0.15138 [I 2022-10-10 21:53:41] ...optimizers/trijoint.py.151: total_norm: 19.564938 [I 2022-10-10 21:53:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 80/2304 [I 2022-10-10 21:53:49] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:55 [I 2022-10-10 21:53:49] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00344 [I 2022-10-10 21:53:49] ...trap/engines/engine.py.230: loss: 0.17071 [I 2022-10-10 21:53:49] ...optimizers/trijoint.py.151: total_norm: 20.504435 [I 2022-10-10 21:53:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 90/2304 [I 2022-10-10 21:53:58] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:54 [I 2022-10-10 21:53:58] ...trap/engines/engine.py.229: process: 0.86673 | load: 0.00367 [I 2022-10-10 21:53:58] ...trap/engines/engine.py.230: loss: 0.11755 [I 2022-10-10 21:53:58] ...optimizers/trijoint.py.151: total_norm: 13.635523 [I 2022-10-10 21:54:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 100/2304 [I 2022-10-10 21:54:07] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:31:54 [I 2022-10-10 21:54:07] ...trap/engines/engine.py.229: process: 0.88672 | load: 0.00323 [I 2022-10-10 21:54:07] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-10 21:54:07] ...optimizers/trijoint.py.151: total_norm: 17.002995 [I 2022-10-10 21:54:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 110/2304 [I 2022-10-10 21:54:16] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:49 [I 2022-10-10 21:54:16] ...trap/engines/engine.py.229: process: 0.86542 | load: 0.00353 [I 2022-10-10 21:54:16] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-10 21:54:16] ...optimizers/trijoint.py.151: total_norm: 24.977013 [I 2022-10-10 21:54:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 120/2304 [I 2022-10-10 21:54:24] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:31 [I 2022-10-10 21:54:24] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00323 [I 2022-10-10 21:54:24] ...trap/engines/engine.py.230: loss: 0.11480 [I 2022-10-10 21:54:24] ...optimizers/trijoint.py.151: total_norm: 31.724974 [I 2022-10-10 21:54:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 130/2304 [I 2022-10-10 21:54:33] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:24 [I 2022-10-10 21:54:33] ...trap/engines/engine.py.229: process: 0.88069 | load: 0.00352 [I 2022-10-10 21:54:33] ...trap/engines/engine.py.230: loss: 0.12420 [I 2022-10-10 21:54:33] ...optimizers/trijoint.py.151: total_norm: 26.695238 [I 2022-10-10 21:54:42] ...trap/engines/engine.py.224: train: epoch 87 | batch 140/2304 [I 2022-10-10 21:54:42] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:04 [I 2022-10-10 21:54:42] ...trap/engines/engine.py.229: process: 0.86993 | load: 0.00311 [I 2022-10-10 21:54:42] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-10 21:54:42] ...optimizers/trijoint.py.151: total_norm: 18.386002 [I 2022-10-10 21:54:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 150/2304 [I 2022-10-10 21:54:50] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:41 [I 2022-10-10 21:54:50] ...trap/engines/engine.py.229: process: 0.86006 | load: 0.00312 [I 2022-10-10 21:54:50] ...trap/engines/engine.py.230: loss: 0.14837 [I 2022-10-10 21:54:50] ...optimizers/trijoint.py.151: total_norm: 23.375118 [I 2022-10-10 21:54:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 160/2304 [I 2022-10-10 21:54:59] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:41 [I 2022-10-10 21:54:59] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00307 [I 2022-10-10 21:54:59] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-10 21:54:59] ...optimizers/trijoint.py.151: total_norm: 14.489818 [I 2022-10-10 21:55:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 170/2304 [I 2022-10-10 21:55:07] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:14 [I 2022-10-10 21:55:07] ...trap/engines/engine.py.229: process: 0.83987 | load: 0.00294 [I 2022-10-10 21:55:07] ...trap/engines/engine.py.230: loss: 0.11933 [I 2022-10-10 21:55:07] ...optimizers/trijoint.py.151: total_norm: 12.195382 [I 2022-10-10 21:55:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 180/2304 [I 2022-10-10 21:55:16] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:13 [I 2022-10-10 21:55:16] ...trap/engines/engine.py.229: process: 0.86385 | load: 0.00294 [I 2022-10-10 21:55:16] ...trap/engines/engine.py.230: loss: 0.08826 [I 2022-10-10 21:55:16] ...optimizers/trijoint.py.151: total_norm: 10.322203 [I 2022-10-10 21:55:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 190/2304 [I 2022-10-10 21:55:24] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:29:54 [I 2022-10-10 21:55:24] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00295 [I 2022-10-10 21:55:24] ...trap/engines/engine.py.230: loss: 0.09082 [I 2022-10-10 21:55:24] ...optimizers/trijoint.py.151: total_norm: 17.871017 [I 2022-10-10 21:55:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 200/2304 [I 2022-10-10 21:55:33] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:29:57 [I 2022-10-10 21:55:33] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00309 [I 2022-10-10 21:55:33] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-10 21:55:33] ...optimizers/trijoint.py.151: total_norm: 24.415211 [I 2022-10-10 21:55:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 210/2304 [I 2022-10-10 21:55:41] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:50 [I 2022-10-10 21:55:41] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00316 [I 2022-10-10 21:55:41] ...trap/engines/engine.py.230: loss: 0.13355 [I 2022-10-10 21:55:41] ...optimizers/trijoint.py.151: total_norm: 18.860994 [I 2022-10-10 21:55:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 220/2304 [I 2022-10-10 21:55:50] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:29 [I 2022-10-10 21:55:50] ...trap/engines/engine.py.229: process: 0.83787 | load: 0.00301 [I 2022-10-10 21:55:50] ...trap/engines/engine.py.230: loss: 0.10031 [I 2022-10-10 21:55:50] ...optimizers/trijoint.py.151: total_norm: 7.033158 [I 2022-10-10 21:55:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 230/2304 [I 2022-10-10 21:55:59] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:31 [I 2022-10-10 21:55:59] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00296 [I 2022-10-10 21:55:59] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-10 21:55:59] ...optimizers/trijoint.py.151: total_norm: 19.523952 [I 2022-10-10 21:56:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 240/2304 [I 2022-10-10 21:56:07] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:20 [I 2022-10-10 21:56:07] ...trap/engines/engine.py.229: process: 0.85781 | load: 0.00320 [I 2022-10-10 21:56:07] ...trap/engines/engine.py.230: loss: 0.15182 [I 2022-10-10 21:56:07] ...optimizers/trijoint.py.151: total_norm: 21.303391 [I 2022-10-10 21:56:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 250/2304 [I 2022-10-10 21:56:16] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:12 [I 2022-10-10 21:56:16] ...trap/engines/engine.py.229: process: 0.85669 | load: 0.00311 [I 2022-10-10 21:56:16] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-10 21:56:16] ...optimizers/trijoint.py.151: total_norm: 12.024401 [I 2022-10-10 21:56:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 260/2304 [I 2022-10-10 21:56:24] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:01 [I 2022-10-10 21:56:24] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00303 [I 2022-10-10 21:56:24] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-10 21:56:24] ...optimizers/trijoint.py.151: total_norm: 8.938293 [I 2022-10-10 21:56:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 270/2304 [I 2022-10-10 21:56:33] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:28:52 [I 2022-10-10 21:56:33] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00303 [I 2022-10-10 21:56:33] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-10 21:56:33] ...optimizers/trijoint.py.151: total_norm: 13.503697 [I 2022-10-10 21:56:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 280/2304 [I 2022-10-10 21:56:41] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:43 [I 2022-10-10 21:56:41] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00295 [I 2022-10-10 21:56:41] ...trap/engines/engine.py.230: loss: 0.07978 [I 2022-10-10 21:56:41] ...optimizers/trijoint.py.151: total_norm: 10.395854 [I 2022-10-10 21:56:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 290/2304 [I 2022-10-10 21:56:50] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:42 [I 2022-10-10 21:56:50] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00302 [I 2022-10-10 21:56:50] ...trap/engines/engine.py.230: loss: 0.09644 [I 2022-10-10 21:56:50] ...optimizers/trijoint.py.151: total_norm: 9.653434 [I 2022-10-10 21:56:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 300/2304 [I 2022-10-10 21:56:58] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:29 [I 2022-10-10 21:56:58] ...trap/engines/engine.py.229: process: 0.84982 | load: 0.00298 [I 2022-10-10 21:56:58] ...trap/engines/engine.py.230: loss: 0.13442 [I 2022-10-10 21:56:58] ...optimizers/trijoint.py.151: total_norm: 16.076441 [I 2022-10-10 21:57:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 310/2304 [I 2022-10-10 21:57:07] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:13 [I 2022-10-10 21:57:07] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00290 [I 2022-10-10 21:57:07] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-10 21:57:07] ...optimizers/trijoint.py.151: total_norm: 26.639051 [I 2022-10-10 21:57:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 320/2304 [I 2022-10-10 21:57:15] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:12 [I 2022-10-10 21:57:15] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00292 [I 2022-10-10 21:57:15] ...trap/engines/engine.py.230: loss: 0.08525 [I 2022-10-10 21:57:16] ...optimizers/trijoint.py.151: total_norm: 14.058285 [I 2022-10-10 21:57:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 330/2304 [I 2022-10-10 21:57:24] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:27:56 [I 2022-10-10 21:57:24] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00293 [I 2022-10-10 21:57:24] ...trap/engines/engine.py.230: loss: 0.08908 [I 2022-10-10 21:57:24] ...optimizers/trijoint.py.151: total_norm: 14.832426 [I 2022-10-10 21:57:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 340/2304 [I 2022-10-10 21:57:33] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:11 [I 2022-10-10 21:57:33] ...trap/engines/engine.py.229: process: 0.85806 | load: 0.00291 [I 2022-10-10 21:57:33] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-10 21:57:33] ...optimizers/trijoint.py.151: total_norm: 22.350122 [I 2022-10-10 21:57:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 350/2304 [I 2022-10-10 21:57:41] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:42 [I 2022-10-10 21:57:41] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00305 [I 2022-10-10 21:57:41] ...trap/engines/engine.py.230: loss: 0.10705 [I 2022-10-10 21:57:41] ...optimizers/trijoint.py.151: total_norm: 12.184732 [I 2022-10-10 21:57:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 360/2304 [I 2022-10-10 21:57:50] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:36 [I 2022-10-10 21:57:50] ...trap/engines/engine.py.229: process: 0.84679 | load: 0.00297 [I 2022-10-10 21:57:50] ...trap/engines/engine.py.230: loss: 0.14558 [I 2022-10-10 21:57:50] ...optimizers/trijoint.py.151: total_norm: 28.649387 [I 2022-10-10 21:57:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 370/2304 [I 2022-10-10 21:57:58] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:31 [I 2022-10-10 21:57:58] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00296 [I 2022-10-10 21:57:58] ...trap/engines/engine.py.230: loss: 0.11407 [I 2022-10-10 21:57:58] ...optimizers/trijoint.py.151: total_norm: 20.268711 [I 2022-10-10 21:58:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 380/2304 [I 2022-10-10 21:58:07] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:19 [I 2022-10-10 21:58:07] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00296 [I 2022-10-10 21:58:07] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-10 21:58:07] ...optimizers/trijoint.py.151: total_norm: 26.122732 [I 2022-10-10 21:58:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 390/2304 [I 2022-10-10 21:58:15] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:08 [I 2022-10-10 21:58:15] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00316 [I 2022-10-10 21:58:15] ...trap/engines/engine.py.230: loss: 0.10741 [I 2022-10-10 21:58:15] ...optimizers/trijoint.py.151: total_norm: 14.665395 [I 2022-10-10 21:58:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 400/2304 [I 2022-10-10 21:58:24] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:08 [I 2022-10-10 21:58:24] ...trap/engines/engine.py.229: process: 0.85661 | load: 0.00299 [I 2022-10-10 21:58:24] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-10 21:58:24] ...optimizers/trijoint.py.151: total_norm: 13.714561 [I 2022-10-10 21:58:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 410/2304 [I 2022-10-10 21:58:32] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:26:56 [I 2022-10-10 21:58:32] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00294 [I 2022-10-10 21:58:32] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-10 21:58:32] ...optimizers/trijoint.py.151: total_norm: 8.362977 [I 2022-10-10 21:58:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 420/2304 [I 2022-10-10 21:58:41] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:26:47 [I 2022-10-10 21:58:41] ...trap/engines/engine.py.229: process: 0.85040 | load: 0.00294 [I 2022-10-10 21:58:41] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-10 21:58:41] ...optimizers/trijoint.py.151: total_norm: 13.104955 [I 2022-10-10 21:58:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 430/2304 [I 2022-10-10 21:58:50] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:26:35 [I 2022-10-10 21:58:50] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00317 [I 2022-10-10 21:58:50] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-10 21:58:50] ...optimizers/trijoint.py.151: total_norm: 13.728763 [I 2022-10-10 21:58:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 440/2304 [I 2022-10-10 21:58:58] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:38 [I 2022-10-10 21:58:58] ...trap/engines/engine.py.229: process: 0.86122 | load: 0.00312 [I 2022-10-10 21:58:58] ...trap/engines/engine.py.230: loss: 0.22556 [I 2022-10-10 21:58:58] ...optimizers/trijoint.py.151: total_norm: 23.254002 [I 2022-10-10 21:59:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 450/2304 [I 2022-10-10 21:59:07] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:24 [I 2022-10-10 21:59:07] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00304 [I 2022-10-10 21:59:07] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-10 21:59:07] ...optimizers/trijoint.py.151: total_norm: 17.176954 [I 2022-10-10 21:59:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 460/2304 [I 2022-10-10 21:59:15] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:26:17 [I 2022-10-10 21:59:15] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00301 [I 2022-10-10 21:59:15] ...trap/engines/engine.py.230: loss: 0.18971 [I 2022-10-10 21:59:15] ...optimizers/trijoint.py.151: total_norm: 17.450886 [I 2022-10-10 21:59:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 470/2304 [I 2022-10-10 21:59:24] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:23 [I 2022-10-10 21:59:24] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00309 [I 2022-10-10 21:59:24] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-10 21:59:24] ...optimizers/trijoint.py.151: total_norm: 34.126804 [I 2022-10-10 21:59:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 480/2304 [I 2022-10-10 21:59:33] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:26:08 [I 2022-10-10 21:59:33] ...trap/engines/engine.py.229: process: 0.84925 | load: 0.00315 [I 2022-10-10 21:59:33] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-10 21:59:33] ...optimizers/trijoint.py.151: total_norm: 12.602035 [I 2022-10-10 21:59:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 490/2304 [I 2022-10-10 21:59:41] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:26:15 [I 2022-10-10 21:59:41] ...trap/engines/engine.py.229: process: 0.87801 | load: 0.00310 [I 2022-10-10 21:59:41] ...trap/engines/engine.py.230: loss: 0.12261 [I 2022-10-10 21:59:41] ...optimizers/trijoint.py.151: total_norm: 12.716061 [I 2022-10-10 21:59:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 500/2304 [I 2022-10-10 21:59:50] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:54 [I 2022-10-10 21:59:50] ...trap/engines/engine.py.229: process: 0.87454 | load: 0.00309 [I 2022-10-10 21:59:50] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-10 21:59:50] ...optimizers/trijoint.py.151: total_norm: 14.339335 [I 2022-10-10 21:59:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 510/2304 [I 2022-10-10 21:59:59] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:49 [I 2022-10-10 21:59:59] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00309 [I 2022-10-10 21:59:59] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-10 21:59:59] ...optimizers/trijoint.py.151: total_norm: 10.646609 [I 2022-10-10 22:00:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 520/2304 [I 2022-10-10 22:00:07] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:25:42 [I 2022-10-10 22:00:07] ...trap/engines/engine.py.229: process: 0.86556 | load: 0.00317 [I 2022-10-10 22:00:07] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-10 22:00:07] ...optimizers/trijoint.py.151: total_norm: 19.880230 [I 2022-10-10 22:00:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 530/2304 [I 2022-10-10 22:00:16] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:36 [I 2022-10-10 22:00:16] ...trap/engines/engine.py.229: process: 0.86557 | load: 0.00307 [I 2022-10-10 22:00:16] ...trap/engines/engine.py.230: loss: 0.09530 [I 2022-10-10 22:00:16] ...optimizers/trijoint.py.151: total_norm: 9.152215 [I 2022-10-10 22:00:25] ...trap/engines/engine.py.224: train: epoch 87 | batch 540/2304 [I 2022-10-10 22:00:25] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:25:27 [I 2022-10-10 22:00:25] ...trap/engines/engine.py.229: process: 0.87266 | load: 0.00308 [I 2022-10-10 22:00:25] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-10 22:00:25] ...optimizers/trijoint.py.151: total_norm: 9.920205 [I 2022-10-10 22:00:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 550/2304 [I 2022-10-10 22:00:33] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:25:25 [I 2022-10-10 22:00:33] ...trap/engines/engine.py.229: process: 0.87965 | load: 0.00309 [I 2022-10-10 22:00:33] ...trap/engines/engine.py.230: loss: 0.12667 [I 2022-10-10 22:00:33] ...optimizers/trijoint.py.151: total_norm: 21.952139 [I 2022-10-10 22:00:42] ...trap/engines/engine.py.224: train: epoch 87 | batch 560/2304 [I 2022-10-10 22:00:42] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:16 [I 2022-10-10 22:00:42] ...trap/engines/engine.py.229: process: 0.87967 | load: 0.00314 [I 2022-10-10 22:00:42] ...trap/engines/engine.py.230: loss: 0.15571 [I 2022-10-10 22:00:42] ...optimizers/trijoint.py.151: total_norm: 13.387199 [I 2022-10-10 22:00:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 570/2304 [I 2022-10-10 22:00:51] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:58 [I 2022-10-10 22:00:51] ...trap/engines/engine.py.229: process: 0.86130 | load: 0.00313 [I 2022-10-10 22:00:51] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-10 22:00:51] ...optimizers/trijoint.py.151: total_norm: 28.965855 [I 2022-10-10 22:00:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 580/2304 [I 2022-10-10 22:00:59] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:56 [I 2022-10-10 22:00:59] ...trap/engines/engine.py.229: process: 0.88601 | load: 0.00313 [I 2022-10-10 22:00:59] ...trap/engines/engine.py.230: loss: 0.15241 [I 2022-10-10 22:00:59] ...optimizers/trijoint.py.151: total_norm: 48.107933 [I 2022-10-10 22:01:08] ...trap/engines/engine.py.224: train: epoch 87 | batch 590/2304 [I 2022-10-10 22:01:08] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:35 [I 2022-10-10 22:01:08] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00310 [I 2022-10-10 22:01:08] ...trap/engines/engine.py.230: loss: 0.09453 [I 2022-10-10 22:01:08] ...optimizers/trijoint.py.151: total_norm: 8.252142 [I 2022-10-10 22:01:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 600/2304 [I 2022-10-10 22:01:17] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:39 [I 2022-10-10 22:01:17] ...trap/engines/engine.py.229: process: 0.88193 | load: 0.00315 [I 2022-10-10 22:01:17] ...trap/engines/engine.py.230: loss: 0.13475 [I 2022-10-10 22:01:17] ...optimizers/trijoint.py.151: total_norm: 21.553476 [I 2022-10-10 22:01:25] ...trap/engines/engine.py.224: train: epoch 87 | batch 610/2304 [I 2022-10-10 22:01:25] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:27 [I 2022-10-10 22:01:25] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00314 [I 2022-10-10 22:01:25] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-10 22:01:26] ...optimizers/trijoint.py.151: total_norm: 9.461390 [I 2022-10-10 22:01:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 620/2304 [I 2022-10-10 22:01:34] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:24:21 [I 2022-10-10 22:01:34] ...trap/engines/engine.py.229: process: 0.86714 | load: 0.00326 [I 2022-10-10 22:01:34] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-10 22:01:34] ...optimizers/trijoint.py.151: total_norm: 10.115353 [I 2022-10-10 22:01:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 630/2304 [I 2022-10-10 22:01:43] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:24:30 [I 2022-10-10 22:01:43] ...trap/engines/engine.py.229: process: 0.88317 | load: 0.00321 [I 2022-10-10 22:01:43] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-10 22:01:43] ...optimizers/trijoint.py.151: total_norm: 14.648104 [I 2022-10-10 22:01:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 640/2304 [I 2022-10-10 22:01:52] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:24:02 [I 2022-10-10 22:01:52] ...trap/engines/engine.py.229: process: 0.85558 | load: 0.00320 [I 2022-10-10 22:01:52] ...trap/engines/engine.py.230: loss: 0.08397 [I 2022-10-10 22:01:52] ...optimizers/trijoint.py.151: total_norm: 12.608894 [I 2022-10-10 22:02:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 650/2304 [I 2022-10-10 22:02:00] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:57 [I 2022-10-10 22:02:00] ...trap/engines/engine.py.229: process: 0.88409 | load: 0.00318 [I 2022-10-10 22:02:00] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-10 22:02:00] ...optimizers/trijoint.py.151: total_norm: 11.309372 [I 2022-10-10 22:02:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 660/2304 [I 2022-10-10 22:02:09] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:44 [I 2022-10-10 22:02:09] ...trap/engines/engine.py.229: process: 0.87192 | load: 0.00324 [I 2022-10-10 22:02:09] ...trap/engines/engine.py.230: loss: 0.11414 [I 2022-10-10 22:02:09] ...optimizers/trijoint.py.151: total_norm: 16.723803 [I 2022-10-10 22:02:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 670/2304 [I 2022-10-10 22:02:18] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:25 [I 2022-10-10 22:02:18] ...trap/engines/engine.py.229: process: 0.86244 | load: 0.00325 [I 2022-10-10 22:02:18] ...trap/engines/engine.py.230: loss: 0.15340 [I 2022-10-10 22:02:18] ...optimizers/trijoint.py.151: total_norm: 18.589636 [I 2022-10-10 22:02:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 680/2304 [I 2022-10-10 22:02:26] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:23 [I 2022-10-10 22:02:26] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00310 [I 2022-10-10 22:02:26] ...trap/engines/engine.py.230: loss: 0.08606 [I 2022-10-10 22:02:26] ...optimizers/trijoint.py.151: total_norm: 10.467983 [I 2022-10-10 22:02:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 690/2304 [I 2022-10-10 22:02:35] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:20 [I 2022-10-10 22:02:35] ...trap/engines/engine.py.229: process: 0.87289 | load: 0.00320 [I 2022-10-10 22:02:35] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-10 22:02:35] ...optimizers/trijoint.py.151: total_norm: 32.798897 [I 2022-10-10 22:02:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 700/2304 [I 2022-10-10 22:02:44] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:23:02 [I 2022-10-10 22:02:44] ...trap/engines/engine.py.229: process: 0.85710 | load: 0.00308 [I 2022-10-10 22:02:44] ...trap/engines/engine.py.230: loss: 0.15096 [I 2022-10-10 22:02:44] ...optimizers/trijoint.py.151: total_norm: 16.740627 [I 2022-10-10 22:02:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 710/2304 [I 2022-10-10 22:02:52] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:57 [I 2022-10-10 22:02:52] ...trap/engines/engine.py.229: process: 0.86058 | load: 0.00308 [I 2022-10-10 22:02:52] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-10 22:02:52] ...optimizers/trijoint.py.151: total_norm: 13.358697 [I 2022-10-10 22:03:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 720/2304 [I 2022-10-10 22:03:01] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:48 [I 2022-10-10 22:03:01] ...trap/engines/engine.py.229: process: 0.87221 | load: 0.00311 [I 2022-10-10 22:03:01] ...trap/engines/engine.py.230: loss: 0.11962 [I 2022-10-10 22:03:01] ...optimizers/trijoint.py.151: total_norm: 30.422178 [I 2022-10-10 22:03:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 730/2304 [I 2022-10-10 22:03:10] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:44 [I 2022-10-10 22:03:10] ...trap/engines/engine.py.229: process: 0.87739 | load: 0.00314 [I 2022-10-10 22:03:10] ...trap/engines/engine.py.230: loss: 0.10713 [I 2022-10-10 22:03:10] ...optimizers/trijoint.py.151: total_norm: 12.511021 [I 2022-10-10 22:03:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 740/2304 [I 2022-10-10 22:03:18] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:26 [I 2022-10-10 22:03:18] ...trap/engines/engine.py.229: process: 0.84926 | load: 0.00295 [I 2022-10-10 22:03:18] ...trap/engines/engine.py.230: loss: 0.18922 [I 2022-10-10 22:03:18] ...optimizers/trijoint.py.151: total_norm: 35.509468 [I 2022-10-10 22:03:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 750/2304 [I 2022-10-10 22:03:27] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:14 [I 2022-10-10 22:03:27] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00304 [I 2022-10-10 22:03:27] ...trap/engines/engine.py.230: loss: 0.09594 [I 2022-10-10 22:03:27] ...optimizers/trijoint.py.151: total_norm: 7.581624 [I 2022-10-10 22:03:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 760/2304 [I 2022-10-10 22:03:35] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:00 [I 2022-10-10 22:03:35] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00293 [I 2022-10-10 22:03:35] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-10 22:03:35] ...optimizers/trijoint.py.151: total_norm: 13.741544 [I 2022-10-10 22:03:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 770/2304 [I 2022-10-10 22:03:44] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:48 [I 2022-10-10 22:03:44] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00296 [I 2022-10-10 22:03:44] ...trap/engines/engine.py.230: loss: 0.13592 [I 2022-10-10 22:03:44] ...optimizers/trijoint.py.151: total_norm: 28.827303 [I 2022-10-10 22:03:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 780/2304 [I 2022-10-10 22:03:53] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:38 [I 2022-10-10 22:03:53] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00298 [I 2022-10-10 22:03:53] ...trap/engines/engine.py.230: loss: 0.09523 [I 2022-10-10 22:03:53] ...optimizers/trijoint.py.151: total_norm: 9.446356 [I 2022-10-10 22:04:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 790/2304 [I 2022-10-10 22:04:01] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:35 [I 2022-10-10 22:04:01] ...trap/engines/engine.py.229: process: 0.85143 | load: 0.00293 [I 2022-10-10 22:04:01] ...trap/engines/engine.py.230: loss: 0.18670 [I 2022-10-10 22:04:01] ...optimizers/trijoint.py.151: total_norm: 19.554024 [I 2022-10-10 22:04:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 800/2304 [I 2022-10-10 22:04:10] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:26 [I 2022-10-10 22:04:10] ...trap/engines/engine.py.229: process: 0.86733 | load: 0.00294 [I 2022-10-10 22:04:10] ...trap/engines/engine.py.230: loss: 0.10751 [I 2022-10-10 22:04:10] ...optimizers/trijoint.py.151: total_norm: 9.089375 [I 2022-10-10 22:04:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 810/2304 [I 2022-10-10 22:04:18] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:12 [I 2022-10-10 22:04:18] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00295 [I 2022-10-10 22:04:18] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-10 22:04:18] ...optimizers/trijoint.py.151: total_norm: 15.761068 [I 2022-10-10 22:04:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 820/2304 [I 2022-10-10 22:04:27] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:21:10 [I 2022-10-10 22:04:27] ...trap/engines/engine.py.229: process: 0.86409 | load: 0.00295 [I 2022-10-10 22:04:27] ...trap/engines/engine.py.230: loss: 0.09056 [I 2022-10-10 22:04:27] ...optimizers/trijoint.py.151: total_norm: 17.889305 [I 2022-10-10 22:04:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 830/2304 [I 2022-10-10 22:04:35] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:57 [I 2022-10-10 22:04:35] ...trap/engines/engine.py.229: process: 0.85709 | load: 0.00299 [I 2022-10-10 22:04:35] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-10 22:04:35] ...optimizers/trijoint.py.151: total_norm: 9.823112 [I 2022-10-10 22:04:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 840/2304 [I 2022-10-10 22:04:44] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:36 [I 2022-10-10 22:04:44] ...trap/engines/engine.py.229: process: 0.82805 | load: 0.00302 [I 2022-10-10 22:04:44] ...trap/engines/engine.py.230: loss: 0.18118 [I 2022-10-10 22:04:44] ...optimizers/trijoint.py.151: total_norm: 8.928224 [I 2022-10-10 22:04:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 850/2304 [I 2022-10-10 22:04:52] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:38 [I 2022-10-10 22:04:52] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00297 [I 2022-10-10 22:04:52] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-10 22:04:52] ...optimizers/trijoint.py.151: total_norm: 14.247897 [I 2022-10-10 22:05:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 860/2304 [I 2022-10-10 22:05:01] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:33 [I 2022-10-10 22:05:01] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00296 [I 2022-10-10 22:05:01] ...trap/engines/engine.py.230: loss: 0.13153 [I 2022-10-10 22:05:01] ...optimizers/trijoint.py.151: total_norm: 14.452968 [I 2022-10-10 22:05:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 870/2304 [I 2022-10-10 22:05:09] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:18 [I 2022-10-10 22:05:09] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00293 [I 2022-10-10 22:05:09] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-10 22:05:09] ...optimizers/trijoint.py.151: total_norm: 7.810890 [I 2022-10-10 22:05:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 880/2304 [I 2022-10-10 22:05:18] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:09 [I 2022-10-10 22:05:18] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00295 [I 2022-10-10 22:05:18] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-10 22:05:18] ...optimizers/trijoint.py.151: total_norm: 15.737761 [I 2022-10-10 22:05:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 890/2304 [I 2022-10-10 22:05:27] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:08 [I 2022-10-10 22:05:27] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00296 [I 2022-10-10 22:05:27] ...trap/engines/engine.py.230: loss: 0.15195 [I 2022-10-10 22:05:27] ...optimizers/trijoint.py.151: total_norm: 17.601273 [I 2022-10-10 22:05:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 900/2304 [I 2022-10-10 22:05:35] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:58 [I 2022-10-10 22:05:35] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00291 [I 2022-10-10 22:05:35] ...trap/engines/engine.py.230: loss: 0.08507 [I 2022-10-10 22:05:35] ...optimizers/trijoint.py.151: total_norm: 11.245261 [I 2022-10-10 22:05:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 910/2304 [I 2022-10-10 22:05:44] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:41 [I 2022-10-10 22:05:44] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00294 [I 2022-10-10 22:05:44] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-10 22:05:44] ...optimizers/trijoint.py.151: total_norm: 9.126612 [I 2022-10-10 22:05:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 920/2304 [I 2022-10-10 22:05:52] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:38 [I 2022-10-10 22:05:52] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00300 [I 2022-10-10 22:05:52] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-10 22:05:52] ...optimizers/trijoint.py.151: total_norm: 10.641733 [I 2022-10-10 22:06:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 930/2304 [I 2022-10-10 22:06:01] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:30 [I 2022-10-10 22:06:01] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00304 [I 2022-10-10 22:06:01] ...trap/engines/engine.py.230: loss: 0.13550 [I 2022-10-10 22:06:01] ...optimizers/trijoint.py.151: total_norm: 21.602055 [I 2022-10-10 22:06:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 940/2304 [I 2022-10-10 22:06:09] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:25 [I 2022-10-10 22:06:09] ...trap/engines/engine.py.229: process: 0.86382 | load: 0.00293 [I 2022-10-10 22:06:09] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-10 22:06:09] ...optimizers/trijoint.py.151: total_norm: 12.099477 [I 2022-10-10 22:06:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 950/2304 [I 2022-10-10 22:06:18] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:16 [I 2022-10-10 22:06:18] ...trap/engines/engine.py.229: process: 0.85779 | load: 0.00296 [I 2022-10-10 22:06:18] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-10 22:06:18] ...optimizers/trijoint.py.151: total_norm: 12.156137 [I 2022-10-10 22:06:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 960/2304 [I 2022-10-10 22:06:26] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:58 [I 2022-10-10 22:06:26] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00291 [I 2022-10-10 22:06:26] ...trap/engines/engine.py.230: loss: 0.09765 [I 2022-10-10 22:06:26] ...optimizers/trijoint.py.151: total_norm: 17.348232 [I 2022-10-10 22:06:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 970/2304 [I 2022-10-10 22:06:35] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:02 [I 2022-10-10 22:06:35] ...trap/engines/engine.py.229: process: 0.86309 | load: 0.00294 [I 2022-10-10 22:06:35] ...trap/engines/engine.py.230: loss: 0.13872 [I 2022-10-10 22:06:35] ...optimizers/trijoint.py.151: total_norm: 19.019764 [I 2022-10-10 22:06:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 980/2304 [I 2022-10-10 22:06:43] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:53 [I 2022-10-10 22:06:43] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00296 [I 2022-10-10 22:06:43] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-10 22:06:43] ...optimizers/trijoint.py.151: total_norm: 9.731971 [I 2022-10-10 22:06:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 990/2304 [I 2022-10-10 22:06:52] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:50 [I 2022-10-10 22:06:52] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00295 [I 2022-10-10 22:06:52] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-10 22:06:52] ...optimizers/trijoint.py.151: total_norm: 8.375372 [I 2022-10-10 22:07:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 1000/2304 [I 2022-10-10 22:07:00] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:26 [I 2022-10-10 22:07:00] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00294 [I 2022-10-10 22:07:00] ...trap/engines/engine.py.230: loss: 0.09001 [I 2022-10-10 22:07:00] ...optimizers/trijoint.py.151: total_norm: 12.226542 [I 2022-10-10 22:07:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1010/2304 [I 2022-10-10 22:07:09] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:20 [I 2022-10-10 22:07:09] ...trap/engines/engine.py.229: process: 0.84828 | load: 0.00297 [I 2022-10-10 22:07:09] ...trap/engines/engine.py.230: loss: 0.13344 [I 2022-10-10 22:07:09] ...optimizers/trijoint.py.151: total_norm: 13.330798 [I 2022-10-10 22:07:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1020/2304 [I 2022-10-10 22:07:18] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:11 [I 2022-10-10 22:07:18] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00307 [I 2022-10-10 22:07:18] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-10 22:07:18] ...optimizers/trijoint.py.151: total_norm: 14.169333 [I 2022-10-10 22:07:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 1030/2304 [I 2022-10-10 22:07:26] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:18:06 [I 2022-10-10 22:07:26] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00293 [I 2022-10-10 22:07:26] ...trap/engines/engine.py.230: loss: 0.13336 [I 2022-10-10 22:07:26] ...optimizers/trijoint.py.151: total_norm: 12.103654 [I 2022-10-10 22:07:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1040/2304 [I 2022-10-10 22:07:35] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:53 [I 2022-10-10 22:07:35] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00298 [I 2022-10-10 22:07:35] ...trap/engines/engine.py.230: loss: 0.09130 [I 2022-10-10 22:07:35] ...optimizers/trijoint.py.151: total_norm: 12.944828 [I 2022-10-10 22:07:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 1050/2304 [I 2022-10-10 22:07:43] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:53 [I 2022-10-10 22:07:43] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00291 [I 2022-10-10 22:07:43] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-10 22:07:43] ...optimizers/trijoint.py.151: total_norm: 11.630410 [I 2022-10-10 22:07:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 1060/2304 [I 2022-10-10 22:07:52] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:40 [I 2022-10-10 22:07:52] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00298 [I 2022-10-10 22:07:52] ...trap/engines/engine.py.230: loss: 0.11582 [I 2022-10-10 22:07:52] ...optimizers/trijoint.py.151: total_norm: 15.349573 [I 2022-10-10 22:08:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 1070/2304 [I 2022-10-10 22:08:00] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:38 [I 2022-10-10 22:08:00] ...trap/engines/engine.py.229: process: 0.86741 | load: 0.00295 [I 2022-10-10 22:08:00] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-10 22:08:00] ...optimizers/trijoint.py.151: total_norm: 11.309770 [I 2022-10-10 22:08:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1080/2304 [I 2022-10-10 22:08:09] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:22 [I 2022-10-10 22:08:09] ...trap/engines/engine.py.229: process: 0.83219 | load: 0.00304 [I 2022-10-10 22:08:09] ...trap/engines/engine.py.230: loss: 0.15930 [I 2022-10-10 22:08:09] ...optimizers/trijoint.py.151: total_norm: 19.124969 [I 2022-10-10 22:08:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 1090/2304 [I 2022-10-10 22:08:17] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:15 [I 2022-10-10 22:08:17] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00293 [I 2022-10-10 22:08:17] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-10 22:08:17] ...optimizers/trijoint.py.151: total_norm: 17.724520 [I 2022-10-10 22:08:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 1100/2304 [I 2022-10-10 22:08:26] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:04 [I 2022-10-10 22:08:26] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00293 [I 2022-10-10 22:08:26] ...trap/engines/engine.py.230: loss: 0.10948 [I 2022-10-10 22:08:26] ...optimizers/trijoint.py.151: total_norm: 18.001442 [I 2022-10-10 22:08:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 1110/2304 [I 2022-10-10 22:08:34] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:53 [I 2022-10-10 22:08:34] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00296 [I 2022-10-10 22:08:34] ...trap/engines/engine.py.230: loss: 0.09502 [I 2022-10-10 22:08:34] ...optimizers/trijoint.py.151: total_norm: 7.762019 [I 2022-10-10 22:08:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 1120/2304 [I 2022-10-10 22:08:43] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:52 [I 2022-10-10 22:08:43] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00297 [I 2022-10-10 22:08:43] ...trap/engines/engine.py.230: loss: 0.10928 [I 2022-10-10 22:08:43] ...optimizers/trijoint.py.151: total_norm: 16.847744 [I 2022-10-10 22:08:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 1130/2304 [I 2022-10-10 22:08:52] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:43 [I 2022-10-10 22:08:52] ...trap/engines/engine.py.229: process: 0.85986 | load: 0.00297 [I 2022-10-10 22:08:52] ...trap/engines/engine.py.230: loss: 0.10951 [I 2022-10-10 22:08:52] ...optimizers/trijoint.py.151: total_norm: 15.932645 [I 2022-10-10 22:09:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 1140/2304 [I 2022-10-10 22:09:00] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:33 [I 2022-10-10 22:09:00] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00296 [I 2022-10-10 22:09:00] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-10 22:09:00] ...optimizers/trijoint.py.151: total_norm: 7.910551 [I 2022-10-10 22:09:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1150/2304 [I 2022-10-10 22:09:09] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:21 [I 2022-10-10 22:09:09] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00298 [I 2022-10-10 22:09:09] ...trap/engines/engine.py.230: loss: 0.13230 [I 2022-10-10 22:09:09] ...optimizers/trijoint.py.151: total_norm: 34.387138 [I 2022-10-10 22:09:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 1160/2304 [I 2022-10-10 22:09:17] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:12 [I 2022-10-10 22:09:17] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00297 [I 2022-10-10 22:09:17] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-10 22:09:17] ...optimizers/trijoint.py.151: total_norm: 12.445990 [I 2022-10-10 22:09:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 1170/2304 [I 2022-10-10 22:09:26] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:57 [I 2022-10-10 22:09:26] ...trap/engines/engine.py.229: process: 0.83440 | load: 0.00298 [I 2022-10-10 22:09:26] ...trap/engines/engine.py.230: loss: 0.15343 [I 2022-10-10 22:09:26] ...optimizers/trijoint.py.151: total_norm: 14.779866 [I 2022-10-10 22:09:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 1180/2304 [I 2022-10-10 22:09:34] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:54 [I 2022-10-10 22:09:34] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00297 [I 2022-10-10 22:09:34] ...trap/engines/engine.py.230: loss: 0.12259 [I 2022-10-10 22:09:34] ...optimizers/trijoint.py.151: total_norm: 24.441710 [I 2022-10-10 22:09:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 1190/2304 [I 2022-10-10 22:09:43] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:49 [I 2022-10-10 22:09:43] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00300 [I 2022-10-10 22:09:43] ...trap/engines/engine.py.230: loss: 0.15845 [I 2022-10-10 22:09:43] ...optimizers/trijoint.py.151: total_norm: 29.285011 [I 2022-10-10 22:09:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 1200/2304 [I 2022-10-10 22:09:51] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:40 [I 2022-10-10 22:09:51] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00293 [I 2022-10-10 22:09:51] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-10 22:09:51] ...optimizers/trijoint.py.151: total_norm: 22.388521 [I 2022-10-10 22:10:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 1210/2304 [I 2022-10-10 22:10:00] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:43 [I 2022-10-10 22:10:00] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00311 [I 2022-10-10 22:10:00] ...trap/engines/engine.py.230: loss: 0.18925 [I 2022-10-10 22:10:00] ...optimizers/trijoint.py.151: total_norm: 36.218662 [I 2022-10-10 22:10:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1220/2304 [I 2022-10-10 22:10:09] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:34 [I 2022-10-10 22:10:09] ...trap/engines/engine.py.229: process: 0.86056 | load: 0.00310 [I 2022-10-10 22:10:09] ...trap/engines/engine.py.230: loss: 0.14616 [I 2022-10-10 22:10:09] ...optimizers/trijoint.py.151: total_norm: 14.764032 [I 2022-10-10 22:10:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 1230/2304 [I 2022-10-10 22:10:17] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:23 [I 2022-10-10 22:10:17] ...trap/engines/engine.py.229: process: 0.85347 | load: 0.00314 [I 2022-10-10 22:10:17] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-10 22:10:17] ...optimizers/trijoint.py.151: total_norm: 27.861715 [I 2022-10-10 22:10:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 1240/2304 [I 2022-10-10 22:10:26] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:13 [I 2022-10-10 22:10:26] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00310 [I 2022-10-10 22:10:26] ...trap/engines/engine.py.230: loss: 0.08315 [I 2022-10-10 22:10:26] ...optimizers/trijoint.py.151: total_norm: 24.904585 [I 2022-10-10 22:10:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1250/2304 [I 2022-10-10 22:10:35] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:15:16 [I 2022-10-10 22:10:35] ...trap/engines/engine.py.229: process: 0.86616 | load: 0.00317 [I 2022-10-10 22:10:35] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-10 22:10:35] ...optimizers/trijoint.py.151: total_norm: 11.010315 [I 2022-10-10 22:10:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 1260/2304 [I 2022-10-10 22:10:43] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:15:07 [I 2022-10-10 22:10:43] ...trap/engines/engine.py.229: process: 0.86968 | load: 0.00310 [I 2022-10-10 22:10:43] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-10 22:10:43] ...optimizers/trijoint.py.151: total_norm: 10.347742 [I 2022-10-10 22:10:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 1270/2304 [I 2022-10-10 22:10:52] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:55 [I 2022-10-10 22:10:52] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00309 [I 2022-10-10 22:10:52] ...trap/engines/engine.py.230: loss: 0.11834 [I 2022-10-10 22:10:52] ...optimizers/trijoint.py.151: total_norm: 15.351760 [I 2022-10-10 22:11:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 1280/2304 [I 2022-10-10 22:11:01] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:46 [I 2022-10-10 22:11:01] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00316 [I 2022-10-10 22:11:01] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-10 22:11:01] ...optimizers/trijoint.py.151: total_norm: 25.890453 [I 2022-10-10 22:11:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1290/2304 [I 2022-10-10 22:11:09] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:35 [I 2022-10-10 22:11:09] ...trap/engines/engine.py.229: process: 0.85851 | load: 0.00308 [I 2022-10-10 22:11:09] ...trap/engines/engine.py.230: loss: 0.09239 [I 2022-10-10 22:11:09] ...optimizers/trijoint.py.151: total_norm: 7.425903 [I 2022-10-10 22:11:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1300/2304 [I 2022-10-10 22:11:18] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:31 [I 2022-10-10 22:11:18] ...trap/engines/engine.py.229: process: 0.88042 | load: 0.00308 [I 2022-10-10 22:11:18] ...trap/engines/engine.py.230: loss: 0.10911 [I 2022-10-10 22:11:18] ...optimizers/trijoint.py.151: total_norm: 11.397655 [I 2022-10-10 22:11:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 1310/2304 [I 2022-10-10 22:11:27] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:24 [I 2022-10-10 22:11:27] ...trap/engines/engine.py.229: process: 0.87969 | load: 0.00310 [I 2022-10-10 22:11:27] ...trap/engines/engine.py.230: loss: 0.10056 [I 2022-10-10 22:11:27] ...optimizers/trijoint.py.151: total_norm: 12.739899 [I 2022-10-10 22:11:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1320/2304 [I 2022-10-10 22:11:35] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:14:11 [I 2022-10-10 22:11:35] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00307 [I 2022-10-10 22:11:35] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-10 22:11:35] ...optimizers/trijoint.py.151: total_norm: 12.150908 [I 2022-10-10 22:11:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 1330/2304 [I 2022-10-10 22:11:44] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:14:01 [I 2022-10-10 22:11:44] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00311 [I 2022-10-10 22:11:44] ...trap/engines/engine.py.230: loss: 0.14793 [I 2022-10-10 22:11:44] ...optimizers/trijoint.py.151: total_norm: 74.558365 [I 2022-10-10 22:11:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 1340/2304 [I 2022-10-10 22:11:53] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:13:57 [I 2022-10-10 22:11:53] ...trap/engines/engine.py.229: process: 0.87369 | load: 0.00309 [I 2022-10-10 22:11:53] ...trap/engines/engine.py.230: loss: 0.08568 [I 2022-10-10 22:11:53] ...optimizers/trijoint.py.151: total_norm: 11.302759 [I 2022-10-10 22:12:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 1350/2304 [I 2022-10-10 22:12:01] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:13:45 [I 2022-10-10 22:12:01] ...trap/engines/engine.py.229: process: 0.86885 | load: 0.00312 [I 2022-10-10 22:12:01] ...trap/engines/engine.py.230: loss: 0.09337 [I 2022-10-10 22:12:01] ...optimizers/trijoint.py.151: total_norm: 14.995890 [I 2022-10-10 22:12:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 1360/2304 [I 2022-10-10 22:12:10] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:13:33 [I 2022-10-10 22:12:10] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00312 [I 2022-10-10 22:12:10] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-10 22:12:10] ...optimizers/trijoint.py.151: total_norm: 16.794100 [I 2022-10-10 22:12:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 1370/2304 [I 2022-10-10 22:12:19] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:13:28 [I 2022-10-10 22:12:19] ...trap/engines/engine.py.229: process: 0.85440 | load: 0.00323 [I 2022-10-10 22:12:19] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-10 22:12:19] ...optimizers/trijoint.py.151: total_norm: 15.634290 [I 2022-10-10 22:12:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1380/2304 [I 2022-10-10 22:12:28] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:13:20 [I 2022-10-10 22:12:28] ...trap/engines/engine.py.229: process: 0.87042 | load: 0.00320 [I 2022-10-10 22:12:28] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-10 22:12:28] ...optimizers/trijoint.py.151: total_norm: 19.661983 [I 2022-10-10 22:12:36] ...trap/engines/engine.py.224: train: epoch 87 | batch 1390/2304 [I 2022-10-10 22:12:36] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:13:10 [I 2022-10-10 22:12:36] ...trap/engines/engine.py.229: process: 0.88188 | load: 0.00312 [I 2022-10-10 22:12:36] ...trap/engines/engine.py.230: loss: 0.12198 [I 2022-10-10 22:12:36] ...optimizers/trijoint.py.151: total_norm: 15.759734 [I 2022-10-10 22:12:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 1400/2304 [I 2022-10-10 22:12:45] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:13:07 [I 2022-10-10 22:12:45] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00312 [I 2022-10-10 22:12:45] ...trap/engines/engine.py.230: loss: 0.08541 [I 2022-10-10 22:12:45] ...optimizers/trijoint.py.151: total_norm: 11.381594 [I 2022-10-10 22:12:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1410/2304 [I 2022-10-10 22:12:54] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:12:57 [I 2022-10-10 22:12:54] ...trap/engines/engine.py.229: process: 0.86888 | load: 0.00324 [I 2022-10-10 22:12:54] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-10 22:12:54] ...optimizers/trijoint.py.151: total_norm: 8.491556 [I 2022-10-10 22:13:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 1420/2304 [I 2022-10-10 22:13:02] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:45 [I 2022-10-10 22:13:02] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00317 [I 2022-10-10 22:13:02] ...trap/engines/engine.py.230: loss: 0.12869 [I 2022-10-10 22:13:02] ...optimizers/trijoint.py.151: total_norm: 15.988305 [I 2022-10-10 22:13:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1430/2304 [I 2022-10-10 22:13:11] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:38 [I 2022-10-10 22:13:11] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00307 [I 2022-10-10 22:13:11] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-10 22:13:11] ...optimizers/trijoint.py.151: total_norm: 15.375091 [I 2022-10-10 22:13:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1440/2304 [I 2022-10-10 22:13:20] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:12:22 [I 2022-10-10 22:13:20] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00313 [I 2022-10-10 22:13:20] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-10 22:13:20] ...optimizers/trijoint.py.151: total_norm: 8.967983 [I 2022-10-10 22:13:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1450/2304 [I 2022-10-10 22:13:28] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:12:18 [I 2022-10-10 22:13:28] ...trap/engines/engine.py.229: process: 0.86702 | load: 0.00316 [I 2022-10-10 22:13:28] ...trap/engines/engine.py.230: loss: 0.12318 [I 2022-10-10 22:13:28] ...optimizers/trijoint.py.151: total_norm: 27.038551 [I 2022-10-10 22:13:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1460/2304 [I 2022-10-10 22:13:37] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:12:16 [I 2022-10-10 22:13:37] ...trap/engines/engine.py.229: process: 0.88039 | load: 0.00311 [I 2022-10-10 22:13:37] ...trap/engines/engine.py.230: loss: 0.14655 [I 2022-10-10 22:13:37] ...optimizers/trijoint.py.151: total_norm: 24.999662 [I 2022-10-10 22:13:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1470/2304 [I 2022-10-10 22:13:46] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:56 [I 2022-10-10 22:13:46] ...trap/engines/engine.py.229: process: 0.85676 | load: 0.00316 [I 2022-10-10 22:13:46] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-10 22:13:46] ...optimizers/trijoint.py.151: total_norm: 17.545008 [I 2022-10-10 22:13:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1480/2304 [I 2022-10-10 22:13:54] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:41 [I 2022-10-10 22:13:54] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00293 [I 2022-10-10 22:13:54] ...trap/engines/engine.py.230: loss: 0.16906 [I 2022-10-10 22:13:54] ...optimizers/trijoint.py.151: total_norm: 12.707678 [I 2022-10-10 22:14:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1490/2304 [I 2022-10-10 22:14:03] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:40 [I 2022-10-10 22:14:03] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00302 [I 2022-10-10 22:14:03] ...trap/engines/engine.py.230: loss: 0.08168 [I 2022-10-10 22:14:03] ...optimizers/trijoint.py.151: total_norm: 13.909333 [I 2022-10-10 22:14:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1500/2304 [I 2022-10-10 22:14:11] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:11:22 [I 2022-10-10 22:14:11] ...trap/engines/engine.py.229: process: 0.84171 | load: 0.00294 [I 2022-10-10 22:14:11] ...trap/engines/engine.py.230: loss: 0.14849 [I 2022-10-10 22:14:11] ...optimizers/trijoint.py.151: total_norm: 17.046824 [I 2022-10-10 22:14:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1510/2304 [I 2022-10-10 22:14:20] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:11:19 [I 2022-10-10 22:14:20] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00295 [I 2022-10-10 22:14:20] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-10 22:14:20] ...optimizers/trijoint.py.151: total_norm: 24.865950 [I 2022-10-10 22:14:29] ...trap/engines/engine.py.224: train: epoch 87 | batch 1520/2304 [I 2022-10-10 22:14:29] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:11:10 [I 2022-10-10 22:14:29] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00301 [I 2022-10-10 22:14:29] ...trap/engines/engine.py.230: loss: 0.11038 [I 2022-10-10 22:14:29] ...optimizers/trijoint.py.151: total_norm: 8.792748 [I 2022-10-10 22:14:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1530/2304 [I 2022-10-10 22:14:37] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:11:03 [I 2022-10-10 22:14:37] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00296 [I 2022-10-10 22:14:37] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-10 22:14:37] ...optimizers/trijoint.py.151: total_norm: 14.760300 [I 2022-10-10 22:14:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1540/2304 [I 2022-10-10 22:14:46] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:10:52 [I 2022-10-10 22:14:46] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00296 [I 2022-10-10 22:14:46] ...trap/engines/engine.py.230: loss: 0.13976 [I 2022-10-10 22:14:46] ...optimizers/trijoint.py.151: total_norm: 12.475650 [I 2022-10-10 22:14:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1550/2304 [I 2022-10-10 22:14:54] ...trap/engines/engine.py.228: elapsed: 0:22:17 | left: 0:10:45 [I 2022-10-10 22:14:54] ...trap/engines/engine.py.229: process: 0.84867 | load: 0.00294 [I 2022-10-10 22:14:54] ...trap/engines/engine.py.230: loss: 0.14428 [I 2022-10-10 22:14:54] ...optimizers/trijoint.py.151: total_norm: 18.702227 [I 2022-10-10 22:15:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1560/2304 [I 2022-10-10 22:15:03] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:35 [I 2022-10-10 22:15:03] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00348 [I 2022-10-10 22:15:03] ...trap/engines/engine.py.230: loss: 0.12281 [I 2022-10-10 22:15:03] ...optimizers/trijoint.py.151: total_norm: 15.344429 [I 2022-10-10 22:15:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1570/2304 [I 2022-10-10 22:15:11] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:10:28 [I 2022-10-10 22:15:11] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00295 [I 2022-10-10 22:15:11] ...trap/engines/engine.py.230: loss: 0.10262 [I 2022-10-10 22:15:11] ...optimizers/trijoint.py.151: total_norm: 15.035680 [I 2022-10-10 22:15:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1580/2304 [I 2022-10-10 22:15:20] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:10:19 [I 2022-10-10 22:15:20] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00298 [I 2022-10-10 22:15:20] ...trap/engines/engine.py.230: loss: 0.18139 [I 2022-10-10 22:15:20] ...optimizers/trijoint.py.151: total_norm: 9.718393 [I 2022-10-10 22:15:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1590/2304 [I 2022-10-10 22:15:28] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:10:10 [I 2022-10-10 22:15:28] ...trap/engines/engine.py.229: process: 0.85992 | load: 0.00294 [I 2022-10-10 22:15:28] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-10 22:15:28] ...optimizers/trijoint.py.151: total_norm: 15.716632 [I 2022-10-10 22:15:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1600/2304 [I 2022-10-10 22:15:37] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:09:58 [I 2022-10-10 22:15:37] ...trap/engines/engine.py.229: process: 0.83502 | load: 0.00296 [I 2022-10-10 22:15:37] ...trap/engines/engine.py.230: loss: 0.12330 [I 2022-10-10 22:15:37] ...optimizers/trijoint.py.151: total_norm: 10.529569 [I 2022-10-10 22:15:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1610/2304 [I 2022-10-10 22:15:46] ...trap/engines/engine.py.228: elapsed: 0:23:08 | left: 0:09:51 [I 2022-10-10 22:15:46] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00299 [I 2022-10-10 22:15:46] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-10 22:15:46] ...optimizers/trijoint.py.151: total_norm: 32.386761 [I 2022-10-10 22:15:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1620/2304 [I 2022-10-10 22:15:54] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:43 [I 2022-10-10 22:15:54] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00297 [I 2022-10-10 22:15:54] ...trap/engines/engine.py.230: loss: 0.14192 [I 2022-10-10 22:15:54] ...optimizers/trijoint.py.151: total_norm: 10.612837 [I 2022-10-10 22:16:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1630/2304 [I 2022-10-10 22:16:03] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:09:33 [I 2022-10-10 22:16:03] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00299 [I 2022-10-10 22:16:03] ...trap/engines/engine.py.230: loss: 0.17022 [I 2022-10-10 22:16:03] ...optimizers/trijoint.py.151: total_norm: 20.425842 [I 2022-10-10 22:16:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1640/2304 [I 2022-10-10 22:16:11] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:09:29 [I 2022-10-10 22:16:11] ...trap/engines/engine.py.229: process: 0.86133 | load: 0.00297 [I 2022-10-10 22:16:11] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-10 22:16:11] ...optimizers/trijoint.py.151: total_norm: 22.862940 [I 2022-10-10 22:16:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1650/2304 [I 2022-10-10 22:16:20] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:09:18 [I 2022-10-10 22:16:20] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00293 [I 2022-10-10 22:16:20] ...trap/engines/engine.py.230: loss: 0.10183 [I 2022-10-10 22:16:20] ...optimizers/trijoint.py.151: total_norm: 10.733141 [I 2022-10-10 22:16:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1660/2304 [I 2022-10-10 22:16:28] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:09:12 [I 2022-10-10 22:16:28] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00296 [I 2022-10-10 22:16:28] ...trap/engines/engine.py.230: loss: 0.09688 [I 2022-10-10 22:16:28] ...optimizers/trijoint.py.151: total_norm: 13.790217 [I 2022-10-10 22:16:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1670/2304 [I 2022-10-10 22:16:37] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:09:03 [I 2022-10-10 22:16:37] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00296 [I 2022-10-10 22:16:37] ...trap/engines/engine.py.230: loss: 0.12432 [I 2022-10-10 22:16:37] ...optimizers/trijoint.py.151: total_norm: 16.691593 [I 2022-10-10 22:16:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1680/2304 [I 2022-10-10 22:16:46] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:08:54 [I 2022-10-10 22:16:46] ...trap/engines/engine.py.229: process: 0.87394 | load: 0.00297 [I 2022-10-10 22:16:46] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-10 22:16:46] ...optimizers/trijoint.py.151: total_norm: 12.508637 [I 2022-10-10 22:16:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1690/2304 [I 2022-10-10 22:16:54] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:41 [I 2022-10-10 22:16:54] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00305 [I 2022-10-10 22:16:54] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-10 22:16:54] ...optimizers/trijoint.py.151: total_norm: 14.089929 [I 2022-10-10 22:17:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1700/2304 [I 2022-10-10 22:17:03] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:08:35 [I 2022-10-10 22:17:03] ...trap/engines/engine.py.229: process: 0.85404 | load: 0.00312 [I 2022-10-10 22:17:03] ...trap/engines/engine.py.230: loss: 0.10742 [I 2022-10-10 22:17:03] ...optimizers/trijoint.py.151: total_norm: 13.357001 [I 2022-10-10 22:17:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1710/2304 [I 2022-10-10 22:17:11] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:08:24 [I 2022-10-10 22:17:11] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00301 [I 2022-10-10 22:17:11] ...trap/engines/engine.py.230: loss: 0.10642 [I 2022-10-10 22:17:11] ...optimizers/trijoint.py.151: total_norm: 8.616717 [I 2022-10-10 22:17:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1720/2304 [I 2022-10-10 22:17:20] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:20 [I 2022-10-10 22:17:20] ...trap/engines/engine.py.229: process: 0.85829 | load: 0.00302 [I 2022-10-10 22:17:20] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-10 22:17:20] ...optimizers/trijoint.py.151: total_norm: 27.889641 [I 2022-10-10 22:17:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1730/2304 [I 2022-10-10 22:17:28] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:08:13 [I 2022-10-10 22:17:28] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00301 [I 2022-10-10 22:17:28] ...trap/engines/engine.py.230: loss: 0.11224 [I 2022-10-10 22:17:28] ...optimizers/trijoint.py.151: total_norm: 10.483537 [I 2022-10-10 22:17:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1740/2304 [I 2022-10-10 22:17:37] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:08:03 [I 2022-10-10 22:17:37] ...trap/engines/engine.py.229: process: 0.86331 | load: 0.00303 [I 2022-10-10 22:17:37] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-10 22:17:37] ...optimizers/trijoint.py.151: total_norm: 12.763754 [I 2022-10-10 22:17:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 1750/2304 [I 2022-10-10 22:17:45] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:07:55 [I 2022-10-10 22:17:45] ...trap/engines/engine.py.229: process: 0.87879 | load: 0.00327 [I 2022-10-10 22:17:45] ...trap/engines/engine.py.230: loss: 0.16780 [I 2022-10-10 22:17:45] ...optimizers/trijoint.py.151: total_norm: 14.630774 [I 2022-10-10 22:17:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1760/2304 [I 2022-10-10 22:17:54] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:07:49 [I 2022-10-10 22:17:54] ...trap/engines/engine.py.229: process: 0.86186 | load: 0.00308 [I 2022-10-10 22:17:54] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-10 22:17:54] ...optimizers/trijoint.py.151: total_norm: 10.052903 [I 2022-10-10 22:18:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1770/2304 [I 2022-10-10 22:18:03] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:44 [I 2022-10-10 22:18:03] ...trap/engines/engine.py.229: process: 0.88759 | load: 0.00311 [I 2022-10-10 22:18:03] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-10 22:18:03] ...optimizers/trijoint.py.151: total_norm: 10.423185 [I 2022-10-10 22:18:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1780/2304 [I 2022-10-10 22:18:11] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:31 [I 2022-10-10 22:18:11] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00309 [I 2022-10-10 22:18:11] ...trap/engines/engine.py.230: loss: 0.16948 [I 2022-10-10 22:18:11] ...optimizers/trijoint.py.151: total_norm: 33.309807 [I 2022-10-10 22:18:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1790/2304 [I 2022-10-10 22:18:20] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:25 [I 2022-10-10 22:18:20] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00311 [I 2022-10-10 22:18:20] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-10 22:18:20] ...optimizers/trijoint.py.151: total_norm: 18.473204 [I 2022-10-10 22:18:29] ...trap/engines/engine.py.224: train: epoch 87 | batch 1800/2304 [I 2022-10-10 22:18:29] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:14 [I 2022-10-10 22:18:29] ...trap/engines/engine.py.229: process: 0.87013 | load: 0.00308 [I 2022-10-10 22:18:29] ...trap/engines/engine.py.230: loss: 0.13507 [I 2022-10-10 22:18:29] ...optimizers/trijoint.py.151: total_norm: 19.485939 [I 2022-10-10 22:18:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1810/2304 [I 2022-10-10 22:18:37] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:06 [I 2022-10-10 22:18:37] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00320 [I 2022-10-10 22:18:37] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-10 22:18:37] ...optimizers/trijoint.py.151: total_norm: 14.132830 [I 2022-10-10 22:18:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1820/2304 [I 2022-10-10 22:18:46] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:06:57 [I 2022-10-10 22:18:46] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00312 [I 2022-10-10 22:18:46] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-10 22:18:46] ...optimizers/trijoint.py.151: total_norm: 20.102766 [I 2022-10-10 22:18:55] ...trap/engines/engine.py.224: train: epoch 87 | batch 1830/2304 [I 2022-10-10 22:18:55] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:49 [I 2022-10-10 22:18:55] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00310 [I 2022-10-10 22:18:55] ...trap/engines/engine.py.230: loss: 0.16100 [I 2022-10-10 22:18:55] ...optimizers/trijoint.py.151: total_norm: 19.186136 [I 2022-10-10 22:19:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 1840/2304 [I 2022-10-10 22:19:04] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:43 [I 2022-10-10 22:19:04] ...trap/engines/engine.py.229: process: 0.87849 | load: 0.00327 [I 2022-10-10 22:19:04] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-10 22:19:04] ...optimizers/trijoint.py.151: total_norm: 9.763061 [I 2022-10-10 22:19:12] ...trap/engines/engine.py.224: train: epoch 87 | batch 1850/2304 [I 2022-10-10 22:19:12] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:34 [I 2022-10-10 22:19:12] ...trap/engines/engine.py.229: process: 0.87575 | load: 0.00313 [I 2022-10-10 22:19:12] ...trap/engines/engine.py.230: loss: 0.09333 [I 2022-10-10 22:19:12] ...optimizers/trijoint.py.151: total_norm: 13.855923 [I 2022-10-10 22:19:21] ...trap/engines/engine.py.224: train: epoch 87 | batch 1860/2304 [I 2022-10-10 22:19:21] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:24 [I 2022-10-10 22:19:21] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00322 [I 2022-10-10 22:19:21] ...trap/engines/engine.py.230: loss: 0.09986 [I 2022-10-10 22:19:21] ...optimizers/trijoint.py.151: total_norm: 17.338036 [I 2022-10-10 22:19:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 1870/2304 [I 2022-10-10 22:19:30] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:15 [I 2022-10-10 22:19:30] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00314 [I 2022-10-10 22:19:30] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-10 22:19:30] ...optimizers/trijoint.py.151: total_norm: 16.520105 [I 2022-10-10 22:19:38] ...trap/engines/engine.py.224: train: epoch 87 | batch 1880/2304 [I 2022-10-10 22:19:38] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:09 [I 2022-10-10 22:19:38] ...trap/engines/engine.py.229: process: 0.86869 | load: 0.00313 [I 2022-10-10 22:19:38] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-10 22:19:38] ...optimizers/trijoint.py.151: total_norm: 17.362717 [I 2022-10-10 22:19:47] ...trap/engines/engine.py.224: train: epoch 87 | batch 1890/2304 [I 2022-10-10 22:19:47] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:59 [I 2022-10-10 22:19:47] ...trap/engines/engine.py.229: process: 0.87919 | load: 0.00316 [I 2022-10-10 22:19:47] ...trap/engines/engine.py.230: loss: 0.19824 [I 2022-10-10 22:19:47] ...optimizers/trijoint.py.151: total_norm: 14.305189 [I 2022-10-10 22:19:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 1900/2304 [I 2022-10-10 22:19:56] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:48 [I 2022-10-10 22:19:56] ...trap/engines/engine.py.229: process: 0.84801 | load: 0.00320 [I 2022-10-10 22:19:56] ...trap/engines/engine.py.230: loss: 0.21025 [I 2022-10-10 22:19:56] ...optimizers/trijoint.py.151: total_norm: 14.007030 [I 2022-10-10 22:20:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 1910/2304 [I 2022-10-10 22:20:04] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:41 [I 2022-10-10 22:20:04] ...trap/engines/engine.py.229: process: 0.87054 | load: 0.00317 [I 2022-10-10 22:20:04] ...trap/engines/engine.py.230: loss: 0.10005 [I 2022-10-10 22:20:04] ...optimizers/trijoint.py.151: total_norm: 30.768408 [I 2022-10-10 22:20:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 1920/2304 [I 2022-10-10 22:20:13] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:31 [I 2022-10-10 22:20:13] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00312 [I 2022-10-10 22:20:13] ...trap/engines/engine.py.230: loss: 0.12685 [I 2022-10-10 22:20:13] ...optimizers/trijoint.py.151: total_norm: 12.986802 [I 2022-10-10 22:20:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 1930/2304 [I 2022-10-10 22:20:22] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:24 [I 2022-10-10 22:20:22] ...trap/engines/engine.py.229: process: 0.86848 | load: 0.00321 [I 2022-10-10 22:20:22] ...trap/engines/engine.py.230: loss: 0.12297 [I 2022-10-10 22:20:22] ...optimizers/trijoint.py.151: total_norm: 12.181628 [I 2022-10-10 22:20:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 1940/2304 [I 2022-10-10 22:20:30] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:05:15 [I 2022-10-10 22:20:30] ...trap/engines/engine.py.229: process: 0.87870 | load: 0.00315 [I 2022-10-10 22:20:30] ...trap/engines/engine.py.230: loss: 0.14669 [I 2022-10-10 22:20:30] ...optimizers/trijoint.py.151: total_norm: 9.679661 [I 2022-10-10 22:20:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 1950/2304 [I 2022-10-10 22:20:39] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:05:05 [I 2022-10-10 22:20:39] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00319 [I 2022-10-10 22:20:39] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-10 22:20:39] ...optimizers/trijoint.py.151: total_norm: 14.920574 [I 2022-10-10 22:20:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 1960/2304 [I 2022-10-10 22:20:48] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:04:57 [I 2022-10-10 22:20:48] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00310 [I 2022-10-10 22:20:48] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-10 22:20:48] ...optimizers/trijoint.py.151: total_norm: 26.086859 [I 2022-10-10 22:20:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 1970/2304 [I 2022-10-10 22:20:56] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:48 [I 2022-10-10 22:20:56] ...trap/engines/engine.py.229: process: 0.86211 | load: 0.00314 [I 2022-10-10 22:20:56] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-10 22:20:56] ...optimizers/trijoint.py.151: total_norm: 11.057117 [I 2022-10-10 22:21:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 1980/2304 [I 2022-10-10 22:21:05] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:39 [I 2022-10-10 22:21:05] ...trap/engines/engine.py.229: process: 0.87479 | load: 0.00318 [I 2022-10-10 22:21:05] ...trap/engines/engine.py.230: loss: 0.13397 [I 2022-10-10 22:21:05] ...optimizers/trijoint.py.151: total_norm: 14.280743 [I 2022-10-10 22:21:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 1990/2304 [I 2022-10-10 22:21:14] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:32 [I 2022-10-10 22:21:14] ...trap/engines/engine.py.229: process: 0.86359 | load: 0.00315 [I 2022-10-10 22:21:14] ...trap/engines/engine.py.230: loss: 0.12817 [I 2022-10-10 22:21:14] ...optimizers/trijoint.py.151: total_norm: 18.857874 [I 2022-10-10 22:21:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2000/2304 [I 2022-10-10 22:21:22] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:23 [I 2022-10-10 22:21:22] ...trap/engines/engine.py.229: process: 0.87269 | load: 0.00310 [I 2022-10-10 22:21:22] ...trap/engines/engine.py.230: loss: 0.12597 [I 2022-10-10 22:21:22] ...optimizers/trijoint.py.151: total_norm: 27.123148 [I 2022-10-10 22:21:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 2010/2304 [I 2022-10-10 22:21:31] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:04:16 [I 2022-10-10 22:21:31] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00321 [I 2022-10-10 22:21:31] ...trap/engines/engine.py.230: loss: 0.17565 [I 2022-10-10 22:21:31] ...optimizers/trijoint.py.151: total_norm: 12.864593 [I 2022-10-10 22:21:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2020/2304 [I 2022-10-10 22:21:40] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:04 [I 2022-10-10 22:21:40] ...trap/engines/engine.py.229: process: 0.85063 | load: 0.00309 [I 2022-10-10 22:21:40] ...trap/engines/engine.py.230: loss: 0.15910 [I 2022-10-10 22:21:40] ...optimizers/trijoint.py.151: total_norm: 20.100418 [I 2022-10-10 22:21:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 2030/2304 [I 2022-10-10 22:21:48] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:53 [I 2022-10-10 22:21:48] ...trap/engines/engine.py.229: process: 0.85705 | load: 0.00295 [I 2022-10-10 22:21:48] ...trap/engines/engine.py.230: loss: 0.16337 [I 2022-10-10 22:21:48] ...optimizers/trijoint.py.151: total_norm: 15.131104 [I 2022-10-10 22:21:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 2040/2304 [I 2022-10-10 22:21:57] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:46 [I 2022-10-10 22:21:57] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00295 [I 2022-10-10 22:21:57] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-10 22:21:57] ...optimizers/trijoint.py.151: total_norm: 16.005960 [I 2022-10-10 22:22:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 2050/2304 [I 2022-10-10 22:22:05] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:37 [I 2022-10-10 22:22:05] ...trap/engines/engine.py.229: process: 0.84793 | load: 0.00292 [I 2022-10-10 22:22:05] ...trap/engines/engine.py.230: loss: 0.10495 [I 2022-10-10 22:22:05] ...optimizers/trijoint.py.151: total_norm: 8.851262 [I 2022-10-10 22:22:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 2060/2304 [I 2022-10-10 22:22:14] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:27 [I 2022-10-10 22:22:14] ...trap/engines/engine.py.229: process: 0.83419 | load: 0.00296 [I 2022-10-10 22:22:14] ...trap/engines/engine.py.230: loss: 0.14636 [I 2022-10-10 22:22:14] ...optimizers/trijoint.py.151: total_norm: 7.739555 [I 2022-10-10 22:22:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2070/2304 [I 2022-10-10 22:22:22] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:21 [I 2022-10-10 22:22:22] ...trap/engines/engine.py.229: process: 0.86431 | load: 0.00293 [I 2022-10-10 22:22:22] ...trap/engines/engine.py.230: loss: 0.10608 [I 2022-10-10 22:22:22] ...optimizers/trijoint.py.151: total_norm: 11.751600 [I 2022-10-10 22:22:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 2080/2304 [I 2022-10-10 22:22:31] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:09 [I 2022-10-10 22:22:31] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00303 [I 2022-10-10 22:22:31] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-10 22:22:31] ...optimizers/trijoint.py.151: total_norm: 15.350774 [I 2022-10-10 22:22:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2090/2304 [I 2022-10-10 22:22:40] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:03 [I 2022-10-10 22:22:40] ...trap/engines/engine.py.229: process: 0.85607 | load: 0.00292 [I 2022-10-10 22:22:40] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-10 22:22:40] ...optimizers/trijoint.py.151: total_norm: 29.015100 [I 2022-10-10 22:22:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 2100/2304 [I 2022-10-10 22:22:48] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:54 [I 2022-10-10 22:22:48] ...trap/engines/engine.py.229: process: 0.86280 | load: 0.00293 [I 2022-10-10 22:22:48] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-10 22:22:48] ...optimizers/trijoint.py.151: total_norm: 12.836104 [I 2022-10-10 22:22:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 2110/2304 [I 2022-10-10 22:22:57] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:45 [I 2022-10-10 22:22:57] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00293 [I 2022-10-10 22:22:57] ...trap/engines/engine.py.230: loss: 0.12584 [I 2022-10-10 22:22:57] ...optimizers/trijoint.py.151: total_norm: 16.679140 [I 2022-10-10 22:23:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 2120/2304 [I 2022-10-10 22:23:05] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:36 [I 2022-10-10 22:23:05] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00293 [I 2022-10-10 22:23:05] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-10 22:23:05] ...optimizers/trijoint.py.151: total_norm: 11.927725 [I 2022-10-10 22:23:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 2130/2304 [I 2022-10-10 22:23:14] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:28 [I 2022-10-10 22:23:14] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00294 [I 2022-10-10 22:23:14] ...trap/engines/engine.py.230: loss: 0.10950 [I 2022-10-10 22:23:14] ...optimizers/trijoint.py.151: total_norm: 11.773323 [I 2022-10-10 22:23:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2140/2304 [I 2022-10-10 22:23:22] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:19 [I 2022-10-10 22:23:22] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00290 [I 2022-10-10 22:23:22] ...trap/engines/engine.py.230: loss: 0.19521 [I 2022-10-10 22:23:22] ...optimizers/trijoint.py.151: total_norm: 17.952179 [I 2022-10-10 22:23:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 2150/2304 [I 2022-10-10 22:23:31] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:12 [I 2022-10-10 22:23:31] ...trap/engines/engine.py.229: process: 0.86333 | load: 0.00297 [I 2022-10-10 22:23:31] ...trap/engines/engine.py.230: loss: 0.12914 [I 2022-10-10 22:23:31] ...optimizers/trijoint.py.151: total_norm: 20.082788 [I 2022-10-10 22:23:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2160/2304 [I 2022-10-10 22:23:40] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:03 [I 2022-10-10 22:23:40] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00293 [I 2022-10-10 22:23:40] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-10 22:23:40] ...optimizers/trijoint.py.151: total_norm: 41.518211 [I 2022-10-10 22:23:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 2170/2304 [I 2022-10-10 22:23:48] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:54 [I 2022-10-10 22:23:48] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00294 [I 2022-10-10 22:23:48] ...trap/engines/engine.py.230: loss: 0.08227 [I 2022-10-10 22:23:48] ...optimizers/trijoint.py.151: total_norm: 13.593632 [I 2022-10-10 22:23:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 2180/2304 [I 2022-10-10 22:23:57] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:45 [I 2022-10-10 22:23:57] ...trap/engines/engine.py.229: process: 0.84308 | load: 0.00304 [I 2022-10-10 22:23:57] ...trap/engines/engine.py.230: loss: 0.09209 [I 2022-10-10 22:23:57] ...optimizers/trijoint.py.151: total_norm: 6.724127 [I 2022-10-10 22:24:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 2190/2304 [I 2022-10-10 22:24:05] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:37 [I 2022-10-10 22:24:05] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00294 [I 2022-10-10 22:24:05] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-10 22:24:05] ...optimizers/trijoint.py.151: total_norm: 10.375992 [I 2022-10-10 22:24:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 2200/2304 [I 2022-10-10 22:24:14] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:28 [I 2022-10-10 22:24:14] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00296 [I 2022-10-10 22:24:14] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-10 22:24:14] ...optimizers/trijoint.py.151: total_norm: 12.844606 [I 2022-10-10 22:24:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2210/2304 [I 2022-10-10 22:24:22] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:20 [I 2022-10-10 22:24:22] ...trap/engines/engine.py.229: process: 0.85761 | load: 0.00296 [I 2022-10-10 22:24:22] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-10 22:24:22] ...optimizers/trijoint.py.151: total_norm: 26.210613 [I 2022-10-10 22:24:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 2220/2304 [I 2022-10-10 22:24:31] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:01:11 [I 2022-10-10 22:24:31] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00299 [I 2022-10-10 22:24:31] ...trap/engines/engine.py.230: loss: 0.14954 [I 2022-10-10 22:24:31] ...optimizers/trijoint.py.151: total_norm: 19.322487 [I 2022-10-10 22:24:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2230/2304 [I 2022-10-10 22:24:40] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:03 [I 2022-10-10 22:24:40] ...trap/engines/engine.py.229: process: 0.85289 | load: 0.00303 [I 2022-10-10 22:24:40] ...trap/engines/engine.py.230: loss: 0.17024 [I 2022-10-10 22:24:40] ...optimizers/trijoint.py.151: total_norm: 10.843786 [I 2022-10-10 22:24:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 2240/2304 [I 2022-10-10 22:24:48] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:54 [I 2022-10-10 22:24:48] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00295 [I 2022-10-10 22:24:48] ...trap/engines/engine.py.230: loss: 0.08410 [I 2022-10-10 22:24:48] ...optimizers/trijoint.py.151: total_norm: 10.232471 [I 2022-10-10 22:24:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 2250/2304 [I 2022-10-10 22:24:57] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:46 [I 2022-10-10 22:24:57] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00295 [I 2022-10-10 22:24:57] ...trap/engines/engine.py.230: loss: 0.09787 [I 2022-10-10 22:24:57] ...optimizers/trijoint.py.151: total_norm: 11.320113 [I 2022-10-10 22:25:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 2260/2304 [I 2022-10-10 22:25:05] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:37 [I 2022-10-10 22:25:05] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00300 [I 2022-10-10 22:25:05] ...trap/engines/engine.py.230: loss: 0.08657 [I 2022-10-10 22:25:05] ...optimizers/trijoint.py.151: total_norm: 13.025983 [I 2022-10-10 22:25:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 2270/2304 [I 2022-10-10 22:25:14] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:29 [I 2022-10-10 22:25:14] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00294 [I 2022-10-10 22:25:14] ...trap/engines/engine.py.230: loss: 0.06839 [I 2022-10-10 22:25:14] ...optimizers/trijoint.py.151: total_norm: 9.601992 [I 2022-10-10 22:25:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2280/2304 [I 2022-10-10 22:25:22] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:20 [I 2022-10-10 22:25:22] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00294 [I 2022-10-10 22:25:22] ...trap/engines/engine.py.230: loss: 0.09732 [I 2022-10-10 22:25:22] ...optimizers/trijoint.py.151: total_norm: 12.350783 [I 2022-10-10 22:25:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 2290/2304 [I 2022-10-10 22:25:31] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:12 [I 2022-10-10 22:25:31] ...trap/engines/engine.py.229: process: 0.84680 | load: 0.00294 [I 2022-10-10 22:25:31] ...trap/engines/engine.py.230: loss: 0.10566 [I 2022-10-10 22:25:31] ...optimizers/trijoint.py.151: total_norm: 9.223289 [I 2022-10-10 22:25:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2300/2304 [I 2022-10-10 22:25:40] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:03 [I 2022-10-10 22:25:40] ...trap/engines/engine.py.229: process: 0.82987 | load: 0.00017 [I 2022-10-10 22:25:40] ...trap/engines/engine.py.230: loss: 0.09983 [I 2022-10-10 22:25:40] ...optimizers/trijoint.py.151: total_norm: 16.246481 [I 2022-10-10 22:25:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 2304/2304 [I 2022-10-10 22:25:43] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:00 [I 2022-10-10 22:25:43] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00016 [I 2022-10-10 22:25:43] ...trap/engines/engine.py.230: loss: 0.10021 [I 2022-10-10 22:25:43] ...optimizers/trijoint.py.151: total_norm: 10.200267 [S 2022-10-10 22:25:44] ...trap/engines/engine.py.129: train_epoch.epoch: 87 [S 2022-10-10 22:25:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.12913502571445742 [I 2022-10-10 22:25:44] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 22:25:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 87 [I 2022-10-10 22:25:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 22:25:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 22:25:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 22:25:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 22:25:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 22:25:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 0/510 [I 2022-10-10 22:25:51] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:13 [I 2022-10-10 22:25:51] ...trap/engines/engine.py.326: process: 5.55638 | load: 5.04266 [I 2022-10-10 22:25:55] ...trap/engines/engine.py.321: eval: epoch 87 | batch 10/510 [I 2022-10-10 22:25:55] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:09 [I 2022-10-10 22:25:55] ...trap/engines/engine.py.326: process: 0.43132 | load: 0.00018 [I 2022-10-10 22:26:00] ...trap/engines/engine.py.321: eval: epoch 87 | batch 20/510 [I 2022-10-10 22:26:00] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:03:54 [I 2022-10-10 22:26:00] ...trap/engines/engine.py.326: process: 0.40847 | load: 0.00018 [I 2022-10-10 22:26:05] ...trap/engines/engine.py.321: eval: epoch 87 | batch 30/510 [I 2022-10-10 22:26:05] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:04:10 [I 2022-10-10 22:26:05] ...trap/engines/engine.py.326: process: 0.57031 | load: 0.00020 [I 2022-10-10 22:26:10] ...trap/engines/engine.py.321: eval: epoch 87 | batch 40/510 [I 2022-10-10 22:26:10] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:08 [I 2022-10-10 22:26:10] ...trap/engines/engine.py.326: process: 0.49906 | load: 0.00020 [I 2022-10-10 22:26:16] ...trap/engines/engine.py.321: eval: epoch 87 | batch 50/510 [I 2022-10-10 22:26:16] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:58 [I 2022-10-10 22:26:16] ...trap/engines/engine.py.326: process: 0.44696 | load: 0.00028 [I 2022-10-10 22:26:21] ...trap/engines/engine.py.321: eval: epoch 87 | batch 60/510 [I 2022-10-10 22:26:21] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:06 [I 2022-10-10 22:26:21] ...trap/engines/engine.py.326: process: 0.56847 | load: 0.00020 [I 2022-10-10 22:26:26] ...trap/engines/engine.py.321: eval: epoch 87 | batch 70/510 [I 2022-10-10 22:26:26] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:27 [I 2022-10-10 22:26:26] ...trap/engines/engine.py.326: process: 0.47604 | load: 0.00021 [I 2022-10-10 22:26:31] ...trap/engines/engine.py.321: eval: epoch 87 | batch 80/510 [I 2022-10-10 22:26:31] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:41 [I 2022-10-10 22:26:31] ...trap/engines/engine.py.326: process: 0.49927 | load: 0.00021 [I 2022-10-10 22:26:36] ...trap/engines/engine.py.321: eval: epoch 87 | batch 90/510 [I 2022-10-10 22:26:36] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:31 [I 2022-10-10 22:26:36] ...trap/engines/engine.py.326: process: 0.50059 | load: 0.00019 [I 2022-10-10 22:26:41] ...trap/engines/engine.py.321: eval: epoch 87 | batch 100/510 [I 2022-10-10 22:26:41] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:09 [I 2022-10-10 22:26:41] ...trap/engines/engine.py.326: process: 0.31451 | load: 0.00020 [I 2022-10-10 22:26:44] ...trap/engines/engine.py.321: eval: epoch 87 | batch 110/510 [I 2022-10-10 22:26:44] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:15 [I 2022-10-10 22:26:44] ...trap/engines/engine.py.326: process: 0.32181 | load: 0.00021 [I 2022-10-10 22:26:47] ...trap/engines/engine.py.321: eval: epoch 87 | batch 120/510 [I 2022-10-10 22:26:47] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:06 [I 2022-10-10 22:26:47] ...trap/engines/engine.py.326: process: 0.32626 | load: 0.00020 [I 2022-10-10 22:26:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 130/510 [I 2022-10-10 22:26:51] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:01 [I 2022-10-10 22:26:51] ...trap/engines/engine.py.326: process: 0.31371 | load: 0.00022 [I 2022-10-10 22:26:54] ...trap/engines/engine.py.321: eval: epoch 87 | batch 140/510 [I 2022-10-10 22:26:54] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:58 [I 2022-10-10 22:26:54] ...trap/engines/engine.py.326: process: 0.32564 | load: 0.00022 [I 2022-10-10 22:26:57] ...trap/engines/engine.py.321: eval: epoch 87 | batch 150/510 [I 2022-10-10 22:26:57] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:56 [I 2022-10-10 22:26:57] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00021 [I 2022-10-10 22:27:00] ...trap/engines/engine.py.321: eval: epoch 87 | batch 160/510 [I 2022-10-10 22:27:00] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:53 [I 2022-10-10 22:27:00] ...trap/engines/engine.py.326: process: 0.32448 | load: 0.00024 [I 2022-10-10 22:27:04] ...trap/engines/engine.py.321: eval: epoch 87 | batch 170/510 [I 2022-10-10 22:27:04] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:49 [I 2022-10-10 22:27:04] ...trap/engines/engine.py.326: process: 0.31789 | load: 0.00021 [I 2022-10-10 22:27:07] ...trap/engines/engine.py.321: eval: epoch 87 | batch 180/510 [I 2022-10-10 22:27:07] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:47 [I 2022-10-10 22:27:07] ...trap/engines/engine.py.326: process: 0.32977 | load: 0.00021 [I 2022-10-10 22:27:10] ...trap/engines/engine.py.321: eval: epoch 87 | batch 190/510 [I 2022-10-10 22:27:10] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:44 [I 2022-10-10 22:27:10] ...trap/engines/engine.py.326: process: 0.32549 | load: 0.00020 [I 2022-10-10 22:27:13] ...trap/engines/engine.py.321: eval: epoch 87 | batch 200/510 [I 2022-10-10 22:27:13] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:40 [I 2022-10-10 22:27:13] ...trap/engines/engine.py.326: process: 0.32847 | load: 0.00021 [I 2022-10-10 22:27:17] ...trap/engines/engine.py.321: eval: epoch 87 | batch 210/510 [I 2022-10-10 22:27:17] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:36 [I 2022-10-10 22:27:17] ...trap/engines/engine.py.326: process: 0.32155 | load: 0.00021 [I 2022-10-10 22:27:20] ...trap/engines/engine.py.321: eval: epoch 87 | batch 220/510 [I 2022-10-10 22:27:20] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:33 [I 2022-10-10 22:27:20] ...trap/engines/engine.py.326: process: 0.32661 | load: 0.00021 [I 2022-10-10 22:27:23] ...trap/engines/engine.py.321: eval: epoch 87 | batch 230/510 [I 2022-10-10 22:27:23] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:30 [I 2022-10-10 22:27:23] ...trap/engines/engine.py.326: process: 0.32268 | load: 0.00021 [I 2022-10-10 22:27:26] ...trap/engines/engine.py.321: eval: epoch 87 | batch 240/510 [I 2022-10-10 22:27:26] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:27 [I 2022-10-10 22:27:26] ...trap/engines/engine.py.326: process: 0.32722 | load: 0.00021 [I 2022-10-10 22:27:29] ...trap/engines/engine.py.321: eval: epoch 87 | batch 250/510 [I 2022-10-10 22:27:29] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-10 22:27:29] ...trap/engines/engine.py.326: process: 0.32767 | load: 0.00021 [I 2022-10-10 22:27:33] ...trap/engines/engine.py.321: eval: epoch 87 | batch 260/510 [I 2022-10-10 22:27:33] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:20 [I 2022-10-10 22:27:33] ...trap/engines/engine.py.326: process: 0.32765 | load: 0.00023 [I 2022-10-10 22:27:36] ...trap/engines/engine.py.321: eval: epoch 87 | batch 270/510 [I 2022-10-10 22:27:36] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:17 [I 2022-10-10 22:27:36] ...trap/engines/engine.py.326: process: 0.31759 | load: 0.00021 [I 2022-10-10 22:27:39] ...trap/engines/engine.py.321: eval: epoch 87 | batch 280/510 [I 2022-10-10 22:27:39] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:14 [I 2022-10-10 22:27:39] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00021 [I 2022-10-10 22:27:42] ...trap/engines/engine.py.321: eval: epoch 87 | batch 290/510 [I 2022-10-10 22:27:42] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:11 [I 2022-10-10 22:27:42] ...trap/engines/engine.py.326: process: 0.32403 | load: 0.00021 [I 2022-10-10 22:27:46] ...trap/engines/engine.py.321: eval: epoch 87 | batch 300/510 [I 2022-10-10 22:27:46] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:07 [I 2022-10-10 22:27:46] ...trap/engines/engine.py.326: process: 0.32014 | load: 0.00021 [I 2022-10-10 22:27:49] ...trap/engines/engine.py.321: eval: epoch 87 | batch 310/510 [I 2022-10-10 22:27:49] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:04 [I 2022-10-10 22:27:49] ...trap/engines/engine.py.326: process: 0.32716 | load: 0.00020 [I 2022-10-10 22:27:52] ...trap/engines/engine.py.321: eval: epoch 87 | batch 320/510 [I 2022-10-10 22:27:52] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:01 [I 2022-10-10 22:27:52] ...trap/engines/engine.py.326: process: 0.32698 | load: 0.00021 [I 2022-10-10 22:27:55] ...trap/engines/engine.py.321: eval: epoch 87 | batch 330/510 [I 2022-10-10 22:27:55] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:58 [I 2022-10-10 22:27:55] ...trap/engines/engine.py.326: process: 0.31782 | load: 0.00020 [I 2022-10-10 22:27:59] ...trap/engines/engine.py.321: eval: epoch 87 | batch 340/510 [I 2022-10-10 22:27:59] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-10 22:27:59] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00021 [I 2022-10-10 22:28:02] ...trap/engines/engine.py.321: eval: epoch 87 | batch 350/510 [I 2022-10-10 22:28:02] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-10 22:28:02] ...trap/engines/engine.py.326: process: 0.31327 | load: 0.00021 [I 2022-10-10 22:28:05] ...trap/engines/engine.py.321: eval: epoch 87 | batch 360/510 [I 2022-10-10 22:28:05] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:48 [I 2022-10-10 22:28:05] ...trap/engines/engine.py.326: process: 0.32765 | load: 0.00023 [I 2022-10-10 22:28:08] ...trap/engines/engine.py.321: eval: epoch 87 | batch 370/510 [I 2022-10-10 22:28:08] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:45 [I 2022-10-10 22:28:08] ...trap/engines/engine.py.326: process: 0.31653 | load: 0.00021 [I 2022-10-10 22:28:12] ...trap/engines/engine.py.321: eval: epoch 87 | batch 380/510 [I 2022-10-10 22:28:12] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-10 22:28:12] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00024 [I 2022-10-10 22:28:15] ...trap/engines/engine.py.321: eval: epoch 87 | batch 390/510 [I 2022-10-10 22:28:15] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-10 22:28:15] ...trap/engines/engine.py.326: process: 0.32887 | load: 0.00022 [I 2022-10-10 22:28:18] ...trap/engines/engine.py.321: eval: epoch 87 | batch 400/510 [I 2022-10-10 22:28:18] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:36 [I 2022-10-10 22:28:18] ...trap/engines/engine.py.326: process: 0.33137 | load: 0.00022 [I 2022-10-10 22:28:21] ...trap/engines/engine.py.321: eval: epoch 87 | batch 410/510 [I 2022-10-10 22:28:21] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:32 [I 2022-10-10 22:28:21] ...trap/engines/engine.py.326: process: 0.32604 | load: 0.00022 [I 2022-10-10 22:28:25] ...trap/engines/engine.py.321: eval: epoch 87 | batch 420/510 [I 2022-10-10 22:28:25] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:29 [I 2022-10-10 22:28:25] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00021 [I 2022-10-10 22:28:28] ...trap/engines/engine.py.321: eval: epoch 87 | batch 430/510 [I 2022-10-10 22:28:28] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:26 [I 2022-10-10 22:28:28] ...trap/engines/engine.py.326: process: 0.33146 | load: 0.00022 [I 2022-10-10 22:28:31] ...trap/engines/engine.py.321: eval: epoch 87 | batch 440/510 [I 2022-10-10 22:28:31] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-10 22:28:31] ...trap/engines/engine.py.326: process: 0.32591 | load: 0.00021 [I 2022-10-10 22:28:34] ...trap/engines/engine.py.321: eval: epoch 87 | batch 450/510 [I 2022-10-10 22:28:34] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-10 22:28:34] ...trap/engines/engine.py.326: process: 0.32796 | load: 0.00021 [I 2022-10-10 22:28:38] ...trap/engines/engine.py.321: eval: epoch 87 | batch 460/510 [I 2022-10-10 22:28:38] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:16 [I 2022-10-10 22:28:38] ...trap/engines/engine.py.326: process: 0.32629 | load: 0.00021 [I 2022-10-10 22:28:41] ...trap/engines/engine.py.321: eval: epoch 87 | batch 470/510 [I 2022-10-10 22:28:41] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:13 [I 2022-10-10 22:28:41] ...trap/engines/engine.py.326: process: 0.33897 | load: 0.00020 [I 2022-10-10 22:28:44] ...trap/engines/engine.py.321: eval: epoch 87 | batch 480/510 [I 2022-10-10 22:28:44] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-10 22:28:44] ...trap/engines/engine.py.326: process: 0.32516 | load: 0.00021 [I 2022-10-10 22:28:47] ...trap/engines/engine.py.321: eval: epoch 87 | batch 490/510 [I 2022-10-10 22:28:47] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-10 22:28:47] ...trap/engines/engine.py.326: process: 0.31812 | load: 0.00021 [I 2022-10-10 22:28:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 500/510 [I 2022-10-10 22:28:51] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-10 22:28:51] ...trap/engines/engine.py.326: process: 0.32593 | load: 0.00020 [I 2022-10-10 22:28:54] ...trap/engines/engine.py.321: eval: epoch 87 | batch 510/510 [I 2022-10-10 22:28:54] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-10 22:28:54] ...trap/engines/engine.py.326: process: 0.32776 | load: 0.00018 [S 2022-10-10 22:28:55] ...trap/engines/engine.py.132: eval_epoch.epoch: 87 [I 2022-10-10 22:28:55] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737100 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908900 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937700 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732600 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909200 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935900 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013057 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009332 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004981 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014941 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009631 [S 2022-10-10 22:29:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005224 [I 2022-10-10 22:29:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 22:29:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 22:29:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 22:29:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 22:29:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 22:29:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 22:29:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 22:29:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 22:29:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 88 [I 2022-10-10 22:29:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 22:29:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 0/2304 [I 2022-10-10 22:29:57] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:08:00 [I 2022-10-10 22:29:57] ...trap/engines/engine.py.229: process: 3.33358 | load: 2.39176 [I 2022-10-10 22:29:57] ...trap/engines/engine.py.230: loss: 0.09070 [I 2022-10-10 22:29:57] ...optimizers/trijoint.py.151: total_norm: 15.901402 [I 2022-10-10 22:30:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 10/2304 [I 2022-10-10 22:30:05] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:48 [I 2022-10-10 22:30:05] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00484 [I 2022-10-10 22:30:05] ...trap/engines/engine.py.230: loss: 0.09614 [I 2022-10-10 22:30:05] ...optimizers/trijoint.py.151: total_norm: 8.140878 [I 2022-10-10 22:30:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 20/2304 [I 2022-10-10 22:30:14] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:32 [I 2022-10-10 22:30:14] ...trap/engines/engine.py.229: process: 0.83898 | load: 0.00401 [I 2022-10-10 22:30:14] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-10 22:30:14] ...optimizers/trijoint.py.151: total_norm: 21.972271 [I 2022-10-10 22:30:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 30/2304 [I 2022-10-10 22:30:23] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:21 [I 2022-10-10 22:30:23] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00379 [I 2022-10-10 22:30:23] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-10 22:30:23] ...optimizers/trijoint.py.151: total_norm: 12.492885 [I 2022-10-10 22:30:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 40/2304 [I 2022-10-10 22:30:31] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:04 [I 2022-10-10 22:30:31] ...trap/engines/engine.py.229: process: 0.84071 | load: 0.00414 [I 2022-10-10 22:30:31] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-10 22:30:31] ...optimizers/trijoint.py.151: total_norm: 25.467323 [I 2022-10-10 22:30:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 50/2304 [I 2022-10-10 22:30:40] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:01 [I 2022-10-10 22:30:40] ...trap/engines/engine.py.229: process: 0.85252 | load: 0.00356 [I 2022-10-10 22:30:40] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-10 22:30:40] ...optimizers/trijoint.py.151: total_norm: 36.323841 [I 2022-10-10 22:30:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 60/2304 [I 2022-10-10 22:30:48] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:51 [I 2022-10-10 22:30:48] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00374 [I 2022-10-10 22:30:48] ...trap/engines/engine.py.230: loss: 0.11090 [I 2022-10-10 22:30:48] ...optimizers/trijoint.py.151: total_norm: 23.593006 [I 2022-10-10 22:30:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 70/2304 [I 2022-10-10 22:30:57] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:46 [I 2022-10-10 22:30:57] ...trap/engines/engine.py.229: process: 0.85157 | load: 0.00327 [I 2022-10-10 22:30:57] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-10 22:30:57] ...optimizers/trijoint.py.151: total_norm: 14.634967 [I 2022-10-10 22:31:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 80/2304 [I 2022-10-10 22:31:05] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:21 [I 2022-10-10 22:31:05] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00340 [I 2022-10-10 22:31:05] ...trap/engines/engine.py.230: loss: 0.17865 [I 2022-10-10 22:31:05] ...optimizers/trijoint.py.151: total_norm: 23.884869 [I 2022-10-10 22:31:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 90/2304 [I 2022-10-10 22:31:14] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:27 [I 2022-10-10 22:31:14] ...trap/engines/engine.py.229: process: 0.85423 | load: 0.00291 [I 2022-10-10 22:31:14] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-10 22:31:14] ...optimizers/trijoint.py.151: total_norm: 13.049370 [I 2022-10-10 22:31:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 100/2304 [I 2022-10-10 22:31:22] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:11 [I 2022-10-10 22:31:22] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00360 [I 2022-10-10 22:31:22] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-10 22:31:22] ...optimizers/trijoint.py.151: total_norm: 18.314957 [I 2022-10-10 22:31:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 110/2304 [I 2022-10-10 22:31:31] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:08 [I 2022-10-10 22:31:31] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00357 [I 2022-10-10 22:31:31] ...trap/engines/engine.py.230: loss: 0.16041 [I 2022-10-10 22:31:31] ...optimizers/trijoint.py.151: total_norm: 8.528922 [I 2022-10-10 22:31:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 120/2304 [I 2022-10-10 22:31:39] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:52 [I 2022-10-10 22:31:39] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00344 [I 2022-10-10 22:31:39] ...trap/engines/engine.py.230: loss: 0.25021 [I 2022-10-10 22:31:39] ...optimizers/trijoint.py.151: total_norm: 21.863054 [I 2022-10-10 22:31:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 130/2304 [I 2022-10-10 22:31:48] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:47 [I 2022-10-10 22:31:48] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00303 [I 2022-10-10 22:31:48] ...trap/engines/engine.py.230: loss: 0.11952 [I 2022-10-10 22:31:48] ...optimizers/trijoint.py.151: total_norm: 9.216839 [I 2022-10-10 22:31:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 140/2304 [I 2022-10-10 22:31:56] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:46 [I 2022-10-10 22:31:56] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00322 [I 2022-10-10 22:31:56] ...trap/engines/engine.py.230: loss: 0.08331 [I 2022-10-10 22:31:56] ...optimizers/trijoint.py.151: total_norm: 13.001556 [I 2022-10-10 22:32:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 150/2304 [I 2022-10-10 22:32:05] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:35 [I 2022-10-10 22:32:05] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00320 [I 2022-10-10 22:32:05] ...trap/engines/engine.py.230: loss: 0.09903 [I 2022-10-10 22:32:05] ...optimizers/trijoint.py.151: total_norm: 20.482746 [I 2022-10-10 22:32:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 160/2304 [I 2022-10-10 22:32:14] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:30 [I 2022-10-10 22:32:14] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00297 [I 2022-10-10 22:32:14] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-10 22:32:14] ...optimizers/trijoint.py.151: total_norm: 15.119107 [I 2022-10-10 22:32:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 170/2304 [I 2022-10-10 22:32:22] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:27 [I 2022-10-10 22:32:22] ...trap/engines/engine.py.229: process: 0.85768 | load: 0.00291 [I 2022-10-10 22:32:22] ...trap/engines/engine.py.230: loss: 0.17231 [I 2022-10-10 22:32:22] ...optimizers/trijoint.py.151: total_norm: 16.144501 [I 2022-10-10 22:32:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 180/2304 [I 2022-10-10 22:32:31] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:11 [I 2022-10-10 22:32:31] ...trap/engines/engine.py.229: process: 0.85431 | load: 0.00299 [I 2022-10-10 22:32:31] ...trap/engines/engine.py.230: loss: 0.25321 [I 2022-10-10 22:32:31] ...optimizers/trijoint.py.151: total_norm: 19.997149 [I 2022-10-10 22:32:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 190/2304 [I 2022-10-10 22:32:39] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:07 [I 2022-10-10 22:32:39] ...trap/engines/engine.py.229: process: 0.84667 | load: 0.00310 [I 2022-10-10 22:32:39] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-10 22:32:39] ...optimizers/trijoint.py.151: total_norm: 19.771973 [I 2022-10-10 22:32:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 200/2304 [I 2022-10-10 22:32:48] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:58 [I 2022-10-10 22:32:48] ...trap/engines/engine.py.229: process: 0.83779 | load: 0.00293 [I 2022-10-10 22:32:48] ...trap/engines/engine.py.230: loss: 0.08156 [I 2022-10-10 22:32:48] ...optimizers/trijoint.py.151: total_norm: 7.967459 [I 2022-10-10 22:32:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 210/2304 [I 2022-10-10 22:32:56] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:41 [I 2022-10-10 22:32:56] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00305 [I 2022-10-10 22:32:56] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-10 22:32:56] ...optimizers/trijoint.py.151: total_norm: 11.516241 [I 2022-10-10 22:33:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 220/2304 [I 2022-10-10 22:33:05] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:33 [I 2022-10-10 22:33:05] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00291 [I 2022-10-10 22:33:05] ...trap/engines/engine.py.230: loss: 0.13569 [I 2022-10-10 22:33:05] ...optimizers/trijoint.py.151: total_norm: 14.452727 [I 2022-10-10 22:33:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 230/2304 [I 2022-10-10 22:33:14] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:38 [I 2022-10-10 22:33:14] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00295 [I 2022-10-10 22:33:14] ...trap/engines/engine.py.230: loss: 0.11133 [I 2022-10-10 22:33:14] ...optimizers/trijoint.py.151: total_norm: 20.485348 [I 2022-10-10 22:33:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 240/2304 [I 2022-10-10 22:33:22] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:20 [I 2022-10-10 22:33:22] ...trap/engines/engine.py.229: process: 0.84403 | load: 0.00297 [I 2022-10-10 22:33:22] ...trap/engines/engine.py.230: loss: 0.14257 [I 2022-10-10 22:33:22] ...optimizers/trijoint.py.151: total_norm: 14.221109 [I 2022-10-10 22:33:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 250/2304 [I 2022-10-10 22:33:31] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:10 [I 2022-10-10 22:33:31] ...trap/engines/engine.py.229: process: 0.83302 | load: 0.00306 [I 2022-10-10 22:33:31] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-10 22:33:31] ...optimizers/trijoint.py.151: total_norm: 15.203951 [I 2022-10-10 22:33:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 260/2304 [I 2022-10-10 22:33:39] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:13 [I 2022-10-10 22:33:39] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00299 [I 2022-10-10 22:33:39] ...trap/engines/engine.py.230: loss: 0.19500 [I 2022-10-10 22:33:39] ...optimizers/trijoint.py.151: total_norm: 16.325550 [I 2022-10-10 22:33:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 270/2304 [I 2022-10-10 22:33:48] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:57 [I 2022-10-10 22:33:48] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00308 [I 2022-10-10 22:33:48] ...trap/engines/engine.py.230: loss: 0.11110 [I 2022-10-10 22:33:48] ...optimizers/trijoint.py.151: total_norm: 8.197505 [I 2022-10-10 22:33:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 280/2304 [I 2022-10-10 22:33:56] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:58 [I 2022-10-10 22:33:56] ...trap/engines/engine.py.229: process: 0.84836 | load: 0.00296 [I 2022-10-10 22:33:56] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-10 22:33:56] ...optimizers/trijoint.py.151: total_norm: 18.415964 [I 2022-10-10 22:34:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 290/2304 [I 2022-10-10 22:34:05] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:42 [I 2022-10-10 22:34:05] ...trap/engines/engine.py.229: process: 0.86190 | load: 0.00322 [I 2022-10-10 22:34:05] ...trap/engines/engine.py.230: loss: 0.09505 [I 2022-10-10 22:34:05] ...optimizers/trijoint.py.151: total_norm: 12.841171 [I 2022-10-10 22:34:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 300/2304 [I 2022-10-10 22:34:14] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:37 [I 2022-10-10 22:34:14] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00296 [I 2022-10-10 22:34:14] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-10 22:34:14] ...optimizers/trijoint.py.151: total_norm: 15.591469 [I 2022-10-10 22:34:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 310/2304 [I 2022-10-10 22:34:22] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:37 [I 2022-10-10 22:34:22] ...trap/engines/engine.py.229: process: 0.86130 | load: 0.00316 [I 2022-10-10 22:34:22] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-10 22:34:22] ...optimizers/trijoint.py.151: total_norm: 10.741964 [I 2022-10-10 22:34:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 320/2304 [I 2022-10-10 22:34:31] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:30 [I 2022-10-10 22:34:31] ...trap/engines/engine.py.229: process: 0.86293 | load: 0.00308 [I 2022-10-10 22:34:31] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-10 22:34:31] ...optimizers/trijoint.py.151: total_norm: 23.919819 [I 2022-10-10 22:34:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 330/2304 [I 2022-10-10 22:34:39] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:26 [I 2022-10-10 22:34:39] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00311 [I 2022-10-10 22:34:39] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-10 22:34:39] ...optimizers/trijoint.py.151: total_norm: 15.155665 [I 2022-10-10 22:34:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 340/2304 [I 2022-10-10 22:34:48] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:19 [I 2022-10-10 22:34:48] ...trap/engines/engine.py.229: process: 0.87399 | load: 0.00311 [I 2022-10-10 22:34:48] ...trap/engines/engine.py.230: loss: 0.18437 [I 2022-10-10 22:34:48] ...optimizers/trijoint.py.151: total_norm: 10.075569 [I 2022-10-10 22:34:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 350/2304 [I 2022-10-10 22:34:57] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:28:27 [I 2022-10-10 22:34:57] ...trap/engines/engine.py.229: process: 0.88239 | load: 0.00307 [I 2022-10-10 22:34:57] ...trap/engines/engine.py.230: loss: 0.10192 [I 2022-10-10 22:34:57] ...optimizers/trijoint.py.151: total_norm: 13.866891 [I 2022-10-10 22:35:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 360/2304 [I 2022-10-10 22:35:06] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:57 [I 2022-10-10 22:35:06] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00312 [I 2022-10-10 22:35:06] ...trap/engines/engine.py.230: loss: 0.13070 [I 2022-10-10 22:35:06] ...optimizers/trijoint.py.151: total_norm: 13.655683 [I 2022-10-10 22:35:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 370/2304 [I 2022-10-10 22:35:14] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:56 [I 2022-10-10 22:35:14] ...trap/engines/engine.py.229: process: 0.85963 | load: 0.00312 [I 2022-10-10 22:35:14] ...trap/engines/engine.py.230: loss: 0.10824 [I 2022-10-10 22:35:14] ...optimizers/trijoint.py.151: total_norm: 15.222184 [I 2022-10-10 22:35:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 380/2304 [I 2022-10-10 22:35:23] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:27:41 [I 2022-10-10 22:35:23] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00309 [I 2022-10-10 22:35:23] ...trap/engines/engine.py.230: loss: 0.15757 [I 2022-10-10 22:35:23] ...optimizers/trijoint.py.151: total_norm: 51.079910 [I 2022-10-10 22:35:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 390/2304 [I 2022-10-10 22:35:32] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:31 [I 2022-10-10 22:35:32] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00309 [I 2022-10-10 22:35:32] ...trap/engines/engine.py.230: loss: 0.09809 [I 2022-10-10 22:35:32] ...optimizers/trijoint.py.151: total_norm: 14.001226 [I 2022-10-10 22:35:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 400/2304 [I 2022-10-10 22:35:40] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:27:27 [I 2022-10-10 22:35:40] ...trap/engines/engine.py.229: process: 0.87164 | load: 0.00329 [I 2022-10-10 22:35:40] ...trap/engines/engine.py.230: loss: 0.10963 [I 2022-10-10 22:35:40] ...optimizers/trijoint.py.151: total_norm: 18.906742 [I 2022-10-10 22:35:49] ...trap/engines/engine.py.224: train: epoch 88 | batch 410/2304 [I 2022-10-10 22:35:49] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:27:17 [I 2022-10-10 22:35:49] ...trap/engines/engine.py.229: process: 0.86153 | load: 0.00318 [I 2022-10-10 22:35:49] ...trap/engines/engine.py.230: loss: 0.12990 [I 2022-10-10 22:35:49] ...optimizers/trijoint.py.151: total_norm: 13.218833 [I 2022-10-10 22:35:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 420/2304 [I 2022-10-10 22:35:58] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:27:08 [I 2022-10-10 22:35:58] ...trap/engines/engine.py.229: process: 0.86898 | load: 0.00315 [I 2022-10-10 22:35:58] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-10 22:35:58] ...optimizers/trijoint.py.151: total_norm: 10.518245 [I 2022-10-10 22:36:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 430/2304 [I 2022-10-10 22:36:06] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:26:58 [I 2022-10-10 22:36:06] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00310 [I 2022-10-10 22:36:06] ...trap/engines/engine.py.230: loss: 0.14055 [I 2022-10-10 22:36:06] ...optimizers/trijoint.py.151: total_norm: 24.213154 [I 2022-10-10 22:36:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 440/2304 [I 2022-10-10 22:36:15] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:26:39 [I 2022-10-10 22:36:15] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00307 [I 2022-10-10 22:36:15] ...trap/engines/engine.py.230: loss: 0.23457 [I 2022-10-10 22:36:15] ...optimizers/trijoint.py.151: total_norm: 14.290708 [I 2022-10-10 22:36:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 450/2304 [I 2022-10-10 22:36:24] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:26:34 [I 2022-10-10 22:36:24] ...trap/engines/engine.py.229: process: 0.85093 | load: 0.00328 [I 2022-10-10 22:36:24] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-10 22:36:24] ...optimizers/trijoint.py.151: total_norm: 11.356656 [I 2022-10-10 22:36:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 460/2304 [I 2022-10-10 22:36:32] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:26:36 [I 2022-10-10 22:36:32] ...trap/engines/engine.py.229: process: 0.86801 | load: 0.00316 [I 2022-10-10 22:36:32] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-10 22:36:32] ...optimizers/trijoint.py.151: total_norm: 18.197708 [I 2022-10-10 22:36:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 470/2304 [I 2022-10-10 22:36:41] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:26:29 [I 2022-10-10 22:36:41] ...trap/engines/engine.py.229: process: 0.85986 | load: 0.00320 [I 2022-10-10 22:36:41] ...trap/engines/engine.py.230: loss: 0.17506 [I 2022-10-10 22:36:41] ...optimizers/trijoint.py.151: total_norm: 13.635915 [I 2022-10-10 22:36:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 480/2304 [I 2022-10-10 22:36:50] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:26:24 [I 2022-10-10 22:36:50] ...trap/engines/engine.py.229: process: 0.87985 | load: 0.00320 [I 2022-10-10 22:36:50] ...trap/engines/engine.py.230: loss: 0.14108 [I 2022-10-10 22:36:50] ...optimizers/trijoint.py.151: total_norm: 13.082578 [I 2022-10-10 22:36:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 490/2304 [I 2022-10-10 22:36:58] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:26:18 [I 2022-10-10 22:36:58] ...trap/engines/engine.py.229: process: 0.87126 | load: 0.00314 [I 2022-10-10 22:36:58] ...trap/engines/engine.py.230: loss: 0.08195 [I 2022-10-10 22:36:58] ...optimizers/trijoint.py.151: total_norm: 15.267919 [I 2022-10-10 22:37:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 500/2304 [I 2022-10-10 22:37:07] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:59 [I 2022-10-10 22:37:07] ...trap/engines/engine.py.229: process: 0.86514 | load: 0.00319 [I 2022-10-10 22:37:07] ...trap/engines/engine.py.230: loss: 0.08749 [I 2022-10-10 22:37:07] ...optimizers/trijoint.py.151: total_norm: 17.606173 [I 2022-10-10 22:37:16] ...trap/engines/engine.py.224: train: epoch 88 | batch 510/2304 [I 2022-10-10 22:37:16] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:54 [I 2022-10-10 22:37:16] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00325 [I 2022-10-10 22:37:16] ...trap/engines/engine.py.230: loss: 0.10260 [I 2022-10-10 22:37:16] ...optimizers/trijoint.py.151: total_norm: 18.607763 [I 2022-10-10 22:37:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 520/2304 [I 2022-10-10 22:37:24] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:26:02 [I 2022-10-10 22:37:24] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00312 [I 2022-10-10 22:37:24] ...trap/engines/engine.py.230: loss: 0.09793 [I 2022-10-10 22:37:24] ...optimizers/trijoint.py.151: total_norm: 62.070675 [I 2022-10-10 22:37:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 530/2304 [I 2022-10-10 22:37:33] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:25:39 [I 2022-10-10 22:37:33] ...trap/engines/engine.py.229: process: 0.87269 | load: 0.00311 [I 2022-10-10 22:37:33] ...trap/engines/engine.py.230: loss: 0.17446 [I 2022-10-10 22:37:33] ...optimizers/trijoint.py.151: total_norm: 12.949183 [I 2022-10-10 22:37:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 540/2304 [I 2022-10-10 22:37:42] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:22 [I 2022-10-10 22:37:42] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00305 [I 2022-10-10 22:37:42] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-10 22:37:42] ...optimizers/trijoint.py.151: total_norm: 12.235148 [I 2022-10-10 22:37:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 550/2304 [I 2022-10-10 22:37:51] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:07 [I 2022-10-10 22:37:51] ...trap/engines/engine.py.229: process: 0.86296 | load: 0.00311 [I 2022-10-10 22:37:51] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-10 22:37:51] ...optimizers/trijoint.py.151: total_norm: 10.712228 [I 2022-10-10 22:37:59] ...trap/engines/engine.py.224: train: epoch 88 | batch 560/2304 [I 2022-10-10 22:37:59] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:17 [I 2022-10-10 22:37:59] ...trap/engines/engine.py.229: process: 0.86327 | load: 0.00338 [I 2022-10-10 22:37:59] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-10 22:37:59] ...optimizers/trijoint.py.151: total_norm: 16.601450 [I 2022-10-10 22:38:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 570/2304 [I 2022-10-10 22:38:08] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:25:10 [I 2022-10-10 22:38:08] ...trap/engines/engine.py.229: process: 0.87896 | load: 0.00357 [I 2022-10-10 22:38:08] ...trap/engines/engine.py.230: loss: 0.13623 [I 2022-10-10 22:38:08] ...optimizers/trijoint.py.151: total_norm: 9.090430 [I 2022-10-10 22:38:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 580/2304 [I 2022-10-10 22:38:17] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:44 [I 2022-10-10 22:38:17] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00295 [I 2022-10-10 22:38:17] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-10 22:38:17] ...optimizers/trijoint.py.151: total_norm: 8.604370 [I 2022-10-10 22:38:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 590/2304 [I 2022-10-10 22:38:25] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:33 [I 2022-10-10 22:38:25] ...trap/engines/engine.py.229: process: 0.86850 | load: 0.00304 [I 2022-10-10 22:38:25] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-10 22:38:25] ...optimizers/trijoint.py.151: total_norm: 14.837486 [I 2022-10-10 22:38:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 600/2304 [I 2022-10-10 22:38:34] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:17 [I 2022-10-10 22:38:34] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00290 [I 2022-10-10 22:38:34] ...trap/engines/engine.py.230: loss: 0.17862 [I 2022-10-10 22:38:34] ...optimizers/trijoint.py.151: total_norm: 16.430044 [I 2022-10-10 22:38:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 610/2304 [I 2022-10-10 22:38:42] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:24:11 [I 2022-10-10 22:38:42] ...trap/engines/engine.py.229: process: 0.86760 | load: 0.00289 [I 2022-10-10 22:38:42] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-10 22:38:42] ...optimizers/trijoint.py.151: total_norm: 21.611292 [I 2022-10-10 22:38:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 620/2304 [I 2022-10-10 22:38:51] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:59 [I 2022-10-10 22:38:51] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00295 [I 2022-10-10 22:38:51] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-10 22:38:51] ...optimizers/trijoint.py.151: total_norm: 17.784327 [I 2022-10-10 22:39:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 630/2304 [I 2022-10-10 22:39:00] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:47 [I 2022-10-10 22:39:00] ...trap/engines/engine.py.229: process: 0.84982 | load: 0.00294 [I 2022-10-10 22:39:00] ...trap/engines/engine.py.230: loss: 0.12573 [I 2022-10-10 22:39:00] ...optimizers/trijoint.py.151: total_norm: 12.679399 [I 2022-10-10 22:39:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 640/2304 [I 2022-10-10 22:39:08] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:39 [I 2022-10-10 22:39:08] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00299 [I 2022-10-10 22:39:08] ...trap/engines/engine.py.230: loss: 0.14150 [I 2022-10-10 22:39:08] ...optimizers/trijoint.py.151: total_norm: 10.863766 [I 2022-10-10 22:39:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 650/2304 [I 2022-10-10 22:39:17] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:32 [I 2022-10-10 22:39:17] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00320 [I 2022-10-10 22:39:17] ...trap/engines/engine.py.230: loss: 0.11094 [I 2022-10-10 22:39:17] ...optimizers/trijoint.py.151: total_norm: 11.432288 [I 2022-10-10 22:39:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 660/2304 [I 2022-10-10 22:39:25] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:16 [I 2022-10-10 22:39:25] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00296 [I 2022-10-10 22:39:25] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-10 22:39:25] ...optimizers/trijoint.py.151: total_norm: 16.457159 [I 2022-10-10 22:39:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 670/2304 [I 2022-10-10 22:39:34] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:26 [I 2022-10-10 22:39:34] ...trap/engines/engine.py.229: process: 0.87416 | load: 0.00297 [I 2022-10-10 22:39:34] ...trap/engines/engine.py.230: loss: 0.12338 [I 2022-10-10 22:39:34] ...optimizers/trijoint.py.151: total_norm: 34.089874 [I 2022-10-10 22:39:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 680/2304 [I 2022-10-10 22:39:42] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:23:17 [I 2022-10-10 22:39:42] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00297 [I 2022-10-10 22:39:42] ...trap/engines/engine.py.230: loss: 0.09115 [I 2022-10-10 22:39:42] ...optimizers/trijoint.py.151: total_norm: 10.564492 [I 2022-10-10 22:39:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 690/2304 [I 2022-10-10 22:39:51] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:23:02 [I 2022-10-10 22:39:51] ...trap/engines/engine.py.229: process: 0.87202 | load: 0.00293 [I 2022-10-10 22:39:51] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-10 22:39:51] ...optimizers/trijoint.py.151: total_norm: 9.743067 [I 2022-10-10 22:40:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 700/2304 [I 2022-10-10 22:40:00] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:54 [I 2022-10-10 22:40:00] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00291 [I 2022-10-10 22:40:00] ...trap/engines/engine.py.230: loss: 0.10754 [I 2022-10-10 22:40:00] ...optimizers/trijoint.py.151: total_norm: 9.037111 [I 2022-10-10 22:40:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 710/2304 [I 2022-10-10 22:40:08] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:22:43 [I 2022-10-10 22:40:08] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00297 [I 2022-10-10 22:40:08] ...trap/engines/engine.py.230: loss: 0.16390 [I 2022-10-10 22:40:08] ...optimizers/trijoint.py.151: total_norm: 17.062191 [I 2022-10-10 22:40:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 720/2304 [I 2022-10-10 22:40:17] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:30 [I 2022-10-10 22:40:17] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00294 [I 2022-10-10 22:40:17] ...trap/engines/engine.py.230: loss: 0.13005 [I 2022-10-10 22:40:17] ...optimizers/trijoint.py.151: total_norm: 36.905857 [I 2022-10-10 22:40:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 730/2304 [I 2022-10-10 22:40:25] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:26 [I 2022-10-10 22:40:25] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00300 [I 2022-10-10 22:40:25] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-10 22:40:25] ...optimizers/trijoint.py.151: total_norm: 19.543854 [I 2022-10-10 22:40:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 740/2304 [I 2022-10-10 22:40:34] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:22:17 [I 2022-10-10 22:40:34] ...trap/engines/engine.py.229: process: 0.85813 | load: 0.00293 [I 2022-10-10 22:40:34] ...trap/engines/engine.py.230: loss: 0.08769 [I 2022-10-10 22:40:34] ...optimizers/trijoint.py.151: total_norm: 23.808786 [I 2022-10-10 22:40:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 750/2304 [I 2022-10-10 22:40:42] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:22:11 [I 2022-10-10 22:40:42] ...trap/engines/engine.py.229: process: 0.87479 | load: 0.00292 [I 2022-10-10 22:40:42] ...trap/engines/engine.py.230: loss: 0.19602 [I 2022-10-10 22:40:42] ...optimizers/trijoint.py.151: total_norm: 18.318289 [I 2022-10-10 22:40:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 760/2304 [I 2022-10-10 22:40:51] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:22:01 [I 2022-10-10 22:40:51] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00291 [I 2022-10-10 22:40:51] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-10 22:40:51] ...optimizers/trijoint.py.151: total_norm: 12.133703 [I 2022-10-10 22:41:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 770/2304 [I 2022-10-10 22:41:00] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:49 [I 2022-10-10 22:41:00] ...trap/engines/engine.py.229: process: 0.84305 | load: 0.00296 [I 2022-10-10 22:41:00] ...trap/engines/engine.py.230: loss: 0.22614 [I 2022-10-10 22:41:00] ...optimizers/trijoint.py.151: total_norm: 16.481432 [I 2022-10-10 22:41:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 780/2304 [I 2022-10-10 22:41:08] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:41 [I 2022-10-10 22:41:08] ...trap/engines/engine.py.229: process: 0.86033 | load: 0.00296 [I 2022-10-10 22:41:08] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-10 22:41:08] ...optimizers/trijoint.py.151: total_norm: 12.293677 [I 2022-10-10 22:41:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 790/2304 [I 2022-10-10 22:41:17] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:37 [I 2022-10-10 22:41:17] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00308 [I 2022-10-10 22:41:17] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-10 22:41:17] ...optimizers/trijoint.py.151: total_norm: 12.704230 [I 2022-10-10 22:41:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 800/2304 [I 2022-10-10 22:41:25] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:37 [I 2022-10-10 22:41:25] ...trap/engines/engine.py.229: process: 0.87096 | load: 0.00310 [I 2022-10-10 22:41:25] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-10 22:41:25] ...optimizers/trijoint.py.151: total_norm: 8.372623 [I 2022-10-10 22:41:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 810/2304 [I 2022-10-10 22:41:34] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:23 [I 2022-10-10 22:41:34] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00310 [I 2022-10-10 22:41:34] ...trap/engines/engine.py.230: loss: 0.12201 [I 2022-10-10 22:41:34] ...optimizers/trijoint.py.151: total_norm: 13.865909 [I 2022-10-10 22:41:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 820/2304 [I 2022-10-10 22:41:43] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:21:20 [I 2022-10-10 22:41:43] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00309 [I 2022-10-10 22:41:43] ...trap/engines/engine.py.230: loss: 0.10088 [I 2022-10-10 22:41:43] ...optimizers/trijoint.py.151: total_norm: 10.881558 [I 2022-10-10 22:41:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 830/2304 [I 2022-10-10 22:41:51] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:21:03 [I 2022-10-10 22:41:51] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00313 [I 2022-10-10 22:41:51] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-10 22:41:51] ...optimizers/trijoint.py.151: total_norm: 11.673189 [I 2022-10-10 22:42:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 840/2304 [I 2022-10-10 22:42:00] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:21:10 [I 2022-10-10 22:42:00] ...trap/engines/engine.py.229: process: 0.86802 | load: 0.00322 [I 2022-10-10 22:42:00] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-10 22:42:00] ...optimizers/trijoint.py.151: total_norm: 17.607403 [I 2022-10-10 22:42:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 850/2304 [I 2022-10-10 22:42:09] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:21:08 [I 2022-10-10 22:42:09] ...trap/engines/engine.py.229: process: 0.87098 | load: 0.00312 [I 2022-10-10 22:42:09] ...trap/engines/engine.py.230: loss: 0.18537 [I 2022-10-10 22:42:09] ...optimizers/trijoint.py.151: total_norm: 18.736065 [I 2022-10-10 22:42:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 860/2304 [I 2022-10-10 22:42:17] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:46 [I 2022-10-10 22:42:17] ...trap/engines/engine.py.229: process: 0.83954 | load: 0.00310 [I 2022-10-10 22:42:17] ...trap/engines/engine.py.230: loss: 0.13681 [I 2022-10-10 22:42:17] ...optimizers/trijoint.py.151: total_norm: 7.424709 [I 2022-10-10 22:42:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 870/2304 [I 2022-10-10 22:42:26] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:46 [I 2022-10-10 22:42:26] ...trap/engines/engine.py.229: process: 0.87217 | load: 0.00312 [I 2022-10-10 22:42:26] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-10 22:42:26] ...optimizers/trijoint.py.151: total_norm: 15.857195 [I 2022-10-10 22:42:35] ...trap/engines/engine.py.224: train: epoch 88 | batch 880/2304 [I 2022-10-10 22:42:35] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:29 [I 2022-10-10 22:42:35] ...trap/engines/engine.py.229: process: 0.86161 | load: 0.00311 [I 2022-10-10 22:42:35] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-10 22:42:35] ...optimizers/trijoint.py.151: total_norm: 9.168444 [I 2022-10-10 22:42:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 890/2304 [I 2022-10-10 22:42:43] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:33 [I 2022-10-10 22:42:43] ...trap/engines/engine.py.229: process: 0.89440 | load: 0.00327 [I 2022-10-10 22:42:43] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-10 22:42:43] ...optimizers/trijoint.py.151: total_norm: 14.623607 [I 2022-10-10 22:42:52] ...trap/engines/engine.py.224: train: epoch 88 | batch 900/2304 [I 2022-10-10 22:42:52] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:20:18 [I 2022-10-10 22:42:52] ...trap/engines/engine.py.229: process: 0.86678 | load: 0.00318 [I 2022-10-10 22:42:52] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-10 22:42:52] ...optimizers/trijoint.py.151: total_norm: 18.098553 [I 2022-10-10 22:43:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 910/2304 [I 2022-10-10 22:43:01] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:20:05 [I 2022-10-10 22:43:01] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00307 [I 2022-10-10 22:43:01] ...trap/engines/engine.py.230: loss: 0.09058 [I 2022-10-10 22:43:01] ...optimizers/trijoint.py.151: total_norm: 12.963746 [I 2022-10-10 22:43:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 920/2304 [I 2022-10-10 22:43:09] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:49 [I 2022-10-10 22:43:09] ...trap/engines/engine.py.229: process: 0.85537 | load: 0.00311 [I 2022-10-10 22:43:09] ...trap/engines/engine.py.230: loss: 0.15838 [I 2022-10-10 22:43:09] ...optimizers/trijoint.py.151: total_norm: 15.132140 [I 2022-10-10 22:43:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 930/2304 [I 2022-10-10 22:43:18] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:50 [I 2022-10-10 22:43:18] ...trap/engines/engine.py.229: process: 0.85428 | load: 0.00309 [I 2022-10-10 22:43:18] ...trap/engines/engine.py.230: loss: 0.14557 [I 2022-10-10 22:43:18] ...optimizers/trijoint.py.151: total_norm: 16.164522 [I 2022-10-10 22:43:27] ...trap/engines/engine.py.224: train: epoch 88 | batch 940/2304 [I 2022-10-10 22:43:27] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:35 [I 2022-10-10 22:43:27] ...trap/engines/engine.py.229: process: 0.86592 | load: 0.00308 [I 2022-10-10 22:43:27] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-10 22:43:27] ...optimizers/trijoint.py.151: total_norm: 16.461842 [I 2022-10-10 22:43:35] ...trap/engines/engine.py.224: train: epoch 88 | batch 950/2304 [I 2022-10-10 22:43:35] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:30 [I 2022-10-10 22:43:35] ...trap/engines/engine.py.229: process: 0.87373 | load: 0.00315 [I 2022-10-10 22:43:35] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-10 22:43:35] ...optimizers/trijoint.py.151: total_norm: 18.369526 [I 2022-10-10 22:43:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 960/2304 [I 2022-10-10 22:43:44] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:26 [I 2022-10-10 22:43:44] ...trap/engines/engine.py.229: process: 0.86992 | load: 0.00310 [I 2022-10-10 22:43:44] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-10 22:43:44] ...optimizers/trijoint.py.151: total_norm: 13.498755 [I 2022-10-10 22:43:53] ...trap/engines/engine.py.224: train: epoch 88 | batch 970/2304 [I 2022-10-10 22:43:53] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:19:15 [I 2022-10-10 22:43:53] ...trap/engines/engine.py.229: process: 0.87444 | load: 0.00316 [I 2022-10-10 22:43:53] ...trap/engines/engine.py.230: loss: 0.08208 [I 2022-10-10 22:43:53] ...optimizers/trijoint.py.151: total_norm: 9.624791 [I 2022-10-10 22:44:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 980/2304 [I 2022-10-10 22:44:01] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:19:07 [I 2022-10-10 22:44:01] ...trap/engines/engine.py.229: process: 0.87199 | load: 0.00310 [I 2022-10-10 22:44:01] ...trap/engines/engine.py.230: loss: 0.13622 [I 2022-10-10 22:44:01] ...optimizers/trijoint.py.151: total_norm: 21.885803 [I 2022-10-10 22:44:10] ...trap/engines/engine.py.224: train: epoch 88 | batch 990/2304 [I 2022-10-10 22:44:10] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:19:02 [I 2022-10-10 22:44:10] ...trap/engines/engine.py.229: process: 0.86753 | load: 0.00316 [I 2022-10-10 22:44:10] ...trap/engines/engine.py.230: loss: 0.14669 [I 2022-10-10 22:44:10] ...optimizers/trijoint.py.151: total_norm: 29.611734 [I 2022-10-10 22:44:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 1000/2304 [I 2022-10-10 22:44:19] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:55 [I 2022-10-10 22:44:19] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00318 [I 2022-10-10 22:44:19] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-10 22:44:19] ...optimizers/trijoint.py.151: total_norm: 21.952518 [I 2022-10-10 22:44:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1010/2304 [I 2022-10-10 22:44:28] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:38 [I 2022-10-10 22:44:28] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00307 [I 2022-10-10 22:44:28] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-10 22:44:28] ...optimizers/trijoint.py.151: total_norm: 8.487437 [I 2022-10-10 22:44:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 1020/2304 [I 2022-10-10 22:44:36] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:26 [I 2022-10-10 22:44:36] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00338 [I 2022-10-10 22:44:36] ...trap/engines/engine.py.230: loss: 0.12076 [I 2022-10-10 22:44:36] ...optimizers/trijoint.py.151: total_norm: 16.116602 [I 2022-10-10 22:44:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1030/2304 [I 2022-10-10 22:44:45] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:18:24 [I 2022-10-10 22:44:45] ...trap/engines/engine.py.229: process: 0.86308 | load: 0.00306 [I 2022-10-10 22:44:45] ...trap/engines/engine.py.230: loss: 0.09574 [I 2022-10-10 22:44:45] ...optimizers/trijoint.py.151: total_norm: 11.800121 [I 2022-10-10 22:44:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1040/2304 [I 2022-10-10 22:44:54] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:18:20 [I 2022-10-10 22:44:54] ...trap/engines/engine.py.229: process: 0.86930 | load: 0.00308 [I 2022-10-10 22:44:54] ...trap/engines/engine.py.230: loss: 0.08948 [I 2022-10-10 22:44:54] ...optimizers/trijoint.py.151: total_norm: 9.443997 [I 2022-10-10 22:45:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 1050/2304 [I 2022-10-10 22:45:02] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:18:05 [I 2022-10-10 22:45:02] ...trap/engines/engine.py.229: process: 0.87012 | load: 0.00308 [I 2022-10-10 22:45:02] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-10 22:45:02] ...optimizers/trijoint.py.151: total_norm: 13.716663 [I 2022-10-10 22:45:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1060/2304 [I 2022-10-10 22:45:11] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:59 [I 2022-10-10 22:45:11] ...trap/engines/engine.py.229: process: 0.87115 | load: 0.00305 [I 2022-10-10 22:45:11] ...trap/engines/engine.py.230: loss: 0.13478 [I 2022-10-10 22:45:11] ...optimizers/trijoint.py.151: total_norm: 21.739363 [I 2022-10-10 22:45:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 1070/2304 [I 2022-10-10 22:45:20] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:37 [I 2022-10-10 22:45:20] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00294 [I 2022-10-10 22:45:20] ...trap/engines/engine.py.230: loss: 0.09463 [I 2022-10-10 22:45:20] ...optimizers/trijoint.py.151: total_norm: 23.360649 [I 2022-10-10 22:45:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1080/2304 [I 2022-10-10 22:45:28] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:27 [I 2022-10-10 22:45:28] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00292 [I 2022-10-10 22:45:28] ...trap/engines/engine.py.230: loss: 0.16969 [I 2022-10-10 22:45:28] ...optimizers/trijoint.py.151: total_norm: 9.185719 [I 2022-10-10 22:45:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 1090/2304 [I 2022-10-10 22:45:37] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:17 [I 2022-10-10 22:45:37] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00292 [I 2022-10-10 22:45:37] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-10 22:45:37] ...optimizers/trijoint.py.151: total_norm: 12.717912 [I 2022-10-10 22:45:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1100/2304 [I 2022-10-10 22:45:45] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:17:03 [I 2022-10-10 22:45:45] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00291 [I 2022-10-10 22:45:45] ...trap/engines/engine.py.230: loss: 0.11942 [I 2022-10-10 22:45:45] ...optimizers/trijoint.py.151: total_norm: 7.558377 [I 2022-10-10 22:45:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1110/2304 [I 2022-10-10 22:45:54] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:17:03 [I 2022-10-10 22:45:54] ...trap/engines/engine.py.229: process: 0.86222 | load: 0.00292 [I 2022-10-10 22:45:54] ...trap/engines/engine.py.230: loss: 0.20592 [I 2022-10-10 22:45:54] ...optimizers/trijoint.py.151: total_norm: 18.530991 [I 2022-10-10 22:46:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 1120/2304 [I 2022-10-10 22:46:02] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:52 [I 2022-10-10 22:46:02] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00290 [I 2022-10-10 22:46:02] ...trap/engines/engine.py.230: loss: 0.13036 [I 2022-10-10 22:46:02] ...optimizers/trijoint.py.151: total_norm: 23.109360 [I 2022-10-10 22:46:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1130/2304 [I 2022-10-10 22:46:11] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:42 [I 2022-10-10 22:46:11] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00304 [I 2022-10-10 22:46:11] ...trap/engines/engine.py.230: loss: 0.09875 [I 2022-10-10 22:46:11] ...optimizers/trijoint.py.151: total_norm: 13.093781 [I 2022-10-10 22:46:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 1140/2304 [I 2022-10-10 22:46:20] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:34 [I 2022-10-10 22:46:20] ...trap/engines/engine.py.229: process: 0.85954 | load: 0.00291 [I 2022-10-10 22:46:20] ...trap/engines/engine.py.230: loss: 0.15920 [I 2022-10-10 22:46:20] ...optimizers/trijoint.py.151: total_norm: 28.327497 [I 2022-10-10 22:46:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1150/2304 [I 2022-10-10 22:46:28] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:24 [I 2022-10-10 22:46:28] ...trap/engines/engine.py.229: process: 0.86175 | load: 0.00295 [I 2022-10-10 22:46:28] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-10 22:46:28] ...optimizers/trijoint.py.151: total_norm: 11.910058 [I 2022-10-10 22:46:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 1160/2304 [I 2022-10-10 22:46:37] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:15 [I 2022-10-10 22:46:37] ...trap/engines/engine.py.229: process: 0.85347 | load: 0.00295 [I 2022-10-10 22:46:37] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-10 22:46:37] ...optimizers/trijoint.py.151: total_norm: 23.402977 [I 2022-10-10 22:46:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1170/2304 [I 2022-10-10 22:46:45] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:02 [I 2022-10-10 22:46:45] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00291 [I 2022-10-10 22:46:45] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-10 22:46:45] ...optimizers/trijoint.py.151: total_norm: 23.457449 [I 2022-10-10 22:46:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1180/2304 [I 2022-10-10 22:46:54] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:15:56 [I 2022-10-10 22:46:54] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00294 [I 2022-10-10 22:46:54] ...trap/engines/engine.py.230: loss: 0.20558 [I 2022-10-10 22:46:54] ...optimizers/trijoint.py.151: total_norm: 35.144344 [I 2022-10-10 22:47:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 1190/2304 [I 2022-10-10 22:47:02] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:52 [I 2022-10-10 22:47:02] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00291 [I 2022-10-10 22:47:02] ...trap/engines/engine.py.230: loss: 0.10602 [I 2022-10-10 22:47:02] ...optimizers/trijoint.py.151: total_norm: 10.637761 [I 2022-10-10 22:47:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1200/2304 [I 2022-10-10 22:47:11] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:39 [I 2022-10-10 22:47:11] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00292 [I 2022-10-10 22:47:11] ...trap/engines/engine.py.230: loss: 0.12411 [I 2022-10-10 22:47:11] ...optimizers/trijoint.py.151: total_norm: 26.839821 [I 2022-10-10 22:47:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 1210/2304 [I 2022-10-10 22:47:19] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:31 [I 2022-10-10 22:47:19] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00290 [I 2022-10-10 22:47:19] ...trap/engines/engine.py.230: loss: 0.21094 [I 2022-10-10 22:47:19] ...optimizers/trijoint.py.151: total_norm: 27.423923 [I 2022-10-10 22:47:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1220/2304 [I 2022-10-10 22:47:28] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:28 [I 2022-10-10 22:47:28] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00292 [I 2022-10-10 22:47:28] ...trap/engines/engine.py.230: loss: 0.14160 [I 2022-10-10 22:47:28] ...optimizers/trijoint.py.151: total_norm: 20.997475 [I 2022-10-10 22:47:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 1230/2304 [I 2022-10-10 22:47:36] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:15 [I 2022-10-10 22:47:36] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00294 [I 2022-10-10 22:47:36] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-10 22:47:36] ...optimizers/trijoint.py.151: total_norm: 18.827799 [I 2022-10-10 22:47:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1240/2304 [I 2022-10-10 22:47:45] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:06 [I 2022-10-10 22:47:45] ...trap/engines/engine.py.229: process: 0.85607 | load: 0.00293 [I 2022-10-10 22:47:45] ...trap/engines/engine.py.230: loss: 0.15458 [I 2022-10-10 22:47:45] ...optimizers/trijoint.py.151: total_norm: 19.828648 [I 2022-10-10 22:47:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1250/2304 [I 2022-10-10 22:47:54] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:00 [I 2022-10-10 22:47:54] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00292 [I 2022-10-10 22:47:54] ...trap/engines/engine.py.230: loss: 0.11746 [I 2022-10-10 22:47:54] ...optimizers/trijoint.py.151: total_norm: 15.833493 [I 2022-10-10 22:48:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 1260/2304 [I 2022-10-10 22:48:02] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:51 [I 2022-10-10 22:48:02] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00301 [I 2022-10-10 22:48:02] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-10 22:48:02] ...optimizers/trijoint.py.151: total_norm: 13.968847 [I 2022-10-10 22:48:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1270/2304 [I 2022-10-10 22:48:11] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:40 [I 2022-10-10 22:48:11] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00296 [I 2022-10-10 22:48:11] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-10 22:48:11] ...optimizers/trijoint.py.151: total_norm: 9.776981 [I 2022-10-10 22:48:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 1280/2304 [I 2022-10-10 22:48:19] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:33 [I 2022-10-10 22:48:19] ...trap/engines/engine.py.229: process: 0.85517 | load: 0.00305 [I 2022-10-10 22:48:19] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-10 22:48:19] ...optimizers/trijoint.py.151: total_norm: 6.497420 [I 2022-10-10 22:48:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1290/2304 [I 2022-10-10 22:48:28] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:23 [I 2022-10-10 22:48:28] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00297 [I 2022-10-10 22:48:28] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-10 22:48:28] ...optimizers/trijoint.py.151: total_norm: 12.461843 [I 2022-10-10 22:48:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 1300/2304 [I 2022-10-10 22:48:36] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:14:17 [I 2022-10-10 22:48:36] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00297 [I 2022-10-10 22:48:36] ...trap/engines/engine.py.230: loss: 0.23026 [I 2022-10-10 22:48:36] ...optimizers/trijoint.py.151: total_norm: 26.474190 [I 2022-10-10 22:48:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1310/2304 [I 2022-10-10 22:48:45] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:14:10 [I 2022-10-10 22:48:45] ...trap/engines/engine.py.229: process: 0.86102 | load: 0.00295 [I 2022-10-10 22:48:45] ...trap/engines/engine.py.230: loss: 0.10399 [I 2022-10-10 22:48:45] ...optimizers/trijoint.py.151: total_norm: 13.629375 [I 2022-10-10 22:48:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1320/2304 [I 2022-10-10 22:48:54] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:08 [I 2022-10-10 22:48:54] ...trap/engines/engine.py.229: process: 0.86132 | load: 0.00306 [I 2022-10-10 22:48:54] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-10 22:48:54] ...optimizers/trijoint.py.151: total_norm: 12.333956 [I 2022-10-10 22:49:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 1330/2304 [I 2022-10-10 22:49:02] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:14:06 [I 2022-10-10 22:49:02] ...trap/engines/engine.py.229: process: 0.88114 | load: 0.00314 [I 2022-10-10 22:49:02] ...trap/engines/engine.py.230: loss: 0.15452 [I 2022-10-10 22:49:02] ...optimizers/trijoint.py.151: total_norm: 18.847107 [I 2022-10-10 22:49:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1340/2304 [I 2022-10-10 22:49:11] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:52 [I 2022-10-10 22:49:11] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00311 [I 2022-10-10 22:49:11] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-10 22:49:11] ...optimizers/trijoint.py.151: total_norm: 18.419764 [I 2022-10-10 22:49:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 1350/2304 [I 2022-10-10 22:49:20] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:47 [I 2022-10-10 22:49:20] ...trap/engines/engine.py.229: process: 0.85774 | load: 0.00315 [I 2022-10-10 22:49:20] ...trap/engines/engine.py.230: loss: 0.10479 [I 2022-10-10 22:49:20] ...optimizers/trijoint.py.151: total_norm: 15.967673 [I 2022-10-10 22:49:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1360/2304 [I 2022-10-10 22:49:28] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:34 [I 2022-10-10 22:49:28] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00310 [I 2022-10-10 22:49:28] ...trap/engines/engine.py.230: loss: 0.10648 [I 2022-10-10 22:49:28] ...optimizers/trijoint.py.151: total_norm: 10.756224 [I 2022-10-10 22:49:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 1370/2304 [I 2022-10-10 22:49:37] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:13:30 [I 2022-10-10 22:49:37] ...trap/engines/engine.py.229: process: 0.87427 | load: 0.00318 [I 2022-10-10 22:49:37] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-10 22:49:37] ...optimizers/trijoint.py.151: total_norm: 11.213781 [I 2022-10-10 22:49:46] ...trap/engines/engine.py.224: train: epoch 88 | batch 1380/2304 [I 2022-10-10 22:49:46] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:23 [I 2022-10-10 22:49:46] ...trap/engines/engine.py.229: process: 0.86478 | load: 0.00321 [I 2022-10-10 22:49:46] ...trap/engines/engine.py.230: loss: 0.18194 [I 2022-10-10 22:49:46] ...optimizers/trijoint.py.151: total_norm: 14.590367 [I 2022-10-10 22:49:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 1390/2304 [I 2022-10-10 22:49:55] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:15 [I 2022-10-10 22:49:55] ...trap/engines/engine.py.229: process: 0.87129 | load: 0.00318 [I 2022-10-10 22:49:55] ...trap/engines/engine.py.230: loss: 0.15364 [I 2022-10-10 22:49:55] ...optimizers/trijoint.py.151: total_norm: 22.729958 [I 2022-10-10 22:50:03] ...trap/engines/engine.py.224: train: epoch 88 | batch 1400/2304 [I 2022-10-10 22:50:03] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:13:04 [I 2022-10-10 22:50:03] ...trap/engines/engine.py.229: process: 0.87310 | load: 0.00309 [I 2022-10-10 22:50:03] ...trap/engines/engine.py.230: loss: 0.20509 [I 2022-10-10 22:50:03] ...optimizers/trijoint.py.151: total_norm: 13.235241 [I 2022-10-10 22:50:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 1410/2304 [I 2022-10-10 22:50:12] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:59 [I 2022-10-10 22:50:12] ...trap/engines/engine.py.229: process: 0.87554 | load: 0.00311 [I 2022-10-10 22:50:12] ...trap/engines/engine.py.230: loss: 0.09199 [I 2022-10-10 22:50:12] ...optimizers/trijoint.py.151: total_norm: 12.673738 [I 2022-10-10 22:50:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 1420/2304 [I 2022-10-10 22:50:21] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:51 [I 2022-10-10 22:50:21] ...trap/engines/engine.py.229: process: 0.87311 | load: 0.00415 [I 2022-10-10 22:50:21] ...trap/engines/engine.py.230: loss: 0.13803 [I 2022-10-10 22:50:21] ...optimizers/trijoint.py.151: total_norm: 38.430279 [I 2022-10-10 22:50:29] ...trap/engines/engine.py.224: train: epoch 88 | batch 1430/2304 [I 2022-10-10 22:50:29] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:38 [I 2022-10-10 22:50:29] ...trap/engines/engine.py.229: process: 0.87465 | load: 0.00308 [I 2022-10-10 22:50:29] ...trap/engines/engine.py.230: loss: 0.17078 [I 2022-10-10 22:50:29] ...optimizers/trijoint.py.151: total_norm: 18.548965 [I 2022-10-10 22:50:38] ...trap/engines/engine.py.224: train: epoch 88 | batch 1440/2304 [I 2022-10-10 22:50:38] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:25 [I 2022-10-10 22:50:38] ...trap/engines/engine.py.229: process: 0.88178 | load: 0.00321 [I 2022-10-10 22:50:38] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-10 22:50:38] ...optimizers/trijoint.py.151: total_norm: 19.279274 [I 2022-10-10 22:50:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 1450/2304 [I 2022-10-10 22:50:47] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:22 [I 2022-10-10 22:50:47] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00321 [I 2022-10-10 22:50:47] ...trap/engines/engine.py.230: loss: 0.12164 [I 2022-10-10 22:50:47] ...optimizers/trijoint.py.151: total_norm: 15.901358 [I 2022-10-10 22:50:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 1460/2304 [I 2022-10-10 22:50:55] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:12:11 [I 2022-10-10 22:50:55] ...trap/engines/engine.py.229: process: 0.86877 | load: 0.00309 [I 2022-10-10 22:50:55] ...trap/engines/engine.py.230: loss: 0.07592 [I 2022-10-10 22:50:55] ...optimizers/trijoint.py.151: total_norm: 11.069267 [I 2022-10-10 22:51:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 1470/2304 [I 2022-10-10 22:51:04] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:12:06 [I 2022-10-10 22:51:04] ...trap/engines/engine.py.229: process: 0.87057 | load: 0.00310 [I 2022-10-10 22:51:04] ...trap/engines/engine.py.230: loss: 0.19820 [I 2022-10-10 22:51:04] ...optimizers/trijoint.py.151: total_norm: 22.611712 [I 2022-10-10 22:51:13] ...trap/engines/engine.py.224: train: epoch 88 | batch 1480/2304 [I 2022-10-10 22:51:13] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:51 [I 2022-10-10 22:51:13] ...trap/engines/engine.py.229: process: 0.85662 | load: 0.00312 [I 2022-10-10 22:51:13] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-10 22:51:13] ...optimizers/trijoint.py.151: total_norm: 10.627292 [I 2022-10-10 22:51:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 1490/2304 [I 2022-10-10 22:51:21] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:41 [I 2022-10-10 22:51:21] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00314 [I 2022-10-10 22:51:21] ...trap/engines/engine.py.230: loss: 0.14197 [I 2022-10-10 22:51:21] ...optimizers/trijoint.py.151: total_norm: 15.599172 [I 2022-10-10 22:51:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 1500/2304 [I 2022-10-10 22:51:30] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:11:36 [I 2022-10-10 22:51:30] ...trap/engines/engine.py.229: process: 0.86806 | load: 0.00401 [I 2022-10-10 22:51:30] ...trap/engines/engine.py.230: loss: 0.08879 [I 2022-10-10 22:51:30] ...optimizers/trijoint.py.151: total_norm: 11.021831 [I 2022-10-10 22:51:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 1510/2304 [I 2022-10-10 22:51:39] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:26 [I 2022-10-10 22:51:39] ...trap/engines/engine.py.229: process: 0.87658 | load: 0.00312 [I 2022-10-10 22:51:39] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-10 22:51:39] ...optimizers/trijoint.py.151: total_norm: 26.342455 [I 2022-10-10 22:51:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 1520/2304 [I 2022-10-10 22:51:47] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:11:19 [I 2022-10-10 22:51:47] ...trap/engines/engine.py.229: process: 0.86880 | load: 0.00319 [I 2022-10-10 22:51:47] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-10 22:51:47] ...optimizers/trijoint.py.151: total_norm: 13.979962 [I 2022-10-10 22:51:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 1530/2304 [I 2022-10-10 22:51:56] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:08 [I 2022-10-10 22:51:56] ...trap/engines/engine.py.229: process: 0.87377 | load: 0.00310 [I 2022-10-10 22:51:56] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-10 22:51:56] ...optimizers/trijoint.py.151: total_norm: 8.763336 [I 2022-10-10 22:52:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1540/2304 [I 2022-10-10 22:52:05] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:11:03 [I 2022-10-10 22:52:05] ...trap/engines/engine.py.229: process: 0.87485 | load: 0.00322 [I 2022-10-10 22:52:05] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-10 22:52:05] ...optimizers/trijoint.py.151: total_norm: 20.984083 [I 2022-10-10 22:52:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 1550/2304 [I 2022-10-10 22:52:14] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:11:00 [I 2022-10-10 22:52:14] ...trap/engines/engine.py.229: process: 0.89529 | load: 0.00312 [I 2022-10-10 22:52:14] ...trap/engines/engine.py.230: loss: 0.16467 [I 2022-10-10 22:52:14] ...optimizers/trijoint.py.151: total_norm: 16.781738 [I 2022-10-10 22:52:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1560/2304 [I 2022-10-10 22:52:22] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:43 [I 2022-10-10 22:52:22] ...trap/engines/engine.py.229: process: 0.86413 | load: 0.00887 [I 2022-10-10 22:52:22] ...trap/engines/engine.py.230: loss: 0.15478 [I 2022-10-10 22:52:22] ...optimizers/trijoint.py.151: total_norm: 23.188211 [I 2022-10-10 22:52:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 1570/2304 [I 2022-10-10 22:52:31] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:38 [I 2022-10-10 22:52:31] ...trap/engines/engine.py.229: process: 0.88127 | load: 0.00311 [I 2022-10-10 22:52:31] ...trap/engines/engine.py.230: loss: 0.12975 [I 2022-10-10 22:52:31] ...optimizers/trijoint.py.151: total_norm: 13.556815 [I 2022-10-10 22:52:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 1580/2304 [I 2022-10-10 22:52:40] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:24 [I 2022-10-10 22:52:40] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00325 [I 2022-10-10 22:52:40] ...trap/engines/engine.py.230: loss: 0.15944 [I 2022-10-10 22:52:40] ...optimizers/trijoint.py.151: total_norm: 11.056056 [I 2022-10-10 22:52:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 1590/2304 [I 2022-10-10 22:52:48] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:08 [I 2022-10-10 22:52:48] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00295 [I 2022-10-10 22:52:48] ...trap/engines/engine.py.230: loss: 0.10378 [I 2022-10-10 22:52:48] ...optimizers/trijoint.py.151: total_norm: 54.607685 [I 2022-10-10 22:52:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 1600/2304 [I 2022-10-10 22:52:57] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:10:03 [I 2022-10-10 22:52:57] ...trap/engines/engine.py.229: process: 0.83466 | load: 0.00296 [I 2022-10-10 22:52:57] ...trap/engines/engine.py.230: loss: 0.14096 [I 2022-10-10 22:52:57] ...optimizers/trijoint.py.151: total_norm: 12.007820 [I 2022-10-10 22:53:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1610/2304 [I 2022-10-10 22:53:05] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:52 [I 2022-10-10 22:53:05] ...trap/engines/engine.py.229: process: 0.84987 | load: 0.00293 [I 2022-10-10 22:53:05] ...trap/engines/engine.py.230: loss: 0.08042 [I 2022-10-10 22:53:05] ...optimizers/trijoint.py.151: total_norm: 9.617908 [I 2022-10-10 22:53:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 1620/2304 [I 2022-10-10 22:53:14] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:43 [I 2022-10-10 22:53:14] ...trap/engines/engine.py.229: process: 0.84434 | load: 0.00293 [I 2022-10-10 22:53:14] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-10 22:53:14] ...optimizers/trijoint.py.151: total_norm: 7.615902 [I 2022-10-10 22:53:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1630/2304 [I 2022-10-10 22:53:22] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:38 [I 2022-10-10 22:53:22] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00296 [I 2022-10-10 22:53:22] ...trap/engines/engine.py.230: loss: 0.14939 [I 2022-10-10 22:53:22] ...optimizers/trijoint.py.151: total_norm: 11.720811 [I 2022-10-10 22:53:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 1640/2304 [I 2022-10-10 22:53:31] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:31 [I 2022-10-10 22:53:31] ...trap/engines/engine.py.229: process: 0.86507 | load: 0.00295 [I 2022-10-10 22:53:31] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-10 22:53:31] ...optimizers/trijoint.py.151: total_norm: 16.569536 [I 2022-10-10 22:53:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 1650/2304 [I 2022-10-10 22:53:40] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:17 [I 2022-10-10 22:53:40] ...trap/engines/engine.py.229: process: 0.84570 | load: 0.00299 [I 2022-10-10 22:53:40] ...trap/engines/engine.py.230: loss: 0.15342 [I 2022-10-10 22:53:40] ...optimizers/trijoint.py.151: total_norm: 15.913172 [I 2022-10-10 22:53:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 1660/2304 [I 2022-10-10 22:53:48] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:09:10 [I 2022-10-10 22:53:48] ...trap/engines/engine.py.229: process: 0.85673 | load: 0.00300 [I 2022-10-10 22:53:48] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-10 22:53:48] ...optimizers/trijoint.py.151: total_norm: 31.314476 [I 2022-10-10 22:53:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 1670/2304 [I 2022-10-10 22:53:57] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:09:00 [I 2022-10-10 22:53:57] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00299 [I 2022-10-10 22:53:57] ...trap/engines/engine.py.230: loss: 0.09802 [I 2022-10-10 22:53:57] ...optimizers/trijoint.py.151: total_norm: 25.930315 [I 2022-10-10 22:54:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1680/2304 [I 2022-10-10 22:54:05] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:54 [I 2022-10-10 22:54:05] ...trap/engines/engine.py.229: process: 0.86305 | load: 0.00296 [I 2022-10-10 22:54:05] ...trap/engines/engine.py.230: loss: 0.08512 [I 2022-10-10 22:54:05] ...optimizers/trijoint.py.151: total_norm: 12.507822 [I 2022-10-10 22:54:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 1690/2304 [I 2022-10-10 22:54:14] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:43 [I 2022-10-10 22:54:14] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00292 [I 2022-10-10 22:54:14] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-10 22:54:14] ...optimizers/trijoint.py.151: total_norm: 15.527707 [I 2022-10-10 22:54:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1700/2304 [I 2022-10-10 22:54:22] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:35 [I 2022-10-10 22:54:22] ...trap/engines/engine.py.229: process: 0.86203 | load: 0.00294 [I 2022-10-10 22:54:22] ...trap/engines/engine.py.230: loss: 0.11168 [I 2022-10-10 22:54:22] ...optimizers/trijoint.py.151: total_norm: 11.457160 [I 2022-10-10 22:54:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 1710/2304 [I 2022-10-10 22:54:31] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:25 [I 2022-10-10 22:54:31] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00293 [I 2022-10-10 22:54:31] ...trap/engines/engine.py.230: loss: 0.09286 [I 2022-10-10 22:54:31] ...optimizers/trijoint.py.151: total_norm: 36.750591 [I 2022-10-10 22:54:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 1720/2304 [I 2022-10-10 22:54:39] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:19 [I 2022-10-10 22:54:39] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00298 [I 2022-10-10 22:54:39] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-10 22:54:39] ...optimizers/trijoint.py.151: total_norm: 43.706825 [I 2022-10-10 22:54:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 1730/2304 [I 2022-10-10 22:54:48] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:08:12 [I 2022-10-10 22:54:48] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00307 [I 2022-10-10 22:54:48] ...trap/engines/engine.py.230: loss: 0.16697 [I 2022-10-10 22:54:48] ...optimizers/trijoint.py.151: total_norm: 28.176706 [I 2022-10-10 22:55:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1740/2304 [I 2022-10-10 22:55:15] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:19:37 [I 2022-10-10 22:55:15] ...trap/engines/engine.py.229: process: 0.86125 | load: 0.00297 [I 2022-10-10 22:55:15] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-10 22:55:15] ...optimizers/trijoint.py.151: total_norm: 18.334028 [I 2022-10-10 22:55:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 1750/2304 [I 2022-10-10 22:55:24] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:09:06 [I 2022-10-10 22:55:24] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00295 [I 2022-10-10 22:55:24] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-10 22:55:24] ...optimizers/trijoint.py.151: total_norm: 16.918629 [I 2022-10-10 22:55:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1760/2304 [I 2022-10-10 22:55:33] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:50 [I 2022-10-10 22:55:33] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00298 [I 2022-10-10 22:55:33] ...trap/engines/engine.py.230: loss: 0.13795 [I 2022-10-10 22:55:33] ...optimizers/trijoint.py.151: total_norm: 24.525484 [I 2022-10-10 22:55:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 1770/2304 [I 2022-10-10 22:55:41] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:33 [I 2022-10-10 22:55:41] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00308 [I 2022-10-10 22:55:41] ...trap/engines/engine.py.230: loss: 0.20538 [I 2022-10-10 22:55:41] ...optimizers/trijoint.py.151: total_norm: 14.849036 [I 2022-10-10 22:55:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 1780/2304 [I 2022-10-10 22:55:50] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:07:23 [I 2022-10-10 22:55:50] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00291 [I 2022-10-10 22:55:50] ...trap/engines/engine.py.230: loss: 0.16305 [I 2022-10-10 22:55:50] ...optimizers/trijoint.py.151: total_norm: 17.032595 [I 2022-10-10 22:55:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 1790/2304 [I 2022-10-10 22:55:58] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:07:17 [I 2022-10-10 22:55:58] ...trap/engines/engine.py.229: process: 0.84013 | load: 0.00295 [I 2022-10-10 22:55:58] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-10 22:55:58] ...optimizers/trijoint.py.151: total_norm: 9.542562 [I 2022-10-10 22:56:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 1800/2304 [I 2022-10-10 22:56:07] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:07:07 [I 2022-10-10 22:56:07] ...trap/engines/engine.py.229: process: 0.85198 | load: 0.00291 [I 2022-10-10 22:56:07] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-10 22:56:07] ...optimizers/trijoint.py.151: total_norm: 10.964008 [I 2022-10-10 22:56:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1810/2304 [I 2022-10-10 22:56:15] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:07:01 [I 2022-10-10 22:56:15] ...trap/engines/engine.py.229: process: 0.84603 | load: 0.00293 [I 2022-10-10 22:56:15] ...trap/engines/engine.py.230: loss: 0.09840 [I 2022-10-10 22:56:15] ...optimizers/trijoint.py.151: total_norm: 11.758237 [I 2022-10-10 22:56:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 1820/2304 [I 2022-10-10 22:56:24] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:50 [I 2022-10-10 22:56:24] ...trap/engines/engine.py.229: process: 0.82941 | load: 0.00294 [I 2022-10-10 22:56:24] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-10 22:56:24] ...optimizers/trijoint.py.151: total_norm: 12.153594 [I 2022-10-10 22:56:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 1830/2304 [I 2022-10-10 22:56:32] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:42 [I 2022-10-10 22:56:32] ...trap/engines/engine.py.229: process: 0.86640 | load: 0.00297 [I 2022-10-10 22:56:32] ...trap/engines/engine.py.230: loss: 0.15264 [I 2022-10-10 22:56:32] ...optimizers/trijoint.py.151: total_norm: 29.655935 [I 2022-10-10 22:56:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 1840/2304 [I 2022-10-10 22:56:41] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:37 [I 2022-10-10 22:56:41] ...trap/engines/engine.py.229: process: 0.87747 | load: 0.00325 [I 2022-10-10 22:56:41] ...trap/engines/engine.py.230: loss: 0.11700 [I 2022-10-10 22:56:41] ...optimizers/trijoint.py.151: total_norm: 14.826015 [I 2022-10-10 22:56:49] ...trap/engines/engine.py.224: train: epoch 88 | batch 1850/2304 [I 2022-10-10 22:56:49] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:06:29 [I 2022-10-10 22:56:49] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00310 [I 2022-10-10 22:56:49] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-10 22:56:49] ...optimizers/trijoint.py.151: total_norm: 14.363529 [I 2022-10-10 22:56:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 1860/2304 [I 2022-10-10 22:56:58] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:06:23 [I 2022-10-10 22:56:58] ...trap/engines/engine.py.229: process: 0.87273 | load: 0.00318 [I 2022-10-10 22:56:58] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-10 22:56:58] ...optimizers/trijoint.py.151: total_norm: 16.189075 [I 2022-10-10 22:57:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 1870/2304 [I 2022-10-10 22:57:07] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:06:14 [I 2022-10-10 22:57:07] ...trap/engines/engine.py.229: process: 0.86480 | load: 0.00315 [I 2022-10-10 22:57:07] ...trap/engines/engine.py.230: loss: 0.13684 [I 2022-10-10 22:57:07] ...optimizers/trijoint.py.151: total_norm: 10.192534 [I 2022-10-10 22:57:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1880/2304 [I 2022-10-10 22:57:15] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:06:06 [I 2022-10-10 22:57:15] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00306 [I 2022-10-10 22:57:15] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-10 22:57:15] ...optimizers/trijoint.py.151: total_norm: 9.801894 [I 2022-10-10 22:57:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 1890/2304 [I 2022-10-10 22:57:24] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:59 [I 2022-10-10 22:57:24] ...trap/engines/engine.py.229: process: 0.86101 | load: 0.00312 [I 2022-10-10 22:57:24] ...trap/engines/engine.py.230: loss: 0.18530 [I 2022-10-10 22:57:24] ...optimizers/trijoint.py.151: total_norm: 25.204947 [I 2022-10-10 22:57:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1900/2304 [I 2022-10-10 22:57:33] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:50 [I 2022-10-10 22:57:33] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00317 [I 2022-10-10 22:57:33] ...trap/engines/engine.py.230: loss: 0.11211 [I 2022-10-10 22:57:33] ...optimizers/trijoint.py.151: total_norm: 14.123350 [I 2022-10-10 22:57:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 1910/2304 [I 2022-10-10 22:57:41] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:39 [I 2022-10-10 22:57:41] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00311 [I 2022-10-10 22:57:41] ...trap/engines/engine.py.230: loss: 0.09048 [I 2022-10-10 22:57:41] ...optimizers/trijoint.py.151: total_norm: 21.708967 [I 2022-10-10 22:57:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 1920/2304 [I 2022-10-10 22:57:50] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:31 [I 2022-10-10 22:57:50] ...trap/engines/engine.py.229: process: 0.86133 | load: 0.00313 [I 2022-10-10 22:57:50] ...trap/engines/engine.py.230: loss: 0.19538 [I 2022-10-10 22:57:50] ...optimizers/trijoint.py.151: total_norm: 23.138075 [I 2022-10-10 22:57:59] ...trap/engines/engine.py.224: train: epoch 88 | batch 1930/2304 [I 2022-10-10 22:57:59] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:05:21 [I 2022-10-10 22:57:59] ...trap/engines/engine.py.229: process: 0.83959 | load: 0.00314 [I 2022-10-10 22:57:59] ...trap/engines/engine.py.230: loss: 0.09419 [I 2022-10-10 22:57:59] ...optimizers/trijoint.py.151: total_norm: 10.240867 [I 2022-10-10 22:58:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 1940/2304 [I 2022-10-10 22:58:07] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:05:14 [I 2022-10-10 22:58:07] ...trap/engines/engine.py.229: process: 0.86363 | load: 0.00310 [I 2022-10-10 22:58:07] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-10 22:58:07] ...optimizers/trijoint.py.151: total_norm: 10.755939 [I 2022-10-10 22:58:16] ...trap/engines/engine.py.224: train: epoch 88 | batch 1950/2304 [I 2022-10-10 22:58:16] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:05:06 [I 2022-10-10 22:58:16] ...trap/engines/engine.py.229: process: 0.87325 | load: 0.00311 [I 2022-10-10 22:58:16] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-10 22:58:16] ...optimizers/trijoint.py.151: total_norm: 25.561754 [I 2022-10-10 22:58:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 1960/2304 [I 2022-10-10 22:58:25] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:57 [I 2022-10-10 22:58:25] ...trap/engines/engine.py.229: process: 0.86008 | load: 0.00315 [I 2022-10-10 22:58:25] ...trap/engines/engine.py.230: loss: 0.09711 [I 2022-10-10 22:58:25] ...optimizers/trijoint.py.151: total_norm: 25.143217 [I 2022-10-10 22:58:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1970/2304 [I 2022-10-10 22:58:33] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:47 [I 2022-10-10 22:58:33] ...trap/engines/engine.py.229: process: 0.87493 | load: 0.00316 [I 2022-10-10 22:58:33] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-10 22:58:33] ...optimizers/trijoint.py.151: total_norm: 18.063547 [I 2022-10-10 22:58:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 1980/2304 [I 2022-10-10 22:58:42] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:04:37 [I 2022-10-10 22:58:42] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00308 [I 2022-10-10 22:58:42] ...trap/engines/engine.py.230: loss: 0.17086 [I 2022-10-10 22:58:42] ...optimizers/trijoint.py.151: total_norm: 24.502407 [I 2022-10-10 22:58:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 1990/2304 [I 2022-10-10 22:58:51] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:04:29 [I 2022-10-10 22:58:51] ...trap/engines/engine.py.229: process: 0.84710 | load: 0.00305 [I 2022-10-10 22:58:51] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-10 22:58:51] ...optimizers/trijoint.py.151: total_norm: 13.969124 [I 2022-10-10 22:58:59] ...trap/engines/engine.py.224: train: epoch 88 | batch 2000/2304 [I 2022-10-10 22:58:59] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:04:23 [I 2022-10-10 22:58:59] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00310 [I 2022-10-10 22:58:59] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-10 22:58:59] ...optimizers/trijoint.py.151: total_norm: 12.402727 [I 2022-10-10 22:59:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 2010/2304 [I 2022-10-10 22:59:08] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:04:14 [I 2022-10-10 22:59:08] ...trap/engines/engine.py.229: process: 0.86992 | load: 0.00306 [I 2022-10-10 22:59:08] ...trap/engines/engine.py.230: loss: 0.11209 [I 2022-10-10 22:59:08] ...optimizers/trijoint.py.151: total_norm: 9.767857 [I 2022-10-10 22:59:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 2020/2304 [I 2022-10-10 22:59:17] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:04:06 [I 2022-10-10 22:59:17] ...trap/engines/engine.py.229: process: 0.86998 | load: 0.00314 [I 2022-10-10 22:59:17] ...trap/engines/engine.py.230: loss: 0.18126 [I 2022-10-10 22:59:17] ...optimizers/trijoint.py.151: total_norm: 23.108744 [I 2022-10-10 22:59:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 2030/2304 [I 2022-10-10 22:59:25] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:56 [I 2022-10-10 22:59:25] ...trap/engines/engine.py.229: process: 0.86943 | load: 0.00323 [I 2022-10-10 22:59:25] ...trap/engines/engine.py.230: loss: 0.12357 [I 2022-10-10 22:59:25] ...optimizers/trijoint.py.151: total_norm: 17.714006 [I 2022-10-10 22:59:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 2040/2304 [I 2022-10-10 22:59:34] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:03:48 [I 2022-10-10 22:59:34] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00312 [I 2022-10-10 22:59:34] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-10 22:59:34] ...optimizers/trijoint.py.151: total_norm: 12.861393 [I 2022-10-10 22:59:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 2050/2304 [I 2022-10-10 22:59:43] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:38 [I 2022-10-10 22:59:43] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00313 [I 2022-10-10 22:59:43] ...trap/engines/engine.py.230: loss: 0.13927 [I 2022-10-10 22:59:43] ...optimizers/trijoint.py.151: total_norm: 17.217972 [I 2022-10-10 22:59:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 2060/2304 [I 2022-10-10 22:59:51] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:30 [I 2022-10-10 22:59:51] ...trap/engines/engine.py.229: process: 0.87680 | load: 0.00307 [I 2022-10-10 22:59:51] ...trap/engines/engine.py.230: loss: 0.14473 [I 2022-10-10 22:59:51] ...optimizers/trijoint.py.151: total_norm: 30.015368 [I 2022-10-10 23:00:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 2070/2304 [I 2022-10-10 23:00:00] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:21 [I 2022-10-10 23:00:00] ...trap/engines/engine.py.229: process: 0.86200 | load: 0.00311 [I 2022-10-10 23:00:00] ...trap/engines/engine.py.230: loss: 0.16749 [I 2022-10-10 23:00:00] ...optimizers/trijoint.py.151: total_norm: 12.113547 [I 2022-10-10 23:00:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 2080/2304 [I 2022-10-10 23:00:09] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:03:13 [I 2022-10-10 23:00:09] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00314 [I 2022-10-10 23:00:09] ...trap/engines/engine.py.230: loss: 0.11730 [I 2022-10-10 23:00:09] ...optimizers/trijoint.py.151: total_norm: 15.806458 [I 2022-10-10 23:00:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 2090/2304 [I 2022-10-10 23:00:17] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:03:05 [I 2022-10-10 23:00:17] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00313 [I 2022-10-10 23:00:17] ...trap/engines/engine.py.230: loss: 0.09558 [I 2022-10-10 23:00:17] ...optimizers/trijoint.py.151: total_norm: 11.676688 [I 2022-10-10 23:00:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 2100/2304 [I 2022-10-10 23:00:26] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:55 [I 2022-10-10 23:00:26] ...trap/engines/engine.py.229: process: 0.86650 | load: 0.00310 [I 2022-10-10 23:00:26] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-10 23:00:26] ...optimizers/trijoint.py.151: total_norm: 13.931049 [I 2022-10-10 23:00:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 2110/2304 [I 2022-10-10 23:00:34] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:46 [I 2022-10-10 23:00:34] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00339 [I 2022-10-10 23:00:34] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-10 23:00:34] ...optimizers/trijoint.py.151: total_norm: 12.818260 [I 2022-10-10 23:00:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 2120/2304 [I 2022-10-10 23:00:43] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:36 [I 2022-10-10 23:00:43] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00298 [I 2022-10-10 23:00:43] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-10 23:00:43] ...optimizers/trijoint.py.151: total_norm: 9.918000 [I 2022-10-10 23:00:52] ...trap/engines/engine.py.224: train: epoch 88 | batch 2130/2304 [I 2022-10-10 23:00:52] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:27 [I 2022-10-10 23:00:52] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00300 [I 2022-10-10 23:00:52] ...trap/engines/engine.py.230: loss: 0.09308 [I 2022-10-10 23:00:52] ...optimizers/trijoint.py.151: total_norm: 23.499626 [I 2022-10-10 23:01:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 2140/2304 [I 2022-10-10 23:01:00] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:18 [I 2022-10-10 23:01:00] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00297 [I 2022-10-10 23:01:00] ...trap/engines/engine.py.230: loss: 0.09545 [I 2022-10-10 23:01:00] ...optimizers/trijoint.py.151: total_norm: 8.963194 [I 2022-10-10 23:01:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 2150/2304 [I 2022-10-10 23:01:09] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:02:10 [I 2022-10-10 23:01:09] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00293 [I 2022-10-10 23:01:09] ...trap/engines/engine.py.230: loss: 0.19496 [I 2022-10-10 23:01:09] ...optimizers/trijoint.py.151: total_norm: 19.738607 [I 2022-10-10 23:01:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 2160/2304 [I 2022-10-10 23:01:17] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:02:02 [I 2022-10-10 23:01:17] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00301 [I 2022-10-10 23:01:17] ...trap/engines/engine.py.230: loss: 0.09866 [I 2022-10-10 23:01:17] ...optimizers/trijoint.py.151: total_norm: 11.191888 [I 2022-10-10 23:01:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 2170/2304 [I 2022-10-10 23:01:26] ...trap/engines/engine.py.228: elapsed: 0:31:32 | left: 0:01:53 [I 2022-10-10 23:01:26] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00297 [I 2022-10-10 23:01:26] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-10 23:01:26] ...optimizers/trijoint.py.151: total_norm: 11.221338 [I 2022-10-10 23:01:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 2180/2304 [I 2022-10-10 23:01:34] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:45 [I 2022-10-10 23:01:34] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00292 [I 2022-10-10 23:01:34] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-10 23:01:34] ...optimizers/trijoint.py.151: total_norm: 9.416031 [I 2022-10-10 23:01:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 2190/2304 [I 2022-10-10 23:01:43] ...trap/engines/engine.py.228: elapsed: 0:31:49 | left: 0:01:36 [I 2022-10-10 23:01:43] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00294 [I 2022-10-10 23:01:43] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-10 23:01:43] ...optimizers/trijoint.py.151: total_norm: 19.942068 [I 2022-10-10 23:01:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 2200/2304 [I 2022-10-10 23:01:51] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:28 [I 2022-10-10 23:01:51] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00296 [I 2022-10-10 23:01:51] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-10 23:01:51] ...optimizers/trijoint.py.151: total_norm: 10.218527 [I 2022-10-10 23:02:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 2210/2304 [I 2022-10-10 23:02:00] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:19 [I 2022-10-10 23:02:00] ...trap/engines/engine.py.229: process: 0.85548 | load: 0.00304 [I 2022-10-10 23:02:00] ...trap/engines/engine.py.230: loss: 0.24338 [I 2022-10-10 23:02:00] ...optimizers/trijoint.py.151: total_norm: 11.848927 [I 2022-10-10 23:02:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 2220/2304 [I 2022-10-10 23:02:08] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:01:11 [I 2022-10-10 23:02:08] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00303 [I 2022-10-10 23:02:08] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-10 23:02:08] ...optimizers/trijoint.py.151: total_norm: 10.718366 [I 2022-10-10 23:02:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 2230/2304 [I 2022-10-10 23:02:17] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:01:03 [I 2022-10-10 23:02:17] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00299 [I 2022-10-10 23:02:17] ...trap/engines/engine.py.230: loss: 0.13549 [I 2022-10-10 23:02:17] ...optimizers/trijoint.py.151: total_norm: 9.992344 [I 2022-10-10 23:02:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 2240/2304 [I 2022-10-10 23:02:25] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:54 [I 2022-10-10 23:02:25] ...trap/engines/engine.py.229: process: 0.85125 | load: 0.00293 [I 2022-10-10 23:02:25] ...trap/engines/engine.py.230: loss: 0.15671 [I 2022-10-10 23:02:25] ...optimizers/trijoint.py.151: total_norm: 32.710876 [I 2022-10-10 23:02:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 2250/2304 [I 2022-10-10 23:02:34] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:45 [I 2022-10-10 23:02:34] ...trap/engines/engine.py.229: process: 0.83991 | load: 0.00296 [I 2022-10-10 23:02:34] ...trap/engines/engine.py.230: loss: 0.10887 [I 2022-10-10 23:02:34] ...optimizers/trijoint.py.151: total_norm: 18.738455 [I 2022-10-10 23:02:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 2260/2304 [I 2022-10-10 23:02:42] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:37 [I 2022-10-10 23:02:42] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00293 [I 2022-10-10 23:02:42] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-10 23:02:42] ...optimizers/trijoint.py.151: total_norm: 7.081088 [I 2022-10-10 23:02:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 2270/2304 [I 2022-10-10 23:02:51] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:29 [I 2022-10-10 23:02:51] ...trap/engines/engine.py.229: process: 0.86080 | load: 0.00301 [I 2022-10-10 23:02:51] ...trap/engines/engine.py.230: loss: 0.12517 [I 2022-10-10 23:02:51] ...optimizers/trijoint.py.151: total_norm: 51.251232 [I 2022-10-10 23:03:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 2280/2304 [I 2022-10-10 23:03:00] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:20 [I 2022-10-10 23:03:00] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00300 [I 2022-10-10 23:03:00] ...trap/engines/engine.py.230: loss: 0.12530 [I 2022-10-10 23:03:00] ...optimizers/trijoint.py.151: total_norm: 8.161878 [I 2022-10-10 23:03:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 2290/2304 [I 2022-10-10 23:03:08] ...trap/engines/engine.py.228: elapsed: 0:33:14 | left: 0:00:11 [I 2022-10-10 23:03:08] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00300 [I 2022-10-10 23:03:08] ...trap/engines/engine.py.230: loss: 0.15860 [I 2022-10-10 23:03:08] ...optimizers/trijoint.py.151: total_norm: 33.291828 [I 2022-10-10 23:03:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 2300/2304 [I 2022-10-10 23:03:17] ...trap/engines/engine.py.228: elapsed: 0:33:23 | left: 0:00:03 [I 2022-10-10 23:03:17] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00017 [I 2022-10-10 23:03:17] ...trap/engines/engine.py.230: loss: 0.09541 [I 2022-10-10 23:03:17] ...optimizers/trijoint.py.151: total_norm: 43.080456 [I 2022-10-10 23:03:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 2304/2304 [I 2022-10-10 23:03:20] ...trap/engines/engine.py.228: elapsed: 0:33:26 | left: 0:00:00 [I 2022-10-10 23:03:20] ...trap/engines/engine.py.229: process: 0.83877 | load: 0.00016 [I 2022-10-10 23:03:20] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-10 23:03:20] ...optimizers/trijoint.py.151: total_norm: 9.743668 [S 2022-10-10 23:03:21] ...trap/engines/engine.py.129: train_epoch.epoch: 88 [S 2022-10-10 23:03:21] ...trap/engines/engine.py.129: train_epoch.loss: 0.1306971890904381 [I 2022-10-10 23:03:21] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 23:03:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 88 [I 2022-10-10 23:03:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 23:03:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 23:03:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 23:03:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 23:03:28] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 23:03:28] ...trap/engines/engine.py.321: eval: epoch 88 | batch 0/510 [I 2022-10-10 23:03:28] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:33 [I 2022-10-10 23:03:28] ...trap/engines/engine.py.326: process: 5.47759 | load: 2.09424 [I 2022-10-10 23:03:34] ...trap/engines/engine.py.321: eval: epoch 88 | batch 10/510 [I 2022-10-10 23:03:34] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:09:18 [I 2022-10-10 23:03:34] ...trap/engines/engine.py.326: process: 0.60056 | load: 0.00019 [I 2022-10-10 23:03:40] ...trap/engines/engine.py.321: eval: epoch 88 | batch 20/510 [I 2022-10-10 23:03:40] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:47 [I 2022-10-10 23:03:40] ...trap/engines/engine.py.326: process: 0.83692 | load: 0.00020 [I 2022-10-10 23:03:47] ...trap/engines/engine.py.321: eval: epoch 88 | batch 30/510 [I 2022-10-10 23:03:47] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:07 [I 2022-10-10 23:03:47] ...trap/engines/engine.py.326: process: 0.59676 | load: 0.00021 [I 2022-10-10 23:03:54] ...trap/engines/engine.py.321: eval: epoch 88 | batch 40/510 [I 2022-10-10 23:03:54] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:05:10 [I 2022-10-10 23:03:54] ...trap/engines/engine.py.326: process: 0.50371 | load: 0.00020 [I 2022-10-10 23:03:59] ...trap/engines/engine.py.321: eval: epoch 88 | batch 50/510 [I 2022-10-10 23:03:59] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:08 [I 2022-10-10 23:03:59] ...trap/engines/engine.py.326: process: 0.49062 | load: 0.00018 [I 2022-10-10 23:04:04] ...trap/engines/engine.py.321: eval: epoch 88 | batch 60/510 [I 2022-10-10 23:04:04] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:34 [I 2022-10-10 23:04:04] ...trap/engines/engine.py.326: process: 0.47716 | load: 0.00021 [I 2022-10-10 23:04:09] ...trap/engines/engine.py.321: eval: epoch 88 | batch 70/510 [I 2022-10-10 23:04:09] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:54 [I 2022-10-10 23:04:09] ...trap/engines/engine.py.326: process: 0.41610 | load: 0.00020 [I 2022-10-10 23:04:15] ...trap/engines/engine.py.321: eval: epoch 88 | batch 80/510 [I 2022-10-10 23:04:15] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:56 [I 2022-10-10 23:04:15] ...trap/engines/engine.py.326: process: 0.47810 | load: 0.00020 [I 2022-10-10 23:04:22] ...trap/engines/engine.py.321: eval: epoch 88 | batch 90/510 [I 2022-10-10 23:04:22] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:06:31 [I 2022-10-10 23:04:22] ...trap/engines/engine.py.326: process: 1.46964 | load: 0.00021 [I 2022-10-10 23:04:29] ...trap/engines/engine.py.321: eval: epoch 88 | batch 100/510 [I 2022-10-10 23:04:29] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:04:19 [I 2022-10-10 23:04:29] ...trap/engines/engine.py.326: process: 0.30873 | load: 0.00028 [I 2022-10-10 23:04:32] ...trap/engines/engine.py.321: eval: epoch 88 | batch 110/510 [I 2022-10-10 23:04:32] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:22 [I 2022-10-10 23:04:32] ...trap/engines/engine.py.326: process: 0.32317 | load: 0.00021 [I 2022-10-10 23:04:35] ...trap/engines/engine.py.321: eval: epoch 88 | batch 120/510 [I 2022-10-10 23:04:35] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:06 [I 2022-10-10 23:04:35] ...trap/engines/engine.py.326: process: 0.31438 | load: 0.00020 [I 2022-10-10 23:04:38] ...trap/engines/engine.py.321: eval: epoch 88 | batch 130/510 [I 2022-10-10 23:04:38] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:02:02 [I 2022-10-10 23:04:38] ...trap/engines/engine.py.326: process: 0.31252 | load: 0.00019 [I 2022-10-10 23:04:42] ...trap/engines/engine.py.321: eval: epoch 88 | batch 140/510 [I 2022-10-10 23:04:42] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:58 [I 2022-10-10 23:04:42] ...trap/engines/engine.py.326: process: 0.32470 | load: 0.00021 [I 2022-10-10 23:04:45] ...trap/engines/engine.py.321: eval: epoch 88 | batch 150/510 [I 2022-10-10 23:04:45] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:56 [I 2022-10-10 23:04:45] ...trap/engines/engine.py.326: process: 0.32152 | load: 0.00021 [I 2022-10-10 23:04:48] ...trap/engines/engine.py.321: eval: epoch 88 | batch 160/510 [I 2022-10-10 23:04:48] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:53 [I 2022-10-10 23:04:48] ...trap/engines/engine.py.326: process: 0.32622 | load: 0.00021 [I 2022-10-10 23:04:51] ...trap/engines/engine.py.321: eval: epoch 88 | batch 170/510 [I 2022-10-10 23:04:51] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:49 [I 2022-10-10 23:04:51] ...trap/engines/engine.py.326: process: 0.32359 | load: 0.00022 [I 2022-10-10 23:04:55] ...trap/engines/engine.py.321: eval: epoch 88 | batch 180/510 [I 2022-10-10 23:04:55] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:46 [I 2022-10-10 23:04:55] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00021 [I 2022-10-10 23:04:58] ...trap/engines/engine.py.321: eval: epoch 88 | batch 190/510 [I 2022-10-10 23:04:58] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:43 [I 2022-10-10 23:04:58] ...trap/engines/engine.py.326: process: 0.33060 | load: 0.00021 [I 2022-10-10 23:05:01] ...trap/engines/engine.py.321: eval: epoch 88 | batch 200/510 [I 2022-10-10 23:05:01] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:40 [I 2022-10-10 23:05:01] ...trap/engines/engine.py.326: process: 0.32426 | load: 0.00020 [I 2022-10-10 23:05:04] ...trap/engines/engine.py.321: eval: epoch 88 | batch 210/510 [I 2022-10-10 23:05:04] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:38 [I 2022-10-10 23:05:04] ...trap/engines/engine.py.326: process: 0.33794 | load: 0.00021 [I 2022-10-10 23:05:08] ...trap/engines/engine.py.321: eval: epoch 88 | batch 220/510 [I 2022-10-10 23:05:08] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:33 [I 2022-10-10 23:05:08] ...trap/engines/engine.py.326: process: 0.31711 | load: 0.00020 [I 2022-10-10 23:05:11] ...trap/engines/engine.py.321: eval: epoch 88 | batch 230/510 [I 2022-10-10 23:05:11] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:30 [I 2022-10-10 23:05:11] ...trap/engines/engine.py.326: process: 0.32339 | load: 0.00020 [I 2022-10-10 23:05:14] ...trap/engines/engine.py.321: eval: epoch 88 | batch 240/510 [I 2022-10-10 23:05:14] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:28 [I 2022-10-10 23:05:14] ...trap/engines/engine.py.326: process: 0.32569 | load: 0.00024 [I 2022-10-10 23:05:17] ...trap/engines/engine.py.321: eval: epoch 88 | batch 250/510 [I 2022-10-10 23:05:17] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:23 [I 2022-10-10 23:05:17] ...trap/engines/engine.py.326: process: 0.31906 | load: 0.00021 [I 2022-10-10 23:05:21] ...trap/engines/engine.py.321: eval: epoch 88 | batch 260/510 [I 2022-10-10 23:05:21] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:21 [I 2022-10-10 23:05:21] ...trap/engines/engine.py.326: process: 0.34167 | load: 0.00021 [I 2022-10-10 23:05:24] ...trap/engines/engine.py.321: eval: epoch 88 | batch 270/510 [I 2022-10-10 23:05:24] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:17 [I 2022-10-10 23:05:24] ...trap/engines/engine.py.326: process: 0.32064 | load: 0.00021 [I 2022-10-10 23:05:27] ...trap/engines/engine.py.321: eval: epoch 88 | batch 280/510 [I 2022-10-10 23:05:27] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:14 [I 2022-10-10 23:05:27] ...trap/engines/engine.py.326: process: 0.32950 | load: 0.00021 [I 2022-10-10 23:05:30] ...trap/engines/engine.py.321: eval: epoch 88 | batch 290/510 [I 2022-10-10 23:05:30] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:10 [I 2022-10-10 23:05:30] ...trap/engines/engine.py.326: process: 0.31251 | load: 0.00020 [I 2022-10-10 23:05:34] ...trap/engines/engine.py.321: eval: epoch 88 | batch 300/510 [I 2022-10-10 23:05:34] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:08 [I 2022-10-10 23:05:34] ...trap/engines/engine.py.326: process: 0.32693 | load: 0.00020 [I 2022-10-10 23:05:37] ...trap/engines/engine.py.321: eval: epoch 88 | batch 310/510 [I 2022-10-10 23:05:37] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:04 [I 2022-10-10 23:05:37] ...trap/engines/engine.py.326: process: 0.32698 | load: 0.00022 [I 2022-10-10 23:05:40] ...trap/engines/engine.py.321: eval: epoch 88 | batch 320/510 [I 2022-10-10 23:05:40] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:01:01 [I 2022-10-10 23:05:40] ...trap/engines/engine.py.326: process: 0.31448 | load: 0.00021 [I 2022-10-10 23:05:43] ...trap/engines/engine.py.321: eval: epoch 88 | batch 330/510 [I 2022-10-10 23:05:43] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:58 [I 2022-10-10 23:05:43] ...trap/engines/engine.py.326: process: 0.32185 | load: 0.00021 [I 2022-10-10 23:05:47] ...trap/engines/engine.py.321: eval: epoch 88 | batch 340/510 [I 2022-10-10 23:05:47] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:55 [I 2022-10-10 23:05:47] ...trap/engines/engine.py.326: process: 0.32760 | load: 0.00021 [I 2022-10-10 23:05:50] ...trap/engines/engine.py.321: eval: epoch 88 | batch 350/510 [I 2022-10-10 23:05:50] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:52 [I 2022-10-10 23:05:50] ...trap/engines/engine.py.326: process: 0.32350 | load: 0.00021 [I 2022-10-10 23:05:53] ...trap/engines/engine.py.321: eval: epoch 88 | batch 360/510 [I 2022-10-10 23:05:53] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:48 [I 2022-10-10 23:05:53] ...trap/engines/engine.py.326: process: 0.31424 | load: 0.00020 [I 2022-10-10 23:05:56] ...trap/engines/engine.py.321: eval: epoch 88 | batch 370/510 [I 2022-10-10 23:05:56] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:45 [I 2022-10-10 23:05:56] ...trap/engines/engine.py.326: process: 0.32337 | load: 0.00021 [I 2022-10-10 23:06:00] ...trap/engines/engine.py.321: eval: epoch 88 | batch 380/510 [I 2022-10-10 23:06:00] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:42 [I 2022-10-10 23:06:00] ...trap/engines/engine.py.326: process: 0.32746 | load: 0.00021 [I 2022-10-10 23:06:03] ...trap/engines/engine.py.321: eval: epoch 88 | batch 390/510 [I 2022-10-10 23:06:03] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:39 [I 2022-10-10 23:06:03] ...trap/engines/engine.py.326: process: 0.32642 | load: 0.00021 [I 2022-10-10 23:06:06] ...trap/engines/engine.py.321: eval: epoch 88 | batch 400/510 [I 2022-10-10 23:06:06] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:35 [I 2022-10-10 23:06:06] ...trap/engines/engine.py.326: process: 0.33134 | load: 0.00021 [I 2022-10-10 23:06:09] ...trap/engines/engine.py.321: eval: epoch 88 | batch 410/510 [I 2022-10-10 23:06:09] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:32 [I 2022-10-10 23:06:09] ...trap/engines/engine.py.326: process: 0.32923 | load: 0.00021 [I 2022-10-10 23:06:13] ...trap/engines/engine.py.321: eval: epoch 88 | batch 420/510 [I 2022-10-10 23:06:13] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:29 [I 2022-10-10 23:06:13] ...trap/engines/engine.py.326: process: 0.31915 | load: 0.00021 [I 2022-10-10 23:06:16] ...trap/engines/engine.py.321: eval: epoch 88 | batch 430/510 [I 2022-10-10 23:06:16] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:25 [I 2022-10-10 23:06:16] ...trap/engines/engine.py.326: process: 0.32347 | load: 0.00021 [I 2022-10-10 23:06:19] ...trap/engines/engine.py.321: eval: epoch 88 | batch 440/510 [I 2022-10-10 23:06:19] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:22 [I 2022-10-10 23:06:19] ...trap/engines/engine.py.326: process: 0.32560 | load: 0.00021 [I 2022-10-10 23:06:22] ...trap/engines/engine.py.321: eval: epoch 88 | batch 450/510 [I 2022-10-10 23:06:22] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:19 [I 2022-10-10 23:06:22] ...trap/engines/engine.py.326: process: 0.32719 | load: 0.00021 [I 2022-10-10 23:06:26] ...trap/engines/engine.py.321: eval: epoch 88 | batch 460/510 [I 2022-10-10 23:06:26] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:16 [I 2022-10-10 23:06:26] ...trap/engines/engine.py.326: process: 0.32527 | load: 0.00022 [I 2022-10-10 23:06:29] ...trap/engines/engine.py.321: eval: epoch 88 | batch 470/510 [I 2022-10-10 23:06:29] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:12 [I 2022-10-10 23:06:29] ...trap/engines/engine.py.326: process: 0.31960 | load: 0.00021 [I 2022-10-10 23:06:32] ...trap/engines/engine.py.321: eval: epoch 88 | batch 480/510 [I 2022-10-10 23:06:32] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:09 [I 2022-10-10 23:06:32] ...trap/engines/engine.py.326: process: 0.32525 | load: 0.00021 [I 2022-10-10 23:06:36] ...trap/engines/engine.py.321: eval: epoch 88 | batch 490/510 [I 2022-10-10 23:06:36] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:06 [I 2022-10-10 23:06:36] ...trap/engines/engine.py.326: process: 0.32037 | load: 0.00022 [I 2022-10-10 23:06:39] ...trap/engines/engine.py.321: eval: epoch 88 | batch 500/510 [I 2022-10-10 23:06:39] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:03 [I 2022-10-10 23:06:39] ...trap/engines/engine.py.326: process: 0.32397 | load: 0.00022 [I 2022-10-10 23:06:42] ...trap/engines/engine.py.321: eval: epoch 88 | batch 510/510 [I 2022-10-10 23:06:42] ...trap/engines/engine.py.325: elapsed: 0:03:19 | left: 0:00:00 [I 2022-10-10 23:06:42] ...trap/engines/engine.py.326: process: 0.31360 | load: 0.00017 [S 2022-10-10 23:06:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 88 [I 2022-10-10 23:06:43] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.747100 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911700 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939600 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.738800 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.913800 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.939900 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011954 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008100 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005851 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013280 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005173 [S 2022-10-10 23:07:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004949 [I 2022-10-10 23:07:27] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-10 23:07:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 23:07:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 23:07:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 23:07:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 23:07:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 23:07:35] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 23:07:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 23:07:39] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 23:07:39] ...trap/engines/engine.py.470: Saving model... [I 2022-10-10 23:07:42] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 23:07:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 23:07:47] ...trap/engines/engine.py.163: Training model on trainset for epoch 89 [I 2022-10-10 23:07:47] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 23:07:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 0/2304 [I 2022-10-10 23:07:50] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:03:17 [I 2022-10-10 23:07:50] ...trap/engines/engine.py.229: process: 3.21058 | load: 2.30699 [I 2022-10-10 23:07:50] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-10 23:07:50] ...optimizers/trijoint.py.151: total_norm: 9.157739 [I 2022-10-10 23:07:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 10/2304 [I 2022-10-10 23:07:59] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:20 [I 2022-10-10 23:07:59] ...trap/engines/engine.py.229: process: 0.85888 | load: 0.00577 [I 2022-10-10 23:07:59] ...trap/engines/engine.py.230: loss: 0.18985 [I 2022-10-10 23:07:59] ...optimizers/trijoint.py.151: total_norm: 47.519451 [I 2022-10-10 23:08:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 20/2304 [I 2022-10-10 23:08:07] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:15 [I 2022-10-10 23:08:07] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00454 [I 2022-10-10 23:08:07] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-10 23:08:07] ...optimizers/trijoint.py.151: total_norm: 20.121292 [I 2022-10-10 23:08:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 30/2304 [I 2022-10-10 23:08:16] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:12 [I 2022-10-10 23:08:16] ...trap/engines/engine.py.229: process: 0.83836 | load: 0.00446 [I 2022-10-10 23:08:16] ...trap/engines/engine.py.230: loss: 0.11834 [I 2022-10-10 23:08:16] ...optimizers/trijoint.py.151: total_norm: 14.554182 [I 2022-10-10 23:08:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 40/2304 [I 2022-10-10 23:08:24] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:46 [I 2022-10-10 23:08:24] ...trap/engines/engine.py.229: process: 0.82916 | load: 0.00357 [I 2022-10-10 23:08:24] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-10 23:08:24] ...optimizers/trijoint.py.151: total_norm: 15.018362 [I 2022-10-10 23:08:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 50/2304 [I 2022-10-10 23:08:33] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:36 [I 2022-10-10 23:08:33] ...trap/engines/engine.py.229: process: 0.85138 | load: 0.00379 [I 2022-10-10 23:08:33] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-10 23:08:33] ...optimizers/trijoint.py.151: total_norm: 14.496661 [I 2022-10-10 23:08:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 60/2304 [I 2022-10-10 23:08:41] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:50 [I 2022-10-10 23:08:41] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00325 [I 2022-10-10 23:08:41] ...trap/engines/engine.py.230: loss: 0.10960 [I 2022-10-10 23:08:41] ...optimizers/trijoint.py.151: total_norm: 10.696982 [I 2022-10-10 23:08:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 70/2304 [I 2022-10-10 23:08:50] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:43 [I 2022-10-10 23:08:50] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00350 [I 2022-10-10 23:08:50] ...trap/engines/engine.py.230: loss: 0.09580 [I 2022-10-10 23:08:50] ...optimizers/trijoint.py.151: total_norm: 18.208242 [I 2022-10-10 23:08:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 80/2304 [I 2022-10-10 23:08:58] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:33 [I 2022-10-10 23:08:58] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00306 [I 2022-10-10 23:08:58] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-10 23:08:58] ...optimizers/trijoint.py.151: total_norm: 27.734756 [I 2022-10-10 23:09:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 90/2304 [I 2022-10-10 23:09:07] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:29 [I 2022-10-10 23:09:07] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00347 [I 2022-10-10 23:09:07] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-10 23:09:07] ...optimizers/trijoint.py.151: total_norm: 16.050432 [I 2022-10-10 23:09:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 100/2304 [I 2022-10-10 23:09:16] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:23 [I 2022-10-10 23:09:16] ...trap/engines/engine.py.229: process: 0.86550 | load: 0.00360 [I 2022-10-10 23:09:16] ...trap/engines/engine.py.230: loss: 0.13293 [I 2022-10-10 23:09:16] ...optimizers/trijoint.py.151: total_norm: 29.510862 [I 2022-10-10 23:09:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 110/2304 [I 2022-10-10 23:09:24] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:05 [I 2022-10-10 23:09:24] ...trap/engines/engine.py.229: process: 0.85868 | load: 0.00339 [I 2022-10-10 23:09:24] ...trap/engines/engine.py.230: loss: 0.09675 [I 2022-10-10 23:09:24] ...optimizers/trijoint.py.151: total_norm: 10.026553 [I 2022-10-10 23:09:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 120/2304 [I 2022-10-10 23:09:33] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:52 [I 2022-10-10 23:09:33] ...trap/engines/engine.py.229: process: 0.84969 | load: 0.00323 [I 2022-10-10 23:09:33] ...trap/engines/engine.py.230: loss: 0.10558 [I 2022-10-10 23:09:33] ...optimizers/trijoint.py.151: total_norm: 13.106580 [I 2022-10-10 23:09:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 130/2304 [I 2022-10-10 23:09:41] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:40 [I 2022-10-10 23:09:41] ...trap/engines/engine.py.229: process: 0.83758 | load: 0.00310 [I 2022-10-10 23:09:41] ...trap/engines/engine.py.230: loss: 0.17025 [I 2022-10-10 23:09:41] ...optimizers/trijoint.py.151: total_norm: 11.521852 [I 2022-10-10 23:09:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 140/2304 [I 2022-10-10 23:09:50] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:40 [I 2022-10-10 23:09:50] ...trap/engines/engine.py.229: process: 0.85149 | load: 0.00310 [I 2022-10-10 23:09:50] ...trap/engines/engine.py.230: loss: 0.10751 [I 2022-10-10 23:09:50] ...optimizers/trijoint.py.151: total_norm: 12.774330 [I 2022-10-10 23:09:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 150/2304 [I 2022-10-10 23:09:58] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:31 [I 2022-10-10 23:09:58] ...trap/engines/engine.py.229: process: 0.83682 | load: 0.00317 [I 2022-10-10 23:09:58] ...trap/engines/engine.py.230: loss: 0.08584 [I 2022-10-10 23:09:58] ...optimizers/trijoint.py.151: total_norm: 10.873329 [I 2022-10-10 23:10:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 160/2304 [I 2022-10-10 23:10:07] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:19 [I 2022-10-10 23:10:07] ...trap/engines/engine.py.229: process: 0.84845 | load: 0.00309 [I 2022-10-10 23:10:07] ...trap/engines/engine.py.230: loss: 0.17912 [I 2022-10-10 23:10:07] ...optimizers/trijoint.py.151: total_norm: 27.833225 [I 2022-10-10 23:10:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 170/2304 [I 2022-10-10 23:10:15] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:09 [I 2022-10-10 23:10:15] ...trap/engines/engine.py.229: process: 0.85588 | load: 0.00311 [I 2022-10-10 23:10:15] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-10 23:10:15] ...optimizers/trijoint.py.151: total_norm: 14.883933 [I 2022-10-10 23:10:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 180/2304 [I 2022-10-10 23:10:24] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:09 [I 2022-10-10 23:10:24] ...trap/engines/engine.py.229: process: 0.86475 | load: 0.00297 [I 2022-10-10 23:10:24] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-10 23:10:24] ...optimizers/trijoint.py.151: total_norm: 9.756541 [I 2022-10-10 23:10:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 190/2304 [I 2022-10-10 23:10:32] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:29:52 [I 2022-10-10 23:10:32] ...trap/engines/engine.py.229: process: 0.83898 | load: 0.00293 [I 2022-10-10 23:10:32] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-10 23:10:32] ...optimizers/trijoint.py.151: total_norm: 7.857113 [I 2022-10-10 23:10:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 200/2304 [I 2022-10-10 23:10:41] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:30:05 [I 2022-10-10 23:10:41] ...trap/engines/engine.py.229: process: 0.85937 | load: 0.00312 [I 2022-10-10 23:10:41] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-10 23:10:41] ...optimizers/trijoint.py.151: total_norm: 10.921168 [I 2022-10-10 23:10:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 210/2304 [I 2022-10-10 23:10:49] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:45 [I 2022-10-10 23:10:49] ...trap/engines/engine.py.229: process: 0.85148 | load: 0.00306 [I 2022-10-10 23:10:49] ...trap/engines/engine.py.230: loss: 0.11647 [I 2022-10-10 23:10:49] ...optimizers/trijoint.py.151: total_norm: 24.444469 [I 2022-10-10 23:10:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 220/2304 [I 2022-10-10 23:10:58] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:41 [I 2022-10-10 23:10:58] ...trap/engines/engine.py.229: process: 0.85855 | load: 0.00295 [I 2022-10-10 23:10:58] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-10 23:10:58] ...optimizers/trijoint.py.151: total_norm: 15.565292 [I 2022-10-10 23:11:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 230/2304 [I 2022-10-10 23:11:07] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:40 [I 2022-10-10 23:11:07] ...trap/engines/engine.py.229: process: 0.85921 | load: 0.00292 [I 2022-10-10 23:11:07] ...trap/engines/engine.py.230: loss: 0.08106 [I 2022-10-10 23:11:07] ...optimizers/trijoint.py.151: total_norm: 11.191443 [I 2022-10-10 23:11:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 240/2304 [I 2022-10-10 23:11:15] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:20 [I 2022-10-10 23:11:15] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00310 [I 2022-10-10 23:11:15] ...trap/engines/engine.py.230: loss: 0.16310 [I 2022-10-10 23:11:15] ...optimizers/trijoint.py.151: total_norm: 12.978477 [I 2022-10-10 23:11:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 250/2304 [I 2022-10-10 23:11:24] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:15 [I 2022-10-10 23:11:24] ...trap/engines/engine.py.229: process: 0.85528 | load: 0.00322 [I 2022-10-10 23:11:24] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-10 23:11:24] ...optimizers/trijoint.py.151: total_norm: 14.424994 [I 2022-10-10 23:11:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 260/2304 [I 2022-10-10 23:11:32] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:06 [I 2022-10-10 23:11:32] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00306 [I 2022-10-10 23:11:32] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-10 23:11:32] ...optimizers/trijoint.py.151: total_norm: 12.096904 [I 2022-10-10 23:11:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 270/2304 [I 2022-10-10 23:11:41] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:57 [I 2022-10-10 23:11:41] ...trap/engines/engine.py.229: process: 0.85250 | load: 0.00313 [I 2022-10-10 23:11:41] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-10 23:11:41] ...optimizers/trijoint.py.151: total_norm: 9.903925 [I 2022-10-10 23:11:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 280/2304 [I 2022-10-10 23:11:49] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:50 [I 2022-10-10 23:11:49] ...trap/engines/engine.py.229: process: 0.86482 | load: 0.00313 [I 2022-10-10 23:11:49] ...trap/engines/engine.py.230: loss: 0.15898 [I 2022-10-10 23:11:49] ...optimizers/trijoint.py.151: total_norm: 15.325531 [I 2022-10-10 23:11:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 290/2304 [I 2022-10-10 23:11:58] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:43 [I 2022-10-10 23:11:58] ...trap/engines/engine.py.229: process: 0.87003 | load: 0.00317 [I 2022-10-10 23:11:58] ...trap/engines/engine.py.230: loss: 0.18199 [I 2022-10-10 23:11:58] ...optimizers/trijoint.py.151: total_norm: 17.696795 [I 2022-10-10 23:12:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 300/2304 [I 2022-10-10 23:12:06] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:39 [I 2022-10-10 23:12:06] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00324 [I 2022-10-10 23:12:06] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-10 23:12:06] ...optimizers/trijoint.py.151: total_norm: 32.836727 [I 2022-10-10 23:12:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 310/2304 [I 2022-10-10 23:12:15] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:31 [I 2022-10-10 23:12:15] ...trap/engines/engine.py.229: process: 0.86656 | load: 0.00312 [I 2022-10-10 23:12:15] ...trap/engines/engine.py.230: loss: 0.10768 [I 2022-10-10 23:12:15] ...optimizers/trijoint.py.151: total_norm: 12.473805 [I 2022-10-10 23:12:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 320/2304 [I 2022-10-10 23:12:24] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:28 [I 2022-10-10 23:12:24] ...trap/engines/engine.py.229: process: 0.84956 | load: 0.00318 [I 2022-10-10 23:12:24] ...trap/engines/engine.py.230: loss: 0.10967 [I 2022-10-10 23:12:24] ...optimizers/trijoint.py.151: total_norm: 10.379097 [I 2022-10-10 23:12:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 330/2304 [I 2022-10-10 23:12:32] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:24 [I 2022-10-10 23:12:32] ...trap/engines/engine.py.229: process: 0.85545 | load: 0.00316 [I 2022-10-10 23:12:32] ...trap/engines/engine.py.230: loss: 0.10043 [I 2022-10-10 23:12:32] ...optimizers/trijoint.py.151: total_norm: 12.933498 [I 2022-10-10 23:12:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 340/2304 [I 2022-10-10 23:12:41] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:28:19 [I 2022-10-10 23:12:41] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00329 [I 2022-10-10 23:12:41] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-10 23:12:41] ...optimizers/trijoint.py.151: total_norm: 12.112154 [I 2022-10-10 23:12:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 350/2304 [I 2022-10-10 23:12:50] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:28:09 [I 2022-10-10 23:12:50] ...trap/engines/engine.py.229: process: 0.86868 | load: 0.00319 [I 2022-10-10 23:12:50] ...trap/engines/engine.py.230: loss: 0.09811 [I 2022-10-10 23:12:50] ...optimizers/trijoint.py.151: total_norm: 10.163731 [I 2022-10-10 23:12:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 360/2304 [I 2022-10-10 23:12:58] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:28:03 [I 2022-10-10 23:12:58] ...trap/engines/engine.py.229: process: 0.87707 | load: 0.00307 [I 2022-10-10 23:12:58] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-10 23:12:58] ...optimizers/trijoint.py.151: total_norm: 11.806951 [I 2022-10-10 23:13:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 370/2304 [I 2022-10-10 23:13:07] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:50 [I 2022-10-10 23:13:07] ...trap/engines/engine.py.229: process: 0.86811 | load: 0.00307 [I 2022-10-10 23:13:07] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-10 23:13:07] ...optimizers/trijoint.py.151: total_norm: 14.073705 [I 2022-10-10 23:13:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 380/2304 [I 2022-10-10 23:13:16] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:48 [I 2022-10-10 23:13:16] ...trap/engines/engine.py.229: process: 0.85893 | load: 0.00311 [I 2022-10-10 23:13:16] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-10 23:13:16] ...optimizers/trijoint.py.151: total_norm: 59.057751 [I 2022-10-10 23:13:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 390/2304 [I 2022-10-10 23:13:24] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:29 [I 2022-10-10 23:13:24] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00318 [I 2022-10-10 23:13:24] ...trap/engines/engine.py.230: loss: 0.15794 [I 2022-10-10 23:13:24] ...optimizers/trijoint.py.151: total_norm: 15.523358 [I 2022-10-10 23:13:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 400/2304 [I 2022-10-10 23:13:33] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:27 [I 2022-10-10 23:13:33] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00309 [I 2022-10-10 23:13:33] ...trap/engines/engine.py.230: loss: 0.10094 [I 2022-10-10 23:13:33] ...optimizers/trijoint.py.151: total_norm: 21.696331 [I 2022-10-10 23:13:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 410/2304 [I 2022-10-10 23:13:42] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:27:18 [I 2022-10-10 23:13:42] ...trap/engines/engine.py.229: process: 0.87032 | load: 0.00318 [I 2022-10-10 23:13:42] ...trap/engines/engine.py.230: loss: 0.08570 [I 2022-10-10 23:13:42] ...optimizers/trijoint.py.151: total_norm: 19.788225 [I 2022-10-10 23:13:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 420/2304 [I 2022-10-10 23:13:50] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:27:02 [I 2022-10-10 23:13:50] ...trap/engines/engine.py.229: process: 0.84928 | load: 0.00320 [I 2022-10-10 23:13:50] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-10 23:13:50] ...optimizers/trijoint.py.151: total_norm: 13.686594 [I 2022-10-10 23:13:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 430/2304 [I 2022-10-10 23:13:59] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:27:00 [I 2022-10-10 23:13:59] ...trap/engines/engine.py.229: process: 0.86744 | load: 0.00337 [I 2022-10-10 23:13:59] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-10 23:13:59] ...optimizers/trijoint.py.151: total_norm: 14.315068 [I 2022-10-10 23:14:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 440/2304 [I 2022-10-10 23:14:08] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:51 [I 2022-10-10 23:14:08] ...trap/engines/engine.py.229: process: 0.87683 | load: 0.00308 [I 2022-10-10 23:14:08] ...trap/engines/engine.py.230: loss: 0.13757 [I 2022-10-10 23:14:08] ...optimizers/trijoint.py.151: total_norm: 16.115826 [I 2022-10-10 23:14:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 450/2304 [I 2022-10-10 23:14:16] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:45 [I 2022-10-10 23:14:16] ...trap/engines/engine.py.229: process: 0.86341 | load: 0.00310 [I 2022-10-10 23:14:16] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-10 23:14:16] ...optimizers/trijoint.py.151: total_norm: 17.319786 [I 2022-10-10 23:14:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 460/2304 [I 2022-10-10 23:14:25] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:26:28 [I 2022-10-10 23:14:25] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00314 [I 2022-10-10 23:14:25] ...trap/engines/engine.py.230: loss: 0.11160 [I 2022-10-10 23:14:25] ...optimizers/trijoint.py.151: total_norm: 12.430078 [I 2022-10-10 23:14:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 470/2304 [I 2022-10-10 23:14:34] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:26:22 [I 2022-10-10 23:14:34] ...trap/engines/engine.py.229: process: 0.86841 | load: 0.00311 [I 2022-10-10 23:14:34] ...trap/engines/engine.py.230: loss: 0.10714 [I 2022-10-10 23:14:34] ...optimizers/trijoint.py.151: total_norm: 24.318613 [I 2022-10-10 23:14:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 480/2304 [I 2022-10-10 23:14:42] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:26:23 [I 2022-10-10 23:14:42] ...trap/engines/engine.py.229: process: 0.87129 | load: 0.00315 [I 2022-10-10 23:14:42] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-10 23:14:42] ...optimizers/trijoint.py.151: total_norm: 22.832926 [I 2022-10-10 23:14:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 490/2304 [I 2022-10-10 23:14:51] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:26:19 [I 2022-10-10 23:14:51] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00308 [I 2022-10-10 23:14:51] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-10 23:14:51] ...optimizers/trijoint.py.151: total_norm: 27.068193 [I 2022-10-10 23:15:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 500/2304 [I 2022-10-10 23:15:00] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:26:01 [I 2022-10-10 23:15:00] ...trap/engines/engine.py.229: process: 0.86475 | load: 0.00344 [I 2022-10-10 23:15:00] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-10 23:15:00] ...optimizers/trijoint.py.151: total_norm: 16.597206 [I 2022-10-10 23:15:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 510/2304 [I 2022-10-10 23:15:08] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:54 [I 2022-10-10 23:15:08] ...trap/engines/engine.py.229: process: 0.88400 | load: 0.00315 [I 2022-10-10 23:15:08] ...trap/engines/engine.py.230: loss: 0.11234 [I 2022-10-10 23:15:08] ...optimizers/trijoint.py.151: total_norm: 11.261580 [I 2022-10-10 23:15:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 520/2304 [I 2022-10-10 23:15:17] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:25:39 [I 2022-10-10 23:15:17] ...trap/engines/engine.py.229: process: 0.86109 | load: 0.00324 [I 2022-10-10 23:15:17] ...trap/engines/engine.py.230: loss: 0.09452 [I 2022-10-10 23:15:17] ...optimizers/trijoint.py.151: total_norm: 12.088986 [I 2022-10-10 23:15:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 530/2304 [I 2022-10-10 23:15:26] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:25:33 [I 2022-10-10 23:15:26] ...trap/engines/engine.py.229: process: 0.85983 | load: 0.00308 [I 2022-10-10 23:15:26] ...trap/engines/engine.py.230: loss: 0.09690 [I 2022-10-10 23:15:26] ...optimizers/trijoint.py.151: total_norm: 13.384376 [I 2022-10-10 23:15:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 540/2304 [I 2022-10-10 23:15:34] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:25:25 [I 2022-10-10 23:15:34] ...trap/engines/engine.py.229: process: 0.86959 | load: 0.00316 [I 2022-10-10 23:15:34] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-10 23:15:34] ...optimizers/trijoint.py.151: total_norm: 32.290283 [I 2022-10-10 23:15:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 550/2304 [I 2022-10-10 23:15:43] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:06 [I 2022-10-10 23:15:43] ...trap/engines/engine.py.229: process: 0.86768 | load: 0.00312 [I 2022-10-10 23:15:43] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-10 23:15:43] ...optimizers/trijoint.py.151: total_norm: 10.689824 [I 2022-10-10 23:15:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 560/2304 [I 2022-10-10 23:15:52] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:25:00 [I 2022-10-10 23:15:52] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00311 [I 2022-10-10 23:15:52] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-10 23:15:52] ...optimizers/trijoint.py.151: total_norm: 14.058023 [I 2022-10-10 23:16:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 570/2304 [I 2022-10-10 23:16:00] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:58 [I 2022-10-10 23:16:00] ...trap/engines/engine.py.229: process: 0.86142 | load: 0.00310 [I 2022-10-10 23:16:00] ...trap/engines/engine.py.230: loss: 0.08123 [I 2022-10-10 23:16:00] ...optimizers/trijoint.py.151: total_norm: 21.687508 [I 2022-10-10 23:16:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 580/2304 [I 2022-10-10 23:16:09] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:24:40 [I 2022-10-10 23:16:09] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00298 [I 2022-10-10 23:16:09] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-10 23:16:09] ...optimizers/trijoint.py.151: total_norm: 11.801784 [I 2022-10-10 23:16:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 590/2304 [I 2022-10-10 23:16:18] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:25 [I 2022-10-10 23:16:18] ...trap/engines/engine.py.229: process: 0.86727 | load: 0.00301 [I 2022-10-10 23:16:18] ...trap/engines/engine.py.230: loss: 0.09377 [I 2022-10-10 23:16:18] ...optimizers/trijoint.py.151: total_norm: 11.338739 [I 2022-10-10 23:16:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 600/2304 [I 2022-10-10 23:16:26] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:24:24 [I 2022-10-10 23:16:26] ...trap/engines/engine.py.229: process: 0.87043 | load: 0.00292 [I 2022-10-10 23:16:26] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-10 23:16:26] ...optimizers/trijoint.py.151: total_norm: 17.865219 [I 2022-10-10 23:16:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 610/2304 [I 2022-10-10 23:16:35] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:24:10 [I 2022-10-10 23:16:35] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00297 [I 2022-10-10 23:16:35] ...trap/engines/engine.py.230: loss: 0.10664 [I 2022-10-10 23:16:35] ...optimizers/trijoint.py.151: total_norm: 14.647496 [I 2022-10-10 23:16:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 620/2304 [I 2022-10-10 23:16:43] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:57 [I 2022-10-10 23:16:43] ...trap/engines/engine.py.229: process: 0.85736 | load: 0.00300 [I 2022-10-10 23:16:43] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-10 23:16:43] ...optimizers/trijoint.py.151: total_norm: 17.712439 [I 2022-10-10 23:16:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 630/2304 [I 2022-10-10 23:16:52] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:48 [I 2022-10-10 23:16:52] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00293 [I 2022-10-10 23:16:52] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-10 23:16:52] ...optimizers/trijoint.py.151: total_norm: 9.647305 [I 2022-10-10 23:17:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 640/2304 [I 2022-10-10 23:17:00] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:36 [I 2022-10-10 23:17:00] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00294 [I 2022-10-10 23:17:00] ...trap/engines/engine.py.230: loss: 0.15168 [I 2022-10-10 23:17:00] ...optimizers/trijoint.py.151: total_norm: 19.670622 [I 2022-10-10 23:17:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 650/2304 [I 2022-10-10 23:17:09] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:28 [I 2022-10-10 23:17:09] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00303 [I 2022-10-10 23:17:09] ...trap/engines/engine.py.230: loss: 0.09179 [I 2022-10-10 23:17:09] ...optimizers/trijoint.py.151: total_norm: 14.189269 [I 2022-10-10 23:17:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 660/2304 [I 2022-10-10 23:17:17] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:18 [I 2022-10-10 23:17:17] ...trap/engines/engine.py.229: process: 0.84959 | load: 0.00291 [I 2022-10-10 23:17:17] ...trap/engines/engine.py.230: loss: 0.15537 [I 2022-10-10 23:17:17] ...optimizers/trijoint.py.151: total_norm: 9.251114 [I 2022-10-10 23:17:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 670/2304 [I 2022-10-10 23:17:26] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:23:15 [I 2022-10-10 23:17:26] ...trap/engines/engine.py.229: process: 0.84844 | load: 0.00293 [I 2022-10-10 23:17:26] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-10 23:17:26] ...optimizers/trijoint.py.151: total_norm: 34.598751 [I 2022-10-10 23:17:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 680/2304 [I 2022-10-10 23:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:23:11 [I 2022-10-10 23:17:35] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00293 [I 2022-10-10 23:17:35] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-10 23:17:35] ...optimizers/trijoint.py.151: total_norm: 20.390312 [I 2022-10-10 23:17:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 690/2304 [I 2022-10-10 23:17:43] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:23:01 [I 2022-10-10 23:17:43] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00303 [I 2022-10-10 23:17:43] ...trap/engines/engine.py.230: loss: 0.22238 [I 2022-10-10 23:17:43] ...optimizers/trijoint.py.151: total_norm: 90.762665 [I 2022-10-10 23:17:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 700/2304 [I 2022-10-10 23:17:52] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:22:46 [I 2022-10-10 23:17:52] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00305 [I 2022-10-10 23:17:52] ...trap/engines/engine.py.230: loss: 0.15359 [I 2022-10-10 23:17:52] ...optimizers/trijoint.py.151: total_norm: 50.189445 [I 2022-10-10 23:18:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 710/2304 [I 2022-10-10 23:18:00] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:22:41 [I 2022-10-10 23:18:00] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00306 [I 2022-10-10 23:18:00] ...trap/engines/engine.py.230: loss: 0.07965 [I 2022-10-10 23:18:00] ...optimizers/trijoint.py.151: total_norm: 10.007197 [I 2022-10-10 23:18:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 720/2304 [I 2022-10-10 23:18:09] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:22:31 [I 2022-10-10 23:18:09] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00307 [I 2022-10-10 23:18:09] ...trap/engines/engine.py.230: loss: 0.14204 [I 2022-10-10 23:18:09] ...optimizers/trijoint.py.151: total_norm: 13.694492 [I 2022-10-10 23:18:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 730/2304 [I 2022-10-10 23:18:17] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:22:20 [I 2022-10-10 23:18:17] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00308 [I 2022-10-10 23:18:17] ...trap/engines/engine.py.230: loss: 0.12557 [I 2022-10-10 23:18:17] ...optimizers/trijoint.py.151: total_norm: 13.780095 [I 2022-10-10 23:18:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 740/2304 [I 2022-10-10 23:18:26] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:22:16 [I 2022-10-10 23:18:26] ...trap/engines/engine.py.229: process: 0.87245 | load: 0.00316 [I 2022-10-10 23:18:26] ...trap/engines/engine.py.230: loss: 0.10457 [I 2022-10-10 23:18:26] ...optimizers/trijoint.py.151: total_norm: 12.039613 [I 2022-10-10 23:18:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 750/2304 [I 2022-10-10 23:18:35] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:22:12 [I 2022-10-10 23:18:35] ...trap/engines/engine.py.229: process: 0.85615 | load: 0.00307 [I 2022-10-10 23:18:35] ...trap/engines/engine.py.230: loss: 0.10537 [I 2022-10-10 23:18:35] ...optimizers/trijoint.py.151: total_norm: 37.520348 [I 2022-10-10 23:18:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 760/2304 [I 2022-10-10 23:18:43] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:22:14 [I 2022-10-10 23:18:43] ...trap/engines/engine.py.229: process: 0.87066 | load: 0.00309 [I 2022-10-10 23:18:43] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-10 23:18:43] ...optimizers/trijoint.py.151: total_norm: 18.255310 [I 2022-10-10 23:18:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 770/2304 [I 2022-10-10 23:18:52] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:22:08 [I 2022-10-10 23:18:52] ...trap/engines/engine.py.229: process: 0.86011 | load: 0.00308 [I 2022-10-10 23:18:52] ...trap/engines/engine.py.230: loss: 0.09749 [I 2022-10-10 23:18:52] ...optimizers/trijoint.py.151: total_norm: 19.005253 [I 2022-10-10 23:19:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 780/2304 [I 2022-10-10 23:19:01] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:22:07 [I 2022-10-10 23:19:01] ...trap/engines/engine.py.229: process: 0.86940 | load: 0.00310 [I 2022-10-10 23:19:01] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-10 23:19:01] ...optimizers/trijoint.py.151: total_norm: 8.169097 [I 2022-10-10 23:19:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 790/2304 [I 2022-10-10 23:19:09] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:40 [I 2022-10-10 23:19:09] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00314 [I 2022-10-10 23:19:09] ...trap/engines/engine.py.230: loss: 0.10351 [I 2022-10-10 23:19:09] ...optimizers/trijoint.py.151: total_norm: 11.534053 [I 2022-10-10 23:19:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 800/2304 [I 2022-10-10 23:19:18] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:41 [I 2022-10-10 23:19:18] ...trap/engines/engine.py.229: process: 0.87281 | load: 0.00309 [I 2022-10-10 23:19:18] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-10 23:19:18] ...optimizers/trijoint.py.151: total_norm: 13.404199 [I 2022-10-10 23:19:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 810/2304 [I 2022-10-10 23:19:27] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:34 [I 2022-10-10 23:19:27] ...trap/engines/engine.py.229: process: 0.87015 | load: 0.00309 [I 2022-10-10 23:19:27] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-10 23:19:27] ...optimizers/trijoint.py.151: total_norm: 16.040176 [I 2022-10-10 23:19:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 820/2304 [I 2022-10-10 23:19:35] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:21:19 [I 2022-10-10 23:19:35] ...trap/engines/engine.py.229: process: 0.86593 | load: 0.00316 [I 2022-10-10 23:19:35] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-10 23:19:35] ...optimizers/trijoint.py.151: total_norm: 21.287561 [I 2022-10-10 23:19:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 830/2304 [I 2022-10-10 23:19:44] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:21:27 [I 2022-10-10 23:19:44] ...trap/engines/engine.py.229: process: 0.86809 | load: 0.00318 [I 2022-10-10 23:19:44] ...trap/engines/engine.py.230: loss: 0.16022 [I 2022-10-10 23:19:44] ...optimizers/trijoint.py.151: total_norm: 23.686800 [I 2022-10-10 23:19:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 840/2304 [I 2022-10-10 23:19:53] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:21:10 [I 2022-10-10 23:19:53] ...trap/engines/engine.py.229: process: 0.86597 | load: 0.00315 [I 2022-10-10 23:19:53] ...trap/engines/engine.py.230: loss: 0.17769 [I 2022-10-10 23:19:53] ...optimizers/trijoint.py.151: total_norm: 24.186163 [I 2022-10-10 23:20:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 850/2304 [I 2022-10-10 23:20:01] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:21:04 [I 2022-10-10 23:20:01] ...trap/engines/engine.py.229: process: 0.87317 | load: 0.00315 [I 2022-10-10 23:20:01] ...trap/engines/engine.py.230: loss: 0.10158 [I 2022-10-10 23:20:01] ...optimizers/trijoint.py.151: total_norm: 14.638102 [I 2022-10-10 23:20:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 860/2304 [I 2022-10-10 23:20:10] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:20:58 [I 2022-10-10 23:20:10] ...trap/engines/engine.py.229: process: 0.87977 | load: 0.00335 [I 2022-10-10 23:20:10] ...trap/engines/engine.py.230: loss: 0.20343 [I 2022-10-10 23:20:10] ...optimizers/trijoint.py.151: total_norm: 31.662317 [I 2022-10-10 23:20:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 870/2304 [I 2022-10-10 23:20:19] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:20:44 [I 2022-10-10 23:20:19] ...trap/engines/engine.py.229: process: 0.87958 | load: 0.00314 [I 2022-10-10 23:20:19] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-10 23:20:19] ...optimizers/trijoint.py.151: total_norm: 13.363042 [I 2022-10-10 23:20:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 880/2304 [I 2022-10-10 23:20:27] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:26 [I 2022-10-10 23:20:27] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00324 [I 2022-10-10 23:20:27] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-10 23:20:28] ...optimizers/trijoint.py.151: total_norm: 16.152233 [I 2022-10-10 23:20:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 890/2304 [I 2022-10-10 23:20:36] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:20:27 [I 2022-10-10 23:20:36] ...trap/engines/engine.py.229: process: 0.86163 | load: 0.00311 [I 2022-10-10 23:20:36] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-10 23:20:36] ...optimizers/trijoint.py.151: total_norm: 26.050961 [I 2022-10-10 23:20:45] ...trap/engines/engine.py.224: train: epoch 89 | batch 900/2304 [I 2022-10-10 23:20:45] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:20:09 [I 2022-10-10 23:20:45] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00327 [I 2022-10-10 23:20:45] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-10 23:20:45] ...optimizers/trijoint.py.151: total_norm: 39.146503 [I 2022-10-10 23:20:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 910/2304 [I 2022-10-10 23:20:54] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:20:01 [I 2022-10-10 23:20:54] ...trap/engines/engine.py.229: process: 0.86297 | load: 0.00314 [I 2022-10-10 23:20:54] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-10 23:20:54] ...optimizers/trijoint.py.151: total_norm: 9.941737 [I 2022-10-10 23:21:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 920/2304 [I 2022-10-10 23:21:02] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:19:59 [I 2022-10-10 23:21:02] ...trap/engines/engine.py.229: process: 0.87441 | load: 0.00350 [I 2022-10-10 23:21:02] ...trap/engines/engine.py.230: loss: 0.23673 [I 2022-10-10 23:21:02] ...optimizers/trijoint.py.151: total_norm: 41.475685 [I 2022-10-10 23:21:11] ...trap/engines/engine.py.224: train: epoch 89 | batch 930/2304 [I 2022-10-10 23:21:11] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:54 [I 2022-10-10 23:21:11] ...trap/engines/engine.py.229: process: 0.87012 | load: 0.00314 [I 2022-10-10 23:21:11] ...trap/engines/engine.py.230: loss: 0.12678 [I 2022-10-10 23:21:11] ...optimizers/trijoint.py.151: total_norm: 16.843513 [I 2022-10-10 23:21:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 940/2304 [I 2022-10-10 23:21:20] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:43 [I 2022-10-10 23:21:20] ...trap/engines/engine.py.229: process: 0.86406 | load: 0.00316 [I 2022-10-10 23:21:20] ...trap/engines/engine.py.230: loss: 0.09219 [I 2022-10-10 23:21:20] ...optimizers/trijoint.py.151: total_norm: 13.224804 [I 2022-10-10 23:21:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 950/2304 [I 2022-10-10 23:21:28] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:32 [I 2022-10-10 23:21:28] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00308 [I 2022-10-10 23:21:28] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-10 23:21:28] ...optimizers/trijoint.py.151: total_norm: 8.089009 [I 2022-10-10 23:21:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 960/2304 [I 2022-10-10 23:21:37] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:20 [I 2022-10-10 23:21:37] ...trap/engines/engine.py.229: process: 0.86842 | load: 0.00324 [I 2022-10-10 23:21:37] ...trap/engines/engine.py.230: loss: 0.11006 [I 2022-10-10 23:21:37] ...optimizers/trijoint.py.151: total_norm: 11.903128 [I 2022-10-10 23:21:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 970/2304 [I 2022-10-10 23:21:46] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:16 [I 2022-10-10 23:21:46] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00325 [I 2022-10-10 23:21:46] ...trap/engines/engine.py.230: loss: 0.11953 [I 2022-10-10 23:21:46] ...optimizers/trijoint.py.151: total_norm: 17.888924 [I 2022-10-10 23:21:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 980/2304 [I 2022-10-10 23:21:54] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:19:15 [I 2022-10-10 23:21:54] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00336 [I 2022-10-10 23:21:54] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-10 23:21:54] ...optimizers/trijoint.py.151: total_norm: 13.246164 [I 2022-10-10 23:22:03] ...trap/engines/engine.py.224: train: epoch 89 | batch 990/2304 [I 2022-10-10 23:22:03] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:19:06 [I 2022-10-10 23:22:03] ...trap/engines/engine.py.229: process: 0.87658 | load: 0.00320 [I 2022-10-10 23:22:03] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-10 23:22:03] ...optimizers/trijoint.py.151: total_norm: 24.051687 [I 2022-10-10 23:22:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1000/2304 [I 2022-10-10 23:22:12] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:51 [I 2022-10-10 23:22:12] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00318 [I 2022-10-10 23:22:12] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-10 23:22:12] ...optimizers/trijoint.py.151: total_norm: 13.923842 [I 2022-10-10 23:22:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 1010/2304 [I 2022-10-10 23:22:20] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:39 [I 2022-10-10 23:22:20] ...trap/engines/engine.py.229: process: 0.86815 | load: 0.00293 [I 2022-10-10 23:22:20] ...trap/engines/engine.py.230: loss: 0.09044 [I 2022-10-10 23:22:20] ...optimizers/trijoint.py.151: total_norm: 16.819757 [I 2022-10-10 23:22:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1020/2304 [I 2022-10-10 23:22:29] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:23 [I 2022-10-10 23:22:29] ...trap/engines/engine.py.229: process: 0.87461 | load: 0.00296 [I 2022-10-10 23:22:29] ...trap/engines/engine.py.230: loss: 0.14887 [I 2022-10-10 23:22:29] ...optimizers/trijoint.py.151: total_norm: 11.754214 [I 2022-10-10 23:22:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 1030/2304 [I 2022-10-10 23:22:38] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:13 [I 2022-10-10 23:22:38] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00295 [I 2022-10-10 23:22:38] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-10 23:22:38] ...optimizers/trijoint.py.151: total_norm: 8.644955 [I 2022-10-10 23:22:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 1040/2304 [I 2022-10-10 23:22:46] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:00 [I 2022-10-10 23:22:46] ...trap/engines/engine.py.229: process: 0.85122 | load: 0.00296 [I 2022-10-10 23:22:46] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-10 23:22:46] ...optimizers/trijoint.py.151: total_norm: 15.961992 [I 2022-10-10 23:22:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 1050/2304 [I 2022-10-10 23:22:55] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:45 [I 2022-10-10 23:22:55] ...trap/engines/engine.py.229: process: 0.84832 | load: 0.00293 [I 2022-10-10 23:22:55] ...trap/engines/engine.py.230: loss: 0.16733 [I 2022-10-10 23:22:55] ...optimizers/trijoint.py.151: total_norm: 18.967331 [I 2022-10-10 23:23:03] ...trap/engines/engine.py.224: train: epoch 89 | batch 1060/2304 [I 2022-10-10 23:23:03] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:33 [I 2022-10-10 23:23:03] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00298 [I 2022-10-10 23:23:03] ...trap/engines/engine.py.230: loss: 0.09557 [I 2022-10-10 23:23:03] ...optimizers/trijoint.py.151: total_norm: 11.690556 [I 2022-10-10 23:23:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1070/2304 [I 2022-10-10 23:23:12] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:34 [I 2022-10-10 23:23:12] ...trap/engines/engine.py.229: process: 0.86055 | load: 0.00297 [I 2022-10-10 23:23:12] ...trap/engines/engine.py.230: loss: 0.16229 [I 2022-10-10 23:23:12] ...optimizers/trijoint.py.151: total_norm: 38.781155 [I 2022-10-10 23:23:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 1080/2304 [I 2022-10-10 23:23:20] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:24 [I 2022-10-10 23:23:20] ...trap/engines/engine.py.229: process: 0.85667 | load: 0.00298 [I 2022-10-10 23:23:20] ...trap/engines/engine.py.230: loss: 0.08861 [I 2022-10-10 23:23:20] ...optimizers/trijoint.py.151: total_norm: 18.624220 [I 2022-10-10 23:23:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1090/2304 [I 2022-10-10 23:23:29] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:17:20 [I 2022-10-10 23:23:29] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00299 [I 2022-10-10 23:23:29] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-10 23:23:29] ...optimizers/trijoint.py.151: total_norm: 11.255346 [I 2022-10-10 23:23:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 1100/2304 [I 2022-10-10 23:23:37] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:17:10 [I 2022-10-10 23:23:37] ...trap/engines/engine.py.229: process: 0.86319 | load: 0.00304 [I 2022-10-10 23:23:37] ...trap/engines/engine.py.230: loss: 0.14333 [I 2022-10-10 23:23:37] ...optimizers/trijoint.py.151: total_norm: 12.841691 [I 2022-10-10 23:23:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 1110/2304 [I 2022-10-10 23:23:46] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:55 [I 2022-10-10 23:23:46] ...trap/engines/engine.py.229: process: 0.84811 | load: 0.00295 [I 2022-10-10 23:23:46] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-10 23:23:46] ...optimizers/trijoint.py.151: total_norm: 19.085058 [I 2022-10-10 23:23:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 1120/2304 [I 2022-10-10 23:23:54] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:43 [I 2022-10-10 23:23:54] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00296 [I 2022-10-10 23:23:54] ...trap/engines/engine.py.230: loss: 0.10156 [I 2022-10-10 23:23:54] ...optimizers/trijoint.py.151: total_norm: 9.643703 [I 2022-10-10 23:24:03] ...trap/engines/engine.py.224: train: epoch 89 | batch 1130/2304 [I 2022-10-10 23:24:03] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:16:49 [I 2022-10-10 23:24:03] ...trap/engines/engine.py.229: process: 0.86165 | load: 0.00338 [I 2022-10-10 23:24:03] ...trap/engines/engine.py.230: loss: 0.08204 [I 2022-10-10 23:24:03] ...optimizers/trijoint.py.151: total_norm: 14.081878 [I 2022-10-10 23:24:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1140/2304 [I 2022-10-10 23:24:12] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:16:40 [I 2022-10-10 23:24:12] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00310 [I 2022-10-10 23:24:12] ...trap/engines/engine.py.230: loss: 0.13699 [I 2022-10-10 23:24:12] ...optimizers/trijoint.py.151: total_norm: 17.852114 [I 2022-10-10 23:24:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 1150/2304 [I 2022-10-10 23:24:20] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:36 [I 2022-10-10 23:24:20] ...trap/engines/engine.py.229: process: 0.86888 | load: 0.00317 [I 2022-10-10 23:24:20] ...trap/engines/engine.py.230: loss: 0.17871 [I 2022-10-10 23:24:20] ...optimizers/trijoint.py.151: total_norm: 16.588774 [I 2022-10-10 23:24:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1160/2304 [I 2022-10-10 23:24:29] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:23 [I 2022-10-10 23:24:29] ...trap/engines/engine.py.229: process: 0.85042 | load: 0.00299 [I 2022-10-10 23:24:29] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-10 23:24:29] ...optimizers/trijoint.py.151: total_norm: 15.579385 [I 2022-10-10 23:24:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 1170/2304 [I 2022-10-10 23:24:38] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:10 [I 2022-10-10 23:24:38] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00298 [I 2022-10-10 23:24:38] ...trap/engines/engine.py.230: loss: 0.10769 [I 2022-10-10 23:24:38] ...optimizers/trijoint.py.151: total_norm: 13.622367 [I 2022-10-10 23:24:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 1180/2304 [I 2022-10-10 23:24:46] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:57 [I 2022-10-10 23:24:46] ...trap/engines/engine.py.229: process: 0.84570 | load: 0.00300 [I 2022-10-10 23:24:46] ...trap/engines/engine.py.230: loss: 0.10318 [I 2022-10-10 23:24:46] ...optimizers/trijoint.py.151: total_norm: 12.943710 [I 2022-10-10 23:24:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 1190/2304 [I 2022-10-10 23:24:55] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:49 [I 2022-10-10 23:24:55] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00298 [I 2022-10-10 23:24:55] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-10 23:24:55] ...optimizers/trijoint.py.151: total_norm: 7.186677 [I 2022-10-10 23:25:03] ...trap/engines/engine.py.224: train: epoch 89 | batch 1200/2304 [I 2022-10-10 23:25:03] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:50 [I 2022-10-10 23:25:03] ...trap/engines/engine.py.229: process: 0.86463 | load: 0.00313 [I 2022-10-10 23:25:03] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-10 23:25:03] ...optimizers/trijoint.py.151: total_norm: 17.615231 [I 2022-10-10 23:25:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1210/2304 [I 2022-10-10 23:25:12] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:49 [I 2022-10-10 23:25:12] ...trap/engines/engine.py.229: process: 0.86693 | load: 0.00319 [I 2022-10-10 23:25:12] ...trap/engines/engine.py.230: loss: 0.11040 [I 2022-10-10 23:25:12] ...optimizers/trijoint.py.151: total_norm: 15.496607 [I 2022-10-10 23:25:21] ...trap/engines/engine.py.224: train: epoch 89 | batch 1220/2304 [I 2022-10-10 23:25:21] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:39 [I 2022-10-10 23:25:21] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00311 [I 2022-10-10 23:25:21] ...trap/engines/engine.py.230: loss: 0.13623 [I 2022-10-10 23:25:21] ...optimizers/trijoint.py.151: total_norm: 20.822474 [I 2022-10-10 23:25:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1230/2304 [I 2022-10-10 23:25:29] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:29 [I 2022-10-10 23:25:29] ...trap/engines/engine.py.229: process: 0.86540 | load: 0.00311 [I 2022-10-10 23:25:29] ...trap/engines/engine.py.230: loss: 0.07958 [I 2022-10-10 23:25:29] ...optimizers/trijoint.py.151: total_norm: 15.906020 [I 2022-10-10 23:25:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 1240/2304 [I 2022-10-10 23:25:38] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:21 [I 2022-10-10 23:25:38] ...trap/engines/engine.py.229: process: 0.86182 | load: 0.00306 [I 2022-10-10 23:25:38] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-10 23:25:38] ...optimizers/trijoint.py.151: total_norm: 24.617661 [I 2022-10-10 23:25:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 1250/2304 [I 2022-10-10 23:25:47] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:15:11 [I 2022-10-10 23:25:47] ...trap/engines/engine.py.229: process: 0.86007 | load: 0.00310 [I 2022-10-10 23:25:47] ...trap/engines/engine.py.230: loss: 0.16385 [I 2022-10-10 23:25:47] ...optimizers/trijoint.py.151: total_norm: 12.474630 [I 2022-10-10 23:25:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 1260/2304 [I 2022-10-10 23:25:55] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:15:03 [I 2022-10-10 23:25:55] ...trap/engines/engine.py.229: process: 0.87292 | load: 0.00307 [I 2022-10-10 23:25:55] ...trap/engines/engine.py.230: loss: 0.10540 [I 2022-10-10 23:25:55] ...optimizers/trijoint.py.151: total_norm: 10.033192 [I 2022-10-10 23:26:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 1270/2304 [I 2022-10-10 23:26:04] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:57 [I 2022-10-10 23:26:04] ...trap/engines/engine.py.229: process: 0.86490 | load: 0.00319 [I 2022-10-10 23:26:04] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-10 23:26:04] ...optimizers/trijoint.py.151: total_norm: 10.801095 [I 2022-10-10 23:26:13] ...trap/engines/engine.py.224: train: epoch 89 | batch 1280/2304 [I 2022-10-10 23:26:13] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:44 [I 2022-10-10 23:26:13] ...trap/engines/engine.py.229: process: 0.86637 | load: 0.00316 [I 2022-10-10 23:26:13] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-10 23:26:13] ...optimizers/trijoint.py.151: total_norm: 16.234299 [I 2022-10-10 23:26:21] ...trap/engines/engine.py.224: train: epoch 89 | batch 1290/2304 [I 2022-10-10 23:26:21] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:34 [I 2022-10-10 23:26:21] ...trap/engines/engine.py.229: process: 0.86731 | load: 0.00317 [I 2022-10-10 23:26:21] ...trap/engines/engine.py.230: loss: 0.14910 [I 2022-10-10 23:26:21] ...optimizers/trijoint.py.151: total_norm: 11.894838 [I 2022-10-10 23:26:30] ...trap/engines/engine.py.224: train: epoch 89 | batch 1300/2304 [I 2022-10-10 23:26:30] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:14:30 [I 2022-10-10 23:26:30] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00309 [I 2022-10-10 23:26:30] ...trap/engines/engine.py.230: loss: 0.14959 [I 2022-10-10 23:26:30] ...optimizers/trijoint.py.151: total_norm: 31.869389 [I 2022-10-10 23:26:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 1310/2304 [I 2022-10-10 23:26:39] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:14:10 [I 2022-10-10 23:26:39] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00317 [I 2022-10-10 23:26:39] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-10 23:26:39] ...optimizers/trijoint.py.151: total_norm: 23.356522 [I 2022-10-10 23:26:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 1320/2304 [I 2022-10-10 23:26:47] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:04 [I 2022-10-10 23:26:47] ...trap/engines/engine.py.229: process: 0.86734 | load: 0.00325 [I 2022-10-10 23:26:47] ...trap/engines/engine.py.230: loss: 0.10985 [I 2022-10-10 23:26:47] ...optimizers/trijoint.py.151: total_norm: 9.881964 [I 2022-10-10 23:26:56] ...trap/engines/engine.py.224: train: epoch 89 | batch 1330/2304 [I 2022-10-10 23:26:56] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:14:02 [I 2022-10-10 23:26:56] ...trap/engines/engine.py.229: process: 0.87262 | load: 0.00316 [I 2022-10-10 23:26:56] ...trap/engines/engine.py.230: loss: 0.18466 [I 2022-10-10 23:26:56] ...optimizers/trijoint.py.151: total_norm: 33.957153 [I 2022-10-10 23:27:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 1340/2304 [I 2022-10-10 23:27:05] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:13:51 [I 2022-10-10 23:27:05] ...trap/engines/engine.py.229: process: 0.87368 | load: 0.00307 [I 2022-10-10 23:27:05] ...trap/engines/engine.py.230: loss: 0.20398 [I 2022-10-10 23:27:05] ...optimizers/trijoint.py.151: total_norm: 15.089583 [I 2022-10-10 23:27:13] ...trap/engines/engine.py.224: train: epoch 89 | batch 1350/2304 [I 2022-10-10 23:27:13] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:43 [I 2022-10-10 23:27:13] ...trap/engines/engine.py.229: process: 0.87252 | load: 0.00316 [I 2022-10-10 23:27:13] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-10 23:27:13] ...optimizers/trijoint.py.151: total_norm: 39.496292 [I 2022-10-10 23:27:22] ...trap/engines/engine.py.224: train: epoch 89 | batch 1360/2304 [I 2022-10-10 23:27:22] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:34 [I 2022-10-10 23:27:22] ...trap/engines/engine.py.229: process: 0.86844 | load: 0.00310 [I 2022-10-10 23:27:22] ...trap/engines/engine.py.230: loss: 0.07346 [I 2022-10-10 23:27:22] ...optimizers/trijoint.py.151: total_norm: 11.582416 [I 2022-10-10 23:27:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 1370/2304 [I 2022-10-10 23:27:31] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:13:21 [I 2022-10-10 23:27:31] ...trap/engines/engine.py.229: process: 0.85782 | load: 0.00310 [I 2022-10-10 23:27:31] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-10 23:27:31] ...optimizers/trijoint.py.151: total_norm: 29.414276 [I 2022-10-10 23:27:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 1380/2304 [I 2022-10-10 23:27:39] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:16 [I 2022-10-10 23:27:39] ...trap/engines/engine.py.229: process: 0.86248 | load: 0.00308 [I 2022-10-10 23:27:39] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-10 23:27:39] ...optimizers/trijoint.py.151: total_norm: 19.543360 [I 2022-10-10 23:27:48] ...trap/engines/engine.py.224: train: epoch 89 | batch 1390/2304 [I 2022-10-10 23:27:48] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:13:10 [I 2022-10-10 23:27:48] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00309 [I 2022-10-10 23:27:48] ...trap/engines/engine.py.230: loss: 0.09479 [I 2022-10-10 23:27:48] ...optimizers/trijoint.py.151: total_norm: 9.694840 [I 2022-10-10 23:27:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1400/2304 [I 2022-10-10 23:27:57] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:13:04 [I 2022-10-10 23:27:57] ...trap/engines/engine.py.229: process: 0.87974 | load: 0.00317 [I 2022-10-10 23:27:57] ...trap/engines/engine.py.230: loss: 0.10794 [I 2022-10-10 23:27:57] ...optimizers/trijoint.py.151: total_norm: 8.992120 [I 2022-10-10 23:28:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 1410/2304 [I 2022-10-10 23:28:05] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:53 [I 2022-10-10 23:28:05] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00323 [I 2022-10-10 23:28:05] ...trap/engines/engine.py.230: loss: 0.10605 [I 2022-10-10 23:28:05] ...optimizers/trijoint.py.151: total_norm: 12.060151 [I 2022-10-10 23:28:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1420/2304 [I 2022-10-10 23:28:14] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:49 [I 2022-10-10 23:28:14] ...trap/engines/engine.py.229: process: 0.87132 | load: 0.00312 [I 2022-10-10 23:28:14] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-10 23:28:14] ...optimizers/trijoint.py.151: total_norm: 11.837919 [I 2022-10-10 23:28:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1430/2304 [I 2022-10-10 23:28:23] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:35 [I 2022-10-10 23:28:23] ...trap/engines/engine.py.229: process: 0.87219 | load: 0.00321 [I 2022-10-10 23:28:23] ...trap/engines/engine.py.230: loss: 0.15515 [I 2022-10-10 23:28:23] ...optimizers/trijoint.py.151: total_norm: 13.167679 [I 2022-10-10 23:28:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 1440/2304 [I 2022-10-10 23:28:31] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:24 [I 2022-10-10 23:28:31] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00323 [I 2022-10-10 23:28:31] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-10 23:28:31] ...optimizers/trijoint.py.151: total_norm: 35.620564 [I 2022-10-10 23:28:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1450/2304 [I 2022-10-10 23:28:40] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:12:16 [I 2022-10-10 23:28:40] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00309 [I 2022-10-10 23:28:40] ...trap/engines/engine.py.230: loss: 0.13929 [I 2022-10-10 23:28:40] ...optimizers/trijoint.py.151: total_norm: 29.103977 [I 2022-10-10 23:28:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 1460/2304 [I 2022-10-10 23:28:49] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:12:10 [I 2022-10-10 23:28:49] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00310 [I 2022-10-10 23:28:49] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-10 23:28:49] ...optimizers/trijoint.py.151: total_norm: 16.793673 [I 2022-10-10 23:28:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1470/2304 [I 2022-10-10 23:28:57] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:12:01 [I 2022-10-10 23:28:57] ...trap/engines/engine.py.229: process: 0.86380 | load: 0.00309 [I 2022-10-10 23:28:57] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-10 23:28:57] ...optimizers/trijoint.py.151: total_norm: 14.313651 [I 2022-10-10 23:29:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1480/2304 [I 2022-10-10 23:29:06] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:37 [I 2022-10-10 23:29:06] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00323 [I 2022-10-10 23:29:06] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-10 23:29:06] ...optimizers/trijoint.py.151: total_norm: 12.948344 [I 2022-10-10 23:29:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1490/2304 [I 2022-10-10 23:29:14] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:37 [I 2022-10-10 23:29:14] ...trap/engines/engine.py.229: process: 0.85922 | load: 0.00307 [I 2022-10-10 23:29:14] ...trap/engines/engine.py.230: loss: 0.12191 [I 2022-10-10 23:29:14] ...optimizers/trijoint.py.151: total_norm: 10.655492 [I 2022-10-10 23:29:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1500/2304 [I 2022-10-10 23:29:23] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:25 [I 2022-10-10 23:29:23] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00297 [I 2022-10-10 23:29:23] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-10 23:29:23] ...optimizers/trijoint.py.151: total_norm: 15.473074 [I 2022-10-10 23:29:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 1510/2304 [I 2022-10-10 23:29:31] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:19 [I 2022-10-10 23:29:31] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00294 [I 2022-10-10 23:29:31] ...trap/engines/engine.py.230: loss: 0.09992 [I 2022-10-10 23:29:31] ...optimizers/trijoint.py.151: total_norm: 9.050400 [I 2022-10-10 23:29:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1520/2304 [I 2022-10-10 23:29:40] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:09 [I 2022-10-10 23:29:40] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00297 [I 2022-10-10 23:29:40] ...trap/engines/engine.py.230: loss: 0.11164 [I 2022-10-10 23:29:40] ...optimizers/trijoint.py.151: total_norm: 20.707331 [I 2022-10-10 23:29:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 1530/2304 [I 2022-10-10 23:29:49] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:11:01 [I 2022-10-10 23:29:49] ...trap/engines/engine.py.229: process: 0.86530 | load: 0.00305 [I 2022-10-10 23:29:49] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-10 23:29:49] ...optimizers/trijoint.py.151: total_norm: 13.817266 [I 2022-10-10 23:29:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1540/2304 [I 2022-10-10 23:29:57] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:54 [I 2022-10-10 23:29:57] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00305 [I 2022-10-10 23:29:57] ...trap/engines/engine.py.230: loss: 0.11308 [I 2022-10-10 23:29:57] ...optimizers/trijoint.py.151: total_norm: 24.393440 [I 2022-10-10 23:30:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1550/2304 [I 2022-10-10 23:30:06] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:41 [I 2022-10-10 23:30:06] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00298 [I 2022-10-10 23:30:06] ...trap/engines/engine.py.230: loss: 0.15964 [I 2022-10-10 23:30:06] ...optimizers/trijoint.py.151: total_norm: 10.669225 [I 2022-10-10 23:30:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1560/2304 [I 2022-10-10 23:30:14] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:34 [I 2022-10-10 23:30:14] ...trap/engines/engine.py.229: process: 0.86327 | load: 0.00294 [I 2022-10-10 23:30:14] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-10 23:30:14] ...optimizers/trijoint.py.151: total_norm: 16.119595 [I 2022-10-10 23:30:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1570/2304 [I 2022-10-10 23:30:23] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:30 [I 2022-10-10 23:30:23] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00295 [I 2022-10-10 23:30:23] ...trap/engines/engine.py.230: loss: 0.10991 [I 2022-10-10 23:30:23] ...optimizers/trijoint.py.151: total_norm: 24.389748 [I 2022-10-10 23:30:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 1580/2304 [I 2022-10-10 23:30:31] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:19 [I 2022-10-10 23:30:31] ...trap/engines/engine.py.229: process: 0.86489 | load: 0.00301 [I 2022-10-10 23:30:31] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-10 23:30:31] ...optimizers/trijoint.py.151: total_norm: 9.638868 [I 2022-10-10 23:30:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1590/2304 [I 2022-10-10 23:30:40] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:10:12 [I 2022-10-10 23:30:40] ...trap/engines/engine.py.229: process: 0.86189 | load: 0.00293 [I 2022-10-10 23:30:40] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-10 23:30:40] ...optimizers/trijoint.py.151: total_norm: 8.997651 [I 2022-10-10 23:30:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 1600/2304 [I 2022-10-10 23:30:49] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:00 [I 2022-10-10 23:30:49] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00298 [I 2022-10-10 23:30:49] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-10 23:30:49] ...optimizers/trijoint.py.151: total_norm: 13.241619 [I 2022-10-10 23:30:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1610/2304 [I 2022-10-10 23:30:57] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:57 [I 2022-10-10 23:30:57] ...trap/engines/engine.py.229: process: 0.86357 | load: 0.00303 [I 2022-10-10 23:30:57] ...trap/engines/engine.py.230: loss: 0.08823 [I 2022-10-10 23:30:57] ...optimizers/trijoint.py.151: total_norm: 12.901101 [I 2022-10-10 23:31:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1620/2304 [I 2022-10-10 23:31:06] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:46 [I 2022-10-10 23:31:06] ...trap/engines/engine.py.229: process: 0.85897 | load: 0.00295 [I 2022-10-10 23:31:06] ...trap/engines/engine.py.230: loss: 0.13533 [I 2022-10-10 23:31:06] ...optimizers/trijoint.py.151: total_norm: 26.806835 [I 2022-10-10 23:31:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1630/2304 [I 2022-10-10 23:31:14] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:41 [I 2022-10-10 23:31:14] ...trap/engines/engine.py.229: process: 0.86037 | load: 0.00311 [I 2022-10-10 23:31:14] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-10 23:31:14] ...optimizers/trijoint.py.151: total_norm: 10.414091 [I 2022-10-10 23:31:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1640/2304 [I 2022-10-10 23:31:23] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:29 [I 2022-10-10 23:31:23] ...trap/engines/engine.py.229: process: 0.86916 | load: 0.00309 [I 2022-10-10 23:31:23] ...trap/engines/engine.py.230: loss: 0.09870 [I 2022-10-10 23:31:23] ...optimizers/trijoint.py.151: total_norm: 15.066953 [I 2022-10-10 23:31:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 1650/2304 [I 2022-10-10 23:31:32] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:24 [I 2022-10-10 23:31:32] ...trap/engines/engine.py.229: process: 0.86443 | load: 0.00312 [I 2022-10-10 23:31:32] ...trap/engines/engine.py.230: loss: 0.12593 [I 2022-10-10 23:31:32] ...optimizers/trijoint.py.151: total_norm: 14.466137 [I 2022-10-10 23:31:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1660/2304 [I 2022-10-10 23:31:40] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:15 [I 2022-10-10 23:31:40] ...trap/engines/engine.py.229: process: 0.86205 | load: 0.00309 [I 2022-10-10 23:31:40] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-10 23:31:40] ...optimizers/trijoint.py.151: total_norm: 17.248373 [I 2022-10-10 23:31:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 1670/2304 [I 2022-10-10 23:31:49] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:09:08 [I 2022-10-10 23:31:49] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00318 [I 2022-10-10 23:31:49] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-10 23:31:49] ...optimizers/trijoint.py.151: total_norm: 19.260542 [I 2022-10-10 23:31:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 1680/2304 [I 2022-10-10 23:31:58] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:09:02 [I 2022-10-10 23:31:58] ...trap/engines/engine.py.229: process: 0.86392 | load: 0.00312 [I 2022-10-10 23:31:58] ...trap/engines/engine.py.230: loss: 0.11401 [I 2022-10-10 23:31:58] ...optimizers/trijoint.py.151: total_norm: 16.278574 [I 2022-10-10 23:32:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1690/2304 [I 2022-10-10 23:32:06] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:52 [I 2022-10-10 23:32:06] ...trap/engines/engine.py.229: process: 0.87643 | load: 0.00318 [I 2022-10-10 23:32:06] ...trap/engines/engine.py.230: loss: 0.14162 [I 2022-10-10 23:32:06] ...optimizers/trijoint.py.151: total_norm: 17.062531 [I 2022-10-10 23:32:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 1700/2304 [I 2022-10-10 23:32:15] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:44 [I 2022-10-10 23:32:15] ...trap/engines/engine.py.229: process: 0.87937 | load: 0.00314 [I 2022-10-10 23:32:15] ...trap/engines/engine.py.230: loss: 0.15842 [I 2022-10-10 23:32:15] ...optimizers/trijoint.py.151: total_norm: 14.257780 [I 2022-10-10 23:32:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 1710/2304 [I 2022-10-10 23:32:24] ...trap/engines/engine.py.228: elapsed: 0:24:36 | left: 0:08:34 [I 2022-10-10 23:32:24] ...trap/engines/engine.py.229: process: 0.86189 | load: 0.00322 [I 2022-10-10 23:32:24] ...trap/engines/engine.py.230: loss: 0.18309 [I 2022-10-10 23:32:24] ...optimizers/trijoint.py.151: total_norm: 45.046066 [I 2022-10-10 23:32:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 1720/2304 [I 2022-10-10 23:32:32] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:24 [I 2022-10-10 23:32:32] ...trap/engines/engine.py.229: process: 0.85430 | load: 0.00311 [I 2022-10-10 23:32:32] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-10 23:32:32] ...optimizers/trijoint.py.151: total_norm: 10.222196 [I 2022-10-10 23:32:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 1730/2304 [I 2022-10-10 23:32:41] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:08:15 [I 2022-10-10 23:32:41] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00308 [I 2022-10-10 23:32:41] ...trap/engines/engine.py.230: loss: 0.14423 [I 2022-10-10 23:32:41] ...optimizers/trijoint.py.151: total_norm: 34.679348 [I 2022-10-10 23:32:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 1740/2304 [I 2022-10-10 23:32:50] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:08:07 [I 2022-10-10 23:32:50] ...trap/engines/engine.py.229: process: 0.86534 | load: 0.00314 [I 2022-10-10 23:32:50] ...trap/engines/engine.py.230: loss: 0.10144 [I 2022-10-10 23:32:50] ...optimizers/trijoint.py.151: total_norm: 11.433002 [I 2022-10-10 23:32:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 1750/2304 [I 2022-10-10 23:32:58] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:58 [I 2022-10-10 23:32:58] ...trap/engines/engine.py.229: process: 0.86995 | load: 0.00309 [I 2022-10-10 23:32:58] ...trap/engines/engine.py.230: loss: 0.08109 [I 2022-10-10 23:32:58] ...optimizers/trijoint.py.151: total_norm: 10.897455 [I 2022-10-10 23:33:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 1760/2304 [I 2022-10-10 23:33:07] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:49 [I 2022-10-10 23:33:07] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00314 [I 2022-10-10 23:33:07] ...trap/engines/engine.py.230: loss: 0.13167 [I 2022-10-10 23:33:07] ...optimizers/trijoint.py.151: total_norm: 12.634409 [I 2022-10-10 23:33:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 1770/2304 [I 2022-10-10 23:33:16] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:39 [I 2022-10-10 23:33:16] ...trap/engines/engine.py.229: process: 0.85881 | load: 0.00313 [I 2022-10-10 23:33:16] ...trap/engines/engine.py.230: loss: 0.08002 [I 2022-10-10 23:33:16] ...optimizers/trijoint.py.151: total_norm: 10.075598 [I 2022-10-10 23:33:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 1780/2304 [I 2022-10-10 23:33:24] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:34 [I 2022-10-10 23:33:24] ...trap/engines/engine.py.229: process: 0.88019 | load: 0.00305 [I 2022-10-10 23:33:24] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-10 23:33:24] ...optimizers/trijoint.py.151: total_norm: 58.575218 [I 2022-10-10 23:33:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 1790/2304 [I 2022-10-10 23:33:33] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:22 [I 2022-10-10 23:33:33] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00347 [I 2022-10-10 23:33:33] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-10 23:33:33] ...optimizers/trijoint.py.151: total_norm: 16.546888 [I 2022-10-10 23:33:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 1800/2304 [I 2022-10-10 23:33:42] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:17 [I 2022-10-10 23:33:42] ...trap/engines/engine.py.229: process: 0.86853 | load: 0.00314 [I 2022-10-10 23:33:42] ...trap/engines/engine.py.230: loss: 0.09099 [I 2022-10-10 23:33:42] ...optimizers/trijoint.py.151: total_norm: 10.418147 [I 2022-10-10 23:33:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 1810/2304 [I 2022-10-10 23:33:50] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:07:06 [I 2022-10-10 23:33:50] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00308 [I 2022-10-10 23:33:50] ...trap/engines/engine.py.230: loss: 0.09259 [I 2022-10-10 23:33:50] ...optimizers/trijoint.py.151: total_norm: 13.201547 [I 2022-10-10 23:33:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 1820/2304 [I 2022-10-10 23:33:59] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:58 [I 2022-10-10 23:33:59] ...trap/engines/engine.py.229: process: 0.85551 | load: 0.00309 [I 2022-10-10 23:33:59] ...trap/engines/engine.py.230: loss: 0.09407 [I 2022-10-10 23:33:59] ...optimizers/trijoint.py.151: total_norm: 10.236881 [I 2022-10-10 23:34:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 1830/2304 [I 2022-10-10 23:34:08] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:48 [I 2022-10-10 23:34:08] ...trap/engines/engine.py.229: process: 0.83645 | load: 0.00310 [I 2022-10-10 23:34:08] ...trap/engines/engine.py.230: loss: 0.08330 [I 2022-10-10 23:34:08] ...optimizers/trijoint.py.151: total_norm: 11.919279 [I 2022-10-10 23:34:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 1840/2304 [I 2022-10-10 23:34:16] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:37 [I 2022-10-10 23:34:16] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00312 [I 2022-10-10 23:34:16] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-10 23:34:16] ...optimizers/trijoint.py.151: total_norm: 19.179951 [I 2022-10-10 23:34:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 1850/2304 [I 2022-10-10 23:34:25] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:29 [I 2022-10-10 23:34:25] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00314 [I 2022-10-10 23:34:25] ...trap/engines/engine.py.230: loss: 0.09684 [I 2022-10-10 23:34:25] ...optimizers/trijoint.py.151: total_norm: 12.151236 [I 2022-10-10 23:34:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 1860/2304 [I 2022-10-10 23:34:33] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:25 [I 2022-10-10 23:34:33] ...trap/engines/engine.py.229: process: 0.86849 | load: 0.00308 [I 2022-10-10 23:34:33] ...trap/engines/engine.py.230: loss: 0.09362 [I 2022-10-10 23:34:33] ...optimizers/trijoint.py.151: total_norm: 41.401386 [I 2022-10-10 23:34:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 1870/2304 [I 2022-10-10 23:34:42] ...trap/engines/engine.py.228: elapsed: 0:26:54 | left: 0:06:16 [I 2022-10-10 23:34:42] ...trap/engines/engine.py.229: process: 0.86806 | load: 0.00314 [I 2022-10-10 23:34:42] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-10 23:34:42] ...optimizers/trijoint.py.151: total_norm: 7.457111 [I 2022-10-10 23:34:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 1880/2304 [I 2022-10-10 23:34:51] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:06:08 [I 2022-10-10 23:34:51] ...trap/engines/engine.py.229: process: 0.86678 | load: 0.00313 [I 2022-10-10 23:34:51] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-10 23:34:51] ...optimizers/trijoint.py.151: total_norm: 15.697064 [I 2022-10-10 23:34:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 1890/2304 [I 2022-10-10 23:34:59] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:06:00 [I 2022-10-10 23:34:59] ...trap/engines/engine.py.229: process: 0.87665 | load: 0.00313 [I 2022-10-10 23:34:59] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-10 23:35:00] ...optimizers/trijoint.py.151: total_norm: 13.310639 [I 2022-10-10 23:35:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 1900/2304 [I 2022-10-10 23:35:08] ...trap/engines/engine.py.228: elapsed: 0:27:20 | left: 0:05:47 [I 2022-10-10 23:35:08] ...trap/engines/engine.py.229: process: 0.85220 | load: 0.00308 [I 2022-10-10 23:35:08] ...trap/engines/engine.py.230: loss: 0.10643 [I 2022-10-10 23:35:08] ...optimizers/trijoint.py.151: total_norm: 10.762545 [I 2022-10-10 23:35:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 1910/2304 [I 2022-10-10 23:35:17] ...trap/engines/engine.py.228: elapsed: 0:27:29 | left: 0:05:35 [I 2022-10-10 23:35:17] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00300 [I 2022-10-10 23:35:17] ...trap/engines/engine.py.230: loss: 0.09405 [I 2022-10-10 23:35:17] ...optimizers/trijoint.py.151: total_norm: 9.820408 [I 2022-10-10 23:35:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 1920/2304 [I 2022-10-10 23:35:25] ...trap/engines/engine.py.228: elapsed: 0:27:37 | left: 0:05:27 [I 2022-10-10 23:35:25] ...trap/engines/engine.py.229: process: 0.82987 | load: 0.00298 [I 2022-10-10 23:35:25] ...trap/engines/engine.py.230: loss: 0.16608 [I 2022-10-10 23:35:25] ...optimizers/trijoint.py.151: total_norm: 11.606039 [I 2022-10-10 23:35:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 1930/2304 [I 2022-10-10 23:35:34] ...trap/engines/engine.py.228: elapsed: 0:27:46 | left: 0:05:19 [I 2022-10-10 23:35:34] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00307 [I 2022-10-10 23:35:34] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-10 23:35:34] ...optimizers/trijoint.py.151: total_norm: 11.234358 [I 2022-10-10 23:35:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 1940/2304 [I 2022-10-10 23:35:42] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:05:11 [I 2022-10-10 23:35:42] ...trap/engines/engine.py.229: process: 0.86123 | load: 0.00303 [I 2022-10-10 23:35:42] ...trap/engines/engine.py.230: loss: 0.09276 [I 2022-10-10 23:35:42] ...optimizers/trijoint.py.151: total_norm: 10.728895 [I 2022-10-10 23:35:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 1950/2304 [I 2022-10-10 23:35:51] ...trap/engines/engine.py.228: elapsed: 0:28:03 | left: 0:05:02 [I 2022-10-10 23:35:51] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00300 [I 2022-10-10 23:35:51] ...trap/engines/engine.py.230: loss: 0.07536 [I 2022-10-10 23:35:51] ...optimizers/trijoint.py.151: total_norm: 11.252615 [I 2022-10-10 23:36:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 1960/2304 [I 2022-10-10 23:36:00] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:54 [I 2022-10-10 23:36:00] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00302 [I 2022-10-10 23:36:00] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-10 23:36:00] ...optimizers/trijoint.py.151: total_norm: 12.456934 [I 2022-10-10 23:36:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 1970/2304 [I 2022-10-10 23:36:08] ...trap/engines/engine.py.228: elapsed: 0:28:20 | left: 0:04:44 [I 2022-10-10 23:36:08] ...trap/engines/engine.py.229: process: 0.85416 | load: 0.00298 [I 2022-10-10 23:36:08] ...trap/engines/engine.py.230: loss: 0.10942 [I 2022-10-10 23:36:08] ...optimizers/trijoint.py.151: total_norm: 13.024302 [I 2022-10-10 23:36:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 1980/2304 [I 2022-10-10 23:36:17] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:34 [I 2022-10-10 23:36:17] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00309 [I 2022-10-10 23:36:17] ...trap/engines/engine.py.230: loss: 0.16887 [I 2022-10-10 23:36:17] ...optimizers/trijoint.py.151: total_norm: 20.751354 [I 2022-10-10 23:36:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 1990/2304 [I 2022-10-10 23:36:25] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:30 [I 2022-10-10 23:36:25] ...trap/engines/engine.py.229: process: 0.87032 | load: 0.00311 [I 2022-10-10 23:36:25] ...trap/engines/engine.py.230: loss: 0.13776 [I 2022-10-10 23:36:25] ...optimizers/trijoint.py.151: total_norm: 26.858534 [I 2022-10-10 23:36:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 2000/2304 [I 2022-10-10 23:36:34] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:19 [I 2022-10-10 23:36:34] ...trap/engines/engine.py.229: process: 0.87333 | load: 0.00300 [I 2022-10-10 23:36:34] ...trap/engines/engine.py.230: loss: 0.15438 [I 2022-10-10 23:36:34] ...optimizers/trijoint.py.151: total_norm: 18.995428 [I 2022-10-10 23:36:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 2010/2304 [I 2022-10-10 23:36:42] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:04:11 [I 2022-10-10 23:36:42] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00305 [I 2022-10-10 23:36:42] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-10 23:36:42] ...optimizers/trijoint.py.151: total_norm: 15.755270 [I 2022-10-10 23:36:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 2020/2304 [I 2022-10-10 23:36:51] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:04:02 [I 2022-10-10 23:36:51] ...trap/engines/engine.py.229: process: 0.85687 | load: 0.00301 [I 2022-10-10 23:36:51] ...trap/engines/engine.py.230: loss: 0.06351 [I 2022-10-10 23:36:51] ...optimizers/trijoint.py.151: total_norm: 11.758773 [I 2022-10-10 23:37:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 2030/2304 [I 2022-10-10 23:37:00] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:55 [I 2022-10-10 23:37:00] ...trap/engines/engine.py.229: process: 0.87527 | load: 0.00302 [I 2022-10-10 23:37:00] ...trap/engines/engine.py.230: loss: 0.18912 [I 2022-10-10 23:37:00] ...optimizers/trijoint.py.151: total_norm: 11.876490 [I 2022-10-10 23:37:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 2040/2304 [I 2022-10-10 23:37:08] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:46 [I 2022-10-10 23:37:08] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00309 [I 2022-10-10 23:37:08] ...trap/engines/engine.py.230: loss: 0.13152 [I 2022-10-10 23:37:08] ...optimizers/trijoint.py.151: total_norm: 10.129932 [I 2022-10-10 23:37:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 2050/2304 [I 2022-10-10 23:37:36] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:13:57 [I 2022-10-10 23:37:36] ...trap/engines/engine.py.229: process: 0.86276 | load: 0.00308 [I 2022-10-10 23:37:36] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-10 23:37:36] ...optimizers/trijoint.py.151: total_norm: 9.504251 [I 2022-10-10 23:37:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 2060/2304 [I 2022-10-10 23:37:44] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:04:30 [I 2022-10-10 23:37:44] ...trap/engines/engine.py.229: process: 0.83781 | load: 0.00310 [I 2022-10-10 23:37:44] ...trap/engines/engine.py.230: loss: 0.10103 [I 2022-10-10 23:37:44] ...optimizers/trijoint.py.151: total_norm: 10.366190 [I 2022-10-10 23:37:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 2070/2304 [I 2022-10-10 23:37:53] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:03:25 [I 2022-10-10 23:37:53] ...trap/engines/engine.py.229: process: 0.85703 | load: 0.00316 [I 2022-10-10 23:37:53] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-10 23:37:53] ...optimizers/trijoint.py.151: total_norm: 12.283946 [I 2022-10-10 23:38:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 2080/2304 [I 2022-10-10 23:38:01] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:03:12 [I 2022-10-10 23:38:01] ...trap/engines/engine.py.229: process: 0.86872 | load: 0.00311 [I 2022-10-10 23:38:01] ...trap/engines/engine.py.230: loss: 0.12439 [I 2022-10-10 23:38:01] ...optimizers/trijoint.py.151: total_norm: 12.974339 [I 2022-10-10 23:38:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 2090/2304 [I 2022-10-10 23:38:10] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:03:01 [I 2022-10-10 23:38:10] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00301 [I 2022-10-10 23:38:10] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-10 23:38:10] ...optimizers/trijoint.py.151: total_norm: 15.774023 [I 2022-10-10 23:38:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 2100/2304 [I 2022-10-10 23:38:19] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:54 [I 2022-10-10 23:38:19] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00310 [I 2022-10-10 23:38:19] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-10 23:38:19] ...optimizers/trijoint.py.151: total_norm: 12.485033 [I 2022-10-10 23:38:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 2110/2304 [I 2022-10-10 23:38:27] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:45 [I 2022-10-10 23:38:27] ...trap/engines/engine.py.229: process: 0.84619 | load: 0.00297 [I 2022-10-10 23:38:27] ...trap/engines/engine.py.230: loss: 0.14157 [I 2022-10-10 23:38:27] ...optimizers/trijoint.py.151: total_norm: 12.944040 [I 2022-10-10 23:38:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 2120/2304 [I 2022-10-10 23:38:36] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:38 [I 2022-10-10 23:38:36] ...trap/engines/engine.py.229: process: 0.86632 | load: 0.00296 [I 2022-10-10 23:38:36] ...trap/engines/engine.py.230: loss: 0.14525 [I 2022-10-10 23:38:36] ...optimizers/trijoint.py.151: total_norm: 12.508576 [I 2022-10-10 23:38:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 2130/2304 [I 2022-10-10 23:38:44] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:28 [I 2022-10-10 23:38:44] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00296 [I 2022-10-10 23:38:44] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-10 23:38:44] ...optimizers/trijoint.py.151: total_norm: 9.575098 [I 2022-10-10 23:38:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 2140/2304 [I 2022-10-10 23:38:53] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:20 [I 2022-10-10 23:38:53] ...trap/engines/engine.py.229: process: 0.86199 | load: 0.00311 [I 2022-10-10 23:38:53] ...trap/engines/engine.py.230: loss: 0.08693 [I 2022-10-10 23:38:53] ...optimizers/trijoint.py.151: total_norm: 10.956407 [I 2022-10-10 23:39:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 2150/2304 [I 2022-10-10 23:39:01] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:02:10 [I 2022-10-10 23:39:01] ...trap/engines/engine.py.229: process: 0.84280 | load: 0.00296 [I 2022-10-10 23:39:01] ...trap/engines/engine.py.230: loss: 0.13273 [I 2022-10-10 23:39:01] ...optimizers/trijoint.py.151: total_norm: 15.737062 [I 2022-10-10 23:39:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 2160/2304 [I 2022-10-10 23:39:10] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:02:03 [I 2022-10-10 23:39:10] ...trap/engines/engine.py.229: process: 0.85769 | load: 0.00297 [I 2022-10-10 23:39:10] ...trap/engines/engine.py.230: loss: 0.07467 [I 2022-10-10 23:39:10] ...optimizers/trijoint.py.151: total_norm: 16.506086 [I 2022-10-10 23:39:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 2170/2304 [I 2022-10-10 23:39:19] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:54 [I 2022-10-10 23:39:19] ...trap/engines/engine.py.229: process: 0.87031 | load: 0.00296 [I 2022-10-10 23:39:19] ...trap/engines/engine.py.230: loss: 0.13743 [I 2022-10-10 23:39:19] ...optimizers/trijoint.py.151: total_norm: 9.694937 [I 2022-10-10 23:39:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 2180/2304 [I 2022-10-10 23:39:27] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:45 [I 2022-10-10 23:39:27] ...trap/engines/engine.py.229: process: 0.83807 | load: 0.00301 [I 2022-10-10 23:39:27] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-10 23:39:27] ...optimizers/trijoint.py.151: total_norm: 20.792006 [I 2022-10-10 23:39:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 2190/2304 [I 2022-10-10 23:39:36] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:37 [I 2022-10-10 23:39:36] ...trap/engines/engine.py.229: process: 0.86363 | load: 0.00310 [I 2022-10-10 23:39:36] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-10 23:39:36] ...optimizers/trijoint.py.151: total_norm: 66.132065 [I 2022-10-10 23:39:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 2200/2304 [I 2022-10-10 23:39:44] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:01:28 [I 2022-10-10 23:39:44] ...trap/engines/engine.py.229: process: 0.85801 | load: 0.00313 [I 2022-10-10 23:39:44] ...trap/engines/engine.py.230: loss: 0.10568 [I 2022-10-10 23:39:44] ...optimizers/trijoint.py.151: total_norm: 13.267104 [I 2022-10-10 23:39:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 2210/2304 [I 2022-10-10 23:39:53] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:20 [I 2022-10-10 23:39:53] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00314 [I 2022-10-10 23:39:53] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-10 23:39:53] ...optimizers/trijoint.py.151: total_norm: 13.954442 [I 2022-10-10 23:40:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 2220/2304 [I 2022-10-10 23:40:01] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:01:12 [I 2022-10-10 23:40:01] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00314 [I 2022-10-10 23:40:01] ...trap/engines/engine.py.230: loss: 0.11268 [I 2022-10-10 23:40:01] ...optimizers/trijoint.py.151: total_norm: 22.115213 [I 2022-10-10 23:40:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 2230/2304 [I 2022-10-10 23:40:10] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:01:03 [I 2022-10-10 23:40:10] ...trap/engines/engine.py.229: process: 0.86769 | load: 0.00312 [I 2022-10-10 23:40:10] ...trap/engines/engine.py.230: loss: 0.11526 [I 2022-10-10 23:40:10] ...optimizers/trijoint.py.151: total_norm: 15.793731 [I 2022-10-10 23:40:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 2240/2304 [I 2022-10-10 23:40:19] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:54 [I 2022-10-10 23:40:19] ...trap/engines/engine.py.229: process: 0.85015 | load: 0.00378 [I 2022-10-10 23:40:19] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-10 23:40:19] ...optimizers/trijoint.py.151: total_norm: 13.035982 [I 2022-10-10 23:40:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 2250/2304 [I 2022-10-10 23:40:27] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:46 [I 2022-10-10 23:40:27] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00307 [I 2022-10-10 23:40:27] ...trap/engines/engine.py.230: loss: 0.09467 [I 2022-10-10 23:40:27] ...optimizers/trijoint.py.151: total_norm: 10.027798 [I 2022-10-10 23:40:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 2260/2304 [I 2022-10-10 23:40:36] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:37 [I 2022-10-10 23:40:36] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00318 [I 2022-10-10 23:40:36] ...trap/engines/engine.py.230: loss: 0.14531 [I 2022-10-10 23:40:36] ...optimizers/trijoint.py.151: total_norm: 22.365656 [I 2022-10-10 23:40:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 2270/2304 [I 2022-10-10 23:40:44] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:29 [I 2022-10-10 23:40:44] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00310 [I 2022-10-10 23:40:44] ...trap/engines/engine.py.230: loss: 0.17439 [I 2022-10-10 23:40:44] ...optimizers/trijoint.py.151: total_norm: 17.631420 [I 2022-10-10 23:40:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 2280/2304 [I 2022-10-10 23:40:53] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:20 [I 2022-10-10 23:40:53] ...trap/engines/engine.py.229: process: 0.87735 | load: 0.00306 [I 2022-10-10 23:40:53] ...trap/engines/engine.py.230: loss: 0.11258 [I 2022-10-10 23:40:53] ...optimizers/trijoint.py.151: total_norm: 12.330901 [I 2022-10-10 23:41:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 2290/2304 [I 2022-10-10 23:41:02] ...trap/engines/engine.py.228: elapsed: 0:33:14 | left: 0:00:12 [I 2022-10-10 23:41:02] ...trap/engines/engine.py.229: process: 0.86319 | load: 0.00315 [I 2022-10-10 23:41:02] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-10 23:41:02] ...optimizers/trijoint.py.151: total_norm: 14.408454 [I 2022-10-10 23:41:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 2300/2304 [I 2022-10-10 23:41:10] ...trap/engines/engine.py.228: elapsed: 0:33:22 | left: 0:00:03 [I 2022-10-10 23:41:10] ...trap/engines/engine.py.229: process: 0.83267 | load: 0.00019 [I 2022-10-10 23:41:10] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-10 23:41:10] ...optimizers/trijoint.py.151: total_norm: 10.383877 [I 2022-10-10 23:41:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 2304/2304 [I 2022-10-10 23:41:14] ...trap/engines/engine.py.228: elapsed: 0:33:26 | left: 0:00:00 [I 2022-10-10 23:41:14] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00019 [I 2022-10-10 23:41:14] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-10 23:41:14] ...optimizers/trijoint.py.151: total_norm: 17.576262 [S 2022-10-10 23:41:15] ...trap/engines/engine.py.129: train_epoch.epoch: 89 [S 2022-10-10 23:41:15] ...trap/engines/engine.py.129: train_epoch.loss: 0.12915324745235113 [I 2022-10-10 23:41:15] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-10 23:41:17] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 89 [I 2022-10-10 23:41:17] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-10 23:41:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 23:41:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 23:41:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 23:41:22] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 23:41:23] ...trap/engines/engine.py.321: eval: epoch 89 | batch 0/510 [I 2022-10-10 23:41:23] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:51:00 [I 2022-10-10 23:41:23] ...trap/engines/engine.py.326: process: 6.00092 | load: 3.74634 [I 2022-10-10 23:41:29] ...trap/engines/engine.py.321: eval: epoch 89 | batch 10/510 [I 2022-10-10 23:41:29] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:10:56 [I 2022-10-10 23:41:29] ...trap/engines/engine.py.326: process: 0.81942 | load: 0.00019 [I 2022-10-10 23:41:36] ...trap/engines/engine.py.321: eval: epoch 89 | batch 20/510 [I 2022-10-10 23:41:36] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:41 [I 2022-10-10 23:41:36] ...trap/engines/engine.py.326: process: 0.78919 | load: 0.00021 [I 2022-10-10 23:41:44] ...trap/engines/engine.py.321: eval: epoch 89 | batch 30/510 [I 2022-10-10 23:41:44] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:06:05 [I 2022-10-10 23:41:44] ...trap/engines/engine.py.326: process: 0.43995 | load: 0.00021 [I 2022-10-10 23:41:52] ...trap/engines/engine.py.321: eval: epoch 89 | batch 40/510 [I 2022-10-10 23:41:52] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:06:13 [I 2022-10-10 23:41:52] ...trap/engines/engine.py.326: process: 0.87193 | load: 0.00021 [I 2022-10-10 23:41:57] ...trap/engines/engine.py.321: eval: epoch 89 | batch 50/510 [I 2022-10-10 23:41:57] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:04:18 [I 2022-10-10 23:41:57] ...trap/engines/engine.py.326: process: 0.54988 | load: 0.00035 [I 2022-10-10 23:42:03] ...trap/engines/engine.py.321: eval: epoch 89 | batch 60/510 [I 2022-10-10 23:42:03] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:26 [I 2022-10-10 23:42:03] ...trap/engines/engine.py.326: process: 0.70136 | load: 0.00019 [I 2022-10-10 23:42:09] ...trap/engines/engine.py.321: eval: epoch 89 | batch 70/510 [I 2022-10-10 23:42:09] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:04:43 [I 2022-10-10 23:42:09] ...trap/engines/engine.py.326: process: 0.50702 | load: 0.00021 [I 2022-10-10 23:42:16] ...trap/engines/engine.py.321: eval: epoch 89 | batch 80/510 [I 2022-10-10 23:42:16] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:04:53 [I 2022-10-10 23:42:16] ...trap/engines/engine.py.326: process: 0.52580 | load: 0.00021 [I 2022-10-10 23:42:22] ...trap/engines/engine.py.321: eval: epoch 89 | batch 90/510 [I 2022-10-10 23:42:22] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:03:42 [I 2022-10-10 23:42:22] ...trap/engines/engine.py.326: process: 0.41311 | load: 0.00025 [I 2022-10-10 23:42:26] ...trap/engines/engine.py.321: eval: epoch 89 | batch 100/510 [I 2022-10-10 23:42:26] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:03:10 [I 2022-10-10 23:42:26] ...trap/engines/engine.py.326: process: 0.31364 | load: 0.00021 [I 2022-10-10 23:42:30] ...trap/engines/engine.py.321: eval: epoch 89 | batch 110/510 [I 2022-10-10 23:42:30] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:15 [I 2022-10-10 23:42:30] ...trap/engines/engine.py.326: process: 0.31621 | load: 0.00020 [I 2022-10-10 23:42:33] ...trap/engines/engine.py.321: eval: epoch 89 | batch 120/510 [I 2022-10-10 23:42:33] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:02:06 [I 2022-10-10 23:42:33] ...trap/engines/engine.py.326: process: 0.32169 | load: 0.00022 [I 2022-10-10 23:42:36] ...trap/engines/engine.py.321: eval: epoch 89 | batch 130/510 [I 2022-10-10 23:42:36] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:02:02 [I 2022-10-10 23:42:36] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00020 [I 2022-10-10 23:42:39] ...trap/engines/engine.py.321: eval: epoch 89 | batch 140/510 [I 2022-10-10 23:42:39] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:59 [I 2022-10-10 23:42:39] ...trap/engines/engine.py.326: process: 0.33070 | load: 0.00022 [I 2022-10-10 23:42:43] ...trap/engines/engine.py.321: eval: epoch 89 | batch 150/510 [I 2022-10-10 23:42:43] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:54 [I 2022-10-10 23:42:43] ...trap/engines/engine.py.326: process: 0.31442 | load: 0.00020 [I 2022-10-10 23:42:46] ...trap/engines/engine.py.321: eval: epoch 89 | batch 160/510 [I 2022-10-10 23:42:46] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:52 [I 2022-10-10 23:42:46] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00020 [I 2022-10-10 23:42:49] ...trap/engines/engine.py.321: eval: epoch 89 | batch 170/510 [I 2022-10-10 23:42:49] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:50 [I 2022-10-10 23:42:49] ...trap/engines/engine.py.326: process: 0.32457 | load: 0.00021 [I 2022-10-10 23:42:52] ...trap/engines/engine.py.321: eval: epoch 89 | batch 180/510 [I 2022-10-10 23:42:52] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:46 [I 2022-10-10 23:42:52] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00020 [I 2022-10-10 23:42:56] ...trap/engines/engine.py.321: eval: epoch 89 | batch 190/510 [I 2022-10-10 23:42:56] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:43 [I 2022-10-10 23:42:56] ...trap/engines/engine.py.326: process: 0.32496 | load: 0.00019 [I 2022-10-10 23:42:59] ...trap/engines/engine.py.321: eval: epoch 89 | batch 200/510 [I 2022-10-10 23:42:59] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:39 [I 2022-10-10 23:42:59] ...trap/engines/engine.py.326: process: 0.32341 | load: 0.00020 [I 2022-10-10 23:43:02] ...trap/engines/engine.py.321: eval: epoch 89 | batch 210/510 [I 2022-10-10 23:43:02] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:35 [I 2022-10-10 23:43:02] ...trap/engines/engine.py.326: process: 0.31375 | load: 0.00021 [I 2022-10-10 23:43:05] ...trap/engines/engine.py.321: eval: epoch 89 | batch 220/510 [I 2022-10-10 23:43:05] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:32 [I 2022-10-10 23:43:05] ...trap/engines/engine.py.326: process: 0.31419 | load: 0.00021 [I 2022-10-10 23:43:08] ...trap/engines/engine.py.321: eval: epoch 89 | batch 230/510 [I 2022-10-10 23:43:08] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:29 [I 2022-10-10 23:43:08] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00020 [I 2022-10-10 23:43:12] ...trap/engines/engine.py.321: eval: epoch 89 | batch 240/510 [I 2022-10-10 23:43:12] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:26 [I 2022-10-10 23:43:12] ...trap/engines/engine.py.326: process: 0.31078 | load: 0.00020 [I 2022-10-10 23:43:15] ...trap/engines/engine.py.321: eval: epoch 89 | batch 250/510 [I 2022-10-10 23:43:15] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:23 [I 2022-10-10 23:43:15] ...trap/engines/engine.py.326: process: 0.32755 | load: 0.00021 [I 2022-10-10 23:43:18] ...trap/engines/engine.py.321: eval: epoch 89 | batch 260/510 [I 2022-10-10 23:43:18] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:20 [I 2022-10-10 23:43:18] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00022 [I 2022-10-10 23:43:21] ...trap/engines/engine.py.321: eval: epoch 89 | batch 270/510 [I 2022-10-10 23:43:21] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:17 [I 2022-10-10 23:43:21] ...trap/engines/engine.py.326: process: 0.32001 | load: 0.00021 [I 2022-10-10 23:43:24] ...trap/engines/engine.py.321: eval: epoch 89 | batch 280/510 [I 2022-10-10 23:43:24] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:13 [I 2022-10-10 23:43:24] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00021 [I 2022-10-10 23:43:28] ...trap/engines/engine.py.321: eval: epoch 89 | batch 290/510 [I 2022-10-10 23:43:28] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:10 [I 2022-10-10 23:43:28] ...trap/engines/engine.py.326: process: 0.31483 | load: 0.00021 [I 2022-10-10 23:43:31] ...trap/engines/engine.py.321: eval: epoch 89 | batch 300/510 [I 2022-10-10 23:43:31] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:07 [I 2022-10-10 23:43:31] ...trap/engines/engine.py.326: process: 0.31702 | load: 0.00017 [I 2022-10-10 23:43:34] ...trap/engines/engine.py.321: eval: epoch 89 | batch 310/510 [I 2022-10-10 23:43:34] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:01:04 [I 2022-10-10 23:43:34] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00021 [I 2022-10-10 23:43:37] ...trap/engines/engine.py.321: eval: epoch 89 | batch 320/510 [I 2022-10-10 23:43:37] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:01:00 [I 2022-10-10 23:43:37] ...trap/engines/engine.py.326: process: 0.32012 | load: 0.00020 [I 2022-10-10 23:43:40] ...trap/engines/engine.py.321: eval: epoch 89 | batch 330/510 [I 2022-10-10 23:43:41] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:57 [I 2022-10-10 23:43:41] ...trap/engines/engine.py.326: process: 0.31913 | load: 0.00020 [I 2022-10-10 23:43:44] ...trap/engines/engine.py.321: eval: epoch 89 | batch 340/510 [I 2022-10-10 23:43:44] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:54 [I 2022-10-10 23:43:44] ...trap/engines/engine.py.326: process: 0.32536 | load: 0.00021 [I 2022-10-10 23:43:47] ...trap/engines/engine.py.321: eval: epoch 89 | batch 350/510 [I 2022-10-10 23:43:47] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:51 [I 2022-10-10 23:43:47] ...trap/engines/engine.py.326: process: 0.32227 | load: 0.00021 [I 2022-10-10 23:43:50] ...trap/engines/engine.py.321: eval: epoch 89 | batch 360/510 [I 2022-10-10 23:43:50] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:48 [I 2022-10-10 23:43:50] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00020 [I 2022-10-10 23:43:53] ...trap/engines/engine.py.321: eval: epoch 89 | batch 370/510 [I 2022-10-10 23:43:53] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:45 [I 2022-10-10 23:43:53] ...trap/engines/engine.py.326: process: 0.32045 | load: 0.00021 [I 2022-10-10 23:43:57] ...trap/engines/engine.py.321: eval: epoch 89 | batch 380/510 [I 2022-10-10 23:43:57] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:41 [I 2022-10-10 23:43:57] ...trap/engines/engine.py.326: process: 0.32619 | load: 0.00022 [I 2022-10-10 23:44:00] ...trap/engines/engine.py.321: eval: epoch 89 | batch 390/510 [I 2022-10-10 23:44:00] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:38 [I 2022-10-10 23:44:00] ...trap/engines/engine.py.326: process: 0.32488 | load: 0.00021 [I 2022-10-10 23:44:03] ...trap/engines/engine.py.321: eval: epoch 89 | batch 400/510 [I 2022-10-10 23:44:03] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:35 [I 2022-10-10 23:44:03] ...trap/engines/engine.py.326: process: 0.32778 | load: 0.00021 [I 2022-10-10 23:44:06] ...trap/engines/engine.py.321: eval: epoch 89 | batch 410/510 [I 2022-10-10 23:44:06] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:32 [I 2022-10-10 23:44:06] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00022 [I 2022-10-10 23:44:10] ...trap/engines/engine.py.321: eval: epoch 89 | batch 420/510 [I 2022-10-10 23:44:10] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:28 [I 2022-10-10 23:44:10] ...trap/engines/engine.py.326: process: 0.32475 | load: 0.00021 [I 2022-10-10 23:44:13] ...trap/engines/engine.py.321: eval: epoch 89 | batch 430/510 [I 2022-10-10 23:44:13] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:25 [I 2022-10-10 23:44:13] ...trap/engines/engine.py.326: process: 0.32765 | load: 0.00025 [I 2022-10-10 23:44:16] ...trap/engines/engine.py.321: eval: epoch 89 | batch 440/510 [I 2022-10-10 23:44:16] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:22 [I 2022-10-10 23:44:16] ...trap/engines/engine.py.326: process: 0.32199 | load: 0.00021 [I 2022-10-10 23:44:19] ...trap/engines/engine.py.321: eval: epoch 89 | batch 450/510 [I 2022-10-10 23:44:19] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:19 [I 2022-10-10 23:44:19] ...trap/engines/engine.py.326: process: 0.32385 | load: 0.00025 [I 2022-10-10 23:44:23] ...trap/engines/engine.py.321: eval: epoch 89 | batch 460/510 [I 2022-10-10 23:44:23] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:16 [I 2022-10-10 23:44:23] ...trap/engines/engine.py.326: process: 0.32836 | load: 0.00021 [I 2022-10-10 23:44:26] ...trap/engines/engine.py.321: eval: epoch 89 | batch 470/510 [I 2022-10-10 23:44:26] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:12 [I 2022-10-10 23:44:26] ...trap/engines/engine.py.326: process: 0.31600 | load: 0.00025 [I 2022-10-10 23:44:29] ...trap/engines/engine.py.321: eval: epoch 89 | batch 480/510 [I 2022-10-10 23:44:29] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:09 [I 2022-10-10 23:44:29] ...trap/engines/engine.py.326: process: 0.31699 | load: 0.00018 [I 2022-10-10 23:44:32] ...trap/engines/engine.py.321: eval: epoch 89 | batch 490/510 [I 2022-10-10 23:44:32] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:06 [I 2022-10-10 23:44:32] ...trap/engines/engine.py.326: process: 0.32271 | load: 0.00022 [I 2022-10-10 23:44:36] ...trap/engines/engine.py.321: eval: epoch 89 | batch 500/510 [I 2022-10-10 23:44:36] ...trap/engines/engine.py.325: elapsed: 0:03:18 | left: 0:00:03 [I 2022-10-10 23:44:36] ...trap/engines/engine.py.326: process: 0.32849 | load: 0.00021 [I 2022-10-10 23:44:39] ...trap/engines/engine.py.321: eval: epoch 89 | batch 510/510 [I 2022-10-10 23:44:39] ...trap/engines/engine.py.325: elapsed: 0:03:22 | left: 0:00:00 [I 2022-10-10 23:44:39] ...trap/engines/engine.py.326: process: 0.31851 | load: 0.00017 [S 2022-10-10 23:44:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 89 [I 2022-10-10 23:44:40] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740000 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.912300 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939200 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.734600 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911600 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937100 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014690 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011145 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009816 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015272 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012339 [S 2022-10-10 23:45:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009628 [I 2022-10-10 23:45:26] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-10 23:45:26] ...trap/engines/engine.py.470: Saving model... [W 2022-10-10 23:45:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-10 23:45:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-10 23:45:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-10 23:45:31] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-10 23:45:34] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-10 23:45:38] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-10 23:45:38] ...trap/engines/engine.py.163: Training model on trainset for epoch 90 [I 2022-10-10 23:45:38] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-10 23:45:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 0/2304 [I 2022-10-10 23:45:41] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:07:31 [I 2022-10-10 23:45:41] ...trap/engines/engine.py.229: process: 3.32095 | load: 2.38638 [I 2022-10-10 23:45:41] ...trap/engines/engine.py.230: loss: 0.19787 [I 2022-10-10 23:45:41] ...optimizers/trijoint.py.151: total_norm: 25.727745 [I 2022-10-10 23:45:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 10/2304 [I 2022-10-10 23:45:50] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:13 [I 2022-10-10 23:45:50] ...trap/engines/engine.py.229: process: 0.86851 | load: 0.00684 [I 2022-10-10 23:45:50] ...trap/engines/engine.py.230: loss: 0.09765 [I 2022-10-10 23:45:50] ...optimizers/trijoint.py.151: total_norm: 12.700268 [I 2022-10-10 23:45:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 20/2304 [I 2022-10-10 23:45:58] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:50 [I 2022-10-10 23:45:58] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00494 [I 2022-10-10 23:45:58] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-10 23:45:58] ...optimizers/trijoint.py.151: total_norm: 18.484627 [I 2022-10-10 23:46:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 30/2304 [I 2022-10-10 23:46:07] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:09 [I 2022-10-10 23:46:07] ...trap/engines/engine.py.229: process: 0.88417 | load: 0.00357 [I 2022-10-10 23:46:07] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-10 23:46:07] ...optimizers/trijoint.py.151: total_norm: 9.541264 [I 2022-10-10 23:46:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 40/2304 [I 2022-10-10 23:46:16] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:23 [I 2022-10-10 23:46:16] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00404 [I 2022-10-10 23:46:16] ...trap/engines/engine.py.230: loss: 0.09775 [I 2022-10-10 23:46:16] ...optimizers/trijoint.py.151: total_norm: 20.431993 [I 2022-10-10 23:46:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 50/2304 [I 2022-10-10 23:46:24] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:34 [I 2022-10-10 23:46:24] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00432 [I 2022-10-10 23:46:24] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-10 23:46:24] ...optimizers/trijoint.py.151: total_norm: 19.108068 [I 2022-10-10 23:46:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 60/2304 [I 2022-10-10 23:46:33] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:16 [I 2022-10-10 23:46:33] ...trap/engines/engine.py.229: process: 0.87290 | load: 0.00407 [I 2022-10-10 23:46:33] ...trap/engines/engine.py.230: loss: 0.11133 [I 2022-10-10 23:46:33] ...optimizers/trijoint.py.151: total_norm: 9.820817 [I 2022-10-10 23:46:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 70/2304 [I 2022-10-10 23:46:42] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:16 [I 2022-10-10 23:46:42] ...trap/engines/engine.py.229: process: 0.86668 | load: 0.00380 [I 2022-10-10 23:46:42] ...trap/engines/engine.py.230: loss: 0.10381 [I 2022-10-10 23:46:42] ...optimizers/trijoint.py.151: total_norm: 27.699808 [I 2022-10-10 23:46:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 80/2304 [I 2022-10-10 23:46:50] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:10 [I 2022-10-10 23:46:50] ...trap/engines/engine.py.229: process: 0.87240 | load: 0.00356 [I 2022-10-10 23:46:50] ...trap/engines/engine.py.230: loss: 0.18119 [I 2022-10-10 23:46:50] ...optimizers/trijoint.py.151: total_norm: 25.518946 [I 2022-10-10 23:46:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 90/2304 [I 2022-10-10 23:46:59] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:55 [I 2022-10-10 23:46:59] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00350 [I 2022-10-10 23:46:59] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-10 23:46:59] ...optimizers/trijoint.py.151: total_norm: 11.434869 [I 2022-10-10 23:47:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 100/2304 [I 2022-10-10 23:47:08] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:31:39 [I 2022-10-10 23:47:08] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00324 [I 2022-10-10 23:47:08] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-10 23:47:08] ...optimizers/trijoint.py.151: total_norm: 34.123543 [I 2022-10-10 23:47:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 110/2304 [I 2022-10-10 23:47:16] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:31 [I 2022-10-10 23:47:16] ...trap/engines/engine.py.229: process: 0.86890 | load: 0.00331 [I 2022-10-10 23:47:16] ...trap/engines/engine.py.230: loss: 0.26275 [I 2022-10-10 23:47:16] ...optimizers/trijoint.py.151: total_norm: 17.920881 [I 2022-10-10 23:47:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 120/2304 [I 2022-10-10 23:47:25] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:25 [I 2022-10-10 23:47:25] ...trap/engines/engine.py.229: process: 0.87271 | load: 0.00310 [I 2022-10-10 23:47:25] ...trap/engines/engine.py.230: loss: 0.08563 [I 2022-10-10 23:47:25] ...optimizers/trijoint.py.151: total_norm: 13.001943 [I 2022-10-10 23:47:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 130/2304 [I 2022-10-10 23:47:34] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:19 [I 2022-10-10 23:47:34] ...trap/engines/engine.py.229: process: 0.85760 | load: 0.00331 [I 2022-10-10 23:47:34] ...trap/engines/engine.py.230: loss: 0.12000 [I 2022-10-10 23:47:34] ...optimizers/trijoint.py.151: total_norm: 15.991519 [I 2022-10-10 23:47:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 140/2304 [I 2022-10-10 23:47:42] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:15 [I 2022-10-10 23:47:42] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00336 [I 2022-10-10 23:47:42] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-10 23:47:42] ...optimizers/trijoint.py.151: total_norm: 10.890669 [I 2022-10-10 23:47:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 150/2304 [I 2022-10-10 23:47:51] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:31:03 [I 2022-10-10 23:47:51] ...trap/engines/engine.py.229: process: 0.86421 | load: 0.00340 [I 2022-10-10 23:47:51] ...trap/engines/engine.py.230: loss: 0.13156 [I 2022-10-10 23:47:51] ...optimizers/trijoint.py.151: total_norm: 25.696623 [I 2022-10-10 23:48:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 160/2304 [I 2022-10-10 23:48:00] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:30:57 [I 2022-10-10 23:48:00] ...trap/engines/engine.py.229: process: 0.86800 | load: 0.00332 [I 2022-10-10 23:48:00] ...trap/engines/engine.py.230: loss: 0.09218 [I 2022-10-10 23:48:00] ...optimizers/trijoint.py.151: total_norm: 11.306977 [I 2022-10-10 23:48:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 170/2304 [I 2022-10-10 23:48:08] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:28 [I 2022-10-10 23:48:08] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00321 [I 2022-10-10 23:48:08] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-10 23:48:08] ...optimizers/trijoint.py.151: total_norm: 26.842937 [I 2022-10-10 23:48:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 180/2304 [I 2022-10-10 23:48:17] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:20 [I 2022-10-10 23:48:17] ...trap/engines/engine.py.229: process: 0.86057 | load: 0.00321 [I 2022-10-10 23:48:17] ...trap/engines/engine.py.230: loss: 0.08482 [I 2022-10-10 23:48:17] ...optimizers/trijoint.py.151: total_norm: 12.393434 [I 2022-10-10 23:48:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 190/2304 [I 2022-10-10 23:48:26] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:18 [I 2022-10-10 23:48:26] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00309 [I 2022-10-10 23:48:26] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-10 23:48:26] ...optimizers/trijoint.py.151: total_norm: 24.265079 [I 2022-10-10 23:48:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 200/2304 [I 2022-10-10 23:48:34] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:18 [I 2022-10-10 23:48:34] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00309 [I 2022-10-10 23:48:34] ...trap/engines/engine.py.230: loss: 0.19556 [I 2022-10-10 23:48:34] ...optimizers/trijoint.py.151: total_norm: 22.295698 [I 2022-10-10 23:48:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 210/2304 [I 2022-10-10 23:48:43] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:30:21 [I 2022-10-10 23:48:43] ...trap/engines/engine.py.229: process: 0.85588 | load: 0.00320 [I 2022-10-10 23:48:43] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-10 23:48:43] ...optimizers/trijoint.py.151: total_norm: 17.965242 [I 2022-10-10 23:48:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 220/2304 [I 2022-10-10 23:48:52] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:57 [I 2022-10-10 23:48:52] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00308 [I 2022-10-10 23:48:52] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-10 23:48:52] ...optimizers/trijoint.py.151: total_norm: 8.546876 [I 2022-10-10 23:49:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 230/2304 [I 2022-10-10 23:49:00] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:59 [I 2022-10-10 23:49:00] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00331 [I 2022-10-10 23:49:00] ...trap/engines/engine.py.230: loss: 0.11593 [I 2022-10-10 23:49:00] ...optimizers/trijoint.py.151: total_norm: 23.410574 [I 2022-10-10 23:49:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 240/2304 [I 2022-10-10 23:49:09] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:29:49 [I 2022-10-10 23:49:09] ...trap/engines/engine.py.229: process: 0.86641 | load: 0.00333 [I 2022-10-10 23:49:09] ...trap/engines/engine.py.230: loss: 0.11871 [I 2022-10-10 23:49:09] ...optimizers/trijoint.py.151: total_norm: 13.788388 [I 2022-10-10 23:49:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 250/2304 [I 2022-10-10 23:49:18] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:38 [I 2022-10-10 23:49:18] ...trap/engines/engine.py.229: process: 0.86007 | load: 0.00332 [I 2022-10-10 23:49:18] ...trap/engines/engine.py.230: loss: 0.17009 [I 2022-10-10 23:49:18] ...optimizers/trijoint.py.151: total_norm: 22.344614 [I 2022-10-10 23:49:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 260/2304 [I 2022-10-10 23:49:26] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:40 [I 2022-10-10 23:49:26] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00338 [I 2022-10-10 23:49:26] ...trap/engines/engine.py.230: loss: 0.13513 [I 2022-10-10 23:49:26] ...optimizers/trijoint.py.151: total_norm: 9.239226 [I 2022-10-10 23:49:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 270/2304 [I 2022-10-10 23:49:35] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:29:15 [I 2022-10-10 23:49:35] ...trap/engines/engine.py.229: process: 0.86488 | load: 0.00308 [I 2022-10-10 23:49:35] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-10 23:49:35] ...optimizers/trijoint.py.151: total_norm: 11.142133 [I 2022-10-10 23:49:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 280/2304 [I 2022-10-10 23:49:44] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:29:10 [I 2022-10-10 23:49:44] ...trap/engines/engine.py.229: process: 0.86902 | load: 0.00314 [I 2022-10-10 23:49:44] ...trap/engines/engine.py.230: loss: 0.11388 [I 2022-10-10 23:49:44] ...optimizers/trijoint.py.151: total_norm: 26.845619 [I 2022-10-10 23:49:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 290/2304 [I 2022-10-10 23:49:52] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:29:09 [I 2022-10-10 23:49:52] ...trap/engines/engine.py.229: process: 0.86698 | load: 0.00309 [I 2022-10-10 23:49:52] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-10 23:49:52] ...optimizers/trijoint.py.151: total_norm: 8.803080 [I 2022-10-10 23:50:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 300/2304 [I 2022-10-10 23:50:01] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:28:42 [I 2022-10-10 23:50:01] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00309 [I 2022-10-10 23:50:01] ...trap/engines/engine.py.230: loss: 0.15652 [I 2022-10-10 23:50:01] ...optimizers/trijoint.py.151: total_norm: 14.340113 [I 2022-10-10 23:50:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 310/2304 [I 2022-10-10 23:50:10] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:39 [I 2022-10-10 23:50:10] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00303 [I 2022-10-10 23:50:10] ...trap/engines/engine.py.230: loss: 0.16087 [I 2022-10-10 23:50:10] ...optimizers/trijoint.py.151: total_norm: 12.912170 [I 2022-10-10 23:50:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 320/2304 [I 2022-10-10 23:50:18] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:28:19 [I 2022-10-10 23:50:18] ...trap/engines/engine.py.229: process: 0.83675 | load: 0.00298 [I 2022-10-10 23:50:18] ...trap/engines/engine.py.230: loss: 0.10002 [I 2022-10-10 23:50:18] ...optimizers/trijoint.py.151: total_norm: 7.558498 [I 2022-10-10 23:50:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 330/2304 [I 2022-10-10 23:50:27] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:28:20 [I 2022-10-10 23:50:27] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00304 [I 2022-10-10 23:50:27] ...trap/engines/engine.py.230: loss: 0.14616 [I 2022-10-10 23:50:27] ...optimizers/trijoint.py.151: total_norm: 9.772861 [I 2022-10-10 23:50:36] ...trap/engines/engine.py.224: train: epoch 90 | batch 340/2304 [I 2022-10-10 23:50:36] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:12 [I 2022-10-10 23:50:36] ...trap/engines/engine.py.229: process: 0.86597 | load: 0.00308 [I 2022-10-10 23:50:36] ...trap/engines/engine.py.230: loss: 0.10081 [I 2022-10-10 23:50:36] ...optimizers/trijoint.py.151: total_norm: 8.009610 [I 2022-10-10 23:50:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 350/2304 [I 2022-10-10 23:50:44] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:28:05 [I 2022-10-10 23:50:44] ...trap/engines/engine.py.229: process: 0.86742 | load: 0.00317 [I 2022-10-10 23:50:44] ...trap/engines/engine.py.230: loss: 0.10900 [I 2022-10-10 23:50:44] ...optimizers/trijoint.py.151: total_norm: 13.095724 [I 2022-10-10 23:50:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 360/2304 [I 2022-10-10 23:50:53] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:28:02 [I 2022-10-10 23:50:53] ...trap/engines/engine.py.229: process: 0.86825 | load: 0.00324 [I 2022-10-10 23:50:53] ...trap/engines/engine.py.230: loss: 0.13254 [I 2022-10-10 23:50:53] ...optimizers/trijoint.py.151: total_norm: 15.112555 [I 2022-10-10 23:51:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 370/2304 [I 2022-10-10 23:51:02] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:56 [I 2022-10-10 23:51:02] ...trap/engines/engine.py.229: process: 0.86906 | load: 0.00321 [I 2022-10-10 23:51:02] ...trap/engines/engine.py.230: loss: 0.11392 [I 2022-10-10 23:51:02] ...optimizers/trijoint.py.151: total_norm: 13.269175 [I 2022-10-10 23:51:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 380/2304 [I 2022-10-10 23:51:10] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:39 [I 2022-10-10 23:51:10] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00308 [I 2022-10-10 23:51:10] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-10 23:51:10] ...optimizers/trijoint.py.151: total_norm: 16.792353 [I 2022-10-10 23:51:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 390/2304 [I 2022-10-10 23:51:19] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:27:45 [I 2022-10-10 23:51:19] ...trap/engines/engine.py.229: process: 0.87482 | load: 0.00316 [I 2022-10-10 23:51:19] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-10 23:51:19] ...optimizers/trijoint.py.151: total_norm: 17.746815 [I 2022-10-10 23:51:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 400/2304 [I 2022-10-10 23:51:28] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:27 [I 2022-10-10 23:51:28] ...trap/engines/engine.py.229: process: 0.86176 | load: 0.00325 [I 2022-10-10 23:51:28] ...trap/engines/engine.py.230: loss: 0.12334 [I 2022-10-10 23:51:28] ...optimizers/trijoint.py.151: total_norm: 12.159719 [I 2022-10-10 23:51:36] ...trap/engines/engine.py.224: train: epoch 90 | batch 410/2304 [I 2022-10-10 23:51:36] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:27:17 [I 2022-10-10 23:51:36] ...trap/engines/engine.py.229: process: 0.85658 | load: 0.00306 [I 2022-10-10 23:51:36] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-10 23:51:36] ...optimizers/trijoint.py.151: total_norm: 12.602002 [I 2022-10-10 23:51:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 420/2304 [I 2022-10-10 23:51:45] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:27:01 [I 2022-10-10 23:51:45] ...trap/engines/engine.py.229: process: 0.85968 | load: 0.00310 [I 2022-10-10 23:51:45] ...trap/engines/engine.py.230: loss: 0.07716 [I 2022-10-10 23:51:45] ...optimizers/trijoint.py.151: total_norm: 9.844158 [I 2022-10-10 23:51:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 430/2304 [I 2022-10-10 23:51:53] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:49 [I 2022-10-10 23:51:53] ...trap/engines/engine.py.229: process: 0.86393 | load: 0.00333 [I 2022-10-10 23:51:53] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-10 23:51:53] ...optimizers/trijoint.py.151: total_norm: 24.196175 [I 2022-10-10 23:52:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 440/2304 [I 2022-10-10 23:52:02] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:44 [I 2022-10-10 23:52:02] ...trap/engines/engine.py.229: process: 0.86191 | load: 0.00316 [I 2022-10-10 23:52:02] ...trap/engines/engine.py.230: loss: 0.17119 [I 2022-10-10 23:52:02] ...optimizers/trijoint.py.151: total_norm: 15.236626 [I 2022-10-10 23:52:11] ...trap/engines/engine.py.224: train: epoch 90 | batch 450/2304 [I 2022-10-10 23:52:11] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:40 [I 2022-10-10 23:52:11] ...trap/engines/engine.py.229: process: 0.87728 | load: 0.00305 [I 2022-10-10 23:52:11] ...trap/engines/engine.py.230: loss: 0.16447 [I 2022-10-10 23:52:11] ...optimizers/trijoint.py.151: total_norm: 13.304707 [I 2022-10-10 23:52:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 460/2304 [I 2022-10-10 23:52:19] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:26:35 [I 2022-10-10 23:52:19] ...trap/engines/engine.py.229: process: 0.86950 | load: 0.00305 [I 2022-10-10 23:52:19] ...trap/engines/engine.py.230: loss: 0.19618 [I 2022-10-10 23:52:19] ...optimizers/trijoint.py.151: total_norm: 28.759281 [I 2022-10-10 23:52:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 470/2304 [I 2022-10-10 23:52:28] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:26:23 [I 2022-10-10 23:52:28] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00306 [I 2022-10-10 23:52:28] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-10 23:52:28] ...optimizers/trijoint.py.151: total_norm: 18.098751 [I 2022-10-10 23:52:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 480/2304 [I 2022-10-10 23:52:37] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:26:22 [I 2022-10-10 23:52:37] ...trap/engines/engine.py.229: process: 0.87675 | load: 0.00312 [I 2022-10-10 23:52:37] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-10 23:52:37] ...optimizers/trijoint.py.151: total_norm: 33.240490 [I 2022-10-10 23:52:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 490/2304 [I 2022-10-10 23:52:45] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:54 [I 2022-10-10 23:52:45] ...trap/engines/engine.py.229: process: 0.85635 | load: 0.00306 [I 2022-10-10 23:52:45] ...trap/engines/engine.py.230: loss: 0.11093 [I 2022-10-10 23:52:45] ...optimizers/trijoint.py.151: total_norm: 9.332050 [I 2022-10-10 23:52:54] ...trap/engines/engine.py.224: train: epoch 90 | batch 500/2304 [I 2022-10-10 23:52:54] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:25:44 [I 2022-10-10 23:52:54] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00321 [I 2022-10-10 23:52:54] ...trap/engines/engine.py.230: loss: 0.14253 [I 2022-10-10 23:52:54] ...optimizers/trijoint.py.151: total_norm: 26.601864 [I 2022-10-10 23:53:03] ...trap/engines/engine.py.224: train: epoch 90 | batch 510/2304 [I 2022-10-10 23:53:03] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:45 [I 2022-10-10 23:53:03] ...trap/engines/engine.py.229: process: 0.86687 | load: 0.00310 [I 2022-10-10 23:53:03] ...trap/engines/engine.py.230: loss: 0.14028 [I 2022-10-10 23:53:03] ...optimizers/trijoint.py.151: total_norm: 30.581436 [I 2022-10-10 23:53:11] ...trap/engines/engine.py.224: train: epoch 90 | batch 520/2304 [I 2022-10-10 23:53:11] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:46 [I 2022-10-10 23:53:11] ...trap/engines/engine.py.229: process: 0.88082 | load: 0.00312 [I 2022-10-10 23:53:11] ...trap/engines/engine.py.230: loss: 0.12179 [I 2022-10-10 23:53:11] ...optimizers/trijoint.py.151: total_norm: 21.733776 [I 2022-10-10 23:53:20] ...trap/engines/engine.py.224: train: epoch 90 | batch 530/2304 [I 2022-10-10 23:53:20] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:25:37 [I 2022-10-10 23:53:20] ...trap/engines/engine.py.229: process: 0.87461 | load: 0.00306 [I 2022-10-10 23:53:20] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-10 23:53:20] ...optimizers/trijoint.py.151: total_norm: 16.705986 [I 2022-10-10 23:53:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 540/2304 [I 2022-10-10 23:53:29] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:25:28 [I 2022-10-10 23:53:29] ...trap/engines/engine.py.229: process: 0.86802 | load: 0.00312 [I 2022-10-10 23:53:29] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-10 23:53:29] ...optimizers/trijoint.py.151: total_norm: 16.225250 [I 2022-10-10 23:53:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 550/2304 [I 2022-10-10 23:53:37] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:25:18 [I 2022-10-10 23:53:37] ...trap/engines/engine.py.229: process: 0.86074 | load: 0.00314 [I 2022-10-10 23:53:37] ...trap/engines/engine.py.230: loss: 0.14835 [I 2022-10-10 23:53:37] ...optimizers/trijoint.py.151: total_norm: 9.134521 [I 2022-10-10 23:53:46] ...trap/engines/engine.py.224: train: epoch 90 | batch 560/2304 [I 2022-10-10 23:53:46] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:25:07 [I 2022-10-10 23:53:46] ...trap/engines/engine.py.229: process: 0.85732 | load: 0.00319 [I 2022-10-10 23:53:46] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-10 23:53:46] ...optimizers/trijoint.py.151: total_norm: 15.259476 [I 2022-10-10 23:53:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 570/2304 [I 2022-10-10 23:53:55] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:50 [I 2022-10-10 23:53:55] ...trap/engines/engine.py.229: process: 0.86070 | load: 0.00326 [I 2022-10-10 23:53:55] ...trap/engines/engine.py.230: loss: 0.20902 [I 2022-10-10 23:53:55] ...optimizers/trijoint.py.151: total_norm: 28.823132 [I 2022-10-10 23:54:03] ...trap/engines/engine.py.224: train: epoch 90 | batch 580/2304 [I 2022-10-10 23:54:03] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:39 [I 2022-10-10 23:54:03] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00314 [I 2022-10-10 23:54:03] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-10 23:54:03] ...optimizers/trijoint.py.151: total_norm: 11.810023 [I 2022-10-10 23:54:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 590/2304 [I 2022-10-10 23:54:12] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:24:33 [I 2022-10-10 23:54:12] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00316 [I 2022-10-10 23:54:12] ...trap/engines/engine.py.230: loss: 0.10072 [I 2022-10-10 23:54:12] ...optimizers/trijoint.py.151: total_norm: 11.149099 [I 2022-10-10 23:54:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 600/2304 [I 2022-10-10 23:54:21] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:23 [I 2022-10-10 23:54:21] ...trap/engines/engine.py.229: process: 0.85893 | load: 0.00307 [I 2022-10-10 23:54:21] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-10 23:54:21] ...optimizers/trijoint.py.151: total_norm: 10.291800 [I 2022-10-10 23:54:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 610/2304 [I 2022-10-10 23:54:29] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:21 [I 2022-10-10 23:54:29] ...trap/engines/engine.py.229: process: 0.86904 | load: 0.00313 [I 2022-10-10 23:54:29] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-10 23:54:29] ...optimizers/trijoint.py.151: total_norm: 12.439356 [I 2022-10-10 23:54:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 620/2304 [I 2022-10-10 23:54:38] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:24:13 [I 2022-10-10 23:54:38] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00308 [I 2022-10-10 23:54:38] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-10 23:54:38] ...optimizers/trijoint.py.151: total_norm: 18.072231 [I 2022-10-10 23:54:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 630/2304 [I 2022-10-10 23:54:47] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:24:02 [I 2022-10-10 23:54:47] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00301 [I 2022-10-10 23:54:47] ...trap/engines/engine.py.230: loss: 0.09200 [I 2022-10-10 23:54:47] ...optimizers/trijoint.py.151: total_norm: 10.659145 [I 2022-10-10 23:54:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 640/2304 [I 2022-10-10 23:54:55] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:52 [I 2022-10-10 23:54:55] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00320 [I 2022-10-10 23:54:55] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-10 23:54:55] ...optimizers/trijoint.py.151: total_norm: 17.608158 [I 2022-10-10 23:55:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 650/2304 [I 2022-10-10 23:55:04] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:53 [I 2022-10-10 23:55:04] ...trap/engines/engine.py.229: process: 0.87745 | load: 0.00330 [I 2022-10-10 23:55:04] ...trap/engines/engine.py.230: loss: 0.16913 [I 2022-10-10 23:55:04] ...optimizers/trijoint.py.151: total_norm: 12.218773 [I 2022-10-10 23:55:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 660/2304 [I 2022-10-10 23:55:13] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:48 [I 2022-10-10 23:55:13] ...trap/engines/engine.py.229: process: 0.87085 | load: 0.00333 [I 2022-10-10 23:55:13] ...trap/engines/engine.py.230: loss: 0.11440 [I 2022-10-10 23:55:13] ...optimizers/trijoint.py.151: total_norm: 16.641155 [I 2022-10-10 23:55:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 670/2304 [I 2022-10-10 23:55:21] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:28 [I 2022-10-10 23:55:21] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00311 [I 2022-10-10 23:55:21] ...trap/engines/engine.py.230: loss: 0.15952 [I 2022-10-10 23:55:21] ...optimizers/trijoint.py.151: total_norm: 32.394123 [I 2022-10-10 23:55:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 680/2304 [I 2022-10-10 23:55:30] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:23:29 [I 2022-10-10 23:55:30] ...trap/engines/engine.py.229: process: 0.86870 | load: 0.00318 [I 2022-10-10 23:55:30] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-10 23:55:30] ...optimizers/trijoint.py.151: total_norm: 8.542903 [I 2022-10-10 23:55:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 690/2304 [I 2022-10-10 23:55:39] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:23:10 [I 2022-10-10 23:55:39] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00305 [I 2022-10-10 23:55:39] ...trap/engines/engine.py.230: loss: 0.08604 [I 2022-10-10 23:55:39] ...optimizers/trijoint.py.151: total_norm: 53.074165 [I 2022-10-10 23:55:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 700/2304 [I 2022-10-10 23:55:47] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:22:51 [I 2022-10-10 23:55:47] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00295 [I 2022-10-10 23:55:47] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-10 23:55:47] ...optimizers/trijoint.py.151: total_norm: 13.604739 [I 2022-10-10 23:55:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 710/2304 [I 2022-10-10 23:55:56] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:32 [I 2022-10-10 23:55:56] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00293 [I 2022-10-10 23:55:56] ...trap/engines/engine.py.230: loss: 0.09017 [I 2022-10-10 23:55:56] ...optimizers/trijoint.py.151: total_norm: 17.410910 [I 2022-10-10 23:56:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 720/2304 [I 2022-10-10 23:56:04] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:27 [I 2022-10-10 23:56:04] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00295 [I 2022-10-10 23:56:04] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-10 23:56:04] ...optimizers/trijoint.py.151: total_norm: 15.575025 [I 2022-10-10 23:56:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 730/2304 [I 2022-10-10 23:56:13] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:21 [I 2022-10-10 23:56:13] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00302 [I 2022-10-10 23:56:13] ...trap/engines/engine.py.230: loss: 0.14526 [I 2022-10-10 23:56:13] ...optimizers/trijoint.py.151: total_norm: 25.641708 [I 2022-10-10 23:56:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 740/2304 [I 2022-10-10 23:56:21] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:05 [I 2022-10-10 23:56:21] ...trap/engines/engine.py.229: process: 0.83390 | load: 0.00294 [I 2022-10-10 23:56:21] ...trap/engines/engine.py.230: loss: 0.17155 [I 2022-10-10 23:56:21] ...optimizers/trijoint.py.151: total_norm: 13.031736 [I 2022-10-10 23:56:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 750/2304 [I 2022-10-10 23:56:30] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:22:09 [I 2022-10-10 23:56:30] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00301 [I 2022-10-10 23:56:30] ...trap/engines/engine.py.230: loss: 0.09778 [I 2022-10-10 23:56:30] ...optimizers/trijoint.py.151: total_norm: 9.916236 [I 2022-10-10 23:56:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 760/2304 [I 2022-10-10 23:56:38] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:22:06 [I 2022-10-10 23:56:38] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00298 [I 2022-10-10 23:56:38] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-10 23:56:38] ...optimizers/trijoint.py.151: total_norm: 14.108548 [I 2022-10-10 23:56:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 770/2304 [I 2022-10-10 23:56:47] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:48 [I 2022-10-10 23:56:47] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00308 [I 2022-10-10 23:56:47] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-10 23:56:47] ...optimizers/trijoint.py.151: total_norm: 8.776482 [I 2022-10-10 23:56:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 780/2304 [I 2022-10-10 23:56:56] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:45 [I 2022-10-10 23:56:56] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00304 [I 2022-10-10 23:56:56] ...trap/engines/engine.py.230: loss: 0.09328 [I 2022-10-10 23:56:56] ...optimizers/trijoint.py.151: total_norm: 14.625532 [I 2022-10-10 23:57:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 790/2304 [I 2022-10-10 23:57:04] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:33 [I 2022-10-10 23:57:04] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00301 [I 2022-10-10 23:57:04] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-10 23:57:04] ...optimizers/trijoint.py.151: total_norm: 12.292857 [I 2022-10-10 23:57:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 800/2304 [I 2022-10-10 23:57:13] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:27 [I 2022-10-10 23:57:13] ...trap/engines/engine.py.229: process: 0.86915 | load: 0.00298 [I 2022-10-10 23:57:13] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-10 23:57:13] ...optimizers/trijoint.py.151: total_norm: 20.281660 [I 2022-10-10 23:57:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 810/2304 [I 2022-10-10 23:57:21] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:17 [I 2022-10-10 23:57:21] ...trap/engines/engine.py.229: process: 0.86429 | load: 0.00302 [I 2022-10-10 23:57:21] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-10 23:57:21] ...optimizers/trijoint.py.151: total_norm: 10.626276 [I 2022-10-10 23:57:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 820/2304 [I 2022-10-10 23:57:30] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:21:13 [I 2022-10-10 23:57:30] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00297 [I 2022-10-10 23:57:30] ...trap/engines/engine.py.230: loss: 0.08969 [I 2022-10-10 23:57:30] ...optimizers/trijoint.py.151: total_norm: 10.200191 [I 2022-10-10 23:57:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 830/2304 [I 2022-10-10 23:57:38] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:56 [I 2022-10-10 23:57:38] ...trap/engines/engine.py.229: process: 0.83719 | load: 0.00310 [I 2022-10-10 23:57:38] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-10 23:57:38] ...optimizers/trijoint.py.151: total_norm: 16.569324 [I 2022-10-10 23:57:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 840/2304 [I 2022-10-10 23:57:47] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:41 [I 2022-10-10 23:57:47] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00301 [I 2022-10-10 23:57:47] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-10 23:57:47] ...optimizers/trijoint.py.151: total_norm: 14.787406 [I 2022-10-10 23:57:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 850/2304 [I 2022-10-10 23:57:55] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:33 [I 2022-10-10 23:57:55] ...trap/engines/engine.py.229: process: 0.83712 | load: 0.00295 [I 2022-10-10 23:57:55] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-10 23:57:55] ...optimizers/trijoint.py.151: total_norm: 10.645862 [I 2022-10-10 23:58:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 860/2304 [I 2022-10-10 23:58:04] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:20 [I 2022-10-10 23:58:04] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00295 [I 2022-10-10 23:58:04] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-10 23:58:04] ...optimizers/trijoint.py.151: total_norm: 18.333761 [I 2022-10-10 23:58:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 870/2304 [I 2022-10-10 23:58:12] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:29 [I 2022-10-10 23:58:12] ...trap/engines/engine.py.229: process: 0.86207 | load: 0.00312 [I 2022-10-10 23:58:13] ...trap/engines/engine.py.230: loss: 0.14915 [I 2022-10-10 23:58:13] ...optimizers/trijoint.py.151: total_norm: 23.381573 [I 2022-10-10 23:58:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 880/2304 [I 2022-10-10 23:58:21] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:36 [I 2022-10-10 23:58:21] ...trap/engines/engine.py.229: process: 0.86838 | load: 0.00313 [I 2022-10-10 23:58:21] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-10 23:58:21] ...optimizers/trijoint.py.151: total_norm: 12.867050 [I 2022-10-10 23:58:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 890/2304 [I 2022-10-10 23:58:30] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:13 [I 2022-10-10 23:58:30] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00306 [I 2022-10-10 23:58:30] ...trap/engines/engine.py.230: loss: 0.21919 [I 2022-10-10 23:58:30] ...optimizers/trijoint.py.151: total_norm: 18.334925 [I 2022-10-10 23:58:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 900/2304 [I 2022-10-10 23:58:38] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:06 [I 2022-10-10 23:58:38] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00309 [I 2022-10-10 23:58:38] ...trap/engines/engine.py.230: loss: 0.12793 [I 2022-10-10 23:58:38] ...optimizers/trijoint.py.151: total_norm: 14.532873 [I 2022-10-10 23:58:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 910/2304 [I 2022-10-10 23:58:47] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:56 [I 2022-10-10 23:58:47] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00307 [I 2022-10-10 23:58:47] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-10 23:58:47] ...optimizers/trijoint.py.151: total_norm: 16.293196 [I 2022-10-10 23:58:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 920/2304 [I 2022-10-10 23:58:56] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:42 [I 2022-10-10 23:58:56] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00306 [I 2022-10-10 23:58:56] ...trap/engines/engine.py.230: loss: 0.22778 [I 2022-10-10 23:58:56] ...optimizers/trijoint.py.151: total_norm: 45.753006 [I 2022-10-10 23:59:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 930/2304 [I 2022-10-10 23:59:04] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:45 [I 2022-10-10 23:59:04] ...trap/engines/engine.py.229: process: 0.87896 | load: 0.00311 [I 2022-10-10 23:59:04] ...trap/engines/engine.py.230: loss: 0.17152 [I 2022-10-10 23:59:04] ...optimizers/trijoint.py.151: total_norm: 17.620882 [I 2022-10-10 23:59:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 940/2304 [I 2022-10-10 23:59:13] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:35 [I 2022-10-10 23:59:13] ...trap/engines/engine.py.229: process: 0.86268 | load: 0.00350 [I 2022-10-10 23:59:13] ...trap/engines/engine.py.230: loss: 0.09789 [I 2022-10-10 23:59:13] ...optimizers/trijoint.py.151: total_norm: 7.192443 [I 2022-10-10 23:59:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 950/2304 [I 2022-10-10 23:59:21] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:21 [I 2022-10-10 23:59:21] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00309 [I 2022-10-10 23:59:21] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-10 23:59:21] ...optimizers/trijoint.py.151: total_norm: 20.459812 [I 2022-10-10 23:59:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 960/2304 [I 2022-10-10 23:59:30] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:23 [I 2022-10-10 23:59:30] ...trap/engines/engine.py.229: process: 0.87278 | load: 0.00319 [I 2022-10-10 23:59:30] ...trap/engines/engine.py.230: loss: 0.11020 [I 2022-10-10 23:59:30] ...optimizers/trijoint.py.151: total_norm: 10.234970 [I 2022-10-10 23:59:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 970/2304 [I 2022-10-10 23:59:39] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:13 [I 2022-10-10 23:59:39] ...trap/engines/engine.py.229: process: 0.87835 | load: 0.00317 [I 2022-10-10 23:59:39] ...trap/engines/engine.py.230: loss: 0.18295 [I 2022-10-10 23:59:39] ...optimizers/trijoint.py.151: total_norm: 15.776630 [I 2022-10-10 23:59:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 980/2304 [I 2022-10-10 23:59:47] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:06 [I 2022-10-10 23:59:47] ...trap/engines/engine.py.229: process: 0.87010 | load: 0.00309 [I 2022-10-10 23:59:47] ...trap/engines/engine.py.230: loss: 0.08606 [I 2022-10-10 23:59:47] ...optimizers/trijoint.py.151: total_norm: 9.284320 [I 2022-10-10 23:59:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 990/2304 [I 2022-10-10 23:59:56] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:48 [I 2022-10-10 23:59:56] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00311 [I 2022-10-10 23:59:56] ...trap/engines/engine.py.230: loss: 0.18371 [I 2022-10-10 23:59:56] ...optimizers/trijoint.py.151: total_norm: 26.053894 [I 2022-10-11 00:00:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 1000/2304 [I 2022-10-11 00:00:05] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:39 [I 2022-10-11 00:00:05] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00318 [I 2022-10-11 00:00:05] ...trap/engines/engine.py.230: loss: 0.10602 [I 2022-10-11 00:00:05] ...optimizers/trijoint.py.151: total_norm: 15.717710 [I 2022-10-11 00:00:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 1010/2304 [I 2022-10-11 00:00:13] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:35 [I 2022-10-11 00:00:13] ...trap/engines/engine.py.229: process: 0.86488 | load: 0.00317 [I 2022-10-11 00:00:13] ...trap/engines/engine.py.230: loss: 0.13793 [I 2022-10-11 00:00:13] ...optimizers/trijoint.py.151: total_norm: 16.006689 [I 2022-10-11 00:00:22] ...trap/engines/engine.py.224: train: epoch 90 | batch 1020/2304 [I 2022-10-11 00:00:22] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:27 [I 2022-10-11 00:00:22] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00314 [I 2022-10-11 00:00:22] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-11 00:00:22] ...optimizers/trijoint.py.151: total_norm: 18.400967 [I 2022-10-11 00:00:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1030/2304 [I 2022-10-11 00:00:31] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:11 [I 2022-10-11 00:00:31] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00309 [I 2022-10-11 00:00:31] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-11 00:00:31] ...optimizers/trijoint.py.151: total_norm: 23.321688 [I 2022-10-11 00:00:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 1040/2304 [I 2022-10-11 00:00:39] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:10 [I 2022-10-11 00:00:39] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00316 [I 2022-10-11 00:00:39] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-11 00:00:39] ...optimizers/trijoint.py.151: total_norm: 15.975888 [I 2022-10-11 00:00:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 1050/2304 [I 2022-10-11 00:00:48] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:04 [I 2022-10-11 00:00:48] ...trap/engines/engine.py.229: process: 0.86208 | load: 0.00317 [I 2022-10-11 00:00:48] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-11 00:00:48] ...optimizers/trijoint.py.151: total_norm: 10.303035 [I 2022-10-11 00:00:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 1060/2304 [I 2022-10-11 00:00:57] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:18:02 [I 2022-10-11 00:00:57] ...trap/engines/engine.py.229: process: 0.87426 | load: 0.00308 [I 2022-10-11 00:00:57] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-11 00:00:57] ...optimizers/trijoint.py.151: total_norm: 9.945852 [I 2022-10-11 00:01:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 1070/2304 [I 2022-10-11 00:01:05] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:45 [I 2022-10-11 00:01:05] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00308 [I 2022-10-11 00:01:05] ...trap/engines/engine.py.230: loss: 0.09320 [I 2022-10-11 00:01:05] ...optimizers/trijoint.py.151: total_norm: 29.195688 [I 2022-10-11 00:01:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 1080/2304 [I 2022-10-11 00:01:14] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:42 [I 2022-10-11 00:01:14] ...trap/engines/engine.py.229: process: 0.87127 | load: 0.00310 [I 2022-10-11 00:01:14] ...trap/engines/engine.py.230: loss: 0.16580 [I 2022-10-11 00:01:14] ...optimizers/trijoint.py.151: total_norm: 17.215908 [I 2022-10-11 00:01:22] ...trap/engines/engine.py.224: train: epoch 90 | batch 1090/2304 [I 2022-10-11 00:01:22] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:21 [I 2022-10-11 00:01:22] ...trap/engines/engine.py.229: process: 0.86068 | load: 0.00314 [I 2022-10-11 00:01:22] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-11 00:01:22] ...optimizers/trijoint.py.151: total_norm: 8.282394 [I 2022-10-11 00:01:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1100/2304 [I 2022-10-11 00:01:31] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:15 [I 2022-10-11 00:01:31] ...trap/engines/engine.py.229: process: 0.86586 | load: 0.00315 [I 2022-10-11 00:01:31] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-11 00:01:31] ...optimizers/trijoint.py.151: total_norm: 13.405676 [I 2022-10-11 00:01:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 1110/2304 [I 2022-10-11 00:01:40] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:17:07 [I 2022-10-11 00:01:40] ...trap/engines/engine.py.229: process: 0.85928 | load: 0.00307 [I 2022-10-11 00:01:40] ...trap/engines/engine.py.230: loss: 0.08424 [I 2022-10-11 00:01:40] ...optimizers/trijoint.py.151: total_norm: 11.590545 [I 2022-10-11 00:01:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 1120/2304 [I 2022-10-11 00:01:48] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:17:04 [I 2022-10-11 00:01:48] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00324 [I 2022-10-11 00:01:48] ...trap/engines/engine.py.230: loss: 0.19832 [I 2022-10-11 00:01:48] ...optimizers/trijoint.py.151: total_norm: 23.349621 [I 2022-10-11 00:01:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 1130/2304 [I 2022-10-11 00:01:57] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:53 [I 2022-10-11 00:01:57] ...trap/engines/engine.py.229: process: 0.86617 | load: 0.00315 [I 2022-10-11 00:01:57] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-11 00:01:57] ...optimizers/trijoint.py.151: total_norm: 12.003099 [I 2022-10-11 00:02:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 1140/2304 [I 2022-10-11 00:02:06] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:39 [I 2022-10-11 00:02:06] ...trap/engines/engine.py.229: process: 0.87297 | load: 0.00299 [I 2022-10-11 00:02:06] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-11 00:02:06] ...optimizers/trijoint.py.151: total_norm: 9.521992 [I 2022-10-11 00:02:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 1150/2304 [I 2022-10-11 00:02:14] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:22 [I 2022-10-11 00:02:14] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00295 [I 2022-10-11 00:02:14] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-11 00:02:14] ...optimizers/trijoint.py.151: total_norm: 20.371464 [I 2022-10-11 00:02:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1160/2304 [I 2022-10-11 00:02:23] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:17 [I 2022-10-11 00:02:23] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00297 [I 2022-10-11 00:02:23] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-11 00:02:23] ...optimizers/trijoint.py.151: total_norm: 10.190780 [I 2022-10-11 00:02:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1170/2304 [I 2022-10-11 00:02:31] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:19 [I 2022-10-11 00:02:31] ...trap/engines/engine.py.229: process: 0.87374 | load: 0.00303 [I 2022-10-11 00:02:31] ...trap/engines/engine.py.230: loss: 0.18266 [I 2022-10-11 00:02:31] ...optimizers/trijoint.py.151: total_norm: 29.625847 [I 2022-10-11 00:02:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 1180/2304 [I 2022-10-11 00:02:40] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:16:02 [I 2022-10-11 00:02:40] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00297 [I 2022-10-11 00:02:40] ...trap/engines/engine.py.230: loss: 0.08568 [I 2022-10-11 00:02:40] ...optimizers/trijoint.py.151: total_norm: 13.862990 [I 2022-10-11 00:02:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 1190/2304 [I 2022-10-11 00:02:48] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:52 [I 2022-10-11 00:02:48] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00301 [I 2022-10-11 00:02:48] ...trap/engines/engine.py.230: loss: 0.10811 [I 2022-10-11 00:02:48] ...optimizers/trijoint.py.151: total_norm: 10.385459 [I 2022-10-11 00:02:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 1200/2304 [I 2022-10-11 00:02:57] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:49 [I 2022-10-11 00:02:57] ...trap/engines/engine.py.229: process: 0.85792 | load: 0.00299 [I 2022-10-11 00:02:57] ...trap/engines/engine.py.230: loss: 0.16442 [I 2022-10-11 00:02:57] ...optimizers/trijoint.py.151: total_norm: 22.142281 [I 2022-10-11 00:03:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 1210/2304 [I 2022-10-11 00:03:06] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:33 [I 2022-10-11 00:03:06] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00300 [I 2022-10-11 00:03:06] ...trap/engines/engine.py.230: loss: 0.10333 [I 2022-10-11 00:03:06] ...optimizers/trijoint.py.151: total_norm: 17.354633 [I 2022-10-11 00:03:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 1220/2304 [I 2022-10-11 00:03:14] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:26 [I 2022-10-11 00:03:14] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00299 [I 2022-10-11 00:03:14] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-11 00:03:14] ...optimizers/trijoint.py.151: total_norm: 20.193356 [I 2022-10-11 00:03:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1230/2304 [I 2022-10-11 00:03:23] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:15:15 [I 2022-10-11 00:03:23] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00297 [I 2022-10-11 00:03:23] ...trap/engines/engine.py.230: loss: 0.14939 [I 2022-10-11 00:03:23] ...optimizers/trijoint.py.151: total_norm: 14.420534 [I 2022-10-11 00:03:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1240/2304 [I 2022-10-11 00:03:31] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:14 [I 2022-10-11 00:03:31] ...trap/engines/engine.py.229: process: 0.85679 | load: 0.00308 [I 2022-10-11 00:03:31] ...trap/engines/engine.py.230: loss: 0.13411 [I 2022-10-11 00:03:31] ...optimizers/trijoint.py.151: total_norm: 13.369433 [I 2022-10-11 00:03:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 1250/2304 [I 2022-10-11 00:03:40] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:15:03 [I 2022-10-11 00:03:40] ...trap/engines/engine.py.229: process: 0.85472 | load: 0.00309 [I 2022-10-11 00:03:40] ...trap/engines/engine.py.230: loss: 0.08083 [I 2022-10-11 00:03:40] ...optimizers/trijoint.py.151: total_norm: 9.006892 [I 2022-10-11 00:03:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 1260/2304 [I 2022-10-11 00:03:49] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:52 [I 2022-10-11 00:03:49] ...trap/engines/engine.py.229: process: 0.84065 | load: 0.00305 [I 2022-10-11 00:03:49] ...trap/engines/engine.py.230: loss: 0.15613 [I 2022-10-11 00:03:49] ...optimizers/trijoint.py.151: total_norm: 18.497166 [I 2022-10-11 00:03:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 1270/2304 [I 2022-10-11 00:03:57] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:51 [I 2022-10-11 00:03:57] ...trap/engines/engine.py.229: process: 0.86508 | load: 0.00327 [I 2022-10-11 00:03:57] ...trap/engines/engine.py.230: loss: 0.14544 [I 2022-10-11 00:03:57] ...optimizers/trijoint.py.151: total_norm: 40.346062 [I 2022-10-11 00:04:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 1280/2304 [I 2022-10-11 00:04:06] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:48 [I 2022-10-11 00:04:06] ...trap/engines/engine.py.229: process: 0.85920 | load: 0.00317 [I 2022-10-11 00:04:06] ...trap/engines/engine.py.230: loss: 0.13283 [I 2022-10-11 00:04:06] ...optimizers/trijoint.py.151: total_norm: 14.720190 [I 2022-10-11 00:04:15] ...trap/engines/engine.py.224: train: epoch 90 | batch 1290/2304 [I 2022-10-11 00:04:15] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:30 [I 2022-10-11 00:04:15] ...trap/engines/engine.py.229: process: 0.84591 | load: 0.00313 [I 2022-10-11 00:04:15] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-11 00:04:15] ...optimizers/trijoint.py.151: total_norm: 20.484903 [I 2022-10-11 00:04:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1300/2304 [I 2022-10-11 00:04:23] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:24 [I 2022-10-11 00:04:23] ...trap/engines/engine.py.229: process: 0.86390 | load: 0.00315 [I 2022-10-11 00:04:23] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-11 00:04:23] ...optimizers/trijoint.py.151: total_norm: 10.449418 [I 2022-10-11 00:04:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 1310/2304 [I 2022-10-11 00:04:32] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:19 [I 2022-10-11 00:04:32] ...trap/engines/engine.py.229: process: 0.86721 | load: 0.00305 [I 2022-10-11 00:04:32] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-11 00:04:32] ...optimizers/trijoint.py.151: total_norm: 8.029321 [I 2022-10-11 00:04:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 1320/2304 [I 2022-10-11 00:04:40] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:05 [I 2022-10-11 00:04:40] ...trap/engines/engine.py.229: process: 0.86088 | load: 0.00327 [I 2022-10-11 00:04:40] ...trap/engines/engine.py.230: loss: 0.08662 [I 2022-10-11 00:04:40] ...optimizers/trijoint.py.151: total_norm: 16.047977 [I 2022-10-11 00:04:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 1330/2304 [I 2022-10-11 00:04:49] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:13:54 [I 2022-10-11 00:04:49] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00310 [I 2022-10-11 00:04:49] ...trap/engines/engine.py.230: loss: 0.09074 [I 2022-10-11 00:04:49] ...optimizers/trijoint.py.151: total_norm: 9.775708 [I 2022-10-11 00:04:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 1340/2304 [I 2022-10-11 00:04:58] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:43 [I 2022-10-11 00:04:58] ...trap/engines/engine.py.229: process: 0.83810 | load: 0.00305 [I 2022-10-11 00:04:58] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-11 00:04:58] ...optimizers/trijoint.py.151: total_norm: 12.561163 [I 2022-10-11 00:05:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 1350/2304 [I 2022-10-11 00:05:06] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:39 [I 2022-10-11 00:05:06] ...trap/engines/engine.py.229: process: 0.86711 | load: 0.00314 [I 2022-10-11 00:05:06] ...trap/engines/engine.py.230: loss: 0.15247 [I 2022-10-11 00:05:06] ...optimizers/trijoint.py.151: total_norm: 11.850801 [I 2022-10-11 00:05:15] ...trap/engines/engine.py.224: train: epoch 90 | batch 1360/2304 [I 2022-10-11 00:05:15] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:27 [I 2022-10-11 00:05:15] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00306 [I 2022-10-11 00:05:15] ...trap/engines/engine.py.230: loss: 0.11892 [I 2022-10-11 00:05:15] ...optimizers/trijoint.py.151: total_norm: 10.438815 [I 2022-10-11 00:05:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1370/2304 [I 2022-10-11 00:05:23] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:20 [I 2022-10-11 00:05:23] ...trap/engines/engine.py.229: process: 0.85237 | load: 0.00311 [I 2022-10-11 00:05:23] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-11 00:05:23] ...optimizers/trijoint.py.151: total_norm: 8.749884 [I 2022-10-11 00:05:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 1380/2304 [I 2022-10-11 00:05:32] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:14 [I 2022-10-11 00:05:32] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00307 [I 2022-10-11 00:05:32] ...trap/engines/engine.py.230: loss: 0.10448 [I 2022-10-11 00:05:32] ...optimizers/trijoint.py.151: total_norm: 19.468388 [I 2022-10-11 00:05:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 1390/2304 [I 2022-10-11 00:05:41] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:13:06 [I 2022-10-11 00:05:41] ...trap/engines/engine.py.229: process: 0.87151 | load: 0.00328 [I 2022-10-11 00:05:41] ...trap/engines/engine.py.230: loss: 0.16310 [I 2022-10-11 00:05:41] ...optimizers/trijoint.py.151: total_norm: 39.113365 [I 2022-10-11 00:05:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 1400/2304 [I 2022-10-11 00:05:49] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:59 [I 2022-10-11 00:05:49] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00317 [I 2022-10-11 00:05:49] ...trap/engines/engine.py.230: loss: 0.19475 [I 2022-10-11 00:05:49] ...optimizers/trijoint.py.151: total_norm: 12.003654 [I 2022-10-11 00:05:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 1410/2304 [I 2022-10-11 00:05:58] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:57 [I 2022-10-11 00:05:58] ...trap/engines/engine.py.229: process: 0.88930 | load: 0.00306 [I 2022-10-11 00:05:58] ...trap/engines/engine.py.230: loss: 0.11229 [I 2022-10-11 00:05:58] ...optimizers/trijoint.py.151: total_norm: 25.474768 [I 2022-10-11 00:06:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1420/2304 [I 2022-10-11 00:06:07] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:40 [I 2022-10-11 00:06:07] ...trap/engines/engine.py.229: process: 0.85956 | load: 0.00311 [I 2022-10-11 00:06:07] ...trap/engines/engine.py.230: loss: 0.09478 [I 2022-10-11 00:06:07] ...optimizers/trijoint.py.151: total_norm: 8.501335 [I 2022-10-11 00:06:15] ...trap/engines/engine.py.224: train: epoch 90 | batch 1430/2304 [I 2022-10-11 00:06:15] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:34 [I 2022-10-11 00:06:15] ...trap/engines/engine.py.229: process: 0.86882 | load: 0.00317 [I 2022-10-11 00:06:15] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-11 00:06:15] ...optimizers/trijoint.py.151: total_norm: 21.091900 [I 2022-10-11 00:06:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 1440/2304 [I 2022-10-11 00:06:24] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:25 [I 2022-10-11 00:06:24] ...trap/engines/engine.py.229: process: 0.87203 | load: 0.00313 [I 2022-10-11 00:06:24] ...trap/engines/engine.py.230: loss: 0.08726 [I 2022-10-11 00:06:24] ...optimizers/trijoint.py.151: total_norm: 8.236513 [I 2022-10-11 00:06:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1450/2304 [I 2022-10-11 00:06:33] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:15 [I 2022-10-11 00:06:33] ...trap/engines/engine.py.229: process: 0.85405 | load: 0.00317 [I 2022-10-11 00:06:33] ...trap/engines/engine.py.230: loss: 0.17922 [I 2022-10-11 00:06:33] ...optimizers/trijoint.py.151: total_norm: 20.524305 [I 2022-10-11 00:06:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 1460/2304 [I 2022-10-11 00:06:41] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:09 [I 2022-10-11 00:06:41] ...trap/engines/engine.py.229: process: 0.87471 | load: 0.00309 [I 2022-10-11 00:06:41] ...trap/engines/engine.py.230: loss: 0.09477 [I 2022-10-11 00:06:41] ...optimizers/trijoint.py.151: total_norm: 12.210163 [I 2022-10-11 00:06:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 1470/2304 [I 2022-10-11 00:06:50] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:12:01 [I 2022-10-11 00:06:50] ...trap/engines/engine.py.229: process: 0.86792 | load: 0.00307 [I 2022-10-11 00:06:50] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-11 00:06:50] ...optimizers/trijoint.py.151: total_norm: 10.486125 [I 2022-10-11 00:06:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1480/2304 [I 2022-10-11 00:06:59] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:50 [I 2022-10-11 00:06:59] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00314 [I 2022-10-11 00:06:59] ...trap/engines/engine.py.230: loss: 0.20304 [I 2022-10-11 00:06:59] ...optimizers/trijoint.py.151: total_norm: 15.691201 [I 2022-10-11 00:07:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1490/2304 [I 2022-10-11 00:07:07] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:40 [I 2022-10-11 00:07:07] ...trap/engines/engine.py.229: process: 0.86540 | load: 0.00311 [I 2022-10-11 00:07:07] ...trap/engines/engine.py.230: loss: 0.08989 [I 2022-10-11 00:07:07] ...optimizers/trijoint.py.151: total_norm: 11.531960 [I 2022-10-11 00:07:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1500/2304 [I 2022-10-11 00:07:16] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:28 [I 2022-10-11 00:07:16] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00306 [I 2022-10-11 00:07:16] ...trap/engines/engine.py.230: loss: 0.12722 [I 2022-10-11 00:07:16] ...optimizers/trijoint.py.151: total_norm: 13.248393 [I 2022-10-11 00:07:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 1510/2304 [I 2022-10-11 00:07:24] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:26 [I 2022-10-11 00:07:24] ...trap/engines/engine.py.229: process: 0.86950 | load: 0.00314 [I 2022-10-11 00:07:24] ...trap/engines/engine.py.230: loss: 0.11517 [I 2022-10-11 00:07:24] ...optimizers/trijoint.py.151: total_norm: 28.942146 [I 2022-10-11 00:07:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1520/2304 [I 2022-10-11 00:07:33] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:20 [I 2022-10-11 00:07:33] ...trap/engines/engine.py.229: process: 0.86581 | load: 0.00310 [I 2022-10-11 00:07:33] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-11 00:07:33] ...optimizers/trijoint.py.151: total_norm: 13.895334 [I 2022-10-11 00:07:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1530/2304 [I 2022-10-11 00:07:42] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:11:05 [I 2022-10-11 00:07:42] ...trap/engines/engine.py.229: process: 0.84646 | load: 0.00330 [I 2022-10-11 00:07:42] ...trap/engines/engine.py.230: loss: 0.11809 [I 2022-10-11 00:07:42] ...optimizers/trijoint.py.151: total_norm: 18.018478 [I 2022-10-11 00:07:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 1540/2304 [I 2022-10-11 00:07:50] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:56 [I 2022-10-11 00:07:50] ...trap/engines/engine.py.229: process: 0.86475 | load: 0.00313 [I 2022-10-11 00:07:50] ...trap/engines/engine.py.230: loss: 0.17749 [I 2022-10-11 00:07:50] ...optimizers/trijoint.py.151: total_norm: 22.891529 [I 2022-10-11 00:07:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1550/2304 [I 2022-10-11 00:07:59] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:43 [I 2022-10-11 00:07:59] ...trap/engines/engine.py.229: process: 0.86035 | load: 0.00296 [I 2022-10-11 00:07:59] ...trap/engines/engine.py.230: loss: 0.09395 [I 2022-10-11 00:07:59] ...optimizers/trijoint.py.151: total_norm: 13.669120 [I 2022-10-11 00:08:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 1560/2304 [I 2022-10-11 00:08:08] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:35 [I 2022-10-11 00:08:08] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00303 [I 2022-10-11 00:08:08] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-11 00:08:08] ...optimizers/trijoint.py.151: total_norm: 19.275692 [I 2022-10-11 00:08:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1570/2304 [I 2022-10-11 00:08:16] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:26 [I 2022-10-11 00:08:16] ...trap/engines/engine.py.229: process: 0.86728 | load: 0.00300 [I 2022-10-11 00:08:16] ...trap/engines/engine.py.230: loss: 0.10277 [I 2022-10-11 00:08:16] ...optimizers/trijoint.py.151: total_norm: 12.458912 [I 2022-10-11 00:08:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 1580/2304 [I 2022-10-11 00:08:25] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:15 [I 2022-10-11 00:08:25] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00313 [I 2022-10-11 00:08:25] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-11 00:08:25] ...optimizers/trijoint.py.151: total_norm: 11.522888 [I 2022-10-11 00:08:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1590/2304 [I 2022-10-11 00:08:33] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:08 [I 2022-10-11 00:08:33] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00299 [I 2022-10-11 00:08:33] ...trap/engines/engine.py.230: loss: 0.08562 [I 2022-10-11 00:08:33] ...optimizers/trijoint.py.151: total_norm: 9.768580 [I 2022-10-11 00:08:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1600/2304 [I 2022-10-11 00:08:42] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:58 [I 2022-10-11 00:08:42] ...trap/engines/engine.py.229: process: 0.84823 | load: 0.00305 [I 2022-10-11 00:08:42] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-11 00:08:42] ...optimizers/trijoint.py.151: total_norm: 14.824261 [I 2022-10-11 00:08:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 1610/2304 [I 2022-10-11 00:08:50] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:54 [I 2022-10-11 00:08:50] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00300 [I 2022-10-11 00:08:50] ...trap/engines/engine.py.230: loss: 0.22054 [I 2022-10-11 00:08:50] ...optimizers/trijoint.py.151: total_norm: 14.526575 [I 2022-10-11 00:08:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1620/2304 [I 2022-10-11 00:08:59] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:43 [I 2022-10-11 00:08:59] ...trap/engines/engine.py.229: process: 0.85248 | load: 0.00308 [I 2022-10-11 00:08:59] ...trap/engines/engine.py.230: loss: 0.08712 [I 2022-10-11 00:08:59] ...optimizers/trijoint.py.151: total_norm: 9.731520 [I 2022-10-11 00:09:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1630/2304 [I 2022-10-11 00:09:07] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:32 [I 2022-10-11 00:09:07] ...trap/engines/engine.py.229: process: 0.83308 | load: 0.00296 [I 2022-10-11 00:09:07] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-11 00:09:07] ...optimizers/trijoint.py.151: total_norm: 14.239158 [I 2022-10-11 00:09:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1640/2304 [I 2022-10-11 00:09:16] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:22 [I 2022-10-11 00:09:16] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00307 [I 2022-10-11 00:09:16] ...trap/engines/engine.py.230: loss: 0.28116 [I 2022-10-11 00:09:16] ...optimizers/trijoint.py.151: total_norm: 28.262617 [I 2022-10-11 00:09:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 1650/2304 [I 2022-10-11 00:09:24] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:17 [I 2022-10-11 00:09:24] ...trap/engines/engine.py.229: process: 0.85670 | load: 0.00300 [I 2022-10-11 00:09:24] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-11 00:09:24] ...optimizers/trijoint.py.151: total_norm: 13.107329 [I 2022-10-11 00:09:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1660/2304 [I 2022-10-11 00:09:33] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:07 [I 2022-10-11 00:09:33] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00304 [I 2022-10-11 00:09:33] ...trap/engines/engine.py.230: loss: 0.11982 [I 2022-10-11 00:09:33] ...optimizers/trijoint.py.151: total_norm: 17.993294 [I 2022-10-11 00:09:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1670/2304 [I 2022-10-11 00:09:42] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:09:06 [I 2022-10-11 00:09:42] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00309 [I 2022-10-11 00:09:42] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-11 00:09:42] ...optimizers/trijoint.py.151: total_norm: 23.749769 [I 2022-10-11 00:09:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 1680/2304 [I 2022-10-11 00:09:50] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:55 [I 2022-10-11 00:09:50] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00321 [I 2022-10-11 00:09:50] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-11 00:09:50] ...optimizers/trijoint.py.151: total_norm: 17.629829 [I 2022-10-11 00:09:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1690/2304 [I 2022-10-11 00:09:59] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:49 [I 2022-10-11 00:09:59] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00353 [I 2022-10-11 00:09:59] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-11 00:09:59] ...optimizers/trijoint.py.151: total_norm: 10.621424 [I 2022-10-11 00:10:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1700/2304 [I 2022-10-11 00:10:07] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:37 [I 2022-10-11 00:10:07] ...trap/engines/engine.py.229: process: 0.86559 | load: 0.00328 [I 2022-10-11 00:10:07] ...trap/engines/engine.py.230: loss: 0.08322 [I 2022-10-11 00:10:07] ...optimizers/trijoint.py.151: total_norm: 9.383519 [I 2022-10-11 00:10:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1710/2304 [I 2022-10-11 00:10:16] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:34 [I 2022-10-11 00:10:16] ...trap/engines/engine.py.229: process: 0.88301 | load: 0.00306 [I 2022-10-11 00:10:16] ...trap/engines/engine.py.230: loss: 0.09205 [I 2022-10-11 00:10:16] ...optimizers/trijoint.py.151: total_norm: 16.952648 [I 2022-10-11 00:10:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 1720/2304 [I 2022-10-11 00:10:25] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:25 [I 2022-10-11 00:10:25] ...trap/engines/engine.py.229: process: 0.85647 | load: 0.00306 [I 2022-10-11 00:10:25] ...trap/engines/engine.py.230: loss: 0.09638 [I 2022-10-11 00:10:25] ...optimizers/trijoint.py.151: total_norm: 16.843552 [I 2022-10-11 00:10:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1730/2304 [I 2022-10-11 00:10:33] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:16 [I 2022-10-11 00:10:33] ...trap/engines/engine.py.229: process: 0.87622 | load: 0.00311 [I 2022-10-11 00:10:33] ...trap/engines/engine.py.230: loss: 0.19470 [I 2022-10-11 00:10:33] ...optimizers/trijoint.py.151: total_norm: 23.256187 [I 2022-10-11 00:10:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1740/2304 [I 2022-10-11 00:10:42] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:08:00 [I 2022-10-11 00:10:42] ...trap/engines/engine.py.229: process: 0.83957 | load: 0.00310 [I 2022-10-11 00:10:42] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-11 00:10:42] ...optimizers/trijoint.py.151: total_norm: 9.375737 [I 2022-10-11 00:10:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 1750/2304 [I 2022-10-11 00:10:51] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:55 [I 2022-10-11 00:10:51] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00311 [I 2022-10-11 00:10:51] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-11 00:10:51] ...optimizers/trijoint.py.151: total_norm: 12.241572 [I 2022-10-11 00:10:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1760/2304 [I 2022-10-11 00:10:59] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:07:47 [I 2022-10-11 00:10:59] ...trap/engines/engine.py.229: process: 0.86607 | load: 0.00306 [I 2022-10-11 00:10:59] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-11 00:10:59] ...optimizers/trijoint.py.151: total_norm: 33.061378 [I 2022-10-11 00:11:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 1770/2304 [I 2022-10-11 00:11:08] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:37 [I 2022-10-11 00:11:08] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00327 [I 2022-10-11 00:11:08] ...trap/engines/engine.py.230: loss: 0.18789 [I 2022-10-11 00:11:08] ...optimizers/trijoint.py.151: total_norm: 15.547350 [I 2022-10-11 00:11:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1780/2304 [I 2022-10-11 00:11:16] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:30 [I 2022-10-11 00:11:16] ...trap/engines/engine.py.229: process: 0.84710 | load: 0.00320 [I 2022-10-11 00:11:16] ...trap/engines/engine.py.230: loss: 0.10468 [I 2022-10-11 00:11:16] ...optimizers/trijoint.py.151: total_norm: 10.396932 [I 2022-10-11 00:11:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 1790/2304 [I 2022-10-11 00:11:25] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:21 [I 2022-10-11 00:11:25] ...trap/engines/engine.py.229: process: 0.87898 | load: 0.00306 [I 2022-10-11 00:11:25] ...trap/engines/engine.py.230: loss: 0.09788 [I 2022-10-11 00:11:25] ...optimizers/trijoint.py.151: total_norm: 8.470934 [I 2022-10-11 00:11:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 1800/2304 [I 2022-10-11 00:11:34] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:07:14 [I 2022-10-11 00:11:34] ...trap/engines/engine.py.229: process: 0.86817 | load: 0.00311 [I 2022-10-11 00:11:34] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-11 00:11:34] ...optimizers/trijoint.py.151: total_norm: 9.063991 [I 2022-10-11 00:11:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1810/2304 [I 2022-10-11 00:11:42] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:07:06 [I 2022-10-11 00:11:42] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00314 [I 2022-10-11 00:11:42] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-11 00:11:42] ...optimizers/trijoint.py.151: total_norm: 17.949455 [I 2022-10-11 00:11:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 1820/2304 [I 2022-10-11 00:11:51] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:56 [I 2022-10-11 00:11:51] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00314 [I 2022-10-11 00:11:51] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-11 00:11:51] ...optimizers/trijoint.py.151: total_norm: 25.438368 [I 2022-10-11 00:12:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 1830/2304 [I 2022-10-11 00:12:00] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:47 [I 2022-10-11 00:12:00] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00329 [I 2022-10-11 00:12:00] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-11 00:12:00] ...optimizers/trijoint.py.151: total_norm: 15.197489 [I 2022-10-11 00:12:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 1840/2304 [I 2022-10-11 00:12:08] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:37 [I 2022-10-11 00:12:08] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00318 [I 2022-10-11 00:12:08] ...trap/engines/engine.py.230: loss: 0.10424 [I 2022-10-11 00:12:08] ...optimizers/trijoint.py.151: total_norm: 13.380507 [I 2022-10-11 00:12:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 1850/2304 [I 2022-10-11 00:12:17] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:28 [I 2022-10-11 00:12:17] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00313 [I 2022-10-11 00:12:17] ...trap/engines/engine.py.230: loss: 0.13860 [I 2022-10-11 00:12:17] ...optimizers/trijoint.py.151: total_norm: 12.882221 [I 2022-10-11 00:12:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 1860/2304 [I 2022-10-11 00:12:25] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:21 [I 2022-10-11 00:12:25] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00306 [I 2022-10-11 00:12:25] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-11 00:12:25] ...optimizers/trijoint.py.151: total_norm: 14.139033 [I 2022-10-11 00:12:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 1870/2304 [I 2022-10-11 00:12:34] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:13 [I 2022-10-11 00:12:34] ...trap/engines/engine.py.229: process: 0.86292 | load: 0.00325 [I 2022-10-11 00:12:34] ...trap/engines/engine.py.230: loss: 0.08912 [I 2022-10-11 00:12:34] ...optimizers/trijoint.py.151: total_norm: 14.399611 [I 2022-10-11 00:12:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 1880/2304 [I 2022-10-11 00:12:43] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:06 [I 2022-10-11 00:12:43] ...trap/engines/engine.py.229: process: 0.86170 | load: 0.00308 [I 2022-10-11 00:12:43] ...trap/engines/engine.py.230: loss: 0.14792 [I 2022-10-11 00:12:43] ...optimizers/trijoint.py.151: total_norm: 20.079527 [I 2022-10-11 00:12:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 1890/2304 [I 2022-10-11 00:12:51] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:55 [I 2022-10-11 00:12:51] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00312 [I 2022-10-11 00:12:51] ...trap/engines/engine.py.230: loss: 0.12417 [I 2022-10-11 00:12:51] ...optimizers/trijoint.py.151: total_norm: 23.688654 [I 2022-10-11 00:13:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 1900/2304 [I 2022-10-11 00:13:00] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:47 [I 2022-10-11 00:13:00] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00313 [I 2022-10-11 00:13:00] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-11 00:13:00] ...optimizers/trijoint.py.151: total_norm: 10.547538 [I 2022-10-11 00:13:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 1910/2304 [I 2022-10-11 00:13:09] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:37 [I 2022-10-11 00:13:09] ...trap/engines/engine.py.229: process: 0.83102 | load: 0.00317 [I 2022-10-11 00:13:09] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-11 00:13:09] ...optimizers/trijoint.py.151: total_norm: 10.838840 [I 2022-10-11 00:13:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 1920/2304 [I 2022-10-11 00:13:17] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:32 [I 2022-10-11 00:13:17] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00309 [I 2022-10-11 00:13:17] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-11 00:13:17] ...optimizers/trijoint.py.151: total_norm: 22.138069 [I 2022-10-11 00:13:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 1930/2304 [I 2022-10-11 00:13:26] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:05:21 [I 2022-10-11 00:13:26] ...trap/engines/engine.py.229: process: 0.86198 | load: 0.00317 [I 2022-10-11 00:13:26] ...trap/engines/engine.py.230: loss: 0.11435 [I 2022-10-11 00:13:26] ...optimizers/trijoint.py.151: total_norm: 21.990429 [I 2022-10-11 00:13:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 1940/2304 [I 2022-10-11 00:13:35] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:13 [I 2022-10-11 00:13:35] ...trap/engines/engine.py.229: process: 0.85590 | load: 0.00304 [I 2022-10-11 00:13:35] ...trap/engines/engine.py.230: loss: 0.17785 [I 2022-10-11 00:13:35] ...optimizers/trijoint.py.151: total_norm: 27.505579 [I 2022-10-11 00:13:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 1950/2304 [I 2022-10-11 00:13:43] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:05:01 [I 2022-10-11 00:13:43] ...trap/engines/engine.py.229: process: 0.85877 | load: 0.00299 [I 2022-10-11 00:13:43] ...trap/engines/engine.py.230: loss: 0.12717 [I 2022-10-11 00:13:43] ...optimizers/trijoint.py.151: total_norm: 18.769110 [I 2022-10-11 00:13:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 1960/2304 [I 2022-10-11 00:13:52] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:55 [I 2022-10-11 00:13:52] ...trap/engines/engine.py.229: process: 0.86463 | load: 0.00296 [I 2022-10-11 00:13:52] ...trap/engines/engine.py.230: loss: 0.18492 [I 2022-10-11 00:13:52] ...optimizers/trijoint.py.151: total_norm: 26.808228 [I 2022-10-11 00:14:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 1970/2304 [I 2022-10-11 00:14:00] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:42 [I 2022-10-11 00:14:00] ...trap/engines/engine.py.229: process: 0.83460 | load: 0.00294 [I 2022-10-11 00:14:00] ...trap/engines/engine.py.230: loss: 0.23027 [I 2022-10-11 00:14:00] ...optimizers/trijoint.py.151: total_norm: 14.939508 [I 2022-10-11 00:14:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 1980/2304 [I 2022-10-11 00:14:09] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:34 [I 2022-10-11 00:14:09] ...trap/engines/engine.py.229: process: 0.82681 | load: 0.00303 [I 2022-10-11 00:14:09] ...trap/engines/engine.py.230: loss: 0.10088 [I 2022-10-11 00:14:09] ...optimizers/trijoint.py.151: total_norm: 10.945108 [I 2022-10-11 00:14:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 1990/2304 [I 2022-10-11 00:14:17] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:26 [I 2022-10-11 00:14:17] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00309 [I 2022-10-11 00:14:17] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-11 00:14:17] ...optimizers/trijoint.py.151: total_norm: 13.401259 [I 2022-10-11 00:14:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 2000/2304 [I 2022-10-11 00:14:26] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:04:18 [I 2022-10-11 00:14:26] ...trap/engines/engine.py.229: process: 0.85732 | load: 0.00294 [I 2022-10-11 00:14:26] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-11 00:14:26] ...optimizers/trijoint.py.151: total_norm: 34.099483 [I 2022-10-11 00:14:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 2010/2304 [I 2022-10-11 00:14:34] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:10 [I 2022-10-11 00:14:34] ...trap/engines/engine.py.229: process: 0.84452 | load: 0.00305 [I 2022-10-11 00:14:34] ...trap/engines/engine.py.230: loss: 0.14222 [I 2022-10-11 00:14:34] ...optimizers/trijoint.py.151: total_norm: 30.354883 [I 2022-10-11 00:14:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 2020/2304 [I 2022-10-11 00:14:43] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:04:01 [I 2022-10-11 00:14:43] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00309 [I 2022-10-11 00:14:43] ...trap/engines/engine.py.230: loss: 0.11941 [I 2022-10-11 00:14:43] ...optimizers/trijoint.py.151: total_norm: 14.437571 [I 2022-10-11 00:14:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 2030/2304 [I 2022-10-11 00:14:51] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:53 [I 2022-10-11 00:14:51] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00301 [I 2022-10-11 00:14:51] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-11 00:14:51] ...optimizers/trijoint.py.151: total_norm: 16.852650 [I 2022-10-11 00:15:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 2040/2304 [I 2022-10-11 00:15:00] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:45 [I 2022-10-11 00:15:00] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00298 [I 2022-10-11 00:15:00] ...trap/engines/engine.py.230: loss: 0.14653 [I 2022-10-11 00:15:00] ...optimizers/trijoint.py.151: total_norm: 16.182835 [I 2022-10-11 00:15:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 2050/2304 [I 2022-10-11 00:15:09] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:37 [I 2022-10-11 00:15:09] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00301 [I 2022-10-11 00:15:09] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-11 00:15:09] ...optimizers/trijoint.py.151: total_norm: 14.443683 [I 2022-10-11 00:15:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 2060/2304 [I 2022-10-11 00:15:17] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:26 [I 2022-10-11 00:15:17] ...trap/engines/engine.py.229: process: 0.84526 | load: 0.00296 [I 2022-10-11 00:15:17] ...trap/engines/engine.py.230: loss: 0.11828 [I 2022-10-11 00:15:17] ...optimizers/trijoint.py.151: total_norm: 13.733623 [I 2022-10-11 00:15:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 2070/2304 [I 2022-10-11 00:15:26] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:20 [I 2022-10-11 00:15:26] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00294 [I 2022-10-11 00:15:26] ...trap/engines/engine.py.230: loss: 0.18316 [I 2022-10-11 00:15:26] ...optimizers/trijoint.py.151: total_norm: 67.506027 [I 2022-10-11 00:15:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 2080/2304 [I 2022-10-11 00:15:34] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:03:10 [I 2022-10-11 00:15:34] ...trap/engines/engine.py.229: process: 0.83445 | load: 0.00303 [I 2022-10-11 00:15:34] ...trap/engines/engine.py.230: loss: 0.12177 [I 2022-10-11 00:15:34] ...optimizers/trijoint.py.151: total_norm: 12.434088 [I 2022-10-11 00:15:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 2090/2304 [I 2022-10-11 00:15:43] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:03:03 [I 2022-10-11 00:15:43] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00313 [I 2022-10-11 00:15:43] ...trap/engines/engine.py.230: loss: 0.16019 [I 2022-10-11 00:15:43] ...optimizers/trijoint.py.151: total_norm: 16.013081 [I 2022-10-11 00:15:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 2100/2304 [I 2022-10-11 00:15:51] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:55 [I 2022-10-11 00:15:51] ...trap/engines/engine.py.229: process: 0.86001 | load: 0.00330 [I 2022-10-11 00:15:51] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-11 00:15:51] ...optimizers/trijoint.py.151: total_norm: 10.732296 [I 2022-10-11 00:16:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 2110/2304 [I 2022-10-11 00:16:00] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:47 [I 2022-10-11 00:16:00] ...trap/engines/engine.py.229: process: 0.86557 | load: 0.00306 [I 2022-10-11 00:16:00] ...trap/engines/engine.py.230: loss: 0.10558 [I 2022-10-11 00:16:00] ...optimizers/trijoint.py.151: total_norm: 11.274681 [I 2022-10-11 00:16:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 2120/2304 [I 2022-10-11 00:16:09] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:38 [I 2022-10-11 00:16:09] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00305 [I 2022-10-11 00:16:09] ...trap/engines/engine.py.230: loss: 0.16545 [I 2022-10-11 00:16:09] ...optimizers/trijoint.py.151: total_norm: 15.586593 [I 2022-10-11 00:16:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 2130/2304 [I 2022-10-11 00:16:17] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:31 [I 2022-10-11 00:16:17] ...trap/engines/engine.py.229: process: 0.88715 | load: 0.00305 [I 2022-10-11 00:16:17] ...trap/engines/engine.py.230: loss: 0.12488 [I 2022-10-11 00:16:17] ...optimizers/trijoint.py.151: total_norm: 11.809957 [I 2022-10-11 00:16:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 2140/2304 [I 2022-10-11 00:16:26] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:21 [I 2022-10-11 00:16:26] ...trap/engines/engine.py.229: process: 0.86439 | load: 0.00336 [I 2022-10-11 00:16:26] ...trap/engines/engine.py.230: loss: 0.13340 [I 2022-10-11 00:16:26] ...optimizers/trijoint.py.151: total_norm: 26.254387 [I 2022-10-11 00:16:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 2150/2304 [I 2022-10-11 00:16:35] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:13 [I 2022-10-11 00:16:35] ...trap/engines/engine.py.229: process: 0.87205 | load: 0.00312 [I 2022-10-11 00:16:35] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-11 00:16:35] ...optimizers/trijoint.py.151: total_norm: 40.489616 [I 2022-10-11 00:16:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 2160/2304 [I 2022-10-11 00:16:43] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:03 [I 2022-10-11 00:16:43] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00378 [I 2022-10-11 00:16:43] ...trap/engines/engine.py.230: loss: 0.10820 [I 2022-10-11 00:16:43] ...optimizers/trijoint.py.151: total_norm: 18.139221 [I 2022-10-11 00:16:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 2170/2304 [I 2022-10-11 00:16:52] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:55 [I 2022-10-11 00:16:52] ...trap/engines/engine.py.229: process: 0.87003 | load: 0.00312 [I 2022-10-11 00:16:52] ...trap/engines/engine.py.230: loss: 0.12632 [I 2022-10-11 00:16:52] ...optimizers/trijoint.py.151: total_norm: 14.940994 [I 2022-10-11 00:17:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 2180/2304 [I 2022-10-11 00:17:01] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:46 [I 2022-10-11 00:17:01] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00308 [I 2022-10-11 00:17:01] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-11 00:17:01] ...optimizers/trijoint.py.151: total_norm: 14.539648 [I 2022-10-11 00:17:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 2190/2304 [I 2022-10-11 00:17:09] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:37 [I 2022-10-11 00:17:09] ...trap/engines/engine.py.229: process: 0.86674 | load: 0.00308 [I 2022-10-11 00:17:09] ...trap/engines/engine.py.230: loss: 0.10626 [I 2022-10-11 00:17:09] ...optimizers/trijoint.py.151: total_norm: 12.639957 [I 2022-10-11 00:17:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 2200/2304 [I 2022-10-11 00:17:18] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:29 [I 2022-10-11 00:17:18] ...trap/engines/engine.py.229: process: 0.86946 | load: 0.00311 [I 2022-10-11 00:17:18] ...trap/engines/engine.py.230: loss: 0.12106 [I 2022-10-11 00:17:18] ...optimizers/trijoint.py.151: total_norm: 13.954772 [I 2022-10-11 00:17:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 2210/2304 [I 2022-10-11 00:17:26] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:20 [I 2022-10-11 00:17:26] ...trap/engines/engine.py.229: process: 0.86714 | load: 0.00324 [I 2022-10-11 00:17:26] ...trap/engines/engine.py.230: loss: 0.09864 [I 2022-10-11 00:17:26] ...optimizers/trijoint.py.151: total_norm: 8.222956 [I 2022-10-11 00:17:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 2220/2304 [I 2022-10-11 00:17:35] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:11 [I 2022-10-11 00:17:35] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00309 [I 2022-10-11 00:17:35] ...trap/engines/engine.py.230: loss: 0.07696 [I 2022-10-11 00:17:35] ...optimizers/trijoint.py.151: total_norm: 9.586083 [I 2022-10-11 00:17:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 2230/2304 [I 2022-10-11 00:17:43] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:03 [I 2022-10-11 00:17:43] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00314 [I 2022-10-11 00:17:43] ...trap/engines/engine.py.230: loss: 0.08871 [I 2022-10-11 00:17:43] ...optimizers/trijoint.py.151: total_norm: 12.812272 [I 2022-10-11 00:17:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 2240/2304 [I 2022-10-11 00:17:52] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:54 [I 2022-10-11 00:17:52] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00320 [I 2022-10-11 00:17:52] ...trap/engines/engine.py.230: loss: 0.08754 [I 2022-10-11 00:17:52] ...optimizers/trijoint.py.151: total_norm: 10.561004 [I 2022-10-11 00:18:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 2250/2304 [I 2022-10-11 00:18:01] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:46 [I 2022-10-11 00:18:01] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00307 [I 2022-10-11 00:18:01] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-11 00:18:01] ...optimizers/trijoint.py.151: total_norm: 27.332973 [I 2022-10-11 00:18:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 2260/2304 [I 2022-10-11 00:18:09] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:37 [I 2022-10-11 00:18:09] ...trap/engines/engine.py.229: process: 0.86927 | load: 0.00319 [I 2022-10-11 00:18:09] ...trap/engines/engine.py.230: loss: 0.17821 [I 2022-10-11 00:18:09] ...optimizers/trijoint.py.151: total_norm: 28.282410 [I 2022-10-11 00:18:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 2270/2304 [I 2022-10-11 00:18:18] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:29 [I 2022-10-11 00:18:18] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00313 [I 2022-10-11 00:18:18] ...trap/engines/engine.py.230: loss: 0.11943 [I 2022-10-11 00:18:18] ...optimizers/trijoint.py.151: total_norm: 17.768526 [I 2022-10-11 00:18:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 2280/2304 [I 2022-10-11 00:18:27] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:20 [I 2022-10-11 00:18:27] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00321 [I 2022-10-11 00:18:27] ...trap/engines/engine.py.230: loss: 0.09927 [I 2022-10-11 00:18:27] ...optimizers/trijoint.py.151: total_norm: 21.529009 [I 2022-10-11 00:18:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 2290/2304 [I 2022-10-11 00:18:35] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:12 [I 2022-10-11 00:18:35] ...trap/engines/engine.py.229: process: 0.86187 | load: 0.00315 [I 2022-10-11 00:18:35] ...trap/engines/engine.py.230: loss: 0.11835 [I 2022-10-11 00:18:35] ...optimizers/trijoint.py.151: total_norm: 21.541676 [I 2022-10-11 00:18:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 2300/2304 [I 2022-10-11 00:18:44] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:03 [I 2022-10-11 00:18:44] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00019 [I 2022-10-11 00:18:44] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-11 00:18:44] ...optimizers/trijoint.py.151: total_norm: 22.612051 [I 2022-10-11 00:18:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 2304/2304 [I 2022-10-11 00:18:47] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:00 [I 2022-10-11 00:18:47] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00018 [I 2022-10-11 00:18:47] ...trap/engines/engine.py.230: loss: 0.09738 [I 2022-10-11 00:18:47] ...optimizers/trijoint.py.151: total_norm: 19.716797 [S 2022-10-11 00:18:49] ...trap/engines/engine.py.129: train_epoch.epoch: 90 [S 2022-10-11 00:18:49] ...trap/engines/engine.py.129: train_epoch.loss: 0.1273128531081299 [I 2022-10-11 00:18:49] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 00:18:50] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 90 [I 2022-10-11 00:18:50] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 00:18:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 00:18:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 00:18:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 00:18:56] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 00:18:56] ...trap/engines/engine.py.321: eval: epoch 90 | batch 0/510 [I 2022-10-11 00:18:56] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:50:43 [I 2022-10-11 00:18:56] ...trap/engines/engine.py.326: process: 5.96810 | load: 5.36474 [I 2022-10-11 00:19:01] ...trap/engines/engine.py.321: eval: epoch 90 | batch 10/510 [I 2022-10-11 00:19:01] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:09:06 [I 2022-10-11 00:19:01] ...trap/engines/engine.py.326: process: 0.60476 | load: 0.00019 [I 2022-10-11 00:19:08] ...trap/engines/engine.py.321: eval: epoch 90 | batch 20/510 [I 2022-10-11 00:19:08] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:38 [I 2022-10-11 00:19:08] ...trap/engines/engine.py.326: process: 0.54813 | load: 0.00020 [I 2022-10-11 00:19:14] ...trap/engines/engine.py.321: eval: epoch 90 | batch 30/510 [I 2022-10-11 00:19:14] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:38 [I 2022-10-11 00:19:14] ...trap/engines/engine.py.326: process: 0.45052 | load: 0.00020 [I 2022-10-11 00:19:28] ...trap/engines/engine.py.321: eval: epoch 90 | batch 40/510 [I 2022-10-11 00:19:28] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:11:09 [I 2022-10-11 00:19:28] ...trap/engines/engine.py.326: process: 0.69078 | load: 0.00020 [I 2022-10-11 00:19:35] ...trap/engines/engine.py.321: eval: epoch 90 | batch 50/510 [I 2022-10-11 00:19:35] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:05:27 [I 2022-10-11 00:19:35] ...trap/engines/engine.py.326: process: 0.55339 | load: 0.00020 [I 2022-10-11 00:19:41] ...trap/engines/engine.py.321: eval: epoch 90 | batch 60/510 [I 2022-10-11 00:19:41] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:04:53 [I 2022-10-11 00:19:41] ...trap/engines/engine.py.326: process: 0.63643 | load: 0.00019 [I 2022-10-11 00:19:46] ...trap/engines/engine.py.321: eval: epoch 90 | batch 70/510 [I 2022-10-11 00:19:46] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:04:17 [I 2022-10-11 00:19:46] ...trap/engines/engine.py.326: process: 0.87875 | load: 0.00019 [I 2022-10-11 00:19:51] ...trap/engines/engine.py.321: eval: epoch 90 | batch 80/510 [I 2022-10-11 00:19:51] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:03:29 [I 2022-10-11 00:19:51] ...trap/engines/engine.py.326: process: 0.58078 | load: 0.00020 [I 2022-10-11 00:19:55] ...trap/engines/engine.py.321: eval: epoch 90 | batch 90/510 [I 2022-10-11 00:19:55] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:03:17 [I 2022-10-11 00:19:55] ...trap/engines/engine.py.326: process: 0.49739 | load: 0.00018 [I 2022-10-11 00:20:01] ...trap/engines/engine.py.321: eval: epoch 90 | batch 100/510 [I 2022-10-11 00:20:01] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:03:34 [I 2022-10-11 00:20:01] ...trap/engines/engine.py.326: process: 0.30882 | load: 0.00018 [I 2022-10-11 00:20:04] ...trap/engines/engine.py.321: eval: epoch 90 | batch 110/510 [I 2022-10-11 00:20:04] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:02:15 [I 2022-10-11 00:20:04] ...trap/engines/engine.py.326: process: 0.31852 | load: 0.00019 [I 2022-10-11 00:20:07] ...trap/engines/engine.py.321: eval: epoch 90 | batch 120/510 [I 2022-10-11 00:20:07] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:02:04 [I 2022-10-11 00:20:07] ...trap/engines/engine.py.326: process: 0.31761 | load: 0.00019 [I 2022-10-11 00:20:11] ...trap/engines/engine.py.321: eval: epoch 90 | batch 130/510 [I 2022-10-11 00:20:11] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:02:01 [I 2022-10-11 00:20:11] ...trap/engines/engine.py.326: process: 0.31838 | load: 0.00020 [I 2022-10-11 00:20:14] ...trap/engines/engine.py.321: eval: epoch 90 | batch 140/510 [I 2022-10-11 00:20:14] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:58 [I 2022-10-11 00:20:14] ...trap/engines/engine.py.326: process: 0.32360 | load: 0.00019 [I 2022-10-11 00:20:17] ...trap/engines/engine.py.321: eval: epoch 90 | batch 150/510 [I 2022-10-11 00:20:17] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:55 [I 2022-10-11 00:20:17] ...trap/engines/engine.py.326: process: 0.32770 | load: 0.00018 [I 2022-10-11 00:20:20] ...trap/engines/engine.py.321: eval: epoch 90 | batch 160/510 [I 2022-10-11 00:20:20] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:52 [I 2022-10-11 00:20:20] ...trap/engines/engine.py.326: process: 0.31710 | load: 0.00020 [I 2022-10-11 00:20:23] ...trap/engines/engine.py.321: eval: epoch 90 | batch 170/510 [I 2022-10-11 00:20:23] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:48 [I 2022-10-11 00:20:23] ...trap/engines/engine.py.326: process: 0.32229 | load: 0.00020 [I 2022-10-11 00:20:27] ...trap/engines/engine.py.321: eval: epoch 90 | batch 180/510 [I 2022-10-11 00:20:27] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:46 [I 2022-10-11 00:20:27] ...trap/engines/engine.py.326: process: 0.32405 | load: 0.00019 [I 2022-10-11 00:20:30] ...trap/engines/engine.py.321: eval: epoch 90 | batch 190/510 [I 2022-10-11 00:20:30] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:42 [I 2022-10-11 00:20:30] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00019 [I 2022-10-11 00:20:33] ...trap/engines/engine.py.321: eval: epoch 90 | batch 200/510 [I 2022-10-11 00:20:33] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:39 [I 2022-10-11 00:20:33] ...trap/engines/engine.py.326: process: 0.32298 | load: 0.00020 [I 2022-10-11 00:20:36] ...trap/engines/engine.py.321: eval: epoch 90 | batch 210/510 [I 2022-10-11 00:20:36] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:36 [I 2022-10-11 00:20:36] ...trap/engines/engine.py.326: process: 0.32707 | load: 0.00020 [I 2022-10-11 00:20:39] ...trap/engines/engine.py.321: eval: epoch 90 | batch 220/510 [I 2022-10-11 00:20:39] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:32 [I 2022-10-11 00:20:39] ...trap/engines/engine.py.326: process: 0.32433 | load: 0.00019 [I 2022-10-11 00:20:43] ...trap/engines/engine.py.321: eval: epoch 90 | batch 230/510 [I 2022-10-11 00:20:43] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:29 [I 2022-10-11 00:20:43] ...trap/engines/engine.py.326: process: 0.32475 | load: 0.00019 [I 2022-10-11 00:20:46] ...trap/engines/engine.py.321: eval: epoch 90 | batch 240/510 [I 2022-10-11 00:20:46] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:26 [I 2022-10-11 00:20:46] ...trap/engines/engine.py.326: process: 0.32231 | load: 0.00020 [I 2022-10-11 00:20:49] ...trap/engines/engine.py.321: eval: epoch 90 | batch 250/510 [I 2022-10-11 00:20:49] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:23 [I 2022-10-11 00:20:49] ...trap/engines/engine.py.326: process: 0.32457 | load: 0.00019 [I 2022-10-11 00:20:52] ...trap/engines/engine.py.321: eval: epoch 90 | batch 260/510 [I 2022-10-11 00:20:52] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:19 [I 2022-10-11 00:20:52] ...trap/engines/engine.py.326: process: 0.31705 | load: 0.00019 [I 2022-10-11 00:20:55] ...trap/engines/engine.py.321: eval: epoch 90 | batch 270/510 [I 2022-10-11 00:20:55] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:16 [I 2022-10-11 00:20:55] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00019 [I 2022-10-11 00:20:59] ...trap/engines/engine.py.321: eval: epoch 90 | batch 280/510 [I 2022-10-11 00:20:59] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:13 [I 2022-10-11 00:20:59] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00020 [I 2022-10-11 00:21:02] ...trap/engines/engine.py.321: eval: epoch 90 | batch 290/510 [I 2022-10-11 00:21:02] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:10 [I 2022-10-11 00:21:02] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00019 [I 2022-10-11 00:21:05] ...trap/engines/engine.py.321: eval: epoch 90 | batch 300/510 [I 2022-10-11 00:21:05] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:06 [I 2022-10-11 00:21:05] ...trap/engines/engine.py.326: process: 0.32049 | load: 0.00019 [I 2022-10-11 00:21:08] ...trap/engines/engine.py.321: eval: epoch 90 | batch 310/510 [I 2022-10-11 00:21:08] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:01:04 [I 2022-10-11 00:21:08] ...trap/engines/engine.py.326: process: 0.32238 | load: 0.00019 [I 2022-10-11 00:21:11] ...trap/engines/engine.py.321: eval: epoch 90 | batch 320/510 [I 2022-10-11 00:21:11] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:01:00 [I 2022-10-11 00:21:11] ...trap/engines/engine.py.326: process: 0.32837 | load: 0.00020 [I 2022-10-11 00:21:15] ...trap/engines/engine.py.321: eval: epoch 90 | batch 330/510 [I 2022-10-11 00:21:15] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:57 [I 2022-10-11 00:21:15] ...trap/engines/engine.py.326: process: 0.31648 | load: 0.00020 [I 2022-10-11 00:21:18] ...trap/engines/engine.py.321: eval: epoch 90 | batch 340/510 [I 2022-10-11 00:21:18] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:54 [I 2022-10-11 00:21:18] ...trap/engines/engine.py.326: process: 0.32574 | load: 0.00019 [I 2022-10-11 00:21:21] ...trap/engines/engine.py.321: eval: epoch 90 | batch 350/510 [I 2022-10-11 00:21:21] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:51 [I 2022-10-11 00:21:21] ...trap/engines/engine.py.326: process: 0.32452 | load: 0.00021 [I 2022-10-11 00:21:24] ...trap/engines/engine.py.321: eval: epoch 90 | batch 360/510 [I 2022-10-11 00:21:24] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:48 [I 2022-10-11 00:21:24] ...trap/engines/engine.py.326: process: 0.31697 | load: 0.00021 [I 2022-10-11 00:21:27] ...trap/engines/engine.py.321: eval: epoch 90 | batch 370/510 [I 2022-10-11 00:21:27] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:45 [I 2022-10-11 00:21:27] ...trap/engines/engine.py.326: process: 0.32357 | load: 0.00021 [I 2022-10-11 00:21:31] ...trap/engines/engine.py.321: eval: epoch 90 | batch 380/510 [I 2022-10-11 00:21:31] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:42 [I 2022-10-11 00:21:31] ...trap/engines/engine.py.326: process: 0.32453 | load: 0.00022 [I 2022-10-11 00:21:34] ...trap/engines/engine.py.321: eval: epoch 90 | batch 390/510 [I 2022-10-11 00:21:34] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:38 [I 2022-10-11 00:21:34] ...trap/engines/engine.py.326: process: 0.32582 | load: 0.00022 [I 2022-10-11 00:21:37] ...trap/engines/engine.py.321: eval: epoch 90 | batch 400/510 [I 2022-10-11 00:21:37] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:35 [I 2022-10-11 00:21:37] ...trap/engines/engine.py.326: process: 0.32545 | load: 0.00060 [I 2022-10-11 00:21:40] ...trap/engines/engine.py.321: eval: epoch 90 | batch 410/510 [I 2022-10-11 00:21:40] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:32 [I 2022-10-11 00:21:40] ...trap/engines/engine.py.326: process: 0.32219 | load: 0.00021 [I 2022-10-11 00:21:44] ...trap/engines/engine.py.321: eval: epoch 90 | batch 420/510 [I 2022-10-11 00:21:44] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:29 [I 2022-10-11 00:21:44] ...trap/engines/engine.py.326: process: 0.32767 | load: 0.00021 [I 2022-10-11 00:21:47] ...trap/engines/engine.py.321: eval: epoch 90 | batch 430/510 [I 2022-10-11 00:21:47] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:25 [I 2022-10-11 00:21:47] ...trap/engines/engine.py.326: process: 0.32682 | load: 0.00021 [I 2022-10-11 00:21:50] ...trap/engines/engine.py.321: eval: epoch 90 | batch 440/510 [I 2022-10-11 00:21:50] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:22 [I 2022-10-11 00:21:50] ...trap/engines/engine.py.326: process: 0.32659 | load: 0.00021 [I 2022-10-11 00:21:53] ...trap/engines/engine.py.321: eval: epoch 90 | batch 450/510 [I 2022-10-11 00:21:53] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:19 [I 2022-10-11 00:21:53] ...trap/engines/engine.py.326: process: 0.31999 | load: 0.00021 [I 2022-10-11 00:21:57] ...trap/engines/engine.py.321: eval: epoch 90 | batch 460/510 [I 2022-10-11 00:21:57] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:16 [I 2022-10-11 00:21:57] ...trap/engines/engine.py.326: process: 0.32540 | load: 0.00022 [I 2022-10-11 00:22:00] ...trap/engines/engine.py.321: eval: epoch 90 | batch 470/510 [I 2022-10-11 00:22:00] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:12 [I 2022-10-11 00:22:00] ...trap/engines/engine.py.326: process: 0.32663 | load: 0.00021 [I 2022-10-11 00:22:03] ...trap/engines/engine.py.321: eval: epoch 90 | batch 480/510 [I 2022-10-11 00:22:03] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:09 [I 2022-10-11 00:22:03] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00022 [I 2022-10-11 00:22:06] ...trap/engines/engine.py.321: eval: epoch 90 | batch 490/510 [I 2022-10-11 00:22:06] ...trap/engines/engine.py.325: elapsed: 0:03:16 | left: 0:00:06 [I 2022-10-11 00:22:06] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00021 [I 2022-10-11 00:22:10] ...trap/engines/engine.py.321: eval: epoch 90 | batch 500/510 [I 2022-10-11 00:22:10] ...trap/engines/engine.py.325: elapsed: 0:03:19 | left: 0:00:03 [I 2022-10-11 00:22:10] ...trap/engines/engine.py.326: process: 0.31220 | load: 0.00021 [I 2022-10-11 00:22:13] ...trap/engines/engine.py.321: eval: epoch 90 | batch 510/510 [I 2022-10-11 00:22:13] ...trap/engines/engine.py.325: elapsed: 0:03:22 | left: 0:00:00 [I 2022-10-11 00:22:13] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00017 [S 2022-10-11 00:22:14] ...trap/engines/engine.py.132: eval_epoch.epoch: 90 [I 2022-10-11 00:22:14] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746800 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909500 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936100 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.740100 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907800 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935000 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014013 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009025 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010728 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011691 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009600 [S 2022-10-11 00:22:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008741 [I 2022-10-11 00:22:57] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 00:22:57] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 00:22:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 00:22:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 00:22:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 00:23:02] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 00:23:05] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 00:23:09] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 00:23:09] ...trap/engines/engine.py.163: Training model on trainset for epoch 91 [I 2022-10-11 00:23:09] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 00:23:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 0/2304 [I 2022-10-11 00:23:13] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:07:02 [I 2022-10-11 00:23:13] ...trap/engines/engine.py.229: process: 3.30857 | load: 2.34449 [I 2022-10-11 00:23:13] ...trap/engines/engine.py.230: loss: 0.13277 [I 2022-10-11 00:23:13] ...optimizers/trijoint.py.151: total_norm: 14.260435 [I 2022-10-11 00:23:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 10/2304 [I 2022-10-11 00:23:21] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:22 [I 2022-10-11 00:23:21] ...trap/engines/engine.py.229: process: 0.87560 | load: 0.00540 [I 2022-10-11 00:23:21] ...trap/engines/engine.py.230: loss: 0.11448 [I 2022-10-11 00:23:21] ...optimizers/trijoint.py.151: total_norm: 14.112710 [I 2022-10-11 00:23:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 20/2304 [I 2022-10-11 00:23:30] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:09 [I 2022-10-11 00:23:30] ...trap/engines/engine.py.229: process: 0.88714 | load: 0.00534 [I 2022-10-11 00:23:30] ...trap/engines/engine.py.230: loss: 0.09542 [I 2022-10-11 00:23:30] ...optimizers/trijoint.py.151: total_norm: 12.415145 [I 2022-10-11 00:23:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 30/2304 [I 2022-10-11 00:23:39] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:53 [I 2022-10-11 00:23:39] ...trap/engines/engine.py.229: process: 0.86964 | load: 0.00468 [I 2022-10-11 00:23:39] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-11 00:23:39] ...optimizers/trijoint.py.151: total_norm: 17.080345 [I 2022-10-11 00:23:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 40/2304 [I 2022-10-11 00:23:47] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:29 [I 2022-10-11 00:23:47] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00379 [I 2022-10-11 00:23:47] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-11 00:23:47] ...optimizers/trijoint.py.151: total_norm: 21.637211 [I 2022-10-11 00:23:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 50/2304 [I 2022-10-11 00:23:56] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:31 [I 2022-10-11 00:23:56] ...trap/engines/engine.py.229: process: 0.86690 | load: 0.00402 [I 2022-10-11 00:23:56] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-11 00:23:56] ...optimizers/trijoint.py.151: total_norm: 16.902933 [I 2022-10-11 00:24:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 60/2304 [I 2022-10-11 00:24:05] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:31 [I 2022-10-11 00:24:05] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00367 [I 2022-10-11 00:24:05] ...trap/engines/engine.py.230: loss: 0.13283 [I 2022-10-11 00:24:05] ...optimizers/trijoint.py.151: total_norm: 26.115446 [I 2022-10-11 00:24:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 70/2304 [I 2022-10-11 00:24:13] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:15 [I 2022-10-11 00:24:13] ...trap/engines/engine.py.229: process: 0.87489 | load: 0.00378 [I 2022-10-11 00:24:13] ...trap/engines/engine.py.230: loss: 0.08979 [I 2022-10-11 00:24:13] ...optimizers/trijoint.py.151: total_norm: 10.099613 [I 2022-10-11 00:24:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 80/2304 [I 2022-10-11 00:24:22] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:13 [I 2022-10-11 00:24:22] ...trap/engines/engine.py.229: process: 0.87484 | load: 0.00419 [I 2022-10-11 00:24:22] ...trap/engines/engine.py.230: loss: 0.09575 [I 2022-10-11 00:24:22] ...optimizers/trijoint.py.151: total_norm: 18.336174 [I 2022-10-11 00:24:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 90/2304 [I 2022-10-11 00:24:31] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:57 [I 2022-10-11 00:24:31] ...trap/engines/engine.py.229: process: 0.86263 | load: 0.00346 [I 2022-10-11 00:24:31] ...trap/engines/engine.py.230: loss: 0.11897 [I 2022-10-11 00:24:31] ...optimizers/trijoint.py.151: total_norm: 16.636772 [I 2022-10-11 00:24:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 100/2304 [I 2022-10-11 00:24:39] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:31:38 [I 2022-10-11 00:24:39] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00328 [I 2022-10-11 00:24:39] ...trap/engines/engine.py.230: loss: 0.11241 [I 2022-10-11 00:24:39] ...optimizers/trijoint.py.151: total_norm: 48.104225 [I 2022-10-11 00:24:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 110/2304 [I 2022-10-11 00:24:48] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:42 [I 2022-10-11 00:24:48] ...trap/engines/engine.py.229: process: 0.86824 | load: 0.00334 [I 2022-10-11 00:24:48] ...trap/engines/engine.py.230: loss: 0.10224 [I 2022-10-11 00:24:48] ...optimizers/trijoint.py.151: total_norm: 11.764408 [I 2022-10-11 00:24:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 120/2304 [I 2022-10-11 00:24:57] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:33 [I 2022-10-11 00:24:57] ...trap/engines/engine.py.229: process: 0.87475 | load: 0.00356 [I 2022-10-11 00:24:57] ...trap/engines/engine.py.230: loss: 0.11278 [I 2022-10-11 00:24:57] ...optimizers/trijoint.py.151: total_norm: 11.428885 [I 2022-10-11 00:25:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 130/2304 [I 2022-10-11 00:25:05] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:18 [I 2022-10-11 00:25:05] ...trap/engines/engine.py.229: process: 0.87405 | load: 0.00341 [I 2022-10-11 00:25:05] ...trap/engines/engine.py.230: loss: 0.15271 [I 2022-10-11 00:25:05] ...optimizers/trijoint.py.151: total_norm: 10.457718 [I 2022-10-11 00:25:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 140/2304 [I 2022-10-11 00:25:14] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:09 [I 2022-10-11 00:25:14] ...trap/engines/engine.py.229: process: 0.86499 | load: 0.00325 [I 2022-10-11 00:25:14] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-11 00:25:14] ...optimizers/trijoint.py.151: total_norm: 19.958710 [I 2022-10-11 00:25:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 150/2304 [I 2022-10-11 00:25:23] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:50 [I 2022-10-11 00:25:23] ...trap/engines/engine.py.229: process: 0.87664 | load: 0.00310 [I 2022-10-11 00:25:23] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-11 00:25:23] ...optimizers/trijoint.py.151: total_norm: 13.959851 [I 2022-10-11 00:25:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 160/2304 [I 2022-10-11 00:25:31] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:31:02 [I 2022-10-11 00:25:31] ...trap/engines/engine.py.229: process: 0.88287 | load: 0.00346 [I 2022-10-11 00:25:31] ...trap/engines/engine.py.230: loss: 0.12000 [I 2022-10-11 00:25:31] ...optimizers/trijoint.py.151: total_norm: 18.636593 [I 2022-10-11 00:25:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 170/2304 [I 2022-10-11 00:25:40] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:40 [I 2022-10-11 00:25:40] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00313 [I 2022-10-11 00:25:40] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-11 00:25:40] ...optimizers/trijoint.py.151: total_norm: 31.647903 [I 2022-10-11 00:25:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 180/2304 [I 2022-10-11 00:25:49] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:21 [I 2022-10-11 00:25:49] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00347 [I 2022-10-11 00:25:49] ...trap/engines/engine.py.230: loss: 0.13987 [I 2022-10-11 00:25:49] ...optimizers/trijoint.py.151: total_norm: 11.106849 [I 2022-10-11 00:25:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 190/2304 [I 2022-10-11 00:25:57] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:03 [I 2022-10-11 00:25:57] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00305 [I 2022-10-11 00:25:57] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-11 00:25:57] ...optimizers/trijoint.py.151: total_norm: 12.572164 [I 2022-10-11 00:26:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 200/2304 [I 2022-10-11 00:26:06] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:05 [I 2022-10-11 00:26:06] ...trap/engines/engine.py.229: process: 0.86432 | load: 0.00298 [I 2022-10-11 00:26:06] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-11 00:26:06] ...optimizers/trijoint.py.151: total_norm: 18.357754 [I 2022-10-11 00:26:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 210/2304 [I 2022-10-11 00:26:14] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:29:59 [I 2022-10-11 00:26:14] ...trap/engines/engine.py.229: process: 0.85548 | load: 0.00304 [I 2022-10-11 00:26:14] ...trap/engines/engine.py.230: loss: 0.10160 [I 2022-10-11 00:26:14] ...optimizers/trijoint.py.151: total_norm: 19.608194 [I 2022-10-11 00:26:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 220/2304 [I 2022-10-11 00:26:23] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:42 [I 2022-10-11 00:26:23] ...trap/engines/engine.py.229: process: 0.86239 | load: 0.00303 [I 2022-10-11 00:26:23] ...trap/engines/engine.py.230: loss: 0.15945 [I 2022-10-11 00:26:23] ...optimizers/trijoint.py.151: total_norm: 9.699323 [I 2022-10-11 00:26:32] ...trap/engines/engine.py.224: train: epoch 91 | batch 230/2304 [I 2022-10-11 00:26:32] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:42 [I 2022-10-11 00:26:32] ...trap/engines/engine.py.229: process: 0.86178 | load: 0.00307 [I 2022-10-11 00:26:32] ...trap/engines/engine.py.230: loss: 0.12927 [I 2022-10-11 00:26:32] ...optimizers/trijoint.py.151: total_norm: 12.874483 [I 2022-10-11 00:26:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 240/2304 [I 2022-10-11 00:26:40] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:32 [I 2022-10-11 00:26:40] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00317 [I 2022-10-11 00:26:40] ...trap/engines/engine.py.230: loss: 0.11464 [I 2022-10-11 00:26:40] ...optimizers/trijoint.py.151: total_norm: 17.177526 [I 2022-10-11 00:26:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 250/2304 [I 2022-10-11 00:26:49] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:19 [I 2022-10-11 00:26:49] ...trap/engines/engine.py.229: process: 0.85980 | load: 0.00318 [I 2022-10-11 00:26:49] ...trap/engines/engine.py.230: loss: 0.08510 [I 2022-10-11 00:26:49] ...optimizers/trijoint.py.151: total_norm: 10.754628 [I 2022-10-11 00:26:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 260/2304 [I 2022-10-11 00:26:57] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:22 [I 2022-10-11 00:26:57] ...trap/engines/engine.py.229: process: 0.86768 | load: 0.00298 [I 2022-10-11 00:26:57] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-11 00:26:57] ...optimizers/trijoint.py.151: total_norm: 11.325442 [I 2022-10-11 00:27:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 270/2304 [I 2022-10-11 00:27:06] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:20 [I 2022-10-11 00:27:06] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00313 [I 2022-10-11 00:27:06] ...trap/engines/engine.py.230: loss: 0.15609 [I 2022-10-11 00:27:06] ...optimizers/trijoint.py.151: total_norm: 20.433447 [I 2022-10-11 00:27:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 280/2304 [I 2022-10-11 00:27:15] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:29:10 [I 2022-10-11 00:27:15] ...trap/engines/engine.py.229: process: 0.85826 | load: 0.00338 [I 2022-10-11 00:27:15] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-11 00:27:15] ...optimizers/trijoint.py.151: total_norm: 16.773855 [I 2022-10-11 00:27:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 290/2304 [I 2022-10-11 00:27:23] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:28:56 [I 2022-10-11 00:27:23] ...trap/engines/engine.py.229: process: 0.85580 | load: 0.00312 [I 2022-10-11 00:27:23] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-11 00:27:23] ...optimizers/trijoint.py.151: total_norm: 15.707807 [I 2022-10-11 00:27:32] ...trap/engines/engine.py.224: train: epoch 91 | batch 300/2304 [I 2022-10-11 00:27:32] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:38 [I 2022-10-11 00:27:32] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00316 [I 2022-10-11 00:27:32] ...trap/engines/engine.py.230: loss: 0.19253 [I 2022-10-11 00:27:32] ...optimizers/trijoint.py.151: total_norm: 17.662598 [I 2022-10-11 00:27:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 310/2304 [I 2022-10-11 00:27:41] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:54 [I 2022-10-11 00:27:41] ...trap/engines/engine.py.229: process: 0.87814 | load: 0.00317 [I 2022-10-11 00:27:41] ...trap/engines/engine.py.230: loss: 0.08669 [I 2022-10-11 00:27:41] ...optimizers/trijoint.py.151: total_norm: 9.881847 [I 2022-10-11 00:27:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 320/2304 [I 2022-10-11 00:27:49] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:28:44 [I 2022-10-11 00:27:49] ...trap/engines/engine.py.229: process: 0.87358 | load: 0.00314 [I 2022-10-11 00:27:49] ...trap/engines/engine.py.230: loss: 0.18119 [I 2022-10-11 00:27:49] ...optimizers/trijoint.py.151: total_norm: 16.630430 [I 2022-10-11 00:27:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 330/2304 [I 2022-10-11 00:27:58] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:25 [I 2022-10-11 00:27:58] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00352 [I 2022-10-11 00:27:58] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-11 00:27:58] ...optimizers/trijoint.py.151: total_norm: 16.211315 [I 2022-10-11 00:28:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 340/2304 [I 2022-10-11 00:28:07] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:21 [I 2022-10-11 00:28:07] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00311 [I 2022-10-11 00:28:07] ...trap/engines/engine.py.230: loss: 0.13130 [I 2022-10-11 00:28:07] ...optimizers/trijoint.py.151: total_norm: 14.294195 [I 2022-10-11 00:28:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 350/2304 [I 2022-10-11 00:28:16] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:28:09 [I 2022-10-11 00:28:16] ...trap/engines/engine.py.229: process: 0.86383 | load: 0.00324 [I 2022-10-11 00:28:16] ...trap/engines/engine.py.230: loss: 0.25695 [I 2022-10-11 00:28:16] ...optimizers/trijoint.py.151: total_norm: 20.788290 [I 2022-10-11 00:28:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 360/2304 [I 2022-10-11 00:28:24] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:28:08 [I 2022-10-11 00:28:24] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00320 [I 2022-10-11 00:28:24] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-11 00:28:24] ...optimizers/trijoint.py.151: total_norm: 15.953053 [I 2022-10-11 00:28:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 370/2304 [I 2022-10-11 00:28:33] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:44 [I 2022-10-11 00:28:33] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00325 [I 2022-10-11 00:28:33] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-11 00:28:33] ...optimizers/trijoint.py.151: total_norm: 17.291174 [I 2022-10-11 00:28:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 380/2304 [I 2022-10-11 00:28:42] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:40 [I 2022-10-11 00:28:42] ...trap/engines/engine.py.229: process: 0.86813 | load: 0.00328 [I 2022-10-11 00:28:42] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-11 00:28:42] ...optimizers/trijoint.py.151: total_norm: 25.344313 [I 2022-10-11 00:28:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 390/2304 [I 2022-10-11 00:28:50] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:31 [I 2022-10-11 00:28:50] ...trap/engines/engine.py.229: process: 0.87060 | load: 0.00311 [I 2022-10-11 00:28:50] ...trap/engines/engine.py.230: loss: 0.10199 [I 2022-10-11 00:28:50] ...optimizers/trijoint.py.151: total_norm: 15.465381 [I 2022-10-11 00:28:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 400/2304 [I 2022-10-11 00:28:59] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:39 [I 2022-10-11 00:28:59] ...trap/engines/engine.py.229: process: 0.86700 | load: 0.00314 [I 2022-10-11 00:28:59] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-11 00:28:59] ...optimizers/trijoint.py.151: total_norm: 13.038953 [I 2022-10-11 00:29:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 410/2304 [I 2022-10-11 00:29:08] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:27:17 [I 2022-10-11 00:29:08] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00310 [I 2022-10-11 00:29:08] ...trap/engines/engine.py.230: loss: 0.21459 [I 2022-10-11 00:29:08] ...optimizers/trijoint.py.151: total_norm: 20.986118 [I 2022-10-11 00:29:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 420/2304 [I 2022-10-11 00:29:16] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:27:06 [I 2022-10-11 00:29:16] ...trap/engines/engine.py.229: process: 0.86392 | load: 0.00319 [I 2022-10-11 00:29:16] ...trap/engines/engine.py.230: loss: 0.14747 [I 2022-10-11 00:29:16] ...optimizers/trijoint.py.151: total_norm: 17.616577 [I 2022-10-11 00:29:25] ...trap/engines/engine.py.224: train: epoch 91 | batch 430/2304 [I 2022-10-11 00:29:25] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:27:03 [I 2022-10-11 00:29:25] ...trap/engines/engine.py.229: process: 0.88044 | load: 0.00311 [I 2022-10-11 00:29:25] ...trap/engines/engine.py.230: loss: 0.12517 [I 2022-10-11 00:29:25] ...optimizers/trijoint.py.151: total_norm: 8.877324 [I 2022-10-11 00:29:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 440/2304 [I 2022-10-11 00:29:34] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:57 [I 2022-10-11 00:29:34] ...trap/engines/engine.py.229: process: 0.86881 | load: 0.00306 [I 2022-10-11 00:29:34] ...trap/engines/engine.py.230: loss: 0.10810 [I 2022-10-11 00:29:34] ...optimizers/trijoint.py.151: total_norm: 16.799261 [I 2022-10-11 00:29:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 450/2304 [I 2022-10-11 00:29:42] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:26:48 [I 2022-10-11 00:29:42] ...trap/engines/engine.py.229: process: 0.87452 | load: 0.00311 [I 2022-10-11 00:29:42] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-11 00:29:42] ...optimizers/trijoint.py.151: total_norm: 9.565414 [I 2022-10-11 00:29:51] ...trap/engines/engine.py.224: train: epoch 91 | batch 460/2304 [I 2022-10-11 00:29:51] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:26:27 [I 2022-10-11 00:29:51] ...trap/engines/engine.py.229: process: 0.85981 | load: 0.00308 [I 2022-10-11 00:29:51] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-11 00:29:51] ...optimizers/trijoint.py.151: total_norm: 10.956223 [I 2022-10-11 00:30:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 470/2304 [I 2022-10-11 00:30:00] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:26:29 [I 2022-10-11 00:30:00] ...trap/engines/engine.py.229: process: 0.87176 | load: 0.00310 [I 2022-10-11 00:30:00] ...trap/engines/engine.py.230: loss: 0.10795 [I 2022-10-11 00:30:00] ...optimizers/trijoint.py.151: total_norm: 15.011867 [I 2022-10-11 00:30:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 480/2304 [I 2022-10-11 00:30:08] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:26:12 [I 2022-10-11 00:30:08] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00328 [I 2022-10-11 00:30:08] ...trap/engines/engine.py.230: loss: 0.11954 [I 2022-10-11 00:30:08] ...optimizers/trijoint.py.151: total_norm: 20.086939 [I 2022-10-11 00:30:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 490/2304 [I 2022-10-11 00:30:17] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:26:05 [I 2022-10-11 00:30:17] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00308 [I 2022-10-11 00:30:17] ...trap/engines/engine.py.230: loss: 0.09338 [I 2022-10-11 00:30:17] ...optimizers/trijoint.py.151: total_norm: 8.249565 [I 2022-10-11 00:30:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 500/2304 [I 2022-10-11 00:30:26] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:25:51 [I 2022-10-11 00:30:26] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00309 [I 2022-10-11 00:30:26] ...trap/engines/engine.py.230: loss: 0.12966 [I 2022-10-11 00:30:26] ...optimizers/trijoint.py.151: total_norm: 16.170607 [I 2022-10-11 00:30:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 510/2304 [I 2022-10-11 00:30:34] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:48 [I 2022-10-11 00:30:34] ...trap/engines/engine.py.229: process: 0.86170 | load: 0.00317 [I 2022-10-11 00:30:34] ...trap/engines/engine.py.230: loss: 0.22115 [I 2022-10-11 00:30:34] ...optimizers/trijoint.py.151: total_norm: 23.151287 [I 2022-10-11 00:30:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 520/2304 [I 2022-10-11 00:30:43] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:42 [I 2022-10-11 00:30:43] ...trap/engines/engine.py.229: process: 0.86396 | load: 0.00311 [I 2022-10-11 00:30:43] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-11 00:30:43] ...optimizers/trijoint.py.151: total_norm: 19.559711 [I 2022-10-11 00:30:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 530/2304 [I 2022-10-11 00:30:52] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:25:40 [I 2022-10-11 00:30:52] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00311 [I 2022-10-11 00:30:52] ...trap/engines/engine.py.230: loss: 0.13850 [I 2022-10-11 00:30:52] ...optimizers/trijoint.py.151: total_norm: 25.024670 [I 2022-10-11 00:31:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 540/2304 [I 2022-10-11 00:31:00] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:11 [I 2022-10-11 00:31:00] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00293 [I 2022-10-11 00:31:00] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-11 00:31:00] ...optimizers/trijoint.py.151: total_norm: 15.588258 [I 2022-10-11 00:31:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 550/2304 [I 2022-10-11 00:31:09] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:25:02 [I 2022-10-11 00:31:09] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00294 [I 2022-10-11 00:31:09] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-11 00:31:09] ...optimizers/trijoint.py.151: total_norm: 28.893658 [I 2022-10-11 00:31:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 560/2304 [I 2022-10-11 00:31:18] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:50 [I 2022-10-11 00:31:18] ...trap/engines/engine.py.229: process: 0.86041 | load: 0.00297 [I 2022-10-11 00:31:18] ...trap/engines/engine.py.230: loss: 0.13476 [I 2022-10-11 00:31:18] ...optimizers/trijoint.py.151: total_norm: 19.618505 [I 2022-10-11 00:31:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 570/2304 [I 2022-10-11 00:31:26] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:45 [I 2022-10-11 00:31:26] ...trap/engines/engine.py.229: process: 0.87380 | load: 0.00292 [I 2022-10-11 00:31:26] ...trap/engines/engine.py.230: loss: 0.11167 [I 2022-10-11 00:31:26] ...optimizers/trijoint.py.151: total_norm: 10.535251 [I 2022-10-11 00:31:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 580/2304 [I 2022-10-11 00:31:35] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:33 [I 2022-10-11 00:31:35] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00299 [I 2022-10-11 00:31:35] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-11 00:31:35] ...optimizers/trijoint.py.151: total_norm: 17.771025 [I 2022-10-11 00:31:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 590/2304 [I 2022-10-11 00:31:43] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:24 [I 2022-10-11 00:31:43] ...trap/engines/engine.py.229: process: 0.85366 | load: 0.00297 [I 2022-10-11 00:31:43] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-11 00:31:43] ...optimizers/trijoint.py.151: total_norm: 11.904389 [I 2022-10-11 00:31:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 600/2304 [I 2022-10-11 00:31:52] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:13 [I 2022-10-11 00:31:52] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00298 [I 2022-10-11 00:31:52] ...trap/engines/engine.py.230: loss: 0.10714 [I 2022-10-11 00:31:52] ...optimizers/trijoint.py.151: total_norm: 66.788979 [I 2022-10-11 00:32:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 610/2304 [I 2022-10-11 00:32:00] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:01 [I 2022-10-11 00:32:00] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00327 [I 2022-10-11 00:32:00] ...trap/engines/engine.py.230: loss: 0.08605 [I 2022-10-11 00:32:00] ...optimizers/trijoint.py.151: total_norm: 9.262506 [I 2022-10-11 00:32:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 620/2304 [I 2022-10-11 00:32:09] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:54 [I 2022-10-11 00:32:09] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00291 [I 2022-10-11 00:32:09] ...trap/engines/engine.py.230: loss: 0.12743 [I 2022-10-11 00:32:09] ...optimizers/trijoint.py.151: total_norm: 11.268352 [I 2022-10-11 00:32:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 630/2304 [I 2022-10-11 00:32:17] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:52 [I 2022-10-11 00:32:17] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00296 [I 2022-10-11 00:32:17] ...trap/engines/engine.py.230: loss: 0.15413 [I 2022-10-11 00:32:17] ...optimizers/trijoint.py.151: total_norm: 11.510858 [I 2022-10-11 00:32:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 640/2304 [I 2022-10-11 00:32:26] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:41 [I 2022-10-11 00:32:26] ...trap/engines/engine.py.229: process: 0.84402 | load: 0.00303 [I 2022-10-11 00:32:26] ...trap/engines/engine.py.230: loss: 0.19975 [I 2022-10-11 00:32:26] ...optimizers/trijoint.py.151: total_norm: 28.704891 [I 2022-10-11 00:32:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 650/2304 [I 2022-10-11 00:32:35] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:33 [I 2022-10-11 00:32:35] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00298 [I 2022-10-11 00:32:35] ...trap/engines/engine.py.230: loss: 0.07139 [I 2022-10-11 00:32:35] ...optimizers/trijoint.py.151: total_norm: 10.124310 [I 2022-10-11 00:32:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 660/2304 [I 2022-10-11 00:32:43] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:25 [I 2022-10-11 00:32:43] ...trap/engines/engine.py.229: process: 0.86168 | load: 0.00301 [I 2022-10-11 00:32:43] ...trap/engines/engine.py.230: loss: 0.15812 [I 2022-10-11 00:32:43] ...optimizers/trijoint.py.151: total_norm: 24.494505 [I 2022-10-11 00:32:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 670/2304 [I 2022-10-11 00:32:52] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:17 [I 2022-10-11 00:32:52] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00297 [I 2022-10-11 00:32:52] ...trap/engines/engine.py.230: loss: 0.10370 [I 2022-10-11 00:32:52] ...optimizers/trijoint.py.151: total_norm: 12.711473 [I 2022-10-11 00:33:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 680/2304 [I 2022-10-11 00:33:00] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:07 [I 2022-10-11 00:33:00] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00295 [I 2022-10-11 00:33:00] ...trap/engines/engine.py.230: loss: 0.15675 [I 2022-10-11 00:33:00] ...optimizers/trijoint.py.151: total_norm: 12.944362 [I 2022-10-11 00:33:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 690/2304 [I 2022-10-11 00:33:09] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:56 [I 2022-10-11 00:33:09] ...trap/engines/engine.py.229: process: 0.86600 | load: 0.00294 [I 2022-10-11 00:33:09] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-11 00:33:09] ...optimizers/trijoint.py.151: total_norm: 15.661260 [I 2022-10-11 00:33:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 700/2304 [I 2022-10-11 00:33:17] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:59 [I 2022-10-11 00:33:17] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00297 [I 2022-10-11 00:33:17] ...trap/engines/engine.py.230: loss: 0.08784 [I 2022-10-11 00:33:17] ...optimizers/trijoint.py.151: total_norm: 14.431966 [I 2022-10-11 00:33:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 710/2304 [I 2022-10-11 00:33:26] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:46 [I 2022-10-11 00:33:26] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00295 [I 2022-10-11 00:33:26] ...trap/engines/engine.py.230: loss: 0.18577 [I 2022-10-11 00:33:26] ...optimizers/trijoint.py.151: total_norm: 10.243331 [I 2022-10-11 00:33:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 720/2304 [I 2022-10-11 00:33:35] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:43 [I 2022-10-11 00:33:35] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00295 [I 2022-10-11 00:33:35] ...trap/engines/engine.py.230: loss: 0.21107 [I 2022-10-11 00:33:35] ...optimizers/trijoint.py.151: total_norm: 39.836536 [I 2022-10-11 00:33:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 730/2304 [I 2022-10-11 00:33:43] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:29 [I 2022-10-11 00:33:43] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00293 [I 2022-10-11 00:33:43] ...trap/engines/engine.py.230: loss: 0.13980 [I 2022-10-11 00:33:43] ...optimizers/trijoint.py.151: total_norm: 16.184206 [I 2022-10-11 00:33:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 740/2304 [I 2022-10-11 00:33:52] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:17 [I 2022-10-11 00:33:52] ...trap/engines/engine.py.229: process: 0.86216 | load: 0.00299 [I 2022-10-11 00:33:52] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-11 00:33:52] ...optimizers/trijoint.py.151: total_norm: 14.715981 [I 2022-10-11 00:34:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 750/2304 [I 2022-10-11 00:34:00] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:28 [I 2022-10-11 00:34:00] ...trap/engines/engine.py.229: process: 0.87350 | load: 0.00312 [I 2022-10-11 00:34:00] ...trap/engines/engine.py.230: loss: 0.18057 [I 2022-10-11 00:34:00] ...optimizers/trijoint.py.151: total_norm: 22.326937 [I 2022-10-11 00:34:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 760/2304 [I 2022-10-11 00:34:09] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:20 [I 2022-10-11 00:34:09] ...trap/engines/engine.py.229: process: 0.88372 | load: 0.00339 [I 2022-10-11 00:34:09] ...trap/engines/engine.py.230: loss: 0.10547 [I 2022-10-11 00:34:09] ...optimizers/trijoint.py.151: total_norm: 14.367575 [I 2022-10-11 00:34:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 770/2304 [I 2022-10-11 00:34:18] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:59 [I 2022-10-11 00:34:18] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00313 [I 2022-10-11 00:34:18] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-11 00:34:18] ...optimizers/trijoint.py.151: total_norm: 8.581672 [I 2022-10-11 00:34:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 780/2304 [I 2022-10-11 00:34:27] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:22:00 [I 2022-10-11 00:34:27] ...trap/engines/engine.py.229: process: 0.87044 | load: 0.00314 [I 2022-10-11 00:34:27] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-11 00:34:27] ...optimizers/trijoint.py.151: total_norm: 20.899813 [I 2022-10-11 00:34:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 790/2304 [I 2022-10-11 00:34:35] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:49 [I 2022-10-11 00:34:35] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00310 [I 2022-10-11 00:34:35] ...trap/engines/engine.py.230: loss: 0.09748 [I 2022-10-11 00:34:35] ...optimizers/trijoint.py.151: total_norm: 16.666424 [I 2022-10-11 00:34:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 800/2304 [I 2022-10-11 00:34:44] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:42 [I 2022-10-11 00:34:44] ...trap/engines/engine.py.229: process: 0.87955 | load: 0.00316 [I 2022-10-11 00:34:44] ...trap/engines/engine.py.230: loss: 0.09408 [I 2022-10-11 00:34:44] ...optimizers/trijoint.py.151: total_norm: 10.044617 [I 2022-10-11 00:34:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 810/2304 [I 2022-10-11 00:34:53] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:31 [I 2022-10-11 00:34:53] ...trap/engines/engine.py.229: process: 0.84998 | load: 0.00307 [I 2022-10-11 00:34:53] ...trap/engines/engine.py.230: loss: 0.16611 [I 2022-10-11 00:34:53] ...optimizers/trijoint.py.151: total_norm: 7.546479 [I 2022-10-11 00:35:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 820/2304 [I 2022-10-11 00:35:01] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:25 [I 2022-10-11 00:35:01] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00318 [I 2022-10-11 00:35:01] ...trap/engines/engine.py.230: loss: 0.10065 [I 2022-10-11 00:35:01] ...optimizers/trijoint.py.151: total_norm: 8.225374 [I 2022-10-11 00:35:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 830/2304 [I 2022-10-11 00:35:10] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:11 [I 2022-10-11 00:35:10] ...trap/engines/engine.py.229: process: 0.87296 | load: 0.00312 [I 2022-10-11 00:35:10] ...trap/engines/engine.py.230: loss: 0.10757 [I 2022-10-11 00:35:10] ...optimizers/trijoint.py.151: total_norm: 18.302170 [I 2022-10-11 00:35:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 840/2304 [I 2022-10-11 00:35:18] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:21:04 [I 2022-10-11 00:35:18] ...trap/engines/engine.py.229: process: 0.87945 | load: 0.00313 [I 2022-10-11 00:35:18] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-11 00:35:18] ...optimizers/trijoint.py.151: total_norm: 19.162706 [I 2022-10-11 00:35:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 850/2304 [I 2022-10-11 00:35:27] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:21:02 [I 2022-10-11 00:35:27] ...trap/engines/engine.py.229: process: 0.87280 | load: 0.00342 [I 2022-10-11 00:35:27] ...trap/engines/engine.py.230: loss: 0.10759 [I 2022-10-11 00:35:27] ...optimizers/trijoint.py.151: total_norm: 9.883698 [I 2022-10-11 00:35:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 860/2304 [I 2022-10-11 00:35:36] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:43 [I 2022-10-11 00:35:36] ...trap/engines/engine.py.229: process: 0.86378 | load: 0.00319 [I 2022-10-11 00:35:36] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-11 00:35:36] ...optimizers/trijoint.py.151: total_norm: 9.226314 [I 2022-10-11 00:35:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 870/2304 [I 2022-10-11 00:35:44] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:41 [I 2022-10-11 00:35:44] ...trap/engines/engine.py.229: process: 0.87759 | load: 0.00311 [I 2022-10-11 00:35:44] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-11 00:35:44] ...optimizers/trijoint.py.151: total_norm: 13.159113 [I 2022-10-11 00:35:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 880/2304 [I 2022-10-11 00:35:53] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:25 [I 2022-10-11 00:35:53] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00311 [I 2022-10-11 00:35:53] ...trap/engines/engine.py.230: loss: 0.12978 [I 2022-10-11 00:35:53] ...optimizers/trijoint.py.151: total_norm: 30.549803 [I 2022-10-11 00:36:02] ...trap/engines/engine.py.224: train: epoch 91 | batch 890/2304 [I 2022-10-11 00:36:02] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:26 [I 2022-10-11 00:36:02] ...trap/engines/engine.py.229: process: 0.87092 | load: 0.00362 [I 2022-10-11 00:36:02] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-11 00:36:02] ...optimizers/trijoint.py.151: total_norm: 10.985422 [I 2022-10-11 00:36:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 900/2304 [I 2022-10-11 00:36:11] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:20:22 [I 2022-10-11 00:36:11] ...trap/engines/engine.py.229: process: 0.87548 | load: 0.00327 [I 2022-10-11 00:36:11] ...trap/engines/engine.py.230: loss: 0.16849 [I 2022-10-11 00:36:11] ...optimizers/trijoint.py.151: total_norm: 12.748603 [I 2022-10-11 00:36:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 910/2304 [I 2022-10-11 00:36:19] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:11 [I 2022-10-11 00:36:19] ...trap/engines/engine.py.229: process: 0.88014 | load: 0.00319 [I 2022-10-11 00:36:19] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-11 00:36:19] ...optimizers/trijoint.py.151: total_norm: 27.736378 [I 2022-10-11 00:36:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 920/2304 [I 2022-10-11 00:36:28] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:59 [I 2022-10-11 00:36:28] ...trap/engines/engine.py.229: process: 0.85622 | load: 0.00312 [I 2022-10-11 00:36:28] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-11 00:36:28] ...optimizers/trijoint.py.151: total_norm: 16.707232 [I 2022-10-11 00:36:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 930/2304 [I 2022-10-11 00:36:37] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:51 [I 2022-10-11 00:36:37] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00321 [I 2022-10-11 00:36:37] ...trap/engines/engine.py.230: loss: 0.16844 [I 2022-10-11 00:36:37] ...optimizers/trijoint.py.151: total_norm: 8.467376 [I 2022-10-11 00:36:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 940/2304 [I 2022-10-11 00:36:45] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:35 [I 2022-10-11 00:36:45] ...trap/engines/engine.py.229: process: 0.85251 | load: 0.00318 [I 2022-10-11 00:36:45] ...trap/engines/engine.py.230: loss: 0.13604 [I 2022-10-11 00:36:45] ...optimizers/trijoint.py.151: total_norm: 18.907831 [I 2022-10-11 00:36:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 950/2304 [I 2022-10-11 00:36:54] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:27 [I 2022-10-11 00:36:54] ...trap/engines/engine.py.229: process: 0.86229 | load: 0.00310 [I 2022-10-11 00:36:54] ...trap/engines/engine.py.230: loss: 0.13526 [I 2022-10-11 00:36:54] ...optimizers/trijoint.py.151: total_norm: 6.350087 [I 2022-10-11 00:37:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 960/2304 [I 2022-10-11 00:37:03] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:22 [I 2022-10-11 00:37:03] ...trap/engines/engine.py.229: process: 0.85190 | load: 0.00310 [I 2022-10-11 00:37:03] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-11 00:37:03] ...optimizers/trijoint.py.151: total_norm: 18.224325 [I 2022-10-11 00:37:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 970/2304 [I 2022-10-11 00:37:11] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:15 [I 2022-10-11 00:37:11] ...trap/engines/engine.py.229: process: 0.86541 | load: 0.00323 [I 2022-10-11 00:37:11] ...trap/engines/engine.py.230: loss: 0.18016 [I 2022-10-11 00:37:11] ...optimizers/trijoint.py.151: total_norm: 13.575898 [I 2022-10-11 00:37:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 980/2304 [I 2022-10-11 00:37:20] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:03 [I 2022-10-11 00:37:20] ...trap/engines/engine.py.229: process: 0.85850 | load: 0.00312 [I 2022-10-11 00:37:20] ...trap/engines/engine.py.230: loss: 0.13048 [I 2022-10-11 00:37:20] ...optimizers/trijoint.py.151: total_norm: 12.737039 [I 2022-10-11 00:37:29] ...trap/engines/engine.py.224: train: epoch 91 | batch 990/2304 [I 2022-10-11 00:37:29] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:18:56 [I 2022-10-11 00:37:29] ...trap/engines/engine.py.229: process: 0.87701 | load: 0.00317 [I 2022-10-11 00:37:29] ...trap/engines/engine.py.230: loss: 0.13982 [I 2022-10-11 00:37:29] ...optimizers/trijoint.py.151: total_norm: 8.251287 [I 2022-10-11 00:37:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 1000/2304 [I 2022-10-11 00:37:37] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:46 [I 2022-10-11 00:37:37] ...trap/engines/engine.py.229: process: 0.86811 | load: 0.00314 [I 2022-10-11 00:37:37] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-11 00:37:37] ...optimizers/trijoint.py.151: total_norm: 14.717671 [I 2022-10-11 00:37:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 1010/2304 [I 2022-10-11 00:37:46] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:40 [I 2022-10-11 00:37:46] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00312 [I 2022-10-11 00:37:46] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-11 00:37:46] ...optimizers/trijoint.py.151: total_norm: 17.608158 [I 2022-10-11 00:37:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 1020/2304 [I 2022-10-11 00:37:54] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:18:18 [I 2022-10-11 00:37:54] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00292 [I 2022-10-11 00:37:54] ...trap/engines/engine.py.230: loss: 0.17475 [I 2022-10-11 00:37:54] ...optimizers/trijoint.py.151: total_norm: 18.161858 [I 2022-10-11 00:38:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 1030/2304 [I 2022-10-11 00:38:03] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:04 [I 2022-10-11 00:38:03] ...trap/engines/engine.py.229: process: 0.84424 | load: 0.00311 [I 2022-10-11 00:38:03] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-11 00:38:03] ...optimizers/trijoint.py.151: total_norm: 20.511707 [I 2022-10-11 00:38:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 1040/2304 [I 2022-10-11 00:38:11] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:17:50 [I 2022-10-11 00:38:11] ...trap/engines/engine.py.229: process: 0.84342 | load: 0.00302 [I 2022-10-11 00:38:11] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-11 00:38:11] ...optimizers/trijoint.py.151: total_norm: 15.225396 [I 2022-10-11 00:38:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 1050/2304 [I 2022-10-11 00:38:38] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:43:05 [I 2022-10-11 00:38:38] ...trap/engines/engine.py.229: process: 0.85602 | load: 0.00295 [I 2022-10-11 00:38:38] ...trap/engines/engine.py.230: loss: 0.08831 [I 2022-10-11 00:38:38] ...optimizers/trijoint.py.151: total_norm: 10.255570 [I 2022-10-11 00:38:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 1060/2304 [I 2022-10-11 00:38:47] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:20:16 [I 2022-10-11 00:38:47] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00302 [I 2022-10-11 00:38:47] ...trap/engines/engine.py.230: loss: 0.25966 [I 2022-10-11 00:38:47] ...optimizers/trijoint.py.151: total_norm: 20.235657 [I 2022-10-11 00:38:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 1070/2304 [I 2022-10-11 00:38:55] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:17:49 [I 2022-10-11 00:38:55] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00312 [I 2022-10-11 00:38:55] ...trap/engines/engine.py.230: loss: 0.08747 [I 2022-10-11 00:38:55] ...optimizers/trijoint.py.151: total_norm: 11.854485 [I 2022-10-11 00:39:04] ...trap/engines/engine.py.224: train: epoch 91 | batch 1080/2304 [I 2022-10-11 00:39:04] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:26 [I 2022-10-11 00:39:04] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00298 [I 2022-10-11 00:39:04] ...trap/engines/engine.py.230: loss: 0.13823 [I 2022-10-11 00:39:04] ...optimizers/trijoint.py.151: total_norm: 23.146015 [I 2022-10-11 00:39:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 1090/2304 [I 2022-10-11 00:39:13] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:17:15 [I 2022-10-11 00:39:13] ...trap/engines/engine.py.229: process: 0.86124 | load: 0.00301 [I 2022-10-11 00:39:13] ...trap/engines/engine.py.230: loss: 0.13756 [I 2022-10-11 00:39:13] ...optimizers/trijoint.py.151: total_norm: 10.247937 [I 2022-10-11 00:39:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 1100/2304 [I 2022-10-11 00:39:21] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:06 [I 2022-10-11 00:39:21] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00295 [I 2022-10-11 00:39:21] ...trap/engines/engine.py.230: loss: 0.14233 [I 2022-10-11 00:39:21] ...optimizers/trijoint.py.151: total_norm: 10.573858 [I 2022-10-11 00:39:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 1110/2304 [I 2022-10-11 00:39:30] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:56 [I 2022-10-11 00:39:30] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00294 [I 2022-10-11 00:39:30] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-11 00:39:30] ...optimizers/trijoint.py.151: total_norm: 16.347374 [I 2022-10-11 00:39:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 1120/2304 [I 2022-10-11 00:39:38] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:49 [I 2022-10-11 00:39:38] ...trap/engines/engine.py.229: process: 0.86266 | load: 0.00298 [I 2022-10-11 00:39:38] ...trap/engines/engine.py.230: loss: 0.11856 [I 2022-10-11 00:39:38] ...optimizers/trijoint.py.151: total_norm: 18.314377 [I 2022-10-11 00:39:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 1130/2304 [I 2022-10-11 00:39:47] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:39 [I 2022-10-11 00:39:47] ...trap/engines/engine.py.229: process: 0.85243 | load: 0.00296 [I 2022-10-11 00:39:47] ...trap/engines/engine.py.230: loss: 0.13871 [I 2022-10-11 00:39:47] ...optimizers/trijoint.py.151: total_norm: 10.926056 [I 2022-10-11 00:39:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 1140/2304 [I 2022-10-11 00:39:55] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:42 [I 2022-10-11 00:39:55] ...trap/engines/engine.py.229: process: 0.86432 | load: 0.00314 [I 2022-10-11 00:39:55] ...trap/engines/engine.py.230: loss: 0.14246 [I 2022-10-11 00:39:55] ...optimizers/trijoint.py.151: total_norm: 28.142254 [I 2022-10-11 00:40:04] ...trap/engines/engine.py.224: train: epoch 91 | batch 1150/2304 [I 2022-10-11 00:40:04] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:41 [I 2022-10-11 00:40:04] ...trap/engines/engine.py.229: process: 0.86798 | load: 0.00316 [I 2022-10-11 00:40:04] ...trap/engines/engine.py.230: loss: 0.12435 [I 2022-10-11 00:40:04] ...optimizers/trijoint.py.151: total_norm: 21.997261 [I 2022-10-11 00:40:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 1160/2304 [I 2022-10-11 00:40:13] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:16:29 [I 2022-10-11 00:40:13] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00315 [I 2022-10-11 00:40:13] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-11 00:40:13] ...optimizers/trijoint.py.151: total_norm: 26.042131 [I 2022-10-11 00:40:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 1170/2304 [I 2022-10-11 00:40:21] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:16:20 [I 2022-10-11 00:40:21] ...trap/engines/engine.py.229: process: 0.85288 | load: 0.00311 [I 2022-10-11 00:40:21] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-11 00:40:21] ...optimizers/trijoint.py.151: total_norm: 8.421722 [I 2022-10-11 00:40:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 1180/2304 [I 2022-10-11 00:40:30] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:16:12 [I 2022-10-11 00:40:30] ...trap/engines/engine.py.229: process: 0.86700 | load: 0.00310 [I 2022-10-11 00:40:30] ...trap/engines/engine.py.230: loss: 0.10597 [I 2022-10-11 00:40:30] ...optimizers/trijoint.py.151: total_norm: 18.240162 [I 2022-10-11 00:40:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 1190/2304 [I 2022-10-11 00:40:39] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:16:02 [I 2022-10-11 00:40:39] ...trap/engines/engine.py.229: process: 0.87830 | load: 0.00316 [I 2022-10-11 00:40:39] ...trap/engines/engine.py.230: loss: 0.13692 [I 2022-10-11 00:40:39] ...optimizers/trijoint.py.151: total_norm: 19.298513 [I 2022-10-11 00:40:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 1200/2304 [I 2022-10-11 00:40:47] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:56 [I 2022-10-11 00:40:47] ...trap/engines/engine.py.229: process: 0.87484 | load: 0.00312 [I 2022-10-11 00:40:47] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-11 00:40:47] ...optimizers/trijoint.py.151: total_norm: 11.899752 [I 2022-10-11 00:40:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 1210/2304 [I 2022-10-11 00:40:56] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:51 [I 2022-10-11 00:40:56] ...trap/engines/engine.py.229: process: 0.88567 | load: 0.00316 [I 2022-10-11 00:40:56] ...trap/engines/engine.py.230: loss: 0.13176 [I 2022-10-11 00:40:56] ...optimizers/trijoint.py.151: total_norm: 14.447610 [I 2022-10-11 00:41:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 1220/2304 [I 2022-10-11 00:41:05] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:39 [I 2022-10-11 00:41:05] ...trap/engines/engine.py.229: process: 0.86870 | load: 0.00313 [I 2022-10-11 00:41:05] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-11 00:41:05] ...optimizers/trijoint.py.151: total_norm: 14.522838 [I 2022-10-11 00:41:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 1230/2304 [I 2022-10-11 00:41:13] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:15:24 [I 2022-10-11 00:41:13] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00320 [I 2022-10-11 00:41:13] ...trap/engines/engine.py.230: loss: 0.12802 [I 2022-10-11 00:41:13] ...optimizers/trijoint.py.151: total_norm: 13.141042 [I 2022-10-11 00:41:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 1240/2304 [I 2022-10-11 00:41:22] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:20 [I 2022-10-11 00:41:22] ...trap/engines/engine.py.229: process: 0.87419 | load: 0.00310 [I 2022-10-11 00:41:22] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-11 00:41:22] ...optimizers/trijoint.py.151: total_norm: 70.820023 [I 2022-10-11 00:41:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 1250/2304 [I 2022-10-11 00:41:31] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:15:16 [I 2022-10-11 00:41:31] ...trap/engines/engine.py.229: process: 0.86220 | load: 0.00315 [I 2022-10-11 00:41:31] ...trap/engines/engine.py.230: loss: 0.10578 [I 2022-10-11 00:41:31] ...optimizers/trijoint.py.151: total_norm: 10.156787 [I 2022-10-11 00:41:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 1260/2304 [I 2022-10-11 00:41:39] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:15:08 [I 2022-10-11 00:41:39] ...trap/engines/engine.py.229: process: 0.87066 | load: 0.00317 [I 2022-10-11 00:41:39] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-11 00:41:39] ...optimizers/trijoint.py.151: total_norm: 11.566701 [I 2022-10-11 00:41:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 1270/2304 [I 2022-10-11 00:41:48] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:53 [I 2022-10-11 00:41:48] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00320 [I 2022-10-11 00:41:48] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-11 00:41:48] ...optimizers/trijoint.py.151: total_norm: 13.955948 [I 2022-10-11 00:41:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 1280/2304 [I 2022-10-11 00:41:57] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:45 [I 2022-10-11 00:41:57] ...trap/engines/engine.py.229: process: 0.87200 | load: 0.00322 [I 2022-10-11 00:41:57] ...trap/engines/engine.py.230: loss: 0.09851 [I 2022-10-11 00:41:57] ...optimizers/trijoint.py.151: total_norm: 9.622662 [I 2022-10-11 00:42:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 1290/2304 [I 2022-10-11 00:42:05] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:37 [I 2022-10-11 00:42:05] ...trap/engines/engine.py.229: process: 0.87325 | load: 0.00310 [I 2022-10-11 00:42:05] ...trap/engines/engine.py.230: loss: 0.09295 [I 2022-10-11 00:42:05] ...optimizers/trijoint.py.151: total_norm: 12.114095 [I 2022-10-11 00:42:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 1300/2304 [I 2022-10-11 00:42:14] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:14:33 [I 2022-10-11 00:42:14] ...trap/engines/engine.py.229: process: 0.86365 | load: 0.00314 [I 2022-10-11 00:42:14] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-11 00:42:14] ...optimizers/trijoint.py.151: total_norm: 34.044178 [I 2022-10-11 00:42:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 1310/2304 [I 2022-10-11 00:42:23] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:14:18 [I 2022-10-11 00:42:23] ...trap/engines/engine.py.229: process: 0.86056 | load: 0.00311 [I 2022-10-11 00:42:23] ...trap/engines/engine.py.230: loss: 0.12878 [I 2022-10-11 00:42:23] ...optimizers/trijoint.py.151: total_norm: 12.903830 [I 2022-10-11 00:42:32] ...trap/engines/engine.py.224: train: epoch 91 | batch 1320/2304 [I 2022-10-11 00:42:32] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:14:08 [I 2022-10-11 00:42:32] ...trap/engines/engine.py.229: process: 0.86502 | load: 0.00311 [I 2022-10-11 00:42:32] ...trap/engines/engine.py.230: loss: 0.13467 [I 2022-10-11 00:42:32] ...optimizers/trijoint.py.151: total_norm: 10.171814 [I 2022-10-11 00:42:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 1330/2304 [I 2022-10-11 00:42:40] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:14:05 [I 2022-10-11 00:42:40] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00315 [I 2022-10-11 00:42:40] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-11 00:42:40] ...optimizers/trijoint.py.151: total_norm: 17.225065 [I 2022-10-11 00:42:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 1340/2304 [I 2022-10-11 00:42:49] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:14:01 [I 2022-10-11 00:42:49] ...trap/engines/engine.py.229: process: 0.86599 | load: 0.00316 [I 2022-10-11 00:42:49] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-11 00:42:49] ...optimizers/trijoint.py.151: total_norm: 14.362054 [I 2022-10-11 00:42:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1350/2304 [I 2022-10-11 00:42:58] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:46 [I 2022-10-11 00:42:58] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00317 [I 2022-10-11 00:42:58] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-11 00:42:58] ...optimizers/trijoint.py.151: total_norm: 10.617913 [I 2022-10-11 00:43:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 1360/2304 [I 2022-10-11 00:43:06] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:13:42 [I 2022-10-11 00:43:06] ...trap/engines/engine.py.229: process: 0.87930 | load: 0.00323 [I 2022-10-11 00:43:06] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-11 00:43:06] ...optimizers/trijoint.py.151: total_norm: 9.166462 [I 2022-10-11 00:43:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 1370/2304 [I 2022-10-11 00:43:15] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:26 [I 2022-10-11 00:43:15] ...trap/engines/engine.py.229: process: 0.86993 | load: 0.00314 [I 2022-10-11 00:43:15] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-11 00:43:15] ...optimizers/trijoint.py.151: total_norm: 20.048843 [I 2022-10-11 00:43:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1380/2304 [I 2022-10-11 00:43:24] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:13:21 [I 2022-10-11 00:43:24] ...trap/engines/engine.py.229: process: 0.86962 | load: 0.00321 [I 2022-10-11 00:43:24] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-11 00:43:24] ...optimizers/trijoint.py.151: total_norm: 9.347716 [I 2022-10-11 00:43:32] ...trap/engines/engine.py.224: train: epoch 91 | batch 1390/2304 [I 2022-10-11 00:43:32] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:13:12 [I 2022-10-11 00:43:32] ...trap/engines/engine.py.229: process: 0.86565 | load: 0.00313 [I 2022-10-11 00:43:32] ...trap/engines/engine.py.230: loss: 0.12024 [I 2022-10-11 00:43:32] ...optimizers/trijoint.py.151: total_norm: 13.169086 [I 2022-10-11 00:43:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 1400/2304 [I 2022-10-11 00:43:41] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:13:07 [I 2022-10-11 00:43:41] ...trap/engines/engine.py.229: process: 0.88435 | load: 0.00310 [I 2022-10-11 00:43:41] ...trap/engines/engine.py.230: loss: 0.09523 [I 2022-10-11 00:43:41] ...optimizers/trijoint.py.151: total_norm: 9.105840 [I 2022-10-11 00:43:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 1410/2304 [I 2022-10-11 00:43:50] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:55 [I 2022-10-11 00:43:50] ...trap/engines/engine.py.229: process: 0.86518 | load: 0.00311 [I 2022-10-11 00:43:50] ...trap/engines/engine.py.230: loss: 0.10094 [I 2022-10-11 00:43:50] ...optimizers/trijoint.py.151: total_norm: 11.953069 [I 2022-10-11 00:43:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1420/2304 [I 2022-10-11 00:43:58] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:32 [I 2022-10-11 00:43:58] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00293 [I 2022-10-11 00:43:58] ...trap/engines/engine.py.230: loss: 0.12586 [I 2022-10-11 00:43:58] ...optimizers/trijoint.py.151: total_norm: 24.228594 [I 2022-10-11 00:44:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 1430/2304 [I 2022-10-11 00:44:07] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:29 [I 2022-10-11 00:44:07] ...trap/engines/engine.py.229: process: 0.86308 | load: 0.00295 [I 2022-10-11 00:44:07] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-11 00:44:07] ...optimizers/trijoint.py.151: total_norm: 12.744526 [I 2022-10-11 00:44:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 1440/2304 [I 2022-10-11 00:44:15] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:12:15 [I 2022-10-11 00:44:15] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00292 [I 2022-10-11 00:44:15] ...trap/engines/engine.py.230: loss: 0.09564 [I 2022-10-11 00:44:15] ...optimizers/trijoint.py.151: total_norm: 9.228836 [I 2022-10-11 00:44:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1450/2304 [I 2022-10-11 00:44:24] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:12:08 [I 2022-10-11 00:44:24] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00293 [I 2022-10-11 00:44:24] ...trap/engines/engine.py.230: loss: 0.12500 [I 2022-10-11 00:44:24] ...optimizers/trijoint.py.151: total_norm: 11.284271 [I 2022-10-11 00:44:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 1460/2304 [I 2022-10-11 00:44:33] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:12:01 [I 2022-10-11 00:44:33] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00292 [I 2022-10-11 00:44:33] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-11 00:44:33] ...optimizers/trijoint.py.151: total_norm: 11.856035 [I 2022-10-11 00:44:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 1470/2304 [I 2022-10-11 00:44:41] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:54 [I 2022-10-11 00:44:41] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00295 [I 2022-10-11 00:44:41] ...trap/engines/engine.py.230: loss: 0.09794 [I 2022-10-11 00:44:41] ...optimizers/trijoint.py.151: total_norm: 17.112196 [I 2022-10-11 00:44:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 1480/2304 [I 2022-10-11 00:44:50] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:47 [I 2022-10-11 00:44:50] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00298 [I 2022-10-11 00:44:50] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-11 00:44:50] ...optimizers/trijoint.py.151: total_norm: 13.502578 [I 2022-10-11 00:44:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1490/2304 [I 2022-10-11 00:44:58] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:35 [I 2022-10-11 00:44:58] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00300 [I 2022-10-11 00:44:58] ...trap/engines/engine.py.230: loss: 0.10057 [I 2022-10-11 00:44:58] ...optimizers/trijoint.py.151: total_norm: 9.547316 [I 2022-10-11 00:45:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 1500/2304 [I 2022-10-11 00:45:07] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:27 [I 2022-10-11 00:45:07] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00385 [I 2022-10-11 00:45:07] ...trap/engines/engine.py.230: loss: 0.09389 [I 2022-10-11 00:45:07] ...optimizers/trijoint.py.151: total_norm: 18.905167 [I 2022-10-11 00:45:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 1510/2304 [I 2022-10-11 00:45:16] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:16 [I 2022-10-11 00:45:16] ...trap/engines/engine.py.229: process: 0.86349 | load: 0.00301 [I 2022-10-11 00:45:16] ...trap/engines/engine.py.230: loss: 0.09795 [I 2022-10-11 00:45:16] ...optimizers/trijoint.py.151: total_norm: 11.100692 [I 2022-10-11 00:45:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1520/2304 [I 2022-10-11 00:45:24] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:11:09 [I 2022-10-11 00:45:24] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00298 [I 2022-10-11 00:45:24] ...trap/engines/engine.py.230: loss: 0.09398 [I 2022-10-11 00:45:24] ...optimizers/trijoint.py.151: total_norm: 13.040557 [I 2022-10-11 00:45:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 1530/2304 [I 2022-10-11 00:45:33] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:11:01 [I 2022-10-11 00:45:33] ...trap/engines/engine.py.229: process: 0.85316 | load: 0.00294 [I 2022-10-11 00:45:33] ...trap/engines/engine.py.230: loss: 0.14262 [I 2022-10-11 00:45:33] ...optimizers/trijoint.py.151: total_norm: 19.419632 [I 2022-10-11 00:45:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 1540/2304 [I 2022-10-11 00:45:41] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:52 [I 2022-10-11 00:45:41] ...trap/engines/engine.py.229: process: 0.85516 | load: 0.00297 [I 2022-10-11 00:45:41] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-11 00:45:41] ...optimizers/trijoint.py.151: total_norm: 16.829447 [I 2022-10-11 00:45:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 1550/2304 [I 2022-10-11 00:45:50] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:46 [I 2022-10-11 00:45:50] ...trap/engines/engine.py.229: process: 0.85844 | load: 0.00303 [I 2022-10-11 00:45:50] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-11 00:45:50] ...optimizers/trijoint.py.151: total_norm: 12.382783 [I 2022-10-11 00:45:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1560/2304 [I 2022-10-11 00:45:58] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:32 [I 2022-10-11 00:45:58] ...trap/engines/engine.py.229: process: 0.83268 | load: 0.00299 [I 2022-10-11 00:45:58] ...trap/engines/engine.py.230: loss: 0.12758 [I 2022-10-11 00:45:58] ...optimizers/trijoint.py.151: total_norm: 12.553456 [I 2022-10-11 00:46:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 1570/2304 [I 2022-10-11 00:46:07] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:31 [I 2022-10-11 00:46:07] ...trap/engines/engine.py.229: process: 0.87604 | load: 0.00295 [I 2022-10-11 00:46:07] ...trap/engines/engine.py.230: loss: 0.09356 [I 2022-10-11 00:46:07] ...optimizers/trijoint.py.151: total_norm: 9.891304 [I 2022-10-11 00:46:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 1580/2304 [I 2022-10-11 00:46:16] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:20 [I 2022-10-11 00:46:16] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00298 [I 2022-10-11 00:46:16] ...trap/engines/engine.py.230: loss: 0.09377 [I 2022-10-11 00:46:16] ...optimizers/trijoint.py.151: total_norm: 12.978531 [I 2022-10-11 00:46:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1590/2304 [I 2022-10-11 00:46:24] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:10:13 [I 2022-10-11 00:46:24] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00398 [I 2022-10-11 00:46:24] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-11 00:46:24] ...optimizers/trijoint.py.151: total_norm: 13.185926 [I 2022-10-11 00:46:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 1600/2304 [I 2022-10-11 00:46:33] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:10:09 [I 2022-10-11 00:46:33] ...trap/engines/engine.py.229: process: 0.87672 | load: 0.00325 [I 2022-10-11 00:46:33] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-11 00:46:33] ...optimizers/trijoint.py.151: total_norm: 35.002296 [I 2022-10-11 00:46:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 1610/2304 [I 2022-10-11 00:46:42] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:59 [I 2022-10-11 00:46:42] ...trap/engines/engine.py.229: process: 0.86777 | load: 0.00308 [I 2022-10-11 00:46:42] ...trap/engines/engine.py.230: loss: 0.13982 [I 2022-10-11 00:46:42] ...optimizers/trijoint.py.151: total_norm: 11.003859 [I 2022-10-11 00:46:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 1620/2304 [I 2022-10-11 00:46:50] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:51 [I 2022-10-11 00:46:50] ...trap/engines/engine.py.229: process: 0.85607 | load: 0.00309 [I 2022-10-11 00:46:50] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-11 00:46:50] ...optimizers/trijoint.py.151: total_norm: 15.533457 [I 2022-10-11 00:46:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 1630/2304 [I 2022-10-11 00:46:59] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:44 [I 2022-10-11 00:46:59] ...trap/engines/engine.py.229: process: 0.86712 | load: 0.00310 [I 2022-10-11 00:46:59] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-11 00:46:59] ...optimizers/trijoint.py.151: total_norm: 16.553553 [I 2022-10-11 00:47:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 1640/2304 [I 2022-10-11 00:47:08] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:35 [I 2022-10-11 00:47:08] ...trap/engines/engine.py.229: process: 0.86438 | load: 0.00313 [I 2022-10-11 00:47:08] ...trap/engines/engine.py.230: loss: 0.10087 [I 2022-10-11 00:47:08] ...optimizers/trijoint.py.151: total_norm: 9.665267 [I 2022-10-11 00:47:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 1650/2304 [I 2022-10-11 00:47:16] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:29 [I 2022-10-11 00:47:16] ...trap/engines/engine.py.229: process: 0.86963 | load: 0.00316 [I 2022-10-11 00:47:16] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-11 00:47:16] ...optimizers/trijoint.py.151: total_norm: 19.979958 [I 2022-10-11 00:47:25] ...trap/engines/engine.py.224: train: epoch 91 | batch 1660/2304 [I 2022-10-11 00:47:25] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:17 [I 2022-10-11 00:47:25] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00314 [I 2022-10-11 00:47:25] ...trap/engines/engine.py.230: loss: 0.12603 [I 2022-10-11 00:47:25] ...optimizers/trijoint.py.151: total_norm: 12.220379 [I 2022-10-11 00:47:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 1670/2304 [I 2022-10-11 00:47:34] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:09:08 [I 2022-10-11 00:47:34] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00313 [I 2022-10-11 00:47:34] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-11 00:47:34] ...optimizers/trijoint.py.151: total_norm: 26.130873 [I 2022-10-11 00:47:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 1680/2304 [I 2022-10-11 00:47:42] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:09:00 [I 2022-10-11 00:47:42] ...trap/engines/engine.py.229: process: 0.86220 | load: 0.00316 [I 2022-10-11 00:47:42] ...trap/engines/engine.py.230: loss: 0.15666 [I 2022-10-11 00:47:42] ...optimizers/trijoint.py.151: total_norm: 18.560423 [I 2022-10-11 00:47:51] ...trap/engines/engine.py.224: train: epoch 91 | batch 1690/2304 [I 2022-10-11 00:47:51] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:52 [I 2022-10-11 00:47:51] ...trap/engines/engine.py.229: process: 0.86631 | load: 0.00311 [I 2022-10-11 00:47:51] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-11 00:47:51] ...optimizers/trijoint.py.151: total_norm: 12.064010 [I 2022-10-11 00:48:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 1700/2304 [I 2022-10-11 00:48:00] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:42 [I 2022-10-11 00:48:00] ...trap/engines/engine.py.229: process: 0.87375 | load: 0.00322 [I 2022-10-11 00:48:00] ...trap/engines/engine.py.230: loss: 0.11904 [I 2022-10-11 00:48:00] ...optimizers/trijoint.py.151: total_norm: 22.483513 [I 2022-10-11 00:48:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 1710/2304 [I 2022-10-11 00:48:08] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:36 [I 2022-10-11 00:48:08] ...trap/engines/engine.py.229: process: 0.86945 | load: 0.00313 [I 2022-10-11 00:48:08] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-11 00:48:08] ...optimizers/trijoint.py.151: total_norm: 14.866015 [I 2022-10-11 00:48:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 1720/2304 [I 2022-10-11 00:48:17] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:08:26 [I 2022-10-11 00:48:17] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00326 [I 2022-10-11 00:48:17] ...trap/engines/engine.py.230: loss: 0.12532 [I 2022-10-11 00:48:17] ...optimizers/trijoint.py.151: total_norm: 12.997266 [I 2022-10-11 00:48:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 1730/2304 [I 2022-10-11 00:48:26] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:08:20 [I 2022-10-11 00:48:26] ...trap/engines/engine.py.229: process: 0.87765 | load: 0.00309 [I 2022-10-11 00:48:26] ...trap/engines/engine.py.230: loss: 0.13609 [I 2022-10-11 00:48:26] ...optimizers/trijoint.py.151: total_norm: 16.376553 [I 2022-10-11 00:48:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 1740/2304 [I 2022-10-11 00:48:34] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:08:09 [I 2022-10-11 00:48:34] ...trap/engines/engine.py.229: process: 0.86729 | load: 0.00313 [I 2022-10-11 00:48:34] ...trap/engines/engine.py.230: loss: 0.08562 [I 2022-10-11 00:48:34] ...optimizers/trijoint.py.151: total_norm: 9.117553 [I 2022-10-11 00:48:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 1750/2304 [I 2022-10-11 00:48:43] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:57 [I 2022-10-11 00:48:43] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00327 [I 2022-10-11 00:48:43] ...trap/engines/engine.py.230: loss: 0.08392 [I 2022-10-11 00:48:43] ...optimizers/trijoint.py.151: total_norm: 14.281639 [I 2022-10-11 00:48:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 1760/2304 [I 2022-10-11 00:48:52] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:52 [I 2022-10-11 00:48:52] ...trap/engines/engine.py.229: process: 0.87062 | load: 0.00358 [I 2022-10-11 00:48:52] ...trap/engines/engine.py.230: loss: 0.14978 [I 2022-10-11 00:48:52] ...optimizers/trijoint.py.151: total_norm: 17.650185 [I 2022-10-11 00:49:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 1770/2304 [I 2022-10-11 00:49:00] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:42 [I 2022-10-11 00:49:00] ...trap/engines/engine.py.229: process: 0.87528 | load: 0.00312 [I 2022-10-11 00:49:00] ...trap/engines/engine.py.230: loss: 0.17482 [I 2022-10-11 00:49:00] ...optimizers/trijoint.py.151: total_norm: 14.957878 [I 2022-10-11 00:49:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 1780/2304 [I 2022-10-11 00:49:09] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:32 [I 2022-10-11 00:49:09] ...trap/engines/engine.py.229: process: 0.86490 | load: 0.00318 [I 2022-10-11 00:49:09] ...trap/engines/engine.py.230: loss: 0.16391 [I 2022-10-11 00:49:09] ...optimizers/trijoint.py.151: total_norm: 36.718220 [I 2022-10-11 00:49:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 1790/2304 [I 2022-10-11 00:49:18] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:07:24 [I 2022-10-11 00:49:18] ...trap/engines/engine.py.229: process: 0.87685 | load: 0.00604 [I 2022-10-11 00:49:18] ...trap/engines/engine.py.230: loss: 0.18381 [I 2022-10-11 00:49:18] ...optimizers/trijoint.py.151: total_norm: 16.458950 [I 2022-10-11 00:49:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1800/2304 [I 2022-10-11 00:49:27] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:07:18 [I 2022-10-11 00:49:27] ...trap/engines/engine.py.229: process: 0.86794 | load: 0.00318 [I 2022-10-11 00:49:27] ...trap/engines/engine.py.230: loss: 0.12686 [I 2022-10-11 00:49:27] ...optimizers/trijoint.py.151: total_norm: 11.791432 [I 2022-10-11 00:49:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 1810/2304 [I 2022-10-11 00:49:35] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:07:05 [I 2022-10-11 00:49:35] ...trap/engines/engine.py.229: process: 0.86011 | load: 0.00320 [I 2022-10-11 00:49:35] ...trap/engines/engine.py.230: loss: 0.10485 [I 2022-10-11 00:49:35] ...optimizers/trijoint.py.151: total_norm: 13.228188 [I 2022-10-11 00:49:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 1820/2304 [I 2022-10-11 00:49:44] ...trap/engines/engine.py.228: elapsed: 0:26:34 | left: 0:07:00 [I 2022-10-11 00:49:44] ...trap/engines/engine.py.229: process: 0.86610 | load: 0.00313 [I 2022-10-11 00:49:44] ...trap/engines/engine.py.230: loss: 0.09866 [I 2022-10-11 00:49:44] ...optimizers/trijoint.py.151: total_norm: 12.422970 [I 2022-10-11 00:49:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 1830/2304 [I 2022-10-11 00:49:53] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:52 [I 2022-10-11 00:49:53] ...trap/engines/engine.py.229: process: 0.87631 | load: 0.00336 [I 2022-10-11 00:49:53] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-11 00:49:53] ...optimizers/trijoint.py.151: total_norm: 26.346092 [I 2022-10-11 00:50:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 1840/2304 [I 2022-10-11 00:50:01] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:39 [I 2022-10-11 00:50:01] ...trap/engines/engine.py.229: process: 0.85637 | load: 0.00306 [I 2022-10-11 00:50:01] ...trap/engines/engine.py.230: loss: 0.14459 [I 2022-10-11 00:50:01] ...optimizers/trijoint.py.151: total_norm: 18.701365 [I 2022-10-11 00:50:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 1850/2304 [I 2022-10-11 00:50:10] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:34 [I 2022-10-11 00:50:10] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00314 [I 2022-10-11 00:50:10] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-11 00:50:10] ...optimizers/trijoint.py.151: total_norm: 77.817940 [I 2022-10-11 00:50:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 1860/2304 [I 2022-10-11 00:50:19] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:24 [I 2022-10-11 00:50:19] ...trap/engines/engine.py.229: process: 0.87380 | load: 0.00313 [I 2022-10-11 00:50:19] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-11 00:50:19] ...optimizers/trijoint.py.151: total_norm: 12.337132 [I 2022-10-11 00:50:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1870/2304 [I 2022-10-11 00:50:27] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:06:12 [I 2022-10-11 00:50:27] ...trap/engines/engine.py.229: process: 0.85359 | load: 0.00301 [I 2022-10-11 00:50:27] ...trap/engines/engine.py.230: loss: 0.18659 [I 2022-10-11 00:50:27] ...optimizers/trijoint.py.151: total_norm: 63.639175 [I 2022-10-11 00:50:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 1880/2304 [I 2022-10-11 00:50:36] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:06:04 [I 2022-10-11 00:50:36] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00300 [I 2022-10-11 00:50:36] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-11 00:50:36] ...optimizers/trijoint.py.151: total_norm: 16.195839 [I 2022-10-11 00:50:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 1890/2304 [I 2022-10-11 00:50:44] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:53 [I 2022-10-11 00:50:44] ...trap/engines/engine.py.229: process: 0.84082 | load: 0.00296 [I 2022-10-11 00:50:44] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-11 00:50:44] ...optimizers/trijoint.py.151: total_norm: 8.624741 [I 2022-10-11 00:50:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 1900/2304 [I 2022-10-11 00:50:53] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:43 [I 2022-10-11 00:50:53] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00301 [I 2022-10-11 00:50:53] ...trap/engines/engine.py.230: loss: 0.17411 [I 2022-10-11 00:50:53] ...optimizers/trijoint.py.151: total_norm: 37.393486 [I 2022-10-11 00:51:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 1910/2304 [I 2022-10-11 00:51:01] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:33 [I 2022-10-11 00:51:01] ...trap/engines/engine.py.229: process: 0.85687 | load: 0.00294 [I 2022-10-11 00:51:01] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-11 00:51:01] ...optimizers/trijoint.py.151: total_norm: 50.235062 [I 2022-10-11 00:51:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 1920/2304 [I 2022-10-11 00:51:10] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:28 [I 2022-10-11 00:51:10] ...trap/engines/engine.py.229: process: 0.85434 | load: 0.00300 [I 2022-10-11 00:51:10] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-11 00:51:10] ...optimizers/trijoint.py.151: total_norm: 16.896187 [I 2022-10-11 00:51:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 1930/2304 [I 2022-10-11 00:51:19] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:18 [I 2022-10-11 00:51:19] ...trap/engines/engine.py.229: process: 0.85229 | load: 0.00295 [I 2022-10-11 00:51:19] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-11 00:51:19] ...optimizers/trijoint.py.151: total_norm: 8.678869 [I 2022-10-11 00:51:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1940/2304 [I 2022-10-11 00:51:27] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:05:10 [I 2022-10-11 00:51:27] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00303 [I 2022-10-11 00:51:27] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-11 00:51:27] ...optimizers/trijoint.py.151: total_norm: 17.736643 [I 2022-10-11 00:51:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 1950/2304 [I 2022-10-11 00:51:36] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:05:02 [I 2022-10-11 00:51:36] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00301 [I 2022-10-11 00:51:36] ...trap/engines/engine.py.230: loss: 0.09723 [I 2022-10-11 00:51:36] ...optimizers/trijoint.py.151: total_norm: 13.574325 [I 2022-10-11 00:51:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 1960/2304 [I 2022-10-11 00:51:44] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:54 [I 2022-10-11 00:51:44] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00306 [I 2022-10-11 00:51:44] ...trap/engines/engine.py.230: loss: 0.09183 [I 2022-10-11 00:51:44] ...optimizers/trijoint.py.151: total_norm: 10.548064 [I 2022-10-11 00:51:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 1970/2304 [I 2022-10-11 00:51:53] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:44 [I 2022-10-11 00:51:53] ...trap/engines/engine.py.229: process: 0.85171 | load: 0.00300 [I 2022-10-11 00:51:53] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-11 00:51:53] ...optimizers/trijoint.py.151: total_norm: 34.602936 [I 2022-10-11 00:52:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 1980/2304 [I 2022-10-11 00:52:01] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:36 [I 2022-10-11 00:52:01] ...trap/engines/engine.py.229: process: 0.84750 | load: 0.00297 [I 2022-10-11 00:52:01] ...trap/engines/engine.py.230: loss: 0.16175 [I 2022-10-11 00:52:01] ...optimizers/trijoint.py.151: total_norm: 11.667298 [I 2022-10-11 00:52:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 1990/2304 [I 2022-10-11 00:52:10] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:28 [I 2022-10-11 00:52:10] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00301 [I 2022-10-11 00:52:10] ...trap/engines/engine.py.230: loss: 0.11597 [I 2022-10-11 00:52:10] ...optimizers/trijoint.py.151: total_norm: 14.051512 [I 2022-10-11 00:52:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 2000/2304 [I 2022-10-11 00:52:19] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:20 [I 2022-10-11 00:52:19] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00313 [I 2022-10-11 00:52:19] ...trap/engines/engine.py.230: loss: 0.11313 [I 2022-10-11 00:52:19] ...optimizers/trijoint.py.151: total_norm: 25.138290 [I 2022-10-11 00:52:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 2010/2304 [I 2022-10-11 00:52:27] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:12 [I 2022-10-11 00:52:27] ...trap/engines/engine.py.229: process: 0.86807 | load: 0.00308 [I 2022-10-11 00:52:27] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-11 00:52:27] ...optimizers/trijoint.py.151: total_norm: 21.030375 [I 2022-10-11 00:52:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 2020/2304 [I 2022-10-11 00:52:36] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:04:04 [I 2022-10-11 00:52:36] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00319 [I 2022-10-11 00:52:36] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-11 00:52:36] ...optimizers/trijoint.py.151: total_norm: 18.210896 [I 2022-10-11 00:52:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 2030/2304 [I 2022-10-11 00:52:44] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:56 [I 2022-10-11 00:52:44] ...trap/engines/engine.py.229: process: 0.86050 | load: 0.00385 [I 2022-10-11 00:52:44] ...trap/engines/engine.py.230: loss: 0.13428 [I 2022-10-11 00:52:44] ...optimizers/trijoint.py.151: total_norm: 15.724205 [I 2022-10-11 00:52:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 2040/2304 [I 2022-10-11 00:52:53] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:46 [I 2022-10-11 00:52:53] ...trap/engines/engine.py.229: process: 0.84973 | load: 0.00306 [I 2022-10-11 00:52:53] ...trap/engines/engine.py.230: loss: 0.08981 [I 2022-10-11 00:52:53] ...optimizers/trijoint.py.151: total_norm: 13.600371 [I 2022-10-11 00:53:02] ...trap/engines/engine.py.224: train: epoch 91 | batch 2050/2304 [I 2022-10-11 00:53:02] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:39 [I 2022-10-11 00:53:02] ...trap/engines/engine.py.229: process: 0.86656 | load: 0.00313 [I 2022-10-11 00:53:02] ...trap/engines/engine.py.230: loss: 0.11269 [I 2022-10-11 00:53:02] ...optimizers/trijoint.py.151: total_norm: 12.757314 [I 2022-10-11 00:53:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 2060/2304 [I 2022-10-11 00:53:10] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:31 [I 2022-10-11 00:53:10] ...trap/engines/engine.py.229: process: 0.87395 | load: 0.00315 [I 2022-10-11 00:53:10] ...trap/engines/engine.py.230: loss: 0.11536 [I 2022-10-11 00:53:10] ...optimizers/trijoint.py.151: total_norm: 17.804354 [I 2022-10-11 00:53:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 2070/2304 [I 2022-10-11 00:53:19] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:03:22 [I 2022-10-11 00:53:19] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00309 [I 2022-10-11 00:53:19] ...trap/engines/engine.py.230: loss: 0.10057 [I 2022-10-11 00:53:19] ...optimizers/trijoint.py.151: total_norm: 15.802496 [I 2022-10-11 00:53:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 2080/2304 [I 2022-10-11 00:53:28] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:03:14 [I 2022-10-11 00:53:28] ...trap/engines/engine.py.229: process: 0.87374 | load: 0.00307 [I 2022-10-11 00:53:28] ...trap/engines/engine.py.230: loss: 0.12325 [I 2022-10-11 00:53:28] ...optimizers/trijoint.py.151: total_norm: 8.693242 [I 2022-10-11 00:53:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 2090/2304 [I 2022-10-11 00:53:36] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:03:05 [I 2022-10-11 00:53:36] ...trap/engines/engine.py.229: process: 0.88785 | load: 0.00319 [I 2022-10-11 00:53:36] ...trap/engines/engine.py.230: loss: 0.23355 [I 2022-10-11 00:53:36] ...optimizers/trijoint.py.151: total_norm: 18.253592 [I 2022-10-11 00:53:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 2100/2304 [I 2022-10-11 00:53:45] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:58 [I 2022-10-11 00:53:45] ...trap/engines/engine.py.229: process: 0.88316 | load: 0.00318 [I 2022-10-11 00:53:45] ...trap/engines/engine.py.230: loss: 0.14967 [I 2022-10-11 00:53:45] ...optimizers/trijoint.py.151: total_norm: 11.341636 [I 2022-10-11 00:53:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 2110/2304 [I 2022-10-11 00:53:54] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:47 [I 2022-10-11 00:53:54] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00311 [I 2022-10-11 00:53:54] ...trap/engines/engine.py.230: loss: 0.20410 [I 2022-10-11 00:53:54] ...optimizers/trijoint.py.151: total_norm: 16.542446 [I 2022-10-11 00:54:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 2120/2304 [I 2022-10-11 00:54:03] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:39 [I 2022-10-11 00:54:03] ...trap/engines/engine.py.229: process: 0.85628 | load: 0.00327 [I 2022-10-11 00:54:03] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-11 00:54:03] ...optimizers/trijoint.py.151: total_norm: 11.938898 [I 2022-10-11 00:54:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 2130/2304 [I 2022-10-11 00:54:11] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:29 [I 2022-10-11 00:54:11] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00310 [I 2022-10-11 00:54:11] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-11 00:54:11] ...optimizers/trijoint.py.151: total_norm: 20.197952 [I 2022-10-11 00:54:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 2140/2304 [I 2022-10-11 00:54:20] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:22 [I 2022-10-11 00:54:20] ...trap/engines/engine.py.229: process: 0.87770 | load: 0.00314 [I 2022-10-11 00:54:20] ...trap/engines/engine.py.230: loss: 0.16841 [I 2022-10-11 00:54:20] ...optimizers/trijoint.py.151: total_norm: 9.485780 [I 2022-10-11 00:54:29] ...trap/engines/engine.py.224: train: epoch 91 | batch 2150/2304 [I 2022-10-11 00:54:29] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:02:13 [I 2022-10-11 00:54:29] ...trap/engines/engine.py.229: process: 0.87827 | load: 0.00311 [I 2022-10-11 00:54:29] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-11 00:54:29] ...optimizers/trijoint.py.151: total_norm: 63.168839 [I 2022-10-11 00:54:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 2160/2304 [I 2022-10-11 00:54:37] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:02:04 [I 2022-10-11 00:54:37] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00308 [I 2022-10-11 00:54:37] ...trap/engines/engine.py.230: loss: 0.17169 [I 2022-10-11 00:54:37] ...optimizers/trijoint.py.151: total_norm: 26.979294 [I 2022-10-11 00:54:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 2170/2304 [I 2022-10-11 00:54:46] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:56 [I 2022-10-11 00:54:46] ...trap/engines/engine.py.229: process: 0.87508 | load: 0.00318 [I 2022-10-11 00:54:46] ...trap/engines/engine.py.230: loss: 0.08798 [I 2022-10-11 00:54:46] ...optimizers/trijoint.py.151: total_norm: 12.492357 [I 2022-10-11 00:54:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 2180/2304 [I 2022-10-11 00:54:55] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:47 [I 2022-10-11 00:54:55] ...trap/engines/engine.py.229: process: 0.86521 | load: 0.00314 [I 2022-10-11 00:54:55] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-11 00:54:55] ...optimizers/trijoint.py.151: total_norm: 18.700283 [I 2022-10-11 00:55:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 2190/2304 [I 2022-10-11 00:55:03] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:38 [I 2022-10-11 00:55:03] ...trap/engines/engine.py.229: process: 0.86660 | load: 0.00318 [I 2022-10-11 00:55:03] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-11 00:55:03] ...optimizers/trijoint.py.151: total_norm: 23.486517 [I 2022-10-11 00:55:12] ...trap/engines/engine.py.224: train: epoch 91 | batch 2200/2304 [I 2022-10-11 00:55:12] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:29 [I 2022-10-11 00:55:12] ...trap/engines/engine.py.229: process: 0.87440 | load: 0.00313 [I 2022-10-11 00:55:12] ...trap/engines/engine.py.230: loss: 0.11609 [I 2022-10-11 00:55:12] ...optimizers/trijoint.py.151: total_norm: 10.104645 [I 2022-10-11 00:55:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 2210/2304 [I 2022-10-11 00:55:20] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:01:20 [I 2022-10-11 00:55:20] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00329 [I 2022-10-11 00:55:20] ...trap/engines/engine.py.230: loss: 0.21437 [I 2022-10-11 00:55:20] ...optimizers/trijoint.py.151: total_norm: 26.528257 [I 2022-10-11 00:55:29] ...trap/engines/engine.py.224: train: epoch 91 | batch 2220/2304 [I 2022-10-11 00:55:29] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:01:12 [I 2022-10-11 00:55:29] ...trap/engines/engine.py.229: process: 0.87342 | load: 0.00354 [I 2022-10-11 00:55:29] ...trap/engines/engine.py.230: loss: 0.10878 [I 2022-10-11 00:55:29] ...optimizers/trijoint.py.151: total_norm: 29.303503 [I 2022-10-11 00:55:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 2230/2304 [I 2022-10-11 00:55:38] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:01:04 [I 2022-10-11 00:55:38] ...trap/engines/engine.py.229: process: 0.87424 | load: 0.00308 [I 2022-10-11 00:55:38] ...trap/engines/engine.py.230: loss: 0.09972 [I 2022-10-11 00:55:38] ...optimizers/trijoint.py.151: total_norm: 13.066783 [I 2022-10-11 00:55:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 2240/2304 [I 2022-10-11 00:55:46] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:55 [I 2022-10-11 00:55:46] ...trap/engines/engine.py.229: process: 0.86008 | load: 0.00316 [I 2022-10-11 00:55:46] ...trap/engines/engine.py.230: loss: 0.16245 [I 2022-10-11 00:55:46] ...optimizers/trijoint.py.151: total_norm: 10.222197 [I 2022-10-11 00:55:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 2250/2304 [I 2022-10-11 00:55:55] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:46 [I 2022-10-11 00:55:55] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00312 [I 2022-10-11 00:55:55] ...trap/engines/engine.py.230: loss: 0.08947 [I 2022-10-11 00:55:55] ...optimizers/trijoint.py.151: total_norm: 17.495394 [I 2022-10-11 00:56:04] ...trap/engines/engine.py.224: train: epoch 91 | batch 2260/2304 [I 2022-10-11 00:56:04] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:38 [I 2022-10-11 00:56:04] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00311 [I 2022-10-11 00:56:04] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-11 00:56:04] ...optimizers/trijoint.py.151: total_norm: 12.799974 [I 2022-10-11 00:56:12] ...trap/engines/engine.py.224: train: epoch 91 | batch 2270/2304 [I 2022-10-11 00:56:12] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:29 [I 2022-10-11 00:56:12] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00312 [I 2022-10-11 00:56:12] ...trap/engines/engine.py.230: loss: 0.11122 [I 2022-10-11 00:56:12] ...optimizers/trijoint.py.151: total_norm: 8.627906 [I 2022-10-11 00:56:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 2280/2304 [I 2022-10-11 00:56:21] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:20 [I 2022-10-11 00:56:21] ...trap/engines/engine.py.229: process: 0.86497 | load: 0.00295 [I 2022-10-11 00:56:21] ...trap/engines/engine.py.230: loss: 0.15214 [I 2022-10-11 00:56:21] ...optimizers/trijoint.py.151: total_norm: 21.756968 [I 2022-10-11 00:56:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 2290/2304 [I 2022-10-11 00:56:30] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:12 [I 2022-10-11 00:56:30] ...trap/engines/engine.py.229: process: 0.87048 | load: 0.00295 [I 2022-10-11 00:56:30] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-11 00:56:30] ...optimizers/trijoint.py.151: total_norm: 12.106609 [I 2022-10-11 00:56:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 2300/2304 [I 2022-10-11 00:56:38] ...trap/engines/engine.py.228: elapsed: 0:33:28 | left: 0:00:03 [I 2022-10-11 00:56:38] ...trap/engines/engine.py.229: process: 0.83626 | load: 0.00015 [I 2022-10-11 00:56:38] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-11 00:56:38] ...optimizers/trijoint.py.151: total_norm: 19.638983 [I 2022-10-11 00:56:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 2304/2304 [I 2022-10-11 00:56:41] ...trap/engines/engine.py.228: elapsed: 0:33:32 | left: 0:00:00 [I 2022-10-11 00:56:41] ...trap/engines/engine.py.229: process: 0.83564 | load: 0.00015 [I 2022-10-11 00:56:41] ...trap/engines/engine.py.230: loss: 0.12361 [I 2022-10-11 00:56:41] ...optimizers/trijoint.py.151: total_norm: 15.463887 [S 2022-10-11 00:56:43] ...trap/engines/engine.py.129: train_epoch.epoch: 91 [S 2022-10-11 00:56:43] ...trap/engines/engine.py.129: train_epoch.loss: 0.1276450987977992 [I 2022-10-11 00:56:43] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 00:56:44] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 91 [I 2022-10-11 00:56:44] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 00:56:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 00:56:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 00:56:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 00:56:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 00:56:50] ...trap/engines/engine.py.321: eval: epoch 91 | batch 0/510 [I 2022-10-11 00:56:50] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:56 [I 2022-10-11 00:56:50] ...trap/engines/engine.py.326: process: 5.63983 | load: 4.93561 [I 2022-10-11 00:56:55] ...trap/engines/engine.py.321: eval: epoch 91 | batch 10/510 [I 2022-10-11 00:56:55] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:16 [I 2022-10-11 00:56:55] ...trap/engines/engine.py.326: process: 0.58438 | load: 0.00018 [I 2022-10-11 00:57:01] ...trap/engines/engine.py.321: eval: epoch 91 | batch 20/510 [I 2022-10-11 00:57:01] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:17 [I 2022-10-11 00:57:01] ...trap/engines/engine.py.326: process: 0.55945 | load: 0.00018 [I 2022-10-11 00:57:08] ...trap/engines/engine.py.321: eval: epoch 91 | batch 30/510 [I 2022-10-11 00:57:08] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:05:15 [I 2022-10-11 00:57:08] ...trap/engines/engine.py.326: process: 0.67533 | load: 0.00018 [I 2022-10-11 00:57:13] ...trap/engines/engine.py.321: eval: epoch 91 | batch 40/510 [I 2022-10-11 00:57:13] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:09 [I 2022-10-11 00:57:13] ...trap/engines/engine.py.326: process: 0.43602 | load: 0.00020 [I 2022-10-11 00:57:18] ...trap/engines/engine.py.321: eval: epoch 91 | batch 50/510 [I 2022-10-11 00:57:18] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:53 [I 2022-10-11 00:57:18] ...trap/engines/engine.py.326: process: 0.52436 | load: 0.00019 [I 2022-10-11 00:57:23] ...trap/engines/engine.py.321: eval: epoch 91 | batch 60/510 [I 2022-10-11 00:57:23] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:30 [I 2022-10-11 00:57:23] ...trap/engines/engine.py.326: process: 0.42324 | load: 0.00019 [I 2022-10-11 00:57:28] ...trap/engines/engine.py.321: eval: epoch 91 | batch 70/510 [I 2022-10-11 00:57:28] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:38 [I 2022-10-11 00:57:28] ...trap/engines/engine.py.326: process: 0.48627 | load: 0.00018 [I 2022-10-11 00:57:33] ...trap/engines/engine.py.321: eval: epoch 91 | batch 80/510 [I 2022-10-11 00:57:33] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:31 [I 2022-10-11 00:57:33] ...trap/engines/engine.py.326: process: 0.51020 | load: 0.00020 [I 2022-10-11 00:57:37] ...trap/engines/engine.py.321: eval: epoch 91 | batch 90/510 [I 2022-10-11 00:57:37] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:02 [I 2022-10-11 00:57:37] ...trap/engines/engine.py.326: process: 0.40896 | load: 0.00019 [I 2022-10-11 00:57:41] ...trap/engines/engine.py.321: eval: epoch 91 | batch 100/510 [I 2022-10-11 00:57:41] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:44 [I 2022-10-11 00:57:41] ...trap/engines/engine.py.326: process: 0.31264 | load: 0.00020 [I 2022-10-11 00:57:45] ...trap/engines/engine.py.321: eval: epoch 91 | batch 110/510 [I 2022-10-11 00:57:45] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:10 [I 2022-10-11 00:57:45] ...trap/engines/engine.py.326: process: 0.32516 | load: 0.00018 [I 2022-10-11 00:57:48] ...trap/engines/engine.py.321: eval: epoch 91 | batch 120/510 [I 2022-10-11 00:57:48] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:03 [I 2022-10-11 00:57:48] ...trap/engines/engine.py.326: process: 0.31627 | load: 0.00019 [I 2022-10-11 00:57:51] ...trap/engines/engine.py.321: eval: epoch 91 | batch 130/510 [I 2022-10-11 00:57:51] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:00 [I 2022-10-11 00:57:51] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00020 [I 2022-10-11 00:57:54] ...trap/engines/engine.py.321: eval: epoch 91 | batch 140/510 [I 2022-10-11 00:57:54] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:56 [I 2022-10-11 00:57:54] ...trap/engines/engine.py.326: process: 0.31113 | load: 0.00018 [I 2022-10-11 00:57:57] ...trap/engines/engine.py.321: eval: epoch 91 | batch 150/510 [I 2022-10-11 00:57:57] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:54 [I 2022-10-11 00:57:57] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00019 [I 2022-10-11 00:58:00] ...trap/engines/engine.py.321: eval: epoch 91 | batch 160/510 [I 2022-10-11 00:58:00] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:50 [I 2022-10-11 00:58:00] ...trap/engines/engine.py.326: process: 0.31053 | load: 0.00019 [I 2022-10-11 00:58:04] ...trap/engines/engine.py.321: eval: epoch 91 | batch 170/510 [I 2022-10-11 00:58:04] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-11 00:58:04] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00019 [I 2022-10-11 00:58:07] ...trap/engines/engine.py.321: eval: epoch 91 | batch 180/510 [I 2022-10-11 00:58:07] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:45 [I 2022-10-11 00:58:07] ...trap/engines/engine.py.326: process: 0.31807 | load: 0.00019 [I 2022-10-11 00:58:10] ...trap/engines/engine.py.321: eval: epoch 91 | batch 190/510 [I 2022-10-11 00:58:10] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:42 [I 2022-10-11 00:58:10] ...trap/engines/engine.py.326: process: 0.32623 | load: 0.00020 [I 2022-10-11 00:58:13] ...trap/engines/engine.py.321: eval: epoch 91 | batch 200/510 [I 2022-10-11 00:58:13] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:39 [I 2022-10-11 00:58:13] ...trap/engines/engine.py.326: process: 0.32631 | load: 0.00022 [I 2022-10-11 00:58:16] ...trap/engines/engine.py.321: eval: epoch 91 | batch 210/510 [I 2022-10-11 00:58:16] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:35 [I 2022-10-11 00:58:16] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00020 [I 2022-10-11 00:58:20] ...trap/engines/engine.py.321: eval: epoch 91 | batch 220/510 [I 2022-10-11 00:58:20] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:32 [I 2022-10-11 00:58:20] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00019 [I 2022-10-11 00:58:23] ...trap/engines/engine.py.321: eval: epoch 91 | batch 230/510 [I 2022-10-11 00:58:23] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-11 00:58:23] ...trap/engines/engine.py.326: process: 0.32099 | load: 0.00020 [I 2022-10-11 00:58:26] ...trap/engines/engine.py.321: eval: epoch 91 | batch 240/510 [I 2022-10-11 00:58:26] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-11 00:58:26] ...trap/engines/engine.py.326: process: 0.32718 | load: 0.00020 [I 2022-10-11 00:58:29] ...trap/engines/engine.py.321: eval: epoch 91 | batch 250/510 [I 2022-10-11 00:58:29] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:22 [I 2022-10-11 00:58:29] ...trap/engines/engine.py.326: process: 0.31180 | load: 0.00020 [I 2022-10-11 00:58:32] ...trap/engines/engine.py.321: eval: epoch 91 | batch 260/510 [I 2022-10-11 00:58:32] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:20 [I 2022-10-11 00:58:32] ...trap/engines/engine.py.326: process: 0.31721 | load: 0.00019 [I 2022-10-11 00:58:36] ...trap/engines/engine.py.321: eval: epoch 91 | batch 270/510 [I 2022-10-11 00:58:36] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-11 00:58:36] ...trap/engines/engine.py.326: process: 0.32157 | load: 0.00021 [I 2022-10-11 00:58:39] ...trap/engines/engine.py.321: eval: epoch 91 | batch 280/510 [I 2022-10-11 00:58:39] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-11 00:58:39] ...trap/engines/engine.py.326: process: 0.32102 | load: 0.00019 [I 2022-10-11 00:58:42] ...trap/engines/engine.py.321: eval: epoch 91 | batch 290/510 [I 2022-10-11 00:58:42] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-11 00:58:42] ...trap/engines/engine.py.326: process: 0.32121 | load: 0.00019 [I 2022-10-11 00:58:45] ...trap/engines/engine.py.321: eval: epoch 91 | batch 300/510 [I 2022-10-11 00:58:45] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:06 [I 2022-10-11 00:58:45] ...trap/engines/engine.py.326: process: 0.31903 | load: 0.00021 [I 2022-10-11 00:58:48] ...trap/engines/engine.py.321: eval: epoch 91 | batch 310/510 [I 2022-10-11 00:58:48] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:04 [I 2022-10-11 00:58:48] ...trap/engines/engine.py.326: process: 0.31844 | load: 0.00023 [I 2022-10-11 00:58:52] ...trap/engines/engine.py.321: eval: epoch 91 | batch 320/510 [I 2022-10-11 00:58:52] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:01 [I 2022-10-11 00:58:52] ...trap/engines/engine.py.326: process: 0.31471 | load: 0.00021 [I 2022-10-11 00:58:55] ...trap/engines/engine.py.321: eval: epoch 91 | batch 330/510 [I 2022-10-11 00:58:55] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:58 [I 2022-10-11 00:58:55] ...trap/engines/engine.py.326: process: 0.32385 | load: 0.00021 [I 2022-10-11 00:58:58] ...trap/engines/engine.py.321: eval: epoch 91 | batch 340/510 [I 2022-10-11 00:58:58] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:55 [I 2022-10-11 00:58:58] ...trap/engines/engine.py.326: process: 0.33282 | load: 0.00022 [I 2022-10-11 00:59:01] ...trap/engines/engine.py.321: eval: epoch 91 | batch 350/510 [I 2022-10-11 00:59:01] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:51 [I 2022-10-11 00:59:01] ...trap/engines/engine.py.326: process: 0.32190 | load: 0.00018 [I 2022-10-11 00:59:05] ...trap/engines/engine.py.321: eval: epoch 91 | batch 360/510 [I 2022-10-11 00:59:05] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:48 [I 2022-10-11 00:59:05] ...trap/engines/engine.py.326: process: 0.32416 | load: 0.00021 [I 2022-10-11 00:59:08] ...trap/engines/engine.py.321: eval: epoch 91 | batch 370/510 [I 2022-10-11 00:59:08] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:45 [I 2022-10-11 00:59:08] ...trap/engines/engine.py.326: process: 0.32577 | load: 0.00021 [I 2022-10-11 00:59:11] ...trap/engines/engine.py.321: eval: epoch 91 | batch 380/510 [I 2022-10-11 00:59:11] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:42 [I 2022-10-11 00:59:11] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00021 [I 2022-10-11 00:59:14] ...trap/engines/engine.py.321: eval: epoch 91 | batch 390/510 [I 2022-10-11 00:59:14] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:39 [I 2022-10-11 00:59:14] ...trap/engines/engine.py.326: process: 0.32791 | load: 0.00022 [I 2022-10-11 00:59:18] ...trap/engines/engine.py.321: eval: epoch 91 | batch 400/510 [I 2022-10-11 00:59:18] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:36 [I 2022-10-11 00:59:18] ...trap/engines/engine.py.326: process: 0.33560 | load: 0.00023 [I 2022-10-11 00:59:21] ...trap/engines/engine.py.321: eval: epoch 91 | batch 410/510 [I 2022-10-11 00:59:21] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:32 [I 2022-10-11 00:59:21] ...trap/engines/engine.py.326: process: 0.32805 | load: 0.00024 [I 2022-10-11 00:59:24] ...trap/engines/engine.py.321: eval: epoch 91 | batch 420/510 [I 2022-10-11 00:59:24] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:29 [I 2022-10-11 00:59:24] ...trap/engines/engine.py.326: process: 0.32467 | load: 0.00021 [I 2022-10-11 00:59:27] ...trap/engines/engine.py.321: eval: epoch 91 | batch 430/510 [I 2022-10-11 00:59:27] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:26 [I 2022-10-11 00:59:27] ...trap/engines/engine.py.326: process: 0.32813 | load: 0.00021 [I 2022-10-11 00:59:31] ...trap/engines/engine.py.321: eval: epoch 91 | batch 440/510 [I 2022-10-11 00:59:31] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:22 [I 2022-10-11 00:59:31] ...trap/engines/engine.py.326: process: 0.32490 | load: 0.00022 [I 2022-10-11 00:59:34] ...trap/engines/engine.py.321: eval: epoch 91 | batch 450/510 [I 2022-10-11 00:59:34] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-11 00:59:34] ...trap/engines/engine.py.326: process: 0.32984 | load: 0.00024 [I 2022-10-11 00:59:37] ...trap/engines/engine.py.321: eval: epoch 91 | batch 460/510 [I 2022-10-11 00:59:37] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:16 [I 2022-10-11 00:59:37] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00023 [I 2022-10-11 00:59:41] ...trap/engines/engine.py.321: eval: epoch 91 | batch 470/510 [I 2022-10-11 00:59:41] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:13 [I 2022-10-11 00:59:41] ...trap/engines/engine.py.326: process: 0.32624 | load: 0.00021 [I 2022-10-11 00:59:44] ...trap/engines/engine.py.321: eval: epoch 91 | batch 480/510 [I 2022-10-11 00:59:44] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:09 [I 2022-10-11 00:59:44] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00021 [I 2022-10-11 00:59:47] ...trap/engines/engine.py.321: eval: epoch 91 | batch 490/510 [I 2022-10-11 00:59:47] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:06 [I 2022-10-11 00:59:47] ...trap/engines/engine.py.326: process: 0.32923 | load: 0.00021 [I 2022-10-11 00:59:50] ...trap/engines/engine.py.321: eval: epoch 91 | batch 500/510 [I 2022-10-11 00:59:50] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:03 [I 2022-10-11 00:59:50] ...trap/engines/engine.py.326: process: 0.32921 | load: 0.00024 [I 2022-10-11 00:59:54] ...trap/engines/engine.py.321: eval: epoch 91 | batch 510/510 [I 2022-10-11 00:59:54] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:00 [I 2022-10-11 00:59:54] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00017 [S 2022-10-11 00:59:54] ...trap/engines/engine.py.132: eval_epoch.epoch: 91 [I 2022-10-11 00:59:54] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.741000 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.915600 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.944400 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.737900 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.916700 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.942300 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014276 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013930 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009058 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016724 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013327 [S 2022-10-11 01:00:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009349 [I 2022-10-11 01:00:40] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 01:00:40] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 01:00:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 01:00:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 01:00:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 01:00:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 01:00:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 01:00:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 01:00:52] ...trap/engines/engine.py.163: Training model on trainset for epoch 92 [I 2022-10-11 01:00:52] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 01:00:55] ...trap/engines/engine.py.224: train: epoch 92 | batch 0/2304 [I 2022-10-11 01:00:55] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:05:53 [I 2022-10-11 01:00:55] ...trap/engines/engine.py.229: process: 3.27834 | load: 2.36212 [I 2022-10-11 01:00:55] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-11 01:00:55] ...optimizers/trijoint.py.151: total_norm: 15.300717 [I 2022-10-11 01:01:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 10/2304 [I 2022-10-11 01:01:04] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:58 [I 2022-10-11 01:01:04] ...trap/engines/engine.py.229: process: 0.85626 | load: 0.00550 [I 2022-10-11 01:01:04] ...trap/engines/engine.py.230: loss: 0.09898 [I 2022-10-11 01:01:04] ...optimizers/trijoint.py.151: total_norm: 18.302799 [I 2022-10-11 01:01:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 20/2304 [I 2022-10-11 01:01:13] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:53 [I 2022-10-11 01:01:13] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00457 [I 2022-10-11 01:01:13] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-11 01:01:13] ...optimizers/trijoint.py.151: total_norm: 15.299057 [I 2022-10-11 01:01:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 30/2304 [I 2022-10-11 01:01:21] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:00 [I 2022-10-11 01:01:21] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00479 [I 2022-10-11 01:01:21] ...trap/engines/engine.py.230: loss: 0.16195 [I 2022-10-11 01:01:21] ...optimizers/trijoint.py.151: total_norm: 25.710150 [I 2022-10-11 01:01:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 40/2304 [I 2022-10-11 01:01:30] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:39 [I 2022-10-11 01:01:30] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00379 [I 2022-10-11 01:01:30] ...trap/engines/engine.py.230: loss: 0.10567 [I 2022-10-11 01:01:30] ...optimizers/trijoint.py.151: total_norm: 34.174789 [I 2022-10-11 01:01:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 50/2304 [I 2022-10-11 01:01:39] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:18 [I 2022-10-11 01:01:39] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00479 [I 2022-10-11 01:01:39] ...trap/engines/engine.py.230: loss: 0.14347 [I 2022-10-11 01:01:39] ...optimizers/trijoint.py.151: total_norm: 22.385515 [I 2022-10-11 01:01:47] ...trap/engines/engine.py.224: train: epoch 92 | batch 60/2304 [I 2022-10-11 01:01:47] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:12 [I 2022-10-11 01:01:47] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00402 [I 2022-10-11 01:01:47] ...trap/engines/engine.py.230: loss: 0.20734 [I 2022-10-11 01:01:47] ...optimizers/trijoint.py.151: total_norm: 28.914532 [I 2022-10-11 01:01:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 70/2304 [I 2022-10-11 01:01:56] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:00 [I 2022-10-11 01:01:56] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00409 [I 2022-10-11 01:01:56] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-11 01:01:56] ...optimizers/trijoint.py.151: total_norm: 13.450698 [I 2022-10-11 01:02:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 80/2304 [I 2022-10-11 01:02:05] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:17 [I 2022-10-11 01:02:05] ...trap/engines/engine.py.229: process: 0.87735 | load: 0.00415 [I 2022-10-11 01:02:05] ...trap/engines/engine.py.230: loss: 0.14468 [I 2022-10-11 01:02:05] ...optimizers/trijoint.py.151: total_norm: 19.928104 [I 2022-10-11 01:02:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 90/2304 [I 2022-10-11 01:02:13] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:57 [I 2022-10-11 01:02:13] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00381 [I 2022-10-11 01:02:13] ...trap/engines/engine.py.230: loss: 0.09630 [I 2022-10-11 01:02:13] ...optimizers/trijoint.py.151: total_norm: 10.728907 [I 2022-10-11 01:02:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 100/2304 [I 2022-10-11 01:02:22] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:42 [I 2022-10-11 01:02:22] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00357 [I 2022-10-11 01:02:22] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-11 01:02:22] ...optimizers/trijoint.py.151: total_norm: 12.893027 [I 2022-10-11 01:02:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 110/2304 [I 2022-10-11 01:02:31] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:35 [I 2022-10-11 01:02:31] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00325 [I 2022-10-11 01:02:31] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-11 01:02:31] ...optimizers/trijoint.py.151: total_norm: 8.917565 [I 2022-10-11 01:02:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 120/2304 [I 2022-10-11 01:02:39] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:26 [I 2022-10-11 01:02:39] ...trap/engines/engine.py.229: process: 0.86994 | load: 0.00347 [I 2022-10-11 01:02:39] ...trap/engines/engine.py.230: loss: 0.10782 [I 2022-10-11 01:02:39] ...optimizers/trijoint.py.151: total_norm: 16.439692 [I 2022-10-11 01:02:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 130/2304 [I 2022-10-11 01:02:48] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:30:49 [I 2022-10-11 01:02:48] ...trap/engines/engine.py.229: process: 0.84137 | load: 0.00308 [I 2022-10-11 01:02:48] ...trap/engines/engine.py.230: loss: 0.09517 [I 2022-10-11 01:02:48] ...optimizers/trijoint.py.151: total_norm: 13.128124 [I 2022-10-11 01:02:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 140/2304 [I 2022-10-11 01:02:56] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:30:47 [I 2022-10-11 01:02:56] ...trap/engines/engine.py.229: process: 0.85512 | load: 0.00305 [I 2022-10-11 01:02:56] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-11 01:02:56] ...optimizers/trijoint.py.151: total_norm: 16.476465 [I 2022-10-11 01:03:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 150/2304 [I 2022-10-11 01:03:05] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:41 [I 2022-10-11 01:03:05] ...trap/engines/engine.py.229: process: 0.85206 | load: 0.00316 [I 2022-10-11 01:03:05] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-11 01:03:05] ...optimizers/trijoint.py.151: total_norm: 18.717909 [I 2022-10-11 01:03:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 160/2304 [I 2022-10-11 01:03:13] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:24 [I 2022-10-11 01:03:13] ...trap/engines/engine.py.229: process: 0.83856 | load: 0.00291 [I 2022-10-11 01:03:13] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-11 01:03:13] ...optimizers/trijoint.py.151: total_norm: 19.389183 [I 2022-10-11 01:03:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 170/2304 [I 2022-10-11 01:03:22] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:18 [I 2022-10-11 01:03:22] ...trap/engines/engine.py.229: process: 0.84624 | load: 0.00303 [I 2022-10-11 01:03:22] ...trap/engines/engine.py.230: loss: 0.10872 [I 2022-10-11 01:03:22] ...optimizers/trijoint.py.151: total_norm: 10.499457 [I 2022-10-11 01:03:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 180/2304 [I 2022-10-11 01:03:31] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:29:59 [I 2022-10-11 01:03:31] ...trap/engines/engine.py.229: process: 0.83139 | load: 0.00297 [I 2022-10-11 01:03:31] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-11 01:03:31] ...optimizers/trijoint.py.151: total_norm: 21.656143 [I 2022-10-11 01:03:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 190/2304 [I 2022-10-11 01:03:39] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:05 [I 2022-10-11 01:03:39] ...trap/engines/engine.py.229: process: 0.85714 | load: 0.00316 [I 2022-10-11 01:03:39] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-11 01:03:39] ...optimizers/trijoint.py.151: total_norm: 13.826873 [I 2022-10-11 01:03:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 200/2304 [I 2022-10-11 01:03:48] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:50 [I 2022-10-11 01:03:48] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00314 [I 2022-10-11 01:03:48] ...trap/engines/engine.py.230: loss: 0.09356 [I 2022-10-11 01:03:48] ...optimizers/trijoint.py.151: total_norm: 12.119903 [I 2022-10-11 01:03:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 210/2304 [I 2022-10-11 01:03:56] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:48 [I 2022-10-11 01:03:56] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00295 [I 2022-10-11 01:03:56] ...trap/engines/engine.py.230: loss: 0.14417 [I 2022-10-11 01:03:56] ...optimizers/trijoint.py.151: total_norm: 18.252001 [I 2022-10-11 01:04:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 220/2304 [I 2022-10-11 01:04:05] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:46 [I 2022-10-11 01:04:05] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00300 [I 2022-10-11 01:04:05] ...trap/engines/engine.py.230: loss: 0.16678 [I 2022-10-11 01:04:05] ...optimizers/trijoint.py.151: total_norm: 20.872852 [I 2022-10-11 01:04:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 230/2304 [I 2022-10-11 01:04:13] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:31 [I 2022-10-11 01:04:13] ...trap/engines/engine.py.229: process: 0.85267 | load: 0.00326 [I 2022-10-11 01:04:13] ...trap/engines/engine.py.230: loss: 0.11802 [I 2022-10-11 01:04:13] ...optimizers/trijoint.py.151: total_norm: 17.461424 [I 2022-10-11 01:04:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 240/2304 [I 2022-10-11 01:04:22] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:30 [I 2022-10-11 01:04:22] ...trap/engines/engine.py.229: process: 0.87622 | load: 0.00315 [I 2022-10-11 01:04:22] ...trap/engines/engine.py.230: loss: 0.14126 [I 2022-10-11 01:04:22] ...optimizers/trijoint.py.151: total_norm: 24.378323 [I 2022-10-11 01:04:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 250/2304 [I 2022-10-11 01:04:30] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:09 [I 2022-10-11 01:04:30] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00305 [I 2022-10-11 01:04:30] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-11 01:04:30] ...optimizers/trijoint.py.151: total_norm: 30.920979 [I 2022-10-11 01:04:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 260/2304 [I 2022-10-11 01:04:39] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:06 [I 2022-10-11 01:04:39] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00296 [I 2022-10-11 01:04:39] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-11 01:04:39] ...optimizers/trijoint.py.151: total_norm: 12.933099 [I 2022-10-11 01:04:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 270/2304 [I 2022-10-11 01:04:48] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:29:04 [I 2022-10-11 01:04:48] ...trap/engines/engine.py.229: process: 0.87088 | load: 0.00294 [I 2022-10-11 01:04:48] ...trap/engines/engine.py.230: loss: 0.08184 [I 2022-10-11 01:04:48] ...optimizers/trijoint.py.151: total_norm: 11.744488 [I 2022-10-11 01:04:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 280/2304 [I 2022-10-11 01:04:56] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:49 [I 2022-10-11 01:04:56] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00307 [I 2022-10-11 01:04:56] ...trap/engines/engine.py.230: loss: 0.18115 [I 2022-10-11 01:04:56] ...optimizers/trijoint.py.151: total_norm: 11.660798 [I 2022-10-11 01:05:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 290/2304 [I 2022-10-11 01:05:05] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:34 [I 2022-10-11 01:05:05] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00308 [I 2022-10-11 01:05:05] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-11 01:05:05] ...optimizers/trijoint.py.151: total_norm: 20.586248 [I 2022-10-11 01:05:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 300/2304 [I 2022-10-11 01:05:13] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:30 [I 2022-10-11 01:05:13] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00306 [I 2022-10-11 01:05:13] ...trap/engines/engine.py.230: loss: 0.16066 [I 2022-10-11 01:05:13] ...optimizers/trijoint.py.151: total_norm: 21.069036 [I 2022-10-11 01:05:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 310/2304 [I 2022-10-11 01:05:22] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:51 [I 2022-10-11 01:05:22] ...trap/engines/engine.py.229: process: 0.88099 | load: 0.00313 [I 2022-10-11 01:05:22] ...trap/engines/engine.py.230: loss: 0.09994 [I 2022-10-11 01:05:22] ...optimizers/trijoint.py.151: total_norm: 15.289296 [I 2022-10-11 01:05:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 320/2304 [I 2022-10-11 01:05:31] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:33 [I 2022-10-11 01:05:31] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00351 [I 2022-10-11 01:05:31] ...trap/engines/engine.py.230: loss: 0.15598 [I 2022-10-11 01:05:31] ...optimizers/trijoint.py.151: total_norm: 11.429264 [I 2022-10-11 01:05:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 330/2304 [I 2022-10-11 01:05:39] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:28:28 [I 2022-10-11 01:05:39] ...trap/engines/engine.py.229: process: 0.86627 | load: 0.00311 [I 2022-10-11 01:05:39] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-11 01:05:39] ...optimizers/trijoint.py.151: total_norm: 20.420042 [I 2022-10-11 01:05:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 340/2304 [I 2022-10-11 01:05:48] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:10 [I 2022-10-11 01:05:48] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00319 [I 2022-10-11 01:05:48] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-11 01:05:48] ...optimizers/trijoint.py.151: total_norm: 28.293701 [I 2022-10-11 01:05:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 350/2304 [I 2022-10-11 01:05:57] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:11 [I 2022-10-11 01:05:57] ...trap/engines/engine.py.229: process: 0.86941 | load: 0.00311 [I 2022-10-11 01:05:57] ...trap/engines/engine.py.230: loss: 0.08512 [I 2022-10-11 01:05:57] ...optimizers/trijoint.py.151: total_norm: 17.106771 [I 2022-10-11 01:06:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 360/2304 [I 2022-10-11 01:06:05] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:28:08 [I 2022-10-11 01:06:05] ...trap/engines/engine.py.229: process: 0.87352 | load: 0.00315 [I 2022-10-11 01:06:05] ...trap/engines/engine.py.230: loss: 0.09164 [I 2022-10-11 01:06:05] ...optimizers/trijoint.py.151: total_norm: 9.694766 [I 2022-10-11 01:06:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 370/2304 [I 2022-10-11 01:06:14] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:28:02 [I 2022-10-11 01:06:14] ...trap/engines/engine.py.229: process: 0.86362 | load: 0.00313 [I 2022-10-11 01:06:14] ...trap/engines/engine.py.230: loss: 0.12368 [I 2022-10-11 01:06:14] ...optimizers/trijoint.py.151: total_norm: 14.920450 [I 2022-10-11 01:06:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 380/2304 [I 2022-10-11 01:06:23] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:42 [I 2022-10-11 01:06:23] ...trap/engines/engine.py.229: process: 0.86656 | load: 0.00313 [I 2022-10-11 01:06:23] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-11 01:06:23] ...optimizers/trijoint.py.151: total_norm: 8.989840 [I 2022-10-11 01:06:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 390/2304 [I 2022-10-11 01:06:31] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:18 [I 2022-10-11 01:06:31] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00305 [I 2022-10-11 01:06:31] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-11 01:06:31] ...optimizers/trijoint.py.151: total_norm: 17.828350 [I 2022-10-11 01:06:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 400/2304 [I 2022-10-11 01:06:40] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:38 [I 2022-10-11 01:06:40] ...trap/engines/engine.py.229: process: 0.87761 | load: 0.00310 [I 2022-10-11 01:06:40] ...trap/engines/engine.py.230: loss: 0.14882 [I 2022-10-11 01:06:40] ...optimizers/trijoint.py.151: total_norm: 9.325890 [I 2022-10-11 01:06:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 410/2304 [I 2022-10-11 01:06:48] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:13 [I 2022-10-11 01:06:48] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00311 [I 2022-10-11 01:06:48] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-11 01:06:48] ...optimizers/trijoint.py.151: total_norm: 17.605768 [I 2022-10-11 01:06:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 420/2304 [I 2022-10-11 01:06:57] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:27:15 [I 2022-10-11 01:06:57] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00314 [I 2022-10-11 01:06:57] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-11 01:06:57] ...optimizers/trijoint.py.151: total_norm: 33.678947 [I 2022-10-11 01:07:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 430/2304 [I 2022-10-11 01:07:06] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:03 [I 2022-10-11 01:07:06] ...trap/engines/engine.py.229: process: 0.87144 | load: 0.00309 [I 2022-10-11 01:07:06] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-11 01:07:06] ...optimizers/trijoint.py.151: total_norm: 15.913065 [I 2022-10-11 01:07:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 440/2304 [I 2022-10-11 01:07:14] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:37 [I 2022-10-11 01:07:14] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00320 [I 2022-10-11 01:07:14] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-11 01:07:14] ...optimizers/trijoint.py.151: total_norm: 23.290949 [I 2022-10-11 01:07:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 450/2304 [I 2022-10-11 01:07:23] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:50 [I 2022-10-11 01:07:23] ...trap/engines/engine.py.229: process: 0.87370 | load: 0.00316 [I 2022-10-11 01:07:23] ...trap/engines/engine.py.230: loss: 0.20692 [I 2022-10-11 01:07:23] ...optimizers/trijoint.py.151: total_norm: 27.891169 [I 2022-10-11 01:07:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 460/2304 [I 2022-10-11 01:07:32] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:23 [I 2022-10-11 01:07:32] ...trap/engines/engine.py.229: process: 0.85696 | load: 0.00327 [I 2022-10-11 01:07:32] ...trap/engines/engine.py.230: loss: 0.10311 [I 2022-10-11 01:07:32] ...optimizers/trijoint.py.151: total_norm: 14.187948 [I 2022-10-11 01:07:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 470/2304 [I 2022-10-11 01:07:40] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:21 [I 2022-10-11 01:07:40] ...trap/engines/engine.py.229: process: 0.85027 | load: 0.00331 [I 2022-10-11 01:07:40] ...trap/engines/engine.py.230: loss: 0.20352 [I 2022-10-11 01:07:40] ...optimizers/trijoint.py.151: total_norm: 16.864698 [I 2022-10-11 01:07:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 480/2304 [I 2022-10-11 01:07:49] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:18 [I 2022-10-11 01:07:49] ...trap/engines/engine.py.229: process: 0.87991 | load: 0.00312 [I 2022-10-11 01:07:49] ...trap/engines/engine.py.230: loss: 0.13498 [I 2022-10-11 01:07:49] ...optimizers/trijoint.py.151: total_norm: 10.661630 [I 2022-10-11 01:07:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 490/2304 [I 2022-10-11 01:07:58] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:15 [I 2022-10-11 01:07:58] ...trap/engines/engine.py.229: process: 0.86348 | load: 0.00326 [I 2022-10-11 01:07:58] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-11 01:07:58] ...optimizers/trijoint.py.151: total_norm: 50.612465 [I 2022-10-11 01:08:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 500/2304 [I 2022-10-11 01:08:07] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:26:01 [I 2022-10-11 01:08:07] ...trap/engines/engine.py.229: process: 0.86786 | load: 0.00331 [I 2022-10-11 01:08:07] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-11 01:08:07] ...optimizers/trijoint.py.151: total_norm: 12.278859 [I 2022-10-11 01:08:15] ...trap/engines/engine.py.224: train: epoch 92 | batch 510/2304 [I 2022-10-11 01:08:15] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:45 [I 2022-10-11 01:08:15] ...trap/engines/engine.py.229: process: 0.86185 | load: 0.00326 [I 2022-10-11 01:08:15] ...trap/engines/engine.py.230: loss: 0.08814 [I 2022-10-11 01:08:15] ...optimizers/trijoint.py.151: total_norm: 9.528659 [I 2022-10-11 01:08:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 520/2304 [I 2022-10-11 01:08:24] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:41 [I 2022-10-11 01:08:24] ...trap/engines/engine.py.229: process: 0.86066 | load: 0.00315 [I 2022-10-11 01:08:24] ...trap/engines/engine.py.230: loss: 0.10018 [I 2022-10-11 01:08:24] ...optimizers/trijoint.py.151: total_norm: 10.418772 [I 2022-10-11 01:08:33] ...trap/engines/engine.py.224: train: epoch 92 | batch 530/2304 [I 2022-10-11 01:08:33] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:34 [I 2022-10-11 01:08:33] ...trap/engines/engine.py.229: process: 0.86980 | load: 0.00311 [I 2022-10-11 01:08:33] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-11 01:08:33] ...optimizers/trijoint.py.151: total_norm: 16.718946 [I 2022-10-11 01:08:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 540/2304 [I 2022-10-11 01:08:41] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:24 [I 2022-10-11 01:08:41] ...trap/engines/engine.py.229: process: 0.86453 | load: 0.00316 [I 2022-10-11 01:08:41] ...trap/engines/engine.py.230: loss: 0.11890 [I 2022-10-11 01:08:41] ...optimizers/trijoint.py.151: total_norm: 19.988323 [I 2022-10-11 01:08:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 550/2304 [I 2022-10-11 01:08:50] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:22 [I 2022-10-11 01:08:50] ...trap/engines/engine.py.229: process: 0.87771 | load: 0.00314 [I 2022-10-11 01:08:50] ...trap/engines/engine.py.230: loss: 0.10373 [I 2022-10-11 01:08:50] ...optimizers/trijoint.py.151: total_norm: 18.111723 [I 2022-10-11 01:08:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 560/2304 [I 2022-10-11 01:08:59] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:11 [I 2022-10-11 01:08:59] ...trap/engines/engine.py.229: process: 0.85904 | load: 0.00321 [I 2022-10-11 01:08:59] ...trap/engines/engine.py.230: loss: 0.20101 [I 2022-10-11 01:08:59] ...optimizers/trijoint.py.151: total_norm: 12.341162 [I 2022-10-11 01:09:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 570/2304 [I 2022-10-11 01:09:07] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:57 [I 2022-10-11 01:09:07] ...trap/engines/engine.py.229: process: 0.86540 | load: 0.00308 [I 2022-10-11 01:09:07] ...trap/engines/engine.py.230: loss: 0.08515 [I 2022-10-11 01:09:07] ...optimizers/trijoint.py.151: total_norm: 10.131746 [I 2022-10-11 01:09:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 580/2304 [I 2022-10-11 01:09:16] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:46 [I 2022-10-11 01:09:16] ...trap/engines/engine.py.229: process: 0.86483 | load: 0.00312 [I 2022-10-11 01:09:16] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-11 01:09:16] ...optimizers/trijoint.py.151: total_norm: 13.800374 [I 2022-10-11 01:09:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 590/2304 [I 2022-10-11 01:09:24] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:25 [I 2022-10-11 01:09:24] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00300 [I 2022-10-11 01:09:24] ...trap/engines/engine.py.230: loss: 0.13079 [I 2022-10-11 01:09:24] ...optimizers/trijoint.py.151: total_norm: 16.010513 [I 2022-10-11 01:09:33] ...trap/engines/engine.py.224: train: epoch 92 | batch 600/2304 [I 2022-10-11 01:09:33] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:16 [I 2022-10-11 01:09:33] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00300 [I 2022-10-11 01:09:33] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-11 01:09:33] ...optimizers/trijoint.py.151: total_norm: 14.414713 [I 2022-10-11 01:09:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 610/2304 [I 2022-10-11 01:09:41] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:23:57 [I 2022-10-11 01:09:41] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00292 [I 2022-10-11 01:09:41] ...trap/engines/engine.py.230: loss: 0.13434 [I 2022-10-11 01:09:41] ...optimizers/trijoint.py.151: total_norm: 11.606612 [I 2022-10-11 01:09:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 620/2304 [I 2022-10-11 01:09:50] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:04 [I 2022-10-11 01:09:50] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00293 [I 2022-10-11 01:09:50] ...trap/engines/engine.py.230: loss: 0.11661 [I 2022-10-11 01:09:50] ...optimizers/trijoint.py.151: total_norm: 17.122326 [I 2022-10-11 01:09:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 630/2304 [I 2022-10-11 01:09:59] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:40 [I 2022-10-11 01:09:59] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00291 [I 2022-10-11 01:09:59] ...trap/engines/engine.py.230: loss: 0.15165 [I 2022-10-11 01:09:59] ...optimizers/trijoint.py.151: total_norm: 16.869511 [I 2022-10-11 01:10:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 640/2304 [I 2022-10-11 01:10:07] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:35 [I 2022-10-11 01:10:07] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00300 [I 2022-10-11 01:10:07] ...trap/engines/engine.py.230: loss: 0.09660 [I 2022-10-11 01:10:07] ...optimizers/trijoint.py.151: total_norm: 12.427644 [I 2022-10-11 01:10:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 650/2304 [I 2022-10-11 01:10:16] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:23 [I 2022-10-11 01:10:16] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00292 [I 2022-10-11 01:10:16] ...trap/engines/engine.py.230: loss: 0.09046 [I 2022-10-11 01:10:16] ...optimizers/trijoint.py.151: total_norm: 10.313192 [I 2022-10-11 01:10:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 660/2304 [I 2022-10-11 01:10:24] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:21 [I 2022-10-11 01:10:24] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00320 [I 2022-10-11 01:10:24] ...trap/engines/engine.py.230: loss: 0.10463 [I 2022-10-11 01:10:24] ...optimizers/trijoint.py.151: total_norm: 14.633213 [I 2022-10-11 01:10:33] ...trap/engines/engine.py.224: train: epoch 92 | batch 670/2304 [I 2022-10-11 01:10:33] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:23 [I 2022-10-11 01:10:33] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00304 [I 2022-10-11 01:10:33] ...trap/engines/engine.py.230: loss: 0.18924 [I 2022-10-11 01:10:33] ...optimizers/trijoint.py.151: total_norm: 11.549408 [I 2022-10-11 01:10:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 680/2304 [I 2022-10-11 01:10:41] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:09 [I 2022-10-11 01:10:41] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00298 [I 2022-10-11 01:10:41] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-11 01:10:41] ...optimizers/trijoint.py.151: total_norm: 14.154904 [I 2022-10-11 01:11:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 690/2304 [I 2022-10-11 01:11:08] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:43:43 [I 2022-10-11 01:11:08] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00320 [I 2022-10-11 01:11:08] ...trap/engines/engine.py.230: loss: 0.14299 [I 2022-10-11 01:11:08] ...optimizers/trijoint.py.151: total_norm: 11.435690 [I 2022-10-11 01:11:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 700/2304 [I 2022-10-11 01:11:17] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:25:03 [I 2022-10-11 01:11:17] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00301 [I 2022-10-11 01:11:17] ...trap/engines/engine.py.230: loss: 0.15607 [I 2022-10-11 01:11:17] ...optimizers/trijoint.py.151: total_norm: 16.731953 [I 2022-10-11 01:11:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 710/2304 [I 2022-10-11 01:11:25] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:59 [I 2022-10-11 01:11:25] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00306 [I 2022-10-11 01:11:25] ...trap/engines/engine.py.230: loss: 0.11167 [I 2022-10-11 01:11:25] ...optimizers/trijoint.py.151: total_norm: 20.196180 [I 2022-10-11 01:11:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 720/2304 [I 2022-10-11 01:11:34] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:33 [I 2022-10-11 01:11:34] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00302 [I 2022-10-11 01:11:34] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-11 01:11:34] ...optimizers/trijoint.py.151: total_norm: 8.180640 [I 2022-10-11 01:11:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 730/2304 [I 2022-10-11 01:11:42] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:25 [I 2022-10-11 01:11:42] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00296 [I 2022-10-11 01:11:42] ...trap/engines/engine.py.230: loss: 0.16275 [I 2022-10-11 01:11:42] ...optimizers/trijoint.py.151: total_norm: 17.682350 [I 2022-10-11 01:11:51] ...trap/engines/engine.py.224: train: epoch 92 | batch 740/2304 [I 2022-10-11 01:11:51] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:12 [I 2022-10-11 01:11:51] ...trap/engines/engine.py.229: process: 0.86014 | load: 0.00295 [I 2022-10-11 01:11:51] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-11 01:11:51] ...optimizers/trijoint.py.151: total_norm: 13.491066 [I 2022-10-11 01:12:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 750/2304 [I 2022-10-11 01:12:00] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:08 [I 2022-10-11 01:12:00] ...trap/engines/engine.py.229: process: 0.85996 | load: 0.00313 [I 2022-10-11 01:12:00] ...trap/engines/engine.py.230: loss: 0.12746 [I 2022-10-11 01:12:00] ...optimizers/trijoint.py.151: total_norm: 17.013397 [I 2022-10-11 01:12:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 760/2304 [I 2022-10-11 01:12:08] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:22:05 [I 2022-10-11 01:12:08] ...trap/engines/engine.py.229: process: 0.85423 | load: 0.00314 [I 2022-10-11 01:12:08] ...trap/engines/engine.py.230: loss: 0.16259 [I 2022-10-11 01:12:08] ...optimizers/trijoint.py.151: total_norm: 15.628424 [I 2022-10-11 01:12:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 770/2304 [I 2022-10-11 01:12:17] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:57 [I 2022-10-11 01:12:17] ...trap/engines/engine.py.229: process: 0.85868 | load: 0.00320 [I 2022-10-11 01:12:17] ...trap/engines/engine.py.230: loss: 0.24785 [I 2022-10-11 01:12:17] ...optimizers/trijoint.py.151: total_norm: 27.887875 [I 2022-10-11 01:12:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 780/2304 [I 2022-10-11 01:12:25] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:57 [I 2022-10-11 01:12:25] ...trap/engines/engine.py.229: process: 0.86489 | load: 0.00323 [I 2022-10-11 01:12:25] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-11 01:12:25] ...optimizers/trijoint.py.151: total_norm: 11.647555 [I 2022-10-11 01:12:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 790/2304 [I 2022-10-11 01:12:34] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:52 [I 2022-10-11 01:12:34] ...trap/engines/engine.py.229: process: 0.85108 | load: 0.00323 [I 2022-10-11 01:12:34] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-11 01:12:34] ...optimizers/trijoint.py.151: total_norm: 18.089207 [I 2022-10-11 01:12:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 800/2304 [I 2022-10-11 01:12:43] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:41 [I 2022-10-11 01:12:43] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00315 [I 2022-10-11 01:12:43] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-11 01:12:43] ...optimizers/trijoint.py.151: total_norm: 13.884912 [I 2022-10-11 01:12:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 810/2304 [I 2022-10-11 01:12:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:33 [I 2022-10-11 01:12:52] ...trap/engines/engine.py.229: process: 0.86222 | load: 0.00308 [I 2022-10-11 01:12:52] ...trap/engines/engine.py.230: loss: 0.14309 [I 2022-10-11 01:12:52] ...optimizers/trijoint.py.151: total_norm: 32.597034 [I 2022-10-11 01:13:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 820/2304 [I 2022-10-11 01:13:00] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:26 [I 2022-10-11 01:13:00] ...trap/engines/engine.py.229: process: 0.86840 | load: 0.00309 [I 2022-10-11 01:13:00] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-11 01:13:00] ...optimizers/trijoint.py.151: total_norm: 9.855110 [I 2022-10-11 01:13:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 830/2304 [I 2022-10-11 01:13:09] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:21:12 [I 2022-10-11 01:13:09] ...trap/engines/engine.py.229: process: 0.87220 | load: 0.00326 [I 2022-10-11 01:13:09] ...trap/engines/engine.py.230: loss: 0.10506 [I 2022-10-11 01:13:09] ...optimizers/trijoint.py.151: total_norm: 15.625375 [I 2022-10-11 01:13:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 840/2304 [I 2022-10-11 01:13:18] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:21:02 [I 2022-10-11 01:13:18] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00332 [I 2022-10-11 01:13:18] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-11 01:13:18] ...optimizers/trijoint.py.151: total_norm: 17.455433 [I 2022-10-11 01:13:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 850/2304 [I 2022-10-11 01:13:26] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:58 [I 2022-10-11 01:13:26] ...trap/engines/engine.py.229: process: 0.86714 | load: 0.00310 [I 2022-10-11 01:13:26] ...trap/engines/engine.py.230: loss: 0.17210 [I 2022-10-11 01:13:26] ...optimizers/trijoint.py.151: total_norm: 57.133808 [I 2022-10-11 01:13:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 860/2304 [I 2022-10-11 01:13:35] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:49 [I 2022-10-11 01:13:35] ...trap/engines/engine.py.229: process: 0.86490 | load: 0.00310 [I 2022-10-11 01:13:35] ...trap/engines/engine.py.230: loss: 0.13796 [I 2022-10-11 01:13:35] ...optimizers/trijoint.py.151: total_norm: 16.865784 [I 2022-10-11 01:13:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 870/2304 [I 2022-10-11 01:13:44] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:37 [I 2022-10-11 01:13:44] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00316 [I 2022-10-11 01:13:44] ...trap/engines/engine.py.230: loss: 0.14738 [I 2022-10-11 01:13:44] ...optimizers/trijoint.py.151: total_norm: 15.595557 [I 2022-10-11 01:13:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 880/2304 [I 2022-10-11 01:13:52] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:39 [I 2022-10-11 01:13:52] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00309 [I 2022-10-11 01:13:52] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-11 01:13:52] ...optimizers/trijoint.py.151: total_norm: 20.446693 [I 2022-10-11 01:14:01] ...trap/engines/engine.py.224: train: epoch 92 | batch 890/2304 [I 2022-10-11 01:14:01] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:20:16 [I 2022-10-11 01:14:01] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00310 [I 2022-10-11 01:14:01] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-11 01:14:01] ...optimizers/trijoint.py.151: total_norm: 9.135313 [I 2022-10-11 01:14:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 900/2304 [I 2022-10-11 01:14:10] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:20:15 [I 2022-10-11 01:14:10] ...trap/engines/engine.py.229: process: 0.86613 | load: 0.00326 [I 2022-10-11 01:14:10] ...trap/engines/engine.py.230: loss: 0.09686 [I 2022-10-11 01:14:10] ...optimizers/trijoint.py.151: total_norm: 14.814191 [I 2022-10-11 01:14:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 910/2304 [I 2022-10-11 01:14:18] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:20:03 [I 2022-10-11 01:14:18] ...trap/engines/engine.py.229: process: 0.86172 | load: 0.00308 [I 2022-10-11 01:14:18] ...trap/engines/engine.py.230: loss: 0.09603 [I 2022-10-11 01:14:18] ...optimizers/trijoint.py.151: total_norm: 9.412101 [I 2022-10-11 01:14:27] ...trap/engines/engine.py.224: train: epoch 92 | batch 920/2304 [I 2022-10-11 01:14:27] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:20:03 [I 2022-10-11 01:14:27] ...trap/engines/engine.py.229: process: 0.87848 | load: 0.00336 [I 2022-10-11 01:14:27] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-11 01:14:27] ...optimizers/trijoint.py.151: total_norm: 23.544125 [I 2022-10-11 01:14:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 930/2304 [I 2022-10-11 01:14:36] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:47 [I 2022-10-11 01:14:36] ...trap/engines/engine.py.229: process: 0.87724 | load: 0.00315 [I 2022-10-11 01:14:36] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-11 01:14:36] ...optimizers/trijoint.py.151: total_norm: 12.155597 [I 2022-10-11 01:14:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 940/2304 [I 2022-10-11 01:14:44] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:42 [I 2022-10-11 01:14:44] ...trap/engines/engine.py.229: process: 0.86975 | load: 0.00311 [I 2022-10-11 01:14:44] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-11 01:14:44] ...optimizers/trijoint.py.151: total_norm: 10.015663 [I 2022-10-11 01:14:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 950/2304 [I 2022-10-11 01:14:53] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:19:25 [I 2022-10-11 01:14:53] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00322 [I 2022-10-11 01:14:53] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-11 01:14:53] ...optimizers/trijoint.py.151: total_norm: 15.681771 [I 2022-10-11 01:15:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 960/2304 [I 2022-10-11 01:15:02] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:27 [I 2022-10-11 01:15:02] ...trap/engines/engine.py.229: process: 0.87803 | load: 0.00313 [I 2022-10-11 01:15:02] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-11 01:15:02] ...optimizers/trijoint.py.151: total_norm: 9.033660 [I 2022-10-11 01:15:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 970/2304 [I 2022-10-11 01:15:10] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:19:11 [I 2022-10-11 01:15:10] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00467 [I 2022-10-11 01:15:10] ...trap/engines/engine.py.230: loss: 0.14384 [I 2022-10-11 01:15:10] ...optimizers/trijoint.py.151: total_norm: 30.647570 [I 2022-10-11 01:15:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 980/2304 [I 2022-10-11 01:15:19] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:19:09 [I 2022-10-11 01:15:19] ...trap/engines/engine.py.229: process: 0.86521 | load: 0.00316 [I 2022-10-11 01:15:19] ...trap/engines/engine.py.230: loss: 0.10311 [I 2022-10-11 01:15:19] ...optimizers/trijoint.py.151: total_norm: 20.024441 [I 2022-10-11 01:15:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 990/2304 [I 2022-10-11 01:15:28] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:50 [I 2022-10-11 01:15:28] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00312 [I 2022-10-11 01:15:28] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-11 01:15:28] ...optimizers/trijoint.py.151: total_norm: 13.224267 [I 2022-10-11 01:15:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 1000/2304 [I 2022-10-11 01:15:36] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:49 [I 2022-10-11 01:15:36] ...trap/engines/engine.py.229: process: 0.86510 | load: 0.00317 [I 2022-10-11 01:15:36] ...trap/engines/engine.py.230: loss: 0.17211 [I 2022-10-11 01:15:36] ...optimizers/trijoint.py.151: total_norm: 16.522137 [I 2022-10-11 01:15:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1010/2304 [I 2022-10-11 01:15:45] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:38 [I 2022-10-11 01:15:45] ...trap/engines/engine.py.229: process: 0.85309 | load: 0.00312 [I 2022-10-11 01:15:45] ...trap/engines/engine.py.230: loss: 0.08382 [I 2022-10-11 01:15:45] ...optimizers/trijoint.py.151: total_norm: 9.221021 [I 2022-10-11 01:15:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 1020/2304 [I 2022-10-11 01:15:54] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:28 [I 2022-10-11 01:15:54] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00317 [I 2022-10-11 01:15:54] ...trap/engines/engine.py.230: loss: 0.13113 [I 2022-10-11 01:15:54] ...optimizers/trijoint.py.151: total_norm: 21.774908 [I 2022-10-11 01:16:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 1030/2304 [I 2022-10-11 01:16:02] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:10 [I 2022-10-11 01:16:02] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00300 [I 2022-10-11 01:16:02] ...trap/engines/engine.py.230: loss: 0.11593 [I 2022-10-11 01:16:02] ...optimizers/trijoint.py.151: total_norm: 17.827667 [I 2022-10-11 01:16:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 1040/2304 [I 2022-10-11 01:16:11] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:54 [I 2022-10-11 01:16:11] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00297 [I 2022-10-11 01:16:11] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-11 01:16:11] ...optimizers/trijoint.py.151: total_norm: 10.867173 [I 2022-10-11 01:16:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 1050/2304 [I 2022-10-11 01:16:19] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:54 [I 2022-10-11 01:16:19] ...trap/engines/engine.py.229: process: 0.84718 | load: 0.00293 [I 2022-10-11 01:16:19] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-11 01:16:19] ...optimizers/trijoint.py.151: total_norm: 11.525646 [I 2022-10-11 01:16:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 1060/2304 [I 2022-10-11 01:16:28] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:49 [I 2022-10-11 01:16:28] ...trap/engines/engine.py.229: process: 0.87743 | load: 0.00296 [I 2022-10-11 01:16:28] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-11 01:16:28] ...optimizers/trijoint.py.151: total_norm: 19.476355 [I 2022-10-11 01:16:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1070/2304 [I 2022-10-11 01:16:37] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:30 [I 2022-10-11 01:16:37] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00294 [I 2022-10-11 01:16:37] ...trap/engines/engine.py.230: loss: 0.21068 [I 2022-10-11 01:16:37] ...optimizers/trijoint.py.151: total_norm: 15.248964 [I 2022-10-11 01:16:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1080/2304 [I 2022-10-11 01:16:45] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:22 [I 2022-10-11 01:16:45] ...trap/engines/engine.py.229: process: 0.85672 | load: 0.00298 [I 2022-10-11 01:16:45] ...trap/engines/engine.py.230: loss: 0.14182 [I 2022-10-11 01:16:45] ...optimizers/trijoint.py.151: total_norm: 29.146795 [I 2022-10-11 01:16:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 1090/2304 [I 2022-10-11 01:16:54] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:17:19 [I 2022-10-11 01:16:54] ...trap/engines/engine.py.229: process: 0.86687 | load: 0.00304 [I 2022-10-11 01:16:54] ...trap/engines/engine.py.230: loss: 0.14519 [I 2022-10-11 01:16:54] ...optimizers/trijoint.py.151: total_norm: 15.798959 [I 2022-10-11 01:17:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 1100/2304 [I 2022-10-11 01:17:02] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:17:12 [I 2022-10-11 01:17:02] ...trap/engines/engine.py.229: process: 0.86834 | load: 0.00295 [I 2022-10-11 01:17:02] ...trap/engines/engine.py.230: loss: 0.10383 [I 2022-10-11 01:17:02] ...optimizers/trijoint.py.151: total_norm: 9.043814 [I 2022-10-11 01:17:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 1110/2304 [I 2022-10-11 01:17:11] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:57 [I 2022-10-11 01:17:11] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00310 [I 2022-10-11 01:17:11] ...trap/engines/engine.py.230: loss: 0.14958 [I 2022-10-11 01:17:11] ...optimizers/trijoint.py.151: total_norm: 18.750261 [I 2022-10-11 01:17:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1120/2304 [I 2022-10-11 01:17:20] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:50 [I 2022-10-11 01:17:20] ...trap/engines/engine.py.229: process: 0.85219 | load: 0.00298 [I 2022-10-11 01:17:20] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-11 01:17:20] ...optimizers/trijoint.py.151: total_norm: 15.532418 [I 2022-10-11 01:17:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 1130/2304 [I 2022-10-11 01:17:28] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:39 [I 2022-10-11 01:17:28] ...trap/engines/engine.py.229: process: 0.85287 | load: 0.00304 [I 2022-10-11 01:17:28] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-11 01:17:28] ...optimizers/trijoint.py.151: total_norm: 12.169951 [I 2022-10-11 01:17:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1140/2304 [I 2022-10-11 01:17:37] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:36 [I 2022-10-11 01:17:37] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00311 [I 2022-10-11 01:17:37] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-11 01:17:37] ...optimizers/trijoint.py.151: total_norm: 11.962644 [I 2022-10-11 01:17:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1150/2304 [I 2022-10-11 01:17:45] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:29 [I 2022-10-11 01:17:45] ...trap/engines/engine.py.229: process: 0.85931 | load: 0.00306 [I 2022-10-11 01:17:45] ...trap/engines/engine.py.230: loss: 0.13922 [I 2022-10-11 01:17:45] ...optimizers/trijoint.py.151: total_norm: 41.138535 [I 2022-10-11 01:17:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 1160/2304 [I 2022-10-11 01:17:54] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:17 [I 2022-10-11 01:17:54] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00304 [I 2022-10-11 01:17:54] ...trap/engines/engine.py.230: loss: 0.13268 [I 2022-10-11 01:17:54] ...optimizers/trijoint.py.151: total_norm: 14.703047 [I 2022-10-11 01:18:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 1170/2304 [I 2022-10-11 01:18:02] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:16:16 [I 2022-10-11 01:18:02] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00292 [I 2022-10-11 01:18:02] ...trap/engines/engine.py.230: loss: 0.09618 [I 2022-10-11 01:18:02] ...optimizers/trijoint.py.151: total_norm: 13.550069 [I 2022-10-11 01:18:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 1180/2304 [I 2022-10-11 01:18:11] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:16:05 [I 2022-10-11 01:18:11] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00298 [I 2022-10-11 01:18:11] ...trap/engines/engine.py.230: loss: 0.16797 [I 2022-10-11 01:18:11] ...optimizers/trijoint.py.151: total_norm: 10.054758 [I 2022-10-11 01:18:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1190/2304 [I 2022-10-11 01:18:20] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:52 [I 2022-10-11 01:18:20] ...trap/engines/engine.py.229: process: 0.84385 | load: 0.00294 [I 2022-10-11 01:18:20] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-11 01:18:20] ...optimizers/trijoint.py.151: total_norm: 7.161838 [I 2022-10-11 01:18:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 1200/2304 [I 2022-10-11 01:18:28] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:47 [I 2022-10-11 01:18:28] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00310 [I 2022-10-11 01:18:28] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-11 01:18:28] ...optimizers/trijoint.py.151: total_norm: 18.774458 [I 2022-10-11 01:18:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1210/2304 [I 2022-10-11 01:18:37] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:44 [I 2022-10-11 01:18:37] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00309 [I 2022-10-11 01:18:37] ...trap/engines/engine.py.230: loss: 0.10103 [I 2022-10-11 01:18:37] ...optimizers/trijoint.py.151: total_norm: 11.262124 [I 2022-10-11 01:18:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 1220/2304 [I 2022-10-11 01:18:46] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:41 [I 2022-10-11 01:18:46] ...trap/engines/engine.py.229: process: 0.86585 | load: 0.00315 [I 2022-10-11 01:18:46] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-11 01:18:46] ...optimizers/trijoint.py.151: total_norm: 17.115383 [I 2022-10-11 01:18:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 1230/2304 [I 2022-10-11 01:18:54] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:15:23 [I 2022-10-11 01:18:54] ...trap/engines/engine.py.229: process: 0.85270 | load: 0.00363 [I 2022-10-11 01:18:54] ...trap/engines/engine.py.230: loss: 0.13970 [I 2022-10-11 01:18:54] ...optimizers/trijoint.py.151: total_norm: 19.230968 [I 2022-10-11 01:19:03] ...trap/engines/engine.py.224: train: epoch 92 | batch 1240/2304 [I 2022-10-11 01:19:03] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:15:20 [I 2022-10-11 01:19:03] ...trap/engines/engine.py.229: process: 0.87920 | load: 0.00310 [I 2022-10-11 01:19:03] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-11 01:19:03] ...optimizers/trijoint.py.151: total_norm: 49.626472 [I 2022-10-11 01:19:12] ...trap/engines/engine.py.224: train: epoch 92 | batch 1250/2304 [I 2022-10-11 01:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:15:12 [I 2022-10-11 01:19:12] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00309 [I 2022-10-11 01:19:12] ...trap/engines/engine.py.230: loss: 0.09065 [I 2022-10-11 01:19:12] ...optimizers/trijoint.py.151: total_norm: 14.345519 [I 2022-10-11 01:19:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1260/2304 [I 2022-10-11 01:19:20] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:15:07 [I 2022-10-11 01:19:20] ...trap/engines/engine.py.229: process: 0.86957 | load: 0.00323 [I 2022-10-11 01:19:20] ...trap/engines/engine.py.230: loss: 0.10752 [I 2022-10-11 01:19:20] ...optimizers/trijoint.py.151: total_norm: 9.925239 [I 2022-10-11 01:19:29] ...trap/engines/engine.py.224: train: epoch 92 | batch 1270/2304 [I 2022-10-11 01:19:29] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:15:01 [I 2022-10-11 01:19:29] ...trap/engines/engine.py.229: process: 0.87165 | load: 0.00308 [I 2022-10-11 01:19:29] ...trap/engines/engine.py.230: loss: 0.16460 [I 2022-10-11 01:19:29] ...optimizers/trijoint.py.151: total_norm: 20.726038 [I 2022-10-11 01:19:38] ...trap/engines/engine.py.224: train: epoch 92 | batch 1280/2304 [I 2022-10-11 01:19:38] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:53 [I 2022-10-11 01:19:38] ...trap/engines/engine.py.229: process: 0.88259 | load: 0.00317 [I 2022-10-11 01:19:38] ...trap/engines/engine.py.230: loss: 0.09389 [I 2022-10-11 01:19:38] ...optimizers/trijoint.py.151: total_norm: 11.155273 [I 2022-10-11 01:19:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 1290/2304 [I 2022-10-11 01:19:46] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:40 [I 2022-10-11 01:19:46] ...trap/engines/engine.py.229: process: 0.87582 | load: 0.00320 [I 2022-10-11 01:19:46] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-11 01:19:46] ...optimizers/trijoint.py.151: total_norm: 12.010874 [I 2022-10-11 01:19:55] ...trap/engines/engine.py.224: train: epoch 92 | batch 1300/2304 [I 2022-10-11 01:19:55] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:14:34 [I 2022-10-11 01:19:55] ...trap/engines/engine.py.229: process: 0.86356 | load: 0.00329 [I 2022-10-11 01:19:55] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-11 01:19:55] ...optimizers/trijoint.py.151: total_norm: 51.210354 [I 2022-10-11 01:20:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 1310/2304 [I 2022-10-11 01:20:04] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:14:19 [I 2022-10-11 01:20:04] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00327 [I 2022-10-11 01:20:04] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-11 01:20:04] ...optimizers/trijoint.py.151: total_norm: 13.570064 [I 2022-10-11 01:20:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 1320/2304 [I 2022-10-11 01:20:13] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:14:11 [I 2022-10-11 01:20:13] ...trap/engines/engine.py.229: process: 0.86187 | load: 0.00329 [I 2022-10-11 01:20:13] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-11 01:20:13] ...optimizers/trijoint.py.151: total_norm: 18.426207 [I 2022-10-11 01:20:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 1330/2304 [I 2022-10-11 01:20:21] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:14:04 [I 2022-10-11 01:20:21] ...trap/engines/engine.py.229: process: 0.87084 | load: 0.00319 [I 2022-10-11 01:20:21] ...trap/engines/engine.py.230: loss: 0.15568 [I 2022-10-11 01:20:21] ...optimizers/trijoint.py.151: total_norm: 14.702026 [I 2022-10-11 01:20:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 1340/2304 [I 2022-10-11 01:20:30] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:52 [I 2022-10-11 01:20:30] ...trap/engines/engine.py.229: process: 0.87350 | load: 0.00321 [I 2022-10-11 01:20:30] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-11 01:20:30] ...optimizers/trijoint.py.151: total_norm: 36.723621 [I 2022-10-11 01:20:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 1350/2304 [I 2022-10-11 01:20:39] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:44 [I 2022-10-11 01:20:39] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00311 [I 2022-10-11 01:20:39] ...trap/engines/engine.py.230: loss: 0.14701 [I 2022-10-11 01:20:39] ...optimizers/trijoint.py.151: total_norm: 26.105297 [I 2022-10-11 01:20:47] ...trap/engines/engine.py.224: train: epoch 92 | batch 1360/2304 [I 2022-10-11 01:20:47] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:13:41 [I 2022-10-11 01:20:47] ...trap/engines/engine.py.229: process: 0.87810 | load: 0.00311 [I 2022-10-11 01:20:47] ...trap/engines/engine.py.230: loss: 0.10168 [I 2022-10-11 01:20:47] ...optimizers/trijoint.py.151: total_norm: 15.233836 [I 2022-10-11 01:20:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 1370/2304 [I 2022-10-11 01:20:56] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:29 [I 2022-10-11 01:20:56] ...trap/engines/engine.py.229: process: 0.86555 | load: 0.00314 [I 2022-10-11 01:20:56] ...trap/engines/engine.py.230: loss: 0.10183 [I 2022-10-11 01:20:56] ...optimizers/trijoint.py.151: total_norm: 14.869645 [I 2022-10-11 01:21:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 1380/2304 [I 2022-10-11 01:21:05] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:13:16 [I 2022-10-11 01:21:05] ...trap/engines/engine.py.229: process: 0.86121 | load: 0.00312 [I 2022-10-11 01:21:05] ...trap/engines/engine.py.230: loss: 0.07664 [I 2022-10-11 01:21:05] ...optimizers/trijoint.py.151: total_norm: 8.866027 [I 2022-10-11 01:21:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 1390/2304 [I 2022-10-11 01:21:13] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:13:12 [I 2022-10-11 01:21:13] ...trap/engines/engine.py.229: process: 0.86177 | load: 0.00316 [I 2022-10-11 01:21:13] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-11 01:21:13] ...optimizers/trijoint.py.151: total_norm: 16.135328 [I 2022-10-11 01:21:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 1400/2304 [I 2022-10-11 01:21:22] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:13:05 [I 2022-10-11 01:21:22] ...trap/engines/engine.py.229: process: 0.88391 | load: 0.00309 [I 2022-10-11 01:21:22] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-11 01:21:22] ...optimizers/trijoint.py.151: total_norm: 28.006205 [I 2022-10-11 01:21:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1410/2304 [I 2022-10-11 01:21:31] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:53 [I 2022-10-11 01:21:31] ...trap/engines/engine.py.229: process: 0.85803 | load: 0.00320 [I 2022-10-11 01:21:31] ...trap/engines/engine.py.230: loss: 0.15777 [I 2022-10-11 01:21:31] ...optimizers/trijoint.py.151: total_norm: 20.828096 [I 2022-10-11 01:21:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 1420/2304 [I 2022-10-11 01:21:39] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:44 [I 2022-10-11 01:21:39] ...trap/engines/engine.py.229: process: 0.87139 | load: 0.00322 [I 2022-10-11 01:21:39] ...trap/engines/engine.py.230: loss: 0.10759 [I 2022-10-11 01:21:39] ...optimizers/trijoint.py.151: total_norm: 15.910699 [I 2022-10-11 01:21:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 1430/2304 [I 2022-10-11 01:21:48] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:31 [I 2022-10-11 01:21:48] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00312 [I 2022-10-11 01:21:48] ...trap/engines/engine.py.230: loss: 0.15001 [I 2022-10-11 01:21:48] ...optimizers/trijoint.py.151: total_norm: 11.790652 [I 2022-10-11 01:21:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 1440/2304 [I 2022-10-11 01:21:57] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:12:24 [I 2022-10-11 01:21:57] ...trap/engines/engine.py.229: process: 0.85849 | load: 0.00315 [I 2022-10-11 01:21:57] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-11 01:21:57] ...optimizers/trijoint.py.151: total_norm: 21.112064 [I 2022-10-11 01:22:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 1450/2304 [I 2022-10-11 01:22:05] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:12:23 [I 2022-10-11 01:22:05] ...trap/engines/engine.py.229: process: 0.86618 | load: 0.00310 [I 2022-10-11 01:22:05] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-11 01:22:05] ...optimizers/trijoint.py.151: total_norm: 16.343182 [I 2022-10-11 01:22:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 1460/2304 [I 2022-10-11 01:22:14] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:12:06 [I 2022-10-11 01:22:14] ...trap/engines/engine.py.229: process: 0.85016 | load: 0.00314 [I 2022-10-11 01:22:14] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-11 01:22:14] ...optimizers/trijoint.py.151: total_norm: 13.286431 [I 2022-10-11 01:22:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 1470/2304 [I 2022-10-11 01:22:23] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:12:01 [I 2022-10-11 01:22:23] ...trap/engines/engine.py.229: process: 0.86251 | load: 0.00302 [I 2022-10-11 01:22:23] ...trap/engines/engine.py.230: loss: 0.10799 [I 2022-10-11 01:22:23] ...optimizers/trijoint.py.151: total_norm: 18.252293 [I 2022-10-11 01:22:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1480/2304 [I 2022-10-11 01:22:31] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:42 [I 2022-10-11 01:22:31] ...trap/engines/engine.py.229: process: 0.85287 | load: 0.00300 [I 2022-10-11 01:22:31] ...trap/engines/engine.py.230: loss: 0.11651 [I 2022-10-11 01:22:31] ...optimizers/trijoint.py.151: total_norm: 27.370056 [I 2022-10-11 01:22:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 1490/2304 [I 2022-10-11 01:22:40] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:34 [I 2022-10-11 01:22:40] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00297 [I 2022-10-11 01:22:40] ...trap/engines/engine.py.230: loss: 0.17577 [I 2022-10-11 01:22:40] ...optimizers/trijoint.py.151: total_norm: 17.801027 [I 2022-10-11 01:22:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 1500/2304 [I 2022-10-11 01:22:48] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:30 [I 2022-10-11 01:22:48] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00297 [I 2022-10-11 01:22:48] ...trap/engines/engine.py.230: loss: 0.08991 [I 2022-10-11 01:22:48] ...optimizers/trijoint.py.151: total_norm: 11.219907 [I 2022-10-11 01:22:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 1510/2304 [I 2022-10-11 01:22:57] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:11:18 [I 2022-10-11 01:22:57] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00295 [I 2022-10-11 01:22:57] ...trap/engines/engine.py.230: loss: 0.13846 [I 2022-10-11 01:22:57] ...optimizers/trijoint.py.151: total_norm: 16.925285 [I 2022-10-11 01:23:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 1520/2304 [I 2022-10-11 01:23:05] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:11:09 [I 2022-10-11 01:23:05] ...trap/engines/engine.py.229: process: 0.86168 | load: 0.00301 [I 2022-10-11 01:23:06] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-11 01:23:06] ...optimizers/trijoint.py.151: total_norm: 8.842610 [I 2022-10-11 01:23:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 1530/2304 [I 2022-10-11 01:23:14] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:59 [I 2022-10-11 01:23:14] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00299 [I 2022-10-11 01:23:14] ...trap/engines/engine.py.230: loss: 0.12676 [I 2022-10-11 01:23:14] ...optimizers/trijoint.py.151: total_norm: 12.718872 [I 2022-10-11 01:23:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 1540/2304 [I 2022-10-11 01:23:23] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:53 [I 2022-10-11 01:23:23] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00312 [I 2022-10-11 01:23:23] ...trap/engines/engine.py.230: loss: 0.17667 [I 2022-10-11 01:23:23] ...optimizers/trijoint.py.151: total_norm: 15.767730 [I 2022-10-11 01:23:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1550/2304 [I 2022-10-11 01:23:31] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:44 [I 2022-10-11 01:23:31] ...trap/engines/engine.py.229: process: 0.87041 | load: 0.00296 [I 2022-10-11 01:23:31] ...trap/engines/engine.py.230: loss: 0.12889 [I 2022-10-11 01:23:31] ...optimizers/trijoint.py.151: total_norm: 28.349813 [I 2022-10-11 01:23:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 1560/2304 [I 2022-10-11 01:23:40] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:39 [I 2022-10-11 01:23:40] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00296 [I 2022-10-11 01:23:40] ...trap/engines/engine.py.230: loss: 0.17384 [I 2022-10-11 01:23:40] ...optimizers/trijoint.py.151: total_norm: 18.502594 [I 2022-10-11 01:23:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 1570/2304 [I 2022-10-11 01:23:48] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:10:28 [I 2022-10-11 01:23:48] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00308 [I 2022-10-11 01:23:48] ...trap/engines/engine.py.230: loss: 0.14147 [I 2022-10-11 01:23:48] ...optimizers/trijoint.py.151: total_norm: 22.467039 [I 2022-10-11 01:23:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 1580/2304 [I 2022-10-11 01:23:57] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:10:19 [I 2022-10-11 01:23:57] ...trap/engines/engine.py.229: process: 0.85226 | load: 0.00297 [I 2022-10-11 01:23:57] ...trap/engines/engine.py.230: loss: 0.12145 [I 2022-10-11 01:23:57] ...optimizers/trijoint.py.151: total_norm: 10.798493 [I 2022-10-11 01:24:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 1590/2304 [I 2022-10-11 01:24:05] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:10:09 [I 2022-10-11 01:24:05] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00311 [I 2022-10-11 01:24:05] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-11 01:24:05] ...optimizers/trijoint.py.151: total_norm: 14.577194 [I 2022-10-11 01:24:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 1600/2304 [I 2022-10-11 01:24:14] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:10:02 [I 2022-10-11 01:24:14] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00315 [I 2022-10-11 01:24:14] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-11 01:24:14] ...optimizers/trijoint.py.151: total_norm: 23.578053 [I 2022-10-11 01:24:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 1610/2304 [I 2022-10-11 01:24:23] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:59 [I 2022-10-11 01:24:23] ...trap/engines/engine.py.229: process: 0.86928 | load: 0.00334 [I 2022-10-11 01:24:23] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-11 01:24:23] ...optimizers/trijoint.py.151: total_norm: 15.039142 [I 2022-10-11 01:24:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1620/2304 [I 2022-10-11 01:24:31] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:55 [I 2022-10-11 01:24:31] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00447 [I 2022-10-11 01:24:31] ...trap/engines/engine.py.230: loss: 0.10102 [I 2022-10-11 01:24:31] ...optimizers/trijoint.py.151: total_norm: 14.936136 [I 2022-10-11 01:24:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 1630/2304 [I 2022-10-11 01:24:40] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:41 [I 2022-10-11 01:24:40] ...trap/engines/engine.py.229: process: 0.87963 | load: 0.00327 [I 2022-10-11 01:24:40] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-11 01:24:40] ...optimizers/trijoint.py.151: total_norm: 10.180480 [I 2022-10-11 01:24:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 1640/2304 [I 2022-10-11 01:24:49] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:09:35 [I 2022-10-11 01:24:49] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00310 [I 2022-10-11 01:24:49] ...trap/engines/engine.py.230: loss: 0.11426 [I 2022-10-11 01:24:49] ...optimizers/trijoint.py.151: total_norm: 13.249282 [I 2022-10-11 01:24:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 1650/2304 [I 2022-10-11 01:24:58] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:09:25 [I 2022-10-11 01:24:58] ...trap/engines/engine.py.229: process: 0.86434 | load: 0.00310 [I 2022-10-11 01:24:58] ...trap/engines/engine.py.230: loss: 0.10961 [I 2022-10-11 01:24:58] ...optimizers/trijoint.py.151: total_norm: 13.775417 [I 2022-10-11 01:25:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 1660/2304 [I 2022-10-11 01:25:06] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:09:15 [I 2022-10-11 01:25:06] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00328 [I 2022-10-11 01:25:06] ...trap/engines/engine.py.230: loss: 0.14385 [I 2022-10-11 01:25:06] ...optimizers/trijoint.py.151: total_norm: 26.485575 [I 2022-10-11 01:25:15] ...trap/engines/engine.py.224: train: epoch 92 | batch 1670/2304 [I 2022-10-11 01:25:15] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:09:07 [I 2022-10-11 01:25:15] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00317 [I 2022-10-11 01:25:15] ...trap/engines/engine.py.230: loss: 0.10979 [I 2022-10-11 01:25:15] ...optimizers/trijoint.py.151: total_norm: 9.862010 [I 2022-10-11 01:25:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 1680/2304 [I 2022-10-11 01:25:23] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:58 [I 2022-10-11 01:25:23] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00314 [I 2022-10-11 01:25:23] ...trap/engines/engine.py.230: loss: 0.18706 [I 2022-10-11 01:25:23] ...optimizers/trijoint.py.151: total_norm: 64.342117 [I 2022-10-11 01:25:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 1690/2304 [I 2022-10-11 01:25:32] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:52 [I 2022-10-11 01:25:32] ...trap/engines/engine.py.229: process: 0.85571 | load: 0.00314 [I 2022-10-11 01:25:32] ...trap/engines/engine.py.230: loss: 0.12146 [I 2022-10-11 01:25:32] ...optimizers/trijoint.py.151: total_norm: 15.126043 [I 2022-10-11 01:25:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 1700/2304 [I 2022-10-11 01:25:41] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:40 [I 2022-10-11 01:25:41] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00312 [I 2022-10-11 01:25:41] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-11 01:25:41] ...optimizers/trijoint.py.151: total_norm: 16.676201 [I 2022-10-11 01:25:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 1710/2304 [I 2022-10-11 01:25:49] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:32 [I 2022-10-11 01:25:49] ...trap/engines/engine.py.229: process: 0.86936 | load: 0.00320 [I 2022-10-11 01:25:49] ...trap/engines/engine.py.230: loss: 0.15537 [I 2022-10-11 01:25:49] ...optimizers/trijoint.py.151: total_norm: 13.577752 [I 2022-10-11 01:25:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 1720/2304 [I 2022-10-11 01:25:58] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:28 [I 2022-10-11 01:25:58] ...trap/engines/engine.py.229: process: 0.89439 | load: 0.00315 [I 2022-10-11 01:25:58] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-11 01:25:58] ...optimizers/trijoint.py.151: total_norm: 20.140364 [I 2022-10-11 01:26:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 1730/2304 [I 2022-10-11 01:26:07] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:08:17 [I 2022-10-11 01:26:07] ...trap/engines/engine.py.229: process: 0.86361 | load: 0.00319 [I 2022-10-11 01:26:07] ...trap/engines/engine.py.230: loss: 0.10228 [I 2022-10-11 01:26:07] ...optimizers/trijoint.py.151: total_norm: 11.146526 [I 2022-10-11 01:26:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 1740/2304 [I 2022-10-11 01:26:16] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:08:09 [I 2022-10-11 01:26:16] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00313 [I 2022-10-11 01:26:16] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-11 01:26:16] ...optimizers/trijoint.py.151: total_norm: 17.353725 [I 2022-10-11 01:26:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 1750/2304 [I 2022-10-11 01:26:24] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:57 [I 2022-10-11 01:26:24] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00313 [I 2022-10-11 01:26:24] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-11 01:26:24] ...optimizers/trijoint.py.151: total_norm: 18.318920 [I 2022-10-11 01:26:33] ...trap/engines/engine.py.224: train: epoch 92 | batch 1760/2304 [I 2022-10-11 01:26:33] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:44 [I 2022-10-11 01:26:33] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00310 [I 2022-10-11 01:26:33] ...trap/engines/engine.py.230: loss: 0.15984 [I 2022-10-11 01:26:33] ...optimizers/trijoint.py.151: total_norm: 32.448421 [I 2022-10-11 01:26:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 1770/2304 [I 2022-10-11 01:26:42] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:46 [I 2022-10-11 01:26:42] ...trap/engines/engine.py.229: process: 0.87410 | load: 0.00316 [I 2022-10-11 01:26:42] ...trap/engines/engine.py.230: loss: 0.09094 [I 2022-10-11 01:26:42] ...optimizers/trijoint.py.151: total_norm: 15.743271 [I 2022-10-11 01:26:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 1780/2304 [I 2022-10-11 01:26:50] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:34 [I 2022-10-11 01:26:50] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00310 [I 2022-10-11 01:26:50] ...trap/engines/engine.py.230: loss: 0.09321 [I 2022-10-11 01:26:50] ...optimizers/trijoint.py.151: total_norm: 10.288390 [I 2022-10-11 01:26:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 1790/2304 [I 2022-10-11 01:26:59] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:22 [I 2022-10-11 01:26:59] ...trap/engines/engine.py.229: process: 0.87226 | load: 0.00309 [I 2022-10-11 01:26:59] ...trap/engines/engine.py.230: loss: 0.09639 [I 2022-10-11 01:26:59] ...optimizers/trijoint.py.151: total_norm: 15.039232 [I 2022-10-11 01:27:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 1800/2304 [I 2022-10-11 01:27:08] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:07:17 [I 2022-10-11 01:27:08] ...trap/engines/engine.py.229: process: 0.87852 | load: 0.00329 [I 2022-10-11 01:27:08] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-11 01:27:08] ...optimizers/trijoint.py.151: total_norm: 24.962143 [I 2022-10-11 01:27:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 1810/2304 [I 2022-10-11 01:27:16] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:07:06 [I 2022-10-11 01:27:16] ...trap/engines/engine.py.229: process: 0.85479 | load: 0.00313 [I 2022-10-11 01:27:16] ...trap/engines/engine.py.230: loss: 0.11631 [I 2022-10-11 01:27:16] ...optimizers/trijoint.py.151: total_norm: 10.188171 [I 2022-10-11 01:27:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 1820/2304 [I 2022-10-11 01:27:25] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:07:01 [I 2022-10-11 01:27:25] ...trap/engines/engine.py.229: process: 0.86719 | load: 0.00319 [I 2022-10-11 01:27:25] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-11 01:27:25] ...optimizers/trijoint.py.151: total_norm: 23.905182 [I 2022-10-11 01:27:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 1830/2304 [I 2022-10-11 01:27:34] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:51 [I 2022-10-11 01:27:34] ...trap/engines/engine.py.229: process: 0.89139 | load: 0.00312 [I 2022-10-11 01:27:34] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-11 01:27:34] ...optimizers/trijoint.py.151: total_norm: 37.778240 [I 2022-10-11 01:27:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 1840/2304 [I 2022-10-11 01:27:42] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:06:45 [I 2022-10-11 01:27:42] ...trap/engines/engine.py.229: process: 0.86667 | load: 0.00317 [I 2022-10-11 01:27:42] ...trap/engines/engine.py.230: loss: 0.13064 [I 2022-10-11 01:27:42] ...optimizers/trijoint.py.151: total_norm: 10.873930 [I 2022-10-11 01:27:51] ...trap/engines/engine.py.224: train: epoch 92 | batch 1850/2304 [I 2022-10-11 01:27:51] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:33 [I 2022-10-11 01:27:51] ...trap/engines/engine.py.229: process: 0.87535 | load: 0.00317 [I 2022-10-11 01:27:51] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-11 01:27:51] ...optimizers/trijoint.py.151: total_norm: 13.232990 [I 2022-10-11 01:28:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 1860/2304 [I 2022-10-11 01:28:00] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:06:25 [I 2022-10-11 01:28:00] ...trap/engines/engine.py.229: process: 0.87317 | load: 0.00313 [I 2022-10-11 01:28:00] ...trap/engines/engine.py.230: loss: 0.08571 [I 2022-10-11 01:28:00] ...optimizers/trijoint.py.151: total_norm: 10.144880 [I 2022-10-11 01:28:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 1870/2304 [I 2022-10-11 01:28:08] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:06:14 [I 2022-10-11 01:28:08] ...trap/engines/engine.py.229: process: 0.86135 | load: 0.00308 [I 2022-10-11 01:28:08] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-11 01:28:08] ...optimizers/trijoint.py.151: total_norm: 46.855419 [I 2022-10-11 01:28:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 1880/2304 [I 2022-10-11 01:28:17] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:06:05 [I 2022-10-11 01:28:17] ...trap/engines/engine.py.229: process: 0.85634 | load: 0.00306 [I 2022-10-11 01:28:17] ...trap/engines/engine.py.230: loss: 0.12843 [I 2022-10-11 01:28:17] ...optimizers/trijoint.py.151: total_norm: 20.062521 [I 2022-10-11 01:28:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 1890/2304 [I 2022-10-11 01:28:26] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:55 [I 2022-10-11 01:28:26] ...trap/engines/engine.py.229: process: 0.85745 | load: 0.00298 [I 2022-10-11 01:28:26] ...trap/engines/engine.py.230: loss: 0.16470 [I 2022-10-11 01:28:26] ...optimizers/trijoint.py.151: total_norm: 18.743221 [I 2022-10-11 01:28:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 1900/2304 [I 2022-10-11 01:28:34] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:45 [I 2022-10-11 01:28:34] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00296 [I 2022-10-11 01:28:34] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-11 01:28:34] ...optimizers/trijoint.py.151: total_norm: 13.024896 [I 2022-10-11 01:28:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 1910/2304 [I 2022-10-11 01:28:43] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:05:37 [I 2022-10-11 01:28:43] ...trap/engines/engine.py.229: process: 0.83642 | load: 0.00296 [I 2022-10-11 01:28:43] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-11 01:28:43] ...optimizers/trijoint.py.151: total_norm: 18.341961 [I 2022-10-11 01:28:51] ...trap/engines/engine.py.224: train: epoch 92 | batch 1920/2304 [I 2022-10-11 01:28:51] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:05:28 [I 2022-10-11 01:28:51] ...trap/engines/engine.py.229: process: 0.85810 | load: 0.00333 [I 2022-10-11 01:28:51] ...trap/engines/engine.py.230: loss: 0.18234 [I 2022-10-11 01:28:51] ...optimizers/trijoint.py.151: total_norm: 34.889709 [I 2022-10-11 01:29:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 1930/2304 [I 2022-10-11 01:29:00] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:05:18 [I 2022-10-11 01:29:00] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00338 [I 2022-10-11 01:29:00] ...trap/engines/engine.py.230: loss: 0.15654 [I 2022-10-11 01:29:00] ...optimizers/trijoint.py.151: total_norm: 49.632504 [I 2022-10-11 01:29:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 1940/2304 [I 2022-10-11 01:29:09] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:05:11 [I 2022-10-11 01:29:09] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00304 [I 2022-10-11 01:29:09] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-11 01:29:09] ...optimizers/trijoint.py.151: total_norm: 15.146385 [I 2022-10-11 01:29:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 1950/2304 [I 2022-10-11 01:29:17] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:05:01 [I 2022-10-11 01:29:17] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00302 [I 2022-10-11 01:29:17] ...trap/engines/engine.py.230: loss: 0.09942 [I 2022-10-11 01:29:17] ...optimizers/trijoint.py.151: total_norm: 8.241807 [I 2022-10-11 01:29:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 1960/2304 [I 2022-10-11 01:29:26] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:55 [I 2022-10-11 01:29:26] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00299 [I 2022-10-11 01:29:26] ...trap/engines/engine.py.230: loss: 0.18748 [I 2022-10-11 01:29:26] ...optimizers/trijoint.py.151: total_norm: 22.832869 [I 2022-10-11 01:29:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 1970/2304 [I 2022-10-11 01:29:34] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:45 [I 2022-10-11 01:29:34] ...trap/engines/engine.py.229: process: 0.85825 | load: 0.00301 [I 2022-10-11 01:29:34] ...trap/engines/engine.py.230: loss: 0.10325 [I 2022-10-11 01:29:34] ...optimizers/trijoint.py.151: total_norm: 19.063820 [I 2022-10-11 01:29:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 1980/2304 [I 2022-10-11 01:29:43] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:04:35 [I 2022-10-11 01:29:43] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00301 [I 2022-10-11 01:29:43] ...trap/engines/engine.py.230: loss: 0.12674 [I 2022-10-11 01:29:43] ...optimizers/trijoint.py.151: total_norm: 16.425640 [I 2022-10-11 01:29:51] ...trap/engines/engine.py.224: train: epoch 92 | batch 1990/2304 [I 2022-10-11 01:29:51] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:04:27 [I 2022-10-11 01:29:51] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00305 [I 2022-10-11 01:29:51] ...trap/engines/engine.py.230: loss: 0.13736 [I 2022-10-11 01:29:51] ...optimizers/trijoint.py.151: total_norm: 8.874172 [I 2022-10-11 01:30:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 2000/2304 [I 2022-10-11 01:30:00] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:17 [I 2022-10-11 01:30:00] ...trap/engines/engine.py.229: process: 0.83559 | load: 0.00305 [I 2022-10-11 01:30:00] ...trap/engines/engine.py.230: loss: 0.15117 [I 2022-10-11 01:30:00] ...optimizers/trijoint.py.151: total_norm: 13.090837 [I 2022-10-11 01:30:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 2010/2304 [I 2022-10-11 01:30:08] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:04:11 [I 2022-10-11 01:30:08] ...trap/engines/engine.py.229: process: 0.85966 | load: 0.00295 [I 2022-10-11 01:30:08] ...trap/engines/engine.py.230: loss: 0.09251 [I 2022-10-11 01:30:08] ...optimizers/trijoint.py.151: total_norm: 12.888770 [I 2022-10-11 01:30:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 2020/2304 [I 2022-10-11 01:30:17] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:04:05 [I 2022-10-11 01:30:17] ...trap/engines/engine.py.229: process: 0.86664 | load: 0.00312 [I 2022-10-11 01:30:17] ...trap/engines/engine.py.230: loss: 0.11347 [I 2022-10-11 01:30:17] ...optimizers/trijoint.py.151: total_norm: 9.388889 [I 2022-10-11 01:30:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 2030/2304 [I 2022-10-11 01:30:26] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:57 [I 2022-10-11 01:30:26] ...trap/engines/engine.py.229: process: 0.86539 | load: 0.00318 [I 2022-10-11 01:30:26] ...trap/engines/engine.py.230: loss: 0.09300 [I 2022-10-11 01:30:26] ...optimizers/trijoint.py.151: total_norm: 7.675262 [I 2022-10-11 01:30:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 2040/2304 [I 2022-10-11 01:30:34] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:46 [I 2022-10-11 01:30:34] ...trap/engines/engine.py.229: process: 0.84856 | load: 0.00308 [I 2022-10-11 01:30:34] ...trap/engines/engine.py.230: loss: 0.11022 [I 2022-10-11 01:30:34] ...optimizers/trijoint.py.151: total_norm: 11.280967 [I 2022-10-11 01:30:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 2050/2304 [I 2022-10-11 01:30:43] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:39 [I 2022-10-11 01:30:43] ...trap/engines/engine.py.229: process: 0.86067 | load: 0.00322 [I 2022-10-11 01:30:43] ...trap/engines/engine.py.230: loss: 0.10402 [I 2022-10-11 01:30:43] ...optimizers/trijoint.py.151: total_norm: 17.878687 [I 2022-10-11 01:30:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 2060/2304 [I 2022-10-11 01:30:52] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:30 [I 2022-10-11 01:30:52] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00309 [I 2022-10-11 01:30:52] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-11 01:30:52] ...optimizers/trijoint.py.151: total_norm: 30.530275 [I 2022-10-11 01:31:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 2070/2304 [I 2022-10-11 01:31:00] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:23 [I 2022-10-11 01:31:00] ...trap/engines/engine.py.229: process: 0.89308 | load: 0.00514 [I 2022-10-11 01:31:00] ...trap/engines/engine.py.230: loss: 0.19062 [I 2022-10-11 01:31:00] ...optimizers/trijoint.py.151: total_norm: 40.997337 [I 2022-10-11 01:31:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 2080/2304 [I 2022-10-11 01:31:09] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:03:12 [I 2022-10-11 01:31:09] ...trap/engines/engine.py.229: process: 0.85836 | load: 0.00311 [I 2022-10-11 01:31:09] ...trap/engines/engine.py.230: loss: 0.09126 [I 2022-10-11 01:31:09] ...optimizers/trijoint.py.151: total_norm: 14.680408 [I 2022-10-11 01:31:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 2090/2304 [I 2022-10-11 01:31:18] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:03:06 [I 2022-10-11 01:31:18] ...trap/engines/engine.py.229: process: 0.87280 | load: 0.00324 [I 2022-10-11 01:31:18] ...trap/engines/engine.py.230: loss: 0.12751 [I 2022-10-11 01:31:18] ...optimizers/trijoint.py.151: total_norm: 15.548702 [I 2022-10-11 01:31:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 2100/2304 [I 2022-10-11 01:31:26] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:56 [I 2022-10-11 01:31:26] ...trap/engines/engine.py.229: process: 0.86338 | load: 0.00330 [I 2022-10-11 01:31:26] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-11 01:31:26] ...optimizers/trijoint.py.151: total_norm: 6.357592 [I 2022-10-11 01:31:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 2110/2304 [I 2022-10-11 01:31:35] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:47 [I 2022-10-11 01:31:35] ...trap/engines/engine.py.229: process: 0.87104 | load: 0.00329 [I 2022-10-11 01:31:35] ...trap/engines/engine.py.230: loss: 0.09745 [I 2022-10-11 01:31:35] ...optimizers/trijoint.py.151: total_norm: 9.121552 [I 2022-10-11 01:31:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 2120/2304 [I 2022-10-11 01:31:44] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:39 [I 2022-10-11 01:31:44] ...trap/engines/engine.py.229: process: 0.86948 | load: 0.00315 [I 2022-10-11 01:31:44] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-11 01:31:44] ...optimizers/trijoint.py.151: total_norm: 12.773100 [I 2022-10-11 01:31:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 2130/2304 [I 2022-10-11 01:31:52] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:30 [I 2022-10-11 01:31:52] ...trap/engines/engine.py.229: process: 0.88036 | load: 0.00312 [I 2022-10-11 01:31:52] ...trap/engines/engine.py.230: loss: 0.14638 [I 2022-10-11 01:31:52] ...optimizers/trijoint.py.151: total_norm: 27.130907 [I 2022-10-11 01:32:01] ...trap/engines/engine.py.224: train: epoch 92 | batch 2140/2304 [I 2022-10-11 01:32:01] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:02:22 [I 2022-10-11 01:32:01] ...trap/engines/engine.py.229: process: 0.87274 | load: 0.00315 [I 2022-10-11 01:32:01] ...trap/engines/engine.py.230: loss: 0.09649 [I 2022-10-11 01:32:01] ...optimizers/trijoint.py.151: total_norm: 12.274086 [I 2022-10-11 01:32:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 2150/2304 [I 2022-10-11 01:32:10] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:02:12 [I 2022-10-11 01:32:10] ...trap/engines/engine.py.229: process: 0.86924 | load: 0.00306 [I 2022-10-11 01:32:10] ...trap/engines/engine.py.230: loss: 0.11090 [I 2022-10-11 01:32:10] ...optimizers/trijoint.py.151: total_norm: 16.012215 [I 2022-10-11 01:32:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 2160/2304 [I 2022-10-11 01:32:18] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:02:04 [I 2022-10-11 01:32:18] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00314 [I 2022-10-11 01:32:18] ...trap/engines/engine.py.230: loss: 0.12039 [I 2022-10-11 01:32:18] ...optimizers/trijoint.py.151: total_norm: 23.514221 [I 2022-10-11 01:32:27] ...trap/engines/engine.py.224: train: epoch 92 | batch 2170/2304 [I 2022-10-11 01:32:27] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:56 [I 2022-10-11 01:32:27] ...trap/engines/engine.py.229: process: 0.87293 | load: 0.00321 [I 2022-10-11 01:32:27] ...trap/engines/engine.py.230: loss: 0.10267 [I 2022-10-11 01:32:27] ...optimizers/trijoint.py.151: total_norm: 21.389959 [I 2022-10-11 01:32:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 2180/2304 [I 2022-10-11 01:32:36] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:47 [I 2022-10-11 01:32:36] ...trap/engines/engine.py.229: process: 0.85883 | load: 0.00309 [I 2022-10-11 01:32:36] ...trap/engines/engine.py.230: loss: 0.18241 [I 2022-10-11 01:32:36] ...optimizers/trijoint.py.151: total_norm: 12.893822 [I 2022-10-11 01:32:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 2190/2304 [I 2022-10-11 01:32:44] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:38 [I 2022-10-11 01:32:44] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00306 [I 2022-10-11 01:32:44] ...trap/engines/engine.py.230: loss: 0.15042 [I 2022-10-11 01:32:44] ...optimizers/trijoint.py.151: total_norm: 20.789801 [I 2022-10-11 01:32:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 2200/2304 [I 2022-10-11 01:32:53] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:29 [I 2022-10-11 01:32:53] ...trap/engines/engine.py.229: process: 0.86707 | load: 0.00313 [I 2022-10-11 01:32:53] ...trap/engines/engine.py.230: loss: 0.11502 [I 2022-10-11 01:32:53] ...optimizers/trijoint.py.151: total_norm: 16.220160 [I 2022-10-11 01:33:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 2210/2304 [I 2022-10-11 01:33:02] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:01:21 [I 2022-10-11 01:33:02] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00331 [I 2022-10-11 01:33:02] ...trap/engines/engine.py.230: loss: 0.15670 [I 2022-10-11 01:33:02] ...optimizers/trijoint.py.151: total_norm: 41.246838 [I 2022-10-11 01:33:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 2220/2304 [I 2022-10-11 01:33:10] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:11 [I 2022-10-11 01:33:10] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00314 [I 2022-10-11 01:33:10] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-11 01:33:10] ...optimizers/trijoint.py.151: total_norm: 8.203158 [I 2022-10-11 01:33:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 2230/2304 [I 2022-10-11 01:33:19] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:01:03 [I 2022-10-11 01:33:19] ...trap/engines/engine.py.229: process: 0.86828 | load: 0.00317 [I 2022-10-11 01:33:19] ...trap/engines/engine.py.230: loss: 0.09102 [I 2022-10-11 01:33:19] ...optimizers/trijoint.py.151: total_norm: 10.895393 [I 2022-10-11 01:33:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 2240/2304 [I 2022-10-11 01:33:28] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:55 [I 2022-10-11 01:33:28] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00310 [I 2022-10-11 01:33:28] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-11 01:33:28] ...optimizers/trijoint.py.151: total_norm: 20.754751 [I 2022-10-11 01:33:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 2250/2304 [I 2022-10-11 01:33:36] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:46 [I 2022-10-11 01:33:36] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00310 [I 2022-10-11 01:33:36] ...trap/engines/engine.py.230: loss: 0.25482 [I 2022-10-11 01:33:36] ...optimizers/trijoint.py.151: total_norm: 31.798485 [I 2022-10-11 01:33:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 2260/2304 [I 2022-10-11 01:33:45] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:37 [I 2022-10-11 01:33:45] ...trap/engines/engine.py.229: process: 0.86990 | load: 0.00315 [I 2022-10-11 01:33:45] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-11 01:33:45] ...optimizers/trijoint.py.151: total_norm: 16.566090 [I 2022-10-11 01:33:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 2270/2304 [I 2022-10-11 01:33:53] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:29 [I 2022-10-11 01:33:53] ...trap/engines/engine.py.229: process: 0.88623 | load: 0.00330 [I 2022-10-11 01:33:53] ...trap/engines/engine.py.230: loss: 0.13315 [I 2022-10-11 01:33:53] ...optimizers/trijoint.py.151: total_norm: 20.508141 [I 2022-10-11 01:34:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 2280/2304 [I 2022-10-11 01:34:02] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:20 [I 2022-10-11 01:34:02] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00308 [I 2022-10-11 01:34:02] ...trap/engines/engine.py.230: loss: 0.09112 [I 2022-10-11 01:34:02] ...optimizers/trijoint.py.151: total_norm: 26.427702 [I 2022-10-11 01:34:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 2290/2304 [I 2022-10-11 01:34:11] ...trap/engines/engine.py.228: elapsed: 0:33:18 | left: 0:00:11 [I 2022-10-11 01:34:11] ...trap/engines/engine.py.229: process: 0.84927 | load: 0.00297 [I 2022-10-11 01:34:11] ...trap/engines/engine.py.230: loss: 0.08417 [I 2022-10-11 01:34:11] ...optimizers/trijoint.py.151: total_norm: 11.938204 [I 2022-10-11 01:34:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 2300/2304 [I 2022-10-11 01:34:19] ...trap/engines/engine.py.228: elapsed: 0:33:27 | left: 0:00:03 [I 2022-10-11 01:34:19] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00016 [I 2022-10-11 01:34:19] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-11 01:34:19] ...optimizers/trijoint.py.151: total_norm: 17.300434 [I 2022-10-11 01:34:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 2304/2304 [I 2022-10-11 01:34:23] ...trap/engines/engine.py.228: elapsed: 0:33:30 | left: 0:00:00 [I 2022-10-11 01:34:23] ...trap/engines/engine.py.229: process: 0.82895 | load: 0.00016 [I 2022-10-11 01:34:23] ...trap/engines/engine.py.230: loss: 0.13451 [I 2022-10-11 01:34:23] ...optimizers/trijoint.py.151: total_norm: 13.947804 [S 2022-10-11 01:34:24] ...trap/engines/engine.py.129: train_epoch.epoch: 92 [S 2022-10-11 01:34:24] ...trap/engines/engine.py.129: train_epoch.loss: 0.12676277099391384 [I 2022-10-11 01:34:24] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 01:34:25] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 92 [I 2022-10-11 01:34:25] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 01:34:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 01:34:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 01:34:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 01:34:31] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 01:34:31] ...trap/engines/engine.py.321: eval: epoch 92 | batch 0/510 [I 2022-10-11 01:34:31] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:24 [I 2022-10-11 01:34:31] ...trap/engines/engine.py.326: process: 5.81346 | load: 5.23366 [I 2022-10-11 01:34:37] ...trap/engines/engine.py.321: eval: epoch 92 | batch 10/510 [I 2022-10-11 01:34:37] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:46 [I 2022-10-11 01:34:37] ...trap/engines/engine.py.326: process: 0.61774 | load: 0.00019 [I 2022-10-11 01:34:43] ...trap/engines/engine.py.321: eval: epoch 92 | batch 20/510 [I 2022-10-11 01:34:43] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:05 [I 2022-10-11 01:34:43] ...trap/engines/engine.py.326: process: 0.49883 | load: 0.00019 [I 2022-10-11 01:34:50] ...trap/engines/engine.py.321: eval: epoch 92 | batch 30/510 [I 2022-10-11 01:34:50] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:19 [I 2022-10-11 01:34:50] ...trap/engines/engine.py.326: process: 0.63101 | load: 0.00019 [I 2022-10-11 01:35:01] ...trap/engines/engine.py.321: eval: epoch 92 | batch 40/510 [I 2022-10-11 01:35:01] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:09:47 [I 2022-10-11 01:35:01] ...trap/engines/engine.py.326: process: 0.46901 | load: 0.00020 [I 2022-10-11 01:35:07] ...trap/engines/engine.py.321: eval: epoch 92 | batch 50/510 [I 2022-10-11 01:35:07] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:05:06 [I 2022-10-11 01:35:07] ...trap/engines/engine.py.326: process: 0.51749 | load: 0.00020 [I 2022-10-11 01:35:17] ...trap/engines/engine.py.321: eval: epoch 92 | batch 60/510 [I 2022-10-11 01:35:17] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:08:22 [I 2022-10-11 01:35:17] ...trap/engines/engine.py.326: process: 0.68339 | load: 0.00021 [I 2022-10-11 01:35:23] ...trap/engines/engine.py.321: eval: epoch 92 | batch 70/510 [I 2022-10-11 01:35:23] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:04:40 [I 2022-10-11 01:35:23] ...trap/engines/engine.py.326: process: 0.59772 | load: 0.00019 [I 2022-10-11 01:35:28] ...trap/engines/engine.py.321: eval: epoch 92 | batch 80/510 [I 2022-10-11 01:35:28] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:03:42 [I 2022-10-11 01:35:28] ...trap/engines/engine.py.326: process: 0.48145 | load: 0.00019 [I 2022-10-11 01:35:33] ...trap/engines/engine.py.321: eval: epoch 92 | batch 90/510 [I 2022-10-11 01:35:33] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:03:07 [I 2022-10-11 01:35:33] ...trap/engines/engine.py.326: process: 0.42991 | load: 0.00020 [I 2022-10-11 01:35:37] ...trap/engines/engine.py.321: eval: epoch 92 | batch 100/510 [I 2022-10-11 01:35:37] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:57 [I 2022-10-11 01:35:37] ...trap/engines/engine.py.326: process: 0.31204 | load: 0.00020 [I 2022-10-11 01:35:40] ...trap/engines/engine.py.321: eval: epoch 92 | batch 110/510 [I 2022-10-11 01:35:40] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:02:11 [I 2022-10-11 01:35:40] ...trap/engines/engine.py.326: process: 0.30984 | load: 0.00018 [I 2022-10-11 01:35:43] ...trap/engines/engine.py.321: eval: epoch 92 | batch 120/510 [I 2022-10-11 01:35:43] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:02:03 [I 2022-10-11 01:35:43] ...trap/engines/engine.py.326: process: 0.31518 | load: 0.00019 [I 2022-10-11 01:35:47] ...trap/engines/engine.py.321: eval: epoch 92 | batch 130/510 [I 2022-10-11 01:35:47] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:02:00 [I 2022-10-11 01:35:47] ...trap/engines/engine.py.326: process: 0.31560 | load: 0.00019 [I 2022-10-11 01:35:50] ...trap/engines/engine.py.321: eval: epoch 92 | batch 140/510 [I 2022-10-11 01:35:50] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:57 [I 2022-10-11 01:35:50] ...trap/engines/engine.py.326: process: 0.31777 | load: 0.00019 [I 2022-10-11 01:35:53] ...trap/engines/engine.py.321: eval: epoch 92 | batch 150/510 [I 2022-10-11 01:35:53] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:53 [I 2022-10-11 01:35:53] ...trap/engines/engine.py.326: process: 0.31515 | load: 0.00019 [I 2022-10-11 01:35:56] ...trap/engines/engine.py.321: eval: epoch 92 | batch 160/510 [I 2022-10-11 01:35:56] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:51 [I 2022-10-11 01:35:56] ...trap/engines/engine.py.326: process: 0.32308 | load: 0.00019 [I 2022-10-11 01:35:59] ...trap/engines/engine.py.321: eval: epoch 92 | batch 170/510 [I 2022-10-11 01:35:59] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:48 [I 2022-10-11 01:35:59] ...trap/engines/engine.py.326: process: 0.32161 | load: 0.00020 [I 2022-10-11 01:36:03] ...trap/engines/engine.py.321: eval: epoch 92 | batch 180/510 [I 2022-10-11 01:36:03] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:44 [I 2022-10-11 01:36:03] ...trap/engines/engine.py.326: process: 0.30673 | load: 0.00019 [I 2022-10-11 01:36:06] ...trap/engines/engine.py.321: eval: epoch 92 | batch 190/510 [I 2022-10-11 01:36:06] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:42 [I 2022-10-11 01:36:06] ...trap/engines/engine.py.326: process: 0.32090 | load: 0.00018 [I 2022-10-11 01:36:09] ...trap/engines/engine.py.321: eval: epoch 92 | batch 200/510 [I 2022-10-11 01:36:09] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:38 [I 2022-10-11 01:36:09] ...trap/engines/engine.py.326: process: 0.32234 | load: 0.00019 [I 2022-10-11 01:36:12] ...trap/engines/engine.py.321: eval: epoch 92 | batch 210/510 [I 2022-10-11 01:36:12] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:34 [I 2022-10-11 01:36:12] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00019 [I 2022-10-11 01:36:15] ...trap/engines/engine.py.321: eval: epoch 92 | batch 220/510 [I 2022-10-11 01:36:15] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:31 [I 2022-10-11 01:36:15] ...trap/engines/engine.py.326: process: 0.31641 | load: 0.00019 [I 2022-10-11 01:36:18] ...trap/engines/engine.py.321: eval: epoch 92 | batch 230/510 [I 2022-10-11 01:36:18] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:27 [I 2022-10-11 01:36:18] ...trap/engines/engine.py.326: process: 0.30219 | load: 0.00018 [I 2022-10-11 01:36:22] ...trap/engines/engine.py.321: eval: epoch 92 | batch 240/510 [I 2022-10-11 01:36:22] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:26 [I 2022-10-11 01:36:22] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00020 [I 2022-10-11 01:36:25] ...trap/engines/engine.py.321: eval: epoch 92 | batch 250/510 [I 2022-10-11 01:36:25] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:22 [I 2022-10-11 01:36:25] ...trap/engines/engine.py.326: process: 0.32008 | load: 0.00020 [I 2022-10-11 01:36:28] ...trap/engines/engine.py.321: eval: epoch 92 | batch 260/510 [I 2022-10-11 01:36:28] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:19 [I 2022-10-11 01:36:28] ...trap/engines/engine.py.326: process: 0.31813 | load: 0.00019 [I 2022-10-11 01:36:31] ...trap/engines/engine.py.321: eval: epoch 92 | batch 270/510 [I 2022-10-11 01:36:31] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:16 [I 2022-10-11 01:36:31] ...trap/engines/engine.py.326: process: 0.31569 | load: 0.00018 [I 2022-10-11 01:36:34] ...trap/engines/engine.py.321: eval: epoch 92 | batch 280/510 [I 2022-10-11 01:36:34] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:13 [I 2022-10-11 01:36:34] ...trap/engines/engine.py.326: process: 0.31474 | load: 0.00018 [I 2022-10-11 01:36:38] ...trap/engines/engine.py.321: eval: epoch 92 | batch 290/510 [I 2022-10-11 01:36:38] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:12 [I 2022-10-11 01:36:38] ...trap/engines/engine.py.326: process: 0.32094 | load: 0.00017 [I 2022-10-11 01:36:41] ...trap/engines/engine.py.321: eval: epoch 92 | batch 300/510 [I 2022-10-11 01:36:41] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:07 [I 2022-10-11 01:36:41] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00023 [I 2022-10-11 01:36:44] ...trap/engines/engine.py.321: eval: epoch 92 | batch 310/510 [I 2022-10-11 01:36:44] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:01:04 [I 2022-10-11 01:36:44] ...trap/engines/engine.py.326: process: 0.33084 | load: 0.00018 [I 2022-10-11 01:36:47] ...trap/engines/engine.py.321: eval: epoch 92 | batch 320/510 [I 2022-10-11 01:36:47] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:01:00 [I 2022-10-11 01:36:47] ...trap/engines/engine.py.326: process: 0.32448 | load: 0.00020 [I 2022-10-11 01:36:51] ...trap/engines/engine.py.321: eval: epoch 92 | batch 330/510 [I 2022-10-11 01:36:51] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:57 [I 2022-10-11 01:36:51] ...trap/engines/engine.py.326: process: 0.31827 | load: 0.00021 [I 2022-10-11 01:36:54] ...trap/engines/engine.py.321: eval: epoch 92 | batch 340/510 [I 2022-10-11 01:36:54] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:55 [I 2022-10-11 01:36:54] ...trap/engines/engine.py.326: process: 0.31953 | load: 0.00021 [I 2022-10-11 01:36:57] ...trap/engines/engine.py.321: eval: epoch 92 | batch 350/510 [I 2022-10-11 01:36:57] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:51 [I 2022-10-11 01:36:57] ...trap/engines/engine.py.326: process: 0.31861 | load: 0.00021 [I 2022-10-11 01:37:00] ...trap/engines/engine.py.321: eval: epoch 92 | batch 360/510 [I 2022-10-11 01:37:00] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:48 [I 2022-10-11 01:37:00] ...trap/engines/engine.py.326: process: 0.32142 | load: 0.00020 [I 2022-10-11 01:37:04] ...trap/engines/engine.py.321: eval: epoch 92 | batch 370/510 [I 2022-10-11 01:37:04] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:45 [I 2022-10-11 01:37:04] ...trap/engines/engine.py.326: process: 0.32268 | load: 0.00021 [I 2022-10-11 01:37:07] ...trap/engines/engine.py.321: eval: epoch 92 | batch 380/510 [I 2022-10-11 01:37:07] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:42 [I 2022-10-11 01:37:07] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00021 [I 2022-10-11 01:37:10] ...trap/engines/engine.py.321: eval: epoch 92 | batch 390/510 [I 2022-10-11 01:37:10] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:39 [I 2022-10-11 01:37:10] ...trap/engines/engine.py.326: process: 0.33585 | load: 0.00023 [I 2022-10-11 01:37:13] ...trap/engines/engine.py.321: eval: epoch 92 | batch 400/510 [I 2022-10-11 01:37:13] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:35 [I 2022-10-11 01:37:13] ...trap/engines/engine.py.326: process: 0.32801 | load: 0.00021 [I 2022-10-11 01:37:17] ...trap/engines/engine.py.321: eval: epoch 92 | batch 410/510 [I 2022-10-11 01:37:17] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:32 [I 2022-10-11 01:37:17] ...trap/engines/engine.py.326: process: 0.32906 | load: 0.00023 [I 2022-10-11 01:37:20] ...trap/engines/engine.py.321: eval: epoch 92 | batch 420/510 [I 2022-10-11 01:37:20] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:29 [I 2022-10-11 01:37:20] ...trap/engines/engine.py.326: process: 0.32048 | load: 0.00020 [I 2022-10-11 01:37:23] ...trap/engines/engine.py.321: eval: epoch 92 | batch 430/510 [I 2022-10-11 01:37:23] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:26 [I 2022-10-11 01:37:23] ...trap/engines/engine.py.326: process: 0.32515 | load: 0.00026 [I 2022-10-11 01:37:26] ...trap/engines/engine.py.321: eval: epoch 92 | batch 440/510 [I 2022-10-11 01:37:26] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:22 [I 2022-10-11 01:37:26] ...trap/engines/engine.py.326: process: 0.32484 | load: 0.00020 [I 2022-10-11 01:37:30] ...trap/engines/engine.py.321: eval: epoch 92 | batch 450/510 [I 2022-10-11 01:37:30] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:19 [I 2022-10-11 01:37:30] ...trap/engines/engine.py.326: process: 0.32410 | load: 0.00022 [I 2022-10-11 01:37:33] ...trap/engines/engine.py.321: eval: epoch 92 | batch 460/510 [I 2022-10-11 01:37:33] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:16 [I 2022-10-11 01:37:33] ...trap/engines/engine.py.326: process: 0.32696 | load: 0.00023 [I 2022-10-11 01:37:36] ...trap/engines/engine.py.321: eval: epoch 92 | batch 470/510 [I 2022-10-11 01:37:36] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:12 [I 2022-10-11 01:37:36] ...trap/engines/engine.py.326: process: 0.32172 | load: 0.00018 [I 2022-10-11 01:37:39] ...trap/engines/engine.py.321: eval: epoch 92 | batch 480/510 [I 2022-10-11 01:37:39] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:09 [I 2022-10-11 01:37:39] ...trap/engines/engine.py.326: process: 0.32717 | load: 0.00019 [I 2022-10-11 01:37:43] ...trap/engines/engine.py.321: eval: epoch 92 | batch 490/510 [I 2022-10-11 01:37:43] ...trap/engines/engine.py.325: elapsed: 0:03:17 | left: 0:00:06 [I 2022-10-11 01:37:43] ...trap/engines/engine.py.326: process: 0.32719 | load: 0.00021 [I 2022-10-11 01:37:46] ...trap/engines/engine.py.321: eval: epoch 92 | batch 500/510 [I 2022-10-11 01:37:46] ...trap/engines/engine.py.325: elapsed: 0:03:20 | left: 0:00:03 [I 2022-10-11 01:37:46] ...trap/engines/engine.py.326: process: 0.31425 | load: 0.00022 [I 2022-10-11 01:37:49] ...trap/engines/engine.py.321: eval: epoch 92 | batch 510/510 [I 2022-10-11 01:37:49] ...trap/engines/engine.py.325: elapsed: 0:03:23 | left: 0:00:00 [I 2022-10-11 01:37:49] ...trap/engines/engine.py.326: process: 0.31668 | load: 0.00017 [S 2022-10-11 01:37:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 92 [I 2022-10-11 01:37:50] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.749700 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.914900 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938500 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.745100 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.913200 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937800 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007457 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005770 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005084 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010319 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007574 [S 2022-10-11 01:38:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005828 [I 2022-10-11 01:38:37] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-11 01:38:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 01:38:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 01:38:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 01:38:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 01:38:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 01:38:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 01:38:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 01:38:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 01:38:48] ...trap/engines/engine.py.470: Saving model... [I 2022-10-11 01:38:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 01:38:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 01:38:58] ...trap/engines/engine.py.163: Training model on trainset for epoch 93 [I 2022-10-11 01:38:58] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 01:39:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 0/2304 [I 2022-10-11 01:39:01] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:06:54 [I 2022-10-11 01:39:01] ...trap/engines/engine.py.229: process: 3.30472 | load: 2.36838 [I 2022-10-11 01:39:01] ...trap/engines/engine.py.230: loss: 0.16960 [I 2022-10-11 01:39:01] ...optimizers/trijoint.py.151: total_norm: 16.438177 [I 2022-10-11 01:39:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 10/2304 [I 2022-10-11 01:39:10] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:06 [I 2022-10-11 01:39:10] ...trap/engines/engine.py.229: process: 0.86225 | load: 0.00519 [I 2022-10-11 01:39:10] ...trap/engines/engine.py.230: loss: 0.18285 [I 2022-10-11 01:39:10] ...optimizers/trijoint.py.151: total_norm: 25.848316 [I 2022-10-11 01:39:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 20/2304 [I 2022-10-11 01:39:19] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:57 [I 2022-10-11 01:39:19] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00507 [I 2022-10-11 01:39:19] ...trap/engines/engine.py.230: loss: 0.19806 [I 2022-10-11 01:39:19] ...optimizers/trijoint.py.151: total_norm: 27.320772 [I 2022-10-11 01:39:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 30/2304 [I 2022-10-11 01:39:27] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:56 [I 2022-10-11 01:39:27] ...trap/engines/engine.py.229: process: 0.87672 | load: 0.00358 [I 2022-10-11 01:39:27] ...trap/engines/engine.py.230: loss: 0.10810 [I 2022-10-11 01:39:27] ...optimizers/trijoint.py.151: total_norm: 12.082699 [I 2022-10-11 01:39:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 40/2304 [I 2022-10-11 01:39:36] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:46 [I 2022-10-11 01:39:36] ...trap/engines/engine.py.229: process: 0.87714 | load: 0.00392 [I 2022-10-11 01:39:36] ...trap/engines/engine.py.230: loss: 0.08425 [I 2022-10-11 01:39:36] ...optimizers/trijoint.py.151: total_norm: 27.879347 [I 2022-10-11 01:39:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 50/2304 [I 2022-10-11 01:39:45] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:38 [I 2022-10-11 01:39:45] ...trap/engines/engine.py.229: process: 0.87793 | load: 0.00354 [I 2022-10-11 01:39:45] ...trap/engines/engine.py.230: loss: 0.15533 [I 2022-10-11 01:39:45] ...optimizers/trijoint.py.151: total_norm: 8.003847 [I 2022-10-11 01:39:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 60/2304 [I 2022-10-11 01:39:54] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:17 [I 2022-10-11 01:39:54] ...trap/engines/engine.py.229: process: 0.86988 | load: 0.00346 [I 2022-10-11 01:39:54] ...trap/engines/engine.py.230: loss: 0.11702 [I 2022-10-11 01:39:54] ...optimizers/trijoint.py.151: total_norm: 16.452076 [I 2022-10-11 01:40:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 70/2304 [I 2022-10-11 01:40:02] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:06 [I 2022-10-11 01:40:02] ...trap/engines/engine.py.229: process: 0.85035 | load: 0.00343 [I 2022-10-11 01:40:02] ...trap/engines/engine.py.230: loss: 0.13148 [I 2022-10-11 01:40:02] ...optimizers/trijoint.py.151: total_norm: 17.230146 [I 2022-10-11 01:40:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 80/2304 [I 2022-10-11 01:40:11] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:02 [I 2022-10-11 01:40:11] ...trap/engines/engine.py.229: process: 0.88021 | load: 0.00368 [I 2022-10-11 01:40:11] ...trap/engines/engine.py.230: loss: 0.11886 [I 2022-10-11 01:40:11] ...optimizers/trijoint.py.151: total_norm: 37.493614 [I 2022-10-11 01:40:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 90/2304 [I 2022-10-11 01:40:19] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:42 [I 2022-10-11 01:40:19] ...trap/engines/engine.py.229: process: 0.86152 | load: 0.00335 [I 2022-10-11 01:40:19] ...trap/engines/engine.py.230: loss: 0.10394 [I 2022-10-11 01:40:19] ...optimizers/trijoint.py.151: total_norm: 14.721714 [I 2022-10-11 01:40:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 100/2304 [I 2022-10-11 01:40:28] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:44 [I 2022-10-11 01:40:28] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00335 [I 2022-10-11 01:40:28] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-11 01:40:28] ...optimizers/trijoint.py.151: total_norm: 18.667048 [I 2022-10-11 01:40:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 110/2304 [I 2022-10-11 01:40:37] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:29 [I 2022-10-11 01:40:37] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00349 [I 2022-10-11 01:40:37] ...trap/engines/engine.py.230: loss: 0.08752 [I 2022-10-11 01:40:37] ...optimizers/trijoint.py.151: total_norm: 12.190158 [I 2022-10-11 01:40:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 120/2304 [I 2022-10-11 01:40:45] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:30:53 [I 2022-10-11 01:40:45] ...trap/engines/engine.py.229: process: 0.85525 | load: 0.00350 [I 2022-10-11 01:40:45] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-11 01:40:45] ...optimizers/trijoint.py.151: total_norm: 22.937761 [I 2022-10-11 01:40:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 130/2304 [I 2022-10-11 01:40:54] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:13 [I 2022-10-11 01:40:54] ...trap/engines/engine.py.229: process: 0.87887 | load: 0.00305 [I 2022-10-11 01:40:54] ...trap/engines/engine.py.230: loss: 0.16140 [I 2022-10-11 01:40:54] ...optimizers/trijoint.py.151: total_norm: 25.729778 [I 2022-10-11 01:41:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 140/2304 [I 2022-10-11 01:41:02] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:07 [I 2022-10-11 01:41:02] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00342 [I 2022-10-11 01:41:02] ...trap/engines/engine.py.230: loss: 0.10255 [I 2022-10-11 01:41:02] ...optimizers/trijoint.py.151: total_norm: 19.615540 [I 2022-10-11 01:41:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 150/2304 [I 2022-10-11 01:41:11] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:45 [I 2022-10-11 01:41:11] ...trap/engines/engine.py.229: process: 0.84156 | load: 0.00330 [I 2022-10-11 01:41:11] ...trap/engines/engine.py.230: loss: 0.20069 [I 2022-10-11 01:41:11] ...optimizers/trijoint.py.151: total_norm: 12.976032 [I 2022-10-11 01:41:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 160/2304 [I 2022-10-11 01:41:20] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:39 [I 2022-10-11 01:41:20] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00292 [I 2022-10-11 01:41:20] ...trap/engines/engine.py.230: loss: 0.11983 [I 2022-10-11 01:41:20] ...optimizers/trijoint.py.151: total_norm: 21.553053 [I 2022-10-11 01:41:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 170/2304 [I 2022-10-11 01:41:28] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:15 [I 2022-10-11 01:41:28] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00314 [I 2022-10-11 01:41:28] ...trap/engines/engine.py.230: loss: 0.20569 [I 2022-10-11 01:41:28] ...optimizers/trijoint.py.151: total_norm: 14.618811 [I 2022-10-11 01:41:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 180/2304 [I 2022-10-11 01:41:37] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:08 [I 2022-10-11 01:41:37] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00312 [I 2022-10-11 01:41:37] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-11 01:41:37] ...optimizers/trijoint.py.151: total_norm: 14.038829 [I 2022-10-11 01:41:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 190/2304 [I 2022-10-11 01:41:45] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:29:53 [I 2022-10-11 01:41:45] ...trap/engines/engine.py.229: process: 0.85461 | load: 0.00305 [I 2022-10-11 01:41:45] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-11 01:41:45] ...optimizers/trijoint.py.151: total_norm: 24.620432 [I 2022-10-11 01:41:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 200/2304 [I 2022-10-11 01:41:54] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:29:45 [I 2022-10-11 01:41:54] ...trap/engines/engine.py.229: process: 0.85138 | load: 0.00312 [I 2022-10-11 01:41:54] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-11 01:41:54] ...optimizers/trijoint.py.151: total_norm: 17.559500 [I 2022-10-11 01:42:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 210/2304 [I 2022-10-11 01:42:02] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:29:37 [I 2022-10-11 01:42:02] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00319 [I 2022-10-11 01:42:02] ...trap/engines/engine.py.230: loss: 0.26449 [I 2022-10-11 01:42:02] ...optimizers/trijoint.py.151: total_norm: 16.962166 [I 2022-10-11 01:42:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 220/2304 [I 2022-10-11 01:42:11] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:36 [I 2022-10-11 01:42:11] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00321 [I 2022-10-11 01:42:11] ...trap/engines/engine.py.230: loss: 0.15668 [I 2022-10-11 01:42:11] ...optimizers/trijoint.py.151: total_norm: 20.666826 [I 2022-10-11 01:42:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 230/2304 [I 2022-10-11 01:42:19] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:24 [I 2022-10-11 01:42:19] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00317 [I 2022-10-11 01:42:19] ...trap/engines/engine.py.230: loss: 0.13102 [I 2022-10-11 01:42:19] ...optimizers/trijoint.py.151: total_norm: 33.270042 [I 2022-10-11 01:42:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 240/2304 [I 2022-10-11 01:42:28] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:34 [I 2022-10-11 01:42:28] ...trap/engines/engine.py.229: process: 0.86661 | load: 0.00294 [I 2022-10-11 01:42:28] ...trap/engines/engine.py.230: loss: 0.18103 [I 2022-10-11 01:42:28] ...optimizers/trijoint.py.151: total_norm: 16.564905 [I 2022-10-11 01:42:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 250/2304 [I 2022-10-11 01:42:36] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:10 [I 2022-10-11 01:42:36] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00301 [I 2022-10-11 01:42:36] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-11 01:42:36] ...optimizers/trijoint.py.151: total_norm: 11.614991 [I 2022-10-11 01:42:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 260/2304 [I 2022-10-11 01:42:45] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:22 [I 2022-10-11 01:42:45] ...trap/engines/engine.py.229: process: 0.86848 | load: 0.00303 [I 2022-10-11 01:42:45] ...trap/engines/engine.py.230: loss: 0.13531 [I 2022-10-11 01:42:45] ...optimizers/trijoint.py.151: total_norm: 17.130070 [I 2022-10-11 01:42:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 270/2304 [I 2022-10-11 01:42:54] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:29:03 [I 2022-10-11 01:42:54] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00314 [I 2022-10-11 01:42:54] ...trap/engines/engine.py.230: loss: 0.10851 [I 2022-10-11 01:42:54] ...optimizers/trijoint.py.151: total_norm: 21.261349 [I 2022-10-11 01:43:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 280/2304 [I 2022-10-11 01:43:02] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:28:52 [I 2022-10-11 01:43:02] ...trap/engines/engine.py.229: process: 0.85469 | load: 0.00313 [I 2022-10-11 01:43:02] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-11 01:43:02] ...optimizers/trijoint.py.151: total_norm: 15.674009 [I 2022-10-11 01:43:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 290/2304 [I 2022-10-11 01:43:11] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:44 [I 2022-10-11 01:43:11] ...trap/engines/engine.py.229: process: 0.85984 | load: 0.00308 [I 2022-10-11 01:43:11] ...trap/engines/engine.py.230: loss: 0.18306 [I 2022-10-11 01:43:11] ...optimizers/trijoint.py.151: total_norm: 30.962460 [I 2022-10-11 01:43:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 300/2304 [I 2022-10-11 01:43:19] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:43 [I 2022-10-11 01:43:19] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00316 [I 2022-10-11 01:43:19] ...trap/engines/engine.py.230: loss: 0.13130 [I 2022-10-11 01:43:20] ...optimizers/trijoint.py.151: total_norm: 34.968109 [I 2022-10-11 01:43:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 310/2304 [I 2022-10-11 01:43:28] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:34 [I 2022-10-11 01:43:28] ...trap/engines/engine.py.229: process: 0.86614 | load: 0.00317 [I 2022-10-11 01:43:28] ...trap/engines/engine.py.230: loss: 0.08978 [I 2022-10-11 01:43:28] ...optimizers/trijoint.py.151: total_norm: 12.347474 [I 2022-10-11 01:43:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 320/2304 [I 2022-10-11 01:43:37] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:35 [I 2022-10-11 01:43:37] ...trap/engines/engine.py.229: process: 0.87967 | load: 0.00308 [I 2022-10-11 01:43:37] ...trap/engines/engine.py.230: loss: 0.19101 [I 2022-10-11 01:43:37] ...optimizers/trijoint.py.151: total_norm: 16.081375 [I 2022-10-11 01:43:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 330/2304 [I 2022-10-11 01:43:45] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:28:17 [I 2022-10-11 01:43:45] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00341 [I 2022-10-11 01:43:45] ...trap/engines/engine.py.230: loss: 0.12425 [I 2022-10-11 01:43:45] ...optimizers/trijoint.py.151: total_norm: 21.852510 [I 2022-10-11 01:43:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 340/2304 [I 2022-10-11 01:43:54] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:27 [I 2022-10-11 01:43:54] ...trap/engines/engine.py.229: process: 0.88737 | load: 0.00329 [I 2022-10-11 01:43:54] ...trap/engines/engine.py.230: loss: 0.10746 [I 2022-10-11 01:43:54] ...optimizers/trijoint.py.151: total_norm: 16.011570 [I 2022-10-11 01:44:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 350/2304 [I 2022-10-11 01:44:03] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:01 [I 2022-10-11 01:44:03] ...trap/engines/engine.py.229: process: 0.86394 | load: 0.00310 [I 2022-10-11 01:44:03] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-11 01:44:03] ...optimizers/trijoint.py.151: total_norm: 18.628532 [I 2022-10-11 01:44:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 360/2304 [I 2022-10-11 01:44:11] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:28:07 [I 2022-10-11 01:44:11] ...trap/engines/engine.py.229: process: 0.87697 | load: 0.00319 [I 2022-10-11 01:44:11] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-11 01:44:11] ...optimizers/trijoint.py.151: total_norm: 15.512389 [I 2022-10-11 01:44:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 370/2304 [I 2022-10-11 01:44:20] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:56 [I 2022-10-11 01:44:20] ...trap/engines/engine.py.229: process: 0.87760 | load: 0.00328 [I 2022-10-11 01:44:20] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-11 01:44:20] ...optimizers/trijoint.py.151: total_norm: 17.675106 [I 2022-10-11 01:44:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 380/2304 [I 2022-10-11 01:44:29] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:37 [I 2022-10-11 01:44:29] ...trap/engines/engine.py.229: process: 0.85964 | load: 0.00341 [I 2022-10-11 01:44:29] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-11 01:44:29] ...optimizers/trijoint.py.151: total_norm: 19.226416 [I 2022-10-11 01:44:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 390/2304 [I 2022-10-11 01:44:37] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:36 [I 2022-10-11 01:44:37] ...trap/engines/engine.py.229: process: 0.87826 | load: 0.00312 [I 2022-10-11 01:44:37] ...trap/engines/engine.py.230: loss: 0.15895 [I 2022-10-11 01:44:37] ...optimizers/trijoint.py.151: total_norm: 19.409227 [I 2022-10-11 01:44:46] ...trap/engines/engine.py.224: train: epoch 93 | batch 400/2304 [I 2022-10-11 01:44:46] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:27:27 [I 2022-10-11 01:44:46] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00319 [I 2022-10-11 01:44:46] ...trap/engines/engine.py.230: loss: 0.07723 [I 2022-10-11 01:44:46] ...optimizers/trijoint.py.151: total_norm: 10.044039 [I 2022-10-11 01:44:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 410/2304 [I 2022-10-11 01:44:55] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:27:28 [I 2022-10-11 01:44:55] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00318 [I 2022-10-11 01:44:55] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-11 01:44:55] ...optimizers/trijoint.py.151: total_norm: 12.010102 [I 2022-10-11 01:45:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 420/2304 [I 2022-10-11 01:45:03] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:27:16 [I 2022-10-11 01:45:03] ...trap/engines/engine.py.229: process: 0.87812 | load: 0.00310 [I 2022-10-11 01:45:03] ...trap/engines/engine.py.230: loss: 0.08951 [I 2022-10-11 01:45:03] ...optimizers/trijoint.py.151: total_norm: 9.311872 [I 2022-10-11 01:45:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 430/2304 [I 2022-10-11 01:45:12] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:52 [I 2022-10-11 01:45:12] ...trap/engines/engine.py.229: process: 0.86247 | load: 0.00336 [I 2022-10-11 01:45:12] ...trap/engines/engine.py.230: loss: 0.14729 [I 2022-10-11 01:45:12] ...optimizers/trijoint.py.151: total_norm: 25.879265 [I 2022-10-11 01:45:21] ...trap/engines/engine.py.224: train: epoch 93 | batch 440/2304 [I 2022-10-11 01:45:21] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:51 [I 2022-10-11 01:45:21] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00311 [I 2022-10-11 01:45:21] ...trap/engines/engine.py.230: loss: 0.11381 [I 2022-10-11 01:45:21] ...optimizers/trijoint.py.151: total_norm: 16.629572 [I 2022-10-11 01:45:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 450/2304 [I 2022-10-11 01:45:29] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:46 [I 2022-10-11 01:45:29] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00314 [I 2022-10-11 01:45:29] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-11 01:45:29] ...optimizers/trijoint.py.151: total_norm: 13.681214 [I 2022-10-11 01:45:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 460/2304 [I 2022-10-11 01:45:38] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:36 [I 2022-10-11 01:45:38] ...trap/engines/engine.py.229: process: 0.86363 | load: 0.00315 [I 2022-10-11 01:45:38] ...trap/engines/engine.py.230: loss: 0.08412 [I 2022-10-11 01:45:38] ...optimizers/trijoint.py.151: total_norm: 11.819994 [I 2022-10-11 01:45:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 470/2304 [I 2022-10-11 01:45:47] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:27 [I 2022-10-11 01:45:47] ...trap/engines/engine.py.229: process: 0.87265 | load: 0.00316 [I 2022-10-11 01:45:47] ...trap/engines/engine.py.230: loss: 0.10287 [I 2022-10-11 01:45:47] ...optimizers/trijoint.py.151: total_norm: 25.915245 [I 2022-10-11 01:45:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 480/2304 [I 2022-10-11 01:45:55] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:08 [I 2022-10-11 01:45:55] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00325 [I 2022-10-11 01:45:55] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-11 01:45:55] ...optimizers/trijoint.py.151: total_norm: 13.117634 [I 2022-10-11 01:46:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 490/2304 [I 2022-10-11 01:46:04] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:26:15 [I 2022-10-11 01:46:04] ...trap/engines/engine.py.229: process: 0.88248 | load: 0.00313 [I 2022-10-11 01:46:04] ...trap/engines/engine.py.230: loss: 0.15595 [I 2022-10-11 01:46:04] ...optimizers/trijoint.py.151: total_norm: 25.333448 [I 2022-10-11 01:46:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 500/2304 [I 2022-10-11 01:46:13] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:26:07 [I 2022-10-11 01:46:13] ...trap/engines/engine.py.229: process: 0.87573 | load: 0.00313 [I 2022-10-11 01:46:13] ...trap/engines/engine.py.230: loss: 0.14925 [I 2022-10-11 01:46:13] ...optimizers/trijoint.py.151: total_norm: 12.562772 [I 2022-10-11 01:46:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 510/2304 [I 2022-10-11 01:46:22] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:26:05 [I 2022-10-11 01:46:22] ...trap/engines/engine.py.229: process: 0.88812 | load: 0.00308 [I 2022-10-11 01:46:22] ...trap/engines/engine.py.230: loss: 0.09527 [I 2022-10-11 01:46:22] ...optimizers/trijoint.py.151: total_norm: 18.067148 [I 2022-10-11 01:46:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 520/2304 [I 2022-10-11 01:46:30] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:52 [I 2022-10-11 01:46:30] ...trap/engines/engine.py.229: process: 0.87346 | load: 0.00313 [I 2022-10-11 01:46:30] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-11 01:46:30] ...optimizers/trijoint.py.151: total_norm: 19.732803 [I 2022-10-11 01:46:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 530/2304 [I 2022-10-11 01:46:39] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:55 [I 2022-10-11 01:46:39] ...trap/engines/engine.py.229: process: 0.88583 | load: 0.00314 [I 2022-10-11 01:46:39] ...trap/engines/engine.py.230: loss: 0.13152 [I 2022-10-11 01:46:39] ...optimizers/trijoint.py.151: total_norm: 12.527441 [I 2022-10-11 01:46:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 540/2304 [I 2022-10-11 01:46:48] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:29 [I 2022-10-11 01:46:48] ...trap/engines/engine.py.229: process: 0.86892 | load: 0.00312 [I 2022-10-11 01:46:48] ...trap/engines/engine.py.230: loss: 0.14468 [I 2022-10-11 01:46:48] ...optimizers/trijoint.py.151: total_norm: 15.956374 [I 2022-10-11 01:46:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 550/2304 [I 2022-10-11 01:46:56] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:12 [I 2022-10-11 01:46:56] ...trap/engines/engine.py.229: process: 0.86503 | load: 0.00313 [I 2022-10-11 01:46:56] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-11 01:46:56] ...optimizers/trijoint.py.151: total_norm: 10.094939 [I 2022-10-11 01:47:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 560/2304 [I 2022-10-11 01:47:05] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:11 [I 2022-10-11 01:47:05] ...trap/engines/engine.py.229: process: 0.87260 | load: 0.00316 [I 2022-10-11 01:47:05] ...trap/engines/engine.py.230: loss: 0.09588 [I 2022-10-11 01:47:05] ...optimizers/trijoint.py.151: total_norm: 17.226938 [I 2022-10-11 01:47:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 570/2304 [I 2022-10-11 01:47:14] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:25:00 [I 2022-10-11 01:47:14] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00317 [I 2022-10-11 01:47:14] ...trap/engines/engine.py.230: loss: 0.12419 [I 2022-10-11 01:47:14] ...optimizers/trijoint.py.151: total_norm: 8.072875 [I 2022-10-11 01:47:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 580/2304 [I 2022-10-11 01:47:22] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:52 [I 2022-10-11 01:47:22] ...trap/engines/engine.py.229: process: 0.86169 | load: 0.00302 [I 2022-10-11 01:47:22] ...trap/engines/engine.py.230: loss: 0.08772 [I 2022-10-11 01:47:22] ...optimizers/trijoint.py.151: total_norm: 19.342394 [I 2022-10-11 01:47:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 590/2304 [I 2022-10-11 01:47:31] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:30 [I 2022-10-11 01:47:31] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00297 [I 2022-10-11 01:47:31] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-11 01:47:31] ...optimizers/trijoint.py.151: total_norm: 29.641821 [I 2022-10-11 01:47:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 600/2304 [I 2022-10-11 01:47:40] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:25 [I 2022-10-11 01:47:40] ...trap/engines/engine.py.229: process: 0.86798 | load: 0.00306 [I 2022-10-11 01:47:40] ...trap/engines/engine.py.230: loss: 0.11682 [I 2022-10-11 01:47:40] ...optimizers/trijoint.py.151: total_norm: 17.764280 [I 2022-10-11 01:47:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 610/2304 [I 2022-10-11 01:47:48] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:10 [I 2022-10-11 01:47:48] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00303 [I 2022-10-11 01:47:48] ...trap/engines/engine.py.230: loss: 0.15475 [I 2022-10-11 01:47:48] ...optimizers/trijoint.py.151: total_norm: 10.327383 [I 2022-10-11 01:47:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 620/2304 [I 2022-10-11 01:47:57] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:59 [I 2022-10-11 01:47:57] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00294 [I 2022-10-11 01:47:57] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-11 01:47:57] ...optimizers/trijoint.py.151: total_norm: 19.428389 [I 2022-10-11 01:48:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 630/2304 [I 2022-10-11 01:48:05] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:24:04 [I 2022-10-11 01:48:05] ...trap/engines/engine.py.229: process: 0.86310 | load: 0.00307 [I 2022-10-11 01:48:05] ...trap/engines/engine.py.230: loss: 0.09997 [I 2022-10-11 01:48:05] ...optimizers/trijoint.py.151: total_norm: 5.938894 [I 2022-10-11 01:48:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 640/2304 [I 2022-10-11 01:48:14] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:46 [I 2022-10-11 01:48:14] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00294 [I 2022-10-11 01:48:14] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-11 01:48:14] ...optimizers/trijoint.py.151: total_norm: 22.227442 [I 2022-10-11 01:48:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 650/2304 [I 2022-10-11 01:48:23] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:26 [I 2022-10-11 01:48:23] ...trap/engines/engine.py.229: process: 0.84222 | load: 0.00296 [I 2022-10-11 01:48:23] ...trap/engines/engine.py.230: loss: 0.12270 [I 2022-10-11 01:48:23] ...optimizers/trijoint.py.151: total_norm: 12.429649 [I 2022-10-11 01:48:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 660/2304 [I 2022-10-11 01:48:31] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:13 [I 2022-10-11 01:48:31] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00297 [I 2022-10-11 01:48:31] ...trap/engines/engine.py.230: loss: 0.19079 [I 2022-10-11 01:48:31] ...optimizers/trijoint.py.151: total_norm: 13.126541 [I 2022-10-11 01:48:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 670/2304 [I 2022-10-11 01:48:40] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:12 [I 2022-10-11 01:48:40] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00300 [I 2022-10-11 01:48:40] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-11 01:48:40] ...optimizers/trijoint.py.151: total_norm: 9.810259 [I 2022-10-11 01:48:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 680/2304 [I 2022-10-11 01:48:48] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:04 [I 2022-10-11 01:48:48] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00301 [I 2022-10-11 01:48:48] ...trap/engines/engine.py.230: loss: 0.16573 [I 2022-10-11 01:48:48] ...optimizers/trijoint.py.151: total_norm: 23.696712 [I 2022-10-11 01:48:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 690/2304 [I 2022-10-11 01:48:57] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:57 [I 2022-10-11 01:48:57] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00305 [I 2022-10-11 01:48:57] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-11 01:48:57] ...optimizers/trijoint.py.151: total_norm: 23.550920 [I 2022-10-11 01:49:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 700/2304 [I 2022-10-11 01:49:05] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:58 [I 2022-10-11 01:49:05] ...trap/engines/engine.py.229: process: 0.86632 | load: 0.00297 [I 2022-10-11 01:49:05] ...trap/engines/engine.py.230: loss: 0.09473 [I 2022-10-11 01:49:05] ...optimizers/trijoint.py.151: total_norm: 12.515353 [I 2022-10-11 01:49:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 710/2304 [I 2022-10-11 01:49:14] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:48 [I 2022-10-11 01:49:14] ...trap/engines/engine.py.229: process: 0.86936 | load: 0.00301 [I 2022-10-11 01:49:14] ...trap/engines/engine.py.230: loss: 0.10163 [I 2022-10-11 01:49:14] ...optimizers/trijoint.py.151: total_norm: 9.106003 [I 2022-10-11 01:49:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 720/2304 [I 2022-10-11 01:49:23] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:29 [I 2022-10-11 01:49:23] ...trap/engines/engine.py.229: process: 0.85092 | load: 0.00299 [I 2022-10-11 01:49:23] ...trap/engines/engine.py.230: loss: 0.17900 [I 2022-10-11 01:49:23] ...optimizers/trijoint.py.151: total_norm: 13.763700 [I 2022-10-11 01:49:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 730/2304 [I 2022-10-11 01:49:31] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:17 [I 2022-10-11 01:49:31] ...trap/engines/engine.py.229: process: 0.85104 | load: 0.00309 [I 2022-10-11 01:49:31] ...trap/engines/engine.py.230: loss: 0.08247 [I 2022-10-11 01:49:31] ...optimizers/trijoint.py.151: total_norm: 13.995620 [I 2022-10-11 01:49:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 740/2304 [I 2022-10-11 01:49:40] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:10 [I 2022-10-11 01:49:40] ...trap/engines/engine.py.229: process: 0.87036 | load: 0.00308 [I 2022-10-11 01:49:40] ...trap/engines/engine.py.230: loss: 0.11889 [I 2022-10-11 01:49:40] ...optimizers/trijoint.py.151: total_norm: 21.837706 [I 2022-10-11 01:49:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 750/2304 [I 2022-10-11 01:49:48] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:07 [I 2022-10-11 01:49:48] ...trap/engines/engine.py.229: process: 0.83150 | load: 0.00302 [I 2022-10-11 01:49:48] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-11 01:49:48] ...optimizers/trijoint.py.151: total_norm: 9.319648 [I 2022-10-11 01:49:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 760/2304 [I 2022-10-11 01:49:57] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:21:55 [I 2022-10-11 01:49:57] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00310 [I 2022-10-11 01:49:57] ...trap/engines/engine.py.230: loss: 0.09399 [I 2022-10-11 01:49:57] ...optimizers/trijoint.py.151: total_norm: 15.265686 [I 2022-10-11 01:50:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 770/2304 [I 2022-10-11 01:50:05] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:47 [I 2022-10-11 01:50:05] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00304 [I 2022-10-11 01:50:05] ...trap/engines/engine.py.230: loss: 0.21360 [I 2022-10-11 01:50:05] ...optimizers/trijoint.py.151: total_norm: 45.110489 [I 2022-10-11 01:50:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 780/2304 [I 2022-10-11 01:50:14] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:44 [I 2022-10-11 01:50:14] ...trap/engines/engine.py.229: process: 0.85695 | load: 0.00309 [I 2022-10-11 01:50:14] ...trap/engines/engine.py.230: loss: 0.11030 [I 2022-10-11 01:50:14] ...optimizers/trijoint.py.151: total_norm: 17.502956 [I 2022-10-11 01:50:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 790/2304 [I 2022-10-11 01:50:23] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:56 [I 2022-10-11 01:50:23] ...trap/engines/engine.py.229: process: 0.86990 | load: 0.00311 [I 2022-10-11 01:50:23] ...trap/engines/engine.py.230: loss: 0.17650 [I 2022-10-11 01:50:23] ...optimizers/trijoint.py.151: total_norm: 12.781417 [I 2022-10-11 01:50:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 800/2304 [I 2022-10-11 01:50:31] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:42 [I 2022-10-11 01:50:31] ...trap/engines/engine.py.229: process: 0.87237 | load: 0.00310 [I 2022-10-11 01:50:31] ...trap/engines/engine.py.230: loss: 0.10290 [I 2022-10-11 01:50:31] ...optimizers/trijoint.py.151: total_norm: 28.124071 [I 2022-10-11 01:50:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 810/2304 [I 2022-10-11 01:50:40] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:39 [I 2022-10-11 01:50:40] ...trap/engines/engine.py.229: process: 0.86755 | load: 0.00316 [I 2022-10-11 01:50:40] ...trap/engines/engine.py.230: loss: 0.17680 [I 2022-10-11 01:50:40] ...optimizers/trijoint.py.151: total_norm: 17.031399 [I 2022-10-11 01:50:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 820/2304 [I 2022-10-11 01:50:49] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:21 [I 2022-10-11 01:50:49] ...trap/engines/engine.py.229: process: 0.85468 | load: 0.00318 [I 2022-10-11 01:50:49] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-11 01:50:49] ...optimizers/trijoint.py.151: total_norm: 9.803858 [I 2022-10-11 01:50:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 830/2304 [I 2022-10-11 01:50:57] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:24 [I 2022-10-11 01:50:57] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00308 [I 2022-10-11 01:50:57] ...trap/engines/engine.py.230: loss: 0.19122 [I 2022-10-11 01:50:57] ...optimizers/trijoint.py.151: total_norm: 26.567129 [I 2022-10-11 01:51:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 840/2304 [I 2022-10-11 01:51:06] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:17 [I 2022-10-11 01:51:06] ...trap/engines/engine.py.229: process: 0.86161 | load: 0.00310 [I 2022-10-11 01:51:06] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-11 01:51:06] ...optimizers/trijoint.py.151: total_norm: 24.094036 [I 2022-10-11 01:51:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 850/2304 [I 2022-10-11 01:51:15] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:21:09 [I 2022-10-11 01:51:15] ...trap/engines/engine.py.229: process: 0.86786 | load: 0.00323 [I 2022-10-11 01:51:15] ...trap/engines/engine.py.230: loss: 0.09157 [I 2022-10-11 01:51:15] ...optimizers/trijoint.py.151: total_norm: 11.839837 [I 2022-10-11 01:51:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 860/2304 [I 2022-10-11 01:51:24] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:54 [I 2022-10-11 01:51:24] ...trap/engines/engine.py.229: process: 0.87192 | load: 0.00317 [I 2022-10-11 01:51:24] ...trap/engines/engine.py.230: loss: 0.12211 [I 2022-10-11 01:51:24] ...optimizers/trijoint.py.151: total_norm: 8.016039 [I 2022-10-11 01:51:32] ...trap/engines/engine.py.224: train: epoch 93 | batch 870/2304 [I 2022-10-11 01:51:32] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:40 [I 2022-10-11 01:51:32] ...trap/engines/engine.py.229: process: 0.86733 | load: 0.00333 [I 2022-10-11 01:51:32] ...trap/engines/engine.py.230: loss: 0.21314 [I 2022-10-11 01:51:32] ...optimizers/trijoint.py.151: total_norm: 25.806553 [I 2022-10-11 01:51:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 880/2304 [I 2022-10-11 01:51:41] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:29 [I 2022-10-11 01:51:41] ...trap/engines/engine.py.229: process: 0.85359 | load: 0.00335 [I 2022-10-11 01:51:41] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-11 01:51:41] ...optimizers/trijoint.py.151: total_norm: 16.884346 [I 2022-10-11 01:51:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 890/2304 [I 2022-10-11 01:51:50] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:31 [I 2022-10-11 01:51:50] ...trap/engines/engine.py.229: process: 0.87392 | load: 0.00312 [I 2022-10-11 01:51:50] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-11 01:51:50] ...optimizers/trijoint.py.151: total_norm: 15.603685 [I 2022-10-11 01:51:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 900/2304 [I 2022-10-11 01:51:58] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:14 [I 2022-10-11 01:51:58] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00311 [I 2022-10-11 01:51:58] ...trap/engines/engine.py.230: loss: 0.10389 [I 2022-10-11 01:51:58] ...optimizers/trijoint.py.151: total_norm: 46.545906 [I 2022-10-11 01:52:07] ...trap/engines/engine.py.224: train: epoch 93 | batch 910/2304 [I 2022-10-11 01:52:07] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:08 [I 2022-10-11 01:52:07] ...trap/engines/engine.py.229: process: 0.86602 | load: 0.00323 [I 2022-10-11 01:52:07] ...trap/engines/engine.py.230: loss: 0.14293 [I 2022-10-11 01:52:07] ...optimizers/trijoint.py.151: total_norm: 26.229010 [I 2022-10-11 01:52:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 920/2304 [I 2022-10-11 01:52:16] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:20:01 [I 2022-10-11 01:52:16] ...trap/engines/engine.py.229: process: 0.87353 | load: 0.00327 [I 2022-10-11 01:52:16] ...trap/engines/engine.py.230: loss: 0.17375 [I 2022-10-11 01:52:16] ...optimizers/trijoint.py.151: total_norm: 23.202593 [I 2022-10-11 01:52:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 930/2304 [I 2022-10-11 01:52:25] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:46 [I 2022-10-11 01:52:25] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00308 [I 2022-10-11 01:52:25] ...trap/engines/engine.py.230: loss: 0.10278 [I 2022-10-11 01:52:25] ...optimizers/trijoint.py.151: total_norm: 8.082683 [I 2022-10-11 01:52:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 940/2304 [I 2022-10-11 01:52:33] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:45 [I 2022-10-11 01:52:33] ...trap/engines/engine.py.229: process: 0.88727 | load: 0.00332 [I 2022-10-11 01:52:33] ...trap/engines/engine.py.230: loss: 0.12189 [I 2022-10-11 01:52:33] ...optimizers/trijoint.py.151: total_norm: 11.669562 [I 2022-10-11 01:52:42] ...trap/engines/engine.py.224: train: epoch 93 | batch 950/2304 [I 2022-10-11 01:52:42] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:32 [I 2022-10-11 01:52:42] ...trap/engines/engine.py.229: process: 0.86789 | load: 0.00319 [I 2022-10-11 01:52:42] ...trap/engines/engine.py.230: loss: 0.09462 [I 2022-10-11 01:52:42] ...optimizers/trijoint.py.151: total_norm: 11.498041 [I 2022-10-11 01:52:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 960/2304 [I 2022-10-11 01:52:51] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:20 [I 2022-10-11 01:52:51] ...trap/engines/engine.py.229: process: 0.86185 | load: 0.00315 [I 2022-10-11 01:52:51] ...trap/engines/engine.py.230: loss: 0.09576 [I 2022-10-11 01:52:51] ...optimizers/trijoint.py.151: total_norm: 9.082423 [I 2022-10-11 01:52:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 970/2304 [I 2022-10-11 01:52:59] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:18 [I 2022-10-11 01:52:59] ...trap/engines/engine.py.229: process: 0.86202 | load: 0.00314 [I 2022-10-11 01:52:59] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-11 01:52:59] ...optimizers/trijoint.py.151: total_norm: 16.170061 [I 2022-10-11 01:53:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 980/2304 [I 2022-10-11 01:53:08] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:01 [I 2022-10-11 01:53:08] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00319 [I 2022-10-11 01:53:08] ...trap/engines/engine.py.230: loss: 0.15108 [I 2022-10-11 01:53:08] ...optimizers/trijoint.py.151: total_norm: 15.222220 [I 2022-10-11 01:53:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 990/2304 [I 2022-10-11 01:53:17] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:55 [I 2022-10-11 01:53:17] ...trap/engines/engine.py.229: process: 0.87296 | load: 0.00311 [I 2022-10-11 01:53:17] ...trap/engines/engine.py.230: loss: 0.15772 [I 2022-10-11 01:53:17] ...optimizers/trijoint.py.151: total_norm: 13.064289 [I 2022-10-11 01:53:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 1000/2304 [I 2022-10-11 01:53:25] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:55 [I 2022-10-11 01:53:25] ...trap/engines/engine.py.229: process: 0.86723 | load: 0.00314 [I 2022-10-11 01:53:25] ...trap/engines/engine.py.230: loss: 0.15736 [I 2022-10-11 01:53:25] ...optimizers/trijoint.py.151: total_norm: 20.708935 [I 2022-10-11 01:53:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 1010/2304 [I 2022-10-11 01:53:34] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:42 [I 2022-10-11 01:53:34] ...trap/engines/engine.py.229: process: 0.86771 | load: 0.00313 [I 2022-10-11 01:53:34] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-11 01:53:34] ...optimizers/trijoint.py.151: total_norm: 11.887696 [I 2022-10-11 01:53:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 1020/2304 [I 2022-10-11 01:53:43] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:34 [I 2022-10-11 01:53:43] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00321 [I 2022-10-11 01:53:43] ...trap/engines/engine.py.230: loss: 0.12191 [I 2022-10-11 01:53:43] ...optimizers/trijoint.py.151: total_norm: 16.085653 [I 2022-10-11 01:53:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 1030/2304 [I 2022-10-11 01:53:51] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:20 [I 2022-10-11 01:53:51] ...trap/engines/engine.py.229: process: 0.86411 | load: 0.00310 [I 2022-10-11 01:53:51] ...trap/engines/engine.py.230: loss: 0.13795 [I 2022-10-11 01:53:51] ...optimizers/trijoint.py.151: total_norm: 9.012094 [I 2022-10-11 01:54:00] ...trap/engines/engine.py.224: train: epoch 93 | batch 1040/2304 [I 2022-10-11 01:54:00] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:04 [I 2022-10-11 01:54:00] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00314 [I 2022-10-11 01:54:00] ...trap/engines/engine.py.230: loss: 0.11557 [I 2022-10-11 01:54:00] ...optimizers/trijoint.py.151: total_norm: 19.714710 [I 2022-10-11 01:54:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 1050/2304 [I 2022-10-11 01:54:09] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:08 [I 2022-10-11 01:54:09] ...trap/engines/engine.py.229: process: 0.86907 | load: 0.00308 [I 2022-10-11 01:54:09] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-11 01:54:09] ...optimizers/trijoint.py.151: total_norm: 10.989378 [I 2022-10-11 01:54:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 1060/2304 [I 2022-10-11 01:54:17] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:48 [I 2022-10-11 01:54:17] ...trap/engines/engine.py.229: process: 0.87618 | load: 0.00296 [I 2022-10-11 01:54:17] ...trap/engines/engine.py.230: loss: 0.17269 [I 2022-10-11 01:54:17] ...optimizers/trijoint.py.151: total_norm: 13.397012 [I 2022-10-11 01:54:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 1070/2304 [I 2022-10-11 01:54:26] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:34 [I 2022-10-11 01:54:26] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00301 [I 2022-10-11 01:54:26] ...trap/engines/engine.py.230: loss: 0.10093 [I 2022-10-11 01:54:26] ...optimizers/trijoint.py.151: total_norm: 10.227209 [I 2022-10-11 01:54:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 1080/2304 [I 2022-10-11 01:54:34] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:34 [I 2022-10-11 01:54:34] ...trap/engines/engine.py.229: process: 0.87536 | load: 0.00296 [I 2022-10-11 01:54:35] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-11 01:54:35] ...optimizers/trijoint.py.151: total_norm: 11.013170 [I 2022-10-11 01:54:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 1090/2304 [I 2022-10-11 01:54:43] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:13 [I 2022-10-11 01:54:43] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00297 [I 2022-10-11 01:54:43] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-11 01:54:43] ...optimizers/trijoint.py.151: total_norm: 13.482974 [I 2022-10-11 01:54:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 1100/2304 [I 2022-10-11 01:54:52] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:07 [I 2022-10-11 01:54:52] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00299 [I 2022-10-11 01:54:52] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-11 01:54:52] ...optimizers/trijoint.py.151: total_norm: 12.942791 [I 2022-10-11 01:55:00] ...trap/engines/engine.py.224: train: epoch 93 | batch 1110/2304 [I 2022-10-11 01:55:00] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:55 [I 2022-10-11 01:55:00] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00302 [I 2022-10-11 01:55:00] ...trap/engines/engine.py.230: loss: 0.08656 [I 2022-10-11 01:55:00] ...optimizers/trijoint.py.151: total_norm: 17.003454 [I 2022-10-11 01:55:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 1120/2304 [I 2022-10-11 01:55:09] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:16:53 [I 2022-10-11 01:55:09] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00302 [I 2022-10-11 01:55:09] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-11 01:55:09] ...optimizers/trijoint.py.151: total_norm: 58.794701 [I 2022-10-11 01:55:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 1130/2304 [I 2022-10-11 01:55:17] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:49 [I 2022-10-11 01:55:17] ...trap/engines/engine.py.229: process: 0.88605 | load: 0.00300 [I 2022-10-11 01:55:17] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-11 01:55:17] ...optimizers/trijoint.py.151: total_norm: 17.207911 [I 2022-10-11 01:55:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 1140/2304 [I 2022-10-11 01:55:26] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:35 [I 2022-10-11 01:55:26] ...trap/engines/engine.py.229: process: 0.83303 | load: 0.00295 [I 2022-10-11 01:55:26] ...trap/engines/engine.py.230: loss: 0.13007 [I 2022-10-11 01:55:26] ...optimizers/trijoint.py.151: total_norm: 22.533033 [I 2022-10-11 01:55:53] ...trap/engines/engine.py.224: train: epoch 93 | batch 1150/2304 [I 2022-10-11 01:55:53] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 1:01:12 [I 2022-10-11 01:55:53] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00299 [I 2022-10-11 01:55:53] ...trap/engines/engine.py.230: loss: 0.11653 [I 2022-10-11 01:55:53] ...optimizers/trijoint.py.151: total_norm: 11.391562 [I 2022-10-11 01:56:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 1160/2304 [I 2022-10-11 01:56:01] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:21:05 [I 2022-10-11 01:56:01] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00309 [I 2022-10-11 01:56:01] ...trap/engines/engine.py.230: loss: 0.10714 [I 2022-10-11 01:56:01] ...optimizers/trijoint.py.151: total_norm: 10.159466 [I 2022-10-11 01:56:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 1170/2304 [I 2022-10-11 01:56:10] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:16:46 [I 2022-10-11 01:56:10] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00318 [I 2022-10-11 01:56:10] ...trap/engines/engine.py.230: loss: 0.09387 [I 2022-10-11 01:56:10] ...optimizers/trijoint.py.151: total_norm: 9.041780 [I 2022-10-11 01:56:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 1180/2304 [I 2022-10-11 01:56:19] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:16:12 [I 2022-10-11 01:56:19] ...trap/engines/engine.py.229: process: 0.86778 | load: 0.00307 [I 2022-10-11 01:56:19] ...trap/engines/engine.py.230: loss: 0.08870 [I 2022-10-11 01:56:19] ...optimizers/trijoint.py.151: total_norm: 10.353920 [I 2022-10-11 01:56:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 1190/2304 [I 2022-10-11 01:56:27] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:16:03 [I 2022-10-11 01:56:27] ...trap/engines/engine.py.229: process: 0.86626 | load: 0.00307 [I 2022-10-11 01:56:27] ...trap/engines/engine.py.230: loss: 0.07848 [I 2022-10-11 01:56:27] ...optimizers/trijoint.py.151: total_norm: 12.556770 [I 2022-10-11 01:56:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 1200/2304 [I 2022-10-11 01:56:36] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:15:55 [I 2022-10-11 01:56:36] ...trap/engines/engine.py.229: process: 0.86155 | load: 0.00315 [I 2022-10-11 01:56:36] ...trap/engines/engine.py.230: loss: 0.09918 [I 2022-10-11 01:56:36] ...optimizers/trijoint.py.151: total_norm: 12.995316 [I 2022-10-11 01:56:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 1210/2304 [I 2022-10-11 01:56:45] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:46 [I 2022-10-11 01:56:45] ...trap/engines/engine.py.229: process: 0.85094 | load: 0.00324 [I 2022-10-11 01:56:45] ...trap/engines/engine.py.230: loss: 0.08675 [I 2022-10-11 01:56:45] ...optimizers/trijoint.py.151: total_norm: 10.846779 [I 2022-10-11 01:56:53] ...trap/engines/engine.py.224: train: epoch 93 | batch 1220/2304 [I 2022-10-11 01:56:53] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:36 [I 2022-10-11 01:56:53] ...trap/engines/engine.py.229: process: 0.87418 | load: 0.00312 [I 2022-10-11 01:56:53] ...trap/engines/engine.py.230: loss: 0.13726 [I 2022-10-11 01:56:53] ...optimizers/trijoint.py.151: total_norm: 15.074732 [I 2022-10-11 01:57:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 1230/2304 [I 2022-10-11 01:57:02] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:25 [I 2022-10-11 01:57:02] ...trap/engines/engine.py.229: process: 0.85931 | load: 0.00314 [I 2022-10-11 01:57:02] ...trap/engines/engine.py.230: loss: 0.14421 [I 2022-10-11 01:57:02] ...optimizers/trijoint.py.151: total_norm: 11.642295 [I 2022-10-11 01:57:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 1240/2304 [I 2022-10-11 01:57:11] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:17 [I 2022-10-11 01:57:11] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00310 [I 2022-10-11 01:57:11] ...trap/engines/engine.py.230: loss: 0.10403 [I 2022-10-11 01:57:11] ...optimizers/trijoint.py.151: total_norm: 17.179661 [I 2022-10-11 01:57:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 1250/2304 [I 2022-10-11 01:57:19] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:15:11 [I 2022-10-11 01:57:19] ...trap/engines/engine.py.229: process: 0.87308 | load: 0.00312 [I 2022-10-11 01:57:19] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-11 01:57:19] ...optimizers/trijoint.py.151: total_norm: 30.244886 [I 2022-10-11 01:57:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 1260/2304 [I 2022-10-11 01:57:28] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:15:08 [I 2022-10-11 01:57:28] ...trap/engines/engine.py.229: process: 0.87965 | load: 0.00315 [I 2022-10-11 01:57:28] ...trap/engines/engine.py.230: loss: 0.09034 [I 2022-10-11 01:57:28] ...optimizers/trijoint.py.151: total_norm: 9.801389 [I 2022-10-11 01:57:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 1270/2304 [I 2022-10-11 01:57:37] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:15:00 [I 2022-10-11 01:57:37] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00345 [I 2022-10-11 01:57:37] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-11 01:57:37] ...optimizers/trijoint.py.151: total_norm: 17.199570 [I 2022-10-11 01:57:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 1280/2304 [I 2022-10-11 01:57:46] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:49 [I 2022-10-11 01:57:46] ...trap/engines/engine.py.229: process: 0.87156 | load: 0.00322 [I 2022-10-11 01:57:46] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-11 01:57:46] ...optimizers/trijoint.py.151: total_norm: 17.436544 [I 2022-10-11 01:57:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 1290/2304 [I 2022-10-11 01:57:54] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:42 [I 2022-10-11 01:57:54] ...trap/engines/engine.py.229: process: 0.88502 | load: 0.00327 [I 2022-10-11 01:57:54] ...trap/engines/engine.py.230: loss: 0.08240 [I 2022-10-11 01:57:54] ...optimizers/trijoint.py.151: total_norm: 14.625642 [I 2022-10-11 01:58:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 1300/2304 [I 2022-10-11 01:58:03] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:14:27 [I 2022-10-11 01:58:03] ...trap/engines/engine.py.229: process: 0.86151 | load: 0.00310 [I 2022-10-11 01:58:03] ...trap/engines/engine.py.230: loss: 0.17408 [I 2022-10-11 01:58:03] ...optimizers/trijoint.py.151: total_norm: 14.853569 [I 2022-10-11 01:58:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 1310/2304 [I 2022-10-11 01:58:12] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:14:20 [I 2022-10-11 01:58:12] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00311 [I 2022-10-11 01:58:12] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-11 01:58:12] ...optimizers/trijoint.py.151: total_norm: 13.450228 [I 2022-10-11 01:58:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 1320/2304 [I 2022-10-11 01:58:20] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:14:16 [I 2022-10-11 01:58:20] ...trap/engines/engine.py.229: process: 0.87188 | load: 0.00314 [I 2022-10-11 01:58:20] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-11 01:58:20] ...optimizers/trijoint.py.151: total_norm: 14.170502 [I 2022-10-11 01:58:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 1330/2304 [I 2022-10-11 01:58:29] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:14:00 [I 2022-10-11 01:58:29] ...trap/engines/engine.py.229: process: 0.86689 | load: 0.00344 [I 2022-10-11 01:58:29] ...trap/engines/engine.py.230: loss: 0.12712 [I 2022-10-11 01:58:29] ...optimizers/trijoint.py.151: total_norm: 16.606993 [I 2022-10-11 01:58:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 1340/2304 [I 2022-10-11 01:58:38] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:14:01 [I 2022-10-11 01:58:38] ...trap/engines/engine.py.229: process: 0.87317 | load: 0.00308 [I 2022-10-11 01:58:38] ...trap/engines/engine.py.230: loss: 0.13929 [I 2022-10-11 01:58:38] ...optimizers/trijoint.py.151: total_norm: 13.215420 [I 2022-10-11 01:58:46] ...trap/engines/engine.py.224: train: epoch 93 | batch 1350/2304 [I 2022-10-11 01:58:46] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:47 [I 2022-10-11 01:58:46] ...trap/engines/engine.py.229: process: 0.88475 | load: 0.00309 [I 2022-10-11 01:58:46] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-11 01:58:46] ...optimizers/trijoint.py.151: total_norm: 18.558487 [I 2022-10-11 01:58:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 1360/2304 [I 2022-10-11 01:58:55] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:38 [I 2022-10-11 01:58:55] ...trap/engines/engine.py.229: process: 0.87658 | load: 0.00313 [I 2022-10-11 01:58:55] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-11 01:58:55] ...optimizers/trijoint.py.151: total_norm: 13.401264 [I 2022-10-11 01:59:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 1370/2304 [I 2022-10-11 01:59:04] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:37 [I 2022-10-11 01:59:04] ...trap/engines/engine.py.229: process: 0.89715 | load: 0.00311 [I 2022-10-11 01:59:04] ...trap/engines/engine.py.230: loss: 0.20022 [I 2022-10-11 01:59:04] ...optimizers/trijoint.py.151: total_norm: 14.969525 [I 2022-10-11 01:59:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 1380/2304 [I 2022-10-11 01:59:12] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:13:19 [I 2022-10-11 01:59:12] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00311 [I 2022-10-11 01:59:12] ...trap/engines/engine.py.230: loss: 0.13033 [I 2022-10-11 01:59:12] ...optimizers/trijoint.py.151: total_norm: 9.005087 [I 2022-10-11 01:59:21] ...trap/engines/engine.py.224: train: epoch 93 | batch 1390/2304 [I 2022-10-11 01:59:21] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:13:12 [I 2022-10-11 01:59:21] ...trap/engines/engine.py.229: process: 0.87340 | load: 0.00314 [I 2022-10-11 01:59:21] ...trap/engines/engine.py.230: loss: 0.23720 [I 2022-10-11 01:59:21] ...optimizers/trijoint.py.151: total_norm: 29.445444 [I 2022-10-11 01:59:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1400/2304 [I 2022-10-11 01:59:30] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:59 [I 2022-10-11 01:59:30] ...trap/engines/engine.py.229: process: 0.86237 | load: 0.00312 [I 2022-10-11 01:59:30] ...trap/engines/engine.py.230: loss: 0.13779 [I 2022-10-11 01:59:30] ...optimizers/trijoint.py.151: total_norm: 17.155678 [I 2022-10-11 01:59:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 1410/2304 [I 2022-10-11 01:59:38] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:49 [I 2022-10-11 01:59:38] ...trap/engines/engine.py.229: process: 0.86260 | load: 0.00322 [I 2022-10-11 01:59:38] ...trap/engines/engine.py.230: loss: 0.09652 [I 2022-10-11 01:59:38] ...optimizers/trijoint.py.151: total_norm: 16.576334 [I 2022-10-11 01:59:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 1420/2304 [I 2022-10-11 01:59:47] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:46 [I 2022-10-11 01:59:47] ...trap/engines/engine.py.229: process: 0.88070 | load: 0.00321 [I 2022-10-11 01:59:47] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-11 01:59:47] ...optimizers/trijoint.py.151: total_norm: 13.550611 [I 2022-10-11 01:59:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1430/2304 [I 2022-10-11 01:59:56] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:34 [I 2022-10-11 01:59:56] ...trap/engines/engine.py.229: process: 0.86735 | load: 0.00310 [I 2022-10-11 01:59:56] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-11 01:59:56] ...optimizers/trijoint.py.151: total_norm: 10.118293 [I 2022-10-11 02:00:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 1440/2304 [I 2022-10-11 02:00:04] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:12:18 [I 2022-10-11 02:00:04] ...trap/engines/engine.py.229: process: 0.86763 | load: 0.00327 [I 2022-10-11 02:00:04] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-11 02:00:04] ...optimizers/trijoint.py.151: total_norm: 9.045426 [I 2022-10-11 02:00:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1450/2304 [I 2022-10-11 02:00:13] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:12:09 [I 2022-10-11 02:00:13] ...trap/engines/engine.py.229: process: 0.86496 | load: 0.00313 [I 2022-10-11 02:00:13] ...trap/engines/engine.py.230: loss: 0.11506 [I 2022-10-11 02:00:13] ...optimizers/trijoint.py.151: total_norm: 16.069309 [I 2022-10-11 02:00:21] ...trap/engines/engine.py.224: train: epoch 93 | batch 1460/2304 [I 2022-10-11 02:00:21] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:12:00 [I 2022-10-11 02:00:21] ...trap/engines/engine.py.229: process: 0.85119 | load: 0.00305 [I 2022-10-11 02:00:21] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-11 02:00:21] ...optimizers/trijoint.py.151: total_norm: 13.252594 [I 2022-10-11 02:00:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1470/2304 [I 2022-10-11 02:00:30] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:57 [I 2022-10-11 02:00:30] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00318 [I 2022-10-11 02:00:30] ...trap/engines/engine.py.230: loss: 0.12038 [I 2022-10-11 02:00:30] ...optimizers/trijoint.py.151: total_norm: 11.140076 [I 2022-10-11 02:00:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1480/2304 [I 2022-10-11 02:00:39] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:46 [I 2022-10-11 02:00:39] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00363 [I 2022-10-11 02:00:39] ...trap/engines/engine.py.230: loss: 0.18410 [I 2022-10-11 02:00:39] ...optimizers/trijoint.py.151: total_norm: 29.457388 [I 2022-10-11 02:00:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 1490/2304 [I 2022-10-11 02:00:47] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:36 [I 2022-10-11 02:00:47] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00305 [I 2022-10-11 02:00:47] ...trap/engines/engine.py.230: loss: 0.21078 [I 2022-10-11 02:00:47] ...optimizers/trijoint.py.151: total_norm: 37.259651 [I 2022-10-11 02:00:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1500/2304 [I 2022-10-11 02:00:56] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:26 [I 2022-10-11 02:00:56] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00301 [I 2022-10-11 02:00:56] ...trap/engines/engine.py.230: loss: 0.16869 [I 2022-10-11 02:00:56] ...optimizers/trijoint.py.151: total_norm: 15.248693 [I 2022-10-11 02:01:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 1510/2304 [I 2022-10-11 02:01:04] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:18 [I 2022-10-11 02:01:04] ...trap/engines/engine.py.229: process: 0.85676 | load: 0.00299 [I 2022-10-11 02:01:04] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-11 02:01:04] ...optimizers/trijoint.py.151: total_norm: 16.489313 [I 2022-10-11 02:01:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1520/2304 [I 2022-10-11 02:01:13] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:11:12 [I 2022-10-11 02:01:13] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00300 [I 2022-10-11 02:01:13] ...trap/engines/engine.py.230: loss: 0.14824 [I 2022-10-11 02:01:13] ...optimizers/trijoint.py.151: total_norm: 15.989098 [I 2022-10-11 02:01:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1530/2304 [I 2022-10-11 02:01:22] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:11:04 [I 2022-10-11 02:01:22] ...trap/engines/engine.py.229: process: 0.85961 | load: 0.00301 [I 2022-10-11 02:01:22] ...trap/engines/engine.py.230: loss: 0.18544 [I 2022-10-11 02:01:22] ...optimizers/trijoint.py.151: total_norm: 23.029955 [I 2022-10-11 02:01:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1540/2304 [I 2022-10-11 02:01:30] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:52 [I 2022-10-11 02:01:30] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00300 [I 2022-10-11 02:01:30] ...trap/engines/engine.py.230: loss: 0.10806 [I 2022-10-11 02:01:30] ...optimizers/trijoint.py.151: total_norm: 25.458162 [I 2022-10-11 02:01:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1550/2304 [I 2022-10-11 02:01:39] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:45 [I 2022-10-11 02:01:39] ...trap/engines/engine.py.229: process: 0.85265 | load: 0.00302 [I 2022-10-11 02:01:39] ...trap/engines/engine.py.230: loss: 0.17866 [I 2022-10-11 02:01:39] ...optimizers/trijoint.py.151: total_norm: 18.469070 [I 2022-10-11 02:01:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 1560/2304 [I 2022-10-11 02:01:47] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:39 [I 2022-10-11 02:01:47] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00303 [I 2022-10-11 02:01:47] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-11 02:01:47] ...optimizers/trijoint.py.151: total_norm: 9.029662 [I 2022-10-11 02:01:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1570/2304 [I 2022-10-11 02:01:56] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:29 [I 2022-10-11 02:01:56] ...trap/engines/engine.py.229: process: 0.86239 | load: 0.00301 [I 2022-10-11 02:01:56] ...trap/engines/engine.py.230: loss: 0.09227 [I 2022-10-11 02:01:56] ...optimizers/trijoint.py.151: total_norm: 20.869976 [I 2022-10-11 02:02:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1580/2304 [I 2022-10-11 02:02:05] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:20 [I 2022-10-11 02:02:05] ...trap/engines/engine.py.229: process: 0.84171 | load: 0.00303 [I 2022-10-11 02:02:05] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-11 02:02:05] ...optimizers/trijoint.py.151: total_norm: 19.066658 [I 2022-10-11 02:02:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1590/2304 [I 2022-10-11 02:02:13] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:10:11 [I 2022-10-11 02:02:13] ...trap/engines/engine.py.229: process: 0.84862 | load: 0.00305 [I 2022-10-11 02:02:13] ...trap/engines/engine.py.230: loss: 0.10004 [I 2022-10-11 02:02:13] ...optimizers/trijoint.py.151: total_norm: 9.445930 [I 2022-10-11 02:02:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1600/2304 [I 2022-10-11 02:02:22] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:10:02 [I 2022-10-11 02:02:22] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00296 [I 2022-10-11 02:02:22] ...trap/engines/engine.py.230: loss: 0.09780 [I 2022-10-11 02:02:22] ...optimizers/trijoint.py.151: total_norm: 12.603385 [I 2022-10-11 02:02:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1610/2304 [I 2022-10-11 02:02:30] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:50 [I 2022-10-11 02:02:30] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00297 [I 2022-10-11 02:02:30] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-11 02:02:30] ...optimizers/trijoint.py.151: total_norm: 8.440144 [I 2022-10-11 02:02:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1620/2304 [I 2022-10-11 02:02:39] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:44 [I 2022-10-11 02:02:39] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00302 [I 2022-10-11 02:02:39] ...trap/engines/engine.py.230: loss: 0.08764 [I 2022-10-11 02:02:39] ...optimizers/trijoint.py.151: total_norm: 13.812651 [I 2022-10-11 02:02:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 1630/2304 [I 2022-10-11 02:02:47] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:36 [I 2022-10-11 02:02:47] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00300 [I 2022-10-11 02:02:47] ...trap/engines/engine.py.230: loss: 0.11496 [I 2022-10-11 02:02:47] ...optimizers/trijoint.py.151: total_norm: 16.839214 [I 2022-10-11 02:02:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1640/2304 [I 2022-10-11 02:02:56] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:09:26 [I 2022-10-11 02:02:56] ...trap/engines/engine.py.229: process: 0.85269 | load: 0.00301 [I 2022-10-11 02:02:56] ...trap/engines/engine.py.230: loss: 0.10301 [I 2022-10-11 02:02:56] ...optimizers/trijoint.py.151: total_norm: 12.900158 [I 2022-10-11 02:03:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1650/2304 [I 2022-10-11 02:03:05] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:09:19 [I 2022-10-11 02:03:05] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00297 [I 2022-10-11 02:03:05] ...trap/engines/engine.py.230: loss: 0.10497 [I 2022-10-11 02:03:05] ...optimizers/trijoint.py.151: total_norm: 19.363562 [I 2022-10-11 02:03:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1660/2304 [I 2022-10-11 02:03:13] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:12 [I 2022-10-11 02:03:13] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00307 [I 2022-10-11 02:03:13] ...trap/engines/engine.py.230: loss: 0.14303 [I 2022-10-11 02:03:13] ...optimizers/trijoint.py.151: total_norm: 13.792204 [I 2022-10-11 02:03:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1670/2304 [I 2022-10-11 02:03:22] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:09:02 [I 2022-10-11 02:03:22] ...trap/engines/engine.py.229: process: 0.86905 | load: 0.00318 [I 2022-10-11 02:03:22] ...trap/engines/engine.py.230: loss: 0.10861 [I 2022-10-11 02:03:22] ...optimizers/trijoint.py.151: total_norm: 34.167484 [I 2022-10-11 02:03:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1680/2304 [I 2022-10-11 02:03:30] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:54 [I 2022-10-11 02:03:30] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00313 [I 2022-10-11 02:03:30] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-11 02:03:30] ...optimizers/trijoint.py.151: total_norm: 11.088101 [I 2022-10-11 02:03:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1690/2304 [I 2022-10-11 02:03:39] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:45 [I 2022-10-11 02:03:39] ...trap/engines/engine.py.229: process: 0.85947 | load: 0.00298 [I 2022-10-11 02:03:39] ...trap/engines/engine.py.230: loss: 0.12132 [I 2022-10-11 02:03:39] ...optimizers/trijoint.py.151: total_norm: 25.308788 [I 2022-10-11 02:03:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1700/2304 [I 2022-10-11 02:03:48] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:37 [I 2022-10-11 02:03:48] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00301 [I 2022-10-11 02:03:48] ...trap/engines/engine.py.230: loss: 0.09365 [I 2022-10-11 02:03:48] ...optimizers/trijoint.py.151: total_norm: 11.812361 [I 2022-10-11 02:03:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1710/2304 [I 2022-10-11 02:03:56] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:08:29 [I 2022-10-11 02:03:56] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00312 [I 2022-10-11 02:03:56] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-11 02:03:56] ...optimizers/trijoint.py.151: total_norm: 18.770966 [I 2022-10-11 02:04:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1720/2304 [I 2022-10-11 02:04:05] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:23 [I 2022-10-11 02:04:05] ...trap/engines/engine.py.229: process: 0.86998 | load: 0.00306 [I 2022-10-11 02:04:05] ...trap/engines/engine.py.230: loss: 0.16422 [I 2022-10-11 02:04:05] ...optimizers/trijoint.py.151: total_norm: 21.251932 [I 2022-10-11 02:04:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1730/2304 [I 2022-10-11 02:04:13] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:08:11 [I 2022-10-11 02:04:13] ...trap/engines/engine.py.229: process: 0.86367 | load: 0.00303 [I 2022-10-11 02:04:13] ...trap/engines/engine.py.230: loss: 0.11365 [I 2022-10-11 02:04:13] ...optimizers/trijoint.py.151: total_norm: 7.985294 [I 2022-10-11 02:04:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1740/2304 [I 2022-10-11 02:04:22] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:08:00 [I 2022-10-11 02:04:22] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00298 [I 2022-10-11 02:04:22] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-11 02:04:22] ...optimizers/trijoint.py.151: total_norm: 12.136273 [I 2022-10-11 02:04:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 1750/2304 [I 2022-10-11 02:04:31] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:08:00 [I 2022-10-11 02:04:31] ...trap/engines/engine.py.229: process: 0.88509 | load: 0.00308 [I 2022-10-11 02:04:31] ...trap/engines/engine.py.230: loss: 0.18745 [I 2022-10-11 02:04:31] ...optimizers/trijoint.py.151: total_norm: 28.220028 [I 2022-10-11 02:04:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1760/2304 [I 2022-10-11 02:04:39] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:44 [I 2022-10-11 02:04:39] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00338 [I 2022-10-11 02:04:39] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-11 02:04:39] ...optimizers/trijoint.py.151: total_norm: 25.116138 [I 2022-10-11 02:04:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1770/2304 [I 2022-10-11 02:04:48] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:38 [I 2022-10-11 02:04:48] ...trap/engines/engine.py.229: process: 0.85638 | load: 0.00307 [I 2022-10-11 02:04:48] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-11 02:04:48] ...optimizers/trijoint.py.151: total_norm: 17.261177 [I 2022-10-11 02:04:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1780/2304 [I 2022-10-11 02:04:56] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:30 [I 2022-10-11 02:04:56] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00322 [I 2022-10-11 02:04:56] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-11 02:04:56] ...optimizers/trijoint.py.151: total_norm: 17.142494 [I 2022-10-11 02:05:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1790/2304 [I 2022-10-11 02:05:05] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:07:22 [I 2022-10-11 02:05:05] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00307 [I 2022-10-11 02:05:05] ...trap/engines/engine.py.230: loss: 0.11699 [I 2022-10-11 02:05:05] ...optimizers/trijoint.py.151: total_norm: 11.500842 [I 2022-10-11 02:05:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 1800/2304 [I 2022-10-11 02:05:14] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:07:13 [I 2022-10-11 02:05:14] ...trap/engines/engine.py.229: process: 0.86790 | load: 0.00314 [I 2022-10-11 02:05:14] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-11 02:05:14] ...optimizers/trijoint.py.151: total_norm: 20.574907 [I 2022-10-11 02:05:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1810/2304 [I 2022-10-11 02:05:22] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:07:05 [I 2022-10-11 02:05:22] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00315 [I 2022-10-11 02:05:22] ...trap/engines/engine.py.230: loss: 0.13329 [I 2022-10-11 02:05:22] ...optimizers/trijoint.py.151: total_norm: 16.623779 [I 2022-10-11 02:05:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 1820/2304 [I 2022-10-11 02:05:31] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:56 [I 2022-10-11 02:05:31] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00306 [I 2022-10-11 02:05:31] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-11 02:05:31] ...optimizers/trijoint.py.151: total_norm: 15.768414 [I 2022-10-11 02:05:39] ...trap/engines/engine.py.224: train: epoch 93 | batch 1830/2304 [I 2022-10-11 02:05:39] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:47 [I 2022-10-11 02:05:39] ...trap/engines/engine.py.229: process: 0.85553 | load: 0.00315 [I 2022-10-11 02:05:39] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-11 02:05:40] ...optimizers/trijoint.py.151: total_norm: 8.132807 [I 2022-10-11 02:05:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1840/2304 [I 2022-10-11 02:05:48] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:06:42 [I 2022-10-11 02:05:48] ...trap/engines/engine.py.229: process: 0.87113 | load: 0.00311 [I 2022-10-11 02:05:48] ...trap/engines/engine.py.230: loss: 0.09141 [I 2022-10-11 02:05:48] ...optimizers/trijoint.py.151: total_norm: 16.390152 [I 2022-10-11 02:05:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 1850/2304 [I 2022-10-11 02:05:57] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:33 [I 2022-10-11 02:05:57] ...trap/engines/engine.py.229: process: 0.87208 | load: 0.00314 [I 2022-10-11 02:05:57] ...trap/engines/engine.py.230: loss: 0.12181 [I 2022-10-11 02:05:57] ...optimizers/trijoint.py.151: total_norm: 31.960220 [I 2022-10-11 02:06:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1860/2304 [I 2022-10-11 02:06:05] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:06:22 [I 2022-10-11 02:06:05] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00314 [I 2022-10-11 02:06:05] ...trap/engines/engine.py.230: loss: 0.14890 [I 2022-10-11 02:06:05] ...optimizers/trijoint.py.151: total_norm: 17.724607 [I 2022-10-11 02:06:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 1870/2304 [I 2022-10-11 02:06:14] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:06:13 [I 2022-10-11 02:06:14] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00303 [I 2022-10-11 02:06:14] ...trap/engines/engine.py.230: loss: 0.09441 [I 2022-10-11 02:06:14] ...optimizers/trijoint.py.151: total_norm: 24.849144 [I 2022-10-11 02:06:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 1880/2304 [I 2022-10-11 02:06:23] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:06:08 [I 2022-10-11 02:06:23] ...trap/engines/engine.py.229: process: 0.88092 | load: 0.00307 [I 2022-10-11 02:06:23] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-11 02:06:23] ...optimizers/trijoint.py.151: total_norm: 8.017588 [I 2022-10-11 02:06:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 1890/2304 [I 2022-10-11 02:06:31] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:56 [I 2022-10-11 02:06:31] ...trap/engines/engine.py.229: process: 0.86628 | load: 0.00307 [I 2022-10-11 02:06:31] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-11 02:06:31] ...optimizers/trijoint.py.151: total_norm: 19.938240 [I 2022-10-11 02:06:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 1900/2304 [I 2022-10-11 02:06:40] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:49 [I 2022-10-11 02:06:40] ...trap/engines/engine.py.229: process: 0.86815 | load: 0.00315 [I 2022-10-11 02:06:40] ...trap/engines/engine.py.230: loss: 0.13226 [I 2022-10-11 02:06:40] ...optimizers/trijoint.py.151: total_norm: 19.625235 [I 2022-10-11 02:06:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 1910/2304 [I 2022-10-11 02:06:49] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:05:39 [I 2022-10-11 02:06:49] ...trap/engines/engine.py.229: process: 0.86986 | load: 0.00309 [I 2022-10-11 02:06:49] ...trap/engines/engine.py.230: loss: 0.14701 [I 2022-10-11 02:06:49] ...optimizers/trijoint.py.151: total_norm: 66.677742 [I 2022-10-11 02:06:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 1920/2304 [I 2022-10-11 02:06:57] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:05:30 [I 2022-10-11 02:06:57] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00304 [I 2022-10-11 02:06:57] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-11 02:06:57] ...optimizers/trijoint.py.151: total_norm: 14.009227 [I 2022-10-11 02:07:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 1930/2304 [I 2022-10-11 02:07:06] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:05:21 [I 2022-10-11 02:07:06] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00303 [I 2022-10-11 02:07:06] ...trap/engines/engine.py.230: loss: 0.11959 [I 2022-10-11 02:07:06] ...optimizers/trijoint.py.151: total_norm: 13.605594 [I 2022-10-11 02:07:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 1940/2304 [I 2022-10-11 02:07:15] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:05:13 [I 2022-10-11 02:07:15] ...trap/engines/engine.py.229: process: 0.86911 | load: 0.00307 [I 2022-10-11 02:07:15] ...trap/engines/engine.py.230: loss: 0.12282 [I 2022-10-11 02:07:15] ...optimizers/trijoint.py.151: total_norm: 15.406702 [I 2022-10-11 02:07:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 1950/2304 [I 2022-10-11 02:07:23] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:05:05 [I 2022-10-11 02:07:23] ...trap/engines/engine.py.229: process: 0.88162 | load: 0.00306 [I 2022-10-11 02:07:23] ...trap/engines/engine.py.230: loss: 0.15290 [I 2022-10-11 02:07:23] ...optimizers/trijoint.py.151: total_norm: 10.003632 [I 2022-10-11 02:07:32] ...trap/engines/engine.py.224: train: epoch 93 | batch 1960/2304 [I 2022-10-11 02:07:32] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:56 [I 2022-10-11 02:07:32] ...trap/engines/engine.py.229: process: 0.85741 | load: 0.00309 [I 2022-10-11 02:07:32] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-11 02:07:32] ...optimizers/trijoint.py.151: total_norm: 14.106525 [I 2022-10-11 02:07:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 1970/2304 [I 2022-10-11 02:07:40] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:45 [I 2022-10-11 02:07:40] ...trap/engines/engine.py.229: process: 0.84945 | load: 0.00312 [I 2022-10-11 02:07:40] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-11 02:07:40] ...optimizers/trijoint.py.151: total_norm: 18.704931 [I 2022-10-11 02:07:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 1980/2304 [I 2022-10-11 02:07:49] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:04:40 [I 2022-10-11 02:07:49] ...trap/engines/engine.py.229: process: 0.86977 | load: 0.00329 [I 2022-10-11 02:07:49] ...trap/engines/engine.py.230: loss: 0.13521 [I 2022-10-11 02:07:49] ...optimizers/trijoint.py.151: total_norm: 13.618173 [I 2022-10-11 02:07:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 1990/2304 [I 2022-10-11 02:07:58] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:04:30 [I 2022-10-11 02:07:58] ...trap/engines/engine.py.229: process: 0.86839 | load: 0.00317 [I 2022-10-11 02:07:58] ...trap/engines/engine.py.230: loss: 0.15737 [I 2022-10-11 02:07:58] ...optimizers/trijoint.py.151: total_norm: 11.936724 [I 2022-10-11 02:08:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 2000/2304 [I 2022-10-11 02:08:06] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:04:22 [I 2022-10-11 02:08:06] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00309 [I 2022-10-11 02:08:06] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-11 02:08:06] ...optimizers/trijoint.py.151: total_norm: 14.618953 [I 2022-10-11 02:08:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 2010/2304 [I 2022-10-11 02:08:15] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:04:13 [I 2022-10-11 02:08:15] ...trap/engines/engine.py.229: process: 0.86216 | load: 0.00315 [I 2022-10-11 02:08:15] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-11 02:08:15] ...optimizers/trijoint.py.151: total_norm: 14.926266 [I 2022-10-11 02:08:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 2020/2304 [I 2022-10-11 02:08:24] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:04:05 [I 2022-10-11 02:08:24] ...trap/engines/engine.py.229: process: 0.87968 | load: 0.00321 [I 2022-10-11 02:08:24] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-11 02:08:24] ...optimizers/trijoint.py.151: total_norm: 21.648939 [I 2022-10-11 02:08:32] ...trap/engines/engine.py.224: train: epoch 93 | batch 2030/2304 [I 2022-10-11 02:08:32] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:57 [I 2022-10-11 02:08:32] ...trap/engines/engine.py.229: process: 0.86590 | load: 0.00321 [I 2022-10-11 02:08:32] ...trap/engines/engine.py.230: loss: 0.13308 [I 2022-10-11 02:08:32] ...optimizers/trijoint.py.151: total_norm: 68.982254 [I 2022-10-11 02:08:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 2040/2304 [I 2022-10-11 02:08:41] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:48 [I 2022-10-11 02:08:41] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00302 [I 2022-10-11 02:08:41] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-11 02:08:41] ...optimizers/trijoint.py.151: total_norm: 10.673567 [I 2022-10-11 02:08:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 2050/2304 [I 2022-10-11 02:08:50] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:39 [I 2022-10-11 02:08:50] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00313 [I 2022-10-11 02:08:50] ...trap/engines/engine.py.230: loss: 0.10148 [I 2022-10-11 02:08:50] ...optimizers/trijoint.py.151: total_norm: 22.100412 [I 2022-10-11 02:08:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 2060/2304 [I 2022-10-11 02:08:58] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:30 [I 2022-10-11 02:08:58] ...trap/engines/engine.py.229: process: 0.86521 | load: 0.00311 [I 2022-10-11 02:08:58] ...trap/engines/engine.py.230: loss: 0.09659 [I 2022-10-11 02:08:58] ...optimizers/trijoint.py.151: total_norm: 11.289024 [I 2022-10-11 02:09:07] ...trap/engines/engine.py.224: train: epoch 93 | batch 2070/2304 [I 2022-10-11 02:09:07] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:21 [I 2022-10-11 02:09:07] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00310 [I 2022-10-11 02:09:07] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-11 02:09:07] ...optimizers/trijoint.py.151: total_norm: 16.437901 [I 2022-10-11 02:09:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 2080/2304 [I 2022-10-11 02:09:15] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:03:12 [I 2022-10-11 02:09:15] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00310 [I 2022-10-11 02:09:15] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-11 02:09:15] ...optimizers/trijoint.py.151: total_norm: 28.884972 [I 2022-10-11 02:09:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 2090/2304 [I 2022-10-11 02:09:24] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:03:04 [I 2022-10-11 02:09:24] ...trap/engines/engine.py.229: process: 0.85931 | load: 0.00316 [I 2022-10-11 02:09:24] ...trap/engines/engine.py.230: loss: 0.16553 [I 2022-10-11 02:09:24] ...optimizers/trijoint.py.151: total_norm: 16.752247 [I 2022-10-11 02:09:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 2100/2304 [I 2022-10-11 02:09:33] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:55 [I 2022-10-11 02:09:33] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00314 [I 2022-10-11 02:09:33] ...trap/engines/engine.py.230: loss: 0.10097 [I 2022-10-11 02:09:33] ...optimizers/trijoint.py.151: total_norm: 11.443732 [I 2022-10-11 02:09:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 2110/2304 [I 2022-10-11 02:09:41] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:47 [I 2022-10-11 02:09:41] ...trap/engines/engine.py.229: process: 0.87271 | load: 0.00305 [I 2022-10-11 02:09:41] ...trap/engines/engine.py.230: loss: 0.07689 [I 2022-10-11 02:09:41] ...optimizers/trijoint.py.151: total_norm: 9.256918 [I 2022-10-11 02:09:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 2120/2304 [I 2022-10-11 02:09:50] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:38 [I 2022-10-11 02:09:50] ...trap/engines/engine.py.229: process: 0.85842 | load: 0.00310 [I 2022-10-11 02:09:50] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-11 02:09:50] ...optimizers/trijoint.py.151: total_norm: 12.365133 [I 2022-10-11 02:09:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 2130/2304 [I 2022-10-11 02:09:59] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:29 [I 2022-10-11 02:09:59] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00306 [I 2022-10-11 02:09:59] ...trap/engines/engine.py.230: loss: 0.19038 [I 2022-10-11 02:09:59] ...optimizers/trijoint.py.151: total_norm: 25.713623 [I 2022-10-11 02:10:07] ...trap/engines/engine.py.224: train: epoch 93 | batch 2140/2304 [I 2022-10-11 02:10:07] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:02:21 [I 2022-10-11 02:10:07] ...trap/engines/engine.py.229: process: 0.87441 | load: 0.00316 [I 2022-10-11 02:10:07] ...trap/engines/engine.py.230: loss: 0.19129 [I 2022-10-11 02:10:07] ...optimizers/trijoint.py.151: total_norm: 12.722700 [I 2022-10-11 02:10:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 2150/2304 [I 2022-10-11 02:10:16] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:02:12 [I 2022-10-11 02:10:16] ...trap/engines/engine.py.229: process: 0.86604 | load: 0.00313 [I 2022-10-11 02:10:16] ...trap/engines/engine.py.230: loss: 0.13103 [I 2022-10-11 02:10:16] ...optimizers/trijoint.py.151: total_norm: 41.516964 [I 2022-10-11 02:10:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 2160/2304 [I 2022-10-11 02:10:25] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:02:04 [I 2022-10-11 02:10:25] ...trap/engines/engine.py.229: process: 0.86448 | load: 0.00300 [I 2022-10-11 02:10:25] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-11 02:10:25] ...optimizers/trijoint.py.151: total_norm: 9.583694 [I 2022-10-11 02:10:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 2170/2304 [I 2022-10-11 02:10:33] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:54 [I 2022-10-11 02:10:33] ...trap/engines/engine.py.229: process: 0.84546 | load: 0.00317 [I 2022-10-11 02:10:33] ...trap/engines/engine.py.230: loss: 0.08446 [I 2022-10-11 02:10:33] ...optimizers/trijoint.py.151: total_norm: 7.158157 [I 2022-10-11 02:10:42] ...trap/engines/engine.py.224: train: epoch 93 | batch 2180/2304 [I 2022-10-11 02:10:42] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:46 [I 2022-10-11 02:10:42] ...trap/engines/engine.py.229: process: 0.86119 | load: 0.00309 [I 2022-10-11 02:10:42] ...trap/engines/engine.py.230: loss: 0.12820 [I 2022-10-11 02:10:42] ...optimizers/trijoint.py.151: total_norm: 14.565648 [I 2022-10-11 02:10:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 2190/2304 [I 2022-10-11 02:10:50] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:37 [I 2022-10-11 02:10:50] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00312 [I 2022-10-11 02:10:50] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-11 02:10:50] ...optimizers/trijoint.py.151: total_norm: 12.015544 [I 2022-10-11 02:10:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 2200/2304 [I 2022-10-11 02:10:59] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:01:29 [I 2022-10-11 02:10:59] ...trap/engines/engine.py.229: process: 0.86183 | load: 0.00305 [I 2022-10-11 02:10:59] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-11 02:10:59] ...optimizers/trijoint.py.151: total_norm: 12.984001 [I 2022-10-11 02:11:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 2210/2304 [I 2022-10-11 02:11:08] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:01:21 [I 2022-10-11 02:11:08] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00306 [I 2022-10-11 02:11:08] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-11 02:11:08] ...optimizers/trijoint.py.151: total_norm: 12.565512 [I 2022-10-11 02:11:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 2220/2304 [I 2022-10-11 02:11:16] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:12 [I 2022-10-11 02:11:16] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00308 [I 2022-10-11 02:11:16] ...trap/engines/engine.py.230: loss: 0.08806 [I 2022-10-11 02:11:16] ...optimizers/trijoint.py.151: total_norm: 9.651745 [I 2022-10-11 02:11:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 2230/2304 [I 2022-10-11 02:11:25] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:01:03 [I 2022-10-11 02:11:25] ...trap/engines/engine.py.229: process: 0.84524 | load: 0.00314 [I 2022-10-11 02:11:25] ...trap/engines/engine.py.230: loss: 0.10765 [I 2022-10-11 02:11:25] ...optimizers/trijoint.py.151: total_norm: 23.317696 [I 2022-10-11 02:11:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 2240/2304 [I 2022-10-11 02:11:33] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:54 [I 2022-10-11 02:11:33] ...trap/engines/engine.py.229: process: 0.86246 | load: 0.00309 [I 2022-10-11 02:11:33] ...trap/engines/engine.py.230: loss: 0.07815 [I 2022-10-11 02:11:33] ...optimizers/trijoint.py.151: total_norm: 12.109511 [I 2022-10-11 02:11:42] ...trap/engines/engine.py.224: train: epoch 93 | batch 2250/2304 [I 2022-10-11 02:11:42] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:46 [I 2022-10-11 02:11:42] ...trap/engines/engine.py.229: process: 0.86913 | load: 0.00312 [I 2022-10-11 02:11:42] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-11 02:11:42] ...optimizers/trijoint.py.151: total_norm: 15.218971 [I 2022-10-11 02:11:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 2260/2304 [I 2022-10-11 02:11:51] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:37 [I 2022-10-11 02:11:51] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00318 [I 2022-10-11 02:11:51] ...trap/engines/engine.py.230: loss: 0.13202 [I 2022-10-11 02:11:51] ...optimizers/trijoint.py.151: total_norm: 18.170090 [I 2022-10-11 02:11:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 2270/2304 [I 2022-10-11 02:11:59] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:29 [I 2022-10-11 02:11:59] ...trap/engines/engine.py.229: process: 0.86435 | load: 0.00316 [I 2022-10-11 02:11:59] ...trap/engines/engine.py.230: loss: 0.13926 [I 2022-10-11 02:11:59] ...optimizers/trijoint.py.151: total_norm: 16.279875 [I 2022-10-11 02:12:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 2280/2304 [I 2022-10-11 02:12:08] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:20 [I 2022-10-11 02:12:08] ...trap/engines/engine.py.229: process: 0.86043 | load: 0.00314 [I 2022-10-11 02:12:08] ...trap/engines/engine.py.230: loss: 0.10987 [I 2022-10-11 02:12:08] ...optimizers/trijoint.py.151: total_norm: 20.894375 [I 2022-10-11 02:12:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 2290/2304 [I 2022-10-11 02:12:16] ...trap/engines/engine.py.228: elapsed: 0:33:18 | left: 0:00:11 [I 2022-10-11 02:12:16] ...trap/engines/engine.py.229: process: 0.85683 | load: 0.00308 [I 2022-10-11 02:12:16] ...trap/engines/engine.py.230: loss: 0.11933 [I 2022-10-11 02:12:16] ...optimizers/trijoint.py.151: total_norm: 12.533648 [I 2022-10-11 02:12:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 2300/2304 [I 2022-10-11 02:12:25] ...trap/engines/engine.py.228: elapsed: 0:33:26 | left: 0:00:03 [I 2022-10-11 02:12:25] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00019 [I 2022-10-11 02:12:25] ...trap/engines/engine.py.230: loss: 0.10348 [I 2022-10-11 02:12:25] ...optimizers/trijoint.py.151: total_norm: 10.949303 [I 2022-10-11 02:12:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 2304/2304 [I 2022-10-11 02:12:29] ...trap/engines/engine.py.228: elapsed: 0:33:30 | left: 0:00:00 [I 2022-10-11 02:12:29] ...trap/engines/engine.py.229: process: 0.86912 | load: 0.00019 [I 2022-10-11 02:12:29] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-11 02:12:29] ...optimizers/trijoint.py.151: total_norm: 15.535265 [S 2022-10-11 02:12:30] ...trap/engines/engine.py.129: train_epoch.epoch: 93 [S 2022-10-11 02:12:30] ...trap/engines/engine.py.129: train_epoch.loss: 0.12658876375883107 [I 2022-10-11 02:12:30] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 02:12:32] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 93 [I 2022-10-11 02:12:32] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 02:12:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 02:12:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 02:12:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 02:12:37] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 02:12:43] ...trap/engines/engine.py.321: eval: epoch 93 | batch 0/510 [I 2022-10-11 02:12:43] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 1:33:11 [I 2022-10-11 02:12:43] ...trap/engines/engine.py.326: process: 10.96331 | load: 3.40858 [I 2022-10-11 02:12:49] ...trap/engines/engine.py.321: eval: epoch 93 | batch 10/510 [I 2022-10-11 02:12:49] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:14:30 [I 2022-10-11 02:12:49] ...trap/engines/engine.py.326: process: 0.49121 | load: 0.00021 [I 2022-10-11 02:12:55] ...trap/engines/engine.py.321: eval: epoch 93 | batch 20/510 [I 2022-10-11 02:12:55] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:06:13 [I 2022-10-11 02:12:55] ...trap/engines/engine.py.326: process: 0.65472 | load: 0.00018 [I 2022-10-11 02:13:10] ...trap/engines/engine.py.321: eval: epoch 93 | batch 30/510 [I 2022-10-11 02:13:10] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:14:07 [I 2022-10-11 02:13:10] ...trap/engines/engine.py.326: process: 0.96021 | load: 0.00021 [I 2022-10-11 02:13:18] ...trap/engines/engine.py.321: eval: epoch 93 | batch 40/510 [I 2022-10-11 02:13:18] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:07:24 [I 2022-10-11 02:13:18] ...trap/engines/engine.py.326: process: 1.09460 | load: 0.00024 [I 2022-10-11 02:13:27] ...trap/engines/engine.py.321: eval: epoch 93 | batch 50/510 [I 2022-10-11 02:13:27] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:06:01 [I 2022-10-11 02:13:27] ...trap/engines/engine.py.326: process: 0.64609 | load: 0.00020 [I 2022-10-11 02:13:34] ...trap/engines/engine.py.321: eval: epoch 93 | batch 60/510 [I 2022-10-11 02:13:34] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:04:50 [I 2022-10-11 02:13:34] ...trap/engines/engine.py.326: process: 0.66152 | load: 0.00020 [I 2022-10-11 02:13:40] ...trap/engines/engine.py.321: eval: epoch 93 | batch 70/510 [I 2022-10-11 02:13:40] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:04:56 [I 2022-10-11 02:13:40] ...trap/engines/engine.py.326: process: 0.68188 | load: 0.00020 [I 2022-10-11 02:13:46] ...trap/engines/engine.py.321: eval: epoch 93 | batch 80/510 [I 2022-10-11 02:13:46] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:04:20 [I 2022-10-11 02:13:46] ...trap/engines/engine.py.326: process: 0.57110 | load: 0.00019 [I 2022-10-11 02:13:58] ...trap/engines/engine.py.321: eval: epoch 93 | batch 90/510 [I 2022-10-11 02:13:58] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:05:34 [I 2022-10-11 02:13:58] ...trap/engines/engine.py.326: process: 0.55184 | load: 0.00027 [I 2022-10-11 02:14:03] ...trap/engines/engine.py.321: eval: epoch 93 | batch 100/510 [I 2022-10-11 02:14:03] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:03:46 [I 2022-10-11 02:14:03] ...trap/engines/engine.py.326: process: 0.31170 | load: 0.00019 [I 2022-10-11 02:14:07] ...trap/engines/engine.py.321: eval: epoch 93 | batch 110/510 [I 2022-10-11 02:14:07] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:02:17 [I 2022-10-11 02:14:07] ...trap/engines/engine.py.326: process: 0.31514 | load: 0.00020 [I 2022-10-11 02:14:10] ...trap/engines/engine.py.321: eval: epoch 93 | batch 120/510 [I 2022-10-11 02:14:10] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:02:05 [I 2022-10-11 02:14:10] ...trap/engines/engine.py.326: process: 0.31233 | load: 0.00020 [I 2022-10-11 02:14:13] ...trap/engines/engine.py.321: eval: epoch 93 | batch 130/510 [I 2022-10-11 02:14:13] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:02:02 [I 2022-10-11 02:14:13] ...trap/engines/engine.py.326: process: 0.31996 | load: 0.00021 [I 2022-10-11 02:14:16] ...trap/engines/engine.py.321: eval: epoch 93 | batch 140/510 [I 2022-10-11 02:14:16] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:59 [I 2022-10-11 02:14:16] ...trap/engines/engine.py.326: process: 0.32409 | load: 0.00021 [I 2022-10-11 02:14:20] ...trap/engines/engine.py.321: eval: epoch 93 | batch 150/510 [I 2022-10-11 02:14:20] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:56 [I 2022-10-11 02:14:20] ...trap/engines/engine.py.326: process: 0.32983 | load: 0.00020 [I 2022-10-11 02:14:23] ...trap/engines/engine.py.321: eval: epoch 93 | batch 160/510 [I 2022-10-11 02:14:23] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:53 [I 2022-10-11 02:14:23] ...trap/engines/engine.py.326: process: 0.32057 | load: 0.00020 [I 2022-10-11 02:14:26] ...trap/engines/engine.py.321: eval: epoch 93 | batch 170/510 [I 2022-10-11 02:14:26] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:49 [I 2022-10-11 02:14:26] ...trap/engines/engine.py.326: process: 0.32452 | load: 0.00020 [I 2022-10-11 02:14:29] ...trap/engines/engine.py.321: eval: epoch 93 | batch 180/510 [I 2022-10-11 02:14:29] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:45 [I 2022-10-11 02:14:29] ...trap/engines/engine.py.326: process: 0.32496 | load: 0.00021 [I 2022-10-11 02:14:32] ...trap/engines/engine.py.321: eval: epoch 93 | batch 190/510 [I 2022-10-11 02:14:32] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:43 [I 2022-10-11 02:14:32] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00022 [I 2022-10-11 02:14:36] ...trap/engines/engine.py.321: eval: epoch 93 | batch 200/510 [I 2022-10-11 02:14:36] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:40 [I 2022-10-11 02:14:36] ...trap/engines/engine.py.326: process: 0.32560 | load: 0.00020 [I 2022-10-11 02:14:39] ...trap/engines/engine.py.321: eval: epoch 93 | batch 210/510 [I 2022-10-11 02:14:39] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:37 [I 2022-10-11 02:14:39] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00021 [I 2022-10-11 02:14:42] ...trap/engines/engine.py.321: eval: epoch 93 | batch 220/510 [I 2022-10-11 02:14:42] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:33 [I 2022-10-11 02:14:42] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00020 [I 2022-10-11 02:14:45] ...trap/engines/engine.py.321: eval: epoch 93 | batch 230/510 [I 2022-10-11 02:14:45] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:01:30 [I 2022-10-11 02:14:45] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00021 [I 2022-10-11 02:14:49] ...trap/engines/engine.py.321: eval: epoch 93 | batch 240/510 [I 2022-10-11 02:14:49] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:01:26 [I 2022-10-11 02:14:49] ...trap/engines/engine.py.326: process: 0.32198 | load: 0.00021 [I 2022-10-11 02:14:52] ...trap/engines/engine.py.321: eval: epoch 93 | batch 250/510 [I 2022-10-11 02:14:52] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:01:23 [I 2022-10-11 02:14:52] ...trap/engines/engine.py.326: process: 0.31189 | load: 0.00022 [I 2022-10-11 02:14:55] ...trap/engines/engine.py.321: eval: epoch 93 | batch 260/510 [I 2022-10-11 02:14:55] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:01:21 [I 2022-10-11 02:14:55] ...trap/engines/engine.py.326: process: 0.32607 | load: 0.00022 [I 2022-10-11 02:14:58] ...trap/engines/engine.py.321: eval: epoch 93 | batch 270/510 [I 2022-10-11 02:14:58] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:01:17 [I 2022-10-11 02:14:58] ...trap/engines/engine.py.326: process: 0.33292 | load: 0.00025 [I 2022-10-11 02:15:02] ...trap/engines/engine.py.321: eval: epoch 93 | batch 280/510 [I 2022-10-11 02:15:02] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:01:13 [I 2022-10-11 02:15:02] ...trap/engines/engine.py.326: process: 0.32321 | load: 0.00022 [I 2022-10-11 02:15:05] ...trap/engines/engine.py.321: eval: epoch 93 | batch 290/510 [I 2022-10-11 02:15:05] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:01:10 [I 2022-10-11 02:15:05] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00021 [I 2022-10-11 02:15:08] ...trap/engines/engine.py.321: eval: epoch 93 | batch 300/510 [I 2022-10-11 02:15:08] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:01:08 [I 2022-10-11 02:15:08] ...trap/engines/engine.py.326: process: 0.32759 | load: 0.00020 [I 2022-10-11 02:15:11] ...trap/engines/engine.py.321: eval: epoch 93 | batch 310/510 [I 2022-10-11 02:15:11] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:01:04 [I 2022-10-11 02:15:11] ...trap/engines/engine.py.326: process: 0.32846 | load: 0.00022 [I 2022-10-11 02:15:14] ...trap/engines/engine.py.321: eval: epoch 93 | batch 320/510 [I 2022-10-11 02:15:14] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:01:02 [I 2022-10-11 02:15:14] ...trap/engines/engine.py.326: process: 0.33621 | load: 0.00021 [I 2022-10-11 02:15:18] ...trap/engines/engine.py.321: eval: epoch 93 | batch 330/510 [I 2022-10-11 02:15:18] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:58 [I 2022-10-11 02:15:18] ...trap/engines/engine.py.326: process: 0.32123 | load: 0.00020 [I 2022-10-11 02:15:21] ...trap/engines/engine.py.321: eval: epoch 93 | batch 340/510 [I 2022-10-11 02:15:21] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:55 [I 2022-10-11 02:15:21] ...trap/engines/engine.py.326: process: 0.32756 | load: 0.00021 [I 2022-10-11 02:15:24] ...trap/engines/engine.py.321: eval: epoch 93 | batch 350/510 [I 2022-10-11 02:15:24] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:51 [I 2022-10-11 02:15:24] ...trap/engines/engine.py.326: process: 0.32237 | load: 0.00018 [I 2022-10-11 02:15:27] ...trap/engines/engine.py.321: eval: epoch 93 | batch 360/510 [I 2022-10-11 02:15:27] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:48 [I 2022-10-11 02:15:27] ...trap/engines/engine.py.326: process: 0.31890 | load: 0.00021 [I 2022-10-11 02:15:31] ...trap/engines/engine.py.321: eval: epoch 93 | batch 370/510 [I 2022-10-11 02:15:31] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:45 [I 2022-10-11 02:15:31] ...trap/engines/engine.py.326: process: 0.32584 | load: 0.00020 [I 2022-10-11 02:15:34] ...trap/engines/engine.py.321: eval: epoch 93 | batch 380/510 [I 2022-10-11 02:15:34] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:42 [I 2022-10-11 02:15:34] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00021 [I 2022-10-11 02:15:37] ...trap/engines/engine.py.321: eval: epoch 93 | batch 390/510 [I 2022-10-11 02:15:37] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:38 [I 2022-10-11 02:15:37] ...trap/engines/engine.py.326: process: 0.31668 | load: 0.00023 [I 2022-10-11 02:15:40] ...trap/engines/engine.py.321: eval: epoch 93 | batch 400/510 [I 2022-10-11 02:15:40] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:35 [I 2022-10-11 02:15:40] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00021 [I 2022-10-11 02:15:44] ...trap/engines/engine.py.321: eval: epoch 93 | batch 410/510 [I 2022-10-11 02:15:44] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:32 [I 2022-10-11 02:15:44] ...trap/engines/engine.py.326: process: 0.32694 | load: 0.00021 [I 2022-10-11 02:15:47] ...trap/engines/engine.py.321: eval: epoch 93 | batch 420/510 [I 2022-10-11 02:15:47] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:28 [I 2022-10-11 02:15:47] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00021 [I 2022-10-11 02:15:50] ...trap/engines/engine.py.321: eval: epoch 93 | batch 430/510 [I 2022-10-11 02:15:50] ...trap/engines/engine.py.325: elapsed: 0:03:18 | left: 0:00:25 [I 2022-10-11 02:15:50] ...trap/engines/engine.py.326: process: 0.32675 | load: 0.00024 [I 2022-10-11 02:15:53] ...trap/engines/engine.py.321: eval: epoch 93 | batch 440/510 [I 2022-10-11 02:15:53] ...trap/engines/engine.py.325: elapsed: 0:03:21 | left: 0:00:22 [I 2022-10-11 02:15:53] ...trap/engines/engine.py.326: process: 0.32480 | load: 0.00020 [I 2022-10-11 02:15:57] ...trap/engines/engine.py.321: eval: epoch 93 | batch 450/510 [I 2022-10-11 02:15:57] ...trap/engines/engine.py.325: elapsed: 0:03:24 | left: 0:00:19 [I 2022-10-11 02:15:57] ...trap/engines/engine.py.326: process: 0.31604 | load: 0.00021 [I 2022-10-11 02:16:00] ...trap/engines/engine.py.321: eval: epoch 93 | batch 460/510 [I 2022-10-11 02:16:00] ...trap/engines/engine.py.325: elapsed: 0:03:28 | left: 0:00:15 [I 2022-10-11 02:16:00] ...trap/engines/engine.py.326: process: 0.31541 | load: 0.00021 [I 2022-10-11 02:16:03] ...trap/engines/engine.py.321: eval: epoch 93 | batch 470/510 [I 2022-10-11 02:16:03] ...trap/engines/engine.py.325: elapsed: 0:03:31 | left: 0:00:12 [I 2022-10-11 02:16:03] ...trap/engines/engine.py.326: process: 0.33586 | load: 0.00021 [I 2022-10-11 02:16:06] ...trap/engines/engine.py.321: eval: epoch 93 | batch 480/510 [I 2022-10-11 02:16:06] ...trap/engines/engine.py.325: elapsed: 0:03:34 | left: 0:00:09 [I 2022-10-11 02:16:06] ...trap/engines/engine.py.326: process: 0.32572 | load: 0.00021 [I 2022-10-11 02:16:09] ...trap/engines/engine.py.321: eval: epoch 93 | batch 490/510 [I 2022-10-11 02:16:09] ...trap/engines/engine.py.325: elapsed: 0:03:37 | left: 0:00:06 [I 2022-10-11 02:16:09] ...trap/engines/engine.py.326: process: 0.30728 | load: 0.00021 [I 2022-10-11 02:16:13] ...trap/engines/engine.py.321: eval: epoch 93 | batch 500/510 [I 2022-10-11 02:16:13] ...trap/engines/engine.py.325: elapsed: 0:03:41 | left: 0:00:03 [I 2022-10-11 02:16:13] ...trap/engines/engine.py.326: process: 0.31804 | load: 0.00021 [I 2022-10-11 02:16:16] ...trap/engines/engine.py.321: eval: epoch 93 | batch 510/510 [I 2022-10-11 02:16:16] ...trap/engines/engine.py.325: elapsed: 0:03:44 | left: 0:00:00 [I 2022-10-11 02:16:16] ...trap/engines/engine.py.326: process: 0.31449 | load: 0.00002 [S 2022-10-11 02:16:17] ...trap/engines/engine.py.132: eval_epoch.epoch: 93 [I 2022-10-11 02:16:17] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746900 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913400 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935400 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.741400 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908900 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934700 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012903 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008789 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008077 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010022 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008252 [S 2022-10-11 02:16:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007430 [I 2022-10-11 02:17:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 02:17:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 02:17:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 02:17:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 02:17:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 02:17:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 02:17:10] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 02:17:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 02:17:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 94 [I 2022-10-11 02:17:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 02:17:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 0/2304 [I 2022-10-11 02:17:17] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:08:30 [I 2022-10-11 02:17:17] ...trap/engines/engine.py.229: process: 3.34640 | load: 2.43220 [I 2022-10-11 02:17:17] ...trap/engines/engine.py.230: loss: 0.11058 [I 2022-10-11 02:17:17] ...optimizers/trijoint.py.151: total_norm: 18.941910 [I 2022-10-11 02:17:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 10/2304 [I 2022-10-11 02:17:26] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:43:07 [I 2022-10-11 02:17:26] ...trap/engines/engine.py.229: process: 0.85371 | load: 0.00581 [I 2022-10-11 02:17:26] ...trap/engines/engine.py.230: loss: 0.09829 [I 2022-10-11 02:17:26] ...optimizers/trijoint.py.151: total_norm: 13.050446 [I 2022-10-11 02:17:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 20/2304 [I 2022-10-11 02:17:35] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:48 [I 2022-10-11 02:17:35] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00613 [I 2022-10-11 02:17:35] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-11 02:17:35] ...optimizers/trijoint.py.151: total_norm: 14.153031 [I 2022-10-11 02:17:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 30/2304 [I 2022-10-11 02:17:43] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:58 [I 2022-10-11 02:17:43] ...trap/engines/engine.py.229: process: 0.88848 | load: 0.00454 [I 2022-10-11 02:17:43] ...trap/engines/engine.py.230: loss: 0.19846 [I 2022-10-11 02:17:43] ...optimizers/trijoint.py.151: total_norm: 33.056976 [I 2022-10-11 02:17:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 40/2304 [I 2022-10-11 02:17:52] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:31 [I 2022-10-11 02:17:52] ...trap/engines/engine.py.229: process: 0.86943 | load: 0.00372 [I 2022-10-11 02:17:52] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-11 02:17:52] ...optimizers/trijoint.py.151: total_norm: 18.020790 [I 2022-10-11 02:18:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 50/2304 [I 2022-10-11 02:18:01] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:32 [I 2022-10-11 02:18:01] ...trap/engines/engine.py.229: process: 0.86754 | load: 0.00430 [I 2022-10-11 02:18:01] ...trap/engines/engine.py.230: loss: 0.09067 [I 2022-10-11 02:18:01] ...optimizers/trijoint.py.151: total_norm: 10.981029 [I 2022-10-11 02:18:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 60/2304 [I 2022-10-11 02:18:09] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:28 [I 2022-10-11 02:18:09] ...trap/engines/engine.py.229: process: 0.87195 | load: 0.00341 [I 2022-10-11 02:18:09] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-11 02:18:09] ...optimizers/trijoint.py.151: total_norm: 39.240715 [I 2022-10-11 02:18:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 70/2304 [I 2022-10-11 02:18:18] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:11 [I 2022-10-11 02:18:18] ...trap/engines/engine.py.229: process: 0.86377 | load: 0.00387 [I 2022-10-11 02:18:18] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-11 02:18:18] ...optimizers/trijoint.py.151: total_norm: 25.601662 [I 2022-10-11 02:18:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 80/2304 [I 2022-10-11 02:18:27] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:55 [I 2022-10-11 02:18:27] ...trap/engines/engine.py.229: process: 0.86321 | load: 0.00415 [I 2022-10-11 02:18:27] ...trap/engines/engine.py.230: loss: 0.14638 [I 2022-10-11 02:18:27] ...optimizers/trijoint.py.151: total_norm: 13.508319 [I 2022-10-11 02:18:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 90/2304 [I 2022-10-11 02:18:35] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:42 [I 2022-10-11 02:18:35] ...trap/engines/engine.py.229: process: 0.85653 | load: 0.00333 [I 2022-10-11 02:18:35] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-11 02:18:35] ...optimizers/trijoint.py.151: total_norm: 15.395744 [I 2022-10-11 02:18:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 100/2304 [I 2022-10-11 02:18:44] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:40 [I 2022-10-11 02:18:44] ...trap/engines/engine.py.229: process: 0.86882 | load: 0.00331 [I 2022-10-11 02:18:44] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-11 02:18:44] ...optimizers/trijoint.py.151: total_norm: 9.039836 [I 2022-10-11 02:18:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 110/2304 [I 2022-10-11 02:18:52] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:30 [I 2022-10-11 02:18:52] ...trap/engines/engine.py.229: process: 0.85500 | load: 0.00336 [I 2022-10-11 02:18:53] ...trap/engines/engine.py.230: loss: 0.20692 [I 2022-10-11 02:18:53] ...optimizers/trijoint.py.151: total_norm: 21.829554 [I 2022-10-11 02:19:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 120/2304 [I 2022-10-11 02:19:01] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:21 [I 2022-10-11 02:19:01] ...trap/engines/engine.py.229: process: 0.87013 | load: 0.00398 [I 2022-10-11 02:19:01] ...trap/engines/engine.py.230: loss: 0.18916 [I 2022-10-11 02:19:01] ...optimizers/trijoint.py.151: total_norm: 11.159670 [I 2022-10-11 02:19:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 130/2304 [I 2022-10-11 02:19:10] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:26 [I 2022-10-11 02:19:10] ...trap/engines/engine.py.229: process: 0.87432 | load: 0.00341 [I 2022-10-11 02:19:10] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-11 02:19:10] ...optimizers/trijoint.py.151: total_norm: 24.640993 [I 2022-10-11 02:19:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 140/2304 [I 2022-10-11 02:19:18] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:31:11 [I 2022-10-11 02:19:18] ...trap/engines/engine.py.229: process: 0.87293 | load: 0.00343 [I 2022-10-11 02:19:18] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-11 02:19:18] ...optimizers/trijoint.py.151: total_norm: 9.076784 [I 2022-10-11 02:19:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 150/2304 [I 2022-10-11 02:19:27] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:31:03 [I 2022-10-11 02:19:27] ...trap/engines/engine.py.229: process: 0.86489 | load: 0.00320 [I 2022-10-11 02:19:27] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-11 02:19:27] ...optimizers/trijoint.py.151: total_norm: 8.628864 [I 2022-10-11 02:19:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 160/2304 [I 2022-10-11 02:19:36] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:59 [I 2022-10-11 02:19:36] ...trap/engines/engine.py.229: process: 0.87785 | load: 0.00347 [I 2022-10-11 02:19:36] ...trap/engines/engine.py.230: loss: 0.12386 [I 2022-10-11 02:19:36] ...optimizers/trijoint.py.151: total_norm: 15.458982 [I 2022-10-11 02:19:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 170/2304 [I 2022-10-11 02:19:44] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:37 [I 2022-10-11 02:19:44] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00332 [I 2022-10-11 02:19:44] ...trap/engines/engine.py.230: loss: 0.07612 [I 2022-10-11 02:19:44] ...optimizers/trijoint.py.151: total_norm: 8.947618 [I 2022-10-11 02:19:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 180/2304 [I 2022-10-11 02:19:53] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:45 [I 2022-10-11 02:19:53] ...trap/engines/engine.py.229: process: 0.87461 | load: 0.00348 [I 2022-10-11 02:19:53] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-11 02:19:53] ...optimizers/trijoint.py.151: total_norm: 15.399247 [I 2022-10-11 02:20:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 190/2304 [I 2022-10-11 02:20:02] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:17 [I 2022-10-11 02:20:02] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00329 [I 2022-10-11 02:20:02] ...trap/engines/engine.py.230: loss: 0.09404 [I 2022-10-11 02:20:02] ...optimizers/trijoint.py.151: total_norm: 16.017513 [I 2022-10-11 02:20:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 200/2304 [I 2022-10-11 02:20:10] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:23 [I 2022-10-11 02:20:10] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00332 [I 2022-10-11 02:20:10] ...trap/engines/engine.py.230: loss: 0.15738 [I 2022-10-11 02:20:10] ...optimizers/trijoint.py.151: total_norm: 10.220283 [I 2022-10-11 02:20:19] ...trap/engines/engine.py.224: train: epoch 94 | batch 210/2304 [I 2022-10-11 02:20:19] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:30:13 [I 2022-10-11 02:20:19] ...trap/engines/engine.py.229: process: 0.86846 | load: 0.00334 [I 2022-10-11 02:20:19] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-11 02:20:19] ...optimizers/trijoint.py.151: total_norm: 44.313667 [I 2022-10-11 02:20:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 220/2304 [I 2022-10-11 02:20:28] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:53 [I 2022-10-11 02:20:28] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00341 [I 2022-10-11 02:20:28] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-11 02:20:28] ...optimizers/trijoint.py.151: total_norm: 22.387392 [I 2022-10-11 02:20:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 230/2304 [I 2022-10-11 02:20:36] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:43 [I 2022-10-11 02:20:36] ...trap/engines/engine.py.229: process: 0.84975 | load: 0.00310 [I 2022-10-11 02:20:36] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-11 02:20:36] ...optimizers/trijoint.py.151: total_norm: 22.356173 [I 2022-10-11 02:20:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 240/2304 [I 2022-10-11 02:20:45] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:29:49 [I 2022-10-11 02:20:45] ...trap/engines/engine.py.229: process: 0.87018 | load: 0.00353 [I 2022-10-11 02:20:45] ...trap/engines/engine.py.230: loss: 0.11075 [I 2022-10-11 02:20:45] ...optimizers/trijoint.py.151: total_norm: 17.416899 [I 2022-10-11 02:20:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 250/2304 [I 2022-10-11 02:20:54] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:28 [I 2022-10-11 02:20:54] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00306 [I 2022-10-11 02:20:54] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-11 02:20:54] ...optimizers/trijoint.py.151: total_norm: 13.180547 [I 2022-10-11 02:21:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 260/2304 [I 2022-10-11 02:21:02] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:33 [I 2022-10-11 02:21:02] ...trap/engines/engine.py.229: process: 0.87644 | load: 0.00324 [I 2022-10-11 02:21:02] ...trap/engines/engine.py.230: loss: 0.10515 [I 2022-10-11 02:21:02] ...optimizers/trijoint.py.151: total_norm: 10.174059 [I 2022-10-11 02:21:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 270/2304 [I 2022-10-11 02:21:11] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:13 [I 2022-10-11 02:21:11] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00313 [I 2022-10-11 02:21:11] ...trap/engines/engine.py.230: loss: 0.15836 [I 2022-10-11 02:21:11] ...optimizers/trijoint.py.151: total_norm: 14.733467 [I 2022-10-11 02:21:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 280/2304 [I 2022-10-11 02:21:20] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:29:19 [I 2022-10-11 02:21:20] ...trap/engines/engine.py.229: process: 0.87178 | load: 0.00336 [I 2022-10-11 02:21:20] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-11 02:21:20] ...optimizers/trijoint.py.151: total_norm: 18.974070 [I 2022-10-11 02:21:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 290/2304 [I 2022-10-11 02:21:28] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:28:56 [I 2022-10-11 02:21:28] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00307 [I 2022-10-11 02:21:28] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-11 02:21:28] ...optimizers/trijoint.py.151: total_norm: 16.377035 [I 2022-10-11 02:21:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 300/2304 [I 2022-10-11 02:21:37] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:35 [I 2022-10-11 02:21:37] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00350 [I 2022-10-11 02:21:37] ...trap/engines/engine.py.230: loss: 0.08292 [I 2022-10-11 02:21:37] ...optimizers/trijoint.py.151: total_norm: 31.167349 [I 2022-10-11 02:21:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 310/2304 [I 2022-10-11 02:21:46] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:27 [I 2022-10-11 02:21:46] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00311 [I 2022-10-11 02:21:46] ...trap/engines/engine.py.230: loss: 0.11481 [I 2022-10-11 02:21:46] ...optimizers/trijoint.py.151: total_norm: 9.832200 [I 2022-10-11 02:21:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 320/2304 [I 2022-10-11 02:21:54] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:28:17 [I 2022-10-11 02:21:54] ...trap/engines/engine.py.229: process: 0.86114 | load: 0.00290 [I 2022-10-11 02:21:54] ...trap/engines/engine.py.230: loss: 0.09638 [I 2022-10-11 02:21:54] ...optimizers/trijoint.py.151: total_norm: 10.029832 [I 2022-10-11 02:22:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 330/2304 [I 2022-10-11 02:22:03] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:03 [I 2022-10-11 02:22:03] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00311 [I 2022-10-11 02:22:03] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-11 02:22:03] ...optimizers/trijoint.py.151: total_norm: 15.106526 [I 2022-10-11 02:22:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 340/2304 [I 2022-10-11 02:22:11] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:05 [I 2022-10-11 02:22:11] ...trap/engines/engine.py.229: process: 0.87418 | load: 0.00295 [I 2022-10-11 02:22:11] ...trap/engines/engine.py.230: loss: 0.10693 [I 2022-10-11 02:22:11] ...optimizers/trijoint.py.151: total_norm: 12.588422 [I 2022-10-11 02:22:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 350/2304 [I 2022-10-11 02:22:20] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:34 [I 2022-10-11 02:22:20] ...trap/engines/engine.py.229: process: 0.85233 | load: 0.00296 [I 2022-10-11 02:22:20] ...trap/engines/engine.py.230: loss: 0.07930 [I 2022-10-11 02:22:20] ...optimizers/trijoint.py.151: total_norm: 8.194038 [I 2022-10-11 02:22:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 360/2304 [I 2022-10-11 02:22:28] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:32 [I 2022-10-11 02:22:28] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00298 [I 2022-10-11 02:22:28] ...trap/engines/engine.py.230: loss: 0.12218 [I 2022-10-11 02:22:28] ...optimizers/trijoint.py.151: total_norm: 19.863939 [I 2022-10-11 02:22:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 370/2304 [I 2022-10-11 02:22:37] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:32 [I 2022-10-11 02:22:37] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00298 [I 2022-10-11 02:22:37] ...trap/engines/engine.py.230: loss: 0.11799 [I 2022-10-11 02:22:37] ...optimizers/trijoint.py.151: total_norm: 27.759857 [I 2022-10-11 02:22:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 380/2304 [I 2022-10-11 02:22:45] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:29 [I 2022-10-11 02:22:45] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00304 [I 2022-10-11 02:22:45] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-11 02:22:45] ...optimizers/trijoint.py.151: total_norm: 13.908357 [I 2022-10-11 02:22:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 390/2304 [I 2022-10-11 02:22:54] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:14 [I 2022-10-11 02:22:54] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00297 [I 2022-10-11 02:22:54] ...trap/engines/engine.py.230: loss: 0.12469 [I 2022-10-11 02:22:54] ...optimizers/trijoint.py.151: total_norm: 17.928589 [I 2022-10-11 02:23:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 400/2304 [I 2022-10-11 02:23:03] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:08 [I 2022-10-11 02:23:03] ...trap/engines/engine.py.229: process: 0.85670 | load: 0.00294 [I 2022-10-11 02:23:03] ...trap/engines/engine.py.230: loss: 0.15591 [I 2022-10-11 02:23:03] ...optimizers/trijoint.py.151: total_norm: 22.063221 [I 2022-10-11 02:23:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 410/2304 [I 2022-10-11 02:23:11] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:39 [I 2022-10-11 02:23:11] ...trap/engines/engine.py.229: process: 0.83104 | load: 0.00300 [I 2022-10-11 02:23:11] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-11 02:23:11] ...optimizers/trijoint.py.151: total_norm: 12.078719 [I 2022-10-11 02:23:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 420/2304 [I 2022-10-11 02:23:20] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:49 [I 2022-10-11 02:23:20] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00302 [I 2022-10-11 02:23:20] ...trap/engines/engine.py.230: loss: 0.10185 [I 2022-10-11 02:23:20] ...optimizers/trijoint.py.151: total_norm: 18.946127 [I 2022-10-11 02:23:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 430/2304 [I 2022-10-11 02:23:28] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:41 [I 2022-10-11 02:23:28] ...trap/engines/engine.py.229: process: 0.86739 | load: 0.00306 [I 2022-10-11 02:23:28] ...trap/engines/engine.py.230: loss: 0.09116 [I 2022-10-11 02:23:28] ...optimizers/trijoint.py.151: total_norm: 9.198979 [I 2022-10-11 02:23:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 440/2304 [I 2022-10-11 02:23:37] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:57 [I 2022-10-11 02:23:37] ...trap/engines/engine.py.229: process: 0.88498 | load: 0.00301 [I 2022-10-11 02:23:37] ...trap/engines/engine.py.230: loss: 0.14096 [I 2022-10-11 02:23:37] ...optimizers/trijoint.py.151: total_norm: 10.663155 [I 2022-10-11 02:23:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 450/2304 [I 2022-10-11 02:23:46] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:41 [I 2022-10-11 02:23:46] ...trap/engines/engine.py.229: process: 0.86108 | load: 0.00294 [I 2022-10-11 02:23:46] ...trap/engines/engine.py.230: loss: 0.12450 [I 2022-10-11 02:23:46] ...optimizers/trijoint.py.151: total_norm: 10.661304 [I 2022-10-11 02:23:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 460/2304 [I 2022-10-11 02:23:54] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:26:11 [I 2022-10-11 02:23:54] ...trap/engines/engine.py.229: process: 0.85662 | load: 0.00306 [I 2022-10-11 02:23:54] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-11 02:23:54] ...optimizers/trijoint.py.151: total_norm: 11.532348 [I 2022-10-11 02:24:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 470/2304 [I 2022-10-11 02:24:03] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:26:04 [I 2022-10-11 02:24:03] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00298 [I 2022-10-11 02:24:03] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-11 02:24:03] ...optimizers/trijoint.py.151: total_norm: 12.516292 [I 2022-10-11 02:24:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 480/2304 [I 2022-10-11 02:24:11] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:51 [I 2022-10-11 02:24:11] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00293 [I 2022-10-11 02:24:11] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-11 02:24:11] ...optimizers/trijoint.py.151: total_norm: 20.124426 [I 2022-10-11 02:24:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 490/2304 [I 2022-10-11 02:24:20] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:56 [I 2022-10-11 02:24:20] ...trap/engines/engine.py.229: process: 0.87757 | load: 0.00302 [I 2022-10-11 02:24:20] ...trap/engines/engine.py.230: loss: 0.09547 [I 2022-10-11 02:24:20] ...optimizers/trijoint.py.151: total_norm: 14.223993 [I 2022-10-11 02:24:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 500/2304 [I 2022-10-11 02:24:28] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:55 [I 2022-10-11 02:24:28] ...trap/engines/engine.py.229: process: 0.86890 | load: 0.00314 [I 2022-10-11 02:24:28] ...trap/engines/engine.py.230: loss: 0.18629 [I 2022-10-11 02:24:28] ...optimizers/trijoint.py.151: total_norm: 12.625035 [I 2022-10-11 02:24:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 510/2304 [I 2022-10-11 02:24:37] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:55 [I 2022-10-11 02:24:37] ...trap/engines/engine.py.229: process: 0.86560 | load: 0.00305 [I 2022-10-11 02:24:37] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-11 02:24:37] ...optimizers/trijoint.py.151: total_norm: 22.564775 [I 2022-10-11 02:24:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 520/2304 [I 2022-10-11 02:24:46] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:25:43 [I 2022-10-11 02:24:46] ...trap/engines/engine.py.229: process: 0.86750 | load: 0.00325 [I 2022-10-11 02:24:46] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-11 02:24:46] ...optimizers/trijoint.py.151: total_norm: 23.444796 [I 2022-10-11 02:24:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 530/2304 [I 2022-10-11 02:24:54] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:34 [I 2022-10-11 02:24:54] ...trap/engines/engine.py.229: process: 0.85835 | load: 0.00307 [I 2022-10-11 02:24:54] ...trap/engines/engine.py.230: loss: 0.14154 [I 2022-10-11 02:24:54] ...optimizers/trijoint.py.151: total_norm: 11.790575 [I 2022-10-11 02:25:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 540/2304 [I 2022-10-11 02:25:03] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:22 [I 2022-10-11 02:25:03] ...trap/engines/engine.py.229: process: 0.86178 | load: 0.00322 [I 2022-10-11 02:25:03] ...trap/engines/engine.py.230: loss: 0.09764 [I 2022-10-11 02:25:03] ...optimizers/trijoint.py.151: total_norm: 8.463818 [I 2022-10-11 02:25:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 550/2304 [I 2022-10-11 02:25:12] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:19 [I 2022-10-11 02:25:12] ...trap/engines/engine.py.229: process: 0.86372 | load: 0.00308 [I 2022-10-11 02:25:12] ...trap/engines/engine.py.230: loss: 0.08253 [I 2022-10-11 02:25:12] ...optimizers/trijoint.py.151: total_norm: 14.962461 [I 2022-10-11 02:25:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 560/2304 [I 2022-10-11 02:25:20] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:17 [I 2022-10-11 02:25:20] ...trap/engines/engine.py.229: process: 0.87075 | load: 0.00327 [I 2022-10-11 02:25:20] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-11 02:25:20] ...optimizers/trijoint.py.151: total_norm: 18.971830 [I 2022-10-11 02:25:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 570/2304 [I 2022-10-11 02:25:29] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:25:02 [I 2022-10-11 02:25:29] ...trap/engines/engine.py.229: process: 0.86977 | load: 0.00308 [I 2022-10-11 02:25:29] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-11 02:25:29] ...optimizers/trijoint.py.151: total_norm: 21.493288 [I 2022-10-11 02:25:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 580/2304 [I 2022-10-11 02:25:38] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:59 [I 2022-10-11 02:25:38] ...trap/engines/engine.py.229: process: 0.87690 | load: 0.00314 [I 2022-10-11 02:25:38] ...trap/engines/engine.py.230: loss: 0.07094 [I 2022-10-11 02:25:38] ...optimizers/trijoint.py.151: total_norm: 10.326835 [I 2022-10-11 02:25:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 590/2304 [I 2022-10-11 02:25:46] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:37 [I 2022-10-11 02:25:46] ...trap/engines/engine.py.229: process: 0.85979 | load: 0.00310 [I 2022-10-11 02:25:46] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-11 02:25:47] ...optimizers/trijoint.py.151: total_norm: 19.338392 [I 2022-10-11 02:25:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 600/2304 [I 2022-10-11 02:25:55] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:29 [I 2022-10-11 02:25:55] ...trap/engines/engine.py.229: process: 0.85836 | load: 0.00314 [I 2022-10-11 02:25:55] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-11 02:25:55] ...optimizers/trijoint.py.151: total_norm: 11.131827 [I 2022-10-11 02:26:04] ...trap/engines/engine.py.224: train: epoch 94 | batch 610/2304 [I 2022-10-11 02:26:04] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:39 [I 2022-10-11 02:26:04] ...trap/engines/engine.py.229: process: 0.89419 | load: 0.00314 [I 2022-10-11 02:26:04] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-11 02:26:04] ...optimizers/trijoint.py.151: total_norm: 21.990215 [I 2022-10-11 02:26:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 620/2304 [I 2022-10-11 02:26:13] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:17 [I 2022-10-11 02:26:13] ...trap/engines/engine.py.229: process: 0.86748 | load: 0.00315 [I 2022-10-11 02:26:13] ...trap/engines/engine.py.230: loss: 0.10732 [I 2022-10-11 02:26:13] ...optimizers/trijoint.py.151: total_norm: 16.230650 [I 2022-10-11 02:26:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 630/2304 [I 2022-10-11 02:26:21] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:24:13 [I 2022-10-11 02:26:21] ...trap/engines/engine.py.229: process: 0.87291 | load: 0.00311 [I 2022-10-11 02:26:21] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-11 02:26:21] ...optimizers/trijoint.py.151: total_norm: 16.423527 [I 2022-10-11 02:26:30] ...trap/engines/engine.py.224: train: epoch 94 | batch 640/2304 [I 2022-10-11 02:26:30] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:24:05 [I 2022-10-11 02:26:30] ...trap/engines/engine.py.229: process: 0.87892 | load: 0.00317 [I 2022-10-11 02:26:30] ...trap/engines/engine.py.230: loss: 0.14684 [I 2022-10-11 02:26:30] ...optimizers/trijoint.py.151: total_norm: 16.378975 [I 2022-10-11 02:26:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 650/2304 [I 2022-10-11 02:26:39] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:54 [I 2022-10-11 02:26:39] ...trap/engines/engine.py.229: process: 0.86851 | load: 0.00331 [I 2022-10-11 02:26:39] ...trap/engines/engine.py.230: loss: 0.14864 [I 2022-10-11 02:26:39] ...optimizers/trijoint.py.151: total_norm: 20.972902 [I 2022-10-11 02:26:47] ...trap/engines/engine.py.224: train: epoch 94 | batch 660/2304 [I 2022-10-11 02:26:47] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:44 [I 2022-10-11 02:26:47] ...trap/engines/engine.py.229: process: 0.87083 | load: 0.00308 [I 2022-10-11 02:26:47] ...trap/engines/engine.py.230: loss: 0.10045 [I 2022-10-11 02:26:47] ...optimizers/trijoint.py.151: total_norm: 21.280689 [I 2022-10-11 02:26:56] ...trap/engines/engine.py.224: train: epoch 94 | batch 670/2304 [I 2022-10-11 02:26:56] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:40 [I 2022-10-11 02:26:56] ...trap/engines/engine.py.229: process: 0.86444 | load: 0.00320 [I 2022-10-11 02:26:56] ...trap/engines/engine.py.230: loss: 0.11252 [I 2022-10-11 02:26:56] ...optimizers/trijoint.py.151: total_norm: 13.309755 [I 2022-10-11 02:27:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 680/2304 [I 2022-10-11 02:27:05] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:29 [I 2022-10-11 02:27:05] ...trap/engines/engine.py.229: process: 0.86510 | load: 0.00331 [I 2022-10-11 02:27:05] ...trap/engines/engine.py.230: loss: 0.08834 [I 2022-10-11 02:27:05] ...optimizers/trijoint.py.151: total_norm: 12.390023 [I 2022-10-11 02:27:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 690/2304 [I 2022-10-11 02:27:13] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:23:18 [I 2022-10-11 02:27:13] ...trap/engines/engine.py.229: process: 0.86812 | load: 0.00319 [I 2022-10-11 02:27:13] ...trap/engines/engine.py.230: loss: 0.09637 [I 2022-10-11 02:27:13] ...optimizers/trijoint.py.151: total_norm: 12.105078 [I 2022-10-11 02:27:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 700/2304 [I 2022-10-11 02:27:22] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:23:06 [I 2022-10-11 02:27:22] ...trap/engines/engine.py.229: process: 0.87153 | load: 0.00312 [I 2022-10-11 02:27:22] ...trap/engines/engine.py.230: loss: 0.11410 [I 2022-10-11 02:27:22] ...optimizers/trijoint.py.151: total_norm: 18.712402 [I 2022-10-11 02:27:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 710/2304 [I 2022-10-11 02:27:31] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:23:02 [I 2022-10-11 02:27:31] ...trap/engines/engine.py.229: process: 0.86358 | load: 0.00310 [I 2022-10-11 02:27:31] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-11 02:27:31] ...optimizers/trijoint.py.151: total_norm: 12.779484 [I 2022-10-11 02:27:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 720/2304 [I 2022-10-11 02:27:39] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:47 [I 2022-10-11 02:27:39] ...trap/engines/engine.py.229: process: 0.86300 | load: 0.00318 [I 2022-10-11 02:27:40] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-11 02:27:40] ...optimizers/trijoint.py.151: total_norm: 30.056128 [I 2022-10-11 02:27:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 730/2304 [I 2022-10-11 02:27:48] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:34 [I 2022-10-11 02:27:48] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00313 [I 2022-10-11 02:27:48] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-11 02:27:48] ...optimizers/trijoint.py.151: total_norm: 17.525000 [I 2022-10-11 02:27:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 740/2304 [I 2022-10-11 02:27:57] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:28 [I 2022-10-11 02:27:57] ...trap/engines/engine.py.229: process: 0.85222 | load: 0.00316 [I 2022-10-11 02:27:57] ...trap/engines/engine.py.230: loss: 0.08319 [I 2022-10-11 02:27:57] ...optimizers/trijoint.py.151: total_norm: 14.063109 [I 2022-10-11 02:28:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 750/2304 [I 2022-10-11 02:28:06] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:34 [I 2022-10-11 02:28:06] ...trap/engines/engine.py.229: process: 0.86824 | load: 0.00305 [I 2022-10-11 02:28:06] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-11 02:28:06] ...optimizers/trijoint.py.151: total_norm: 17.881155 [I 2022-10-11 02:28:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 760/2304 [I 2022-10-11 02:28:14] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:22:19 [I 2022-10-11 02:28:14] ...trap/engines/engine.py.229: process: 0.88658 | load: 0.00310 [I 2022-10-11 02:28:14] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-11 02:28:14] ...optimizers/trijoint.py.151: total_norm: 79.680183 [I 2022-10-11 02:28:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 770/2304 [I 2022-10-11 02:28:23] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:22:00 [I 2022-10-11 02:28:23] ...trap/engines/engine.py.229: process: 0.86350 | load: 0.00291 [I 2022-10-11 02:28:23] ...trap/engines/engine.py.230: loss: 0.10523 [I 2022-10-11 02:28:23] ...optimizers/trijoint.py.151: total_norm: 18.239349 [I 2022-10-11 02:28:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 780/2304 [I 2022-10-11 02:28:31] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:40 [I 2022-10-11 02:28:31] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00299 [I 2022-10-11 02:28:31] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-11 02:28:31] ...optimizers/trijoint.py.151: total_norm: 15.744209 [I 2022-10-11 02:28:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 790/2304 [I 2022-10-11 02:28:40] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:48 [I 2022-10-11 02:28:40] ...trap/engines/engine.py.229: process: 0.88405 | load: 0.00305 [I 2022-10-11 02:28:40] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-11 02:28:40] ...optimizers/trijoint.py.151: total_norm: 29.951708 [I 2022-10-11 02:28:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 800/2304 [I 2022-10-11 02:28:49] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:26 [I 2022-10-11 02:28:49] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00307 [I 2022-10-11 02:28:49] ...trap/engines/engine.py.230: loss: 0.15663 [I 2022-10-11 02:28:49] ...optimizers/trijoint.py.151: total_norm: 21.053907 [I 2022-10-11 02:28:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 810/2304 [I 2022-10-11 02:28:57] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:32 [I 2022-10-11 02:28:57] ...trap/engines/engine.py.229: process: 0.86698 | load: 0.00297 [I 2022-10-11 02:28:57] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-11 02:28:57] ...optimizers/trijoint.py.151: total_norm: 40.947830 [I 2022-10-11 02:29:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 820/2304 [I 2022-10-11 02:29:06] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:25 [I 2022-10-11 02:29:06] ...trap/engines/engine.py.229: process: 0.87508 | load: 0.00315 [I 2022-10-11 02:29:06] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-11 02:29:06] ...optimizers/trijoint.py.151: total_norm: 13.135831 [I 2022-10-11 02:29:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 830/2304 [I 2022-10-11 02:29:15] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:12 [I 2022-10-11 02:29:15] ...trap/engines/engine.py.229: process: 0.86526 | load: 0.00306 [I 2022-10-11 02:29:15] ...trap/engines/engine.py.230: loss: 0.17044 [I 2022-10-11 02:29:15] ...optimizers/trijoint.py.151: total_norm: 30.567083 [I 2022-10-11 02:29:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 840/2304 [I 2022-10-11 02:29:23] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:21:04 [I 2022-10-11 02:29:23] ...trap/engines/engine.py.229: process: 0.87655 | load: 0.00316 [I 2022-10-11 02:29:23] ...trap/engines/engine.py.230: loss: 0.13709 [I 2022-10-11 02:29:23] ...optimizers/trijoint.py.151: total_norm: 13.600938 [I 2022-10-11 02:29:32] ...trap/engines/engine.py.224: train: epoch 94 | batch 850/2304 [I 2022-10-11 02:29:32] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:46 [I 2022-10-11 02:29:32] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00296 [I 2022-10-11 02:29:32] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-11 02:29:32] ...optimizers/trijoint.py.151: total_norm: 18.321213 [I 2022-10-11 02:29:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 860/2304 [I 2022-10-11 02:29:40] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:31 [I 2022-10-11 02:29:40] ...trap/engines/engine.py.229: process: 0.85459 | load: 0.00307 [I 2022-10-11 02:29:40] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-11 02:29:40] ...optimizers/trijoint.py.151: total_norm: 9.599463 [I 2022-10-11 02:29:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 870/2304 [I 2022-10-11 02:29:49] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:28 [I 2022-10-11 02:29:49] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00296 [I 2022-10-11 02:29:49] ...trap/engines/engine.py.230: loss: 0.11678 [I 2022-10-11 02:29:49] ...optimizers/trijoint.py.151: total_norm: 68.039932 [I 2022-10-11 02:29:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 880/2304 [I 2022-10-11 02:29:57] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:18 [I 2022-10-11 02:29:57] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00306 [I 2022-10-11 02:29:57] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-11 02:29:57] ...optimizers/trijoint.py.151: total_norm: 14.700712 [I 2022-10-11 02:30:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 890/2304 [I 2022-10-11 02:30:06] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:13 [I 2022-10-11 02:30:06] ...trap/engines/engine.py.229: process: 0.86213 | load: 0.00309 [I 2022-10-11 02:30:06] ...trap/engines/engine.py.230: loss: 0.18711 [I 2022-10-11 02:30:06] ...optimizers/trijoint.py.151: total_norm: 37.330715 [I 2022-10-11 02:30:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 900/2304 [I 2022-10-11 02:30:15] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:11 [I 2022-10-11 02:30:15] ...trap/engines/engine.py.229: process: 0.86875 | load: 0.00318 [I 2022-10-11 02:30:15] ...trap/engines/engine.py.230: loss: 0.16273 [I 2022-10-11 02:30:15] ...optimizers/trijoint.py.151: total_norm: 19.193220 [I 2022-10-11 02:30:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 910/2304 [I 2022-10-11 02:30:23] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:20:09 [I 2022-10-11 02:30:23] ...trap/engines/engine.py.229: process: 0.87175 | load: 0.00306 [I 2022-10-11 02:30:23] ...trap/engines/engine.py.230: loss: 0.14374 [I 2022-10-11 02:30:23] ...optimizers/trijoint.py.151: total_norm: 14.316461 [I 2022-10-11 02:30:32] ...trap/engines/engine.py.224: train: epoch 94 | batch 920/2304 [I 2022-10-11 02:30:32] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:56 [I 2022-10-11 02:30:32] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00307 [I 2022-10-11 02:30:32] ...trap/engines/engine.py.230: loss: 0.17951 [I 2022-10-11 02:30:32] ...optimizers/trijoint.py.151: total_norm: 13.030046 [I 2022-10-11 02:30:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 930/2304 [I 2022-10-11 02:30:41] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:43 [I 2022-10-11 02:30:41] ...trap/engines/engine.py.229: process: 0.87150 | load: 0.00320 [I 2022-10-11 02:30:41] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-11 02:30:41] ...optimizers/trijoint.py.151: total_norm: 13.663308 [I 2022-10-11 02:30:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 940/2304 [I 2022-10-11 02:30:49] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:37 [I 2022-10-11 02:30:49] ...trap/engines/engine.py.229: process: 0.85607 | load: 0.00315 [I 2022-10-11 02:30:49] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-11 02:30:49] ...optimizers/trijoint.py.151: total_norm: 11.530216 [I 2022-10-11 02:30:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 950/2304 [I 2022-10-11 02:30:58] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:46 [I 2022-10-11 02:30:58] ...trap/engines/engine.py.229: process: 0.87652 | load: 0.00331 [I 2022-10-11 02:30:58] ...trap/engines/engine.py.230: loss: 0.17940 [I 2022-10-11 02:30:58] ...optimizers/trijoint.py.151: total_norm: 40.153534 [I 2022-10-11 02:31:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 960/2304 [I 2022-10-11 02:31:07] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:29 [I 2022-10-11 02:31:07] ...trap/engines/engine.py.229: process: 0.86153 | load: 0.00332 [I 2022-10-11 02:31:07] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-11 02:31:07] ...optimizers/trijoint.py.151: total_norm: 7.568899 [I 2022-10-11 02:31:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 970/2304 [I 2022-10-11 02:31:16] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:20 [I 2022-10-11 02:31:16] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00355 [I 2022-10-11 02:31:16] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-11 02:31:16] ...optimizers/trijoint.py.151: total_norm: 14.830297 [I 2022-10-11 02:31:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 980/2304 [I 2022-10-11 02:31:24] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:08 [I 2022-10-11 02:31:24] ...trap/engines/engine.py.229: process: 0.86491 | load: 0.00310 [I 2022-10-11 02:31:24] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-11 02:31:24] ...optimizers/trijoint.py.151: total_norm: 34.138905 [I 2022-10-11 02:31:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 990/2304 [I 2022-10-11 02:31:33] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:58 [I 2022-10-11 02:31:33] ...trap/engines/engine.py.229: process: 0.86391 | load: 0.00381 [I 2022-10-11 02:31:33] ...trap/engines/engine.py.230: loss: 0.15288 [I 2022-10-11 02:31:33] ...optimizers/trijoint.py.151: total_norm: 12.611502 [I 2022-10-11 02:31:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1000/2304 [I 2022-10-11 02:31:42] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:45 [I 2022-10-11 02:31:42] ...trap/engines/engine.py.229: process: 0.86444 | load: 0.00314 [I 2022-10-11 02:31:42] ...trap/engines/engine.py.230: loss: 0.16941 [I 2022-10-11 02:31:42] ...optimizers/trijoint.py.151: total_norm: 16.720913 [I 2022-10-11 02:31:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 1010/2304 [I 2022-10-11 02:31:50] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:36 [I 2022-10-11 02:31:50] ...trap/engines/engine.py.229: process: 0.86408 | load: 0.00316 [I 2022-10-11 02:31:50] ...trap/engines/engine.py.230: loss: 0.11435 [I 2022-10-11 02:31:50] ...optimizers/trijoint.py.151: total_norm: 21.960794 [I 2022-10-11 02:31:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1020/2304 [I 2022-10-11 02:31:59] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:25 [I 2022-10-11 02:31:59] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00316 [I 2022-10-11 02:31:59] ...trap/engines/engine.py.230: loss: 0.14566 [I 2022-10-11 02:31:59] ...optimizers/trijoint.py.151: total_norm: 13.059388 [I 2022-10-11 02:32:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 1030/2304 [I 2022-10-11 02:32:08] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:25 [I 2022-10-11 02:32:08] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00308 [I 2022-10-11 02:32:08] ...trap/engines/engine.py.230: loss: 0.13297 [I 2022-10-11 02:32:08] ...optimizers/trijoint.py.151: total_norm: 17.483198 [I 2022-10-11 02:32:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 1040/2304 [I 2022-10-11 02:32:16] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:18:09 [I 2022-10-11 02:32:16] ...trap/engines/engine.py.229: process: 0.86403 | load: 0.00334 [I 2022-10-11 02:32:16] ...trap/engines/engine.py.230: loss: 0.11410 [I 2022-10-11 02:32:16] ...optimizers/trijoint.py.151: total_norm: 13.900022 [I 2022-10-11 02:32:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 1050/2304 [I 2022-10-11 02:32:25] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:01 [I 2022-10-11 02:32:25] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00325 [I 2022-10-11 02:32:25] ...trap/engines/engine.py.230: loss: 0.11552 [I 2022-10-11 02:32:25] ...optimizers/trijoint.py.151: total_norm: 71.354729 [I 2022-10-11 02:32:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 1060/2304 [I 2022-10-11 02:32:34] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:18:02 [I 2022-10-11 02:32:34] ...trap/engines/engine.py.229: process: 0.87354 | load: 0.00311 [I 2022-10-11 02:32:34] ...trap/engines/engine.py.230: loss: 0.09818 [I 2022-10-11 02:32:34] ...optimizers/trijoint.py.151: total_norm: 9.279614 [I 2022-10-11 02:32:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1070/2304 [I 2022-10-11 02:32:42] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:45 [I 2022-10-11 02:32:42] ...trap/engines/engine.py.229: process: 0.86624 | load: 0.00310 [I 2022-10-11 02:32:42] ...trap/engines/engine.py.230: loss: 0.09428 [I 2022-10-11 02:32:42] ...optimizers/trijoint.py.151: total_norm: 19.820227 [I 2022-10-11 02:32:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 1080/2304 [I 2022-10-11 02:32:51] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:40 [I 2022-10-11 02:32:51] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00311 [I 2022-10-11 02:32:51] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-11 02:32:51] ...optimizers/trijoint.py.151: total_norm: 13.425692 [I 2022-10-11 02:33:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 1090/2304 [I 2022-10-11 02:33:00] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:32 [I 2022-10-11 02:33:00] ...trap/engines/engine.py.229: process: 0.86581 | load: 0.00314 [I 2022-10-11 02:33:00] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-11 02:33:00] ...optimizers/trijoint.py.151: total_norm: 10.660784 [I 2022-10-11 02:33:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 1100/2304 [I 2022-10-11 02:33:08] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:23 [I 2022-10-11 02:33:08] ...trap/engines/engine.py.229: process: 0.86607 | load: 0.00307 [I 2022-10-11 02:33:08] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-11 02:33:08] ...optimizers/trijoint.py.151: total_norm: 14.103064 [I 2022-10-11 02:33:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 1110/2304 [I 2022-10-11 02:33:17] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:19 [I 2022-10-11 02:33:17] ...trap/engines/engine.py.229: process: 0.86591 | load: 0.00323 [I 2022-10-11 02:33:17] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-11 02:33:17] ...optimizers/trijoint.py.151: total_norm: 14.079413 [I 2022-10-11 02:33:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 1120/2304 [I 2022-10-11 02:33:26] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:15 [I 2022-10-11 02:33:26] ...trap/engines/engine.py.229: process: 0.87617 | load: 0.00324 [I 2022-10-11 02:33:26] ...trap/engines/engine.py.230: loss: 0.16148 [I 2022-10-11 02:33:26] ...optimizers/trijoint.py.151: total_norm: 26.423288 [I 2022-10-11 02:33:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 1130/2304 [I 2022-10-11 02:33:34] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:17:01 [I 2022-10-11 02:33:34] ...trap/engines/engine.py.229: process: 0.86285 | load: 0.00313 [I 2022-10-11 02:33:34] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-11 02:33:34] ...optimizers/trijoint.py.151: total_norm: 13.900697 [I 2022-10-11 02:33:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 1140/2304 [I 2022-10-11 02:33:43] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:16:52 [I 2022-10-11 02:33:43] ...trap/engines/engine.py.229: process: 0.88016 | load: 0.00317 [I 2022-10-11 02:33:43] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-11 02:33:43] ...optimizers/trijoint.py.151: total_norm: 44.341476 [I 2022-10-11 02:33:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1150/2304 [I 2022-10-11 02:33:52] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:42 [I 2022-10-11 02:33:52] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00316 [I 2022-10-11 02:33:52] ...trap/engines/engine.py.230: loss: 0.10228 [I 2022-10-11 02:33:52] ...optimizers/trijoint.py.151: total_norm: 18.296148 [I 2022-10-11 02:34:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 1160/2304 [I 2022-10-11 02:34:00] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:29 [I 2022-10-11 02:34:00] ...trap/engines/engine.py.229: process: 0.87186 | load: 0.00313 [I 2022-10-11 02:34:00] ...trap/engines/engine.py.230: loss: 0.18010 [I 2022-10-11 02:34:00] ...optimizers/trijoint.py.151: total_norm: 14.386516 [I 2022-10-11 02:34:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 1170/2304 [I 2022-10-11 02:34:09] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:16:15 [I 2022-10-11 02:34:09] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00299 [I 2022-10-11 02:34:09] ...trap/engines/engine.py.230: loss: 0.10273 [I 2022-10-11 02:34:09] ...optimizers/trijoint.py.151: total_norm: 12.530446 [I 2022-10-11 02:34:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1180/2304 [I 2022-10-11 02:34:18] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:16:02 [I 2022-10-11 02:34:18] ...trap/engines/engine.py.229: process: 0.85323 | load: 0.00294 [I 2022-10-11 02:34:18] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-11 02:34:18] ...optimizers/trijoint.py.151: total_norm: 10.560178 [I 2022-10-11 02:34:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 1190/2304 [I 2022-10-11 02:34:26] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:56 [I 2022-10-11 02:34:26] ...trap/engines/engine.py.229: process: 0.85028 | load: 0.00312 [I 2022-10-11 02:34:26] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-11 02:34:26] ...optimizers/trijoint.py.151: total_norm: 13.715683 [I 2022-10-11 02:34:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1200/2304 [I 2022-10-11 02:34:35] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:47 [I 2022-10-11 02:34:35] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00308 [I 2022-10-11 02:34:35] ...trap/engines/engine.py.230: loss: 0.16202 [I 2022-10-11 02:34:35] ...optimizers/trijoint.py.151: total_norm: 17.167757 [I 2022-10-11 02:34:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 1210/2304 [I 2022-10-11 02:34:44] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:35 [I 2022-10-11 02:34:44] ...trap/engines/engine.py.229: process: 0.84243 | load: 0.00296 [I 2022-10-11 02:34:44] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-11 02:34:44] ...optimizers/trijoint.py.151: total_norm: 11.276757 [I 2022-10-11 02:34:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1220/2304 [I 2022-10-11 02:34:52] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:21 [I 2022-10-11 02:34:52] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00297 [I 2022-10-11 02:34:52] ...trap/engines/engine.py.230: loss: 0.25189 [I 2022-10-11 02:34:52] ...optimizers/trijoint.py.151: total_norm: 18.999741 [I 2022-10-11 02:35:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1230/2304 [I 2022-10-11 02:35:01] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:16 [I 2022-10-11 02:35:01] ...trap/engines/engine.py.229: process: 0.85840 | load: 0.00296 [I 2022-10-11 02:35:01] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-11 02:35:01] ...optimizers/trijoint.py.151: total_norm: 23.546085 [I 2022-10-11 02:35:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 1240/2304 [I 2022-10-11 02:35:09] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:06 [I 2022-10-11 02:35:09] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00296 [I 2022-10-11 02:35:09] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-11 02:35:09] ...optimizers/trijoint.py.151: total_norm: 10.168279 [I 2022-10-11 02:35:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1250/2304 [I 2022-10-11 02:35:18] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:00 [I 2022-10-11 02:35:18] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00318 [I 2022-10-11 02:35:18] ...trap/engines/engine.py.230: loss: 0.09485 [I 2022-10-11 02:35:18] ...optimizers/trijoint.py.151: total_norm: 9.809831 [I 2022-10-11 02:35:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 1260/2304 [I 2022-10-11 02:35:26] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:51 [I 2022-10-11 02:35:26] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00298 [I 2022-10-11 02:35:26] ...trap/engines/engine.py.230: loss: 0.14723 [I 2022-10-11 02:35:26] ...optimizers/trijoint.py.151: total_norm: 8.404240 [I 2022-10-11 02:35:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1270/2304 [I 2022-10-11 02:35:35] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:14:45 [I 2022-10-11 02:35:35] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00303 [I 2022-10-11 02:35:35] ...trap/engines/engine.py.230: loss: 0.15970 [I 2022-10-11 02:35:35] ...optimizers/trijoint.py.151: total_norm: 16.438675 [I 2022-10-11 02:35:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 1280/2304 [I 2022-10-11 02:35:43] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:35 [I 2022-10-11 02:35:43] ...trap/engines/engine.py.229: process: 0.85716 | load: 0.00299 [I 2022-10-11 02:35:43] ...trap/engines/engine.py.230: loss: 0.20456 [I 2022-10-11 02:35:43] ...optimizers/trijoint.py.151: total_norm: 51.681679 [I 2022-10-11 02:35:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1290/2304 [I 2022-10-11 02:35:52] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:35 [I 2022-10-11 02:35:52] ...trap/engines/engine.py.229: process: 0.87373 | load: 0.00299 [I 2022-10-11 02:35:52] ...trap/engines/engine.py.230: loss: 0.13333 [I 2022-10-11 02:35:52] ...optimizers/trijoint.py.151: total_norm: 31.000410 [I 2022-10-11 02:36:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1300/2304 [I 2022-10-11 02:36:01] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:22 [I 2022-10-11 02:36:01] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00307 [I 2022-10-11 02:36:01] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-11 02:36:01] ...optimizers/trijoint.py.151: total_norm: 17.177204 [I 2022-10-11 02:36:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 1310/2304 [I 2022-10-11 02:36:09] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:22 [I 2022-10-11 02:36:09] ...trap/engines/engine.py.229: process: 0.86908 | load: 0.00307 [I 2022-10-11 02:36:09] ...trap/engines/engine.py.230: loss: 0.08452 [I 2022-10-11 02:36:09] ...optimizers/trijoint.py.151: total_norm: 9.937191 [I 2022-10-11 02:36:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1320/2304 [I 2022-10-11 02:36:18] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:14:04 [I 2022-10-11 02:36:18] ...trap/engines/engine.py.229: process: 0.86843 | load: 0.00316 [I 2022-10-11 02:36:18] ...trap/engines/engine.py.230: loss: 0.12599 [I 2022-10-11 02:36:18] ...optimizers/trijoint.py.151: total_norm: 18.439762 [I 2022-10-11 02:36:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 1330/2304 [I 2022-10-11 02:36:27] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:14:02 [I 2022-10-11 02:36:27] ...trap/engines/engine.py.229: process: 0.86122 | load: 0.00334 [I 2022-10-11 02:36:27] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-11 02:36:27] ...optimizers/trijoint.py.151: total_norm: 19.976791 [I 2022-10-11 02:36:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1340/2304 [I 2022-10-11 02:36:35] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:57 [I 2022-10-11 02:36:35] ...trap/engines/engine.py.229: process: 0.88326 | load: 0.00308 [I 2022-10-11 02:36:35] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-11 02:36:35] ...optimizers/trijoint.py.151: total_norm: 18.088766 [I 2022-10-11 02:36:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 1350/2304 [I 2022-10-11 02:36:44] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:48 [I 2022-10-11 02:36:44] ...trap/engines/engine.py.229: process: 0.87293 | load: 0.00305 [I 2022-10-11 02:36:44] ...trap/engines/engine.py.230: loss: 0.24770 [I 2022-10-11 02:36:44] ...optimizers/trijoint.py.151: total_norm: 24.916777 [I 2022-10-11 02:36:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 1360/2304 [I 2022-10-11 02:36:53] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:37 [I 2022-10-11 02:36:53] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00308 [I 2022-10-11 02:36:53] ...trap/engines/engine.py.230: loss: 0.08435 [I 2022-10-11 02:36:53] ...optimizers/trijoint.py.151: total_norm: 18.995165 [I 2022-10-11 02:37:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1370/2304 [I 2022-10-11 02:37:01] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:28 [I 2022-10-11 02:37:01] ...trap/engines/engine.py.229: process: 0.85610 | load: 0.00309 [I 2022-10-11 02:37:01] ...trap/engines/engine.py.230: loss: 0.12544 [I 2022-10-11 02:37:01] ...optimizers/trijoint.py.151: total_norm: 10.414645 [I 2022-10-11 02:37:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 1380/2304 [I 2022-10-11 02:37:10] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:19 [I 2022-10-11 02:37:10] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00311 [I 2022-10-11 02:37:10] ...trap/engines/engine.py.230: loss: 0.13942 [I 2022-10-11 02:37:10] ...optimizers/trijoint.py.151: total_norm: 12.558910 [I 2022-10-11 02:37:19] ...trap/engines/engine.py.224: train: epoch 94 | batch 1390/2304 [I 2022-10-11 02:37:19] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:08 [I 2022-10-11 02:37:19] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00311 [I 2022-10-11 02:37:19] ...trap/engines/engine.py.230: loss: 0.10767 [I 2022-10-11 02:37:19] ...optimizers/trijoint.py.151: total_norm: 9.339157 [I 2022-10-11 02:37:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 1400/2304 [I 2022-10-11 02:37:27] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:13:01 [I 2022-10-11 02:37:27] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00313 [I 2022-10-11 02:37:27] ...trap/engines/engine.py.230: loss: 0.11020 [I 2022-10-11 02:37:27] ...optimizers/trijoint.py.151: total_norm: 13.603301 [I 2022-10-11 02:37:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 1410/2304 [I 2022-10-11 02:37:36] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:53 [I 2022-10-11 02:37:36] ...trap/engines/engine.py.229: process: 0.85848 | load: 0.00313 [I 2022-10-11 02:37:36] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-11 02:37:36] ...optimizers/trijoint.py.151: total_norm: 10.663349 [I 2022-10-11 02:37:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 1420/2304 [I 2022-10-11 02:37:45] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:45 [I 2022-10-11 02:37:45] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00313 [I 2022-10-11 02:37:45] ...trap/engines/engine.py.230: loss: 0.14999 [I 2022-10-11 02:37:45] ...optimizers/trijoint.py.151: total_norm: 28.648962 [I 2022-10-11 02:37:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 1430/2304 [I 2022-10-11 02:37:54] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:39 [I 2022-10-11 02:37:54] ...trap/engines/engine.py.229: process: 0.87410 | load: 0.00313 [I 2022-10-11 02:37:54] ...trap/engines/engine.py.230: loss: 0.09828 [I 2022-10-11 02:37:54] ...optimizers/trijoint.py.151: total_norm: 10.460152 [I 2022-10-11 02:38:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 1440/2304 [I 2022-10-11 02:38:02] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:30 [I 2022-10-11 02:38:02] ...trap/engines/engine.py.229: process: 0.87256 | load: 0.00306 [I 2022-10-11 02:38:02] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-11 02:38:02] ...optimizers/trijoint.py.151: total_norm: 38.525955 [I 2022-10-11 02:38:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 1450/2304 [I 2022-10-11 02:38:11] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:20 [I 2022-10-11 02:38:11] ...trap/engines/engine.py.229: process: 0.86424 | load: 0.00311 [I 2022-10-11 02:38:11] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-11 02:38:11] ...optimizers/trijoint.py.151: total_norm: 8.680979 [I 2022-10-11 02:38:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 1460/2304 [I 2022-10-11 02:38:20] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:12:12 [I 2022-10-11 02:38:20] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00310 [I 2022-10-11 02:38:20] ...trap/engines/engine.py.230: loss: 0.14850 [I 2022-10-11 02:38:20] ...optimizers/trijoint.py.151: total_norm: 41.931259 [I 2022-10-11 02:38:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 1470/2304 [I 2022-10-11 02:38:28] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:12:03 [I 2022-10-11 02:38:28] ...trap/engines/engine.py.229: process: 0.87189 | load: 0.00328 [I 2022-10-11 02:38:28] ...trap/engines/engine.py.230: loss: 0.10267 [I 2022-10-11 02:38:28] ...optimizers/trijoint.py.151: total_norm: 15.984650 [I 2022-10-11 02:38:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 1480/2304 [I 2022-10-11 02:38:37] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:56 [I 2022-10-11 02:38:37] ...trap/engines/engine.py.229: process: 0.87092 | load: 0.00319 [I 2022-10-11 02:38:37] ...trap/engines/engine.py.230: loss: 0.12964 [I 2022-10-11 02:38:37] ...optimizers/trijoint.py.151: total_norm: 12.850105 [I 2022-10-11 02:38:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 1490/2304 [I 2022-10-11 02:38:46] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:47 [I 2022-10-11 02:38:46] ...trap/engines/engine.py.229: process: 0.87083 | load: 0.00314 [I 2022-10-11 02:38:46] ...trap/engines/engine.py.230: loss: 0.11104 [I 2022-10-11 02:38:46] ...optimizers/trijoint.py.151: total_norm: 16.621965 [I 2022-10-11 02:38:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 1500/2304 [I 2022-10-11 02:38:54] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:41 [I 2022-10-11 02:38:54] ...trap/engines/engine.py.229: process: 0.88566 | load: 0.00314 [I 2022-10-11 02:38:54] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-11 02:38:54] ...optimizers/trijoint.py.151: total_norm: 14.688327 [I 2022-10-11 02:39:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 1510/2304 [I 2022-10-11 02:39:03] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:32 [I 2022-10-11 02:39:03] ...trap/engines/engine.py.229: process: 0.88706 | load: 0.00317 [I 2022-10-11 02:39:03] ...trap/engines/engine.py.230: loss: 0.12706 [I 2022-10-11 02:39:03] ...optimizers/trijoint.py.151: total_norm: 14.067789 [I 2022-10-11 02:39:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 1520/2304 [I 2022-10-11 02:39:12] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:18 [I 2022-10-11 02:39:12] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00310 [I 2022-10-11 02:39:12] ...trap/engines/engine.py.230: loss: 0.08493 [I 2022-10-11 02:39:12] ...optimizers/trijoint.py.151: total_norm: 15.049389 [I 2022-10-11 02:39:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 1530/2304 [I 2022-10-11 02:39:21] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:05 [I 2022-10-11 02:39:21] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00306 [I 2022-10-11 02:39:21] ...trap/engines/engine.py.230: loss: 0.09773 [I 2022-10-11 02:39:21] ...optimizers/trijoint.py.151: total_norm: 8.815021 [I 2022-10-11 02:39:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 1540/2304 [I 2022-10-11 02:39:29] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:11:03 [I 2022-10-11 02:39:29] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00319 [I 2022-10-11 02:39:29] ...trap/engines/engine.py.230: loss: 0.12599 [I 2022-10-11 02:39:29] ...optimizers/trijoint.py.151: total_norm: 20.335018 [I 2022-10-11 02:39:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 1550/2304 [I 2022-10-11 02:39:38] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:51 [I 2022-10-11 02:39:38] ...trap/engines/engine.py.229: process: 0.86143 | load: 0.00308 [I 2022-10-11 02:39:38] ...trap/engines/engine.py.230: loss: 0.11641 [I 2022-10-11 02:39:38] ...optimizers/trijoint.py.151: total_norm: 12.252614 [I 2022-10-11 02:39:47] ...trap/engines/engine.py.224: train: epoch 94 | batch 1560/2304 [I 2022-10-11 02:39:47] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:41 [I 2022-10-11 02:39:47] ...trap/engines/engine.py.229: process: 0.86400 | load: 0.00305 [I 2022-10-11 02:39:47] ...trap/engines/engine.py.230: loss: 0.14073 [I 2022-10-11 02:39:47] ...optimizers/trijoint.py.151: total_norm: 13.073519 [I 2022-10-11 02:39:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 1570/2304 [I 2022-10-11 02:39:55] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:27 [I 2022-10-11 02:39:55] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00316 [I 2022-10-11 02:39:55] ...trap/engines/engine.py.230: loss: 0.12802 [I 2022-10-11 02:39:55] ...optimizers/trijoint.py.151: total_norm: 15.263746 [I 2022-10-11 02:40:04] ...trap/engines/engine.py.224: train: epoch 94 | batch 1580/2304 [I 2022-10-11 02:40:04] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:18 [I 2022-10-11 02:40:04] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00292 [I 2022-10-11 02:40:04] ...trap/engines/engine.py.230: loss: 0.10755 [I 2022-10-11 02:40:04] ...optimizers/trijoint.py.151: total_norm: 20.148672 [I 2022-10-11 02:40:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 1590/2304 [I 2022-10-11 02:40:12] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:16 [I 2022-10-11 02:40:12] ...trap/engines/engine.py.229: process: 0.87746 | load: 0.00306 [I 2022-10-11 02:40:12] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-11 02:40:12] ...optimizers/trijoint.py.151: total_norm: 11.538119 [I 2022-10-11 02:40:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 1600/2304 [I 2022-10-11 02:40:21] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:00 [I 2022-10-11 02:40:21] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00296 [I 2022-10-11 02:40:21] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-11 02:40:21] ...optimizers/trijoint.py.151: total_norm: 27.232372 [I 2022-10-11 02:40:30] ...trap/engines/engine.py.224: train: epoch 94 | batch 1610/2304 [I 2022-10-11 02:40:30] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:52 [I 2022-10-11 02:40:30] ...trap/engines/engine.py.229: process: 0.85900 | load: 0.00307 [I 2022-10-11 02:40:30] ...trap/engines/engine.py.230: loss: 0.09185 [I 2022-10-11 02:40:30] ...optimizers/trijoint.py.151: total_norm: 8.691756 [I 2022-10-11 02:40:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 1620/2304 [I 2022-10-11 02:40:38] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:47 [I 2022-10-11 02:40:38] ...trap/engines/engine.py.229: process: 0.85812 | load: 0.00310 [I 2022-10-11 02:40:38] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-11 02:40:38] ...optimizers/trijoint.py.151: total_norm: 12.397239 [I 2022-10-11 02:40:47] ...trap/engines/engine.py.224: train: epoch 94 | batch 1630/2304 [I 2022-10-11 02:40:47] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:40 [I 2022-10-11 02:40:47] ...trap/engines/engine.py.229: process: 0.86411 | load: 0.00329 [I 2022-10-11 02:40:47] ...trap/engines/engine.py.230: loss: 0.21088 [I 2022-10-11 02:40:47] ...optimizers/trijoint.py.151: total_norm: 54.103493 [I 2022-10-11 02:40:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 1640/2304 [I 2022-10-11 02:40:55] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:31 [I 2022-10-11 02:40:55] ...trap/engines/engine.py.229: process: 0.87157 | load: 0.00296 [I 2022-10-11 02:40:55] ...trap/engines/engine.py.230: loss: 0.16947 [I 2022-10-11 02:40:55] ...optimizers/trijoint.py.151: total_norm: 9.677499 [I 2022-10-11 02:41:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 1650/2304 [I 2022-10-11 02:41:22] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:25:30 [I 2022-10-11 02:41:22] ...trap/engines/engine.py.229: process: 0.84556 | load: 0.00297 [I 2022-10-11 02:41:22] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-11 02:41:22] ...optimizers/trijoint.py.151: total_norm: 11.862196 [I 2022-10-11 02:41:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 1660/2304 [I 2022-10-11 02:41:31] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:10:54 [I 2022-10-11 02:41:31] ...trap/engines/engine.py.229: process: 0.87342 | load: 0.00303 [I 2022-10-11 02:41:31] ...trap/engines/engine.py.230: loss: 0.09755 [I 2022-10-11 02:41:31] ...optimizers/trijoint.py.151: total_norm: 16.315781 [I 2022-10-11 02:41:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 1670/2304 [I 2022-10-11 02:41:39] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:09:16 [I 2022-10-11 02:41:39] ...trap/engines/engine.py.229: process: 0.86836 | load: 0.00342 [I 2022-10-11 02:41:39] ...trap/engines/engine.py.230: loss: 0.08835 [I 2022-10-11 02:41:39] ...optimizers/trijoint.py.151: total_norm: 9.775569 [I 2022-10-11 02:41:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 1680/2304 [I 2022-10-11 02:41:48] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:59 [I 2022-10-11 02:41:48] ...trap/engines/engine.py.229: process: 0.87986 | load: 0.00310 [I 2022-10-11 02:41:48] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-11 02:41:48] ...optimizers/trijoint.py.151: total_norm: 10.594785 [I 2022-10-11 02:41:56] ...trap/engines/engine.py.224: train: epoch 94 | batch 1690/2304 [I 2022-10-11 02:41:56] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:48 [I 2022-10-11 02:41:56] ...trap/engines/engine.py.229: process: 0.87229 | load: 0.00307 [I 2022-10-11 02:41:56] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-11 02:41:56] ...optimizers/trijoint.py.151: total_norm: 35.596615 [I 2022-10-11 02:42:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 1700/2304 [I 2022-10-11 02:42:05] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:38 [I 2022-10-11 02:42:05] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00349 [I 2022-10-11 02:42:05] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-11 02:42:05] ...optimizers/trijoint.py.151: total_norm: 19.205698 [I 2022-10-11 02:42:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 1710/2304 [I 2022-10-11 02:42:14] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:31 [I 2022-10-11 02:42:14] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00314 [I 2022-10-11 02:42:14] ...trap/engines/engine.py.230: loss: 0.11830 [I 2022-10-11 02:42:14] ...optimizers/trijoint.py.151: total_norm: 20.279667 [I 2022-10-11 02:42:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 1720/2304 [I 2022-10-11 02:42:22] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:08:25 [I 2022-10-11 02:42:22] ...trap/engines/engine.py.229: process: 0.85633 | load: 0.00316 [I 2022-10-11 02:42:22] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-11 02:42:22] ...optimizers/trijoint.py.151: total_norm: 14.865950 [I 2022-10-11 02:42:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 1730/2304 [I 2022-10-11 02:42:31] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:08:17 [I 2022-10-11 02:42:31] ...trap/engines/engine.py.229: process: 0.87602 | load: 0.00333 [I 2022-10-11 02:42:31] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-11 02:42:31] ...optimizers/trijoint.py.151: total_norm: 15.128414 [I 2022-10-11 02:42:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 1740/2304 [I 2022-10-11 02:42:40] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:08:04 [I 2022-10-11 02:42:40] ...trap/engines/engine.py.229: process: 0.85453 | load: 0.00310 [I 2022-10-11 02:42:40] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-11 02:42:40] ...optimizers/trijoint.py.151: total_norm: 7.784252 [I 2022-10-11 02:42:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 1750/2304 [I 2022-10-11 02:42:48] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:08:01 [I 2022-10-11 02:42:48] ...trap/engines/engine.py.229: process: 0.86273 | load: 0.00321 [I 2022-10-11 02:42:48] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-11 02:42:48] ...optimizers/trijoint.py.151: total_norm: 23.616755 [I 2022-10-11 02:42:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 1760/2304 [I 2022-10-11 02:42:57] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:48 [I 2022-10-11 02:42:57] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00309 [I 2022-10-11 02:42:57] ...trap/engines/engine.py.230: loss: 0.08218 [I 2022-10-11 02:42:57] ...optimizers/trijoint.py.151: total_norm: 7.610281 [I 2022-10-11 02:43:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 1770/2304 [I 2022-10-11 02:43:06] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:48 [I 2022-10-11 02:43:06] ...trap/engines/engine.py.229: process: 0.90478 | load: 0.00325 [I 2022-10-11 02:43:06] ...trap/engines/engine.py.230: loss: 0.10534 [I 2022-10-11 02:43:06] ...optimizers/trijoint.py.151: total_norm: 14.269543 [I 2022-10-11 02:43:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 1780/2304 [I 2022-10-11 02:43:15] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:34 [I 2022-10-11 02:43:15] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00314 [I 2022-10-11 02:43:15] ...trap/engines/engine.py.230: loss: 0.10024 [I 2022-10-11 02:43:15] ...optimizers/trijoint.py.151: total_norm: 32.114662 [I 2022-10-11 02:43:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 1790/2304 [I 2022-10-11 02:43:23] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:24 [I 2022-10-11 02:43:23] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00342 [I 2022-10-11 02:43:23] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-11 02:43:23] ...optimizers/trijoint.py.151: total_norm: 11.272938 [I 2022-10-11 02:43:32] ...trap/engines/engine.py.224: train: epoch 94 | batch 1800/2304 [I 2022-10-11 02:43:32] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:07:14 [I 2022-10-11 02:43:32] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00320 [I 2022-10-11 02:43:32] ...trap/engines/engine.py.230: loss: 0.09852 [I 2022-10-11 02:43:32] ...optimizers/trijoint.py.151: total_norm: 9.156698 [I 2022-10-11 02:43:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 1810/2304 [I 2022-10-11 02:43:41] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:07:09 [I 2022-10-11 02:43:41] ...trap/engines/engine.py.229: process: 0.86684 | load: 0.00316 [I 2022-10-11 02:43:41] ...trap/engines/engine.py.230: loss: 0.10531 [I 2022-10-11 02:43:41] ...optimizers/trijoint.py.151: total_norm: 12.833138 [I 2022-10-11 02:43:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 1820/2304 [I 2022-10-11 02:43:49] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:57 [I 2022-10-11 02:43:49] ...trap/engines/engine.py.229: process: 0.84862 | load: 0.00317 [I 2022-10-11 02:43:49] ...trap/engines/engine.py.230: loss: 0.11327 [I 2022-10-11 02:43:49] ...optimizers/trijoint.py.151: total_norm: 30.022818 [I 2022-10-11 02:43:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 1830/2304 [I 2022-10-11 02:43:58] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:52 [I 2022-10-11 02:43:58] ...trap/engines/engine.py.229: process: 0.86950 | load: 0.00324 [I 2022-10-11 02:43:58] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-11 02:43:58] ...optimizers/trijoint.py.151: total_norm: 12.519743 [I 2022-10-11 02:44:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 1840/2304 [I 2022-10-11 02:44:07] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:42 [I 2022-10-11 02:44:07] ...trap/engines/engine.py.229: process: 0.86738 | load: 0.00337 [I 2022-10-11 02:44:07] ...trap/engines/engine.py.230: loss: 0.10067 [I 2022-10-11 02:44:07] ...optimizers/trijoint.py.151: total_norm: 12.536452 [I 2022-10-11 02:44:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 1850/2304 [I 2022-10-11 02:44:15] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:34 [I 2022-10-11 02:44:15] ...trap/engines/engine.py.229: process: 0.87702 | load: 0.00314 [I 2022-10-11 02:44:15] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-11 02:44:15] ...optimizers/trijoint.py.151: total_norm: 23.550711 [I 2022-10-11 02:44:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 1860/2304 [I 2022-10-11 02:44:24] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:06:24 [I 2022-10-11 02:44:24] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00310 [I 2022-10-11 02:44:24] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-11 02:44:24] ...optimizers/trijoint.py.151: total_norm: 10.919380 [I 2022-10-11 02:44:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 1870/2304 [I 2022-10-11 02:44:33] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:06:14 [I 2022-10-11 02:44:33] ...trap/engines/engine.py.229: process: 0.86231 | load: 0.00310 [I 2022-10-11 02:44:33] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-11 02:44:33] ...optimizers/trijoint.py.151: total_norm: 12.867712 [I 2022-10-11 02:44:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 1880/2304 [I 2022-10-11 02:44:41] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:06:07 [I 2022-10-11 02:44:41] ...trap/engines/engine.py.229: process: 0.88006 | load: 0.00313 [I 2022-10-11 02:44:41] ...trap/engines/engine.py.230: loss: 0.11465 [I 2022-10-11 02:44:41] ...optimizers/trijoint.py.151: total_norm: 15.759764 [I 2022-10-11 02:44:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 1890/2304 [I 2022-10-11 02:44:50] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:06:00 [I 2022-10-11 02:44:50] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00313 [I 2022-10-11 02:44:50] ...trap/engines/engine.py.230: loss: 0.13619 [I 2022-10-11 02:44:50] ...optimizers/trijoint.py.151: total_norm: 24.826097 [I 2022-10-11 02:44:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1900/2304 [I 2022-10-11 02:44:59] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:50 [I 2022-10-11 02:44:59] ...trap/engines/engine.py.229: process: 0.88370 | load: 0.00330 [I 2022-10-11 02:44:59] ...trap/engines/engine.py.230: loss: 0.15781 [I 2022-10-11 02:44:59] ...optimizers/trijoint.py.151: total_norm: 13.371083 [I 2022-10-11 02:45:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 1910/2304 [I 2022-10-11 02:45:08] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:05:41 [I 2022-10-11 02:45:08] ...trap/engines/engine.py.229: process: 0.86780 | load: 0.00317 [I 2022-10-11 02:45:08] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-11 02:45:08] ...optimizers/trijoint.py.151: total_norm: 13.563055 [I 2022-10-11 02:45:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 1920/2304 [I 2022-10-11 02:45:16] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:05:30 [I 2022-10-11 02:45:16] ...trap/engines/engine.py.229: process: 0.86862 | load: 0.00326 [I 2022-10-11 02:45:16] ...trap/engines/engine.py.230: loss: 0.09959 [I 2022-10-11 02:45:16] ...optimizers/trijoint.py.151: total_norm: 13.581989 [I 2022-10-11 02:45:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 1930/2304 [I 2022-10-11 02:45:25] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:05:25 [I 2022-10-11 02:45:25] ...trap/engines/engine.py.229: process: 0.87351 | load: 0.00310 [I 2022-10-11 02:45:25] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-11 02:45:25] ...optimizers/trijoint.py.151: total_norm: 17.587849 [I 2022-10-11 02:45:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 1940/2304 [I 2022-10-11 02:45:34] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:05:13 [I 2022-10-11 02:45:34] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00315 [I 2022-10-11 02:45:34] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-11 02:45:34] ...optimizers/trijoint.py.151: total_norm: 22.486494 [I 2022-10-11 02:45:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1950/2304 [I 2022-10-11 02:45:42] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:05:05 [I 2022-10-11 02:45:42] ...trap/engines/engine.py.229: process: 0.86554 | load: 0.00307 [I 2022-10-11 02:45:42] ...trap/engines/engine.py.230: loss: 0.19794 [I 2022-10-11 02:45:42] ...optimizers/trijoint.py.151: total_norm: 20.668694 [I 2022-10-11 02:45:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 1960/2304 [I 2022-10-11 02:45:51] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:54 [I 2022-10-11 02:45:51] ...trap/engines/engine.py.229: process: 0.85824 | load: 0.00297 [I 2022-10-11 02:45:51] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-11 02:45:51] ...optimizers/trijoint.py.151: total_norm: 9.181858 [I 2022-10-11 02:45:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1970/2304 [I 2022-10-11 02:45:59] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:47 [I 2022-10-11 02:45:59] ...trap/engines/engine.py.229: process: 0.87116 | load: 0.00298 [I 2022-10-11 02:45:59] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-11 02:45:59] ...optimizers/trijoint.py.151: total_norm: 15.821033 [I 2022-10-11 02:46:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 1980/2304 [I 2022-10-11 02:46:08] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:04:39 [I 2022-10-11 02:46:08] ...trap/engines/engine.py.229: process: 0.87483 | load: 0.00310 [I 2022-10-11 02:46:08] ...trap/engines/engine.py.230: loss: 0.09904 [I 2022-10-11 02:46:08] ...optimizers/trijoint.py.151: total_norm: 13.535190 [I 2022-10-11 02:46:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 1990/2304 [I 2022-10-11 02:46:17] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:29 [I 2022-10-11 02:46:17] ...trap/engines/engine.py.229: process: 0.85912 | load: 0.00337 [I 2022-10-11 02:46:17] ...trap/engines/engine.py.230: loss: 0.09354 [I 2022-10-11 02:46:17] ...optimizers/trijoint.py.151: total_norm: 11.575658 [I 2022-10-11 02:46:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 2000/2304 [I 2022-10-11 02:46:25] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:04:21 [I 2022-10-11 02:46:25] ...trap/engines/engine.py.229: process: 0.86349 | load: 0.00298 [I 2022-10-11 02:46:25] ...trap/engines/engine.py.230: loss: 0.10704 [I 2022-10-11 02:46:25] ...optimizers/trijoint.py.151: total_norm: 10.569091 [I 2022-10-11 02:46:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 2010/2304 [I 2022-10-11 02:46:34] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:04:11 [I 2022-10-11 02:46:34] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00297 [I 2022-10-11 02:46:34] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-11 02:46:34] ...optimizers/trijoint.py.151: total_norm: 28.744541 [I 2022-10-11 02:46:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 2020/2304 [I 2022-10-11 02:46:42] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:04:03 [I 2022-10-11 02:46:42] ...trap/engines/engine.py.229: process: 0.86339 | load: 0.00297 [I 2022-10-11 02:46:42] ...trap/engines/engine.py.230: loss: 0.08042 [I 2022-10-11 02:46:42] ...optimizers/trijoint.py.151: total_norm: 10.283097 [I 2022-10-11 02:46:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 2030/2304 [I 2022-10-11 02:46:51] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:03:53 [I 2022-10-11 02:46:51] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00301 [I 2022-10-11 02:46:51] ...trap/engines/engine.py.230: loss: 0.19473 [I 2022-10-11 02:46:51] ...optimizers/trijoint.py.151: total_norm: 23.603666 [I 2022-10-11 02:47:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 2040/2304 [I 2022-10-11 02:47:00] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:46 [I 2022-10-11 02:47:00] ...trap/engines/engine.py.229: process: 0.87156 | load: 0.00297 [I 2022-10-11 02:47:00] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-11 02:47:00] ...optimizers/trijoint.py.151: total_norm: 12.555224 [I 2022-10-11 02:47:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 2050/2304 [I 2022-10-11 02:47:08] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:03:36 [I 2022-10-11 02:47:08] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00307 [I 2022-10-11 02:47:08] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-11 02:47:08] ...optimizers/trijoint.py.151: total_norm: 13.957635 [I 2022-10-11 02:47:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 2060/2304 [I 2022-10-11 02:47:17] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:29 [I 2022-10-11 02:47:17] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00298 [I 2022-10-11 02:47:17] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-11 02:47:17] ...optimizers/trijoint.py.151: total_norm: 41.398533 [I 2022-10-11 02:47:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 2070/2304 [I 2022-10-11 02:47:25] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:03:19 [I 2022-10-11 02:47:25] ...trap/engines/engine.py.229: process: 0.85807 | load: 0.00308 [I 2022-10-11 02:47:25] ...trap/engines/engine.py.230: loss: 0.09766 [I 2022-10-11 02:47:25] ...optimizers/trijoint.py.151: total_norm: 10.018929 [I 2022-10-11 02:47:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 2080/2304 [I 2022-10-11 02:47:34] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:03:12 [I 2022-10-11 02:47:34] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00339 [I 2022-10-11 02:47:34] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-11 02:47:34] ...optimizers/trijoint.py.151: total_norm: 58.468349 [I 2022-10-11 02:47:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 2090/2304 [I 2022-10-11 02:47:43] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:03:05 [I 2022-10-11 02:47:43] ...trap/engines/engine.py.229: process: 0.86678 | load: 0.00309 [I 2022-10-11 02:47:43] ...trap/engines/engine.py.230: loss: 0.09891 [I 2022-10-11 02:47:43] ...optimizers/trijoint.py.151: total_norm: 10.241003 [I 2022-10-11 02:47:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 2100/2304 [I 2022-10-11 02:47:51] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:56 [I 2022-10-11 02:47:51] ...trap/engines/engine.py.229: process: 0.85106 | load: 0.00308 [I 2022-10-11 02:47:51] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-11 02:47:51] ...optimizers/trijoint.py.151: total_norm: 22.810814 [I 2022-10-11 02:48:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 2110/2304 [I 2022-10-11 02:48:00] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:47 [I 2022-10-11 02:48:00] ...trap/engines/engine.py.229: process: 0.86036 | load: 0.00308 [I 2022-10-11 02:48:00] ...trap/engines/engine.py.230: loss: 0.09272 [I 2022-10-11 02:48:00] ...optimizers/trijoint.py.151: total_norm: 31.527967 [I 2022-10-11 02:48:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 2120/2304 [I 2022-10-11 02:48:09] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:38 [I 2022-10-11 02:48:09] ...trap/engines/engine.py.229: process: 0.86501 | load: 0.00309 [I 2022-10-11 02:48:09] ...trap/engines/engine.py.230: loss: 0.12087 [I 2022-10-11 02:48:09] ...optimizers/trijoint.py.151: total_norm: 11.126741 [I 2022-10-11 02:48:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 2130/2304 [I 2022-10-11 02:48:17] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:02:29 [I 2022-10-11 02:48:17] ...trap/engines/engine.py.229: process: 0.85901 | load: 0.00316 [I 2022-10-11 02:48:17] ...trap/engines/engine.py.230: loss: 0.14456 [I 2022-10-11 02:48:17] ...optimizers/trijoint.py.151: total_norm: 36.304779 [I 2022-10-11 02:48:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 2140/2304 [I 2022-10-11 02:48:26] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:02:21 [I 2022-10-11 02:48:26] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00317 [I 2022-10-11 02:48:26] ...trap/engines/engine.py.230: loss: 0.09588 [I 2022-10-11 02:48:26] ...optimizers/trijoint.py.151: total_norm: 9.675303 [I 2022-10-11 02:48:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 2150/2304 [I 2022-10-11 02:48:34] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:02:12 [I 2022-10-11 02:48:34] ...trap/engines/engine.py.229: process: 0.86411 | load: 0.00305 [I 2022-10-11 02:48:34] ...trap/engines/engine.py.230: loss: 0.08680 [I 2022-10-11 02:48:34] ...optimizers/trijoint.py.151: total_norm: 12.887750 [I 2022-10-11 02:48:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 2160/2304 [I 2022-10-11 02:48:43] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:02:04 [I 2022-10-11 02:48:43] ...trap/engines/engine.py.229: process: 0.86603 | load: 0.00369 [I 2022-10-11 02:48:43] ...trap/engines/engine.py.230: loss: 0.17914 [I 2022-10-11 02:48:43] ...optimizers/trijoint.py.151: total_norm: 21.352070 [I 2022-10-11 02:48:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 2170/2304 [I 2022-10-11 02:48:52] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:55 [I 2022-10-11 02:48:52] ...trap/engines/engine.py.229: process: 0.85022 | load: 0.00305 [I 2022-10-11 02:48:52] ...trap/engines/engine.py.230: loss: 0.09111 [I 2022-10-11 02:48:52] ...optimizers/trijoint.py.151: total_norm: 14.483690 [I 2022-10-11 02:49:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 2180/2304 [I 2022-10-11 02:49:01] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:47 [I 2022-10-11 02:49:01] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00401 [I 2022-10-11 02:49:01] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-11 02:49:01] ...optimizers/trijoint.py.151: total_norm: 26.990978 [I 2022-10-11 02:49:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 2190/2304 [I 2022-10-11 02:49:09] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:38 [I 2022-10-11 02:49:09] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00325 [I 2022-10-11 02:49:09] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-11 02:49:09] ...optimizers/trijoint.py.151: total_norm: 17.986048 [I 2022-10-11 02:49:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 2200/2304 [I 2022-10-11 02:49:18] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:01:29 [I 2022-10-11 02:49:18] ...trap/engines/engine.py.229: process: 0.85158 | load: 0.00309 [I 2022-10-11 02:49:18] ...trap/engines/engine.py.230: loss: 0.20137 [I 2022-10-11 02:49:18] ...optimizers/trijoint.py.151: total_norm: 16.654068 [I 2022-10-11 02:49:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 2210/2304 [I 2022-10-11 02:49:26] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:01:21 [I 2022-10-11 02:49:26] ...trap/engines/engine.py.229: process: 0.86497 | load: 0.00309 [I 2022-10-11 02:49:26] ...trap/engines/engine.py.230: loss: 0.11464 [I 2022-10-11 02:49:26] ...optimizers/trijoint.py.151: total_norm: 59.025795 [I 2022-10-11 02:49:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 2220/2304 [I 2022-10-11 02:49:35] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:01:12 [I 2022-10-11 02:49:35] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00309 [I 2022-10-11 02:49:35] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-11 02:49:35] ...optimizers/trijoint.py.151: total_norm: 16.555899 [I 2022-10-11 02:49:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 2230/2304 [I 2022-10-11 02:49:44] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:01:04 [I 2022-10-11 02:49:44] ...trap/engines/engine.py.229: process: 0.86605 | load: 0.00322 [I 2022-10-11 02:49:44] ...trap/engines/engine.py.230: loss: 0.12692 [I 2022-10-11 02:49:44] ...optimizers/trijoint.py.151: total_norm: 11.629149 [I 2022-10-11 02:49:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 2240/2304 [I 2022-10-11 02:49:52] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:55 [I 2022-10-11 02:49:52] ...trap/engines/engine.py.229: process: 0.86615 | load: 0.00307 [I 2022-10-11 02:49:52] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-11 02:49:52] ...optimizers/trijoint.py.151: total_norm: 8.448386 [I 2022-10-11 02:50:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 2250/2304 [I 2022-10-11 02:50:01] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:46 [I 2022-10-11 02:50:01] ...trap/engines/engine.py.229: process: 0.87899 | load: 0.00321 [I 2022-10-11 02:50:01] ...trap/engines/engine.py.230: loss: 0.24194 [I 2022-10-11 02:50:01] ...optimizers/trijoint.py.151: total_norm: 35.675774 [I 2022-10-11 02:50:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 2260/2304 [I 2022-10-11 02:50:10] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:38 [I 2022-10-11 02:50:10] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00337 [I 2022-10-11 02:50:10] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-11 02:50:10] ...optimizers/trijoint.py.151: total_norm: 16.366484 [I 2022-10-11 02:50:19] ...trap/engines/engine.py.224: train: epoch 94 | batch 2270/2304 [I 2022-10-11 02:50:19] ...trap/engines/engine.py.228: elapsed: 0:33:04 | left: 0:00:29 [I 2022-10-11 02:50:19] ...trap/engines/engine.py.229: process: 0.86535 | load: 0.00330 [I 2022-10-11 02:50:19] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-11 02:50:19] ...optimizers/trijoint.py.151: total_norm: 15.468891 [I 2022-10-11 02:50:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 2280/2304 [I 2022-10-11 02:50:27] ...trap/engines/engine.py.228: elapsed: 0:33:13 | left: 0:00:20 [I 2022-10-11 02:50:27] ...trap/engines/engine.py.229: process: 0.87147 | load: 0.00318 [I 2022-10-11 02:50:27] ...trap/engines/engine.py.230: loss: 0.15492 [I 2022-10-11 02:50:27] ...optimizers/trijoint.py.151: total_norm: 11.494120 [I 2022-10-11 02:50:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 2290/2304 [I 2022-10-11 02:50:36] ...trap/engines/engine.py.228: elapsed: 0:33:21 | left: 0:00:12 [I 2022-10-11 02:50:36] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00312 [I 2022-10-11 02:50:36] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-11 02:50:36] ...optimizers/trijoint.py.151: total_norm: 9.624318 [I 2022-10-11 02:50:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 2300/2304 [I 2022-10-11 02:50:45] ...trap/engines/engine.py.228: elapsed: 0:33:30 | left: 0:00:03 [I 2022-10-11 02:50:45] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00020 [I 2022-10-11 02:50:45] ...trap/engines/engine.py.230: loss: 0.11073 [I 2022-10-11 02:50:45] ...optimizers/trijoint.py.151: total_norm: 14.187206 [I 2022-10-11 02:50:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 2304/2304 [I 2022-10-11 02:50:48] ...trap/engines/engine.py.228: elapsed: 0:33:33 | left: 0:00:00 [I 2022-10-11 02:50:48] ...trap/engines/engine.py.229: process: 0.83847 | load: 0.00018 [I 2022-10-11 02:50:48] ...trap/engines/engine.py.230: loss: 0.09693 [I 2022-10-11 02:50:48] ...optimizers/trijoint.py.151: total_norm: 7.879032 [S 2022-10-11 02:50:49] ...trap/engines/engine.py.129: train_epoch.epoch: 94 [S 2022-10-11 02:50:49] ...trap/engines/engine.py.129: train_epoch.loss: 0.12708380004783 [I 2022-10-11 02:50:50] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 02:50:51] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 94 [I 2022-10-11 02:50:51] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 02:50:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 02:50:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 02:50:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 02:50:57] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 02:50:57] ...trap/engines/engine.py.321: eval: epoch 94 | batch 0/510 [I 2022-10-11 02:50:57] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:51:40 [I 2022-10-11 02:50:57] ...trap/engines/engine.py.326: process: 6.07971 | load: 5.48729 [I 2022-10-11 02:51:02] ...trap/engines/engine.py.321: eval: epoch 94 | batch 10/510 [I 2022-10-11 02:51:02] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:38 [I 2022-10-11 02:51:02] ...trap/engines/engine.py.326: process: 0.62121 | load: 0.00021 [I 2022-10-11 02:51:07] ...trap/engines/engine.py.321: eval: epoch 94 | batch 20/510 [I 2022-10-11 02:51:07] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:36 [I 2022-10-11 02:51:07] ...trap/engines/engine.py.326: process: 0.55774 | load: 0.00020 [I 2022-10-11 02:51:14] ...trap/engines/engine.py.321: eval: epoch 94 | batch 30/510 [I 2022-10-11 02:51:14] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:50 [I 2022-10-11 02:51:14] ...trap/engines/engine.py.326: process: 0.58429 | load: 0.00018 [I 2022-10-11 02:51:20] ...trap/engines/engine.py.321: eval: epoch 94 | batch 40/510 [I 2022-10-11 02:51:20] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:04:17 [I 2022-10-11 02:51:20] ...trap/engines/engine.py.326: process: 0.46123 | load: 0.00021 [I 2022-10-11 02:51:25] ...trap/engines/engine.py.321: eval: epoch 94 | batch 50/510 [I 2022-10-11 02:51:25] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:43 [I 2022-10-11 02:51:25] ...trap/engines/engine.py.326: process: 0.48677 | load: 0.00022 [I 2022-10-11 02:51:31] ...trap/engines/engine.py.321: eval: epoch 94 | batch 60/510 [I 2022-10-11 02:51:31] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:04:24 [I 2022-10-11 02:51:31] ...trap/engines/engine.py.326: process: 0.54974 | load: 0.00019 [I 2022-10-11 02:51:37] ...trap/engines/engine.py.321: eval: epoch 94 | batch 70/510 [I 2022-10-11 02:51:37] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:04:22 [I 2022-10-11 02:51:37] ...trap/engines/engine.py.326: process: 0.65696 | load: 0.00019 [I 2022-10-11 02:51:43] ...trap/engines/engine.py.321: eval: epoch 94 | batch 80/510 [I 2022-10-11 02:51:43] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:04:02 [I 2022-10-11 02:51:43] ...trap/engines/engine.py.326: process: 0.51340 | load: 0.00020 [I 2022-10-11 02:51:47] ...trap/engines/engine.py.321: eval: epoch 94 | batch 90/510 [I 2022-10-11 02:51:47] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:06 [I 2022-10-11 02:51:47] ...trap/engines/engine.py.326: process: 0.41174 | load: 0.00018 [I 2022-10-11 02:51:52] ...trap/engines/engine.py.321: eval: epoch 94 | batch 100/510 [I 2022-10-11 02:51:52] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:57 [I 2022-10-11 02:51:52] ...trap/engines/engine.py.326: process: 0.30755 | load: 0.00018 [I 2022-10-11 02:51:55] ...trap/engines/engine.py.321: eval: epoch 94 | batch 110/510 [I 2022-10-11 02:51:55] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:12 [I 2022-10-11 02:51:55] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00019 [I 2022-10-11 02:51:58] ...trap/engines/engine.py.321: eval: epoch 94 | batch 120/510 [I 2022-10-11 02:51:58] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:04 [I 2022-10-11 02:51:58] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00019 [I 2022-10-11 02:52:01] ...trap/engines/engine.py.321: eval: epoch 94 | batch 130/510 [I 2022-10-11 02:52:01] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:00 [I 2022-10-11 02:52:01] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00021 [I 2022-10-11 02:52:05] ...trap/engines/engine.py.321: eval: epoch 94 | batch 140/510 [I 2022-10-11 02:52:05] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:57 [I 2022-10-11 02:52:05] ...trap/engines/engine.py.326: process: 0.32445 | load: 0.00019 [I 2022-10-11 02:52:08] ...trap/engines/engine.py.321: eval: epoch 94 | batch 150/510 [I 2022-10-11 02:52:08] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:55 [I 2022-10-11 02:52:08] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00020 [I 2022-10-11 02:52:11] ...trap/engines/engine.py.321: eval: epoch 94 | batch 160/510 [I 2022-10-11 02:52:11] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:51 [I 2022-10-11 02:52:11] ...trap/engines/engine.py.326: process: 0.31286 | load: 0.00019 [I 2022-10-11 02:52:14] ...trap/engines/engine.py.321: eval: epoch 94 | batch 170/510 [I 2022-10-11 02:52:14] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:49 [I 2022-10-11 02:52:14] ...trap/engines/engine.py.326: process: 0.32460 | load: 0.00020 [I 2022-10-11 02:52:17] ...trap/engines/engine.py.321: eval: epoch 94 | batch 180/510 [I 2022-10-11 02:52:17] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:45 [I 2022-10-11 02:52:17] ...trap/engines/engine.py.326: process: 0.31895 | load: 0.00019 [I 2022-10-11 02:52:21] ...trap/engines/engine.py.321: eval: epoch 94 | batch 190/510 [I 2022-10-11 02:52:21] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:42 [I 2022-10-11 02:52:21] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00020 [I 2022-10-11 02:52:24] ...trap/engines/engine.py.321: eval: epoch 94 | batch 200/510 [I 2022-10-11 02:52:24] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:39 [I 2022-10-11 02:52:24] ...trap/engines/engine.py.326: process: 0.31793 | load: 0.00021 [I 2022-10-11 02:52:27] ...trap/engines/engine.py.321: eval: epoch 94 | batch 210/510 [I 2022-10-11 02:52:27] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:36 [I 2022-10-11 02:52:27] ...trap/engines/engine.py.326: process: 0.31487 | load: 0.00020 [I 2022-10-11 02:52:30] ...trap/engines/engine.py.321: eval: epoch 94 | batch 220/510 [I 2022-10-11 02:52:30] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:32 [I 2022-10-11 02:52:30] ...trap/engines/engine.py.326: process: 0.32348 | load: 0.00021 [I 2022-10-11 02:52:33] ...trap/engines/engine.py.321: eval: epoch 94 | batch 230/510 [I 2022-10-11 02:52:33] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:29 [I 2022-10-11 02:52:33] ...trap/engines/engine.py.326: process: 0.32135 | load: 0.00019 [I 2022-10-11 02:52:37] ...trap/engines/engine.py.321: eval: epoch 94 | batch 240/510 [I 2022-10-11 02:52:37] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:26 [I 2022-10-11 02:52:37] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00020 [I 2022-10-11 02:52:40] ...trap/engines/engine.py.321: eval: epoch 94 | batch 250/510 [I 2022-10-11 02:52:40] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:23 [I 2022-10-11 02:52:40] ...trap/engines/engine.py.326: process: 0.32532 | load: 0.00020 [I 2022-10-11 02:52:43] ...trap/engines/engine.py.321: eval: epoch 94 | batch 260/510 [I 2022-10-11 02:52:43] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:20 [I 2022-10-11 02:52:43] ...trap/engines/engine.py.326: process: 0.32276 | load: 0.00019 [I 2022-10-11 02:52:46] ...trap/engines/engine.py.321: eval: epoch 94 | batch 270/510 [I 2022-10-11 02:52:46] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:17 [I 2022-10-11 02:52:46] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00020 [I 2022-10-11 02:52:49] ...trap/engines/engine.py.321: eval: epoch 94 | batch 280/510 [I 2022-10-11 02:52:49] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:13 [I 2022-10-11 02:52:49] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00020 [I 2022-10-11 02:52:53] ...trap/engines/engine.py.321: eval: epoch 94 | batch 290/510 [I 2022-10-11 02:52:53] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:10 [I 2022-10-11 02:52:53] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00020 [I 2022-10-11 02:52:56] ...trap/engines/engine.py.321: eval: epoch 94 | batch 300/510 [I 2022-10-11 02:52:56] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:07 [I 2022-10-11 02:52:56] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00020 [I 2022-10-11 02:52:59] ...trap/engines/engine.py.321: eval: epoch 94 | batch 310/510 [I 2022-10-11 02:52:59] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:03 [I 2022-10-11 02:52:59] ...trap/engines/engine.py.326: process: 0.32271 | load: 0.00021 [I 2022-10-11 02:53:02] ...trap/engines/engine.py.321: eval: epoch 94 | batch 320/510 [I 2022-10-11 02:53:02] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:00 [I 2022-10-11 02:53:02] ...trap/engines/engine.py.326: process: 0.31202 | load: 0.00020 [I 2022-10-11 02:53:05] ...trap/engines/engine.py.321: eval: epoch 94 | batch 330/510 [I 2022-10-11 02:53:06] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:57 [I 2022-10-11 02:53:06] ...trap/engines/engine.py.326: process: 0.31730 | load: 0.00020 [I 2022-10-11 02:53:09] ...trap/engines/engine.py.321: eval: epoch 94 | batch 340/510 [I 2022-10-11 02:53:09] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:54 [I 2022-10-11 02:53:09] ...trap/engines/engine.py.326: process: 0.32303 | load: 0.00019 [I 2022-10-11 02:53:12] ...trap/engines/engine.py.321: eval: epoch 94 | batch 350/510 [I 2022-10-11 02:53:12] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:51 [I 2022-10-11 02:53:12] ...trap/engines/engine.py.326: process: 0.32318 | load: 0.00020 [I 2022-10-11 02:53:15] ...trap/engines/engine.py.321: eval: epoch 94 | batch 360/510 [I 2022-10-11 02:53:15] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:48 [I 2022-10-11 02:53:15] ...trap/engines/engine.py.326: process: 0.31945 | load: 0.00022 [I 2022-10-11 02:53:18] ...trap/engines/engine.py.321: eval: epoch 94 | batch 370/510 [I 2022-10-11 02:53:18] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:45 [I 2022-10-11 02:53:18] ...trap/engines/engine.py.326: process: 0.31742 | load: 0.00023 [I 2022-10-11 02:53:22] ...trap/engines/engine.py.321: eval: epoch 94 | batch 380/510 [I 2022-10-11 02:53:22] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:42 [I 2022-10-11 02:53:22] ...trap/engines/engine.py.326: process: 0.33188 | load: 0.00021 [I 2022-10-11 02:53:25] ...trap/engines/engine.py.321: eval: epoch 94 | batch 390/510 [I 2022-10-11 02:53:25] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:39 [I 2022-10-11 02:53:25] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00021 [I 2022-10-11 02:53:28] ...trap/engines/engine.py.321: eval: epoch 94 | batch 400/510 [I 2022-10-11 02:53:28] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:35 [I 2022-10-11 02:53:28] ...trap/engines/engine.py.326: process: 0.32339 | load: 0.00021 [I 2022-10-11 02:53:31] ...trap/engines/engine.py.321: eval: epoch 94 | batch 410/510 [I 2022-10-11 02:53:31] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:32 [I 2022-10-11 02:53:31] ...trap/engines/engine.py.326: process: 0.32678 | load: 0.00021 [I 2022-10-11 02:53:35] ...trap/engines/engine.py.321: eval: epoch 94 | batch 420/510 [I 2022-10-11 02:53:35] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:29 [I 2022-10-11 02:53:35] ...trap/engines/engine.py.326: process: 0.32409 | load: 0.00023 [I 2022-10-11 02:53:38] ...trap/engines/engine.py.321: eval: epoch 94 | batch 430/510 [I 2022-10-11 02:53:38] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:25 [I 2022-10-11 02:53:38] ...trap/engines/engine.py.326: process: 0.32432 | load: 0.00021 [I 2022-10-11 02:53:41] ...trap/engines/engine.py.321: eval: epoch 94 | batch 440/510 [I 2022-10-11 02:53:41] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:22 [I 2022-10-11 02:53:41] ...trap/engines/engine.py.326: process: 0.32509 | load: 0.00021 [I 2022-10-11 02:53:44] ...trap/engines/engine.py.321: eval: epoch 94 | batch 450/510 [I 2022-10-11 02:53:44] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:19 [I 2022-10-11 02:53:44] ...trap/engines/engine.py.326: process: 0.31613 | load: 0.00021 [I 2022-10-11 02:53:48] ...trap/engines/engine.py.321: eval: epoch 94 | batch 460/510 [I 2022-10-11 02:53:48] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:16 [I 2022-10-11 02:53:48] ...trap/engines/engine.py.326: process: 0.33276 | load: 0.00019 [I 2022-10-11 02:53:51] ...trap/engines/engine.py.321: eval: epoch 94 | batch 470/510 [I 2022-10-11 02:53:51] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:12 [I 2022-10-11 02:53:51] ...trap/engines/engine.py.326: process: 0.32128 | load: 0.00021 [I 2022-10-11 02:53:54] ...trap/engines/engine.py.321: eval: epoch 94 | batch 480/510 [I 2022-10-11 02:53:54] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:09 [I 2022-10-11 02:53:54] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00021 [I 2022-10-11 02:53:58] ...trap/engines/engine.py.321: eval: epoch 94 | batch 490/510 [I 2022-10-11 02:53:58] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:06 [I 2022-10-11 02:53:58] ...trap/engines/engine.py.326: process: 0.31991 | load: 0.00021 [I 2022-10-11 02:54:01] ...trap/engines/engine.py.321: eval: epoch 94 | batch 500/510 [I 2022-10-11 02:54:01] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:03 [I 2022-10-11 02:54:01] ...trap/engines/engine.py.326: process: 0.32206 | load: 0.00021 [I 2022-10-11 02:54:04] ...trap/engines/engine.py.321: eval: epoch 94 | batch 510/510 [I 2022-10-11 02:54:04] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:00 [I 2022-10-11 02:54:04] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00016 [S 2022-10-11 02:54:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 94 [I 2022-10-11 02:54:05] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.744800 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907200 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936100 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.731300 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905800 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934700 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012742 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006063 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004700 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012642 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007560 [S 2022-10-11 02:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006278 [I 2022-10-11 02:54:52] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 02:54:52] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 02:54:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 02:54:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 02:54:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 02:54:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 02:55:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 02:55:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 02:55:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 95 [I 2022-10-11 02:55:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 02:55:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 0/2304 [I 2022-10-11 02:55:07] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:10:41 [I 2022-10-11 02:55:07] ...trap/engines/engine.py.229: process: 3.40350 | load: 2.49021 [I 2022-10-11 02:55:07] ...trap/engines/engine.py.230: loss: 0.09224 [I 2022-10-11 02:55:07] ...optimizers/trijoint.py.151: total_norm: 13.289238 [I 2022-10-11 02:55:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 10/2304 [I 2022-10-11 02:55:16] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:24 [I 2022-10-11 02:55:16] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00660 [I 2022-10-11 02:55:16] ...trap/engines/engine.py.230: loss: 0.09287 [I 2022-10-11 02:55:16] ...optimizers/trijoint.py.151: total_norm: 10.063861 [I 2022-10-11 02:55:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 20/2304 [I 2022-10-11 02:55:24] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:34:05 [I 2022-10-11 02:55:24] ...trap/engines/engine.py.229: process: 0.87402 | load: 0.00666 [I 2022-10-11 02:55:24] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-11 02:55:24] ...optimizers/trijoint.py.151: total_norm: 22.840689 [I 2022-10-11 02:55:33] ...trap/engines/engine.py.224: train: epoch 95 | batch 30/2304 [I 2022-10-11 02:55:33] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:00 [I 2022-10-11 02:55:33] ...trap/engines/engine.py.229: process: 0.87840 | load: 0.00527 [I 2022-10-11 02:55:33] ...trap/engines/engine.py.230: loss: 0.07950 [I 2022-10-11 02:55:33] ...optimizers/trijoint.py.151: total_norm: 10.599990 [I 2022-10-11 02:55:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 40/2304 [I 2022-10-11 02:55:42] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:37 [I 2022-10-11 02:55:42] ...trap/engines/engine.py.229: process: 0.85377 | load: 0.00422 [I 2022-10-11 02:55:42] ...trap/engines/engine.py.230: loss: 0.14214 [I 2022-10-11 02:55:42] ...optimizers/trijoint.py.151: total_norm: 14.616142 [I 2022-10-11 02:55:50] ...trap/engines/engine.py.224: train: epoch 95 | batch 50/2304 [I 2022-10-11 02:55:50] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:26 [I 2022-10-11 02:55:50] ...trap/engines/engine.py.229: process: 0.85984 | load: 0.00409 [I 2022-10-11 02:55:50] ...trap/engines/engine.py.230: loss: 0.13648 [I 2022-10-11 02:55:50] ...optimizers/trijoint.py.151: total_norm: 13.240534 [I 2022-10-11 02:55:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 60/2304 [I 2022-10-11 02:55:59] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:28 [I 2022-10-11 02:55:59] ...trap/engines/engine.py.229: process: 0.86912 | load: 0.00398 [I 2022-10-11 02:55:59] ...trap/engines/engine.py.230: loss: 0.09830 [I 2022-10-11 02:55:59] ...optimizers/trijoint.py.151: total_norm: 9.129929 [I 2022-10-11 02:56:08] ...trap/engines/engine.py.224: train: epoch 95 | batch 70/2304 [I 2022-10-11 02:56:08] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:24 [I 2022-10-11 02:56:08] ...trap/engines/engine.py.229: process: 0.87210 | load: 0.00423 [I 2022-10-11 02:56:08] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-11 02:56:08] ...optimizers/trijoint.py.151: total_norm: 15.551807 [I 2022-10-11 02:56:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 80/2304 [I 2022-10-11 02:56:16] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:12 [I 2022-10-11 02:56:16] ...trap/engines/engine.py.229: process: 0.87229 | load: 0.00414 [I 2022-10-11 02:56:16] ...trap/engines/engine.py.230: loss: 0.11922 [I 2022-10-11 02:56:16] ...optimizers/trijoint.py.151: total_norm: 14.896091 [I 2022-10-11 02:56:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 90/2304 [I 2022-10-11 02:56:25] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:32:03 [I 2022-10-11 02:56:25] ...trap/engines/engine.py.229: process: 0.86922 | load: 0.00365 [I 2022-10-11 02:56:25] ...trap/engines/engine.py.230: loss: 0.13264 [I 2022-10-11 02:56:25] ...optimizers/trijoint.py.151: total_norm: 25.653011 [I 2022-10-11 02:56:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 100/2304 [I 2022-10-11 02:56:34] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:32:04 [I 2022-10-11 02:56:34] ...trap/engines/engine.py.229: process: 0.88438 | load: 0.00331 [I 2022-10-11 02:56:34] ...trap/engines/engine.py.230: loss: 0.10668 [I 2022-10-11 02:56:34] ...optimizers/trijoint.py.151: total_norm: 9.836535 [I 2022-10-11 02:56:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 110/2304 [I 2022-10-11 02:56:43] ...trap/engines/engine.py.228: elapsed: 0:01:39 | left: 0:31:48 [I 2022-10-11 02:56:43] ...trap/engines/engine.py.229: process: 0.86597 | load: 0.00357 [I 2022-10-11 02:56:43] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-11 02:56:43] ...optimizers/trijoint.py.151: total_norm: 14.128368 [I 2022-10-11 02:56:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 120/2304 [I 2022-10-11 02:56:51] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:34 [I 2022-10-11 02:56:51] ...trap/engines/engine.py.229: process: 0.85853 | load: 0.00370 [I 2022-10-11 02:56:51] ...trap/engines/engine.py.230: loss: 0.24266 [I 2022-10-11 02:56:51] ...optimizers/trijoint.py.151: total_norm: 25.900232 [I 2022-10-11 02:57:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 130/2304 [I 2022-10-11 02:57:00] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:25 [I 2022-10-11 02:57:00] ...trap/engines/engine.py.229: process: 0.88304 | load: 0.00354 [I 2022-10-11 02:57:00] ...trap/engines/engine.py.230: loss: 0.12515 [I 2022-10-11 02:57:00] ...optimizers/trijoint.py.151: total_norm: 14.835322 [I 2022-10-11 02:57:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 140/2304 [I 2022-10-11 02:57:09] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:31:05 [I 2022-10-11 02:57:09] ...trap/engines/engine.py.229: process: 0.87187 | load: 0.00328 [I 2022-10-11 02:57:09] ...trap/engines/engine.py.230: loss: 0.11488 [I 2022-10-11 02:57:09] ...optimizers/trijoint.py.151: total_norm: 13.380548 [I 2022-10-11 02:57:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 150/2304 [I 2022-10-11 02:57:17] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:30:47 [I 2022-10-11 02:57:17] ...trap/engines/engine.py.229: process: 0.86481 | load: 0.00342 [I 2022-10-11 02:57:17] ...trap/engines/engine.py.230: loss: 0.10483 [I 2022-10-11 02:57:17] ...optimizers/trijoint.py.151: total_norm: 24.383379 [I 2022-10-11 02:57:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 160/2304 [I 2022-10-11 02:57:26] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:30:37 [I 2022-10-11 02:57:26] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00310 [I 2022-10-11 02:57:26] ...trap/engines/engine.py.230: loss: 0.09471 [I 2022-10-11 02:57:26] ...optimizers/trijoint.py.151: total_norm: 10.280983 [I 2022-10-11 02:57:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 170/2304 [I 2022-10-11 02:57:34] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:30:19 [I 2022-10-11 02:57:34] ...trap/engines/engine.py.229: process: 0.85939 | load: 0.00302 [I 2022-10-11 02:57:34] ...trap/engines/engine.py.230: loss: 0.08548 [I 2022-10-11 02:57:34] ...optimizers/trijoint.py.151: total_norm: 19.054871 [I 2022-10-11 02:57:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 180/2304 [I 2022-10-11 02:57:43] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:14 [I 2022-10-11 02:57:43] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00298 [I 2022-10-11 02:57:43] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-11 02:57:43] ...optimizers/trijoint.py.151: total_norm: 18.454052 [I 2022-10-11 02:57:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 190/2304 [I 2022-10-11 02:57:52] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:10 [I 2022-10-11 02:57:52] ...trap/engines/engine.py.229: process: 0.86766 | load: 0.00305 [I 2022-10-11 02:57:52] ...trap/engines/engine.py.230: loss: 0.12149 [I 2022-10-11 02:57:52] ...optimizers/trijoint.py.151: total_norm: 16.098516 [I 2022-10-11 02:58:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 200/2304 [I 2022-10-11 02:58:00] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:29:56 [I 2022-10-11 02:58:00] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00305 [I 2022-10-11 02:58:00] ...trap/engines/engine.py.230: loss: 0.07952 [I 2022-10-11 02:58:00] ...optimizers/trijoint.py.151: total_norm: 15.085217 [I 2022-10-11 02:58:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 210/2304 [I 2022-10-11 02:58:09] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:29:51 [I 2022-10-11 02:58:09] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00296 [I 2022-10-11 02:58:09] ...trap/engines/engine.py.230: loss: 0.13669 [I 2022-10-11 02:58:09] ...optimizers/trijoint.py.151: total_norm: 27.329855 [I 2022-10-11 02:58:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 220/2304 [I 2022-10-11 02:58:17] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:44 [I 2022-10-11 02:58:17] ...trap/engines/engine.py.229: process: 0.83828 | load: 0.00297 [I 2022-10-11 02:58:17] ...trap/engines/engine.py.230: loss: 0.11834 [I 2022-10-11 02:58:17] ...optimizers/trijoint.py.151: total_norm: 39.839214 [I 2022-10-11 02:58:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 230/2304 [I 2022-10-11 02:58:26] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:43 [I 2022-10-11 02:58:26] ...trap/engines/engine.py.229: process: 0.85569 | load: 0.00310 [I 2022-10-11 02:58:26] ...trap/engines/engine.py.230: loss: 0.10296 [I 2022-10-11 02:58:26] ...optimizers/trijoint.py.151: total_norm: 12.023735 [I 2022-10-11 02:58:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 240/2304 [I 2022-10-11 02:58:35] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:24 [I 2022-10-11 02:58:35] ...trap/engines/engine.py.229: process: 0.85142 | load: 0.00298 [I 2022-10-11 02:58:35] ...trap/engines/engine.py.230: loss: 0.14209 [I 2022-10-11 02:58:35] ...optimizers/trijoint.py.151: total_norm: 11.792340 [I 2022-10-11 02:58:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 250/2304 [I 2022-10-11 02:58:43] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:18 [I 2022-10-11 02:58:43] ...trap/engines/engine.py.229: process: 0.85594 | load: 0.00299 [I 2022-10-11 02:58:43] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-11 02:58:43] ...optimizers/trijoint.py.151: total_norm: 19.786188 [I 2022-10-11 02:58:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 260/2304 [I 2022-10-11 02:58:52] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:00 [I 2022-10-11 02:58:52] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00311 [I 2022-10-11 02:58:52] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-11 02:58:52] ...optimizers/trijoint.py.151: total_norm: 23.924921 [I 2022-10-11 02:59:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 270/2304 [I 2022-10-11 02:59:00] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:05 [I 2022-10-11 02:59:00] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00305 [I 2022-10-11 02:59:00] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-11 02:59:00] ...optimizers/trijoint.py.151: total_norm: 9.994713 [I 2022-10-11 02:59:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 280/2304 [I 2022-10-11 02:59:09] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:28:58 [I 2022-10-11 02:59:09] ...trap/engines/engine.py.229: process: 0.86008 | load: 0.00294 [I 2022-10-11 02:59:09] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-11 02:59:09] ...optimizers/trijoint.py.151: total_norm: 12.586920 [I 2022-10-11 02:59:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 290/2304 [I 2022-10-11 02:59:17] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:44 [I 2022-10-11 02:59:17] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00307 [I 2022-10-11 02:59:17] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-11 02:59:17] ...optimizers/trijoint.py.151: total_norm: 17.939556 [I 2022-10-11 02:59:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 300/2304 [I 2022-10-11 02:59:26] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:18 [I 2022-10-11 02:59:26] ...trap/engines/engine.py.229: process: 0.83999 | load: 0.00309 [I 2022-10-11 02:59:26] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-11 02:59:26] ...optimizers/trijoint.py.151: total_norm: 14.059030 [I 2022-10-11 02:59:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 310/2304 [I 2022-10-11 02:59:35] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:36 [I 2022-10-11 02:59:35] ...trap/engines/engine.py.229: process: 0.87255 | load: 0.00308 [I 2022-10-11 02:59:35] ...trap/engines/engine.py.230: loss: 0.09873 [I 2022-10-11 02:59:35] ...optimizers/trijoint.py.151: total_norm: 9.245909 [I 2022-10-11 02:59:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 320/2304 [I 2022-10-11 02:59:43] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:31 [I 2022-10-11 02:59:43] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00324 [I 2022-10-11 02:59:43] ...trap/engines/engine.py.230: loss: 0.10060 [I 2022-10-11 02:59:43] ...optimizers/trijoint.py.151: total_norm: 35.291119 [I 2022-10-11 02:59:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 330/2304 [I 2022-10-11 02:59:52] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:25 [I 2022-10-11 02:59:52] ...trap/engines/engine.py.229: process: 0.87530 | load: 0.00308 [I 2022-10-11 02:59:52] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-11 02:59:52] ...optimizers/trijoint.py.151: total_norm: 10.829476 [I 2022-10-11 03:00:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 340/2304 [I 2022-10-11 03:00:01] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:28:27 [I 2022-10-11 03:00:01] ...trap/engines/engine.py.229: process: 0.86018 | load: 0.00348 [I 2022-10-11 03:00:01] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-11 03:00:01] ...optimizers/trijoint.py.151: total_norm: 10.888276 [I 2022-10-11 03:00:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 350/2304 [I 2022-10-11 03:00:09] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:13 [I 2022-10-11 03:00:09] ...trap/engines/engine.py.229: process: 0.86286 | load: 0.00307 [I 2022-10-11 03:00:09] ...trap/engines/engine.py.230: loss: 0.15907 [I 2022-10-11 03:00:09] ...optimizers/trijoint.py.151: total_norm: 19.539518 [I 2022-10-11 03:00:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 360/2304 [I 2022-10-11 03:00:18] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:47 [I 2022-10-11 03:00:18] ...trap/engines/engine.py.229: process: 0.84414 | load: 0.00322 [I 2022-10-11 03:00:18] ...trap/engines/engine.py.230: loss: 0.14904 [I 2022-10-11 03:00:18] ...optimizers/trijoint.py.151: total_norm: 16.977230 [I 2022-10-11 03:00:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 370/2304 [I 2022-10-11 03:00:27] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:53 [I 2022-10-11 03:00:27] ...trap/engines/engine.py.229: process: 0.86088 | load: 0.00308 [I 2022-10-11 03:00:27] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-11 03:00:27] ...optimizers/trijoint.py.151: total_norm: 20.409132 [I 2022-10-11 03:00:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 380/2304 [I 2022-10-11 03:00:35] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:33 [I 2022-10-11 03:00:35] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00306 [I 2022-10-11 03:00:35] ...trap/engines/engine.py.230: loss: 0.09331 [I 2022-10-11 03:00:35] ...optimizers/trijoint.py.151: total_norm: 12.074144 [I 2022-10-11 03:00:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 390/2304 [I 2022-10-11 03:00:44] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:33 [I 2022-10-11 03:00:44] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00307 [I 2022-10-11 03:00:44] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-11 03:00:44] ...optimizers/trijoint.py.151: total_norm: 11.529691 [I 2022-10-11 03:00:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 400/2304 [I 2022-10-11 03:00:53] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:35 [I 2022-10-11 03:00:53] ...trap/engines/engine.py.229: process: 0.89385 | load: 0.00313 [I 2022-10-11 03:00:53] ...trap/engines/engine.py.230: loss: 0.08815 [I 2022-10-11 03:00:53] ...optimizers/trijoint.py.151: total_norm: 10.686652 [I 2022-10-11 03:01:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 410/2304 [I 2022-10-11 03:01:01] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:18 [I 2022-10-11 03:01:01] ...trap/engines/engine.py.229: process: 0.86382 | load: 0.00322 [I 2022-10-11 03:01:01] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-11 03:01:01] ...optimizers/trijoint.py.151: total_norm: 17.497557 [I 2022-10-11 03:01:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 420/2304 [I 2022-10-11 03:01:10] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:59 [I 2022-10-11 03:01:10] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00312 [I 2022-10-11 03:01:10] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-11 03:01:10] ...optimizers/trijoint.py.151: total_norm: 11.935269 [I 2022-10-11 03:01:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 430/2304 [I 2022-10-11 03:01:19] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:27:02 [I 2022-10-11 03:01:19] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00321 [I 2022-10-11 03:01:19] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-11 03:01:19] ...optimizers/trijoint.py.151: total_norm: 14.514004 [I 2022-10-11 03:01:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 440/2304 [I 2022-10-11 03:01:27] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:27:02 [I 2022-10-11 03:01:27] ...trap/engines/engine.py.229: process: 0.87025 | load: 0.00336 [I 2022-10-11 03:01:27] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-11 03:01:27] ...optimizers/trijoint.py.151: total_norm: 14.462387 [I 2022-10-11 03:01:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 450/2304 [I 2022-10-11 03:01:36] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:48 [I 2022-10-11 03:01:36] ...trap/engines/engine.py.229: process: 0.86716 | load: 0.00310 [I 2022-10-11 03:01:36] ...trap/engines/engine.py.230: loss: 0.22174 [I 2022-10-11 03:01:36] ...optimizers/trijoint.py.151: total_norm: 21.822901 [I 2022-10-11 03:01:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 460/2304 [I 2022-10-11 03:01:45] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:26:33 [I 2022-10-11 03:01:45] ...trap/engines/engine.py.229: process: 0.86263 | load: 0.00307 [I 2022-10-11 03:01:45] ...trap/engines/engine.py.230: loss: 0.13196 [I 2022-10-11 03:01:45] ...optimizers/trijoint.py.151: total_norm: 20.952089 [I 2022-10-11 03:01:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 470/2304 [I 2022-10-11 03:01:53] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:28 [I 2022-10-11 03:01:53] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00311 [I 2022-10-11 03:01:53] ...trap/engines/engine.py.230: loss: 0.12519 [I 2022-10-11 03:01:53] ...optimizers/trijoint.py.151: total_norm: 17.869041 [I 2022-10-11 03:02:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 480/2304 [I 2022-10-11 03:02:02] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:26:09 [I 2022-10-11 03:02:02] ...trap/engines/engine.py.229: process: 0.87429 | load: 0.00309 [I 2022-10-11 03:02:02] ...trap/engines/engine.py.230: loss: 0.14308 [I 2022-10-11 03:02:02] ...optimizers/trijoint.py.151: total_norm: 12.362926 [I 2022-10-11 03:02:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 490/2304 [I 2022-10-11 03:02:11] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:26:09 [I 2022-10-11 03:02:11] ...trap/engines/engine.py.229: process: 0.86746 | load: 0.00326 [I 2022-10-11 03:02:11] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-11 03:02:11] ...optimizers/trijoint.py.151: total_norm: 40.022930 [I 2022-10-11 03:02:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 500/2304 [I 2022-10-11 03:02:19] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:57 [I 2022-10-11 03:02:19] ...trap/engines/engine.py.229: process: 0.86635 | load: 0.00316 [I 2022-10-11 03:02:19] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-11 03:02:19] ...optimizers/trijoint.py.151: total_norm: 18.800816 [I 2022-10-11 03:02:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 510/2304 [I 2022-10-11 03:02:28] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:49 [I 2022-10-11 03:02:28] ...trap/engines/engine.py.229: process: 0.87553 | load: 0.00313 [I 2022-10-11 03:02:28] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-11 03:02:28] ...optimizers/trijoint.py.151: total_norm: 12.017773 [I 2022-10-11 03:02:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 520/2304 [I 2022-10-11 03:02:37] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:49 [I 2022-10-11 03:02:37] ...trap/engines/engine.py.229: process: 0.86848 | load: 0.00352 [I 2022-10-11 03:02:37] ...trap/engines/engine.py.230: loss: 0.19468 [I 2022-10-11 03:02:37] ...optimizers/trijoint.py.151: total_norm: 33.069275 [I 2022-10-11 03:02:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 530/2304 [I 2022-10-11 03:02:45] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:25:36 [I 2022-10-11 03:02:45] ...trap/engines/engine.py.229: process: 0.85823 | load: 0.00314 [I 2022-10-11 03:02:45] ...trap/engines/engine.py.230: loss: 0.12180 [I 2022-10-11 03:02:45] ...optimizers/trijoint.py.151: total_norm: 23.358761 [I 2022-10-11 03:02:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 540/2304 [I 2022-10-11 03:02:54] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:25:24 [I 2022-10-11 03:02:54] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00315 [I 2022-10-11 03:02:54] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-11 03:02:54] ...optimizers/trijoint.py.151: total_norm: 12.037846 [I 2022-10-11 03:03:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 550/2304 [I 2022-10-11 03:03:03] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:25:16 [I 2022-10-11 03:03:03] ...trap/engines/engine.py.229: process: 0.86573 | load: 0.00311 [I 2022-10-11 03:03:03] ...trap/engines/engine.py.230: loss: 0.17200 [I 2022-10-11 03:03:03] ...optimizers/trijoint.py.151: total_norm: 14.039211 [I 2022-10-11 03:03:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 560/2304 [I 2022-10-11 03:03:11] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:25:16 [I 2022-10-11 03:03:11] ...trap/engines/engine.py.229: process: 0.88420 | load: 0.00312 [I 2022-10-11 03:03:11] ...trap/engines/engine.py.230: loss: 0.15960 [I 2022-10-11 03:03:11] ...optimizers/trijoint.py.151: total_norm: 33.270218 [I 2022-10-11 03:03:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 570/2304 [I 2022-10-11 03:03:20] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:25:06 [I 2022-10-11 03:03:20] ...trap/engines/engine.py.229: process: 0.87427 | load: 0.00306 [I 2022-10-11 03:03:20] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-11 03:03:20] ...optimizers/trijoint.py.151: total_norm: 11.439971 [I 2022-10-11 03:03:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 580/2304 [I 2022-10-11 03:03:29] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:35 [I 2022-10-11 03:03:29] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00297 [I 2022-10-11 03:03:29] ...trap/engines/engine.py.230: loss: 0.12896 [I 2022-10-11 03:03:29] ...optimizers/trijoint.py.151: total_norm: 33.085327 [I 2022-10-11 03:03:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 590/2304 [I 2022-10-11 03:03:37] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:20 [I 2022-10-11 03:03:37] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00313 [I 2022-10-11 03:03:37] ...trap/engines/engine.py.230: loss: 0.11127 [I 2022-10-11 03:03:37] ...optimizers/trijoint.py.151: total_norm: 22.978392 [I 2022-10-11 03:03:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 600/2304 [I 2022-10-11 03:03:46] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:21 [I 2022-10-11 03:03:46] ...trap/engines/engine.py.229: process: 0.86998 | load: 0.00296 [I 2022-10-11 03:03:46] ...trap/engines/engine.py.230: loss: 0.13676 [I 2022-10-11 03:03:46] ...optimizers/trijoint.py.151: total_norm: 17.201221 [I 2022-10-11 03:03:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 610/2304 [I 2022-10-11 03:03:54] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:15 [I 2022-10-11 03:03:54] ...trap/engines/engine.py.229: process: 0.86694 | load: 0.00310 [I 2022-10-11 03:03:54] ...trap/engines/engine.py.230: loss: 0.17374 [I 2022-10-11 03:03:54] ...optimizers/trijoint.py.151: total_norm: 15.181548 [I 2022-10-11 03:04:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 620/2304 [I 2022-10-11 03:04:03] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:24:04 [I 2022-10-11 03:04:03] ...trap/engines/engine.py.229: process: 0.86601 | load: 0.00328 [I 2022-10-11 03:04:03] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-11 03:04:03] ...optimizers/trijoint.py.151: total_norm: 8.727078 [I 2022-10-11 03:04:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 630/2304 [I 2022-10-11 03:04:12] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:48 [I 2022-10-11 03:04:12] ...trap/engines/engine.py.229: process: 0.84287 | load: 0.00312 [I 2022-10-11 03:04:12] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-11 03:04:12] ...optimizers/trijoint.py.151: total_norm: 11.037056 [I 2022-10-11 03:04:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 640/2304 [I 2022-10-11 03:04:20] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:44 [I 2022-10-11 03:04:20] ...trap/engines/engine.py.229: process: 0.85565 | load: 0.00307 [I 2022-10-11 03:04:20] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-11 03:04:20] ...optimizers/trijoint.py.151: total_norm: 9.665817 [I 2022-10-11 03:04:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 650/2304 [I 2022-10-11 03:04:29] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:35 [I 2022-10-11 03:04:29] ...trap/engines/engine.py.229: process: 0.86072 | load: 0.00299 [I 2022-10-11 03:04:29] ...trap/engines/engine.py.230: loss: 0.14106 [I 2022-10-11 03:04:29] ...optimizers/trijoint.py.151: total_norm: 26.256777 [I 2022-10-11 03:04:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 660/2304 [I 2022-10-11 03:04:37] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:34 [I 2022-10-11 03:04:37] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00298 [I 2022-10-11 03:04:37] ...trap/engines/engine.py.230: loss: 0.12646 [I 2022-10-11 03:04:37] ...optimizers/trijoint.py.151: total_norm: 9.852642 [I 2022-10-11 03:04:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 670/2304 [I 2022-10-11 03:04:46] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:21 [I 2022-10-11 03:04:46] ...trap/engines/engine.py.229: process: 0.86018 | load: 0.00297 [I 2022-10-11 03:04:46] ...trap/engines/engine.py.230: loss: 0.18959 [I 2022-10-11 03:04:46] ...optimizers/trijoint.py.151: total_norm: 17.706350 [I 2022-10-11 03:04:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 680/2304 [I 2022-10-11 03:04:55] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:08 [I 2022-10-11 03:04:55] ...trap/engines/engine.py.229: process: 0.87126 | load: 0.00295 [I 2022-10-11 03:04:55] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-11 03:04:55] ...optimizers/trijoint.py.151: total_norm: 11.424965 [I 2022-10-11 03:05:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 690/2304 [I 2022-10-11 03:05:03] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:23:09 [I 2022-10-11 03:05:03] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00299 [I 2022-10-11 03:05:03] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-11 03:05:03] ...optimizers/trijoint.py.151: total_norm: 10.218239 [I 2022-10-11 03:05:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 700/2304 [I 2022-10-11 03:05:12] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:54 [I 2022-10-11 03:05:12] ...trap/engines/engine.py.229: process: 0.86980 | load: 0.00295 [I 2022-10-11 03:05:12] ...trap/engines/engine.py.230: loss: 0.18239 [I 2022-10-11 03:05:12] ...optimizers/trijoint.py.151: total_norm: 31.115467 [I 2022-10-11 03:05:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 710/2304 [I 2022-10-11 03:05:20] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:31 [I 2022-10-11 03:05:20] ...trap/engines/engine.py.229: process: 0.83639 | load: 0.00294 [I 2022-10-11 03:05:20] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-11 03:05:20] ...optimizers/trijoint.py.151: total_norm: 17.406326 [I 2022-10-11 03:05:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 720/2304 [I 2022-10-11 03:05:29] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:33 [I 2022-10-11 03:05:29] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00300 [I 2022-10-11 03:05:29] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-11 03:05:29] ...optimizers/trijoint.py.151: total_norm: 21.296947 [I 2022-10-11 03:05:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 730/2304 [I 2022-10-11 03:05:37] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:18 [I 2022-10-11 03:05:37] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00293 [I 2022-10-11 03:05:37] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-11 03:05:38] ...optimizers/trijoint.py.151: total_norm: 21.980062 [I 2022-10-11 03:05:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 740/2304 [I 2022-10-11 03:05:46] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:17 [I 2022-10-11 03:05:46] ...trap/engines/engine.py.229: process: 0.86617 | load: 0.00304 [I 2022-10-11 03:05:46] ...trap/engines/engine.py.230: loss: 0.11168 [I 2022-10-11 03:05:46] ...optimizers/trijoint.py.151: total_norm: 9.243575 [I 2022-10-11 03:05:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 750/2304 [I 2022-10-11 03:05:55] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:25 [I 2022-10-11 03:05:55] ...trap/engines/engine.py.229: process: 0.87651 | load: 0.00312 [I 2022-10-11 03:05:55] ...trap/engines/engine.py.230: loss: 0.15164 [I 2022-10-11 03:05:55] ...optimizers/trijoint.py.151: total_norm: 22.276617 [I 2022-10-11 03:06:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 760/2304 [I 2022-10-11 03:06:03] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:26 [I 2022-10-11 03:06:03] ...trap/engines/engine.py.229: process: 0.86786 | load: 0.00316 [I 2022-10-11 03:06:03] ...trap/engines/engine.py.230: loss: 0.10494 [I 2022-10-11 03:06:03] ...optimizers/trijoint.py.151: total_norm: 18.699520 [I 2022-10-11 03:06:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 770/2304 [I 2022-10-11 03:06:12] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:22:12 [I 2022-10-11 03:06:12] ...trap/engines/engine.py.229: process: 0.86817 | load: 0.00323 [I 2022-10-11 03:06:12] ...trap/engines/engine.py.230: loss: 0.08565 [I 2022-10-11 03:06:12] ...optimizers/trijoint.py.151: total_norm: 33.040047 [I 2022-10-11 03:06:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 780/2304 [I 2022-10-11 03:06:21] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:22:04 [I 2022-10-11 03:06:21] ...trap/engines/engine.py.229: process: 0.86387 | load: 0.00314 [I 2022-10-11 03:06:21] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-11 03:06:21] ...optimizers/trijoint.py.151: total_norm: 18.057985 [I 2022-10-11 03:06:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 790/2304 [I 2022-10-11 03:06:30] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:45 [I 2022-10-11 03:06:30] ...trap/engines/engine.py.229: process: 0.86141 | load: 0.00305 [I 2022-10-11 03:06:30] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-11 03:06:30] ...optimizers/trijoint.py.151: total_norm: 15.543413 [I 2022-10-11 03:06:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 800/2304 [I 2022-10-11 03:06:38] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:38 [I 2022-10-11 03:06:38] ...trap/engines/engine.py.229: process: 0.86631 | load: 0.00316 [I 2022-10-11 03:06:38] ...trap/engines/engine.py.230: loss: 0.10392 [I 2022-10-11 03:06:38] ...optimizers/trijoint.py.151: total_norm: 16.971872 [I 2022-10-11 03:06:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 810/2304 [I 2022-10-11 03:06:47] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:31 [I 2022-10-11 03:06:47] ...trap/engines/engine.py.229: process: 0.86262 | load: 0.00308 [I 2022-10-11 03:06:47] ...trap/engines/engine.py.230: loss: 0.11337 [I 2022-10-11 03:06:47] ...optimizers/trijoint.py.151: total_norm: 15.449469 [I 2022-10-11 03:06:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 820/2304 [I 2022-10-11 03:06:56] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:30 [I 2022-10-11 03:06:56] ...trap/engines/engine.py.229: process: 0.87526 | load: 0.00307 [I 2022-10-11 03:06:56] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-11 03:06:56] ...optimizers/trijoint.py.151: total_norm: 16.297112 [I 2022-10-11 03:07:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 830/2304 [I 2022-10-11 03:07:04] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:15 [I 2022-10-11 03:07:04] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00588 [I 2022-10-11 03:07:04] ...trap/engines/engine.py.230: loss: 0.11283 [I 2022-10-11 03:07:04] ...optimizers/trijoint.py.151: total_norm: 18.004704 [I 2022-10-11 03:07:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 840/2304 [I 2022-10-11 03:07:13] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:21:07 [I 2022-10-11 03:07:13] ...trap/engines/engine.py.229: process: 0.86652 | load: 0.00314 [I 2022-10-11 03:07:13] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-11 03:07:13] ...optimizers/trijoint.py.151: total_norm: 15.742605 [I 2022-10-11 03:07:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 850/2304 [I 2022-10-11 03:07:22] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:21:02 [I 2022-10-11 03:07:22] ...trap/engines/engine.py.229: process: 0.86945 | load: 0.00325 [I 2022-10-11 03:07:22] ...trap/engines/engine.py.230: loss: 0.10414 [I 2022-10-11 03:07:22] ...optimizers/trijoint.py.151: total_norm: 8.926846 [I 2022-10-11 03:07:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 860/2304 [I 2022-10-11 03:07:30] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:49 [I 2022-10-11 03:07:30] ...trap/engines/engine.py.229: process: 0.86314 | load: 0.00315 [I 2022-10-11 03:07:30] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-11 03:07:30] ...optimizers/trijoint.py.151: total_norm: 15.129049 [I 2022-10-11 03:07:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 870/2304 [I 2022-10-11 03:07:39] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:39 [I 2022-10-11 03:07:39] ...trap/engines/engine.py.229: process: 0.86333 | load: 0.00313 [I 2022-10-11 03:07:39] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-11 03:07:39] ...optimizers/trijoint.py.151: total_norm: 36.107498 [I 2022-10-11 03:07:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 880/2304 [I 2022-10-11 03:07:48] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:29 [I 2022-10-11 03:07:48] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00316 [I 2022-10-11 03:07:48] ...trap/engines/engine.py.230: loss: 0.14264 [I 2022-10-11 03:07:48] ...optimizers/trijoint.py.151: total_norm: 26.789852 [I 2022-10-11 03:07:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 890/2304 [I 2022-10-11 03:07:56] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:40 [I 2022-10-11 03:07:56] ...trap/engines/engine.py.229: process: 0.89212 | load: 0.00311 [I 2022-10-11 03:07:56] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-11 03:07:56] ...optimizers/trijoint.py.151: total_norm: 10.892282 [I 2022-10-11 03:08:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 900/2304 [I 2022-10-11 03:08:05] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:20:14 [I 2022-10-11 03:08:05] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00310 [I 2022-10-11 03:08:05] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-11 03:08:05] ...optimizers/trijoint.py.151: total_norm: 8.962742 [I 2022-10-11 03:08:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 910/2304 [I 2022-10-11 03:08:14] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:20:12 [I 2022-10-11 03:08:14] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00318 [I 2022-10-11 03:08:14] ...trap/engines/engine.py.230: loss: 0.21609 [I 2022-10-11 03:08:14] ...optimizers/trijoint.py.151: total_norm: 34.821182 [I 2022-10-11 03:08:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 920/2304 [I 2022-10-11 03:08:23] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:59 [I 2022-10-11 03:08:23] ...trap/engines/engine.py.229: process: 0.87429 | load: 0.00310 [I 2022-10-11 03:08:23] ...trap/engines/engine.py.230: loss: 0.11275 [I 2022-10-11 03:08:23] ...optimizers/trijoint.py.151: total_norm: 17.486694 [I 2022-10-11 03:08:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 930/2304 [I 2022-10-11 03:08:31] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:50 [I 2022-10-11 03:08:31] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00385 [I 2022-10-11 03:08:31] ...trap/engines/engine.py.230: loss: 0.09141 [I 2022-10-11 03:08:31] ...optimizers/trijoint.py.151: total_norm: 15.581540 [I 2022-10-11 03:08:40] ...trap/engines/engine.py.224: train: epoch 95 | batch 940/2304 [I 2022-10-11 03:08:40] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:19:39 [I 2022-10-11 03:08:40] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00317 [I 2022-10-11 03:08:40] ...trap/engines/engine.py.230: loss: 0.09385 [I 2022-10-11 03:08:40] ...optimizers/trijoint.py.151: total_norm: 12.509194 [I 2022-10-11 03:08:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 950/2304 [I 2022-10-11 03:08:49] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:29 [I 2022-10-11 03:08:49] ...trap/engines/engine.py.229: process: 0.86253 | load: 0.00310 [I 2022-10-11 03:08:49] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-11 03:08:49] ...optimizers/trijoint.py.151: total_norm: 13.657666 [I 2022-10-11 03:08:57] ...trap/engines/engine.py.224: train: epoch 95 | batch 960/2304 [I 2022-10-11 03:08:57] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:28 [I 2022-10-11 03:08:57] ...trap/engines/engine.py.229: process: 0.87862 | load: 0.00312 [I 2022-10-11 03:08:57] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-11 03:08:57] ...optimizers/trijoint.py.151: total_norm: 13.040027 [I 2022-10-11 03:09:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 970/2304 [I 2022-10-11 03:09:06] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:19:13 [I 2022-10-11 03:09:06] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00310 [I 2022-10-11 03:09:06] ...trap/engines/engine.py.230: loss: 0.14041 [I 2022-10-11 03:09:06] ...optimizers/trijoint.py.151: total_norm: 13.677037 [I 2022-10-11 03:09:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 980/2304 [I 2022-10-11 03:09:15] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:07 [I 2022-10-11 03:09:15] ...trap/engines/engine.py.229: process: 0.86576 | load: 0.00321 [I 2022-10-11 03:09:15] ...trap/engines/engine.py.230: loss: 0.10678 [I 2022-10-11 03:09:15] ...optimizers/trijoint.py.151: total_norm: 18.728548 [I 2022-10-11 03:09:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 990/2304 [I 2022-10-11 03:09:23] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:19:03 [I 2022-10-11 03:09:23] ...trap/engines/engine.py.229: process: 0.87623 | load: 0.00307 [I 2022-10-11 03:09:23] ...trap/engines/engine.py.230: loss: 0.13944 [I 2022-10-11 03:09:23] ...optimizers/trijoint.py.151: total_norm: 22.861710 [I 2022-10-11 03:09:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 1000/2304 [I 2022-10-11 03:09:32] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:51 [I 2022-10-11 03:09:32] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00325 [I 2022-10-11 03:09:32] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-11 03:09:32] ...optimizers/trijoint.py.151: total_norm: 13.735945 [I 2022-10-11 03:09:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 1010/2304 [I 2022-10-11 03:09:41] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:18:40 [I 2022-10-11 03:09:41] ...trap/engines/engine.py.229: process: 0.87919 | load: 0.00310 [I 2022-10-11 03:09:41] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-11 03:09:41] ...optimizers/trijoint.py.151: total_norm: 23.314238 [I 2022-10-11 03:09:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 1020/2304 [I 2022-10-11 03:09:49] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:18:27 [I 2022-10-11 03:09:49] ...trap/engines/engine.py.229: process: 0.86961 | load: 0.00298 [I 2022-10-11 03:09:49] ...trap/engines/engine.py.230: loss: 0.09446 [I 2022-10-11 03:09:49] ...optimizers/trijoint.py.151: total_norm: 9.037989 [I 2022-10-11 03:09:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 1030/2304 [I 2022-10-11 03:09:58] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:18:13 [I 2022-10-11 03:09:58] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00301 [I 2022-10-11 03:09:58] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-11 03:09:58] ...optimizers/trijoint.py.151: total_norm: 8.954077 [I 2022-10-11 03:10:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 1040/2304 [I 2022-10-11 03:10:06] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:17:59 [I 2022-10-11 03:10:06] ...trap/engines/engine.py.229: process: 0.85068 | load: 0.00294 [I 2022-10-11 03:10:06] ...trap/engines/engine.py.230: loss: 0.09920 [I 2022-10-11 03:10:06] ...optimizers/trijoint.py.151: total_norm: 10.869228 [I 2022-10-11 03:10:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 1050/2304 [I 2022-10-11 03:10:15] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:17:47 [I 2022-10-11 03:10:15] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00301 [I 2022-10-11 03:10:15] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-11 03:10:15] ...optimizers/trijoint.py.151: total_norm: 10.594239 [I 2022-10-11 03:10:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 1060/2304 [I 2022-10-11 03:10:24] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:46 [I 2022-10-11 03:10:24] ...trap/engines/engine.py.229: process: 0.84400 | load: 0.00298 [I 2022-10-11 03:10:24] ...trap/engines/engine.py.230: loss: 0.14333 [I 2022-10-11 03:10:24] ...optimizers/trijoint.py.151: total_norm: 21.836237 [I 2022-10-11 03:10:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 1070/2304 [I 2022-10-11 03:10:32] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:29 [I 2022-10-11 03:10:32] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00295 [I 2022-10-11 03:10:32] ...trap/engines/engine.py.230: loss: 0.10097 [I 2022-10-11 03:10:32] ...optimizers/trijoint.py.151: total_norm: 15.106323 [I 2022-10-11 03:10:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 1080/2304 [I 2022-10-11 03:10:41] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:17:33 [I 2022-10-11 03:10:41] ...trap/engines/engine.py.229: process: 0.86632 | load: 0.00311 [I 2022-10-11 03:10:41] ...trap/engines/engine.py.230: loss: 0.11951 [I 2022-10-11 03:10:41] ...optimizers/trijoint.py.151: total_norm: 18.410488 [I 2022-10-11 03:10:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 1090/2304 [I 2022-10-11 03:10:49] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:21 [I 2022-10-11 03:10:49] ...trap/engines/engine.py.229: process: 0.86302 | load: 0.00311 [I 2022-10-11 03:10:49] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-11 03:10:49] ...optimizers/trijoint.py.151: total_norm: 25.012890 [I 2022-10-11 03:10:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 1100/2304 [I 2022-10-11 03:10:58] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:22 [I 2022-10-11 03:10:58] ...trap/engines/engine.py.229: process: 0.86772 | load: 0.00297 [I 2022-10-11 03:10:58] ...trap/engines/engine.py.230: loss: 0.11098 [I 2022-10-11 03:10:58] ...optimizers/trijoint.py.151: total_norm: 14.993518 [I 2022-10-11 03:11:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 1110/2304 [I 2022-10-11 03:11:07] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:03 [I 2022-10-11 03:11:07] ...trap/engines/engine.py.229: process: 0.85605 | load: 0.00309 [I 2022-10-11 03:11:07] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-11 03:11:07] ...optimizers/trijoint.py.151: total_norm: 15.161177 [I 2022-10-11 03:11:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 1120/2304 [I 2022-10-11 03:11:15] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:01 [I 2022-10-11 03:11:15] ...trap/engines/engine.py.229: process: 0.86430 | load: 0.00310 [I 2022-10-11 03:11:15] ...trap/engines/engine.py.230: loss: 0.08887 [I 2022-10-11 03:11:15] ...optimizers/trijoint.py.151: total_norm: 15.008409 [I 2022-10-11 03:11:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 1130/2304 [I 2022-10-11 03:11:24] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:44 [I 2022-10-11 03:11:24] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00299 [I 2022-10-11 03:11:24] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-11 03:11:24] ...optimizers/trijoint.py.151: total_norm: 14.987761 [I 2022-10-11 03:11:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 1140/2304 [I 2022-10-11 03:11:32] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:34 [I 2022-10-11 03:11:32] ...trap/engines/engine.py.229: process: 0.85160 | load: 0.00297 [I 2022-10-11 03:11:32] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-11 03:11:32] ...optimizers/trijoint.py.151: total_norm: 10.284908 [I 2022-10-11 03:11:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 1150/2304 [I 2022-10-11 03:11:41] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:36 [I 2022-10-11 03:11:41] ...trap/engines/engine.py.229: process: 0.87271 | load: 0.00308 [I 2022-10-11 03:11:41] ...trap/engines/engine.py.230: loss: 0.11942 [I 2022-10-11 03:11:41] ...optimizers/trijoint.py.151: total_norm: 9.463085 [I 2022-10-11 03:11:50] ...trap/engines/engine.py.224: train: epoch 95 | batch 1160/2304 [I 2022-10-11 03:11:50] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:23 [I 2022-10-11 03:11:50] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00309 [I 2022-10-11 03:11:50] ...trap/engines/engine.py.230: loss: 0.11082 [I 2022-10-11 03:11:50] ...optimizers/trijoint.py.151: total_norm: 14.576178 [I 2022-10-11 03:11:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 1170/2304 [I 2022-10-11 03:11:58] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:23 [I 2022-10-11 03:11:58] ...trap/engines/engine.py.229: process: 0.88831 | load: 0.00306 [I 2022-10-11 03:11:58] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-11 03:11:58] ...optimizers/trijoint.py.151: total_norm: 16.809925 [I 2022-10-11 03:12:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 1180/2304 [I 2022-10-11 03:12:07] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:16:13 [I 2022-10-11 03:12:07] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00306 [I 2022-10-11 03:12:07] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-11 03:12:07] ...optimizers/trijoint.py.151: total_norm: 80.176674 [I 2022-10-11 03:12:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 1190/2304 [I 2022-10-11 03:12:16] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:16:03 [I 2022-10-11 03:12:16] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00316 [I 2022-10-11 03:12:16] ...trap/engines/engine.py.230: loss: 0.09069 [I 2022-10-11 03:12:16] ...optimizers/trijoint.py.151: total_norm: 9.324356 [I 2022-10-11 03:12:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 1200/2304 [I 2022-10-11 03:12:24] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:59 [I 2022-10-11 03:12:24] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00311 [I 2022-10-11 03:12:24] ...trap/engines/engine.py.230: loss: 0.16010 [I 2022-10-11 03:12:24] ...optimizers/trijoint.py.151: total_norm: 15.273624 [I 2022-10-11 03:12:33] ...trap/engines/engine.py.224: train: epoch 95 | batch 1210/2304 [I 2022-10-11 03:12:33] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:45 [I 2022-10-11 03:12:33] ...trap/engines/engine.py.229: process: 0.86641 | load: 0.00330 [I 2022-10-11 03:12:33] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-11 03:12:33] ...optimizers/trijoint.py.151: total_norm: 27.808157 [I 2022-10-11 03:12:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 1220/2304 [I 2022-10-11 03:12:42] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:38 [I 2022-10-11 03:12:42] ...trap/engines/engine.py.229: process: 0.88005 | load: 0.00315 [I 2022-10-11 03:12:42] ...trap/engines/engine.py.230: loss: 0.09626 [I 2022-10-11 03:12:42] ...optimizers/trijoint.py.151: total_norm: 13.228065 [I 2022-10-11 03:12:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 1230/2304 [I 2022-10-11 03:12:51] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:30 [I 2022-10-11 03:12:51] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00310 [I 2022-10-11 03:12:51] ...trap/engines/engine.py.230: loss: 0.08570 [I 2022-10-11 03:12:51] ...optimizers/trijoint.py.151: total_norm: 11.036822 [I 2022-10-11 03:12:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 1240/2304 [I 2022-10-11 03:12:59] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:18 [I 2022-10-11 03:12:59] ...trap/engines/engine.py.229: process: 0.85875 | load: 0.00319 [I 2022-10-11 03:12:59] ...trap/engines/engine.py.230: loss: 0.08845 [I 2022-10-11 03:12:59] ...optimizers/trijoint.py.151: total_norm: 15.312505 [I 2022-10-11 03:13:08] ...trap/engines/engine.py.224: train: epoch 95 | batch 1250/2304 [I 2022-10-11 03:13:08] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:15:13 [I 2022-10-11 03:13:08] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00311 [I 2022-10-11 03:13:08] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-11 03:13:08] ...optimizers/trijoint.py.151: total_norm: 8.888137 [I 2022-10-11 03:13:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 1260/2304 [I 2022-10-11 03:13:17] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:06 [I 2022-10-11 03:13:17] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00324 [I 2022-10-11 03:13:17] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-11 03:13:17] ...optimizers/trijoint.py.151: total_norm: 13.823838 [I 2022-10-11 03:13:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 1270/2304 [I 2022-10-11 03:13:25] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:52 [I 2022-10-11 03:13:25] ...trap/engines/engine.py.229: process: 0.86682 | load: 0.00316 [I 2022-10-11 03:13:25] ...trap/engines/engine.py.230: loss: 0.23026 [I 2022-10-11 03:13:25] ...optimizers/trijoint.py.151: total_norm: 23.481531 [I 2022-10-11 03:13:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 1280/2304 [I 2022-10-11 03:13:34] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:46 [I 2022-10-11 03:13:34] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00314 [I 2022-10-11 03:13:34] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-11 03:13:34] ...optimizers/trijoint.py.151: total_norm: 9.849901 [I 2022-10-11 03:13:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 1290/2304 [I 2022-10-11 03:13:43] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:32 [I 2022-10-11 03:13:43] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00310 [I 2022-10-11 03:13:43] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-11 03:13:43] ...optimizers/trijoint.py.151: total_norm: 12.366217 [I 2022-10-11 03:13:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 1300/2304 [I 2022-10-11 03:13:51] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:31 [I 2022-10-11 03:13:51] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00317 [I 2022-10-11 03:13:51] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-11 03:13:51] ...optimizers/trijoint.py.151: total_norm: 17.908150 [I 2022-10-11 03:14:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 1310/2304 [I 2022-10-11 03:14:00] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:17 [I 2022-10-11 03:14:00] ...trap/engines/engine.py.229: process: 0.87229 | load: 0.00309 [I 2022-10-11 03:14:00] ...trap/engines/engine.py.230: loss: 0.15264 [I 2022-10-11 03:14:00] ...optimizers/trijoint.py.151: total_norm: 26.032913 [I 2022-10-11 03:14:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 1320/2304 [I 2022-10-11 03:14:09] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:14:18 [I 2022-10-11 03:14:09] ...trap/engines/engine.py.229: process: 0.86881 | load: 0.00316 [I 2022-10-11 03:14:09] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-11 03:14:09] ...optimizers/trijoint.py.151: total_norm: 26.987347 [I 2022-10-11 03:14:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 1330/2304 [I 2022-10-11 03:14:17] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:14:01 [I 2022-10-11 03:14:17] ...trap/engines/engine.py.229: process: 0.86987 | load: 0.00307 [I 2022-10-11 03:14:17] ...trap/engines/engine.py.230: loss: 0.22344 [I 2022-10-11 03:14:17] ...optimizers/trijoint.py.151: total_norm: 16.304153 [I 2022-10-11 03:14:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 1340/2304 [I 2022-10-11 03:14:26] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:58 [I 2022-10-11 03:14:26] ...trap/engines/engine.py.229: process: 0.87208 | load: 0.00315 [I 2022-10-11 03:14:26] ...trap/engines/engine.py.230: loss: 0.09149 [I 2022-10-11 03:14:26] ...optimizers/trijoint.py.151: total_norm: 11.256067 [I 2022-10-11 03:14:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 1350/2304 [I 2022-10-11 03:14:35] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:48 [I 2022-10-11 03:14:35] ...trap/engines/engine.py.229: process: 0.88863 | load: 0.00325 [I 2022-10-11 03:14:35] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-11 03:14:35] ...optimizers/trijoint.py.151: total_norm: 32.519630 [I 2022-10-11 03:14:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 1360/2304 [I 2022-10-11 03:14:43] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:42 [I 2022-10-11 03:14:43] ...trap/engines/engine.py.229: process: 0.87318 | load: 0.00318 [I 2022-10-11 03:14:43] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-11 03:14:43] ...optimizers/trijoint.py.151: total_norm: 14.079777 [I 2022-10-11 03:14:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 1370/2304 [I 2022-10-11 03:14:52] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:32 [I 2022-10-11 03:14:52] ...trap/engines/engine.py.229: process: 0.87378 | load: 0.00322 [I 2022-10-11 03:14:52] ...trap/engines/engine.py.230: loss: 0.18311 [I 2022-10-11 03:14:52] ...optimizers/trijoint.py.151: total_norm: 18.114338 [I 2022-10-11 03:15:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 1380/2304 [I 2022-10-11 03:15:01] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:13:18 [I 2022-10-11 03:15:01] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00312 [I 2022-10-11 03:15:01] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-11 03:15:01] ...optimizers/trijoint.py.151: total_norm: 18.867598 [I 2022-10-11 03:15:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 1390/2304 [I 2022-10-11 03:15:09] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:13:11 [I 2022-10-11 03:15:09] ...trap/engines/engine.py.229: process: 0.86319 | load: 0.00310 [I 2022-10-11 03:15:09] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-11 03:15:09] ...optimizers/trijoint.py.151: total_norm: 12.985747 [I 2022-10-11 03:15:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 1400/2304 [I 2022-10-11 03:15:18] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:13:02 [I 2022-10-11 03:15:18] ...trap/engines/engine.py.229: process: 0.87551 | load: 0.00311 [I 2022-10-11 03:15:18] ...trap/engines/engine.py.230: loss: 0.09893 [I 2022-10-11 03:15:18] ...optimizers/trijoint.py.151: total_norm: 15.987191 [I 2022-10-11 03:15:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 1410/2304 [I 2022-10-11 03:15:27] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:56 [I 2022-10-11 03:15:27] ...trap/engines/engine.py.229: process: 0.86988 | load: 0.00317 [I 2022-10-11 03:15:27] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-11 03:15:27] ...optimizers/trijoint.py.151: total_norm: 11.510137 [I 2022-10-11 03:15:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 1420/2304 [I 2022-10-11 03:15:35] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:39 [I 2022-10-11 03:15:35] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00294 [I 2022-10-11 03:15:35] ...trap/engines/engine.py.230: loss: 0.16294 [I 2022-10-11 03:15:35] ...optimizers/trijoint.py.151: total_norm: 15.174292 [I 2022-10-11 03:15:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 1430/2304 [I 2022-10-11 03:15:44] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:27 [I 2022-10-11 03:15:44] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00294 [I 2022-10-11 03:15:44] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-11 03:15:44] ...optimizers/trijoint.py.151: total_norm: 18.175276 [I 2022-10-11 03:15:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 1440/2304 [I 2022-10-11 03:15:53] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:21 [I 2022-10-11 03:15:53] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00299 [I 2022-10-11 03:15:53] ...trap/engines/engine.py.230: loss: 0.11510 [I 2022-10-11 03:15:53] ...optimizers/trijoint.py.151: total_norm: 8.958964 [I 2022-10-11 03:16:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 1450/2304 [I 2022-10-11 03:16:01] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:08 [I 2022-10-11 03:16:01] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00298 [I 2022-10-11 03:16:01] ...trap/engines/engine.py.230: loss: 0.19910 [I 2022-10-11 03:16:01] ...optimizers/trijoint.py.151: total_norm: 22.323952 [I 2022-10-11 03:16:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 1460/2304 [I 2022-10-11 03:16:10] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:12:04 [I 2022-10-11 03:16:10] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00298 [I 2022-10-11 03:16:10] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-11 03:16:10] ...optimizers/trijoint.py.151: total_norm: 20.630152 [I 2022-10-11 03:16:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 1470/2304 [I 2022-10-11 03:16:18] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:11:54 [I 2022-10-11 03:16:18] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00300 [I 2022-10-11 03:16:18] ...trap/engines/engine.py.230: loss: 0.18677 [I 2022-10-11 03:16:18] ...optimizers/trijoint.py.151: total_norm: 19.270336 [I 2022-10-11 03:16:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 1480/2304 [I 2022-10-11 03:16:27] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:48 [I 2022-10-11 03:16:27] ...trap/engines/engine.py.229: process: 0.86725 | load: 0.00297 [I 2022-10-11 03:16:27] ...trap/engines/engine.py.230: loss: 0.10417 [I 2022-10-11 03:16:27] ...optimizers/trijoint.py.151: total_norm: 10.430265 [I 2022-10-11 03:16:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 1490/2304 [I 2022-10-11 03:16:36] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:38 [I 2022-10-11 03:16:36] ...trap/engines/engine.py.229: process: 0.85362 | load: 0.00299 [I 2022-10-11 03:16:36] ...trap/engines/engine.py.230: loss: 0.08860 [I 2022-10-11 03:16:36] ...optimizers/trijoint.py.151: total_norm: 11.225689 [I 2022-10-11 03:16:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 1500/2304 [I 2022-10-11 03:16:44] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:31 [I 2022-10-11 03:16:44] ...trap/engines/engine.py.229: process: 0.86337 | load: 0.00301 [I 2022-10-11 03:16:44] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-11 03:16:44] ...optimizers/trijoint.py.151: total_norm: 12.725046 [I 2022-10-11 03:16:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 1510/2304 [I 2022-10-11 03:16:53] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:20 [I 2022-10-11 03:16:53] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00297 [I 2022-10-11 03:16:53] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-11 03:16:53] ...optimizers/trijoint.py.151: total_norm: 25.580086 [I 2022-10-11 03:17:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 1520/2304 [I 2022-10-11 03:17:01] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:08 [I 2022-10-11 03:17:01] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00294 [I 2022-10-11 03:17:01] ...trap/engines/engine.py.230: loss: 0.10573 [I 2022-10-11 03:17:01] ...optimizers/trijoint.py.151: total_norm: 15.084730 [I 2022-10-11 03:17:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 1530/2304 [I 2022-10-11 03:17:10] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:02 [I 2022-10-11 03:17:10] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00298 [I 2022-10-11 03:17:10] ...trap/engines/engine.py.230: loss: 0.11653 [I 2022-10-11 03:17:10] ...optimizers/trijoint.py.151: total_norm: 8.013310 [I 2022-10-11 03:17:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 1540/2304 [I 2022-10-11 03:17:19] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:52 [I 2022-10-11 03:17:19] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00300 [I 2022-10-11 03:17:19] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-11 03:17:19] ...optimizers/trijoint.py.151: total_norm: 11.449542 [I 2022-10-11 03:17:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 1550/2304 [I 2022-10-11 03:17:27] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:45 [I 2022-10-11 03:17:27] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00299 [I 2022-10-11 03:17:27] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-11 03:17:27] ...optimizers/trijoint.py.151: total_norm: 10.213286 [I 2022-10-11 03:17:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 1560/2304 [I 2022-10-11 03:17:36] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:45 [I 2022-10-11 03:17:36] ...trap/engines/engine.py.229: process: 0.86795 | load: 0.00320 [I 2022-10-11 03:17:36] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-11 03:17:36] ...optimizers/trijoint.py.151: total_norm: 31.980433 [I 2022-10-11 03:17:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 1570/2304 [I 2022-10-11 03:17:45] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:35 [I 2022-10-11 03:17:45] ...trap/engines/engine.py.229: process: 0.87355 | load: 0.00312 [I 2022-10-11 03:17:45] ...trap/engines/engine.py.230: loss: 0.17501 [I 2022-10-11 03:17:45] ...optimizers/trijoint.py.151: total_norm: 28.678879 [I 2022-10-11 03:17:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 1580/2304 [I 2022-10-11 03:17:53] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:26 [I 2022-10-11 03:17:53] ...trap/engines/engine.py.229: process: 0.87687 | load: 0.00354 [I 2022-10-11 03:17:53] ...trap/engines/engine.py.230: loss: 0.06989 [I 2022-10-11 03:17:53] ...optimizers/trijoint.py.151: total_norm: 13.066447 [I 2022-10-11 03:18:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 1590/2304 [I 2022-10-11 03:18:02] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:16 [I 2022-10-11 03:18:02] ...trap/engines/engine.py.229: process: 0.85474 | load: 0.00311 [I 2022-10-11 03:18:02] ...trap/engines/engine.py.230: loss: 0.11504 [I 2022-10-11 03:18:02] ...optimizers/trijoint.py.151: total_norm: 11.368933 [I 2022-10-11 03:18:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 1600/2304 [I 2022-10-11 03:18:11] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:08 [I 2022-10-11 03:18:11] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00309 [I 2022-10-11 03:18:11] ...trap/engines/engine.py.230: loss: 0.13429 [I 2022-10-11 03:18:11] ...optimizers/trijoint.py.151: total_norm: 14.875572 [I 2022-10-11 03:18:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 1610/2304 [I 2022-10-11 03:18:19] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:10:00 [I 2022-10-11 03:18:19] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00322 [I 2022-10-11 03:18:19] ...trap/engines/engine.py.230: loss: 0.09453 [I 2022-10-11 03:18:19] ...optimizers/trijoint.py.151: total_norm: 10.410995 [I 2022-10-11 03:18:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 1620/2304 [I 2022-10-11 03:18:28] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:51 [I 2022-10-11 03:18:28] ...trap/engines/engine.py.229: process: 0.86558 | load: 0.00323 [I 2022-10-11 03:18:28] ...trap/engines/engine.py.230: loss: 0.12409 [I 2022-10-11 03:18:28] ...optimizers/trijoint.py.151: total_norm: 20.909498 [I 2022-10-11 03:18:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 1630/2304 [I 2022-10-11 03:18:37] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:42 [I 2022-10-11 03:18:37] ...trap/engines/engine.py.229: process: 0.87345 | load: 0.00311 [I 2022-10-11 03:18:37] ...trap/engines/engine.py.230: loss: 0.15750 [I 2022-10-11 03:18:37] ...optimizers/trijoint.py.151: total_norm: 10.250780 [I 2022-10-11 03:18:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 1640/2304 [I 2022-10-11 03:18:45] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:30 [I 2022-10-11 03:18:45] ...trap/engines/engine.py.229: process: 0.85446 | load: 0.00305 [I 2022-10-11 03:18:45] ...trap/engines/engine.py.230: loss: 0.14937 [I 2022-10-11 03:18:45] ...optimizers/trijoint.py.151: total_norm: 16.220657 [I 2022-10-11 03:18:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 1650/2304 [I 2022-10-11 03:18:54] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:26 [I 2022-10-11 03:18:54] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00307 [I 2022-10-11 03:18:54] ...trap/engines/engine.py.230: loss: 0.11064 [I 2022-10-11 03:18:54] ...optimizers/trijoint.py.151: total_norm: 12.830784 [I 2022-10-11 03:19:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 1660/2304 [I 2022-10-11 03:19:03] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:17 [I 2022-10-11 03:19:03] ...trap/engines/engine.py.229: process: 0.86597 | load: 0.00315 [I 2022-10-11 03:19:03] ...trap/engines/engine.py.230: loss: 0.08516 [I 2022-10-11 03:19:03] ...optimizers/trijoint.py.151: total_norm: 19.860641 [I 2022-10-11 03:19:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 1670/2304 [I 2022-10-11 03:19:11] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:07 [I 2022-10-11 03:19:11] ...trap/engines/engine.py.229: process: 0.86856 | load: 0.00315 [I 2022-10-11 03:19:11] ...trap/engines/engine.py.230: loss: 0.17494 [I 2022-10-11 03:19:11] ...optimizers/trijoint.py.151: total_norm: 18.947466 [I 2022-10-11 03:19:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 1680/2304 [I 2022-10-11 03:19:20] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:08:58 [I 2022-10-11 03:19:20] ...trap/engines/engine.py.229: process: 0.87324 | load: 0.00333 [I 2022-10-11 03:19:20] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-11 03:19:20] ...optimizers/trijoint.py.151: total_norm: 13.823944 [I 2022-10-11 03:19:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 1690/2304 [I 2022-10-11 03:19:29] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:49 [I 2022-10-11 03:19:29] ...trap/engines/engine.py.229: process: 0.86533 | load: 0.00307 [I 2022-10-11 03:19:29] ...trap/engines/engine.py.230: loss: 0.08738 [I 2022-10-11 03:19:29] ...optimizers/trijoint.py.151: total_norm: 7.568057 [I 2022-10-11 03:19:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 1700/2304 [I 2022-10-11 03:19:37] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:43 [I 2022-10-11 03:19:37] ...trap/engines/engine.py.229: process: 0.86927 | load: 0.00313 [I 2022-10-11 03:19:37] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-11 03:19:37] ...optimizers/trijoint.py.151: total_norm: 11.944034 [I 2022-10-11 03:19:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 1710/2304 [I 2022-10-11 03:19:46] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:33 [I 2022-10-11 03:19:46] ...trap/engines/engine.py.229: process: 0.85590 | load: 0.00306 [I 2022-10-11 03:19:46] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-11 03:19:46] ...optimizers/trijoint.py.151: total_norm: 11.374234 [I 2022-10-11 03:19:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 1720/2304 [I 2022-10-11 03:19:55] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:24 [I 2022-10-11 03:19:55] ...trap/engines/engine.py.229: process: 0.86246 | load: 0.00309 [I 2022-10-11 03:19:55] ...trap/engines/engine.py.230: loss: 0.11918 [I 2022-10-11 03:19:55] ...optimizers/trijoint.py.151: total_norm: 15.342406 [I 2022-10-11 03:20:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 1730/2304 [I 2022-10-11 03:20:03] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:21 [I 2022-10-11 03:20:03] ...trap/engines/engine.py.229: process: 0.89226 | load: 0.00309 [I 2022-10-11 03:20:03] ...trap/engines/engine.py.230: loss: 0.12552 [I 2022-10-11 03:20:03] ...optimizers/trijoint.py.151: total_norm: 15.428948 [I 2022-10-11 03:20:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 1740/2304 [I 2022-10-11 03:20:12] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:08:09 [I 2022-10-11 03:20:12] ...trap/engines/engine.py.229: process: 0.85909 | load: 0.00312 [I 2022-10-11 03:20:12] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-11 03:20:12] ...optimizers/trijoint.py.151: total_norm: 16.949051 [I 2022-10-11 03:20:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 1750/2304 [I 2022-10-11 03:20:21] ...trap/engines/engine.py.228: elapsed: 0:25:17 | left: 0:08:00 [I 2022-10-11 03:20:21] ...trap/engines/engine.py.229: process: 0.86496 | load: 0.00330 [I 2022-10-11 03:20:21] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-11 03:20:21] ...optimizers/trijoint.py.151: total_norm: 15.957623 [I 2022-10-11 03:20:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1760/2304 [I 2022-10-11 03:20:30] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:54 [I 2022-10-11 03:20:30] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00314 [I 2022-10-11 03:20:30] ...trap/engines/engine.py.230: loss: 0.09902 [I 2022-10-11 03:20:30] ...optimizers/trijoint.py.151: total_norm: 15.498476 [I 2022-10-11 03:20:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 1770/2304 [I 2022-10-11 03:20:38] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:38 [I 2022-10-11 03:20:38] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00321 [I 2022-10-11 03:20:38] ...trap/engines/engine.py.230: loss: 0.14640 [I 2022-10-11 03:20:38] ...optimizers/trijoint.py.151: total_norm: 28.892576 [I 2022-10-11 03:20:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 1780/2304 [I 2022-10-11 03:20:47] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:31 [I 2022-10-11 03:20:47] ...trap/engines/engine.py.229: process: 0.86536 | load: 0.00310 [I 2022-10-11 03:20:47] ...trap/engines/engine.py.230: loss: 0.10436 [I 2022-10-11 03:20:47] ...optimizers/trijoint.py.151: total_norm: 11.096580 [I 2022-10-11 03:20:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 1790/2304 [I 2022-10-11 03:20:55] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:22 [I 2022-10-11 03:20:55] ...trap/engines/engine.py.229: process: 0.84631 | load: 0.00308 [I 2022-10-11 03:20:55] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-11 03:20:55] ...optimizers/trijoint.py.151: total_norm: 29.141680 [I 2022-10-11 03:21:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 1800/2304 [I 2022-10-11 03:21:04] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:18 [I 2022-10-11 03:21:04] ...trap/engines/engine.py.229: process: 0.87015 | load: 0.00317 [I 2022-10-11 03:21:04] ...trap/engines/engine.py.230: loss: 0.17008 [I 2022-10-11 03:21:04] ...optimizers/trijoint.py.151: total_norm: 26.708937 [I 2022-10-11 03:21:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 1810/2304 [I 2022-10-11 03:21:13] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:08 [I 2022-10-11 03:21:13] ...trap/engines/engine.py.229: process: 0.86575 | load: 0.00320 [I 2022-10-11 03:21:13] ...trap/engines/engine.py.230: loss: 0.18023 [I 2022-10-11 03:21:13] ...optimizers/trijoint.py.151: total_norm: 25.108244 [I 2022-10-11 03:21:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 1820/2304 [I 2022-10-11 03:21:22] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:57 [I 2022-10-11 03:21:22] ...trap/engines/engine.py.229: process: 0.85371 | load: 0.00313 [I 2022-10-11 03:21:22] ...trap/engines/engine.py.230: loss: 0.11853 [I 2022-10-11 03:21:22] ...optimizers/trijoint.py.151: total_norm: 10.724688 [I 2022-10-11 03:21:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1830/2304 [I 2022-10-11 03:21:30] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:44 [I 2022-10-11 03:21:30] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00303 [I 2022-10-11 03:21:30] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-11 03:21:30] ...optimizers/trijoint.py.151: total_norm: 12.199569 [I 2022-10-11 03:21:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 1840/2304 [I 2022-10-11 03:21:39] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:36 [I 2022-10-11 03:21:39] ...trap/engines/engine.py.229: process: 0.86103 | load: 0.00296 [I 2022-10-11 03:21:39] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-11 03:21:39] ...optimizers/trijoint.py.151: total_norm: 15.297542 [I 2022-10-11 03:21:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 1850/2304 [I 2022-10-11 03:21:47] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:31 [I 2022-10-11 03:21:47] ...trap/engines/engine.py.229: process: 0.86191 | load: 0.00304 [I 2022-10-11 03:21:47] ...trap/engines/engine.py.230: loss: 0.08542 [I 2022-10-11 03:21:47] ...optimizers/trijoint.py.151: total_norm: 9.568587 [I 2022-10-11 03:21:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 1860/2304 [I 2022-10-11 03:21:56] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:20 [I 2022-10-11 03:21:56] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00300 [I 2022-10-11 03:21:56] ...trap/engines/engine.py.230: loss: 0.11921 [I 2022-10-11 03:21:56] ...optimizers/trijoint.py.151: total_norm: 12.473083 [I 2022-10-11 03:22:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 1870/2304 [I 2022-10-11 03:22:05] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:13 [I 2022-10-11 03:22:05] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00300 [I 2022-10-11 03:22:05] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-11 03:22:05] ...optimizers/trijoint.py.151: total_norm: 24.175472 [I 2022-10-11 03:22:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 1880/2304 [I 2022-10-11 03:22:13] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:02 [I 2022-10-11 03:22:13] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00295 [I 2022-10-11 03:22:13] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-11 03:22:13] ...optimizers/trijoint.py.151: total_norm: 15.255885 [I 2022-10-11 03:22:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 1890/2304 [I 2022-10-11 03:22:22] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:52 [I 2022-10-11 03:22:22] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00296 [I 2022-10-11 03:22:22] ...trap/engines/engine.py.230: loss: 0.10458 [I 2022-10-11 03:22:22] ...optimizers/trijoint.py.151: total_norm: 13.897188 [I 2022-10-11 03:22:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1900/2304 [I 2022-10-11 03:22:30] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:46 [I 2022-10-11 03:22:30] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00295 [I 2022-10-11 03:22:30] ...trap/engines/engine.py.230: loss: 0.13604 [I 2022-10-11 03:22:30] ...optimizers/trijoint.py.151: total_norm: 11.999537 [I 2022-10-11 03:22:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 1910/2304 [I 2022-10-11 03:22:39] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:38 [I 2022-10-11 03:22:39] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00301 [I 2022-10-11 03:22:39] ...trap/engines/engine.py.230: loss: 0.08606 [I 2022-10-11 03:22:39] ...optimizers/trijoint.py.151: total_norm: 11.954921 [I 2022-10-11 03:22:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 1920/2304 [I 2022-10-11 03:22:47] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:30 [I 2022-10-11 03:22:47] ...trap/engines/engine.py.229: process: 0.86167 | load: 0.00296 [I 2022-10-11 03:22:47] ...trap/engines/engine.py.230: loss: 0.10944 [I 2022-10-11 03:22:47] ...optimizers/trijoint.py.151: total_norm: 15.683384 [I 2022-10-11 03:22:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 1930/2304 [I 2022-10-11 03:22:56] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:21 [I 2022-10-11 03:22:56] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00298 [I 2022-10-11 03:22:56] ...trap/engines/engine.py.230: loss: 0.18009 [I 2022-10-11 03:22:56] ...optimizers/trijoint.py.151: total_norm: 21.935564 [I 2022-10-11 03:23:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 1940/2304 [I 2022-10-11 03:23:05] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:10 [I 2022-10-11 03:23:05] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00298 [I 2022-10-11 03:23:05] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-11 03:23:05] ...optimizers/trijoint.py.151: total_norm: 15.763380 [I 2022-10-11 03:23:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 1950/2304 [I 2022-10-11 03:23:13] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:03 [I 2022-10-11 03:23:13] ...trap/engines/engine.py.229: process: 0.86267 | load: 0.00298 [I 2022-10-11 03:23:13] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-11 03:23:13] ...optimizers/trijoint.py.151: total_norm: 10.130744 [I 2022-10-11 03:23:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 1960/2304 [I 2022-10-11 03:23:22] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:53 [I 2022-10-11 03:23:22] ...trap/engines/engine.py.229: process: 0.84924 | load: 0.00297 [I 2022-10-11 03:23:22] ...trap/engines/engine.py.230: loss: 0.12801 [I 2022-10-11 03:23:22] ...optimizers/trijoint.py.151: total_norm: 12.696066 [I 2022-10-11 03:23:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1970/2304 [I 2022-10-11 03:23:30] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:49 [I 2022-10-11 03:23:30] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00317 [I 2022-10-11 03:23:30] ...trap/engines/engine.py.230: loss: 0.17318 [I 2022-10-11 03:23:30] ...optimizers/trijoint.py.151: total_norm: 12.772618 [I 2022-10-11 03:23:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 1980/2304 [I 2022-10-11 03:23:39] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:40 [I 2022-10-11 03:23:39] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00330 [I 2022-10-11 03:23:39] ...trap/engines/engine.py.230: loss: 0.10851 [I 2022-10-11 03:23:39] ...optimizers/trijoint.py.151: total_norm: 10.704522 [I 2022-10-11 03:23:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 1990/2304 [I 2022-10-11 03:23:48] ...trap/engines/engine.py.228: elapsed: 0:28:44 | left: 0:04:30 [I 2022-10-11 03:23:48] ...trap/engines/engine.py.229: process: 0.86600 | load: 0.00307 [I 2022-10-11 03:23:48] ...trap/engines/engine.py.230: loss: 0.13074 [I 2022-10-11 03:23:48] ...optimizers/trijoint.py.151: total_norm: 18.819283 [I 2022-10-11 03:23:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 2000/2304 [I 2022-10-11 03:23:56] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:24 [I 2022-10-11 03:23:56] ...trap/engines/engine.py.229: process: 0.87688 | load: 0.00310 [I 2022-10-11 03:23:56] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-11 03:23:56] ...optimizers/trijoint.py.151: total_norm: 17.793394 [I 2022-10-11 03:24:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 2010/2304 [I 2022-10-11 03:24:05] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:04:15 [I 2022-10-11 03:24:05] ...trap/engines/engine.py.229: process: 0.87355 | load: 0.00308 [I 2022-10-11 03:24:05] ...trap/engines/engine.py.230: loss: 0.08029 [I 2022-10-11 03:24:05] ...optimizers/trijoint.py.151: total_norm: 10.959796 [I 2022-10-11 03:24:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 2020/2304 [I 2022-10-11 03:24:14] ...trap/engines/engine.py.228: elapsed: 0:29:10 | left: 0:04:06 [I 2022-10-11 03:24:14] ...trap/engines/engine.py.229: process: 0.85748 | load: 0.00308 [I 2022-10-11 03:24:14] ...trap/engines/engine.py.230: loss: 0.12495 [I 2022-10-11 03:24:14] ...optimizers/trijoint.py.151: total_norm: 14.775491 [I 2022-10-11 03:24:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 2030/2304 [I 2022-10-11 03:24:23] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:57 [I 2022-10-11 03:24:23] ...trap/engines/engine.py.229: process: 0.86759 | load: 0.00309 [I 2022-10-11 03:24:23] ...trap/engines/engine.py.230: loss: 0.25939 [I 2022-10-11 03:24:23] ...optimizers/trijoint.py.151: total_norm: 19.604786 [I 2022-10-11 03:24:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 2040/2304 [I 2022-10-11 03:24:31] ...trap/engines/engine.py.228: elapsed: 0:29:27 | left: 0:03:48 [I 2022-10-11 03:24:31] ...trap/engines/engine.py.229: process: 0.87227 | load: 0.00307 [I 2022-10-11 03:24:31] ...trap/engines/engine.py.230: loss: 0.10106 [I 2022-10-11 03:24:31] ...optimizers/trijoint.py.151: total_norm: 19.575855 [I 2022-10-11 03:24:40] ...trap/engines/engine.py.224: train: epoch 95 | batch 2050/2304 [I 2022-10-11 03:24:40] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:03:40 [I 2022-10-11 03:24:40] ...trap/engines/engine.py.229: process: 0.88115 | load: 0.00308 [I 2022-10-11 03:24:40] ...trap/engines/engine.py.230: loss: 0.11444 [I 2022-10-11 03:24:40] ...optimizers/trijoint.py.151: total_norm: 14.341168 [I 2022-10-11 03:24:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 2060/2304 [I 2022-10-11 03:24:49] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:32 [I 2022-10-11 03:24:49] ...trap/engines/engine.py.229: process: 0.86876 | load: 0.00323 [I 2022-10-11 03:24:49] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-11 03:24:49] ...optimizers/trijoint.py.151: total_norm: 10.565355 [I 2022-10-11 03:24:57] ...trap/engines/engine.py.224: train: epoch 95 | batch 2070/2304 [I 2022-10-11 03:24:57] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:03:23 [I 2022-10-11 03:24:57] ...trap/engines/engine.py.229: process: 0.86900 | load: 0.00313 [I 2022-10-11 03:24:57] ...trap/engines/engine.py.230: loss: 0.18738 [I 2022-10-11 03:24:57] ...optimizers/trijoint.py.151: total_norm: 20.550545 [I 2022-10-11 03:25:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 2080/2304 [I 2022-10-11 03:25:06] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:14 [I 2022-10-11 03:25:06] ...trap/engines/engine.py.229: process: 0.87144 | load: 0.00328 [I 2022-10-11 03:25:06] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-11 03:25:06] ...optimizers/trijoint.py.151: total_norm: 22.553799 [I 2022-10-11 03:25:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 2090/2304 [I 2022-10-11 03:25:34] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:05:59 [I 2022-10-11 03:25:34] ...trap/engines/engine.py.229: process: 0.86614 | load: 0.00308 [I 2022-10-11 03:25:34] ...trap/engines/engine.py.230: loss: 0.09325 [I 2022-10-11 03:25:34] ...optimizers/trijoint.py.151: total_norm: 7.400262 [I 2022-10-11 03:25:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 2100/2304 [I 2022-10-11 03:25:43] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:03:15 [I 2022-10-11 03:25:43] ...trap/engines/engine.py.229: process: 0.87210 | load: 0.00321 [I 2022-10-11 03:25:43] ...trap/engines/engine.py.230: loss: 0.18467 [I 2022-10-11 03:25:43] ...optimizers/trijoint.py.151: total_norm: 29.434086 [I 2022-10-11 03:25:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 2110/2304 [I 2022-10-11 03:25:52] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:02:48 [I 2022-10-11 03:25:52] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00309 [I 2022-10-11 03:25:52] ...trap/engines/engine.py.230: loss: 0.09060 [I 2022-10-11 03:25:52] ...optimizers/trijoint.py.151: total_norm: 9.817046 [I 2022-10-11 03:26:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 2120/2304 [I 2022-10-11 03:26:00] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:02:38 [I 2022-10-11 03:26:00] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00308 [I 2022-10-11 03:26:00] ...trap/engines/engine.py.230: loss: 0.08604 [I 2022-10-11 03:26:00] ...optimizers/trijoint.py.151: total_norm: 14.422213 [I 2022-10-11 03:26:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 2130/2304 [I 2022-10-11 03:26:09] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:30 [I 2022-10-11 03:26:09] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00315 [I 2022-10-11 03:26:09] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-11 03:26:09] ...optimizers/trijoint.py.151: total_norm: 8.897075 [I 2022-10-11 03:26:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 2140/2304 [I 2022-10-11 03:26:18] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:02:22 [I 2022-10-11 03:26:18] ...trap/engines/engine.py.229: process: 0.86963 | load: 0.00321 [I 2022-10-11 03:26:18] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-11 03:26:18] ...optimizers/trijoint.py.151: total_norm: 11.873370 [I 2022-10-11 03:26:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 2150/2304 [I 2022-10-11 03:26:26] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:02:14 [I 2022-10-11 03:26:26] ...trap/engines/engine.py.229: process: 0.87233 | load: 0.00308 [I 2022-10-11 03:26:26] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-11 03:26:26] ...optimizers/trijoint.py.151: total_norm: 21.193083 [I 2022-10-11 03:26:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 2160/2304 [I 2022-10-11 03:26:35] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:02:05 [I 2022-10-11 03:26:35] ...trap/engines/engine.py.229: process: 0.87532 | load: 0.00309 [I 2022-10-11 03:26:35] ...trap/engines/engine.py.230: loss: 0.18138 [I 2022-10-11 03:26:35] ...optimizers/trijoint.py.151: total_norm: 16.861145 [I 2022-10-11 03:26:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 2170/2304 [I 2022-10-11 03:26:44] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:56 [I 2022-10-11 03:26:44] ...trap/engines/engine.py.229: process: 0.86958 | load: 0.00310 [I 2022-10-11 03:26:44] ...trap/engines/engine.py.230: loss: 0.18486 [I 2022-10-11 03:26:44] ...optimizers/trijoint.py.151: total_norm: 17.797390 [I 2022-10-11 03:26:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 2180/2304 [I 2022-10-11 03:26:52] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:47 [I 2022-10-11 03:26:52] ...trap/engines/engine.py.229: process: 0.86377 | load: 0.00312 [I 2022-10-11 03:26:52] ...trap/engines/engine.py.230: loss: 0.09897 [I 2022-10-11 03:26:52] ...optimizers/trijoint.py.151: total_norm: 15.595940 [I 2022-10-11 03:27:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 2190/2304 [I 2022-10-11 03:27:01] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:39 [I 2022-10-11 03:27:01] ...trap/engines/engine.py.229: process: 0.86516 | load: 0.00311 [I 2022-10-11 03:27:01] ...trap/engines/engine.py.230: loss: 0.15938 [I 2022-10-11 03:27:01] ...optimizers/trijoint.py.151: total_norm: 25.530712 [I 2022-10-11 03:27:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 2200/2304 [I 2022-10-11 03:27:10] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:01:30 [I 2022-10-11 03:27:10] ...trap/engines/engine.py.229: process: 0.87725 | load: 0.00320 [I 2022-10-11 03:27:10] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-11 03:27:10] ...optimizers/trijoint.py.151: total_norm: 25.716013 [I 2022-10-11 03:27:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 2210/2304 [I 2022-10-11 03:27:18] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:01:21 [I 2022-10-11 03:27:18] ...trap/engines/engine.py.229: process: 0.87053 | load: 0.00307 [I 2022-10-11 03:27:18] ...trap/engines/engine.py.230: loss: 0.12325 [I 2022-10-11 03:27:18] ...optimizers/trijoint.py.151: total_norm: 19.710455 [I 2022-10-11 03:27:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 2220/2304 [I 2022-10-11 03:27:27] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:01:11 [I 2022-10-11 03:27:27] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00294 [I 2022-10-11 03:27:27] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-11 03:27:27] ...optimizers/trijoint.py.151: total_norm: 15.003091 [I 2022-10-11 03:27:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 2230/2304 [I 2022-10-11 03:27:36] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:01:03 [I 2022-10-11 03:27:36] ...trap/engines/engine.py.229: process: 0.85913 | load: 0.00303 [I 2022-10-11 03:27:36] ...trap/engines/engine.py.230: loss: 0.12972 [I 2022-10-11 03:27:36] ...optimizers/trijoint.py.151: total_norm: 16.033049 [I 2022-10-11 03:27:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 2240/2304 [I 2022-10-11 03:27:44] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:54 [I 2022-10-11 03:27:44] ...trap/engines/engine.py.229: process: 0.86062 | load: 0.00299 [I 2022-10-11 03:27:44] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-11 03:27:44] ...optimizers/trijoint.py.151: total_norm: 10.748343 [I 2022-10-11 03:27:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 2250/2304 [I 2022-10-11 03:27:53] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:46 [I 2022-10-11 03:27:53] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00299 [I 2022-10-11 03:27:53] ...trap/engines/engine.py.230: loss: 0.11423 [I 2022-10-11 03:27:53] ...optimizers/trijoint.py.151: total_norm: 7.642997 [I 2022-10-11 03:28:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 2260/2304 [I 2022-10-11 03:28:01] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:37 [I 2022-10-11 03:28:01] ...trap/engines/engine.py.229: process: 0.85996 | load: 0.00296 [I 2022-10-11 03:28:01] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-11 03:28:01] ...optimizers/trijoint.py.151: total_norm: 9.811406 [I 2022-10-11 03:28:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 2270/2304 [I 2022-10-11 03:28:10] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:29 [I 2022-10-11 03:28:10] ...trap/engines/engine.py.229: process: 0.85966 | load: 0.00301 [I 2022-10-11 03:28:10] ...trap/engines/engine.py.230: loss: 0.11628 [I 2022-10-11 03:28:10] ...optimizers/trijoint.py.151: total_norm: 12.819115 [I 2022-10-11 03:28:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 2280/2304 [I 2022-10-11 03:28:19] ...trap/engines/engine.py.228: elapsed: 0:33:14 | left: 0:00:20 [I 2022-10-11 03:28:19] ...trap/engines/engine.py.229: process: 0.85690 | load: 0.00297 [I 2022-10-11 03:28:19] ...trap/engines/engine.py.230: loss: 0.07966 [I 2022-10-11 03:28:19] ...optimizers/trijoint.py.151: total_norm: 13.558726 [I 2022-10-11 03:28:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 2290/2304 [I 2022-10-11 03:28:27] ...trap/engines/engine.py.228: elapsed: 0:33:23 | left: 0:00:11 [I 2022-10-11 03:28:27] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00301 [I 2022-10-11 03:28:27] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-11 03:28:27] ...optimizers/trijoint.py.151: total_norm: 15.011609 [I 2022-10-11 03:28:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 2300/2304 [I 2022-10-11 03:28:36] ...trap/engines/engine.py.228: elapsed: 0:33:32 | left: 0:00:03 [I 2022-10-11 03:28:36] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00016 [I 2022-10-11 03:28:36] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-11 03:28:36] ...optimizers/trijoint.py.151: total_norm: 11.059312 [I 2022-10-11 03:28:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 2304/2304 [I 2022-10-11 03:28:39] ...trap/engines/engine.py.228: elapsed: 0:33:35 | left: 0:00:00 [I 2022-10-11 03:28:39] ...trap/engines/engine.py.229: process: 0.82692 | load: 0.00016 [I 2022-10-11 03:28:39] ...trap/engines/engine.py.230: loss: 0.12706 [I 2022-10-11 03:28:39] ...optimizers/trijoint.py.151: total_norm: 11.240959 [S 2022-10-11 03:28:40] ...trap/engines/engine.py.129: train_epoch.epoch: 95 [S 2022-10-11 03:28:40] ...trap/engines/engine.py.129: train_epoch.loss: 0.12576329099769964 [I 2022-10-11 03:28:40] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 03:28:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 95 [I 2022-10-11 03:28:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 03:28:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 03:28:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 03:28:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 03:28:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 03:28:48] ...trap/engines/engine.py.321: eval: epoch 95 | batch 0/510 [I 2022-10-11 03:28:48] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:56 [I 2022-10-11 03:28:48] ...trap/engines/engine.py.326: process: 5.87597 | load: 3.46406 [I 2022-10-11 03:28:53] ...trap/engines/engine.py.321: eval: epoch 95 | batch 10/510 [I 2022-10-11 03:28:53] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:09:16 [I 2022-10-11 03:28:53] ...trap/engines/engine.py.326: process: 0.61279 | load: 0.00019 [I 2022-10-11 03:28:59] ...trap/engines/engine.py.321: eval: epoch 95 | batch 20/510 [I 2022-10-11 03:28:59] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:19 [I 2022-10-11 03:28:59] ...trap/engines/engine.py.326: process: 0.63678 | load: 0.00020 [I 2022-10-11 03:29:05] ...trap/engines/engine.py.321: eval: epoch 95 | batch 30/510 [I 2022-10-11 03:29:05] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:44 [I 2022-10-11 03:29:05] ...trap/engines/engine.py.326: process: 0.61381 | load: 0.00019 [I 2022-10-11 03:29:11] ...trap/engines/engine.py.321: eval: epoch 95 | batch 40/510 [I 2022-10-11 03:29:11] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:05:01 [I 2022-10-11 03:29:11] ...trap/engines/engine.py.326: process: 0.50120 | load: 0.00019 [I 2022-10-11 03:29:17] ...trap/engines/engine.py.321: eval: epoch 95 | batch 50/510 [I 2022-10-11 03:29:17] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:18 [I 2022-10-11 03:29:17] ...trap/engines/engine.py.326: process: 0.54854 | load: 0.00022 [I 2022-10-11 03:29:22] ...trap/engines/engine.py.321: eval: epoch 95 | batch 60/510 [I 2022-10-11 03:29:22] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:55 [I 2022-10-11 03:29:22] ...trap/engines/engine.py.326: process: 0.49857 | load: 0.00032 [I 2022-10-11 03:29:28] ...trap/engines/engine.py.321: eval: epoch 95 | batch 70/510 [I 2022-10-11 03:29:28] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:04:10 [I 2022-10-11 03:29:28] ...trap/engines/engine.py.326: process: 0.56842 | load: 0.00020 [I 2022-10-11 03:29:34] ...trap/engines/engine.py.321: eval: epoch 95 | batch 80/510 [I 2022-10-11 03:29:34] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:04:08 [I 2022-10-11 03:29:34] ...trap/engines/engine.py.326: process: 0.50621 | load: 0.00021 [I 2022-10-11 03:29:39] ...trap/engines/engine.py.321: eval: epoch 95 | batch 90/510 [I 2022-10-11 03:29:39] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:33 [I 2022-10-11 03:29:39] ...trap/engines/engine.py.326: process: 0.48668 | load: 0.00020 [I 2022-10-11 03:29:44] ...trap/engines/engine.py.321: eval: epoch 95 | batch 100/510 [I 2022-10-11 03:29:44] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:03:01 [I 2022-10-11 03:29:44] ...trap/engines/engine.py.326: process: 0.31968 | load: 0.00023 [I 2022-10-11 03:29:47] ...trap/engines/engine.py.321: eval: epoch 95 | batch 110/510 [I 2022-10-11 03:29:47] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:14 [I 2022-10-11 03:29:47] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00020 [I 2022-10-11 03:29:50] ...trap/engines/engine.py.321: eval: epoch 95 | batch 120/510 [I 2022-10-11 03:29:50] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:05 [I 2022-10-11 03:29:50] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00020 [I 2022-10-11 03:29:53] ...trap/engines/engine.py.321: eval: epoch 95 | batch 130/510 [I 2022-10-11 03:29:53] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:02 [I 2022-10-11 03:29:53] ...trap/engines/engine.py.326: process: 0.33574 | load: 0.00022 [I 2022-10-11 03:29:57] ...trap/engines/engine.py.321: eval: epoch 95 | batch 140/510 [I 2022-10-11 03:29:57] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:59 [I 2022-10-11 03:29:57] ...trap/engines/engine.py.326: process: 0.32104 | load: 0.00024 [I 2022-10-11 03:30:00] ...trap/engines/engine.py.321: eval: epoch 95 | batch 150/510 [I 2022-10-11 03:30:00] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:57 [I 2022-10-11 03:30:00] ...trap/engines/engine.py.326: process: 0.33763 | load: 0.00021 [I 2022-10-11 03:30:03] ...trap/engines/engine.py.321: eval: epoch 95 | batch 160/510 [I 2022-10-11 03:30:03] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:53 [I 2022-10-11 03:30:03] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00020 [I 2022-10-11 03:30:06] ...trap/engines/engine.py.321: eval: epoch 95 | batch 170/510 [I 2022-10-11 03:30:06] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:49 [I 2022-10-11 03:30:06] ...trap/engines/engine.py.326: process: 0.31730 | load: 0.00022 [I 2022-10-11 03:30:10] ...trap/engines/engine.py.321: eval: epoch 95 | batch 180/510 [I 2022-10-11 03:30:10] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:45 [I 2022-10-11 03:30:10] ...trap/engines/engine.py.326: process: 0.31446 | load: 0.00026 [I 2022-10-11 03:30:13] ...trap/engines/engine.py.321: eval: epoch 95 | batch 190/510 [I 2022-10-11 03:30:13] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:43 [I 2022-10-11 03:30:13] ...trap/engines/engine.py.326: process: 0.32952 | load: 0.00020 [I 2022-10-11 03:30:16] ...trap/engines/engine.py.321: eval: epoch 95 | batch 200/510 [I 2022-10-11 03:30:16] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:39 [I 2022-10-11 03:30:16] ...trap/engines/engine.py.326: process: 0.31703 | load: 0.00021 [I 2022-10-11 03:30:19] ...trap/engines/engine.py.321: eval: epoch 95 | batch 210/510 [I 2022-10-11 03:30:19] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:36 [I 2022-10-11 03:30:19] ...trap/engines/engine.py.326: process: 0.32144 | load: 0.00021 [I 2022-10-11 03:30:23] ...trap/engines/engine.py.321: eval: epoch 95 | batch 220/510 [I 2022-10-11 03:30:23] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:33 [I 2022-10-11 03:30:23] ...trap/engines/engine.py.326: process: 0.31499 | load: 0.00022 [I 2022-10-11 03:30:26] ...trap/engines/engine.py.321: eval: epoch 95 | batch 230/510 [I 2022-10-11 03:30:26] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:30 [I 2022-10-11 03:30:26] ...trap/engines/engine.py.326: process: 0.32810 | load: 0.00021 [I 2022-10-11 03:30:29] ...trap/engines/engine.py.321: eval: epoch 95 | batch 240/510 [I 2022-10-11 03:30:29] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:27 [I 2022-10-11 03:30:29] ...trap/engines/engine.py.326: process: 0.32490 | load: 0.00020 [I 2022-10-11 03:30:32] ...trap/engines/engine.py.321: eval: epoch 95 | batch 250/510 [I 2022-10-11 03:30:32] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:24 [I 2022-10-11 03:30:32] ...trap/engines/engine.py.326: process: 0.32680 | load: 0.00021 [I 2022-10-11 03:30:35] ...trap/engines/engine.py.321: eval: epoch 95 | batch 260/510 [I 2022-10-11 03:30:35] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:20 [I 2022-10-11 03:30:35] ...trap/engines/engine.py.326: process: 0.31694 | load: 0.00021 [I 2022-10-11 03:30:39] ...trap/engines/engine.py.321: eval: epoch 95 | batch 270/510 [I 2022-10-11 03:30:39] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:16 [I 2022-10-11 03:30:39] ...trap/engines/engine.py.326: process: 0.31819 | load: 0.00028 [I 2022-10-11 03:30:42] ...trap/engines/engine.py.321: eval: epoch 95 | batch 280/510 [I 2022-10-11 03:30:42] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:14 [I 2022-10-11 03:30:42] ...trap/engines/engine.py.326: process: 0.32319 | load: 0.00022 [I 2022-10-11 03:30:45] ...trap/engines/engine.py.321: eval: epoch 95 | batch 290/510 [I 2022-10-11 03:30:45] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:10 [I 2022-10-11 03:30:45] ...trap/engines/engine.py.326: process: 0.32719 | load: 0.00023 [I 2022-10-11 03:30:48] ...trap/engines/engine.py.321: eval: epoch 95 | batch 300/510 [I 2022-10-11 03:30:48] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:07 [I 2022-10-11 03:30:48] ...trap/engines/engine.py.326: process: 0.32565 | load: 0.00022 [I 2022-10-11 03:30:52] ...trap/engines/engine.py.321: eval: epoch 95 | batch 310/510 [I 2022-10-11 03:30:52] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:04 [I 2022-10-11 03:30:52] ...trap/engines/engine.py.326: process: 0.32056 | load: 0.00022 [I 2022-10-11 03:30:55] ...trap/engines/engine.py.321: eval: epoch 95 | batch 320/510 [I 2022-10-11 03:30:55] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:01 [I 2022-10-11 03:30:55] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00021 [I 2022-10-11 03:30:58] ...trap/engines/engine.py.321: eval: epoch 95 | batch 330/510 [I 2022-10-11 03:30:58] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:58 [I 2022-10-11 03:30:58] ...trap/engines/engine.py.326: process: 0.32641 | load: 0.00021 [I 2022-10-11 03:31:01] ...trap/engines/engine.py.321: eval: epoch 95 | batch 340/510 [I 2022-10-11 03:31:01] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:55 [I 2022-10-11 03:31:01] ...trap/engines/engine.py.326: process: 0.32322 | load: 0.00022 [I 2022-10-11 03:31:05] ...trap/engines/engine.py.321: eval: epoch 95 | batch 350/510 [I 2022-10-11 03:31:05] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:51 [I 2022-10-11 03:31:05] ...trap/engines/engine.py.326: process: 0.32602 | load: 0.00021 [I 2022-10-11 03:31:08] ...trap/engines/engine.py.321: eval: epoch 95 | batch 360/510 [I 2022-10-11 03:31:08] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:48 [I 2022-10-11 03:31:08] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00021 [I 2022-10-11 03:31:11] ...trap/engines/engine.py.321: eval: epoch 95 | batch 370/510 [I 2022-10-11 03:31:11] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:45 [I 2022-10-11 03:31:11] ...trap/engines/engine.py.326: process: 0.32666 | load: 0.00022 [I 2022-10-11 03:31:14] ...trap/engines/engine.py.321: eval: epoch 95 | batch 380/510 [I 2022-10-11 03:31:14] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:42 [I 2022-10-11 03:31:14] ...trap/engines/engine.py.326: process: 0.35133 | load: 0.00022 [I 2022-10-11 03:31:18] ...trap/engines/engine.py.321: eval: epoch 95 | batch 390/510 [I 2022-10-11 03:31:18] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:38 [I 2022-10-11 03:31:18] ...trap/engines/engine.py.326: process: 0.32731 | load: 0.00021 [I 2022-10-11 03:31:21] ...trap/engines/engine.py.321: eval: epoch 95 | batch 400/510 [I 2022-10-11 03:31:21] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:35 [I 2022-10-11 03:31:21] ...trap/engines/engine.py.326: process: 0.31520 | load: 0.00021 [I 2022-10-11 03:31:24] ...trap/engines/engine.py.321: eval: epoch 95 | batch 410/510 [I 2022-10-11 03:31:24] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:32 [I 2022-10-11 03:31:24] ...trap/engines/engine.py.326: process: 0.32101 | load: 0.00021 [I 2022-10-11 03:31:27] ...trap/engines/engine.py.321: eval: epoch 95 | batch 420/510 [I 2022-10-11 03:31:27] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:29 [I 2022-10-11 03:31:27] ...trap/engines/engine.py.326: process: 0.32367 | load: 0.00021 [I 2022-10-11 03:31:31] ...trap/engines/engine.py.321: eval: epoch 95 | batch 430/510 [I 2022-10-11 03:31:31] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:26 [I 2022-10-11 03:31:31] ...trap/engines/engine.py.326: process: 0.32450 | load: 0.00023 [I 2022-10-11 03:31:34] ...trap/engines/engine.py.321: eval: epoch 95 | batch 440/510 [I 2022-10-11 03:31:34] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:22 [I 2022-10-11 03:31:34] ...trap/engines/engine.py.326: process: 0.32552 | load: 0.00023 [I 2022-10-11 03:31:37] ...trap/engines/engine.py.321: eval: epoch 95 | batch 450/510 [I 2022-10-11 03:31:37] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:19 [I 2022-10-11 03:31:37] ...trap/engines/engine.py.326: process: 0.32215 | load: 0.00023 [I 2022-10-11 03:31:40] ...trap/engines/engine.py.321: eval: epoch 95 | batch 460/510 [I 2022-10-11 03:31:40] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:16 [I 2022-10-11 03:31:40] ...trap/engines/engine.py.326: process: 0.32121 | load: 0.00022 [I 2022-10-11 03:31:44] ...trap/engines/engine.py.321: eval: epoch 95 | batch 470/510 [I 2022-10-11 03:31:44] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:12 [I 2022-10-11 03:31:44] ...trap/engines/engine.py.326: process: 0.32007 | load: 0.00021 [I 2022-10-11 03:31:47] ...trap/engines/engine.py.321: eval: epoch 95 | batch 480/510 [I 2022-10-11 03:31:47] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:09 [I 2022-10-11 03:31:47] ...trap/engines/engine.py.326: process: 0.32399 | load: 0.00021 [I 2022-10-11 03:31:50] ...trap/engines/engine.py.321: eval: epoch 95 | batch 490/510 [I 2022-10-11 03:31:50] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:06 [I 2022-10-11 03:31:50] ...trap/engines/engine.py.326: process: 0.32597 | load: 0.00023 [I 2022-10-11 03:31:53] ...trap/engines/engine.py.321: eval: epoch 95 | batch 500/510 [I 2022-10-11 03:31:53] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:03 [I 2022-10-11 03:31:53] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00021 [I 2022-10-11 03:31:56] ...trap/engines/engine.py.321: eval: epoch 95 | batch 510/510 [I 2022-10-11 03:31:56] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:00 [I 2022-10-11 03:31:56] ...trap/engines/engine.py.326: process: 0.31740 | load: 0.00017 [S 2022-10-11 03:31:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 95 [I 2022-10-11 03:31:57] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.742200 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910700 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938500 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.738800 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910400 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937500 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009464 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011731 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010337 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013600 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012101 [S 2022-10-11 03:32:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010249 [I 2022-10-11 03:32:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 03:32:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 03:32:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 03:32:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 03:32:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 03:32:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 03:32:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 03:32:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 03:32:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 96 [I 2022-10-11 03:32:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 03:32:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 0/2304 [I 2022-10-11 03:32:59] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:10:18 [I 2022-10-11 03:32:59] ...trap/engines/engine.py.229: process: 3.39354 | load: 2.43623 [I 2022-10-11 03:32:59] ...trap/engines/engine.py.230: loss: 0.11284 [I 2022-10-11 03:32:59] ...optimizers/trijoint.py.151: total_norm: 12.581839 [I 2022-10-11 03:33:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 10/2304 [I 2022-10-11 03:33:08] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:43:14 [I 2022-10-11 03:33:08] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00555 [I 2022-10-11 03:33:08] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-11 03:33:08] ...optimizers/trijoint.py.151: total_norm: 12.791915 [I 2022-10-11 03:33:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 20/2304 [I 2022-10-11 03:33:16] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:47 [I 2022-10-11 03:33:16] ...trap/engines/engine.py.229: process: 0.84976 | load: 0.00506 [I 2022-10-11 03:33:16] ...trap/engines/engine.py.230: loss: 0.16741 [I 2022-10-11 03:33:16] ...optimizers/trijoint.py.151: total_norm: 23.832647 [I 2022-10-11 03:33:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 30/2304 [I 2022-10-11 03:33:25] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:30 [I 2022-10-11 03:33:25] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00452 [I 2022-10-11 03:33:25] ...trap/engines/engine.py.230: loss: 0.09495 [I 2022-10-11 03:33:25] ...optimizers/trijoint.py.151: total_norm: 10.237630 [I 2022-10-11 03:33:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 40/2304 [I 2022-10-11 03:33:33] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:22 [I 2022-10-11 03:33:33] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00388 [I 2022-10-11 03:33:33] ...trap/engines/engine.py.230: loss: 0.24728 [I 2022-10-11 03:33:33] ...optimizers/trijoint.py.151: total_norm: 53.194427 [I 2022-10-11 03:33:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 50/2304 [I 2022-10-11 03:33:42] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:08 [I 2022-10-11 03:33:42] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00479 [I 2022-10-11 03:33:42] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-11 03:33:42] ...optimizers/trijoint.py.151: total_norm: 16.842953 [I 2022-10-11 03:33:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 60/2304 [I 2022-10-11 03:33:51] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:21 [I 2022-10-11 03:33:51] ...trap/engines/engine.py.229: process: 0.87592 | load: 0.00401 [I 2022-10-11 03:33:51] ...trap/engines/engine.py.230: loss: 0.13053 [I 2022-10-11 03:33:51] ...optimizers/trijoint.py.151: total_norm: 9.769059 [I 2022-10-11 03:33:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 70/2304 [I 2022-10-11 03:33:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:59 [I 2022-10-11 03:33:59] ...trap/engines/engine.py.229: process: 0.85723 | load: 0.00352 [I 2022-10-11 03:33:59] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-11 03:33:59] ...optimizers/trijoint.py.151: total_norm: 17.640055 [I 2022-10-11 03:34:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 80/2304 [I 2022-10-11 03:34:08] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:50 [I 2022-10-11 03:34:08] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00339 [I 2022-10-11 03:34:08] ...trap/engines/engine.py.230: loss: 0.13443 [I 2022-10-11 03:34:08] ...optimizers/trijoint.py.151: total_norm: 13.939614 [I 2022-10-11 03:34:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 90/2304 [I 2022-10-11 03:34:16] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:16 [I 2022-10-11 03:34:16] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00320 [I 2022-10-11 03:34:16] ...trap/engines/engine.py.230: loss: 0.07240 [I 2022-10-11 03:34:16] ...optimizers/trijoint.py.151: total_norm: 14.002786 [I 2022-10-11 03:34:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 100/2304 [I 2022-10-11 03:34:25] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:22 [I 2022-10-11 03:34:25] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00331 [I 2022-10-11 03:34:25] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-11 03:34:25] ...optimizers/trijoint.py.151: total_norm: 21.305975 [I 2022-10-11 03:34:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 110/2304 [I 2022-10-11 03:34:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:15 [I 2022-10-11 03:34:34] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00376 [I 2022-10-11 03:34:34] ...trap/engines/engine.py.230: loss: 0.11455 [I 2022-10-11 03:34:34] ...optimizers/trijoint.py.151: total_norm: 8.566648 [I 2022-10-11 03:34:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 120/2304 [I 2022-10-11 03:34:42] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:07 [I 2022-10-11 03:34:42] ...trap/engines/engine.py.229: process: 0.87674 | load: 0.00351 [I 2022-10-11 03:34:42] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-11 03:34:42] ...optimizers/trijoint.py.151: total_norm: 17.792137 [I 2022-10-11 03:34:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 130/2304 [I 2022-10-11 03:34:51] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:30:58 [I 2022-10-11 03:34:51] ...trap/engines/engine.py.229: process: 0.85636 | load: 0.00294 [I 2022-10-11 03:34:51] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-11 03:34:51] ...optimizers/trijoint.py.151: total_norm: 15.197773 [I 2022-10-11 03:34:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 140/2304 [I 2022-10-11 03:34:59] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:45 [I 2022-10-11 03:34:59] ...trap/engines/engine.py.229: process: 0.86442 | load: 0.00305 [I 2022-10-11 03:34:59] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-11 03:34:59] ...optimizers/trijoint.py.151: total_norm: 36.680050 [I 2022-10-11 03:35:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 150/2304 [I 2022-10-11 03:35:08] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:48 [I 2022-10-11 03:35:08] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00338 [I 2022-10-11 03:35:08] ...trap/engines/engine.py.230: loss: 0.08588 [I 2022-10-11 03:35:08] ...optimizers/trijoint.py.151: total_norm: 10.076150 [I 2022-10-11 03:35:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 160/2304 [I 2022-10-11 03:35:17] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:55 [I 2022-10-11 03:35:17] ...trap/engines/engine.py.229: process: 0.86615 | load: 0.00346 [I 2022-10-11 03:35:17] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-11 03:35:17] ...optimizers/trijoint.py.151: total_norm: 15.615681 [I 2022-10-11 03:35:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 170/2304 [I 2022-10-11 03:35:25] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:45 [I 2022-10-11 03:35:25] ...trap/engines/engine.py.229: process: 0.86202 | load: 0.00318 [I 2022-10-11 03:35:25] ...trap/engines/engine.py.230: loss: 0.07717 [I 2022-10-11 03:35:25] ...optimizers/trijoint.py.151: total_norm: 11.622593 [I 2022-10-11 03:35:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 180/2304 [I 2022-10-11 03:35:34] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:34 [I 2022-10-11 03:35:34] ...trap/engines/engine.py.229: process: 0.87138 | load: 0.00336 [I 2022-10-11 03:35:34] ...trap/engines/engine.py.230: loss: 0.18849 [I 2022-10-11 03:35:34] ...optimizers/trijoint.py.151: total_norm: 21.479036 [I 2022-10-11 03:35:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 190/2304 [I 2022-10-11 03:35:43] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:32 [I 2022-10-11 03:35:43] ...trap/engines/engine.py.229: process: 0.85770 | load: 0.00348 [I 2022-10-11 03:35:43] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-11 03:35:43] ...optimizers/trijoint.py.151: total_norm: 19.307924 [I 2022-10-11 03:35:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 200/2304 [I 2022-10-11 03:35:51] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:16 [I 2022-10-11 03:35:51] ...trap/engines/engine.py.229: process: 0.86707 | load: 0.00309 [I 2022-10-11 03:35:51] ...trap/engines/engine.py.230: loss: 0.13242 [I 2022-10-11 03:35:51] ...optimizers/trijoint.py.151: total_norm: 16.416830 [I 2022-10-11 03:36:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 210/2304 [I 2022-10-11 03:36:00] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:30:04 [I 2022-10-11 03:36:00] ...trap/engines/engine.py.229: process: 0.86583 | load: 0.00360 [I 2022-10-11 03:36:00] ...trap/engines/engine.py.230: loss: 0.09429 [I 2022-10-11 03:36:00] ...optimizers/trijoint.py.151: total_norm: 16.173237 [I 2022-10-11 03:36:09] ...trap/engines/engine.py.224: train: epoch 96 | batch 220/2304 [I 2022-10-11 03:36:09] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:30:09 [I 2022-10-11 03:36:09] ...trap/engines/engine.py.229: process: 0.88095 | load: 0.00308 [I 2022-10-11 03:36:09] ...trap/engines/engine.py.230: loss: 0.09128 [I 2022-10-11 03:36:09] ...optimizers/trijoint.py.151: total_norm: 12.924100 [I 2022-10-11 03:36:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 230/2304 [I 2022-10-11 03:36:17] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:29:56 [I 2022-10-11 03:36:17] ...trap/engines/engine.py.229: process: 0.86858 | load: 0.00326 [I 2022-10-11 03:36:17] ...trap/engines/engine.py.230: loss: 0.11428 [I 2022-10-11 03:36:17] ...optimizers/trijoint.py.151: total_norm: 18.813250 [I 2022-10-11 03:36:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 240/2304 [I 2022-10-11 03:36:26] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:32 [I 2022-10-11 03:36:26] ...trap/engines/engine.py.229: process: 0.86667 | load: 0.00340 [I 2022-10-11 03:36:26] ...trap/engines/engine.py.230: loss: 0.09378 [I 2022-10-11 03:36:26] ...optimizers/trijoint.py.151: total_norm: 19.911753 [I 2022-10-11 03:36:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 250/2304 [I 2022-10-11 03:36:35] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:37 [I 2022-10-11 03:36:35] ...trap/engines/engine.py.229: process: 0.87095 | load: 0.00307 [I 2022-10-11 03:36:35] ...trap/engines/engine.py.230: loss: 0.09839 [I 2022-10-11 03:36:35] ...optimizers/trijoint.py.151: total_norm: 9.799250 [I 2022-10-11 03:36:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 260/2304 [I 2022-10-11 03:36:43] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:36 [I 2022-10-11 03:36:43] ...trap/engines/engine.py.229: process: 0.86891 | load: 0.00309 [I 2022-10-11 03:36:43] ...trap/engines/engine.py.230: loss: 0.12760 [I 2022-10-11 03:36:43] ...optimizers/trijoint.py.151: total_norm: 10.182199 [I 2022-10-11 03:36:52] ...trap/engines/engine.py.224: train: epoch 96 | batch 270/2304 [I 2022-10-11 03:36:52] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:19 [I 2022-10-11 03:36:52] ...trap/engines/engine.py.229: process: 0.86918 | load: 0.00326 [I 2022-10-11 03:36:52] ...trap/engines/engine.py.230: loss: 0.09862 [I 2022-10-11 03:36:52] ...optimizers/trijoint.py.151: total_norm: 12.385447 [I 2022-10-11 03:37:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 280/2304 [I 2022-10-11 03:37:01] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:29:15 [I 2022-10-11 03:37:01] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00319 [I 2022-10-11 03:37:01] ...trap/engines/engine.py.230: loss: 0.08417 [I 2022-10-11 03:37:01] ...optimizers/trijoint.py.151: total_norm: 12.916615 [I 2022-10-11 03:37:09] ...trap/engines/engine.py.224: train: epoch 96 | batch 290/2304 [I 2022-10-11 03:37:09] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:29:08 [I 2022-10-11 03:37:09] ...trap/engines/engine.py.229: process: 0.87415 | load: 0.00325 [I 2022-10-11 03:37:09] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-11 03:37:09] ...optimizers/trijoint.py.151: total_norm: 14.615457 [I 2022-10-11 03:37:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 300/2304 [I 2022-10-11 03:37:18] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:58 [I 2022-10-11 03:37:18] ...trap/engines/engine.py.229: process: 0.86270 | load: 0.00308 [I 2022-10-11 03:37:18] ...trap/engines/engine.py.230: loss: 0.10209 [I 2022-10-11 03:37:18] ...optimizers/trijoint.py.151: total_norm: 27.006308 [I 2022-10-11 03:37:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 310/2304 [I 2022-10-11 03:37:27] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:50 [I 2022-10-11 03:37:27] ...trap/engines/engine.py.229: process: 0.86676 | load: 0.00321 [I 2022-10-11 03:37:27] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-11 03:37:27] ...optimizers/trijoint.py.151: total_norm: 12.488550 [I 2022-10-11 03:37:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 320/2304 [I 2022-10-11 03:37:35] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:45 [I 2022-10-11 03:37:35] ...trap/engines/engine.py.229: process: 0.88246 | load: 0.00361 [I 2022-10-11 03:37:35] ...trap/engines/engine.py.230: loss: 0.09255 [I 2022-10-11 03:37:35] ...optimizers/trijoint.py.151: total_norm: 11.219332 [I 2022-10-11 03:37:44] ...trap/engines/engine.py.224: train: epoch 96 | batch 330/2304 [I 2022-10-11 03:37:44] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:32 [I 2022-10-11 03:37:44] ...trap/engines/engine.py.229: process: 0.87253 | load: 0.00682 [I 2022-10-11 03:37:44] ...trap/engines/engine.py.230: loss: 0.11459 [I 2022-10-11 03:37:44] ...optimizers/trijoint.py.151: total_norm: 10.304244 [I 2022-10-11 03:37:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 340/2304 [I 2022-10-11 03:37:53] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:27 [I 2022-10-11 03:37:53] ...trap/engines/engine.py.229: process: 0.86385 | load: 0.00321 [I 2022-10-11 03:37:53] ...trap/engines/engine.py.230: loss: 0.13016 [I 2022-10-11 03:37:53] ...optimizers/trijoint.py.151: total_norm: 20.168245 [I 2022-10-11 03:38:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 350/2304 [I 2022-10-11 03:38:01] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:07 [I 2022-10-11 03:38:01] ...trap/engines/engine.py.229: process: 0.85774 | load: 0.00311 [I 2022-10-11 03:38:01] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-11 03:38:01] ...optimizers/trijoint.py.151: total_norm: 26.806833 [I 2022-10-11 03:38:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 360/2304 [I 2022-10-11 03:38:10] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:28:03 [I 2022-10-11 03:38:10] ...trap/engines/engine.py.229: process: 0.86798 | load: 0.00312 [I 2022-10-11 03:38:10] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-11 03:38:10] ...optimizers/trijoint.py.151: total_norm: 11.300830 [I 2022-10-11 03:38:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 370/2304 [I 2022-10-11 03:38:19] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:45 [I 2022-10-11 03:38:19] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00310 [I 2022-10-11 03:38:19] ...trap/engines/engine.py.230: loss: 0.08617 [I 2022-10-11 03:38:19] ...optimizers/trijoint.py.151: total_norm: 9.917903 [I 2022-10-11 03:38:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 380/2304 [I 2022-10-11 03:38:27] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:40 [I 2022-10-11 03:38:27] ...trap/engines/engine.py.229: process: 0.85670 | load: 0.00318 [I 2022-10-11 03:38:27] ...trap/engines/engine.py.230: loss: 0.09622 [I 2022-10-11 03:38:27] ...optimizers/trijoint.py.151: total_norm: 11.722397 [I 2022-10-11 03:38:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 390/2304 [I 2022-10-11 03:38:36] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:34 [I 2022-10-11 03:38:36] ...trap/engines/engine.py.229: process: 0.86535 | load: 0.00319 [I 2022-10-11 03:38:36] ...trap/engines/engine.py.230: loss: 0.11123 [I 2022-10-11 03:38:36] ...optimizers/trijoint.py.151: total_norm: 10.055507 [I 2022-10-11 03:38:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 400/2304 [I 2022-10-11 03:38:45] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:32 [I 2022-10-11 03:38:45] ...trap/engines/engine.py.229: process: 0.86947 | load: 0.00312 [I 2022-10-11 03:38:45] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-11 03:38:45] ...optimizers/trijoint.py.151: total_norm: 15.272271 [I 2022-10-11 03:38:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 410/2304 [I 2022-10-11 03:38:53] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:15 [I 2022-10-11 03:38:53] ...trap/engines/engine.py.229: process: 0.87599 | load: 0.00303 [I 2022-10-11 03:38:53] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-11 03:38:53] ...optimizers/trijoint.py.151: total_norm: 18.745771 [I 2022-10-11 03:39:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 420/2304 [I 2022-10-11 03:39:02] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:50 [I 2022-10-11 03:39:02] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00296 [I 2022-10-11 03:39:02] ...trap/engines/engine.py.230: loss: 0.10055 [I 2022-10-11 03:39:02] ...optimizers/trijoint.py.151: total_norm: 18.804056 [I 2022-10-11 03:39:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 430/2304 [I 2022-10-11 03:39:11] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:45 [I 2022-10-11 03:39:11] ...trap/engines/engine.py.229: process: 0.85816 | load: 0.00315 [I 2022-10-11 03:39:11] ...trap/engines/engine.py.230: loss: 0.13850 [I 2022-10-11 03:39:11] ...optimizers/trijoint.py.151: total_norm: 25.914518 [I 2022-10-11 03:39:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 440/2304 [I 2022-10-11 03:39:19] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:39 [I 2022-10-11 03:39:19] ...trap/engines/engine.py.229: process: 0.86156 | load: 0.00318 [I 2022-10-11 03:39:19] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-11 03:39:19] ...optimizers/trijoint.py.151: total_norm: 18.576479 [I 2022-10-11 03:39:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 450/2304 [I 2022-10-11 03:39:28] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:32 [I 2022-10-11 03:39:28] ...trap/engines/engine.py.229: process: 0.85441 | load: 0.00296 [I 2022-10-11 03:39:28] ...trap/engines/engine.py.230: loss: 0.17377 [I 2022-10-11 03:39:28] ...optimizers/trijoint.py.151: total_norm: 31.715618 [I 2022-10-11 03:39:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 460/2304 [I 2022-10-11 03:39:36] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:13 [I 2022-10-11 03:39:36] ...trap/engines/engine.py.229: process: 0.85108 | load: 0.00303 [I 2022-10-11 03:39:36] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-11 03:39:36] ...optimizers/trijoint.py.151: total_norm: 12.481974 [I 2022-10-11 03:39:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 470/2304 [I 2022-10-11 03:39:45] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:08 [I 2022-10-11 03:39:45] ...trap/engines/engine.py.229: process: 0.85885 | load: 0.00305 [I 2022-10-11 03:39:45] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-11 03:39:45] ...optimizers/trijoint.py.151: total_norm: 14.903150 [I 2022-10-11 03:39:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 480/2304 [I 2022-10-11 03:39:53] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:56 [I 2022-10-11 03:39:53] ...trap/engines/engine.py.229: process: 0.85556 | load: 0.00300 [I 2022-10-11 03:39:53] ...trap/engines/engine.py.230: loss: 0.08472 [I 2022-10-11 03:39:53] ...optimizers/trijoint.py.151: total_norm: 9.212150 [I 2022-10-11 03:40:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 490/2304 [I 2022-10-11 03:40:02] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:44 [I 2022-10-11 03:40:02] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00296 [I 2022-10-11 03:40:02] ...trap/engines/engine.py.230: loss: 0.15566 [I 2022-10-11 03:40:02] ...optimizers/trijoint.py.151: total_norm: 17.240242 [I 2022-10-11 03:40:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 500/2304 [I 2022-10-11 03:40:11] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:50 [I 2022-10-11 03:40:11] ...trap/engines/engine.py.229: process: 0.87496 | load: 0.00297 [I 2022-10-11 03:40:11] ...trap/engines/engine.py.230: loss: 0.13677 [I 2022-10-11 03:40:11] ...optimizers/trijoint.py.151: total_norm: 19.970274 [I 2022-10-11 03:40:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 510/2304 [I 2022-10-11 03:40:19] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:38 [I 2022-10-11 03:40:19] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00297 [I 2022-10-11 03:40:19] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-11 03:40:19] ...optimizers/trijoint.py.151: total_norm: 14.199302 [I 2022-10-11 03:40:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 520/2304 [I 2022-10-11 03:40:28] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:20 [I 2022-10-11 03:40:28] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00300 [I 2022-10-11 03:40:28] ...trap/engines/engine.py.230: loss: 0.12924 [I 2022-10-11 03:40:28] ...optimizers/trijoint.py.151: total_norm: 16.683998 [I 2022-10-11 03:40:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 530/2304 [I 2022-10-11 03:40:36] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:16 [I 2022-10-11 03:40:36] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00296 [I 2022-10-11 03:40:36] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-11 03:40:36] ...optimizers/trijoint.py.151: total_norm: 11.118696 [I 2022-10-11 03:40:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 540/2304 [I 2022-10-11 03:40:45] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:24 [I 2022-10-11 03:40:45] ...trap/engines/engine.py.229: process: 0.87172 | load: 0.00311 [I 2022-10-11 03:40:45] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-11 03:40:45] ...optimizers/trijoint.py.151: total_norm: 34.292675 [I 2022-10-11 03:40:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 550/2304 [I 2022-10-11 03:40:54] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:25 [I 2022-10-11 03:40:54] ...trap/engines/engine.py.229: process: 0.86985 | load: 0.00308 [I 2022-10-11 03:40:54] ...trap/engines/engine.py.230: loss: 0.07459 [I 2022-10-11 03:40:54] ...optimizers/trijoint.py.151: total_norm: 11.932981 [I 2022-10-11 03:41:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 560/2304 [I 2022-10-11 03:41:02] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:09 [I 2022-10-11 03:41:02] ...trap/engines/engine.py.229: process: 0.87575 | load: 0.00314 [I 2022-10-11 03:41:02] ...trap/engines/engine.py.230: loss: 0.09127 [I 2022-10-11 03:41:02] ...optimizers/trijoint.py.151: total_norm: 13.953726 [I 2022-10-11 03:41:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 570/2304 [I 2022-10-11 03:41:11] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:25:00 [I 2022-10-11 03:41:11] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00311 [I 2022-10-11 03:41:11] ...trap/engines/engine.py.230: loss: 0.10062 [I 2022-10-11 03:41:11] ...optimizers/trijoint.py.151: total_norm: 11.043639 [I 2022-10-11 03:41:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 580/2304 [I 2022-10-11 03:41:20] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:47 [I 2022-10-11 03:41:20] ...trap/engines/engine.py.229: process: 0.86088 | load: 0.00327 [I 2022-10-11 03:41:20] ...trap/engines/engine.py.230: loss: 0.16496 [I 2022-10-11 03:41:20] ...optimizers/trijoint.py.151: total_norm: 16.754883 [I 2022-10-11 03:41:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 590/2304 [I 2022-10-11 03:41:28] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:30 [I 2022-10-11 03:41:28] ...trap/engines/engine.py.229: process: 0.85546 | load: 0.00319 [I 2022-10-11 03:41:28] ...trap/engines/engine.py.230: loss: 0.07914 [I 2022-10-11 03:41:28] ...optimizers/trijoint.py.151: total_norm: 14.590258 [I 2022-10-11 03:41:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 600/2304 [I 2022-10-11 03:41:37] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:44 [I 2022-10-11 03:41:37] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00309 [I 2022-10-11 03:41:37] ...trap/engines/engine.py.230: loss: 0.10540 [I 2022-10-11 03:41:37] ...optimizers/trijoint.py.151: total_norm: 10.150832 [I 2022-10-11 03:41:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 610/2304 [I 2022-10-11 03:41:46] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:30 [I 2022-10-11 03:41:46] ...trap/engines/engine.py.229: process: 0.86757 | load: 0.00310 [I 2022-10-11 03:41:46] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-11 03:41:46] ...optimizers/trijoint.py.151: total_norm: 40.372185 [I 2022-10-11 03:41:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 620/2304 [I 2022-10-11 03:41:55] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:26 [I 2022-10-11 03:41:55] ...trap/engines/engine.py.229: process: 0.87836 | load: 0.00311 [I 2022-10-11 03:41:55] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-11 03:41:55] ...optimizers/trijoint.py.151: total_norm: 13.655623 [I 2022-10-11 03:42:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 630/2304 [I 2022-10-11 03:42:03] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:24:10 [I 2022-10-11 03:42:03] ...trap/engines/engine.py.229: process: 0.87281 | load: 0.00308 [I 2022-10-11 03:42:03] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-11 03:42:03] ...optimizers/trijoint.py.151: total_norm: 15.040932 [I 2022-10-11 03:42:12] ...trap/engines/engine.py.224: train: epoch 96 | batch 640/2304 [I 2022-10-11 03:42:12] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:24:00 [I 2022-10-11 03:42:12] ...trap/engines/engine.py.229: process: 0.86342 | load: 0.00313 [I 2022-10-11 03:42:12] ...trap/engines/engine.py.230: loss: 0.08605 [I 2022-10-11 03:42:12] ...optimizers/trijoint.py.151: total_norm: 11.831803 [I 2022-10-11 03:42:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 650/2304 [I 2022-10-11 03:42:21] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:53 [I 2022-10-11 03:42:21] ...trap/engines/engine.py.229: process: 0.86296 | load: 0.00308 [I 2022-10-11 03:42:21] ...trap/engines/engine.py.230: loss: 0.18722 [I 2022-10-11 03:42:21] ...optimizers/trijoint.py.151: total_norm: 10.087046 [I 2022-10-11 03:42:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 660/2304 [I 2022-10-11 03:42:29] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:38 [I 2022-10-11 03:42:29] ...trap/engines/engine.py.229: process: 0.86657 | load: 0.00306 [I 2022-10-11 03:42:29] ...trap/engines/engine.py.230: loss: 0.18001 [I 2022-10-11 03:42:29] ...optimizers/trijoint.py.151: total_norm: 13.155516 [I 2022-10-11 03:42:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 670/2304 [I 2022-10-11 03:42:38] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:31 [I 2022-10-11 03:42:38] ...trap/engines/engine.py.229: process: 0.86611 | load: 0.00311 [I 2022-10-11 03:42:38] ...trap/engines/engine.py.230: loss: 0.10824 [I 2022-10-11 03:42:38] ...optimizers/trijoint.py.151: total_norm: 12.287004 [I 2022-10-11 03:42:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 680/2304 [I 2022-10-11 03:42:47] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:20 [I 2022-10-11 03:42:47] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00307 [I 2022-10-11 03:42:47] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-11 03:42:47] ...optimizers/trijoint.py.151: total_norm: 21.812466 [I 2022-10-11 03:42:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 690/2304 [I 2022-10-11 03:42:55] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:23:11 [I 2022-10-11 03:42:55] ...trap/engines/engine.py.229: process: 0.86242 | load: 0.00319 [I 2022-10-11 03:42:55] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-11 03:42:55] ...optimizers/trijoint.py.151: total_norm: 7.322369 [I 2022-10-11 03:43:04] ...trap/engines/engine.py.224: train: epoch 96 | batch 700/2304 [I 2022-10-11 03:43:04] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:23:16 [I 2022-10-11 03:43:04] ...trap/engines/engine.py.229: process: 0.87016 | load: 0.00485 [I 2022-10-11 03:43:04] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-11 03:43:04] ...optimizers/trijoint.py.151: total_norm: 18.278553 [I 2022-10-11 03:43:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 710/2304 [I 2022-10-11 03:43:13] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:23:01 [I 2022-10-11 03:43:13] ...trap/engines/engine.py.229: process: 0.87643 | load: 0.00309 [I 2022-10-11 03:43:13] ...trap/engines/engine.py.230: loss: 0.14066 [I 2022-10-11 03:43:13] ...optimizers/trijoint.py.151: total_norm: 20.466564 [I 2022-10-11 03:43:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 720/2304 [I 2022-10-11 03:43:21] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:48 [I 2022-10-11 03:43:21] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00306 [I 2022-10-11 03:43:21] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-11 03:43:21] ...optimizers/trijoint.py.151: total_norm: 20.331236 [I 2022-10-11 03:43:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 730/2304 [I 2022-10-11 03:43:30] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:42 [I 2022-10-11 03:43:30] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00320 [I 2022-10-11 03:43:30] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-11 03:43:30] ...optimizers/trijoint.py.151: total_norm: 17.516386 [I 2022-10-11 03:43:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 740/2304 [I 2022-10-11 03:43:39] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:42 [I 2022-10-11 03:43:39] ...trap/engines/engine.py.229: process: 0.87497 | load: 0.00333 [I 2022-10-11 03:43:39] ...trap/engines/engine.py.230: loss: 0.10659 [I 2022-10-11 03:43:39] ...optimizers/trijoint.py.151: total_norm: 11.057740 [I 2022-10-11 03:43:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 750/2304 [I 2022-10-11 03:43:47] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:32 [I 2022-10-11 03:43:47] ...trap/engines/engine.py.229: process: 0.87711 | load: 0.00326 [I 2022-10-11 03:43:47] ...trap/engines/engine.py.230: loss: 0.16358 [I 2022-10-11 03:43:47] ...optimizers/trijoint.py.151: total_norm: 14.123035 [I 2022-10-11 03:43:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 760/2304 [I 2022-10-11 03:43:56] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:22:21 [I 2022-10-11 03:43:56] ...trap/engines/engine.py.229: process: 0.87171 | load: 0.00327 [I 2022-10-11 03:43:56] ...trap/engines/engine.py.230: loss: 0.15301 [I 2022-10-11 03:43:56] ...optimizers/trijoint.py.151: total_norm: 13.297328 [I 2022-10-11 03:44:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 770/2304 [I 2022-10-11 03:44:05] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:22:13 [I 2022-10-11 03:44:05] ...trap/engines/engine.py.229: process: 0.86878 | load: 0.00319 [I 2022-10-11 03:44:05] ...trap/engines/engine.py.230: loss: 0.10224 [I 2022-10-11 03:44:05] ...optimizers/trijoint.py.151: total_norm: 8.932048 [I 2022-10-11 03:44:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 780/2304 [I 2022-10-11 03:44:13] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:56 [I 2022-10-11 03:44:13] ...trap/engines/engine.py.229: process: 0.84981 | load: 0.00307 [I 2022-10-11 03:44:13] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-11 03:44:13] ...optimizers/trijoint.py.151: total_norm: 15.982647 [I 2022-10-11 03:44:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 790/2304 [I 2022-10-11 03:44:22] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:40 [I 2022-10-11 03:44:22] ...trap/engines/engine.py.229: process: 0.85673 | load: 0.00308 [I 2022-10-11 03:44:22] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-11 03:44:22] ...optimizers/trijoint.py.151: total_norm: 16.265839 [I 2022-10-11 03:44:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 800/2304 [I 2022-10-11 03:44:31] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:38 [I 2022-10-11 03:44:31] ...trap/engines/engine.py.229: process: 0.87050 | load: 0.00307 [I 2022-10-11 03:44:31] ...trap/engines/engine.py.230: loss: 0.11694 [I 2022-10-11 03:44:31] ...optimizers/trijoint.py.151: total_norm: 10.240693 [I 2022-10-11 03:44:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 810/2304 [I 2022-10-11 03:44:39] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:23 [I 2022-10-11 03:44:39] ...trap/engines/engine.py.229: process: 0.87613 | load: 0.00299 [I 2022-10-11 03:44:39] ...trap/engines/engine.py.230: loss: 0.17387 [I 2022-10-11 03:44:39] ...optimizers/trijoint.py.151: total_norm: 18.405491 [I 2022-10-11 03:44:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 820/2304 [I 2022-10-11 03:44:48] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:21:20 [I 2022-10-11 03:44:48] ...trap/engines/engine.py.229: process: 0.86583 | load: 0.00302 [I 2022-10-11 03:44:48] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-11 03:44:48] ...optimizers/trijoint.py.151: total_norm: 9.561088 [I 2022-10-11 03:44:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 830/2304 [I 2022-10-11 03:44:57] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:00 [I 2022-10-11 03:44:57] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00297 [I 2022-10-11 03:44:57] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-11 03:44:57] ...optimizers/trijoint.py.151: total_norm: 13.010592 [I 2022-10-11 03:45:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 840/2304 [I 2022-10-11 03:45:05] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:42 [I 2022-10-11 03:45:05] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00303 [I 2022-10-11 03:45:05] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-11 03:45:05] ...optimizers/trijoint.py.151: total_norm: 16.514881 [I 2022-10-11 03:45:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 850/2304 [I 2022-10-11 03:45:14] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:41 [I 2022-10-11 03:45:14] ...trap/engines/engine.py.229: process: 0.85511 | load: 0.00299 [I 2022-10-11 03:45:14] ...trap/engines/engine.py.230: loss: 0.09827 [I 2022-10-11 03:45:14] ...optimizers/trijoint.py.151: total_norm: 12.391267 [I 2022-10-11 03:45:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 860/2304 [I 2022-10-11 03:45:22] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:28 [I 2022-10-11 03:45:22] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00305 [I 2022-10-11 03:45:22] ...trap/engines/engine.py.230: loss: 0.07080 [I 2022-10-11 03:45:22] ...optimizers/trijoint.py.151: total_norm: 9.954037 [I 2022-10-11 03:45:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 870/2304 [I 2022-10-11 03:45:31] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:27 [I 2022-10-11 03:45:31] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00296 [I 2022-10-11 03:45:31] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-11 03:45:31] ...optimizers/trijoint.py.151: total_norm: 27.796803 [I 2022-10-11 03:45:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 880/2304 [I 2022-10-11 03:45:39] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:04 [I 2022-10-11 03:45:39] ...trap/engines/engine.py.229: process: 0.83825 | load: 0.00298 [I 2022-10-11 03:45:39] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-11 03:45:39] ...optimizers/trijoint.py.151: total_norm: 15.251263 [I 2022-10-11 03:45:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 890/2304 [I 2022-10-11 03:45:48] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:14 [I 2022-10-11 03:45:48] ...trap/engines/engine.py.229: process: 0.87007 | load: 0.00306 [I 2022-10-11 03:45:48] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-11 03:45:48] ...optimizers/trijoint.py.151: total_norm: 23.892673 [I 2022-10-11 03:45:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 900/2304 [I 2022-10-11 03:45:56] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:00 [I 2022-10-11 03:45:56] ...trap/engines/engine.py.229: process: 0.84617 | load: 0.00314 [I 2022-10-11 03:45:56] ...trap/engines/engine.py.230: loss: 0.08665 [I 2022-10-11 03:45:56] ...optimizers/trijoint.py.151: total_norm: 14.177242 [I 2022-10-11 03:46:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 910/2304 [I 2022-10-11 03:46:05] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:55 [I 2022-10-11 03:46:05] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00302 [I 2022-10-11 03:46:05] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 03:46:05] ...optimizers/trijoint.py.151: total_norm: 9.898303 [I 2022-10-11 03:46:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 920/2304 [I 2022-10-11 03:46:14] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:50 [I 2022-10-11 03:46:14] ...trap/engines/engine.py.229: process: 0.87801 | load: 0.00303 [I 2022-10-11 03:46:14] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-11 03:46:14] ...optimizers/trijoint.py.151: total_norm: 10.674609 [I 2022-10-11 03:46:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 930/2304 [I 2022-10-11 03:46:22] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:49 [I 2022-10-11 03:46:22] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00311 [I 2022-10-11 03:46:22] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-11 03:46:22] ...optimizers/trijoint.py.151: total_norm: 16.898653 [I 2022-10-11 03:46:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 940/2304 [I 2022-10-11 03:46:31] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:39 [I 2022-10-11 03:46:31] ...trap/engines/engine.py.229: process: 0.87560 | load: 0.00322 [I 2022-10-11 03:46:31] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-11 03:46:31] ...optimizers/trijoint.py.151: total_norm: 26.248325 [I 2022-10-11 03:46:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 950/2304 [I 2022-10-11 03:46:40] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:34 [I 2022-10-11 03:46:40] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00312 [I 2022-10-11 03:46:40] ...trap/engines/engine.py.230: loss: 0.09859 [I 2022-10-11 03:46:40] ...optimizers/trijoint.py.151: total_norm: 14.957047 [I 2022-10-11 03:46:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 960/2304 [I 2022-10-11 03:46:48] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:21 [I 2022-10-11 03:46:48] ...trap/engines/engine.py.229: process: 0.87771 | load: 0.00317 [I 2022-10-11 03:46:48] ...trap/engines/engine.py.230: loss: 0.08171 [I 2022-10-11 03:46:48] ...optimizers/trijoint.py.151: total_norm: 19.023138 [I 2022-10-11 03:46:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 970/2304 [I 2022-10-11 03:46:57] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:19 [I 2022-10-11 03:46:57] ...trap/engines/engine.py.229: process: 0.87582 | load: 0.00315 [I 2022-10-11 03:46:57] ...trap/engines/engine.py.230: loss: 0.21656 [I 2022-10-11 03:46:57] ...optimizers/trijoint.py.151: total_norm: 19.502394 [I 2022-10-11 03:47:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 980/2304 [I 2022-10-11 03:47:06] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:19:12 [I 2022-10-11 03:47:06] ...trap/engines/engine.py.229: process: 0.86654 | load: 0.00309 [I 2022-10-11 03:47:06] ...trap/engines/engine.py.230: loss: 0.13178 [I 2022-10-11 03:47:06] ...optimizers/trijoint.py.151: total_norm: 10.132936 [I 2022-10-11 03:47:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 990/2304 [I 2022-10-11 03:47:14] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:19:00 [I 2022-10-11 03:47:14] ...trap/engines/engine.py.229: process: 0.88123 | load: 0.00308 [I 2022-10-11 03:47:14] ...trap/engines/engine.py.230: loss: 0.12121 [I 2022-10-11 03:47:14] ...optimizers/trijoint.py.151: total_norm: 16.444572 [I 2022-10-11 03:47:23] ...trap/engines/engine.py.224: train: epoch 96 | batch 1000/2304 [I 2022-10-11 03:47:23] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:49 [I 2022-10-11 03:47:23] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00312 [I 2022-10-11 03:47:23] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-11 03:47:23] ...optimizers/trijoint.py.151: total_norm: 25.272453 [I 2022-10-11 03:47:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 1010/2304 [I 2022-10-11 03:47:32] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:18:40 [I 2022-10-11 03:47:32] ...trap/engines/engine.py.229: process: 0.87442 | load: 0.00311 [I 2022-10-11 03:47:32] ...trap/engines/engine.py.230: loss: 0.09692 [I 2022-10-11 03:47:32] ...optimizers/trijoint.py.151: total_norm: 10.703881 [I 2022-10-11 03:47:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 1020/2304 [I 2022-10-11 03:47:40] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:34 [I 2022-10-11 03:47:40] ...trap/engines/engine.py.229: process: 0.88516 | load: 0.00310 [I 2022-10-11 03:47:40] ...trap/engines/engine.py.230: loss: 0.12791 [I 2022-10-11 03:47:40] ...optimizers/trijoint.py.151: total_norm: 14.828425 [I 2022-10-11 03:47:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 1030/2304 [I 2022-10-11 03:47:49] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:28 [I 2022-10-11 03:47:49] ...trap/engines/engine.py.229: process: 0.86935 | load: 0.00312 [I 2022-10-11 03:47:49] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-11 03:47:49] ...optimizers/trijoint.py.151: total_norm: 37.074215 [I 2022-10-11 03:47:58] ...trap/engines/engine.py.224: train: epoch 96 | batch 1040/2304 [I 2022-10-11 03:47:58] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:18:13 [I 2022-10-11 03:47:58] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00308 [I 2022-10-11 03:47:58] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-11 03:47:58] ...optimizers/trijoint.py.151: total_norm: 17.961521 [I 2022-10-11 03:48:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 1050/2304 [I 2022-10-11 03:48:06] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:10 [I 2022-10-11 03:48:06] ...trap/engines/engine.py.229: process: 0.88060 | load: 0.00317 [I 2022-10-11 03:48:06] ...trap/engines/engine.py.230: loss: 0.14673 [I 2022-10-11 03:48:06] ...optimizers/trijoint.py.151: total_norm: 32.394886 [I 2022-10-11 03:48:15] ...trap/engines/engine.py.224: train: epoch 96 | batch 1060/2304 [I 2022-10-11 03:48:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:17:55 [I 2022-10-11 03:48:15] ...trap/engines/engine.py.229: process: 0.86485 | load: 0.00313 [I 2022-10-11 03:48:15] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-11 03:48:15] ...optimizers/trijoint.py.151: total_norm: 62.777336 [I 2022-10-11 03:48:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 1070/2304 [I 2022-10-11 03:48:24] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:51 [I 2022-10-11 03:48:24] ...trap/engines/engine.py.229: process: 0.87198 | load: 0.00308 [I 2022-10-11 03:48:24] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-11 03:48:24] ...optimizers/trijoint.py.151: total_norm: 13.407120 [I 2022-10-11 03:48:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 1080/2304 [I 2022-10-11 03:48:33] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:17:39 [I 2022-10-11 03:48:33] ...trap/engines/engine.py.229: process: 0.87533 | load: 0.00311 [I 2022-10-11 03:48:33] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-11 03:48:33] ...optimizers/trijoint.py.151: total_norm: 10.484379 [I 2022-10-11 03:48:41] ...trap/engines/engine.py.224: train: epoch 96 | batch 1090/2304 [I 2022-10-11 03:48:41] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:30 [I 2022-10-11 03:48:41] ...trap/engines/engine.py.229: process: 0.86506 | load: 0.00313 [I 2022-10-11 03:48:41] ...trap/engines/engine.py.230: loss: 0.12290 [I 2022-10-11 03:48:41] ...optimizers/trijoint.py.151: total_norm: 14.684007 [I 2022-10-11 03:48:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 1100/2304 [I 2022-10-11 03:48:50] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:20 [I 2022-10-11 03:48:50] ...trap/engines/engine.py.229: process: 0.86086 | load: 0.00310 [I 2022-10-11 03:48:50] ...trap/engines/engine.py.230: loss: 0.11871 [I 2022-10-11 03:48:50] ...optimizers/trijoint.py.151: total_norm: 18.071877 [I 2022-10-11 03:48:58] ...trap/engines/engine.py.224: train: epoch 96 | batch 1110/2304 [I 2022-10-11 03:48:58] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:11 [I 2022-10-11 03:48:58] ...trap/engines/engine.py.229: process: 0.86107 | load: 0.00308 [I 2022-10-11 03:48:59] ...trap/engines/engine.py.230: loss: 0.10371 [I 2022-10-11 03:48:59] ...optimizers/trijoint.py.151: total_norm: 13.169619 [I 2022-10-11 03:49:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 1120/2304 [I 2022-10-11 03:49:07] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:03 [I 2022-10-11 03:49:07] ...trap/engines/engine.py.229: process: 0.86219 | load: 0.00308 [I 2022-10-11 03:49:07] ...trap/engines/engine.py.230: loss: 0.12191 [I 2022-10-11 03:49:07] ...optimizers/trijoint.py.151: total_norm: 17.527369 [I 2022-10-11 03:49:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 1130/2304 [I 2022-10-11 03:49:16] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:56 [I 2022-10-11 03:49:16] ...trap/engines/engine.py.229: process: 0.87160 | load: 0.00311 [I 2022-10-11 03:49:16] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-11 03:49:16] ...optimizers/trijoint.py.151: total_norm: 11.552899 [I 2022-10-11 03:49:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 1140/2304 [I 2022-10-11 03:49:25] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:46 [I 2022-10-11 03:49:25] ...trap/engines/engine.py.229: process: 0.86592 | load: 0.00348 [I 2022-10-11 03:49:25] ...trap/engines/engine.py.230: loss: 0.13337 [I 2022-10-11 03:49:25] ...optimizers/trijoint.py.151: total_norm: 18.335884 [I 2022-10-11 03:49:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 1150/2304 [I 2022-10-11 03:49:33] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:32 [I 2022-10-11 03:49:33] ...trap/engines/engine.py.229: process: 0.85162 | load: 0.00313 [I 2022-10-11 03:49:33] ...trap/engines/engine.py.230: loss: 0.14020 [I 2022-10-11 03:49:33] ...optimizers/trijoint.py.151: total_norm: 17.113569 [I 2022-10-11 03:49:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 1160/2304 [I 2022-10-11 03:49:42] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:37 [I 2022-10-11 03:49:42] ...trap/engines/engine.py.229: process: 0.87151 | load: 0.00322 [I 2022-10-11 03:49:42] ...trap/engines/engine.py.230: loss: 0.15703 [I 2022-10-11 03:49:42] ...optimizers/trijoint.py.151: total_norm: 25.508844 [I 2022-10-11 03:49:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 1170/2304 [I 2022-10-11 03:49:51] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:16:25 [I 2022-10-11 03:49:51] ...trap/engines/engine.py.229: process: 0.87713 | load: 0.00318 [I 2022-10-11 03:49:51] ...trap/engines/engine.py.230: loss: 0.10031 [I 2022-10-11 03:49:51] ...optimizers/trijoint.py.151: total_norm: 16.755381 [I 2022-10-11 03:49:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 1180/2304 [I 2022-10-11 03:49:59] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:16:12 [I 2022-10-11 03:49:59] ...trap/engines/engine.py.229: process: 0.87079 | load: 0.00324 [I 2022-10-11 03:49:59] ...trap/engines/engine.py.230: loss: 0.08182 [I 2022-10-11 03:49:59] ...optimizers/trijoint.py.151: total_norm: 13.916034 [I 2022-10-11 03:50:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 1190/2304 [I 2022-10-11 03:50:08] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:54 [I 2022-10-11 03:50:08] ...trap/engines/engine.py.229: process: 0.83510 | load: 0.00298 [I 2022-10-11 03:50:08] ...trap/engines/engine.py.230: loss: 0.09401 [I 2022-10-11 03:50:08] ...optimizers/trijoint.py.151: total_norm: 10.963157 [I 2022-10-11 03:50:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 1200/2304 [I 2022-10-11 03:50:16] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:41 [I 2022-10-11 03:50:16] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00293 [I 2022-10-11 03:50:16] ...trap/engines/engine.py.230: loss: 0.18449 [I 2022-10-11 03:50:16] ...optimizers/trijoint.py.151: total_norm: 18.033775 [I 2022-10-11 03:50:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 1210/2304 [I 2022-10-11 03:50:25] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:33 [I 2022-10-11 03:50:25] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00301 [I 2022-10-11 03:50:25] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-11 03:50:25] ...optimizers/trijoint.py.151: total_norm: 48.392422 [I 2022-10-11 03:50:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1220/2304 [I 2022-10-11 03:50:34] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:25 [I 2022-10-11 03:50:34] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00298 [I 2022-10-11 03:50:34] ...trap/engines/engine.py.230: loss: 0.08066 [I 2022-10-11 03:50:34] ...optimizers/trijoint.py.151: total_norm: 9.521518 [I 2022-10-11 03:50:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 1230/2304 [I 2022-10-11 03:50:42] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:23 [I 2022-10-11 03:50:42] ...trap/engines/engine.py.229: process: 0.86293 | load: 0.00318 [I 2022-10-11 03:50:42] ...trap/engines/engine.py.230: loss: 0.12758 [I 2022-10-11 03:50:42] ...optimizers/trijoint.py.151: total_norm: 16.439857 [I 2022-10-11 03:50:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 1240/2304 [I 2022-10-11 03:50:51] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:11 [I 2022-10-11 03:50:51] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00315 [I 2022-10-11 03:50:51] ...trap/engines/engine.py.230: loss: 0.14343 [I 2022-10-11 03:50:51] ...optimizers/trijoint.py.151: total_norm: 8.176755 [I 2022-10-11 03:50:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 1250/2304 [I 2022-10-11 03:50:59] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:15:05 [I 2022-10-11 03:50:59] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00309 [I 2022-10-11 03:50:59] ...trap/engines/engine.py.230: loss: 0.10621 [I 2022-10-11 03:50:59] ...optimizers/trijoint.py.151: total_norm: 27.775116 [I 2022-10-11 03:51:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 1260/2304 [I 2022-10-11 03:51:08] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:55 [I 2022-10-11 03:51:08] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00314 [I 2022-10-11 03:51:08] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-11 03:51:08] ...optimizers/trijoint.py.151: total_norm: 29.858828 [I 2022-10-11 03:51:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 1270/2304 [I 2022-10-11 03:51:17] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:46 [I 2022-10-11 03:51:17] ...trap/engines/engine.py.229: process: 0.85307 | load: 0.00301 [I 2022-10-11 03:51:17] ...trap/engines/engine.py.230: loss: 0.09983 [I 2022-10-11 03:51:17] ...optimizers/trijoint.py.151: total_norm: 12.208735 [I 2022-10-11 03:51:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 1280/2304 [I 2022-10-11 03:51:25] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:37 [I 2022-10-11 03:51:25] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00297 [I 2022-10-11 03:51:25] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-11 03:51:25] ...optimizers/trijoint.py.151: total_norm: 8.195295 [I 2022-10-11 03:51:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1290/2304 [I 2022-10-11 03:51:34] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:28 [I 2022-10-11 03:51:34] ...trap/engines/engine.py.229: process: 0.85114 | load: 0.00299 [I 2022-10-11 03:51:34] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-11 03:51:34] ...optimizers/trijoint.py.151: total_norm: 8.399942 [I 2022-10-11 03:51:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 1300/2304 [I 2022-10-11 03:51:42] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:14:22 [I 2022-10-11 03:51:42] ...trap/engines/engine.py.229: process: 0.86310 | load: 0.00298 [I 2022-10-11 03:51:42] ...trap/engines/engine.py.230: loss: 0.15112 [I 2022-10-11 03:51:42] ...optimizers/trijoint.py.151: total_norm: 15.288105 [I 2022-10-11 03:51:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 1310/2304 [I 2022-10-11 03:51:51] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:12 [I 2022-10-11 03:51:51] ...trap/engines/engine.py.229: process: 0.85673 | load: 0.00305 [I 2022-10-11 03:51:51] ...trap/engines/engine.py.230: loss: 0.13049 [I 2022-10-11 03:51:51] ...optimizers/trijoint.py.151: total_norm: 39.899261 [I 2022-10-11 03:52:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 1320/2304 [I 2022-10-11 03:52:00] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:14:08 [I 2022-10-11 03:52:00] ...trap/engines/engine.py.229: process: 0.85787 | load: 0.00315 [I 2022-10-11 03:52:00] ...trap/engines/engine.py.230: loss: 0.09364 [I 2022-10-11 03:52:00] ...optimizers/trijoint.py.151: total_norm: 14.286727 [I 2022-10-11 03:52:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 1330/2304 [I 2022-10-11 03:52:08] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:14:00 [I 2022-10-11 03:52:08] ...trap/engines/engine.py.229: process: 0.86651 | load: 0.00313 [I 2022-10-11 03:52:08] ...trap/engines/engine.py.230: loss: 0.10713 [I 2022-10-11 03:52:08] ...optimizers/trijoint.py.151: total_norm: 10.888037 [I 2022-10-11 03:52:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 1340/2304 [I 2022-10-11 03:52:17] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:53 [I 2022-10-11 03:52:17] ...trap/engines/engine.py.229: process: 0.85750 | load: 0.00310 [I 2022-10-11 03:52:17] ...trap/engines/engine.py.230: loss: 0.08914 [I 2022-10-11 03:52:17] ...optimizers/trijoint.py.151: total_norm: 8.115736 [I 2022-10-11 03:52:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 1350/2304 [I 2022-10-11 03:52:26] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:44 [I 2022-10-11 03:52:26] ...trap/engines/engine.py.229: process: 0.86910 | load: 0.00323 [I 2022-10-11 03:52:26] ...trap/engines/engine.py.230: loss: 0.08821 [I 2022-10-11 03:52:26] ...optimizers/trijoint.py.151: total_norm: 10.926332 [I 2022-10-11 03:52:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1360/2304 [I 2022-10-11 03:52:34] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:37 [I 2022-10-11 03:52:34] ...trap/engines/engine.py.229: process: 0.87119 | load: 0.00318 [I 2022-10-11 03:52:34] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-11 03:52:34] ...optimizers/trijoint.py.151: total_norm: 13.756859 [I 2022-10-11 03:52:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 1370/2304 [I 2022-10-11 03:52:43] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:27 [I 2022-10-11 03:52:43] ...trap/engines/engine.py.229: process: 0.87328 | load: 0.00309 [I 2022-10-11 03:52:43] ...trap/engines/engine.py.230: loss: 0.12254 [I 2022-10-11 03:52:43] ...optimizers/trijoint.py.151: total_norm: 28.743809 [I 2022-10-11 03:52:52] ...trap/engines/engine.py.224: train: epoch 96 | batch 1380/2304 [I 2022-10-11 03:52:52] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:21 [I 2022-10-11 03:52:52] ...trap/engines/engine.py.229: process: 0.86345 | load: 0.00324 [I 2022-10-11 03:52:52] ...trap/engines/engine.py.230: loss: 0.10452 [I 2022-10-11 03:52:52] ...optimizers/trijoint.py.151: total_norm: 22.064497 [I 2022-10-11 03:53:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 1390/2304 [I 2022-10-11 03:53:00] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:05 [I 2022-10-11 03:53:00] ...trap/engines/engine.py.229: process: 0.86295 | load: 0.00308 [I 2022-10-11 03:53:00] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-11 03:53:00] ...optimizers/trijoint.py.151: total_norm: 19.163090 [I 2022-10-11 03:53:09] ...trap/engines/engine.py.224: train: epoch 96 | batch 1400/2304 [I 2022-10-11 03:53:09] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:13:01 [I 2022-10-11 03:53:09] ...trap/engines/engine.py.229: process: 0.86718 | load: 0.00312 [I 2022-10-11 03:53:09] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-11 03:53:09] ...optimizers/trijoint.py.151: total_norm: 13.349206 [I 2022-10-11 03:53:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 1410/2304 [I 2022-10-11 03:53:18] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:13:03 [I 2022-10-11 03:53:18] ...trap/engines/engine.py.229: process: 0.86873 | load: 0.00314 [I 2022-10-11 03:53:18] ...trap/engines/engine.py.230: loss: 0.09635 [I 2022-10-11 03:53:18] ...optimizers/trijoint.py.151: total_norm: 12.129398 [I 2022-10-11 03:53:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 1420/2304 [I 2022-10-11 03:53:27] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:50 [I 2022-10-11 03:53:27] ...trap/engines/engine.py.229: process: 0.87817 | load: 0.00319 [I 2022-10-11 03:53:27] ...trap/engines/engine.py.230: loss: 0.10781 [I 2022-10-11 03:53:27] ...optimizers/trijoint.py.151: total_norm: 16.724773 [I 2022-10-11 03:53:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 1430/2304 [I 2022-10-11 03:53:35] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:35 [I 2022-10-11 03:53:35] ...trap/engines/engine.py.229: process: 0.86251 | load: 0.00310 [I 2022-10-11 03:53:35] ...trap/engines/engine.py.230: loss: 0.10670 [I 2022-10-11 03:53:35] ...optimizers/trijoint.py.151: total_norm: 34.217812 [I 2022-10-11 03:53:44] ...trap/engines/engine.py.224: train: epoch 96 | batch 1440/2304 [I 2022-10-11 03:53:44] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:34 [I 2022-10-11 03:53:44] ...trap/engines/engine.py.229: process: 0.89093 | load: 0.00315 [I 2022-10-11 03:53:44] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-11 03:53:44] ...optimizers/trijoint.py.151: total_norm: 15.129894 [I 2022-10-11 03:53:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 1450/2304 [I 2022-10-11 03:53:53] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:12:26 [I 2022-10-11 03:53:53] ...trap/engines/engine.py.229: process: 0.88337 | load: 0.00305 [I 2022-10-11 03:53:53] ...trap/engines/engine.py.230: loss: 0.13033 [I 2022-10-11 03:53:53] ...optimizers/trijoint.py.151: total_norm: 15.130301 [I 2022-10-11 03:54:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 1460/2304 [I 2022-10-11 03:54:01] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:12:14 [I 2022-10-11 03:54:01] ...trap/engines/engine.py.229: process: 0.87730 | load: 0.00316 [I 2022-10-11 03:54:01] ...trap/engines/engine.py.230: loss: 0.13697 [I 2022-10-11 03:54:01] ...optimizers/trijoint.py.151: total_norm: 12.145983 [I 2022-10-11 03:54:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 1470/2304 [I 2022-10-11 03:54:10] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:12:02 [I 2022-10-11 03:54:10] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00321 [I 2022-10-11 03:54:10] ...trap/engines/engine.py.230: loss: 0.12843 [I 2022-10-11 03:54:10] ...optimizers/trijoint.py.151: total_norm: 19.840117 [I 2022-10-11 03:54:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 1480/2304 [I 2022-10-11 03:54:19] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:11:53 [I 2022-10-11 03:54:19] ...trap/engines/engine.py.229: process: 0.86694 | load: 0.00322 [I 2022-10-11 03:54:19] ...trap/engines/engine.py.230: loss: 0.19204 [I 2022-10-11 03:54:19] ...optimizers/trijoint.py.151: total_norm: 10.757216 [I 2022-10-11 03:54:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 1490/2304 [I 2022-10-11 03:54:27] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:11:49 [I 2022-10-11 03:54:27] ...trap/engines/engine.py.229: process: 0.87535 | load: 0.00335 [I 2022-10-11 03:54:27] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-11 03:54:27] ...optimizers/trijoint.py.151: total_norm: 12.753223 [I 2022-10-11 03:54:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 1500/2304 [I 2022-10-11 03:54:36] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:35 [I 2022-10-11 03:54:36] ...trap/engines/engine.py.229: process: 0.86529 | load: 0.00322 [I 2022-10-11 03:54:36] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-11 03:54:36] ...optimizers/trijoint.py.151: total_norm: 12.238134 [I 2022-10-11 03:54:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 1510/2304 [I 2022-10-11 03:54:45] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:28 [I 2022-10-11 03:54:45] ...trap/engines/engine.py.229: process: 0.87910 | load: 0.00325 [I 2022-10-11 03:54:45] ...trap/engines/engine.py.230: loss: 0.08213 [I 2022-10-11 03:54:45] ...optimizers/trijoint.py.151: total_norm: 16.243885 [I 2022-10-11 03:54:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 1520/2304 [I 2022-10-11 03:54:53] ...trap/engines/engine.py.228: elapsed: 0:21:57 | left: 0:11:17 [I 2022-10-11 03:54:53] ...trap/engines/engine.py.229: process: 0.85858 | load: 0.00365 [I 2022-10-11 03:54:53] ...trap/engines/engine.py.230: loss: 0.08362 [I 2022-10-11 03:54:53] ...optimizers/trijoint.py.151: total_norm: 9.569807 [I 2022-10-11 03:55:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 1530/2304 [I 2022-10-11 03:55:02] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:10 [I 2022-10-11 03:55:02] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00305 [I 2022-10-11 03:55:02] ...trap/engines/engine.py.230: loss: 0.12329 [I 2022-10-11 03:55:02] ...optimizers/trijoint.py.151: total_norm: 12.747394 [I 2022-10-11 03:55:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 1540/2304 [I 2022-10-11 03:55:11] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:11:01 [I 2022-10-11 03:55:11] ...trap/engines/engine.py.229: process: 0.89347 | load: 0.00308 [I 2022-10-11 03:55:11] ...trap/engines/engine.py.230: loss: 0.11516 [I 2022-10-11 03:55:11] ...optimizers/trijoint.py.151: total_norm: 9.858720 [I 2022-10-11 03:55:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 1550/2304 [I 2022-10-11 03:55:19] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:51 [I 2022-10-11 03:55:19] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00310 [I 2022-10-11 03:55:19] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-11 03:55:19] ...optimizers/trijoint.py.151: total_norm: 14.998989 [I 2022-10-11 03:55:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 1560/2304 [I 2022-10-11 03:55:28] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:40 [I 2022-10-11 03:55:28] ...trap/engines/engine.py.229: process: 0.85759 | load: 0.00311 [I 2022-10-11 03:55:28] ...trap/engines/engine.py.230: loss: 0.09179 [I 2022-10-11 03:55:28] ...optimizers/trijoint.py.151: total_norm: 9.169376 [I 2022-10-11 03:55:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 1570/2304 [I 2022-10-11 03:55:37] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:34 [I 2022-10-11 03:55:37] ...trap/engines/engine.py.229: process: 0.86107 | load: 0.00311 [I 2022-10-11 03:55:37] ...trap/engines/engine.py.230: loss: 0.15330 [I 2022-10-11 03:55:37] ...optimizers/trijoint.py.151: total_norm: 8.839006 [I 2022-10-11 03:55:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 1580/2304 [I 2022-10-11 03:55:45] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:22 [I 2022-10-11 03:55:45] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00315 [I 2022-10-11 03:55:45] ...trap/engines/engine.py.230: loss: 0.14431 [I 2022-10-11 03:55:45] ...optimizers/trijoint.py.151: total_norm: 13.760121 [I 2022-10-11 03:55:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 1590/2304 [I 2022-10-11 03:55:54] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:13 [I 2022-10-11 03:55:54] ...trap/engines/engine.py.229: process: 0.84653 | load: 0.00296 [I 2022-10-11 03:55:54] ...trap/engines/engine.py.230: loss: 0.10760 [I 2022-10-11 03:55:54] ...optimizers/trijoint.py.151: total_norm: 15.265059 [I 2022-10-11 03:56:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 1600/2304 [I 2022-10-11 03:56:03] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:59 [I 2022-10-11 03:56:03] ...trap/engines/engine.py.229: process: 0.85181 | load: 0.00294 [I 2022-10-11 03:56:03] ...trap/engines/engine.py.230: loss: 0.15269 [I 2022-10-11 03:56:03] ...optimizers/trijoint.py.151: total_norm: 24.832201 [I 2022-10-11 03:56:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 1610/2304 [I 2022-10-11 03:56:11] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:55 [I 2022-10-11 03:56:11] ...trap/engines/engine.py.229: process: 0.86243 | load: 0.00293 [I 2022-10-11 03:56:11] ...trap/engines/engine.py.230: loss: 0.08038 [I 2022-10-11 03:56:11] ...optimizers/trijoint.py.151: total_norm: 9.225395 [I 2022-10-11 03:56:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 1620/2304 [I 2022-10-11 03:56:20] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:46 [I 2022-10-11 03:56:20] ...trap/engines/engine.py.229: process: 0.86701 | load: 0.00302 [I 2022-10-11 03:56:20] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-11 03:56:20] ...optimizers/trijoint.py.151: total_norm: 24.135714 [I 2022-10-11 03:56:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 1630/2304 [I 2022-10-11 03:56:28] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:31 [I 2022-10-11 03:56:28] ...trap/engines/engine.py.229: process: 0.83644 | load: 0.00303 [I 2022-10-11 03:56:28] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-11 03:56:28] ...optimizers/trijoint.py.151: total_norm: 14.009941 [I 2022-10-11 03:56:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 1640/2304 [I 2022-10-11 03:56:37] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:28 [I 2022-10-11 03:56:37] ...trap/engines/engine.py.229: process: 0.85634 | load: 0.00307 [I 2022-10-11 03:56:37] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-11 03:56:37] ...optimizers/trijoint.py.151: total_norm: 15.068187 [I 2022-10-11 03:56:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 1650/2304 [I 2022-10-11 03:56:45] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:17 [I 2022-10-11 03:56:45] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00300 [I 2022-10-11 03:56:45] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-11 03:56:45] ...optimizers/trijoint.py.151: total_norm: 10.139155 [I 2022-10-11 03:56:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 1660/2304 [I 2022-10-11 03:56:54] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:08 [I 2022-10-11 03:56:54] ...trap/engines/engine.py.229: process: 0.84362 | load: 0.00297 [I 2022-10-11 03:56:54] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-11 03:56:54] ...optimizers/trijoint.py.151: total_norm: 10.259199 [I 2022-10-11 03:57:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 1670/2304 [I 2022-10-11 03:57:02] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:59 [I 2022-10-11 03:57:02] ...trap/engines/engine.py.229: process: 0.86152 | load: 0.00294 [I 2022-10-11 03:57:02] ...trap/engines/engine.py.230: loss: 0.10461 [I 2022-10-11 03:57:02] ...optimizers/trijoint.py.151: total_norm: 14.524433 [I 2022-10-11 03:57:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 1680/2304 [I 2022-10-11 03:57:11] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:51 [I 2022-10-11 03:57:11] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00300 [I 2022-10-11 03:57:11] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-11 03:57:11] ...optimizers/trijoint.py.151: total_norm: 11.556343 [I 2022-10-11 03:57:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 1690/2304 [I 2022-10-11 03:57:20] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:43 [I 2022-10-11 03:57:20] ...trap/engines/engine.py.229: process: 0.83712 | load: 0.00297 [I 2022-10-11 03:57:20] ...trap/engines/engine.py.230: loss: 0.17490 [I 2022-10-11 03:57:20] ...optimizers/trijoint.py.151: total_norm: 26.591448 [I 2022-10-11 03:57:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 1700/2304 [I 2022-10-11 03:57:28] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:35 [I 2022-10-11 03:57:28] ...trap/engines/engine.py.229: process: 0.83949 | load: 0.00308 [I 2022-10-11 03:57:28] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-11 03:57:28] ...optimizers/trijoint.py.151: total_norm: 29.746647 [I 2022-10-11 03:57:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 1710/2304 [I 2022-10-11 03:57:37] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:27 [I 2022-10-11 03:57:37] ...trap/engines/engine.py.229: process: 0.84786 | load: 0.00305 [I 2022-10-11 03:57:37] ...trap/engines/engine.py.230: loss: 0.08301 [I 2022-10-11 03:57:37] ...optimizers/trijoint.py.151: total_norm: 8.379122 [I 2022-10-11 03:57:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 1720/2304 [I 2022-10-11 03:57:45] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:18 [I 2022-10-11 03:57:45] ...trap/engines/engine.py.229: process: 0.84573 | load: 0.00318 [I 2022-10-11 03:57:45] ...trap/engines/engine.py.230: loss: 0.10358 [I 2022-10-11 03:57:45] ...optimizers/trijoint.py.151: total_norm: 9.668566 [I 2022-10-11 03:57:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 1730/2304 [I 2022-10-11 03:57:54] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:08:13 [I 2022-10-11 03:57:54] ...trap/engines/engine.py.229: process: 0.87031 | load: 0.00309 [I 2022-10-11 03:57:54] ...trap/engines/engine.py.230: loss: 0.09761 [I 2022-10-11 03:57:54] ...optimizers/trijoint.py.151: total_norm: 8.223899 [I 2022-10-11 03:58:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 1740/2304 [I 2022-10-11 03:58:03] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:08:07 [I 2022-10-11 03:58:03] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00309 [I 2022-10-11 03:58:03] ...trap/engines/engine.py.230: loss: 0.19841 [I 2022-10-11 03:58:03] ...optimizers/trijoint.py.151: total_norm: 16.683277 [I 2022-10-11 03:58:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 1750/2304 [I 2022-10-11 03:58:11] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:58 [I 2022-10-11 03:58:11] ...trap/engines/engine.py.229: process: 0.87269 | load: 0.00309 [I 2022-10-11 03:58:11] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-11 03:58:11] ...optimizers/trijoint.py.151: total_norm: 13.510132 [I 2022-10-11 03:58:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 1760/2304 [I 2022-10-11 03:58:20] ...trap/engines/engine.py.228: elapsed: 0:25:24 | left: 0:07:51 [I 2022-10-11 03:58:20] ...trap/engines/engine.py.229: process: 0.86625 | load: 0.00309 [I 2022-10-11 03:58:20] ...trap/engines/engine.py.230: loss: 0.16141 [I 2022-10-11 03:58:20] ...optimizers/trijoint.py.151: total_norm: 23.529776 [I 2022-10-11 03:58:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 1770/2304 [I 2022-10-11 03:58:29] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:41 [I 2022-10-11 03:58:29] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00306 [I 2022-10-11 03:58:29] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-11 03:58:29] ...optimizers/trijoint.py.151: total_norm: 15.191794 [I 2022-10-11 03:58:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 1780/2304 [I 2022-10-11 03:58:37] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:33 [I 2022-10-11 03:58:37] ...trap/engines/engine.py.229: process: 0.86830 | load: 0.00329 [I 2022-10-11 03:58:37] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-11 03:58:37] ...optimizers/trijoint.py.151: total_norm: 14.988975 [I 2022-10-11 03:58:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 1790/2304 [I 2022-10-11 03:58:46] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:07:23 [I 2022-10-11 03:58:46] ...trap/engines/engine.py.229: process: 0.87051 | load: 0.00311 [I 2022-10-11 03:58:46] ...trap/engines/engine.py.230: loss: 0.09920 [I 2022-10-11 03:58:46] ...optimizers/trijoint.py.151: total_norm: 13.979854 [I 2022-10-11 03:58:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 1800/2304 [I 2022-10-11 03:58:55] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:19 [I 2022-10-11 03:58:55] ...trap/engines/engine.py.229: process: 0.88651 | load: 0.00309 [I 2022-10-11 03:58:55] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-11 03:58:55] ...optimizers/trijoint.py.151: total_norm: 50.403664 [I 2022-10-11 03:59:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 1810/2304 [I 2022-10-11 03:59:03] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:07:08 [I 2022-10-11 03:59:03] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00313 [I 2022-10-11 03:59:03] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-11 03:59:03] ...optimizers/trijoint.py.151: total_norm: 13.586069 [I 2022-10-11 03:59:12] ...trap/engines/engine.py.224: train: epoch 96 | batch 1820/2304 [I 2022-10-11 03:59:12] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:59 [I 2022-10-11 03:59:12] ...trap/engines/engine.py.229: process: 0.87168 | load: 0.00311 [I 2022-10-11 03:59:12] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-11 03:59:12] ...optimizers/trijoint.py.151: total_norm: 17.459784 [I 2022-10-11 03:59:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 1830/2304 [I 2022-10-11 03:59:21] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:52 [I 2022-10-11 03:59:21] ...trap/engines/engine.py.229: process: 0.86773 | load: 0.00315 [I 2022-10-11 03:59:21] ...trap/engines/engine.py.230: loss: 0.09078 [I 2022-10-11 03:59:21] ...optimizers/trijoint.py.151: total_norm: 20.542921 [I 2022-10-11 03:59:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 1840/2304 [I 2022-10-11 03:59:29] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:45 [I 2022-10-11 03:59:29] ...trap/engines/engine.py.229: process: 0.87752 | load: 0.00314 [I 2022-10-11 03:59:29] ...trap/engines/engine.py.230: loss: 0.09643 [I 2022-10-11 03:59:29] ...optimizers/trijoint.py.151: total_norm: 14.378823 [I 2022-10-11 03:59:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 1850/2304 [I 2022-10-11 03:59:38] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:35 [I 2022-10-11 03:59:38] ...trap/engines/engine.py.229: process: 0.87812 | load: 0.00320 [I 2022-10-11 03:59:38] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-11 03:59:38] ...optimizers/trijoint.py.151: total_norm: 40.996803 [I 2022-10-11 03:59:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 1860/2304 [I 2022-10-11 03:59:47] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:24 [I 2022-10-11 03:59:47] ...trap/engines/engine.py.229: process: 0.85870 | load: 0.00308 [I 2022-10-11 03:59:47] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-11 03:59:47] ...optimizers/trijoint.py.151: total_norm: 13.953712 [I 2022-10-11 03:59:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 1870/2304 [I 2022-10-11 03:59:55] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:06:13 [I 2022-10-11 03:59:55] ...trap/engines/engine.py.229: process: 0.86561 | load: 0.00325 [I 2022-10-11 03:59:55] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-11 03:59:55] ...optimizers/trijoint.py.151: total_norm: 12.299681 [I 2022-10-11 04:00:04] ...trap/engines/engine.py.224: train: epoch 96 | batch 1880/2304 [I 2022-10-11 04:00:04] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:06:05 [I 2022-10-11 04:00:04] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00309 [I 2022-10-11 04:00:04] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-11 04:00:04] ...optimizers/trijoint.py.151: total_norm: 16.261786 [I 2022-10-11 04:00:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 1890/2304 [I 2022-10-11 04:00:13] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:59 [I 2022-10-11 04:00:13] ...trap/engines/engine.py.229: process: 0.86593 | load: 0.00312 [I 2022-10-11 04:00:13] ...trap/engines/engine.py.230: loss: 0.09661 [I 2022-10-11 04:00:13] ...optimizers/trijoint.py.151: total_norm: 11.608816 [I 2022-10-11 04:00:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 1900/2304 [I 2022-10-11 04:00:21] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:49 [I 2022-10-11 04:00:21] ...trap/engines/engine.py.229: process: 0.85455 | load: 0.00315 [I 2022-10-11 04:00:21] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-11 04:00:21] ...optimizers/trijoint.py.151: total_norm: 17.851362 [I 2022-10-11 04:00:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 1910/2304 [I 2022-10-11 04:00:30] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:40 [I 2022-10-11 04:00:30] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00315 [I 2022-10-11 04:00:30] ...trap/engines/engine.py.230: loss: 0.07946 [I 2022-10-11 04:00:30] ...optimizers/trijoint.py.151: total_norm: 11.217093 [I 2022-10-11 04:00:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 1920/2304 [I 2022-10-11 04:00:39] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:34 [I 2022-10-11 04:00:39] ...trap/engines/engine.py.229: process: 0.87712 | load: 0.00312 [I 2022-10-11 04:00:39] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-11 04:00:39] ...optimizers/trijoint.py.151: total_norm: 27.043465 [I 2022-10-11 04:00:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 1930/2304 [I 2022-10-11 04:00:48] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:24 [I 2022-10-11 04:00:48] ...trap/engines/engine.py.229: process: 0.86938 | load: 0.00340 [I 2022-10-11 04:00:48] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-11 04:00:48] ...optimizers/trijoint.py.151: total_norm: 12.187495 [I 2022-10-11 04:00:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 1940/2304 [I 2022-10-11 04:00:56] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:14 [I 2022-10-11 04:00:56] ...trap/engines/engine.py.229: process: 0.86361 | load: 0.00315 [I 2022-10-11 04:00:56] ...trap/engines/engine.py.230: loss: 0.10292 [I 2022-10-11 04:00:56] ...optimizers/trijoint.py.151: total_norm: 14.919003 [I 2022-10-11 04:01:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 1950/2304 [I 2022-10-11 04:01:05] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:06 [I 2022-10-11 04:01:05] ...trap/engines/engine.py.229: process: 0.86940 | load: 0.00309 [I 2022-10-11 04:01:05] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-11 04:01:05] ...optimizers/trijoint.py.151: total_norm: 11.197948 [I 2022-10-11 04:01:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 1960/2304 [I 2022-10-11 04:01:14] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:58 [I 2022-10-11 04:01:14] ...trap/engines/engine.py.229: process: 0.86661 | load: 0.00305 [I 2022-10-11 04:01:14] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-11 04:01:14] ...optimizers/trijoint.py.151: total_norm: 14.182853 [I 2022-10-11 04:01:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 1970/2304 [I 2022-10-11 04:01:22] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:48 [I 2022-10-11 04:01:22] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00321 [I 2022-10-11 04:01:22] ...trap/engines/engine.py.230: loss: 0.07333 [I 2022-10-11 04:01:22] ...optimizers/trijoint.py.151: total_norm: 11.208524 [I 2022-10-11 04:01:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 1980/2304 [I 2022-10-11 04:01:31] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:41 [I 2022-10-11 04:01:31] ...trap/engines/engine.py.229: process: 0.88358 | load: 0.00310 [I 2022-10-11 04:01:31] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-11 04:01:31] ...optimizers/trijoint.py.151: total_norm: 26.110186 [I 2022-10-11 04:01:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 1990/2304 [I 2022-10-11 04:01:39] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:30 [I 2022-10-11 04:01:39] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00305 [I 2022-10-11 04:01:39] ...trap/engines/engine.py.230: loss: 0.14397 [I 2022-10-11 04:01:39] ...optimizers/trijoint.py.151: total_norm: 13.996184 [I 2022-10-11 04:01:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 2000/2304 [I 2022-10-11 04:01:48] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:22 [I 2022-10-11 04:01:48] ...trap/engines/engine.py.229: process: 0.86864 | load: 0.00294 [I 2022-10-11 04:01:48] ...trap/engines/engine.py.230: loss: 0.12472 [I 2022-10-11 04:01:48] ...optimizers/trijoint.py.151: total_norm: 11.672313 [I 2022-10-11 04:01:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 2010/2304 [I 2022-10-11 04:01:57] ...trap/engines/engine.py.228: elapsed: 0:29:01 | left: 0:04:12 [I 2022-10-11 04:01:57] ...trap/engines/engine.py.229: process: 0.86195 | load: 0.00297 [I 2022-10-11 04:01:57] ...trap/engines/engine.py.230: loss: 0.14379 [I 2022-10-11 04:01:57] ...optimizers/trijoint.py.151: total_norm: 18.474394 [I 2022-10-11 04:02:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 2020/2304 [I 2022-10-11 04:02:05] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:03 [I 2022-10-11 04:02:05] ...trap/engines/engine.py.229: process: 0.85816 | load: 0.00303 [I 2022-10-11 04:02:05] ...trap/engines/engine.py.230: loss: 0.09089 [I 2022-10-11 04:02:05] ...optimizers/trijoint.py.151: total_norm: 17.223589 [I 2022-10-11 04:02:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 2030/2304 [I 2022-10-11 04:02:14] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:03:52 [I 2022-10-11 04:02:14] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00298 [I 2022-10-11 04:02:14] ...trap/engines/engine.py.230: loss: 0.16304 [I 2022-10-11 04:02:14] ...optimizers/trijoint.py.151: total_norm: 15.857050 [I 2022-10-11 04:02:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 2040/2304 [I 2022-10-11 04:02:22] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:46 [I 2022-10-11 04:02:22] ...trap/engines/engine.py.229: process: 0.85827 | load: 0.00303 [I 2022-10-11 04:02:22] ...trap/engines/engine.py.230: loss: 0.10699 [I 2022-10-11 04:02:22] ...optimizers/trijoint.py.151: total_norm: 7.505900 [I 2022-10-11 04:02:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 2050/2304 [I 2022-10-11 04:02:31] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:37 [I 2022-10-11 04:02:31] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00298 [I 2022-10-11 04:02:31] ...trap/engines/engine.py.230: loss: 0.11645 [I 2022-10-11 04:02:31] ...optimizers/trijoint.py.151: total_norm: 11.857823 [I 2022-10-11 04:02:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 2060/2304 [I 2022-10-11 04:02:40] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:27 [I 2022-10-11 04:02:40] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00300 [I 2022-10-11 04:02:40] ...trap/engines/engine.py.230: loss: 0.13749 [I 2022-10-11 04:02:40] ...optimizers/trijoint.py.151: total_norm: 13.571178 [I 2022-10-11 04:02:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 2070/2304 [I 2022-10-11 04:02:48] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:20 [I 2022-10-11 04:02:48] ...trap/engines/engine.py.229: process: 0.84477 | load: 0.00321 [I 2022-10-11 04:02:48] ...trap/engines/engine.py.230: loss: 0.13294 [I 2022-10-11 04:02:48] ...optimizers/trijoint.py.151: total_norm: 18.727549 [I 2022-10-11 04:02:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 2080/2304 [I 2022-10-11 04:02:57] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:03:11 [I 2022-10-11 04:02:57] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00302 [I 2022-10-11 04:02:57] ...trap/engines/engine.py.230: loss: 0.15949 [I 2022-10-11 04:02:57] ...optimizers/trijoint.py.151: total_norm: 15.465770 [I 2022-10-11 04:03:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 2090/2304 [I 2022-10-11 04:03:05] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:03:02 [I 2022-10-11 04:03:05] ...trap/engines/engine.py.229: process: 0.83656 | load: 0.00317 [I 2022-10-11 04:03:05] ...trap/engines/engine.py.230: loss: 0.13243 [I 2022-10-11 04:03:05] ...optimizers/trijoint.py.151: total_norm: 14.935868 [I 2022-10-11 04:03:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 2100/2304 [I 2022-10-11 04:03:14] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:02:53 [I 2022-10-11 04:03:14] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00304 [I 2022-10-11 04:03:14] ...trap/engines/engine.py.230: loss: 0.10431 [I 2022-10-11 04:03:14] ...optimizers/trijoint.py.151: total_norm: 16.356552 [I 2022-10-11 04:03:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 2110/2304 [I 2022-10-11 04:03:22] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:45 [I 2022-10-11 04:03:22] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00308 [I 2022-10-11 04:03:22] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-11 04:03:22] ...optimizers/trijoint.py.151: total_norm: 22.523798 [I 2022-10-11 04:03:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 2120/2304 [I 2022-10-11 04:03:31] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:37 [I 2022-10-11 04:03:31] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00309 [I 2022-10-11 04:03:31] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-11 04:03:31] ...optimizers/trijoint.py.151: total_norm: 8.635399 [I 2022-10-11 04:03:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 2130/2304 [I 2022-10-11 04:03:40] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:30 [I 2022-10-11 04:03:40] ...trap/engines/engine.py.229: process: 0.87903 | load: 0.00313 [I 2022-10-11 04:03:40] ...trap/engines/engine.py.230: loss: 0.08966 [I 2022-10-11 04:03:40] ...optimizers/trijoint.py.151: total_norm: 22.623001 [I 2022-10-11 04:03:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 2140/2304 [I 2022-10-11 04:03:48] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:21 [I 2022-10-11 04:03:48] ...trap/engines/engine.py.229: process: 0.87048 | load: 0.00321 [I 2022-10-11 04:03:48] ...trap/engines/engine.py.230: loss: 0.10688 [I 2022-10-11 04:03:48] ...optimizers/trijoint.py.151: total_norm: 16.266554 [I 2022-10-11 04:03:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 2150/2304 [I 2022-10-11 04:03:57] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:14 [I 2022-10-11 04:03:57] ...trap/engines/engine.py.229: process: 0.87589 | load: 0.00308 [I 2022-10-11 04:03:57] ...trap/engines/engine.py.230: loss: 0.09919 [I 2022-10-11 04:03:57] ...optimizers/trijoint.py.151: total_norm: 34.708401 [I 2022-10-11 04:04:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 2160/2304 [I 2022-10-11 04:04:06] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:04 [I 2022-10-11 04:04:06] ...trap/engines/engine.py.229: process: 0.88112 | load: 0.00313 [I 2022-10-11 04:04:06] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-11 04:04:06] ...optimizers/trijoint.py.151: total_norm: 20.895586 [I 2022-10-11 04:04:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 2170/2304 [I 2022-10-11 04:04:14] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:56 [I 2022-10-11 04:04:14] ...trap/engines/engine.py.229: process: 0.86846 | load: 0.00319 [I 2022-10-11 04:04:14] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-11 04:04:14] ...optimizers/trijoint.py.151: total_norm: 16.176235 [I 2022-10-11 04:04:23] ...trap/engines/engine.py.224: train: epoch 96 | batch 2180/2304 [I 2022-10-11 04:04:23] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:46 [I 2022-10-11 04:04:23] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00322 [I 2022-10-11 04:04:23] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-11 04:04:23] ...optimizers/trijoint.py.151: total_norm: 31.453987 [I 2022-10-11 04:04:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 2190/2304 [I 2022-10-11 04:04:32] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:38 [I 2022-10-11 04:04:32] ...trap/engines/engine.py.229: process: 0.86335 | load: 0.00323 [I 2022-10-11 04:04:32] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-11 04:04:32] ...optimizers/trijoint.py.151: total_norm: 25.254818 [I 2022-10-11 04:04:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 2200/2304 [I 2022-10-11 04:04:40] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:30 [I 2022-10-11 04:04:40] ...trap/engines/engine.py.229: process: 0.87811 | load: 0.00330 [I 2022-10-11 04:04:40] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-11 04:04:40] ...optimizers/trijoint.py.151: total_norm: 14.269316 [I 2022-10-11 04:04:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 2210/2304 [I 2022-10-11 04:04:49] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:21 [I 2022-10-11 04:04:49] ...trap/engines/engine.py.229: process: 0.86979 | load: 0.00312 [I 2022-10-11 04:04:49] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-11 04:04:49] ...optimizers/trijoint.py.151: total_norm: 14.581297 [I 2022-10-11 04:04:58] ...trap/engines/engine.py.224: train: epoch 96 | batch 2220/2304 [I 2022-10-11 04:04:58] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:01:12 [I 2022-10-11 04:04:58] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00333 [I 2022-10-11 04:04:58] ...trap/engines/engine.py.230: loss: 0.12107 [I 2022-10-11 04:04:58] ...optimizers/trijoint.py.151: total_norm: 14.772975 [I 2022-10-11 04:05:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 2230/2304 [I 2022-10-11 04:05:06] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:03 [I 2022-10-11 04:05:06] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00311 [I 2022-10-11 04:05:06] ...trap/engines/engine.py.230: loss: 0.16105 [I 2022-10-11 04:05:06] ...optimizers/trijoint.py.151: total_norm: 18.985495 [I 2022-10-11 04:05:15] ...trap/engines/engine.py.224: train: epoch 96 | batch 2240/2304 [I 2022-10-11 04:05:15] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:55 [I 2022-10-11 04:05:15] ...trap/engines/engine.py.229: process: 0.86363 | load: 0.00307 [I 2022-10-11 04:05:15] ...trap/engines/engine.py.230: loss: 0.22899 [I 2022-10-11 04:05:15] ...optimizers/trijoint.py.151: total_norm: 34.300461 [I 2022-10-11 04:05:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 2250/2304 [I 2022-10-11 04:05:24] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:46 [I 2022-10-11 04:05:24] ...trap/engines/engine.py.229: process: 0.86154 | load: 0.00322 [I 2022-10-11 04:05:24] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-11 04:05:24] ...optimizers/trijoint.py.151: total_norm: 23.999117 [I 2022-10-11 04:05:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 2260/2304 [I 2022-10-11 04:05:33] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:38 [I 2022-10-11 04:05:33] ...trap/engines/engine.py.229: process: 0.86295 | load: 0.00315 [I 2022-10-11 04:05:33] ...trap/engines/engine.py.230: loss: 0.14459 [I 2022-10-11 04:05:33] ...optimizers/trijoint.py.151: total_norm: 12.520074 [I 2022-10-11 04:05:41] ...trap/engines/engine.py.224: train: epoch 96 | batch 2270/2304 [I 2022-10-11 04:05:41] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:29 [I 2022-10-11 04:05:41] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00311 [I 2022-10-11 04:05:41] ...trap/engines/engine.py.230: loss: 0.10048 [I 2022-10-11 04:05:41] ...optimizers/trijoint.py.151: total_norm: 13.270607 [I 2022-10-11 04:05:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 2280/2304 [I 2022-10-11 04:05:50] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:20 [I 2022-10-11 04:05:50] ...trap/engines/engine.py.229: process: 0.85916 | load: 0.00315 [I 2022-10-11 04:05:50] ...trap/engines/engine.py.230: loss: 0.09581 [I 2022-10-11 04:05:50] ...optimizers/trijoint.py.151: total_norm: 7.954445 [I 2022-10-11 04:06:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 2290/2304 [I 2022-10-11 04:06:18] ...trap/engines/engine.py.228: elapsed: 0:33:22 | left: 0:00:40 [I 2022-10-11 04:06:18] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00311 [I 2022-10-11 04:06:18] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-11 04:06:18] ...optimizers/trijoint.py.151: total_norm: 15.232344 [I 2022-10-11 04:06:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 2300/2304 [I 2022-10-11 04:06:27] ...trap/engines/engine.py.228: elapsed: 0:33:31 | left: 0:00:04 [I 2022-10-11 04:06:27] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00019 [I 2022-10-11 04:06:27] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-11 04:06:27] ...optimizers/trijoint.py.151: total_norm: 13.600517 [I 2022-10-11 04:06:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 2304/2304 [I 2022-10-11 04:06:30] ...trap/engines/engine.py.228: elapsed: 0:33:34 | left: 0:00:00 [I 2022-10-11 04:06:30] ...trap/engines/engine.py.229: process: 0.83027 | load: 0.00019 [I 2022-10-11 04:06:30] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-11 04:06:30] ...optimizers/trijoint.py.151: total_norm: 15.589513 [S 2022-10-11 04:06:32] ...trap/engines/engine.py.129: train_epoch.epoch: 96 [S 2022-10-11 04:06:32] ...trap/engines/engine.py.129: train_epoch.loss: 0.1253321561372461 [I 2022-10-11 04:06:32] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 04:06:33] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 96 [I 2022-10-11 04:06:33] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 04:06:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 04:06:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 04:06:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 04:06:39] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 04:06:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 0/510 [I 2022-10-11 04:06:40] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:26 [I 2022-10-11 04:06:40] ...trap/engines/engine.py.326: process: 6.16904 | load: 5.60164 [I 2022-10-11 04:06:44] ...trap/engines/engine.py.321: eval: epoch 96 | batch 10/510 [I 2022-10-11 04:06:44] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:15 [I 2022-10-11 04:06:44] ...trap/engines/engine.py.326: process: 0.51661 | load: 0.00019 [I 2022-10-11 04:06:51] ...trap/engines/engine.py.321: eval: epoch 96 | batch 20/510 [I 2022-10-11 04:06:51] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:48 [I 2022-10-11 04:06:51] ...trap/engines/engine.py.326: process: 0.62932 | load: 0.00020 [I 2022-10-11 04:06:58] ...trap/engines/engine.py.321: eval: epoch 96 | batch 30/510 [I 2022-10-11 04:06:58] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:26 [I 2022-10-11 04:06:58] ...trap/engines/engine.py.326: process: 0.78389 | load: 0.00021 [I 2022-10-11 04:07:05] ...trap/engines/engine.py.321: eval: epoch 96 | batch 40/510 [I 2022-10-11 04:07:05] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:05:05 [I 2022-10-11 04:07:05] ...trap/engines/engine.py.326: process: 0.55776 | load: 0.00020 [I 2022-10-11 04:07:09] ...trap/engines/engine.py.321: eval: epoch 96 | batch 50/510 [I 2022-10-11 04:07:09] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:01 [I 2022-10-11 04:07:09] ...trap/engines/engine.py.326: process: 0.55641 | load: 0.00021 [I 2022-10-11 04:07:15] ...trap/engines/engine.py.321: eval: epoch 96 | batch 60/510 [I 2022-10-11 04:07:15] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:00 [I 2022-10-11 04:07:15] ...trap/engines/engine.py.326: process: 0.56996 | load: 0.00021 [I 2022-10-11 04:07:20] ...trap/engines/engine.py.321: eval: epoch 96 | batch 70/510 [I 2022-10-11 04:07:20] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:47 [I 2022-10-11 04:07:20] ...trap/engines/engine.py.326: process: 0.42122 | load: 0.00023 [I 2022-10-11 04:07:25] ...trap/engines/engine.py.321: eval: epoch 96 | batch 80/510 [I 2022-10-11 04:07:25] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:28 [I 2022-10-11 04:07:25] ...trap/engines/engine.py.326: process: 0.49931 | load: 0.00019 [I 2022-10-11 04:07:29] ...trap/engines/engine.py.321: eval: epoch 96 | batch 90/510 [I 2022-10-11 04:07:29] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:03:20 [I 2022-10-11 04:07:29] ...trap/engines/engine.py.326: process: 0.51779 | load: 0.00018 [I 2022-10-11 04:07:34] ...trap/engines/engine.py.321: eval: epoch 96 | batch 100/510 [I 2022-10-11 04:07:34] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:47 [I 2022-10-11 04:07:34] ...trap/engines/engine.py.326: process: 0.30702 | load: 0.00019 [I 2022-10-11 04:07:37] ...trap/engines/engine.py.321: eval: epoch 96 | batch 110/510 [I 2022-10-11 04:07:37] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:12 [I 2022-10-11 04:07:37] ...trap/engines/engine.py.326: process: 0.31914 | load: 0.00020 [I 2022-10-11 04:07:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 120/510 [I 2022-10-11 04:07:40] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:04 [I 2022-10-11 04:07:40] ...trap/engines/engine.py.326: process: 0.31428 | load: 0.00018 [I 2022-10-11 04:07:43] ...trap/engines/engine.py.321: eval: epoch 96 | batch 130/510 [I 2022-10-11 04:07:43] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:00 [I 2022-10-11 04:07:43] ...trap/engines/engine.py.326: process: 0.32073 | load: 0.00020 [I 2022-10-11 04:07:46] ...trap/engines/engine.py.321: eval: epoch 96 | batch 140/510 [I 2022-10-11 04:07:46] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:57 [I 2022-10-11 04:07:46] ...trap/engines/engine.py.326: process: 0.31494 | load: 0.00019 [I 2022-10-11 04:07:50] ...trap/engines/engine.py.321: eval: epoch 96 | batch 150/510 [I 2022-10-11 04:07:50] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:54 [I 2022-10-11 04:07:50] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00019 [I 2022-10-11 04:07:53] ...trap/engines/engine.py.321: eval: epoch 96 | batch 160/510 [I 2022-10-11 04:07:53] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:51 [I 2022-10-11 04:07:53] ...trap/engines/engine.py.326: process: 0.32108 | load: 0.00018 [I 2022-10-11 04:07:56] ...trap/engines/engine.py.321: eval: epoch 96 | batch 170/510 [I 2022-10-11 04:07:56] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:47 [I 2022-10-11 04:07:56] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00019 [I 2022-10-11 04:07:59] ...trap/engines/engine.py.321: eval: epoch 96 | batch 180/510 [I 2022-10-11 04:07:59] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:43 [I 2022-10-11 04:07:59] ...trap/engines/engine.py.326: process: 0.31303 | load: 0.00019 [I 2022-10-11 04:08:02] ...trap/engines/engine.py.321: eval: epoch 96 | batch 190/510 [I 2022-10-11 04:08:02] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:41 [I 2022-10-11 04:08:02] ...trap/engines/engine.py.326: process: 0.31611 | load: 0.00020 [I 2022-10-11 04:08:05] ...trap/engines/engine.py.321: eval: epoch 96 | batch 200/510 [I 2022-10-11 04:08:05] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:37 [I 2022-10-11 04:08:05] ...trap/engines/engine.py.326: process: 0.31380 | load: 0.00019 [I 2022-10-11 04:08:09] ...trap/engines/engine.py.321: eval: epoch 96 | batch 210/510 [I 2022-10-11 04:08:09] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:34 [I 2022-10-11 04:08:09] ...trap/engines/engine.py.326: process: 0.31361 | load: 0.00020 [I 2022-10-11 04:08:12] ...trap/engines/engine.py.321: eval: epoch 96 | batch 220/510 [I 2022-10-11 04:08:12] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:31 [I 2022-10-11 04:08:12] ...trap/engines/engine.py.326: process: 0.31280 | load: 0.00019 [I 2022-10-11 04:08:15] ...trap/engines/engine.py.321: eval: epoch 96 | batch 230/510 [I 2022-10-11 04:08:15] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:29 [I 2022-10-11 04:08:15] ...trap/engines/engine.py.326: process: 0.31836 | load: 0.00020 [I 2022-10-11 04:08:18] ...trap/engines/engine.py.321: eval: epoch 96 | batch 240/510 [I 2022-10-11 04:08:18] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:25 [I 2022-10-11 04:08:18] ...trap/engines/engine.py.326: process: 0.31868 | load: 0.00019 [I 2022-10-11 04:08:21] ...trap/engines/engine.py.321: eval: epoch 96 | batch 250/510 [I 2022-10-11 04:08:21] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:22 [I 2022-10-11 04:08:21] ...trap/engines/engine.py.326: process: 0.31835 | load: 0.00020 [I 2022-10-11 04:08:24] ...trap/engines/engine.py.321: eval: epoch 96 | batch 260/510 [I 2022-10-11 04:08:24] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:19 [I 2022-10-11 04:08:24] ...trap/engines/engine.py.326: process: 0.32410 | load: 0.00021 [I 2022-10-11 04:08:28] ...trap/engines/engine.py.321: eval: epoch 96 | batch 270/510 [I 2022-10-11 04:08:28] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:16 [I 2022-10-11 04:08:28] ...trap/engines/engine.py.326: process: 0.31495 | load: 0.00019 [I 2022-10-11 04:08:31] ...trap/engines/engine.py.321: eval: epoch 96 | batch 280/510 [I 2022-10-11 04:08:31] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:13 [I 2022-10-11 04:08:31] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00021 [I 2022-10-11 04:08:34] ...trap/engines/engine.py.321: eval: epoch 96 | batch 290/510 [I 2022-10-11 04:08:34] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:09 [I 2022-10-11 04:08:34] ...trap/engines/engine.py.326: process: 0.31281 | load: 0.00019 [I 2022-10-11 04:08:37] ...trap/engines/engine.py.321: eval: epoch 96 | batch 300/510 [I 2022-10-11 04:08:37] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:06 [I 2022-10-11 04:08:37] ...trap/engines/engine.py.326: process: 0.32020 | load: 0.00019 [I 2022-10-11 04:08:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 310/510 [I 2022-10-11 04:08:40] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:03 [I 2022-10-11 04:08:40] ...trap/engines/engine.py.326: process: 0.31087 | load: 0.00020 [I 2022-10-11 04:08:44] ...trap/engines/engine.py.321: eval: epoch 96 | batch 320/510 [I 2022-10-11 04:08:44] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:00 [I 2022-10-11 04:08:44] ...trap/engines/engine.py.326: process: 0.31594 | load: 0.00020 [I 2022-10-11 04:08:47] ...trap/engines/engine.py.321: eval: epoch 96 | batch 330/510 [I 2022-10-11 04:08:47] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:57 [I 2022-10-11 04:08:47] ...trap/engines/engine.py.326: process: 0.31810 | load: 0.00019 [I 2022-10-11 04:08:50] ...trap/engines/engine.py.321: eval: epoch 96 | batch 340/510 [I 2022-10-11 04:08:50] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:54 [I 2022-10-11 04:08:50] ...trap/engines/engine.py.326: process: 0.31520 | load: 0.00020 [I 2022-10-11 04:08:53] ...trap/engines/engine.py.321: eval: epoch 96 | batch 350/510 [I 2022-10-11 04:08:53] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:51 [I 2022-10-11 04:08:53] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00020 [I 2022-10-11 04:08:56] ...trap/engines/engine.py.321: eval: epoch 96 | batch 360/510 [I 2022-10-11 04:08:56] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:47 [I 2022-10-11 04:08:56] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00019 [I 2022-10-11 04:09:00] ...trap/engines/engine.py.321: eval: epoch 96 | batch 370/510 [I 2022-10-11 04:09:00] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:44 [I 2022-10-11 04:09:00] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00020 [I 2022-10-11 04:09:03] ...trap/engines/engine.py.321: eval: epoch 96 | batch 380/510 [I 2022-10-11 04:09:03] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:41 [I 2022-10-11 04:09:03] ...trap/engines/engine.py.326: process: 0.31783 | load: 0.00020 [I 2022-10-11 04:09:06] ...trap/engines/engine.py.321: eval: epoch 96 | batch 390/510 [I 2022-10-11 04:09:06] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-11 04:09:06] ...trap/engines/engine.py.326: process: 0.31725 | load: 0.00020 [I 2022-10-11 04:09:09] ...trap/engines/engine.py.321: eval: epoch 96 | batch 400/510 [I 2022-10-11 04:09:09] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:34 [I 2022-10-11 04:09:09] ...trap/engines/engine.py.326: process: 0.31263 | load: 0.00020 [I 2022-10-11 04:09:12] ...trap/engines/engine.py.321: eval: epoch 96 | batch 410/510 [I 2022-10-11 04:09:12] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-11 04:09:12] ...trap/engines/engine.py.326: process: 0.31367 | load: 0.00019 [I 2022-10-11 04:09:15] ...trap/engines/engine.py.321: eval: epoch 96 | batch 420/510 [I 2022-10-11 04:09:15] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:28 [I 2022-10-11 04:09:15] ...trap/engines/engine.py.326: process: 0.31202 | load: 0.00019 [I 2022-10-11 04:09:19] ...trap/engines/engine.py.321: eval: epoch 96 | batch 430/510 [I 2022-10-11 04:09:19] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:25 [I 2022-10-11 04:09:19] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00019 [I 2022-10-11 04:09:22] ...trap/engines/engine.py.321: eval: epoch 96 | batch 440/510 [I 2022-10-11 04:09:22] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:22 [I 2022-10-11 04:09:22] ...trap/engines/engine.py.326: process: 0.31332 | load: 0.00020 [I 2022-10-11 04:09:25] ...trap/engines/engine.py.321: eval: epoch 96 | batch 450/510 [I 2022-10-11 04:09:25] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:19 [I 2022-10-11 04:09:25] ...trap/engines/engine.py.326: process: 0.31415 | load: 0.00019 [I 2022-10-11 04:09:28] ...trap/engines/engine.py.321: eval: epoch 96 | batch 460/510 [I 2022-10-11 04:09:28] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:15 [I 2022-10-11 04:09:28] ...trap/engines/engine.py.326: process: 0.31074 | load: 0.00020 [I 2022-10-11 04:09:31] ...trap/engines/engine.py.321: eval: epoch 96 | batch 470/510 [I 2022-10-11 04:09:31] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:12 [I 2022-10-11 04:09:31] ...trap/engines/engine.py.326: process: 0.31518 | load: 0.00019 [I 2022-10-11 04:09:35] ...trap/engines/engine.py.321: eval: epoch 96 | batch 480/510 [I 2022-10-11 04:09:35] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:09 [I 2022-10-11 04:09:35] ...trap/engines/engine.py.326: process: 0.31935 | load: 0.00019 [I 2022-10-11 04:09:38] ...trap/engines/engine.py.321: eval: epoch 96 | batch 490/510 [I 2022-10-11 04:09:38] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:06 [I 2022-10-11 04:09:38] ...trap/engines/engine.py.326: process: 0.32674 | load: 0.00020 [I 2022-10-11 04:09:41] ...trap/engines/engine.py.321: eval: epoch 96 | batch 500/510 [I 2022-10-11 04:09:41] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:03 [I 2022-10-11 04:09:41] ...trap/engines/engine.py.326: process: 0.32023 | load: 0.00020 [I 2022-10-11 04:09:44] ...trap/engines/engine.py.321: eval: epoch 96 | batch 510/510 [I 2022-10-11 04:09:44] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:00 [I 2022-10-11 04:09:44] ...trap/engines/engine.py.326: process: 0.31665 | load: 0.00017 [S 2022-10-11 04:09:45] ...trap/engines/engine.py.132: eval_epoch.epoch: 96 [I 2022-10-11 04:09:45] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.751200 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.912500 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936300 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.744200 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911700 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937000 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.021193 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008065 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007043 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018291 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007976 [S 2022-10-11 04:10:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004561 [I 2022-10-11 04:10:30] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-11 04:10:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 04:10:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 04:10:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 04:10:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 04:10:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 04:10:39] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 04:10:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 04:10:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 04:10:42] ...trap/engines/engine.py.470: Saving model... [I 2022-10-11 04:10:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 04:10:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 04:10:52] ...trap/engines/engine.py.163: Training model on trainset for epoch 97 [I 2022-10-11 04:10:52] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 04:10:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 0/2304 [I 2022-10-11 04:10:55] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:08:37 [I 2022-10-11 04:10:55] ...trap/engines/engine.py.229: process: 3.34982 | load: 2.41771 [I 2022-10-11 04:10:55] ...trap/engines/engine.py.230: loss: 0.14226 [I 2022-10-11 04:10:55] ...optimizers/trijoint.py.151: total_norm: 15.728865 [I 2022-10-11 04:11:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 10/2304 [I 2022-10-11 04:11:04] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:22 [I 2022-10-11 04:11:04] ...trap/engines/engine.py.229: process: 0.87568 | load: 0.00665 [I 2022-10-11 04:11:04] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-11 04:11:04] ...optimizers/trijoint.py.151: total_norm: 11.372201 [I 2022-10-11 04:11:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 20/2304 [I 2022-10-11 04:11:13] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:58 [I 2022-10-11 04:11:13] ...trap/engines/engine.py.229: process: 0.85989 | load: 0.00544 [I 2022-10-11 04:11:13] ...trap/engines/engine.py.230: loss: 0.09774 [I 2022-10-11 04:11:13] ...optimizers/trijoint.py.151: total_norm: 9.841487 [I 2022-10-11 04:11:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 30/2304 [I 2022-10-11 04:11:21] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:48 [I 2022-10-11 04:11:21] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00433 [I 2022-10-11 04:11:21] ...trap/engines/engine.py.230: loss: 0.18897 [I 2022-10-11 04:11:21] ...optimizers/trijoint.py.151: total_norm: 11.485145 [I 2022-10-11 04:11:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 40/2304 [I 2022-10-11 04:11:30] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:45 [I 2022-10-11 04:11:30] ...trap/engines/engine.py.229: process: 0.87728 | load: 0.00405 [I 2022-10-11 04:11:30] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-11 04:11:30] ...optimizers/trijoint.py.151: total_norm: 8.104083 [I 2022-10-11 04:11:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 50/2304 [I 2022-10-11 04:11:39] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:29 [I 2022-10-11 04:11:39] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00407 [I 2022-10-11 04:11:39] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-11 04:11:39] ...optimizers/trijoint.py.151: total_norm: 25.289404 [I 2022-10-11 04:11:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 60/2304 [I 2022-10-11 04:11:47] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:25 [I 2022-10-11 04:11:47] ...trap/engines/engine.py.229: process: 0.87545 | load: 0.00421 [I 2022-10-11 04:11:47] ...trap/engines/engine.py.230: loss: 0.10866 [I 2022-10-11 04:11:47] ...optimizers/trijoint.py.151: total_norm: 8.128033 [I 2022-10-11 04:11:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 70/2304 [I 2022-10-11 04:11:56] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:32:31 [I 2022-10-11 04:11:56] ...trap/engines/engine.py.229: process: 0.87219 | load: 0.00363 [I 2022-10-11 04:11:56] ...trap/engines/engine.py.230: loss: 0.10470 [I 2022-10-11 04:11:56] ...optimizers/trijoint.py.151: total_norm: 9.905895 [I 2022-10-11 04:12:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 80/2304 [I 2022-10-11 04:12:05] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:32:12 [I 2022-10-11 04:12:05] ...trap/engines/engine.py.229: process: 0.86227 | load: 0.00354 [I 2022-10-11 04:12:05] ...trap/engines/engine.py.230: loss: 0.11465 [I 2022-10-11 04:12:05] ...optimizers/trijoint.py.151: total_norm: 11.603593 [I 2022-10-11 04:12:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 90/2304 [I 2022-10-11 04:12:13] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:32:03 [I 2022-10-11 04:12:13] ...trap/engines/engine.py.229: process: 0.86521 | load: 0.00408 [I 2022-10-11 04:12:13] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-11 04:12:13] ...optimizers/trijoint.py.151: total_norm: 20.418295 [I 2022-10-11 04:12:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 100/2304 [I 2022-10-11 04:12:22] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:31:52 [I 2022-10-11 04:12:22] ...trap/engines/engine.py.229: process: 0.86478 | load: 0.00341 [I 2022-10-11 04:12:22] ...trap/engines/engine.py.230: loss: 0.13189 [I 2022-10-11 04:12:22] ...optimizers/trijoint.py.151: total_norm: 10.682788 [I 2022-10-11 04:12:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 110/2304 [I 2022-10-11 04:12:31] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:31:43 [I 2022-10-11 04:12:31] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00363 [I 2022-10-11 04:12:31] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-11 04:12:31] ...optimizers/trijoint.py.151: total_norm: 9.619420 [I 2022-10-11 04:12:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 120/2304 [I 2022-10-11 04:12:40] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:31:32 [I 2022-10-11 04:12:40] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00365 [I 2022-10-11 04:12:40] ...trap/engines/engine.py.230: loss: 0.11864 [I 2022-10-11 04:12:40] ...optimizers/trijoint.py.151: total_norm: 23.500788 [I 2022-10-11 04:12:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 130/2304 [I 2022-10-11 04:12:48] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:31:25 [I 2022-10-11 04:12:48] ...trap/engines/engine.py.229: process: 0.87988 | load: 0.00360 [I 2022-10-11 04:12:48] ...trap/engines/engine.py.230: loss: 0.14574 [I 2022-10-11 04:12:48] ...optimizers/trijoint.py.151: total_norm: 18.637440 [I 2022-10-11 04:12:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 140/2304 [I 2022-10-11 04:12:57] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:31:20 [I 2022-10-11 04:12:57] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00341 [I 2022-10-11 04:12:57] ...trap/engines/engine.py.230: loss: 0.12403 [I 2022-10-11 04:12:57] ...optimizers/trijoint.py.151: total_norm: 33.369236 [I 2022-10-11 04:13:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 150/2304 [I 2022-10-11 04:13:06] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:31:06 [I 2022-10-11 04:13:06] ...trap/engines/engine.py.229: process: 0.86331 | load: 0.00342 [I 2022-10-11 04:13:06] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-11 04:13:06] ...optimizers/trijoint.py.151: total_norm: 20.943838 [I 2022-10-11 04:13:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 160/2304 [I 2022-10-11 04:13:14] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:31:00 [I 2022-10-11 04:13:14] ...trap/engines/engine.py.229: process: 0.87229 | load: 0.00329 [I 2022-10-11 04:13:14] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-11 04:13:14] ...optimizers/trijoint.py.151: total_norm: 13.790658 [I 2022-10-11 04:13:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 170/2304 [I 2022-10-11 04:13:23] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:30:45 [I 2022-10-11 04:13:23] ...trap/engines/engine.py.229: process: 0.87850 | load: 0.00344 [I 2022-10-11 04:13:23] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-11 04:13:23] ...optimizers/trijoint.py.151: total_norm: 20.974400 [I 2022-10-11 04:13:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 180/2304 [I 2022-10-11 04:13:32] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:37 [I 2022-10-11 04:13:32] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00318 [I 2022-10-11 04:13:32] ...trap/engines/engine.py.230: loss: 0.10666 [I 2022-10-11 04:13:32] ...optimizers/trijoint.py.151: total_norm: 15.186754 [I 2022-10-11 04:13:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 190/2304 [I 2022-10-11 04:13:40] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:30:31 [I 2022-10-11 04:13:40] ...trap/engines/engine.py.229: process: 0.87745 | load: 0.00328 [I 2022-10-11 04:13:40] ...trap/engines/engine.py.230: loss: 0.17969 [I 2022-10-11 04:13:40] ...optimizers/trijoint.py.151: total_norm: 13.904910 [I 2022-10-11 04:13:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 200/2304 [I 2022-10-11 04:13:49] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:29:58 [I 2022-10-11 04:13:49] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00308 [I 2022-10-11 04:13:49] ...trap/engines/engine.py.230: loss: 0.09276 [I 2022-10-11 04:13:49] ...optimizers/trijoint.py.151: total_norm: 19.013527 [I 2022-10-11 04:13:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 210/2304 [I 2022-10-11 04:13:57] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:29:40 [I 2022-10-11 04:13:57] ...trap/engines/engine.py.229: process: 0.83142 | load: 0.00321 [I 2022-10-11 04:13:57] ...trap/engines/engine.py.230: loss: 0.09692 [I 2022-10-11 04:13:57] ...optimizers/trijoint.py.151: total_norm: 37.542747 [I 2022-10-11 04:14:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 220/2304 [I 2022-10-11 04:14:06] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:29:38 [I 2022-10-11 04:14:06] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00292 [I 2022-10-11 04:14:06] ...trap/engines/engine.py.230: loss: 0.09071 [I 2022-10-11 04:14:06] ...optimizers/trijoint.py.151: total_norm: 10.590597 [I 2022-10-11 04:14:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 230/2304 [I 2022-10-11 04:14:14] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:36 [I 2022-10-11 04:14:14] ...trap/engines/engine.py.229: process: 0.87081 | load: 0.00317 [I 2022-10-11 04:14:14] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-11 04:14:14] ...optimizers/trijoint.py.151: total_norm: 14.745296 [I 2022-10-11 04:14:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 240/2304 [I 2022-10-11 04:14:23] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:29:24 [I 2022-10-11 04:14:23] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00313 [I 2022-10-11 04:14:23] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-11 04:14:23] ...optimizers/trijoint.py.151: total_norm: 8.475004 [I 2022-10-11 04:14:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 250/2304 [I 2022-10-11 04:14:32] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:17 [I 2022-10-11 04:14:32] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00297 [I 2022-10-11 04:14:32] ...trap/engines/engine.py.230: loss: 0.19669 [I 2022-10-11 04:14:32] ...optimizers/trijoint.py.151: total_norm: 38.585670 [I 2022-10-11 04:14:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 260/2304 [I 2022-10-11 04:14:40] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:22 [I 2022-10-11 04:14:40] ...trap/engines/engine.py.229: process: 0.87550 | load: 0.00305 [I 2022-10-11 04:14:40] ...trap/engines/engine.py.230: loss: 0.09210 [I 2022-10-11 04:14:40] ...optimizers/trijoint.py.151: total_norm: 15.463371 [I 2022-10-11 04:14:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 270/2304 [I 2022-10-11 04:14:49] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:28:54 [I 2022-10-11 04:14:49] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00311 [I 2022-10-11 04:14:49] ...trap/engines/engine.py.230: loss: 0.16688 [I 2022-10-11 04:14:49] ...optimizers/trijoint.py.151: total_norm: 15.540807 [I 2022-10-11 04:14:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 280/2304 [I 2022-10-11 04:14:57] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:28:45 [I 2022-10-11 04:14:57] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00328 [I 2022-10-11 04:14:57] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-11 04:14:57] ...optimizers/trijoint.py.151: total_norm: 14.097631 [I 2022-10-11 04:15:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 290/2304 [I 2022-10-11 04:15:06] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:28:42 [I 2022-10-11 04:15:06] ...trap/engines/engine.py.229: process: 0.86033 | load: 0.00306 [I 2022-10-11 04:15:06] ...trap/engines/engine.py.230: loss: 0.14346 [I 2022-10-11 04:15:06] ...optimizers/trijoint.py.151: total_norm: 13.634646 [I 2022-10-11 04:15:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 300/2304 [I 2022-10-11 04:15:14] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:23 [I 2022-10-11 04:15:14] ...trap/engines/engine.py.229: process: 0.84288 | load: 0.00307 [I 2022-10-11 04:15:14] ...trap/engines/engine.py.230: loss: 0.09964 [I 2022-10-11 04:15:14] ...optimizers/trijoint.py.151: total_norm: 12.369668 [I 2022-10-11 04:15:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 310/2304 [I 2022-10-11 04:15:23] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:17 [I 2022-10-11 04:15:23] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00295 [I 2022-10-11 04:15:23] ...trap/engines/engine.py.230: loss: 0.16817 [I 2022-10-11 04:15:23] ...optimizers/trijoint.py.151: total_norm: 9.146976 [I 2022-10-11 04:15:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 320/2304 [I 2022-10-11 04:15:32] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:16 [I 2022-10-11 04:15:32] ...trap/engines/engine.py.229: process: 0.87274 | load: 0.00319 [I 2022-10-11 04:15:32] ...trap/engines/engine.py.230: loss: 0.11336 [I 2022-10-11 04:15:32] ...optimizers/trijoint.py.151: total_norm: 14.206994 [I 2022-10-11 04:15:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 330/2304 [I 2022-10-11 04:15:40] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:01 [I 2022-10-11 04:15:40] ...trap/engines/engine.py.229: process: 0.84467 | load: 0.00293 [I 2022-10-11 04:15:40] ...trap/engines/engine.py.230: loss: 0.09642 [I 2022-10-11 04:15:40] ...optimizers/trijoint.py.151: total_norm: 10.778093 [I 2022-10-11 04:15:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 340/2304 [I 2022-10-11 04:15:49] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:28:00 [I 2022-10-11 04:15:49] ...trap/engines/engine.py.229: process: 0.85192 | load: 0.00298 [I 2022-10-11 04:15:49] ...trap/engines/engine.py.230: loss: 0.08612 [I 2022-10-11 04:15:49] ...optimizers/trijoint.py.151: total_norm: 29.570290 [I 2022-10-11 04:15:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 350/2304 [I 2022-10-11 04:15:57] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:52 [I 2022-10-11 04:15:57] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00307 [I 2022-10-11 04:15:57] ...trap/engines/engine.py.230: loss: 0.11595 [I 2022-10-11 04:15:57] ...optimizers/trijoint.py.151: total_norm: 10.593837 [I 2022-10-11 04:16:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 360/2304 [I 2022-10-11 04:16:06] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:28:01 [I 2022-10-11 04:16:06] ...trap/engines/engine.py.229: process: 0.87005 | load: 0.00307 [I 2022-10-11 04:16:06] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-11 04:16:06] ...optimizers/trijoint.py.151: total_norm: 8.502848 [I 2022-10-11 04:16:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 370/2304 [I 2022-10-11 04:16:15] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:57 [I 2022-10-11 04:16:15] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00310 [I 2022-10-11 04:16:15] ...trap/engines/engine.py.230: loss: 0.16585 [I 2022-10-11 04:16:15] ...optimizers/trijoint.py.151: total_norm: 18.112432 [I 2022-10-11 04:16:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 380/2304 [I 2022-10-11 04:16:23] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:41 [I 2022-10-11 04:16:23] ...trap/engines/engine.py.229: process: 0.86529 | load: 0.00309 [I 2022-10-11 04:16:23] ...trap/engines/engine.py.230: loss: 0.08561 [I 2022-10-11 04:16:23] ...optimizers/trijoint.py.151: total_norm: 10.912197 [I 2022-10-11 04:16:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 390/2304 [I 2022-10-11 04:16:32] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:51 [I 2022-10-11 04:16:32] ...trap/engines/engine.py.229: process: 0.88620 | load: 0.00339 [I 2022-10-11 04:16:32] ...trap/engines/engine.py.230: loss: 0.15113 [I 2022-10-11 04:16:32] ...optimizers/trijoint.py.151: total_norm: 14.255227 [I 2022-10-11 04:16:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 400/2304 [I 2022-10-11 04:16:41] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:34 [I 2022-10-11 04:16:41] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00326 [I 2022-10-11 04:16:41] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-11 04:16:41] ...optimizers/trijoint.py.151: total_norm: 11.493210 [I 2022-10-11 04:16:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 410/2304 [I 2022-10-11 04:16:49] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:30 [I 2022-10-11 04:16:49] ...trap/engines/engine.py.229: process: 0.87924 | load: 0.00312 [I 2022-10-11 04:16:49] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-11 04:16:49] ...optimizers/trijoint.py.151: total_norm: 14.816089 [I 2022-10-11 04:16:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 420/2304 [I 2022-10-11 04:16:58] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:27:12 [I 2022-10-11 04:16:58] ...trap/engines/engine.py.229: process: 0.87746 | load: 0.00322 [I 2022-10-11 04:16:58] ...trap/engines/engine.py.230: loss: 0.09270 [I 2022-10-11 04:16:58] ...optimizers/trijoint.py.151: total_norm: 19.110025 [I 2022-10-11 04:17:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 430/2304 [I 2022-10-11 04:17:07] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:27:13 [I 2022-10-11 04:17:07] ...trap/engines/engine.py.229: process: 0.87691 | load: 0.00308 [I 2022-10-11 04:17:07] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-11 04:17:07] ...optimizers/trijoint.py.151: total_norm: 12.581419 [I 2022-10-11 04:17:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 440/2304 [I 2022-10-11 04:17:15] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:53 [I 2022-10-11 04:17:15] ...trap/engines/engine.py.229: process: 0.85864 | load: 0.00314 [I 2022-10-11 04:17:15] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-11 04:17:15] ...optimizers/trijoint.py.151: total_norm: 22.321615 [I 2022-10-11 04:17:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 450/2304 [I 2022-10-11 04:17:24] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:42 [I 2022-10-11 04:17:24] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00320 [I 2022-10-11 04:17:24] ...trap/engines/engine.py.230: loss: 0.09837 [I 2022-10-11 04:17:24] ...optimizers/trijoint.py.151: total_norm: 12.059978 [I 2022-10-11 04:17:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 460/2304 [I 2022-10-11 04:17:33] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:31 [I 2022-10-11 04:17:33] ...trap/engines/engine.py.229: process: 0.86274 | load: 0.00322 [I 2022-10-11 04:17:33] ...trap/engines/engine.py.230: loss: 0.24727 [I 2022-10-11 04:17:33] ...optimizers/trijoint.py.151: total_norm: 38.413967 [I 2022-10-11 04:17:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 470/2304 [I 2022-10-11 04:17:41] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:26 [I 2022-10-11 04:17:41] ...trap/engines/engine.py.229: process: 0.87017 | load: 0.00314 [I 2022-10-11 04:17:41] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-11 04:17:41] ...optimizers/trijoint.py.151: total_norm: 14.071419 [I 2022-10-11 04:17:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 480/2304 [I 2022-10-11 04:17:50] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:26:16 [I 2022-10-11 04:17:50] ...trap/engines/engine.py.229: process: 0.88116 | load: 0.00329 [I 2022-10-11 04:17:50] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-11 04:17:50] ...optimizers/trijoint.py.151: total_norm: 28.669123 [I 2022-10-11 04:17:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 490/2304 [I 2022-10-11 04:17:59] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:26:17 [I 2022-10-11 04:17:59] ...trap/engines/engine.py.229: process: 0.88573 | load: 0.00328 [I 2022-10-11 04:17:59] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-11 04:17:59] ...optimizers/trijoint.py.151: total_norm: 33.143856 [I 2022-10-11 04:18:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 500/2304 [I 2022-10-11 04:18:07] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:26:14 [I 2022-10-11 04:18:07] ...trap/engines/engine.py.229: process: 0.87753 | load: 0.00319 [I 2022-10-11 04:18:07] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-11 04:18:07] ...optimizers/trijoint.py.151: total_norm: 14.736133 [I 2022-10-11 04:18:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 510/2304 [I 2022-10-11 04:18:16] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:25:59 [I 2022-10-11 04:18:16] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00314 [I 2022-10-11 04:18:16] ...trap/engines/engine.py.230: loss: 0.14683 [I 2022-10-11 04:18:16] ...optimizers/trijoint.py.151: total_norm: 53.345509 [I 2022-10-11 04:18:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 520/2304 [I 2022-10-11 04:18:25] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:50 [I 2022-10-11 04:18:25] ...trap/engines/engine.py.229: process: 0.86306 | load: 0.00314 [I 2022-10-11 04:18:25] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-11 04:18:25] ...optimizers/trijoint.py.151: total_norm: 11.571765 [I 2022-10-11 04:18:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 530/2304 [I 2022-10-11 04:18:34] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:25:40 [I 2022-10-11 04:18:34] ...trap/engines/engine.py.229: process: 0.86776 | load: 0.00318 [I 2022-10-11 04:18:34] ...trap/engines/engine.py.230: loss: 0.08662 [I 2022-10-11 04:18:34] ...optimizers/trijoint.py.151: total_norm: 10.585173 [I 2022-10-11 04:18:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 540/2304 [I 2022-10-11 04:18:42] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:25:41 [I 2022-10-11 04:18:42] ...trap/engines/engine.py.229: process: 0.87236 | load: 0.00310 [I 2022-10-11 04:18:42] ...trap/engines/engine.py.230: loss: 0.14890 [I 2022-10-11 04:18:42] ...optimizers/trijoint.py.151: total_norm: 14.773719 [I 2022-10-11 04:18:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 550/2304 [I 2022-10-11 04:18:51] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:25:12 [I 2022-10-11 04:18:51] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00312 [I 2022-10-11 04:18:51] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-11 04:18:51] ...optimizers/trijoint.py.151: total_norm: 63.345436 [I 2022-10-11 04:19:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 560/2304 [I 2022-10-11 04:19:00] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:25:06 [I 2022-10-11 04:19:00] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00317 [I 2022-10-11 04:19:00] ...trap/engines/engine.py.230: loss: 0.07036 [I 2022-10-11 04:19:00] ...optimizers/trijoint.py.151: total_norm: 8.760483 [I 2022-10-11 04:19:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 570/2304 [I 2022-10-11 04:19:08] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:51 [I 2022-10-11 04:19:08] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00330 [I 2022-10-11 04:19:08] ...trap/engines/engine.py.230: loss: 0.09102 [I 2022-10-11 04:19:08] ...optimizers/trijoint.py.151: total_norm: 14.851172 [I 2022-10-11 04:19:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 580/2304 [I 2022-10-11 04:19:17] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:52 [I 2022-10-11 04:19:17] ...trap/engines/engine.py.229: process: 0.87482 | load: 0.00306 [I 2022-10-11 04:19:17] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-11 04:19:17] ...optimizers/trijoint.py.151: total_norm: 11.996097 [I 2022-10-11 04:19:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 590/2304 [I 2022-10-11 04:19:26] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:42 [I 2022-10-11 04:19:26] ...trap/engines/engine.py.229: process: 0.87026 | load: 0.00314 [I 2022-10-11 04:19:26] ...trap/engines/engine.py.230: loss: 0.12222 [I 2022-10-11 04:19:26] ...optimizers/trijoint.py.151: total_norm: 15.907275 [I 2022-10-11 04:19:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 600/2304 [I 2022-10-11 04:19:34] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:28 [I 2022-10-11 04:19:34] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00315 [I 2022-10-11 04:19:34] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-11 04:19:34] ...optimizers/trijoint.py.151: total_norm: 13.557629 [I 2022-10-11 04:19:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 610/2304 [I 2022-10-11 04:19:43] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:29 [I 2022-10-11 04:19:43] ...trap/engines/engine.py.229: process: 0.87382 | load: 0.00308 [I 2022-10-11 04:19:43] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-11 04:19:43] ...optimizers/trijoint.py.151: total_norm: 11.298431 [I 2022-10-11 04:19:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 620/2304 [I 2022-10-11 04:19:52] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:24:33 [I 2022-10-11 04:19:52] ...trap/engines/engine.py.229: process: 0.88326 | load: 0.00312 [I 2022-10-11 04:19:52] ...trap/engines/engine.py.230: loss: 0.09880 [I 2022-10-11 04:19:52] ...optimizers/trijoint.py.151: total_norm: 18.865171 [I 2022-10-11 04:20:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 630/2304 [I 2022-10-11 04:20:00] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:58 [I 2022-10-11 04:20:00] ...trap/engines/engine.py.229: process: 0.85672 | load: 0.00310 [I 2022-10-11 04:20:00] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-11 04:20:00] ...optimizers/trijoint.py.151: total_norm: 36.135216 [I 2022-10-11 04:20:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 640/2304 [I 2022-10-11 04:20:09] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:36 [I 2022-10-11 04:20:09] ...trap/engines/engine.py.229: process: 0.86049 | load: 0.00300 [I 2022-10-11 04:20:09] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-11 04:20:09] ...optimizers/trijoint.py.151: total_norm: 9.811584 [I 2022-10-11 04:20:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 650/2304 [I 2022-10-11 04:20:18] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:37 [I 2022-10-11 04:20:18] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00296 [I 2022-10-11 04:20:18] ...trap/engines/engine.py.230: loss: 0.11283 [I 2022-10-11 04:20:18] ...optimizers/trijoint.py.151: total_norm: 8.079685 [I 2022-10-11 04:20:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 660/2304 [I 2022-10-11 04:20:26] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:20 [I 2022-10-11 04:20:26] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00295 [I 2022-10-11 04:20:26] ...trap/engines/engine.py.230: loss: 0.18286 [I 2022-10-11 04:20:26] ...optimizers/trijoint.py.151: total_norm: 11.556670 [I 2022-10-11 04:20:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 670/2304 [I 2022-10-11 04:20:35] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:24 [I 2022-10-11 04:20:35] ...trap/engines/engine.py.229: process: 0.87008 | load: 0.00300 [I 2022-10-11 04:20:35] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-11 04:20:35] ...optimizers/trijoint.py.151: total_norm: 13.592698 [I 2022-10-11 04:20:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 680/2304 [I 2022-10-11 04:20:43] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:23:01 [I 2022-10-11 04:20:43] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00292 [I 2022-10-11 04:20:43] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-11 04:20:43] ...optimizers/trijoint.py.151: total_norm: 16.037962 [I 2022-10-11 04:20:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 690/2304 [I 2022-10-11 04:20:52] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:54 [I 2022-10-11 04:20:52] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00293 [I 2022-10-11 04:20:52] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-11 04:20:52] ...optimizers/trijoint.py.151: total_norm: 19.791208 [I 2022-10-11 04:21:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 700/2304 [I 2022-10-11 04:21:00] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:48 [I 2022-10-11 04:21:00] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00293 [I 2022-10-11 04:21:00] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-11 04:21:00] ...optimizers/trijoint.py.151: total_norm: 9.071585 [I 2022-10-11 04:21:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 710/2304 [I 2022-10-11 04:21:09] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:36 [I 2022-10-11 04:21:09] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00297 [I 2022-10-11 04:21:09] ...trap/engines/engine.py.230: loss: 0.18353 [I 2022-10-11 04:21:09] ...optimizers/trijoint.py.151: total_norm: 23.055260 [I 2022-10-11 04:21:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 720/2304 [I 2022-10-11 04:21:17] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:41 [I 2022-10-11 04:21:17] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00295 [I 2022-10-11 04:21:17] ...trap/engines/engine.py.230: loss: 0.14737 [I 2022-10-11 04:21:17] ...optimizers/trijoint.py.151: total_norm: 19.403223 [I 2022-10-11 04:21:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 730/2304 [I 2022-10-11 04:21:26] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:18 [I 2022-10-11 04:21:26] ...trap/engines/engine.py.229: process: 0.85446 | load: 0.00298 [I 2022-10-11 04:21:26] ...trap/engines/engine.py.230: loss: 0.09712 [I 2022-10-11 04:21:26] ...optimizers/trijoint.py.151: total_norm: 15.664648 [I 2022-10-11 04:21:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 740/2304 [I 2022-10-11 04:21:34] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:11 [I 2022-10-11 04:21:34] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00305 [I 2022-10-11 04:21:34] ...trap/engines/engine.py.230: loss: 0.20407 [I 2022-10-11 04:21:34] ...optimizers/trijoint.py.151: total_norm: 59.175270 [I 2022-10-11 04:21:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 750/2304 [I 2022-10-11 04:21:43] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:01 [I 2022-10-11 04:21:43] ...trap/engines/engine.py.229: process: 0.83705 | load: 0.00301 [I 2022-10-11 04:21:43] ...trap/engines/engine.py.230: loss: 0.10252 [I 2022-10-11 04:21:43] ...optimizers/trijoint.py.151: total_norm: 8.558947 [I 2022-10-11 04:21:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 760/2304 [I 2022-10-11 04:21:52] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:03 [I 2022-10-11 04:21:52] ...trap/engines/engine.py.229: process: 0.86103 | load: 0.00307 [I 2022-10-11 04:21:52] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-11 04:21:52] ...optimizers/trijoint.py.151: total_norm: 15.599707 [I 2022-10-11 04:22:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 770/2304 [I 2022-10-11 04:22:00] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:58 [I 2022-10-11 04:22:00] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00296 [I 2022-10-11 04:22:00] ...trap/engines/engine.py.230: loss: 0.09299 [I 2022-10-11 04:22:00] ...optimizers/trijoint.py.151: total_norm: 12.484344 [I 2022-10-11 04:22:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 780/2304 [I 2022-10-11 04:22:09] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:46 [I 2022-10-11 04:22:09] ...trap/engines/engine.py.229: process: 0.85626 | load: 0.00296 [I 2022-10-11 04:22:09] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-11 04:22:09] ...optimizers/trijoint.py.151: total_norm: 11.050829 [I 2022-10-11 04:22:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 790/2304 [I 2022-10-11 04:22:17] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:33 [I 2022-10-11 04:22:17] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00294 [I 2022-10-11 04:22:17] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-11 04:22:17] ...optimizers/trijoint.py.151: total_norm: 11.757618 [I 2022-10-11 04:22:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 800/2304 [I 2022-10-11 04:22:26] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:25 [I 2022-10-11 04:22:26] ...trap/engines/engine.py.229: process: 0.86645 | load: 0.00294 [I 2022-10-11 04:22:26] ...trap/engines/engine.py.230: loss: 0.08814 [I 2022-10-11 04:22:26] ...optimizers/trijoint.py.151: total_norm: 11.920846 [I 2022-10-11 04:22:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 810/2304 [I 2022-10-11 04:22:35] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:23 [I 2022-10-11 04:22:35] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00303 [I 2022-10-11 04:22:35] ...trap/engines/engine.py.230: loss: 0.19945 [I 2022-10-11 04:22:35] ...optimizers/trijoint.py.151: total_norm: 22.357889 [I 2022-10-11 04:22:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 820/2304 [I 2022-10-11 04:22:43] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:07 [I 2022-10-11 04:22:43] ...trap/engines/engine.py.229: process: 0.85534 | load: 0.00295 [I 2022-10-11 04:22:43] ...trap/engines/engine.py.230: loss: 0.15655 [I 2022-10-11 04:22:43] ...optimizers/trijoint.py.151: total_norm: 42.392483 [I 2022-10-11 04:22:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 830/2304 [I 2022-10-11 04:22:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:08 [I 2022-10-11 04:22:52] ...trap/engines/engine.py.229: process: 0.87878 | load: 0.00295 [I 2022-10-11 04:22:52] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-11 04:22:52] ...optimizers/trijoint.py.151: total_norm: 17.523790 [I 2022-10-11 04:23:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 840/2304 [I 2022-10-11 04:23:00] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:04 [I 2022-10-11 04:23:00] ...trap/engines/engine.py.229: process: 0.88144 | load: 0.00324 [I 2022-10-11 04:23:00] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-11 04:23:00] ...optimizers/trijoint.py.151: total_norm: 16.752970 [I 2022-10-11 04:23:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 850/2304 [I 2022-10-11 04:23:09] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:54 [I 2022-10-11 04:23:09] ...trap/engines/engine.py.229: process: 0.86878 | load: 0.00293 [I 2022-10-11 04:23:09] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-11 04:23:09] ...optimizers/trijoint.py.151: total_norm: 19.054228 [I 2022-10-11 04:23:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 860/2304 [I 2022-10-11 04:23:18] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:32 [I 2022-10-11 04:23:18] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00294 [I 2022-10-11 04:23:18] ...trap/engines/engine.py.230: loss: 0.09905 [I 2022-10-11 04:23:18] ...optimizers/trijoint.py.151: total_norm: 17.217735 [I 2022-10-11 04:23:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 870/2304 [I 2022-10-11 04:23:26] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:24 [I 2022-10-11 04:23:26] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00305 [I 2022-10-11 04:23:26] ...trap/engines/engine.py.230: loss: 0.08670 [I 2022-10-11 04:23:26] ...optimizers/trijoint.py.151: total_norm: 14.230959 [I 2022-10-11 04:23:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 880/2304 [I 2022-10-11 04:23:35] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:14 [I 2022-10-11 04:23:35] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00307 [I 2022-10-11 04:23:35] ...trap/engines/engine.py.230: loss: 0.10818 [I 2022-10-11 04:23:35] ...optimizers/trijoint.py.151: total_norm: 8.889289 [I 2022-10-11 04:23:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 890/2304 [I 2022-10-11 04:23:43] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:15 [I 2022-10-11 04:23:43] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00306 [I 2022-10-11 04:23:43] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-11 04:23:43] ...optimizers/trijoint.py.151: total_norm: 30.309317 [I 2022-10-11 04:23:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 900/2304 [I 2022-10-11 04:23:52] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:10 [I 2022-10-11 04:23:52] ...trap/engines/engine.py.229: process: 0.87431 | load: 0.00300 [I 2022-10-11 04:23:52] ...trap/engines/engine.py.230: loss: 0.14703 [I 2022-10-11 04:23:52] ...optimizers/trijoint.py.151: total_norm: 23.470638 [I 2022-10-11 04:24:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 910/2304 [I 2022-10-11 04:24:01] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:57 [I 2022-10-11 04:24:01] ...trap/engines/engine.py.229: process: 0.86068 | load: 0.00296 [I 2022-10-11 04:24:01] ...trap/engines/engine.py.230: loss: 0.21017 [I 2022-10-11 04:24:01] ...optimizers/trijoint.py.151: total_norm: 19.290268 [I 2022-10-11 04:24:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 920/2304 [I 2022-10-11 04:24:09] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:34 [I 2022-10-11 04:24:09] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00301 [I 2022-10-11 04:24:09] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-11 04:24:09] ...optimizers/trijoint.py.151: total_norm: 17.067175 [I 2022-10-11 04:24:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 930/2304 [I 2022-10-11 04:24:18] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:31 [I 2022-10-11 04:24:18] ...trap/engines/engine.py.229: process: 0.85401 | load: 0.00297 [I 2022-10-11 04:24:18] ...trap/engines/engine.py.230: loss: 0.11695 [I 2022-10-11 04:24:18] ...optimizers/trijoint.py.151: total_norm: 12.517389 [I 2022-10-11 04:24:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 940/2304 [I 2022-10-11 04:24:26] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:25 [I 2022-10-11 04:24:26] ...trap/engines/engine.py.229: process: 0.86415 | load: 0.00301 [I 2022-10-11 04:24:26] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-11 04:24:26] ...optimizers/trijoint.py.151: total_norm: 15.269912 [I 2022-10-11 04:24:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 950/2304 [I 2022-10-11 04:24:35] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:15 [I 2022-10-11 04:24:35] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00295 [I 2022-10-11 04:24:35] ...trap/engines/engine.py.230: loss: 0.12509 [I 2022-10-11 04:24:35] ...optimizers/trijoint.py.151: total_norm: 13.585245 [I 2022-10-11 04:24:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 960/2304 [I 2022-10-11 04:24:43] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:19:08 [I 2022-10-11 04:24:43] ...trap/engines/engine.py.229: process: 0.85450 | load: 0.00304 [I 2022-10-11 04:24:43] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-11 04:24:43] ...optimizers/trijoint.py.151: total_norm: 11.215226 [I 2022-10-11 04:24:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 970/2304 [I 2022-10-11 04:24:52] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:59 [I 2022-10-11 04:24:52] ...trap/engines/engine.py.229: process: 0.84520 | load: 0.00294 [I 2022-10-11 04:24:52] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-11 04:24:52] ...optimizers/trijoint.py.151: total_norm: 19.938091 [I 2022-10-11 04:25:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 980/2304 [I 2022-10-11 04:25:01] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:53 [I 2022-10-11 04:25:01] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00295 [I 2022-10-11 04:25:01] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-11 04:25:01] ...optimizers/trijoint.py.151: total_norm: 14.315677 [I 2022-10-11 04:25:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 990/2304 [I 2022-10-11 04:25:09] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:50 [I 2022-10-11 04:25:09] ...trap/engines/engine.py.229: process: 0.86212 | load: 0.00296 [I 2022-10-11 04:25:09] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-11 04:25:09] ...optimizers/trijoint.py.151: total_norm: 10.655341 [I 2022-10-11 04:25:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 1000/2304 [I 2022-10-11 04:25:18] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:42 [I 2022-10-11 04:25:18] ...trap/engines/engine.py.229: process: 0.86039 | load: 0.00311 [I 2022-10-11 04:25:18] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-11 04:25:18] ...optimizers/trijoint.py.151: total_norm: 12.017719 [I 2022-10-11 04:25:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1010/2304 [I 2022-10-11 04:25:26] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:41 [I 2022-10-11 04:25:26] ...trap/engines/engine.py.229: process: 0.88428 | load: 0.00310 [I 2022-10-11 04:25:26] ...trap/engines/engine.py.230: loss: 0.11085 [I 2022-10-11 04:25:26] ...optimizers/trijoint.py.151: total_norm: 16.735834 [I 2022-10-11 04:25:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 1020/2304 [I 2022-10-11 04:25:35] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:31 [I 2022-10-11 04:25:35] ...trap/engines/engine.py.229: process: 0.86981 | load: 0.00313 [I 2022-10-11 04:25:35] ...trap/engines/engine.py.230: loss: 0.12853 [I 2022-10-11 04:25:35] ...optimizers/trijoint.py.151: total_norm: 21.038248 [I 2022-10-11 04:25:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 1030/2304 [I 2022-10-11 04:25:44] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:18:17 [I 2022-10-11 04:25:44] ...trap/engines/engine.py.229: process: 0.85889 | load: 0.00308 [I 2022-10-11 04:25:44] ...trap/engines/engine.py.230: loss: 0.10836 [I 2022-10-11 04:25:44] ...optimizers/trijoint.py.151: total_norm: 13.617848 [I 2022-10-11 04:25:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 1040/2304 [I 2022-10-11 04:25:52] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:18:11 [I 2022-10-11 04:25:52] ...trap/engines/engine.py.229: process: 0.87460 | load: 0.00321 [I 2022-10-11 04:25:52] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-11 04:25:52] ...optimizers/trijoint.py.151: total_norm: 11.700398 [I 2022-10-11 04:26:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 1050/2304 [I 2022-10-11 04:26:01] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:18:02 [I 2022-10-11 04:26:01] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00325 [I 2022-10-11 04:26:01] ...trap/engines/engine.py.230: loss: 0.09428 [I 2022-10-11 04:26:01] ...optimizers/trijoint.py.151: total_norm: 13.705643 [I 2022-10-11 04:26:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 1060/2304 [I 2022-10-11 04:26:10] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:58 [I 2022-10-11 04:26:10] ...trap/engines/engine.py.229: process: 0.86930 | load: 0.00321 [I 2022-10-11 04:26:10] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-11 04:26:10] ...optimizers/trijoint.py.151: total_norm: 15.232985 [I 2022-10-11 04:26:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 1070/2304 [I 2022-10-11 04:26:18] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:47 [I 2022-10-11 04:26:18] ...trap/engines/engine.py.229: process: 0.87486 | load: 0.00311 [I 2022-10-11 04:26:18] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-11 04:26:18] ...optimizers/trijoint.py.151: total_norm: 11.424847 [I 2022-10-11 04:26:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 1080/2304 [I 2022-10-11 04:26:27] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:40 [I 2022-10-11 04:26:27] ...trap/engines/engine.py.229: process: 0.87151 | load: 0.00321 [I 2022-10-11 04:26:27] ...trap/engines/engine.py.230: loss: 0.11752 [I 2022-10-11 04:26:27] ...optimizers/trijoint.py.151: total_norm: 19.208939 [I 2022-10-11 04:26:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 1090/2304 [I 2022-10-11 04:26:36] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:29 [I 2022-10-11 04:26:36] ...trap/engines/engine.py.229: process: 0.86160 | load: 0.00305 [I 2022-10-11 04:26:36] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-11 04:26:36] ...optimizers/trijoint.py.151: total_norm: 12.832914 [I 2022-10-11 04:26:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 1100/2304 [I 2022-10-11 04:26:44] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:13 [I 2022-10-11 04:26:44] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00309 [I 2022-10-11 04:26:44] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-11 04:26:44] ...optimizers/trijoint.py.151: total_norm: 18.766102 [I 2022-10-11 04:26:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 1110/2304 [I 2022-10-11 04:26:53] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:17:09 [I 2022-10-11 04:26:53] ...trap/engines/engine.py.229: process: 0.87182 | load: 0.00310 [I 2022-10-11 04:26:53] ...trap/engines/engine.py.230: loss: 0.08904 [I 2022-10-11 04:26:53] ...optimizers/trijoint.py.151: total_norm: 12.459641 [I 2022-10-11 04:27:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 1120/2304 [I 2022-10-11 04:27:02] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:57 [I 2022-10-11 04:27:02] ...trap/engines/engine.py.229: process: 0.86703 | load: 0.00311 [I 2022-10-11 04:27:02] ...trap/engines/engine.py.230: loss: 0.07615 [I 2022-10-11 04:27:02] ...optimizers/trijoint.py.151: total_norm: 16.214094 [I 2022-10-11 04:27:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 1130/2304 [I 2022-10-11 04:27:10] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:55 [I 2022-10-11 04:27:10] ...trap/engines/engine.py.229: process: 0.85667 | load: 0.00315 [I 2022-10-11 04:27:10] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-11 04:27:10] ...optimizers/trijoint.py.151: total_norm: 20.540363 [I 2022-10-11 04:27:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 1140/2304 [I 2022-10-11 04:27:19] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:47 [I 2022-10-11 04:27:19] ...trap/engines/engine.py.229: process: 0.87768 | load: 0.00316 [I 2022-10-11 04:27:19] ...trap/engines/engine.py.230: loss: 0.20640 [I 2022-10-11 04:27:19] ...optimizers/trijoint.py.151: total_norm: 32.569809 [I 2022-10-11 04:27:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 1150/2304 [I 2022-10-11 04:27:28] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:33 [I 2022-10-11 04:27:28] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00317 [I 2022-10-11 04:27:28] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-11 04:27:28] ...optimizers/trijoint.py.151: total_norm: 33.796646 [I 2022-10-11 04:27:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 1160/2304 [I 2022-10-11 04:27:36] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:18 [I 2022-10-11 04:27:36] ...trap/engines/engine.py.229: process: 0.86066 | load: 0.00313 [I 2022-10-11 04:27:36] ...trap/engines/engine.py.230: loss: 0.11009 [I 2022-10-11 04:27:36] ...optimizers/trijoint.py.151: total_norm: 11.641001 [I 2022-10-11 04:27:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 1170/2304 [I 2022-10-11 04:27:45] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:27 [I 2022-10-11 04:27:45] ...trap/engines/engine.py.229: process: 0.86645 | load: 0.00313 [I 2022-10-11 04:27:45] ...trap/engines/engine.py.230: loss: 0.08336 [I 2022-10-11 04:27:45] ...optimizers/trijoint.py.151: total_norm: 8.088580 [I 2022-10-11 04:27:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 1180/2304 [I 2022-10-11 04:27:54] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:13 [I 2022-10-11 04:27:54] ...trap/engines/engine.py.229: process: 0.86281 | load: 0.00299 [I 2022-10-11 04:27:54] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-11 04:27:54] ...optimizers/trijoint.py.151: total_norm: 18.180696 [I 2022-10-11 04:28:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 1190/2304 [I 2022-10-11 04:28:02] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:16:01 [I 2022-10-11 04:28:02] ...trap/engines/engine.py.229: process: 0.85637 | load: 0.00308 [I 2022-10-11 04:28:02] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-11 04:28:02] ...optimizers/trijoint.py.151: total_norm: 16.144836 [I 2022-10-11 04:28:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 1200/2304 [I 2022-10-11 04:28:11] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:50 [I 2022-10-11 04:28:11] ...trap/engines/engine.py.229: process: 0.83434 | load: 0.00311 [I 2022-10-11 04:28:11] ...trap/engines/engine.py.230: loss: 0.17746 [I 2022-10-11 04:28:11] ...optimizers/trijoint.py.151: total_norm: 21.075418 [I 2022-10-11 04:28:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 1210/2304 [I 2022-10-11 04:28:20] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:44 [I 2022-10-11 04:28:20] ...trap/engines/engine.py.229: process: 0.86942 | load: 0.00307 [I 2022-10-11 04:28:20] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-11 04:28:20] ...optimizers/trijoint.py.151: total_norm: 38.771671 [I 2022-10-11 04:28:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 1220/2304 [I 2022-10-11 04:28:28] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:31 [I 2022-10-11 04:28:28] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00306 [I 2022-10-11 04:28:28] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-11 04:28:28] ...optimizers/trijoint.py.151: total_norm: 13.460150 [I 2022-10-11 04:28:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 1230/2304 [I 2022-10-11 04:28:37] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:26 [I 2022-10-11 04:28:37] ...trap/engines/engine.py.229: process: 0.86379 | load: 0.00308 [I 2022-10-11 04:28:37] ...trap/engines/engine.py.230: loss: 0.08579 [I 2022-10-11 04:28:37] ...optimizers/trijoint.py.151: total_norm: 9.865964 [I 2022-10-11 04:28:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 1240/2304 [I 2022-10-11 04:28:45] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:13 [I 2022-10-11 04:28:45] ...trap/engines/engine.py.229: process: 0.86216 | load: 0.00311 [I 2022-10-11 04:28:45] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-11 04:28:45] ...optimizers/trijoint.py.151: total_norm: 12.044395 [I 2022-10-11 04:28:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 1250/2304 [I 2022-10-11 04:28:54] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:15:04 [I 2022-10-11 04:28:54] ...trap/engines/engine.py.229: process: 0.84646 | load: 0.00310 [I 2022-10-11 04:28:54] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-11 04:28:54] ...optimizers/trijoint.py.151: total_norm: 20.645874 [I 2022-10-11 04:29:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 1260/2304 [I 2022-10-11 04:29:03] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:57 [I 2022-10-11 04:29:03] ...trap/engines/engine.py.229: process: 0.85883 | load: 0.00301 [I 2022-10-11 04:29:03] ...trap/engines/engine.py.230: loss: 0.10385 [I 2022-10-11 04:29:03] ...optimizers/trijoint.py.151: total_norm: 20.453094 [I 2022-10-11 04:29:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 1270/2304 [I 2022-10-11 04:29:11] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:52 [I 2022-10-11 04:29:11] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00314 [I 2022-10-11 04:29:11] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-11 04:29:11] ...optimizers/trijoint.py.151: total_norm: 24.424068 [I 2022-10-11 04:29:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 1280/2304 [I 2022-10-11 04:29:20] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:42 [I 2022-10-11 04:29:20] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00324 [I 2022-10-11 04:29:20] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-11 04:29:20] ...optimizers/trijoint.py.151: total_norm: 11.843140 [I 2022-10-11 04:29:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 1290/2304 [I 2022-10-11 04:29:29] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:37 [I 2022-10-11 04:29:29] ...trap/engines/engine.py.229: process: 0.86629 | load: 0.00309 [I 2022-10-11 04:29:29] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-11 04:29:29] ...optimizers/trijoint.py.151: total_norm: 11.281671 [I 2022-10-11 04:29:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 1300/2304 [I 2022-10-11 04:29:37] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:22 [I 2022-10-11 04:29:37] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00320 [I 2022-10-11 04:29:37] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-11 04:29:37] ...optimizers/trijoint.py.151: total_norm: 25.752270 [I 2022-10-11 04:29:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 1310/2304 [I 2022-10-11 04:29:46] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:20 [I 2022-10-11 04:29:46] ...trap/engines/engine.py.229: process: 0.87668 | load: 0.00309 [I 2022-10-11 04:29:46] ...trap/engines/engine.py.230: loss: 0.15662 [I 2022-10-11 04:29:46] ...optimizers/trijoint.py.151: total_norm: 10.740898 [I 2022-10-11 04:29:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 1320/2304 [I 2022-10-11 04:29:55] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:14:07 [I 2022-10-11 04:29:55] ...trap/engines/engine.py.229: process: 0.87112 | load: 0.00312 [I 2022-10-11 04:29:55] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-11 04:29:55] ...optimizers/trijoint.py.151: total_norm: 25.493757 [I 2022-10-11 04:30:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 1330/2304 [I 2022-10-11 04:30:03] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:13:59 [I 2022-10-11 04:30:03] ...trap/engines/engine.py.229: process: 0.86921 | load: 0.00308 [I 2022-10-11 04:30:03] ...trap/engines/engine.py.230: loss: 0.09309 [I 2022-10-11 04:30:03] ...optimizers/trijoint.py.151: total_norm: 20.466902 [I 2022-10-11 04:30:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 1340/2304 [I 2022-10-11 04:30:12] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:49 [I 2022-10-11 04:30:12] ...trap/engines/engine.py.229: process: 0.86673 | load: 0.00311 [I 2022-10-11 04:30:12] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-11 04:30:12] ...optimizers/trijoint.py.151: total_norm: 35.145313 [I 2022-10-11 04:30:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 1350/2304 [I 2022-10-11 04:30:20] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:13:46 [I 2022-10-11 04:30:20] ...trap/engines/engine.py.229: process: 0.87260 | load: 0.00315 [I 2022-10-11 04:30:20] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-11 04:30:20] ...optimizers/trijoint.py.151: total_norm: 21.139404 [I 2022-10-11 04:30:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 1360/2304 [I 2022-10-11 04:30:29] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:13:34 [I 2022-10-11 04:30:29] ...trap/engines/engine.py.229: process: 0.86903 | load: 0.00318 [I 2022-10-11 04:30:29] ...trap/engines/engine.py.230: loss: 0.09787 [I 2022-10-11 04:30:29] ...optimizers/trijoint.py.151: total_norm: 14.106775 [I 2022-10-11 04:30:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 1370/2304 [I 2022-10-11 04:30:38] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:13:27 [I 2022-10-11 04:30:38] ...trap/engines/engine.py.229: process: 0.86570 | load: 0.00307 [I 2022-10-11 04:30:38] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-11 04:30:38] ...optimizers/trijoint.py.151: total_norm: 63.161465 [I 2022-10-11 04:30:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 1380/2304 [I 2022-10-11 04:30:46] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:13:15 [I 2022-10-11 04:30:46] ...trap/engines/engine.py.229: process: 0.86803 | load: 0.00298 [I 2022-10-11 04:30:46] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-11 04:30:46] ...optimizers/trijoint.py.151: total_norm: 65.482552 [I 2022-10-11 04:30:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 1390/2304 [I 2022-10-11 04:30:55] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:13:02 [I 2022-10-11 04:30:55] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00310 [I 2022-10-11 04:30:55] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-11 04:30:55] ...optimizers/trijoint.py.151: total_norm: 14.470906 [I 2022-10-11 04:31:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 1400/2304 [I 2022-10-11 04:31:04] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:57 [I 2022-10-11 04:31:04] ...trap/engines/engine.py.229: process: 0.86437 | load: 0.00309 [I 2022-10-11 04:31:04] ...trap/engines/engine.py.230: loss: 0.08921 [I 2022-10-11 04:31:04] ...optimizers/trijoint.py.151: total_norm: 20.558689 [I 2022-10-11 04:31:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 1410/2304 [I 2022-10-11 04:31:12] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:48 [I 2022-10-11 04:31:12] ...trap/engines/engine.py.229: process: 0.86335 | load: 0.00308 [I 2022-10-11 04:31:12] ...trap/engines/engine.py.230: loss: 0.10377 [I 2022-10-11 04:31:12] ...optimizers/trijoint.py.151: total_norm: 13.380274 [I 2022-10-11 04:31:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 1420/2304 [I 2022-10-11 04:31:21] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:46 [I 2022-10-11 04:31:21] ...trap/engines/engine.py.229: process: 0.87176 | load: 0.00309 [I 2022-10-11 04:31:21] ...trap/engines/engine.py.230: loss: 0.10162 [I 2022-10-11 04:31:21] ...optimizers/trijoint.py.151: total_norm: 10.831190 [I 2022-10-11 04:31:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 1430/2304 [I 2022-10-11 04:31:29] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:30 [I 2022-10-11 04:31:29] ...trap/engines/engine.py.229: process: 0.86553 | load: 0.00311 [I 2022-10-11 04:31:29] ...trap/engines/engine.py.230: loss: 0.11952 [I 2022-10-11 04:31:29] ...optimizers/trijoint.py.151: total_norm: 18.336815 [I 2022-10-11 04:31:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 1440/2304 [I 2022-10-11 04:31:38] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:22 [I 2022-10-11 04:31:38] ...trap/engines/engine.py.229: process: 0.86269 | load: 0.00304 [I 2022-10-11 04:31:38] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-11 04:31:38] ...optimizers/trijoint.py.151: total_norm: 17.727636 [I 2022-10-11 04:31:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 1450/2304 [I 2022-10-11 04:31:47] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:14 [I 2022-10-11 04:31:47] ...trap/engines/engine.py.229: process: 0.85744 | load: 0.00314 [I 2022-10-11 04:31:47] ...trap/engines/engine.py.230: loss: 0.15868 [I 2022-10-11 04:31:47] ...optimizers/trijoint.py.151: total_norm: 23.264572 [I 2022-10-11 04:31:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 1460/2304 [I 2022-10-11 04:31:55] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:12:07 [I 2022-10-11 04:31:55] ...trap/engines/engine.py.229: process: 0.86540 | load: 0.00309 [I 2022-10-11 04:31:55] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-11 04:31:55] ...optimizers/trijoint.py.151: total_norm: 12.964440 [I 2022-10-11 04:32:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 1470/2304 [I 2022-10-11 04:32:04] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:12:01 [I 2022-10-11 04:32:04] ...trap/engines/engine.py.229: process: 0.86235 | load: 0.00317 [I 2022-10-11 04:32:04] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-11 04:32:04] ...optimizers/trijoint.py.151: total_norm: 10.531759 [I 2022-10-11 04:32:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1480/2304 [I 2022-10-11 04:32:13] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:50 [I 2022-10-11 04:32:13] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00384 [I 2022-10-11 04:32:13] ...trap/engines/engine.py.230: loss: 0.10242 [I 2022-10-11 04:32:13] ...optimizers/trijoint.py.151: total_norm: 14.363688 [I 2022-10-11 04:32:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 1490/2304 [I 2022-10-11 04:32:21] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:44 [I 2022-10-11 04:32:21] ...trap/engines/engine.py.229: process: 0.86187 | load: 0.00309 [I 2022-10-11 04:32:21] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-11 04:32:21] ...optimizers/trijoint.py.151: total_norm: 150.279617 [I 2022-10-11 04:32:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 1500/2304 [I 2022-10-11 04:32:30] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:36 [I 2022-10-11 04:32:30] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00312 [I 2022-10-11 04:32:30] ...trap/engines/engine.py.230: loss: 0.11998 [I 2022-10-11 04:32:30] ...optimizers/trijoint.py.151: total_norm: 8.960071 [I 2022-10-11 04:32:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 1510/2304 [I 2022-10-11 04:32:39] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:23 [I 2022-10-11 04:32:39] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00308 [I 2022-10-11 04:32:39] ...trap/engines/engine.py.230: loss: 0.09490 [I 2022-10-11 04:32:39] ...optimizers/trijoint.py.151: total_norm: 10.973343 [I 2022-10-11 04:32:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 1520/2304 [I 2022-10-11 04:32:47] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:14 [I 2022-10-11 04:32:47] ...trap/engines/engine.py.229: process: 0.88050 | load: 0.00310 [I 2022-10-11 04:32:47] ...trap/engines/engine.py.230: loss: 0.08097 [I 2022-10-11 04:32:47] ...optimizers/trijoint.py.151: total_norm: 15.072486 [I 2022-10-11 04:32:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 1530/2304 [I 2022-10-11 04:32:56] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:11:04 [I 2022-10-11 04:32:56] ...trap/engines/engine.py.229: process: 0.85042 | load: 0.00308 [I 2022-10-11 04:32:56] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-11 04:32:56] ...optimizers/trijoint.py.151: total_norm: 14.067467 [I 2022-10-11 04:33:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 1540/2304 [I 2022-10-11 04:33:04] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:55 [I 2022-10-11 04:33:04] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00300 [I 2022-10-11 04:33:04] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-11 04:33:04] ...optimizers/trijoint.py.151: total_norm: 8.560211 [I 2022-10-11 04:33:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1550/2304 [I 2022-10-11 04:33:13] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:53 [I 2022-10-11 04:33:13] ...trap/engines/engine.py.229: process: 0.86422 | load: 0.00313 [I 2022-10-11 04:33:13] ...trap/engines/engine.py.230: loss: 0.10524 [I 2022-10-11 04:33:13] ...optimizers/trijoint.py.151: total_norm: 12.284203 [I 2022-10-11 04:33:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 1560/2304 [I 2022-10-11 04:33:22] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:40 [I 2022-10-11 04:33:22] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00304 [I 2022-10-11 04:33:22] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-11 04:33:22] ...optimizers/trijoint.py.151: total_norm: 17.237551 [I 2022-10-11 04:33:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 1570/2304 [I 2022-10-11 04:33:30] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:33 [I 2022-10-11 04:33:30] ...trap/engines/engine.py.229: process: 0.86644 | load: 0.00312 [I 2022-10-11 04:33:30] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-11 04:33:30] ...optimizers/trijoint.py.151: total_norm: 16.025862 [I 2022-10-11 04:33:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 1580/2304 [I 2022-10-11 04:33:39] ...trap/engines/engine.py.228: elapsed: 0:22:46 | left: 0:10:21 [I 2022-10-11 04:33:39] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00313 [I 2022-10-11 04:33:39] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-11 04:33:39] ...optimizers/trijoint.py.151: total_norm: 17.832178 [I 2022-10-11 04:33:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 1590/2304 [I 2022-10-11 04:33:48] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:10:14 [I 2022-10-11 04:33:48] ...trap/engines/engine.py.229: process: 0.86005 | load: 0.00309 [I 2022-10-11 04:33:48] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-11 04:33:48] ...optimizers/trijoint.py.151: total_norm: 11.074399 [I 2022-10-11 04:33:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 1600/2304 [I 2022-10-11 04:33:56] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:10:05 [I 2022-10-11 04:33:56] ...trap/engines/engine.py.229: process: 0.86284 | load: 0.00311 [I 2022-10-11 04:33:56] ...trap/engines/engine.py.230: loss: 0.18682 [I 2022-10-11 04:33:56] ...optimizers/trijoint.py.151: total_norm: 90.141701 [I 2022-10-11 04:34:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 1610/2304 [I 2022-10-11 04:34:05] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:54 [I 2022-10-11 04:34:05] ...trap/engines/engine.py.229: process: 0.83333 | load: 0.00325 [I 2022-10-11 04:34:05] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-11 04:34:05] ...optimizers/trijoint.py.151: total_norm: 17.750313 [I 2022-10-11 04:34:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1620/2304 [I 2022-10-11 04:34:13] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:47 [I 2022-10-11 04:34:13] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00307 [I 2022-10-11 04:34:13] ...trap/engines/engine.py.230: loss: 0.11613 [I 2022-10-11 04:34:13] ...optimizers/trijoint.py.151: total_norm: 7.596663 [I 2022-10-11 04:34:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 1630/2304 [I 2022-10-11 04:34:22] ...trap/engines/engine.py.228: elapsed: 0:23:30 | left: 0:09:39 [I 2022-10-11 04:34:22] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00307 [I 2022-10-11 04:34:22] ...trap/engines/engine.py.230: loss: 0.08433 [I 2022-10-11 04:34:22] ...optimizers/trijoint.py.151: total_norm: 11.002724 [I 2022-10-11 04:34:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 1640/2304 [I 2022-10-11 04:34:31] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:28 [I 2022-10-11 04:34:31] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00310 [I 2022-10-11 04:34:31] ...trap/engines/engine.py.230: loss: 0.14770 [I 2022-10-11 04:34:31] ...optimizers/trijoint.py.151: total_norm: 14.335897 [I 2022-10-11 04:34:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 1650/2304 [I 2022-10-11 04:34:39] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:24 [I 2022-10-11 04:34:39] ...trap/engines/engine.py.229: process: 0.87847 | load: 0.00306 [I 2022-10-11 04:34:39] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-11 04:34:39] ...optimizers/trijoint.py.151: total_norm: 13.219161 [I 2022-10-11 04:34:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 1660/2304 [I 2022-10-11 04:34:48] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:15 [I 2022-10-11 04:34:48] ...trap/engines/engine.py.229: process: 0.86328 | load: 0.00306 [I 2022-10-11 04:34:48] ...trap/engines/engine.py.230: loss: 0.12866 [I 2022-10-11 04:34:48] ...optimizers/trijoint.py.151: total_norm: 14.941332 [I 2022-10-11 04:34:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 1670/2304 [I 2022-10-11 04:34:56] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:09:08 [I 2022-10-11 04:34:56] ...trap/engines/engine.py.229: process: 0.86801 | load: 0.00310 [I 2022-10-11 04:34:56] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-11 04:34:56] ...optimizers/trijoint.py.151: total_norm: 12.269556 [I 2022-10-11 04:35:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 1680/2304 [I 2022-10-11 04:35:05] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:58 [I 2022-10-11 04:35:05] ...trap/engines/engine.py.229: process: 0.86341 | load: 0.00312 [I 2022-10-11 04:35:05] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-11 04:35:05] ...optimizers/trijoint.py.151: total_norm: 10.986388 [I 2022-10-11 04:35:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 1690/2304 [I 2022-10-11 04:35:14] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:50 [I 2022-10-11 04:35:14] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00309 [I 2022-10-11 04:35:14] ...trap/engines/engine.py.230: loss: 0.14782 [I 2022-10-11 04:35:14] ...optimizers/trijoint.py.151: total_norm: 9.591470 [I 2022-10-11 04:35:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 1700/2304 [I 2022-10-11 04:35:22] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:41 [I 2022-10-11 04:35:22] ...trap/engines/engine.py.229: process: 0.86125 | load: 0.00312 [I 2022-10-11 04:35:22] ...trap/engines/engine.py.230: loss: 0.16458 [I 2022-10-11 04:35:22] ...optimizers/trijoint.py.151: total_norm: 13.338547 [I 2022-10-11 04:35:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 1710/2304 [I 2022-10-11 04:35:31] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:35 [I 2022-10-11 04:35:31] ...trap/engines/engine.py.229: process: 0.87528 | load: 0.00308 [I 2022-10-11 04:35:31] ...trap/engines/engine.py.230: loss: 0.21514 [I 2022-10-11 04:35:31] ...optimizers/trijoint.py.151: total_norm: 24.753939 [I 2022-10-11 04:35:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 1720/2304 [I 2022-10-11 04:35:40] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:24 [I 2022-10-11 04:35:40] ...trap/engines/engine.py.229: process: 0.85633 | load: 0.00319 [I 2022-10-11 04:35:40] ...trap/engines/engine.py.230: loss: 0.10681 [I 2022-10-11 04:35:40] ...optimizers/trijoint.py.151: total_norm: 9.618359 [I 2022-10-11 04:35:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 1730/2304 [I 2022-10-11 04:35:48] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:18 [I 2022-10-11 04:35:48] ...trap/engines/engine.py.229: process: 0.87350 | load: 0.00324 [I 2022-10-11 04:35:48] ...trap/engines/engine.py.230: loss: 0.14918 [I 2022-10-11 04:35:48] ...optimizers/trijoint.py.151: total_norm: 23.540413 [I 2022-10-11 04:35:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 1740/2304 [I 2022-10-11 04:35:57] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:08:06 [I 2022-10-11 04:35:57] ...trap/engines/engine.py.229: process: 0.85525 | load: 0.00319 [I 2022-10-11 04:35:57] ...trap/engines/engine.py.230: loss: 0.09220 [I 2022-10-11 04:35:57] ...optimizers/trijoint.py.151: total_norm: 9.334756 [I 2022-10-11 04:36:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1750/2304 [I 2022-10-11 04:36:06] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:08:00 [I 2022-10-11 04:36:06] ...trap/engines/engine.py.229: process: 0.87514 | load: 0.00308 [I 2022-10-11 04:36:06] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-11 04:36:06] ...optimizers/trijoint.py.151: total_norm: 20.857504 [I 2022-10-11 04:36:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 1760/2304 [I 2022-10-11 04:36:14] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:47 [I 2022-10-11 04:36:14] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00311 [I 2022-10-11 04:36:14] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-11 04:36:14] ...optimizers/trijoint.py.151: total_norm: 22.673574 [I 2022-10-11 04:36:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1770/2304 [I 2022-10-11 04:36:23] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:39 [I 2022-10-11 04:36:23] ...trap/engines/engine.py.229: process: 0.86648 | load: 0.00316 [I 2022-10-11 04:36:23] ...trap/engines/engine.py.230: loss: 0.13358 [I 2022-10-11 04:36:23] ...optimizers/trijoint.py.151: total_norm: 26.492769 [I 2022-10-11 04:36:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1780/2304 [I 2022-10-11 04:36:32] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:32 [I 2022-10-11 04:36:32] ...trap/engines/engine.py.229: process: 0.87455 | load: 0.00314 [I 2022-10-11 04:36:32] ...trap/engines/engine.py.230: loss: 0.09736 [I 2022-10-11 04:36:32] ...optimizers/trijoint.py.151: total_norm: 23.926083 [I 2022-10-11 04:36:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 1790/2304 [I 2022-10-11 04:36:40] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:20 [I 2022-10-11 04:36:40] ...trap/engines/engine.py.229: process: 0.84280 | load: 0.00317 [I 2022-10-11 04:36:40] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-11 04:36:40] ...optimizers/trijoint.py.151: total_norm: 12.832088 [I 2022-10-11 04:36:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1800/2304 [I 2022-10-11 04:36:49] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:12 [I 2022-10-11 04:36:49] ...trap/engines/engine.py.229: process: 0.86031 | load: 0.00300 [I 2022-10-11 04:36:49] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-11 04:36:49] ...optimizers/trijoint.py.151: total_norm: 10.635714 [I 2022-10-11 04:36:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 1810/2304 [I 2022-10-11 04:36:57] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:05 [I 2022-10-11 04:36:57] ...trap/engines/engine.py.229: process: 0.86340 | load: 0.00313 [I 2022-10-11 04:36:57] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-11 04:36:58] ...optimizers/trijoint.py.151: total_norm: 13.316138 [I 2022-10-11 04:37:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1820/2304 [I 2022-10-11 04:37:06] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:55 [I 2022-10-11 04:37:06] ...trap/engines/engine.py.229: process: 0.86820 | load: 0.00320 [I 2022-10-11 04:37:06] ...trap/engines/engine.py.230: loss: 0.17069 [I 2022-10-11 04:37:06] ...optimizers/trijoint.py.151: total_norm: 17.285629 [I 2022-10-11 04:37:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1830/2304 [I 2022-10-11 04:37:15] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:48 [I 2022-10-11 04:37:15] ...trap/engines/engine.py.229: process: 0.87192 | load: 0.00316 [I 2022-10-11 04:37:15] ...trap/engines/engine.py.230: loss: 0.11138 [I 2022-10-11 04:37:15] ...optimizers/trijoint.py.151: total_norm: 14.792089 [I 2022-10-11 04:37:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1840/2304 [I 2022-10-11 04:37:23] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:37 [I 2022-10-11 04:37:23] ...trap/engines/engine.py.229: process: 0.86745 | load: 0.00308 [I 2022-10-11 04:37:23] ...trap/engines/engine.py.230: loss: 0.10658 [I 2022-10-11 04:37:23] ...optimizers/trijoint.py.151: total_norm: 11.913452 [I 2022-10-11 04:37:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1850/2304 [I 2022-10-11 04:37:32] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:29 [I 2022-10-11 04:37:32] ...trap/engines/engine.py.229: process: 0.85432 | load: 0.00310 [I 2022-10-11 04:37:32] ...trap/engines/engine.py.230: loss: 0.18360 [I 2022-10-11 04:37:32] ...optimizers/trijoint.py.151: total_norm: 25.021622 [I 2022-10-11 04:37:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1860/2304 [I 2022-10-11 04:37:41] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:22 [I 2022-10-11 04:37:41] ...trap/engines/engine.py.229: process: 0.84888 | load: 0.00305 [I 2022-10-11 04:37:41] ...trap/engines/engine.py.230: loss: 0.09811 [I 2022-10-11 04:37:41] ...optimizers/trijoint.py.151: total_norm: 13.254811 [I 2022-10-11 04:37:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1870/2304 [I 2022-10-11 04:37:49] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:12 [I 2022-10-11 04:37:49] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00308 [I 2022-10-11 04:37:49] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-11 04:37:49] ...optimizers/trijoint.py.151: total_norm: 28.546957 [I 2022-10-11 04:37:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1880/2304 [I 2022-10-11 04:37:58] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:05 [I 2022-10-11 04:37:58] ...trap/engines/engine.py.229: process: 0.86651 | load: 0.00311 [I 2022-10-11 04:37:58] ...trap/engines/engine.py.230: loss: 0.12295 [I 2022-10-11 04:37:58] ...optimizers/trijoint.py.151: total_norm: 18.580294 [I 2022-10-11 04:38:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1890/2304 [I 2022-10-11 04:38:06] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:05:58 [I 2022-10-11 04:38:06] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00307 [I 2022-10-11 04:38:06] ...trap/engines/engine.py.230: loss: 0.08762 [I 2022-10-11 04:38:06] ...optimizers/trijoint.py.151: total_norm: 14.181978 [I 2022-10-11 04:38:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1900/2304 [I 2022-10-11 04:38:15] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:48 [I 2022-10-11 04:38:15] ...trap/engines/engine.py.229: process: 0.86875 | load: 0.00309 [I 2022-10-11 04:38:15] ...trap/engines/engine.py.230: loss: 0.10297 [I 2022-10-11 04:38:15] ...optimizers/trijoint.py.151: total_norm: 9.668003 [I 2022-10-11 04:38:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1910/2304 [I 2022-10-11 04:38:24] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:37 [I 2022-10-11 04:38:24] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00303 [I 2022-10-11 04:38:24] ...trap/engines/engine.py.230: loss: 0.19039 [I 2022-10-11 04:38:24] ...optimizers/trijoint.py.151: total_norm: 15.301243 [I 2022-10-11 04:38:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1920/2304 [I 2022-10-11 04:38:32] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:05:31 [I 2022-10-11 04:38:32] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00308 [I 2022-10-11 04:38:32] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-11 04:38:32] ...optimizers/trijoint.py.151: total_norm: 32.153374 [I 2022-10-11 04:38:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1930/2304 [I 2022-10-11 04:38:41] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:25 [I 2022-10-11 04:38:41] ...trap/engines/engine.py.229: process: 0.88727 | load: 0.00312 [I 2022-10-11 04:38:41] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 04:38:41] ...optimizers/trijoint.py.151: total_norm: 14.222795 [I 2022-10-11 04:38:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 1940/2304 [I 2022-10-11 04:38:50] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:05:13 [I 2022-10-11 04:38:50] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00308 [I 2022-10-11 04:38:50] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-11 04:38:50] ...optimizers/trijoint.py.151: total_norm: 11.695612 [I 2022-10-11 04:38:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1950/2304 [I 2022-10-11 04:38:58] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:05:04 [I 2022-10-11 04:38:58] ...trap/engines/engine.py.229: process: 0.86203 | load: 0.00307 [I 2022-10-11 04:38:58] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-11 04:38:58] ...optimizers/trijoint.py.151: total_norm: 22.178064 [I 2022-10-11 04:39:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1960/2304 [I 2022-10-11 04:39:07] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:04:56 [I 2022-10-11 04:39:07] ...trap/engines/engine.py.229: process: 0.86546 | load: 0.00309 [I 2022-10-11 04:39:07] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-11 04:39:07] ...optimizers/trijoint.py.151: total_norm: 25.189066 [I 2022-10-11 04:39:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1970/2304 [I 2022-10-11 04:39:15] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:46 [I 2022-10-11 04:39:15] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00312 [I 2022-10-11 04:39:15] ...trap/engines/engine.py.230: loss: 0.15719 [I 2022-10-11 04:39:15] ...optimizers/trijoint.py.151: total_norm: 10.104259 [I 2022-10-11 04:39:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1980/2304 [I 2022-10-11 04:39:24] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:39 [I 2022-10-11 04:39:24] ...trap/engines/engine.py.229: process: 0.85956 | load: 0.00310 [I 2022-10-11 04:39:24] ...trap/engines/engine.py.230: loss: 0.10566 [I 2022-10-11 04:39:24] ...optimizers/trijoint.py.151: total_norm: 17.399872 [I 2022-10-11 04:39:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1990/2304 [I 2022-10-11 04:39:33] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:04:29 [I 2022-10-11 04:39:33] ...trap/engines/engine.py.229: process: 0.85996 | load: 0.00309 [I 2022-10-11 04:39:33] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-11 04:39:33] ...optimizers/trijoint.py.151: total_norm: 16.279312 [I 2022-10-11 04:39:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 2000/2304 [I 2022-10-11 04:39:41] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:21 [I 2022-10-11 04:39:41] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00311 [I 2022-10-11 04:39:41] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-11 04:39:41] ...optimizers/trijoint.py.151: total_norm: 20.311129 [I 2022-10-11 04:39:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 2010/2304 [I 2022-10-11 04:39:50] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:14 [I 2022-10-11 04:39:50] ...trap/engines/engine.py.229: process: 0.87587 | load: 0.00319 [I 2022-10-11 04:39:50] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-11 04:39:50] ...optimizers/trijoint.py.151: total_norm: 17.667751 [I 2022-10-11 04:39:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 2020/2304 [I 2022-10-11 04:39:59] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:04:05 [I 2022-10-11 04:39:59] ...trap/engines/engine.py.229: process: 0.86221 | load: 0.00309 [I 2022-10-11 04:39:59] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-11 04:39:59] ...optimizers/trijoint.py.151: total_norm: 20.567734 [I 2022-10-11 04:40:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 2030/2304 [I 2022-10-11 04:40:07] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:56 [I 2022-10-11 04:40:07] ...trap/engines/engine.py.229: process: 0.86383 | load: 0.00312 [I 2022-10-11 04:40:07] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-11 04:40:07] ...optimizers/trijoint.py.151: total_norm: 22.919979 [I 2022-10-11 04:40:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 2040/2304 [I 2022-10-11 04:40:16] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:48 [I 2022-10-11 04:40:16] ...trap/engines/engine.py.229: process: 0.86826 | load: 0.00321 [I 2022-10-11 04:40:16] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-11 04:40:16] ...optimizers/trijoint.py.151: total_norm: 44.066071 [I 2022-10-11 04:40:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 2050/2304 [I 2022-10-11 04:40:25] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:38 [I 2022-10-11 04:40:25] ...trap/engines/engine.py.229: process: 0.86912 | load: 0.00314 [I 2022-10-11 04:40:25] ...trap/engines/engine.py.230: loss: 0.12406 [I 2022-10-11 04:40:25] ...optimizers/trijoint.py.151: total_norm: 21.520962 [I 2022-10-11 04:40:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 2060/2304 [I 2022-10-11 04:40:33] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:31 [I 2022-10-11 04:40:33] ...trap/engines/engine.py.229: process: 0.86802 | load: 0.00316 [I 2022-10-11 04:40:33] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-11 04:40:33] ...optimizers/trijoint.py.151: total_norm: 11.176386 [I 2022-10-11 04:40:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 2070/2304 [I 2022-10-11 04:40:42] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:20 [I 2022-10-11 04:40:42] ...trap/engines/engine.py.229: process: 0.86324 | load: 0.00316 [I 2022-10-11 04:40:42] ...trap/engines/engine.py.230: loss: 0.12878 [I 2022-10-11 04:40:42] ...optimizers/trijoint.py.151: total_norm: 29.805614 [I 2022-10-11 04:40:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 2080/2304 [I 2022-10-11 04:40:50] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:12 [I 2022-10-11 04:40:50] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00324 [I 2022-10-11 04:40:50] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-11 04:40:50] ...optimizers/trijoint.py.151: total_norm: 22.793583 [I 2022-10-11 04:40:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 2090/2304 [I 2022-10-11 04:40:59] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:03:04 [I 2022-10-11 04:40:59] ...trap/engines/engine.py.229: process: 0.86161 | load: 0.00332 [I 2022-10-11 04:40:59] ...trap/engines/engine.py.230: loss: 0.09830 [I 2022-10-11 04:40:59] ...optimizers/trijoint.py.151: total_norm: 13.184810 [I 2022-10-11 04:41:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 2100/2304 [I 2022-10-11 04:41:08] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:55 [I 2022-10-11 04:41:08] ...trap/engines/engine.py.229: process: 0.85819 | load: 0.00322 [I 2022-10-11 04:41:08] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-11 04:41:08] ...optimizers/trijoint.py.151: total_norm: 14.570951 [I 2022-10-11 04:41:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 2110/2304 [I 2022-10-11 04:41:16] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:47 [I 2022-10-11 04:41:16] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00318 [I 2022-10-11 04:41:16] ...trap/engines/engine.py.230: loss: 0.10289 [I 2022-10-11 04:41:16] ...optimizers/trijoint.py.151: total_norm: 13.900625 [I 2022-10-11 04:41:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 2120/2304 [I 2022-10-11 04:41:25] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:38 [I 2022-10-11 04:41:25] ...trap/engines/engine.py.229: process: 0.85632 | load: 0.00326 [I 2022-10-11 04:41:25] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-11 04:41:25] ...optimizers/trijoint.py.151: total_norm: 11.318669 [I 2022-10-11 04:41:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 2130/2304 [I 2022-10-11 04:41:34] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:31 [I 2022-10-11 04:41:34] ...trap/engines/engine.py.229: process: 0.87736 | load: 0.00323 [I 2022-10-11 04:41:34] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 04:41:34] ...optimizers/trijoint.py.151: total_norm: 13.384078 [I 2022-10-11 04:41:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 2140/2304 [I 2022-10-11 04:41:42] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:20 [I 2022-10-11 04:41:42] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00321 [I 2022-10-11 04:41:42] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-11 04:41:42] ...optimizers/trijoint.py.151: total_norm: 18.813131 [I 2022-10-11 04:41:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 2150/2304 [I 2022-10-11 04:41:51] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:11 [I 2022-10-11 04:41:51] ...trap/engines/engine.py.229: process: 0.84008 | load: 0.00312 [I 2022-10-11 04:41:51] ...trap/engines/engine.py.230: loss: 0.14873 [I 2022-10-11 04:41:51] ...optimizers/trijoint.py.151: total_norm: 32.413109 [I 2022-10-11 04:41:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 2160/2304 [I 2022-10-11 04:41:59] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:02:03 [I 2022-10-11 04:41:59] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00320 [I 2022-10-11 04:41:59] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-11 04:41:59] ...optimizers/trijoint.py.151: total_norm: 29.397190 [I 2022-10-11 04:42:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 2170/2304 [I 2022-10-11 04:42:08] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:55 [I 2022-10-11 04:42:08] ...trap/engines/engine.py.229: process: 0.86422 | load: 0.00306 [I 2022-10-11 04:42:08] ...trap/engines/engine.py.230: loss: 0.10386 [I 2022-10-11 04:42:08] ...optimizers/trijoint.py.151: total_norm: 10.284778 [I 2022-10-11 04:42:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 2180/2304 [I 2022-10-11 04:42:17] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:47 [I 2022-10-11 04:42:17] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00309 [I 2022-10-11 04:42:17] ...trap/engines/engine.py.230: loss: 0.14370 [I 2022-10-11 04:42:17] ...optimizers/trijoint.py.151: total_norm: 9.833242 [I 2022-10-11 04:42:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 2190/2304 [I 2022-10-11 04:42:45] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:03:10 [I 2022-10-11 04:42:45] ...trap/engines/engine.py.229: process: 0.87096 | load: 0.00323 [I 2022-10-11 04:42:45] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-11 04:42:45] ...optimizers/trijoint.py.151: total_norm: 16.104643 [I 2022-10-11 04:42:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 2200/2304 [I 2022-10-11 04:42:53] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:37 [I 2022-10-11 04:42:53] ...trap/engines/engine.py.229: process: 0.85406 | load: 0.00312 [I 2022-10-11 04:42:53] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-11 04:42:53] ...optimizers/trijoint.py.151: total_norm: 29.204308 [I 2022-10-11 04:43:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 2210/2304 [I 2022-10-11 04:43:02] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:01:21 [I 2022-10-11 04:43:02] ...trap/engines/engine.py.229: process: 0.85455 | load: 0.00309 [I 2022-10-11 04:43:02] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-11 04:43:02] ...optimizers/trijoint.py.151: total_norm: 18.077610 [I 2022-10-11 04:43:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 2220/2304 [I 2022-10-11 04:43:10] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:11 [I 2022-10-11 04:43:10] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00318 [I 2022-10-11 04:43:10] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-11 04:43:10] ...optimizers/trijoint.py.151: total_norm: 16.751783 [I 2022-10-11 04:43:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 2230/2304 [I 2022-10-11 04:43:19] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:01:03 [I 2022-10-11 04:43:19] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00325 [I 2022-10-11 04:43:19] ...trap/engines/engine.py.230: loss: 0.15623 [I 2022-10-11 04:43:19] ...optimizers/trijoint.py.151: total_norm: 23.560345 [I 2022-10-11 04:43:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 2240/2304 [I 2022-10-11 04:43:28] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:55 [I 2022-10-11 04:43:28] ...trap/engines/engine.py.229: process: 0.85989 | load: 0.00308 [I 2022-10-11 04:43:28] ...trap/engines/engine.py.230: loss: 0.13610 [I 2022-10-11 04:43:28] ...optimizers/trijoint.py.151: total_norm: 13.183584 [I 2022-10-11 04:43:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 2250/2304 [I 2022-10-11 04:43:36] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:46 [I 2022-10-11 04:43:36] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00310 [I 2022-10-11 04:43:36] ...trap/engines/engine.py.230: loss: 0.09575 [I 2022-10-11 04:43:36] ...optimizers/trijoint.py.151: total_norm: 9.995465 [I 2022-10-11 04:43:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 2260/2304 [I 2022-10-11 04:43:45] ...trap/engines/engine.py.228: elapsed: 0:32:53 | left: 0:00:38 [I 2022-10-11 04:43:45] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00319 [I 2022-10-11 04:43:45] ...trap/engines/engine.py.230: loss: 0.12867 [I 2022-10-11 04:43:45] ...optimizers/trijoint.py.151: total_norm: 22.117220 [I 2022-10-11 04:43:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 2270/2304 [I 2022-10-11 04:43:53] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:28 [I 2022-10-11 04:43:53] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00293 [I 2022-10-11 04:43:53] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-11 04:43:53] ...optimizers/trijoint.py.151: total_norm: 19.692507 [I 2022-10-11 04:44:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 2280/2304 [I 2022-10-11 04:44:02] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:20 [I 2022-10-11 04:44:02] ...trap/engines/engine.py.229: process: 0.84618 | load: 0.00291 [I 2022-10-11 04:44:02] ...trap/engines/engine.py.230: loss: 0.08582 [I 2022-10-11 04:44:02] ...optimizers/trijoint.py.151: total_norm: 8.481562 [I 2022-10-11 07:28:24] ...ytorch/bootstrap/run.py.73: Saving environment info [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: options: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dataset: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: aux_kw_path: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: batch_sampler: triplet_classif [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: batch_size: 100 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: database: lmdb [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: eval_split: val [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: freq_mismatch: 0.0 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: image_from: database [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: import: recipe1m.datasets.factory [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: kw_path: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: recipe1m [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_classes: 1048 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_threads: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: randkw_p: 0.3 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: randkw_p_aux: 0.5 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: tokenized_raw_text: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: train_split: train [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: use_vcs: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: engine: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: logger [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_epochs: 120 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: print_freq: 10 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max'] [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: exp: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: resume: last [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: misc: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cuda: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: data_parrallel: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: logs_name: None [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: overrite: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: seed: 1338 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: model: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: criterion: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: tri_cross_encoder: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: itm_loss_weight: 1 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: keep_background: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: retrieval_strategy: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: aggregation: valid [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: margin: 0.05 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: margin_params: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: increment: 0.005 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: increment_margin: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: max_margin: 0.3 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: triplet [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_samples: 9999 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: sampling: max_negative [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: substrategy: ['IRR', 'RII', 'SIRR', 'SRII'] [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: substrategy_weights: [1.0, 1.0, 0.1, 0.1] [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: import: recipe1m.models.factory [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: metric: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: k_test: 10 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_bags: 10 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: nb_matchs_per_bag: 1000 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: trijoint: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: network: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: activations: ['tanh', 'normalize'] [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: aux_kwords: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: aux_kwords_encoder: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: avg_concat: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cat_pos: 0 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: class_attention_fus: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cls_token_fus: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_decoder_image: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_decoder_recipe: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_attention: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_decoder: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: double_decoder: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: get_tokens: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: load_backbones: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_layers: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: query_img: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: cross_transformer: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dim_emb: 1024 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dim_image_out: 2048 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dim_ingr_out: 300 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dim_instr_in: 1024 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: dim_instr_out: 1024 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: embedding_names: ['proj_image'] [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: freeze_im: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: freeze_rec: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: get_tokens_cross_decoder_recipe: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: hidden_size: 512 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: image_backbone_name: clip [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: image_res: 224 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: kwords_same_level: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: last_cat_aux: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_heads_cross: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_heads_cross_image: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_heads_single: 4 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_layers: 2 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_layers_cross: 2 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_layers_cross_image: 1 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: n_layers_single: 2 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: num_hidden_layers_kw: 2 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: old: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: recipe_encoder: tfoodvicha-transformer [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: text_encoder: bert-base-uncased [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: vision_width: 768 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: vit_all_tokens: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: with_ingrs: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: with_instrs: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: with_titles: True [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: with_classif: False [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: optimizer: [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: batch_size_factor: None [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: clip_grad: 8.0 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: import: recipe1m.optimizers.factory [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: lr: 1e-05 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: lr_img: 1e-06 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: name: trijoint_fixed_fine_tune [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: switch_epoch: 20 [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml [S 2022-10-11 07:28:32] ...ytorch/bootstrap/run.py.75: view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm'] [I 2022-10-11 07:28:34] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-a101', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64') [I 2022-10-11 07:28:34] ...ytorch/bootstrap/run.py.80: Available GPUs: [0, 1] [I 2022-10-11 07:28:34] ...strap/engines/factory.py.9: Creating engine... [I 2022-10-11 07:28:34] ...trap/datasets/factory.py.8: Creating dataset... [I 2022-10-11 07:28:49] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-11 07:28:53] ...m/datasets/recipe1m.py.165: randkw_p... 0.3 [I 2022-10-11 07:30:00] ...m/datasets/recipe1m.py.370: Calculate indices by class... [I 2022-10-11 07:30:03] ...m/datasets/recipe1m.py.375: Done! [I 2022-10-11 07:30:04] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-11 07:30:04] ...m/datasets/recipe1m.py.165: randkw_p... None [I 2022-10-11 07:31:17] ...rap/datasets/factory.py.18: Training will take place on trainset (238399 items) [I 2022-10-11 07:31:17] ...rap/datasets/factory.py.21: Evaluation will take place on valset (51119 items) [I 2022-10-11 07:31:17] ...strap/models/factory.py.21: Creating model... [I 2022-10-11 07:31:21] ...e_networks/networks.py.203: Loading CLIP Visual model... [I 2022-10-11 07:31:25] ...e_networks/networks.py.145: cuda [I 2022-10-11 07:31:26] ...networks/tfood_vicha.py.49: Load VCs encoder... [I 2022-10-11 07:31:28] .../criterions/trijoint.py.30: ignore_index=0 [I 2022-10-11 07:31:28] ...strap/models/factory.py.44: Enabling CUDA mode... [I 2022-10-11 07:31:34] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-11 07:31:34] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-11 07:31:34] ...otstrap/views/factory.py.8: Creating views... [I 2022-10-11 07:31:39] ...strap/engines/engine.py.98: Loading last checkpoint [I 2022-10-11 07:31:39] ...trap/engines/engine.py.440: Loading model... [I 2022-10-11 07:31:45] ...trap/engines/engine.py.446: Loading optimizer... [I 2022-10-11 07:31:50] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-11 07:31:50] ...trap/engines/engine.py.453: Loading engine... [I 2022-10-11 07:31:50] ...trap/engines/engine.py.125: Launching training procedures [I 2022-10-11 07:31:50] ...trap/engines/engine.py.163: Training model on trainset for epoch 97 [I 2022-10-11 07:31:50] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 07:32:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 0/2304 [I 2022-10-11 07:32:00] ...trap/engines/engine.py.228: elapsed: 0:00:09 | left: 6:22:27 [I 2022-10-11 07:32:00] ...trap/engines/engine.py.229: process: 9.95979 | load: 4.70065 [I 2022-10-11 07:32:00] ...trap/engines/engine.py.230: loss: 0.14226 [I 2022-10-11 07:32:00] ...optimizers/trijoint.py.151: total_norm: 15.728865 [I 2022-10-11 07:32:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 10/2304 [I 2022-10-11 07:32:09] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 1:10:43 [I 2022-10-11 07:32:09] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00671 [I 2022-10-11 07:32:09] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-11 07:32:09] ...optimizers/trijoint.py.151: total_norm: 11.372201 [I 2022-10-11 07:32:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 20/2304 [I 2022-10-11 07:32:17] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:36:23 [I 2022-10-11 07:32:17] ...trap/engines/engine.py.229: process: 0.83071 | load: 0.00521 [I 2022-10-11 07:32:17] ...trap/engines/engine.py.230: loss: 0.09774 [I 2022-10-11 07:32:17] ...optimizers/trijoint.py.151: total_norm: 9.841487 [I 2022-10-11 07:32:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 30/2304 [I 2022-10-11 07:32:26] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:19 [I 2022-10-11 07:32:26] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00386 [I 2022-10-11 07:32:26] ...trap/engines/engine.py.230: loss: 0.18897 [I 2022-10-11 07:32:26] ...optimizers/trijoint.py.151: total_norm: 11.485145 [I 2022-10-11 07:32:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 40/2304 [I 2022-10-11 07:32:34] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:31:56 [I 2022-10-11 07:32:34] ...trap/engines/engine.py.229: process: 0.87671 | load: 0.00372 [I 2022-10-11 07:32:34] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-11 07:32:34] ...optimizers/trijoint.py.151: total_norm: 8.104083 [I 2022-10-11 07:32:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 50/2304 [I 2022-10-11 07:32:42] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:31:25 [I 2022-10-11 07:32:42] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00364 [I 2022-10-11 07:32:42] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-11 07:32:43] ...optimizers/trijoint.py.151: total_norm: 25.289404 [I 2022-10-11 07:32:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 60/2304 [I 2022-10-11 07:32:51] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:20 [I 2022-10-11 07:32:51] ...trap/engines/engine.py.229: process: 0.83414 | load: 0.00366 [I 2022-10-11 07:32:51] ...trap/engines/engine.py.230: loss: 0.10866 [I 2022-10-11 07:32:51] ...optimizers/trijoint.py.151: total_norm: 8.128033 [I 2022-10-11 07:32:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 70/2304 [I 2022-10-11 07:32:59] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:31:32 [I 2022-10-11 07:32:59] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00330 [I 2022-10-11 07:32:59] ...trap/engines/engine.py.230: loss: 0.10470 [I 2022-10-11 07:32:59] ...optimizers/trijoint.py.151: total_norm: 9.905895 [I 2022-10-11 07:33:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 80/2304 [I 2022-10-11 07:33:08] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:21 [I 2022-10-11 07:33:08] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00319 [I 2022-10-11 07:33:08] ...trap/engines/engine.py.230: loss: 0.11465 [I 2022-10-11 07:33:08] ...optimizers/trijoint.py.151: total_norm: 11.603593 [I 2022-10-11 07:33:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 90/2304 [I 2022-10-11 07:33:16] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:31:01 [I 2022-10-11 07:33:16] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00362 [I 2022-10-11 07:33:16] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-11 07:33:16] ...optimizers/trijoint.py.151: total_norm: 20.418295 [I 2022-10-11 07:33:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 100/2304 [I 2022-10-11 07:33:25] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:31:59 [I 2022-10-11 07:33:25] ...trap/engines/engine.py.229: process: 0.86958 | load: 0.00405 [I 2022-10-11 07:33:25] ...trap/engines/engine.py.230: loss: 0.13189 [I 2022-10-11 07:33:25] ...optimizers/trijoint.py.151: total_norm: 10.682788 [I 2022-10-11 07:33:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 110/2304 [I 2022-10-11 07:33:34] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:31:47 [I 2022-10-11 07:33:34] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00420 [I 2022-10-11 07:33:34] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-11 07:33:34] ...optimizers/trijoint.py.151: total_norm: 9.619420 [I 2022-10-11 07:33:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 120/2304 [I 2022-10-11 07:33:42] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:54 [I 2022-10-11 07:33:42] ...trap/engines/engine.py.229: process: 0.83686 | load: 0.00339 [I 2022-10-11 07:33:42] ...trap/engines/engine.py.230: loss: 0.11864 [I 2022-10-11 07:33:42] ...optimizers/trijoint.py.151: total_norm: 23.500788 [I 2022-10-11 07:33:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 130/2304 [I 2022-10-11 07:33:51] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:29 [I 2022-10-11 07:33:51] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00339 [I 2022-10-11 07:33:51] ...trap/engines/engine.py.230: loss: 0.14574 [I 2022-10-11 07:33:51] ...optimizers/trijoint.py.151: total_norm: 18.637440 [I 2022-10-11 07:33:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 140/2304 [I 2022-10-11 07:33:59] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:13 [I 2022-10-11 07:33:59] ...trap/engines/engine.py.229: process: 0.83661 | load: 0.00307 [I 2022-10-11 07:33:59] ...trap/engines/engine.py.230: loss: 0.12403 [I 2022-10-11 07:33:59] ...optimizers/trijoint.py.151: total_norm: 33.369236 [I 2022-10-11 07:34:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 150/2304 [I 2022-10-11 07:34:08] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:00 [I 2022-10-11 07:34:08] ...trap/engines/engine.py.229: process: 0.84169 | load: 0.00314 [I 2022-10-11 07:34:08] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-11 07:34:08] ...optimizers/trijoint.py.151: total_norm: 20.943838 [I 2022-10-11 07:34:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 160/2304 [I 2022-10-11 07:34:16] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:19 [I 2022-10-11 07:34:16] ...trap/engines/engine.py.229: process: 0.85263 | load: 0.00312 [I 2022-10-11 07:34:16] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-11 07:34:16] ...optimizers/trijoint.py.151: total_norm: 13.790658 [I 2022-10-11 07:34:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 170/2304 [I 2022-10-11 07:34:25] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:30:33 [I 2022-10-11 07:34:25] ...trap/engines/engine.py.229: process: 0.87185 | load: 0.00407 [I 2022-10-11 07:34:25] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-11 07:34:25] ...optimizers/trijoint.py.151: total_norm: 20.974400 [I 2022-10-11 07:34:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 180/2304 [I 2022-10-11 07:34:33] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:30:42 [I 2022-10-11 07:34:33] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00388 [I 2022-10-11 07:34:33] ...trap/engines/engine.py.230: loss: 0.10666 [I 2022-10-11 07:34:33] ...optimizers/trijoint.py.151: total_norm: 15.186754 [I 2022-10-11 07:34:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 190/2304 [I 2022-10-11 07:34:42] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:30:18 [I 2022-10-11 07:34:42] ...trap/engines/engine.py.229: process: 0.88682 | load: 0.00397 [I 2022-10-11 07:34:42] ...trap/engines/engine.py.230: loss: 0.17969 [I 2022-10-11 07:34:42] ...optimizers/trijoint.py.151: total_norm: 13.904910 [I 2022-10-11 07:34:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 200/2304 [I 2022-10-11 07:34:50] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:31 [I 2022-10-11 07:34:50] ...trap/engines/engine.py.229: process: 0.83933 | load: 0.00311 [I 2022-10-11 07:34:50] ...trap/engines/engine.py.230: loss: 0.09276 [I 2022-10-11 07:34:50] ...optimizers/trijoint.py.151: total_norm: 19.013527 [I 2022-10-11 07:34:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 210/2304 [I 2022-10-11 07:34:59] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:02 [I 2022-10-11 07:34:59] ...trap/engines/engine.py.229: process: 0.81326 | load: 0.00313 [I 2022-10-11 07:34:59] ...trap/engines/engine.py.230: loss: 0.09692 [I 2022-10-11 07:34:59] ...optimizers/trijoint.py.151: total_norm: 37.542747 [I 2022-10-11 07:35:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 220/2304 [I 2022-10-11 07:35:07] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:16 [I 2022-10-11 07:35:07] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00298 [I 2022-10-11 07:35:07] ...trap/engines/engine.py.230: loss: 0.09071 [I 2022-10-11 07:35:07] ...optimizers/trijoint.py.151: total_norm: 10.590597 [I 2022-10-11 07:35:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 230/2304 [I 2022-10-11 07:35:16] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:21 [I 2022-10-11 07:35:16] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00328 [I 2022-10-11 07:35:16] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-11 07:35:16] ...optimizers/trijoint.py.151: total_norm: 14.745296 [I 2022-10-11 07:35:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 240/2304 [I 2022-10-11 07:35:24] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:12 [I 2022-10-11 07:35:24] ...trap/engines/engine.py.229: process: 0.84175 | load: 0.00299 [I 2022-10-11 07:35:24] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-11 07:35:24] ...optimizers/trijoint.py.151: total_norm: 8.475004 [I 2022-10-11 07:35:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 250/2304 [I 2022-10-11 07:35:33] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:58 [I 2022-10-11 07:35:33] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00297 [I 2022-10-11 07:35:33] ...trap/engines/engine.py.230: loss: 0.19669 [I 2022-10-11 07:35:33] ...optimizers/trijoint.py.151: total_norm: 38.585670 [I 2022-10-11 07:35:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 260/2304 [I 2022-10-11 07:35:41] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:39 [I 2022-10-11 07:35:41] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00297 [I 2022-10-11 07:35:41] ...trap/engines/engine.py.230: loss: 0.09210 [I 2022-10-11 07:35:41] ...optimizers/trijoint.py.151: total_norm: 15.463371 [I 2022-10-11 07:35:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 270/2304 [I 2022-10-11 07:35:49] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:13 [I 2022-10-11 07:35:49] ...trap/engines/engine.py.229: process: 0.83135 | load: 0.00310 [I 2022-10-11 07:35:49] ...trap/engines/engine.py.230: loss: 0.16688 [I 2022-10-11 07:35:49] ...optimizers/trijoint.py.151: total_norm: 15.540807 [I 2022-10-11 07:35:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 280/2304 [I 2022-10-11 07:35:58] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:21 [I 2022-10-11 07:35:58] ...trap/engines/engine.py.229: process: 0.83739 | load: 0.00318 [I 2022-10-11 07:35:58] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-11 07:35:58] ...optimizers/trijoint.py.151: total_norm: 14.097631 [I 2022-10-11 07:36:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 290/2304 [I 2022-10-11 07:36:06] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:28 [I 2022-10-11 07:36:06] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00298 [I 2022-10-11 07:36:06] ...trap/engines/engine.py.230: loss: 0.14346 [I 2022-10-11 07:36:06] ...optimizers/trijoint.py.151: total_norm: 13.634646 [I 2022-10-11 07:36:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 300/2304 [I 2022-10-11 07:36:15] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:00 [I 2022-10-11 07:36:15] ...trap/engines/engine.py.229: process: 0.83113 | load: 0.00303 [I 2022-10-11 07:36:15] ...trap/engines/engine.py.230: loss: 0.09964 [I 2022-10-11 07:36:15] ...optimizers/trijoint.py.151: total_norm: 12.369668 [I 2022-10-11 07:36:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 310/2304 [I 2022-10-11 07:36:23] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:28:09 [I 2022-10-11 07:36:23] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00493 [I 2022-10-11 07:36:23] ...trap/engines/engine.py.230: loss: 0.16817 [I 2022-10-11 07:36:23] ...optimizers/trijoint.py.151: total_norm: 9.146976 [I 2022-10-11 07:36:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 320/2304 [I 2022-10-11 07:36:32] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:28:00 [I 2022-10-11 07:36:32] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00313 [I 2022-10-11 07:36:32] ...trap/engines/engine.py.230: loss: 0.11336 [I 2022-10-11 07:36:32] ...optimizers/trijoint.py.151: total_norm: 14.206994 [I 2022-10-11 07:36:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 330/2304 [I 2022-10-11 07:36:40] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:31 [I 2022-10-11 07:36:40] ...trap/engines/engine.py.229: process: 0.83264 | load: 0.00301 [I 2022-10-11 07:36:40] ...trap/engines/engine.py.230: loss: 0.09642 [I 2022-10-11 07:36:40] ...optimizers/trijoint.py.151: total_norm: 10.778093 [I 2022-10-11 07:36:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 340/2304 [I 2022-10-11 07:36:49] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:36 [I 2022-10-11 07:36:49] ...trap/engines/engine.py.229: process: 0.83821 | load: 0.00301 [I 2022-10-11 07:36:49] ...trap/engines/engine.py.230: loss: 0.08612 [I 2022-10-11 07:36:49] ...optimizers/trijoint.py.151: total_norm: 29.570290 [I 2022-10-11 07:36:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 350/2304 [I 2022-10-11 07:36:57] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:23 [I 2022-10-11 07:36:57] ...trap/engines/engine.py.229: process: 0.83330 | load: 0.00308 [I 2022-10-11 07:36:57] ...trap/engines/engine.py.230: loss: 0.11595 [I 2022-10-11 07:36:57] ...optimizers/trijoint.py.151: total_norm: 10.593837 [I 2022-10-11 07:37:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 360/2304 [I 2022-10-11 07:37:06] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:28:12 [I 2022-10-11 07:37:06] ...trap/engines/engine.py.229: process: 0.88275 | load: 0.00374 [I 2022-10-11 07:37:06] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-11 07:37:06] ...optimizers/trijoint.py.151: total_norm: 8.502848 [I 2022-10-11 07:37:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 370/2304 [I 2022-10-11 07:37:14] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:28:10 [I 2022-10-11 07:37:14] ...trap/engines/engine.py.229: process: 0.87694 | load: 0.00410 [I 2022-10-11 07:37:14] ...trap/engines/engine.py.230: loss: 0.16585 [I 2022-10-11 07:37:14] ...optimizers/trijoint.py.151: total_norm: 18.112432 [I 2022-10-11 07:37:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 380/2304 [I 2022-10-11 07:37:23] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:27:50 [I 2022-10-11 07:37:23] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00301 [I 2022-10-11 07:37:23] ...trap/engines/engine.py.230: loss: 0.08561 [I 2022-10-11 07:37:23] ...optimizers/trijoint.py.151: total_norm: 10.912197 [I 2022-10-11 07:37:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 390/2304 [I 2022-10-11 07:37:32] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:27:03 [I 2022-10-11 07:37:32] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00311 [I 2022-10-11 07:37:32] ...trap/engines/engine.py.230: loss: 0.15113 [I 2022-10-11 07:37:32] ...optimizers/trijoint.py.151: total_norm: 14.255227 [I 2022-10-11 07:37:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 400/2304 [I 2022-10-11 07:37:40] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:26:45 [I 2022-10-11 07:37:40] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00297 [I 2022-10-11 07:37:40] ...trap/engines/engine.py.230: loss: 0.12585 [I 2022-10-11 07:37:40] ...optimizers/trijoint.py.151: total_norm: 11.493210 [I 2022-10-11 07:37:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 410/2304 [I 2022-10-11 07:37:49] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:26:56 [I 2022-10-11 07:37:49] ...trap/engines/engine.py.229: process: 0.86907 | load: 0.00389 [I 2022-10-11 07:37:49] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-11 07:37:49] ...optimizers/trijoint.py.151: total_norm: 14.816089 [I 2022-10-11 07:37:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 420/2304 [I 2022-10-11 07:37:57] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:26 [I 2022-10-11 07:37:57] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00299 [I 2022-10-11 07:37:57] ...trap/engines/engine.py.230: loss: 0.09270 [I 2022-10-11 07:37:57] ...optimizers/trijoint.py.151: total_norm: 19.110025 [I 2022-10-11 07:38:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 430/2304 [I 2022-10-11 07:38:05] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:26:15 [I 2022-10-11 07:38:05] ...trap/engines/engine.py.229: process: 0.83966 | load: 0.00297 [I 2022-10-11 07:38:05] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-11 07:38:05] ...optimizers/trijoint.py.151: total_norm: 12.581419 [I 2022-10-11 07:38:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 440/2304 [I 2022-10-11 07:38:14] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:26:10 [I 2022-10-11 07:38:14] ...trap/engines/engine.py.229: process: 0.84071 | load: 0.00298 [I 2022-10-11 07:38:14] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-11 07:38:14] ...optimizers/trijoint.py.151: total_norm: 22.321615 [I 2022-10-11 07:38:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 450/2304 [I 2022-10-11 07:38:22] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:08 [I 2022-10-11 07:38:22] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00304 [I 2022-10-11 07:38:22] ...trap/engines/engine.py.230: loss: 0.09837 [I 2022-10-11 07:38:22] ...optimizers/trijoint.py.151: total_norm: 12.059978 [I 2022-10-11 07:38:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 460/2304 [I 2022-10-11 07:38:31] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:25:50 [I 2022-10-11 07:38:31] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00297 [I 2022-10-11 07:38:31] ...trap/engines/engine.py.230: loss: 0.24727 [I 2022-10-11 07:38:31] ...optimizers/trijoint.py.151: total_norm: 38.413967 [I 2022-10-11 07:38:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 470/2304 [I 2022-10-11 07:38:39] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:25:43 [I 2022-10-11 07:38:39] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00297 [I 2022-10-11 07:38:39] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-11 07:38:39] ...optimizers/trijoint.py.151: total_norm: 14.071419 [I 2022-10-11 07:38:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 480/2304 [I 2022-10-11 07:38:48] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:42 [I 2022-10-11 07:38:48] ...trap/engines/engine.py.229: process: 0.87906 | load: 0.00306 [I 2022-10-11 07:38:48] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-11 07:38:48] ...optimizers/trijoint.py.151: total_norm: 28.669123 [I 2022-10-11 07:38:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 490/2304 [I 2022-10-11 07:38:56] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:36 [I 2022-10-11 07:38:56] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00347 [I 2022-10-11 07:38:56] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-11 07:38:56] ...optimizers/trijoint.py.151: total_norm: 33.143856 [I 2022-10-11 07:39:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 500/2304 [I 2022-10-11 07:39:05] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:29 [I 2022-10-11 07:39:05] ...trap/engines/engine.py.229: process: 0.84246 | load: 0.00301 [I 2022-10-11 07:39:05] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-11 07:39:05] ...optimizers/trijoint.py.151: total_norm: 14.736133 [I 2022-10-11 07:39:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 510/2304 [I 2022-10-11 07:39:13] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:02 [I 2022-10-11 07:39:13] ...trap/engines/engine.py.229: process: 0.83393 | load: 0.00295 [I 2022-10-11 07:39:13] ...trap/engines/engine.py.230: loss: 0.14683 [I 2022-10-11 07:39:13] ...optimizers/trijoint.py.151: total_norm: 53.345509 [I 2022-10-11 07:39:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 520/2304 [I 2022-10-11 07:39:21] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:24:49 [I 2022-10-11 07:39:21] ...trap/engines/engine.py.229: process: 0.83314 | load: 0.00297 [I 2022-10-11 07:39:21] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-11 07:39:21] ...optimizers/trijoint.py.151: total_norm: 11.571765 [I 2022-10-11 07:39:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 530/2304 [I 2022-10-11 07:39:30] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:07 [I 2022-10-11 07:39:30] ...trap/engines/engine.py.229: process: 0.86723 | load: 0.00297 [I 2022-10-11 07:39:30] ...trap/engines/engine.py.230: loss: 0.08662 [I 2022-10-11 07:39:30] ...optimizers/trijoint.py.151: total_norm: 10.585173 [I 2022-10-11 07:39:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 540/2304 [I 2022-10-11 07:39:38] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:59 [I 2022-10-11 07:39:38] ...trap/engines/engine.py.229: process: 0.84585 | load: 0.00296 [I 2022-10-11 07:39:38] ...trap/engines/engine.py.230: loss: 0.14890 [I 2022-10-11 07:39:38] ...optimizers/trijoint.py.151: total_norm: 14.773719 [I 2022-10-11 07:39:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 550/2304 [I 2022-10-11 07:39:47] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:24:47 [I 2022-10-11 07:39:47] ...trap/engines/engine.py.229: process: 0.86047 | load: 0.00298 [I 2022-10-11 07:39:47] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-11 07:39:47] ...optimizers/trijoint.py.151: total_norm: 63.345436 [I 2022-10-11 07:39:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 560/2304 [I 2022-10-11 07:39:55] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:24:30 [I 2022-10-11 07:39:55] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00296 [I 2022-10-11 07:39:55] ...trap/engines/engine.py.230: loss: 0.07036 [I 2022-10-11 07:39:55] ...optimizers/trijoint.py.151: total_norm: 8.760483 [I 2022-10-11 07:40:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 570/2304 [I 2022-10-11 07:40:04] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:17 [I 2022-10-11 07:40:04] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00303 [I 2022-10-11 07:40:04] ...trap/engines/engine.py.230: loss: 0.09102 [I 2022-10-11 07:40:04] ...optimizers/trijoint.py.151: total_norm: 14.851172 [I 2022-10-11 07:40:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 580/2304 [I 2022-10-11 07:40:12] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:04 [I 2022-10-11 07:40:12] ...trap/engines/engine.py.229: process: 0.81842 | load: 0.00295 [I 2022-10-11 07:40:12] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-11 07:40:12] ...optimizers/trijoint.py.151: total_norm: 11.996097 [I 2022-10-11 07:40:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 590/2304 [I 2022-10-11 07:40:21] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:24:14 [I 2022-10-11 07:40:21] ...trap/engines/engine.py.229: process: 0.86846 | load: 0.00297 [I 2022-10-11 07:40:21] ...trap/engines/engine.py.230: loss: 0.12222 [I 2022-10-11 07:40:21] ...optimizers/trijoint.py.151: total_norm: 15.907275 [I 2022-10-11 07:40:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 600/2304 [I 2022-10-11 07:40:29] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:24:01 [I 2022-10-11 07:40:29] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00294 [I 2022-10-11 07:40:29] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-11 07:40:29] ...optimizers/trijoint.py.151: total_norm: 13.557629 [I 2022-10-11 07:40:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 610/2304 [I 2022-10-11 07:40:37] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:23:43 [I 2022-10-11 07:40:37] ...trap/engines/engine.py.229: process: 0.84399 | load: 0.00300 [I 2022-10-11 07:40:37] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-11 07:40:37] ...optimizers/trijoint.py.151: total_norm: 11.298431 [I 2022-10-11 07:40:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 620/2304 [I 2022-10-11 07:40:46] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:39 [I 2022-10-11 07:40:46] ...trap/engines/engine.py.229: process: 0.84575 | load: 0.00296 [I 2022-10-11 07:40:46] ...trap/engines/engine.py.230: loss: 0.09880 [I 2022-10-11 07:40:46] ...optimizers/trijoint.py.151: total_norm: 18.865171 [I 2022-10-11 07:40:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 630/2304 [I 2022-10-11 07:40:54] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:23:35 [I 2022-10-11 07:40:54] ...trap/engines/engine.py.229: process: 0.85558 | load: 0.00308 [I 2022-10-11 07:40:54] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-11 07:40:54] ...optimizers/trijoint.py.151: total_norm: 36.135216 [I 2022-10-11 07:41:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 640/2304 [I 2022-10-11 07:41:03] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:17 [I 2022-10-11 07:41:03] ...trap/engines/engine.py.229: process: 0.84356 | load: 0.00300 [I 2022-10-11 07:41:03] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-11 07:41:03] ...optimizers/trijoint.py.151: total_norm: 9.811584 [I 2022-10-11 07:41:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 650/2304 [I 2022-10-11 07:41:11] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:23:14 [I 2022-10-11 07:41:11] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00297 [I 2022-10-11 07:41:11] ...trap/engines/engine.py.230: loss: 0.11283 [I 2022-10-11 07:41:11] ...optimizers/trijoint.py.151: total_norm: 8.079685 [I 2022-10-11 07:41:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 660/2304 [I 2022-10-11 07:41:20] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:02 [I 2022-10-11 07:41:20] ...trap/engines/engine.py.229: process: 0.84085 | load: 0.00300 [I 2022-10-11 07:41:20] ...trap/engines/engine.py.230: loss: 0.18286 [I 2022-10-11 07:41:20] ...optimizers/trijoint.py.151: total_norm: 11.556670 [I 2022-10-11 07:41:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 670/2304 [I 2022-10-11 07:41:28] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:23:00 [I 2022-10-11 07:41:28] ...trap/engines/engine.py.229: process: 0.83778 | load: 0.00297 [I 2022-10-11 07:41:28] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-11 07:41:28] ...optimizers/trijoint.py.151: total_norm: 13.592698 [I 2022-10-11 07:41:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 680/2304 [I 2022-10-11 07:41:37] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:22:52 [I 2022-10-11 07:41:37] ...trap/engines/engine.py.229: process: 0.86270 | load: 0.00302 [I 2022-10-11 07:41:37] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-11 07:41:37] ...optimizers/trijoint.py.151: total_norm: 16.037962 [I 2022-10-11 07:41:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 690/2304 [I 2022-10-11 07:41:45] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:22:47 [I 2022-10-11 07:41:45] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00300 [I 2022-10-11 07:41:45] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-11 07:41:45] ...optimizers/trijoint.py.151: total_norm: 19.791208 [I 2022-10-11 07:41:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 700/2304 [I 2022-10-11 07:41:54] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:45 [I 2022-10-11 07:41:54] ...trap/engines/engine.py.229: process: 0.85467 | load: 0.00300 [I 2022-10-11 07:41:54] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-11 07:41:54] ...optimizers/trijoint.py.151: total_norm: 9.071585 [I 2022-10-11 07:42:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 710/2304 [I 2022-10-11 07:42:02] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:22:26 [I 2022-10-11 07:42:02] ...trap/engines/engine.py.229: process: 0.83579 | load: 0.00297 [I 2022-10-11 07:42:02] ...trap/engines/engine.py.230: loss: 0.18353 [I 2022-10-11 07:42:02] ...optimizers/trijoint.py.151: total_norm: 23.055260 [I 2022-10-11 07:42:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 720/2304 [I 2022-10-11 07:42:11] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:31 [I 2022-10-11 07:42:11] ...trap/engines/engine.py.229: process: 0.86232 | load: 0.00297 [I 2022-10-11 07:42:11] ...trap/engines/engine.py.230: loss: 0.14737 [I 2022-10-11 07:42:11] ...optimizers/trijoint.py.151: total_norm: 19.403223 [I 2022-10-11 07:42:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 730/2304 [I 2022-10-11 07:42:19] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:22:07 [I 2022-10-11 07:42:19] ...trap/engines/engine.py.229: process: 0.85143 | load: 0.00296 [I 2022-10-11 07:42:19] ...trap/engines/engine.py.230: loss: 0.09712 [I 2022-10-11 07:42:19] ...optimizers/trijoint.py.151: total_norm: 15.664648 [I 2022-10-11 07:42:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 740/2304 [I 2022-10-11 07:42:27] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:46 [I 2022-10-11 07:42:27] ...trap/engines/engine.py.229: process: 0.83840 | load: 0.00296 [I 2022-10-11 07:42:27] ...trap/engines/engine.py.230: loss: 0.20407 [I 2022-10-11 07:42:27] ...optimizers/trijoint.py.151: total_norm: 59.175270 [I 2022-10-11 07:42:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 750/2304 [I 2022-10-11 07:42:36] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:21:44 [I 2022-10-11 07:42:36] ...trap/engines/engine.py.229: process: 0.82967 | load: 0.00297 [I 2022-10-11 07:42:36] ...trap/engines/engine.py.230: loss: 0.10252 [I 2022-10-11 07:42:36] ...optimizers/trijoint.py.151: total_norm: 8.558947 [I 2022-10-11 07:42:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 760/2304 [I 2022-10-11 07:42:44] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:21:57 [I 2022-10-11 07:42:44] ...trap/engines/engine.py.229: process: 0.85777 | load: 0.00306 [I 2022-10-11 07:42:44] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-11 07:42:44] ...optimizers/trijoint.py.151: total_norm: 15.599707 [I 2022-10-11 07:42:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 770/2304 [I 2022-10-11 07:42:53] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:21:28 [I 2022-10-11 07:42:53] ...trap/engines/engine.py.229: process: 0.81839 | load: 0.00296 [I 2022-10-11 07:42:53] ...trap/engines/engine.py.230: loss: 0.09299 [I 2022-10-11 07:42:53] ...optimizers/trijoint.py.151: total_norm: 12.484344 [I 2022-10-11 07:43:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 780/2304 [I 2022-10-11 07:43:01] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:25 [I 2022-10-11 07:43:01] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00296 [I 2022-10-11 07:43:01] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-11 07:43:01] ...optimizers/trijoint.py.151: total_norm: 11.050829 [I 2022-10-11 07:43:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 790/2304 [I 2022-10-11 07:43:10] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:17 [I 2022-10-11 07:43:10] ...trap/engines/engine.py.229: process: 0.84387 | load: 0.00294 [I 2022-10-11 07:43:10] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-11 07:43:10] ...optimizers/trijoint.py.151: total_norm: 11.757618 [I 2022-10-11 07:43:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 800/2304 [I 2022-10-11 07:43:18] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:21:06 [I 2022-10-11 07:43:18] ...trap/engines/engine.py.229: process: 0.85143 | load: 0.00296 [I 2022-10-11 07:43:18] ...trap/engines/engine.py.230: loss: 0.08814 [I 2022-10-11 07:43:18] ...optimizers/trijoint.py.151: total_norm: 11.920846 [I 2022-10-11 07:43:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 810/2304 [I 2022-10-11 07:43:27] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:21:02 [I 2022-10-11 07:43:27] ...trap/engines/engine.py.229: process: 0.84352 | load: 0.00295 [I 2022-10-11 07:43:27] ...trap/engines/engine.py.230: loss: 0.19945 [I 2022-10-11 07:43:27] ...optimizers/trijoint.py.151: total_norm: 22.357889 [I 2022-10-11 07:43:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 820/2304 [I 2022-10-11 07:43:35] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:20:42 [I 2022-10-11 07:43:35] ...trap/engines/engine.py.229: process: 0.84031 | load: 0.00297 [I 2022-10-11 07:43:35] ...trap/engines/engine.py.230: loss: 0.15655 [I 2022-10-11 07:43:35] ...optimizers/trijoint.py.151: total_norm: 42.392483 [I 2022-10-11 07:43:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 830/2304 [I 2022-10-11 07:43:43] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:45 [I 2022-10-11 07:43:43] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00295 [I 2022-10-11 07:43:43] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-11 07:43:43] ...optimizers/trijoint.py.151: total_norm: 17.523790 [I 2022-10-11 07:43:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 840/2304 [I 2022-10-11 07:43:52] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:20:39 [I 2022-10-11 07:43:52] ...trap/engines/engine.py.229: process: 0.84988 | load: 0.00293 [I 2022-10-11 07:43:52] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-11 07:43:52] ...optimizers/trijoint.py.151: total_norm: 16.752970 [I 2022-10-11 07:44:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 850/2304 [I 2022-10-11 07:44:00] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:22 [I 2022-10-11 07:44:00] ...trap/engines/engine.py.229: process: 0.83177 | load: 0.00305 [I 2022-10-11 07:44:00] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-11 07:44:00] ...optimizers/trijoint.py.151: total_norm: 19.054228 [I 2022-10-11 07:44:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 860/2304 [I 2022-10-11 07:44:09] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:20:23 [I 2022-10-11 07:44:09] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00297 [I 2022-10-11 07:44:09] ...trap/engines/engine.py.230: loss: 0.09905 [I 2022-10-11 07:44:09] ...optimizers/trijoint.py.151: total_norm: 17.217735 [I 2022-10-11 07:44:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 870/2304 [I 2022-10-11 07:44:17] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:15 [I 2022-10-11 07:44:17] ...trap/engines/engine.py.229: process: 0.86559 | load: 0.00307 [I 2022-10-11 07:44:17] ...trap/engines/engine.py.230: loss: 0.08670 [I 2022-10-11 07:44:17] ...optimizers/trijoint.py.151: total_norm: 14.230959 [I 2022-10-11 07:44:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 880/2304 [I 2022-10-11 07:44:26] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:00 [I 2022-10-11 07:44:26] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00296 [I 2022-10-11 07:44:26] ...trap/engines/engine.py.230: loss: 0.10818 [I 2022-10-11 07:44:26] ...optimizers/trijoint.py.151: total_norm: 8.889289 [I 2022-10-11 07:44:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 890/2304 [I 2022-10-11 07:44:34] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:20:18 [I 2022-10-11 07:44:34] ...trap/engines/engine.py.229: process: 0.87991 | load: 0.00383 [I 2022-10-11 07:44:34] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-11 07:44:34] ...optimizers/trijoint.py.151: total_norm: 30.309317 [I 2022-10-11 07:44:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 900/2304 [I 2022-10-11 07:44:43] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:19:59 [I 2022-10-11 07:44:43] ...trap/engines/engine.py.229: process: 0.83524 | load: 0.00296 [I 2022-10-11 07:44:43] ...trap/engines/engine.py.230: loss: 0.14703 [I 2022-10-11 07:44:43] ...optimizers/trijoint.py.151: total_norm: 23.470638 [I 2022-10-11 07:44:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 910/2304 [I 2022-10-11 07:44:51] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:41 [I 2022-10-11 07:44:51] ...trap/engines/engine.py.229: process: 0.86090 | load: 0.00296 [I 2022-10-11 07:44:51] ...trap/engines/engine.py.230: loss: 0.21017 [I 2022-10-11 07:44:51] ...optimizers/trijoint.py.151: total_norm: 19.290268 [I 2022-10-11 07:45:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 920/2304 [I 2022-10-11 07:45:00] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:25 [I 2022-10-11 07:45:00] ...trap/engines/engine.py.229: process: 0.82419 | load: 0.00300 [I 2022-10-11 07:45:00] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-11 07:45:00] ...optimizers/trijoint.py.151: total_norm: 17.067175 [I 2022-10-11 07:45:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 930/2304 [I 2022-10-11 07:45:08] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:17 [I 2022-10-11 07:45:08] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00293 [I 2022-10-11 07:45:08] ...trap/engines/engine.py.230: loss: 0.11695 [I 2022-10-11 07:45:08] ...optimizers/trijoint.py.151: total_norm: 12.517389 [I 2022-10-11 07:45:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 940/2304 [I 2022-10-11 07:45:17] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:09 [I 2022-10-11 07:45:17] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00294 [I 2022-10-11 07:45:17] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-11 07:45:17] ...optimizers/trijoint.py.151: total_norm: 15.269912 [I 2022-10-11 07:45:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 950/2304 [I 2022-10-11 07:45:25] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:18:57 [I 2022-10-11 07:45:25] ...trap/engines/engine.py.229: process: 0.84045 | load: 0.00295 [I 2022-10-11 07:45:25] ...trap/engines/engine.py.230: loss: 0.12509 [I 2022-10-11 07:45:25] ...optimizers/trijoint.py.151: total_norm: 13.585245 [I 2022-10-11 07:45:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 960/2304 [I 2022-10-11 07:45:34] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:18:52 [I 2022-10-11 07:45:34] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00297 [I 2022-10-11 07:45:34] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-11 07:45:34] ...optimizers/trijoint.py.151: total_norm: 11.215226 [I 2022-10-11 07:45:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 970/2304 [I 2022-10-11 07:45:42] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:18:33 [I 2022-10-11 07:45:42] ...trap/engines/engine.py.229: process: 0.82771 | load: 0.00303 [I 2022-10-11 07:45:42] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-11 07:45:42] ...optimizers/trijoint.py.151: total_norm: 19.938091 [I 2022-10-11 07:45:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 980/2304 [I 2022-10-11 07:45:50] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:18:37 [I 2022-10-11 07:45:50] ...trap/engines/engine.py.229: process: 0.84896 | load: 0.00296 [I 2022-10-11 07:45:50] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-11 07:45:50] ...optimizers/trijoint.py.151: total_norm: 14.315677 [I 2022-10-11 07:45:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 990/2304 [I 2022-10-11 07:45:59] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:30 [I 2022-10-11 07:45:59] ...trap/engines/engine.py.229: process: 0.83911 | load: 0.00303 [I 2022-10-11 07:45:59] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-11 07:45:59] ...optimizers/trijoint.py.151: total_norm: 10.655341 [I 2022-10-11 07:46:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1000/2304 [I 2022-10-11 07:46:07] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:23 [I 2022-10-11 07:46:07] ...trap/engines/engine.py.229: process: 0.84463 | load: 0.00297 [I 2022-10-11 07:46:07] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-11 07:46:07] ...optimizers/trijoint.py.151: total_norm: 12.017719 [I 2022-10-11 07:46:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1010/2304 [I 2022-10-11 07:46:16] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:24 [I 2022-10-11 07:46:16] ...trap/engines/engine.py.229: process: 0.89428 | load: 0.00375 [I 2022-10-11 07:46:16] ...trap/engines/engine.py.230: loss: 0.11085 [I 2022-10-11 07:46:16] ...optimizers/trijoint.py.151: total_norm: 16.735834 [I 2022-10-11 07:46:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1020/2304 [I 2022-10-11 07:46:25] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:18:37 [I 2022-10-11 07:46:25] ...trap/engines/engine.py.229: process: 0.88429 | load: 0.00376 [I 2022-10-11 07:46:25] ...trap/engines/engine.py.230: loss: 0.12853 [I 2022-10-11 07:46:25] ...optimizers/trijoint.py.151: total_norm: 21.038248 [I 2022-10-11 07:46:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1030/2304 [I 2022-10-11 07:46:33] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:08 [I 2022-10-11 07:46:33] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00298 [I 2022-10-11 07:46:33] ...trap/engines/engine.py.230: loss: 0.10836 [I 2022-10-11 07:46:33] ...optimizers/trijoint.py.151: total_norm: 13.617848 [I 2022-10-11 07:46:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 1040/2304 [I 2022-10-11 07:46:42] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:52 [I 2022-10-11 07:46:42] ...trap/engines/engine.py.229: process: 0.86704 | load: 0.00306 [I 2022-10-11 07:46:42] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-11 07:46:42] ...optimizers/trijoint.py.151: total_norm: 11.700398 [I 2022-10-11 07:46:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 1050/2304 [I 2022-10-11 07:46:50] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:48 [I 2022-10-11 07:46:50] ...trap/engines/engine.py.229: process: 0.85129 | load: 0.00294 [I 2022-10-11 07:46:50] ...trap/engines/engine.py.230: loss: 0.09428 [I 2022-10-11 07:46:50] ...optimizers/trijoint.py.151: total_norm: 13.705643 [I 2022-10-11 07:46:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 1060/2304 [I 2022-10-11 07:46:59] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:28 [I 2022-10-11 07:46:59] ...trap/engines/engine.py.229: process: 0.84540 | load: 0.00295 [I 2022-10-11 07:46:59] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-11 07:46:59] ...optimizers/trijoint.py.151: total_norm: 15.232985 [I 2022-10-11 07:47:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1070/2304 [I 2022-10-11 07:47:07] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:20 [I 2022-10-11 07:47:07] ...trap/engines/engine.py.229: process: 0.84996 | load: 0.00296 [I 2022-10-11 07:47:07] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-11 07:47:07] ...optimizers/trijoint.py.151: total_norm: 11.424847 [I 2022-10-11 07:47:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1080/2304 [I 2022-10-11 07:47:16] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:17:15 [I 2022-10-11 07:47:16] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00297 [I 2022-10-11 07:47:16] ...trap/engines/engine.py.230: loss: 0.11752 [I 2022-10-11 07:47:16] ...optimizers/trijoint.py.151: total_norm: 19.208939 [I 2022-10-11 07:47:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1090/2304 [I 2022-10-11 07:47:24] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:02 [I 2022-10-11 07:47:24] ...trap/engines/engine.py.229: process: 0.83469 | load: 0.00303 [I 2022-10-11 07:47:24] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-11 07:47:24] ...optimizers/trijoint.py.151: total_norm: 12.832914 [I 2022-10-11 07:47:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1100/2304 [I 2022-10-11 07:47:32] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:51 [I 2022-10-11 07:47:32] ...trap/engines/engine.py.229: process: 0.84461 | load: 0.00296 [I 2022-10-11 07:47:32] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-11 07:47:32] ...optimizers/trijoint.py.151: total_norm: 18.766102 [I 2022-10-11 07:47:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1110/2304 [I 2022-10-11 07:47:41] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:51 [I 2022-10-11 07:47:41] ...trap/engines/engine.py.229: process: 0.85045 | load: 0.00293 [I 2022-10-11 07:47:41] ...trap/engines/engine.py.230: loss: 0.08904 [I 2022-10-11 07:47:41] ...optimizers/trijoint.py.151: total_norm: 12.459641 [I 2022-10-11 07:48:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1120/2304 [I 2022-10-11 07:48:07] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:28:22 [I 2022-10-11 07:48:07] ...trap/engines/engine.py.229: process: 0.84443 | load: 0.00294 [I 2022-10-11 07:48:07] ...trap/engines/engine.py.230: loss: 0.07615 [I 2022-10-11 07:48:07] ...optimizers/trijoint.py.151: total_norm: 16.214094 [I 2022-10-11 07:48:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1130/2304 [I 2022-10-11 07:48:16] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:17:52 [I 2022-10-11 07:48:16] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00375 [I 2022-10-11 07:48:16] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-11 07:48:16] ...optimizers/trijoint.py.151: total_norm: 20.540363 [I 2022-10-11 07:48:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1140/2304 [I 2022-10-11 07:48:24] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:48 [I 2022-10-11 07:48:24] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00297 [I 2022-10-11 07:48:24] ...trap/engines/engine.py.230: loss: 0.20640 [I 2022-10-11 07:48:24] ...optimizers/trijoint.py.151: total_norm: 32.569809 [I 2022-10-11 07:48:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1150/2304 [I 2022-10-11 07:48:33] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:23 [I 2022-10-11 07:48:33] ...trap/engines/engine.py.229: process: 0.86473 | load: 0.00377 [I 2022-10-11 07:48:33] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-11 07:48:33] ...optimizers/trijoint.py.151: total_norm: 33.796646 [I 2022-10-11 07:48:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1160/2304 [I 2022-10-11 07:48:41] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:36 [I 2022-10-11 07:48:41] ...trap/engines/engine.py.229: process: 0.88189 | load: 0.00375 [I 2022-10-11 07:48:41] ...trap/engines/engine.py.230: loss: 0.11009 [I 2022-10-11 07:48:41] ...optimizers/trijoint.py.151: total_norm: 11.641001 [I 2022-10-11 07:48:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 1170/2304 [I 2022-10-11 07:48:50] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:16:31 [I 2022-10-11 07:48:50] ...trap/engines/engine.py.229: process: 0.86407 | load: 0.00386 [I 2022-10-11 07:48:50] ...trap/engines/engine.py.230: loss: 0.08336 [I 2022-10-11 07:48:50] ...optimizers/trijoint.py.151: total_norm: 8.088580 [I 2022-10-11 07:48:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 1180/2304 [I 2022-10-11 07:48:59] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:15:52 [I 2022-10-11 07:48:59] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00306 [I 2022-10-11 07:48:59] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-11 07:48:59] ...optimizers/trijoint.py.151: total_norm: 18.180696 [I 2022-10-11 07:49:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1190/2304 [I 2022-10-11 07:49:07] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:46 [I 2022-10-11 07:49:07] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00292 [I 2022-10-11 07:49:07] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-11 07:49:07] ...optimizers/trijoint.py.151: total_norm: 16.144836 [I 2022-10-11 07:49:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1200/2304 [I 2022-10-11 07:49:16] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:52 [I 2022-10-11 07:49:16] ...trap/engines/engine.py.229: process: 0.86918 | load: 0.00378 [I 2022-10-11 07:49:16] ...trap/engines/engine.py.230: loss: 0.17746 [I 2022-10-11 07:49:16] ...optimizers/trijoint.py.151: total_norm: 21.075418 [I 2022-10-11 07:49:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1210/2304 [I 2022-10-11 07:49:25] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:55 [I 2022-10-11 07:49:25] ...trap/engines/engine.py.229: process: 0.87416 | load: 0.00377 [I 2022-10-11 07:49:25] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-11 07:49:25] ...optimizers/trijoint.py.151: total_norm: 38.771671 [I 2022-10-11 07:49:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1220/2304 [I 2022-10-11 07:49:33] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:45 [I 2022-10-11 07:49:33] ...trap/engines/engine.py.229: process: 0.87133 | load: 0.00373 [I 2022-10-11 07:49:33] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-11 07:49:33] ...optimizers/trijoint.py.151: total_norm: 13.460150 [I 2022-10-11 07:49:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 1230/2304 [I 2022-10-11 07:49:42] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:39 [I 2022-10-11 07:49:42] ...trap/engines/engine.py.229: process: 0.87885 | load: 0.00377 [I 2022-10-11 07:49:42] ...trap/engines/engine.py.230: loss: 0.08579 [I 2022-10-11 07:49:42] ...optimizers/trijoint.py.151: total_norm: 9.865964 [I 2022-10-11 07:49:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1240/2304 [I 2022-10-11 07:49:51] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:27 [I 2022-10-11 07:49:51] ...trap/engines/engine.py.229: process: 0.87660 | load: 0.00377 [I 2022-10-11 07:49:51] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-11 07:49:51] ...optimizers/trijoint.py.151: total_norm: 12.044395 [I 2022-10-11 07:50:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1250/2304 [I 2022-10-11 07:50:00] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:15:22 [I 2022-10-11 07:50:00] ...trap/engines/engine.py.229: process: 0.86696 | load: 0.00388 [I 2022-10-11 07:50:00] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-11 07:50:00] ...optimizers/trijoint.py.151: total_norm: 20.645874 [I 2022-10-11 07:50:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1260/2304 [I 2022-10-11 07:50:08] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:15:15 [I 2022-10-11 07:50:08] ...trap/engines/engine.py.229: process: 0.87111 | load: 0.00377 [I 2022-10-11 07:50:08] ...trap/engines/engine.py.230: loss: 0.10385 [I 2022-10-11 07:50:08] ...optimizers/trijoint.py.151: total_norm: 20.453094 [I 2022-10-11 07:50:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 1270/2304 [I 2022-10-11 07:50:17] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:15:06 [I 2022-10-11 07:50:17] ...trap/engines/engine.py.229: process: 0.86515 | load: 0.00376 [I 2022-10-11 07:50:17] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-11 07:50:17] ...optimizers/trijoint.py.151: total_norm: 24.424068 [I 2022-10-11 07:50:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1280/2304 [I 2022-10-11 07:50:26] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:36 [I 2022-10-11 07:50:26] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00296 [I 2022-10-11 07:50:26] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-11 07:50:26] ...optimizers/trijoint.py.151: total_norm: 11.843140 [I 2022-10-11 07:50:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1290/2304 [I 2022-10-11 07:50:34] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:21 [I 2022-10-11 07:50:34] ...trap/engines/engine.py.229: process: 0.85755 | load: 0.00296 [I 2022-10-11 07:50:34] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-11 07:50:34] ...optimizers/trijoint.py.151: total_norm: 11.281671 [I 2022-10-11 07:50:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 1300/2304 [I 2022-10-11 07:50:43] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:02 [I 2022-10-11 07:50:43] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00304 [I 2022-10-11 07:50:43] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-11 07:50:43] ...optimizers/trijoint.py.151: total_norm: 25.752270 [I 2022-10-11 07:50:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1310/2304 [I 2022-10-11 07:50:51] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:14:00 [I 2022-10-11 07:50:51] ...trap/engines/engine.py.229: process: 0.85229 | load: 0.00294 [I 2022-10-11 07:50:51] ...trap/engines/engine.py.230: loss: 0.15662 [I 2022-10-11 07:50:51] ...optimizers/trijoint.py.151: total_norm: 10.740898 [I 2022-10-11 07:51:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1320/2304 [I 2022-10-11 07:51:00] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:52 [I 2022-10-11 07:51:00] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00296 [I 2022-10-11 07:51:00] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-11 07:51:00] ...optimizers/trijoint.py.151: total_norm: 25.493757 [I 2022-10-11 07:51:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1330/2304 [I 2022-10-11 07:51:08] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:41 [I 2022-10-11 07:51:08] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00294 [I 2022-10-11 07:51:08] ...trap/engines/engine.py.230: loss: 0.09309 [I 2022-10-11 07:51:08] ...optimizers/trijoint.py.151: total_norm: 20.466902 [I 2022-10-11 07:51:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1340/2304 [I 2022-10-11 07:51:16] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:37 [I 2022-10-11 07:51:16] ...trap/engines/engine.py.229: process: 0.87041 | load: 0.00377 [I 2022-10-11 07:51:16] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-11 07:51:16] ...optimizers/trijoint.py.151: total_norm: 35.145313 [I 2022-10-11 07:51:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1350/2304 [I 2022-10-11 07:51:25] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:51 [I 2022-10-11 07:51:25] ...trap/engines/engine.py.229: process: 0.86556 | load: 0.00379 [I 2022-10-11 07:51:25] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-11 07:51:25] ...optimizers/trijoint.py.151: total_norm: 21.139404 [I 2022-10-11 07:51:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1360/2304 [I 2022-10-11 07:51:34] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:45 [I 2022-10-11 07:51:34] ...trap/engines/engine.py.229: process: 0.87844 | load: 0.00376 [I 2022-10-11 07:51:34] ...trap/engines/engine.py.230: loss: 0.09787 [I 2022-10-11 07:51:34] ...optimizers/trijoint.py.151: total_norm: 14.106775 [I 2022-10-11 07:51:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 1370/2304 [I 2022-10-11 07:51:43] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:33 [I 2022-10-11 07:51:43] ...trap/engines/engine.py.229: process: 0.87009 | load: 0.00373 [I 2022-10-11 07:51:43] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-11 07:51:43] ...optimizers/trijoint.py.151: total_norm: 63.161465 [I 2022-10-11 07:51:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1380/2304 [I 2022-10-11 07:51:51] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:01 [I 2022-10-11 07:51:51] ...trap/engines/engine.py.229: process: 0.84813 | load: 0.00297 [I 2022-10-11 07:51:51] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-11 07:51:51] ...optimizers/trijoint.py.151: total_norm: 65.482552 [I 2022-10-11 07:52:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1390/2304 [I 2022-10-11 07:52:00] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:13:10 [I 2022-10-11 07:52:00] ...trap/engines/engine.py.229: process: 0.87243 | load: 0.00374 [I 2022-10-11 07:52:00] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-11 07:52:00] ...optimizers/trijoint.py.151: total_norm: 14.470906 [I 2022-10-11 07:52:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1400/2304 [I 2022-10-11 07:52:08] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:13:12 [I 2022-10-11 07:52:08] ...trap/engines/engine.py.229: process: 0.89567 | load: 0.00381 [I 2022-10-11 07:52:08] ...trap/engines/engine.py.230: loss: 0.08921 [I 2022-10-11 07:52:08] ...optimizers/trijoint.py.151: total_norm: 20.558689 [I 2022-10-11 07:52:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 1410/2304 [I 2022-10-11 07:52:17] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:13:00 [I 2022-10-11 07:52:17] ...trap/engines/engine.py.229: process: 0.87789 | load: 0.00388 [I 2022-10-11 07:52:17] ...trap/engines/engine.py.230: loss: 0.10377 [I 2022-10-11 07:52:17] ...optimizers/trijoint.py.151: total_norm: 13.380274 [I 2022-10-11 07:52:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1420/2304 [I 2022-10-11 07:52:26] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:55 [I 2022-10-11 07:52:26] ...trap/engines/engine.py.229: process: 0.87523 | load: 0.00375 [I 2022-10-11 07:52:26] ...trap/engines/engine.py.230: loss: 0.10162 [I 2022-10-11 07:52:26] ...optimizers/trijoint.py.151: total_norm: 10.831190 [I 2022-10-11 07:52:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1430/2304 [I 2022-10-11 07:52:34] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:16 [I 2022-10-11 07:52:34] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00297 [I 2022-10-11 07:52:34] ...trap/engines/engine.py.230: loss: 0.11952 [I 2022-10-11 07:52:34] ...optimizers/trijoint.py.151: total_norm: 18.336815 [I 2022-10-11 07:52:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 1440/2304 [I 2022-10-11 07:52:43] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:01 [I 2022-10-11 07:52:43] ...trap/engines/engine.py.229: process: 0.83396 | load: 0.00296 [I 2022-10-11 07:52:43] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-11 07:52:43] ...optimizers/trijoint.py.151: total_norm: 17.727636 [I 2022-10-11 07:52:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1450/2304 [I 2022-10-11 07:52:51] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:12:02 [I 2022-10-11 07:52:51] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00294 [I 2022-10-11 07:52:51] ...trap/engines/engine.py.230: loss: 0.15868 [I 2022-10-11 07:52:51] ...optimizers/trijoint.py.151: total_norm: 23.264572 [I 2022-10-11 07:53:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1460/2304 [I 2022-10-11 07:53:00] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:11:53 [I 2022-10-11 07:53:00] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00296 [I 2022-10-11 07:53:00] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-11 07:53:00] ...optimizers/trijoint.py.151: total_norm: 12.964440 [I 2022-10-11 07:53:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1470/2304 [I 2022-10-11 07:53:08] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:51 [I 2022-10-11 07:53:08] ...trap/engines/engine.py.229: process: 0.84572 | load: 0.00297 [I 2022-10-11 07:53:08] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-11 07:53:08] ...optimizers/trijoint.py.151: total_norm: 10.531759 [I 2022-10-11 07:53:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 1480/2304 [I 2022-10-11 07:53:17] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:38 [I 2022-10-11 07:53:17] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00296 [I 2022-10-11 07:53:17] ...trap/engines/engine.py.230: loss: 0.10242 [I 2022-10-11 07:53:17] ...optimizers/trijoint.py.151: total_norm: 14.363688 [I 2022-10-11 07:53:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1490/2304 [I 2022-10-11 07:53:25] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:27 [I 2022-10-11 07:53:25] ...trap/engines/engine.py.229: process: 0.83789 | load: 0.00296 [I 2022-10-11 07:53:25] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-11 07:53:25] ...optimizers/trijoint.py.151: total_norm: 150.279617 [I 2022-10-11 07:53:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1500/2304 [I 2022-10-11 07:53:34] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:37 [I 2022-10-11 07:53:34] ...trap/engines/engine.py.229: process: 0.87109 | load: 0.00302 [I 2022-10-11 07:53:34] ...trap/engines/engine.py.230: loss: 0.11998 [I 2022-10-11 07:53:34] ...optimizers/trijoint.py.151: total_norm: 8.960071 [I 2022-10-11 07:53:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 1510/2304 [I 2022-10-11 07:53:42] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:13 [I 2022-10-11 07:53:42] ...trap/engines/engine.py.229: process: 0.84499 | load: 0.00293 [I 2022-10-11 07:53:42] ...trap/engines/engine.py.230: loss: 0.09490 [I 2022-10-11 07:53:42] ...optimizers/trijoint.py.151: total_norm: 10.973343 [I 2022-10-11 07:53:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1520/2304 [I 2022-10-11 07:53:51] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:11:57 [I 2022-10-11 07:53:51] ...trap/engines/engine.py.229: process: 1.20565 | load: 0.34578 [I 2022-10-11 07:53:51] ...trap/engines/engine.py.230: loss: 0.08097 [I 2022-10-11 07:53:51] ...optimizers/trijoint.py.151: total_norm: 15.072486 [I 2022-10-11 07:54:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1530/2304 [I 2022-10-11 07:54:00] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:54 [I 2022-10-11 07:54:00] ...trap/engines/engine.py.229: process: 0.82581 | load: 0.00293 [I 2022-10-11 07:54:00] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-11 07:54:00] ...optimizers/trijoint.py.151: total_norm: 14.067467 [I 2022-10-11 07:54:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1540/2304 [I 2022-10-11 07:54:08] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:51 [I 2022-10-11 07:54:08] ...trap/engines/engine.py.229: process: 0.88143 | load: 0.00375 [I 2022-10-11 07:54:08] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-11 07:54:08] ...optimizers/trijoint.py.151: total_norm: 8.560211 [I 2022-10-11 07:54:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 1550/2304 [I 2022-10-11 07:54:17] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:11:00 [I 2022-10-11 07:54:17] ...trap/engines/engine.py.229: process: 0.88356 | load: 0.00376 [I 2022-10-11 07:54:17] ...trap/engines/engine.py.230: loss: 0.10524 [I 2022-10-11 07:54:17] ...optimizers/trijoint.py.151: total_norm: 12.284203 [I 2022-10-11 07:54:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1560/2304 [I 2022-10-11 07:54:26] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:54 [I 2022-10-11 07:54:26] ...trap/engines/engine.py.229: process: 0.87067 | load: 0.00377 [I 2022-10-11 07:54:26] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-11 07:54:26] ...optimizers/trijoint.py.151: total_norm: 17.237551 [I 2022-10-11 07:54:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1570/2304 [I 2022-10-11 07:54:34] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:23 [I 2022-10-11 07:54:34] ...trap/engines/engine.py.229: process: 0.83647 | load: 0.00304 [I 2022-10-11 07:54:34] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-11 07:54:34] ...optimizers/trijoint.py.151: total_norm: 16.025862 [I 2022-10-11 07:54:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 1580/2304 [I 2022-10-11 07:54:43] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:10:11 [I 2022-10-11 07:54:43] ...trap/engines/engine.py.229: process: 0.83181 | load: 0.00297 [I 2022-10-11 07:54:43] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-11 07:54:43] ...optimizers/trijoint.py.151: total_norm: 17.832178 [I 2022-10-11 07:54:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1590/2304 [I 2022-10-11 07:54:51] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:04 [I 2022-10-11 07:54:51] ...trap/engines/engine.py.229: process: 0.84723 | load: 0.00300 [I 2022-10-11 07:54:51] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-11 07:54:51] ...optimizers/trijoint.py.151: total_norm: 11.074399 [I 2022-10-11 07:55:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1600/2304 [I 2022-10-11 07:55:00] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:56 [I 2022-10-11 07:55:00] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00298 [I 2022-10-11 07:55:00] ...trap/engines/engine.py.230: loss: 0.18682 [I 2022-10-11 07:55:00] ...optimizers/trijoint.py.151: total_norm: 90.141701 [I 2022-10-11 07:55:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1610/2304 [I 2022-10-11 07:55:08] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:42 [I 2022-10-11 07:55:08] ...trap/engines/engine.py.229: process: 0.81828 | load: 0.00298 [I 2022-10-11 07:55:08] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-11 07:55:08] ...optimizers/trijoint.py.151: total_norm: 17.750313 [I 2022-10-11 07:55:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1620/2304 [I 2022-10-11 07:55:16] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:34 [I 2022-10-11 07:55:16] ...trap/engines/engine.py.229: process: 0.84413 | load: 0.00300 [I 2022-10-11 07:55:16] ...trap/engines/engine.py.230: loss: 0.11613 [I 2022-10-11 07:55:16] ...optimizers/trijoint.py.151: total_norm: 7.596663 [I 2022-10-11 07:55:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1630/2304 [I 2022-10-11 07:55:25] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:22 [I 2022-10-11 07:55:25] ...trap/engines/engine.py.229: process: 0.83068 | load: 0.00299 [I 2022-10-11 07:55:25] ...trap/engines/engine.py.230: loss: 0.08433 [I 2022-10-11 07:55:25] ...optimizers/trijoint.py.151: total_norm: 11.002724 [I 2022-10-11 07:55:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1640/2304 [I 2022-10-11 07:55:33] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:09:16 [I 2022-10-11 07:55:33] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00301 [I 2022-10-11 07:55:33] ...trap/engines/engine.py.230: loss: 0.14770 [I 2022-10-11 07:55:33] ...optimizers/trijoint.py.151: total_norm: 14.335897 [I 2022-10-11 07:55:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 1650/2304 [I 2022-10-11 07:55:42] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:09:12 [I 2022-10-11 07:55:42] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00302 [I 2022-10-11 07:55:42] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-11 07:55:42] ...optimizers/trijoint.py.151: total_norm: 13.219161 [I 2022-10-11 07:55:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 1660/2304 [I 2022-10-11 07:55:50] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:09:04 [I 2022-10-11 07:55:50] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00304 [I 2022-10-11 07:55:50] ...trap/engines/engine.py.230: loss: 0.12866 [I 2022-10-11 07:55:50] ...optimizers/trijoint.py.151: total_norm: 14.941332 [I 2022-10-11 07:55:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 1670/2304 [I 2022-10-11 07:55:59] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:08:54 [I 2022-10-11 07:55:59] ...trap/engines/engine.py.229: process: 0.83239 | load: 0.00296 [I 2022-10-11 07:55:59] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-11 07:55:59] ...optimizers/trijoint.py.151: total_norm: 12.269556 [I 2022-10-11 07:56:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1680/2304 [I 2022-10-11 07:56:07] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:45 [I 2022-10-11 07:56:07] ...trap/engines/engine.py.229: process: 0.84835 | load: 0.00297 [I 2022-10-11 07:56:07] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-11 07:56:07] ...optimizers/trijoint.py.151: total_norm: 10.986388 [I 2022-10-11 07:56:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1690/2304 [I 2022-10-11 07:56:15] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:08:39 [I 2022-10-11 07:56:15] ...trap/engines/engine.py.229: process: 0.84420 | load: 0.00298 [I 2022-10-11 07:56:15] ...trap/engines/engine.py.230: loss: 0.14782 [I 2022-10-11 07:56:15] ...optimizers/trijoint.py.151: total_norm: 9.591470 [I 2022-10-11 07:56:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1700/2304 [I 2022-10-11 07:56:24] ...trap/engines/engine.py.228: elapsed: 0:24:34 | left: 0:08:30 [I 2022-10-11 07:56:24] ...trap/engines/engine.py.229: process: 0.83705 | load: 0.00300 [I 2022-10-11 07:56:24] ...trap/engines/engine.py.230: loss: 0.16458 [I 2022-10-11 07:56:24] ...optimizers/trijoint.py.151: total_norm: 13.338547 [I 2022-10-11 07:56:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1710/2304 [I 2022-10-11 07:56:32] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:23 [I 2022-10-11 07:56:32] ...trap/engines/engine.py.229: process: 0.84795 | load: 0.00298 [I 2022-10-11 07:56:32] ...trap/engines/engine.py.230: loss: 0.21514 [I 2022-10-11 07:56:32] ...optimizers/trijoint.py.151: total_norm: 24.753939 [I 2022-10-11 07:56:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1720/2304 [I 2022-10-11 07:56:41] ...trap/engines/engine.py.228: elapsed: 0:24:51 | left: 0:08:15 [I 2022-10-11 07:56:41] ...trap/engines/engine.py.229: process: 0.85463 | load: 0.00295 [I 2022-10-11 07:56:41] ...trap/engines/engine.py.230: loss: 0.10681 [I 2022-10-11 07:56:41] ...optimizers/trijoint.py.151: total_norm: 9.618359 [I 2022-10-11 07:56:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1730/2304 [I 2022-10-11 07:56:49] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:06 [I 2022-10-11 07:56:49] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00304 [I 2022-10-11 07:56:49] ...trap/engines/engine.py.230: loss: 0.14918 [I 2022-10-11 07:56:49] ...optimizers/trijoint.py.151: total_norm: 23.540413 [I 2022-10-11 07:56:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1740/2304 [I 2022-10-11 07:56:58] ...trap/engines/engine.py.228: elapsed: 0:25:08 | left: 0:07:55 [I 2022-10-11 07:56:58] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00298 [I 2022-10-11 07:56:58] ...trap/engines/engine.py.230: loss: 0.09220 [I 2022-10-11 07:56:58] ...optimizers/trijoint.py.151: total_norm: 9.334756 [I 2022-10-11 07:57:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1750/2304 [I 2022-10-11 07:57:07] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:08:01 [I 2022-10-11 07:57:07] ...trap/engines/engine.py.229: process: 0.86120 | load: 0.00376 [I 2022-10-11 07:57:07] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-11 07:57:07] ...optimizers/trijoint.py.151: total_norm: 20.857504 [I 2022-10-11 07:57:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1760/2304 [I 2022-10-11 07:57:15] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:55 [I 2022-10-11 07:57:15] ...trap/engines/engine.py.229: process: 0.87386 | load: 0.00374 [I 2022-10-11 07:57:15] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-11 07:57:15] ...optimizers/trijoint.py.151: total_norm: 22.673574 [I 2022-10-11 07:57:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1770/2304 [I 2022-10-11 07:57:24] ...trap/engines/engine.py.228: elapsed: 0:25:34 | left: 0:07:31 [I 2022-10-11 07:57:24] ...trap/engines/engine.py.229: process: 0.83922 | load: 0.00300 [I 2022-10-11 07:57:24] ...trap/engines/engine.py.230: loss: 0.13358 [I 2022-10-11 07:57:24] ...optimizers/trijoint.py.151: total_norm: 26.492769 [I 2022-10-11 07:57:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1780/2304 [I 2022-10-11 07:57:32] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:25 [I 2022-10-11 07:57:32] ...trap/engines/engine.py.229: process: 0.86056 | load: 0.00295 [I 2022-10-11 07:57:32] ...trap/engines/engine.py.230: loss: 0.09736 [I 2022-10-11 07:57:32] ...optimizers/trijoint.py.151: total_norm: 23.926083 [I 2022-10-11 07:57:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1790/2304 [I 2022-10-11 07:57:41] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:07:09 [I 2022-10-11 07:57:41] ...trap/engines/engine.py.229: process: 0.82183 | load: 0.00296 [I 2022-10-11 07:57:41] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-11 07:57:41] ...optimizers/trijoint.py.151: total_norm: 12.832088 [I 2022-10-11 07:57:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1800/2304 [I 2022-10-11 07:57:49] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:05 [I 2022-10-11 07:57:49] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00301 [I 2022-10-11 07:57:49] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-11 07:57:49] ...optimizers/trijoint.py.151: total_norm: 10.635714 [I 2022-10-11 07:57:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1810/2304 [I 2022-10-11 07:57:58] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:06:54 [I 2022-10-11 07:57:58] ...trap/engines/engine.py.229: process: 0.83025 | load: 0.00298 [I 2022-10-11 07:57:58] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-11 07:57:58] ...optimizers/trijoint.py.151: total_norm: 13.316138 [I 2022-10-11 07:58:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1820/2304 [I 2022-10-11 07:58:06] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:46 [I 2022-10-11 07:58:06] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00298 [I 2022-10-11 07:58:06] ...trap/engines/engine.py.230: loss: 0.17069 [I 2022-10-11 07:58:06] ...optimizers/trijoint.py.151: total_norm: 17.285629 [I 2022-10-11 07:58:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1830/2304 [I 2022-10-11 07:58:15] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:51 [I 2022-10-11 07:58:15] ...trap/engines/engine.py.229: process: 0.87912 | load: 0.00376 [I 2022-10-11 07:58:15] ...trap/engines/engine.py.230: loss: 0.11138 [I 2022-10-11 07:58:15] ...optimizers/trijoint.py.151: total_norm: 14.792089 [I 2022-10-11 07:58:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1840/2304 [I 2022-10-11 07:58:23] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:31 [I 2022-10-11 07:58:23] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00297 [I 2022-10-11 07:58:23] ...trap/engines/engine.py.230: loss: 0.10658 [I 2022-10-11 07:58:23] ...optimizers/trijoint.py.151: total_norm: 11.913452 [I 2022-10-11 07:58:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1850/2304 [I 2022-10-11 07:58:32] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:06:22 [I 2022-10-11 07:58:32] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00296 [I 2022-10-11 07:58:32] ...trap/engines/engine.py.230: loss: 0.18360 [I 2022-10-11 07:58:32] ...optimizers/trijoint.py.151: total_norm: 25.021622 [I 2022-10-11 07:58:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 1860/2304 [I 2022-10-11 07:58:40] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:06:16 [I 2022-10-11 07:58:40] ...trap/engines/engine.py.229: process: 0.83967 | load: 0.00297 [I 2022-10-11 07:58:40] ...trap/engines/engine.py.230: loss: 0.09811 [I 2022-10-11 07:58:40] ...optimizers/trijoint.py.151: total_norm: 13.254811 [I 2022-10-11 07:58:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1870/2304 [I 2022-10-11 07:58:49] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:07 [I 2022-10-11 07:58:49] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00294 [I 2022-10-11 07:58:49] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-11 07:58:49] ...optimizers/trijoint.py.151: total_norm: 28.546957 [I 2022-10-11 07:58:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 1880/2304 [I 2022-10-11 07:58:57] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:06:00 [I 2022-10-11 07:58:57] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00297 [I 2022-10-11 07:58:57] ...trap/engines/engine.py.230: loss: 0.12295 [I 2022-10-11 07:58:57] ...optimizers/trijoint.py.151: total_norm: 18.580294 [I 2022-10-11 07:59:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1890/2304 [I 2022-10-11 07:59:06] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:53 [I 2022-10-11 07:59:06] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00292 [I 2022-10-11 07:59:06] ...trap/engines/engine.py.230: loss: 0.08762 [I 2022-10-11 07:59:06] ...optimizers/trijoint.py.151: total_norm: 14.181978 [I 2022-10-11 07:59:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 1900/2304 [I 2022-10-11 07:59:14] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:51 [I 2022-10-11 07:59:14] ...trap/engines/engine.py.229: process: 0.88687 | load: 0.00374 [I 2022-10-11 07:59:14] ...trap/engines/engine.py.230: loss: 0.10297 [I 2022-10-11 07:59:14] ...optimizers/trijoint.py.151: total_norm: 9.668003 [I 2022-10-11 07:59:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1910/2304 [I 2022-10-11 07:59:23] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:44 [I 2022-10-11 07:59:23] ...trap/engines/engine.py.229: process: 0.88072 | load: 0.00381 [I 2022-10-11 07:59:23] ...trap/engines/engine.py.230: loss: 0.19039 [I 2022-10-11 07:59:23] ...optimizers/trijoint.py.151: total_norm: 15.301243 [I 2022-10-11 07:59:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 1920/2304 [I 2022-10-11 07:59:32] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:35 [I 2022-10-11 07:59:32] ...trap/engines/engine.py.229: process: 0.86722 | load: 0.00378 [I 2022-10-11 07:59:32] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-11 07:59:32] ...optimizers/trijoint.py.151: total_norm: 32.153374 [I 2022-10-11 07:59:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 1930/2304 [I 2022-10-11 07:59:40] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:05:23 [I 2022-10-11 07:59:40] ...trap/engines/engine.py.229: process: 0.85100 | load: 0.00297 [I 2022-10-11 07:59:40] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 07:59:40] ...optimizers/trijoint.py.151: total_norm: 14.222795 [I 2022-10-11 07:59:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1940/2304 [I 2022-10-11 07:59:49] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:05:10 [I 2022-10-11 07:59:49] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00297 [I 2022-10-11 07:59:49] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-11 07:59:49] ...optimizers/trijoint.py.151: total_norm: 11.695612 [I 2022-10-11 07:59:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 1950/2304 [I 2022-10-11 07:59:57] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:56 [I 2022-10-11 07:59:57] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00296 [I 2022-10-11 07:59:57] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-11 07:59:57] ...optimizers/trijoint.py.151: total_norm: 22.178064 [I 2022-10-11 08:00:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1960/2304 [I 2022-10-11 08:00:06] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:50 [I 2022-10-11 08:00:06] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00298 [I 2022-10-11 08:00:06] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-11 08:00:06] ...optimizers/trijoint.py.151: total_norm: 25.189066 [I 2022-10-11 08:00:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 1970/2304 [I 2022-10-11 08:00:14] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:39 [I 2022-10-11 08:00:14] ...trap/engines/engine.py.229: process: 0.83944 | load: 0.00298 [I 2022-10-11 08:00:14] ...trap/engines/engine.py.230: loss: 0.15719 [I 2022-10-11 08:00:14] ...optimizers/trijoint.py.151: total_norm: 10.104259 [I 2022-10-11 08:00:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1980/2304 [I 2022-10-11 08:00:23] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:31 [I 2022-10-11 08:00:23] ...trap/engines/engine.py.229: process: 0.83380 | load: 0.00297 [I 2022-10-11 08:00:23] ...trap/engines/engine.py.230: loss: 0.10566 [I 2022-10-11 08:00:23] ...optimizers/trijoint.py.151: total_norm: 17.399872 [I 2022-10-11 08:00:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 1990/2304 [I 2022-10-11 08:00:31] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:24 [I 2022-10-11 08:00:31] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00297 [I 2022-10-11 08:00:31] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-11 08:00:31] ...optimizers/trijoint.py.151: total_norm: 16.279312 [I 2022-10-11 08:00:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 2000/2304 [I 2022-10-11 08:00:39] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:15 [I 2022-10-11 08:00:39] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00297 [I 2022-10-11 08:00:39] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-11 08:00:39] ...optimizers/trijoint.py.151: total_norm: 20.311129 [I 2022-10-11 08:00:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 2010/2304 [I 2022-10-11 08:00:48] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:11 [I 2022-10-11 08:00:48] ...trap/engines/engine.py.229: process: 0.85075 | load: 0.00295 [I 2022-10-11 08:00:48] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-11 08:00:48] ...optimizers/trijoint.py.151: total_norm: 17.667751 [I 2022-10-11 08:00:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 2020/2304 [I 2022-10-11 08:00:56] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:04:01 [I 2022-10-11 08:00:56] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00298 [I 2022-10-11 08:00:56] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-11 08:00:56] ...optimizers/trijoint.py.151: total_norm: 20.567734 [I 2022-10-11 08:01:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 2030/2304 [I 2022-10-11 08:01:05] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:51 [I 2022-10-11 08:01:05] ...trap/engines/engine.py.229: process: 0.83041 | load: 0.00298 [I 2022-10-11 08:01:05] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-11 08:01:05] ...optimizers/trijoint.py.151: total_norm: 22.919979 [I 2022-10-11 08:01:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 2040/2304 [I 2022-10-11 08:01:13] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:42 [I 2022-10-11 08:01:13] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00295 [I 2022-10-11 08:01:13] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-11 08:01:13] ...optimizers/trijoint.py.151: total_norm: 44.066071 [I 2022-10-11 08:01:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 2050/2304 [I 2022-10-11 08:01:22] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:35 [I 2022-10-11 08:01:22] ...trap/engines/engine.py.229: process: 0.85030 | load: 0.00294 [I 2022-10-11 08:01:22] ...trap/engines/engine.py.230: loss: 0.12406 [I 2022-10-11 08:01:22] ...optimizers/trijoint.py.151: total_norm: 21.520962 [I 2022-10-11 08:01:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 2060/2304 [I 2022-10-11 08:01:30] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:03:27 [I 2022-10-11 08:01:30] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00304 [I 2022-10-11 08:01:30] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-11 08:01:30] ...optimizers/trijoint.py.151: total_norm: 11.176386 [I 2022-10-11 08:01:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 2070/2304 [I 2022-10-11 08:01:39] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:18 [I 2022-10-11 08:01:39] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00297 [I 2022-10-11 08:01:39] ...trap/engines/engine.py.230: loss: 0.12878 [I 2022-10-11 08:01:39] ...optimizers/trijoint.py.151: total_norm: 29.805614 [I 2022-10-11 08:01:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 2080/2304 [I 2022-10-11 08:01:47] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:08 [I 2022-10-11 08:01:47] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00307 [I 2022-10-11 08:01:47] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-11 08:01:47] ...optimizers/trijoint.py.151: total_norm: 22.793583 [I 2022-10-11 08:01:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 2090/2304 [I 2022-10-11 08:01:56] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:01 [I 2022-10-11 08:01:56] ...trap/engines/engine.py.229: process: 0.83113 | load: 0.00298 [I 2022-10-11 08:01:56] ...trap/engines/engine.py.230: loss: 0.09830 [I 2022-10-11 08:01:56] ...optimizers/trijoint.py.151: total_norm: 13.184810 [I 2022-10-11 08:02:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 2100/2304 [I 2022-10-11 08:02:04] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:52 [I 2022-10-11 08:02:04] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00298 [I 2022-10-11 08:02:04] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-11 08:02:04] ...optimizers/trijoint.py.151: total_norm: 14.570951 [I 2022-10-11 08:02:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 2110/2304 [I 2022-10-11 08:02:13] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:44 [I 2022-10-11 08:02:13] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00296 [I 2022-10-11 08:02:13] ...trap/engines/engine.py.230: loss: 0.10289 [I 2022-10-11 08:02:13] ...optimizers/trijoint.py.151: total_norm: 13.900625 [I 2022-10-11 08:02:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 2120/2304 [I 2022-10-11 08:02:21] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:35 [I 2022-10-11 08:02:21] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00297 [I 2022-10-11 08:02:21] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-11 08:02:21] ...optimizers/trijoint.py.151: total_norm: 11.318669 [I 2022-10-11 08:02:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 2130/2304 [I 2022-10-11 08:02:30] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:27 [I 2022-10-11 08:02:30] ...trap/engines/engine.py.229: process: 0.84076 | load: 0.00297 [I 2022-10-11 08:02:30] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 08:02:30] ...optimizers/trijoint.py.151: total_norm: 13.384078 [I 2022-10-11 08:02:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 2140/2304 [I 2022-10-11 08:02:38] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:18 [I 2022-10-11 08:02:38] ...trap/engines/engine.py.229: process: 0.82752 | load: 0.00294 [I 2022-10-11 08:02:38] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-11 08:02:38] ...optimizers/trijoint.py.151: total_norm: 18.813131 [I 2022-10-11 08:02:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 2150/2304 [I 2022-10-11 08:02:47] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:02:09 [I 2022-10-11 08:02:47] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00296 [I 2022-10-11 08:02:47] ...trap/engines/engine.py.230: loss: 0.14873 [I 2022-10-11 08:02:47] ...optimizers/trijoint.py.151: total_norm: 32.413109 [I 2022-10-11 08:02:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 2160/2304 [I 2022-10-11 08:02:55] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:01 [I 2022-10-11 08:02:55] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00296 [I 2022-10-11 08:02:55] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-11 08:02:55] ...optimizers/trijoint.py.151: total_norm: 29.397190 [I 2022-10-11 08:03:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 2170/2304 [I 2022-10-11 08:03:03] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:52 [I 2022-10-11 08:03:03] ...trap/engines/engine.py.229: process: 0.84288 | load: 0.00296 [I 2022-10-11 08:03:03] ...trap/engines/engine.py.230: loss: 0.10386 [I 2022-10-11 08:03:03] ...optimizers/trijoint.py.151: total_norm: 10.284778 [I 2022-10-11 08:03:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 2180/2304 [I 2022-10-11 08:03:12] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:44 [I 2022-10-11 08:03:12] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00304 [I 2022-10-11 08:03:12] ...trap/engines/engine.py.230: loss: 0.14370 [I 2022-10-11 08:03:12] ...optimizers/trijoint.py.151: total_norm: 9.833242 [I 2022-10-11 08:03:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 2190/2304 [I 2022-10-11 08:03:20] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:37 [I 2022-10-11 08:03:20] ...trap/engines/engine.py.229: process: 0.87155 | load: 0.00385 [I 2022-10-11 08:03:20] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-11 08:03:20] ...optimizers/trijoint.py.151: total_norm: 16.104643 [I 2022-10-11 08:03:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 2200/2304 [I 2022-10-11 08:03:29] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:30 [I 2022-10-11 08:03:29] ...trap/engines/engine.py.229: process: 0.87204 | load: 0.00377 [I 2022-10-11 08:03:29] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-11 08:03:29] ...optimizers/trijoint.py.151: total_norm: 29.204308 [I 2022-10-11 08:03:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 2210/2304 [I 2022-10-11 08:03:38] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:21 [I 2022-10-11 08:03:38] ...trap/engines/engine.py.229: process: 0.86832 | load: 0.00370 [I 2022-10-11 08:03:38] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-11 08:03:38] ...optimizers/trijoint.py.151: total_norm: 18.077610 [I 2022-10-11 08:03:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 2220/2304 [I 2022-10-11 08:03:46] ...trap/engines/engine.py.228: elapsed: 0:31:56 | left: 0:01:12 [I 2022-10-11 08:03:46] ...trap/engines/engine.py.229: process: 0.85489 | load: 0.00376 [I 2022-10-11 08:03:46] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-11 08:03:46] ...optimizers/trijoint.py.151: total_norm: 16.751783 [I 2022-10-11 08:03:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 2230/2304 [I 2022-10-11 08:03:55] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:04 [I 2022-10-11 08:03:55] ...trap/engines/engine.py.229: process: 0.88027 | load: 0.00384 [I 2022-10-11 08:03:55] ...trap/engines/engine.py.230: loss: 0.15623 [I 2022-10-11 08:03:55] ...optimizers/trijoint.py.151: total_norm: 23.560345 [I 2022-10-11 08:04:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 2240/2304 [I 2022-10-11 08:04:04] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:54 [I 2022-10-11 08:04:04] ...trap/engines/engine.py.229: process: 0.85725 | load: 0.00307 [I 2022-10-11 08:04:04] ...trap/engines/engine.py.230: loss: 0.13610 [I 2022-10-11 08:04:04] ...optimizers/trijoint.py.151: total_norm: 13.183584 [I 2022-10-11 08:04:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 2250/2304 [I 2022-10-11 08:04:13] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:47 [I 2022-10-11 08:04:13] ...trap/engines/engine.py.229: process: 0.87647 | load: 0.00387 [I 2022-10-11 08:04:13] ...trap/engines/engine.py.230: loss: 0.09575 [I 2022-10-11 08:04:13] ...optimizers/trijoint.py.151: total_norm: 9.995465 [I 2022-10-11 08:04:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 2260/2304 [I 2022-10-11 08:04:21] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:37 [I 2022-10-11 08:04:21] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00297 [I 2022-10-11 08:04:21] ...trap/engines/engine.py.230: loss: 0.12867 [I 2022-10-11 08:04:21] ...optimizers/trijoint.py.151: total_norm: 22.117220 [I 2022-10-11 08:04:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 2270/2304 [I 2022-10-11 08:04:30] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:28 [I 2022-10-11 08:04:30] ...trap/engines/engine.py.229: process: 0.85914 | load: 0.00297 [I 2022-10-11 08:04:30] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-11 08:04:30] ...optimizers/trijoint.py.151: total_norm: 19.692507 [I 2022-10-11 08:04:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 2280/2304 [I 2022-10-11 08:04:38] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:20 [I 2022-10-11 08:04:38] ...trap/engines/engine.py.229: process: 0.83232 | load: 0.00298 [I 2022-10-11 08:04:38] ...trap/engines/engine.py.230: loss: 0.08582 [I 2022-10-11 08:04:38] ...optimizers/trijoint.py.151: total_norm: 8.481562 [I 2022-10-11 08:04:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 2290/2304 [I 2022-10-11 08:04:46] ...trap/engines/engine.py.228: elapsed: 0:32:56 | left: 0:00:11 [I 2022-10-11 08:04:46] ...trap/engines/engine.py.229: process: 0.81895 | load: 0.00296 [I 2022-10-11 08:04:46] ...trap/engines/engine.py.230: loss: 0.09033 [I 2022-10-11 08:04:46] ...optimizers/trijoint.py.151: total_norm: 12.198960 [I 2022-10-11 08:04:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 2300/2304 [I 2022-10-11 08:04:55] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:03 [I 2022-10-11 08:04:55] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00013 [I 2022-10-11 08:04:55] ...trap/engines/engine.py.230: loss: 0.17957 [I 2022-10-11 08:04:55] ...optimizers/trijoint.py.151: total_norm: 11.033021 [I 2022-10-11 08:04:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 2304/2304 [I 2022-10-11 08:04:58] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:00 [I 2022-10-11 08:04:58] ...trap/engines/engine.py.229: process: 0.82728 | load: 0.00011 [I 2022-10-11 08:04:58] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-11 08:04:58] ...optimizers/trijoint.py.151: total_norm: 27.515406 [S 2022-10-11 08:04:59] ...trap/engines/engine.py.129: train_epoch.epoch: 97 [S 2022-10-11 08:04:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.12526666581178178 [I 2022-10-11 08:05:00] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 08:05:01] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 97 [I 2022-10-11 08:05:01] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [I 2022-10-11 08:05:06] ...trap/engines/engine.py.321: eval: epoch 97 | batch 0/510 [I 2022-10-11 08:05:06] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:39:22 [I 2022-10-11 08:05:06] ...trap/engines/engine.py.326: process: 4.63195 | load: 3.48572 [W 2022-10-11 08:05:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 08:05:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 08:05:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 08:05:13] ...trap/engines/engine.py.321: eval: epoch 97 | batch 10/510 [I 2022-10-11 08:05:13] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:08:17 [I 2022-10-11 08:05:13] ...trap/engines/engine.py.326: process: 0.44397 | load: 0.00019 [I 2022-10-11 08:05:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 08:05:19] ...trap/engines/engine.py.321: eval: epoch 97 | batch 20/510 [I 2022-10-11 08:05:19] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:04:31 [I 2022-10-11 08:05:19] ...trap/engines/engine.py.326: process: 0.60289 | load: 0.00018 [I 2022-10-11 08:05:26] ...trap/engines/engine.py.321: eval: epoch 97 | batch 30/510 [I 2022-10-11 08:05:26] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:04:55 [I 2022-10-11 08:05:26] ...trap/engines/engine.py.326: process: 0.41372 | load: 0.00018 [I 2022-10-11 08:05:31] ...trap/engines/engine.py.321: eval: epoch 97 | batch 40/510 [I 2022-10-11 08:05:31] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:04:50 [I 2022-10-11 08:05:31] ...trap/engines/engine.py.326: process: 0.66958 | load: 0.00019 [I 2022-10-11 08:05:36] ...trap/engines/engine.py.321: eval: epoch 97 | batch 50/510 [I 2022-10-11 08:05:36] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:42 [I 2022-10-11 08:05:36] ...trap/engines/engine.py.326: process: 0.52336 | load: 0.00017 [I 2022-10-11 08:05:41] ...trap/engines/engine.py.321: eval: epoch 97 | batch 60/510 [I 2022-10-11 08:05:41] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:03:41 [I 2022-10-11 08:05:41] ...trap/engines/engine.py.326: process: 0.41932 | load: 0.00019 [I 2022-10-11 08:05:46] ...trap/engines/engine.py.321: eval: epoch 97 | batch 70/510 [I 2022-10-11 08:05:46] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:36 [I 2022-10-11 08:05:46] ...trap/engines/engine.py.326: process: 0.48950 | load: 0.00018 [I 2022-10-11 08:05:51] ...trap/engines/engine.py.321: eval: epoch 97 | batch 80/510 [I 2022-10-11 08:05:51] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:40 [I 2022-10-11 08:05:51] ...trap/engines/engine.py.326: process: 0.45368 | load: 0.00019 [I 2022-10-11 08:05:56] ...trap/engines/engine.py.321: eval: epoch 97 | batch 90/510 [I 2022-10-11 08:05:56] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:25 [I 2022-10-11 08:05:56] ...trap/engines/engine.py.326: process: 0.45430 | load: 0.00016 [I 2022-10-11 08:06:01] ...trap/engines/engine.py.321: eval: epoch 97 | batch 100/510 [I 2022-10-11 08:06:01] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:58 [I 2022-10-11 08:06:01] ...trap/engines/engine.py.326: process: 0.30602 | load: 0.00018 [I 2022-10-11 08:06:04] ...trap/engines/engine.py.321: eval: epoch 97 | batch 110/510 [I 2022-10-11 08:06:04] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:12 [I 2022-10-11 08:06:04] ...trap/engines/engine.py.326: process: 0.32118 | load: 0.00019 [I 2022-10-11 08:06:07] ...trap/engines/engine.py.321: eval: epoch 97 | batch 120/510 [I 2022-10-11 08:06:07] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:04 [I 2022-10-11 08:06:07] ...trap/engines/engine.py.326: process: 0.31568 | load: 0.00019 [I 2022-10-11 08:06:10] ...trap/engines/engine.py.321: eval: epoch 97 | batch 130/510 [I 2022-10-11 08:06:10] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:00 [I 2022-10-11 08:06:10] ...trap/engines/engine.py.326: process: 0.31565 | load: 0.00020 [I 2022-10-11 08:06:13] ...trap/engines/engine.py.321: eval: epoch 97 | batch 140/510 [I 2022-10-11 08:06:13] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:57 [I 2022-10-11 08:06:13] ...trap/engines/engine.py.326: process: 0.31875 | load: 0.00019 [I 2022-10-11 08:06:17] ...trap/engines/engine.py.321: eval: epoch 97 | batch 150/510 [I 2022-10-11 08:06:17] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:55 [I 2022-10-11 08:06:17] ...trap/engines/engine.py.326: process: 0.32307 | load: 0.00018 [I 2022-10-11 08:06:20] ...trap/engines/engine.py.321: eval: epoch 97 | batch 160/510 [I 2022-10-11 08:06:20] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:51 [I 2022-10-11 08:06:20] ...trap/engines/engine.py.326: process: 0.31570 | load: 0.00019 [I 2022-10-11 08:06:23] ...trap/engines/engine.py.321: eval: epoch 97 | batch 170/510 [I 2022-10-11 08:06:23] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:48 [I 2022-10-11 08:06:23] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00019 [I 2022-10-11 08:06:26] ...trap/engines/engine.py.321: eval: epoch 97 | batch 180/510 [I 2022-10-11 08:06:26] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:45 [I 2022-10-11 08:06:26] ...trap/engines/engine.py.326: process: 0.31428 | load: 0.00018 [I 2022-10-11 08:06:29] ...trap/engines/engine.py.321: eval: epoch 97 | batch 190/510 [I 2022-10-11 08:06:29] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:41 [I 2022-10-11 08:06:29] ...trap/engines/engine.py.326: process: 0.31825 | load: 0.00019 [I 2022-10-11 08:06:33] ...trap/engines/engine.py.321: eval: epoch 97 | batch 200/510 [I 2022-10-11 08:06:33] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:38 [I 2022-10-11 08:06:33] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00018 [I 2022-10-11 08:06:36] ...trap/engines/engine.py.321: eval: epoch 97 | batch 210/510 [I 2022-10-11 08:06:36] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:34 [I 2022-10-11 08:06:36] ...trap/engines/engine.py.326: process: 0.30707 | load: 0.00019 [I 2022-10-11 08:06:39] ...trap/engines/engine.py.321: eval: epoch 97 | batch 220/510 [I 2022-10-11 08:06:39] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:50 [I 2022-10-11 08:06:39] ...trap/engines/engine.py.326: process: 0.60459 | load: 0.28503 [I 2022-10-11 08:06:42] ...trap/engines/engine.py.321: eval: epoch 97 | batch 230/510 [I 2022-10-11 08:06:42] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:31 [I 2022-10-11 08:06:42] ...trap/engines/engine.py.326: process: 0.32183 | load: 0.00020 [I 2022-10-11 08:06:46] ...trap/engines/engine.py.321: eval: epoch 97 | batch 240/510 [I 2022-10-11 08:06:46] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:28 [I 2022-10-11 08:06:46] ...trap/engines/engine.py.326: process: 0.32747 | load: 0.00019 [I 2022-10-11 08:06:49] ...trap/engines/engine.py.321: eval: epoch 97 | batch 250/510 [I 2022-10-11 08:06:49] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:22 [I 2022-10-11 08:06:49] ...trap/engines/engine.py.326: process: 0.31651 | load: 0.00018 [I 2022-10-11 08:06:52] ...trap/engines/engine.py.321: eval: epoch 97 | batch 260/510 [I 2022-10-11 08:06:52] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:28 [I 2022-10-11 08:06:52] ...trap/engines/engine.py.326: process: 0.31195 | load: 0.00016 [I 2022-10-11 08:06:56] ...trap/engines/engine.py.321: eval: epoch 97 | batch 270/510 [I 2022-10-11 08:06:56] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:17 [I 2022-10-11 08:06:56] ...trap/engines/engine.py.326: process: 0.32293 | load: 0.00021 [I 2022-10-11 08:06:59] ...trap/engines/engine.py.321: eval: epoch 97 | batch 280/510 [I 2022-10-11 08:06:59] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:13 [I 2022-10-11 08:06:59] ...trap/engines/engine.py.326: process: 0.31768 | load: 0.00017 [I 2022-10-11 08:07:02] ...trap/engines/engine.py.321: eval: epoch 97 | batch 290/510 [I 2022-10-11 08:07:02] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:10 [I 2022-10-11 08:07:02] ...trap/engines/engine.py.326: process: 0.31393 | load: 0.00026 [I 2022-10-11 08:07:05] ...trap/engines/engine.py.321: eval: epoch 97 | batch 300/510 [I 2022-10-11 08:07:05] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:06 [I 2022-10-11 08:07:05] ...trap/engines/engine.py.326: process: 0.31284 | load: 0.00020 [I 2022-10-11 08:07:08] ...trap/engines/engine.py.321: eval: epoch 97 | batch 310/510 [I 2022-10-11 08:07:08] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:03 [I 2022-10-11 08:07:08] ...trap/engines/engine.py.326: process: 0.31340 | load: 0.00018 [I 2022-10-11 08:07:13] ...trap/engines/engine.py.321: eval: epoch 97 | batch 320/510 [I 2022-10-11 08:07:13] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:46 [I 2022-10-11 08:07:13] ...trap/engines/engine.py.326: process: 0.89096 | load: 0.57246 [I 2022-10-11 08:07:16] ...trap/engines/engine.py.321: eval: epoch 97 | batch 330/510 [I 2022-10-11 08:07:16] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:02 [I 2022-10-11 08:07:16] ...trap/engines/engine.py.326: process: 0.31625 | load: 0.00021 [I 2022-10-11 08:07:19] ...trap/engines/engine.py.321: eval: epoch 97 | batch 340/510 [I 2022-10-11 08:07:19] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:55 [I 2022-10-11 08:07:19] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00019 [I 2022-10-11 08:07:23] ...trap/engines/engine.py.321: eval: epoch 97 | batch 350/510 [I 2022-10-11 08:07:23] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:50 [I 2022-10-11 08:07:23] ...trap/engines/engine.py.326: process: 0.30723 | load: 0.00019 [I 2022-10-11 08:07:26] ...trap/engines/engine.py.321: eval: epoch 97 | batch 360/510 [I 2022-10-11 08:07:26] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:47 [I 2022-10-11 08:07:26] ...trap/engines/engine.py.326: process: 0.31454 | load: 0.00019 [I 2022-10-11 08:07:29] ...trap/engines/engine.py.321: eval: epoch 97 | batch 370/510 [I 2022-10-11 08:07:29] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:44 [I 2022-10-11 08:07:29] ...trap/engines/engine.py.326: process: 0.32335 | load: 0.00019 [I 2022-10-11 08:07:32] ...trap/engines/engine.py.321: eval: epoch 97 | batch 380/510 [I 2022-10-11 08:07:32] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:41 [I 2022-10-11 08:07:32] ...trap/engines/engine.py.326: process: 0.32407 | load: 0.00018 [I 2022-10-11 08:07:35] ...trap/engines/engine.py.321: eval: epoch 97 | batch 390/510 [I 2022-10-11 08:07:35] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:37 [I 2022-10-11 08:07:35] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00019 [I 2022-10-11 08:07:40] ...trap/engines/engine.py.321: eval: epoch 97 | batch 400/510 [I 2022-10-11 08:07:40] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:52 [I 2022-10-11 08:07:40] ...trap/engines/engine.py.326: process: 0.31432 | load: 0.00007 [I 2022-10-11 08:07:44] ...trap/engines/engine.py.321: eval: epoch 97 | batch 410/510 [I 2022-10-11 08:07:44] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:40 [I 2022-10-11 08:07:44] ...trap/engines/engine.py.326: process: 0.30764 | load: 0.00020 [I 2022-10-11 08:07:50] ...trap/engines/engine.py.321: eval: epoch 97 | batch 420/510 [I 2022-10-11 08:07:50] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:56 [I 2022-10-11 08:07:50] ...trap/engines/engine.py.326: process: 0.31251 | load: 0.00016 [I 2022-10-11 08:07:54] ...trap/engines/engine.py.321: eval: epoch 97 | batch 430/510 [I 2022-10-11 08:07:54] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:31 [I 2022-10-11 08:07:54] ...trap/engines/engine.py.326: process: 0.30993 | load: 0.00016 [I 2022-10-11 08:08:00] ...trap/engines/engine.py.321: eval: epoch 97 | batch 440/510 [I 2022-10-11 08:08:00] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:33 [I 2022-10-11 08:08:00] ...trap/engines/engine.py.326: process: 0.32510 | load: 0.00017 [I 2022-10-11 08:08:03] ...trap/engines/engine.py.321: eval: epoch 97 | batch 450/510 [I 2022-10-11 08:08:03] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:20 [I 2022-10-11 08:08:03] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00017 [I 2022-10-11 08:08:07] ...trap/engines/engine.py.321: eval: epoch 97 | batch 460/510 [I 2022-10-11 08:08:07] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:15 [I 2022-10-11 08:08:07] ...trap/engines/engine.py.326: process: 0.31204 | load: 0.00018 [I 2022-10-11 08:08:12] ...trap/engines/engine.py.321: eval: epoch 97 | batch 470/510 [I 2022-10-11 08:08:12] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:18 [I 2022-10-11 08:08:12] ...trap/engines/engine.py.326: process: 0.45177 | load: 0.12812 [I 2022-10-11 08:08:15] ...trap/engines/engine.py.321: eval: epoch 97 | batch 480/510 [I 2022-10-11 08:08:15] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:10 [I 2022-10-11 08:08:15] ...trap/engines/engine.py.326: process: 0.33348 | load: 0.00031 [I 2022-10-11 08:08:19] ...trap/engines/engine.py.321: eval: epoch 97 | batch 490/510 [I 2022-10-11 08:08:19] ...trap/engines/engine.py.325: elapsed: 0:03:17 | left: 0:00:06 [I 2022-10-11 08:08:19] ...trap/engines/engine.py.326: process: 0.32606 | load: 0.00017 [I 2022-10-11 08:08:22] ...trap/engines/engine.py.321: eval: epoch 97 | batch 500/510 [I 2022-10-11 08:08:22] ...trap/engines/engine.py.325: elapsed: 0:03:20 | left: 0:00:03 [I 2022-10-11 08:08:22] ...trap/engines/engine.py.326: process: 0.32127 | load: 0.00017 [I 2022-10-11 08:08:25] ...trap/engines/engine.py.321: eval: epoch 97 | batch 510/510 [I 2022-10-11 08:08:25] ...trap/engines/engine.py.325: elapsed: 0:03:24 | left: 0:00:00 [I 2022-10-11 08:08:25] ...trap/engines/engine.py.326: process: 0.32634 | load: 0.00001 [S 2022-10-11 08:08:26] ...trap/engines/engine.py.132: eval_epoch.epoch: 97 [I 2022-10-11 08:08:26] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.745600 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.912300 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.941000 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.742600 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.913400 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.939800 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014108 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008956 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005441 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016107 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007889 [S 2022-10-11 08:08:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006384 [I 2022-10-11 08:09:08] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 08:09:08] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 08:09:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 08:09:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 08:09:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 08:09:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 08:09:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 08:09:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 08:09:19] ...trap/engines/engine.py.163: Training model on trainset for epoch 98 [I 2022-10-11 08:09:19] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 08:09:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 0/2304 [I 2022-10-11 08:09:22] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:16:08 [I 2022-10-11 08:09:22] ...trap/engines/engine.py.229: process: 3.54536 | load: 2.36054 [I 2022-10-11 08:09:22] ...trap/engines/engine.py.230: loss: 0.09334 [I 2022-10-11 08:09:22] ...optimizers/trijoint.py.151: total_norm: 12.406425 [I 2022-10-11 08:09:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 10/2304 [I 2022-10-11 08:09:31] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:43:40 [I 2022-10-11 08:09:31] ...trap/engines/engine.py.229: process: 0.82707 | load: 0.00512 [I 2022-10-11 08:09:31] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-11 08:09:31] ...optimizers/trijoint.py.151: total_norm: 8.722400 [I 2022-10-11 08:09:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 20/2304 [I 2022-10-11 08:09:39] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:04 [I 2022-10-11 08:09:39] ...trap/engines/engine.py.229: process: 0.83041 | load: 0.00350 [I 2022-10-11 08:09:39] ...trap/engines/engine.py.230: loss: 0.15462 [I 2022-10-11 08:09:39] ...optimizers/trijoint.py.151: total_norm: 26.762840 [I 2022-10-11 08:09:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 30/2304 [I 2022-10-11 08:09:48] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:43 [I 2022-10-11 08:09:48] ...trap/engines/engine.py.229: process: 0.83295 | load: 0.00400 [I 2022-10-11 08:09:48] ...trap/engines/engine.py.230: loss: 0.12786 [I 2022-10-11 08:09:48] ...optimizers/trijoint.py.151: total_norm: 9.376012 [I 2022-10-11 08:09:56] ...trap/engines/engine.py.224: train: epoch 98 | batch 40/2304 [I 2022-10-11 08:09:56] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:26 [I 2022-10-11 08:09:56] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00443 [I 2022-10-11 08:09:56] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-11 08:09:56] ...optimizers/trijoint.py.151: total_norm: 15.146416 [I 2022-10-11 08:10:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 50/2304 [I 2022-10-11 08:10:04] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:17 [I 2022-10-11 08:10:04] ...trap/engines/engine.py.229: process: 0.82669 | load: 0.00358 [I 2022-10-11 08:10:04] ...trap/engines/engine.py.230: loss: 0.08198 [I 2022-10-11 08:10:04] ...optimizers/trijoint.py.151: total_norm: 10.227467 [I 2022-10-11 08:10:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 60/2304 [I 2022-10-11 08:10:13] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:05 [I 2022-10-11 08:10:13] ...trap/engines/engine.py.229: process: 0.83113 | load: 0.00328 [I 2022-10-11 08:10:13] ...trap/engines/engine.py.230: loss: 0.10976 [I 2022-10-11 08:10:13] ...optimizers/trijoint.py.151: total_norm: 19.603998 [I 2022-10-11 08:10:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 70/2304 [I 2022-10-11 08:10:21] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:00 [I 2022-10-11 08:10:21] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00364 [I 2022-10-11 08:10:21] ...trap/engines/engine.py.230: loss: 0.10729 [I 2022-10-11 08:10:21] ...optimizers/trijoint.py.151: total_norm: 9.654262 [I 2022-10-11 08:10:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 80/2304 [I 2022-10-11 08:10:29] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:30:47 [I 2022-10-11 08:10:29] ...trap/engines/engine.py.229: process: 0.81899 | load: 0.00365 [I 2022-10-11 08:10:29] ...trap/engines/engine.py.230: loss: 0.12076 [I 2022-10-11 08:10:29] ...optimizers/trijoint.py.151: total_norm: 25.937508 [I 2022-10-11 08:10:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 90/2304 [I 2022-10-11 08:10:38] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:00 [I 2022-10-11 08:10:38] ...trap/engines/engine.py.229: process: 0.85633 | load: 0.00320 [I 2022-10-11 08:10:38] ...trap/engines/engine.py.230: loss: 0.14929 [I 2022-10-11 08:10:38] ...optimizers/trijoint.py.151: total_norm: 23.639963 [I 2022-10-11 08:10:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 100/2304 [I 2022-10-11 08:10:46] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:47 [I 2022-10-11 08:10:46] ...trap/engines/engine.py.229: process: 0.83382 | load: 0.00308 [I 2022-10-11 08:10:46] ...trap/engines/engine.py.230: loss: 0.13329 [I 2022-10-11 08:10:46] ...optimizers/trijoint.py.151: total_norm: 41.743683 [I 2022-10-11 08:10:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 110/2304 [I 2022-10-11 08:10:55] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:42 [I 2022-10-11 08:10:55] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00347 [I 2022-10-11 08:10:55] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-11 08:10:55] ...optimizers/trijoint.py.151: total_norm: 8.626858 [I 2022-10-11 08:11:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 120/2304 [I 2022-10-11 08:11:03] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:30 [I 2022-10-11 08:11:03] ...trap/engines/engine.py.229: process: 0.83207 | load: 0.00306 [I 2022-10-11 08:11:03] ...trap/engines/engine.py.230: loss: 0.17484 [I 2022-10-11 08:11:03] ...optimizers/trijoint.py.151: total_norm: 27.056517 [I 2022-10-11 08:11:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 130/2304 [I 2022-10-11 08:11:11] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:34 [I 2022-10-11 08:11:11] ...trap/engines/engine.py.229: process: 0.84116 | load: 0.00316 [I 2022-10-11 08:11:11] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-11 08:11:11] ...optimizers/trijoint.py.151: total_norm: 9.816988 [I 2022-10-11 08:11:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 140/2304 [I 2022-10-11 08:11:20] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:29 [I 2022-10-11 08:11:20] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00331 [I 2022-10-11 08:11:20] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-11 08:11:20] ...optimizers/trijoint.py.151: total_norm: 16.088692 [I 2022-10-11 08:11:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 150/2304 [I 2022-10-11 08:11:28] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:11 [I 2022-10-11 08:11:28] ...trap/engines/engine.py.229: process: 0.83449 | load: 0.00295 [I 2022-10-11 08:11:28] ...trap/engines/engine.py.230: loss: 0.09471 [I 2022-10-11 08:11:28] ...optimizers/trijoint.py.151: total_norm: 16.098263 [I 2022-10-11 08:11:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 160/2304 [I 2022-10-11 08:11:37] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:23 [I 2022-10-11 08:11:37] ...trap/engines/engine.py.229: process: 0.85988 | load: 0.00302 [I 2022-10-11 08:11:37] ...trap/engines/engine.py.230: loss: 0.14950 [I 2022-10-11 08:11:37] ...optimizers/trijoint.py.151: total_norm: 9.192432 [I 2022-10-11 08:11:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 170/2304 [I 2022-10-11 08:11:45] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:49 [I 2022-10-11 08:11:45] ...trap/engines/engine.py.229: process: 0.82709 | load: 0.00317 [I 2022-10-11 08:11:45] ...trap/engines/engine.py.230: loss: 0.12232 [I 2022-10-11 08:11:45] ...optimizers/trijoint.py.151: total_norm: 17.358759 [I 2022-10-11 08:11:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 180/2304 [I 2022-10-11 08:11:54] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:39 [I 2022-10-11 08:11:54] ...trap/engines/engine.py.229: process: 0.84000 | load: 0.00305 [I 2022-10-11 08:11:54] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-11 08:11:54] ...optimizers/trijoint.py.151: total_norm: 11.129869 [I 2022-10-11 08:12:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 190/2304 [I 2022-10-11 08:12:02] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:24 [I 2022-10-11 08:12:02] ...trap/engines/engine.py.229: process: 0.81939 | load: 0.00294 [I 2022-10-11 08:12:02] ...trap/engines/engine.py.230: loss: 0.11031 [I 2022-10-11 08:12:02] ...optimizers/trijoint.py.151: total_norm: 16.230040 [I 2022-10-11 08:12:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 200/2304 [I 2022-10-11 08:12:10] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:08 [I 2022-10-11 08:12:10] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00330 [I 2022-10-11 08:12:10] ...trap/engines/engine.py.230: loss: 0.09871 [I 2022-10-11 08:12:10] ...optimizers/trijoint.py.151: total_norm: 12.563608 [I 2022-10-11 08:12:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 210/2304 [I 2022-10-11 08:12:19] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:10 [I 2022-10-11 08:12:19] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00294 [I 2022-10-11 08:12:19] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-11 08:12:19] ...optimizers/trijoint.py.151: total_norm: 12.825265 [I 2022-10-11 08:12:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 220/2304 [I 2022-10-11 08:12:27] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:05 [I 2022-10-11 08:12:27] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00304 [I 2022-10-11 08:12:27] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-11 08:12:27] ...optimizers/trijoint.py.151: total_norm: 11.712869 [I 2022-10-11 08:12:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 230/2304 [I 2022-10-11 08:12:36] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:19 [I 2022-10-11 08:12:36] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00295 [I 2022-10-11 08:12:36] ...trap/engines/engine.py.230: loss: 0.08635 [I 2022-10-11 08:12:36] ...optimizers/trijoint.py.151: total_norm: 13.877415 [I 2022-10-11 08:12:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 240/2304 [I 2022-10-11 08:12:44] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:05 [I 2022-10-11 08:12:44] ...trap/engines/engine.py.229: process: 0.84708 | load: 0.00314 [I 2022-10-11 08:12:44] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-11 08:12:44] ...optimizers/trijoint.py.151: total_norm: 26.394741 [I 2022-10-11 08:12:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 250/2304 [I 2022-10-11 08:12:52] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:38 [I 2022-10-11 08:12:52] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00302 [I 2022-10-11 08:12:52] ...trap/engines/engine.py.230: loss: 0.16529 [I 2022-10-11 08:12:52] ...optimizers/trijoint.py.151: total_norm: 13.453722 [I 2022-10-11 08:13:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 260/2304 [I 2022-10-11 08:13:01] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:29 [I 2022-10-11 08:13:01] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00294 [I 2022-10-11 08:13:01] ...trap/engines/engine.py.230: loss: 0.20895 [I 2022-10-11 08:13:01] ...optimizers/trijoint.py.151: total_norm: 17.592279 [I 2022-10-11 08:13:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 270/2304 [I 2022-10-11 08:13:09] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:03 [I 2022-10-11 08:13:09] ...trap/engines/engine.py.229: process: 0.82828 | load: 0.00321 [I 2022-10-11 08:13:09] ...trap/engines/engine.py.230: loss: 0.11936 [I 2022-10-11 08:13:09] ...optimizers/trijoint.py.151: total_norm: 10.268805 [I 2022-10-11 08:13:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 280/2304 [I 2022-10-11 08:13:17] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:17 [I 2022-10-11 08:13:17] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00304 [I 2022-10-11 08:13:17] ...trap/engines/engine.py.230: loss: 0.13133 [I 2022-10-11 08:13:17] ...optimizers/trijoint.py.151: total_norm: 16.499285 [I 2022-10-11 08:13:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 290/2304 [I 2022-10-11 08:13:26] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:02 [I 2022-10-11 08:13:26] ...trap/engines/engine.py.229: process: 0.83122 | load: 0.00307 [I 2022-10-11 08:13:26] ...trap/engines/engine.py.230: loss: 0.23708 [I 2022-10-11 08:13:26] ...optimizers/trijoint.py.151: total_norm: 12.653601 [I 2022-10-11 08:13:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 300/2304 [I 2022-10-11 08:13:34] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:49 [I 2022-10-11 08:13:34] ...trap/engines/engine.py.229: process: 0.83060 | load: 0.00292 [I 2022-10-11 08:13:34] ...trap/engines/engine.py.230: loss: 0.09974 [I 2022-10-11 08:13:34] ...optimizers/trijoint.py.151: total_norm: 14.183752 [I 2022-10-11 08:13:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 310/2304 [I 2022-10-11 08:13:43] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:27:52 [I 2022-10-11 08:13:43] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00293 [I 2022-10-11 08:13:43] ...trap/engines/engine.py.230: loss: 0.09511 [I 2022-10-11 08:13:43] ...optimizers/trijoint.py.151: total_norm: 7.582818 [I 2022-10-11 08:13:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 320/2304 [I 2022-10-11 08:13:51] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:27:38 [I 2022-10-11 08:13:51] ...trap/engines/engine.py.229: process: 0.85001 | load: 0.00295 [I 2022-10-11 08:13:51] ...trap/engines/engine.py.230: loss: 0.10914 [I 2022-10-11 08:13:51] ...optimizers/trijoint.py.151: total_norm: 15.521972 [I 2022-10-11 08:13:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 330/2304 [I 2022-10-11 08:13:59] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:33 [I 2022-10-11 08:13:59] ...trap/engines/engine.py.229: process: 0.84010 | load: 0.00312 [I 2022-10-11 08:13:59] ...trap/engines/engine.py.230: loss: 0.13929 [I 2022-10-11 08:13:59] ...optimizers/trijoint.py.151: total_norm: 11.526414 [I 2022-10-11 08:14:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 340/2304 [I 2022-10-11 08:14:08] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:42 [I 2022-10-11 08:14:08] ...trap/engines/engine.py.229: process: 0.84539 | load: 0.00300 [I 2022-10-11 08:14:08] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-11 08:14:08] ...optimizers/trijoint.py.151: total_norm: 11.003263 [I 2022-10-11 08:14:16] ...trap/engines/engine.py.224: train: epoch 98 | batch 350/2304 [I 2022-10-11 08:14:16] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:13 [I 2022-10-11 08:14:16] ...trap/engines/engine.py.229: process: 0.82126 | load: 0.00296 [I 2022-10-11 08:14:16] ...trap/engines/engine.py.230: loss: 0.17090 [I 2022-10-11 08:14:16] ...optimizers/trijoint.py.151: total_norm: 18.242977 [I 2022-10-11 08:14:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 360/2304 [I 2022-10-11 08:14:25] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:27:20 [I 2022-10-11 08:14:25] ...trap/engines/engine.py.229: process: 0.84632 | load: 0.00291 [I 2022-10-11 08:14:25] ...trap/engines/engine.py.230: loss: 0.09494 [I 2022-10-11 08:14:25] ...optimizers/trijoint.py.151: total_norm: 9.339080 [I 2022-10-11 08:14:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 370/2304 [I 2022-10-11 08:14:33] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:00 [I 2022-10-11 08:14:33] ...trap/engines/engine.py.229: process: 0.83140 | load: 0.00293 [I 2022-10-11 08:14:33] ...trap/engines/engine.py.230: loss: 0.08828 [I 2022-10-11 08:14:33] ...optimizers/trijoint.py.151: total_norm: 9.944439 [I 2022-10-11 08:14:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 380/2304 [I 2022-10-11 08:14:41] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:01 [I 2022-10-11 08:14:41] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00293 [I 2022-10-11 08:14:41] ...trap/engines/engine.py.230: loss: 0.18667 [I 2022-10-11 08:14:41] ...optimizers/trijoint.py.151: total_norm: 11.394411 [I 2022-10-11 08:14:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 390/2304 [I 2022-10-11 08:14:50] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:26:52 [I 2022-10-11 08:14:50] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00294 [I 2022-10-11 08:14:50] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-11 08:14:50] ...optimizers/trijoint.py.151: total_norm: 12.078197 [I 2022-10-11 08:14:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 400/2304 [I 2022-10-11 08:14:58] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:26:31 [I 2022-10-11 08:14:58] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00293 [I 2022-10-11 08:14:58] ...trap/engines/engine.py.230: loss: 0.11056 [I 2022-10-11 08:14:58] ...optimizers/trijoint.py.151: total_norm: 9.101481 [I 2022-10-11 08:15:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 410/2304 [I 2022-10-11 08:15:07] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:26:33 [I 2022-10-11 08:15:07] ...trap/engines/engine.py.229: process: 0.83444 | load: 0.00305 [I 2022-10-11 08:15:07] ...trap/engines/engine.py.230: loss: 0.07590 [I 2022-10-11 08:15:07] ...optimizers/trijoint.py.151: total_norm: 11.129728 [I 2022-10-11 08:15:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 420/2304 [I 2022-10-11 08:15:15] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:26:48 [I 2022-10-11 08:15:15] ...trap/engines/engine.py.229: process: 0.85889 | load: 0.00374 [I 2022-10-11 08:15:15] ...trap/engines/engine.py.230: loss: 0.10098 [I 2022-10-11 08:15:15] ...optimizers/trijoint.py.151: total_norm: 9.090347 [I 2022-10-11 08:15:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 430/2304 [I 2022-10-11 08:15:24] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:59 [I 2022-10-11 08:15:24] ...trap/engines/engine.py.229: process: 0.86075 | load: 0.00374 [I 2022-10-11 08:15:24] ...trap/engines/engine.py.230: loss: 0.19135 [I 2022-10-11 08:15:24] ...optimizers/trijoint.py.151: total_norm: 14.614803 [I 2022-10-11 08:15:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 440/2304 [I 2022-10-11 08:15:33] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:26:55 [I 2022-10-11 08:15:33] ...trap/engines/engine.py.229: process: 0.86587 | load: 0.00373 [I 2022-10-11 08:15:33] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-11 08:15:33] ...optimizers/trijoint.py.151: total_norm: 11.515730 [I 2022-10-11 08:15:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 450/2304 [I 2022-10-11 08:15:41] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:26:44 [I 2022-10-11 08:15:41] ...trap/engines/engine.py.229: process: 0.87034 | load: 0.00373 [I 2022-10-11 08:15:41] ...trap/engines/engine.py.230: loss: 0.09647 [I 2022-10-11 08:15:41] ...optimizers/trijoint.py.151: total_norm: 12.365298 [I 2022-10-11 08:15:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 460/2304 [I 2022-10-11 08:15:50] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:25 [I 2022-10-11 08:15:50] ...trap/engines/engine.py.229: process: 0.85554 | load: 0.00294 [I 2022-10-11 08:15:50] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-11 08:15:50] ...optimizers/trijoint.py.151: total_norm: 15.728381 [I 2022-10-11 08:15:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 470/2304 [I 2022-10-11 08:15:58] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:41 [I 2022-10-11 08:15:58] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00295 [I 2022-10-11 08:15:58] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-11 08:15:58] ...optimizers/trijoint.py.151: total_norm: 23.143196 [I 2022-10-11 08:16:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 480/2304 [I 2022-10-11 08:16:07] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:15 [I 2022-10-11 08:16:07] ...trap/engines/engine.py.229: process: 0.83358 | load: 0.00296 [I 2022-10-11 08:16:07] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-11 08:16:07] ...optimizers/trijoint.py.151: total_norm: 22.096800 [I 2022-10-11 08:16:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 490/2304 [I 2022-10-11 08:16:15] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:12 [I 2022-10-11 08:16:15] ...trap/engines/engine.py.229: process: 0.82921 | load: 0.00293 [I 2022-10-11 08:16:15] ...trap/engines/engine.py.230: loss: 0.10700 [I 2022-10-11 08:16:15] ...optimizers/trijoint.py.151: total_norm: 11.749179 [I 2022-10-11 08:16:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 500/2304 [I 2022-10-11 08:16:23] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:03 [I 2022-10-11 08:16:23] ...trap/engines/engine.py.229: process: 0.83512 | load: 0.00293 [I 2022-10-11 08:16:23] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-11 08:16:23] ...optimizers/trijoint.py.151: total_norm: 8.108169 [I 2022-10-11 08:16:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 510/2304 [I 2022-10-11 08:16:32] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:24:53 [I 2022-10-11 08:16:32] ...trap/engines/engine.py.229: process: 0.82929 | load: 0.00294 [I 2022-10-11 08:16:32] ...trap/engines/engine.py.230: loss: 0.11141 [I 2022-10-11 08:16:32] ...optimizers/trijoint.py.151: total_norm: 24.680801 [I 2022-10-11 08:16:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 520/2304 [I 2022-10-11 08:16:40] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:24:39 [I 2022-10-11 08:16:40] ...trap/engines/engine.py.229: process: 0.82777 | load: 0.00293 [I 2022-10-11 08:16:40] ...trap/engines/engine.py.230: loss: 0.09918 [I 2022-10-11 08:16:40] ...optimizers/trijoint.py.151: total_norm: 12.538960 [I 2022-10-11 08:16:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 530/2304 [I 2022-10-11 08:16:48] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:24:45 [I 2022-10-11 08:16:48] ...trap/engines/engine.py.229: process: 0.82844 | load: 0.00301 [I 2022-10-11 08:16:48] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-11 08:16:48] ...optimizers/trijoint.py.151: total_norm: 12.020291 [I 2022-10-11 08:16:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 540/2304 [I 2022-10-11 08:16:57] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:55 [I 2022-10-11 08:16:57] ...trap/engines/engine.py.229: process: 0.87330 | load: 0.00373 [I 2022-10-11 08:16:57] ...trap/engines/engine.py.230: loss: 0.10002 [I 2022-10-11 08:16:57] ...optimizers/trijoint.py.151: total_norm: 15.169513 [I 2022-10-11 08:17:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 550/2304 [I 2022-10-11 08:17:05] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:25:18 [I 2022-10-11 08:17:05] ...trap/engines/engine.py.229: process: 0.86660 | load: 0.00373 [I 2022-10-11 08:17:05] ...trap/engines/engine.py.230: loss: 0.11695 [I 2022-10-11 08:17:05] ...optimizers/trijoint.py.151: total_norm: 23.398726 [I 2022-10-11 08:17:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 560/2304 [I 2022-10-11 08:17:14] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:11 [I 2022-10-11 08:17:14] ...trap/engines/engine.py.229: process: 0.86933 | load: 0.00376 [I 2022-10-11 08:17:14] ...trap/engines/engine.py.230: loss: 0.10305 [I 2022-10-11 08:17:14] ...optimizers/trijoint.py.151: total_norm: 11.356402 [I 2022-10-11 08:17:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 570/2304 [I 2022-10-11 08:17:23] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:58 [I 2022-10-11 08:17:23] ...trap/engines/engine.py.229: process: 0.87351 | load: 0.00372 [I 2022-10-11 08:17:23] ...trap/engines/engine.py.230: loss: 0.14983 [I 2022-10-11 08:17:23] ...optimizers/trijoint.py.151: total_norm: 26.352760 [I 2022-10-11 08:17:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 580/2304 [I 2022-10-11 08:17:31] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:43 [I 2022-10-11 08:17:31] ...trap/engines/engine.py.229: process: 0.86797 | load: 0.00373 [I 2022-10-11 08:17:31] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-11 08:17:31] ...optimizers/trijoint.py.151: total_norm: 9.432463 [I 2022-10-11 08:17:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 590/2304 [I 2022-10-11 08:17:40] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:23:50 [I 2022-10-11 08:17:40] ...trap/engines/engine.py.229: process: 0.82633 | load: 0.00295 [I 2022-10-11 08:17:40] ...trap/engines/engine.py.230: loss: 0.12398 [I 2022-10-11 08:17:40] ...optimizers/trijoint.py.151: total_norm: 11.079288 [I 2022-10-11 08:17:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 600/2304 [I 2022-10-11 08:17:48] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:23:44 [I 2022-10-11 08:17:48] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00294 [I 2022-10-11 08:17:48] ...trap/engines/engine.py.230: loss: 0.10870 [I 2022-10-11 08:17:48] ...optimizers/trijoint.py.151: total_norm: 28.034632 [I 2022-10-11 08:17:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 610/2304 [I 2022-10-11 08:17:57] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:37 [I 2022-10-11 08:17:57] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00295 [I 2022-10-11 08:17:57] ...trap/engines/engine.py.230: loss: 0.11680 [I 2022-10-11 08:17:57] ...optimizers/trijoint.py.151: total_norm: 18.800180 [I 2022-10-11 08:18:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 620/2304 [I 2022-10-11 08:18:05] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:38 [I 2022-10-11 08:18:05] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00296 [I 2022-10-11 08:18:05] ...trap/engines/engine.py.230: loss: 0.15504 [I 2022-10-11 08:18:05] ...optimizers/trijoint.py.151: total_norm: 14.967399 [I 2022-10-11 08:18:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 630/2304 [I 2022-10-11 08:18:13] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:24 [I 2022-10-11 08:18:13] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00300 [I 2022-10-11 08:18:13] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-11 08:18:13] ...optimizers/trijoint.py.151: total_norm: 9.715059 [I 2022-10-11 08:18:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 640/2304 [I 2022-10-11 08:18:22] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:18 [I 2022-10-11 08:18:22] ...trap/engines/engine.py.229: process: 0.83255 | load: 0.00295 [I 2022-10-11 08:18:22] ...trap/engines/engine.py.230: loss: 0.09040 [I 2022-10-11 08:18:22] ...optimizers/trijoint.py.151: total_norm: 10.577036 [I 2022-10-11 08:18:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 650/2304 [I 2022-10-11 08:18:30] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:22:59 [I 2022-10-11 08:18:30] ...trap/engines/engine.py.229: process: 0.84771 | load: 0.00297 [I 2022-10-11 08:18:30] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-11 08:18:30] ...optimizers/trijoint.py.151: total_norm: 19.142918 [I 2022-10-11 08:18:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 660/2304 [I 2022-10-11 08:18:39] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:08 [I 2022-10-11 08:18:39] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00296 [I 2022-10-11 08:18:39] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-11 08:18:39] ...optimizers/trijoint.py.151: total_norm: 13.470062 [I 2022-10-11 08:18:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 670/2304 [I 2022-10-11 08:18:47] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:22:55 [I 2022-10-11 08:18:47] ...trap/engines/engine.py.229: process: 0.85245 | load: 0.00295 [I 2022-10-11 08:18:47] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-11 08:18:47] ...optimizers/trijoint.py.151: total_norm: 43.761303 [I 2022-10-11 08:18:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 680/2304 [I 2022-10-11 08:18:55] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:55 [I 2022-10-11 08:18:55] ...trap/engines/engine.py.229: process: 0.85471 | load: 0.00292 [I 2022-10-11 08:18:55] ...trap/engines/engine.py.230: loss: 0.17525 [I 2022-10-11 08:18:55] ...optimizers/trijoint.py.151: total_norm: 17.505758 [I 2022-10-11 08:19:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 690/2304 [I 2022-10-11 08:19:04] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:22:33 [I 2022-10-11 08:19:04] ...trap/engines/engine.py.229: process: 0.83626 | load: 0.00296 [I 2022-10-11 08:19:04] ...trap/engines/engine.py.230: loss: 0.19300 [I 2022-10-11 08:19:04] ...optimizers/trijoint.py.151: total_norm: 23.280439 [I 2022-10-11 08:19:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 700/2304 [I 2022-10-11 08:19:12] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:17 [I 2022-10-11 08:19:12] ...trap/engines/engine.py.229: process: 0.83418 | load: 0.00294 [I 2022-10-11 08:19:12] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-11 08:19:12] ...optimizers/trijoint.py.151: total_norm: 14.757357 [I 2022-10-11 08:19:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 710/2304 [I 2022-10-11 08:19:21] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:12 [I 2022-10-11 08:19:21] ...trap/engines/engine.py.229: process: 0.83402 | load: 0.00295 [I 2022-10-11 08:19:21] ...trap/engines/engine.py.230: loss: 0.11816 [I 2022-10-11 08:19:21] ...optimizers/trijoint.py.151: total_norm: 26.143143 [I 2022-10-11 08:19:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 720/2304 [I 2022-10-11 08:19:29] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:21:56 [I 2022-10-11 08:19:29] ...trap/engines/engine.py.229: process: 0.82636 | load: 0.00294 [I 2022-10-11 08:19:29] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-11 08:19:29] ...optimizers/trijoint.py.151: total_norm: 8.779632 [I 2022-10-11 08:19:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 730/2304 [I 2022-10-11 08:19:37] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:21:55 [I 2022-10-11 08:19:37] ...trap/engines/engine.py.229: process: 0.83988 | load: 0.00294 [I 2022-10-11 08:19:37] ...trap/engines/engine.py.230: loss: 0.17563 [I 2022-10-11 08:19:37] ...optimizers/trijoint.py.151: total_norm: 12.417146 [I 2022-10-11 08:19:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 740/2304 [I 2022-10-11 08:19:46] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:46 [I 2022-10-11 08:19:46] ...trap/engines/engine.py.229: process: 0.83157 | load: 0.00296 [I 2022-10-11 08:19:46] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-11 08:19:46] ...optimizers/trijoint.py.151: total_norm: 11.391092 [I 2022-10-11 08:19:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 750/2304 [I 2022-10-11 08:19:54] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:38 [I 2022-10-11 08:19:54] ...trap/engines/engine.py.229: process: 0.83247 | load: 0.00295 [I 2022-10-11 08:19:54] ...trap/engines/engine.py.230: loss: 0.10896 [I 2022-10-11 08:19:54] ...optimizers/trijoint.py.151: total_norm: 11.638392 [I 2022-10-11 08:20:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 760/2304 [I 2022-10-11 08:20:02] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:32 [I 2022-10-11 08:20:02] ...trap/engines/engine.py.229: process: 0.82904 | load: 0.00296 [I 2022-10-11 08:20:02] ...trap/engines/engine.py.230: loss: 0.17039 [I 2022-10-11 08:20:02] ...optimizers/trijoint.py.151: total_norm: 19.300110 [I 2022-10-11 08:20:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 770/2304 [I 2022-10-11 08:20:11] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:21:23 [I 2022-10-11 08:20:11] ...trap/engines/engine.py.229: process: 0.83441 | load: 0.00295 [I 2022-10-11 08:20:11] ...trap/engines/engine.py.230: loss: 0.10301 [I 2022-10-11 08:20:11] ...optimizers/trijoint.py.151: total_norm: 15.479106 [I 2022-10-11 08:20:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 780/2304 [I 2022-10-11 08:20:19] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:17 [I 2022-10-11 08:20:19] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00295 [I 2022-10-11 08:20:19] ...trap/engines/engine.py.230: loss: 0.14228 [I 2022-10-11 08:20:19] ...optimizers/trijoint.py.151: total_norm: 25.151331 [I 2022-10-11 08:20:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 790/2304 [I 2022-10-11 08:20:27] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:05 [I 2022-10-11 08:20:27] ...trap/engines/engine.py.229: process: 0.83426 | load: 0.00296 [I 2022-10-11 08:20:27] ...trap/engines/engine.py.230: loss: 0.11193 [I 2022-10-11 08:20:27] ...optimizers/trijoint.py.151: total_norm: 11.127513 [I 2022-10-11 08:20:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 800/2304 [I 2022-10-11 08:20:36] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:20:49 [I 2022-10-11 08:20:36] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00296 [I 2022-10-11 08:20:36] ...trap/engines/engine.py.230: loss: 0.13498 [I 2022-10-11 08:20:36] ...optimizers/trijoint.py.151: total_norm: 19.250900 [I 2022-10-11 08:20:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 810/2304 [I 2022-10-11 08:20:44] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:43 [I 2022-10-11 08:20:44] ...trap/engines/engine.py.229: process: 0.83035 | load: 0.00295 [I 2022-10-11 08:20:44] ...trap/engines/engine.py.230: loss: 0.08769 [I 2022-10-11 08:20:44] ...optimizers/trijoint.py.151: total_norm: 10.672813 [I 2022-10-11 08:20:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 820/2304 [I 2022-10-11 08:20:52] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:20:40 [I 2022-10-11 08:20:52] ...trap/engines/engine.py.229: process: 0.83664 | load: 0.00293 [I 2022-10-11 08:20:52] ...trap/engines/engine.py.230: loss: 0.10386 [I 2022-10-11 08:20:52] ...optimizers/trijoint.py.151: total_norm: 14.941300 [I 2022-10-11 08:21:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 830/2304 [I 2022-10-11 08:21:01] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:32 [I 2022-10-11 08:21:01] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00297 [I 2022-10-11 08:21:01] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-11 08:21:01] ...optimizers/trijoint.py.151: total_norm: 8.991756 [I 2022-10-11 08:21:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 840/2304 [I 2022-10-11 08:21:09] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:57 [I 2022-10-11 08:21:09] ...trap/engines/engine.py.229: process: 0.86740 | load: 0.00373 [I 2022-10-11 08:21:09] ...trap/engines/engine.py.230: loss: 0.11698 [I 2022-10-11 08:21:09] ...optimizers/trijoint.py.151: total_norm: 44.778362 [I 2022-10-11 08:21:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 850/2304 [I 2022-10-11 08:21:18] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:56 [I 2022-10-11 08:21:18] ...trap/engines/engine.py.229: process: 0.87279 | load: 0.00374 [I 2022-10-11 08:21:18] ...trap/engines/engine.py.230: loss: 0.12326 [I 2022-10-11 08:21:18] ...optimizers/trijoint.py.151: total_norm: 33.271305 [I 2022-10-11 08:21:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 860/2304 [I 2022-10-11 08:21:27] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:48 [I 2022-10-11 08:21:27] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00372 [I 2022-10-11 08:21:27] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-11 08:21:27] ...optimizers/trijoint.py.151: total_norm: 30.339180 [I 2022-10-11 08:21:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 870/2304 [I 2022-10-11 08:21:35] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:20:42 [I 2022-10-11 08:21:35] ...trap/engines/engine.py.229: process: 0.86858 | load: 0.00373 [I 2022-10-11 08:21:35] ...trap/engines/engine.py.230: loss: 0.11286 [I 2022-10-11 08:21:35] ...optimizers/trijoint.py.151: total_norm: 39.664352 [I 2022-10-11 08:21:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 880/2304 [I 2022-10-11 08:21:44] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:34 [I 2022-10-11 08:21:44] ...trap/engines/engine.py.229: process: 0.86104 | load: 0.00372 [I 2022-10-11 08:21:44] ...trap/engines/engine.py.230: loss: 0.22224 [I 2022-10-11 08:21:44] ...optimizers/trijoint.py.151: total_norm: 32.365372 [I 2022-10-11 08:21:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 890/2304 [I 2022-10-11 08:21:53] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:23 [I 2022-10-11 08:21:53] ...trap/engines/engine.py.229: process: 0.87281 | load: 0.00375 [I 2022-10-11 08:21:53] ...trap/engines/engine.py.230: loss: 0.12176 [I 2022-10-11 08:21:53] ...optimizers/trijoint.py.151: total_norm: 16.585732 [I 2022-10-11 08:22:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 900/2304 [I 2022-10-11 08:22:01] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:19:56 [I 2022-10-11 08:22:01] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00294 [I 2022-10-11 08:22:01] ...trap/engines/engine.py.230: loss: 0.07773 [I 2022-10-11 08:22:01] ...optimizers/trijoint.py.151: total_norm: 11.024022 [I 2022-10-11 08:22:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 910/2304 [I 2022-10-11 08:22:10] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:26 [I 2022-10-11 08:22:10] ...trap/engines/engine.py.229: process: 0.83356 | load: 0.00295 [I 2022-10-11 08:22:10] ...trap/engines/engine.py.230: loss: 0.08815 [I 2022-10-11 08:22:10] ...optimizers/trijoint.py.151: total_norm: 10.444331 [I 2022-10-11 08:22:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 920/2304 [I 2022-10-11 08:22:18] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:19:29 [I 2022-10-11 08:22:18] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00293 [I 2022-10-11 08:22:18] ...trap/engines/engine.py.230: loss: 0.09235 [I 2022-10-11 08:22:18] ...optimizers/trijoint.py.151: total_norm: 10.681911 [I 2022-10-11 08:22:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 930/2304 [I 2022-10-11 08:22:27] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:57 [I 2022-10-11 08:22:27] ...trap/engines/engine.py.229: process: 0.87592 | load: 0.00374 [I 2022-10-11 08:22:27] ...trap/engines/engine.py.230: loss: 0.20276 [I 2022-10-11 08:22:27] ...optimizers/trijoint.py.151: total_norm: 12.488588 [I 2022-10-11 08:22:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 940/2304 [I 2022-10-11 08:22:36] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:19:55 [I 2022-10-11 08:22:36] ...trap/engines/engine.py.229: process: 0.87136 | load: 0.00373 [I 2022-10-11 08:22:36] ...trap/engines/engine.py.230: loss: 0.08739 [I 2022-10-11 08:22:36] ...optimizers/trijoint.py.151: total_norm: 12.535651 [I 2022-10-11 08:22:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 950/2304 [I 2022-10-11 08:22:44] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:38 [I 2022-10-11 08:22:44] ...trap/engines/engine.py.229: process: 0.87069 | load: 0.00373 [I 2022-10-11 08:22:44] ...trap/engines/engine.py.230: loss: 0.09509 [I 2022-10-11 08:22:44] ...optimizers/trijoint.py.151: total_norm: 11.310381 [I 2022-10-11 08:22:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 960/2304 [I 2022-10-11 08:22:53] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:22 [I 2022-10-11 08:22:53] ...trap/engines/engine.py.229: process: 0.85822 | load: 0.00374 [I 2022-10-11 08:22:53] ...trap/engines/engine.py.230: loss: 0.10002 [I 2022-10-11 08:22:53] ...optimizers/trijoint.py.151: total_norm: 20.482504 [I 2022-10-11 08:23:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 970/2304 [I 2022-10-11 08:23:02] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:16 [I 2022-10-11 08:23:02] ...trap/engines/engine.py.229: process: 0.87739 | load: 0.00373 [I 2022-10-11 08:23:02] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-11 08:23:02] ...optimizers/trijoint.py.151: total_norm: 10.170882 [I 2022-10-11 08:23:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 980/2304 [I 2022-10-11 08:23:10] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:18:47 [I 2022-10-11 08:23:10] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00295 [I 2022-10-11 08:23:10] ...trap/engines/engine.py.230: loss: 0.08457 [I 2022-10-11 08:23:10] ...optimizers/trijoint.py.151: total_norm: 11.522520 [I 2022-10-11 08:23:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 990/2304 [I 2022-10-11 08:23:19] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:13 [I 2022-10-11 08:23:19] ...trap/engines/engine.py.229: process: 0.82880 | load: 0.00294 [I 2022-10-11 08:23:19] ...trap/engines/engine.py.230: loss: 0.09741 [I 2022-10-11 08:23:19] ...optimizers/trijoint.py.151: total_norm: 12.857328 [I 2022-10-11 08:23:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 1000/2304 [I 2022-10-11 08:23:27] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:11 [I 2022-10-11 08:23:27] ...trap/engines/engine.py.229: process: 0.83436 | load: 0.00293 [I 2022-10-11 08:23:27] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-11 08:23:27] ...optimizers/trijoint.py.151: total_norm: 10.701320 [I 2022-10-11 08:23:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 1010/2304 [I 2022-10-11 08:23:35] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:17:56 [I 2022-10-11 08:23:35] ...trap/engines/engine.py.229: process: 0.82442 | load: 0.00293 [I 2022-10-11 08:23:35] ...trap/engines/engine.py.230: loss: 0.12863 [I 2022-10-11 08:23:35] ...optimizers/trijoint.py.151: total_norm: 14.724751 [I 2022-10-11 08:23:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 1020/2304 [I 2022-10-11 08:23:44] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:17:55 [I 2022-10-11 08:23:44] ...trap/engines/engine.py.229: process: 0.83890 | load: 0.00295 [I 2022-10-11 08:23:44] ...trap/engines/engine.py.230: loss: 0.10058 [I 2022-10-11 08:23:44] ...optimizers/trijoint.py.151: total_norm: 15.565481 [I 2022-10-11 08:23:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 1030/2304 [I 2022-10-11 08:23:52] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:17:53 [I 2022-10-11 08:23:52] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00299 [I 2022-10-11 08:23:52] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-11 08:23:52] ...optimizers/trijoint.py.151: total_norm: 17.224358 [I 2022-10-11 08:24:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 1040/2304 [I 2022-10-11 08:24:01] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:17:35 [I 2022-10-11 08:24:01] ...trap/engines/engine.py.229: process: 0.82340 | load: 0.00293 [I 2022-10-11 08:24:01] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-11 08:24:01] ...optimizers/trijoint.py.151: total_norm: 8.839292 [I 2022-10-11 08:24:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 1050/2304 [I 2022-10-11 08:24:09] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:17:42 [I 2022-10-11 08:24:09] ...trap/engines/engine.py.229: process: 0.87230 | load: 0.00295 [I 2022-10-11 08:24:09] ...trap/engines/engine.py.230: loss: 0.14437 [I 2022-10-11 08:24:09] ...optimizers/trijoint.py.151: total_norm: 17.640282 [I 2022-10-11 08:24:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 1060/2304 [I 2022-10-11 08:24:17] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:26 [I 2022-10-11 08:24:17] ...trap/engines/engine.py.229: process: 0.84569 | load: 0.00295 [I 2022-10-11 08:24:17] ...trap/engines/engine.py.230: loss: 0.11006 [I 2022-10-11 08:24:17] ...optimizers/trijoint.py.151: total_norm: 8.919969 [I 2022-10-11 08:24:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 1070/2304 [I 2022-10-11 08:24:26] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:09 [I 2022-10-11 08:24:26] ...trap/engines/engine.py.229: process: 0.84347 | load: 0.00296 [I 2022-10-11 08:24:26] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-11 08:24:26] ...optimizers/trijoint.py.151: total_norm: 18.577572 [I 2022-10-11 08:24:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 1080/2304 [I 2022-10-11 08:24:34] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:08 [I 2022-10-11 08:24:34] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00293 [I 2022-10-11 08:24:34] ...trap/engines/engine.py.230: loss: 0.08601 [I 2022-10-11 08:24:34] ...optimizers/trijoint.py.151: total_norm: 10.129511 [I 2022-10-11 08:24:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 1090/2304 [I 2022-10-11 08:24:43] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:01 [I 2022-10-11 08:24:43] ...trap/engines/engine.py.229: process: 0.82946 | load: 0.00295 [I 2022-10-11 08:24:43] ...trap/engines/engine.py.230: loss: 0.08009 [I 2022-10-11 08:24:43] ...optimizers/trijoint.py.151: total_norm: 18.802927 [I 2022-10-11 08:24:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 1100/2304 [I 2022-10-11 08:24:51] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:16:49 [I 2022-10-11 08:24:51] ...trap/engines/engine.py.229: process: 0.82396 | load: 0.00297 [I 2022-10-11 08:24:51] ...trap/engines/engine.py.230: loss: 0.18587 [I 2022-10-11 08:24:51] ...optimizers/trijoint.py.151: total_norm: 7.966029 [I 2022-10-11 08:24:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 1110/2304 [I 2022-10-11 08:24:59] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:16:41 [I 2022-10-11 08:24:59] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00293 [I 2022-10-11 08:24:59] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-11 08:24:59] ...optimizers/trijoint.py.151: total_norm: 14.791017 [I 2022-10-11 08:25:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 1120/2304 [I 2022-10-11 08:25:08] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:37 [I 2022-10-11 08:25:08] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00294 [I 2022-10-11 08:25:08] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-11 08:25:08] ...optimizers/trijoint.py.151: total_norm: 18.320398 [I 2022-10-11 08:25:16] ...trap/engines/engine.py.224: train: epoch 98 | batch 1130/2304 [I 2022-10-11 08:25:16] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:32 [I 2022-10-11 08:25:16] ...trap/engines/engine.py.229: process: 0.83221 | load: 0.00292 [I 2022-10-11 08:25:16] ...trap/engines/engine.py.230: loss: 0.12474 [I 2022-10-11 08:25:16] ...optimizers/trijoint.py.151: total_norm: 15.618676 [I 2022-10-11 08:25:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 1140/2304 [I 2022-10-11 08:25:25] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:22 [I 2022-10-11 08:25:25] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00294 [I 2022-10-11 08:25:25] ...trap/engines/engine.py.230: loss: 0.10068 [I 2022-10-11 08:25:25] ...optimizers/trijoint.py.151: total_norm: 8.969853 [I 2022-10-11 08:25:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 1150/2304 [I 2022-10-11 08:25:33] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:13 [I 2022-10-11 08:25:33] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00293 [I 2022-10-11 08:25:33] ...trap/engines/engine.py.230: loss: 0.10777 [I 2022-10-11 08:25:33] ...optimizers/trijoint.py.151: total_norm: 14.567968 [I 2022-10-11 08:25:42] ...trap/engines/engine.py.224: train: epoch 98 | batch 1160/2304 [I 2022-10-11 08:25:42] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:05 [I 2022-10-11 08:25:42] ...trap/engines/engine.py.229: process: 0.83958 | load: 0.00295 [I 2022-10-11 08:25:42] ...trap/engines/engine.py.230: loss: 0.11123 [I 2022-10-11 08:25:42] ...optimizers/trijoint.py.151: total_norm: 22.035885 [I 2022-10-11 08:25:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 1170/2304 [I 2022-10-11 08:25:50] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:15:52 [I 2022-10-11 08:25:50] ...trap/engines/engine.py.229: process: 0.82561 | load: 0.00294 [I 2022-10-11 08:25:50] ...trap/engines/engine.py.230: loss: 0.16782 [I 2022-10-11 08:25:50] ...optimizers/trijoint.py.151: total_norm: 72.590347 [I 2022-10-11 08:25:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 1180/2304 [I 2022-10-11 08:25:59] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:15:51 [I 2022-10-11 08:25:59] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00293 [I 2022-10-11 08:25:59] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-11 08:25:59] ...optimizers/trijoint.py.151: total_norm: 19.225368 [I 2022-10-11 08:26:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 1190/2304 [I 2022-10-11 08:26:07] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:38 [I 2022-10-11 08:26:07] ...trap/engines/engine.py.229: process: 0.84795 | load: 0.00294 [I 2022-10-11 08:26:07] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-11 08:26:07] ...optimizers/trijoint.py.151: total_norm: 21.551918 [I 2022-10-11 08:26:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 1200/2304 [I 2022-10-11 08:26:15] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:23 [I 2022-10-11 08:26:15] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00294 [I 2022-10-11 08:26:15] ...trap/engines/engine.py.230: loss: 0.10974 [I 2022-10-11 08:26:15] ...optimizers/trijoint.py.151: total_norm: 11.828959 [I 2022-10-11 08:26:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 1210/2304 [I 2022-10-11 08:26:24] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:06 [I 2022-10-11 08:26:24] ...trap/engines/engine.py.229: process: 0.82369 | load: 0.00293 [I 2022-10-11 08:26:24] ...trap/engines/engine.py.230: loss: 0.10656 [I 2022-10-11 08:26:24] ...optimizers/trijoint.py.151: total_norm: 12.367695 [I 2022-10-11 08:26:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 1220/2304 [I 2022-10-11 08:26:32] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:28 [I 2022-10-11 08:26:32] ...trap/engines/engine.py.229: process: 0.86106 | load: 0.00373 [I 2022-10-11 08:26:32] ...trap/engines/engine.py.230: loss: 0.09208 [I 2022-10-11 08:26:32] ...optimizers/trijoint.py.151: total_norm: 8.873531 [I 2022-10-11 08:26:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 1230/2304 [I 2022-10-11 08:26:41] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:29 [I 2022-10-11 08:26:41] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00373 [I 2022-10-11 08:26:41] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-11 08:26:41] ...optimizers/trijoint.py.151: total_norm: 8.703267 [I 2022-10-11 08:26:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 1240/2304 [I 2022-10-11 08:26:50] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:21 [I 2022-10-11 08:26:50] ...trap/engines/engine.py.229: process: 0.86395 | load: 0.00373 [I 2022-10-11 08:26:50] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-11 08:26:50] ...optimizers/trijoint.py.151: total_norm: 12.009245 [I 2022-10-11 08:26:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 1250/2304 [I 2022-10-11 08:26:58] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:14:48 [I 2022-10-11 08:26:58] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00296 [I 2022-10-11 08:26:58] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-11 08:26:58] ...optimizers/trijoint.py.151: total_norm: 27.883204 [I 2022-10-11 08:27:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 1260/2304 [I 2022-10-11 08:27:06] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:39 [I 2022-10-11 08:27:06] ...trap/engines/engine.py.229: process: 0.83633 | load: 0.00294 [I 2022-10-11 08:27:06] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-11 08:27:06] ...optimizers/trijoint.py.151: total_norm: 9.665783 [I 2022-10-11 08:27:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 1270/2304 [I 2022-10-11 08:27:15] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:21 [I 2022-10-11 08:27:15] ...trap/engines/engine.py.229: process: 0.82234 | load: 0.00294 [I 2022-10-11 08:27:15] ...trap/engines/engine.py.230: loss: 0.12586 [I 2022-10-11 08:27:15] ...optimizers/trijoint.py.151: total_norm: 14.261647 [I 2022-10-11 08:27:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 1280/2304 [I 2022-10-11 08:27:23] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:11 [I 2022-10-11 08:27:23] ...trap/engines/engine.py.229: process: 0.83324 | load: 0.00294 [I 2022-10-11 08:27:23] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-11 08:27:23] ...optimizers/trijoint.py.151: total_norm: 18.655630 [I 2022-10-11 08:27:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 1290/2304 [I 2022-10-11 08:27:32] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:08 [I 2022-10-11 08:27:32] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00294 [I 2022-10-11 08:27:32] ...trap/engines/engine.py.230: loss: 0.09300 [I 2022-10-11 08:27:32] ...optimizers/trijoint.py.151: total_norm: 15.082001 [I 2022-10-11 08:27:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 1300/2304 [I 2022-10-11 08:27:40] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:10 [I 2022-10-11 08:27:40] ...trap/engines/engine.py.229: process: 0.83918 | load: 0.00292 [I 2022-10-11 08:27:40] ...trap/engines/engine.py.230: loss: 0.09259 [I 2022-10-11 08:27:40] ...optimizers/trijoint.py.151: total_norm: 9.482322 [I 2022-10-11 08:27:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 1310/2304 [I 2022-10-11 08:27:49] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:01 [I 2022-10-11 08:27:49] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00294 [I 2022-10-11 08:27:49] ...trap/engines/engine.py.230: loss: 0.11523 [I 2022-10-11 08:27:49] ...optimizers/trijoint.py.151: total_norm: 10.740067 [I 2022-10-11 08:27:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 1320/2304 [I 2022-10-11 08:27:57] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:13:42 [I 2022-10-11 08:27:57] ...trap/engines/engine.py.229: process: 0.82439 | load: 0.00294 [I 2022-10-11 08:27:57] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-11 08:27:57] ...optimizers/trijoint.py.151: total_norm: 23.048914 [I 2022-10-11 08:28:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 1330/2304 [I 2022-10-11 08:28:05] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:13:43 [I 2022-10-11 08:28:05] ...trap/engines/engine.py.229: process: 0.83728 | load: 0.00295 [I 2022-10-11 08:28:05] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-11 08:28:05] ...optimizers/trijoint.py.151: total_norm: 15.881174 [I 2022-10-11 08:28:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 1340/2304 [I 2022-10-11 08:28:14] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:20 [I 2022-10-11 08:28:14] ...trap/engines/engine.py.229: process: 0.81643 | load: 0.00297 [I 2022-10-11 08:28:14] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-11 08:28:14] ...optimizers/trijoint.py.151: total_norm: 9.850189 [I 2022-10-11 08:28:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 1350/2304 [I 2022-10-11 08:28:22] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:13:14 [I 2022-10-11 08:28:22] ...trap/engines/engine.py.229: process: 0.83162 | load: 0.00294 [I 2022-10-11 08:28:22] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-11 08:28:22] ...optimizers/trijoint.py.151: total_norm: 9.536098 [I 2022-10-11 08:28:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 1360/2304 [I 2022-10-11 08:28:30] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:13:06 [I 2022-10-11 08:28:30] ...trap/engines/engine.py.229: process: 0.83529 | load: 0.00294 [I 2022-10-11 08:28:30] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-11 08:28:30] ...optimizers/trijoint.py.151: total_norm: 6.845977 [I 2022-10-11 08:28:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 1370/2304 [I 2022-10-11 08:28:39] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:01 [I 2022-10-11 08:28:39] ...trap/engines/engine.py.229: process: 0.82866 | load: 0.00295 [I 2022-10-11 08:28:39] ...trap/engines/engine.py.230: loss: 0.10659 [I 2022-10-11 08:28:39] ...optimizers/trijoint.py.151: total_norm: 8.840277 [I 2022-10-11 08:28:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 1380/2304 [I 2022-10-11 08:28:47] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:12:48 [I 2022-10-11 08:28:47] ...trap/engines/engine.py.229: process: 0.83017 | load: 0.00295 [I 2022-10-11 08:28:47] ...trap/engines/engine.py.230: loss: 0.14489 [I 2022-10-11 08:28:47] ...optimizers/trijoint.py.151: total_norm: 25.462065 [I 2022-10-11 08:28:56] ...trap/engines/engine.py.224: train: epoch 98 | batch 1390/2304 [I 2022-10-11 08:28:56] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:12:56 [I 2022-10-11 08:28:56] ...trap/engines/engine.py.229: process: 0.87022 | load: 0.00371 [I 2022-10-11 08:28:56] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-11 08:28:56] ...optimizers/trijoint.py.151: total_norm: 11.886060 [I 2022-10-11 08:29:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 1400/2304 [I 2022-10-11 08:29:04] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:12:34 [I 2022-10-11 08:29:04] ...trap/engines/engine.py.229: process: 0.82826 | load: 0.00293 [I 2022-10-11 08:29:04] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-11 08:29:04] ...optimizers/trijoint.py.151: total_norm: 7.951855 [I 2022-10-11 08:29:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1410/2304 [I 2022-10-11 08:29:12] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:12:26 [I 2022-10-11 08:29:12] ...trap/engines/engine.py.229: process: 0.83630 | load: 0.00319 [I 2022-10-11 08:29:12] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-11 08:29:12] ...optimizers/trijoint.py.151: total_norm: 11.268905 [I 2022-10-11 08:29:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 1420/2304 [I 2022-10-11 08:29:21] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:12:12 [I 2022-10-11 08:29:21] ...trap/engines/engine.py.229: process: 0.81590 | load: 0.00297 [I 2022-10-11 08:29:21] ...trap/engines/engine.py.230: loss: 0.09219 [I 2022-10-11 08:29:21] ...optimizers/trijoint.py.151: total_norm: 12.032293 [I 2022-10-11 08:29:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 1430/2304 [I 2022-10-11 08:29:29] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:14 [I 2022-10-11 08:29:29] ...trap/engines/engine.py.229: process: 0.84340 | load: 0.00297 [I 2022-10-11 08:29:29] ...trap/engines/engine.py.230: loss: 0.10500 [I 2022-10-11 08:29:29] ...optimizers/trijoint.py.151: total_norm: 10.055195 [I 2022-10-11 08:29:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1440/2304 [I 2022-10-11 08:29:37] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:03 [I 2022-10-11 08:29:37] ...trap/engines/engine.py.229: process: 0.83853 | load: 0.00297 [I 2022-10-11 08:29:37] ...trap/engines/engine.py.230: loss: 0.15081 [I 2022-10-11 08:29:37] ...optimizers/trijoint.py.151: total_norm: 15.438083 [I 2022-10-11 08:29:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1450/2304 [I 2022-10-11 08:29:46] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:00 [I 2022-10-11 08:29:46] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00294 [I 2022-10-11 08:29:46] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-11 08:29:46] ...optimizers/trijoint.py.151: total_norm: 38.011688 [I 2022-10-11 08:29:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1460/2304 [I 2022-10-11 08:29:54] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:11:49 [I 2022-10-11 08:29:54] ...trap/engines/engine.py.229: process: 0.83904 | load: 0.00295 [I 2022-10-11 08:29:54] ...trap/engines/engine.py.230: loss: 0.11151 [I 2022-10-11 08:29:54] ...optimizers/trijoint.py.151: total_norm: 17.889143 [I 2022-10-11 08:30:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1470/2304 [I 2022-10-11 08:30:03] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:11:41 [I 2022-10-11 08:30:03] ...trap/engines/engine.py.229: process: 0.82855 | load: 0.00298 [I 2022-10-11 08:30:03] ...trap/engines/engine.py.230: loss: 0.09445 [I 2022-10-11 08:30:03] ...optimizers/trijoint.py.151: total_norm: 12.200951 [I 2022-10-11 08:30:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 1480/2304 [I 2022-10-11 08:30:11] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:11:28 [I 2022-10-11 08:30:11] ...trap/engines/engine.py.229: process: 0.83915 | load: 0.00295 [I 2022-10-11 08:30:11] ...trap/engines/engine.py.230: loss: 0.10781 [I 2022-10-11 08:30:11] ...optimizers/trijoint.py.151: total_norm: 18.237995 [I 2022-10-11 08:30:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1490/2304 [I 2022-10-11 08:30:20] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:39 [I 2022-10-11 08:30:20] ...trap/engines/engine.py.229: process: 0.87705 | load: 0.00373 [I 2022-10-11 08:30:20] ...trap/engines/engine.py.230: loss: 0.14844 [I 2022-10-11 08:30:20] ...optimizers/trijoint.py.151: total_norm: 14.280111 [I 2022-10-11 08:30:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 1500/2304 [I 2022-10-11 08:30:28] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:11:42 [I 2022-10-11 08:30:28] ...trap/engines/engine.py.229: process: 0.89482 | load: 0.00372 [I 2022-10-11 08:30:28] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-11 08:30:28] ...optimizers/trijoint.py.151: total_norm: 20.621855 [I 2022-10-11 08:30:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1510/2304 [I 2022-10-11 08:30:37] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:27 [I 2022-10-11 08:30:37] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00373 [I 2022-10-11 08:30:37] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-11 08:30:37] ...optimizers/trijoint.py.151: total_norm: 11.782486 [I 2022-10-11 08:30:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1520/2304 [I 2022-10-11 08:30:46] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:25 [I 2022-10-11 08:30:46] ...trap/engines/engine.py.229: process: 0.90478 | load: 0.00374 [I 2022-10-11 08:30:46] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-11 08:30:46] ...optimizers/trijoint.py.151: total_norm: 14.711953 [I 2022-10-11 08:30:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1530/2304 [I 2022-10-11 08:30:55] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:11 [I 2022-10-11 08:30:55] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00376 [I 2022-10-11 08:30:55] ...trap/engines/engine.py.230: loss: 0.09879 [I 2022-10-11 08:30:55] ...optimizers/trijoint.py.151: total_norm: 9.167560 [I 2022-10-11 08:31:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1540/2304 [I 2022-10-11 08:31:03] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:02 [I 2022-10-11 08:31:03] ...trap/engines/engine.py.229: process: 0.84406 | load: 0.00377 [I 2022-10-11 08:31:03] ...trap/engines/engine.py.230: loss: 0.14738 [I 2022-10-11 08:31:03] ...optimizers/trijoint.py.151: total_norm: 13.092172 [I 2022-10-11 08:31:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1550/2304 [I 2022-10-11 08:31:12] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:11:00 [I 2022-10-11 08:31:12] ...trap/engines/engine.py.229: process: 0.88820 | load: 0.00377 [I 2022-10-11 08:31:12] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-11 08:31:12] ...optimizers/trijoint.py.151: total_norm: 11.028293 [I 2022-10-11 08:31:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 1560/2304 [I 2022-10-11 08:31:21] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:33 [I 2022-10-11 08:31:21] ...trap/engines/engine.py.229: process: 0.85207 | load: 0.00295 [I 2022-10-11 08:31:21] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-11 08:31:21] ...optimizers/trijoint.py.151: total_norm: 12.566830 [I 2022-10-11 08:31:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 1570/2304 [I 2022-10-11 08:31:29] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:10:17 [I 2022-10-11 08:31:29] ...trap/engines/engine.py.229: process: 0.83481 | load: 0.00295 [I 2022-10-11 08:31:29] ...trap/engines/engine.py.230: loss: 0.12421 [I 2022-10-11 08:31:29] ...optimizers/trijoint.py.151: total_norm: 38.455509 [I 2022-10-11 08:31:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1580/2304 [I 2022-10-11 08:31:37] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:03 [I 2022-10-11 08:31:37] ...trap/engines/engine.py.229: process: 0.83189 | load: 0.00295 [I 2022-10-11 08:31:37] ...trap/engines/engine.py.230: loss: 0.16065 [I 2022-10-11 08:31:37] ...optimizers/trijoint.py.151: total_norm: 13.821656 [I 2022-10-11 08:31:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1590/2304 [I 2022-10-11 08:31:46] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:03 [I 2022-10-11 08:31:46] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00295 [I 2022-10-11 08:31:46] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-11 08:31:46] ...optimizers/trijoint.py.151: total_norm: 12.251413 [I 2022-10-11 08:31:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1600/2304 [I 2022-10-11 08:31:54] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:09:51 [I 2022-10-11 08:31:54] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00294 [I 2022-10-11 08:31:54] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-11 08:31:54] ...optimizers/trijoint.py.151: total_norm: 18.891832 [I 2022-10-11 08:32:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1610/2304 [I 2022-10-11 08:32:03] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:09:47 [I 2022-10-11 08:32:03] ...trap/engines/engine.py.229: process: 0.83850 | load: 0.00300 [I 2022-10-11 08:32:03] ...trap/engines/engine.py.230: loss: 0.13752 [I 2022-10-11 08:32:03] ...optimizers/trijoint.py.151: total_norm: 19.103098 [I 2022-10-11 08:32:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 1620/2304 [I 2022-10-11 08:32:11] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:09:31 [I 2022-10-11 08:32:11] ...trap/engines/engine.py.229: process: 0.83903 | load: 0.00298 [I 2022-10-11 08:32:11] ...trap/engines/engine.py.230: loss: 0.08142 [I 2022-10-11 08:32:11] ...optimizers/trijoint.py.151: total_norm: 10.643206 [I 2022-10-11 08:32:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1630/2304 [I 2022-10-11 08:32:20] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:09:30 [I 2022-10-11 08:32:20] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00295 [I 2022-10-11 08:32:20] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-11 08:32:20] ...optimizers/trijoint.py.151: total_norm: 13.632573 [I 2022-10-11 08:32:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 1640/2304 [I 2022-10-11 08:32:28] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:34 [I 2022-10-11 08:32:28] ...trap/engines/engine.py.229: process: 0.87729 | load: 0.00374 [I 2022-10-11 08:32:28] ...trap/engines/engine.py.230: loss: 0.11110 [I 2022-10-11 08:32:28] ...optimizers/trijoint.py.151: total_norm: 14.172554 [I 2022-10-11 08:32:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1650/2304 [I 2022-10-11 08:32:37] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:26 [I 2022-10-11 08:32:37] ...trap/engines/engine.py.229: process: 0.86946 | load: 0.00376 [I 2022-10-11 08:32:37] ...trap/engines/engine.py.230: loss: 0.09163 [I 2022-10-11 08:32:37] ...optimizers/trijoint.py.151: total_norm: 12.281349 [I 2022-10-11 08:32:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 1660/2304 [I 2022-10-11 08:32:45] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:18 [I 2022-10-11 08:32:45] ...trap/engines/engine.py.229: process: 0.87685 | load: 0.00374 [I 2022-10-11 08:32:46] ...trap/engines/engine.py.230: loss: 0.10488 [I 2022-10-11 08:32:46] ...optimizers/trijoint.py.151: total_norm: 10.351185 [I 2022-10-11 08:32:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1670/2304 [I 2022-10-11 08:32:54] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:07 [I 2022-10-11 08:32:54] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00373 [I 2022-10-11 08:32:54] ...trap/engines/engine.py.230: loss: 0.11493 [I 2022-10-11 08:32:54] ...optimizers/trijoint.py.151: total_norm: 15.554644 [I 2022-10-11 08:33:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1680/2304 [I 2022-10-11 08:33:03] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:08:45 [I 2022-10-11 08:33:03] ...trap/engines/engine.py.229: process: 0.81639 | load: 0.00294 [I 2022-10-11 08:33:03] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-11 08:33:03] ...optimizers/trijoint.py.151: total_norm: 8.787994 [I 2022-10-11 08:33:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 1690/2304 [I 2022-10-11 08:33:11] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:08:39 [I 2022-10-11 08:33:11] ...trap/engines/engine.py.229: process: 0.86396 | load: 0.00372 [I 2022-10-11 08:33:11] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-11 08:33:11] ...optimizers/trijoint.py.151: total_norm: 12.999845 [I 2022-10-11 08:33:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1700/2304 [I 2022-10-11 08:33:20] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:08:46 [I 2022-10-11 08:33:20] ...trap/engines/engine.py.229: process: 0.87599 | load: 0.00374 [I 2022-10-11 08:33:20] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-11 08:33:20] ...optimizers/trijoint.py.151: total_norm: 13.661957 [I 2022-10-11 08:33:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 1710/2304 [I 2022-10-11 08:33:28] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:08:33 [I 2022-10-11 08:33:28] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00374 [I 2022-10-11 08:33:28] ...trap/engines/engine.py.230: loss: 0.11130 [I 2022-10-11 08:33:28] ...optimizers/trijoint.py.151: total_norm: 7.798350 [I 2022-10-11 08:33:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1720/2304 [I 2022-10-11 08:33:37] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:28 [I 2022-10-11 08:33:37] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00374 [I 2022-10-11 08:33:37] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-11 08:33:37] ...optimizers/trijoint.py.151: total_norm: 9.717835 [I 2022-10-11 08:33:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1730/2304 [I 2022-10-11 08:33:46] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:16 [I 2022-10-11 08:33:46] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00375 [I 2022-10-11 08:33:46] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-11 08:33:46] ...optimizers/trijoint.py.151: total_norm: 25.840435 [I 2022-10-11 08:33:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1740/2304 [I 2022-10-11 08:33:55] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:08 [I 2022-10-11 08:33:55] ...trap/engines/engine.py.229: process: 0.88238 | load: 0.00374 [I 2022-10-11 08:33:55] ...trap/engines/engine.py.230: loss: 0.15802 [I 2022-10-11 08:33:55] ...optimizers/trijoint.py.151: total_norm: 28.154085 [I 2022-10-11 08:34:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1750/2304 [I 2022-10-11 08:34:03] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:07:58 [I 2022-10-11 08:34:03] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00374 [I 2022-10-11 08:34:03] ...trap/engines/engine.py.230: loss: 0.25534 [I 2022-10-11 08:34:03] ...optimizers/trijoint.py.151: total_norm: 36.676605 [I 2022-10-11 08:34:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1760/2304 [I 2022-10-11 08:34:12] ...trap/engines/engine.py.228: elapsed: 0:24:53 | left: 0:07:51 [I 2022-10-11 08:34:12] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00374 [I 2022-10-11 08:34:12] ...trap/engines/engine.py.230: loss: 0.09252 [I 2022-10-11 08:34:12] ...optimizers/trijoint.py.151: total_norm: 9.297613 [I 2022-10-11 08:34:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 1770/2304 [I 2022-10-11 08:34:21] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:07:42 [I 2022-10-11 08:34:21] ...trap/engines/engine.py.229: process: 0.86353 | load: 0.00374 [I 2022-10-11 08:34:21] ...trap/engines/engine.py.230: loss: 0.08528 [I 2022-10-11 08:34:21] ...optimizers/trijoint.py.151: total_norm: 23.797043 [I 2022-10-11 08:34:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 1780/2304 [I 2022-10-11 08:34:29] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:07:35 [I 2022-10-11 08:34:29] ...trap/engines/engine.py.229: process: 0.86351 | load: 0.00374 [I 2022-10-11 08:34:29] ...trap/engines/engine.py.230: loss: 0.08615 [I 2022-10-11 08:34:29] ...optimizers/trijoint.py.151: total_norm: 10.472900 [I 2022-10-11 08:34:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 1790/2304 [I 2022-10-11 08:34:38] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:25 [I 2022-10-11 08:34:38] ...trap/engines/engine.py.229: process: 0.86994 | load: 0.00373 [I 2022-10-11 08:34:38] ...trap/engines/engine.py.230: loss: 0.11130 [I 2022-10-11 08:34:38] ...optimizers/trijoint.py.151: total_norm: 13.771803 [I 2022-10-11 08:34:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 1800/2304 [I 2022-10-11 08:34:47] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:18 [I 2022-10-11 08:34:47] ...trap/engines/engine.py.229: process: 0.87912 | load: 0.00382 [I 2022-10-11 08:34:47] ...trap/engines/engine.py.230: loss: 0.12972 [I 2022-10-11 08:34:47] ...optimizers/trijoint.py.151: total_norm: 43.581898 [I 2022-10-11 08:34:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1810/2304 [I 2022-10-11 08:34:55] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:09 [I 2022-10-11 08:34:55] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00373 [I 2022-10-11 08:34:55] ...trap/engines/engine.py.230: loss: 0.09663 [I 2022-10-11 08:34:55] ...optimizers/trijoint.py.151: total_norm: 13.046148 [I 2022-10-11 08:35:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 1820/2304 [I 2022-10-11 08:35:04] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:06:59 [I 2022-10-11 08:35:04] ...trap/engines/engine.py.229: process: 0.86726 | load: 0.00374 [I 2022-10-11 08:35:04] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-11 08:35:04] ...optimizers/trijoint.py.151: total_norm: 12.384569 [I 2022-10-11 08:35:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 1830/2304 [I 2022-10-11 08:35:13] ...trap/engines/engine.py.228: elapsed: 0:25:53 | left: 0:06:53 [I 2022-10-11 08:35:13] ...trap/engines/engine.py.229: process: 0.88674 | load: 0.00374 [I 2022-10-11 08:35:13] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-11 08:35:13] ...optimizers/trijoint.py.151: total_norm: 34.710400 [I 2022-10-11 08:35:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 1840/2304 [I 2022-10-11 08:35:21] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:42 [I 2022-10-11 08:35:21] ...trap/engines/engine.py.229: process: 0.87225 | load: 0.00374 [I 2022-10-11 08:35:21] ...trap/engines/engine.py.230: loss: 0.09056 [I 2022-10-11 08:35:21] ...optimizers/trijoint.py.151: total_norm: 9.636148 [I 2022-10-11 08:35:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 1850/2304 [I 2022-10-11 08:35:30] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:22 [I 2022-10-11 08:35:30] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00298 [I 2022-10-11 08:35:30] ...trap/engines/engine.py.230: loss: 0.16098 [I 2022-10-11 08:35:30] ...optimizers/trijoint.py.151: total_norm: 16.702843 [I 2022-10-11 08:35:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 1860/2304 [I 2022-10-11 08:35:38] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:14 [I 2022-10-11 08:35:38] ...trap/engines/engine.py.229: process: 0.85537 | load: 0.00296 [I 2022-10-11 08:35:38] ...trap/engines/engine.py.230: loss: 0.11413 [I 2022-10-11 08:35:38] ...optimizers/trijoint.py.151: total_norm: 11.472903 [I 2022-10-11 08:35:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 1870/2304 [I 2022-10-11 08:35:47] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:05 [I 2022-10-11 08:35:47] ...trap/engines/engine.py.229: process: 0.82974 | load: 0.00296 [I 2022-10-11 08:35:47] ...trap/engines/engine.py.230: loss: 0.09733 [I 2022-10-11 08:35:47] ...optimizers/trijoint.py.151: total_norm: 17.097904 [I 2022-10-11 08:35:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1880/2304 [I 2022-10-11 08:35:55] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:05:59 [I 2022-10-11 08:35:55] ...trap/engines/engine.py.229: process: 0.85066 | load: 0.00298 [I 2022-10-11 08:35:55] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-11 08:35:55] ...optimizers/trijoint.py.151: total_norm: 22.380280 [I 2022-10-11 08:36:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 1890/2304 [I 2022-10-11 08:36:04] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:05:50 [I 2022-10-11 08:36:04] ...trap/engines/engine.py.229: process: 0.85259 | load: 0.00297 [I 2022-10-11 08:36:04] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-11 08:36:04] ...optimizers/trijoint.py.151: total_norm: 14.145950 [I 2022-10-11 08:36:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1900/2304 [I 2022-10-11 08:36:12] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:40 [I 2022-10-11 08:36:12] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00295 [I 2022-10-11 08:36:12] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-11 08:36:12] ...optimizers/trijoint.py.151: total_norm: 10.237341 [I 2022-10-11 08:36:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 1910/2304 [I 2022-10-11 08:36:21] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:32 [I 2022-10-11 08:36:21] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00295 [I 2022-10-11 08:36:21] ...trap/engines/engine.py.230: loss: 0.11522 [I 2022-10-11 08:36:21] ...optimizers/trijoint.py.151: total_norm: 24.624601 [I 2022-10-11 08:36:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 1920/2304 [I 2022-10-11 08:36:29] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:23 [I 2022-10-11 08:36:29] ...trap/engines/engine.py.229: process: 0.85535 | load: 0.00296 [I 2022-10-11 08:36:29] ...trap/engines/engine.py.230: loss: 0.16680 [I 2022-10-11 08:36:29] ...optimizers/trijoint.py.151: total_norm: 15.959199 [I 2022-10-11 08:36:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1930/2304 [I 2022-10-11 08:36:37] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:13 [I 2022-10-11 08:36:37] ...trap/engines/engine.py.229: process: 0.83289 | load: 0.00297 [I 2022-10-11 08:36:37] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-11 08:36:37] ...optimizers/trijoint.py.151: total_norm: 12.516451 [I 2022-10-11 08:36:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1940/2304 [I 2022-10-11 08:36:46] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:05 [I 2022-10-11 08:36:46] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00294 [I 2022-10-11 08:36:46] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-11 08:36:46] ...optimizers/trijoint.py.151: total_norm: 9.537180 [I 2022-10-11 08:36:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1950/2304 [I 2022-10-11 08:36:54] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:04:56 [I 2022-10-11 08:36:54] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00299 [I 2022-10-11 08:36:54] ...trap/engines/engine.py.230: loss: 0.10257 [I 2022-10-11 08:36:54] ...optimizers/trijoint.py.151: total_norm: 17.261354 [I 2022-10-11 08:37:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 1960/2304 [I 2022-10-11 08:37:02] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:04:45 [I 2022-10-11 08:37:02] ...trap/engines/engine.py.229: process: 0.82256 | load: 0.00295 [I 2022-10-11 08:37:02] ...trap/engines/engine.py.230: loss: 0.11904 [I 2022-10-11 08:37:02] ...optimizers/trijoint.py.151: total_norm: 10.873939 [I 2022-10-11 08:37:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 1970/2304 [I 2022-10-11 08:37:11] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:38 [I 2022-10-11 08:37:11] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00295 [I 2022-10-11 08:37:11] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-11 08:37:11] ...optimizers/trijoint.py.151: total_norm: 15.655793 [I 2022-10-11 08:37:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 1980/2304 [I 2022-10-11 08:37:19] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:31 [I 2022-10-11 08:37:19] ...trap/engines/engine.py.229: process: 0.83693 | load: 0.00295 [I 2022-10-11 08:37:19] ...trap/engines/engine.py.230: loss: 0.15664 [I 2022-10-11 08:37:19] ...optimizers/trijoint.py.151: total_norm: 15.751726 [I 2022-10-11 08:37:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 1990/2304 [I 2022-10-11 08:37:28] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:22 [I 2022-10-11 08:37:28] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00295 [I 2022-10-11 08:37:28] ...trap/engines/engine.py.230: loss: 0.09620 [I 2022-10-11 08:37:28] ...optimizers/trijoint.py.151: total_norm: 13.008160 [I 2022-10-11 08:37:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 2000/2304 [I 2022-10-11 08:37:36] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:13 [I 2022-10-11 08:37:36] ...trap/engines/engine.py.229: process: 0.81911 | load: 0.00295 [I 2022-10-11 08:37:36] ...trap/engines/engine.py.230: loss: 0.15013 [I 2022-10-11 08:37:36] ...optimizers/trijoint.py.151: total_norm: 15.644355 [I 2022-10-11 08:37:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 2010/2304 [I 2022-10-11 08:37:44] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:05 [I 2022-10-11 08:37:44] ...trap/engines/engine.py.229: process: 0.83778 | load: 0.00294 [I 2022-10-11 08:37:44] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-11 08:37:44] ...optimizers/trijoint.py.151: total_norm: 11.490276 [I 2022-10-11 08:37:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 2020/2304 [I 2022-10-11 08:37:53] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:03:57 [I 2022-10-11 08:37:53] ...trap/engines/engine.py.229: process: 0.83855 | load: 0.00308 [I 2022-10-11 08:37:53] ...trap/engines/engine.py.230: loss: 0.10189 [I 2022-10-11 08:37:53] ...optimizers/trijoint.py.151: total_norm: 15.323185 [I 2022-10-11 08:38:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 2030/2304 [I 2022-10-11 08:38:01] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:47 [I 2022-10-11 08:38:01] ...trap/engines/engine.py.229: process: 0.81619 | load: 0.00299 [I 2022-10-11 08:38:01] ...trap/engines/engine.py.230: loss: 0.09273 [I 2022-10-11 08:38:01] ...optimizers/trijoint.py.151: total_norm: 10.978847 [I 2022-10-11 08:38:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 2040/2304 [I 2022-10-11 08:38:09] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:03:41 [I 2022-10-11 08:38:09] ...trap/engines/engine.py.229: process: 0.83506 | load: 0.00295 [I 2022-10-11 08:38:09] ...trap/engines/engine.py.230: loss: 0.08272 [I 2022-10-11 08:38:09] ...optimizers/trijoint.py.151: total_norm: 11.315362 [I 2022-10-11 08:38:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 2050/2304 [I 2022-10-11 08:38:18] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:32 [I 2022-10-11 08:38:18] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00296 [I 2022-10-11 08:38:18] ...trap/engines/engine.py.230: loss: 0.17169 [I 2022-10-11 08:38:18] ...optimizers/trijoint.py.151: total_norm: 14.412231 [I 2022-10-11 08:38:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 2060/2304 [I 2022-10-11 08:38:26] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:24 [I 2022-10-11 08:38:26] ...trap/engines/engine.py.229: process: 0.84193 | load: 0.00297 [I 2022-10-11 08:38:26] ...trap/engines/engine.py.230: loss: 0.09363 [I 2022-10-11 08:38:26] ...optimizers/trijoint.py.151: total_norm: 17.904032 [I 2022-10-11 08:38:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 2070/2304 [I 2022-10-11 08:38:35] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:15 [I 2022-10-11 08:38:35] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00296 [I 2022-10-11 08:38:35] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-11 08:38:35] ...optimizers/trijoint.py.151: total_norm: 18.852549 [I 2022-10-11 08:38:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 2080/2304 [I 2022-10-11 08:38:43] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:08 [I 2022-10-11 08:38:43] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00295 [I 2022-10-11 08:38:43] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-11 08:38:43] ...optimizers/trijoint.py.151: total_norm: 11.144270 [I 2022-10-11 08:38:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 2090/2304 [I 2022-10-11 08:38:51] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:02:58 [I 2022-10-11 08:38:51] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00298 [I 2022-10-11 08:38:51] ...trap/engines/engine.py.230: loss: 0.13551 [I 2022-10-11 08:38:51] ...optimizers/trijoint.py.151: total_norm: 14.480180 [I 2022-10-11 08:39:00] ...trap/engines/engine.py.224: train: epoch 98 | batch 2100/2304 [I 2022-10-11 08:39:00] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:51 [I 2022-10-11 08:39:00] ...trap/engines/engine.py.229: process: 0.85590 | load: 0.00294 [I 2022-10-11 08:39:00] ...trap/engines/engine.py.230: loss: 0.13097 [I 2022-10-11 08:39:00] ...optimizers/trijoint.py.151: total_norm: 13.358262 [I 2022-10-11 08:39:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 2110/2304 [I 2022-10-11 08:39:08] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:02:43 [I 2022-10-11 08:39:08] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00295 [I 2022-10-11 08:39:08] ...trap/engines/engine.py.230: loss: 0.08630 [I 2022-10-11 08:39:08] ...optimizers/trijoint.py.151: total_norm: 10.826777 [I 2022-10-11 08:39:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 2120/2304 [I 2022-10-11 08:39:17] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:02:35 [I 2022-10-11 08:39:17] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00298 [I 2022-10-11 08:39:17] ...trap/engines/engine.py.230: loss: 0.08801 [I 2022-10-11 08:39:17] ...optimizers/trijoint.py.151: total_norm: 14.015346 [I 2022-10-11 08:39:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 2130/2304 [I 2022-10-11 08:39:25] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:27 [I 2022-10-11 08:39:25] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00294 [I 2022-10-11 08:39:25] ...trap/engines/engine.py.230: loss: 0.08864 [I 2022-10-11 08:39:25] ...optimizers/trijoint.py.151: total_norm: 21.088062 [I 2022-10-11 08:39:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 2140/2304 [I 2022-10-11 08:39:33] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:17 [I 2022-10-11 08:39:33] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00296 [I 2022-10-11 08:39:33] ...trap/engines/engine.py.230: loss: 0.15326 [I 2022-10-11 08:39:33] ...optimizers/trijoint.py.151: total_norm: 18.476116 [I 2022-10-11 08:39:42] ...trap/engines/engine.py.224: train: epoch 98 | batch 2150/2304 [I 2022-10-11 08:39:42] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:08 [I 2022-10-11 08:39:42] ...trap/engines/engine.py.229: process: 0.84505 | load: 0.00295 [I 2022-10-11 08:39:42] ...trap/engines/engine.py.230: loss: 0.15107 [I 2022-10-11 08:39:42] ...optimizers/trijoint.py.151: total_norm: 18.121826 [I 2022-10-11 08:39:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 2160/2304 [I 2022-10-11 08:39:50] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:00 [I 2022-10-11 08:39:50] ...trap/engines/engine.py.229: process: 0.83553 | load: 0.00294 [I 2022-10-11 08:39:50] ...trap/engines/engine.py.230: loss: 0.08387 [I 2022-10-11 08:39:50] ...optimizers/trijoint.py.151: total_norm: 13.853561 [I 2022-10-11 08:39:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 2170/2304 [I 2022-10-11 08:39:59] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:01:51 [I 2022-10-11 08:39:59] ...trap/engines/engine.py.229: process: 0.82666 | load: 0.00304 [I 2022-10-11 08:39:59] ...trap/engines/engine.py.230: loss: 0.11796 [I 2022-10-11 08:39:59] ...optimizers/trijoint.py.151: total_norm: 12.893250 [I 2022-10-11 08:40:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 2180/2304 [I 2022-10-11 08:40:07] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:01:43 [I 2022-10-11 08:40:07] ...trap/engines/engine.py.229: process: 0.83610 | load: 0.00307 [I 2022-10-11 08:40:07] ...trap/engines/engine.py.230: loss: 0.15117 [I 2022-10-11 08:40:07] ...optimizers/trijoint.py.151: total_norm: 15.327628 [I 2022-10-11 08:40:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 2190/2304 [I 2022-10-11 08:40:15] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:34 [I 2022-10-11 08:40:15] ...trap/engines/engine.py.229: process: 0.83036 | load: 0.00296 [I 2022-10-11 08:40:15] ...trap/engines/engine.py.230: loss: 0.18023 [I 2022-10-11 08:40:15] ...optimizers/trijoint.py.151: total_norm: 13.119688 [I 2022-10-11 08:40:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 2200/2304 [I 2022-10-11 08:40:24] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:27 [I 2022-10-11 08:40:24] ...trap/engines/engine.py.229: process: 0.84746 | load: 0.00300 [I 2022-10-11 08:40:24] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-11 08:40:24] ...optimizers/trijoint.py.151: total_norm: 10.152930 [I 2022-10-11 08:40:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 2210/2304 [I 2022-10-11 08:40:32] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:18 [I 2022-10-11 08:40:32] ...trap/engines/engine.py.229: process: 0.82587 | load: 0.00292 [I 2022-10-11 08:40:32] ...trap/engines/engine.py.230: loss: 0.09083 [I 2022-10-11 08:40:32] ...optimizers/trijoint.py.151: total_norm: 12.507016 [I 2022-10-11 08:40:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 2220/2304 [I 2022-10-11 08:40:40] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:10 [I 2022-10-11 08:40:40] ...trap/engines/engine.py.229: process: 0.82496 | load: 0.00297 [I 2022-10-11 08:40:40] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-11 08:40:40] ...optimizers/trijoint.py.151: total_norm: 24.466116 [I 2022-10-11 08:40:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 2230/2304 [I 2022-10-11 08:40:49] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:01 [I 2022-10-11 08:40:49] ...trap/engines/engine.py.229: process: 0.83530 | load: 0.00297 [I 2022-10-11 08:40:49] ...trap/engines/engine.py.230: loss: 0.08386 [I 2022-10-11 08:40:49] ...optimizers/trijoint.py.151: total_norm: 18.412699 [I 2022-10-11 08:40:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 2240/2304 [I 2022-10-11 08:40:57] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:00:53 [I 2022-10-11 08:40:57] ...trap/engines/engine.py.229: process: 0.84272 | load: 0.00295 [I 2022-10-11 08:40:57] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-11 08:40:57] ...optimizers/trijoint.py.151: total_norm: 7.489093 [I 2022-10-11 08:41:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 2250/2304 [I 2022-10-11 08:41:05] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:00:44 [I 2022-10-11 08:41:05] ...trap/engines/engine.py.229: process: 0.81512 | load: 0.00295 [I 2022-10-11 08:41:05] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-11 08:41:05] ...optimizers/trijoint.py.151: total_norm: 16.959204 [I 2022-10-11 08:41:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 2260/2304 [I 2022-10-11 08:41:14] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:36 [I 2022-10-11 08:41:14] ...trap/engines/engine.py.229: process: 0.84256 | load: 0.00295 [I 2022-10-11 08:41:14] ...trap/engines/engine.py.230: loss: 0.08584 [I 2022-10-11 08:41:14] ...optimizers/trijoint.py.151: total_norm: 13.654436 [I 2022-10-11 08:41:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 2270/2304 [I 2022-10-11 08:41:22] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:28 [I 2022-10-11 08:41:22] ...trap/engines/engine.py.229: process: 0.83275 | load: 0.00299 [I 2022-10-11 08:41:22] ...trap/engines/engine.py.230: loss: 0.12377 [I 2022-10-11 08:41:22] ...optimizers/trijoint.py.151: total_norm: 13.577976 [I 2022-10-11 08:41:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 2280/2304 [I 2022-10-11 08:41:31] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:20 [I 2022-10-11 08:41:31] ...trap/engines/engine.py.229: process: 0.83667 | load: 0.00294 [I 2022-10-11 08:41:31] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-11 08:41:31] ...optimizers/trijoint.py.151: total_norm: 9.029900 [I 2022-10-11 08:41:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 2290/2304 [I 2022-10-11 08:41:39] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:11 [I 2022-10-11 08:41:39] ...trap/engines/engine.py.229: process: 0.82610 | load: 0.00295 [I 2022-10-11 08:41:39] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-11 08:41:39] ...optimizers/trijoint.py.151: total_norm: 13.110382 [I 2022-10-11 08:41:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 2300/2304 [I 2022-10-11 08:41:47] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:03 [I 2022-10-11 08:41:47] ...trap/engines/engine.py.229: process: 0.83096 | load: 0.00012 [I 2022-10-11 08:41:47] ...trap/engines/engine.py.230: loss: 0.11350 [I 2022-10-11 08:41:47] ...optimizers/trijoint.py.151: total_norm: 24.573999 [I 2022-10-11 08:41:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 2304/2304 [I 2022-10-11 08:41:51] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:00 [I 2022-10-11 08:41:51] ...trap/engines/engine.py.229: process: 0.82405 | load: 0.00015 [I 2022-10-11 08:41:51] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-11 08:41:51] ...optimizers/trijoint.py.151: total_norm: 15.368168 [S 2022-10-11 08:41:52] ...trap/engines/engine.py.129: train_epoch.epoch: 98 [S 2022-10-11 08:41:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.12445193149003962 [I 2022-10-11 08:41:52] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 08:41:53] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 98 [I 2022-10-11 08:41:53] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 08:41:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 08:41:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 08:41:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 08:41:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 08:41:59] ...trap/engines/engine.py.321: eval: epoch 98 | batch 0/510 [I 2022-10-11 08:41:59] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:54 [I 2022-10-11 08:41:59] ...trap/engines/engine.py.326: process: 5.63583 | load: 3.55193 [I 2022-10-11 08:42:05] ...trap/engines/engine.py.321: eval: epoch 98 | batch 10/510 [I 2022-10-11 08:42:05] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:34 [I 2022-10-11 08:42:05] ...trap/engines/engine.py.326: process: 0.69137 | load: 0.00019 [I 2022-10-11 08:42:11] ...trap/engines/engine.py.321: eval: epoch 98 | batch 20/510 [I 2022-10-11 08:42:11] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:25 [I 2022-10-11 08:42:11] ...trap/engines/engine.py.326: process: 0.58561 | load: 0.00018 [I 2022-10-11 08:42:17] ...trap/engines/engine.py.321: eval: epoch 98 | batch 30/510 [I 2022-10-11 08:42:17] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:32 [I 2022-10-11 08:42:17] ...trap/engines/engine.py.326: process: 0.68134 | load: 0.00018 [I 2022-10-11 08:42:25] ...trap/engines/engine.py.321: eval: epoch 98 | batch 40/510 [I 2022-10-11 08:42:25] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:05:49 [I 2022-10-11 08:42:25] ...trap/engines/engine.py.326: process: 0.66573 | load: 0.00017 [I 2022-10-11 08:42:30] ...trap/engines/engine.py.321: eval: epoch 98 | batch 50/510 [I 2022-10-11 08:42:30] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:04:21 [I 2022-10-11 08:42:30] ...trap/engines/engine.py.326: process: 0.57523 | load: 0.00018 [I 2022-10-11 08:42:36] ...trap/engines/engine.py.321: eval: epoch 98 | batch 60/510 [I 2022-10-11 08:42:36] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:12 [I 2022-10-11 08:42:36] ...trap/engines/engine.py.326: process: 0.58175 | load: 0.00018 [I 2022-10-11 08:42:42] ...trap/engines/engine.py.321: eval: epoch 98 | batch 70/510 [I 2022-10-11 08:42:42] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:04:09 [I 2022-10-11 08:42:42] ...trap/engines/engine.py.326: process: 0.59059 | load: 0.00018 [I 2022-10-11 08:42:47] ...trap/engines/engine.py.321: eval: epoch 98 | batch 80/510 [I 2022-10-11 08:42:47] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:01 [I 2022-10-11 08:42:47] ...trap/engines/engine.py.326: process: 0.54213 | load: 0.00018 [I 2022-10-11 08:42:52] ...trap/engines/engine.py.321: eval: epoch 98 | batch 90/510 [I 2022-10-11 08:42:52] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:26 [I 2022-10-11 08:42:52] ...trap/engines/engine.py.326: process: 0.50777 | load: 0.00018 [I 2022-10-11 08:42:57] ...trap/engines/engine.py.321: eval: epoch 98 | batch 100/510 [I 2022-10-11 08:42:57] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:07 [I 2022-10-11 08:42:57] ...trap/engines/engine.py.326: process: 0.32539 | load: 0.00020 [I 2022-10-11 08:43:00] ...trap/engines/engine.py.321: eval: epoch 98 | batch 110/510 [I 2022-10-11 08:43:00] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:17 [I 2022-10-11 08:43:00] ...trap/engines/engine.py.326: process: 0.33574 | load: 0.00020 [I 2022-10-11 08:43:03] ...trap/engines/engine.py.321: eval: epoch 98 | batch 120/510 [I 2022-10-11 08:43:03] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:08 [I 2022-10-11 08:43:03] ...trap/engines/engine.py.326: process: 0.32791 | load: 0.00020 [I 2022-10-11 08:43:06] ...trap/engines/engine.py.321: eval: epoch 98 | batch 130/510 [I 2022-10-11 08:43:06] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:04 [I 2022-10-11 08:43:06] ...trap/engines/engine.py.326: process: 0.33289 | load: 0.00019 [I 2022-10-11 08:43:10] ...trap/engines/engine.py.321: eval: epoch 98 | batch 140/510 [I 2022-10-11 08:43:10] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:02:00 [I 2022-10-11 08:43:10] ...trap/engines/engine.py.326: process: 0.32431 | load: 0.00020 [I 2022-10-11 08:43:13] ...trap/engines/engine.py.321: eval: epoch 98 | batch 150/510 [I 2022-10-11 08:43:13] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:58 [I 2022-10-11 08:43:13] ...trap/engines/engine.py.326: process: 0.32874 | load: 0.00020 [I 2022-10-11 08:43:16] ...trap/engines/engine.py.321: eval: epoch 98 | batch 160/510 [I 2022-10-11 08:43:16] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:55 [I 2022-10-11 08:43:16] ...trap/engines/engine.py.326: process: 0.32957 | load: 0.00020 [I 2022-10-11 08:43:20] ...trap/engines/engine.py.321: eval: epoch 98 | batch 170/510 [I 2022-10-11 08:43:20] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:51 [I 2022-10-11 08:43:20] ...trap/engines/engine.py.326: process: 0.33042 | load: 0.00021 [I 2022-10-11 08:43:23] ...trap/engines/engine.py.321: eval: epoch 98 | batch 180/510 [I 2022-10-11 08:43:23] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:47 [I 2022-10-11 08:43:23] ...trap/engines/engine.py.326: process: 0.32485 | load: 0.00020 [I 2022-10-11 08:43:26] ...trap/engines/engine.py.321: eval: epoch 98 | batch 190/510 [I 2022-10-11 08:43:26] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:45 [I 2022-10-11 08:43:26] ...trap/engines/engine.py.326: process: 0.33381 | load: 0.00020 [I 2022-10-11 08:43:29] ...trap/engines/engine.py.321: eval: epoch 98 | batch 200/510 [I 2022-10-11 08:43:29] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:42 [I 2022-10-11 08:43:29] ...trap/engines/engine.py.326: process: 0.33658 | load: 0.00020 [I 2022-10-11 08:43:33] ...trap/engines/engine.py.321: eval: epoch 98 | batch 210/510 [I 2022-10-11 08:43:33] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:40 [I 2022-10-11 08:43:33] ...trap/engines/engine.py.326: process: 0.34236 | load: 0.00018 [I 2022-10-11 08:43:36] ...trap/engines/engine.py.321: eval: epoch 98 | batch 220/510 [I 2022-10-11 08:43:36] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:35 [I 2022-10-11 08:43:36] ...trap/engines/engine.py.326: process: 0.32276 | load: 0.00019 [I 2022-10-11 08:43:39] ...trap/engines/engine.py.321: eval: epoch 98 | batch 230/510 [I 2022-10-11 08:43:39] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:32 [I 2022-10-11 08:43:39] ...trap/engines/engine.py.326: process: 0.33515 | load: 0.00021 [I 2022-10-11 08:43:43] ...trap/engines/engine.py.321: eval: epoch 98 | batch 240/510 [I 2022-10-11 08:43:43] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:28 [I 2022-10-11 08:43:43] ...trap/engines/engine.py.326: process: 0.32752 | load: 0.00020 [I 2022-10-11 08:43:46] ...trap/engines/engine.py.321: eval: epoch 98 | batch 250/510 [I 2022-10-11 08:43:46] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:24 [I 2022-10-11 08:43:46] ...trap/engines/engine.py.326: process: 0.32550 | load: 0.00020 [I 2022-10-11 08:43:49] ...trap/engines/engine.py.321: eval: epoch 98 | batch 260/510 [I 2022-10-11 08:43:49] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:23 [I 2022-10-11 08:43:49] ...trap/engines/engine.py.326: process: 0.33351 | load: 0.00019 [I 2022-10-11 08:43:53] ...trap/engines/engine.py.321: eval: epoch 98 | batch 270/510 [I 2022-10-11 08:43:53] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:19 [I 2022-10-11 08:43:53] ...trap/engines/engine.py.326: process: 0.32779 | load: 0.00021 [I 2022-10-11 08:43:56] ...trap/engines/engine.py.321: eval: epoch 98 | batch 280/510 [I 2022-10-11 08:43:56] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:15 [I 2022-10-11 08:43:56] ...trap/engines/engine.py.326: process: 0.32238 | load: 0.00020 [I 2022-10-11 08:43:59] ...trap/engines/engine.py.321: eval: epoch 98 | batch 290/510 [I 2022-10-11 08:43:59] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:12 [I 2022-10-11 08:43:59] ...trap/engines/engine.py.326: process: 0.33583 | load: 0.00020 [I 2022-10-11 08:44:02] ...trap/engines/engine.py.321: eval: epoch 98 | batch 300/510 [I 2022-10-11 08:44:02] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:09 [I 2022-10-11 08:44:02] ...trap/engines/engine.py.326: process: 0.33056 | load: 0.00019 [I 2022-10-11 08:44:06] ...trap/engines/engine.py.321: eval: epoch 98 | batch 310/510 [I 2022-10-11 08:44:06] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:06 [I 2022-10-11 08:44:06] ...trap/engines/engine.py.326: process: 0.32907 | load: 0.00021 [I 2022-10-11 08:44:09] ...trap/engines/engine.py.321: eval: epoch 98 | batch 320/510 [I 2022-10-11 08:44:09] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:02 [I 2022-10-11 08:44:09] ...trap/engines/engine.py.326: process: 0.32622 | load: 0.00020 [I 2022-10-11 08:44:12] ...trap/engines/engine.py.321: eval: epoch 98 | batch 330/510 [I 2022-10-11 08:44:12] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:59 [I 2022-10-11 08:44:12] ...trap/engines/engine.py.326: process: 0.34018 | load: 0.00021 [I 2022-10-11 08:44:16] ...trap/engines/engine.py.321: eval: epoch 98 | batch 340/510 [I 2022-10-11 08:44:16] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:56 [I 2022-10-11 08:44:16] ...trap/engines/engine.py.326: process: 0.32678 | load: 0.00019 [I 2022-10-11 08:44:19] ...trap/engines/engine.py.321: eval: epoch 98 | batch 350/510 [I 2022-10-11 08:44:19] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:53 [I 2022-10-11 08:44:19] ...trap/engines/engine.py.326: process: 0.34211 | load: 0.00020 [I 2022-10-11 08:44:22] ...trap/engines/engine.py.321: eval: epoch 98 | batch 360/510 [I 2022-10-11 08:44:22] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:49 [I 2022-10-11 08:44:22] ...trap/engines/engine.py.326: process: 0.33730 | load: 0.00020 [I 2022-10-11 08:44:26] ...trap/engines/engine.py.321: eval: epoch 98 | batch 370/510 [I 2022-10-11 08:44:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:46 [I 2022-10-11 08:44:26] ...trap/engines/engine.py.326: process: 0.32792 | load: 0.00021 [I 2022-10-11 08:44:29] ...trap/engines/engine.py.321: eval: epoch 98 | batch 380/510 [I 2022-10-11 08:44:29] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:42 [I 2022-10-11 08:44:29] ...trap/engines/engine.py.326: process: 0.32659 | load: 0.00019 [I 2022-10-11 08:44:32] ...trap/engines/engine.py.321: eval: epoch 98 | batch 390/510 [I 2022-10-11 08:44:32] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:39 [I 2022-10-11 08:44:32] ...trap/engines/engine.py.326: process: 0.33139 | load: 0.00020 [I 2022-10-11 08:44:36] ...trap/engines/engine.py.321: eval: epoch 98 | batch 400/510 [I 2022-10-11 08:44:36] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:36 [I 2022-10-11 08:44:36] ...trap/engines/engine.py.326: process: 0.32443 | load: 0.00019 [I 2022-10-11 08:44:39] ...trap/engines/engine.py.321: eval: epoch 98 | batch 410/510 [I 2022-10-11 08:44:39] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:32 [I 2022-10-11 08:44:39] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00019 [I 2022-10-11 08:44:42] ...trap/engines/engine.py.321: eval: epoch 98 | batch 420/510 [I 2022-10-11 08:44:42] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:29 [I 2022-10-11 08:44:42] ...trap/engines/engine.py.326: process: 0.33416 | load: 0.00020 [I 2022-10-11 08:44:45] ...trap/engines/engine.py.321: eval: epoch 98 | batch 430/510 [I 2022-10-11 08:44:45] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:26 [I 2022-10-11 08:44:45] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00019 [I 2022-10-11 08:44:49] ...trap/engines/engine.py.321: eval: epoch 98 | batch 440/510 [I 2022-10-11 08:44:49] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:23 [I 2022-10-11 08:44:49] ...trap/engines/engine.py.326: process: 0.32520 | load: 0.00019 [I 2022-10-11 08:44:52] ...trap/engines/engine.py.321: eval: epoch 98 | batch 450/510 [I 2022-10-11 08:44:52] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:19 [I 2022-10-11 08:44:52] ...trap/engines/engine.py.326: process: 0.33097 | load: 0.00020 [I 2022-10-11 08:44:55] ...trap/engines/engine.py.321: eval: epoch 98 | batch 460/510 [I 2022-10-11 08:44:55] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:16 [I 2022-10-11 08:44:55] ...trap/engines/engine.py.326: process: 0.32663 | load: 0.00020 [I 2022-10-11 08:44:59] ...trap/engines/engine.py.321: eval: epoch 98 | batch 470/510 [I 2022-10-11 08:44:59] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:13 [I 2022-10-11 08:44:59] ...trap/engines/engine.py.326: process: 0.32356 | load: 0.00020 [I 2022-10-11 08:45:02] ...trap/engines/engine.py.321: eval: epoch 98 | batch 480/510 [I 2022-10-11 08:45:02] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:09 [I 2022-10-11 08:45:02] ...trap/engines/engine.py.326: process: 0.32354 | load: 0.00020 [I 2022-10-11 08:45:05] ...trap/engines/engine.py.321: eval: epoch 98 | batch 490/510 [I 2022-10-11 08:45:05] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:06 [I 2022-10-11 08:45:05] ...trap/engines/engine.py.326: process: 0.33750 | load: 0.00020 [I 2022-10-11 08:45:08] ...trap/engines/engine.py.321: eval: epoch 98 | batch 500/510 [I 2022-10-11 08:45:08] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:03 [I 2022-10-11 08:45:08] ...trap/engines/engine.py.326: process: 0.33284 | load: 0.00020 [I 2022-10-11 08:45:12] ...trap/engines/engine.py.321: eval: epoch 98 | batch 510/510 [I 2022-10-11 08:45:12] ...trap/engines/engine.py.325: elapsed: 0:03:18 | left: 0:00:00 [I 2022-10-11 08:45:12] ...trap/engines/engine.py.326: process: 0.32681 | load: 0.00012 [S 2022-10-11 08:45:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 98 [I 2022-10-11 08:45:12] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737900 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905900 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934300 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735000 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906300 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008431 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006670 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007015 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007169 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006310 [S 2022-10-11 08:45:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006789 [I 2022-10-11 08:45:53] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 08:45:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 08:45:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 08:45:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 08:45:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 08:45:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 08:46:01] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 08:46:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 08:46:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 99 [I 2022-10-11 08:46:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 08:46:07] ...trap/engines/engine.py.224: train: epoch 99 | batch 0/2304 [I 2022-10-11 08:46:07] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:28 [I 2022-10-11 08:46:07] ...trap/engines/engine.py.229: process: 3.24170 | load: 2.37351 [I 2022-10-11 08:46:07] ...trap/engines/engine.py.230: loss: 0.10125 [I 2022-10-11 08:46:07] ...optimizers/trijoint.py.151: total_norm: 12.588766 [I 2022-10-11 08:46:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 10/2304 [I 2022-10-11 08:46:16] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:56 [I 2022-10-11 08:46:16] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00512 [I 2022-10-11 08:46:16] ...trap/engines/engine.py.230: loss: 0.08947 [I 2022-10-11 08:46:16] ...optimizers/trijoint.py.151: total_norm: 12.578707 [I 2022-10-11 08:46:24] ...trap/engines/engine.py.224: train: epoch 99 | batch 20/2304 [I 2022-10-11 08:46:24] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:48 [I 2022-10-11 08:46:24] ...trap/engines/engine.py.229: process: 0.82615 | load: 0.00408 [I 2022-10-11 08:46:24] ...trap/engines/engine.py.230: loss: 0.10161 [I 2022-10-11 08:46:24] ...optimizers/trijoint.py.151: total_norm: 15.348120 [I 2022-10-11 08:46:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 30/2304 [I 2022-10-11 08:46:33] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:37 [I 2022-10-11 08:46:33] ...trap/engines/engine.py.229: process: 0.82345 | load: 0.00419 [I 2022-10-11 08:46:33] ...trap/engines/engine.py.230: loss: 0.10741 [I 2022-10-11 08:46:33] ...optimizers/trijoint.py.151: total_norm: 12.139340 [I 2022-10-11 08:46:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 40/2304 [I 2022-10-11 08:46:41] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:23 [I 2022-10-11 08:46:41] ...trap/engines/engine.py.229: process: 0.83176 | load: 0.00321 [I 2022-10-11 08:46:41] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-11 08:46:41] ...optimizers/trijoint.py.151: total_norm: 20.632318 [I 2022-10-11 08:46:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 50/2304 [I 2022-10-11 08:46:49] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:40 [I 2022-10-11 08:46:49] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00326 [I 2022-10-11 08:46:49] ...trap/engines/engine.py.230: loss: 0.10396 [I 2022-10-11 08:46:49] ...optimizers/trijoint.py.151: total_norm: 10.272841 [I 2022-10-11 08:46:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 60/2304 [I 2022-10-11 08:46:58] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:24 [I 2022-10-11 08:46:58] ...trap/engines/engine.py.229: process: 0.84792 | load: 0.00372 [I 2022-10-11 08:46:58] ...trap/engines/engine.py.230: loss: 0.19315 [I 2022-10-11 08:46:58] ...optimizers/trijoint.py.151: total_norm: 16.719082 [I 2022-10-11 08:47:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 70/2304 [I 2022-10-11 08:47:06] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:10 [I 2022-10-11 08:47:06] ...trap/engines/engine.py.229: process: 0.82766 | load: 0.00319 [I 2022-10-11 08:47:06] ...trap/engines/engine.py.230: loss: 0.11982 [I 2022-10-11 08:47:06] ...optimizers/trijoint.py.151: total_norm: 13.974921 [I 2022-10-11 08:47:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 80/2304 [I 2022-10-11 08:47:15] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:10 [I 2022-10-11 08:47:15] ...trap/engines/engine.py.229: process: 0.83086 | load: 0.00313 [I 2022-10-11 08:47:15] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-11 08:47:15] ...optimizers/trijoint.py.151: total_norm: 12.374681 [I 2022-10-11 08:47:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 90/2304 [I 2022-10-11 08:47:23] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:54 [I 2022-10-11 08:47:23] ...trap/engines/engine.py.229: process: 0.82790 | load: 0.00335 [I 2022-10-11 08:47:23] ...trap/engines/engine.py.230: loss: 0.14636 [I 2022-10-11 08:47:23] ...optimizers/trijoint.py.151: total_norm: 17.047451 [I 2022-10-11 08:47:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 100/2304 [I 2022-10-11 08:47:31] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:58 [I 2022-10-11 08:47:31] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00329 [I 2022-10-11 08:47:31] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-11 08:47:31] ...optimizers/trijoint.py.151: total_norm: 12.829158 [I 2022-10-11 08:47:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 110/2304 [I 2022-10-11 08:47:40] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:29 [I 2022-10-11 08:47:40] ...trap/engines/engine.py.229: process: 0.82691 | load: 0.00314 [I 2022-10-11 08:47:40] ...trap/engines/engine.py.230: loss: 0.08900 [I 2022-10-11 08:47:40] ...optimizers/trijoint.py.151: total_norm: 25.715672 [I 2022-10-11 08:47:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 120/2304 [I 2022-10-11 08:47:48] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:30:29 [I 2022-10-11 08:47:48] ...trap/engines/engine.py.229: process: 0.83505 | load: 0.00305 [I 2022-10-11 08:47:48] ...trap/engines/engine.py.230: loss: 0.15714 [I 2022-10-11 08:47:48] ...optimizers/trijoint.py.151: total_norm: 27.471733 [I 2022-10-11 08:47:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 130/2304 [I 2022-10-11 08:47:56] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:11 [I 2022-10-11 08:47:56] ...trap/engines/engine.py.229: process: 0.82679 | load: 0.00328 [I 2022-10-11 08:47:56] ...trap/engines/engine.py.230: loss: 0.10311 [I 2022-10-11 08:47:56] ...optimizers/trijoint.py.151: total_norm: 9.587268 [I 2022-10-11 08:48:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 140/2304 [I 2022-10-11 08:48:05] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:29:56 [I 2022-10-11 08:48:05] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00325 [I 2022-10-11 08:48:05] ...trap/engines/engine.py.230: loss: 0.11341 [I 2022-10-11 08:48:05] ...optimizers/trijoint.py.151: total_norm: 62.365158 [I 2022-10-11 08:48:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 150/2304 [I 2022-10-11 08:48:13] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:29:56 [I 2022-10-11 08:48:13] ...trap/engines/engine.py.229: process: 0.83374 | load: 0.00293 [I 2022-10-11 08:48:13] ...trap/engines/engine.py.230: loss: 0.14925 [I 2022-10-11 08:48:13] ...optimizers/trijoint.py.151: total_norm: 20.003071 [I 2022-10-11 08:48:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 160/2304 [I 2022-10-11 08:48:22] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:30:04 [I 2022-10-11 08:48:22] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00293 [I 2022-10-11 08:48:22] ...trap/engines/engine.py.230: loss: 0.14132 [I 2022-10-11 08:48:22] ...optimizers/trijoint.py.151: total_norm: 91.665886 [I 2022-10-11 08:48:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 170/2304 [I 2022-10-11 08:48:30] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:29:55 [I 2022-10-11 08:48:30] ...trap/engines/engine.py.229: process: 0.83381 | load: 0.00304 [I 2022-10-11 08:48:30] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-11 08:48:30] ...optimizers/trijoint.py.151: total_norm: 43.865658 [I 2022-10-11 08:48:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 180/2304 [I 2022-10-11 08:48:39] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:44 [I 2022-10-11 08:48:39] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00305 [I 2022-10-11 08:48:39] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-11 08:48:39] ...optimizers/trijoint.py.151: total_norm: 13.843575 [I 2022-10-11 08:48:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 190/2304 [I 2022-10-11 08:48:47] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:34 [I 2022-10-11 08:48:47] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00294 [I 2022-10-11 08:48:47] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-11 08:48:47] ...optimizers/trijoint.py.151: total_norm: 17.944649 [I 2022-10-11 08:48:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 200/2304 [I 2022-10-11 08:48:56] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:27 [I 2022-10-11 08:48:56] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00303 [I 2022-10-11 08:48:56] ...trap/engines/engine.py.230: loss: 0.10546 [I 2022-10-11 08:48:56] ...optimizers/trijoint.py.151: total_norm: 17.988279 [I 2022-10-11 08:49:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 210/2304 [I 2022-10-11 08:49:04] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:20 [I 2022-10-11 08:49:04] ...trap/engines/engine.py.229: process: 0.86383 | load: 0.00305 [I 2022-10-11 08:49:04] ...trap/engines/engine.py.230: loss: 0.12366 [I 2022-10-11 08:49:04] ...optimizers/trijoint.py.151: total_norm: 10.446879 [I 2022-10-11 08:49:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 220/2304 [I 2022-10-11 08:49:12] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:11 [I 2022-10-11 08:49:12] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00307 [I 2022-10-11 08:49:12] ...trap/engines/engine.py.230: loss: 0.20765 [I 2022-10-11 08:49:12] ...optimizers/trijoint.py.151: total_norm: 17.712318 [I 2022-10-11 08:49:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 230/2304 [I 2022-10-11 08:49:21] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:28:59 [I 2022-10-11 08:49:21] ...trap/engines/engine.py.229: process: 0.83522 | load: 0.00307 [I 2022-10-11 08:49:21] ...trap/engines/engine.py.230: loss: 0.09268 [I 2022-10-11 08:49:21] ...optimizers/trijoint.py.151: total_norm: 13.702596 [I 2022-10-11 08:49:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 240/2304 [I 2022-10-11 08:49:29] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:52 [I 2022-10-11 08:49:29] ...trap/engines/engine.py.229: process: 0.82918 | load: 0.00295 [I 2022-10-11 08:49:29] ...trap/engines/engine.py.230: loss: 0.14296 [I 2022-10-11 08:49:29] ...optimizers/trijoint.py.151: total_norm: 11.330285 [I 2022-10-11 08:49:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 250/2304 [I 2022-10-11 08:49:38] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:28:46 [I 2022-10-11 08:49:38] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00290 [I 2022-10-11 08:49:38] ...trap/engines/engine.py.230: loss: 0.13893 [I 2022-10-11 08:49:38] ...optimizers/trijoint.py.151: total_norm: 20.610075 [I 2022-10-11 08:49:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 260/2304 [I 2022-10-11 08:49:46] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:28:53 [I 2022-10-11 08:49:46] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00294 [I 2022-10-11 08:49:46] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-11 08:49:46] ...optimizers/trijoint.py.151: total_norm: 20.236967 [I 2022-10-11 08:49:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 270/2304 [I 2022-10-11 08:49:55] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:38 [I 2022-10-11 08:49:55] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00294 [I 2022-10-11 08:49:55] ...trap/engines/engine.py.230: loss: 0.09168 [I 2022-10-11 08:49:55] ...optimizers/trijoint.py.151: total_norm: 8.200101 [I 2022-10-11 08:50:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 280/2304 [I 2022-10-11 08:50:03] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:28:36 [I 2022-10-11 08:50:03] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00304 [I 2022-10-11 08:50:03] ...trap/engines/engine.py.230: loss: 0.07902 [I 2022-10-11 08:50:03] ...optimizers/trijoint.py.151: total_norm: 8.059033 [I 2022-10-11 08:50:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 290/2304 [I 2022-10-11 08:50:12] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:20 [I 2022-10-11 08:50:12] ...trap/engines/engine.py.229: process: 0.84669 | load: 0.00291 [I 2022-10-11 08:50:12] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-11 08:50:12] ...optimizers/trijoint.py.151: total_norm: 24.957104 [I 2022-10-11 08:50:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 300/2304 [I 2022-10-11 08:50:20] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:54 [I 2022-10-11 08:50:20] ...trap/engines/engine.py.229: process: 0.83047 | load: 0.00293 [I 2022-10-11 08:50:20] ...trap/engines/engine.py.230: loss: 0.06984 [I 2022-10-11 08:50:20] ...optimizers/trijoint.py.151: total_norm: 13.023537 [I 2022-10-11 08:50:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 310/2304 [I 2022-10-11 08:50:28] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:03 [I 2022-10-11 08:50:28] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00294 [I 2022-10-11 08:50:28] ...trap/engines/engine.py.230: loss: 0.09258 [I 2022-10-11 08:50:28] ...optimizers/trijoint.py.151: total_norm: 10.834908 [I 2022-10-11 08:50:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 320/2304 [I 2022-10-11 08:50:37] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:28:02 [I 2022-10-11 08:50:37] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00292 [I 2022-10-11 08:50:37] ...trap/engines/engine.py.230: loss: 0.19223 [I 2022-10-11 08:50:37] ...optimizers/trijoint.py.151: total_norm: 11.609046 [I 2022-10-11 08:50:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 330/2304 [I 2022-10-11 08:50:45] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:47 [I 2022-10-11 08:50:45] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00293 [I 2022-10-11 08:50:45] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-11 08:50:45] ...optimizers/trijoint.py.151: total_norm: 10.394712 [I 2022-10-11 08:50:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 340/2304 [I 2022-10-11 08:50:54] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:34 [I 2022-10-11 08:50:54] ...trap/engines/engine.py.229: process: 0.83273 | load: 0.00293 [I 2022-10-11 08:50:54] ...trap/engines/engine.py.230: loss: 0.08831 [I 2022-10-11 08:50:54] ...optimizers/trijoint.py.151: total_norm: 17.000851 [I 2022-10-11 08:51:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 350/2304 [I 2022-10-11 08:51:02] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:26:56 [I 2022-10-11 08:51:02] ...trap/engines/engine.py.229: process: 0.82569 | load: 0.00308 [I 2022-10-11 08:51:02] ...trap/engines/engine.py.230: loss: 0.15490 [I 2022-10-11 08:51:02] ...optimizers/trijoint.py.151: total_norm: 22.617321 [I 2022-10-11 08:51:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 360/2304 [I 2022-10-11 08:51:10] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:05 [I 2022-10-11 08:51:10] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00292 [I 2022-10-11 08:51:10] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-11 08:51:10] ...optimizers/trijoint.py.151: total_norm: 18.933344 [I 2022-10-11 08:51:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 370/2304 [I 2022-10-11 08:51:19] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:52 [I 2022-10-11 08:51:19] ...trap/engines/engine.py.229: process: 0.87612 | load: 0.00398 [I 2022-10-11 08:51:19] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-11 08:51:19] ...optimizers/trijoint.py.151: total_norm: 14.556865 [I 2022-10-11 08:51:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 380/2304 [I 2022-10-11 08:51:28] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:41 [I 2022-10-11 08:51:28] ...trap/engines/engine.py.229: process: 0.86626 | load: 0.00374 [I 2022-10-11 08:51:28] ...trap/engines/engine.py.230: loss: 0.12432 [I 2022-10-11 08:51:28] ...optimizers/trijoint.py.151: total_norm: 9.120459 [I 2022-10-11 08:51:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 390/2304 [I 2022-10-11 08:51:36] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:33 [I 2022-10-11 08:51:36] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00372 [I 2022-10-11 08:51:36] ...trap/engines/engine.py.230: loss: 0.11115 [I 2022-10-11 08:51:36] ...optimizers/trijoint.py.151: total_norm: 13.101585 [I 2022-10-11 08:51:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 400/2304 [I 2022-10-11 08:51:45] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:34 [I 2022-10-11 08:51:45] ...trap/engines/engine.py.229: process: 0.86621 | load: 0.00379 [I 2022-10-11 08:51:45] ...trap/engines/engine.py.230: loss: 0.08943 [I 2022-10-11 08:51:45] ...optimizers/trijoint.py.151: total_norm: 19.607912 [I 2022-10-11 08:51:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 410/2304 [I 2022-10-11 08:51:54] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:18 [I 2022-10-11 08:51:54] ...trap/engines/engine.py.229: process: 0.86602 | load: 0.00371 [I 2022-10-11 08:51:54] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-11 08:51:54] ...optimizers/trijoint.py.151: total_norm: 25.974056 [I 2022-10-11 08:52:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 420/2304 [I 2022-10-11 08:52:02] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:27:11 [I 2022-10-11 08:52:02] ...trap/engines/engine.py.229: process: 0.86403 | load: 0.00372 [I 2022-10-11 08:52:02] ...trap/engines/engine.py.230: loss: 0.09654 [I 2022-10-11 08:52:02] ...optimizers/trijoint.py.151: total_norm: 12.426964 [I 2022-10-11 08:52:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 430/2304 [I 2022-10-11 08:52:11] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:55 [I 2022-10-11 08:52:11] ...trap/engines/engine.py.229: process: 0.86152 | load: 0.00372 [I 2022-10-11 08:52:11] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-11 08:52:11] ...optimizers/trijoint.py.151: total_norm: 20.794436 [I 2022-10-11 08:52:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 440/2304 [I 2022-10-11 08:52:20] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:27:06 [I 2022-10-11 08:52:20] ...trap/engines/engine.py.229: process: 0.87247 | load: 0.00370 [I 2022-10-11 08:52:20] ...trap/engines/engine.py.230: loss: 0.07885 [I 2022-10-11 08:52:20] ...optimizers/trijoint.py.151: total_norm: 15.473513 [I 2022-10-11 08:52:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 450/2304 [I 2022-10-11 08:52:28] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:25:54 [I 2022-10-11 08:52:28] ...trap/engines/engine.py.229: process: 0.84011 | load: 0.00294 [I 2022-10-11 08:52:28] ...trap/engines/engine.py.230: loss: 0.06668 [I 2022-10-11 08:52:28] ...optimizers/trijoint.py.151: total_norm: 12.330143 [I 2022-10-11 08:52:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 460/2304 [I 2022-10-11 08:52:36] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:25:39 [I 2022-10-11 08:52:36] ...trap/engines/engine.py.229: process: 0.82815 | load: 0.00294 [I 2022-10-11 08:52:36] ...trap/engines/engine.py.230: loss: 0.13722 [I 2022-10-11 08:52:36] ...optimizers/trijoint.py.151: total_norm: 16.846849 [I 2022-10-11 08:52:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 470/2304 [I 2022-10-11 08:52:45] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:32 [I 2022-10-11 08:52:45] ...trap/engines/engine.py.229: process: 0.81840 | load: 0.00295 [I 2022-10-11 08:52:45] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-11 08:52:45] ...optimizers/trijoint.py.151: total_norm: 23.208519 [I 2022-10-11 08:52:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 480/2304 [I 2022-10-11 08:52:53] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:22 [I 2022-10-11 08:52:53] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00293 [I 2022-10-11 08:52:53] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-11 08:52:53] ...optimizers/trijoint.py.151: total_norm: 9.408678 [I 2022-10-11 08:53:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 490/2304 [I 2022-10-11 08:53:02] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:19 [I 2022-10-11 08:53:02] ...trap/engines/engine.py.229: process: 0.83155 | load: 0.00294 [I 2022-10-11 08:53:02] ...trap/engines/engine.py.230: loss: 0.10131 [I 2022-10-11 08:53:02] ...optimizers/trijoint.py.151: total_norm: 21.623955 [I 2022-10-11 08:53:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 500/2304 [I 2022-10-11 08:53:10] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:08 [I 2022-10-11 08:53:10] ...trap/engines/engine.py.229: process: 0.84759 | load: 0.00293 [I 2022-10-11 08:53:10] ...trap/engines/engine.py.230: loss: 0.11022 [I 2022-10-11 08:53:10] ...optimizers/trijoint.py.151: total_norm: 12.759580 [I 2022-10-11 08:53:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 510/2304 [I 2022-10-11 08:53:18] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:25:07 [I 2022-10-11 08:53:18] ...trap/engines/engine.py.229: process: 0.83560 | load: 0.00294 [I 2022-10-11 08:53:18] ...trap/engines/engine.py.230: loss: 0.13029 [I 2022-10-11 08:53:18] ...optimizers/trijoint.py.151: total_norm: 14.740440 [I 2022-10-11 08:53:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 520/2304 [I 2022-10-11 08:53:27] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:24:51 [I 2022-10-11 08:53:27] ...trap/engines/engine.py.229: process: 0.82750 | load: 0.00308 [I 2022-10-11 08:53:27] ...trap/engines/engine.py.230: loss: 0.17831 [I 2022-10-11 08:53:27] ...optimizers/trijoint.py.151: total_norm: 16.596743 [I 2022-10-11 08:53:35] ...trap/engines/engine.py.224: train: epoch 99 | batch 530/2304 [I 2022-10-11 08:53:35] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:24:42 [I 2022-10-11 08:53:35] ...trap/engines/engine.py.229: process: 0.82996 | load: 0.00295 [I 2022-10-11 08:53:35] ...trap/engines/engine.py.230: loss: 0.10039 [I 2022-10-11 08:53:35] ...optimizers/trijoint.py.151: total_norm: 18.547115 [I 2022-10-11 08:53:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 540/2304 [I 2022-10-11 08:53:43] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:40 [I 2022-10-11 08:53:43] ...trap/engines/engine.py.229: process: 0.82855 | load: 0.00300 [I 2022-10-11 08:53:43] ...trap/engines/engine.py.230: loss: 0.12751 [I 2022-10-11 08:53:43] ...optimizers/trijoint.py.151: total_norm: 20.427319 [I 2022-10-11 08:53:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 550/2304 [I 2022-10-11 08:53:52] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:19 [I 2022-10-11 08:53:52] ...trap/engines/engine.py.229: process: 0.82103 | load: 0.00294 [I 2022-10-11 08:53:52] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-11 08:53:52] ...optimizers/trijoint.py.151: total_norm: 9.906202 [I 2022-10-11 08:54:00] ...trap/engines/engine.py.224: train: epoch 99 | batch 560/2304 [I 2022-10-11 08:54:00] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:08 [I 2022-10-11 08:54:00] ...trap/engines/engine.py.229: process: 0.83578 | load: 0.00305 [I 2022-10-11 08:54:00] ...trap/engines/engine.py.230: loss: 0.13029 [I 2022-10-11 08:54:00] ...optimizers/trijoint.py.151: total_norm: 23.704622 [I 2022-10-11 08:54:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 570/2304 [I 2022-10-11 08:54:08] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:08 [I 2022-10-11 08:54:08] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00293 [I 2022-10-11 08:54:08] ...trap/engines/engine.py.230: loss: 0.14789 [I 2022-10-11 08:54:08] ...optimizers/trijoint.py.151: total_norm: 15.698488 [I 2022-10-11 08:54:17] ...trap/engines/engine.py.224: train: epoch 99 | batch 580/2304 [I 2022-10-11 08:54:17] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:23:57 [I 2022-10-11 08:54:17] ...trap/engines/engine.py.229: process: 0.83115 | load: 0.00296 [I 2022-10-11 08:54:17] ...trap/engines/engine.py.230: loss: 0.16341 [I 2022-10-11 08:54:17] ...optimizers/trijoint.py.151: total_norm: 22.355040 [I 2022-10-11 08:54:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 590/2304 [I 2022-10-11 08:54:25] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:23:56 [I 2022-10-11 08:54:25] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00294 [I 2022-10-11 08:54:25] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-11 08:54:25] ...optimizers/trijoint.py.151: total_norm: 16.510633 [I 2022-10-11 08:54:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 600/2304 [I 2022-10-11 08:54:34] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:23:44 [I 2022-10-11 08:54:34] ...trap/engines/engine.py.229: process: 0.83882 | load: 0.00295 [I 2022-10-11 08:54:34] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-11 08:54:34] ...optimizers/trijoint.py.151: total_norm: 25.294205 [I 2022-10-11 08:54:42] ...trap/engines/engine.py.224: train: epoch 99 | batch 610/2304 [I 2022-10-11 08:54:42] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:37 [I 2022-10-11 08:54:42] ...trap/engines/engine.py.229: process: 0.83701 | load: 0.00294 [I 2022-10-11 08:54:42] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-11 08:54:42] ...optimizers/trijoint.py.151: total_norm: 83.839241 [I 2022-10-11 08:54:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 620/2304 [I 2022-10-11 08:54:50] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:35 [I 2022-10-11 08:54:50] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00293 [I 2022-10-11 08:54:50] ...trap/engines/engine.py.230: loss: 0.09804 [I 2022-10-11 08:54:50] ...optimizers/trijoint.py.151: total_norm: 13.640960 [I 2022-10-11 08:54:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 630/2304 [I 2022-10-11 08:54:59] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:21 [I 2022-10-11 08:54:59] ...trap/engines/engine.py.229: process: 0.83696 | load: 0.00294 [I 2022-10-11 08:54:59] ...trap/engines/engine.py.230: loss: 0.09950 [I 2022-10-11 08:54:59] ...optimizers/trijoint.py.151: total_norm: 10.715512 [I 2022-10-11 08:55:07] ...trap/engines/engine.py.224: train: epoch 99 | batch 640/2304 [I 2022-10-11 08:55:07] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:22:57 [I 2022-10-11 08:55:07] ...trap/engines/engine.py.229: process: 0.81983 | load: 0.00294 [I 2022-10-11 08:55:07] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-11 08:55:07] ...optimizers/trijoint.py.151: total_norm: 17.334517 [I 2022-10-11 08:55:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 650/2304 [I 2022-10-11 08:55:15] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:03 [I 2022-10-11 08:55:15] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00295 [I 2022-10-11 08:55:15] ...trap/engines/engine.py.230: loss: 0.13710 [I 2022-10-11 08:55:15] ...optimizers/trijoint.py.151: total_norm: 13.916337 [I 2022-10-11 08:55:24] ...trap/engines/engine.py.224: train: epoch 99 | batch 660/2304 [I 2022-10-11 08:55:24] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:22:50 [I 2022-10-11 08:55:24] ...trap/engines/engine.py.229: process: 0.83220 | load: 0.00295 [I 2022-10-11 08:55:24] ...trap/engines/engine.py.230: loss: 0.16228 [I 2022-10-11 08:55:24] ...optimizers/trijoint.py.151: total_norm: 15.444990 [I 2022-10-11 08:55:32] ...trap/engines/engine.py.224: train: epoch 99 | batch 670/2304 [I 2022-10-11 08:55:32] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:22:43 [I 2022-10-11 08:55:32] ...trap/engines/engine.py.229: process: 0.82646 | load: 0.00298 [I 2022-10-11 08:55:32] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-11 08:55:32] ...optimizers/trijoint.py.151: total_norm: 18.923666 [I 2022-10-11 08:55:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 680/2304 [I 2022-10-11 08:55:40] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:37 [I 2022-10-11 08:55:40] ...trap/engines/engine.py.229: process: 0.83675 | load: 0.00293 [I 2022-10-11 08:55:40] ...trap/engines/engine.py.230: loss: 0.08545 [I 2022-10-11 08:55:40] ...optimizers/trijoint.py.151: total_norm: 9.360898 [I 2022-10-11 08:55:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 690/2304 [I 2022-10-11 08:55:49] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:22:29 [I 2022-10-11 08:55:49] ...trap/engines/engine.py.229: process: 0.83520 | load: 0.00294 [I 2022-10-11 08:55:49] ...trap/engines/engine.py.230: loss: 0.21212 [I 2022-10-11 08:55:49] ...optimizers/trijoint.py.151: total_norm: 26.731064 [I 2022-10-11 08:55:57] ...trap/engines/engine.py.224: train: epoch 99 | batch 700/2304 [I 2022-10-11 08:55:57] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:25 [I 2022-10-11 08:55:57] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00293 [I 2022-10-11 08:55:57] ...trap/engines/engine.py.230: loss: 0.09556 [I 2022-10-11 08:55:57] ...optimizers/trijoint.py.151: total_norm: 14.877518 [I 2022-10-11 08:56:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 710/2304 [I 2022-10-11 08:56:06] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:45 [I 2022-10-11 08:56:06] ...trap/engines/engine.py.229: process: 0.86771 | load: 0.00371 [I 2022-10-11 08:56:06] ...trap/engines/engine.py.230: loss: 0.14695 [I 2022-10-11 08:56:06] ...optimizers/trijoint.py.151: total_norm: 19.289062 [I 2022-10-11 08:56:14] ...trap/engines/engine.py.224: train: epoch 99 | batch 720/2304 [I 2022-10-11 08:56:14] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:17 [I 2022-10-11 08:56:14] ...trap/engines/engine.py.229: process: 0.84637 | load: 0.00299 [I 2022-10-11 08:56:14] ...trap/engines/engine.py.230: loss: 0.10782 [I 2022-10-11 08:56:14] ...optimizers/trijoint.py.151: total_norm: 14.089211 [I 2022-10-11 08:56:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 730/2304 [I 2022-10-11 08:56:23] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:02 [I 2022-10-11 08:56:23] ...trap/engines/engine.py.229: process: 0.83052 | load: 0.00298 [I 2022-10-11 08:56:23] ...trap/engines/engine.py.230: loss: 0.24260 [I 2022-10-11 08:56:23] ...optimizers/trijoint.py.151: total_norm: 18.478136 [I 2022-10-11 08:56:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 740/2304 [I 2022-10-11 08:56:31] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:21:47 [I 2022-10-11 08:56:31] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00298 [I 2022-10-11 08:56:31] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-11 08:56:31] ...optimizers/trijoint.py.151: total_norm: 12.632508 [I 2022-10-11 08:56:39] ...trap/engines/engine.py.224: train: epoch 99 | batch 750/2304 [I 2022-10-11 08:56:39] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:51 [I 2022-10-11 08:56:39] ...trap/engines/engine.py.229: process: 0.84944 | load: 0.00293 [I 2022-10-11 08:56:39] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-11 08:56:39] ...optimizers/trijoint.py.151: total_norm: 16.440878 [I 2022-10-11 08:56:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 760/2304 [I 2022-10-11 08:56:48] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:21:34 [I 2022-10-11 08:56:48] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00298 [I 2022-10-11 08:56:48] ...trap/engines/engine.py.230: loss: 0.08872 [I 2022-10-11 08:56:48] ...optimizers/trijoint.py.151: total_norm: 17.904741 [I 2022-10-11 08:56:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 770/2304 [I 2022-10-11 08:56:56] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:31 [I 2022-10-11 08:56:56] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00294 [I 2022-10-11 08:56:56] ...trap/engines/engine.py.230: loss: 0.13137 [I 2022-10-11 08:56:56] ...optimizers/trijoint.py.151: total_norm: 10.530592 [I 2022-10-11 08:57:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 780/2304 [I 2022-10-11 08:57:05] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:12 [I 2022-10-11 08:57:05] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00295 [I 2022-10-11 08:57:05] ...trap/engines/engine.py.230: loss: 0.13711 [I 2022-10-11 08:57:05] ...optimizers/trijoint.py.151: total_norm: 13.112033 [I 2022-10-11 08:57:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 790/2304 [I 2022-10-11 08:57:13] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:10 [I 2022-10-11 08:57:13] ...trap/engines/engine.py.229: process: 0.83144 | load: 0.00292 [I 2022-10-11 08:57:13] ...trap/engines/engine.py.230: loss: 0.16275 [I 2022-10-11 08:57:13] ...optimizers/trijoint.py.151: total_norm: 23.111862 [I 2022-10-11 08:57:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 800/2304 [I 2022-10-11 08:57:21] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:20:54 [I 2022-10-11 08:57:21] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00295 [I 2022-10-11 08:57:21] ...trap/engines/engine.py.230: loss: 0.08436 [I 2022-10-11 08:57:21] ...optimizers/trijoint.py.151: total_norm: 9.132350 [I 2022-10-11 08:57:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 810/2304 [I 2022-10-11 08:57:30] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:20:48 [I 2022-10-11 08:57:30] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00294 [I 2022-10-11 08:57:30] ...trap/engines/engine.py.230: loss: 0.09547 [I 2022-10-11 08:57:30] ...optimizers/trijoint.py.151: total_norm: 12.389205 [I 2022-10-11 08:57:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 820/2304 [I 2022-10-11 08:57:38] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:20:38 [I 2022-10-11 08:57:38] ...trap/engines/engine.py.229: process: 0.83589 | load: 0.00296 [I 2022-10-11 08:57:38] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-11 08:57:38] ...optimizers/trijoint.py.151: total_norm: 8.643463 [I 2022-10-11 08:57:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 830/2304 [I 2022-10-11 08:57:47] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:20:37 [I 2022-10-11 08:57:47] ...trap/engines/engine.py.229: process: 0.83743 | load: 0.00296 [I 2022-10-11 08:57:47] ...trap/engines/engine.py.230: loss: 0.17393 [I 2022-10-11 08:57:47] ...optimizers/trijoint.py.151: total_norm: 12.997681 [I 2022-10-11 08:57:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 840/2304 [I 2022-10-11 08:57:55] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:16 [I 2022-10-11 08:57:55] ...trap/engines/engine.py.229: process: 0.83145 | load: 0.00297 [I 2022-10-11 08:57:55] ...trap/engines/engine.py.230: loss: 0.12425 [I 2022-10-11 08:57:55] ...optimizers/trijoint.py.151: total_norm: 10.008054 [I 2022-10-11 08:58:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 850/2304 [I 2022-10-11 08:58:03] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:20:21 [I 2022-10-11 08:58:03] ...trap/engines/engine.py.229: process: 0.85156 | load: 0.00295 [I 2022-10-11 08:58:03] ...trap/engines/engine.py.230: loss: 0.10370 [I 2022-10-11 08:58:03] ...optimizers/trijoint.py.151: total_norm: 12.678746 [I 2022-10-11 08:58:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 860/2304 [I 2022-10-11 08:58:12] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:07 [I 2022-10-11 08:58:12] ...trap/engines/engine.py.229: process: 0.83839 | load: 0.00296 [I 2022-10-11 08:58:12] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-11 08:58:12] ...optimizers/trijoint.py.151: total_norm: 15.251911 [I 2022-10-11 08:58:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 870/2304 [I 2022-10-11 08:58:20] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:01 [I 2022-10-11 08:58:20] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00294 [I 2022-10-11 08:58:20] ...trap/engines/engine.py.230: loss: 0.10083 [I 2022-10-11 08:58:20] ...optimizers/trijoint.py.151: total_norm: 18.383371 [I 2022-10-11 08:58:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 880/2304 [I 2022-10-11 08:58:28] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:19:53 [I 2022-10-11 08:58:28] ...trap/engines/engine.py.229: process: 0.84883 | load: 0.00295 [I 2022-10-11 08:58:28] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-11 08:58:28] ...optimizers/trijoint.py.151: total_norm: 12.557533 [I 2022-10-11 08:58:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 890/2304 [I 2022-10-11 08:58:37] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:19:47 [I 2022-10-11 08:58:37] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00293 [I 2022-10-11 08:58:37] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-11 08:58:37] ...optimizers/trijoint.py.151: total_norm: 28.243935 [I 2022-10-11 08:58:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 900/2304 [I 2022-10-11 08:58:45] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:15 [I 2022-10-11 08:58:45] ...trap/engines/engine.py.229: process: 0.86572 | load: 0.00372 [I 2022-10-11 08:58:45] ...trap/engines/engine.py.230: loss: 0.08063 [I 2022-10-11 08:58:45] ...optimizers/trijoint.py.151: total_norm: 14.426180 [I 2022-10-11 08:58:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 910/2304 [I 2022-10-11 08:58:54] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:59 [I 2022-10-11 08:58:54] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00371 [I 2022-10-11 08:58:54] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-11 08:58:54] ...optimizers/trijoint.py.151: total_norm: 11.369429 [I 2022-10-11 08:59:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 920/2304 [I 2022-10-11 08:59:03] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:54 [I 2022-10-11 08:59:03] ...trap/engines/engine.py.229: process: 0.86087 | load: 0.00372 [I 2022-10-11 08:59:03] ...trap/engines/engine.py.230: loss: 0.11335 [I 2022-10-11 08:59:03] ...optimizers/trijoint.py.151: total_norm: 15.082452 [I 2022-10-11 08:59:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 930/2304 [I 2022-10-11 08:59:11] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:48 [I 2022-10-11 08:59:11] ...trap/engines/engine.py.229: process: 0.86747 | load: 0.00373 [I 2022-10-11 08:59:11] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-11 08:59:11] ...optimizers/trijoint.py.151: total_norm: 10.600366 [I 2022-10-11 08:59:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 940/2304 [I 2022-10-11 08:59:20] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:36 [I 2022-10-11 08:59:20] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00372 [I 2022-10-11 08:59:20] ...trap/engines/engine.py.230: loss: 0.13728 [I 2022-10-11 08:59:20] ...optimizers/trijoint.py.151: total_norm: 23.729370 [I 2022-10-11 08:59:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 950/2304 [I 2022-10-11 08:59:29] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:32 [I 2022-10-11 08:59:29] ...trap/engines/engine.py.229: process: 0.86936 | load: 0.00380 [I 2022-10-11 08:59:29] ...trap/engines/engine.py.230: loss: 0.12507 [I 2022-10-11 08:59:29] ...optimizers/trijoint.py.151: total_norm: 41.492649 [I 2022-10-11 08:59:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 960/2304 [I 2022-10-11 08:59:37] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:19:10 [I 2022-10-11 08:59:37] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00372 [I 2022-10-11 08:59:37] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-11 08:59:37] ...optimizers/trijoint.py.151: total_norm: 21.052662 [I 2022-10-11 08:59:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 970/2304 [I 2022-10-11 08:59:46] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:14 [I 2022-10-11 08:59:46] ...trap/engines/engine.py.229: process: 0.85922 | load: 0.00371 [I 2022-10-11 08:59:46] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-11 08:59:46] ...optimizers/trijoint.py.151: total_norm: 16.510975 [I 2022-10-11 08:59:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 980/2304 [I 2022-10-11 08:59:55] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:19:07 [I 2022-10-11 08:59:55] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00373 [I 2022-10-11 08:59:55] ...trap/engines/engine.py.230: loss: 0.13979 [I 2022-10-11 08:59:55] ...optimizers/trijoint.py.151: total_norm: 12.716658 [I 2022-10-11 09:00:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 990/2304 [I 2022-10-11 09:00:03] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:33 [I 2022-10-11 09:00:03] ...trap/engines/engine.py.229: process: 0.82869 | load: 0.00297 [I 2022-10-11 09:00:03] ...trap/engines/engine.py.230: loss: 0.08530 [I 2022-10-11 09:00:03] ...optimizers/trijoint.py.151: total_norm: 7.891743 [I 2022-10-11 09:00:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 1000/2304 [I 2022-10-11 09:00:12] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:11 [I 2022-10-11 09:00:12] ...trap/engines/engine.py.229: process: 0.84660 | load: 0.00295 [I 2022-10-11 09:00:12] ...trap/engines/engine.py.230: loss: 0.15720 [I 2022-10-11 09:00:12] ...optimizers/trijoint.py.151: total_norm: 36.468933 [I 2022-10-11 09:00:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 1010/2304 [I 2022-10-11 09:00:20] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:01 [I 2022-10-11 09:00:20] ...trap/engines/engine.py.229: process: 0.84115 | load: 0.00294 [I 2022-10-11 09:00:20] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-11 09:00:20] ...optimizers/trijoint.py.151: total_norm: 17.385839 [I 2022-10-11 09:00:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 1020/2304 [I 2022-10-11 09:00:28] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:17:45 [I 2022-10-11 09:00:28] ...trap/engines/engine.py.229: process: 0.82386 | load: 0.00296 [I 2022-10-11 09:00:28] ...trap/engines/engine.py.230: loss: 0.12755 [I 2022-10-11 09:00:28] ...optimizers/trijoint.py.151: total_norm: 23.793674 [I 2022-10-11 09:00:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 1030/2304 [I 2022-10-11 09:00:37] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:17:46 [I 2022-10-11 09:00:37] ...trap/engines/engine.py.229: process: 0.83261 | load: 0.00293 [I 2022-10-11 09:00:37] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-11 09:00:37] ...optimizers/trijoint.py.151: total_norm: 10.841549 [I 2022-10-11 09:00:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 1040/2304 [I 2022-10-11 09:00:45] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:17:35 [I 2022-10-11 09:00:45] ...trap/engines/engine.py.229: process: 0.82726 | load: 0.00293 [I 2022-10-11 09:00:45] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-11 09:00:45] ...optimizers/trijoint.py.151: total_norm: 20.314693 [I 2022-10-11 09:00:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 1050/2304 [I 2022-10-11 09:00:53] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:23 [I 2022-10-11 09:00:53] ...trap/engines/engine.py.229: process: 0.82149 | load: 0.00297 [I 2022-10-11 09:00:53] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-11 09:00:53] ...optimizers/trijoint.py.151: total_norm: 19.847126 [I 2022-10-11 09:01:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 1060/2304 [I 2022-10-11 09:01:02] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:29 [I 2022-10-11 09:01:02] ...trap/engines/engine.py.229: process: 0.86670 | load: 0.00297 [I 2022-10-11 09:01:02] ...trap/engines/engine.py.230: loss: 0.15628 [I 2022-10-11 09:01:02] ...optimizers/trijoint.py.151: total_norm: 42.762619 [I 2022-10-11 09:01:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 1070/2304 [I 2022-10-11 09:01:10] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:12 [I 2022-10-11 09:01:10] ...trap/engines/engine.py.229: process: 0.84107 | load: 0.00298 [I 2022-10-11 09:01:10] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-11 09:01:10] ...optimizers/trijoint.py.151: total_norm: 24.717285 [I 2022-10-11 09:01:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 1080/2304 [I 2022-10-11 09:01:19] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:00 [I 2022-10-11 09:01:19] ...trap/engines/engine.py.229: process: 0.83704 | load: 0.00297 [I 2022-10-11 09:01:19] ...trap/engines/engine.py.230: loss: 0.09333 [I 2022-10-11 09:01:19] ...optimizers/trijoint.py.151: total_norm: 10.423089 [I 2022-10-11 09:01:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 1090/2304 [I 2022-10-11 09:01:27] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:16:56 [I 2022-10-11 09:01:27] ...trap/engines/engine.py.229: process: 0.83532 | load: 0.00296 [I 2022-10-11 09:01:27] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-11 09:01:27] ...optimizers/trijoint.py.151: total_norm: 11.484818 [I 2022-10-11 09:01:35] ...trap/engines/engine.py.224: train: epoch 99 | batch 1100/2304 [I 2022-10-11 09:01:35] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:16:49 [I 2022-10-11 09:01:35] ...trap/engines/engine.py.229: process: 0.84892 | load: 0.00296 [I 2022-10-11 09:01:35] ...trap/engines/engine.py.230: loss: 0.15658 [I 2022-10-11 09:01:35] ...optimizers/trijoint.py.151: total_norm: 19.411106 [I 2022-10-11 09:01:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 1110/2304 [I 2022-10-11 09:01:44] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:16:46 [I 2022-10-11 09:01:44] ...trap/engines/engine.py.229: process: 0.84834 | load: 0.00294 [I 2022-10-11 09:01:44] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-11 09:01:44] ...optimizers/trijoint.py.151: total_norm: 10.577569 [I 2022-10-11 09:01:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 1120/2304 [I 2022-10-11 09:01:52] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:42 [I 2022-10-11 09:01:52] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00296 [I 2022-10-11 09:01:52] ...trap/engines/engine.py.230: loss: 0.15038 [I 2022-10-11 09:01:52] ...optimizers/trijoint.py.151: total_norm: 19.607004 [I 2022-10-11 09:02:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 1130/2304 [I 2022-10-11 09:02:01] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:16:36 [I 2022-10-11 09:02:01] ...trap/engines/engine.py.229: process: 0.86059 | load: 0.00292 [I 2022-10-11 09:02:01] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-11 09:02:01] ...optimizers/trijoint.py.151: total_norm: 13.251691 [I 2022-10-11 09:02:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 1140/2304 [I 2022-10-11 09:02:09] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:16:17 [I 2022-10-11 09:02:09] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00297 [I 2022-10-11 09:02:09] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-11 09:02:09] ...optimizers/trijoint.py.151: total_norm: 12.961777 [I 2022-10-11 09:02:17] ...trap/engines/engine.py.224: train: epoch 99 | batch 1150/2304 [I 2022-10-11 09:02:17] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:16:01 [I 2022-10-11 09:02:17] ...trap/engines/engine.py.229: process: 0.83874 | load: 0.00306 [I 2022-10-11 09:02:17] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-11 09:02:17] ...optimizers/trijoint.py.151: total_norm: 16.240189 [I 2022-10-11 09:02:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 1160/2304 [I 2022-10-11 09:02:26] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:00 [I 2022-10-11 09:02:26] ...trap/engines/engine.py.229: process: 0.83789 | load: 0.00294 [I 2022-10-11 09:02:26] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-11 09:02:26] ...optimizers/trijoint.py.151: total_norm: 26.885777 [I 2022-10-11 09:02:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1170/2304 [I 2022-10-11 09:02:34] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:15:46 [I 2022-10-11 09:02:34] ...trap/engines/engine.py.229: process: 0.84295 | load: 0.00297 [I 2022-10-11 09:02:34] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-11 09:02:34] ...optimizers/trijoint.py.151: total_norm: 10.021702 [I 2022-10-11 09:02:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 1180/2304 [I 2022-10-11 09:02:43] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:15:37 [I 2022-10-11 09:02:43] ...trap/engines/engine.py.229: process: 0.81973 | load: 0.00296 [I 2022-10-11 09:02:43] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-11 09:02:43] ...optimizers/trijoint.py.151: total_norm: 22.884115 [I 2022-10-11 09:02:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 1190/2304 [I 2022-10-11 09:02:51] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:15:37 [I 2022-10-11 09:02:51] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00296 [I 2022-10-11 09:02:51] ...trap/engines/engine.py.230: loss: 0.13675 [I 2022-10-11 09:02:51] ...optimizers/trijoint.py.151: total_norm: 21.025309 [I 2022-10-11 09:02:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 1200/2304 [I 2022-10-11 09:02:59] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:15:27 [I 2022-10-11 09:02:59] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00294 [I 2022-10-11 09:02:59] ...trap/engines/engine.py.230: loss: 0.09010 [I 2022-10-11 09:02:59] ...optimizers/trijoint.py.151: total_norm: 24.543741 [I 2022-10-11 09:03:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 1210/2304 [I 2022-10-11 09:03:08] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:17 [I 2022-10-11 09:03:08] ...trap/engines/engine.py.229: process: 0.83588 | load: 0.00292 [I 2022-10-11 09:03:08] ...trap/engines/engine.py.230: loss: 0.11549 [I 2022-10-11 09:03:08] ...optimizers/trijoint.py.151: total_norm: 12.997518 [I 2022-10-11 09:03:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 1220/2304 [I 2022-10-11 09:03:16] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:04 [I 2022-10-11 09:03:16] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00295 [I 2022-10-11 09:03:16] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-11 09:03:16] ...optimizers/trijoint.py.151: total_norm: 12.471859 [I 2022-10-11 09:03:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 1230/2304 [I 2022-10-11 09:03:25] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:14:58 [I 2022-10-11 09:03:25] ...trap/engines/engine.py.229: process: 0.82864 | load: 0.00293 [I 2022-10-11 09:03:25] ...trap/engines/engine.py.230: loss: 0.14745 [I 2022-10-11 09:03:25] ...optimizers/trijoint.py.151: total_norm: 16.704250 [I 2022-10-11 09:03:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 1240/2304 [I 2022-10-11 09:03:33] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:14:49 [I 2022-10-11 09:03:33] ...trap/engines/engine.py.229: process: 0.84462 | load: 0.00294 [I 2022-10-11 09:03:33] ...trap/engines/engine.py.230: loss: 0.16560 [I 2022-10-11 09:03:33] ...optimizers/trijoint.py.151: total_norm: 16.095533 [I 2022-10-11 09:03:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 1250/2304 [I 2022-10-11 09:03:41] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:14:42 [I 2022-10-11 09:03:41] ...trap/engines/engine.py.229: process: 0.84167 | load: 0.00293 [I 2022-10-11 09:03:41] ...trap/engines/engine.py.230: loss: 0.16471 [I 2022-10-11 09:03:41] ...optimizers/trijoint.py.151: total_norm: 25.870901 [I 2022-10-11 09:03:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 1260/2304 [I 2022-10-11 09:03:50] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:14:27 [I 2022-10-11 09:03:50] ...trap/engines/engine.py.229: process: 0.82861 | load: 0.00295 [I 2022-10-11 09:03:50] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-11 09:03:50] ...optimizers/trijoint.py.151: total_norm: 14.652135 [I 2022-10-11 09:03:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 1270/2304 [I 2022-10-11 09:03:58] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:14:26 [I 2022-10-11 09:03:58] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00296 [I 2022-10-11 09:03:58] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-11 09:03:58] ...optimizers/trijoint.py.151: total_norm: 20.810520 [I 2022-10-11 09:04:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1280/2304 [I 2022-10-11 09:04:06] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:16 [I 2022-10-11 09:04:06] ...trap/engines/engine.py.229: process: 0.82933 | load: 0.00294 [I 2022-10-11 09:04:06] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-11 09:04:06] ...optimizers/trijoint.py.151: total_norm: 13.229457 [I 2022-10-11 09:04:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 1290/2304 [I 2022-10-11 09:04:15] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:14:12 [I 2022-10-11 09:04:15] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00295 [I 2022-10-11 09:04:15] ...trap/engines/engine.py.230: loss: 0.08916 [I 2022-10-11 09:04:15] ...optimizers/trijoint.py.151: total_norm: 10.886167 [I 2022-10-11 09:04:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 1300/2304 [I 2022-10-11 09:04:41] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:33:44 [I 2022-10-11 09:04:41] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00294 [I 2022-10-11 09:04:41] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-11 09:04:41] ...optimizers/trijoint.py.151: total_norm: 60.251289 [I 2022-10-11 09:04:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 1310/2304 [I 2022-10-11 09:04:49] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:15:54 [I 2022-10-11 09:04:49] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00296 [I 2022-10-11 09:04:49] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-11 09:04:49] ...optimizers/trijoint.py.151: total_norm: 16.005287 [I 2022-10-11 09:04:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 1320/2304 [I 2022-10-11 09:04:58] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:56 [I 2022-10-11 09:04:58] ...trap/engines/engine.py.229: process: 0.86066 | load: 0.00295 [I 2022-10-11 09:04:58] ...trap/engines/engine.py.230: loss: 0.13478 [I 2022-10-11 09:04:58] ...optimizers/trijoint.py.151: total_norm: 16.497700 [I 2022-10-11 09:05:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1330/2304 [I 2022-10-11 09:05:06] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:13:37 [I 2022-10-11 09:05:06] ...trap/engines/engine.py.229: process: 0.83623 | load: 0.00293 [I 2022-10-11 09:05:06] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-11 09:05:06] ...optimizers/trijoint.py.151: total_norm: 21.936666 [I 2022-10-11 09:05:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 1340/2304 [I 2022-10-11 09:05:15] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:25 [I 2022-10-11 09:05:15] ...trap/engines/engine.py.229: process: 0.82790 | load: 0.00295 [I 2022-10-11 09:05:15] ...trap/engines/engine.py.230: loss: 0.18284 [I 2022-10-11 09:05:15] ...optimizers/trijoint.py.151: total_norm: 44.518539 [I 2022-10-11 09:05:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 1350/2304 [I 2022-10-11 09:05:23] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:22 [I 2022-10-11 09:05:23] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00297 [I 2022-10-11 09:05:23] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-11 09:05:23] ...optimizers/trijoint.py.151: total_norm: 15.490337 [I 2022-10-11 09:05:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 1360/2304 [I 2022-10-11 09:05:31] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:12 [I 2022-10-11 09:05:31] ...trap/engines/engine.py.229: process: 0.83213 | load: 0.00296 [I 2022-10-11 09:05:31] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-11 09:05:31] ...optimizers/trijoint.py.151: total_norm: 13.567005 [I 2022-10-11 09:05:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 1370/2304 [I 2022-10-11 09:05:40] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:05 [I 2022-10-11 09:05:40] ...trap/engines/engine.py.229: process: 0.83782 | load: 0.00292 [I 2022-10-11 09:05:40] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-11 09:05:40] ...optimizers/trijoint.py.151: total_norm: 11.289655 [I 2022-10-11 09:05:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 1380/2304 [I 2022-10-11 09:05:48] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:12:55 [I 2022-10-11 09:05:48] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00296 [I 2022-10-11 09:05:48] ...trap/engines/engine.py.230: loss: 0.16402 [I 2022-10-11 09:05:48] ...optimizers/trijoint.py.151: total_norm: 21.430342 [I 2022-10-11 09:05:57] ...trap/engines/engine.py.224: train: epoch 99 | batch 1390/2304 [I 2022-10-11 09:05:57] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:12:44 [I 2022-10-11 09:05:57] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00294 [I 2022-10-11 09:05:57] ...trap/engines/engine.py.230: loss: 0.21958 [I 2022-10-11 09:05:57] ...optimizers/trijoint.py.151: total_norm: 30.651381 [I 2022-10-11 09:06:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 1400/2304 [I 2022-10-11 09:06:05] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:12:34 [I 2022-10-11 09:06:05] ...trap/engines/engine.py.229: process: 0.83336 | load: 0.00293 [I 2022-10-11 09:06:05] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-11 09:06:05] ...optimizers/trijoint.py.151: total_norm: 10.838565 [I 2022-10-11 09:06:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 1410/2304 [I 2022-10-11 09:06:13] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:35 [I 2022-10-11 09:06:13] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00296 [I 2022-10-11 09:06:13] ...trap/engines/engine.py.230: loss: 0.11008 [I 2022-10-11 09:06:13] ...optimizers/trijoint.py.151: total_norm: 17.941696 [I 2022-10-11 09:06:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 1420/2304 [I 2022-10-11 09:06:22] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:25 [I 2022-10-11 09:06:22] ...trap/engines/engine.py.229: process: 0.84329 | load: 0.00297 [I 2022-10-11 09:06:22] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-11 09:06:22] ...optimizers/trijoint.py.151: total_norm: 10.836887 [I 2022-10-11 09:06:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 1430/2304 [I 2022-10-11 09:06:30] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:15 [I 2022-10-11 09:06:30] ...trap/engines/engine.py.229: process: 0.85081 | load: 0.00290 [I 2022-10-11 09:06:30] ...trap/engines/engine.py.230: loss: 0.10130 [I 2022-10-11 09:06:30] ...optimizers/trijoint.py.151: total_norm: 11.381977 [I 2022-10-11 09:06:39] ...trap/engines/engine.py.224: train: epoch 99 | batch 1440/2304 [I 2022-10-11 09:06:39] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:12 [I 2022-10-11 09:06:39] ...trap/engines/engine.py.229: process: 0.86544 | load: 0.00373 [I 2022-10-11 09:06:39] ...trap/engines/engine.py.230: loss: 0.20238 [I 2022-10-11 09:06:39] ...optimizers/trijoint.py.151: total_norm: 16.644747 [I 2022-10-11 09:06:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 1450/2304 [I 2022-10-11 09:06:47] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:18 [I 2022-10-11 09:06:47] ...trap/engines/engine.py.229: process: 0.86990 | load: 0.00375 [I 2022-10-11 09:06:47] ...trap/engines/engine.py.230: loss: 0.09708 [I 2022-10-11 09:06:47] ...optimizers/trijoint.py.151: total_norm: 10.199183 [I 2022-10-11 09:06:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 1460/2304 [I 2022-10-11 09:06:56] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:11:53 [I 2022-10-11 09:06:56] ...trap/engines/engine.py.229: process: 0.84253 | load: 0.00291 [I 2022-10-11 09:06:56] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-11 09:06:56] ...optimizers/trijoint.py.151: total_norm: 23.536251 [I 2022-10-11 09:07:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 1470/2304 [I 2022-10-11 09:07:04] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:11:41 [I 2022-10-11 09:07:04] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00293 [I 2022-10-11 09:07:04] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-11 09:07:04] ...optimizers/trijoint.py.151: total_norm: 31.872780 [I 2022-10-11 09:07:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 1480/2304 [I 2022-10-11 09:07:13] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:34 [I 2022-10-11 09:07:13] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00294 [I 2022-10-11 09:07:13] ...trap/engines/engine.py.230: loss: 0.06540 [I 2022-10-11 09:07:13] ...optimizers/trijoint.py.151: total_norm: 18.192997 [I 2022-10-11 09:07:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 1490/2304 [I 2022-10-11 09:07:21] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:11:20 [I 2022-10-11 09:07:21] ...trap/engines/engine.py.229: process: 0.83118 | load: 0.00293 [I 2022-10-11 09:07:21] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-11 09:07:21] ...optimizers/trijoint.py.151: total_norm: 18.383204 [I 2022-10-11 09:07:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 1500/2304 [I 2022-10-11 09:07:30] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:12 [I 2022-10-11 09:07:30] ...trap/engines/engine.py.229: process: 0.83934 | load: 0.00304 [I 2022-10-11 09:07:30] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-11 09:07:30] ...optimizers/trijoint.py.151: total_norm: 13.411304 [I 2022-10-11 09:07:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 1510/2304 [I 2022-10-11 09:07:38] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:12 [I 2022-10-11 09:07:38] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00293 [I 2022-10-11 09:07:38] ...trap/engines/engine.py.230: loss: 0.13981 [I 2022-10-11 09:07:38] ...optimizers/trijoint.py.151: total_norm: 20.318027 [I 2022-10-11 09:07:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 1520/2304 [I 2022-10-11 09:07:46] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:10:56 [I 2022-10-11 09:07:46] ...trap/engines/engine.py.229: process: 0.83331 | load: 0.00294 [I 2022-10-11 09:07:46] ...trap/engines/engine.py.230: loss: 0.17368 [I 2022-10-11 09:07:46] ...optimizers/trijoint.py.151: total_norm: 15.588650 [I 2022-10-11 09:07:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 1530/2304 [I 2022-10-11 09:07:55] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:10:50 [I 2022-10-11 09:07:55] ...trap/engines/engine.py.229: process: 0.83672 | load: 0.00293 [I 2022-10-11 09:07:55] ...trap/engines/engine.py.230: loss: 0.14857 [I 2022-10-11 09:07:55] ...optimizers/trijoint.py.151: total_norm: 19.144146 [I 2022-10-11 09:08:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 1540/2304 [I 2022-10-11 09:08:03] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:10:51 [I 2022-10-11 09:08:03] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00294 [I 2022-10-11 09:08:03] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-11 09:08:03] ...optimizers/trijoint.py.151: total_norm: 17.647568 [I 2022-10-11 09:08:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 1550/2304 [I 2022-10-11 09:08:12] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:37 [I 2022-10-11 09:08:12] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00293 [I 2022-10-11 09:08:12] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-11 09:08:12] ...optimizers/trijoint.py.151: total_norm: 8.976834 [I 2022-10-11 09:08:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 1560/2304 [I 2022-10-11 09:08:20] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:10:30 [I 2022-10-11 09:08:20] ...trap/engines/engine.py.229: process: 0.84211 | load: 0.00293 [I 2022-10-11 09:08:20] ...trap/engines/engine.py.230: loss: 0.09067 [I 2022-10-11 09:08:20] ...optimizers/trijoint.py.151: total_norm: 22.934694 [I 2022-10-11 09:08:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 1570/2304 [I 2022-10-11 09:08:29] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:20 [I 2022-10-11 09:08:29] ...trap/engines/engine.py.229: process: 0.83162 | load: 0.00292 [I 2022-10-11 09:08:29] ...trap/engines/engine.py.230: loss: 0.12608 [I 2022-10-11 09:08:29] ...optimizers/trijoint.py.151: total_norm: 19.246538 [I 2022-10-11 09:08:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 1580/2304 [I 2022-10-11 09:08:37] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:11 [I 2022-10-11 09:08:37] ...trap/engines/engine.py.229: process: 0.82429 | load: 0.00295 [I 2022-10-11 09:08:37] ...trap/engines/engine.py.230: loss: 0.12889 [I 2022-10-11 09:08:37] ...optimizers/trijoint.py.151: total_norm: 11.889398 [I 2022-10-11 09:08:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 1590/2304 [I 2022-10-11 09:08:46] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:00 [I 2022-10-11 09:08:46] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00294 [I 2022-10-11 09:08:46] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-11 09:08:46] ...optimizers/trijoint.py.151: total_norm: 9.461473 [I 2022-10-11 09:08:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 1600/2304 [I 2022-10-11 09:08:54] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:09:50 [I 2022-10-11 09:08:54] ...trap/engines/engine.py.229: process: 0.83676 | load: 0.00294 [I 2022-10-11 09:08:54] ...trap/engines/engine.py.230: loss: 0.11377 [I 2022-10-11 09:08:54] ...optimizers/trijoint.py.151: total_norm: 16.590525 [I 2022-10-11 09:09:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 1610/2304 [I 2022-10-11 09:09:02] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:09:40 [I 2022-10-11 09:09:02] ...trap/engines/engine.py.229: process: 0.84046 | load: 0.00293 [I 2022-10-11 09:09:02] ...trap/engines/engine.py.230: loss: 0.18364 [I 2022-10-11 09:09:02] ...optimizers/trijoint.py.151: total_norm: 9.656598 [I 2022-10-11 09:09:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 1620/2304 [I 2022-10-11 09:09:11] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:32 [I 2022-10-11 09:09:11] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00293 [I 2022-10-11 09:09:11] ...trap/engines/engine.py.230: loss: 0.17190 [I 2022-10-11 09:09:11] ...optimizers/trijoint.py.151: total_norm: 16.827068 [I 2022-10-11 09:09:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 1630/2304 [I 2022-10-11 09:09:19] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:22 [I 2022-10-11 09:09:19] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00294 [I 2022-10-11 09:09:19] ...trap/engines/engine.py.230: loss: 0.10081 [I 2022-10-11 09:09:19] ...optimizers/trijoint.py.151: total_norm: 10.659828 [I 2022-10-11 09:09:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 1640/2304 [I 2022-10-11 09:09:28] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:15 [I 2022-10-11 09:09:28] ...trap/engines/engine.py.229: process: 0.84448 | load: 0.00296 [I 2022-10-11 09:09:28] ...trap/engines/engine.py.230: loss: 0.11559 [I 2022-10-11 09:09:28] ...optimizers/trijoint.py.151: total_norm: 15.339924 [I 2022-10-11 09:09:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 1650/2304 [I 2022-10-11 09:09:36] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:05 [I 2022-10-11 09:09:36] ...trap/engines/engine.py.229: process: 0.82453 | load: 0.00293 [I 2022-10-11 09:09:36] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-11 09:09:36] ...optimizers/trijoint.py.151: total_norm: 14.893540 [I 2022-10-11 09:09:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 1660/2304 [I 2022-10-11 09:09:44] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:08:55 [I 2022-10-11 09:09:44] ...trap/engines/engine.py.229: process: 0.83471 | load: 0.00294 [I 2022-10-11 09:09:44] ...trap/engines/engine.py.230: loss: 0.15065 [I 2022-10-11 09:09:44] ...optimizers/trijoint.py.151: total_norm: 31.421783 [I 2022-10-11 09:09:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 1670/2304 [I 2022-10-11 09:09:53] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:48 [I 2022-10-11 09:09:53] ...trap/engines/engine.py.229: process: 0.83597 | load: 0.00295 [I 2022-10-11 09:09:53] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-11 09:09:53] ...optimizers/trijoint.py.151: total_norm: 21.628553 [I 2022-10-11 09:10:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 1680/2304 [I 2022-10-11 09:10:01] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:39 [I 2022-10-11 09:10:01] ...trap/engines/engine.py.229: process: 0.83467 | load: 0.00294 [I 2022-10-11 09:10:01] ...trap/engines/engine.py.230: loss: 0.08860 [I 2022-10-11 09:10:01] ...optimizers/trijoint.py.151: total_norm: 8.035799 [I 2022-10-11 09:10:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 1690/2304 [I 2022-10-11 09:10:09] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:33 [I 2022-10-11 09:10:09] ...trap/engines/engine.py.229: process: 0.82255 | load: 0.00293 [I 2022-10-11 09:10:09] ...trap/engines/engine.py.230: loss: 0.12530 [I 2022-10-11 09:10:09] ...optimizers/trijoint.py.151: total_norm: 16.178415 [I 2022-10-11 09:10:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 1700/2304 [I 2022-10-11 09:10:18] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:21 [I 2022-10-11 09:10:18] ...trap/engines/engine.py.229: process: 0.84084 | load: 0.00294 [I 2022-10-11 09:10:18] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-11 09:10:18] ...optimizers/trijoint.py.151: total_norm: 22.363886 [I 2022-10-11 09:10:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 1710/2304 [I 2022-10-11 09:10:26] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:16 [I 2022-10-11 09:10:26] ...trap/engines/engine.py.229: process: 0.83861 | load: 0.00294 [I 2022-10-11 09:10:26] ...trap/engines/engine.py.230: loss: 0.09822 [I 2022-10-11 09:10:26] ...optimizers/trijoint.py.151: total_norm: 14.192524 [I 2022-10-11 09:10:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1720/2304 [I 2022-10-11 09:10:34] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:08 [I 2022-10-11 09:10:34] ...trap/engines/engine.py.229: process: 0.83623 | load: 0.00296 [I 2022-10-11 09:10:34] ...trap/engines/engine.py.230: loss: 0.25294 [I 2022-10-11 09:10:34] ...optimizers/trijoint.py.151: total_norm: 38.272858 [I 2022-10-11 09:10:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 1730/2304 [I 2022-10-11 09:10:43] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:01 [I 2022-10-11 09:10:43] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00295 [I 2022-10-11 09:10:43] ...trap/engines/engine.py.230: loss: 0.12704 [I 2022-10-11 09:10:43] ...optimizers/trijoint.py.151: total_norm: 23.956289 [I 2022-10-11 09:10:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 1740/2304 [I 2022-10-11 09:10:51] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:07:55 [I 2022-10-11 09:10:51] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00295 [I 2022-10-11 09:10:51] ...trap/engines/engine.py.230: loss: 0.12104 [I 2022-10-11 09:10:51] ...optimizers/trijoint.py.151: total_norm: 18.437922 [I 2022-10-11 09:11:00] ...trap/engines/engine.py.224: train: epoch 99 | batch 1750/2304 [I 2022-10-11 09:11:00] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:42 [I 2022-10-11 09:11:00] ...trap/engines/engine.py.229: process: 0.83034 | load: 0.00295 [I 2022-10-11 09:11:00] ...trap/engines/engine.py.230: loss: 0.15753 [I 2022-10-11 09:11:00] ...optimizers/trijoint.py.151: total_norm: 17.364960 [I 2022-10-11 09:11:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 1760/2304 [I 2022-10-11 09:11:08] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:07:56 [I 2022-10-11 09:11:08] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00293 [I 2022-10-11 09:11:08] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-11 09:11:08] ...optimizers/trijoint.py.151: total_norm: 10.304744 [I 2022-10-11 09:11:17] ...trap/engines/engine.py.224: train: epoch 99 | batch 1770/2304 [I 2022-10-11 09:11:17] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:30 [I 2022-10-11 09:11:17] ...trap/engines/engine.py.229: process: 0.82794 | load: 0.00293 [I 2022-10-11 09:11:17] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-11 09:11:17] ...optimizers/trijoint.py.151: total_norm: 22.314804 [I 2022-10-11 09:11:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 1780/2304 [I 2022-10-11 09:11:25] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:16 [I 2022-10-11 09:11:25] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00293 [I 2022-10-11 09:11:25] ...trap/engines/engine.py.230: loss: 0.08414 [I 2022-10-11 09:11:25] ...optimizers/trijoint.py.151: total_norm: 17.071688 [I 2022-10-11 09:11:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1790/2304 [I 2022-10-11 09:11:34] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:10 [I 2022-10-11 09:11:34] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00293 [I 2022-10-11 09:11:34] ...trap/engines/engine.py.230: loss: 0.09429 [I 2022-10-11 09:11:34] ...optimizers/trijoint.py.151: total_norm: 10.242883 [I 2022-10-11 09:11:42] ...trap/engines/engine.py.224: train: epoch 99 | batch 1800/2304 [I 2022-10-11 09:11:42] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:01 [I 2022-10-11 09:11:42] ...trap/engines/engine.py.229: process: 0.83721 | load: 0.00294 [I 2022-10-11 09:11:42] ...trap/engines/engine.py.230: loss: 0.11291 [I 2022-10-11 09:11:42] ...optimizers/trijoint.py.151: total_norm: 21.575367 [I 2022-10-11 09:11:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 1810/2304 [I 2022-10-11 09:11:50] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:06:50 [I 2022-10-11 09:11:50] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00295 [I 2022-10-11 09:11:50] ...trap/engines/engine.py.230: loss: 0.08417 [I 2022-10-11 09:11:50] ...optimizers/trijoint.py.151: total_norm: 10.828962 [I 2022-10-11 09:11:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 1820/2304 [I 2022-10-11 09:11:59] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:48 [I 2022-10-11 09:11:59] ...trap/engines/engine.py.229: process: 0.85507 | load: 0.00293 [I 2022-10-11 09:11:59] ...trap/engines/engine.py.230: loss: 0.10711 [I 2022-10-11 09:11:59] ...optimizers/trijoint.py.151: total_norm: 11.763965 [I 2022-10-11 09:12:07] ...trap/engines/engine.py.224: train: epoch 99 | batch 1830/2304 [I 2022-10-11 09:12:07] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:39 [I 2022-10-11 09:12:07] ...trap/engines/engine.py.229: process: 0.83592 | load: 0.00294 [I 2022-10-11 09:12:07] ...trap/engines/engine.py.230: loss: 0.08971 [I 2022-10-11 09:12:07] ...optimizers/trijoint.py.151: total_norm: 25.088455 [I 2022-10-11 09:12:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 1840/2304 [I 2022-10-11 09:12:16] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:30 [I 2022-10-11 09:12:16] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00293 [I 2022-10-11 09:12:16] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-11 09:12:16] ...optimizers/trijoint.py.151: total_norm: 9.508084 [I 2022-10-11 09:12:24] ...trap/engines/engine.py.224: train: epoch 99 | batch 1850/2304 [I 2022-10-11 09:12:24] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:19 [I 2022-10-11 09:12:24] ...trap/engines/engine.py.229: process: 0.83941 | load: 0.00295 [I 2022-10-11 09:12:24] ...trap/engines/engine.py.230: loss: 0.13094 [I 2022-10-11 09:12:24] ...optimizers/trijoint.py.151: total_norm: 16.979576 [I 2022-10-11 09:12:32] ...trap/engines/engine.py.224: train: epoch 99 | batch 1860/2304 [I 2022-10-11 09:12:32] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:10 [I 2022-10-11 09:12:32] ...trap/engines/engine.py.229: process: 0.83034 | load: 0.00299 [I 2022-10-11 09:12:32] ...trap/engines/engine.py.230: loss: 0.16670 [I 2022-10-11 09:12:32] ...optimizers/trijoint.py.151: total_norm: 9.072829 [I 2022-10-11 09:12:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 1870/2304 [I 2022-10-11 09:12:41] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:02 [I 2022-10-11 09:12:41] ...trap/engines/engine.py.229: process: 0.81813 | load: 0.00295 [I 2022-10-11 09:12:41] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-11 09:12:41] ...optimizers/trijoint.py.151: total_norm: 11.093316 [I 2022-10-11 09:12:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 1880/2304 [I 2022-10-11 09:12:49] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:05:56 [I 2022-10-11 09:12:49] ...trap/engines/engine.py.229: process: 0.84032 | load: 0.00294 [I 2022-10-11 09:12:49] ...trap/engines/engine.py.230: loss: 0.13704 [I 2022-10-11 09:12:49] ...optimizers/trijoint.py.151: total_norm: 19.237347 [I 2022-10-11 09:12:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 1890/2304 [I 2022-10-11 09:12:58] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:05:48 [I 2022-10-11 09:12:58] ...trap/engines/engine.py.229: process: 0.83526 | load: 0.00295 [I 2022-10-11 09:12:58] ...trap/engines/engine.py.230: loss: 0.18160 [I 2022-10-11 09:12:58] ...optimizers/trijoint.py.151: total_norm: 16.075283 [I 2022-10-11 09:13:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1900/2304 [I 2022-10-11 09:13:06] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:38 [I 2022-10-11 09:13:06] ...trap/engines/engine.py.229: process: 0.83107 | load: 0.00298 [I 2022-10-11 09:13:06] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-11 09:13:06] ...optimizers/trijoint.py.151: total_norm: 12.860319 [I 2022-10-11 09:13:14] ...trap/engines/engine.py.224: train: epoch 99 | batch 1910/2304 [I 2022-10-11 09:13:14] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:29 [I 2022-10-11 09:13:14] ...trap/engines/engine.py.229: process: 0.82743 | load: 0.00297 [I 2022-10-11 09:13:14] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-11 09:13:14] ...optimizers/trijoint.py.151: total_norm: 8.974067 [I 2022-10-11 09:13:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 1920/2304 [I 2022-10-11 09:13:23] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:21 [I 2022-10-11 09:13:23] ...trap/engines/engine.py.229: process: 0.83654 | load: 0.00297 [I 2022-10-11 09:13:23] ...trap/engines/engine.py.230: loss: 0.11434 [I 2022-10-11 09:13:23] ...optimizers/trijoint.py.151: total_norm: 17.121294 [I 2022-10-11 09:13:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 1930/2304 [I 2022-10-11 09:13:31] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:13 [I 2022-10-11 09:13:31] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00292 [I 2022-10-11 09:13:31] ...trap/engines/engine.py.230: loss: 0.10735 [I 2022-10-11 09:13:31] ...optimizers/trijoint.py.151: total_norm: 15.965187 [I 2022-10-11 09:13:39] ...trap/engines/engine.py.224: train: epoch 99 | batch 1940/2304 [I 2022-10-11 09:13:39] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:04 [I 2022-10-11 09:13:39] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00294 [I 2022-10-11 09:13:39] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-11 09:13:39] ...optimizers/trijoint.py.151: total_norm: 14.433125 [I 2022-10-11 09:13:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 1950/2304 [I 2022-10-11 09:13:48] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:04:56 [I 2022-10-11 09:13:48] ...trap/engines/engine.py.229: process: 0.83360 | load: 0.00293 [I 2022-10-11 09:13:48] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-11 09:13:48] ...optimizers/trijoint.py.151: total_norm: 13.979463 [I 2022-10-11 09:13:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 1960/2304 [I 2022-10-11 09:13:56] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:48 [I 2022-10-11 09:13:56] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00292 [I 2022-10-11 09:13:56] ...trap/engines/engine.py.230: loss: 0.16015 [I 2022-10-11 09:13:56] ...optimizers/trijoint.py.151: total_norm: 8.022505 [I 2022-10-11 09:14:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 1970/2304 [I 2022-10-11 09:14:05] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:04:41 [I 2022-10-11 09:14:05] ...trap/engines/engine.py.229: process: 0.84153 | load: 0.00292 [I 2022-10-11 09:14:05] ...trap/engines/engine.py.230: loss: 0.14100 [I 2022-10-11 09:14:05] ...optimizers/trijoint.py.151: total_norm: 18.071182 [I 2022-10-11 09:14:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 1980/2304 [I 2022-10-11 09:14:13] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:39 [I 2022-10-11 09:14:13] ...trap/engines/engine.py.229: process: 0.90439 | load: 0.00374 [I 2022-10-11 09:14:13] ...trap/engines/engine.py.230: loss: 0.11307 [I 2022-10-11 09:14:13] ...optimizers/trijoint.py.151: total_norm: 8.416075 [I 2022-10-11 09:14:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 1990/2304 [I 2022-10-11 09:14:22] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:33 [I 2022-10-11 09:14:22] ...trap/engines/engine.py.229: process: 0.88741 | load: 0.00378 [I 2022-10-11 09:14:22] ...trap/engines/engine.py.230: loss: 0.10496 [I 2022-10-11 09:14:22] ...optimizers/trijoint.py.151: total_norm: 9.363534 [I 2022-10-11 09:14:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 2000/2304 [I 2022-10-11 09:14:31] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:23 [I 2022-10-11 09:14:31] ...trap/engines/engine.py.229: process: 0.86175 | load: 0.00372 [I 2022-10-11 09:14:31] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-11 09:14:31] ...optimizers/trijoint.py.151: total_norm: 10.657446 [I 2022-10-11 09:14:39] ...trap/engines/engine.py.224: train: epoch 99 | batch 2010/2304 [I 2022-10-11 09:14:39] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:16 [I 2022-10-11 09:14:39] ...trap/engines/engine.py.229: process: 0.87705 | load: 0.00375 [I 2022-10-11 09:14:39] ...trap/engines/engine.py.230: loss: 0.21111 [I 2022-10-11 09:14:39] ...optimizers/trijoint.py.151: total_norm: 58.924412 [I 2022-10-11 09:14:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 2020/2304 [I 2022-10-11 09:14:48] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:00 [I 2022-10-11 09:14:48] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00293 [I 2022-10-11 09:14:48] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-11 09:14:48] ...optimizers/trijoint.py.151: total_norm: 36.334183 [I 2022-10-11 09:14:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 2030/2304 [I 2022-10-11 09:14:56] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:49 [I 2022-10-11 09:14:56] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00294 [I 2022-10-11 09:14:56] ...trap/engines/engine.py.230: loss: 0.07147 [I 2022-10-11 09:14:56] ...optimizers/trijoint.py.151: total_norm: 9.159707 [I 2022-10-11 09:15:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 2040/2304 [I 2022-10-11 09:15:05] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:40 [I 2022-10-11 09:15:05] ...trap/engines/engine.py.229: process: 0.83344 | load: 0.00295 [I 2022-10-11 09:15:05] ...trap/engines/engine.py.230: loss: 0.09194 [I 2022-10-11 09:15:05] ...optimizers/trijoint.py.151: total_norm: 8.961857 [I 2022-10-11 09:15:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 2050/2304 [I 2022-10-11 09:15:13] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:30 [I 2022-10-11 09:15:13] ...trap/engines/engine.py.229: process: 0.81754 | load: 0.00295 [I 2022-10-11 09:15:13] ...trap/engines/engine.py.230: loss: 0.08208 [I 2022-10-11 09:15:13] ...optimizers/trijoint.py.151: total_norm: 16.224285 [I 2022-10-11 09:15:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 2060/2304 [I 2022-10-11 09:15:21] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:24 [I 2022-10-11 09:15:21] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00295 [I 2022-10-11 09:15:21] ...trap/engines/engine.py.230: loss: 0.10430 [I 2022-10-11 09:15:21] ...optimizers/trijoint.py.151: total_norm: 13.196086 [I 2022-10-11 09:15:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 2070/2304 [I 2022-10-11 09:15:30] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:14 [I 2022-10-11 09:15:30] ...trap/engines/engine.py.229: process: 0.81921 | load: 0.00296 [I 2022-10-11 09:15:30] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-11 09:15:30] ...optimizers/trijoint.py.151: total_norm: 22.703274 [I 2022-10-11 09:15:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 2080/2304 [I 2022-10-11 09:15:38] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:07 [I 2022-10-11 09:15:38] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00294 [I 2022-10-11 09:15:38] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-11 09:15:38] ...optimizers/trijoint.py.151: total_norm: 11.353301 [I 2022-10-11 09:15:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 2090/2304 [I 2022-10-11 09:15:46] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:02:58 [I 2022-10-11 09:15:46] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00295 [I 2022-10-11 09:15:46] ...trap/engines/engine.py.230: loss: 0.13111 [I 2022-10-11 09:15:46] ...optimizers/trijoint.py.151: total_norm: 10.898652 [I 2022-10-11 09:15:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 2100/2304 [I 2022-10-11 09:15:55] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:02:50 [I 2022-10-11 09:15:55] ...trap/engines/engine.py.229: process: 0.82326 | load: 0.00292 [I 2022-10-11 09:15:55] ...trap/engines/engine.py.230: loss: 0.14026 [I 2022-10-11 09:15:55] ...optimizers/trijoint.py.151: total_norm: 22.578114 [I 2022-10-11 09:16:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 2110/2304 [I 2022-10-11 09:16:03] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:42 [I 2022-10-11 09:16:03] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00292 [I 2022-10-11 09:16:03] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-11 09:16:03] ...optimizers/trijoint.py.151: total_norm: 11.980536 [I 2022-10-11 09:16:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 2120/2304 [I 2022-10-11 09:16:12] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:35 [I 2022-10-11 09:16:12] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00294 [I 2022-10-11 09:16:12] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-11 09:16:12] ...optimizers/trijoint.py.151: total_norm: 16.394836 [I 2022-10-11 09:16:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 2130/2304 [I 2022-10-11 09:16:20] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:25 [I 2022-10-11 09:16:20] ...trap/engines/engine.py.229: process: 0.82397 | load: 0.00295 [I 2022-10-11 09:16:20] ...trap/engines/engine.py.230: loss: 0.10832 [I 2022-10-11 09:16:20] ...optimizers/trijoint.py.151: total_norm: 9.485771 [I 2022-10-11 09:16:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 2140/2304 [I 2022-10-11 09:16:28] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:16 [I 2022-10-11 09:16:28] ...trap/engines/engine.py.229: process: 0.82722 | load: 0.00294 [I 2022-10-11 09:16:28] ...trap/engines/engine.py.230: loss: 0.07158 [I 2022-10-11 09:16:28] ...optimizers/trijoint.py.151: total_norm: 11.709404 [I 2022-10-11 09:16:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 2150/2304 [I 2022-10-11 09:16:37] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:02:08 [I 2022-10-11 09:16:37] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00294 [I 2022-10-11 09:16:37] ...trap/engines/engine.py.230: loss: 0.16417 [I 2022-10-11 09:16:37] ...optimizers/trijoint.py.151: total_norm: 44.084789 [I 2022-10-11 09:16:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 2160/2304 [I 2022-10-11 09:16:45] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:00 [I 2022-10-11 09:16:45] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00296 [I 2022-10-11 09:16:45] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-11 09:16:45] ...optimizers/trijoint.py.151: total_norm: 24.119091 [I 2022-10-11 09:16:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 2170/2304 [I 2022-10-11 09:16:54] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:01:52 [I 2022-10-11 09:16:54] ...trap/engines/engine.py.229: process: 0.82443 | load: 0.00294 [I 2022-10-11 09:16:54] ...trap/engines/engine.py.230: loss: 0.11850 [I 2022-10-11 09:16:54] ...optimizers/trijoint.py.151: total_norm: 24.062172 [I 2022-10-11 09:17:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 2180/2304 [I 2022-10-11 09:17:02] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:01:44 [I 2022-10-11 09:17:02] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00295 [I 2022-10-11 09:17:02] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-11 09:17:02] ...optimizers/trijoint.py.151: total_norm: 14.279575 [I 2022-10-11 09:17:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 2190/2304 [I 2022-10-11 09:17:11] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:01:36 [I 2022-10-11 09:17:11] ...trap/engines/engine.py.229: process: 0.83087 | load: 0.00295 [I 2022-10-11 09:17:11] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-11 09:17:11] ...optimizers/trijoint.py.151: total_norm: 16.628408 [I 2022-10-11 09:17:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 2200/2304 [I 2022-10-11 09:17:19] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:28 [I 2022-10-11 09:17:19] ...trap/engines/engine.py.229: process: 0.84211 | load: 0.00292 [I 2022-10-11 09:17:19] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-11 09:17:19] ...optimizers/trijoint.py.151: total_norm: 19.758295 [I 2022-10-11 09:17:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 2210/2304 [I 2022-10-11 09:17:28] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:01:19 [I 2022-10-11 09:17:28] ...trap/engines/engine.py.229: process: 0.84751 | load: 0.00293 [I 2022-10-11 09:17:28] ...trap/engines/engine.py.230: loss: 0.11004 [I 2022-10-11 09:17:28] ...optimizers/trijoint.py.151: total_norm: 11.240407 [I 2022-10-11 09:17:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 2220/2304 [I 2022-10-11 09:17:36] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:11 [I 2022-10-11 09:17:36] ...trap/engines/engine.py.229: process: 0.85095 | load: 0.00297 [I 2022-10-11 09:17:36] ...trap/engines/engine.py.230: loss: 0.13443 [I 2022-10-11 09:17:36] ...optimizers/trijoint.py.151: total_norm: 12.163302 [I 2022-10-11 09:17:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 2230/2304 [I 2022-10-11 09:17:45] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:02 [I 2022-10-11 09:17:45] ...trap/engines/engine.py.229: process: 0.85333 | load: 0.00295 [I 2022-10-11 09:17:45] ...trap/engines/engine.py.230: loss: 0.08335 [I 2022-10-11 09:17:45] ...optimizers/trijoint.py.151: total_norm: 11.591978 [I 2022-10-11 09:17:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 2240/2304 [I 2022-10-11 09:17:53] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:00:54 [I 2022-10-11 09:17:53] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00294 [I 2022-10-11 09:17:53] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-11 09:17:53] ...optimizers/trijoint.py.151: total_norm: 14.427289 [I 2022-10-11 09:18:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 2250/2304 [I 2022-10-11 09:18:01] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:00:45 [I 2022-10-11 09:18:01] ...trap/engines/engine.py.229: process: 0.84204 | load: 0.00295 [I 2022-10-11 09:18:01] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-11 09:18:01] ...optimizers/trijoint.py.151: total_norm: 15.198072 [I 2022-10-11 09:18:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 2260/2304 [I 2022-10-11 09:18:10] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:36 [I 2022-10-11 09:18:10] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00293 [I 2022-10-11 09:18:10] ...trap/engines/engine.py.230: loss: 0.11919 [I 2022-10-11 09:18:10] ...optimizers/trijoint.py.151: total_norm: 11.931458 [I 2022-10-11 09:18:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 2270/2304 [I 2022-10-11 09:18:18] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:28 [I 2022-10-11 09:18:18] ...trap/engines/engine.py.229: process: 0.82563 | load: 0.00294 [I 2022-10-11 09:18:18] ...trap/engines/engine.py.230: loss: 0.09783 [I 2022-10-11 09:18:18] ...optimizers/trijoint.py.151: total_norm: 9.880112 [I 2022-10-11 09:18:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 2280/2304 [I 2022-10-11 09:18:27] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:00:19 [I 2022-10-11 09:18:27] ...trap/engines/engine.py.229: process: 0.83618 | load: 0.00297 [I 2022-10-11 09:18:27] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-11 09:18:27] ...optimizers/trijoint.py.151: total_norm: 16.764946 [I 2022-10-11 09:18:35] ...trap/engines/engine.py.224: train: epoch 99 | batch 2290/2304 [I 2022-10-11 09:18:35] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:11 [I 2022-10-11 09:18:35] ...trap/engines/engine.py.229: process: 0.82717 | load: 0.00294 [I 2022-10-11 09:18:35] ...trap/engines/engine.py.230: loss: 0.12611 [I 2022-10-11 09:18:35] ...optimizers/trijoint.py.151: total_norm: 15.480561 [I 2022-10-11 09:18:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 2300/2304 [I 2022-10-11 09:18:43] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:03 [I 2022-10-11 09:18:43] ...trap/engines/engine.py.229: process: 0.81758 | load: 0.00011 [I 2022-10-11 09:18:43] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-11 09:18:43] ...optimizers/trijoint.py.151: total_norm: 19.385309 [I 2022-10-11 09:18:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 2304/2304 [I 2022-10-11 09:18:47] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:00 [I 2022-10-11 09:18:47] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00012 [I 2022-10-11 09:18:47] ...trap/engines/engine.py.230: loss: 0.12384 [I 2022-10-11 09:18:47] ...optimizers/trijoint.py.151: total_norm: 14.783556 [S 2022-10-11 09:18:48] ...trap/engines/engine.py.129: train_epoch.epoch: 99 [S 2022-10-11 09:18:48] ...trap/engines/engine.py.129: train_epoch.loss: 0.12328868781796247 [I 2022-10-11 09:18:48] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 09:18:49] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 99 [I 2022-10-11 09:18:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 09:18:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 09:18:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 09:18:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 09:18:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 09:18:55] ...trap/engines/engine.py.321: eval: epoch 99 | batch 0/510 [I 2022-10-11 09:18:55] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:23 [I 2022-10-11 09:18:55] ...trap/engines/engine.py.326: process: 5.69265 | load: 4.87980 [I 2022-10-11 09:19:01] ...trap/engines/engine.py.321: eval: epoch 99 | batch 10/510 [I 2022-10-11 09:19:01] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:49 [I 2022-10-11 09:19:01] ...trap/engines/engine.py.326: process: 0.66670 | load: 0.00017 [I 2022-10-11 09:19:08] ...trap/engines/engine.py.321: eval: epoch 99 | batch 20/510 [I 2022-10-11 09:19:08] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:06:30 [I 2022-10-11 09:19:08] ...trap/engines/engine.py.326: process: 0.74827 | load: 0.00018 [I 2022-10-11 09:19:22] ...trap/engines/engine.py.321: eval: epoch 99 | batch 30/510 [I 2022-10-11 09:19:22] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:08:27 [I 2022-10-11 09:19:22] ...trap/engines/engine.py.326: process: 0.87787 | load: 0.00021 [I 2022-10-11 09:19:30] ...trap/engines/engine.py.321: eval: epoch 99 | batch 40/510 [I 2022-10-11 09:19:30] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:05:45 [I 2022-10-11 09:19:30] ...trap/engines/engine.py.326: process: 0.62045 | load: 0.00019 [I 2022-10-11 09:19:35] ...trap/engines/engine.py.321: eval: epoch 99 | batch 50/510 [I 2022-10-11 09:19:35] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:04:07 [I 2022-10-11 09:19:35] ...trap/engines/engine.py.326: process: 0.54845 | load: 0.00021 [I 2022-10-11 09:19:40] ...trap/engines/engine.py.321: eval: epoch 99 | batch 60/510 [I 2022-10-11 09:19:40] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:55 [I 2022-10-11 09:19:40] ...trap/engines/engine.py.326: process: 0.51900 | load: 0.00019 [I 2022-10-11 09:19:45] ...trap/engines/engine.py.321: eval: epoch 99 | batch 70/510 [I 2022-10-11 09:19:45] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:44 [I 2022-10-11 09:19:45] ...trap/engines/engine.py.326: process: 0.56583 | load: 0.00020 [I 2022-10-11 09:19:52] ...trap/engines/engine.py.321: eval: epoch 99 | batch 80/510 [I 2022-10-11 09:19:52] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:05:16 [I 2022-10-11 09:19:52] ...trap/engines/engine.py.326: process: 0.60570 | load: 0.00020 [I 2022-10-11 09:20:03] ...trap/engines/engine.py.321: eval: epoch 99 | batch 90/510 [I 2022-10-11 09:20:03] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:11:26 [I 2022-10-11 09:20:03] ...trap/engines/engine.py.326: process: 5.82873 | load: 0.00018 [I 2022-10-11 09:20:14] ...trap/engines/engine.py.321: eval: epoch 99 | batch 100/510 [I 2022-10-11 09:20:14] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:06:06 [I 2022-10-11 09:20:14] ...trap/engines/engine.py.326: process: 0.30056 | load: 0.00018 [I 2022-10-11 09:20:17] ...trap/engines/engine.py.321: eval: epoch 99 | batch 110/510 [I 2022-10-11 09:20:17] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:02:31 [I 2022-10-11 09:20:17] ...trap/engines/engine.py.326: process: 0.30979 | load: 0.00018 [I 2022-10-11 09:20:20] ...trap/engines/engine.py.321: eval: epoch 99 | batch 120/510 [I 2022-10-11 09:20:20] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:02:06 [I 2022-10-11 09:20:20] ...trap/engines/engine.py.326: process: 0.31534 | load: 0.00018 [I 2022-10-11 09:20:23] ...trap/engines/engine.py.321: eval: epoch 99 | batch 130/510 [I 2022-10-11 09:20:23] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:02:00 [I 2022-10-11 09:20:23] ...trap/engines/engine.py.326: process: 0.31154 | load: 0.00018 [I 2022-10-11 09:20:26] ...trap/engines/engine.py.321: eval: epoch 99 | batch 140/510 [I 2022-10-11 09:20:26] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:56 [I 2022-10-11 09:20:26] ...trap/engines/engine.py.326: process: 0.31702 | load: 0.00018 [I 2022-10-11 09:20:29] ...trap/engines/engine.py.321: eval: epoch 99 | batch 150/510 [I 2022-10-11 09:20:29] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:52 [I 2022-10-11 09:20:29] ...trap/engines/engine.py.326: process: 0.31766 | load: 0.00018 [I 2022-10-11 09:20:33] ...trap/engines/engine.py.321: eval: epoch 99 | batch 160/510 [I 2022-10-11 09:20:33] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:50 [I 2022-10-11 09:20:33] ...trap/engines/engine.py.326: process: 0.31969 | load: 0.00018 [I 2022-10-11 09:20:36] ...trap/engines/engine.py.321: eval: epoch 99 | batch 170/510 [I 2022-10-11 09:20:36] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:47 [I 2022-10-11 09:20:36] ...trap/engines/engine.py.326: process: 0.31340 | load: 0.00019 [I 2022-10-11 09:20:39] ...trap/engines/engine.py.321: eval: epoch 99 | batch 180/510 [I 2022-10-11 09:20:39] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:43 [I 2022-10-11 09:20:39] ...trap/engines/engine.py.326: process: 0.31227 | load: 0.00019 [I 2022-10-11 09:20:42] ...trap/engines/engine.py.321: eval: epoch 99 | batch 190/510 [I 2022-10-11 09:20:42] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:40 [I 2022-10-11 09:20:42] ...trap/engines/engine.py.326: process: 0.31656 | load: 0.00019 [I 2022-10-11 09:20:45] ...trap/engines/engine.py.321: eval: epoch 99 | batch 200/510 [I 2022-10-11 09:20:45] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:37 [I 2022-10-11 09:20:45] ...trap/engines/engine.py.326: process: 0.31059 | load: 0.00018 [I 2022-10-11 09:20:48] ...trap/engines/engine.py.321: eval: epoch 99 | batch 210/510 [I 2022-10-11 09:20:48] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:34 [I 2022-10-11 09:20:48] ...trap/engines/engine.py.326: process: 0.31620 | load: 0.00020 [I 2022-10-11 09:20:51] ...trap/engines/engine.py.321: eval: epoch 99 | batch 220/510 [I 2022-10-11 09:20:51] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:32 [I 2022-10-11 09:20:51] ...trap/engines/engine.py.326: process: 0.32157 | load: 0.00020 [I 2022-10-11 09:20:55] ...trap/engines/engine.py.321: eval: epoch 99 | batch 230/510 [I 2022-10-11 09:20:55] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:28 [I 2022-10-11 09:20:55] ...trap/engines/engine.py.326: process: 0.31818 | load: 0.00018 [I 2022-10-11 09:20:58] ...trap/engines/engine.py.321: eval: epoch 99 | batch 240/510 [I 2022-10-11 09:20:58] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:24 [I 2022-10-11 09:20:58] ...trap/engines/engine.py.326: process: 0.30699 | load: 0.00018 [I 2022-10-11 09:21:01] ...trap/engines/engine.py.321: eval: epoch 99 | batch 250/510 [I 2022-10-11 09:21:01] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:21 [I 2022-10-11 09:21:01] ...trap/engines/engine.py.326: process: 0.31395 | load: 0.00017 [I 2022-10-11 09:21:04] ...trap/engines/engine.py.321: eval: epoch 99 | batch 260/510 [I 2022-10-11 09:21:04] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:18 [I 2022-10-11 09:21:04] ...trap/engines/engine.py.326: process: 0.30863 | load: 0.00018 [I 2022-10-11 09:21:07] ...trap/engines/engine.py.321: eval: epoch 99 | batch 270/510 [I 2022-10-11 09:21:07] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:01:15 [I 2022-10-11 09:21:07] ...trap/engines/engine.py.326: process: 0.31814 | load: 0.00019 [I 2022-10-11 09:21:10] ...trap/engines/engine.py.321: eval: epoch 99 | batch 280/510 [I 2022-10-11 09:21:10] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:01:12 [I 2022-10-11 09:21:10] ...trap/engines/engine.py.326: process: 0.32319 | load: 0.00019 [I 2022-10-11 09:21:13] ...trap/engines/engine.py.321: eval: epoch 99 | batch 290/510 [I 2022-10-11 09:21:13] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:01:08 [I 2022-10-11 09:21:13] ...trap/engines/engine.py.326: process: 0.31082 | load: 0.00019 [I 2022-10-11 09:21:17] ...trap/engines/engine.py.321: eval: epoch 99 | batch 300/510 [I 2022-10-11 09:21:17] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:01:06 [I 2022-10-11 09:21:17] ...trap/engines/engine.py.326: process: 0.31764 | load: 0.00019 [I 2022-10-11 09:21:20] ...trap/engines/engine.py.321: eval: epoch 99 | batch 310/510 [I 2022-10-11 09:21:20] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:01:03 [I 2022-10-11 09:21:20] ...trap/engines/engine.py.326: process: 0.30966 | load: 0.00019 [I 2022-10-11 09:21:23] ...trap/engines/engine.py.321: eval: epoch 99 | batch 320/510 [I 2022-10-11 09:21:23] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:59 [I 2022-10-11 09:21:23] ...trap/engines/engine.py.326: process: 0.30807 | load: 0.00016 [I 2022-10-11 09:21:26] ...trap/engines/engine.py.321: eval: epoch 99 | batch 330/510 [I 2022-10-11 09:21:26] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:56 [I 2022-10-11 09:21:26] ...trap/engines/engine.py.326: process: 0.32033 | load: 0.00021 [I 2022-10-11 09:21:29] ...trap/engines/engine.py.321: eval: epoch 99 | batch 340/510 [I 2022-10-11 09:21:29] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:53 [I 2022-10-11 09:21:29] ...trap/engines/engine.py.326: process: 0.31319 | load: 0.00018 [I 2022-10-11 09:21:32] ...trap/engines/engine.py.321: eval: epoch 99 | batch 350/510 [I 2022-10-11 09:21:32] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:50 [I 2022-10-11 09:21:32] ...trap/engines/engine.py.326: process: 0.31332 | load: 0.00019 [I 2022-10-11 09:21:36] ...trap/engines/engine.py.321: eval: epoch 99 | batch 360/510 [I 2022-10-11 09:21:36] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:46 [I 2022-10-11 09:21:36] ...trap/engines/engine.py.326: process: 0.31452 | load: 0.00018 [I 2022-10-11 09:21:39] ...trap/engines/engine.py.321: eval: epoch 99 | batch 370/510 [I 2022-10-11 09:21:39] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:44 [I 2022-10-11 09:21:39] ...trap/engines/engine.py.326: process: 0.31334 | load: 0.00019 [I 2022-10-11 09:21:42] ...trap/engines/engine.py.321: eval: epoch 99 | batch 380/510 [I 2022-10-11 09:21:42] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:41 [I 2022-10-11 09:21:42] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00019 [I 2022-10-11 09:21:45] ...trap/engines/engine.py.321: eval: epoch 99 | batch 390/510 [I 2022-10-11 09:21:45] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:37 [I 2022-10-11 09:21:45] ...trap/engines/engine.py.326: process: 0.31865 | load: 0.00019 [I 2022-10-11 09:21:48] ...trap/engines/engine.py.321: eval: epoch 99 | batch 400/510 [I 2022-10-11 09:21:48] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:34 [I 2022-10-11 09:21:48] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00020 [I 2022-10-11 09:21:51] ...trap/engines/engine.py.321: eval: epoch 99 | batch 410/510 [I 2022-10-11 09:21:51] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:31 [I 2022-10-11 09:21:51] ...trap/engines/engine.py.326: process: 0.31496 | load: 0.00019 [I 2022-10-11 09:21:55] ...trap/engines/engine.py.321: eval: epoch 99 | batch 420/510 [I 2022-10-11 09:21:55] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:28 [I 2022-10-11 09:21:55] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00019 [I 2022-10-11 09:21:58] ...trap/engines/engine.py.321: eval: epoch 99 | batch 430/510 [I 2022-10-11 09:21:58] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:25 [I 2022-10-11 09:21:58] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00019 [I 2022-10-11 09:22:01] ...trap/engines/engine.py.321: eval: epoch 99 | batch 440/510 [I 2022-10-11 09:22:01] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:21 [I 2022-10-11 09:22:01] ...trap/engines/engine.py.326: process: 0.31733 | load: 0.00019 [I 2022-10-11 09:22:04] ...trap/engines/engine.py.321: eval: epoch 99 | batch 450/510 [I 2022-10-11 09:22:04] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:18 [I 2022-10-11 09:22:04] ...trap/engines/engine.py.326: process: 0.31235 | load: 0.00019 [I 2022-10-11 09:22:07] ...trap/engines/engine.py.321: eval: epoch 99 | batch 460/510 [I 2022-10-11 09:22:07] ...trap/engines/engine.py.325: elapsed: 0:03:17 | left: 0:00:15 [I 2022-10-11 09:22:07] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00019 [I 2022-10-11 09:22:10] ...trap/engines/engine.py.321: eval: epoch 99 | batch 470/510 [I 2022-10-11 09:22:10] ...trap/engines/engine.py.325: elapsed: 0:03:20 | left: 0:00:12 [I 2022-10-11 09:22:10] ...trap/engines/engine.py.326: process: 0.31857 | load: 0.00019 [I 2022-10-11 09:22:13] ...trap/engines/engine.py.321: eval: epoch 99 | batch 480/510 [I 2022-10-11 09:22:13] ...trap/engines/engine.py.325: elapsed: 0:03:24 | left: 0:00:09 [I 2022-10-11 09:22:13] ...trap/engines/engine.py.326: process: 0.30757 | load: 0.00019 [I 2022-10-11 09:22:17] ...trap/engines/engine.py.321: eval: epoch 99 | batch 490/510 [I 2022-10-11 09:22:17] ...trap/engines/engine.py.325: elapsed: 0:03:27 | left: 0:00:06 [I 2022-10-11 09:22:17] ...trap/engines/engine.py.326: process: 0.31829 | load: 0.00020 [I 2022-10-11 09:22:20] ...trap/engines/engine.py.321: eval: epoch 99 | batch 500/510 [I 2022-10-11 09:22:20] ...trap/engines/engine.py.325: elapsed: 0:03:30 | left: 0:00:03 [I 2022-10-11 09:22:20] ...trap/engines/engine.py.326: process: 0.31658 | load: 0.00018 [I 2022-10-11 09:22:23] ...trap/engines/engine.py.321: eval: epoch 99 | batch 510/510 [I 2022-10-11 09:22:23] ...trap/engines/engine.py.325: elapsed: 0:03:33 | left: 0:00:00 [I 2022-10-11 09:22:23] ...trap/engines/engine.py.326: process: 0.31069 | load: 0.00011 [S 2022-10-11 09:22:24] ...trap/engines/engine.py.132: eval_epoch.epoch: 99 [I 2022-10-11 09:22:24] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740500 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913800 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939500 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.736000 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910700 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938400 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010966 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005056 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005045 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010686 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005274 [S 2022-10-11 09:22:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005919 [I 2022-10-11 09:23:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 09:23:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 09:23:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 09:23:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 09:23:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 09:23:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 09:23:09] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 09:23:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 09:23:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 100 [I 2022-10-11 09:23:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 09:23:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 0/2304 [I 2022-10-11 09:23:17] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:54:48 [I 2022-10-11 09:23:17] ...trap/engines/engine.py.229: process: 2.98961 | load: 2.07818 [I 2022-10-11 09:23:17] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-11 09:23:17] ...optimizers/trijoint.py.151: total_norm: 14.714581 [I 2022-10-11 09:23:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 10/2304 [I 2022-10-11 09:23:25] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:54 [I 2022-10-11 09:23:25] ...trap/engines/engine.py.229: process: 0.83214 | load: 0.00480 [I 2022-10-11 09:23:25] ...trap/engines/engine.py.230: loss: 0.14687 [I 2022-10-11 09:23:25] ...optimizers/trijoint.py.151: total_norm: 11.929357 [I 2022-10-11 09:23:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 20/2304 [I 2022-10-11 09:23:34] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-11 09:23:34] ...trap/engines/engine.py.229: process: 0.84143 | load: 0.00489 [I 2022-10-11 09:23:34] ...trap/engines/engine.py.230: loss: 0.08993 [I 2022-10-11 09:23:34] ...optimizers/trijoint.py.151: total_norm: 9.017122 [I 2022-10-11 09:23:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 30/2304 [I 2022-10-11 09:23:42] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:10 [I 2022-10-11 09:23:42] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00425 [I 2022-10-11 09:23:42] ...trap/engines/engine.py.230: loss: 0.13246 [I 2022-10-11 09:23:42] ...optimizers/trijoint.py.151: total_norm: 15.372032 [I 2022-10-11 09:23:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 40/2304 [I 2022-10-11 09:23:51] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:02 [I 2022-10-11 09:23:51] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00421 [I 2022-10-11 09:23:51] ...trap/engines/engine.py.230: loss: 0.09596 [I 2022-10-11 09:23:51] ...optimizers/trijoint.py.151: total_norm: 15.770192 [I 2022-10-11 09:23:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 50/2304 [I 2022-10-11 09:23:59] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:24 [I 2022-10-11 09:23:59] ...trap/engines/engine.py.229: process: 0.83423 | load: 0.00368 [I 2022-10-11 09:23:59] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-11 09:23:59] ...optimizers/trijoint.py.151: total_norm: 13.443298 [I 2022-10-11 09:24:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 60/2304 [I 2022-10-11 09:24:07] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:27 [I 2022-10-11 09:24:07] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00396 [I 2022-10-11 09:24:07] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-11 09:24:07] ...optimizers/trijoint.py.151: total_norm: 12.189701 [I 2022-10-11 09:24:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 70/2304 [I 2022-10-11 09:24:16] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:31 [I 2022-10-11 09:24:16] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00328 [I 2022-10-11 09:24:16] ...trap/engines/engine.py.230: loss: 0.09806 [I 2022-10-11 09:24:16] ...optimizers/trijoint.py.151: total_norm: 8.210241 [I 2022-10-11 09:24:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 80/2304 [I 2022-10-11 09:24:24] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:25 [I 2022-10-11 09:24:24] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00321 [I 2022-10-11 09:24:24] ...trap/engines/engine.py.230: loss: 0.08093 [I 2022-10-11 09:24:24] ...optimizers/trijoint.py.151: total_norm: 9.274574 [I 2022-10-11 09:24:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 90/2304 [I 2022-10-11 09:24:33] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:10 [I 2022-10-11 09:24:33] ...trap/engines/engine.py.229: process: 0.84716 | load: 0.00325 [I 2022-10-11 09:24:33] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-11 09:24:33] ...optimizers/trijoint.py.151: total_norm: 9.060912 [I 2022-10-11 09:24:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 100/2304 [I 2022-10-11 09:24:41] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:51 [I 2022-10-11 09:24:41] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00321 [I 2022-10-11 09:24:41] ...trap/engines/engine.py.230: loss: 0.08760 [I 2022-10-11 09:24:41] ...optimizers/trijoint.py.151: total_norm: 11.398849 [I 2022-10-11 09:24:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 110/2304 [I 2022-10-11 09:24:50] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:41 [I 2022-10-11 09:24:50] ...trap/engines/engine.py.229: process: 0.83423 | load: 0.00338 [I 2022-10-11 09:24:50] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-11 09:24:50] ...optimizers/trijoint.py.151: total_norm: 21.137329 [I 2022-10-11 09:24:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 120/2304 [I 2022-10-11 09:24:58] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:42 [I 2022-10-11 09:24:58] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00306 [I 2022-10-11 09:24:58] ...trap/engines/engine.py.230: loss: 0.12886 [I 2022-10-11 09:24:58] ...optimizers/trijoint.py.151: total_norm: 15.232058 [I 2022-10-11 09:25:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 130/2304 [I 2022-10-11 09:25:07] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:16 [I 2022-10-11 09:25:07] ...trap/engines/engine.py.229: process: 0.82375 | load: 0.00307 [I 2022-10-11 09:25:07] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-11 09:25:07] ...optimizers/trijoint.py.151: total_norm: 10.100743 [I 2022-10-11 09:25:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 140/2304 [I 2022-10-11 09:25:15] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:15 [I 2022-10-11 09:25:15] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00300 [I 2022-10-11 09:25:15] ...trap/engines/engine.py.230: loss: 0.09386 [I 2022-10-11 09:25:15] ...optimizers/trijoint.py.151: total_norm: 31.154718 [I 2022-10-11 09:25:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 150/2304 [I 2022-10-11 09:25:23] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:06 [I 2022-10-11 09:25:23] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00302 [I 2022-10-11 09:25:23] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-11 09:25:23] ...optimizers/trijoint.py.151: total_norm: 13.393075 [I 2022-10-11 09:25:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 160/2304 [I 2022-10-11 09:25:32] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:29:53 [I 2022-10-11 09:25:32] ...trap/engines/engine.py.229: process: 0.83935 | load: 0.00292 [I 2022-10-11 09:25:32] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-11 09:25:32] ...optimizers/trijoint.py.151: total_norm: 7.315122 [I 2022-10-11 09:25:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 170/2304 [I 2022-10-11 09:25:40] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:08 [I 2022-10-11 09:25:40] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00318 [I 2022-10-11 09:25:40] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-11 09:25:40] ...optimizers/trijoint.py.151: total_norm: 19.114281 [I 2022-10-11 09:25:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 180/2304 [I 2022-10-11 09:25:49] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:42 [I 2022-10-11 09:25:49] ...trap/engines/engine.py.229: process: 0.84407 | load: 0.00294 [I 2022-10-11 09:25:49] ...trap/engines/engine.py.230: loss: 0.08778 [I 2022-10-11 09:25:49] ...optimizers/trijoint.py.151: total_norm: 12.047164 [I 2022-10-11 09:25:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 190/2304 [I 2022-10-11 09:25:57] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:35 [I 2022-10-11 09:25:57] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00303 [I 2022-10-11 09:25:57] ...trap/engines/engine.py.230: loss: 0.16408 [I 2022-10-11 09:25:57] ...optimizers/trijoint.py.151: total_norm: 9.025266 [I 2022-10-11 09:26:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 200/2304 [I 2022-10-11 09:26:05] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:16 [I 2022-10-11 09:26:05] ...trap/engines/engine.py.229: process: 0.82790 | load: 0.00311 [I 2022-10-11 09:26:05] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-11 09:26:05] ...optimizers/trijoint.py.151: total_norm: 17.366589 [I 2022-10-11 09:26:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 210/2304 [I 2022-10-11 09:26:14] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:29:11 [I 2022-10-11 09:26:14] ...trap/engines/engine.py.229: process: 0.83832 | load: 0.00317 [I 2022-10-11 09:26:14] ...trap/engines/engine.py.230: loss: 0.11399 [I 2022-10-11 09:26:14] ...optimizers/trijoint.py.151: total_norm: 10.988276 [I 2022-10-11 09:26:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 220/2304 [I 2022-10-11 09:26:22] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:27 [I 2022-10-11 09:26:22] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00292 [I 2022-10-11 09:26:22] ...trap/engines/engine.py.230: loss: 0.17761 [I 2022-10-11 09:26:22] ...optimizers/trijoint.py.151: total_norm: 31.910650 [I 2022-10-11 09:26:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 230/2304 [I 2022-10-11 09:26:31] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:29:04 [I 2022-10-11 09:26:31] ...trap/engines/engine.py.229: process: 0.84805 | load: 0.00296 [I 2022-10-11 09:26:31] ...trap/engines/engine.py.230: loss: 0.08858 [I 2022-10-11 09:26:31] ...optimizers/trijoint.py.151: total_norm: 12.414865 [I 2022-10-11 09:26:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 240/2304 [I 2022-10-11 09:26:39] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:28:46 [I 2022-10-11 09:26:39] ...trap/engines/engine.py.229: process: 0.83210 | load: 0.00294 [I 2022-10-11 09:26:39] ...trap/engines/engine.py.230: loss: 0.10366 [I 2022-10-11 09:26:39] ...optimizers/trijoint.py.151: total_norm: 8.641564 [I 2022-10-11 09:26:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 250/2304 [I 2022-10-11 09:26:48] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:43 [I 2022-10-11 09:26:48] ...trap/engines/engine.py.229: process: 0.83897 | load: 0.00292 [I 2022-10-11 09:26:48] ...trap/engines/engine.py.230: loss: 0.06930 [I 2022-10-11 09:26:48] ...optimizers/trijoint.py.151: total_norm: 15.233945 [I 2022-10-11 09:26:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 260/2304 [I 2022-10-11 09:26:56] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:33 [I 2022-10-11 09:26:56] ...trap/engines/engine.py.229: process: 0.82714 | load: 0.00307 [I 2022-10-11 09:26:56] ...trap/engines/engine.py.230: loss: 0.11920 [I 2022-10-11 09:26:56] ...optimizers/trijoint.py.151: total_norm: 13.307796 [I 2022-10-11 09:27:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 270/2304 [I 2022-10-11 09:27:05] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:28:17 [I 2022-10-11 09:27:05] ...trap/engines/engine.py.229: process: 0.82999 | load: 0.00295 [I 2022-10-11 09:27:05] ...trap/engines/engine.py.230: loss: 0.09159 [I 2022-10-11 09:27:05] ...optimizers/trijoint.py.151: total_norm: 12.385266 [I 2022-10-11 09:27:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 280/2304 [I 2022-10-11 09:27:13] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:31 [I 2022-10-11 09:27:13] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00295 [I 2022-10-11 09:27:13] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-11 09:27:13] ...optimizers/trijoint.py.151: total_norm: 12.155286 [I 2022-10-11 09:27:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 290/2304 [I 2022-10-11 09:27:22] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:28:25 [I 2022-10-11 09:27:22] ...trap/engines/engine.py.229: process: 0.85444 | load: 0.00296 [I 2022-10-11 09:27:22] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-11 09:27:22] ...optimizers/trijoint.py.151: total_norm: 36.683193 [I 2022-10-11 09:27:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 300/2304 [I 2022-10-11 09:27:30] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:14 [I 2022-10-11 09:27:30] ...trap/engines/engine.py.229: process: 0.84288 | load: 0.00293 [I 2022-10-11 09:27:30] ...trap/engines/engine.py.230: loss: 0.11074 [I 2022-10-11 09:27:30] ...optimizers/trijoint.py.151: total_norm: 12.672708 [I 2022-10-11 09:27:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 310/2304 [I 2022-10-11 09:27:39] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:05 [I 2022-10-11 09:27:39] ...trap/engines/engine.py.229: process: 0.83798 | load: 0.00293 [I 2022-10-11 09:27:39] ...trap/engines/engine.py.230: loss: 0.09427 [I 2022-10-11 09:27:39] ...optimizers/trijoint.py.151: total_norm: 10.902631 [I 2022-10-11 09:27:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 320/2304 [I 2022-10-11 09:27:47] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:49 [I 2022-10-11 09:27:47] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00302 [I 2022-10-11 09:27:47] ...trap/engines/engine.py.230: loss: 0.08964 [I 2022-10-11 09:27:47] ...optimizers/trijoint.py.151: total_norm: 12.808146 [I 2022-10-11 09:27:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 330/2304 [I 2022-10-11 09:27:55] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:46 [I 2022-10-11 09:27:55] ...trap/engines/engine.py.229: process: 0.83440 | load: 0.00293 [I 2022-10-11 09:27:55] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-11 09:27:55] ...optimizers/trijoint.py.151: total_norm: 27.295174 [I 2022-10-11 09:28:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 340/2304 [I 2022-10-11 09:28:04] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:37 [I 2022-10-11 09:28:04] ...trap/engines/engine.py.229: process: 0.85279 | load: 0.00294 [I 2022-10-11 09:28:04] ...trap/engines/engine.py.230: loss: 0.09773 [I 2022-10-11 09:28:04] ...optimizers/trijoint.py.151: total_norm: 18.474407 [I 2022-10-11 09:28:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 350/2304 [I 2022-10-11 09:28:12] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:22 [I 2022-10-11 09:28:12] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00304 [I 2022-10-11 09:28:12] ...trap/engines/engine.py.230: loss: 0.10727 [I 2022-10-11 09:28:12] ...optimizers/trijoint.py.151: total_norm: 14.892390 [I 2022-10-11 09:28:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 360/2304 [I 2022-10-11 09:28:21] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:10 [I 2022-10-11 09:28:21] ...trap/engines/engine.py.229: process: 0.82978 | load: 0.00294 [I 2022-10-11 09:28:21] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-11 09:28:21] ...optimizers/trijoint.py.151: total_norm: 16.681717 [I 2022-10-11 09:28:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 370/2304 [I 2022-10-11 09:28:29] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:08 [I 2022-10-11 09:28:29] ...trap/engines/engine.py.229: process: 0.82151 | load: 0.00295 [I 2022-10-11 09:28:29] ...trap/engines/engine.py.230: loss: 0.07800 [I 2022-10-11 09:28:29] ...optimizers/trijoint.py.151: total_norm: 15.047400 [I 2022-10-11 09:28:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 380/2304 [I 2022-10-11 09:28:38] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:26:56 [I 2022-10-11 09:28:38] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00294 [I 2022-10-11 09:28:38] ...trap/engines/engine.py.230: loss: 0.12751 [I 2022-10-11 09:28:38] ...optimizers/trijoint.py.151: total_norm: 26.169014 [I 2022-10-11 09:28:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 390/2304 [I 2022-10-11 09:28:46] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:50 [I 2022-10-11 09:28:46] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00293 [I 2022-10-11 09:28:46] ...trap/engines/engine.py.230: loss: 0.12259 [I 2022-10-11 09:28:46] ...optimizers/trijoint.py.151: total_norm: 21.137917 [I 2022-10-11 09:28:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 400/2304 [I 2022-10-11 09:28:55] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:26:30 [I 2022-10-11 09:28:55] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00295 [I 2022-10-11 09:28:55] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-11 09:28:55] ...optimizers/trijoint.py.151: total_norm: 23.646671 [I 2022-10-11 09:29:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 410/2304 [I 2022-10-11 09:29:03] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:33 [I 2022-10-11 09:29:03] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00295 [I 2022-10-11 09:29:03] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-11 09:29:03] ...optimizers/trijoint.py.151: total_norm: 11.021956 [I 2022-10-11 09:29:11] ...trap/engines/engine.py.224: train: epoch 100 | batch 420/2304 [I 2022-10-11 09:29:11] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:16 [I 2022-10-11 09:29:11] ...trap/engines/engine.py.229: process: 0.82550 | load: 0.00292 [I 2022-10-11 09:29:11] ...trap/engines/engine.py.230: loss: 0.13379 [I 2022-10-11 09:29:11] ...optimizers/trijoint.py.151: total_norm: 10.512623 [I 2022-10-11 09:29:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 430/2304 [I 2022-10-11 09:29:20] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:26:19 [I 2022-10-11 09:29:20] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00293 [I 2022-10-11 09:29:20] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-11 09:29:20] ...optimizers/trijoint.py.151: total_norm: 13.049983 [I 2022-10-11 09:29:28] ...trap/engines/engine.py.224: train: epoch 100 | batch 440/2304 [I 2022-10-11 09:29:28] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:25:56 [I 2022-10-11 09:29:28] ...trap/engines/engine.py.229: process: 0.83095 | load: 0.00293 [I 2022-10-11 09:29:28] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-11 09:29:28] ...optimizers/trijoint.py.151: total_norm: 10.312037 [I 2022-10-11 09:29:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 450/2304 [I 2022-10-11 09:29:37] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:25:53 [I 2022-10-11 09:29:37] ...trap/engines/engine.py.229: process: 0.83245 | load: 0.00292 [I 2022-10-11 09:29:37] ...trap/engines/engine.py.230: loss: 0.09731 [I 2022-10-11 09:29:37] ...optimizers/trijoint.py.151: total_norm: 12.951972 [I 2022-10-11 09:29:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 460/2304 [I 2022-10-11 09:29:45] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:25:50 [I 2022-10-11 09:29:45] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00297 [I 2022-10-11 09:29:45] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-11 09:29:45] ...optimizers/trijoint.py.151: total_norm: 17.954805 [I 2022-10-11 09:29:53] ...trap/engines/engine.py.224: train: epoch 100 | batch 470/2304 [I 2022-10-11 09:29:53] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:25:43 [I 2022-10-11 09:29:53] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00295 [I 2022-10-11 09:29:53] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-11 09:29:53] ...optimizers/trijoint.py.151: total_norm: 20.808043 [I 2022-10-11 09:30:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 480/2304 [I 2022-10-11 09:30:02] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:25:32 [I 2022-10-11 09:30:02] ...trap/engines/engine.py.229: process: 0.84227 | load: 0.00296 [I 2022-10-11 09:30:02] ...trap/engines/engine.py.230: loss: 0.18269 [I 2022-10-11 09:30:02] ...optimizers/trijoint.py.151: total_norm: 17.496830 [I 2022-10-11 09:30:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 490/2304 [I 2022-10-11 09:30:10] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:25:25 [I 2022-10-11 09:30:10] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00293 [I 2022-10-11 09:30:10] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-11 09:30:10] ...optimizers/trijoint.py.151: total_norm: 26.314337 [I 2022-10-11 09:30:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 500/2304 [I 2022-10-11 09:30:19] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:25:33 [I 2022-10-11 09:30:19] ...trap/engines/engine.py.229: process: 0.87138 | load: 0.00305 [I 2022-10-11 09:30:19] ...trap/engines/engine.py.230: loss: 0.13526 [I 2022-10-11 09:30:19] ...optimizers/trijoint.py.151: total_norm: 38.029800 [I 2022-10-11 09:30:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 510/2304 [I 2022-10-11 09:30:27] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:14 [I 2022-10-11 09:30:27] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00292 [I 2022-10-11 09:30:27] ...trap/engines/engine.py.230: loss: 0.09878 [I 2022-10-11 09:30:27] ...optimizers/trijoint.py.151: total_norm: 10.152349 [I 2022-10-11 09:30:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 520/2304 [I 2022-10-11 09:30:36] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:25:06 [I 2022-10-11 09:30:36] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00293 [I 2022-10-11 09:30:36] ...trap/engines/engine.py.230: loss: 0.14078 [I 2022-10-11 09:30:36] ...optimizers/trijoint.py.151: total_norm: 22.813784 [I 2022-10-11 09:30:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 530/2304 [I 2022-10-11 09:30:44] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:24:49 [I 2022-10-11 09:30:44] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00296 [I 2022-10-11 09:30:44] ...trap/engines/engine.py.230: loss: 0.09346 [I 2022-10-11 09:30:44] ...optimizers/trijoint.py.151: total_norm: 9.473389 [I 2022-10-11 09:30:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 540/2304 [I 2022-10-11 09:30:52] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:24:41 [I 2022-10-11 09:30:52] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00294 [I 2022-10-11 09:30:52] ...trap/engines/engine.py.230: loss: 0.14984 [I 2022-10-11 09:30:52] ...optimizers/trijoint.py.151: total_norm: 9.702577 [I 2022-10-11 09:31:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 550/2304 [I 2022-10-11 09:31:01] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:24:30 [I 2022-10-11 09:31:01] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00295 [I 2022-10-11 09:31:01] ...trap/engines/engine.py.230: loss: 0.10487 [I 2022-10-11 09:31:01] ...optimizers/trijoint.py.151: total_norm: 19.802830 [I 2022-10-11 09:31:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 560/2304 [I 2022-10-11 09:31:09] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:24:35 [I 2022-10-11 09:31:09] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00292 [I 2022-10-11 09:31:09] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-11 09:31:09] ...optimizers/trijoint.py.151: total_norm: 16.674053 [I 2022-10-11 09:31:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 570/2304 [I 2022-10-11 09:31:18] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:24:25 [I 2022-10-11 09:31:18] ...trap/engines/engine.py.229: process: 0.84943 | load: 0.00292 [I 2022-10-11 09:31:18] ...trap/engines/engine.py.230: loss: 0.08548 [I 2022-10-11 09:31:18] ...optimizers/trijoint.py.151: total_norm: 16.237093 [I 2022-10-11 09:31:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 580/2304 [I 2022-10-11 09:31:26] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:13 [I 2022-10-11 09:31:26] ...trap/engines/engine.py.229: process: 0.85060 | load: 0.00296 [I 2022-10-11 09:31:26] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-11 09:31:26] ...optimizers/trijoint.py.151: total_norm: 10.850911 [I 2022-10-11 09:31:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 590/2304 [I 2022-10-11 09:31:35] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:15 [I 2022-10-11 09:31:35] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00294 [I 2022-10-11 09:31:35] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-11 09:31:35] ...optimizers/trijoint.py.151: total_norm: 19.638540 [I 2022-10-11 09:31:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 600/2304 [I 2022-10-11 09:31:43] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:24:11 [I 2022-10-11 09:31:43] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00292 [I 2022-10-11 09:31:43] ...trap/engines/engine.py.230: loss: 0.10573 [I 2022-10-11 09:31:43] ...optimizers/trijoint.py.151: total_norm: 23.425245 [I 2022-10-11 09:31:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 610/2304 [I 2022-10-11 09:31:52] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:23:59 [I 2022-10-11 09:31:52] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00291 [I 2022-10-11 09:31:52] ...trap/engines/engine.py.230: loss: 0.10224 [I 2022-10-11 09:31:52] ...optimizers/trijoint.py.151: total_norm: 12.198864 [I 2022-10-11 09:32:00] ...trap/engines/engine.py.224: train: epoch 100 | batch 620/2304 [I 2022-10-11 09:32:00] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:54 [I 2022-10-11 09:32:00] ...trap/engines/engine.py.229: process: 0.84168 | load: 0.00292 [I 2022-10-11 09:32:00] ...trap/engines/engine.py.230: loss: 0.10344 [I 2022-10-11 09:32:00] ...optimizers/trijoint.py.151: total_norm: 11.783534 [I 2022-10-11 09:32:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 630/2304 [I 2022-10-11 09:32:09] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:25 [I 2022-10-11 09:32:09] ...trap/engines/engine.py.229: process: 0.83147 | load: 0.00297 [I 2022-10-11 09:32:09] ...trap/engines/engine.py.230: loss: 0.11099 [I 2022-10-11 09:32:09] ...optimizers/trijoint.py.151: total_norm: 23.979254 [I 2022-10-11 09:32:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 640/2304 [I 2022-10-11 09:32:17] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:08 [I 2022-10-11 09:32:17] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00295 [I 2022-10-11 09:32:17] ...trap/engines/engine.py.230: loss: 0.16168 [I 2022-10-11 09:32:17] ...optimizers/trijoint.py.151: total_norm: 8.494605 [I 2022-10-11 09:32:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 650/2304 [I 2022-10-11 09:32:25] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:01 [I 2022-10-11 09:32:25] ...trap/engines/engine.py.229: process: 0.81965 | load: 0.00292 [I 2022-10-11 09:32:25] ...trap/engines/engine.py.230: loss: 0.11445 [I 2022-10-11 09:32:25] ...optimizers/trijoint.py.151: total_norm: 11.451062 [I 2022-10-11 09:32:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 660/2304 [I 2022-10-11 09:32:34] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:01 [I 2022-10-11 09:32:34] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00292 [I 2022-10-11 09:32:34] ...trap/engines/engine.py.230: loss: 0.14057 [I 2022-10-11 09:32:34] ...optimizers/trijoint.py.151: total_norm: 12.181876 [I 2022-10-11 09:32:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 670/2304 [I 2022-10-11 09:32:42] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:22:49 [I 2022-10-11 09:32:42] ...trap/engines/engine.py.229: process: 0.85027 | load: 0.00294 [I 2022-10-11 09:32:42] ...trap/engines/engine.py.230: loss: 0.10955 [I 2022-10-11 09:32:42] ...optimizers/trijoint.py.151: total_norm: 11.837975 [I 2022-10-11 09:32:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 680/2304 [I 2022-10-11 09:32:51] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:22:36 [I 2022-10-11 09:32:51] ...trap/engines/engine.py.229: process: 0.83731 | load: 0.00293 [I 2022-10-11 09:32:51] ...trap/engines/engine.py.230: loss: 0.12095 [I 2022-10-11 09:32:51] ...optimizers/trijoint.py.151: total_norm: 19.527069 [I 2022-10-11 09:32:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 690/2304 [I 2022-10-11 09:32:59] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:22:37 [I 2022-10-11 09:32:59] ...trap/engines/engine.py.229: process: 0.84447 | load: 0.00294 [I 2022-10-11 09:32:59] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-11 09:32:59] ...optimizers/trijoint.py.151: total_norm: 8.290770 [I 2022-10-11 09:33:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 700/2304 [I 2022-10-11 09:33:07] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:22:24 [I 2022-10-11 09:33:07] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00294 [I 2022-10-11 09:33:07] ...trap/engines/engine.py.230: loss: 0.13673 [I 2022-10-11 09:33:07] ...optimizers/trijoint.py.151: total_norm: 16.192390 [I 2022-10-11 09:33:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 710/2304 [I 2022-10-11 09:33:16] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:21 [I 2022-10-11 09:33:16] ...trap/engines/engine.py.229: process: 0.83450 | load: 0.00293 [I 2022-10-11 09:33:16] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-11 09:33:16] ...optimizers/trijoint.py.151: total_norm: 10.029509 [I 2022-10-11 09:33:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 720/2304 [I 2022-10-11 09:33:24] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:20 [I 2022-10-11 09:33:24] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00294 [I 2022-10-11 09:33:24] ...trap/engines/engine.py.230: loss: 0.08590 [I 2022-10-11 09:33:24] ...optimizers/trijoint.py.151: total_norm: 15.483815 [I 2022-10-11 09:33:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 730/2304 [I 2022-10-11 09:33:33] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:16 [I 2022-10-11 09:33:33] ...trap/engines/engine.py.229: process: 0.87220 | load: 0.00294 [I 2022-10-11 09:33:33] ...trap/engines/engine.py.230: loss: 0.06929 [I 2022-10-11 09:33:33] ...optimizers/trijoint.py.151: total_norm: 11.874983 [I 2022-10-11 09:33:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 740/2304 [I 2022-10-11 09:33:41] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:21:53 [I 2022-10-11 09:33:41] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00295 [I 2022-10-11 09:33:41] ...trap/engines/engine.py.230: loss: 0.10007 [I 2022-10-11 09:33:41] ...optimizers/trijoint.py.151: total_norm: 14.548460 [I 2022-10-11 09:33:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 750/2304 [I 2022-10-11 09:33:49] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:21:43 [I 2022-10-11 09:33:49] ...trap/engines/engine.py.229: process: 0.83093 | load: 0.00295 [I 2022-10-11 09:33:49] ...trap/engines/engine.py.230: loss: 0.17107 [I 2022-10-11 09:33:49] ...optimizers/trijoint.py.151: total_norm: 29.957111 [I 2022-10-11 09:33:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 760/2304 [I 2022-10-11 09:33:58] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:38 [I 2022-10-11 09:33:58] ...trap/engines/engine.py.229: process: 0.86470 | load: 0.00294 [I 2022-10-11 09:33:58] ...trap/engines/engine.py.230: loss: 0.13527 [I 2022-10-11 09:33:58] ...optimizers/trijoint.py.151: total_norm: 18.051731 [I 2022-10-11 09:34:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 770/2304 [I 2022-10-11 09:34:06] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:21:42 [I 2022-10-11 09:34:06] ...trap/engines/engine.py.229: process: 0.86650 | load: 0.00292 [I 2022-10-11 09:34:06] ...trap/engines/engine.py.230: loss: 0.17124 [I 2022-10-11 09:34:06] ...optimizers/trijoint.py.151: total_norm: 23.530811 [I 2022-10-11 09:34:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 780/2304 [I 2022-10-11 09:34:15] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:19 [I 2022-10-11 09:34:15] ...trap/engines/engine.py.229: process: 0.83165 | load: 0.00291 [I 2022-10-11 09:34:15] ...trap/engines/engine.py.230: loss: 0.07495 [I 2022-10-11 09:34:15] ...optimizers/trijoint.py.151: total_norm: 15.623915 [I 2022-10-11 09:34:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 790/2304 [I 2022-10-11 09:34:27] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:31:11 [I 2022-10-11 09:34:27] ...trap/engines/engine.py.229: process: 0.90314 | load: 0.00377 [I 2022-10-11 09:34:27] ...trap/engines/engine.py.230: loss: 0.14473 [I 2022-10-11 09:34:27] ...optimizers/trijoint.py.151: total_norm: 11.544534 [I 2022-10-11 09:34:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 800/2304 [I 2022-10-11 09:34:36] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:23:13 [I 2022-10-11 09:34:36] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00375 [I 2022-10-11 09:34:36] ...trap/engines/engine.py.230: loss: 0.12188 [I 2022-10-11 09:34:36] ...optimizers/trijoint.py.151: total_norm: 11.479912 [I 2022-10-11 09:34:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 810/2304 [I 2022-10-11 09:34:45] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:46 [I 2022-10-11 09:34:45] ...trap/engines/engine.py.229: process: 0.87623 | load: 0.00374 [I 2022-10-11 09:34:45] ...trap/engines/engine.py.230: loss: 0.14887 [I 2022-10-11 09:34:45] ...optimizers/trijoint.py.151: total_norm: 11.196569 [I 2022-10-11 09:34:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 820/2304 [I 2022-10-11 09:34:54] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:20:37 [I 2022-10-11 09:34:54] ...trap/engines/engine.py.229: process: 0.81212 | load: 0.00294 [I 2022-10-11 09:34:54] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-11 09:34:54] ...optimizers/trijoint.py.151: total_norm: 10.152532 [I 2022-10-11 09:35:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 830/2304 [I 2022-10-11 09:35:02] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:20:38 [I 2022-10-11 09:35:02] ...trap/engines/engine.py.229: process: 0.84351 | load: 0.00291 [I 2022-10-11 09:35:02] ...trap/engines/engine.py.230: loss: 0.10141 [I 2022-10-11 09:35:02] ...optimizers/trijoint.py.151: total_norm: 13.024748 [I 2022-10-11 09:35:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 840/2304 [I 2022-10-11 09:35:10] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:20 [I 2022-10-11 09:35:10] ...trap/engines/engine.py.229: process: 0.83789 | load: 0.00294 [I 2022-10-11 09:35:10] ...trap/engines/engine.py.230: loss: 0.14505 [I 2022-10-11 09:35:10] ...optimizers/trijoint.py.151: total_norm: 19.852013 [I 2022-10-11 09:35:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 850/2304 [I 2022-10-11 09:35:19] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:12 [I 2022-10-11 09:35:19] ...trap/engines/engine.py.229: process: 0.82778 | load: 0.00293 [I 2022-10-11 09:35:19] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-11 09:35:19] ...optimizers/trijoint.py.151: total_norm: 7.768661 [I 2022-10-11 09:35:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 860/2304 [I 2022-10-11 09:35:27] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:11 [I 2022-10-11 09:35:27] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00291 [I 2022-10-11 09:35:27] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-11 09:35:27] ...optimizers/trijoint.py.151: total_norm: 10.975199 [I 2022-10-11 09:35:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 870/2304 [I 2022-10-11 09:35:35] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:19:53 [I 2022-10-11 09:35:35] ...trap/engines/engine.py.229: process: 0.82588 | load: 0.00294 [I 2022-10-11 09:35:35] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-11 09:35:35] ...optimizers/trijoint.py.151: total_norm: 15.076245 [I 2022-10-11 09:35:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 880/2304 [I 2022-10-11 09:35:44] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:19:50 [I 2022-10-11 09:35:44] ...trap/engines/engine.py.229: process: 0.82459 | load: 0.00295 [I 2022-10-11 09:35:44] ...trap/engines/engine.py.230: loss: 0.17084 [I 2022-10-11 09:35:44] ...optimizers/trijoint.py.151: total_norm: 13.763750 [I 2022-10-11 09:35:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 890/2304 [I 2022-10-11 09:35:52] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:19:40 [I 2022-10-11 09:35:52] ...trap/engines/engine.py.229: process: 0.83256 | load: 0.00296 [I 2022-10-11 09:35:52] ...trap/engines/engine.py.230: loss: 0.09556 [I 2022-10-11 09:35:52] ...optimizers/trijoint.py.151: total_norm: 11.080866 [I 2022-10-11 09:36:00] ...trap/engines/engine.py.224: train: epoch 100 | batch 900/2304 [I 2022-10-11 09:36:00] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:28 [I 2022-10-11 09:36:00] ...trap/engines/engine.py.229: process: 0.82465 | load: 0.00295 [I 2022-10-11 09:36:00] ...trap/engines/engine.py.230: loss: 0.11190 [I 2022-10-11 09:36:00] ...optimizers/trijoint.py.151: total_norm: 11.581205 [I 2022-10-11 09:36:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 910/2304 [I 2022-10-11 09:36:09] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:19:29 [I 2022-10-11 09:36:09] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00292 [I 2022-10-11 09:36:09] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-11 09:36:09] ...optimizers/trijoint.py.151: total_norm: 13.723083 [I 2022-10-11 09:36:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 920/2304 [I 2022-10-11 09:36:17] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:17 [I 2022-10-11 09:36:17] ...trap/engines/engine.py.229: process: 0.83088 | load: 0.00292 [I 2022-10-11 09:36:17] ...trap/engines/engine.py.230: loss: 0.12167 [I 2022-10-11 09:36:17] ...optimizers/trijoint.py.151: total_norm: 12.890527 [I 2022-10-11 09:36:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 930/2304 [I 2022-10-11 09:36:26] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:15 [I 2022-10-11 09:36:26] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00293 [I 2022-10-11 09:36:26] ...trap/engines/engine.py.230: loss: 0.14026 [I 2022-10-11 09:36:26] ...optimizers/trijoint.py.151: total_norm: 11.595093 [I 2022-10-11 09:36:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 940/2304 [I 2022-10-11 09:36:34] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:01 [I 2022-10-11 09:36:34] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00293 [I 2022-10-11 09:36:34] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-11 09:36:34] ...optimizers/trijoint.py.151: total_norm: 25.309710 [I 2022-10-11 09:36:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 950/2304 [I 2022-10-11 09:36:42] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:18:58 [I 2022-10-11 09:36:42] ...trap/engines/engine.py.229: process: 0.82825 | load: 0.00291 [I 2022-10-11 09:36:42] ...trap/engines/engine.py.230: loss: 0.17275 [I 2022-10-11 09:36:42] ...optimizers/trijoint.py.151: total_norm: 20.005116 [I 2022-10-11 09:36:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 960/2304 [I 2022-10-11 09:36:51] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:18:48 [I 2022-10-11 09:36:51] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00293 [I 2022-10-11 09:36:51] ...trap/engines/engine.py.230: loss: 0.09788 [I 2022-10-11 09:36:51] ...optimizers/trijoint.py.151: total_norm: 25.977261 [I 2022-10-11 09:36:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 970/2304 [I 2022-10-11 09:36:59] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:18:40 [I 2022-10-11 09:36:59] ...trap/engines/engine.py.229: process: 0.84104 | load: 0.00295 [I 2022-10-11 09:36:59] ...trap/engines/engine.py.230: loss: 0.17327 [I 2022-10-11 09:36:59] ...optimizers/trijoint.py.151: total_norm: 17.004744 [I 2022-10-11 09:37:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 980/2304 [I 2022-10-11 09:37:08] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:18:23 [I 2022-10-11 09:37:08] ...trap/engines/engine.py.229: process: 0.83549 | load: 0.00295 [I 2022-10-11 09:37:08] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-11 09:37:08] ...optimizers/trijoint.py.151: total_norm: 8.856838 [I 2022-10-11 09:37:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 990/2304 [I 2022-10-11 09:37:16] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:10 [I 2022-10-11 09:37:16] ...trap/engines/engine.py.229: process: 0.81873 | load: 0.00295 [I 2022-10-11 09:37:16] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-11 09:37:16] ...optimizers/trijoint.py.151: total_norm: 10.552038 [I 2022-10-11 09:37:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 1000/2304 [I 2022-10-11 09:37:24] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:18:08 [I 2022-10-11 09:37:24] ...trap/engines/engine.py.229: process: 0.82771 | load: 0.00296 [I 2022-10-11 09:37:24] ...trap/engines/engine.py.230: loss: 0.16059 [I 2022-10-11 09:37:24] ...optimizers/trijoint.py.151: total_norm: 16.098156 [I 2022-10-11 09:37:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 1010/2304 [I 2022-10-11 09:37:33] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:00 [I 2022-10-11 09:37:33] ...trap/engines/engine.py.229: process: 0.84109 | load: 0.00294 [I 2022-10-11 09:37:33] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-11 09:37:33] ...optimizers/trijoint.py.151: total_norm: 8.699674 [I 2022-10-11 09:37:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 1020/2304 [I 2022-10-11 09:37:41] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:17:57 [I 2022-10-11 09:37:41] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00293 [I 2022-10-11 09:37:41] ...trap/engines/engine.py.230: loss: 0.09800 [I 2022-10-11 09:37:41] ...optimizers/trijoint.py.151: total_norm: 13.006936 [I 2022-10-11 09:37:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 1030/2304 [I 2022-10-11 09:37:49] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:17:48 [I 2022-10-11 09:37:49] ...trap/engines/engine.py.229: process: 0.85351 | load: 0.00295 [I 2022-10-11 09:37:49] ...trap/engines/engine.py.230: loss: 0.07701 [I 2022-10-11 09:37:49] ...optimizers/trijoint.py.151: total_norm: 10.584517 [I 2022-10-11 09:37:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 1040/2304 [I 2022-10-11 09:37:58] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:43 [I 2022-10-11 09:37:58] ...trap/engines/engine.py.229: process: 0.85892 | load: 0.00294 [I 2022-10-11 09:37:58] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-11 09:37:58] ...optimizers/trijoint.py.151: total_norm: 15.004738 [I 2022-10-11 09:38:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 1050/2304 [I 2022-10-11 09:38:06] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:32 [I 2022-10-11 09:38:06] ...trap/engines/engine.py.229: process: 0.84305 | load: 0.00296 [I 2022-10-11 09:38:06] ...trap/engines/engine.py.230: loss: 0.13493 [I 2022-10-11 09:38:06] ...optimizers/trijoint.py.151: total_norm: 21.473747 [I 2022-10-11 09:38:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 1060/2304 [I 2022-10-11 09:38:15] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:19 [I 2022-10-11 09:38:15] ...trap/engines/engine.py.229: process: 0.82881 | load: 0.00295 [I 2022-10-11 09:38:15] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-11 09:38:15] ...optimizers/trijoint.py.151: total_norm: 20.803766 [I 2022-10-11 09:38:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 1070/2304 [I 2022-10-11 09:38:23] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:21 [I 2022-10-11 09:38:23] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00293 [I 2022-10-11 09:38:23] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-11 09:38:23] ...optimizers/trijoint.py.151: total_norm: 15.121852 [I 2022-10-11 09:38:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 1080/2304 [I 2022-10-11 09:38:31] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:13 [I 2022-10-11 09:38:31] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00295 [I 2022-10-11 09:38:31] ...trap/engines/engine.py.230: loss: 0.14912 [I 2022-10-11 09:38:31] ...optimizers/trijoint.py.151: total_norm: 17.620369 [I 2022-10-11 09:38:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 1090/2304 [I 2022-10-11 09:38:40] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:16:59 [I 2022-10-11 09:38:40] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00297 [I 2022-10-11 09:38:40] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-11 09:38:40] ...optimizers/trijoint.py.151: total_norm: 20.376736 [I 2022-10-11 09:38:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 1100/2304 [I 2022-10-11 09:38:48] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:16:57 [I 2022-10-11 09:38:48] ...trap/engines/engine.py.229: process: 0.84164 | load: 0.00295 [I 2022-10-11 09:38:48] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-11 09:38:48] ...optimizers/trijoint.py.151: total_norm: 21.232904 [I 2022-10-11 09:38:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 1110/2304 [I 2022-10-11 09:38:57] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:16:44 [I 2022-10-11 09:38:57] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00297 [I 2022-10-11 09:38:57] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-11 09:38:57] ...optimizers/trijoint.py.151: total_norm: 25.168943 [I 2022-10-11 09:39:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 1120/2304 [I 2022-10-11 09:39:05] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:37 [I 2022-10-11 09:39:05] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00295 [I 2022-10-11 09:39:05] ...trap/engines/engine.py.230: loss: 0.18798 [I 2022-10-11 09:39:05] ...optimizers/trijoint.py.151: total_norm: 24.681763 [I 2022-10-11 09:39:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 1130/2304 [I 2022-10-11 09:39:14] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:19 [I 2022-10-11 09:39:14] ...trap/engines/engine.py.229: process: 0.83456 | load: 0.00295 [I 2022-10-11 09:39:14] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-11 09:39:14] ...optimizers/trijoint.py.151: total_norm: 14.169566 [I 2022-10-11 09:39:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 1140/2304 [I 2022-10-11 09:39:22] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:11 [I 2022-10-11 09:39:22] ...trap/engines/engine.py.229: process: 0.83716 | load: 0.00294 [I 2022-10-11 09:39:22] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-11 09:39:22] ...optimizers/trijoint.py.151: total_norm: 15.657003 [I 2022-10-11 09:39:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 1150/2304 [I 2022-10-11 09:39:30] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:11 [I 2022-10-11 09:39:30] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00291 [I 2022-10-11 09:39:30] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-11 09:39:30] ...optimizers/trijoint.py.151: total_norm: 11.123819 [I 2022-10-11 09:39:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 1160/2304 [I 2022-10-11 09:39:39] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:15:54 [I 2022-10-11 09:39:39] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00292 [I 2022-10-11 09:39:39] ...trap/engines/engine.py.230: loss: 0.10298 [I 2022-10-11 09:39:39] ...optimizers/trijoint.py.151: total_norm: 11.283854 [I 2022-10-11 09:39:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1170/2304 [I 2022-10-11 09:39:47] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:15:52 [I 2022-10-11 09:39:47] ...trap/engines/engine.py.229: process: 0.82285 | load: 0.00293 [I 2022-10-11 09:39:47] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-11 09:39:47] ...optimizers/trijoint.py.151: total_norm: 19.468693 [I 2022-10-11 09:39:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 1180/2304 [I 2022-10-11 09:39:55] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:15:36 [I 2022-10-11 09:39:55] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00295 [I 2022-10-11 09:39:55] ...trap/engines/engine.py.230: loss: 0.14875 [I 2022-10-11 09:39:55] ...optimizers/trijoint.py.151: total_norm: 30.660446 [I 2022-10-11 09:40:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 1190/2304 [I 2022-10-11 09:40:04] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:15:34 [I 2022-10-11 09:40:04] ...trap/engines/engine.py.229: process: 0.83940 | load: 0.00295 [I 2022-10-11 09:40:04] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-11 09:40:04] ...optimizers/trijoint.py.151: total_norm: 10.086390 [I 2022-10-11 09:40:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1200/2304 [I 2022-10-11 09:40:12] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:19 [I 2022-10-11 09:40:12] ...trap/engines/engine.py.229: process: 0.83977 | load: 0.00295 [I 2022-10-11 09:40:12] ...trap/engines/engine.py.230: loss: 0.09881 [I 2022-10-11 09:40:12] ...optimizers/trijoint.py.151: total_norm: 14.422537 [I 2022-10-11 09:40:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 1210/2304 [I 2022-10-11 09:40:21] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:15 [I 2022-10-11 09:40:21] ...trap/engines/engine.py.229: process: 0.83508 | load: 0.00292 [I 2022-10-11 09:40:21] ...trap/engines/engine.py.230: loss: 0.13495 [I 2022-10-11 09:40:21] ...optimizers/trijoint.py.151: total_norm: 11.110322 [I 2022-10-11 09:40:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1220/2304 [I 2022-10-11 09:40:29] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:07 [I 2022-10-11 09:40:29] ...trap/engines/engine.py.229: process: 0.82743 | load: 0.00293 [I 2022-10-11 09:40:29] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-11 09:40:29] ...optimizers/trijoint.py.151: total_norm: 11.039327 [I 2022-10-11 09:40:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 1230/2304 [I 2022-10-11 09:40:37] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:14:56 [I 2022-10-11 09:40:37] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00294 [I 2022-10-11 09:40:37] ...trap/engines/engine.py.230: loss: 0.09446 [I 2022-10-11 09:40:37] ...optimizers/trijoint.py.151: total_norm: 14.092843 [I 2022-10-11 09:40:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 1240/2304 [I 2022-10-11 09:40:46] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:14:55 [I 2022-10-11 09:40:46] ...trap/engines/engine.py.229: process: 0.83923 | load: 0.00292 [I 2022-10-11 09:40:46] ...trap/engines/engine.py.230: loss: 0.08166 [I 2022-10-11 09:40:46] ...optimizers/trijoint.py.151: total_norm: 9.657953 [I 2022-10-11 09:40:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 1250/2304 [I 2022-10-11 09:40:54] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:14:48 [I 2022-10-11 09:40:54] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00294 [I 2022-10-11 09:40:54] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-11 09:40:54] ...optimizers/trijoint.py.151: total_norm: 11.824986 [I 2022-10-11 09:41:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 1260/2304 [I 2022-10-11 09:41:03] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:47 [I 2022-10-11 09:41:03] ...trap/engines/engine.py.229: process: 0.87084 | load: 0.00296 [I 2022-10-11 09:41:03] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-11 09:41:03] ...optimizers/trijoint.py.151: total_norm: 18.485914 [I 2022-10-11 09:41:11] ...trap/engines/engine.py.224: train: epoch 100 | batch 1270/2304 [I 2022-10-11 09:41:11] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:53 [I 2022-10-11 09:41:11] ...trap/engines/engine.py.229: process: 0.88001 | load: 0.00375 [I 2022-10-11 09:41:11] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-11 09:41:11] ...optimizers/trijoint.py.151: total_norm: 11.167659 [I 2022-10-11 09:41:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 1280/2304 [I 2022-10-11 09:41:20] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:52 [I 2022-10-11 09:41:20] ...trap/engines/engine.py.229: process: 0.86413 | load: 0.00368 [I 2022-10-11 09:41:20] ...trap/engines/engine.py.230: loss: 0.16977 [I 2022-10-11 09:41:20] ...optimizers/trijoint.py.151: total_norm: 50.614468 [I 2022-10-11 09:41:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1290/2304 [I 2022-10-11 09:41:29] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:18 [I 2022-10-11 09:41:29] ...trap/engines/engine.py.229: process: 0.82979 | load: 0.00293 [I 2022-10-11 09:41:29] ...trap/engines/engine.py.230: loss: 0.23116 [I 2022-10-11 09:41:29] ...optimizers/trijoint.py.151: total_norm: 37.975906 [I 2022-10-11 09:41:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 1300/2304 [I 2022-10-11 09:41:37] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:00 [I 2022-10-11 09:41:37] ...trap/engines/engine.py.229: process: 0.83268 | load: 0.00294 [I 2022-10-11 09:41:37] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-11 09:41:37] ...optimizers/trijoint.py.151: total_norm: 21.044451 [I 2022-10-11 09:41:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 1310/2304 [I 2022-10-11 09:41:45] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:13:52 [I 2022-10-11 09:41:45] ...trap/engines/engine.py.229: process: 0.83755 | load: 0.00293 [I 2022-10-11 09:41:45] ...trap/engines/engine.py.230: loss: 0.15787 [I 2022-10-11 09:41:45] ...optimizers/trijoint.py.151: total_norm: 24.934616 [I 2022-10-11 09:41:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 1320/2304 [I 2022-10-11 09:41:54] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:13:40 [I 2022-10-11 09:41:54] ...trap/engines/engine.py.229: process: 0.81536 | load: 0.00292 [I 2022-10-11 09:41:54] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-11 09:41:54] ...optimizers/trijoint.py.151: total_norm: 12.883521 [I 2022-10-11 09:42:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 1330/2304 [I 2022-10-11 09:42:02] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:13:35 [I 2022-10-11 09:42:02] ...trap/engines/engine.py.229: process: 0.83196 | load: 0.00293 [I 2022-10-11 09:42:02] ...trap/engines/engine.py.230: loss: 0.08800 [I 2022-10-11 09:42:02] ...optimizers/trijoint.py.151: total_norm: 11.775737 [I 2022-10-11 09:42:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 1340/2304 [I 2022-10-11 09:42:10] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:26 [I 2022-10-11 09:42:10] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00295 [I 2022-10-11 09:42:10] ...trap/engines/engine.py.230: loss: 0.16627 [I 2022-10-11 09:42:10] ...optimizers/trijoint.py.151: total_norm: 43.973263 [I 2022-10-11 09:42:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 1350/2304 [I 2022-10-11 09:42:19] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:17 [I 2022-10-11 09:42:19] ...trap/engines/engine.py.229: process: 0.83820 | load: 0.00293 [I 2022-10-11 09:42:19] ...trap/engines/engine.py.230: loss: 0.08691 [I 2022-10-11 09:42:19] ...optimizers/trijoint.py.151: total_norm: 12.085166 [I 2022-10-11 09:42:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 1360/2304 [I 2022-10-11 09:42:27] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:10 [I 2022-10-11 09:42:27] ...trap/engines/engine.py.229: process: 0.84808 | load: 0.00293 [I 2022-10-11 09:42:27] ...trap/engines/engine.py.230: loss: 0.09796 [I 2022-10-11 09:42:27] ...optimizers/trijoint.py.151: total_norm: 11.510442 [I 2022-10-11 09:42:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 1370/2304 [I 2022-10-11 09:42:36] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:12:59 [I 2022-10-11 09:42:36] ...trap/engines/engine.py.229: process: 0.82728 | load: 0.00294 [I 2022-10-11 09:42:36] ...trap/engines/engine.py.230: loss: 0.13711 [I 2022-10-11 09:42:36] ...optimizers/trijoint.py.151: total_norm: 13.094631 [I 2022-10-11 09:42:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 1380/2304 [I 2022-10-11 09:42:44] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:12:46 [I 2022-10-11 09:42:44] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00295 [I 2022-10-11 09:42:44] ...trap/engines/engine.py.230: loss: 0.10596 [I 2022-10-11 09:42:44] ...optimizers/trijoint.py.151: total_norm: 10.185792 [I 2022-10-11 09:42:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 1390/2304 [I 2022-10-11 09:42:52] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:45 [I 2022-10-11 09:42:52] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00295 [I 2022-10-11 09:42:52] ...trap/engines/engine.py.230: loss: 0.14388 [I 2022-10-11 09:42:52] ...optimizers/trijoint.py.151: total_norm: 27.341118 [I 2022-10-11 09:43:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 1400/2304 [I 2022-10-11 09:43:01] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:38 [I 2022-10-11 09:43:01] ...trap/engines/engine.py.229: process: 0.85520 | load: 0.00293 [I 2022-10-11 09:43:01] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 09:43:01] ...optimizers/trijoint.py.151: total_norm: 15.900857 [I 2022-10-11 09:43:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 1410/2304 [I 2022-10-11 09:43:09] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:33 [I 2022-10-11 09:43:09] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00296 [I 2022-10-11 09:43:09] ...trap/engines/engine.py.230: loss: 0.15494 [I 2022-10-11 09:43:09] ...optimizers/trijoint.py.151: total_norm: 11.274408 [I 2022-10-11 09:43:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 1420/2304 [I 2022-10-11 09:43:17] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:26 [I 2022-10-11 09:43:17] ...trap/engines/engine.py.229: process: 0.84739 | load: 0.00293 [I 2022-10-11 09:43:17] ...trap/engines/engine.py.230: loss: 0.09845 [I 2022-10-11 09:43:17] ...optimizers/trijoint.py.151: total_norm: 25.314034 [I 2022-10-11 09:43:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 1430/2304 [I 2022-10-11 09:43:26] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:16 [I 2022-10-11 09:43:26] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00291 [I 2022-10-11 09:43:26] ...trap/engines/engine.py.230: loss: 0.10637 [I 2022-10-11 09:43:26] ...optimizers/trijoint.py.151: total_norm: 11.878704 [I 2022-10-11 09:43:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 1440/2304 [I 2022-10-11 09:43:34] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:09 [I 2022-10-11 09:43:34] ...trap/engines/engine.py.229: process: 0.82420 | load: 0.00294 [I 2022-10-11 09:43:34] ...trap/engines/engine.py.230: loss: 0.08356 [I 2022-10-11 09:43:34] ...optimizers/trijoint.py.151: total_norm: 15.004289 [I 2022-10-11 09:43:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 1450/2304 [I 2022-10-11 09:43:43] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:11:55 [I 2022-10-11 09:43:43] ...trap/engines/engine.py.229: process: 0.82340 | load: 0.00294 [I 2022-10-11 09:43:43] ...trap/engines/engine.py.230: loss: 0.11608 [I 2022-10-11 09:43:43] ...optimizers/trijoint.py.151: total_norm: 20.418938 [I 2022-10-11 09:43:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 1460/2304 [I 2022-10-11 09:43:51] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:11:45 [I 2022-10-11 09:43:51] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00295 [I 2022-10-11 09:43:51] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-11 09:43:51] ...optimizers/trijoint.py.151: total_norm: 8.014838 [I 2022-10-11 09:43:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 1470/2304 [I 2022-10-11 09:43:59] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:11:38 [I 2022-10-11 09:43:59] ...trap/engines/engine.py.229: process: 0.84690 | load: 0.00293 [I 2022-10-11 09:43:59] ...trap/engines/engine.py.230: loss: 0.13655 [I 2022-10-11 09:43:59] ...optimizers/trijoint.py.151: total_norm: 11.545321 [I 2022-10-11 09:44:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 1480/2304 [I 2022-10-11 09:44:08] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:32 [I 2022-10-11 09:44:08] ...trap/engines/engine.py.229: process: 0.82211 | load: 0.00293 [I 2022-10-11 09:44:08] ...trap/engines/engine.py.230: loss: 0.15387 [I 2022-10-11 09:44:08] ...optimizers/trijoint.py.151: total_norm: 32.523476 [I 2022-10-11 09:44:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 1490/2304 [I 2022-10-11 09:44:16] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:18 [I 2022-10-11 09:44:16] ...trap/engines/engine.py.229: process: 0.81899 | load: 0.00293 [I 2022-10-11 09:44:16] ...trap/engines/engine.py.230: loss: 0.13245 [I 2022-10-11 09:44:16] ...optimizers/trijoint.py.151: total_norm: 15.119677 [I 2022-10-11 09:44:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 1500/2304 [I 2022-10-11 09:44:25] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:15 [I 2022-10-11 09:44:25] ...trap/engines/engine.py.229: process: 0.86119 | load: 0.00294 [I 2022-10-11 09:44:25] ...trap/engines/engine.py.230: loss: 0.10569 [I 2022-10-11 09:44:25] ...optimizers/trijoint.py.151: total_norm: 8.918539 [I 2022-10-11 09:44:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 1510/2304 [I 2022-10-11 09:44:33] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:05 [I 2022-10-11 09:44:33] ...trap/engines/engine.py.229: process: 0.83104 | load: 0.00293 [I 2022-10-11 09:44:33] ...trap/engines/engine.py.230: loss: 0.14146 [I 2022-10-11 09:44:33] ...optimizers/trijoint.py.151: total_norm: 24.471434 [I 2022-10-11 09:44:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 1520/2304 [I 2022-10-11 09:44:41] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:10:59 [I 2022-10-11 09:44:41] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00294 [I 2022-10-11 09:44:41] ...trap/engines/engine.py.230: loss: 0.23569 [I 2022-10-11 09:44:41] ...optimizers/trijoint.py.151: total_norm: 22.858858 [I 2022-10-11 09:44:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 1530/2304 [I 2022-10-11 09:44:50] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:10:53 [I 2022-10-11 09:44:50] ...trap/engines/engine.py.229: process: 0.87943 | load: 0.00294 [I 2022-10-11 09:44:50] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-11 09:44:50] ...optimizers/trijoint.py.151: total_norm: 9.685013 [I 2022-10-11 09:44:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 1540/2304 [I 2022-10-11 09:44:58] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:10:38 [I 2022-10-11 09:44:58] ...trap/engines/engine.py.229: process: 0.82691 | load: 0.00295 [I 2022-10-11 09:44:58] ...trap/engines/engine.py.230: loss: 0.12989 [I 2022-10-11 09:44:58] ...optimizers/trijoint.py.151: total_norm: 11.000648 [I 2022-10-11 09:45:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 1550/2304 [I 2022-10-11 09:45:07] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:10:34 [I 2022-10-11 09:45:07] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00296 [I 2022-10-11 09:45:07] ...trap/engines/engine.py.230: loss: 0.18876 [I 2022-10-11 09:45:07] ...optimizers/trijoint.py.151: total_norm: 12.309053 [I 2022-10-11 09:45:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 1560/2304 [I 2022-10-11 09:45:15] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:24 [I 2022-10-11 09:45:15] ...trap/engines/engine.py.229: process: 0.85734 | load: 0.00293 [I 2022-10-11 09:45:15] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-11 09:45:15] ...optimizers/trijoint.py.151: total_norm: 10.786796 [I 2022-10-11 09:45:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 1570/2304 [I 2022-10-11 09:45:23] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:11 [I 2022-10-11 09:45:23] ...trap/engines/engine.py.229: process: 0.82788 | load: 0.00294 [I 2022-10-11 09:45:23] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-11 09:45:23] ...optimizers/trijoint.py.151: total_norm: 18.294754 [I 2022-10-11 09:45:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 1580/2304 [I 2022-10-11 09:45:32] ...trap/engines/engine.py.228: elapsed: 0:22:17 | left: 0:10:03 [I 2022-10-11 09:45:32] ...trap/engines/engine.py.229: process: 0.82901 | load: 0.00292 [I 2022-10-11 09:45:32] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-11 09:45:32] ...optimizers/trijoint.py.151: total_norm: 24.685923 [I 2022-10-11 09:45:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 1590/2304 [I 2022-10-11 09:45:40] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:00 [I 2022-10-11 09:45:40] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00294 [I 2022-10-11 09:45:40] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-11 09:45:40] ...optimizers/trijoint.py.151: total_norm: 11.213355 [I 2022-10-11 09:45:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 1600/2304 [I 2022-10-11 09:45:49] ...trap/engines/engine.py.228: elapsed: 0:22:34 | left: 0:09:50 [I 2022-10-11 09:45:49] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00291 [I 2022-10-11 09:45:49] ...trap/engines/engine.py.230: loss: 0.10860 [I 2022-10-11 09:45:49] ...optimizers/trijoint.py.151: total_norm: 16.940716 [I 2022-10-11 09:45:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 1610/2304 [I 2022-10-11 09:45:57] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:09:37 [I 2022-10-11 09:45:57] ...trap/engines/engine.py.229: process: 0.82749 | load: 0.00293 [I 2022-10-11 09:45:57] ...trap/engines/engine.py.230: loss: 0.08571 [I 2022-10-11 09:45:57] ...optimizers/trijoint.py.151: total_norm: 11.884026 [I 2022-10-11 09:46:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 1620/2304 [I 2022-10-11 09:46:05] ...trap/engines/engine.py.228: elapsed: 0:22:51 | left: 0:09:30 [I 2022-10-11 09:46:05] ...trap/engines/engine.py.229: process: 0.82881 | load: 0.00294 [I 2022-10-11 09:46:05] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-11 09:46:05] ...optimizers/trijoint.py.151: total_norm: 66.709862 [I 2022-10-11 09:46:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 1630/2304 [I 2022-10-11 09:46:14] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:09:21 [I 2022-10-11 09:46:14] ...trap/engines/engine.py.229: process: 0.83244 | load: 0.00291 [I 2022-10-11 09:46:14] ...trap/engines/engine.py.230: loss: 0.09903 [I 2022-10-11 09:46:14] ...optimizers/trijoint.py.151: total_norm: 11.477018 [I 2022-10-11 09:46:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 1640/2304 [I 2022-10-11 09:46:22] ...trap/engines/engine.py.228: elapsed: 0:23:08 | left: 0:09:25 [I 2022-10-11 09:46:22] ...trap/engines/engine.py.229: process: 0.86837 | load: 0.00293 [I 2022-10-11 09:46:22] ...trap/engines/engine.py.230: loss: 0.12281 [I 2022-10-11 09:46:22] ...optimizers/trijoint.py.151: total_norm: 11.932427 [I 2022-10-11 09:46:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 1650/2304 [I 2022-10-11 09:46:31] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:10 [I 2022-10-11 09:46:31] ...trap/engines/engine.py.229: process: 0.83851 | load: 0.00294 [I 2022-10-11 09:46:31] ...trap/engines/engine.py.230: loss: 0.08194 [I 2022-10-11 09:46:31] ...optimizers/trijoint.py.151: total_norm: 12.955255 [I 2022-10-11 09:46:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 1660/2304 [I 2022-10-11 09:46:39] ...trap/engines/engine.py.228: elapsed: 0:23:25 | left: 0:09:04 [I 2022-10-11 09:46:39] ...trap/engines/engine.py.229: process: 0.85211 | load: 0.00295 [I 2022-10-11 09:46:39] ...trap/engines/engine.py.230: loss: 0.11594 [I 2022-10-11 09:46:39] ...optimizers/trijoint.py.151: total_norm: 57.206112 [I 2022-10-11 09:46:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1670/2304 [I 2022-10-11 09:46:47] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:08:51 [I 2022-10-11 09:46:47] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00294 [I 2022-10-11 09:46:47] ...trap/engines/engine.py.230: loss: 0.14623 [I 2022-10-11 09:46:47] ...optimizers/trijoint.py.151: total_norm: 13.292893 [I 2022-10-11 09:46:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 1680/2304 [I 2022-10-11 09:46:56] ...trap/engines/engine.py.228: elapsed: 0:23:42 | left: 0:08:47 [I 2022-10-11 09:46:56] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00293 [I 2022-10-11 09:46:56] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-11 09:46:56] ...optimizers/trijoint.py.151: total_norm: 47.157223 [I 2022-10-11 09:47:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 1690/2304 [I 2022-10-11 09:47:04] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:08:38 [I 2022-10-11 09:47:04] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00295 [I 2022-10-11 09:47:04] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-11 09:47:04] ...optimizers/trijoint.py.151: total_norm: 13.801859 [I 2022-10-11 09:47:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 1700/2304 [I 2022-10-11 09:47:13] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:08:31 [I 2022-10-11 09:47:13] ...trap/engines/engine.py.229: process: 0.83394 | load: 0.00291 [I 2022-10-11 09:47:13] ...trap/engines/engine.py.230: loss: 0.19410 [I 2022-10-11 09:47:13] ...optimizers/trijoint.py.151: total_norm: 10.144910 [I 2022-10-11 09:47:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 1710/2304 [I 2022-10-11 09:47:21] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:22 [I 2022-10-11 09:47:21] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00292 [I 2022-10-11 09:47:21] ...trap/engines/engine.py.230: loss: 0.10078 [I 2022-10-11 09:47:21] ...optimizers/trijoint.py.151: total_norm: 17.207186 [I 2022-10-11 09:47:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 1720/2304 [I 2022-10-11 09:47:30] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:13 [I 2022-10-11 09:47:30] ...trap/engines/engine.py.229: process: 0.83404 | load: 0.00292 [I 2022-10-11 09:47:30] ...trap/engines/engine.py.230: loss: 0.15470 [I 2022-10-11 09:47:30] ...optimizers/trijoint.py.151: total_norm: 17.698929 [I 2022-10-11 09:47:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 1730/2304 [I 2022-10-11 09:47:38] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:02 [I 2022-10-11 09:47:38] ...trap/engines/engine.py.229: process: 0.82568 | load: 0.00295 [I 2022-10-11 09:47:38] ...trap/engines/engine.py.230: loss: 0.17273 [I 2022-10-11 09:47:38] ...optimizers/trijoint.py.151: total_norm: 51.986633 [I 2022-10-11 09:47:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1740/2304 [I 2022-10-11 09:47:47] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:07:54 [I 2022-10-11 09:47:47] ...trap/engines/engine.py.229: process: 0.84036 | load: 0.00295 [I 2022-10-11 09:47:47] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-11 09:47:47] ...optimizers/trijoint.py.151: total_norm: 7.808929 [I 2022-10-11 09:47:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 1750/2304 [I 2022-10-11 09:47:55] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:07:43 [I 2022-10-11 09:47:55] ...trap/engines/engine.py.229: process: 0.83189 | load: 0.00293 [I 2022-10-11 09:47:55] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-11 09:47:55] ...optimizers/trijoint.py.151: total_norm: 15.161040 [I 2022-10-11 09:48:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 1760/2304 [I 2022-10-11 09:48:03] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:07:39 [I 2022-10-11 09:48:03] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00290 [I 2022-10-11 09:48:03] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-11 09:48:03] ...optimizers/trijoint.py.151: total_norm: 20.126379 [I 2022-10-11 09:48:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1770/2304 [I 2022-10-11 09:48:12] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:07:29 [I 2022-10-11 09:48:12] ...trap/engines/engine.py.229: process: 0.82990 | load: 0.00293 [I 2022-10-11 09:48:12] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-11 09:48:12] ...optimizers/trijoint.py.151: total_norm: 18.016693 [I 2022-10-11 09:48:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 1780/2304 [I 2022-10-11 09:48:20] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:21 [I 2022-10-11 09:48:20] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00293 [I 2022-10-11 09:48:20] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-11 09:48:20] ...optimizers/trijoint.py.151: total_norm: 11.328743 [I 2022-10-11 09:48:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1790/2304 [I 2022-10-11 09:48:29] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:09 [I 2022-10-11 09:48:29] ...trap/engines/engine.py.229: process: 0.82307 | load: 0.00293 [I 2022-10-11 09:48:29] ...trap/engines/engine.py.230: loss: 0.10336 [I 2022-10-11 09:48:29] ...optimizers/trijoint.py.151: total_norm: 6.568822 [I 2022-10-11 09:48:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 1800/2304 [I 2022-10-11 09:48:37] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:04 [I 2022-10-11 09:48:37] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00292 [I 2022-10-11 09:48:37] ...trap/engines/engine.py.230: loss: 0.13107 [I 2022-10-11 09:48:37] ...optimizers/trijoint.py.151: total_norm: 39.327824 [I 2022-10-11 09:48:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 1810/2304 [I 2022-10-11 09:48:46] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:03 [I 2022-10-11 09:48:46] ...trap/engines/engine.py.229: process: 0.88034 | load: 0.00373 [I 2022-10-11 09:48:46] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-11 09:48:46] ...optimizers/trijoint.py.151: total_norm: 15.174343 [I 2022-10-11 09:48:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 1820/2304 [I 2022-10-11 09:48:54] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:06:59 [I 2022-10-11 09:48:54] ...trap/engines/engine.py.229: process: 0.86961 | load: 0.00373 [I 2022-10-11 09:48:54] ...trap/engines/engine.py.230: loss: 0.19775 [I 2022-10-11 09:48:54] ...optimizers/trijoint.py.151: total_norm: 35.873383 [I 2022-10-11 09:49:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 1830/2304 [I 2022-10-11 09:49:03] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:06:52 [I 2022-10-11 09:49:03] ...trap/engines/engine.py.229: process: 0.87194 | load: 0.00371 [I 2022-10-11 09:49:03] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-11 09:49:03] ...optimizers/trijoint.py.151: total_norm: 15.573270 [I 2022-10-11 09:49:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1840/2304 [I 2022-10-11 09:49:12] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:06:41 [I 2022-10-11 09:49:12] ...trap/engines/engine.py.229: process: 0.87072 | load: 0.00375 [I 2022-10-11 09:49:12] ...trap/engines/engine.py.230: loss: 0.10597 [I 2022-10-11 09:49:12] ...optimizers/trijoint.py.151: total_norm: 11.698244 [I 2022-10-11 09:49:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 1850/2304 [I 2022-10-11 09:49:20] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:33 [I 2022-10-11 09:49:20] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00374 [I 2022-10-11 09:49:20] ...trap/engines/engine.py.230: loss: 0.13283 [I 2022-10-11 09:49:20] ...optimizers/trijoint.py.151: total_norm: 19.666334 [I 2022-10-11 09:49:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1860/2304 [I 2022-10-11 09:49:29] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:25 [I 2022-10-11 09:49:29] ...trap/engines/engine.py.229: process: 0.88047 | load: 0.00370 [I 2022-10-11 09:49:29] ...trap/engines/engine.py.230: loss: 0.11695 [I 2022-10-11 09:49:29] ...optimizers/trijoint.py.151: total_norm: 8.241720 [I 2022-10-11 09:49:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 1870/2304 [I 2022-10-11 09:49:38] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:17 [I 2022-10-11 09:49:38] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00374 [I 2022-10-11 09:49:38] ...trap/engines/engine.py.230: loss: 0.11224 [I 2022-10-11 09:49:38] ...optimizers/trijoint.py.151: total_norm: 8.974895 [I 2022-10-11 09:49:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 1880/2304 [I 2022-10-11 09:49:46] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:07 [I 2022-10-11 09:49:46] ...trap/engines/engine.py.229: process: 0.85877 | load: 0.00372 [I 2022-10-11 09:49:46] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-11 09:49:46] ...optimizers/trijoint.py.151: total_norm: 13.586839 [I 2022-10-11 09:49:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 1890/2304 [I 2022-10-11 09:49:55] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:05:58 [I 2022-10-11 09:49:55] ...trap/engines/engine.py.229: process: 0.85983 | load: 0.00370 [I 2022-10-11 09:49:55] ...trap/engines/engine.py.230: loss: 0.13213 [I 2022-10-11 09:49:55] ...optimizers/trijoint.py.151: total_norm: 72.910149 [I 2022-10-11 09:50:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 1900/2304 [I 2022-10-11 09:50:04] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:05:50 [I 2022-10-11 09:50:04] ...trap/engines/engine.py.229: process: 0.86738 | load: 0.00374 [I 2022-10-11 09:50:04] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-11 09:50:04] ...optimizers/trijoint.py.151: total_norm: 15.386927 [I 2022-10-11 09:50:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1910/2304 [I 2022-10-11 09:50:12] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:05:41 [I 2022-10-11 09:50:12] ...trap/engines/engine.py.229: process: 0.86141 | load: 0.00369 [I 2022-10-11 09:50:12] ...trap/engines/engine.py.230: loss: 0.11090 [I 2022-10-11 09:50:12] ...optimizers/trijoint.py.151: total_norm: 10.403405 [I 2022-10-11 09:50:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 1920/2304 [I 2022-10-11 09:50:21] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:05:31 [I 2022-10-11 09:50:21] ...trap/engines/engine.py.229: process: 0.86439 | load: 0.00373 [I 2022-10-11 09:50:21] ...trap/engines/engine.py.230: loss: 0.11718 [I 2022-10-11 09:50:21] ...optimizers/trijoint.py.151: total_norm: 15.862468 [I 2022-10-11 09:50:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 1930/2304 [I 2022-10-11 09:50:30] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:22 [I 2022-10-11 09:50:30] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00371 [I 2022-10-11 09:50:30] ...trap/engines/engine.py.230: loss: 0.09360 [I 2022-10-11 09:50:30] ...optimizers/trijoint.py.151: total_norm: 19.591728 [I 2022-10-11 09:50:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 1940/2304 [I 2022-10-11 09:50:38] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:15 [I 2022-10-11 09:50:38] ...trap/engines/engine.py.229: process: 0.87067 | load: 0.00374 [I 2022-10-11 09:50:38] ...trap/engines/engine.py.230: loss: 0.13629 [I 2022-10-11 09:50:38] ...optimizers/trijoint.py.151: total_norm: 13.100998 [I 2022-10-11 09:50:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1950/2304 [I 2022-10-11 09:50:47] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:06 [I 2022-10-11 09:50:47] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00373 [I 2022-10-11 09:50:47] ...trap/engines/engine.py.230: loss: 0.13532 [I 2022-10-11 09:50:47] ...optimizers/trijoint.py.151: total_norm: 17.699554 [I 2022-10-11 09:50:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 1960/2304 [I 2022-10-11 09:50:56] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:04:57 [I 2022-10-11 09:50:56] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00375 [I 2022-10-11 09:50:56] ...trap/engines/engine.py.230: loss: 0.11535 [I 2022-10-11 09:50:56] ...optimizers/trijoint.py.151: total_norm: 15.761585 [I 2022-10-11 09:51:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 1970/2304 [I 2022-10-11 09:51:05] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:50 [I 2022-10-11 09:51:05] ...trap/engines/engine.py.229: process: 0.86496 | load: 0.00374 [I 2022-10-11 09:51:05] ...trap/engines/engine.py.230: loss: 0.13062 [I 2022-10-11 09:51:05] ...optimizers/trijoint.py.151: total_norm: 19.025728 [I 2022-10-11 09:51:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 1980/2304 [I 2022-10-11 09:51:13] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:40 [I 2022-10-11 09:51:13] ...trap/engines/engine.py.229: process: 0.86205 | load: 0.00374 [I 2022-10-11 09:51:13] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-11 09:51:13] ...optimizers/trijoint.py.151: total_norm: 17.470358 [I 2022-10-11 09:51:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 1990/2304 [I 2022-10-11 09:51:22] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:31 [I 2022-10-11 09:51:22] ...trap/engines/engine.py.229: process: 0.85970 | load: 0.00371 [I 2022-10-11 09:51:22] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-11 09:51:22] ...optimizers/trijoint.py.151: total_norm: 20.706217 [I 2022-10-11 09:51:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 2000/2304 [I 2022-10-11 09:51:31] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:23 [I 2022-10-11 09:51:31] ...trap/engines/engine.py.229: process: 0.86706 | load: 0.00370 [I 2022-10-11 09:51:31] ...trap/engines/engine.py.230: loss: 0.12670 [I 2022-10-11 09:51:31] ...optimizers/trijoint.py.151: total_norm: 18.795019 [I 2022-10-11 09:51:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 2010/2304 [I 2022-10-11 09:51:39] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:14 [I 2022-10-11 09:51:39] ...trap/engines/engine.py.229: process: 0.86131 | load: 0.00375 [I 2022-10-11 09:51:39] ...trap/engines/engine.py.230: loss: 0.14014 [I 2022-10-11 09:51:39] ...optimizers/trijoint.py.151: total_norm: 13.196211 [I 2022-10-11 09:51:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 2020/2304 [I 2022-10-11 09:51:48] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:06 [I 2022-10-11 09:51:48] ...trap/engines/engine.py.229: process: 0.88053 | load: 0.00372 [I 2022-10-11 09:51:48] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-11 09:51:48] ...optimizers/trijoint.py.151: total_norm: 12.052933 [I 2022-10-11 09:51:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 2030/2304 [I 2022-10-11 09:51:57] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:03:57 [I 2022-10-11 09:51:57] ...trap/engines/engine.py.229: process: 0.87277 | load: 0.00372 [I 2022-10-11 09:51:57] ...trap/engines/engine.py.230: loss: 0.15503 [I 2022-10-11 09:51:57] ...optimizers/trijoint.py.151: total_norm: 10.525232 [I 2022-10-11 09:52:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 2040/2304 [I 2022-10-11 09:52:05] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:03:49 [I 2022-10-11 09:52:05] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00373 [I 2022-10-11 09:52:05] ...trap/engines/engine.py.230: loss: 0.08876 [I 2022-10-11 09:52:05] ...optimizers/trijoint.py.151: total_norm: 10.831471 [I 2022-10-11 09:52:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 2050/2304 [I 2022-10-11 09:52:14] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:03:38 [I 2022-10-11 09:52:14] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00372 [I 2022-10-11 09:52:14] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-11 09:52:14] ...optimizers/trijoint.py.151: total_norm: 20.160955 [I 2022-10-11 09:52:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 2060/2304 [I 2022-10-11 09:52:23] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:31 [I 2022-10-11 09:52:23] ...trap/engines/engine.py.229: process: 0.87600 | load: 0.00376 [I 2022-10-11 09:52:23] ...trap/engines/engine.py.230: loss: 0.17882 [I 2022-10-11 09:52:23] ...optimizers/trijoint.py.151: total_norm: 26.855433 [I 2022-10-11 09:52:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 2070/2304 [I 2022-10-11 09:52:35] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:04:18 [I 2022-10-11 09:52:35] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00371 [I 2022-10-11 09:52:35] ...trap/engines/engine.py.230: loss: 0.08919 [I 2022-10-11 09:52:35] ...optimizers/trijoint.py.151: total_norm: 11.902551 [I 2022-10-11 09:52:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 2080/2304 [I 2022-10-11 09:52:43] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:14 [I 2022-10-11 09:52:43] ...trap/engines/engine.py.229: process: 0.82944 | load: 0.00295 [I 2022-10-11 09:52:43] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-11 09:52:43] ...optimizers/trijoint.py.151: total_norm: 25.854528 [I 2022-10-11 09:52:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 2090/2304 [I 2022-10-11 09:52:52] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:02:59 [I 2022-10-11 09:52:52] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00298 [I 2022-10-11 09:52:52] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-11 09:52:52] ...optimizers/trijoint.py.151: total_norm: 13.720938 [I 2022-10-11 09:53:00] ...trap/engines/engine.py.224: train: epoch 100 | batch 2100/2304 [I 2022-10-11 09:53:00] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:02:49 [I 2022-10-11 09:53:00] ...trap/engines/engine.py.229: process: 0.83343 | load: 0.00293 [I 2022-10-11 09:53:00] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-11 09:53:00] ...optimizers/trijoint.py.151: total_norm: 10.752809 [I 2022-10-11 09:53:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 2110/2304 [I 2022-10-11 09:53:08] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:02:42 [I 2022-10-11 09:53:08] ...trap/engines/engine.py.229: process: 0.82893 | load: 0.00292 [I 2022-10-11 09:53:08] ...trap/engines/engine.py.230: loss: 0.13501 [I 2022-10-11 09:53:08] ...optimizers/trijoint.py.151: total_norm: 11.940646 [I 2022-10-11 09:53:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 2120/2304 [I 2022-10-11 09:53:17] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:33 [I 2022-10-11 09:53:17] ...trap/engines/engine.py.229: process: 0.82892 | load: 0.00293 [I 2022-10-11 09:53:17] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-11 09:53:17] ...optimizers/trijoint.py.151: total_norm: 14.982947 [I 2022-10-11 09:53:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 2130/2304 [I 2022-10-11 09:53:25] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:25 [I 2022-10-11 09:53:25] ...trap/engines/engine.py.229: process: 0.81686 | load: 0.00292 [I 2022-10-11 09:53:25] ...trap/engines/engine.py.230: loss: 0.10453 [I 2022-10-11 09:53:25] ...optimizers/trijoint.py.151: total_norm: 11.115452 [I 2022-10-11 09:53:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 2140/2304 [I 2022-10-11 09:53:34] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:17 [I 2022-10-11 09:53:34] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00293 [I 2022-10-11 09:53:34] ...trap/engines/engine.py.230: loss: 0.09939 [I 2022-10-11 09:53:34] ...optimizers/trijoint.py.151: total_norm: 9.429300 [I 2022-10-11 09:53:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 2150/2304 [I 2022-10-11 09:53:42] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:09 [I 2022-10-11 09:53:42] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00292 [I 2022-10-11 09:53:42] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-11 09:53:42] ...optimizers/trijoint.py.151: total_norm: 13.504545 [I 2022-10-11 09:53:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 2160/2304 [I 2022-10-11 09:53:50] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:01 [I 2022-10-11 09:53:50] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00294 [I 2022-10-11 09:53:50] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-11 09:53:50] ...optimizers/trijoint.py.151: total_norm: 13.748412 [I 2022-10-11 09:53:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 2170/2304 [I 2022-10-11 09:53:59] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:01:52 [I 2022-10-11 09:53:59] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00292 [I 2022-10-11 09:53:59] ...trap/engines/engine.py.230: loss: 0.10029 [I 2022-10-11 09:53:59] ...optimizers/trijoint.py.151: total_norm: 8.494878 [I 2022-10-11 09:54:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 2180/2304 [I 2022-10-11 09:54:07] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:01:43 [I 2022-10-11 09:54:07] ...trap/engines/engine.py.229: process: 0.83299 | load: 0.00294 [I 2022-10-11 09:54:07] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-11 09:54:07] ...optimizers/trijoint.py.151: total_norm: 11.205965 [I 2022-10-11 09:54:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 2190/2304 [I 2022-10-11 09:54:16] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:35 [I 2022-10-11 09:54:16] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00293 [I 2022-10-11 09:54:16] ...trap/engines/engine.py.230: loss: 0.08720 [I 2022-10-11 09:54:16] ...optimizers/trijoint.py.151: total_norm: 11.585104 [I 2022-10-11 09:54:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 2200/2304 [I 2022-10-11 09:54:24] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:01:27 [I 2022-10-11 09:54:24] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00294 [I 2022-10-11 09:54:24] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-11 09:54:24] ...optimizers/trijoint.py.151: total_norm: 18.852261 [I 2022-10-11 09:54:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 2210/2304 [I 2022-10-11 09:54:32] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:18 [I 2022-10-11 09:54:32] ...trap/engines/engine.py.229: process: 0.84394 | load: 0.00294 [I 2022-10-11 09:54:32] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-11 09:54:32] ...optimizers/trijoint.py.151: total_norm: 36.825592 [I 2022-10-11 09:54:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 2220/2304 [I 2022-10-11 09:54:41] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:10 [I 2022-10-11 09:54:41] ...trap/engines/engine.py.229: process: 0.83972 | load: 0.00293 [I 2022-10-11 09:54:41] ...trap/engines/engine.py.230: loss: 0.06533 [I 2022-10-11 09:54:41] ...optimizers/trijoint.py.151: total_norm: 12.188129 [I 2022-10-11 09:54:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 2230/2304 [I 2022-10-11 09:54:49] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:02 [I 2022-10-11 09:54:49] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00295 [I 2022-10-11 09:54:49] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-11 09:54:49] ...optimizers/trijoint.py.151: total_norm: 11.148486 [I 2022-10-11 09:54:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 2240/2304 [I 2022-10-11 09:54:58] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:00:53 [I 2022-10-11 09:54:58] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00297 [I 2022-10-11 09:54:58] ...trap/engines/engine.py.230: loss: 0.11559 [I 2022-10-11 09:54:58] ...optimizers/trijoint.py.151: total_norm: 13.953396 [I 2022-10-11 09:55:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 2250/2304 [I 2022-10-11 09:55:06] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:00:45 [I 2022-10-11 09:55:06] ...trap/engines/engine.py.229: process: 0.83753 | load: 0.00294 [I 2022-10-11 09:55:06] ...trap/engines/engine.py.230: loss: 0.14822 [I 2022-10-11 09:55:06] ...optimizers/trijoint.py.151: total_norm: 21.096031 [I 2022-10-11 09:55:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 2260/2304 [I 2022-10-11 09:55:15] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:37 [I 2022-10-11 09:55:15] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00295 [I 2022-10-11 09:55:15] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-11 09:55:15] ...optimizers/trijoint.py.151: total_norm: 23.162884 [I 2022-10-11 09:55:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 2270/2304 [I 2022-10-11 09:55:23] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:28 [I 2022-10-11 09:55:23] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00294 [I 2022-10-11 09:55:23] ...trap/engines/engine.py.230: loss: 0.14861 [I 2022-10-11 09:55:23] ...optimizers/trijoint.py.151: total_norm: 12.610063 [I 2022-10-11 09:55:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 2280/2304 [I 2022-10-11 09:55:32] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:20 [I 2022-10-11 09:55:32] ...trap/engines/engine.py.229: process: 0.85490 | load: 0.00292 [I 2022-10-11 09:55:32] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-11 09:55:32] ...optimizers/trijoint.py.151: total_norm: 20.588835 [I 2022-10-11 09:55:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 2290/2304 [I 2022-10-11 09:55:40] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:11 [I 2022-10-11 09:55:40] ...trap/engines/engine.py.229: process: 0.83810 | load: 0.00294 [I 2022-10-11 09:55:40] ...trap/engines/engine.py.230: loss: 0.08535 [I 2022-10-11 09:55:40] ...optimizers/trijoint.py.151: total_norm: 9.727853 [I 2022-10-11 09:55:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 2300/2304 [I 2022-10-11 09:55:49] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:03 [I 2022-10-11 09:55:49] ...trap/engines/engine.py.229: process: 0.81435 | load: 0.00012 [I 2022-10-11 09:55:49] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-11 09:55:49] ...optimizers/trijoint.py.151: total_norm: 14.537544 [I 2022-10-11 09:55:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 2304/2304 [I 2022-10-11 09:55:52] ...trap/engines/engine.py.228: elapsed: 0:32:38 | left: 0:00:00 [I 2022-10-11 09:55:52] ...trap/engines/engine.py.229: process: 0.82355 | load: 0.00011 [I 2022-10-11 09:55:52] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-11 09:55:52] ...optimizers/trijoint.py.151: total_norm: 13.033832 [S 2022-10-11 09:55:53] ...trap/engines/engine.py.129: train_epoch.epoch: 100 [S 2022-10-11 09:55:53] ...trap/engines/engine.py.129: train_epoch.loss: 0.12275121363926347 [I 2022-10-11 09:55:53] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 09:55:55] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 100 [I 2022-10-11 09:55:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 09:55:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 09:55:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 09:55:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 09:56:00] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 09:56:00] ...trap/engines/engine.py.321: eval: epoch 100 | batch 0/510 [I 2022-10-11 09:56:00] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:14 [I 2022-10-11 09:56:00] ...trap/engines/engine.py.326: process: 5.67627 | load: 3.19226 [I 2022-10-11 09:56:06] ...trap/engines/engine.py.321: eval: epoch 100 | batch 10/510 [I 2022-10-11 09:56:06] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:10 [I 2022-10-11 09:56:06] ...trap/engines/engine.py.326: process: 0.53759 | load: 0.00018 [I 2022-10-11 09:56:11] ...trap/engines/engine.py.321: eval: epoch 100 | batch 20/510 [I 2022-10-11 09:56:11] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:29 [I 2022-10-11 09:56:11] ...trap/engines/engine.py.326: process: 0.47973 | load: 0.00018 [I 2022-10-11 09:56:15] ...trap/engines/engine.py.321: eval: epoch 100 | batch 30/510 [I 2022-10-11 09:56:15] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:26 [I 2022-10-11 09:56:15] ...trap/engines/engine.py.326: process: 0.41706 | load: 0.00018 [I 2022-10-11 09:56:19] ...trap/engines/engine.py.321: eval: epoch 100 | batch 40/510 [I 2022-10-11 09:56:19] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:32 [I 2022-10-11 09:56:19] ...trap/engines/engine.py.326: process: 0.45743 | load: 0.00019 [I 2022-10-11 09:56:26] ...trap/engines/engine.py.321: eval: epoch 100 | batch 50/510 [I 2022-10-11 09:56:26] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:24 [I 2022-10-11 09:56:26] ...trap/engines/engine.py.326: process: 0.47438 | load: 0.00018 [I 2022-10-11 09:56:32] ...trap/engines/engine.py.321: eval: epoch 100 | batch 60/510 [I 2022-10-11 09:56:32] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:04:21 [I 2022-10-11 09:56:32] ...trap/engines/engine.py.326: process: 0.58968 | load: 0.00018 [I 2022-10-11 09:56:37] ...trap/engines/engine.py.321: eval: epoch 100 | batch 70/510 [I 2022-10-11 09:56:37] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:43 [I 2022-10-11 09:56:37] ...trap/engines/engine.py.326: process: 0.45434 | load: 0.00019 [I 2022-10-11 09:56:43] ...trap/engines/engine.py.321: eval: epoch 100 | batch 80/510 [I 2022-10-11 09:56:43] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:58 [I 2022-10-11 09:56:43] ...trap/engines/engine.py.326: process: 0.58264 | load: 0.00018 [I 2022-10-11 09:56:47] ...trap/engines/engine.py.321: eval: epoch 100 | batch 90/510 [I 2022-10-11 09:56:47] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:13 [I 2022-10-11 09:56:47] ...trap/engines/engine.py.326: process: 0.44513 | load: 0.00020 [I 2022-10-11 09:56:52] ...trap/engines/engine.py.321: eval: epoch 100 | batch 100/510 [I 2022-10-11 09:56:52] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:56 [I 2022-10-11 09:56:52] ...trap/engines/engine.py.326: process: 0.31164 | load: 0.00019 [I 2022-10-11 09:56:55] ...trap/engines/engine.py.321: eval: epoch 100 | batch 110/510 [I 2022-10-11 09:56:55] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:11 [I 2022-10-11 09:56:55] ...trap/engines/engine.py.326: process: 0.31238 | load: 0.00018 [I 2022-10-11 09:56:58] ...trap/engines/engine.py.321: eval: epoch 100 | batch 120/510 [I 2022-10-11 09:56:58] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:03 [I 2022-10-11 09:56:58] ...trap/engines/engine.py.326: process: 0.31641 | load: 0.00018 [I 2022-10-11 09:57:01] ...trap/engines/engine.py.321: eval: epoch 100 | batch 130/510 [I 2022-10-11 09:57:01] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:58 [I 2022-10-11 09:57:01] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00019 [I 2022-10-11 09:57:04] ...trap/engines/engine.py.321: eval: epoch 100 | batch 140/510 [I 2022-10-11 09:57:04] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:56 [I 2022-10-11 09:57:04] ...trap/engines/engine.py.326: process: 0.31748 | load: 0.00019 [I 2022-10-11 09:57:07] ...trap/engines/engine.py.321: eval: epoch 100 | batch 150/510 [I 2022-10-11 09:57:07] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:52 [I 2022-10-11 09:57:07] ...trap/engines/engine.py.326: process: 0.31111 | load: 0.00018 [I 2022-10-11 09:57:11] ...trap/engines/engine.py.321: eval: epoch 100 | batch 160/510 [I 2022-10-11 09:57:11] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:50 [I 2022-10-11 09:57:11] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00019 [I 2022-10-11 09:57:14] ...trap/engines/engine.py.321: eval: epoch 100 | batch 170/510 [I 2022-10-11 09:57:14] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:46 [I 2022-10-11 09:57:14] ...trap/engines/engine.py.326: process: 0.30995 | load: 0.00019 [I 2022-10-11 09:57:17] ...trap/engines/engine.py.321: eval: epoch 100 | batch 180/510 [I 2022-10-11 09:57:17] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:43 [I 2022-10-11 09:57:17] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00019 [I 2022-10-11 09:57:20] ...trap/engines/engine.py.321: eval: epoch 100 | batch 190/510 [I 2022-10-11 09:57:20] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:40 [I 2022-10-11 09:57:20] ...trap/engines/engine.py.326: process: 0.31618 | load: 0.00019 [I 2022-10-11 09:57:23] ...trap/engines/engine.py.321: eval: epoch 100 | batch 200/510 [I 2022-10-11 09:57:23] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:36 [I 2022-10-11 09:57:23] ...trap/engines/engine.py.326: process: 0.31262 | load: 0.00020 [I 2022-10-11 09:57:26] ...trap/engines/engine.py.321: eval: epoch 100 | batch 210/510 [I 2022-10-11 09:57:26] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:34 [I 2022-10-11 09:57:26] ...trap/engines/engine.py.326: process: 0.31680 | load: 0.00020 [I 2022-10-11 09:57:29] ...trap/engines/engine.py.321: eval: epoch 100 | batch 220/510 [I 2022-10-11 09:57:29] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:30 [I 2022-10-11 09:57:29] ...trap/engines/engine.py.326: process: 0.30904 | load: 0.00019 [I 2022-10-11 09:57:33] ...trap/engines/engine.py.321: eval: epoch 100 | batch 230/510 [I 2022-10-11 09:57:33] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:27 [I 2022-10-11 09:57:33] ...trap/engines/engine.py.326: process: 0.30826 | load: 0.00019 [I 2022-10-11 09:57:36] ...trap/engines/engine.py.321: eval: epoch 100 | batch 240/510 [I 2022-10-11 09:57:36] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:24 [I 2022-10-11 09:57:36] ...trap/engines/engine.py.326: process: 0.31359 | load: 0.00020 [I 2022-10-11 09:57:39] ...trap/engines/engine.py.321: eval: epoch 100 | batch 250/510 [I 2022-10-11 09:57:39] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:21 [I 2022-10-11 09:57:39] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00019 [I 2022-10-11 09:57:42] ...trap/engines/engine.py.321: eval: epoch 100 | batch 260/510 [I 2022-10-11 09:57:42] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:18 [I 2022-10-11 09:57:42] ...trap/engines/engine.py.326: process: 0.31418 | load: 0.00020 [I 2022-10-11 09:57:45] ...trap/engines/engine.py.321: eval: epoch 100 | batch 270/510 [I 2022-10-11 09:57:45] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:15 [I 2022-10-11 09:57:45] ...trap/engines/engine.py.326: process: 0.31170 | load: 0.00020 [I 2022-10-11 09:57:48] ...trap/engines/engine.py.321: eval: epoch 100 | batch 280/510 [I 2022-10-11 09:57:48] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:12 [I 2022-10-11 09:57:48] ...trap/engines/engine.py.326: process: 0.31562 | load: 0.00019 [I 2022-10-11 09:57:52] ...trap/engines/engine.py.321: eval: epoch 100 | batch 290/510 [I 2022-10-11 09:57:52] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:09 [I 2022-10-11 09:57:52] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00019 [I 2022-10-11 09:57:55] ...trap/engines/engine.py.321: eval: epoch 100 | batch 300/510 [I 2022-10-11 09:57:55] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:06 [I 2022-10-11 09:57:55] ...trap/engines/engine.py.326: process: 0.31768 | load: 0.00019 [I 2022-10-11 09:57:58] ...trap/engines/engine.py.321: eval: epoch 100 | batch 310/510 [I 2022-10-11 09:57:58] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:02 [I 2022-10-11 09:57:58] ...trap/engines/engine.py.326: process: 0.31986 | load: 0.00019 [I 2022-10-11 09:58:01] ...trap/engines/engine.py.321: eval: epoch 100 | batch 320/510 [I 2022-10-11 09:58:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:59 [I 2022-10-11 09:58:01] ...trap/engines/engine.py.326: process: 0.31888 | load: 0.00020 [I 2022-10-11 09:58:04] ...trap/engines/engine.py.321: eval: epoch 100 | batch 330/510 [I 2022-10-11 09:58:04] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:56 [I 2022-10-11 09:58:04] ...trap/engines/engine.py.326: process: 0.30982 | load: 0.00019 [I 2022-10-11 09:58:07] ...trap/engines/engine.py.321: eval: epoch 100 | batch 340/510 [I 2022-10-11 09:58:07] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:53 [I 2022-10-11 09:58:07] ...trap/engines/engine.py.326: process: 0.31172 | load: 0.00019 [I 2022-10-11 09:58:10] ...trap/engines/engine.py.321: eval: epoch 100 | batch 350/510 [I 2022-10-11 09:58:10] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:50 [I 2022-10-11 09:58:10] ...trap/engines/engine.py.326: process: 0.31465 | load: 0.00019 [I 2022-10-11 09:58:14] ...trap/engines/engine.py.321: eval: epoch 100 | batch 360/510 [I 2022-10-11 09:58:14] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:47 [I 2022-10-11 09:58:14] ...trap/engines/engine.py.326: process: 0.31817 | load: 0.00019 [I 2022-10-11 09:58:17] ...trap/engines/engine.py.321: eval: epoch 100 | batch 370/510 [I 2022-10-11 09:58:17] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:44 [I 2022-10-11 09:58:17] ...trap/engines/engine.py.326: process: 0.31943 | load: 0.00019 [I 2022-10-11 09:58:20] ...trap/engines/engine.py.321: eval: epoch 100 | batch 380/510 [I 2022-10-11 09:58:20] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-11 09:58:20] ...trap/engines/engine.py.326: process: 0.31815 | load: 0.00019 [I 2022-10-11 09:58:23] ...trap/engines/engine.py.321: eval: epoch 100 | batch 390/510 [I 2022-10-11 09:58:23] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:37 [I 2022-10-11 09:58:23] ...trap/engines/engine.py.326: process: 0.31503 | load: 0.00019 [I 2022-10-11 09:58:26] ...trap/engines/engine.py.321: eval: epoch 100 | batch 400/510 [I 2022-10-11 09:58:26] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:34 [I 2022-10-11 09:58:26] ...trap/engines/engine.py.326: process: 0.32136 | load: 0.00019 [I 2022-10-11 09:58:29] ...trap/engines/engine.py.321: eval: epoch 100 | batch 410/510 [I 2022-10-11 09:58:29] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:31 [I 2022-10-11 09:58:29] ...trap/engines/engine.py.326: process: 0.30965 | load: 0.00020 [I 2022-10-11 09:58:32] ...trap/engines/engine.py.321: eval: epoch 100 | batch 420/510 [I 2022-10-11 09:58:32] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-11 09:58:32] ...trap/engines/engine.py.326: process: 0.32064 | load: 0.00020 [I 2022-10-11 09:58:36] ...trap/engines/engine.py.321: eval: epoch 100 | batch 430/510 [I 2022-10-11 09:58:36] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-11 09:58:36] ...trap/engines/engine.py.326: process: 0.31398 | load: 0.00021 [I 2022-10-11 09:58:39] ...trap/engines/engine.py.321: eval: epoch 100 | batch 440/510 [I 2022-10-11 09:58:39] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-11 09:58:39] ...trap/engines/engine.py.326: process: 0.31656 | load: 0.00019 [I 2022-10-11 09:58:42] ...trap/engines/engine.py.321: eval: epoch 100 | batch 450/510 [I 2022-10-11 09:58:42] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:18 [I 2022-10-11 09:58:42] ...trap/engines/engine.py.326: process: 0.31485 | load: 0.00019 [I 2022-10-11 09:58:45] ...trap/engines/engine.py.321: eval: epoch 100 | batch 460/510 [I 2022-10-11 09:58:45] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-11 09:58:45] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00019 [I 2022-10-11 09:58:48] ...trap/engines/engine.py.321: eval: epoch 100 | batch 470/510 [I 2022-10-11 09:58:48] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-11 09:58:48] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00020 [I 2022-10-11 09:58:51] ...trap/engines/engine.py.321: eval: epoch 100 | batch 480/510 [I 2022-10-11 09:58:51] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-11 09:58:51] ...trap/engines/engine.py.326: process: 0.31548 | load: 0.00020 [I 2022-10-11 09:58:55] ...trap/engines/engine.py.321: eval: epoch 100 | batch 490/510 [I 2022-10-11 09:58:55] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:06 [I 2022-10-11 09:58:55] ...trap/engines/engine.py.326: process: 0.31705 | load: 0.00019 [I 2022-10-11 09:58:58] ...trap/engines/engine.py.321: eval: epoch 100 | batch 500/510 [I 2022-10-11 09:58:58] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-11 09:58:58] ...trap/engines/engine.py.326: process: 0.30916 | load: 0.00020 [I 2022-10-11 09:59:01] ...trap/engines/engine.py.321: eval: epoch 100 | batch 510/510 [I 2022-10-11 09:59:01] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-11 09:59:01] ...trap/engines/engine.py.326: process: 0.31309 | load: 0.00011 [S 2022-10-11 09:59:02] ...trap/engines/engine.py.132: eval_epoch.epoch: 100 [I 2022-10-11 09:59:02] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.747200 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.914700 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939200 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.743200 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.915900 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.942300 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.020247 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009508 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008807 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014162 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010004 [S 2022-10-11 09:59:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007198 [I 2022-10-11 09:59:41] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 09:59:41] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 09:59:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 09:59:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 09:59:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 09:59:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 09:59:49] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 09:59:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 09:59:52] ...trap/engines/engine.py.163: Training model on trainset for epoch 101 [I 2022-10-11 09:59:52] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 09:59:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 0/2304 [I 2022-10-11 09:59:56] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:02:55 [I 2022-10-11 09:59:56] ...trap/engines/engine.py.229: process: 3.20119 | load: 2.31396 [I 2022-10-11 09:59:56] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-11 09:59:56] ...optimizers/trijoint.py.151: total_norm: 24.622799 [I 2022-10-11 10:00:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 10/2304 [I 2022-10-11 10:00:04] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:06 [I 2022-10-11 10:00:04] ...trap/engines/engine.py.229: process: 0.82966 | load: 0.00525 [I 2022-10-11 10:00:04] ...trap/engines/engine.py.230: loss: 0.10113 [I 2022-10-11 10:00:04] ...optimizers/trijoint.py.151: total_norm: 9.162613 [I 2022-10-11 10:00:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 20/2304 [I 2022-10-11 10:00:13] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:00 [I 2022-10-11 10:00:13] ...trap/engines/engine.py.229: process: 0.83404 | load: 0.00497 [I 2022-10-11 10:00:13] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-11 10:00:13] ...optimizers/trijoint.py.151: total_norm: 14.413566 [I 2022-10-11 10:00:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 30/2304 [I 2022-10-11 10:00:21] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:07 [I 2022-10-11 10:00:21] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00366 [I 2022-10-11 10:00:21] ...trap/engines/engine.py.230: loss: 0.09348 [I 2022-10-11 10:00:21] ...optimizers/trijoint.py.151: total_norm: 11.814837 [I 2022-10-11 10:00:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 40/2304 [I 2022-10-11 10:00:29] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:52 [I 2022-10-11 10:00:29] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00343 [I 2022-10-11 10:00:29] ...trap/engines/engine.py.230: loss: 0.10893 [I 2022-10-11 10:00:29] ...optimizers/trijoint.py.151: total_norm: 11.288799 [I 2022-10-11 10:00:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 50/2304 [I 2022-10-11 10:00:38] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:40 [I 2022-10-11 10:00:38] ...trap/engines/engine.py.229: process: 0.84510 | load: 0.00395 [I 2022-10-11 10:00:38] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-11 10:00:38] ...optimizers/trijoint.py.151: total_norm: 10.735334 [I 2022-10-11 10:00:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 60/2304 [I 2022-10-11 10:00:46] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:34 [I 2022-10-11 10:00:46] ...trap/engines/engine.py.229: process: 0.84206 | load: 0.00364 [I 2022-10-11 10:00:46] ...trap/engines/engine.py.230: loss: 0.10154 [I 2022-10-11 10:00:46] ...optimizers/trijoint.py.151: total_norm: 20.859724 [I 2022-10-11 10:00:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 70/2304 [I 2022-10-11 10:00:55] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:03 [I 2022-10-11 10:00:55] ...trap/engines/engine.py.229: process: 0.84022 | load: 0.00334 [I 2022-10-11 10:00:55] ...trap/engines/engine.py.230: loss: 0.13016 [I 2022-10-11 10:00:55] ...optimizers/trijoint.py.151: total_norm: 18.308342 [I 2022-10-11 10:01:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 80/2304 [I 2022-10-11 10:01:03] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:06 [I 2022-10-11 10:01:03] ...trap/engines/engine.py.229: process: 0.82734 | load: 0.00329 [I 2022-10-11 10:01:03] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-11 10:01:03] ...optimizers/trijoint.py.151: total_norm: 11.735513 [I 2022-10-11 10:01:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 90/2304 [I 2022-10-11 10:01:11] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:04 [I 2022-10-11 10:01:11] ...trap/engines/engine.py.229: process: 0.83912 | load: 0.00362 [I 2022-10-11 10:01:11] ...trap/engines/engine.py.230: loss: 0.18707 [I 2022-10-11 10:01:11] ...optimizers/trijoint.py.151: total_norm: 20.498074 [I 2022-10-11 10:01:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 100/2304 [I 2022-10-11 10:01:20] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:12 [I 2022-10-11 10:01:20] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00331 [I 2022-10-11 10:01:20] ...trap/engines/engine.py.230: loss: 0.11326 [I 2022-10-11 10:01:20] ...optimizers/trijoint.py.151: total_norm: 20.729338 [I 2022-10-11 10:01:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 110/2304 [I 2022-10-11 10:01:28] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:01 [I 2022-10-11 10:01:28] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00293 [I 2022-10-11 10:01:28] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-11 10:01:28] ...optimizers/trijoint.py.151: total_norm: 9.796840 [I 2022-10-11 10:01:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 120/2304 [I 2022-10-11 10:01:37] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:48 [I 2022-10-11 10:01:37] ...trap/engines/engine.py.229: process: 0.84530 | load: 0.00351 [I 2022-10-11 10:01:37] ...trap/engines/engine.py.230: loss: 0.14823 [I 2022-10-11 10:01:37] ...optimizers/trijoint.py.151: total_norm: 13.743707 [I 2022-10-11 10:01:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 130/2304 [I 2022-10-11 10:01:45] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:41 [I 2022-10-11 10:01:45] ...trap/engines/engine.py.229: process: 0.85272 | load: 0.00336 [I 2022-10-11 10:01:45] ...trap/engines/engine.py.230: loss: 0.11928 [I 2022-10-11 10:01:45] ...optimizers/trijoint.py.151: total_norm: 9.037952 [I 2022-10-11 10:01:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 140/2304 [I 2022-10-11 10:01:54] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:36 [I 2022-10-11 10:01:54] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00292 [I 2022-10-11 10:01:54] ...trap/engines/engine.py.230: loss: 0.09182 [I 2022-10-11 10:01:54] ...optimizers/trijoint.py.151: total_norm: 7.607338 [I 2022-10-11 10:02:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 150/2304 [I 2022-10-11 10:02:02] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:32 [I 2022-10-11 10:02:02] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00303 [I 2022-10-11 10:02:02] ...trap/engines/engine.py.230: loss: 0.16782 [I 2022-10-11 10:02:02] ...optimizers/trijoint.py.151: total_norm: 14.345956 [I 2022-10-11 10:02:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 160/2304 [I 2022-10-11 10:02:11] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:23 [I 2022-10-11 10:02:11] ...trap/engines/engine.py.229: process: 0.87017 | load: 0.00386 [I 2022-10-11 10:02:11] ...trap/engines/engine.py.230: loss: 0.12631 [I 2022-10-11 10:02:11] ...optimizers/trijoint.py.151: total_norm: 20.361326 [I 2022-10-11 10:02:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 170/2304 [I 2022-10-11 10:02:20] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:58 [I 2022-10-11 10:02:20] ...trap/engines/engine.py.229: process: 0.86420 | load: 0.00404 [I 2022-10-11 10:02:20] ...trap/engines/engine.py.230: loss: 0.16106 [I 2022-10-11 10:02:20] ...optimizers/trijoint.py.151: total_norm: 12.024020 [I 2022-10-11 10:02:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 180/2304 [I 2022-10-11 10:02:28] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:36 [I 2022-10-11 10:02:28] ...trap/engines/engine.py.229: process: 0.86828 | load: 0.00384 [I 2022-10-11 10:02:28] ...trap/engines/engine.py.230: loss: 0.15760 [I 2022-10-11 10:02:28] ...optimizers/trijoint.py.151: total_norm: 11.543850 [I 2022-10-11 10:02:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 190/2304 [I 2022-10-11 10:02:37] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:42 [I 2022-10-11 10:02:37] ...trap/engines/engine.py.229: process: 0.87716 | load: 0.00401 [I 2022-10-11 10:02:37] ...trap/engines/engine.py.230: loss: 0.08316 [I 2022-10-11 10:02:37] ...optimizers/trijoint.py.151: total_norm: 13.824411 [I 2022-10-11 10:02:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 200/2304 [I 2022-10-11 10:02:46] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:30:31 [I 2022-10-11 10:02:46] ...trap/engines/engine.py.229: process: 0.86632 | load: 0.00372 [I 2022-10-11 10:02:46] ...trap/engines/engine.py.230: loss: 0.08959 [I 2022-10-11 10:02:46] ...optimizers/trijoint.py.151: total_norm: 19.446537 [I 2022-10-11 10:02:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 210/2304 [I 2022-10-11 10:02:55] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:30:23 [I 2022-10-11 10:02:55] ...trap/engines/engine.py.229: process: 0.86700 | load: 0.00372 [I 2022-10-11 10:02:55] ...trap/engines/engine.py.230: loss: 0.10397 [I 2022-10-11 10:02:55] ...optimizers/trijoint.py.151: total_norm: 16.797562 [I 2022-10-11 10:03:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 220/2304 [I 2022-10-11 10:03:03] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:30:20 [I 2022-10-11 10:03:03] ...trap/engines/engine.py.229: process: 0.87098 | load: 0.00371 [I 2022-10-11 10:03:03] ...trap/engines/engine.py.230: loss: 0.07720 [I 2022-10-11 10:03:03] ...optimizers/trijoint.py.151: total_norm: 11.314320 [I 2022-10-11 10:03:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 230/2304 [I 2022-10-11 10:03:12] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:30:00 [I 2022-10-11 10:03:12] ...trap/engines/engine.py.229: process: 0.85880 | load: 0.00370 [I 2022-10-11 10:03:12] ...trap/engines/engine.py.230: loss: 0.13453 [I 2022-10-11 10:03:12] ...optimizers/trijoint.py.151: total_norm: 15.409636 [I 2022-10-11 10:03:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 240/2304 [I 2022-10-11 10:03:21] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:53 [I 2022-10-11 10:03:21] ...trap/engines/engine.py.229: process: 0.85665 | load: 0.00374 [I 2022-10-11 10:03:21] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-11 10:03:21] ...optimizers/trijoint.py.151: total_norm: 14.209517 [I 2022-10-11 10:03:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 250/2304 [I 2022-10-11 10:03:29] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:35 [I 2022-10-11 10:03:29] ...trap/engines/engine.py.229: process: 0.86606 | load: 0.00372 [I 2022-10-11 10:03:29] ...trap/engines/engine.py.230: loss: 0.14869 [I 2022-10-11 10:03:29] ...optimizers/trijoint.py.151: total_norm: 21.040943 [I 2022-10-11 10:03:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 260/2304 [I 2022-10-11 10:03:38] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:43 [I 2022-10-11 10:03:38] ...trap/engines/engine.py.229: process: 0.86522 | load: 0.00391 [I 2022-10-11 10:03:38] ...trap/engines/engine.py.230: loss: 0.10368 [I 2022-10-11 10:03:38] ...optimizers/trijoint.py.151: total_norm: 14.490636 [I 2022-10-11 10:03:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 270/2304 [I 2022-10-11 10:03:47] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:37 [I 2022-10-11 10:03:47] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00399 [I 2022-10-11 10:03:47] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-11 10:03:47] ...optimizers/trijoint.py.151: total_norm: 13.847352 [I 2022-10-11 10:03:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 280/2304 [I 2022-10-11 10:03:56] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:29:13 [I 2022-10-11 10:03:56] ...trap/engines/engine.py.229: process: 0.86366 | load: 0.00371 [I 2022-10-11 10:03:56] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-11 10:03:56] ...optimizers/trijoint.py.151: total_norm: 10.178177 [I 2022-10-11 10:04:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 290/2304 [I 2022-10-11 10:04:04] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:29:20 [I 2022-10-11 10:04:04] ...trap/engines/engine.py.229: process: 0.88609 | load: 0.00373 [I 2022-10-11 10:04:04] ...trap/engines/engine.py.230: loss: 0.07871 [I 2022-10-11 10:04:04] ...optimizers/trijoint.py.151: total_norm: 14.674369 [I 2022-10-11 10:04:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 300/2304 [I 2022-10-11 10:04:13] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:29:06 [I 2022-10-11 10:04:13] ...trap/engines/engine.py.229: process: 0.87451 | load: 0.00392 [I 2022-10-11 10:04:13] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-11 10:04:13] ...optimizers/trijoint.py.151: total_norm: 48.225029 [I 2022-10-11 10:04:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 310/2304 [I 2022-10-11 10:04:22] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:57 [I 2022-10-11 10:04:22] ...trap/engines/engine.py.229: process: 0.87515 | load: 0.00383 [I 2022-10-11 10:04:22] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-11 10:04:22] ...optimizers/trijoint.py.151: total_norm: 15.758027 [I 2022-10-11 10:04:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 320/2304 [I 2022-10-11 10:04:31] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:28:53 [I 2022-10-11 10:04:31] ...trap/engines/engine.py.229: process: 0.87687 | load: 0.00381 [I 2022-10-11 10:04:31] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-11 10:04:31] ...optimizers/trijoint.py.151: total_norm: 18.507748 [I 2022-10-11 10:04:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 330/2304 [I 2022-10-11 10:04:39] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:35 [I 2022-10-11 10:04:39] ...trap/engines/engine.py.229: process: 0.86503 | load: 0.00373 [I 2022-10-11 10:04:39] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-11 10:04:39] ...optimizers/trijoint.py.151: total_norm: 12.832109 [I 2022-10-11 10:04:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 340/2304 [I 2022-10-11 10:04:48] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:28:33 [I 2022-10-11 10:04:48] ...trap/engines/engine.py.229: process: 0.87365 | load: 0.00372 [I 2022-10-11 10:04:48] ...trap/engines/engine.py.230: loss: 0.08904 [I 2022-10-11 10:04:48] ...optimizers/trijoint.py.151: total_norm: 13.072442 [I 2022-10-11 10:04:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 350/2304 [I 2022-10-11 10:04:57] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:28:22 [I 2022-10-11 10:04:57] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00369 [I 2022-10-11 10:04:57] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-11 10:04:57] ...optimizers/trijoint.py.151: total_norm: 9.616051 [I 2022-10-11 10:05:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 360/2304 [I 2022-10-11 10:05:05] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:28:22 [I 2022-10-11 10:05:05] ...trap/engines/engine.py.229: process: 0.86547 | load: 0.00373 [I 2022-10-11 10:05:05] ...trap/engines/engine.py.230: loss: 0.10809 [I 2022-10-11 10:05:05] ...optimizers/trijoint.py.151: total_norm: 15.170341 [I 2022-10-11 10:05:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 370/2304 [I 2022-10-11 10:05:14] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:22 [I 2022-10-11 10:05:14] ...trap/engines/engine.py.229: process: 0.84219 | load: 0.00293 [I 2022-10-11 10:05:14] ...trap/engines/engine.py.230: loss: 0.09305 [I 2022-10-11 10:05:14] ...optimizers/trijoint.py.151: total_norm: 11.160048 [I 2022-10-11 10:05:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 380/2304 [I 2022-10-11 10:05:22] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:27:03 [I 2022-10-11 10:05:22] ...trap/engines/engine.py.229: process: 0.85004 | load: 0.00292 [I 2022-10-11 10:05:22] ...trap/engines/engine.py.230: loss: 0.11786 [I 2022-10-11 10:05:22] ...optimizers/trijoint.py.151: total_norm: 15.815232 [I 2022-10-11 10:05:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 390/2304 [I 2022-10-11 10:05:31] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:26:43 [I 2022-10-11 10:05:31] ...trap/engines/engine.py.229: process: 0.83474 | load: 0.00302 [I 2022-10-11 10:05:31] ...trap/engines/engine.py.230: loss: 0.11257 [I 2022-10-11 10:05:31] ...optimizers/trijoint.py.151: total_norm: 25.087393 [I 2022-10-11 10:05:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 400/2304 [I 2022-10-11 10:05:39] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:26:39 [I 2022-10-11 10:05:39] ...trap/engines/engine.py.229: process: 0.83507 | load: 0.00291 [I 2022-10-11 10:05:39] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-11 10:05:39] ...optimizers/trijoint.py.151: total_norm: 8.825768 [I 2022-10-11 10:05:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 410/2304 [I 2022-10-11 10:05:48] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:26:36 [I 2022-10-11 10:05:48] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00293 [I 2022-10-11 10:05:48] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-11 10:05:48] ...optimizers/trijoint.py.151: total_norm: 9.741892 [I 2022-10-11 10:05:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 420/2304 [I 2022-10-11 10:05:56] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:26:36 [I 2022-10-11 10:05:56] ...trap/engines/engine.py.229: process: 0.84430 | load: 0.00293 [I 2022-10-11 10:05:56] ...trap/engines/engine.py.230: loss: 0.09567 [I 2022-10-11 10:05:56] ...optimizers/trijoint.py.151: total_norm: 11.345017 [I 2022-10-11 10:06:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 430/2304 [I 2022-10-11 10:06:05] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:26:17 [I 2022-10-11 10:06:05] ...trap/engines/engine.py.229: process: 0.83877 | load: 0.00292 [I 2022-10-11 10:06:05] ...trap/engines/engine.py.230: loss: 0.12977 [I 2022-10-11 10:06:05] ...optimizers/trijoint.py.151: total_norm: 17.064434 [I 2022-10-11 10:06:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 440/2304 [I 2022-10-11 10:06:13] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:00 [I 2022-10-11 10:06:13] ...trap/engines/engine.py.229: process: 0.84061 | load: 0.00294 [I 2022-10-11 10:06:13] ...trap/engines/engine.py.230: loss: 0.08684 [I 2022-10-11 10:06:13] ...optimizers/trijoint.py.151: total_norm: 11.441761 [I 2022-10-11 10:06:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 450/2304 [I 2022-10-11 10:06:21] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:26:01 [I 2022-10-11 10:06:21] ...trap/engines/engine.py.229: process: 0.84117 | load: 0.00293 [I 2022-10-11 10:06:21] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-11 10:06:21] ...optimizers/trijoint.py.151: total_norm: 18.472538 [I 2022-10-11 10:06:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 460/2304 [I 2022-10-11 10:06:30] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:25:49 [I 2022-10-11 10:06:30] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00293 [I 2022-10-11 10:06:30] ...trap/engines/engine.py.230: loss: 0.07711 [I 2022-10-11 10:06:30] ...optimizers/trijoint.py.151: total_norm: 8.392587 [I 2022-10-11 10:06:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 470/2304 [I 2022-10-11 10:06:38] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:25:47 [I 2022-10-11 10:06:38] ...trap/engines/engine.py.229: process: 0.85786 | load: 0.00304 [I 2022-10-11 10:06:38] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-11 10:06:38] ...optimizers/trijoint.py.151: total_norm: 25.159712 [I 2022-10-11 10:06:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 480/2304 [I 2022-10-11 10:06:47] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:25:38 [I 2022-10-11 10:06:47] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00293 [I 2022-10-11 10:06:47] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-11 10:06:47] ...optimizers/trijoint.py.151: total_norm: 20.209246 [I 2022-10-11 10:06:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 490/2304 [I 2022-10-11 10:06:55] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:25:42 [I 2022-10-11 10:06:55] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00292 [I 2022-10-11 10:06:55] ...trap/engines/engine.py.230: loss: 0.09290 [I 2022-10-11 10:06:55] ...optimizers/trijoint.py.151: total_norm: 15.524759 [I 2022-10-11 10:07:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 500/2304 [I 2022-10-11 10:07:04] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:25:27 [I 2022-10-11 10:07:04] ...trap/engines/engine.py.229: process: 0.83965 | load: 0.00292 [I 2022-10-11 10:07:04] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-11 10:07:04] ...optimizers/trijoint.py.151: total_norm: 12.913346 [I 2022-10-11 10:07:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 510/2304 [I 2022-10-11 10:07:12] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:25:13 [I 2022-10-11 10:07:12] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00292 [I 2022-10-11 10:07:12] ...trap/engines/engine.py.230: loss: 0.09385 [I 2022-10-11 10:07:12] ...optimizers/trijoint.py.151: total_norm: 12.642156 [I 2022-10-11 10:07:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 520/2304 [I 2022-10-11 10:07:21] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:25:18 [I 2022-10-11 10:07:21] ...trap/engines/engine.py.229: process: 0.84935 | load: 0.00294 [I 2022-10-11 10:07:21] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-11 10:07:21] ...optimizers/trijoint.py.151: total_norm: 12.714767 [I 2022-10-11 10:07:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 530/2304 [I 2022-10-11 10:07:29] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:24:57 [I 2022-10-11 10:07:29] ...trap/engines/engine.py.229: process: 0.83266 | load: 0.00293 [I 2022-10-11 10:07:29] ...trap/engines/engine.py.230: loss: 0.07607 [I 2022-10-11 10:07:29] ...optimizers/trijoint.py.151: total_norm: 9.088165 [I 2022-10-11 10:07:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 540/2304 [I 2022-10-11 10:07:38] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:24:41 [I 2022-10-11 10:07:38] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00291 [I 2022-10-11 10:07:38] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-11 10:07:38] ...optimizers/trijoint.py.151: total_norm: 14.011612 [I 2022-10-11 10:07:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 550/2304 [I 2022-10-11 10:07:46] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:24:44 [I 2022-10-11 10:07:46] ...trap/engines/engine.py.229: process: 0.84254 | load: 0.00301 [I 2022-10-11 10:07:46] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-11 10:07:46] ...optimizers/trijoint.py.151: total_norm: 16.180878 [I 2022-10-11 10:07:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 560/2304 [I 2022-10-11 10:07:55] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:24:33 [I 2022-10-11 10:07:55] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00292 [I 2022-10-11 10:07:55] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-11 10:07:55] ...optimizers/trijoint.py.151: total_norm: 11.625494 [I 2022-10-11 10:08:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 570/2304 [I 2022-10-11 10:08:03] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:24:16 [I 2022-10-11 10:08:03] ...trap/engines/engine.py.229: process: 0.83275 | load: 0.00293 [I 2022-10-11 10:08:03] ...trap/engines/engine.py.230: loss: 0.10278 [I 2022-10-11 10:08:03] ...optimizers/trijoint.py.151: total_norm: 9.500430 [I 2022-10-11 10:08:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 580/2304 [I 2022-10-11 10:08:11] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:01 [I 2022-10-11 10:08:11] ...trap/engines/engine.py.229: process: 0.83367 | load: 0.00292 [I 2022-10-11 10:08:11] ...trap/engines/engine.py.230: loss: 0.12730 [I 2022-10-11 10:08:11] ...optimizers/trijoint.py.151: total_norm: 20.081585 [I 2022-10-11 10:08:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 590/2304 [I 2022-10-11 10:08:20] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:24:18 [I 2022-10-11 10:08:20] ...trap/engines/engine.py.229: process: 0.85811 | load: 0.00293 [I 2022-10-11 10:08:20] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-11 10:08:20] ...optimizers/trijoint.py.151: total_norm: 24.743071 [I 2022-10-11 10:08:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 600/2304 [I 2022-10-11 10:08:28] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:23:59 [I 2022-10-11 10:08:28] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00293 [I 2022-10-11 10:08:28] ...trap/engines/engine.py.230: loss: 0.08716 [I 2022-10-11 10:08:28] ...optimizers/trijoint.py.151: total_norm: 19.407719 [I 2022-10-11 10:08:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 610/2304 [I 2022-10-11 10:08:37] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:54 [I 2022-10-11 10:08:37] ...trap/engines/engine.py.229: process: 0.84537 | load: 0.00292 [I 2022-10-11 10:08:37] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-11 10:08:37] ...optimizers/trijoint.py.151: total_norm: 12.631728 [I 2022-10-11 10:08:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 620/2304 [I 2022-10-11 10:08:45] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:46 [I 2022-10-11 10:08:45] ...trap/engines/engine.py.229: process: 0.83805 | load: 0.00292 [I 2022-10-11 10:08:45] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-11 10:08:45] ...optimizers/trijoint.py.151: total_norm: 21.184629 [I 2022-10-11 10:08:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 630/2304 [I 2022-10-11 10:08:54] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:34 [I 2022-10-11 10:08:54] ...trap/engines/engine.py.229: process: 0.84238 | load: 0.00293 [I 2022-10-11 10:08:54] ...trap/engines/engine.py.230: loss: 0.11368 [I 2022-10-11 10:08:54] ...optimizers/trijoint.py.151: total_norm: 18.015352 [I 2022-10-11 10:09:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 640/2304 [I 2022-10-11 10:09:02] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:20 [I 2022-10-11 10:09:02] ...trap/engines/engine.py.229: process: 0.84004 | load: 0.00293 [I 2022-10-11 10:09:02] ...trap/engines/engine.py.230: loss: 0.18911 [I 2022-10-11 10:09:02] ...optimizers/trijoint.py.151: total_norm: 8.557601 [I 2022-10-11 10:09:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 650/2304 [I 2022-10-11 10:09:10] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:08 [I 2022-10-11 10:09:10] ...trap/engines/engine.py.229: process: 0.84480 | load: 0.00293 [I 2022-10-11 10:09:10] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-11 10:09:10] ...optimizers/trijoint.py.151: total_norm: 16.310465 [I 2022-10-11 10:09:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 660/2304 [I 2022-10-11 10:09:19] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:22:50 [I 2022-10-11 10:09:19] ...trap/engines/engine.py.229: process: 0.82911 | load: 0.00293 [I 2022-10-11 10:09:19] ...trap/engines/engine.py.230: loss: 0.10155 [I 2022-10-11 10:09:19] ...optimizers/trijoint.py.151: total_norm: 27.290373 [I 2022-10-11 10:09:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 670/2304 [I 2022-10-11 10:09:30] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:33:28 [I 2022-10-11 10:09:30] ...trap/engines/engine.py.229: process: 0.83679 | load: 0.00291 [I 2022-10-11 10:09:30] ...trap/engines/engine.py.230: loss: 0.14481 [I 2022-10-11 10:09:30] ...optimizers/trijoint.py.151: total_norm: 16.362000 [I 2022-10-11 10:09:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 680/2304 [I 2022-10-11 10:09:39] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:23:56 [I 2022-10-11 10:09:39] ...trap/engines/engine.py.229: process: 0.82001 | load: 0.00293 [I 2022-10-11 10:09:39] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-11 10:09:39] ...optimizers/trijoint.py.151: total_norm: 12.123025 [I 2022-10-11 10:09:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 690/2304 [I 2022-10-11 10:09:47] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:56 [I 2022-10-11 10:09:47] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00293 [I 2022-10-11 10:09:47] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-11 10:09:47] ...optimizers/trijoint.py.151: total_norm: 35.907124 [I 2022-10-11 10:09:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 700/2304 [I 2022-10-11 10:09:56] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:22:32 [I 2022-10-11 10:09:56] ...trap/engines/engine.py.229: process: 0.82610 | load: 0.00295 [I 2022-10-11 10:09:56] ...trap/engines/engine.py.230: loss: 0.09097 [I 2022-10-11 10:09:56] ...optimizers/trijoint.py.151: total_norm: 8.642102 [I 2022-10-11 10:10:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 710/2304 [I 2022-10-11 10:10:04] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:22 [I 2022-10-11 10:10:04] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00294 [I 2022-10-11 10:10:04] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-11 10:10:04] ...optimizers/trijoint.py.151: total_norm: 14.309584 [I 2022-10-11 10:10:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 720/2304 [I 2022-10-11 10:10:13] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:22:11 [I 2022-10-11 10:10:13] ...trap/engines/engine.py.229: process: 0.84432 | load: 0.00293 [I 2022-10-11 10:10:13] ...trap/engines/engine.py.230: loss: 0.09649 [I 2022-10-11 10:10:13] ...optimizers/trijoint.py.151: total_norm: 21.720797 [I 2022-10-11 10:10:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 730/2304 [I 2022-10-11 10:10:21] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:22:03 [I 2022-10-11 10:10:21] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00294 [I 2022-10-11 10:10:21] ...trap/engines/engine.py.230: loss: 0.09753 [I 2022-10-11 10:10:21] ...optimizers/trijoint.py.151: total_norm: 8.217966 [I 2022-10-11 10:10:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 740/2304 [I 2022-10-11 10:10:29] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:21:59 [I 2022-10-11 10:10:29] ...trap/engines/engine.py.229: process: 0.86457 | load: 0.00295 [I 2022-10-11 10:10:29] ...trap/engines/engine.py.230: loss: 0.16145 [I 2022-10-11 10:10:29] ...optimizers/trijoint.py.151: total_norm: 14.276541 [I 2022-10-11 10:10:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 750/2304 [I 2022-10-11 10:10:38] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:22:21 [I 2022-10-11 10:10:38] ...trap/engines/engine.py.229: process: 0.87848 | load: 0.00373 [I 2022-10-11 10:10:38] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-11 10:10:38] ...optimizers/trijoint.py.151: total_norm: 9.265898 [I 2022-10-11 10:10:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 760/2304 [I 2022-10-11 10:10:47] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:22:22 [I 2022-10-11 10:10:47] ...trap/engines/engine.py.229: process: 0.87022 | load: 0.00372 [I 2022-10-11 10:10:47] ...trap/engines/engine.py.230: loss: 0.15214 [I 2022-10-11 10:10:47] ...optimizers/trijoint.py.151: total_norm: 25.768440 [I 2022-10-11 10:10:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 770/2304 [I 2022-10-11 10:10:55] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:22:14 [I 2022-10-11 10:10:55] ...trap/engines/engine.py.229: process: 0.87122 | load: 0.00373 [I 2022-10-11 10:10:55] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-11 10:10:55] ...optimizers/trijoint.py.151: total_norm: 9.820882 [I 2022-10-11 10:11:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 780/2304 [I 2022-10-11 10:11:04] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:22:03 [I 2022-10-11 10:11:04] ...trap/engines/engine.py.229: process: 0.87235 | load: 0.00373 [I 2022-10-11 10:11:04] ...trap/engines/engine.py.230: loss: 0.20121 [I 2022-10-11 10:11:04] ...optimizers/trijoint.py.151: total_norm: 24.118963 [I 2022-10-11 10:11:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 790/2304 [I 2022-10-11 10:11:13] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:55 [I 2022-10-11 10:11:13] ...trap/engines/engine.py.229: process: 0.88227 | load: 0.00371 [I 2022-10-11 10:11:13] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-11 10:11:13] ...optimizers/trijoint.py.151: total_norm: 15.517571 [I 2022-10-11 10:11:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 800/2304 [I 2022-10-11 10:11:22] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:21:56 [I 2022-10-11 10:11:22] ...trap/engines/engine.py.229: process: 0.88806 | load: 0.00372 [I 2022-10-11 10:11:22] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-11 10:11:22] ...optimizers/trijoint.py.151: total_norm: 24.505432 [I 2022-10-11 10:11:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 810/2304 [I 2022-10-11 10:11:30] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:21:10 [I 2022-10-11 10:11:30] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00293 [I 2022-10-11 10:11:30] ...trap/engines/engine.py.230: loss: 0.10141 [I 2022-10-11 10:11:30] ...optimizers/trijoint.py.151: total_norm: 16.937449 [I 2022-10-11 10:11:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 820/2304 [I 2022-10-11 10:11:39] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:21:05 [I 2022-10-11 10:11:39] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00293 [I 2022-10-11 10:11:39] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-11 10:11:39] ...optimizers/trijoint.py.151: total_norm: 9.363453 [I 2022-10-11 10:11:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 830/2304 [I 2022-10-11 10:11:47] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:20:53 [I 2022-10-11 10:11:47] ...trap/engines/engine.py.229: process: 0.85825 | load: 0.00294 [I 2022-10-11 10:11:47] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-11 10:11:47] ...optimizers/trijoint.py.151: total_norm: 9.051036 [I 2022-10-11 10:11:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 840/2304 [I 2022-10-11 10:11:56] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:21:12 [I 2022-10-11 10:11:56] ...trap/engines/engine.py.229: process: 0.87018 | load: 0.00373 [I 2022-10-11 10:11:56] ...trap/engines/engine.py.230: loss: 0.14048 [I 2022-10-11 10:11:56] ...optimizers/trijoint.py.151: total_norm: 17.059547 [I 2022-10-11 10:12:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 850/2304 [I 2022-10-11 10:12:05] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:21:00 [I 2022-10-11 10:12:05] ...trap/engines/engine.py.229: process: 0.87943 | load: 0.00373 [I 2022-10-11 10:12:05] ...trap/engines/engine.py.230: loss: 0.09289 [I 2022-10-11 10:12:05] ...optimizers/trijoint.py.151: total_norm: 11.132320 [I 2022-10-11 10:12:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 860/2304 [I 2022-10-11 10:12:13] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:21:02 [I 2022-10-11 10:12:13] ...trap/engines/engine.py.229: process: 0.87567 | load: 0.00370 [I 2022-10-11 10:12:13] ...trap/engines/engine.py.230: loss: 0.21851 [I 2022-10-11 10:12:13] ...optimizers/trijoint.py.151: total_norm: 27.014906 [I 2022-10-11 10:12:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 870/2304 [I 2022-10-11 10:12:22] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:20:49 [I 2022-10-11 10:12:22] ...trap/engines/engine.py.229: process: 0.87864 | load: 0.00373 [I 2022-10-11 10:12:22] ...trap/engines/engine.py.230: loss: 0.16656 [I 2022-10-11 10:12:22] ...optimizers/trijoint.py.151: total_norm: 17.603779 [I 2022-10-11 10:12:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 880/2304 [I 2022-10-11 10:12:31] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:20:41 [I 2022-10-11 10:12:31] ...trap/engines/engine.py.229: process: 0.87398 | load: 0.00372 [I 2022-10-11 10:12:31] ...trap/engines/engine.py.230: loss: 0.12940 [I 2022-10-11 10:12:31] ...optimizers/trijoint.py.151: total_norm: 26.501152 [I 2022-10-11 10:12:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 890/2304 [I 2022-10-11 10:12:39] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:23 [I 2022-10-11 10:12:39] ...trap/engines/engine.py.229: process: 0.85550 | load: 0.00373 [I 2022-10-11 10:12:39] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-11 10:12:39] ...optimizers/trijoint.py.151: total_norm: 22.956602 [I 2022-10-11 10:12:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 900/2304 [I 2022-10-11 10:12:48] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:20:24 [I 2022-10-11 10:12:48] ...trap/engines/engine.py.229: process: 0.86668 | load: 0.00371 [I 2022-10-11 10:12:48] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-11 10:12:48] ...optimizers/trijoint.py.151: total_norm: 13.725914 [I 2022-10-11 10:12:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 910/2304 [I 2022-10-11 10:12:57] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:57 [I 2022-10-11 10:12:57] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00292 [I 2022-10-11 10:12:57] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-11 10:12:57] ...optimizers/trijoint.py.151: total_norm: 38.307022 [I 2022-10-11 10:13:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 920/2304 [I 2022-10-11 10:13:05] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:19:21 [I 2022-10-11 10:13:05] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00294 [I 2022-10-11 10:13:05] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-11 10:13:05] ...optimizers/trijoint.py.151: total_norm: 17.211664 [I 2022-10-11 10:13:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 930/2304 [I 2022-10-11 10:13:14] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:10 [I 2022-10-11 10:13:14] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00292 [I 2022-10-11 10:13:14] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-11 10:13:14] ...optimizers/trijoint.py.151: total_norm: 20.899994 [I 2022-10-11 10:13:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 940/2304 [I 2022-10-11 10:13:22] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:19:10 [I 2022-10-11 10:13:22] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00295 [I 2022-10-11 10:13:22] ...trap/engines/engine.py.230: loss: 0.16326 [I 2022-10-11 10:13:22] ...optimizers/trijoint.py.151: total_norm: 15.473827 [I 2022-10-11 10:13:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 950/2304 [I 2022-10-11 10:13:30] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:18:48 [I 2022-10-11 10:13:30] ...trap/engines/engine.py.229: process: 0.82875 | load: 0.00293 [I 2022-10-11 10:13:30] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-11 10:13:30] ...optimizers/trijoint.py.151: total_norm: 12.965320 [I 2022-10-11 10:13:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 960/2304 [I 2022-10-11 10:13:39] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:18:48 [I 2022-10-11 10:13:39] ...trap/engines/engine.py.229: process: 0.84140 | load: 0.00293 [I 2022-10-11 10:13:39] ...trap/engines/engine.py.230: loss: 0.12015 [I 2022-10-11 10:13:39] ...optimizers/trijoint.py.151: total_norm: 11.633708 [I 2022-10-11 10:13:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 970/2304 [I 2022-10-11 10:13:47] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:42 [I 2022-10-11 10:13:47] ...trap/engines/engine.py.229: process: 0.83482 | load: 0.00292 [I 2022-10-11 10:13:47] ...trap/engines/engine.py.230: loss: 0.09525 [I 2022-10-11 10:13:47] ...optimizers/trijoint.py.151: total_norm: 17.107718 [I 2022-10-11 10:13:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 980/2304 [I 2022-10-11 10:13:56] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:18:50 [I 2022-10-11 10:13:56] ...trap/engines/engine.py.229: process: 0.87603 | load: 0.00295 [I 2022-10-11 10:13:56] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-11 10:13:56] ...optimizers/trijoint.py.151: total_norm: 11.904107 [I 2022-10-11 10:14:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 990/2304 [I 2022-10-11 10:14:04] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:41 [I 2022-10-11 10:14:04] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00293 [I 2022-10-11 10:14:04] ...trap/engines/engine.py.230: loss: 0.18196 [I 2022-10-11 10:14:04] ...optimizers/trijoint.py.151: total_norm: 24.065145 [I 2022-10-11 10:14:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 1000/2304 [I 2022-10-11 10:14:13] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:18:30 [I 2022-10-11 10:14:13] ...trap/engines/engine.py.229: process: 0.86363 | load: 0.00295 [I 2022-10-11 10:14:13] ...trap/engines/engine.py.230: loss: 0.11528 [I 2022-10-11 10:14:13] ...optimizers/trijoint.py.151: total_norm: 11.306432 [I 2022-10-11 10:14:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 1010/2304 [I 2022-10-11 10:14:39] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:57:51 [I 2022-10-11 10:14:39] ...trap/engines/engine.py.229: process: 0.86215 | load: 0.00294 [I 2022-10-11 10:14:39] ...trap/engines/engine.py.230: loss: 0.12541 [I 2022-10-11 10:14:39] ...optimizers/trijoint.py.151: total_norm: 22.224541 [I 2022-10-11 10:14:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 1020/2304 [I 2022-10-11 10:14:47] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:22:15 [I 2022-10-11 10:14:47] ...trap/engines/engine.py.229: process: 0.84137 | load: 0.00292 [I 2022-10-11 10:14:47] ...trap/engines/engine.py.230: loss: 0.13100 [I 2022-10-11 10:14:47] ...optimizers/trijoint.py.151: total_norm: 19.621342 [I 2022-10-11 10:14:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 1030/2304 [I 2022-10-11 10:14:56] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:18:26 [I 2022-10-11 10:14:56] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00294 [I 2022-10-11 10:14:56] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-11 10:14:56] ...optimizers/trijoint.py.151: total_norm: 10.314574 [I 2022-10-11 10:15:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 1040/2304 [I 2022-10-11 10:15:04] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:51 [I 2022-10-11 10:15:04] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00294 [I 2022-10-11 10:15:04] ...trap/engines/engine.py.230: loss: 0.18958 [I 2022-10-11 10:15:04] ...optimizers/trijoint.py.151: total_norm: 14.195499 [I 2022-10-11 10:15:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 1050/2304 [I 2022-10-11 10:15:13] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:31 [I 2022-10-11 10:15:13] ...trap/engines/engine.py.229: process: 0.83134 | load: 0.00294 [I 2022-10-11 10:15:13] ...trap/engines/engine.py.230: loss: 0.13408 [I 2022-10-11 10:15:13] ...optimizers/trijoint.py.151: total_norm: 11.531826 [I 2022-10-11 10:15:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 1060/2304 [I 2022-10-11 10:15:21] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:25 [I 2022-10-11 10:15:21] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00292 [I 2022-10-11 10:15:21] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-11 10:15:21] ...optimizers/trijoint.py.151: total_norm: 14.385760 [I 2022-10-11 10:15:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 1070/2304 [I 2022-10-11 10:15:30] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:17:32 [I 2022-10-11 10:15:30] ...trap/engines/engine.py.229: process: 0.86437 | load: 0.00291 [I 2022-10-11 10:15:30] ...trap/engines/engine.py.230: loss: 0.10848 [I 2022-10-11 10:15:30] ...optimizers/trijoint.py.151: total_norm: 16.917652 [I 2022-10-11 10:15:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 1080/2304 [I 2022-10-11 10:15:38] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:15 [I 2022-10-11 10:15:38] ...trap/engines/engine.py.229: process: 0.85007 | load: 0.00293 [I 2022-10-11 10:15:38] ...trap/engines/engine.py.230: loss: 0.09777 [I 2022-10-11 10:15:38] ...optimizers/trijoint.py.151: total_norm: 14.732342 [I 2022-10-11 10:15:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 1090/2304 [I 2022-10-11 10:15:47] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:00 [I 2022-10-11 10:15:47] ...trap/engines/engine.py.229: process: 0.84035 | load: 0.00292 [I 2022-10-11 10:15:47] ...trap/engines/engine.py.230: loss: 0.10136 [I 2022-10-11 10:15:47] ...optimizers/trijoint.py.151: total_norm: 14.851274 [I 2022-10-11 10:15:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 1100/2304 [I 2022-10-11 10:15:55] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:50 [I 2022-10-11 10:15:55] ...trap/engines/engine.py.229: process: 0.83361 | load: 0.00291 [I 2022-10-11 10:15:55] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-11 10:15:55] ...optimizers/trijoint.py.151: total_norm: 17.038761 [I 2022-10-11 10:16:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1110/2304 [I 2022-10-11 10:16:03] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:16:37 [I 2022-10-11 10:16:03] ...trap/engines/engine.py.229: process: 0.83078 | load: 0.00294 [I 2022-10-11 10:16:03] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-11 10:16:03] ...optimizers/trijoint.py.151: total_norm: 14.509273 [I 2022-10-11 10:16:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 1120/2304 [I 2022-10-11 10:16:12] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:16:34 [I 2022-10-11 10:16:12] ...trap/engines/engine.py.229: process: 0.83504 | load: 0.00294 [I 2022-10-11 10:16:12] ...trap/engines/engine.py.230: loss: 0.13674 [I 2022-10-11 10:16:12] ...optimizers/trijoint.py.151: total_norm: 13.497918 [I 2022-10-11 10:16:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1130/2304 [I 2022-10-11 10:16:20] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:18 [I 2022-10-11 10:16:20] ...trap/engines/engine.py.229: process: 0.83385 | load: 0.00292 [I 2022-10-11 10:16:20] ...trap/engines/engine.py.230: loss: 0.15732 [I 2022-10-11 10:16:20] ...optimizers/trijoint.py.151: total_norm: 18.015867 [I 2022-10-11 10:16:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 1140/2304 [I 2022-10-11 10:16:29] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:15 [I 2022-10-11 10:16:29] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00292 [I 2022-10-11 10:16:29] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-11 10:16:29] ...optimizers/trijoint.py.151: total_norm: 26.875874 [I 2022-10-11 10:16:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 1150/2304 [I 2022-10-11 10:16:37] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:12 [I 2022-10-11 10:16:37] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00293 [I 2022-10-11 10:16:37] ...trap/engines/engine.py.230: loss: 0.19607 [I 2022-10-11 10:16:37] ...optimizers/trijoint.py.151: total_norm: 27.216021 [I 2022-10-11 10:16:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 1160/2304 [I 2022-10-11 10:16:46] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:07 [I 2022-10-11 10:16:46] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00292 [I 2022-10-11 10:16:46] ...trap/engines/engine.py.230: loss: 0.14358 [I 2022-10-11 10:16:46] ...optimizers/trijoint.py.151: total_norm: 17.827221 [I 2022-10-11 10:16:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 1170/2304 [I 2022-10-11 10:16:54] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:05 [I 2022-10-11 10:16:54] ...trap/engines/engine.py.229: process: 0.83876 | load: 0.00292 [I 2022-10-11 10:16:54] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-11 10:16:54] ...optimizers/trijoint.py.151: total_norm: 23.358549 [I 2022-10-11 10:17:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1180/2304 [I 2022-10-11 10:17:03] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:58 [I 2022-10-11 10:17:03] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00292 [I 2022-10-11 10:17:03] ...trap/engines/engine.py.230: loss: 0.15150 [I 2022-10-11 10:17:03] ...optimizers/trijoint.py.151: total_norm: 31.646048 [I 2022-10-11 10:17:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 1190/2304 [I 2022-10-11 10:17:11] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:15:46 [I 2022-10-11 10:17:11] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00294 [I 2022-10-11 10:17:11] ...trap/engines/engine.py.230: loss: 0.08850 [I 2022-10-11 10:17:11] ...optimizers/trijoint.py.151: total_norm: 15.313185 [I 2022-10-11 10:17:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1200/2304 [I 2022-10-11 10:17:20] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:30 [I 2022-10-11 10:17:20] ...trap/engines/engine.py.229: process: 0.83090 | load: 0.00294 [I 2022-10-11 10:17:20] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-11 10:17:20] ...optimizers/trijoint.py.151: total_norm: 13.317948 [I 2022-10-11 10:17:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 1210/2304 [I 2022-10-11 10:17:28] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:23 [I 2022-10-11 10:17:28] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00293 [I 2022-10-11 10:17:28] ...trap/engines/engine.py.230: loss: 0.09790 [I 2022-10-11 10:17:28] ...optimizers/trijoint.py.151: total_norm: 18.003916 [I 2022-10-11 10:17:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 1220/2304 [I 2022-10-11 10:17:37] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:40 [I 2022-10-11 10:17:37] ...trap/engines/engine.py.229: process: 0.86883 | load: 0.00369 [I 2022-10-11 10:17:37] ...trap/engines/engine.py.230: loss: 0.14098 [I 2022-10-11 10:17:37] ...optimizers/trijoint.py.151: total_norm: 29.725769 [I 2022-10-11 10:17:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 1230/2304 [I 2022-10-11 10:17:45] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:30 [I 2022-10-11 10:17:45] ...trap/engines/engine.py.229: process: 0.85355 | load: 0.00374 [I 2022-10-11 10:17:45] ...trap/engines/engine.py.230: loss: 0.11361 [I 2022-10-11 10:17:45] ...optimizers/trijoint.py.151: total_norm: 17.089489 [I 2022-10-11 10:17:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 1240/2304 [I 2022-10-11 10:17:54] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:25 [I 2022-10-11 10:17:54] ...trap/engines/engine.py.229: process: 0.87323 | load: 0.00373 [I 2022-10-11 10:17:54] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-11 10:17:54] ...optimizers/trijoint.py.151: total_norm: 15.133321 [I 2022-10-11 10:18:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1250/2304 [I 2022-10-11 10:18:03] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:15:13 [I 2022-10-11 10:18:03] ...trap/engines/engine.py.229: process: 0.86394 | load: 0.00378 [I 2022-10-11 10:18:03] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-11 10:18:03] ...optimizers/trijoint.py.151: total_norm: 56.586441 [I 2022-10-11 10:18:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 1260/2304 [I 2022-10-11 10:18:12] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:15:05 [I 2022-10-11 10:18:12] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00373 [I 2022-10-11 10:18:12] ...trap/engines/engine.py.230: loss: 0.08972 [I 2022-10-11 10:18:12] ...optimizers/trijoint.py.151: total_norm: 17.876150 [I 2022-10-11 10:18:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1270/2304 [I 2022-10-11 10:18:20] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:32 [I 2022-10-11 10:18:20] ...trap/engines/engine.py.229: process: 0.84363 | load: 0.00293 [I 2022-10-11 10:18:20] ...trap/engines/engine.py.230: loss: 0.09100 [I 2022-10-11 10:18:20] ...optimizers/trijoint.py.151: total_norm: 10.955283 [I 2022-10-11 10:18:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 1280/2304 [I 2022-10-11 10:18:28] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:18 [I 2022-10-11 10:18:28] ...trap/engines/engine.py.229: process: 0.83552 | load: 0.00295 [I 2022-10-11 10:18:28] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-11 10:18:28] ...optimizers/trijoint.py.151: total_norm: 13.153669 [I 2022-10-11 10:18:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 1290/2304 [I 2022-10-11 10:18:37] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:09 [I 2022-10-11 10:18:37] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00294 [I 2022-10-11 10:18:37] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-11 10:18:37] ...optimizers/trijoint.py.151: total_norm: 19.473501 [I 2022-10-11 10:18:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 1300/2304 [I 2022-10-11 10:18:45] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:01 [I 2022-10-11 10:18:45] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00294 [I 2022-10-11 10:18:45] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-11 10:18:45] ...optimizers/trijoint.py.151: total_norm: 19.456001 [I 2022-10-11 10:18:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 1310/2304 [I 2022-10-11 10:18:54] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:56 [I 2022-10-11 10:18:54] ...trap/engines/engine.py.229: process: 0.84377 | load: 0.00295 [I 2022-10-11 10:18:54] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-11 10:18:54] ...optimizers/trijoint.py.151: total_norm: 21.572090 [I 2022-10-11 10:19:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 1320/2304 [I 2022-10-11 10:19:02] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:13:43 [I 2022-10-11 10:19:02] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00294 [I 2022-10-11 10:19:02] ...trap/engines/engine.py.230: loss: 0.08219 [I 2022-10-11 10:19:02] ...optimizers/trijoint.py.151: total_norm: 9.085276 [I 2022-10-11 10:19:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 1330/2304 [I 2022-10-11 10:19:10] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:48 [I 2022-10-11 10:19:10] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00293 [I 2022-10-11 10:19:10] ...trap/engines/engine.py.230: loss: 0.16400 [I 2022-10-11 10:19:10] ...optimizers/trijoint.py.151: total_norm: 86.982788 [I 2022-10-11 10:19:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 1340/2304 [I 2022-10-11 10:19:19] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:34 [I 2022-10-11 10:19:19] ...trap/engines/engine.py.229: process: 0.83980 | load: 0.00294 [I 2022-10-11 10:19:19] ...trap/engines/engine.py.230: loss: 0.10059 [I 2022-10-11 10:19:19] ...optimizers/trijoint.py.151: total_norm: 14.928761 [I 2022-10-11 10:19:27] ...trap/engines/engine.py.224: train: epoch 101 | batch 1350/2304 [I 2022-10-11 10:19:27] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:13:28 [I 2022-10-11 10:19:27] ...trap/engines/engine.py.229: process: 0.85888 | load: 0.00292 [I 2022-10-11 10:19:27] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-11 10:19:27] ...optimizers/trijoint.py.151: total_norm: 9.300315 [I 2022-10-11 10:19:36] ...trap/engines/engine.py.224: train: epoch 101 | batch 1360/2304 [I 2022-10-11 10:19:36] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:13:16 [I 2022-10-11 10:19:36] ...trap/engines/engine.py.229: process: 0.84750 | load: 0.00292 [I 2022-10-11 10:19:36] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-11 10:19:36] ...optimizers/trijoint.py.151: total_norm: 17.325899 [I 2022-10-11 10:19:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 1370/2304 [I 2022-10-11 10:19:44] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:13:12 [I 2022-10-11 10:19:44] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00294 [I 2022-10-11 10:19:44] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-11 10:19:44] ...optimizers/trijoint.py.151: total_norm: 8.851342 [I 2022-10-11 10:19:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 1380/2304 [I 2022-10-11 10:19:53] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:13:03 [I 2022-10-11 10:19:53] ...trap/engines/engine.py.229: process: 0.85864 | load: 0.00295 [I 2022-10-11 10:19:53] ...trap/engines/engine.py.230: loss: 0.11068 [I 2022-10-11 10:19:53] ...optimizers/trijoint.py.151: total_norm: 19.714117 [I 2022-10-11 10:20:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 1390/2304 [I 2022-10-11 10:20:01] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:12:57 [I 2022-10-11 10:20:01] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00294 [I 2022-10-11 10:20:01] ...trap/engines/engine.py.230: loss: 0.08384 [I 2022-10-11 10:20:01] ...optimizers/trijoint.py.151: total_norm: 9.568177 [I 2022-10-11 10:20:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 1400/2304 [I 2022-10-11 10:20:10] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:12:49 [I 2022-10-11 10:20:10] ...trap/engines/engine.py.229: process: 0.84916 | load: 0.00293 [I 2022-10-11 10:20:10] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-11 10:20:10] ...optimizers/trijoint.py.151: total_norm: 11.255548 [I 2022-10-11 10:20:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 1410/2304 [I 2022-10-11 10:20:18] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:12:40 [I 2022-10-11 10:20:18] ...trap/engines/engine.py.229: process: 0.85088 | load: 0.00292 [I 2022-10-11 10:20:18] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-11 10:20:18] ...optimizers/trijoint.py.151: total_norm: 17.693167 [I 2022-10-11 10:20:27] ...trap/engines/engine.py.224: train: epoch 101 | batch 1420/2304 [I 2022-10-11 10:20:27] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:12:31 [I 2022-10-11 10:20:27] ...trap/engines/engine.py.229: process: 0.86184 | load: 0.00293 [I 2022-10-11 10:20:27] ...trap/engines/engine.py.230: loss: 0.11081 [I 2022-10-11 10:20:27] ...optimizers/trijoint.py.151: total_norm: 16.392424 [I 2022-10-11 10:20:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1430/2304 [I 2022-10-11 10:20:35] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:26 [I 2022-10-11 10:20:35] ...trap/engines/engine.py.229: process: 0.86636 | load: 0.00294 [I 2022-10-11 10:20:35] ...trap/engines/engine.py.230: loss: 0.14497 [I 2022-10-11 10:20:35] ...optimizers/trijoint.py.151: total_norm: 13.710135 [I 2022-10-11 10:20:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 1440/2304 [I 2022-10-11 10:20:44] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:12:18 [I 2022-10-11 10:20:44] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00294 [I 2022-10-11 10:20:44] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-11 10:20:44] ...optimizers/trijoint.py.151: total_norm: 13.788132 [I 2022-10-11 10:20:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1450/2304 [I 2022-10-11 10:20:52] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:12:06 [I 2022-10-11 10:20:52] ...trap/engines/engine.py.229: process: 0.85758 | load: 0.00293 [I 2022-10-11 10:20:52] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-11 10:20:52] ...optimizers/trijoint.py.151: total_norm: 22.301584 [I 2022-10-11 10:21:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 1460/2304 [I 2022-10-11 10:21:01] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:11:56 [I 2022-10-11 10:21:01] ...trap/engines/engine.py.229: process: 0.85241 | load: 0.00293 [I 2022-10-11 10:21:01] ...trap/engines/engine.py.230: loss: 0.06978 [I 2022-10-11 10:21:01] ...optimizers/trijoint.py.151: total_norm: 9.581296 [I 2022-10-11 10:21:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1470/2304 [I 2022-10-11 10:21:09] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:11:44 [I 2022-10-11 10:21:09] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00294 [I 2022-10-11 10:21:09] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-11 10:21:09] ...optimizers/trijoint.py.151: total_norm: 14.580010 [I 2022-10-11 10:21:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 1480/2304 [I 2022-10-11 10:21:18] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:11:31 [I 2022-10-11 10:21:18] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00292 [I 2022-10-11 10:21:18] ...trap/engines/engine.py.230: loss: 0.10014 [I 2022-10-11 10:21:18] ...optimizers/trijoint.py.151: total_norm: 23.504318 [I 2022-10-11 10:21:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 1490/2304 [I 2022-10-11 10:21:26] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:29 [I 2022-10-11 10:21:26] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00292 [I 2022-10-11 10:21:26] ...trap/engines/engine.py.230: loss: 0.16130 [I 2022-10-11 10:21:26] ...optimizers/trijoint.py.151: total_norm: 15.296349 [I 2022-10-11 10:21:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1500/2304 [I 2022-10-11 10:21:35] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:11:12 [I 2022-10-11 10:21:35] ...trap/engines/engine.py.229: process: 0.83272 | load: 0.00292 [I 2022-10-11 10:21:35] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-11 10:21:35] ...optimizers/trijoint.py.151: total_norm: 13.763367 [I 2022-10-11 10:21:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 1510/2304 [I 2022-10-11 10:21:43] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:11:08 [I 2022-10-11 10:21:43] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00292 [I 2022-10-11 10:21:43] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-11 10:21:43] ...optimizers/trijoint.py.151: total_norm: 20.580658 [I 2022-10-11 10:21:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1520/2304 [I 2022-10-11 10:21:52] ...trap/engines/engine.py.228: elapsed: 0:21:59 | left: 0:11:00 [I 2022-10-11 10:21:52] ...trap/engines/engine.py.229: process: 0.85170 | load: 0.00293 [I 2022-10-11 10:21:52] ...trap/engines/engine.py.230: loss: 0.16178 [I 2022-10-11 10:21:52] ...optimizers/trijoint.py.151: total_norm: 19.006062 [I 2022-10-11 10:22:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1530/2304 [I 2022-10-11 10:22:00] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:10:58 [I 2022-10-11 10:22:00] ...trap/engines/engine.py.229: process: 0.86642 | load: 0.00371 [I 2022-10-11 10:22:00] ...trap/engines/engine.py.230: loss: 0.08938 [I 2022-10-11 10:22:00] ...optimizers/trijoint.py.151: total_norm: 10.901658 [I 2022-10-11 10:22:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1540/2304 [I 2022-10-11 10:22:09] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:11:01 [I 2022-10-11 10:22:09] ...trap/engines/engine.py.229: process: 0.85874 | load: 0.00372 [I 2022-10-11 10:22:09] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-11 10:22:09] ...optimizers/trijoint.py.151: total_norm: 24.124378 [I 2022-10-11 10:22:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1550/2304 [I 2022-10-11 10:22:17] ...trap/engines/engine.py.228: elapsed: 0:22:25 | left: 0:10:58 [I 2022-10-11 10:22:17] ...trap/engines/engine.py.229: process: 0.87427 | load: 0.00374 [I 2022-10-11 10:22:17] ...trap/engines/engine.py.230: loss: 0.08265 [I 2022-10-11 10:22:17] ...optimizers/trijoint.py.151: total_norm: 17.471636 [I 2022-10-11 10:22:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 1560/2304 [I 2022-10-11 10:22:26] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:45 [I 2022-10-11 10:22:26] ...trap/engines/engine.py.229: process: 0.86509 | load: 0.00371 [I 2022-10-11 10:22:26] ...trap/engines/engine.py.230: loss: 0.14200 [I 2022-10-11 10:22:26] ...optimizers/trijoint.py.151: total_norm: 10.185724 [I 2022-10-11 10:22:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1570/2304 [I 2022-10-11 10:22:35] ...trap/engines/engine.py.228: elapsed: 0:22:42 | left: 0:10:29 [I 2022-10-11 10:22:35] ...trap/engines/engine.py.229: process: 0.82620 | load: 0.00291 [I 2022-10-11 10:22:35] ...trap/engines/engine.py.230: loss: 0.09246 [I 2022-10-11 10:22:35] ...optimizers/trijoint.py.151: total_norm: 16.248096 [I 2022-10-11 10:22:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 1580/2304 [I 2022-10-11 10:22:43] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:10:12 [I 2022-10-11 10:22:43] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00294 [I 2022-10-11 10:22:43] ...trap/engines/engine.py.230: loss: 0.09779 [I 2022-10-11 10:22:43] ...optimizers/trijoint.py.151: total_norm: 11.607515 [I 2022-10-11 10:22:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1590/2304 [I 2022-10-11 10:22:52] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:09:59 [I 2022-10-11 10:22:52] ...trap/engines/engine.py.229: process: 0.84173 | load: 0.00293 [I 2022-10-11 10:22:52] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-11 10:22:52] ...optimizers/trijoint.py.151: total_norm: 26.775425 [I 2022-10-11 10:23:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1600/2304 [I 2022-10-11 10:23:00] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:09:50 [I 2022-10-11 10:23:00] ...trap/engines/engine.py.229: process: 0.83098 | load: 0.00293 [I 2022-10-11 10:23:00] ...trap/engines/engine.py.230: loss: 0.07923 [I 2022-10-11 10:23:00] ...optimizers/trijoint.py.151: total_norm: 10.830859 [I 2022-10-11 10:23:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 1610/2304 [I 2022-10-11 10:23:08] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:41 [I 2022-10-11 10:23:08] ...trap/engines/engine.py.229: process: 0.82952 | load: 0.00293 [I 2022-10-11 10:23:08] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-11 10:23:08] ...optimizers/trijoint.py.151: total_norm: 15.312034 [I 2022-10-11 10:23:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1620/2304 [I 2022-10-11 10:23:17] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:33 [I 2022-10-11 10:23:17] ...trap/engines/engine.py.229: process: 0.84025 | load: 0.00292 [I 2022-10-11 10:23:17] ...trap/engines/engine.py.230: loss: 0.08184 [I 2022-10-11 10:23:17] ...optimizers/trijoint.py.151: total_norm: 11.865485 [I 2022-10-11 10:23:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 1630/2304 [I 2022-10-11 10:23:25] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:31 [I 2022-10-11 10:23:25] ...trap/engines/engine.py.229: process: 0.86262 | load: 0.00297 [I 2022-10-11 10:23:25] ...trap/engines/engine.py.230: loss: 0.08784 [I 2022-10-11 10:23:25] ...optimizers/trijoint.py.151: total_norm: 9.007880 [I 2022-10-11 10:23:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1640/2304 [I 2022-10-11 10:23:34] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:25 [I 2022-10-11 10:23:34] ...trap/engines/engine.py.229: process: 0.84282 | load: 0.00294 [I 2022-10-11 10:23:34] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-11 10:23:34] ...optimizers/trijoint.py.151: total_norm: 13.428405 [I 2022-10-11 10:23:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 1650/2304 [I 2022-10-11 10:23:42] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:17 [I 2022-10-11 10:23:42] ...trap/engines/engine.py.229: process: 0.84939 | load: 0.00292 [I 2022-10-11 10:23:42] ...trap/engines/engine.py.230: loss: 0.09142 [I 2022-10-11 10:23:42] ...optimizers/trijoint.py.151: total_norm: 10.020135 [I 2022-10-11 10:23:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 1660/2304 [I 2022-10-11 10:23:51] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:08 [I 2022-10-11 10:23:51] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00295 [I 2022-10-11 10:23:51] ...trap/engines/engine.py.230: loss: 0.11456 [I 2022-10-11 10:23:51] ...optimizers/trijoint.py.151: total_norm: 13.561423 [I 2022-10-11 10:24:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1670/2304 [I 2022-10-11 10:24:00] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:58 [I 2022-10-11 10:24:00] ...trap/engines/engine.py.229: process: 0.84144 | load: 0.00294 [I 2022-10-11 10:24:00] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-11 10:24:00] ...optimizers/trijoint.py.151: total_norm: 11.468123 [I 2022-10-11 10:24:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 1680/2304 [I 2022-10-11 10:24:08] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:56 [I 2022-10-11 10:24:08] ...trap/engines/engine.py.229: process: 0.86021 | load: 0.00295 [I 2022-10-11 10:24:08] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-11 10:24:08] ...optimizers/trijoint.py.151: total_norm: 23.053370 [I 2022-10-11 10:24:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1690/2304 [I 2022-10-11 10:24:17] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:49 [I 2022-10-11 10:24:17] ...trap/engines/engine.py.229: process: 0.85936 | load: 0.00296 [I 2022-10-11 10:24:17] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-11 10:24:17] ...optimizers/trijoint.py.151: total_norm: 12.663942 [I 2022-10-11 10:24:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 1700/2304 [I 2022-10-11 10:24:25] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:33 [I 2022-10-11 10:24:25] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00294 [I 2022-10-11 10:24:25] ...trap/engines/engine.py.230: loss: 0.08128 [I 2022-10-11 10:24:25] ...optimizers/trijoint.py.151: total_norm: 16.890600 [I 2022-10-11 10:24:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1710/2304 [I 2022-10-11 10:24:34] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:24 [I 2022-10-11 10:24:34] ...trap/engines/engine.py.229: process: 0.84049 | load: 0.00293 [I 2022-10-11 10:24:34] ...trap/engines/engine.py.230: loss: 0.14774 [I 2022-10-11 10:24:34] ...optimizers/trijoint.py.151: total_norm: 19.223276 [I 2022-10-11 10:24:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 1720/2304 [I 2022-10-11 10:24:42] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:19 [I 2022-10-11 10:24:42] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00292 [I 2022-10-11 10:24:42] ...trap/engines/engine.py.230: loss: 0.11308 [I 2022-10-11 10:24:42] ...optimizers/trijoint.py.151: total_norm: 12.157907 [I 2022-10-11 10:24:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 1730/2304 [I 2022-10-11 10:24:51] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:08:06 [I 2022-10-11 10:24:51] ...trap/engines/engine.py.229: process: 0.84262 | load: 0.00293 [I 2022-10-11 10:24:51] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-11 10:24:51] ...optimizers/trijoint.py.151: total_norm: 33.043304 [I 2022-10-11 10:24:59] ...trap/engines/engine.py.224: train: epoch 101 | batch 1740/2304 [I 2022-10-11 10:24:59] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:00 [I 2022-10-11 10:24:59] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00294 [I 2022-10-11 10:24:59] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-11 10:24:59] ...optimizers/trijoint.py.151: total_norm: 10.195455 [I 2022-10-11 10:25:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 1750/2304 [I 2022-10-11 10:25:08] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:49 [I 2022-10-11 10:25:08] ...trap/engines/engine.py.229: process: 0.85482 | load: 0.00292 [I 2022-10-11 10:25:08] ...trap/engines/engine.py.230: loss: 0.10751 [I 2022-10-11 10:25:08] ...optimizers/trijoint.py.151: total_norm: 12.720967 [I 2022-10-11 10:25:16] ...trap/engines/engine.py.224: train: epoch 101 | batch 1760/2304 [I 2022-10-11 10:25:16] ...trap/engines/engine.py.228: elapsed: 0:25:24 | left: 0:07:50 [I 2022-10-11 10:25:16] ...trap/engines/engine.py.229: process: 0.88281 | load: 0.00374 [I 2022-10-11 10:25:16] ...trap/engines/engine.py.230: loss: 0.10964 [I 2022-10-11 10:25:16] ...optimizers/trijoint.py.151: total_norm: 26.655094 [I 2022-10-11 10:25:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 1770/2304 [I 2022-10-11 10:25:25] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:48 [I 2022-10-11 10:25:25] ...trap/engines/engine.py.229: process: 0.88441 | load: 0.00372 [I 2022-10-11 10:25:25] ...trap/engines/engine.py.230: loss: 0.11669 [I 2022-10-11 10:25:25] ...optimizers/trijoint.py.151: total_norm: 25.492506 [I 2022-10-11 10:25:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1780/2304 [I 2022-10-11 10:25:34] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:37 [I 2022-10-11 10:25:34] ...trap/engines/engine.py.229: process: 0.86730 | load: 0.00372 [I 2022-10-11 10:25:34] ...trap/engines/engine.py.230: loss: 0.11630 [I 2022-10-11 10:25:34] ...optimizers/trijoint.py.151: total_norm: 18.524679 [I 2022-10-11 10:25:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 1790/2304 [I 2022-10-11 10:25:43] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:07:27 [I 2022-10-11 10:25:43] ...trap/engines/engine.py.229: process: 0.88570 | load: 0.00374 [I 2022-10-11 10:25:43] ...trap/engines/engine.py.230: loss: 0.09226 [I 2022-10-11 10:25:43] ...optimizers/trijoint.py.151: total_norm: 10.020373 [I 2022-10-11 10:25:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 1800/2304 [I 2022-10-11 10:25:51] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:18 [I 2022-10-11 10:25:51] ...trap/engines/engine.py.229: process: 0.87264 | load: 0.00374 [I 2022-10-11 10:25:51] ...trap/engines/engine.py.230: loss: 0.16041 [I 2022-10-11 10:25:51] ...optimizers/trijoint.py.151: total_norm: 15.415661 [I 2022-10-11 10:26:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1810/2304 [I 2022-10-11 10:26:00] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:07:09 [I 2022-10-11 10:26:00] ...trap/engines/engine.py.229: process: 0.87350 | load: 0.00374 [I 2022-10-11 10:26:00] ...trap/engines/engine.py.230: loss: 0.15631 [I 2022-10-11 10:26:00] ...optimizers/trijoint.py.151: total_norm: 13.739594 [I 2022-10-11 10:26:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1820/2304 [I 2022-10-11 10:26:09] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:07:01 [I 2022-10-11 10:26:09] ...trap/engines/engine.py.229: process: 0.86637 | load: 0.00373 [I 2022-10-11 10:26:09] ...trap/engines/engine.py.230: loss: 0.10393 [I 2022-10-11 10:26:09] ...optimizers/trijoint.py.151: total_norm: 13.544785 [I 2022-10-11 10:26:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 1830/2304 [I 2022-10-11 10:26:18] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:54 [I 2022-10-11 10:26:18] ...trap/engines/engine.py.229: process: 0.87647 | load: 0.00372 [I 2022-10-11 10:26:18] ...trap/engines/engine.py.230: loss: 0.10250 [I 2022-10-11 10:26:18] ...optimizers/trijoint.py.151: total_norm: 12.459167 [I 2022-10-11 10:26:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 1840/2304 [I 2022-10-11 10:26:26] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:42 [I 2022-10-11 10:26:26] ...trap/engines/engine.py.229: process: 0.87007 | load: 0.00372 [I 2022-10-11 10:26:26] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-11 10:26:26] ...optimizers/trijoint.py.151: total_norm: 25.065489 [I 2022-10-11 10:26:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1850/2304 [I 2022-10-11 10:26:35] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:37 [I 2022-10-11 10:26:35] ...trap/engines/engine.py.229: process: 0.87767 | load: 0.00371 [I 2022-10-11 10:26:35] ...trap/engines/engine.py.230: loss: 0.09174 [I 2022-10-11 10:26:35] ...optimizers/trijoint.py.151: total_norm: 17.525177 [I 2022-10-11 10:26:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 1860/2304 [I 2022-10-11 10:26:44] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:24 [I 2022-10-11 10:26:44] ...trap/engines/engine.py.229: process: 0.87381 | load: 0.00374 [I 2022-10-11 10:26:44] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-11 10:26:44] ...optimizers/trijoint.py.151: total_norm: 19.625877 [I 2022-10-11 10:26:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1870/2304 [I 2022-10-11 10:26:52] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:18 [I 2022-10-11 10:26:52] ...trap/engines/engine.py.229: process: 0.87312 | load: 0.00371 [I 2022-10-11 10:26:52] ...trap/engines/engine.py.230: loss: 0.09468 [I 2022-10-11 10:26:52] ...optimizers/trijoint.py.151: total_norm: 11.768148 [I 2022-10-11 10:27:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 1880/2304 [I 2022-10-11 10:27:01] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:06:07 [I 2022-10-11 10:27:01] ...trap/engines/engine.py.229: process: 0.86289 | load: 0.00371 [I 2022-10-11 10:27:01] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-11 10:27:01] ...optimizers/trijoint.py.151: total_norm: 16.424887 [I 2022-10-11 10:27:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 1890/2304 [I 2022-10-11 10:27:10] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:06:00 [I 2022-10-11 10:27:10] ...trap/engines/engine.py.229: process: 0.86854 | load: 0.00372 [I 2022-10-11 10:27:10] ...trap/engines/engine.py.230: loss: 0.14106 [I 2022-10-11 10:27:10] ...optimizers/trijoint.py.151: total_norm: 15.786112 [I 2022-10-11 10:27:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 1900/2304 [I 2022-10-11 10:27:18] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:45 [I 2022-10-11 10:27:18] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00292 [I 2022-10-11 10:27:18] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-11 10:27:18] ...optimizers/trijoint.py.151: total_norm: 18.759600 [I 2022-10-11 10:27:27] ...trap/engines/engine.py.224: train: epoch 101 | batch 1910/2304 [I 2022-10-11 10:27:27] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:31 [I 2022-10-11 10:27:27] ...trap/engines/engine.py.229: process: 0.83545 | load: 0.00293 [I 2022-10-11 10:27:27] ...trap/engines/engine.py.230: loss: 0.08293 [I 2022-10-11 10:27:27] ...optimizers/trijoint.py.151: total_norm: 13.670406 [I 2022-10-11 10:27:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1920/2304 [I 2022-10-11 10:27:35] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:21 [I 2022-10-11 10:27:35] ...trap/engines/engine.py.229: process: 0.84366 | load: 0.00294 [I 2022-10-11 10:27:35] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 10:27:35] ...optimizers/trijoint.py.151: total_norm: 14.585629 [I 2022-10-11 10:27:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 1930/2304 [I 2022-10-11 10:27:44] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:12 [I 2022-10-11 10:27:44] ...trap/engines/engine.py.229: process: 0.83307 | load: 0.00294 [I 2022-10-11 10:27:44] ...trap/engines/engine.py.230: loss: 0.10936 [I 2022-10-11 10:27:44] ...optimizers/trijoint.py.151: total_norm: 21.556189 [I 2022-10-11 10:27:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1940/2304 [I 2022-10-11 10:27:52] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:05:04 [I 2022-10-11 10:27:52] ...trap/engines/engine.py.229: process: 0.83527 | load: 0.00292 [I 2022-10-11 10:27:52] ...trap/engines/engine.py.230: loss: 0.12870 [I 2022-10-11 10:27:52] ...optimizers/trijoint.py.151: total_norm: 12.834065 [I 2022-10-11 10:28:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1950/2304 [I 2022-10-11 10:28:00] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:56 [I 2022-10-11 10:28:00] ...trap/engines/engine.py.229: process: 0.83913 | load: 0.00292 [I 2022-10-11 10:28:00] ...trap/engines/engine.py.230: loss: 0.09016 [I 2022-10-11 10:28:00] ...optimizers/trijoint.py.151: total_norm: 21.938705 [I 2022-10-11 10:28:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1960/2304 [I 2022-10-11 10:28:09] ...trap/engines/engine.py.228: elapsed: 0:28:16 | left: 0:04:48 [I 2022-10-11 10:28:09] ...trap/engines/engine.py.229: process: 0.83395 | load: 0.00294 [I 2022-10-11 10:28:09] ...trap/engines/engine.py.230: loss: 0.13353 [I 2022-10-11 10:28:09] ...optimizers/trijoint.py.151: total_norm: 15.086871 [I 2022-10-11 10:28:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1970/2304 [I 2022-10-11 10:28:17] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:40 [I 2022-10-11 10:28:17] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00294 [I 2022-10-11 10:28:17] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-11 10:28:17] ...optimizers/trijoint.py.151: total_norm: 18.221451 [I 2022-10-11 10:28:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 1980/2304 [I 2022-10-11 10:28:26] ...trap/engines/engine.py.228: elapsed: 0:28:33 | left: 0:04:32 [I 2022-10-11 10:28:26] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00293 [I 2022-10-11 10:28:26] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-11 10:28:26] ...optimizers/trijoint.py.151: total_norm: 23.660061 [I 2022-10-11 10:28:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1990/2304 [I 2022-10-11 10:28:34] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:23 [I 2022-10-11 10:28:34] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00292 [I 2022-10-11 10:28:34] ...trap/engines/engine.py.230: loss: 0.18155 [I 2022-10-11 10:28:34] ...optimizers/trijoint.py.151: total_norm: 34.627632 [I 2022-10-11 10:28:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 2000/2304 [I 2022-10-11 10:28:42] ...trap/engines/engine.py.228: elapsed: 0:28:50 | left: 0:04:14 [I 2022-10-11 10:28:42] ...trap/engines/engine.py.229: process: 0.84863 | load: 0.00292 [I 2022-10-11 10:28:42] ...trap/engines/engine.py.230: loss: 0.08507 [I 2022-10-11 10:28:42] ...optimizers/trijoint.py.151: total_norm: 19.118954 [I 2022-10-11 10:28:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 2010/2304 [I 2022-10-11 10:28:51] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:06 [I 2022-10-11 10:28:51] ...trap/engines/engine.py.229: process: 0.84991 | load: 0.00292 [I 2022-10-11 10:28:51] ...trap/engines/engine.py.230: loss: 0.10795 [I 2022-10-11 10:28:51] ...optimizers/trijoint.py.151: total_norm: 19.701096 [I 2022-10-11 10:28:59] ...trap/engines/engine.py.224: train: epoch 101 | batch 2020/2304 [I 2022-10-11 10:28:59] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:58 [I 2022-10-11 10:28:59] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00292 [I 2022-10-11 10:28:59] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-11 10:28:59] ...optimizers/trijoint.py.151: total_norm: 16.746990 [I 2022-10-11 10:29:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 2030/2304 [I 2022-10-11 10:29:08] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:49 [I 2022-10-11 10:29:08] ...trap/engines/engine.py.229: process: 0.84158 | load: 0.00293 [I 2022-10-11 10:29:08] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 10:29:08] ...optimizers/trijoint.py.151: total_norm: 9.359060 [I 2022-10-11 10:29:16] ...trap/engines/engine.py.224: train: epoch 101 | batch 2040/2304 [I 2022-10-11 10:29:16] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:41 [I 2022-10-11 10:29:16] ...trap/engines/engine.py.229: process: 0.83052 | load: 0.00291 [I 2022-10-11 10:29:16] ...trap/engines/engine.py.230: loss: 0.10207 [I 2022-10-11 10:29:16] ...optimizers/trijoint.py.151: total_norm: 13.774139 [I 2022-10-11 10:29:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 2050/2304 [I 2022-10-11 10:29:24] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:33 [I 2022-10-11 10:29:24] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00293 [I 2022-10-11 10:29:24] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-11 10:29:24] ...optimizers/trijoint.py.151: total_norm: 16.513329 [I 2022-10-11 10:29:33] ...trap/engines/engine.py.224: train: epoch 101 | batch 2060/2304 [I 2022-10-11 10:29:33] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:03:25 [I 2022-10-11 10:29:33] ...trap/engines/engine.py.229: process: 0.84211 | load: 0.00291 [I 2022-10-11 10:29:33] ...trap/engines/engine.py.230: loss: 0.09474 [I 2022-10-11 10:29:33] ...optimizers/trijoint.py.151: total_norm: 11.531661 [I 2022-10-11 10:29:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 2070/2304 [I 2022-10-11 10:29:41] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:03:16 [I 2022-10-11 10:29:41] ...trap/engines/engine.py.229: process: 0.83318 | load: 0.00293 [I 2022-10-11 10:29:41] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-11 10:29:41] ...optimizers/trijoint.py.151: total_norm: 11.426904 [I 2022-10-11 10:29:50] ...trap/engines/engine.py.224: train: epoch 101 | batch 2080/2304 [I 2022-10-11 10:29:50] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:09 [I 2022-10-11 10:29:50] ...trap/engines/engine.py.229: process: 0.83293 | load: 0.00294 [I 2022-10-11 10:29:50] ...trap/engines/engine.py.230: loss: 0.18279 [I 2022-10-11 10:29:50] ...optimizers/trijoint.py.151: total_norm: 27.676250 [I 2022-10-11 10:29:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 2090/2304 [I 2022-10-11 10:29:58] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:59 [I 2022-10-11 10:29:58] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00292 [I 2022-10-11 10:29:58] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-11 10:29:58] ...optimizers/trijoint.py.151: total_norm: 21.893785 [I 2022-10-11 10:30:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 2100/2304 [I 2022-10-11 10:30:07] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:51 [I 2022-10-11 10:30:07] ...trap/engines/engine.py.229: process: 0.82526 | load: 0.00294 [I 2022-10-11 10:30:07] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-11 10:30:07] ...optimizers/trijoint.py.151: total_norm: 20.771723 [I 2022-10-11 10:30:15] ...trap/engines/engine.py.224: train: epoch 101 | batch 2110/2304 [I 2022-10-11 10:30:15] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:44 [I 2022-10-11 10:30:15] ...trap/engines/engine.py.229: process: 0.87042 | load: 0.00296 [I 2022-10-11 10:30:15] ...trap/engines/engine.py.230: loss: 0.11281 [I 2022-10-11 10:30:15] ...optimizers/trijoint.py.151: total_norm: 8.482667 [I 2022-10-11 10:30:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 2120/2304 [I 2022-10-11 10:30:24] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:37 [I 2022-10-11 10:30:24] ...trap/engines/engine.py.229: process: 0.84650 | load: 0.00293 [I 2022-10-11 10:30:24] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-11 10:30:24] ...optimizers/trijoint.py.151: total_norm: 26.343996 [I 2022-10-11 10:30:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 2130/2304 [I 2022-10-11 10:30:32] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:27 [I 2022-10-11 10:30:32] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00295 [I 2022-10-11 10:30:32] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-11 10:30:32] ...optimizers/trijoint.py.151: total_norm: 18.277634 [I 2022-10-11 10:30:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 2140/2304 [I 2022-10-11 10:30:41] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:18 [I 2022-10-11 10:30:41] ...trap/engines/engine.py.229: process: 0.82925 | load: 0.00293 [I 2022-10-11 10:30:41] ...trap/engines/engine.py.230: loss: 0.12190 [I 2022-10-11 10:30:41] ...optimizers/trijoint.py.151: total_norm: 20.851336 [I 2022-10-11 10:30:49] ...trap/engines/engine.py.224: train: epoch 101 | batch 2150/2304 [I 2022-10-11 10:30:49] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:02:09 [I 2022-10-11 10:30:49] ...trap/engines/engine.py.229: process: 0.84247 | load: 0.00293 [I 2022-10-11 10:30:49] ...trap/engines/engine.py.230: loss: 0.16669 [I 2022-10-11 10:30:49] ...optimizers/trijoint.py.151: total_norm: 22.509567 [I 2022-10-11 10:30:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 2160/2304 [I 2022-10-11 10:30:57] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:02:00 [I 2022-10-11 10:30:57] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00293 [I 2022-10-11 10:30:57] ...trap/engines/engine.py.230: loss: 0.08524 [I 2022-10-11 10:30:57] ...optimizers/trijoint.py.151: total_norm: 10.210395 [I 2022-10-11 10:31:06] ...trap/engines/engine.py.224: train: epoch 101 | batch 2170/2304 [I 2022-10-11 10:31:06] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:52 [I 2022-10-11 10:31:06] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00294 [I 2022-10-11 10:31:06] ...trap/engines/engine.py.230: loss: 0.14037 [I 2022-10-11 10:31:06] ...optimizers/trijoint.py.151: total_norm: 26.562042 [I 2022-10-11 10:31:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 2180/2304 [I 2022-10-11 10:31:14] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:42 [I 2022-10-11 10:31:14] ...trap/engines/engine.py.229: process: 0.82190 | load: 0.00295 [I 2022-10-11 10:31:14] ...trap/engines/engine.py.230: loss: 0.09285 [I 2022-10-11 10:31:14] ...optimizers/trijoint.py.151: total_norm: 16.379448 [I 2022-10-11 10:31:23] ...trap/engines/engine.py.224: train: epoch 101 | batch 2190/2304 [I 2022-10-11 10:31:23] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:35 [I 2022-10-11 10:31:23] ...trap/engines/engine.py.229: process: 0.84059 | load: 0.00294 [I 2022-10-11 10:31:23] ...trap/engines/engine.py.230: loss: 0.09738 [I 2022-10-11 10:31:23] ...optimizers/trijoint.py.151: total_norm: 19.114498 [I 2022-10-11 10:31:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 2200/2304 [I 2022-10-11 10:31:31] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:01:27 [I 2022-10-11 10:31:31] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00294 [I 2022-10-11 10:31:31] ...trap/engines/engine.py.230: loss: 0.10530 [I 2022-10-11 10:31:31] ...optimizers/trijoint.py.151: total_norm: 17.596228 [I 2022-10-11 10:31:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 2210/2304 [I 2022-10-11 10:31:39] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:18 [I 2022-10-11 10:31:39] ...trap/engines/engine.py.229: process: 0.83555 | load: 0.00295 [I 2022-10-11 10:31:39] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-11 10:31:39] ...optimizers/trijoint.py.151: total_norm: 21.050772 [I 2022-10-11 10:31:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 2220/2304 [I 2022-10-11 10:31:48] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:10 [I 2022-10-11 10:31:48] ...trap/engines/engine.py.229: process: 0.83381 | load: 0.00294 [I 2022-10-11 10:31:48] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-11 10:31:48] ...optimizers/trijoint.py.151: total_norm: 23.147423 [I 2022-10-11 10:31:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 2230/2304 [I 2022-10-11 10:31:56] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:01:02 [I 2022-10-11 10:31:56] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00294 [I 2022-10-11 10:31:56] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-11 10:31:56] ...optimizers/trijoint.py.151: total_norm: 10.633076 [I 2022-10-11 10:32:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 2240/2304 [I 2022-10-11 10:32:05] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:53 [I 2022-10-11 10:32:05] ...trap/engines/engine.py.229: process: 0.83009 | load: 0.00296 [I 2022-10-11 10:32:05] ...trap/engines/engine.py.230: loss: 0.13417 [I 2022-10-11 10:32:05] ...optimizers/trijoint.py.151: total_norm: 12.137683 [I 2022-10-11 10:32:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 2250/2304 [I 2022-10-11 10:32:13] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:45 [I 2022-10-11 10:32:13] ...trap/engines/engine.py.229: process: 0.84012 | load: 0.00295 [I 2022-10-11 10:32:13] ...trap/engines/engine.py.230: loss: 0.10911 [I 2022-10-11 10:32:13] ...optimizers/trijoint.py.151: total_norm: 20.473091 [I 2022-10-11 10:32:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 2260/2304 [I 2022-10-11 10:32:21] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:37 [I 2022-10-11 10:32:21] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00293 [I 2022-10-11 10:32:21] ...trap/engines/engine.py.230: loss: 0.10505 [I 2022-10-11 10:32:21] ...optimizers/trijoint.py.151: total_norm: 9.770909 [I 2022-10-11 10:32:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 2270/2304 [I 2022-10-11 10:32:30] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:28 [I 2022-10-11 10:32:30] ...trap/engines/engine.py.229: process: 0.84576 | load: 0.00294 [I 2022-10-11 10:32:30] ...trap/engines/engine.py.230: loss: 0.09660 [I 2022-10-11 10:32:30] ...optimizers/trijoint.py.151: total_norm: 12.351073 [I 2022-10-11 10:32:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 2280/2304 [I 2022-10-11 10:32:38] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:20 [I 2022-10-11 10:32:38] ...trap/engines/engine.py.229: process: 0.83355 | load: 0.00292 [I 2022-10-11 10:32:38] ...trap/engines/engine.py.230: loss: 0.16961 [I 2022-10-11 10:32:38] ...optimizers/trijoint.py.151: total_norm: 14.247366 [I 2022-10-11 10:32:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 2290/2304 [I 2022-10-11 10:32:47] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:11 [I 2022-10-11 10:32:47] ...trap/engines/engine.py.229: process: 0.84471 | load: 0.00295 [I 2022-10-11 10:32:47] ...trap/engines/engine.py.230: loss: 0.10684 [I 2022-10-11 10:32:47] ...optimizers/trijoint.py.151: total_norm: 27.027981 [I 2022-10-11 10:32:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 2300/2304 [I 2022-10-11 10:32:55] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:03 [I 2022-10-11 10:32:55] ...trap/engines/engine.py.229: process: 0.83447 | load: 0.00012 [I 2022-10-11 10:32:55] ...trap/engines/engine.py.230: loss: 0.09680 [I 2022-10-11 10:32:55] ...optimizers/trijoint.py.151: total_norm: 13.217403 [I 2022-10-11 10:32:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 2304/2304 [I 2022-10-11 10:32:58] ...trap/engines/engine.py.228: elapsed: 0:33:05 | left: 0:00:00 [I 2022-10-11 10:32:58] ...trap/engines/engine.py.229: process: 0.81363 | load: 0.00012 [I 2022-10-11 10:32:58] ...trap/engines/engine.py.230: loss: 0.15073 [I 2022-10-11 10:32:58] ...optimizers/trijoint.py.151: total_norm: 22.981209 [S 2022-10-11 10:33:00] ...trap/engines/engine.py.129: train_epoch.epoch: 101 [S 2022-10-11 10:33:00] ...trap/engines/engine.py.129: train_epoch.loss: 0.12270280403002223 [I 2022-10-11 10:33:00] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 10:33:01] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 101 [I 2022-10-11 10:33:01] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 10:33:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 10:33:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 10:33:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 10:33:06] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 10:33:07] ...trap/engines/engine.py.321: eval: epoch 101 | batch 0/510 [I 2022-10-11 10:33:07] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:52 [I 2022-10-11 10:33:07] ...trap/engines/engine.py.326: process: 5.75006 | load: 4.82119 [I 2022-10-11 10:33:12] ...trap/engines/engine.py.321: eval: epoch 101 | batch 10/510 [I 2022-10-11 10:33:12] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:46 [I 2022-10-11 10:33:12] ...trap/engines/engine.py.326: process: 0.46824 | load: 0.00019 [I 2022-10-11 10:33:16] ...trap/engines/engine.py.321: eval: epoch 101 | batch 20/510 [I 2022-10-11 10:33:16] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:14 [I 2022-10-11 10:33:16] ...trap/engines/engine.py.326: process: 0.44521 | load: 0.00019 [I 2022-10-11 10:33:21] ...trap/engines/engine.py.321: eval: epoch 101 | batch 30/510 [I 2022-10-11 10:33:21] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:03:34 [I 2022-10-11 10:33:21] ...trap/engines/engine.py.326: process: 0.43332 | load: 0.00019 [I 2022-10-11 10:33:25] ...trap/engines/engine.py.321: eval: epoch 101 | batch 40/510 [I 2022-10-11 10:33:25] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:03:27 [I 2022-10-11 10:33:25] ...trap/engines/engine.py.326: process: 0.44681 | load: 0.00019 [I 2022-10-11 10:33:29] ...trap/engines/engine.py.321: eval: epoch 101 | batch 50/510 [I 2022-10-11 10:33:29] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:03:22 [I 2022-10-11 10:33:29] ...trap/engines/engine.py.326: process: 0.44166 | load: 0.00020 [I 2022-10-11 10:33:34] ...trap/engines/engine.py.321: eval: epoch 101 | batch 60/510 [I 2022-10-11 10:33:34] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:41 [I 2022-10-11 10:33:34] ...trap/engines/engine.py.326: process: 0.53185 | load: 0.00018 [I 2022-10-11 10:33:40] ...trap/engines/engine.py.321: eval: epoch 101 | batch 70/510 [I 2022-10-11 10:33:40] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:04:16 [I 2022-10-11 10:33:40] ...trap/engines/engine.py.326: process: 0.55326 | load: 0.00018 [I 2022-10-11 10:33:46] ...trap/engines/engine.py.321: eval: epoch 101 | batch 80/510 [I 2022-10-11 10:33:46] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:54 [I 2022-10-11 10:33:46] ...trap/engines/engine.py.326: process: 0.55898 | load: 0.00018 [I 2022-10-11 10:33:50] ...trap/engines/engine.py.321: eval: epoch 101 | batch 90/510 [I 2022-10-11 10:33:50] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:12 [I 2022-10-11 10:33:50] ...trap/engines/engine.py.326: process: 0.46978 | load: 0.00018 [I 2022-10-11 10:33:55] ...trap/engines/engine.py.321: eval: epoch 101 | batch 100/510 [I 2022-10-11 10:33:55] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:00 [I 2022-10-11 10:33:55] ...trap/engines/engine.py.326: process: 0.30555 | load: 0.00018 [I 2022-10-11 10:33:58] ...trap/engines/engine.py.321: eval: epoch 101 | batch 110/510 [I 2022-10-11 10:33:58] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:10 [I 2022-10-11 10:33:58] ...trap/engines/engine.py.326: process: 0.31162 | load: 0.00018 [I 2022-10-11 10:34:01] ...trap/engines/engine.py.321: eval: epoch 101 | batch 120/510 [I 2022-10-11 10:34:01] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:02 [I 2022-10-11 10:34:01] ...trap/engines/engine.py.326: process: 0.31423 | load: 0.00019 [I 2022-10-11 10:34:04] ...trap/engines/engine.py.321: eval: epoch 101 | batch 130/510 [I 2022-10-11 10:34:04] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:59 [I 2022-10-11 10:34:04] ...trap/engines/engine.py.326: process: 0.31594 | load: 0.00018 [I 2022-10-11 10:34:07] ...trap/engines/engine.py.321: eval: epoch 101 | batch 140/510 [I 2022-10-11 10:34:07] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:56 [I 2022-10-11 10:34:07] ...trap/engines/engine.py.326: process: 0.31176 | load: 0.00018 [I 2022-10-11 10:34:10] ...trap/engines/engine.py.321: eval: epoch 101 | batch 150/510 [I 2022-10-11 10:34:10] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:53 [I 2022-10-11 10:34:10] ...trap/engines/engine.py.326: process: 0.31437 | load: 0.00018 [I 2022-10-11 10:34:13] ...trap/engines/engine.py.321: eval: epoch 101 | batch 160/510 [I 2022-10-11 10:34:13] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:49 [I 2022-10-11 10:34:13] ...trap/engines/engine.py.326: process: 0.31348 | load: 0.00019 [I 2022-10-11 10:34:17] ...trap/engines/engine.py.321: eval: epoch 101 | batch 170/510 [I 2022-10-11 10:34:17] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:46 [I 2022-10-11 10:34:17] ...trap/engines/engine.py.326: process: 0.30836 | load: 0.00019 [I 2022-10-11 10:34:20] ...trap/engines/engine.py.321: eval: epoch 101 | batch 180/510 [I 2022-10-11 10:34:20] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:43 [I 2022-10-11 10:34:20] ...trap/engines/engine.py.326: process: 0.31668 | load: 0.00019 [I 2022-10-11 10:34:23] ...trap/engines/engine.py.321: eval: epoch 101 | batch 190/510 [I 2022-10-11 10:34:23] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:39 [I 2022-10-11 10:34:23] ...trap/engines/engine.py.326: process: 0.30723 | load: 0.00018 [I 2022-10-11 10:34:26] ...trap/engines/engine.py.321: eval: epoch 101 | batch 200/510 [I 2022-10-11 10:34:26] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:37 [I 2022-10-11 10:34:26] ...trap/engines/engine.py.326: process: 0.31823 | load: 0.00019 [I 2022-10-11 10:34:29] ...trap/engines/engine.py.321: eval: epoch 101 | batch 210/510 [I 2022-10-11 10:34:29] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:33 [I 2022-10-11 10:34:29] ...trap/engines/engine.py.326: process: 0.31446 | load: 0.00019 [I 2022-10-11 10:34:32] ...trap/engines/engine.py.321: eval: epoch 101 | batch 220/510 [I 2022-10-11 10:34:32] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:30 [I 2022-10-11 10:34:32] ...trap/engines/engine.py.326: process: 0.31404 | load: 0.00019 [I 2022-10-11 10:34:35] ...trap/engines/engine.py.321: eval: epoch 101 | batch 230/510 [I 2022-10-11 10:34:35] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:28 [I 2022-10-11 10:34:35] ...trap/engines/engine.py.326: process: 0.31744 | load: 0.00018 [I 2022-10-11 10:34:39] ...trap/engines/engine.py.321: eval: epoch 101 | batch 240/510 [I 2022-10-11 10:34:39] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:24 [I 2022-10-11 10:34:39] ...trap/engines/engine.py.326: process: 0.30897 | load: 0.00019 [I 2022-10-11 10:34:42] ...trap/engines/engine.py.321: eval: epoch 101 | batch 250/510 [I 2022-10-11 10:34:42] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:22 [I 2022-10-11 10:34:42] ...trap/engines/engine.py.326: process: 0.32362 | load: 0.00018 [I 2022-10-11 10:34:45] ...trap/engines/engine.py.321: eval: epoch 101 | batch 260/510 [I 2022-10-11 10:34:45] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:18 [I 2022-10-11 10:34:45] ...trap/engines/engine.py.326: process: 0.31302 | load: 0.00019 [I 2022-10-11 10:34:48] ...trap/engines/engine.py.321: eval: epoch 101 | batch 270/510 [I 2022-10-11 10:34:48] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:16 [I 2022-10-11 10:34:48] ...trap/engines/engine.py.326: process: 0.31893 | load: 0.00020 [I 2022-10-11 10:34:51] ...trap/engines/engine.py.321: eval: epoch 101 | batch 280/510 [I 2022-10-11 10:34:51] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:12 [I 2022-10-11 10:34:51] ...trap/engines/engine.py.326: process: 0.31060 | load: 0.00019 [I 2022-10-11 10:34:54] ...trap/engines/engine.py.321: eval: epoch 101 | batch 290/510 [I 2022-10-11 10:34:54] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:09 [I 2022-10-11 10:34:54] ...trap/engines/engine.py.326: process: 0.31688 | load: 0.00019 [I 2022-10-11 10:34:58] ...trap/engines/engine.py.321: eval: epoch 101 | batch 300/510 [I 2022-10-11 10:34:58] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:06 [I 2022-10-11 10:34:58] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00020 [I 2022-10-11 10:35:01] ...trap/engines/engine.py.321: eval: epoch 101 | batch 310/510 [I 2022-10-11 10:35:01] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:03 [I 2022-10-11 10:35:01] ...trap/engines/engine.py.326: process: 0.31187 | load: 0.00019 [I 2022-10-11 10:35:04] ...trap/engines/engine.py.321: eval: epoch 101 | batch 320/510 [I 2022-10-11 10:35:04] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:00 [I 2022-10-11 10:35:04] ...trap/engines/engine.py.326: process: 0.31822 | load: 0.00019 [I 2022-10-11 10:35:07] ...trap/engines/engine.py.321: eval: epoch 101 | batch 330/510 [I 2022-10-11 10:35:07] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-11 10:35:07] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00019 [I 2022-10-11 10:35:10] ...trap/engines/engine.py.321: eval: epoch 101 | batch 340/510 [I 2022-10-11 10:35:10] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:53 [I 2022-10-11 10:35:10] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00019 [I 2022-10-11 10:35:13] ...trap/engines/engine.py.321: eval: epoch 101 | batch 350/510 [I 2022-10-11 10:35:13] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:50 [I 2022-10-11 10:35:13] ...trap/engines/engine.py.326: process: 0.32552 | load: 0.00020 [I 2022-10-11 10:35:17] ...trap/engines/engine.py.321: eval: epoch 101 | batch 360/510 [I 2022-10-11 10:35:17] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:47 [I 2022-10-11 10:35:17] ...trap/engines/engine.py.326: process: 0.30888 | load: 0.00018 [I 2022-10-11 10:35:20] ...trap/engines/engine.py.321: eval: epoch 101 | batch 370/510 [I 2022-10-11 10:35:20] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:44 [I 2022-10-11 10:35:20] ...trap/engines/engine.py.326: process: 0.31479 | load: 0.00019 [I 2022-10-11 10:35:23] ...trap/engines/engine.py.321: eval: epoch 101 | batch 380/510 [I 2022-10-11 10:35:23] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-11 10:35:23] ...trap/engines/engine.py.326: process: 0.30926 | load: 0.00019 [I 2022-10-11 10:35:26] ...trap/engines/engine.py.321: eval: epoch 101 | batch 390/510 [I 2022-10-11 10:35:26] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-11 10:35:26] ...trap/engines/engine.py.326: process: 0.32526 | load: 0.00020 [I 2022-10-11 10:35:29] ...trap/engines/engine.py.321: eval: epoch 101 | batch 400/510 [I 2022-10-11 10:35:29] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:34 [I 2022-10-11 10:35:29] ...trap/engines/engine.py.326: process: 0.31751 | load: 0.00019 [I 2022-10-11 10:35:32] ...trap/engines/engine.py.321: eval: epoch 101 | batch 410/510 [I 2022-10-11 10:35:32] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:31 [I 2022-10-11 10:35:32] ...trap/engines/engine.py.326: process: 0.31928 | load: 0.00019 [I 2022-10-11 10:35:35] ...trap/engines/engine.py.321: eval: epoch 101 | batch 420/510 [I 2022-10-11 10:35:35] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:28 [I 2022-10-11 10:35:35] ...trap/engines/engine.py.326: process: 0.31487 | load: 0.00017 [I 2022-10-11 10:35:39] ...trap/engines/engine.py.321: eval: epoch 101 | batch 430/510 [I 2022-10-11 10:35:39] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:25 [I 2022-10-11 10:35:39] ...trap/engines/engine.py.326: process: 0.31952 | load: 0.00019 [I 2022-10-11 10:35:42] ...trap/engines/engine.py.321: eval: epoch 101 | batch 440/510 [I 2022-10-11 10:35:42] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-11 10:35:42] ...trap/engines/engine.py.326: process: 0.31396 | load: 0.00018 [I 2022-10-11 10:35:45] ...trap/engines/engine.py.321: eval: epoch 101 | batch 450/510 [I 2022-10-11 10:35:45] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-11 10:35:45] ...trap/engines/engine.py.326: process: 0.31351 | load: 0.00019 [I 2022-10-11 10:35:48] ...trap/engines/engine.py.321: eval: epoch 101 | batch 460/510 [I 2022-10-11 10:35:48] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:15 [I 2022-10-11 10:35:48] ...trap/engines/engine.py.326: process: 0.31657 | load: 0.00017 [I 2022-10-11 10:35:51] ...trap/engines/engine.py.321: eval: epoch 101 | batch 470/510 [I 2022-10-11 10:35:51] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:12 [I 2022-10-11 10:35:51] ...trap/engines/engine.py.326: process: 0.30744 | load: 0.00019 [I 2022-10-11 10:35:54] ...trap/engines/engine.py.321: eval: epoch 101 | batch 480/510 [I 2022-10-11 10:35:54] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:09 [I 2022-10-11 10:35:54] ...trap/engines/engine.py.326: process: 0.31809 | load: 0.00019 [I 2022-10-11 10:35:58] ...trap/engines/engine.py.321: eval: epoch 101 | batch 490/510 [I 2022-10-11 10:35:58] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-11 10:35:58] ...trap/engines/engine.py.326: process: 0.31354 | load: 0.00020 [I 2022-10-11 10:36:01] ...trap/engines/engine.py.321: eval: epoch 101 | batch 500/510 [I 2022-10-11 10:36:01] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-11 10:36:01] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00019 [I 2022-10-11 10:36:04] ...trap/engines/engine.py.321: eval: epoch 101 | batch 510/510 [I 2022-10-11 10:36:04] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-11 10:36:04] ...trap/engines/engine.py.326: process: 0.31243 | load: 0.00011 [S 2022-10-11 10:36:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 101 [I 2022-10-11 10:36:05] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740600 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908800 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933700 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.739500 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907400 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935600 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017585 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005896 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004941 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011218 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005851 [S 2022-10-11 10:36:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004695 [I 2022-10-11 10:36:45] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 10:36:45] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 10:36:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 10:36:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 10:36:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 10:36:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 10:36:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 10:36:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 10:36:57] ...trap/engines/engine.py.163: Training model on trainset for epoch 102 [I 2022-10-11 10:36:57] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 10:37:00] ...trap/engines/engine.py.224: train: epoch 102 | batch 0/2304 [I 2022-10-11 10:37:00] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:55:56 [I 2022-10-11 10:37:00] ...trap/engines/engine.py.229: process: 3.01952 | load: 2.10492 [I 2022-10-11 10:37:00] ...trap/engines/engine.py.230: loss: 0.08766 [I 2022-10-11 10:37:00] ...optimizers/trijoint.py.151: total_norm: 18.465876 [I 2022-10-11 10:37:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 10/2304 [I 2022-10-11 10:37:08] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:10 [I 2022-10-11 10:37:08] ...trap/engines/engine.py.229: process: 0.83645 | load: 0.00441 [I 2022-10-11 10:37:08] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-11 10:37:08] ...optimizers/trijoint.py.151: total_norm: 14.148288 [I 2022-10-11 10:37:17] ...trap/engines/engine.py.224: train: epoch 102 | batch 20/2304 [I 2022-10-11 10:37:17] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:48 [I 2022-10-11 10:37:17] ...trap/engines/engine.py.229: process: 0.83916 | load: 0.00443 [I 2022-10-11 10:37:17] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-11 10:37:17] ...optimizers/trijoint.py.151: total_norm: 32.213097 [I 2022-10-11 10:37:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 30/2304 [I 2022-10-11 10:37:25] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:50 [I 2022-10-11 10:37:25] ...trap/engines/engine.py.229: process: 0.82592 | load: 0.00374 [I 2022-10-11 10:37:25] ...trap/engines/engine.py.230: loss: 0.23065 [I 2022-10-11 10:37:25] ...optimizers/trijoint.py.151: total_norm: 36.300056 [I 2022-10-11 10:37:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 40/2304 [I 2022-10-11 10:37:33] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:36 [I 2022-10-11 10:37:33] ...trap/engines/engine.py.229: process: 0.82992 | load: 0.00481 [I 2022-10-11 10:37:33] ...trap/engines/engine.py.230: loss: 0.10079 [I 2022-10-11 10:37:33] ...optimizers/trijoint.py.151: total_norm: 13.726620 [I 2022-10-11 10:37:42] ...trap/engines/engine.py.224: train: epoch 102 | batch 50/2304 [I 2022-10-11 10:37:42] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:32 [I 2022-10-11 10:37:42] ...trap/engines/engine.py.229: process: 0.83393 | load: 0.00356 [I 2022-10-11 10:37:42] ...trap/engines/engine.py.230: loss: 0.07945 [I 2022-10-11 10:37:42] ...optimizers/trijoint.py.151: total_norm: 12.343707 [I 2022-10-11 10:37:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 60/2304 [I 2022-10-11 10:37:50] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:19 [I 2022-10-11 10:37:50] ...trap/engines/engine.py.229: process: 0.82807 | load: 0.00345 [I 2022-10-11 10:37:50] ...trap/engines/engine.py.230: loss: 0.11610 [I 2022-10-11 10:37:50] ...optimizers/trijoint.py.151: total_norm: 8.800457 [I 2022-10-11 10:37:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 70/2304 [I 2022-10-11 10:37:59] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:22 [I 2022-10-11 10:37:59] ...trap/engines/engine.py.229: process: 0.85867 | load: 0.00343 [I 2022-10-11 10:37:59] ...trap/engines/engine.py.230: loss: 0.15294 [I 2022-10-11 10:37:59] ...optimizers/trijoint.py.151: total_norm: 22.360352 [I 2022-10-11 10:38:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 80/2304 [I 2022-10-11 10:38:07] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:21 [I 2022-10-11 10:38:07] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00323 [I 2022-10-11 10:38:07] ...trap/engines/engine.py.230: loss: 0.10533 [I 2022-10-11 10:38:07] ...optimizers/trijoint.py.151: total_norm: 10.872457 [I 2022-10-11 10:38:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 90/2304 [I 2022-10-11 10:38:16] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:31:26 [I 2022-10-11 10:38:16] ...trap/engines/engine.py.229: process: 0.84930 | load: 0.00316 [I 2022-10-11 10:38:16] ...trap/engines/engine.py.230: loss: 0.15927 [I 2022-10-11 10:38:16] ...optimizers/trijoint.py.151: total_norm: 23.052317 [I 2022-10-11 10:38:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 100/2304 [I 2022-10-11 10:38:24] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:18 [I 2022-10-11 10:38:24] ...trap/engines/engine.py.229: process: 0.86761 | load: 0.00374 [I 2022-10-11 10:38:24] ...trap/engines/engine.py.230: loss: 0.17312 [I 2022-10-11 10:38:24] ...optimizers/trijoint.py.151: total_norm: 72.158104 [I 2022-10-11 10:38:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 110/2304 [I 2022-10-11 10:38:33] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:55 [I 2022-10-11 10:38:33] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00311 [I 2022-10-11 10:38:33] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-11 10:38:33] ...optimizers/trijoint.py.151: total_norm: 12.270161 [I 2022-10-11 10:38:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 120/2304 [I 2022-10-11 10:38:41] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:50 [I 2022-10-11 10:38:41] ...trap/engines/engine.py.229: process: 0.84612 | load: 0.00323 [I 2022-10-11 10:38:41] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-11 10:38:41] ...optimizers/trijoint.py.151: total_norm: 18.101686 [I 2022-10-11 10:38:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 130/2304 [I 2022-10-11 10:38:50] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:47 [I 2022-10-11 10:38:50] ...trap/engines/engine.py.229: process: 0.83132 | load: 0.00327 [I 2022-10-11 10:38:50] ...trap/engines/engine.py.230: loss: 0.12829 [I 2022-10-11 10:38:50] ...optimizers/trijoint.py.151: total_norm: 28.645823 [I 2022-10-11 10:38:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 140/2304 [I 2022-10-11 10:38:58] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:33 [I 2022-10-11 10:38:58] ...trap/engines/engine.py.229: process: 0.83899 | load: 0.00338 [I 2022-10-11 10:38:58] ...trap/engines/engine.py.230: loss: 0.07346 [I 2022-10-11 10:38:58] ...optimizers/trijoint.py.151: total_norm: 10.823369 [I 2022-10-11 10:39:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 150/2304 [I 2022-10-11 10:39:07] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:20 [I 2022-10-11 10:39:07] ...trap/engines/engine.py.229: process: 0.83629 | load: 0.00309 [I 2022-10-11 10:39:07] ...trap/engines/engine.py.230: loss: 0.10565 [I 2022-10-11 10:39:07] ...optimizers/trijoint.py.151: total_norm: 45.669907 [I 2022-10-11 10:39:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 160/2304 [I 2022-10-11 10:39:15] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:09 [I 2022-10-11 10:39:15] ...trap/engines/engine.py.229: process: 0.83507 | load: 0.00308 [I 2022-10-11 10:39:15] ...trap/engines/engine.py.230: loss: 0.15378 [I 2022-10-11 10:39:15] ...optimizers/trijoint.py.151: total_norm: 9.180518 [I 2022-10-11 10:39:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 170/2304 [I 2022-10-11 10:39:24] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:29:55 [I 2022-10-11 10:39:24] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00310 [I 2022-10-11 10:39:24] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-11 10:39:24] ...optimizers/trijoint.py.151: total_norm: 29.165836 [I 2022-10-11 10:39:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 180/2304 [I 2022-10-11 10:39:32] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:42 [I 2022-10-11 10:39:32] ...trap/engines/engine.py.229: process: 0.84023 | load: 0.00294 [I 2022-10-11 10:39:32] ...trap/engines/engine.py.230: loss: 0.08733 [I 2022-10-11 10:39:32] ...optimizers/trijoint.py.151: total_norm: 10.042551 [I 2022-10-11 10:39:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 190/2304 [I 2022-10-11 10:39:41] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:45 [I 2022-10-11 10:39:41] ...trap/engines/engine.py.229: process: 0.85523 | load: 0.00307 [I 2022-10-11 10:39:41] ...trap/engines/engine.py.230: loss: 0.09872 [I 2022-10-11 10:39:41] ...optimizers/trijoint.py.151: total_norm: 10.035410 [I 2022-10-11 10:39:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 200/2304 [I 2022-10-11 10:39:49] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:22 [I 2022-10-11 10:39:49] ...trap/engines/engine.py.229: process: 0.82829 | load: 0.00323 [I 2022-10-11 10:39:49] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-11 10:39:49] ...optimizers/trijoint.py.151: total_norm: 16.485947 [I 2022-10-11 10:39:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 210/2304 [I 2022-10-11 10:39:57] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:41 [I 2022-10-11 10:39:57] ...trap/engines/engine.py.229: process: 0.84693 | load: 0.00305 [I 2022-10-11 10:39:57] ...trap/engines/engine.py.230: loss: 0.15571 [I 2022-10-11 10:39:57] ...optimizers/trijoint.py.151: total_norm: 28.611889 [I 2022-10-11 10:40:06] ...trap/engines/engine.py.224: train: epoch 102 | batch 220/2304 [I 2022-10-11 10:40:06] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:12 [I 2022-10-11 10:40:06] ...trap/engines/engine.py.229: process: 0.82163 | load: 0.00294 [I 2022-10-11 10:40:06] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-11 10:40:06] ...optimizers/trijoint.py.151: total_norm: 13.436486 [I 2022-10-11 10:40:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 230/2304 [I 2022-10-11 10:40:14] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:30 [I 2022-10-11 10:40:14] ...trap/engines/engine.py.229: process: 0.85753 | load: 0.00318 [I 2022-10-11 10:40:14] ...trap/engines/engine.py.230: loss: 0.14855 [I 2022-10-11 10:40:14] ...optimizers/trijoint.py.151: total_norm: 14.247565 [I 2022-10-11 10:40:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 240/2304 [I 2022-10-11 10:40:23] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:13 [I 2022-10-11 10:40:23] ...trap/engines/engine.py.229: process: 0.83314 | load: 0.00308 [I 2022-10-11 10:40:23] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-11 10:40:23] ...optimizers/trijoint.py.151: total_norm: 29.737370 [I 2022-10-11 10:40:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 250/2304 [I 2022-10-11 10:40:31] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:29:02 [I 2022-10-11 10:40:31] ...trap/engines/engine.py.229: process: 0.86195 | load: 0.00292 [I 2022-10-11 10:40:31] ...trap/engines/engine.py.230: loss: 0.10280 [I 2022-10-11 10:40:31] ...optimizers/trijoint.py.151: total_norm: 18.637320 [I 2022-10-11 10:40:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 260/2304 [I 2022-10-11 10:40:40] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:54 [I 2022-10-11 10:40:40] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00319 [I 2022-10-11 10:40:40] ...trap/engines/engine.py.230: loss: 0.12397 [I 2022-10-11 10:40:40] ...optimizers/trijoint.py.151: total_norm: 17.177784 [I 2022-10-11 10:40:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 270/2304 [I 2022-10-11 10:40:48] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:35 [I 2022-10-11 10:40:48] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00293 [I 2022-10-11 10:40:48] ...trap/engines/engine.py.230: loss: 0.09173 [I 2022-10-11 10:40:48] ...optimizers/trijoint.py.151: total_norm: 18.629313 [I 2022-10-11 10:40:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 280/2304 [I 2022-10-11 10:40:57] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:23 [I 2022-10-11 10:40:57] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00292 [I 2022-10-11 10:40:57] ...trap/engines/engine.py.230: loss: 0.11361 [I 2022-10-11 10:40:57] ...optimizers/trijoint.py.151: total_norm: 29.662178 [I 2022-10-11 10:41:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 290/2304 [I 2022-10-11 10:41:05] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:17 [I 2022-10-11 10:41:05] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00323 [I 2022-10-11 10:41:05] ...trap/engines/engine.py.230: loss: 0.12569 [I 2022-10-11 10:41:05] ...optimizers/trijoint.py.151: total_norm: 38.156517 [I 2022-10-11 10:41:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 300/2304 [I 2022-10-11 10:41:14] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:27:59 [I 2022-10-11 10:41:14] ...trap/engines/engine.py.229: process: 0.83816 | load: 0.00298 [I 2022-10-11 10:41:14] ...trap/engines/engine.py.230: loss: 0.08577 [I 2022-10-11 10:41:14] ...optimizers/trijoint.py.151: total_norm: 12.243505 [I 2022-10-11 10:41:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 310/2304 [I 2022-10-11 10:41:22] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:27:58 [I 2022-10-11 10:41:22] ...trap/engines/engine.py.229: process: 0.85101 | load: 0.00292 [I 2022-10-11 10:41:22] ...trap/engines/engine.py.230: loss: 0.09075 [I 2022-10-11 10:41:22] ...optimizers/trijoint.py.151: total_norm: 15.749438 [I 2022-10-11 10:41:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 320/2304 [I 2022-10-11 10:41:31] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:47 [I 2022-10-11 10:41:31] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00300 [I 2022-10-11 10:41:31] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-11 10:41:31] ...optimizers/trijoint.py.151: total_norm: 20.623682 [I 2022-10-11 10:41:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 330/2304 [I 2022-10-11 10:41:39] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:27:47 [I 2022-10-11 10:41:39] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00293 [I 2022-10-11 10:41:39] ...trap/engines/engine.py.230: loss: 0.14095 [I 2022-10-11 10:41:39] ...optimizers/trijoint.py.151: total_norm: 22.226234 [I 2022-10-11 10:41:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 340/2304 [I 2022-10-11 10:41:48] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:28:23 [I 2022-10-11 10:41:48] ...trap/engines/engine.py.229: process: 0.86742 | load: 0.00395 [I 2022-10-11 10:41:48] ...trap/engines/engine.py.230: loss: 0.14204 [I 2022-10-11 10:41:48] ...optimizers/trijoint.py.151: total_norm: 12.033938 [I 2022-10-11 10:41:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 350/2304 [I 2022-10-11 10:41:56] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:28:19 [I 2022-10-11 10:41:56] ...trap/engines/engine.py.229: process: 0.85454 | load: 0.00371 [I 2022-10-11 10:41:56] ...trap/engines/engine.py.230: loss: 0.08872 [I 2022-10-11 10:41:56] ...optimizers/trijoint.py.151: total_norm: 29.275589 [I 2022-10-11 10:42:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 360/2304 [I 2022-10-11 10:42:05] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:28:10 [I 2022-10-11 10:42:05] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00372 [I 2022-10-11 10:42:05] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-11 10:42:05] ...optimizers/trijoint.py.151: total_norm: 20.256504 [I 2022-10-11 10:42:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 370/2304 [I 2022-10-11 10:42:14] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:28:02 [I 2022-10-11 10:42:14] ...trap/engines/engine.py.229: process: 0.87525 | load: 0.00374 [I 2022-10-11 10:42:14] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-11 10:42:14] ...optimizers/trijoint.py.151: total_norm: 24.813766 [I 2022-10-11 10:42:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 380/2304 [I 2022-10-11 10:42:23] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:27:51 [I 2022-10-11 10:42:23] ...trap/engines/engine.py.229: process: 0.86137 | load: 0.00371 [I 2022-10-11 10:42:23] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-11 10:42:23] ...optimizers/trijoint.py.151: total_norm: 18.423756 [I 2022-10-11 10:42:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 390/2304 [I 2022-10-11 10:42:31] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:27:03 [I 2022-10-11 10:42:31] ...trap/engines/engine.py.229: process: 0.83690 | load: 0.00292 [I 2022-10-11 10:42:31] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-11 10:42:31] ...optimizers/trijoint.py.151: total_norm: 30.732740 [I 2022-10-11 10:42:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 400/2304 [I 2022-10-11 10:42:40] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:26:51 [I 2022-10-11 10:42:40] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00303 [I 2022-10-11 10:42:40] ...trap/engines/engine.py.230: loss: 0.19391 [I 2022-10-11 10:42:40] ...optimizers/trijoint.py.151: total_norm: 20.167700 [I 2022-10-11 10:42:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 410/2304 [I 2022-10-11 10:42:48] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:21 [I 2022-10-11 10:42:48] ...trap/engines/engine.py.229: process: 0.82794 | load: 0.00294 [I 2022-10-11 10:42:48] ...trap/engines/engine.py.230: loss: 0.13598 [I 2022-10-11 10:42:48] ...optimizers/trijoint.py.151: total_norm: 18.119761 [I 2022-10-11 10:42:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 420/2304 [I 2022-10-11 10:42:56] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:26:22 [I 2022-10-11 10:42:56] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00291 [I 2022-10-11 10:42:56] ...trap/engines/engine.py.230: loss: 0.10684 [I 2022-10-11 10:42:56] ...optimizers/trijoint.py.151: total_norm: 12.015544 [I 2022-10-11 10:43:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 430/2304 [I 2022-10-11 10:43:05] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:26:23 [I 2022-10-11 10:43:05] ...trap/engines/engine.py.229: process: 0.84769 | load: 0.00291 [I 2022-10-11 10:43:05] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-11 10:43:05] ...optimizers/trijoint.py.151: total_norm: 25.860413 [I 2022-10-11 10:43:13] ...trap/engines/engine.py.224: train: epoch 102 | batch 440/2304 [I 2022-10-11 10:43:13] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:07 [I 2022-10-11 10:43:13] ...trap/engines/engine.py.229: process: 0.84706 | load: 0.00293 [I 2022-10-11 10:43:13] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-11 10:43:13] ...optimizers/trijoint.py.151: total_norm: 9.921329 [I 2022-10-11 10:43:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 450/2304 [I 2022-10-11 10:43:22] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:25:56 [I 2022-10-11 10:43:22] ...trap/engines/engine.py.229: process: 0.82734 | load: 0.00293 [I 2022-10-11 10:43:22] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-11 10:43:22] ...optimizers/trijoint.py.151: total_norm: 30.097794 [I 2022-10-11 10:43:30] ...trap/engines/engine.py.224: train: epoch 102 | batch 460/2304 [I 2022-10-11 10:43:30] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:25:53 [I 2022-10-11 10:43:30] ...trap/engines/engine.py.229: process: 0.84673 | load: 0.00293 [I 2022-10-11 10:43:30] ...trap/engines/engine.py.230: loss: 0.10797 [I 2022-10-11 10:43:30] ...optimizers/trijoint.py.151: total_norm: 20.342197 [I 2022-10-11 10:43:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 470/2304 [I 2022-10-11 10:43:38] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:25:27 [I 2022-10-11 10:43:38] ...trap/engines/engine.py.229: process: 0.82150 | load: 0.00293 [I 2022-10-11 10:43:38] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-11 10:43:38] ...optimizers/trijoint.py.151: total_norm: 11.844515 [I 2022-10-11 10:43:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 480/2304 [I 2022-10-11 10:43:47] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:30 [I 2022-10-11 10:43:47] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00294 [I 2022-10-11 10:43:47] ...trap/engines/engine.py.230: loss: 0.09795 [I 2022-10-11 10:43:47] ...optimizers/trijoint.py.151: total_norm: 14.030124 [I 2022-10-11 10:43:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 490/2304 [I 2022-10-11 10:43:55] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:25:27 [I 2022-10-11 10:43:55] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00292 [I 2022-10-11 10:43:55] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-11 10:43:55] ...optimizers/trijoint.py.151: total_norm: 10.265670 [I 2022-10-11 10:44:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 500/2304 [I 2022-10-11 10:44:04] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:25:08 [I 2022-10-11 10:44:04] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00296 [I 2022-10-11 10:44:04] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-11 10:44:04] ...optimizers/trijoint.py.151: total_norm: 13.676423 [I 2022-10-11 10:44:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 510/2304 [I 2022-10-11 10:44:12] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:09 [I 2022-10-11 10:44:12] ...trap/engines/engine.py.229: process: 0.84854 | load: 0.00292 [I 2022-10-11 10:44:12] ...trap/engines/engine.py.230: loss: 0.08514 [I 2022-10-11 10:44:12] ...optimizers/trijoint.py.151: total_norm: 8.124972 [I 2022-10-11 10:44:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 520/2304 [I 2022-10-11 10:44:20] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:24:58 [I 2022-10-11 10:44:20] ...trap/engines/engine.py.229: process: 0.84096 | load: 0.00292 [I 2022-10-11 10:44:20] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-11 10:44:20] ...optimizers/trijoint.py.151: total_norm: 20.347269 [I 2022-10-11 10:44:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 530/2304 [I 2022-10-11 10:44:29] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:24:44 [I 2022-10-11 10:44:29] ...trap/engines/engine.py.229: process: 0.83755 | load: 0.00294 [I 2022-10-11 10:44:29] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-11 10:44:29] ...optimizers/trijoint.py.151: total_norm: 35.483498 [I 2022-10-11 10:44:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 540/2304 [I 2022-10-11 10:44:37] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:24:36 [I 2022-10-11 10:44:37] ...trap/engines/engine.py.229: process: 0.83121 | load: 0.00295 [I 2022-10-11 10:44:37] ...trap/engines/engine.py.230: loss: 0.08859 [I 2022-10-11 10:44:37] ...optimizers/trijoint.py.151: total_norm: 21.018961 [I 2022-10-11 10:44:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 550/2304 [I 2022-10-11 10:44:46] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:24:29 [I 2022-10-11 10:44:46] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00292 [I 2022-10-11 10:44:46] ...trap/engines/engine.py.230: loss: 0.09730 [I 2022-10-11 10:44:46] ...optimizers/trijoint.py.151: total_norm: 11.917727 [I 2022-10-11 10:44:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 560/2304 [I 2022-10-11 10:44:54] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:24:29 [I 2022-10-11 10:44:54] ...trap/engines/engine.py.229: process: 0.84803 | load: 0.00296 [I 2022-10-11 10:44:54] ...trap/engines/engine.py.230: loss: 0.17666 [I 2022-10-11 10:44:54] ...optimizers/trijoint.py.151: total_norm: 33.175663 [I 2022-10-11 10:45:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 570/2304 [I 2022-10-11 10:45:02] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:24:18 [I 2022-10-11 10:45:02] ...trap/engines/engine.py.229: process: 0.85192 | load: 0.00294 [I 2022-10-11 10:45:02] ...trap/engines/engine.py.230: loss: 0.10657 [I 2022-10-11 10:45:02] ...optimizers/trijoint.py.151: total_norm: 11.554905 [I 2022-10-11 10:45:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 580/2304 [I 2022-10-11 10:45:11] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:21 [I 2022-10-11 10:45:11] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00294 [I 2022-10-11 10:45:11] ...trap/engines/engine.py.230: loss: 0.09519 [I 2022-10-11 10:45:11] ...optimizers/trijoint.py.151: total_norm: 12.150934 [I 2022-10-11 10:45:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 590/2304 [I 2022-10-11 10:45:19] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:24:17 [I 2022-10-11 10:45:19] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00295 [I 2022-10-11 10:45:19] ...trap/engines/engine.py.230: loss: 0.10498 [I 2022-10-11 10:45:19] ...optimizers/trijoint.py.151: total_norm: 11.591013 [I 2022-10-11 10:45:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 600/2304 [I 2022-10-11 10:45:28] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:24:05 [I 2022-10-11 10:45:28] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00296 [I 2022-10-11 10:45:28] ...trap/engines/engine.py.230: loss: 0.17454 [I 2022-10-11 10:45:28] ...optimizers/trijoint.py.151: total_norm: 21.980450 [I 2022-10-11 10:45:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 610/2304 [I 2022-10-11 10:45:36] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:23:52 [I 2022-10-11 10:45:36] ...trap/engines/engine.py.229: process: 0.85210 | load: 0.00294 [I 2022-10-11 10:45:36] ...trap/engines/engine.py.230: loss: 0.11991 [I 2022-10-11 10:45:36] ...optimizers/trijoint.py.151: total_norm: 30.047310 [I 2022-10-11 10:45:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 620/2304 [I 2022-10-11 10:45:45] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:23:45 [I 2022-10-11 10:45:45] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00294 [I 2022-10-11 10:45:45] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-11 10:45:45] ...optimizers/trijoint.py.151: total_norm: 17.593029 [I 2022-10-11 10:45:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 630/2304 [I 2022-10-11 10:45:53] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:23:35 [I 2022-10-11 10:45:53] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00295 [I 2022-10-11 10:45:53] ...trap/engines/engine.py.230: loss: 0.09010 [I 2022-10-11 10:45:53] ...optimizers/trijoint.py.151: total_norm: 8.693872 [I 2022-10-11 10:46:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 640/2304 [I 2022-10-11 10:46:02] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:44 [I 2022-10-11 10:46:02] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00373 [I 2022-10-11 10:46:02] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-11 10:46:02] ...optimizers/trijoint.py.151: total_norm: 27.807751 [I 2022-10-11 10:46:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 650/2304 [I 2022-10-11 10:46:11] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:48 [I 2022-10-11 10:46:11] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00373 [I 2022-10-11 10:46:11] ...trap/engines/engine.py.230: loss: 0.13563 [I 2022-10-11 10:46:11] ...optimizers/trijoint.py.151: total_norm: 27.898119 [I 2022-10-11 10:46:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 660/2304 [I 2022-10-11 10:46:19] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:48 [I 2022-10-11 10:46:19] ...trap/engines/engine.py.229: process: 0.86535 | load: 0.00371 [I 2022-10-11 10:46:19] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-11 10:46:19] ...optimizers/trijoint.py.151: total_norm: 21.666115 [I 2022-10-11 10:46:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 670/2304 [I 2022-10-11 10:46:28] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:47 [I 2022-10-11 10:46:28] ...trap/engines/engine.py.229: process: 0.87983 | load: 0.00372 [I 2022-10-11 10:46:28] ...trap/engines/engine.py.230: loss: 0.11399 [I 2022-10-11 10:46:28] ...optimizers/trijoint.py.151: total_norm: 16.034149 [I 2022-10-11 10:46:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 680/2304 [I 2022-10-11 10:46:37] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:23:40 [I 2022-10-11 10:46:37] ...trap/engines/engine.py.229: process: 0.88029 | load: 0.00373 [I 2022-10-11 10:46:37] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-11 10:46:37] ...optimizers/trijoint.py.151: total_norm: 28.372255 [I 2022-10-11 10:46:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 690/2304 [I 2022-10-11 10:46:45] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:23:22 [I 2022-10-11 10:46:45] ...trap/engines/engine.py.229: process: 0.87417 | load: 0.00375 [I 2022-10-11 10:46:45] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-11 10:46:45] ...optimizers/trijoint.py.151: total_norm: 18.924463 [I 2022-10-11 10:46:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 700/2304 [I 2022-10-11 10:46:54] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:23:25 [I 2022-10-11 10:46:54] ...trap/engines/engine.py.229: process: 0.88092 | load: 0.00371 [I 2022-10-11 10:46:54] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-11 10:46:54] ...optimizers/trijoint.py.151: total_norm: 48.688354 [I 2022-10-11 10:47:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 710/2304 [I 2022-10-11 10:47:03] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:23:15 [I 2022-10-11 10:47:03] ...trap/engines/engine.py.229: process: 0.88304 | load: 0.00371 [I 2022-10-11 10:47:03] ...trap/engines/engine.py.230: loss: 0.11171 [I 2022-10-11 10:47:03] ...optimizers/trijoint.py.151: total_norm: 12.975972 [I 2022-10-11 10:47:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 720/2304 [I 2022-10-11 10:47:12] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:23:02 [I 2022-10-11 10:47:12] ...trap/engines/engine.py.229: process: 0.87192 | load: 0.00371 [I 2022-10-11 10:47:12] ...trap/engines/engine.py.230: loss: 0.16297 [I 2022-10-11 10:47:12] ...optimizers/trijoint.py.151: total_norm: 16.679848 [I 2022-10-11 10:47:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 730/2304 [I 2022-10-11 10:47:20] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:15 [I 2022-10-11 10:47:20] ...trap/engines/engine.py.229: process: 0.82823 | load: 0.00294 [I 2022-10-11 10:47:20] ...trap/engines/engine.py.230: loss: 0.08216 [I 2022-10-11 10:47:20] ...optimizers/trijoint.py.151: total_norm: 12.805013 [I 2022-10-11 10:47:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 740/2304 [I 2022-10-11 10:47:29] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:21:53 [I 2022-10-11 10:47:29] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00293 [I 2022-10-11 10:47:29] ...trap/engines/engine.py.230: loss: 0.09956 [I 2022-10-11 10:47:29] ...optimizers/trijoint.py.151: total_norm: 8.766157 [I 2022-10-11 10:47:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 750/2304 [I 2022-10-11 10:47:37] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:21:42 [I 2022-10-11 10:47:37] ...trap/engines/engine.py.229: process: 0.83749 | load: 0.00293 [I 2022-10-11 10:47:37] ...trap/engines/engine.py.230: loss: 0.11027 [I 2022-10-11 10:47:37] ...optimizers/trijoint.py.151: total_norm: 14.343414 [I 2022-10-11 10:47:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 760/2304 [I 2022-10-11 10:47:46] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:41 [I 2022-10-11 10:47:46] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00295 [I 2022-10-11 10:47:46] ...trap/engines/engine.py.230: loss: 0.16858 [I 2022-10-11 10:47:46] ...optimizers/trijoint.py.151: total_norm: 11.339534 [I 2022-10-11 10:47:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 770/2304 [I 2022-10-11 10:47:54] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:21 [I 2022-10-11 10:47:54] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00294 [I 2022-10-11 10:47:54] ...trap/engines/engine.py.230: loss: 0.08143 [I 2022-10-11 10:47:54] ...optimizers/trijoint.py.151: total_norm: 11.076110 [I 2022-10-11 10:48:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 780/2304 [I 2022-10-11 10:48:02] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:21 [I 2022-10-11 10:48:02] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00292 [I 2022-10-11 10:48:02] ...trap/engines/engine.py.230: loss: 0.08646 [I 2022-10-11 10:48:02] ...optimizers/trijoint.py.151: total_norm: 9.685780 [I 2022-10-11 10:48:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 790/2304 [I 2022-10-11 10:48:11] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:09 [I 2022-10-11 10:48:11] ...trap/engines/engine.py.229: process: 0.83742 | load: 0.00293 [I 2022-10-11 10:48:11] ...trap/engines/engine.py.230: loss: 0.14556 [I 2022-10-11 10:48:11] ...optimizers/trijoint.py.151: total_norm: 21.561687 [I 2022-10-11 10:48:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 800/2304 [I 2022-10-11 10:48:19] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:05 [I 2022-10-11 10:48:19] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00292 [I 2022-10-11 10:48:19] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-11 10:48:19] ...optimizers/trijoint.py.151: total_norm: 12.369419 [I 2022-10-11 10:48:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 810/2304 [I 2022-10-11 10:48:28] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:00 [I 2022-10-11 10:48:28] ...trap/engines/engine.py.229: process: 0.83665 | load: 0.00293 [I 2022-10-11 10:48:28] ...trap/engines/engine.py.230: loss: 0.08818 [I 2022-10-11 10:48:28] ...optimizers/trijoint.py.151: total_norm: 11.316683 [I 2022-10-11 10:48:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 820/2304 [I 2022-10-11 10:48:36] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:20:42 [I 2022-10-11 10:48:36] ...trap/engines/engine.py.229: process: 0.81794 | load: 0.00293 [I 2022-10-11 10:48:36] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-11 10:48:36] ...optimizers/trijoint.py.151: total_norm: 20.068829 [I 2022-10-11 10:48:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 830/2304 [I 2022-10-11 10:48:44] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:37 [I 2022-10-11 10:48:44] ...trap/engines/engine.py.229: process: 0.84867 | load: 0.00296 [I 2022-10-11 10:48:44] ...trap/engines/engine.py.230: loss: 0.11168 [I 2022-10-11 10:48:44] ...optimizers/trijoint.py.151: total_norm: 19.048677 [I 2022-10-11 10:48:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 840/2304 [I 2022-10-11 10:48:53] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:20:21 [I 2022-10-11 10:48:53] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00299 [I 2022-10-11 10:48:53] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-11 10:48:53] ...optimizers/trijoint.py.151: total_norm: 13.524192 [I 2022-10-11 10:49:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 850/2304 [I 2022-10-11 10:49:01] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:17 [I 2022-10-11 10:49:01] ...trap/engines/engine.py.229: process: 0.83127 | load: 0.00296 [I 2022-10-11 10:49:01] ...trap/engines/engine.py.230: loss: 0.09373 [I 2022-10-11 10:49:01] ...optimizers/trijoint.py.151: total_norm: 12.012847 [I 2022-10-11 10:49:09] ...trap/engines/engine.py.224: train: epoch 102 | batch 860/2304 [I 2022-10-11 10:49:09] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:07 [I 2022-10-11 10:49:09] ...trap/engines/engine.py.229: process: 0.82470 | load: 0.00292 [I 2022-10-11 10:49:09] ...trap/engines/engine.py.230: loss: 0.11505 [I 2022-10-11 10:49:09] ...optimizers/trijoint.py.151: total_norm: 27.838755 [I 2022-10-11 10:49:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 870/2304 [I 2022-10-11 10:49:18] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:19:57 [I 2022-10-11 10:49:18] ...trap/engines/engine.py.229: process: 0.83435 | load: 0.00295 [I 2022-10-11 10:49:18] ...trap/engines/engine.py.230: loss: 0.09757 [I 2022-10-11 10:49:18] ...optimizers/trijoint.py.151: total_norm: 12.693822 [I 2022-10-11 10:49:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 880/2304 [I 2022-10-11 10:49:26] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:19:54 [I 2022-10-11 10:49:26] ...trap/engines/engine.py.229: process: 0.83533 | load: 0.00295 [I 2022-10-11 10:49:26] ...trap/engines/engine.py.230: loss: 0.13135 [I 2022-10-11 10:49:26] ...optimizers/trijoint.py.151: total_norm: 16.778141 [I 2022-10-11 10:49:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 890/2304 [I 2022-10-11 10:49:35] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:19:40 [I 2022-10-11 10:49:35] ...trap/engines/engine.py.229: process: 0.83493 | load: 0.00294 [I 2022-10-11 10:49:35] ...trap/engines/engine.py.230: loss: 0.07743 [I 2022-10-11 10:49:35] ...optimizers/trijoint.py.151: total_norm: 9.818637 [I 2022-10-11 10:49:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 900/2304 [I 2022-10-11 10:49:43] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:36 [I 2022-10-11 10:49:43] ...trap/engines/engine.py.229: process: 0.83891 | load: 0.00292 [I 2022-10-11 10:49:43] ...trap/engines/engine.py.230: loss: 0.15307 [I 2022-10-11 10:49:43] ...optimizers/trijoint.py.151: total_norm: 10.504883 [I 2022-10-11 10:49:51] ...trap/engines/engine.py.224: train: epoch 102 | batch 910/2304 [I 2022-10-11 10:49:51] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:19:29 [I 2022-10-11 10:49:51] ...trap/engines/engine.py.229: process: 0.82207 | load: 0.00294 [I 2022-10-11 10:49:51] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-11 10:49:51] ...optimizers/trijoint.py.151: total_norm: 12.304231 [I 2022-10-11 10:50:00] ...trap/engines/engine.py.224: train: epoch 102 | batch 920/2304 [I 2022-10-11 10:50:00] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:25 [I 2022-10-11 10:50:00] ...trap/engines/engine.py.229: process: 0.83250 | load: 0.00292 [I 2022-10-11 10:50:00] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-11 10:50:00] ...optimizers/trijoint.py.151: total_norm: 8.967094 [I 2022-10-11 10:50:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 930/2304 [I 2022-10-11 10:50:08] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:19:12 [I 2022-10-11 10:50:08] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00294 [I 2022-10-11 10:50:08] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-11 10:50:08] ...optimizers/trijoint.py.151: total_norm: 8.329204 [I 2022-10-11 10:50:17] ...trap/engines/engine.py.224: train: epoch 102 | batch 940/2304 [I 2022-10-11 10:50:17] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:19:15 [I 2022-10-11 10:50:17] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00292 [I 2022-10-11 10:50:17] ...trap/engines/engine.py.230: loss: 0.13896 [I 2022-10-11 10:50:17] ...optimizers/trijoint.py.151: total_norm: 11.388787 [I 2022-10-11 10:50:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 950/2304 [I 2022-10-11 10:50:25] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:02 [I 2022-10-11 10:50:25] ...trap/engines/engine.py.229: process: 0.83846 | load: 0.00294 [I 2022-10-11 10:50:25] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-11 10:50:25] ...optimizers/trijoint.py.151: total_norm: 15.700710 [I 2022-10-11 10:50:34] ...trap/engines/engine.py.224: train: epoch 102 | batch 960/2304 [I 2022-10-11 10:50:34] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:18:49 [I 2022-10-11 10:50:34] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00294 [I 2022-10-11 10:50:34] ...trap/engines/engine.py.230: loss: 0.08890 [I 2022-10-11 10:50:34] ...optimizers/trijoint.py.151: total_norm: 11.518982 [I 2022-10-11 10:50:42] ...trap/engines/engine.py.224: train: epoch 102 | batch 970/2304 [I 2022-10-11 10:50:42] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:18:41 [I 2022-10-11 10:50:42] ...trap/engines/engine.py.229: process: 0.83328 | load: 0.00294 [I 2022-10-11 10:50:42] ...trap/engines/engine.py.230: loss: 0.16156 [I 2022-10-11 10:50:42] ...optimizers/trijoint.py.151: total_norm: 13.882936 [I 2022-10-11 10:50:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 980/2304 [I 2022-10-11 10:50:50] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:18:33 [I 2022-10-11 10:50:50] ...trap/engines/engine.py.229: process: 0.84353 | load: 0.00295 [I 2022-10-11 10:50:50] ...trap/engines/engine.py.230: loss: 0.11687 [I 2022-10-11 10:50:50] ...optimizers/trijoint.py.151: total_norm: 11.718102 [I 2022-10-11 10:50:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 990/2304 [I 2022-10-11 10:50:59] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:18 [I 2022-10-11 10:50:59] ...trap/engines/engine.py.229: process: 0.82861 | load: 0.00297 [I 2022-10-11 10:50:59] ...trap/engines/engine.py.230: loss: 0.07134 [I 2022-10-11 10:50:59] ...optimizers/trijoint.py.151: total_norm: 10.009706 [I 2022-10-11 10:51:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 1000/2304 [I 2022-10-11 10:51:07] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:18:17 [I 2022-10-11 10:51:07] ...trap/engines/engine.py.229: process: 0.83612 | load: 0.00293 [I 2022-10-11 10:51:07] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-11 10:51:07] ...optimizers/trijoint.py.151: total_norm: 12.065708 [I 2022-10-11 10:51:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 1010/2304 [I 2022-10-11 10:51:16] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:11 [I 2022-10-11 10:51:16] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00295 [I 2022-10-11 10:51:16] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-11 10:51:16] ...optimizers/trijoint.py.151: total_norm: 16.239475 [I 2022-10-11 10:51:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 1020/2304 [I 2022-10-11 10:51:24] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:02 [I 2022-10-11 10:51:24] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00295 [I 2022-10-11 10:51:24] ...trap/engines/engine.py.230: loss: 0.16088 [I 2022-10-11 10:51:24] ...optimizers/trijoint.py.151: total_norm: 17.318504 [I 2022-10-11 10:51:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 1030/2304 [I 2022-10-11 10:51:33] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:17:54 [I 2022-10-11 10:51:33] ...trap/engines/engine.py.229: process: 0.83301 | load: 0.00294 [I 2022-10-11 10:51:33] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-11 10:51:33] ...optimizers/trijoint.py.151: total_norm: 10.768408 [I 2022-10-11 10:51:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 1040/2304 [I 2022-10-11 10:51:41] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:17:52 [I 2022-10-11 10:51:41] ...trap/engines/engine.py.229: process: 0.85918 | load: 0.00294 [I 2022-10-11 10:51:41] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-11 10:51:41] ...optimizers/trijoint.py.151: total_norm: 22.719564 [I 2022-10-11 10:51:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 1050/2304 [I 2022-10-11 10:51:50] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:37 [I 2022-10-11 10:51:50] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00293 [I 2022-10-11 10:51:50] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-11 10:51:50] ...optimizers/trijoint.py.151: total_norm: 18.808588 [I 2022-10-11 10:51:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 1060/2304 [I 2022-10-11 10:51:58] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:23 [I 2022-10-11 10:51:58] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00298 [I 2022-10-11 10:51:58] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-11 10:51:58] ...optimizers/trijoint.py.151: total_norm: 15.829301 [I 2022-10-11 10:52:06] ...trap/engines/engine.py.224: train: epoch 102 | batch 1070/2304 [I 2022-10-11 10:52:06] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:17 [I 2022-10-11 10:52:06] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00296 [I 2022-10-11 10:52:06] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-11 10:52:06] ...optimizers/trijoint.py.151: total_norm: 12.661501 [I 2022-10-11 10:52:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 1080/2304 [I 2022-10-11 10:52:15] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:10 [I 2022-10-11 10:52:15] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00295 [I 2022-10-11 10:52:15] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-11 10:52:15] ...optimizers/trijoint.py.151: total_norm: 8.315742 [I 2022-10-11 10:52:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 1090/2304 [I 2022-10-11 10:52:23] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:04 [I 2022-10-11 10:52:23] ...trap/engines/engine.py.229: process: 0.84290 | load: 0.00295 [I 2022-10-11 10:52:23] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-11 10:52:23] ...optimizers/trijoint.py.151: total_norm: 16.410999 [I 2022-10-11 10:52:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 1100/2304 [I 2022-10-11 10:52:32] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:16:55 [I 2022-10-11 10:52:32] ...trap/engines/engine.py.229: process: 0.83211 | load: 0.00296 [I 2022-10-11 10:52:32] ...trap/engines/engine.py.230: loss: 0.17613 [I 2022-10-11 10:52:32] ...optimizers/trijoint.py.151: total_norm: 13.857238 [I 2022-10-11 10:52:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 1110/2304 [I 2022-10-11 10:52:40] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:16:52 [I 2022-10-11 10:52:40] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00295 [I 2022-10-11 10:52:40] ...trap/engines/engine.py.230: loss: 0.12678 [I 2022-10-11 10:52:40] ...optimizers/trijoint.py.151: total_norm: 29.156612 [I 2022-10-11 10:52:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 1120/2304 [I 2022-10-11 10:52:49] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:36 [I 2022-10-11 10:52:49] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00294 [I 2022-10-11 10:52:49] ...trap/engines/engine.py.230: loss: 0.09170 [I 2022-10-11 10:52:49] ...optimizers/trijoint.py.151: total_norm: 9.974236 [I 2022-10-11 10:52:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 1130/2304 [I 2022-10-11 10:52:57] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:28 [I 2022-10-11 10:52:57] ...trap/engines/engine.py.229: process: 0.85057 | load: 0.00295 [I 2022-10-11 10:52:57] ...trap/engines/engine.py.230: loss: 0.14912 [I 2022-10-11 10:52:57] ...optimizers/trijoint.py.151: total_norm: 20.902697 [I 2022-10-11 10:53:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 1140/2304 [I 2022-10-11 10:53:05] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:16 [I 2022-10-11 10:53:05] ...trap/engines/engine.py.229: process: 0.83809 | load: 0.00296 [I 2022-10-11 10:53:05] ...trap/engines/engine.py.230: loss: 0.15216 [I 2022-10-11 10:53:05] ...optimizers/trijoint.py.151: total_norm: 14.518310 [I 2022-10-11 10:53:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 1150/2304 [I 2022-10-11 10:53:14] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:10 [I 2022-10-11 10:53:14] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00294 [I 2022-10-11 10:53:14] ...trap/engines/engine.py.230: loss: 0.08986 [I 2022-10-11 10:53:14] ...optimizers/trijoint.py.151: total_norm: 13.198130 [I 2022-10-11 10:53:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 1160/2304 [I 2022-10-11 10:53:22] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:03 [I 2022-10-11 10:53:22] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00294 [I 2022-10-11 10:53:22] ...trap/engines/engine.py.230: loss: 0.11677 [I 2022-10-11 10:53:22] ...optimizers/trijoint.py.151: total_norm: 22.540592 [I 2022-10-11 10:53:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 1170/2304 [I 2022-10-11 10:53:31] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:15:52 [I 2022-10-11 10:53:31] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00294 [I 2022-10-11 10:53:31] ...trap/engines/engine.py.230: loss: 0.09520 [I 2022-10-11 10:53:31] ...optimizers/trijoint.py.151: total_norm: 13.597268 [I 2022-10-11 10:53:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 1180/2304 [I 2022-10-11 10:53:39] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:15:47 [I 2022-10-11 10:53:39] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00294 [I 2022-10-11 10:53:39] ...trap/engines/engine.py.230: loss: 0.14725 [I 2022-10-11 10:53:39] ...optimizers/trijoint.py.151: total_norm: 14.101909 [I 2022-10-11 10:53:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 1190/2304 [I 2022-10-11 10:53:48] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:15:37 [I 2022-10-11 10:53:48] ...trap/engines/engine.py.229: process: 0.84825 | load: 0.00294 [I 2022-10-11 10:53:48] ...trap/engines/engine.py.230: loss: 0.09157 [I 2022-10-11 10:53:48] ...optimizers/trijoint.py.151: total_norm: 17.821606 [I 2022-10-11 10:53:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 1200/2304 [I 2022-10-11 10:53:56] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:29 [I 2022-10-11 10:53:56] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00293 [I 2022-10-11 10:53:56] ...trap/engines/engine.py.230: loss: 0.11092 [I 2022-10-11 10:53:56] ...optimizers/trijoint.py.151: total_norm: 19.990501 [I 2022-10-11 10:54:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 1210/2304 [I 2022-10-11 10:54:04] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:14 [I 2022-10-11 10:54:04] ...trap/engines/engine.py.229: process: 0.82861 | load: 0.00294 [I 2022-10-11 10:54:04] ...trap/engines/engine.py.230: loss: 0.14907 [I 2022-10-11 10:54:04] ...optimizers/trijoint.py.151: total_norm: 58.152702 [I 2022-10-11 10:54:13] ...trap/engines/engine.py.224: train: epoch 102 | batch 1220/2304 [I 2022-10-11 10:54:13] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:17 [I 2022-10-11 10:54:13] ...trap/engines/engine.py.229: process: 0.84490 | load: 0.00295 [I 2022-10-11 10:54:13] ...trap/engines/engine.py.230: loss: 0.11942 [I 2022-10-11 10:54:13] ...optimizers/trijoint.py.151: total_norm: 11.589287 [I 2022-10-11 10:54:21] ...trap/engines/engine.py.224: train: epoch 102 | batch 1230/2304 [I 2022-10-11 10:54:21] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:15:02 [I 2022-10-11 10:54:21] ...trap/engines/engine.py.229: process: 0.85019 | load: 0.00294 [I 2022-10-11 10:54:21] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-11 10:54:21] ...optimizers/trijoint.py.151: total_norm: 11.804509 [I 2022-10-11 10:54:30] ...trap/engines/engine.py.224: train: epoch 102 | batch 1240/2304 [I 2022-10-11 10:54:30] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:14:57 [I 2022-10-11 10:54:30] ...trap/engines/engine.py.229: process: 0.83702 | load: 0.00293 [I 2022-10-11 10:54:30] ...trap/engines/engine.py.230: loss: 0.21933 [I 2022-10-11 10:54:30] ...optimizers/trijoint.py.151: total_norm: 23.749380 [I 2022-10-11 10:54:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 1250/2304 [I 2022-10-11 10:54:38] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:14:47 [I 2022-10-11 10:54:38] ...trap/engines/engine.py.229: process: 0.86051 | load: 0.00295 [I 2022-10-11 10:54:38] ...trap/engines/engine.py.230: loss: 0.12718 [I 2022-10-11 10:54:38] ...optimizers/trijoint.py.151: total_norm: 13.934569 [I 2022-10-11 10:54:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 1260/2304 [I 2022-10-11 10:54:46] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:14:38 [I 2022-10-11 10:54:46] ...trap/engines/engine.py.229: process: 0.84923 | load: 0.00290 [I 2022-10-11 10:54:46] ...trap/engines/engine.py.230: loss: 0.15156 [I 2022-10-11 10:54:46] ...optimizers/trijoint.py.151: total_norm: 13.650299 [I 2022-10-11 10:54:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 1270/2304 [I 2022-10-11 10:54:55] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:14:29 [I 2022-10-11 10:54:55] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00298 [I 2022-10-11 10:54:55] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-11 10:54:55] ...optimizers/trijoint.py.151: total_norm: 12.322373 [I 2022-10-11 10:55:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 1280/2304 [I 2022-10-11 10:55:03] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:24 [I 2022-10-11 10:55:03] ...trap/engines/engine.py.229: process: 0.83796 | load: 0.00295 [I 2022-10-11 10:55:03] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-11 10:55:03] ...optimizers/trijoint.py.151: total_norm: 12.242727 [I 2022-10-11 10:55:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 1290/2304 [I 2022-10-11 10:55:12] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:14:19 [I 2022-10-11 10:55:12] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00295 [I 2022-10-11 10:55:12] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-11 10:55:12] ...optimizers/trijoint.py.151: total_norm: 16.787882 [I 2022-10-11 10:55:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 1300/2304 [I 2022-10-11 10:55:20] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:12 [I 2022-10-11 10:55:20] ...trap/engines/engine.py.229: process: 0.84696 | load: 0.00295 [I 2022-10-11 10:55:20] ...trap/engines/engine.py.230: loss: 0.13041 [I 2022-10-11 10:55:20] ...optimizers/trijoint.py.151: total_norm: 10.848508 [I 2022-10-11 10:55:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 1310/2304 [I 2022-10-11 10:55:29] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:03 [I 2022-10-11 10:55:29] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00295 [I 2022-10-11 10:55:29] ...trap/engines/engine.py.230: loss: 0.10838 [I 2022-10-11 10:55:29] ...optimizers/trijoint.py.151: total_norm: 11.425853 [I 2022-10-11 10:55:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 1320/2304 [I 2022-10-11 10:55:37] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:13:54 [I 2022-10-11 10:55:37] ...trap/engines/engine.py.229: process: 0.85641 | load: 0.00297 [I 2022-10-11 10:55:37] ...trap/engines/engine.py.230: loss: 0.09277 [I 2022-10-11 10:55:37] ...optimizers/trijoint.py.151: total_norm: 15.031892 [I 2022-10-11 10:55:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 1330/2304 [I 2022-10-11 10:55:46] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:13:44 [I 2022-10-11 10:55:46] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00295 [I 2022-10-11 10:55:46] ...trap/engines/engine.py.230: loss: 0.10384 [I 2022-10-11 10:55:46] ...optimizers/trijoint.py.151: total_norm: 9.722460 [I 2022-10-11 10:55:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 1340/2304 [I 2022-10-11 10:55:54] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:39 [I 2022-10-11 10:55:54] ...trap/engines/engine.py.229: process: 0.85684 | load: 0.00294 [I 2022-10-11 10:55:54] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-11 10:55:54] ...optimizers/trijoint.py.151: total_norm: 12.919686 [I 2022-10-11 10:56:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 1350/2304 [I 2022-10-11 10:56:03] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:28 [I 2022-10-11 10:56:03] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00292 [I 2022-10-11 10:56:03] ...trap/engines/engine.py.230: loss: 0.10207 [I 2022-10-11 10:56:03] ...optimizers/trijoint.py.151: total_norm: 8.514971 [I 2022-10-11 10:56:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 1360/2304 [I 2022-10-11 10:56:11] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:13:12 [I 2022-10-11 10:56:11] ...trap/engines/engine.py.229: process: 0.83434 | load: 0.00294 [I 2022-10-11 10:56:11] ...trap/engines/engine.py.230: loss: 0.15022 [I 2022-10-11 10:56:11] ...optimizers/trijoint.py.151: total_norm: 20.210213 [I 2022-10-11 10:56:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 1370/2304 [I 2022-10-11 10:56:20] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:04 [I 2022-10-11 10:56:20] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00292 [I 2022-10-11 10:56:20] ...trap/engines/engine.py.230: loss: 0.08829 [I 2022-10-11 10:56:20] ...optimizers/trijoint.py.151: total_norm: 13.806593 [I 2022-10-11 10:56:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 1380/2304 [I 2022-10-11 10:56:28] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:12:51 [I 2022-10-11 10:56:28] ...trap/engines/engine.py.229: process: 0.82770 | load: 0.00292 [I 2022-10-11 10:56:28] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-11 10:56:28] ...optimizers/trijoint.py.151: total_norm: 7.919742 [I 2022-10-11 10:56:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 1390/2304 [I 2022-10-11 10:56:36] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:12:43 [I 2022-10-11 10:56:36] ...trap/engines/engine.py.229: process: 0.83673 | load: 0.00296 [I 2022-10-11 10:56:36] ...trap/engines/engine.py.230: loss: 0.17968 [I 2022-10-11 10:56:36] ...optimizers/trijoint.py.151: total_norm: 15.153809 [I 2022-10-11 10:56:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 1400/2304 [I 2022-10-11 10:56:45] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:40 [I 2022-10-11 10:56:45] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00292 [I 2022-10-11 10:56:45] ...trap/engines/engine.py.230: loss: 0.10003 [I 2022-10-11 10:56:45] ...optimizers/trijoint.py.151: total_norm: 12.388789 [I 2022-10-11 10:56:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 1410/2304 [I 2022-10-11 10:56:53] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:29 [I 2022-10-11 10:56:53] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00295 [I 2022-10-11 10:56:53] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-11 10:56:53] ...optimizers/trijoint.py.151: total_norm: 11.949890 [I 2022-10-11 10:57:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 1420/2304 [I 2022-10-11 10:57:02] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:26 [I 2022-10-11 10:57:02] ...trap/engines/engine.py.229: process: 0.84873 | load: 0.00295 [I 2022-10-11 10:57:02] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-11 10:57:02] ...optimizers/trijoint.py.151: total_norm: 24.114012 [I 2022-10-11 10:57:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 1430/2304 [I 2022-10-11 10:57:10] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:15 [I 2022-10-11 10:57:10] ...trap/engines/engine.py.229: process: 0.84243 | load: 0.00293 [I 2022-10-11 10:57:10] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-11 10:57:10] ...optimizers/trijoint.py.151: total_norm: 10.469403 [I 2022-10-11 10:57:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 1440/2304 [I 2022-10-11 10:57:18] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:05 [I 2022-10-11 10:57:18] ...trap/engines/engine.py.229: process: 0.83857 | load: 0.00294 [I 2022-10-11 10:57:18] ...trap/engines/engine.py.230: loss: 0.08328 [I 2022-10-11 10:57:18] ...optimizers/trijoint.py.151: total_norm: 12.248384 [I 2022-10-11 10:57:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 1450/2304 [I 2022-10-11 10:57:27] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:11:54 [I 2022-10-11 10:57:27] ...trap/engines/engine.py.229: process: 0.82125 | load: 0.00293 [I 2022-10-11 10:57:27] ...trap/engines/engine.py.230: loss: 0.09947 [I 2022-10-11 10:57:27] ...optimizers/trijoint.py.151: total_norm: 30.476011 [I 2022-10-11 10:57:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 1460/2304 [I 2022-10-11 10:57:35] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:11:47 [I 2022-10-11 10:57:35] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00292 [I 2022-10-11 10:57:35] ...trap/engines/engine.py.230: loss: 0.15671 [I 2022-10-11 10:57:35] ...optimizers/trijoint.py.151: total_norm: 14.093869 [I 2022-10-11 10:57:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 1470/2304 [I 2022-10-11 10:57:44] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:11:55 [I 2022-10-11 10:57:44] ...trap/engines/engine.py.229: process: 0.88967 | load: 0.00373 [I 2022-10-11 10:57:44] ...trap/engines/engine.py.230: loss: 0.09252 [I 2022-10-11 10:57:44] ...optimizers/trijoint.py.151: total_norm: 9.217936 [I 2022-10-11 10:57:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 1480/2304 [I 2022-10-11 10:57:52] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:55 [I 2022-10-11 10:57:52] ...trap/engines/engine.py.229: process: 0.86277 | load: 0.00375 [I 2022-10-11 10:57:52] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 10:57:52] ...optimizers/trijoint.py.151: total_norm: 16.685778 [I 2022-10-11 10:58:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 1490/2304 [I 2022-10-11 10:58:01] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:49 [I 2022-10-11 10:58:01] ...trap/engines/engine.py.229: process: 0.87027 | load: 0.00376 [I 2022-10-11 10:58:01] ...trap/engines/engine.py.230: loss: 0.10962 [I 2022-10-11 10:58:01] ...optimizers/trijoint.py.151: total_norm: 9.716207 [I 2022-10-11 10:58:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 1500/2304 [I 2022-10-11 10:58:10] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:40 [I 2022-10-11 10:58:10] ...trap/engines/engine.py.229: process: 0.86758 | load: 0.00373 [I 2022-10-11 10:58:10] ...trap/engines/engine.py.230: loss: 0.14388 [I 2022-10-11 10:58:10] ...optimizers/trijoint.py.151: total_norm: 29.842888 [I 2022-10-11 10:58:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 1510/2304 [I 2022-10-11 10:58:19] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:30 [I 2022-10-11 10:58:19] ...trap/engines/engine.py.229: process: 0.87475 | load: 0.00372 [I 2022-10-11 10:58:19] ...trap/engines/engine.py.230: loss: 0.09840 [I 2022-10-11 10:58:19] ...optimizers/trijoint.py.151: total_norm: 12.451214 [I 2022-10-11 10:58:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 1520/2304 [I 2022-10-11 10:58:27] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:21 [I 2022-10-11 10:58:27] ...trap/engines/engine.py.229: process: 0.87079 | load: 0.00373 [I 2022-10-11 10:58:27] ...trap/engines/engine.py.230: loss: 0.12460 [I 2022-10-11 10:58:27] ...optimizers/trijoint.py.151: total_norm: 78.632965 [I 2022-10-11 10:58:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 1530/2304 [I 2022-10-11 10:58:36] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:14 [I 2022-10-11 10:58:36] ...trap/engines/engine.py.229: process: 0.87136 | load: 0.00371 [I 2022-10-11 10:58:36] ...trap/engines/engine.py.230: loss: 0.10381 [I 2022-10-11 10:58:36] ...optimizers/trijoint.py.151: total_norm: 12.674853 [I 2022-10-11 10:58:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 1540/2304 [I 2022-10-11 10:58:45] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:01 [I 2022-10-11 10:58:45] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00372 [I 2022-10-11 10:58:45] ...trap/engines/engine.py.230: loss: 0.12557 [I 2022-10-11 10:58:45] ...optimizers/trijoint.py.151: total_norm: 25.707661 [I 2022-10-11 10:58:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 1550/2304 [I 2022-10-11 10:58:53] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:55 [I 2022-10-11 10:58:53] ...trap/engines/engine.py.229: process: 0.87198 | load: 0.00373 [I 2022-10-11 10:58:53] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-11 10:58:53] ...optimizers/trijoint.py.151: total_norm: 20.880068 [I 2022-10-11 10:59:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 1560/2304 [I 2022-10-11 10:59:02] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:45 [I 2022-10-11 10:59:02] ...trap/engines/engine.py.229: process: 0.86752 | load: 0.00375 [I 2022-10-11 10:59:02] ...trap/engines/engine.py.230: loss: 0.10592 [I 2022-10-11 10:59:02] ...optimizers/trijoint.py.151: total_norm: 14.336880 [I 2022-10-11 10:59:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 1570/2304 [I 2022-10-11 10:59:11] ...trap/engines/engine.py.228: elapsed: 0:22:14 | left: 0:10:36 [I 2022-10-11 10:59:11] ...trap/engines/engine.py.229: process: 0.86721 | load: 0.00371 [I 2022-10-11 10:59:11] ...trap/engines/engine.py.230: loss: 0.18203 [I 2022-10-11 10:59:11] ...optimizers/trijoint.py.151: total_norm: 18.592827 [I 2022-10-11 10:59:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 1580/2304 [I 2022-10-11 10:59:19] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:27 [I 2022-10-11 10:59:19] ...trap/engines/engine.py.229: process: 0.87622 | load: 0.00376 [I 2022-10-11 10:59:20] ...trap/engines/engine.py.230: loss: 0.13700 [I 2022-10-11 10:59:20] ...optimizers/trijoint.py.151: total_norm: 12.305344 [I 2022-10-11 10:59:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 1590/2304 [I 2022-10-11 10:59:28] ...trap/engines/engine.py.228: elapsed: 0:22:31 | left: 0:10:21 [I 2022-10-11 10:59:28] ...trap/engines/engine.py.229: process: 0.87830 | load: 0.00373 [I 2022-10-11 10:59:28] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-11 10:59:28] ...optimizers/trijoint.py.151: total_norm: 11.831130 [I 2022-10-11 10:59:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 1600/2304 [I 2022-10-11 10:59:37] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:10 [I 2022-10-11 10:59:37] ...trap/engines/engine.py.229: process: 0.86285 | load: 0.00372 [I 2022-10-11 10:59:37] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-11 10:59:37] ...optimizers/trijoint.py.151: total_norm: 8.046041 [I 2022-10-11 10:59:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 1610/2304 [I 2022-10-11 10:59:46] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:00 [I 2022-10-11 10:59:46] ...trap/engines/engine.py.229: process: 0.86188 | load: 0.00374 [I 2022-10-11 10:59:46] ...trap/engines/engine.py.230: loss: 0.25810 [I 2022-10-11 10:59:46] ...optimizers/trijoint.py.151: total_norm: 35.585609 [I 2022-10-11 10:59:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 1620/2304 [I 2022-10-11 10:59:54] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:09:55 [I 2022-10-11 10:59:54] ...trap/engines/engine.py.229: process: 0.86563 | load: 0.00371 [I 2022-10-11 10:59:54] ...trap/engines/engine.py.230: loss: 0.12514 [I 2022-10-11 10:59:54] ...optimizers/trijoint.py.151: total_norm: 19.147614 [I 2022-10-11 11:00:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 1630/2304 [I 2022-10-11 11:00:03] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:46 [I 2022-10-11 11:00:03] ...trap/engines/engine.py.229: process: 0.87228 | load: 0.00373 [I 2022-10-11 11:00:03] ...trap/engines/engine.py.230: loss: 0.11828 [I 2022-10-11 11:00:03] ...optimizers/trijoint.py.151: total_norm: 13.830823 [I 2022-10-11 11:00:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 1640/2304 [I 2022-10-11 11:00:12] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:27 [I 2022-10-11 11:00:12] ...trap/engines/engine.py.229: process: 0.82976 | load: 0.00294 [I 2022-10-11 11:00:12] ...trap/engines/engine.py.230: loss: 0.13100 [I 2022-10-11 11:00:12] ...optimizers/trijoint.py.151: total_norm: 15.104451 [I 2022-10-11 11:00:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 1650/2304 [I 2022-10-11 11:00:20] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:04 [I 2022-10-11 11:00:20] ...trap/engines/engine.py.229: process: 0.83317 | load: 0.00294 [I 2022-10-11 11:00:20] ...trap/engines/engine.py.230: loss: 0.12601 [I 2022-10-11 11:00:20] ...optimizers/trijoint.py.151: total_norm: 8.238777 [I 2022-10-11 11:00:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 1660/2304 [I 2022-10-11 11:00:28] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:08:57 [I 2022-10-11 11:00:28] ...trap/engines/engine.py.229: process: 0.83785 | load: 0.00294 [I 2022-10-11 11:00:28] ...trap/engines/engine.py.230: loss: 0.08407 [I 2022-10-11 11:00:28] ...optimizers/trijoint.py.151: total_norm: 10.187303 [I 2022-10-11 11:00:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 1670/2304 [I 2022-10-11 11:00:37] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:08:49 [I 2022-10-11 11:00:37] ...trap/engines/engine.py.229: process: 0.83827 | load: 0.00292 [I 2022-10-11 11:00:37] ...trap/engines/engine.py.230: loss: 0.09032 [I 2022-10-11 11:00:37] ...optimizers/trijoint.py.151: total_norm: 19.740923 [I 2022-10-11 11:00:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 1680/2304 [I 2022-10-11 11:00:45] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:08:46 [I 2022-10-11 11:00:45] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00295 [I 2022-10-11 11:00:45] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-11 11:00:45] ...optimizers/trijoint.py.151: total_norm: 11.898751 [I 2022-10-11 11:00:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 1690/2304 [I 2022-10-11 11:00:54] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:08:34 [I 2022-10-11 11:00:54] ...trap/engines/engine.py.229: process: 0.83775 | load: 0.00295 [I 2022-10-11 11:00:54] ...trap/engines/engine.py.230: loss: 0.10366 [I 2022-10-11 11:00:54] ...optimizers/trijoint.py.151: total_norm: 12.330970 [I 2022-10-11 11:01:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 1700/2304 [I 2022-10-11 11:01:02] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:22 [I 2022-10-11 11:01:02] ...trap/engines/engine.py.229: process: 0.81593 | load: 0.00293 [I 2022-10-11 11:01:02] ...trap/engines/engine.py.230: loss: 0.10913 [I 2022-10-11 11:01:02] ...optimizers/trijoint.py.151: total_norm: 6.856036 [I 2022-10-11 11:01:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 1710/2304 [I 2022-10-11 11:01:10] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:13 [I 2022-10-11 11:01:10] ...trap/engines/engine.py.229: process: 0.82289 | load: 0.00292 [I 2022-10-11 11:01:10] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-11 11:01:10] ...optimizers/trijoint.py.151: total_norm: 13.936035 [I 2022-10-11 11:01:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 1720/2304 [I 2022-10-11 11:01:19] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:08 [I 2022-10-11 11:01:19] ...trap/engines/engine.py.229: process: 0.83938 | load: 0.00293 [I 2022-10-11 11:01:19] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-11 11:01:19] ...optimizers/trijoint.py.151: total_norm: 14.225842 [I 2022-10-11 11:01:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 1730/2304 [I 2022-10-11 11:01:27] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:07:59 [I 2022-10-11 11:01:27] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00292 [I 2022-10-11 11:01:27] ...trap/engines/engine.py.230: loss: 0.16905 [I 2022-10-11 11:01:27] ...optimizers/trijoint.py.151: total_norm: 21.026104 [I 2022-10-11 11:01:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 1740/2304 [I 2022-10-11 11:01:35] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:07:51 [I 2022-10-11 11:01:35] ...trap/engines/engine.py.229: process: 0.83354 | load: 0.00290 [I 2022-10-11 11:01:35] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-11 11:01:35] ...optimizers/trijoint.py.151: total_norm: 13.180440 [I 2022-10-11 11:01:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 1750/2304 [I 2022-10-11 11:01:44] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:07:44 [I 2022-10-11 11:01:44] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00293 [I 2022-10-11 11:01:44] ...trap/engines/engine.py.230: loss: 0.14414 [I 2022-10-11 11:01:44] ...optimizers/trijoint.py.151: total_norm: 21.222609 [I 2022-10-11 11:01:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 1760/2304 [I 2022-10-11 11:01:52] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:07:35 [I 2022-10-11 11:01:52] ...trap/engines/engine.py.229: process: 0.84553 | load: 0.00295 [I 2022-10-11 11:01:52] ...trap/engines/engine.py.230: loss: 0.11025 [I 2022-10-11 11:01:52] ...optimizers/trijoint.py.151: total_norm: 9.851609 [I 2022-10-11 11:02:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 1770/2304 [I 2022-10-11 11:02:01] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:28 [I 2022-10-11 11:02:01] ...trap/engines/engine.py.229: process: 0.84089 | load: 0.00293 [I 2022-10-11 11:02:01] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-11 11:02:01] ...optimizers/trijoint.py.151: total_norm: 18.869743 [I 2022-10-11 11:02:09] ...trap/engines/engine.py.224: train: epoch 102 | batch 1780/2304 [I 2022-10-11 11:02:09] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:20 [I 2022-10-11 11:02:09] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00296 [I 2022-10-11 11:02:09] ...trap/engines/engine.py.230: loss: 0.15628 [I 2022-10-11 11:02:09] ...optimizers/trijoint.py.151: total_norm: 25.740866 [I 2022-10-11 11:02:17] ...trap/engines/engine.py.224: train: epoch 102 | batch 1790/2304 [I 2022-10-11 11:02:17] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:11 [I 2022-10-11 11:02:17] ...trap/engines/engine.py.229: process: 0.84726 | load: 0.00294 [I 2022-10-11 11:02:17] ...trap/engines/engine.py.230: loss: 0.08881 [I 2022-10-11 11:02:17] ...optimizers/trijoint.py.151: total_norm: 12.563924 [I 2022-10-11 11:02:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 1800/2304 [I 2022-10-11 11:02:26] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:01 [I 2022-10-11 11:02:26] ...trap/engines/engine.py.229: process: 0.84289 | load: 0.00294 [I 2022-10-11 11:02:26] ...trap/engines/engine.py.230: loss: 0.12322 [I 2022-10-11 11:02:26] ...optimizers/trijoint.py.151: total_norm: 13.919144 [I 2022-10-11 11:02:34] ...trap/engines/engine.py.224: train: epoch 102 | batch 1810/2304 [I 2022-10-11 11:02:34] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:06:56 [I 2022-10-11 11:02:34] ...trap/engines/engine.py.229: process: 0.84434 | load: 0.00293 [I 2022-10-11 11:02:34] ...trap/engines/engine.py.230: loss: 0.10004 [I 2022-10-11 11:02:34] ...optimizers/trijoint.py.151: total_norm: 19.194740 [I 2022-10-11 11:02:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 1820/2304 [I 2022-10-11 11:02:43] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:06:47 [I 2022-10-11 11:02:43] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00294 [I 2022-10-11 11:02:43] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-11 11:02:43] ...optimizers/trijoint.py.151: total_norm: 36.641258 [I 2022-10-11 11:02:51] ...trap/engines/engine.py.224: train: epoch 102 | batch 1830/2304 [I 2022-10-11 11:02:51] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:06:38 [I 2022-10-11 11:02:51] ...trap/engines/engine.py.229: process: 0.83375 | load: 0.00294 [I 2022-10-11 11:02:51] ...trap/engines/engine.py.230: loss: 0.10637 [I 2022-10-11 11:02:51] ...optimizers/trijoint.py.151: total_norm: 9.880189 [I 2022-10-11 11:02:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 1840/2304 [I 2022-10-11 11:02:59] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:06:29 [I 2022-10-11 11:02:59] ...trap/engines/engine.py.229: process: 0.82967 | load: 0.00293 [I 2022-10-11 11:02:59] ...trap/engines/engine.py.230: loss: 0.15522 [I 2022-10-11 11:02:59] ...optimizers/trijoint.py.151: total_norm: 11.404591 [I 2022-10-11 11:03:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 1850/2304 [I 2022-10-11 11:03:08] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:06:19 [I 2022-10-11 11:03:08] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00295 [I 2022-10-11 11:03:08] ...trap/engines/engine.py.230: loss: 0.22973 [I 2022-10-11 11:03:08] ...optimizers/trijoint.py.151: total_norm: 25.679901 [I 2022-10-11 11:03:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 1860/2304 [I 2022-10-11 11:03:16] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:09 [I 2022-10-11 11:03:16] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00296 [I 2022-10-11 11:03:16] ...trap/engines/engine.py.230: loss: 0.12683 [I 2022-10-11 11:03:16] ...optimizers/trijoint.py.151: total_norm: 9.509348 [I 2022-10-11 11:03:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 1870/2304 [I 2022-10-11 11:03:25] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:06 [I 2022-10-11 11:03:25] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00293 [I 2022-10-11 11:03:25] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-11 11:03:25] ...optimizers/trijoint.py.151: total_norm: 20.525291 [I 2022-10-11 11:03:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 1880/2304 [I 2022-10-11 11:03:33] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:05:54 [I 2022-10-11 11:03:33] ...trap/engines/engine.py.229: process: 0.83431 | load: 0.00295 [I 2022-10-11 11:03:33] ...trap/engines/engine.py.230: loss: 0.12801 [I 2022-10-11 11:03:33] ...optimizers/trijoint.py.151: total_norm: 7.239618 [I 2022-10-11 11:03:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 1890/2304 [I 2022-10-11 11:03:41] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:05:48 [I 2022-10-11 11:03:41] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00293 [I 2022-10-11 11:03:41] ...trap/engines/engine.py.230: loss: 0.11922 [I 2022-10-11 11:03:41] ...optimizers/trijoint.py.151: total_norm: 19.042517 [I 2022-10-11 11:03:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 1900/2304 [I 2022-10-11 11:03:50] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:05:38 [I 2022-10-11 11:03:50] ...trap/engines/engine.py.229: process: 0.83329 | load: 0.00295 [I 2022-10-11 11:03:50] ...trap/engines/engine.py.230: loss: 0.10448 [I 2022-10-11 11:03:50] ...optimizers/trijoint.py.151: total_norm: 16.979134 [I 2022-10-11 11:03:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 1910/2304 [I 2022-10-11 11:03:58] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:05:31 [I 2022-10-11 11:03:58] ...trap/engines/engine.py.229: process: 0.84110 | load: 0.00293 [I 2022-10-11 11:03:58] ...trap/engines/engine.py.230: loss: 0.14909 [I 2022-10-11 11:03:58] ...optimizers/trijoint.py.151: total_norm: 26.592171 [I 2022-10-11 11:04:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 1920/2304 [I 2022-10-11 11:04:07] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:05:23 [I 2022-10-11 11:04:07] ...trap/engines/engine.py.229: process: 0.83037 | load: 0.00293 [I 2022-10-11 11:04:07] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-11 11:04:07] ...optimizers/trijoint.py.151: total_norm: 16.808519 [I 2022-10-11 11:04:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 1930/2304 [I 2022-10-11 11:04:15] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:14 [I 2022-10-11 11:04:15] ...trap/engines/engine.py.229: process: 0.83622 | load: 0.00296 [I 2022-10-11 11:04:15] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-11 11:04:15] ...optimizers/trijoint.py.151: total_norm: 9.997764 [I 2022-10-11 11:04:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 1940/2304 [I 2022-10-11 11:04:23] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:06 [I 2022-10-11 11:04:23] ...trap/engines/engine.py.229: process: 0.84950 | load: 0.00295 [I 2022-10-11 11:04:23] ...trap/engines/engine.py.230: loss: 0.07069 [I 2022-10-11 11:04:23] ...optimizers/trijoint.py.151: total_norm: 10.310343 [I 2022-10-11 11:04:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 1950/2304 [I 2022-10-11 11:04:32] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:01 [I 2022-10-11 11:04:32] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00294 [I 2022-10-11 11:04:32] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-11 11:04:32] ...optimizers/trijoint.py.151: total_norm: 18.071661 [I 2022-10-11 11:04:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 1960/2304 [I 2022-10-11 11:04:41] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:04:54 [I 2022-10-11 11:04:41] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00296 [I 2022-10-11 11:04:41] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-11 11:04:41] ...optimizers/trijoint.py.151: total_norm: 9.392339 [I 2022-10-11 11:04:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 1970/2304 [I 2022-10-11 11:04:49] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:04:51 [I 2022-10-11 11:04:49] ...trap/engines/engine.py.229: process: 0.88123 | load: 0.00372 [I 2022-10-11 11:04:49] ...trap/engines/engine.py.230: loss: 0.08889 [I 2022-10-11 11:04:49] ...optimizers/trijoint.py.151: total_norm: 18.252707 [I 2022-10-11 11:04:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 1980/2304 [I 2022-10-11 11:04:58] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:04:40 [I 2022-10-11 11:04:58] ...trap/engines/engine.py.229: process: 0.85827 | load: 0.00372 [I 2022-10-11 11:04:58] ...trap/engines/engine.py.230: loss: 0.10494 [I 2022-10-11 11:04:58] ...optimizers/trijoint.py.151: total_norm: 15.287814 [I 2022-10-11 11:05:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 1990/2304 [I 2022-10-11 11:05:07] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:33 [I 2022-10-11 11:05:07] ...trap/engines/engine.py.229: process: 0.87216 | load: 0.00371 [I 2022-10-11 11:05:07] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-11 11:05:07] ...optimizers/trijoint.py.151: total_norm: 16.205187 [I 2022-10-11 11:05:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 2000/2304 [I 2022-10-11 11:05:15] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:20 [I 2022-10-11 11:05:15] ...trap/engines/engine.py.229: process: 0.85371 | load: 0.00294 [I 2022-10-11 11:05:15] ...trap/engines/engine.py.230: loss: 0.10878 [I 2022-10-11 11:05:15] ...optimizers/trijoint.py.151: total_norm: 10.482629 [I 2022-10-11 11:05:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 2010/2304 [I 2022-10-11 11:05:24] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:08 [I 2022-10-11 11:05:24] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00294 [I 2022-10-11 11:05:24] ...trap/engines/engine.py.230: loss: 0.11023 [I 2022-10-11 11:05:24] ...optimizers/trijoint.py.151: total_norm: 12.363641 [I 2022-10-11 11:05:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 2020/2304 [I 2022-10-11 11:05:32] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:03:57 [I 2022-10-11 11:05:32] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00295 [I 2022-10-11 11:05:32] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-11 11:05:32] ...optimizers/trijoint.py.151: total_norm: 26.183456 [I 2022-10-11 11:05:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 2030/2304 [I 2022-10-11 11:05:41] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:03:50 [I 2022-10-11 11:05:41] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00295 [I 2022-10-11 11:05:41] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-11 11:05:41] ...optimizers/trijoint.py.151: total_norm: 14.968677 [I 2022-10-11 11:05:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 2040/2304 [I 2022-10-11 11:05:49] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:03:40 [I 2022-10-11 11:05:49] ...trap/engines/engine.py.229: process: 0.82834 | load: 0.00293 [I 2022-10-11 11:05:49] ...trap/engines/engine.py.230: loss: 0.12767 [I 2022-10-11 11:05:49] ...optimizers/trijoint.py.151: total_norm: 15.783885 [I 2022-10-11 11:05:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 2050/2304 [I 2022-10-11 11:05:57] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:33 [I 2022-10-11 11:05:57] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00295 [I 2022-10-11 11:05:57] ...trap/engines/engine.py.230: loss: 0.13732 [I 2022-10-11 11:05:57] ...optimizers/trijoint.py.151: total_norm: 20.954533 [I 2022-10-11 11:06:06] ...trap/engines/engine.py.224: train: epoch 102 | batch 2060/2304 [I 2022-10-11 11:06:06] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:24 [I 2022-10-11 11:06:06] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00295 [I 2022-10-11 11:06:06] ...trap/engines/engine.py.230: loss: 0.10078 [I 2022-10-11 11:06:06] ...optimizers/trijoint.py.151: total_norm: 9.436046 [I 2022-10-11 11:06:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 2070/2304 [I 2022-10-11 11:06:14] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:17 [I 2022-10-11 11:06:14] ...trap/engines/engine.py.229: process: 0.83917 | load: 0.00294 [I 2022-10-11 11:06:14] ...trap/engines/engine.py.230: loss: 0.12889 [I 2022-10-11 11:06:14] ...optimizers/trijoint.py.151: total_norm: 11.018783 [I 2022-10-11 11:06:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 2080/2304 [I 2022-10-11 11:06:23] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:08 [I 2022-10-11 11:06:23] ...trap/engines/engine.py.229: process: 0.83596 | load: 0.00294 [I 2022-10-11 11:06:23] ...trap/engines/engine.py.230: loss: 0.06444 [I 2022-10-11 11:06:23] ...optimizers/trijoint.py.151: total_norm: 7.965391 [I 2022-10-11 11:06:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 2090/2304 [I 2022-10-11 11:06:31] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:01 [I 2022-10-11 11:06:31] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00293 [I 2022-10-11 11:06:31] ...trap/engines/engine.py.230: loss: 0.10309 [I 2022-10-11 11:06:31] ...optimizers/trijoint.py.151: total_norm: 30.767221 [I 2022-10-11 11:06:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 2100/2304 [I 2022-10-11 11:06:40] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:02:51 [I 2022-10-11 11:06:40] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00294 [I 2022-10-11 11:06:40] ...trap/engines/engine.py.230: loss: 0.25137 [I 2022-10-11 11:06:40] ...optimizers/trijoint.py.151: total_norm: 36.832546 [I 2022-10-11 11:06:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 2110/2304 [I 2022-10-11 11:06:48] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:02:42 [I 2022-10-11 11:06:48] ...trap/engines/engine.py.229: process: 0.85003 | load: 0.00293 [I 2022-10-11 11:06:48] ...trap/engines/engine.py.230: loss: 0.12904 [I 2022-10-11 11:06:48] ...optimizers/trijoint.py.151: total_norm: 25.675268 [I 2022-10-11 11:06:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 2120/2304 [I 2022-10-11 11:06:56] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:02:33 [I 2022-10-11 11:06:56] ...trap/engines/engine.py.229: process: 0.82893 | load: 0.00293 [I 2022-10-11 11:06:56] ...trap/engines/engine.py.230: loss: 0.12756 [I 2022-10-11 11:06:56] ...optimizers/trijoint.py.151: total_norm: 13.467699 [I 2022-10-11 11:07:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 2130/2304 [I 2022-10-11 11:07:05] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:25 [I 2022-10-11 11:07:05] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00293 [I 2022-10-11 11:07:05] ...trap/engines/engine.py.230: loss: 0.08982 [I 2022-10-11 11:07:05] ...optimizers/trijoint.py.151: total_norm: 12.139073 [I 2022-10-11 11:07:13] ...trap/engines/engine.py.224: train: epoch 102 | batch 2140/2304 [I 2022-10-11 11:07:13] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:18 [I 2022-10-11 11:07:13] ...trap/engines/engine.py.229: process: 0.83907 | load: 0.00293 [I 2022-10-11 11:07:13] ...trap/engines/engine.py.230: loss: 0.10765 [I 2022-10-11 11:07:13] ...optimizers/trijoint.py.151: total_norm: 25.613968 [I 2022-10-11 11:07:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 2150/2304 [I 2022-10-11 11:07:22] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:08 [I 2022-10-11 11:07:22] ...trap/engines/engine.py.229: process: 0.83658 | load: 0.00293 [I 2022-10-11 11:07:22] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-11 11:07:22] ...optimizers/trijoint.py.151: total_norm: 13.448825 [I 2022-10-11 11:07:30] ...trap/engines/engine.py.224: train: epoch 102 | batch 2160/2304 [I 2022-10-11 11:07:30] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:01 [I 2022-10-11 11:07:30] ...trap/engines/engine.py.229: process: 0.83571 | load: 0.00296 [I 2022-10-11 11:07:30] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-11 11:07:30] ...optimizers/trijoint.py.151: total_norm: 10.126398 [I 2022-10-11 11:07:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 2170/2304 [I 2022-10-11 11:07:38] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:01:53 [I 2022-10-11 11:07:38] ...trap/engines/engine.py.229: process: 0.85346 | load: 0.00294 [I 2022-10-11 11:07:38] ...trap/engines/engine.py.230: loss: 0.09079 [I 2022-10-11 11:07:38] ...optimizers/trijoint.py.151: total_norm: 22.465185 [I 2022-10-11 11:07:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 2180/2304 [I 2022-10-11 11:07:47] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:01:44 [I 2022-10-11 11:07:47] ...trap/engines/engine.py.229: process: 0.83641 | load: 0.00294 [I 2022-10-11 11:07:47] ...trap/engines/engine.py.230: loss: 0.10048 [I 2022-10-11 11:07:47] ...optimizers/trijoint.py.151: total_norm: 11.291511 [I 2022-10-11 11:07:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 2190/2304 [I 2022-10-11 11:07:55] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:01:36 [I 2022-10-11 11:07:55] ...trap/engines/engine.py.229: process: 0.82857 | load: 0.00295 [I 2022-10-11 11:07:55] ...trap/engines/engine.py.230: loss: 0.15494 [I 2022-10-11 11:07:55] ...optimizers/trijoint.py.151: total_norm: 36.141140 [I 2022-10-11 11:08:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 2200/2304 [I 2022-10-11 11:08:04] ...trap/engines/engine.py.228: elapsed: 0:31:07 | left: 0:01:28 [I 2022-10-11 11:08:04] ...trap/engines/engine.py.229: process: 0.83638 | load: 0.00293 [I 2022-10-11 11:08:04] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-11 11:08:04] ...optimizers/trijoint.py.151: total_norm: 17.081453 [I 2022-10-11 11:08:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 2210/2304 [I 2022-10-11 11:08:12] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:01:19 [I 2022-10-11 11:08:12] ...trap/engines/engine.py.229: process: 0.86025 | load: 0.00297 [I 2022-10-11 11:08:12] ...trap/engines/engine.py.230: loss: 0.17957 [I 2022-10-11 11:08:12] ...optimizers/trijoint.py.151: total_norm: 15.406710 [I 2022-10-11 11:08:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 2220/2304 [I 2022-10-11 11:08:39] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:04:23 [I 2022-10-11 11:08:39] ...trap/engines/engine.py.229: process: 0.84134 | load: 0.00294 [I 2022-10-11 11:08:39] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-11 11:08:39] ...optimizers/trijoint.py.151: total_norm: 19.070379 [I 2022-10-11 11:08:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 2230/2304 [I 2022-10-11 11:08:47] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:21 [I 2022-10-11 11:08:47] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00292 [I 2022-10-11 11:08:47] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-11 11:08:47] ...optimizers/trijoint.py.151: total_norm: 19.829514 [I 2022-10-11 11:08:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 2240/2304 [I 2022-10-11 11:08:56] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:00:55 [I 2022-10-11 11:08:56] ...trap/engines/engine.py.229: process: 0.82475 | load: 0.00296 [I 2022-10-11 11:08:56] ...trap/engines/engine.py.230: loss: 0.11260 [I 2022-10-11 11:08:56] ...optimizers/trijoint.py.151: total_norm: 10.273247 [I 2022-10-11 11:09:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 2250/2304 [I 2022-10-11 11:09:04] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:45 [I 2022-10-11 11:09:04] ...trap/engines/engine.py.229: process: 0.86245 | load: 0.00294 [I 2022-10-11 11:09:04] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-11 11:09:04] ...optimizers/trijoint.py.151: total_norm: 9.217537 [I 2022-10-11 11:09:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 2260/2304 [I 2022-10-11 11:09:12] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:36 [I 2022-10-11 11:09:13] ...trap/engines/engine.py.229: process: 0.83416 | load: 0.00295 [I 2022-10-11 11:09:13] ...trap/engines/engine.py.230: loss: 0.19653 [I 2022-10-11 11:09:13] ...optimizers/trijoint.py.151: total_norm: 15.882644 [I 2022-10-11 11:09:21] ...trap/engines/engine.py.224: train: epoch 102 | batch 2270/2304 [I 2022-10-11 11:09:21] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:28 [I 2022-10-11 11:09:21] ...trap/engines/engine.py.229: process: 0.84224 | load: 0.00292 [I 2022-10-11 11:09:21] ...trap/engines/engine.py.230: loss: 0.10444 [I 2022-10-11 11:09:21] ...optimizers/trijoint.py.151: total_norm: 16.093077 [I 2022-10-11 11:09:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 2280/2304 [I 2022-10-11 11:09:29] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:20 [I 2022-10-11 11:09:29] ...trap/engines/engine.py.229: process: 0.83738 | load: 0.00293 [I 2022-10-11 11:09:29] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-11 11:09:29] ...optimizers/trijoint.py.151: total_norm: 20.845156 [I 2022-10-11 11:09:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 2290/2304 [I 2022-10-11 11:09:38] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:11 [I 2022-10-11 11:09:38] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00292 [I 2022-10-11 11:09:38] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-11 11:09:38] ...optimizers/trijoint.py.151: total_norm: 22.713781 [I 2022-10-11 11:09:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 2300/2304 [I 2022-10-11 11:09:46] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:03 [I 2022-10-11 11:09:46] ...trap/engines/engine.py.229: process: 0.84106 | load: 0.00012 [I 2022-10-11 11:09:46] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-11 11:09:46] ...optimizers/trijoint.py.151: total_norm: 11.305260 [I 2022-10-11 11:09:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 2304/2304 [I 2022-10-11 11:09:50] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:00 [I 2022-10-11 11:09:50] ...trap/engines/engine.py.229: process: 0.83015 | load: 0.00011 [I 2022-10-11 11:09:50] ...trap/engines/engine.py.230: loss: 0.09080 [I 2022-10-11 11:09:50] ...optimizers/trijoint.py.151: total_norm: 11.654054 [S 2022-10-11 11:09:51] ...trap/engines/engine.py.129: train_epoch.epoch: 102 [S 2022-10-11 11:09:51] ...trap/engines/engine.py.129: train_epoch.loss: 0.12348957540479504 [I 2022-10-11 11:09:51] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 11:09:52] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 102 [I 2022-10-11 11:09:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 11:09:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 11:09:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 11:09:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 11:09:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 11:09:58] ...trap/engines/engine.py.321: eval: epoch 102 | batch 0/510 [I 2022-10-11 11:09:58] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:36 [I 2022-10-11 11:09:58] ...trap/engines/engine.py.326: process: 5.83536 | load: 3.30249 [I 2022-10-11 11:10:04] ...trap/engines/engine.py.321: eval: epoch 102 | batch 10/510 [I 2022-10-11 11:10:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:27 [I 2022-10-11 11:10:04] ...trap/engines/engine.py.326: process: 0.58831 | load: 0.00019 [I 2022-10-11 11:10:10] ...trap/engines/engine.py.321: eval: epoch 102 | batch 20/510 [I 2022-10-11 11:10:10] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:24 [I 2022-10-11 11:10:10] ...trap/engines/engine.py.326: process: 0.67694 | load: 0.00017 [I 2022-10-11 11:10:17] ...trap/engines/engine.py.321: eval: epoch 102 | batch 30/510 [I 2022-10-11 11:10:17] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:06:19 [I 2022-10-11 11:10:17] ...trap/engines/engine.py.326: process: 0.83197 | load: 0.00018 [I 2022-10-11 11:10:25] ...trap/engines/engine.py.321: eval: epoch 102 | batch 40/510 [I 2022-10-11 11:10:25] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:05:41 [I 2022-10-11 11:10:25] ...trap/engines/engine.py.326: process: 0.61739 | load: 0.00018 [I 2022-10-11 11:10:31] ...trap/engines/engine.py.321: eval: epoch 102 | batch 50/510 [I 2022-10-11 11:10:31] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:38 [I 2022-10-11 11:10:31] ...trap/engines/engine.py.326: process: 0.60056 | load: 0.00017 [I 2022-10-11 11:10:37] ...trap/engines/engine.py.321: eval: epoch 102 | batch 60/510 [I 2022-10-11 11:10:37] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:04:22 [I 2022-10-11 11:10:37] ...trap/engines/engine.py.326: process: 0.57643 | load: 0.00018 [I 2022-10-11 11:10:43] ...trap/engines/engine.py.321: eval: epoch 102 | batch 70/510 [I 2022-10-11 11:10:43] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:04:21 [I 2022-10-11 11:10:43] ...trap/engines/engine.py.326: process: 0.52080 | load: 0.00018 [I 2022-10-11 11:10:50] ...trap/engines/engine.py.321: eval: epoch 102 | batch 80/510 [I 2022-10-11 11:10:50] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:05:06 [I 2022-10-11 11:10:50] ...trap/engines/engine.py.326: process: 0.48714 | load: 0.00017 [I 2022-10-11 11:10:55] ...trap/engines/engine.py.321: eval: epoch 102 | batch 90/510 [I 2022-10-11 11:10:55] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:51 [I 2022-10-11 11:10:55] ...trap/engines/engine.py.326: process: 0.57447 | load: 0.00018 [I 2022-10-11 11:11:01] ...trap/engines/engine.py.321: eval: epoch 102 | batch 100/510 [I 2022-10-11 11:11:01] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:03:10 [I 2022-10-11 11:11:01] ...trap/engines/engine.py.326: process: 0.30787 | load: 0.00018 [I 2022-10-11 11:11:04] ...trap/engines/engine.py.321: eval: epoch 102 | batch 110/510 [I 2022-10-11 11:11:04] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:11 [I 2022-10-11 11:11:04] ...trap/engines/engine.py.326: process: 0.31701 | load: 0.00018 [I 2022-10-11 11:11:07] ...trap/engines/engine.py.321: eval: epoch 102 | batch 120/510 [I 2022-10-11 11:11:07] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:02:02 [I 2022-10-11 11:11:07] ...trap/engines/engine.py.326: process: 0.30980 | load: 0.00018 [I 2022-10-11 11:11:10] ...trap/engines/engine.py.321: eval: epoch 102 | batch 130/510 [I 2022-10-11 11:11:10] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:59 [I 2022-10-11 11:11:10] ...trap/engines/engine.py.326: process: 0.31300 | load: 0.00018 [I 2022-10-11 11:11:13] ...trap/engines/engine.py.321: eval: epoch 102 | batch 140/510 [I 2022-10-11 11:11:13] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:56 [I 2022-10-11 11:11:13] ...trap/engines/engine.py.326: process: 0.32062 | load: 0.00019 [I 2022-10-11 11:11:16] ...trap/engines/engine.py.321: eval: epoch 102 | batch 150/510 [I 2022-10-11 11:11:16] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:55 [I 2022-10-11 11:11:16] ...trap/engines/engine.py.326: process: 0.32295 | load: 0.00021 [I 2022-10-11 11:11:20] ...trap/engines/engine.py.321: eval: epoch 102 | batch 160/510 [I 2022-10-11 11:11:20] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:54 [I 2022-10-11 11:11:20] ...trap/engines/engine.py.326: process: 0.32378 | load: 0.00020 [I 2022-10-11 11:11:23] ...trap/engines/engine.py.321: eval: epoch 102 | batch 170/510 [I 2022-10-11 11:11:23] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:51 [I 2022-10-11 11:11:23] ...trap/engines/engine.py.326: process: 0.32891 | load: 0.00021 [I 2022-10-11 11:11:26] ...trap/engines/engine.py.321: eval: epoch 102 | batch 180/510 [I 2022-10-11 11:11:26] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:48 [I 2022-10-11 11:11:26] ...trap/engines/engine.py.326: process: 0.32730 | load: 0.00021 [I 2022-10-11 11:11:29] ...trap/engines/engine.py.321: eval: epoch 102 | batch 190/510 [I 2022-10-11 11:11:29] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:44 [I 2022-10-11 11:11:29] ...trap/engines/engine.py.326: process: 0.32422 | load: 0.00021 [I 2022-10-11 11:11:33] ...trap/engines/engine.py.321: eval: epoch 102 | batch 200/510 [I 2022-10-11 11:11:33] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:41 [I 2022-10-11 11:11:33] ...trap/engines/engine.py.326: process: 0.33086 | load: 0.00021 [I 2022-10-11 11:11:36] ...trap/engines/engine.py.321: eval: epoch 102 | batch 210/510 [I 2022-10-11 11:11:36] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:38 [I 2022-10-11 11:11:36] ...trap/engines/engine.py.326: process: 0.33777 | load: 0.00022 [I 2022-10-11 11:11:39] ...trap/engines/engine.py.321: eval: epoch 102 | batch 220/510 [I 2022-10-11 11:11:39] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:35 [I 2022-10-11 11:11:39] ...trap/engines/engine.py.326: process: 0.32846 | load: 0.00020 [I 2022-10-11 11:11:43] ...trap/engines/engine.py.321: eval: epoch 102 | batch 230/510 [I 2022-10-11 11:11:43] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:31 [I 2022-10-11 11:11:43] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00021 [I 2022-10-11 11:11:46] ...trap/engines/engine.py.321: eval: epoch 102 | batch 240/510 [I 2022-10-11 11:11:46] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:28 [I 2022-10-11 11:11:46] ...trap/engines/engine.py.326: process: 0.32374 | load: 0.00021 [I 2022-10-11 11:11:49] ...trap/engines/engine.py.321: eval: epoch 102 | batch 250/510 [I 2022-10-11 11:11:49] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:25 [I 2022-10-11 11:11:49] ...trap/engines/engine.py.326: process: 0.33163 | load: 0.00021 [I 2022-10-11 11:11:52] ...trap/engines/engine.py.321: eval: epoch 102 | batch 260/510 [I 2022-10-11 11:11:52] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:21 [I 2022-10-11 11:11:52] ...trap/engines/engine.py.326: process: 0.32808 | load: 0.00021 [I 2022-10-11 11:11:56] ...trap/engines/engine.py.321: eval: epoch 102 | batch 270/510 [I 2022-10-11 11:11:56] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:19 [I 2022-10-11 11:11:56] ...trap/engines/engine.py.326: process: 0.34088 | load: 0.00021 [I 2022-10-11 11:11:59] ...trap/engines/engine.py.321: eval: epoch 102 | batch 280/510 [I 2022-10-11 11:11:59] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:15 [I 2022-10-11 11:11:59] ...trap/engines/engine.py.326: process: 0.32885 | load: 0.00020 [I 2022-10-11 11:12:02] ...trap/engines/engine.py.321: eval: epoch 102 | batch 290/510 [I 2022-10-11 11:12:02] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:11 [I 2022-10-11 11:12:02] ...trap/engines/engine.py.326: process: 0.32744 | load: 0.00022 [I 2022-10-11 11:12:06] ...trap/engines/engine.py.321: eval: epoch 102 | batch 300/510 [I 2022-10-11 11:12:06] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:01:08 [I 2022-10-11 11:12:06] ...trap/engines/engine.py.326: process: 0.31993 | load: 0.00021 [I 2022-10-11 11:12:09] ...trap/engines/engine.py.321: eval: epoch 102 | batch 310/510 [I 2022-10-11 11:12:09] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:01:05 [I 2022-10-11 11:12:09] ...trap/engines/engine.py.326: process: 0.32880 | load: 0.00022 [I 2022-10-11 11:12:12] ...trap/engines/engine.py.321: eval: epoch 102 | batch 320/510 [I 2022-10-11 11:12:12] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:01:02 [I 2022-10-11 11:12:12] ...trap/engines/engine.py.326: process: 0.33253 | load: 0.00021 [I 2022-10-11 11:12:15] ...trap/engines/engine.py.321: eval: epoch 102 | batch 330/510 [I 2022-10-11 11:12:15] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:58 [I 2022-10-11 11:12:15] ...trap/engines/engine.py.326: process: 0.32452 | load: 0.00021 [I 2022-10-11 11:12:19] ...trap/engines/engine.py.321: eval: epoch 102 | batch 340/510 [I 2022-10-11 11:12:19] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:53 [I 2022-10-11 11:12:19] ...trap/engines/engine.py.326: process: 0.31037 | load: 0.00020 [I 2022-10-11 11:12:22] ...trap/engines/engine.py.321: eval: epoch 102 | batch 350/510 [I 2022-10-11 11:12:22] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:50 [I 2022-10-11 11:12:22] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00019 [I 2022-10-11 11:12:25] ...trap/engines/engine.py.321: eval: epoch 102 | batch 360/510 [I 2022-10-11 11:12:25] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:47 [I 2022-10-11 11:12:25] ...trap/engines/engine.py.326: process: 0.31809 | load: 0.00019 [I 2022-10-11 11:12:28] ...trap/engines/engine.py.321: eval: epoch 102 | batch 370/510 [I 2022-10-11 11:12:28] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:44 [I 2022-10-11 11:12:28] ...trap/engines/engine.py.326: process: 0.31268 | load: 0.00020 [I 2022-10-11 11:12:31] ...trap/engines/engine.py.321: eval: epoch 102 | batch 380/510 [I 2022-10-11 11:12:31] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:41 [I 2022-10-11 11:12:31] ...trap/engines/engine.py.326: process: 0.31557 | load: 0.00019 [I 2022-10-11 11:12:34] ...trap/engines/engine.py.321: eval: epoch 102 | batch 390/510 [I 2022-10-11 11:12:34] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:37 [I 2022-10-11 11:12:34] ...trap/engines/engine.py.326: process: 0.31317 | load: 0.00019 [I 2022-10-11 11:12:38] ...trap/engines/engine.py.321: eval: epoch 102 | batch 400/510 [I 2022-10-11 11:12:38] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:34 [I 2022-10-11 11:12:38] ...trap/engines/engine.py.326: process: 0.31079 | load: 0.00020 [I 2022-10-11 11:12:41] ...trap/engines/engine.py.321: eval: epoch 102 | batch 410/510 [I 2022-10-11 11:12:41] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:31 [I 2022-10-11 11:12:41] ...trap/engines/engine.py.326: process: 0.31820 | load: 0.00019 [I 2022-10-11 11:12:44] ...trap/engines/engine.py.321: eval: epoch 102 | batch 420/510 [I 2022-10-11 11:12:44] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:28 [I 2022-10-11 11:12:44] ...trap/engines/engine.py.326: process: 0.31443 | load: 0.00019 [I 2022-10-11 11:12:47] ...trap/engines/engine.py.321: eval: epoch 102 | batch 430/510 [I 2022-10-11 11:12:47] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:25 [I 2022-10-11 11:12:47] ...trap/engines/engine.py.326: process: 0.31841 | load: 0.00019 [I 2022-10-11 11:12:50] ...trap/engines/engine.py.321: eval: epoch 102 | batch 440/510 [I 2022-10-11 11:12:50] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:22 [I 2022-10-11 11:12:50] ...trap/engines/engine.py.326: process: 0.32098 | load: 0.00019 [I 2022-10-11 11:12:53] ...trap/engines/engine.py.321: eval: epoch 102 | batch 450/510 [I 2022-10-11 11:12:53] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:19 [I 2022-10-11 11:12:53] ...trap/engines/engine.py.326: process: 0.31702 | load: 0.00019 [I 2022-10-11 11:12:56] ...trap/engines/engine.py.321: eval: epoch 102 | batch 460/510 [I 2022-10-11 11:12:56] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:15 [I 2022-10-11 11:12:56] ...trap/engines/engine.py.326: process: 0.31615 | load: 0.00020 [I 2022-10-11 11:13:00] ...trap/engines/engine.py.321: eval: epoch 102 | batch 470/510 [I 2022-10-11 11:13:00] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:12 [I 2022-10-11 11:13:00] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00019 [I 2022-10-11 11:13:03] ...trap/engines/engine.py.321: eval: epoch 102 | batch 480/510 [I 2022-10-11 11:13:03] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:09 [I 2022-10-11 11:13:03] ...trap/engines/engine.py.326: process: 0.31416 | load: 0.00020 [I 2022-10-11 11:13:06] ...trap/engines/engine.py.321: eval: epoch 102 | batch 490/510 [I 2022-10-11 11:13:06] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:06 [I 2022-10-11 11:13:06] ...trap/engines/engine.py.326: process: 0.31665 | load: 0.00020 [I 2022-10-11 11:13:09] ...trap/engines/engine.py.321: eval: epoch 102 | batch 500/510 [I 2022-10-11 11:13:09] ...trap/engines/engine.py.325: elapsed: 0:03:16 | left: 0:00:03 [I 2022-10-11 11:13:09] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00019 [I 2022-10-11 11:13:12] ...trap/engines/engine.py.321: eval: epoch 102 | batch 510/510 [I 2022-10-11 11:13:12] ...trap/engines/engine.py.325: elapsed: 0:03:19 | left: 0:00:00 [I 2022-10-11 11:13:12] ...trap/engines/engine.py.326: process: 0.31543 | load: 0.00011 [S 2022-10-11 11:13:13] ...trap/engines/engine.py.132: eval_epoch.epoch: 102 [I 2022-10-11 11:13:13] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.744100 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907800 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936700 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735300 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909300 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937500 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.006347 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007795 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007471 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012232 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009122 [S 2022-10-11 11:13:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006233 [I 2022-10-11 11:13:54] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 11:13:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 11:13:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 11:13:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 11:13:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 11:13:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 11:14:02] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 11:14:05] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 11:14:05] ...trap/engines/engine.py.163: Training model on trainset for epoch 103 [I 2022-10-11 11:14:05] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 11:14:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 0/2304 [I 2022-10-11 11:14:08] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:55:00 [I 2022-10-11 11:14:08] ...trap/engines/engine.py.229: process: 2.99491 | load: 2.10561 [I 2022-10-11 11:14:08] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-11 11:14:08] ...optimizers/trijoint.py.151: total_norm: 19.770245 [I 2022-10-11 11:14:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 10/2304 [I 2022-10-11 11:14:17] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:13 [I 2022-10-11 11:14:17] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00592 [I 2022-10-11 11:14:17] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-11 11:14:17] ...optimizers/trijoint.py.151: total_norm: 17.955164 [I 2022-10-11 11:14:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 20/2304 [I 2022-10-11 11:14:25] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-11 11:14:25] ...trap/engines/engine.py.229: process: 0.83305 | load: 0.00548 [I 2022-10-11 11:14:25] ...trap/engines/engine.py.230: loss: 0.13277 [I 2022-10-11 11:14:25] ...optimizers/trijoint.py.151: total_norm: 24.881966 [I 2022-10-11 11:14:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 30/2304 [I 2022-10-11 11:14:33] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:05 [I 2022-10-11 11:14:33] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00427 [I 2022-10-11 11:14:33] ...trap/engines/engine.py.230: loss: 0.08118 [I 2022-10-11 11:14:33] ...optimizers/trijoint.py.151: total_norm: 22.947872 [I 2022-10-11 11:14:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 40/2304 [I 2022-10-11 11:14:42] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:00 [I 2022-10-11 11:14:42] ...trap/engines/engine.py.229: process: 0.85216 | load: 0.00352 [I 2022-10-11 11:14:42] ...trap/engines/engine.py.230: loss: 0.09520 [I 2022-10-11 11:14:42] ...optimizers/trijoint.py.151: total_norm: 9.909472 [I 2022-10-11 11:14:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 50/2304 [I 2022-10-11 11:14:50] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:26 [I 2022-10-11 11:14:50] ...trap/engines/engine.py.229: process: 0.84012 | load: 0.00366 [I 2022-10-11 11:14:50] ...trap/engines/engine.py.230: loss: 0.08159 [I 2022-10-11 11:14:50] ...optimizers/trijoint.py.151: total_norm: 6.757601 [I 2022-10-11 11:14:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 60/2304 [I 2022-10-11 11:14:59] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:36 [I 2022-10-11 11:14:59] ...trap/engines/engine.py.229: process: 0.85577 | load: 0.00345 [I 2022-10-11 11:14:59] ...trap/engines/engine.py.230: loss: 0.10640 [I 2022-10-11 11:14:59] ...optimizers/trijoint.py.151: total_norm: 12.847035 [I 2022-10-11 11:15:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 70/2304 [I 2022-10-11 11:15:07] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:15 [I 2022-10-11 11:15:07] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00346 [I 2022-10-11 11:15:07] ...trap/engines/engine.py.230: loss: 0.12638 [I 2022-10-11 11:15:07] ...optimizers/trijoint.py.151: total_norm: 12.186657 [I 2022-10-11 11:15:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 80/2304 [I 2022-10-11 11:15:16] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:02 [I 2022-10-11 11:15:16] ...trap/engines/engine.py.229: process: 0.83287 | load: 0.00338 [I 2022-10-11 11:15:16] ...trap/engines/engine.py.230: loss: 0.07772 [I 2022-10-11 11:15:16] ...optimizers/trijoint.py.151: total_norm: 10.520760 [I 2022-10-11 11:15:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 90/2304 [I 2022-10-11 11:15:24] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:53 [I 2022-10-11 11:15:24] ...trap/engines/engine.py.229: process: 0.82778 | load: 0.00335 [I 2022-10-11 11:15:24] ...trap/engines/engine.py.230: loss: 0.10823 [I 2022-10-11 11:15:24] ...optimizers/trijoint.py.151: total_norm: 22.203485 [I 2022-10-11 11:15:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 100/2304 [I 2022-10-11 11:15:32] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:04 [I 2022-10-11 11:15:32] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00349 [I 2022-10-11 11:15:32] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-11 11:15:32] ...optimizers/trijoint.py.151: total_norm: 16.901203 [I 2022-10-11 11:15:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 110/2304 [I 2022-10-11 11:15:41] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:30:54 [I 2022-10-11 11:15:41] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00294 [I 2022-10-11 11:15:41] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-11 11:15:41] ...optimizers/trijoint.py.151: total_norm: 13.263749 [I 2022-10-11 11:15:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 120/2304 [I 2022-10-11 11:15:49] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:51 [I 2022-10-11 11:15:49] ...trap/engines/engine.py.229: process: 0.87071 | load: 0.00329 [I 2022-10-11 11:15:49] ...trap/engines/engine.py.230: loss: 0.08855 [I 2022-10-11 11:15:49] ...optimizers/trijoint.py.151: total_norm: 10.154162 [I 2022-10-11 11:15:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 130/2304 [I 2022-10-11 11:15:58] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:36 [I 2022-10-11 11:15:58] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00292 [I 2022-10-11 11:15:58] ...trap/engines/engine.py.230: loss: 0.20816 [I 2022-10-11 11:15:58] ...optimizers/trijoint.py.151: total_norm: 24.481012 [I 2022-10-11 11:16:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 140/2304 [I 2022-10-11 11:16:06] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:20 [I 2022-10-11 11:16:06] ...trap/engines/engine.py.229: process: 0.83448 | load: 0.00295 [I 2022-10-11 11:16:06] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-11 11:16:06] ...optimizers/trijoint.py.151: total_norm: 13.701741 [I 2022-10-11 11:16:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 150/2304 [I 2022-10-11 11:16:15] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:23 [I 2022-10-11 11:16:15] ...trap/engines/engine.py.229: process: 0.86019 | load: 0.00304 [I 2022-10-11 11:16:15] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-11 11:16:15] ...optimizers/trijoint.py.151: total_norm: 9.409922 [I 2022-10-11 11:16:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 160/2304 [I 2022-10-11 11:16:23] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:07 [I 2022-10-11 11:16:23] ...trap/engines/engine.py.229: process: 0.84838 | load: 0.00294 [I 2022-10-11 11:16:23] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-11 11:16:23] ...optimizers/trijoint.py.151: total_norm: 10.122118 [I 2022-10-11 11:16:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 170/2304 [I 2022-10-11 11:16:32] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:06 [I 2022-10-11 11:16:32] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00292 [I 2022-10-11 11:16:32] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-11 11:16:32] ...optimizers/trijoint.py.151: total_norm: 11.788717 [I 2022-10-11 11:16:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 180/2304 [I 2022-10-11 11:16:40] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:29:51 [I 2022-10-11 11:16:40] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00319 [I 2022-10-11 11:16:40] ...trap/engines/engine.py.230: loss: 0.20642 [I 2022-10-11 11:16:40] ...optimizers/trijoint.py.151: total_norm: 30.258871 [I 2022-10-11 11:16:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 190/2304 [I 2022-10-11 11:16:49] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:47 [I 2022-10-11 11:16:49] ...trap/engines/engine.py.229: process: 0.85643 | load: 0.00294 [I 2022-10-11 11:16:49] ...trap/engines/engine.py.230: loss: 0.16709 [I 2022-10-11 11:16:49] ...optimizers/trijoint.py.151: total_norm: 12.852533 [I 2022-10-11 11:16:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 200/2304 [I 2022-10-11 11:16:57] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:29:44 [I 2022-10-11 11:16:57] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00294 [I 2022-10-11 11:16:57] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-11 11:16:57] ...optimizers/trijoint.py.151: total_norm: 18.261129 [I 2022-10-11 11:17:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 210/2304 [I 2022-10-11 11:17:05] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:31 [I 2022-10-11 11:17:05] ...trap/engines/engine.py.229: process: 0.83865 | load: 0.00294 [I 2022-10-11 11:17:05] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-11 11:17:05] ...optimizers/trijoint.py.151: total_norm: 10.255072 [I 2022-10-11 11:17:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 220/2304 [I 2022-10-11 11:17:14] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:29:25 [I 2022-10-11 11:17:14] ...trap/engines/engine.py.229: process: 0.84060 | load: 0.00311 [I 2022-10-11 11:17:14] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-11 11:17:14] ...optimizers/trijoint.py.151: total_norm: 14.691503 [I 2022-10-11 11:17:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 230/2304 [I 2022-10-11 11:17:22] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:29:18 [I 2022-10-11 11:17:22] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00293 [I 2022-10-11 11:17:22] ...trap/engines/engine.py.230: loss: 0.15243 [I 2022-10-11 11:17:22] ...optimizers/trijoint.py.151: total_norm: 22.825733 [I 2022-10-11 11:17:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 240/2304 [I 2022-10-11 11:17:31] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:29:05 [I 2022-10-11 11:17:31] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00296 [I 2022-10-11 11:17:31] ...trap/engines/engine.py.230: loss: 0.09590 [I 2022-10-11 11:17:31] ...optimizers/trijoint.py.151: total_norm: 8.297585 [I 2022-10-11 11:17:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 250/2304 [I 2022-10-11 11:17:39] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:28:55 [I 2022-10-11 11:17:39] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00294 [I 2022-10-11 11:17:39] ...trap/engines/engine.py.230: loss: 0.08876 [I 2022-10-11 11:17:39] ...optimizers/trijoint.py.151: total_norm: 21.169754 [I 2022-10-11 11:17:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 260/2304 [I 2022-10-11 11:17:48] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:28:54 [I 2022-10-11 11:17:48] ...trap/engines/engine.py.229: process: 0.84210 | load: 0.00293 [I 2022-10-11 11:17:48] ...trap/engines/engine.py.230: loss: 0.11448 [I 2022-10-11 11:17:48] ...optimizers/trijoint.py.151: total_norm: 18.025255 [I 2022-10-11 11:17:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 270/2304 [I 2022-10-11 11:17:56] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:51 [I 2022-10-11 11:17:56] ...trap/engines/engine.py.229: process: 0.85554 | load: 0.00295 [I 2022-10-11 11:17:56] ...trap/engines/engine.py.230: loss: 0.08647 [I 2022-10-11 11:17:56] ...optimizers/trijoint.py.151: total_norm: 11.528483 [I 2022-10-11 11:18:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 280/2304 [I 2022-10-11 11:18:05] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:28:39 [I 2022-10-11 11:18:05] ...trap/engines/engine.py.229: process: 0.83924 | load: 0.00303 [I 2022-10-11 11:18:05] ...trap/engines/engine.py.230: loss: 0.24919 [I 2022-10-11 11:18:05] ...optimizers/trijoint.py.151: total_norm: 14.471562 [I 2022-10-11 11:18:13] ...trap/engines/engine.py.224: train: epoch 103 | batch 290/2304 [I 2022-10-11 11:18:13] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:28:13 [I 2022-10-11 11:18:13] ...trap/engines/engine.py.229: process: 0.84136 | load: 0.00314 [I 2022-10-11 11:18:13] ...trap/engines/engine.py.230: loss: 0.12054 [I 2022-10-11 11:18:13] ...optimizers/trijoint.py.151: total_norm: 15.729329 [I 2022-10-11 11:18:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 300/2304 [I 2022-10-11 11:18:22] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:28:12 [I 2022-10-11 11:18:22] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00293 [I 2022-10-11 11:18:22] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-11 11:18:22] ...optimizers/trijoint.py.151: total_norm: 78.727364 [I 2022-10-11 11:18:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 310/2304 [I 2022-10-11 11:18:30] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:28:02 [I 2022-10-11 11:18:30] ...trap/engines/engine.py.229: process: 0.85413 | load: 0.00304 [I 2022-10-11 11:18:30] ...trap/engines/engine.py.230: loss: 0.08583 [I 2022-10-11 11:18:30] ...optimizers/trijoint.py.151: total_norm: 16.991184 [I 2022-10-11 11:18:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 320/2304 [I 2022-10-11 11:18:39] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:27:44 [I 2022-10-11 11:18:39] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00293 [I 2022-10-11 11:18:39] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-11 11:18:39] ...optimizers/trijoint.py.151: total_norm: 23.617979 [I 2022-10-11 11:18:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 330/2304 [I 2022-10-11 11:18:47] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:27:43 [I 2022-10-11 11:18:47] ...trap/engines/engine.py.229: process: 0.83575 | load: 0.00304 [I 2022-10-11 11:18:47] ...trap/engines/engine.py.230: loss: 0.10140 [I 2022-10-11 11:18:47] ...optimizers/trijoint.py.151: total_norm: 11.196876 [I 2022-10-11 11:18:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 340/2304 [I 2022-10-11 11:18:56] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:27:34 [I 2022-10-11 11:18:56] ...trap/engines/engine.py.229: process: 0.83199 | load: 0.00293 [I 2022-10-11 11:18:56] ...trap/engines/engine.py.230: loss: 0.11657 [I 2022-10-11 11:18:56] ...optimizers/trijoint.py.151: total_norm: 9.791880 [I 2022-10-11 11:19:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 350/2304 [I 2022-10-11 11:19:04] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:27:27 [I 2022-10-11 11:19:04] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00296 [I 2022-10-11 11:19:04] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-11 11:19:04] ...optimizers/trijoint.py.151: total_norm: 24.028454 [I 2022-10-11 11:19:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 360/2304 [I 2022-10-11 11:19:12] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:27:14 [I 2022-10-11 11:19:12] ...trap/engines/engine.py.229: process: 0.82309 | load: 0.00293 [I 2022-10-11 11:19:12] ...trap/engines/engine.py.230: loss: 0.09020 [I 2022-10-11 11:19:12] ...optimizers/trijoint.py.151: total_norm: 16.517382 [I 2022-10-11 11:19:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 370/2304 [I 2022-10-11 11:19:21] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:27:05 [I 2022-10-11 11:19:21] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00305 [I 2022-10-11 11:19:21] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-11 11:19:21] ...optimizers/trijoint.py.151: total_norm: 11.776918 [I 2022-10-11 11:19:29] ...trap/engines/engine.py.224: train: epoch 103 | batch 380/2304 [I 2022-10-11 11:19:29] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:26:57 [I 2022-10-11 11:19:29] ...trap/engines/engine.py.229: process: 0.83877 | load: 0.00293 [I 2022-10-11 11:19:29] ...trap/engines/engine.py.230: loss: 0.10491 [I 2022-10-11 11:19:29] ...optimizers/trijoint.py.151: total_norm: 13.903914 [I 2022-10-11 11:19:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 390/2304 [I 2022-10-11 11:19:38] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:26:51 [I 2022-10-11 11:19:38] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00293 [I 2022-10-11 11:19:38] ...trap/engines/engine.py.230: loss: 0.16990 [I 2022-10-11 11:19:38] ...optimizers/trijoint.py.151: total_norm: 8.440618 [I 2022-10-11 11:19:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 400/2304 [I 2022-10-11 11:19:46] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:45 [I 2022-10-11 11:19:46] ...trap/engines/engine.py.229: process: 0.83263 | load: 0.00293 [I 2022-10-11 11:19:46] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-11 11:19:46] ...optimizers/trijoint.py.151: total_norm: 51.329800 [I 2022-10-11 11:19:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 410/2304 [I 2022-10-11 11:19:55] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:36 [I 2022-10-11 11:19:55] ...trap/engines/engine.py.229: process: 0.84899 | load: 0.00294 [I 2022-10-11 11:19:55] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-11 11:19:55] ...optimizers/trijoint.py.151: total_norm: 10.863019 [I 2022-10-11 11:20:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 420/2304 [I 2022-10-11 11:20:03] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:27 [I 2022-10-11 11:20:03] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00293 [I 2022-10-11 11:20:03] ...trap/engines/engine.py.230: loss: 0.21702 [I 2022-10-11 11:20:03] ...optimizers/trijoint.py.151: total_norm: 28.847271 [I 2022-10-11 11:20:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 430/2304 [I 2022-10-11 11:20:11] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:17 [I 2022-10-11 11:20:11] ...trap/engines/engine.py.229: process: 0.84952 | load: 0.00297 [I 2022-10-11 11:20:11] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-11 11:20:11] ...optimizers/trijoint.py.151: total_norm: 16.120129 [I 2022-10-11 11:20:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 440/2304 [I 2022-10-11 11:20:20] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:14 [I 2022-10-11 11:20:20] ...trap/engines/engine.py.229: process: 0.83041 | load: 0.00291 [I 2022-10-11 11:20:20] ...trap/engines/engine.py.230: loss: 0.15066 [I 2022-10-11 11:20:20] ...optimizers/trijoint.py.151: total_norm: 9.933305 [I 2022-10-11 11:20:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 450/2304 [I 2022-10-11 11:20:28] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:09 [I 2022-10-11 11:20:28] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00294 [I 2022-10-11 11:20:28] ...trap/engines/engine.py.230: loss: 0.10287 [I 2022-10-11 11:20:28] ...optimizers/trijoint.py.151: total_norm: 28.826801 [I 2022-10-11 11:20:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 460/2304 [I 2022-10-11 11:20:37] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:03 [I 2022-10-11 11:20:37] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00292 [I 2022-10-11 11:20:37] ...trap/engines/engine.py.230: loss: 0.20088 [I 2022-10-11 11:20:37] ...optimizers/trijoint.py.151: total_norm: 31.383883 [I 2022-10-11 11:20:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 470/2304 [I 2022-10-11 11:20:45] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:25:48 [I 2022-10-11 11:20:45] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00293 [I 2022-10-11 11:20:45] ...trap/engines/engine.py.230: loss: 0.11927 [I 2022-10-11 11:20:45] ...optimizers/trijoint.py.151: total_norm: 14.513674 [I 2022-10-11 11:20:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 480/2304 [I 2022-10-11 11:20:54] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:25:40 [I 2022-10-11 11:20:54] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00294 [I 2022-10-11 11:20:54] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-11 11:20:54] ...optimizers/trijoint.py.151: total_norm: 14.428907 [I 2022-10-11 11:21:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 490/2304 [I 2022-10-11 11:21:02] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:25:28 [I 2022-10-11 11:21:02] ...trap/engines/engine.py.229: process: 0.83210 | load: 0.00293 [I 2022-10-11 11:21:02] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-11 11:21:02] ...optimizers/trijoint.py.151: total_norm: 19.280050 [I 2022-10-11 11:21:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 500/2304 [I 2022-10-11 11:21:11] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:25:20 [I 2022-10-11 11:21:11] ...trap/engines/engine.py.229: process: 0.82621 | load: 0.00294 [I 2022-10-11 11:21:11] ...trap/engines/engine.py.230: loss: 0.08244 [I 2022-10-11 11:21:11] ...optimizers/trijoint.py.151: total_norm: 15.870815 [I 2022-10-11 11:21:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 510/2304 [I 2022-10-11 11:21:19] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:25:07 [I 2022-10-11 11:21:19] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00300 [I 2022-10-11 11:21:19] ...trap/engines/engine.py.230: loss: 0.11621 [I 2022-10-11 11:21:19] ...optimizers/trijoint.py.151: total_norm: 13.906226 [I 2022-10-11 11:21:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 520/2304 [I 2022-10-11 11:21:27] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:25:08 [I 2022-10-11 11:21:27] ...trap/engines/engine.py.229: process: 0.84630 | load: 0.00293 [I 2022-10-11 11:21:27] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-11 11:21:27] ...optimizers/trijoint.py.151: total_norm: 12.508650 [I 2022-10-11 11:21:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 530/2304 [I 2022-10-11 11:21:36] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:24:56 [I 2022-10-11 11:21:36] ...trap/engines/engine.py.229: process: 0.84002 | load: 0.00292 [I 2022-10-11 11:21:36] ...trap/engines/engine.py.230: loss: 0.14401 [I 2022-10-11 11:21:36] ...optimizers/trijoint.py.151: total_norm: 16.136816 [I 2022-10-11 11:21:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 540/2304 [I 2022-10-11 11:21:44] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:24:44 [I 2022-10-11 11:21:44] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00294 [I 2022-10-11 11:21:44] ...trap/engines/engine.py.230: loss: 0.08104 [I 2022-10-11 11:21:44] ...optimizers/trijoint.py.151: total_norm: 11.449399 [I 2022-10-11 11:21:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 550/2304 [I 2022-10-11 11:21:53] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:24:44 [I 2022-10-11 11:21:53] ...trap/engines/engine.py.229: process: 0.84989 | load: 0.00294 [I 2022-10-11 11:21:53] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-11 11:21:53] ...optimizers/trijoint.py.151: total_norm: 18.955254 [I 2022-10-11 11:22:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 560/2304 [I 2022-10-11 11:22:01] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:24:37 [I 2022-10-11 11:22:01] ...trap/engines/engine.py.229: process: 0.84601 | load: 0.00294 [I 2022-10-11 11:22:01] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-11 11:22:01] ...optimizers/trijoint.py.151: total_norm: 9.387774 [I 2022-10-11 11:22:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 570/2304 [I 2022-10-11 11:22:10] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:24:18 [I 2022-10-11 11:22:10] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00294 [I 2022-10-11 11:22:10] ...trap/engines/engine.py.230: loss: 0.10399 [I 2022-10-11 11:22:10] ...optimizers/trijoint.py.151: total_norm: 22.916637 [I 2022-10-11 11:22:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 580/2304 [I 2022-10-11 11:22:18] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:24:17 [I 2022-10-11 11:22:18] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00291 [I 2022-10-11 11:22:18] ...trap/engines/engine.py.230: loss: 0.23385 [I 2022-10-11 11:22:18] ...optimizers/trijoint.py.151: total_norm: 13.526457 [I 2022-10-11 11:22:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 590/2304 [I 2022-10-11 11:22:27] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:23:56 [I 2022-10-11 11:22:27] ...trap/engines/engine.py.229: process: 0.83799 | load: 0.00296 [I 2022-10-11 11:22:27] ...trap/engines/engine.py.230: loss: 0.10192 [I 2022-10-11 11:22:27] ...optimizers/trijoint.py.151: total_norm: 18.008213 [I 2022-10-11 11:22:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 600/2304 [I 2022-10-11 11:22:35] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:23:54 [I 2022-10-11 11:22:35] ...trap/engines/engine.py.229: process: 0.84567 | load: 0.00292 [I 2022-10-11 11:22:35] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-11 11:22:35] ...optimizers/trijoint.py.151: total_norm: 20.298191 [I 2022-10-11 11:22:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 610/2304 [I 2022-10-11 11:22:43] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:23:40 [I 2022-10-11 11:22:43] ...trap/engines/engine.py.229: process: 0.82926 | load: 0.00293 [I 2022-10-11 11:22:43] ...trap/engines/engine.py.230: loss: 0.10933 [I 2022-10-11 11:22:43] ...optimizers/trijoint.py.151: total_norm: 9.846547 [I 2022-10-11 11:22:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 620/2304 [I 2022-10-11 11:22:52] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:23:35 [I 2022-10-11 11:22:52] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00295 [I 2022-10-11 11:22:52] ...trap/engines/engine.py.230: loss: 0.13094 [I 2022-10-11 11:22:52] ...optimizers/trijoint.py.151: total_norm: 18.719303 [I 2022-10-11 11:23:00] ...trap/engines/engine.py.224: train: epoch 103 | batch 630/2304 [I 2022-10-11 11:23:00] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:23:38 [I 2022-10-11 11:23:00] ...trap/engines/engine.py.229: process: 0.85518 | load: 0.00293 [I 2022-10-11 11:23:00] ...trap/engines/engine.py.230: loss: 0.11393 [I 2022-10-11 11:23:00] ...optimizers/trijoint.py.151: total_norm: 12.185748 [I 2022-10-11 11:23:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 640/2304 [I 2022-10-11 11:23:09] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:34 [I 2022-10-11 11:23:09] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00293 [I 2022-10-11 11:23:09] ...trap/engines/engine.py.230: loss: 0.09828 [I 2022-10-11 11:23:09] ...optimizers/trijoint.py.151: total_norm: 23.414825 [I 2022-10-11 11:23:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 650/2304 [I 2022-10-11 11:23:17] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:23:12 [I 2022-10-11 11:23:17] ...trap/engines/engine.py.229: process: 0.84127 | load: 0.00294 [I 2022-10-11 11:23:17] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-11 11:23:17] ...optimizers/trijoint.py.151: total_norm: 18.739601 [I 2022-10-11 11:23:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 660/2304 [I 2022-10-11 11:23:26] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:15 [I 2022-10-11 11:23:26] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00294 [I 2022-10-11 11:23:26] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-11 11:23:26] ...optimizers/trijoint.py.151: total_norm: 30.865913 [I 2022-10-11 11:23:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 670/2304 [I 2022-10-11 11:23:34] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:23:02 [I 2022-10-11 11:23:34] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00302 [I 2022-10-11 11:23:34] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-11 11:23:34] ...optimizers/trijoint.py.151: total_norm: 18.485500 [I 2022-10-11 11:23:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 680/2304 [I 2022-10-11 11:23:43] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:22:45 [I 2022-10-11 11:23:43] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00293 [I 2022-10-11 11:23:43] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-11 11:23:43] ...optimizers/trijoint.py.151: total_norm: 12.125471 [I 2022-10-11 11:23:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 690/2304 [I 2022-10-11 11:23:51] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:22:39 [I 2022-10-11 11:23:51] ...trap/engines/engine.py.229: process: 0.83797 | load: 0.00293 [I 2022-10-11 11:23:51] ...trap/engines/engine.py.230: loss: 0.09628 [I 2022-10-11 11:23:51] ...optimizers/trijoint.py.151: total_norm: 13.049479 [I 2022-10-11 11:23:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 700/2304 [I 2022-10-11 11:23:59] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:22:27 [I 2022-10-11 11:23:59] ...trap/engines/engine.py.229: process: 0.82698 | load: 0.00293 [I 2022-10-11 11:23:59] ...trap/engines/engine.py.230: loss: 0.17512 [I 2022-10-11 11:23:59] ...optimizers/trijoint.py.151: total_norm: 44.227276 [I 2022-10-11 11:24:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 710/2304 [I 2022-10-11 11:24:08] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:22 [I 2022-10-11 11:24:08] ...trap/engines/engine.py.229: process: 0.82258 | load: 0.00292 [I 2022-10-11 11:24:08] ...trap/engines/engine.py.230: loss: 0.10776 [I 2022-10-11 11:24:08] ...optimizers/trijoint.py.151: total_norm: 22.057859 [I 2022-10-11 11:24:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 720/2304 [I 2022-10-11 11:24:16] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:22:11 [I 2022-10-11 11:24:16] ...trap/engines/engine.py.229: process: 0.82616 | load: 0.00294 [I 2022-10-11 11:24:16] ...trap/engines/engine.py.230: loss: 0.08174 [I 2022-10-11 11:24:16] ...optimizers/trijoint.py.151: total_norm: 11.633900 [I 2022-10-11 11:24:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 730/2304 [I 2022-10-11 11:24:25] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:08 [I 2022-10-11 11:24:25] ...trap/engines/engine.py.229: process: 0.84138 | load: 0.00294 [I 2022-10-11 11:24:25] ...trap/engines/engine.py.230: loss: 0.08460 [I 2022-10-11 11:24:25] ...optimizers/trijoint.py.151: total_norm: 9.511847 [I 2022-10-11 11:24:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 740/2304 [I 2022-10-11 11:24:33] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:21:55 [I 2022-10-11 11:24:33] ...trap/engines/engine.py.229: process: 0.84310 | load: 0.00296 [I 2022-10-11 11:24:33] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-11 11:24:33] ...optimizers/trijoint.py.151: total_norm: 9.529089 [I 2022-10-11 11:24:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 750/2304 [I 2022-10-11 11:24:42] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:21:47 [I 2022-10-11 11:24:42] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00295 [I 2022-10-11 11:24:42] ...trap/engines/engine.py.230: loss: 0.13001 [I 2022-10-11 11:24:42] ...optimizers/trijoint.py.151: total_norm: 35.831543 [I 2022-10-11 11:24:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 760/2304 [I 2022-10-11 11:24:50] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:21:37 [I 2022-10-11 11:24:50] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00294 [I 2022-10-11 11:24:50] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-11 11:24:50] ...optimizers/trijoint.py.151: total_norm: 11.198791 [I 2022-10-11 11:24:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 770/2304 [I 2022-10-11 11:24:58] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:21:23 [I 2022-10-11 11:24:58] ...trap/engines/engine.py.229: process: 0.83101 | load: 0.00293 [I 2022-10-11 11:24:58] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-11 11:24:58] ...optimizers/trijoint.py.151: total_norm: 7.876508 [I 2022-10-11 11:25:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 780/2304 [I 2022-10-11 11:25:07] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:14 [I 2022-10-11 11:25:07] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00295 [I 2022-10-11 11:25:07] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-11 11:25:07] ...optimizers/trijoint.py.151: total_norm: 16.435415 [I 2022-10-11 11:25:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 790/2304 [I 2022-10-11 11:25:15] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:07 [I 2022-10-11 11:25:15] ...trap/engines/engine.py.229: process: 0.83622 | load: 0.00291 [I 2022-10-11 11:25:15] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-11 11:25:15] ...optimizers/trijoint.py.151: total_norm: 13.738195 [I 2022-10-11 11:25:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 800/2304 [I 2022-10-11 11:25:24] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:07 [I 2022-10-11 11:25:24] ...trap/engines/engine.py.229: process: 0.84240 | load: 0.00293 [I 2022-10-11 11:25:24] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-11 11:25:24] ...optimizers/trijoint.py.151: total_norm: 47.558147 [I 2022-10-11 11:25:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 810/2304 [I 2022-10-11 11:25:32] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:03 [I 2022-10-11 11:25:32] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00292 [I 2022-10-11 11:25:32] ...trap/engines/engine.py.230: loss: 0.11213 [I 2022-10-11 11:25:32] ...optimizers/trijoint.py.151: total_norm: 9.615281 [I 2022-10-11 11:25:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 820/2304 [I 2022-10-11 11:25:41] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:05 [I 2022-10-11 11:25:41] ...trap/engines/engine.py.229: process: 0.86241 | load: 0.00292 [I 2022-10-11 11:25:41] ...trap/engines/engine.py.230: loss: 0.11609 [I 2022-10-11 11:25:41] ...optimizers/trijoint.py.151: total_norm: 18.894096 [I 2022-10-11 11:25:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 830/2304 [I 2022-10-11 11:25:49] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:20:49 [I 2022-10-11 11:25:49] ...trap/engines/engine.py.229: process: 0.84107 | load: 0.00292 [I 2022-10-11 11:25:49] ...trap/engines/engine.py.230: loss: 0.09776 [I 2022-10-11 11:25:49] ...optimizers/trijoint.py.151: total_norm: 21.426432 [I 2022-10-11 11:25:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 840/2304 [I 2022-10-11 11:25:58] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:20:28 [I 2022-10-11 11:25:58] ...trap/engines/engine.py.229: process: 0.83547 | load: 0.00293 [I 2022-10-11 11:25:58] ...trap/engines/engine.py.230: loss: 0.09209 [I 2022-10-11 11:25:58] ...optimizers/trijoint.py.151: total_norm: 13.796715 [I 2022-10-11 11:26:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 850/2304 [I 2022-10-11 11:26:06] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:25 [I 2022-10-11 11:26:06] ...trap/engines/engine.py.229: process: 0.84770 | load: 0.00293 [I 2022-10-11 11:26:06] ...trap/engines/engine.py.230: loss: 0.08438 [I 2022-10-11 11:26:06] ...optimizers/trijoint.py.151: total_norm: 16.705753 [I 2022-10-11 11:26:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 860/2304 [I 2022-10-11 11:26:14] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:16 [I 2022-10-11 11:26:14] ...trap/engines/engine.py.229: process: 0.84656 | load: 0.00294 [I 2022-10-11 11:26:14] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-11 11:26:14] ...optimizers/trijoint.py.151: total_norm: 17.218142 [I 2022-10-11 11:26:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 870/2304 [I 2022-10-11 11:26:23] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:07 [I 2022-10-11 11:26:23] ...trap/engines/engine.py.229: process: 0.83886 | load: 0.00291 [I 2022-10-11 11:26:23] ...trap/engines/engine.py.230: loss: 0.08852 [I 2022-10-11 11:26:23] ...optimizers/trijoint.py.151: total_norm: 12.268405 [I 2022-10-11 11:26:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 880/2304 [I 2022-10-11 11:26:31] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:19:59 [I 2022-10-11 11:26:31] ...trap/engines/engine.py.229: process: 0.84161 | load: 0.00292 [I 2022-10-11 11:26:31] ...trap/engines/engine.py.230: loss: 0.07777 [I 2022-10-11 11:26:31] ...optimizers/trijoint.py.151: total_norm: 15.847305 [I 2022-10-11 11:26:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 890/2304 [I 2022-10-11 11:26:40] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:00 [I 2022-10-11 11:26:40] ...trap/engines/engine.py.229: process: 0.86615 | load: 0.00291 [I 2022-10-11 11:26:40] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-11 11:26:40] ...optimizers/trijoint.py.151: total_norm: 37.285194 [I 2022-10-11 11:26:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 900/2304 [I 2022-10-11 11:26:48] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:19:51 [I 2022-10-11 11:26:48] ...trap/engines/engine.py.229: process: 0.83659 | load: 0.00291 [I 2022-10-11 11:26:48] ...trap/engines/engine.py.230: loss: 0.08731 [I 2022-10-11 11:26:48] ...optimizers/trijoint.py.151: total_norm: 13.104275 [I 2022-10-11 11:26:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 910/2304 [I 2022-10-11 11:26:57] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:30 [I 2022-10-11 11:26:57] ...trap/engines/engine.py.229: process: 0.83774 | load: 0.00294 [I 2022-10-11 11:26:57] ...trap/engines/engine.py.230: loss: 0.14069 [I 2022-10-11 11:26:57] ...optimizers/trijoint.py.151: total_norm: 22.700029 [I 2022-10-11 11:27:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 920/2304 [I 2022-10-11 11:27:05] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:19:27 [I 2022-10-11 11:27:05] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00292 [I 2022-10-11 11:27:05] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-11 11:27:05] ...optimizers/trijoint.py.151: total_norm: 49.851059 [I 2022-10-11 11:27:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 930/2304 [I 2022-10-11 11:27:14] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:29 [I 2022-10-11 11:27:14] ...trap/engines/engine.py.229: process: 0.84027 | load: 0.00292 [I 2022-10-11 11:27:14] ...trap/engines/engine.py.230: loss: 0.12690 [I 2022-10-11 11:27:14] ...optimizers/trijoint.py.151: total_norm: 21.194143 [I 2022-10-11 11:27:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 940/2304 [I 2022-10-11 11:27:22] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:18:58 [I 2022-10-11 11:27:22] ...trap/engines/engine.py.229: process: 0.82382 | load: 0.00292 [I 2022-10-11 11:27:22] ...trap/engines/engine.py.230: loss: 0.10394 [I 2022-10-11 11:27:22] ...optimizers/trijoint.py.151: total_norm: 12.838045 [I 2022-10-11 11:27:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 950/2304 [I 2022-10-11 11:27:30] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:18:51 [I 2022-10-11 11:27:30] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00293 [I 2022-10-11 11:27:30] ...trap/engines/engine.py.230: loss: 0.12657 [I 2022-10-11 11:27:30] ...optimizers/trijoint.py.151: total_norm: 36.435768 [I 2022-10-11 11:27:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 960/2304 [I 2022-10-11 11:27:39] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:18:50 [I 2022-10-11 11:27:39] ...trap/engines/engine.py.229: process: 0.84955 | load: 0.00293 [I 2022-10-11 11:27:39] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-11 11:27:39] ...optimizers/trijoint.py.151: total_norm: 12.945835 [I 2022-10-11 11:27:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 970/2304 [I 2022-10-11 11:27:47] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:18:41 [I 2022-10-11 11:27:47] ...trap/engines/engine.py.229: process: 0.83831 | load: 0.00292 [I 2022-10-11 11:27:47] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-11 11:27:47] ...optimizers/trijoint.py.151: total_norm: 11.061836 [I 2022-10-11 11:27:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 980/2304 [I 2022-10-11 11:27:56] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:18:29 [I 2022-10-11 11:27:56] ...trap/engines/engine.py.229: process: 0.83646 | load: 0.00295 [I 2022-10-11 11:27:56] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-11 11:27:56] ...optimizers/trijoint.py.151: total_norm: 9.512275 [I 2022-10-11 11:28:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 990/2304 [I 2022-10-11 11:28:04] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:24 [I 2022-10-11 11:28:04] ...trap/engines/engine.py.229: process: 0.84724 | load: 0.00293 [I 2022-10-11 11:28:04] ...trap/engines/engine.py.230: loss: 0.09683 [I 2022-10-11 11:28:04] ...optimizers/trijoint.py.151: total_norm: 12.886488 [I 2022-10-11 11:28:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 1000/2304 [I 2022-10-11 11:28:12] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:18 [I 2022-10-11 11:28:12] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00292 [I 2022-10-11 11:28:12] ...trap/engines/engine.py.230: loss: 0.10755 [I 2022-10-11 11:28:12] ...optimizers/trijoint.py.151: total_norm: 13.878675 [I 2022-10-11 11:28:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 1010/2304 [I 2022-10-11 11:28:21] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:08 [I 2022-10-11 11:28:21] ...trap/engines/engine.py.229: process: 0.83621 | load: 0.00291 [I 2022-10-11 11:28:21] ...trap/engines/engine.py.230: loss: 0.14457 [I 2022-10-11 11:28:21] ...optimizers/trijoint.py.151: total_norm: 57.171902 [I 2022-10-11 11:28:29] ...trap/engines/engine.py.224: train: epoch 103 | batch 1020/2304 [I 2022-10-11 11:28:29] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:17:53 [I 2022-10-11 11:28:29] ...trap/engines/engine.py.229: process: 0.82778 | load: 0.00296 [I 2022-10-11 11:28:29] ...trap/engines/engine.py.230: loss: 0.09166 [I 2022-10-11 11:28:29] ...optimizers/trijoint.py.151: total_norm: 9.637312 [I 2022-10-11 11:28:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 1030/2304 [I 2022-10-11 11:28:38] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:17:47 [I 2022-10-11 11:28:38] ...trap/engines/engine.py.229: process: 0.84974 | load: 0.00292 [I 2022-10-11 11:28:38] ...trap/engines/engine.py.230: loss: 0.09345 [I 2022-10-11 11:28:38] ...optimizers/trijoint.py.151: total_norm: 9.091070 [I 2022-10-11 11:28:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 1040/2304 [I 2022-10-11 11:28:46] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:17:45 [I 2022-10-11 11:28:46] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00292 [I 2022-10-11 11:28:46] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-11 11:28:46] ...optimizers/trijoint.py.151: total_norm: 10.459673 [I 2022-10-11 11:28:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 1050/2304 [I 2022-10-11 11:28:55] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:17:28 [I 2022-10-11 11:28:55] ...trap/engines/engine.py.229: process: 0.83538 | load: 0.00292 [I 2022-10-11 11:28:55] ...trap/engines/engine.py.230: loss: 0.12963 [I 2022-10-11 11:28:55] ...optimizers/trijoint.py.151: total_norm: 12.552299 [I 2022-10-11 11:29:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 1060/2304 [I 2022-10-11 11:29:03] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:17:24 [I 2022-10-11 11:29:03] ...trap/engines/engine.py.229: process: 0.82927 | load: 0.00294 [I 2022-10-11 11:29:03] ...trap/engines/engine.py.230: loss: 0.09287 [I 2022-10-11 11:29:03] ...optimizers/trijoint.py.151: total_norm: 10.523988 [I 2022-10-11 11:29:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 1070/2304 [I 2022-10-11 11:29:11] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:13 [I 2022-10-11 11:29:11] ...trap/engines/engine.py.229: process: 0.83311 | load: 0.00294 [I 2022-10-11 11:29:11] ...trap/engines/engine.py.230: loss: 0.14399 [I 2022-10-11 11:29:11] ...optimizers/trijoint.py.151: total_norm: 16.986254 [I 2022-10-11 11:29:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 1080/2304 [I 2022-10-11 11:29:20] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:08 [I 2022-10-11 11:29:20] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00291 [I 2022-10-11 11:29:20] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-11 11:29:20] ...optimizers/trijoint.py.151: total_norm: 27.894070 [I 2022-10-11 11:29:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 1090/2304 [I 2022-10-11 11:29:28] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:04 [I 2022-10-11 11:29:28] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00294 [I 2022-10-11 11:29:28] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-11 11:29:28] ...optimizers/trijoint.py.151: total_norm: 12.708509 [I 2022-10-11 11:29:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 1100/2304 [I 2022-10-11 11:29:37] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:00 [I 2022-10-11 11:29:37] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00295 [I 2022-10-11 11:29:37] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-11 11:29:37] ...optimizers/trijoint.py.151: total_norm: 11.171683 [I 2022-10-11 11:29:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 1110/2304 [I 2022-10-11 11:29:45] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:03 [I 2022-10-11 11:29:45] ...trap/engines/engine.py.229: process: 0.86406 | load: 0.00294 [I 2022-10-11 11:29:45] ...trap/engines/engine.py.230: loss: 0.09584 [I 2022-10-11 11:29:45] ...optimizers/trijoint.py.151: total_norm: 15.891391 [I 2022-10-11 11:29:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 1120/2304 [I 2022-10-11 11:29:54] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:16:48 [I 2022-10-11 11:29:54] ...trap/engines/engine.py.229: process: 0.85412 | load: 0.00292 [I 2022-10-11 11:29:54] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-11 11:29:54] ...optimizers/trijoint.py.151: total_norm: 20.103971 [I 2022-10-11 11:30:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 1130/2304 [I 2022-10-11 11:30:02] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:44 [I 2022-10-11 11:30:02] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00294 [I 2022-10-11 11:30:02] ...trap/engines/engine.py.230: loss: 0.10678 [I 2022-10-11 11:30:02] ...optimizers/trijoint.py.151: total_norm: 19.747265 [I 2022-10-11 11:30:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 1140/2304 [I 2022-10-11 11:30:11] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:29 [I 2022-10-11 11:30:11] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00294 [I 2022-10-11 11:30:11] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-11 11:30:11] ...optimizers/trijoint.py.151: total_norm: 22.946909 [I 2022-10-11 11:30:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 1150/2304 [I 2022-10-11 11:30:19] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:20 [I 2022-10-11 11:30:19] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00296 [I 2022-10-11 11:30:19] ...trap/engines/engine.py.230: loss: 0.10247 [I 2022-10-11 11:30:19] ...optimizers/trijoint.py.151: total_norm: 16.009396 [I 2022-10-11 11:30:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 1160/2304 [I 2022-10-11 11:30:28] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:14 [I 2022-10-11 11:30:28] ...trap/engines/engine.py.229: process: 0.85719 | load: 0.00298 [I 2022-10-11 11:30:28] ...trap/engines/engine.py.230: loss: 0.16897 [I 2022-10-11 11:30:28] ...optimizers/trijoint.py.151: total_norm: 17.110844 [I 2022-10-11 11:30:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 1170/2304 [I 2022-10-11 11:30:36] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:06 [I 2022-10-11 11:30:36] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00296 [I 2022-10-11 11:30:36] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-11 11:30:36] ...optimizers/trijoint.py.151: total_norm: 13.552488 [I 2022-10-11 11:30:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 1180/2304 [I 2022-10-11 11:30:45] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:15:59 [I 2022-10-11 11:30:45] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00298 [I 2022-10-11 11:30:45] ...trap/engines/engine.py.230: loss: 0.07742 [I 2022-10-11 11:30:45] ...optimizers/trijoint.py.151: total_norm: 11.798356 [I 2022-10-11 11:30:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 1190/2304 [I 2022-10-11 11:30:53] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:45 [I 2022-10-11 11:30:53] ...trap/engines/engine.py.229: process: 0.83668 | load: 0.00296 [I 2022-10-11 11:30:53] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-11 11:30:53] ...optimizers/trijoint.py.151: total_norm: 14.199495 [I 2022-10-11 11:31:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 1200/2304 [I 2022-10-11 11:31:02] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:36 [I 2022-10-11 11:31:02] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00298 [I 2022-10-11 11:31:02] ...trap/engines/engine.py.230: loss: 0.14504 [I 2022-10-11 11:31:02] ...optimizers/trijoint.py.151: total_norm: 14.781116 [I 2022-10-11 11:31:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 1210/2304 [I 2022-10-11 11:31:10] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:22 [I 2022-10-11 11:31:10] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00298 [I 2022-10-11 11:31:10] ...trap/engines/engine.py.230: loss: 0.09150 [I 2022-10-11 11:31:10] ...optimizers/trijoint.py.151: total_norm: 7.275663 [I 2022-10-11 11:31:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 1220/2304 [I 2022-10-11 11:31:19] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:37 [I 2022-10-11 11:31:19] ...trap/engines/engine.py.229: process: 0.87429 | load: 0.00371 [I 2022-10-11 11:31:19] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-11 11:31:19] ...optimizers/trijoint.py.151: total_norm: 22.888655 [I 2022-10-11 11:31:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 1230/2304 [I 2022-10-11 11:31:28] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:42 [I 2022-10-11 11:31:28] ...trap/engines/engine.py.229: process: 0.87813 | load: 0.00375 [I 2022-10-11 11:31:28] ...trap/engines/engine.py.230: loss: 0.11227 [I 2022-10-11 11:31:28] ...optimizers/trijoint.py.151: total_norm: 10.943785 [I 2022-10-11 11:31:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 1240/2304 [I 2022-10-11 11:31:36] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:27 [I 2022-10-11 11:31:36] ...trap/engines/engine.py.229: process: 0.87976 | load: 0.00371 [I 2022-10-11 11:31:36] ...trap/engines/engine.py.230: loss: 0.19765 [I 2022-10-11 11:31:36] ...optimizers/trijoint.py.151: total_norm: 10.957632 [I 2022-10-11 11:31:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 1250/2304 [I 2022-10-11 11:31:45] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:09 [I 2022-10-11 11:31:45] ...trap/engines/engine.py.229: process: 0.84633 | load: 0.00293 [I 2022-10-11 11:31:45] ...trap/engines/engine.py.230: loss: 0.08898 [I 2022-10-11 11:31:45] ...optimizers/trijoint.py.151: total_norm: 11.211371 [I 2022-10-11 11:31:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 1260/2304 [I 2022-10-11 11:31:54] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:14:39 [I 2022-10-11 11:31:54] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00293 [I 2022-10-11 11:31:54] ...trap/engines/engine.py.230: loss: 0.11286 [I 2022-10-11 11:31:54] ...optimizers/trijoint.py.151: total_norm: 19.880701 [I 2022-10-11 11:32:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 1270/2304 [I 2022-10-11 11:32:02] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:20 [I 2022-10-11 11:32:02] ...trap/engines/engine.py.229: process: 0.81415 | load: 0.00296 [I 2022-10-11 11:32:02] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-11 11:32:02] ...optimizers/trijoint.py.151: total_norm: 10.871653 [I 2022-10-11 11:32:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 1280/2304 [I 2022-10-11 11:32:10] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:14:14 [I 2022-10-11 11:32:10] ...trap/engines/engine.py.229: process: 0.83395 | load: 0.00293 [I 2022-10-11 11:32:10] ...trap/engines/engine.py.230: loss: 0.08246 [I 2022-10-11 11:32:10] ...optimizers/trijoint.py.151: total_norm: 14.703164 [I 2022-10-11 11:32:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 1290/2304 [I 2022-10-11 11:32:19] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:11 [I 2022-10-11 11:32:19] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00294 [I 2022-10-11 11:32:19] ...trap/engines/engine.py.230: loss: 0.09068 [I 2022-10-11 11:32:19] ...optimizers/trijoint.py.151: total_norm: 14.018098 [I 2022-10-11 11:32:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 1300/2304 [I 2022-10-11 11:32:27] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:07 [I 2022-10-11 11:32:27] ...trap/engines/engine.py.229: process: 0.84047 | load: 0.00294 [I 2022-10-11 11:32:27] ...trap/engines/engine.py.230: loss: 0.09779 [I 2022-10-11 11:32:27] ...optimizers/trijoint.py.151: total_norm: 13.658906 [I 2022-10-11 11:32:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 1310/2304 [I 2022-10-11 11:32:36] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:13:56 [I 2022-10-11 11:32:36] ...trap/engines/engine.py.229: process: 0.83801 | load: 0.00294 [I 2022-10-11 11:32:36] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-11 11:32:36] ...optimizers/trijoint.py.151: total_norm: 22.467180 [I 2022-10-11 11:32:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 1320/2304 [I 2022-10-11 11:32:44] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:13:48 [I 2022-10-11 11:32:44] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00292 [I 2022-10-11 11:32:44] ...trap/engines/engine.py.230: loss: 0.09114 [I 2022-10-11 11:32:44] ...optimizers/trijoint.py.151: total_norm: 10.377947 [I 2022-10-11 11:32:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 1330/2304 [I 2022-10-11 11:32:52] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:13:34 [I 2022-10-11 11:32:52] ...trap/engines/engine.py.229: process: 0.82902 | load: 0.00291 [I 2022-10-11 11:32:52] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-11 11:32:52] ...optimizers/trijoint.py.151: total_norm: 25.035107 [I 2022-10-11 11:33:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 1340/2304 [I 2022-10-11 11:33:01] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:30 [I 2022-10-11 11:33:01] ...trap/engines/engine.py.229: process: 0.84085 | load: 0.00293 [I 2022-10-11 11:33:01] ...trap/engines/engine.py.230: loss: 0.12495 [I 2022-10-11 11:33:01] ...optimizers/trijoint.py.151: total_norm: 13.213569 [I 2022-10-11 11:33:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 1350/2304 [I 2022-10-11 11:33:09] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:20 [I 2022-10-11 11:33:09] ...trap/engines/engine.py.229: process: 0.81298 | load: 0.00300 [I 2022-10-11 11:33:09] ...trap/engines/engine.py.230: loss: 0.10177 [I 2022-10-11 11:33:09] ...optimizers/trijoint.py.151: total_norm: 10.887943 [I 2022-10-11 11:33:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 1360/2304 [I 2022-10-11 11:33:18] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:16 [I 2022-10-11 11:33:18] ...trap/engines/engine.py.229: process: 0.83837 | load: 0.00293 [I 2022-10-11 11:33:18] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-11 11:33:18] ...optimizers/trijoint.py.151: total_norm: 21.154051 [I 2022-10-11 11:33:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 1370/2304 [I 2022-10-11 11:33:26] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:13 [I 2022-10-11 11:33:26] ...trap/engines/engine.py.229: process: 0.85990 | load: 0.00293 [I 2022-10-11 11:33:26] ...trap/engines/engine.py.230: loss: 0.09143 [I 2022-10-11 11:33:26] ...optimizers/trijoint.py.151: total_norm: 11.004263 [I 2022-10-11 11:33:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 1380/2304 [I 2022-10-11 11:33:35] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:01 [I 2022-10-11 11:33:35] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00292 [I 2022-10-11 11:33:35] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-11 11:33:35] ...optimizers/trijoint.py.151: total_norm: 19.679703 [I 2022-10-11 11:33:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 1390/2304 [I 2022-10-11 11:33:43] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:12:48 [I 2022-10-11 11:33:43] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00292 [I 2022-10-11 11:33:43] ...trap/engines/engine.py.230: loss: 0.09303 [I 2022-10-11 11:33:43] ...optimizers/trijoint.py.151: total_norm: 15.082608 [I 2022-10-11 11:33:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 1400/2304 [I 2022-10-11 11:33:51] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:39 [I 2022-10-11 11:33:51] ...trap/engines/engine.py.229: process: 0.83131 | load: 0.00293 [I 2022-10-11 11:33:51] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-11 11:33:51] ...optimizers/trijoint.py.151: total_norm: 7.579095 [I 2022-10-11 11:34:00] ...trap/engines/engine.py.224: train: epoch 103 | batch 1410/2304 [I 2022-10-11 11:34:00] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:12:31 [I 2022-10-11 11:34:00] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00296 [I 2022-10-11 11:34:00] ...trap/engines/engine.py.230: loss: 0.17773 [I 2022-10-11 11:34:00] ...optimizers/trijoint.py.151: total_norm: 13.243761 [I 2022-10-11 11:34:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 1420/2304 [I 2022-10-11 11:34:08] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:22 [I 2022-10-11 11:34:08] ...trap/engines/engine.py.229: process: 0.84192 | load: 0.00294 [I 2022-10-11 11:34:08] ...trap/engines/engine.py.230: loss: 0.14639 [I 2022-10-11 11:34:08] ...optimizers/trijoint.py.151: total_norm: 16.708275 [I 2022-10-11 11:34:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 1430/2304 [I 2022-10-11 11:34:17] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:12:18 [I 2022-10-11 11:34:17] ...trap/engines/engine.py.229: process: 0.84266 | load: 0.00293 [I 2022-10-11 11:34:17] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-11 11:34:17] ...optimizers/trijoint.py.151: total_norm: 8.571423 [I 2022-10-11 11:34:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 1440/2304 [I 2022-10-11 11:34:25] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:12:07 [I 2022-10-11 11:34:25] ...trap/engines/engine.py.229: process: 0.84488 | load: 0.00294 [I 2022-10-11 11:34:25] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-11 11:34:25] ...optimizers/trijoint.py.151: total_norm: 10.083219 [I 2022-10-11 11:34:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 1450/2304 [I 2022-10-11 11:34:33] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:11:55 [I 2022-10-11 11:34:33] ...trap/engines/engine.py.229: process: 0.84651 | load: 0.00293 [I 2022-10-11 11:34:33] ...trap/engines/engine.py.230: loss: 0.17263 [I 2022-10-11 11:34:33] ...optimizers/trijoint.py.151: total_norm: 14.596712 [I 2022-10-11 11:34:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 1460/2304 [I 2022-10-11 11:34:42] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:11:54 [I 2022-10-11 11:34:42] ...trap/engines/engine.py.229: process: 0.84864 | load: 0.00292 [I 2022-10-11 11:34:42] ...trap/engines/engine.py.230: loss: 0.15782 [I 2022-10-11 11:34:42] ...optimizers/trijoint.py.151: total_norm: 13.389688 [I 2022-10-11 11:34:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 1470/2304 [I 2022-10-11 11:34:50] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:11:44 [I 2022-10-11 11:34:50] ...trap/engines/engine.py.229: process: 0.84053 | load: 0.00293 [I 2022-10-11 11:34:50] ...trap/engines/engine.py.230: loss: 0.09577 [I 2022-10-11 11:34:50] ...optimizers/trijoint.py.151: total_norm: 10.955353 [I 2022-10-11 11:34:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 1480/2304 [I 2022-10-11 11:34:59] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:11:35 [I 2022-10-11 11:34:59] ...trap/engines/engine.py.229: process: 0.83748 | load: 0.00294 [I 2022-10-11 11:34:59] ...trap/engines/engine.py.230: loss: 0.07891 [I 2022-10-11 11:34:59] ...optimizers/trijoint.py.151: total_norm: 12.758842 [I 2022-10-11 11:35:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 1490/2304 [I 2022-10-11 11:35:07] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:30 [I 2022-10-11 11:35:07] ...trap/engines/engine.py.229: process: 0.85530 | load: 0.00294 [I 2022-10-11 11:35:07] ...trap/engines/engine.py.230: loss: 0.09862 [I 2022-10-11 11:35:07] ...optimizers/trijoint.py.151: total_norm: 17.305981 [I 2022-10-11 11:35:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 1500/2304 [I 2022-10-11 11:35:16] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:17 [I 2022-10-11 11:35:16] ...trap/engines/engine.py.229: process: 0.83107 | load: 0.00294 [I 2022-10-11 11:35:16] ...trap/engines/engine.py.230: loss: 0.15146 [I 2022-10-11 11:35:16] ...optimizers/trijoint.py.151: total_norm: 8.885247 [I 2022-10-11 11:35:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 1510/2304 [I 2022-10-11 11:35:24] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:11:10 [I 2022-10-11 11:35:24] ...trap/engines/engine.py.229: process: 0.82954 | load: 0.00292 [I 2022-10-11 11:35:24] ...trap/engines/engine.py.230: loss: 0.09468 [I 2022-10-11 11:35:24] ...optimizers/trijoint.py.151: total_norm: 7.214899 [I 2022-10-11 11:35:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 1520/2304 [I 2022-10-11 11:35:33] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:03 [I 2022-10-11 11:35:33] ...trap/engines/engine.py.229: process: 0.84428 | load: 0.00293 [I 2022-10-11 11:35:33] ...trap/engines/engine.py.230: loss: 0.13978 [I 2022-10-11 11:35:33] ...optimizers/trijoint.py.151: total_norm: 19.153906 [I 2022-10-11 11:35:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 1530/2304 [I 2022-10-11 11:35:41] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:10:58 [I 2022-10-11 11:35:41] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00294 [I 2022-10-11 11:35:41] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-11 11:35:41] ...optimizers/trijoint.py.151: total_norm: 8.739611 [I 2022-10-11 11:35:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 1540/2304 [I 2022-10-11 11:35:50] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:10:43 [I 2022-10-11 11:35:50] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00293 [I 2022-10-11 11:35:50] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-11 11:35:50] ...optimizers/trijoint.py.151: total_norm: 16.688721 [I 2022-10-11 11:35:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 1550/2304 [I 2022-10-11 11:35:58] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:10:37 [I 2022-10-11 11:35:58] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00293 [I 2022-10-11 11:35:58] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-11 11:35:58] ...optimizers/trijoint.py.151: total_norm: 9.174675 [I 2022-10-11 11:36:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 1560/2304 [I 2022-10-11 11:36:07] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:29 [I 2022-10-11 11:36:07] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00294 [I 2022-10-11 11:36:07] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-11 11:36:07] ...optimizers/trijoint.py.151: total_norm: 16.212002 [I 2022-10-11 11:36:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 1570/2304 [I 2022-10-11 11:36:15] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:18 [I 2022-10-11 11:36:15] ...trap/engines/engine.py.229: process: 0.84540 | load: 0.00294 [I 2022-10-11 11:36:15] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-11 11:36:15] ...optimizers/trijoint.py.151: total_norm: 13.650321 [I 2022-10-11 11:36:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 1580/2304 [I 2022-10-11 11:36:24] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:13 [I 2022-10-11 11:36:24] ...trap/engines/engine.py.229: process: 0.84064 | load: 0.00295 [I 2022-10-11 11:36:24] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-11 11:36:24] ...optimizers/trijoint.py.151: total_norm: 12.644975 [I 2022-10-11 11:36:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 1590/2304 [I 2022-10-11 11:36:32] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:06 [I 2022-10-11 11:36:32] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00293 [I 2022-10-11 11:36:32] ...trap/engines/engine.py.230: loss: 0.13750 [I 2022-10-11 11:36:32] ...optimizers/trijoint.py.151: total_norm: 18.044546 [I 2022-10-11 11:36:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 1600/2304 [I 2022-10-11 11:36:41] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:09:58 [I 2022-10-11 11:36:41] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00293 [I 2022-10-11 11:36:41] ...trap/engines/engine.py.230: loss: 0.09243 [I 2022-10-11 11:36:41] ...optimizers/trijoint.py.151: total_norm: 10.794250 [I 2022-10-11 11:36:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 1610/2304 [I 2022-10-11 11:36:49] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:09:49 [I 2022-10-11 11:36:49] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00293 [I 2022-10-11 11:36:49] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-11 11:36:49] ...optimizers/trijoint.py.151: total_norm: 29.359795 [I 2022-10-11 11:36:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 1620/2304 [I 2022-10-11 11:36:58] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:09:42 [I 2022-10-11 11:36:58] ...trap/engines/engine.py.229: process: 0.86841 | load: 0.00295 [I 2022-10-11 11:36:58] ...trap/engines/engine.py.230: loss: 0.14971 [I 2022-10-11 11:36:58] ...optimizers/trijoint.py.151: total_norm: 16.415203 [I 2022-10-11 11:37:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 1630/2304 [I 2022-10-11 11:37:06] ...trap/engines/engine.py.228: elapsed: 0:23:00 | left: 0:09:34 [I 2022-10-11 11:37:06] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00295 [I 2022-10-11 11:37:06] ...trap/engines/engine.py.230: loss: 0.09209 [I 2022-10-11 11:37:06] ...optimizers/trijoint.py.151: total_norm: 18.460848 [I 2022-10-11 11:37:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 1640/2304 [I 2022-10-11 11:37:15] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:09:23 [I 2022-10-11 11:37:15] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00295 [I 2022-10-11 11:37:15] ...trap/engines/engine.py.230: loss: 0.12442 [I 2022-10-11 11:37:15] ...optimizers/trijoint.py.151: total_norm: 10.820484 [I 2022-10-11 11:37:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 1650/2304 [I 2022-10-11 11:37:23] ...trap/engines/engine.py.228: elapsed: 0:23:17 | left: 0:09:13 [I 2022-10-11 11:37:23] ...trap/engines/engine.py.229: process: 0.85927 | load: 0.00292 [I 2022-10-11 11:37:23] ...trap/engines/engine.py.230: loss: 0.14206 [I 2022-10-11 11:37:23] ...optimizers/trijoint.py.151: total_norm: 14.741250 [I 2022-10-11 11:37:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 1660/2304 [I 2022-10-11 11:37:31] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:02 [I 2022-10-11 11:37:31] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00294 [I 2022-10-11 11:37:31] ...trap/engines/engine.py.230: loss: 0.07864 [I 2022-10-11 11:37:31] ...optimizers/trijoint.py.151: total_norm: 12.438131 [I 2022-10-11 11:37:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 1670/2304 [I 2022-10-11 11:37:40] ...trap/engines/engine.py.228: elapsed: 0:23:34 | left: 0:08:55 [I 2022-10-11 11:37:40] ...trap/engines/engine.py.229: process: 0.83794 | load: 0.00293 [I 2022-10-11 11:37:40] ...trap/engines/engine.py.230: loss: 0.12632 [I 2022-10-11 11:37:40] ...optimizers/trijoint.py.151: total_norm: 10.003773 [I 2022-10-11 11:37:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 1680/2304 [I 2022-10-11 11:37:48] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:08:46 [I 2022-10-11 11:37:48] ...trap/engines/engine.py.229: process: 0.84692 | load: 0.00295 [I 2022-10-11 11:37:48] ...trap/engines/engine.py.230: loss: 0.13273 [I 2022-10-11 11:37:48] ...optimizers/trijoint.py.151: total_norm: 12.436762 [I 2022-10-11 11:37:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 1690/2304 [I 2022-10-11 11:37:57] ...trap/engines/engine.py.228: elapsed: 0:23:51 | left: 0:08:37 [I 2022-10-11 11:37:57] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00294 [I 2022-10-11 11:37:57] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-11 11:37:57] ...optimizers/trijoint.py.151: total_norm: 21.687178 [I 2022-10-11 11:38:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 1700/2304 [I 2022-10-11 11:38:05] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:08:30 [I 2022-10-11 11:38:05] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00294 [I 2022-10-11 11:38:05] ...trap/engines/engine.py.230: loss: 0.14630 [I 2022-10-11 11:38:05] ...optimizers/trijoint.py.151: total_norm: 19.637205 [I 2022-10-11 11:38:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 1710/2304 [I 2022-10-11 11:38:14] ...trap/engines/engine.py.228: elapsed: 0:24:08 | left: 0:08:22 [I 2022-10-11 11:38:14] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00295 [I 2022-10-11 11:38:14] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-11 11:38:14] ...optimizers/trijoint.py.151: total_norm: 11.145863 [I 2022-10-11 11:38:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 1720/2304 [I 2022-10-11 11:38:22] ...trap/engines/engine.py.228: elapsed: 0:24:17 | left: 0:08:15 [I 2022-10-11 11:38:22] ...trap/engines/engine.py.229: process: 0.84659 | load: 0.00294 [I 2022-10-11 11:38:22] ...trap/engines/engine.py.230: loss: 0.11092 [I 2022-10-11 11:38:22] ...optimizers/trijoint.py.151: total_norm: 15.721430 [I 2022-10-11 11:38:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 1730/2304 [I 2022-10-11 11:38:31] ...trap/engines/engine.py.228: elapsed: 0:24:25 | left: 0:08:01 [I 2022-10-11 11:38:31] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00300 [I 2022-10-11 11:38:31] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-11 11:38:31] ...optimizers/trijoint.py.151: total_norm: 15.625672 [I 2022-10-11 11:38:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 1740/2304 [I 2022-10-11 11:38:39] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:07:52 [I 2022-10-11 11:38:39] ...trap/engines/engine.py.229: process: 0.83974 | load: 0.00295 [I 2022-10-11 11:38:39] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-11 11:38:39] ...optimizers/trijoint.py.151: total_norm: 8.451735 [I 2022-10-11 11:38:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 1750/2304 [I 2022-10-11 11:38:47] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:07:45 [I 2022-10-11 11:38:47] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00294 [I 2022-10-11 11:38:47] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-11 11:38:47] ...optimizers/trijoint.py.151: total_norm: 34.579601 [I 2022-10-11 11:38:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1760/2304 [I 2022-10-11 11:38:56] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:07:38 [I 2022-10-11 11:38:56] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00292 [I 2022-10-11 11:38:56] ...trap/engines/engine.py.230: loss: 0.12770 [I 2022-10-11 11:38:56] ...optimizers/trijoint.py.151: total_norm: 12.122322 [I 2022-10-11 11:39:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 1770/2304 [I 2022-10-11 11:39:04] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:07:30 [I 2022-10-11 11:39:04] ...trap/engines/engine.py.229: process: 0.84171 | load: 0.00293 [I 2022-10-11 11:39:04] ...trap/engines/engine.py.230: loss: 0.16133 [I 2022-10-11 11:39:04] ...optimizers/trijoint.py.151: total_norm: 27.982473 [I 2022-10-11 11:39:13] ...trap/engines/engine.py.224: train: epoch 103 | batch 1780/2304 [I 2022-10-11 11:39:13] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:07:23 [I 2022-10-11 11:39:13] ...trap/engines/engine.py.229: process: 0.83415 | load: 0.00294 [I 2022-10-11 11:39:13] ...trap/engines/engine.py.230: loss: 0.09844 [I 2022-10-11 11:39:13] ...optimizers/trijoint.py.151: total_norm: 13.024459 [I 2022-10-11 11:39:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 1790/2304 [I 2022-10-11 11:39:22] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:07:26 [I 2022-10-11 11:39:22] ...trap/engines/engine.py.229: process: 0.86992 | load: 0.00372 [I 2022-10-11 11:39:22] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-11 11:39:22] ...optimizers/trijoint.py.151: total_norm: 26.104975 [I 2022-10-11 11:39:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 1800/2304 [I 2022-10-11 11:39:30] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:20 [I 2022-10-11 11:39:30] ...trap/engines/engine.py.229: process: 0.87033 | load: 0.00372 [I 2022-10-11 11:39:30] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-11 11:39:30] ...optimizers/trijoint.py.151: total_norm: 14.009295 [I 2022-10-11 11:39:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 1810/2304 [I 2022-10-11 11:39:39] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:09 [I 2022-10-11 11:39:39] ...trap/engines/engine.py.229: process: 0.86230 | load: 0.00373 [I 2022-10-11 11:39:39] ...trap/engines/engine.py.230: loss: 0.10165 [I 2022-10-11 11:39:39] ...optimizers/trijoint.py.151: total_norm: 10.701458 [I 2022-10-11 11:39:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 1820/2304 [I 2022-10-11 11:39:48] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:01 [I 2022-10-11 11:39:48] ...trap/engines/engine.py.229: process: 0.86672 | load: 0.00373 [I 2022-10-11 11:39:48] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-11 11:39:48] ...optimizers/trijoint.py.151: total_norm: 28.092373 [I 2022-10-11 11:39:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1830/2304 [I 2022-10-11 11:39:56] ...trap/engines/engine.py.228: elapsed: 0:25:51 | left: 0:06:52 [I 2022-10-11 11:39:56] ...trap/engines/engine.py.229: process: 0.85923 | load: 0.00372 [I 2022-10-11 11:39:56] ...trap/engines/engine.py.230: loss: 0.08276 [I 2022-10-11 11:39:56] ...optimizers/trijoint.py.151: total_norm: 9.641661 [I 2022-10-11 11:40:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 1840/2304 [I 2022-10-11 11:40:05] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:06:44 [I 2022-10-11 11:40:05] ...trap/engines/engine.py.229: process: 0.88248 | load: 0.00372 [I 2022-10-11 11:40:05] ...trap/engines/engine.py.230: loss: 0.10492 [I 2022-10-11 11:40:05] ...optimizers/trijoint.py.151: total_norm: 11.967807 [I 2022-10-11 11:40:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 1850/2304 [I 2022-10-11 11:40:14] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:06:30 [I 2022-10-11 11:40:14] ...trap/engines/engine.py.229: process: 0.83965 | load: 0.00294 [I 2022-10-11 11:40:14] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-11 11:40:14] ...optimizers/trijoint.py.151: total_norm: 23.308558 [I 2022-10-11 11:40:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 1860/2304 [I 2022-10-11 11:40:22] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:14 [I 2022-10-11 11:40:22] ...trap/engines/engine.py.229: process: 0.84758 | load: 0.00294 [I 2022-10-11 11:40:22] ...trap/engines/engine.py.230: loss: 0.15161 [I 2022-10-11 11:40:22] ...optimizers/trijoint.py.151: total_norm: 37.375034 [I 2022-10-11 11:40:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 1870/2304 [I 2022-10-11 11:40:31] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:04 [I 2022-10-11 11:40:31] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00294 [I 2022-10-11 11:40:31] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-11 11:40:31] ...optimizers/trijoint.py.151: total_norm: 17.943520 [I 2022-10-11 11:40:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 1880/2304 [I 2022-10-11 11:40:39] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:05:57 [I 2022-10-11 11:40:39] ...trap/engines/engine.py.229: process: 0.84139 | load: 0.00292 [I 2022-10-11 11:40:39] ...trap/engines/engine.py.230: loss: 0.10082 [I 2022-10-11 11:40:39] ...optimizers/trijoint.py.151: total_norm: 16.814369 [I 2022-10-11 11:40:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 1890/2304 [I 2022-10-11 11:40:48] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:05:49 [I 2022-10-11 11:40:48] ...trap/engines/engine.py.229: process: 0.84472 | load: 0.00293 [I 2022-10-11 11:40:48] ...trap/engines/engine.py.230: loss: 0.08690 [I 2022-10-11 11:40:48] ...optimizers/trijoint.py.151: total_norm: 11.516400 [I 2022-10-11 11:40:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1900/2304 [I 2022-10-11 11:40:56] ...trap/engines/engine.py.228: elapsed: 0:26:50 | left: 0:05:38 [I 2022-10-11 11:40:56] ...trap/engines/engine.py.229: process: 0.83383 | load: 0.00294 [I 2022-10-11 11:40:56] ...trap/engines/engine.py.230: loss: 0.15026 [I 2022-10-11 11:40:56] ...optimizers/trijoint.py.151: total_norm: 20.839420 [I 2022-10-11 11:41:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 1910/2304 [I 2022-10-11 11:41:04] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:05:30 [I 2022-10-11 11:41:04] ...trap/engines/engine.py.229: process: 0.83557 | load: 0.00293 [I 2022-10-11 11:41:04] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-11 11:41:04] ...optimizers/trijoint.py.151: total_norm: 12.349097 [I 2022-10-11 11:41:13] ...trap/engines/engine.py.224: train: epoch 103 | batch 1920/2304 [I 2022-10-11 11:41:13] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:05:22 [I 2022-10-11 11:41:13] ...trap/engines/engine.py.229: process: 0.84972 | load: 0.00293 [I 2022-10-11 11:41:13] ...trap/engines/engine.py.230: loss: 0.10848 [I 2022-10-11 11:41:13] ...optimizers/trijoint.py.151: total_norm: 17.412066 [I 2022-10-11 11:41:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 1930/2304 [I 2022-10-11 11:41:21] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:05:13 [I 2022-10-11 11:41:21] ...trap/engines/engine.py.229: process: 0.83895 | load: 0.00295 [I 2022-10-11 11:41:21] ...trap/engines/engine.py.230: loss: 0.12147 [I 2022-10-11 11:41:21] ...optimizers/trijoint.py.151: total_norm: 12.498040 [I 2022-10-11 11:41:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 1940/2304 [I 2022-10-11 11:41:30] ...trap/engines/engine.py.228: elapsed: 0:27:24 | left: 0:05:10 [I 2022-10-11 11:41:30] ...trap/engines/engine.py.229: process: 0.86932 | load: 0.00371 [I 2022-10-11 11:41:30] ...trap/engines/engine.py.230: loss: 0.09122 [I 2022-10-11 11:41:30] ...optimizers/trijoint.py.151: total_norm: 20.372992 [I 2022-10-11 11:41:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 1950/2304 [I 2022-10-11 11:41:38] ...trap/engines/engine.py.228: elapsed: 0:27:33 | left: 0:05:08 [I 2022-10-11 11:41:38] ...trap/engines/engine.py.229: process: 0.87049 | load: 0.00372 [I 2022-10-11 11:41:38] ...trap/engines/engine.py.230: loss: 0.10965 [I 2022-10-11 11:41:38] ...optimizers/trijoint.py.151: total_norm: 10.099468 [I 2022-10-11 11:41:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 1960/2304 [I 2022-10-11 11:41:47] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:04:57 [I 2022-10-11 11:41:47] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00372 [I 2022-10-11 11:41:47] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-11 11:41:47] ...optimizers/trijoint.py.151: total_norm: 14.638227 [I 2022-10-11 11:41:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1970/2304 [I 2022-10-11 11:41:56] ...trap/engines/engine.py.228: elapsed: 0:27:50 | left: 0:04:44 [I 2022-10-11 11:41:56] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00293 [I 2022-10-11 11:41:56] ...trap/engines/engine.py.230: loss: 0.09437 [I 2022-10-11 11:41:56] ...optimizers/trijoint.py.151: total_norm: 25.610580 [I 2022-10-11 11:42:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 1980/2304 [I 2022-10-11 11:42:04] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:04:32 [I 2022-10-11 11:42:04] ...trap/engines/engine.py.229: process: 0.83272 | load: 0.00294 [I 2022-10-11 11:42:04] ...trap/engines/engine.py.230: loss: 0.10748 [I 2022-10-11 11:42:04] ...optimizers/trijoint.py.151: total_norm: 11.166482 [I 2022-10-11 11:42:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 1990/2304 [I 2022-10-11 11:42:12] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:04:24 [I 2022-10-11 11:42:12] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00294 [I 2022-10-11 11:42:12] ...trap/engines/engine.py.230: loss: 0.13008 [I 2022-10-11 11:42:12] ...optimizers/trijoint.py.151: total_norm: 21.003416 [I 2022-10-11 11:42:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 2000/2304 [I 2022-10-11 11:42:21] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:15 [I 2022-10-11 11:42:21] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00292 [I 2022-10-11 11:42:21] ...trap/engines/engine.py.230: loss: 0.06976 [I 2022-10-11 11:42:21] ...optimizers/trijoint.py.151: total_norm: 19.708931 [I 2022-10-11 11:42:29] ...trap/engines/engine.py.224: train: epoch 103 | batch 2010/2304 [I 2022-10-11 11:42:29] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:07 [I 2022-10-11 11:42:29] ...trap/engines/engine.py.229: process: 0.84017 | load: 0.00296 [I 2022-10-11 11:42:29] ...trap/engines/engine.py.230: loss: 0.14731 [I 2022-10-11 11:42:29] ...optimizers/trijoint.py.151: total_norm: 24.423126 [I 2022-10-11 11:42:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 2020/2304 [I 2022-10-11 11:42:38] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:03:57 [I 2022-10-11 11:42:38] ...trap/engines/engine.py.229: process: 0.84099 | load: 0.00292 [I 2022-10-11 11:42:38] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-11 11:42:38] ...optimizers/trijoint.py.151: total_norm: 43.227814 [I 2022-10-11 11:42:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 2030/2304 [I 2022-10-11 11:42:46] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:03:49 [I 2022-10-11 11:42:46] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00293 [I 2022-10-11 11:42:46] ...trap/engines/engine.py.230: loss: 0.07575 [I 2022-10-11 11:42:46] ...optimizers/trijoint.py.151: total_norm: 12.597960 [I 2022-10-11 11:42:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 2040/2304 [I 2022-10-11 11:42:55] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:03:42 [I 2022-10-11 11:42:55] ...trap/engines/engine.py.229: process: 0.83684 | load: 0.00293 [I 2022-10-11 11:42:55] ...trap/engines/engine.py.230: loss: 0.09675 [I 2022-10-11 11:42:55] ...optimizers/trijoint.py.151: total_norm: 10.418015 [I 2022-10-11 11:43:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 2050/2304 [I 2022-10-11 11:43:03] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:03:33 [I 2022-10-11 11:43:03] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00293 [I 2022-10-11 11:43:03] ...trap/engines/engine.py.230: loss: 0.15941 [I 2022-10-11 11:43:03] ...optimizers/trijoint.py.151: total_norm: 27.902571 [I 2022-10-11 11:43:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 2060/2304 [I 2022-10-11 11:43:11] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:23 [I 2022-10-11 11:43:11] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00292 [I 2022-10-11 11:43:11] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-11 11:43:11] ...optimizers/trijoint.py.151: total_norm: 15.801151 [I 2022-10-11 11:43:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 2070/2304 [I 2022-10-11 11:43:20] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:17 [I 2022-10-11 11:43:20] ...trap/engines/engine.py.229: process: 0.85435 | load: 0.00297 [I 2022-10-11 11:43:20] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-11 11:43:20] ...optimizers/trijoint.py.151: total_norm: 11.191877 [I 2022-10-11 11:43:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 2080/2304 [I 2022-10-11 11:43:28] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:09 [I 2022-10-11 11:43:28] ...trap/engines/engine.py.229: process: 0.85703 | load: 0.00296 [I 2022-10-11 11:43:28] ...trap/engines/engine.py.230: loss: 0.15717 [I 2022-10-11 11:43:28] ...optimizers/trijoint.py.151: total_norm: 14.623601 [I 2022-10-11 11:43:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 2090/2304 [I 2022-10-11 11:43:37] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:02 [I 2022-10-11 11:43:37] ...trap/engines/engine.py.229: process: 0.86197 | load: 0.00294 [I 2022-10-11 11:43:37] ...trap/engines/engine.py.230: loss: 0.15367 [I 2022-10-11 11:43:37] ...optimizers/trijoint.py.151: total_norm: 15.680670 [I 2022-10-11 11:43:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 2100/2304 [I 2022-10-11 11:43:45] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:02:52 [I 2022-10-11 11:43:45] ...trap/engines/engine.py.229: process: 0.83591 | load: 0.00294 [I 2022-10-11 11:43:45] ...trap/engines/engine.py.230: loss: 0.13914 [I 2022-10-11 11:43:45] ...optimizers/trijoint.py.151: total_norm: 9.666983 [I 2022-10-11 11:43:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 2110/2304 [I 2022-10-11 11:43:54] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:02:43 [I 2022-10-11 11:43:54] ...trap/engines/engine.py.229: process: 0.87572 | load: 0.00292 [I 2022-10-11 11:43:54] ...trap/engines/engine.py.230: loss: 0.13654 [I 2022-10-11 11:43:54] ...optimizers/trijoint.py.151: total_norm: 18.924162 [I 2022-10-11 11:44:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 2120/2304 [I 2022-10-11 11:44:02] ...trap/engines/engine.py.228: elapsed: 0:29:56 | left: 0:02:34 [I 2022-10-11 11:44:02] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00293 [I 2022-10-11 11:44:02] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-11 11:44:02] ...optimizers/trijoint.py.151: total_norm: 12.784234 [I 2022-10-11 11:44:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 2130/2304 [I 2022-10-11 11:44:10] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:26 [I 2022-10-11 11:44:10] ...trap/engines/engine.py.229: process: 0.83973 | load: 0.00294 [I 2022-10-11 11:44:10] ...trap/engines/engine.py.230: loss: 0.10507 [I 2022-10-11 11:44:10] ...optimizers/trijoint.py.151: total_norm: 10.630618 [I 2022-10-11 11:44:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 2140/2304 [I 2022-10-11 11:44:19] ...trap/engines/engine.py.228: elapsed: 0:30:13 | left: 0:02:17 [I 2022-10-11 11:44:19] ...trap/engines/engine.py.229: process: 0.83409 | load: 0.00294 [I 2022-10-11 11:44:19] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-11 11:44:19] ...optimizers/trijoint.py.151: total_norm: 14.399703 [I 2022-10-11 11:44:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 2150/2304 [I 2022-10-11 11:44:27] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:09 [I 2022-10-11 11:44:27] ...trap/engines/engine.py.229: process: 0.83345 | load: 0.00292 [I 2022-10-11 11:44:27] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-11 11:44:27] ...optimizers/trijoint.py.151: total_norm: 11.595643 [I 2022-10-11 11:44:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 2160/2304 [I 2022-10-11 11:44:36] ...trap/engines/engine.py.228: elapsed: 0:30:30 | left: 0:02:00 [I 2022-10-11 11:44:36] ...trap/engines/engine.py.229: process: 0.83665 | load: 0.00292 [I 2022-10-11 11:44:36] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-11 11:44:36] ...optimizers/trijoint.py.151: total_norm: 20.513235 [I 2022-10-11 11:44:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 2170/2304 [I 2022-10-11 11:44:44] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:01:52 [I 2022-10-11 11:44:44] ...trap/engines/engine.py.229: process: 0.84698 | load: 0.00294 [I 2022-10-11 11:44:44] ...trap/engines/engine.py.230: loss: 0.09105 [I 2022-10-11 11:44:44] ...optimizers/trijoint.py.151: total_norm: 13.928442 [I 2022-10-11 11:44:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 2180/2304 [I 2022-10-11 11:44:53] ...trap/engines/engine.py.228: elapsed: 0:30:47 | left: 0:01:44 [I 2022-10-11 11:44:53] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00294 [I 2022-10-11 11:44:53] ...trap/engines/engine.py.230: loss: 0.09587 [I 2022-10-11 11:44:53] ...optimizers/trijoint.py.151: total_norm: 12.873980 [I 2022-10-11 11:45:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 2190/2304 [I 2022-10-11 11:45:01] ...trap/engines/engine.py.228: elapsed: 0:30:56 | left: 0:01:37 [I 2022-10-11 11:45:01] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00293 [I 2022-10-11 11:45:01] ...trap/engines/engine.py.230: loss: 0.08911 [I 2022-10-11 11:45:01] ...optimizers/trijoint.py.151: total_norm: 8.037936 [I 2022-10-11 11:45:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 2200/2304 [I 2022-10-11 11:45:10] ...trap/engines/engine.py.228: elapsed: 0:31:04 | left: 0:01:29 [I 2022-10-11 11:45:10] ...trap/engines/engine.py.229: process: 0.84300 | load: 0.00298 [I 2022-10-11 11:45:10] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-11 11:45:10] ...optimizers/trijoint.py.151: total_norm: 8.906669 [I 2022-10-11 11:45:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 2210/2304 [I 2022-10-11 11:45:18] ...trap/engines/engine.py.228: elapsed: 0:31:13 | left: 0:01:20 [I 2022-10-11 11:45:18] ...trap/engines/engine.py.229: process: 0.86300 | load: 0.00301 [I 2022-10-11 11:45:18] ...trap/engines/engine.py.230: loss: 0.12959 [I 2022-10-11 11:45:18] ...optimizers/trijoint.py.151: total_norm: 176.992645 [I 2022-10-11 11:45:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 2220/2304 [I 2022-10-11 11:45:27] ...trap/engines/engine.py.228: elapsed: 0:31:21 | left: 0:01:11 [I 2022-10-11 11:45:27] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00298 [I 2022-10-11 11:45:27] ...trap/engines/engine.py.230: loss: 0.08456 [I 2022-10-11 11:45:27] ...optimizers/trijoint.py.151: total_norm: 19.165215 [I 2022-10-11 11:45:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 2230/2304 [I 2022-10-11 11:45:35] ...trap/engines/engine.py.228: elapsed: 0:31:30 | left: 0:01:02 [I 2022-10-11 11:45:35] ...trap/engines/engine.py.229: process: 0.86618 | load: 0.00301 [I 2022-10-11 11:45:35] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-11 11:45:35] ...optimizers/trijoint.py.151: total_norm: 24.567543 [I 2022-10-11 11:45:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 2240/2304 [I 2022-10-11 11:45:44] ...trap/engines/engine.py.228: elapsed: 0:31:38 | left: 0:00:54 [I 2022-10-11 11:45:44] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00302 [I 2022-10-11 11:45:44] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-11 11:45:44] ...optimizers/trijoint.py.151: total_norm: 10.679874 [I 2022-10-11 11:45:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 2250/2304 [I 2022-10-11 11:45:52] ...trap/engines/engine.py.228: elapsed: 0:31:47 | left: 0:00:45 [I 2022-10-11 11:45:52] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00305 [I 2022-10-11 11:45:52] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-11 11:45:52] ...optimizers/trijoint.py.151: total_norm: 12.207201 [I 2022-10-11 11:46:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 2260/2304 [I 2022-10-11 11:46:01] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:00:37 [I 2022-10-11 11:46:01] ...trap/engines/engine.py.229: process: 0.83562 | load: 0.00293 [I 2022-10-11 11:46:01] ...trap/engines/engine.py.230: loss: 0.18186 [I 2022-10-11 11:46:01] ...optimizers/trijoint.py.151: total_norm: 21.269445 [I 2022-10-11 11:46:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 2270/2304 [I 2022-10-11 11:46:09] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:28 [I 2022-10-11 11:46:09] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00295 [I 2022-10-11 11:46:09] ...trap/engines/engine.py.230: loss: 0.11904 [I 2022-10-11 11:46:09] ...optimizers/trijoint.py.151: total_norm: 14.202341 [I 2022-10-11 11:46:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 2280/2304 [I 2022-10-11 11:46:18] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:20 [I 2022-10-11 11:46:18] ...trap/engines/engine.py.229: process: 0.83960 | load: 0.00293 [I 2022-10-11 11:46:18] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-11 11:46:18] ...optimizers/trijoint.py.151: total_norm: 30.168697 [I 2022-10-11 11:46:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 2290/2304 [I 2022-10-11 11:46:26] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:11 [I 2022-10-11 11:46:26] ...trap/engines/engine.py.229: process: 0.84367 | load: 0.00294 [I 2022-10-11 11:46:26] ...trap/engines/engine.py.230: loss: 0.11675 [I 2022-10-11 11:46:26] ...optimizers/trijoint.py.151: total_norm: 9.978572 [I 2022-10-11 11:46:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 2300/2304 [I 2022-10-11 11:46:34] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:03 [I 2022-10-11 11:46:34] ...trap/engines/engine.py.229: process: 0.81837 | load: 0.00012 [I 2022-10-11 11:46:34] ...trap/engines/engine.py.230: loss: 0.06919 [I 2022-10-11 11:46:34] ...optimizers/trijoint.py.151: total_norm: 8.569843 [I 2022-10-11 11:46:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 2304/2304 [I 2022-10-11 11:46:38] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:00 [I 2022-10-11 11:46:38] ...trap/engines/engine.py.229: process: 0.83477 | load: 0.00012 [I 2022-10-11 11:46:38] ...trap/engines/engine.py.230: loss: 0.18366 [I 2022-10-11 11:46:38] ...optimizers/trijoint.py.151: total_norm: 22.795813 [S 2022-10-11 11:46:39] ...trap/engines/engine.py.129: train_epoch.epoch: 103 [S 2022-10-11 11:46:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.12267362515476417 [I 2022-10-11 11:46:39] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 11:46:41] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 103 [I 2022-10-11 11:46:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 11:46:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 11:46:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 11:46:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 11:46:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 11:46:46] ...trap/engines/engine.py.321: eval: epoch 103 | batch 0/510 [I 2022-10-11 11:46:47] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:50:45 [I 2022-10-11 11:46:47] ...trap/engines/engine.py.326: process: 5.97229 | load: 4.88564 [I 2022-10-11 11:46:51] ...trap/engines/engine.py.321: eval: epoch 103 | batch 10/510 [I 2022-10-11 11:46:51] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:47 [I 2022-10-11 11:46:51] ...trap/engines/engine.py.326: process: 0.53704 | load: 0.00017 [I 2022-10-11 11:46:57] ...trap/engines/engine.py.321: eval: epoch 103 | batch 20/510 [I 2022-10-11 11:46:57] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:35 [I 2022-10-11 11:46:57] ...trap/engines/engine.py.326: process: 0.68561 | load: 0.00018 [I 2022-10-11 11:47:09] ...trap/engines/engine.py.321: eval: epoch 103 | batch 30/510 [I 2022-10-11 11:47:09] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:06:31 [I 2022-10-11 11:47:09] ...trap/engines/engine.py.326: process: 0.53306 | load: 0.00017 [I 2022-10-11 11:47:20] ...trap/engines/engine.py.321: eval: epoch 103 | batch 40/510 [I 2022-10-11 11:47:20] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:06:21 [I 2022-10-11 11:47:20] ...trap/engines/engine.py.326: process: 0.75234 | load: 0.00018 [I 2022-10-11 11:47:26] ...trap/engines/engine.py.321: eval: epoch 103 | batch 50/510 [I 2022-10-11 11:47:26] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:04:20 [I 2022-10-11 11:47:26] ...trap/engines/engine.py.326: process: 0.40184 | load: 0.00018 [I 2022-10-11 11:47:31] ...trap/engines/engine.py.321: eval: epoch 103 | batch 60/510 [I 2022-10-11 11:47:31] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:55 [I 2022-10-11 11:47:31] ...trap/engines/engine.py.326: process: 0.53396 | load: 0.00019 [I 2022-10-11 11:47:36] ...trap/engines/engine.py.321: eval: epoch 103 | batch 70/510 [I 2022-10-11 11:47:36] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:04:06 [I 2022-10-11 11:47:36] ...trap/engines/engine.py.326: process: 0.55119 | load: 0.00018 [I 2022-10-11 11:47:42] ...trap/engines/engine.py.321: eval: epoch 103 | batch 80/510 [I 2022-10-11 11:47:42] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:03:53 [I 2022-10-11 11:47:42] ...trap/engines/engine.py.326: process: 0.42394 | load: 0.00019 [I 2022-10-11 11:47:47] ...trap/engines/engine.py.321: eval: epoch 103 | batch 90/510 [I 2022-10-11 11:47:47] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:03:35 [I 2022-10-11 11:47:47] ...trap/engines/engine.py.326: process: 0.49442 | load: 0.00018 [I 2022-10-11 11:47:52] ...trap/engines/engine.py.321: eval: epoch 103 | batch 100/510 [I 2022-10-11 11:47:52] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:03:16 [I 2022-10-11 11:47:52] ...trap/engines/engine.py.326: process: 0.30412 | load: 0.00017 [I 2022-10-11 11:47:55] ...trap/engines/engine.py.321: eval: epoch 103 | batch 110/510 [I 2022-10-11 11:47:55] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:02:12 [I 2022-10-11 11:47:55] ...trap/engines/engine.py.326: process: 0.31567 | load: 0.00019 [I 2022-10-11 11:47:59] ...trap/engines/engine.py.321: eval: epoch 103 | batch 120/510 [I 2022-10-11 11:47:59] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:02:02 [I 2022-10-11 11:47:59] ...trap/engines/engine.py.326: process: 0.30795 | load: 0.00017 [I 2022-10-11 11:48:02] ...trap/engines/engine.py.321: eval: epoch 103 | batch 130/510 [I 2022-10-11 11:48:02] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:58 [I 2022-10-11 11:48:02] ...trap/engines/engine.py.326: process: 0.31886 | load: 0.00020 [I 2022-10-11 11:48:05] ...trap/engines/engine.py.321: eval: epoch 103 | batch 140/510 [I 2022-10-11 11:48:05] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:55 [I 2022-10-11 11:48:05] ...trap/engines/engine.py.326: process: 0.31894 | load: 0.00018 [I 2022-10-11 11:48:08] ...trap/engines/engine.py.321: eval: epoch 103 | batch 150/510 [I 2022-10-11 11:48:08] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:53 [I 2022-10-11 11:48:08] ...trap/engines/engine.py.326: process: 0.31598 | load: 0.00019 [I 2022-10-11 11:48:11] ...trap/engines/engine.py.321: eval: epoch 103 | batch 160/510 [I 2022-10-11 11:48:11] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:49 [I 2022-10-11 11:48:11] ...trap/engines/engine.py.326: process: 0.31260 | load: 0.00019 [I 2022-10-11 11:48:14] ...trap/engines/engine.py.321: eval: epoch 103 | batch 170/510 [I 2022-10-11 11:48:14] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:46 [I 2022-10-11 11:48:14] ...trap/engines/engine.py.326: process: 0.31610 | load: 0.00019 [I 2022-10-11 11:48:17] ...trap/engines/engine.py.321: eval: epoch 103 | batch 180/510 [I 2022-10-11 11:48:17] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:43 [I 2022-10-11 11:48:17] ...trap/engines/engine.py.326: process: 0.31479 | load: 0.00019 [I 2022-10-11 11:48:20] ...trap/engines/engine.py.321: eval: epoch 103 | batch 190/510 [I 2022-10-11 11:48:20] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:39 [I 2022-10-11 11:48:20] ...trap/engines/engine.py.326: process: 0.31270 | load: 0.00019 [I 2022-10-11 11:48:24] ...trap/engines/engine.py.321: eval: epoch 103 | batch 200/510 [I 2022-10-11 11:48:24] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:37 [I 2022-10-11 11:48:24] ...trap/engines/engine.py.326: process: 0.31663 | load: 0.00019 [I 2022-10-11 11:48:27] ...trap/engines/engine.py.321: eval: epoch 103 | batch 210/510 [I 2022-10-11 11:48:27] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:34 [I 2022-10-11 11:48:27] ...trap/engines/engine.py.326: process: 0.32294 | load: 0.00019 [I 2022-10-11 11:48:30] ...trap/engines/engine.py.321: eval: epoch 103 | batch 220/510 [I 2022-10-11 11:48:30] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:31 [I 2022-10-11 11:48:30] ...trap/engines/engine.py.326: process: 0.32043 | load: 0.00019 [I 2022-10-11 11:48:33] ...trap/engines/engine.py.321: eval: epoch 103 | batch 230/510 [I 2022-10-11 11:48:33] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:28 [I 2022-10-11 11:48:33] ...trap/engines/engine.py.326: process: 0.31374 | load: 0.00020 [I 2022-10-11 11:48:36] ...trap/engines/engine.py.321: eval: epoch 103 | batch 240/510 [I 2022-10-11 11:48:36] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:24 [I 2022-10-11 11:48:36] ...trap/engines/engine.py.326: process: 0.30635 | load: 0.00020 [I 2022-10-11 11:48:39] ...trap/engines/engine.py.321: eval: epoch 103 | batch 250/510 [I 2022-10-11 11:48:39] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:22 [I 2022-10-11 11:48:39] ...trap/engines/engine.py.326: process: 0.32009 | load: 0.00019 [I 2022-10-11 11:48:43] ...trap/engines/engine.py.321: eval: epoch 103 | batch 260/510 [I 2022-10-11 11:48:43] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:19 [I 2022-10-11 11:48:43] ...trap/engines/engine.py.326: process: 0.31901 | load: 0.00019 [I 2022-10-11 11:48:46] ...trap/engines/engine.py.321: eval: epoch 103 | batch 270/510 [I 2022-10-11 11:48:46] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:15 [I 2022-10-11 11:48:46] ...trap/engines/engine.py.326: process: 0.31359 | load: 0.00019 [I 2022-10-11 11:48:49] ...trap/engines/engine.py.321: eval: epoch 103 | batch 280/510 [I 2022-10-11 11:48:49] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:12 [I 2022-10-11 11:48:49] ...trap/engines/engine.py.326: process: 0.31460 | load: 0.00019 [I 2022-10-11 11:48:52] ...trap/engines/engine.py.321: eval: epoch 103 | batch 290/510 [I 2022-10-11 11:48:52] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:08 [I 2022-10-11 11:48:52] ...trap/engines/engine.py.326: process: 0.31044 | load: 0.00020 [I 2022-10-11 11:48:55] ...trap/engines/engine.py.321: eval: epoch 103 | batch 300/510 [I 2022-10-11 11:48:55] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:05 [I 2022-10-11 11:48:55] ...trap/engines/engine.py.326: process: 0.31791 | load: 0.00020 [I 2022-10-11 11:48:58] ...trap/engines/engine.py.321: eval: epoch 103 | batch 310/510 [I 2022-10-11 11:48:58] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:01:03 [I 2022-10-11 11:48:58] ...trap/engines/engine.py.326: process: 0.31689 | load: 0.00019 [I 2022-10-11 11:49:01] ...trap/engines/engine.py.321: eval: epoch 103 | batch 320/510 [I 2022-10-11 11:49:01] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:59 [I 2022-10-11 11:49:01] ...trap/engines/engine.py.326: process: 0.31852 | load: 0.00019 [I 2022-10-11 11:49:05] ...trap/engines/engine.py.321: eval: epoch 103 | batch 330/510 [I 2022-10-11 11:49:05] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:56 [I 2022-10-11 11:49:05] ...trap/engines/engine.py.326: process: 0.31718 | load: 0.00020 [I 2022-10-11 11:49:08] ...trap/engines/engine.py.321: eval: epoch 103 | batch 340/510 [I 2022-10-11 11:49:08] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:53 [I 2022-10-11 11:49:08] ...trap/engines/engine.py.326: process: 0.32038 | load: 0.00019 [I 2022-10-11 11:49:11] ...trap/engines/engine.py.321: eval: epoch 103 | batch 350/510 [I 2022-10-11 11:49:11] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:50 [I 2022-10-11 11:49:11] ...trap/engines/engine.py.326: process: 0.32028 | load: 0.00019 [I 2022-10-11 11:49:14] ...trap/engines/engine.py.321: eval: epoch 103 | batch 360/510 [I 2022-10-11 11:49:14] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:47 [I 2022-10-11 11:49:14] ...trap/engines/engine.py.326: process: 0.31241 | load: 0.00019 [I 2022-10-11 11:49:17] ...trap/engines/engine.py.321: eval: epoch 103 | batch 370/510 [I 2022-10-11 11:49:17] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:44 [I 2022-10-11 11:49:17] ...trap/engines/engine.py.326: process: 0.31877 | load: 0.00019 [I 2022-10-11 11:49:20] ...trap/engines/engine.py.321: eval: epoch 103 | batch 380/510 [I 2022-10-11 11:49:20] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:41 [I 2022-10-11 11:49:20] ...trap/engines/engine.py.326: process: 0.31193 | load: 0.00020 [I 2022-10-11 11:49:24] ...trap/engines/engine.py.321: eval: epoch 103 | batch 390/510 [I 2022-10-11 11:49:24] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:37 [I 2022-10-11 11:49:24] ...trap/engines/engine.py.326: process: 0.30883 | load: 0.00019 [I 2022-10-11 11:49:27] ...trap/engines/engine.py.321: eval: epoch 103 | batch 400/510 [I 2022-10-11 11:49:27] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:34 [I 2022-10-11 11:49:27] ...trap/engines/engine.py.326: process: 0.31409 | load: 0.00019 [I 2022-10-11 11:49:30] ...trap/engines/engine.py.321: eval: epoch 103 | batch 410/510 [I 2022-10-11 11:49:30] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:31 [I 2022-10-11 11:49:30] ...trap/engines/engine.py.326: process: 0.31628 | load: 0.00020 [I 2022-10-11 11:49:33] ...trap/engines/engine.py.321: eval: epoch 103 | batch 420/510 [I 2022-10-11 11:49:33] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:28 [I 2022-10-11 11:49:33] ...trap/engines/engine.py.326: process: 0.31482 | load: 0.00020 [I 2022-10-11 11:49:36] ...trap/engines/engine.py.321: eval: epoch 103 | batch 430/510 [I 2022-10-11 11:49:36] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:25 [I 2022-10-11 11:49:36] ...trap/engines/engine.py.326: process: 0.30805 | load: 0.00019 [I 2022-10-11 11:49:39] ...trap/engines/engine.py.321: eval: epoch 103 | batch 440/510 [I 2022-10-11 11:49:39] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:22 [I 2022-10-11 11:49:39] ...trap/engines/engine.py.326: process: 0.31663 | load: 0.00019 [I 2022-10-11 11:49:42] ...trap/engines/engine.py.321: eval: epoch 103 | batch 450/510 [I 2022-10-11 11:49:42] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:18 [I 2022-10-11 11:49:42] ...trap/engines/engine.py.326: process: 0.31104 | load: 0.00019 [I 2022-10-11 11:49:46] ...trap/engines/engine.py.321: eval: epoch 103 | batch 460/510 [I 2022-10-11 11:49:46] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:15 [I 2022-10-11 11:49:46] ...trap/engines/engine.py.326: process: 0.30651 | load: 0.00021 [I 2022-10-11 11:49:49] ...trap/engines/engine.py.321: eval: epoch 103 | batch 470/510 [I 2022-10-11 11:49:49] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:12 [I 2022-10-11 11:49:49] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00020 [I 2022-10-11 11:49:52] ...trap/engines/engine.py.321: eval: epoch 103 | batch 480/510 [I 2022-10-11 11:49:52] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:09 [I 2022-10-11 11:49:52] ...trap/engines/engine.py.326: process: 0.31843 | load: 0.00019 [I 2022-10-11 11:49:55] ...trap/engines/engine.py.321: eval: epoch 103 | batch 490/510 [I 2022-10-11 11:49:55] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:06 [I 2022-10-11 11:49:55] ...trap/engines/engine.py.326: process: 0.31377 | load: 0.00020 [I 2022-10-11 11:49:58] ...trap/engines/engine.py.321: eval: epoch 103 | batch 500/510 [I 2022-10-11 11:49:58] ...trap/engines/engine.py.325: elapsed: 0:03:17 | left: 0:00:03 [I 2022-10-11 11:49:58] ...trap/engines/engine.py.326: process: 0.31660 | load: 0.00020 [I 2022-10-11 11:50:01] ...trap/engines/engine.py.321: eval: epoch 103 | batch 510/510 [I 2022-10-11 11:50:01] ...trap/engines/engine.py.325: elapsed: 0:03:20 | left: 0:00:00 [I 2022-10-11 11:50:01] ...trap/engines/engine.py.326: process: 0.30945 | load: 0.00011 [S 2022-10-11 11:50:02] ...trap/engines/engine.py.132: eval_epoch.epoch: 103 [I 2022-10-11 11:50:02] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.748700 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910300 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937500 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.743500 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910500 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937300 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.020015 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013365 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011307 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.019976 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011985 [S 2022-10-11 11:50:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009155 [I 2022-10-11 11:50:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 11:50:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 11:50:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 11:50:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 11:50:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 11:50:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 11:50:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 11:50:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 11:50:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 104 [I 2022-10-11 11:50:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 11:50:57] ...trap/engines/engine.py.224: train: epoch 104 | batch 0/2304 [I 2022-10-11 11:50:57] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:56:14 [I 2022-10-11 11:50:57] ...trap/engines/engine.py.229: process: 3.02703 | load: 2.12099 [I 2022-10-11 11:50:57] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-11 11:50:57] ...optimizers/trijoint.py.151: total_norm: 20.186733 [I 2022-10-11 11:51:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 10/2304 [I 2022-10-11 11:51:06] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:58 [I 2022-10-11 11:51:06] ...trap/engines/engine.py.229: process: 0.81784 | load: 0.00423 [I 2022-10-11 11:51:06] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-11 11:51:06] ...optimizers/trijoint.py.151: total_norm: 10.584164 [I 2022-10-11 11:51:14] ...trap/engines/engine.py.224: train: epoch 104 | batch 20/2304 [I 2022-10-11 11:51:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:32:39 [I 2022-10-11 11:51:14] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00556 [I 2022-10-11 11:51:14] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-11 11:51:14] ...optimizers/trijoint.py.151: total_norm: 14.095917 [I 2022-10-11 11:51:23] ...trap/engines/engine.py.224: train: epoch 104 | batch 30/2304 [I 2022-10-11 11:51:23] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:52 [I 2022-10-11 11:51:23] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00438 [I 2022-10-11 11:51:23] ...trap/engines/engine.py.230: loss: 0.08942 [I 2022-10-11 11:51:23] ...optimizers/trijoint.py.151: total_norm: 18.120798 [I 2022-10-11 11:51:31] ...trap/engines/engine.py.224: train: epoch 104 | batch 40/2304 [I 2022-10-11 11:51:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:45 [I 2022-10-11 11:51:31] ...trap/engines/engine.py.229: process: 0.84024 | load: 0.00417 [I 2022-10-11 11:51:31] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-11 11:51:31] ...optimizers/trijoint.py.151: total_norm: 18.628481 [I 2022-10-11 11:51:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 50/2304 [I 2022-10-11 11:51:39] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:38 [I 2022-10-11 11:51:39] ...trap/engines/engine.py.229: process: 0.83747 | load: 0.00340 [I 2022-10-11 11:51:39] ...trap/engines/engine.py.230: loss: 0.11745 [I 2022-10-11 11:51:39] ...optimizers/trijoint.py.151: total_norm: 28.431467 [I 2022-10-11 11:51:48] ...trap/engines/engine.py.224: train: epoch 104 | batch 60/2304 [I 2022-10-11 11:51:48] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:30 [I 2022-10-11 11:51:48] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00323 [I 2022-10-11 11:51:48] ...trap/engines/engine.py.230: loss: 0.10091 [I 2022-10-11 11:51:48] ...optimizers/trijoint.py.151: total_norm: 20.891268 [I 2022-10-11 11:51:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 70/2304 [I 2022-10-11 11:51:56] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:31:17 [I 2022-10-11 11:51:56] ...trap/engines/engine.py.229: process: 0.83010 | load: 0.00338 [I 2022-10-11 11:51:56] ...trap/engines/engine.py.230: loss: 0.08861 [I 2022-10-11 11:51:56] ...optimizers/trijoint.py.151: total_norm: 9.943996 [I 2022-10-11 11:52:05] ...trap/engines/engine.py.224: train: epoch 104 | batch 80/2304 [I 2022-10-11 11:52:05] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:14 [I 2022-10-11 11:52:05] ...trap/engines/engine.py.229: process: 0.83204 | load: 0.00305 [I 2022-10-11 11:52:05] ...trap/engines/engine.py.230: loss: 0.13970 [I 2022-10-11 11:52:05] ...optimizers/trijoint.py.151: total_norm: 23.808462 [I 2022-10-11 11:52:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 90/2304 [I 2022-10-11 11:52:13] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:30:59 [I 2022-10-11 11:52:13] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00312 [I 2022-10-11 11:52:13] ...trap/engines/engine.py.230: loss: 0.08635 [I 2022-10-11 11:52:13] ...optimizers/trijoint.py.151: total_norm: 11.880676 [I 2022-10-11 11:52:22] ...trap/engines/engine.py.224: train: epoch 104 | batch 100/2304 [I 2022-10-11 11:52:22] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:56 [I 2022-10-11 11:52:22] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00331 [I 2022-10-11 11:52:22] ...trap/engines/engine.py.230: loss: 0.10135 [I 2022-10-11 11:52:22] ...optimizers/trijoint.py.151: total_norm: 18.997761 [I 2022-10-11 11:52:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 110/2304 [I 2022-10-11 11:52:30] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:31:01 [I 2022-10-11 11:52:30] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00304 [I 2022-10-11 11:52:30] ...trap/engines/engine.py.230: loss: 0.17355 [I 2022-10-11 11:52:30] ...optimizers/trijoint.py.151: total_norm: 24.544460 [I 2022-10-11 11:52:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 120/2304 [I 2022-10-11 11:52:39] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:31:05 [I 2022-10-11 11:52:39] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00313 [I 2022-10-11 11:52:39] ...trap/engines/engine.py.230: loss: 0.09704 [I 2022-10-11 11:52:39] ...optimizers/trijoint.py.151: total_norm: 22.489334 [I 2022-10-11 11:52:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 130/2304 [I 2022-10-11 11:52:47] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:30:46 [I 2022-10-11 11:52:47] ...trap/engines/engine.py.229: process: 0.84541 | load: 0.00290 [I 2022-10-11 11:52:47] ...trap/engines/engine.py.230: loss: 0.12295 [I 2022-10-11 11:52:47] ...optimizers/trijoint.py.151: total_norm: 26.830730 [I 2022-10-11 11:52:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 140/2304 [I 2022-10-11 11:52:56] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:28 [I 2022-10-11 11:52:56] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00323 [I 2022-10-11 11:52:56] ...trap/engines/engine.py.230: loss: 0.18953 [I 2022-10-11 11:52:56] ...optimizers/trijoint.py.151: total_norm: 23.370705 [I 2022-10-11 11:53:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 150/2304 [I 2022-10-11 11:53:04] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:30:31 [I 2022-10-11 11:53:04] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00302 [I 2022-10-11 11:53:04] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-11 11:53:04] ...optimizers/trijoint.py.151: total_norm: 29.425867 [I 2022-10-11 11:53:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 160/2304 [I 2022-10-11 11:53:13] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:28 [I 2022-10-11 11:53:13] ...trap/engines/engine.py.229: process: 0.85319 | load: 0.00299 [I 2022-10-11 11:53:13] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-11 11:53:13] ...optimizers/trijoint.py.151: total_norm: 12.276441 [I 2022-10-11 11:53:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 170/2304 [I 2022-10-11 11:53:21] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:30:48 [I 2022-10-11 11:53:21] ...trap/engines/engine.py.229: process: 0.87533 | load: 0.00425 [I 2022-10-11 11:53:21] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-11 11:53:21] ...optimizers/trijoint.py.151: total_norm: 19.404076 [I 2022-10-11 11:53:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 180/2304 [I 2022-10-11 11:53:30] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:30:46 [I 2022-10-11 11:53:30] ...trap/engines/engine.py.229: process: 0.87450 | load: 0.00392 [I 2022-10-11 11:53:30] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-11 11:53:30] ...optimizers/trijoint.py.151: total_norm: 24.416607 [I 2022-10-11 11:53:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 190/2304 [I 2022-10-11 11:53:39] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:42 [I 2022-10-11 11:53:39] ...trap/engines/engine.py.229: process: 0.87335 | load: 0.00383 [I 2022-10-11 11:53:39] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-11 11:53:39] ...optimizers/trijoint.py.151: total_norm: 18.891066 [I 2022-10-11 11:53:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 200/2304 [I 2022-10-11 11:53:47] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:49 [I 2022-10-11 11:53:47] ...trap/engines/engine.py.229: process: 0.83462 | load: 0.00316 [I 2022-10-11 11:53:47] ...trap/engines/engine.py.230: loss: 0.09448 [I 2022-10-11 11:53:47] ...optimizers/trijoint.py.151: total_norm: 14.498718 [I 2022-10-11 11:53:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 210/2304 [I 2022-10-11 11:53:56] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:37 [I 2022-10-11 11:53:56] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00293 [I 2022-10-11 11:53:56] ...trap/engines/engine.py.230: loss: 0.09129 [I 2022-10-11 11:53:56] ...optimizers/trijoint.py.151: total_norm: 10.638805 [I 2022-10-11 11:54:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 220/2304 [I 2022-10-11 11:54:04] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:07 [I 2022-10-11 11:54:04] ...trap/engines/engine.py.229: process: 0.83318 | load: 0.00308 [I 2022-10-11 11:54:04] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-11 11:54:04] ...optimizers/trijoint.py.151: total_norm: 15.140344 [I 2022-10-11 11:54:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 230/2304 [I 2022-10-11 11:54:13] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:06 [I 2022-10-11 11:54:13] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00304 [I 2022-10-11 11:54:13] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-11 11:54:13] ...optimizers/trijoint.py.151: total_norm: 45.737099 [I 2022-10-11 11:54:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 240/2304 [I 2022-10-11 11:54:21] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:28:54 [I 2022-10-11 11:54:21] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00315 [I 2022-10-11 11:54:21] ...trap/engines/engine.py.230: loss: 0.06786 [I 2022-10-11 11:54:21] ...optimizers/trijoint.py.151: total_norm: 14.983532 [I 2022-10-11 11:54:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 250/2304 [I 2022-10-11 11:54:29] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:28:39 [I 2022-10-11 11:54:29] ...trap/engines/engine.py.229: process: 0.83678 | load: 0.00307 [I 2022-10-11 11:54:29] ...trap/engines/engine.py.230: loss: 0.10888 [I 2022-10-11 11:54:29] ...optimizers/trijoint.py.151: total_norm: 14.393336 [I 2022-10-11 11:54:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 260/2304 [I 2022-10-11 11:54:38] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:28:31 [I 2022-10-11 11:54:38] ...trap/engines/engine.py.229: process: 0.84503 | load: 0.00292 [I 2022-10-11 11:54:38] ...trap/engines/engine.py.230: loss: 0.14397 [I 2022-10-11 11:54:38] ...optimizers/trijoint.py.151: total_norm: 10.695332 [I 2022-10-11 11:54:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 270/2304 [I 2022-10-11 11:54:46] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:28:26 [I 2022-10-11 11:54:46] ...trap/engines/engine.py.229: process: 0.82656 | load: 0.00312 [I 2022-10-11 11:54:46] ...trap/engines/engine.py.230: loss: 0.09335 [I 2022-10-11 11:54:46] ...optimizers/trijoint.py.151: total_norm: 11.313917 [I 2022-10-11 11:54:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 280/2304 [I 2022-10-11 11:54:55] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:28:39 [I 2022-10-11 11:54:55] ...trap/engines/engine.py.229: process: 0.86005 | load: 0.00371 [I 2022-10-11 11:54:55] ...trap/engines/engine.py.230: loss: 0.11687 [I 2022-10-11 11:54:55] ...optimizers/trijoint.py.151: total_norm: 20.418388 [I 2022-10-11 11:55:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 290/2304 [I 2022-10-11 11:55:03] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:29:15 [I 2022-10-11 11:55:03] ...trap/engines/engine.py.229: process: 0.87918 | load: 0.00373 [I 2022-10-11 11:55:03] ...trap/engines/engine.py.230: loss: 0.07235 [I 2022-10-11 11:55:03] ...optimizers/trijoint.py.151: total_norm: 12.944901 [I 2022-10-11 11:55:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 300/2304 [I 2022-10-11 11:55:12] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:29:03 [I 2022-10-11 11:55:12] ...trap/engines/engine.py.229: process: 0.86354 | load: 0.00383 [I 2022-10-11 11:55:12] ...trap/engines/engine.py.230: loss: 0.10605 [I 2022-10-11 11:55:12] ...optimizers/trijoint.py.151: total_norm: 28.086121 [I 2022-10-11 11:55:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 310/2304 [I 2022-10-11 11:55:21] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:53 [I 2022-10-11 11:55:21] ...trap/engines/engine.py.229: process: 0.87530 | load: 0.00371 [I 2022-10-11 11:55:21] ...trap/engines/engine.py.230: loss: 0.10362 [I 2022-10-11 11:55:21] ...optimizers/trijoint.py.151: total_norm: 24.598385 [I 2022-10-11 11:55:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 320/2304 [I 2022-10-11 11:55:29] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:51 [I 2022-10-11 11:55:29] ...trap/engines/engine.py.229: process: 0.87966 | load: 0.00371 [I 2022-10-11 11:55:29] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-11 11:55:29] ...optimizers/trijoint.py.151: total_norm: 32.863052 [I 2022-10-11 11:55:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 330/2304 [I 2022-10-11 11:55:38] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:33 [I 2022-10-11 11:55:38] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00374 [I 2022-10-11 11:55:38] ...trap/engines/engine.py.230: loss: 0.22117 [I 2022-10-11 11:55:38] ...optimizers/trijoint.py.151: total_norm: 56.277435 [I 2022-10-11 11:55:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 340/2304 [I 2022-10-11 11:55:47] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:28:38 [I 2022-10-11 11:55:47] ...trap/engines/engine.py.229: process: 0.87659 | load: 0.00371 [I 2022-10-11 11:55:47] ...trap/engines/engine.py.230: loss: 0.08622 [I 2022-10-11 11:55:47] ...optimizers/trijoint.py.151: total_norm: 8.603425 [I 2022-10-11 11:55:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 350/2304 [I 2022-10-11 11:55:56] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:28:20 [I 2022-10-11 11:55:56] ...trap/engines/engine.py.229: process: 0.85990 | load: 0.00371 [I 2022-10-11 11:55:56] ...trap/engines/engine.py.230: loss: 0.09542 [I 2022-10-11 11:55:56] ...optimizers/trijoint.py.151: total_norm: 10.302514 [I 2022-10-11 11:56:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 360/2304 [I 2022-10-11 11:56:04] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:13 [I 2022-10-11 11:56:04] ...trap/engines/engine.py.229: process: 0.83882 | load: 0.00294 [I 2022-10-11 11:56:04] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-11 11:56:04] ...optimizers/trijoint.py.151: total_norm: 10.078820 [I 2022-10-11 11:56:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 370/2304 [I 2022-10-11 11:56:12] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:06 [I 2022-10-11 11:56:12] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00294 [I 2022-10-11 11:56:12] ...trap/engines/engine.py.230: loss: 0.12194 [I 2022-10-11 11:56:12] ...optimizers/trijoint.py.151: total_norm: 26.267525 [I 2022-10-11 11:56:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 380/2304 [I 2022-10-11 11:56:21] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:26:57 [I 2022-10-11 11:56:21] ...trap/engines/engine.py.229: process: 0.84113 | load: 0.00298 [I 2022-10-11 11:56:21] ...trap/engines/engine.py.230: loss: 0.19915 [I 2022-10-11 11:56:21] ...optimizers/trijoint.py.151: total_norm: 29.815966 [I 2022-10-11 11:56:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 390/2304 [I 2022-10-11 11:56:29] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:26:41 [I 2022-10-11 11:56:29] ...trap/engines/engine.py.229: process: 0.83580 | load: 0.00303 [I 2022-10-11 11:56:29] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-11 11:56:29] ...optimizers/trijoint.py.151: total_norm: 13.172194 [I 2022-10-11 11:56:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 400/2304 [I 2022-10-11 11:56:38] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:26:44 [I 2022-10-11 11:56:38] ...trap/engines/engine.py.229: process: 0.83238 | load: 0.00293 [I 2022-10-11 11:56:38] ...trap/engines/engine.py.230: loss: 0.09006 [I 2022-10-11 11:56:38] ...optimizers/trijoint.py.151: total_norm: 13.342468 [I 2022-10-11 11:56:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 410/2304 [I 2022-10-11 11:56:46] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:26:24 [I 2022-10-11 11:56:46] ...trap/engines/engine.py.229: process: 0.83845 | load: 0.00292 [I 2022-10-11 11:56:46] ...trap/engines/engine.py.230: loss: 0.17026 [I 2022-10-11 11:56:46] ...optimizers/trijoint.py.151: total_norm: 21.011215 [I 2022-10-11 11:56:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 420/2304 [I 2022-10-11 11:56:54] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:26:22 [I 2022-10-11 11:56:54] ...trap/engines/engine.py.229: process: 0.84350 | load: 0.00296 [I 2022-10-11 11:56:54] ...trap/engines/engine.py.230: loss: 0.10850 [I 2022-10-11 11:56:54] ...optimizers/trijoint.py.151: total_norm: 21.987234 [I 2022-10-11 11:57:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 430/2304 [I 2022-10-11 11:57:03] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:26:21 [I 2022-10-11 11:57:03] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00293 [I 2022-10-11 11:57:03] ...trap/engines/engine.py.230: loss: 0.09610 [I 2022-10-11 11:57:03] ...optimizers/trijoint.py.151: total_norm: 29.221054 [I 2022-10-11 11:57:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 440/2304 [I 2022-10-11 11:57:11] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:26:04 [I 2022-10-11 11:57:11] ...trap/engines/engine.py.229: process: 0.83615 | load: 0.00292 [I 2022-10-11 11:57:11] ...trap/engines/engine.py.230: loss: 0.07582 [I 2022-10-11 11:57:11] ...optimizers/trijoint.py.151: total_norm: 9.461172 [I 2022-10-11 11:57:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 450/2304 [I 2022-10-11 11:57:20] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:25:54 [I 2022-10-11 11:57:20] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00292 [I 2022-10-11 11:57:20] ...trap/engines/engine.py.230: loss: 0.08891 [I 2022-10-11 11:57:20] ...optimizers/trijoint.py.151: total_norm: 12.479923 [I 2022-10-11 11:57:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 460/2304 [I 2022-10-11 11:57:28] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:25:35 [I 2022-10-11 11:57:28] ...trap/engines/engine.py.229: process: 0.82120 | load: 0.00292 [I 2022-10-11 11:57:28] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-11 11:57:28] ...optimizers/trijoint.py.151: total_norm: 25.170156 [I 2022-10-11 11:57:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 470/2304 [I 2022-10-11 11:57:36] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:25:38 [I 2022-10-11 11:57:36] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00292 [I 2022-10-11 11:57:36] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-11 11:57:36] ...optimizers/trijoint.py.151: total_norm: 14.688142 [I 2022-10-11 11:57:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 480/2304 [I 2022-10-11 11:57:45] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:25:38 [I 2022-10-11 11:57:45] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00293 [I 2022-10-11 11:57:45] ...trap/engines/engine.py.230: loss: 0.09513 [I 2022-10-11 11:57:45] ...optimizers/trijoint.py.151: total_norm: 9.471535 [I 2022-10-11 11:57:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 490/2304 [I 2022-10-11 11:57:53] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:25:17 [I 2022-10-11 11:57:53] ...trap/engines/engine.py.229: process: 0.84056 | load: 0.00302 [I 2022-10-11 11:57:53] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-11 11:57:53] ...optimizers/trijoint.py.151: total_norm: 22.244375 [I 2022-10-11 11:58:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 500/2304 [I 2022-10-11 11:58:02] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:25:08 [I 2022-10-11 11:58:02] ...trap/engines/engine.py.229: process: 0.83233 | load: 0.00292 [I 2022-10-11 11:58:02] ...trap/engines/engine.py.230: loss: 0.08571 [I 2022-10-11 11:58:02] ...optimizers/trijoint.py.151: total_norm: 19.025534 [I 2022-10-11 11:58:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 510/2304 [I 2022-10-11 11:58:10] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:10 [I 2022-10-11 11:58:10] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00299 [I 2022-10-11 11:58:10] ...trap/engines/engine.py.230: loss: 0.08170 [I 2022-10-11 11:58:10] ...optimizers/trijoint.py.151: total_norm: 13.049217 [I 2022-10-11 11:58:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 520/2304 [I 2022-10-11 11:58:19] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:24:54 [I 2022-10-11 11:58:19] ...trap/engines/engine.py.229: process: 0.82740 | load: 0.00295 [I 2022-10-11 11:58:19] ...trap/engines/engine.py.230: loss: 0.11281 [I 2022-10-11 11:58:19] ...optimizers/trijoint.py.151: total_norm: 9.889844 [I 2022-10-11 11:58:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 530/2304 [I 2022-10-11 11:58:27] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:24:48 [I 2022-10-11 11:58:27] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00303 [I 2022-10-11 11:58:27] ...trap/engines/engine.py.230: loss: 0.23016 [I 2022-10-11 11:58:27] ...optimizers/trijoint.py.151: total_norm: 16.055752 [I 2022-10-11 11:58:35] ...trap/engines/engine.py.224: train: epoch 104 | batch 540/2304 [I 2022-10-11 11:58:35] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:24:41 [I 2022-10-11 11:58:35] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00304 [I 2022-10-11 11:58:35] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-11 11:58:35] ...optimizers/trijoint.py.151: total_norm: 22.681349 [I 2022-10-11 11:58:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 550/2304 [I 2022-10-11 11:58:44] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:24:38 [I 2022-10-11 11:58:44] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00299 [I 2022-10-11 11:58:44] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-11 11:58:44] ...optimizers/trijoint.py.151: total_norm: 8.674776 [I 2022-10-11 11:58:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 560/2304 [I 2022-10-11 11:58:52] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:24:30 [I 2022-10-11 11:58:52] ...trap/engines/engine.py.229: process: 0.83804 | load: 0.00299 [I 2022-10-11 11:58:52] ...trap/engines/engine.py.230: loss: 0.14287 [I 2022-10-11 11:58:52] ...optimizers/trijoint.py.151: total_norm: 13.413604 [I 2022-10-11 11:59:01] ...trap/engines/engine.py.224: train: epoch 104 | batch 570/2304 [I 2022-10-11 11:59:01] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:16 [I 2022-10-11 11:59:01] ...trap/engines/engine.py.229: process: 0.83252 | load: 0.00298 [I 2022-10-11 11:59:01] ...trap/engines/engine.py.230: loss: 0.19557 [I 2022-10-11 11:59:01] ...optimizers/trijoint.py.151: total_norm: 14.818156 [I 2022-10-11 11:59:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 580/2304 [I 2022-10-11 11:59:09] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:24:24 [I 2022-10-11 11:59:09] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00307 [I 2022-10-11 11:59:09] ...trap/engines/engine.py.230: loss: 0.09610 [I 2022-10-11 11:59:09] ...optimizers/trijoint.py.151: total_norm: 16.015329 [I 2022-10-11 11:59:18] ...trap/engines/engine.py.224: train: epoch 104 | batch 590/2304 [I 2022-10-11 11:59:18] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:10 [I 2022-10-11 11:59:18] ...trap/engines/engine.py.229: process: 0.85244 | load: 0.00298 [I 2022-10-11 11:59:18] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-11 11:59:18] ...optimizers/trijoint.py.151: total_norm: 13.883651 [I 2022-10-11 11:59:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 600/2304 [I 2022-10-11 11:59:26] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:23:48 [I 2022-10-11 11:59:26] ...trap/engines/engine.py.229: process: 0.84383 | load: 0.00304 [I 2022-10-11 11:59:26] ...trap/engines/engine.py.230: loss: 0.10007 [I 2022-10-11 11:59:26] ...optimizers/trijoint.py.151: total_norm: 11.905720 [I 2022-10-11 11:59:35] ...trap/engines/engine.py.224: train: epoch 104 | batch 610/2304 [I 2022-10-11 11:59:35] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:23:45 [I 2022-10-11 11:59:35] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00299 [I 2022-10-11 11:59:35] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-11 11:59:35] ...optimizers/trijoint.py.151: total_norm: 12.504636 [I 2022-10-11 11:59:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 620/2304 [I 2022-10-11 11:59:43] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:23:41 [I 2022-10-11 11:59:43] ...trap/engines/engine.py.229: process: 0.84634 | load: 0.00297 [I 2022-10-11 11:59:43] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-11 11:59:43] ...optimizers/trijoint.py.151: total_norm: 19.540520 [I 2022-10-11 11:59:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 630/2304 [I 2022-10-11 11:59:52] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:23:35 [I 2022-10-11 11:59:52] ...trap/engines/engine.py.229: process: 0.85789 | load: 0.00304 [I 2022-10-11 11:59:52] ...trap/engines/engine.py.230: loss: 0.14320 [I 2022-10-11 11:59:52] ...optimizers/trijoint.py.151: total_norm: 11.085095 [I 2022-10-11 12:00:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 640/2304 [I 2022-10-11 12:00:00] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:23:29 [I 2022-10-11 12:00:00] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00307 [I 2022-10-11 12:00:00] ...trap/engines/engine.py.230: loss: 0.10452 [I 2022-10-11 12:00:00] ...optimizers/trijoint.py.151: total_norm: 32.273258 [I 2022-10-11 12:00:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 650/2304 [I 2022-10-11 12:00:09] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:23:28 [I 2022-10-11 12:00:09] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00301 [I 2022-10-11 12:00:09] ...trap/engines/engine.py.230: loss: 0.15342 [I 2022-10-11 12:00:09] ...optimizers/trijoint.py.151: total_norm: 14.453222 [I 2022-10-11 12:00:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 660/2304 [I 2022-10-11 12:00:17] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:11 [I 2022-10-11 12:00:17] ...trap/engines/engine.py.229: process: 0.83024 | load: 0.00299 [I 2022-10-11 12:00:17] ...trap/engines/engine.py.230: loss: 0.10667 [I 2022-10-11 12:00:17] ...optimizers/trijoint.py.151: total_norm: 15.259061 [I 2022-10-11 12:00:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 670/2304 [I 2022-10-11 12:00:26] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:23:00 [I 2022-10-11 12:00:26] ...trap/engines/engine.py.229: process: 0.83895 | load: 0.00295 [I 2022-10-11 12:00:26] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-11 12:00:26] ...optimizers/trijoint.py.151: total_norm: 19.076633 [I 2022-10-11 12:00:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 680/2304 [I 2022-10-11 12:00:52] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 1:12:26 [I 2022-10-11 12:00:52] ...trap/engines/engine.py.229: process: 0.85338 | load: 0.00307 [I 2022-10-11 12:00:52] ...trap/engines/engine.py.230: loss: 0.10072 [I 2022-10-11 12:00:52] ...optimizers/trijoint.py.151: total_norm: 13.781255 [I 2022-10-11 12:01:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 690/2304 [I 2022-10-11 12:01:00] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:28:03 [I 2022-10-11 12:01:00] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00312 [I 2022-10-11 12:01:00] ...trap/engines/engine.py.230: loss: 0.15921 [I 2022-10-11 12:01:00] ...optimizers/trijoint.py.151: total_norm: 13.998912 [I 2022-10-11 12:01:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 700/2304 [I 2022-10-11 12:01:09] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:23:04 [I 2022-10-11 12:01:09] ...trap/engines/engine.py.229: process: 0.84495 | load: 0.00299 [I 2022-10-11 12:01:09] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-11 12:01:09] ...optimizers/trijoint.py.151: total_norm: 14.567181 [I 2022-10-11 12:01:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 710/2304 [I 2022-10-11 12:01:17] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:29 [I 2022-10-11 12:01:17] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00299 [I 2022-10-11 12:01:17] ...trap/engines/engine.py.230: loss: 0.16103 [I 2022-10-11 12:01:17] ...optimizers/trijoint.py.151: total_norm: 13.129773 [I 2022-10-11 12:01:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 720/2304 [I 2022-10-11 12:01:26] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:15 [I 2022-10-11 12:01:26] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00313 [I 2022-10-11 12:01:26] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-11 12:01:26] ...optimizers/trijoint.py.151: total_norm: 20.255301 [I 2022-10-11 12:01:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 730/2304 [I 2022-10-11 12:01:34] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:22:06 [I 2022-10-11 12:01:34] ...trap/engines/engine.py.229: process: 0.83982 | load: 0.00302 [I 2022-10-11 12:01:34] ...trap/engines/engine.py.230: loss: 0.09095 [I 2022-10-11 12:01:34] ...optimizers/trijoint.py.151: total_norm: 16.710623 [I 2022-10-11 12:01:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 740/2304 [I 2022-10-11 12:01:43] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:54 [I 2022-10-11 12:01:43] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00299 [I 2022-10-11 12:01:43] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-11 12:01:43] ...optimizers/trijoint.py.151: total_norm: 18.485670 [I 2022-10-11 12:01:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 750/2304 [I 2022-10-11 12:01:51] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:21:44 [I 2022-10-11 12:01:51] ...trap/engines/engine.py.229: process: 0.84100 | load: 0.00299 [I 2022-10-11 12:01:51] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-11 12:01:51] ...optimizers/trijoint.py.151: total_norm: 13.664467 [I 2022-10-11 12:02:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 760/2304 [I 2022-10-11 12:02:00] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:47 [I 2022-10-11 12:02:00] ...trap/engines/engine.py.229: process: 0.84414 | load: 0.00298 [I 2022-10-11 12:02:00] ...trap/engines/engine.py.230: loss: 0.17974 [I 2022-10-11 12:02:00] ...optimizers/trijoint.py.151: total_norm: 18.078110 [I 2022-10-11 12:02:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 770/2304 [I 2022-10-11 12:02:08] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:21:43 [I 2022-10-11 12:02:08] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00302 [I 2022-10-11 12:02:08] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-11 12:02:08] ...optimizers/trijoint.py.151: total_norm: 22.591328 [I 2022-10-11 12:02:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 780/2304 [I 2022-10-11 12:02:17] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:28 [I 2022-10-11 12:02:17] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00299 [I 2022-10-11 12:02:17] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-11 12:02:17] ...optimizers/trijoint.py.151: total_norm: 11.928393 [I 2022-10-11 12:02:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 790/2304 [I 2022-10-11 12:02:25] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:21 [I 2022-10-11 12:02:25] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00300 [I 2022-10-11 12:02:25] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-11 12:02:25] ...optimizers/trijoint.py.151: total_norm: 14.622396 [I 2022-10-11 12:02:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 800/2304 [I 2022-10-11 12:02:34] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:11 [I 2022-10-11 12:02:34] ...trap/engines/engine.py.229: process: 0.84426 | load: 0.00302 [I 2022-10-11 12:02:34] ...trap/engines/engine.py.230: loss: 0.07565 [I 2022-10-11 12:02:34] ...optimizers/trijoint.py.151: total_norm: 10.290689 [I 2022-10-11 12:02:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 810/2304 [I 2022-10-11 12:02:42] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:21:14 [I 2022-10-11 12:02:42] ...trap/engines/engine.py.229: process: 0.84821 | load: 0.00300 [I 2022-10-11 12:02:42] ...trap/engines/engine.py.230: loss: 0.15171 [I 2022-10-11 12:02:42] ...optimizers/trijoint.py.151: total_norm: 11.142954 [I 2022-10-11 12:02:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 820/2304 [I 2022-10-11 12:02:51] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:51 [I 2022-10-11 12:02:51] ...trap/engines/engine.py.229: process: 0.84037 | load: 0.00299 [I 2022-10-11 12:02:51] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-11 12:02:51] ...optimizers/trijoint.py.151: total_norm: 17.261431 [I 2022-10-11 12:02:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 830/2304 [I 2022-10-11 12:02:59] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:21:01 [I 2022-10-11 12:02:59] ...trap/engines/engine.py.229: process: 0.86984 | load: 0.00309 [I 2022-10-11 12:02:59] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-11 12:02:59] ...optimizers/trijoint.py.151: total_norm: 18.614414 [I 2022-10-11 12:03:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 840/2304 [I 2022-10-11 12:03:08] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:53 [I 2022-10-11 12:03:08] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00314 [I 2022-10-11 12:03:08] ...trap/engines/engine.py.230: loss: 0.17535 [I 2022-10-11 12:03:08] ...optimizers/trijoint.py.151: total_norm: 9.660385 [I 2022-10-11 12:03:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 850/2304 [I 2022-10-11 12:03:16] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:49 [I 2022-10-11 12:03:16] ...trap/engines/engine.py.229: process: 0.86010 | load: 0.00320 [I 2022-10-11 12:03:16] ...trap/engines/engine.py.230: loss: 0.09587 [I 2022-10-11 12:03:16] ...optimizers/trijoint.py.151: total_norm: 9.919588 [I 2022-10-11 12:03:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 860/2304 [I 2022-10-11 12:03:25] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:39 [I 2022-10-11 12:03:25] ...trap/engines/engine.py.229: process: 0.85998 | load: 0.00313 [I 2022-10-11 12:03:25] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-11 12:03:25] ...optimizers/trijoint.py.151: total_norm: 16.967001 [I 2022-10-11 12:03:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 870/2304 [I 2022-10-11 12:03:34] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:20:34 [I 2022-10-11 12:03:34] ...trap/engines/engine.py.229: process: 0.87975 | load: 0.00313 [I 2022-10-11 12:03:34] ...trap/engines/engine.py.230: loss: 0.09292 [I 2022-10-11 12:03:34] ...optimizers/trijoint.py.151: total_norm: 14.117910 [I 2022-10-11 12:03:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 880/2304 [I 2022-10-11 12:03:42] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:23 [I 2022-10-11 12:03:42] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00313 [I 2022-10-11 12:03:42] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-11 12:03:42] ...optimizers/trijoint.py.151: total_norm: 13.433242 [I 2022-10-11 12:03:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 890/2304 [I 2022-10-11 12:03:51] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:20:16 [I 2022-10-11 12:03:51] ...trap/engines/engine.py.229: process: 0.86028 | load: 0.00313 [I 2022-10-11 12:03:51] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-11 12:03:51] ...optimizers/trijoint.py.151: total_norm: 15.793988 [I 2022-10-11 12:03:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 900/2304 [I 2022-10-11 12:03:59] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:20:03 [I 2022-10-11 12:03:59] ...trap/engines/engine.py.229: process: 0.85766 | load: 0.00314 [I 2022-10-11 12:03:59] ...trap/engines/engine.py.230: loss: 0.10848 [I 2022-10-11 12:03:59] ...optimizers/trijoint.py.151: total_norm: 10.926847 [I 2022-10-11 12:04:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 910/2304 [I 2022-10-11 12:04:08] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:20:04 [I 2022-10-11 12:04:08] ...trap/engines/engine.py.229: process: 0.87363 | load: 0.00314 [I 2022-10-11 12:04:08] ...trap/engines/engine.py.230: loss: 0.10354 [I 2022-10-11 12:04:08] ...optimizers/trijoint.py.151: total_norm: 9.604696 [I 2022-10-11 12:04:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 920/2304 [I 2022-10-11 12:04:17] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:19:45 [I 2022-10-11 12:04:17] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00314 [I 2022-10-11 12:04:17] ...trap/engines/engine.py.230: loss: 0.11122 [I 2022-10-11 12:04:17] ...optimizers/trijoint.py.151: total_norm: 12.990127 [I 2022-10-11 12:04:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 930/2304 [I 2022-10-11 12:04:25] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:19:38 [I 2022-10-11 12:04:25] ...trap/engines/engine.py.229: process: 0.86484 | load: 0.00312 [I 2022-10-11 12:04:25] ...trap/engines/engine.py.230: loss: 0.08852 [I 2022-10-11 12:04:25] ...optimizers/trijoint.py.151: total_norm: 28.323332 [I 2022-10-11 12:04:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 940/2304 [I 2022-10-11 12:04:34] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:19:29 [I 2022-10-11 12:04:34] ...trap/engines/engine.py.229: process: 0.85605 | load: 0.00314 [I 2022-10-11 12:04:34] ...trap/engines/engine.py.230: loss: 0.08708 [I 2022-10-11 12:04:34] ...optimizers/trijoint.py.151: total_norm: 7.909090 [I 2022-10-11 12:04:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 950/2304 [I 2022-10-11 12:04:42] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:19:23 [I 2022-10-11 12:04:42] ...trap/engines/engine.py.229: process: 0.85721 | load: 0.00317 [I 2022-10-11 12:04:42] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-11 12:04:42] ...optimizers/trijoint.py.151: total_norm: 13.790362 [I 2022-10-11 12:04:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 960/2304 [I 2022-10-11 12:04:51] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:19:14 [I 2022-10-11 12:04:51] ...trap/engines/engine.py.229: process: 0.85726 | load: 0.00315 [I 2022-10-11 12:04:51] ...trap/engines/engine.py.230: loss: 0.08081 [I 2022-10-11 12:04:51] ...optimizers/trijoint.py.151: total_norm: 12.115898 [I 2022-10-11 12:04:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 970/2304 [I 2022-10-11 12:04:59] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:19:00 [I 2022-10-11 12:04:59] ...trap/engines/engine.py.229: process: 0.86448 | load: 0.00319 [I 2022-10-11 12:04:59] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-11 12:04:59] ...optimizers/trijoint.py.151: total_norm: 15.201214 [I 2022-10-11 12:05:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 980/2304 [I 2022-10-11 12:05:08] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:19:10 [I 2022-10-11 12:05:08] ...trap/engines/engine.py.229: process: 0.87196 | load: 0.00322 [I 2022-10-11 12:05:08] ...trap/engines/engine.py.230: loss: 0.14494 [I 2022-10-11 12:05:08] ...optimizers/trijoint.py.151: total_norm: 17.665504 [I 2022-10-11 12:05:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 990/2304 [I 2022-10-11 12:05:17] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:57 [I 2022-10-11 12:05:17] ...trap/engines/engine.py.229: process: 0.86498 | load: 0.00314 [I 2022-10-11 12:05:17] ...trap/engines/engine.py.230: loss: 0.10597 [I 2022-10-11 12:05:17] ...optimizers/trijoint.py.151: total_norm: 25.396881 [I 2022-10-11 12:05:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 1000/2304 [I 2022-10-11 12:05:25] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:18:47 [I 2022-10-11 12:05:25] ...trap/engines/engine.py.229: process: 0.85929 | load: 0.00315 [I 2022-10-11 12:05:25] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-11 12:05:26] ...optimizers/trijoint.py.151: total_norm: 10.750045 [I 2022-10-11 12:05:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 1010/2304 [I 2022-10-11 12:05:34] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:42 [I 2022-10-11 12:05:34] ...trap/engines/engine.py.229: process: 0.87594 | load: 0.00313 [I 2022-10-11 12:05:34] ...trap/engines/engine.py.230: loss: 0.10214 [I 2022-10-11 12:05:34] ...optimizers/trijoint.py.151: total_norm: 12.289557 [I 2022-10-11 12:05:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 1020/2304 [I 2022-10-11 12:05:43] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:18:32 [I 2022-10-11 12:05:43] ...trap/engines/engine.py.229: process: 0.87502 | load: 0.00314 [I 2022-10-11 12:05:43] ...trap/engines/engine.py.230: loss: 0.15961 [I 2022-10-11 12:05:43] ...optimizers/trijoint.py.151: total_norm: 29.960854 [I 2022-10-11 12:05:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 1030/2304 [I 2022-10-11 12:05:51] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:18:10 [I 2022-10-11 12:05:51] ...trap/engines/engine.py.229: process: 0.84542 | load: 0.00313 [I 2022-10-11 12:05:51] ...trap/engines/engine.py.230: loss: 0.14839 [I 2022-10-11 12:05:51] ...optimizers/trijoint.py.151: total_norm: 18.970917 [I 2022-10-11 12:06:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 1040/2304 [I 2022-10-11 12:06:00] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:18:04 [I 2022-10-11 12:06:00] ...trap/engines/engine.py.229: process: 0.86443 | load: 0.00313 [I 2022-10-11 12:06:00] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-11 12:06:00] ...optimizers/trijoint.py.151: total_norm: 9.215905 [I 2022-10-11 12:06:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 1050/2304 [I 2022-10-11 12:06:09] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:17:48 [I 2022-10-11 12:06:09] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00321 [I 2022-10-11 12:06:09] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-11 12:06:09] ...optimizers/trijoint.py.151: total_norm: 23.698027 [I 2022-10-11 12:06:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 1060/2304 [I 2022-10-11 12:06:17] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:17:46 [I 2022-10-11 12:06:17] ...trap/engines/engine.py.229: process: 0.85934 | load: 0.00312 [I 2022-10-11 12:06:17] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-11 12:06:17] ...optimizers/trijoint.py.151: total_norm: 10.648074 [I 2022-10-11 12:06:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 1070/2304 [I 2022-10-11 12:06:26] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:17:35 [I 2022-10-11 12:06:26] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00311 [I 2022-10-11 12:06:26] ...trap/engines/engine.py.230: loss: 0.15752 [I 2022-10-11 12:06:26] ...optimizers/trijoint.py.151: total_norm: 22.010637 [I 2022-10-11 12:06:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 1080/2304 [I 2022-10-11 12:06:34] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:17:24 [I 2022-10-11 12:06:34] ...trap/engines/engine.py.229: process: 0.85886 | load: 0.00312 [I 2022-10-11 12:06:34] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-11 12:06:34] ...optimizers/trijoint.py.151: total_norm: 20.912786 [I 2022-10-11 12:06:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 1090/2304 [I 2022-10-11 12:06:43] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:17:21 [I 2022-10-11 12:06:43] ...trap/engines/engine.py.229: process: 0.86577 | load: 0.00311 [I 2022-10-11 12:06:43] ...trap/engines/engine.py.230: loss: 0.08773 [I 2022-10-11 12:06:43] ...optimizers/trijoint.py.151: total_norm: 16.915277 [I 2022-10-11 12:06:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 1100/2304 [I 2022-10-11 12:06:51] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:17:21 [I 2022-10-11 12:06:51] ...trap/engines/engine.py.229: process: 0.86134 | load: 0.00313 [I 2022-10-11 12:06:51] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-11 12:06:51] ...optimizers/trijoint.py.151: total_norm: 9.556890 [I 2022-10-11 12:07:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 1110/2304 [I 2022-10-11 12:07:00] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:17:15 [I 2022-10-11 12:07:00] ...trap/engines/engine.py.229: process: 0.86933 | load: 0.00311 [I 2022-10-11 12:07:00] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-11 12:07:00] ...optimizers/trijoint.py.151: total_norm: 18.218723 [I 2022-10-11 12:07:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 1120/2304 [I 2022-10-11 12:07:09] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:17:22 [I 2022-10-11 12:07:09] ...trap/engines/engine.py.229: process: 0.89385 | load: 0.00393 [I 2022-10-11 12:07:09] ...trap/engines/engine.py.230: loss: 0.11041 [I 2022-10-11 12:07:09] ...optimizers/trijoint.py.151: total_norm: 25.380611 [I 2022-10-11 12:07:18] ...trap/engines/engine.py.224: train: epoch 104 | batch 1130/2304 [I 2022-10-11 12:07:18] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:17:25 [I 2022-10-11 12:07:18] ...trap/engines/engine.py.229: process: 0.89586 | load: 0.00398 [I 2022-10-11 12:07:18] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-11 12:07:18] ...optimizers/trijoint.py.151: total_norm: 12.290266 [I 2022-10-11 12:07:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 1140/2304 [I 2022-10-11 12:07:27] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:17:12 [I 2022-10-11 12:07:27] ...trap/engines/engine.py.229: process: 0.88828 | load: 0.00391 [I 2022-10-11 12:07:27] ...trap/engines/engine.py.230: loss: 0.07697 [I 2022-10-11 12:07:27] ...optimizers/trijoint.py.151: total_norm: 22.749870 [I 2022-10-11 12:07:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 1150/2304 [I 2022-10-11 12:07:36] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:17:04 [I 2022-10-11 12:07:36] ...trap/engines/engine.py.229: process: 0.88771 | load: 0.00398 [I 2022-10-11 12:07:36] ...trap/engines/engine.py.230: loss: 0.07427 [I 2022-10-11 12:07:36] ...optimizers/trijoint.py.151: total_norm: 13.021814 [I 2022-10-11 12:07:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 1160/2304 [I 2022-10-11 12:07:44] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:58 [I 2022-10-11 12:07:44] ...trap/engines/engine.py.229: process: 0.90107 | load: 0.00396 [I 2022-10-11 12:07:44] ...trap/engines/engine.py.230: loss: 0.09577 [I 2022-10-11 12:07:44] ...optimizers/trijoint.py.151: total_norm: 11.005688 [I 2022-10-11 12:07:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 1170/2304 [I 2022-10-11 12:07:53] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:16:44 [I 2022-10-11 12:07:53] ...trap/engines/engine.py.229: process: 0.87564 | load: 0.00396 [I 2022-10-11 12:07:53] ...trap/engines/engine.py.230: loss: 0.13002 [I 2022-10-11 12:07:53] ...optimizers/trijoint.py.151: total_norm: 39.212784 [I 2022-10-11 12:08:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 1180/2304 [I 2022-10-11 12:08:02] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:16:38 [I 2022-10-11 12:08:02] ...trap/engines/engine.py.229: process: 0.88449 | load: 0.00397 [I 2022-10-11 12:08:02] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-11 12:08:02] ...optimizers/trijoint.py.151: total_norm: 10.330503 [I 2022-10-11 12:08:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 1190/2304 [I 2022-10-11 12:08:11] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:16:30 [I 2022-10-11 12:08:11] ...trap/engines/engine.py.229: process: 0.89132 | load: 0.00397 [I 2022-10-11 12:08:11] ...trap/engines/engine.py.230: loss: 0.11611 [I 2022-10-11 12:08:11] ...optimizers/trijoint.py.151: total_norm: 13.254760 [I 2022-10-11 12:08:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 1200/2304 [I 2022-10-11 12:08:20] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:16:17 [I 2022-10-11 12:08:20] ...trap/engines/engine.py.229: process: 0.87500 | load: 0.00393 [I 2022-10-11 12:08:20] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-11 12:08:20] ...optimizers/trijoint.py.151: total_norm: 16.379776 [I 2022-10-11 12:08:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1210/2304 [I 2022-10-11 12:08:29] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:16:13 [I 2022-10-11 12:08:29] ...trap/engines/engine.py.229: process: 0.89144 | load: 0.00401 [I 2022-10-11 12:08:29] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-11 12:08:29] ...optimizers/trijoint.py.151: total_norm: 23.293425 [I 2022-10-11 12:08:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1220/2304 [I 2022-10-11 12:08:38] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:31 [I 2022-10-11 12:08:38] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00311 [I 2022-10-11 12:08:38] ...trap/engines/engine.py.230: loss: 0.12614 [I 2022-10-11 12:08:38] ...optimizers/trijoint.py.151: total_norm: 14.761378 [I 2022-10-11 12:08:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 1230/2304 [I 2022-10-11 12:08:46] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:25 [I 2022-10-11 12:08:46] ...trap/engines/engine.py.229: process: 0.88555 | load: 0.00311 [I 2022-10-11 12:08:46] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-11 12:08:46] ...optimizers/trijoint.py.151: total_norm: 17.192015 [I 2022-10-11 12:08:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1240/2304 [I 2022-10-11 12:08:55] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:14 [I 2022-10-11 12:08:55] ...trap/engines/engine.py.229: process: 0.85788 | load: 0.00313 [I 2022-10-11 12:08:55] ...trap/engines/engine.py.230: loss: 0.08731 [I 2022-10-11 12:08:55] ...optimizers/trijoint.py.151: total_norm: 15.810784 [I 2022-10-11 12:09:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 1250/2304 [I 2022-10-11 12:09:03] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:58 [I 2022-10-11 12:09:03] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00312 [I 2022-10-11 12:09:03] ...trap/engines/engine.py.230: loss: 0.10377 [I 2022-10-11 12:09:03] ...optimizers/trijoint.py.151: total_norm: 7.821081 [I 2022-10-11 12:09:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1260/2304 [I 2022-10-11 12:09:12] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:50 [I 2022-10-11 12:09:12] ...trap/engines/engine.py.229: process: 0.86023 | load: 0.00313 [I 2022-10-11 12:09:12] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-11 12:09:12] ...optimizers/trijoint.py.151: total_norm: 11.702142 [I 2022-10-11 12:09:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 1270/2304 [I 2022-10-11 12:09:20] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:14:48 [I 2022-10-11 12:09:20] ...trap/engines/engine.py.229: process: 0.86495 | load: 0.00319 [I 2022-10-11 12:09:20] ...trap/engines/engine.py.230: loss: 0.13029 [I 2022-10-11 12:09:20] ...optimizers/trijoint.py.151: total_norm: 18.681150 [I 2022-10-11 12:09:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1280/2304 [I 2022-10-11 12:09:29] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:46 [I 2022-10-11 12:09:29] ...trap/engines/engine.py.229: process: 0.86831 | load: 0.00313 [I 2022-10-11 12:09:29] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-11 12:09:29] ...optimizers/trijoint.py.151: total_norm: 13.524998 [I 2022-10-11 12:09:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1290/2304 [I 2022-10-11 12:09:38] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:14:39 [I 2022-10-11 12:09:38] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00314 [I 2022-10-11 12:09:38] ...trap/engines/engine.py.230: loss: 0.10884 [I 2022-10-11 12:09:38] ...optimizers/trijoint.py.151: total_norm: 15.306726 [I 2022-10-11 12:09:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 1300/2304 [I 2022-10-11 12:09:46] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:30 [I 2022-10-11 12:09:46] ...trap/engines/engine.py.229: process: 0.86679 | load: 0.00315 [I 2022-10-11 12:09:46] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-11 12:09:46] ...optimizers/trijoint.py.151: total_norm: 16.363138 [I 2022-10-11 12:09:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1310/2304 [I 2022-10-11 12:09:55] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:38 [I 2022-10-11 12:09:55] ...trap/engines/engine.py.229: process: 0.89400 | load: 0.00401 [I 2022-10-11 12:09:55] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-11 12:09:55] ...optimizers/trijoint.py.151: total_norm: 24.106276 [I 2022-10-11 12:10:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1320/2304 [I 2022-10-11 12:10:04] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:14:10 [I 2022-10-11 12:10:04] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00313 [I 2022-10-11 12:10:04] ...trap/engines/engine.py.230: loss: 0.13652 [I 2022-10-11 12:10:04] ...optimizers/trijoint.py.151: total_norm: 17.920010 [I 2022-10-11 12:10:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1330/2304 [I 2022-10-11 12:10:13] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:57 [I 2022-10-11 12:10:13] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00312 [I 2022-10-11 12:10:13] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-11 12:10:13] ...optimizers/trijoint.py.151: total_norm: 12.147349 [I 2022-10-11 12:10:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1340/2304 [I 2022-10-11 12:10:21] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:13:46 [I 2022-10-11 12:10:21] ...trap/engines/engine.py.229: process: 0.86258 | load: 0.00309 [I 2022-10-11 12:10:21] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-11 12:10:21] ...optimizers/trijoint.py.151: total_norm: 13.522437 [I 2022-10-11 12:10:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1350/2304 [I 2022-10-11 12:10:30] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:38 [I 2022-10-11 12:10:30] ...trap/engines/engine.py.229: process: 0.86908 | load: 0.00311 [I 2022-10-11 12:10:30] ...trap/engines/engine.py.230: loss: 0.12753 [I 2022-10-11 12:10:30] ...optimizers/trijoint.py.151: total_norm: 15.224211 [I 2022-10-11 12:10:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1360/2304 [I 2022-10-11 12:10:38] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:13:27 [I 2022-10-11 12:10:38] ...trap/engines/engine.py.229: process: 0.85418 | load: 0.00308 [I 2022-10-11 12:10:38] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-11 12:10:38] ...optimizers/trijoint.py.151: total_norm: 16.367987 [I 2022-10-11 12:10:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1370/2304 [I 2022-10-11 12:10:47] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:19 [I 2022-10-11 12:10:47] ...trap/engines/engine.py.229: process: 0.85409 | load: 0.00312 [I 2022-10-11 12:10:47] ...trap/engines/engine.py.230: loss: 0.09955 [I 2022-10-11 12:10:47] ...optimizers/trijoint.py.151: total_norm: 18.079775 [I 2022-10-11 12:10:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1380/2304 [I 2022-10-11 12:10:55] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:11 [I 2022-10-11 12:10:55] ...trap/engines/engine.py.229: process: 0.85882 | load: 0.00315 [I 2022-10-11 12:10:55] ...trap/engines/engine.py.230: loss: 0.11133 [I 2022-10-11 12:10:55] ...optimizers/trijoint.py.151: total_norm: 74.152405 [I 2022-10-11 12:11:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1390/2304 [I 2022-10-11 12:11:04] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:13:03 [I 2022-10-11 12:11:04] ...trap/engines/engine.py.229: process: 0.86784 | load: 0.00314 [I 2022-10-11 12:11:04] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-11 12:11:04] ...optimizers/trijoint.py.151: total_norm: 10.016014 [I 2022-10-11 12:11:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1400/2304 [I 2022-10-11 12:11:12] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:49 [I 2022-10-11 12:11:12] ...trap/engines/engine.py.229: process: 0.85625 | load: 0.00312 [I 2022-10-11 12:11:12] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-11 12:11:12] ...optimizers/trijoint.py.151: total_norm: 14.616943 [I 2022-10-11 12:11:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1410/2304 [I 2022-10-11 12:11:21] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:51 [I 2022-10-11 12:11:21] ...trap/engines/engine.py.229: process: 0.86401 | load: 0.00313 [I 2022-10-11 12:11:21] ...trap/engines/engine.py.230: loss: 0.15731 [I 2022-10-11 12:11:21] ...optimizers/trijoint.py.151: total_norm: 16.788883 [I 2022-10-11 12:11:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1420/2304 [I 2022-10-11 12:11:30] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:33 [I 2022-10-11 12:11:30] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00313 [I 2022-10-11 12:11:30] ...trap/engines/engine.py.230: loss: 0.09337 [I 2022-10-11 12:11:30] ...optimizers/trijoint.py.151: total_norm: 11.097244 [I 2022-10-11 12:11:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1430/2304 [I 2022-10-11 12:11:38] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:12:23 [I 2022-10-11 12:11:38] ...trap/engines/engine.py.229: process: 0.83931 | load: 0.00312 [I 2022-10-11 12:11:38] ...trap/engines/engine.py.230: loss: 0.19815 [I 2022-10-11 12:11:38] ...optimizers/trijoint.py.151: total_norm: 16.070524 [I 2022-10-11 12:11:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1440/2304 [I 2022-10-11 12:11:47] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:12:17 [I 2022-10-11 12:11:47] ...trap/engines/engine.py.229: process: 0.85620 | load: 0.00312 [I 2022-10-11 12:11:47] ...trap/engines/engine.py.230: loss: 0.10116 [I 2022-10-11 12:11:47] ...optimizers/trijoint.py.151: total_norm: 14.094872 [I 2022-10-11 12:11:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1450/2304 [I 2022-10-11 12:11:55] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:12:13 [I 2022-10-11 12:11:55] ...trap/engines/engine.py.229: process: 0.85084 | load: 0.00311 [I 2022-10-11 12:11:55] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-11 12:11:55] ...optimizers/trijoint.py.151: total_norm: 13.267416 [I 2022-10-11 12:12:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1460/2304 [I 2022-10-11 12:12:04] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:12:02 [I 2022-10-11 12:12:04] ...trap/engines/engine.py.229: process: 0.86222 | load: 0.00311 [I 2022-10-11 12:12:04] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-11 12:12:04] ...optimizers/trijoint.py.151: total_norm: 22.732283 [I 2022-10-11 12:12:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1470/2304 [I 2022-10-11 12:12:13] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:59 [I 2022-10-11 12:12:13] ...trap/engines/engine.py.229: process: 0.87153 | load: 0.00318 [I 2022-10-11 12:12:13] ...trap/engines/engine.py.230: loss: 0.14186 [I 2022-10-11 12:12:13] ...optimizers/trijoint.py.151: total_norm: 11.796093 [I 2022-10-11 12:12:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1480/2304 [I 2022-10-11 12:12:21] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:11:42 [I 2022-10-11 12:12:21] ...trap/engines/engine.py.229: process: 0.83483 | load: 0.00313 [I 2022-10-11 12:12:21] ...trap/engines/engine.py.230: loss: 0.13681 [I 2022-10-11 12:12:21] ...optimizers/trijoint.py.151: total_norm: 8.026730 [I 2022-10-11 12:12:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1490/2304 [I 2022-10-11 12:12:30] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:33 [I 2022-10-11 12:12:30] ...trap/engines/engine.py.229: process: 0.84966 | load: 0.00314 [I 2022-10-11 12:12:30] ...trap/engines/engine.py.230: loss: 0.08930 [I 2022-10-11 12:12:30] ...optimizers/trijoint.py.151: total_norm: 16.787918 [I 2022-10-11 12:12:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1500/2304 [I 2022-10-11 12:12:38] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:11:26 [I 2022-10-11 12:12:38] ...trap/engines/engine.py.229: process: 0.84706 | load: 0.00312 [I 2022-10-11 12:12:38] ...trap/engines/engine.py.230: loss: 0.07724 [I 2022-10-11 12:12:38] ...optimizers/trijoint.py.151: total_norm: 12.215777 [I 2022-10-11 12:12:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1510/2304 [I 2022-10-11 12:12:47] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:18 [I 2022-10-11 12:12:47] ...trap/engines/engine.py.229: process: 0.86626 | load: 0.00312 [I 2022-10-11 12:12:47] ...trap/engines/engine.py.230: loss: 0.15039 [I 2022-10-11 12:12:47] ...optimizers/trijoint.py.151: total_norm: 15.420400 [I 2022-10-11 12:12:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1520/2304 [I 2022-10-11 12:12:55] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:11:16 [I 2022-10-11 12:12:55] ...trap/engines/engine.py.229: process: 0.86528 | load: 0.00314 [I 2022-10-11 12:12:55] ...trap/engines/engine.py.230: loss: 0.12954 [I 2022-10-11 12:12:55] ...optimizers/trijoint.py.151: total_norm: 10.000686 [I 2022-10-11 12:13:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1530/2304 [I 2022-10-11 12:13:04] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:11:04 [I 2022-10-11 12:13:04] ...trap/engines/engine.py.229: process: 0.86775 | load: 0.00308 [I 2022-10-11 12:13:04] ...trap/engines/engine.py.230: loss: 0.07444 [I 2022-10-11 12:13:04] ...optimizers/trijoint.py.151: total_norm: 12.945522 [I 2022-10-11 12:13:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1540/2304 [I 2022-10-11 12:13:13] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:51 [I 2022-10-11 12:13:13] ...trap/engines/engine.py.229: process: 0.84678 | load: 0.00311 [I 2022-10-11 12:13:13] ...trap/engines/engine.py.230: loss: 0.13861 [I 2022-10-11 12:13:13] ...optimizers/trijoint.py.151: total_norm: 29.843796 [I 2022-10-11 12:13:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1550/2304 [I 2022-10-11 12:13:21] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:10:47 [I 2022-10-11 12:13:21] ...trap/engines/engine.py.229: process: 0.86352 | load: 0.00317 [I 2022-10-11 12:13:21] ...trap/engines/engine.py.230: loss: 0.18408 [I 2022-10-11 12:13:21] ...optimizers/trijoint.py.151: total_norm: 21.914957 [I 2022-10-11 12:13:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1560/2304 [I 2022-10-11 12:13:30] ...trap/engines/engine.py.228: elapsed: 0:22:35 | left: 0:10:34 [I 2022-10-11 12:13:30] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00311 [I 2022-10-11 12:13:30] ...trap/engines/engine.py.230: loss: 0.18534 [I 2022-10-11 12:13:30] ...optimizers/trijoint.py.151: total_norm: 14.749439 [I 2022-10-11 12:13:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1570/2304 [I 2022-10-11 12:13:38] ...trap/engines/engine.py.228: elapsed: 0:22:43 | left: 0:10:27 [I 2022-10-11 12:13:38] ...trap/engines/engine.py.229: process: 0.83990 | load: 0.00328 [I 2022-10-11 12:13:38] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-11 12:13:38] ...optimizers/trijoint.py.151: total_norm: 21.272558 [I 2022-10-11 12:13:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1580/2304 [I 2022-10-11 12:13:47] ...trap/engines/engine.py.228: elapsed: 0:22:52 | left: 0:10:19 [I 2022-10-11 12:13:47] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00313 [I 2022-10-11 12:13:47] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-11 12:13:47] ...optimizers/trijoint.py.151: total_norm: 15.049734 [I 2022-10-11 12:13:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1590/2304 [I 2022-10-11 12:13:55] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:10 [I 2022-10-11 12:13:55] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00316 [I 2022-10-11 12:13:55] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-11 12:13:55] ...optimizers/trijoint.py.151: total_norm: 41.664501 [I 2022-10-11 12:14:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1600/2304 [I 2022-10-11 12:14:04] ...trap/engines/engine.py.228: elapsed: 0:23:09 | left: 0:10:01 [I 2022-10-11 12:14:04] ...trap/engines/engine.py.229: process: 0.86459 | load: 0.00314 [I 2022-10-11 12:14:04] ...trap/engines/engine.py.230: loss: 0.09982 [I 2022-10-11 12:14:04] ...optimizers/trijoint.py.151: total_norm: 15.095888 [I 2022-10-11 12:14:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1610/2304 [I 2022-10-11 12:14:13] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:54 [I 2022-10-11 12:14:13] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00314 [I 2022-10-11 12:14:13] ...trap/engines/engine.py.230: loss: 0.10749 [I 2022-10-11 12:14:13] ...optimizers/trijoint.py.151: total_norm: 10.575715 [I 2022-10-11 12:14:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1620/2304 [I 2022-10-11 12:14:21] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:46 [I 2022-10-11 12:14:21] ...trap/engines/engine.py.229: process: 0.86897 | load: 0.00314 [I 2022-10-11 12:14:21] ...trap/engines/engine.py.230: loss: 0.07641 [I 2022-10-11 12:14:21] ...optimizers/trijoint.py.151: total_norm: 13.579749 [I 2022-10-11 12:14:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1630/2304 [I 2022-10-11 12:14:30] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:39 [I 2022-10-11 12:14:30] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00318 [I 2022-10-11 12:14:30] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-11 12:14:30] ...optimizers/trijoint.py.151: total_norm: 16.582851 [I 2022-10-11 12:14:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1640/2304 [I 2022-10-11 12:14:38] ...trap/engines/engine.py.228: elapsed: 0:23:43 | left: 0:09:31 [I 2022-10-11 12:14:38] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00310 [I 2022-10-11 12:14:38] ...trap/engines/engine.py.230: loss: 0.11469 [I 2022-10-11 12:14:38] ...optimizers/trijoint.py.151: total_norm: 22.025915 [I 2022-10-11 12:14:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1650/2304 [I 2022-10-11 12:14:47] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:09:16 [I 2022-10-11 12:14:47] ...trap/engines/engine.py.229: process: 0.83933 | load: 0.00313 [I 2022-10-11 12:14:47] ...trap/engines/engine.py.230: loss: 0.08983 [I 2022-10-11 12:14:47] ...optimizers/trijoint.py.151: total_norm: 17.356493 [I 2022-10-11 12:14:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1660/2304 [I 2022-10-11 12:14:55] ...trap/engines/engine.py.228: elapsed: 0:24:00 | left: 0:09:07 [I 2022-10-11 12:14:55] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00310 [I 2022-10-11 12:14:55] ...trap/engines/engine.py.230: loss: 0.15614 [I 2022-10-11 12:14:55] ...optimizers/trijoint.py.151: total_norm: 25.032948 [I 2022-10-11 12:15:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1670/2304 [I 2022-10-11 12:15:04] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:09:02 [I 2022-10-11 12:15:04] ...trap/engines/engine.py.229: process: 0.85787 | load: 0.00315 [I 2022-10-11 12:15:04] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-11 12:15:04] ...optimizers/trijoint.py.151: total_norm: 10.311172 [I 2022-10-11 12:15:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1680/2304 [I 2022-10-11 12:15:13] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:52 [I 2022-10-11 12:15:13] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00314 [I 2022-10-11 12:15:13] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-11 12:15:13] ...optimizers/trijoint.py.151: total_norm: 14.445325 [I 2022-10-11 12:15:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1690/2304 [I 2022-10-11 12:15:21] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:45 [I 2022-10-11 12:15:21] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00312 [I 2022-10-11 12:15:21] ...trap/engines/engine.py.230: loss: 0.13516 [I 2022-10-11 12:15:21] ...optimizers/trijoint.py.151: total_norm: 29.059425 [I 2022-10-11 12:15:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1700/2304 [I 2022-10-11 12:15:30] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:37 [I 2022-10-11 12:15:30] ...trap/engines/engine.py.229: process: 0.86090 | load: 0.00312 [I 2022-10-11 12:15:30] ...trap/engines/engine.py.230: loss: 0.07703 [I 2022-10-11 12:15:30] ...optimizers/trijoint.py.151: total_norm: 12.650495 [I 2022-10-11 12:15:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1710/2304 [I 2022-10-11 12:15:38] ...trap/engines/engine.py.228: elapsed: 0:24:43 | left: 0:08:24 [I 2022-10-11 12:15:38] ...trap/engines/engine.py.229: process: 0.84730 | load: 0.00308 [I 2022-10-11 12:15:38] ...trap/engines/engine.py.230: loss: 0.07992 [I 2022-10-11 12:15:38] ...optimizers/trijoint.py.151: total_norm: 9.853304 [I 2022-10-11 12:15:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1720/2304 [I 2022-10-11 12:15:47] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:08:17 [I 2022-10-11 12:15:47] ...trap/engines/engine.py.229: process: 0.84228 | load: 0.00312 [I 2022-10-11 12:15:47] ...trap/engines/engine.py.230: loss: 0.14543 [I 2022-10-11 12:15:47] ...optimizers/trijoint.py.151: total_norm: 15.930437 [I 2022-10-11 12:15:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1730/2304 [I 2022-10-11 12:15:55] ...trap/engines/engine.py.228: elapsed: 0:25:00 | left: 0:08:08 [I 2022-10-11 12:15:55] ...trap/engines/engine.py.229: process: 0.84669 | load: 0.00312 [I 2022-10-11 12:15:55] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-11 12:15:55] ...optimizers/trijoint.py.151: total_norm: 11.774055 [I 2022-10-11 12:16:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1740/2304 [I 2022-10-11 12:16:04] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:08:02 [I 2022-10-11 12:16:04] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00312 [I 2022-10-11 12:16:04] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-11 12:16:04] ...optimizers/trijoint.py.151: total_norm: 12.451577 [I 2022-10-11 12:16:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1750/2304 [I 2022-10-11 12:16:12] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:50 [I 2022-10-11 12:16:12] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00314 [I 2022-10-11 12:16:12] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-11 12:16:12] ...optimizers/trijoint.py.151: total_norm: 19.835909 [I 2022-10-11 12:16:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1760/2304 [I 2022-10-11 12:16:21] ...trap/engines/engine.py.228: elapsed: 0:25:26 | left: 0:07:44 [I 2022-10-11 12:16:21] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00312 [I 2022-10-11 12:16:21] ...trap/engines/engine.py.230: loss: 0.12775 [I 2022-10-11 12:16:21] ...optimizers/trijoint.py.151: total_norm: 38.683315 [I 2022-10-11 12:16:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1770/2304 [I 2022-10-11 12:16:30] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:37 [I 2022-10-11 12:16:30] ...trap/engines/engine.py.229: process: 0.85312 | load: 0.00313 [I 2022-10-11 12:16:30] ...trap/engines/engine.py.230: loss: 0.06527 [I 2022-10-11 12:16:30] ...optimizers/trijoint.py.151: total_norm: 9.867451 [I 2022-10-11 12:16:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1780/2304 [I 2022-10-11 12:16:38] ...trap/engines/engine.py.228: elapsed: 0:25:43 | left: 0:07:29 [I 2022-10-11 12:16:38] ...trap/engines/engine.py.229: process: 0.86516 | load: 0.00312 [I 2022-10-11 12:16:38] ...trap/engines/engine.py.230: loss: 0.13710 [I 2022-10-11 12:16:38] ...optimizers/trijoint.py.151: total_norm: 9.597561 [I 2022-10-11 12:16:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1790/2304 [I 2022-10-11 12:16:47] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:19 [I 2022-10-11 12:16:47] ...trap/engines/engine.py.229: process: 0.85544 | load: 0.00313 [I 2022-10-11 12:16:47] ...trap/engines/engine.py.230: loss: 0.08232 [I 2022-10-11 12:16:47] ...optimizers/trijoint.py.151: total_norm: 27.959755 [I 2022-10-11 12:16:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1800/2304 [I 2022-10-11 12:16:55] ...trap/engines/engine.py.228: elapsed: 0:26:00 | left: 0:07:09 [I 2022-10-11 12:16:55] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00310 [I 2022-10-11 12:16:55] ...trap/engines/engine.py.230: loss: 0.12327 [I 2022-10-11 12:16:55] ...optimizers/trijoint.py.151: total_norm: 17.315308 [I 2022-10-11 12:17:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1810/2304 [I 2022-10-11 12:17:04] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:01 [I 2022-10-11 12:17:04] ...trap/engines/engine.py.229: process: 0.85912 | load: 0.00310 [I 2022-10-11 12:17:04] ...trap/engines/engine.py.230: loss: 0.08600 [I 2022-10-11 12:17:04] ...optimizers/trijoint.py.151: total_norm: 15.198404 [I 2022-10-11 12:17:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1820/2304 [I 2022-10-11 12:17:12] ...trap/engines/engine.py.228: elapsed: 0:26:17 | left: 0:06:53 [I 2022-10-11 12:17:12] ...trap/engines/engine.py.229: process: 0.85692 | load: 0.00314 [I 2022-10-11 12:17:12] ...trap/engines/engine.py.230: loss: 0.07510 [I 2022-10-11 12:17:12] ...optimizers/trijoint.py.151: total_norm: 10.862312 [I 2022-10-11 12:17:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1830/2304 [I 2022-10-11 12:17:21] ...trap/engines/engine.py.228: elapsed: 0:26:26 | left: 0:06:43 [I 2022-10-11 12:17:21] ...trap/engines/engine.py.229: process: 0.85248 | load: 0.00313 [I 2022-10-11 12:17:21] ...trap/engines/engine.py.230: loss: 0.08710 [I 2022-10-11 12:17:21] ...optimizers/trijoint.py.151: total_norm: 10.337598 [I 2022-10-11 12:17:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1840/2304 [I 2022-10-11 12:17:29] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:35 [I 2022-10-11 12:17:29] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00311 [I 2022-10-11 12:17:29] ...trap/engines/engine.py.230: loss: 0.08681 [I 2022-10-11 12:17:29] ...optimizers/trijoint.py.151: total_norm: 12.453600 [I 2022-10-11 12:17:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1850/2304 [I 2022-10-11 12:17:38] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:06:26 [I 2022-10-11 12:17:38] ...trap/engines/engine.py.229: process: 0.85543 | load: 0.00311 [I 2022-10-11 12:17:38] ...trap/engines/engine.py.230: loss: 0.11116 [I 2022-10-11 12:17:38] ...optimizers/trijoint.py.151: total_norm: 14.137857 [I 2022-10-11 12:17:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 1860/2304 [I 2022-10-11 12:17:46] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:20 [I 2022-10-11 12:17:46] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00317 [I 2022-10-11 12:17:46] ...trap/engines/engine.py.230: loss: 0.15952 [I 2022-10-11 12:17:46] ...optimizers/trijoint.py.151: total_norm: 14.052871 [I 2022-10-11 12:17:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1870/2304 [I 2022-10-11 12:17:55] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:06:15 [I 2022-10-11 12:17:55] ...trap/engines/engine.py.229: process: 0.88731 | load: 0.00311 [I 2022-10-11 12:17:55] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-11 12:17:55] ...optimizers/trijoint.py.151: total_norm: 17.297756 [I 2022-10-11 12:18:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1880/2304 [I 2022-10-11 12:18:04] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:01 [I 2022-10-11 12:18:04] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00308 [I 2022-10-11 12:18:04] ...trap/engines/engine.py.230: loss: 0.10113 [I 2022-10-11 12:18:04] ...optimizers/trijoint.py.151: total_norm: 9.979424 [I 2022-10-11 12:18:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1890/2304 [I 2022-10-11 12:18:12] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:51 [I 2022-10-11 12:18:12] ...trap/engines/engine.py.229: process: 0.82910 | load: 0.00297 [I 2022-10-11 12:18:12] ...trap/engines/engine.py.230: loss: 0.11933 [I 2022-10-11 12:18:12] ...optimizers/trijoint.py.151: total_norm: 29.014513 [I 2022-10-11 12:18:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1900/2304 [I 2022-10-11 12:18:21] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:53 [I 2022-10-11 12:18:21] ...trap/engines/engine.py.229: process: 0.88648 | load: 0.00373 [I 2022-10-11 12:18:21] ...trap/engines/engine.py.230: loss: 0.10306 [I 2022-10-11 12:18:21] ...optimizers/trijoint.py.151: total_norm: 19.492064 [I 2022-10-11 12:18:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1910/2304 [I 2022-10-11 12:18:30] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:41 [I 2022-10-11 12:18:30] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00372 [I 2022-10-11 12:18:30] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-11 12:18:30] ...optimizers/trijoint.py.151: total_norm: 40.728542 [I 2022-10-11 12:18:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1920/2304 [I 2022-10-11 12:18:38] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:26 [I 2022-10-11 12:18:38] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00295 [I 2022-10-11 12:18:38] ...trap/engines/engine.py.230: loss: 0.12762 [I 2022-10-11 12:18:38] ...optimizers/trijoint.py.151: total_norm: 10.554713 [I 2022-10-11 12:18:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1930/2304 [I 2022-10-11 12:18:47] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:15 [I 2022-10-11 12:18:47] ...trap/engines/engine.py.229: process: 0.83386 | load: 0.00295 [I 2022-10-11 12:18:47] ...trap/engines/engine.py.230: loss: 0.10902 [I 2022-10-11 12:18:47] ...optimizers/trijoint.py.151: total_norm: 35.839005 [I 2022-10-11 12:18:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1940/2304 [I 2022-10-11 12:18:55] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:05 [I 2022-10-11 12:18:55] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00294 [I 2022-10-11 12:18:55] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-11 12:18:55] ...optimizers/trijoint.py.151: total_norm: 11.975702 [I 2022-10-11 12:19:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1950/2304 [I 2022-10-11 12:19:04] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:04:57 [I 2022-10-11 12:19:04] ...trap/engines/engine.py.229: process: 0.84566 | load: 0.00297 [I 2022-10-11 12:19:04] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-11 12:19:04] ...optimizers/trijoint.py.151: total_norm: 16.349985 [I 2022-10-11 12:19:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1960/2304 [I 2022-10-11 12:19:12] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:48 [I 2022-10-11 12:19:12] ...trap/engines/engine.py.229: process: 0.82599 | load: 0.00296 [I 2022-10-11 12:19:12] ...trap/engines/engine.py.230: loss: 0.13553 [I 2022-10-11 12:19:12] ...optimizers/trijoint.py.151: total_norm: 17.900164 [I 2022-10-11 12:19:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 1970/2304 [I 2022-10-11 12:19:20] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:41 [I 2022-10-11 12:19:20] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00298 [I 2022-10-11 12:19:20] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-11 12:19:20] ...optimizers/trijoint.py.151: total_norm: 11.711888 [I 2022-10-11 12:19:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1980/2304 [I 2022-10-11 12:19:29] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:32 [I 2022-10-11 12:19:29] ...trap/engines/engine.py.229: process: 0.83085 | load: 0.00299 [I 2022-10-11 12:19:29] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-11 12:19:29] ...optimizers/trijoint.py.151: total_norm: 10.048765 [I 2022-10-11 12:19:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 1990/2304 [I 2022-10-11 12:19:37] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:24 [I 2022-10-11 12:19:37] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00297 [I 2022-10-11 12:19:37] ...trap/engines/engine.py.230: loss: 0.07777 [I 2022-10-11 12:19:37] ...optimizers/trijoint.py.151: total_norm: 9.284971 [I 2022-10-11 12:19:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 2000/2304 [I 2022-10-11 12:19:46] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:14 [I 2022-10-11 12:19:46] ...trap/engines/engine.py.229: process: 0.83778 | load: 0.00299 [I 2022-10-11 12:19:46] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-11 12:19:46] ...optimizers/trijoint.py.151: total_norm: 45.411026 [I 2022-10-11 12:19:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 2010/2304 [I 2022-10-11 12:19:54] ...trap/engines/engine.py.228: elapsed: 0:28:59 | left: 0:04:08 [I 2022-10-11 12:19:54] ...trap/engines/engine.py.229: process: 0.84281 | load: 0.00301 [I 2022-10-11 12:19:54] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-11 12:19:54] ...optimizers/trijoint.py.151: total_norm: 13.794643 [I 2022-10-11 12:20:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 2020/2304 [I 2022-10-11 12:20:03] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:57 [I 2022-10-11 12:20:03] ...trap/engines/engine.py.229: process: 0.84960 | load: 0.00303 [I 2022-10-11 12:20:03] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-11 12:20:03] ...optimizers/trijoint.py.151: total_norm: 14.746783 [I 2022-10-11 12:20:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 2030/2304 [I 2022-10-11 12:20:11] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:51 [I 2022-10-11 12:20:11] ...trap/engines/engine.py.229: process: 0.84130 | load: 0.00299 [I 2022-10-11 12:20:11] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-11 12:20:11] ...optimizers/trijoint.py.151: total_norm: 17.724072 [I 2022-10-11 12:20:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 2040/2304 [I 2022-10-11 12:20:20] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:42 [I 2022-10-11 12:20:20] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00298 [I 2022-10-11 12:20:20] ...trap/engines/engine.py.230: loss: 0.14160 [I 2022-10-11 12:20:20] ...optimizers/trijoint.py.151: total_norm: 13.808014 [I 2022-10-11 12:20:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 2050/2304 [I 2022-10-11 12:20:28] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:34 [I 2022-10-11 12:20:28] ...trap/engines/engine.py.229: process: 0.83787 | load: 0.00300 [I 2022-10-11 12:20:28] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-11 12:20:28] ...optimizers/trijoint.py.151: total_norm: 18.218859 [I 2022-10-11 12:20:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 2060/2304 [I 2022-10-11 12:20:37] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:26 [I 2022-10-11 12:20:37] ...trap/engines/engine.py.229: process: 0.84186 | load: 0.00309 [I 2022-10-11 12:20:37] ...trap/engines/engine.py.230: loss: 0.11743 [I 2022-10-11 12:20:37] ...optimizers/trijoint.py.151: total_norm: 10.408080 [I 2022-10-11 12:20:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 2070/2304 [I 2022-10-11 12:20:45] ...trap/engines/engine.py.228: elapsed: 0:29:50 | left: 0:03:18 [I 2022-10-11 12:20:45] ...trap/engines/engine.py.229: process: 0.83258 | load: 0.00317 [I 2022-10-11 12:20:45] ...trap/engines/engine.py.230: loss: 0.10288 [I 2022-10-11 12:20:45] ...optimizers/trijoint.py.151: total_norm: 20.428436 [I 2022-10-11 12:20:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 2080/2304 [I 2022-10-11 12:20:54] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:11 [I 2022-10-11 12:20:54] ...trap/engines/engine.py.229: process: 0.85984 | load: 0.00315 [I 2022-10-11 12:20:54] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-11 12:20:54] ...optimizers/trijoint.py.151: total_norm: 22.135870 [I 2022-10-11 12:21:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 2090/2304 [I 2022-10-11 12:21:02] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:03:04 [I 2022-10-11 12:21:02] ...trap/engines/engine.py.229: process: 0.85875 | load: 0.00309 [I 2022-10-11 12:21:02] ...trap/engines/engine.py.230: loss: 0.15125 [I 2022-10-11 12:21:02] ...optimizers/trijoint.py.151: total_norm: 21.752550 [I 2022-10-11 12:21:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 2100/2304 [I 2022-10-11 12:21:11] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:02:55 [I 2022-10-11 12:21:11] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00809 [I 2022-10-11 12:21:11] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-11 12:21:11] ...optimizers/trijoint.py.151: total_norm: 11.293274 [I 2022-10-11 12:21:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 2110/2304 [I 2022-10-11 12:21:19] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:46 [I 2022-10-11 12:21:19] ...trap/engines/engine.py.229: process: 0.85791 | load: 0.00313 [I 2022-10-11 12:21:19] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-11 12:21:19] ...optimizers/trijoint.py.151: total_norm: 22.401651 [I 2022-10-11 12:21:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 2120/2304 [I 2022-10-11 12:21:28] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:38 [I 2022-10-11 12:21:28] ...trap/engines/engine.py.229: process: 0.85074 | load: 0.00311 [I 2022-10-11 12:21:28] ...trap/engines/engine.py.230: loss: 0.09528 [I 2022-10-11 12:21:28] ...optimizers/trijoint.py.151: total_norm: 17.856001 [I 2022-10-11 12:21:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 2130/2304 [I 2022-10-11 12:21:37] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:29 [I 2022-10-11 12:21:37] ...trap/engines/engine.py.229: process: 0.85696 | load: 0.00313 [I 2022-10-11 12:21:37] ...trap/engines/engine.py.230: loss: 0.08302 [I 2022-10-11 12:21:37] ...optimizers/trijoint.py.151: total_norm: 8.842306 [I 2022-10-11 12:21:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 2140/2304 [I 2022-10-11 12:21:45] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:21 [I 2022-10-11 12:21:45] ...trap/engines/engine.py.229: process: 0.86802 | load: 0.00311 [I 2022-10-11 12:21:45] ...trap/engines/engine.py.230: loss: 0.08103 [I 2022-10-11 12:21:45] ...optimizers/trijoint.py.151: total_norm: 10.720509 [I 2022-10-11 12:21:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 2150/2304 [I 2022-10-11 12:21:54] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:13 [I 2022-10-11 12:21:54] ...trap/engines/engine.py.229: process: 0.87595 | load: 0.00312 [I 2022-10-11 12:21:54] ...trap/engines/engine.py.230: loss: 0.07283 [I 2022-10-11 12:21:54] ...optimizers/trijoint.py.151: total_norm: 9.861037 [I 2022-10-11 12:22:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 2160/2304 [I 2022-10-11 12:22:03] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:02:03 [I 2022-10-11 12:22:03] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00314 [I 2022-10-11 12:22:03] ...trap/engines/engine.py.230: loss: 0.17885 [I 2022-10-11 12:22:03] ...optimizers/trijoint.py.151: total_norm: 38.171864 [I 2022-10-11 12:22:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 2170/2304 [I 2022-10-11 12:22:11] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:54 [I 2022-10-11 12:22:11] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00314 [I 2022-10-11 12:22:11] ...trap/engines/engine.py.230: loss: 0.14934 [I 2022-10-11 12:22:11] ...optimizers/trijoint.py.151: total_norm: 19.531885 [I 2022-10-11 12:22:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 2180/2304 [I 2022-10-11 12:22:20] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:45 [I 2022-10-11 12:22:20] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00311 [I 2022-10-11 12:22:20] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-11 12:22:20] ...optimizers/trijoint.py.151: total_norm: 11.676539 [I 2022-10-11 12:22:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 2190/2304 [I 2022-10-11 12:22:28] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:37 [I 2022-10-11 12:22:28] ...trap/engines/engine.py.229: process: 0.86045 | load: 0.00312 [I 2022-10-11 12:22:28] ...trap/engines/engine.py.230: loss: 0.13521 [I 2022-10-11 12:22:28] ...optimizers/trijoint.py.151: total_norm: 30.978359 [I 2022-10-11 12:22:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 2200/2304 [I 2022-10-11 12:22:37] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:28 [I 2022-10-11 12:22:37] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00313 [I 2022-10-11 12:22:37] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-11 12:22:37] ...optimizers/trijoint.py.151: total_norm: 10.555074 [I 2022-10-11 12:22:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 2210/2304 [I 2022-10-11 12:22:45] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:20 [I 2022-10-11 12:22:45] ...trap/engines/engine.py.229: process: 0.84126 | load: 0.00312 [I 2022-10-11 12:22:45] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-11 12:22:45] ...optimizers/trijoint.py.151: total_norm: 10.328438 [I 2022-10-11 12:22:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 2220/2304 [I 2022-10-11 12:22:54] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:11 [I 2022-10-11 12:22:54] ...trap/engines/engine.py.229: process: 0.84013 | load: 0.00312 [I 2022-10-11 12:22:54] ...trap/engines/engine.py.230: loss: 0.20327 [I 2022-10-11 12:22:54] ...optimizers/trijoint.py.151: total_norm: 24.138475 [I 2022-10-11 12:23:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 2230/2304 [I 2022-10-11 12:23:02] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:03 [I 2022-10-11 12:23:02] ...trap/engines/engine.py.229: process: 0.86765 | load: 0.00311 [I 2022-10-11 12:23:02] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-11 12:23:02] ...optimizers/trijoint.py.151: total_norm: 7.646016 [I 2022-10-11 12:23:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 2240/2304 [I 2022-10-11 12:23:11] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:00:54 [I 2022-10-11 12:23:11] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00311 [I 2022-10-11 12:23:11] ...trap/engines/engine.py.230: loss: 0.11310 [I 2022-10-11 12:23:11] ...optimizers/trijoint.py.151: total_norm: 12.889728 [I 2022-10-11 12:23:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 2250/2304 [I 2022-10-11 12:23:19] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:45 [I 2022-10-11 12:23:19] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00311 [I 2022-10-11 12:23:19] ...trap/engines/engine.py.230: loss: 0.14999 [I 2022-10-11 12:23:19] ...optimizers/trijoint.py.151: total_norm: 15.428313 [I 2022-10-11 12:23:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 2260/2304 [I 2022-10-11 12:23:28] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:37 [I 2022-10-11 12:23:28] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00312 [I 2022-10-11 12:23:28] ...trap/engines/engine.py.230: loss: 0.10397 [I 2022-10-11 12:23:28] ...optimizers/trijoint.py.151: total_norm: 14.593102 [I 2022-10-11 12:23:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 2270/2304 [I 2022-10-11 12:23:37] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:28 [I 2022-10-11 12:23:37] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00310 [I 2022-10-11 12:23:37] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-11 12:23:37] ...optimizers/trijoint.py.151: total_norm: 12.218454 [I 2022-10-11 12:23:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 2280/2304 [I 2022-10-11 12:23:45] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:20 [I 2022-10-11 12:23:45] ...trap/engines/engine.py.229: process: 0.85134 | load: 0.00313 [I 2022-10-11 12:23:45] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-11 12:23:45] ...optimizers/trijoint.py.151: total_norm: 12.950430 [I 2022-10-11 12:23:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 2290/2304 [I 2022-10-11 12:23:54] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:12 [I 2022-10-11 12:23:54] ...trap/engines/engine.py.229: process: 0.87035 | load: 0.00312 [I 2022-10-11 12:23:54] ...trap/engines/engine.py.230: loss: 0.13752 [I 2022-10-11 12:23:54] ...optimizers/trijoint.py.151: total_norm: 13.970139 [I 2022-10-11 12:24:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 2300/2304 [I 2022-10-11 12:24:02] ...trap/engines/engine.py.228: elapsed: 0:33:07 | left: 0:00:03 [I 2022-10-11 12:24:02] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00018 [I 2022-10-11 12:24:02] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-11 12:24:02] ...optimizers/trijoint.py.151: total_norm: 14.702692 [I 2022-10-11 12:24:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 2304/2304 [I 2022-10-11 12:24:06] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:00 [I 2022-10-11 12:24:06] ...trap/engines/engine.py.229: process: 0.83251 | load: 0.00025 [I 2022-10-11 12:24:06] ...trap/engines/engine.py.230: loss: 0.11718 [I 2022-10-11 12:24:06] ...optimizers/trijoint.py.151: total_norm: 21.416746 [S 2022-10-11 12:24:07] ...trap/engines/engine.py.129: train_epoch.epoch: 104 [S 2022-10-11 12:24:07] ...trap/engines/engine.py.129: train_epoch.loss: 0.12179727611857226 [I 2022-10-11 12:24:07] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 12:24:09] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 104 [I 2022-10-11 12:24:09] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 12:24:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 12:24:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 12:24:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 12:24:15] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 12:24:15] ...trap/engines/engine.py.321: eval: epoch 104 | batch 0/510 [I 2022-10-11 12:24:15] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:58 [I 2022-10-11 12:24:15] ...trap/engines/engine.py.326: process: 6.35093 | load: 5.80694 [I 2022-10-11 12:24:24] ...trap/engines/engine.py.321: eval: epoch 104 | batch 10/510 [I 2022-10-11 12:24:24] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:10:17 [I 2022-10-11 12:24:24] ...trap/engines/engine.py.326: process: 0.42613 | load: 0.00019 [I 2022-10-11 12:24:30] ...trap/engines/engine.py.321: eval: epoch 104 | batch 20/510 [I 2022-10-11 12:24:30] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:52 [I 2022-10-11 12:24:30] ...trap/engines/engine.py.326: process: 0.53504 | load: 0.00020 [I 2022-10-11 12:24:36] ...trap/engines/engine.py.321: eval: epoch 104 | batch 30/510 [I 2022-10-11 12:24:36] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:47 [I 2022-10-11 12:24:36] ...trap/engines/engine.py.326: process: 0.40815 | load: 0.00018 [I 2022-10-11 12:24:41] ...trap/engines/engine.py.321: eval: epoch 104 | batch 40/510 [I 2022-10-11 12:24:41] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:03:34 [I 2022-10-11 12:24:41] ...trap/engines/engine.py.326: process: 0.43087 | load: 0.00018 [I 2022-10-11 12:24:45] ...trap/engines/engine.py.321: eval: epoch 104 | batch 50/510 [I 2022-10-11 12:24:45] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:35 [I 2022-10-11 12:24:45] ...trap/engines/engine.py.326: process: 0.43423 | load: 0.00019 [I 2022-10-11 12:24:50] ...trap/engines/engine.py.321: eval: epoch 104 | batch 60/510 [I 2022-10-11 12:24:50] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:55 [I 2022-10-11 12:24:50] ...trap/engines/engine.py.326: process: 0.59993 | load: 0.00018 [I 2022-10-11 12:24:55] ...trap/engines/engine.py.321: eval: epoch 104 | batch 70/510 [I 2022-10-11 12:24:55] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:16 [I 2022-10-11 12:24:55] ...trap/engines/engine.py.326: process: 0.42169 | load: 0.00018 [I 2022-10-11 12:25:00] ...trap/engines/engine.py.321: eval: epoch 104 | batch 80/510 [I 2022-10-11 12:25:00] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:47 [I 2022-10-11 12:25:00] ...trap/engines/engine.py.326: process: 0.56301 | load: 0.00018 [I 2022-10-11 12:25:05] ...trap/engines/engine.py.321: eval: epoch 104 | batch 90/510 [I 2022-10-11 12:25:05] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:09 [I 2022-10-11 12:25:05] ...trap/engines/engine.py.326: process: 0.47187 | load: 0.00017 [I 2022-10-11 12:25:09] ...trap/engines/engine.py.321: eval: epoch 104 | batch 100/510 [I 2022-10-11 12:25:09] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:53 [I 2022-10-11 12:25:09] ...trap/engines/engine.py.326: process: 0.30308 | load: 0.00018 [I 2022-10-11 12:25:12] ...trap/engines/engine.py.321: eval: epoch 104 | batch 110/510 [I 2022-10-11 12:25:12] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:10 [I 2022-10-11 12:25:12] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00019 [I 2022-10-11 12:25:15] ...trap/engines/engine.py.321: eval: epoch 104 | batch 120/510 [I 2022-10-11 12:25:15] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:02 [I 2022-10-11 12:25:15] ...trap/engines/engine.py.326: process: 0.30896 | load: 0.00019 [I 2022-10-11 12:25:19] ...trap/engines/engine.py.321: eval: epoch 104 | batch 130/510 [I 2022-10-11 12:25:19] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:59 [I 2022-10-11 12:25:19] ...trap/engines/engine.py.326: process: 0.31507 | load: 0.00019 [I 2022-10-11 12:25:22] ...trap/engines/engine.py.321: eval: epoch 104 | batch 140/510 [I 2022-10-11 12:25:22] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:56 [I 2022-10-11 12:25:22] ...trap/engines/engine.py.326: process: 0.31002 | load: 0.00019 [I 2022-10-11 12:25:25] ...trap/engines/engine.py.321: eval: epoch 104 | batch 150/510 [I 2022-10-11 12:25:25] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:52 [I 2022-10-11 12:25:25] ...trap/engines/engine.py.326: process: 0.31223 | load: 0.00019 [I 2022-10-11 12:25:28] ...trap/engines/engine.py.321: eval: epoch 104 | batch 160/510 [I 2022-10-11 12:25:28] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:50 [I 2022-10-11 12:25:28] ...trap/engines/engine.py.326: process: 0.31828 | load: 0.00019 [I 2022-10-11 12:25:31] ...trap/engines/engine.py.321: eval: epoch 104 | batch 170/510 [I 2022-10-11 12:25:31] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:48 [I 2022-10-11 12:25:31] ...trap/engines/engine.py.326: process: 0.32268 | load: 0.00019 [I 2022-10-11 12:25:34] ...trap/engines/engine.py.321: eval: epoch 104 | batch 180/510 [I 2022-10-11 12:25:34] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:44 [I 2022-10-11 12:25:34] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00019 [I 2022-10-11 12:25:37] ...trap/engines/engine.py.321: eval: epoch 104 | batch 190/510 [I 2022-10-11 12:25:37] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:40 [I 2022-10-11 12:25:37] ...trap/engines/engine.py.326: process: 0.31299 | load: 0.00019 [I 2022-10-11 12:25:41] ...trap/engines/engine.py.321: eval: epoch 104 | batch 200/510 [I 2022-10-11 12:25:41] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:37 [I 2022-10-11 12:25:41] ...trap/engines/engine.py.326: process: 0.31396 | load: 0.00019 [I 2022-10-11 12:25:44] ...trap/engines/engine.py.321: eval: epoch 104 | batch 210/510 [I 2022-10-11 12:25:44] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:35 [I 2022-10-11 12:25:44] ...trap/engines/engine.py.326: process: 0.32222 | load: 0.00019 [I 2022-10-11 12:25:47] ...trap/engines/engine.py.321: eval: epoch 104 | batch 220/510 [I 2022-10-11 12:25:47] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:31 [I 2022-10-11 12:25:47] ...trap/engines/engine.py.326: process: 0.31590 | load: 0.00018 [I 2022-10-11 12:25:50] ...trap/engines/engine.py.321: eval: epoch 104 | batch 230/510 [I 2022-10-11 12:25:50] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:28 [I 2022-10-11 12:25:50] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00019 [I 2022-10-11 12:25:53] ...trap/engines/engine.py.321: eval: epoch 104 | batch 240/510 [I 2022-10-11 12:25:53] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:25 [I 2022-10-11 12:25:53] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00021 [I 2022-10-11 12:25:56] ...trap/engines/engine.py.321: eval: epoch 104 | batch 250/510 [I 2022-10-11 12:25:56] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:22 [I 2022-10-11 12:25:56] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00019 [I 2022-10-11 12:26:00] ...trap/engines/engine.py.321: eval: epoch 104 | batch 260/510 [I 2022-10-11 12:26:00] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:19 [I 2022-10-11 12:26:00] ...trap/engines/engine.py.326: process: 0.32021 | load: 0.00019 [I 2022-10-11 12:26:03] ...trap/engines/engine.py.321: eval: epoch 104 | batch 270/510 [I 2022-10-11 12:26:03] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:16 [I 2022-10-11 12:26:03] ...trap/engines/engine.py.326: process: 0.32407 | load: 0.00020 [I 2022-10-11 12:26:06] ...trap/engines/engine.py.321: eval: epoch 104 | batch 280/510 [I 2022-10-11 12:26:06] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:12 [I 2022-10-11 12:26:06] ...trap/engines/engine.py.326: process: 0.31754 | load: 0.00019 [I 2022-10-11 12:26:09] ...trap/engines/engine.py.321: eval: epoch 104 | batch 290/510 [I 2022-10-11 12:26:09] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:09 [I 2022-10-11 12:26:09] ...trap/engines/engine.py.326: process: 0.31468 | load: 0.00020 [I 2022-10-11 12:26:12] ...trap/engines/engine.py.321: eval: epoch 104 | batch 300/510 [I 2022-10-11 12:26:12] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:06 [I 2022-10-11 12:26:12] ...trap/engines/engine.py.326: process: 0.31344 | load: 0.00020 [I 2022-10-11 12:26:15] ...trap/engines/engine.py.321: eval: epoch 104 | batch 310/510 [I 2022-10-11 12:26:15] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:03 [I 2022-10-11 12:26:15] ...trap/engines/engine.py.326: process: 0.31965 | load: 0.00020 [I 2022-10-11 12:26:19] ...trap/engines/engine.py.321: eval: epoch 104 | batch 320/510 [I 2022-10-11 12:26:19] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:00 [I 2022-10-11 12:26:19] ...trap/engines/engine.py.326: process: 0.32077 | load: 0.00020 [I 2022-10-11 12:26:22] ...trap/engines/engine.py.321: eval: epoch 104 | batch 330/510 [I 2022-10-11 12:26:22] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:57 [I 2022-10-11 12:26:22] ...trap/engines/engine.py.326: process: 0.32355 | load: 0.00020 [I 2022-10-11 12:26:25] ...trap/engines/engine.py.321: eval: epoch 104 | batch 340/510 [I 2022-10-11 12:26:25] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:53 [I 2022-10-11 12:26:25] ...trap/engines/engine.py.326: process: 0.31172 | load: 0.00019 [I 2022-10-11 12:26:28] ...trap/engines/engine.py.321: eval: epoch 104 | batch 350/510 [I 2022-10-11 12:26:28] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:50 [I 2022-10-11 12:26:28] ...trap/engines/engine.py.326: process: 0.31999 | load: 0.00020 [I 2022-10-11 12:26:31] ...trap/engines/engine.py.321: eval: epoch 104 | batch 360/510 [I 2022-10-11 12:26:31] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:47 [I 2022-10-11 12:26:31] ...trap/engines/engine.py.326: process: 0.31946 | load: 0.00020 [I 2022-10-11 12:26:34] ...trap/engines/engine.py.321: eval: epoch 104 | batch 370/510 [I 2022-10-11 12:26:34] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:44 [I 2022-10-11 12:26:34] ...trap/engines/engine.py.326: process: 0.31661 | load: 0.00021 [I 2022-10-11 12:26:38] ...trap/engines/engine.py.321: eval: epoch 104 | batch 380/510 [I 2022-10-11 12:26:38] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:41 [I 2022-10-11 12:26:38] ...trap/engines/engine.py.326: process: 0.31966 | load: 0.00020 [I 2022-10-11 12:26:41] ...trap/engines/engine.py.321: eval: epoch 104 | batch 390/510 [I 2022-10-11 12:26:41] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:38 [I 2022-10-11 12:26:41] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00019 [I 2022-10-11 12:26:44] ...trap/engines/engine.py.321: eval: epoch 104 | batch 400/510 [I 2022-10-11 12:26:44] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:34 [I 2022-10-11 12:26:44] ...trap/engines/engine.py.326: process: 0.31674 | load: 0.00019 [I 2022-10-11 12:26:47] ...trap/engines/engine.py.321: eval: epoch 104 | batch 410/510 [I 2022-10-11 12:26:47] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:31 [I 2022-10-11 12:26:47] ...trap/engines/engine.py.326: process: 0.32625 | load: 0.00020 [I 2022-10-11 12:26:50] ...trap/engines/engine.py.321: eval: epoch 104 | batch 420/510 [I 2022-10-11 12:26:50] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:28 [I 2022-10-11 12:26:50] ...trap/engines/engine.py.326: process: 0.31017 | load: 0.00019 [I 2022-10-11 12:26:54] ...trap/engines/engine.py.321: eval: epoch 104 | batch 430/510 [I 2022-10-11 12:26:54] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:25 [I 2022-10-11 12:26:54] ...trap/engines/engine.py.326: process: 0.31717 | load: 0.00020 [I 2022-10-11 12:26:57] ...trap/engines/engine.py.321: eval: epoch 104 | batch 440/510 [I 2022-10-11 12:26:57] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:22 [I 2022-10-11 12:26:57] ...trap/engines/engine.py.326: process: 0.32053 | load: 0.00019 [I 2022-10-11 12:27:00] ...trap/engines/engine.py.321: eval: epoch 104 | batch 450/510 [I 2022-10-11 12:27:00] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:18 [I 2022-10-11 12:27:00] ...trap/engines/engine.py.326: process: 0.31251 | load: 0.00021 [I 2022-10-11 12:27:03] ...trap/engines/engine.py.321: eval: epoch 104 | batch 460/510 [I 2022-10-11 12:27:03] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:15 [I 2022-10-11 12:27:03] ...trap/engines/engine.py.326: process: 0.31949 | load: 0.00019 [I 2022-10-11 12:27:06] ...trap/engines/engine.py.321: eval: epoch 104 | batch 470/510 [I 2022-10-11 12:27:06] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:12 [I 2022-10-11 12:27:06] ...trap/engines/engine.py.326: process: 0.32001 | load: 0.00020 [I 2022-10-11 12:27:09] ...trap/engines/engine.py.321: eval: epoch 104 | batch 480/510 [I 2022-10-11 12:27:09] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:09 [I 2022-10-11 12:27:09] ...trap/engines/engine.py.326: process: 0.31414 | load: 0.00019 [I 2022-10-11 12:27:13] ...trap/engines/engine.py.321: eval: epoch 104 | batch 490/510 [I 2022-10-11 12:27:13] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:06 [I 2022-10-11 12:27:13] ...trap/engines/engine.py.326: process: 0.32464 | load: 0.00021 [I 2022-10-11 12:27:16] ...trap/engines/engine.py.321: eval: epoch 104 | batch 500/510 [I 2022-10-11 12:27:16] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:03 [I 2022-10-11 12:27:16] ...trap/engines/engine.py.326: process: 0.32014 | load: 0.00020 [I 2022-10-11 12:27:19] ...trap/engines/engine.py.321: eval: epoch 104 | batch 510/510 [I 2022-10-11 12:27:19] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:00 [I 2022-10-11 12:27:19] ...trap/engines/engine.py.326: process: 0.31747 | load: 0.00015 [S 2022-10-11 12:27:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 104 [I 2022-10-11 12:27:20] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.747100 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.916500 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939900 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.744800 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.912300 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.939500 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013634 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007406 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005485 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012081 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007925 [S 2022-10-11 12:27:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005869 [I 2022-10-11 12:28:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 12:28:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 12:28:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 12:28:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 12:28:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 12:28:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 12:28:09] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 12:28:13] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 12:28:13] ...trap/engines/engine.py.163: Training model on trainset for epoch 105 [I 2022-10-11 12:28:13] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 12:28:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 0/2304 [I 2022-10-11 12:28:16] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:55:04 [I 2022-10-11 12:28:16] ...trap/engines/engine.py.229: process: 2.99681 | load: 2.09826 [I 2022-10-11 12:28:16] ...trap/engines/engine.py.230: loss: 0.09104 [I 2022-10-11 12:28:16] ...optimizers/trijoint.py.151: total_norm: 11.037843 [I 2022-10-11 12:28:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 10/2304 [I 2022-10-11 12:28:24] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:40:51 [I 2022-10-11 12:28:24] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00583 [I 2022-10-11 12:28:24] ...trap/engines/engine.py.230: loss: 0.10696 [I 2022-10-11 12:28:24] ...optimizers/trijoint.py.151: total_norm: 18.916275 [I 2022-10-11 12:28:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 20/2304 [I 2022-10-11 12:28:32] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:33:00 [I 2022-10-11 12:28:32] ...trap/engines/engine.py.229: process: 0.83440 | load: 0.00375 [I 2022-10-11 12:28:32] ...trap/engines/engine.py.230: loss: 0.12441 [I 2022-10-11 12:28:32] ...optimizers/trijoint.py.151: total_norm: 14.873344 [I 2022-10-11 12:28:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 30/2304 [I 2022-10-11 12:28:41] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:31:56 [I 2022-10-11 12:28:41] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00334 [I 2022-10-11 12:28:41] ...trap/engines/engine.py.230: loss: 0.11598 [I 2022-10-11 12:28:41] ...optimizers/trijoint.py.151: total_norm: 21.965027 [I 2022-10-11 12:28:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 40/2304 [I 2022-10-11 12:28:49] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:31:37 [I 2022-10-11 12:28:49] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00377 [I 2022-10-11 12:28:49] ...trap/engines/engine.py.230: loss: 0.13407 [I 2022-10-11 12:28:49] ...optimizers/trijoint.py.151: total_norm: 23.401747 [I 2022-10-11 12:28:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 50/2304 [I 2022-10-11 12:28:58] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:05 [I 2022-10-11 12:28:58] ...trap/engines/engine.py.229: process: 0.86788 | load: 0.00377 [I 2022-10-11 12:28:58] ...trap/engines/engine.py.230: loss: 0.13709 [I 2022-10-11 12:28:58] ...optimizers/trijoint.py.151: total_norm: 14.810486 [I 2022-10-11 12:29:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 60/2304 [I 2022-10-11 12:29:06] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:41 [I 2022-10-11 12:29:06] ...trap/engines/engine.py.229: process: 0.84412 | load: 0.00351 [I 2022-10-11 12:29:06] ...trap/engines/engine.py.230: loss: 0.14972 [I 2022-10-11 12:29:06] ...optimizers/trijoint.py.151: total_norm: 23.625401 [I 2022-10-11 12:29:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 70/2304 [I 2022-10-11 12:29:15] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:50 [I 2022-10-11 12:29:15] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00403 [I 2022-10-11 12:29:15] ...trap/engines/engine.py.230: loss: 0.11969 [I 2022-10-11 12:29:15] ...optimizers/trijoint.py.151: total_norm: 17.413324 [I 2022-10-11 12:29:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 80/2304 [I 2022-10-11 12:29:23] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:37 [I 2022-10-11 12:29:23] ...trap/engines/engine.py.229: process: 0.85277 | load: 0.00363 [I 2022-10-11 12:29:23] ...trap/engines/engine.py.230: loss: 0.10696 [I 2022-10-11 12:29:23] ...optimizers/trijoint.py.151: total_norm: 14.980534 [I 2022-10-11 12:29:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 90/2304 [I 2022-10-11 12:29:32] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:19 [I 2022-10-11 12:29:32] ...trap/engines/engine.py.229: process: 0.83370 | load: 0.00367 [I 2022-10-11 12:29:32] ...trap/engines/engine.py.230: loss: 0.09706 [I 2022-10-11 12:29:32] ...optimizers/trijoint.py.151: total_norm: 9.333620 [I 2022-10-11 12:29:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 100/2304 [I 2022-10-11 12:29:40] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:20 [I 2022-10-11 12:29:40] ...trap/engines/engine.py.229: process: 0.86836 | load: 0.00380 [I 2022-10-11 12:29:40] ...trap/engines/engine.py.230: loss: 0.18189 [I 2022-10-11 12:29:40] ...optimizers/trijoint.py.151: total_norm: 10.193717 [I 2022-10-11 12:29:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 110/2304 [I 2022-10-11 12:29:49] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:16 [I 2022-10-11 12:29:49] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00373 [I 2022-10-11 12:29:49] ...trap/engines/engine.py.230: loss: 0.07858 [I 2022-10-11 12:29:49] ...optimizers/trijoint.py.151: total_norm: 10.330184 [I 2022-10-11 12:29:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 120/2304 [I 2022-10-11 12:29:58] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:04 [I 2022-10-11 12:29:58] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00341 [I 2022-10-11 12:29:58] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-11 12:29:58] ...optimizers/trijoint.py.151: total_norm: 7.849591 [I 2022-10-11 12:30:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 130/2304 [I 2022-10-11 12:30:06] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:31:01 [I 2022-10-11 12:30:06] ...trap/engines/engine.py.229: process: 0.86602 | load: 0.00353 [I 2022-10-11 12:30:06] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-11 12:30:06] ...optimizers/trijoint.py.151: total_norm: 25.239557 [I 2022-10-11 12:30:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 140/2304 [I 2022-10-11 12:30:15] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:42 [I 2022-10-11 12:30:15] ...trap/engines/engine.py.229: process: 0.85560 | load: 0.00341 [I 2022-10-11 12:30:15] ...trap/engines/engine.py.230: loss: 0.08801 [I 2022-10-11 12:30:15] ...optimizers/trijoint.py.151: total_norm: 15.294288 [I 2022-10-11 12:30:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 150/2304 [I 2022-10-11 12:30:23] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:52 [I 2022-10-11 12:30:23] ...trap/engines/engine.py.229: process: 0.86138 | load: 0.00341 [I 2022-10-11 12:30:23] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-11 12:30:23] ...optimizers/trijoint.py.151: total_norm: 15.763071 [I 2022-10-11 12:30:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 160/2304 [I 2022-10-11 12:30:32] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:31 [I 2022-10-11 12:30:32] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00348 [I 2022-10-11 12:30:32] ...trap/engines/engine.py.230: loss: 0.11323 [I 2022-10-11 12:30:32] ...optimizers/trijoint.py.151: total_norm: 18.370094 [I 2022-10-11 12:30:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 170/2304 [I 2022-10-11 12:30:40] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:19 [I 2022-10-11 12:30:40] ...trap/engines/engine.py.229: process: 0.85797 | load: 0.00316 [I 2022-10-11 12:30:40] ...trap/engines/engine.py.230: loss: 0.20529 [I 2022-10-11 12:30:40] ...optimizers/trijoint.py.151: total_norm: 21.001539 [I 2022-10-11 12:30:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 180/2304 [I 2022-10-11 12:30:49] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:12 [I 2022-10-11 12:30:49] ...trap/engines/engine.py.229: process: 0.83873 | load: 0.00312 [I 2022-10-11 12:30:49] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-11 12:30:49] ...optimizers/trijoint.py.151: total_norm: 24.506060 [I 2022-10-11 12:30:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 190/2304 [I 2022-10-11 12:30:58] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:12 [I 2022-10-11 12:30:58] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00360 [I 2022-10-11 12:30:58] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-11 12:30:58] ...optimizers/trijoint.py.151: total_norm: 41.072979 [I 2022-10-11 12:31:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 200/2304 [I 2022-10-11 12:31:06] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:51 [I 2022-10-11 12:31:06] ...trap/engines/engine.py.229: process: 0.83380 | load: 0.00312 [I 2022-10-11 12:31:06] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-11 12:31:06] ...optimizers/trijoint.py.151: total_norm: 16.120028 [I 2022-10-11 12:31:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 210/2304 [I 2022-10-11 12:31:15] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:30:00 [I 2022-10-11 12:31:15] ...trap/engines/engine.py.229: process: 0.87729 | load: 0.00331 [I 2022-10-11 12:31:15] ...trap/engines/engine.py.230: loss: 0.09147 [I 2022-10-11 12:31:15] ...optimizers/trijoint.py.151: total_norm: 14.635656 [I 2022-10-11 12:31:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 220/2304 [I 2022-10-11 12:31:23] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:49 [I 2022-10-11 12:31:23] ...trap/engines/engine.py.229: process: 0.86139 | load: 0.00327 [I 2022-10-11 12:31:23] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-11 12:31:23] ...optimizers/trijoint.py.151: total_norm: 20.906536 [I 2022-10-11 12:31:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 230/2304 [I 2022-10-11 12:31:32] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:48 [I 2022-10-11 12:31:32] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00312 [I 2022-10-11 12:31:32] ...trap/engines/engine.py.230: loss: 0.10918 [I 2022-10-11 12:31:32] ...optimizers/trijoint.py.151: total_norm: 11.519757 [I 2022-10-11 12:31:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 240/2304 [I 2022-10-11 12:31:40] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:18 [I 2022-10-11 12:31:40] ...trap/engines/engine.py.229: process: 0.84040 | load: 0.00314 [I 2022-10-11 12:31:40] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-11 12:31:40] ...optimizers/trijoint.py.151: total_norm: 9.802160 [I 2022-10-11 12:31:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 250/2304 [I 2022-10-11 12:31:49] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:08 [I 2022-10-11 12:31:49] ...trap/engines/engine.py.229: process: 0.83603 | load: 0.00325 [I 2022-10-11 12:31:49] ...trap/engines/engine.py.230: loss: 0.08330 [I 2022-10-11 12:31:49] ...optimizers/trijoint.py.151: total_norm: 12.324978 [I 2022-10-11 12:31:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 260/2304 [I 2022-10-11 12:31:57] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:29:01 [I 2022-10-11 12:31:57] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00317 [I 2022-10-11 12:31:57] ...trap/engines/engine.py.230: loss: 0.07186 [I 2022-10-11 12:31:57] ...optimizers/trijoint.py.151: total_norm: 14.698872 [I 2022-10-11 12:32:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 270/2304 [I 2022-10-11 12:32:06] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:56 [I 2022-10-11 12:32:06] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00313 [I 2022-10-11 12:32:06] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-11 12:32:06] ...optimizers/trijoint.py.151: total_norm: 13.581187 [I 2022-10-11 12:32:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 280/2304 [I 2022-10-11 12:32:15] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:52 [I 2022-10-11 12:32:15] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00327 [I 2022-10-11 12:32:15] ...trap/engines/engine.py.230: loss: 0.09422 [I 2022-10-11 12:32:15] ...optimizers/trijoint.py.151: total_norm: 32.876587 [I 2022-10-11 12:32:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 290/2304 [I 2022-10-11 12:32:23] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:29:14 [I 2022-10-11 12:32:23] ...trap/engines/engine.py.229: process: 0.91169 | load: 0.00314 [I 2022-10-11 12:32:23] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-11 12:32:23] ...optimizers/trijoint.py.151: total_norm: 16.143858 [I 2022-10-11 12:32:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 300/2304 [I 2022-10-11 12:32:32] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:38 [I 2022-10-11 12:32:32] ...trap/engines/engine.py.229: process: 0.84052 | load: 0.00318 [I 2022-10-11 12:32:32] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-11 12:32:32] ...optimizers/trijoint.py.151: total_norm: 26.220942 [I 2022-10-11 12:32:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 310/2304 [I 2022-10-11 12:32:41] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:32 [I 2022-10-11 12:32:41] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00315 [I 2022-10-11 12:32:41] ...trap/engines/engine.py.230: loss: 0.15876 [I 2022-10-11 12:32:41] ...optimizers/trijoint.py.151: total_norm: 16.831242 [I 2022-10-11 12:32:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 320/2304 [I 2022-10-11 12:32:49] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:57 [I 2022-10-11 12:32:49] ...trap/engines/engine.py.229: process: 0.87828 | load: 0.00396 [I 2022-10-11 12:32:49] ...trap/engines/engine.py.230: loss: 0.09061 [I 2022-10-11 12:32:49] ...optimizers/trijoint.py.151: total_norm: 10.375252 [I 2022-10-11 12:32:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 330/2304 [I 2022-10-11 12:32:58] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:55 [I 2022-10-11 12:32:58] ...trap/engines/engine.py.229: process: 0.87835 | load: 0.00377 [I 2022-10-11 12:32:58] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-11 12:32:58] ...optimizers/trijoint.py.151: total_norm: 9.950953 [I 2022-10-11 12:33:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 340/2304 [I 2022-10-11 12:33:07] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:31 [I 2022-10-11 12:33:07] ...trap/engines/engine.py.229: process: 0.86068 | load: 0.00312 [I 2022-10-11 12:33:07] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-11 12:33:07] ...optimizers/trijoint.py.151: total_norm: 13.102912 [I 2022-10-11 12:33:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 350/2304 [I 2022-10-11 12:33:15] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:40 [I 2022-10-11 12:33:15] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00292 [I 2022-10-11 12:33:15] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-11 12:33:15] ...optimizers/trijoint.py.151: total_norm: 25.796642 [I 2022-10-11 12:33:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 360/2304 [I 2022-10-11 12:33:24] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:16 [I 2022-10-11 12:33:24] ...trap/engines/engine.py.229: process: 0.84450 | load: 0.00294 [I 2022-10-11 12:33:24] ...trap/engines/engine.py.230: loss: 0.12886 [I 2022-10-11 12:33:24] ...optimizers/trijoint.py.151: total_norm: 20.525114 [I 2022-10-11 12:33:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 370/2304 [I 2022-10-11 12:33:32] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:11 [I 2022-10-11 12:33:32] ...trap/engines/engine.py.229: process: 0.82232 | load: 0.00296 [I 2022-10-11 12:33:32] ...trap/engines/engine.py.230: loss: 0.14416 [I 2022-10-11 12:33:32] ...optimizers/trijoint.py.151: total_norm: 14.632263 [I 2022-10-11 12:33:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 380/2304 [I 2022-10-11 12:33:41] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:26:57 [I 2022-10-11 12:33:41] ...trap/engines/engine.py.229: process: 0.85415 | load: 0.00303 [I 2022-10-11 12:33:41] ...trap/engines/engine.py.230: loss: 0.22526 [I 2022-10-11 12:33:41] ...optimizers/trijoint.py.151: total_norm: 14.982634 [I 2022-10-11 12:33:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 390/2304 [I 2022-10-11 12:33:49] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:26:37 [I 2022-10-11 12:33:49] ...trap/engines/engine.py.229: process: 0.84345 | load: 0.00296 [I 2022-10-11 12:33:49] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-11 12:33:49] ...optimizers/trijoint.py.151: total_norm: 20.205679 [I 2022-10-11 12:33:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 400/2304 [I 2022-10-11 12:33:57] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:26:40 [I 2022-10-11 12:33:57] ...trap/engines/engine.py.229: process: 0.84712 | load: 0.00300 [I 2022-10-11 12:33:57] ...trap/engines/engine.py.230: loss: 0.09127 [I 2022-10-11 12:33:57] ...optimizers/trijoint.py.151: total_norm: 8.274526 [I 2022-10-11 12:34:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 410/2304 [I 2022-10-11 12:34:06] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:26:42 [I 2022-10-11 12:34:06] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00323 [I 2022-10-11 12:34:06] ...trap/engines/engine.py.230: loss: 0.10357 [I 2022-10-11 12:34:06] ...optimizers/trijoint.py.151: total_norm: 8.291382 [I 2022-10-11 12:34:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 420/2304 [I 2022-10-11 12:34:14] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:30 [I 2022-10-11 12:34:14] ...trap/engines/engine.py.229: process: 0.83219 | load: 0.00308 [I 2022-10-11 12:34:14] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-11 12:34:14] ...optimizers/trijoint.py.151: total_norm: 12.278264 [I 2022-10-11 12:34:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 430/2304 [I 2022-10-11 12:34:23] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:21 [I 2022-10-11 12:34:23] ...trap/engines/engine.py.229: process: 0.85270 | load: 0.00314 [I 2022-10-11 12:34:23] ...trap/engines/engine.py.230: loss: 0.15539 [I 2022-10-11 12:34:23] ...optimizers/trijoint.py.151: total_norm: 13.830472 [I 2022-10-11 12:34:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 440/2304 [I 2022-10-11 12:34:31] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:12 [I 2022-10-11 12:34:31] ...trap/engines/engine.py.229: process: 0.84454 | load: 0.00297 [I 2022-10-11 12:34:31] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-11 12:34:31] ...optimizers/trijoint.py.151: total_norm: 22.056486 [I 2022-10-11 12:34:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 450/2304 [I 2022-10-11 12:34:40] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:19 [I 2022-10-11 12:34:40] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00296 [I 2022-10-11 12:34:40] ...trap/engines/engine.py.230: loss: 0.09593 [I 2022-10-11 12:34:40] ...optimizers/trijoint.py.151: total_norm: 13.782449 [I 2022-10-11 12:34:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 460/2304 [I 2022-10-11 12:34:48] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:25:57 [I 2022-10-11 12:34:48] ...trap/engines/engine.py.229: process: 0.84587 | load: 0.00303 [I 2022-10-11 12:34:48] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-11 12:34:48] ...optimizers/trijoint.py.151: total_norm: 12.676647 [I 2022-10-11 12:34:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 470/2304 [I 2022-10-11 12:34:57] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:39 [I 2022-10-11 12:34:57] ...trap/engines/engine.py.229: process: 0.83216 | load: 0.00309 [I 2022-10-11 12:34:57] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-11 12:34:57] ...optimizers/trijoint.py.151: total_norm: 27.924723 [I 2022-10-11 12:35:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 480/2304 [I 2022-10-11 12:35:05] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:25:28 [I 2022-10-11 12:35:05] ...trap/engines/engine.py.229: process: 0.84207 | load: 0.00298 [I 2022-10-11 12:35:05] ...trap/engines/engine.py.230: loss: 0.09604 [I 2022-10-11 12:35:05] ...optimizers/trijoint.py.151: total_norm: 11.032341 [I 2022-10-11 12:35:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 490/2304 [I 2022-10-11 12:35:14] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:51 [I 2022-10-11 12:35:14] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00298 [I 2022-10-11 12:35:14] ...trap/engines/engine.py.230: loss: 0.09795 [I 2022-10-11 12:35:14] ...optimizers/trijoint.py.151: total_norm: 9.093257 [I 2022-10-11 12:35:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 500/2304 [I 2022-10-11 12:35:22] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:25:36 [I 2022-10-11 12:35:22] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00296 [I 2022-10-11 12:35:22] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-11 12:35:22] ...optimizers/trijoint.py.151: total_norm: 21.545486 [I 2022-10-11 12:35:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 510/2304 [I 2022-10-11 12:35:31] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:29 [I 2022-10-11 12:35:31] ...trap/engines/engine.py.229: process: 0.86400 | load: 0.00299 [I 2022-10-11 12:35:31] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-11 12:35:31] ...optimizers/trijoint.py.151: total_norm: 25.035839 [I 2022-10-11 12:35:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 520/2304 [I 2022-10-11 12:35:39] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:25:09 [I 2022-10-11 12:35:39] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00296 [I 2022-10-11 12:35:39] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-11 12:35:39] ...optimizers/trijoint.py.151: total_norm: 14.923725 [I 2022-10-11 12:35:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 530/2304 [I 2022-10-11 12:35:48] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:11 [I 2022-10-11 12:35:48] ...trap/engines/engine.py.229: process: 0.87808 | load: 0.00306 [I 2022-10-11 12:35:48] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-11 12:35:48] ...optimizers/trijoint.py.151: total_norm: 16.202410 [I 2022-10-11 12:35:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 540/2304 [I 2022-10-11 12:35:56] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:25:21 [I 2022-10-11 12:35:57] ...trap/engines/engine.py.229: process: 0.84790 | load: 0.00309 [I 2022-10-11 12:35:57] ...trap/engines/engine.py.230: loss: 0.16158 [I 2022-10-11 12:35:57] ...optimizers/trijoint.py.151: total_norm: 16.688295 [I 2022-10-11 12:36:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 550/2304 [I 2022-10-11 12:36:05] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:25:03 [I 2022-10-11 12:36:05] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00312 [I 2022-10-11 12:36:05] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-11 12:36:05] ...optimizers/trijoint.py.151: total_norm: 34.193413 [I 2022-10-11 12:36:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 560/2304 [I 2022-10-11 12:36:14] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:25:04 [I 2022-10-11 12:36:14] ...trap/engines/engine.py.229: process: 0.85465 | load: 0.00311 [I 2022-10-11 12:36:14] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-11 12:36:14] ...optimizers/trijoint.py.151: total_norm: 18.745502 [I 2022-10-11 12:36:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 570/2304 [I 2022-10-11 12:36:22] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:54 [I 2022-10-11 12:36:22] ...trap/engines/engine.py.229: process: 0.86984 | load: 0.00314 [I 2022-10-11 12:36:22] ...trap/engines/engine.py.230: loss: 0.10685 [I 2022-10-11 12:36:22] ...optimizers/trijoint.py.151: total_norm: 11.316960 [I 2022-10-11 12:36:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 580/2304 [I 2022-10-11 12:36:31] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:40 [I 2022-10-11 12:36:31] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00312 [I 2022-10-11 12:36:31] ...trap/engines/engine.py.230: loss: 0.13230 [I 2022-10-11 12:36:31] ...optimizers/trijoint.py.151: total_norm: 7.745822 [I 2022-10-11 12:36:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 590/2304 [I 2022-10-11 12:36:40] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:27 [I 2022-10-11 12:36:40] ...trap/engines/engine.py.229: process: 0.85113 | load: 0.00313 [I 2022-10-11 12:36:40] ...trap/engines/engine.py.230: loss: 0.09366 [I 2022-10-11 12:36:40] ...optimizers/trijoint.py.151: total_norm: 11.319477 [I 2022-10-11 12:36:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 600/2304 [I 2022-10-11 12:36:48] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:29 [I 2022-10-11 12:36:48] ...trap/engines/engine.py.229: process: 0.88524 | load: 0.00318 [I 2022-10-11 12:36:48] ...trap/engines/engine.py.230: loss: 0.08576 [I 2022-10-11 12:36:48] ...optimizers/trijoint.py.151: total_norm: 14.018536 [I 2022-10-11 12:36:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 610/2304 [I 2022-10-11 12:36:57] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:24:12 [I 2022-10-11 12:36:57] ...trap/engines/engine.py.229: process: 0.85278 | load: 0.00324 [I 2022-10-11 12:36:57] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-11 12:36:57] ...optimizers/trijoint.py.151: total_norm: 19.246616 [I 2022-10-11 12:37:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 620/2304 [I 2022-10-11 12:37:05] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:24:03 [I 2022-10-11 12:37:05] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00314 [I 2022-10-11 12:37:05] ...trap/engines/engine.py.230: loss: 0.14716 [I 2022-10-11 12:37:05] ...optimizers/trijoint.py.151: total_norm: 44.342396 [I 2022-10-11 12:37:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 630/2304 [I 2022-10-11 12:37:14] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:50 [I 2022-10-11 12:37:14] ...trap/engines/engine.py.229: process: 0.85763 | load: 0.00312 [I 2022-10-11 12:37:14] ...trap/engines/engine.py.230: loss: 0.13755 [I 2022-10-11 12:37:14] ...optimizers/trijoint.py.151: total_norm: 11.919401 [I 2022-10-11 12:37:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 640/2304 [I 2022-10-11 12:37:22] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:44 [I 2022-10-11 12:37:22] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00319 [I 2022-10-11 12:37:22] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-11 12:37:22] ...optimizers/trijoint.py.151: total_norm: 13.856318 [I 2022-10-11 12:37:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 650/2304 [I 2022-10-11 12:37:31] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:40 [I 2022-10-11 12:37:31] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00318 [I 2022-10-11 12:37:31] ...trap/engines/engine.py.230: loss: 0.13114 [I 2022-10-11 12:37:31] ...optimizers/trijoint.py.151: total_norm: 21.578964 [I 2022-10-11 12:37:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 660/2304 [I 2022-10-11 12:37:40] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:32 [I 2022-10-11 12:37:40] ...trap/engines/engine.py.229: process: 0.84604 | load: 0.00312 [I 2022-10-11 12:37:40] ...trap/engines/engine.py.230: loss: 0.08217 [I 2022-10-11 12:37:40] ...optimizers/trijoint.py.151: total_norm: 13.974066 [I 2022-10-11 12:37:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 670/2304 [I 2022-10-11 12:37:48] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:28 [I 2022-10-11 12:37:48] ...trap/engines/engine.py.229: process: 0.86596 | load: 0.00313 [I 2022-10-11 12:37:48] ...trap/engines/engine.py.230: loss: 0.10884 [I 2022-10-11 12:37:48] ...optimizers/trijoint.py.151: total_norm: 35.861404 [I 2022-10-11 12:37:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 680/2304 [I 2022-10-11 12:37:57] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:23 [I 2022-10-11 12:37:57] ...trap/engines/engine.py.229: process: 0.87926 | load: 0.00325 [I 2022-10-11 12:37:57] ...trap/engines/engine.py.230: loss: 0.11214 [I 2022-10-11 12:37:57] ...optimizers/trijoint.py.151: total_norm: 8.724597 [I 2022-10-11 12:38:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 690/2304 [I 2022-10-11 12:38:06] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:22:56 [I 2022-10-11 12:38:06] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00314 [I 2022-10-11 12:38:06] ...trap/engines/engine.py.230: loss: 0.13610 [I 2022-10-11 12:38:06] ...optimizers/trijoint.py.151: total_norm: 28.436460 [I 2022-10-11 12:38:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 700/2304 [I 2022-10-11 12:38:14] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:52 [I 2022-10-11 12:38:14] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00317 [I 2022-10-11 12:38:14] ...trap/engines/engine.py.230: loss: 0.09149 [I 2022-10-11 12:38:14] ...optimizers/trijoint.py.151: total_norm: 10.922692 [I 2022-10-11 12:38:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 710/2304 [I 2022-10-11 12:38:23] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:42 [I 2022-10-11 12:38:23] ...trap/engines/engine.py.229: process: 0.86381 | load: 0.00315 [I 2022-10-11 12:38:23] ...trap/engines/engine.py.230: loss: 0.12430 [I 2022-10-11 12:38:23] ...optimizers/trijoint.py.151: total_norm: 25.608910 [I 2022-10-11 12:38:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 720/2304 [I 2022-10-11 12:38:31] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:32 [I 2022-10-11 12:38:31] ...trap/engines/engine.py.229: process: 0.85408 | load: 0.00319 [I 2022-10-11 12:38:31] ...trap/engines/engine.py.230: loss: 0.25343 [I 2022-10-11 12:38:31] ...optimizers/trijoint.py.151: total_norm: 28.926083 [I 2022-10-11 12:38:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 730/2304 [I 2022-10-11 12:38:40] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:31 [I 2022-10-11 12:38:40] ...trap/engines/engine.py.229: process: 0.85705 | load: 0.00317 [I 2022-10-11 12:38:40] ...trap/engines/engine.py.230: loss: 0.08390 [I 2022-10-11 12:38:40] ...optimizers/trijoint.py.151: total_norm: 16.198675 [I 2022-10-11 12:38:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 740/2304 [I 2022-10-11 12:38:48] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:24 [I 2022-10-11 12:38:48] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00314 [I 2022-10-11 12:38:48] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-11 12:38:48] ...optimizers/trijoint.py.151: total_norm: 11.362167 [I 2022-10-11 12:38:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 750/2304 [I 2022-10-11 12:38:57] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:22:12 [I 2022-10-11 12:38:57] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00315 [I 2022-10-11 12:38:57] ...trap/engines/engine.py.230: loss: 0.10088 [I 2022-10-11 12:38:57] ...optimizers/trijoint.py.151: total_norm: 8.764716 [I 2022-10-11 12:39:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 760/2304 [I 2022-10-11 12:39:06] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:22:18 [I 2022-10-11 12:39:06] ...trap/engines/engine.py.229: process: 0.87099 | load: 0.00318 [I 2022-10-11 12:39:06] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-11 12:39:06] ...optimizers/trijoint.py.151: total_norm: 8.799902 [I 2022-10-11 12:39:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 770/2304 [I 2022-10-11 12:39:14] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:21:40 [I 2022-10-11 12:39:14] ...trap/engines/engine.py.229: process: 0.83429 | load: 0.00313 [I 2022-10-11 12:39:14] ...trap/engines/engine.py.230: loss: 0.10234 [I 2022-10-11 12:39:14] ...optimizers/trijoint.py.151: total_norm: 24.593994 [I 2022-10-11 12:39:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 780/2304 [I 2022-10-11 12:39:23] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:37 [I 2022-10-11 12:39:23] ...trap/engines/engine.py.229: process: 0.85088 | load: 0.00313 [I 2022-10-11 12:39:23] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-11 12:39:23] ...optimizers/trijoint.py.151: total_norm: 12.165973 [I 2022-10-11 12:39:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 790/2304 [I 2022-10-11 12:39:31] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:37 [I 2022-10-11 12:39:31] ...trap/engines/engine.py.229: process: 0.85111 | load: 0.00316 [I 2022-10-11 12:39:31] ...trap/engines/engine.py.230: loss: 0.17651 [I 2022-10-11 12:39:31] ...optimizers/trijoint.py.151: total_norm: 30.788187 [I 2022-10-11 12:39:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 800/2304 [I 2022-10-11 12:39:40] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:21:40 [I 2022-10-11 12:39:40] ...trap/engines/engine.py.229: process: 0.90930 | load: 0.00316 [I 2022-10-11 12:39:40] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-11 12:39:40] ...optimizers/trijoint.py.151: total_norm: 25.140146 [I 2022-10-11 12:39:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 810/2304 [I 2022-10-11 12:39:49] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:22:00 [I 2022-10-11 12:39:49] ...trap/engines/engine.py.229: process: 0.88275 | load: 0.00389 [I 2022-10-11 12:39:49] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-11 12:39:49] ...optimizers/trijoint.py.151: total_norm: 14.888135 [I 2022-10-11 12:39:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 820/2304 [I 2022-10-11 12:39:57] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:20:59 [I 2022-10-11 12:39:57] ...trap/engines/engine.py.229: process: 0.83121 | load: 0.00296 [I 2022-10-11 12:39:57] ...trap/engines/engine.py.230: loss: 0.09178 [I 2022-10-11 12:39:57] ...optimizers/trijoint.py.151: total_norm: 10.210819 [I 2022-10-11 12:40:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 830/2304 [I 2022-10-11 12:40:06] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:43 [I 2022-10-11 12:40:06] ...trap/engines/engine.py.229: process: 0.82521 | load: 0.00298 [I 2022-10-11 12:40:06] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-11 12:40:06] ...optimizers/trijoint.py.151: total_norm: 13.297893 [I 2022-10-11 12:40:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 840/2304 [I 2022-10-11 12:40:14] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:20:31 [I 2022-10-11 12:40:14] ...trap/engines/engine.py.229: process: 0.86470 | load: 0.00298 [I 2022-10-11 12:40:14] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-11 12:40:14] ...optimizers/trijoint.py.151: total_norm: 13.086597 [I 2022-10-11 12:40:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 850/2304 [I 2022-10-11 12:40:23] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:33 [I 2022-10-11 12:40:23] ...trap/engines/engine.py.229: process: 0.84433 | load: 0.00297 [I 2022-10-11 12:40:23] ...trap/engines/engine.py.230: loss: 0.10857 [I 2022-10-11 12:40:23] ...optimizers/trijoint.py.151: total_norm: 19.036425 [I 2022-10-11 12:40:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 860/2304 [I 2022-10-11 12:40:31] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:22 [I 2022-10-11 12:40:31] ...trap/engines/engine.py.229: process: 0.84069 | load: 0.00299 [I 2022-10-11 12:40:31] ...trap/engines/engine.py.230: loss: 0.11436 [I 2022-10-11 12:40:31] ...optimizers/trijoint.py.151: total_norm: 23.218971 [I 2022-10-11 12:40:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 870/2304 [I 2022-10-11 12:40:40] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:10 [I 2022-10-11 12:40:40] ...trap/engines/engine.py.229: process: 0.83079 | load: 0.00300 [I 2022-10-11 12:40:40] ...trap/engines/engine.py.230: loss: 0.12978 [I 2022-10-11 12:40:40] ...optimizers/trijoint.py.151: total_norm: 46.774246 [I 2022-10-11 12:40:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 880/2304 [I 2022-10-11 12:40:48] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:19:59 [I 2022-10-11 12:40:48] ...trap/engines/engine.py.229: process: 0.84427 | load: 0.00299 [I 2022-10-11 12:40:48] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-11 12:40:48] ...optimizers/trijoint.py.151: total_norm: 8.149407 [I 2022-10-11 12:40:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 890/2304 [I 2022-10-11 12:40:57] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:19:51 [I 2022-10-11 12:40:57] ...trap/engines/engine.py.229: process: 0.83902 | load: 0.00299 [I 2022-10-11 12:40:57] ...trap/engines/engine.py.230: loss: 0.09880 [I 2022-10-11 12:40:57] ...optimizers/trijoint.py.151: total_norm: 8.406686 [I 2022-10-11 12:41:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 900/2304 [I 2022-10-11 12:41:05] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:19:45 [I 2022-10-11 12:41:05] ...trap/engines/engine.py.229: process: 0.83777 | load: 0.00298 [I 2022-10-11 12:41:05] ...trap/engines/engine.py.230: loss: 0.08548 [I 2022-10-11 12:41:05] ...optimizers/trijoint.py.151: total_norm: 17.567011 [I 2022-10-11 12:41:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 910/2304 [I 2022-10-11 12:41:14] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:19:46 [I 2022-10-11 12:41:14] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00304 [I 2022-10-11 12:41:14] ...trap/engines/engine.py.230: loss: 0.07213 [I 2022-10-11 12:41:14] ...optimizers/trijoint.py.151: total_norm: 11.249368 [I 2022-10-11 12:41:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 920/2304 [I 2022-10-11 12:41:22] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:29 [I 2022-10-11 12:41:22] ...trap/engines/engine.py.229: process: 0.85020 | load: 0.00297 [I 2022-10-11 12:41:22] ...trap/engines/engine.py.230: loss: 0.09244 [I 2022-10-11 12:41:22] ...optimizers/trijoint.py.151: total_norm: 15.220636 [I 2022-10-11 12:41:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 930/2304 [I 2022-10-11 12:41:31] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:32 [I 2022-10-11 12:41:31] ...trap/engines/engine.py.229: process: 0.86520 | load: 0.00300 [I 2022-10-11 12:41:31] ...trap/engines/engine.py.230: loss: 0.10439 [I 2022-10-11 12:41:31] ...optimizers/trijoint.py.151: total_norm: 17.466324 [I 2022-10-11 12:41:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 940/2304 [I 2022-10-11 12:41:39] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:17 [I 2022-10-11 12:41:39] ...trap/engines/engine.py.229: process: 0.84586 | load: 0.00302 [I 2022-10-11 12:41:39] ...trap/engines/engine.py.230: loss: 0.13222 [I 2022-10-11 12:41:39] ...optimizers/trijoint.py.151: total_norm: 12.849345 [I 2022-10-11 12:41:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 950/2304 [I 2022-10-11 12:41:48] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:07 [I 2022-10-11 12:41:48] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00310 [I 2022-10-11 12:41:48] ...trap/engines/engine.py.230: loss: 0.16365 [I 2022-10-11 12:41:48] ...optimizers/trijoint.py.151: total_norm: 16.931833 [I 2022-10-11 12:41:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 960/2304 [I 2022-10-11 12:41:56] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:00 [I 2022-10-11 12:41:56] ...trap/engines/engine.py.229: process: 0.83087 | load: 0.00310 [I 2022-10-11 12:41:56] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-11 12:41:56] ...optimizers/trijoint.py.151: total_norm: 16.706821 [I 2022-10-11 12:42:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 970/2304 [I 2022-10-11 12:42:05] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:07 [I 2022-10-11 12:42:05] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00314 [I 2022-10-11 12:42:05] ...trap/engines/engine.py.230: loss: 0.17420 [I 2022-10-11 12:42:05] ...optimizers/trijoint.py.151: total_norm: 15.697555 [I 2022-10-11 12:42:13] ...trap/engines/engine.py.224: train: epoch 105 | batch 980/2304 [I 2022-10-11 12:42:13] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:19:00 [I 2022-10-11 12:42:13] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00311 [I 2022-10-11 12:42:13] ...trap/engines/engine.py.230: loss: 0.19109 [I 2022-10-11 12:42:13] ...optimizers/trijoint.py.151: total_norm: 19.657244 [I 2022-10-11 12:42:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 990/2304 [I 2022-10-11 12:42:22] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:37 [I 2022-10-11 12:42:22] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00313 [I 2022-10-11 12:42:22] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-11 12:42:22] ...optimizers/trijoint.py.151: total_norm: 34.469734 [I 2022-10-11 12:42:30] ...trap/engines/engine.py.224: train: epoch 105 | batch 1000/2304 [I 2022-10-11 12:42:30] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:18:38 [I 2022-10-11 12:42:30] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00319 [I 2022-10-11 12:42:30] ...trap/engines/engine.py.230: loss: 0.09356 [I 2022-10-11 12:42:30] ...optimizers/trijoint.py.151: total_norm: 10.125415 [I 2022-10-11 12:42:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 1010/2304 [I 2022-10-11 12:42:39] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:34 [I 2022-10-11 12:42:39] ...trap/engines/engine.py.229: process: 0.86127 | load: 0.00314 [I 2022-10-11 12:42:39] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-11 12:42:39] ...optimizers/trijoint.py.151: total_norm: 10.546430 [I 2022-10-11 12:42:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 1020/2304 [I 2022-10-11 12:42:48] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:23 [I 2022-10-11 12:42:48] ...trap/engines/engine.py.229: process: 0.87257 | load: 0.00315 [I 2022-10-11 12:42:48] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-11 12:42:48] ...optimizers/trijoint.py.151: total_norm: 13.914578 [I 2022-10-11 12:42:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 1030/2304 [I 2022-10-11 12:42:56] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:21 [I 2022-10-11 12:42:56] ...trap/engines/engine.py.229: process: 0.85903 | load: 0.00315 [I 2022-10-11 12:42:56] ...trap/engines/engine.py.230: loss: 0.11827 [I 2022-10-11 12:42:56] ...optimizers/trijoint.py.151: total_norm: 20.233604 [I 2022-10-11 12:43:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 1040/2304 [I 2022-10-11 12:43:05] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:59 [I 2022-10-11 12:43:05] ...trap/engines/engine.py.229: process: 0.83928 | load: 0.00313 [I 2022-10-11 12:43:05] ...trap/engines/engine.py.230: loss: 0.07832 [I 2022-10-11 12:43:05] ...optimizers/trijoint.py.151: total_norm: 14.903607 [I 2022-10-11 12:43:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 1050/2304 [I 2022-10-11 12:43:14] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:54 [I 2022-10-11 12:43:14] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00311 [I 2022-10-11 12:43:14] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-11 12:43:14] ...optimizers/trijoint.py.151: total_norm: 18.553452 [I 2022-10-11 12:43:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 1060/2304 [I 2022-10-11 12:43:22] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:18:18 [I 2022-10-11 12:43:22] ...trap/engines/engine.py.229: process: 0.90316 | load: 0.00392 [I 2022-10-11 12:43:22] ...trap/engines/engine.py.230: loss: 0.10895 [I 2022-10-11 12:43:22] ...optimizers/trijoint.py.151: total_norm: 17.003788 [I 2022-10-11 12:43:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 1070/2304 [I 2022-10-11 12:43:31] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:53 [I 2022-10-11 12:43:31] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00316 [I 2022-10-11 12:43:31] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-11 12:43:31] ...optimizers/trijoint.py.151: total_norm: 11.655627 [I 2022-10-11 12:43:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 1080/2304 [I 2022-10-11 12:43:40] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:17:40 [I 2022-10-11 12:43:40] ...trap/engines/engine.py.229: process: 0.87880 | load: 0.00320 [I 2022-10-11 12:43:40] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-11 12:43:40] ...optimizers/trijoint.py.151: total_norm: 53.398922 [I 2022-10-11 12:43:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 1090/2304 [I 2022-10-11 12:43:48] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:23 [I 2022-10-11 12:43:48] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00314 [I 2022-10-11 12:43:48] ...trap/engines/engine.py.230: loss: 0.12836 [I 2022-10-11 12:43:48] ...optimizers/trijoint.py.151: total_norm: 25.431576 [I 2022-10-11 12:43:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 1100/2304 [I 2022-10-11 12:43:57] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:09 [I 2022-10-11 12:43:57] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00313 [I 2022-10-11 12:43:57] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-11 12:43:57] ...optimizers/trijoint.py.151: total_norm: 23.903715 [I 2022-10-11 12:44:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 1110/2304 [I 2022-10-11 12:44:06] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:07 [I 2022-10-11 12:44:06] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00313 [I 2022-10-11 12:44:06] ...trap/engines/engine.py.230: loss: 0.11711 [I 2022-10-11 12:44:06] ...optimizers/trijoint.py.151: total_norm: 21.433355 [I 2022-10-11 12:44:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 1120/2304 [I 2022-10-11 12:44:14] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:16:55 [I 2022-10-11 12:44:14] ...trap/engines/engine.py.229: process: 0.84616 | load: 0.00314 [I 2022-10-11 12:44:14] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-11 12:44:14] ...optimizers/trijoint.py.151: total_norm: 14.972533 [I 2022-10-11 12:44:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 1130/2304 [I 2022-10-11 12:44:23] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:16:39 [I 2022-10-11 12:44:23] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00313 [I 2022-10-11 12:44:23] ...trap/engines/engine.py.230: loss: 0.15329 [I 2022-10-11 12:44:23] ...optimizers/trijoint.py.151: total_norm: 16.666433 [I 2022-10-11 12:44:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 1140/2304 [I 2022-10-11 12:44:31] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:16:34 [I 2022-10-11 12:44:31] ...trap/engines/engine.py.229: process: 0.85276 | load: 0.00315 [I 2022-10-11 12:44:31] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-11 12:44:31] ...optimizers/trijoint.py.151: total_norm: 14.924865 [I 2022-10-11 12:44:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 1150/2304 [I 2022-10-11 12:44:40] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:16:25 [I 2022-10-11 12:44:40] ...trap/engines/engine.py.229: process: 0.85977 | load: 0.00318 [I 2022-10-11 12:44:40] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-11 12:44:40] ...optimizers/trijoint.py.151: total_norm: 15.750062 [I 2022-10-11 12:44:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 1160/2304 [I 2022-10-11 12:44:48] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:16:18 [I 2022-10-11 12:44:48] ...trap/engines/engine.py.229: process: 0.86294 | load: 0.00312 [I 2022-10-11 12:44:48] ...trap/engines/engine.py.230: loss: 0.09330 [I 2022-10-11 12:44:48] ...optimizers/trijoint.py.151: total_norm: 9.602767 [I 2022-10-11 12:44:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 1170/2304 [I 2022-10-11 12:44:57] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:38 [I 2022-10-11 12:44:57] ...trap/engines/engine.py.229: process: 0.88599 | load: 0.00394 [I 2022-10-11 12:44:57] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-11 12:44:57] ...optimizers/trijoint.py.151: total_norm: 11.383739 [I 2022-10-11 12:45:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 1180/2304 [I 2022-10-11 12:45:06] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:39 [I 2022-10-11 12:45:06] ...trap/engines/engine.py.229: process: 0.90543 | load: 0.00399 [I 2022-10-11 12:45:06] ...trap/engines/engine.py.230: loss: 0.08556 [I 2022-10-11 12:45:06] ...optimizers/trijoint.py.151: total_norm: 10.468880 [I 2022-10-11 12:45:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1190/2304 [I 2022-10-11 12:45:15] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:16:10 [I 2022-10-11 12:45:15] ...trap/engines/engine.py.229: process: 0.85035 | load: 0.00315 [I 2022-10-11 12:45:15] ...trap/engines/engine.py.230: loss: 0.13374 [I 2022-10-11 12:45:15] ...optimizers/trijoint.py.151: total_norm: 23.919609 [I 2022-10-11 12:45:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 1200/2304 [I 2022-10-11 12:45:23] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:44 [I 2022-10-11 12:45:23] ...trap/engines/engine.py.229: process: 0.86227 | load: 0.00313 [I 2022-10-11 12:45:23] ...trap/engines/engine.py.230: loss: 0.09865 [I 2022-10-11 12:45:23] ...optimizers/trijoint.py.151: total_norm: 9.052396 [I 2022-10-11 12:45:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1210/2304 [I 2022-10-11 12:45:32] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:36 [I 2022-10-11 12:45:32] ...trap/engines/engine.py.229: process: 0.86778 | load: 0.00316 [I 2022-10-11 12:45:32] ...trap/engines/engine.py.230: loss: 0.11591 [I 2022-10-11 12:45:32] ...optimizers/trijoint.py.151: total_norm: 12.393826 [I 2022-10-11 12:45:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1220/2304 [I 2022-10-11 12:45:41] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:39 [I 2022-10-11 12:45:41] ...trap/engines/engine.py.229: process: 0.88898 | load: 0.00404 [I 2022-10-11 12:45:41] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-11 12:45:41] ...optimizers/trijoint.py.151: total_norm: 15.182612 [I 2022-10-11 12:45:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 1230/2304 [I 2022-10-11 12:45:49] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:41 [I 2022-10-11 12:45:49] ...trap/engines/engine.py.229: process: 0.86055 | load: 0.00378 [I 2022-10-11 12:45:49] ...trap/engines/engine.py.230: loss: 0.09669 [I 2022-10-11 12:45:49] ...optimizers/trijoint.py.151: total_norm: 11.715117 [I 2022-10-11 12:45:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1240/2304 [I 2022-10-11 12:45:58] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:15:27 [I 2022-10-11 12:45:58] ...trap/engines/engine.py.229: process: 0.87535 | load: 0.00385 [I 2022-10-11 12:45:58] ...trap/engines/engine.py.230: loss: 0.10345 [I 2022-10-11 12:45:58] ...optimizers/trijoint.py.151: total_norm: 10.593393 [I 2022-10-11 12:46:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1250/2304 [I 2022-10-11 12:46:07] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:14:59 [I 2022-10-11 12:46:07] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00299 [I 2022-10-11 12:46:07] ...trap/engines/engine.py.230: loss: 0.10792 [I 2022-10-11 12:46:07] ...optimizers/trijoint.py.151: total_norm: 9.891351 [I 2022-10-11 12:46:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1260/2304 [I 2022-10-11 12:46:15] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:14:42 [I 2022-10-11 12:46:15] ...trap/engines/engine.py.229: process: 0.84605 | load: 0.00300 [I 2022-10-11 12:46:15] ...trap/engines/engine.py.230: loss: 0.08619 [I 2022-10-11 12:46:15] ...optimizers/trijoint.py.151: total_norm: 23.345369 [I 2022-10-11 12:46:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1270/2304 [I 2022-10-11 12:46:24] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:33 [I 2022-10-11 12:46:24] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00298 [I 2022-10-11 12:46:24] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-11 12:46:24] ...optimizers/trijoint.py.151: total_norm: 29.929016 [I 2022-10-11 12:46:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1280/2304 [I 2022-10-11 12:46:32] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:31 [I 2022-10-11 12:46:32] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00298 [I 2022-10-11 12:46:32] ...trap/engines/engine.py.230: loss: 0.11369 [I 2022-10-11 12:46:32] ...optimizers/trijoint.py.151: total_norm: 13.064014 [I 2022-10-11 12:46:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1290/2304 [I 2022-10-11 12:46:41] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:22 [I 2022-10-11 12:46:41] ...trap/engines/engine.py.229: process: 0.84132 | load: 0.00298 [I 2022-10-11 12:46:41] ...trap/engines/engine.py.230: loss: 0.09494 [I 2022-10-11 12:46:41] ...optimizers/trijoint.py.151: total_norm: 10.157145 [I 2022-10-11 12:46:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 1300/2304 [I 2022-10-11 12:46:49] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:18 [I 2022-10-11 12:46:49] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00298 [I 2022-10-11 12:46:49] ...trap/engines/engine.py.230: loss: 0.08460 [I 2022-10-11 12:46:49] ...optimizers/trijoint.py.151: total_norm: 19.265318 [I 2022-10-11 12:46:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1310/2304 [I 2022-10-11 12:46:58] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:06 [I 2022-10-11 12:46:58] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00298 [I 2022-10-11 12:46:58] ...trap/engines/engine.py.230: loss: 0.14287 [I 2022-10-11 12:46:58] ...optimizers/trijoint.py.151: total_norm: 18.188854 [I 2022-10-11 12:47:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1320/2304 [I 2022-10-11 12:47:07] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:13:50 [I 2022-10-11 12:47:07] ...trap/engines/engine.py.229: process: 0.82816 | load: 0.00301 [I 2022-10-11 12:47:07] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-11 12:47:07] ...optimizers/trijoint.py.151: total_norm: 23.868814 [I 2022-10-11 12:47:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1330/2304 [I 2022-10-11 12:47:15] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:13:40 [I 2022-10-11 12:47:15] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00302 [I 2022-10-11 12:47:15] ...trap/engines/engine.py.230: loss: 0.12036 [I 2022-10-11 12:47:15] ...optimizers/trijoint.py.151: total_norm: 11.792206 [I 2022-10-11 12:47:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 1340/2304 [I 2022-10-11 12:47:23] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:32 [I 2022-10-11 12:47:23] ...trap/engines/engine.py.229: process: 0.84626 | load: 0.00300 [I 2022-10-11 12:47:23] ...trap/engines/engine.py.230: loss: 0.12812 [I 2022-10-11 12:47:23] ...optimizers/trijoint.py.151: total_norm: 10.217132 [I 2022-10-11 12:47:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1350/2304 [I 2022-10-11 12:47:32] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:13:27 [I 2022-10-11 12:47:32] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00310 [I 2022-10-11 12:47:32] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-11 12:47:32] ...optimizers/trijoint.py.151: total_norm: 14.618032 [I 2022-10-11 12:47:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 1360/2304 [I 2022-10-11 12:47:40] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:23 [I 2022-10-11 12:47:40] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00314 [I 2022-10-11 12:47:40] ...trap/engines/engine.py.230: loss: 0.10049 [I 2022-10-11 12:47:40] ...optimizers/trijoint.py.151: total_norm: 11.860374 [I 2022-10-11 12:47:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 1370/2304 [I 2022-10-11 12:47:49] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:22 [I 2022-10-11 12:47:49] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00312 [I 2022-10-11 12:47:49] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-11 12:47:49] ...optimizers/trijoint.py.151: total_norm: 14.654919 [I 2022-10-11 12:47:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1380/2304 [I 2022-10-11 12:47:58] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:09 [I 2022-10-11 12:47:58] ...trap/engines/engine.py.229: process: 0.86397 | load: 0.00312 [I 2022-10-11 12:47:58] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-11 12:47:58] ...optimizers/trijoint.py.151: total_norm: 11.747147 [I 2022-10-11 12:48:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 1390/2304 [I 2022-10-11 12:48:06] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:13:04 [I 2022-10-11 12:48:06] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00313 [I 2022-10-11 12:48:06] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-11 12:48:06] ...optimizers/trijoint.py.151: total_norm: 16.809601 [I 2022-10-11 12:48:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1400/2304 [I 2022-10-11 12:48:15] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:13:00 [I 2022-10-11 12:48:15] ...trap/engines/engine.py.229: process: 0.86692 | load: 0.00313 [I 2022-10-11 12:48:15] ...trap/engines/engine.py.230: loss: 0.11952 [I 2022-10-11 12:48:15] ...optimizers/trijoint.py.151: total_norm: 12.766108 [I 2022-10-11 12:48:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 1410/2304 [I 2022-10-11 12:48:23] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:54 [I 2022-10-11 12:48:23] ...trap/engines/engine.py.229: process: 0.87470 | load: 0.00315 [I 2022-10-11 12:48:23] ...trap/engines/engine.py.230: loss: 0.10021 [I 2022-10-11 12:48:23] ...optimizers/trijoint.py.151: total_norm: 23.200132 [I 2022-10-11 12:48:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1420/2304 [I 2022-10-11 12:48:32] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:12:46 [I 2022-10-11 12:48:32] ...trap/engines/engine.py.229: process: 0.87236 | load: 0.00313 [I 2022-10-11 12:48:32] ...trap/engines/engine.py.230: loss: 0.12753 [I 2022-10-11 12:48:32] ...optimizers/trijoint.py.151: total_norm: 12.092923 [I 2022-10-11 12:48:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1430/2304 [I 2022-10-11 12:48:41] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:12:36 [I 2022-10-11 12:48:41] ...trap/engines/engine.py.229: process: 0.86647 | load: 0.00317 [I 2022-10-11 12:48:41] ...trap/engines/engine.py.230: loss: 0.11196 [I 2022-10-11 12:48:41] ...optimizers/trijoint.py.151: total_norm: 8.706150 [I 2022-10-11 12:48:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 1440/2304 [I 2022-10-11 12:48:49] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:21 [I 2022-10-11 12:48:49] ...trap/engines/engine.py.229: process: 0.84857 | load: 0.00315 [I 2022-10-11 12:48:49] ...trap/engines/engine.py.230: loss: 0.09391 [I 2022-10-11 12:48:49] ...optimizers/trijoint.py.151: total_norm: 18.023203 [I 2022-10-11 12:48:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1450/2304 [I 2022-10-11 12:48:58] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:12:31 [I 2022-10-11 12:48:58] ...trap/engines/engine.py.229: process: 0.88422 | load: 0.00395 [I 2022-10-11 12:48:58] ...trap/engines/engine.py.230: loss: 0.09173 [I 2022-10-11 12:48:58] ...optimizers/trijoint.py.151: total_norm: 10.131031 [I 2022-10-11 12:49:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1460/2304 [I 2022-10-11 12:49:07] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:12:20 [I 2022-10-11 12:49:07] ...trap/engines/engine.py.229: process: 0.86645 | load: 0.00314 [I 2022-10-11 12:49:07] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-11 12:49:07] ...optimizers/trijoint.py.151: total_norm: 14.482514 [I 2022-10-11 12:49:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 1470/2304 [I 2022-10-11 12:49:16] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:11:55 [I 2022-10-11 12:49:16] ...trap/engines/engine.py.229: process: 0.84871 | load: 0.00311 [I 2022-10-11 12:49:16] ...trap/engines/engine.py.230: loss: 0.09410 [I 2022-10-11 12:49:16] ...optimizers/trijoint.py.151: total_norm: 13.367551 [I 2022-10-11 12:49:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1480/2304 [I 2022-10-11 12:49:24] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:11:42 [I 2022-10-11 12:49:24] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00314 [I 2022-10-11 12:49:24] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-11 12:49:24] ...optimizers/trijoint.py.151: total_norm: 10.330571 [I 2022-10-11 12:49:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1490/2304 [I 2022-10-11 12:49:33] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:30 [I 2022-10-11 12:49:33] ...trap/engines/engine.py.229: process: 0.84086 | load: 0.00311 [I 2022-10-11 12:49:33] ...trap/engines/engine.py.230: loss: 0.09114 [I 2022-10-11 12:49:33] ...optimizers/trijoint.py.151: total_norm: 10.503631 [I 2022-10-11 12:49:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1500/2304 [I 2022-10-11 12:49:41] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:23 [I 2022-10-11 12:49:41] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00313 [I 2022-10-11 12:49:41] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-11 12:49:41] ...optimizers/trijoint.py.151: total_norm: 15.990883 [I 2022-10-11 12:49:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1510/2304 [I 2022-10-11 12:49:50] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:19 [I 2022-10-11 12:49:50] ...trap/engines/engine.py.229: process: 0.85483 | load: 0.00314 [I 2022-10-11 12:49:50] ...trap/engines/engine.py.230: loss: 0.18165 [I 2022-10-11 12:49:50] ...optimizers/trijoint.py.151: total_norm: 38.979214 [I 2022-10-11 12:49:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1520/2304 [I 2022-10-11 12:49:58] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:10 [I 2022-10-11 12:49:58] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00357 [I 2022-10-11 12:49:58] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-11 12:49:58] ...optimizers/trijoint.py.151: total_norm: 12.089170 [I 2022-10-11 12:50:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1530/2304 [I 2022-10-11 12:50:07] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:10:56 [I 2022-10-11 12:50:07] ...trap/engines/engine.py.229: process: 0.82767 | load: 0.00315 [I 2022-10-11 12:50:07] ...trap/engines/engine.py.230: loss: 0.08115 [I 2022-10-11 12:50:07] ...optimizers/trijoint.py.151: total_norm: 12.322337 [I 2022-10-11 12:50:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1540/2304 [I 2022-10-11 12:50:15] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:10:50 [I 2022-10-11 12:50:15] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00317 [I 2022-10-11 12:50:15] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-11 12:50:15] ...optimizers/trijoint.py.151: total_norm: 13.613894 [I 2022-10-11 12:50:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1550/2304 [I 2022-10-11 12:50:24] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:10:46 [I 2022-10-11 12:50:24] ...trap/engines/engine.py.229: process: 0.84998 | load: 0.00313 [I 2022-10-11 12:50:24] ...trap/engines/engine.py.230: loss: 0.12499 [I 2022-10-11 12:50:24] ...optimizers/trijoint.py.151: total_norm: 14.863338 [I 2022-10-11 12:50:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1560/2304 [I 2022-10-11 12:50:33] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:39 [I 2022-10-11 12:50:33] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00314 [I 2022-10-11 12:50:33] ...trap/engines/engine.py.230: loss: 0.09688 [I 2022-10-11 12:50:33] ...optimizers/trijoint.py.151: total_norm: 9.142553 [I 2022-10-11 12:50:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1570/2304 [I 2022-10-11 12:50:41] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:29 [I 2022-10-11 12:50:41] ...trap/engines/engine.py.229: process: 0.86723 | load: 0.00311 [I 2022-10-11 12:50:41] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-11 12:50:41] ...optimizers/trijoint.py.151: total_norm: 19.362448 [I 2022-10-11 12:50:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1580/2304 [I 2022-10-11 12:50:50] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:23 [I 2022-10-11 12:50:50] ...trap/engines/engine.py.229: process: 0.85651 | load: 0.00314 [I 2022-10-11 12:50:50] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 12:50:50] ...optimizers/trijoint.py.151: total_norm: 9.158257 [I 2022-10-11 12:50:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1590/2304 [I 2022-10-11 12:50:58] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:11 [I 2022-10-11 12:50:58] ...trap/engines/engine.py.229: process: 0.85992 | load: 0.00413 [I 2022-10-11 12:50:58] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-11 12:50:58] ...optimizers/trijoint.py.151: total_norm: 20.187027 [I 2022-10-11 12:51:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1600/2304 [I 2022-10-11 12:51:07] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:04 [I 2022-10-11 12:51:07] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00314 [I 2022-10-11 12:51:07] ...trap/engines/engine.py.230: loss: 0.16053 [I 2022-10-11 12:51:07] ...optimizers/trijoint.py.151: total_norm: 21.077085 [I 2022-10-11 12:51:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 1610/2304 [I 2022-10-11 12:51:16] ...trap/engines/engine.py.228: elapsed: 0:23:03 | left: 0:09:50 [I 2022-10-11 12:51:16] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00315 [I 2022-10-11 12:51:16] ...trap/engines/engine.py.230: loss: 0.12838 [I 2022-10-11 12:51:16] ...optimizers/trijoint.py.151: total_norm: 14.049956 [I 2022-10-11 12:51:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1620/2304 [I 2022-10-11 12:51:24] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:47 [I 2022-10-11 12:51:24] ...trap/engines/engine.py.229: process: 0.87264 | load: 0.00314 [I 2022-10-11 12:51:24] ...trap/engines/engine.py.230: loss: 0.15965 [I 2022-10-11 12:51:24] ...optimizers/trijoint.py.151: total_norm: 26.430391 [I 2022-10-11 12:51:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1630/2304 [I 2022-10-11 12:51:33] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:36 [I 2022-10-11 12:51:33] ...trap/engines/engine.py.229: process: 0.86038 | load: 0.00301 [I 2022-10-11 12:51:33] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-11 12:51:33] ...optimizers/trijoint.py.151: total_norm: 14.795379 [I 2022-10-11 12:51:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1640/2304 [I 2022-10-11 12:51:41] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:28 [I 2022-10-11 12:51:41] ...trap/engines/engine.py.229: process: 0.84393 | load: 0.00320 [I 2022-10-11 12:51:41] ...trap/engines/engine.py.230: loss: 0.15319 [I 2022-10-11 12:51:41] ...optimizers/trijoint.py.151: total_norm: 15.469168 [I 2022-10-11 12:51:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1650/2304 [I 2022-10-11 12:51:50] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:18 [I 2022-10-11 12:51:50] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00294 [I 2022-10-11 12:51:50] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-11 12:51:50] ...optimizers/trijoint.py.151: total_norm: 11.024527 [I 2022-10-11 12:51:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1660/2304 [I 2022-10-11 12:51:58] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:10 [I 2022-10-11 12:51:58] ...trap/engines/engine.py.229: process: 0.86259 | load: 0.00294 [I 2022-10-11 12:51:58] ...trap/engines/engine.py.230: loss: 0.10213 [I 2022-10-11 12:51:58] ...optimizers/trijoint.py.151: total_norm: 67.662590 [I 2022-10-11 12:52:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1670/2304 [I 2022-10-11 12:52:07] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:08:59 [I 2022-10-11 12:52:07] ...trap/engines/engine.py.229: process: 0.84555 | load: 0.00295 [I 2022-10-11 12:52:07] ...trap/engines/engine.py.230: loss: 0.09526 [I 2022-10-11 12:52:07] ...optimizers/trijoint.py.151: total_norm: 13.239856 [I 2022-10-11 12:52:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1680/2304 [I 2022-10-11 12:52:15] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:08:49 [I 2022-10-11 12:52:15] ...trap/engines/engine.py.229: process: 0.85588 | load: 0.00294 [I 2022-10-11 12:52:15] ...trap/engines/engine.py.230: loss: 0.15621 [I 2022-10-11 12:52:15] ...optimizers/trijoint.py.151: total_norm: 35.656380 [I 2022-10-11 12:52:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1690/2304 [I 2022-10-11 12:52:24] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:08:41 [I 2022-10-11 12:52:24] ...trap/engines/engine.py.229: process: 0.84582 | load: 0.00295 [I 2022-10-11 12:52:24] ...trap/engines/engine.py.230: loss: 0.11629 [I 2022-10-11 12:52:24] ...optimizers/trijoint.py.151: total_norm: 14.403650 [I 2022-10-11 12:52:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1700/2304 [I 2022-10-11 12:52:33] ...trap/engines/engine.py.228: elapsed: 0:24:19 | left: 0:08:35 [I 2022-10-11 12:52:33] ...trap/engines/engine.py.229: process: 0.85186 | load: 0.00296 [I 2022-10-11 12:52:33] ...trap/engines/engine.py.230: loss: 0.14746 [I 2022-10-11 12:52:33] ...optimizers/trijoint.py.151: total_norm: 64.942436 [I 2022-10-11 12:52:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1710/2304 [I 2022-10-11 12:52:41] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:29 [I 2022-10-11 12:52:41] ...trap/engines/engine.py.229: process: 0.85803 | load: 0.00299 [I 2022-10-11 12:52:41] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-11 12:52:41] ...optimizers/trijoint.py.151: total_norm: 11.316982 [I 2022-10-11 12:52:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1720/2304 [I 2022-10-11 12:52:50] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:17 [I 2022-10-11 12:52:50] ...trap/engines/engine.py.229: process: 0.85801 | load: 0.00302 [I 2022-10-11 12:52:50] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-11 12:52:50] ...optimizers/trijoint.py.151: total_norm: 37.404961 [I 2022-10-11 12:52:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1730/2304 [I 2022-10-11 12:52:58] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:10 [I 2022-10-11 12:52:58] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00303 [I 2022-10-11 12:52:58] ...trap/engines/engine.py.230: loss: 0.14728 [I 2022-10-11 12:52:58] ...optimizers/trijoint.py.151: total_norm: 26.602463 [I 2022-10-11 12:53:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1740/2304 [I 2022-10-11 12:53:07] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:07:59 [I 2022-10-11 12:53:07] ...trap/engines/engine.py.229: process: 0.86393 | load: 0.00298 [I 2022-10-11 12:53:07] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-11 12:53:07] ...optimizers/trijoint.py.151: total_norm: 15.639156 [I 2022-10-11 12:53:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1750/2304 [I 2022-10-11 12:53:15] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:07:52 [I 2022-10-11 12:53:15] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00299 [I 2022-10-11 12:53:15] ...trap/engines/engine.py.230: loss: 0.10920 [I 2022-10-11 12:53:15] ...optimizers/trijoint.py.151: total_norm: 23.752558 [I 2022-10-11 12:53:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1760/2304 [I 2022-10-11 12:53:24] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:47 [I 2022-10-11 12:53:24] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00298 [I 2022-10-11 12:53:24] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-11 12:53:24] ...optimizers/trijoint.py.151: total_norm: 15.298949 [I 2022-10-11 12:53:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1770/2304 [I 2022-10-11 12:53:32] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:37 [I 2022-10-11 12:53:32] ...trap/engines/engine.py.229: process: 0.85926 | load: 0.00297 [I 2022-10-11 12:53:32] ...trap/engines/engine.py.230: loss: 0.13367 [I 2022-10-11 12:53:33] ...optimizers/trijoint.py.151: total_norm: 14.642221 [I 2022-10-11 12:53:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1780/2304 [I 2022-10-11 12:53:41] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:27 [I 2022-10-11 12:53:41] ...trap/engines/engine.py.229: process: 0.86156 | load: 0.00302 [I 2022-10-11 12:53:41] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-11 12:53:41] ...optimizers/trijoint.py.151: total_norm: 16.379360 [I 2022-10-11 12:53:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1790/2304 [I 2022-10-11 12:53:50] ...trap/engines/engine.py.228: elapsed: 0:25:36 | left: 0:07:16 [I 2022-10-11 12:53:50] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00300 [I 2022-10-11 12:53:50] ...trap/engines/engine.py.230: loss: 0.09518 [I 2022-10-11 12:53:50] ...optimizers/trijoint.py.151: total_norm: 13.705711 [I 2022-10-11 12:53:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1800/2304 [I 2022-10-11 12:53:58] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:14 [I 2022-10-11 12:53:58] ...trap/engines/engine.py.229: process: 0.87143 | load: 0.00311 [I 2022-10-11 12:53:58] ...trap/engines/engine.py.230: loss: 0.20733 [I 2022-10-11 12:53:58] ...optimizers/trijoint.py.151: total_norm: 26.176796 [I 2022-10-11 12:54:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1810/2304 [I 2022-10-11 12:54:07] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:06 [I 2022-10-11 12:54:07] ...trap/engines/engine.py.229: process: 0.84440 | load: 0.00313 [I 2022-10-11 12:54:07] ...trap/engines/engine.py.230: loss: 0.09124 [I 2022-10-11 12:54:07] ...optimizers/trijoint.py.151: total_norm: 7.310332 [I 2022-10-11 12:54:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 1820/2304 [I 2022-10-11 12:54:16] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:07:05 [I 2022-10-11 12:54:16] ...trap/engines/engine.py.229: process: 0.89667 | load: 0.00392 [I 2022-10-11 12:54:16] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-11 12:54:16] ...optimizers/trijoint.py.151: total_norm: 11.911591 [I 2022-10-11 12:54:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1830/2304 [I 2022-10-11 12:54:24] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:57 [I 2022-10-11 12:54:24] ...trap/engines/engine.py.229: process: 0.88427 | load: 0.00393 [I 2022-10-11 12:54:24] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-11 12:54:24] ...optimizers/trijoint.py.151: total_norm: 14.124972 [I 2022-10-11 12:54:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1840/2304 [I 2022-10-11 12:54:33] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:53 [I 2022-10-11 12:54:33] ...trap/engines/engine.py.229: process: 0.89527 | load: 0.00396 [I 2022-10-11 12:54:33] ...trap/engines/engine.py.230: loss: 0.09632 [I 2022-10-11 12:54:33] ...optimizers/trijoint.py.151: total_norm: 9.177050 [I 2022-10-11 12:54:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 1850/2304 [I 2022-10-11 12:54:42] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:41 [I 2022-10-11 12:54:42] ...trap/engines/engine.py.229: process: 0.89046 | load: 0.00396 [I 2022-10-11 12:54:42] ...trap/engines/engine.py.230: loss: 0.19839 [I 2022-10-11 12:54:42] ...optimizers/trijoint.py.151: total_norm: 25.174200 [I 2022-10-11 12:54:51] ...trap/engines/engine.py.224: train: epoch 105 | batch 1860/2304 [I 2022-10-11 12:54:51] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:32 [I 2022-10-11 12:54:51] ...trap/engines/engine.py.229: process: 0.88116 | load: 0.00394 [I 2022-10-11 12:54:51] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-11 12:54:51] ...optimizers/trijoint.py.151: total_norm: 15.385266 [I 2022-10-11 12:55:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 1870/2304 [I 2022-10-11 12:55:00] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:24 [I 2022-10-11 12:55:00] ...trap/engines/engine.py.229: process: 0.88475 | load: 0.00398 [I 2022-10-11 12:55:00] ...trap/engines/engine.py.230: loss: 0.12531 [I 2022-10-11 12:55:00] ...optimizers/trijoint.py.151: total_norm: 10.812655 [I 2022-10-11 12:55:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 1880/2304 [I 2022-10-11 12:55:09] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:15 [I 2022-10-11 12:55:09] ...trap/engines/engine.py.229: process: 0.88856 | load: 0.00395 [I 2022-10-11 12:55:09] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-11 12:55:09] ...optimizers/trijoint.py.151: total_norm: 11.246422 [I 2022-10-11 12:55:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 1890/2304 [I 2022-10-11 12:55:18] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:06 [I 2022-10-11 12:55:18] ...trap/engines/engine.py.229: process: 0.88648 | load: 0.00397 [I 2022-10-11 12:55:18] ...trap/engines/engine.py.230: loss: 0.11953 [I 2022-10-11 12:55:18] ...optimizers/trijoint.py.151: total_norm: 9.789225 [I 2022-10-11 12:55:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 1900/2304 [I 2022-10-11 12:55:27] ...trap/engines/engine.py.228: elapsed: 0:27:14 | left: 0:05:57 [I 2022-10-11 12:55:27] ...trap/engines/engine.py.229: process: 0.89048 | load: 0.00396 [I 2022-10-11 12:55:27] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-11 12:55:27] ...optimizers/trijoint.py.151: total_norm: 11.317479 [I 2022-10-11 12:55:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 1910/2304 [I 2022-10-11 12:55:36] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:50 [I 2022-10-11 12:55:36] ...trap/engines/engine.py.229: process: 0.89559 | load: 0.00396 [I 2022-10-11 12:55:36] ...trap/engines/engine.py.230: loss: 0.07985 [I 2022-10-11 12:55:36] ...optimizers/trijoint.py.151: total_norm: 15.104571 [I 2022-10-11 12:55:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 1920/2304 [I 2022-10-11 12:55:44] ...trap/engines/engine.py.228: elapsed: 0:27:31 | left: 0:05:38 [I 2022-10-11 12:55:44] ...trap/engines/engine.py.229: process: 0.85647 | load: 0.00397 [I 2022-10-11 12:55:44] ...trap/engines/engine.py.230: loss: 0.13915 [I 2022-10-11 12:55:44] ...optimizers/trijoint.py.151: total_norm: 12.711228 [I 2022-10-11 12:55:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 1930/2304 [I 2022-10-11 12:55:53] ...trap/engines/engine.py.228: elapsed: 0:27:40 | left: 0:05:22 [I 2022-10-11 12:55:53] ...trap/engines/engine.py.229: process: 0.85512 | load: 0.00313 [I 2022-10-11 12:55:53] ...trap/engines/engine.py.230: loss: 0.13979 [I 2022-10-11 12:55:53] ...optimizers/trijoint.py.151: total_norm: 19.724413 [I 2022-10-11 12:56:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 1940/2304 [I 2022-10-11 12:56:02] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:05:11 [I 2022-10-11 12:56:02] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00313 [I 2022-10-11 12:56:02] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-11 12:56:02] ...optimizers/trijoint.py.151: total_norm: 11.522834 [I 2022-10-11 12:56:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 1950/2304 [I 2022-10-11 12:56:10] ...trap/engines/engine.py.228: elapsed: 0:27:57 | left: 0:05:03 [I 2022-10-11 12:56:10] ...trap/engines/engine.py.229: process: 0.85971 | load: 0.00315 [I 2022-10-11 12:56:10] ...trap/engines/engine.py.230: loss: 0.08829 [I 2022-10-11 12:56:10] ...optimizers/trijoint.py.151: total_norm: 24.785769 [I 2022-10-11 12:56:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 1960/2304 [I 2022-10-11 12:56:19] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:50 [I 2022-10-11 12:56:19] ...trap/engines/engine.py.229: process: 0.83489 | load: 0.00314 [I 2022-10-11 12:56:19] ...trap/engines/engine.py.230: loss: 0.09640 [I 2022-10-11 12:56:19] ...optimizers/trijoint.py.151: total_norm: 13.484959 [I 2022-10-11 12:56:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 1970/2304 [I 2022-10-11 12:56:27] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:04:43 [I 2022-10-11 12:56:27] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00313 [I 2022-10-11 12:56:27] ...trap/engines/engine.py.230: loss: 0.11898 [I 2022-10-11 12:56:27] ...optimizers/trijoint.py.151: total_norm: 14.927083 [I 2022-10-11 12:56:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 1980/2304 [I 2022-10-11 12:56:36] ...trap/engines/engine.py.228: elapsed: 0:28:23 | left: 0:04:38 [I 2022-10-11 12:56:36] ...trap/engines/engine.py.229: process: 0.86410 | load: 0.00312 [I 2022-10-11 12:56:36] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-11 12:56:36] ...optimizers/trijoint.py.151: total_norm: 28.297766 [I 2022-10-11 12:56:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 1990/2304 [I 2022-10-11 12:56:44] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:32 [I 2022-10-11 12:56:44] ...trap/engines/engine.py.229: process: 0.87361 | load: 0.00314 [I 2022-10-11 12:56:44] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-11 12:56:44] ...optimizers/trijoint.py.151: total_norm: 76.421181 [I 2022-10-11 12:56:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2000/2304 [I 2022-10-11 12:56:53] ...trap/engines/engine.py.228: elapsed: 0:28:40 | left: 0:04:22 [I 2022-10-11 12:56:53] ...trap/engines/engine.py.229: process: 0.87813 | load: 0.00316 [I 2022-10-11 12:56:53] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-11 12:56:53] ...optimizers/trijoint.py.151: total_norm: 10.433661 [I 2022-10-11 12:57:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 2010/2304 [I 2022-10-11 12:57:02] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:13 [I 2022-10-11 12:57:02] ...trap/engines/engine.py.229: process: 0.86297 | load: 0.00314 [I 2022-10-11 12:57:02] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-11 12:57:02] ...optimizers/trijoint.py.151: total_norm: 20.955585 [I 2022-10-11 12:57:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 2020/2304 [I 2022-10-11 12:57:10] ...trap/engines/engine.py.228: elapsed: 0:28:57 | left: 0:04:02 [I 2022-10-11 12:57:10] ...trap/engines/engine.py.229: process: 0.86723 | load: 0.00314 [I 2022-10-11 12:57:10] ...trap/engines/engine.py.230: loss: 0.11284 [I 2022-10-11 12:57:10] ...optimizers/trijoint.py.151: total_norm: 20.334526 [I 2022-10-11 12:57:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 2030/2304 [I 2022-10-11 12:57:19] ...trap/engines/engine.py.228: elapsed: 0:29:06 | left: 0:03:55 [I 2022-10-11 12:57:19] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00315 [I 2022-10-11 12:57:19] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-11 12:57:19] ...optimizers/trijoint.py.151: total_norm: 11.771186 [I 2022-10-11 12:57:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 2040/2304 [I 2022-10-11 12:57:27] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:47 [I 2022-10-11 12:57:27] ...trap/engines/engine.py.229: process: 0.86058 | load: 0.00315 [I 2022-10-11 12:57:27] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-11 12:57:27] ...optimizers/trijoint.py.151: total_norm: 60.083740 [I 2022-10-11 12:57:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 2050/2304 [I 2022-10-11 12:57:36] ...trap/engines/engine.py.228: elapsed: 0:29:23 | left: 0:03:39 [I 2022-10-11 12:57:36] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00316 [I 2022-10-11 12:57:36] ...trap/engines/engine.py.230: loss: 0.09338 [I 2022-10-11 12:57:36] ...optimizers/trijoint.py.151: total_norm: 8.378293 [I 2022-10-11 12:57:45] ...trap/engines/engine.py.224: train: epoch 105 | batch 2060/2304 [I 2022-10-11 12:57:45] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:31 [I 2022-10-11 12:57:45] ...trap/engines/engine.py.229: process: 0.87432 | load: 0.00312 [I 2022-10-11 12:57:45] ...trap/engines/engine.py.230: loss: 0.16078 [I 2022-10-11 12:57:45] ...optimizers/trijoint.py.151: total_norm: 16.069002 [I 2022-10-11 12:57:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2070/2304 [I 2022-10-11 12:57:53] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:03:21 [I 2022-10-11 12:57:53] ...trap/engines/engine.py.229: process: 0.86031 | load: 0.00312 [I 2022-10-11 12:57:53] ...trap/engines/engine.py.230: loss: 0.08303 [I 2022-10-11 12:57:53] ...optimizers/trijoint.py.151: total_norm: 14.054561 [I 2022-10-11 12:58:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 2080/2304 [I 2022-10-11 12:58:02] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:10 [I 2022-10-11 12:58:02] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00297 [I 2022-10-11 12:58:02] ...trap/engines/engine.py.230: loss: 0.08242 [I 2022-10-11 12:58:02] ...optimizers/trijoint.py.151: total_norm: 11.498049 [I 2022-10-11 12:58:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 2090/2304 [I 2022-10-11 12:58:10] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:02 [I 2022-10-11 12:58:10] ...trap/engines/engine.py.229: process: 0.86180 | load: 0.00297 [I 2022-10-11 12:58:10] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-11 12:58:10] ...optimizers/trijoint.py.151: total_norm: 9.929678 [I 2022-10-11 12:58:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 2100/2304 [I 2022-10-11 12:58:19] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:02:53 [I 2022-10-11 12:58:19] ...trap/engines/engine.py.229: process: 0.86344 | load: 0.00297 [I 2022-10-11 12:58:19] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-11 12:58:19] ...optimizers/trijoint.py.151: total_norm: 12.010705 [I 2022-10-11 12:58:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 2110/2304 [I 2022-10-11 12:58:27] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:45 [I 2022-10-11 12:58:27] ...trap/engines/engine.py.229: process: 0.84078 | load: 0.00297 [I 2022-10-11 12:58:27] ...trap/engines/engine.py.230: loss: 0.10072 [I 2022-10-11 12:58:27] ...optimizers/trijoint.py.151: total_norm: 14.154546 [I 2022-10-11 12:58:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 2120/2304 [I 2022-10-11 12:58:36] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:02:36 [I 2022-10-11 12:58:36] ...trap/engines/engine.py.229: process: 0.85243 | load: 0.00294 [I 2022-10-11 12:58:36] ...trap/engines/engine.py.230: loss: 0.09158 [I 2022-10-11 12:58:36] ...optimizers/trijoint.py.151: total_norm: 16.080906 [I 2022-10-11 12:58:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 2130/2304 [I 2022-10-11 12:58:44] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:27 [I 2022-10-11 12:58:44] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00296 [I 2022-10-11 12:58:44] ...trap/engines/engine.py.230: loss: 0.10607 [I 2022-10-11 12:58:44] ...optimizers/trijoint.py.151: total_norm: 17.261679 [I 2022-10-11 12:58:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2140/2304 [I 2022-10-11 12:58:53] ...trap/engines/engine.py.228: elapsed: 0:30:40 | left: 0:02:18 [I 2022-10-11 12:58:53] ...trap/engines/engine.py.229: process: 0.84763 | load: 0.00298 [I 2022-10-11 12:58:53] ...trap/engines/engine.py.230: loss: 0.11924 [I 2022-10-11 12:58:53] ...optimizers/trijoint.py.151: total_norm: 13.745337 [I 2022-10-11 12:59:01] ...trap/engines/engine.py.224: train: epoch 105 | batch 2150/2304 [I 2022-10-11 12:59:01] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:10 [I 2022-10-11 12:59:01] ...trap/engines/engine.py.229: process: 0.85118 | load: 0.00295 [I 2022-10-11 12:59:01] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-11 12:59:01] ...optimizers/trijoint.py.151: total_norm: 22.120789 [I 2022-10-11 12:59:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 2160/2304 [I 2022-10-11 12:59:10] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:02 [I 2022-10-11 12:59:10] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00297 [I 2022-10-11 12:59:10] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-11 12:59:10] ...optimizers/trijoint.py.151: total_norm: 22.112398 [I 2022-10-11 12:59:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 2170/2304 [I 2022-10-11 12:59:18] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:01:54 [I 2022-10-11 12:59:18] ...trap/engines/engine.py.229: process: 0.86438 | load: 0.00298 [I 2022-10-11 12:59:18] ...trap/engines/engine.py.230: loss: 0.10710 [I 2022-10-11 12:59:18] ...optimizers/trijoint.py.151: total_norm: 12.223842 [I 2022-10-11 12:59:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 2180/2304 [I 2022-10-11 12:59:27] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:45 [I 2022-10-11 12:59:27] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00298 [I 2022-10-11 12:59:27] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-11 12:59:27] ...optimizers/trijoint.py.151: total_norm: 15.114133 [I 2022-10-11 12:59:35] ...trap/engines/engine.py.224: train: epoch 105 | batch 2190/2304 [I 2022-10-11 12:59:35] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:36 [I 2022-10-11 12:59:35] ...trap/engines/engine.py.229: process: 0.83627 | load: 0.00300 [I 2022-10-11 12:59:35] ...trap/engines/engine.py.230: loss: 0.09202 [I 2022-10-11 12:59:35] ...optimizers/trijoint.py.151: total_norm: 16.578054 [I 2022-10-11 12:59:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 2200/2304 [I 2022-10-11 12:59:44] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:28 [I 2022-10-11 12:59:44] ...trap/engines/engine.py.229: process: 0.83345 | load: 0.00303 [I 2022-10-11 12:59:44] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-11 12:59:44] ...optimizers/trijoint.py.151: total_norm: 36.425594 [I 2022-10-11 12:59:52] ...trap/engines/engine.py.224: train: epoch 105 | batch 2210/2304 [I 2022-10-11 12:59:52] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:19 [I 2022-10-11 12:59:52] ...trap/engines/engine.py.229: process: 0.86079 | load: 0.00298 [I 2022-10-11 12:59:52] ...trap/engines/engine.py.230: loss: 0.15759 [I 2022-10-11 12:59:52] ...optimizers/trijoint.py.151: total_norm: 13.238156 [I 2022-10-11 13:00:01] ...trap/engines/engine.py.224: train: epoch 105 | batch 2220/2304 [I 2022-10-11 13:00:01] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:13 [I 2022-10-11 13:00:01] ...trap/engines/engine.py.229: process: 0.87902 | load: 0.00378 [I 2022-10-11 13:00:01] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-11 13:00:01] ...optimizers/trijoint.py.151: total_norm: 13.811707 [I 2022-10-11 13:00:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 2230/2304 [I 2022-10-11 13:00:10] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:04 [I 2022-10-11 13:00:10] ...trap/engines/engine.py.229: process: 0.87913 | load: 0.00379 [I 2022-10-11 13:00:10] ...trap/engines/engine.py.230: loss: 0.08324 [I 2022-10-11 13:00:10] ...optimizers/trijoint.py.151: total_norm: 10.666613 [I 2022-10-11 13:00:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 2240/2304 [I 2022-10-11 13:00:19] ...trap/engines/engine.py.228: elapsed: 0:32:06 | left: 0:00:56 [I 2022-10-11 13:00:19] ...trap/engines/engine.py.229: process: 0.88981 | load: 0.00378 [I 2022-10-11 13:00:19] ...trap/engines/engine.py.230: loss: 0.12651 [I 2022-10-11 13:00:19] ...optimizers/trijoint.py.151: total_norm: 150.797165 [I 2022-10-11 13:00:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 2250/2304 [I 2022-10-11 13:00:27] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:46 [I 2022-10-11 13:00:27] ...trap/engines/engine.py.229: process: 0.85537 | load: 0.00300 [I 2022-10-11 13:00:27] ...trap/engines/engine.py.230: loss: 0.15497 [I 2022-10-11 13:00:27] ...optimizers/trijoint.py.151: total_norm: 14.901283 [I 2022-10-11 13:00:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 2260/2304 [I 2022-10-11 13:00:36] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:37 [I 2022-10-11 13:00:36] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00300 [I 2022-10-11 13:00:36] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-11 13:00:36] ...optimizers/trijoint.py.151: total_norm: 10.843677 [I 2022-10-11 13:00:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 2270/2304 [I 2022-10-11 13:00:44] ...trap/engines/engine.py.228: elapsed: 0:32:31 | left: 0:00:28 [I 2022-10-11 13:00:44] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00312 [I 2022-10-11 13:00:44] ...trap/engines/engine.py.230: loss: 0.14160 [I 2022-10-11 13:00:44] ...optimizers/trijoint.py.151: total_norm: 10.346652 [I 2022-10-11 13:00:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2280/2304 [I 2022-10-11 13:00:53] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:20 [I 2022-10-11 13:00:53] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00314 [I 2022-10-11 13:00:53] ...trap/engines/engine.py.230: loss: 0.10688 [I 2022-10-11 13:00:53] ...optimizers/trijoint.py.151: total_norm: 7.273320 [I 2022-10-11 13:01:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 2290/2304 [I 2022-10-11 13:01:02] ...trap/engines/engine.py.228: elapsed: 0:32:48 | left: 0:00:11 [I 2022-10-11 13:01:02] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00313 [I 2022-10-11 13:01:02] ...trap/engines/engine.py.230: loss: 0.09515 [I 2022-10-11 13:01:02] ...optimizers/trijoint.py.151: total_norm: 17.041727 [I 2022-10-11 13:01:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 2300/2304 [I 2022-10-11 13:01:10] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:03 [I 2022-10-11 13:01:10] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00019 [I 2022-10-11 13:01:10] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-11 13:01:10] ...optimizers/trijoint.py.151: total_norm: 18.914743 [I 2022-10-11 13:01:13] ...trap/engines/engine.py.224: train: epoch 105 | batch 2304/2304 [I 2022-10-11 13:01:13] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:00 [I 2022-10-11 13:01:13] ...trap/engines/engine.py.229: process: 0.81872 | load: 0.00018 [I 2022-10-11 13:01:13] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-11 13:01:13] ...optimizers/trijoint.py.151: total_norm: 19.954756 [S 2022-10-11 13:01:15] ...trap/engines/engine.py.129: train_epoch.epoch: 105 [S 2022-10-11 13:01:15] ...trap/engines/engine.py.129: train_epoch.loss: 0.12189019273909985 [I 2022-10-11 13:01:15] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 13:01:16] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 105 [I 2022-10-11 13:01:16] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 13:01:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 13:01:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 13:01:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 13:01:22] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 13:01:23] ...trap/engines/engine.py.321: eval: epoch 105 | batch 0/510 [I 2022-10-11 13:01:23] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:34 [I 2022-10-11 13:01:23] ...trap/engines/engine.py.326: process: 6.30227 | load: 5.79905 [I 2022-10-11 13:01:30] ...trap/engines/engine.py.321: eval: epoch 105 | batch 10/510 [I 2022-10-11 13:01:30] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:11:20 [I 2022-10-11 13:01:30] ...trap/engines/engine.py.326: process: 0.72776 | load: 0.00020 [I 2022-10-11 13:01:36] ...trap/engines/engine.py.321: eval: epoch 105 | batch 20/510 [I 2022-10-11 13:01:36] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:29 [I 2022-10-11 13:01:36] ...trap/engines/engine.py.326: process: 0.64283 | load: 0.00021 [I 2022-10-11 13:01:43] ...trap/engines/engine.py.321: eval: epoch 105 | batch 30/510 [I 2022-10-11 13:01:43] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:28 [I 2022-10-11 13:01:43] ...trap/engines/engine.py.326: process: 0.74908 | load: 0.00021 [I 2022-10-11 13:01:49] ...trap/engines/engine.py.321: eval: epoch 105 | batch 40/510 [I 2022-10-11 13:01:49] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:05:04 [I 2022-10-11 13:01:49] ...trap/engines/engine.py.326: process: 0.56920 | load: 0.00022 [I 2022-10-11 13:01:55] ...trap/engines/engine.py.321: eval: epoch 105 | batch 50/510 [I 2022-10-11 13:01:55] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:11 [I 2022-10-11 13:01:55] ...trap/engines/engine.py.326: process: 0.48822 | load: 0.00023 [I 2022-10-11 13:02:00] ...trap/engines/engine.py.321: eval: epoch 105 | batch 60/510 [I 2022-10-11 13:02:00] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:46 [I 2022-10-11 13:02:00] ...trap/engines/engine.py.326: process: 0.43643 | load: 0.00019 [I 2022-10-11 13:02:06] ...trap/engines/engine.py.321: eval: epoch 105 | batch 70/510 [I 2022-10-11 13:02:06] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:04:12 [I 2022-10-11 13:02:06] ...trap/engines/engine.py.326: process: 0.59576 | load: 0.00021 [I 2022-10-11 13:02:12] ...trap/engines/engine.py.321: eval: epoch 105 | batch 80/510 [I 2022-10-11 13:02:12] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:04:37 [I 2022-10-11 13:02:12] ...trap/engines/engine.py.326: process: 0.63939 | load: 0.00023 [I 2022-10-11 13:02:17] ...trap/engines/engine.py.321: eval: epoch 105 | batch 90/510 [I 2022-10-11 13:02:17] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:03:40 [I 2022-10-11 13:02:17] ...trap/engines/engine.py.326: process: 0.49665 | load: 0.00020 [I 2022-10-11 13:02:22] ...trap/engines/engine.py.321: eval: epoch 105 | batch 100/510 [I 2022-10-11 13:02:22] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:03:14 [I 2022-10-11 13:02:22] ...trap/engines/engine.py.326: process: 0.32754 | load: 0.00022 [I 2022-10-11 13:02:25] ...trap/engines/engine.py.321: eval: epoch 105 | batch 110/510 [I 2022-10-11 13:02:25] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:18 [I 2022-10-11 13:02:25] ...trap/engines/engine.py.326: process: 0.32968 | load: 0.00022 [I 2022-10-11 13:02:29] ...trap/engines/engine.py.321: eval: epoch 105 | batch 120/510 [I 2022-10-11 13:02:29] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:10 [I 2022-10-11 13:02:29] ...trap/engines/engine.py.326: process: 0.33187 | load: 0.00022 [I 2022-10-11 13:02:32] ...trap/engines/engine.py.321: eval: epoch 105 | batch 130/510 [I 2022-10-11 13:02:32] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:02:06 [I 2022-10-11 13:02:32] ...trap/engines/engine.py.326: process: 0.33312 | load: 0.00028 [I 2022-10-11 13:02:35] ...trap/engines/engine.py.321: eval: epoch 105 | batch 140/510 [I 2022-10-11 13:02:35] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:02:03 [I 2022-10-11 13:02:35] ...trap/engines/engine.py.326: process: 0.33647 | load: 0.00023 [I 2022-10-11 13:02:39] ...trap/engines/engine.py.321: eval: epoch 105 | batch 150/510 [I 2022-10-11 13:02:39] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:02:00 [I 2022-10-11 13:02:39] ...trap/engines/engine.py.326: process: 0.33840 | load: 0.00023 [I 2022-10-11 13:02:42] ...trap/engines/engine.py.321: eval: epoch 105 | batch 160/510 [I 2022-10-11 13:02:42] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:56 [I 2022-10-11 13:02:42] ...trap/engines/engine.py.326: process: 0.33570 | load: 0.00022 [I 2022-10-11 13:02:45] ...trap/engines/engine.py.321: eval: epoch 105 | batch 170/510 [I 2022-10-11 13:02:45] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:53 [I 2022-10-11 13:02:45] ...trap/engines/engine.py.326: process: 0.33321 | load: 0.00023 [I 2022-10-11 13:02:49] ...trap/engines/engine.py.321: eval: epoch 105 | batch 180/510 [I 2022-10-11 13:02:49] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:51 [I 2022-10-11 13:02:49] ...trap/engines/engine.py.326: process: 0.33903 | load: 0.00025 [I 2022-10-11 13:02:52] ...trap/engines/engine.py.321: eval: epoch 105 | batch 190/510 [I 2022-10-11 13:02:52] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:47 [I 2022-10-11 13:02:52] ...trap/engines/engine.py.326: process: 0.33354 | load: 0.00022 [I 2022-10-11 13:02:55] ...trap/engines/engine.py.321: eval: epoch 105 | batch 200/510 [I 2022-10-11 13:02:55] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:43 [I 2022-10-11 13:02:55] ...trap/engines/engine.py.326: process: 0.33228 | load: 0.00024 [I 2022-10-11 13:02:59] ...trap/engines/engine.py.321: eval: epoch 105 | batch 210/510 [I 2022-10-11 13:02:59] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:40 [I 2022-10-11 13:02:59] ...trap/engines/engine.py.326: process: 0.33448 | load: 0.00023 [I 2022-10-11 13:03:02] ...trap/engines/engine.py.321: eval: epoch 105 | batch 220/510 [I 2022-10-11 13:03:02] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:37 [I 2022-10-11 13:03:02] ...trap/engines/engine.py.326: process: 0.33750 | load: 0.00022 [I 2022-10-11 13:03:06] ...trap/engines/engine.py.321: eval: epoch 105 | batch 230/510 [I 2022-10-11 13:03:06] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:33 [I 2022-10-11 13:03:06] ...trap/engines/engine.py.326: process: 0.33709 | load: 0.00022 [I 2022-10-11 13:03:09] ...trap/engines/engine.py.321: eval: epoch 105 | batch 240/510 [I 2022-10-11 13:03:09] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:30 [I 2022-10-11 13:03:09] ...trap/engines/engine.py.326: process: 0.33677 | load: 0.00022 [I 2022-10-11 13:03:12] ...trap/engines/engine.py.321: eval: epoch 105 | batch 250/510 [I 2022-10-11 13:03:12] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:26 [I 2022-10-11 13:03:12] ...trap/engines/engine.py.326: process: 0.33288 | load: 0.00023 [I 2022-10-11 13:03:16] ...trap/engines/engine.py.321: eval: epoch 105 | batch 260/510 [I 2022-10-11 13:03:16] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:23 [I 2022-10-11 13:03:16] ...trap/engines/engine.py.326: process: 0.32872 | load: 0.00023 [I 2022-10-11 13:03:19] ...trap/engines/engine.py.321: eval: epoch 105 | batch 270/510 [I 2022-10-11 13:03:19] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:20 [I 2022-10-11 13:03:19] ...trap/engines/engine.py.326: process: 0.33323 | load: 0.00023 [I 2022-10-11 13:03:22] ...trap/engines/engine.py.321: eval: epoch 105 | batch 280/510 [I 2022-10-11 13:03:22] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:17 [I 2022-10-11 13:03:22] ...trap/engines/engine.py.326: process: 0.33373 | load: 0.00023 [I 2022-10-11 13:03:26] ...trap/engines/engine.py.321: eval: epoch 105 | batch 290/510 [I 2022-10-11 13:03:26] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:13 [I 2022-10-11 13:03:26] ...trap/engines/engine.py.326: process: 0.33502 | load: 0.00023 [I 2022-10-11 13:03:29] ...trap/engines/engine.py.321: eval: epoch 105 | batch 300/510 [I 2022-10-11 13:03:29] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:10 [I 2022-10-11 13:03:29] ...trap/engines/engine.py.326: process: 0.32720 | load: 0.00022 [I 2022-10-11 13:03:32] ...trap/engines/engine.py.321: eval: epoch 105 | batch 310/510 [I 2022-10-11 13:03:32] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:06 [I 2022-10-11 13:03:32] ...trap/engines/engine.py.326: process: 0.33179 | load: 0.00022 [I 2022-10-11 13:03:36] ...trap/engines/engine.py.321: eval: epoch 105 | batch 320/510 [I 2022-10-11 13:03:36] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:01:03 [I 2022-10-11 13:03:36] ...trap/engines/engine.py.326: process: 0.33710 | load: 0.00023 [I 2022-10-11 13:03:39] ...trap/engines/engine.py.321: eval: epoch 105 | batch 330/510 [I 2022-10-11 13:03:39] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:59 [I 2022-10-11 13:03:39] ...trap/engines/engine.py.326: process: 0.33714 | load: 0.00023 [I 2022-10-11 13:03:42] ...trap/engines/engine.py.321: eval: epoch 105 | batch 340/510 [I 2022-10-11 13:03:42] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:56 [I 2022-10-11 13:03:42] ...trap/engines/engine.py.326: process: 0.33552 | load: 0.00023 [I 2022-10-11 13:03:46] ...trap/engines/engine.py.321: eval: epoch 105 | batch 350/510 [I 2022-10-11 13:03:46] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:53 [I 2022-10-11 13:03:46] ...trap/engines/engine.py.326: process: 0.33824 | load: 0.00023 [I 2022-10-11 13:03:49] ...trap/engines/engine.py.321: eval: epoch 105 | batch 360/510 [I 2022-10-11 13:03:49] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:50 [I 2022-10-11 13:03:49] ...trap/engines/engine.py.326: process: 0.34034 | load: 0.00023 [I 2022-10-11 13:03:52] ...trap/engines/engine.py.321: eval: epoch 105 | batch 370/510 [I 2022-10-11 13:03:52] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:46 [I 2022-10-11 13:03:52] ...trap/engines/engine.py.326: process: 0.33276 | load: 0.00025 [I 2022-10-11 13:03:56] ...trap/engines/engine.py.321: eval: epoch 105 | batch 380/510 [I 2022-10-11 13:03:56] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:42 [I 2022-10-11 13:03:56] ...trap/engines/engine.py.326: process: 0.33972 | load: 0.00027 [I 2022-10-11 13:03:59] ...trap/engines/engine.py.321: eval: epoch 105 | batch 390/510 [I 2022-10-11 13:03:59] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:38 [I 2022-10-11 13:03:59] ...trap/engines/engine.py.326: process: 0.32293 | load: 0.00021 [I 2022-10-11 13:04:02] ...trap/engines/engine.py.321: eval: epoch 105 | batch 400/510 [I 2022-10-11 13:04:02] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:35 [I 2022-10-11 13:04:02] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00025 [I 2022-10-11 13:04:05] ...trap/engines/engine.py.321: eval: epoch 105 | batch 410/510 [I 2022-10-11 13:04:05] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:32 [I 2022-10-11 13:04:05] ...trap/engines/engine.py.326: process: 0.32481 | load: 0.00022 [I 2022-10-11 13:04:08] ...trap/engines/engine.py.321: eval: epoch 105 | batch 420/510 [I 2022-10-11 13:04:08] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:28 [I 2022-10-11 13:04:08] ...trap/engines/engine.py.326: process: 0.31874 | load: 0.00021 [I 2022-10-11 13:04:12] ...trap/engines/engine.py.321: eval: epoch 105 | batch 430/510 [I 2022-10-11 13:04:12] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:25 [I 2022-10-11 13:04:12] ...trap/engines/engine.py.326: process: 0.32256 | load: 0.00022 [I 2022-10-11 13:04:15] ...trap/engines/engine.py.321: eval: epoch 105 | batch 440/510 [I 2022-10-11 13:04:15] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:22 [I 2022-10-11 13:04:15] ...trap/engines/engine.py.326: process: 0.33031 | load: 0.00022 [I 2022-10-11 13:04:18] ...trap/engines/engine.py.321: eval: epoch 105 | batch 450/510 [I 2022-10-11 13:04:18] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:19 [I 2022-10-11 13:04:18] ...trap/engines/engine.py.326: process: 0.32758 | load: 0.00025 [I 2022-10-11 13:04:21] ...trap/engines/engine.py.321: eval: epoch 105 | batch 460/510 [I 2022-10-11 13:04:21] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:16 [I 2022-10-11 13:04:21] ...trap/engines/engine.py.326: process: 0.32785 | load: 0.00021 [I 2022-10-11 13:04:25] ...trap/engines/engine.py.321: eval: epoch 105 | batch 470/510 [I 2022-10-11 13:04:25] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:12 [I 2022-10-11 13:04:25] ...trap/engines/engine.py.326: process: 0.31896 | load: 0.00022 [I 2022-10-11 13:04:28] ...trap/engines/engine.py.321: eval: epoch 105 | batch 480/510 [I 2022-10-11 13:04:28] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:09 [I 2022-10-11 13:04:28] ...trap/engines/engine.py.326: process: 0.32384 | load: 0.00021 [I 2022-10-11 13:04:31] ...trap/engines/engine.py.321: eval: epoch 105 | batch 490/510 [I 2022-10-11 13:04:31] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:06 [I 2022-10-11 13:04:31] ...trap/engines/engine.py.326: process: 0.32307 | load: 0.00022 [I 2022-10-11 13:04:34] ...trap/engines/engine.py.321: eval: epoch 105 | batch 500/510 [I 2022-10-11 13:04:34] ...trap/engines/engine.py.325: elapsed: 0:03:17 | left: 0:00:03 [I 2022-10-11 13:04:34] ...trap/engines/engine.py.326: process: 0.32707 | load: 0.00022 [I 2022-10-11 13:04:37] ...trap/engines/engine.py.321: eval: epoch 105 | batch 510/510 [I 2022-10-11 13:04:37] ...trap/engines/engine.py.325: elapsed: 0:03:21 | left: 0:00:00 [I 2022-10-11 13:04:37] ...trap/engines/engine.py.326: process: 0.32095 | load: 0.00017 [S 2022-10-11 13:04:38] ...trap/engines/engine.py.132: eval_epoch.epoch: 105 [I 2022-10-11 13:04:38] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.748300 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911300 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935000 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.741100 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911700 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936600 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013799 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008649 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006116 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015846 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005460 [S 2022-10-11 13:05:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004386 [I 2022-10-11 13:05:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 13:05:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 13:05:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 13:05:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 13:05:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 13:05:25] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 13:05:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 13:05:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 13:05:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 106 [I 2022-10-11 13:05:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 13:05:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 0/2304 [I 2022-10-11 13:05:36] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:57:51 [I 2022-10-11 13:05:36] ...trap/engines/engine.py.229: process: 3.06913 | load: 2.15575 [I 2022-10-11 13:05:36] ...trap/engines/engine.py.230: loss: 0.09237 [I 2022-10-11 13:05:36] ...optimizers/trijoint.py.151: total_norm: 11.256762 [I 2022-10-11 13:05:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 10/2304 [I 2022-10-11 13:05:45] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:50 [I 2022-10-11 13:05:45] ...trap/engines/engine.py.229: process: 0.87020 | load: 0.00586 [I 2022-10-11 13:05:45] ...trap/engines/engine.py.230: loss: 0.16252 [I 2022-10-11 13:05:45] ...optimizers/trijoint.py.151: total_norm: 27.743389 [I 2022-10-11 13:05:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 20/2304 [I 2022-10-11 13:05:53] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:03 [I 2022-10-11 13:05:53] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00384 [I 2022-10-11 13:05:53] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-11 13:05:53] ...optimizers/trijoint.py.151: total_norm: 28.176537 [I 2022-10-11 13:06:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 30/2304 [I 2022-10-11 13:06:02] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:17 [I 2022-10-11 13:06:02] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00479 [I 2022-10-11 13:06:02] ...trap/engines/engine.py.230: loss: 0.08820 [I 2022-10-11 13:06:02] ...optimizers/trijoint.py.151: total_norm: 12.819173 [I 2022-10-11 13:06:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 40/2304 [I 2022-10-11 13:06:10] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:39 [I 2022-10-11 13:06:10] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00374 [I 2022-10-11 13:06:10] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-11 13:06:10] ...optimizers/trijoint.py.151: total_norm: 34.320526 [I 2022-10-11 13:06:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 50/2304 [I 2022-10-11 13:06:19] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:40 [I 2022-10-11 13:06:19] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00373 [I 2022-10-11 13:06:19] ...trap/engines/engine.py.230: loss: 0.08202 [I 2022-10-11 13:06:19] ...optimizers/trijoint.py.151: total_norm: 10.200414 [I 2022-10-11 13:06:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 60/2304 [I 2022-10-11 13:06:27] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:31:29 [I 2022-10-11 13:06:27] ...trap/engines/engine.py.229: process: 0.85421 | load: 0.00297 [I 2022-10-11 13:06:27] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-11 13:06:27] ...optimizers/trijoint.py.151: total_norm: 13.549296 [I 2022-10-11 13:06:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 70/2304 [I 2022-10-11 13:06:36] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:22 [I 2022-10-11 13:06:36] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00379 [I 2022-10-11 13:06:36] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-11 13:06:36] ...optimizers/trijoint.py.151: total_norm: 15.766502 [I 2022-10-11 13:06:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 80/2304 [I 2022-10-11 13:06:44] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:16 [I 2022-10-11 13:06:44] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00336 [I 2022-10-11 13:06:44] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-11 13:06:44] ...optimizers/trijoint.py.151: total_norm: 10.408793 [I 2022-10-11 13:06:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 90/2304 [I 2022-10-11 13:06:53] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:09 [I 2022-10-11 13:06:53] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00335 [I 2022-10-11 13:06:53] ...trap/engines/engine.py.230: loss: 0.11146 [I 2022-10-11 13:06:53] ...optimizers/trijoint.py.151: total_norm: 11.926268 [I 2022-10-11 13:07:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 100/2304 [I 2022-10-11 13:07:01] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:31:05 [I 2022-10-11 13:07:01] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00320 [I 2022-10-11 13:07:01] ...trap/engines/engine.py.230: loss: 0.13685 [I 2022-10-11 13:07:01] ...optimizers/trijoint.py.151: total_norm: 14.387546 [I 2022-10-11 13:07:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 110/2304 [I 2022-10-11 13:07:10] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:50 [I 2022-10-11 13:07:10] ...trap/engines/engine.py.229: process: 0.84298 | load: 0.00357 [I 2022-10-11 13:07:10] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-11 13:07:10] ...optimizers/trijoint.py.151: total_norm: 24.433666 [I 2022-10-11 13:07:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 120/2304 [I 2022-10-11 13:07:18] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:47 [I 2022-10-11 13:07:18] ...trap/engines/engine.py.229: process: 0.84276 | load: 0.00325 [I 2022-10-11 13:07:18] ...trap/engines/engine.py.230: loss: 0.14604 [I 2022-10-11 13:07:18] ...optimizers/trijoint.py.151: total_norm: 26.428305 [I 2022-10-11 13:07:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 130/2304 [I 2022-10-11 13:07:27] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:31:31 [I 2022-10-11 13:07:27] ...trap/engines/engine.py.229: process: 0.87073 | load: 0.00422 [I 2022-10-11 13:07:27] ...trap/engines/engine.py.230: loss: 0.10848 [I 2022-10-11 13:07:27] ...optimizers/trijoint.py.151: total_norm: 11.644258 [I 2022-10-11 13:07:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 140/2304 [I 2022-10-11 13:07:35] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:49 [I 2022-10-11 13:07:35] ...trap/engines/engine.py.229: process: 0.84958 | load: 0.00362 [I 2022-10-11 13:07:35] ...trap/engines/engine.py.230: loss: 0.11450 [I 2022-10-11 13:07:35] ...optimizers/trijoint.py.151: total_norm: 29.435219 [I 2022-10-11 13:07:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 150/2304 [I 2022-10-11 13:07:44] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:37 [I 2022-10-11 13:07:44] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00323 [I 2022-10-11 13:07:44] ...trap/engines/engine.py.230: loss: 0.10390 [I 2022-10-11 13:07:44] ...optimizers/trijoint.py.151: total_norm: 11.123030 [I 2022-10-11 13:07:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 160/2304 [I 2022-10-11 13:07:53] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:35 [I 2022-10-11 13:07:53] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00329 [I 2022-10-11 13:07:53] ...trap/engines/engine.py.230: loss: 0.08409 [I 2022-10-11 13:07:53] ...optimizers/trijoint.py.151: total_norm: 14.877559 [I 2022-10-11 13:08:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 170/2304 [I 2022-10-11 13:08:01] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:30:30 [I 2022-10-11 13:08:01] ...trap/engines/engine.py.229: process: 0.85567 | load: 0.00339 [I 2022-10-11 13:08:01] ...trap/engines/engine.py.230: loss: 0.10696 [I 2022-10-11 13:08:01] ...optimizers/trijoint.py.151: total_norm: 17.417292 [I 2022-10-11 13:08:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 180/2304 [I 2022-10-11 13:08:10] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:30:08 [I 2022-10-11 13:08:10] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00312 [I 2022-10-11 13:08:10] ...trap/engines/engine.py.230: loss: 0.18744 [I 2022-10-11 13:08:10] ...optimizers/trijoint.py.151: total_norm: 37.484722 [I 2022-10-11 13:08:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 190/2304 [I 2022-10-11 13:08:18] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:30:20 [I 2022-10-11 13:08:18] ...trap/engines/engine.py.229: process: 0.87601 | load: 0.00333 [I 2022-10-11 13:08:18] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-11 13:08:18] ...optimizers/trijoint.py.151: total_norm: 19.929916 [I 2022-10-11 13:08:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 200/2304 [I 2022-10-11 13:08:27] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:30:02 [I 2022-10-11 13:08:27] ...trap/engines/engine.py.229: process: 0.84341 | load: 0.00329 [I 2022-10-11 13:08:27] ...trap/engines/engine.py.230: loss: 0.07704 [I 2022-10-11 13:08:27] ...optimizers/trijoint.py.151: total_norm: 14.426527 [I 2022-10-11 13:08:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 210/2304 [I 2022-10-11 13:08:36] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:30:03 [I 2022-10-11 13:08:36] ...trap/engines/engine.py.229: process: 0.86018 | load: 0.00316 [I 2022-10-11 13:08:36] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-11 13:08:36] ...optimizers/trijoint.py.151: total_norm: 8.123409 [I 2022-10-11 13:08:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 220/2304 [I 2022-10-11 13:08:44] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:43 [I 2022-10-11 13:08:44] ...trap/engines/engine.py.229: process: 0.84964 | load: 0.00312 [I 2022-10-11 13:08:44] ...trap/engines/engine.py.230: loss: 0.15003 [I 2022-10-11 13:08:44] ...optimizers/trijoint.py.151: total_norm: 37.499634 [I 2022-10-11 13:08:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 230/2304 [I 2022-10-11 13:08:53] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:43 [I 2022-10-11 13:08:53] ...trap/engines/engine.py.229: process: 0.86318 | load: 0.00312 [I 2022-10-11 13:08:53] ...trap/engines/engine.py.230: loss: 0.10233 [I 2022-10-11 13:08:53] ...optimizers/trijoint.py.151: total_norm: 8.931614 [I 2022-10-11 13:09:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 240/2304 [I 2022-10-11 13:09:02] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:59 [I 2022-10-11 13:09:02] ...trap/engines/engine.py.229: process: 0.88186 | load: 0.00324 [I 2022-10-11 13:09:02] ...trap/engines/engine.py.230: loss: 0.14527 [I 2022-10-11 13:09:02] ...optimizers/trijoint.py.151: total_norm: 16.140108 [I 2022-10-11 13:09:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 250/2304 [I 2022-10-11 13:09:10] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:19 [I 2022-10-11 13:09:10] ...trap/engines/engine.py.229: process: 0.85059 | load: 0.00329 [I 2022-10-11 13:09:10] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-11 13:09:10] ...optimizers/trijoint.py.151: total_norm: 19.320093 [I 2022-10-11 13:09:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 260/2304 [I 2022-10-11 13:09:19] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:13 [I 2022-10-11 13:09:19] ...trap/engines/engine.py.229: process: 0.85443 | load: 0.00318 [I 2022-10-11 13:09:19] ...trap/engines/engine.py.230: loss: 0.10755 [I 2022-10-11 13:09:19] ...optimizers/trijoint.py.151: total_norm: 14.164173 [I 2022-10-11 13:09:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 270/2304 [I 2022-10-11 13:09:27] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:29:01 [I 2022-10-11 13:09:27] ...trap/engines/engine.py.229: process: 0.86842 | load: 0.00311 [I 2022-10-11 13:09:27] ...trap/engines/engine.py.230: loss: 0.12323 [I 2022-10-11 13:09:27] ...optimizers/trijoint.py.151: total_norm: 30.770117 [I 2022-10-11 13:09:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 280/2304 [I 2022-10-11 13:09:36] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:43 [I 2022-10-11 13:09:36] ...trap/engines/engine.py.229: process: 0.84318 | load: 0.00310 [I 2022-10-11 13:09:36] ...trap/engines/engine.py.230: loss: 0.15005 [I 2022-10-11 13:09:36] ...optimizers/trijoint.py.151: total_norm: 12.541012 [I 2022-10-11 13:09:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 290/2304 [I 2022-10-11 13:09:44] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:36 [I 2022-10-11 13:09:44] ...trap/engines/engine.py.229: process: 0.85568 | load: 0.00312 [I 2022-10-11 13:09:44] ...trap/engines/engine.py.230: loss: 0.08794 [I 2022-10-11 13:09:44] ...optimizers/trijoint.py.151: total_norm: 10.666284 [I 2022-10-11 13:09:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 300/2304 [I 2022-10-11 13:09:53] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:28:24 [I 2022-10-11 13:09:53] ...trap/engines/engine.py.229: process: 0.84820 | load: 0.00315 [I 2022-10-11 13:09:53] ...trap/engines/engine.py.230: loss: 0.10477 [I 2022-10-11 13:09:53] ...optimizers/trijoint.py.151: total_norm: 11.091128 [I 2022-10-11 13:10:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 310/2304 [I 2022-10-11 13:10:01] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:23 [I 2022-10-11 13:10:01] ...trap/engines/engine.py.229: process: 0.86413 | load: 0.00330 [I 2022-10-11 13:10:01] ...trap/engines/engine.py.230: loss: 0.08135 [I 2022-10-11 13:10:01] ...optimizers/trijoint.py.151: total_norm: 9.752412 [I 2022-10-11 13:10:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 320/2304 [I 2022-10-11 13:10:10] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:28:24 [I 2022-10-11 13:10:10] ...trap/engines/engine.py.229: process: 0.86057 | load: 0.00312 [I 2022-10-11 13:10:10] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-11 13:10:10] ...optimizers/trijoint.py.151: total_norm: 17.027676 [I 2022-10-11 13:10:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 330/2304 [I 2022-10-11 13:10:19] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:09 [I 2022-10-11 13:10:19] ...trap/engines/engine.py.229: process: 0.86478 | load: 0.00312 [I 2022-10-11 13:10:19] ...trap/engines/engine.py.230: loss: 0.08381 [I 2022-10-11 13:10:19] ...optimizers/trijoint.py.151: total_norm: 20.427065 [I 2022-10-11 13:10:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 340/2304 [I 2022-10-11 13:10:27] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:28:07 [I 2022-10-11 13:10:27] ...trap/engines/engine.py.229: process: 0.86450 | load: 0.00314 [I 2022-10-11 13:10:27] ...trap/engines/engine.py.230: loss: 0.10105 [I 2022-10-11 13:10:27] ...optimizers/trijoint.py.151: total_norm: 11.470299 [I 2022-10-11 13:10:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 350/2304 [I 2022-10-11 13:10:36] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:27:59 [I 2022-10-11 13:10:36] ...trap/engines/engine.py.229: process: 0.87066 | load: 0.00330 [I 2022-10-11 13:10:36] ...trap/engines/engine.py.230: loss: 0.16540 [I 2022-10-11 13:10:36] ...optimizers/trijoint.py.151: total_norm: 16.022278 [I 2022-10-11 13:10:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 360/2304 [I 2022-10-11 13:10:44] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:28:02 [I 2022-10-11 13:10:44] ...trap/engines/engine.py.229: process: 0.87119 | load: 0.00312 [I 2022-10-11 13:10:44] ...trap/engines/engine.py.230: loss: 0.09046 [I 2022-10-11 13:10:44] ...optimizers/trijoint.py.151: total_norm: 13.194722 [I 2022-10-11 13:10:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 370/2304 [I 2022-10-11 13:10:53] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:27:37 [I 2022-10-11 13:10:53] ...trap/engines/engine.py.229: process: 0.86886 | load: 0.00328 [I 2022-10-11 13:10:53] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-11 13:10:53] ...optimizers/trijoint.py.151: total_norm: 31.400328 [I 2022-10-11 13:11:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 380/2304 [I 2022-10-11 13:11:02] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:30 [I 2022-10-11 13:11:02] ...trap/engines/engine.py.229: process: 0.85174 | load: 0.00313 [I 2022-10-11 13:11:02] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-11 13:11:02] ...optimizers/trijoint.py.151: total_norm: 20.750050 [I 2022-10-11 13:11:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 390/2304 [I 2022-10-11 13:11:10] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:27:28 [I 2022-10-11 13:11:10] ...trap/engines/engine.py.229: process: 0.86727 | load: 0.00313 [I 2022-10-11 13:11:10] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-11 13:11:10] ...optimizers/trijoint.py.151: total_norm: 8.705154 [I 2022-10-11 13:11:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 400/2304 [I 2022-10-11 13:11:19] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:21 [I 2022-10-11 13:11:19] ...trap/engines/engine.py.229: process: 0.85218 | load: 0.00315 [I 2022-10-11 13:11:19] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-11 13:11:19] ...optimizers/trijoint.py.151: total_norm: 12.217370 [I 2022-10-11 13:11:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 410/2304 [I 2022-10-11 13:11:27] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:26:52 [I 2022-10-11 13:11:27] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00300 [I 2022-10-11 13:11:27] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-11 13:11:27] ...optimizers/trijoint.py.151: total_norm: 17.052458 [I 2022-10-11 13:11:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 420/2304 [I 2022-10-11 13:11:36] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:26:41 [I 2022-10-11 13:11:36] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00310 [I 2022-10-11 13:11:36] ...trap/engines/engine.py.230: loss: 0.13486 [I 2022-10-11 13:11:36] ...optimizers/trijoint.py.151: total_norm: 24.382891 [I 2022-10-11 13:11:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 430/2304 [I 2022-10-11 13:11:44] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:24 [I 2022-10-11 13:11:44] ...trap/engines/engine.py.229: process: 0.84863 | load: 0.00305 [I 2022-10-11 13:11:44] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-11 13:11:44] ...optimizers/trijoint.py.151: total_norm: 19.141882 [I 2022-10-11 13:11:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 440/2304 [I 2022-10-11 13:11:53] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:26:34 [I 2022-10-11 13:11:53] ...trap/engines/engine.py.229: process: 0.87639 | load: 0.00290 [I 2022-10-11 13:11:53] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-11 13:11:53] ...optimizers/trijoint.py.151: total_norm: 25.679897 [I 2022-10-11 13:12:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 450/2304 [I 2022-10-11 13:12:02] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:26:54 [I 2022-10-11 13:12:02] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00375 [I 2022-10-11 13:12:02] ...trap/engines/engine.py.230: loss: 0.08414 [I 2022-10-11 13:12:02] ...optimizers/trijoint.py.151: total_norm: 12.482006 [I 2022-10-11 13:12:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 460/2304 [I 2022-10-11 13:12:10] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:25:56 [I 2022-10-11 13:12:10] ...trap/engines/engine.py.229: process: 0.83368 | load: 0.00293 [I 2022-10-11 13:12:10] ...trap/engines/engine.py.230: loss: 0.15942 [I 2022-10-11 13:12:10] ...optimizers/trijoint.py.151: total_norm: 16.213409 [I 2022-10-11 13:12:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 470/2304 [I 2022-10-11 13:12:19] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:25:54 [I 2022-10-11 13:12:19] ...trap/engines/engine.py.229: process: 0.85198 | load: 0.00297 [I 2022-10-11 13:12:19] ...trap/engines/engine.py.230: loss: 0.07504 [I 2022-10-11 13:12:19] ...optimizers/trijoint.py.151: total_norm: 9.610739 [I 2022-10-11 13:12:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 480/2304 [I 2022-10-11 13:12:27] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:25:43 [I 2022-10-11 13:12:27] ...trap/engines/engine.py.229: process: 0.83495 | load: 0.00297 [I 2022-10-11 13:12:27] ...trap/engines/engine.py.230: loss: 0.08995 [I 2022-10-11 13:12:27] ...optimizers/trijoint.py.151: total_norm: 9.031721 [I 2022-10-11 13:12:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 490/2304 [I 2022-10-11 13:12:36] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:25:33 [I 2022-10-11 13:12:36] ...trap/engines/engine.py.229: process: 0.85103 | load: 0.00295 [I 2022-10-11 13:12:36] ...trap/engines/engine.py.230: loss: 0.07788 [I 2022-10-11 13:12:36] ...optimizers/trijoint.py.151: total_norm: 9.714890 [I 2022-10-11 13:12:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 500/2304 [I 2022-10-11 13:12:44] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:20 [I 2022-10-11 13:12:44] ...trap/engines/engine.py.229: process: 0.84689 | load: 0.00300 [I 2022-10-11 13:12:44] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-11 13:12:44] ...optimizers/trijoint.py.151: total_norm: 14.025687 [I 2022-10-11 13:12:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 510/2304 [I 2022-10-11 13:12:53] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:25:14 [I 2022-10-11 13:12:53] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00299 [I 2022-10-11 13:12:53] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-11 13:12:53] ...optimizers/trijoint.py.151: total_norm: 13.305431 [I 2022-10-11 13:13:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 520/2304 [I 2022-10-11 13:13:01] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:25:16 [I 2022-10-11 13:13:01] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00314 [I 2022-10-11 13:13:01] ...trap/engines/engine.py.230: loss: 0.10022 [I 2022-10-11 13:13:01] ...optimizers/trijoint.py.151: total_norm: 14.485929 [I 2022-10-11 13:13:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 530/2304 [I 2022-10-11 13:13:10] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:25:05 [I 2022-10-11 13:13:10] ...trap/engines/engine.py.229: process: 0.84121 | load: 0.00297 [I 2022-10-11 13:13:10] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-11 13:13:10] ...optimizers/trijoint.py.151: total_norm: 29.399471 [I 2022-10-11 13:13:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 540/2304 [I 2022-10-11 13:13:18] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:24:50 [I 2022-10-11 13:13:18] ...trap/engines/engine.py.229: process: 0.83349 | load: 0.00300 [I 2022-10-11 13:13:18] ...trap/engines/engine.py.230: loss: 0.13244 [I 2022-10-11 13:13:18] ...optimizers/trijoint.py.151: total_norm: 13.930626 [I 2022-10-11 13:13:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 550/2304 [I 2022-10-11 13:13:27] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:24:42 [I 2022-10-11 13:13:27] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00302 [I 2022-10-11 13:13:27] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-11 13:13:27] ...optimizers/trijoint.py.151: total_norm: 13.689652 [I 2022-10-11 13:13:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 560/2304 [I 2022-10-11 13:13:35] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:24:29 [I 2022-10-11 13:13:35] ...trap/engines/engine.py.229: process: 0.84244 | load: 0.00300 [I 2022-10-11 13:13:35] ...trap/engines/engine.py.230: loss: 0.11657 [I 2022-10-11 13:13:35] ...optimizers/trijoint.py.151: total_norm: 7.753677 [I 2022-10-11 13:13:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 570/2304 [I 2022-10-11 13:13:43] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:14 [I 2022-10-11 13:13:43] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00301 [I 2022-10-11 13:13:43] ...trap/engines/engine.py.230: loss: 0.08000 [I 2022-10-11 13:13:43] ...optimizers/trijoint.py.151: total_norm: 9.936034 [I 2022-10-11 13:13:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 580/2304 [I 2022-10-11 13:13:52] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:10 [I 2022-10-11 13:13:52] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00298 [I 2022-10-11 13:13:52] ...trap/engines/engine.py.230: loss: 0.10585 [I 2022-10-11 13:13:52] ...optimizers/trijoint.py.151: total_norm: 27.039360 [I 2022-10-11 13:14:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 590/2304 [I 2022-10-11 13:14:00] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:26 [I 2022-10-11 13:14:00] ...trap/engines/engine.py.229: process: 0.86636 | load: 0.00315 [I 2022-10-11 13:14:00] ...trap/engines/engine.py.230: loss: 0.08952 [I 2022-10-11 13:14:00] ...optimizers/trijoint.py.151: total_norm: 12.668934 [I 2022-10-11 13:14:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 600/2304 [I 2022-10-11 13:14:09] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:24:13 [I 2022-10-11 13:14:09] ...trap/engines/engine.py.229: process: 0.84285 | load: 0.00313 [I 2022-10-11 13:14:09] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-11 13:14:09] ...optimizers/trijoint.py.151: total_norm: 17.752022 [I 2022-10-11 13:14:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 610/2304 [I 2022-10-11 13:14:18] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:24:20 [I 2022-10-11 13:14:18] ...trap/engines/engine.py.229: process: 0.87321 | load: 0.00326 [I 2022-10-11 13:14:18] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-11 13:14:18] ...optimizers/trijoint.py.151: total_norm: 19.238710 [I 2022-10-11 13:14:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 620/2304 [I 2022-10-11 13:14:26] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:58 [I 2022-10-11 13:14:26] ...trap/engines/engine.py.229: process: 0.85289 | load: 0.00310 [I 2022-10-11 13:14:26] ...trap/engines/engine.py.230: loss: 0.09941 [I 2022-10-11 13:14:26] ...optimizers/trijoint.py.151: total_norm: 20.495153 [I 2022-10-11 13:14:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 630/2304 [I 2022-10-11 13:14:35] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:23:44 [I 2022-10-11 13:14:35] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00311 [I 2022-10-11 13:14:35] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-11 13:14:35] ...optimizers/trijoint.py.151: total_norm: 9.962250 [I 2022-10-11 13:14:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 640/2304 [I 2022-10-11 13:14:43] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:55 [I 2022-10-11 13:14:43] ...trap/engines/engine.py.229: process: 0.86996 | load: 0.00317 [I 2022-10-11 13:14:43] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-11 13:14:43] ...optimizers/trijoint.py.151: total_norm: 13.128208 [I 2022-10-11 13:14:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 650/2304 [I 2022-10-11 13:14:52] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:23:29 [I 2022-10-11 13:14:52] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00313 [I 2022-10-11 13:14:52] ...trap/engines/engine.py.230: loss: 0.07957 [I 2022-10-11 13:14:52] ...optimizers/trijoint.py.151: total_norm: 15.401028 [I 2022-10-11 13:15:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 660/2304 [I 2022-10-11 13:15:01] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:23:44 [I 2022-10-11 13:15:01] ...trap/engines/engine.py.229: process: 0.87126 | load: 0.00312 [I 2022-10-11 13:15:01] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-11 13:15:01] ...optimizers/trijoint.py.151: total_norm: 21.425644 [I 2022-10-11 13:15:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 670/2304 [I 2022-10-11 13:15:09] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:23:20 [I 2022-10-11 13:15:09] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00347 [I 2022-10-11 13:15:09] ...trap/engines/engine.py.230: loss: 0.08591 [I 2022-10-11 13:15:09] ...optimizers/trijoint.py.151: total_norm: 14.934558 [I 2022-10-11 13:15:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 680/2304 [I 2022-10-11 13:15:18] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:23:20 [I 2022-10-11 13:15:18] ...trap/engines/engine.py.229: process: 0.86819 | load: 0.00314 [I 2022-10-11 13:15:18] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-11 13:15:18] ...optimizers/trijoint.py.151: total_norm: 12.082325 [I 2022-10-11 13:15:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 690/2304 [I 2022-10-11 13:15:26] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:23:09 [I 2022-10-11 13:15:26] ...trap/engines/engine.py.229: process: 0.87085 | load: 0.00313 [I 2022-10-11 13:15:26] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-11 13:15:26] ...optimizers/trijoint.py.151: total_norm: 10.532352 [I 2022-10-11 13:15:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 700/2304 [I 2022-10-11 13:15:35] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:22:47 [I 2022-10-11 13:15:35] ...trap/engines/engine.py.229: process: 0.85641 | load: 0.00313 [I 2022-10-11 13:15:35] ...trap/engines/engine.py.230: loss: 0.14507 [I 2022-10-11 13:15:35] ...optimizers/trijoint.py.151: total_norm: 40.047337 [I 2022-10-11 13:15:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 710/2304 [I 2022-10-11 13:15:43] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:40 [I 2022-10-11 13:15:43] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00315 [I 2022-10-11 13:15:43] ...trap/engines/engine.py.230: loss: 0.14413 [I 2022-10-11 13:15:43] ...optimizers/trijoint.py.151: total_norm: 15.842645 [I 2022-10-11 13:15:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 720/2304 [I 2022-10-11 13:15:52] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:22:33 [I 2022-10-11 13:15:52] ...trap/engines/engine.py.229: process: 0.84738 | load: 0.00314 [I 2022-10-11 13:15:52] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-11 13:15:52] ...optimizers/trijoint.py.151: total_norm: 9.994514 [I 2022-10-11 13:16:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 730/2304 [I 2022-10-11 13:16:01] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:22:31 [I 2022-10-11 13:16:01] ...trap/engines/engine.py.229: process: 0.87242 | load: 0.00313 [I 2022-10-11 13:16:01] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-11 13:16:01] ...optimizers/trijoint.py.151: total_norm: 17.372084 [I 2022-10-11 13:16:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 740/2304 [I 2022-10-11 13:16:09] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:22:16 [I 2022-10-11 13:16:09] ...trap/engines/engine.py.229: process: 0.85536 | load: 0.00316 [I 2022-10-11 13:16:09] ...trap/engines/engine.py.230: loss: 0.09175 [I 2022-10-11 13:16:09] ...optimizers/trijoint.py.151: total_norm: 14.538408 [I 2022-10-11 13:16:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 750/2304 [I 2022-10-11 13:16:18] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:22:23 [I 2022-10-11 13:16:18] ...trap/engines/engine.py.229: process: 0.87761 | load: 0.00315 [I 2022-10-11 13:16:18] ...trap/engines/engine.py.230: loss: 0.20291 [I 2022-10-11 13:16:18] ...optimizers/trijoint.py.151: total_norm: 31.257086 [I 2022-10-11 13:16:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 760/2304 [I 2022-10-11 13:16:26] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:22:03 [I 2022-10-11 13:16:26] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00311 [I 2022-10-11 13:16:26] ...trap/engines/engine.py.230: loss: 0.09083 [I 2022-10-11 13:16:26] ...optimizers/trijoint.py.151: total_norm: 21.154636 [I 2022-10-11 13:16:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 770/2304 [I 2022-10-11 13:16:35] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:22:10 [I 2022-10-11 13:16:35] ...trap/engines/engine.py.229: process: 0.88183 | load: 0.00311 [I 2022-10-11 13:16:35] ...trap/engines/engine.py.230: loss: 0.13564 [I 2022-10-11 13:16:35] ...optimizers/trijoint.py.151: total_norm: 20.054195 [I 2022-10-11 13:16:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 780/2304 [I 2022-10-11 13:16:44] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:21:53 [I 2022-10-11 13:16:44] ...trap/engines/engine.py.229: process: 0.87054 | load: 0.00315 [I 2022-10-11 13:16:44] ...trap/engines/engine.py.230: loss: 0.13423 [I 2022-10-11 13:16:44] ...optimizers/trijoint.py.151: total_norm: 25.167593 [I 2022-10-11 13:16:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 790/2304 [I 2022-10-11 13:16:52] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:39 [I 2022-10-11 13:16:52] ...trap/engines/engine.py.229: process: 0.85840 | load: 0.00311 [I 2022-10-11 13:16:52] ...trap/engines/engine.py.230: loss: 0.10637 [I 2022-10-11 13:16:52] ...optimizers/trijoint.py.151: total_norm: 17.596167 [I 2022-10-11 13:17:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 800/2304 [I 2022-10-11 13:17:01] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:21:23 [I 2022-10-11 13:17:01] ...trap/engines/engine.py.229: process: 0.84529 | load: 0.00317 [I 2022-10-11 13:17:01] ...trap/engines/engine.py.230: loss: 0.16979 [I 2022-10-11 13:17:01] ...optimizers/trijoint.py.151: total_norm: 22.073235 [I 2022-10-11 13:17:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 810/2304 [I 2022-10-11 13:17:09] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:20 [I 2022-10-11 13:17:09] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00318 [I 2022-10-11 13:17:09] ...trap/engines/engine.py.230: loss: 0.08271 [I 2022-10-11 13:17:09] ...optimizers/trijoint.py.151: total_norm: 10.507485 [I 2022-10-11 13:17:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 820/2304 [I 2022-10-11 13:17:18] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:21:16 [I 2022-10-11 13:17:18] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00312 [I 2022-10-11 13:17:18] ...trap/engines/engine.py.230: loss: 0.13536 [I 2022-10-11 13:17:18] ...optimizers/trijoint.py.151: total_norm: 18.354595 [I 2022-10-11 13:17:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 830/2304 [I 2022-10-11 13:17:27] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:20:55 [I 2022-10-11 13:17:27] ...trap/engines/engine.py.229: process: 0.84266 | load: 0.00317 [I 2022-10-11 13:17:27] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-11 13:17:27] ...optimizers/trijoint.py.151: total_norm: 17.093853 [I 2022-10-11 13:17:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 840/2304 [I 2022-10-11 13:17:35] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:20:51 [I 2022-10-11 13:17:35] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00313 [I 2022-10-11 13:17:35] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-11 13:17:35] ...optimizers/trijoint.py.151: total_norm: 16.552013 [I 2022-10-11 13:17:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 850/2304 [I 2022-10-11 13:17:44] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:20:45 [I 2022-10-11 13:17:44] ...trap/engines/engine.py.229: process: 0.86466 | load: 0.00315 [I 2022-10-11 13:17:44] ...trap/engines/engine.py.230: loss: 0.09792 [I 2022-10-11 13:17:44] ...optimizers/trijoint.py.151: total_norm: 15.768234 [I 2022-10-11 13:17:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 860/2304 [I 2022-10-11 13:17:52] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:20:42 [I 2022-10-11 13:17:52] ...trap/engines/engine.py.229: process: 0.86706 | load: 0.00314 [I 2022-10-11 13:17:52] ...trap/engines/engine.py.230: loss: 0.11045 [I 2022-10-11 13:17:52] ...optimizers/trijoint.py.151: total_norm: 8.484142 [I 2022-10-11 13:18:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 870/2304 [I 2022-10-11 13:18:01] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:20:15 [I 2022-10-11 13:18:01] ...trap/engines/engine.py.229: process: 0.83764 | load: 0.00295 [I 2022-10-11 13:18:01] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-11 13:18:01] ...optimizers/trijoint.py.151: total_norm: 29.624659 [I 2022-10-11 13:18:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 880/2304 [I 2022-10-11 13:18:09] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:02 [I 2022-10-11 13:18:09] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00302 [I 2022-10-11 13:18:09] ...trap/engines/engine.py.230: loss: 0.07335 [I 2022-10-11 13:18:09] ...optimizers/trijoint.py.151: total_norm: 12.918976 [I 2022-10-11 13:18:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 890/2304 [I 2022-10-11 13:18:18] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:19:56 [I 2022-10-11 13:18:18] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00298 [I 2022-10-11 13:18:18] ...trap/engines/engine.py.230: loss: 0.16693 [I 2022-10-11 13:18:18] ...optimizers/trijoint.py.151: total_norm: 19.915108 [I 2022-10-11 13:18:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 900/2304 [I 2022-10-11 13:18:26] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:19:41 [I 2022-10-11 13:18:26] ...trap/engines/engine.py.229: process: 0.83780 | load: 0.00297 [I 2022-10-11 13:18:26] ...trap/engines/engine.py.230: loss: 0.20362 [I 2022-10-11 13:18:26] ...optimizers/trijoint.py.151: total_norm: 22.505779 [I 2022-10-11 13:18:34] ...trap/engines/engine.py.224: train: epoch 106 | batch 910/2304 [I 2022-10-11 13:18:34] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:19:30 [I 2022-10-11 13:18:34] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00299 [I 2022-10-11 13:18:35] ...trap/engines/engine.py.230: loss: 0.18355 [I 2022-10-11 13:18:35] ...optimizers/trijoint.py.151: total_norm: 30.559813 [I 2022-10-11 13:18:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 920/2304 [I 2022-10-11 13:18:43] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:25 [I 2022-10-11 13:18:43] ...trap/engines/engine.py.229: process: 0.84477 | load: 0.00298 [I 2022-10-11 13:18:43] ...trap/engines/engine.py.230: loss: 0.09581 [I 2022-10-11 13:18:43] ...optimizers/trijoint.py.151: total_norm: 9.043173 [I 2022-10-11 13:18:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 930/2304 [I 2022-10-11 13:18:52] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:36 [I 2022-10-11 13:18:52] ...trap/engines/engine.py.229: process: 0.87111 | load: 0.00322 [I 2022-10-11 13:18:52] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-11 13:18:52] ...optimizers/trijoint.py.151: total_norm: 12.010411 [I 2022-10-11 13:19:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 940/2304 [I 2022-10-11 13:19:00] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:08 [I 2022-10-11 13:19:00] ...trap/engines/engine.py.229: process: 0.84088 | load: 0.00303 [I 2022-10-11 13:19:00] ...trap/engines/engine.py.230: loss: 0.12174 [I 2022-10-11 13:19:00] ...optimizers/trijoint.py.151: total_norm: 11.527560 [I 2022-10-11 13:19:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 950/2304 [I 2022-10-11 13:19:26] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:52:05 [I 2022-10-11 13:19:26] ...trap/engines/engine.py.229: process: 0.82317 | load: 0.00299 [I 2022-10-11 13:19:26] ...trap/engines/engine.py.230: loss: 0.13246 [I 2022-10-11 13:19:26] ...optimizers/trijoint.py.151: total_norm: 17.621637 [I 2022-10-11 13:19:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 960/2304 [I 2022-10-11 13:19:35] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:22:27 [I 2022-10-11 13:19:35] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00298 [I 2022-10-11 13:19:35] ...trap/engines/engine.py.230: loss: 0.09609 [I 2022-10-11 13:19:35] ...optimizers/trijoint.py.151: total_norm: 14.890862 [I 2022-10-11 13:19:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 970/2304 [I 2022-10-11 13:19:43] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:05 [I 2022-10-11 13:19:43] ...trap/engines/engine.py.229: process: 0.84103 | load: 0.00301 [I 2022-10-11 13:19:43] ...trap/engines/engine.py.230: loss: 0.10324 [I 2022-10-11 13:19:43] ...optimizers/trijoint.py.151: total_norm: 14.277640 [I 2022-10-11 13:19:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 980/2304 [I 2022-10-11 13:19:52] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:41 [I 2022-10-11 13:19:52] ...trap/engines/engine.py.229: process: 0.84841 | load: 0.00300 [I 2022-10-11 13:19:52] ...trap/engines/engine.py.230: loss: 0.07064 [I 2022-10-11 13:19:52] ...optimizers/trijoint.py.151: total_norm: 10.997078 [I 2022-10-11 13:20:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 990/2304 [I 2022-10-11 13:20:00] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:36 [I 2022-10-11 13:20:00] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00319 [I 2022-10-11 13:20:00] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-11 13:20:00] ...optimizers/trijoint.py.151: total_norm: 17.890341 [I 2022-10-11 13:20:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1000/2304 [I 2022-10-11 13:20:09] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:31 [I 2022-10-11 13:20:09] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00314 [I 2022-10-11 13:20:09] ...trap/engines/engine.py.230: loss: 0.08396 [I 2022-10-11 13:20:09] ...optimizers/trijoint.py.151: total_norm: 12.149661 [I 2022-10-11 13:20:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1010/2304 [I 2022-10-11 13:20:17] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:18:18 [I 2022-10-11 13:20:17] ...trap/engines/engine.py.229: process: 0.84174 | load: 0.00313 [I 2022-10-11 13:20:17] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-11 13:20:17] ...optimizers/trijoint.py.151: total_norm: 33.145958 [I 2022-10-11 13:20:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 1020/2304 [I 2022-10-11 13:20:26] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:15 [I 2022-10-11 13:20:26] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00315 [I 2022-10-11 13:20:26] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-11 13:20:26] ...optimizers/trijoint.py.151: total_norm: 28.036818 [I 2022-10-11 13:20:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1030/2304 [I 2022-10-11 13:20:35] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:12 [I 2022-10-11 13:20:35] ...trap/engines/engine.py.229: process: 0.86908 | load: 0.00314 [I 2022-10-11 13:20:35] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-11 13:20:35] ...optimizers/trijoint.py.151: total_norm: 13.241945 [I 2022-10-11 13:20:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 1040/2304 [I 2022-10-11 13:20:43] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:59 [I 2022-10-11 13:20:43] ...trap/engines/engine.py.229: process: 0.84750 | load: 0.00312 [I 2022-10-11 13:20:43] ...trap/engines/engine.py.230: loss: 0.13605 [I 2022-10-11 13:20:43] ...optimizers/trijoint.py.151: total_norm: 15.276142 [I 2022-10-11 13:20:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1050/2304 [I 2022-10-11 13:20:52] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:46 [I 2022-10-11 13:20:52] ...trap/engines/engine.py.229: process: 0.84825 | load: 0.00317 [I 2022-10-11 13:20:52] ...trap/engines/engine.py.230: loss: 0.07026 [I 2022-10-11 13:20:52] ...optimizers/trijoint.py.151: total_norm: 16.129543 [I 2022-10-11 13:21:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 1060/2304 [I 2022-10-11 13:21:00] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:47 [I 2022-10-11 13:21:00] ...trap/engines/engine.py.229: process: 0.88006 | load: 0.00312 [I 2022-10-11 13:21:00] ...trap/engines/engine.py.230: loss: 0.07312 [I 2022-10-11 13:21:00] ...optimizers/trijoint.py.151: total_norm: 11.794328 [I 2022-10-11 13:21:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1070/2304 [I 2022-10-11 13:21:09] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:43 [I 2022-10-11 13:21:09] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00314 [I 2022-10-11 13:21:09] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-11 13:21:09] ...optimizers/trijoint.py.151: total_norm: 21.247799 [I 2022-10-11 13:21:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1080/2304 [I 2022-10-11 13:21:17] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:17:31 [I 2022-10-11 13:21:17] ...trap/engines/engine.py.229: process: 0.85699 | load: 0.00313 [I 2022-10-11 13:21:17] ...trap/engines/engine.py.230: loss: 0.11972 [I 2022-10-11 13:21:17] ...optimizers/trijoint.py.151: total_norm: 13.313182 [I 2022-10-11 13:21:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 1090/2304 [I 2022-10-11 13:21:26] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:26 [I 2022-10-11 13:21:26] ...trap/engines/engine.py.229: process: 0.87569 | load: 0.00314 [I 2022-10-11 13:21:26] ...trap/engines/engine.py.230: loss: 0.17982 [I 2022-10-11 13:21:26] ...optimizers/trijoint.py.151: total_norm: 19.270620 [I 2022-10-11 13:21:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1100/2304 [I 2022-10-11 13:21:35] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:17:19 [I 2022-10-11 13:21:35] ...trap/engines/engine.py.229: process: 0.85856 | load: 0.00312 [I 2022-10-11 13:21:35] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-11 13:21:35] ...optimizers/trijoint.py.151: total_norm: 24.066553 [I 2022-10-11 13:21:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 1110/2304 [I 2022-10-11 13:21:43] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:17:13 [I 2022-10-11 13:21:43] ...trap/engines/engine.py.229: process: 0.87512 | load: 0.00312 [I 2022-10-11 13:21:43] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-11 13:21:43] ...optimizers/trijoint.py.151: total_norm: 15.832455 [I 2022-10-11 13:21:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1120/2304 [I 2022-10-11 13:21:52] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:17:27 [I 2022-10-11 13:21:52] ...trap/engines/engine.py.229: process: 0.92287 | load: 0.00393 [I 2022-10-11 13:21:52] ...trap/engines/engine.py.230: loss: 0.16078 [I 2022-10-11 13:21:52] ...optimizers/trijoint.py.151: total_norm: 8.082655 [I 2022-10-11 13:22:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1130/2304 [I 2022-10-11 13:22:01] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:17:03 [I 2022-10-11 13:22:01] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00312 [I 2022-10-11 13:22:01] ...trap/engines/engine.py.230: loss: 0.18095 [I 2022-10-11 13:22:01] ...optimizers/trijoint.py.151: total_norm: 12.856032 [I 2022-10-11 13:22:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1140/2304 [I 2022-10-11 13:22:10] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:16:41 [I 2022-10-11 13:22:10] ...trap/engines/engine.py.229: process: 0.85194 | load: 0.00314 [I 2022-10-11 13:22:10] ...trap/engines/engine.py.230: loss: 0.09442 [I 2022-10-11 13:22:10] ...optimizers/trijoint.py.151: total_norm: 20.684830 [I 2022-10-11 13:22:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 1150/2304 [I 2022-10-11 13:22:18] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:16:36 [I 2022-10-11 13:22:18] ...trap/engines/engine.py.229: process: 0.86911 | load: 0.00313 [I 2022-10-11 13:22:18] ...trap/engines/engine.py.230: loss: 0.09471 [I 2022-10-11 13:22:18] ...optimizers/trijoint.py.151: total_norm: 9.853193 [I 2022-10-11 13:22:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1160/2304 [I 2022-10-11 13:22:27] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:16:21 [I 2022-10-11 13:22:27] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00310 [I 2022-10-11 13:22:27] ...trap/engines/engine.py.230: loss: 0.16878 [I 2022-10-11 13:22:27] ...optimizers/trijoint.py.151: total_norm: 17.758633 [I 2022-10-11 13:22:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1170/2304 [I 2022-10-11 13:22:35] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:16:12 [I 2022-10-11 13:22:35] ...trap/engines/engine.py.229: process: 0.85823 | load: 0.00320 [I 2022-10-11 13:22:35] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-11 13:22:35] ...optimizers/trijoint.py.151: total_norm: 8.230248 [I 2022-10-11 13:22:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1180/2304 [I 2022-10-11 13:22:44] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:15:55 [I 2022-10-11 13:22:44] ...trap/engines/engine.py.229: process: 0.84360 | load: 0.00313 [I 2022-10-11 13:22:44] ...trap/engines/engine.py.230: loss: 0.13193 [I 2022-10-11 13:22:44] ...optimizers/trijoint.py.151: total_norm: 18.450472 [I 2022-10-11 13:22:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1190/2304 [I 2022-10-11 13:22:52] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:51 [I 2022-10-11 13:22:52] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00319 [I 2022-10-11 13:22:52] ...trap/engines/engine.py.230: loss: 0.08217 [I 2022-10-11 13:22:52] ...optimizers/trijoint.py.151: total_norm: 14.837226 [I 2022-10-11 13:23:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1200/2304 [I 2022-10-11 13:23:01] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:15:39 [I 2022-10-11 13:23:01] ...trap/engines/engine.py.229: process: 0.85555 | load: 0.00314 [I 2022-10-11 13:23:01] ...trap/engines/engine.py.230: loss: 0.09876 [I 2022-10-11 13:23:01] ...optimizers/trijoint.py.151: total_norm: 13.096573 [I 2022-10-11 13:23:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1210/2304 [I 2022-10-11 13:23:09] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:33 [I 2022-10-11 13:23:09] ...trap/engines/engine.py.229: process: 0.86649 | load: 0.00315 [I 2022-10-11 13:23:09] ...trap/engines/engine.py.230: loss: 0.09553 [I 2022-10-11 13:23:09] ...optimizers/trijoint.py.151: total_norm: 11.277862 [I 2022-10-11 13:23:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 1220/2304 [I 2022-10-11 13:23:18] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:15:36 [I 2022-10-11 13:23:18] ...trap/engines/engine.py.229: process: 0.86447 | load: 0.00313 [I 2022-10-11 13:23:18] ...trap/engines/engine.py.230: loss: 0.08006 [I 2022-10-11 13:23:18] ...optimizers/trijoint.py.151: total_norm: 8.807913 [I 2022-10-11 13:23:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1230/2304 [I 2022-10-11 13:23:27] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:22 [I 2022-10-11 13:23:27] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00313 [I 2022-10-11 13:23:27] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-11 13:23:27] ...optimizers/trijoint.py.151: total_norm: 12.913841 [I 2022-10-11 13:23:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1240/2304 [I 2022-10-11 13:23:35] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:11 [I 2022-10-11 13:23:35] ...trap/engines/engine.py.229: process: 0.86196 | load: 0.00312 [I 2022-10-11 13:23:35] ...trap/engines/engine.py.230: loss: 0.10582 [I 2022-10-11 13:23:35] ...optimizers/trijoint.py.151: total_norm: 10.433705 [I 2022-10-11 13:23:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1250/2304 [I 2022-10-11 13:23:44] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:15:04 [I 2022-10-11 13:23:44] ...trap/engines/engine.py.229: process: 0.86151 | load: 0.00312 [I 2022-10-11 13:23:44] ...trap/engines/engine.py.230: loss: 0.12349 [I 2022-10-11 13:23:44] ...optimizers/trijoint.py.151: total_norm: 15.134861 [I 2022-10-11 13:23:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1260/2304 [I 2022-10-11 13:23:53] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:54 [I 2022-10-11 13:23:53] ...trap/engines/engine.py.229: process: 0.82701 | load: 0.00314 [I 2022-10-11 13:23:53] ...trap/engines/engine.py.230: loss: 0.10483 [I 2022-10-11 13:23:53] ...optimizers/trijoint.py.151: total_norm: 12.429297 [I 2022-10-11 13:24:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1270/2304 [I 2022-10-11 13:24:01] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:14:30 [I 2022-10-11 13:24:01] ...trap/engines/engine.py.229: process: 0.83463 | load: 0.00298 [I 2022-10-11 13:24:01] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-11 13:24:01] ...optimizers/trijoint.py.151: total_norm: 19.338755 [I 2022-10-11 13:24:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1280/2304 [I 2022-10-11 13:24:09] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:29 [I 2022-10-11 13:24:09] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00300 [I 2022-10-11 13:24:09] ...trap/engines/engine.py.230: loss: 0.11345 [I 2022-10-11 13:24:10] ...optimizers/trijoint.py.151: total_norm: 22.580498 [I 2022-10-11 13:24:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 1290/2304 [I 2022-10-11 13:24:18] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:22 [I 2022-10-11 13:24:18] ...trap/engines/engine.py.229: process: 0.85164 | load: 0.00304 [I 2022-10-11 13:24:18] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-11 13:24:18] ...optimizers/trijoint.py.151: total_norm: 26.784016 [I 2022-10-11 13:24:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1300/2304 [I 2022-10-11 13:24:27] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:14:14 [I 2022-10-11 13:24:27] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00301 [I 2022-10-11 13:24:27] ...trap/engines/engine.py.230: loss: 0.12609 [I 2022-10-11 13:24:27] ...optimizers/trijoint.py.151: total_norm: 15.075265 [I 2022-10-11 13:24:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1310/2304 [I 2022-10-11 13:24:35] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:13:57 [I 2022-10-11 13:24:35] ...trap/engines/engine.py.229: process: 0.83694 | load: 0.00299 [I 2022-10-11 13:24:35] ...trap/engines/engine.py.230: loss: 0.09464 [I 2022-10-11 13:24:35] ...optimizers/trijoint.py.151: total_norm: 10.582760 [I 2022-10-11 13:24:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1320/2304 [I 2022-10-11 13:24:44] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:13:54 [I 2022-10-11 13:24:44] ...trap/engines/engine.py.229: process: 0.86236 | load: 0.00300 [I 2022-10-11 13:24:44] ...trap/engines/engine.py.230: loss: 0.10704 [I 2022-10-11 13:24:44] ...optimizers/trijoint.py.151: total_norm: 8.996909 [I 2022-10-11 13:24:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1330/2304 [I 2022-10-11 13:24:52] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:13:46 [I 2022-10-11 13:24:52] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00299 [I 2022-10-11 13:24:52] ...trap/engines/engine.py.230: loss: 0.09051 [I 2022-10-11 13:24:52] ...optimizers/trijoint.py.151: total_norm: 12.058002 [I 2022-10-11 13:25:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1340/2304 [I 2022-10-11 13:25:01] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:40 [I 2022-10-11 13:25:01] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00306 [I 2022-10-11 13:25:01] ...trap/engines/engine.py.230: loss: 0.13068 [I 2022-10-11 13:25:01] ...optimizers/trijoint.py.151: total_norm: 22.970968 [I 2022-10-11 13:25:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1350/2304 [I 2022-10-11 13:25:09] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:26 [I 2022-10-11 13:25:09] ...trap/engines/engine.py.229: process: 0.84442 | load: 0.00299 [I 2022-10-11 13:25:09] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-11 13:25:09] ...optimizers/trijoint.py.151: total_norm: 34.489750 [I 2022-10-11 13:25:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1360/2304 [I 2022-10-11 13:25:17] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:16 [I 2022-10-11 13:25:17] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00299 [I 2022-10-11 13:25:17] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-11 13:25:17] ...optimizers/trijoint.py.151: total_norm: 18.920813 [I 2022-10-11 13:25:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 1370/2304 [I 2022-10-11 13:25:26] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:08 [I 2022-10-11 13:25:26] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00299 [I 2022-10-11 13:25:26] ...trap/engines/engine.py.230: loss: 0.08949 [I 2022-10-11 13:25:26] ...optimizers/trijoint.py.151: total_norm: 25.629789 [I 2022-10-11 13:25:34] ...trap/engines/engine.py.224: train: epoch 106 | batch 1380/2304 [I 2022-10-11 13:25:34] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:02 [I 2022-10-11 13:25:34] ...trap/engines/engine.py.229: process: 0.84509 | load: 0.00299 [I 2022-10-11 13:25:34] ...trap/engines/engine.py.230: loss: 0.15466 [I 2022-10-11 13:25:34] ...optimizers/trijoint.py.151: total_norm: 23.937321 [I 2022-10-11 13:25:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 1390/2304 [I 2022-10-11 13:25:43] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:13:02 [I 2022-10-11 13:25:43] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00316 [I 2022-10-11 13:25:43] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-11 13:25:43] ...optimizers/trijoint.py.151: total_norm: 20.693230 [I 2022-10-11 13:25:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1400/2304 [I 2022-10-11 13:25:52] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:12:57 [I 2022-10-11 13:25:52] ...trap/engines/engine.py.229: process: 0.86430 | load: 0.00313 [I 2022-10-11 13:25:52] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-11 13:25:52] ...optimizers/trijoint.py.151: total_norm: 16.724903 [I 2022-10-11 13:26:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 1410/2304 [I 2022-10-11 13:26:00] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:12:46 [I 2022-10-11 13:26:00] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00312 [I 2022-10-11 13:26:00] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-11 13:26:00] ...optimizers/trijoint.py.151: total_norm: 14.132483 [I 2022-10-11 13:26:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1420/2304 [I 2022-10-11 13:26:09] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:44 [I 2022-10-11 13:26:09] ...trap/engines/engine.py.229: process: 0.86906 | load: 0.00316 [I 2022-10-11 13:26:09] ...trap/engines/engine.py.230: loss: 0.14508 [I 2022-10-11 13:26:09] ...optimizers/trijoint.py.151: total_norm: 19.607332 [I 2022-10-11 13:26:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1430/2304 [I 2022-10-11 13:26:17] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:31 [I 2022-10-11 13:26:17] ...trap/engines/engine.py.229: process: 0.86105 | load: 0.00311 [I 2022-10-11 13:26:17] ...trap/engines/engine.py.230: loss: 0.11741 [I 2022-10-11 13:26:17] ...optimizers/trijoint.py.151: total_norm: 12.220752 [I 2022-10-11 13:26:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 1440/2304 [I 2022-10-11 13:26:26] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:12:26 [I 2022-10-11 13:26:26] ...trap/engines/engine.py.229: process: 0.86647 | load: 0.00312 [I 2022-10-11 13:26:26] ...trap/engines/engine.py.230: loss: 0.11218 [I 2022-10-11 13:26:26] ...optimizers/trijoint.py.151: total_norm: 14.192037 [I 2022-10-11 13:26:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1450/2304 [I 2022-10-11 13:26:35] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:12:06 [I 2022-10-11 13:26:35] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00316 [I 2022-10-11 13:26:35] ...trap/engines/engine.py.230: loss: 0.08741 [I 2022-10-11 13:26:35] ...optimizers/trijoint.py.151: total_norm: 15.074636 [I 2022-10-11 13:26:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 1460/2304 [I 2022-10-11 13:26:43] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:12:15 [I 2022-10-11 13:26:43] ...trap/engines/engine.py.229: process: 0.89898 | load: 0.00392 [I 2022-10-11 13:26:43] ...trap/engines/engine.py.230: loss: 0.10802 [I 2022-10-11 13:26:43] ...optimizers/trijoint.py.151: total_norm: 9.518491 [I 2022-10-11 13:26:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1470/2304 [I 2022-10-11 13:26:52] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:12:18 [I 2022-10-11 13:26:52] ...trap/engines/engine.py.229: process: 0.88673 | load: 0.00397 [I 2022-10-11 13:26:52] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-11 13:26:52] ...optimizers/trijoint.py.151: total_norm: 12.623158 [I 2022-10-11 13:27:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1480/2304 [I 2022-10-11 13:27:01] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:12:10 [I 2022-10-11 13:27:01] ...trap/engines/engine.py.229: process: 0.87314 | load: 0.00394 [I 2022-10-11 13:27:01] ...trap/engines/engine.py.230: loss: 0.15484 [I 2022-10-11 13:27:01] ...optimizers/trijoint.py.151: total_norm: 19.785065 [I 2022-10-11 13:27:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1490/2304 [I 2022-10-11 13:27:10] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:11:59 [I 2022-10-11 13:27:10] ...trap/engines/engine.py.229: process: 0.87896 | load: 0.00394 [I 2022-10-11 13:27:10] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-11 13:27:10] ...optimizers/trijoint.py.151: total_norm: 10.937418 [I 2022-10-11 13:27:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1500/2304 [I 2022-10-11 13:27:19] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:50 [I 2022-10-11 13:27:19] ...trap/engines/engine.py.229: process: 0.88088 | load: 0.00395 [I 2022-10-11 13:27:19] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-11 13:27:19] ...optimizers/trijoint.py.151: total_norm: 10.984897 [I 2022-10-11 13:27:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1510/2304 [I 2022-10-11 13:27:28] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:31 [I 2022-10-11 13:27:28] ...trap/engines/engine.py.229: process: 0.85869 | load: 0.00322 [I 2022-10-11 13:27:28] ...trap/engines/engine.py.230: loss: 0.12102 [I 2022-10-11 13:27:28] ...optimizers/trijoint.py.151: total_norm: 25.085440 [I 2022-10-11 13:27:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1520/2304 [I 2022-10-11 13:27:36] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:10 [I 2022-10-11 13:27:36] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00313 [I 2022-10-11 13:27:36] ...trap/engines/engine.py.230: loss: 0.08949 [I 2022-10-11 13:27:36] ...optimizers/trijoint.py.151: total_norm: 8.979628 [I 2022-10-11 13:27:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1530/2304 [I 2022-10-11 13:27:45] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:11:03 [I 2022-10-11 13:27:45] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00315 [I 2022-10-11 13:27:45] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-11 13:27:45] ...optimizers/trijoint.py.151: total_norm: 15.116054 [I 2022-10-11 13:27:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1540/2304 [I 2022-10-11 13:27:53] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:56 [I 2022-10-11 13:27:53] ...trap/engines/engine.py.229: process: 0.86344 | load: 0.00314 [I 2022-10-11 13:27:53] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-11 13:27:53] ...optimizers/trijoint.py.151: total_norm: 9.488549 [I 2022-10-11 13:28:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1550/2304 [I 2022-10-11 13:28:02] ...trap/engines/engine.py.228: elapsed: 0:22:28 | left: 0:10:46 [I 2022-10-11 13:28:02] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00313 [I 2022-10-11 13:28:02] ...trap/engines/engine.py.230: loss: 0.10743 [I 2022-10-11 13:28:02] ...optimizers/trijoint.py.151: total_norm: 10.292508 [I 2022-10-11 13:28:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1560/2304 [I 2022-10-11 13:28:11] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:37 [I 2022-10-11 13:28:11] ...trap/engines/engine.py.229: process: 0.86101 | load: 0.00312 [I 2022-10-11 13:28:11] ...trap/engines/engine.py.230: loss: 0.09274 [I 2022-10-11 13:28:11] ...optimizers/trijoint.py.151: total_norm: 12.601871 [I 2022-10-11 13:28:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1570/2304 [I 2022-10-11 13:28:19] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:29 [I 2022-10-11 13:28:19] ...trap/engines/engine.py.229: process: 0.86912 | load: 0.00312 [I 2022-10-11 13:28:19] ...trap/engines/engine.py.230: loss: 0.11032 [I 2022-10-11 13:28:19] ...optimizers/trijoint.py.151: total_norm: 20.758108 [I 2022-10-11 13:28:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1580/2304 [I 2022-10-11 13:28:28] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:10:20 [I 2022-10-11 13:28:28] ...trap/engines/engine.py.229: process: 0.86116 | load: 0.00314 [I 2022-10-11 13:28:28] ...trap/engines/engine.py.230: loss: 0.13298 [I 2022-10-11 13:28:28] ...optimizers/trijoint.py.151: total_norm: 16.814014 [I 2022-10-11 13:28:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1590/2304 [I 2022-10-11 13:28:36] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:10:13 [I 2022-10-11 13:28:36] ...trap/engines/engine.py.229: process: 0.87264 | load: 0.00312 [I 2022-10-11 13:28:36] ...trap/engines/engine.py.230: loss: 0.19054 [I 2022-10-11 13:28:36] ...optimizers/trijoint.py.151: total_norm: 17.653788 [I 2022-10-11 13:28:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1600/2304 [I 2022-10-11 13:28:45] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:10:07 [I 2022-10-11 13:28:45] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00314 [I 2022-10-11 13:28:45] ...trap/engines/engine.py.230: loss: 0.10806 [I 2022-10-11 13:28:45] ...optimizers/trijoint.py.151: total_norm: 22.153849 [I 2022-10-11 13:28:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 1610/2304 [I 2022-10-11 13:28:54] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:54 [I 2022-10-11 13:28:54] ...trap/engines/engine.py.229: process: 0.83456 | load: 0.00312 [I 2022-10-11 13:28:54] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-11 13:28:54] ...optimizers/trijoint.py.151: total_norm: 7.914356 [I 2022-10-11 13:29:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1620/2304 [I 2022-10-11 13:29:02] ...trap/engines/engine.py.228: elapsed: 0:23:28 | left: 0:09:49 [I 2022-10-11 13:29:02] ...trap/engines/engine.py.229: process: 0.86839 | load: 0.00311 [I 2022-10-11 13:29:02] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-11 13:29:02] ...optimizers/trijoint.py.151: total_norm: 23.471693 [I 2022-10-11 13:29:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1630/2304 [I 2022-10-11 13:29:11] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:42 [I 2022-10-11 13:29:11] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00313 [I 2022-10-11 13:29:11] ...trap/engines/engine.py.230: loss: 0.14565 [I 2022-10-11 13:29:11] ...optimizers/trijoint.py.151: total_norm: 25.470757 [I 2022-10-11 13:29:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1640/2304 [I 2022-10-11 13:29:19] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:34 [I 2022-10-11 13:29:19] ...trap/engines/engine.py.229: process: 0.86657 | load: 0.00309 [I 2022-10-11 13:29:19] ...trap/engines/engine.py.230: loss: 0.13977 [I 2022-10-11 13:29:19] ...optimizers/trijoint.py.151: total_norm: 13.690880 [I 2022-10-11 13:29:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1650/2304 [I 2022-10-11 13:29:28] ...trap/engines/engine.py.228: elapsed: 0:23:54 | left: 0:09:22 [I 2022-10-11 13:29:28] ...trap/engines/engine.py.229: process: 0.87242 | load: 0.00311 [I 2022-10-11 13:29:28] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-11 13:29:28] ...optimizers/trijoint.py.151: total_norm: 28.435854 [I 2022-10-11 13:29:37] ...trap/engines/engine.py.224: train: epoch 106 | batch 1660/2304 [I 2022-10-11 13:29:37] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:09:15 [I 2022-10-11 13:29:37] ...trap/engines/engine.py.229: process: 0.85438 | load: 0.00310 [I 2022-10-11 13:29:37] ...trap/engines/engine.py.230: loss: 0.11027 [I 2022-10-11 13:29:37] ...optimizers/trijoint.py.151: total_norm: 27.095642 [I 2022-10-11 13:29:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1670/2304 [I 2022-10-11 13:29:45] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:09:03 [I 2022-10-11 13:29:45] ...trap/engines/engine.py.229: process: 0.85584 | load: 0.00295 [I 2022-10-11 13:29:45] ...trap/engines/engine.py.230: loss: 0.08565 [I 2022-10-11 13:29:45] ...optimizers/trijoint.py.151: total_norm: 9.874965 [I 2022-10-11 13:29:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 1680/2304 [I 2022-10-11 13:29:54] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:50 [I 2022-10-11 13:29:54] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00294 [I 2022-10-11 13:29:54] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-11 13:29:54] ...optimizers/trijoint.py.151: total_norm: 115.671280 [I 2022-10-11 13:30:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1690/2304 [I 2022-10-11 13:30:02] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:08:38 [I 2022-10-11 13:30:02] ...trap/engines/engine.py.229: process: 0.83910 | load: 0.00293 [I 2022-10-11 13:30:02] ...trap/engines/engine.py.230: loss: 0.15117 [I 2022-10-11 13:30:02] ...optimizers/trijoint.py.151: total_norm: 22.568272 [I 2022-10-11 13:30:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1700/2304 [I 2022-10-11 13:30:11] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:31 [I 2022-10-11 13:30:11] ...trap/engines/engine.py.229: process: 0.84042 | load: 0.00293 [I 2022-10-11 13:30:11] ...trap/engines/engine.py.230: loss: 0.15106 [I 2022-10-11 13:30:11] ...optimizers/trijoint.py.151: total_norm: 24.115719 [I 2022-10-11 13:30:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1710/2304 [I 2022-10-11 13:30:19] ...trap/engines/engine.py.228: elapsed: 0:24:45 | left: 0:08:21 [I 2022-10-11 13:30:19] ...trap/engines/engine.py.229: process: 0.84639 | load: 0.00293 [I 2022-10-11 13:30:19] ...trap/engines/engine.py.230: loss: 0.13232 [I 2022-10-11 13:30:19] ...optimizers/trijoint.py.151: total_norm: 12.120440 [I 2022-10-11 13:30:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1720/2304 [I 2022-10-11 13:30:28] ...trap/engines/engine.py.228: elapsed: 0:24:54 | left: 0:08:14 [I 2022-10-11 13:30:28] ...trap/engines/engine.py.229: process: 0.84765 | load: 0.00296 [I 2022-10-11 13:30:28] ...trap/engines/engine.py.230: loss: 0.14521 [I 2022-10-11 13:30:28] ...optimizers/trijoint.py.151: total_norm: 16.800751 [I 2022-10-11 13:30:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1730/2304 [I 2022-10-11 13:30:36] ...trap/engines/engine.py.228: elapsed: 0:25:02 | left: 0:08:05 [I 2022-10-11 13:30:36] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00294 [I 2022-10-11 13:30:36] ...trap/engines/engine.py.230: loss: 0.08712 [I 2022-10-11 13:30:36] ...optimizers/trijoint.py.151: total_norm: 10.085795 [I 2022-10-11 13:30:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1740/2304 [I 2022-10-11 13:30:45] ...trap/engines/engine.py.228: elapsed: 0:25:11 | left: 0:07:54 [I 2022-10-11 13:30:45] ...trap/engines/engine.py.229: process: 0.83430 | load: 0.00297 [I 2022-10-11 13:30:45] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-11 13:30:45] ...optimizers/trijoint.py.151: total_norm: 12.827729 [I 2022-10-11 13:30:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1750/2304 [I 2022-10-11 13:30:53] ...trap/engines/engine.py.228: elapsed: 0:25:19 | left: 0:07:48 [I 2022-10-11 13:30:53] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00304 [I 2022-10-11 13:30:53] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-11 13:30:53] ...optimizers/trijoint.py.151: total_norm: 8.713362 [I 2022-10-11 13:31:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1760/2304 [I 2022-10-11 13:31:02] ...trap/engines/engine.py.228: elapsed: 0:25:28 | left: 0:07:49 [I 2022-10-11 13:31:02] ...trap/engines/engine.py.229: process: 0.87655 | load: 0.00377 [I 2022-10-11 13:31:02] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-11 13:31:02] ...optimizers/trijoint.py.151: total_norm: 13.215108 [I 2022-10-11 13:31:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1770/2304 [I 2022-10-11 13:31:11] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:45 [I 2022-10-11 13:31:11] ...trap/engines/engine.py.229: process: 0.86753 | load: 0.00378 [I 2022-10-11 13:31:11] ...trap/engines/engine.py.230: loss: 0.12387 [I 2022-10-11 13:31:11] ...optimizers/trijoint.py.151: total_norm: 23.932768 [I 2022-10-11 13:31:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1780/2304 [I 2022-10-11 13:31:19] ...trap/engines/engine.py.228: elapsed: 0:25:45 | left: 0:07:38 [I 2022-10-11 13:31:19] ...trap/engines/engine.py.229: process: 0.87030 | load: 0.00379 [I 2022-10-11 13:31:19] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-11 13:31:19] ...optimizers/trijoint.py.151: total_norm: 47.028175 [I 2022-10-11 13:31:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1790/2304 [I 2022-10-11 13:31:28] ...trap/engines/engine.py.228: elapsed: 0:25:54 | left: 0:07:16 [I 2022-10-11 13:31:28] ...trap/engines/engine.py.229: process: 0.84937 | load: 0.00294 [I 2022-10-11 13:31:28] ...trap/engines/engine.py.230: loss: 0.12769 [I 2022-10-11 13:31:28] ...optimizers/trijoint.py.151: total_norm: 8.322129 [I 2022-10-11 13:31:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1800/2304 [I 2022-10-11 13:31:36] ...trap/engines/engine.py.228: elapsed: 0:26:02 | left: 0:07:05 [I 2022-10-11 13:31:36] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00299 [I 2022-10-11 13:31:36] ...trap/engines/engine.py.230: loss: 0.10078 [I 2022-10-11 13:31:36] ...optimizers/trijoint.py.151: total_norm: 13.421742 [I 2022-10-11 13:31:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1810/2304 [I 2022-10-11 13:31:45] ...trap/engines/engine.py.228: elapsed: 0:26:11 | left: 0:06:58 [I 2022-10-11 13:31:45] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00297 [I 2022-10-11 13:31:45] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-11 13:31:45] ...optimizers/trijoint.py.151: total_norm: 10.986333 [I 2022-10-11 13:31:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1820/2304 [I 2022-10-11 13:31:53] ...trap/engines/engine.py.228: elapsed: 0:26:19 | left: 0:06:47 [I 2022-10-11 13:31:53] ...trap/engines/engine.py.229: process: 0.82971 | load: 0.00299 [I 2022-10-11 13:31:53] ...trap/engines/engine.py.230: loss: 0.08990 [I 2022-10-11 13:31:53] ...optimizers/trijoint.py.151: total_norm: 16.375612 [I 2022-10-11 13:32:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1830/2304 [I 2022-10-11 13:32:02] ...trap/engines/engine.py.228: elapsed: 0:26:28 | left: 0:06:39 [I 2022-10-11 13:32:02] ...trap/engines/engine.py.229: process: 0.83900 | load: 0.00301 [I 2022-10-11 13:32:02] ...trap/engines/engine.py.230: loss: 0.14246 [I 2022-10-11 13:32:02] ...optimizers/trijoint.py.151: total_norm: 26.432486 [I 2022-10-11 13:32:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1840/2304 [I 2022-10-11 13:32:10] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:28 [I 2022-10-11 13:32:10] ...trap/engines/engine.py.229: process: 0.83458 | load: 0.00301 [I 2022-10-11 13:32:10] ...trap/engines/engine.py.230: loss: 0.09643 [I 2022-10-11 13:32:10] ...optimizers/trijoint.py.151: total_norm: 10.015862 [I 2022-10-11 13:32:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 1850/2304 [I 2022-10-11 13:32:18] ...trap/engines/engine.py.228: elapsed: 0:26:45 | left: 0:06:23 [I 2022-10-11 13:32:18] ...trap/engines/engine.py.229: process: 0.84594 | load: 0.00298 [I 2022-10-11 13:32:18] ...trap/engines/engine.py.230: loss: 0.08812 [I 2022-10-11 13:32:19] ...optimizers/trijoint.py.151: total_norm: 12.426596 [I 2022-10-11 13:32:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1860/2304 [I 2022-10-11 13:32:27] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:06:18 [I 2022-10-11 13:32:27] ...trap/engines/engine.py.229: process: 0.84608 | load: 0.00313 [I 2022-10-11 13:32:27] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-11 13:32:27] ...optimizers/trijoint.py.151: total_norm: 13.612972 [I 2022-10-11 13:32:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1870/2304 [I 2022-10-11 13:32:36] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:06:11 [I 2022-10-11 13:32:36] ...trap/engines/engine.py.229: process: 0.86941 | load: 0.00313 [I 2022-10-11 13:32:36] ...trap/engines/engine.py.230: loss: 0.08240 [I 2022-10-11 13:32:36] ...optimizers/trijoint.py.151: total_norm: 8.845105 [I 2022-10-11 13:32:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1880/2304 [I 2022-10-11 13:32:44] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:06:06 [I 2022-10-11 13:32:44] ...trap/engines/engine.py.229: process: 0.86317 | load: 0.00311 [I 2022-10-11 13:32:44] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-11 13:32:44] ...optimizers/trijoint.py.151: total_norm: 10.900803 [I 2022-10-11 13:32:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1890/2304 [I 2022-10-11 13:32:53] ...trap/engines/engine.py.228: elapsed: 0:27:19 | left: 0:05:59 [I 2022-10-11 13:32:53] ...trap/engines/engine.py.229: process: 0.87431 | load: 0.00313 [I 2022-10-11 13:32:53] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-11 13:32:53] ...optimizers/trijoint.py.151: total_norm: 15.525501 [I 2022-10-11 13:33:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1900/2304 [I 2022-10-11 13:33:02] ...trap/engines/engine.py.228: elapsed: 0:27:28 | left: 0:05:48 [I 2022-10-11 13:33:02] ...trap/engines/engine.py.229: process: 0.85208 | load: 0.00310 [I 2022-10-11 13:33:02] ...trap/engines/engine.py.230: loss: 0.07944 [I 2022-10-11 13:33:02] ...optimizers/trijoint.py.151: total_norm: 12.519581 [I 2022-10-11 13:33:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1910/2304 [I 2022-10-11 13:33:10] ...trap/engines/engine.py.228: elapsed: 0:27:36 | left: 0:05:38 [I 2022-10-11 13:33:10] ...trap/engines/engine.py.229: process: 0.86409 | load: 0.00314 [I 2022-10-11 13:33:10] ...trap/engines/engine.py.230: loss: 0.11707 [I 2022-10-11 13:33:10] ...optimizers/trijoint.py.151: total_norm: 8.010822 [I 2022-10-11 13:33:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1920/2304 [I 2022-10-11 13:33:19] ...trap/engines/engine.py.228: elapsed: 0:27:45 | left: 0:05:30 [I 2022-10-11 13:33:19] ...trap/engines/engine.py.229: process: 0.86726 | load: 0.00311 [I 2022-10-11 13:33:19] ...trap/engines/engine.py.230: loss: 0.07430 [I 2022-10-11 13:33:19] ...optimizers/trijoint.py.151: total_norm: 12.139693 [I 2022-10-11 13:33:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1930/2304 [I 2022-10-11 13:33:27] ...trap/engines/engine.py.228: elapsed: 0:27:54 | left: 0:05:21 [I 2022-10-11 13:33:27] ...trap/engines/engine.py.229: process: 0.87819 | load: 0.00312 [I 2022-10-11 13:33:27] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-11 13:33:27] ...optimizers/trijoint.py.151: total_norm: 9.717764 [I 2022-10-11 13:33:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1940/2304 [I 2022-10-11 13:33:36] ...trap/engines/engine.py.228: elapsed: 0:28:02 | left: 0:05:15 [I 2022-10-11 13:33:36] ...trap/engines/engine.py.229: process: 0.85982 | load: 0.00314 [I 2022-10-11 13:33:36] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-11 13:33:36] ...optimizers/trijoint.py.151: total_norm: 16.383341 [I 2022-10-11 13:33:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1950/2304 [I 2022-10-11 13:33:45] ...trap/engines/engine.py.228: elapsed: 0:28:11 | left: 0:05:02 [I 2022-10-11 13:33:45] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00311 [I 2022-10-11 13:33:45] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-11 13:33:45] ...optimizers/trijoint.py.151: total_norm: 11.299085 [I 2022-10-11 13:33:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1960/2304 [I 2022-10-11 13:33:53] ...trap/engines/engine.py.228: elapsed: 0:28:19 | left: 0:04:53 [I 2022-10-11 13:33:53] ...trap/engines/engine.py.229: process: 0.84853 | load: 0.00314 [I 2022-10-11 13:33:53] ...trap/engines/engine.py.230: loss: 0.23347 [I 2022-10-11 13:33:53] ...optimizers/trijoint.py.151: total_norm: 13.107506 [I 2022-10-11 13:34:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1970/2304 [I 2022-10-11 13:34:02] ...trap/engines/engine.py.228: elapsed: 0:28:28 | left: 0:04:44 [I 2022-10-11 13:34:02] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00311 [I 2022-10-11 13:34:02] ...trap/engines/engine.py.230: loss: 0.10223 [I 2022-10-11 13:34:02] ...optimizers/trijoint.py.151: total_norm: 21.198076 [I 2022-10-11 13:34:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1980/2304 [I 2022-10-11 13:34:10] ...trap/engines/engine.py.228: elapsed: 0:28:37 | left: 0:04:38 [I 2022-10-11 13:34:10] ...trap/engines/engine.py.229: process: 0.85718 | load: 0.00313 [I 2022-10-11 13:34:10] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-11 13:34:10] ...optimizers/trijoint.py.151: total_norm: 53.135986 [I 2022-10-11 13:34:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1990/2304 [I 2022-10-11 13:34:19] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:28 [I 2022-10-11 13:34:19] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00314 [I 2022-10-11 13:34:19] ...trap/engines/engine.py.230: loss: 0.16960 [I 2022-10-11 13:34:19] ...optimizers/trijoint.py.151: total_norm: 16.968256 [I 2022-10-11 13:34:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 2000/2304 [I 2022-10-11 13:34:28] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:04:21 [I 2022-10-11 13:34:28] ...trap/engines/engine.py.229: process: 0.85872 | load: 0.00314 [I 2022-10-11 13:34:28] ...trap/engines/engine.py.230: loss: 0.09063 [I 2022-10-11 13:34:28] ...optimizers/trijoint.py.151: total_norm: 9.988869 [I 2022-10-11 13:34:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 2010/2304 [I 2022-10-11 13:34:36] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:12 [I 2022-10-11 13:34:36] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00314 [I 2022-10-11 13:34:36] ...trap/engines/engine.py.230: loss: 0.07566 [I 2022-10-11 13:34:36] ...optimizers/trijoint.py.151: total_norm: 13.439833 [I 2022-10-11 13:34:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 2020/2304 [I 2022-10-11 13:34:45] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:04:02 [I 2022-10-11 13:34:45] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00314 [I 2022-10-11 13:34:45] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-11 13:34:45] ...optimizers/trijoint.py.151: total_norm: 22.037815 [I 2022-10-11 13:34:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 2030/2304 [I 2022-10-11 13:34:53] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:03:53 [I 2022-10-11 13:34:53] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00311 [I 2022-10-11 13:34:53] ...trap/engines/engine.py.230: loss: 0.09646 [I 2022-10-11 13:34:53] ...optimizers/trijoint.py.151: total_norm: 13.714240 [I 2022-10-11 13:35:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 2040/2304 [I 2022-10-11 13:35:02] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:46 [I 2022-10-11 13:35:02] ...trap/engines/engine.py.229: process: 0.85640 | load: 0.00315 [I 2022-10-11 13:35:02] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-11 13:35:02] ...optimizers/trijoint.py.151: total_norm: 36.839897 [I 2022-10-11 13:35:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 2050/2304 [I 2022-10-11 13:35:10] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:37 [I 2022-10-11 13:35:10] ...trap/engines/engine.py.229: process: 0.86078 | load: 0.00313 [I 2022-10-11 13:35:10] ...trap/engines/engine.py.230: loss: 0.08868 [I 2022-10-11 13:35:10] ...optimizers/trijoint.py.151: total_norm: 12.676006 [I 2022-10-11 13:35:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 2060/2304 [I 2022-10-11 13:35:19] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:30 [I 2022-10-11 13:35:19] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00313 [I 2022-10-11 13:35:19] ...trap/engines/engine.py.230: loss: 0.13140 [I 2022-10-11 13:35:19] ...optimizers/trijoint.py.151: total_norm: 9.856877 [I 2022-10-11 13:35:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 2070/2304 [I 2022-10-11 13:35:28] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:21 [I 2022-10-11 13:35:28] ...trap/engines/engine.py.229: process: 0.87039 | load: 0.00314 [I 2022-10-11 13:35:28] ...trap/engines/engine.py.230: loss: 0.13073 [I 2022-10-11 13:35:28] ...optimizers/trijoint.py.151: total_norm: 20.651276 [I 2022-10-11 13:35:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 2080/2304 [I 2022-10-11 13:35:36] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:11 [I 2022-10-11 13:35:36] ...trap/engines/engine.py.229: process: 0.85863 | load: 0.00313 [I 2022-10-11 13:35:36] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-11 13:35:36] ...optimizers/trijoint.py.151: total_norm: 11.424602 [I 2022-10-11 13:35:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 2090/2304 [I 2022-10-11 13:35:45] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:03:03 [I 2022-10-11 13:35:45] ...trap/engines/engine.py.229: process: 0.86686 | load: 0.00317 [I 2022-10-11 13:35:45] ...trap/engines/engine.py.230: loss: 0.09677 [I 2022-10-11 13:35:45] ...optimizers/trijoint.py.151: total_norm: 8.582954 [I 2022-10-11 13:35:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 2100/2304 [I 2022-10-11 13:35:53] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:55 [I 2022-10-11 13:35:53] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00311 [I 2022-10-11 13:35:53] ...trap/engines/engine.py.230: loss: 0.08763 [I 2022-10-11 13:35:53] ...optimizers/trijoint.py.151: total_norm: 31.927086 [I 2022-10-11 13:36:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 2110/2304 [I 2022-10-11 13:36:02] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:46 [I 2022-10-11 13:36:02] ...trap/engines/engine.py.229: process: 0.86082 | load: 0.00318 [I 2022-10-11 13:36:02] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-11 13:36:02] ...optimizers/trijoint.py.151: total_norm: 18.295401 [I 2022-10-11 13:36:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 2120/2304 [I 2022-10-11 13:36:11] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:37 [I 2022-10-11 13:36:11] ...trap/engines/engine.py.229: process: 0.84867 | load: 0.00312 [I 2022-10-11 13:36:11] ...trap/engines/engine.py.230: loss: 0.08976 [I 2022-10-11 13:36:11] ...optimizers/trijoint.py.151: total_norm: 13.774085 [I 2022-10-11 13:36:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 2130/2304 [I 2022-10-11 13:36:19] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:26 [I 2022-10-11 13:36:19] ...trap/engines/engine.py.229: process: 0.85505 | load: 0.00307 [I 2022-10-11 13:36:19] ...trap/engines/engine.py.230: loss: 0.18441 [I 2022-10-11 13:36:19] ...optimizers/trijoint.py.151: total_norm: 30.997051 [I 2022-10-11 13:36:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 2140/2304 [I 2022-10-11 13:36:27] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:17 [I 2022-10-11 13:36:27] ...trap/engines/engine.py.229: process: 0.83715 | load: 0.00295 [I 2022-10-11 13:36:27] ...trap/engines/engine.py.230: loss: 0.08920 [I 2022-10-11 13:36:27] ...optimizers/trijoint.py.151: total_norm: 22.878225 [I 2022-10-11 13:36:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 2150/2304 [I 2022-10-11 13:36:36] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:02:10 [I 2022-10-11 13:36:36] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00297 [I 2022-10-11 13:36:36] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-11 13:36:36] ...optimizers/trijoint.py.151: total_norm: 11.415561 [I 2022-10-11 13:36:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 2160/2304 [I 2022-10-11 13:36:44] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:02:01 [I 2022-10-11 13:36:44] ...trap/engines/engine.py.229: process: 0.83937 | load: 0.00294 [I 2022-10-11 13:36:44] ...trap/engines/engine.py.230: loss: 0.09578 [I 2022-10-11 13:36:44] ...optimizers/trijoint.py.151: total_norm: 12.811403 [I 2022-10-11 13:36:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 2170/2304 [I 2022-10-11 13:36:53] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:52 [I 2022-10-11 13:36:53] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00296 [I 2022-10-11 13:36:53] ...trap/engines/engine.py.230: loss: 0.08736 [I 2022-10-11 13:36:53] ...optimizers/trijoint.py.151: total_norm: 11.867548 [I 2022-10-11 13:37:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 2180/2304 [I 2022-10-11 13:37:01] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:43 [I 2022-10-11 13:37:01] ...trap/engines/engine.py.229: process: 0.82720 | load: 0.00294 [I 2022-10-11 13:37:01] ...trap/engines/engine.py.230: loss: 0.11186 [I 2022-10-11 13:37:01] ...optimizers/trijoint.py.151: total_norm: 12.414742 [I 2022-10-11 13:37:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 2190/2304 [I 2022-10-11 13:37:10] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:35 [I 2022-10-11 13:37:10] ...trap/engines/engine.py.229: process: 0.83616 | load: 0.00297 [I 2022-10-11 13:37:10] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-11 13:37:10] ...optimizers/trijoint.py.151: total_norm: 14.443971 [I 2022-10-11 13:37:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 2200/2304 [I 2022-10-11 13:37:18] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:27 [I 2022-10-11 13:37:18] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00293 [I 2022-10-11 13:37:18] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 13:37:18] ...optimizers/trijoint.py.151: total_norm: 11.817257 [I 2022-10-11 13:37:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 2210/2304 [I 2022-10-11 13:37:27] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:19 [I 2022-10-11 13:37:27] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00296 [I 2022-10-11 13:37:27] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-11 13:37:27] ...optimizers/trijoint.py.151: total_norm: 16.882572 [I 2022-10-11 13:37:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 2220/2304 [I 2022-10-11 13:37:35] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:10 [I 2022-10-11 13:37:35] ...trap/engines/engine.py.229: process: 0.83286 | load: 0.00298 [I 2022-10-11 13:37:35] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-11 13:37:35] ...optimizers/trijoint.py.151: total_norm: 13.767700 [I 2022-10-11 13:37:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 2230/2304 [I 2022-10-11 13:37:44] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:03 [I 2022-10-11 13:37:44] ...trap/engines/engine.py.229: process: 0.87181 | load: 0.00302 [I 2022-10-11 13:37:44] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-11 13:37:44] ...optimizers/trijoint.py.151: total_norm: 21.811501 [I 2022-10-11 13:37:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 2240/2304 [I 2022-10-11 13:37:52] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:00:54 [I 2022-10-11 13:37:52] ...trap/engines/engine.py.229: process: 0.85156 | load: 0.00300 [I 2022-10-11 13:37:52] ...trap/engines/engine.py.230: loss: 0.19713 [I 2022-10-11 13:37:52] ...optimizers/trijoint.py.151: total_norm: 29.504950 [I 2022-10-11 13:38:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 2250/2304 [I 2022-10-11 13:38:01] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:45 [I 2022-10-11 13:38:01] ...trap/engines/engine.py.229: process: 0.82880 | load: 0.00296 [I 2022-10-11 13:38:01] ...trap/engines/engine.py.230: loss: 0.09383 [I 2022-10-11 13:38:01] ...optimizers/trijoint.py.151: total_norm: 8.088948 [I 2022-10-11 13:38:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 2260/2304 [I 2022-10-11 13:38:09] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:37 [I 2022-10-11 13:38:09] ...trap/engines/engine.py.229: process: 0.83717 | load: 0.00296 [I 2022-10-11 13:38:09] ...trap/engines/engine.py.230: loss: 0.20969 [I 2022-10-11 13:38:09] ...optimizers/trijoint.py.151: total_norm: 18.402510 [I 2022-10-11 13:38:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 2270/2304 [I 2022-10-11 13:38:18] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:28 [I 2022-10-11 13:38:18] ...trap/engines/engine.py.229: process: 0.84200 | load: 0.00299 [I 2022-10-11 13:38:18] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-11 13:38:18] ...optimizers/trijoint.py.151: total_norm: 14.546533 [I 2022-10-11 13:38:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 2280/2304 [I 2022-10-11 13:38:26] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:20 [I 2022-10-11 13:38:26] ...trap/engines/engine.py.229: process: 0.82460 | load: 0.00299 [I 2022-10-11 13:38:26] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-11 13:38:26] ...optimizers/trijoint.py.151: total_norm: 76.751076 [I 2022-10-11 13:38:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 2290/2304 [I 2022-10-11 13:38:35] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:11 [I 2022-10-11 13:38:35] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00299 [I 2022-10-11 13:38:35] ...trap/engines/engine.py.230: loss: 0.13182 [I 2022-10-11 13:38:35] ...optimizers/trijoint.py.151: total_norm: 24.432484 [I 2022-10-11 13:38:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 2300/2304 [I 2022-10-11 13:38:43] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:00:03 [I 2022-10-11 13:38:43] ...trap/engines/engine.py.229: process: 0.82766 | load: 0.00015 [I 2022-10-11 13:38:43] ...trap/engines/engine.py.230: loss: 0.13184 [I 2022-10-11 13:38:43] ...optimizers/trijoint.py.151: total_norm: 42.398598 [I 2022-10-11 13:38:46] ...trap/engines/engine.py.224: train: epoch 106 | batch 2304/2304 [I 2022-10-11 13:38:46] ...trap/engines/engine.py.228: elapsed: 0:33:12 | left: 0:00:00 [I 2022-10-11 13:38:46] ...trap/engines/engine.py.229: process: 0.83571 | load: 0.00018 [I 2022-10-11 13:38:46] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-11 13:38:46] ...optimizers/trijoint.py.151: total_norm: 10.979403 [S 2022-10-11 13:38:48] ...trap/engines/engine.py.129: train_epoch.epoch: 106 [S 2022-10-11 13:38:48] ...trap/engines/engine.py.129: train_epoch.loss: 0.12094243936192188 [I 2022-10-11 13:38:48] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 13:38:49] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 106 [I 2022-10-11 13:38:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 13:38:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 13:38:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 13:38:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 13:38:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 13:38:55] ...trap/engines/engine.py.321: eval: epoch 106 | batch 0/510 [I 2022-10-11 13:38:55] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:50:53 [I 2022-10-11 13:38:55] ...trap/engines/engine.py.326: process: 5.98709 | load: 2.12206 [I 2022-10-11 13:39:01] ...trap/engines/engine.py.321: eval: epoch 106 | batch 10/510 [I 2022-10-11 13:39:01] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:42 [I 2022-10-11 13:39:01] ...trap/engines/engine.py.326: process: 0.70500 | load: 0.00018 [I 2022-10-11 13:39:06] ...trap/engines/engine.py.321: eval: epoch 106 | batch 20/510 [I 2022-10-11 13:39:06] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:04:54 [I 2022-10-11 13:39:06] ...trap/engines/engine.py.326: process: 0.41991 | load: 0.00019 [I 2022-10-11 13:39:12] ...trap/engines/engine.py.321: eval: epoch 106 | batch 30/510 [I 2022-10-11 13:39:12] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:10 [I 2022-10-11 13:39:12] ...trap/engines/engine.py.326: process: 0.42075 | load: 0.00019 [I 2022-10-11 13:39:18] ...trap/engines/engine.py.321: eval: epoch 106 | batch 40/510 [I 2022-10-11 13:39:18] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:04:41 [I 2022-10-11 13:39:18] ...trap/engines/engine.py.326: process: 0.43629 | load: 0.00018 [I 2022-10-11 13:39:22] ...trap/engines/engine.py.321: eval: epoch 106 | batch 50/510 [I 2022-10-11 13:39:22] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:24 [I 2022-10-11 13:39:22] ...trap/engines/engine.py.326: process: 0.42310 | load: 0.00018 [I 2022-10-11 13:39:27] ...trap/engines/engine.py.321: eval: epoch 106 | batch 60/510 [I 2022-10-11 13:39:27] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:36 [I 2022-10-11 13:39:27] ...trap/engines/engine.py.326: process: 0.52457 | load: 0.00019 [I 2022-10-11 13:39:32] ...trap/engines/engine.py.321: eval: epoch 106 | batch 70/510 [I 2022-10-11 13:39:32] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:53 [I 2022-10-11 13:39:32] ...trap/engines/engine.py.326: process: 0.58224 | load: 0.00019 [I 2022-10-11 13:39:37] ...trap/engines/engine.py.321: eval: epoch 106 | batch 80/510 [I 2022-10-11 13:39:37] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:36 [I 2022-10-11 13:39:37] ...trap/engines/engine.py.326: process: 0.45289 | load: 0.00020 [I 2022-10-11 13:39:42] ...trap/engines/engine.py.321: eval: epoch 106 | batch 90/510 [I 2022-10-11 13:39:42] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:09 [I 2022-10-11 13:39:42] ...trap/engines/engine.py.326: process: 0.45428 | load: 0.00019 [I 2022-10-11 13:39:46] ...trap/engines/engine.py.321: eval: epoch 106 | batch 100/510 [I 2022-10-11 13:39:46] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:50 [I 2022-10-11 13:39:46] ...trap/engines/engine.py.326: process: 0.31407 | load: 0.00019 [I 2022-10-11 13:39:49] ...trap/engines/engine.py.321: eval: epoch 106 | batch 110/510 [I 2022-10-11 13:39:49] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:10 [I 2022-10-11 13:39:49] ...trap/engines/engine.py.326: process: 0.30991 | load: 0.00020 [I 2022-10-11 13:39:53] ...trap/engines/engine.py.321: eval: epoch 106 | batch 120/510 [I 2022-10-11 13:39:53] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:02:04 [I 2022-10-11 13:39:53] ...trap/engines/engine.py.326: process: 0.31556 | load: 0.00019 [I 2022-10-11 13:39:56] ...trap/engines/engine.py.321: eval: epoch 106 | batch 130/510 [I 2022-10-11 13:39:56] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:00 [I 2022-10-11 13:39:56] ...trap/engines/engine.py.326: process: 0.31744 | load: 0.00020 [I 2022-10-11 13:39:59] ...trap/engines/engine.py.321: eval: epoch 106 | batch 140/510 [I 2022-10-11 13:39:59] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:57 [I 2022-10-11 13:39:59] ...trap/engines/engine.py.326: process: 0.31269 | load: 0.00019 [I 2022-10-11 13:40:02] ...trap/engines/engine.py.321: eval: epoch 106 | batch 150/510 [I 2022-10-11 13:40:02] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:53 [I 2022-10-11 13:40:02] ...trap/engines/engine.py.326: process: 0.31096 | load: 0.00020 [I 2022-10-11 13:40:05] ...trap/engines/engine.py.321: eval: epoch 106 | batch 160/510 [I 2022-10-11 13:40:05] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:50 [I 2022-10-11 13:40:05] ...trap/engines/engine.py.326: process: 0.30889 | load: 0.00022 [I 2022-10-11 13:40:08] ...trap/engines/engine.py.321: eval: epoch 106 | batch 170/510 [I 2022-10-11 13:40:08] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:48 [I 2022-10-11 13:40:08] ...trap/engines/engine.py.326: process: 0.31481 | load: 0.00022 [I 2022-10-11 13:40:12] ...trap/engines/engine.py.321: eval: epoch 106 | batch 180/510 [I 2022-10-11 13:40:12] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:45 [I 2022-10-11 13:40:12] ...trap/engines/engine.py.326: process: 0.31322 | load: 0.00022 [I 2022-10-11 13:40:15] ...trap/engines/engine.py.321: eval: epoch 106 | batch 190/510 [I 2022-10-11 13:40:15] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:42 [I 2022-10-11 13:40:15] ...trap/engines/engine.py.326: process: 0.32748 | load: 0.00021 [I 2022-10-11 13:40:18] ...trap/engines/engine.py.321: eval: epoch 106 | batch 200/510 [I 2022-10-11 13:40:18] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:38 [I 2022-10-11 13:40:18] ...trap/engines/engine.py.326: process: 0.31782 | load: 0.00022 [I 2022-10-11 13:40:21] ...trap/engines/engine.py.321: eval: epoch 106 | batch 210/510 [I 2022-10-11 13:40:21] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:35 [I 2022-10-11 13:40:21] ...trap/engines/engine.py.326: process: 0.31673 | load: 0.00022 [I 2022-10-11 13:40:24] ...trap/engines/engine.py.321: eval: epoch 106 | batch 220/510 [I 2022-10-11 13:40:24] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:33 [I 2022-10-11 13:40:24] ...trap/engines/engine.py.326: process: 0.32752 | load: 0.00022 [I 2022-10-11 13:40:28] ...trap/engines/engine.py.321: eval: epoch 106 | batch 230/510 [I 2022-10-11 13:40:28] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:29 [I 2022-10-11 13:40:28] ...trap/engines/engine.py.326: process: 0.31963 | load: 0.00021 [I 2022-10-11 13:40:31] ...trap/engines/engine.py.321: eval: epoch 106 | batch 240/510 [I 2022-10-11 13:40:31] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:26 [I 2022-10-11 13:40:31] ...trap/engines/engine.py.326: process: 0.32497 | load: 0.00022 [I 2022-10-11 13:40:34] ...trap/engines/engine.py.321: eval: epoch 106 | batch 250/510 [I 2022-10-11 13:40:34] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:23 [I 2022-10-11 13:40:34] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00022 [I 2022-10-11 13:40:37] ...trap/engines/engine.py.321: eval: epoch 106 | batch 260/510 [I 2022-10-11 13:40:37] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:20 [I 2022-10-11 13:40:37] ...trap/engines/engine.py.326: process: 0.31997 | load: 0.00022 [I 2022-10-11 13:40:40] ...trap/engines/engine.py.321: eval: epoch 106 | batch 270/510 [I 2022-10-11 13:40:40] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:16 [I 2022-10-11 13:40:40] ...trap/engines/engine.py.326: process: 0.31777 | load: 0.00022 [I 2022-10-11 13:40:44] ...trap/engines/engine.py.321: eval: epoch 106 | batch 280/510 [I 2022-10-11 13:40:44] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:13 [I 2022-10-11 13:40:44] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00023 [I 2022-10-11 13:40:47] ...trap/engines/engine.py.321: eval: epoch 106 | batch 290/510 [I 2022-10-11 13:40:47] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:10 [I 2022-10-11 13:40:47] ...trap/engines/engine.py.326: process: 0.32398 | load: 0.00022 [I 2022-10-11 13:40:50] ...trap/engines/engine.py.321: eval: epoch 106 | batch 300/510 [I 2022-10-11 13:40:50] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:07 [I 2022-10-11 13:40:50] ...trap/engines/engine.py.326: process: 0.32076 | load: 0.00022 [I 2022-10-11 13:40:53] ...trap/engines/engine.py.321: eval: epoch 106 | batch 310/510 [I 2022-10-11 13:40:53] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:04 [I 2022-10-11 13:40:53] ...trap/engines/engine.py.326: process: 0.32133 | load: 0.00021 [I 2022-10-11 13:40:57] ...trap/engines/engine.py.321: eval: epoch 106 | batch 320/510 [I 2022-10-11 13:40:57] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:01 [I 2022-10-11 13:40:57] ...trap/engines/engine.py.326: process: 0.33669 | load: 0.00022 [I 2022-10-11 13:41:00] ...trap/engines/engine.py.321: eval: epoch 106 | batch 330/510 [I 2022-10-11 13:41:00] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:58 [I 2022-10-11 13:41:00] ...trap/engines/engine.py.326: process: 0.32519 | load: 0.00022 [I 2022-10-11 13:41:03] ...trap/engines/engine.py.321: eval: epoch 106 | batch 340/510 [I 2022-10-11 13:41:03] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:54 [I 2022-10-11 13:41:03] ...trap/engines/engine.py.326: process: 0.31841 | load: 0.00025 [I 2022-10-11 13:41:06] ...trap/engines/engine.py.321: eval: epoch 106 | batch 350/510 [I 2022-10-11 13:41:06] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:51 [I 2022-10-11 13:41:06] ...trap/engines/engine.py.326: process: 0.32325 | load: 0.00022 [I 2022-10-11 13:41:09] ...trap/engines/engine.py.321: eval: epoch 106 | batch 360/510 [I 2022-10-11 13:41:09] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:48 [I 2022-10-11 13:41:09] ...trap/engines/engine.py.326: process: 0.31615 | load: 0.00022 [I 2022-10-11 13:41:13] ...trap/engines/engine.py.321: eval: epoch 106 | batch 370/510 [I 2022-10-11 13:41:13] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:44 [I 2022-10-11 13:41:13] ...trap/engines/engine.py.326: process: 0.31354 | load: 0.00021 [I 2022-10-11 13:41:16] ...trap/engines/engine.py.321: eval: epoch 106 | batch 380/510 [I 2022-10-11 13:41:16] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-11 13:41:16] ...trap/engines/engine.py.326: process: 0.32507 | load: 0.00022 [I 2022-10-11 13:41:19] ...trap/engines/engine.py.321: eval: epoch 106 | batch 390/510 [I 2022-10-11 13:41:19] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-11 13:41:19] ...trap/engines/engine.py.326: process: 0.32383 | load: 0.00023 [I 2022-10-11 13:41:22] ...trap/engines/engine.py.321: eval: epoch 106 | batch 400/510 [I 2022-10-11 13:41:22] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:35 [I 2022-10-11 13:41:22] ...trap/engines/engine.py.326: process: 0.31582 | load: 0.00021 [I 2022-10-11 13:41:26] ...trap/engines/engine.py.321: eval: epoch 106 | batch 410/510 [I 2022-10-11 13:41:26] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:32 [I 2022-10-11 13:41:26] ...trap/engines/engine.py.326: process: 0.31682 | load: 0.00024 [I 2022-10-11 13:41:29] ...trap/engines/engine.py.321: eval: epoch 106 | batch 420/510 [I 2022-10-11 13:41:29] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:29 [I 2022-10-11 13:41:29] ...trap/engines/engine.py.326: process: 0.32302 | load: 0.00024 [I 2022-10-11 13:41:32] ...trap/engines/engine.py.321: eval: epoch 106 | batch 430/510 [I 2022-10-11 13:41:32] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:26 [I 2022-10-11 13:41:32] ...trap/engines/engine.py.326: process: 0.32877 | load: 0.00022 [I 2022-10-11 13:41:35] ...trap/engines/engine.py.321: eval: epoch 106 | batch 440/510 [I 2022-10-11 13:41:35] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:22 [I 2022-10-11 13:41:35] ...trap/engines/engine.py.326: process: 0.32058 | load: 0.00021 [I 2022-10-11 13:41:39] ...trap/engines/engine.py.321: eval: epoch 106 | batch 450/510 [I 2022-10-11 13:41:39] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:19 [I 2022-10-11 13:41:39] ...trap/engines/engine.py.326: process: 0.31439 | load: 0.00024 [I 2022-10-11 13:41:42] ...trap/engines/engine.py.321: eval: epoch 106 | batch 460/510 [I 2022-10-11 13:41:42] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:16 [I 2022-10-11 13:41:42] ...trap/engines/engine.py.326: process: 0.32341 | load: 0.00022 [I 2022-10-11 13:41:45] ...trap/engines/engine.py.321: eval: epoch 106 | batch 470/510 [I 2022-10-11 13:41:45] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:12 [I 2022-10-11 13:41:45] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00023 [I 2022-10-11 13:41:48] ...trap/engines/engine.py.321: eval: epoch 106 | batch 480/510 [I 2022-10-11 13:41:48] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-11 13:41:48] ...trap/engines/engine.py.326: process: 0.32200 | load: 0.00022 [I 2022-10-11 13:41:52] ...trap/engines/engine.py.321: eval: epoch 106 | batch 490/510 [I 2022-10-11 13:41:52] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:06 [I 2022-10-11 13:41:52] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00023 [I 2022-10-11 13:41:55] ...trap/engines/engine.py.321: eval: epoch 106 | batch 500/510 [I 2022-10-11 13:41:55] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:03 [I 2022-10-11 13:41:55] ...trap/engines/engine.py.326: process: 0.32535 | load: 0.00021 [I 2022-10-11 13:41:58] ...trap/engines/engine.py.321: eval: epoch 106 | batch 510/510 [I 2022-10-11 13:41:58] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:00 [I 2022-10-11 13:41:58] ...trap/engines/engine.py.326: process: 0.31274 | load: 0.00017 [S 2022-10-11 13:41:59] ...trap/engines/engine.py.132: eval_epoch.epoch: 106 [I 2022-10-11 13:41:59] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.743900 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908300 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935400 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.739100 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907700 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935600 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016337 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008707 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009489 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010578 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009778 [S 2022-10-11 13:42:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010947 [I 2022-10-11 13:42:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 13:42:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 13:42:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 13:42:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 13:42:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 13:42:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 13:42:50] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 13:42:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 13:42:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 107 [I 2022-10-11 13:42:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 13:42:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 0/2304 [I 2022-10-11 13:42:57] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:08:24 [I 2022-10-11 13:42:57] ...trap/engines/engine.py.229: process: 3.34381 | load: 2.41732 [I 2022-10-11 13:42:57] ...trap/engines/engine.py.230: loss: 0.08754 [I 2022-10-11 13:42:57] ...optimizers/trijoint.py.151: total_norm: 16.423950 [I 2022-10-11 13:43:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 10/2304 [I 2022-10-11 13:43:06] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:43:17 [I 2022-10-11 13:43:06] ...trap/engines/engine.py.229: process: 0.87538 | load: 0.00582 [I 2022-10-11 13:43:06] ...trap/engines/engine.py.230: loss: 0.12828 [I 2022-10-11 13:43:06] ...optimizers/trijoint.py.151: total_norm: 16.272779 [I 2022-10-11 13:43:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 20/2304 [I 2022-10-11 13:43:15] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:59 [I 2022-10-11 13:43:15] ...trap/engines/engine.py.229: process: 0.87083 | load: 0.00437 [I 2022-10-11 13:43:15] ...trap/engines/engine.py.230: loss: 0.09587 [I 2022-10-11 13:43:15] ...optimizers/trijoint.py.151: total_norm: 13.902131 [I 2022-10-11 13:43:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 30/2304 [I 2022-10-11 13:43:23] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:37 [I 2022-10-11 13:43:23] ...trap/engines/engine.py.229: process: 0.86040 | load: 0.00365 [I 2022-10-11 13:43:23] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-11 13:43:23] ...optimizers/trijoint.py.151: total_norm: 10.123940 [I 2022-10-11 13:43:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 40/2304 [I 2022-10-11 13:43:32] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:39 [I 2022-10-11 13:43:32] ...trap/engines/engine.py.229: process: 0.86693 | load: 0.00391 [I 2022-10-11 13:43:32] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-11 13:43:32] ...optimizers/trijoint.py.151: total_norm: 31.541260 [I 2022-10-11 13:43:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 50/2304 [I 2022-10-11 13:43:41] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:27 [I 2022-10-11 13:43:41] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00448 [I 2022-10-11 13:43:41] ...trap/engines/engine.py.230: loss: 0.15002 [I 2022-10-11 13:43:41] ...optimizers/trijoint.py.151: total_norm: 54.285789 [I 2022-10-11 13:43:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 60/2304 [I 2022-10-11 13:43:49] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:17 [I 2022-10-11 13:43:49] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00385 [I 2022-10-11 13:43:49] ...trap/engines/engine.py.230: loss: 0.10166 [I 2022-10-11 13:43:49] ...optimizers/trijoint.py.151: total_norm: 17.893820 [I 2022-10-11 13:43:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 70/2304 [I 2022-10-11 13:43:58] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:16 [I 2022-10-11 13:43:58] ...trap/engines/engine.py.229: process: 0.87668 | load: 0.00320 [I 2022-10-11 13:43:58] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-11 13:43:58] ...optimizers/trijoint.py.151: total_norm: 13.083506 [I 2022-10-11 13:44:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 80/2304 [I 2022-10-11 13:44:07] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:30 [I 2022-10-11 13:44:07] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00405 [I 2022-10-11 13:44:07] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-11 13:44:07] ...optimizers/trijoint.py.151: total_norm: 10.941242 [I 2022-10-11 13:44:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 90/2304 [I 2022-10-11 13:44:15] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:27 [I 2022-10-11 13:44:15] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00375 [I 2022-10-11 13:44:15] ...trap/engines/engine.py.230: loss: 0.09521 [I 2022-10-11 13:44:15] ...optimizers/trijoint.py.151: total_norm: 9.900832 [I 2022-10-11 13:44:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 100/2304 [I 2022-10-11 13:44:24] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:09 [I 2022-10-11 13:44:24] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00324 [I 2022-10-11 13:44:24] ...trap/engines/engine.py.230: loss: 0.10399 [I 2022-10-11 13:44:24] ...optimizers/trijoint.py.151: total_norm: 14.453641 [I 2022-10-11 13:44:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 110/2304 [I 2022-10-11 13:44:32] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:30:58 [I 2022-10-11 13:44:32] ...trap/engines/engine.py.229: process: 0.85606 | load: 0.00310 [I 2022-10-11 13:44:32] ...trap/engines/engine.py.230: loss: 0.10996 [I 2022-10-11 13:44:32] ...optimizers/trijoint.py.151: total_norm: 15.641827 [I 2022-10-11 13:44:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 120/2304 [I 2022-10-11 13:44:40] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:30:48 [I 2022-10-11 13:44:40] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00306 [I 2022-10-11 13:44:40] ...trap/engines/engine.py.230: loss: 0.10332 [I 2022-10-11 13:44:40] ...optimizers/trijoint.py.151: total_norm: 8.569490 [I 2022-10-11 13:44:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 130/2304 [I 2022-10-11 13:44:49] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:41 [I 2022-10-11 13:44:49] ...trap/engines/engine.py.229: process: 0.86061 | load: 0.00327 [I 2022-10-11 13:44:49] ...trap/engines/engine.py.230: loss: 0.09934 [I 2022-10-11 13:44:49] ...optimizers/trijoint.py.151: total_norm: 13.572730 [I 2022-10-11 13:44:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 140/2304 [I 2022-10-11 13:44:57] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:21 [I 2022-10-11 13:44:57] ...trap/engines/engine.py.229: process: 0.84740 | load: 0.00329 [I 2022-10-11 13:44:57] ...trap/engines/engine.py.230: loss: 0.15717 [I 2022-10-11 13:44:57] ...optimizers/trijoint.py.151: total_norm: 8.529001 [I 2022-10-11 13:45:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 150/2304 [I 2022-10-11 13:45:06] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:05 [I 2022-10-11 13:45:06] ...trap/engines/engine.py.229: process: 0.83467 | load: 0.00305 [I 2022-10-11 13:45:06] ...trap/engines/engine.py.230: loss: 0.08194 [I 2022-10-11 13:45:06] ...optimizers/trijoint.py.151: total_norm: 13.248179 [I 2022-10-11 13:45:14] ...trap/engines/engine.py.224: train: epoch 107 | batch 160/2304 [I 2022-10-11 13:45:14] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:09 [I 2022-10-11 13:45:14] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00303 [I 2022-10-11 13:45:14] ...trap/engines/engine.py.230: loss: 0.14327 [I 2022-10-11 13:45:14] ...optimizers/trijoint.py.151: total_norm: 12.507261 [I 2022-10-11 13:45:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 170/2304 [I 2022-10-11 13:45:23] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:13 [I 2022-10-11 13:45:23] ...trap/engines/engine.py.229: process: 0.85714 | load: 0.00303 [I 2022-10-11 13:45:23] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-11 13:45:23] ...optimizers/trijoint.py.151: total_norm: 10.444439 [I 2022-10-11 13:45:31] ...trap/engines/engine.py.224: train: epoch 107 | batch 180/2304 [I 2022-10-11 13:45:31] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:16 [I 2022-10-11 13:45:31] ...trap/engines/engine.py.229: process: 0.84191 | load: 0.00294 [I 2022-10-11 13:45:31] ...trap/engines/engine.py.230: loss: 0.10854 [I 2022-10-11 13:45:31] ...optimizers/trijoint.py.151: total_norm: 13.061125 [I 2022-10-11 13:45:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 190/2304 [I 2022-10-11 13:45:40] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:02 [I 2022-10-11 13:45:40] ...trap/engines/engine.py.229: process: 0.85898 | load: 0.00309 [I 2022-10-11 13:45:40] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-11 13:45:40] ...optimizers/trijoint.py.151: total_norm: 16.755653 [I 2022-10-11 13:45:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 200/2304 [I 2022-10-11 13:45:48] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:42 [I 2022-10-11 13:45:48] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00308 [I 2022-10-11 13:45:48] ...trap/engines/engine.py.230: loss: 0.13679 [I 2022-10-11 13:45:48] ...optimizers/trijoint.py.151: total_norm: 9.144351 [I 2022-10-11 13:45:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 210/2304 [I 2022-10-11 13:45:57] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:37 [I 2022-10-11 13:45:57] ...trap/engines/engine.py.229: process: 0.85447 | load: 0.00295 [I 2022-10-11 13:45:57] ...trap/engines/engine.py.230: loss: 0.10142 [I 2022-10-11 13:45:57] ...optimizers/trijoint.py.151: total_norm: 15.782663 [I 2022-10-11 13:46:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 220/2304 [I 2022-10-11 13:46:05] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:31 [I 2022-10-11 13:46:05] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00296 [I 2022-10-11 13:46:05] ...trap/engines/engine.py.230: loss: 0.14390 [I 2022-10-11 13:46:05] ...optimizers/trijoint.py.151: total_norm: 12.170602 [I 2022-10-11 13:46:14] ...trap/engines/engine.py.224: train: epoch 107 | batch 230/2304 [I 2022-10-11 13:46:14] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:16 [I 2022-10-11 13:46:14] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00316 [I 2022-10-11 13:46:14] ...trap/engines/engine.py.230: loss: 0.14235 [I 2022-10-11 13:46:14] ...optimizers/trijoint.py.151: total_norm: 18.429590 [I 2022-10-11 13:46:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 240/2304 [I 2022-10-11 13:46:22] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:16 [I 2022-10-11 13:46:22] ...trap/engines/engine.py.229: process: 0.85196 | load: 0.00317 [I 2022-10-11 13:46:22] ...trap/engines/engine.py.230: loss: 0.09905 [I 2022-10-11 13:46:22] ...optimizers/trijoint.py.151: total_norm: 26.229509 [I 2022-10-11 13:46:31] ...trap/engines/engine.py.224: train: epoch 107 | batch 250/2304 [I 2022-10-11 13:46:31] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:29:02 [I 2022-10-11 13:46:31] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00314 [I 2022-10-11 13:46:31] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-11 13:46:31] ...optimizers/trijoint.py.151: total_norm: 28.337626 [I 2022-10-11 13:46:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 260/2304 [I 2022-10-11 13:46:39] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:52 [I 2022-10-11 13:46:39] ...trap/engines/engine.py.229: process: 0.84545 | load: 0.00296 [I 2022-10-11 13:46:39] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-11 13:46:39] ...optimizers/trijoint.py.151: total_norm: 12.921599 [I 2022-10-11 13:46:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 270/2304 [I 2022-10-11 13:46:48] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:28:39 [I 2022-10-11 13:46:48] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00299 [I 2022-10-11 13:46:48] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-11 13:46:48] ...optimizers/trijoint.py.151: total_norm: 15.640005 [I 2022-10-11 13:46:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 280/2304 [I 2022-10-11 13:46:56] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:28:32 [I 2022-10-11 13:46:56] ...trap/engines/engine.py.229: process: 0.83749 | load: 0.00300 [I 2022-10-11 13:46:56] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-11 13:46:56] ...optimizers/trijoint.py.151: total_norm: 16.759403 [I 2022-10-11 13:47:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 290/2304 [I 2022-10-11 13:47:05] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:28:24 [I 2022-10-11 13:47:05] ...trap/engines/engine.py.229: process: 0.85184 | load: 0.00298 [I 2022-10-11 13:47:05] ...trap/engines/engine.py.230: loss: 0.10016 [I 2022-10-11 13:47:05] ...optimizers/trijoint.py.151: total_norm: 16.183100 [I 2022-10-11 13:47:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 300/2304 [I 2022-10-11 13:47:13] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:01 [I 2022-10-11 13:47:13] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00305 [I 2022-10-11 13:47:13] ...trap/engines/engine.py.230: loss: 0.13340 [I 2022-10-11 13:47:13] ...optimizers/trijoint.py.151: total_norm: 28.805250 [I 2022-10-11 13:47:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 310/2304 [I 2022-10-11 13:47:22] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:28:10 [I 2022-10-11 13:47:22] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00320 [I 2022-10-11 13:47:22] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-11 13:47:22] ...optimizers/trijoint.py.151: total_norm: 24.724237 [I 2022-10-11 13:47:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 320/2304 [I 2022-10-11 13:47:30] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:02 [I 2022-10-11 13:47:30] ...trap/engines/engine.py.229: process: 0.85232 | load: 0.00322 [I 2022-10-11 13:47:30] ...trap/engines/engine.py.230: loss: 0.08121 [I 2022-10-11 13:47:30] ...optimizers/trijoint.py.151: total_norm: 12.758551 [I 2022-10-11 13:47:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 330/2304 [I 2022-10-11 13:47:39] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:02 [I 2022-10-11 13:47:39] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00312 [I 2022-10-11 13:47:39] ...trap/engines/engine.py.230: loss: 0.18443 [I 2022-10-11 13:47:39] ...optimizers/trijoint.py.151: total_norm: 13.844848 [I 2022-10-11 13:47:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 340/2304 [I 2022-10-11 13:47:47] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:27:53 [I 2022-10-11 13:47:47] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00325 [I 2022-10-11 13:47:47] ...trap/engines/engine.py.230: loss: 0.20616 [I 2022-10-11 13:47:47] ...optimizers/trijoint.py.151: total_norm: 34.682793 [I 2022-10-11 13:47:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 350/2304 [I 2022-10-11 13:47:56] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:58 [I 2022-10-11 13:47:56] ...trap/engines/engine.py.229: process: 0.86639 | load: 0.00314 [I 2022-10-11 13:47:56] ...trap/engines/engine.py.230: loss: 0.10715 [I 2022-10-11 13:47:56] ...optimizers/trijoint.py.151: total_norm: 17.523287 [I 2022-10-11 13:48:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 360/2304 [I 2022-10-11 13:48:04] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:27:35 [I 2022-10-11 13:48:04] ...trap/engines/engine.py.229: process: 0.85137 | load: 0.00314 [I 2022-10-11 13:48:04] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-11 13:48:04] ...optimizers/trijoint.py.151: total_norm: 12.879403 [I 2022-10-11 13:48:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 370/2304 [I 2022-10-11 13:48:13] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:41 [I 2022-10-11 13:48:13] ...trap/engines/engine.py.229: process: 0.84151 | load: 0.00338 [I 2022-10-11 13:48:13] ...trap/engines/engine.py.230: loss: 0.09486 [I 2022-10-11 13:48:13] ...optimizers/trijoint.py.151: total_norm: 38.608906 [I 2022-10-11 13:48:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 380/2304 [I 2022-10-11 13:48:21] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:17 [I 2022-10-11 13:48:21] ...trap/engines/engine.py.229: process: 0.84128 | load: 0.00314 [I 2022-10-11 13:48:21] ...trap/engines/engine.py.230: loss: 0.11391 [I 2022-10-11 13:48:22] ...optimizers/trijoint.py.151: total_norm: 31.055168 [I 2022-10-11 13:48:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 390/2304 [I 2022-10-11 13:48:30] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:13 [I 2022-10-11 13:48:30] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00315 [I 2022-10-11 13:48:30] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-11 13:48:30] ...optimizers/trijoint.py.151: total_norm: 15.729316 [I 2022-10-11 13:48:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 400/2304 [I 2022-10-11 13:48:39] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:07 [I 2022-10-11 13:48:39] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00312 [I 2022-10-11 13:48:39] ...trap/engines/engine.py.230: loss: 0.10491 [I 2022-10-11 13:48:39] ...optimizers/trijoint.py.151: total_norm: 16.445524 [I 2022-10-11 13:48:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 410/2304 [I 2022-10-11 13:48:47] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:27:01 [I 2022-10-11 13:48:47] ...trap/engines/engine.py.229: process: 0.83443 | load: 0.00314 [I 2022-10-11 13:48:47] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-11 13:48:47] ...optimizers/trijoint.py.151: total_norm: 39.231209 [I 2022-10-11 13:48:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 420/2304 [I 2022-10-11 13:48:56] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:40 [I 2022-10-11 13:48:56] ...trap/engines/engine.py.229: process: 0.85866 | load: 0.00314 [I 2022-10-11 13:48:56] ...trap/engines/engine.py.230: loss: 0.09008 [I 2022-10-11 13:48:56] ...optimizers/trijoint.py.151: total_norm: 10.664893 [I 2022-10-11 13:49:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 430/2304 [I 2022-10-11 13:49:04] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:26:38 [I 2022-10-11 13:49:04] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00315 [I 2022-10-11 13:49:04] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-11 13:49:04] ...optimizers/trijoint.py.151: total_norm: 11.217010 [I 2022-10-11 13:49:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 440/2304 [I 2022-10-11 13:49:13] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:37 [I 2022-10-11 13:49:13] ...trap/engines/engine.py.229: process: 0.85473 | load: 0.00312 [I 2022-10-11 13:49:13] ...trap/engines/engine.py.230: loss: 0.10066 [I 2022-10-11 13:49:13] ...optimizers/trijoint.py.151: total_norm: 19.374462 [I 2022-10-11 13:49:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 450/2304 [I 2022-10-11 13:49:21] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:21 [I 2022-10-11 13:49:21] ...trap/engines/engine.py.229: process: 0.83759 | load: 0.00313 [I 2022-10-11 13:49:21] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-11 13:49:21] ...optimizers/trijoint.py.151: total_norm: 12.701347 [I 2022-10-11 13:49:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 460/2304 [I 2022-10-11 13:49:30] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:10 [I 2022-10-11 13:49:30] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00314 [I 2022-10-11 13:49:30] ...trap/engines/engine.py.230: loss: 0.09761 [I 2022-10-11 13:49:30] ...optimizers/trijoint.py.151: total_norm: 11.278811 [I 2022-10-11 13:49:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 470/2304 [I 2022-10-11 13:49:38] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:25:57 [I 2022-10-11 13:49:38] ...trap/engines/engine.py.229: process: 0.84810 | load: 0.00312 [I 2022-10-11 13:49:38] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-11 13:49:38] ...optimizers/trijoint.py.151: total_norm: 59.930248 [I 2022-10-11 13:49:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 480/2304 [I 2022-10-11 13:49:47] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:26:03 [I 2022-10-11 13:49:47] ...trap/engines/engine.py.229: process: 0.86568 | load: 0.00317 [I 2022-10-11 13:49:47] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-11 13:49:47] ...optimizers/trijoint.py.151: total_norm: 19.739889 [I 2022-10-11 13:49:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 490/2304 [I 2022-10-11 13:49:56] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:26:01 [I 2022-10-11 13:49:56] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00313 [I 2022-10-11 13:49:56] ...trap/engines/engine.py.230: loss: 0.07624 [I 2022-10-11 13:49:56] ...optimizers/trijoint.py.151: total_norm: 10.002024 [I 2022-10-11 13:50:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 500/2304 [I 2022-10-11 13:50:04] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:45 [I 2022-10-11 13:50:04] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00313 [I 2022-10-11 13:50:04] ...trap/engines/engine.py.230: loss: 0.12620 [I 2022-10-11 13:50:04] ...optimizers/trijoint.py.151: total_norm: 22.456417 [I 2022-10-11 13:50:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 510/2304 [I 2022-10-11 13:50:13] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:42 [I 2022-10-11 13:50:13] ...trap/engines/engine.py.229: process: 0.86509 | load: 0.00310 [I 2022-10-11 13:50:13] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-11 13:50:13] ...optimizers/trijoint.py.151: total_norm: 11.959135 [I 2022-10-11 13:50:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 520/2304 [I 2022-10-11 13:50:22] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:25:27 [I 2022-10-11 13:50:22] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00313 [I 2022-10-11 13:50:22] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-11 13:50:22] ...optimizers/trijoint.py.151: total_norm: 16.377975 [I 2022-10-11 13:50:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 530/2304 [I 2022-10-11 13:50:30] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:25:13 [I 2022-10-11 13:50:30] ...trap/engines/engine.py.229: process: 0.86182 | load: 0.00313 [I 2022-10-11 13:50:30] ...trap/engines/engine.py.230: loss: 0.09270 [I 2022-10-11 13:50:30] ...optimizers/trijoint.py.151: total_norm: 17.875448 [I 2022-10-11 13:50:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 540/2304 [I 2022-10-11 13:50:39] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:25:16 [I 2022-10-11 13:50:39] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00317 [I 2022-10-11 13:50:39] ...trap/engines/engine.py.230: loss: 0.14214 [I 2022-10-11 13:50:39] ...optimizers/trijoint.py.151: total_norm: 15.059148 [I 2022-10-11 13:50:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 550/2304 [I 2022-10-11 13:50:47] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:24:57 [I 2022-10-11 13:50:47] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00313 [I 2022-10-11 13:50:47] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-11 13:50:47] ...optimizers/trijoint.py.151: total_norm: 41.666382 [I 2022-10-11 13:50:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 560/2304 [I 2022-10-11 13:50:56] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:24:42 [I 2022-10-11 13:50:56] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00324 [I 2022-10-11 13:50:56] ...trap/engines/engine.py.230: loss: 0.09692 [I 2022-10-11 13:50:56] ...optimizers/trijoint.py.151: total_norm: 21.711273 [I 2022-10-11 13:51:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 570/2304 [I 2022-10-11 13:51:04] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:24:33 [I 2022-10-11 13:51:04] ...trap/engines/engine.py.229: process: 0.85091 | load: 0.00314 [I 2022-10-11 13:51:04] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-11 13:51:04] ...optimizers/trijoint.py.151: total_norm: 8.139411 [I 2022-10-11 13:51:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 580/2304 [I 2022-10-11 13:51:13] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:24:21 [I 2022-10-11 13:51:13] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00315 [I 2022-10-11 13:51:13] ...trap/engines/engine.py.230: loss: 0.09870 [I 2022-10-11 13:51:13] ...optimizers/trijoint.py.151: total_norm: 13.974378 [I 2022-10-11 13:51:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 590/2304 [I 2022-10-11 13:51:21] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:24:26 [I 2022-10-11 13:51:21] ...trap/engines/engine.py.229: process: 0.85369 | load: 0.00313 [I 2022-10-11 13:51:21] ...trap/engines/engine.py.230: loss: 0.16257 [I 2022-10-11 13:51:21] ...optimizers/trijoint.py.151: total_norm: 20.864597 [I 2022-10-11 13:51:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 600/2304 [I 2022-10-11 13:51:30] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:23:58 [I 2022-10-11 13:51:30] ...trap/engines/engine.py.229: process: 0.84756 | load: 0.00296 [I 2022-10-11 13:51:30] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-11 13:51:30] ...optimizers/trijoint.py.151: total_norm: 15.067266 [I 2022-10-11 13:51:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 610/2304 [I 2022-10-11 13:51:38] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:23:44 [I 2022-10-11 13:51:38] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00302 [I 2022-10-11 13:51:38] ...trap/engines/engine.py.230: loss: 0.11738 [I 2022-10-11 13:51:38] ...optimizers/trijoint.py.151: total_norm: 12.242245 [I 2022-10-11 13:51:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 620/2304 [I 2022-10-11 13:51:47] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:23:35 [I 2022-10-11 13:51:47] ...trap/engines/engine.py.229: process: 0.83195 | load: 0.00296 [I 2022-10-11 13:51:47] ...trap/engines/engine.py.230: loss: 0.14033 [I 2022-10-11 13:51:47] ...optimizers/trijoint.py.151: total_norm: 14.492474 [I 2022-10-11 13:51:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 630/2304 [I 2022-10-11 13:51:55] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:28 [I 2022-10-11 13:51:55] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00298 [I 2022-10-11 13:51:55] ...trap/engines/engine.py.230: loss: 0.07486 [I 2022-10-11 13:51:55] ...optimizers/trijoint.py.151: total_norm: 11.992278 [I 2022-10-11 13:52:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 640/2304 [I 2022-10-11 13:52:04] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:23:26 [I 2022-10-11 13:52:04] ...trap/engines/engine.py.229: process: 0.85395 | load: 0.00298 [I 2022-10-11 13:52:04] ...trap/engines/engine.py.230: loss: 0.18878 [I 2022-10-11 13:52:04] ...optimizers/trijoint.py.151: total_norm: 24.090145 [I 2022-10-11 13:52:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 650/2304 [I 2022-10-11 13:52:12] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:09 [I 2022-10-11 13:52:12] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00295 [I 2022-10-11 13:52:12] ...trap/engines/engine.py.230: loss: 0.11310 [I 2022-10-11 13:52:12] ...optimizers/trijoint.py.151: total_norm: 11.042745 [I 2022-10-11 13:52:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 660/2304 [I 2022-10-11 13:52:20] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:23:05 [I 2022-10-11 13:52:20] ...trap/engines/engine.py.229: process: 0.85828 | load: 0.00295 [I 2022-10-11 13:52:20] ...trap/engines/engine.py.230: loss: 0.10083 [I 2022-10-11 13:52:20] ...optimizers/trijoint.py.151: total_norm: 12.899900 [I 2022-10-11 13:52:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 670/2304 [I 2022-10-11 13:52:29] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:22:57 [I 2022-10-11 13:52:29] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00297 [I 2022-10-11 13:52:29] ...trap/engines/engine.py.230: loss: 0.08514 [I 2022-10-11 13:52:29] ...optimizers/trijoint.py.151: total_norm: 15.417983 [I 2022-10-11 13:52:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 680/2304 [I 2022-10-11 13:52:37] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:22:53 [I 2022-10-11 13:52:37] ...trap/engines/engine.py.229: process: 0.86260 | load: 0.00294 [I 2022-10-11 13:52:37] ...trap/engines/engine.py.230: loss: 0.12266 [I 2022-10-11 13:52:37] ...optimizers/trijoint.py.151: total_norm: 13.177755 [I 2022-10-11 13:52:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 690/2304 [I 2022-10-11 13:52:46] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:22:40 [I 2022-10-11 13:52:46] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00294 [I 2022-10-11 13:52:46] ...trap/engines/engine.py.230: loss: 0.17915 [I 2022-10-11 13:52:46] ...optimizers/trijoint.py.151: total_norm: 17.687031 [I 2022-10-11 13:52:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 700/2304 [I 2022-10-11 13:52:54] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:22:25 [I 2022-10-11 13:52:54] ...trap/engines/engine.py.229: process: 0.83997 | load: 0.00296 [I 2022-10-11 13:52:54] ...trap/engines/engine.py.230: loss: 0.11049 [I 2022-10-11 13:52:54] ...optimizers/trijoint.py.151: total_norm: 17.720667 [I 2022-10-11 13:53:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 710/2304 [I 2022-10-11 13:53:03] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:16 [I 2022-10-11 13:53:03] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00298 [I 2022-10-11 13:53:03] ...trap/engines/engine.py.230: loss: 0.11272 [I 2022-10-11 13:53:03] ...optimizers/trijoint.py.151: total_norm: 8.377052 [I 2022-10-11 13:53:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 720/2304 [I 2022-10-11 13:53:11] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:22:11 [I 2022-10-11 13:53:11] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00300 [I 2022-10-11 13:53:11] ...trap/engines/engine.py.230: loss: 0.18143 [I 2022-10-11 13:53:11] ...optimizers/trijoint.py.151: total_norm: 21.969612 [I 2022-10-11 13:53:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 730/2304 [I 2022-10-11 13:53:19] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:22:04 [I 2022-10-11 13:53:19] ...trap/engines/engine.py.229: process: 0.84320 | load: 0.00299 [I 2022-10-11 13:53:19] ...trap/engines/engine.py.230: loss: 0.11357 [I 2022-10-11 13:53:19] ...optimizers/trijoint.py.151: total_norm: 38.384541 [I 2022-10-11 13:53:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 740/2304 [I 2022-10-11 13:53:28] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:21:52 [I 2022-10-11 13:53:28] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00302 [I 2022-10-11 13:53:28] ...trap/engines/engine.py.230: loss: 0.11254 [I 2022-10-11 13:53:28] ...optimizers/trijoint.py.151: total_norm: 19.618954 [I 2022-10-11 13:53:36] ...trap/engines/engine.py.224: train: epoch 107 | batch 750/2304 [I 2022-10-11 13:53:36] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:21:54 [I 2022-10-11 13:53:36] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00302 [I 2022-10-11 13:53:36] ...trap/engines/engine.py.230: loss: 0.16056 [I 2022-10-11 13:53:36] ...optimizers/trijoint.py.151: total_norm: 14.071046 [I 2022-10-11 13:53:45] ...trap/engines/engine.py.224: train: epoch 107 | batch 760/2304 [I 2022-10-11 13:53:45] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:21:40 [I 2022-10-11 13:53:45] ...trap/engines/engine.py.229: process: 0.83706 | load: 0.00300 [I 2022-10-11 13:53:45] ...trap/engines/engine.py.230: loss: 0.15071 [I 2022-10-11 13:53:45] ...optimizers/trijoint.py.151: total_norm: 24.000681 [I 2022-10-11 13:53:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 770/2304 [I 2022-10-11 13:53:53] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:21:36 [I 2022-10-11 13:53:53] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00298 [I 2022-10-11 13:53:53] ...trap/engines/engine.py.230: loss: 0.07164 [I 2022-10-11 13:53:53] ...optimizers/trijoint.py.151: total_norm: 13.757793 [I 2022-10-11 13:54:02] ...trap/engines/engine.py.224: train: epoch 107 | batch 780/2304 [I 2022-10-11 13:54:02] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:17 [I 2022-10-11 13:54:02] ...trap/engines/engine.py.229: process: 0.84152 | load: 0.00299 [I 2022-10-11 13:54:02] ...trap/engines/engine.py.230: loss: 0.10087 [I 2022-10-11 13:54:02] ...optimizers/trijoint.py.151: total_norm: 13.745780 [I 2022-10-11 13:54:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 790/2304 [I 2022-10-11 13:54:10] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:11 [I 2022-10-11 13:54:10] ...trap/engines/engine.py.229: process: 0.84173 | load: 0.00302 [I 2022-10-11 13:54:10] ...trap/engines/engine.py.230: loss: 0.07882 [I 2022-10-11 13:54:10] ...optimizers/trijoint.py.151: total_norm: 9.180756 [I 2022-10-11 13:54:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 800/2304 [I 2022-10-11 13:54:18] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:01 [I 2022-10-11 13:54:18] ...trap/engines/engine.py.229: process: 0.83728 | load: 0.00297 [I 2022-10-11 13:54:18] ...trap/engines/engine.py.230: loss: 0.16986 [I 2022-10-11 13:54:18] ...optimizers/trijoint.py.151: total_norm: 16.924742 [I 2022-10-11 13:54:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 810/2304 [I 2022-10-11 13:54:27] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:06 [I 2022-10-11 13:54:27] ...trap/engines/engine.py.229: process: 0.84181 | load: 0.00310 [I 2022-10-11 13:54:27] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-11 13:54:27] ...optimizers/trijoint.py.151: total_norm: 23.796825 [I 2022-10-11 13:54:36] ...trap/engines/engine.py.224: train: epoch 107 | batch 820/2304 [I 2022-10-11 13:54:36] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:04 [I 2022-10-11 13:54:36] ...trap/engines/engine.py.229: process: 0.84593 | load: 0.00315 [I 2022-10-11 13:54:36] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-11 13:54:36] ...optimizers/trijoint.py.151: total_norm: 9.202487 [I 2022-10-11 13:54:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 830/2304 [I 2022-10-11 13:54:44] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:21:08 [I 2022-10-11 13:54:44] ...trap/engines/engine.py.229: process: 0.85997 | load: 0.00311 [I 2022-10-11 13:54:44] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-11 13:54:44] ...optimizers/trijoint.py.151: total_norm: 11.944448 [I 2022-10-11 13:54:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 840/2304 [I 2022-10-11 13:54:53] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:53 [I 2022-10-11 13:54:53] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00316 [I 2022-10-11 13:54:53] ...trap/engines/engine.py.230: loss: 0.10781 [I 2022-10-11 13:54:53] ...optimizers/trijoint.py.151: total_norm: 30.756491 [I 2022-10-11 13:55:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 850/2304 [I 2022-10-11 13:55:01] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:40 [I 2022-10-11 13:55:01] ...trap/engines/engine.py.229: process: 0.86660 | load: 0.00315 [I 2022-10-11 13:55:01] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-11 13:55:01] ...optimizers/trijoint.py.151: total_norm: 37.158234 [I 2022-10-11 13:55:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 860/2304 [I 2022-10-11 13:55:10] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:38 [I 2022-10-11 13:55:10] ...trap/engines/engine.py.229: process: 0.87344 | load: 0.00314 [I 2022-10-11 13:55:10] ...trap/engines/engine.py.230: loss: 0.12682 [I 2022-10-11 13:55:10] ...optimizers/trijoint.py.151: total_norm: 16.789429 [I 2022-10-11 13:55:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 870/2304 [I 2022-10-11 13:55:18] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:24 [I 2022-10-11 13:55:18] ...trap/engines/engine.py.229: process: 0.85706 | load: 0.00314 [I 2022-10-11 13:55:18] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-11 13:55:18] ...optimizers/trijoint.py.151: total_norm: 27.019842 [I 2022-10-11 13:55:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 880/2304 [I 2022-10-11 13:55:27] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:16 [I 2022-10-11 13:55:27] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00314 [I 2022-10-11 13:55:27] ...trap/engines/engine.py.230: loss: 0.08366 [I 2022-10-11 13:55:27] ...optimizers/trijoint.py.151: total_norm: 9.845260 [I 2022-10-11 13:55:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 890/2304 [I 2022-10-11 13:55:35] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:19:55 [I 2022-10-11 13:55:35] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00312 [I 2022-10-11 13:55:35] ...trap/engines/engine.py.230: loss: 0.17986 [I 2022-10-11 13:55:35] ...optimizers/trijoint.py.151: total_norm: 18.645540 [I 2022-10-11 13:55:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 900/2304 [I 2022-10-11 13:55:44] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:19:58 [I 2022-10-11 13:55:44] ...trap/engines/engine.py.229: process: 0.84823 | load: 0.00310 [I 2022-10-11 13:55:44] ...trap/engines/engine.py.230: loss: 0.15727 [I 2022-10-11 13:55:44] ...optimizers/trijoint.py.151: total_norm: 15.541330 [I 2022-10-11 13:55:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 910/2304 [I 2022-10-11 13:55:52] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:49 [I 2022-10-11 13:55:52] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00312 [I 2022-10-11 13:55:52] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-11 13:55:52] ...optimizers/trijoint.py.151: total_norm: 10.161105 [I 2022-10-11 13:56:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 920/2304 [I 2022-10-11 13:56:01] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:38 [I 2022-10-11 13:56:01] ...trap/engines/engine.py.229: process: 0.86658 | load: 0.00312 [I 2022-10-11 13:56:01] ...trap/engines/engine.py.230: loss: 0.08675 [I 2022-10-11 13:56:01] ...optimizers/trijoint.py.151: total_norm: 14.589828 [I 2022-10-11 13:56:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 930/2304 [I 2022-10-11 13:56:10] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:27 [I 2022-10-11 13:56:10] ...trap/engines/engine.py.229: process: 0.83852 | load: 0.00312 [I 2022-10-11 13:56:10] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-11 13:56:10] ...optimizers/trijoint.py.151: total_norm: 12.906393 [I 2022-10-11 13:56:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 940/2304 [I 2022-10-11 13:56:18] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:27 [I 2022-10-11 13:56:18] ...trap/engines/engine.py.229: process: 0.86587 | load: 0.00312 [I 2022-10-11 13:56:18] ...trap/engines/engine.py.230: loss: 0.10965 [I 2022-10-11 13:56:18] ...optimizers/trijoint.py.151: total_norm: 10.964878 [I 2022-10-11 13:56:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 950/2304 [I 2022-10-11 13:56:27] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:16 [I 2022-10-11 13:56:27] ...trap/engines/engine.py.229: process: 0.84768 | load: 0.00312 [I 2022-10-11 13:56:27] ...trap/engines/engine.py.230: loss: 0.11296 [I 2022-10-11 13:56:27] ...optimizers/trijoint.py.151: total_norm: 11.244981 [I 2022-10-11 13:56:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 960/2304 [I 2022-10-11 13:56:35] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:09 [I 2022-10-11 13:56:35] ...trap/engines/engine.py.229: process: 0.87214 | load: 0.00314 [I 2022-10-11 13:56:35] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-11 13:56:35] ...optimizers/trijoint.py.151: total_norm: 12.177017 [I 2022-10-11 13:56:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 970/2304 [I 2022-10-11 13:56:44] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:18:54 [I 2022-10-11 13:56:44] ...trap/engines/engine.py.229: process: 0.85357 | load: 0.00311 [I 2022-10-11 13:56:44] ...trap/engines/engine.py.230: loss: 0.12983 [I 2022-10-11 13:56:44] ...optimizers/trijoint.py.151: total_norm: 18.488964 [I 2022-10-11 13:56:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 980/2304 [I 2022-10-11 13:56:52] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:18:45 [I 2022-10-11 13:56:52] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00320 [I 2022-10-11 13:56:52] ...trap/engines/engine.py.230: loss: 0.12361 [I 2022-10-11 13:56:52] ...optimizers/trijoint.py.151: total_norm: 12.721490 [I 2022-10-11 13:57:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 990/2304 [I 2022-10-11 13:57:01] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:41 [I 2022-10-11 13:57:01] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00312 [I 2022-10-11 13:57:01] ...trap/engines/engine.py.230: loss: 0.09697 [I 2022-10-11 13:57:01] ...optimizers/trijoint.py.151: total_norm: 11.653974 [I 2022-10-11 13:57:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1000/2304 [I 2022-10-11 13:57:09] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:40 [I 2022-10-11 13:57:09] ...trap/engines/engine.py.229: process: 0.86983 | load: 0.00313 [I 2022-10-11 13:57:09] ...trap/engines/engine.py.230: loss: 0.12314 [I 2022-10-11 13:57:09] ...optimizers/trijoint.py.151: total_norm: 17.745548 [I 2022-10-11 13:57:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1010/2304 [I 2022-10-11 13:57:18] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:32 [I 2022-10-11 13:57:18] ...trap/engines/engine.py.229: process: 0.85800 | load: 0.00315 [I 2022-10-11 13:57:18] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-11 13:57:18] ...optimizers/trijoint.py.151: total_norm: 18.373785 [I 2022-10-11 13:57:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 1020/2304 [I 2022-10-11 13:57:27] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:24 [I 2022-10-11 13:57:27] ...trap/engines/engine.py.229: process: 0.86612 | load: 0.00313 [I 2022-10-11 13:57:27] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-11 13:57:27] ...optimizers/trijoint.py.151: total_norm: 18.166323 [I 2022-10-11 13:57:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1030/2304 [I 2022-10-11 13:57:35] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:12 [I 2022-10-11 13:57:35] ...trap/engines/engine.py.229: process: 0.85326 | load: 0.00317 [I 2022-10-11 13:57:35] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-11 13:57:35] ...optimizers/trijoint.py.151: total_norm: 17.181160 [I 2022-10-11 13:57:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 1040/2304 [I 2022-10-11 13:57:44] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:18:02 [I 2022-10-11 13:57:44] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00313 [I 2022-10-11 13:57:44] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-11 13:57:44] ...optimizers/trijoint.py.151: total_norm: 37.121387 [I 2022-10-11 13:57:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 1050/2304 [I 2022-10-11 13:57:52] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:17:43 [I 2022-10-11 13:57:52] ...trap/engines/engine.py.229: process: 0.84994 | load: 0.00314 [I 2022-10-11 13:57:52] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-11 13:57:52] ...optimizers/trijoint.py.151: total_norm: 45.456768 [I 2022-10-11 13:58:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 1060/2304 [I 2022-10-11 13:58:01] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:35 [I 2022-10-11 13:58:01] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00313 [I 2022-10-11 13:58:01] ...trap/engines/engine.py.230: loss: 0.08251 [I 2022-10-11 13:58:01] ...optimizers/trijoint.py.151: total_norm: 7.907832 [I 2022-10-11 13:58:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1070/2304 [I 2022-10-11 13:58:09] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:33 [I 2022-10-11 13:58:09] ...trap/engines/engine.py.229: process: 0.85355 | load: 0.00312 [I 2022-10-11 13:58:09] ...trap/engines/engine.py.230: loss: 0.18645 [I 2022-10-11 13:58:09] ...optimizers/trijoint.py.151: total_norm: 40.308826 [I 2022-10-11 13:58:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1080/2304 [I 2022-10-11 13:58:18] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:17 [I 2022-10-11 13:58:18] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00308 [I 2022-10-11 13:58:18] ...trap/engines/engine.py.230: loss: 0.13820 [I 2022-10-11 13:58:18] ...optimizers/trijoint.py.151: total_norm: 27.344522 [I 2022-10-11 13:58:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 1090/2304 [I 2022-10-11 13:58:26] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:04 [I 2022-10-11 13:58:26] ...trap/engines/engine.py.229: process: 0.83703 | load: 0.00297 [I 2022-10-11 13:58:26] ...trap/engines/engine.py.230: loss: 0.08702 [I 2022-10-11 13:58:26] ...optimizers/trijoint.py.151: total_norm: 8.755301 [I 2022-10-11 13:58:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1100/2304 [I 2022-10-11 13:58:35] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:16:45 [I 2022-10-11 13:58:35] ...trap/engines/engine.py.229: process: 0.83209 | load: 0.00296 [I 2022-10-11 13:58:35] ...trap/engines/engine.py.230: loss: 0.10340 [I 2022-10-11 13:58:35] ...optimizers/trijoint.py.151: total_norm: 13.242465 [I 2022-10-11 13:58:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 1110/2304 [I 2022-10-11 13:58:43] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:16:37 [I 2022-10-11 13:58:43] ...trap/engines/engine.py.229: process: 0.82185 | load: 0.00297 [I 2022-10-11 13:58:43] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-11 13:58:43] ...optimizers/trijoint.py.151: total_norm: 13.711705 [I 2022-10-11 13:58:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 1120/2304 [I 2022-10-11 13:58:52] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:40 [I 2022-10-11 13:58:52] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00295 [I 2022-10-11 13:58:52] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-11 13:58:52] ...optimizers/trijoint.py.151: total_norm: 7.662285 [I 2022-10-11 13:59:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 1130/2304 [I 2022-10-11 13:59:00] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:16:26 [I 2022-10-11 13:59:00] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00293 [I 2022-10-11 13:59:00] ...trap/engines/engine.py.230: loss: 0.09338 [I 2022-10-11 13:59:00] ...optimizers/trijoint.py.151: total_norm: 12.203105 [I 2022-10-11 13:59:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 1140/2304 [I 2022-10-11 13:59:08] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:15 [I 2022-10-11 13:59:08] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00294 [I 2022-10-11 13:59:08] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-11 13:59:08] ...optimizers/trijoint.py.151: total_norm: 9.887481 [I 2022-10-11 13:59:17] ...trap/engines/engine.py.224: train: epoch 107 | batch 1150/2304 [I 2022-10-11 13:59:17] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:16:06 [I 2022-10-11 13:59:17] ...trap/engines/engine.py.229: process: 0.84185 | load: 0.00296 [I 2022-10-11 13:59:17] ...trap/engines/engine.py.230: loss: 0.19735 [I 2022-10-11 13:59:17] ...optimizers/trijoint.py.151: total_norm: 13.126262 [I 2022-10-11 13:59:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 1160/2304 [I 2022-10-11 13:59:25] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:15:55 [I 2022-10-11 13:59:25] ...trap/engines/engine.py.229: process: 0.83319 | load: 0.00296 [I 2022-10-11 13:59:25] ...trap/engines/engine.py.230: loss: 0.17184 [I 2022-10-11 13:59:25] ...optimizers/trijoint.py.151: total_norm: 14.491086 [I 2022-10-11 13:59:34] ...trap/engines/engine.py.224: train: epoch 107 | batch 1170/2304 [I 2022-10-11 13:59:34] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:15:54 [I 2022-10-11 13:59:34] ...trap/engines/engine.py.229: process: 0.83192 | load: 0.00295 [I 2022-10-11 13:59:34] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-11 13:59:34] ...optimizers/trijoint.py.151: total_norm: 38.754032 [I 2022-10-11 13:59:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1180/2304 [I 2022-10-11 13:59:42] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:15:50 [I 2022-10-11 13:59:42] ...trap/engines/engine.py.229: process: 0.84725 | load: 0.00300 [I 2022-10-11 13:59:42] ...trap/engines/engine.py.230: loss: 0.09492 [I 2022-10-11 13:59:42] ...optimizers/trijoint.py.151: total_norm: 8.287947 [I 2022-10-11 13:59:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 1190/2304 [I 2022-10-11 13:59:51] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:15:49 [I 2022-10-11 13:59:51] ...trap/engines/engine.py.229: process: 0.85686 | load: 0.00294 [I 2022-10-11 13:59:51] ...trap/engines/engine.py.230: loss: 0.08126 [I 2022-10-11 13:59:51] ...optimizers/trijoint.py.151: total_norm: 10.394883 [I 2022-10-11 13:59:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 1200/2304 [I 2022-10-11 13:59:59] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:40 [I 2022-10-11 13:59:59] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00298 [I 2022-10-11 13:59:59] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-11 13:59:59] ...optimizers/trijoint.py.151: total_norm: 28.583832 [I 2022-10-11 14:00:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 1210/2304 [I 2022-10-11 14:00:08] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:33 [I 2022-10-11 14:00:08] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00297 [I 2022-10-11 14:00:08] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-11 14:00:08] ...optimizers/trijoint.py.151: total_norm: 10.524745 [I 2022-10-11 14:00:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 1220/2304 [I 2022-10-11 14:00:16] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:22 [I 2022-10-11 14:00:16] ...trap/engines/engine.py.229: process: 0.84766 | load: 0.00300 [I 2022-10-11 14:00:16] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-11 14:00:16] ...optimizers/trijoint.py.151: total_norm: 13.640543 [I 2022-10-11 14:00:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 1230/2304 [I 2022-10-11 14:00:25] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:12 [I 2022-10-11 14:00:25] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00301 [I 2022-10-11 14:00:25] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-11 14:00:25] ...optimizers/trijoint.py.151: total_norm: 22.090582 [I 2022-10-11 14:00:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 1240/2304 [I 2022-10-11 14:00:33] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:06 [I 2022-10-11 14:00:33] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00299 [I 2022-10-11 14:00:33] ...trap/engines/engine.py.230: loss: 0.11596 [I 2022-10-11 14:00:33] ...optimizers/trijoint.py.151: total_norm: 24.094620 [I 2022-10-11 14:00:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1250/2304 [I 2022-10-11 14:00:42] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:50 [I 2022-10-11 14:00:42] ...trap/engines/engine.py.229: process: 0.83518 | load: 0.00301 [I 2022-10-11 14:00:42] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-11 14:00:42] ...optimizers/trijoint.py.151: total_norm: 21.283760 [I 2022-10-11 14:00:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1260/2304 [I 2022-10-11 14:00:50] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:41 [I 2022-10-11 14:00:50] ...trap/engines/engine.py.229: process: 0.83736 | load: 0.00298 [I 2022-10-11 14:00:50] ...trap/engines/engine.py.230: loss: 0.12112 [I 2022-10-11 14:00:50] ...optimizers/trijoint.py.151: total_norm: 10.651146 [I 2022-10-11 14:00:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 1270/2304 [I 2022-10-11 14:00:59] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:28 [I 2022-10-11 14:00:59] ...trap/engines/engine.py.229: process: 0.84237 | load: 0.00303 [I 2022-10-11 14:00:59] ...trap/engines/engine.py.230: loss: 0.11193 [I 2022-10-11 14:00:59] ...optimizers/trijoint.py.151: total_norm: 9.959514 [I 2022-10-11 14:01:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 1280/2304 [I 2022-10-11 14:01:07] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:14:16 [I 2022-10-11 14:01:07] ...trap/engines/engine.py.229: process: 0.83167 | load: 0.00296 [I 2022-10-11 14:01:07] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-11 14:01:07] ...optimizers/trijoint.py.151: total_norm: 18.381205 [I 2022-10-11 14:01:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 1290/2304 [I 2022-10-11 14:01:16] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:21 [I 2022-10-11 14:01:16] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00298 [I 2022-10-11 14:01:16] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-11 14:01:16] ...optimizers/trijoint.py.151: total_norm: 11.444529 [I 2022-10-11 14:01:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1300/2304 [I 2022-10-11 14:01:24] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:22 [I 2022-10-11 14:01:24] ...trap/engines/engine.py.229: process: 0.86745 | load: 0.00377 [I 2022-10-11 14:01:24] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-11 14:01:24] ...optimizers/trijoint.py.151: total_norm: 14.259400 [I 2022-10-11 14:01:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 1310/2304 [I 2022-10-11 14:01:33] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:23 [I 2022-10-11 14:01:33] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00380 [I 2022-10-11 14:01:33] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-11 14:01:33] ...optimizers/trijoint.py.151: total_norm: 33.773891 [I 2022-10-11 14:01:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1320/2304 [I 2022-10-11 14:01:42] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:16 [I 2022-10-11 14:01:42] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00380 [I 2022-10-11 14:01:42] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-11 14:01:42] ...optimizers/trijoint.py.151: total_norm: 10.825253 [I 2022-10-11 14:01:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1330/2304 [I 2022-10-11 14:01:50] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:11 [I 2022-10-11 14:01:50] ...trap/engines/engine.py.229: process: 0.88189 | load: 0.00380 [I 2022-10-11 14:01:50] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-11 14:01:50] ...optimizers/trijoint.py.151: total_norm: 19.144308 [I 2022-10-11 14:01:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 1340/2304 [I 2022-10-11 14:01:59] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:33 [I 2022-10-11 14:01:59] ...trap/engines/engine.py.229: process: 0.84119 | load: 0.00300 [I 2022-10-11 14:01:59] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-11 14:01:59] ...optimizers/trijoint.py.151: total_norm: 10.641555 [I 2022-10-11 14:02:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 1350/2304 [I 2022-10-11 14:02:07] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:19 [I 2022-10-11 14:02:07] ...trap/engines/engine.py.229: process: 0.83487 | load: 0.00298 [I 2022-10-11 14:02:07] ...trap/engines/engine.py.230: loss: 0.13925 [I 2022-10-11 14:02:07] ...optimizers/trijoint.py.151: total_norm: 22.561979 [I 2022-10-11 14:02:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 1360/2304 [I 2022-10-11 14:02:16] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:10 [I 2022-10-11 14:02:16] ...trap/engines/engine.py.229: process: 0.85215 | load: 0.00298 [I 2022-10-11 14:02:16] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-11 14:02:16] ...optimizers/trijoint.py.151: total_norm: 40.920708 [I 2022-10-11 14:02:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1370/2304 [I 2022-10-11 14:02:24] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:07 [I 2022-10-11 14:02:24] ...trap/engines/engine.py.229: process: 0.84863 | load: 0.00294 [I 2022-10-11 14:02:24] ...trap/engines/engine.py.230: loss: 0.09137 [I 2022-10-11 14:02:24] ...optimizers/trijoint.py.151: total_norm: 8.840755 [I 2022-10-11 14:02:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 1380/2304 [I 2022-10-11 14:02:33] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:12:54 [I 2022-10-11 14:02:33] ...trap/engines/engine.py.229: process: 0.81936 | load: 0.00295 [I 2022-10-11 14:02:33] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-11 14:02:33] ...optimizers/trijoint.py.151: total_norm: 25.860252 [I 2022-10-11 14:02:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 1390/2304 [I 2022-10-11 14:02:41] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:12:47 [I 2022-10-11 14:02:41] ...trap/engines/engine.py.229: process: 0.83302 | load: 0.00312 [I 2022-10-11 14:02:41] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-11 14:02:41] ...optimizers/trijoint.py.151: total_norm: 21.514471 [I 2022-10-11 14:02:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1400/2304 [I 2022-10-11 14:02:50] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:12:54 [I 2022-10-11 14:02:50] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00313 [I 2022-10-11 14:02:50] ...trap/engines/engine.py.230: loss: 0.16518 [I 2022-10-11 14:02:50] ...optimizers/trijoint.py.151: total_norm: 14.121440 [I 2022-10-11 14:02:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1410/2304 [I 2022-10-11 14:02:58] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:41 [I 2022-10-11 14:02:58] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00312 [I 2022-10-11 14:02:58] ...trap/engines/engine.py.230: loss: 0.10228 [I 2022-10-11 14:02:58] ...optimizers/trijoint.py.151: total_norm: 16.491892 [I 2022-10-11 14:03:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 1420/2304 [I 2022-10-11 14:03:07] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:38 [I 2022-10-11 14:03:07] ...trap/engines/engine.py.229: process: 0.85674 | load: 0.00313 [I 2022-10-11 14:03:07] ...trap/engines/engine.py.230: loss: 0.14273 [I 2022-10-11 14:03:07] ...optimizers/trijoint.py.151: total_norm: 27.715851 [I 2022-10-11 14:03:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 1430/2304 [I 2022-10-11 14:03:15] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:12:24 [I 2022-10-11 14:03:15] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00315 [I 2022-10-11 14:03:15] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-11 14:03:15] ...optimizers/trijoint.py.151: total_norm: 17.266685 [I 2022-10-11 14:03:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1440/2304 [I 2022-10-11 14:03:24] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:18 [I 2022-10-11 14:03:24] ...trap/engines/engine.py.229: process: 0.86077 | load: 0.00316 [I 2022-10-11 14:03:24] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-11 14:03:24] ...optimizers/trijoint.py.151: total_norm: 14.050712 [I 2022-10-11 14:03:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 1450/2304 [I 2022-10-11 14:03:32] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:12:15 [I 2022-10-11 14:03:32] ...trap/engines/engine.py.229: process: 0.88416 | load: 0.00313 [I 2022-10-11 14:03:32] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-11 14:03:32] ...optimizers/trijoint.py.151: total_norm: 16.984404 [I 2022-10-11 14:03:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 1460/2304 [I 2022-10-11 14:03:41] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:08 [I 2022-10-11 14:03:41] ...trap/engines/engine.py.229: process: 0.86524 | load: 0.00313 [I 2022-10-11 14:03:41] ...trap/engines/engine.py.230: loss: 0.14490 [I 2022-10-11 14:03:41] ...optimizers/trijoint.py.151: total_norm: 21.219782 [I 2022-10-11 14:03:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1470/2304 [I 2022-10-11 14:03:50] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:11:54 [I 2022-10-11 14:03:50] ...trap/engines/engine.py.229: process: 0.86130 | load: 0.00311 [I 2022-10-11 14:03:50] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-11 14:03:50] ...optimizers/trijoint.py.151: total_norm: 13.930327 [I 2022-10-11 14:03:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1480/2304 [I 2022-10-11 14:03:58] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:47 [I 2022-10-11 14:03:58] ...trap/engines/engine.py.229: process: 0.85572 | load: 0.00312 [I 2022-10-11 14:03:58] ...trap/engines/engine.py.230: loss: 0.11517 [I 2022-10-11 14:03:58] ...optimizers/trijoint.py.151: total_norm: 15.675797 [I 2022-10-11 14:04:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 1490/2304 [I 2022-10-11 14:04:07] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:39 [I 2022-10-11 14:04:07] ...trap/engines/engine.py.229: process: 0.86583 | load: 0.00312 [I 2022-10-11 14:04:07] ...trap/engines/engine.py.230: loss: 0.09503 [I 2022-10-11 14:04:07] ...optimizers/trijoint.py.151: total_norm: 9.260290 [I 2022-10-11 14:04:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 1500/2304 [I 2022-10-11 14:04:15] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:11:28 [I 2022-10-11 14:04:15] ...trap/engines/engine.py.229: process: 0.86936 | load: 0.00313 [I 2022-10-11 14:04:15] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-11 14:04:15] ...optimizers/trijoint.py.151: total_norm: 17.407209 [I 2022-10-11 14:04:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1510/2304 [I 2022-10-11 14:04:24] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:16 [I 2022-10-11 14:04:24] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00315 [I 2022-10-11 14:04:24] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-11 14:04:24] ...optimizers/trijoint.py.151: total_norm: 12.166650 [I 2022-10-11 14:04:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 1520/2304 [I 2022-10-11 14:04:32] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:11:10 [I 2022-10-11 14:04:32] ...trap/engines/engine.py.229: process: 0.86650 | load: 0.00313 [I 2022-10-11 14:04:32] ...trap/engines/engine.py.230: loss: 0.09015 [I 2022-10-11 14:04:32] ...optimizers/trijoint.py.151: total_norm: 12.164593 [I 2022-10-11 14:04:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 1530/2304 [I 2022-10-11 14:04:41] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:10:57 [I 2022-10-11 14:04:41] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00313 [I 2022-10-11 14:04:41] ...trap/engines/engine.py.230: loss: 0.17700 [I 2022-10-11 14:04:41] ...optimizers/trijoint.py.151: total_norm: 40.617332 [I 2022-10-11 14:04:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1540/2304 [I 2022-10-11 14:04:50] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:10:52 [I 2022-10-11 14:04:50] ...trap/engines/engine.py.229: process: 0.86227 | load: 0.00311 [I 2022-10-11 14:04:50] ...trap/engines/engine.py.230: loss: 0.10076 [I 2022-10-11 14:04:50] ...optimizers/trijoint.py.151: total_norm: 14.865247 [I 2022-10-11 14:04:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1550/2304 [I 2022-10-11 14:04:58] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:43 [I 2022-10-11 14:04:58] ...trap/engines/engine.py.229: process: 0.85286 | load: 0.00317 [I 2022-10-11 14:04:58] ...trap/engines/engine.py.230: loss: 0.09599 [I 2022-10-11 14:04:58] ...optimizers/trijoint.py.151: total_norm: 13.006701 [I 2022-10-11 14:05:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 1560/2304 [I 2022-10-11 14:05:07] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:36 [I 2022-10-11 14:05:07] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00312 [I 2022-10-11 14:05:07] ...trap/engines/engine.py.230: loss: 0.15411 [I 2022-10-11 14:05:07] ...optimizers/trijoint.py.151: total_norm: 19.182598 [I 2022-10-11 14:05:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 1570/2304 [I 2022-10-11 14:05:15] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:31 [I 2022-10-11 14:05:15] ...trap/engines/engine.py.229: process: 0.85583 | load: 0.00316 [I 2022-10-11 14:05:15] ...trap/engines/engine.py.230: loss: 0.13527 [I 2022-10-11 14:05:15] ...optimizers/trijoint.py.151: total_norm: 10.321499 [I 2022-10-11 14:05:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1580/2304 [I 2022-10-11 14:05:24] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:39 [I 2022-10-11 14:05:24] ...trap/engines/engine.py.229: process: 0.89175 | load: 0.00392 [I 2022-10-11 14:05:24] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-11 14:05:24] ...optimizers/trijoint.py.151: total_norm: 14.052677 [I 2022-10-11 14:05:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 1590/2304 [I 2022-10-11 14:05:33] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:32 [I 2022-10-11 14:05:33] ...trap/engines/engine.py.229: process: 0.88726 | load: 0.00396 [I 2022-10-11 14:05:33] ...trap/engines/engine.py.230: loss: 0.09380 [I 2022-10-11 14:05:33] ...optimizers/trijoint.py.151: total_norm: 37.443420 [I 2022-10-11 14:05:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1600/2304 [I 2022-10-11 14:05:42] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:26 [I 2022-10-11 14:05:42] ...trap/engines/engine.py.229: process: 0.88246 | load: 0.00401 [I 2022-10-11 14:05:42] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-11 14:05:42] ...optimizers/trijoint.py.151: total_norm: 9.137438 [I 2022-10-11 14:05:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 1610/2304 [I 2022-10-11 14:05:51] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:10:16 [I 2022-10-11 14:05:51] ...trap/engines/engine.py.229: process: 0.89010 | load: 0.00393 [I 2022-10-11 14:05:51] ...trap/engines/engine.py.230: loss: 0.08455 [I 2022-10-11 14:05:51] ...optimizers/trijoint.py.151: total_norm: 9.804450 [I 2022-10-11 14:06:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 1620/2304 [I 2022-10-11 14:06:00] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:10:03 [I 2022-10-11 14:06:00] ...trap/engines/engine.py.229: process: 0.88036 | load: 0.00396 [I 2022-10-11 14:06:00] ...trap/engines/engine.py.230: loss: 0.08375 [I 2022-10-11 14:06:00] ...optimizers/trijoint.py.151: total_norm: 10.119701 [I 2022-10-11 14:06:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1630/2304 [I 2022-10-11 14:06:09] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:55 [I 2022-10-11 14:06:09] ...trap/engines/engine.py.229: process: 0.89281 | load: 0.00395 [I 2022-10-11 14:06:09] ...trap/engines/engine.py.230: loss: 0.15354 [I 2022-10-11 14:06:09] ...optimizers/trijoint.py.151: total_norm: 11.680706 [I 2022-10-11 14:06:17] ...trap/engines/engine.py.224: train: epoch 107 | batch 1640/2304 [I 2022-10-11 14:06:17] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:49 [I 2022-10-11 14:06:17] ...trap/engines/engine.py.229: process: 0.87891 | load: 0.00399 [I 2022-10-11 14:06:17] ...trap/engines/engine.py.230: loss: 0.07101 [I 2022-10-11 14:06:17] ...optimizers/trijoint.py.151: total_norm: 16.957251 [I 2022-10-11 14:06:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 1650/2304 [I 2022-10-11 14:06:26] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:42 [I 2022-10-11 14:06:26] ...trap/engines/engine.py.229: process: 0.90577 | load: 0.00395 [I 2022-10-11 14:06:26] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-11 14:06:26] ...optimizers/trijoint.py.151: total_norm: 32.842598 [I 2022-10-11 14:06:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1660/2304 [I 2022-10-11 14:06:35] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:34 [I 2022-10-11 14:06:35] ...trap/engines/engine.py.229: process: 0.90205 | load: 0.00394 [I 2022-10-11 14:06:35] ...trap/engines/engine.py.230: loss: 0.20302 [I 2022-10-11 14:06:35] ...optimizers/trijoint.py.151: total_norm: 17.205496 [I 2022-10-11 14:06:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 1670/2304 [I 2022-10-11 14:06:44] ...trap/engines/engine.py.228: elapsed: 0:23:49 | left: 0:09:12 [I 2022-10-11 14:06:44] ...trap/engines/engine.py.229: process: 0.87354 | load: 0.00373 [I 2022-10-11 14:06:44] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-11 14:06:44] ...optimizers/trijoint.py.151: total_norm: 16.775143 [I 2022-10-11 14:06:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 1680/2304 [I 2022-10-11 14:06:53] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:08:48 [I 2022-10-11 14:06:53] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00317 [I 2022-10-11 14:06:53] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-11 14:06:53] ...optimizers/trijoint.py.151: total_norm: 34.868542 [I 2022-10-11 14:07:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 1690/2304 [I 2022-10-11 14:07:01] ...trap/engines/engine.py.228: elapsed: 0:24:06 | left: 0:08:42 [I 2022-10-11 14:07:01] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00295 [I 2022-10-11 14:07:01] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-11 14:07:01] ...optimizers/trijoint.py.151: total_norm: 14.301223 [I 2022-10-11 14:07:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 1700/2304 [I 2022-10-11 14:07:10] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:32 [I 2022-10-11 14:07:10] ...trap/engines/engine.py.229: process: 0.84511 | load: 0.00297 [I 2022-10-11 14:07:10] ...trap/engines/engine.py.230: loss: 0.09408 [I 2022-10-11 14:07:10] ...optimizers/trijoint.py.151: total_norm: 11.439791 [I 2022-10-11 14:07:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1710/2304 [I 2022-10-11 14:07:18] ...trap/engines/engine.py.228: elapsed: 0:24:23 | left: 0:08:23 [I 2022-10-11 14:07:18] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00295 [I 2022-10-11 14:07:18] ...trap/engines/engine.py.230: loss: 0.14965 [I 2022-10-11 14:07:18] ...optimizers/trijoint.py.151: total_norm: 10.095449 [I 2022-10-11 14:07:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 1720/2304 [I 2022-10-11 14:07:27] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:19 [I 2022-10-11 14:07:27] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00298 [I 2022-10-11 14:07:27] ...trap/engines/engine.py.230: loss: 0.10906 [I 2022-10-11 14:07:27] ...optimizers/trijoint.py.151: total_norm: 10.572308 [I 2022-10-11 14:07:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1730/2304 [I 2022-10-11 14:07:35] ...trap/engines/engine.py.228: elapsed: 0:24:40 | left: 0:08:08 [I 2022-10-11 14:07:35] ...trap/engines/engine.py.229: process: 0.86002 | load: 0.00297 [I 2022-10-11 14:07:35] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-11 14:07:35] ...optimizers/trijoint.py.151: total_norm: 41.261475 [I 2022-10-11 14:07:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 1740/2304 [I 2022-10-11 14:07:44] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:00 [I 2022-10-11 14:07:44] ...trap/engines/engine.py.229: process: 0.85394 | load: 0.00296 [I 2022-10-11 14:07:44] ...trap/engines/engine.py.230: loss: 0.12812 [I 2022-10-11 14:07:44] ...optimizers/trijoint.py.151: total_norm: 10.898839 [I 2022-10-11 14:07:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 1750/2304 [I 2022-10-11 14:07:52] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:07:49 [I 2022-10-11 14:07:52] ...trap/engines/engine.py.229: process: 0.84479 | load: 0.00295 [I 2022-10-11 14:07:52] ...trap/engines/engine.py.230: loss: 0.10868 [I 2022-10-11 14:07:52] ...optimizers/trijoint.py.151: total_norm: 8.769701 [I 2022-10-11 14:08:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 1760/2304 [I 2022-10-11 14:08:01] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:41 [I 2022-10-11 14:08:01] ...trap/engines/engine.py.229: process: 0.86166 | load: 0.00297 [I 2022-10-11 14:08:01] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-11 14:08:01] ...optimizers/trijoint.py.151: total_norm: 19.968287 [I 2022-10-11 14:08:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1770/2304 [I 2022-10-11 14:08:09] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:34 [I 2022-10-11 14:08:09] ...trap/engines/engine.py.229: process: 0.87749 | load: 0.00301 [I 2022-10-11 14:08:09] ...trap/engines/engine.py.230: loss: 0.09076 [I 2022-10-11 14:08:09] ...optimizers/trijoint.py.151: total_norm: 20.061771 [I 2022-10-11 14:08:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1780/2304 [I 2022-10-11 14:08:18] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:35 [I 2022-10-11 14:08:18] ...trap/engines/engine.py.229: process: 0.84850 | load: 0.00381 [I 2022-10-11 14:08:18] ...trap/engines/engine.py.230: loss: 0.15841 [I 2022-10-11 14:08:18] ...optimizers/trijoint.py.151: total_norm: 10.321909 [I 2022-10-11 14:08:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 1790/2304 [I 2022-10-11 14:08:27] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:26 [I 2022-10-11 14:08:27] ...trap/engines/engine.py.229: process: 0.88631 | load: 0.00300 [I 2022-10-11 14:08:27] ...trap/engines/engine.py.230: loss: 0.13620 [I 2022-10-11 14:08:27] ...optimizers/trijoint.py.151: total_norm: 32.891594 [I 2022-10-11 14:08:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1800/2304 [I 2022-10-11 14:08:35] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:21 [I 2022-10-11 14:08:35] ...trap/engines/engine.py.229: process: 0.88689 | load: 0.00389 [I 2022-10-11 14:08:35] ...trap/engines/engine.py.230: loss: 0.15280 [I 2022-10-11 14:08:35] ...optimizers/trijoint.py.151: total_norm: 16.774746 [I 2022-10-11 14:08:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 1810/2304 [I 2022-10-11 14:08:44] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:08 [I 2022-10-11 14:08:44] ...trap/engines/engine.py.229: process: 0.85317 | load: 0.00381 [I 2022-10-11 14:08:44] ...trap/engines/engine.py.230: loss: 0.15691 [I 2022-10-11 14:08:44] ...optimizers/trijoint.py.151: total_norm: 7.933036 [I 2022-10-11 14:08:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 1820/2304 [I 2022-10-11 14:08:53] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:00 [I 2022-10-11 14:08:53] ...trap/engines/engine.py.229: process: 0.86896 | load: 0.00381 [I 2022-10-11 14:08:53] ...trap/engines/engine.py.230: loss: 0.08359 [I 2022-10-11 14:08:53] ...optimizers/trijoint.py.151: total_norm: 9.373768 [I 2022-10-11 14:09:02] ...trap/engines/engine.py.224: train: epoch 107 | batch 1830/2304 [I 2022-10-11 14:09:02] ...trap/engines/engine.py.228: elapsed: 0:26:07 | left: 0:06:53 [I 2022-10-11 14:09:02] ...trap/engines/engine.py.229: process: 0.86479 | load: 0.00381 [I 2022-10-11 14:09:02] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-11 14:09:02] ...optimizers/trijoint.py.151: total_norm: 30.548849 [I 2022-10-11 14:09:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 1840/2304 [I 2022-10-11 14:09:10] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:06:45 [I 2022-10-11 14:09:10] ...trap/engines/engine.py.229: process: 0.87942 | load: 0.00378 [I 2022-10-11 14:09:10] ...trap/engines/engine.py.230: loss: 0.15833 [I 2022-10-11 14:09:10] ...optimizers/trijoint.py.151: total_norm: 13.922183 [I 2022-10-11 14:09:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 1850/2304 [I 2022-10-11 14:09:19] ...trap/engines/engine.py.228: elapsed: 0:26:24 | left: 0:06:36 [I 2022-10-11 14:09:19] ...trap/engines/engine.py.229: process: 0.86065 | load: 0.00381 [I 2022-10-11 14:09:19] ...trap/engines/engine.py.230: loss: 0.13984 [I 2022-10-11 14:09:19] ...optimizers/trijoint.py.151: total_norm: 14.112106 [I 2022-10-11 14:09:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 1860/2304 [I 2022-10-11 14:09:28] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:26 [I 2022-10-11 14:09:28] ...trap/engines/engine.py.229: process: 0.87976 | load: 0.00394 [I 2022-10-11 14:09:28] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-11 14:09:28] ...optimizers/trijoint.py.151: total_norm: 9.764999 [I 2022-10-11 14:09:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 1870/2304 [I 2022-10-11 14:09:37] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:24 [I 2022-10-11 14:09:37] ...trap/engines/engine.py.229: process: 0.88764 | load: 0.00397 [I 2022-10-11 14:09:37] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-11 14:09:37] ...optimizers/trijoint.py.151: total_norm: 10.289506 [I 2022-10-11 14:09:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 1880/2304 [I 2022-10-11 14:09:46] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:13 [I 2022-10-11 14:09:46] ...trap/engines/engine.py.229: process: 0.87021 | load: 0.00393 [I 2022-10-11 14:09:46] ...trap/engines/engine.py.230: loss: 0.12871 [I 2022-10-11 14:09:46] ...optimizers/trijoint.py.151: total_norm: 16.122742 [I 2022-10-11 14:09:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 1890/2304 [I 2022-10-11 14:09:54] ...trap/engines/engine.py.228: elapsed: 0:27:00 | left: 0:05:58 [I 2022-10-11 14:09:54] ...trap/engines/engine.py.229: process: 0.84325 | load: 0.00310 [I 2022-10-11 14:09:54] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-11 14:09:54] ...optimizers/trijoint.py.151: total_norm: 10.870549 [I 2022-10-11 14:10:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 1900/2304 [I 2022-10-11 14:10:03] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:05:42 [I 2022-10-11 14:10:03] ...trap/engines/engine.py.229: process: 0.83567 | load: 0.00309 [I 2022-10-11 14:10:03] ...trap/engines/engine.py.230: loss: 0.09085 [I 2022-10-11 14:10:03] ...optimizers/trijoint.py.151: total_norm: 13.275286 [I 2022-10-11 14:10:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 1910/2304 [I 2022-10-11 14:10:11] ...trap/engines/engine.py.228: elapsed: 0:27:17 | left: 0:05:36 [I 2022-10-11 14:10:11] ...trap/engines/engine.py.229: process: 0.86081 | load: 0.00311 [I 2022-10-11 14:10:11] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-11 14:10:11] ...optimizers/trijoint.py.151: total_norm: 19.107935 [I 2022-10-11 14:10:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 1920/2304 [I 2022-10-11 14:10:20] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:26 [I 2022-10-11 14:10:20] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00311 [I 2022-10-11 14:10:20] ...trap/engines/engine.py.230: loss: 0.07294 [I 2022-10-11 14:10:20] ...optimizers/trijoint.py.151: total_norm: 16.534933 [I 2022-10-11 14:10:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 1930/2304 [I 2022-10-11 14:10:28] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:19 [I 2022-10-11 14:10:28] ...trap/engines/engine.py.229: process: 0.84767 | load: 0.00316 [I 2022-10-11 14:10:28] ...trap/engines/engine.py.230: loss: 0.15404 [I 2022-10-11 14:10:28] ...optimizers/trijoint.py.151: total_norm: 21.491329 [I 2022-10-11 14:10:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 1940/2304 [I 2022-10-11 14:10:37] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:13 [I 2022-10-11 14:10:37] ...trap/engines/engine.py.229: process: 0.87543 | load: 0.00313 [I 2022-10-11 14:10:37] ...trap/engines/engine.py.230: loss: 0.10832 [I 2022-10-11 14:10:37] ...optimizers/trijoint.py.151: total_norm: 14.750432 [I 2022-10-11 14:10:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 1950/2304 [I 2022-10-11 14:10:46] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:01 [I 2022-10-11 14:10:46] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00312 [I 2022-10-11 14:10:46] ...trap/engines/engine.py.230: loss: 0.10358 [I 2022-10-11 14:10:46] ...optimizers/trijoint.py.151: total_norm: 10.374400 [I 2022-10-11 14:10:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 1960/2304 [I 2022-10-11 14:10:54] ...trap/engines/engine.py.228: elapsed: 0:27:59 | left: 0:04:52 [I 2022-10-11 14:10:54] ...trap/engines/engine.py.229: process: 0.84083 | load: 0.00317 [I 2022-10-11 14:10:54] ...trap/engines/engine.py.230: loss: 0.08273 [I 2022-10-11 14:10:54] ...optimizers/trijoint.py.151: total_norm: 11.157778 [I 2022-10-11 14:11:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 1970/2304 [I 2022-10-11 14:11:03] ...trap/engines/engine.py.228: elapsed: 0:28:08 | left: 0:04:44 [I 2022-10-11 14:11:03] ...trap/engines/engine.py.229: process: 0.84640 | load: 0.00314 [I 2022-10-11 14:11:03] ...trap/engines/engine.py.230: loss: 0.13590 [I 2022-10-11 14:11:03] ...optimizers/trijoint.py.151: total_norm: 13.351027 [I 2022-10-11 14:11:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 1980/2304 [I 2022-10-11 14:11:11] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:04:36 [I 2022-10-11 14:11:11] ...trap/engines/engine.py.229: process: 0.83881 | load: 0.00314 [I 2022-10-11 14:11:11] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-11 14:11:11] ...optimizers/trijoint.py.151: total_norm: 33.557446 [I 2022-10-11 14:11:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 1990/2304 [I 2022-10-11 14:11:20] ...trap/engines/engine.py.228: elapsed: 0:28:25 | left: 0:04:29 [I 2022-10-11 14:11:20] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00312 [I 2022-10-11 14:11:20] ...trap/engines/engine.py.230: loss: 0.15209 [I 2022-10-11 14:11:20] ...optimizers/trijoint.py.151: total_norm: 26.425440 [I 2022-10-11 14:11:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 2000/2304 [I 2022-10-11 14:11:28] ...trap/engines/engine.py.228: elapsed: 0:28:34 | left: 0:04:19 [I 2022-10-11 14:11:28] ...trap/engines/engine.py.229: process: 0.86148 | load: 0.00314 [I 2022-10-11 14:11:28] ...trap/engines/engine.py.230: loss: 0.10068 [I 2022-10-11 14:11:28] ...optimizers/trijoint.py.151: total_norm: 11.327847 [I 2022-10-11 14:11:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 2010/2304 [I 2022-10-11 14:11:37] ...trap/engines/engine.py.228: elapsed: 0:28:42 | left: 0:04:11 [I 2022-10-11 14:11:37] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00314 [I 2022-10-11 14:11:37] ...trap/engines/engine.py.230: loss: 0.11841 [I 2022-10-11 14:11:37] ...optimizers/trijoint.py.151: total_norm: 14.712668 [I 2022-10-11 14:11:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 2020/2304 [I 2022-10-11 14:11:46] ...trap/engines/engine.py.228: elapsed: 0:28:51 | left: 0:04:04 [I 2022-10-11 14:11:46] ...trap/engines/engine.py.229: process: 0.86125 | load: 0.00321 [I 2022-10-11 14:11:46] ...trap/engines/engine.py.230: loss: 0.11281 [I 2022-10-11 14:11:46] ...optimizers/trijoint.py.151: total_norm: 10.557524 [I 2022-10-11 14:11:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 2030/2304 [I 2022-10-11 14:11:54] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:03:55 [I 2022-10-11 14:11:54] ...trap/engines/engine.py.229: process: 0.85999 | load: 0.00311 [I 2022-10-11 14:11:54] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-11 14:11:54] ...optimizers/trijoint.py.151: total_norm: 24.175150 [I 2022-10-11 14:12:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 2040/2304 [I 2022-10-11 14:12:03] ...trap/engines/engine.py.228: elapsed: 0:29:08 | left: 0:03:48 [I 2022-10-11 14:12:03] ...trap/engines/engine.py.229: process: 0.88231 | load: 0.00394 [I 2022-10-11 14:12:03] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-11 14:12:03] ...optimizers/trijoint.py.151: total_norm: 25.866156 [I 2022-10-11 14:12:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 2050/2304 [I 2022-10-11 14:12:12] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:43 [I 2022-10-11 14:12:12] ...trap/engines/engine.py.229: process: 0.88349 | load: 0.00399 [I 2022-10-11 14:12:12] ...trap/engines/engine.py.230: loss: 0.10385 [I 2022-10-11 14:12:12] ...optimizers/trijoint.py.151: total_norm: 18.045061 [I 2022-10-11 14:12:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 2060/2304 [I 2022-10-11 14:12:21] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:36 [I 2022-10-11 14:12:21] ...trap/engines/engine.py.229: process: 0.89661 | load: 0.00395 [I 2022-10-11 14:12:21] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-11 14:12:21] ...optimizers/trijoint.py.151: total_norm: 8.764356 [I 2022-10-11 14:12:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 2070/2304 [I 2022-10-11 14:12:29] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:27 [I 2022-10-11 14:12:29] ...trap/engines/engine.py.229: process: 0.89111 | load: 0.00396 [I 2022-10-11 14:12:29] ...trap/engines/engine.py.230: loss: 0.09673 [I 2022-10-11 14:12:29] ...optimizers/trijoint.py.151: total_norm: 10.856091 [I 2022-10-11 14:12:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 2080/2304 [I 2022-10-11 14:12:38] ...trap/engines/engine.py.228: elapsed: 0:29:44 | left: 0:03:18 [I 2022-10-11 14:12:38] ...trap/engines/engine.py.229: process: 0.88562 | load: 0.00396 [I 2022-10-11 14:12:38] ...trap/engines/engine.py.230: loss: 0.09087 [I 2022-10-11 14:12:38] ...optimizers/trijoint.py.151: total_norm: 14.904814 [I 2022-10-11 14:12:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 2090/2304 [I 2022-10-11 14:12:47] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:03:09 [I 2022-10-11 14:12:47] ...trap/engines/engine.py.229: process: 0.87764 | load: 0.00397 [I 2022-10-11 14:12:47] ...trap/engines/engine.py.230: loss: 0.10444 [I 2022-10-11 14:12:47] ...optimizers/trijoint.py.151: total_norm: 29.825184 [I 2022-10-11 14:12:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 2100/2304 [I 2022-10-11 14:12:56] ...trap/engines/engine.py.228: elapsed: 0:30:01 | left: 0:02:58 [I 2022-10-11 14:12:56] ...trap/engines/engine.py.229: process: 0.85953 | load: 0.00316 [I 2022-10-11 14:12:56] ...trap/engines/engine.py.230: loss: 0.17263 [I 2022-10-11 14:12:56] ...optimizers/trijoint.py.151: total_norm: 32.437401 [I 2022-10-11 14:13:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 2110/2304 [I 2022-10-11 14:13:05] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:02:47 [I 2022-10-11 14:13:05] ...trap/engines/engine.py.229: process: 0.87132 | load: 0.00316 [I 2022-10-11 14:13:05] ...trap/engines/engine.py.230: loss: 0.10049 [I 2022-10-11 14:13:05] ...optimizers/trijoint.py.151: total_norm: 21.923805 [I 2022-10-11 14:13:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 2120/2304 [I 2022-10-11 14:13:13] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:38 [I 2022-10-11 14:13:13] ...trap/engines/engine.py.229: process: 0.86244 | load: 0.00318 [I 2022-10-11 14:13:13] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-11 14:13:13] ...optimizers/trijoint.py.151: total_norm: 10.539366 [I 2022-10-11 14:13:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 2130/2304 [I 2022-10-11 14:13:22] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:28 [I 2022-10-11 14:13:22] ...trap/engines/engine.py.229: process: 0.86174 | load: 0.00312 [I 2022-10-11 14:13:22] ...trap/engines/engine.py.230: loss: 0.07987 [I 2022-10-11 14:13:22] ...optimizers/trijoint.py.151: total_norm: 9.022448 [I 2022-10-11 14:13:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 2140/2304 [I 2022-10-11 14:13:30] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:18 [I 2022-10-11 14:13:30] ...trap/engines/engine.py.229: process: 0.82939 | load: 0.00294 [I 2022-10-11 14:13:30] ...trap/engines/engine.py.230: loss: 0.13216 [I 2022-10-11 14:13:30] ...optimizers/trijoint.py.151: total_norm: 18.013758 [I 2022-10-11 14:13:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 2150/2304 [I 2022-10-11 14:13:39] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:09 [I 2022-10-11 14:13:39] ...trap/engines/engine.py.229: process: 0.83517 | load: 0.00304 [I 2022-10-11 14:13:39] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-11 14:13:39] ...optimizers/trijoint.py.151: total_norm: 17.773428 [I 2022-10-11 14:13:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 2160/2304 [I 2022-10-11 14:13:47] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:02 [I 2022-10-11 14:13:47] ...trap/engines/engine.py.229: process: 0.85474 | load: 0.00295 [I 2022-10-11 14:13:47] ...trap/engines/engine.py.230: loss: 0.08896 [I 2022-10-11 14:13:47] ...optimizers/trijoint.py.151: total_norm: 11.809509 [I 2022-10-11 14:13:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 2170/2304 [I 2022-10-11 14:13:56] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:01:52 [I 2022-10-11 14:13:56] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00294 [I 2022-10-11 14:13:56] ...trap/engines/engine.py.230: loss: 0.08367 [I 2022-10-11 14:13:56] ...optimizers/trijoint.py.151: total_norm: 18.187445 [I 2022-10-11 14:14:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 2180/2304 [I 2022-10-11 14:14:04] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:01:44 [I 2022-10-11 14:14:04] ...trap/engines/engine.py.229: process: 0.85262 | load: 0.00292 [I 2022-10-11 14:14:04] ...trap/engines/engine.py.230: loss: 0.14252 [I 2022-10-11 14:14:04] ...optimizers/trijoint.py.151: total_norm: 33.257065 [I 2022-10-11 14:14:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 2190/2304 [I 2022-10-11 14:14:12] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:01:35 [I 2022-10-11 14:14:12] ...trap/engines/engine.py.229: process: 0.83718 | load: 0.00295 [I 2022-10-11 14:14:12] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-11 14:14:12] ...optimizers/trijoint.py.151: total_norm: 20.762175 [I 2022-10-11 14:14:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 2200/2304 [I 2022-10-11 14:14:21] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:28 [I 2022-10-11 14:14:21] ...trap/engines/engine.py.229: process: 0.86008 | load: 0.00295 [I 2022-10-11 14:14:21] ...trap/engines/engine.py.230: loss: 0.11587 [I 2022-10-11 14:14:21] ...optimizers/trijoint.py.151: total_norm: 15.406460 [I 2022-10-11 14:14:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 2210/2304 [I 2022-10-11 14:14:29] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:20 [I 2022-10-11 14:14:29] ...trap/engines/engine.py.229: process: 0.85126 | load: 0.00292 [I 2022-10-11 14:14:29] ...trap/engines/engine.py.230: loss: 0.09340 [I 2022-10-11 14:14:29] ...optimizers/trijoint.py.151: total_norm: 8.789164 [I 2022-10-11 14:14:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 2220/2304 [I 2022-10-11 14:14:38] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:11 [I 2022-10-11 14:14:38] ...trap/engines/engine.py.229: process: 0.85240 | load: 0.00294 [I 2022-10-11 14:14:38] ...trap/engines/engine.py.230: loss: 0.12777 [I 2022-10-11 14:14:38] ...optimizers/trijoint.py.151: total_norm: 15.357065 [I 2022-10-11 14:14:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 2230/2304 [I 2022-10-11 14:14:46] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:02 [I 2022-10-11 14:14:46] ...trap/engines/engine.py.229: process: 0.83253 | load: 0.00294 [I 2022-10-11 14:14:46] ...trap/engines/engine.py.230: loss: 0.11710 [I 2022-10-11 14:14:46] ...optimizers/trijoint.py.151: total_norm: 25.286900 [I 2022-10-11 14:14:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 2240/2304 [I 2022-10-11 14:14:55] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:00:54 [I 2022-10-11 14:14:55] ...trap/engines/engine.py.229: process: 0.82497 | load: 0.00293 [I 2022-10-11 14:14:55] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-11 14:14:55] ...optimizers/trijoint.py.151: total_norm: 14.806932 [I 2022-10-11 14:15:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 2250/2304 [I 2022-10-11 14:15:03] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:00:45 [I 2022-10-11 14:15:03] ...trap/engines/engine.py.229: process: 0.84032 | load: 0.00299 [I 2022-10-11 14:15:03] ...trap/engines/engine.py.230: loss: 0.18107 [I 2022-10-11 14:15:03] ...optimizers/trijoint.py.151: total_norm: 31.168041 [I 2022-10-11 14:15:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 2260/2304 [I 2022-10-11 14:15:12] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:37 [I 2022-10-11 14:15:12] ...trap/engines/engine.py.229: process: 0.85430 | load: 0.00293 [I 2022-10-11 14:15:12] ...trap/engines/engine.py.230: loss: 0.13048 [I 2022-10-11 14:15:12] ...optimizers/trijoint.py.151: total_norm: 12.223845 [I 2022-10-11 14:15:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 2270/2304 [I 2022-10-11 14:15:20] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:28 [I 2022-10-11 14:15:20] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00297 [I 2022-10-11 14:15:20] ...trap/engines/engine.py.230: loss: 0.10616 [I 2022-10-11 14:15:20] ...optimizers/trijoint.py.151: total_norm: 8.718492 [I 2022-10-11 14:15:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 2280/2304 [I 2022-10-11 14:15:29] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:20 [I 2022-10-11 14:15:29] ...trap/engines/engine.py.229: process: 0.83086 | load: 0.00292 [I 2022-10-11 14:15:29] ...trap/engines/engine.py.230: loss: 0.13536 [I 2022-10-11 14:15:29] ...optimizers/trijoint.py.151: total_norm: 17.663979 [I 2022-10-11 14:15:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 2290/2304 [I 2022-10-11 14:15:37] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:11 [I 2022-10-11 14:15:37] ...trap/engines/engine.py.229: process: 0.84516 | load: 0.00294 [I 2022-10-11 14:15:37] ...trap/engines/engine.py.230: loss: 0.12185 [I 2022-10-11 14:15:37] ...optimizers/trijoint.py.151: total_norm: 23.296865 [I 2022-10-11 14:15:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 2300/2304 [I 2022-10-11 14:15:46] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:03 [I 2022-10-11 14:15:46] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00016 [I 2022-10-11 14:15:46] ...trap/engines/engine.py.230: loss: 0.14660 [I 2022-10-11 14:15:46] ...optimizers/trijoint.py.151: total_norm: 36.725471 [I 2022-10-11 14:15:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 2304/2304 [I 2022-10-11 14:15:49] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:00 [I 2022-10-11 14:15:49] ...trap/engines/engine.py.229: process: 0.82752 | load: 0.00015 [I 2022-10-11 14:15:49] ...trap/engines/engine.py.230: loss: 0.11736 [I 2022-10-11 14:15:49] ...optimizers/trijoint.py.151: total_norm: 19.740215 [S 2022-10-11 14:15:51] ...trap/engines/engine.py.129: train_epoch.epoch: 107 [S 2022-10-11 14:15:51] ...trap/engines/engine.py.129: train_epoch.loss: 0.12195730279977565 [I 2022-10-11 14:15:51] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 14:15:52] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 107 [I 2022-10-11 14:15:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 14:15:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 14:15:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 14:15:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 14:15:58] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 14:15:58] ...trap/engines/engine.py.321: eval: epoch 107 | batch 0/510 [I 2022-10-11 14:15:58] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:13 [I 2022-10-11 14:15:58] ...trap/engines/engine.py.326: process: 6.14501 | load: 5.60622 [I 2022-10-11 14:16:04] ...trap/engines/engine.py.321: eval: epoch 107 | batch 10/510 [I 2022-10-11 14:16:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:47 [I 2022-10-11 14:16:04] ...trap/engines/engine.py.326: process: 0.65694 | load: 0.00018 [I 2022-10-11 14:16:09] ...trap/engines/engine.py.321: eval: epoch 107 | batch 20/510 [I 2022-10-11 14:16:09] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:09 [I 2022-10-11 14:16:09] ...trap/engines/engine.py.326: process: 0.69937 | load: 0.00018 [I 2022-10-11 14:16:15] ...trap/engines/engine.py.321: eval: epoch 107 | batch 30/510 [I 2022-10-11 14:16:15] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:22 [I 2022-10-11 14:16:15] ...trap/engines/engine.py.326: process: 0.49551 | load: 0.00018 [I 2022-10-11 14:16:20] ...trap/engines/engine.py.321: eval: epoch 107 | batch 40/510 [I 2022-10-11 14:16:20] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:00 [I 2022-10-11 14:16:20] ...trap/engines/engine.py.326: process: 0.45179 | load: 0.00018 [I 2022-10-11 14:16:24] ...trap/engines/engine.py.321: eval: epoch 107 | batch 50/510 [I 2022-10-11 14:16:24] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:27 [I 2022-10-11 14:16:24] ...trap/engines/engine.py.326: process: 0.41451 | load: 0.00019 [I 2022-10-11 14:16:29] ...trap/engines/engine.py.321: eval: epoch 107 | batch 60/510 [I 2022-10-11 14:16:29] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:26 [I 2022-10-11 14:16:29] ...trap/engines/engine.py.326: process: 0.47476 | load: 0.00018 [I 2022-10-11 14:16:33] ...trap/engines/engine.py.321: eval: epoch 107 | batch 70/510 [I 2022-10-11 14:16:33] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:03:09 [I 2022-10-11 14:16:33] ...trap/engines/engine.py.326: process: 0.43129 | load: 0.00018 [I 2022-10-11 14:16:38] ...trap/engines/engine.py.321: eval: epoch 107 | batch 80/510 [I 2022-10-11 14:16:38] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:03:26 [I 2022-10-11 14:16:38] ...trap/engines/engine.py.326: process: 0.47636 | load: 0.00019 [I 2022-10-11 14:16:43] ...trap/engines/engine.py.321: eval: epoch 107 | batch 90/510 [I 2022-10-11 14:16:43] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:18 [I 2022-10-11 14:16:43] ...trap/engines/engine.py.326: process: 0.47215 | load: 0.00020 [I 2022-10-11 14:16:47] ...trap/engines/engine.py.321: eval: epoch 107 | batch 100/510 [I 2022-10-11 14:16:47] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:03:14 [I 2022-10-11 14:16:47] ...trap/engines/engine.py.326: process: 0.30799 | load: 0.00019 [I 2022-10-11 14:16:51] ...trap/engines/engine.py.321: eval: epoch 107 | batch 110/510 [I 2022-10-11 14:16:51] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-11 14:16:51] ...trap/engines/engine.py.326: process: 0.31329 | load: 0.00018 [I 2022-10-11 14:16:54] ...trap/engines/engine.py.321: eval: epoch 107 | batch 120/510 [I 2022-10-11 14:16:54] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:03 [I 2022-10-11 14:16:54] ...trap/engines/engine.py.326: process: 0.31404 | load: 0.00019 [I 2022-10-11 14:16:57] ...trap/engines/engine.py.321: eval: epoch 107 | batch 130/510 [I 2022-10-11 14:16:57] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:59 [I 2022-10-11 14:16:57] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00019 [I 2022-10-11 14:17:00] ...trap/engines/engine.py.321: eval: epoch 107 | batch 140/510 [I 2022-10-11 14:17:00] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:57 [I 2022-10-11 14:17:00] ...trap/engines/engine.py.326: process: 0.31752 | load: 0.00020 [I 2022-10-11 14:17:03] ...trap/engines/engine.py.321: eval: epoch 107 | batch 150/510 [I 2022-10-11 14:17:03] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:53 [I 2022-10-11 14:17:03] ...trap/engines/engine.py.326: process: 0.31477 | load: 0.00018 [I 2022-10-11 14:17:06] ...trap/engines/engine.py.321: eval: epoch 107 | batch 160/510 [I 2022-10-11 14:17:06] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:49 [I 2022-10-11 14:17:06] ...trap/engines/engine.py.326: process: 0.31250 | load: 0.00020 [I 2022-10-11 14:17:09] ...trap/engines/engine.py.321: eval: epoch 107 | batch 170/510 [I 2022-10-11 14:17:09] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:47 [I 2022-10-11 14:17:09] ...trap/engines/engine.py.326: process: 0.31801 | load: 0.00019 [I 2022-10-11 14:17:13] ...trap/engines/engine.py.321: eval: epoch 107 | batch 180/510 [I 2022-10-11 14:17:13] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:44 [I 2022-10-11 14:17:13] ...trap/engines/engine.py.326: process: 0.32289 | load: 0.00022 [I 2022-10-11 14:17:16] ...trap/engines/engine.py.321: eval: epoch 107 | batch 190/510 [I 2022-10-11 14:17:16] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:42 [I 2022-10-11 14:17:16] ...trap/engines/engine.py.326: process: 0.31704 | load: 0.00020 [I 2022-10-11 14:17:19] ...trap/engines/engine.py.321: eval: epoch 107 | batch 200/510 [I 2022-10-11 14:17:19] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:39 [I 2022-10-11 14:17:19] ...trap/engines/engine.py.326: process: 0.31662 | load: 0.00021 [I 2022-10-11 14:17:22] ...trap/engines/engine.py.321: eval: epoch 107 | batch 210/510 [I 2022-10-11 14:17:22] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-11 14:17:22] ...trap/engines/engine.py.326: process: 0.31852 | load: 0.00020 [I 2022-10-11 14:17:25] ...trap/engines/engine.py.321: eval: epoch 107 | batch 220/510 [I 2022-10-11 14:17:25] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-11 14:17:25] ...trap/engines/engine.py.326: process: 0.31853 | load: 0.00020 [I 2022-10-11 14:17:29] ...trap/engines/engine.py.321: eval: epoch 107 | batch 230/510 [I 2022-10-11 14:17:29] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-11 14:17:29] ...trap/engines/engine.py.326: process: 0.32180 | load: 0.00020 [I 2022-10-11 14:17:32] ...trap/engines/engine.py.321: eval: epoch 107 | batch 240/510 [I 2022-10-11 14:17:32] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:26 [I 2022-10-11 14:17:32] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00021 [I 2022-10-11 14:17:35] ...trap/engines/engine.py.321: eval: epoch 107 | batch 250/510 [I 2022-10-11 14:17:35] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:23 [I 2022-10-11 14:17:35] ...trap/engines/engine.py.326: process: 0.31811 | load: 0.00021 [I 2022-10-11 14:17:38] ...trap/engines/engine.py.321: eval: epoch 107 | batch 260/510 [I 2022-10-11 14:17:38] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:20 [I 2022-10-11 14:17:38] ...trap/engines/engine.py.326: process: 0.31651 | load: 0.00021 [I 2022-10-11 14:17:41] ...trap/engines/engine.py.321: eval: epoch 107 | batch 270/510 [I 2022-10-11 14:17:41] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-11 14:17:41] ...trap/engines/engine.py.326: process: 0.32124 | load: 0.00021 [I 2022-10-11 14:17:45] ...trap/engines/engine.py.321: eval: epoch 107 | batch 280/510 [I 2022-10-11 14:17:45] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-11 14:17:45] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00022 [I 2022-10-11 14:17:48] ...trap/engines/engine.py.321: eval: epoch 107 | batch 290/510 [I 2022-10-11 14:17:48] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:10 [I 2022-10-11 14:17:48] ...trap/engines/engine.py.326: process: 0.31789 | load: 0.00021 [I 2022-10-11 14:17:51] ...trap/engines/engine.py.321: eval: epoch 107 | batch 300/510 [I 2022-10-11 14:17:51] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:07 [I 2022-10-11 14:17:51] ...trap/engines/engine.py.326: process: 0.32478 | load: 0.00021 [I 2022-10-11 14:17:54] ...trap/engines/engine.py.321: eval: epoch 107 | batch 310/510 [I 2022-10-11 14:17:54] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-11 14:17:54] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00021 [I 2022-10-11 14:17:58] ...trap/engines/engine.py.321: eval: epoch 107 | batch 320/510 [I 2022-10-11 14:17:58] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:01 [I 2022-10-11 14:17:58] ...trap/engines/engine.py.326: process: 0.32315 | load: 0.00020 [I 2022-10-11 14:18:01] ...trap/engines/engine.py.321: eval: epoch 107 | batch 330/510 [I 2022-10-11 14:18:01] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:58 [I 2022-10-11 14:18:01] ...trap/engines/engine.py.326: process: 0.32173 | load: 0.00021 [I 2022-10-11 14:18:04] ...trap/engines/engine.py.321: eval: epoch 107 | batch 340/510 [I 2022-10-11 14:18:04] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:54 [I 2022-10-11 14:18:04] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00021 [I 2022-10-11 14:18:07] ...trap/engines/engine.py.321: eval: epoch 107 | batch 350/510 [I 2022-10-11 14:18:07] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-11 14:18:07] ...trap/engines/engine.py.326: process: 0.32292 | load: 0.00021 [I 2022-10-11 14:18:10] ...trap/engines/engine.py.321: eval: epoch 107 | batch 360/510 [I 2022-10-11 14:18:10] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-11 14:18:10] ...trap/engines/engine.py.326: process: 0.32266 | load: 0.00022 [I 2022-10-11 14:18:14] ...trap/engines/engine.py.321: eval: epoch 107 | batch 370/510 [I 2022-10-11 14:18:14] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:44 [I 2022-10-11 14:18:14] ...trap/engines/engine.py.326: process: 0.31711 | load: 0.00022 [I 2022-10-11 14:18:17] ...trap/engines/engine.py.321: eval: epoch 107 | batch 380/510 [I 2022-10-11 14:18:17] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:41 [I 2022-10-11 14:18:17] ...trap/engines/engine.py.326: process: 0.32440 | load: 0.00025 [I 2022-10-11 14:18:20] ...trap/engines/engine.py.321: eval: epoch 107 | batch 390/510 [I 2022-10-11 14:18:20] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:38 [I 2022-10-11 14:18:20] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00022 [I 2022-10-11 14:18:23] ...trap/engines/engine.py.321: eval: epoch 107 | batch 400/510 [I 2022-10-11 14:18:23] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-11 14:18:23] ...trap/engines/engine.py.326: process: 0.32171 | load: 0.00021 [I 2022-10-11 14:18:27] ...trap/engines/engine.py.321: eval: epoch 107 | batch 410/510 [I 2022-10-11 14:18:27] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-11 14:18:27] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00021 [I 2022-10-11 14:18:30] ...trap/engines/engine.py.321: eval: epoch 107 | batch 420/510 [I 2022-10-11 14:18:30] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:29 [I 2022-10-11 14:18:30] ...trap/engines/engine.py.326: process: 0.32931 | load: 0.00021 [I 2022-10-11 14:18:33] ...trap/engines/engine.py.321: eval: epoch 107 | batch 430/510 [I 2022-10-11 14:18:33] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:25 [I 2022-10-11 14:18:33] ...trap/engines/engine.py.326: process: 0.32418 | load: 0.00022 [I 2022-10-11 14:18:36] ...trap/engines/engine.py.321: eval: epoch 107 | batch 440/510 [I 2022-10-11 14:18:36] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:22 [I 2022-10-11 14:18:36] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00021 [I 2022-10-11 14:18:39] ...trap/engines/engine.py.321: eval: epoch 107 | batch 450/510 [I 2022-10-11 14:18:39] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-11 14:18:39] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00021 [I 2022-10-11 14:18:43] ...trap/engines/engine.py.321: eval: epoch 107 | batch 460/510 [I 2022-10-11 14:18:43] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:16 [I 2022-10-11 14:18:43] ...trap/engines/engine.py.326: process: 0.31548 | load: 0.00022 [I 2022-10-11 14:18:46] ...trap/engines/engine.py.321: eval: epoch 107 | batch 470/510 [I 2022-10-11 14:18:46] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:12 [I 2022-10-11 14:18:46] ...trap/engines/engine.py.326: process: 0.32388 | load: 0.00021 [I 2022-10-11 14:18:49] ...trap/engines/engine.py.321: eval: epoch 107 | batch 480/510 [I 2022-10-11 14:18:49] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:09 [I 2022-10-11 14:18:49] ...trap/engines/engine.py.326: process: 0.32466 | load: 0.00021 [I 2022-10-11 14:18:52] ...trap/engines/engine.py.321: eval: epoch 107 | batch 490/510 [I 2022-10-11 14:18:52] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-11 14:18:52] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00021 [I 2022-10-11 14:18:56] ...trap/engines/engine.py.321: eval: epoch 107 | batch 500/510 [I 2022-10-11 14:18:56] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-11 14:18:56] ...trap/engines/engine.py.326: process: 0.34203 | load: 0.00022 [I 2022-10-11 14:18:59] ...trap/engines/engine.py.321: eval: epoch 107 | batch 510/510 [I 2022-10-11 14:18:59] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-11 14:18:59] ...trap/engines/engine.py.326: process: 0.31731 | load: 0.00017 [S 2022-10-11 14:19:00] ...trap/engines/engine.py.132: eval_epoch.epoch: 107 [I 2022-10-11 14:19:00] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.741000 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905100 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932800 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735900 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904000 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933300 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010982 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009544 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006353 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012186 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008355 [S 2022-10-11 14:19:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007198 [I 2022-10-11 14:19:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 14:19:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 14:19:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 14:19:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 14:19:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 14:19:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 14:19:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 14:19:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 14:19:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 108 [I 2022-10-11 14:19:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 14:19:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 0/2304 [I 2022-10-11 14:19:59] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:12:40 [I 2022-10-11 14:19:59] ...trap/engines/engine.py.229: process: 3.45493 | load: 2.52479 [I 2022-10-11 14:19:59] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-11 14:19:59] ...optimizers/trijoint.py.151: total_norm: 10.914678 [I 2022-10-11 14:20:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 10/2304 [I 2022-10-11 14:20:08] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:44:24 [I 2022-10-11 14:20:08] ...trap/engines/engine.py.229: process: 0.88793 | load: 0.00688 [I 2022-10-11 14:20:08] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-11 14:20:08] ...optimizers/trijoint.py.151: total_norm: 21.573200 [I 2022-10-11 14:20:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 20/2304 [I 2022-10-11 14:20:17] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:49 [I 2022-10-11 14:20:17] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00502 [I 2022-10-11 14:20:17] ...trap/engines/engine.py.230: loss: 0.11675 [I 2022-10-11 14:20:17] ...optimizers/trijoint.py.151: total_norm: 14.279189 [I 2022-10-11 14:20:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 30/2304 [I 2022-10-11 14:20:25] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:31 [I 2022-10-11 14:20:25] ...trap/engines/engine.py.229: process: 0.85176 | load: 0.00397 [I 2022-10-11 14:20:25] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-11 14:20:25] ...optimizers/trijoint.py.151: total_norm: 18.274029 [I 2022-10-11 14:20:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 40/2304 [I 2022-10-11 14:20:34] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:09 [I 2022-10-11 14:20:34] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00418 [I 2022-10-11 14:20:34] ...trap/engines/engine.py.230: loss: 0.13834 [I 2022-10-11 14:20:34] ...optimizers/trijoint.py.151: total_norm: 12.128901 [I 2022-10-11 14:20:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 50/2304 [I 2022-10-11 14:20:42] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:31:57 [I 2022-10-11 14:20:42] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00388 [I 2022-10-11 14:20:42] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-11 14:20:42] ...optimizers/trijoint.py.151: total_norm: 9.943865 [I 2022-10-11 14:20:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 60/2304 [I 2022-10-11 14:20:51] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:31:53 [I 2022-10-11 14:20:51] ...trap/engines/engine.py.229: process: 0.85089 | load: 0.00402 [I 2022-10-11 14:20:51] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-11 14:20:51] ...optimizers/trijoint.py.151: total_norm: 18.397755 [I 2022-10-11 14:20:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 70/2304 [I 2022-10-11 14:20:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:55 [I 2022-10-11 14:20:59] ...trap/engines/engine.py.229: process: 0.85978 | load: 0.00371 [I 2022-10-11 14:20:59] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-11 14:20:59] ...optimizers/trijoint.py.151: total_norm: 13.894792 [I 2022-10-11 14:21:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 80/2304 [I 2022-10-11 14:21:08] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:23 [I 2022-10-11 14:21:08] ...trap/engines/engine.py.229: process: 0.84668 | load: 0.00353 [I 2022-10-11 14:21:08] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-11 14:21:08] ...optimizers/trijoint.py.151: total_norm: 17.956947 [I 2022-10-11 14:21:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 90/2304 [I 2022-10-11 14:21:16] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:02 [I 2022-10-11 14:21:16] ...trap/engines/engine.py.229: process: 0.84419 | load: 0.00319 [I 2022-10-11 14:21:16] ...trap/engines/engine.py.230: loss: 0.10834 [I 2022-10-11 14:21:16] ...optimizers/trijoint.py.151: total_norm: 18.700739 [I 2022-10-11 14:21:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 100/2304 [I 2022-10-11 14:21:25] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:30:51 [I 2022-10-11 14:21:25] ...trap/engines/engine.py.229: process: 0.84138 | load: 0.00331 [I 2022-10-11 14:21:25] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-11 14:21:25] ...optimizers/trijoint.py.151: total_norm: 13.603697 [I 2022-10-11 14:21:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 110/2304 [I 2022-10-11 14:21:33] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:30:49 [I 2022-10-11 14:21:33] ...trap/engines/engine.py.229: process: 0.85379 | load: 0.00341 [I 2022-10-11 14:21:33] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-11 14:21:33] ...optimizers/trijoint.py.151: total_norm: 18.713327 [I 2022-10-11 14:21:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 120/2304 [I 2022-10-11 14:21:42] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:30:47 [I 2022-10-11 14:21:42] ...trap/engines/engine.py.229: process: 0.84149 | load: 0.00306 [I 2022-10-11 14:21:42] ...trap/engines/engine.py.230: loss: 0.19714 [I 2022-10-11 14:21:42] ...optimizers/trijoint.py.151: total_norm: 21.968397 [I 2022-10-11 14:21:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 130/2304 [I 2022-10-11 14:21:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:38 [I 2022-10-11 14:21:50] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00295 [I 2022-10-11 14:21:50] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-11 14:21:50] ...optimizers/trijoint.py.151: total_norm: 12.403164 [I 2022-10-11 14:21:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 140/2304 [I 2022-10-11 14:21:58] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:30 [I 2022-10-11 14:21:58] ...trap/engines/engine.py.229: process: 0.84202 | load: 0.00330 [I 2022-10-11 14:21:58] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-11 14:21:58] ...optimizers/trijoint.py.151: total_norm: 26.675554 [I 2022-10-11 14:22:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 150/2304 [I 2022-10-11 14:22:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:29:59 [I 2022-10-11 14:22:07] ...trap/engines/engine.py.229: process: 0.83892 | load: 0.00296 [I 2022-10-11 14:22:07] ...trap/engines/engine.py.230: loss: 0.09740 [I 2022-10-11 14:22:07] ...optimizers/trijoint.py.151: total_norm: 39.384686 [I 2022-10-11 14:22:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 160/2304 [I 2022-10-11 14:22:15] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:30:03 [I 2022-10-11 14:22:15] ...trap/engines/engine.py.229: process: 0.83306 | load: 0.00305 [I 2022-10-11 14:22:15] ...trap/engines/engine.py.230: loss: 0.07801 [I 2022-10-11 14:22:15] ...optimizers/trijoint.py.151: total_norm: 11.451088 [I 2022-10-11 14:22:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 170/2304 [I 2022-10-11 14:22:24] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:00 [I 2022-10-11 14:22:24] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00325 [I 2022-10-11 14:22:24] ...trap/engines/engine.py.230: loss: 0.09875 [I 2022-10-11 14:22:24] ...optimizers/trijoint.py.151: total_norm: 14.033401 [I 2022-10-11 14:22:32] ...trap/engines/engine.py.224: train: epoch 108 | batch 180/2304 [I 2022-10-11 14:22:32] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:29:54 [I 2022-10-11 14:22:32] ...trap/engines/engine.py.229: process: 0.84087 | load: 0.00306 [I 2022-10-11 14:22:32] ...trap/engines/engine.py.230: loss: 0.13032 [I 2022-10-11 14:22:32] ...optimizers/trijoint.py.151: total_norm: 9.044736 [I 2022-10-11 14:22:41] ...trap/engines/engine.py.224: train: epoch 108 | batch 190/2304 [I 2022-10-11 14:22:41] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:36 [I 2022-10-11 14:22:41] ...trap/engines/engine.py.229: process: 0.83166 | load: 0.00304 [I 2022-10-11 14:22:41] ...trap/engines/engine.py.230: loss: 0.14620 [I 2022-10-11 14:22:41] ...optimizers/trijoint.py.151: total_norm: 21.161766 [I 2022-10-11 14:22:49] ...trap/engines/engine.py.224: train: epoch 108 | batch 200/2304 [I 2022-10-11 14:22:49] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:29:27 [I 2022-10-11 14:22:49] ...trap/engines/engine.py.229: process: 0.83660 | load: 0.00297 [I 2022-10-11 14:22:49] ...trap/engines/engine.py.230: loss: 0.08510 [I 2022-10-11 14:22:49] ...optimizers/trijoint.py.151: total_norm: 12.636468 [I 2022-10-11 14:22:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 210/2304 [I 2022-10-11 14:22:58] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:29:34 [I 2022-10-11 14:22:58] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00299 [I 2022-10-11 14:22:58] ...trap/engines/engine.py.230: loss: 0.08840 [I 2022-10-11 14:22:58] ...optimizers/trijoint.py.151: total_norm: 18.726280 [I 2022-10-11 14:23:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 220/2304 [I 2022-10-11 14:23:06] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:29:21 [I 2022-10-11 14:23:06] ...trap/engines/engine.py.229: process: 0.82869 | load: 0.00301 [I 2022-10-11 14:23:06] ...trap/engines/engine.py.230: loss: 0.15703 [I 2022-10-11 14:23:06] ...optimizers/trijoint.py.151: total_norm: 9.035312 [I 2022-10-11 14:23:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 230/2304 [I 2022-10-11 14:23:15] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:08 [I 2022-10-11 14:23:15] ...trap/engines/engine.py.229: process: 0.84184 | load: 0.00310 [I 2022-10-11 14:23:15] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-11 14:23:15] ...optimizers/trijoint.py.151: total_norm: 8.355292 [I 2022-10-11 14:23:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 240/2304 [I 2022-10-11 14:23:23] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:29:06 [I 2022-10-11 14:23:23] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00300 [I 2022-10-11 14:23:23] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-11 14:23:23] ...optimizers/trijoint.py.151: total_norm: 10.622611 [I 2022-10-11 14:23:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 250/2304 [I 2022-10-11 14:23:31] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:03 [I 2022-10-11 14:23:31] ...trap/engines/engine.py.229: process: 0.86184 | load: 0.00299 [I 2022-10-11 14:23:31] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-11 14:23:31] ...optimizers/trijoint.py.151: total_norm: 9.380467 [I 2022-10-11 14:23:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 260/2304 [I 2022-10-11 14:23:40] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:28:47 [I 2022-10-11 14:23:40] ...trap/engines/engine.py.229: process: 0.83121 | load: 0.00310 [I 2022-10-11 14:23:40] ...trap/engines/engine.py.230: loss: 0.08122 [I 2022-10-11 14:23:40] ...optimizers/trijoint.py.151: total_norm: 11.308780 [I 2022-10-11 14:23:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 270/2304 [I 2022-10-11 14:23:48] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:33 [I 2022-10-11 14:23:48] ...trap/engines/engine.py.229: process: 0.84880 | load: 0.00300 [I 2022-10-11 14:23:48] ...trap/engines/engine.py.230: loss: 0.10019 [I 2022-10-11 14:23:48] ...optimizers/trijoint.py.151: total_norm: 13.222754 [I 2022-10-11 14:23:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 280/2304 [I 2022-10-11 14:23:57] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:28:28 [I 2022-10-11 14:23:57] ...trap/engines/engine.py.229: process: 0.84921 | load: 0.00302 [I 2022-10-11 14:23:57] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-11 14:23:57] ...optimizers/trijoint.py.151: total_norm: 12.785491 [I 2022-10-11 14:24:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 290/2304 [I 2022-10-11 14:24:05] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:28:34 [I 2022-10-11 14:24:05] ...trap/engines/engine.py.229: process: 0.84859 | load: 0.00329 [I 2022-10-11 14:24:05] ...trap/engines/engine.py.230: loss: 0.11915 [I 2022-10-11 14:24:05] ...optimizers/trijoint.py.151: total_norm: 14.414223 [I 2022-10-11 14:24:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 300/2304 [I 2022-10-11 14:24:14] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:33 [I 2022-10-11 14:24:14] ...trap/engines/engine.py.229: process: 0.85420 | load: 0.00313 [I 2022-10-11 14:24:14] ...trap/engines/engine.py.230: loss: 0.10450 [I 2022-10-11 14:24:14] ...optimizers/trijoint.py.151: total_norm: 16.037863 [I 2022-10-11 14:24:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 310/2304 [I 2022-10-11 14:24:23] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:31 [I 2022-10-11 14:24:23] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00315 [I 2022-10-11 14:24:23] ...trap/engines/engine.py.230: loss: 0.12211 [I 2022-10-11 14:24:23] ...optimizers/trijoint.py.151: total_norm: 12.417313 [I 2022-10-11 14:24:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 320/2304 [I 2022-10-11 14:24:31] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:11 [I 2022-10-11 14:24:31] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00329 [I 2022-10-11 14:24:31] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-11 14:24:31] ...optimizers/trijoint.py.151: total_norm: 28.521975 [I 2022-10-11 14:24:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 330/2304 [I 2022-10-11 14:24:40] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:28:02 [I 2022-10-11 14:24:40] ...trap/engines/engine.py.229: process: 0.85015 | load: 0.00328 [I 2022-10-11 14:24:40] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-11 14:24:40] ...optimizers/trijoint.py.151: total_norm: 37.960213 [I 2022-10-11 14:24:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 340/2304 [I 2022-10-11 14:24:48] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:27:59 [I 2022-10-11 14:24:48] ...trap/engines/engine.py.229: process: 0.84985 | load: 0.00315 [I 2022-10-11 14:24:48] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-11 14:24:48] ...optimizers/trijoint.py.151: total_norm: 18.961658 [I 2022-10-11 14:24:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 350/2304 [I 2022-10-11 14:24:57] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:56 [I 2022-10-11 14:24:57] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00326 [I 2022-10-11 14:24:57] ...trap/engines/engine.py.230: loss: 0.16015 [I 2022-10-11 14:24:57] ...optimizers/trijoint.py.151: total_norm: 25.836359 [I 2022-10-11 14:25:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 360/2304 [I 2022-10-11 14:25:05] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:55 [I 2022-10-11 14:25:05] ...trap/engines/engine.py.229: process: 0.87219 | load: 0.00312 [I 2022-10-11 14:25:05] ...trap/engines/engine.py.230: loss: 0.14487 [I 2022-10-11 14:25:05] ...optimizers/trijoint.py.151: total_norm: 9.587815 [I 2022-10-11 14:25:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 370/2304 [I 2022-10-11 14:25:14] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:43 [I 2022-10-11 14:25:14] ...trap/engines/engine.py.229: process: 0.85767 | load: 0.00498 [I 2022-10-11 14:25:14] ...trap/engines/engine.py.230: loss: 0.19696 [I 2022-10-11 14:25:14] ...optimizers/trijoint.py.151: total_norm: 13.861792 [I 2022-10-11 14:25:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 380/2304 [I 2022-10-11 14:25:23] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:27:22 [I 2022-10-11 14:25:23] ...trap/engines/engine.py.229: process: 0.86076 | load: 0.00314 [I 2022-10-11 14:25:23] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-11 14:25:23] ...optimizers/trijoint.py.151: total_norm: 28.807600 [I 2022-10-11 14:25:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 390/2304 [I 2022-10-11 14:25:31] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:29 [I 2022-10-11 14:25:31] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00312 [I 2022-10-11 14:25:31] ...trap/engines/engine.py.230: loss: 0.10659 [I 2022-10-11 14:25:31] ...optimizers/trijoint.py.151: total_norm: 29.248209 [I 2022-10-11 14:25:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 400/2304 [I 2022-10-11 14:25:40] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:22 [I 2022-10-11 14:25:40] ...trap/engines/engine.py.229: process: 0.86833 | load: 0.00313 [I 2022-10-11 14:25:40] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-11 14:25:40] ...optimizers/trijoint.py.151: total_norm: 10.778930 [I 2022-10-11 14:25:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 410/2304 [I 2022-10-11 14:25:48] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:27:05 [I 2022-10-11 14:25:48] ...trap/engines/engine.py.229: process: 0.86522 | load: 0.00313 [I 2022-10-11 14:25:48] ...trap/engines/engine.py.230: loss: 0.14575 [I 2022-10-11 14:25:48] ...optimizers/trijoint.py.151: total_norm: 22.036364 [I 2022-10-11 14:25:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 420/2304 [I 2022-10-11 14:25:57] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:51 [I 2022-10-11 14:25:57] ...trap/engines/engine.py.229: process: 0.85817 | load: 0.00314 [I 2022-10-11 14:25:57] ...trap/engines/engine.py.230: loss: 0.07602 [I 2022-10-11 14:25:57] ...optimizers/trijoint.py.151: total_norm: 10.059559 [I 2022-10-11 14:26:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 430/2304 [I 2022-10-11 14:26:06] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:38 [I 2022-10-11 14:26:06] ...trap/engines/engine.py.229: process: 0.84671 | load: 0.00315 [I 2022-10-11 14:26:06] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-11 14:26:06] ...optimizers/trijoint.py.151: total_norm: 26.589754 [I 2022-10-11 14:26:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 440/2304 [I 2022-10-11 14:26:14] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:40 [I 2022-10-11 14:26:14] ...trap/engines/engine.py.229: process: 0.85085 | load: 0.00312 [I 2022-10-11 14:26:14] ...trap/engines/engine.py.230: loss: 0.08383 [I 2022-10-11 14:26:14] ...optimizers/trijoint.py.151: total_norm: 13.831160 [I 2022-10-11 14:26:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 450/2304 [I 2022-10-11 14:26:23] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:26:24 [I 2022-10-11 14:26:23] ...trap/engines/engine.py.229: process: 0.84314 | load: 0.00316 [I 2022-10-11 14:26:23] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-11 14:26:23] ...optimizers/trijoint.py.151: total_norm: 20.485991 [I 2022-10-11 14:26:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 460/2304 [I 2022-10-11 14:26:31] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:20 [I 2022-10-11 14:26:31] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00314 [I 2022-10-11 14:26:31] ...trap/engines/engine.py.230: loss: 0.13202 [I 2022-10-11 14:26:31] ...optimizers/trijoint.py.151: total_norm: 24.038013 [I 2022-10-11 14:26:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 470/2304 [I 2022-10-11 14:26:40] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:26:05 [I 2022-10-11 14:26:40] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00314 [I 2022-10-11 14:26:40] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-11 14:26:40] ...optimizers/trijoint.py.151: total_norm: 24.063866 [I 2022-10-11 14:26:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 480/2304 [I 2022-10-11 14:26:48] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:26:07 [I 2022-10-11 14:26:48] ...trap/engines/engine.py.229: process: 0.87256 | load: 0.00313 [I 2022-10-11 14:26:48] ...trap/engines/engine.py.230: loss: 0.09623 [I 2022-10-11 14:26:48] ...optimizers/trijoint.py.151: total_norm: 16.077335 [I 2022-10-11 14:26:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 490/2304 [I 2022-10-11 14:26:57] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:47 [I 2022-10-11 14:26:57] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00316 [I 2022-10-11 14:26:57] ...trap/engines/engine.py.230: loss: 0.09517 [I 2022-10-11 14:26:57] ...optimizers/trijoint.py.151: total_norm: 13.532055 [I 2022-10-11 14:27:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 500/2304 [I 2022-10-11 14:27:06] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:44 [I 2022-10-11 14:27:06] ...trap/engines/engine.py.229: process: 0.86768 | load: 0.00317 [I 2022-10-11 14:27:06] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-11 14:27:06] ...optimizers/trijoint.py.151: total_norm: 16.603666 [I 2022-10-11 14:27:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 510/2304 [I 2022-10-11 14:27:34] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 1:03:33 [I 2022-10-11 14:27:34] ...trap/engines/engine.py.229: process: 0.85498 | load: 0.00319 [I 2022-10-11 14:27:34] ...trap/engines/engine.py.230: loss: 0.10404 [I 2022-10-11 14:27:34] ...optimizers/trijoint.py.151: total_norm: 18.911898 [I 2022-10-11 14:27:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 520/2304 [I 2022-10-11 14:27:42] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:29:32 [I 2022-10-11 14:27:42] ...trap/engines/engine.py.229: process: 0.86008 | load: 0.00313 [I 2022-10-11 14:27:42] ...trap/engines/engine.py.230: loss: 0.08772 [I 2022-10-11 14:27:42] ...optimizers/trijoint.py.151: total_norm: 10.596240 [I 2022-10-11 14:27:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 530/2304 [I 2022-10-11 14:27:51] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:40 [I 2022-10-11 14:27:51] ...trap/engines/engine.py.229: process: 0.86503 | load: 0.00318 [I 2022-10-11 14:27:51] ...trap/engines/engine.py.230: loss: 0.14374 [I 2022-10-11 14:27:51] ...optimizers/trijoint.py.151: total_norm: 23.942696 [I 2022-10-11 14:27:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 540/2304 [I 2022-10-11 14:27:59] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:25:06 [I 2022-10-11 14:27:59] ...trap/engines/engine.py.229: process: 0.84005 | load: 0.00311 [I 2022-10-11 14:27:59] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-11 14:27:59] ...optimizers/trijoint.py.151: total_norm: 7.636998 [I 2022-10-11 14:28:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 550/2304 [I 2022-10-11 14:28:08] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:24:49 [I 2022-10-11 14:28:08] ...trap/engines/engine.py.229: process: 0.83134 | load: 0.00297 [I 2022-10-11 14:28:08] ...trap/engines/engine.py.230: loss: 0.17420 [I 2022-10-11 14:28:08] ...optimizers/trijoint.py.151: total_norm: 24.989954 [I 2022-10-11 14:28:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 560/2304 [I 2022-10-11 14:28:16] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:45 [I 2022-10-11 14:28:16] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00297 [I 2022-10-11 14:28:16] ...trap/engines/engine.py.230: loss: 0.13137 [I 2022-10-11 14:28:16] ...optimizers/trijoint.py.151: total_norm: 13.946803 [I 2022-10-11 14:28:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 570/2304 [I 2022-10-11 14:28:25] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:24:26 [I 2022-10-11 14:28:25] ...trap/engines/engine.py.229: process: 0.83596 | load: 0.00297 [I 2022-10-11 14:28:25] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-11 14:28:25] ...optimizers/trijoint.py.151: total_norm: 12.278439 [I 2022-10-11 14:28:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 580/2304 [I 2022-10-11 14:28:33] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:24:25 [I 2022-10-11 14:28:33] ...trap/engines/engine.py.229: process: 0.85399 | load: 0.00303 [I 2022-10-11 14:28:33] ...trap/engines/engine.py.230: loss: 0.09539 [I 2022-10-11 14:28:33] ...optimizers/trijoint.py.151: total_norm: 18.410501 [I 2022-10-11 14:28:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 590/2304 [I 2022-10-11 14:28:42] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:24:12 [I 2022-10-11 14:28:42] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00296 [I 2022-10-11 14:28:42] ...trap/engines/engine.py.230: loss: 0.11890 [I 2022-10-11 14:28:42] ...optimizers/trijoint.py.151: total_norm: 9.637402 [I 2022-10-11 14:28:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 600/2304 [I 2022-10-11 14:28:50] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:24:13 [I 2022-10-11 14:28:50] ...trap/engines/engine.py.229: process: 0.85370 | load: 0.00303 [I 2022-10-11 14:28:50] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-11 14:28:50] ...optimizers/trijoint.py.151: total_norm: 12.544114 [I 2022-10-11 14:28:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 610/2304 [I 2022-10-11 14:28:59] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:23:51 [I 2022-10-11 14:28:59] ...trap/engines/engine.py.229: process: 0.83275 | load: 0.00297 [I 2022-10-11 14:28:59] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-11 14:28:59] ...optimizers/trijoint.py.151: total_norm: 13.012549 [I 2022-10-11 14:29:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 620/2304 [I 2022-10-11 14:29:08] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:49 [I 2022-10-11 14:29:08] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00297 [I 2022-10-11 14:29:08] ...trap/engines/engine.py.230: loss: 0.19482 [I 2022-10-11 14:29:08] ...optimizers/trijoint.py.151: total_norm: 12.526087 [I 2022-10-11 14:29:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 630/2304 [I 2022-10-11 14:29:16] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:23:41 [I 2022-10-11 14:29:16] ...trap/engines/engine.py.229: process: 0.84907 | load: 0.00298 [I 2022-10-11 14:29:16] ...trap/engines/engine.py.230: loss: 0.11560 [I 2022-10-11 14:29:16] ...optimizers/trijoint.py.151: total_norm: 12.382936 [I 2022-10-11 14:29:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 640/2304 [I 2022-10-11 14:29:24] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:23:30 [I 2022-10-11 14:29:24] ...trap/engines/engine.py.229: process: 0.85515 | load: 0.00299 [I 2022-10-11 14:29:24] ...trap/engines/engine.py.230: loss: 0.12407 [I 2022-10-11 14:29:24] ...optimizers/trijoint.py.151: total_norm: 25.536894 [I 2022-10-11 14:29:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 650/2304 [I 2022-10-11 14:29:33] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:23:25 [I 2022-10-11 14:29:33] ...trap/engines/engine.py.229: process: 0.85011 | load: 0.00298 [I 2022-10-11 14:29:33] ...trap/engines/engine.py.230: loss: 0.08192 [I 2022-10-11 14:29:33] ...optimizers/trijoint.py.151: total_norm: 12.055036 [I 2022-10-11 14:29:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 660/2304 [I 2022-10-11 14:29:42] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:23:18 [I 2022-10-11 14:29:42] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00296 [I 2022-10-11 14:29:42] ...trap/engines/engine.py.230: loss: 0.20655 [I 2022-10-11 14:29:42] ...optimizers/trijoint.py.151: total_norm: 50.112530 [I 2022-10-11 14:29:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 670/2304 [I 2022-10-11 14:29:50] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:23:58 [I 2022-10-11 14:29:50] ...trap/engines/engine.py.229: process: 0.88972 | load: 0.00375 [I 2022-10-11 14:29:50] ...trap/engines/engine.py.230: loss: 0.11774 [I 2022-10-11 14:29:50] ...optimizers/trijoint.py.151: total_norm: 10.542220 [I 2022-10-11 14:29:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 680/2304 [I 2022-10-11 14:29:59] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:23:47 [I 2022-10-11 14:29:59] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00373 [I 2022-10-11 14:29:59] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-11 14:29:59] ...optimizers/trijoint.py.151: total_norm: 26.563793 [I 2022-10-11 14:30:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 690/2304 [I 2022-10-11 14:30:08] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:23:56 [I 2022-10-11 14:30:08] ...trap/engines/engine.py.229: process: 0.90127 | load: 0.00378 [I 2022-10-11 14:30:08] ...trap/engines/engine.py.230: loss: 0.18775 [I 2022-10-11 14:30:08] ...optimizers/trijoint.py.151: total_norm: 18.648293 [I 2022-10-11 14:30:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 700/2304 [I 2022-10-11 14:30:17] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:23:46 [I 2022-10-11 14:30:17] ...trap/engines/engine.py.229: process: 0.88278 | load: 0.00373 [I 2022-10-11 14:30:17] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-11 14:30:17] ...optimizers/trijoint.py.151: total_norm: 14.095167 [I 2022-10-11 14:30:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 710/2304 [I 2022-10-11 14:30:26] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:23:37 [I 2022-10-11 14:30:26] ...trap/engines/engine.py.229: process: 0.89575 | load: 0.00374 [I 2022-10-11 14:30:26] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-11 14:30:26] ...optimizers/trijoint.py.151: total_norm: 16.208014 [I 2022-10-11 14:30:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 720/2304 [I 2022-10-11 14:30:35] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:22:32 [I 2022-10-11 14:30:35] ...trap/engines/engine.py.229: process: 0.84688 | load: 0.00297 [I 2022-10-11 14:30:35] ...trap/engines/engine.py.230: loss: 0.13477 [I 2022-10-11 14:30:35] ...optimizers/trijoint.py.151: total_norm: 12.068933 [I 2022-10-11 14:30:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 730/2304 [I 2022-10-11 14:30:43] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:22:05 [I 2022-10-11 14:30:43] ...trap/engines/engine.py.229: process: 0.83853 | load: 0.00297 [I 2022-10-11 14:30:43] ...trap/engines/engine.py.230: loss: 0.09488 [I 2022-10-11 14:30:43] ...optimizers/trijoint.py.151: total_norm: 20.967211 [I 2022-10-11 14:30:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 740/2304 [I 2022-10-11 14:30:51] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:21:57 [I 2022-10-11 14:30:51] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00298 [I 2022-10-11 14:30:51] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-11 14:30:51] ...optimizers/trijoint.py.151: total_norm: 21.319986 [I 2022-10-11 14:31:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 750/2304 [I 2022-10-11 14:31:00] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:21:50 [I 2022-10-11 14:31:00] ...trap/engines/engine.py.229: process: 0.83811 | load: 0.00298 [I 2022-10-11 14:31:00] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-11 14:31:00] ...optimizers/trijoint.py.151: total_norm: 9.720953 [I 2022-10-11 14:31:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 760/2304 [I 2022-10-11 14:31:08] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:21:45 [I 2022-10-11 14:31:08] ...trap/engines/engine.py.229: process: 0.85405 | load: 0.00296 [I 2022-10-11 14:31:08] ...trap/engines/engine.py.230: loss: 0.12628 [I 2022-10-11 14:31:08] ...optimizers/trijoint.py.151: total_norm: 12.892468 [I 2022-10-11 14:31:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 770/2304 [I 2022-10-11 14:31:17] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:21:31 [I 2022-10-11 14:31:17] ...trap/engines/engine.py.229: process: 0.85053 | load: 0.00294 [I 2022-10-11 14:31:17] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-11 14:31:17] ...optimizers/trijoint.py.151: total_norm: 14.806333 [I 2022-10-11 14:31:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 780/2304 [I 2022-10-11 14:31:25] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:21:23 [I 2022-10-11 14:31:25] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00294 [I 2022-10-11 14:31:25] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-11 14:31:25] ...optimizers/trijoint.py.151: total_norm: 12.018634 [I 2022-10-11 14:31:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 790/2304 [I 2022-10-11 14:31:34] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:21:13 [I 2022-10-11 14:31:34] ...trap/engines/engine.py.229: process: 0.82978 | load: 0.00298 [I 2022-10-11 14:31:34] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-11 14:31:34] ...optimizers/trijoint.py.151: total_norm: 17.015875 [I 2022-10-11 14:31:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 800/2304 [I 2022-10-11 14:31:42] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:21:05 [I 2022-10-11 14:31:42] ...trap/engines/engine.py.229: process: 0.84441 | load: 0.00298 [I 2022-10-11 14:31:42] ...trap/engines/engine.py.230: loss: 0.09302 [I 2022-10-11 14:31:42] ...optimizers/trijoint.py.151: total_norm: 22.206062 [I 2022-10-11 14:31:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 810/2304 [I 2022-10-11 14:31:51] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:21:04 [I 2022-10-11 14:31:51] ...trap/engines/engine.py.229: process: 0.84195 | load: 0.00314 [I 2022-10-11 14:31:51] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-11 14:31:51] ...optimizers/trijoint.py.151: total_norm: 22.799927 [I 2022-10-11 14:31:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 820/2304 [I 2022-10-11 14:31:59] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:21:06 [I 2022-10-11 14:31:59] ...trap/engines/engine.py.229: process: 0.86538 | load: 0.00311 [I 2022-10-11 14:31:59] ...trap/engines/engine.py.230: loss: 0.13971 [I 2022-10-11 14:31:59] ...optimizers/trijoint.py.151: total_norm: 17.757719 [I 2022-10-11 14:32:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 830/2304 [I 2022-10-11 14:32:08] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:20:53 [I 2022-10-11 14:32:08] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00322 [I 2022-10-11 14:32:08] ...trap/engines/engine.py.230: loss: 0.16679 [I 2022-10-11 14:32:08] ...optimizers/trijoint.py.151: total_norm: 22.491060 [I 2022-10-11 14:32:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 840/2304 [I 2022-10-11 14:32:16] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:20:56 [I 2022-10-11 14:32:16] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00317 [I 2022-10-11 14:32:16] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-11 14:32:16] ...optimizers/trijoint.py.151: total_norm: 12.713924 [I 2022-10-11 14:32:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 850/2304 [I 2022-10-11 14:32:25] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:20:44 [I 2022-10-11 14:32:25] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00317 [I 2022-10-11 14:32:25] ...trap/engines/engine.py.230: loss: 0.11663 [I 2022-10-11 14:32:25] ...optimizers/trijoint.py.151: total_norm: 11.519131 [I 2022-10-11 14:32:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 860/2304 [I 2022-10-11 14:32:34] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:20:43 [I 2022-10-11 14:32:34] ...trap/engines/engine.py.229: process: 0.87277 | load: 0.00316 [I 2022-10-11 14:32:34] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-11 14:32:34] ...optimizers/trijoint.py.151: total_norm: 12.448584 [I 2022-10-11 14:32:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 870/2304 [I 2022-10-11 14:32:42] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:20:33 [I 2022-10-11 14:32:42] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00314 [I 2022-10-11 14:32:42] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-11 14:32:42] ...optimizers/trijoint.py.151: total_norm: 9.284825 [I 2022-10-11 14:32:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 880/2304 [I 2022-10-11 14:32:51] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:20:24 [I 2022-10-11 14:32:51] ...trap/engines/engine.py.229: process: 0.86509 | load: 0.00311 [I 2022-10-11 14:32:51] ...trap/engines/engine.py.230: loss: 0.14026 [I 2022-10-11 14:32:51] ...optimizers/trijoint.py.151: total_norm: 21.689207 [I 2022-10-11 14:32:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 890/2304 [I 2022-10-11 14:32:59] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:20:24 [I 2022-10-11 14:32:59] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00314 [I 2022-10-11 14:32:59] ...trap/engines/engine.py.230: loss: 0.24709 [I 2022-10-11 14:32:59] ...optimizers/trijoint.py.151: total_norm: 48.933506 [I 2022-10-11 14:33:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 900/2304 [I 2022-10-11 14:33:08] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:20:00 [I 2022-10-11 14:33:08] ...trap/engines/engine.py.229: process: 0.84227 | load: 0.00315 [I 2022-10-11 14:33:08] ...trap/engines/engine.py.230: loss: 0.05771 [I 2022-10-11 14:33:08] ...optimizers/trijoint.py.151: total_norm: 12.817915 [I 2022-10-11 14:33:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 910/2304 [I 2022-10-11 14:33:17] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:54 [I 2022-10-11 14:33:17] ...trap/engines/engine.py.229: process: 0.85064 | load: 0.00315 [I 2022-10-11 14:33:17] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-11 14:33:17] ...optimizers/trijoint.py.151: total_norm: 16.380882 [I 2022-10-11 14:33:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 920/2304 [I 2022-10-11 14:33:25] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:19:42 [I 2022-10-11 14:33:25] ...trap/engines/engine.py.229: process: 0.85642 | load: 0.00313 [I 2022-10-11 14:33:25] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-11 14:33:25] ...optimizers/trijoint.py.151: total_norm: 12.321104 [I 2022-10-11 14:33:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 930/2304 [I 2022-10-11 14:33:34] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:19:35 [I 2022-10-11 14:33:34] ...trap/engines/engine.py.229: process: 0.85417 | load: 0.00316 [I 2022-10-11 14:33:34] ...trap/engines/engine.py.230: loss: 0.09776 [I 2022-10-11 14:33:34] ...optimizers/trijoint.py.151: total_norm: 21.200186 [I 2022-10-11 14:33:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 940/2304 [I 2022-10-11 14:33:42] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:19:26 [I 2022-10-11 14:33:42] ...trap/engines/engine.py.229: process: 0.86602 | load: 0.00315 [I 2022-10-11 14:33:42] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-11 14:33:42] ...optimizers/trijoint.py.151: total_norm: 23.451969 [I 2022-10-11 14:33:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 950/2304 [I 2022-10-11 14:33:51] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:19:25 [I 2022-10-11 14:33:51] ...trap/engines/engine.py.229: process: 0.85173 | load: 0.00314 [I 2022-10-11 14:33:51] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-11 14:33:51] ...optimizers/trijoint.py.151: total_norm: 30.721264 [I 2022-10-11 14:34:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 960/2304 [I 2022-10-11 14:34:00] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:19:20 [I 2022-10-11 14:34:00] ...trap/engines/engine.py.229: process: 0.86243 | load: 0.00315 [I 2022-10-11 14:34:00] ...trap/engines/engine.py.230: loss: 0.12792 [I 2022-10-11 14:34:00] ...optimizers/trijoint.py.151: total_norm: 19.072668 [I 2022-10-11 14:34:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 970/2304 [I 2022-10-11 14:34:08] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:19:02 [I 2022-10-11 14:34:08] ...trap/engines/engine.py.229: process: 0.85877 | load: 0.00312 [I 2022-10-11 14:34:08] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-11 14:34:08] ...optimizers/trijoint.py.151: total_norm: 10.220735 [I 2022-10-11 14:34:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 980/2304 [I 2022-10-11 14:34:17] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:18:51 [I 2022-10-11 14:34:17] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00313 [I 2022-10-11 14:34:17] ...trap/engines/engine.py.230: loss: 0.06447 [I 2022-10-11 14:34:17] ...optimizers/trijoint.py.151: total_norm: 9.236150 [I 2022-10-11 14:34:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 990/2304 [I 2022-10-11 14:34:25] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:18:47 [I 2022-10-11 14:34:25] ...trap/engines/engine.py.229: process: 0.86208 | load: 0.00313 [I 2022-10-11 14:34:25] ...trap/engines/engine.py.230: loss: 0.18456 [I 2022-10-11 14:34:25] ...optimizers/trijoint.py.151: total_norm: 13.992153 [I 2022-10-11 14:34:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 1000/2304 [I 2022-10-11 14:34:34] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:18:44 [I 2022-10-11 14:34:34] ...trap/engines/engine.py.229: process: 0.85912 | load: 0.00312 [I 2022-10-11 14:34:34] ...trap/engines/engine.py.230: loss: 0.16961 [I 2022-10-11 14:34:34] ...optimizers/trijoint.py.151: total_norm: 20.087072 [I 2022-10-11 14:34:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 1010/2304 [I 2022-10-11 14:34:43] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:18:29 [I 2022-10-11 14:34:43] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00313 [I 2022-10-11 14:34:43] ...trap/engines/engine.py.230: loss: 0.10486 [I 2022-10-11 14:34:43] ...optimizers/trijoint.py.151: total_norm: 49.527637 [I 2022-10-11 14:34:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 1020/2304 [I 2022-10-11 14:34:51] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:18:22 [I 2022-10-11 14:34:51] ...trap/engines/engine.py.229: process: 0.86538 | load: 0.00324 [I 2022-10-11 14:34:51] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-11 14:34:51] ...optimizers/trijoint.py.151: total_norm: 18.496582 [I 2022-10-11 14:35:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 1030/2304 [I 2022-10-11 14:35:00] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:18:50 [I 2022-10-11 14:35:00] ...trap/engines/engine.py.229: process: 0.91059 | load: 0.00399 [I 2022-10-11 14:35:00] ...trap/engines/engine.py.230: loss: 0.09800 [I 2022-10-11 14:35:00] ...optimizers/trijoint.py.151: total_norm: 7.936543 [I 2022-10-11 14:35:09] ...trap/engines/engine.py.224: train: epoch 108 | batch 1040/2304 [I 2022-10-11 14:35:09] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:18:48 [I 2022-10-11 14:35:09] ...trap/engines/engine.py.229: process: 0.89024 | load: 0.00395 [I 2022-10-11 14:35:09] ...trap/engines/engine.py.230: loss: 0.09380 [I 2022-10-11 14:35:09] ...optimizers/trijoint.py.151: total_norm: 12.977715 [I 2022-10-11 14:35:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 1050/2304 [I 2022-10-11 14:35:18] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:18:30 [I 2022-10-11 14:35:18] ...trap/engines/engine.py.229: process: 0.88703 | load: 0.00395 [I 2022-10-11 14:35:18] ...trap/engines/engine.py.230: loss: 0.15671 [I 2022-10-11 14:35:18] ...optimizers/trijoint.py.151: total_norm: 21.997105 [I 2022-10-11 14:35:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 1060/2304 [I 2022-10-11 14:35:27] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:18:25 [I 2022-10-11 14:35:27] ...trap/engines/engine.py.229: process: 0.89798 | load: 0.00398 [I 2022-10-11 14:35:27] ...trap/engines/engine.py.230: loss: 0.07560 [I 2022-10-11 14:35:27] ...optimizers/trijoint.py.151: total_norm: 8.207164 [I 2022-10-11 14:35:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1070/2304 [I 2022-10-11 14:35:36] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:18:20 [I 2022-10-11 14:35:36] ...trap/engines/engine.py.229: process: 0.88458 | load: 0.00398 [I 2022-10-11 14:35:36] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-11 14:35:36] ...optimizers/trijoint.py.151: total_norm: 9.963488 [I 2022-10-11 14:35:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 1080/2304 [I 2022-10-11 14:35:45] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:17:54 [I 2022-10-11 14:35:45] ...trap/engines/engine.py.229: process: 0.88128 | load: 0.00386 [I 2022-10-11 14:35:45] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-11 14:35:45] ...optimizers/trijoint.py.151: total_norm: 11.769404 [I 2022-10-11 14:35:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1090/2304 [I 2022-10-11 14:35:53] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:17:41 [I 2022-10-11 14:35:53] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00382 [I 2022-10-11 14:35:53] ...trap/engines/engine.py.230: loss: 0.12247 [I 2022-10-11 14:35:53] ...optimizers/trijoint.py.151: total_norm: 12.887119 [I 2022-10-11 14:36:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 1100/2304 [I 2022-10-11 14:36:02] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:17:04 [I 2022-10-11 14:36:02] ...trap/engines/engine.py.229: process: 0.85385 | load: 0.00297 [I 2022-10-11 14:36:02] ...trap/engines/engine.py.230: loss: 0.12962 [I 2022-10-11 14:36:02] ...optimizers/trijoint.py.151: total_norm: 9.821955 [I 2022-10-11 14:36:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 1110/2304 [I 2022-10-11 14:36:10] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:17:00 [I 2022-10-11 14:36:10] ...trap/engines/engine.py.229: process: 0.86932 | load: 0.00296 [I 2022-10-11 14:36:10] ...trap/engines/engine.py.230: loss: 0.14075 [I 2022-10-11 14:36:10] ...optimizers/trijoint.py.151: total_norm: 13.243823 [I 2022-10-11 14:36:19] ...trap/engines/engine.py.224: train: epoch 108 | batch 1120/2304 [I 2022-10-11 14:36:19] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:42 [I 2022-10-11 14:36:19] ...trap/engines/engine.py.229: process: 0.85198 | load: 0.00294 [I 2022-10-11 14:36:19] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-11 14:36:19] ...optimizers/trijoint.py.151: total_norm: 12.301277 [I 2022-10-11 14:36:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 1130/2304 [I 2022-10-11 14:36:27] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:43 [I 2022-10-11 14:36:27] ...trap/engines/engine.py.229: process: 0.86436 | load: 0.00297 [I 2022-10-11 14:36:27] ...trap/engines/engine.py.230: loss: 0.11037 [I 2022-10-11 14:36:27] ...optimizers/trijoint.py.151: total_norm: 15.960025 [I 2022-10-11 14:36:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1140/2304 [I 2022-10-11 14:36:36] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:34 [I 2022-10-11 14:36:36] ...trap/engines/engine.py.229: process: 0.86432 | load: 0.00295 [I 2022-10-11 14:36:36] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-11 14:36:36] ...optimizers/trijoint.py.151: total_norm: 14.932525 [I 2022-10-11 14:36:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 1150/2304 [I 2022-10-11 14:36:45] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:16:26 [I 2022-10-11 14:36:45] ...trap/engines/engine.py.229: process: 0.85830 | load: 0.00294 [I 2022-10-11 14:36:45] ...trap/engines/engine.py.230: loss: 0.09764 [I 2022-10-11 14:36:45] ...optimizers/trijoint.py.151: total_norm: 20.164373 [I 2022-10-11 14:36:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1160/2304 [I 2022-10-11 14:36:53] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:16:17 [I 2022-10-11 14:36:53] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00294 [I 2022-10-11 14:36:53] ...trap/engines/engine.py.230: loss: 0.14901 [I 2022-10-11 14:36:53] ...optimizers/trijoint.py.151: total_norm: 39.950562 [I 2022-10-11 14:37:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 1170/2304 [I 2022-10-11 14:37:02] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:16:13 [I 2022-10-11 14:37:02] ...trap/engines/engine.py.229: process: 0.85870 | load: 0.00299 [I 2022-10-11 14:37:02] ...trap/engines/engine.py.230: loss: 0.17339 [I 2022-10-11 14:37:02] ...optimizers/trijoint.py.151: total_norm: 22.772038 [I 2022-10-11 14:37:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 1180/2304 [I 2022-10-11 14:37:10] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:16:07 [I 2022-10-11 14:37:10] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00298 [I 2022-10-11 14:37:10] ...trap/engines/engine.py.230: loss: 0.14089 [I 2022-10-11 14:37:10] ...optimizers/trijoint.py.151: total_norm: 16.895454 [I 2022-10-11 14:37:19] ...trap/engines/engine.py.224: train: epoch 108 | batch 1190/2304 [I 2022-10-11 14:37:19] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:52 [I 2022-10-11 14:37:19] ...trap/engines/engine.py.229: process: 0.85304 | load: 0.00300 [I 2022-10-11 14:37:19] ...trap/engines/engine.py.230: loss: 0.16779 [I 2022-10-11 14:37:19] ...optimizers/trijoint.py.151: total_norm: 25.675339 [I 2022-10-11 14:37:28] ...trap/engines/engine.py.224: train: epoch 108 | batch 1200/2304 [I 2022-10-11 14:37:28] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:15:42 [I 2022-10-11 14:37:28] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00302 [I 2022-10-11 14:37:28] ...trap/engines/engine.py.230: loss: 0.12665 [I 2022-10-11 14:37:28] ...optimizers/trijoint.py.151: total_norm: 18.247444 [I 2022-10-11 14:37:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1210/2304 [I 2022-10-11 14:37:36] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:28 [I 2022-10-11 14:37:36] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00304 [I 2022-10-11 14:37:36] ...trap/engines/engine.py.230: loss: 0.09448 [I 2022-10-11 14:37:36] ...optimizers/trijoint.py.151: total_norm: 15.787994 [I 2022-10-11 14:37:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 1220/2304 [I 2022-10-11 14:37:45] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:15:19 [I 2022-10-11 14:37:45] ...trap/engines/engine.py.229: process: 0.84212 | load: 0.00295 [I 2022-10-11 14:37:45] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-11 14:37:45] ...optimizers/trijoint.py.151: total_norm: 13.910568 [I 2022-10-11 14:37:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1230/2304 [I 2022-10-11 14:37:53] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:15:08 [I 2022-10-11 14:37:53] ...trap/engines/engine.py.229: process: 0.85673 | load: 0.00294 [I 2022-10-11 14:37:53] ...trap/engines/engine.py.230: loss: 0.07697 [I 2022-10-11 14:37:53] ...optimizers/trijoint.py.151: total_norm: 12.222726 [I 2022-10-11 14:38:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 1240/2304 [I 2022-10-11 14:38:02] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:15:07 [I 2022-10-11 14:38:02] ...trap/engines/engine.py.229: process: 0.86452 | load: 0.00297 [I 2022-10-11 14:38:02] ...trap/engines/engine.py.230: loss: 0.17708 [I 2022-10-11 14:38:02] ...optimizers/trijoint.py.151: total_norm: 9.928214 [I 2022-10-11 14:38:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 1250/2304 [I 2022-10-11 14:38:10] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:44 [I 2022-10-11 14:38:10] ...trap/engines/engine.py.229: process: 0.81422 | load: 0.00297 [I 2022-10-11 14:38:10] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-11 14:38:10] ...optimizers/trijoint.py.151: total_norm: 20.781548 [I 2022-10-11 14:38:19] ...trap/engines/engine.py.224: train: epoch 108 | batch 1260/2304 [I 2022-10-11 14:38:19] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:47 [I 2022-10-11 14:38:19] ...trap/engines/engine.py.229: process: 0.84583 | load: 0.00301 [I 2022-10-11 14:38:19] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-11 14:38:19] ...optimizers/trijoint.py.151: total_norm: 26.992830 [I 2022-10-11 14:38:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 1270/2304 [I 2022-10-11 14:38:27] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:41 [I 2022-10-11 14:38:27] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00302 [I 2022-10-11 14:38:27] ...trap/engines/engine.py.230: loss: 0.08458 [I 2022-10-11 14:38:27] ...optimizers/trijoint.py.151: total_norm: 18.316847 [I 2022-10-11 14:38:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1280/2304 [I 2022-10-11 14:38:36] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:40 [I 2022-10-11 14:38:36] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00302 [I 2022-10-11 14:38:36] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-11 14:38:36] ...optimizers/trijoint.py.151: total_norm: 18.226318 [I 2022-10-11 14:38:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 1290/2304 [I 2022-10-11 14:38:44] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:30 [I 2022-10-11 14:38:44] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00305 [I 2022-10-11 14:38:44] ...trap/engines/engine.py.230: loss: 0.20620 [I 2022-10-11 14:38:44] ...optimizers/trijoint.py.151: total_norm: 23.237534 [I 2022-10-11 14:38:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1300/2304 [I 2022-10-11 14:38:53] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:14:23 [I 2022-10-11 14:38:53] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00300 [I 2022-10-11 14:38:53] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-11 14:38:53] ...optimizers/trijoint.py.151: total_norm: 14.564633 [I 2022-10-11 14:39:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 1310/2304 [I 2022-10-11 14:39:02] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:14:14 [I 2022-10-11 14:39:02] ...trap/engines/engine.py.229: process: 0.85387 | load: 0.00298 [I 2022-10-11 14:39:02] ...trap/engines/engine.py.230: loss: 0.11741 [I 2022-10-11 14:39:02] ...optimizers/trijoint.py.151: total_norm: 18.173086 [I 2022-10-11 14:39:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 1320/2304 [I 2022-10-11 14:39:10] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:14:05 [I 2022-10-11 14:39:10] ...trap/engines/engine.py.229: process: 0.86249 | load: 0.00299 [I 2022-10-11 14:39:10] ...trap/engines/engine.py.230: loss: 0.16487 [I 2022-10-11 14:39:10] ...optimizers/trijoint.py.151: total_norm: 21.080574 [I 2022-10-11 14:39:19] ...trap/engines/engine.py.224: train: epoch 108 | batch 1330/2304 [I 2022-10-11 14:39:19] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:58 [I 2022-10-11 14:39:19] ...trap/engines/engine.py.229: process: 0.85247 | load: 0.00307 [I 2022-10-11 14:39:19] ...trap/engines/engine.py.230: loss: 0.12492 [I 2022-10-11 14:39:19] ...optimizers/trijoint.py.151: total_norm: 15.004922 [I 2022-10-11 14:39:28] ...trap/engines/engine.py.224: train: epoch 108 | batch 1340/2304 [I 2022-10-11 14:39:28] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:14:01 [I 2022-10-11 14:39:28] ...trap/engines/engine.py.229: process: 0.88321 | load: 0.00376 [I 2022-10-11 14:39:28] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-11 14:39:28] ...optimizers/trijoint.py.151: total_norm: 39.168129 [I 2022-10-11 14:39:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1350/2304 [I 2022-10-11 14:39:36] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:56 [I 2022-10-11 14:39:36] ...trap/engines/engine.py.229: process: 0.86881 | load: 0.00378 [I 2022-10-11 14:39:36] ...trap/engines/engine.py.230: loss: 0.12281 [I 2022-10-11 14:39:36] ...optimizers/trijoint.py.151: total_norm: 19.299398 [I 2022-10-11 14:39:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 1360/2304 [I 2022-10-11 14:39:45] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:48 [I 2022-10-11 14:39:45] ...trap/engines/engine.py.229: process: 0.86927 | load: 0.00379 [I 2022-10-11 14:39:45] ...trap/engines/engine.py.230: loss: 0.10948 [I 2022-10-11 14:39:45] ...optimizers/trijoint.py.151: total_norm: 9.826240 [I 2022-10-11 14:39:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 1370/2304 [I 2022-10-11 14:39:54] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:13:35 [I 2022-10-11 14:39:54] ...trap/engines/engine.py.229: process: 0.87472 | load: 0.00380 [I 2022-10-11 14:39:54] ...trap/engines/engine.py.230: loss: 0.14662 [I 2022-10-11 14:39:54] ...optimizers/trijoint.py.151: total_norm: 12.787418 [I 2022-10-11 14:40:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1380/2304 [I 2022-10-11 14:40:03] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:13:27 [I 2022-10-11 14:40:03] ...trap/engines/engine.py.229: process: 0.86704 | load: 0.00380 [I 2022-10-11 14:40:03] ...trap/engines/engine.py.230: loss: 0.06358 [I 2022-10-11 14:40:03] ...optimizers/trijoint.py.151: total_norm: 14.392642 [I 2022-10-11 14:40:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1390/2304 [I 2022-10-11 14:40:12] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:13:16 [I 2022-10-11 14:40:12] ...trap/engines/engine.py.229: process: 0.87436 | load: 0.00375 [I 2022-10-11 14:40:12] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-11 14:40:12] ...optimizers/trijoint.py.151: total_norm: 21.881350 [I 2022-10-11 14:40:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1400/2304 [I 2022-10-11 14:40:20] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:13:12 [I 2022-10-11 14:40:20] ...trap/engines/engine.py.229: process: 0.87860 | load: 0.00381 [I 2022-10-11 14:40:20] ...trap/engines/engine.py.230: loss: 0.09581 [I 2022-10-11 14:40:20] ...optimizers/trijoint.py.151: total_norm: 8.483785 [I 2022-10-11 14:40:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1410/2304 [I 2022-10-11 14:40:29] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:12:44 [I 2022-10-11 14:40:29] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00301 [I 2022-10-11 14:40:29] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-11 14:40:29] ...optimizers/trijoint.py.151: total_norm: 11.931063 [I 2022-10-11 14:40:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1420/2304 [I 2022-10-11 14:40:37] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:26 [I 2022-10-11 14:40:37] ...trap/engines/engine.py.229: process: 0.84705 | load: 0.00299 [I 2022-10-11 14:40:37] ...trap/engines/engine.py.230: loss: 0.09773 [I 2022-10-11 14:40:37] ...optimizers/trijoint.py.151: total_norm: 9.605117 [I 2022-10-11 14:40:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1430/2304 [I 2022-10-11 14:40:46] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:12:24 [I 2022-10-11 14:40:46] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00299 [I 2022-10-11 14:40:46] ...trap/engines/engine.py.230: loss: 0.12964 [I 2022-10-11 14:40:46] ...optimizers/trijoint.py.151: total_norm: 15.551338 [I 2022-10-11 14:40:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 1440/2304 [I 2022-10-11 14:40:54] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:12:07 [I 2022-10-11 14:40:54] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00298 [I 2022-10-11 14:40:54] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-11 14:40:54] ...optimizers/trijoint.py.151: total_norm: 29.553675 [I 2022-10-11 14:41:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1450/2304 [I 2022-10-11 14:41:03] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:12:07 [I 2022-10-11 14:41:03] ...trap/engines/engine.py.229: process: 0.85305 | load: 0.00314 [I 2022-10-11 14:41:03] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-11 14:41:03] ...optimizers/trijoint.py.151: total_norm: 15.103081 [I 2022-10-11 14:41:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1460/2304 [I 2022-10-11 14:41:12] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:12:02 [I 2022-10-11 14:41:12] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00313 [I 2022-10-11 14:41:12] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-11 14:41:12] ...optimizers/trijoint.py.151: total_norm: 10.584457 [I 2022-10-11 14:41:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1470/2304 [I 2022-10-11 14:41:20] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:55 [I 2022-10-11 14:41:20] ...trap/engines/engine.py.229: process: 0.85540 | load: 0.00313 [I 2022-10-11 14:41:20] ...trap/engines/engine.py.230: loss: 0.08621 [I 2022-10-11 14:41:20] ...optimizers/trijoint.py.151: total_norm: 18.518129 [I 2022-10-11 14:41:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1480/2304 [I 2022-10-11 14:41:29] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:11:44 [I 2022-10-11 14:41:29] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00317 [I 2022-10-11 14:41:29] ...trap/engines/engine.py.230: loss: 0.11923 [I 2022-10-11 14:41:29] ...optimizers/trijoint.py.151: total_norm: 31.033703 [I 2022-10-11 14:41:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1490/2304 [I 2022-10-11 14:41:37] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:39 [I 2022-10-11 14:41:37] ...trap/engines/engine.py.229: process: 0.87144 | load: 0.00312 [I 2022-10-11 14:41:37] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-11 14:41:37] ...optimizers/trijoint.py.151: total_norm: 12.609766 [I 2022-10-11 14:41:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1500/2304 [I 2022-10-11 14:41:46] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:11:27 [I 2022-10-11 14:41:46] ...trap/engines/engine.py.229: process: 0.85146 | load: 0.00314 [I 2022-10-11 14:41:46] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-11 14:41:46] ...optimizers/trijoint.py.151: total_norm: 17.617706 [I 2022-10-11 14:41:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 1510/2304 [I 2022-10-11 14:41:54] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:11:19 [I 2022-10-11 14:41:54] ...trap/engines/engine.py.229: process: 0.84714 | load: 0.00315 [I 2022-10-11 14:41:54] ...trap/engines/engine.py.230: loss: 0.10644 [I 2022-10-11 14:41:54] ...optimizers/trijoint.py.151: total_norm: 20.961395 [I 2022-10-11 14:42:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1520/2304 [I 2022-10-11 14:42:03] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:11:09 [I 2022-10-11 14:42:03] ...trap/engines/engine.py.229: process: 0.83190 | load: 0.00314 [I 2022-10-11 14:42:03] ...trap/engines/engine.py.230: loss: 0.09330 [I 2022-10-11 14:42:03] ...optimizers/trijoint.py.151: total_norm: 22.458612 [I 2022-10-11 14:42:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1530/2304 [I 2022-10-11 14:42:12] ...trap/engines/engine.py.228: elapsed: 0:22:16 | left: 0:11:03 [I 2022-10-11 14:42:12] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00317 [I 2022-10-11 14:42:12] ...trap/engines/engine.py.230: loss: 0.20326 [I 2022-10-11 14:42:12] ...optimizers/trijoint.py.151: total_norm: 17.775484 [I 2022-10-11 14:42:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1540/2304 [I 2022-10-11 14:42:20] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:54 [I 2022-10-11 14:42:20] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00317 [I 2022-10-11 14:42:20] ...trap/engines/engine.py.230: loss: 0.10636 [I 2022-10-11 14:42:20] ...optimizers/trijoint.py.151: total_norm: 10.438137 [I 2022-10-11 14:42:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1550/2304 [I 2022-10-11 14:42:29] ...trap/engines/engine.py.228: elapsed: 0:22:33 | left: 0:10:45 [I 2022-10-11 14:42:29] ...trap/engines/engine.py.229: process: 0.84904 | load: 0.00314 [I 2022-10-11 14:42:29] ...trap/engines/engine.py.230: loss: 0.16821 [I 2022-10-11 14:42:29] ...optimizers/trijoint.py.151: total_norm: 11.876548 [I 2022-10-11 14:42:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1560/2304 [I 2022-10-11 14:42:37] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:44 [I 2022-10-11 14:42:37] ...trap/engines/engine.py.229: process: 0.89957 | load: 0.00313 [I 2022-10-11 14:42:37] ...trap/engines/engine.py.230: loss: 0.09500 [I 2022-10-11 14:42:37] ...optimizers/trijoint.py.151: total_norm: 20.296328 [I 2022-10-11 14:42:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1570/2304 [I 2022-10-11 14:42:46] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:10:29 [I 2022-10-11 14:42:46] ...trap/engines/engine.py.229: process: 0.85222 | load: 0.00315 [I 2022-10-11 14:42:46] ...trap/engines/engine.py.230: loss: 0.09937 [I 2022-10-11 14:42:46] ...optimizers/trijoint.py.151: total_norm: 15.551794 [I 2022-10-11 14:42:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 1580/2304 [I 2022-10-11 14:42:55] ...trap/engines/engine.py.228: elapsed: 0:22:59 | left: 0:10:16 [I 2022-10-11 14:42:55] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00324 [I 2022-10-11 14:42:55] ...trap/engines/engine.py.230: loss: 0.13486 [I 2022-10-11 14:42:55] ...optimizers/trijoint.py.151: total_norm: 23.364573 [I 2022-10-11 14:43:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1590/2304 [I 2022-10-11 14:43:03] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:10:11 [I 2022-10-11 14:43:03] ...trap/engines/engine.py.229: process: 0.85340 | load: 0.00314 [I 2022-10-11 14:43:03] ...trap/engines/engine.py.230: loss: 0.13481 [I 2022-10-11 14:43:03] ...optimizers/trijoint.py.151: total_norm: 13.604082 [I 2022-10-11 14:43:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1600/2304 [I 2022-10-11 14:43:12] ...trap/engines/engine.py.228: elapsed: 0:23:16 | left: 0:09:59 [I 2022-10-11 14:43:12] ...trap/engines/engine.py.229: process: 0.84684 | load: 0.00312 [I 2022-10-11 14:43:12] ...trap/engines/engine.py.230: loss: 0.11634 [I 2022-10-11 14:43:12] ...optimizers/trijoint.py.151: total_norm: 23.805368 [I 2022-10-11 14:43:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1610/2304 [I 2022-10-11 14:43:20] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:52 [I 2022-10-11 14:43:20] ...trap/engines/engine.py.229: process: 0.85741 | load: 0.00313 [I 2022-10-11 14:43:20] ...trap/engines/engine.py.230: loss: 0.12163 [I 2022-10-11 14:43:20] ...optimizers/trijoint.py.151: total_norm: 10.377737 [I 2022-10-11 14:43:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1620/2304 [I 2022-10-11 14:43:29] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:09:45 [I 2022-10-11 14:43:29] ...trap/engines/engine.py.229: process: 0.85861 | load: 0.00314 [I 2022-10-11 14:43:29] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-11 14:43:29] ...optimizers/trijoint.py.151: total_norm: 10.536700 [I 2022-10-11 14:43:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1630/2304 [I 2022-10-11 14:43:37] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:37 [I 2022-10-11 14:43:37] ...trap/engines/engine.py.229: process: 0.86094 | load: 0.00312 [I 2022-10-11 14:43:37] ...trap/engines/engine.py.230: loss: 0.09834 [I 2022-10-11 14:43:37] ...optimizers/trijoint.py.151: total_norm: 8.139073 [I 2022-10-11 14:43:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1640/2304 [I 2022-10-11 14:43:46] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:29 [I 2022-10-11 14:43:46] ...trap/engines/engine.py.229: process: 0.85763 | load: 0.00315 [I 2022-10-11 14:43:46] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-11 14:43:46] ...optimizers/trijoint.py.151: total_norm: 29.049595 [I 2022-10-11 14:43:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 1650/2304 [I 2022-10-11 14:43:55] ...trap/engines/engine.py.228: elapsed: 0:23:59 | left: 0:09:21 [I 2022-10-11 14:43:55] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00318 [I 2022-10-11 14:43:55] ...trap/engines/engine.py.230: loss: 0.15170 [I 2022-10-11 14:43:55] ...optimizers/trijoint.py.151: total_norm: 31.212553 [I 2022-10-11 14:44:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1660/2304 [I 2022-10-11 14:44:03] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:15 [I 2022-10-11 14:44:03] ...trap/engines/engine.py.229: process: 0.85806 | load: 0.00316 [I 2022-10-11 14:44:03] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-11 14:44:03] ...optimizers/trijoint.py.151: total_norm: 15.943172 [I 2022-10-11 14:44:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1670/2304 [I 2022-10-11 14:44:12] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:09:06 [I 2022-10-11 14:44:12] ...trap/engines/engine.py.229: process: 0.87344 | load: 0.00315 [I 2022-10-11 14:44:12] ...trap/engines/engine.py.230: loss: 0.13588 [I 2022-10-11 14:44:12] ...optimizers/trijoint.py.151: total_norm: 18.304054 [I 2022-10-11 14:44:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 1680/2304 [I 2022-10-11 14:44:21] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:57 [I 2022-10-11 14:44:21] ...trap/engines/engine.py.229: process: 0.86788 | load: 0.00318 [I 2022-10-11 14:44:21] ...trap/engines/engine.py.230: loss: 0.11452 [I 2022-10-11 14:44:21] ...optimizers/trijoint.py.151: total_norm: 10.956059 [I 2022-10-11 14:44:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1690/2304 [I 2022-10-11 14:44:29] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:47 [I 2022-10-11 14:44:29] ...trap/engines/engine.py.229: process: 0.85635 | load: 0.00314 [I 2022-10-11 14:44:29] ...trap/engines/engine.py.230: loss: 0.18529 [I 2022-10-11 14:44:29] ...optimizers/trijoint.py.151: total_norm: 26.844482 [I 2022-10-11 14:44:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 1700/2304 [I 2022-10-11 14:44:38] ...trap/engines/engine.py.228: elapsed: 0:24:42 | left: 0:08:37 [I 2022-10-11 14:44:38] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00309 [I 2022-10-11 14:44:38] ...trap/engines/engine.py.230: loss: 0.12756 [I 2022-10-11 14:44:38] ...optimizers/trijoint.py.151: total_norm: 17.778816 [I 2022-10-11 14:44:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1710/2304 [I 2022-10-11 14:44:46] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:30 [I 2022-10-11 14:44:46] ...trap/engines/engine.py.229: process: 0.86696 | load: 0.00314 [I 2022-10-11 14:44:46] ...trap/engines/engine.py.230: loss: 0.14081 [I 2022-10-11 14:44:46] ...optimizers/trijoint.py.151: total_norm: 33.622616 [I 2022-10-11 14:44:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 1720/2304 [I 2022-10-11 14:44:55] ...trap/engines/engine.py.228: elapsed: 0:24:59 | left: 0:08:22 [I 2022-10-11 14:44:55] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00314 [I 2022-10-11 14:44:55] ...trap/engines/engine.py.230: loss: 0.10215 [I 2022-10-11 14:44:55] ...optimizers/trijoint.py.151: total_norm: 46.817871 [I 2022-10-11 14:45:04] ...trap/engines/engine.py.224: train: epoch 108 | batch 1730/2304 [I 2022-10-11 14:45:04] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:08:11 [I 2022-10-11 14:45:04] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00315 [I 2022-10-11 14:45:04] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-11 14:45:04] ...optimizers/trijoint.py.151: total_norm: 21.800262 [I 2022-10-11 14:45:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1740/2304 [I 2022-10-11 14:45:12] ...trap/engines/engine.py.228: elapsed: 0:25:16 | left: 0:08:01 [I 2022-10-11 14:45:12] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00314 [I 2022-10-11 14:45:12] ...trap/engines/engine.py.230: loss: 0.13026 [I 2022-10-11 14:45:12] ...optimizers/trijoint.py.151: total_norm: 10.311843 [I 2022-10-11 14:45:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 1750/2304 [I 2022-10-11 14:45:21] ...trap/engines/engine.py.228: elapsed: 0:25:25 | left: 0:07:57 [I 2022-10-11 14:45:21] ...trap/engines/engine.py.229: process: 0.86346 | load: 0.00320 [I 2022-10-11 14:45:21] ...trap/engines/engine.py.230: loss: 0.19586 [I 2022-10-11 14:45:21] ...optimizers/trijoint.py.151: total_norm: 15.120059 [I 2022-10-11 14:45:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1760/2304 [I 2022-10-11 14:45:29] ...trap/engines/engine.py.228: elapsed: 0:25:33 | left: 0:07:46 [I 2022-10-11 14:45:29] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00313 [I 2022-10-11 14:45:29] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-11 14:45:29] ...optimizers/trijoint.py.151: total_norm: 9.310503 [I 2022-10-11 14:45:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 1770/2304 [I 2022-10-11 14:45:38] ...trap/engines/engine.py.228: elapsed: 0:25:42 | left: 0:07:38 [I 2022-10-11 14:45:38] ...trap/engines/engine.py.229: process: 0.87431 | load: 0.00315 [I 2022-10-11 14:45:38] ...trap/engines/engine.py.230: loss: 0.20067 [I 2022-10-11 14:45:38] ...optimizers/trijoint.py.151: total_norm: 19.313711 [I 2022-10-11 14:45:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 1780/2304 [I 2022-10-11 14:45:47] ...trap/engines/engine.py.228: elapsed: 0:25:50 | left: 0:07:30 [I 2022-10-11 14:45:47] ...trap/engines/engine.py.229: process: 0.85246 | load: 0.00312 [I 2022-10-11 14:45:47] ...trap/engines/engine.py.230: loss: 0.08483 [I 2022-10-11 14:45:47] ...optimizers/trijoint.py.151: total_norm: 13.986905 [I 2022-10-11 14:45:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 1790/2304 [I 2022-10-11 14:45:55] ...trap/engines/engine.py.228: elapsed: 0:25:59 | left: 0:07:22 [I 2022-10-11 14:45:55] ...trap/engines/engine.py.229: process: 0.87073 | load: 0.00321 [I 2022-10-11 14:45:55] ...trap/engines/engine.py.230: loss: 0.13053 [I 2022-10-11 14:45:55] ...optimizers/trijoint.py.151: total_norm: 23.266233 [I 2022-10-11 14:46:04] ...trap/engines/engine.py.224: train: epoch 108 | batch 1800/2304 [I 2022-10-11 14:46:04] ...trap/engines/engine.py.228: elapsed: 0:26:08 | left: 0:07:09 [I 2022-10-11 14:46:04] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00314 [I 2022-10-11 14:46:04] ...trap/engines/engine.py.230: loss: 0.11786 [I 2022-10-11 14:46:04] ...optimizers/trijoint.py.151: total_norm: 14.728441 [I 2022-10-11 14:46:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1810/2304 [I 2022-10-11 14:46:12] ...trap/engines/engine.py.228: elapsed: 0:26:16 | left: 0:07:02 [I 2022-10-11 14:46:12] ...trap/engines/engine.py.229: process: 0.84111 | load: 0.00316 [I 2022-10-11 14:46:12] ...trap/engines/engine.py.230: loss: 0.10765 [I 2022-10-11 14:46:12] ...optimizers/trijoint.py.151: total_norm: 17.209995 [I 2022-10-11 14:46:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 1820/2304 [I 2022-10-11 14:46:21] ...trap/engines/engine.py.228: elapsed: 0:26:25 | left: 0:06:55 [I 2022-10-11 14:46:21] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00313 [I 2022-10-11 14:46:21] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-11 14:46:21] ...optimizers/trijoint.py.151: total_norm: 18.069881 [I 2022-10-11 14:46:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1830/2304 [I 2022-10-11 14:46:29] ...trap/engines/engine.py.228: elapsed: 0:26:33 | left: 0:06:44 [I 2022-10-11 14:46:29] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00314 [I 2022-10-11 14:46:29] ...trap/engines/engine.py.230: loss: 0.08669 [I 2022-10-11 14:46:29] ...optimizers/trijoint.py.151: total_norm: 9.278934 [I 2022-10-11 14:46:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 1840/2304 [I 2022-10-11 14:46:38] ...trap/engines/engine.py.228: elapsed: 0:26:42 | left: 0:06:40 [I 2022-10-11 14:46:38] ...trap/engines/engine.py.229: process: 0.85549 | load: 0.00316 [I 2022-10-11 14:46:38] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-11 14:46:38] ...optimizers/trijoint.py.151: total_norm: 80.642143 [I 2022-10-11 14:46:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 1850/2304 [I 2022-10-11 14:46:47] ...trap/engines/engine.py.228: elapsed: 0:26:51 | left: 0:06:30 [I 2022-10-11 14:46:47] ...trap/engines/engine.py.229: process: 0.86159 | load: 0.00312 [I 2022-10-11 14:46:47] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-11 14:46:47] ...optimizers/trijoint.py.151: total_norm: 41.072960 [I 2022-10-11 14:46:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 1860/2304 [I 2022-10-11 14:46:55] ...trap/engines/engine.py.228: elapsed: 0:26:59 | left: 0:06:20 [I 2022-10-11 14:46:55] ...trap/engines/engine.py.229: process: 0.84500 | load: 0.00312 [I 2022-10-11 14:46:55] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-11 14:46:55] ...optimizers/trijoint.py.151: total_norm: 11.420977 [I 2022-10-11 14:47:04] ...trap/engines/engine.py.224: train: epoch 108 | batch 1870/2304 [I 2022-10-11 14:47:04] ...trap/engines/engine.py.228: elapsed: 0:27:08 | left: 0:06:13 [I 2022-10-11 14:47:04] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00313 [I 2022-10-11 14:47:04] ...trap/engines/engine.py.230: loss: 0.17374 [I 2022-10-11 14:47:04] ...optimizers/trijoint.py.151: total_norm: 14.022923 [I 2022-10-11 14:47:13] ...trap/engines/engine.py.224: train: epoch 108 | batch 1880/2304 [I 2022-10-11 14:47:13] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:06:10 [I 2022-10-11 14:47:13] ...trap/engines/engine.py.229: process: 0.89325 | load: 0.00395 [I 2022-10-11 14:47:13] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-11 14:47:13] ...optimizers/trijoint.py.151: total_norm: 54.281879 [I 2022-10-11 14:47:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 1890/2304 [I 2022-10-11 14:47:21] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:05:58 [I 2022-10-11 14:47:21] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00313 [I 2022-10-11 14:47:21] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-11 14:47:21] ...optimizers/trijoint.py.151: total_norm: 13.459566 [I 2022-10-11 14:47:30] ...trap/engines/engine.py.224: train: epoch 108 | batch 1900/2304 [I 2022-10-11 14:47:30] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:05:44 [I 2022-10-11 14:47:30] ...trap/engines/engine.py.229: process: 0.83462 | load: 0.00310 [I 2022-10-11 14:47:30] ...trap/engines/engine.py.230: loss: 0.09280 [I 2022-10-11 14:47:30] ...optimizers/trijoint.py.151: total_norm: 14.948154 [I 2022-10-11 14:47:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 1910/2304 [I 2022-10-11 14:47:38] ...trap/engines/engine.py.228: elapsed: 0:27:42 | left: 0:05:36 [I 2022-10-11 14:47:38] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00314 [I 2022-10-11 14:47:38] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-11 14:47:38] ...optimizers/trijoint.py.151: total_norm: 19.233418 [I 2022-10-11 14:47:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 1920/2304 [I 2022-10-11 14:47:47] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:34 [I 2022-10-11 14:47:47] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00397 [I 2022-10-11 14:47:47] ...trap/engines/engine.py.230: loss: 0.09872 [I 2022-10-11 14:47:47] ...optimizers/trijoint.py.151: total_norm: 7.994688 [I 2022-10-11 14:47:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 1930/2304 [I 2022-10-11 14:47:56] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:31 [I 2022-10-11 14:47:56] ...trap/engines/engine.py.229: process: 0.89195 | load: 0.00393 [I 2022-10-11 14:47:56] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-11 14:47:56] ...optimizers/trijoint.py.151: total_norm: 18.402063 [I 2022-10-11 14:48:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 1940/2304 [I 2022-10-11 14:48:05] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:23 [I 2022-10-11 14:48:05] ...trap/engines/engine.py.229: process: 0.88291 | load: 0.00394 [I 2022-10-11 14:48:05] ...trap/engines/engine.py.230: loss: 0.10541 [I 2022-10-11 14:48:05] ...optimizers/trijoint.py.151: total_norm: 12.084984 [I 2022-10-11 14:48:13] ...trap/engines/engine.py.224: train: epoch 108 | batch 1950/2304 [I 2022-10-11 14:48:13] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:05:03 [I 2022-10-11 14:48:13] ...trap/engines/engine.py.229: process: 0.84236 | load: 0.00311 [I 2022-10-11 14:48:13] ...trap/engines/engine.py.230: loss: 0.09934 [I 2022-10-11 14:48:13] ...optimizers/trijoint.py.151: total_norm: 13.519668 [I 2022-10-11 14:48:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 1960/2304 [I 2022-10-11 14:48:22] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:54 [I 2022-10-11 14:48:22] ...trap/engines/engine.py.229: process: 0.85425 | load: 0.00312 [I 2022-10-11 14:48:22] ...trap/engines/engine.py.230: loss: 0.10449 [I 2022-10-11 14:48:22] ...optimizers/trijoint.py.151: total_norm: 15.373781 [I 2022-10-11 14:48:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 1970/2304 [I 2022-10-11 14:48:31] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:46 [I 2022-10-11 14:48:31] ...trap/engines/engine.py.229: process: 0.85475 | load: 0.00310 [I 2022-10-11 14:48:31] ...trap/engines/engine.py.230: loss: 0.12059 [I 2022-10-11 14:48:31] ...optimizers/trijoint.py.151: total_norm: 15.370200 [I 2022-10-11 14:48:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 1980/2304 [I 2022-10-11 14:48:39] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:38 [I 2022-10-11 14:48:39] ...trap/engines/engine.py.229: process: 0.85730 | load: 0.00314 [I 2022-10-11 14:48:39] ...trap/engines/engine.py.230: loss: 0.10255 [I 2022-10-11 14:48:39] ...optimizers/trijoint.py.151: total_norm: 12.549965 [I 2022-10-11 14:48:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 1990/2304 [I 2022-10-11 14:48:48] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:31 [I 2022-10-11 14:48:48] ...trap/engines/engine.py.229: process: 0.86595 | load: 0.00312 [I 2022-10-11 14:48:48] ...trap/engines/engine.py.230: loss: 0.16774 [I 2022-10-11 14:48:48] ...optimizers/trijoint.py.151: total_norm: 25.400671 [I 2022-10-11 14:48:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 2000/2304 [I 2022-10-11 14:48:56] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:19 [I 2022-10-11 14:48:56] ...trap/engines/engine.py.229: process: 0.85959 | load: 0.00311 [I 2022-10-11 14:48:56] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-11 14:48:56] ...optimizers/trijoint.py.151: total_norm: 16.216957 [I 2022-10-11 14:49:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 2010/2304 [I 2022-10-11 14:49:05] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:12 [I 2022-10-11 14:49:05] ...trap/engines/engine.py.229: process: 0.86687 | load: 0.00314 [I 2022-10-11 14:49:05] ...trap/engines/engine.py.230: loss: 0.12199 [I 2022-10-11 14:49:05] ...optimizers/trijoint.py.151: total_norm: 25.546633 [I 2022-10-11 14:49:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2020/2304 [I 2022-10-11 14:49:14] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:03 [I 2022-10-11 14:49:14] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00315 [I 2022-10-11 14:49:14] ...trap/engines/engine.py.230: loss: 0.13472 [I 2022-10-11 14:49:14] ...optimizers/trijoint.py.151: total_norm: 19.611683 [I 2022-10-11 14:49:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 2030/2304 [I 2022-10-11 14:49:22] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:55 [I 2022-10-11 14:49:22] ...trap/engines/engine.py.229: process: 0.86295 | load: 0.00312 [I 2022-10-11 14:49:22] ...trap/engines/engine.py.230: loss: 0.15434 [I 2022-10-11 14:49:22] ...optimizers/trijoint.py.151: total_norm: 12.753733 [I 2022-10-11 14:49:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2040/2304 [I 2022-10-11 14:49:31] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:44 [I 2022-10-11 14:49:31] ...trap/engines/engine.py.229: process: 0.85342 | load: 0.00313 [I 2022-10-11 14:49:31] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-11 14:49:31] ...optimizers/trijoint.py.151: total_norm: 10.629337 [I 2022-10-11 14:49:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 2050/2304 [I 2022-10-11 14:49:39] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:37 [I 2022-10-11 14:49:39] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00311 [I 2022-10-11 14:49:39] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-11 14:49:39] ...optimizers/trijoint.py.151: total_norm: 14.331187 [I 2022-10-11 14:49:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 2060/2304 [I 2022-10-11 14:49:48] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:30 [I 2022-10-11 14:49:48] ...trap/engines/engine.py.229: process: 0.87165 | load: 0.00316 [I 2022-10-11 14:49:48] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-11 14:49:48] ...optimizers/trijoint.py.151: total_norm: 25.075672 [I 2022-10-11 14:49:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 2070/2304 [I 2022-10-11 14:49:57] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:20 [I 2022-10-11 14:49:57] ...trap/engines/engine.py.229: process: 0.86668 | load: 0.00314 [I 2022-10-11 14:49:57] ...trap/engines/engine.py.230: loss: 0.15903 [I 2022-10-11 14:49:57] ...optimizers/trijoint.py.151: total_norm: 35.293156 [I 2022-10-11 14:50:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 2080/2304 [I 2022-10-11 14:50:05] ...trap/engines/engine.py.228: elapsed: 0:30:09 | left: 0:03:13 [I 2022-10-11 14:50:05] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00310 [I 2022-10-11 14:50:05] ...trap/engines/engine.py.230: loss: 0.11373 [I 2022-10-11 14:50:05] ...optimizers/trijoint.py.151: total_norm: 8.440205 [I 2022-10-11 14:50:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2090/2304 [I 2022-10-11 14:50:14] ...trap/engines/engine.py.228: elapsed: 0:30:18 | left: 0:03:04 [I 2022-10-11 14:50:14] ...trap/engines/engine.py.229: process: 0.86299 | load: 0.00319 [I 2022-10-11 14:50:14] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-11 14:50:14] ...optimizers/trijoint.py.151: total_norm: 29.605156 [I 2022-10-11 14:50:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 2100/2304 [I 2022-10-11 14:50:22] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:56 [I 2022-10-11 14:50:22] ...trap/engines/engine.py.229: process: 0.85388 | load: 0.00317 [I 2022-10-11 14:50:22] ...trap/engines/engine.py.230: loss: 0.10430 [I 2022-10-11 14:50:22] ...optimizers/trijoint.py.151: total_norm: 20.476662 [I 2022-10-11 14:50:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2110/2304 [I 2022-10-11 14:50:31] ...trap/engines/engine.py.228: elapsed: 0:30:35 | left: 0:02:46 [I 2022-10-11 14:50:31] ...trap/engines/engine.py.229: process: 0.84596 | load: 0.00315 [I 2022-10-11 14:50:31] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-11 14:50:31] ...optimizers/trijoint.py.151: total_norm: 16.563946 [I 2022-10-11 14:50:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 2120/2304 [I 2022-10-11 14:50:40] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:39 [I 2022-10-11 14:50:40] ...trap/engines/engine.py.229: process: 0.86941 | load: 0.00315 [I 2022-10-11 14:50:40] ...trap/engines/engine.py.230: loss: 0.10050 [I 2022-10-11 14:50:40] ...optimizers/trijoint.py.151: total_norm: 18.215454 [I 2022-10-11 14:50:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 2130/2304 [I 2022-10-11 14:50:48] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:30 [I 2022-10-11 14:50:48] ...trap/engines/engine.py.229: process: 0.85488 | load: 0.00315 [I 2022-10-11 14:50:48] ...trap/engines/engine.py.230: loss: 0.10514 [I 2022-10-11 14:50:48] ...optimizers/trijoint.py.151: total_norm: 14.230950 [I 2022-10-11 14:50:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 2140/2304 [I 2022-10-11 14:50:57] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:22 [I 2022-10-11 14:50:57] ...trap/engines/engine.py.229: process: 0.89045 | load: 0.00317 [I 2022-10-11 14:50:57] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-11 14:50:57] ...optimizers/trijoint.py.151: total_norm: 20.282513 [I 2022-10-11 14:51:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 2150/2304 [I 2022-10-11 14:51:06] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:14 [I 2022-10-11 14:51:06] ...trap/engines/engine.py.229: process: 0.87950 | load: 0.00315 [I 2022-10-11 14:51:06] ...trap/engines/engine.py.230: loss: 0.15798 [I 2022-10-11 14:51:06] ...optimizers/trijoint.py.151: total_norm: 16.796434 [I 2022-10-11 14:51:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2160/2304 [I 2022-10-11 14:51:14] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:02:02 [I 2022-10-11 14:51:14] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00315 [I 2022-10-11 14:51:14] ...trap/engines/engine.py.230: loss: 0.17922 [I 2022-10-11 14:51:14] ...optimizers/trijoint.py.151: total_norm: 10.202255 [I 2022-10-11 14:51:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 2170/2304 [I 2022-10-11 14:51:23] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:54 [I 2022-10-11 14:51:23] ...trap/engines/engine.py.229: process: 0.84525 | load: 0.00313 [I 2022-10-11 14:51:23] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-11 14:51:23] ...optimizers/trijoint.py.151: total_norm: 25.830559 [I 2022-10-11 14:51:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2180/2304 [I 2022-10-11 14:51:31] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:46 [I 2022-10-11 14:51:31] ...trap/engines/engine.py.229: process: 0.86907 | load: 0.00313 [I 2022-10-11 14:51:31] ...trap/engines/engine.py.230: loss: 0.11712 [I 2022-10-11 14:51:31] ...optimizers/trijoint.py.151: total_norm: 68.867088 [I 2022-10-11 14:51:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 2190/2304 [I 2022-10-11 14:51:40] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:36 [I 2022-10-11 14:51:40] ...trap/engines/engine.py.229: process: 0.85621 | load: 0.00315 [I 2022-10-11 14:51:40] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-11 14:51:40] ...optimizers/trijoint.py.151: total_norm: 17.606857 [I 2022-10-11 14:51:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 2200/2304 [I 2022-10-11 14:51:48] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:28 [I 2022-10-11 14:51:48] ...trap/engines/engine.py.229: process: 0.86905 | load: 0.00315 [I 2022-10-11 14:51:48] ...trap/engines/engine.py.230: loss: 0.08770 [I 2022-10-11 14:51:48] ...optimizers/trijoint.py.151: total_norm: 11.128276 [I 2022-10-11 14:51:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 2210/2304 [I 2022-10-11 14:51:57] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:20 [I 2022-10-11 14:51:57] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00313 [I 2022-10-11 14:51:57] ...trap/engines/engine.py.230: loss: 0.11051 [I 2022-10-11 14:51:57] ...optimizers/trijoint.py.151: total_norm: 8.887880 [I 2022-10-11 14:52:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 2220/2304 [I 2022-10-11 14:52:06] ...trap/engines/engine.py.228: elapsed: 0:32:09 | left: 0:01:11 [I 2022-10-11 14:52:06] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00311 [I 2022-10-11 14:52:06] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-11 14:52:06] ...optimizers/trijoint.py.151: total_norm: 10.771688 [I 2022-10-11 14:52:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2230/2304 [I 2022-10-11 14:52:14] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:03 [I 2022-10-11 14:52:14] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00314 [I 2022-10-11 14:52:14] ...trap/engines/engine.py.230: loss: 0.19247 [I 2022-10-11 14:52:14] ...optimizers/trijoint.py.151: total_norm: 20.438726 [I 2022-10-11 14:52:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 2240/2304 [I 2022-10-11 14:52:23] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:54 [I 2022-10-11 14:52:23] ...trap/engines/engine.py.229: process: 0.83517 | load: 0.00311 [I 2022-10-11 14:52:23] ...trap/engines/engine.py.230: loss: 0.20128 [I 2022-10-11 14:52:23] ...optimizers/trijoint.py.151: total_norm: 11.160308 [I 2022-10-11 14:52:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2250/2304 [I 2022-10-11 14:52:31] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:45 [I 2022-10-11 14:52:31] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00311 [I 2022-10-11 14:52:31] ...trap/engines/engine.py.230: loss: 0.10978 [I 2022-10-11 14:52:31] ...optimizers/trijoint.py.151: total_norm: 16.725069 [I 2022-10-11 14:52:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 2260/2304 [I 2022-10-11 14:52:40] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:37 [I 2022-10-11 14:52:40] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00313 [I 2022-10-11 14:52:40] ...trap/engines/engine.py.230: loss: 0.10093 [I 2022-10-11 14:52:40] ...optimizers/trijoint.py.151: total_norm: 26.446020 [I 2022-10-11 14:53:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 2270/2304 [I 2022-10-11 14:53:05] ...trap/engines/engine.py.228: elapsed: 0:33:09 | left: 0:01:43 [I 2022-10-11 14:53:05] ...trap/engines/engine.py.229: process: 0.84686 | load: 0.00313 [I 2022-10-11 14:53:05] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-11 14:53:05] ...optimizers/trijoint.py.151: total_norm: 6.702460 [I 2022-10-11 14:53:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2280/2304 [I 2022-10-11 14:53:14] ...trap/engines/engine.py.228: elapsed: 0:33:18 | left: 0:00:25 [I 2022-10-11 14:53:14] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00313 [I 2022-10-11 14:53:14] ...trap/engines/engine.py.230: loss: 0.09704 [I 2022-10-11 14:53:14] ...optimizers/trijoint.py.151: total_norm: 15.767072 [I 2022-10-11 14:53:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 2290/2304 [I 2022-10-11 14:53:22] ...trap/engines/engine.py.228: elapsed: 0:33:26 | left: 0:00:12 [I 2022-10-11 14:53:22] ...trap/engines/engine.py.229: process: 0.84870 | load: 0.00315 [I 2022-10-11 14:53:22] ...trap/engines/engine.py.230: loss: 0.17637 [I 2022-10-11 14:53:22] ...optimizers/trijoint.py.151: total_norm: 19.693041 [I 2022-10-11 14:53:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2300/2304 [I 2022-10-11 14:53:31] ...trap/engines/engine.py.228: elapsed: 0:33:35 | left: 0:00:03 [I 2022-10-11 14:53:31] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00019 [I 2022-10-11 14:53:31] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-11 14:53:31] ...optimizers/trijoint.py.151: total_norm: 18.562618 [I 2022-10-11 14:53:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 2304/2304 [I 2022-10-11 14:53:34] ...trap/engines/engine.py.228: elapsed: 0:33:38 | left: 0:00:00 [I 2022-10-11 14:53:34] ...trap/engines/engine.py.229: process: 0.84508 | load: 0.00019 [I 2022-10-11 14:53:34] ...trap/engines/engine.py.230: loss: 0.10590 [I 2022-10-11 14:53:34] ...optimizers/trijoint.py.151: total_norm: 15.215591 [S 2022-10-11 14:53:36] ...trap/engines/engine.py.129: train_epoch.epoch: 108 [S 2022-10-11 14:53:36] ...trap/engines/engine.py.129: train_epoch.loss: 0.12180967338851889 [I 2022-10-11 14:53:36] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 14:53:37] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 108 [I 2022-10-11 14:53:37] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 14:53:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 14:53:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 14:53:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 14:53:44] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 14:53:44] ...trap/engines/engine.py.321: eval: epoch 108 | batch 0/510 [I 2022-10-11 14:53:44] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:55:20 [I 2022-10-11 14:53:44] ...trap/engines/engine.py.326: process: 6.51098 | load: 3.51487 [I 2022-10-11 14:53:50] ...trap/engines/engine.py.321: eval: epoch 108 | batch 10/510 [I 2022-10-11 14:53:50] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:10:24 [I 2022-10-11 14:53:50] ...trap/engines/engine.py.326: process: 0.67367 | load: 0.00021 [I 2022-10-11 14:53:56] ...trap/engines/engine.py.321: eval: epoch 108 | batch 20/510 [I 2022-10-11 14:53:56] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:05:18 [I 2022-10-11 14:53:56] ...trap/engines/engine.py.326: process: 0.64382 | load: 0.00023 [I 2022-10-11 14:54:04] ...trap/engines/engine.py.321: eval: epoch 108 | batch 30/510 [I 2022-10-11 14:54:04] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:06:12 [I 2022-10-11 14:54:04] ...trap/engines/engine.py.326: process: 0.72251 | load: 0.00021 [I 2022-10-11 14:54:10] ...trap/engines/engine.py.321: eval: epoch 108 | batch 40/510 [I 2022-10-11 14:54:10] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:05:27 [I 2022-10-11 14:54:10] ...trap/engines/engine.py.326: process: 0.55813 | load: 0.00026 [I 2022-10-11 14:54:16] ...trap/engines/engine.py.321: eval: epoch 108 | batch 50/510 [I 2022-10-11 14:54:16] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:24 [I 2022-10-11 14:54:16] ...trap/engines/engine.py.326: process: 0.53550 | load: 0.00021 [I 2022-10-11 14:54:21] ...trap/engines/engine.py.321: eval: epoch 108 | batch 60/510 [I 2022-10-11 14:54:21] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:55 [I 2022-10-11 14:54:21] ...trap/engines/engine.py.326: process: 0.50347 | load: 0.00022 [I 2022-10-11 14:54:26] ...trap/engines/engine.py.321: eval: epoch 108 | batch 70/510 [I 2022-10-11 14:54:26] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:24 [I 2022-10-11 14:54:26] ...trap/engines/engine.py.326: process: 0.46628 | load: 0.00022 [I 2022-10-11 14:54:31] ...trap/engines/engine.py.321: eval: epoch 108 | batch 80/510 [I 2022-10-11 14:54:31] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:42 [I 2022-10-11 14:54:31] ...trap/engines/engine.py.326: process: 0.44915 | load: 0.00021 [I 2022-10-11 14:54:36] ...trap/engines/engine.py.321: eval: epoch 108 | batch 90/510 [I 2022-10-11 14:54:36] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:21 [I 2022-10-11 14:54:36] ...trap/engines/engine.py.326: process: 0.52306 | load: 0.00020 [I 2022-10-11 14:54:41] ...trap/engines/engine.py.321: eval: epoch 108 | batch 100/510 [I 2022-10-11 14:54:41] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:08 [I 2022-10-11 14:54:41] ...trap/engines/engine.py.326: process: 0.30712 | load: 0.00020 [I 2022-10-11 14:54:44] ...trap/engines/engine.py.321: eval: epoch 108 | batch 110/510 [I 2022-10-11 14:54:44] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:13 [I 2022-10-11 14:54:44] ...trap/engines/engine.py.326: process: 0.32050 | load: 0.00021 [I 2022-10-11 14:54:47] ...trap/engines/engine.py.321: eval: epoch 108 | batch 120/510 [I 2022-10-11 14:54:47] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:04 [I 2022-10-11 14:54:47] ...trap/engines/engine.py.326: process: 0.31927 | load: 0.00021 [I 2022-10-11 14:54:51] ...trap/engines/engine.py.321: eval: epoch 108 | batch 130/510 [I 2022-10-11 14:54:51] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:02 [I 2022-10-11 14:54:51] ...trap/engines/engine.py.326: process: 0.32256 | load: 0.00023 [I 2022-10-11 14:54:54] ...trap/engines/engine.py.321: eval: epoch 108 | batch 140/510 [I 2022-10-11 14:54:54] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:58 [I 2022-10-11 14:54:54] ...trap/engines/engine.py.326: process: 0.32318 | load: 0.00020 [I 2022-10-11 14:54:57] ...trap/engines/engine.py.321: eval: epoch 108 | batch 150/510 [I 2022-10-11 14:54:57] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-11 14:54:57] ...trap/engines/engine.py.326: process: 0.31960 | load: 0.00020 [I 2022-10-11 14:55:00] ...trap/engines/engine.py.321: eval: epoch 108 | batch 160/510 [I 2022-10-11 14:55:00] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:52 [I 2022-10-11 14:55:00] ...trap/engines/engine.py.326: process: 0.32317 | load: 0.00022 [I 2022-10-11 14:55:03] ...trap/engines/engine.py.321: eval: epoch 108 | batch 170/510 [I 2022-10-11 14:55:03] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:49 [I 2022-10-11 14:55:03] ...trap/engines/engine.py.326: process: 0.32310 | load: 0.00021 [I 2022-10-11 14:55:07] ...trap/engines/engine.py.321: eval: epoch 108 | batch 180/510 [I 2022-10-11 14:55:07] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:46 [I 2022-10-11 14:55:07] ...trap/engines/engine.py.326: process: 0.32517 | load: 0.00021 [I 2022-10-11 14:55:10] ...trap/engines/engine.py.321: eval: epoch 108 | batch 190/510 [I 2022-10-11 14:55:10] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:42 [I 2022-10-11 14:55:10] ...trap/engines/engine.py.326: process: 0.32079 | load: 0.00021 [I 2022-10-11 14:55:13] ...trap/engines/engine.py.321: eval: epoch 108 | batch 200/510 [I 2022-10-11 14:55:13] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:39 [I 2022-10-11 14:55:13] ...trap/engines/engine.py.326: process: 0.31999 | load: 0.00024 [I 2022-10-11 14:55:16] ...trap/engines/engine.py.321: eval: epoch 108 | batch 210/510 [I 2022-10-11 14:55:16] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:36 [I 2022-10-11 14:55:16] ...trap/engines/engine.py.326: process: 0.32352 | load: 0.00021 [I 2022-10-11 14:55:19] ...trap/engines/engine.py.321: eval: epoch 108 | batch 220/510 [I 2022-10-11 14:55:19] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:33 [I 2022-10-11 14:55:19] ...trap/engines/engine.py.326: process: 0.32095 | load: 0.00021 [I 2022-10-11 14:55:23] ...trap/engines/engine.py.321: eval: epoch 108 | batch 230/510 [I 2022-10-11 14:55:23] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:29 [I 2022-10-11 14:55:23] ...trap/engines/engine.py.326: process: 0.31274 | load: 0.00022 [I 2022-10-11 14:55:26] ...trap/engines/engine.py.321: eval: epoch 108 | batch 240/510 [I 2022-10-11 14:55:26] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:27 [I 2022-10-11 14:55:26] ...trap/engines/engine.py.326: process: 0.32767 | load: 0.00021 [I 2022-10-11 14:55:29] ...trap/engines/engine.py.321: eval: epoch 108 | batch 250/510 [I 2022-10-11 14:55:29] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:23 [I 2022-10-11 14:55:29] ...trap/engines/engine.py.326: process: 0.32386 | load: 0.00023 [I 2022-10-11 14:55:32] ...trap/engines/engine.py.321: eval: epoch 108 | batch 260/510 [I 2022-10-11 14:55:32] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:19 [I 2022-10-11 14:55:32] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00024 [I 2022-10-11 14:55:36] ...trap/engines/engine.py.321: eval: epoch 108 | batch 270/510 [I 2022-10-11 14:55:36] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:16 [I 2022-10-11 14:55:36] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00021 [I 2022-10-11 14:55:39] ...trap/engines/engine.py.321: eval: epoch 108 | batch 280/510 [I 2022-10-11 14:55:39] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:13 [I 2022-10-11 14:55:39] ...trap/engines/engine.py.326: process: 0.31632 | load: 0.00021 [I 2022-10-11 14:55:42] ...trap/engines/engine.py.321: eval: epoch 108 | batch 290/510 [I 2022-10-11 14:55:42] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:10 [I 2022-10-11 14:55:42] ...trap/engines/engine.py.326: process: 0.32083 | load: 0.00021 [I 2022-10-11 14:55:45] ...trap/engines/engine.py.321: eval: epoch 108 | batch 300/510 [I 2022-10-11 14:55:45] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:07 [I 2022-10-11 14:55:45] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00021 [I 2022-10-11 14:55:48] ...trap/engines/engine.py.321: eval: epoch 108 | batch 310/510 [I 2022-10-11 14:55:48] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:01:04 [I 2022-10-11 14:55:48] ...trap/engines/engine.py.326: process: 0.31940 | load: 0.00022 [I 2022-10-11 14:55:52] ...trap/engines/engine.py.321: eval: epoch 108 | batch 320/510 [I 2022-10-11 14:55:52] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:01 [I 2022-10-11 14:55:52] ...trap/engines/engine.py.326: process: 0.31760 | load: 0.00021 [I 2022-10-11 14:55:55] ...trap/engines/engine.py.321: eval: epoch 108 | batch 330/510 [I 2022-10-11 14:55:55] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:58 [I 2022-10-11 14:55:55] ...trap/engines/engine.py.326: process: 0.32490 | load: 0.00021 [I 2022-10-11 14:55:58] ...trap/engines/engine.py.321: eval: epoch 108 | batch 340/510 [I 2022-10-11 14:55:58] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:54 [I 2022-10-11 14:55:58] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00021 [I 2022-10-11 14:56:01] ...trap/engines/engine.py.321: eval: epoch 108 | batch 350/510 [I 2022-10-11 14:56:01] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:51 [I 2022-10-11 14:56:01] ...trap/engines/engine.py.326: process: 0.31622 | load: 0.00021 [I 2022-10-11 14:56:04] ...trap/engines/engine.py.321: eval: epoch 108 | batch 360/510 [I 2022-10-11 14:56:04] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:48 [I 2022-10-11 14:56:04] ...trap/engines/engine.py.326: process: 0.32248 | load: 0.00021 [I 2022-10-11 14:56:08] ...trap/engines/engine.py.321: eval: epoch 108 | batch 370/510 [I 2022-10-11 14:56:08] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:44 [I 2022-10-11 14:56:08] ...trap/engines/engine.py.326: process: 0.31729 | load: 0.00020 [I 2022-10-11 14:56:11] ...trap/engines/engine.py.321: eval: epoch 108 | batch 380/510 [I 2022-10-11 14:56:11] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:41 [I 2022-10-11 14:56:11] ...trap/engines/engine.py.326: process: 0.31721 | load: 0.00021 [I 2022-10-11 14:56:14] ...trap/engines/engine.py.321: eval: epoch 108 | batch 390/510 [I 2022-10-11 14:56:14] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:38 [I 2022-10-11 14:56:14] ...trap/engines/engine.py.326: process: 0.32086 | load: 0.00020 [I 2022-10-11 14:56:17] ...trap/engines/engine.py.321: eval: epoch 108 | batch 400/510 [I 2022-10-11 14:56:17] ...trap/engines/engine.py.325: elapsed: 0:02:39 | left: 0:00:34 [I 2022-10-11 14:56:17] ...trap/engines/engine.py.326: process: 0.31793 | load: 0.00021 [I 2022-10-11 14:56:20] ...trap/engines/engine.py.321: eval: epoch 108 | batch 410/510 [I 2022-10-11 14:56:20] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:31 [I 2022-10-11 14:56:20] ...trap/engines/engine.py.326: process: 0.31242 | load: 0.00020 [I 2022-10-11 14:56:24] ...trap/engines/engine.py.321: eval: epoch 108 | batch 420/510 [I 2022-10-11 14:56:24] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:28 [I 2022-10-11 14:56:24] ...trap/engines/engine.py.326: process: 0.31500 | load: 0.00020 [I 2022-10-11 14:56:27] ...trap/engines/engine.py.321: eval: epoch 108 | batch 430/510 [I 2022-10-11 14:56:27] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:25 [I 2022-10-11 14:56:27] ...trap/engines/engine.py.326: process: 0.31294 | load: 0.00020 [I 2022-10-11 14:56:30] ...trap/engines/engine.py.321: eval: epoch 108 | batch 440/510 [I 2022-10-11 14:56:30] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:22 [I 2022-10-11 14:56:30] ...trap/engines/engine.py.326: process: 0.31976 | load: 0.00020 [I 2022-10-11 14:56:33] ...trap/engines/engine.py.321: eval: epoch 108 | batch 450/510 [I 2022-10-11 14:56:33] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:18 [I 2022-10-11 14:56:33] ...trap/engines/engine.py.326: process: 0.31479 | load: 0.00020 [I 2022-10-11 14:56:36] ...trap/engines/engine.py.321: eval: epoch 108 | batch 460/510 [I 2022-10-11 14:56:36] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:15 [I 2022-10-11 14:56:36] ...trap/engines/engine.py.326: process: 0.31246 | load: 0.00020 [I 2022-10-11 14:56:39] ...trap/engines/engine.py.321: eval: epoch 108 | batch 470/510 [I 2022-10-11 14:56:39] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-11 14:56:39] ...trap/engines/engine.py.326: process: 0.31599 | load: 0.00020 [I 2022-10-11 14:56:43] ...trap/engines/engine.py.321: eval: epoch 108 | batch 480/510 [I 2022-10-11 14:56:43] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:09 [I 2022-10-11 14:56:43] ...trap/engines/engine.py.326: process: 0.31924 | load: 0.00020 [I 2022-10-11 14:56:46] ...trap/engines/engine.py.321: eval: epoch 108 | batch 490/510 [I 2022-10-11 14:56:46] ...trap/engines/engine.py.325: elapsed: 0:03:08 | left: 0:00:06 [I 2022-10-11 14:56:46] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00021 [I 2022-10-11 14:56:49] ...trap/engines/engine.py.321: eval: epoch 108 | batch 500/510 [I 2022-10-11 14:56:49] ...trap/engines/engine.py.325: elapsed: 0:03:11 | left: 0:00:03 [I 2022-10-11 14:56:49] ...trap/engines/engine.py.326: process: 0.32090 | load: 0.00023 [I 2022-10-11 14:56:52] ...trap/engines/engine.py.321: eval: epoch 108 | batch 510/510 [I 2022-10-11 14:56:52] ...trap/engines/engine.py.325: elapsed: 0:03:14 | left: 0:00:00 [I 2022-10-11 14:56:52] ...trap/engines/engine.py.326: process: 0.31630 | load: 0.00018 [S 2022-10-11 14:56:53] ...trap/engines/engine.py.132: eval_epoch.epoch: 108 [I 2022-10-11 14:56:53] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.745000 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911900 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935500 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.742200 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910600 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936200 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014401 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005147 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006771 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009464 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007940 [S 2022-10-11 14:57:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006570 [I 2022-10-11 14:57:35] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 14:57:35] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 14:57:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 14:57:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 14:57:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 14:57:40] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 14:57:43] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 14:57:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 14:57:47] ...trap/engines/engine.py.163: Training model on trainset for epoch 109 [I 2022-10-11 14:57:47] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 14:57:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 0/2304 [I 2022-10-11 14:57:50] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:03:20 [I 2022-10-11 14:57:50] ...trap/engines/engine.py.229: process: 3.21188 | load: 2.27296 [I 2022-10-11 14:57:50] ...trap/engines/engine.py.230: loss: 0.19154 [I 2022-10-11 14:57:50] ...optimizers/trijoint.py.151: total_norm: 29.544071 [I 2022-10-11 14:57:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 10/2304 [I 2022-10-11 14:57:58] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:45 [I 2022-10-11 14:57:58] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00505 [I 2022-10-11 14:57:58] ...trap/engines/engine.py.230: loss: 0.09412 [I 2022-10-11 14:57:58] ...optimizers/trijoint.py.151: total_norm: 14.409397 [I 2022-10-11 14:58:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 20/2304 [I 2022-10-11 14:58:07] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:32:55 [I 2022-10-11 14:58:07] ...trap/engines/engine.py.229: process: 0.83909 | load: 0.00486 [I 2022-10-11 14:58:07] ...trap/engines/engine.py.230: loss: 0.11211 [I 2022-10-11 14:58:07] ...optimizers/trijoint.py.151: total_norm: 9.279378 [I 2022-10-11 14:58:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 30/2304 [I 2022-10-11 14:58:15] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:18 [I 2022-10-11 14:58:15] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00360 [I 2022-10-11 14:58:15] ...trap/engines/engine.py.230: loss: 0.10745 [I 2022-10-11 14:58:15] ...optimizers/trijoint.py.151: total_norm: 22.811319 [I 2022-10-11 14:58:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 40/2304 [I 2022-10-11 14:58:24] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:05 [I 2022-10-11 14:58:24] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00451 [I 2022-10-11 14:58:24] ...trap/engines/engine.py.230: loss: 0.08878 [I 2022-10-11 14:58:24] ...optimizers/trijoint.py.151: total_norm: 15.273291 [I 2022-10-11 14:58:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 50/2304 [I 2022-10-11 14:58:32] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:32:05 [I 2022-10-11 14:58:32] ...trap/engines/engine.py.229: process: 0.85717 | load: 0.00460 [I 2022-10-11 14:58:32] ...trap/engines/engine.py.230: loss: 0.14793 [I 2022-10-11 14:58:32] ...optimizers/trijoint.py.151: total_norm: 21.683252 [I 2022-10-11 14:58:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 60/2304 [I 2022-10-11 14:58:41] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:32:01 [I 2022-10-11 14:58:41] ...trap/engines/engine.py.229: process: 0.86778 | load: 0.00367 [I 2022-10-11 14:58:41] ...trap/engines/engine.py.230: loss: 0.13965 [I 2022-10-11 14:58:41] ...optimizers/trijoint.py.151: total_norm: 18.979691 [I 2022-10-11 14:58:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 70/2304 [I 2022-10-11 14:58:49] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:32:06 [I 2022-10-11 14:58:49] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00369 [I 2022-10-11 14:58:49] ...trap/engines/engine.py.230: loss: 0.08440 [I 2022-10-11 14:58:49] ...optimizers/trijoint.py.151: total_norm: 9.444602 [I 2022-10-11 14:58:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 80/2304 [I 2022-10-11 14:58:58] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:37 [I 2022-10-11 14:58:58] ...trap/engines/engine.py.229: process: 0.85140 | load: 0.00381 [I 2022-10-11 14:58:58] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-11 14:58:58] ...optimizers/trijoint.py.151: total_norm: 13.098143 [I 2022-10-11 14:59:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 90/2304 [I 2022-10-11 14:59:07] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:37 [I 2022-10-11 14:59:07] ...trap/engines/engine.py.229: process: 0.86135 | load: 0.00364 [I 2022-10-11 14:59:07] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-11 14:59:07] ...optimizers/trijoint.py.151: total_norm: 21.885069 [I 2022-10-11 14:59:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 100/2304 [I 2022-10-11 14:59:15] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:21 [I 2022-10-11 14:59:15] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00371 [I 2022-10-11 14:59:15] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-11 14:59:15] ...optimizers/trijoint.py.151: total_norm: 19.752594 [I 2022-10-11 14:59:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 110/2304 [I 2022-10-11 14:59:24] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:15 [I 2022-10-11 14:59:24] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00328 [I 2022-10-11 14:59:24] ...trap/engines/engine.py.230: loss: 0.07285 [I 2022-10-11 14:59:24] ...optimizers/trijoint.py.151: total_norm: 11.766758 [I 2022-10-11 14:59:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 120/2304 [I 2022-10-11 14:59:32] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:10 [I 2022-10-11 14:59:32] ...trap/engines/engine.py.229: process: 0.85243 | load: 0.00349 [I 2022-10-11 14:59:32] ...trap/engines/engine.py.230: loss: 0.07923 [I 2022-10-11 14:59:32] ...optimizers/trijoint.py.151: total_norm: 8.555539 [I 2022-10-11 14:59:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 130/2304 [I 2022-10-11 14:59:41] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:04 [I 2022-10-11 14:59:41] ...trap/engines/engine.py.229: process: 0.85740 | load: 0.00344 [I 2022-10-11 14:59:41] ...trap/engines/engine.py.230: loss: 0.12160 [I 2022-10-11 14:59:41] ...optimizers/trijoint.py.151: total_norm: 15.798369 [I 2022-10-11 14:59:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 140/2304 [I 2022-10-11 14:59:49] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:57 [I 2022-10-11 14:59:49] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00314 [I 2022-10-11 14:59:49] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-11 14:59:49] ...optimizers/trijoint.py.151: total_norm: 15.130121 [I 2022-10-11 14:59:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 150/2304 [I 2022-10-11 14:59:58] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:44 [I 2022-10-11 14:59:58] ...trap/engines/engine.py.229: process: 0.85491 | load: 0.00325 [I 2022-10-11 14:59:58] ...trap/engines/engine.py.230: loss: 0.09310 [I 2022-10-11 14:59:58] ...optimizers/trijoint.py.151: total_norm: 17.828424 [I 2022-10-11 15:00:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 160/2304 [I 2022-10-11 15:00:07] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:29 [I 2022-10-11 15:00:07] ...trap/engines/engine.py.229: process: 0.83340 | load: 0.00337 [I 2022-10-11 15:00:07] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-11 15:00:07] ...optimizers/trijoint.py.151: total_norm: 11.565368 [I 2022-10-11 15:00:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 170/2304 [I 2022-10-11 15:00:15] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:33 [I 2022-10-11 15:00:15] ...trap/engines/engine.py.229: process: 0.88560 | load: 0.00344 [I 2022-10-11 15:00:15] ...trap/engines/engine.py.230: loss: 0.11588 [I 2022-10-11 15:00:15] ...optimizers/trijoint.py.151: total_norm: 15.132252 [I 2022-10-11 15:00:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 180/2304 [I 2022-10-11 15:00:24] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:31:35 [I 2022-10-11 15:00:24] ...trap/engines/engine.py.229: process: 0.90132 | load: 0.00392 [I 2022-10-11 15:00:24] ...trap/engines/engine.py.230: loss: 0.13900 [I 2022-10-11 15:00:24] ...optimizers/trijoint.py.151: total_norm: 9.582149 [I 2022-10-11 15:00:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 190/2304 [I 2022-10-11 15:00:33] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:31:06 [I 2022-10-11 15:00:33] ...trap/engines/engine.py.229: process: 0.88700 | load: 0.00427 [I 2022-10-11 15:00:33] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-11 15:00:33] ...optimizers/trijoint.py.151: total_norm: 29.403488 [I 2022-10-11 15:00:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 200/2304 [I 2022-10-11 15:00:42] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:48 [I 2022-10-11 15:00:42] ...trap/engines/engine.py.229: process: 0.85948 | load: 0.00343 [I 2022-10-11 15:00:42] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-11 15:00:42] ...optimizers/trijoint.py.151: total_norm: 22.554184 [I 2022-10-11 15:00:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 210/2304 [I 2022-10-11 15:00:50] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:52 [I 2022-10-11 15:00:50] ...trap/engines/engine.py.229: process: 0.84797 | load: 0.00312 [I 2022-10-11 15:00:50] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-11 15:00:50] ...optimizers/trijoint.py.151: total_norm: 18.742935 [I 2022-10-11 15:00:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 220/2304 [I 2022-10-11 15:00:59] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:45 [I 2022-10-11 15:00:59] ...trap/engines/engine.py.229: process: 0.85144 | load: 0.00328 [I 2022-10-11 15:00:59] ...trap/engines/engine.py.230: loss: 0.14252 [I 2022-10-11 15:00:59] ...optimizers/trijoint.py.151: total_norm: 15.919569 [I 2022-10-11 15:01:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 230/2304 [I 2022-10-11 15:01:08] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:26 [I 2022-10-11 15:01:08] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00336 [I 2022-10-11 15:01:08] ...trap/engines/engine.py.230: loss: 0.11801 [I 2022-10-11 15:01:08] ...optimizers/trijoint.py.151: total_norm: 16.156420 [I 2022-10-11 15:01:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 240/2304 [I 2022-10-11 15:01:16] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:34 [I 2022-10-11 15:01:16] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00313 [I 2022-10-11 15:01:16] ...trap/engines/engine.py.230: loss: 0.09044 [I 2022-10-11 15:01:16] ...optimizers/trijoint.py.151: total_norm: 14.069579 [I 2022-10-11 15:01:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 250/2304 [I 2022-10-11 15:01:25] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:20 [I 2022-10-11 15:01:25] ...trap/engines/engine.py.229: process: 0.84658 | load: 0.00323 [I 2022-10-11 15:01:25] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-11 15:01:25] ...optimizers/trijoint.py.151: total_norm: 80.318848 [I 2022-10-11 15:01:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 260/2304 [I 2022-10-11 15:01:33] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:15 [I 2022-10-11 15:01:33] ...trap/engines/engine.py.229: process: 0.84879 | load: 0.00312 [I 2022-10-11 15:01:33] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-11 15:01:33] ...optimizers/trijoint.py.151: total_norm: 29.509224 [I 2022-10-11 15:01:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 270/2304 [I 2022-10-11 15:01:42] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:29:08 [I 2022-10-11 15:01:42] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00311 [I 2022-10-11 15:01:42] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-11 15:01:42] ...optimizers/trijoint.py.151: total_norm: 12.017455 [I 2022-10-11 15:01:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 280/2304 [I 2022-10-11 15:01:51] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:54 [I 2022-10-11 15:01:51] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00327 [I 2022-10-11 15:01:51] ...trap/engines/engine.py.230: loss: 0.08959 [I 2022-10-11 15:01:51] ...optimizers/trijoint.py.151: total_norm: 12.427119 [I 2022-10-11 15:01:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 290/2304 [I 2022-10-11 15:01:59] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:35 [I 2022-10-11 15:01:59] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00311 [I 2022-10-11 15:01:59] ...trap/engines/engine.py.230: loss: 0.09788 [I 2022-10-11 15:01:59] ...optimizers/trijoint.py.151: total_norm: 12.332876 [I 2022-10-11 15:02:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 300/2304 [I 2022-10-11 15:02:08] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:28:28 [I 2022-10-11 15:02:08] ...trap/engines/engine.py.229: process: 0.86006 | load: 0.00322 [I 2022-10-11 15:02:08] ...trap/engines/engine.py.230: loss: 0.08487 [I 2022-10-11 15:02:08] ...optimizers/trijoint.py.151: total_norm: 11.919856 [I 2022-10-11 15:02:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 310/2304 [I 2022-10-11 15:02:16] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:02 [I 2022-10-11 15:02:16] ...trap/engines/engine.py.229: process: 0.84271 | load: 0.00302 [I 2022-10-11 15:02:16] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-11 15:02:16] ...optimizers/trijoint.py.151: total_norm: 14.996184 [I 2022-10-11 15:02:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 320/2304 [I 2022-10-11 15:02:24] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:27:45 [I 2022-10-11 15:02:25] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00294 [I 2022-10-11 15:02:25] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-11 15:02:25] ...optimizers/trijoint.py.151: total_norm: 36.155151 [I 2022-10-11 15:02:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 330/2304 [I 2022-10-11 15:02:33] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:27:31 [I 2022-10-11 15:02:33] ...trap/engines/engine.py.229: process: 0.83056 | load: 0.00299 [I 2022-10-11 15:02:33] ...trap/engines/engine.py.230: loss: 0.10652 [I 2022-10-11 15:02:33] ...optimizers/trijoint.py.151: total_norm: 16.863476 [I 2022-10-11 15:02:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 340/2304 [I 2022-10-11 15:02:41] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:27:33 [I 2022-10-11 15:02:41] ...trap/engines/engine.py.229: process: 0.85260 | load: 0.00310 [I 2022-10-11 15:02:41] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-11 15:02:41] ...optimizers/trijoint.py.151: total_norm: 15.062098 [I 2022-10-11 15:02:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 350/2304 [I 2022-10-11 15:02:50] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:27:30 [I 2022-10-11 15:02:50] ...trap/engines/engine.py.229: process: 0.84559 | load: 0.00300 [I 2022-10-11 15:02:50] ...trap/engines/engine.py.230: loss: 0.11804 [I 2022-10-11 15:02:50] ...optimizers/trijoint.py.151: total_norm: 18.051018 [I 2022-10-11 15:02:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 360/2304 [I 2022-10-11 15:02:58] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:20 [I 2022-10-11 15:02:58] ...trap/engines/engine.py.229: process: 0.84239 | load: 0.00299 [I 2022-10-11 15:02:58] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-11 15:02:58] ...optimizers/trijoint.py.151: total_norm: 12.421973 [I 2022-10-11 15:03:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 370/2304 [I 2022-10-11 15:03:07] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:29 [I 2022-10-11 15:03:07] ...trap/engines/engine.py.229: process: 0.86744 | load: 0.00307 [I 2022-10-11 15:03:07] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-11 15:03:07] ...optimizers/trijoint.py.151: total_norm: 30.601524 [I 2022-10-11 15:03:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 380/2304 [I 2022-10-11 15:03:15] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:06 [I 2022-10-11 15:03:15] ...trap/engines/engine.py.229: process: 0.83878 | load: 0.00300 [I 2022-10-11 15:03:15] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-11 15:03:15] ...optimizers/trijoint.py.151: total_norm: 10.888207 [I 2022-10-11 15:03:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 390/2304 [I 2022-10-11 15:03:24] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:17 [I 2022-10-11 15:03:24] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00309 [I 2022-10-11 15:03:24] ...trap/engines/engine.py.230: loss: 0.08471 [I 2022-10-11 15:03:24] ...optimizers/trijoint.py.151: total_norm: 15.220787 [I 2022-10-11 15:03:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 400/2304 [I 2022-10-11 15:03:32] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:06 [I 2022-10-11 15:03:32] ...trap/engines/engine.py.229: process: 0.86338 | load: 0.00299 [I 2022-10-11 15:03:32] ...trap/engines/engine.py.230: loss: 0.15066 [I 2022-10-11 15:03:32] ...optimizers/trijoint.py.151: total_norm: 24.843872 [I 2022-10-11 15:03:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 410/2304 [I 2022-10-11 15:03:41] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:26:53 [I 2022-10-11 15:03:41] ...trap/engines/engine.py.229: process: 0.85324 | load: 0.00300 [I 2022-10-11 15:03:41] ...trap/engines/engine.py.230: loss: 0.10280 [I 2022-10-11 15:03:41] ...optimizers/trijoint.py.151: total_norm: 9.587221 [I 2022-10-11 15:03:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 420/2304 [I 2022-10-11 15:03:49] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:26:39 [I 2022-10-11 15:03:49] ...trap/engines/engine.py.229: process: 0.84399 | load: 0.00311 [I 2022-10-11 15:03:49] ...trap/engines/engine.py.230: loss: 0.10352 [I 2022-10-11 15:03:49] ...optimizers/trijoint.py.151: total_norm: 10.380687 [I 2022-10-11 15:03:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 430/2304 [I 2022-10-11 15:03:58] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:21 [I 2022-10-11 15:03:58] ...trap/engines/engine.py.229: process: 0.83936 | load: 0.00301 [I 2022-10-11 15:03:58] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-11 15:03:58] ...optimizers/trijoint.py.151: total_norm: 9.026012 [I 2022-10-11 15:04:06] ...trap/engines/engine.py.224: train: epoch 109 | batch 440/2304 [I 2022-10-11 15:04:06] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:26:17 [I 2022-10-11 15:04:06] ...trap/engines/engine.py.229: process: 0.84417 | load: 0.00310 [I 2022-10-11 15:04:06] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-11 15:04:06] ...optimizers/trijoint.py.151: total_norm: 12.050481 [I 2022-10-11 15:04:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 450/2304 [I 2022-10-11 15:04:15] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:26:32 [I 2022-10-11 15:04:15] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00313 [I 2022-10-11 15:04:15] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-11 15:04:15] ...optimizers/trijoint.py.151: total_norm: 14.192091 [I 2022-10-11 15:04:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 460/2304 [I 2022-10-11 15:04:24] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:26:26 [I 2022-10-11 15:04:24] ...trap/engines/engine.py.229: process: 0.86140 | load: 0.00312 [I 2022-10-11 15:04:24] ...trap/engines/engine.py.230: loss: 0.09300 [I 2022-10-11 15:04:24] ...optimizers/trijoint.py.151: total_norm: 62.906876 [I 2022-10-11 15:04:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 470/2304 [I 2022-10-11 15:04:32] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:26:07 [I 2022-10-11 15:04:32] ...trap/engines/engine.py.229: process: 0.84749 | load: 0.00319 [I 2022-10-11 15:04:32] ...trap/engines/engine.py.230: loss: 0.10832 [I 2022-10-11 15:04:32] ...optimizers/trijoint.py.151: total_norm: 24.140488 [I 2022-10-11 15:04:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 480/2304 [I 2022-10-11 15:04:41] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:26:14 [I 2022-10-11 15:04:41] ...trap/engines/engine.py.229: process: 0.85437 | load: 0.00314 [I 2022-10-11 15:04:41] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-11 15:04:41] ...optimizers/trijoint.py.151: total_norm: 17.777788 [I 2022-10-11 15:04:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 490/2304 [I 2022-10-11 15:04:50] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:26:13 [I 2022-10-11 15:04:50] ...trap/engines/engine.py.229: process: 0.87137 | load: 0.00314 [I 2022-10-11 15:04:50] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-11 15:04:50] ...optimizers/trijoint.py.151: total_norm: 15.839027 [I 2022-10-11 15:04:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 500/2304 [I 2022-10-11 15:04:58] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:26:04 [I 2022-10-11 15:04:58] ...trap/engines/engine.py.229: process: 0.86132 | load: 0.00314 [I 2022-10-11 15:04:58] ...trap/engines/engine.py.230: loss: 0.08805 [I 2022-10-11 15:04:58] ...optimizers/trijoint.py.151: total_norm: 20.023533 [I 2022-10-11 15:05:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 510/2304 [I 2022-10-11 15:05:07] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:26:24 [I 2022-10-11 15:05:07] ...trap/engines/engine.py.229: process: 0.89171 | load: 0.00394 [I 2022-10-11 15:05:07] ...trap/engines/engine.py.230: loss: 0.08829 [I 2022-10-11 15:05:07] ...optimizers/trijoint.py.151: total_norm: 17.494074 [I 2022-10-11 15:05:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 520/2304 [I 2022-10-11 15:05:16] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:26:20 [I 2022-10-11 15:05:16] ...trap/engines/engine.py.229: process: 0.88863 | load: 0.00395 [I 2022-10-11 15:05:16] ...trap/engines/engine.py.230: loss: 0.13310 [I 2022-10-11 15:05:16] ...optimizers/trijoint.py.151: total_norm: 19.551266 [I 2022-10-11 15:05:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 530/2304 [I 2022-10-11 15:05:25] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:26:07 [I 2022-10-11 15:05:25] ...trap/engines/engine.py.229: process: 0.89021 | load: 0.00393 [I 2022-10-11 15:05:25] ...trap/engines/engine.py.230: loss: 0.11110 [I 2022-10-11 15:05:25] ...optimizers/trijoint.py.151: total_norm: 15.050261 [I 2022-10-11 15:05:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 540/2304 [I 2022-10-11 15:05:34] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:26:15 [I 2022-10-11 15:05:34] ...trap/engines/engine.py.229: process: 0.89566 | load: 0.00394 [I 2022-10-11 15:05:34] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-11 15:05:34] ...optimizers/trijoint.py.151: total_norm: 20.429152 [I 2022-10-11 15:05:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 550/2304 [I 2022-10-11 15:05:43] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:26:01 [I 2022-10-11 15:05:43] ...trap/engines/engine.py.229: process: 0.89389 | load: 0.00401 [I 2022-10-11 15:05:43] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-11 15:05:43] ...optimizers/trijoint.py.151: total_norm: 12.269166 [I 2022-10-11 15:05:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 560/2304 [I 2022-10-11 15:05:52] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:25:45 [I 2022-10-11 15:05:52] ...trap/engines/engine.py.229: process: 0.88833 | load: 0.00413 [I 2022-10-11 15:05:52] ...trap/engines/engine.py.230: loss: 0.09891 [I 2022-10-11 15:05:52] ...optimizers/trijoint.py.151: total_norm: 11.010855 [I 2022-10-11 15:06:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 570/2304 [I 2022-10-11 15:06:01] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:25:46 [I 2022-10-11 15:06:01] ...trap/engines/engine.py.229: process: 0.88727 | load: 0.00395 [I 2022-10-11 15:06:01] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-11 15:06:01] ...optimizers/trijoint.py.151: total_norm: 10.646769 [I 2022-10-11 15:06:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 580/2304 [I 2022-10-11 15:06:09] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:25:28 [I 2022-10-11 15:06:09] ...trap/engines/engine.py.229: process: 0.89424 | load: 0.00396 [I 2022-10-11 15:06:09] ...trap/engines/engine.py.230: loss: 0.12657 [I 2022-10-11 15:06:09] ...optimizers/trijoint.py.151: total_norm: 31.292269 [I 2022-10-11 15:06:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 590/2304 [I 2022-10-11 15:06:18] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:25:06 [I 2022-10-11 15:06:18] ...trap/engines/engine.py.229: process: 0.86111 | load: 0.00396 [I 2022-10-11 15:06:18] ...trap/engines/engine.py.230: loss: 0.13137 [I 2022-10-11 15:06:18] ...optimizers/trijoint.py.151: total_norm: 15.995966 [I 2022-10-11 15:06:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 600/2304 [I 2022-10-11 15:06:27] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:25:16 [I 2022-10-11 15:06:27] ...trap/engines/engine.py.229: process: 0.90256 | load: 0.00396 [I 2022-10-11 15:06:27] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-11 15:06:27] ...optimizers/trijoint.py.151: total_norm: 33.559826 [I 2022-10-11 15:06:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 610/2304 [I 2022-10-11 15:06:36] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:54 [I 2022-10-11 15:06:36] ...trap/engines/engine.py.229: process: 0.85253 | load: 0.00396 [I 2022-10-11 15:06:36] ...trap/engines/engine.py.230: loss: 0.14504 [I 2022-10-11 15:06:36] ...optimizers/trijoint.py.151: total_norm: 15.022051 [I 2022-10-11 15:06:45] ...trap/engines/engine.py.224: train: epoch 109 | batch 620/2304 [I 2022-10-11 15:06:45] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:12 [I 2022-10-11 15:06:45] ...trap/engines/engine.py.229: process: 0.86124 | load: 0.00316 [I 2022-10-11 15:06:45] ...trap/engines/engine.py.230: loss: 0.13259 [I 2022-10-11 15:06:45] ...optimizers/trijoint.py.151: total_norm: 16.520466 [I 2022-10-11 15:06:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 630/2304 [I 2022-10-11 15:06:53] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:48 [I 2022-10-11 15:06:53] ...trap/engines/engine.py.229: process: 0.85777 | load: 0.00330 [I 2022-10-11 15:06:53] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-11 15:06:53] ...optimizers/trijoint.py.151: total_norm: 13.799056 [I 2022-10-11 15:07:02] ...trap/engines/engine.py.224: train: epoch 109 | batch 640/2304 [I 2022-10-11 15:07:02] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:53 [I 2022-10-11 15:07:02] ...trap/engines/engine.py.229: process: 0.86403 | load: 0.00314 [I 2022-10-11 15:07:02] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-11 15:07:02] ...optimizers/trijoint.py.151: total_norm: 24.372885 [I 2022-10-11 15:07:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 650/2304 [I 2022-10-11 15:07:10] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:33 [I 2022-10-11 15:07:10] ...trap/engines/engine.py.229: process: 0.86797 | load: 0.00311 [I 2022-10-11 15:07:10] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-11 15:07:10] ...optimizers/trijoint.py.151: total_norm: 13.801108 [I 2022-10-11 15:07:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 660/2304 [I 2022-10-11 15:07:19] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:22 [I 2022-10-11 15:07:19] ...trap/engines/engine.py.229: process: 0.84734 | load: 0.00317 [I 2022-10-11 15:07:19] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-11 15:07:19] ...optimizers/trijoint.py.151: total_norm: 19.055462 [I 2022-10-11 15:07:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 670/2304 [I 2022-10-11 15:07:27] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:16 [I 2022-10-11 15:07:27] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00314 [I 2022-10-11 15:07:27] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-11 15:07:27] ...optimizers/trijoint.py.151: total_norm: 12.310022 [I 2022-10-11 15:07:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 680/2304 [I 2022-10-11 15:07:36] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:10 [I 2022-10-11 15:07:36] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00316 [I 2022-10-11 15:07:36] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-11 15:07:36] ...optimizers/trijoint.py.151: total_norm: 12.815122 [I 2022-10-11 15:07:45] ...trap/engines/engine.py.224: train: epoch 109 | batch 690/2304 [I 2022-10-11 15:07:45] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:57 [I 2022-10-11 15:07:45] ...trap/engines/engine.py.229: process: 0.85907 | load: 0.00319 [I 2022-10-11 15:07:45] ...trap/engines/engine.py.230: loss: 0.14962 [I 2022-10-11 15:07:45] ...optimizers/trijoint.py.151: total_norm: 15.062728 [I 2022-10-11 15:07:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 700/2304 [I 2022-10-11 15:07:53] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:45 [I 2022-10-11 15:07:53] ...trap/engines/engine.py.229: process: 0.84437 | load: 0.00312 [I 2022-10-11 15:07:53] ...trap/engines/engine.py.230: loss: 0.08473 [I 2022-10-11 15:07:53] ...optimizers/trijoint.py.151: total_norm: 15.185977 [I 2022-10-11 15:08:02] ...trap/engines/engine.py.224: train: epoch 109 | batch 710/2304 [I 2022-10-11 15:08:02] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:38 [I 2022-10-11 15:08:02] ...trap/engines/engine.py.229: process: 0.83572 | load: 0.00309 [I 2022-10-11 15:08:02] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-11 15:08:02] ...optimizers/trijoint.py.151: total_norm: 24.281130 [I 2022-10-11 15:08:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 720/2304 [I 2022-10-11 15:08:10] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:11 [I 2022-10-11 15:08:10] ...trap/engines/engine.py.229: process: 0.83864 | load: 0.00295 [I 2022-10-11 15:08:10] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 15:08:10] ...optimizers/trijoint.py.151: total_norm: 10.111850 [I 2022-10-11 15:08:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 730/2304 [I 2022-10-11 15:08:19] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:08 [I 2022-10-11 15:08:19] ...trap/engines/engine.py.229: process: 0.84560 | load: 0.00307 [I 2022-10-11 15:08:19] ...trap/engines/engine.py.230: loss: 0.09288 [I 2022-10-11 15:08:19] ...optimizers/trijoint.py.151: total_norm: 14.429839 [I 2022-10-11 15:08:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 740/2304 [I 2022-10-11 15:08:27] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:21:54 [I 2022-10-11 15:08:27] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00294 [I 2022-10-11 15:08:27] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-11 15:08:27] ...optimizers/trijoint.py.151: total_norm: 14.802231 [I 2022-10-11 15:08:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 750/2304 [I 2022-10-11 15:08:35] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:46 [I 2022-10-11 15:08:35] ...trap/engines/engine.py.229: process: 0.84147 | load: 0.00294 [I 2022-10-11 15:08:35] ...trap/engines/engine.py.230: loss: 0.12508 [I 2022-10-11 15:08:35] ...optimizers/trijoint.py.151: total_norm: 8.381719 [I 2022-10-11 15:08:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 760/2304 [I 2022-10-11 15:08:44] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:45 [I 2022-10-11 15:08:44] ...trap/engines/engine.py.229: process: 0.85564 | load: 0.00295 [I 2022-10-11 15:08:44] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-11 15:08:44] ...optimizers/trijoint.py.151: total_norm: 9.790938 [I 2022-10-11 15:08:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 770/2304 [I 2022-10-11 15:08:52] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:35 [I 2022-10-11 15:08:52] ...trap/engines/engine.py.229: process: 0.84903 | load: 0.00295 [I 2022-10-11 15:08:52] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-11 15:08:52] ...optimizers/trijoint.py.151: total_norm: 18.655777 [I 2022-10-11 15:09:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 780/2304 [I 2022-10-11 15:09:01] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:18 [I 2022-10-11 15:09:01] ...trap/engines/engine.py.229: process: 0.81933 | load: 0.00293 [I 2022-10-11 15:09:01] ...trap/engines/engine.py.230: loss: 0.14318 [I 2022-10-11 15:09:01] ...optimizers/trijoint.py.151: total_norm: 18.366266 [I 2022-10-11 15:09:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 790/2304 [I 2022-10-11 15:09:09] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:10 [I 2022-10-11 15:09:09] ...trap/engines/engine.py.229: process: 0.83470 | load: 0.00297 [I 2022-10-11 15:09:09] ...trap/engines/engine.py.230: loss: 0.13823 [I 2022-10-11 15:09:09] ...optimizers/trijoint.py.151: total_norm: 36.810379 [I 2022-10-11 15:09:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 800/2304 [I 2022-10-11 15:09:18] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:00 [I 2022-10-11 15:09:18] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00294 [I 2022-10-11 15:09:18] ...trap/engines/engine.py.230: loss: 0.21976 [I 2022-10-11 15:09:18] ...optimizers/trijoint.py.151: total_norm: 10.043406 [I 2022-10-11 15:09:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 810/2304 [I 2022-10-11 15:09:26] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:20:59 [I 2022-10-11 15:09:26] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00299 [I 2022-10-11 15:09:26] ...trap/engines/engine.py.230: loss: 0.10554 [I 2022-10-11 15:09:26] ...optimizers/trijoint.py.151: total_norm: 9.689009 [I 2022-10-11 15:09:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 820/2304 [I 2022-10-11 15:09:34] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:56 [I 2022-10-11 15:09:34] ...trap/engines/engine.py.229: process: 0.83224 | load: 0.00296 [I 2022-10-11 15:09:34] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-11 15:09:34] ...optimizers/trijoint.py.151: total_norm: 8.605203 [I 2022-10-11 15:09:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 830/2304 [I 2022-10-11 15:09:43] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:43 [I 2022-10-11 15:09:43] ...trap/engines/engine.py.229: process: 0.83545 | load: 0.00295 [I 2022-10-11 15:09:43] ...trap/engines/engine.py.230: loss: 0.10991 [I 2022-10-11 15:09:43] ...optimizers/trijoint.py.151: total_norm: 18.223654 [I 2022-10-11 15:09:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 840/2304 [I 2022-10-11 15:09:51] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:40 [I 2022-10-11 15:09:51] ...trap/engines/engine.py.229: process: 0.86207 | load: 0.00304 [I 2022-10-11 15:09:51] ...trap/engines/engine.py.230: loss: 0.12410 [I 2022-10-11 15:09:51] ...optimizers/trijoint.py.151: total_norm: 12.601981 [I 2022-10-11 15:10:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 850/2304 [I 2022-10-11 15:10:00] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:25 [I 2022-10-11 15:10:00] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00301 [I 2022-10-11 15:10:00] ...trap/engines/engine.py.230: loss: 0.12628 [I 2022-10-11 15:10:00] ...optimizers/trijoint.py.151: total_norm: 11.385942 [I 2022-10-11 15:10:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 860/2304 [I 2022-10-11 15:10:08] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:20:10 [I 2022-10-11 15:10:08] ...trap/engines/engine.py.229: process: 0.84191 | load: 0.00301 [I 2022-10-11 15:10:08] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-11 15:10:08] ...optimizers/trijoint.py.151: total_norm: 15.876056 [I 2022-10-11 15:10:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 870/2304 [I 2022-10-11 15:10:17] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:07 [I 2022-10-11 15:10:17] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00298 [I 2022-10-11 15:10:17] ...trap/engines/engine.py.230: loss: 0.11718 [I 2022-10-11 15:10:17] ...optimizers/trijoint.py.151: total_norm: 12.955469 [I 2022-10-11 15:10:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 880/2304 [I 2022-10-11 15:10:25] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:19:51 [I 2022-10-11 15:10:25] ...trap/engines/engine.py.229: process: 0.83209 | load: 0.00295 [I 2022-10-11 15:10:25] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-11 15:10:25] ...optimizers/trijoint.py.151: total_norm: 15.806735 [I 2022-10-11 15:10:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 890/2304 [I 2022-10-11 15:10:34] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:19:49 [I 2022-10-11 15:10:34] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00298 [I 2022-10-11 15:10:34] ...trap/engines/engine.py.230: loss: 0.07168 [I 2022-10-11 15:10:34] ...optimizers/trijoint.py.151: total_norm: 9.978981 [I 2022-10-11 15:10:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 900/2304 [I 2022-10-11 15:10:42] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:19:43 [I 2022-10-11 15:10:42] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00295 [I 2022-10-11 15:10:42] ...trap/engines/engine.py.230: loss: 0.10057 [I 2022-10-11 15:10:42] ...optimizers/trijoint.py.151: total_norm: 8.861895 [I 2022-10-11 15:10:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 910/2304 [I 2022-10-11 15:10:50] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:19:39 [I 2022-10-11 15:10:50] ...trap/engines/engine.py.229: process: 0.83989 | load: 0.00300 [I 2022-10-11 15:10:50] ...trap/engines/engine.py.230: loss: 0.08680 [I 2022-10-11 15:10:50] ...optimizers/trijoint.py.151: total_norm: 9.979358 [I 2022-10-11 15:10:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 920/2304 [I 2022-10-11 15:10:59] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:19:31 [I 2022-10-11 15:10:59] ...trap/engines/engine.py.229: process: 0.86423 | load: 0.00295 [I 2022-10-11 15:10:59] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-11 15:10:59] ...optimizers/trijoint.py.151: total_norm: 20.724472 [I 2022-10-11 15:11:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 930/2304 [I 2022-10-11 15:11:07] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:19:11 [I 2022-10-11 15:11:07] ...trap/engines/engine.py.229: process: 0.83018 | load: 0.00300 [I 2022-10-11 15:11:07] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-11 15:11:07] ...optimizers/trijoint.py.151: total_norm: 11.915782 [I 2022-10-11 15:11:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 940/2304 [I 2022-10-11 15:11:16] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:19:08 [I 2022-10-11 15:11:16] ...trap/engines/engine.py.229: process: 0.84335 | load: 0.00300 [I 2022-10-11 15:11:16] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-11 15:11:16] ...optimizers/trijoint.py.151: total_norm: 10.369140 [I 2022-10-11 15:11:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 950/2304 [I 2022-10-11 15:11:24] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:18:56 [I 2022-10-11 15:11:24] ...trap/engines/engine.py.229: process: 0.82849 | load: 0.00300 [I 2022-10-11 15:11:24] ...trap/engines/engine.py.230: loss: 0.08855 [I 2022-10-11 15:11:24] ...optimizers/trijoint.py.151: total_norm: 10.410493 [I 2022-10-11 15:11:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 960/2304 [I 2022-10-11 15:11:33] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:18:53 [I 2022-10-11 15:11:33] ...trap/engines/engine.py.229: process: 0.85460 | load: 0.00298 [I 2022-10-11 15:11:33] ...trap/engines/engine.py.230: loss: 0.14471 [I 2022-10-11 15:11:33] ...optimizers/trijoint.py.151: total_norm: 29.715469 [I 2022-10-11 15:11:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 970/2304 [I 2022-10-11 15:11:41] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:18:45 [I 2022-10-11 15:11:41] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00303 [I 2022-10-11 15:11:41] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-11 15:11:41] ...optimizers/trijoint.py.151: total_norm: 10.700845 [I 2022-10-11 15:11:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 980/2304 [I 2022-10-11 15:11:50] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:18:37 [I 2022-10-11 15:11:50] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00304 [I 2022-10-11 15:11:50] ...trap/engines/engine.py.230: loss: 0.15048 [I 2022-10-11 15:11:50] ...optimizers/trijoint.py.151: total_norm: 12.056158 [I 2022-10-11 15:11:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 990/2304 [I 2022-10-11 15:11:58] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:18:51 [I 2022-10-11 15:11:58] ...trap/engines/engine.py.229: process: 0.86868 | load: 0.00310 [I 2022-10-11 15:11:58] ...trap/engines/engine.py.230: loss: 0.07622 [I 2022-10-11 15:11:58] ...optimizers/trijoint.py.151: total_norm: 10.176964 [I 2022-10-11 15:12:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 1000/2304 [I 2022-10-11 15:12:07] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:18:42 [I 2022-10-11 15:12:07] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00311 [I 2022-10-11 15:12:07] ...trap/engines/engine.py.230: loss: 0.13471 [I 2022-10-11 15:12:07] ...optimizers/trijoint.py.151: total_norm: 14.440581 [I 2022-10-11 15:12:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 1010/2304 [I 2022-10-11 15:12:15] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:18:37 [I 2022-10-11 15:12:15] ...trap/engines/engine.py.229: process: 0.86222 | load: 0.00315 [I 2022-10-11 15:12:15] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-11 15:12:15] ...optimizers/trijoint.py.151: total_norm: 11.235013 [I 2022-10-11 15:12:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 1020/2304 [I 2022-10-11 15:12:24] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:18:21 [I 2022-10-11 15:12:24] ...trap/engines/engine.py.229: process: 0.84615 | load: 0.00315 [I 2022-10-11 15:12:24] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-11 15:12:24] ...optimizers/trijoint.py.151: total_norm: 10.627447 [I 2022-10-11 15:12:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1030/2304 [I 2022-10-11 15:12:33] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:18:21 [I 2022-10-11 15:12:33] ...trap/engines/engine.py.229: process: 0.87354 | load: 0.00313 [I 2022-10-11 15:12:33] ...trap/engines/engine.py.230: loss: 0.07880 [I 2022-10-11 15:12:33] ...optimizers/trijoint.py.151: total_norm: 11.394095 [I 2022-10-11 15:12:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 1040/2304 [I 2022-10-11 15:12:41] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:18:04 [I 2022-10-11 15:12:41] ...trap/engines/engine.py.229: process: 0.87201 | load: 0.00314 [I 2022-10-11 15:12:41] ...trap/engines/engine.py.230: loss: 0.11617 [I 2022-10-11 15:12:41] ...optimizers/trijoint.py.151: total_norm: 11.757946 [I 2022-10-11 15:12:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 1050/2304 [I 2022-10-11 15:12:50] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:56 [I 2022-10-11 15:12:50] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00311 [I 2022-10-11 15:12:50] ...trap/engines/engine.py.230: loss: 0.08902 [I 2022-10-11 15:12:50] ...optimizers/trijoint.py.151: total_norm: 11.635356 [I 2022-10-11 15:12:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 1060/2304 [I 2022-10-11 15:12:58] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:17:54 [I 2022-10-11 15:12:58] ...trap/engines/engine.py.229: process: 0.87871 | load: 0.00325 [I 2022-10-11 15:12:58] ...trap/engines/engine.py.230: loss: 0.16689 [I 2022-10-11 15:12:58] ...optimizers/trijoint.py.151: total_norm: 18.600367 [I 2022-10-11 15:13:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 1070/2304 [I 2022-10-11 15:13:07] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:37 [I 2022-10-11 15:13:07] ...trap/engines/engine.py.229: process: 0.86863 | load: 0.00312 [I 2022-10-11 15:13:07] ...trap/engines/engine.py.230: loss: 0.17776 [I 2022-10-11 15:13:07] ...optimizers/trijoint.py.151: total_norm: 11.129248 [I 2022-10-11 15:13:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 1080/2304 [I 2022-10-11 15:13:16] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:17:29 [I 2022-10-11 15:13:16] ...trap/engines/engine.py.229: process: 0.85592 | load: 0.00332 [I 2022-10-11 15:13:16] ...trap/engines/engine.py.230: loss: 0.18879 [I 2022-10-11 15:13:16] ...optimizers/trijoint.py.151: total_norm: 16.716396 [I 2022-10-11 15:13:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 1090/2304 [I 2022-10-11 15:13:24] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:17:25 [I 2022-10-11 15:13:24] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00310 [I 2022-10-11 15:13:24] ...trap/engines/engine.py.230: loss: 0.06179 [I 2022-10-11 15:13:24] ...optimizers/trijoint.py.151: total_norm: 8.458585 [I 2022-10-11 15:13:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1100/2304 [I 2022-10-11 15:13:33] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:17:18 [I 2022-10-11 15:13:33] ...trap/engines/engine.py.229: process: 0.86283 | load: 0.00313 [I 2022-10-11 15:13:33] ...trap/engines/engine.py.230: loss: 0.08347 [I 2022-10-11 15:13:33] ...optimizers/trijoint.py.151: total_norm: 16.274225 [I 2022-10-11 15:13:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 1110/2304 [I 2022-10-11 15:13:41] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:01 [I 2022-10-11 15:13:41] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00313 [I 2022-10-11 15:13:41] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-11 15:13:41] ...optimizers/trijoint.py.151: total_norm: 18.073547 [I 2022-10-11 15:13:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 1120/2304 [I 2022-10-11 15:13:50] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:17:02 [I 2022-10-11 15:13:50] ...trap/engines/engine.py.229: process: 0.87285 | load: 0.00379 [I 2022-10-11 15:13:50] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-11 15:13:50] ...optimizers/trijoint.py.151: total_norm: 13.651498 [I 2022-10-11 15:13:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 1130/2304 [I 2022-10-11 15:13:59] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:16:53 [I 2022-10-11 15:13:59] ...trap/engines/engine.py.229: process: 0.87732 | load: 0.00313 [I 2022-10-11 15:13:59] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-11 15:13:59] ...optimizers/trijoint.py.151: total_norm: 14.241257 [I 2022-10-11 15:14:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 1140/2304 [I 2022-10-11 15:14:07] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:50 [I 2022-10-11 15:14:07] ...trap/engines/engine.py.229: process: 0.86402 | load: 0.00318 [I 2022-10-11 15:14:07] ...trap/engines/engine.py.230: loss: 0.08976 [I 2022-10-11 15:14:07] ...optimizers/trijoint.py.151: total_norm: 14.256021 [I 2022-10-11 15:14:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 1150/2304 [I 2022-10-11 15:14:16] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:16:31 [I 2022-10-11 15:14:16] ...trap/engines/engine.py.229: process: 0.86095 | load: 0.00316 [I 2022-10-11 15:14:16] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-11 15:14:16] ...optimizers/trijoint.py.151: total_norm: 12.152616 [I 2022-10-11 15:14:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 1160/2304 [I 2022-10-11 15:14:25] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:22 [I 2022-10-11 15:14:25] ...trap/engines/engine.py.229: process: 0.88021 | load: 0.00316 [I 2022-10-11 15:14:25] ...trap/engines/engine.py.230: loss: 0.11114 [I 2022-10-11 15:14:25] ...optimizers/trijoint.py.151: total_norm: 29.623158 [I 2022-10-11 15:14:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1170/2304 [I 2022-10-11 15:14:33] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:16:34 [I 2022-10-11 15:14:33] ...trap/engines/engine.py.229: process: 0.88414 | load: 0.00395 [I 2022-10-11 15:14:33] ...trap/engines/engine.py.230: loss: 0.15001 [I 2022-10-11 15:14:33] ...optimizers/trijoint.py.151: total_norm: 11.202602 [I 2022-10-11 15:14:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 1180/2304 [I 2022-10-11 15:14:42] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:16:05 [I 2022-10-11 15:14:42] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00311 [I 2022-10-11 15:14:42] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-11 15:14:42] ...optimizers/trijoint.py.151: total_norm: 9.937787 [I 2022-10-11 15:14:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 1190/2304 [I 2022-10-11 15:14:50] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:15:54 [I 2022-10-11 15:14:50] ...trap/engines/engine.py.229: process: 0.86979 | load: 0.00315 [I 2022-10-11 15:14:50] ...trap/engines/engine.py.230: loss: 0.17047 [I 2022-10-11 15:14:50] ...optimizers/trijoint.py.151: total_norm: 18.545502 [I 2022-10-11 15:14:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 1200/2304 [I 2022-10-11 15:14:59] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:15:40 [I 2022-10-11 15:14:59] ...trap/engines/engine.py.229: process: 0.84787 | load: 0.00311 [I 2022-10-11 15:14:59] ...trap/engines/engine.py.230: loss: 0.10967 [I 2022-10-11 15:14:59] ...optimizers/trijoint.py.151: total_norm: 21.304811 [I 2022-10-11 15:15:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 1210/2304 [I 2022-10-11 15:15:08] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:15:36 [I 2022-10-11 15:15:08] ...trap/engines/engine.py.229: process: 0.85952 | load: 0.00313 [I 2022-10-11 15:15:08] ...trap/engines/engine.py.230: loss: 0.12365 [I 2022-10-11 15:15:08] ...optimizers/trijoint.py.151: total_norm: 26.792219 [I 2022-10-11 15:15:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 1220/2304 [I 2022-10-11 15:15:16] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:15:28 [I 2022-10-11 15:15:16] ...trap/engines/engine.py.229: process: 0.85898 | load: 0.00314 [I 2022-10-11 15:15:16] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-11 15:15:16] ...optimizers/trijoint.py.151: total_norm: 20.408722 [I 2022-10-11 15:15:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 1230/2304 [I 2022-10-11 15:15:25] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:15:43 [I 2022-10-11 15:15:25] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00412 [I 2022-10-11 15:15:25] ...trap/engines/engine.py.230: loss: 0.08826 [I 2022-10-11 15:15:25] ...optimizers/trijoint.py.151: total_norm: 11.423072 [I 2022-10-11 15:15:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1240/2304 [I 2022-10-11 15:15:33] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:15:16 [I 2022-10-11 15:15:33] ...trap/engines/engine.py.229: process: 0.86705 | load: 0.00314 [I 2022-10-11 15:15:33] ...trap/engines/engine.py.230: loss: 0.10287 [I 2022-10-11 15:15:33] ...optimizers/trijoint.py.151: total_norm: 11.737484 [I 2022-10-11 15:15:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 1250/2304 [I 2022-10-11 15:15:42] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:15:26 [I 2022-10-11 15:15:42] ...trap/engines/engine.py.229: process: 0.89734 | load: 0.00394 [I 2022-10-11 15:15:42] ...trap/engines/engine.py.230: loss: 0.13243 [I 2022-10-11 15:15:42] ...optimizers/trijoint.py.151: total_norm: 17.372717 [I 2022-10-11 15:15:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 1260/2304 [I 2022-10-11 15:15:51] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:15:20 [I 2022-10-11 15:15:51] ...trap/engines/engine.py.229: process: 0.87737 | load: 0.00378 [I 2022-10-11 15:15:51] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-11 15:15:51] ...optimizers/trijoint.py.151: total_norm: 11.935352 [I 2022-10-11 15:16:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 1270/2304 [I 2022-10-11 15:16:00] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:15:02 [I 2022-10-11 15:16:00] ...trap/engines/engine.py.229: process: 0.88081 | load: 0.00404 [I 2022-10-11 15:16:00] ...trap/engines/engine.py.230: loss: 0.09837 [I 2022-10-11 15:16:00] ...optimizers/trijoint.py.151: total_norm: 10.534970 [I 2022-10-11 15:16:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1280/2304 [I 2022-10-11 15:16:09] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:59 [I 2022-10-11 15:16:09] ...trap/engines/engine.py.229: process: 0.87996 | load: 0.00376 [I 2022-10-11 15:16:09] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-11 15:16:09] ...optimizers/trijoint.py.151: total_norm: 37.660667 [I 2022-10-11 15:16:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 1290/2304 [I 2022-10-11 15:16:17] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:41 [I 2022-10-11 15:16:17] ...trap/engines/engine.py.229: process: 0.87381 | load: 0.00379 [I 2022-10-11 15:16:17] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-11 15:16:17] ...optimizers/trijoint.py.151: total_norm: 13.193960 [I 2022-10-11 15:16:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1300/2304 [I 2022-10-11 15:16:26] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:35 [I 2022-10-11 15:16:26] ...trap/engines/engine.py.229: process: 0.87726 | load: 0.00376 [I 2022-10-11 15:16:26] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-11 15:16:26] ...optimizers/trijoint.py.151: total_norm: 30.728054 [I 2022-10-11 15:16:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1310/2304 [I 2022-10-11 15:16:35] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:28 [I 2022-10-11 15:16:35] ...trap/engines/engine.py.229: process: 0.87812 | load: 0.00375 [I 2022-10-11 15:16:35] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-11 15:16:35] ...optimizers/trijoint.py.151: total_norm: 25.381899 [I 2022-10-11 15:16:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1320/2304 [I 2022-10-11 15:16:43] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:14:18 [I 2022-10-11 15:16:43] ...trap/engines/engine.py.229: process: 0.87915 | load: 0.00374 [I 2022-10-11 15:16:43] ...trap/engines/engine.py.230: loss: 0.09564 [I 2022-10-11 15:16:44] ...optimizers/trijoint.py.151: total_norm: 9.465906 [I 2022-10-11 15:16:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1330/2304 [I 2022-10-11 15:16:52] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:14:11 [I 2022-10-11 15:16:52] ...trap/engines/engine.py.229: process: 0.87489 | load: 0.00378 [I 2022-10-11 15:16:52] ...trap/engines/engine.py.230: loss: 0.09603 [I 2022-10-11 15:16:52] ...optimizers/trijoint.py.151: total_norm: 11.820355 [I 2022-10-11 15:17:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1340/2304 [I 2022-10-11 15:17:01] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:14:06 [I 2022-10-11 15:17:01] ...trap/engines/engine.py.229: process: 0.87454 | load: 0.00380 [I 2022-10-11 15:17:01] ...trap/engines/engine.py.230: loss: 0.07983 [I 2022-10-11 15:17:01] ...optimizers/trijoint.py.151: total_norm: 9.021122 [I 2022-10-11 15:17:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 1350/2304 [I 2022-10-11 15:17:10] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:54 [I 2022-10-11 15:17:10] ...trap/engines/engine.py.229: process: 0.86524 | load: 0.00379 [I 2022-10-11 15:17:10] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-11 15:17:10] ...optimizers/trijoint.py.151: total_norm: 13.759372 [I 2022-10-11 15:17:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 1360/2304 [I 2022-10-11 15:17:19] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:44 [I 2022-10-11 15:17:19] ...trap/engines/engine.py.229: process: 0.88134 | load: 0.00382 [I 2022-10-11 15:17:19] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-11 15:17:19] ...optimizers/trijoint.py.151: total_norm: 17.833649 [I 2022-10-11 15:17:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 1370/2304 [I 2022-10-11 15:17:27] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:34 [I 2022-10-11 15:17:27] ...trap/engines/engine.py.229: process: 0.87759 | load: 0.00378 [I 2022-10-11 15:17:27] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-11 15:17:27] ...optimizers/trijoint.py.151: total_norm: 10.206272 [I 2022-10-11 15:17:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 1380/2304 [I 2022-10-11 15:17:36] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:05 [I 2022-10-11 15:17:36] ...trap/engines/engine.py.229: process: 0.84682 | load: 0.00297 [I 2022-10-11 15:17:36] ...trap/engines/engine.py.230: loss: 0.09760 [I 2022-10-11 15:17:36] ...optimizers/trijoint.py.151: total_norm: 12.117918 [I 2022-10-11 15:17:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1390/2304 [I 2022-10-11 15:17:44] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:48 [I 2022-10-11 15:17:44] ...trap/engines/engine.py.229: process: 0.84051 | load: 0.00296 [I 2022-10-11 15:17:44] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-11 15:17:44] ...optimizers/trijoint.py.151: total_norm: 11.712729 [I 2022-10-11 15:17:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 1400/2304 [I 2022-10-11 15:17:53] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:41 [I 2022-10-11 15:17:53] ...trap/engines/engine.py.229: process: 0.84181 | load: 0.00294 [I 2022-10-11 15:17:53] ...trap/engines/engine.py.230: loss: 0.10348 [I 2022-10-11 15:17:53] ...optimizers/trijoint.py.151: total_norm: 11.234772 [I 2022-10-11 15:18:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1410/2304 [I 2022-10-11 15:18:01] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:33 [I 2022-10-11 15:18:01] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00295 [I 2022-10-11 15:18:01] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-11 15:18:01] ...optimizers/trijoint.py.151: total_norm: 11.301590 [I 2022-10-11 15:18:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 1420/2304 [I 2022-10-11 15:18:10] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:24 [I 2022-10-11 15:18:10] ...trap/engines/engine.py.229: process: 0.83405 | load: 0.00297 [I 2022-10-11 15:18:10] ...trap/engines/engine.py.230: loss: 0.14268 [I 2022-10-11 15:18:10] ...optimizers/trijoint.py.151: total_norm: 13.933431 [I 2022-10-11 15:18:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1430/2304 [I 2022-10-11 15:18:18] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:12 [I 2022-10-11 15:18:18] ...trap/engines/engine.py.229: process: 0.84422 | load: 0.00294 [I 2022-10-11 15:18:18] ...trap/engines/engine.py.230: loss: 0.09172 [I 2022-10-11 15:18:18] ...optimizers/trijoint.py.151: total_norm: 11.435774 [I 2022-10-11 15:18:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1440/2304 [I 2022-10-11 15:18:26] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:03 [I 2022-10-11 15:18:26] ...trap/engines/engine.py.229: process: 0.83572 | load: 0.00296 [I 2022-10-11 15:18:26] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-11 15:18:26] ...optimizers/trijoint.py.151: total_norm: 21.938202 [I 2022-10-11 15:18:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1450/2304 [I 2022-10-11 15:18:35] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:00 [I 2022-10-11 15:18:35] ...trap/engines/engine.py.229: process: 0.84912 | load: 0.00295 [I 2022-10-11 15:18:35] ...trap/engines/engine.py.230: loss: 0.08875 [I 2022-10-11 15:18:35] ...optimizers/trijoint.py.151: total_norm: 19.481426 [I 2022-10-11 15:18:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1460/2304 [I 2022-10-11 15:18:43] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:50 [I 2022-10-11 15:18:43] ...trap/engines/engine.py.229: process: 0.83697 | load: 0.00295 [I 2022-10-11 15:18:43] ...trap/engines/engine.py.230: loss: 0.14271 [I 2022-10-11 15:18:43] ...optimizers/trijoint.py.151: total_norm: 11.425375 [I 2022-10-11 15:18:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1470/2304 [I 2022-10-11 15:18:52] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:43 [I 2022-10-11 15:18:52] ...trap/engines/engine.py.229: process: 0.83995 | load: 0.00295 [I 2022-10-11 15:18:52] ...trap/engines/engine.py.230: loss: 0.12519 [I 2022-10-11 15:18:52] ...optimizers/trijoint.py.151: total_norm: 15.741623 [I 2022-10-11 15:19:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 1480/2304 [I 2022-10-11 15:19:00] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:30 [I 2022-10-11 15:19:00] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00296 [I 2022-10-11 15:19:00] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-11 15:19:00] ...optimizers/trijoint.py.151: total_norm: 14.937955 [I 2022-10-11 15:19:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1490/2304 [I 2022-10-11 15:19:09] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:28 [I 2022-10-11 15:19:09] ...trap/engines/engine.py.229: process: 0.85039 | load: 0.00325 [I 2022-10-11 15:19:09] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-11 15:19:09] ...optimizers/trijoint.py.151: total_norm: 12.732003 [I 2022-10-11 15:19:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 1500/2304 [I 2022-10-11 15:19:17] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:16 [I 2022-10-11 15:19:17] ...trap/engines/engine.py.229: process: 0.83561 | load: 0.00296 [I 2022-10-11 15:19:17] ...trap/engines/engine.py.230: loss: 0.12732 [I 2022-10-11 15:19:17] ...optimizers/trijoint.py.151: total_norm: 9.734021 [I 2022-10-11 15:19:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1510/2304 [I 2022-10-11 15:19:26] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:13 [I 2022-10-11 15:19:26] ...trap/engines/engine.py.229: process: 0.85256 | load: 0.00310 [I 2022-10-11 15:19:26] ...trap/engines/engine.py.230: loss: 0.07917 [I 2022-10-11 15:19:26] ...optimizers/trijoint.py.151: total_norm: 12.185163 [I 2022-10-11 15:19:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 1520/2304 [I 2022-10-11 15:19:34] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:13 [I 2022-10-11 15:19:34] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00312 [I 2022-10-11 15:19:34] ...trap/engines/engine.py.230: loss: 0.09848 [I 2022-10-11 15:19:34] ...optimizers/trijoint.py.151: total_norm: 8.705233 [I 2022-10-11 15:19:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1530/2304 [I 2022-10-11 15:19:43] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:01 [I 2022-10-11 15:19:43] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00314 [I 2022-10-11 15:19:43] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-11 15:19:43] ...optimizers/trijoint.py.151: total_norm: 17.058762 [I 2022-10-11 15:19:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 1540/2304 [I 2022-10-11 15:19:51] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:52 [I 2022-10-11 15:19:51] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00316 [I 2022-10-11 15:19:51] ...trap/engines/engine.py.230: loss: 0.09388 [I 2022-10-11 15:19:51] ...optimizers/trijoint.py.151: total_norm: 11.288280 [I 2022-10-11 15:20:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 1550/2304 [I 2022-10-11 15:20:00] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:48 [I 2022-10-11 15:20:00] ...trap/engines/engine.py.229: process: 0.87546 | load: 0.00315 [I 2022-10-11 15:20:00] ...trap/engines/engine.py.230: loss: 0.11194 [I 2022-10-11 15:20:00] ...optimizers/trijoint.py.151: total_norm: 9.539702 [I 2022-10-11 15:20:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1560/2304 [I 2022-10-11 15:20:09] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:38 [I 2022-10-11 15:20:09] ...trap/engines/engine.py.229: process: 0.86225 | load: 0.00317 [I 2022-10-11 15:20:09] ...trap/engines/engine.py.230: loss: 0.10715 [I 2022-10-11 15:20:09] ...optimizers/trijoint.py.151: total_norm: 13.316215 [I 2022-10-11 15:20:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 1570/2304 [I 2022-10-11 15:20:17] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:30 [I 2022-10-11 15:20:17] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00314 [I 2022-10-11 15:20:17] ...trap/engines/engine.py.230: loss: 0.07939 [I 2022-10-11 15:20:17] ...optimizers/trijoint.py.151: total_norm: 10.522352 [I 2022-10-11 15:20:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1580/2304 [I 2022-10-11 15:20:26] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:23 [I 2022-10-11 15:20:26] ...trap/engines/engine.py.229: process: 0.86219 | load: 0.00315 [I 2022-10-11 15:20:26] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-11 15:20:26] ...optimizers/trijoint.py.151: total_norm: 27.235115 [I 2022-10-11 15:20:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 1590/2304 [I 2022-10-11 15:20:34] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:11 [I 2022-10-11 15:20:34] ...trap/engines/engine.py.229: process: 0.85416 | load: 0.00312 [I 2022-10-11 15:20:35] ...trap/engines/engine.py.230: loss: 0.17365 [I 2022-10-11 15:20:35] ...optimizers/trijoint.py.151: total_norm: 17.216560 [I 2022-10-11 15:20:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1600/2304 [I 2022-10-11 15:20:43] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:10:09 [I 2022-10-11 15:20:43] ...trap/engines/engine.py.229: process: 0.89758 | load: 0.00394 [I 2022-10-11 15:20:43] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-11 15:20:43] ...optimizers/trijoint.py.151: total_norm: 20.823538 [I 2022-10-11 15:20:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1610/2304 [I 2022-10-11 15:20:52] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:09:59 [I 2022-10-11 15:20:52] ...trap/engines/engine.py.229: process: 0.86206 | load: 0.00314 [I 2022-10-11 15:20:52] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-11 15:20:52] ...optimizers/trijoint.py.151: total_norm: 15.136882 [I 2022-10-11 15:21:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1620/2304 [I 2022-10-11 15:21:01] ...trap/engines/engine.py.228: elapsed: 0:23:13 | left: 0:09:51 [I 2022-10-11 15:21:01] ...trap/engines/engine.py.229: process: 0.86733 | load: 0.00313 [I 2022-10-11 15:21:01] ...trap/engines/engine.py.230: loss: 0.20098 [I 2022-10-11 15:21:01] ...optimizers/trijoint.py.151: total_norm: 25.819260 [I 2022-10-11 15:21:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1630/2304 [I 2022-10-11 15:21:09] ...trap/engines/engine.py.228: elapsed: 0:23:22 | left: 0:09:39 [I 2022-10-11 15:21:09] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00317 [I 2022-10-11 15:21:09] ...trap/engines/engine.py.230: loss: 0.12682 [I 2022-10-11 15:21:09] ...optimizers/trijoint.py.151: total_norm: 13.296373 [I 2022-10-11 15:21:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1640/2304 [I 2022-10-11 15:21:18] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:25 [I 2022-10-11 15:21:18] ...trap/engines/engine.py.229: process: 0.85071 | load: 0.00314 [I 2022-10-11 15:21:18] ...trap/engines/engine.py.230: loss: 0.13278 [I 2022-10-11 15:21:18] ...optimizers/trijoint.py.151: total_norm: 16.063669 [I 2022-10-11 15:21:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1650/2304 [I 2022-10-11 15:21:26] ...trap/engines/engine.py.228: elapsed: 0:23:39 | left: 0:09:23 [I 2022-10-11 15:21:26] ...trap/engines/engine.py.229: process: 0.87500 | load: 0.00316 [I 2022-10-11 15:21:26] ...trap/engines/engine.py.230: loss: 0.11764 [I 2022-10-11 15:21:26] ...optimizers/trijoint.py.151: total_norm: 14.564215 [I 2022-10-11 15:21:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1660/2304 [I 2022-10-11 15:21:35] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:10 [I 2022-10-11 15:21:35] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00311 [I 2022-10-11 15:21:35] ...trap/engines/engine.py.230: loss: 0.11392 [I 2022-10-11 15:21:35] ...optimizers/trijoint.py.151: total_norm: 24.861349 [I 2022-10-11 15:21:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1670/2304 [I 2022-10-11 15:21:43] ...trap/engines/engine.py.228: elapsed: 0:23:56 | left: 0:09:02 [I 2022-10-11 15:21:43] ...trap/engines/engine.py.229: process: 0.86305 | load: 0.00314 [I 2022-10-11 15:21:43] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-11 15:21:43] ...optimizers/trijoint.py.151: total_norm: 11.852464 [I 2022-10-11 15:21:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1680/2304 [I 2022-10-11 15:21:52] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:56 [I 2022-10-11 15:21:52] ...trap/engines/engine.py.229: process: 0.85857 | load: 0.00314 [I 2022-10-11 15:21:52] ...trap/engines/engine.py.230: loss: 0.15080 [I 2022-10-11 15:21:52] ...optimizers/trijoint.py.151: total_norm: 15.494341 [I 2022-10-11 15:22:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1690/2304 [I 2022-10-11 15:22:01] ...trap/engines/engine.py.228: elapsed: 0:24:14 | left: 0:08:44 [I 2022-10-11 15:22:01] ...trap/engines/engine.py.229: process: 0.84618 | load: 0.00320 [I 2022-10-11 15:22:01] ...trap/engines/engine.py.230: loss: 0.13275 [I 2022-10-11 15:22:01] ...optimizers/trijoint.py.151: total_norm: 19.162531 [I 2022-10-11 15:22:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1700/2304 [I 2022-10-11 15:22:09] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:39 [I 2022-10-11 15:22:09] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00316 [I 2022-10-11 15:22:09] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-11 15:22:09] ...optimizers/trijoint.py.151: total_norm: 10.640115 [I 2022-10-11 15:22:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1710/2304 [I 2022-10-11 15:22:18] ...trap/engines/engine.py.228: elapsed: 0:24:31 | left: 0:08:27 [I 2022-10-11 15:22:18] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00311 [I 2022-10-11 15:22:18] ...trap/engines/engine.py.230: loss: 0.13274 [I 2022-10-11 15:22:18] ...optimizers/trijoint.py.151: total_norm: 11.196431 [I 2022-10-11 15:22:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1720/2304 [I 2022-10-11 15:22:26] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:24 [I 2022-10-11 15:22:26] ...trap/engines/engine.py.229: process: 0.86759 | load: 0.00311 [I 2022-10-11 15:22:26] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-11 15:22:26] ...optimizers/trijoint.py.151: total_norm: 17.448635 [I 2022-10-11 15:22:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1730/2304 [I 2022-10-11 15:22:35] ...trap/engines/engine.py.228: elapsed: 0:24:48 | left: 0:08:11 [I 2022-10-11 15:22:35] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00310 [I 2022-10-11 15:22:35] ...trap/engines/engine.py.230: loss: 0.10794 [I 2022-10-11 15:22:35] ...optimizers/trijoint.py.151: total_norm: 21.133476 [I 2022-10-11 15:22:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1740/2304 [I 2022-10-11 15:22:44] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:02 [I 2022-10-11 15:22:44] ...trap/engines/engine.py.229: process: 0.86072 | load: 0.00315 [I 2022-10-11 15:22:44] ...trap/engines/engine.py.230: loss: 0.10063 [I 2022-10-11 15:22:44] ...optimizers/trijoint.py.151: total_norm: 19.274702 [I 2022-10-11 15:22:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1750/2304 [I 2022-10-11 15:22:52] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:07:56 [I 2022-10-11 15:22:52] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00314 [I 2022-10-11 15:22:52] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-11 15:22:52] ...optimizers/trijoint.py.151: total_norm: 23.704082 [I 2022-10-11 15:23:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1760/2304 [I 2022-10-11 15:23:01] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:07:48 [I 2022-10-11 15:23:01] ...trap/engines/engine.py.229: process: 0.86068 | load: 0.00316 [I 2022-10-11 15:23:01] ...trap/engines/engine.py.230: loss: 0.11855 [I 2022-10-11 15:23:01] ...optimizers/trijoint.py.151: total_norm: 27.479021 [I 2022-10-11 15:23:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 1770/2304 [I 2022-10-11 15:23:10] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:43 [I 2022-10-11 15:23:10] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00315 [I 2022-10-11 15:23:10] ...trap/engines/engine.py.230: loss: 0.08159 [I 2022-10-11 15:23:10] ...optimizers/trijoint.py.151: total_norm: 14.632620 [I 2022-10-11 15:23:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1780/2304 [I 2022-10-11 15:23:18] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:31 [I 2022-10-11 15:23:18] ...trap/engines/engine.py.229: process: 0.85696 | load: 0.00311 [I 2022-10-11 15:23:18] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-11 15:23:18] ...optimizers/trijoint.py.151: total_norm: 21.748474 [I 2022-10-11 15:23:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 1790/2304 [I 2022-10-11 15:23:27] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:19 [I 2022-10-11 15:23:27] ...trap/engines/engine.py.229: process: 0.84374 | load: 0.00297 [I 2022-10-11 15:23:27] ...trap/engines/engine.py.230: loss: 0.09607 [I 2022-10-11 15:23:27] ...optimizers/trijoint.py.151: total_norm: 8.877742 [I 2022-10-11 15:23:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1800/2304 [I 2022-10-11 15:23:35] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:10 [I 2022-10-11 15:23:35] ...trap/engines/engine.py.229: process: 0.85932 | load: 0.00300 [I 2022-10-11 15:23:35] ...trap/engines/engine.py.230: loss: 0.11980 [I 2022-10-11 15:23:35] ...optimizers/trijoint.py.151: total_norm: 9.238106 [I 2022-10-11 15:23:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1810/2304 [I 2022-10-11 15:23:44] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:00 [I 2022-10-11 15:23:44] ...trap/engines/engine.py.229: process: 0.83969 | load: 0.00297 [I 2022-10-11 15:23:44] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-11 15:23:44] ...optimizers/trijoint.py.151: total_norm: 14.192017 [I 2022-10-11 15:23:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1820/2304 [I 2022-10-11 15:23:52] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:47 [I 2022-10-11 15:23:52] ...trap/engines/engine.py.229: process: 0.83488 | load: 0.00296 [I 2022-10-11 15:23:52] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-11 15:23:52] ...optimizers/trijoint.py.151: total_norm: 22.565979 [I 2022-10-11 15:24:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1830/2304 [I 2022-10-11 15:24:01] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:42 [I 2022-10-11 15:24:01] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00297 [I 2022-10-11 15:24:01] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-11 15:24:01] ...optimizers/trijoint.py.151: total_norm: 10.331829 [I 2022-10-11 15:24:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1840/2304 [I 2022-10-11 15:24:09] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:35 [I 2022-10-11 15:24:09] ...trap/engines/engine.py.229: process: 0.85282 | load: 0.00293 [I 2022-10-11 15:24:09] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-11 15:24:09] ...optimizers/trijoint.py.151: total_norm: 20.948265 [I 2022-10-11 15:24:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1850/2304 [I 2022-10-11 15:24:18] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:25 [I 2022-10-11 15:24:18] ...trap/engines/engine.py.229: process: 0.85100 | load: 0.00296 [I 2022-10-11 15:24:18] ...trap/engines/engine.py.230: loss: 0.14211 [I 2022-10-11 15:24:18] ...optimizers/trijoint.py.151: total_norm: 24.744293 [I 2022-10-11 15:24:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1860/2304 [I 2022-10-11 15:24:26] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:17 [I 2022-10-11 15:24:26] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00295 [I 2022-10-11 15:24:26] ...trap/engines/engine.py.230: loss: 0.09018 [I 2022-10-11 15:24:26] ...optimizers/trijoint.py.151: total_norm: 9.046474 [I 2022-10-11 15:24:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1870/2304 [I 2022-10-11 15:24:35] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:06 [I 2022-10-11 15:24:35] ...trap/engines/engine.py.229: process: 0.83883 | load: 0.00296 [I 2022-10-11 15:24:35] ...trap/engines/engine.py.230: loss: 0.18939 [I 2022-10-11 15:24:35] ...optimizers/trijoint.py.151: total_norm: 13.394989 [I 2022-10-11 15:24:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1880/2304 [I 2022-10-11 15:24:43] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:05:57 [I 2022-10-11 15:24:43] ...trap/engines/engine.py.229: process: 0.83025 | load: 0.00298 [I 2022-10-11 15:24:43] ...trap/engines/engine.py.230: loss: 0.08874 [I 2022-10-11 15:24:43] ...optimizers/trijoint.py.151: total_norm: 8.853645 [I 2022-10-11 15:24:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1890/2304 [I 2022-10-11 15:24:52] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:50 [I 2022-10-11 15:24:52] ...trap/engines/engine.py.229: process: 0.85065 | load: 0.00297 [I 2022-10-11 15:24:52] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-11 15:24:52] ...optimizers/trijoint.py.151: total_norm: 10.207647 [I 2022-10-11 15:25:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 1900/2304 [I 2022-10-11 15:25:00] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:41 [I 2022-10-11 15:25:00] ...trap/engines/engine.py.229: process: 0.83443 | load: 0.00296 [I 2022-10-11 15:25:00] ...trap/engines/engine.py.230: loss: 0.12416 [I 2022-10-11 15:25:00] ...optimizers/trijoint.py.151: total_norm: 14.994630 [I 2022-10-11 15:25:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1910/2304 [I 2022-10-11 15:25:09] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:31 [I 2022-10-11 15:25:09] ...trap/engines/engine.py.229: process: 0.83992 | load: 0.00299 [I 2022-10-11 15:25:09] ...trap/engines/engine.py.230: loss: 0.10650 [I 2022-10-11 15:25:09] ...optimizers/trijoint.py.151: total_norm: 13.059573 [I 2022-10-11 15:25:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 1920/2304 [I 2022-10-11 15:25:17] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:27 [I 2022-10-11 15:25:17] ...trap/engines/engine.py.229: process: 0.86876 | load: 0.00378 [I 2022-10-11 15:25:17] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-11 15:25:17] ...optimizers/trijoint.py.151: total_norm: 15.398210 [I 2022-10-11 15:25:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1930/2304 [I 2022-10-11 15:25:26] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:25 [I 2022-10-11 15:25:26] ...trap/engines/engine.py.229: process: 0.87582 | load: 0.00377 [I 2022-10-11 15:25:26] ...trap/engines/engine.py.230: loss: 0.15785 [I 2022-10-11 15:25:26] ...optimizers/trijoint.py.151: total_norm: 21.452616 [I 2022-10-11 15:25:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1940/2304 [I 2022-10-11 15:25:35] ...trap/engines/engine.py.228: elapsed: 0:27:48 | left: 0:05:17 [I 2022-10-11 15:25:35] ...trap/engines/engine.py.229: process: 0.88328 | load: 0.00378 [I 2022-10-11 15:25:35] ...trap/engines/engine.py.230: loss: 0.08243 [I 2022-10-11 15:25:35] ...optimizers/trijoint.py.151: total_norm: 29.197504 [I 2022-10-11 15:25:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 1950/2304 [I 2022-10-11 15:25:43] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:07 [I 2022-10-11 15:25:43] ...trap/engines/engine.py.229: process: 0.86527 | load: 0.00394 [I 2022-10-11 15:25:43] ...trap/engines/engine.py.230: loss: 0.12993 [I 2022-10-11 15:25:43] ...optimizers/trijoint.py.151: total_norm: 16.787411 [I 2022-10-11 15:25:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1960/2304 [I 2022-10-11 15:25:52] ...trap/engines/engine.py.228: elapsed: 0:28:05 | left: 0:05:00 [I 2022-10-11 15:25:52] ...trap/engines/engine.py.229: process: 0.88482 | load: 0.00384 [I 2022-10-11 15:25:52] ...trap/engines/engine.py.230: loss: 0.18723 [I 2022-10-11 15:25:52] ...optimizers/trijoint.py.151: total_norm: 24.490438 [I 2022-10-11 15:26:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1970/2304 [I 2022-10-11 15:26:01] ...trap/engines/engine.py.228: elapsed: 0:28:14 | left: 0:04:47 [I 2022-10-11 15:26:01] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00297 [I 2022-10-11 15:26:01] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-11 15:26:01] ...optimizers/trijoint.py.151: total_norm: 12.269629 [I 2022-10-11 15:26:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1980/2304 [I 2022-10-11 15:26:09] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:04:34 [I 2022-10-11 15:26:09] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00299 [I 2022-10-11 15:26:09] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-11 15:26:09] ...optimizers/trijoint.py.151: total_norm: 10.111399 [I 2022-10-11 15:26:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1990/2304 [I 2022-10-11 15:26:18] ...trap/engines/engine.py.228: elapsed: 0:28:31 | left: 0:04:26 [I 2022-10-11 15:26:18] ...trap/engines/engine.py.229: process: 0.83887 | load: 0.00313 [I 2022-10-11 15:26:18] ...trap/engines/engine.py.230: loss: 0.09800 [I 2022-10-11 15:26:18] ...optimizers/trijoint.py.151: total_norm: 10.496251 [I 2022-10-11 15:26:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 2000/2304 [I 2022-10-11 15:26:26] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:17 [I 2022-10-11 15:26:26] ...trap/engines/engine.py.229: process: 0.84584 | load: 0.00311 [I 2022-10-11 15:26:26] ...trap/engines/engine.py.230: loss: 0.17574 [I 2022-10-11 15:26:26] ...optimizers/trijoint.py.151: total_norm: 25.267359 [I 2022-10-11 15:26:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 2010/2304 [I 2022-10-11 15:26:35] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:04:08 [I 2022-10-11 15:26:35] ...trap/engines/engine.py.229: process: 0.83481 | load: 0.00316 [I 2022-10-11 15:26:35] ...trap/engines/engine.py.230: loss: 0.09312 [I 2022-10-11 15:26:35] ...optimizers/trijoint.py.151: total_norm: 21.147892 [I 2022-10-11 15:26:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 2020/2304 [I 2022-10-11 15:26:43] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:02 [I 2022-10-11 15:26:43] ...trap/engines/engine.py.229: process: 0.87277 | load: 0.00310 [I 2022-10-11 15:26:43] ...trap/engines/engine.py.230: loss: 0.14166 [I 2022-10-11 15:26:43] ...optimizers/trijoint.py.151: total_norm: 29.684458 [I 2022-10-11 15:26:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 2030/2304 [I 2022-10-11 15:26:52] ...trap/engines/engine.py.228: elapsed: 0:29:05 | left: 0:03:56 [I 2022-10-11 15:26:52] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00316 [I 2022-10-11 15:26:52] ...trap/engines/engine.py.230: loss: 0.19341 [I 2022-10-11 15:26:52] ...optimizers/trijoint.py.151: total_norm: 27.153986 [I 2022-10-11 15:27:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 2040/2304 [I 2022-10-11 15:27:01] ...trap/engines/engine.py.228: elapsed: 0:29:14 | left: 0:03:47 [I 2022-10-11 15:27:01] ...trap/engines/engine.py.229: process: 0.87060 | load: 0.00316 [I 2022-10-11 15:27:01] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-11 15:27:01] ...optimizers/trijoint.py.151: total_norm: 14.488096 [I 2022-10-11 15:27:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 2050/2304 [I 2022-10-11 15:27:09] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:03:39 [I 2022-10-11 15:27:09] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00313 [I 2022-10-11 15:27:09] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-11 15:27:09] ...optimizers/trijoint.py.151: total_norm: 18.105404 [I 2022-10-11 15:27:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 2060/2304 [I 2022-10-11 15:27:18] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:03:32 [I 2022-10-11 15:27:18] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00315 [I 2022-10-11 15:27:18] ...trap/engines/engine.py.230: loss: 0.12245 [I 2022-10-11 15:27:18] ...optimizers/trijoint.py.151: total_norm: 11.477199 [I 2022-10-11 15:27:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 2070/2304 [I 2022-10-11 15:27:27] ...trap/engines/engine.py.228: elapsed: 0:29:39 | left: 0:03:20 [I 2022-10-11 15:27:27] ...trap/engines/engine.py.229: process: 0.84194 | load: 0.00312 [I 2022-10-11 15:27:27] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-11 15:27:27] ...optimizers/trijoint.py.151: total_norm: 25.814674 [I 2022-10-11 15:27:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 2080/2304 [I 2022-10-11 15:27:35] ...trap/engines/engine.py.228: elapsed: 0:29:48 | left: 0:03:11 [I 2022-10-11 15:27:35] ...trap/engines/engine.py.229: process: 0.85950 | load: 0.00317 [I 2022-10-11 15:27:35] ...trap/engines/engine.py.230: loss: 0.10271 [I 2022-10-11 15:27:35] ...optimizers/trijoint.py.151: total_norm: 20.518681 [I 2022-10-11 15:27:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 2090/2304 [I 2022-10-11 15:27:44] ...trap/engines/engine.py.228: elapsed: 0:29:57 | left: 0:03:02 [I 2022-10-11 15:27:44] ...trap/engines/engine.py.229: process: 0.85668 | load: 0.00313 [I 2022-10-11 15:27:44] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-11 15:27:44] ...optimizers/trijoint.py.151: total_norm: 13.729412 [I 2022-10-11 15:27:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 2100/2304 [I 2022-10-11 15:27:52] ...trap/engines/engine.py.228: elapsed: 0:30:05 | left: 0:02:53 [I 2022-10-11 15:27:52] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00313 [I 2022-10-11 15:27:52] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-11 15:27:52] ...optimizers/trijoint.py.151: total_norm: 20.706612 [I 2022-10-11 15:28:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 2110/2304 [I 2022-10-11 15:28:01] ...trap/engines/engine.py.228: elapsed: 0:30:14 | left: 0:02:47 [I 2022-10-11 15:28:01] ...trap/engines/engine.py.229: process: 0.86904 | load: 0.00314 [I 2022-10-11 15:28:01] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-11 15:28:01] ...optimizers/trijoint.py.151: total_norm: 12.133723 [I 2022-10-11 15:28:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 2120/2304 [I 2022-10-11 15:28:09] ...trap/engines/engine.py.228: elapsed: 0:30:22 | left: 0:02:36 [I 2022-10-11 15:28:09] ...trap/engines/engine.py.229: process: 0.82785 | load: 0.00315 [I 2022-10-11 15:28:09] ...trap/engines/engine.py.230: loss: 0.08821 [I 2022-10-11 15:28:09] ...optimizers/trijoint.py.151: total_norm: 8.239273 [I 2022-10-11 15:28:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 2130/2304 [I 2022-10-11 15:28:18] ...trap/engines/engine.py.228: elapsed: 0:30:31 | left: 0:02:28 [I 2022-10-11 15:28:18] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00312 [I 2022-10-11 15:28:18] ...trap/engines/engine.py.230: loss: 0.18899 [I 2022-10-11 15:28:18] ...optimizers/trijoint.py.151: total_norm: 28.703993 [I 2022-10-11 15:28:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 2140/2304 [I 2022-10-11 15:28:26] ...trap/engines/engine.py.228: elapsed: 0:30:39 | left: 0:02:19 [I 2022-10-11 15:28:26] ...trap/engines/engine.py.229: process: 0.85708 | load: 0.00315 [I 2022-10-11 15:28:26] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-11 15:28:26] ...optimizers/trijoint.py.151: total_norm: 18.090179 [I 2022-10-11 15:28:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 2150/2304 [I 2022-10-11 15:28:35] ...trap/engines/engine.py.228: elapsed: 0:30:48 | left: 0:02:11 [I 2022-10-11 15:28:35] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00313 [I 2022-10-11 15:28:35] ...trap/engines/engine.py.230: loss: 0.10445 [I 2022-10-11 15:28:35] ...optimizers/trijoint.py.151: total_norm: 16.226954 [I 2022-10-11 15:28:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 2160/2304 [I 2022-10-11 15:28:44] ...trap/engines/engine.py.228: elapsed: 0:30:57 | left: 0:02:03 [I 2022-10-11 15:28:44] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00313 [I 2022-10-11 15:28:44] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-11 15:28:44] ...optimizers/trijoint.py.151: total_norm: 15.000812 [I 2022-10-11 15:28:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 2170/2304 [I 2022-10-11 15:28:52] ...trap/engines/engine.py.228: elapsed: 0:31:05 | left: 0:01:54 [I 2022-10-11 15:28:52] ...trap/engines/engine.py.229: process: 0.86452 | load: 0.00314 [I 2022-10-11 15:28:52] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-11 15:28:52] ...optimizers/trijoint.py.151: total_norm: 23.253057 [I 2022-10-11 15:29:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 2180/2304 [I 2022-10-11 15:29:01] ...trap/engines/engine.py.228: elapsed: 0:31:14 | left: 0:01:46 [I 2022-10-11 15:29:01] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00313 [I 2022-10-11 15:29:01] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-11 15:29:01] ...optimizers/trijoint.py.151: total_norm: 38.295910 [I 2022-10-11 15:29:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 2190/2304 [I 2022-10-11 15:29:09] ...trap/engines/engine.py.228: elapsed: 0:31:22 | left: 0:01:37 [I 2022-10-11 15:29:09] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00312 [I 2022-10-11 15:29:09] ...trap/engines/engine.py.230: loss: 0.16902 [I 2022-10-11 15:29:09] ...optimizers/trijoint.py.151: total_norm: 30.833982 [I 2022-10-11 15:29:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 2200/2304 [I 2022-10-11 15:29:18] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:01:29 [I 2022-10-11 15:29:18] ...trap/engines/engine.py.229: process: 0.85344 | load: 0.00310 [I 2022-10-11 15:29:18] ...trap/engines/engine.py.230: loss: 0.11794 [I 2022-10-11 15:29:18] ...optimizers/trijoint.py.151: total_norm: 11.420183 [I 2022-10-11 15:29:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 2210/2304 [I 2022-10-11 15:29:26] ...trap/engines/engine.py.228: elapsed: 0:31:39 | left: 0:01:20 [I 2022-10-11 15:29:26] ...trap/engines/engine.py.229: process: 0.86462 | load: 0.00315 [I 2022-10-11 15:29:26] ...trap/engines/engine.py.230: loss: 0.08904 [I 2022-10-11 15:29:26] ...optimizers/trijoint.py.151: total_norm: 15.928869 [I 2022-10-11 15:29:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 2220/2304 [I 2022-10-11 15:29:35] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:12 [I 2022-10-11 15:29:35] ...trap/engines/engine.py.229: process: 0.85820 | load: 0.00312 [I 2022-10-11 15:29:35] ...trap/engines/engine.py.230: loss: 0.14250 [I 2022-10-11 15:29:35] ...optimizers/trijoint.py.151: total_norm: 16.039030 [I 2022-10-11 15:29:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 2230/2304 [I 2022-10-11 15:29:44] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:03 [I 2022-10-11 15:29:44] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00316 [I 2022-10-11 15:29:44] ...trap/engines/engine.py.230: loss: 0.09393 [I 2022-10-11 15:29:44] ...optimizers/trijoint.py.151: total_norm: 9.959744 [I 2022-10-11 15:29:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 2240/2304 [I 2022-10-11 15:29:52] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:00:55 [I 2022-10-11 15:29:52] ...trap/engines/engine.py.229: process: 0.88758 | load: 0.00393 [I 2022-10-11 15:29:52] ...trap/engines/engine.py.230: loss: 0.12609 [I 2022-10-11 15:29:52] ...optimizers/trijoint.py.151: total_norm: 11.967928 [I 2022-10-11 15:30:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 2250/2304 [I 2022-10-11 15:30:01] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:00:47 [I 2022-10-11 15:30:01] ...trap/engines/engine.py.229: process: 0.88392 | load: 0.00396 [I 2022-10-11 15:30:01] ...trap/engines/engine.py.230: loss: 0.10129 [I 2022-10-11 15:30:01] ...optimizers/trijoint.py.151: total_norm: 11.516946 [I 2022-10-11 15:30:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 2260/2304 [I 2022-10-11 15:30:10] ...trap/engines/engine.py.228: elapsed: 0:32:23 | left: 0:00:38 [I 2022-10-11 15:30:10] ...trap/engines/engine.py.229: process: 0.87635 | load: 0.00382 [I 2022-10-11 15:30:10] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-11 15:30:10] ...optimizers/trijoint.py.151: total_norm: 16.118589 [I 2022-10-11 15:30:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 2270/2304 [I 2022-10-11 15:30:19] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:29 [I 2022-10-11 15:30:19] ...trap/engines/engine.py.229: process: 0.88352 | load: 0.00376 [I 2022-10-11 15:30:19] ...trap/engines/engine.py.230: loss: 0.15999 [I 2022-10-11 15:30:19] ...optimizers/trijoint.py.151: total_norm: 18.533472 [I 2022-10-11 15:30:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 2280/2304 [I 2022-10-11 15:30:27] ...trap/engines/engine.py.228: elapsed: 0:32:40 | left: 0:00:20 [I 2022-10-11 15:30:27] ...trap/engines/engine.py.229: process: 0.86394 | load: 0.00298 [I 2022-10-11 15:30:27] ...trap/engines/engine.py.230: loss: 0.10388 [I 2022-10-11 15:30:27] ...optimizers/trijoint.py.151: total_norm: 14.721525 [I 2022-10-11 15:30:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 2290/2304 [I 2022-10-11 15:30:36] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:11 [I 2022-10-11 15:30:36] ...trap/engines/engine.py.229: process: 0.84293 | load: 0.00305 [I 2022-10-11 15:30:36] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-11 15:30:36] ...optimizers/trijoint.py.151: total_norm: 18.830374 [I 2022-10-11 15:30:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 2300/2304 [I 2022-10-11 15:30:44] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:03 [I 2022-10-11 15:30:44] ...trap/engines/engine.py.229: process: 0.82776 | load: 0.00016 [I 2022-10-11 15:30:44] ...trap/engines/engine.py.230: loss: 0.19325 [I 2022-10-11 15:30:44] ...optimizers/trijoint.py.151: total_norm: 15.666198 [I 2022-10-11 15:30:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 2304/2304 [I 2022-10-11 15:30:48] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:00 [I 2022-10-11 15:30:48] ...trap/engines/engine.py.229: process: 0.83398 | load: 0.00016 [I 2022-10-11 15:30:48] ...trap/engines/engine.py.230: loss: 0.13906 [I 2022-10-11 15:30:48] ...optimizers/trijoint.py.151: total_norm: 32.395378 [S 2022-10-11 15:30:49] ...trap/engines/engine.py.129: train_epoch.epoch: 109 [S 2022-10-11 15:30:49] ...trap/engines/engine.py.129: train_epoch.loss: 0.12046423301488614 [I 2022-10-11 15:30:49] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 15:30:50] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 109 [I 2022-10-11 15:30:51] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 15:30:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 15:30:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 15:30:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 15:30:56] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 15:30:57] ...trap/engines/engine.py.321: eval: epoch 109 | batch 0/510 [I 2022-10-11 15:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:03 [I 2022-10-11 15:30:57] ...trap/engines/engine.py.326: process: 6.24181 | load: 5.71072 [I 2022-10-11 15:31:02] ...trap/engines/engine.py.321: eval: epoch 109 | batch 10/510 [I 2022-10-11 15:31:02] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:18 [I 2022-10-11 15:31:02] ...trap/engines/engine.py.326: process: 0.46418 | load: 0.00018 [I 2022-10-11 15:31:07] ...trap/engines/engine.py.321: eval: epoch 109 | batch 20/510 [I 2022-10-11 15:31:07] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:04:53 [I 2022-10-11 15:31:07] ...trap/engines/engine.py.326: process: 0.58744 | load: 0.00018 [I 2022-10-11 15:31:12] ...trap/engines/engine.py.321: eval: epoch 109 | batch 30/510 [I 2022-10-11 15:31:12] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:22 [I 2022-10-11 15:31:12] ...trap/engines/engine.py.326: process: 0.68640 | load: 0.00017 [I 2022-10-11 15:31:18] ...trap/engines/engine.py.321: eval: epoch 109 | batch 40/510 [I 2022-10-11 15:31:18] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:20 [I 2022-10-11 15:31:18] ...trap/engines/engine.py.326: process: 0.42503 | load: 0.00018 [I 2022-10-11 15:31:23] ...trap/engines/engine.py.321: eval: epoch 109 | batch 50/510 [I 2022-10-11 15:31:23] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:50 [I 2022-10-11 15:31:23] ...trap/engines/engine.py.326: process: 0.45150 | load: 0.00019 [I 2022-10-11 15:31:28] ...trap/engines/engine.py.321: eval: epoch 109 | batch 60/510 [I 2022-10-11 15:31:28] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:03:33 [I 2022-10-11 15:31:28] ...trap/engines/engine.py.326: process: 0.49918 | load: 0.00017 [I 2022-10-11 15:31:33] ...trap/engines/engine.py.321: eval: epoch 109 | batch 70/510 [I 2022-10-11 15:31:33] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:22 [I 2022-10-11 15:31:33] ...trap/engines/engine.py.326: process: 0.46124 | load: 0.00019 [I 2022-10-11 15:31:37] ...trap/engines/engine.py.321: eval: epoch 109 | batch 80/510 [I 2022-10-11 15:31:37] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:32 [I 2022-10-11 15:31:37] ...trap/engines/engine.py.326: process: 0.54148 | load: 0.00018 [I 2022-10-11 15:31:42] ...trap/engines/engine.py.321: eval: epoch 109 | batch 90/510 [I 2022-10-11 15:31:42] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:03:03 [I 2022-10-11 15:31:42] ...trap/engines/engine.py.326: process: 0.43175 | load: 0.00017 [I 2022-10-11 15:31:46] ...trap/engines/engine.py.321: eval: epoch 109 | batch 100/510 [I 2022-10-11 15:31:46] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:49 [I 2022-10-11 15:31:46] ...trap/engines/engine.py.326: process: 0.31056 | load: 0.00018 [I 2022-10-11 15:31:49] ...trap/engines/engine.py.321: eval: epoch 109 | batch 110/510 [I 2022-10-11 15:31:49] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:09 [I 2022-10-11 15:31:49] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00018 [I 2022-10-11 15:31:52] ...trap/engines/engine.py.321: eval: epoch 109 | batch 120/510 [I 2022-10-11 15:31:52] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:03 [I 2022-10-11 15:31:52] ...trap/engines/engine.py.326: process: 0.31254 | load: 0.00018 [I 2022-10-11 15:31:55] ...trap/engines/engine.py.321: eval: epoch 109 | batch 130/510 [I 2022-10-11 15:31:55] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:59 [I 2022-10-11 15:31:55] ...trap/engines/engine.py.326: process: 0.31355 | load: 0.00020 [I 2022-10-11 15:31:59] ...trap/engines/engine.py.321: eval: epoch 109 | batch 140/510 [I 2022-10-11 15:31:59] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:56 [I 2022-10-11 15:31:59] ...trap/engines/engine.py.326: process: 0.31333 | load: 0.00018 [I 2022-10-11 15:32:02] ...trap/engines/engine.py.321: eval: epoch 109 | batch 150/510 [I 2022-10-11 15:32:02] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:53 [I 2022-10-11 15:32:02] ...trap/engines/engine.py.326: process: 0.31579 | load: 0.00019 [I 2022-10-11 15:32:05] ...trap/engines/engine.py.321: eval: epoch 109 | batch 160/510 [I 2022-10-11 15:32:05] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:50 [I 2022-10-11 15:32:05] ...trap/engines/engine.py.326: process: 0.31407 | load: 0.00020 [I 2022-10-11 15:32:08] ...trap/engines/engine.py.321: eval: epoch 109 | batch 170/510 [I 2022-10-11 15:32:08] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-11 15:32:08] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00020 [I 2022-10-11 15:32:11] ...trap/engines/engine.py.321: eval: epoch 109 | batch 180/510 [I 2022-10-11 15:32:11] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:46 [I 2022-10-11 15:32:11] ...trap/engines/engine.py.326: process: 0.32566 | load: 0.00020 [I 2022-10-11 15:32:15] ...trap/engines/engine.py.321: eval: epoch 109 | batch 190/510 [I 2022-10-11 15:32:15] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:42 [I 2022-10-11 15:32:15] ...trap/engines/engine.py.326: process: 0.32249 | load: 0.00020 [I 2022-10-11 15:32:18] ...trap/engines/engine.py.321: eval: epoch 109 | batch 200/510 [I 2022-10-11 15:32:18] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:39 [I 2022-10-11 15:32:18] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00020 [I 2022-10-11 15:32:21] ...trap/engines/engine.py.321: eval: epoch 109 | batch 210/510 [I 2022-10-11 15:32:21] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:35 [I 2022-10-11 15:32:21] ...trap/engines/engine.py.326: process: 0.32017 | load: 0.00021 [I 2022-10-11 15:32:24] ...trap/engines/engine.py.321: eval: epoch 109 | batch 220/510 [I 2022-10-11 15:32:24] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:32 [I 2022-10-11 15:32:24] ...trap/engines/engine.py.326: process: 0.31938 | load: 0.00021 [I 2022-10-11 15:32:27] ...trap/engines/engine.py.321: eval: epoch 109 | batch 230/510 [I 2022-10-11 15:32:27] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-11 15:32:27] ...trap/engines/engine.py.326: process: 0.32149 | load: 0.00021 [I 2022-10-11 15:32:31] ...trap/engines/engine.py.321: eval: epoch 109 | batch 240/510 [I 2022-10-11 15:32:31] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:26 [I 2022-10-11 15:32:31] ...trap/engines/engine.py.326: process: 0.31863 | load: 0.00022 [I 2022-10-11 15:32:34] ...trap/engines/engine.py.321: eval: epoch 109 | batch 250/510 [I 2022-10-11 15:32:34] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-11 15:32:34] ...trap/engines/engine.py.326: process: 0.32537 | load: 0.00023 [I 2022-10-11 15:32:37] ...trap/engines/engine.py.321: eval: epoch 109 | batch 260/510 [I 2022-10-11 15:32:37] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:19 [I 2022-10-11 15:32:37] ...trap/engines/engine.py.326: process: 0.31535 | load: 0.00022 [I 2022-10-11 15:32:40] ...trap/engines/engine.py.321: eval: epoch 109 | batch 270/510 [I 2022-10-11 15:32:40] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-11 15:32:40] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00021 [I 2022-10-11 15:32:43] ...trap/engines/engine.py.321: eval: epoch 109 | batch 280/510 [I 2022-10-11 15:32:43] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-11 15:32:43] ...trap/engines/engine.py.326: process: 0.32278 | load: 0.00021 [I 2022-10-11 15:32:47] ...trap/engines/engine.py.321: eval: epoch 109 | batch 290/510 [I 2022-10-11 15:32:47] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:10 [I 2022-10-11 15:32:47] ...trap/engines/engine.py.326: process: 0.32552 | load: 0.00021 [I 2022-10-11 15:32:50] ...trap/engines/engine.py.321: eval: epoch 109 | batch 300/510 [I 2022-10-11 15:32:50] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:07 [I 2022-10-11 15:32:50] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00022 [I 2022-10-11 15:32:53] ...trap/engines/engine.py.321: eval: epoch 109 | batch 310/510 [I 2022-10-11 15:32:53] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-11 15:32:53] ...trap/engines/engine.py.326: process: 0.32472 | load: 0.00022 [I 2022-10-11 15:32:56] ...trap/engines/engine.py.321: eval: epoch 109 | batch 320/510 [I 2022-10-11 15:32:56] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:01 [I 2022-10-11 15:32:56] ...trap/engines/engine.py.326: process: 0.32567 | load: 0.00023 [I 2022-10-11 15:33:00] ...trap/engines/engine.py.321: eval: epoch 109 | batch 330/510 [I 2022-10-11 15:33:00] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:57 [I 2022-10-11 15:33:00] ...trap/engines/engine.py.326: process: 0.31985 | load: 0.00021 [I 2022-10-11 15:33:03] ...trap/engines/engine.py.321: eval: epoch 109 | batch 340/510 [I 2022-10-11 15:33:03] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-11 15:33:03] ...trap/engines/engine.py.326: process: 0.31846 | load: 0.00021 [I 2022-10-11 15:33:06] ...trap/engines/engine.py.321: eval: epoch 109 | batch 350/510 [I 2022-10-11 15:33:06] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-11 15:33:06] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00022 [I 2022-10-11 15:33:09] ...trap/engines/engine.py.321: eval: epoch 109 | batch 360/510 [I 2022-10-11 15:33:09] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-11 15:33:09] ...trap/engines/engine.py.326: process: 0.32103 | load: 0.00022 [I 2022-10-11 15:33:12] ...trap/engines/engine.py.321: eval: epoch 109 | batch 370/510 [I 2022-10-11 15:33:12] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:45 [I 2022-10-11 15:33:12] ...trap/engines/engine.py.326: process: 0.31941 | load: 0.00021 [I 2022-10-11 15:33:16] ...trap/engines/engine.py.321: eval: epoch 109 | batch 380/510 [I 2022-10-11 15:33:16] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-11 15:33:16] ...trap/engines/engine.py.326: process: 0.32132 | load: 0.00022 [I 2022-10-11 15:33:19] ...trap/engines/engine.py.321: eval: epoch 109 | batch 390/510 [I 2022-10-11 15:33:19] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-11 15:33:19] ...trap/engines/engine.py.326: process: 0.32261 | load: 0.00021 [I 2022-10-11 15:33:22] ...trap/engines/engine.py.321: eval: epoch 109 | batch 400/510 [I 2022-10-11 15:33:22] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-11 15:33:22] ...trap/engines/engine.py.326: process: 0.32493 | load: 0.00022 [I 2022-10-11 15:33:25] ...trap/engines/engine.py.321: eval: epoch 109 | batch 410/510 [I 2022-10-11 15:33:25] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-11 15:33:25] ...trap/engines/engine.py.326: process: 0.32214 | load: 0.00021 [I 2022-10-11 15:33:28] ...trap/engines/engine.py.321: eval: epoch 109 | batch 420/510 [I 2022-10-11 15:33:28] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:28 [I 2022-10-11 15:33:28] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00021 [I 2022-10-11 15:33:32] ...trap/engines/engine.py.321: eval: epoch 109 | batch 430/510 [I 2022-10-11 15:33:32] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-11 15:33:32] ...trap/engines/engine.py.326: process: 0.32678 | load: 0.00022 [I 2022-10-11 15:33:35] ...trap/engines/engine.py.321: eval: epoch 109 | batch 440/510 [I 2022-10-11 15:33:35] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-11 15:33:35] ...trap/engines/engine.py.326: process: 0.32358 | load: 0.00021 [I 2022-10-11 15:33:38] ...trap/engines/engine.py.321: eval: epoch 109 | batch 450/510 [I 2022-10-11 15:33:38] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-11 15:33:38] ...trap/engines/engine.py.326: process: 0.31745 | load: 0.00021 [I 2022-10-11 15:33:41] ...trap/engines/engine.py.321: eval: epoch 109 | batch 460/510 [I 2022-10-11 15:33:41] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-11 15:33:41] ...trap/engines/engine.py.326: process: 0.32379 | load: 0.00023 [I 2022-10-11 15:33:44] ...trap/engines/engine.py.321: eval: epoch 109 | batch 470/510 [I 2022-10-11 15:33:44] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-11 15:33:44] ...trap/engines/engine.py.326: process: 0.32310 | load: 0.00021 [I 2022-10-11 15:33:48] ...trap/engines/engine.py.321: eval: epoch 109 | batch 480/510 [I 2022-10-11 15:33:48] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-11 15:33:48] ...trap/engines/engine.py.326: process: 0.32402 | load: 0.00021 [I 2022-10-11 15:33:51] ...trap/engines/engine.py.321: eval: epoch 109 | batch 490/510 [I 2022-10-11 15:33:51] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-11 15:33:51] ...trap/engines/engine.py.326: process: 0.32154 | load: 0.00021 [I 2022-10-11 15:33:54] ...trap/engines/engine.py.321: eval: epoch 109 | batch 500/510 [I 2022-10-11 15:33:54] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-11 15:33:54] ...trap/engines/engine.py.326: process: 0.31655 | load: 0.00022 [I 2022-10-11 15:33:57] ...trap/engines/engine.py.321: eval: epoch 109 | batch 510/510 [I 2022-10-11 15:33:57] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-11 15:33:57] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00017 [S 2022-10-11 15:33:58] ...trap/engines/engine.py.132: eval_epoch.epoch: 109 [I 2022-10-11 15:33:58] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.749900 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909400 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934800 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735600 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909800 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934200 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016902 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010229 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008483 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011612 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008195 [S 2022-10-11 15:34:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007960 [I 2022-10-11 15:34:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 15:34:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 15:34:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 15:34:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 15:34:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 15:34:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 15:34:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 15:34:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 15:34:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 110 [I 2022-10-11 15:34:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 15:34:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 0/2304 [I 2022-10-11 15:34:58] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:09:09 [I 2022-10-11 15:34:58] ...trap/engines/engine.py.229: process: 3.36333 | load: 2.40288 [I 2022-10-11 15:34:58] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-11 15:34:58] ...optimizers/trijoint.py.151: total_norm: 10.820123 [I 2022-10-11 15:35:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 10/2304 [I 2022-10-11 15:35:06] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:52 [I 2022-10-11 15:35:06] ...trap/engines/engine.py.229: process: 0.84548 | load: 0.00643 [I 2022-10-11 15:35:06] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-11 15:35:06] ...optimizers/trijoint.py.151: total_norm: 17.733500 [I 2022-10-11 15:35:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 20/2304 [I 2022-10-11 15:35:15] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:46 [I 2022-10-11 15:35:15] ...trap/engines/engine.py.229: process: 0.85373 | load: 0.00475 [I 2022-10-11 15:35:15] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-11 15:35:15] ...optimizers/trijoint.py.151: total_norm: 12.068673 [I 2022-10-11 15:35:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 30/2304 [I 2022-10-11 15:35:23] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:31 [I 2022-10-11 15:35:23] ...trap/engines/engine.py.229: process: 0.85710 | load: 0.00523 [I 2022-10-11 15:35:23] ...trap/engines/engine.py.230: loss: 0.11269 [I 2022-10-11 15:35:23] ...optimizers/trijoint.py.151: total_norm: 22.582277 [I 2022-10-11 15:35:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 40/2304 [I 2022-10-11 15:35:32] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:16 [I 2022-10-11 15:35:32] ...trap/engines/engine.py.229: process: 0.85868 | load: 0.00463 [I 2022-10-11 15:35:32] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-11 15:35:32] ...optimizers/trijoint.py.151: total_norm: 12.000497 [I 2022-10-11 15:35:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 50/2304 [I 2022-10-11 15:35:41] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:02 [I 2022-10-11 15:35:41] ...trap/engines/engine.py.229: process: 0.85341 | load: 0.00582 [I 2022-10-11 15:35:41] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-11 15:35:41] ...optimizers/trijoint.py.151: total_norm: 11.295815 [I 2022-10-11 15:35:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 60/2304 [I 2022-10-11 15:35:49] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:53 [I 2022-10-11 15:35:49] ...trap/engines/engine.py.229: process: 0.84493 | load: 0.00386 [I 2022-10-11 15:35:49] ...trap/engines/engine.py.230: loss: 0.12515 [I 2022-10-11 15:35:49] ...optimizers/trijoint.py.151: total_norm: 14.898965 [I 2022-10-11 15:35:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 70/2304 [I 2022-10-11 15:35:58] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:46 [I 2022-10-11 15:35:58] ...trap/engines/engine.py.229: process: 0.86594 | load: 0.00360 [I 2022-10-11 15:35:58] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-11 15:35:58] ...optimizers/trijoint.py.151: total_norm: 8.970990 [I 2022-10-11 15:36:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 80/2304 [I 2022-10-11 15:36:06] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:29 [I 2022-10-11 15:36:06] ...trap/engines/engine.py.229: process: 0.84561 | load: 0.00355 [I 2022-10-11 15:36:06] ...trap/engines/engine.py.230: loss: 0.14746 [I 2022-10-11 15:36:06] ...optimizers/trijoint.py.151: total_norm: 11.578680 [I 2022-10-11 15:36:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 90/2304 [I 2022-10-11 15:36:15] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:05 [I 2022-10-11 15:36:15] ...trap/engines/engine.py.229: process: 0.84376 | load: 0.00354 [I 2022-10-11 15:36:15] ...trap/engines/engine.py.230: loss: 0.13562 [I 2022-10-11 15:36:15] ...optimizers/trijoint.py.151: total_norm: 11.162048 [I 2022-10-11 15:36:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 100/2304 [I 2022-10-11 15:36:23] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:14 [I 2022-10-11 15:36:23] ...trap/engines/engine.py.229: process: 0.85258 | load: 0.00321 [I 2022-10-11 15:36:23] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-11 15:36:23] ...optimizers/trijoint.py.151: total_norm: 13.012463 [I 2022-10-11 15:36:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 110/2304 [I 2022-10-11 15:36:32] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:15 [I 2022-10-11 15:36:32] ...trap/engines/engine.py.229: process: 0.86286 | load: 0.00294 [I 2022-10-11 15:36:32] ...trap/engines/engine.py.230: loss: 0.08259 [I 2022-10-11 15:36:32] ...optimizers/trijoint.py.151: total_norm: 16.431040 [I 2022-10-11 15:36:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 120/2304 [I 2022-10-11 15:36:40] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:08 [I 2022-10-11 15:36:40] ...trap/engines/engine.py.229: process: 0.86589 | load: 0.00314 [I 2022-10-11 15:36:40] ...trap/engines/engine.py.230: loss: 0.22257 [I 2022-10-11 15:36:40] ...optimizers/trijoint.py.151: total_norm: 19.550543 [I 2022-10-11 15:36:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 130/2304 [I 2022-10-11 15:36:49] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:46 [I 2022-10-11 15:36:49] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00332 [I 2022-10-11 15:36:49] ...trap/engines/engine.py.230: loss: 0.09552 [I 2022-10-11 15:36:49] ...optimizers/trijoint.py.151: total_norm: 16.742699 [I 2022-10-11 15:36:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 140/2304 [I 2022-10-11 15:36:57] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:30:41 [I 2022-10-11 15:36:57] ...trap/engines/engine.py.229: process: 0.85574 | load: 0.00294 [I 2022-10-11 15:36:57] ...trap/engines/engine.py.230: loss: 0.18775 [I 2022-10-11 15:36:57] ...optimizers/trijoint.py.151: total_norm: 18.468302 [I 2022-10-11 15:37:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 150/2304 [I 2022-10-11 15:37:06] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:39 [I 2022-10-11 15:37:06] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00314 [I 2022-10-11 15:37:06] ...trap/engines/engine.py.230: loss: 0.13420 [I 2022-10-11 15:37:06] ...optimizers/trijoint.py.151: total_norm: 11.147709 [I 2022-10-11 15:37:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 160/2304 [I 2022-10-11 15:37:14] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:32 [I 2022-10-11 15:37:14] ...trap/engines/engine.py.229: process: 0.85935 | load: 0.00317 [I 2022-10-11 15:37:14] ...trap/engines/engine.py.230: loss: 0.12655 [I 2022-10-11 15:37:14] ...optimizers/trijoint.py.151: total_norm: 23.105904 [I 2022-10-11 15:37:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 170/2304 [I 2022-10-11 15:37:23] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:17 [I 2022-10-11 15:37:23] ...trap/engines/engine.py.229: process: 0.85497 | load: 0.00351 [I 2022-10-11 15:37:23] ...trap/engines/engine.py.230: loss: 0.11440 [I 2022-10-11 15:37:23] ...optimizers/trijoint.py.151: total_norm: 11.521346 [I 2022-10-11 15:37:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 180/2304 [I 2022-10-11 15:37:31] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:12 [I 2022-10-11 15:37:31] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00312 [I 2022-10-11 15:37:31] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-11 15:37:31] ...optimizers/trijoint.py.151: total_norm: 31.521723 [I 2022-10-11 15:37:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 190/2304 [I 2022-10-11 15:37:40] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:30:03 [I 2022-10-11 15:37:40] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00323 [I 2022-10-11 15:37:40] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-11 15:37:40] ...optimizers/trijoint.py.151: total_norm: 11.844834 [I 2022-10-11 15:37:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 200/2304 [I 2022-10-11 15:37:49] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:07 [I 2022-10-11 15:37:49] ...trap/engines/engine.py.229: process: 0.85702 | load: 0.00315 [I 2022-10-11 15:37:49] ...trap/engines/engine.py.230: loss: 0.09082 [I 2022-10-11 15:37:49] ...optimizers/trijoint.py.151: total_norm: 10.241278 [I 2022-10-11 15:37:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 210/2304 [I 2022-10-11 15:37:57] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:55 [I 2022-10-11 15:37:57] ...trap/engines/engine.py.229: process: 0.86132 | load: 0.00302 [I 2022-10-11 15:37:57] ...trap/engines/engine.py.230: loss: 0.08783 [I 2022-10-11 15:37:57] ...optimizers/trijoint.py.151: total_norm: 10.913979 [I 2022-10-11 15:38:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 220/2304 [I 2022-10-11 15:38:06] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:27 [I 2022-10-11 15:38:06] ...trap/engines/engine.py.229: process: 0.83201 | load: 0.00299 [I 2022-10-11 15:38:06] ...trap/engines/engine.py.230: loss: 0.09178 [I 2022-10-11 15:38:06] ...optimizers/trijoint.py.151: total_norm: 18.817402 [I 2022-10-11 15:38:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 230/2304 [I 2022-10-11 15:38:14] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:23 [I 2022-10-11 15:38:14] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00304 [I 2022-10-11 15:38:14] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-11 15:38:14] ...optimizers/trijoint.py.151: total_norm: 16.877735 [I 2022-10-11 15:38:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 240/2304 [I 2022-10-11 15:38:23] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:22 [I 2022-10-11 15:38:23] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00302 [I 2022-10-11 15:38:23] ...trap/engines/engine.py.230: loss: 0.24005 [I 2022-10-11 15:38:23] ...optimizers/trijoint.py.151: total_norm: 30.568403 [I 2022-10-11 15:38:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 250/2304 [I 2022-10-11 15:38:31] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:12 [I 2022-10-11 15:38:31] ...trap/engines/engine.py.229: process: 0.85231 | load: 0.00317 [I 2022-10-11 15:38:31] ...trap/engines/engine.py.230: loss: 0.20496 [I 2022-10-11 15:38:31] ...optimizers/trijoint.py.151: total_norm: 21.312113 [I 2022-10-11 15:38:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 260/2304 [I 2022-10-11 15:38:40] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:10 [I 2022-10-11 15:38:40] ...trap/engines/engine.py.229: process: 0.85141 | load: 0.00323 [I 2022-10-11 15:38:40] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-11 15:38:40] ...optimizers/trijoint.py.151: total_norm: 7.008710 [I 2022-10-11 15:38:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 270/2304 [I 2022-10-11 15:38:49] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:00 [I 2022-10-11 15:38:49] ...trap/engines/engine.py.229: process: 0.85650 | load: 0.00333 [I 2022-10-11 15:38:49] ...trap/engines/engine.py.230: loss: 0.10856 [I 2022-10-11 15:38:49] ...optimizers/trijoint.py.151: total_norm: 9.171930 [I 2022-10-11 15:38:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 280/2304 [I 2022-10-11 15:38:57] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:29:05 [I 2022-10-11 15:38:57] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00314 [I 2022-10-11 15:38:57] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-11 15:38:57] ...optimizers/trijoint.py.151: total_norm: 33.768589 [I 2022-10-11 15:39:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 290/2304 [I 2022-10-11 15:39:06] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:29:12 [I 2022-10-11 15:39:06] ...trap/engines/engine.py.229: process: 0.89343 | load: 0.00312 [I 2022-10-11 15:39:06] ...trap/engines/engine.py.230: loss: 0.09077 [I 2022-10-11 15:39:06] ...optimizers/trijoint.py.151: total_norm: 14.940001 [I 2022-10-11 15:39:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 300/2304 [I 2022-10-11 15:39:15] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:53 [I 2022-10-11 15:39:15] ...trap/engines/engine.py.229: process: 0.87314 | load: 0.00316 [I 2022-10-11 15:39:15] ...trap/engines/engine.py.230: loss: 0.09332 [I 2022-10-11 15:39:15] ...optimizers/trijoint.py.151: total_norm: 16.953577 [I 2022-10-11 15:39:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 310/2304 [I 2022-10-11 15:39:23] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:38 [I 2022-10-11 15:39:23] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00313 [I 2022-10-11 15:39:23] ...trap/engines/engine.py.230: loss: 0.07099 [I 2022-10-11 15:39:23] ...optimizers/trijoint.py.151: total_norm: 15.470665 [I 2022-10-11 15:39:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 320/2304 [I 2022-10-11 15:39:32] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:24 [I 2022-10-11 15:39:32] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00311 [I 2022-10-11 15:39:32] ...trap/engines/engine.py.230: loss: 0.11302 [I 2022-10-11 15:39:32] ...optimizers/trijoint.py.151: total_norm: 12.104653 [I 2022-10-11 15:39:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 330/2304 [I 2022-10-11 15:39:40] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:21 [I 2022-10-11 15:39:40] ...trap/engines/engine.py.229: process: 0.87811 | load: 0.00312 [I 2022-10-11 15:39:40] ...trap/engines/engine.py.230: loss: 0.20449 [I 2022-10-11 15:39:40] ...optimizers/trijoint.py.151: total_norm: 19.135899 [I 2022-10-11 15:39:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 340/2304 [I 2022-10-11 15:39:49] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:18 [I 2022-10-11 15:39:49] ...trap/engines/engine.py.229: process: 0.85097 | load: 0.00321 [I 2022-10-11 15:39:49] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-11 15:39:49] ...optimizers/trijoint.py.151: total_norm: 22.307329 [I 2022-10-11 15:39:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 350/2304 [I 2022-10-11 15:39:58] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:28:07 [I 2022-10-11 15:39:58] ...trap/engines/engine.py.229: process: 0.85313 | load: 0.00314 [I 2022-10-11 15:39:58] ...trap/engines/engine.py.230: loss: 0.25079 [I 2022-10-11 15:39:58] ...optimizers/trijoint.py.151: total_norm: 15.672800 [I 2022-10-11 15:40:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 360/2304 [I 2022-10-11 15:40:06] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:28:08 [I 2022-10-11 15:40:06] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00312 [I 2022-10-11 15:40:06] ...trap/engines/engine.py.230: loss: 0.07481 [I 2022-10-11 15:40:06] ...optimizers/trijoint.py.151: total_norm: 11.784652 [I 2022-10-11 15:40:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 370/2304 [I 2022-10-11 15:40:15] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:28:03 [I 2022-10-11 15:40:15] ...trap/engines/engine.py.229: process: 0.87959 | load: 0.00313 [I 2022-10-11 15:40:15] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-11 15:40:15] ...optimizers/trijoint.py.151: total_norm: 17.948669 [I 2022-10-11 15:40:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 380/2304 [I 2022-10-11 15:40:24] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:27:47 [I 2022-10-11 15:40:24] ...trap/engines/engine.py.229: process: 0.87918 | load: 0.00320 [I 2022-10-11 15:40:24] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-11 15:40:24] ...optimizers/trijoint.py.151: total_norm: 14.417325 [I 2022-10-11 15:40:33] ...trap/engines/engine.py.224: train: epoch 110 | batch 390/2304 [I 2022-10-11 15:40:33] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:27:50 [I 2022-10-11 15:40:33] ...trap/engines/engine.py.229: process: 0.86969 | load: 0.00329 [I 2022-10-11 15:40:33] ...trap/engines/engine.py.230: loss: 0.09523 [I 2022-10-11 15:40:33] ...optimizers/trijoint.py.151: total_norm: 8.295447 [I 2022-10-11 15:40:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 400/2304 [I 2022-10-11 15:40:41] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:27:31 [I 2022-10-11 15:40:41] ...trap/engines/engine.py.229: process: 0.87462 | load: 0.00314 [I 2022-10-11 15:40:41] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-11 15:40:41] ...optimizers/trijoint.py.151: total_norm: 12.568612 [I 2022-10-11 15:40:50] ...trap/engines/engine.py.224: train: epoch 110 | batch 410/2304 [I 2022-10-11 15:40:50] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:27:37 [I 2022-10-11 15:40:50] ...trap/engines/engine.py.229: process: 0.87513 | load: 0.00315 [I 2022-10-11 15:40:50] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-11 15:40:50] ...optimizers/trijoint.py.151: total_norm: 10.509412 [I 2022-10-11 15:40:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 420/2304 [I 2022-10-11 15:40:59] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:27:06 [I 2022-10-11 15:40:59] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00316 [I 2022-10-11 15:40:59] ...trap/engines/engine.py.230: loss: 0.07700 [I 2022-10-11 15:40:59] ...optimizers/trijoint.py.151: total_norm: 12.093528 [I 2022-10-11 15:41:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 430/2304 [I 2022-10-11 15:41:07] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:27:27 [I 2022-10-11 15:41:07] ...trap/engines/engine.py.229: process: 0.89009 | load: 0.00393 [I 2022-10-11 15:41:07] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-11 15:41:07] ...optimizers/trijoint.py.151: total_norm: 17.782574 [I 2022-10-11 15:41:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 440/2304 [I 2022-10-11 15:41:16] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:27:25 [I 2022-10-11 15:41:16] ...trap/engines/engine.py.229: process: 0.87788 | load: 0.00400 [I 2022-10-11 15:41:16] ...trap/engines/engine.py.230: loss: 0.16839 [I 2022-10-11 15:41:16] ...optimizers/trijoint.py.151: total_norm: 20.347670 [I 2022-10-11 15:41:25] ...trap/engines/engine.py.224: train: epoch 110 | batch 450/2304 [I 2022-10-11 15:41:25] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:27:26 [I 2022-10-11 15:41:25] ...trap/engines/engine.py.229: process: 0.89777 | load: 0.00393 [I 2022-10-11 15:41:25] ...trap/engines/engine.py.230: loss: 0.15799 [I 2022-10-11 15:41:25] ...optimizers/trijoint.py.151: total_norm: 10.402053 [I 2022-10-11 15:41:34] ...trap/engines/engine.py.224: train: epoch 110 | batch 460/2304 [I 2022-10-11 15:41:34] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:27:18 [I 2022-10-11 15:41:34] ...trap/engines/engine.py.229: process: 0.88841 | load: 0.00392 [I 2022-10-11 15:41:34] ...trap/engines/engine.py.230: loss: 0.08089 [I 2022-10-11 15:41:34] ...optimizers/trijoint.py.151: total_norm: 12.686050 [I 2022-10-11 15:41:43] ...trap/engines/engine.py.224: train: epoch 110 | batch 470/2304 [I 2022-10-11 15:41:43] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:27:06 [I 2022-10-11 15:41:43] ...trap/engines/engine.py.229: process: 0.88270 | load: 0.00394 [I 2022-10-11 15:41:43] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-11 15:41:43] ...optimizers/trijoint.py.151: total_norm: 9.707964 [I 2022-10-11 15:41:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 480/2304 [I 2022-10-11 15:41:52] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:27:04 [I 2022-10-11 15:41:52] ...trap/engines/engine.py.229: process: 0.89294 | load: 0.00394 [I 2022-10-11 15:41:52] ...trap/engines/engine.py.230: loss: 0.09649 [I 2022-10-11 15:41:52] ...optimizers/trijoint.py.151: total_norm: 13.129945 [I 2022-10-11 15:42:01] ...trap/engines/engine.py.224: train: epoch 110 | batch 490/2304 [I 2022-10-11 15:42:01] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:26:52 [I 2022-10-11 15:42:01] ...trap/engines/engine.py.229: process: 0.88224 | load: 0.00393 [I 2022-10-11 15:42:01] ...trap/engines/engine.py.230: loss: 0.15329 [I 2022-10-11 15:42:01] ...optimizers/trijoint.py.151: total_norm: 20.451992 [I 2022-10-11 15:42:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 500/2304 [I 2022-10-11 15:42:10] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:26:50 [I 2022-10-11 15:42:10] ...trap/engines/engine.py.229: process: 0.89427 | load: 0.00392 [I 2022-10-11 15:42:10] ...trap/engines/engine.py.230: loss: 0.11221 [I 2022-10-11 15:42:10] ...optimizers/trijoint.py.151: total_norm: 36.097649 [I 2022-10-11 15:42:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 510/2304 [I 2022-10-11 15:42:19] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:26:38 [I 2022-10-11 15:42:19] ...trap/engines/engine.py.229: process: 0.90633 | load: 0.00393 [I 2022-10-11 15:42:19] ...trap/engines/engine.py.230: loss: 0.10215 [I 2022-10-11 15:42:19] ...optimizers/trijoint.py.151: total_norm: 10.081822 [I 2022-10-11 15:42:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 520/2304 [I 2022-10-11 15:42:28] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:26:21 [I 2022-10-11 15:42:28] ...trap/engines/engine.py.229: process: 0.88138 | load: 0.00392 [I 2022-10-11 15:42:28] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-11 15:42:28] ...optimizers/trijoint.py.151: total_norm: 18.447569 [I 2022-10-11 15:42:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 530/2304 [I 2022-10-11 15:42:36] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:25:46 [I 2022-10-11 15:42:36] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00312 [I 2022-10-11 15:42:36] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-11 15:42:36] ...optimizers/trijoint.py.151: total_norm: 10.765556 [I 2022-10-11 15:42:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 540/2304 [I 2022-10-11 15:42:45] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:48 [I 2022-10-11 15:42:45] ...trap/engines/engine.py.229: process: 0.84146 | load: 0.00295 [I 2022-10-11 15:42:45] ...trap/engines/engine.py.230: loss: 0.08219 [I 2022-10-11 15:42:45] ...optimizers/trijoint.py.151: total_norm: 11.749964 [I 2022-10-11 15:42:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 550/2304 [I 2022-10-11 15:42:53] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:24:54 [I 2022-10-11 15:42:53] ...trap/engines/engine.py.229: process: 0.86307 | load: 0.00295 [I 2022-10-11 15:42:53] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-11 15:42:53] ...optimizers/trijoint.py.151: total_norm: 20.792870 [I 2022-10-11 15:43:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 560/2304 [I 2022-10-11 15:43:02] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:24:35 [I 2022-10-11 15:43:02] ...trap/engines/engine.py.229: process: 0.86002 | load: 0.00305 [I 2022-10-11 15:43:02] ...trap/engines/engine.py.230: loss: 0.10803 [I 2022-10-11 15:43:02] ...optimizers/trijoint.py.151: total_norm: 24.885391 [I 2022-10-11 15:43:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 570/2304 [I 2022-10-11 15:43:10] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:25 [I 2022-10-11 15:43:10] ...trap/engines/engine.py.229: process: 0.84317 | load: 0.00295 [I 2022-10-11 15:43:10] ...trap/engines/engine.py.230: loss: 0.13596 [I 2022-10-11 15:43:10] ...optimizers/trijoint.py.151: total_norm: 13.712025 [I 2022-10-11 15:43:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 580/2304 [I 2022-10-11 15:43:19] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:24:22 [I 2022-10-11 15:43:19] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00292 [I 2022-10-11 15:43:19] ...trap/engines/engine.py.230: loss: 0.09058 [I 2022-10-11 15:43:19] ...optimizers/trijoint.py.151: total_norm: 12.935055 [I 2022-10-11 15:43:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 590/2304 [I 2022-10-11 15:43:27] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:16 [I 2022-10-11 15:43:27] ...trap/engines/engine.py.229: process: 0.85519 | load: 0.00295 [I 2022-10-11 15:43:27] ...trap/engines/engine.py.230: loss: 0.08943 [I 2022-10-11 15:43:27] ...optimizers/trijoint.py.151: total_norm: 9.354460 [I 2022-10-11 15:43:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 600/2304 [I 2022-10-11 15:43:36] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:23:54 [I 2022-10-11 15:43:36] ...trap/engines/engine.py.229: process: 0.83961 | load: 0.00296 [I 2022-10-11 15:43:36] ...trap/engines/engine.py.230: loss: 0.17900 [I 2022-10-11 15:43:36] ...optimizers/trijoint.py.151: total_norm: 11.113502 [I 2022-10-11 15:43:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 610/2304 [I 2022-10-11 15:43:44] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:23:57 [I 2022-10-11 15:43:44] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00294 [I 2022-10-11 15:43:44] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-11 15:43:44] ...optimizers/trijoint.py.151: total_norm: 18.299221 [I 2022-10-11 15:43:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 620/2304 [I 2022-10-11 15:43:53] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:42 [I 2022-10-11 15:43:53] ...trap/engines/engine.py.229: process: 0.84646 | load: 0.00295 [I 2022-10-11 15:43:53] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-11 15:43:53] ...optimizers/trijoint.py.151: total_norm: 30.647167 [I 2022-10-11 15:44:01] ...trap/engines/engine.py.224: train: epoch 110 | batch 630/2304 [I 2022-10-11 15:44:01] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:31 [I 2022-10-11 15:44:01] ...trap/engines/engine.py.229: process: 0.84055 | load: 0.00295 [I 2022-10-11 15:44:01] ...trap/engines/engine.py.230: loss: 0.14830 [I 2022-10-11 15:44:01] ...optimizers/trijoint.py.151: total_norm: 26.509825 [I 2022-10-11 15:44:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 640/2304 [I 2022-10-11 15:44:10] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:35 [I 2022-10-11 15:44:10] ...trap/engines/engine.py.229: process: 0.86254 | load: 0.00297 [I 2022-10-11 15:44:10] ...trap/engines/engine.py.230: loss: 0.11278 [I 2022-10-11 15:44:10] ...optimizers/trijoint.py.151: total_norm: 24.878151 [I 2022-10-11 15:44:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 650/2304 [I 2022-10-11 15:44:18] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:23 [I 2022-10-11 15:44:18] ...trap/engines/engine.py.229: process: 0.83565 | load: 0.00299 [I 2022-10-11 15:44:18] ...trap/engines/engine.py.230: loss: 0.08136 [I 2022-10-11 15:44:18] ...optimizers/trijoint.py.151: total_norm: 13.558695 [I 2022-10-11 15:44:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 660/2304 [I 2022-10-11 15:44:27] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:47 [I 2022-10-11 15:44:27] ...trap/engines/engine.py.229: process: 0.88222 | load: 0.00380 [I 2022-10-11 15:44:27] ...trap/engines/engine.py.230: loss: 0.19432 [I 2022-10-11 15:44:27] ...optimizers/trijoint.py.151: total_norm: 15.118224 [I 2022-10-11 15:44:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 670/2304 [I 2022-10-11 15:44:36] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:24:05 [I 2022-10-11 15:44:36] ...trap/engines/engine.py.229: process: 0.89921 | load: 0.00390 [I 2022-10-11 15:44:36] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-11 15:44:36] ...optimizers/trijoint.py.151: total_norm: 11.727274 [I 2022-10-11 15:44:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 680/2304 [I 2022-10-11 15:44:44] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:23:47 [I 2022-10-11 15:44:44] ...trap/engines/engine.py.229: process: 0.86764 | load: 0.00378 [I 2022-10-11 15:44:44] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-11 15:44:44] ...optimizers/trijoint.py.151: total_norm: 12.253704 [I 2022-10-11 15:44:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 690/2304 [I 2022-10-11 15:44:53] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:29 [I 2022-10-11 15:44:53] ...trap/engines/engine.py.229: process: 0.86530 | load: 0.00378 [I 2022-10-11 15:44:53] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-11 15:44:53] ...optimizers/trijoint.py.151: total_norm: 16.219048 [I 2022-10-11 15:45:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 700/2304 [I 2022-10-11 15:45:02] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:23:23 [I 2022-10-11 15:45:02] ...trap/engines/engine.py.229: process: 0.87114 | load: 0.00377 [I 2022-10-11 15:45:02] ...trap/engines/engine.py.230: loss: 0.09132 [I 2022-10-11 15:45:02] ...optimizers/trijoint.py.151: total_norm: 9.967523 [I 2022-10-11 15:45:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 710/2304 [I 2022-10-11 15:45:11] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:23:11 [I 2022-10-11 15:45:11] ...trap/engines/engine.py.229: process: 0.87451 | load: 0.00378 [I 2022-10-11 15:45:11] ...trap/engines/engine.py.230: loss: 0.09487 [I 2022-10-11 15:45:11] ...optimizers/trijoint.py.151: total_norm: 26.721769 [I 2022-10-11 15:45:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 720/2304 [I 2022-10-11 15:45:20] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:23:10 [I 2022-10-11 15:45:20] ...trap/engines/engine.py.229: process: 0.88655 | load: 0.00377 [I 2022-10-11 15:45:20] ...trap/engines/engine.py.230: loss: 0.18904 [I 2022-10-11 15:45:20] ...optimizers/trijoint.py.151: total_norm: 31.441666 [I 2022-10-11 15:45:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 730/2304 [I 2022-10-11 15:45:28] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:57 [I 2022-10-11 15:45:28] ...trap/engines/engine.py.229: process: 0.87301 | load: 0.00379 [I 2022-10-11 15:45:28] ...trap/engines/engine.py.230: loss: 0.09087 [I 2022-10-11 15:45:28] ...optimizers/trijoint.py.151: total_norm: 17.576723 [I 2022-10-11 15:45:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 740/2304 [I 2022-10-11 15:45:37] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:48 [I 2022-10-11 15:45:37] ...trap/engines/engine.py.229: process: 0.88740 | load: 0.00376 [I 2022-10-11 15:45:37] ...trap/engines/engine.py.230: loss: 0.13271 [I 2022-10-11 15:45:37] ...optimizers/trijoint.py.151: total_norm: 18.241186 [I 2022-10-11 15:45:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 750/2304 [I 2022-10-11 15:45:46] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:25 [I 2022-10-11 15:45:46] ...trap/engines/engine.py.229: process: 0.87221 | load: 0.00311 [I 2022-10-11 15:45:46] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-11 15:45:46] ...optimizers/trijoint.py.151: total_norm: 10.960976 [I 2022-10-11 15:45:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 760/2304 [I 2022-10-11 15:45:54] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:22:06 [I 2022-10-11 15:45:54] ...trap/engines/engine.py.229: process: 0.85559 | load: 0.00318 [I 2022-10-11 15:45:54] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-11 15:45:54] ...optimizers/trijoint.py.151: total_norm: 14.815757 [I 2022-10-11 15:46:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 770/2304 [I 2022-10-11 15:46:03] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:21:55 [I 2022-10-11 15:46:03] ...trap/engines/engine.py.229: process: 0.85053 | load: 0.00316 [I 2022-10-11 15:46:03] ...trap/engines/engine.py.230: loss: 0.08305 [I 2022-10-11 15:46:03] ...optimizers/trijoint.py.151: total_norm: 8.916035 [I 2022-10-11 15:46:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 780/2304 [I 2022-10-11 15:46:12] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:42 [I 2022-10-11 15:46:12] ...trap/engines/engine.py.229: process: 0.85630 | load: 0.00339 [I 2022-10-11 15:46:12] ...trap/engines/engine.py.230: loss: 0.09221 [I 2022-10-11 15:46:12] ...optimizers/trijoint.py.151: total_norm: 9.273265 [I 2022-10-11 15:46:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 790/2304 [I 2022-10-11 15:46:20] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:21:38 [I 2022-10-11 15:46:20] ...trap/engines/engine.py.229: process: 0.86260 | load: 0.00312 [I 2022-10-11 15:46:20] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-11 15:46:20] ...optimizers/trijoint.py.151: total_norm: 13.940330 [I 2022-10-11 15:46:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 800/2304 [I 2022-10-11 15:46:29] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:24 [I 2022-10-11 15:46:29] ...trap/engines/engine.py.229: process: 0.85873 | load: 0.00311 [I 2022-10-11 15:46:29] ...trap/engines/engine.py.230: loss: 0.13987 [I 2022-10-11 15:46:29] ...optimizers/trijoint.py.151: total_norm: 8.900986 [I 2022-10-11 15:46:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 810/2304 [I 2022-10-11 15:46:37] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:17 [I 2022-10-11 15:46:37] ...trap/engines/engine.py.229: process: 0.85764 | load: 0.00310 [I 2022-10-11 15:46:37] ...trap/engines/engine.py.230: loss: 0.16176 [I 2022-10-11 15:46:37] ...optimizers/trijoint.py.151: total_norm: 17.636269 [I 2022-10-11 15:46:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 820/2304 [I 2022-10-11 15:46:46] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:21:13 [I 2022-10-11 15:46:46] ...trap/engines/engine.py.229: process: 0.86399 | load: 0.00324 [I 2022-10-11 15:46:46] ...trap/engines/engine.py.230: loss: 0.14397 [I 2022-10-11 15:46:46] ...optimizers/trijoint.py.151: total_norm: 10.630713 [I 2022-10-11 15:46:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 830/2304 [I 2022-10-11 15:46:54] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:10 [I 2022-10-11 15:46:54] ...trap/engines/engine.py.229: process: 0.89629 | load: 0.00312 [I 2022-10-11 15:46:54] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-11 15:46:54] ...optimizers/trijoint.py.151: total_norm: 9.438057 [I 2022-10-11 15:47:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 840/2304 [I 2022-10-11 15:47:03] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:00 [I 2022-10-11 15:47:03] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00313 [I 2022-10-11 15:47:03] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-11 15:47:03] ...optimizers/trijoint.py.151: total_norm: 9.055186 [I 2022-10-11 15:47:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 850/2304 [I 2022-10-11 15:47:12] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:50 [I 2022-10-11 15:47:12] ...trap/engines/engine.py.229: process: 0.88913 | load: 0.00315 [I 2022-10-11 15:47:12] ...trap/engines/engine.py.230: loss: 0.13273 [I 2022-10-11 15:47:12] ...optimizers/trijoint.py.151: total_norm: 12.631688 [I 2022-10-11 15:47:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 860/2304 [I 2022-10-11 15:47:20] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:20:45 [I 2022-10-11 15:47:20] ...trap/engines/engine.py.229: process: 0.87221 | load: 0.00312 [I 2022-10-11 15:47:20] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-11 15:47:20] ...optimizers/trijoint.py.151: total_norm: 16.116085 [I 2022-10-11 15:47:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 870/2304 [I 2022-10-11 15:47:29] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:17 [I 2022-10-11 15:47:29] ...trap/engines/engine.py.229: process: 0.84628 | load: 0.00313 [I 2022-10-11 15:47:29] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-11 15:47:29] ...optimizers/trijoint.py.151: total_norm: 17.358599 [I 2022-10-11 15:47:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 880/2304 [I 2022-10-11 15:47:37] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:20:13 [I 2022-10-11 15:47:37] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00310 [I 2022-10-11 15:47:37] ...trap/engines/engine.py.230: loss: 0.09900 [I 2022-10-11 15:47:37] ...optimizers/trijoint.py.151: total_norm: 15.038709 [I 2022-10-11 15:47:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 890/2304 [I 2022-10-11 15:47:46] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:20:11 [I 2022-10-11 15:47:46] ...trap/engines/engine.py.229: process: 0.86790 | load: 0.00312 [I 2022-10-11 15:47:46] ...trap/engines/engine.py.230: loss: 0.16930 [I 2022-10-11 15:47:46] ...optimizers/trijoint.py.151: total_norm: 27.183559 [I 2022-10-11 15:47:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 900/2304 [I 2022-10-11 15:47:54] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:20:03 [I 2022-10-11 15:47:54] ...trap/engines/engine.py.229: process: 0.87131 | load: 0.00311 [I 2022-10-11 15:47:54] ...trap/engines/engine.py.230: loss: 0.14296 [I 2022-10-11 15:47:54] ...optimizers/trijoint.py.151: total_norm: 15.081243 [I 2022-10-11 15:48:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 910/2304 [I 2022-10-11 15:48:03] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:47 [I 2022-10-11 15:48:03] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00313 [I 2022-10-11 15:48:03] ...trap/engines/engine.py.230: loss: 0.10205 [I 2022-10-11 15:48:03] ...optimizers/trijoint.py.151: total_norm: 15.635466 [I 2022-10-11 15:48:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 920/2304 [I 2022-10-11 15:48:12] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:51 [I 2022-10-11 15:48:12] ...trap/engines/engine.py.229: process: 0.87960 | load: 0.00311 [I 2022-10-11 15:48:12] ...trap/engines/engine.py.230: loss: 0.09557 [I 2022-10-11 15:48:12] ...optimizers/trijoint.py.151: total_norm: 11.985493 [I 2022-10-11 15:48:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 930/2304 [I 2022-10-11 15:48:20] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:31 [I 2022-10-11 15:48:20] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00311 [I 2022-10-11 15:48:20] ...trap/engines/engine.py.230: loss: 0.18880 [I 2022-10-11 15:48:20] ...optimizers/trijoint.py.151: total_norm: 35.186485 [I 2022-10-11 15:48:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 940/2304 [I 2022-10-11 15:48:29] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:22 [I 2022-10-11 15:48:29] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00313 [I 2022-10-11 15:48:29] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-11 15:48:29] ...optimizers/trijoint.py.151: total_norm: 16.064484 [I 2022-10-11 15:48:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 950/2304 [I 2022-10-11 15:48:37] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:50 [I 2022-10-11 15:48:37] ...trap/engines/engine.py.229: process: 0.89846 | load: 0.00393 [I 2022-10-11 15:48:37] ...trap/engines/engine.py.230: loss: 0.11187 [I 2022-10-11 15:48:37] ...optimizers/trijoint.py.151: total_norm: 14.615176 [I 2022-10-11 15:48:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 960/2304 [I 2022-10-11 15:48:46] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:19:52 [I 2022-10-11 15:48:46] ...trap/engines/engine.py.229: process: 0.89364 | load: 0.00394 [I 2022-10-11 15:48:46] ...trap/engines/engine.py.230: loss: 0.09499 [I 2022-10-11 15:48:46] ...optimizers/trijoint.py.151: total_norm: 9.814168 [I 2022-10-11 15:48:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 970/2304 [I 2022-10-11 15:48:55] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:19:41 [I 2022-10-11 15:48:55] ...trap/engines/engine.py.229: process: 0.88194 | load: 0.00396 [I 2022-10-11 15:48:55] ...trap/engines/engine.py.230: loss: 0.18066 [I 2022-10-11 15:48:55] ...optimizers/trijoint.py.151: total_norm: 24.213224 [I 2022-10-11 15:49:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 980/2304 [I 2022-10-11 15:49:04] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:19:29 [I 2022-10-11 15:49:04] ...trap/engines/engine.py.229: process: 0.86774 | load: 0.00396 [I 2022-10-11 15:49:04] ...trap/engines/engine.py.230: loss: 0.13401 [I 2022-10-11 15:49:04] ...optimizers/trijoint.py.151: total_norm: 11.478723 [I 2022-10-11 15:49:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 990/2304 [I 2022-10-11 15:49:13] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:19:25 [I 2022-10-11 15:49:13] ...trap/engines/engine.py.229: process: 0.89012 | load: 0.00396 [I 2022-10-11 15:49:13] ...trap/engines/engine.py.230: loss: 0.08752 [I 2022-10-11 15:49:13] ...optimizers/trijoint.py.151: total_norm: 11.557669 [I 2022-10-11 15:49:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1000/2304 [I 2022-10-11 15:49:22] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:53 [I 2022-10-11 15:49:22] ...trap/engines/engine.py.229: process: 0.85363 | load: 0.00314 [I 2022-10-11 15:49:22] ...trap/engines/engine.py.230: loss: 0.11712 [I 2022-10-11 15:49:22] ...optimizers/trijoint.py.151: total_norm: 14.195786 [I 2022-10-11 15:49:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 1010/2304 [I 2022-10-11 15:49:30] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:24 [I 2022-10-11 15:49:30] ...trap/engines/engine.py.229: process: 0.84418 | load: 0.00314 [I 2022-10-11 15:49:30] ...trap/engines/engine.py.230: loss: 0.12730 [I 2022-10-11 15:49:30] ...optimizers/trijoint.py.151: total_norm: 28.413244 [I 2022-10-11 15:49:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1020/2304 [I 2022-10-11 15:49:39] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:05 [I 2022-10-11 15:49:39] ...trap/engines/engine.py.229: process: 0.83866 | load: 0.00297 [I 2022-10-11 15:49:39] ...trap/engines/engine.py.230: loss: 0.13058 [I 2022-10-11 15:49:39] ...optimizers/trijoint.py.151: total_norm: 13.674106 [I 2022-10-11 15:49:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 1030/2304 [I 2022-10-11 15:49:47] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:18:04 [I 2022-10-11 15:49:47] ...trap/engines/engine.py.229: process: 0.85770 | load: 0.00303 [I 2022-10-11 15:49:47] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-11 15:49:47] ...optimizers/trijoint.py.151: total_norm: 21.454454 [I 2022-10-11 15:49:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 1040/2304 [I 2022-10-11 15:49:56] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:17:48 [I 2022-10-11 15:49:56] ...trap/engines/engine.py.229: process: 0.84906 | load: 0.00294 [I 2022-10-11 15:49:56] ...trap/engines/engine.py.230: loss: 0.08855 [I 2022-10-11 15:49:56] ...optimizers/trijoint.py.151: total_norm: 20.337791 [I 2022-10-11 15:50:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 1050/2304 [I 2022-10-11 15:50:04] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:37 [I 2022-10-11 15:50:04] ...trap/engines/engine.py.229: process: 0.83741 | load: 0.00295 [I 2022-10-11 15:50:04] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-11 15:50:04] ...optimizers/trijoint.py.151: total_norm: 12.187373 [I 2022-10-11 15:50:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 1060/2304 [I 2022-10-11 15:50:13] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:17:25 [I 2022-10-11 15:50:13] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00296 [I 2022-10-11 15:50:13] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-11 15:50:13] ...optimizers/trijoint.py.151: total_norm: 23.688105 [I 2022-10-11 15:50:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 1070/2304 [I 2022-10-11 15:50:21] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:23 [I 2022-10-11 15:50:21] ...trap/engines/engine.py.229: process: 0.84365 | load: 0.00298 [I 2022-10-11 15:50:21] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-11 15:50:21] ...optimizers/trijoint.py.151: total_norm: 8.629354 [I 2022-10-11 15:50:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 1080/2304 [I 2022-10-11 15:50:29] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:17:06 [I 2022-10-11 15:50:29] ...trap/engines/engine.py.229: process: 0.84773 | load: 0.00297 [I 2022-10-11 15:50:29] ...trap/engines/engine.py.230: loss: 0.10395 [I 2022-10-11 15:50:29] ...optimizers/trijoint.py.151: total_norm: 18.261431 [I 2022-10-11 15:50:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1090/2304 [I 2022-10-11 15:50:38] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:17:09 [I 2022-10-11 15:50:38] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00295 [I 2022-10-11 15:50:38] ...trap/engines/engine.py.230: loss: 0.09068 [I 2022-10-11 15:50:38] ...optimizers/trijoint.py.151: total_norm: 10.927827 [I 2022-10-11 15:50:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 1100/2304 [I 2022-10-11 15:50:47] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:17:00 [I 2022-10-11 15:50:47] ...trap/engines/engine.py.229: process: 0.85209 | load: 0.00294 [I 2022-10-11 15:50:47] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-11 15:50:47] ...optimizers/trijoint.py.151: total_norm: 15.872519 [I 2022-10-11 15:50:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1110/2304 [I 2022-10-11 15:50:55] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:49 [I 2022-10-11 15:50:55] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00294 [I 2022-10-11 15:50:55] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-11 15:50:55] ...optimizers/trijoint.py.151: total_norm: 22.304962 [I 2022-10-11 15:51:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 1120/2304 [I 2022-10-11 15:51:03] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:16:43 [I 2022-10-11 15:51:03] ...trap/engines/engine.py.229: process: 0.85735 | load: 0.00293 [I 2022-10-11 15:51:03] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-11 15:51:03] ...optimizers/trijoint.py.151: total_norm: 14.776411 [I 2022-10-11 15:51:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 1130/2304 [I 2022-10-11 15:51:12] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:16:40 [I 2022-10-11 15:51:12] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00297 [I 2022-10-11 15:51:12] ...trap/engines/engine.py.230: loss: 0.09231 [I 2022-10-11 15:51:12] ...optimizers/trijoint.py.151: total_norm: 17.000320 [I 2022-10-11 15:51:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 1140/2304 [I 2022-10-11 15:51:21] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:16:31 [I 2022-10-11 15:51:21] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00300 [I 2022-10-11 15:51:21] ...trap/engines/engine.py.230: loss: 0.10572 [I 2022-10-11 15:51:21] ...optimizers/trijoint.py.151: total_norm: 14.086550 [I 2022-10-11 15:51:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 1150/2304 [I 2022-10-11 15:51:29] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:16:25 [I 2022-10-11 15:51:29] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00303 [I 2022-10-11 15:51:29] ...trap/engines/engine.py.230: loss: 0.10989 [I 2022-10-11 15:51:29] ...optimizers/trijoint.py.151: total_norm: 18.331741 [I 2022-10-11 15:51:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1160/2304 [I 2022-10-11 15:51:38] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:16:16 [I 2022-10-11 15:51:38] ...trap/engines/engine.py.229: process: 0.86223 | load: 0.00301 [I 2022-10-11 15:51:38] ...trap/engines/engine.py.230: loss: 0.18542 [I 2022-10-11 15:51:38] ...optimizers/trijoint.py.151: total_norm: 29.760368 [I 2022-10-11 15:51:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 1170/2304 [I 2022-10-11 15:51:46] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:02 [I 2022-10-11 15:51:46] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00327 [I 2022-10-11 15:51:46] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-11 15:51:46] ...optimizers/trijoint.py.151: total_norm: 8.688139 [I 2022-10-11 15:51:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1180/2304 [I 2022-10-11 15:51:55] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:15:53 [I 2022-10-11 15:51:55] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00299 [I 2022-10-11 15:51:55] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-11 15:51:55] ...optimizers/trijoint.py.151: total_norm: 20.245714 [I 2022-10-11 15:52:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 1190/2304 [I 2022-10-11 15:52:03] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:51 [I 2022-10-11 15:52:03] ...trap/engines/engine.py.229: process: 0.86255 | load: 0.00297 [I 2022-10-11 15:52:03] ...trap/engines/engine.py.230: loss: 0.15209 [I 2022-10-11 15:52:03] ...optimizers/trijoint.py.151: total_norm: 14.832590 [I 2022-10-11 15:52:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 1200/2304 [I 2022-10-11 15:52:30] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:24:38 [I 2022-10-11 15:52:30] ...trap/engines/engine.py.229: process: 0.84077 | load: 0.00301 [I 2022-10-11 15:52:30] ...trap/engines/engine.py.230: loss: 0.09616 [I 2022-10-11 15:52:30] ...optimizers/trijoint.py.151: total_norm: 16.812523 [I 2022-10-11 15:52:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1210/2304 [I 2022-10-11 15:52:39] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:16:31 [I 2022-10-11 15:52:39] ...trap/engines/engine.py.229: process: 0.85993 | load: 0.00310 [I 2022-10-11 15:52:39] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 15:52:39] ...optimizers/trijoint.py.151: total_norm: 12.238762 [I 2022-10-11 15:52:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 1220/2304 [I 2022-10-11 15:52:47] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:35 [I 2022-10-11 15:52:47] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00314 [I 2022-10-11 15:52:47] ...trap/engines/engine.py.230: loss: 0.12151 [I 2022-10-11 15:52:47] ...optimizers/trijoint.py.151: total_norm: 13.794413 [I 2022-10-11 15:52:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 1230/2304 [I 2022-10-11 15:52:56] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:15:18 [I 2022-10-11 15:52:56] ...trap/engines/engine.py.229: process: 0.85917 | load: 0.00312 [I 2022-10-11 15:52:56] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-11 15:52:56] ...optimizers/trijoint.py.151: total_norm: 12.831759 [I 2022-10-11 15:53:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 1240/2304 [I 2022-10-11 15:53:04] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:15:17 [I 2022-10-11 15:53:04] ...trap/engines/engine.py.229: process: 0.87208 | load: 0.00317 [I 2022-10-11 15:53:04] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-11 15:53:04] ...optimizers/trijoint.py.151: total_norm: 13.599902 [I 2022-10-11 15:53:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 1250/2304 [I 2022-10-11 15:53:13] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:15:06 [I 2022-10-11 15:53:13] ...trap/engines/engine.py.229: process: 0.87005 | load: 0.00314 [I 2022-10-11 15:53:13] ...trap/engines/engine.py.230: loss: 0.09440 [I 2022-10-11 15:53:13] ...optimizers/trijoint.py.151: total_norm: 13.278683 [I 2022-10-11 15:53:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1260/2304 [I 2022-10-11 15:53:22] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:15:04 [I 2022-10-11 15:53:22] ...trap/engines/engine.py.229: process: 0.87469 | load: 0.00314 [I 2022-10-11 15:53:22] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-11 15:53:22] ...optimizers/trijoint.py.151: total_norm: 13.948570 [I 2022-10-11 15:53:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 1270/2304 [I 2022-10-11 15:53:30] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:46 [I 2022-10-11 15:53:30] ...trap/engines/engine.py.229: process: 0.84321 | load: 0.00310 [I 2022-10-11 15:53:30] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-11 15:53:30] ...optimizers/trijoint.py.151: total_norm: 12.883705 [I 2022-10-11 15:53:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1280/2304 [I 2022-10-11 15:53:39] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:14:47 [I 2022-10-11 15:53:39] ...trap/engines/engine.py.229: process: 0.87723 | load: 0.00320 [I 2022-10-11 15:53:39] ...trap/engines/engine.py.230: loss: 0.10596 [I 2022-10-11 15:53:39] ...optimizers/trijoint.py.151: total_norm: 10.790856 [I 2022-10-11 15:53:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1290/2304 [I 2022-10-11 15:53:48] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:14:36 [I 2022-10-11 15:53:48] ...trap/engines/engine.py.229: process: 0.87046 | load: 0.00312 [I 2022-10-11 15:53:48] ...trap/engines/engine.py.230: loss: 0.12619 [I 2022-10-11 15:53:48] ...optimizers/trijoint.py.151: total_norm: 17.192350 [I 2022-10-11 15:53:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 1300/2304 [I 2022-10-11 15:53:56] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:14:33 [I 2022-10-11 15:53:56] ...trap/engines/engine.py.229: process: 0.87681 | load: 0.00313 [I 2022-10-11 15:53:56] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-11 15:53:56] ...optimizers/trijoint.py.151: total_norm: 23.374382 [I 2022-10-11 15:54:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1310/2304 [I 2022-10-11 15:54:05] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:14:11 [I 2022-10-11 15:54:05] ...trap/engines/engine.py.229: process: 0.85573 | load: 0.00327 [I 2022-10-11 15:54:05] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-11 15:54:05] ...optimizers/trijoint.py.151: total_norm: 12.073122 [I 2022-10-11 15:54:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 1320/2304 [I 2022-10-11 15:54:13] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:14:08 [I 2022-10-11 15:54:13] ...trap/engines/engine.py.229: process: 0.88424 | load: 0.00315 [I 2022-10-11 15:54:13] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-11 15:54:13] ...optimizers/trijoint.py.151: total_norm: 9.226731 [I 2022-10-11 15:54:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1330/2304 [I 2022-10-11 15:54:22] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:56 [I 2022-10-11 15:54:22] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00311 [I 2022-10-11 15:54:22] ...trap/engines/engine.py.230: loss: 0.15054 [I 2022-10-11 15:54:22] ...optimizers/trijoint.py.151: total_norm: 27.394751 [I 2022-10-11 15:54:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1340/2304 [I 2022-10-11 15:54:31] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:13:44 [I 2022-10-11 15:54:31] ...trap/engines/engine.py.229: process: 0.84274 | load: 0.00314 [I 2022-10-11 15:54:31] ...trap/engines/engine.py.230: loss: 0.15340 [I 2022-10-11 15:54:31] ...optimizers/trijoint.py.151: total_norm: 12.114844 [I 2022-10-11 15:54:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1350/2304 [I 2022-10-11 15:54:39] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:34 [I 2022-10-11 15:54:39] ...trap/engines/engine.py.229: process: 0.85567 | load: 0.00318 [I 2022-10-11 15:54:39] ...trap/engines/engine.py.230: loss: 0.11037 [I 2022-10-11 15:54:39] ...optimizers/trijoint.py.151: total_norm: 21.190058 [I 2022-10-11 15:54:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1360/2304 [I 2022-10-11 15:54:48] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:13:30 [I 2022-10-11 15:54:48] ...trap/engines/engine.py.229: process: 0.85778 | load: 0.00314 [I 2022-10-11 15:54:48] ...trap/engines/engine.py.230: loss: 0.08569 [I 2022-10-11 15:54:48] ...optimizers/trijoint.py.151: total_norm: 8.490257 [I 2022-10-11 15:54:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 1370/2304 [I 2022-10-11 15:54:56] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:18 [I 2022-10-11 15:54:56] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00313 [I 2022-10-11 15:54:56] ...trap/engines/engine.py.230: loss: 0.18228 [I 2022-10-11 15:54:56] ...optimizers/trijoint.py.151: total_norm: 14.912440 [I 2022-10-11 15:55:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1380/2304 [I 2022-10-11 15:55:05] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:13:15 [I 2022-10-11 15:55:05] ...trap/engines/engine.py.229: process: 0.86916 | load: 0.00315 [I 2022-10-11 15:55:05] ...trap/engines/engine.py.230: loss: 0.05574 [I 2022-10-11 15:55:05] ...optimizers/trijoint.py.151: total_norm: 10.386508 [I 2022-10-11 15:55:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 1390/2304 [I 2022-10-11 15:55:13] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:13:03 [I 2022-10-11 15:55:13] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00312 [I 2022-10-11 15:55:13] ...trap/engines/engine.py.230: loss: 0.23132 [I 2022-10-11 15:55:13] ...optimizers/trijoint.py.151: total_norm: 45.959778 [I 2022-10-11 15:55:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1400/2304 [I 2022-10-11 15:55:22] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:13:01 [I 2022-10-11 15:55:22] ...trap/engines/engine.py.229: process: 0.86155 | load: 0.00314 [I 2022-10-11 15:55:22] ...trap/engines/engine.py.230: loss: 0.12894 [I 2022-10-11 15:55:22] ...optimizers/trijoint.py.151: total_norm: 16.614975 [I 2022-10-11 15:55:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1410/2304 [I 2022-10-11 15:55:31] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:47 [I 2022-10-11 15:55:31] ...trap/engines/engine.py.229: process: 0.85692 | load: 0.00313 [I 2022-10-11 15:55:31] ...trap/engines/engine.py.230: loss: 0.10770 [I 2022-10-11 15:55:31] ...optimizers/trijoint.py.151: total_norm: 12.941792 [I 2022-10-11 15:55:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1420/2304 [I 2022-10-11 15:55:39] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:12:40 [I 2022-10-11 15:55:39] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00312 [I 2022-10-11 15:55:39] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-11 15:55:39] ...optimizers/trijoint.py.151: total_norm: 17.605663 [I 2022-10-11 15:55:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1430/2304 [I 2022-10-11 15:55:48] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:32 [I 2022-10-11 15:55:48] ...trap/engines/engine.py.229: process: 0.86077 | load: 0.00314 [I 2022-10-11 15:55:48] ...trap/engines/engine.py.230: loss: 0.14201 [I 2022-10-11 15:55:48] ...optimizers/trijoint.py.151: total_norm: 30.920465 [I 2022-10-11 15:55:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 1440/2304 [I 2022-10-11 15:55:57] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:12:18 [I 2022-10-11 15:55:57] ...trap/engines/engine.py.229: process: 0.84570 | load: 0.00313 [I 2022-10-11 15:55:57] ...trap/engines/engine.py.230: loss: 0.11268 [I 2022-10-11 15:55:57] ...optimizers/trijoint.py.151: total_norm: 12.174754 [I 2022-10-11 15:56:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1450/2304 [I 2022-10-11 15:56:05] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:12:11 [I 2022-10-11 15:56:05] ...trap/engines/engine.py.229: process: 0.85120 | load: 0.00313 [I 2022-10-11 15:56:05] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-11 15:56:05] ...optimizers/trijoint.py.151: total_norm: 7.632648 [I 2022-10-11 15:56:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1460/2304 [I 2022-10-11 15:56:14] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:12:03 [I 2022-10-11 15:56:14] ...trap/engines/engine.py.229: process: 0.84812 | load: 0.00312 [I 2022-10-11 15:56:14] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-11 15:56:14] ...optimizers/trijoint.py.151: total_norm: 14.800264 [I 2022-10-11 15:56:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1470/2304 [I 2022-10-11 15:56:22] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:11:55 [I 2022-10-11 15:56:22] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00314 [I 2022-10-11 15:56:22] ...trap/engines/engine.py.230: loss: 0.13935 [I 2022-10-11 15:56:22] ...optimizers/trijoint.py.151: total_norm: 13.455029 [I 2022-10-11 15:56:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1480/2304 [I 2022-10-11 15:56:31] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:11:46 [I 2022-10-11 15:56:31] ...trap/engines/engine.py.229: process: 0.85611 | load: 0.00310 [I 2022-10-11 15:56:31] ...trap/engines/engine.py.230: loss: 0.10123 [I 2022-10-11 15:56:31] ...optimizers/trijoint.py.151: total_norm: 18.264463 [I 2022-10-11 15:56:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 1490/2304 [I 2022-10-11 15:56:40] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:36 [I 2022-10-11 15:56:40] ...trap/engines/engine.py.229: process: 0.85558 | load: 0.00296 [I 2022-10-11 15:56:40] ...trap/engines/engine.py.230: loss: 0.09295 [I 2022-10-11 15:56:40] ...optimizers/trijoint.py.151: total_norm: 10.878652 [I 2022-10-11 15:56:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1500/2304 [I 2022-10-11 15:56:48] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:11:28 [I 2022-10-11 15:56:48] ...trap/engines/engine.py.229: process: 0.85271 | load: 0.00298 [I 2022-10-11 15:56:48] ...trap/engines/engine.py.230: loss: 0.09187 [I 2022-10-11 15:56:48] ...optimizers/trijoint.py.151: total_norm: 12.982629 [I 2022-10-11 15:56:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 1510/2304 [I 2022-10-11 15:56:57] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:13 [I 2022-10-11 15:56:57] ...trap/engines/engine.py.229: process: 0.83308 | load: 0.00296 [I 2022-10-11 15:56:57] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-11 15:56:57] ...optimizers/trijoint.py.151: total_norm: 27.680428 [I 2022-10-11 15:57:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1520/2304 [I 2022-10-11 15:57:05] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:11:10 [I 2022-10-11 15:57:05] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00296 [I 2022-10-11 15:57:05] ...trap/engines/engine.py.230: loss: 0.10048 [I 2022-10-11 15:57:05] ...optimizers/trijoint.py.151: total_norm: 13.312732 [I 2022-10-11 15:57:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1530/2304 [I 2022-10-11 15:57:14] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:11:02 [I 2022-10-11 15:57:14] ...trap/engines/engine.py.229: process: 0.85674 | load: 0.00294 [I 2022-10-11 15:57:14] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-11 15:57:14] ...optimizers/trijoint.py.151: total_norm: 29.526745 [I 2022-10-11 15:57:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1540/2304 [I 2022-10-11 15:57:22] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:51 [I 2022-10-11 15:57:22] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00298 [I 2022-10-11 15:57:22] ...trap/engines/engine.py.230: loss: 0.09465 [I 2022-10-11 15:57:22] ...optimizers/trijoint.py.151: total_norm: 15.371747 [I 2022-10-11 15:57:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1550/2304 [I 2022-10-11 15:57:31] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:10:39 [I 2022-10-11 15:57:31] ...trap/engines/engine.py.229: process: 0.84336 | load: 0.00308 [I 2022-10-11 15:57:31] ...trap/engines/engine.py.230: loss: 0.08794 [I 2022-10-11 15:57:31] ...optimizers/trijoint.py.151: total_norm: 7.933418 [I 2022-10-11 15:57:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1560/2304 [I 2022-10-11 15:57:39] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:35 [I 2022-10-11 15:57:39] ...trap/engines/engine.py.229: process: 0.85275 | load: 0.00297 [I 2022-10-11 15:57:39] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-11 15:57:39] ...optimizers/trijoint.py.151: total_norm: 18.802345 [I 2022-10-11 15:57:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1570/2304 [I 2022-10-11 15:57:48] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:10:25 [I 2022-10-11 15:57:48] ...trap/engines/engine.py.229: process: 0.85694 | load: 0.00294 [I 2022-10-11 15:57:48] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-11 15:57:48] ...optimizers/trijoint.py.151: total_norm: 14.628410 [I 2022-10-11 15:57:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 1580/2304 [I 2022-10-11 15:57:57] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:10:16 [I 2022-10-11 15:57:57] ...trap/engines/engine.py.229: process: 0.84776 | load: 0.00292 [I 2022-10-11 15:57:57] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-11 15:57:57] ...optimizers/trijoint.py.151: total_norm: 13.686519 [I 2022-10-11 15:58:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1590/2304 [I 2022-10-11 15:58:05] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:10:17 [I 2022-10-11 15:58:05] ...trap/engines/engine.py.229: process: 0.86063 | load: 0.00374 [I 2022-10-11 15:58:05] ...trap/engines/engine.py.230: loss: 0.08452 [I 2022-10-11 15:58:05] ...optimizers/trijoint.py.151: total_norm: 8.951054 [I 2022-10-11 15:58:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1600/2304 [I 2022-10-11 15:58:14] ...trap/engines/engine.py.228: elapsed: 0:23:19 | left: 0:09:59 [I 2022-10-11 15:58:14] ...trap/engines/engine.py.229: process: 0.83787 | load: 0.00297 [I 2022-10-11 15:58:14] ...trap/engines/engine.py.230: loss: 0.10692 [I 2022-10-11 15:58:14] ...optimizers/trijoint.py.151: total_norm: 9.536972 [I 2022-10-11 15:58:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1610/2304 [I 2022-10-11 15:58:22] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:48 [I 2022-10-11 15:58:22] ...trap/engines/engine.py.229: process: 0.85025 | load: 0.00305 [I 2022-10-11 15:58:22] ...trap/engines/engine.py.230: loss: 0.14281 [I 2022-10-11 15:58:22] ...optimizers/trijoint.py.151: total_norm: 15.529830 [I 2022-10-11 15:58:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1620/2304 [I 2022-10-11 15:58:31] ...trap/engines/engine.py.228: elapsed: 0:23:36 | left: 0:09:43 [I 2022-10-11 15:58:31] ...trap/engines/engine.py.229: process: 0.85453 | load: 0.00297 [I 2022-10-11 15:58:31] ...trap/engines/engine.py.230: loss: 0.09007 [I 2022-10-11 15:58:31] ...optimizers/trijoint.py.151: total_norm: 13.163895 [I 2022-10-11 15:58:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1630/2304 [I 2022-10-11 15:58:39] ...trap/engines/engine.py.228: elapsed: 0:23:45 | left: 0:09:35 [I 2022-10-11 15:58:39] ...trap/engines/engine.py.229: process: 0.86528 | load: 0.00305 [I 2022-10-11 15:58:39] ...trap/engines/engine.py.230: loss: 0.13901 [I 2022-10-11 15:58:39] ...optimizers/trijoint.py.151: total_norm: 16.505945 [I 2022-10-11 15:58:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1640/2304 [I 2022-10-11 15:58:48] ...trap/engines/engine.py.228: elapsed: 0:23:53 | left: 0:09:40 [I 2022-10-11 15:58:48] ...trap/engines/engine.py.229: process: 0.88303 | load: 0.00378 [I 2022-10-11 15:58:48] ...trap/engines/engine.py.230: loss: 0.12280 [I 2022-10-11 15:58:48] ...optimizers/trijoint.py.151: total_norm: 16.497040 [I 2022-10-11 15:58:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 1650/2304 [I 2022-10-11 15:58:57] ...trap/engines/engine.py.228: elapsed: 0:24:02 | left: 0:09:29 [I 2022-10-11 15:58:57] ...trap/engines/engine.py.229: process: 0.86773 | load: 0.00381 [I 2022-10-11 15:58:57] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-11 15:58:57] ...optimizers/trijoint.py.151: total_norm: 10.563023 [I 2022-10-11 15:59:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 1660/2304 [I 2022-10-11 15:59:06] ...trap/engines/engine.py.228: elapsed: 0:24:11 | left: 0:09:22 [I 2022-10-11 15:59:06] ...trap/engines/engine.py.229: process: 0.86613 | load: 0.00381 [I 2022-10-11 15:59:06] ...trap/engines/engine.py.230: loss: 0.08133 [I 2022-10-11 15:59:06] ...optimizers/trijoint.py.151: total_norm: 10.555648 [I 2022-10-11 15:59:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1670/2304 [I 2022-10-11 15:59:14] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:09:15 [I 2022-10-11 15:59:14] ...trap/engines/engine.py.229: process: 0.88337 | load: 0.00378 [I 2022-10-11 15:59:14] ...trap/engines/engine.py.230: loss: 0.12049 [I 2022-10-11 15:59:14] ...optimizers/trijoint.py.151: total_norm: 16.624207 [I 2022-10-11 15:59:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 1680/2304 [I 2022-10-11 15:59:23] ...trap/engines/engine.py.228: elapsed: 0:24:28 | left: 0:09:04 [I 2022-10-11 15:59:23] ...trap/engines/engine.py.229: process: 0.87722 | load: 0.00375 [I 2022-10-11 15:59:23] ...trap/engines/engine.py.230: loss: 0.09709 [I 2022-10-11 15:59:23] ...optimizers/trijoint.py.151: total_norm: 17.664982 [I 2022-10-11 15:59:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 1690/2304 [I 2022-10-11 15:59:32] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:56 [I 2022-10-11 15:59:32] ...trap/engines/engine.py.229: process: 0.88066 | load: 0.00374 [I 2022-10-11 15:59:32] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-11 15:59:32] ...optimizers/trijoint.py.151: total_norm: 21.242840 [I 2022-10-11 15:59:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 1700/2304 [I 2022-10-11 15:59:41] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:45 [I 2022-10-11 15:59:41] ...trap/engines/engine.py.229: process: 0.86994 | load: 0.00373 [I 2022-10-11 15:59:41] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-11 15:59:41] ...optimizers/trijoint.py.151: total_norm: 16.719866 [I 2022-10-11 15:59:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 1710/2304 [I 2022-10-11 15:59:49] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:37 [I 2022-10-11 15:59:49] ...trap/engines/engine.py.229: process: 0.86325 | load: 0.00375 [I 2022-10-11 15:59:49] ...trap/engines/engine.py.230: loss: 0.17303 [I 2022-10-11 15:59:49] ...optimizers/trijoint.py.151: total_norm: 77.037399 [I 2022-10-11 15:59:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 1720/2304 [I 2022-10-11 15:59:58] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:08:26 [I 2022-10-11 15:59:58] ...trap/engines/engine.py.229: process: 0.86315 | load: 0.00401 [I 2022-10-11 15:59:58] ...trap/engines/engine.py.230: loss: 0.09666 [I 2022-10-11 15:59:58] ...optimizers/trijoint.py.151: total_norm: 10.798105 [I 2022-10-11 16:00:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 1730/2304 [I 2022-10-11 16:00:07] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:08:22 [I 2022-10-11 16:00:07] ...trap/engines/engine.py.229: process: 0.88086 | load: 0.00378 [I 2022-10-11 16:00:07] ...trap/engines/engine.py.230: loss: 0.16611 [I 2022-10-11 16:00:07] ...optimizers/trijoint.py.151: total_norm: 14.793648 [I 2022-10-11 16:00:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 1740/2304 [I 2022-10-11 16:00:16] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:08:13 [I 2022-10-11 16:00:16] ...trap/engines/engine.py.229: process: 0.87643 | load: 0.00380 [I 2022-10-11 16:00:16] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-11 16:00:16] ...optimizers/trijoint.py.151: total_norm: 18.711807 [I 2022-10-11 16:00:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 1750/2304 [I 2022-10-11 16:00:24] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:08:07 [I 2022-10-11 16:00:24] ...trap/engines/engine.py.229: process: 0.87952 | load: 0.00392 [I 2022-10-11 16:00:24] ...trap/engines/engine.py.230: loss: 0.12710 [I 2022-10-11 16:00:24] ...optimizers/trijoint.py.151: total_norm: 28.116264 [I 2022-10-11 16:00:33] ...trap/engines/engine.py.224: train: epoch 110 | batch 1760/2304 [I 2022-10-11 16:00:33] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:08:02 [I 2022-10-11 16:00:33] ...trap/engines/engine.py.229: process: 0.88504 | load: 0.00403 [I 2022-10-11 16:00:33] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-11 16:00:33] ...optimizers/trijoint.py.151: total_norm: 14.059831 [I 2022-10-11 16:00:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 1770/2304 [I 2022-10-11 16:00:42] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:52 [I 2022-10-11 16:00:42] ...trap/engines/engine.py.229: process: 0.88229 | load: 0.00399 [I 2022-10-11 16:00:42] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-11 16:00:42] ...optimizers/trijoint.py.151: total_norm: 11.536485 [I 2022-10-11 16:00:51] ...trap/engines/engine.py.224: train: epoch 110 | batch 1780/2304 [I 2022-10-11 16:00:51] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:43 [I 2022-10-11 16:00:51] ...trap/engines/engine.py.229: process: 0.88262 | load: 0.00401 [I 2022-10-11 16:00:51] ...trap/engines/engine.py.230: loss: 0.08399 [I 2022-10-11 16:00:51] ...optimizers/trijoint.py.151: total_norm: 13.852634 [I 2022-10-11 16:01:00] ...trap/engines/engine.py.224: train: epoch 110 | batch 1790/2304 [I 2022-10-11 16:01:00] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:38 [I 2022-10-11 16:01:00] ...trap/engines/engine.py.229: process: 0.88978 | load: 0.00396 [I 2022-10-11 16:01:00] ...trap/engines/engine.py.230: loss: 0.09721 [I 2022-10-11 16:01:00] ...optimizers/trijoint.py.151: total_norm: 11.646259 [I 2022-10-11 16:01:09] ...trap/engines/engine.py.224: train: epoch 110 | batch 1800/2304 [I 2022-10-11 16:01:09] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:07:24 [I 2022-10-11 16:01:09] ...trap/engines/engine.py.229: process: 0.87561 | load: 0.00395 [I 2022-10-11 16:01:09] ...trap/engines/engine.py.230: loss: 0.09519 [I 2022-10-11 16:01:09] ...optimizers/trijoint.py.151: total_norm: 10.770413 [I 2022-10-11 16:01:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 1810/2304 [I 2022-10-11 16:01:18] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:07:15 [I 2022-10-11 16:01:18] ...trap/engines/engine.py.229: process: 0.88705 | load: 0.00395 [I 2022-10-11 16:01:18] ...trap/engines/engine.py.230: loss: 0.09983 [I 2022-10-11 16:01:18] ...optimizers/trijoint.py.151: total_norm: 15.490766 [I 2022-10-11 16:01:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 1820/2304 [I 2022-10-11 16:01:27] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:07:05 [I 2022-10-11 16:01:27] ...trap/engines/engine.py.229: process: 0.88130 | load: 0.00394 [I 2022-10-11 16:01:27] ...trap/engines/engine.py.230: loss: 0.09738 [I 2022-10-11 16:01:27] ...optimizers/trijoint.py.151: total_norm: 17.198000 [I 2022-10-11 16:01:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 1830/2304 [I 2022-10-11 16:01:35] ...trap/engines/engine.py.228: elapsed: 0:26:41 | left: 0:07:01 [I 2022-10-11 16:01:35] ...trap/engines/engine.py.229: process: 0.89722 | load: 0.00399 [I 2022-10-11 16:01:35] ...trap/engines/engine.py.230: loss: 0.11482 [I 2022-10-11 16:01:35] ...optimizers/trijoint.py.151: total_norm: 18.161659 [I 2022-10-11 16:01:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 1840/2304 [I 2022-10-11 16:01:44] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:48 [I 2022-10-11 16:01:44] ...trap/engines/engine.py.229: process: 0.88070 | load: 0.00396 [I 2022-10-11 16:01:44] ...trap/engines/engine.py.230: loss: 0.08299 [I 2022-10-11 16:01:44] ...optimizers/trijoint.py.151: total_norm: 8.310733 [I 2022-10-11 16:01:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 1850/2304 [I 2022-10-11 16:01:53] ...trap/engines/engine.py.228: elapsed: 0:26:58 | left: 0:06:42 [I 2022-10-11 16:01:53] ...trap/engines/engine.py.229: process: 0.88004 | load: 0.00393 [I 2022-10-11 16:01:53] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-11 16:01:53] ...optimizers/trijoint.py.151: total_norm: 9.492283 [I 2022-10-11 16:02:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 1860/2304 [I 2022-10-11 16:02:02] ...trap/engines/engine.py.228: elapsed: 0:27:07 | left: 0:06:33 [I 2022-10-11 16:02:02] ...trap/engines/engine.py.229: process: 0.87734 | load: 0.00394 [I 2022-10-11 16:02:02] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-11 16:02:02] ...optimizers/trijoint.py.151: total_norm: 14.196482 [I 2022-10-11 16:02:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 1870/2304 [I 2022-10-11 16:02:11] ...trap/engines/engine.py.228: elapsed: 0:27:16 | left: 0:06:24 [I 2022-10-11 16:02:11] ...trap/engines/engine.py.229: process: 0.89270 | load: 0.00395 [I 2022-10-11 16:02:11] ...trap/engines/engine.py.230: loss: 0.14315 [I 2022-10-11 16:02:11] ...optimizers/trijoint.py.151: total_norm: 21.943485 [I 2022-10-11 16:02:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 1880/2304 [I 2022-10-11 16:02:20] ...trap/engines/engine.py.228: elapsed: 0:27:25 | left: 0:06:13 [I 2022-10-11 16:02:20] ...trap/engines/engine.py.229: process: 0.87536 | load: 0.00397 [I 2022-10-11 16:02:20] ...trap/engines/engine.py.230: loss: 0.17347 [I 2022-10-11 16:02:20] ...optimizers/trijoint.py.151: total_norm: 24.145296 [I 2022-10-11 16:02:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 1890/2304 [I 2022-10-11 16:02:29] ...trap/engines/engine.py.228: elapsed: 0:27:34 | left: 0:06:06 [I 2022-10-11 16:02:29] ...trap/engines/engine.py.229: process: 0.88922 | load: 0.00395 [I 2022-10-11 16:02:29] ...trap/engines/engine.py.230: loss: 0.09771 [I 2022-10-11 16:02:29] ...optimizers/trijoint.py.151: total_norm: 10.927784 [I 2022-10-11 16:02:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 1900/2304 [I 2022-10-11 16:02:37] ...trap/engines/engine.py.228: elapsed: 0:27:43 | left: 0:05:57 [I 2022-10-11 16:02:37] ...trap/engines/engine.py.229: process: 0.87152 | load: 0.00394 [I 2022-10-11 16:02:37] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-11 16:02:37] ...optimizers/trijoint.py.151: total_norm: 12.901474 [I 2022-10-11 16:02:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 1910/2304 [I 2022-10-11 16:02:46] ...trap/engines/engine.py.228: elapsed: 0:27:51 | left: 0:05:48 [I 2022-10-11 16:02:46] ...trap/engines/engine.py.229: process: 0.89567 | load: 0.00396 [I 2022-10-11 16:02:46] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-11 16:02:46] ...optimizers/trijoint.py.151: total_norm: 12.150109 [I 2022-10-11 16:02:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1920/2304 [I 2022-10-11 16:02:55] ...trap/engines/engine.py.228: elapsed: 0:28:00 | left: 0:05:35 [I 2022-10-11 16:02:55] ...trap/engines/engine.py.229: process: 0.86409 | load: 0.00312 [I 2022-10-11 16:02:55] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-11 16:02:55] ...optimizers/trijoint.py.151: total_norm: 13.041585 [I 2022-10-11 16:03:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 1930/2304 [I 2022-10-11 16:03:04] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:19 [I 2022-10-11 16:03:04] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00313 [I 2022-10-11 16:03:04] ...trap/engines/engine.py.230: loss: 0.19768 [I 2022-10-11 16:03:04] ...optimizers/trijoint.py.151: total_norm: 13.670252 [I 2022-10-11 16:03:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 1940/2304 [I 2022-10-11 16:03:12] ...trap/engines/engine.py.228: elapsed: 0:28:17 | left: 0:05:09 [I 2022-10-11 16:03:12] ...trap/engines/engine.py.229: process: 0.86205 | load: 0.00313 [I 2022-10-11 16:03:12] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-11 16:03:12] ...optimizers/trijoint.py.151: total_norm: 8.390542 [I 2022-10-11 16:03:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 1950/2304 [I 2022-10-11 16:03:21] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:05:02 [I 2022-10-11 16:03:21] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00315 [I 2022-10-11 16:03:21] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-11 16:03:21] ...optimizers/trijoint.py.151: total_norm: 12.208260 [I 2022-10-11 16:03:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 1960/2304 [I 2022-10-11 16:03:29] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:55 [I 2022-10-11 16:03:29] ...trap/engines/engine.py.229: process: 0.86831 | load: 0.00312 [I 2022-10-11 16:03:29] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-11 16:03:29] ...optimizers/trijoint.py.151: total_norm: 17.208143 [I 2022-10-11 16:03:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1970/2304 [I 2022-10-11 16:03:38] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:45 [I 2022-10-11 16:03:38] ...trap/engines/engine.py.229: process: 0.84243 | load: 0.00315 [I 2022-10-11 16:03:38] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-11 16:03:38] ...optimizers/trijoint.py.151: total_norm: 15.911137 [I 2022-10-11 16:03:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 1980/2304 [I 2022-10-11 16:03:47] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:37 [I 2022-10-11 16:03:47] ...trap/engines/engine.py.229: process: 0.85166 | load: 0.00313 [I 2022-10-11 16:03:47] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-11 16:03:47] ...optimizers/trijoint.py.151: total_norm: 23.080015 [I 2022-10-11 16:03:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1990/2304 [I 2022-10-11 16:03:55] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:28 [I 2022-10-11 16:03:55] ...trap/engines/engine.py.229: process: 0.86029 | load: 0.00318 [I 2022-10-11 16:03:55] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-11 16:03:55] ...optimizers/trijoint.py.151: total_norm: 13.927422 [I 2022-10-11 16:04:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 2000/2304 [I 2022-10-11 16:04:04] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:22 [I 2022-10-11 16:04:04] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00311 [I 2022-10-11 16:04:04] ...trap/engines/engine.py.230: loss: 0.07314 [I 2022-10-11 16:04:04] ...optimizers/trijoint.py.151: total_norm: 9.347382 [I 2022-10-11 16:04:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 2010/2304 [I 2022-10-11 16:04:12] ...trap/engines/engine.py.228: elapsed: 0:29:18 | left: 0:04:14 [I 2022-10-11 16:04:12] ...trap/engines/engine.py.229: process: 0.88334 | load: 0.00315 [I 2022-10-11 16:04:12] ...trap/engines/engine.py.230: loss: 0.09683 [I 2022-10-11 16:04:12] ...optimizers/trijoint.py.151: total_norm: 20.590897 [I 2022-10-11 16:04:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 2020/2304 [I 2022-10-11 16:04:21] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:04:00 [I 2022-10-11 16:04:21] ...trap/engines/engine.py.229: process: 0.83499 | load: 0.00297 [I 2022-10-11 16:04:21] ...trap/engines/engine.py.230: loss: 0.12831 [I 2022-10-11 16:04:21] ...optimizers/trijoint.py.151: total_norm: 10.179829 [I 2022-10-11 16:04:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 2030/2304 [I 2022-10-11 16:04:29] ...trap/engines/engine.py.228: elapsed: 0:29:35 | left: 0:03:53 [I 2022-10-11 16:04:29] ...trap/engines/engine.py.229: process: 0.84802 | load: 0.00295 [I 2022-10-11 16:04:29] ...trap/engines/engine.py.230: loss: 0.15859 [I 2022-10-11 16:04:29] ...optimizers/trijoint.py.151: total_norm: 10.561687 [I 2022-10-11 16:04:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 2040/2304 [I 2022-10-11 16:04:38] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:43 [I 2022-10-11 16:04:38] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00297 [I 2022-10-11 16:04:38] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-11 16:04:38] ...optimizers/trijoint.py.151: total_norm: 18.583679 [I 2022-10-11 16:04:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 2050/2304 [I 2022-10-11 16:04:46] ...trap/engines/engine.py.228: elapsed: 0:29:52 | left: 0:03:33 [I 2022-10-11 16:04:46] ...trap/engines/engine.py.229: process: 0.84487 | load: 0.00300 [I 2022-10-11 16:04:46] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-11 16:04:46] ...optimizers/trijoint.py.151: total_norm: 28.211412 [I 2022-10-11 16:04:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 2060/2304 [I 2022-10-11 16:04:55] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:25 [I 2022-10-11 16:04:55] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00294 [I 2022-10-11 16:04:55] ...trap/engines/engine.py.230: loss: 0.22108 [I 2022-10-11 16:04:55] ...optimizers/trijoint.py.151: total_norm: 14.817783 [I 2022-10-11 16:05:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 2070/2304 [I 2022-10-11 16:05:03] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:16 [I 2022-10-11 16:05:03] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00300 [I 2022-10-11 16:05:03] ...trap/engines/engine.py.230: loss: 0.15449 [I 2022-10-11 16:05:03] ...optimizers/trijoint.py.151: total_norm: 27.517937 [I 2022-10-11 16:05:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 2080/2304 [I 2022-10-11 16:05:12] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:03:08 [I 2022-10-11 16:05:12] ...trap/engines/engine.py.229: process: 0.85411 | load: 0.00295 [I 2022-10-11 16:05:12] ...trap/engines/engine.py.230: loss: 0.13470 [I 2022-10-11 16:05:12] ...optimizers/trijoint.py.151: total_norm: 31.800146 [I 2022-10-11 16:05:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 2090/2304 [I 2022-10-11 16:05:20] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:03:00 [I 2022-10-11 16:05:20] ...trap/engines/engine.py.229: process: 0.84794 | load: 0.00297 [I 2022-10-11 16:05:20] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-11 16:05:20] ...optimizers/trijoint.py.151: total_norm: 9.664539 [I 2022-10-11 16:05:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 2100/2304 [I 2022-10-11 16:05:29] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:51 [I 2022-10-11 16:05:29] ...trap/engines/engine.py.229: process: 0.83579 | load: 0.00295 [I 2022-10-11 16:05:29] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-11 16:05:29] ...optimizers/trijoint.py.151: total_norm: 32.010925 [I 2022-10-11 16:05:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 2110/2304 [I 2022-10-11 16:05:37] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:44 [I 2022-10-11 16:05:37] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00299 [I 2022-10-11 16:05:37] ...trap/engines/engine.py.230: loss: 0.16134 [I 2022-10-11 16:05:37] ...optimizers/trijoint.py.151: total_norm: 13.839355 [I 2022-10-11 16:05:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 2120/2304 [I 2022-10-11 16:05:46] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:35 [I 2022-10-11 16:05:46] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00298 [I 2022-10-11 16:05:46] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-11 16:05:46] ...optimizers/trijoint.py.151: total_norm: 59.723576 [I 2022-10-11 16:05:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 2130/2304 [I 2022-10-11 16:05:54] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:27 [I 2022-10-11 16:05:54] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00303 [I 2022-10-11 16:05:54] ...trap/engines/engine.py.230: loss: 0.12731 [I 2022-10-11 16:05:54] ...optimizers/trijoint.py.151: total_norm: 16.713495 [I 2022-10-11 16:06:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 2140/2304 [I 2022-10-11 16:06:03] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:02:18 [I 2022-10-11 16:06:03] ...trap/engines/engine.py.229: process: 0.84026 | load: 0.00300 [I 2022-10-11 16:06:03] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-11 16:06:03] ...optimizers/trijoint.py.151: total_norm: 26.396896 [I 2022-10-11 16:06:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 2150/2304 [I 2022-10-11 16:06:11] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:02:09 [I 2022-10-11 16:06:11] ...trap/engines/engine.py.229: process: 0.85589 | load: 0.00301 [I 2022-10-11 16:06:11] ...trap/engines/engine.py.230: loss: 0.11786 [I 2022-10-11 16:06:11] ...optimizers/trijoint.py.151: total_norm: 9.525010 [I 2022-10-11 16:06:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 2160/2304 [I 2022-10-11 16:06:19] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:02:00 [I 2022-10-11 16:06:19] ...trap/engines/engine.py.229: process: 0.83701 | load: 0.00310 [I 2022-10-11 16:06:19] ...trap/engines/engine.py.230: loss: 0.10471 [I 2022-10-11 16:06:19] ...optimizers/trijoint.py.151: total_norm: 12.358508 [I 2022-10-11 16:06:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 2170/2304 [I 2022-10-11 16:06:28] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:52 [I 2022-10-11 16:06:28] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00299 [I 2022-10-11 16:06:28] ...trap/engines/engine.py.230: loss: 0.09458 [I 2022-10-11 16:06:28] ...optimizers/trijoint.py.151: total_norm: 10.266229 [I 2022-10-11 16:06:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 2180/2304 [I 2022-10-11 16:06:36] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:44 [I 2022-10-11 16:06:36] ...trap/engines/engine.py.229: process: 0.85426 | load: 0.00301 [I 2022-10-11 16:06:36] ...trap/engines/engine.py.230: loss: 0.16396 [I 2022-10-11 16:06:36] ...optimizers/trijoint.py.151: total_norm: 15.822779 [I 2022-10-11 16:06:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 2190/2304 [I 2022-10-11 16:06:45] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:36 [I 2022-10-11 16:06:45] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00305 [I 2022-10-11 16:06:45] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-11 16:06:45] ...optimizers/trijoint.py.151: total_norm: 14.978786 [I 2022-10-11 16:06:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 2200/2304 [I 2022-10-11 16:06:53] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:27 [I 2022-10-11 16:06:53] ...trap/engines/engine.py.229: process: 0.83385 | load: 0.00297 [I 2022-10-11 16:06:53] ...trap/engines/engine.py.230: loss: 0.15957 [I 2022-10-11 16:06:53] ...optimizers/trijoint.py.151: total_norm: 18.761951 [I 2022-10-11 16:07:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 2210/2304 [I 2022-10-11 16:07:02] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:01:19 [I 2022-10-11 16:07:02] ...trap/engines/engine.py.229: process: 0.84722 | load: 0.00314 [I 2022-10-11 16:07:02] ...trap/engines/engine.py.230: loss: 0.09295 [I 2022-10-11 16:07:02] ...optimizers/trijoint.py.151: total_norm: 12.561835 [I 2022-10-11 16:07:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 2220/2304 [I 2022-10-11 16:07:10] ...trap/engines/engine.py.228: elapsed: 0:32:16 | left: 0:01:11 [I 2022-10-11 16:07:10] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00315 [I 2022-10-11 16:07:10] ...trap/engines/engine.py.230: loss: 0.08683 [I 2022-10-11 16:07:10] ...optimizers/trijoint.py.151: total_norm: 20.222090 [I 2022-10-11 16:07:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 2230/2304 [I 2022-10-11 16:07:19] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:01:02 [I 2022-10-11 16:07:19] ...trap/engines/engine.py.229: process: 0.84557 | load: 0.00312 [I 2022-10-11 16:07:19] ...trap/engines/engine.py.230: loss: 0.08096 [I 2022-10-11 16:07:19] ...optimizers/trijoint.py.151: total_norm: 22.313726 [I 2022-10-11 16:07:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 2240/2304 [I 2022-10-11 16:07:27] ...trap/engines/engine.py.228: elapsed: 0:32:32 | left: 0:00:54 [I 2022-10-11 16:07:27] ...trap/engines/engine.py.229: process: 0.83847 | load: 0.00315 [I 2022-10-11 16:07:27] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-11 16:07:27] ...optimizers/trijoint.py.151: total_norm: 8.464965 [I 2022-10-11 16:07:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 2250/2304 [I 2022-10-11 16:07:36] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:46 [I 2022-10-11 16:07:36] ...trap/engines/engine.py.229: process: 0.85859 | load: 0.00312 [I 2022-10-11 16:07:36] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-11 16:07:36] ...optimizers/trijoint.py.151: total_norm: 10.070291 [I 2022-10-11 16:07:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 2260/2304 [I 2022-10-11 16:07:44] ...trap/engines/engine.py.228: elapsed: 0:32:49 | left: 0:00:37 [I 2022-10-11 16:07:44] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00314 [I 2022-10-11 16:07:44] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-11 16:07:44] ...optimizers/trijoint.py.151: total_norm: 14.899608 [I 2022-10-11 16:07:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 2270/2304 [I 2022-10-11 16:07:53] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:28 [I 2022-10-11 16:07:53] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00314 [I 2022-10-11 16:07:53] ...trap/engines/engine.py.230: loss: 0.11104 [I 2022-10-11 16:07:53] ...optimizers/trijoint.py.151: total_norm: 22.843025 [I 2022-10-11 16:08:01] ...trap/engines/engine.py.224: train: epoch 110 | batch 2280/2304 [I 2022-10-11 16:08:01] ...trap/engines/engine.py.228: elapsed: 0:33:06 | left: 0:00:20 [I 2022-10-11 16:08:01] ...trap/engines/engine.py.229: process: 0.84610 | load: 0.00315 [I 2022-10-11 16:08:01] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-11 16:08:01] ...optimizers/trijoint.py.151: total_norm: 26.929806 [I 2022-10-11 16:08:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 2290/2304 [I 2022-10-11 16:08:10] ...trap/engines/engine.py.228: elapsed: 0:33:15 | left: 0:00:11 [I 2022-10-11 16:08:10] ...trap/engines/engine.py.229: process: 0.84701 | load: 0.00313 [I 2022-10-11 16:08:10] ...trap/engines/engine.py.230: loss: 0.08274 [I 2022-10-11 16:08:10] ...optimizers/trijoint.py.151: total_norm: 19.539707 [I 2022-10-11 16:08:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 2300/2304 [I 2022-10-11 16:08:18] ...trap/engines/engine.py.228: elapsed: 0:33:24 | left: 0:00:03 [I 2022-10-11 16:08:18] ...trap/engines/engine.py.229: process: 0.83844 | load: 0.00019 [I 2022-10-11 16:08:18] ...trap/engines/engine.py.230: loss: 0.14173 [I 2022-10-11 16:08:18] ...optimizers/trijoint.py.151: total_norm: 16.122185 [I 2022-10-11 16:08:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 2304/2304 [I 2022-10-11 16:08:22] ...trap/engines/engine.py.228: elapsed: 0:33:27 | left: 0:00:00 [I 2022-10-11 16:08:22] ...trap/engines/engine.py.229: process: 0.83273 | load: 0.00018 [I 2022-10-11 16:08:22] ...trap/engines/engine.py.230: loss: 0.12261 [I 2022-10-11 16:08:22] ...optimizers/trijoint.py.151: total_norm: 63.674549 [S 2022-10-11 16:08:23] ...trap/engines/engine.py.129: train_epoch.epoch: 110 [S 2022-10-11 16:08:23] ...trap/engines/engine.py.129: train_epoch.loss: 0.11976184241813072 [I 2022-10-11 16:08:23] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 16:08:25] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 110 [I 2022-10-11 16:08:25] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 16:08:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 16:08:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 16:08:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 16:08:31] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 16:08:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 0/510 [I 2022-10-11 16:08:32] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:57:22 [I 2022-10-11 16:08:32] ...trap/engines/engine.py.326: process: 6.75086 | load: 6.15976 [I 2022-10-11 16:08:37] ...trap/engines/engine.py.321: eval: epoch 110 | batch 10/510 [I 2022-10-11 16:08:37] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:34 [I 2022-10-11 16:08:37] ...trap/engines/engine.py.326: process: 0.42819 | load: 0.00019 [I 2022-10-11 16:08:44] ...trap/engines/engine.py.321: eval: epoch 110 | batch 20/510 [I 2022-10-11 16:08:44] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:05:49 [I 2022-10-11 16:08:44] ...trap/engines/engine.py.326: process: 0.45830 | load: 0.00019 [I 2022-10-11 16:08:51] ...trap/engines/engine.py.321: eval: epoch 110 | batch 30/510 [I 2022-10-11 16:08:51] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:06:13 [I 2022-10-11 16:08:51] ...trap/engines/engine.py.326: process: 0.83286 | load: 0.00021 [I 2022-10-11 16:08:57] ...trap/engines/engine.py.321: eval: epoch 110 | batch 40/510 [I 2022-10-11 16:08:57] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:04:44 [I 2022-10-11 16:08:57] ...trap/engines/engine.py.326: process: 0.44242 | load: 0.00023 [I 2022-10-11 16:09:02] ...trap/engines/engine.py.321: eval: epoch 110 | batch 50/510 [I 2022-10-11 16:09:02] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:47 [I 2022-10-11 16:09:02] ...trap/engines/engine.py.326: process: 0.57312 | load: 0.00020 [I 2022-10-11 16:09:07] ...trap/engines/engine.py.321: eval: epoch 110 | batch 60/510 [I 2022-10-11 16:09:07] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:03 [I 2022-10-11 16:09:07] ...trap/engines/engine.py.326: process: 0.62140 | load: 0.00021 [I 2022-10-11 16:09:12] ...trap/engines/engine.py.321: eval: epoch 110 | batch 70/510 [I 2022-10-11 16:09:12] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:50 [I 2022-10-11 16:09:12] ...trap/engines/engine.py.326: process: 0.44887 | load: 0.00020 [I 2022-10-11 16:09:18] ...trap/engines/engine.py.321: eval: epoch 110 | batch 80/510 [I 2022-10-11 16:09:18] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:32 [I 2022-10-11 16:09:18] ...trap/engines/engine.py.326: process: 0.47141 | load: 0.00022 [I 2022-10-11 16:09:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 90/510 [I 2022-10-11 16:09:23] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:32 [I 2022-10-11 16:09:23] ...trap/engines/engine.py.326: process: 0.54300 | load: 0.00021 [I 2022-10-11 16:09:27] ...trap/engines/engine.py.321: eval: epoch 110 | batch 100/510 [I 2022-10-11 16:09:27] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:53 [I 2022-10-11 16:09:27] ...trap/engines/engine.py.326: process: 0.31320 | load: 0.00020 [I 2022-10-11 16:09:30] ...trap/engines/engine.py.321: eval: epoch 110 | batch 110/510 [I 2022-10-11 16:09:30] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:13 [I 2022-10-11 16:09:30] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00021 [I 2022-10-11 16:09:33] ...trap/engines/engine.py.321: eval: epoch 110 | batch 120/510 [I 2022-10-11 16:09:33] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:04 [I 2022-10-11 16:09:33] ...trap/engines/engine.py.326: process: 0.31453 | load: 0.00021 [I 2022-10-11 16:09:37] ...trap/engines/engine.py.321: eval: epoch 110 | batch 130/510 [I 2022-10-11 16:09:37] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:06 [I 2022-10-11 16:09:37] ...trap/engines/engine.py.326: process: 0.33424 | load: 0.00023 [I 2022-10-11 16:09:40] ...trap/engines/engine.py.321: eval: epoch 110 | batch 140/510 [I 2022-10-11 16:09:40] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:02:01 [I 2022-10-11 16:09:40] ...trap/engines/engine.py.326: process: 0.32029 | load: 0.00021 [I 2022-10-11 16:09:43] ...trap/engines/engine.py.321: eval: epoch 110 | batch 150/510 [I 2022-10-11 16:09:43] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:56 [I 2022-10-11 16:09:43] ...trap/engines/engine.py.326: process: 0.32377 | load: 0.00021 [I 2022-10-11 16:09:47] ...trap/engines/engine.py.321: eval: epoch 110 | batch 160/510 [I 2022-10-11 16:09:47] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:51 [I 2022-10-11 16:09:47] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00022 [I 2022-10-11 16:09:50] ...trap/engines/engine.py.321: eval: epoch 110 | batch 170/510 [I 2022-10-11 16:09:50] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:49 [I 2022-10-11 16:09:50] ...trap/engines/engine.py.326: process: 0.31495 | load: 0.00021 [I 2022-10-11 16:09:53] ...trap/engines/engine.py.321: eval: epoch 110 | batch 180/510 [I 2022-10-11 16:09:53] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:45 [I 2022-10-11 16:09:53] ...trap/engines/engine.py.326: process: 0.31642 | load: 0.00021 [I 2022-10-11 16:09:56] ...trap/engines/engine.py.321: eval: epoch 110 | batch 190/510 [I 2022-10-11 16:09:56] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:43 [I 2022-10-11 16:09:56] ...trap/engines/engine.py.326: process: 0.32978 | load: 0.00022 [I 2022-10-11 16:09:59] ...trap/engines/engine.py.321: eval: epoch 110 | batch 200/510 [I 2022-10-11 16:09:59] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:39 [I 2022-10-11 16:09:59] ...trap/engines/engine.py.326: process: 0.31869 | load: 0.00022 [I 2022-10-11 16:10:03] ...trap/engines/engine.py.321: eval: epoch 110 | batch 210/510 [I 2022-10-11 16:10:03] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:35 [I 2022-10-11 16:10:03] ...trap/engines/engine.py.326: process: 0.32247 | load: 0.00021 [I 2022-10-11 16:10:06] ...trap/engines/engine.py.321: eval: epoch 110 | batch 220/510 [I 2022-10-11 16:10:06] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:33 [I 2022-10-11 16:10:06] ...trap/engines/engine.py.326: process: 0.31741 | load: 0.00023 [I 2022-10-11 16:10:09] ...trap/engines/engine.py.321: eval: epoch 110 | batch 230/510 [I 2022-10-11 16:10:09] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:30 [I 2022-10-11 16:10:09] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00021 [I 2022-10-11 16:10:12] ...trap/engines/engine.py.321: eval: epoch 110 | batch 240/510 [I 2022-10-11 16:10:12] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:26 [I 2022-10-11 16:10:12] ...trap/engines/engine.py.326: process: 0.31785 | load: 0.00021 [I 2022-10-11 16:10:15] ...trap/engines/engine.py.321: eval: epoch 110 | batch 250/510 [I 2022-10-11 16:10:15] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:23 [I 2022-10-11 16:10:15] ...trap/engines/engine.py.326: process: 0.32146 | load: 0.00023 [I 2022-10-11 16:10:19] ...trap/engines/engine.py.321: eval: epoch 110 | batch 260/510 [I 2022-10-11 16:10:19] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:20 [I 2022-10-11 16:10:19] ...trap/engines/engine.py.326: process: 0.31799 | load: 0.00025 [I 2022-10-11 16:10:22] ...trap/engines/engine.py.321: eval: epoch 110 | batch 270/510 [I 2022-10-11 16:10:22] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:16 [I 2022-10-11 16:10:22] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00025 [I 2022-10-11 16:10:25] ...trap/engines/engine.py.321: eval: epoch 110 | batch 280/510 [I 2022-10-11 16:10:25] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:13 [I 2022-10-11 16:10:25] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00025 [I 2022-10-11 16:10:28] ...trap/engines/engine.py.321: eval: epoch 110 | batch 290/510 [I 2022-10-11 16:10:28] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:10 [I 2022-10-11 16:10:28] ...trap/engines/engine.py.326: process: 0.32247 | load: 0.00022 [I 2022-10-11 16:10:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 300/510 [I 2022-10-11 16:10:32] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:07 [I 2022-10-11 16:10:32] ...trap/engines/engine.py.326: process: 0.32364 | load: 0.00020 [I 2022-10-11 16:10:35] ...trap/engines/engine.py.321: eval: epoch 110 | batch 310/510 [I 2022-10-11 16:10:35] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:04 [I 2022-10-11 16:10:35] ...trap/engines/engine.py.326: process: 0.31380 | load: 0.00021 [I 2022-10-11 16:10:38] ...trap/engines/engine.py.321: eval: epoch 110 | batch 320/510 [I 2022-10-11 16:10:38] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:01 [I 2022-10-11 16:10:38] ...trap/engines/engine.py.326: process: 0.32260 | load: 0.00023 [I 2022-10-11 16:10:41] ...trap/engines/engine.py.321: eval: epoch 110 | batch 330/510 [I 2022-10-11 16:10:41] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:57 [I 2022-10-11 16:10:41] ...trap/engines/engine.py.326: process: 0.32122 | load: 0.00025 [I 2022-10-11 16:10:44] ...trap/engines/engine.py.321: eval: epoch 110 | batch 340/510 [I 2022-10-11 16:10:44] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:54 [I 2022-10-11 16:10:44] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00022 [I 2022-10-11 16:10:48] ...trap/engines/engine.py.321: eval: epoch 110 | batch 350/510 [I 2022-10-11 16:10:48] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:51 [I 2022-10-11 16:10:48] ...trap/engines/engine.py.326: process: 0.32371 | load: 0.00024 [I 2022-10-11 16:10:51] ...trap/engines/engine.py.321: eval: epoch 110 | batch 360/510 [I 2022-10-11 16:10:51] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:48 [I 2022-10-11 16:10:51] ...trap/engines/engine.py.326: process: 0.31850 | load: 0.00022 [I 2022-10-11 16:10:54] ...trap/engines/engine.py.321: eval: epoch 110 | batch 370/510 [I 2022-10-11 16:10:54] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:44 [I 2022-10-11 16:10:54] ...trap/engines/engine.py.326: process: 0.31163 | load: 0.00023 [I 2022-10-11 16:10:57] ...trap/engines/engine.py.321: eval: epoch 110 | batch 380/510 [I 2022-10-11 16:10:57] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:41 [I 2022-10-11 16:10:57] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00021 [I 2022-10-11 16:11:00] ...trap/engines/engine.py.321: eval: epoch 110 | batch 390/510 [I 2022-10-11 16:11:00] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:38 [I 2022-10-11 16:11:00] ...trap/engines/engine.py.326: process: 0.31987 | load: 0.00022 [I 2022-10-11 16:11:04] ...trap/engines/engine.py.321: eval: epoch 110 | batch 400/510 [I 2022-10-11 16:11:04] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:34 [I 2022-10-11 16:11:04] ...trap/engines/engine.py.326: process: 0.32030 | load: 0.00020 [I 2022-10-11 16:11:07] ...trap/engines/engine.py.321: eval: epoch 110 | batch 410/510 [I 2022-10-11 16:11:07] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:31 [I 2022-10-11 16:11:07] ...trap/engines/engine.py.326: process: 0.31719 | load: 0.00020 [I 2022-10-11 16:11:10] ...trap/engines/engine.py.321: eval: epoch 110 | batch 420/510 [I 2022-10-11 16:11:10] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:28 [I 2022-10-11 16:11:10] ...trap/engines/engine.py.326: process: 0.31531 | load: 0.00019 [I 2022-10-11 16:11:13] ...trap/engines/engine.py.321: eval: epoch 110 | batch 430/510 [I 2022-10-11 16:11:13] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:25 [I 2022-10-11 16:11:13] ...trap/engines/engine.py.326: process: 0.31056 | load: 0.00020 [I 2022-10-11 16:11:16] ...trap/engines/engine.py.321: eval: epoch 110 | batch 440/510 [I 2022-10-11 16:11:16] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:22 [I 2022-10-11 16:11:16] ...trap/engines/engine.py.326: process: 0.31420 | load: 0.00021 [I 2022-10-11 16:11:19] ...trap/engines/engine.py.321: eval: epoch 110 | batch 450/510 [I 2022-10-11 16:11:19] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:19 [I 2022-10-11 16:11:19] ...trap/engines/engine.py.326: process: 0.32010 | load: 0.00020 [I 2022-10-11 16:11:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 460/510 [I 2022-10-11 16:11:23] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:15 [I 2022-10-11 16:11:23] ...trap/engines/engine.py.326: process: 0.31305 | load: 0.00020 [I 2022-10-11 16:11:26] ...trap/engines/engine.py.321: eval: epoch 110 | batch 470/510 [I 2022-10-11 16:11:26] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:12 [I 2022-10-11 16:11:26] ...trap/engines/engine.py.326: process: 0.30882 | load: 0.00020 [I 2022-10-11 16:11:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 480/510 [I 2022-10-11 16:11:29] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:09 [I 2022-10-11 16:11:29] ...trap/engines/engine.py.326: process: 0.31671 | load: 0.00020 [I 2022-10-11 16:11:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 490/510 [I 2022-10-11 16:11:32] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:06 [I 2022-10-11 16:11:32] ...trap/engines/engine.py.326: process: 0.31312 | load: 0.00020 [I 2022-10-11 16:11:35] ...trap/engines/engine.py.321: eval: epoch 110 | batch 500/510 [I 2022-10-11 16:11:35] ...trap/engines/engine.py.325: elapsed: 0:03:10 | left: 0:00:03 [I 2022-10-11 16:11:35] ...trap/engines/engine.py.326: process: 0.31624 | load: 0.00020 [I 2022-10-11 16:11:38] ...trap/engines/engine.py.321: eval: epoch 110 | batch 510/510 [I 2022-10-11 16:11:38] ...trap/engines/engine.py.325: elapsed: 0:03:13 | left: 0:00:00 [I 2022-10-11 16:11:38] ...trap/engines/engine.py.326: process: 0.31416 | load: 0.00015 [S 2022-10-11 16:11:39] ...trap/engines/engine.py.132: eval_epoch.epoch: 110 [I 2022-10-11 16:11:39] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.751800 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909600 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935000 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.742900 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910300 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935800 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013090 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007172 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007772 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009575 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010040 [S 2022-10-11 16:12:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008183 [I 2022-10-11 16:12:22] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-11 16:12:22] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 16:12:23] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 16:12:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 16:12:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 16:12:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 16:12:31] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 16:12:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 16:12:34] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 16:12:34] ...trap/engines/engine.py.470: Saving model... [I 2022-10-11 16:12:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 16:12:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 16:12:42] ...trap/engines/engine.py.163: Training model on trainset for epoch 111 [I 2022-10-11 16:12:42] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 16:12:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 0/2304 [I 2022-10-11 16:12:45] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:01:44 [I 2022-10-11 16:12:45] ...trap/engines/engine.py.229: process: 3.17039 | load: 2.25858 [I 2022-10-11 16:12:45] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-11 16:12:45] ...optimizers/trijoint.py.151: total_norm: 17.151293 [I 2022-10-11 16:12:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 10/2304 [I 2022-10-11 16:12:53] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:05 [I 2022-10-11 16:12:53] ...trap/engines/engine.py.229: process: 0.85419 | load: 0.00486 [I 2022-10-11 16:12:53] ...trap/engines/engine.py.230: loss: 0.10749 [I 2022-10-11 16:12:53] ...optimizers/trijoint.py.151: total_norm: 14.112800 [I 2022-10-11 16:13:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 20/2304 [I 2022-10-11 16:13:02] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:10 [I 2022-10-11 16:13:02] ...trap/engines/engine.py.229: process: 0.83466 | load: 0.00459 [I 2022-10-11 16:13:02] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-11 16:13:02] ...optimizers/trijoint.py.151: total_norm: 17.703083 [I 2022-10-11 16:13:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 30/2304 [I 2022-10-11 16:13:10] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:21 [I 2022-10-11 16:13:10] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00468 [I 2022-10-11 16:13:10] ...trap/engines/engine.py.230: loss: 0.16203 [I 2022-10-11 16:13:10] ...optimizers/trijoint.py.151: total_norm: 13.241068 [I 2022-10-11 16:13:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 40/2304 [I 2022-10-11 16:13:19] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:03 [I 2022-10-11 16:13:19] ...trap/engines/engine.py.229: process: 0.84613 | load: 0.00342 [I 2022-10-11 16:13:19] ...trap/engines/engine.py.230: loss: 0.06995 [I 2022-10-11 16:13:19] ...optimizers/trijoint.py.151: total_norm: 14.672270 [I 2022-10-11 16:13:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 50/2304 [I 2022-10-11 16:13:27] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:48 [I 2022-10-11 16:13:27] ...trap/engines/engine.py.229: process: 0.85737 | load: 0.00407 [I 2022-10-11 16:13:27] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-11 16:13:27] ...optimizers/trijoint.py.151: total_norm: 27.022602 [I 2022-10-11 16:13:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 60/2304 [I 2022-10-11 16:13:36] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:40 [I 2022-10-11 16:13:36] ...trap/engines/engine.py.229: process: 0.83653 | load: 0.00351 [I 2022-10-11 16:13:36] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-11 16:13:36] ...optimizers/trijoint.py.151: total_norm: 9.442219 [I 2022-10-11 16:13:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 70/2304 [I 2022-10-11 16:13:44] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:44 [I 2022-10-11 16:13:44] ...trap/engines/engine.py.229: process: 0.86625 | load: 0.00369 [I 2022-10-11 16:13:44] ...trap/engines/engine.py.230: loss: 0.08730 [I 2022-10-11 16:13:44] ...optimizers/trijoint.py.151: total_norm: 10.080705 [I 2022-10-11 16:13:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 80/2304 [I 2022-10-11 16:13:53] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:50 [I 2022-10-11 16:13:53] ...trap/engines/engine.py.229: process: 0.85151 | load: 0.00322 [I 2022-10-11 16:13:53] ...trap/engines/engine.py.230: loss: 0.09587 [I 2022-10-11 16:13:53] ...optimizers/trijoint.py.151: total_norm: 12.043185 [I 2022-10-11 16:14:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 90/2304 [I 2022-10-11 16:14:02] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:56 [I 2022-10-11 16:14:02] ...trap/engines/engine.py.229: process: 0.87458 | load: 0.00354 [I 2022-10-11 16:14:02] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-11 16:14:02] ...optimizers/trijoint.py.151: total_norm: 32.469330 [I 2022-10-11 16:14:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 100/2304 [I 2022-10-11 16:14:10] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:53 [I 2022-10-11 16:14:10] ...trap/engines/engine.py.229: process: 0.88590 | load: 0.00355 [I 2022-10-11 16:14:10] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 16:14:10] ...optimizers/trijoint.py.151: total_norm: 11.406029 [I 2022-10-11 16:14:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 110/2304 [I 2022-10-11 16:14:19] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:55 [I 2022-10-11 16:14:19] ...trap/engines/engine.py.229: process: 0.84928 | load: 0.00395 [I 2022-10-11 16:14:19] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-11 16:14:19] ...optimizers/trijoint.py.151: total_norm: 17.524298 [I 2022-10-11 16:14:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 120/2304 [I 2022-10-11 16:14:28] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:08 [I 2022-10-11 16:14:28] ...trap/engines/engine.py.229: process: 0.85023 | load: 0.00367 [I 2022-10-11 16:14:28] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-11 16:14:28] ...optimizers/trijoint.py.151: total_norm: 55.248405 [I 2022-10-11 16:14:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 130/2304 [I 2022-10-11 16:14:36] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:03 [I 2022-10-11 16:14:36] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00377 [I 2022-10-11 16:14:36] ...trap/engines/engine.py.230: loss: 0.18107 [I 2022-10-11 16:14:36] ...optimizers/trijoint.py.151: total_norm: 21.074324 [I 2022-10-11 16:14:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 140/2304 [I 2022-10-11 16:14:45] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:57 [I 2022-10-11 16:14:45] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00329 [I 2022-10-11 16:14:45] ...trap/engines/engine.py.230: loss: 0.10080 [I 2022-10-11 16:14:45] ...optimizers/trijoint.py.151: total_norm: 12.870695 [I 2022-10-11 16:14:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 150/2304 [I 2022-10-11 16:14:54] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:34 [I 2022-10-11 16:14:54] ...trap/engines/engine.py.229: process: 0.84638 | load: 0.00331 [I 2022-10-11 16:14:54] ...trap/engines/engine.py.230: loss: 0.15321 [I 2022-10-11 16:14:54] ...optimizers/trijoint.py.151: total_norm: 27.491247 [I 2022-10-11 16:15:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 160/2304 [I 2022-10-11 16:15:02] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:43 [I 2022-10-11 16:15:02] ...trap/engines/engine.py.229: process: 0.85145 | load: 0.00353 [I 2022-10-11 16:15:02] ...trap/engines/engine.py.230: loss: 0.13453 [I 2022-10-11 16:15:02] ...optimizers/trijoint.py.151: total_norm: 29.329590 [I 2022-10-11 16:15:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 170/2304 [I 2022-10-11 16:15:11] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:37 [I 2022-10-11 16:15:11] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00313 [I 2022-10-11 16:15:11] ...trap/engines/engine.py.230: loss: 0.09707 [I 2022-10-11 16:15:11] ...optimizers/trijoint.py.151: total_norm: 11.416548 [I 2022-10-11 16:15:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 180/2304 [I 2022-10-11 16:15:19] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:18 [I 2022-10-11 16:15:19] ...trap/engines/engine.py.229: process: 0.84675 | load: 0.00329 [I 2022-10-11 16:15:19] ...trap/engines/engine.py.230: loss: 0.11522 [I 2022-10-11 16:15:19] ...optimizers/trijoint.py.151: total_norm: 8.624559 [I 2022-10-11 16:15:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 190/2304 [I 2022-10-11 16:15:28] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:09 [I 2022-10-11 16:15:28] ...trap/engines/engine.py.229: process: 0.85745 | load: 0.00326 [I 2022-10-11 16:15:28] ...trap/engines/engine.py.230: loss: 0.17141 [I 2022-10-11 16:15:28] ...optimizers/trijoint.py.151: total_norm: 17.621275 [I 2022-10-11 16:15:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 200/2304 [I 2022-10-11 16:15:36] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:51 [I 2022-10-11 16:15:36] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00329 [I 2022-10-11 16:15:36] ...trap/engines/engine.py.230: loss: 0.07663 [I 2022-10-11 16:15:36] ...optimizers/trijoint.py.151: total_norm: 11.975312 [I 2022-10-11 16:15:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 210/2304 [I 2022-10-11 16:15:45] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:30:01 [I 2022-10-11 16:15:45] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00327 [I 2022-10-11 16:15:45] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-11 16:15:45] ...optimizers/trijoint.py.151: total_norm: 14.746975 [I 2022-10-11 16:15:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 220/2304 [I 2022-10-11 16:15:54] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:33 [I 2022-10-11 16:15:54] ...trap/engines/engine.py.229: process: 0.84676 | load: 0.00357 [I 2022-10-11 16:15:54] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-11 16:15:54] ...optimizers/trijoint.py.151: total_norm: 46.422688 [I 2022-10-11 16:16:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 230/2304 [I 2022-10-11 16:16:02] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:51 [I 2022-10-11 16:16:02] ...trap/engines/engine.py.229: process: 0.87379 | load: 0.00312 [I 2022-10-11 16:16:02] ...trap/engines/engine.py.230: loss: 0.08396 [I 2022-10-11 16:16:02] ...optimizers/trijoint.py.151: total_norm: 15.668949 [I 2022-10-11 16:16:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 240/2304 [I 2022-10-11 16:16:11] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:29:37 [I 2022-10-11 16:16:11] ...trap/engines/engine.py.229: process: 0.86004 | load: 0.00310 [I 2022-10-11 16:16:11] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-11 16:16:11] ...optimizers/trijoint.py.151: total_norm: 17.022051 [I 2022-10-11 16:16:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 250/2304 [I 2022-10-11 16:16:19] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:18 [I 2022-10-11 16:16:19] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00313 [I 2022-10-11 16:16:19] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-11 16:16:19] ...optimizers/trijoint.py.151: total_norm: 13.865449 [I 2022-10-11 16:16:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 260/2304 [I 2022-10-11 16:16:28] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:29:14 [I 2022-10-11 16:16:28] ...trap/engines/engine.py.229: process: 0.84830 | load: 0.00333 [I 2022-10-11 16:16:28] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-11 16:16:28] ...optimizers/trijoint.py.151: total_norm: 13.795216 [I 2022-10-11 16:16:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 270/2304 [I 2022-10-11 16:16:37] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:00 [I 2022-10-11 16:16:37] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00313 [I 2022-10-11 16:16:37] ...trap/engines/engine.py.230: loss: 0.09610 [I 2022-10-11 16:16:37] ...optimizers/trijoint.py.151: total_norm: 10.404243 [I 2022-10-11 16:16:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 280/2304 [I 2022-10-11 16:16:45] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:57 [I 2022-10-11 16:16:45] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00312 [I 2022-10-11 16:16:45] ...trap/engines/engine.py.230: loss: 0.19348 [I 2022-10-11 16:16:45] ...optimizers/trijoint.py.151: total_norm: 20.568676 [I 2022-10-11 16:16:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 290/2304 [I 2022-10-11 16:16:54] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:28:34 [I 2022-10-11 16:16:54] ...trap/engines/engine.py.229: process: 0.84747 | load: 0.00313 [I 2022-10-11 16:16:54] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-11 16:16:54] ...optimizers/trijoint.py.151: total_norm: 30.021418 [I 2022-10-11 16:17:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 300/2304 [I 2022-10-11 16:17:02] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:40 [I 2022-10-11 16:17:02] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00313 [I 2022-10-11 16:17:02] ...trap/engines/engine.py.230: loss: 0.04750 [I 2022-10-11 16:17:02] ...optimizers/trijoint.py.151: total_norm: 30.141415 [I 2022-10-11 16:17:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 310/2304 [I 2022-10-11 16:17:11] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:28:41 [I 2022-10-11 16:17:11] ...trap/engines/engine.py.229: process: 0.88317 | load: 0.00315 [I 2022-10-11 16:17:11] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-11 16:17:11] ...optimizers/trijoint.py.151: total_norm: 16.120853 [I 2022-10-11 16:17:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 320/2304 [I 2022-10-11 16:17:19] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:08 [I 2022-10-11 16:17:19] ...trap/engines/engine.py.229: process: 0.84377 | load: 0.00315 [I 2022-10-11 16:17:20] ...trap/engines/engine.py.230: loss: 0.09628 [I 2022-10-11 16:17:20] ...optimizers/trijoint.py.151: total_norm: 16.315636 [I 2022-10-11 16:17:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 330/2304 [I 2022-10-11 16:17:28] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:28:00 [I 2022-10-11 16:17:28] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00316 [I 2022-10-11 16:17:28] ...trap/engines/engine.py.230: loss: 0.10802 [I 2022-10-11 16:17:28] ...optimizers/trijoint.py.151: total_norm: 18.516449 [I 2022-10-11 16:17:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 340/2304 [I 2022-10-11 16:17:37] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:27:59 [I 2022-10-11 16:17:37] ...trap/engines/engine.py.229: process: 0.84915 | load: 0.00308 [I 2022-10-11 16:17:37] ...trap/engines/engine.py.230: loss: 0.10908 [I 2022-10-11 16:17:37] ...optimizers/trijoint.py.151: total_norm: 15.800602 [I 2022-10-11 16:17:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 350/2304 [I 2022-10-11 16:17:45] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:27:54 [I 2022-10-11 16:17:45] ...trap/engines/engine.py.229: process: 0.84980 | load: 0.00295 [I 2022-10-11 16:17:45] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-11 16:17:45] ...optimizers/trijoint.py.151: total_norm: 15.043341 [I 2022-10-11 16:17:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 360/2304 [I 2022-10-11 16:17:54] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:20 [I 2022-10-11 16:17:54] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00295 [I 2022-10-11 16:17:54] ...trap/engines/engine.py.230: loss: 0.10157 [I 2022-10-11 16:17:54] ...optimizers/trijoint.py.151: total_norm: 15.559968 [I 2022-10-11 16:18:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 370/2304 [I 2022-10-11 16:18:02] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:26 [I 2022-10-11 16:18:02] ...trap/engines/engine.py.229: process: 0.84504 | load: 0.00294 [I 2022-10-11 16:18:02] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-11 16:18:02] ...optimizers/trijoint.py.151: total_norm: 17.214043 [I 2022-10-11 16:18:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 380/2304 [I 2022-10-11 16:18:11] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:12 [I 2022-10-11 16:18:11] ...trap/engines/engine.py.229: process: 0.86320 | load: 0.00294 [I 2022-10-11 16:18:11] ...trap/engines/engine.py.230: loss: 0.14233 [I 2022-10-11 16:18:11] ...optimizers/trijoint.py.151: total_norm: 33.690426 [I 2022-10-11 16:18:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 390/2304 [I 2022-10-11 16:18:19] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:04 [I 2022-10-11 16:18:19] ...trap/engines/engine.py.229: process: 0.85285 | load: 0.00294 [I 2022-10-11 16:18:19] ...trap/engines/engine.py.230: loss: 0.08050 [I 2022-10-11 16:18:19] ...optimizers/trijoint.py.151: total_norm: 9.953202 [I 2022-10-11 16:18:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 400/2304 [I 2022-10-11 16:18:28] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:09 [I 2022-10-11 16:18:28] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00311 [I 2022-10-11 16:18:28] ...trap/engines/engine.py.230: loss: 0.08574 [I 2022-10-11 16:18:28] ...optimizers/trijoint.py.151: total_norm: 10.908837 [I 2022-10-11 16:18:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 410/2304 [I 2022-10-11 16:18:36] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:26:42 [I 2022-10-11 16:18:36] ...trap/engines/engine.py.229: process: 0.84476 | load: 0.00293 [I 2022-10-11 16:18:36] ...trap/engines/engine.py.230: loss: 0.13909 [I 2022-10-11 16:18:36] ...optimizers/trijoint.py.151: total_norm: 16.688383 [I 2022-10-11 16:18:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 420/2304 [I 2022-10-11 16:18:45] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:26:48 [I 2022-10-11 16:18:45] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00294 [I 2022-10-11 16:18:45] ...trap/engines/engine.py.230: loss: 0.12779 [I 2022-10-11 16:18:45] ...optimizers/trijoint.py.151: total_norm: 9.386828 [I 2022-10-11 16:18:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 430/2304 [I 2022-10-11 16:18:53] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:32 [I 2022-10-11 16:18:53] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00299 [I 2022-10-11 16:18:53] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-11 16:18:53] ...optimizers/trijoint.py.151: total_norm: 14.927242 [I 2022-10-11 16:19:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 440/2304 [I 2022-10-11 16:19:02] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:31 [I 2022-10-11 16:19:02] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00299 [I 2022-10-11 16:19:02] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-11 16:19:02] ...optimizers/trijoint.py.151: total_norm: 22.710960 [I 2022-10-11 16:19:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 450/2304 [I 2022-10-11 16:19:10] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:26:06 [I 2022-10-11 16:19:10] ...trap/engines/engine.py.229: process: 0.83582 | load: 0.00302 [I 2022-10-11 16:19:10] ...trap/engines/engine.py.230: loss: 0.08594 [I 2022-10-11 16:19:10] ...optimizers/trijoint.py.151: total_norm: 11.853951 [I 2022-10-11 16:19:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 460/2304 [I 2022-10-11 16:19:19] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:26:05 [I 2022-10-11 16:19:19] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00298 [I 2022-10-11 16:19:19] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-11 16:19:19] ...optimizers/trijoint.py.151: total_norm: 17.754745 [I 2022-10-11 16:19:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 470/2304 [I 2022-10-11 16:19:27] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:25:45 [I 2022-10-11 16:19:27] ...trap/engines/engine.py.229: process: 0.84367 | load: 0.00296 [I 2022-10-11 16:19:27] ...trap/engines/engine.py.230: loss: 0.12058 [I 2022-10-11 16:19:27] ...optimizers/trijoint.py.151: total_norm: 20.094189 [I 2022-10-11 16:19:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 480/2304 [I 2022-10-11 16:19:36] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:25:50 [I 2022-10-11 16:19:36] ...trap/engines/engine.py.229: process: 0.85433 | load: 0.00295 [I 2022-10-11 16:19:36] ...trap/engines/engine.py.230: loss: 0.13868 [I 2022-10-11 16:19:36] ...optimizers/trijoint.py.151: total_norm: 17.269386 [I 2022-10-11 16:19:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 490/2304 [I 2022-10-11 16:19:44] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:25:28 [I 2022-10-11 16:19:44] ...trap/engines/engine.py.229: process: 0.82949 | load: 0.00295 [I 2022-10-11 16:19:44] ...trap/engines/engine.py.230: loss: 0.09019 [I 2022-10-11 16:19:44] ...optimizers/trijoint.py.151: total_norm: 11.562724 [I 2022-10-11 16:19:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 500/2304 [I 2022-10-11 16:19:53] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:25:14 [I 2022-10-11 16:19:53] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00297 [I 2022-10-11 16:19:53] ...trap/engines/engine.py.230: loss: 0.13188 [I 2022-10-11 16:19:53] ...optimizers/trijoint.py.151: total_norm: 15.416168 [I 2022-10-11 16:20:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 510/2304 [I 2022-10-11 16:20:01] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:25:12 [I 2022-10-11 16:20:01] ...trap/engines/engine.py.229: process: 0.84090 | load: 0.00297 [I 2022-10-11 16:20:01] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-11 16:20:01] ...optimizers/trijoint.py.151: total_norm: 33.987686 [I 2022-10-11 16:20:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 520/2304 [I 2022-10-11 16:20:10] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:25:46 [I 2022-10-11 16:20:10] ...trap/engines/engine.py.229: process: 0.87603 | load: 0.00382 [I 2022-10-11 16:20:10] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-11 16:20:10] ...optimizers/trijoint.py.151: total_norm: 26.489372 [I 2022-10-11 16:20:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 530/2304 [I 2022-10-11 16:20:18] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:25:39 [I 2022-10-11 16:20:18] ...trap/engines/engine.py.229: process: 0.86279 | load: 0.00382 [I 2022-10-11 16:20:18] ...trap/engines/engine.py.230: loss: 0.10129 [I 2022-10-11 16:20:18] ...optimizers/trijoint.py.151: total_norm: 9.726209 [I 2022-10-11 16:20:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 540/2304 [I 2022-10-11 16:20:27] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:25:47 [I 2022-10-11 16:20:27] ...trap/engines/engine.py.229: process: 0.87301 | load: 0.00373 [I 2022-10-11 16:20:27] ...trap/engines/engine.py.230: loss: 0.11123 [I 2022-10-11 16:20:27] ...optimizers/trijoint.py.151: total_norm: 25.400240 [I 2022-10-11 16:20:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 550/2304 [I 2022-10-11 16:20:36] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:25:31 [I 2022-10-11 16:20:36] ...trap/engines/engine.py.229: process: 0.87785 | load: 0.00377 [I 2022-10-11 16:20:36] ...trap/engines/engine.py.230: loss: 0.14243 [I 2022-10-11 16:20:36] ...optimizers/trijoint.py.151: total_norm: 16.070845 [I 2022-10-11 16:20:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 560/2304 [I 2022-10-11 16:20:45] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:25:30 [I 2022-10-11 16:20:45] ...trap/engines/engine.py.229: process: 0.88058 | load: 0.00373 [I 2022-10-11 16:20:45] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-11 16:20:45] ...optimizers/trijoint.py.151: total_norm: 10.559738 [I 2022-10-11 16:20:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 570/2304 [I 2022-10-11 16:20:54] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:25:15 [I 2022-10-11 16:20:54] ...trap/engines/engine.py.229: process: 0.88026 | load: 0.00376 [I 2022-10-11 16:20:54] ...trap/engines/engine.py.230: loss: 0.14085 [I 2022-10-11 16:20:54] ...optimizers/trijoint.py.151: total_norm: 15.413089 [I 2022-10-11 16:21:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 580/2304 [I 2022-10-11 16:21:02] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:24:25 [I 2022-10-11 16:21:02] ...trap/engines/engine.py.229: process: 0.85775 | load: 0.00294 [I 2022-10-11 16:21:02] ...trap/engines/engine.py.230: loss: 0.08586 [I 2022-10-11 16:21:02] ...optimizers/trijoint.py.151: total_norm: 11.039037 [I 2022-10-11 16:21:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 590/2304 [I 2022-10-11 16:21:10] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:24:09 [I 2022-10-11 16:21:10] ...trap/engines/engine.py.229: process: 0.85562 | load: 0.00295 [I 2022-10-11 16:21:10] ...trap/engines/engine.py.230: loss: 0.11196 [I 2022-10-11 16:21:10] ...optimizers/trijoint.py.151: total_norm: 37.466095 [I 2022-10-11 16:21:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 600/2304 [I 2022-10-11 16:21:19] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:24:02 [I 2022-10-11 16:21:19] ...trap/engines/engine.py.229: process: 0.84265 | load: 0.00296 [I 2022-10-11 16:21:19] ...trap/engines/engine.py.230: loss: 0.10518 [I 2022-10-11 16:21:19] ...optimizers/trijoint.py.151: total_norm: 25.341925 [I 2022-10-11 16:21:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 610/2304 [I 2022-10-11 16:21:27] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:24:05 [I 2022-10-11 16:21:27] ...trap/engines/engine.py.229: process: 0.85760 | load: 0.00297 [I 2022-10-11 16:21:27] ...trap/engines/engine.py.230: loss: 0.06499 [I 2022-10-11 16:21:27] ...optimizers/trijoint.py.151: total_norm: 11.196997 [I 2022-10-11 16:21:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 620/2304 [I 2022-10-11 16:21:36] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:23:40 [I 2022-10-11 16:21:36] ...trap/engines/engine.py.229: process: 0.84337 | load: 0.00298 [I 2022-10-11 16:21:36] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-11 16:21:36] ...optimizers/trijoint.py.151: total_norm: 15.698300 [I 2022-10-11 16:21:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 630/2304 [I 2022-10-11 16:21:44] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:23:30 [I 2022-10-11 16:21:44] ...trap/engines/engine.py.229: process: 0.83182 | load: 0.00301 [I 2022-10-11 16:21:44] ...trap/engines/engine.py.230: loss: 0.07770 [I 2022-10-11 16:21:44] ...optimizers/trijoint.py.151: total_norm: 7.811033 [I 2022-10-11 16:21:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 640/2304 [I 2022-10-11 16:21:53] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:23:25 [I 2022-10-11 16:21:53] ...trap/engines/engine.py.229: process: 0.83379 | load: 0.00297 [I 2022-10-11 16:21:53] ...trap/engines/engine.py.230: loss: 0.15290 [I 2022-10-11 16:21:53] ...optimizers/trijoint.py.151: total_norm: 20.749596 [I 2022-10-11 16:22:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 650/2304 [I 2022-10-11 16:22:01] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:23:22 [I 2022-10-11 16:22:01] ...trap/engines/engine.py.229: process: 0.85111 | load: 0.00311 [I 2022-10-11 16:22:01] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-11 16:22:01] ...optimizers/trijoint.py.151: total_norm: 12.959744 [I 2022-10-11 16:22:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 660/2304 [I 2022-10-11 16:22:10] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:23:19 [I 2022-10-11 16:22:10] ...trap/engines/engine.py.229: process: 0.85602 | load: 0.00311 [I 2022-10-11 16:22:10] ...trap/engines/engine.py.230: loss: 0.10284 [I 2022-10-11 16:22:10] ...optimizers/trijoint.py.151: total_norm: 15.181855 [I 2022-10-11 16:22:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 670/2304 [I 2022-10-11 16:22:18] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:23:18 [I 2022-10-11 16:22:18] ...trap/engines/engine.py.229: process: 0.86433 | load: 0.00312 [I 2022-10-11 16:22:18] ...trap/engines/engine.py.230: loss: 0.09221 [I 2022-10-11 16:22:18] ...optimizers/trijoint.py.151: total_norm: 8.675318 [I 2022-10-11 16:22:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 680/2304 [I 2022-10-11 16:22:27] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:23:04 [I 2022-10-11 16:22:27] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00310 [I 2022-10-11 16:22:27] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-11 16:22:27] ...optimizers/trijoint.py.151: total_norm: 26.104450 [I 2022-10-11 16:22:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 690/2304 [I 2022-10-11 16:22:36] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:23:02 [I 2022-10-11 16:22:36] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00311 [I 2022-10-11 16:22:36] ...trap/engines/engine.py.230: loss: 0.10103 [I 2022-10-11 16:22:36] ...optimizers/trijoint.py.151: total_norm: 10.929551 [I 2022-10-11 16:22:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 700/2304 [I 2022-10-11 16:22:44] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:22:53 [I 2022-10-11 16:22:44] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00310 [I 2022-10-11 16:22:44] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-11 16:22:44] ...optimizers/trijoint.py.151: total_norm: 9.183652 [I 2022-10-11 16:22:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 710/2304 [I 2022-10-11 16:22:53] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:22:36 [I 2022-10-11 16:22:53] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00311 [I 2022-10-11 16:22:53] ...trap/engines/engine.py.230: loss: 0.09847 [I 2022-10-11 16:22:53] ...optimizers/trijoint.py.151: total_norm: 11.543133 [I 2022-10-11 16:23:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 720/2304 [I 2022-10-11 16:23:01] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:22:28 [I 2022-10-11 16:23:01] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00314 [I 2022-10-11 16:23:01] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-11 16:23:01] ...optimizers/trijoint.py.151: total_norm: 7.932688 [I 2022-10-11 16:23:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 730/2304 [I 2022-10-11 16:23:10] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:22:26 [I 2022-10-11 16:23:10] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00312 [I 2022-10-11 16:23:10] ...trap/engines/engine.py.230: loss: 0.17253 [I 2022-10-11 16:23:10] ...optimizers/trijoint.py.151: total_norm: 15.468230 [I 2022-10-11 16:23:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 740/2304 [I 2022-10-11 16:23:18] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:22:23 [I 2022-10-11 16:23:18] ...trap/engines/engine.py.229: process: 0.85494 | load: 0.00315 [I 2022-10-11 16:23:18] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-11 16:23:18] ...optimizers/trijoint.py.151: total_norm: 13.231133 [I 2022-10-11 16:23:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 750/2304 [I 2022-10-11 16:23:27] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:22:28 [I 2022-10-11 16:23:27] ...trap/engines/engine.py.229: process: 0.87207 | load: 0.00312 [I 2022-10-11 16:23:27] ...trap/engines/engine.py.230: loss: 0.09312 [I 2022-10-11 16:23:27] ...optimizers/trijoint.py.151: total_norm: 13.108741 [I 2022-10-11 16:23:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 760/2304 [I 2022-10-11 16:23:36] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:22:13 [I 2022-10-11 16:23:36] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00314 [I 2022-10-11 16:23:36] ...trap/engines/engine.py.230: loss: 0.10304 [I 2022-10-11 16:23:36] ...optimizers/trijoint.py.151: total_norm: 33.905052 [I 2022-10-11 16:23:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 770/2304 [I 2022-10-11 16:23:44] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:22:07 [I 2022-10-11 16:23:44] ...trap/engines/engine.py.229: process: 0.87125 | load: 0.00311 [I 2022-10-11 16:23:44] ...trap/engines/engine.py.230: loss: 0.09859 [I 2022-10-11 16:23:44] ...optimizers/trijoint.py.151: total_norm: 19.087811 [I 2022-10-11 16:23:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 780/2304 [I 2022-10-11 16:23:53] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:21:52 [I 2022-10-11 16:23:53] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00313 [I 2022-10-11 16:23:53] ...trap/engines/engine.py.230: loss: 0.08708 [I 2022-10-11 16:23:53] ...optimizers/trijoint.py.151: total_norm: 14.544311 [I 2022-10-11 16:24:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 790/2304 [I 2022-10-11 16:24:02] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:21:48 [I 2022-10-11 16:24:02] ...trap/engines/engine.py.229: process: 0.87106 | load: 0.00309 [I 2022-10-11 16:24:02] ...trap/engines/engine.py.230: loss: 0.15073 [I 2022-10-11 16:24:02] ...optimizers/trijoint.py.151: total_norm: 21.837063 [I 2022-10-11 16:24:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 800/2304 [I 2022-10-11 16:24:10] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:21:33 [I 2022-10-11 16:24:10] ...trap/engines/engine.py.229: process: 0.86064 | load: 0.00311 [I 2022-10-11 16:24:10] ...trap/engines/engine.py.230: loss: 0.20736 [I 2022-10-11 16:24:10] ...optimizers/trijoint.py.151: total_norm: 26.098577 [I 2022-10-11 16:24:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 810/2304 [I 2022-10-11 16:24:19] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:21:52 [I 2022-10-11 16:24:19] ...trap/engines/engine.py.229: process: 0.89005 | load: 0.00393 [I 2022-10-11 16:24:19] ...trap/engines/engine.py.230: loss: 0.13105 [I 2022-10-11 16:24:19] ...optimizers/trijoint.py.151: total_norm: 14.975321 [I 2022-10-11 16:24:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 820/2304 [I 2022-10-11 16:24:28] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:21:48 [I 2022-10-11 16:24:28] ...trap/engines/engine.py.229: process: 0.87552 | load: 0.00392 [I 2022-10-11 16:24:28] ...trap/engines/engine.py.230: loss: 0.21584 [I 2022-10-11 16:24:28] ...optimizers/trijoint.py.151: total_norm: 29.372393 [I 2022-10-11 16:24:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 830/2304 [I 2022-10-11 16:24:37] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:21:49 [I 2022-10-11 16:24:37] ...trap/engines/engine.py.229: process: 0.88985 | load: 0.00396 [I 2022-10-11 16:24:37] ...trap/engines/engine.py.230: loss: 0.10050 [I 2022-10-11 16:24:37] ...optimizers/trijoint.py.151: total_norm: 14.158893 [I 2022-10-11 16:24:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 840/2304 [I 2022-10-11 16:24:46] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:21:41 [I 2022-10-11 16:24:46] ...trap/engines/engine.py.229: process: 0.89080 | load: 0.00396 [I 2022-10-11 16:24:46] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-11 16:24:46] ...optimizers/trijoint.py.151: total_norm: 15.374106 [I 2022-10-11 16:24:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 850/2304 [I 2022-10-11 16:24:55] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:21:26 [I 2022-10-11 16:24:55] ...trap/engines/engine.py.229: process: 0.88848 | load: 0.00396 [I 2022-10-11 16:24:55] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-11 16:24:55] ...optimizers/trijoint.py.151: total_norm: 21.721085 [I 2022-10-11 16:25:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 860/2304 [I 2022-10-11 16:25:03] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:21:06 [I 2022-10-11 16:25:03] ...trap/engines/engine.py.229: process: 0.86141 | load: 0.00309 [I 2022-10-11 16:25:03] ...trap/engines/engine.py.230: loss: 0.09390 [I 2022-10-11 16:25:03] ...optimizers/trijoint.py.151: total_norm: 16.844418 [I 2022-10-11 16:25:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 870/2304 [I 2022-10-11 16:25:12] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:26 [I 2022-10-11 16:25:12] ...trap/engines/engine.py.229: process: 0.86636 | load: 0.00314 [I 2022-10-11 16:25:12] ...trap/engines/engine.py.230: loss: 0.08277 [I 2022-10-11 16:25:12] ...optimizers/trijoint.py.151: total_norm: 11.743642 [I 2022-10-11 16:25:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 880/2304 [I 2022-10-11 16:25:21] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:20:28 [I 2022-10-11 16:25:21] ...trap/engines/engine.py.229: process: 0.85293 | load: 0.00313 [I 2022-10-11 16:25:21] ...trap/engines/engine.py.230: loss: 0.09023 [I 2022-10-11 16:25:21] ...optimizers/trijoint.py.151: total_norm: 15.753892 [I 2022-10-11 16:25:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 890/2304 [I 2022-10-11 16:25:29] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:13 [I 2022-10-11 16:25:29] ...trap/engines/engine.py.229: process: 0.85486 | load: 0.00313 [I 2022-10-11 16:25:29] ...trap/engines/engine.py.230: loss: 0.09175 [I 2022-10-11 16:25:29] ...optimizers/trijoint.py.151: total_norm: 18.747122 [I 2022-10-11 16:25:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 900/2304 [I 2022-10-11 16:25:38] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:20:02 [I 2022-10-11 16:25:38] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00313 [I 2022-10-11 16:25:38] ...trap/engines/engine.py.230: loss: 0.08364 [I 2022-10-11 16:25:38] ...optimizers/trijoint.py.151: total_norm: 13.105929 [I 2022-10-11 16:25:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 910/2304 [I 2022-10-11 16:25:46] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:19:43 [I 2022-10-11 16:25:46] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00311 [I 2022-10-11 16:25:46] ...trap/engines/engine.py.230: loss: 0.10124 [I 2022-10-11 16:25:46] ...optimizers/trijoint.py.151: total_norm: 14.179894 [I 2022-10-11 16:25:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 920/2304 [I 2022-10-11 16:25:55] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:19:44 [I 2022-10-11 16:25:55] ...trap/engines/engine.py.229: process: 0.85576 | load: 0.00318 [I 2022-10-11 16:25:55] ...trap/engines/engine.py.230: loss: 0.08346 [I 2022-10-11 16:25:55] ...optimizers/trijoint.py.151: total_norm: 14.631161 [I 2022-10-11 16:26:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 930/2304 [I 2022-10-11 16:26:03] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:19:32 [I 2022-10-11 16:26:03] ...trap/engines/engine.py.229: process: 0.85118 | load: 0.00308 [I 2022-10-11 16:26:03] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-11 16:26:03] ...optimizers/trijoint.py.151: total_norm: 11.215833 [I 2022-10-11 16:26:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 940/2304 [I 2022-10-11 16:26:12] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:19:06 [I 2022-10-11 16:26:12] ...trap/engines/engine.py.229: process: 0.83468 | load: 0.00297 [I 2022-10-11 16:26:12] ...trap/engines/engine.py.230: loss: 0.08649 [I 2022-10-11 16:26:12] ...optimizers/trijoint.py.151: total_norm: 10.598226 [I 2022-10-11 16:26:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 950/2304 [I 2022-10-11 16:26:20] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:19:03 [I 2022-10-11 16:26:20] ...trap/engines/engine.py.229: process: 0.83774 | load: 0.00299 [I 2022-10-11 16:26:20] ...trap/engines/engine.py.230: loss: 0.09139 [I 2022-10-11 16:26:20] ...optimizers/trijoint.py.151: total_norm: 14.021687 [I 2022-10-11 16:26:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 960/2304 [I 2022-10-11 16:26:29] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:19:09 [I 2022-10-11 16:26:29] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00293 [I 2022-10-11 16:26:29] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-11 16:26:29] ...optimizers/trijoint.py.151: total_norm: 22.528830 [I 2022-10-11 16:26:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 970/2304 [I 2022-10-11 16:26:37] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:18:47 [I 2022-10-11 16:26:37] ...trap/engines/engine.py.229: process: 0.82593 | load: 0.00295 [I 2022-10-11 16:26:37] ...trap/engines/engine.py.230: loss: 0.17702 [I 2022-10-11 16:26:37] ...optimizers/trijoint.py.151: total_norm: 20.526823 [I 2022-10-11 16:26:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 980/2304 [I 2022-10-11 16:26:46] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:18:35 [I 2022-10-11 16:26:46] ...trap/engines/engine.py.229: process: 0.84243 | load: 0.00291 [I 2022-10-11 16:26:46] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-11 16:26:46] ...optimizers/trijoint.py.151: total_norm: 19.061426 [I 2022-10-11 16:26:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 990/2304 [I 2022-10-11 16:26:54] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:18:21 [I 2022-10-11 16:26:54] ...trap/engines/engine.py.229: process: 0.84215 | load: 0.00293 [I 2022-10-11 16:26:54] ...trap/engines/engine.py.230: loss: 0.18257 [I 2022-10-11 16:26:54] ...optimizers/trijoint.py.151: total_norm: 17.487814 [I 2022-10-11 16:27:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1000/2304 [I 2022-10-11 16:27:03] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:18:22 [I 2022-10-11 16:27:03] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00292 [I 2022-10-11 16:27:03] ...trap/engines/engine.py.230: loss: 0.12794 [I 2022-10-11 16:27:03] ...optimizers/trijoint.py.151: total_norm: 18.224075 [I 2022-10-11 16:27:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1010/2304 [I 2022-10-11 16:27:11] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:18:04 [I 2022-10-11 16:27:11] ...trap/engines/engine.py.229: process: 0.83946 | load: 0.00293 [I 2022-10-11 16:27:11] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-11 16:27:11] ...optimizers/trijoint.py.151: total_norm: 13.358502 [I 2022-10-11 16:27:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1020/2304 [I 2022-10-11 16:27:20] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:18:12 [I 2022-10-11 16:27:20] ...trap/engines/engine.py.229: process: 0.84750 | load: 0.00294 [I 2022-10-11 16:27:20] ...trap/engines/engine.py.230: loss: 0.07181 [I 2022-10-11 16:27:20] ...optimizers/trijoint.py.151: total_norm: 16.909399 [I 2022-10-11 16:27:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 1030/2304 [I 2022-10-11 16:27:28] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:18:03 [I 2022-10-11 16:27:28] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00295 [I 2022-10-11 16:27:28] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-11 16:27:28] ...optimizers/trijoint.py.151: total_norm: 12.735455 [I 2022-10-11 16:27:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1040/2304 [I 2022-10-11 16:27:37] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:17:54 [I 2022-10-11 16:27:37] ...trap/engines/engine.py.229: process: 0.85006 | load: 0.00295 [I 2022-10-11 16:27:37] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-11 16:27:37] ...optimizers/trijoint.py.151: total_norm: 25.532211 [I 2022-10-11 16:27:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 1050/2304 [I 2022-10-11 16:27:45] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:44 [I 2022-10-11 16:27:45] ...trap/engines/engine.py.229: process: 0.83929 | load: 0.00295 [I 2022-10-11 16:27:45] ...trap/engines/engine.py.230: loss: 0.10169 [I 2022-10-11 16:27:45] ...optimizers/trijoint.py.151: total_norm: 8.868827 [I 2022-10-11 16:27:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1060/2304 [I 2022-10-11 16:27:54] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:17:39 [I 2022-10-11 16:27:54] ...trap/engines/engine.py.229: process: 0.85452 | load: 0.00295 [I 2022-10-11 16:27:54] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-11 16:27:54] ...optimizers/trijoint.py.151: total_norm: 9.364554 [I 2022-10-11 16:28:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 1070/2304 [I 2022-10-11 16:28:02] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:33 [I 2022-10-11 16:28:02] ...trap/engines/engine.py.229: process: 0.86324 | load: 0.00298 [I 2022-10-11 16:28:02] ...trap/engines/engine.py.230: loss: 0.08453 [I 2022-10-11 16:28:02] ...optimizers/trijoint.py.151: total_norm: 13.149993 [I 2022-10-11 16:28:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1080/2304 [I 2022-10-11 16:28:11] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:17:21 [I 2022-10-11 16:28:11] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00297 [I 2022-10-11 16:28:11] ...trap/engines/engine.py.230: loss: 0.10058 [I 2022-10-11 16:28:11] ...optimizers/trijoint.py.151: total_norm: 21.162804 [I 2022-10-11 16:28:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 1090/2304 [I 2022-10-11 16:28:19] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:17:18 [I 2022-10-11 16:28:19] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00304 [I 2022-10-11 16:28:19] ...trap/engines/engine.py.230: loss: 0.17054 [I 2022-10-11 16:28:19] ...optimizers/trijoint.py.151: total_norm: 36.101368 [I 2022-10-11 16:28:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 1100/2304 [I 2022-10-11 16:28:28] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:17:29 [I 2022-10-11 16:28:28] ...trap/engines/engine.py.229: process: 0.85716 | load: 0.00379 [I 2022-10-11 16:28:28] ...trap/engines/engine.py.230: loss: 0.13926 [I 2022-10-11 16:28:28] ...optimizers/trijoint.py.151: total_norm: 19.870237 [I 2022-10-11 16:28:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1110/2304 [I 2022-10-11 16:28:37] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:17:18 [I 2022-10-11 16:28:37] ...trap/engines/engine.py.229: process: 0.87510 | load: 0.00381 [I 2022-10-11 16:28:37] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-11 16:28:37] ...optimizers/trijoint.py.151: total_norm: 9.560826 [I 2022-10-11 16:28:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1120/2304 [I 2022-10-11 16:28:46] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:17:13 [I 2022-10-11 16:28:46] ...trap/engines/engine.py.229: process: 0.87122 | load: 0.00376 [I 2022-10-11 16:28:46] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-11 16:28:46] ...optimizers/trijoint.py.151: total_norm: 17.221409 [I 2022-10-11 16:28:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1130/2304 [I 2022-10-11 16:28:54] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:17:12 [I 2022-10-11 16:28:54] ...trap/engines/engine.py.229: process: 0.87897 | load: 0.00378 [I 2022-10-11 16:28:54] ...trap/engines/engine.py.230: loss: 0.15384 [I 2022-10-11 16:28:54] ...optimizers/trijoint.py.151: total_norm: 29.204845 [I 2022-10-11 16:29:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1140/2304 [I 2022-10-11 16:29:03] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:16:53 [I 2022-10-11 16:29:03] ...trap/engines/engine.py.229: process: 0.86696 | load: 0.00378 [I 2022-10-11 16:29:03] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-11 16:29:03] ...optimizers/trijoint.py.151: total_norm: 12.098042 [I 2022-10-11 16:29:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1150/2304 [I 2022-10-11 16:29:12] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:16:54 [I 2022-10-11 16:29:12] ...trap/engines/engine.py.229: process: 0.87500 | load: 0.00404 [I 2022-10-11 16:29:12] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-11 16:29:12] ...optimizers/trijoint.py.151: total_norm: 31.536737 [I 2022-10-11 16:29:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1160/2304 [I 2022-10-11 16:29:21] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:16:25 [I 2022-10-11 16:29:21] ...trap/engines/engine.py.229: process: 0.85052 | load: 0.00312 [I 2022-10-11 16:29:21] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-11 16:29:21] ...optimizers/trijoint.py.151: total_norm: 15.789268 [I 2022-10-11 16:29:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1170/2304 [I 2022-10-11 16:29:29] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:16:07 [I 2022-10-11 16:29:29] ...trap/engines/engine.py.229: process: 0.85729 | load: 0.00310 [I 2022-10-11 16:29:29] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-11 16:29:29] ...optimizers/trijoint.py.151: total_norm: 12.037417 [I 2022-10-11 16:29:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1180/2304 [I 2022-10-11 16:29:38] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:16:09 [I 2022-10-11 16:29:38] ...trap/engines/engine.py.229: process: 0.88882 | load: 0.00315 [I 2022-10-11 16:29:38] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-11 16:29:38] ...optimizers/trijoint.py.151: total_norm: 11.490904 [I 2022-10-11 16:29:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1190/2304 [I 2022-10-11 16:29:46] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:15:52 [I 2022-10-11 16:29:46] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00316 [I 2022-10-11 16:29:46] ...trap/engines/engine.py.230: loss: 0.09307 [I 2022-10-11 16:29:46] ...optimizers/trijoint.py.151: total_norm: 14.782136 [I 2022-10-11 16:29:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1200/2304 [I 2022-10-11 16:29:55] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:15:48 [I 2022-10-11 16:29:55] ...trap/engines/engine.py.229: process: 0.87315 | load: 0.00316 [I 2022-10-11 16:29:55] ...trap/engines/engine.py.230: loss: 0.15895 [I 2022-10-11 16:29:55] ...optimizers/trijoint.py.151: total_norm: 10.789246 [I 2022-10-11 16:30:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1210/2304 [I 2022-10-11 16:30:04] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:15:43 [I 2022-10-11 16:30:04] ...trap/engines/engine.py.229: process: 0.86577 | load: 0.00314 [I 2022-10-11 16:30:04] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-11 16:30:04] ...optimizers/trijoint.py.151: total_norm: 9.393757 [I 2022-10-11 16:30:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1220/2304 [I 2022-10-11 16:30:12] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:32 [I 2022-10-11 16:30:12] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00312 [I 2022-10-11 16:30:12] ...trap/engines/engine.py.230: loss: 0.08454 [I 2022-10-11 16:30:12] ...optimizers/trijoint.py.151: total_norm: 18.696781 [I 2022-10-11 16:30:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1230/2304 [I 2022-10-11 16:30:21] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:24 [I 2022-10-11 16:30:21] ...trap/engines/engine.py.229: process: 0.86853 | load: 0.00320 [I 2022-10-11 16:30:21] ...trap/engines/engine.py.230: loss: 0.09322 [I 2022-10-11 16:30:21] ...optimizers/trijoint.py.151: total_norm: 16.726572 [I 2022-10-11 16:30:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1240/2304 [I 2022-10-11 16:30:29] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:15:11 [I 2022-10-11 16:30:29] ...trap/engines/engine.py.229: process: 0.85794 | load: 0.00314 [I 2022-10-11 16:30:29] ...trap/engines/engine.py.230: loss: 0.12371 [I 2022-10-11 16:30:29] ...optimizers/trijoint.py.151: total_norm: 16.015076 [I 2022-10-11 16:30:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1250/2304 [I 2022-10-11 16:30:38] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:15:03 [I 2022-10-11 16:30:38] ...trap/engines/engine.py.229: process: 0.87187 | load: 0.00311 [I 2022-10-11 16:30:38] ...trap/engines/engine.py.230: loss: 0.10190 [I 2022-10-11 16:30:38] ...optimizers/trijoint.py.151: total_norm: 11.712820 [I 2022-10-11 16:30:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1260/2304 [I 2022-10-11 16:30:47] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:55 [I 2022-10-11 16:30:47] ...trap/engines/engine.py.229: process: 0.85736 | load: 0.00312 [I 2022-10-11 16:30:47] ...trap/engines/engine.py.230: loss: 0.12401 [I 2022-10-11 16:30:47] ...optimizers/trijoint.py.151: total_norm: 21.425732 [I 2022-10-11 16:30:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1270/2304 [I 2022-10-11 16:30:55] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:51 [I 2022-10-11 16:30:55] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00318 [I 2022-10-11 16:30:55] ...trap/engines/engine.py.230: loss: 0.09925 [I 2022-10-11 16:30:55] ...optimizers/trijoint.py.151: total_norm: 9.617686 [I 2022-10-11 16:31:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1280/2304 [I 2022-10-11 16:31:04] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:14:41 [I 2022-10-11 16:31:04] ...trap/engines/engine.py.229: process: 0.84946 | load: 0.00312 [I 2022-10-11 16:31:04] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-11 16:31:04] ...optimizers/trijoint.py.151: total_norm: 16.736813 [I 2022-10-11 16:31:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1290/2304 [I 2022-10-11 16:31:13] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:14:35 [I 2022-10-11 16:31:13] ...trap/engines/engine.py.229: process: 0.87555 | load: 0.00312 [I 2022-10-11 16:31:13] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-11 16:31:13] ...optimizers/trijoint.py.151: total_norm: 35.788677 [I 2022-10-11 16:31:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1300/2304 [I 2022-10-11 16:31:21] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:14:31 [I 2022-10-11 16:31:21] ...trap/engines/engine.py.229: process: 0.86433 | load: 0.00310 [I 2022-10-11 16:31:21] ...trap/engines/engine.py.230: loss: 0.11437 [I 2022-10-11 16:31:21] ...optimizers/trijoint.py.151: total_norm: 15.813620 [I 2022-10-11 16:31:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 1310/2304 [I 2022-10-11 16:31:30] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:24 [I 2022-10-11 16:31:30] ...trap/engines/engine.py.229: process: 0.87320 | load: 0.00313 [I 2022-10-11 16:31:30] ...trap/engines/engine.py.230: loss: 0.13558 [I 2022-10-11 16:31:30] ...optimizers/trijoint.py.151: total_norm: 32.220547 [I 2022-10-11 16:31:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1320/2304 [I 2022-10-11 16:31:38] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:13:58 [I 2022-10-11 16:31:38] ...trap/engines/engine.py.229: process: 0.84826 | load: 0.00315 [I 2022-10-11 16:31:38] ...trap/engines/engine.py.230: loss: 0.10206 [I 2022-10-11 16:31:38] ...optimizers/trijoint.py.151: total_norm: 21.075813 [I 2022-10-11 16:31:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1330/2304 [I 2022-10-11 16:31:47] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:55 [I 2022-10-11 16:31:47] ...trap/engines/engine.py.229: process: 0.84848 | load: 0.00311 [I 2022-10-11 16:31:47] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-11 16:31:47] ...optimizers/trijoint.py.151: total_norm: 16.426266 [I 2022-10-11 16:31:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1340/2304 [I 2022-10-11 16:31:56] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:45 [I 2022-10-11 16:31:56] ...trap/engines/engine.py.229: process: 0.87662 | load: 0.00313 [I 2022-10-11 16:31:56] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-11 16:31:56] ...optimizers/trijoint.py.151: total_norm: 27.693485 [I 2022-10-11 16:32:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1350/2304 [I 2022-10-11 16:32:04] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:34 [I 2022-10-11 16:32:04] ...trap/engines/engine.py.229: process: 0.84882 | load: 0.00310 [I 2022-10-11 16:32:04] ...trap/engines/engine.py.230: loss: 0.09554 [I 2022-10-11 16:32:04] ...optimizers/trijoint.py.151: total_norm: 8.975291 [I 2022-10-11 16:32:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1360/2304 [I 2022-10-11 16:32:13] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:13:29 [I 2022-10-11 16:32:13] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00313 [I 2022-10-11 16:32:13] ...trap/engines/engine.py.230: loss: 0.17129 [I 2022-10-11 16:32:13] ...optimizers/trijoint.py.151: total_norm: 12.375582 [I 2022-10-11 16:32:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1370/2304 [I 2022-10-11 16:32:21] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:22 [I 2022-10-11 16:32:21] ...trap/engines/engine.py.229: process: 0.87162 | load: 0.00314 [I 2022-10-11 16:32:21] ...trap/engines/engine.py.230: loss: 0.15781 [I 2022-10-11 16:32:21] ...optimizers/trijoint.py.151: total_norm: 9.641276 [I 2022-10-11 16:32:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 1380/2304 [I 2022-10-11 16:32:30] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:13:08 [I 2022-10-11 16:32:30] ...trap/engines/engine.py.229: process: 0.84890 | load: 0.00311 [I 2022-10-11 16:32:30] ...trap/engines/engine.py.230: loss: 0.08807 [I 2022-10-11 16:32:30] ...optimizers/trijoint.py.151: total_norm: 11.873337 [I 2022-10-11 16:32:39] ...trap/engines/engine.py.224: train: epoch 111 | batch 1390/2304 [I 2022-10-11 16:32:39] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:59 [I 2022-10-11 16:32:39] ...trap/engines/engine.py.229: process: 0.84709 | load: 0.00314 [I 2022-10-11 16:32:39] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-11 16:32:39] ...optimizers/trijoint.py.151: total_norm: 10.455177 [I 2022-10-11 16:32:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1400/2304 [I 2022-10-11 16:32:47] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:12:54 [I 2022-10-11 16:32:47] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00313 [I 2022-10-11 16:32:47] ...trap/engines/engine.py.230: loss: 0.17180 [I 2022-10-11 16:32:47] ...optimizers/trijoint.py.151: total_norm: 11.592181 [I 2022-10-11 16:32:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1410/2304 [I 2022-10-11 16:32:56] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:46 [I 2022-10-11 16:32:56] ...trap/engines/engine.py.229: process: 0.84997 | load: 0.00311 [I 2022-10-11 16:32:56] ...trap/engines/engine.py.230: loss: 0.14063 [I 2022-10-11 16:32:56] ...optimizers/trijoint.py.151: total_norm: 19.082970 [I 2022-10-11 16:33:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1420/2304 [I 2022-10-11 16:33:04] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:34 [I 2022-10-11 16:33:04] ...trap/engines/engine.py.229: process: 0.85296 | load: 0.00314 [I 2022-10-11 16:33:04] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-11 16:33:04] ...optimizers/trijoint.py.151: total_norm: 22.381231 [I 2022-10-11 16:33:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1430/2304 [I 2022-10-11 16:33:13] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:18 [I 2022-10-11 16:33:13] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00298 [I 2022-10-11 16:33:13] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-11 16:33:13] ...optimizers/trijoint.py.151: total_norm: 21.279552 [I 2022-10-11 16:33:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1440/2304 [I 2022-10-11 16:33:21] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:13 [I 2022-10-11 16:33:21] ...trap/engines/engine.py.229: process: 0.84984 | load: 0.00297 [I 2022-10-11 16:33:21] ...trap/engines/engine.py.230: loss: 0.15648 [I 2022-10-11 16:33:21] ...optimizers/trijoint.py.151: total_norm: 15.854898 [I 2022-10-11 16:33:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 1450/2304 [I 2022-10-11 16:33:30] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:05 [I 2022-10-11 16:33:30] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00295 [I 2022-10-11 16:33:30] ...trap/engines/engine.py.230: loss: 0.10634 [I 2022-10-11 16:33:30] ...optimizers/trijoint.py.151: total_norm: 17.009729 [I 2022-10-11 16:33:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1460/2304 [I 2022-10-11 16:33:38] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:54 [I 2022-10-11 16:33:38] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00296 [I 2022-10-11 16:33:38] ...trap/engines/engine.py.230: loss: 0.09829 [I 2022-10-11 16:33:38] ...optimizers/trijoint.py.151: total_norm: 14.557939 [I 2022-10-11 16:33:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1470/2304 [I 2022-10-11 16:33:47] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:11:40 [I 2022-10-11 16:33:47] ...trap/engines/engine.py.229: process: 0.85069 | load: 0.00293 [I 2022-10-11 16:33:47] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-11 16:33:47] ...optimizers/trijoint.py.151: total_norm: 10.322459 [I 2022-10-11 16:33:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1480/2304 [I 2022-10-11 16:33:55] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:43 [I 2022-10-11 16:33:55] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00296 [I 2022-10-11 16:33:55] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-11 16:33:55] ...optimizers/trijoint.py.151: total_norm: 11.356495 [I 2022-10-11 16:34:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1490/2304 [I 2022-10-11 16:34:04] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:27 [I 2022-10-11 16:34:04] ...trap/engines/engine.py.229: process: 0.84588 | load: 0.00295 [I 2022-10-11 16:34:04] ...trap/engines/engine.py.230: loss: 0.16035 [I 2022-10-11 16:34:04] ...optimizers/trijoint.py.151: total_norm: 18.313896 [I 2022-10-11 16:34:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1500/2304 [I 2022-10-11 16:34:12] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:27 [I 2022-10-11 16:34:12] ...trap/engines/engine.py.229: process: 0.85846 | load: 0.00293 [I 2022-10-11 16:34:12] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-11 16:34:12] ...optimizers/trijoint.py.151: total_norm: 32.685417 [I 2022-10-11 16:34:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1510/2304 [I 2022-10-11 16:34:21] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:34 [I 2022-10-11 16:34:21] ...trap/engines/engine.py.229: process: 0.89177 | load: 0.00377 [I 2022-10-11 16:34:21] ...trap/engines/engine.py.230: loss: 0.16310 [I 2022-10-11 16:34:21] ...optimizers/trijoint.py.151: total_norm: 12.093124 [I 2022-10-11 16:34:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 1520/2304 [I 2022-10-11 16:34:30] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:10 [I 2022-10-11 16:34:30] ...trap/engines/engine.py.229: process: 0.85780 | load: 0.00298 [I 2022-10-11 16:34:30] ...trap/engines/engine.py.230: loss: 0.08772 [I 2022-10-11 16:34:30] ...optimizers/trijoint.py.151: total_norm: 9.302954 [I 2022-10-11 16:34:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1530/2304 [I 2022-10-11 16:34:38] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:54 [I 2022-10-11 16:34:38] ...trap/engines/engine.py.229: process: 0.83628 | load: 0.00299 [I 2022-10-11 16:34:38] ...trap/engines/engine.py.230: loss: 0.08793 [I 2022-10-11 16:34:38] ...optimizers/trijoint.py.151: total_norm: 11.196948 [I 2022-10-11 16:34:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1540/2304 [I 2022-10-11 16:34:47] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:46 [I 2022-10-11 16:34:47] ...trap/engines/engine.py.229: process: 0.83248 | load: 0.00301 [I 2022-10-11 16:34:47] ...trap/engines/engine.py.230: loss: 0.12646 [I 2022-10-11 16:34:47] ...optimizers/trijoint.py.151: total_norm: 12.421966 [I 2022-10-11 16:34:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1550/2304 [I 2022-10-11 16:34:55] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:36 [I 2022-10-11 16:34:55] ...trap/engines/engine.py.229: process: 0.85299 | load: 0.00301 [I 2022-10-11 16:34:55] ...trap/engines/engine.py.230: loss: 0.09122 [I 2022-10-11 16:34:55] ...optimizers/trijoint.py.151: total_norm: 17.484333 [I 2022-10-11 16:35:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1560/2304 [I 2022-10-11 16:35:04] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:30 [I 2022-10-11 16:35:04] ...trap/engines/engine.py.229: process: 0.83819 | load: 0.00297 [I 2022-10-11 16:35:04] ...trap/engines/engine.py.230: loss: 0.07727 [I 2022-10-11 16:35:04] ...optimizers/trijoint.py.151: total_norm: 12.306734 [I 2022-10-11 16:35:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1570/2304 [I 2022-10-11 16:35:12] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:17 [I 2022-10-11 16:35:12] ...trap/engines/engine.py.229: process: 0.84782 | load: 0.00298 [I 2022-10-11 16:35:12] ...trap/engines/engine.py.230: loss: 0.09800 [I 2022-10-11 16:35:12] ...optimizers/trijoint.py.151: total_norm: 12.368997 [I 2022-10-11 16:35:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1580/2304 [I 2022-10-11 16:35:20] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:08 [I 2022-10-11 16:35:20] ...trap/engines/engine.py.229: process: 0.82371 | load: 0.00300 [I 2022-10-11 16:35:20] ...trap/engines/engine.py.230: loss: 0.12545 [I 2022-10-11 16:35:20] ...optimizers/trijoint.py.151: total_norm: 14.560431 [I 2022-10-11 16:35:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1590/2304 [I 2022-10-11 16:35:29] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:02 [I 2022-10-11 16:35:29] ...trap/engines/engine.py.229: process: 0.84148 | load: 0.00302 [I 2022-10-11 16:35:29] ...trap/engines/engine.py.230: loss: 0.13103 [I 2022-10-11 16:35:29] ...optimizers/trijoint.py.151: total_norm: 20.862684 [I 2022-10-11 16:35:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1600/2304 [I 2022-10-11 16:35:37] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:56 [I 2022-10-11 16:35:37] ...trap/engines/engine.py.229: process: 0.83584 | load: 0.00299 [I 2022-10-11 16:35:37] ...trap/engines/engine.py.230: loss: 0.09741 [I 2022-10-11 16:35:37] ...optimizers/trijoint.py.151: total_norm: 15.036116 [I 2022-10-11 16:35:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1610/2304 [I 2022-10-11 16:35:46] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:42 [I 2022-10-11 16:35:46] ...trap/engines/engine.py.229: process: 0.84642 | load: 0.00297 [I 2022-10-11 16:35:46] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-11 16:35:46] ...optimizers/trijoint.py.151: total_norm: 35.430737 [I 2022-10-11 16:35:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1620/2304 [I 2022-10-11 16:35:54] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:33 [I 2022-10-11 16:35:54] ...trap/engines/engine.py.229: process: 0.83771 | load: 0.00296 [I 2022-10-11 16:35:54] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-11 16:35:54] ...optimizers/trijoint.py.151: total_norm: 17.309816 [I 2022-10-11 16:36:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1630/2304 [I 2022-10-11 16:36:03] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:27 [I 2022-10-11 16:36:03] ...trap/engines/engine.py.229: process: 0.84908 | load: 0.00298 [I 2022-10-11 16:36:03] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-11 16:36:03] ...optimizers/trijoint.py.151: total_norm: 14.479618 [I 2022-10-11 16:36:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1640/2304 [I 2022-10-11 16:36:11] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:25 [I 2022-10-11 16:36:11] ...trap/engines/engine.py.229: process: 0.85207 | load: 0.00316 [I 2022-10-11 16:36:11] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-11 16:36:11] ...optimizers/trijoint.py.151: total_norm: 17.210363 [I 2022-10-11 16:36:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1650/2304 [I 2022-10-11 16:36:20] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:17 [I 2022-10-11 16:36:20] ...trap/engines/engine.py.229: process: 0.84533 | load: 0.00312 [I 2022-10-11 16:36:20] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-11 16:36:20] ...optimizers/trijoint.py.151: total_norm: 18.706015 [I 2022-10-11 16:36:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 1660/2304 [I 2022-10-11 16:36:28] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:10 [I 2022-10-11 16:36:28] ...trap/engines/engine.py.229: process: 0.84378 | load: 0.00311 [I 2022-10-11 16:36:28] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-11 16:36:28] ...optimizers/trijoint.py.151: total_norm: 11.329402 [I 2022-10-11 16:36:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1670/2304 [I 2022-10-11 16:36:37] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:07 [I 2022-10-11 16:36:37] ...trap/engines/engine.py.229: process: 0.87177 | load: 0.00316 [I 2022-10-11 16:36:37] ...trap/engines/engine.py.230: loss: 0.14734 [I 2022-10-11 16:36:37] ...optimizers/trijoint.py.151: total_norm: 24.068913 [I 2022-10-11 16:36:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 1680/2304 [I 2022-10-11 16:36:45] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:55 [I 2022-10-11 16:36:45] ...trap/engines/engine.py.229: process: 0.87247 | load: 0.00315 [I 2022-10-11 16:36:45] ...trap/engines/engine.py.230: loss: 0.10630 [I 2022-10-11 16:36:45] ...optimizers/trijoint.py.151: total_norm: 29.945570 [I 2022-10-11 16:36:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1690/2304 [I 2022-10-11 16:36:54] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:58 [I 2022-10-11 16:36:54] ...trap/engines/engine.py.229: process: 0.89380 | load: 0.00392 [I 2022-10-11 16:36:54] ...trap/engines/engine.py.230: loss: 0.13240 [I 2022-10-11 16:36:54] ...optimizers/trijoint.py.151: total_norm: 24.919170 [I 2022-10-11 16:37:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1700/2304 [I 2022-10-11 16:37:03] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:50 [I 2022-10-11 16:37:03] ...trap/engines/engine.py.229: process: 0.86404 | load: 0.00315 [I 2022-10-11 16:37:03] ...trap/engines/engine.py.230: loss: 0.13080 [I 2022-10-11 16:37:03] ...optimizers/trijoint.py.151: total_norm: 16.029497 [I 2022-10-11 16:37:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1710/2304 [I 2022-10-11 16:37:12] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:26 [I 2022-10-11 16:37:12] ...trap/engines/engine.py.229: process: 0.84286 | load: 0.00319 [I 2022-10-11 16:37:12] ...trap/engines/engine.py.230: loss: 0.10748 [I 2022-10-11 16:37:12] ...optimizers/trijoint.py.151: total_norm: 14.775274 [I 2022-10-11 16:37:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1720/2304 [I 2022-10-11 16:37:20] ...trap/engines/engine.py.228: elapsed: 0:24:38 | left: 0:08:20 [I 2022-10-11 16:37:20] ...trap/engines/engine.py.229: process: 0.86751 | load: 0.00312 [I 2022-10-11 16:37:20] ...trap/engines/engine.py.230: loss: 0.08869 [I 2022-10-11 16:37:20] ...optimizers/trijoint.py.151: total_norm: 12.883888 [I 2022-10-11 16:37:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1730/2304 [I 2022-10-11 16:37:29] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:27 [I 2022-10-11 16:37:29] ...trap/engines/engine.py.229: process: 0.89240 | load: 0.00398 [I 2022-10-11 16:37:29] ...trap/engines/engine.py.230: loss: 0.18343 [I 2022-10-11 16:37:29] ...optimizers/trijoint.py.151: total_norm: 9.979051 [I 2022-10-11 16:37:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1740/2304 [I 2022-10-11 16:37:38] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:05 [I 2022-10-11 16:37:38] ...trap/engines/engine.py.229: process: 0.87048 | load: 0.00311 [I 2022-10-11 16:37:38] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-11 16:37:38] ...optimizers/trijoint.py.151: total_norm: 8.725537 [I 2022-10-11 16:37:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1750/2304 [I 2022-10-11 16:37:46] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:07:58 [I 2022-10-11 16:37:46] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00312 [I 2022-10-11 16:37:46] ...trap/engines/engine.py.230: loss: 0.12609 [I 2022-10-11 16:37:46] ...optimizers/trijoint.py.151: total_norm: 12.087028 [I 2022-10-11 16:37:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1760/2304 [I 2022-10-11 16:37:55] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:49 [I 2022-10-11 16:37:55] ...trap/engines/engine.py.229: process: 0.86782 | load: 0.00311 [I 2022-10-11 16:37:55] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-11 16:37:55] ...optimizers/trijoint.py.151: total_norm: 41.006527 [I 2022-10-11 16:38:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1770/2304 [I 2022-10-11 16:38:04] ...trap/engines/engine.py.228: elapsed: 0:25:21 | left: 0:07:34 [I 2022-10-11 16:38:04] ...trap/engines/engine.py.229: process: 0.83389 | load: 0.00312 [I 2022-10-11 16:38:04] ...trap/engines/engine.py.230: loss: 0.20531 [I 2022-10-11 16:38:04] ...optimizers/trijoint.py.151: total_norm: 10.477204 [I 2022-10-11 16:38:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1780/2304 [I 2022-10-11 16:38:12] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:28 [I 2022-10-11 16:38:12] ...trap/engines/engine.py.229: process: 0.86071 | load: 0.00312 [I 2022-10-11 16:38:12] ...trap/engines/engine.py.230: loss: 0.13787 [I 2022-10-11 16:38:12] ...optimizers/trijoint.py.151: total_norm: 23.217396 [I 2022-10-11 16:38:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1790/2304 [I 2022-10-11 16:38:21] ...trap/engines/engine.py.228: elapsed: 0:25:38 | left: 0:07:17 [I 2022-10-11 16:38:21] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00313 [I 2022-10-11 16:38:21] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-11 16:38:21] ...optimizers/trijoint.py.151: total_norm: 51.414848 [I 2022-10-11 16:38:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1800/2304 [I 2022-10-11 16:38:29] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:12 [I 2022-10-11 16:38:29] ...trap/engines/engine.py.229: process: 0.85596 | load: 0.00311 [I 2022-10-11 16:38:29] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-11 16:38:29] ...optimizers/trijoint.py.151: total_norm: 15.838247 [I 2022-10-11 16:38:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1810/2304 [I 2022-10-11 16:38:38] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:03 [I 2022-10-11 16:38:38] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00311 [I 2022-10-11 16:38:38] ...trap/engines/engine.py.230: loss: 0.09075 [I 2022-10-11 16:38:38] ...optimizers/trijoint.py.151: total_norm: 10.422879 [I 2022-10-11 16:38:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1820/2304 [I 2022-10-11 16:38:46] ...trap/engines/engine.py.228: elapsed: 0:26:04 | left: 0:06:56 [I 2022-10-11 16:38:46] ...trap/engines/engine.py.229: process: 0.86976 | load: 0.00312 [I 2022-10-11 16:38:46] ...trap/engines/engine.py.230: loss: 0.08575 [I 2022-10-11 16:38:46] ...optimizers/trijoint.py.151: total_norm: 8.699184 [I 2022-10-11 16:38:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1830/2304 [I 2022-10-11 16:38:55] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:45 [I 2022-10-11 16:38:55] ...trap/engines/engine.py.229: process: 0.85697 | load: 0.00311 [I 2022-10-11 16:38:55] ...trap/engines/engine.py.230: loss: 0.13296 [I 2022-10-11 16:38:55] ...optimizers/trijoint.py.151: total_norm: 22.768909 [I 2022-10-11 16:39:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1840/2304 [I 2022-10-11 16:39:03] ...trap/engines/engine.py.228: elapsed: 0:26:21 | left: 0:06:36 [I 2022-10-11 16:39:03] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00311 [I 2022-10-11 16:39:03] ...trap/engines/engine.py.230: loss: 0.10574 [I 2022-10-11 16:39:03] ...optimizers/trijoint.py.151: total_norm: 16.073952 [I 2022-10-11 16:39:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1850/2304 [I 2022-10-11 16:39:12] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:26 [I 2022-10-11 16:39:12] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00311 [I 2022-10-11 16:39:12] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-11 16:39:12] ...optimizers/trijoint.py.151: total_norm: 11.247932 [I 2022-10-11 16:39:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1860/2304 [I 2022-10-11 16:39:21] ...trap/engines/engine.py.228: elapsed: 0:26:38 | left: 0:06:18 [I 2022-10-11 16:39:21] ...trap/engines/engine.py.229: process: 0.85990 | load: 0.00312 [I 2022-10-11 16:39:21] ...trap/engines/engine.py.230: loss: 0.08064 [I 2022-10-11 16:39:21] ...optimizers/trijoint.py.151: total_norm: 23.566730 [I 2022-10-11 16:39:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1870/2304 [I 2022-10-11 16:39:29] ...trap/engines/engine.py.228: elapsed: 0:26:47 | left: 0:06:11 [I 2022-10-11 16:39:29] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00313 [I 2022-10-11 16:39:29] ...trap/engines/engine.py.230: loss: 0.07626 [I 2022-10-11 16:39:29] ...optimizers/trijoint.py.151: total_norm: 15.679730 [I 2022-10-11 16:39:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1880/2304 [I 2022-10-11 16:39:38] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:06 [I 2022-10-11 16:39:38] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00312 [I 2022-10-11 16:39:38] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-11 16:39:38] ...optimizers/trijoint.py.151: total_norm: 19.245535 [I 2022-10-11 16:39:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1890/2304 [I 2022-10-11 16:39:46] ...trap/engines/engine.py.228: elapsed: 0:27:04 | left: 0:05:54 [I 2022-10-11 16:39:46] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00314 [I 2022-10-11 16:39:46] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-11 16:39:46] ...optimizers/trijoint.py.151: total_norm: 18.829926 [I 2022-10-11 16:39:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1900/2304 [I 2022-10-11 16:39:55] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:46 [I 2022-10-11 16:39:55] ...trap/engines/engine.py.229: process: 0.84488 | load: 0.00310 [I 2022-10-11 16:39:55] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-11 16:39:55] ...optimizers/trijoint.py.151: total_norm: 17.728859 [I 2022-10-11 16:40:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 1910/2304 [I 2022-10-11 16:40:04] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:41 [I 2022-10-11 16:40:04] ...trap/engines/engine.py.229: process: 0.86911 | load: 0.00310 [I 2022-10-11 16:40:04] ...trap/engines/engine.py.230: loss: 0.08712 [I 2022-10-11 16:40:04] ...optimizers/trijoint.py.151: total_norm: 13.630195 [I 2022-10-11 16:40:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 1920/2304 [I 2022-10-11 16:40:12] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:26 [I 2022-10-11 16:40:12] ...trap/engines/engine.py.229: process: 0.86907 | load: 0.00296 [I 2022-10-11 16:40:12] ...trap/engines/engine.py.230: loss: 0.16531 [I 2022-10-11 16:40:12] ...optimizers/trijoint.py.151: total_norm: 45.201790 [I 2022-10-11 16:40:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1930/2304 [I 2022-10-11 16:40:21] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:17 [I 2022-10-11 16:40:21] ...trap/engines/engine.py.229: process: 0.85459 | load: 0.00295 [I 2022-10-11 16:40:21] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-11 16:40:21] ...optimizers/trijoint.py.151: total_norm: 10.985753 [I 2022-10-11 16:40:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1940/2304 [I 2022-10-11 16:40:29] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:06 [I 2022-10-11 16:40:29] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00293 [I 2022-10-11 16:40:29] ...trap/engines/engine.py.230: loss: 0.10177 [I 2022-10-11 16:40:29] ...optimizers/trijoint.py.151: total_norm: 8.229358 [I 2022-10-11 16:40:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1950/2304 [I 2022-10-11 16:40:38] ...trap/engines/engine.py.228: elapsed: 0:27:55 | left: 0:04:58 [I 2022-10-11 16:40:38] ...trap/engines/engine.py.229: process: 0.86136 | load: 0.00298 [I 2022-10-11 16:40:38] ...trap/engines/engine.py.230: loss: 0.10767 [I 2022-10-11 16:40:38] ...optimizers/trijoint.py.151: total_norm: 16.253674 [I 2022-10-11 16:40:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1960/2304 [I 2022-10-11 16:40:46] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:48 [I 2022-10-11 16:40:46] ...trap/engines/engine.py.229: process: 0.84166 | load: 0.00293 [I 2022-10-11 16:40:46] ...trap/engines/engine.py.230: loss: 0.12578 [I 2022-10-11 16:40:46] ...optimizers/trijoint.py.151: total_norm: 11.255553 [I 2022-10-11 16:40:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1970/2304 [I 2022-10-11 16:40:54] ...trap/engines/engine.py.228: elapsed: 0:28:12 | left: 0:04:41 [I 2022-10-11 16:40:54] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00295 [I 2022-10-11 16:40:54] ...trap/engines/engine.py.230: loss: 0.11049 [I 2022-10-11 16:40:54] ...optimizers/trijoint.py.151: total_norm: 11.250597 [I 2022-10-11 16:41:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1980/2304 [I 2022-10-11 16:41:03] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:32 [I 2022-10-11 16:41:03] ...trap/engines/engine.py.229: process: 0.83576 | load: 0.00296 [I 2022-10-11 16:41:03] ...trap/engines/engine.py.230: loss: 0.12676 [I 2022-10-11 16:41:03] ...optimizers/trijoint.py.151: total_norm: 13.017694 [I 2022-10-11 16:41:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1990/2304 [I 2022-10-11 16:41:11] ...trap/engines/engine.py.228: elapsed: 0:28:29 | left: 0:04:23 [I 2022-10-11 16:41:11] ...trap/engines/engine.py.229: process: 0.82909 | load: 0.00295 [I 2022-10-11 16:41:11] ...trap/engines/engine.py.230: loss: 0.08879 [I 2022-10-11 16:41:11] ...optimizers/trijoint.py.151: total_norm: 20.203278 [I 2022-10-11 16:41:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 2000/2304 [I 2022-10-11 16:41:20] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:16 [I 2022-10-11 16:41:20] ...trap/engines/engine.py.229: process: 0.83858 | load: 0.00294 [I 2022-10-11 16:41:20] ...trap/engines/engine.py.230: loss: 0.08607 [I 2022-10-11 16:41:20] ...optimizers/trijoint.py.151: total_norm: 10.467028 [I 2022-10-11 16:41:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 2010/2304 [I 2022-10-11 16:41:28] ...trap/engines/engine.py.228: elapsed: 0:28:46 | left: 0:04:08 [I 2022-10-11 16:41:28] ...trap/engines/engine.py.229: process: 0.84214 | load: 0.00294 [I 2022-10-11 16:41:28] ...trap/engines/engine.py.230: loss: 0.18937 [I 2022-10-11 16:41:28] ...optimizers/trijoint.py.151: total_norm: 12.810933 [I 2022-10-11 16:41:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 2020/2304 [I 2022-10-11 16:41:37] ...trap/engines/engine.py.228: elapsed: 0:28:54 | left: 0:03:59 [I 2022-10-11 16:41:37] ...trap/engines/engine.py.229: process: 0.85854 | load: 0.00297 [I 2022-10-11 16:41:37] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-11 16:41:37] ...optimizers/trijoint.py.151: total_norm: 25.050533 [I 2022-10-11 16:41:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 2030/2304 [I 2022-10-11 16:41:45] ...trap/engines/engine.py.228: elapsed: 0:29:03 | left: 0:03:50 [I 2022-10-11 16:41:45] ...trap/engines/engine.py.229: process: 0.82905 | load: 0.00294 [I 2022-10-11 16:41:45] ...trap/engines/engine.py.230: loss: 0.09207 [I 2022-10-11 16:41:45] ...optimizers/trijoint.py.151: total_norm: 10.119729 [I 2022-10-11 16:41:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 2040/2304 [I 2022-10-11 16:41:54] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:03:41 [I 2022-10-11 16:41:54] ...trap/engines/engine.py.229: process: 0.83524 | load: 0.00301 [I 2022-10-11 16:41:54] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-11 16:41:54] ...optimizers/trijoint.py.151: total_norm: 30.472166 [I 2022-10-11 16:42:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 2050/2304 [I 2022-10-11 16:42:02] ...trap/engines/engine.py.228: elapsed: 0:29:20 | left: 0:03:33 [I 2022-10-11 16:42:02] ...trap/engines/engine.py.229: process: 0.82894 | load: 0.00301 [I 2022-10-11 16:42:02] ...trap/engines/engine.py.230: loss: 0.13750 [I 2022-10-11 16:42:02] ...optimizers/trijoint.py.151: total_norm: 18.755079 [I 2022-10-11 16:42:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 2060/2304 [I 2022-10-11 16:42:10] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:24 [I 2022-10-11 16:42:10] ...trap/engines/engine.py.229: process: 0.82819 | load: 0.00296 [I 2022-10-11 16:42:10] ...trap/engines/engine.py.230: loss: 0.08229 [I 2022-10-11 16:42:10] ...optimizers/trijoint.py.151: total_norm: 11.061147 [I 2022-10-11 16:42:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 2070/2304 [I 2022-10-11 16:42:19] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:17 [I 2022-10-11 16:42:19] ...trap/engines/engine.py.229: process: 0.83774 | load: 0.00297 [I 2022-10-11 16:42:19] ...trap/engines/engine.py.230: loss: 0.15690 [I 2022-10-11 16:42:19] ...optimizers/trijoint.py.151: total_norm: 20.909079 [I 2022-10-11 16:42:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 2080/2304 [I 2022-10-11 16:42:27] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:08 [I 2022-10-11 16:42:27] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00297 [I 2022-10-11 16:42:27] ...trap/engines/engine.py.230: loss: 0.12250 [I 2022-10-11 16:42:27] ...optimizers/trijoint.py.151: total_norm: 11.216644 [I 2022-10-11 16:42:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 2090/2304 [I 2022-10-11 16:42:36] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:01 [I 2022-10-11 16:42:36] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00297 [I 2022-10-11 16:42:36] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-11 16:42:36] ...optimizers/trijoint.py.151: total_norm: 17.485718 [I 2022-10-11 16:42:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 2100/2304 [I 2022-10-11 16:42:44] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:02:51 [I 2022-10-11 16:42:44] ...trap/engines/engine.py.229: process: 0.85531 | load: 0.00298 [I 2022-10-11 16:42:44] ...trap/engines/engine.py.230: loss: 0.18318 [I 2022-10-11 16:42:44] ...optimizers/trijoint.py.151: total_norm: 13.384572 [I 2022-10-11 16:42:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 2110/2304 [I 2022-10-11 16:42:53] ...trap/engines/engine.py.228: elapsed: 0:30:11 | left: 0:02:45 [I 2022-10-11 16:42:53] ...trap/engines/engine.py.229: process: 0.86977 | load: 0.00301 [I 2022-10-11 16:42:53] ...trap/engines/engine.py.230: loss: 0.12769 [I 2022-10-11 16:42:53] ...optimizers/trijoint.py.151: total_norm: 22.103333 [I 2022-10-11 16:43:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 2120/2304 [I 2022-10-11 16:43:01] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:02:36 [I 2022-10-11 16:43:01] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00295 [I 2022-10-11 16:43:01] ...trap/engines/engine.py.230: loss: 0.10046 [I 2022-10-11 16:43:01] ...optimizers/trijoint.py.151: total_norm: 14.385876 [I 2022-10-11 16:43:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 2130/2304 [I 2022-10-11 16:43:10] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:27 [I 2022-10-11 16:43:10] ...trap/engines/engine.py.229: process: 0.85496 | load: 0.00297 [I 2022-10-11 16:43:10] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-11 16:43:10] ...optimizers/trijoint.py.151: total_norm: 8.514036 [I 2022-10-11 16:43:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 2140/2304 [I 2022-10-11 16:43:18] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:18 [I 2022-10-11 16:43:18] ...trap/engines/engine.py.229: process: 0.85378 | load: 0.00306 [I 2022-10-11 16:43:18] ...trap/engines/engine.py.230: loss: 0.09123 [I 2022-10-11 16:43:18] ...optimizers/trijoint.py.151: total_norm: 18.907616 [I 2022-10-11 16:43:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 2150/2304 [I 2022-10-11 16:43:27] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:10 [I 2022-10-11 16:43:27] ...trap/engines/engine.py.229: process: 0.83948 | load: 0.00305 [I 2022-10-11 16:43:27] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-11 16:43:27] ...optimizers/trijoint.py.151: total_norm: 11.149497 [I 2022-10-11 16:43:35] ...trap/engines/engine.py.224: train: epoch 111 | batch 2160/2304 [I 2022-10-11 16:43:35] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:02 [I 2022-10-11 16:43:35] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00310 [I 2022-10-11 16:43:35] ...trap/engines/engine.py.230: loss: 0.11663 [I 2022-10-11 16:43:35] ...optimizers/trijoint.py.151: total_norm: 16.868366 [I 2022-10-11 16:43:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 2170/2304 [I 2022-10-11 16:43:44] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:01:55 [I 2022-10-11 16:43:44] ...trap/engines/engine.py.229: process: 0.86132 | load: 0.00310 [I 2022-10-11 16:43:44] ...trap/engines/engine.py.230: loss: 0.08003 [I 2022-10-11 16:43:44] ...optimizers/trijoint.py.151: total_norm: 10.517788 [I 2022-10-11 16:43:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 2180/2304 [I 2022-10-11 16:43:52] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:01:46 [I 2022-10-11 16:43:52] ...trap/engines/engine.py.229: process: 0.86984 | load: 0.00314 [I 2022-10-11 16:43:52] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-11 16:43:52] ...optimizers/trijoint.py.151: total_norm: 17.347759 [I 2022-10-11 16:44:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 2190/2304 [I 2022-10-11 16:44:01] ...trap/engines/engine.py.228: elapsed: 0:31:19 | left: 0:01:37 [I 2022-10-11 16:44:01] ...trap/engines/engine.py.229: process: 0.84804 | load: 0.00311 [I 2022-10-11 16:44:01] ...trap/engines/engine.py.230: loss: 0.08859 [I 2022-10-11 16:44:01] ...optimizers/trijoint.py.151: total_norm: 14.845406 [I 2022-10-11 16:44:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 2200/2304 [I 2022-10-11 16:44:10] ...trap/engines/engine.py.228: elapsed: 0:31:28 | left: 0:01:29 [I 2022-10-11 16:44:10] ...trap/engines/engine.py.229: process: 0.86236 | load: 0.00315 [I 2022-10-11 16:44:10] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-11 16:44:10] ...optimizers/trijoint.py.151: total_norm: 9.485893 [I 2022-10-11 16:44:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 2210/2304 [I 2022-10-11 16:44:18] ...trap/engines/engine.py.228: elapsed: 0:31:36 | left: 0:01:21 [I 2022-10-11 16:44:18] ...trap/engines/engine.py.229: process: 0.86893 | load: 0.00313 [I 2022-10-11 16:44:18] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-11 16:44:18] ...optimizers/trijoint.py.151: total_norm: 10.806271 [I 2022-10-11 16:44:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 2220/2304 [I 2022-10-11 16:44:27] ...trap/engines/engine.py.228: elapsed: 0:31:45 | left: 0:01:13 [I 2022-10-11 16:44:27] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00313 [I 2022-10-11 16:44:27] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-11 16:44:27] ...optimizers/trijoint.py.151: total_norm: 14.954319 [I 2022-10-11 16:44:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 2230/2304 [I 2022-10-11 16:44:36] ...trap/engines/engine.py.228: elapsed: 0:31:53 | left: 0:01:03 [I 2022-10-11 16:44:36] ...trap/engines/engine.py.229: process: 0.85965 | load: 0.00313 [I 2022-10-11 16:44:36] ...trap/engines/engine.py.230: loss: 0.21158 [I 2022-10-11 16:44:36] ...optimizers/trijoint.py.151: total_norm: 27.730427 [I 2022-10-11 16:44:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 2240/2304 [I 2022-10-11 16:44:44] ...trap/engines/engine.py.228: elapsed: 0:32:02 | left: 0:00:54 [I 2022-10-11 16:44:44] ...trap/engines/engine.py.229: process: 0.84691 | load: 0.00315 [I 2022-10-11 16:44:44] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-11 16:44:44] ...optimizers/trijoint.py.151: total_norm: 18.286545 [I 2022-10-11 16:44:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 2250/2304 [I 2022-10-11 16:44:53] ...trap/engines/engine.py.228: elapsed: 0:32:11 | left: 0:00:46 [I 2022-10-11 16:44:53] ...trap/engines/engine.py.229: process: 0.85983 | load: 0.00311 [I 2022-10-11 16:44:53] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-11 16:44:53] ...optimizers/trijoint.py.151: total_norm: 16.136349 [I 2022-10-11 16:45:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 2260/2304 [I 2022-10-11 16:45:01] ...trap/engines/engine.py.228: elapsed: 0:32:19 | left: 0:00:37 [I 2022-10-11 16:45:01] ...trap/engines/engine.py.229: process: 0.86319 | load: 0.00311 [I 2022-10-11 16:45:01] ...trap/engines/engine.py.230: loss: 0.11557 [I 2022-10-11 16:45:01] ...optimizers/trijoint.py.151: total_norm: 12.014224 [I 2022-10-11 16:45:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 2270/2304 [I 2022-10-11 16:45:10] ...trap/engines/engine.py.228: elapsed: 0:32:28 | left: 0:00:28 [I 2022-10-11 16:45:10] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00312 [I 2022-10-11 16:45:10] ...trap/engines/engine.py.230: loss: 0.09873 [I 2022-10-11 16:45:10] ...optimizers/trijoint.py.151: total_norm: 45.182018 [I 2022-10-11 16:45:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 2280/2304 [I 2022-10-11 16:45:19] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:20 [I 2022-10-11 16:45:19] ...trap/engines/engine.py.229: process: 0.87202 | load: 0.00314 [I 2022-10-11 16:45:19] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-11 16:45:19] ...optimizers/trijoint.py.151: total_norm: 30.356243 [I 2022-10-11 16:45:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 2290/2304 [I 2022-10-11 16:45:27] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:12 [I 2022-10-11 16:45:27] ...trap/engines/engine.py.229: process: 0.87249 | load: 0.00313 [I 2022-10-11 16:45:27] ...trap/engines/engine.py.230: loss: 0.14070 [I 2022-10-11 16:45:27] ...optimizers/trijoint.py.151: total_norm: 14.810745 [I 2022-10-11 16:45:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 2300/2304 [I 2022-10-11 16:45:36] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:03 [I 2022-10-11 16:45:36] ...trap/engines/engine.py.229: process: 0.83544 | load: 0.00020 [I 2022-10-11 16:45:36] ...trap/engines/engine.py.230: loss: 0.07702 [I 2022-10-11 16:45:36] ...optimizers/trijoint.py.151: total_norm: 29.177860 [I 2022-10-11 16:45:39] ...trap/engines/engine.py.224: train: epoch 111 | batch 2304/2304 [I 2022-10-11 16:45:39] ...trap/engines/engine.py.228: elapsed: 0:32:57 | left: 0:00:00 [I 2022-10-11 16:45:39] ...trap/engines/engine.py.229: process: 0.82995 | load: 0.00020 [I 2022-10-11 16:45:39] ...trap/engines/engine.py.230: loss: 0.14177 [I 2022-10-11 16:45:39] ...optimizers/trijoint.py.151: total_norm: 19.927568 [S 2022-10-11 16:45:41] ...trap/engines/engine.py.129: train_epoch.epoch: 111 [S 2022-10-11 16:45:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.12004696174227751 [I 2022-10-11 16:45:41] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 16:45:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 111 [I 2022-10-11 16:45:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 16:45:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 16:45:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 16:45:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 16:45:48] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 16:45:49] ...trap/engines/engine.py.321: eval: epoch 111 | batch 0/510 [I 2022-10-11 16:45:49] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:56:38 [I 2022-10-11 16:45:49] ...trap/engines/engine.py.326: process: 6.66467 | load: 4.30452 [I 2022-10-11 16:45:55] ...trap/engines/engine.py.321: eval: epoch 111 | batch 10/510 [I 2022-10-11 16:45:55] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:10:40 [I 2022-10-11 16:45:55] ...trap/engines/engine.py.326: process: 0.70221 | load: 0.00021 [I 2022-10-11 16:46:02] ...trap/engines/engine.py.321: eval: epoch 111 | batch 20/510 [I 2022-10-11 16:46:02] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:48 [I 2022-10-11 16:46:02] ...trap/engines/engine.py.326: process: 0.82247 | load: 0.00020 [I 2022-10-11 16:46:08] ...trap/engines/engine.py.321: eval: epoch 111 | batch 30/510 [I 2022-10-11 16:46:08] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:04:59 [I 2022-10-11 16:46:08] ...trap/engines/engine.py.326: process: 0.48553 | load: 0.00021 [I 2022-10-11 16:46:13] ...trap/engines/engine.py.321: eval: epoch 111 | batch 40/510 [I 2022-10-11 16:46:13] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:04:13 [I 2022-10-11 16:46:13] ...trap/engines/engine.py.326: process: 0.53883 | load: 0.00020 [I 2022-10-11 16:46:19] ...trap/engines/engine.py.321: eval: epoch 111 | batch 50/510 [I 2022-10-11 16:46:19] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:04:06 [I 2022-10-11 16:46:19] ...trap/engines/engine.py.326: process: 0.46807 | load: 0.00021 [I 2022-10-11 16:46:24] ...trap/engines/engine.py.321: eval: epoch 111 | batch 60/510 [I 2022-10-11 16:46:24] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:04:01 [I 2022-10-11 16:46:24] ...trap/engines/engine.py.326: process: 0.54998 | load: 0.00025 [I 2022-10-11 16:46:30] ...trap/engines/engine.py.321: eval: epoch 111 | batch 70/510 [I 2022-10-11 16:46:30] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:04:02 [I 2022-10-11 16:46:30] ...trap/engines/engine.py.326: process: 0.58178 | load: 0.00021 [I 2022-10-11 16:46:35] ...trap/engines/engine.py.321: eval: epoch 111 | batch 80/510 [I 2022-10-11 16:46:35] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:03 [I 2022-10-11 16:46:35] ...trap/engines/engine.py.326: process: 0.54306 | load: 0.00020 [I 2022-10-11 16:46:40] ...trap/engines/engine.py.321: eval: epoch 111 | batch 90/510 [I 2022-10-11 16:46:40] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:03:16 [I 2022-10-11 16:46:40] ...trap/engines/engine.py.326: process: 0.43543 | load: 0.00021 [I 2022-10-11 16:46:45] ...trap/engines/engine.py.321: eval: epoch 111 | batch 100/510 [I 2022-10-11 16:46:45] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:59 [I 2022-10-11 16:46:45] ...trap/engines/engine.py.326: process: 0.31357 | load: 0.00021 [I 2022-10-11 16:46:48] ...trap/engines/engine.py.321: eval: epoch 111 | batch 110/510 [I 2022-10-11 16:46:48] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:13 [I 2022-10-11 16:46:48] ...trap/engines/engine.py.326: process: 0.32176 | load: 0.00021 [I 2022-10-11 16:46:51] ...trap/engines/engine.py.321: eval: epoch 111 | batch 120/510 [I 2022-10-11 16:46:51] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:05 [I 2022-10-11 16:46:51] ...trap/engines/engine.py.326: process: 0.32273 | load: 0.00021 [I 2022-10-11 16:46:54] ...trap/engines/engine.py.321: eval: epoch 111 | batch 130/510 [I 2022-10-11 16:46:54] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:02:01 [I 2022-10-11 16:46:54] ...trap/engines/engine.py.326: process: 0.31580 | load: 0.00023 [I 2022-10-11 16:46:57] ...trap/engines/engine.py.321: eval: epoch 111 | batch 140/510 [I 2022-10-11 16:46:57] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:58 [I 2022-10-11 16:46:57] ...trap/engines/engine.py.326: process: 0.32525 | load: 0.00021 [I 2022-10-11 16:47:01] ...trap/engines/engine.py.321: eval: epoch 111 | batch 150/510 [I 2022-10-11 16:47:01] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:55 [I 2022-10-11 16:47:01] ...trap/engines/engine.py.326: process: 0.31859 | load: 0.00023 [I 2022-10-11 16:47:04] ...trap/engines/engine.py.321: eval: epoch 111 | batch 160/510 [I 2022-10-11 16:47:04] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:52 [I 2022-10-11 16:47:04] ...trap/engines/engine.py.326: process: 0.32530 | load: 0.00025 [I 2022-10-11 16:47:07] ...trap/engines/engine.py.321: eval: epoch 111 | batch 170/510 [I 2022-10-11 16:47:07] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:49 [I 2022-10-11 16:47:07] ...trap/engines/engine.py.326: process: 0.32066 | load: 0.00021 [I 2022-10-11 16:47:10] ...trap/engines/engine.py.321: eval: epoch 111 | batch 180/510 [I 2022-10-11 16:47:10] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:46 [I 2022-10-11 16:47:10] ...trap/engines/engine.py.326: process: 0.32205 | load: 0.00021 [I 2022-10-11 16:47:14] ...trap/engines/engine.py.321: eval: epoch 111 | batch 190/510 [I 2022-10-11 16:47:14] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:43 [I 2022-10-11 16:47:14] ...trap/engines/engine.py.326: process: 0.32297 | load: 0.00022 [I 2022-10-11 16:47:17] ...trap/engines/engine.py.321: eval: epoch 111 | batch 200/510 [I 2022-10-11 16:47:17] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:39 [I 2022-10-11 16:47:17] ...trap/engines/engine.py.326: process: 0.32306 | load: 0.00021 [I 2022-10-11 16:47:20] ...trap/engines/engine.py.321: eval: epoch 111 | batch 210/510 [I 2022-10-11 16:47:20] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:35 [I 2022-10-11 16:47:20] ...trap/engines/engine.py.326: process: 0.31918 | load: 0.00021 [I 2022-10-11 16:47:23] ...trap/engines/engine.py.321: eval: epoch 111 | batch 220/510 [I 2022-10-11 16:47:23] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:01:32 [I 2022-10-11 16:47:23] ...trap/engines/engine.py.326: process: 0.31872 | load: 0.00022 [I 2022-10-11 16:47:26] ...trap/engines/engine.py.321: eval: epoch 111 | batch 230/510 [I 2022-10-11 16:47:26] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:28 [I 2022-10-11 16:47:26] ...trap/engines/engine.py.326: process: 0.32112 | load: 0.00021 [I 2022-10-11 16:47:29] ...trap/engines/engine.py.321: eval: epoch 111 | batch 240/510 [I 2022-10-11 16:47:29] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:25 [I 2022-10-11 16:47:29] ...trap/engines/engine.py.326: process: 0.31895 | load: 0.00020 [I 2022-10-11 16:47:33] ...trap/engines/engine.py.321: eval: epoch 111 | batch 250/510 [I 2022-10-11 16:47:33] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:22 [I 2022-10-11 16:47:33] ...trap/engines/engine.py.326: process: 0.32178 | load: 0.00021 [I 2022-10-11 16:47:36] ...trap/engines/engine.py.321: eval: epoch 111 | batch 260/510 [I 2022-10-11 16:47:36] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:01:19 [I 2022-10-11 16:47:36] ...trap/engines/engine.py.326: process: 0.31883 | load: 0.00020 [I 2022-10-11 16:47:39] ...trap/engines/engine.py.321: eval: epoch 111 | batch 270/510 [I 2022-10-11 16:47:39] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:15 [I 2022-10-11 16:47:39] ...trap/engines/engine.py.326: process: 0.31919 | load: 0.00020 [I 2022-10-11 16:47:42] ...trap/engines/engine.py.321: eval: epoch 111 | batch 280/510 [I 2022-10-11 16:47:42] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:13 [I 2022-10-11 16:47:42] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00019 [I 2022-10-11 16:47:45] ...trap/engines/engine.py.321: eval: epoch 111 | batch 290/510 [I 2022-10-11 16:47:45] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:09 [I 2022-10-11 16:47:45] ...trap/engines/engine.py.326: process: 0.31452 | load: 0.00020 [I 2022-10-11 16:47:49] ...trap/engines/engine.py.321: eval: epoch 111 | batch 300/510 [I 2022-10-11 16:47:49] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:06 [I 2022-10-11 16:47:49] ...trap/engines/engine.py.326: process: 0.31715 | load: 0.00021 [I 2022-10-11 16:47:52] ...trap/engines/engine.py.321: eval: epoch 111 | batch 310/510 [I 2022-10-11 16:47:52] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:03 [I 2022-10-11 16:47:52] ...trap/engines/engine.py.326: process: 0.31727 | load: 0.00020 [I 2022-10-11 16:47:55] ...trap/engines/engine.py.321: eval: epoch 111 | batch 320/510 [I 2022-10-11 16:47:55] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:59 [I 2022-10-11 16:47:55] ...trap/engines/engine.py.326: process: 0.30770 | load: 0.00020 [I 2022-10-11 16:47:58] ...trap/engines/engine.py.321: eval: epoch 111 | batch 330/510 [I 2022-10-11 16:47:58] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:56 [I 2022-10-11 16:47:58] ...trap/engines/engine.py.326: process: 0.31781 | load: 0.00020 [I 2022-10-11 16:48:01] ...trap/engines/engine.py.321: eval: epoch 111 | batch 340/510 [I 2022-10-11 16:48:01] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:53 [I 2022-10-11 16:48:01] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00020 [I 2022-10-11 16:48:04] ...trap/engines/engine.py.321: eval: epoch 111 | batch 350/510 [I 2022-10-11 16:48:04] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:50 [I 2022-10-11 16:48:04] ...trap/engines/engine.py.326: process: 0.31744 | load: 0.00021 [I 2022-10-11 16:48:08] ...trap/engines/engine.py.321: eval: epoch 111 | batch 360/510 [I 2022-10-11 16:48:08] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:47 [I 2022-10-11 16:48:08] ...trap/engines/engine.py.326: process: 0.30661 | load: 0.00020 [I 2022-10-11 16:48:11] ...trap/engines/engine.py.321: eval: epoch 111 | batch 370/510 [I 2022-10-11 16:48:11] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:44 [I 2022-10-11 16:48:11] ...trap/engines/engine.py.326: process: 0.31372 | load: 0.00020 [I 2022-10-11 16:48:14] ...trap/engines/engine.py.321: eval: epoch 111 | batch 380/510 [I 2022-10-11 16:48:14] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:41 [I 2022-10-11 16:48:14] ...trap/engines/engine.py.326: process: 0.32004 | load: 0.00020 [I 2022-10-11 16:48:17] ...trap/engines/engine.py.321: eval: epoch 111 | batch 390/510 [I 2022-10-11 16:48:17] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:37 [I 2022-10-11 16:48:17] ...trap/engines/engine.py.326: process: 0.31348 | load: 0.00020 [I 2022-10-11 16:48:20] ...trap/engines/engine.py.321: eval: epoch 111 | batch 400/510 [I 2022-10-11 16:48:20] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:34 [I 2022-10-11 16:48:20] ...trap/engines/engine.py.326: process: 0.31441 | load: 0.00019 [I 2022-10-11 16:48:23] ...trap/engines/engine.py.321: eval: epoch 111 | batch 410/510 [I 2022-10-11 16:48:23] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:31 [I 2022-10-11 16:48:23] ...trap/engines/engine.py.326: process: 0.32566 | load: 0.00020 [I 2022-10-11 16:48:27] ...trap/engines/engine.py.321: eval: epoch 111 | batch 420/510 [I 2022-10-11 16:48:27] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:28 [I 2022-10-11 16:48:27] ...trap/engines/engine.py.326: process: 0.31873 | load: 0.00020 [I 2022-10-11 16:48:30] ...trap/engines/engine.py.321: eval: epoch 111 | batch 430/510 [I 2022-10-11 16:48:30] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:25 [I 2022-10-11 16:48:30] ...trap/engines/engine.py.326: process: 0.32444 | load: 0.00020 [I 2022-10-11 16:48:33] ...trap/engines/engine.py.321: eval: epoch 111 | batch 440/510 [I 2022-10-11 16:48:33] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:21 [I 2022-10-11 16:48:33] ...trap/engines/engine.py.326: process: 0.30789 | load: 0.00019 [I 2022-10-11 16:48:36] ...trap/engines/engine.py.321: eval: epoch 111 | batch 450/510 [I 2022-10-11 16:48:36] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:19 [I 2022-10-11 16:48:36] ...trap/engines/engine.py.326: process: 0.31910 | load: 0.00020 [I 2022-10-11 16:48:39] ...trap/engines/engine.py.321: eval: epoch 111 | batch 460/510 [I 2022-10-11 16:48:39] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:15 [I 2022-10-11 16:48:39] ...trap/engines/engine.py.326: process: 0.32090 | load: 0.00019 [I 2022-10-11 16:48:42] ...trap/engines/engine.py.321: eval: epoch 111 | batch 470/510 [I 2022-10-11 16:48:42] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:12 [I 2022-10-11 16:48:42] ...trap/engines/engine.py.326: process: 0.31702 | load: 0.00020 [I 2022-10-11 16:48:46] ...trap/engines/engine.py.321: eval: epoch 111 | batch 480/510 [I 2022-10-11 16:48:46] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:09 [I 2022-10-11 16:48:46] ...trap/engines/engine.py.326: process: 0.31899 | load: 0.00020 [I 2022-10-11 16:48:49] ...trap/engines/engine.py.321: eval: epoch 111 | batch 490/510 [I 2022-10-11 16:48:49] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:06 [I 2022-10-11 16:48:49] ...trap/engines/engine.py.326: process: 0.31705 | load: 0.00020 [I 2022-10-11 16:48:52] ...trap/engines/engine.py.321: eval: epoch 111 | batch 500/510 [I 2022-10-11 16:48:52] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:03 [I 2022-10-11 16:48:52] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00019 [I 2022-10-11 16:48:55] ...trap/engines/engine.py.321: eval: epoch 111 | batch 510/510 [I 2022-10-11 16:48:55] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:00 [I 2022-10-11 16:48:55] ...trap/engines/engine.py.326: process: 0.31444 | load: 0.00014 [S 2022-10-11 16:48:56] ...trap/engines/engine.py.132: eval_epoch.epoch: 111 [I 2022-10-11 16:48:56] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.743400 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908500 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934100 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.738100 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909300 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934900 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009394 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006054 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005394 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011929 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008013 [S 2022-10-11 16:49:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005907 [I 2022-10-11 16:49:39] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 16:49:39] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 16:49:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 16:49:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 16:49:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 16:49:44] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 16:49:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 16:49:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 16:49:51] ...trap/engines/engine.py.163: Training model on trainset for epoch 112 [I 2022-10-11 16:49:51] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 16:49:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 0/2304 [I 2022-10-11 16:49:54] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:00:34 [I 2022-10-11 16:49:54] ...trap/engines/engine.py.229: process: 3.14008 | load: 2.21549 [I 2022-10-11 16:49:54] ...trap/engines/engine.py.230: loss: 0.13298 [I 2022-10-11 16:49:54] ...optimizers/trijoint.py.151: total_norm: 21.830578 [I 2022-10-11 16:50:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 10/2304 [I 2022-10-11 16:50:02] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:36 [I 2022-10-11 16:50:02] ...trap/engines/engine.py.229: process: 0.85191 | load: 0.00546 [I 2022-10-11 16:50:02] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-11 16:50:02] ...optimizers/trijoint.py.151: total_norm: 16.988512 [I 2022-10-11 16:50:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 20/2304 [I 2022-10-11 16:50:11] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:04 [I 2022-10-11 16:50:11] ...trap/engines/engine.py.229: process: 0.84070 | load: 0.00438 [I 2022-10-11 16:50:11] ...trap/engines/engine.py.230: loss: 0.16736 [I 2022-10-11 16:50:11] ...optimizers/trijoint.py.151: total_norm: 20.498438 [I 2022-10-11 16:50:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 30/2304 [I 2022-10-11 16:50:19] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:03 [I 2022-10-11 16:50:19] ...trap/engines/engine.py.229: process: 0.84506 | load: 0.00428 [I 2022-10-11 16:50:19] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-11 16:50:19] ...optimizers/trijoint.py.151: total_norm: 21.653648 [I 2022-10-11 16:50:28] ...trap/engines/engine.py.224: train: epoch 112 | batch 40/2304 [I 2022-10-11 16:50:28] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:32:00 [I 2022-10-11 16:50:28] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00373 [I 2022-10-11 16:50:28] ...trap/engines/engine.py.230: loss: 0.13038 [I 2022-10-11 16:50:28] ...optimizers/trijoint.py.151: total_norm: 16.926985 [I 2022-10-11 16:50:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 50/2304 [I 2022-10-11 16:50:36] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:53 [I 2022-10-11 16:50:36] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00441 [I 2022-10-11 16:50:36] ...trap/engines/engine.py.230: loss: 0.10955 [I 2022-10-11 16:50:36] ...optimizers/trijoint.py.151: total_norm: 17.445850 [I 2022-10-11 16:50:45] ...trap/engines/engine.py.224: train: epoch 112 | batch 60/2304 [I 2022-10-11 16:50:45] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:55 [I 2022-10-11 16:50:45] ...trap/engines/engine.py.229: process: 0.84322 | load: 0.00349 [I 2022-10-11 16:50:45] ...trap/engines/engine.py.230: loss: 0.08213 [I 2022-10-11 16:50:45] ...optimizers/trijoint.py.151: total_norm: 7.989018 [I 2022-10-11 16:50:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 70/2304 [I 2022-10-11 16:50:53] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:48 [I 2022-10-11 16:50:53] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00364 [I 2022-10-11 16:50:53] ...trap/engines/engine.py.230: loss: 0.08326 [I 2022-10-11 16:50:53] ...optimizers/trijoint.py.151: total_norm: 9.573030 [I 2022-10-11 16:51:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 80/2304 [I 2022-10-11 16:51:02] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:53 [I 2022-10-11 16:51:02] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00361 [I 2022-10-11 16:51:02] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-11 16:51:02] ...optimizers/trijoint.py.151: total_norm: 13.966032 [I 2022-10-11 16:51:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 90/2304 [I 2022-10-11 16:51:11] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:32:38 [I 2022-10-11 16:51:11] ...trap/engines/engine.py.229: process: 0.88360 | load: 0.00481 [I 2022-10-11 16:51:11] ...trap/engines/engine.py.230: loss: 0.14868 [I 2022-10-11 16:51:11] ...optimizers/trijoint.py.151: total_norm: 13.894681 [I 2022-10-11 16:51:20] ...trap/engines/engine.py.224: train: epoch 112 | batch 100/2304 [I 2022-10-11 16:51:20] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:32:16 [I 2022-10-11 16:51:20] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00398 [I 2022-10-11 16:51:20] ...trap/engines/engine.py.230: loss: 0.09221 [I 2022-10-11 16:51:20] ...optimizers/trijoint.py.151: total_norm: 11.745960 [I 2022-10-11 16:51:28] ...trap/engines/engine.py.224: train: epoch 112 | batch 110/2304 [I 2022-10-11 16:51:28] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:40 [I 2022-10-11 16:51:28] ...trap/engines/engine.py.229: process: 0.85297 | load: 0.00335 [I 2022-10-11 16:51:28] ...trap/engines/engine.py.230: loss: 0.11888 [I 2022-10-11 16:51:28] ...optimizers/trijoint.py.151: total_norm: 15.353558 [I 2022-10-11 16:51:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 120/2304 [I 2022-10-11 16:51:37] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:47 [I 2022-10-11 16:51:37] ...trap/engines/engine.py.229: process: 0.87775 | load: 0.00336 [I 2022-10-11 16:51:37] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-11 16:51:37] ...optimizers/trijoint.py.151: total_norm: 11.887966 [I 2022-10-11 16:51:46] ...trap/engines/engine.py.224: train: epoch 112 | batch 130/2304 [I 2022-10-11 16:51:46] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:22 [I 2022-10-11 16:51:46] ...trap/engines/engine.py.229: process: 0.86781 | load: 0.00330 [I 2022-10-11 16:51:46] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-11 16:51:46] ...optimizers/trijoint.py.151: total_norm: 14.167856 [I 2022-10-11 16:51:55] ...trap/engines/engine.py.224: train: epoch 112 | batch 140/2304 [I 2022-10-11 16:51:55] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:31:06 [I 2022-10-11 16:51:55] ...trap/engines/engine.py.229: process: 0.86386 | load: 0.00316 [I 2022-10-11 16:51:55] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-11 16:51:55] ...optimizers/trijoint.py.151: total_norm: 11.205123 [I 2022-10-11 16:52:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 150/2304 [I 2022-10-11 16:52:03] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:31:07 [I 2022-10-11 16:52:03] ...trap/engines/engine.py.229: process: 0.85930 | load: 0.00330 [I 2022-10-11 16:52:03] ...trap/engines/engine.py.230: loss: 0.09206 [I 2022-10-11 16:52:03] ...optimizers/trijoint.py.151: total_norm: 10.203496 [I 2022-10-11 16:52:12] ...trap/engines/engine.py.224: train: epoch 112 | batch 160/2304 [I 2022-10-11 16:52:12] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:31:04 [I 2022-10-11 16:52:12] ...trap/engines/engine.py.229: process: 0.87209 | load: 0.00328 [I 2022-10-11 16:52:12] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-11 16:52:12] ...optimizers/trijoint.py.151: total_norm: 10.551844 [I 2022-10-11 16:52:21] ...trap/engines/engine.py.224: train: epoch 112 | batch 170/2304 [I 2022-10-11 16:52:21] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:48 [I 2022-10-11 16:52:21] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00369 [I 2022-10-11 16:52:21] ...trap/engines/engine.py.230: loss: 0.14957 [I 2022-10-11 16:52:21] ...optimizers/trijoint.py.151: total_norm: 9.917716 [I 2022-10-11 16:52:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 180/2304 [I 2022-10-11 16:52:29] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:23 [I 2022-10-11 16:52:29] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00338 [I 2022-10-11 16:52:29] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-11 16:52:29] ...optimizers/trijoint.py.151: total_norm: 23.368284 [I 2022-10-11 16:52:38] ...trap/engines/engine.py.224: train: epoch 112 | batch 190/2304 [I 2022-10-11 16:52:38] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:30:22 [I 2022-10-11 16:52:38] ...trap/engines/engine.py.229: process: 0.86150 | load: 0.00311 [I 2022-10-11 16:52:38] ...trap/engines/engine.py.230: loss: 0.18620 [I 2022-10-11 16:52:38] ...optimizers/trijoint.py.151: total_norm: 21.983641 [I 2022-10-11 16:52:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 200/2304 [I 2022-10-11 16:52:47] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:29 [I 2022-10-11 16:52:47] ...trap/engines/engine.py.229: process: 0.87923 | load: 0.00323 [I 2022-10-11 16:52:47] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-11 16:52:47] ...optimizers/trijoint.py.151: total_norm: 21.102644 [I 2022-10-11 16:52:55] ...trap/engines/engine.py.224: train: epoch 112 | batch 210/2304 [I 2022-10-11 16:52:55] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:30:22 [I 2022-10-11 16:52:55] ...trap/engines/engine.py.229: process: 0.87437 | load: 0.00325 [I 2022-10-11 16:52:55] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-11 16:52:55] ...optimizers/trijoint.py.151: total_norm: 23.107510 [I 2022-10-11 16:53:04] ...trap/engines/engine.py.224: train: epoch 112 | batch 220/2304 [I 2022-10-11 16:53:04] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:30:09 [I 2022-10-11 16:53:04] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00322 [I 2022-10-11 16:53:04] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-11 16:53:04] ...optimizers/trijoint.py.151: total_norm: 11.043202 [I 2022-10-11 16:53:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 230/2304 [I 2022-10-11 16:53:13] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:30:01 [I 2022-10-11 16:53:13] ...trap/engines/engine.py.229: process: 0.86264 | load: 0.00314 [I 2022-10-11 16:53:13] ...trap/engines/engine.py.230: loss: 0.08937 [I 2022-10-11 16:53:13] ...optimizers/trijoint.py.151: total_norm: 14.935088 [I 2022-10-11 16:53:21] ...trap/engines/engine.py.224: train: epoch 112 | batch 240/2304 [I 2022-10-11 16:53:21] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:57 [I 2022-10-11 16:53:21] ...trap/engines/engine.py.229: process: 0.86693 | load: 0.00315 [I 2022-10-11 16:53:21] ...trap/engines/engine.py.230: loss: 0.11127 [I 2022-10-11 16:53:21] ...optimizers/trijoint.py.151: total_norm: 8.358746 [I 2022-10-11 16:53:30] ...trap/engines/engine.py.224: train: epoch 112 | batch 250/2304 [I 2022-10-11 16:53:30] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:37 [I 2022-10-11 16:53:30] ...trap/engines/engine.py.229: process: 0.87259 | load: 0.00312 [I 2022-10-11 16:53:30] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-11 16:53:30] ...optimizers/trijoint.py.151: total_norm: 21.661791 [I 2022-10-11 16:53:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 260/2304 [I 2022-10-11 16:53:39] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:37 [I 2022-10-11 16:53:39] ...trap/engines/engine.py.229: process: 0.86595 | load: 0.00337 [I 2022-10-11 16:53:39] ...trap/engines/engine.py.230: loss: 0.19595 [I 2022-10-11 16:53:39] ...optimizers/trijoint.py.151: total_norm: 18.383717 [I 2022-10-11 16:53:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 270/2304 [I 2022-10-11 16:53:48] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:07 [I 2022-10-11 16:53:48] ...trap/engines/engine.py.229: process: 0.84484 | load: 0.00329 [I 2022-10-11 16:53:48] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-11 16:53:48] ...optimizers/trijoint.py.151: total_norm: 13.770079 [I 2022-10-11 16:53:56] ...trap/engines/engine.py.224: train: epoch 112 | batch 280/2304 [I 2022-10-11 16:53:56] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:28:54 [I 2022-10-11 16:53:56] ...trap/engines/engine.py.229: process: 0.85539 | load: 0.00317 [I 2022-10-11 16:53:56] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-11 16:53:56] ...optimizers/trijoint.py.151: total_norm: 17.383507 [I 2022-10-11 16:54:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 290/2304 [I 2022-10-11 16:54:05] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:28:45 [I 2022-10-11 16:54:05] ...trap/engines/engine.py.229: process: 0.85047 | load: 0.00311 [I 2022-10-11 16:54:05] ...trap/engines/engine.py.230: loss: 0.17273 [I 2022-10-11 16:54:05] ...optimizers/trijoint.py.151: total_norm: 14.160352 [I 2022-10-11 16:54:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 300/2304 [I 2022-10-11 16:54:13] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:27 [I 2022-10-11 16:54:13] ...trap/engines/engine.py.229: process: 0.84816 | load: 0.00313 [I 2022-10-11 16:54:13] ...trap/engines/engine.py.230: loss: 0.11866 [I 2022-10-11 16:54:13] ...optimizers/trijoint.py.151: total_norm: 15.477497 [I 2022-10-11 16:54:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 310/2304 [I 2022-10-11 16:54:22] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:26 [I 2022-10-11 16:54:22] ...trap/engines/engine.py.229: process: 0.85612 | load: 0.00313 [I 2022-10-11 16:54:22] ...trap/engines/engine.py.230: loss: 0.22322 [I 2022-10-11 16:54:22] ...optimizers/trijoint.py.151: total_norm: 41.507549 [I 2022-10-11 16:54:30] ...trap/engines/engine.py.224: train: epoch 112 | batch 320/2304 [I 2022-10-11 16:54:30] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:28:27 [I 2022-10-11 16:54:30] ...trap/engines/engine.py.229: process: 0.87200 | load: 0.00386 [I 2022-10-11 16:54:30] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-11 16:54:30] ...optimizers/trijoint.py.151: total_norm: 13.473948 [I 2022-10-11 16:54:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 330/2304 [I 2022-10-11 16:54:39] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:48 [I 2022-10-11 16:54:39] ...trap/engines/engine.py.229: process: 0.87824 | load: 0.00414 [I 2022-10-11 16:54:39] ...trap/engines/engine.py.230: loss: 0.09964 [I 2022-10-11 16:54:39] ...optimizers/trijoint.py.151: total_norm: 11.698484 [I 2022-10-11 16:54:48] ...trap/engines/engine.py.224: train: epoch 112 | batch 340/2304 [I 2022-10-11 16:54:48] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:34 [I 2022-10-11 16:54:48] ...trap/engines/engine.py.229: process: 0.86926 | load: 0.00373 [I 2022-10-11 16:54:48] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-11 16:54:48] ...optimizers/trijoint.py.151: total_norm: 40.644634 [I 2022-10-11 16:54:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 350/2304 [I 2022-10-11 16:54:57] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:24 [I 2022-10-11 16:54:57] ...trap/engines/engine.py.229: process: 0.87040 | load: 0.00373 [I 2022-10-11 16:54:57] ...trap/engines/engine.py.230: loss: 0.18047 [I 2022-10-11 16:54:57] ...optimizers/trijoint.py.151: total_norm: 8.755519 [I 2022-10-11 16:55:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 360/2304 [I 2022-10-11 16:55:05] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:28:20 [I 2022-10-11 16:55:05] ...trap/engines/engine.py.229: process: 0.85579 | load: 0.00373 [I 2022-10-11 16:55:05] ...trap/engines/engine.py.230: loss: 0.10394 [I 2022-10-11 16:55:05] ...optimizers/trijoint.py.151: total_norm: 10.163720 [I 2022-10-11 16:55:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 370/2304 [I 2022-10-11 16:55:14] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:28:07 [I 2022-10-11 16:55:14] ...trap/engines/engine.py.229: process: 0.86852 | load: 0.00384 [I 2022-10-11 16:55:14] ...trap/engines/engine.py.230: loss: 0.14757 [I 2022-10-11 16:55:14] ...optimizers/trijoint.py.151: total_norm: 31.906874 [I 2022-10-11 16:55:23] ...trap/engines/engine.py.224: train: epoch 112 | batch 380/2304 [I 2022-10-11 16:55:23] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:28:07 [I 2022-10-11 16:55:23] ...trap/engines/engine.py.229: process: 0.87732 | load: 0.00376 [I 2022-10-11 16:55:23] ...trap/engines/engine.py.230: loss: 0.07320 [I 2022-10-11 16:55:23] ...optimizers/trijoint.py.151: total_norm: 15.270205 [I 2022-10-11 16:55:32] ...trap/engines/engine.py.224: train: epoch 112 | batch 390/2304 [I 2022-10-11 16:55:32] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:53 [I 2022-10-11 16:55:32] ...trap/engines/engine.py.229: process: 0.86752 | load: 0.00388 [I 2022-10-11 16:55:32] ...trap/engines/engine.py.230: loss: 0.18300 [I 2022-10-11 16:55:32] ...optimizers/trijoint.py.151: total_norm: 10.523321 [I 2022-10-11 16:55:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 400/2304 [I 2022-10-11 16:55:40] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:42 [I 2022-10-11 16:55:40] ...trap/engines/engine.py.229: process: 0.88334 | load: 0.00374 [I 2022-10-11 16:55:40] ...trap/engines/engine.py.230: loss: 0.11308 [I 2022-10-11 16:55:40] ...optimizers/trijoint.py.151: total_norm: 12.010843 [I 2022-10-11 16:55:49] ...trap/engines/engine.py.224: train: epoch 112 | batch 410/2304 [I 2022-10-11 16:55:49] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:27:35 [I 2022-10-11 16:55:49] ...trap/engines/engine.py.229: process: 0.87629 | load: 0.00373 [I 2022-10-11 16:55:49] ...trap/engines/engine.py.230: loss: 0.10078 [I 2022-10-11 16:55:49] ...optimizers/trijoint.py.151: total_norm: 13.745449 [I 2022-10-11 16:55:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 420/2304 [I 2022-10-11 16:55:58] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:27:28 [I 2022-10-11 16:55:58] ...trap/engines/engine.py.229: process: 0.87903 | load: 0.00379 [I 2022-10-11 16:55:58] ...trap/engines/engine.py.230: loss: 0.07181 [I 2022-10-11 16:55:58] ...optimizers/trijoint.py.151: total_norm: 15.561543 [I 2022-10-11 16:56:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 430/2304 [I 2022-10-11 16:56:07] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:27:18 [I 2022-10-11 16:56:07] ...trap/engines/engine.py.229: process: 0.86370 | load: 0.00401 [I 2022-10-11 16:56:07] ...trap/engines/engine.py.230: loss: 0.17072 [I 2022-10-11 16:56:07] ...optimizers/trijoint.py.151: total_norm: 19.286510 [I 2022-10-11 16:56:15] ...trap/engines/engine.py.224: train: epoch 112 | batch 440/2304 [I 2022-10-11 16:56:15] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:27:09 [I 2022-10-11 16:56:15] ...trap/engines/engine.py.229: process: 0.88433 | load: 0.00379 [I 2022-10-11 16:56:15] ...trap/engines/engine.py.230: loss: 0.16721 [I 2022-10-11 16:56:15] ...optimizers/trijoint.py.151: total_norm: 26.721590 [I 2022-10-11 16:56:24] ...trap/engines/engine.py.224: train: epoch 112 | batch 450/2304 [I 2022-10-11 16:56:24] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:26:53 [I 2022-10-11 16:56:24] ...trap/engines/engine.py.229: process: 0.86275 | load: 0.00378 [I 2022-10-11 16:56:24] ...trap/engines/engine.py.230: loss: 0.16518 [I 2022-10-11 16:56:24] ...optimizers/trijoint.py.151: total_norm: 19.254463 [I 2022-10-11 16:56:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 460/2304 [I 2022-10-11 16:56:33] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:26:53 [I 2022-10-11 16:56:33] ...trap/engines/engine.py.229: process: 0.88377 | load: 0.00379 [I 2022-10-11 16:56:33] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-11 16:56:33] ...optimizers/trijoint.py.151: total_norm: 13.673323 [I 2022-10-11 16:56:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 470/2304 [I 2022-10-11 16:56:42] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:26:43 [I 2022-10-11 16:56:42] ...trap/engines/engine.py.229: process: 0.86829 | load: 0.00381 [I 2022-10-11 16:56:42] ...trap/engines/engine.py.230: loss: 0.10585 [I 2022-10-11 16:56:42] ...optimizers/trijoint.py.151: total_norm: 8.381500 [I 2022-10-11 16:56:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 480/2304 [I 2022-10-11 16:56:51] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:26:39 [I 2022-10-11 16:56:51] ...trap/engines/engine.py.229: process: 0.86580 | load: 0.00379 [I 2022-10-11 16:56:51] ...trap/engines/engine.py.230: loss: 0.23811 [I 2022-10-11 16:56:51] ...optimizers/trijoint.py.151: total_norm: 16.343874 [I 2022-10-11 16:56:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 490/2304 [I 2022-10-11 16:56:59] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:26:29 [I 2022-10-11 16:56:59] ...trap/engines/engine.py.229: process: 0.87474 | load: 0.00378 [I 2022-10-11 16:56:59] ...trap/engines/engine.py.230: loss: 0.13608 [I 2022-10-11 16:56:59] ...optimizers/trijoint.py.151: total_norm: 16.888861 [I 2022-10-11 16:57:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 500/2304 [I 2022-10-11 16:57:08] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:25:39 [I 2022-10-11 16:57:08] ...trap/engines/engine.py.229: process: 0.83648 | load: 0.00296 [I 2022-10-11 16:57:08] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-11 16:57:08] ...optimizers/trijoint.py.151: total_norm: 26.986441 [I 2022-10-11 16:57:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 510/2304 [I 2022-10-11 16:57:16] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:15 [I 2022-10-11 16:57:16] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00294 [I 2022-10-11 16:57:16] ...trap/engines/engine.py.230: loss: 0.08590 [I 2022-10-11 16:57:16] ...optimizers/trijoint.py.151: total_norm: 19.884453 [I 2022-10-11 16:57:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 520/2304 [I 2022-10-11 16:57:25] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:24:57 [I 2022-10-11 16:57:25] ...trap/engines/engine.py.229: process: 0.82373 | load: 0.00298 [I 2022-10-11 16:57:25] ...trap/engines/engine.py.230: loss: 0.09529 [I 2022-10-11 16:57:25] ...optimizers/trijoint.py.151: total_norm: 9.673985 [I 2022-10-11 16:57:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 530/2304 [I 2022-10-11 16:57:33] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:24:51 [I 2022-10-11 16:57:33] ...trap/engines/engine.py.229: process: 0.83073 | load: 0.00298 [I 2022-10-11 16:57:33] ...trap/engines/engine.py.230: loss: 0.08147 [I 2022-10-11 16:57:33] ...optimizers/trijoint.py.151: total_norm: 10.623234 [I 2022-10-11 16:57:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 540/2304 [I 2022-10-11 16:57:42] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:24:40 [I 2022-10-11 16:57:42] ...trap/engines/engine.py.229: process: 0.83925 | load: 0.00307 [I 2022-10-11 16:57:42] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-11 16:57:42] ...optimizers/trijoint.py.151: total_norm: 17.117620 [I 2022-10-11 16:57:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 550/2304 [I 2022-10-11 16:57:50] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:50 [I 2022-10-11 16:57:50] ...trap/engines/engine.py.229: process: 0.84502 | load: 0.00311 [I 2022-10-11 16:57:50] ...trap/engines/engine.py.230: loss: 0.16933 [I 2022-10-11 16:57:50] ...optimizers/trijoint.py.151: total_norm: 12.498672 [I 2022-10-11 16:57:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 560/2304 [I 2022-10-11 16:57:59] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:53 [I 2022-10-11 16:57:59] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00323 [I 2022-10-11 16:57:59] ...trap/engines/engine.py.230: loss: 0.11677 [I 2022-10-11 16:57:59] ...optimizers/trijoint.py.151: total_norm: 11.702681 [I 2022-10-11 16:58:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 570/2304 [I 2022-10-11 16:58:07] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:45 [I 2022-10-11 16:58:07] ...trap/engines/engine.py.229: process: 0.86020 | load: 0.00328 [I 2022-10-11 16:58:07] ...trap/engines/engine.py.230: loss: 0.13248 [I 2022-10-11 16:58:07] ...optimizers/trijoint.py.151: total_norm: 13.756709 [I 2022-10-11 16:58:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 580/2304 [I 2022-10-11 16:58:16] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:37 [I 2022-10-11 16:58:16] ...trap/engines/engine.py.229: process: 0.84685 | load: 0.00314 [I 2022-10-11 16:58:16] ...trap/engines/engine.py.230: loss: 0.15998 [I 2022-10-11 16:58:16] ...optimizers/trijoint.py.151: total_norm: 10.914480 [I 2022-10-11 16:58:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 590/2304 [I 2022-10-11 16:58:25] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:29 [I 2022-10-11 16:58:25] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00313 [I 2022-10-11 16:58:25] ...trap/engines/engine.py.230: loss: 0.22120 [I 2022-10-11 16:58:25] ...optimizers/trijoint.py.151: total_norm: 17.038214 [I 2022-10-11 16:58:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 600/2304 [I 2022-10-11 16:58:33] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:17 [I 2022-10-11 16:58:33] ...trap/engines/engine.py.229: process: 0.84968 | load: 0.00313 [I 2022-10-11 16:58:33] ...trap/engines/engine.py.230: loss: 0.18034 [I 2022-10-11 16:58:33] ...optimizers/trijoint.py.151: total_norm: 12.346606 [I 2022-10-11 16:58:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 610/2304 [I 2022-10-11 16:58:42] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:24:31 [I 2022-10-11 16:58:42] ...trap/engines/engine.py.229: process: 0.87131 | load: 0.00312 [I 2022-10-11 16:58:42] ...trap/engines/engine.py.230: loss: 0.08552 [I 2022-10-11 16:58:42] ...optimizers/trijoint.py.151: total_norm: 9.416087 [I 2022-10-11 16:58:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 620/2304 [I 2022-10-11 16:58:51] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:24:16 [I 2022-10-11 16:58:51] ...trap/engines/engine.py.229: process: 0.87145 | load: 0.00311 [I 2022-10-11 16:58:51] ...trap/engines/engine.py.230: loss: 0.11259 [I 2022-10-11 16:58:51] ...optimizers/trijoint.py.151: total_norm: 10.013775 [I 2022-10-11 16:58:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 630/2304 [I 2022-10-11 16:58:59] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:24:17 [I 2022-10-11 16:58:59] ...trap/engines/engine.py.229: process: 0.86756 | load: 0.00314 [I 2022-10-11 16:58:59] ...trap/engines/engine.py.230: loss: 0.14741 [I 2022-10-11 16:58:59] ...optimizers/trijoint.py.151: total_norm: 13.599428 [I 2022-10-11 16:59:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 640/2304 [I 2022-10-11 16:59:08] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:24:02 [I 2022-10-11 16:59:08] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00311 [I 2022-10-11 16:59:08] ...trap/engines/engine.py.230: loss: 0.09360 [I 2022-10-11 16:59:08] ...optimizers/trijoint.py.151: total_norm: 15.472268 [I 2022-10-11 16:59:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 650/2304 [I 2022-10-11 16:59:17] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:58 [I 2022-10-11 16:59:17] ...trap/engines/engine.py.229: process: 0.86815 | load: 0.00312 [I 2022-10-11 16:59:17] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-11 16:59:17] ...optimizers/trijoint.py.151: total_norm: 16.637148 [I 2022-10-11 16:59:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 660/2304 [I 2022-10-11 16:59:25] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:40 [I 2022-10-11 16:59:25] ...trap/engines/engine.py.229: process: 0.85710 | load: 0.00325 [I 2022-10-11 16:59:25] ...trap/engines/engine.py.230: loss: 0.08512 [I 2022-10-11 16:59:25] ...optimizers/trijoint.py.151: total_norm: 10.174676 [I 2022-10-11 16:59:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 670/2304 [I 2022-10-11 16:59:34] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:23:23 [I 2022-10-11 16:59:34] ...trap/engines/engine.py.229: process: 0.85326 | load: 0.00312 [I 2022-10-11 16:59:34] ...trap/engines/engine.py.230: loss: 0.08042 [I 2022-10-11 16:59:34] ...optimizers/trijoint.py.151: total_norm: 9.775146 [I 2022-10-11 16:59:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 680/2304 [I 2022-10-11 16:59:43] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:23:19 [I 2022-10-11 16:59:43] ...trap/engines/engine.py.229: process: 0.86748 | load: 0.00312 [I 2022-10-11 16:59:43] ...trap/engines/engine.py.230: loss: 0.09705 [I 2022-10-11 16:59:43] ...optimizers/trijoint.py.151: total_norm: 11.545567 [I 2022-10-11 16:59:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 690/2304 [I 2022-10-11 16:59:51] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:23:01 [I 2022-10-11 16:59:51] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00314 [I 2022-10-11 16:59:51] ...trap/engines/engine.py.230: loss: 0.08545 [I 2022-10-11 16:59:51] ...optimizers/trijoint.py.151: total_norm: 27.101542 [I 2022-10-11 17:00:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 700/2304 [I 2022-10-11 17:00:00] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:22:54 [I 2022-10-11 17:00:00] ...trap/engines/engine.py.229: process: 0.86766 | load: 0.00316 [I 2022-10-11 17:00:00] ...trap/engines/engine.py.230: loss: 0.08725 [I 2022-10-11 17:00:00] ...optimizers/trijoint.py.151: total_norm: 16.290539 [I 2022-10-11 17:00:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 710/2304 [I 2022-10-11 17:00:08] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:54 [I 2022-10-11 17:00:08] ...trap/engines/engine.py.229: process: 0.85964 | load: 0.00312 [I 2022-10-11 17:00:08] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-11 17:00:08] ...optimizers/trijoint.py.151: total_norm: 15.197025 [I 2022-10-11 17:00:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 720/2304 [I 2022-10-11 17:00:17] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:51 [I 2022-10-11 17:00:17] ...trap/engines/engine.py.229: process: 0.87013 | load: 0.00314 [I 2022-10-11 17:00:17] ...trap/engines/engine.py.230: loss: 0.21911 [I 2022-10-11 17:00:17] ...optimizers/trijoint.py.151: total_norm: 10.414207 [I 2022-10-11 17:00:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 730/2304 [I 2022-10-11 17:00:26] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:38 [I 2022-10-11 17:00:26] ...trap/engines/engine.py.229: process: 0.86993 | load: 0.00315 [I 2022-10-11 17:00:26] ...trap/engines/engine.py.230: loss: 0.11594 [I 2022-10-11 17:00:26] ...optimizers/trijoint.py.151: total_norm: 14.491438 [I 2022-10-11 17:00:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 740/2304 [I 2022-10-11 17:00:34] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:32 [I 2022-10-11 17:00:34] ...trap/engines/engine.py.229: process: 0.86112 | load: 0.00315 [I 2022-10-11 17:00:34] ...trap/engines/engine.py.230: loss: 0.14073 [I 2022-10-11 17:00:34] ...optimizers/trijoint.py.151: total_norm: 7.505563 [I 2022-10-11 17:00:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 750/2304 [I 2022-10-11 17:00:43] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:22:26 [I 2022-10-11 17:00:43] ...trap/engines/engine.py.229: process: 0.86859 | load: 0.00312 [I 2022-10-11 17:00:43] ...trap/engines/engine.py.230: loss: 0.10102 [I 2022-10-11 17:00:43] ...optimizers/trijoint.py.151: total_norm: 9.066426 [I 2022-10-11 17:00:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 760/2304 [I 2022-10-11 17:00:52] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:22:24 [I 2022-10-11 17:00:52] ...trap/engines/engine.py.229: process: 0.88548 | load: 0.00312 [I 2022-10-11 17:00:52] ...trap/engines/engine.py.230: loss: 0.10435 [I 2022-10-11 17:00:52] ...optimizers/trijoint.py.151: total_norm: 13.941915 [I 2022-10-11 17:01:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 770/2304 [I 2022-10-11 17:01:00] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:57 [I 2022-10-11 17:01:00] ...trap/engines/engine.py.229: process: 0.86103 | load: 0.00314 [I 2022-10-11 17:01:00] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-11 17:01:00] ...optimizers/trijoint.py.151: total_norm: 30.811724 [I 2022-10-11 17:01:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 780/2304 [I 2022-10-11 17:01:09] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:21:39 [I 2022-10-11 17:01:09] ...trap/engines/engine.py.229: process: 0.85318 | load: 0.00314 [I 2022-10-11 17:01:09] ...trap/engines/engine.py.230: loss: 0.17032 [I 2022-10-11 17:01:09] ...optimizers/trijoint.py.151: total_norm: 22.477852 [I 2022-10-11 17:01:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 790/2304 [I 2022-10-11 17:01:17] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:34 [I 2022-10-11 17:01:17] ...trap/engines/engine.py.229: process: 0.87039 | load: 0.00315 [I 2022-10-11 17:01:17] ...trap/engines/engine.py.230: loss: 0.08015 [I 2022-10-11 17:01:17] ...optimizers/trijoint.py.151: total_norm: 16.425385 [I 2022-10-11 17:01:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 800/2304 [I 2022-10-11 17:01:26] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:21:29 [I 2022-10-11 17:01:26] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00312 [I 2022-10-11 17:01:26] ...trap/engines/engine.py.230: loss: 0.10518 [I 2022-10-11 17:01:26] ...optimizers/trijoint.py.151: total_norm: 24.408560 [I 2022-10-11 17:01:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 810/2304 [I 2022-10-11 17:01:35] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:16 [I 2022-10-11 17:01:35] ...trap/engines/engine.py.229: process: 0.86012 | load: 0.00313 [I 2022-10-11 17:01:35] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-11 17:01:35] ...optimizers/trijoint.py.151: total_norm: 10.284764 [I 2022-10-11 17:01:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 820/2304 [I 2022-10-11 17:01:43] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:21:04 [I 2022-10-11 17:01:43] ...trap/engines/engine.py.229: process: 0.85062 | load: 0.00299 [I 2022-10-11 17:01:43] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-11 17:01:43] ...optimizers/trijoint.py.151: total_norm: 9.993035 [I 2022-10-11 17:01:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 830/2304 [I 2022-10-11 17:01:52] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:20:46 [I 2022-10-11 17:01:52] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00296 [I 2022-10-11 17:01:52] ...trap/engines/engine.py.230: loss: 0.11409 [I 2022-10-11 17:01:52] ...optimizers/trijoint.py.151: total_norm: 15.116448 [I 2022-10-11 17:02:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 840/2304 [I 2022-10-11 17:02:00] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:50 [I 2022-10-11 17:02:00] ...trap/engines/engine.py.229: process: 0.84434 | load: 0.00294 [I 2022-10-11 17:02:00] ...trap/engines/engine.py.230: loss: 0.13990 [I 2022-10-11 17:02:00] ...optimizers/trijoint.py.151: total_norm: 8.777791 [I 2022-10-11 17:02:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 850/2304 [I 2022-10-11 17:02:09] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:38 [I 2022-10-11 17:02:09] ...trap/engines/engine.py.229: process: 0.84910 | load: 0.00294 [I 2022-10-11 17:02:09] ...trap/engines/engine.py.230: loss: 0.09593 [I 2022-10-11 17:02:09] ...optimizers/trijoint.py.151: total_norm: 14.185779 [I 2022-10-11 17:02:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 860/2304 [I 2022-10-11 17:02:17] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:29 [I 2022-10-11 17:02:17] ...trap/engines/engine.py.229: process: 0.85693 | load: 0.00292 [I 2022-10-11 17:02:17] ...trap/engines/engine.py.230: loss: 0.09229 [I 2022-10-11 17:02:17] ...optimizers/trijoint.py.151: total_norm: 11.681898 [I 2022-10-11 17:02:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 870/2304 [I 2022-10-11 17:02:26] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:20:18 [I 2022-10-11 17:02:26] ...trap/engines/engine.py.229: process: 0.85894 | load: 0.00295 [I 2022-10-11 17:02:26] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-11 17:02:26] ...optimizers/trijoint.py.151: total_norm: 17.444122 [I 2022-10-11 17:02:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 880/2304 [I 2022-10-11 17:02:34] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:19:59 [I 2022-10-11 17:02:34] ...trap/engines/engine.py.229: process: 0.83502 | load: 0.00295 [I 2022-10-11 17:02:34] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-11 17:02:34] ...optimizers/trijoint.py.151: total_norm: 15.555996 [I 2022-10-11 17:02:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 890/2304 [I 2022-10-11 17:02:43] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:19:54 [I 2022-10-11 17:02:43] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00296 [I 2022-10-11 17:02:43] ...trap/engines/engine.py.230: loss: 0.10564 [I 2022-10-11 17:02:43] ...optimizers/trijoint.py.151: total_norm: 76.860397 [I 2022-10-11 17:02:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 900/2304 [I 2022-10-11 17:02:51] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:19:40 [I 2022-10-11 17:02:51] ...trap/engines/engine.py.229: process: 0.84562 | load: 0.00295 [I 2022-10-11 17:02:51] ...trap/engines/engine.py.230: loss: 0.07630 [I 2022-10-11 17:02:51] ...optimizers/trijoint.py.151: total_norm: 11.200188 [I 2022-10-11 17:03:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 910/2304 [I 2022-10-11 17:03:00] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:19:38 [I 2022-10-11 17:03:00] ...trap/engines/engine.py.229: process: 0.83732 | load: 0.00295 [I 2022-10-11 17:03:00] ...trap/engines/engine.py.230: loss: 0.09064 [I 2022-10-11 17:03:00] ...optimizers/trijoint.py.151: total_norm: 11.150803 [I 2022-10-11 17:03:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 920/2304 [I 2022-10-11 17:03:08] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:19:31 [I 2022-10-11 17:03:08] ...trap/engines/engine.py.229: process: 0.84009 | load: 0.00297 [I 2022-10-11 17:03:08] ...trap/engines/engine.py.230: loss: 0.13684 [I 2022-10-11 17:03:08] ...optimizers/trijoint.py.151: total_norm: 15.302899 [I 2022-10-11 17:03:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 930/2304 [I 2022-10-11 17:03:17] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:19:22 [I 2022-10-11 17:03:17] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00295 [I 2022-10-11 17:03:17] ...trap/engines/engine.py.230: loss: 0.09845 [I 2022-10-11 17:03:17] ...optimizers/trijoint.py.151: total_norm: 8.371182 [I 2022-10-11 17:03:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 940/2304 [I 2022-10-11 17:03:25] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:19:06 [I 2022-10-11 17:03:25] ...trap/engines/engine.py.229: process: 0.84114 | load: 0.00295 [I 2022-10-11 17:03:25] ...trap/engines/engine.py.230: loss: 0.16022 [I 2022-10-11 17:03:25] ...optimizers/trijoint.py.151: total_norm: 37.080425 [I 2022-10-11 17:03:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 950/2304 [I 2022-10-11 17:03:34] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:19:10 [I 2022-10-11 17:03:34] ...trap/engines/engine.py.229: process: 0.84869 | load: 0.00296 [I 2022-10-11 17:03:34] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-11 17:03:34] ...optimizers/trijoint.py.151: total_norm: 26.665379 [I 2022-10-11 17:03:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 960/2304 [I 2022-10-11 17:03:42] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:19:07 [I 2022-10-11 17:03:42] ...trap/engines/engine.py.229: process: 0.86758 | load: 0.00296 [I 2022-10-11 17:03:42] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-11 17:03:42] ...optimizers/trijoint.py.151: total_norm: 13.731348 [I 2022-10-11 17:03:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 970/2304 [I 2022-10-11 17:03:51] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:18:57 [I 2022-10-11 17:03:51] ...trap/engines/engine.py.229: process: 0.86997 | load: 0.00297 [I 2022-10-11 17:03:51] ...trap/engines/engine.py.230: loss: 0.20635 [I 2022-10-11 17:03:51] ...optimizers/trijoint.py.151: total_norm: 29.942076 [I 2022-10-11 17:03:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 980/2304 [I 2022-10-11 17:03:59] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:18:54 [I 2022-10-11 17:03:59] ...trap/engines/engine.py.229: process: 0.86601 | load: 0.00296 [I 2022-10-11 17:03:59] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-11 17:03:59] ...optimizers/trijoint.py.151: total_norm: 15.326422 [I 2022-10-11 17:04:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 990/2304 [I 2022-10-11 17:04:08] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:18:28 [I 2022-10-11 17:04:08] ...trap/engines/engine.py.229: process: 0.82175 | load: 0.00296 [I 2022-10-11 17:04:08] ...trap/engines/engine.py.230: loss: 0.09941 [I 2022-10-11 17:04:08] ...optimizers/trijoint.py.151: total_norm: 17.479250 [I 2022-10-11 17:04:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 1000/2304 [I 2022-10-11 17:04:16] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:18:28 [I 2022-10-11 17:04:16] ...trap/engines/engine.py.229: process: 0.85653 | load: 0.00301 [I 2022-10-11 17:04:16] ...trap/engines/engine.py.230: loss: 0.10152 [I 2022-10-11 17:04:16] ...optimizers/trijoint.py.151: total_norm: 17.790434 [I 2022-10-11 17:04:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 1010/2304 [I 2022-10-11 17:04:25] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:18:08 [I 2022-10-11 17:04:25] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00296 [I 2022-10-11 17:04:25] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-11 17:04:25] ...optimizers/trijoint.py.151: total_norm: 15.032941 [I 2022-10-11 17:04:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 1020/2304 [I 2022-10-11 17:04:33] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:18:12 [I 2022-10-11 17:04:33] ...trap/engines/engine.py.229: process: 0.83989 | load: 0.00301 [I 2022-10-11 17:04:33] ...trap/engines/engine.py.230: loss: 0.07819 [I 2022-10-11 17:04:33] ...optimizers/trijoint.py.151: total_norm: 18.226257 [I 2022-10-11 17:04:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 1030/2304 [I 2022-10-11 17:04:42] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:17:54 [I 2022-10-11 17:04:42] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00296 [I 2022-10-11 17:04:42] ...trap/engines/engine.py.230: loss: 0.09396 [I 2022-10-11 17:04:42] ...optimizers/trijoint.py.151: total_norm: 16.782234 [I 2022-10-11 17:04:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 1040/2304 [I 2022-10-11 17:04:50] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:17:52 [I 2022-10-11 17:04:50] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00299 [I 2022-10-11 17:04:50] ...trap/engines/engine.py.230: loss: 0.08730 [I 2022-10-11 17:04:50] ...optimizers/trijoint.py.151: total_norm: 24.985731 [I 2022-10-11 17:04:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 1050/2304 [I 2022-10-11 17:04:59] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:37 [I 2022-10-11 17:04:59] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00304 [I 2022-10-11 17:04:59] ...trap/engines/engine.py.230: loss: 0.09969 [I 2022-10-11 17:04:59] ...optimizers/trijoint.py.151: total_norm: 14.628018 [I 2022-10-11 17:05:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 1060/2304 [I 2022-10-11 17:05:07] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:31 [I 2022-10-11 17:05:07] ...trap/engines/engine.py.229: process: 0.83015 | load: 0.00300 [I 2022-10-11 17:05:07] ...trap/engines/engine.py.230: loss: 0.19636 [I 2022-10-11 17:05:07] ...optimizers/trijoint.py.151: total_norm: 27.445847 [I 2022-10-11 17:05:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 1070/2304 [I 2022-10-11 17:05:16] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:28 [I 2022-10-11 17:05:16] ...trap/engines/engine.py.229: process: 0.85270 | load: 0.00300 [I 2022-10-11 17:05:16] ...trap/engines/engine.py.230: loss: 0.12696 [I 2022-10-11 17:05:16] ...optimizers/trijoint.py.151: total_norm: 26.818790 [I 2022-10-11 17:05:24] ...trap/engines/engine.py.224: train: epoch 112 | batch 1080/2304 [I 2022-10-11 17:05:24] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:12 [I 2022-10-11 17:05:24] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00299 [I 2022-10-11 17:05:24] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-11 17:05:24] ...optimizers/trijoint.py.151: total_norm: 47.494370 [I 2022-10-11 17:05:32] ...trap/engines/engine.py.224: train: epoch 112 | batch 1090/2304 [I 2022-10-11 17:05:32] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:16:56 [I 2022-10-11 17:05:32] ...trap/engines/engine.py.229: process: 0.83716 | load: 0.00297 [I 2022-10-11 17:05:32] ...trap/engines/engine.py.230: loss: 0.18992 [I 2022-10-11 17:05:32] ...optimizers/trijoint.py.151: total_norm: 28.304304 [I 2022-10-11 17:05:41] ...trap/engines/engine.py.224: train: epoch 112 | batch 1100/2304 [I 2022-10-11 17:05:41] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:16:54 [I 2022-10-11 17:05:41] ...trap/engines/engine.py.229: process: 0.84481 | load: 0.00301 [I 2022-10-11 17:05:41] ...trap/engines/engine.py.230: loss: 0.07565 [I 2022-10-11 17:05:41] ...optimizers/trijoint.py.151: total_norm: 9.456129 [I 2022-10-11 17:05:49] ...trap/engines/engine.py.224: train: epoch 112 | batch 1110/2304 [I 2022-10-11 17:05:49] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:58 [I 2022-10-11 17:05:49] ...trap/engines/engine.py.229: process: 0.86722 | load: 0.00312 [I 2022-10-11 17:05:49] ...trap/engines/engine.py.230: loss: 0.12786 [I 2022-10-11 17:05:49] ...optimizers/trijoint.py.151: total_norm: 26.755651 [I 2022-10-11 17:05:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 1120/2304 [I 2022-10-11 17:05:58] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:51 [I 2022-10-11 17:05:58] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00313 [I 2022-10-11 17:05:58] ...trap/engines/engine.py.230: loss: 0.11169 [I 2022-10-11 17:05:58] ...optimizers/trijoint.py.151: total_norm: 26.257183 [I 2022-10-11 17:06:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1130/2304 [I 2022-10-11 17:06:26] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:29:35 [I 2022-10-11 17:06:26] ...trap/engines/engine.py.229: process: 0.85503 | load: 0.00313 [I 2022-10-11 17:06:26] ...trap/engines/engine.py.230: loss: 0.10108 [I 2022-10-11 17:06:26] ...optimizers/trijoint.py.151: total_norm: 13.708532 [I 2022-10-11 17:06:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1140/2304 [I 2022-10-11 17:06:35] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:17:57 [I 2022-10-11 17:06:35] ...trap/engines/engine.py.229: process: 0.85906 | load: 0.00313 [I 2022-10-11 17:06:35] ...trap/engines/engine.py.230: loss: 0.10773 [I 2022-10-11 17:06:35] ...optimizers/trijoint.py.151: total_norm: 10.367386 [I 2022-10-11 17:06:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1150/2304 [I 2022-10-11 17:06:43] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:16:32 [I 2022-10-11 17:06:43] ...trap/engines/engine.py.229: process: 0.86188 | load: 0.00316 [I 2022-10-11 17:06:43] ...trap/engines/engine.py.230: loss: 0.15428 [I 2022-10-11 17:06:43] ...optimizers/trijoint.py.151: total_norm: 15.817286 [I 2022-10-11 17:06:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1160/2304 [I 2022-10-11 17:06:52] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:16:16 [I 2022-10-11 17:06:52] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00315 [I 2022-10-11 17:06:52] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-11 17:06:52] ...optimizers/trijoint.py.151: total_norm: 16.574665 [I 2022-10-11 17:07:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 1170/2304 [I 2022-10-11 17:07:00] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:16:05 [I 2022-10-11 17:07:00] ...trap/engines/engine.py.229: process: 0.83976 | load: 0.00314 [I 2022-10-11 17:07:00] ...trap/engines/engine.py.230: loss: 0.11707 [I 2022-10-11 17:07:00] ...optimizers/trijoint.py.151: total_norm: 15.799091 [I 2022-10-11 17:07:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1180/2304 [I 2022-10-11 17:07:09] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:16:02 [I 2022-10-11 17:07:09] ...trap/engines/engine.py.229: process: 0.87205 | load: 0.00321 [I 2022-10-11 17:07:09] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-11 17:07:09] ...optimizers/trijoint.py.151: total_norm: 13.188988 [I 2022-10-11 17:07:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 1190/2304 [I 2022-10-11 17:07:17] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:15:52 [I 2022-10-11 17:07:17] ...trap/engines/engine.py.229: process: 0.84468 | load: 0.00313 [I 2022-10-11 17:07:17] ...trap/engines/engine.py.230: loss: 0.12863 [I 2022-10-11 17:07:17] ...optimizers/trijoint.py.151: total_norm: 10.146685 [I 2022-10-11 17:07:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1200/2304 [I 2022-10-11 17:07:26] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:15:45 [I 2022-10-11 17:07:26] ...trap/engines/engine.py.229: process: 0.85281 | load: 0.00312 [I 2022-10-11 17:07:26] ...trap/engines/engine.py.230: loss: 0.09713 [I 2022-10-11 17:07:26] ...optimizers/trijoint.py.151: total_norm: 24.562597 [I 2022-10-11 17:07:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1210/2304 [I 2022-10-11 17:07:35] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:15:37 [I 2022-10-11 17:07:35] ...trap/engines/engine.py.229: process: 0.84789 | load: 0.00312 [I 2022-10-11 17:07:35] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-11 17:07:35] ...optimizers/trijoint.py.151: total_norm: 41.613228 [I 2022-10-11 17:07:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1220/2304 [I 2022-10-11 17:07:43] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:15:28 [I 2022-10-11 17:07:43] ...trap/engines/engine.py.229: process: 0.86354 | load: 0.00315 [I 2022-10-11 17:07:43] ...trap/engines/engine.py.230: loss: 0.09884 [I 2022-10-11 17:07:43] ...optimizers/trijoint.py.151: total_norm: 10.053001 [I 2022-10-11 17:07:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1230/2304 [I 2022-10-11 17:07:52] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:16 [I 2022-10-11 17:07:52] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00315 [I 2022-10-11 17:07:52] ...trap/engines/engine.py.230: loss: 0.10850 [I 2022-10-11 17:07:52] ...optimizers/trijoint.py.151: total_norm: 27.731560 [I 2022-10-11 17:08:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 1240/2304 [I 2022-10-11 17:08:00] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:15:08 [I 2022-10-11 17:08:00] ...trap/engines/engine.py.229: process: 0.86026 | load: 0.00312 [I 2022-10-11 17:08:00] ...trap/engines/engine.py.230: loss: 0.10852 [I 2022-10-11 17:08:00] ...optimizers/trijoint.py.151: total_norm: 21.457676 [I 2022-10-11 17:08:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1250/2304 [I 2022-10-11 17:08:09] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:15:07 [I 2022-10-11 17:08:09] ...trap/engines/engine.py.229: process: 0.87637 | load: 0.00315 [I 2022-10-11 17:08:09] ...trap/engines/engine.py.230: loss: 0.09338 [I 2022-10-11 17:08:09] ...optimizers/trijoint.py.151: total_norm: 16.365396 [I 2022-10-11 17:08:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1260/2304 [I 2022-10-11 17:08:18] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:14:51 [I 2022-10-11 17:08:18] ...trap/engines/engine.py.229: process: 0.83627 | load: 0.00314 [I 2022-10-11 17:08:18] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-11 17:08:18] ...optimizers/trijoint.py.151: total_norm: 7.029048 [I 2022-10-11 17:08:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1270/2304 [I 2022-10-11 17:08:26] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:14:45 [I 2022-10-11 17:08:26] ...trap/engines/engine.py.229: process: 0.85873 | load: 0.00311 [I 2022-10-11 17:08:26] ...trap/engines/engine.py.230: loss: 0.07040 [I 2022-10-11 17:08:26] ...optimizers/trijoint.py.151: total_norm: 18.752563 [I 2022-10-11 17:08:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1280/2304 [I 2022-10-11 17:08:35] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:14:34 [I 2022-10-11 17:08:35] ...trap/engines/engine.py.229: process: 0.86096 | load: 0.00315 [I 2022-10-11 17:08:35] ...trap/engines/engine.py.230: loss: 0.19429 [I 2022-10-11 17:08:35] ...optimizers/trijoint.py.151: total_norm: 26.752522 [I 2022-10-11 17:08:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1290/2304 [I 2022-10-11 17:08:43] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:14:32 [I 2022-10-11 17:08:43] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00313 [I 2022-10-11 17:08:43] ...trap/engines/engine.py.230: loss: 0.18754 [I 2022-10-11 17:08:43] ...optimizers/trijoint.py.151: total_norm: 14.535630 [I 2022-10-11 17:08:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1300/2304 [I 2022-10-11 17:08:52] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:14:22 [I 2022-10-11 17:08:52] ...trap/engines/engine.py.229: process: 0.88096 | load: 0.00315 [I 2022-10-11 17:08:52] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-11 17:08:52] ...optimizers/trijoint.py.151: total_norm: 21.860067 [I 2022-10-11 17:09:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 1310/2304 [I 2022-10-11 17:09:00] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:14:14 [I 2022-10-11 17:09:00] ...trap/engines/engine.py.229: process: 0.85700 | load: 0.00315 [I 2022-10-11 17:09:00] ...trap/engines/engine.py.230: loss: 0.08796 [I 2022-10-11 17:09:00] ...optimizers/trijoint.py.151: total_norm: 15.043601 [I 2022-10-11 17:09:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1320/2304 [I 2022-10-11 17:09:09] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:14:08 [I 2022-10-11 17:09:09] ...trap/engines/engine.py.229: process: 0.86116 | load: 0.00316 [I 2022-10-11 17:09:09] ...trap/engines/engine.py.230: loss: 0.13636 [I 2022-10-11 17:09:09] ...optimizers/trijoint.py.151: total_norm: 18.733227 [I 2022-10-11 17:09:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1330/2304 [I 2022-10-11 17:09:18] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:14:00 [I 2022-10-11 17:09:18] ...trap/engines/engine.py.229: process: 0.86494 | load: 0.00313 [I 2022-10-11 17:09:18] ...trap/engines/engine.py.230: loss: 0.12647 [I 2022-10-11 17:09:18] ...optimizers/trijoint.py.151: total_norm: 15.884205 [I 2022-10-11 17:09:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1340/2304 [I 2022-10-11 17:09:26] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:46 [I 2022-10-11 17:09:26] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00313 [I 2022-10-11 17:09:26] ...trap/engines/engine.py.230: loss: 0.13707 [I 2022-10-11 17:09:26] ...optimizers/trijoint.py.151: total_norm: 36.824951 [I 2022-10-11 17:09:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1350/2304 [I 2022-10-11 17:09:35] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:34 [I 2022-10-11 17:09:35] ...trap/engines/engine.py.229: process: 0.85608 | load: 0.00309 [I 2022-10-11 17:09:35] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-11 17:09:35] ...optimizers/trijoint.py.151: total_norm: 15.148798 [I 2022-10-11 17:09:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1360/2304 [I 2022-10-11 17:09:43] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:13:26 [I 2022-10-11 17:09:43] ...trap/engines/engine.py.229: process: 0.85871 | load: 0.00311 [I 2022-10-11 17:09:43] ...trap/engines/engine.py.230: loss: 0.23815 [I 2022-10-11 17:09:43] ...optimizers/trijoint.py.151: total_norm: 68.049309 [I 2022-10-11 17:09:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1370/2304 [I 2022-10-11 17:09:52] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:13:06 [I 2022-10-11 17:09:52] ...trap/engines/engine.py.229: process: 0.84331 | load: 0.00294 [I 2022-10-11 17:09:52] ...trap/engines/engine.py.230: loss: 0.10467 [I 2022-10-11 17:09:52] ...optimizers/trijoint.py.151: total_norm: 8.966731 [I 2022-10-11 17:10:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 1380/2304 [I 2022-10-11 17:10:00] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:12:59 [I 2022-10-11 17:10:00] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00301 [I 2022-10-11 17:10:00] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-11 17:10:00] ...optimizers/trijoint.py.151: total_norm: 22.752407 [I 2022-10-11 17:10:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1390/2304 [I 2022-10-11 17:10:09] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:13:18 [I 2022-10-11 17:10:09] ...trap/engines/engine.py.229: process: 0.88840 | load: 0.00376 [I 2022-10-11 17:10:09] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-11 17:10:09] ...optimizers/trijoint.py.151: total_norm: 11.781206 [I 2022-10-11 17:10:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1400/2304 [I 2022-10-11 17:10:18] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:13:09 [I 2022-10-11 17:10:18] ...trap/engines/engine.py.229: process: 0.87481 | load: 0.00372 [I 2022-10-11 17:10:18] ...trap/engines/engine.py.230: loss: 0.20803 [I 2022-10-11 17:10:18] ...optimizers/trijoint.py.151: total_norm: 11.352864 [I 2022-10-11 17:10:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1410/2304 [I 2022-10-11 17:10:27] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:12:58 [I 2022-10-11 17:10:27] ...trap/engines/engine.py.229: process: 0.86931 | load: 0.00372 [I 2022-10-11 17:10:27] ...trap/engines/engine.py.230: loss: 0.17111 [I 2022-10-11 17:10:27] ...optimizers/trijoint.py.151: total_norm: 17.351694 [I 2022-10-11 17:10:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1420/2304 [I 2022-10-11 17:10:35] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:57 [I 2022-10-11 17:10:35] ...trap/engines/engine.py.229: process: 0.88705 | load: 0.00376 [I 2022-10-11 17:10:35] ...trap/engines/engine.py.230: loss: 0.06889 [I 2022-10-11 17:10:35] ...optimizers/trijoint.py.151: total_norm: 14.718564 [I 2022-10-11 17:10:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1430/2304 [I 2022-10-11 17:10:44] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:12:47 [I 2022-10-11 17:10:44] ...trap/engines/engine.py.229: process: 0.87693 | load: 0.00373 [I 2022-10-11 17:10:44] ...trap/engines/engine.py.230: loss: 0.10058 [I 2022-10-11 17:10:44] ...optimizers/trijoint.py.151: total_norm: 10.590805 [I 2022-10-11 17:10:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 1440/2304 [I 2022-10-11 17:10:53] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:12:37 [I 2022-10-11 17:10:53] ...trap/engines/engine.py.229: process: 0.88020 | load: 0.00383 [I 2022-10-11 17:10:53] ...trap/engines/engine.py.230: loss: 0.09187 [I 2022-10-11 17:10:53] ...optimizers/trijoint.py.151: total_norm: 19.759279 [I 2022-10-11 17:11:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 1450/2304 [I 2022-10-11 17:11:02] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:12:26 [I 2022-10-11 17:11:02] ...trap/engines/engine.py.229: process: 0.87874 | load: 0.00378 [I 2022-10-11 17:11:02] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-11 17:11:02] ...optimizers/trijoint.py.151: total_norm: 12.477874 [I 2022-10-11 17:11:10] ...trap/engines/engine.py.224: train: epoch 112 | batch 1460/2304 [I 2022-10-11 17:11:10] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:12:20 [I 2022-10-11 17:11:10] ...trap/engines/engine.py.229: process: 0.87800 | load: 0.00375 [I 2022-10-11 17:11:10] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-11 17:11:10] ...optimizers/trijoint.py.151: total_norm: 13.426205 [I 2022-10-11 17:11:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 1470/2304 [I 2022-10-11 17:11:19] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:12:10 [I 2022-10-11 17:11:19] ...trap/engines/engine.py.229: process: 0.88527 | load: 0.00375 [I 2022-10-11 17:11:19] ...trap/engines/engine.py.230: loss: 0.08931 [I 2022-10-11 17:11:19] ...optimizers/trijoint.py.151: total_norm: 20.119366 [I 2022-10-11 17:11:28] ...trap/engines/engine.py.224: train: epoch 112 | batch 1480/2304 [I 2022-10-11 17:11:28] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:44 [I 2022-10-11 17:11:28] ...trap/engines/engine.py.229: process: 0.85179 | load: 0.00296 [I 2022-10-11 17:11:28] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-11 17:11:28] ...optimizers/trijoint.py.151: total_norm: 49.999401 [I 2022-10-11 17:11:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 1490/2304 [I 2022-10-11 17:11:36] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:11:30 [I 2022-10-11 17:11:36] ...trap/engines/engine.py.229: process: 0.84547 | load: 0.00300 [I 2022-10-11 17:11:36] ...trap/engines/engine.py.230: loss: 0.13991 [I 2022-10-11 17:11:36] ...optimizers/trijoint.py.151: total_norm: 35.018803 [I 2022-10-11 17:11:45] ...trap/engines/engine.py.224: train: epoch 112 | batch 1500/2304 [I 2022-10-11 17:11:45] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:11:21 [I 2022-10-11 17:11:45] ...trap/engines/engine.py.229: process: 0.85336 | load: 0.00297 [I 2022-10-11 17:11:45] ...trap/engines/engine.py.230: loss: 0.11476 [I 2022-10-11 17:11:45] ...optimizers/trijoint.py.151: total_norm: 24.243326 [I 2022-10-11 17:11:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 1510/2304 [I 2022-10-11 17:11:53] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:11:12 [I 2022-10-11 17:11:53] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00298 [I 2022-10-11 17:11:53] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-11 17:11:53] ...optimizers/trijoint.py.151: total_norm: 11.939315 [I 2022-10-11 17:12:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 1520/2304 [I 2022-10-11 17:12:02] ...trap/engines/engine.py.228: elapsed: 0:22:11 | left: 0:11:02 [I 2022-10-11 17:12:02] ...trap/engines/engine.py.229: process: 0.84962 | load: 0.00298 [I 2022-10-11 17:12:02] ...trap/engines/engine.py.230: loss: 0.07479 [I 2022-10-11 17:12:02] ...optimizers/trijoint.py.151: total_norm: 8.153560 [I 2022-10-11 17:12:10] ...trap/engines/engine.py.224: train: epoch 112 | batch 1530/2304 [I 2022-10-11 17:12:10] ...trap/engines/engine.py.228: elapsed: 0:22:19 | left: 0:10:55 [I 2022-10-11 17:12:10] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00296 [I 2022-10-11 17:12:10] ...trap/engines/engine.py.230: loss: 0.09347 [I 2022-10-11 17:12:10] ...optimizers/trijoint.py.151: total_norm: 10.967269 [I 2022-10-11 17:12:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 1540/2304 [I 2022-10-11 17:12:19] ...trap/engines/engine.py.228: elapsed: 0:22:27 | left: 0:10:41 [I 2022-10-11 17:12:19] ...trap/engines/engine.py.229: process: 0.83701 | load: 0.00296 [I 2022-10-11 17:12:19] ...trap/engines/engine.py.230: loss: 0.12894 [I 2022-10-11 17:12:19] ...optimizers/trijoint.py.151: total_norm: 11.693881 [I 2022-10-11 17:12:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1550/2304 [I 2022-10-11 17:12:27] ...trap/engines/engine.py.228: elapsed: 0:22:36 | left: 0:10:33 [I 2022-10-11 17:12:27] ...trap/engines/engine.py.229: process: 0.83428 | load: 0.00296 [I 2022-10-11 17:12:27] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-11 17:12:27] ...optimizers/trijoint.py.151: total_norm: 11.975136 [I 2022-10-11 17:12:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 1560/2304 [I 2022-10-11 17:12:36] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:10:24 [I 2022-10-11 17:12:36] ...trap/engines/engine.py.229: process: 0.83342 | load: 0.00296 [I 2022-10-11 17:12:36] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-11 17:12:36] ...optimizers/trijoint.py.151: total_norm: 9.283498 [I 2022-10-11 17:12:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1570/2304 [I 2022-10-11 17:12:44] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:10:16 [I 2022-10-11 17:12:44] ...trap/engines/engine.py.229: process: 0.83404 | load: 0.00303 [I 2022-10-11 17:12:44] ...trap/engines/engine.py.230: loss: 0.14693 [I 2022-10-11 17:12:44] ...optimizers/trijoint.py.151: total_norm: 11.985714 [I 2022-10-11 17:12:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 1580/2304 [I 2022-10-11 17:12:53] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:10 [I 2022-10-11 17:12:53] ...trap/engines/engine.py.229: process: 0.83699 | load: 0.00298 [I 2022-10-11 17:12:53] ...trap/engines/engine.py.230: loss: 0.07552 [I 2022-10-11 17:12:53] ...optimizers/trijoint.py.151: total_norm: 12.509741 [I 2022-10-11 17:13:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1590/2304 [I 2022-10-11 17:13:01] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:09:59 [I 2022-10-11 17:13:01] ...trap/engines/engine.py.229: process: 0.84655 | load: 0.00303 [I 2022-10-11 17:13:01] ...trap/engines/engine.py.230: loss: 0.09543 [I 2022-10-11 17:13:01] ...optimizers/trijoint.py.151: total_norm: 11.843130 [I 2022-10-11 17:13:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1600/2304 [I 2022-10-11 17:13:09] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:54 [I 2022-10-11 17:13:09] ...trap/engines/engine.py.229: process: 0.83075 | load: 0.00301 [I 2022-10-11 17:13:09] ...trap/engines/engine.py.230: loss: 0.12308 [I 2022-10-11 17:13:09] ...optimizers/trijoint.py.151: total_norm: 10.337737 [I 2022-10-11 17:13:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1610/2304 [I 2022-10-11 17:13:18] ...trap/engines/engine.py.228: elapsed: 0:23:26 | left: 0:09:43 [I 2022-10-11 17:13:18] ...trap/engines/engine.py.229: process: 0.82483 | load: 0.00299 [I 2022-10-11 17:13:18] ...trap/engines/engine.py.230: loss: 0.11388 [I 2022-10-11 17:13:18] ...optimizers/trijoint.py.151: total_norm: 13.563104 [I 2022-10-11 17:13:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1620/2304 [I 2022-10-11 17:13:27] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:50 [I 2022-10-11 17:13:27] ...trap/engines/engine.py.229: process: 0.86569 | load: 0.00311 [I 2022-10-11 17:13:27] ...trap/engines/engine.py.230: loss: 0.09238 [I 2022-10-11 17:13:27] ...optimizers/trijoint.py.151: total_norm: 14.470122 [I 2022-10-11 17:13:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1630/2304 [I 2022-10-11 17:13:35] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:40 [I 2022-10-11 17:13:35] ...trap/engines/engine.py.229: process: 0.86314 | load: 0.00313 [I 2022-10-11 17:13:35] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-11 17:13:35] ...optimizers/trijoint.py.151: total_norm: 27.593588 [I 2022-10-11 17:13:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1640/2304 [I 2022-10-11 17:13:44] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:09:23 [I 2022-10-11 17:13:44] ...trap/engines/engine.py.229: process: 0.83506 | load: 0.00323 [I 2022-10-11 17:13:44] ...trap/engines/engine.py.230: loss: 0.09486 [I 2022-10-11 17:13:44] ...optimizers/trijoint.py.151: total_norm: 7.698177 [I 2022-10-11 17:13:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1650/2304 [I 2022-10-11 17:13:52] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:09:18 [I 2022-10-11 17:13:52] ...trap/engines/engine.py.229: process: 0.84717 | load: 0.00312 [I 2022-10-11 17:13:52] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-11 17:13:52] ...optimizers/trijoint.py.151: total_norm: 10.702717 [I 2022-10-11 17:14:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1660/2304 [I 2022-10-11 17:14:01] ...trap/engines/engine.py.228: elapsed: 0:24:10 | left: 0:09:10 [I 2022-10-11 17:14:01] ...trap/engines/engine.py.229: process: 0.85817 | load: 0.00313 [I 2022-10-11 17:14:01] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-11 17:14:01] ...optimizers/trijoint.py.151: total_norm: 21.743498 [I 2022-10-11 17:14:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1670/2304 [I 2022-10-11 17:14:09] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:09:02 [I 2022-10-11 17:14:09] ...trap/engines/engine.py.229: process: 0.84264 | load: 0.00314 [I 2022-10-11 17:14:09] ...trap/engines/engine.py.230: loss: 0.11963 [I 2022-10-11 17:14:09] ...optimizers/trijoint.py.151: total_norm: 8.722469 [I 2022-10-11 17:14:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1680/2304 [I 2022-10-11 17:14:18] ...trap/engines/engine.py.228: elapsed: 0:24:27 | left: 0:08:55 [I 2022-10-11 17:14:18] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00315 [I 2022-10-11 17:14:18] ...trap/engines/engine.py.230: loss: 0.09642 [I 2022-10-11 17:14:18] ...optimizers/trijoint.py.151: total_norm: 12.309871 [I 2022-10-11 17:14:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1690/2304 [I 2022-10-11 17:14:27] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:44 [I 2022-10-11 17:14:27] ...trap/engines/engine.py.229: process: 0.85083 | load: 0.00311 [I 2022-10-11 17:14:27] ...trap/engines/engine.py.230: loss: 0.14800 [I 2022-10-11 17:14:27] ...optimizers/trijoint.py.151: total_norm: 15.358152 [I 2022-10-11 17:14:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1700/2304 [I 2022-10-11 17:14:35] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:38 [I 2022-10-11 17:14:35] ...trap/engines/engine.py.229: process: 0.86353 | load: 0.00314 [I 2022-10-11 17:14:35] ...trap/engines/engine.py.230: loss: 0.12282 [I 2022-10-11 17:14:35] ...optimizers/trijoint.py.151: total_norm: 18.677071 [I 2022-10-11 17:14:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1710/2304 [I 2022-10-11 17:14:44] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:08:29 [I 2022-10-11 17:14:44] ...trap/engines/engine.py.229: process: 0.83473 | load: 0.00312 [I 2022-10-11 17:14:44] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-11 17:14:44] ...optimizers/trijoint.py.151: total_norm: 15.532075 [I 2022-10-11 17:14:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1720/2304 [I 2022-10-11 17:14:52] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:08:17 [I 2022-10-11 17:14:52] ...trap/engines/engine.py.229: process: 0.85146 | load: 0.00315 [I 2022-10-11 17:14:52] ...trap/engines/engine.py.230: loss: 0.10191 [I 2022-10-11 17:14:52] ...optimizers/trijoint.py.151: total_norm: 12.548058 [I 2022-10-11 17:15:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1730/2304 [I 2022-10-11 17:15:01] ...trap/engines/engine.py.228: elapsed: 0:25:10 | left: 0:08:12 [I 2022-10-11 17:15:01] ...trap/engines/engine.py.229: process: 0.86819 | load: 0.00310 [I 2022-10-11 17:15:01] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-11 17:15:01] ...optimizers/trijoint.py.151: total_norm: 36.364014 [I 2022-10-11 17:15:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1740/2304 [I 2022-10-11 17:15:09] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:08:03 [I 2022-10-11 17:15:09] ...trap/engines/engine.py.229: process: 0.85354 | load: 0.00316 [I 2022-10-11 17:15:09] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-11 17:15:09] ...optimizers/trijoint.py.151: total_norm: 39.045345 [I 2022-10-11 17:15:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1750/2304 [I 2022-10-11 17:15:18] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:53 [I 2022-10-11 17:15:18] ...trap/engines/engine.py.229: process: 0.85364 | load: 0.00357 [I 2022-10-11 17:15:18] ...trap/engines/engine.py.230: loss: 0.23929 [I 2022-10-11 17:15:18] ...optimizers/trijoint.py.151: total_norm: 80.179161 [I 2022-10-11 17:15:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1760/2304 [I 2022-10-11 17:15:27] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:43 [I 2022-10-11 17:15:27] ...trap/engines/engine.py.229: process: 0.84199 | load: 0.00314 [I 2022-10-11 17:15:27] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-11 17:15:27] ...optimizers/trijoint.py.151: total_norm: 13.905600 [I 2022-10-11 17:15:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1770/2304 [I 2022-10-11 17:15:35] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:34 [I 2022-10-11 17:15:35] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00314 [I 2022-10-11 17:15:35] ...trap/engines/engine.py.230: loss: 0.13651 [I 2022-10-11 17:15:35] ...optimizers/trijoint.py.151: total_norm: 18.265820 [I 2022-10-11 17:15:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1780/2304 [I 2022-10-11 17:15:44] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:27 [I 2022-10-11 17:15:44] ...trap/engines/engine.py.229: process: 0.85837 | load: 0.00317 [I 2022-10-11 17:15:44] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-11 17:15:44] ...optimizers/trijoint.py.151: total_norm: 7.856717 [I 2022-10-11 17:15:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1790/2304 [I 2022-10-11 17:15:52] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:07:20 [I 2022-10-11 17:15:52] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00313 [I 2022-10-11 17:15:52] ...trap/engines/engine.py.230: loss: 0.10144 [I 2022-10-11 17:15:52] ...optimizers/trijoint.py.151: total_norm: 13.671988 [I 2022-10-11 17:16:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1800/2304 [I 2022-10-11 17:16:01] ...trap/engines/engine.py.228: elapsed: 0:26:09 | left: 0:07:09 [I 2022-10-11 17:16:01] ...trap/engines/engine.py.229: process: 0.86409 | load: 0.00313 [I 2022-10-11 17:16:01] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-11 17:16:01] ...optimizers/trijoint.py.151: total_norm: 20.712584 [I 2022-10-11 17:16:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1810/2304 [I 2022-10-11 17:16:09] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:07:00 [I 2022-10-11 17:16:09] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00311 [I 2022-10-11 17:16:09] ...trap/engines/engine.py.230: loss: 0.14743 [I 2022-10-11 17:16:09] ...optimizers/trijoint.py.151: total_norm: 21.849239 [I 2022-10-11 17:16:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1820/2304 [I 2022-10-11 17:16:18] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:53 [I 2022-10-11 17:16:18] ...trap/engines/engine.py.229: process: 0.85230 | load: 0.00318 [I 2022-10-11 17:16:18] ...trap/engines/engine.py.230: loss: 0.11355 [I 2022-10-11 17:16:18] ...optimizers/trijoint.py.151: total_norm: 13.365003 [I 2022-10-11 17:16:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1830/2304 [I 2022-10-11 17:16:26] ...trap/engines/engine.py.228: elapsed: 0:26:35 | left: 0:06:46 [I 2022-10-11 17:16:26] ...trap/engines/engine.py.229: process: 0.86179 | load: 0.00313 [I 2022-10-11 17:16:26] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-11 17:16:26] ...optimizers/trijoint.py.151: total_norm: 13.441900 [I 2022-10-11 17:16:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1840/2304 [I 2022-10-11 17:16:35] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:37 [I 2022-10-11 17:16:35] ...trap/engines/engine.py.229: process: 0.86648 | load: 0.00314 [I 2022-10-11 17:16:35] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-11 17:16:35] ...optimizers/trijoint.py.151: total_norm: 18.081728 [I 2022-10-11 17:16:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1850/2304 [I 2022-10-11 17:16:44] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:29 [I 2022-10-11 17:16:44] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00319 [I 2022-10-11 17:16:44] ...trap/engines/engine.py.230: loss: 0.07184 [I 2022-10-11 17:16:44] ...optimizers/trijoint.py.151: total_norm: 12.731410 [I 2022-10-11 17:16:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1860/2304 [I 2022-10-11 17:16:52] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:21 [I 2022-10-11 17:16:52] ...trap/engines/engine.py.229: process: 0.85669 | load: 0.00312 [I 2022-10-11 17:16:52] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-11 17:16:52] ...optimizers/trijoint.py.151: total_norm: 10.728391 [I 2022-10-11 17:17:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1870/2304 [I 2022-10-11 17:17:01] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:06:12 [I 2022-10-11 17:17:01] ...trap/engines/engine.py.229: process: 0.85809 | load: 0.00312 [I 2022-10-11 17:17:01] ...trap/engines/engine.py.230: loss: 0.10436 [I 2022-10-11 17:17:01] ...optimizers/trijoint.py.151: total_norm: 10.267911 [I 2022-10-11 17:17:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1880/2304 [I 2022-10-11 17:17:09] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:06:02 [I 2022-10-11 17:17:09] ...trap/engines/engine.py.229: process: 0.84468 | load: 0.00313 [I 2022-10-11 17:17:09] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-11 17:17:09] ...optimizers/trijoint.py.151: total_norm: 24.715973 [I 2022-10-11 17:17:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1890/2304 [I 2022-10-11 17:17:18] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:53 [I 2022-10-11 17:17:18] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00314 [I 2022-10-11 17:17:18] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-11 17:17:18] ...optimizers/trijoint.py.151: total_norm: 12.019970 [I 2022-10-11 17:17:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1900/2304 [I 2022-10-11 17:17:26] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:42 [I 2022-10-11 17:17:26] ...trap/engines/engine.py.229: process: 0.85480 | load: 0.00298 [I 2022-10-11 17:17:26] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-11 17:17:26] ...optimizers/trijoint.py.151: total_norm: 20.520720 [I 2022-10-11 17:17:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1910/2304 [I 2022-10-11 17:17:35] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:35 [I 2022-10-11 17:17:35] ...trap/engines/engine.py.229: process: 0.84154 | load: 0.00319 [I 2022-10-11 17:17:35] ...trap/engines/engine.py.230: loss: 0.07260 [I 2022-10-11 17:17:35] ...optimizers/trijoint.py.151: total_norm: 9.072610 [I 2022-10-11 17:17:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1920/2304 [I 2022-10-11 17:17:43] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:25 [I 2022-10-11 17:17:43] ...trap/engines/engine.py.229: process: 0.85588 | load: 0.00295 [I 2022-10-11 17:17:43] ...trap/engines/engine.py.230: loss: 0.06707 [I 2022-10-11 17:17:43] ...optimizers/trijoint.py.151: total_norm: 12.577118 [I 2022-10-11 17:17:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1930/2304 [I 2022-10-11 17:17:52] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:17 [I 2022-10-11 17:17:52] ...trap/engines/engine.py.229: process: 0.83986 | load: 0.00298 [I 2022-10-11 17:17:52] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-11 17:17:52] ...optimizers/trijoint.py.151: total_norm: 21.398771 [I 2022-10-11 17:18:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 1940/2304 [I 2022-10-11 17:18:00] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:08 [I 2022-10-11 17:18:00] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00295 [I 2022-10-11 17:18:00] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-11 17:18:00] ...optimizers/trijoint.py.151: total_norm: 13.657006 [I 2022-10-11 17:18:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1950/2304 [I 2022-10-11 17:18:09] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:05:01 [I 2022-10-11 17:18:09] ...trap/engines/engine.py.229: process: 0.84607 | load: 0.00292 [I 2022-10-11 17:18:09] ...trap/engines/engine.py.230: loss: 0.21249 [I 2022-10-11 17:18:09] ...optimizers/trijoint.py.151: total_norm: 16.508257 [I 2022-10-11 17:18:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1960/2304 [I 2022-10-11 17:18:18] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:52 [I 2022-10-11 17:18:18] ...trap/engines/engine.py.229: process: 0.86455 | load: 0.00294 [I 2022-10-11 17:18:18] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-11 17:18:18] ...optimizers/trijoint.py.151: total_norm: 14.567657 [I 2022-10-11 17:18:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1970/2304 [I 2022-10-11 17:18:26] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:44 [I 2022-10-11 17:18:26] ...trap/engines/engine.py.229: process: 0.86502 | load: 0.00294 [I 2022-10-11 17:18:26] ...trap/engines/engine.py.230: loss: 0.08964 [I 2022-10-11 17:18:26] ...optimizers/trijoint.py.151: total_norm: 14.824492 [I 2022-10-11 17:18:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1980/2304 [I 2022-10-11 17:18:35] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:33 [I 2022-10-11 17:18:35] ...trap/engines/engine.py.229: process: 0.84754 | load: 0.00301 [I 2022-10-11 17:18:35] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-11 17:18:35] ...optimizers/trijoint.py.151: total_norm: 28.108652 [I 2022-10-11 17:18:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1990/2304 [I 2022-10-11 17:18:43] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:25 [I 2022-10-11 17:18:43] ...trap/engines/engine.py.229: process: 0.83326 | load: 0.00297 [I 2022-10-11 17:18:43] ...trap/engines/engine.py.230: loss: 0.07527 [I 2022-10-11 17:18:43] ...optimizers/trijoint.py.151: total_norm: 9.121486 [I 2022-10-11 17:18:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 2000/2304 [I 2022-10-11 17:18:51] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:16 [I 2022-10-11 17:18:51] ...trap/engines/engine.py.229: process: 0.84543 | load: 0.00302 [I 2022-10-11 17:18:51] ...trap/engines/engine.py.230: loss: 0.08968 [I 2022-10-11 17:18:51] ...optimizers/trijoint.py.151: total_norm: 10.775762 [I 2022-10-11 17:19:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 2010/2304 [I 2022-10-11 17:19:00] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:08 [I 2022-10-11 17:19:00] ...trap/engines/engine.py.229: process: 0.84614 | load: 0.00308 [I 2022-10-11 17:19:00] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-11 17:19:00] ...optimizers/trijoint.py.151: total_norm: 18.970657 [I 2022-10-11 17:19:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 2020/2304 [I 2022-10-11 17:19:08] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:04:01 [I 2022-10-11 17:19:08] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00306 [I 2022-10-11 17:19:08] ...trap/engines/engine.py.230: loss: 0.18110 [I 2022-10-11 17:19:08] ...optimizers/trijoint.py.151: total_norm: 27.333986 [I 2022-10-11 17:19:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 2030/2304 [I 2022-10-11 17:19:17] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:49 [I 2022-10-11 17:19:17] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00302 [I 2022-10-11 17:19:17] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-11 17:19:17] ...optimizers/trijoint.py.151: total_norm: 10.509092 [I 2022-10-11 17:19:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 2040/2304 [I 2022-10-11 17:19:25] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:43 [I 2022-10-11 17:19:25] ...trap/engines/engine.py.229: process: 0.84672 | load: 0.00302 [I 2022-10-11 17:19:25] ...trap/engines/engine.py.230: loss: 0.16331 [I 2022-10-11 17:19:25] ...optimizers/trijoint.py.151: total_norm: 28.070580 [I 2022-10-11 17:19:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 2050/2304 [I 2022-10-11 17:19:34] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:36 [I 2022-10-11 17:19:34] ...trap/engines/engine.py.229: process: 0.87999 | load: 0.00301 [I 2022-10-11 17:19:34] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-11 17:19:34] ...optimizers/trijoint.py.151: total_norm: 15.234846 [I 2022-10-11 17:19:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 2060/2304 [I 2022-10-11 17:19:42] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:25 [I 2022-10-11 17:19:42] ...trap/engines/engine.py.229: process: 0.84662 | load: 0.00300 [I 2022-10-11 17:19:42] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-11 17:19:42] ...optimizers/trijoint.py.151: total_norm: 10.333305 [I 2022-10-11 17:19:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 2070/2304 [I 2022-10-11 17:19:51] ...trap/engines/engine.py.228: elapsed: 0:29:59 | left: 0:03:18 [I 2022-10-11 17:19:51] ...trap/engines/engine.py.229: process: 0.83421 | load: 0.00308 [I 2022-10-11 17:19:51] ...trap/engines/engine.py.230: loss: 0.13370 [I 2022-10-11 17:19:51] ...optimizers/trijoint.py.151: total_norm: 16.468214 [I 2022-10-11 17:19:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 2080/2304 [I 2022-10-11 17:19:59] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:08 [I 2022-10-11 17:19:59] ...trap/engines/engine.py.229: process: 0.84371 | load: 0.00297 [I 2022-10-11 17:19:59] ...trap/engines/engine.py.230: loss: 0.19670 [I 2022-10-11 17:19:59] ...optimizers/trijoint.py.151: total_norm: 50.827782 [I 2022-10-11 17:20:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 2090/2304 [I 2022-10-11 17:20:08] ...trap/engines/engine.py.228: elapsed: 0:30:16 | left: 0:03:01 [I 2022-10-11 17:20:08] ...trap/engines/engine.py.229: process: 0.84517 | load: 0.00299 [I 2022-10-11 17:20:08] ...trap/engines/engine.py.230: loss: 0.09999 [I 2022-10-11 17:20:08] ...optimizers/trijoint.py.151: total_norm: 14.235312 [I 2022-10-11 17:20:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 2100/2304 [I 2022-10-11 17:20:16] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:52 [I 2022-10-11 17:20:16] ...trap/engines/engine.py.229: process: 0.84977 | load: 0.00296 [I 2022-10-11 17:20:16] ...trap/engines/engine.py.230: loss: 0.15138 [I 2022-10-11 17:20:16] ...optimizers/trijoint.py.151: total_norm: 29.229311 [I 2022-10-11 17:20:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 2110/2304 [I 2022-10-11 17:20:25] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:44 [I 2022-10-11 17:20:25] ...trap/engines/engine.py.229: process: 0.85472 | load: 0.00296 [I 2022-10-11 17:20:25] ...trap/engines/engine.py.230: loss: 0.08051 [I 2022-10-11 17:20:25] ...optimizers/trijoint.py.151: total_norm: 12.504850 [I 2022-10-11 17:20:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 2120/2304 [I 2022-10-11 17:20:33] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:36 [I 2022-10-11 17:20:33] ...trap/engines/engine.py.229: process: 0.85682 | load: 0.00300 [I 2022-10-11 17:20:33] ...trap/engines/engine.py.230: loss: 0.07497 [I 2022-10-11 17:20:33] ...optimizers/trijoint.py.151: total_norm: 12.548121 [I 2022-10-11 17:20:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 2130/2304 [I 2022-10-11 17:20:42] ...trap/engines/engine.py.228: elapsed: 0:30:50 | left: 0:02:28 [I 2022-10-11 17:20:42] ...trap/engines/engine.py.229: process: 0.85499 | load: 0.00296 [I 2022-10-11 17:20:42] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-11 17:20:42] ...optimizers/trijoint.py.151: total_norm: 15.022064 [I 2022-10-11 17:20:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 2140/2304 [I 2022-10-11 17:20:50] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:20 [I 2022-10-11 17:20:50] ...trap/engines/engine.py.229: process: 0.86182 | load: 0.00298 [I 2022-10-11 17:20:50] ...trap/engines/engine.py.230: loss: 0.09314 [I 2022-10-11 17:20:50] ...optimizers/trijoint.py.151: total_norm: 10.653433 [I 2022-10-11 17:20:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 2150/2304 [I 2022-10-11 17:20:59] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:02:14 [I 2022-10-11 17:20:59] ...trap/engines/engine.py.229: process: 0.88711 | load: 0.00383 [I 2022-10-11 17:20:59] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-11 17:20:59] ...optimizers/trijoint.py.151: total_norm: 10.227527 [I 2022-10-11 17:21:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 2160/2304 [I 2022-10-11 17:21:08] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:02:05 [I 2022-10-11 17:21:08] ...trap/engines/engine.py.229: process: 0.87072 | load: 0.00387 [I 2022-10-11 17:21:08] ...trap/engines/engine.py.230: loss: 0.15868 [I 2022-10-11 17:21:08] ...optimizers/trijoint.py.151: total_norm: 11.710019 [I 2022-10-11 17:21:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 2170/2304 [I 2022-10-11 17:21:17] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:57 [I 2022-10-11 17:21:17] ...trap/engines/engine.py.229: process: 0.87555 | load: 0.00381 [I 2022-10-11 17:21:17] ...trap/engines/engine.py.230: loss: 0.09850 [I 2022-10-11 17:21:17] ...optimizers/trijoint.py.151: total_norm: 11.625132 [I 2022-10-11 17:21:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 2180/2304 [I 2022-10-11 17:21:25] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:46 [I 2022-10-11 17:21:25] ...trap/engines/engine.py.229: process: 0.83723 | load: 0.00298 [I 2022-10-11 17:21:25] ...trap/engines/engine.py.230: loss: 0.17012 [I 2022-10-11 17:21:25] ...optimizers/trijoint.py.151: total_norm: 22.973764 [I 2022-10-11 17:21:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 2190/2304 [I 2022-10-11 17:21:34] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:36 [I 2022-10-11 17:21:34] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00299 [I 2022-10-11 17:21:34] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-11 17:21:34] ...optimizers/trijoint.py.151: total_norm: 19.212423 [I 2022-10-11 17:21:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 2200/2304 [I 2022-10-11 17:21:42] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:28 [I 2022-10-11 17:21:42] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00300 [I 2022-10-11 17:21:42] ...trap/engines/engine.py.230: loss: 0.09825 [I 2022-10-11 17:21:42] ...optimizers/trijoint.py.151: total_norm: 11.484536 [I 2022-10-11 17:21:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 2210/2304 [I 2022-10-11 17:21:51] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:01:19 [I 2022-10-11 17:21:51] ...trap/engines/engine.py.229: process: 0.85186 | load: 0.00295 [I 2022-10-11 17:21:51] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-11 17:21:51] ...optimizers/trijoint.py.151: total_norm: 14.481231 [I 2022-10-11 17:22:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 2220/2304 [I 2022-10-11 17:22:00] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:11 [I 2022-10-11 17:22:00] ...trap/engines/engine.py.229: process: 0.83905 | load: 0.00294 [I 2022-10-11 17:22:00] ...trap/engines/engine.py.230: loss: 0.08121 [I 2022-10-11 17:22:00] ...optimizers/trijoint.py.151: total_norm: 17.834469 [I 2022-10-11 17:22:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 2230/2304 [I 2022-10-11 17:22:08] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:01:02 [I 2022-10-11 17:22:08] ...trap/engines/engine.py.229: process: 0.85106 | load: 0.00296 [I 2022-10-11 17:22:08] ...trap/engines/engine.py.230: loss: 0.14246 [I 2022-10-11 17:22:08] ...optimizers/trijoint.py.151: total_norm: 15.734163 [I 2022-10-11 17:22:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 2240/2304 [I 2022-10-11 17:22:16] ...trap/engines/engine.py.228: elapsed: 0:32:25 | left: 0:00:54 [I 2022-10-11 17:22:16] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00297 [I 2022-10-11 17:22:16] ...trap/engines/engine.py.230: loss: 0.15269 [I 2022-10-11 17:22:16] ...optimizers/trijoint.py.151: total_norm: 53.304523 [I 2022-10-11 17:22:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 2250/2304 [I 2022-10-11 17:22:25] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:45 [I 2022-10-11 17:22:25] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00297 [I 2022-10-11 17:22:25] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-11 17:22:25] ...optimizers/trijoint.py.151: total_norm: 18.549412 [I 2022-10-11 17:22:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 2260/2304 [I 2022-10-11 17:22:33] ...trap/engines/engine.py.228: elapsed: 0:32:42 | left: 0:00:37 [I 2022-10-11 17:22:33] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00297 [I 2022-10-11 17:22:33] ...trap/engines/engine.py.230: loss: 0.11072 [I 2022-10-11 17:22:33] ...optimizers/trijoint.py.151: total_norm: 11.619306 [I 2022-10-11 17:22:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 2270/2304 [I 2022-10-11 17:22:42] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:28 [I 2022-10-11 17:22:42] ...trap/engines/engine.py.229: process: 0.83396 | load: 0.00297 [I 2022-10-11 17:22:42] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-11 17:22:42] ...optimizers/trijoint.py.151: total_norm: 17.609043 [I 2022-10-11 17:22:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 2280/2304 [I 2022-10-11 17:22:50] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:20 [I 2022-10-11 17:22:50] ...trap/engines/engine.py.229: process: 0.83870 | load: 0.00301 [I 2022-10-11 17:22:50] ...trap/engines/engine.py.230: loss: 0.09857 [I 2022-10-11 17:22:50] ...optimizers/trijoint.py.151: total_norm: 15.393070 [I 2022-10-11 17:22:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 2290/2304 [I 2022-10-11 17:22:59] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:11 [I 2022-10-11 17:22:59] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00300 [I 2022-10-11 17:22:59] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-11 17:22:59] ...optimizers/trijoint.py.151: total_norm: 10.276098 [I 2022-10-11 17:23:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 2300/2304 [I 2022-10-11 17:23:07] ...trap/engines/engine.py.228: elapsed: 0:33:16 | left: 0:00:03 [I 2022-10-11 17:23:07] ...trap/engines/engine.py.229: process: 0.83868 | load: 0.00017 [I 2022-10-11 17:23:07] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-11 17:23:07] ...optimizers/trijoint.py.151: total_norm: 19.977564 [I 2022-10-11 17:23:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 2304/2304 [I 2022-10-11 17:23:11] ...trap/engines/engine.py.228: elapsed: 0:33:19 | left: 0:00:00 [I 2022-10-11 17:23:11] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00016 [I 2022-10-11 17:23:11] ...trap/engines/engine.py.230: loss: 0.09243 [I 2022-10-11 17:23:11] ...optimizers/trijoint.py.151: total_norm: 10.217710 [S 2022-10-11 17:23:12] ...trap/engines/engine.py.129: train_epoch.epoch: 112 [S 2022-10-11 17:23:12] ...trap/engines/engine.py.129: train_epoch.loss: 0.1195596445505376 [I 2022-10-11 17:23:12] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 17:23:14] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 112 [I 2022-10-11 17:23:14] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 17:23:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 17:23:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 17:23:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 17:23:20] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 17:23:20] ...trap/engines/engine.py.321: eval: epoch 112 | batch 0/510 [I 2022-10-11 17:23:20] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:26 [I 2022-10-11 17:23:20] ...trap/engines/engine.py.326: process: 6.28761 | load: 5.41143 [I 2022-10-11 17:23:25] ...trap/engines/engine.py.321: eval: epoch 112 | batch 10/510 [I 2022-10-11 17:23:25] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:06 [I 2022-10-11 17:23:25] ...trap/engines/engine.py.326: process: 0.41991 | load: 0.00017 [I 2022-10-11 17:23:29] ...trap/engines/engine.py.321: eval: epoch 112 | batch 20/510 [I 2022-10-11 17:23:29] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:23 [I 2022-10-11 17:23:29] ...trap/engines/engine.py.326: process: 0.49715 | load: 0.00017 [I 2022-10-11 17:23:35] ...trap/engines/engine.py.321: eval: epoch 112 | batch 30/510 [I 2022-10-11 17:23:35] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:04:22 [I 2022-10-11 17:23:35] ...trap/engines/engine.py.326: process: 0.50284 | load: 0.00017 [I 2022-10-11 17:23:41] ...trap/engines/engine.py.321: eval: epoch 112 | batch 40/510 [I 2022-10-11 17:23:41] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:04:29 [I 2022-10-11 17:23:41] ...trap/engines/engine.py.326: process: 0.50523 | load: 0.00018 [I 2022-10-11 17:23:46] ...trap/engines/engine.py.321: eval: epoch 112 | batch 50/510 [I 2022-10-11 17:23:46] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:03:54 [I 2022-10-11 17:23:46] ...trap/engines/engine.py.326: process: 0.51745 | load: 0.00024 [I 2022-10-11 17:23:51] ...trap/engines/engine.py.321: eval: epoch 112 | batch 60/510 [I 2022-10-11 17:23:51] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:03:44 [I 2022-10-11 17:23:51] ...trap/engines/engine.py.326: process: 0.55576 | load: 0.00018 [I 2022-10-11 17:23:55] ...trap/engines/engine.py.321: eval: epoch 112 | batch 70/510 [I 2022-10-11 17:23:55] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:03:30 [I 2022-10-11 17:23:55] ...trap/engines/engine.py.326: process: 0.46128 | load: 0.00020 [I 2022-10-11 17:24:00] ...trap/engines/engine.py.321: eval: epoch 112 | batch 80/510 [I 2022-10-11 17:24:00] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:08 [I 2022-10-11 17:24:00] ...trap/engines/engine.py.326: process: 0.45161 | load: 0.00021 [I 2022-10-11 17:24:04] ...trap/engines/engine.py.321: eval: epoch 112 | batch 90/510 [I 2022-10-11 17:24:04] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:03:11 [I 2022-10-11 17:24:04] ...trap/engines/engine.py.326: process: 0.46458 | load: 0.00019 [I 2022-10-11 17:24:09] ...trap/engines/engine.py.321: eval: epoch 112 | batch 100/510 [I 2022-10-11 17:24:09] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:53 [I 2022-10-11 17:24:09] ...trap/engines/engine.py.326: process: 0.31081 | load: 0.00019 [I 2022-10-11 17:24:12] ...trap/engines/engine.py.321: eval: epoch 112 | batch 110/510 [I 2022-10-11 17:24:12] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:11 [I 2022-10-11 17:24:12] ...trap/engines/engine.py.326: process: 0.31046 | load: 0.00020 [I 2022-10-11 17:24:15] ...trap/engines/engine.py.321: eval: epoch 112 | batch 120/510 [I 2022-10-11 17:24:15] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:02:05 [I 2022-10-11 17:24:15] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00021 [I 2022-10-11 17:24:18] ...trap/engines/engine.py.321: eval: epoch 112 | batch 130/510 [I 2022-10-11 17:24:18] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:02:00 [I 2022-10-11 17:24:18] ...trap/engines/engine.py.326: process: 0.30411 | load: 0.00020 [I 2022-10-11 17:24:22] ...trap/engines/engine.py.321: eval: epoch 112 | batch 140/510 [I 2022-10-11 17:24:22] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:58 [I 2022-10-11 17:24:22] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00019 [I 2022-10-11 17:24:25] ...trap/engines/engine.py.321: eval: epoch 112 | batch 150/510 [I 2022-10-11 17:24:25] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:55 [I 2022-10-11 17:24:25] ...trap/engines/engine.py.326: process: 0.31672 | load: 0.00020 [I 2022-10-11 17:24:28] ...trap/engines/engine.py.321: eval: epoch 112 | batch 160/510 [I 2022-10-11 17:24:28] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:52 [I 2022-10-11 17:24:28] ...trap/engines/engine.py.326: process: 0.32240 | load: 0.00020 [I 2022-10-11 17:24:31] ...trap/engines/engine.py.321: eval: epoch 112 | batch 170/510 [I 2022-10-11 17:24:31] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:48 [I 2022-10-11 17:24:31] ...trap/engines/engine.py.326: process: 0.31489 | load: 0.00020 [I 2022-10-11 17:24:34] ...trap/engines/engine.py.321: eval: epoch 112 | batch 180/510 [I 2022-10-11 17:24:34] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:45 [I 2022-10-11 17:24:34] ...trap/engines/engine.py.326: process: 0.31795 | load: 0.00020 [I 2022-10-11 17:24:38] ...trap/engines/engine.py.321: eval: epoch 112 | batch 190/510 [I 2022-10-11 17:24:38] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:43 [I 2022-10-11 17:24:38] ...trap/engines/engine.py.326: process: 0.32419 | load: 0.00021 [I 2022-10-11 17:24:41] ...trap/engines/engine.py.321: eval: epoch 112 | batch 200/510 [I 2022-10-11 17:24:41] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:39 [I 2022-10-11 17:24:41] ...trap/engines/engine.py.326: process: 0.32340 | load: 0.00020 [I 2022-10-11 17:24:44] ...trap/engines/engine.py.321: eval: epoch 112 | batch 210/510 [I 2022-10-11 17:24:44] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:36 [I 2022-10-11 17:24:44] ...trap/engines/engine.py.326: process: 0.32111 | load: 0.00021 [I 2022-10-11 17:24:47] ...trap/engines/engine.py.321: eval: epoch 112 | batch 220/510 [I 2022-10-11 17:24:47] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:33 [I 2022-10-11 17:24:47] ...trap/engines/engine.py.326: process: 0.32404 | load: 0.00020 [I 2022-10-11 17:24:50] ...trap/engines/engine.py.321: eval: epoch 112 | batch 230/510 [I 2022-10-11 17:24:50] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:29 [I 2022-10-11 17:24:50] ...trap/engines/engine.py.326: process: 0.32444 | load: 0.00024 [I 2022-10-11 17:24:54] ...trap/engines/engine.py.321: eval: epoch 112 | batch 240/510 [I 2022-10-11 17:24:54] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:27 [I 2022-10-11 17:24:54] ...trap/engines/engine.py.326: process: 0.32437 | load: 0.00020 [I 2022-10-11 17:24:57] ...trap/engines/engine.py.321: eval: epoch 112 | batch 250/510 [I 2022-10-11 17:24:57] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:23 [I 2022-10-11 17:24:57] ...trap/engines/engine.py.326: process: 0.32044 | load: 0.00020 [I 2022-10-11 17:25:00] ...trap/engines/engine.py.321: eval: epoch 112 | batch 260/510 [I 2022-10-11 17:25:00] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:21 [I 2022-10-11 17:25:00] ...trap/engines/engine.py.326: process: 0.33373 | load: 0.00021 [I 2022-10-11 17:25:03] ...trap/engines/engine.py.321: eval: epoch 112 | batch 270/510 [I 2022-10-11 17:25:03] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:16 [I 2022-10-11 17:25:03] ...trap/engines/engine.py.326: process: 0.32192 | load: 0.00023 [I 2022-10-11 17:25:07] ...trap/engines/engine.py.321: eval: epoch 112 | batch 280/510 [I 2022-10-11 17:25:07] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:13 [I 2022-10-11 17:25:07] ...trap/engines/engine.py.326: process: 0.31709 | load: 0.00022 [I 2022-10-11 17:25:10] ...trap/engines/engine.py.321: eval: epoch 112 | batch 290/510 [I 2022-10-11 17:25:10] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:10 [I 2022-10-11 17:25:10] ...trap/engines/engine.py.326: process: 0.32592 | load: 0.00022 [I 2022-10-11 17:25:13] ...trap/engines/engine.py.321: eval: epoch 112 | batch 300/510 [I 2022-10-11 17:25:13] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:07 [I 2022-10-11 17:25:13] ...trap/engines/engine.py.326: process: 0.32535 | load: 0.00022 [I 2022-10-11 17:25:16] ...trap/engines/engine.py.321: eval: epoch 112 | batch 310/510 [I 2022-10-11 17:25:16] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:04 [I 2022-10-11 17:25:16] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00020 [I 2022-10-11 17:25:19] ...trap/engines/engine.py.321: eval: epoch 112 | batch 320/510 [I 2022-10-11 17:25:19] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:00 [I 2022-10-11 17:25:19] ...trap/engines/engine.py.326: process: 0.32604 | load: 0.00020 [I 2022-10-11 17:25:23] ...trap/engines/engine.py.321: eval: epoch 112 | batch 330/510 [I 2022-10-11 17:25:23] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:57 [I 2022-10-11 17:25:23] ...trap/engines/engine.py.326: process: 0.31768 | load: 0.00022 [I 2022-10-11 17:25:26] ...trap/engines/engine.py.321: eval: epoch 112 | batch 340/510 [I 2022-10-11 17:25:26] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:54 [I 2022-10-11 17:25:26] ...trap/engines/engine.py.326: process: 0.32650 | load: 0.00020 [I 2022-10-11 17:25:29] ...trap/engines/engine.py.321: eval: epoch 112 | batch 350/510 [I 2022-10-11 17:25:29] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:51 [I 2022-10-11 17:25:29] ...trap/engines/engine.py.326: process: 0.32158 | load: 0.00021 [I 2022-10-11 17:25:32] ...trap/engines/engine.py.321: eval: epoch 112 | batch 360/510 [I 2022-10-11 17:25:32] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:48 [I 2022-10-11 17:25:32] ...trap/engines/engine.py.326: process: 0.32502 | load: 0.00022 [I 2022-10-11 17:25:36] ...trap/engines/engine.py.321: eval: epoch 112 | batch 370/510 [I 2022-10-11 17:25:36] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:45 [I 2022-10-11 17:25:36] ...trap/engines/engine.py.326: process: 0.31854 | load: 0.00021 [I 2022-10-11 17:25:39] ...trap/engines/engine.py.321: eval: epoch 112 | batch 380/510 [I 2022-10-11 17:25:39] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:41 [I 2022-10-11 17:25:39] ...trap/engines/engine.py.326: process: 0.32395 | load: 0.00021 [I 2022-10-11 17:25:42] ...trap/engines/engine.py.321: eval: epoch 112 | batch 390/510 [I 2022-10-11 17:25:42] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:38 [I 2022-10-11 17:25:42] ...trap/engines/engine.py.326: process: 0.32197 | load: 0.00020 [I 2022-10-11 17:25:45] ...trap/engines/engine.py.321: eval: epoch 112 | batch 400/510 [I 2022-10-11 17:25:45] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:35 [I 2022-10-11 17:25:45] ...trap/engines/engine.py.326: process: 0.31761 | load: 0.00021 [I 2022-10-11 17:25:48] ...trap/engines/engine.py.321: eval: epoch 112 | batch 410/510 [I 2022-10-11 17:25:48] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:32 [I 2022-10-11 17:25:48] ...trap/engines/engine.py.326: process: 0.32139 | load: 0.00021 [I 2022-10-11 17:25:52] ...trap/engines/engine.py.321: eval: epoch 112 | batch 420/510 [I 2022-10-11 17:25:52] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:29 [I 2022-10-11 17:25:52] ...trap/engines/engine.py.326: process: 0.31998 | load: 0.00021 [I 2022-10-11 17:25:55] ...trap/engines/engine.py.321: eval: epoch 112 | batch 430/510 [I 2022-10-11 17:25:55] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:25 [I 2022-10-11 17:25:55] ...trap/engines/engine.py.326: process: 0.32445 | load: 0.00021 [I 2022-10-11 17:25:58] ...trap/engines/engine.py.321: eval: epoch 112 | batch 440/510 [I 2022-10-11 17:25:58] ...trap/engines/engine.py.325: elapsed: 0:02:44 | left: 0:00:22 [I 2022-10-11 17:25:58] ...trap/engines/engine.py.326: process: 0.32012 | load: 0.00020 [I 2022-10-11 17:26:01] ...trap/engines/engine.py.321: eval: epoch 112 | batch 450/510 [I 2022-10-11 17:26:01] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:19 [I 2022-10-11 17:26:01] ...trap/engines/engine.py.326: process: 0.32070 | load: 0.00021 [I 2022-10-11 17:26:05] ...trap/engines/engine.py.321: eval: epoch 112 | batch 460/510 [I 2022-10-11 17:26:05] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:15 [I 2022-10-11 17:26:05] ...trap/engines/engine.py.326: process: 0.32363 | load: 0.00020 [I 2022-10-11 17:26:08] ...trap/engines/engine.py.321: eval: epoch 112 | batch 470/510 [I 2022-10-11 17:26:08] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-11 17:26:08] ...trap/engines/engine.py.326: process: 0.32842 | load: 0.00021 [I 2022-10-11 17:26:11] ...trap/engines/engine.py.321: eval: epoch 112 | batch 480/510 [I 2022-10-11 17:26:11] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:09 [I 2022-10-11 17:26:11] ...trap/engines/engine.py.326: process: 0.32371 | load: 0.00021 [I 2022-10-11 17:26:14] ...trap/engines/engine.py.321: eval: epoch 112 | batch 490/510 [I 2022-10-11 17:26:14] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:06 [I 2022-10-11 17:26:14] ...trap/engines/engine.py.326: process: 0.32314 | load: 0.00021 [I 2022-10-11 17:26:18] ...trap/engines/engine.py.321: eval: epoch 112 | batch 500/510 [I 2022-10-11 17:26:18] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:03 [I 2022-10-11 17:26:18] ...trap/engines/engine.py.326: process: 0.31762 | load: 0.00024 [I 2022-10-11 17:26:21] ...trap/engines/engine.py.321: eval: epoch 112 | batch 510/510 [I 2022-10-11 17:26:21] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:00 [I 2022-10-11 17:26:21] ...trap/engines/engine.py.326: process: 0.30849 | load: 0.00016 [S 2022-10-11 17:26:21] ...trap/engines/engine.py.132: eval_epoch.epoch: 112 [I 2022-10-11 17:26:21] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.748100 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910500 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933800 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.740100 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908700 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935100 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013080 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008868 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008875 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016416 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009177 [S 2022-10-11 17:26:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008904 [I 2022-10-11 17:27:05] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 17:27:05] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 17:27:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 17:27:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 17:27:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 17:27:10] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 17:27:14] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 17:27:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 17:27:18] ...trap/engines/engine.py.163: Training model on trainset for epoch 113 [I 2022-10-11 17:27:18] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 17:27:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 0/2304 [I 2022-10-11 17:27:21] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:09:38 [I 2022-10-11 17:27:21] ...trap/engines/engine.py.229: process: 3.37589 | load: 2.40335 [I 2022-10-11 17:27:21] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-11 17:27:21] ...optimizers/trijoint.py.151: total_norm: 11.185107 [I 2022-10-11 17:27:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 10/2304 [I 2022-10-11 17:27:30] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:44:10 [I 2022-10-11 17:27:30] ...trap/engines/engine.py.229: process: 0.90355 | load: 0.00805 [I 2022-10-11 17:27:30] ...trap/engines/engine.py.230: loss: 0.10025 [I 2022-10-11 17:27:30] ...optimizers/trijoint.py.151: total_norm: 15.849896 [I 2022-10-11 17:27:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 20/2304 [I 2022-10-11 17:27:39] ...trap/engines/engine.py.228: elapsed: 0:00:21 | left: 0:34:49 [I 2022-10-11 17:27:39] ...trap/engines/engine.py.229: process: 0.88560 | load: 0.00468 [I 2022-10-11 17:27:39] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-11 17:27:39] ...optimizers/trijoint.py.151: total_norm: 11.542364 [I 2022-10-11 17:27:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 30/2304 [I 2022-10-11 17:27:47] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:33:11 [I 2022-10-11 17:27:47] ...trap/engines/engine.py.229: process: 0.85778 | load: 0.00490 [I 2022-10-11 17:27:47] ...trap/engines/engine.py.230: loss: 0.13561 [I 2022-10-11 17:27:47] ...optimizers/trijoint.py.151: total_norm: 12.995688 [I 2022-10-11 17:27:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 40/2304 [I 2022-10-11 17:27:56] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:32:19 [I 2022-10-11 17:27:56] ...trap/engines/engine.py.229: process: 0.83930 | load: 0.00421 [I 2022-10-11 17:27:56] ...trap/engines/engine.py.230: loss: 0.08728 [I 2022-10-11 17:27:56] ...optimizers/trijoint.py.151: total_norm: 11.496416 [I 2022-10-11 17:28:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 50/2304 [I 2022-10-11 17:28:05] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:32:18 [I 2022-10-11 17:28:05] ...trap/engines/engine.py.229: process: 0.86453 | load: 0.00415 [I 2022-10-11 17:28:05] ...trap/engines/engine.py.230: loss: 0.08923 [I 2022-10-11 17:28:05] ...optimizers/trijoint.py.151: total_norm: 10.625254 [I 2022-10-11 17:28:13] ...trap/engines/engine.py.224: train: epoch 113 | batch 60/2304 [I 2022-10-11 17:28:13] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:32:05 [I 2022-10-11 17:28:13] ...trap/engines/engine.py.229: process: 0.85000 | load: 0.00389 [I 2022-10-11 17:28:13] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-11 17:28:13] ...optimizers/trijoint.py.151: total_norm: 23.908455 [I 2022-10-11 17:28:22] ...trap/engines/engine.py.224: train: epoch 113 | batch 70/2304 [I 2022-10-11 17:28:22] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:31:49 [I 2022-10-11 17:28:22] ...trap/engines/engine.py.229: process: 0.86331 | load: 0.00437 [I 2022-10-11 17:28:22] ...trap/engines/engine.py.230: loss: 0.06550 [I 2022-10-11 17:28:22] ...optimizers/trijoint.py.151: total_norm: 17.942841 [I 2022-10-11 17:28:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 80/2304 [I 2022-10-11 17:28:30] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:31 [I 2022-10-11 17:28:30] ...trap/engines/engine.py.229: process: 0.84492 | load: 0.00366 [I 2022-10-11 17:28:30] ...trap/engines/engine.py.230: loss: 0.10942 [I 2022-10-11 17:28:30] ...optimizers/trijoint.py.151: total_norm: 11.305828 [I 2022-10-11 17:28:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 90/2304 [I 2022-10-11 17:28:39] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:31:44 [I 2022-10-11 17:28:39] ...trap/engines/engine.py.229: process: 0.84961 | load: 0.00336 [I 2022-10-11 17:28:39] ...trap/engines/engine.py.230: loss: 0.12330 [I 2022-10-11 17:28:39] ...optimizers/trijoint.py.151: total_norm: 8.768317 [I 2022-10-11 17:28:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 100/2304 [I 2022-10-11 17:28:47] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:35 [I 2022-10-11 17:28:47] ...trap/engines/engine.py.229: process: 0.84846 | load: 0.00338 [I 2022-10-11 17:28:47] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-11 17:28:47] ...optimizers/trijoint.py.151: total_norm: 19.982794 [I 2022-10-11 17:28:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 110/2304 [I 2022-10-11 17:28:56] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:32:19 [I 2022-10-11 17:28:56] ...trap/engines/engine.py.229: process: 0.88868 | load: 0.00430 [I 2022-10-11 17:28:56] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-11 17:28:56] ...optimizers/trijoint.py.151: total_norm: 11.594236 [I 2022-10-11 17:29:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 120/2304 [I 2022-10-11 17:29:05] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:32:13 [I 2022-10-11 17:29:05] ...trap/engines/engine.py.229: process: 0.87780 | load: 0.00430 [I 2022-10-11 17:29:05] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-11 17:29:05] ...optimizers/trijoint.py.151: total_norm: 13.032395 [I 2022-10-11 17:29:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 130/2304 [I 2022-10-11 17:29:14] ...trap/engines/engine.py.228: elapsed: 0:01:56 | left: 0:32:03 [I 2022-10-11 17:29:14] ...trap/engines/engine.py.229: process: 0.88500 | load: 0.00424 [I 2022-10-11 17:29:14] ...trap/engines/engine.py.230: loss: 0.21441 [I 2022-10-11 17:29:14] ...optimizers/trijoint.py.151: total_norm: 22.109665 [I 2022-10-11 17:29:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 140/2304 [I 2022-10-11 17:29:23] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:32:05 [I 2022-10-11 17:29:23] ...trap/engines/engine.py.229: process: 0.88514 | load: 0.00429 [I 2022-10-11 17:29:23] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-11 17:29:23] ...optimizers/trijoint.py.151: total_norm: 14.395717 [I 2022-10-11 17:29:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 150/2304 [I 2022-10-11 17:29:32] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:30:43 [I 2022-10-11 17:29:32] ...trap/engines/engine.py.229: process: 0.85367 | load: 0.00349 [I 2022-10-11 17:29:32] ...trap/engines/engine.py.230: loss: 0.10257 [I 2022-10-11 17:29:32] ...optimizers/trijoint.py.151: total_norm: 12.923224 [I 2022-10-11 17:29:40] ...trap/engines/engine.py.224: train: epoch 113 | batch 160/2304 [I 2022-10-11 17:29:40] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:30:23 [I 2022-10-11 17:29:40] ...trap/engines/engine.py.229: process: 0.85032 | load: 0.00315 [I 2022-10-11 17:29:40] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-11 17:29:40] ...optimizers/trijoint.py.151: total_norm: 10.699972 [I 2022-10-11 17:29:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 170/2304 [I 2022-10-11 17:29:49] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:30:39 [I 2022-10-11 17:29:49] ...trap/engines/engine.py.229: process: 0.86714 | load: 0.00349 [I 2022-10-11 17:29:49] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-11 17:29:49] ...optimizers/trijoint.py.151: total_norm: 14.079927 [I 2022-10-11 17:29:57] ...trap/engines/engine.py.224: train: epoch 113 | batch 180/2304 [I 2022-10-11 17:29:57] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:30:23 [I 2022-10-11 17:29:57] ...trap/engines/engine.py.229: process: 0.84893 | load: 0.00329 [I 2022-10-11 17:29:57] ...trap/engines/engine.py.230: loss: 0.08308 [I 2022-10-11 17:29:57] ...optimizers/trijoint.py.151: total_norm: 21.127628 [I 2022-10-11 17:30:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 190/2304 [I 2022-10-11 17:30:06] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:30:06 [I 2022-10-11 17:30:06] ...trap/engines/engine.py.229: process: 0.84592 | load: 0.00341 [I 2022-10-11 17:30:06] ...trap/engines/engine.py.230: loss: 0.13505 [I 2022-10-11 17:30:06] ...optimizers/trijoint.py.151: total_norm: 26.762861 [I 2022-10-11 17:30:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 200/2304 [I 2022-10-11 17:30:14] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:30:03 [I 2022-10-11 17:30:14] ...trap/engines/engine.py.229: process: 0.89197 | load: 0.00326 [I 2022-10-11 17:30:14] ...trap/engines/engine.py.230: loss: 0.09399 [I 2022-10-11 17:30:14] ...optimizers/trijoint.py.151: total_norm: 19.597965 [I 2022-10-11 17:30:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 210/2304 [I 2022-10-11 17:30:23] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:30:03 [I 2022-10-11 17:30:23] ...trap/engines/engine.py.229: process: 0.85835 | load: 0.00333 [I 2022-10-11 17:30:23] ...trap/engines/engine.py.230: loss: 0.10852 [I 2022-10-11 17:30:23] ...optimizers/trijoint.py.151: total_norm: 25.411541 [I 2022-10-11 17:30:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 220/2304 [I 2022-10-11 17:30:32] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:29:46 [I 2022-10-11 17:30:32] ...trap/engines/engine.py.229: process: 0.87109 | load: 0.00320 [I 2022-10-11 17:30:32] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-11 17:30:32] ...optimizers/trijoint.py.151: total_norm: 17.361872 [I 2022-10-11 17:30:40] ...trap/engines/engine.py.224: train: epoch 113 | batch 230/2304 [I 2022-10-11 17:30:40] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:29:54 [I 2022-10-11 17:30:40] ...trap/engines/engine.py.229: process: 0.86477 | load: 0.00318 [I 2022-10-11 17:30:40] ...trap/engines/engine.py.230: loss: 0.13274 [I 2022-10-11 17:30:40] ...optimizers/trijoint.py.151: total_norm: 26.936161 [I 2022-10-11 17:30:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 240/2304 [I 2022-10-11 17:30:49] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:29:37 [I 2022-10-11 17:30:49] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00330 [I 2022-10-11 17:30:49] ...trap/engines/engine.py.230: loss: 0.18225 [I 2022-10-11 17:30:49] ...optimizers/trijoint.py.151: total_norm: 39.144928 [I 2022-10-11 17:30:57] ...trap/engines/engine.py.224: train: epoch 113 | batch 250/2304 [I 2022-10-11 17:30:58] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:29:15 [I 2022-10-11 17:30:58] ...trap/engines/engine.py.229: process: 0.84299 | load: 0.00314 [I 2022-10-11 17:30:58] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-11 17:30:58] ...optimizers/trijoint.py.151: total_norm: 13.330317 [I 2022-10-11 17:31:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 260/2304 [I 2022-10-11 17:31:06] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:29:27 [I 2022-10-11 17:31:06] ...trap/engines/engine.py.229: process: 0.86384 | load: 0.00314 [I 2022-10-11 17:31:06] ...trap/engines/engine.py.230: loss: 0.08601 [I 2022-10-11 17:31:06] ...optimizers/trijoint.py.151: total_norm: 40.062752 [I 2022-10-11 17:31:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 270/2304 [I 2022-10-11 17:31:15] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:29:14 [I 2022-10-11 17:31:15] ...trap/engines/engine.py.229: process: 0.87991 | load: 0.00333 [I 2022-10-11 17:31:15] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-11 17:31:15] ...optimizers/trijoint.py.151: total_norm: 10.909133 [I 2022-10-11 17:31:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 280/2304 [I 2022-10-11 17:31:23] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:29:00 [I 2022-10-11 17:31:23] ...trap/engines/engine.py.229: process: 0.85148 | load: 0.00324 [I 2022-10-11 17:31:23] ...trap/engines/engine.py.230: loss: 0.26321 [I 2022-10-11 17:31:23] ...optimizers/trijoint.py.151: total_norm: 23.365959 [I 2022-10-11 17:31:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 290/2304 [I 2022-10-11 17:31:32] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:28:31 [I 2022-10-11 17:31:32] ...trap/engines/engine.py.229: process: 0.84401 | load: 0.00317 [I 2022-10-11 17:31:32] ...trap/engines/engine.py.230: loss: 0.09235 [I 2022-10-11 17:31:32] ...optimizers/trijoint.py.151: total_norm: 7.106370 [I 2022-10-11 17:31:40] ...trap/engines/engine.py.224: train: epoch 113 | batch 300/2304 [I 2022-10-11 17:31:40] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:23 [I 2022-10-11 17:31:40] ...trap/engines/engine.py.229: process: 0.84449 | load: 0.00328 [I 2022-10-11 17:31:40] ...trap/engines/engine.py.230: loss: 0.17182 [I 2022-10-11 17:31:40] ...optimizers/trijoint.py.151: total_norm: 14.422502 [I 2022-10-11 17:31:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 310/2304 [I 2022-10-11 17:31:49] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:28:38 [I 2022-10-11 17:31:49] ...trap/engines/engine.py.229: process: 0.85393 | load: 0.00315 [I 2022-10-11 17:31:49] ...trap/engines/engine.py.230: loss: 0.06413 [I 2022-10-11 17:31:49] ...optimizers/trijoint.py.151: total_norm: 11.873930 [I 2022-10-11 17:31:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 320/2304 [I 2022-10-11 17:31:58] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:28:26 [I 2022-10-11 17:31:58] ...trap/engines/engine.py.229: process: 0.85214 | load: 0.00325 [I 2022-10-11 17:31:58] ...trap/engines/engine.py.230: loss: 0.08757 [I 2022-10-11 17:31:58] ...optimizers/trijoint.py.151: total_norm: 12.781384 [I 2022-10-11 17:32:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 330/2304 [I 2022-10-11 17:32:06] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:28:16 [I 2022-10-11 17:32:06] ...trap/engines/engine.py.229: process: 0.86494 | load: 0.00328 [I 2022-10-11 17:32:06] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-11 17:32:06] ...optimizers/trijoint.py.151: total_norm: 15.031659 [I 2022-10-11 17:32:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 340/2304 [I 2022-10-11 17:32:15] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:28:01 [I 2022-10-11 17:32:15] ...trap/engines/engine.py.229: process: 0.84075 | load: 0.00316 [I 2022-10-11 17:32:15] ...trap/engines/engine.py.230: loss: 0.14394 [I 2022-10-11 17:32:15] ...optimizers/trijoint.py.151: total_norm: 40.021080 [I 2022-10-11 17:32:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 350/2304 [I 2022-10-11 17:32:24] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:49 [I 2022-10-11 17:32:24] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00319 [I 2022-10-11 17:32:24] ...trap/engines/engine.py.230: loss: 0.11505 [I 2022-10-11 17:32:24] ...optimizers/trijoint.py.151: total_norm: 15.627377 [I 2022-10-11 17:32:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 360/2304 [I 2022-10-11 17:32:32] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:37 [I 2022-10-11 17:32:32] ...trap/engines/engine.py.229: process: 0.84057 | load: 0.00328 [I 2022-10-11 17:32:32] ...trap/engines/engine.py.230: loss: 0.11422 [I 2022-10-11 17:32:32] ...optimizers/trijoint.py.151: total_norm: 23.698982 [I 2022-10-11 17:32:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 370/2304 [I 2022-10-11 17:32:41] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:26 [I 2022-10-11 17:32:41] ...trap/engines/engine.py.229: process: 0.84229 | load: 0.00315 [I 2022-10-11 17:32:41] ...trap/engines/engine.py.230: loss: 0.16758 [I 2022-10-11 17:32:41] ...optimizers/trijoint.py.151: total_norm: 19.148108 [I 2022-10-11 17:32:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 380/2304 [I 2022-10-11 17:32:49] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:33 [I 2022-10-11 17:32:49] ...trap/engines/engine.py.229: process: 0.86582 | load: 0.00315 [I 2022-10-11 17:32:49] ...trap/engines/engine.py.230: loss: 0.17876 [I 2022-10-11 17:32:49] ...optimizers/trijoint.py.151: total_norm: 16.351669 [I 2022-10-11 17:32:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 390/2304 [I 2022-10-11 17:32:58] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:27 [I 2022-10-11 17:32:58] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00316 [I 2022-10-11 17:32:58] ...trap/engines/engine.py.230: loss: 0.08071 [I 2022-10-11 17:32:58] ...optimizers/trijoint.py.151: total_norm: 15.128877 [I 2022-10-11 17:33:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 400/2304 [I 2022-10-11 17:33:07] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:27:16 [I 2022-10-11 17:33:07] ...trap/engines/engine.py.229: process: 0.84872 | load: 0.00314 [I 2022-10-11 17:33:07] ...trap/engines/engine.py.230: loss: 0.10441 [I 2022-10-11 17:33:07] ...optimizers/trijoint.py.151: total_norm: 12.036152 [I 2022-10-11 17:33:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 410/2304 [I 2022-10-11 17:33:15] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:15 [I 2022-10-11 17:33:15] ...trap/engines/engine.py.229: process: 0.89355 | load: 0.00330 [I 2022-10-11 17:33:15] ...trap/engines/engine.py.230: loss: 0.08325 [I 2022-10-11 17:33:15] ...optimizers/trijoint.py.151: total_norm: 11.505638 [I 2022-10-11 17:33:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 420/2304 [I 2022-10-11 17:33:24] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:27:11 [I 2022-10-11 17:33:24] ...trap/engines/engine.py.229: process: 0.86492 | load: 0.00315 [I 2022-10-11 17:33:24] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-11 17:33:24] ...optimizers/trijoint.py.151: total_norm: 13.022607 [I 2022-10-11 17:33:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 430/2304 [I 2022-10-11 17:33:33] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:27:01 [I 2022-10-11 17:33:33] ...trap/engines/engine.py.229: process: 0.84841 | load: 0.00872 [I 2022-10-11 17:33:33] ...trap/engines/engine.py.230: loss: 0.09853 [I 2022-10-11 17:33:33] ...optimizers/trijoint.py.151: total_norm: 15.630817 [I 2022-10-11 17:33:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 440/2304 [I 2022-10-11 17:33:41] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:30 [I 2022-10-11 17:33:41] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00314 [I 2022-10-11 17:33:41] ...trap/engines/engine.py.230: loss: 0.15576 [I 2022-10-11 17:33:41] ...optimizers/trijoint.py.151: total_norm: 19.693293 [I 2022-10-11 17:33:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 450/2304 [I 2022-10-11 17:33:50] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:26:31 [I 2022-10-11 17:33:50] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00313 [I 2022-10-11 17:33:50] ...trap/engines/engine.py.230: loss: 0.13135 [I 2022-10-11 17:33:50] ...optimizers/trijoint.py.151: total_norm: 14.205329 [I 2022-10-11 17:33:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 460/2304 [I 2022-10-11 17:33:58] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:16 [I 2022-10-11 17:33:58] ...trap/engines/engine.py.229: process: 0.85805 | load: 0.00310 [I 2022-10-11 17:33:58] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-11 17:33:58] ...optimizers/trijoint.py.151: total_norm: 11.131356 [I 2022-10-11 17:34:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 470/2304 [I 2022-10-11 17:34:07] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:09 [I 2022-10-11 17:34:07] ...trap/engines/engine.py.229: process: 0.85018 | load: 0.00310 [I 2022-10-11 17:34:07] ...trap/engines/engine.py.230: loss: 0.12449 [I 2022-10-11 17:34:07] ...optimizers/trijoint.py.151: total_norm: 13.650405 [I 2022-10-11 17:34:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 480/2304 [I 2022-10-11 17:34:15] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:08 [I 2022-10-11 17:34:15] ...trap/engines/engine.py.229: process: 0.86671 | load: 0.00313 [I 2022-10-11 17:34:15] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-11 17:34:15] ...optimizers/trijoint.py.151: total_norm: 20.339504 [I 2022-10-11 17:34:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 490/2304 [I 2022-10-11 17:34:24] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:26:11 [I 2022-10-11 17:34:24] ...trap/engines/engine.py.229: process: 0.85921 | load: 0.00312 [I 2022-10-11 17:34:24] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-11 17:34:24] ...optimizers/trijoint.py.151: total_norm: 27.828028 [I 2022-10-11 17:34:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 500/2304 [I 2022-10-11 17:34:33] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:52 [I 2022-10-11 17:34:33] ...trap/engines/engine.py.229: process: 0.86023 | load: 0.00311 [I 2022-10-11 17:34:33] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-11 17:34:33] ...optimizers/trijoint.py.151: total_norm: 32.183701 [I 2022-10-11 17:34:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 510/2304 [I 2022-10-11 17:34:41] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:43 [I 2022-10-11 17:34:41] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00312 [I 2022-10-11 17:34:41] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-11 17:34:41] ...optimizers/trijoint.py.151: total_norm: 12.104973 [I 2022-10-11 17:34:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 520/2304 [I 2022-10-11 17:34:50] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:41 [I 2022-10-11 17:34:50] ...trap/engines/engine.py.229: process: 0.85933 | load: 0.00313 [I 2022-10-11 17:34:50] ...trap/engines/engine.py.230: loss: 0.16175 [I 2022-10-11 17:34:50] ...optimizers/trijoint.py.151: total_norm: 14.880544 [I 2022-10-11 17:34:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 530/2304 [I 2022-10-11 17:34:58] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:15 [I 2022-10-11 17:34:58] ...trap/engines/engine.py.229: process: 0.85186 | load: 0.00314 [I 2022-10-11 17:34:58] ...trap/engines/engine.py.230: loss: 0.17969 [I 2022-10-11 17:34:58] ...optimizers/trijoint.py.151: total_norm: 26.175625 [I 2022-10-11 17:35:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 540/2304 [I 2022-10-11 17:35:07] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:01 [I 2022-10-11 17:35:07] ...trap/engines/engine.py.229: process: 0.85077 | load: 0.00316 [I 2022-10-11 17:35:07] ...trap/engines/engine.py.230: loss: 0.11528 [I 2022-10-11 17:35:07] ...optimizers/trijoint.py.151: total_norm: 19.621759 [I 2022-10-11 17:35:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 550/2304 [I 2022-10-11 17:35:16] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:04 [I 2022-10-11 17:35:16] ...trap/engines/engine.py.229: process: 0.85878 | load: 0.00318 [I 2022-10-11 17:35:16] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-11 17:35:16] ...optimizers/trijoint.py.151: total_norm: 30.851934 [I 2022-10-11 17:35:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 560/2304 [I 2022-10-11 17:35:24] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:24:50 [I 2022-10-11 17:35:24] ...trap/engines/engine.py.229: process: 0.86058 | load: 0.00312 [I 2022-10-11 17:35:24] ...trap/engines/engine.py.230: loss: 0.09184 [I 2022-10-11 17:35:24] ...optimizers/trijoint.py.151: total_norm: 17.581438 [I 2022-10-11 17:35:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 570/2304 [I 2022-10-11 17:35:33] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:47 [I 2022-10-11 17:35:33] ...trap/engines/engine.py.229: process: 0.86792 | load: 0.00314 [I 2022-10-11 17:35:33] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-11 17:35:33] ...optimizers/trijoint.py.151: total_norm: 20.626638 [I 2022-10-11 17:35:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 580/2304 [I 2022-10-11 17:35:41] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:40 [I 2022-10-11 17:35:41] ...trap/engines/engine.py.229: process: 0.85021 | load: 0.00313 [I 2022-10-11 17:35:41] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-11 17:35:41] ...optimizers/trijoint.py.151: total_norm: 10.505367 [I 2022-10-11 17:35:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 590/2304 [I 2022-10-11 17:35:50] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:23 [I 2022-10-11 17:35:50] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00314 [I 2022-10-11 17:35:50] ...trap/engines/engine.py.230: loss: 0.08652 [I 2022-10-11 17:35:50] ...optimizers/trijoint.py.151: total_norm: 11.202003 [I 2022-10-11 17:35:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 600/2304 [I 2022-10-11 17:35:58] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:24:13 [I 2022-10-11 17:35:58] ...trap/engines/engine.py.229: process: 0.83431 | load: 0.00314 [I 2022-10-11 17:35:58] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-11 17:35:58] ...optimizers/trijoint.py.151: total_norm: 14.318327 [I 2022-10-11 17:36:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 610/2304 [I 2022-10-11 17:36:07] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:19 [I 2022-10-11 17:36:07] ...trap/engines/engine.py.229: process: 0.86619 | load: 0.00311 [I 2022-10-11 17:36:07] ...trap/engines/engine.py.230: loss: 0.09054 [I 2022-10-11 17:36:07] ...optimizers/trijoint.py.151: total_norm: 12.115932 [I 2022-10-11 17:36:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 620/2304 [I 2022-10-11 17:36:16] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:10 [I 2022-10-11 17:36:16] ...trap/engines/engine.py.229: process: 0.86522 | load: 0.00337 [I 2022-10-11 17:36:16] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-11 17:36:16] ...optimizers/trijoint.py.151: total_norm: 12.378124 [I 2022-10-11 17:36:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 630/2304 [I 2022-10-11 17:36:24] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:24:17 [I 2022-10-11 17:36:24] ...trap/engines/engine.py.229: process: 0.87206 | load: 0.00315 [I 2022-10-11 17:36:24] ...trap/engines/engine.py.230: loss: 0.11753 [I 2022-10-11 17:36:24] ...optimizers/trijoint.py.151: total_norm: 15.647498 [I 2022-10-11 17:36:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 640/2304 [I 2022-10-11 17:36:33] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:24:04 [I 2022-10-11 17:36:33] ...trap/engines/engine.py.229: process: 0.87163 | load: 0.00313 [I 2022-10-11 17:36:33] ...trap/engines/engine.py.230: loss: 0.08827 [I 2022-10-11 17:36:33] ...optimizers/trijoint.py.151: total_norm: 10.499047 [I 2022-10-11 17:36:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 650/2304 [I 2022-10-11 17:36:42] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:55 [I 2022-10-11 17:36:42] ...trap/engines/engine.py.229: process: 0.87236 | load: 0.00316 [I 2022-10-11 17:36:42] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-11 17:36:42] ...optimizers/trijoint.py.151: total_norm: 11.926791 [I 2022-10-11 17:36:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 660/2304 [I 2022-10-11 17:36:50] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:35 [I 2022-10-11 17:36:50] ...trap/engines/engine.py.229: process: 0.85781 | load: 0.00315 [I 2022-10-11 17:36:50] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-11 17:36:50] ...optimizers/trijoint.py.151: total_norm: 11.405679 [I 2022-10-11 17:36:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 670/2304 [I 2022-10-11 17:36:59] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:14 [I 2022-10-11 17:36:59] ...trap/engines/engine.py.229: process: 0.85634 | load: 0.00314 [I 2022-10-11 17:36:59] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-11 17:36:59] ...optimizers/trijoint.py.151: total_norm: 9.245206 [I 2022-10-11 17:37:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 680/2304 [I 2022-10-11 17:37:08] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:16 [I 2022-10-11 17:37:08] ...trap/engines/engine.py.229: process: 0.85382 | load: 0.00316 [I 2022-10-11 17:37:08] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-11 17:37:08] ...optimizers/trijoint.py.151: total_norm: 10.123100 [I 2022-10-11 17:37:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 690/2304 [I 2022-10-11 17:37:16] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:23:00 [I 2022-10-11 17:37:16] ...trap/engines/engine.py.229: process: 0.86082 | load: 0.00314 [I 2022-10-11 17:37:16] ...trap/engines/engine.py.230: loss: 0.15622 [I 2022-10-11 17:37:16] ...optimizers/trijoint.py.151: total_norm: 13.218962 [I 2022-10-11 17:37:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 700/2304 [I 2022-10-11 17:37:25] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:53 [I 2022-10-11 17:37:25] ...trap/engines/engine.py.229: process: 0.85451 | load: 0.00311 [I 2022-10-11 17:37:25] ...trap/engines/engine.py.230: loss: 0.09082 [I 2022-10-11 17:37:25] ...optimizers/trijoint.py.151: total_norm: 9.826422 [I 2022-10-11 17:37:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 710/2304 [I 2022-10-11 17:37:33] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:46 [I 2022-10-11 17:37:33] ...trap/engines/engine.py.229: process: 0.85692 | load: 0.00318 [I 2022-10-11 17:37:33] ...trap/engines/engine.py.230: loss: 0.13101 [I 2022-10-11 17:37:33] ...optimizers/trijoint.py.151: total_norm: 15.162672 [I 2022-10-11 17:37:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 720/2304 [I 2022-10-11 17:37:42] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:48 [I 2022-10-11 17:37:42] ...trap/engines/engine.py.229: process: 0.88397 | load: 0.00317 [I 2022-10-11 17:37:42] ...trap/engines/engine.py.230: loss: 0.09583 [I 2022-10-11 17:37:42] ...optimizers/trijoint.py.151: total_norm: 9.103262 [I 2022-10-11 17:37:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 730/2304 [I 2022-10-11 17:37:51] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:22:36 [I 2022-10-11 17:37:51] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00314 [I 2022-10-11 17:37:51] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-11 17:37:51] ...optimizers/trijoint.py.151: total_norm: 23.696100 [I 2022-10-11 17:37:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 740/2304 [I 2022-10-11 17:37:59] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:31 [I 2022-10-11 17:37:59] ...trap/engines/engine.py.229: process: 0.87086 | load: 0.00312 [I 2022-10-11 17:37:59] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-11 17:37:59] ...optimizers/trijoint.py.151: total_norm: 9.822344 [I 2022-10-11 17:38:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 750/2304 [I 2022-10-11 17:38:08] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:09 [I 2022-10-11 17:38:08] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00313 [I 2022-10-11 17:38:08] ...trap/engines/engine.py.230: loss: 0.09830 [I 2022-10-11 17:38:08] ...optimizers/trijoint.py.151: total_norm: 12.181963 [I 2022-10-11 17:38:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 760/2304 [I 2022-10-11 17:38:16] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:16 [I 2022-10-11 17:38:16] ...trap/engines/engine.py.229: process: 0.87821 | load: 0.00315 [I 2022-10-11 17:38:16] ...trap/engines/engine.py.230: loss: 0.14833 [I 2022-10-11 17:38:16] ...optimizers/trijoint.py.151: total_norm: 11.025850 [I 2022-10-11 17:38:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 770/2304 [I 2022-10-11 17:38:25] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:21:59 [I 2022-10-11 17:38:25] ...trap/engines/engine.py.229: process: 0.86671 | load: 0.00317 [I 2022-10-11 17:38:25] ...trap/engines/engine.py.230: loss: 0.07845 [I 2022-10-11 17:38:25] ...optimizers/trijoint.py.151: total_norm: 20.048082 [I 2022-10-11 17:38:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 780/2304 [I 2022-10-11 17:38:34] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:21:56 [I 2022-10-11 17:38:34] ...trap/engines/engine.py.229: process: 0.87624 | load: 0.00312 [I 2022-10-11 17:38:34] ...trap/engines/engine.py.230: loss: 0.10934 [I 2022-10-11 17:38:34] ...optimizers/trijoint.py.151: total_norm: 17.855160 [I 2022-10-11 17:38:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 790/2304 [I 2022-10-11 17:38:42] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:39 [I 2022-10-11 17:38:42] ...trap/engines/engine.py.229: process: 0.85865 | load: 0.00314 [I 2022-10-11 17:38:42] ...trap/engines/engine.py.230: loss: 0.10441 [I 2022-10-11 17:38:42] ...optimizers/trijoint.py.151: total_norm: 11.225926 [I 2022-10-11 17:38:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 800/2304 [I 2022-10-11 17:38:51] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:21:34 [I 2022-10-11 17:38:51] ...trap/engines/engine.py.229: process: 0.86636 | load: 0.00309 [I 2022-10-11 17:38:51] ...trap/engines/engine.py.230: loss: 0.08629 [I 2022-10-11 17:38:51] ...optimizers/trijoint.py.151: total_norm: 9.479548 [I 2022-10-11 17:38:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 810/2304 [I 2022-10-11 17:38:59] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:21:09 [I 2022-10-11 17:38:59] ...trap/engines/engine.py.229: process: 0.85858 | load: 0.00296 [I 2022-10-11 17:38:59] ...trap/engines/engine.py.230: loss: 0.17131 [I 2022-10-11 17:38:59] ...optimizers/trijoint.py.151: total_norm: 19.765545 [I 2022-10-11 17:39:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 820/2304 [I 2022-10-11 17:39:08] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:20:56 [I 2022-10-11 17:39:08] ...trap/engines/engine.py.229: process: 0.86478 | load: 0.00295 [I 2022-10-11 17:39:08] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-11 17:39:08] ...optimizers/trijoint.py.151: total_norm: 38.341526 [I 2022-10-11 17:39:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 830/2304 [I 2022-10-11 17:39:16] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:20:42 [I 2022-10-11 17:39:16] ...trap/engines/engine.py.229: process: 0.84148 | load: 0.00295 [I 2022-10-11 17:39:16] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-11 17:39:16] ...optimizers/trijoint.py.151: total_norm: 10.360760 [I 2022-10-11 17:39:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 840/2304 [I 2022-10-11 17:39:25] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:20:40 [I 2022-10-11 17:39:25] ...trap/engines/engine.py.229: process: 0.85334 | load: 0.00297 [I 2022-10-11 17:39:25] ...trap/engines/engine.py.230: loss: 0.10842 [I 2022-10-11 17:39:25] ...optimizers/trijoint.py.151: total_norm: 20.968721 [I 2022-10-11 17:39:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 850/2304 [I 2022-10-11 17:39:33] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:20:51 [I 2022-10-11 17:39:33] ...trap/engines/engine.py.229: process: 0.88305 | load: 0.00378 [I 2022-10-11 17:39:33] ...trap/engines/engine.py.230: loss: 0.11387 [I 2022-10-11 17:39:33] ...optimizers/trijoint.py.151: total_norm: 11.410157 [I 2022-10-11 17:39:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 860/2304 [I 2022-10-11 17:39:42] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:20:40 [I 2022-10-11 17:39:42] ...trap/engines/engine.py.229: process: 0.85123 | load: 0.00301 [I 2022-10-11 17:39:42] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-11 17:39:42] ...optimizers/trijoint.py.151: total_norm: 41.991783 [I 2022-10-11 17:39:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 870/2304 [I 2022-10-11 17:39:51] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:20:25 [I 2022-10-11 17:39:51] ...trap/engines/engine.py.229: process: 0.86243 | load: 0.00302 [I 2022-10-11 17:39:51] ...trap/engines/engine.py.230: loss: 0.09362 [I 2022-10-11 17:39:51] ...optimizers/trijoint.py.151: total_norm: 9.366655 [I 2022-10-11 17:39:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 880/2304 [I 2022-10-11 17:39:59] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:20:12 [I 2022-10-11 17:39:59] ...trap/engines/engine.py.229: process: 0.86944 | load: 0.00298 [I 2022-10-11 17:39:59] ...trap/engines/engine.py.230: loss: 0.12608 [I 2022-10-11 17:39:59] ...optimizers/trijoint.py.151: total_norm: 31.562878 [I 2022-10-11 17:40:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 890/2304 [I 2022-10-11 17:40:08] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:19:55 [I 2022-10-11 17:40:08] ...trap/engines/engine.py.229: process: 0.83984 | load: 0.00301 [I 2022-10-11 17:40:08] ...trap/engines/engine.py.230: loss: 0.13064 [I 2022-10-11 17:40:08] ...optimizers/trijoint.py.151: total_norm: 15.422267 [I 2022-10-11 17:40:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 900/2304 [I 2022-10-11 17:40:16] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:19:50 [I 2022-10-11 17:40:16] ...trap/engines/engine.py.229: process: 0.84947 | load: 0.00300 [I 2022-10-11 17:40:16] ...trap/engines/engine.py.230: loss: 0.08951 [I 2022-10-11 17:40:16] ...optimizers/trijoint.py.151: total_norm: 15.897164 [I 2022-10-11 17:40:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 910/2304 [I 2022-10-11 17:40:25] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:19:43 [I 2022-10-11 17:40:25] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00301 [I 2022-10-11 17:40:25] ...trap/engines/engine.py.230: loss: 0.18259 [I 2022-10-11 17:40:25] ...optimizers/trijoint.py.151: total_norm: 20.806879 [I 2022-10-11 17:40:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 920/2304 [I 2022-10-11 17:40:33] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:36 [I 2022-10-11 17:40:33] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00299 [I 2022-10-11 17:40:33] ...trap/engines/engine.py.230: loss: 0.09516 [I 2022-10-11 17:40:33] ...optimizers/trijoint.py.151: total_norm: 11.429183 [I 2022-10-11 17:40:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 930/2304 [I 2022-10-11 17:40:42] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:19:20 [I 2022-10-11 17:40:42] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00300 [I 2022-10-11 17:40:42] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-11 17:40:42] ...optimizers/trijoint.py.151: total_norm: 17.255318 [I 2022-10-11 17:40:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 940/2304 [I 2022-10-11 17:40:50] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:17 [I 2022-10-11 17:40:50] ...trap/engines/engine.py.229: process: 0.83645 | load: 0.00314 [I 2022-10-11 17:40:50] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-11 17:40:50] ...optimizers/trijoint.py.151: total_norm: 21.927706 [I 2022-10-11 17:40:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 950/2304 [I 2022-10-11 17:40:59] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:26 [I 2022-10-11 17:40:59] ...trap/engines/engine.py.229: process: 0.86531 | load: 0.00314 [I 2022-10-11 17:40:59] ...trap/engines/engine.py.230: loss: 0.07965 [I 2022-10-11 17:40:59] ...optimizers/trijoint.py.151: total_norm: 11.287037 [I 2022-10-11 17:41:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 960/2304 [I 2022-10-11 17:41:07] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:23 [I 2022-10-11 17:41:07] ...trap/engines/engine.py.229: process: 0.86428 | load: 0.00316 [I 2022-10-11 17:41:07] ...trap/engines/engine.py.230: loss: 0.07243 [I 2022-10-11 17:41:07] ...optimizers/trijoint.py.151: total_norm: 13.133354 [I 2022-10-11 17:41:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 970/2304 [I 2022-10-11 17:41:16] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:14 [I 2022-10-11 17:41:16] ...trap/engines/engine.py.229: process: 0.87504 | load: 0.00316 [I 2022-10-11 17:41:16] ...trap/engines/engine.py.230: loss: 0.08893 [I 2022-10-11 17:41:16] ...optimizers/trijoint.py.151: total_norm: 9.016912 [I 2022-10-11 17:41:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 980/2304 [I 2022-10-11 17:41:25] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:53 [I 2022-10-11 17:41:25] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00311 [I 2022-10-11 17:41:25] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-11 17:41:25] ...optimizers/trijoint.py.151: total_norm: 21.517056 [I 2022-10-11 17:41:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 990/2304 [I 2022-10-11 17:41:33] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:40 [I 2022-10-11 17:41:33] ...trap/engines/engine.py.229: process: 0.85350 | load: 0.00316 [I 2022-10-11 17:41:33] ...trap/engines/engine.py.230: loss: 0.14353 [I 2022-10-11 17:41:33] ...optimizers/trijoint.py.151: total_norm: 27.448183 [I 2022-10-11 17:41:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1000/2304 [I 2022-10-11 17:41:42] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:44 [I 2022-10-11 17:41:42] ...trap/engines/engine.py.229: process: 0.85790 | load: 0.00316 [I 2022-10-11 17:41:42] ...trap/engines/engine.py.230: loss: 0.09096 [I 2022-10-11 17:41:42] ...optimizers/trijoint.py.151: total_norm: 11.183196 [I 2022-10-11 17:41:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1010/2304 [I 2022-10-11 17:41:50] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:30 [I 2022-10-11 17:41:50] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00316 [I 2022-10-11 17:41:50] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-11 17:41:50] ...optimizers/trijoint.py.151: total_norm: 12.809359 [I 2022-10-11 17:41:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1020/2304 [I 2022-10-11 17:41:59] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:26 [I 2022-10-11 17:41:59] ...trap/engines/engine.py.229: process: 0.86417 | load: 0.00313 [I 2022-10-11 17:41:59] ...trap/engines/engine.py.230: loss: 0.08727 [I 2022-10-11 17:41:59] ...optimizers/trijoint.py.151: total_norm: 14.392811 [I 2022-10-11 17:42:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1030/2304 [I 2022-10-11 17:42:08] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:27 [I 2022-10-11 17:42:08] ...trap/engines/engine.py.229: process: 0.87205 | load: 0.00313 [I 2022-10-11 17:42:08] ...trap/engines/engine.py.230: loss: 0.08067 [I 2022-10-11 17:42:08] ...optimizers/trijoint.py.151: total_norm: 14.752123 [I 2022-10-11 17:42:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 1040/2304 [I 2022-10-11 17:42:17] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:19 [I 2022-10-11 17:42:17] ...trap/engines/engine.py.229: process: 0.87736 | load: 0.00311 [I 2022-10-11 17:42:17] ...trap/engines/engine.py.230: loss: 0.13908 [I 2022-10-11 17:42:17] ...optimizers/trijoint.py.151: total_norm: 15.794801 [I 2022-10-11 17:42:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1050/2304 [I 2022-10-11 17:42:25] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:18:01 [I 2022-10-11 17:42:25] ...trap/engines/engine.py.229: process: 0.87852 | load: 0.00311 [I 2022-10-11 17:42:25] ...trap/engines/engine.py.230: loss: 0.14855 [I 2022-10-11 17:42:25] ...optimizers/trijoint.py.151: total_norm: 33.994053 [I 2022-10-11 17:42:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1060/2304 [I 2022-10-11 17:42:34] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:51 [I 2022-10-11 17:42:34] ...trap/engines/engine.py.229: process: 0.85939 | load: 0.00315 [I 2022-10-11 17:42:34] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-11 17:42:34] ...optimizers/trijoint.py.151: total_norm: 35.362198 [I 2022-10-11 17:42:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1070/2304 [I 2022-10-11 17:42:42] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:28 [I 2022-10-11 17:42:42] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00313 [I 2022-10-11 17:42:42] ...trap/engines/engine.py.230: loss: 0.13619 [I 2022-10-11 17:42:42] ...optimizers/trijoint.py.151: total_norm: 24.970213 [I 2022-10-11 17:42:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1080/2304 [I 2022-10-11 17:42:51] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:38 [I 2022-10-11 17:42:51] ...trap/engines/engine.py.229: process: 0.88068 | load: 0.00315 [I 2022-10-11 17:42:51] ...trap/engines/engine.py.230: loss: 0.10460 [I 2022-10-11 17:42:51] ...optimizers/trijoint.py.151: total_norm: 9.110469 [I 2022-10-11 17:43:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 1090/2304 [I 2022-10-11 17:43:00] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:14 [I 2022-10-11 17:43:00] ...trap/engines/engine.py.229: process: 0.84259 | load: 0.00313 [I 2022-10-11 17:43:00] ...trap/engines/engine.py.230: loss: 0.10520 [I 2022-10-11 17:43:00] ...optimizers/trijoint.py.151: total_norm: 14.075435 [I 2022-10-11 17:43:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1100/2304 [I 2022-10-11 17:43:08] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:17:17 [I 2022-10-11 17:43:08] ...trap/engines/engine.py.229: process: 0.86855 | load: 0.00314 [I 2022-10-11 17:43:08] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-11 17:43:08] ...optimizers/trijoint.py.151: total_norm: 13.502930 [I 2022-10-11 17:43:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 1110/2304 [I 2022-10-11 17:43:17] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:16:56 [I 2022-10-11 17:43:17] ...trap/engines/engine.py.229: process: 0.85237 | load: 0.00315 [I 2022-10-11 17:43:17] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-11 17:43:17] ...optimizers/trijoint.py.151: total_norm: 11.345958 [I 2022-10-11 17:43:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1120/2304 [I 2022-10-11 17:43:25] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:16:54 [I 2022-10-11 17:43:25] ...trap/engines/engine.py.229: process: 0.85835 | load: 0.00315 [I 2022-10-11 17:43:25] ...trap/engines/engine.py.230: loss: 0.08991 [I 2022-10-11 17:43:25] ...optimizers/trijoint.py.151: total_norm: 17.903013 [I 2022-10-11 17:43:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1130/2304 [I 2022-10-11 17:43:34] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:54 [I 2022-10-11 17:43:34] ...trap/engines/engine.py.229: process: 0.87063 | load: 0.00321 [I 2022-10-11 17:43:34] ...trap/engines/engine.py.230: loss: 0.08786 [I 2022-10-11 17:43:34] ...optimizers/trijoint.py.151: total_norm: 8.571829 [I 2022-10-11 17:43:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 1140/2304 [I 2022-10-11 17:43:43] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:41 [I 2022-10-11 17:43:43] ...trap/engines/engine.py.229: process: 0.86304 | load: 0.00315 [I 2022-10-11 17:43:43] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-11 17:43:43] ...optimizers/trijoint.py.151: total_norm: 16.704895 [I 2022-10-11 17:43:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1150/2304 [I 2022-10-11 17:43:51] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:36 [I 2022-10-11 17:43:51] ...trap/engines/engine.py.229: process: 0.84940 | load: 0.00317 [I 2022-10-11 17:43:51] ...trap/engines/engine.py.230: loss: 0.13111 [I 2022-10-11 17:43:51] ...optimizers/trijoint.py.151: total_norm: 20.063543 [I 2022-10-11 17:44:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 1160/2304 [I 2022-10-11 17:44:00] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:30 [I 2022-10-11 17:44:00] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00316 [I 2022-10-11 17:44:00] ...trap/engines/engine.py.230: loss: 0.13786 [I 2022-10-11 17:44:00] ...optimizers/trijoint.py.151: total_norm: 13.697198 [I 2022-10-11 17:44:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1170/2304 [I 2022-10-11 17:44:08] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:07 [I 2022-10-11 17:44:08] ...trap/engines/engine.py.229: process: 0.85619 | load: 0.00314 [I 2022-10-11 17:44:08] ...trap/engines/engine.py.230: loss: 0.10872 [I 2022-10-11 17:44:08] ...optimizers/trijoint.py.151: total_norm: 11.394040 [I 2022-10-11 17:44:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 1180/2304 [I 2022-10-11 17:44:17] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:16:00 [I 2022-10-11 17:44:17] ...trap/engines/engine.py.229: process: 0.86082 | load: 0.00313 [I 2022-10-11 17:44:17] ...trap/engines/engine.py.230: loss: 0.15107 [I 2022-10-11 17:44:17] ...optimizers/trijoint.py.151: total_norm: 9.521568 [I 2022-10-11 17:44:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 1190/2304 [I 2022-10-11 17:44:26] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:15:54 [I 2022-10-11 17:44:26] ...trap/engines/engine.py.229: process: 0.84521 | load: 0.00317 [I 2022-10-11 17:44:26] ...trap/engines/engine.py.230: loss: 0.08981 [I 2022-10-11 17:44:26] ...optimizers/trijoint.py.151: total_norm: 20.323416 [I 2022-10-11 17:44:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1200/2304 [I 2022-10-11 17:44:34] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:39 [I 2022-10-11 17:44:34] ...trap/engines/engine.py.229: process: 0.84277 | load: 0.00318 [I 2022-10-11 17:44:34] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-11 17:44:34] ...optimizers/trijoint.py.151: total_norm: 14.191067 [I 2022-10-11 17:44:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 1210/2304 [I 2022-10-11 17:44:43] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:43 [I 2022-10-11 17:44:43] ...trap/engines/engine.py.229: process: 0.85307 | load: 0.00296 [I 2022-10-11 17:44:43] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-11 17:44:43] ...optimizers/trijoint.py.151: total_norm: 42.906322 [I 2022-10-11 17:44:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1220/2304 [I 2022-10-11 17:44:51] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:19 [I 2022-10-11 17:44:51] ...trap/engines/engine.py.229: process: 0.85188 | load: 0.00298 [I 2022-10-11 17:44:51] ...trap/engines/engine.py.230: loss: 0.09331 [I 2022-10-11 17:44:51] ...optimizers/trijoint.py.151: total_norm: 13.080956 [I 2022-10-11 17:45:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 1230/2304 [I 2022-10-11 17:45:00] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:04 [I 2022-10-11 17:45:00] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00295 [I 2022-10-11 17:45:00] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-11 17:45:00] ...optimizers/trijoint.py.151: total_norm: 21.287172 [I 2022-10-11 17:45:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1240/2304 [I 2022-10-11 17:45:08] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:14:49 [I 2022-10-11 17:45:08] ...trap/engines/engine.py.229: process: 0.83862 | load: 0.00294 [I 2022-10-11 17:45:08] ...trap/engines/engine.py.230: loss: 0.08141 [I 2022-10-11 17:45:08] ...optimizers/trijoint.py.151: total_norm: 13.991620 [I 2022-10-11 17:45:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 1250/2304 [I 2022-10-11 17:45:17] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:53 [I 2022-10-11 17:45:17] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00296 [I 2022-10-11 17:45:17] ...trap/engines/engine.py.230: loss: 0.08844 [I 2022-10-11 17:45:17] ...optimizers/trijoint.py.151: total_norm: 15.826576 [I 2022-10-11 17:45:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1260/2304 [I 2022-10-11 17:45:25] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:44 [I 2022-10-11 17:45:25] ...trap/engines/engine.py.229: process: 0.83632 | load: 0.00296 [I 2022-10-11 17:45:25] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-11 17:45:25] ...optimizers/trijoint.py.151: total_norm: 23.608385 [I 2022-10-11 17:45:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1270/2304 [I 2022-10-11 17:45:34] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:36 [I 2022-10-11 17:45:34] ...trap/engines/engine.py.229: process: 0.85161 | load: 0.00296 [I 2022-10-11 17:45:34] ...trap/engines/engine.py.230: loss: 0.08479 [I 2022-10-11 17:45:34] ...optimizers/trijoint.py.151: total_norm: 10.592810 [I 2022-10-11 17:45:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1280/2304 [I 2022-10-11 17:45:42] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:36 [I 2022-10-11 17:45:42] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00296 [I 2022-10-11 17:45:42] ...trap/engines/engine.py.230: loss: 0.07795 [I 2022-10-11 17:45:42] ...optimizers/trijoint.py.151: total_norm: 10.825644 [I 2022-10-11 17:45:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1290/2304 [I 2022-10-11 17:45:51] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:14:16 [I 2022-10-11 17:45:51] ...trap/engines/engine.py.229: process: 0.85942 | load: 0.00302 [I 2022-10-11 17:45:51] ...trap/engines/engine.py.230: loss: 0.11871 [I 2022-10-11 17:45:51] ...optimizers/trijoint.py.151: total_norm: 15.190819 [I 2022-10-11 17:45:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1300/2304 [I 2022-10-11 17:45:59] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:07 [I 2022-10-11 17:45:59] ...trap/engines/engine.py.229: process: 0.83337 | load: 0.00296 [I 2022-10-11 17:45:59] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-11 17:45:59] ...optimizers/trijoint.py.151: total_norm: 27.993998 [I 2022-10-11 17:46:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1310/2304 [I 2022-10-11 17:46:07] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:00 [I 2022-10-11 17:46:07] ...trap/engines/engine.py.229: process: 0.84622 | load: 0.00302 [I 2022-10-11 17:46:08] ...trap/engines/engine.py.230: loss: 0.09554 [I 2022-10-11 17:46:08] ...optimizers/trijoint.py.151: total_norm: 18.247583 [I 2022-10-11 17:46:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1320/2304 [I 2022-10-11 17:46:16] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:13:50 [I 2022-10-11 17:46:16] ...trap/engines/engine.py.229: process: 0.84283 | load: 0.00300 [I 2022-10-11 17:46:16] ...trap/engines/engine.py.230: loss: 0.08638 [I 2022-10-11 17:46:16] ...optimizers/trijoint.py.151: total_norm: 9.335423 [I 2022-10-11 17:46:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1330/2304 [I 2022-10-11 17:46:24] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:43 [I 2022-10-11 17:46:24] ...trap/engines/engine.py.229: process: 0.84091 | load: 0.00300 [I 2022-10-11 17:46:24] ...trap/engines/engine.py.230: loss: 0.10309 [I 2022-10-11 17:46:24] ...optimizers/trijoint.py.151: total_norm: 10.059298 [I 2022-10-11 17:46:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1340/2304 [I 2022-10-11 17:46:33] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:33 [I 2022-10-11 17:46:33] ...trap/engines/engine.py.229: process: 0.84218 | load: 0.00304 [I 2022-10-11 17:46:33] ...trap/engines/engine.py.230: loss: 0.08307 [I 2022-10-11 17:46:33] ...optimizers/trijoint.py.151: total_norm: 37.730247 [I 2022-10-11 17:46:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1350/2304 [I 2022-10-11 17:46:41] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:30 [I 2022-10-11 17:46:41] ...trap/engines/engine.py.229: process: 0.85765 | load: 0.00301 [I 2022-10-11 17:46:41] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-11 17:46:41] ...optimizers/trijoint.py.151: total_norm: 30.094772 [I 2022-10-11 17:46:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1360/2304 [I 2022-10-11 17:46:50] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:22 [I 2022-10-11 17:46:50] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00300 [I 2022-10-11 17:46:50] ...trap/engines/engine.py.230: loss: 0.12460 [I 2022-10-11 17:46:50] ...optimizers/trijoint.py.151: total_norm: 23.274391 [I 2022-10-11 17:46:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1370/2304 [I 2022-10-11 17:46:59] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:13:18 [I 2022-10-11 17:46:59] ...trap/engines/engine.py.229: process: 0.86743 | load: 0.00300 [I 2022-10-11 17:46:59] ...trap/engines/engine.py.230: loss: 0.09667 [I 2022-10-11 17:46:59] ...optimizers/trijoint.py.151: total_norm: 25.031063 [I 2022-10-11 17:47:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1380/2304 [I 2022-10-11 17:47:07] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:13:00 [I 2022-10-11 17:47:07] ...trap/engines/engine.py.229: process: 0.82743 | load: 0.00305 [I 2022-10-11 17:47:07] ...trap/engines/engine.py.230: loss: 0.10013 [I 2022-10-11 17:47:07] ...optimizers/trijoint.py.151: total_norm: 7.689413 [I 2022-10-11 17:47:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1390/2304 [I 2022-10-11 17:47:16] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:12:56 [I 2022-10-11 17:47:16] ...trap/engines/engine.py.229: process: 0.85487 | load: 0.00300 [I 2022-10-11 17:47:16] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-11 17:47:16] ...optimizers/trijoint.py.151: total_norm: 22.798691 [I 2022-10-11 17:47:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1400/2304 [I 2022-10-11 17:47:24] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:44 [I 2022-10-11 17:47:24] ...trap/engines/engine.py.229: process: 0.83708 | load: 0.00299 [I 2022-10-11 17:47:24] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-11 17:47:24] ...optimizers/trijoint.py.151: total_norm: 12.806868 [I 2022-10-11 17:47:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1410/2304 [I 2022-10-11 17:47:33] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:12:36 [I 2022-10-11 17:47:33] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00308 [I 2022-10-11 17:47:33] ...trap/engines/engine.py.230: loss: 0.13887 [I 2022-10-11 17:47:33] ...optimizers/trijoint.py.151: total_norm: 11.618631 [I 2022-10-11 17:47:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1420/2304 [I 2022-10-11 17:47:41] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:56 [I 2022-10-11 17:47:41] ...trap/engines/engine.py.229: process: 0.89470 | load: 0.00379 [I 2022-10-11 17:47:41] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-11 17:47:41] ...optimizers/trijoint.py.151: total_norm: 17.195381 [I 2022-10-11 17:47:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1430/2304 [I 2022-10-11 17:47:50] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:12:35 [I 2022-10-11 17:47:50] ...trap/engines/engine.py.229: process: 0.85992 | load: 0.00311 [I 2022-10-11 17:47:50] ...trap/engines/engine.py.230: loss: 0.14468 [I 2022-10-11 17:47:50] ...optimizers/trijoint.py.151: total_norm: 33.900436 [I 2022-10-11 17:47:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1440/2304 [I 2022-10-11 17:47:59] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:12:20 [I 2022-10-11 17:47:59] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00318 [I 2022-10-11 17:47:59] ...trap/engines/engine.py.230: loss: 0.09207 [I 2022-10-11 17:47:59] ...optimizers/trijoint.py.151: total_norm: 17.213682 [I 2022-10-11 17:48:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1450/2304 [I 2022-10-11 17:48:07] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:12:12 [I 2022-10-11 17:48:07] ...trap/engines/engine.py.229: process: 0.86698 | load: 0.00320 [I 2022-10-11 17:48:07] ...trap/engines/engine.py.230: loss: 0.09702 [I 2022-10-11 17:48:07] ...optimizers/trijoint.py.151: total_norm: 8.566142 [I 2022-10-11 17:48:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1460/2304 [I 2022-10-11 17:48:16] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:12:02 [I 2022-10-11 17:48:16] ...trap/engines/engine.py.229: process: 0.86086 | load: 0.00325 [I 2022-10-11 17:48:16] ...trap/engines/engine.py.230: loss: 0.07955 [I 2022-10-11 17:48:16] ...optimizers/trijoint.py.151: total_norm: 12.157902 [I 2022-10-11 17:48:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1470/2304 [I 2022-10-11 17:48:24] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:11:57 [I 2022-10-11 17:48:24] ...trap/engines/engine.py.229: process: 0.85613 | load: 0.00314 [I 2022-10-11 17:48:24] ...trap/engines/engine.py.230: loss: 0.12646 [I 2022-10-11 17:48:24] ...optimizers/trijoint.py.151: total_norm: 10.859488 [I 2022-10-11 17:48:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1480/2304 [I 2022-10-11 17:48:33] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:11:43 [I 2022-10-11 17:48:33] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00319 [I 2022-10-11 17:48:33] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-11 17:48:33] ...optimizers/trijoint.py.151: total_norm: 10.142427 [I 2022-10-11 17:48:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1490/2304 [I 2022-10-11 17:48:42] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:11:35 [I 2022-10-11 17:48:42] ...trap/engines/engine.py.229: process: 0.84501 | load: 0.00324 [I 2022-10-11 17:48:42] ...trap/engines/engine.py.230: loss: 0.17040 [I 2022-10-11 17:48:42] ...optimizers/trijoint.py.151: total_norm: 12.135010 [I 2022-10-11 17:48:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1500/2304 [I 2022-10-11 17:48:50] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:11:32 [I 2022-10-11 17:48:50] ...trap/engines/engine.py.229: process: 0.86614 | load: 0.00318 [I 2022-10-11 17:48:50] ...trap/engines/engine.py.230: loss: 0.10385 [I 2022-10-11 17:48:50] ...optimizers/trijoint.py.151: total_norm: 14.381294 [I 2022-10-11 17:48:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1510/2304 [I 2022-10-11 17:48:59] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:23 [I 2022-10-11 17:48:59] ...trap/engines/engine.py.229: process: 0.86323 | load: 0.00320 [I 2022-10-11 17:48:59] ...trap/engines/engine.py.230: loss: 0.15177 [I 2022-10-11 17:48:59] ...optimizers/trijoint.py.151: total_norm: 11.384547 [I 2022-10-11 17:49:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1520/2304 [I 2022-10-11 17:49:07] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:19 [I 2022-10-11 17:49:07] ...trap/engines/engine.py.229: process: 0.87058 | load: 0.00312 [I 2022-10-11 17:49:07] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-11 17:49:08] ...optimizers/trijoint.py.151: total_norm: 18.419542 [I 2022-10-11 17:49:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1530/2304 [I 2022-10-11 17:49:16] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:11:04 [I 2022-10-11 17:49:16] ...trap/engines/engine.py.229: process: 0.85121 | load: 0.00314 [I 2022-10-11 17:49:16] ...trap/engines/engine.py.230: loss: 0.10578 [I 2022-10-11 17:49:16] ...optimizers/trijoint.py.151: total_norm: 13.926859 [I 2022-10-11 17:49:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1540/2304 [I 2022-10-11 17:49:25] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:11:02 [I 2022-10-11 17:49:25] ...trap/engines/engine.py.229: process: 0.88008 | load: 0.00314 [I 2022-10-11 17:49:25] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-11 17:49:25] ...optimizers/trijoint.py.151: total_norm: 16.939621 [I 2022-10-11 17:49:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1550/2304 [I 2022-10-11 17:49:33] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:45 [I 2022-10-11 17:49:33] ...trap/engines/engine.py.229: process: 0.85361 | load: 0.00314 [I 2022-10-11 17:49:33] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-11 17:49:33] ...optimizers/trijoint.py.151: total_norm: 11.153494 [I 2022-10-11 17:49:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1560/2304 [I 2022-10-11 17:49:42] ...trap/engines/engine.py.228: elapsed: 0:22:24 | left: 0:10:37 [I 2022-10-11 17:49:42] ...trap/engines/engine.py.229: process: 0.85701 | load: 0.00314 [I 2022-10-11 17:49:42] ...trap/engines/engine.py.230: loss: 0.12446 [I 2022-10-11 17:49:42] ...optimizers/trijoint.py.151: total_norm: 16.739861 [I 2022-10-11 17:49:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1570/2304 [I 2022-10-11 17:49:50] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:28 [I 2022-10-11 17:49:50] ...trap/engines/engine.py.229: process: 0.87660 | load: 0.00312 [I 2022-10-11 17:49:50] ...trap/engines/engine.py.230: loss: 0.18239 [I 2022-10-11 17:49:50] ...optimizers/trijoint.py.151: total_norm: 21.716402 [I 2022-10-11 17:49:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1580/2304 [I 2022-10-11 17:49:59] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:19 [I 2022-10-11 17:49:59] ...trap/engines/engine.py.229: process: 0.84354 | load: 0.00317 [I 2022-10-11 17:49:59] ...trap/engines/engine.py.230: loss: 0.12059 [I 2022-10-11 17:49:59] ...optimizers/trijoint.py.151: total_norm: 12.627493 [I 2022-10-11 17:50:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1590/2304 [I 2022-10-11 17:50:08] ...trap/engines/engine.py.228: elapsed: 0:22:50 | left: 0:10:11 [I 2022-10-11 17:50:08] ...trap/engines/engine.py.229: process: 0.85904 | load: 0.00312 [I 2022-10-11 17:50:08] ...trap/engines/engine.py.230: loss: 0.14496 [I 2022-10-11 17:50:08] ...optimizers/trijoint.py.151: total_norm: 12.289818 [I 2022-10-11 17:50:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1600/2304 [I 2022-10-11 17:50:16] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:02 [I 2022-10-11 17:50:16] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00309 [I 2022-10-11 17:50:16] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-11 17:50:16] ...optimizers/trijoint.py.151: total_norm: 15.933720 [I 2022-10-11 17:50:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1610/2304 [I 2022-10-11 17:50:25] ...trap/engines/engine.py.228: elapsed: 0:23:07 | left: 0:09:57 [I 2022-10-11 17:50:25] ...trap/engines/engine.py.229: process: 0.87479 | load: 0.00313 [I 2022-10-11 17:50:25] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-11 17:50:25] ...optimizers/trijoint.py.151: total_norm: 33.529243 [I 2022-10-11 17:50:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1620/2304 [I 2022-10-11 17:50:33] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:09:47 [I 2022-10-11 17:50:33] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00318 [I 2022-10-11 17:50:33] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-11 17:50:33] ...optimizers/trijoint.py.151: total_norm: 13.465559 [I 2022-10-11 17:50:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1630/2304 [I 2022-10-11 17:50:42] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:09:38 [I 2022-10-11 17:50:42] ...trap/engines/engine.py.229: process: 0.86399 | load: 0.00320 [I 2022-10-11 17:50:42] ...trap/engines/engine.py.230: loss: 0.09364 [I 2022-10-11 17:50:42] ...optimizers/trijoint.py.151: total_norm: 18.724771 [I 2022-10-11 17:50:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1640/2304 [I 2022-10-11 17:50:51] ...trap/engines/engine.py.228: elapsed: 0:23:33 | left: 0:09:29 [I 2022-10-11 17:50:51] ...trap/engines/engine.py.229: process: 0.85853 | load: 0.00317 [I 2022-10-11 17:50:51] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-11 17:50:51] ...optimizers/trijoint.py.151: total_norm: 20.507004 [I 2022-10-11 17:50:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1650/2304 [I 2022-10-11 17:50:59] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:21 [I 2022-10-11 17:50:59] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00318 [I 2022-10-11 17:50:59] ...trap/engines/engine.py.230: loss: 0.14801 [I 2022-10-11 17:50:59] ...optimizers/trijoint.py.151: total_norm: 20.245701 [I 2022-10-11 17:51:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1660/2304 [I 2022-10-11 17:51:08] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:09 [I 2022-10-11 17:51:08] ...trap/engines/engine.py.229: process: 0.83447 | load: 0.00314 [I 2022-10-11 17:51:08] ...trap/engines/engine.py.230: loss: 0.14306 [I 2022-10-11 17:51:08] ...optimizers/trijoint.py.151: total_norm: 18.292940 [I 2022-10-11 17:51:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1670/2304 [I 2022-10-11 17:51:16] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:00 [I 2022-10-11 17:51:16] ...trap/engines/engine.py.229: process: 0.84781 | load: 0.00316 [I 2022-10-11 17:51:16] ...trap/engines/engine.py.230: loss: 0.12341 [I 2022-10-11 17:51:16] ...optimizers/trijoint.py.151: total_norm: 17.400167 [I 2022-10-11 17:51:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1680/2304 [I 2022-10-11 17:51:25] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:08:53 [I 2022-10-11 17:51:25] ...trap/engines/engine.py.229: process: 0.85072 | load: 0.00319 [I 2022-10-11 17:51:25] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-11 17:51:25] ...optimizers/trijoint.py.151: total_norm: 29.879753 [I 2022-10-11 17:51:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1690/2304 [I 2022-10-11 17:51:34] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:08:47 [I 2022-10-11 17:51:34] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00317 [I 2022-10-11 17:51:34] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-11 17:51:34] ...optimizers/trijoint.py.151: total_norm: 15.857176 [I 2022-10-11 17:51:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1700/2304 [I 2022-10-11 17:51:42] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:42 [I 2022-10-11 17:51:42] ...trap/engines/engine.py.229: process: 0.85547 | load: 0.00316 [I 2022-10-11 17:51:42] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-11 17:51:42] ...optimizers/trijoint.py.151: total_norm: 26.547426 [I 2022-10-11 17:51:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1710/2304 [I 2022-10-11 17:51:51] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:24 [I 2022-10-11 17:51:51] ...trap/engines/engine.py.229: process: 0.84357 | load: 0.00298 [I 2022-10-11 17:51:51] ...trap/engines/engine.py.230: loss: 0.18962 [I 2022-10-11 17:51:51] ...optimizers/trijoint.py.151: total_norm: 90.587074 [I 2022-10-11 17:51:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1720/2304 [I 2022-10-11 17:51:59] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:14 [I 2022-10-11 17:51:59] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00305 [I 2022-10-11 17:51:59] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-11 17:51:59] ...optimizers/trijoint.py.151: total_norm: 20.825258 [I 2022-10-11 17:52:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1730/2304 [I 2022-10-11 17:52:08] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:01 [I 2022-10-11 17:52:08] ...trap/engines/engine.py.229: process: 0.83545 | load: 0.00299 [I 2022-10-11 17:52:08] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-11 17:52:08] ...optimizers/trijoint.py.151: total_norm: 19.222090 [I 2022-10-11 17:52:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1740/2304 [I 2022-10-11 17:52:16] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:07:57 [I 2022-10-11 17:52:16] ...trap/engines/engine.py.229: process: 0.85751 | load: 0.00296 [I 2022-10-11 17:52:16] ...trap/engines/engine.py.230: loss: 0.07118 [I 2022-10-11 17:52:16] ...optimizers/trijoint.py.151: total_norm: 10.676038 [I 2022-10-11 17:52:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1750/2304 [I 2022-10-11 17:52:25] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:07:46 [I 2022-10-11 17:52:25] ...trap/engines/engine.py.229: process: 0.84411 | load: 0.00296 [I 2022-10-11 17:52:25] ...trap/engines/engine.py.230: loss: 0.15783 [I 2022-10-11 17:52:25] ...optimizers/trijoint.py.151: total_norm: 14.666590 [I 2022-10-11 17:52:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1760/2304 [I 2022-10-11 17:52:33] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:44 [I 2022-10-11 17:52:33] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00298 [I 2022-10-11 17:52:33] ...trap/engines/engine.py.230: loss: 0.16511 [I 2022-10-11 17:52:33] ...optimizers/trijoint.py.151: total_norm: 45.065563 [I 2022-10-11 17:52:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1770/2304 [I 2022-10-11 17:52:42] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:07:30 [I 2022-10-11 17:52:42] ...trap/engines/engine.py.229: process: 0.83963 | load: 0.00301 [I 2022-10-11 17:52:42] ...trap/engines/engine.py.230: loss: 0.16754 [I 2022-10-11 17:52:42] ...optimizers/trijoint.py.151: total_norm: 17.543678 [I 2022-10-11 17:52:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1780/2304 [I 2022-10-11 17:52:50] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:22 [I 2022-10-11 17:52:50] ...trap/engines/engine.py.229: process: 0.83576 | load: 0.00303 [I 2022-10-11 17:52:50] ...trap/engines/engine.py.230: loss: 0.12629 [I 2022-10-11 17:52:50] ...optimizers/trijoint.py.151: total_norm: 22.099001 [I 2022-10-11 17:52:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1790/2304 [I 2022-10-11 17:52:59] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:16 [I 2022-10-11 17:52:59] ...trap/engines/engine.py.229: process: 0.84970 | load: 0.00300 [I 2022-10-11 17:52:59] ...trap/engines/engine.py.230: loss: 0.12767 [I 2022-10-11 17:52:59] ...optimizers/trijoint.py.151: total_norm: 28.450094 [I 2022-10-11 17:53:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1800/2304 [I 2022-10-11 17:53:07] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:06 [I 2022-10-11 17:53:07] ...trap/engines/engine.py.229: process: 0.83570 | load: 0.00299 [I 2022-10-11 17:53:07] ...trap/engines/engine.py.230: loss: 0.14979 [I 2022-10-11 17:53:07] ...optimizers/trijoint.py.151: total_norm: 27.143358 [I 2022-10-11 17:53:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1810/2304 [I 2022-10-11 17:53:16] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:00 [I 2022-10-11 17:53:16] ...trap/engines/engine.py.229: process: 0.84663 | load: 0.00310 [I 2022-10-11 17:53:16] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-11 17:53:16] ...optimizers/trijoint.py.151: total_norm: 12.344684 [I 2022-10-11 17:53:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1820/2304 [I 2022-10-11 17:53:24] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:49 [I 2022-10-11 17:53:24] ...trap/engines/engine.py.229: process: 0.85070 | load: 0.00300 [I 2022-10-11 17:53:24] ...trap/engines/engine.py.230: loss: 0.09786 [I 2022-10-11 17:53:24] ...optimizers/trijoint.py.151: total_norm: 12.183439 [I 2022-10-11 17:53:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1830/2304 [I 2022-10-11 17:53:33] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:42 [I 2022-10-11 17:53:33] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00299 [I 2022-10-11 17:53:33] ...trap/engines/engine.py.230: loss: 0.10817 [I 2022-10-11 17:53:33] ...optimizers/trijoint.py.151: total_norm: 13.579167 [I 2022-10-11 17:53:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1840/2304 [I 2022-10-11 17:53:41] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:35 [I 2022-10-11 17:53:41] ...trap/engines/engine.py.229: process: 0.85154 | load: 0.00298 [I 2022-10-11 17:53:41] ...trap/engines/engine.py.230: loss: 0.08582 [I 2022-10-11 17:53:41] ...optimizers/trijoint.py.151: total_norm: 13.925184 [I 2022-10-11 17:53:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1850/2304 [I 2022-10-11 17:53:50] ...trap/engines/engine.py.228: elapsed: 0:26:32 | left: 0:06:28 [I 2022-10-11 17:53:50] ...trap/engines/engine.py.229: process: 0.86671 | load: 0.00304 [I 2022-10-11 17:53:50] ...trap/engines/engine.py.230: loss: 0.09608 [I 2022-10-11 17:53:50] ...optimizers/trijoint.py.151: total_norm: 30.869070 [I 2022-10-11 17:53:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 1860/2304 [I 2022-10-11 17:53:58] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:18 [I 2022-10-11 17:53:58] ...trap/engines/engine.py.229: process: 0.85155 | load: 0.00312 [I 2022-10-11 17:53:58] ...trap/engines/engine.py.230: loss: 0.08043 [I 2022-10-11 17:53:58] ...optimizers/trijoint.py.151: total_norm: 15.710388 [I 2022-10-11 17:54:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1870/2304 [I 2022-10-11 17:54:07] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:13 [I 2022-10-11 17:54:07] ...trap/engines/engine.py.229: process: 0.88033 | load: 0.00315 [I 2022-10-11 17:54:07] ...trap/engines/engine.py.230: loss: 0.10140 [I 2022-10-11 17:54:07] ...optimizers/trijoint.py.151: total_norm: 12.945706 [I 2022-10-11 17:54:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 1880/2304 [I 2022-10-11 17:54:15] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:04 [I 2022-10-11 17:54:15] ...trap/engines/engine.py.229: process: 0.87282 | load: 0.00316 [I 2022-10-11 17:54:15] ...trap/engines/engine.py.230: loss: 0.15577 [I 2022-10-11 17:54:15] ...optimizers/trijoint.py.151: total_norm: 11.510914 [I 2022-10-11 17:54:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1890/2304 [I 2022-10-11 17:54:24] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:05:57 [I 2022-10-11 17:54:24] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00316 [I 2022-10-11 17:54:24] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-11 17:54:24] ...optimizers/trijoint.py.151: total_norm: 13.640862 [I 2022-10-11 17:54:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1900/2304 [I 2022-10-11 17:54:33] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:47 [I 2022-10-11 17:54:33] ...trap/engines/engine.py.229: process: 0.83713 | load: 0.00316 [I 2022-10-11 17:54:33] ...trap/engines/engine.py.230: loss: 0.09982 [I 2022-10-11 17:54:33] ...optimizers/trijoint.py.151: total_norm: 14.056318 [I 2022-10-11 17:54:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1910/2304 [I 2022-10-11 17:54:41] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:39 [I 2022-10-11 17:54:41] ...trap/engines/engine.py.229: process: 0.85215 | load: 0.00313 [I 2022-10-11 17:54:41] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-11 17:54:41] ...optimizers/trijoint.py.151: total_norm: 20.983257 [I 2022-10-11 17:54:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1920/2304 [I 2022-10-11 17:54:50] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:31 [I 2022-10-11 17:54:50] ...trap/engines/engine.py.229: process: 0.86113 | load: 0.00315 [I 2022-10-11 17:54:50] ...trap/engines/engine.py.230: loss: 0.15150 [I 2022-10-11 17:54:50] ...optimizers/trijoint.py.151: total_norm: 16.603056 [I 2022-10-11 17:54:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1930/2304 [I 2022-10-11 17:54:59] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:21 [I 2022-10-11 17:54:59] ...trap/engines/engine.py.229: process: 0.85199 | load: 0.00313 [I 2022-10-11 17:54:59] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-11 17:54:59] ...optimizers/trijoint.py.151: total_norm: 8.040556 [I 2022-10-11 17:55:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1940/2304 [I 2022-10-11 17:55:07] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:13 [I 2022-10-11 17:55:07] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00313 [I 2022-10-11 17:55:07] ...trap/engines/engine.py.230: loss: 0.09253 [I 2022-10-11 17:55:07] ...optimizers/trijoint.py.151: total_norm: 9.511015 [I 2022-10-11 17:55:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1950/2304 [I 2022-10-11 17:55:16] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:05 [I 2022-10-11 17:55:16] ...trap/engines/engine.py.229: process: 0.87318 | load: 0.00313 [I 2022-10-11 17:55:16] ...trap/engines/engine.py.230: loss: 0.09691 [I 2022-10-11 17:55:16] ...optimizers/trijoint.py.151: total_norm: 13.155400 [I 2022-10-11 17:55:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1960/2304 [I 2022-10-11 17:55:24] ...trap/engines/engine.py.228: elapsed: 0:28:06 | left: 0:04:55 [I 2022-10-11 17:55:24] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00316 [I 2022-10-11 17:55:24] ...trap/engines/engine.py.230: loss: 0.10704 [I 2022-10-11 17:55:24] ...optimizers/trijoint.py.151: total_norm: 13.543402 [I 2022-10-11 17:55:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1970/2304 [I 2022-10-11 17:55:33] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:46 [I 2022-10-11 17:55:33] ...trap/engines/engine.py.229: process: 0.83983 | load: 0.00318 [I 2022-10-11 17:55:33] ...trap/engines/engine.py.230: loss: 0.10090 [I 2022-10-11 17:55:33] ...optimizers/trijoint.py.151: total_norm: 9.342178 [I 2022-10-11 17:55:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1980/2304 [I 2022-10-11 17:55:42] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:38 [I 2022-10-11 17:55:42] ...trap/engines/engine.py.229: process: 0.85754 | load: 0.00314 [I 2022-10-11 17:55:42] ...trap/engines/engine.py.230: loss: 0.15730 [I 2022-10-11 17:55:42] ...optimizers/trijoint.py.151: total_norm: 25.222019 [I 2022-10-11 17:55:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1990/2304 [I 2022-10-11 17:55:50] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:29 [I 2022-10-11 17:55:50] ...trap/engines/engine.py.229: process: 0.83926 | load: 0.00313 [I 2022-10-11 17:55:50] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-11 17:55:50] ...optimizers/trijoint.py.151: total_norm: 9.912772 [I 2022-10-11 17:55:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 2000/2304 [I 2022-10-11 17:55:59] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:20 [I 2022-10-11 17:55:59] ...trap/engines/engine.py.229: process: 0.85442 | load: 0.00315 [I 2022-10-11 17:55:59] ...trap/engines/engine.py.230: loss: 0.16485 [I 2022-10-11 17:55:59] ...optimizers/trijoint.py.151: total_norm: 13.156429 [I 2022-10-11 17:56:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 2010/2304 [I 2022-10-11 17:56:07] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:12 [I 2022-10-11 17:56:07] ...trap/engines/engine.py.229: process: 0.86763 | load: 0.00317 [I 2022-10-11 17:56:07] ...trap/engines/engine.py.230: loss: 0.09529 [I 2022-10-11 17:56:07] ...optimizers/trijoint.py.151: total_norm: 21.388849 [I 2022-10-11 17:56:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 2020/2304 [I 2022-10-11 17:56:16] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:03 [I 2022-10-11 17:56:16] ...trap/engines/engine.py.229: process: 0.86189 | load: 0.00317 [I 2022-10-11 17:56:16] ...trap/engines/engine.py.230: loss: 0.09223 [I 2022-10-11 17:56:16] ...optimizers/trijoint.py.151: total_norm: 13.382770 [I 2022-10-11 17:56:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 2030/2304 [I 2022-10-11 17:56:25] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:03:54 [I 2022-10-11 17:56:25] ...trap/engines/engine.py.229: process: 0.85372 | load: 0.00312 [I 2022-10-11 17:56:25] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-11 17:56:25] ...optimizers/trijoint.py.151: total_norm: 21.798145 [I 2022-10-11 17:56:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 2040/2304 [I 2022-10-11 17:56:33] ...trap/engines/engine.py.228: elapsed: 0:29:15 | left: 0:03:45 [I 2022-10-11 17:56:33] ...trap/engines/engine.py.229: process: 0.86946 | load: 0.00312 [I 2022-10-11 17:56:33] ...trap/engines/engine.py.230: loss: 0.08789 [I 2022-10-11 17:56:33] ...optimizers/trijoint.py.151: total_norm: 11.605836 [I 2022-10-11 17:56:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 2050/2304 [I 2022-10-11 17:56:42] ...trap/engines/engine.py.228: elapsed: 0:29:24 | left: 0:03:36 [I 2022-10-11 17:56:42] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00312 [I 2022-10-11 17:56:42] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-11 17:56:42] ...optimizers/trijoint.py.151: total_norm: 33.053009 [I 2022-10-11 17:56:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 2060/2304 [I 2022-10-11 17:56:50] ...trap/engines/engine.py.228: elapsed: 0:29:32 | left: 0:03:28 [I 2022-10-11 17:56:50] ...trap/engines/engine.py.229: process: 0.85445 | load: 0.00317 [I 2022-10-11 17:56:50] ...trap/engines/engine.py.230: loss: 0.15349 [I 2022-10-11 17:56:50] ...optimizers/trijoint.py.151: total_norm: 14.147565 [I 2022-10-11 17:56:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 2070/2304 [I 2022-10-11 17:56:59] ...trap/engines/engine.py.228: elapsed: 0:29:41 | left: 0:03:20 [I 2022-10-11 17:56:59] ...trap/engines/engine.py.229: process: 0.84957 | load: 0.00311 [I 2022-10-11 17:56:59] ...trap/engines/engine.py.230: loss: 0.09015 [I 2022-10-11 17:56:59] ...optimizers/trijoint.py.151: total_norm: 17.760555 [I 2022-10-11 17:57:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 2080/2304 [I 2022-10-11 17:57:07] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:11 [I 2022-10-11 17:57:07] ...trap/engines/engine.py.229: process: 0.86358 | load: 0.00318 [I 2022-10-11 17:57:07] ...trap/engines/engine.py.230: loss: 0.12099 [I 2022-10-11 17:57:07] ...optimizers/trijoint.py.151: total_norm: 18.600760 [I 2022-10-11 17:57:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 2090/2304 [I 2022-10-11 17:57:16] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:02 [I 2022-10-11 17:57:16] ...trap/engines/engine.py.229: process: 0.84071 | load: 0.00314 [I 2022-10-11 17:57:16] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-11 17:57:16] ...optimizers/trijoint.py.151: total_norm: 12.313767 [I 2022-10-11 17:57:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 2100/2304 [I 2022-10-11 17:57:25] ...trap/engines/engine.py.228: elapsed: 0:30:07 | left: 0:02:53 [I 2022-10-11 17:57:25] ...trap/engines/engine.py.229: process: 0.84535 | load: 0.00314 [I 2022-10-11 17:57:25] ...trap/engines/engine.py.230: loss: 0.09615 [I 2022-10-11 17:57:25] ...optimizers/trijoint.py.151: total_norm: 9.271443 [I 2022-10-11 17:57:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 2110/2304 [I 2022-10-11 17:57:33] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:02:48 [I 2022-10-11 17:57:33] ...trap/engines/engine.py.229: process: 0.88106 | load: 0.00397 [I 2022-10-11 17:57:33] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-11 17:57:33] ...optimizers/trijoint.py.151: total_norm: 16.272978 [I 2022-10-11 17:57:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 2120/2304 [I 2022-10-11 17:57:42] ...trap/engines/engine.py.228: elapsed: 0:30:24 | left: 0:02:42 [I 2022-10-11 17:57:42] ...trap/engines/engine.py.229: process: 0.88734 | load: 0.00394 [I 2022-10-11 17:57:42] ...trap/engines/engine.py.230: loss: 0.09652 [I 2022-10-11 17:57:42] ...optimizers/trijoint.py.151: total_norm: 12.623722 [I 2022-10-11 17:57:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 2130/2304 [I 2022-10-11 17:57:51] ...trap/engines/engine.py.228: elapsed: 0:30:33 | left: 0:02:34 [I 2022-10-11 17:57:51] ...trap/engines/engine.py.229: process: 0.87610 | load: 0.00397 [I 2022-10-11 17:57:51] ...trap/engines/engine.py.230: loss: 0.08075 [I 2022-10-11 17:57:51] ...optimizers/trijoint.py.151: total_norm: 14.730869 [I 2022-10-11 17:58:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 2140/2304 [I 2022-10-11 17:58:00] ...trap/engines/engine.py.228: elapsed: 0:30:42 | left: 0:02:24 [I 2022-10-11 17:58:00] ...trap/engines/engine.py.229: process: 0.86666 | load: 0.00378 [I 2022-10-11 17:58:00] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-11 17:58:00] ...optimizers/trijoint.py.151: total_norm: 19.517200 [I 2022-10-11 17:58:09] ...trap/engines/engine.py.224: train: epoch 113 | batch 2150/2304 [I 2022-10-11 17:58:09] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:13 [I 2022-10-11 17:58:09] ...trap/engines/engine.py.229: process: 0.84453 | load: 0.00392 [I 2022-10-11 17:58:09] ...trap/engines/engine.py.230: loss: 0.20654 [I 2022-10-11 17:58:09] ...optimizers/trijoint.py.151: total_norm: 17.568609 [I 2022-10-11 17:58:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 2160/2304 [I 2022-10-11 17:58:17] ...trap/engines/engine.py.228: elapsed: 0:30:59 | left: 0:02:01 [I 2022-10-11 17:58:17] ...trap/engines/engine.py.229: process: 0.83633 | load: 0.00295 [I 2022-10-11 17:58:17] ...trap/engines/engine.py.230: loss: 0.09138 [I 2022-10-11 17:58:17] ...optimizers/trijoint.py.151: total_norm: 9.497700 [I 2022-10-11 17:58:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 2170/2304 [I 2022-10-11 17:58:26] ...trap/engines/engine.py.228: elapsed: 0:31:08 | left: 0:01:53 [I 2022-10-11 17:58:26] ...trap/engines/engine.py.229: process: 0.84245 | load: 0.00297 [I 2022-10-11 17:58:26] ...trap/engines/engine.py.230: loss: 0.07916 [I 2022-10-11 17:58:26] ...optimizers/trijoint.py.151: total_norm: 12.194919 [I 2022-10-11 17:58:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 2180/2304 [I 2022-10-11 17:58:34] ...trap/engines/engine.py.228: elapsed: 0:31:16 | left: 0:01:44 [I 2022-10-11 17:58:34] ...trap/engines/engine.py.229: process: 0.83750 | load: 0.00296 [I 2022-10-11 17:58:34] ...trap/engines/engine.py.230: loss: 0.12810 [I 2022-10-11 17:58:34] ...optimizers/trijoint.py.151: total_norm: 14.226142 [I 2022-10-11 17:58:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 2190/2304 [I 2022-10-11 17:58:43] ...trap/engines/engine.py.228: elapsed: 0:31:24 | left: 0:01:37 [I 2022-10-11 17:58:43] ...trap/engines/engine.py.229: process: 0.85644 | load: 0.00300 [I 2022-10-11 17:58:43] ...trap/engines/engine.py.230: loss: 0.13441 [I 2022-10-11 17:58:43] ...optimizers/trijoint.py.151: total_norm: 13.327219 [I 2022-10-11 17:58:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 2200/2304 [I 2022-10-11 17:58:51] ...trap/engines/engine.py.228: elapsed: 0:31:33 | left: 0:01:27 [I 2022-10-11 17:58:51] ...trap/engines/engine.py.229: process: 0.84063 | load: 0.00301 [I 2022-10-11 17:58:51] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-11 17:58:51] ...optimizers/trijoint.py.151: total_norm: 23.045252 [I 2022-10-11 17:58:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 2210/2304 [I 2022-10-11 17:58:59] ...trap/engines/engine.py.228: elapsed: 0:31:41 | left: 0:01:19 [I 2022-10-11 17:58:59] ...trap/engines/engine.py.229: process: 0.83812 | load: 0.00301 [I 2022-10-11 17:58:59] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-11 17:58:59] ...optimizers/trijoint.py.151: total_norm: 27.833025 [I 2022-10-11 17:59:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 2220/2304 [I 2022-10-11 17:59:08] ...trap/engines/engine.py.228: elapsed: 0:31:50 | left: 0:01:10 [I 2022-10-11 17:59:08] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00299 [I 2022-10-11 17:59:08] ...trap/engines/engine.py.230: loss: 0.17394 [I 2022-10-11 17:59:08] ...optimizers/trijoint.py.151: total_norm: 19.186275 [I 2022-10-11 17:59:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 2230/2304 [I 2022-10-11 17:59:16] ...trap/engines/engine.py.228: elapsed: 0:31:58 | left: 0:01:02 [I 2022-10-11 17:59:16] ...trap/engines/engine.py.229: process: 0.84269 | load: 0.00299 [I 2022-10-11 17:59:16] ...trap/engines/engine.py.230: loss: 0.10095 [I 2022-10-11 17:59:16] ...optimizers/trijoint.py.151: total_norm: 17.143667 [I 2022-10-11 17:59:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 2240/2304 [I 2022-10-11 17:59:25] ...trap/engines/engine.py.228: elapsed: 0:32:07 | left: 0:00:53 [I 2022-10-11 17:59:25] ...trap/engines/engine.py.229: process: 0.83084 | load: 0.00302 [I 2022-10-11 17:59:25] ...trap/engines/engine.py.230: loss: 0.11134 [I 2022-10-11 17:59:25] ...optimizers/trijoint.py.151: total_norm: 22.976809 [I 2022-10-11 17:59:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 2250/2304 [I 2022-10-11 17:59:33] ...trap/engines/engine.py.228: elapsed: 0:32:15 | left: 0:00:45 [I 2022-10-11 17:59:33] ...trap/engines/engine.py.229: process: 0.85150 | load: 0.00302 [I 2022-10-11 17:59:33] ...trap/engines/engine.py.230: loss: 0.15667 [I 2022-10-11 17:59:33] ...optimizers/trijoint.py.151: total_norm: 10.110682 [I 2022-10-11 17:59:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 2260/2304 [I 2022-10-11 17:59:42] ...trap/engines/engine.py.228: elapsed: 0:32:24 | left: 0:00:37 [I 2022-10-11 17:59:42] ...trap/engines/engine.py.229: process: 0.85924 | load: 0.00298 [I 2022-10-11 17:59:42] ...trap/engines/engine.py.230: loss: 0.11590 [I 2022-10-11 17:59:42] ...optimizers/trijoint.py.151: total_norm: 9.899043 [I 2022-10-11 17:59:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 2270/2304 [I 2022-10-11 17:59:51] ...trap/engines/engine.py.228: elapsed: 0:32:33 | left: 0:00:29 [I 2022-10-11 17:59:51] ...trap/engines/engine.py.229: process: 0.90568 | load: 0.00377 [I 2022-10-11 17:59:51] ...trap/engines/engine.py.230: loss: 0.09048 [I 2022-10-11 17:59:51] ...optimizers/trijoint.py.151: total_norm: 11.286725 [I 2022-10-11 17:59:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 2280/2304 [I 2022-10-11 17:59:59] ...trap/engines/engine.py.228: elapsed: 0:32:41 | left: 0:00:21 [I 2022-10-11 17:59:59] ...trap/engines/engine.py.229: process: 0.87086 | load: 0.00377 [I 2022-10-11 17:59:59] ...trap/engines/engine.py.230: loss: 0.07432 [I 2022-10-11 17:59:59] ...optimizers/trijoint.py.151: total_norm: 13.211743 [I 2022-10-11 18:00:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 2290/2304 [I 2022-10-11 18:00:08] ...trap/engines/engine.py.228: elapsed: 0:32:50 | left: 0:00:12 [I 2022-10-11 18:00:08] ...trap/engines/engine.py.229: process: 0.88387 | load: 0.00378 [I 2022-10-11 18:00:08] ...trap/engines/engine.py.230: loss: 0.10173 [I 2022-10-11 18:00:08] ...optimizers/trijoint.py.151: total_norm: 11.839322 [I 2022-10-11 18:00:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 2300/2304 [I 2022-10-11 18:00:17] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:03 [I 2022-10-11 18:00:17] ...trap/engines/engine.py.229: process: 0.88082 | load: 0.00021 [I 2022-10-11 18:00:17] ...trap/engines/engine.py.230: loss: 0.15458 [I 2022-10-11 18:00:17] ...optimizers/trijoint.py.151: total_norm: 62.298424 [I 2022-10-11 18:00:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 2304/2304 [I 2022-10-11 18:00:21] ...trap/engines/engine.py.228: elapsed: 0:33:03 | left: 0:00:00 [I 2022-10-11 18:00:21] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00021 [I 2022-10-11 18:00:21] ...trap/engines/engine.py.230: loss: 0.13441 [I 2022-10-11 18:00:21] ...optimizers/trijoint.py.151: total_norm: 25.020714 [S 2022-10-11 18:00:22] ...trap/engines/engine.py.129: train_epoch.epoch: 113 [S 2022-10-11 18:00:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.11972446105162131 [I 2022-10-11 18:00:22] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 18:00:24] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 113 [I 2022-10-11 18:00:24] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 18:00:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 18:00:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 18:00:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 18:00:30] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 18:00:30] ...trap/engines/engine.py.321: eval: epoch 113 | batch 0/510 [I 2022-10-11 18:00:30] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:56:15 [I 2022-10-11 18:00:30] ...trap/engines/engine.py.326: process: 6.61775 | load: 5.70680 [I 2022-10-11 18:00:36] ...trap/engines/engine.py.321: eval: epoch 113 | batch 10/510 [I 2022-10-11 18:00:36] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:49 [I 2022-10-11 18:00:36] ...trap/engines/engine.py.326: process: 0.52831 | load: 0.00019 [I 2022-10-11 18:00:42] ...trap/engines/engine.py.321: eval: epoch 113 | batch 20/510 [I 2022-10-11 18:00:42] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:05:33 [I 2022-10-11 18:00:42] ...trap/engines/engine.py.326: process: 0.67739 | load: 0.00020 [I 2022-10-11 18:00:48] ...trap/engines/engine.py.321: eval: epoch 113 | batch 30/510 [I 2022-10-11 18:00:48] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:05:26 [I 2022-10-11 18:00:48] ...trap/engines/engine.py.326: process: 0.66746 | load: 0.00021 [I 2022-10-11 18:00:54] ...trap/engines/engine.py.321: eval: epoch 113 | batch 40/510 [I 2022-10-11 18:00:54] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:04:53 [I 2022-10-11 18:00:54] ...trap/engines/engine.py.326: process: 0.52795 | load: 0.00022 [I 2022-10-11 18:01:00] ...trap/engines/engine.py.321: eval: epoch 113 | batch 50/510 [I 2022-10-11 18:01:00] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:04:10 [I 2022-10-11 18:01:00] ...trap/engines/engine.py.326: process: 0.66150 | load: 0.00020 [I 2022-10-11 18:01:07] ...trap/engines/engine.py.321: eval: epoch 113 | batch 60/510 [I 2022-10-11 18:01:07] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:04:58 [I 2022-10-11 18:01:07] ...trap/engines/engine.py.326: process: 0.60900 | load: 0.00020 [I 2022-10-11 18:01:12] ...trap/engines/engine.py.321: eval: epoch 113 | batch 70/510 [I 2022-10-11 18:01:12] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:04:26 [I 2022-10-11 18:01:12] ...trap/engines/engine.py.326: process: 0.64028 | load: 0.00020 [I 2022-10-11 18:01:17] ...trap/engines/engine.py.321: eval: epoch 113 | batch 80/510 [I 2022-10-11 18:01:17] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:03:35 [I 2022-10-11 18:01:17] ...trap/engines/engine.py.326: process: 0.44190 | load: 0.00023 [I 2022-10-11 18:01:22] ...trap/engines/engine.py.321: eval: epoch 113 | batch 90/510 [I 2022-10-11 18:01:22] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:09 [I 2022-10-11 18:01:22] ...trap/engines/engine.py.326: process: 0.44519 | load: 0.00022 [I 2022-10-11 18:01:27] ...trap/engines/engine.py.321: eval: epoch 113 | batch 100/510 [I 2022-10-11 18:01:27] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:04 [I 2022-10-11 18:01:27] ...trap/engines/engine.py.326: process: 0.30978 | load: 0.00019 [I 2022-10-11 18:01:30] ...trap/engines/engine.py.321: eval: epoch 113 | batch 110/510 [I 2022-10-11 18:01:30] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:13 [I 2022-10-11 18:01:30] ...trap/engines/engine.py.326: process: 0.31133 | load: 0.00020 [I 2022-10-11 18:01:33] ...trap/engines/engine.py.321: eval: epoch 113 | batch 120/510 [I 2022-10-11 18:01:33] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:05 [I 2022-10-11 18:01:33] ...trap/engines/engine.py.326: process: 0.31995 | load: 0.00021 [I 2022-10-11 18:01:36] ...trap/engines/engine.py.321: eval: epoch 113 | batch 130/510 [I 2022-10-11 18:01:36] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:02 [I 2022-10-11 18:01:36] ...trap/engines/engine.py.326: process: 0.32450 | load: 0.00020 [I 2022-10-11 18:01:40] ...trap/engines/engine.py.321: eval: epoch 113 | batch 140/510 [I 2022-10-11 18:01:40] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:02:01 [I 2022-10-11 18:01:40] ...trap/engines/engine.py.326: process: 0.33680 | load: 0.00024 [I 2022-10-11 18:01:43] ...trap/engines/engine.py.321: eval: epoch 113 | batch 150/510 [I 2022-10-11 18:01:43] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:59 [I 2022-10-11 18:01:43] ...trap/engines/engine.py.326: process: 0.32475 | load: 0.00023 [I 2022-10-11 18:01:46] ...trap/engines/engine.py.321: eval: epoch 113 | batch 160/510 [I 2022-10-11 18:01:46] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:53 [I 2022-10-11 18:01:46] ...trap/engines/engine.py.326: process: 0.32362 | load: 0.00022 [I 2022-10-11 18:01:49] ...trap/engines/engine.py.321: eval: epoch 113 | batch 170/510 [I 2022-10-11 18:01:49] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:48 [I 2022-10-11 18:01:49] ...trap/engines/engine.py.326: process: 0.32150 | load: 0.00021 [I 2022-10-11 18:01:53] ...trap/engines/engine.py.321: eval: epoch 113 | batch 180/510 [I 2022-10-11 18:01:53] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:46 [I 2022-10-11 18:01:53] ...trap/engines/engine.py.326: process: 0.32299 | load: 0.00022 [I 2022-10-11 18:01:56] ...trap/engines/engine.py.321: eval: epoch 113 | batch 190/510 [I 2022-10-11 18:01:56] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:42 [I 2022-10-11 18:01:56] ...trap/engines/engine.py.326: process: 0.31920 | load: 0.00021 [I 2022-10-11 18:01:59] ...trap/engines/engine.py.321: eval: epoch 113 | batch 200/510 [I 2022-10-11 18:01:59] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:39 [I 2022-10-11 18:01:59] ...trap/engines/engine.py.326: process: 0.32222 | load: 0.00022 [I 2022-10-11 18:02:02] ...trap/engines/engine.py.321: eval: epoch 113 | batch 210/510 [I 2022-10-11 18:02:02] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:35 [I 2022-10-11 18:02:02] ...trap/engines/engine.py.326: process: 0.32178 | load: 0.00021 [I 2022-10-11 18:02:06] ...trap/engines/engine.py.321: eval: epoch 113 | batch 220/510 [I 2022-10-11 18:02:06] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:36 [I 2022-10-11 18:02:06] ...trap/engines/engine.py.326: process: 0.33389 | load: 0.00023 [I 2022-10-11 18:02:09] ...trap/engines/engine.py.321: eval: epoch 113 | batch 230/510 [I 2022-10-11 18:02:09] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:33 [I 2022-10-11 18:02:09] ...trap/engines/engine.py.326: process: 0.33516 | load: 0.00025 [I 2022-10-11 18:02:12] ...trap/engines/engine.py.321: eval: epoch 113 | batch 240/510 [I 2022-10-11 18:02:12] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:30 [I 2022-10-11 18:02:12] ...trap/engines/engine.py.326: process: 0.32982 | load: 0.00023 [I 2022-10-11 18:02:16] ...trap/engines/engine.py.321: eval: epoch 113 | batch 250/510 [I 2022-10-11 18:02:16] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:26 [I 2022-10-11 18:02:16] ...trap/engines/engine.py.326: process: 0.33074 | load: 0.00022 [I 2022-10-11 18:02:19] ...trap/engines/engine.py.321: eval: epoch 113 | batch 260/510 [I 2022-10-11 18:02:19] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:23 [I 2022-10-11 18:02:19] ...trap/engines/engine.py.326: process: 0.33436 | load: 0.00023 [I 2022-10-11 18:02:22] ...trap/engines/engine.py.321: eval: epoch 113 | batch 270/510 [I 2022-10-11 18:02:22] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:20 [I 2022-10-11 18:02:22] ...trap/engines/engine.py.326: process: 0.32951 | load: 0.00025 [I 2022-10-11 18:02:26] ...trap/engines/engine.py.321: eval: epoch 113 | batch 280/510 [I 2022-10-11 18:02:26] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:17 [I 2022-10-11 18:02:26] ...trap/engines/engine.py.326: process: 0.33452 | load: 0.00026 [I 2022-10-11 18:02:29] ...trap/engines/engine.py.321: eval: epoch 113 | batch 290/510 [I 2022-10-11 18:02:29] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:13 [I 2022-10-11 18:02:29] ...trap/engines/engine.py.326: process: 0.33229 | load: 0.00022 [I 2022-10-11 18:02:32] ...trap/engines/engine.py.321: eval: epoch 113 | batch 300/510 [I 2022-10-11 18:02:32] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:10 [I 2022-10-11 18:02:32] ...trap/engines/engine.py.326: process: 0.33801 | load: 0.00024 [I 2022-10-11 18:02:36] ...trap/engines/engine.py.321: eval: epoch 113 | batch 310/510 [I 2022-10-11 18:02:36] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:01:07 [I 2022-10-11 18:02:36] ...trap/engines/engine.py.326: process: 0.33988 | load: 0.00025 [I 2022-10-11 18:02:39] ...trap/engines/engine.py.321: eval: epoch 113 | batch 320/510 [I 2022-10-11 18:02:39] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:01:03 [I 2022-10-11 18:02:39] ...trap/engines/engine.py.326: process: 0.33317 | load: 0.00025 [I 2022-10-11 18:02:42] ...trap/engines/engine.py.321: eval: epoch 113 | batch 330/510 [I 2022-10-11 18:02:42] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:01:00 [I 2022-10-11 18:02:42] ...trap/engines/engine.py.326: process: 0.33858 | load: 0.00023 [I 2022-10-11 18:02:46] ...trap/engines/engine.py.321: eval: epoch 113 | batch 340/510 [I 2022-10-11 18:02:46] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:56 [I 2022-10-11 18:02:46] ...trap/engines/engine.py.326: process: 0.33318 | load: 0.00022 [I 2022-10-11 18:02:49] ...trap/engines/engine.py.321: eval: epoch 113 | batch 350/510 [I 2022-10-11 18:02:49] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:53 [I 2022-10-11 18:02:49] ...trap/engines/engine.py.326: process: 0.33804 | load: 0.00023 [I 2022-10-11 18:02:53] ...trap/engines/engine.py.321: eval: epoch 113 | batch 360/510 [I 2022-10-11 18:02:53] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:50 [I 2022-10-11 18:02:53] ...trap/engines/engine.py.326: process: 0.34117 | load: 0.00022 [I 2022-10-11 18:02:56] ...trap/engines/engine.py.321: eval: epoch 113 | batch 370/510 [I 2022-10-11 18:02:56] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:47 [I 2022-10-11 18:02:56] ...trap/engines/engine.py.326: process: 0.33584 | load: 0.00023 [I 2022-10-11 18:02:59] ...trap/engines/engine.py.321: eval: epoch 113 | batch 380/510 [I 2022-10-11 18:02:59] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:43 [I 2022-10-11 18:02:59] ...trap/engines/engine.py.326: process: 0.33904 | load: 0.00022 [I 2022-10-11 18:03:03] ...trap/engines/engine.py.321: eval: epoch 113 | batch 390/510 [I 2022-10-11 18:03:03] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:40 [I 2022-10-11 18:03:03] ...trap/engines/engine.py.326: process: 0.32951 | load: 0.00025 [I 2022-10-11 18:03:06] ...trap/engines/engine.py.321: eval: epoch 113 | batch 400/510 [I 2022-10-11 18:03:06] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:36 [I 2022-10-11 18:03:06] ...trap/engines/engine.py.326: process: 0.33284 | load: 0.00023 [I 2022-10-11 18:03:09] ...trap/engines/engine.py.321: eval: epoch 113 | batch 410/510 [I 2022-10-11 18:03:09] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:33 [I 2022-10-11 18:03:09] ...trap/engines/engine.py.326: process: 0.33361 | load: 0.00023 [I 2022-10-11 18:03:13] ...trap/engines/engine.py.321: eval: epoch 113 | batch 420/510 [I 2022-10-11 18:03:13] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:30 [I 2022-10-11 18:03:13] ...trap/engines/engine.py.326: process: 0.34006 | load: 0.00023 [I 2022-10-11 18:03:16] ...trap/engines/engine.py.321: eval: epoch 113 | batch 430/510 [I 2022-10-11 18:03:16] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:27 [I 2022-10-11 18:03:16] ...trap/engines/engine.py.326: process: 0.33617 | load: 0.00024 [I 2022-10-11 18:03:19] ...trap/engines/engine.py.321: eval: epoch 113 | batch 440/510 [I 2022-10-11 18:03:19] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:23 [I 2022-10-11 18:03:19] ...trap/engines/engine.py.326: process: 0.33555 | load: 0.00023 [I 2022-10-11 18:03:23] ...trap/engines/engine.py.321: eval: epoch 113 | batch 450/510 [I 2022-10-11 18:03:23] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:20 [I 2022-10-11 18:03:23] ...trap/engines/engine.py.326: process: 0.34512 | load: 0.00022 [I 2022-10-11 18:03:26] ...trap/engines/engine.py.321: eval: epoch 113 | batch 460/510 [I 2022-10-11 18:03:26] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:17 [I 2022-10-11 18:03:26] ...trap/engines/engine.py.326: process: 0.34896 | load: 0.00029 [I 2022-10-11 18:03:30] ...trap/engines/engine.py.321: eval: epoch 113 | batch 470/510 [I 2022-10-11 18:03:30] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:13 [I 2022-10-11 18:03:30] ...trap/engines/engine.py.326: process: 0.33107 | load: 0.00023 [I 2022-10-11 18:03:33] ...trap/engines/engine.py.321: eval: epoch 113 | batch 480/510 [I 2022-10-11 18:03:33] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:10 [I 2022-10-11 18:03:33] ...trap/engines/engine.py.326: process: 0.33434 | load: 0.00023 [I 2022-10-11 18:03:36] ...trap/engines/engine.py.321: eval: epoch 113 | batch 490/510 [I 2022-10-11 18:03:36] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:06 [I 2022-10-11 18:03:36] ...trap/engines/engine.py.326: process: 0.33582 | load: 0.00023 [I 2022-10-11 18:03:40] ...trap/engines/engine.py.321: eval: epoch 113 | batch 500/510 [I 2022-10-11 18:03:40] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:03 [I 2022-10-11 18:03:40] ...trap/engines/engine.py.326: process: 0.33488 | load: 0.00024 [I 2022-10-11 18:03:43] ...trap/engines/engine.py.321: eval: epoch 113 | batch 510/510 [I 2022-10-11 18:03:43] ...trap/engines/engine.py.325: elapsed: 0:03:19 | left: 0:00:00 [I 2022-10-11 18:03:43] ...trap/engines/engine.py.326: process: 0.32881 | load: 0.00019 [S 2022-10-11 18:03:44] ...trap/engines/engine.py.132: eval_epoch.epoch: 113 [I 2022-10-11 18:03:44] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.747700 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907600 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932100 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.739700 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908400 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933500 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007226 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008499 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007582 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007184 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008027 [S 2022-10-11 18:04:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006087 [I 2022-10-11 18:04:27] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 18:04:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 18:04:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 18:04:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 18:04:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 18:04:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 18:04:34] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 18:04:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 18:04:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 114 [I 2022-10-11 18:04:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 18:04:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 0/2304 [I 2022-10-11 18:04:43] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:57:39 [I 2022-10-11 18:04:43] ...trap/engines/engine.py.229: process: 3.06398 | load: 2.15990 [I 2022-10-11 18:04:43] ...trap/engines/engine.py.230: loss: 0.08046 [I 2022-10-11 18:04:43] ...optimizers/trijoint.py.151: total_norm: 17.513309 [I 2022-10-11 18:04:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 10/2304 [I 2022-10-11 18:04:51] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:18 [I 2022-10-11 18:04:51] ...trap/engines/engine.py.229: process: 0.84209 | load: 0.00623 [I 2022-10-11 18:04:51] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-11 18:04:51] ...optimizers/trijoint.py.151: total_norm: 12.645634 [I 2022-10-11 18:05:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 20/2304 [I 2022-10-11 18:05:00] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:14 [I 2022-10-11 18:05:00] ...trap/engines/engine.py.229: process: 0.86818 | load: 0.00450 [I 2022-10-11 18:05:00] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-11 18:05:00] ...optimizers/trijoint.py.151: total_norm: 8.071008 [I 2022-10-11 18:05:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 30/2304 [I 2022-10-11 18:05:08] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:06 [I 2022-10-11 18:05:08] ...trap/engines/engine.py.229: process: 0.84187 | load: 0.00412 [I 2022-10-11 18:05:08] ...trap/engines/engine.py.230: loss: 0.09256 [I 2022-10-11 18:05:08] ...optimizers/trijoint.py.151: total_norm: 17.645781 [I 2022-10-11 18:05:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 40/2304 [I 2022-10-11 18:05:17] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:03 [I 2022-10-11 18:05:17] ...trap/engines/engine.py.229: process: 0.85183 | load: 0.00516 [I 2022-10-11 18:05:17] ...trap/engines/engine.py.230: loss: 0.09752 [I 2022-10-11 18:05:17] ...optimizers/trijoint.py.151: total_norm: 29.764406 [I 2022-10-11 18:05:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 50/2304 [I 2022-10-11 18:05:25] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:52 [I 2022-10-11 18:05:25] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00372 [I 2022-10-11 18:05:25] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-11 18:05:25] ...optimizers/trijoint.py.151: total_norm: 21.533554 [I 2022-10-11 18:05:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 60/2304 [I 2022-10-11 18:05:34] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:38 [I 2022-10-11 18:05:34] ...trap/engines/engine.py.229: process: 0.84155 | load: 0.00357 [I 2022-10-11 18:05:34] ...trap/engines/engine.py.230: loss: 0.08533 [I 2022-10-11 18:05:34] ...optimizers/trijoint.py.151: total_norm: 17.361269 [I 2022-10-11 18:05:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 70/2304 [I 2022-10-11 18:05:42] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:26 [I 2022-10-11 18:05:42] ...trap/engines/engine.py.229: process: 0.83894 | load: 0.00351 [I 2022-10-11 18:05:42] ...trap/engines/engine.py.230: loss: 0.13655 [I 2022-10-11 18:05:42] ...optimizers/trijoint.py.151: total_norm: 15.677318 [I 2022-10-11 18:05:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 80/2304 [I 2022-10-11 18:05:51] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:19 [I 2022-10-11 18:05:51] ...trap/engines/engine.py.229: process: 0.83211 | load: 0.00371 [I 2022-10-11 18:05:51] ...trap/engines/engine.py.230: loss: 0.09915 [I 2022-10-11 18:05:51] ...optimizers/trijoint.py.151: total_norm: 75.129829 [I 2022-10-11 18:05:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 90/2304 [I 2022-10-11 18:05:59] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:31:11 [I 2022-10-11 18:05:59] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00320 [I 2022-10-11 18:05:59] ...trap/engines/engine.py.230: loss: 0.12887 [I 2022-10-11 18:05:59] ...optimizers/trijoint.py.151: total_norm: 12.611409 [I 2022-10-11 18:06:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 100/2304 [I 2022-10-11 18:06:08] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:19 [I 2022-10-11 18:06:08] ...trap/engines/engine.py.229: process: 0.85049 | load: 0.00353 [I 2022-10-11 18:06:08] ...trap/engines/engine.py.230: loss: 0.15709 [I 2022-10-11 18:06:08] ...optimizers/trijoint.py.151: total_norm: 11.485553 [I 2022-10-11 18:06:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 110/2304 [I 2022-10-11 18:06:16] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:31:08 [I 2022-10-11 18:06:16] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00347 [I 2022-10-11 18:06:16] ...trap/engines/engine.py.230: loss: 0.08522 [I 2022-10-11 18:06:16] ...optimizers/trijoint.py.151: total_norm: 9.950235 [I 2022-10-11 18:06:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 120/2304 [I 2022-10-11 18:06:25] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:17 [I 2022-10-11 18:06:25] ...trap/engines/engine.py.229: process: 0.86533 | load: 0.00357 [I 2022-10-11 18:06:25] ...trap/engines/engine.py.230: loss: 0.09169 [I 2022-10-11 18:06:25] ...optimizers/trijoint.py.151: total_norm: 26.005322 [I 2022-10-11 18:06:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 130/2304 [I 2022-10-11 18:06:34] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:31:03 [I 2022-10-11 18:06:34] ...trap/engines/engine.py.229: process: 0.86329 | load: 0.00323 [I 2022-10-11 18:06:34] ...trap/engines/engine.py.230: loss: 0.10095 [I 2022-10-11 18:06:34] ...optimizers/trijoint.py.151: total_norm: 9.295981 [I 2022-10-11 18:06:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 140/2304 [I 2022-10-11 18:06:42] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:31:25 [I 2022-10-11 18:06:42] ...trap/engines/engine.py.229: process: 0.88660 | load: 0.00394 [I 2022-10-11 18:06:42] ...trap/engines/engine.py.230: loss: 0.10893 [I 2022-10-11 18:06:42] ...optimizers/trijoint.py.151: total_norm: 12.281939 [I 2022-10-11 18:06:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 150/2304 [I 2022-10-11 18:06:51] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:31:40 [I 2022-10-11 18:06:51] ...trap/engines/engine.py.229: process: 0.87848 | load: 0.00414 [I 2022-10-11 18:06:51] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-11 18:06:51] ...optimizers/trijoint.py.151: total_norm: 27.379894 [I 2022-10-11 18:07:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 160/2304 [I 2022-10-11 18:07:00] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:31:29 [I 2022-10-11 18:07:00] ...trap/engines/engine.py.229: process: 0.88488 | load: 0.00432 [I 2022-10-11 18:07:00] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-11 18:07:00] ...optimizers/trijoint.py.151: total_norm: 9.161013 [I 2022-10-11 18:07:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 170/2304 [I 2022-10-11 18:07:09] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:31:25 [I 2022-10-11 18:07:09] ...trap/engines/engine.py.229: process: 0.88594 | load: 0.00425 [I 2022-10-11 18:07:09] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-11 18:07:09] ...optimizers/trijoint.py.151: total_norm: 27.883051 [I 2022-10-11 18:07:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 180/2304 [I 2022-10-11 18:07:18] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:31:33 [I 2022-10-11 18:07:18] ...trap/engines/engine.py.229: process: 0.89753 | load: 0.00392 [I 2022-10-11 18:07:18] ...trap/engines/engine.py.230: loss: 0.08508 [I 2022-10-11 18:07:18] ...optimizers/trijoint.py.151: total_norm: 14.289027 [I 2022-10-11 18:07:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 190/2304 [I 2022-10-11 18:07:27] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:31:06 [I 2022-10-11 18:07:27] ...trap/engines/engine.py.229: process: 0.88336 | load: 0.00394 [I 2022-10-11 18:07:27] ...trap/engines/engine.py.230: loss: 0.12282 [I 2022-10-11 18:07:27] ...optimizers/trijoint.py.151: total_norm: 16.785372 [I 2022-10-11 18:07:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 200/2304 [I 2022-10-11 18:07:36] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:56 [I 2022-10-11 18:07:36] ...trap/engines/engine.py.229: process: 0.87103 | load: 0.00411 [I 2022-10-11 18:07:36] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-11 18:07:36] ...optimizers/trijoint.py.151: total_norm: 12.229763 [I 2022-10-11 18:07:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 210/2304 [I 2022-10-11 18:07:44] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:30:52 [I 2022-10-11 18:07:44] ...trap/engines/engine.py.229: process: 0.87914 | load: 0.00431 [I 2022-10-11 18:07:44] ...trap/engines/engine.py.230: loss: 0.08370 [I 2022-10-11 18:07:44] ...optimizers/trijoint.py.151: total_norm: 40.597015 [I 2022-10-11 18:07:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 220/2304 [I 2022-10-11 18:07:53] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:30:51 [I 2022-10-11 18:07:53] ...trap/engines/engine.py.229: process: 0.88419 | load: 0.00395 [I 2022-10-11 18:07:53] ...trap/engines/engine.py.230: loss: 0.24083 [I 2022-10-11 18:07:53] ...optimizers/trijoint.py.151: total_norm: 25.115728 [I 2022-10-11 18:08:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 230/2304 [I 2022-10-11 18:08:02] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:30:29 [I 2022-10-11 18:08:02] ...trap/engines/engine.py.229: process: 0.87561 | load: 0.00397 [I 2022-10-11 18:08:02] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-11 18:08:02] ...optimizers/trijoint.py.151: total_norm: 17.490389 [I 2022-10-11 18:08:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 240/2304 [I 2022-10-11 18:08:11] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:30:24 [I 2022-10-11 18:08:11] ...trap/engines/engine.py.229: process: 0.88052 | load: 0.00397 [I 2022-10-11 18:08:11] ...trap/engines/engine.py.230: loss: 0.13144 [I 2022-10-11 18:08:11] ...optimizers/trijoint.py.151: total_norm: 31.763723 [I 2022-10-11 18:08:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 250/2304 [I 2022-10-11 18:08:20] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:30:15 [I 2022-10-11 18:08:20] ...trap/engines/engine.py.229: process: 0.87894 | load: 0.00394 [I 2022-10-11 18:08:20] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-11 18:08:20] ...optimizers/trijoint.py.151: total_norm: 13.042720 [I 2022-10-11 18:08:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 260/2304 [I 2022-10-11 18:08:29] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:30:08 [I 2022-10-11 18:08:29] ...trap/engines/engine.py.229: process: 0.87593 | load: 0.00417 [I 2022-10-11 18:08:29] ...trap/engines/engine.py.230: loss: 0.07380 [I 2022-10-11 18:08:29] ...optimizers/trijoint.py.151: total_norm: 14.961956 [I 2022-10-11 18:08:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 270/2304 [I 2022-10-11 18:08:38] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:30:02 [I 2022-10-11 18:08:38] ...trap/engines/engine.py.229: process: 0.89382 | load: 0.00398 [I 2022-10-11 18:08:38] ...trap/engines/engine.py.230: loss: 0.11018 [I 2022-10-11 18:08:38] ...optimizers/trijoint.py.151: total_norm: 21.801754 [I 2022-10-11 18:08:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 280/2304 [I 2022-10-11 18:08:46] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:30:02 [I 2022-10-11 18:08:46] ...trap/engines/engine.py.229: process: 0.90203 | load: 0.00422 [I 2022-10-11 18:08:46] ...trap/engines/engine.py.230: loss: 0.09589 [I 2022-10-11 18:08:46] ...optimizers/trijoint.py.151: total_norm: 9.186941 [I 2022-10-11 18:08:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 290/2304 [I 2022-10-11 18:08:55] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:29:34 [I 2022-10-11 18:08:55] ...trap/engines/engine.py.229: process: 0.87247 | load: 0.00396 [I 2022-10-11 18:08:55] ...trap/engines/engine.py.230: loss: 0.09901 [I 2022-10-11 18:08:55] ...optimizers/trijoint.py.151: total_norm: 8.733085 [I 2022-10-11 18:09:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 300/2304 [I 2022-10-11 18:09:04] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:29:39 [I 2022-10-11 18:09:04] ...trap/engines/engine.py.229: process: 0.88440 | load: 0.00401 [I 2022-10-11 18:09:04] ...trap/engines/engine.py.230: loss: 0.11280 [I 2022-10-11 18:09:04] ...optimizers/trijoint.py.151: total_norm: 19.755505 [I 2022-10-11 18:09:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 310/2304 [I 2022-10-11 18:09:13] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:29:32 [I 2022-10-11 18:09:13] ...trap/engines/engine.py.229: process: 0.89494 | load: 0.00393 [I 2022-10-11 18:09:13] ...trap/engines/engine.py.230: loss: 0.08375 [I 2022-10-11 18:09:13] ...optimizers/trijoint.py.151: total_norm: 15.212292 [I 2022-10-11 18:09:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 320/2304 [I 2022-10-11 18:09:22] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:29:18 [I 2022-10-11 18:09:22] ...trap/engines/engine.py.229: process: 0.88196 | load: 0.00411 [I 2022-10-11 18:09:22] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-11 18:09:22] ...optimizers/trijoint.py.151: total_norm: 11.772148 [I 2022-10-11 18:09:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 330/2304 [I 2022-10-11 18:09:31] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:29:14 [I 2022-10-11 18:09:31] ...trap/engines/engine.py.229: process: 0.89822 | load: 0.00396 [I 2022-10-11 18:09:31] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 18:09:31] ...optimizers/trijoint.py.151: total_norm: 18.611286 [I 2022-10-11 18:09:40] ...trap/engines/engine.py.224: train: epoch 114 | batch 340/2304 [I 2022-10-11 18:09:40] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:28:50 [I 2022-10-11 18:09:40] ...trap/engines/engine.py.229: process: 0.88391 | load: 0.00426 [I 2022-10-11 18:09:40] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-11 18:09:40] ...optimizers/trijoint.py.151: total_norm: 18.955862 [I 2022-10-11 18:09:49] ...trap/engines/engine.py.224: train: epoch 114 | batch 350/2304 [I 2022-10-11 18:09:49] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:28:49 [I 2022-10-11 18:09:49] ...trap/engines/engine.py.229: process: 0.88364 | load: 0.00391 [I 2022-10-11 18:09:49] ...trap/engines/engine.py.230: loss: 0.08739 [I 2022-10-11 18:09:49] ...optimizers/trijoint.py.151: total_norm: 17.378431 [I 2022-10-11 18:09:57] ...trap/engines/engine.py.224: train: epoch 114 | batch 360/2304 [I 2022-10-11 18:09:57] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:28:40 [I 2022-10-11 18:09:57] ...trap/engines/engine.py.229: process: 0.89348 | load: 0.00411 [I 2022-10-11 18:09:57] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-11 18:09:57] ...optimizers/trijoint.py.151: total_norm: 18.409737 [I 2022-10-11 18:10:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 370/2304 [I 2022-10-11 18:10:06] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:28:37 [I 2022-10-11 18:10:06] ...trap/engines/engine.py.229: process: 0.89888 | load: 0.00411 [I 2022-10-11 18:10:06] ...trap/engines/engine.py.230: loss: 0.09596 [I 2022-10-11 18:10:06] ...optimizers/trijoint.py.151: total_norm: 10.359383 [I 2022-10-11 18:10:15] ...trap/engines/engine.py.224: train: epoch 114 | batch 380/2304 [I 2022-10-11 18:10:15] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:28:02 [I 2022-10-11 18:10:15] ...trap/engines/engine.py.229: process: 0.86171 | load: 0.00380 [I 2022-10-11 18:10:15] ...trap/engines/engine.py.230: loss: 0.07190 [I 2022-10-11 18:10:15] ...optimizers/trijoint.py.151: total_norm: 10.896143 [I 2022-10-11 18:10:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 390/2304 [I 2022-10-11 18:10:24] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:28:04 [I 2022-10-11 18:10:24] ...trap/engines/engine.py.229: process: 0.87483 | load: 0.00394 [I 2022-10-11 18:10:24] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-11 18:10:24] ...optimizers/trijoint.py.151: total_norm: 24.761074 [I 2022-10-11 18:10:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 400/2304 [I 2022-10-11 18:10:33] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:27:36 [I 2022-10-11 18:10:33] ...trap/engines/engine.py.229: process: 0.86584 | load: 0.00376 [I 2022-10-11 18:10:33] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-11 18:10:33] ...optimizers/trijoint.py.151: total_norm: 16.107269 [I 2022-10-11 18:10:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 410/2304 [I 2022-10-11 18:10:41] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:27:37 [I 2022-10-11 18:10:41] ...trap/engines/engine.py.229: process: 0.87715 | load: 0.00381 [I 2022-10-11 18:10:41] ...trap/engines/engine.py.230: loss: 0.07903 [I 2022-10-11 18:10:41] ...optimizers/trijoint.py.151: total_norm: 11.808803 [I 2022-10-11 18:10:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 420/2304 [I 2022-10-11 18:10:50] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:27:34 [I 2022-10-11 18:10:50] ...trap/engines/engine.py.229: process: 0.88580 | load: 0.00374 [I 2022-10-11 18:10:50] ...trap/engines/engine.py.230: loss: 0.10924 [I 2022-10-11 18:10:50] ...optimizers/trijoint.py.151: total_norm: 14.280021 [I 2022-10-11 18:10:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 430/2304 [I 2022-10-11 18:10:59] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:27:25 [I 2022-10-11 18:10:59] ...trap/engines/engine.py.229: process: 0.86832 | load: 0.00374 [I 2022-10-11 18:10:59] ...trap/engines/engine.py.230: loss: 0.18393 [I 2022-10-11 18:10:59] ...optimizers/trijoint.py.151: total_norm: 14.110106 [I 2022-10-11 18:11:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 440/2304 [I 2022-10-11 18:11:08] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:27:18 [I 2022-10-11 18:11:08] ...trap/engines/engine.py.229: process: 0.88339 | load: 0.00372 [I 2022-10-11 18:11:08] ...trap/engines/engine.py.230: loss: 0.11916 [I 2022-10-11 18:11:08] ...optimizers/trijoint.py.151: total_norm: 16.408489 [I 2022-10-11 18:11:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 450/2304 [I 2022-10-11 18:11:17] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:27:06 [I 2022-10-11 18:11:17] ...trap/engines/engine.py.229: process: 0.88635 | load: 0.00376 [I 2022-10-11 18:11:17] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-11 18:11:17] ...optimizers/trijoint.py.151: total_norm: 12.684912 [I 2022-10-11 18:11:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 460/2304 [I 2022-10-11 18:11:25] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:26:53 [I 2022-10-11 18:11:25] ...trap/engines/engine.py.229: process: 0.88071 | load: 0.00375 [I 2022-10-11 18:11:25] ...trap/engines/engine.py.230: loss: 0.14162 [I 2022-10-11 18:11:25] ...optimizers/trijoint.py.151: total_norm: 20.490583 [I 2022-10-11 18:11:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 470/2304 [I 2022-10-11 18:11:34] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:26:47 [I 2022-10-11 18:11:34] ...trap/engines/engine.py.229: process: 0.87841 | load: 0.00380 [I 2022-10-11 18:11:34] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-11 18:11:34] ...optimizers/trijoint.py.151: total_norm: 15.250585 [I 2022-10-11 18:11:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 480/2304 [I 2022-10-11 18:11:43] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:26:29 [I 2022-10-11 18:11:43] ...trap/engines/engine.py.229: process: 0.87075 | load: 0.00379 [I 2022-10-11 18:11:43] ...trap/engines/engine.py.230: loss: 0.10567 [I 2022-10-11 18:11:43] ...optimizers/trijoint.py.151: total_norm: 18.123743 [I 2022-10-11 18:11:52] ...trap/engines/engine.py.224: train: epoch 114 | batch 490/2304 [I 2022-10-11 18:11:52] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:26:20 [I 2022-10-11 18:11:52] ...trap/engines/engine.py.229: process: 0.87284 | load: 0.00380 [I 2022-10-11 18:11:52] ...trap/engines/engine.py.230: loss: 0.10775 [I 2022-10-11 18:11:52] ...optimizers/trijoint.py.151: total_norm: 16.072359 [I 2022-10-11 18:12:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 500/2304 [I 2022-10-11 18:12:00] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:26:19 [I 2022-10-11 18:12:00] ...trap/engines/engine.py.229: process: 0.86371 | load: 0.00383 [I 2022-10-11 18:12:00] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-11 18:12:00] ...optimizers/trijoint.py.151: total_norm: 19.686722 [I 2022-10-11 18:12:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 510/2304 [I 2022-10-11 18:12:09] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:26:04 [I 2022-10-11 18:12:09] ...trap/engines/engine.py.229: process: 0.87583 | load: 0.00376 [I 2022-10-11 18:12:09] ...trap/engines/engine.py.230: loss: 0.08589 [I 2022-10-11 18:12:09] ...optimizers/trijoint.py.151: total_norm: 11.636071 [I 2022-10-11 18:12:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 520/2304 [I 2022-10-11 18:12:18] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:25:59 [I 2022-10-11 18:12:18] ...trap/engines/engine.py.229: process: 0.86755 | load: 0.00377 [I 2022-10-11 18:12:18] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-11 18:12:18] ...optimizers/trijoint.py.151: total_norm: 19.866913 [I 2022-10-11 18:12:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 530/2304 [I 2022-10-11 18:12:27] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:25:52 [I 2022-10-11 18:12:27] ...trap/engines/engine.py.229: process: 0.87908 | load: 0.00380 [I 2022-10-11 18:12:27] ...trap/engines/engine.py.230: loss: 0.07365 [I 2022-10-11 18:12:27] ...optimizers/trijoint.py.151: total_norm: 8.316107 [I 2022-10-11 18:12:35] ...trap/engines/engine.py.224: train: epoch 114 | batch 540/2304 [I 2022-10-11 18:12:35] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:25:43 [I 2022-10-11 18:12:35] ...trap/engines/engine.py.229: process: 0.88362 | load: 0.00378 [I 2022-10-11 18:12:35] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-11 18:12:35] ...optimizers/trijoint.py.151: total_norm: 19.311405 [I 2022-10-11 18:12:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 550/2304 [I 2022-10-11 18:12:44] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:25:30 [I 2022-10-11 18:12:44] ...trap/engines/engine.py.229: process: 0.87381 | load: 0.00381 [I 2022-10-11 18:12:44] ...trap/engines/engine.py.230: loss: 0.12766 [I 2022-10-11 18:12:44] ...optimizers/trijoint.py.151: total_norm: 16.419640 [I 2022-10-11 18:12:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 560/2304 [I 2022-10-11 18:12:53] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:25:28 [I 2022-10-11 18:12:53] ...trap/engines/engine.py.229: process: 0.87790 | load: 0.00381 [I 2022-10-11 18:12:53] ...trap/engines/engine.py.230: loss: 0.15604 [I 2022-10-11 18:12:53] ...optimizers/trijoint.py.151: total_norm: 32.703144 [I 2022-10-11 18:13:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 570/2304 [I 2022-10-11 18:13:02] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:25:19 [I 2022-10-11 18:13:02] ...trap/engines/engine.py.229: process: 0.88547 | load: 0.00376 [I 2022-10-11 18:13:02] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-11 18:13:02] ...optimizers/trijoint.py.151: total_norm: 14.926723 [I 2022-10-11 18:13:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 580/2304 [I 2022-10-11 18:13:11] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:25:15 [I 2022-10-11 18:13:11] ...trap/engines/engine.py.229: process: 0.87287 | load: 0.00392 [I 2022-10-11 18:13:11] ...trap/engines/engine.py.230: loss: 0.09031 [I 2022-10-11 18:13:11] ...optimizers/trijoint.py.151: total_norm: 11.900790 [I 2022-10-11 18:13:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 590/2304 [I 2022-10-11 18:13:19] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:24:41 [I 2022-10-11 18:13:19] ...trap/engines/engine.py.229: process: 0.85896 | load: 0.00308 [I 2022-10-11 18:13:19] ...trap/engines/engine.py.230: loss: 0.10827 [I 2022-10-11 18:13:19] ...optimizers/trijoint.py.151: total_norm: 16.608562 [I 2022-10-11 18:13:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 600/2304 [I 2022-10-11 18:13:28] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:24:09 [I 2022-10-11 18:13:28] ...trap/engines/engine.py.229: process: 0.85330 | load: 0.00311 [I 2022-10-11 18:13:28] ...trap/engines/engine.py.230: loss: 0.07258 [I 2022-10-11 18:13:28] ...optimizers/trijoint.py.151: total_norm: 10.083795 [I 2022-10-11 18:13:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 610/2304 [I 2022-10-11 18:13:36] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:24:13 [I 2022-10-11 18:13:36] ...trap/engines/engine.py.229: process: 0.86101 | load: 0.00318 [I 2022-10-11 18:13:36] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-11 18:13:36] ...optimizers/trijoint.py.151: total_norm: 14.676237 [I 2022-10-11 18:13:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 620/2304 [I 2022-10-11 18:13:45] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:24:02 [I 2022-10-11 18:13:45] ...trap/engines/engine.py.229: process: 0.85381 | load: 0.00312 [I 2022-10-11 18:13:45] ...trap/engines/engine.py.230: loss: 0.19824 [I 2022-10-11 18:13:45] ...optimizers/trijoint.py.151: total_norm: 34.614353 [I 2022-10-11 18:13:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 630/2304 [I 2022-10-11 18:13:54] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:23:51 [I 2022-10-11 18:13:54] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00311 [I 2022-10-11 18:13:54] ...trap/engines/engine.py.230: loss: 0.11089 [I 2022-10-11 18:13:54] ...optimizers/trijoint.py.151: total_norm: 18.117727 [I 2022-10-11 18:14:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 640/2304 [I 2022-10-11 18:14:02] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:40 [I 2022-10-11 18:14:02] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00312 [I 2022-10-11 18:14:02] ...trap/engines/engine.py.230: loss: 0.07020 [I 2022-10-11 18:14:02] ...optimizers/trijoint.py.151: total_norm: 9.273823 [I 2022-10-11 18:14:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 650/2304 [I 2022-10-11 18:14:11] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:33 [I 2022-10-11 18:14:11] ...trap/engines/engine.py.229: process: 0.85893 | load: 0.00317 [I 2022-10-11 18:14:11] ...trap/engines/engine.py.230: loss: 0.13460 [I 2022-10-11 18:14:11] ...optimizers/trijoint.py.151: total_norm: 20.849146 [I 2022-10-11 18:14:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 660/2304 [I 2022-10-11 18:14:19] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:23:32 [I 2022-10-11 18:14:19] ...trap/engines/engine.py.229: process: 0.87000 | load: 0.00315 [I 2022-10-11 18:14:19] ...trap/engines/engine.py.230: loss: 0.14048 [I 2022-10-11 18:14:19] ...optimizers/trijoint.py.151: total_norm: 10.372377 [I 2022-10-11 18:14:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 670/2304 [I 2022-10-11 18:14:47] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:37:35 [I 2022-10-11 18:14:47] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00315 [I 2022-10-11 18:14:47] ...trap/engines/engine.py.230: loss: 0.11261 [I 2022-10-11 18:14:47] ...optimizers/trijoint.py.151: total_norm: 13.128492 [I 2022-10-11 18:14:56] ...trap/engines/engine.py.224: train: epoch 114 | batch 680/2304 [I 2022-10-11 18:14:56] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:24:53 [I 2022-10-11 18:14:56] ...trap/engines/engine.py.229: process: 0.86030 | load: 0.00317 [I 2022-10-11 18:14:56] ...trap/engines/engine.py.230: loss: 0.13296 [I 2022-10-11 18:14:56] ...optimizers/trijoint.py.151: total_norm: 11.243501 [I 2022-10-11 18:15:05] ...trap/engines/engine.py.224: train: epoch 114 | batch 690/2304 [I 2022-10-11 18:15:05] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:23:22 [I 2022-10-11 18:15:05] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00311 [I 2022-10-11 18:15:05] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-11 18:15:05] ...optimizers/trijoint.py.151: total_norm: 28.249998 [I 2022-10-11 18:15:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 700/2304 [I 2022-10-11 18:15:13] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:23:16 [I 2022-10-11 18:15:13] ...trap/engines/engine.py.229: process: 0.87155 | load: 0.00312 [I 2022-10-11 18:15:13] ...trap/engines/engine.py.230: loss: 0.19185 [I 2022-10-11 18:15:13] ...optimizers/trijoint.py.151: total_norm: 14.594479 [I 2022-10-11 18:15:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 710/2304 [I 2022-10-11 18:15:22] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:22:55 [I 2022-10-11 18:15:22] ...trap/engines/engine.py.229: process: 0.85332 | load: 0.00314 [I 2022-10-11 18:15:22] ...trap/engines/engine.py.230: loss: 0.10381 [I 2022-10-11 18:15:22] ...optimizers/trijoint.py.151: total_norm: 12.373736 [I 2022-10-11 18:15:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 720/2304 [I 2022-10-11 18:15:31] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:47 [I 2022-10-11 18:15:31] ...trap/engines/engine.py.229: process: 0.86338 | load: 0.00311 [I 2022-10-11 18:15:31] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-11 18:15:31] ...optimizers/trijoint.py.151: total_norm: 13.580071 [I 2022-10-11 18:15:39] ...trap/engines/engine.py.224: train: epoch 114 | batch 730/2304 [I 2022-10-11 18:15:39] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:22:32 [I 2022-10-11 18:15:39] ...trap/engines/engine.py.229: process: 0.86916 | load: 0.00310 [I 2022-10-11 18:15:39] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-11 18:15:39] ...optimizers/trijoint.py.151: total_norm: 9.072445 [I 2022-10-11 18:15:48] ...trap/engines/engine.py.224: train: epoch 114 | batch 740/2304 [I 2022-10-11 18:15:48] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:15 [I 2022-10-11 18:15:48] ...trap/engines/engine.py.229: process: 0.84317 | load: 0.00319 [I 2022-10-11 18:15:48] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-11 18:15:48] ...optimizers/trijoint.py.151: total_norm: 17.343748 [I 2022-10-11 18:15:57] ...trap/engines/engine.py.224: train: epoch 114 | batch 750/2304 [I 2022-10-11 18:15:57] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:22:55 [I 2022-10-11 18:15:57] ...trap/engines/engine.py.229: process: 0.89564 | load: 0.00394 [I 2022-10-11 18:15:57] ...trap/engines/engine.py.230: loss: 0.09494 [I 2022-10-11 18:15:57] ...optimizers/trijoint.py.151: total_norm: 23.398336 [I 2022-10-11 18:16:05] ...trap/engines/engine.py.224: train: epoch 114 | batch 760/2304 [I 2022-10-11 18:16:05] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:22:43 [I 2022-10-11 18:16:05] ...trap/engines/engine.py.229: process: 0.87006 | load: 0.00397 [I 2022-10-11 18:16:05] ...trap/engines/engine.py.230: loss: 0.18573 [I 2022-10-11 18:16:05] ...optimizers/trijoint.py.151: total_norm: 21.777168 [I 2022-10-11 18:16:14] ...trap/engines/engine.py.224: train: epoch 114 | batch 770/2304 [I 2022-10-11 18:16:14] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:22:41 [I 2022-10-11 18:16:14] ...trap/engines/engine.py.229: process: 0.88874 | load: 0.00396 [I 2022-10-11 18:16:14] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-11 18:16:14] ...optimizers/trijoint.py.151: total_norm: 14.469632 [I 2022-10-11 18:16:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 780/2304 [I 2022-10-11 18:16:23] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:22:01 [I 2022-10-11 18:16:23] ...trap/engines/engine.py.229: process: 0.86300 | load: 0.00312 [I 2022-10-11 18:16:23] ...trap/engines/engine.py.230: loss: 0.08140 [I 2022-10-11 18:16:23] ...optimizers/trijoint.py.151: total_norm: 14.440866 [I 2022-10-11 18:16:32] ...trap/engines/engine.py.224: train: epoch 114 | batch 790/2304 [I 2022-10-11 18:16:32] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:22:21 [I 2022-10-11 18:16:32] ...trap/engines/engine.py.229: process: 0.89861 | load: 0.00396 [I 2022-10-11 18:16:32] ...trap/engines/engine.py.230: loss: 0.08116 [I 2022-10-11 18:16:32] ...optimizers/trijoint.py.151: total_norm: 12.913045 [I 2022-10-11 18:16:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 800/2304 [I 2022-10-11 18:16:41] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:22:09 [I 2022-10-11 18:16:41] ...trap/engines/engine.py.229: process: 0.88423 | load: 0.00394 [I 2022-10-11 18:16:41] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-11 18:16:41] ...optimizers/trijoint.py.151: total_norm: 31.973856 [I 2022-10-11 18:16:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 810/2304 [I 2022-10-11 18:16:50] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:22:09 [I 2022-10-11 18:16:50] ...trap/engines/engine.py.229: process: 0.89548 | load: 0.00392 [I 2022-10-11 18:16:50] ...trap/engines/engine.py.230: loss: 0.16786 [I 2022-10-11 18:16:50] ...optimizers/trijoint.py.151: total_norm: 29.538725 [I 2022-10-11 18:16:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 820/2304 [I 2022-10-11 18:16:58] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:21:17 [I 2022-10-11 18:16:58] ...trap/engines/engine.py.229: process: 0.84291 | load: 0.00315 [I 2022-10-11 18:16:58] ...trap/engines/engine.py.230: loss: 0.15334 [I 2022-10-11 18:16:58] ...optimizers/trijoint.py.151: total_norm: 11.448691 [I 2022-10-11 18:17:07] ...trap/engines/engine.py.224: train: epoch 114 | batch 830/2304 [I 2022-10-11 18:17:07] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:48 [I 2022-10-11 18:17:07] ...trap/engines/engine.py.229: process: 0.84390 | load: 0.00299 [I 2022-10-11 18:17:07] ...trap/engines/engine.py.230: loss: 0.20460 [I 2022-10-11 18:17:07] ...optimizers/trijoint.py.151: total_norm: 39.632427 [I 2022-10-11 18:17:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 840/2304 [I 2022-10-11 18:17:16] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:21:22 [I 2022-10-11 18:17:16] ...trap/engines/engine.py.229: process: 0.87786 | load: 0.00383 [I 2022-10-11 18:17:16] ...trap/engines/engine.py.230: loss: 0.10609 [I 2022-10-11 18:17:16] ...optimizers/trijoint.py.151: total_norm: 11.552288 [I 2022-10-11 18:17:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 850/2304 [I 2022-10-11 18:17:24] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:21:11 [I 2022-10-11 18:17:24] ...trap/engines/engine.py.229: process: 0.87435 | load: 0.00377 [I 2022-10-11 18:17:24] ...trap/engines/engine.py.230: loss: 0.08693 [I 2022-10-11 18:17:24] ...optimizers/trijoint.py.151: total_norm: 12.456206 [I 2022-10-11 18:17:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 860/2304 [I 2022-10-11 18:17:33] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:20:56 [I 2022-10-11 18:17:33] ...trap/engines/engine.py.229: process: 0.87186 | load: 0.00376 [I 2022-10-11 18:17:33] ...trap/engines/engine.py.230: loss: 0.15416 [I 2022-10-11 18:17:33] ...optimizers/trijoint.py.151: total_norm: 14.727998 [I 2022-10-11 18:17:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 870/2304 [I 2022-10-11 18:17:42] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:20:56 [I 2022-10-11 18:17:42] ...trap/engines/engine.py.229: process: 0.88214 | load: 0.00381 [I 2022-10-11 18:17:42] ...trap/engines/engine.py.230: loss: 0.13647 [I 2022-10-11 18:17:42] ...optimizers/trijoint.py.151: total_norm: 17.782553 [I 2022-10-11 18:17:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 880/2304 [I 2022-10-11 18:17:51] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:20:51 [I 2022-10-11 18:17:51] ...trap/engines/engine.py.229: process: 0.88934 | load: 0.00380 [I 2022-10-11 18:17:51] ...trap/engines/engine.py.230: loss: 0.08608 [I 2022-10-11 18:17:51] ...optimizers/trijoint.py.151: total_norm: 13.813611 [I 2022-10-11 18:17:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 890/2304 [I 2022-10-11 18:17:59] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:20:36 [I 2022-10-11 18:17:59] ...trap/engines/engine.py.229: process: 0.87381 | load: 0.00377 [I 2022-10-11 18:17:59] ...trap/engines/engine.py.230: loss: 0.13425 [I 2022-10-11 18:17:59] ...optimizers/trijoint.py.151: total_norm: 8.157834 [I 2022-10-11 18:18:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 900/2304 [I 2022-10-11 18:18:08] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:19:56 [I 2022-10-11 18:18:08] ...trap/engines/engine.py.229: process: 0.85890 | load: 0.00300 [I 2022-10-11 18:18:08] ...trap/engines/engine.py.230: loss: 0.09248 [I 2022-10-11 18:18:08] ...optimizers/trijoint.py.151: total_norm: 19.433846 [I 2022-10-11 18:18:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 910/2304 [I 2022-10-11 18:18:16] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:19:41 [I 2022-10-11 18:18:16] ...trap/engines/engine.py.229: process: 0.84897 | load: 0.00300 [I 2022-10-11 18:18:16] ...trap/engines/engine.py.230: loss: 0.11775 [I 2022-10-11 18:18:16] ...optimizers/trijoint.py.151: total_norm: 18.600578 [I 2022-10-11 18:18:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 920/2304 [I 2022-10-11 18:18:25] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:19:29 [I 2022-10-11 18:18:25] ...trap/engines/engine.py.229: process: 0.83509 | load: 0.00300 [I 2022-10-11 18:18:25] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-11 18:18:25] ...optimizers/trijoint.py.151: total_norm: 11.213956 [I 2022-10-11 18:18:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 930/2304 [I 2022-10-11 18:18:33] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:17 [I 2022-10-11 18:18:33] ...trap/engines/engine.py.229: process: 0.84349 | load: 0.00302 [I 2022-10-11 18:18:33] ...trap/engines/engine.py.230: loss: 0.07684 [I 2022-10-11 18:18:33] ...optimizers/trijoint.py.151: total_norm: 11.760373 [I 2022-10-11 18:18:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 940/2304 [I 2022-10-11 18:18:42] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:19:10 [I 2022-10-11 18:18:42] ...trap/engines/engine.py.229: process: 0.84326 | load: 0.00299 [I 2022-10-11 18:18:42] ...trap/engines/engine.py.230: loss: 0.12210 [I 2022-10-11 18:18:42] ...optimizers/trijoint.py.151: total_norm: 17.176905 [I 2022-10-11 18:18:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 950/2304 [I 2022-10-11 18:18:50] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:18:55 [I 2022-10-11 18:18:50] ...trap/engines/engine.py.229: process: 0.83178 | load: 0.00299 [I 2022-10-11 18:18:50] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-11 18:18:50] ...optimizers/trijoint.py.151: total_norm: 21.132101 [I 2022-10-11 18:18:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 960/2304 [I 2022-10-11 18:18:59] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:56 [I 2022-10-11 18:18:59] ...trap/engines/engine.py.229: process: 0.85510 | load: 0.00312 [I 2022-10-11 18:18:59] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-11 18:18:59] ...optimizers/trijoint.py.151: total_norm: 17.246134 [I 2022-10-11 18:19:07] ...trap/engines/engine.py.224: train: epoch 114 | batch 970/2304 [I 2022-10-11 18:19:07] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:18:57 [I 2022-10-11 18:19:07] ...trap/engines/engine.py.229: process: 0.85731 | load: 0.00313 [I 2022-10-11 18:19:07] ...trap/engines/engine.py.230: loss: 0.09169 [I 2022-10-11 18:19:07] ...optimizers/trijoint.py.151: total_norm: 14.522894 [I 2022-10-11 18:19:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 980/2304 [I 2022-10-11 18:19:16] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:54 [I 2022-10-11 18:19:16] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00315 [I 2022-10-11 18:19:16] ...trap/engines/engine.py.230: loss: 0.11491 [I 2022-10-11 18:19:16] ...optimizers/trijoint.py.151: total_norm: 13.484713 [I 2022-10-11 18:19:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 990/2304 [I 2022-10-11 18:19:24] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:18:49 [I 2022-10-11 18:19:24] ...trap/engines/engine.py.229: process: 0.86861 | load: 0.00316 [I 2022-10-11 18:19:24] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-11 18:19:24] ...optimizers/trijoint.py.151: total_norm: 8.451682 [I 2022-10-11 18:19:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 1000/2304 [I 2022-10-11 18:19:33] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:18:41 [I 2022-10-11 18:19:33] ...trap/engines/engine.py.229: process: 0.88256 | load: 0.00310 [I 2022-10-11 18:19:33] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-11 18:19:33] ...optimizers/trijoint.py.151: total_norm: 18.256817 [I 2022-10-11 18:19:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 1010/2304 [I 2022-10-11 18:19:42] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:18:33 [I 2022-10-11 18:19:42] ...trap/engines/engine.py.229: process: 0.86077 | load: 0.00314 [I 2022-10-11 18:19:42] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-11 18:19:42] ...optimizers/trijoint.py.151: total_norm: 12.618489 [I 2022-10-11 18:19:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 1020/2304 [I 2022-10-11 18:19:50] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:18:29 [I 2022-10-11 18:19:50] ...trap/engines/engine.py.229: process: 0.87366 | load: 0.00318 [I 2022-10-11 18:19:50] ...trap/engines/engine.py.230: loss: 0.18788 [I 2022-10-11 18:19:50] ...optimizers/trijoint.py.151: total_norm: 14.613703 [I 2022-10-11 18:19:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 1030/2304 [I 2022-10-11 18:19:59] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:18:34 [I 2022-10-11 18:19:59] ...trap/engines/engine.py.229: process: 0.88065 | load: 0.00312 [I 2022-10-11 18:19:59] ...trap/engines/engine.py.230: loss: 0.14574 [I 2022-10-11 18:19:59] ...optimizers/trijoint.py.151: total_norm: 14.671184 [I 2022-10-11 18:20:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 1040/2304 [I 2022-10-11 18:20:08] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:18:16 [I 2022-10-11 18:20:08] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00313 [I 2022-10-11 18:20:08] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-11 18:20:08] ...optimizers/trijoint.py.151: total_norm: 14.122673 [I 2022-10-11 18:20:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 1050/2304 [I 2022-10-11 18:20:16] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:18:00 [I 2022-10-11 18:20:16] ...trap/engines/engine.py.229: process: 0.86638 | load: 0.00309 [I 2022-10-11 18:20:16] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-11 18:20:16] ...optimizers/trijoint.py.151: total_norm: 9.926013 [I 2022-10-11 18:20:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 1060/2304 [I 2022-10-11 18:20:25] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:52 [I 2022-10-11 18:20:25] ...trap/engines/engine.py.229: process: 0.86463 | load: 0.00314 [I 2022-10-11 18:20:25] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-11 18:20:25] ...optimizers/trijoint.py.151: total_norm: 14.888527 [I 2022-10-11 18:20:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 1070/2304 [I 2022-10-11 18:20:34] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:17:37 [I 2022-10-11 18:20:34] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00319 [I 2022-10-11 18:20:34] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-11 18:20:34] ...optimizers/trijoint.py.151: total_norm: 37.767117 [I 2022-10-11 18:20:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 1080/2304 [I 2022-10-11 18:20:42] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:17:25 [I 2022-10-11 18:20:42] ...trap/engines/engine.py.229: process: 0.84248 | load: 0.00317 [I 2022-10-11 18:20:42] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-11 18:20:42] ...optimizers/trijoint.py.151: total_norm: 9.204605 [I 2022-10-11 18:20:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 1090/2304 [I 2022-10-11 18:20:51] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:17:24 [I 2022-10-11 18:20:51] ...trap/engines/engine.py.229: process: 0.87397 | load: 0.00315 [I 2022-10-11 18:20:51] ...trap/engines/engine.py.230: loss: 0.16468 [I 2022-10-11 18:20:51] ...optimizers/trijoint.py.151: total_norm: 10.832375 [I 2022-10-11 18:20:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 1100/2304 [I 2022-10-11 18:20:59] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:17:24 [I 2022-10-11 18:20:59] ...trap/engines/engine.py.229: process: 0.87027 | load: 0.00309 [I 2022-10-11 18:20:59] ...trap/engines/engine.py.230: loss: 0.11788 [I 2022-10-11 18:20:59] ...optimizers/trijoint.py.151: total_norm: 25.010958 [I 2022-10-11 18:21:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 1110/2304 [I 2022-10-11 18:21:08] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:17:17 [I 2022-10-11 18:21:08] ...trap/engines/engine.py.229: process: 0.87338 | load: 0.00312 [I 2022-10-11 18:21:08] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-11 18:21:08] ...optimizers/trijoint.py.151: total_norm: 19.511061 [I 2022-10-11 18:21:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 1120/2304 [I 2022-10-11 18:21:17] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:17:09 [I 2022-10-11 18:21:17] ...trap/engines/engine.py.229: process: 0.87385 | load: 0.00314 [I 2022-10-11 18:21:17] ...trap/engines/engine.py.230: loss: 0.08015 [I 2022-10-11 18:21:17] ...optimizers/trijoint.py.151: total_norm: 11.876838 [I 2022-10-11 18:21:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 1130/2304 [I 2022-10-11 18:21:25] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:17:01 [I 2022-10-11 18:21:25] ...trap/engines/engine.py.229: process: 0.87673 | load: 0.00311 [I 2022-10-11 18:21:25] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-11 18:21:25] ...optimizers/trijoint.py.151: total_norm: 15.086214 [I 2022-10-11 18:21:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 1140/2304 [I 2022-10-11 18:21:34] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:43 [I 2022-10-11 18:21:34] ...trap/engines/engine.py.229: process: 0.85439 | load: 0.00318 [I 2022-10-11 18:21:34] ...trap/engines/engine.py.230: loss: 0.08852 [I 2022-10-11 18:21:34] ...optimizers/trijoint.py.151: total_norm: 16.704199 [I 2022-10-11 18:21:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 1150/2304 [I 2022-10-11 18:21:43] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:16:27 [I 2022-10-11 18:21:43] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00310 [I 2022-10-11 18:21:43] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-11 18:21:43] ...optimizers/trijoint.py.151: total_norm: 10.157204 [I 2022-10-11 18:21:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 1160/2304 [I 2022-10-11 18:21:51] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:16:25 [I 2022-10-11 18:21:51] ...trap/engines/engine.py.229: process: 0.88005 | load: 0.00313 [I 2022-10-11 18:21:51] ...trap/engines/engine.py.230: loss: 0.09333 [I 2022-10-11 18:21:51] ...optimizers/trijoint.py.151: total_norm: 13.090020 [I 2022-10-11 18:22:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 1170/2304 [I 2022-10-11 18:22:00] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:16:08 [I 2022-10-11 18:22:00] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00313 [I 2022-10-11 18:22:00] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-11 18:22:00] ...optimizers/trijoint.py.151: total_norm: 12.575810 [I 2022-10-11 18:22:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 1180/2304 [I 2022-10-11 18:22:08] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:16:02 [I 2022-10-11 18:22:08] ...trap/engines/engine.py.229: process: 0.85742 | load: 0.00317 [I 2022-10-11 18:22:08] ...trap/engines/engine.py.230: loss: 0.08031 [I 2022-10-11 18:22:08] ...optimizers/trijoint.py.151: total_norm: 10.961002 [I 2022-10-11 18:22:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 1190/2304 [I 2022-10-11 18:22:17] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:16:16 [I 2022-10-11 18:22:17] ...trap/engines/engine.py.229: process: 0.89313 | load: 0.00398 [I 2022-10-11 18:22:17] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-11 18:22:17] ...optimizers/trijoint.py.151: total_norm: 8.619831 [I 2022-10-11 18:22:26] ...trap/engines/engine.py.224: train: epoch 114 | batch 1200/2304 [I 2022-10-11 18:22:26] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:16:17 [I 2022-10-11 18:22:26] ...trap/engines/engine.py.229: process: 0.87839 | load: 0.00396 [I 2022-10-11 18:22:26] ...trap/engines/engine.py.230: loss: 0.10621 [I 2022-10-11 18:22:26] ...optimizers/trijoint.py.151: total_norm: 12.565985 [I 2022-10-11 18:22:35] ...trap/engines/engine.py.224: train: epoch 114 | batch 1210/2304 [I 2022-10-11 18:22:35] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:16:10 [I 2022-10-11 18:22:35] ...trap/engines/engine.py.229: process: 0.87949 | load: 0.00396 [I 2022-10-11 18:22:35] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-11 18:22:35] ...optimizers/trijoint.py.151: total_norm: 16.139214 [I 2022-10-11 18:22:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 1220/2304 [I 2022-10-11 18:22:44] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:16:02 [I 2022-10-11 18:22:44] ...trap/engines/engine.py.229: process: 0.88225 | load: 0.00397 [I 2022-10-11 18:22:44] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-11 18:22:44] ...optimizers/trijoint.py.151: total_norm: 15.921266 [I 2022-10-11 18:22:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 1230/2304 [I 2022-10-11 18:22:53] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:15:47 [I 2022-10-11 18:22:53] ...trap/engines/engine.py.229: process: 0.87668 | load: 0.00393 [I 2022-10-11 18:22:53] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-11 18:22:53] ...optimizers/trijoint.py.151: total_norm: 27.221470 [I 2022-10-11 18:23:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 1240/2304 [I 2022-10-11 18:23:01] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:15:26 [I 2022-10-11 18:23:01] ...trap/engines/engine.py.229: process: 0.87411 | load: 0.00376 [I 2022-10-11 18:23:01] ...trap/engines/engine.py.230: loss: 0.11832 [I 2022-10-11 18:23:01] ...optimizers/trijoint.py.151: total_norm: 25.948578 [I 2022-10-11 18:23:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 1250/2304 [I 2022-10-11 18:23:10] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:15:11 [I 2022-10-11 18:23:10] ...trap/engines/engine.py.229: process: 0.84755 | load: 0.00298 [I 2022-10-11 18:23:10] ...trap/engines/engine.py.230: loss: 0.12670 [I 2022-10-11 18:23:10] ...optimizers/trijoint.py.151: total_norm: 9.008654 [I 2022-10-11 18:23:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1260/2304 [I 2022-10-11 18:23:19] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:52 [I 2022-10-11 18:23:19] ...trap/engines/engine.py.229: process: 0.87982 | load: 0.00315 [I 2022-10-11 18:23:19] ...trap/engines/engine.py.230: loss: 0.07762 [I 2022-10-11 18:23:19] ...optimizers/trijoint.py.151: total_norm: 20.978306 [I 2022-10-11 18:23:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 1270/2304 [I 2022-10-11 18:23:27] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:40 [I 2022-10-11 18:23:27] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00300 [I 2022-10-11 18:23:27] ...trap/engines/engine.py.230: loss: 0.14424 [I 2022-10-11 18:23:27] ...optimizers/trijoint.py.151: total_norm: 9.215836 [I 2022-10-11 18:23:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1280/2304 [I 2022-10-11 18:23:36] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:14:31 [I 2022-10-11 18:23:36] ...trap/engines/engine.py.229: process: 0.83808 | load: 0.00298 [I 2022-10-11 18:23:36] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-11 18:23:36] ...optimizers/trijoint.py.151: total_norm: 10.015756 [I 2022-10-11 18:23:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 1290/2304 [I 2022-10-11 18:23:44] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:14:19 [I 2022-10-11 18:23:44] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00303 [I 2022-10-11 18:23:44] ...trap/engines/engine.py.230: loss: 0.10374 [I 2022-10-11 18:23:44] ...optimizers/trijoint.py.151: total_norm: 25.164207 [I 2022-10-11 18:23:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 1300/2304 [I 2022-10-11 18:23:53] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:14:15 [I 2022-10-11 18:23:53] ...trap/engines/engine.py.229: process: 0.83944 | load: 0.00302 [I 2022-10-11 18:23:53] ...trap/engines/engine.py.230: loss: 0.10228 [I 2022-10-11 18:23:53] ...optimizers/trijoint.py.151: total_norm: 13.755634 [I 2022-10-11 18:24:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 1310/2304 [I 2022-10-11 18:24:01] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:57 [I 2022-10-11 18:24:01] ...trap/engines/engine.py.229: process: 0.83072 | load: 0.00302 [I 2022-10-11 18:24:01] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-11 18:24:01] ...optimizers/trijoint.py.151: total_norm: 27.116201 [I 2022-10-11 18:24:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 1320/2304 [I 2022-10-11 18:24:10] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:14:00 [I 2022-10-11 18:24:10] ...trap/engines/engine.py.229: process: 0.85891 | load: 0.00300 [I 2022-10-11 18:24:10] ...trap/engines/engine.py.230: loss: 0.16514 [I 2022-10-11 18:24:10] ...optimizers/trijoint.py.151: total_norm: 21.293661 [I 2022-10-11 18:24:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1330/2304 [I 2022-10-11 18:24:19] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:55 [I 2022-10-11 18:24:19] ...trap/engines/engine.py.229: process: 0.86918 | load: 0.00298 [I 2022-10-11 18:24:19] ...trap/engines/engine.py.230: loss: 0.14670 [I 2022-10-11 18:24:19] ...optimizers/trijoint.py.151: total_norm: 10.546941 [I 2022-10-11 18:24:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 1340/2304 [I 2022-10-11 18:24:27] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:48 [I 2022-10-11 18:24:27] ...trap/engines/engine.py.229: process: 0.86610 | load: 0.00385 [I 2022-10-11 18:24:27] ...trap/engines/engine.py.230: loss: 0.08192 [I 2022-10-11 18:24:27] ...optimizers/trijoint.py.151: total_norm: 9.851008 [I 2022-10-11 18:24:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1350/2304 [I 2022-10-11 18:24:36] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:49 [I 2022-10-11 18:24:36] ...trap/engines/engine.py.229: process: 0.87884 | load: 0.00379 [I 2022-10-11 18:24:36] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-11 18:24:36] ...optimizers/trijoint.py.151: total_norm: 9.849052 [I 2022-10-11 18:24:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1360/2304 [I 2022-10-11 18:24:45] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:53 [I 2022-10-11 18:24:45] ...trap/engines/engine.py.229: process: 0.89106 | load: 0.00393 [I 2022-10-11 18:24:45] ...trap/engines/engine.py.230: loss: 0.09169 [I 2022-10-11 18:24:45] ...optimizers/trijoint.py.151: total_norm: 9.888690 [I 2022-10-11 18:24:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 1370/2304 [I 2022-10-11 18:24:54] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:13:46 [I 2022-10-11 18:24:54] ...trap/engines/engine.py.229: process: 0.89040 | load: 0.00397 [I 2022-10-11 18:24:54] ...trap/engines/engine.py.230: loss: 0.18081 [I 2022-10-11 18:24:54] ...optimizers/trijoint.py.151: total_norm: 15.527688 [I 2022-10-11 18:25:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1380/2304 [I 2022-10-11 18:25:02] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:13:35 [I 2022-10-11 18:25:02] ...trap/engines/engine.py.229: process: 0.88549 | load: 0.00394 [I 2022-10-11 18:25:02] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-11 18:25:02] ...optimizers/trijoint.py.151: total_norm: 17.065975 [I 2022-10-11 18:25:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 1390/2304 [I 2022-10-11 18:25:11] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:13:28 [I 2022-10-11 18:25:11] ...trap/engines/engine.py.229: process: 0.88705 | load: 0.00393 [I 2022-10-11 18:25:11] ...trap/engines/engine.py.230: loss: 0.09927 [I 2022-10-11 18:25:11] ...optimizers/trijoint.py.151: total_norm: 14.489265 [I 2022-10-11 18:25:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 1400/2304 [I 2022-10-11 18:25:20] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:13:26 [I 2022-10-11 18:25:20] ...trap/engines/engine.py.229: process: 0.89532 | load: 0.00397 [I 2022-10-11 18:25:20] ...trap/engines/engine.py.230: loss: 0.08518 [I 2022-10-11 18:25:20] ...optimizers/trijoint.py.151: total_norm: 16.005560 [I 2022-10-11 18:25:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 1410/2304 [I 2022-10-11 18:25:29] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:13:14 [I 2022-10-11 18:25:29] ...trap/engines/engine.py.229: process: 0.89252 | load: 0.00393 [I 2022-10-11 18:25:29] ...trap/engines/engine.py.230: loss: 0.07457 [I 2022-10-11 18:25:29] ...optimizers/trijoint.py.151: total_norm: 13.969024 [I 2022-10-11 18:25:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1420/2304 [I 2022-10-11 18:25:38] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:13:04 [I 2022-10-11 18:25:38] ...trap/engines/engine.py.229: process: 0.88288 | load: 0.00393 [I 2022-10-11 18:25:38] ...trap/engines/engine.py.230: loss: 0.09450 [I 2022-10-11 18:25:38] ...optimizers/trijoint.py.151: total_norm: 12.152009 [I 2022-10-11 18:25:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1430/2304 [I 2022-10-11 18:25:47] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:12:43 [I 2022-10-11 18:25:47] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00312 [I 2022-10-11 18:25:47] ...trap/engines/engine.py.230: loss: 0.13987 [I 2022-10-11 18:25:47] ...optimizers/trijoint.py.151: total_norm: 43.483498 [I 2022-10-11 18:25:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1440/2304 [I 2022-10-11 18:25:55] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:12:20 [I 2022-10-11 18:25:55] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00313 [I 2022-10-11 18:25:55] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-11 18:25:55] ...optimizers/trijoint.py.151: total_norm: 36.894875 [I 2022-10-11 18:26:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 1450/2304 [I 2022-10-11 18:26:04] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:12:09 [I 2022-10-11 18:26:04] ...trap/engines/engine.py.229: process: 0.84482 | load: 0.00312 [I 2022-10-11 18:26:04] ...trap/engines/engine.py.230: loss: 0.10700 [I 2022-10-11 18:26:04] ...optimizers/trijoint.py.151: total_norm: 23.090050 [I 2022-10-11 18:26:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 1460/2304 [I 2022-10-11 18:26:12] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:12:01 [I 2022-10-11 18:26:12] ...trap/engines/engine.py.229: process: 0.86487 | load: 0.00312 [I 2022-10-11 18:26:12] ...trap/engines/engine.py.230: loss: 0.18069 [I 2022-10-11 18:26:12] ...optimizers/trijoint.py.151: total_norm: 13.601301 [I 2022-10-11 18:26:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 1470/2304 [I 2022-10-11 18:26:21] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:11:53 [I 2022-10-11 18:26:21] ...trap/engines/engine.py.229: process: 0.86517 | load: 0.00314 [I 2022-10-11 18:26:21] ...trap/engines/engine.py.230: loss: 0.20650 [I 2022-10-11 18:26:21] ...optimizers/trijoint.py.151: total_norm: 22.553541 [I 2022-10-11 18:26:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 1480/2304 [I 2022-10-11 18:26:30] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:42 [I 2022-10-11 18:26:30] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00312 [I 2022-10-11 18:26:30] ...trap/engines/engine.py.230: loss: 0.08693 [I 2022-10-11 18:26:30] ...optimizers/trijoint.py.151: total_norm: 10.644905 [I 2022-10-11 18:26:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1490/2304 [I 2022-10-11 18:26:38] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:11:37 [I 2022-10-11 18:26:38] ...trap/engines/engine.py.229: process: 0.85014 | load: 0.00312 [I 2022-10-11 18:26:38] ...trap/engines/engine.py.230: loss: 0.08342 [I 2022-10-11 18:26:38] ...optimizers/trijoint.py.151: total_norm: 14.880196 [I 2022-10-11 18:26:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1500/2304 [I 2022-10-11 18:26:47] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:26 [I 2022-10-11 18:26:47] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00314 [I 2022-10-11 18:26:47] ...trap/engines/engine.py.230: loss: 0.11531 [I 2022-10-11 18:26:47] ...optimizers/trijoint.py.151: total_norm: 13.211059 [I 2022-10-11 18:26:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1510/2304 [I 2022-10-11 18:26:55] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:11:20 [I 2022-10-11 18:26:55] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00312 [I 2022-10-11 18:26:55] ...trap/engines/engine.py.230: loss: 0.09081 [I 2022-10-11 18:26:55] ...optimizers/trijoint.py.151: total_norm: 10.655002 [I 2022-10-11 18:27:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 1520/2304 [I 2022-10-11 18:27:04] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:11:09 [I 2022-10-11 18:27:04] ...trap/engines/engine.py.229: process: 0.85400 | load: 0.00313 [I 2022-10-11 18:27:04] ...trap/engines/engine.py.230: loss: 0.09604 [I 2022-10-11 18:27:04] ...optimizers/trijoint.py.151: total_norm: 12.978558 [I 2022-10-11 18:27:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 1530/2304 [I 2022-10-11 18:27:12] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:11:04 [I 2022-10-11 18:27:12] ...trap/engines/engine.py.229: process: 0.85760 | load: 0.00312 [I 2022-10-11 18:27:12] ...trap/engines/engine.py.230: loss: 0.12473 [I 2022-10-11 18:27:12] ...optimizers/trijoint.py.151: total_norm: 16.352333 [I 2022-10-11 18:27:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 1540/2304 [I 2022-10-11 18:27:21] ...trap/engines/engine.py.228: elapsed: 0:22:41 | left: 0:10:54 [I 2022-10-11 18:27:21] ...trap/engines/engine.py.229: process: 0.85818 | load: 0.00316 [I 2022-10-11 18:27:21] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-11 18:27:21] ...optimizers/trijoint.py.151: total_norm: 22.921413 [I 2022-10-11 18:27:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 1550/2304 [I 2022-10-11 18:27:29] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:42 [I 2022-10-11 18:27:29] ...trap/engines/engine.py.229: process: 0.85217 | load: 0.00314 [I 2022-10-11 18:27:29] ...trap/engines/engine.py.230: loss: 0.09281 [I 2022-10-11 18:27:29] ...optimizers/trijoint.py.151: total_norm: 12.261102 [I 2022-10-11 18:27:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1560/2304 [I 2022-10-11 18:27:38] ...trap/engines/engine.py.228: elapsed: 0:22:58 | left: 0:10:39 [I 2022-10-11 18:27:38] ...trap/engines/engine.py.229: process: 0.85739 | load: 0.00313 [I 2022-10-11 18:27:38] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-11 18:27:38] ...optimizers/trijoint.py.151: total_norm: 65.608780 [I 2022-10-11 18:27:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1570/2304 [I 2022-10-11 18:27:47] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:10:32 [I 2022-10-11 18:27:47] ...trap/engines/engine.py.229: process: 0.87086 | load: 0.00318 [I 2022-10-11 18:27:47] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-11 18:27:47] ...optimizers/trijoint.py.151: total_norm: 12.261669 [I 2022-10-11 18:27:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1580/2304 [I 2022-10-11 18:27:55] ...trap/engines/engine.py.228: elapsed: 0:23:15 | left: 0:10:24 [I 2022-10-11 18:27:55] ...trap/engines/engine.py.229: process: 0.85781 | load: 0.00312 [I 2022-10-11 18:27:55] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-11 18:27:55] ...optimizers/trijoint.py.151: total_norm: 13.801837 [I 2022-10-11 18:28:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 1590/2304 [I 2022-10-11 18:28:04] ...trap/engines/engine.py.228: elapsed: 0:23:24 | left: 0:10:19 [I 2022-10-11 18:28:04] ...trap/engines/engine.py.229: process: 0.88058 | load: 0.00316 [I 2022-10-11 18:28:04] ...trap/engines/engine.py.230: loss: 0.15479 [I 2022-10-11 18:28:04] ...optimizers/trijoint.py.151: total_norm: 16.600935 [I 2022-10-11 18:28:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 1600/2304 [I 2022-10-11 18:28:13] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:10:08 [I 2022-10-11 18:28:13] ...trap/engines/engine.py.229: process: 0.88326 | load: 0.00313 [I 2022-10-11 18:28:13] ...trap/engines/engine.py.230: loss: 0.16771 [I 2022-10-11 18:28:13] ...optimizers/trijoint.py.151: total_norm: 11.520577 [I 2022-10-11 18:28:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 1610/2304 [I 2022-10-11 18:28:22] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:10:12 [I 2022-10-11 18:28:22] ...trap/engines/engine.py.229: process: 0.87835 | load: 0.00398 [I 2022-10-11 18:28:22] ...trap/engines/engine.py.230: loss: 0.09343 [I 2022-10-11 18:28:22] ...optimizers/trijoint.py.151: total_norm: 15.176414 [I 2022-10-11 18:28:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 1620/2304 [I 2022-10-11 18:28:30] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:47 [I 2022-10-11 18:28:30] ...trap/engines/engine.py.229: process: 0.84742 | load: 0.00315 [I 2022-10-11 18:28:30] ...trap/engines/engine.py.230: loss: 0.10104 [I 2022-10-11 18:28:30] ...optimizers/trijoint.py.151: total_norm: 10.275100 [I 2022-10-11 18:28:39] ...trap/engines/engine.py.224: train: epoch 114 | batch 1630/2304 [I 2022-10-11 18:28:39] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:32 [I 2022-10-11 18:28:39] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00306 [I 2022-10-11 18:28:39] ...trap/engines/engine.py.230: loss: 0.09856 [I 2022-10-11 18:28:39] ...optimizers/trijoint.py.151: total_norm: 11.459742 [I 2022-10-11 18:28:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1640/2304 [I 2022-10-11 18:28:47] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:19 [I 2022-10-11 18:28:47] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00296 [I 2022-10-11 18:28:47] ...trap/engines/engine.py.230: loss: 0.08211 [I 2022-10-11 18:28:47] ...optimizers/trijoint.py.151: total_norm: 13.423921 [I 2022-10-11 18:28:56] ...trap/engines/engine.py.224: train: epoch 114 | batch 1650/2304 [I 2022-10-11 18:28:56] ...trap/engines/engine.py.228: elapsed: 0:24:15 | left: 0:09:10 [I 2022-10-11 18:28:56] ...trap/engines/engine.py.229: process: 0.83893 | load: 0.00297 [I 2022-10-11 18:28:56] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-11 18:28:56] ...optimizers/trijoint.py.151: total_norm: 20.913134 [I 2022-10-11 18:29:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 1660/2304 [I 2022-10-11 18:29:04] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:09:05 [I 2022-10-11 18:29:04] ...trap/engines/engine.py.229: process: 0.84312 | load: 0.00294 [I 2022-10-11 18:29:04] ...trap/engines/engine.py.230: loss: 0.09919 [I 2022-10-11 18:29:04] ...optimizers/trijoint.py.151: total_norm: 15.399091 [I 2022-10-11 18:29:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 1670/2304 [I 2022-10-11 18:29:13] ...trap/engines/engine.py.228: elapsed: 0:24:32 | left: 0:08:53 [I 2022-10-11 18:29:13] ...trap/engines/engine.py.229: process: 0.83433 | load: 0.00293 [I 2022-10-11 18:29:13] ...trap/engines/engine.py.230: loss: 0.10422 [I 2022-10-11 18:29:13] ...optimizers/trijoint.py.151: total_norm: 17.497416 [I 2022-10-11 18:29:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 1680/2304 [I 2022-10-11 18:29:21] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:45 [I 2022-10-11 18:29:21] ...trap/engines/engine.py.229: process: 0.83179 | load: 0.00295 [I 2022-10-11 18:29:21] ...trap/engines/engine.py.230: loss: 0.11467 [I 2022-10-11 18:29:21] ...optimizers/trijoint.py.151: total_norm: 8.647420 [I 2022-10-11 18:29:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 1690/2304 [I 2022-10-11 18:29:29] ...trap/engines/engine.py.228: elapsed: 0:24:49 | left: 0:08:35 [I 2022-10-11 18:29:29] ...trap/engines/engine.py.229: process: 0.83739 | load: 0.00294 [I 2022-10-11 18:29:29] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-11 18:29:29] ...optimizers/trijoint.py.151: total_norm: 19.320925 [I 2022-10-11 18:29:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1700/2304 [I 2022-10-11 18:29:38] ...trap/engines/engine.py.228: elapsed: 0:24:57 | left: 0:08:27 [I 2022-10-11 18:29:38] ...trap/engines/engine.py.229: process: 0.83490 | load: 0.00294 [I 2022-10-11 18:29:38] ...trap/engines/engine.py.230: loss: 0.17213 [I 2022-10-11 18:29:38] ...optimizers/trijoint.py.151: total_norm: 20.089781 [I 2022-10-11 18:29:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 1710/2304 [I 2022-10-11 18:29:46] ...trap/engines/engine.py.228: elapsed: 0:25:06 | left: 0:08:19 [I 2022-10-11 18:29:46] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00296 [I 2022-10-11 18:29:46] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-11 18:29:46] ...optimizers/trijoint.py.151: total_norm: 15.590778 [I 2022-10-11 18:29:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1720/2304 [I 2022-10-11 18:29:55] ...trap/engines/engine.py.228: elapsed: 0:25:14 | left: 0:08:10 [I 2022-10-11 18:29:55] ...trap/engines/engine.py.229: process: 0.83773 | load: 0.00294 [I 2022-10-11 18:29:55] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-11 18:29:55] ...optimizers/trijoint.py.151: total_norm: 14.258359 [I 2022-10-11 18:30:03] ...trap/engines/engine.py.224: train: epoch 114 | batch 1730/2304 [I 2022-10-11 18:30:03] ...trap/engines/engine.py.228: elapsed: 0:25:23 | left: 0:08:04 [I 2022-10-11 18:30:03] ...trap/engines/engine.py.229: process: 0.86299 | load: 0.00296 [I 2022-10-11 18:30:03] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-11 18:30:03] ...optimizers/trijoint.py.151: total_norm: 13.077009 [I 2022-10-11 18:30:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 1740/2304 [I 2022-10-11 18:30:12] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:55 [I 2022-10-11 18:30:12] ...trap/engines/engine.py.229: process: 0.84720 | load: 0.00295 [I 2022-10-11 18:30:12] ...trap/engines/engine.py.230: loss: 0.12136 [I 2022-10-11 18:30:12] ...optimizers/trijoint.py.151: total_norm: 15.530987 [I 2022-10-11 18:30:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 1750/2304 [I 2022-10-11 18:30:20] ...trap/engines/engine.py.228: elapsed: 0:25:40 | left: 0:07:43 [I 2022-10-11 18:30:20] ...trap/engines/engine.py.229: process: 0.83407 | load: 0.00298 [I 2022-10-11 18:30:20] ...trap/engines/engine.py.230: loss: 0.11702 [I 2022-10-11 18:30:20] ...optimizers/trijoint.py.151: total_norm: 19.041557 [I 2022-10-11 18:30:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 1760/2304 [I 2022-10-11 18:30:28] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:39 [I 2022-10-11 18:30:28] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00298 [I 2022-10-11 18:30:28] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-11 18:30:28] ...optimizers/trijoint.py.151: total_norm: 8.679764 [I 2022-10-11 18:30:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 1770/2304 [I 2022-10-11 18:30:37] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:32 [I 2022-10-11 18:30:37] ...trap/engines/engine.py.229: process: 0.86534 | load: 0.00303 [I 2022-10-11 18:30:37] ...trap/engines/engine.py.230: loss: 0.07595 [I 2022-10-11 18:30:37] ...optimizers/trijoint.py.151: total_norm: 27.529192 [I 2022-10-11 18:30:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1780/2304 [I 2022-10-11 18:30:45] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:20 [I 2022-10-11 18:30:45] ...trap/engines/engine.py.229: process: 0.83420 | load: 0.00313 [I 2022-10-11 18:30:45] ...trap/engines/engine.py.230: loss: 0.17212 [I 2022-10-11 18:30:45] ...optimizers/trijoint.py.151: total_norm: 25.625275 [I 2022-10-11 18:30:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 1790/2304 [I 2022-10-11 18:30:54] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:07:13 [I 2022-10-11 18:30:54] ...trap/engines/engine.py.229: process: 0.84538 | load: 0.00301 [I 2022-10-11 18:30:54] ...trap/engines/engine.py.230: loss: 0.18371 [I 2022-10-11 18:30:54] ...optimizers/trijoint.py.151: total_norm: 14.804940 [I 2022-10-11 18:31:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1800/2304 [I 2022-10-11 18:31:02] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:07:07 [I 2022-10-11 18:31:02] ...trap/engines/engine.py.229: process: 0.84849 | load: 0.00299 [I 2022-10-11 18:31:02] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-11 18:31:02] ...optimizers/trijoint.py.151: total_norm: 17.356060 [I 2022-10-11 18:31:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 1810/2304 [I 2022-10-11 18:31:11] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:07:00 [I 2022-10-11 18:31:11] ...trap/engines/engine.py.229: process: 0.84369 | load: 0.00298 [I 2022-10-11 18:31:11] ...trap/engines/engine.py.230: loss: 0.16940 [I 2022-10-11 18:31:11] ...optimizers/trijoint.py.151: total_norm: 22.501575 [I 2022-10-11 18:31:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1820/2304 [I 2022-10-11 18:31:19] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:49 [I 2022-10-11 18:31:19] ...trap/engines/engine.py.229: process: 0.83585 | load: 0.00302 [I 2022-10-11 18:31:19] ...trap/engines/engine.py.230: loss: 0.09614 [I 2022-10-11 18:31:19] ...optimizers/trijoint.py.151: total_norm: 16.317930 [I 2022-10-11 18:31:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 1830/2304 [I 2022-10-11 18:31:28] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:41 [I 2022-10-11 18:31:28] ...trap/engines/engine.py.229: process: 0.84059 | load: 0.00302 [I 2022-10-11 18:31:28] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-11 18:31:28] ...optimizers/trijoint.py.151: total_norm: 14.681523 [I 2022-10-11 18:31:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1840/2304 [I 2022-10-11 18:31:36] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:31 [I 2022-10-11 18:31:36] ...trap/engines/engine.py.229: process: 0.83077 | load: 0.00300 [I 2022-10-11 18:31:36] ...trap/engines/engine.py.230: loss: 0.09597 [I 2022-10-11 18:31:36] ...optimizers/trijoint.py.151: total_norm: 13.385509 [I 2022-10-11 18:31:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1850/2304 [I 2022-10-11 18:31:45] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:06:25 [I 2022-10-11 18:31:45] ...trap/engines/engine.py.229: process: 0.84213 | load: 0.00313 [I 2022-10-11 18:31:45] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-11 18:31:45] ...optimizers/trijoint.py.151: total_norm: 14.641985 [I 2022-10-11 18:31:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 1860/2304 [I 2022-10-11 18:31:53] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:06:21 [I 2022-10-11 18:31:53] ...trap/engines/engine.py.229: process: 0.85413 | load: 0.00315 [I 2022-10-11 18:31:53] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-11 18:31:53] ...optimizers/trijoint.py.151: total_norm: 26.635666 [I 2022-10-11 18:32:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1870/2304 [I 2022-10-11 18:32:02] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:06:11 [I 2022-10-11 18:32:02] ...trap/engines/engine.py.229: process: 0.84855 | load: 0.00315 [I 2022-10-11 18:32:02] ...trap/engines/engine.py.230: loss: 0.08663 [I 2022-10-11 18:32:02] ...optimizers/trijoint.py.151: total_norm: 14.802135 [I 2022-10-11 18:32:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 1880/2304 [I 2022-10-11 18:32:11] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:59 [I 2022-10-11 18:32:11] ...trap/engines/engine.py.229: process: 0.86288 | load: 0.00310 [I 2022-10-11 18:32:11] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-11 18:32:11] ...optimizers/trijoint.py.151: total_norm: 17.587011 [I 2022-10-11 18:32:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1890/2304 [I 2022-10-11 18:32:19] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:52 [I 2022-10-11 18:32:19] ...trap/engines/engine.py.229: process: 0.83767 | load: 0.00317 [I 2022-10-11 18:32:19] ...trap/engines/engine.py.230: loss: 0.08795 [I 2022-10-11 18:32:19] ...optimizers/trijoint.py.151: total_norm: 12.760900 [I 2022-10-11 18:32:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 1900/2304 [I 2022-10-11 18:32:28] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:44 [I 2022-10-11 18:32:28] ...trap/engines/engine.py.229: process: 0.85110 | load: 0.00316 [I 2022-10-11 18:32:28] ...trap/engines/engine.py.230: loss: 0.15895 [I 2022-10-11 18:32:28] ...optimizers/trijoint.py.151: total_norm: 21.176891 [I 2022-10-11 18:32:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1910/2304 [I 2022-10-11 18:32:36] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:35 [I 2022-10-11 18:32:36] ...trap/engines/engine.py.229: process: 0.85117 | load: 0.00314 [I 2022-10-11 18:32:36] ...trap/engines/engine.py.230: loss: 0.10102 [I 2022-10-11 18:32:36] ...optimizers/trijoint.py.151: total_norm: 15.655989 [I 2022-10-11 18:32:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1920/2304 [I 2022-10-11 18:32:45] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:05:28 [I 2022-10-11 18:32:45] ...trap/engines/engine.py.229: process: 0.87486 | load: 0.00315 [I 2022-10-11 18:32:45] ...trap/engines/engine.py.230: loss: 0.15113 [I 2022-10-11 18:32:45] ...optimizers/trijoint.py.151: total_norm: 14.239311 [I 2022-10-11 18:32:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 1930/2304 [I 2022-10-11 18:32:53] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:05:20 [I 2022-10-11 18:32:53] ...trap/engines/engine.py.229: process: 0.85139 | load: 0.00319 [I 2022-10-11 18:32:53] ...trap/engines/engine.py.230: loss: 0.23131 [I 2022-10-11 18:32:53] ...optimizers/trijoint.py.151: total_norm: 16.022484 [I 2022-10-11 18:33:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1940/2304 [I 2022-10-11 18:33:02] ...trap/engines/engine.py.228: elapsed: 0:28:22 | left: 0:05:13 [I 2022-10-11 18:33:02] ...trap/engines/engine.py.229: process: 0.87393 | load: 0.00319 [I 2022-10-11 18:33:02] ...trap/engines/engine.py.230: loss: 0.09005 [I 2022-10-11 18:33:02] ...optimizers/trijoint.py.151: total_norm: 57.741814 [I 2022-10-11 18:33:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 1950/2304 [I 2022-10-11 18:33:10] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:05:01 [I 2022-10-11 18:33:10] ...trap/engines/engine.py.229: process: 0.84451 | load: 0.00315 [I 2022-10-11 18:33:10] ...trap/engines/engine.py.230: loss: 0.14201 [I 2022-10-11 18:33:10] ...optimizers/trijoint.py.151: total_norm: 33.927090 [I 2022-10-11 18:33:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1960/2304 [I 2022-10-11 18:33:19] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:54 [I 2022-10-11 18:33:19] ...trap/engines/engine.py.229: process: 0.87607 | load: 0.00313 [I 2022-10-11 18:33:19] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-11 18:33:19] ...optimizers/trijoint.py.151: total_norm: 14.589862 [I 2022-10-11 18:33:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 1970/2304 [I 2022-10-11 18:33:28] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:44 [I 2022-10-11 18:33:28] ...trap/engines/engine.py.229: process: 0.83947 | load: 0.00318 [I 2022-10-11 18:33:28] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-11 18:33:28] ...optimizers/trijoint.py.151: total_norm: 14.857689 [I 2022-10-11 18:33:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1980/2304 [I 2022-10-11 18:33:36] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:04:39 [I 2022-10-11 18:33:36] ...trap/engines/engine.py.229: process: 0.87669 | load: 0.00315 [I 2022-10-11 18:33:36] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-11 18:33:36] ...optimizers/trijoint.py.151: total_norm: 21.292418 [I 2022-10-11 18:33:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1990/2304 [I 2022-10-11 18:33:45] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:04:27 [I 2022-10-11 18:33:45] ...trap/engines/engine.py.229: process: 0.83116 | load: 0.00313 [I 2022-10-11 18:33:45] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-11 18:33:45] ...optimizers/trijoint.py.151: total_norm: 26.259121 [I 2022-10-11 18:33:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 2000/2304 [I 2022-10-11 18:33:53] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:04:27 [I 2022-10-11 18:33:53] ...trap/engines/engine.py.229: process: 0.89344 | load: 0.00400 [I 2022-10-11 18:33:53] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-11 18:33:53] ...optimizers/trijoint.py.151: total_norm: 9.813031 [I 2022-10-11 18:34:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 2010/2304 [I 2022-10-11 18:34:02] ...trap/engines/engine.py.228: elapsed: 0:29:22 | left: 0:04:21 [I 2022-10-11 18:34:02] ...trap/engines/engine.py.229: process: 0.88547 | load: 0.00399 [I 2022-10-11 18:34:02] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-11 18:34:02] ...optimizers/trijoint.py.151: total_norm: 19.847458 [I 2022-10-11 18:34:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 2020/2304 [I 2022-10-11 18:34:11] ...trap/engines/engine.py.228: elapsed: 0:29:31 | left: 0:04:10 [I 2022-10-11 18:34:11] ...trap/engines/engine.py.229: process: 0.86432 | load: 0.00398 [I 2022-10-11 18:34:11] ...trap/engines/engine.py.230: loss: 0.09270 [I 2022-10-11 18:34:11] ...optimizers/trijoint.py.151: total_norm: 9.846401 [I 2022-10-11 18:34:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 2030/2304 [I 2022-10-11 18:34:20] ...trap/engines/engine.py.228: elapsed: 0:29:40 | left: 0:04:03 [I 2022-10-11 18:34:20] ...trap/engines/engine.py.229: process: 0.89869 | load: 0.00399 [I 2022-10-11 18:34:20] ...trap/engines/engine.py.230: loss: 0.09902 [I 2022-10-11 18:34:20] ...optimizers/trijoint.py.151: total_norm: 11.643096 [I 2022-10-11 18:34:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 2040/2304 [I 2022-10-11 18:34:29] ...trap/engines/engine.py.228: elapsed: 0:29:49 | left: 0:03:54 [I 2022-10-11 18:34:29] ...trap/engines/engine.py.229: process: 0.88395 | load: 0.00399 [I 2022-10-11 18:34:29] ...trap/engines/engine.py.230: loss: 0.12274 [I 2022-10-11 18:34:29] ...optimizers/trijoint.py.151: total_norm: 32.752361 [I 2022-10-11 18:34:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 2050/2304 [I 2022-10-11 18:34:38] ...trap/engines/engine.py.228: elapsed: 0:29:58 | left: 0:03:44 [I 2022-10-11 18:34:38] ...trap/engines/engine.py.229: process: 0.87408 | load: 0.00398 [I 2022-10-11 18:34:38] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-11 18:34:38] ...optimizers/trijoint.py.151: total_norm: 13.250944 [I 2022-10-11 18:34:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 2060/2304 [I 2022-10-11 18:34:47] ...trap/engines/engine.py.228: elapsed: 0:30:06 | left: 0:03:34 [I 2022-10-11 18:34:47] ...trap/engines/engine.py.229: process: 0.89092 | load: 0.00398 [I 2022-10-11 18:34:47] ...trap/engines/engine.py.230: loss: 0.08973 [I 2022-10-11 18:34:47] ...optimizers/trijoint.py.151: total_norm: 9.475577 [I 2022-10-11 18:34:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 2070/2304 [I 2022-10-11 18:34:55] ...trap/engines/engine.py.228: elapsed: 0:30:15 | left: 0:03:20 [I 2022-10-11 18:34:55] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00313 [I 2022-10-11 18:34:55] ...trap/engines/engine.py.230: loss: 0.09392 [I 2022-10-11 18:34:55] ...optimizers/trijoint.py.151: total_norm: 21.259329 [I 2022-10-11 18:35:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 2080/2304 [I 2022-10-11 18:35:04] ...trap/engines/engine.py.228: elapsed: 0:30:23 | left: 0:03:10 [I 2022-10-11 18:35:04] ...trap/engines/engine.py.229: process: 0.84267 | load: 0.00311 [I 2022-10-11 18:35:04] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-11 18:35:04] ...optimizers/trijoint.py.151: total_norm: 9.863849 [I 2022-10-11 18:35:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 2090/2304 [I 2022-10-11 18:35:12] ...trap/engines/engine.py.228: elapsed: 0:30:32 | left: 0:03:03 [I 2022-10-11 18:35:12] ...trap/engines/engine.py.229: process: 0.84225 | load: 0.00318 [I 2022-10-11 18:35:12] ...trap/engines/engine.py.230: loss: 0.08645 [I 2022-10-11 18:35:12] ...optimizers/trijoint.py.151: total_norm: 10.159154 [I 2022-10-11 18:35:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 2100/2304 [I 2022-10-11 18:35:21] ...trap/engines/engine.py.228: elapsed: 0:30:41 | left: 0:02:55 [I 2022-10-11 18:35:21] ...trap/engines/engine.py.229: process: 0.86815 | load: 0.00314 [I 2022-10-11 18:35:21] ...trap/engines/engine.py.230: loss: 0.10674 [I 2022-10-11 18:35:21] ...optimizers/trijoint.py.151: total_norm: 12.197072 [I 2022-10-11 18:35:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 2110/2304 [I 2022-10-11 18:35:30] ...trap/engines/engine.py.228: elapsed: 0:30:49 | left: 0:02:45 [I 2022-10-11 18:35:30] ...trap/engines/engine.py.229: process: 0.84699 | load: 0.00314 [I 2022-10-11 18:35:30] ...trap/engines/engine.py.230: loss: 0.09072 [I 2022-10-11 18:35:30] ...optimizers/trijoint.py.151: total_norm: 10.777395 [I 2022-10-11 18:35:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 2120/2304 [I 2022-10-11 18:35:38] ...trap/engines/engine.py.228: elapsed: 0:30:58 | left: 0:02:35 [I 2022-10-11 18:35:38] ...trap/engines/engine.py.229: process: 0.85182 | load: 0.00307 [I 2022-10-11 18:35:38] ...trap/engines/engine.py.230: loss: 0.13336 [I 2022-10-11 18:35:38] ...optimizers/trijoint.py.151: total_norm: 15.814145 [I 2022-10-11 18:35:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 2130/2304 [I 2022-10-11 18:35:46] ...trap/engines/engine.py.228: elapsed: 0:31:06 | left: 0:02:25 [I 2022-10-11 18:35:46] ...trap/engines/engine.py.229: process: 0.83082 | load: 0.00297 [I 2022-10-11 18:35:46] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-11 18:35:46] ...optimizers/trijoint.py.151: total_norm: 17.157488 [I 2022-10-11 18:35:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 2140/2304 [I 2022-10-11 18:35:55] ...trap/engines/engine.py.228: elapsed: 0:31:15 | left: 0:02:18 [I 2022-10-11 18:35:55] ...trap/engines/engine.py.229: process: 0.84551 | load: 0.00297 [I 2022-10-11 18:35:55] ...trap/engines/engine.py.230: loss: 0.12421 [I 2022-10-11 18:35:55] ...optimizers/trijoint.py.151: total_norm: 29.637468 [I 2022-10-11 18:36:03] ...trap/engines/engine.py.224: train: epoch 114 | batch 2150/2304 [I 2022-10-11 18:36:03] ...trap/engines/engine.py.228: elapsed: 0:31:23 | left: 0:02:10 [I 2022-10-11 18:36:03] ...trap/engines/engine.py.229: process: 0.85787 | load: 0.00296 [I 2022-10-11 18:36:03] ...trap/engines/engine.py.230: loss: 0.11951 [I 2022-10-11 18:36:03] ...optimizers/trijoint.py.151: total_norm: 8.584849 [I 2022-10-11 18:36:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 2160/2304 [I 2022-10-11 18:36:12] ...trap/engines/engine.py.228: elapsed: 0:31:31 | left: 0:02:01 [I 2022-10-11 18:36:12] ...trap/engines/engine.py.229: process: 0.84941 | load: 0.00299 [I 2022-10-11 18:36:12] ...trap/engines/engine.py.230: loss: 0.16927 [I 2022-10-11 18:36:12] ...optimizers/trijoint.py.151: total_norm: 19.200962 [I 2022-10-11 18:36:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 2170/2304 [I 2022-10-11 18:36:20] ...trap/engines/engine.py.228: elapsed: 0:31:40 | left: 0:01:52 [I 2022-10-11 18:36:20] ...trap/engines/engine.py.229: process: 0.82549 | load: 0.00296 [I 2022-10-11 18:36:20] ...trap/engines/engine.py.230: loss: 0.10933 [I 2022-10-11 18:36:20] ...optimizers/trijoint.py.151: total_norm: 34.015083 [I 2022-10-11 18:36:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 2180/2304 [I 2022-10-11 18:36:29] ...trap/engines/engine.py.228: elapsed: 0:31:48 | left: 0:01:44 [I 2022-10-11 18:36:29] ...trap/engines/engine.py.229: process: 0.84336 | load: 0.00295 [I 2022-10-11 18:36:29] ...trap/engines/engine.py.230: loss: 0.09522 [I 2022-10-11 18:36:29] ...optimizers/trijoint.py.151: total_norm: 14.145549 [I 2022-10-11 18:36:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 2190/2304 [I 2022-10-11 18:36:37] ...trap/engines/engine.py.228: elapsed: 0:31:57 | left: 0:01:35 [I 2022-10-11 18:36:37] ...trap/engines/engine.py.229: process: 0.84489 | load: 0.00296 [I 2022-10-11 18:36:37] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-11 18:36:37] ...optimizers/trijoint.py.151: total_norm: 9.687912 [I 2022-10-11 18:36:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 2200/2304 [I 2022-10-11 18:36:45] ...trap/engines/engine.py.228: elapsed: 0:32:05 | left: 0:01:27 [I 2022-10-11 18:36:45] ...trap/engines/engine.py.229: process: 0.83035 | load: 0.00297 [I 2022-10-11 18:36:45] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-11 18:36:45] ...optimizers/trijoint.py.151: total_norm: 17.949223 [I 2022-10-11 18:36:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 2210/2304 [I 2022-10-11 18:36:54] ...trap/engines/engine.py.228: elapsed: 0:32:14 | left: 0:01:19 [I 2022-10-11 18:36:54] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00295 [I 2022-10-11 18:36:54] ...trap/engines/engine.py.230: loss: 0.13824 [I 2022-10-11 18:36:54] ...optimizers/trijoint.py.151: total_norm: 9.678365 [I 2022-10-11 18:37:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 2220/2304 [I 2022-10-11 18:37:02] ...trap/engines/engine.py.228: elapsed: 0:32:22 | left: 0:01:10 [I 2022-10-11 18:37:02] ...trap/engines/engine.py.229: process: 0.82584 | load: 0.00296 [I 2022-10-11 18:37:02] ...trap/engines/engine.py.230: loss: 0.09708 [I 2022-10-11 18:37:02] ...optimizers/trijoint.py.151: total_norm: 10.763421 [I 2022-10-11 18:37:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 2230/2304 [I 2022-10-11 18:37:11] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:01:01 [I 2022-10-11 18:37:11] ...trap/engines/engine.py.229: process: 0.81400 | load: 0.00298 [I 2022-10-11 18:37:11] ...trap/engines/engine.py.230: loss: 0.09939 [I 2022-10-11 18:37:11] ...optimizers/trijoint.py.151: total_norm: 36.039932 [I 2022-10-11 18:37:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 2240/2304 [I 2022-10-11 18:37:19] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:53 [I 2022-10-11 18:37:19] ...trap/engines/engine.py.229: process: 0.85356 | load: 0.00301 [I 2022-10-11 18:37:19] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-11 18:37:19] ...optimizers/trijoint.py.151: total_norm: 17.095203 [I 2022-10-11 18:37:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 2250/2304 [I 2022-10-11 18:37:27] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:45 [I 2022-10-11 18:37:27] ...trap/engines/engine.py.229: process: 0.84196 | load: 0.00298 [I 2022-10-11 18:37:27] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-11 18:37:27] ...optimizers/trijoint.py.151: total_norm: 15.737529 [I 2022-10-11 18:37:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 2260/2304 [I 2022-10-11 18:37:36] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:36 [I 2022-10-11 18:37:36] ...trap/engines/engine.py.229: process: 0.82926 | load: 0.00299 [I 2022-10-11 18:37:36] ...trap/engines/engine.py.230: loss: 0.12892 [I 2022-10-11 18:37:36] ...optimizers/trijoint.py.151: total_norm: 26.118685 [I 2022-10-11 18:37:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 2270/2304 [I 2022-10-11 18:37:44] ...trap/engines/engine.py.228: elapsed: 0:33:04 | left: 0:00:28 [I 2022-10-11 18:37:44] ...trap/engines/engine.py.229: process: 0.84455 | load: 0.00299 [I 2022-10-11 18:37:44] ...trap/engines/engine.py.230: loss: 0.13695 [I 2022-10-11 18:37:44] ...optimizers/trijoint.py.151: total_norm: 20.845116 [I 2022-10-11 18:37:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 2280/2304 [I 2022-10-11 18:37:53] ...trap/engines/engine.py.228: elapsed: 0:33:12 | left: 0:00:20 [I 2022-10-11 18:37:53] ...trap/engines/engine.py.229: process: 0.84979 | load: 0.00295 [I 2022-10-11 18:37:53] ...trap/engines/engine.py.230: loss: 0.11979 [I 2022-10-11 18:37:53] ...optimizers/trijoint.py.151: total_norm: 11.006068 [I 2022-10-11 18:38:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 2290/2304 [I 2022-10-11 18:38:01] ...trap/engines/engine.py.228: elapsed: 0:33:21 | left: 0:00:11 [I 2022-10-11 18:38:01] ...trap/engines/engine.py.229: process: 0.83652 | load: 0.00299 [I 2022-10-11 18:38:01] ...trap/engines/engine.py.230: loss: 0.08964 [I 2022-10-11 18:38:01] ...optimizers/trijoint.py.151: total_norm: 12.105738 [I 2022-10-11 18:38:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 2300/2304 [I 2022-10-11 18:38:10] ...trap/engines/engine.py.228: elapsed: 0:33:29 | left: 0:00:03 [I 2022-10-11 18:38:10] ...trap/engines/engine.py.229: process: 0.83327 | load: 0.00016 [I 2022-10-11 18:38:10] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-11 18:38:10] ...optimizers/trijoint.py.151: total_norm: 20.053452 [I 2022-10-11 18:38:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 2304/2304 [I 2022-10-11 18:38:13] ...trap/engines/engine.py.228: elapsed: 0:33:33 | left: 0:00:00 [I 2022-10-11 18:38:13] ...trap/engines/engine.py.229: process: 0.83246 | load: 0.00016 [I 2022-10-11 18:38:13] ...trap/engines/engine.py.230: loss: 0.14226 [I 2022-10-11 18:38:13] ...optimizers/trijoint.py.151: total_norm: 20.294415 [S 2022-10-11 18:38:14] ...trap/engines/engine.py.129: train_epoch.epoch: 114 [S 2022-10-11 18:38:14] ...trap/engines/engine.py.129: train_epoch.loss: 0.11909992772916134 [I 2022-10-11 18:38:14] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 18:38:16] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 114 [I 2022-10-11 18:38:16] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 18:38:18] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 18:38:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 18:38:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 18:38:22] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 18:38:22] ...trap/engines/engine.py.321: eval: epoch 114 | batch 0/510 [I 2022-10-11 18:38:22] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:54:16 [I 2022-10-11 18:38:22] ...trap/engines/engine.py.326: process: 6.38617 | load: 4.37242 [I 2022-10-11 18:38:27] ...trap/engines/engine.py.321: eval: epoch 114 | batch 10/510 [I 2022-10-11 18:38:27] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:24 [I 2022-10-11 18:38:27] ...trap/engines/engine.py.326: process: 0.48305 | load: 0.00019 [I 2022-10-11 18:38:33] ...trap/engines/engine.py.321: eval: epoch 114 | batch 20/510 [I 2022-10-11 18:38:33] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:05:15 [I 2022-10-11 18:38:33] ...trap/engines/engine.py.326: process: 0.62108 | load: 0.00020 [I 2022-10-11 18:38:39] ...trap/engines/engine.py.321: eval: epoch 114 | batch 30/510 [I 2022-10-11 18:38:39] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:04:43 [I 2022-10-11 18:38:39] ...trap/engines/engine.py.326: process: 0.57143 | load: 0.00019 [I 2022-10-11 18:38:49] ...trap/engines/engine.py.321: eval: epoch 114 | batch 40/510 [I 2022-10-11 18:38:49] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:08:22 [I 2022-10-11 18:38:49] ...trap/engines/engine.py.326: process: 0.56510 | load: 0.00019 [I 2022-10-11 18:38:54] ...trap/engines/engine.py.321: eval: epoch 114 | batch 50/510 [I 2022-10-11 18:38:54] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:20 [I 2022-10-11 18:38:54] ...trap/engines/engine.py.326: process: 0.50049 | load: 0.00022 [I 2022-10-11 18:38:59] ...trap/engines/engine.py.321: eval: epoch 114 | batch 60/510 [I 2022-10-11 18:38:59] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:32 [I 2022-10-11 18:38:59] ...trap/engines/engine.py.326: process: 0.42679 | load: 0.00023 [I 2022-10-11 18:39:04] ...trap/engines/engine.py.321: eval: epoch 114 | batch 70/510 [I 2022-10-11 18:39:04] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:17 [I 2022-10-11 18:39:04] ...trap/engines/engine.py.326: process: 0.47856 | load: 0.00019 [I 2022-10-11 18:39:09] ...trap/engines/engine.py.321: eval: epoch 114 | batch 80/510 [I 2022-10-11 18:39:09] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:04:01 [I 2022-10-11 18:39:09] ...trap/engines/engine.py.326: process: 0.51340 | load: 0.00020 [I 2022-10-11 18:39:15] ...trap/engines/engine.py.321: eval: epoch 114 | batch 90/510 [I 2022-10-11 18:39:15] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:03:39 [I 2022-10-11 18:39:15] ...trap/engines/engine.py.326: process: 0.44930 | load: 0.00021 [I 2022-10-11 18:39:19] ...trap/engines/engine.py.321: eval: epoch 114 | batch 100/510 [I 2022-10-11 18:39:19] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:01 [I 2022-10-11 18:39:19] ...trap/engines/engine.py.326: process: 0.30650 | load: 0.00020 [I 2022-10-11 18:39:22] ...trap/engines/engine.py.321: eval: epoch 114 | batch 110/510 [I 2022-10-11 18:39:22] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:02:13 [I 2022-10-11 18:39:22] ...trap/engines/engine.py.326: process: 0.31603 | load: 0.00021 [I 2022-10-11 18:39:25] ...trap/engines/engine.py.321: eval: epoch 114 | batch 120/510 [I 2022-10-11 18:39:25] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:02:05 [I 2022-10-11 18:39:25] ...trap/engines/engine.py.326: process: 0.32413 | load: 0.00020 [I 2022-10-11 18:39:29] ...trap/engines/engine.py.321: eval: epoch 114 | batch 130/510 [I 2022-10-11 18:39:29] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:02:02 [I 2022-10-11 18:39:29] ...trap/engines/engine.py.326: process: 0.32286 | load: 0.00021 [I 2022-10-11 18:39:32] ...trap/engines/engine.py.321: eval: epoch 114 | batch 140/510 [I 2022-10-11 18:39:32] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:58 [I 2022-10-11 18:39:32] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00020 [I 2022-10-11 18:39:35] ...trap/engines/engine.py.321: eval: epoch 114 | batch 150/510 [I 2022-10-11 18:39:35] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-11 18:39:35] ...trap/engines/engine.py.326: process: 0.31890 | load: 0.00022 [I 2022-10-11 18:39:38] ...trap/engines/engine.py.321: eval: epoch 114 | batch 160/510 [I 2022-10-11 18:39:38] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:53 [I 2022-10-11 18:39:38] ...trap/engines/engine.py.326: process: 0.32582 | load: 0.00021 [I 2022-10-11 18:39:42] ...trap/engines/engine.py.321: eval: epoch 114 | batch 170/510 [I 2022-10-11 18:39:42] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:50 [I 2022-10-11 18:39:42] ...trap/engines/engine.py.326: process: 0.32917 | load: 0.00021 [I 2022-10-11 18:39:45] ...trap/engines/engine.py.321: eval: epoch 114 | batch 180/510 [I 2022-10-11 18:39:45] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:47 [I 2022-10-11 18:39:45] ...trap/engines/engine.py.326: process: 0.33824 | load: 0.00021 [I 2022-10-11 18:39:48] ...trap/engines/engine.py.321: eval: epoch 114 | batch 190/510 [I 2022-10-11 18:39:48] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:46 [I 2022-10-11 18:39:48] ...trap/engines/engine.py.326: process: 0.33660 | load: 0.00024 [I 2022-10-11 18:39:52] ...trap/engines/engine.py.321: eval: epoch 114 | batch 200/510 [I 2022-10-11 18:39:52] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:44 [I 2022-10-11 18:39:52] ...trap/engines/engine.py.326: process: 0.33481 | load: 0.00023 [I 2022-10-11 18:39:55] ...trap/engines/engine.py.321: eval: epoch 114 | batch 210/510 [I 2022-10-11 18:39:55] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:40 [I 2022-10-11 18:39:55] ...trap/engines/engine.py.326: process: 0.33035 | load: 0.00023 [I 2022-10-11 18:39:58] ...trap/engines/engine.py.321: eval: epoch 114 | batch 220/510 [I 2022-10-11 18:39:58] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:34 [I 2022-10-11 18:39:58] ...trap/engines/engine.py.326: process: 0.32162 | load: 0.00022 [I 2022-10-11 18:40:01] ...trap/engines/engine.py.321: eval: epoch 114 | batch 230/510 [I 2022-10-11 18:40:01] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:29 [I 2022-10-11 18:40:01] ...trap/engines/engine.py.326: process: 0.31990 | load: 0.00026 [I 2022-10-11 18:40:05] ...trap/engines/engine.py.321: eval: epoch 114 | batch 240/510 [I 2022-10-11 18:40:05] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:26 [I 2022-10-11 18:40:05] ...trap/engines/engine.py.326: process: 0.32138 | load: 0.00020 [I 2022-10-11 18:40:08] ...trap/engines/engine.py.321: eval: epoch 114 | batch 250/510 [I 2022-10-11 18:40:08] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:01:23 [I 2022-10-11 18:40:08] ...trap/engines/engine.py.326: process: 0.31939 | load: 0.00026 [I 2022-10-11 18:40:11] ...trap/engines/engine.py.321: eval: epoch 114 | batch 260/510 [I 2022-10-11 18:40:11] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:20 [I 2022-10-11 18:40:11] ...trap/engines/engine.py.326: process: 0.32577 | load: 0.00021 [I 2022-10-11 18:40:14] ...trap/engines/engine.py.321: eval: epoch 114 | batch 270/510 [I 2022-10-11 18:40:14] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:17 [I 2022-10-11 18:40:14] ...trap/engines/engine.py.326: process: 0.31891 | load: 0.00024 [I 2022-10-11 18:40:17] ...trap/engines/engine.py.321: eval: epoch 114 | batch 280/510 [I 2022-10-11 18:40:17] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:14 [I 2022-10-11 18:40:17] ...trap/engines/engine.py.326: process: 0.32063 | load: 0.00021 [I 2022-10-11 18:40:21] ...trap/engines/engine.py.321: eval: epoch 114 | batch 290/510 [I 2022-10-11 18:40:21] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:01:10 [I 2022-10-11 18:40:21] ...trap/engines/engine.py.326: process: 0.32084 | load: 0.00025 [I 2022-10-11 18:40:24] ...trap/engines/engine.py.321: eval: epoch 114 | batch 300/510 [I 2022-10-11 18:40:24] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:01:07 [I 2022-10-11 18:40:24] ...trap/engines/engine.py.326: process: 0.32089 | load: 0.00021 [I 2022-10-11 18:40:27] ...trap/engines/engine.py.321: eval: epoch 114 | batch 310/510 [I 2022-10-11 18:40:27] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-11 18:40:27] ...trap/engines/engine.py.326: process: 0.32380 | load: 0.00022 [I 2022-10-11 18:40:30] ...trap/engines/engine.py.321: eval: epoch 114 | batch 320/510 [I 2022-10-11 18:40:30] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:01 [I 2022-10-11 18:40:30] ...trap/engines/engine.py.326: process: 0.31677 | load: 0.00021 [I 2022-10-11 18:40:34] ...trap/engines/engine.py.321: eval: epoch 114 | batch 330/510 [I 2022-10-11 18:40:34] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:58 [I 2022-10-11 18:40:34] ...trap/engines/engine.py.326: process: 0.32393 | load: 0.00022 [I 2022-10-11 18:40:37] ...trap/engines/engine.py.321: eval: epoch 114 | batch 340/510 [I 2022-10-11 18:40:37] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:55 [I 2022-10-11 18:40:37] ...trap/engines/engine.py.326: process: 0.32477 | load: 0.00024 [I 2022-10-11 18:40:40] ...trap/engines/engine.py.321: eval: epoch 114 | batch 350/510 [I 2022-10-11 18:40:40] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:51 [I 2022-10-11 18:40:40] ...trap/engines/engine.py.326: process: 0.32343 | load: 0.00023 [I 2022-10-11 18:40:43] ...trap/engines/engine.py.321: eval: epoch 114 | batch 360/510 [I 2022-10-11 18:40:43] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-11 18:40:43] ...trap/engines/engine.py.326: process: 0.32822 | load: 0.00021 [I 2022-10-11 18:40:46] ...trap/engines/engine.py.321: eval: epoch 114 | batch 370/510 [I 2022-10-11 18:40:46] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:45 [I 2022-10-11 18:40:46] ...trap/engines/engine.py.326: process: 0.31983 | load: 0.00021 [I 2022-10-11 18:40:50] ...trap/engines/engine.py.321: eval: epoch 114 | batch 380/510 [I 2022-10-11 18:40:50] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:41 [I 2022-10-11 18:40:50] ...trap/engines/engine.py.326: process: 0.32270 | load: 0.00021 [I 2022-10-11 18:40:53] ...trap/engines/engine.py.321: eval: epoch 114 | batch 390/510 [I 2022-10-11 18:40:53] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:38 [I 2022-10-11 18:40:53] ...trap/engines/engine.py.326: process: 0.32425 | load: 0.00024 [I 2022-10-11 18:40:56] ...trap/engines/engine.py.321: eval: epoch 114 | batch 400/510 [I 2022-10-11 18:40:56] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-11 18:40:56] ...trap/engines/engine.py.326: process: 0.32813 | load: 0.00023 [I 2022-10-11 18:40:59] ...trap/engines/engine.py.321: eval: epoch 114 | batch 410/510 [I 2022-10-11 18:40:59] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-11 18:40:59] ...trap/engines/engine.py.326: process: 0.32194 | load: 0.00022 [I 2022-10-11 18:41:03] ...trap/engines/engine.py.321: eval: epoch 114 | batch 420/510 [I 2022-10-11 18:41:03] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:28 [I 2022-10-11 18:41:03] ...trap/engines/engine.py.326: process: 0.32380 | load: 0.00022 [I 2022-10-11 18:41:06] ...trap/engines/engine.py.321: eval: epoch 114 | batch 430/510 [I 2022-10-11 18:41:06] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:25 [I 2022-10-11 18:41:06] ...trap/engines/engine.py.326: process: 0.33050 | load: 0.00023 [I 2022-10-11 18:41:09] ...trap/engines/engine.py.321: eval: epoch 114 | batch 440/510 [I 2022-10-11 18:41:09] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-11 18:41:09] ...trap/engines/engine.py.326: process: 0.32000 | load: 0.00022 [I 2022-10-11 18:41:12] ...trap/engines/engine.py.321: eval: epoch 114 | batch 450/510 [I 2022-10-11 18:41:12] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-11 18:41:12] ...trap/engines/engine.py.326: process: 0.32474 | load: 0.00024 [I 2022-10-11 18:41:15] ...trap/engines/engine.py.321: eval: epoch 114 | batch 460/510 [I 2022-10-11 18:41:15] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:16 [I 2022-10-11 18:41:15] ...trap/engines/engine.py.326: process: 0.33693 | load: 0.00022 [I 2022-10-11 18:41:19] ...trap/engines/engine.py.321: eval: epoch 114 | batch 470/510 [I 2022-10-11 18:41:19] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:12 [I 2022-10-11 18:41:19] ...trap/engines/engine.py.326: process: 0.32011 | load: 0.00022 [I 2022-10-11 18:41:22] ...trap/engines/engine.py.321: eval: epoch 114 | batch 480/510 [I 2022-10-11 18:41:22] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-11 18:41:22] ...trap/engines/engine.py.326: process: 0.32134 | load: 0.00021 [I 2022-10-11 18:41:25] ...trap/engines/engine.py.321: eval: epoch 114 | batch 490/510 [I 2022-10-11 18:41:25] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-11 18:41:25] ...trap/engines/engine.py.326: process: 0.32175 | load: 0.00021 [I 2022-10-11 18:41:28] ...trap/engines/engine.py.321: eval: epoch 114 | batch 500/510 [I 2022-10-11 18:41:28] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-11 18:41:28] ...trap/engines/engine.py.326: process: 0.32380 | load: 0.00021 [I 2022-10-11 18:41:32] ...trap/engines/engine.py.321: eval: epoch 114 | batch 510/510 [I 2022-10-11 18:41:32] ...trap/engines/engine.py.325: elapsed: 0:03:15 | left: 0:00:00 [I 2022-10-11 18:41:32] ...trap/engines/engine.py.326: process: 0.31149 | load: 0.00017 [S 2022-10-11 18:41:32] ...trap/engines/engine.py.132: eval_epoch.epoch: 114 [I 2022-10-11 18:41:32] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.747500 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907300 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933700 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.740000 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906300 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009912 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009023 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009870 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016291 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010050 [S 2022-10-11 18:42:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007294 [I 2022-10-11 18:42:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 18:42:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 18:42:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 18:42:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 18:42:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 18:42:26] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 18:42:30] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 18:42:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 18:42:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 115 [I 2022-10-11 18:42:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 18:42:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 0/2304 [I 2022-10-11 18:42:37] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:09:13 [I 2022-10-11 18:42:37] ...trap/engines/engine.py.229: process: 3.36534 | load: 2.45918 [I 2022-10-11 18:42:37] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-11 18:42:37] ...optimizers/trijoint.py.151: total_norm: 14.686269 [I 2022-10-11 18:42:45] ...trap/engines/engine.py.224: train: epoch 115 | batch 10/2304 [I 2022-10-11 18:42:45] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:48 [I 2022-10-11 18:42:45] ...trap/engines/engine.py.229: process: 0.85984 | load: 0.00486 [I 2022-10-11 18:42:45] ...trap/engines/engine.py.230: loss: 0.07565 [I 2022-10-11 18:42:45] ...optimizers/trijoint.py.151: total_norm: 13.394905 [I 2022-10-11 18:42:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 20/2304 [I 2022-10-11 18:42:54] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:11 [I 2022-10-11 18:42:54] ...trap/engines/engine.py.229: process: 0.83988 | load: 0.00442 [I 2022-10-11 18:42:54] ...trap/engines/engine.py.230: loss: 0.09467 [I 2022-10-11 18:42:54] ...optimizers/trijoint.py.151: total_norm: 13.229263 [I 2022-10-11 18:43:02] ...trap/engines/engine.py.224: train: epoch 115 | batch 30/2304 [I 2022-10-11 18:43:02] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:01 [I 2022-10-11 18:43:02] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00354 [I 2022-10-11 18:43:02] ...trap/engines/engine.py.230: loss: 0.08152 [I 2022-10-11 18:43:02] ...optimizers/trijoint.py.151: total_norm: 14.123884 [I 2022-10-11 18:43:10] ...trap/engines/engine.py.224: train: epoch 115 | batch 40/2304 [I 2022-10-11 18:43:10] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:31:45 [I 2022-10-11 18:43:10] ...trap/engines/engine.py.229: process: 0.83825 | load: 0.00404 [I 2022-10-11 18:43:10] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-11 18:43:10] ...optimizers/trijoint.py.151: total_norm: 13.129058 [I 2022-10-11 18:43:19] ...trap/engines/engine.py.224: train: epoch 115 | batch 50/2304 [I 2022-10-11 18:43:19] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:47 [I 2022-10-11 18:43:19] ...trap/engines/engine.py.229: process: 0.86018 | load: 0.00397 [I 2022-10-11 18:43:19] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-11 18:43:19] ...optimizers/trijoint.py.151: total_norm: 12.554317 [I 2022-10-11 18:43:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 60/2304 [I 2022-10-11 18:43:27] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:24 [I 2022-10-11 18:43:27] ...trap/engines/engine.py.229: process: 0.83743 | load: 0.00366 [I 2022-10-11 18:43:27] ...trap/engines/engine.py.230: loss: 0.10855 [I 2022-10-11 18:43:27] ...optimizers/trijoint.py.151: total_norm: 11.569965 [I 2022-10-11 18:43:36] ...trap/engines/engine.py.224: train: epoch 115 | batch 70/2304 [I 2022-10-11 18:43:36] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:18 [I 2022-10-11 18:43:36] ...trap/engines/engine.py.229: process: 0.84530 | load: 0.00358 [I 2022-10-11 18:43:36] ...trap/engines/engine.py.230: loss: 0.10104 [I 2022-10-11 18:43:36] ...optimizers/trijoint.py.151: total_norm: 10.974992 [I 2022-10-11 18:43:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 80/2304 [I 2022-10-11 18:43:44] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:31:08 [I 2022-10-11 18:43:44] ...trap/engines/engine.py.229: process: 0.83943 | load: 0.00357 [I 2022-10-11 18:43:44] ...trap/engines/engine.py.230: loss: 0.12058 [I 2022-10-11 18:43:44] ...optimizers/trijoint.py.151: total_norm: 12.724812 [I 2022-10-11 18:43:53] ...trap/engines/engine.py.224: train: epoch 115 | batch 90/2304 [I 2022-10-11 18:43:53] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:30:53 [I 2022-10-11 18:43:53] ...trap/engines/engine.py.229: process: 0.84707 | load: 0.00327 [I 2022-10-11 18:43:53] ...trap/engines/engine.py.230: loss: 0.08755 [I 2022-10-11 18:43:53] ...optimizers/trijoint.py.151: total_norm: 8.234410 [I 2022-10-11 18:44:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 100/2304 [I 2022-10-11 18:44:01] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:30:46 [I 2022-10-11 18:44:01] ...trap/engines/engine.py.229: process: 0.83519 | load: 0.00319 [I 2022-10-11 18:44:01] ...trap/engines/engine.py.230: loss: 0.15895 [I 2022-10-11 18:44:01] ...optimizers/trijoint.py.151: total_norm: 11.617998 [I 2022-10-11 18:44:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 110/2304 [I 2022-10-11 18:44:09] ...trap/engines/engine.py.228: elapsed: 0:01:36 | left: 0:30:51 [I 2022-10-11 18:44:09] ...trap/engines/engine.py.229: process: 0.83379 | load: 0.00344 [I 2022-10-11 18:44:09] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-11 18:44:09] ...optimizers/trijoint.py.151: total_norm: 20.037373 [I 2022-10-11 18:44:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 120/2304 [I 2022-10-11 18:44:18] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:30:35 [I 2022-10-11 18:44:18] ...trap/engines/engine.py.229: process: 0.83788 | load: 0.00357 [I 2022-10-11 18:44:18] ...trap/engines/engine.py.230: loss: 0.06814 [I 2022-10-11 18:44:18] ...optimizers/trijoint.py.151: total_norm: 18.199417 [I 2022-10-11 18:44:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 130/2304 [I 2022-10-11 18:44:26] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:30:32 [I 2022-10-11 18:44:26] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00297 [I 2022-10-11 18:44:26] ...trap/engines/engine.py.230: loss: 0.13217 [I 2022-10-11 18:44:26] ...optimizers/trijoint.py.151: total_norm: 13.352121 [I 2022-10-11 18:44:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 140/2304 [I 2022-10-11 18:44:35] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:30:53 [I 2022-10-11 18:44:35] ...trap/engines/engine.py.229: process: 0.85825 | load: 0.00310 [I 2022-10-11 18:44:35] ...trap/engines/engine.py.230: loss: 0.09430 [I 2022-10-11 18:44:35] ...optimizers/trijoint.py.151: total_norm: 10.170432 [I 2022-10-11 18:44:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 150/2304 [I 2022-10-11 18:44:43] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:30:20 [I 2022-10-11 18:44:43] ...trap/engines/engine.py.229: process: 0.83601 | load: 0.00308 [I 2022-10-11 18:44:43] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-11 18:44:43] ...optimizers/trijoint.py.151: total_norm: 12.402421 [I 2022-10-11 18:44:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 160/2304 [I 2022-10-11 18:44:52] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:30:03 [I 2022-10-11 18:44:52] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00321 [I 2022-10-11 18:44:52] ...trap/engines/engine.py.230: loss: 0.10364 [I 2022-10-11 18:44:52] ...optimizers/trijoint.py.151: total_norm: 15.159328 [I 2022-10-11 18:45:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 170/2304 [I 2022-10-11 18:45:00] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:29:58 [I 2022-10-11 18:45:00] ...trap/engines/engine.py.229: process: 0.83039 | load: 0.00326 [I 2022-10-11 18:45:00] ...trap/engines/engine.py.230: loss: 0.11646 [I 2022-10-11 18:45:00] ...optimizers/trijoint.py.151: total_norm: 13.829684 [I 2022-10-11 18:45:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 180/2304 [I 2022-10-11 18:45:09] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:29:39 [I 2022-10-11 18:45:09] ...trap/engines/engine.py.229: process: 0.84518 | load: 0.00300 [I 2022-10-11 18:45:09] ...trap/engines/engine.py.230: loss: 0.11031 [I 2022-10-11 18:45:09] ...optimizers/trijoint.py.151: total_norm: 13.610455 [I 2022-10-11 18:45:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 190/2304 [I 2022-10-11 18:45:17] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:29:58 [I 2022-10-11 18:45:17] ...trap/engines/engine.py.229: process: 0.84760 | load: 0.00334 [I 2022-10-11 18:45:17] ...trap/engines/engine.py.230: loss: 0.10727 [I 2022-10-11 18:45:17] ...optimizers/trijoint.py.151: total_norm: 19.251774 [I 2022-10-11 18:45:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 200/2304 [I 2022-10-11 18:45:26] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:29:27 [I 2022-10-11 18:45:26] ...trap/engines/engine.py.229: process: 0.84028 | load: 0.00314 [I 2022-10-11 18:45:26] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-11 18:45:26] ...optimizers/trijoint.py.151: total_norm: 24.847305 [I 2022-10-11 18:45:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 210/2304 [I 2022-10-11 18:45:34] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:29:37 [I 2022-10-11 18:45:34] ...trap/engines/engine.py.229: process: 0.85895 | load: 0.00314 [I 2022-10-11 18:45:34] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-11 18:45:34] ...optimizers/trijoint.py.151: total_norm: 13.146960 [I 2022-10-11 18:45:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 220/2304 [I 2022-10-11 18:45:43] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:29:22 [I 2022-10-11 18:45:43] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00315 [I 2022-10-11 18:45:43] ...trap/engines/engine.py.230: loss: 0.13158 [I 2022-10-11 18:45:43] ...optimizers/trijoint.py.151: total_norm: 25.141357 [I 2022-10-11 18:45:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 230/2304 [I 2022-10-11 18:45:51] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:29:46 [I 2022-10-11 18:45:51] ...trap/engines/engine.py.229: process: 0.86550 | load: 0.00324 [I 2022-10-11 18:45:51] ...trap/engines/engine.py.230: loss: 0.09086 [I 2022-10-11 18:45:51] ...optimizers/trijoint.py.151: total_norm: 11.260345 [I 2022-10-11 18:46:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 240/2304 [I 2022-10-11 18:46:00] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:29:45 [I 2022-10-11 18:46:00] ...trap/engines/engine.py.229: process: 0.86488 | load: 0.00312 [I 2022-10-11 18:46:00] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-11 18:46:00] ...optimizers/trijoint.py.151: total_norm: 23.014467 [I 2022-10-11 18:46:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 250/2304 [I 2022-10-11 18:46:09] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:29:21 [I 2022-10-11 18:46:09] ...trap/engines/engine.py.229: process: 0.86290 | load: 0.00317 [I 2022-10-11 18:46:09] ...trap/engines/engine.py.230: loss: 0.09700 [I 2022-10-11 18:46:09] ...optimizers/trijoint.py.151: total_norm: 10.125741 [I 2022-10-11 18:46:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 260/2304 [I 2022-10-11 18:46:17] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:29:13 [I 2022-10-11 18:46:17] ...trap/engines/engine.py.229: process: 0.85557 | load: 0.00324 [I 2022-10-11 18:46:17] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-11 18:46:17] ...optimizers/trijoint.py.151: total_norm: 21.466553 [I 2022-10-11 18:46:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 270/2304 [I 2022-10-11 18:46:26] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:28:48 [I 2022-10-11 18:46:26] ...trap/engines/engine.py.229: process: 0.84093 | load: 0.00329 [I 2022-10-11 18:46:26] ...trap/engines/engine.py.230: loss: 0.10104 [I 2022-10-11 18:46:26] ...optimizers/trijoint.py.151: total_norm: 14.458190 [I 2022-10-11 18:46:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 280/2304 [I 2022-10-11 18:46:34] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:29:10 [I 2022-10-11 18:46:34] ...trap/engines/engine.py.229: process: 0.87528 | load: 0.00336 [I 2022-10-11 18:46:34] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-11 18:46:34] ...optimizers/trijoint.py.151: total_norm: 15.443297 [I 2022-10-11 18:46:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 290/2304 [I 2022-10-11 18:46:43] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:29:00 [I 2022-10-11 18:46:43] ...trap/engines/engine.py.229: process: 0.86441 | load: 0.00330 [I 2022-10-11 18:46:43] ...trap/engines/engine.py.230: loss: 0.08635 [I 2022-10-11 18:46:43] ...optimizers/trijoint.py.151: total_norm: 15.071886 [I 2022-10-11 18:46:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 300/2304 [I 2022-10-11 18:46:52] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:28:38 [I 2022-10-11 18:46:52] ...trap/engines/engine.py.229: process: 0.85578 | load: 0.00331 [I 2022-10-11 18:46:52] ...trap/engines/engine.py.230: loss: 0.13891 [I 2022-10-11 18:46:52] ...optimizers/trijoint.py.151: total_norm: 29.962770 [I 2022-10-11 18:47:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 310/2304 [I 2022-10-11 18:47:00] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:28:15 [I 2022-10-11 18:47:00] ...trap/engines/engine.py.229: process: 0.85078 | load: 0.00334 [I 2022-10-11 18:47:00] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-11 18:47:00] ...optimizers/trijoint.py.151: total_norm: 13.134983 [I 2022-10-11 18:47:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 320/2304 [I 2022-10-11 18:47:09] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:28:16 [I 2022-10-11 18:47:09] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00532 [I 2022-10-11 18:47:09] ...trap/engines/engine.py.230: loss: 0.09760 [I 2022-10-11 18:47:09] ...optimizers/trijoint.py.151: total_norm: 9.984984 [I 2022-10-11 18:47:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 330/2304 [I 2022-10-11 18:47:17] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:28:19 [I 2022-10-11 18:47:17] ...trap/engines/engine.py.229: process: 0.87307 | load: 0.00315 [I 2022-10-11 18:47:17] ...trap/engines/engine.py.230: loss: 0.12662 [I 2022-10-11 18:47:17] ...optimizers/trijoint.py.151: total_norm: 22.381460 [I 2022-10-11 18:47:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 340/2304 [I 2022-10-11 18:47:26] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:28:00 [I 2022-10-11 18:47:26] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00320 [I 2022-10-11 18:47:26] ...trap/engines/engine.py.230: loss: 0.09048 [I 2022-10-11 18:47:26] ...optimizers/trijoint.py.151: total_norm: 17.633232 [I 2022-10-11 18:47:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 350/2304 [I 2022-10-11 18:47:34] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:27:48 [I 2022-10-11 18:47:34] ...trap/engines/engine.py.229: process: 0.86020 | load: 0.00314 [I 2022-10-11 18:47:34] ...trap/engines/engine.py.230: loss: 0.11684 [I 2022-10-11 18:47:34] ...optimizers/trijoint.py.151: total_norm: 13.078381 [I 2022-10-11 18:47:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 360/2304 [I 2022-10-11 18:47:43] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:27:54 [I 2022-10-11 18:47:43] ...trap/engines/engine.py.229: process: 0.87664 | load: 0.00312 [I 2022-10-11 18:47:43] ...trap/engines/engine.py.230: loss: 0.09457 [I 2022-10-11 18:47:43] ...optimizers/trijoint.py.151: total_norm: 12.053123 [I 2022-10-11 18:47:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 370/2304 [I 2022-10-11 18:47:52] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:27:37 [I 2022-10-11 18:47:52] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00328 [I 2022-10-11 18:47:52] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-11 18:47:52] ...optimizers/trijoint.py.151: total_norm: 24.527781 [I 2022-10-11 18:48:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 380/2304 [I 2022-10-11 18:48:00] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:27:31 [I 2022-10-11 18:48:00] ...trap/engines/engine.py.229: process: 0.87057 | load: 0.00316 [I 2022-10-11 18:48:00] ...trap/engines/engine.py.230: loss: 0.05773 [I 2022-10-11 18:48:00] ...optimizers/trijoint.py.151: total_norm: 15.285199 [I 2022-10-11 18:48:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 390/2304 [I 2022-10-11 18:48:09] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:27:16 [I 2022-10-11 18:48:09] ...trap/engines/engine.py.229: process: 0.84319 | load: 0.00313 [I 2022-10-11 18:48:09] ...trap/engines/engine.py.230: loss: 0.10158 [I 2022-10-11 18:48:09] ...optimizers/trijoint.py.151: total_norm: 12.150745 [I 2022-10-11 18:48:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 400/2304 [I 2022-10-11 18:48:17] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:27:35 [I 2022-10-11 18:48:17] ...trap/engines/engine.py.229: process: 0.88399 | load: 0.00395 [I 2022-10-11 18:48:17] ...trap/engines/engine.py.230: loss: 0.08748 [I 2022-10-11 18:48:17] ...optimizers/trijoint.py.151: total_norm: 12.417397 [I 2022-10-11 18:48:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 410/2304 [I 2022-10-11 18:48:26] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:26:52 [I 2022-10-11 18:48:26] ...trap/engines/engine.py.229: process: 0.83175 | load: 0.00315 [I 2022-10-11 18:48:26] ...trap/engines/engine.py.230: loss: 0.18256 [I 2022-10-11 18:48:26] ...optimizers/trijoint.py.151: total_norm: 15.312136 [I 2022-10-11 18:48:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 420/2304 [I 2022-10-11 18:48:35] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:26:47 [I 2022-10-11 18:48:35] ...trap/engines/engine.py.229: process: 0.85533 | load: 0.00317 [I 2022-10-11 18:48:35] ...trap/engines/engine.py.230: loss: 0.13559 [I 2022-10-11 18:48:35] ...optimizers/trijoint.py.151: total_norm: 13.536282 [I 2022-10-11 18:48:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 430/2304 [I 2022-10-11 18:48:43] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:26:48 [I 2022-10-11 18:48:43] ...trap/engines/engine.py.229: process: 0.87030 | load: 0.00313 [I 2022-10-11 18:48:43] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-11 18:48:43] ...optimizers/trijoint.py.151: total_norm: 9.863930 [I 2022-10-11 18:48:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 440/2304 [I 2022-10-11 18:48:52] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:26:37 [I 2022-10-11 18:48:52] ...trap/engines/engine.py.229: process: 0.86580 | load: 0.00311 [I 2022-10-11 18:48:52] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-11 18:48:52] ...optimizers/trijoint.py.151: total_norm: 12.166073 [I 2022-10-11 18:49:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 450/2304 [I 2022-10-11 18:49:00] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:26:13 [I 2022-10-11 18:49:00] ...trap/engines/engine.py.229: process: 0.83714 | load: 0.00313 [I 2022-10-11 18:49:00] ...trap/engines/engine.py.230: loss: 0.08827 [I 2022-10-11 18:49:00] ...optimizers/trijoint.py.151: total_norm: 10.079025 [I 2022-10-11 18:49:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 460/2304 [I 2022-10-11 18:49:09] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:26:19 [I 2022-10-11 18:49:09] ...trap/engines/engine.py.229: process: 0.86591 | load: 0.00314 [I 2022-10-11 18:49:09] ...trap/engines/engine.py.230: loss: 0.07931 [I 2022-10-11 18:49:09] ...optimizers/trijoint.py.151: total_norm: 14.141380 [I 2022-10-11 18:49:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 470/2304 [I 2022-10-11 18:49:17] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:26:30 [I 2022-10-11 18:49:17] ...trap/engines/engine.py.229: process: 0.86798 | load: 0.00312 [I 2022-10-11 18:49:17] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-11 18:49:17] ...optimizers/trijoint.py.151: total_norm: 41.239758 [I 2022-10-11 18:49:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 480/2304 [I 2022-10-11 18:49:26] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:26:04 [I 2022-10-11 18:49:26] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00315 [I 2022-10-11 18:49:26] ...trap/engines/engine.py.230: loss: 0.13194 [I 2022-10-11 18:49:26] ...optimizers/trijoint.py.151: total_norm: 14.861547 [I 2022-10-11 18:49:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 490/2304 [I 2022-10-11 18:49:35] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:25:54 [I 2022-10-11 18:49:35] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00316 [I 2022-10-11 18:49:35] ...trap/engines/engine.py.230: loss: 0.11437 [I 2022-10-11 18:49:35] ...optimizers/trijoint.py.151: total_norm: 9.144959 [I 2022-10-11 18:49:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 500/2304 [I 2022-10-11 18:49:43] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:26:26 [I 2022-10-11 18:49:43] ...trap/engines/engine.py.229: process: 0.89179 | load: 0.00391 [I 2022-10-11 18:49:43] ...trap/engines/engine.py.230: loss: 0.06603 [I 2022-10-11 18:49:43] ...optimizers/trijoint.py.151: total_norm: 17.251532 [I 2022-10-11 18:49:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 510/2304 [I 2022-10-11 18:49:52] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:25:42 [I 2022-10-11 18:49:52] ...trap/engines/engine.py.229: process: 0.83244 | load: 0.00296 [I 2022-10-11 18:49:52] ...trap/engines/engine.py.230: loss: 0.10001 [I 2022-10-11 18:49:52] ...optimizers/trijoint.py.151: total_norm: 11.683938 [I 2022-10-11 18:50:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 520/2304 [I 2022-10-11 18:50:01] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:25:12 [I 2022-10-11 18:50:01] ...trap/engines/engine.py.229: process: 0.83215 | load: 0.00301 [I 2022-10-11 18:50:01] ...trap/engines/engine.py.230: loss: 0.10380 [I 2022-10-11 18:50:01] ...optimizers/trijoint.py.151: total_norm: 16.767614 [I 2022-10-11 18:50:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 530/2304 [I 2022-10-11 18:50:09] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:24:52 [I 2022-10-11 18:50:09] ...trap/engines/engine.py.229: process: 0.83639 | load: 0.00295 [I 2022-10-11 18:50:09] ...trap/engines/engine.py.230: loss: 0.12375 [I 2022-10-11 18:50:09] ...optimizers/trijoint.py.151: total_norm: 12.374516 [I 2022-10-11 18:50:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 540/2304 [I 2022-10-11 18:50:17] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:24:50 [I 2022-10-11 18:50:17] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00294 [I 2022-10-11 18:50:17] ...trap/engines/engine.py.230: loss: 0.10469 [I 2022-10-11 18:50:17] ...optimizers/trijoint.py.151: total_norm: 23.054434 [I 2022-10-11 18:50:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 550/2304 [I 2022-10-11 18:50:26] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:24:43 [I 2022-10-11 18:50:26] ...trap/engines/engine.py.229: process: 0.84486 | load: 0.00293 [I 2022-10-11 18:50:26] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-11 18:50:26] ...optimizers/trijoint.py.151: total_norm: 33.040081 [I 2022-10-11 18:50:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 560/2304 [I 2022-10-11 18:50:34] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:24:22 [I 2022-10-11 18:50:34] ...trap/engines/engine.py.229: process: 0.83345 | load: 0.00296 [I 2022-10-11 18:50:34] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-11 18:50:34] ...optimizers/trijoint.py.151: total_norm: 12.162786 [I 2022-10-11 18:50:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 570/2304 [I 2022-10-11 18:50:43] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:24:25 [I 2022-10-11 18:50:43] ...trap/engines/engine.py.229: process: 0.85026 | load: 0.00295 [I 2022-10-11 18:50:43] ...trap/engines/engine.py.230: loss: 0.15102 [I 2022-10-11 18:50:43] ...optimizers/trijoint.py.151: total_norm: 16.119097 [I 2022-10-11 18:50:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 580/2304 [I 2022-10-11 18:50:51] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:24:16 [I 2022-10-11 18:50:51] ...trap/engines/engine.py.229: process: 0.84415 | load: 0.00296 [I 2022-10-11 18:50:51] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-11 18:50:51] ...optimizers/trijoint.py.151: total_norm: 36.949020 [I 2022-10-11 18:51:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 590/2304 [I 2022-10-11 18:51:00] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:24:04 [I 2022-10-11 18:51:00] ...trap/engines/engine.py.229: process: 0.83513 | load: 0.00293 [I 2022-10-11 18:51:00] ...trap/engines/engine.py.230: loss: 0.14488 [I 2022-10-11 18:51:00] ...optimizers/trijoint.py.151: total_norm: 20.050249 [I 2022-10-11 18:51:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 600/2304 [I 2022-10-11 18:51:08] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:23:58 [I 2022-10-11 18:51:08] ...trap/engines/engine.py.229: process: 0.85263 | load: 0.00298 [I 2022-10-11 18:51:08] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-11 18:51:08] ...optimizers/trijoint.py.151: total_norm: 9.154637 [I 2022-10-11 18:51:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 610/2304 [I 2022-10-11 18:51:17] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:23:42 [I 2022-10-11 18:51:17] ...trap/engines/engine.py.229: process: 0.83460 | load: 0.00294 [I 2022-10-11 18:51:17] ...trap/engines/engine.py.230: loss: 0.08423 [I 2022-10-11 18:51:17] ...optimizers/trijoint.py.151: total_norm: 12.832866 [I 2022-10-11 18:51:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 620/2304 [I 2022-10-11 18:51:25] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:23:34 [I 2022-10-11 18:51:25] ...trap/engines/engine.py.229: process: 0.84125 | load: 0.00295 [I 2022-10-11 18:51:25] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-11 18:51:25] ...optimizers/trijoint.py.151: total_norm: 12.247431 [I 2022-10-11 18:51:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 630/2304 [I 2022-10-11 18:51:33] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:23:39 [I 2022-10-11 18:51:33] ...trap/engines/engine.py.229: process: 0.86238 | load: 0.00298 [I 2022-10-11 18:51:33] ...trap/engines/engine.py.230: loss: 0.18587 [I 2022-10-11 18:51:33] ...optimizers/trijoint.py.151: total_norm: 10.801972 [I 2022-10-11 18:51:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 640/2304 [I 2022-10-11 18:51:42] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:23:22 [I 2022-10-11 18:51:42] ...trap/engines/engine.py.229: process: 0.85419 | load: 0.00299 [I 2022-10-11 18:51:42] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-11 18:51:42] ...optimizers/trijoint.py.151: total_norm: 16.921549 [I 2022-10-11 18:51:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 650/2304 [I 2022-10-11 18:51:50] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:23:11 [I 2022-10-11 18:51:50] ...trap/engines/engine.py.229: process: 0.85362 | load: 0.00304 [I 2022-10-11 18:51:50] ...trap/engines/engine.py.230: loss: 0.12161 [I 2022-10-11 18:51:50] ...optimizers/trijoint.py.151: total_norm: 12.830993 [I 2022-10-11 18:51:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 660/2304 [I 2022-10-11 18:51:59] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:23:25 [I 2022-10-11 18:51:59] ...trap/engines/engine.py.229: process: 0.86699 | load: 0.00297 [I 2022-10-11 18:51:59] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-11 18:51:59] ...optimizers/trijoint.py.151: total_norm: 9.166863 [I 2022-10-11 18:52:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 670/2304 [I 2022-10-11 18:52:07] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:23:18 [I 2022-10-11 18:52:07] ...trap/engines/engine.py.229: process: 0.86376 | load: 0.00299 [I 2022-10-11 18:52:07] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-11 18:52:07] ...optimizers/trijoint.py.151: total_norm: 9.837419 [I 2022-10-11 18:52:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 680/2304 [I 2022-10-11 18:52:16] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:23:05 [I 2022-10-11 18:52:16] ...trap/engines/engine.py.229: process: 0.85752 | load: 0.00303 [I 2022-10-11 18:52:16] ...trap/engines/engine.py.230: loss: 0.09546 [I 2022-10-11 18:52:16] ...optimizers/trijoint.py.151: total_norm: 11.031026 [I 2022-10-11 18:52:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 690/2304 [I 2022-10-11 18:52:25] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:23:07 [I 2022-10-11 18:52:25] ...trap/engines/engine.py.229: process: 0.85321 | load: 0.00295 [I 2022-10-11 18:52:25] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-11 18:52:25] ...optimizers/trijoint.py.151: total_norm: 17.995399 [I 2022-10-11 18:52:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 700/2304 [I 2022-10-11 18:52:33] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:23:10 [I 2022-10-11 18:52:33] ...trap/engines/engine.py.229: process: 0.87779 | load: 0.00381 [I 2022-10-11 18:52:33] ...trap/engines/engine.py.230: loss: 0.10349 [I 2022-10-11 18:52:33] ...optimizers/trijoint.py.151: total_norm: 35.663303 [I 2022-10-11 18:52:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 710/2304 [I 2022-10-11 18:52:42] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:23:11 [I 2022-10-11 18:52:42] ...trap/engines/engine.py.229: process: 0.87657 | load: 0.00383 [I 2022-10-11 18:52:42] ...trap/engines/engine.py.230: loss: 0.16773 [I 2022-10-11 18:52:42] ...optimizers/trijoint.py.151: total_norm: 9.542125 [I 2022-10-11 18:52:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 720/2304 [I 2022-10-11 18:52:51] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:22:48 [I 2022-10-11 18:52:51] ...trap/engines/engine.py.229: process: 0.84425 | load: 0.00304 [I 2022-10-11 18:52:51] ...trap/engines/engine.py.230: loss: 0.09825 [I 2022-10-11 18:52:51] ...optimizers/trijoint.py.151: total_norm: 16.906816 [I 2022-10-11 18:52:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 730/2304 [I 2022-10-11 18:52:59] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:22:55 [I 2022-10-11 18:52:59] ...trap/engines/engine.py.229: process: 0.85255 | load: 0.00328 [I 2022-10-11 18:52:59] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-11 18:52:59] ...optimizers/trijoint.py.151: total_norm: 15.992455 [I 2022-10-11 18:53:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 740/2304 [I 2022-10-11 18:53:08] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:22:20 [I 2022-10-11 18:53:08] ...trap/engines/engine.py.229: process: 0.85389 | load: 0.00313 [I 2022-10-11 18:53:08] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-11 18:53:08] ...optimizers/trijoint.py.151: total_norm: 21.997326 [I 2022-10-11 18:53:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 750/2304 [I 2022-10-11 18:53:17] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:22:15 [I 2022-10-11 18:53:17] ...trap/engines/engine.py.229: process: 0.87318 | load: 0.00314 [I 2022-10-11 18:53:17] ...trap/engines/engine.py.230: loss: 0.13280 [I 2022-10-11 18:53:17] ...optimizers/trijoint.py.151: total_norm: 14.077078 [I 2022-10-11 18:53:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 760/2304 [I 2022-10-11 18:53:25] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:22:00 [I 2022-10-11 18:53:25] ...trap/engines/engine.py.229: process: 0.84165 | load: 0.00317 [I 2022-10-11 18:53:25] ...trap/engines/engine.py.230: loss: 0.09340 [I 2022-10-11 18:53:25] ...optimizers/trijoint.py.151: total_norm: 18.582304 [I 2022-10-11 18:53:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 770/2304 [I 2022-10-11 18:53:34] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:21:49 [I 2022-10-11 18:53:34] ...trap/engines/engine.py.229: process: 0.84695 | load: 0.00311 [I 2022-10-11 18:53:34] ...trap/engines/engine.py.230: loss: 0.09482 [I 2022-10-11 18:53:34] ...optimizers/trijoint.py.151: total_norm: 11.190538 [I 2022-10-11 18:53:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 780/2304 [I 2022-10-11 18:53:42] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:21:42 [I 2022-10-11 18:53:42] ...trap/engines/engine.py.229: process: 0.84831 | load: 0.00314 [I 2022-10-11 18:53:42] ...trap/engines/engine.py.230: loss: 0.10351 [I 2022-10-11 18:53:42] ...optimizers/trijoint.py.151: total_norm: 17.176577 [I 2022-10-11 18:53:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 790/2304 [I 2022-10-11 18:53:51] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:21:34 [I 2022-10-11 18:53:51] ...trap/engines/engine.py.229: process: 0.85945 | load: 0.00314 [I 2022-10-11 18:53:51] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-11 18:53:51] ...optimizers/trijoint.py.151: total_norm: 30.430080 [I 2022-10-11 18:53:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 800/2304 [I 2022-10-11 18:53:59] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:21:23 [I 2022-10-11 18:53:59] ...trap/engines/engine.py.229: process: 0.85698 | load: 0.00312 [I 2022-10-11 18:53:59] ...trap/engines/engine.py.230: loss: 0.10302 [I 2022-10-11 18:53:59] ...optimizers/trijoint.py.151: total_norm: 8.276663 [I 2022-10-11 18:54:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 810/2304 [I 2022-10-11 18:54:08] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:21:21 [I 2022-10-11 18:54:08] ...trap/engines/engine.py.229: process: 0.85799 | load: 0.00314 [I 2022-10-11 18:54:08] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-11 18:54:08] ...optimizers/trijoint.py.151: total_norm: 23.767065 [I 2022-10-11 18:54:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 820/2304 [I 2022-10-11 18:54:17] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:21:18 [I 2022-10-11 18:54:17] ...trap/engines/engine.py.229: process: 0.87395 | load: 0.00313 [I 2022-10-11 18:54:17] ...trap/engines/engine.py.230: loss: 0.08670 [I 2022-10-11 18:54:17] ...optimizers/trijoint.py.151: total_norm: 14.503785 [I 2022-10-11 18:54:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 830/2304 [I 2022-10-11 18:54:25] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:21:15 [I 2022-10-11 18:54:25] ...trap/engines/engine.py.229: process: 0.85733 | load: 0.00313 [I 2022-10-11 18:54:25] ...trap/engines/engine.py.230: loss: 0.11551 [I 2022-10-11 18:54:25] ...optimizers/trijoint.py.151: total_norm: 14.738727 [I 2022-10-11 18:54:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 840/2304 [I 2022-10-11 18:54:34] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:21:04 [I 2022-10-11 18:54:34] ...trap/engines/engine.py.229: process: 0.87529 | load: 0.00314 [I 2022-10-11 18:54:34] ...trap/engines/engine.py.230: loss: 0.10355 [I 2022-10-11 18:54:34] ...optimizers/trijoint.py.151: total_norm: 11.564636 [I 2022-10-11 18:54:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 850/2304 [I 2022-10-11 18:54:43] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:20:49 [I 2022-10-11 18:54:43] ...trap/engines/engine.py.229: process: 0.84951 | load: 0.00317 [I 2022-10-11 18:54:43] ...trap/engines/engine.py.230: loss: 0.09165 [I 2022-10-11 18:54:43] ...optimizers/trijoint.py.151: total_norm: 15.792703 [I 2022-10-11 18:54:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 860/2304 [I 2022-10-11 18:54:51] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:20:37 [I 2022-10-11 18:54:51] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00313 [I 2022-10-11 18:54:51] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-11 18:54:51] ...optimizers/trijoint.py.151: total_norm: 11.909833 [I 2022-10-11 18:55:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 870/2304 [I 2022-10-11 18:55:00] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:20:37 [I 2022-10-11 18:55:00] ...trap/engines/engine.py.229: process: 0.85721 | load: 0.00315 [I 2022-10-11 18:55:00] ...trap/engines/engine.py.230: loss: 0.09500 [I 2022-10-11 18:55:00] ...optimizers/trijoint.py.151: total_norm: 11.535903 [I 2022-10-11 18:55:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 880/2304 [I 2022-10-11 18:55:09] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:20:24 [I 2022-10-11 18:55:09] ...trap/engines/engine.py.229: process: 0.85723 | load: 0.00313 [I 2022-10-11 18:55:09] ...trap/engines/engine.py.230: loss: 0.17254 [I 2022-10-11 18:55:09] ...optimizers/trijoint.py.151: total_norm: 20.948498 [I 2022-10-11 18:55:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 890/2304 [I 2022-10-11 18:55:17] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:20:10 [I 2022-10-11 18:55:17] ...trap/engines/engine.py.229: process: 0.85884 | load: 0.00313 [I 2022-10-11 18:55:17] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-11 18:55:17] ...optimizers/trijoint.py.151: total_norm: 10.284453 [I 2022-10-11 18:55:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 900/2304 [I 2022-10-11 18:55:26] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:20:02 [I 2022-10-11 18:55:26] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00311 [I 2022-10-11 18:55:26] ...trap/engines/engine.py.230: loss: 0.10804 [I 2022-10-11 18:55:26] ...optimizers/trijoint.py.151: total_norm: 11.505042 [I 2022-10-11 18:55:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 910/2304 [I 2022-10-11 18:55:34] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:19:57 [I 2022-10-11 18:55:34] ...trap/engines/engine.py.229: process: 0.86135 | load: 0.00319 [I 2022-10-11 18:55:34] ...trap/engines/engine.py.230: loss: 0.10303 [I 2022-10-11 18:55:34] ...optimizers/trijoint.py.151: total_norm: 11.036874 [I 2022-10-11 18:55:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 920/2304 [I 2022-10-11 18:55:43] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:19:46 [I 2022-10-11 18:55:43] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00313 [I 2022-10-11 18:55:43] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-11 18:55:43] ...optimizers/trijoint.py.151: total_norm: 9.533660 [I 2022-10-11 18:55:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 930/2304 [I 2022-10-11 18:55:51] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:35 [I 2022-10-11 18:55:51] ...trap/engines/engine.py.229: process: 0.84381 | load: 0.00312 [I 2022-10-11 18:55:51] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-11 18:55:51] ...optimizers/trijoint.py.151: total_norm: 17.535568 [I 2022-10-11 18:56:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 940/2304 [I 2022-10-11 18:56:00] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:19:31 [I 2022-10-11 18:56:00] ...trap/engines/engine.py.229: process: 0.84866 | load: 0.00310 [I 2022-10-11 18:56:00] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-11 18:56:00] ...optimizers/trijoint.py.151: total_norm: 17.549294 [I 2022-10-11 18:56:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 950/2304 [I 2022-10-11 18:56:09] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:19:21 [I 2022-10-11 18:56:09] ...trap/engines/engine.py.229: process: 0.86129 | load: 0.00314 [I 2022-10-11 18:56:09] ...trap/engines/engine.py.230: loss: 0.11094 [I 2022-10-11 18:56:09] ...optimizers/trijoint.py.151: total_norm: 13.084922 [I 2022-10-11 18:56:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 960/2304 [I 2022-10-11 18:56:17] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:19:14 [I 2022-10-11 18:56:17] ...trap/engines/engine.py.229: process: 0.85527 | load: 0.00314 [I 2022-10-11 18:56:17] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-11 18:56:17] ...optimizers/trijoint.py.151: total_norm: 12.385071 [I 2022-10-11 18:56:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 970/2304 [I 2022-10-11 18:56:26] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:19:00 [I 2022-10-11 18:56:26] ...trap/engines/engine.py.229: process: 0.85189 | load: 0.00313 [I 2022-10-11 18:56:26] ...trap/engines/engine.py.230: loss: 0.10014 [I 2022-10-11 18:56:26] ...optimizers/trijoint.py.151: total_norm: 15.225704 [I 2022-10-11 18:56:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 980/2304 [I 2022-10-11 18:56:34] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:18:56 [I 2022-10-11 18:56:34] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00311 [I 2022-10-11 18:56:34] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-11 18:56:34] ...optimizers/trijoint.py.151: total_norm: 17.439520 [I 2022-10-11 18:56:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 990/2304 [I 2022-10-11 18:56:43] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:18:42 [I 2022-10-11 18:56:43] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00315 [I 2022-10-11 18:56:43] ...trap/engines/engine.py.230: loss: 0.07431 [I 2022-10-11 18:56:43] ...optimizers/trijoint.py.151: total_norm: 34.907085 [I 2022-10-11 18:56:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1000/2304 [I 2022-10-11 18:56:52] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:18:30 [I 2022-10-11 18:56:52] ...trap/engines/engine.py.229: process: 0.84066 | load: 0.00306 [I 2022-10-11 18:56:52] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-11 18:56:52] ...optimizers/trijoint.py.151: total_norm: 12.143115 [I 2022-10-11 18:57:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1010/2304 [I 2022-10-11 18:57:00] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:18:13 [I 2022-10-11 18:57:00] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00296 [I 2022-10-11 18:57:00] ...trap/engines/engine.py.230: loss: 0.14866 [I 2022-10-11 18:57:00] ...optimizers/trijoint.py.151: total_norm: 14.777707 [I 2022-10-11 18:57:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1020/2304 [I 2022-10-11 18:57:09] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:18:14 [I 2022-10-11 18:57:09] ...trap/engines/engine.py.229: process: 0.85234 | load: 0.00297 [I 2022-10-11 18:57:09] ...trap/engines/engine.py.230: loss: 0.08520 [I 2022-10-11 18:57:09] ...optimizers/trijoint.py.151: total_norm: 10.324224 [I 2022-10-11 18:57:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1030/2304 [I 2022-10-11 18:57:17] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:17:51 [I 2022-10-11 18:57:17] ...trap/engines/engine.py.229: process: 0.84255 | load: 0.00294 [I 2022-10-11 18:57:17] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-11 18:57:17] ...optimizers/trijoint.py.151: total_norm: 11.266549 [I 2022-10-11 18:57:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1040/2304 [I 2022-10-11 18:57:25] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:17:47 [I 2022-10-11 18:57:25] ...trap/engines/engine.py.229: process: 0.84122 | load: 0.00300 [I 2022-10-11 18:57:25] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-11 18:57:25] ...optimizers/trijoint.py.151: total_norm: 20.673040 [I 2022-10-11 18:57:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 1050/2304 [I 2022-10-11 18:57:34] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:17:41 [I 2022-10-11 18:57:34] ...trap/engines/engine.py.229: process: 0.85648 | load: 0.00296 [I 2022-10-11 18:57:34] ...trap/engines/engine.py.230: loss: 0.08981 [I 2022-10-11 18:57:34] ...optimizers/trijoint.py.151: total_norm: 19.938509 [I 2022-10-11 18:57:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 1060/2304 [I 2022-10-11 18:57:42] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:17:33 [I 2022-10-11 18:57:42] ...trap/engines/engine.py.229: process: 0.84043 | load: 0.00295 [I 2022-10-11 18:57:42] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-11 18:57:42] ...optimizers/trijoint.py.151: total_norm: 21.990631 [I 2022-10-11 18:57:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1070/2304 [I 2022-10-11 18:57:51] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:17:27 [I 2022-10-11 18:57:51] ...trap/engines/engine.py.229: process: 0.86334 | load: 0.00296 [I 2022-10-11 18:57:51] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-11 18:57:51] ...optimizers/trijoint.py.151: total_norm: 15.403270 [I 2022-10-11 18:57:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 1080/2304 [I 2022-10-11 18:57:59] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:17:16 [I 2022-10-11 18:57:59] ...trap/engines/engine.py.229: process: 0.84297 | load: 0.00296 [I 2022-10-11 18:57:59] ...trap/engines/engine.py.230: loss: 0.09242 [I 2022-10-11 18:57:59] ...optimizers/trijoint.py.151: total_norm: 10.805563 [I 2022-10-11 18:58:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 1090/2304 [I 2022-10-11 18:58:08] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:17:09 [I 2022-10-11 18:58:08] ...trap/engines/engine.py.229: process: 0.85368 | load: 0.00302 [I 2022-10-11 18:58:08] ...trap/engines/engine.py.230: loss: 0.16534 [I 2022-10-11 18:58:08] ...optimizers/trijoint.py.151: total_norm: 18.707632 [I 2022-10-11 18:58:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 1100/2304 [I 2022-10-11 18:58:16] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:16:55 [I 2022-10-11 18:58:16] ...trap/engines/engine.py.229: process: 0.84252 | load: 0.00300 [I 2022-10-11 18:58:16] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-11 18:58:16] ...optimizers/trijoint.py.151: total_norm: 15.742019 [I 2022-10-11 18:58:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1110/2304 [I 2022-10-11 18:58:25] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:16:53 [I 2022-10-11 18:58:25] ...trap/engines/engine.py.229: process: 0.85056 | load: 0.00298 [I 2022-10-11 18:58:25] ...trap/engines/engine.py.230: loss: 0.12145 [I 2022-10-11 18:58:25] ...optimizers/trijoint.py.151: total_norm: 9.121859 [I 2022-10-11 18:58:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1120/2304 [I 2022-10-11 18:58:33] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:16:35 [I 2022-10-11 18:58:33] ...trap/engines/engine.py.229: process: 0.84355 | load: 0.00298 [I 2022-10-11 18:58:33] ...trap/engines/engine.py.230: loss: 0.08211 [I 2022-10-11 18:58:33] ...optimizers/trijoint.py.151: total_norm: 14.037789 [I 2022-10-11 18:58:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 1130/2304 [I 2022-10-11 18:58:42] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:16:30 [I 2022-10-11 18:58:42] ...trap/engines/engine.py.229: process: 0.84410 | load: 0.00299 [I 2022-10-11 18:58:42] ...trap/engines/engine.py.230: loss: 0.08164 [I 2022-10-11 18:58:42] ...optimizers/trijoint.py.151: total_norm: 33.742580 [I 2022-10-11 18:58:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 1140/2304 [I 2022-10-11 18:58:50] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:23 [I 2022-10-11 18:58:50] ...trap/engines/engine.py.229: process: 0.85202 | load: 0.00302 [I 2022-10-11 18:58:50] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-11 18:58:50] ...optimizers/trijoint.py.151: total_norm: 11.378968 [I 2022-10-11 18:58:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 1150/2304 [I 2022-10-11 18:58:59] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:14 [I 2022-10-11 18:58:59] ...trap/engines/engine.py.229: process: 0.84552 | load: 0.00298 [I 2022-10-11 18:58:59] ...trap/engines/engine.py.230: loss: 0.09958 [I 2022-10-11 18:58:59] ...optimizers/trijoint.py.151: total_norm: 10.658416 [I 2022-10-11 18:59:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 1160/2304 [I 2022-10-11 18:59:07] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:20 [I 2022-10-11 18:59:07] ...trap/engines/engine.py.229: process: 0.86757 | load: 0.00317 [I 2022-10-11 18:59:07] ...trap/engines/engine.py.230: loss: 0.13201 [I 2022-10-11 18:59:07] ...optimizers/trijoint.py.151: total_norm: 16.820755 [I 2022-10-11 18:59:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 1170/2304 [I 2022-10-11 18:59:16] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:10 [I 2022-10-11 18:59:16] ...trap/engines/engine.py.229: process: 0.85222 | load: 0.00314 [I 2022-10-11 18:59:16] ...trap/engines/engine.py.230: loss: 0.09693 [I 2022-10-11 18:59:16] ...optimizers/trijoint.py.151: total_norm: 13.774802 [I 2022-10-11 18:59:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 1180/2304 [I 2022-10-11 18:59:24] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:16:02 [I 2022-10-11 18:59:24] ...trap/engines/engine.py.229: process: 0.84375 | load: 0.00313 [I 2022-10-11 18:59:24] ...trap/engines/engine.py.230: loss: 0.24771 [I 2022-10-11 18:59:24] ...optimizers/trijoint.py.151: total_norm: 11.524655 [I 2022-10-11 18:59:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1190/2304 [I 2022-10-11 18:59:33] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:15:57 [I 2022-10-11 18:59:33] ...trap/engines/engine.py.229: process: 0.84201 | load: 0.00318 [I 2022-10-11 18:59:33] ...trap/engines/engine.py.230: loss: 0.10142 [I 2022-10-11 18:59:33] ...optimizers/trijoint.py.151: total_norm: 17.976219 [I 2022-10-11 18:59:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 1200/2304 [I 2022-10-11 18:59:42] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:54 [I 2022-10-11 18:59:42] ...trap/engines/engine.py.229: process: 0.86510 | load: 0.00315 [I 2022-10-11 18:59:42] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-11 18:59:42] ...optimizers/trijoint.py.151: total_norm: 20.605448 [I 2022-10-11 18:59:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 1210/2304 [I 2022-10-11 18:59:50] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:15:37 [I 2022-10-11 18:59:50] ...trap/engines/engine.py.229: process: 0.85979 | load: 0.00313 [I 2022-10-11 18:59:50] ...trap/engines/engine.py.230: loss: 0.10447 [I 2022-10-11 18:59:50] ...optimizers/trijoint.py.151: total_norm: 14.644144 [I 2022-10-11 18:59:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 1220/2304 [I 2022-10-11 18:59:59] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:22 [I 2022-10-11 18:59:59] ...trap/engines/engine.py.229: process: 0.84905 | load: 0.00316 [I 2022-10-11 18:59:59] ...trap/engines/engine.py.230: loss: 0.08043 [I 2022-10-11 18:59:59] ...optimizers/trijoint.py.151: total_norm: 12.433267 [I 2022-10-11 19:00:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 1230/2304 [I 2022-10-11 19:00:07] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:15:18 [I 2022-10-11 19:00:07] ...trap/engines/engine.py.229: process: 0.85152 | load: 0.00316 [I 2022-10-11 19:00:07] ...trap/engines/engine.py.230: loss: 0.11816 [I 2022-10-11 19:00:07] ...optimizers/trijoint.py.151: total_norm: 21.804190 [I 2022-10-11 19:00:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 1240/2304 [I 2022-10-11 19:00:16] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:14 [I 2022-10-11 19:00:16] ...trap/engines/engine.py.229: process: 0.86257 | load: 0.00314 [I 2022-10-11 19:00:16] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-11 19:00:16] ...optimizers/trijoint.py.151: total_norm: 14.035042 [I 2022-10-11 19:00:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1250/2304 [I 2022-10-11 19:00:25] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:15:10 [I 2022-10-11 19:00:25] ...trap/engines/engine.py.229: process: 0.84596 | load: 0.00321 [I 2022-10-11 19:00:25] ...trap/engines/engine.py.230: loss: 0.08589 [I 2022-10-11 19:00:25] ...optimizers/trijoint.py.151: total_norm: 14.777447 [I 2022-10-11 19:00:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1260/2304 [I 2022-10-11 19:00:33] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:15:05 [I 2022-10-11 19:00:33] ...trap/engines/engine.py.229: process: 0.86697 | load: 0.00312 [I 2022-10-11 19:00:33] ...trap/engines/engine.py.230: loss: 0.10210 [I 2022-10-11 19:00:33] ...optimizers/trijoint.py.151: total_norm: 25.302612 [I 2022-10-11 19:00:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 1270/2304 [I 2022-10-11 19:00:42] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:14:45 [I 2022-10-11 19:00:42] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00313 [I 2022-10-11 19:00:42] ...trap/engines/engine.py.230: loss: 0.08658 [I 2022-10-11 19:00:42] ...optimizers/trijoint.py.151: total_norm: 16.157223 [I 2022-10-11 19:00:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1280/2304 [I 2022-10-11 19:00:51] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:14:40 [I 2022-10-11 19:00:51] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00316 [I 2022-10-11 19:00:51] ...trap/engines/engine.py.230: loss: 0.09479 [I 2022-10-11 19:00:51] ...optimizers/trijoint.py.151: total_norm: 11.310054 [I 2022-10-11 19:00:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 1290/2304 [I 2022-10-11 19:00:59] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:14:34 [I 2022-10-11 19:00:59] ...trap/engines/engine.py.229: process: 0.87542 | load: 0.00314 [I 2022-10-11 19:00:59] ...trap/engines/engine.py.230: loss: 0.07333 [I 2022-10-11 19:00:59] ...optimizers/trijoint.py.151: total_norm: 13.140162 [I 2022-10-11 19:01:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 1300/2304 [I 2022-10-11 19:01:08] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:14:20 [I 2022-10-11 19:01:08] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00316 [I 2022-10-11 19:01:08] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-11 19:01:08] ...optimizers/trijoint.py.151: total_norm: 17.998491 [I 2022-10-11 19:01:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 1310/2304 [I 2022-10-11 19:01:16] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:14:18 [I 2022-10-11 19:01:16] ...trap/engines/engine.py.229: process: 0.86782 | load: 0.00313 [I 2022-10-11 19:01:16] ...trap/engines/engine.py.230: loss: 0.12446 [I 2022-10-11 19:01:16] ...optimizers/trijoint.py.151: total_norm: 11.346864 [I 2022-10-11 19:01:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1320/2304 [I 2022-10-11 19:01:25] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:14:07 [I 2022-10-11 19:01:25] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00314 [I 2022-10-11 19:01:25] ...trap/engines/engine.py.230: loss: 0.08750 [I 2022-10-11 19:01:25] ...optimizers/trijoint.py.151: total_norm: 10.317024 [I 2022-10-11 19:01:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 1330/2304 [I 2022-10-11 19:01:34] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:14:02 [I 2022-10-11 19:01:34] ...trap/engines/engine.py.229: process: 0.87063 | load: 0.00313 [I 2022-10-11 19:01:34] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-11 19:01:34] ...optimizers/trijoint.py.151: total_norm: 23.596321 [I 2022-10-11 19:01:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1340/2304 [I 2022-10-11 19:01:43] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:14:10 [I 2022-10-11 19:01:43] ...trap/engines/engine.py.229: process: 0.88228 | load: 0.00401 [I 2022-10-11 19:01:43] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-11 19:01:43] ...optimizers/trijoint.py.151: total_norm: 12.846418 [I 2022-10-11 19:01:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1350/2304 [I 2022-10-11 19:01:51] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:14:04 [I 2022-10-11 19:01:51] ...trap/engines/engine.py.229: process: 0.88361 | load: 0.00396 [I 2022-10-11 19:01:51] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-11 19:01:51] ...optimizers/trijoint.py.151: total_norm: 10.640113 [I 2022-10-11 19:02:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1360/2304 [I 2022-10-11 19:02:00] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:13:58 [I 2022-10-11 19:02:00] ...trap/engines/engine.py.229: process: 0.88420 | load: 0.00398 [I 2022-10-11 19:02:00] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-11 19:02:00] ...optimizers/trijoint.py.151: total_norm: 16.071842 [I 2022-10-11 19:02:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1370/2304 [I 2022-10-11 19:02:09] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:13:48 [I 2022-10-11 19:02:09] ...trap/engines/engine.py.229: process: 0.88845 | load: 0.00396 [I 2022-10-11 19:02:09] ...trap/engines/engine.py.230: loss: 0.08864 [I 2022-10-11 19:02:09] ...optimizers/trijoint.py.151: total_norm: 17.602795 [I 2022-10-11 19:02:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1380/2304 [I 2022-10-11 19:02:18] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:13:38 [I 2022-10-11 19:02:18] ...trap/engines/engine.py.229: process: 0.86728 | load: 0.00393 [I 2022-10-11 19:02:18] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-11 19:02:18] ...optimizers/trijoint.py.151: total_norm: 23.814388 [I 2022-10-11 19:02:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 1390/2304 [I 2022-10-11 19:02:27] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:13:16 [I 2022-10-11 19:02:27] ...trap/engines/engine.py.229: process: 0.90771 | load: 0.00312 [I 2022-10-11 19:02:27] ...trap/engines/engine.py.230: loss: 0.14869 [I 2022-10-11 19:02:27] ...optimizers/trijoint.py.151: total_norm: 16.690504 [I 2022-10-11 19:02:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1400/2304 [I 2022-10-11 19:02:35] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:13:06 [I 2022-10-11 19:02:35] ...trap/engines/engine.py.229: process: 0.87471 | load: 0.00399 [I 2022-10-11 19:02:35] ...trap/engines/engine.py.230: loss: 0.16371 [I 2022-10-11 19:02:35] ...optimizers/trijoint.py.151: total_norm: 18.097631 [I 2022-10-11 19:02:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 1410/2304 [I 2022-10-11 19:02:44] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:12:47 [I 2022-10-11 19:02:44] ...trap/engines/engine.py.229: process: 0.86160 | load: 0.00317 [I 2022-10-11 19:02:44] ...trap/engines/engine.py.230: loss: 0.16513 [I 2022-10-11 19:02:44] ...optimizers/trijoint.py.151: total_norm: 13.887097 [I 2022-10-11 19:02:53] ...trap/engines/engine.py.224: train: epoch 115 | batch 1420/2304 [I 2022-10-11 19:02:53] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:12:32 [I 2022-10-11 19:02:53] ...trap/engines/engine.py.229: process: 0.84741 | load: 0.00313 [I 2022-10-11 19:02:53] ...trap/engines/engine.py.230: loss: 0.12155 [I 2022-10-11 19:02:53] ...optimizers/trijoint.py.151: total_norm: 10.085262 [I 2022-10-11 19:03:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1430/2304 [I 2022-10-11 19:03:01] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:12:25 [I 2022-10-11 19:03:01] ...trap/engines/engine.py.229: process: 0.84728 | load: 0.00313 [I 2022-10-11 19:03:01] ...trap/engines/engine.py.230: loss: 0.13003 [I 2022-10-11 19:03:01] ...optimizers/trijoint.py.151: total_norm: 14.042956 [I 2022-10-11 19:03:10] ...trap/engines/engine.py.224: train: epoch 115 | batch 1440/2304 [I 2022-10-11 19:03:10] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:12:12 [I 2022-10-11 19:03:10] ...trap/engines/engine.py.229: process: 0.83991 | load: 0.00296 [I 2022-10-11 19:03:10] ...trap/engines/engine.py.230: loss: 0.11608 [I 2022-10-11 19:03:10] ...optimizers/trijoint.py.151: total_norm: 21.148170 [I 2022-10-11 19:03:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1450/2304 [I 2022-10-11 19:03:18] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:12:07 [I 2022-10-11 19:03:18] ...trap/engines/engine.py.229: process: 0.84918 | load: 0.00299 [I 2022-10-11 19:03:18] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-11 19:03:18] ...optimizers/trijoint.py.151: total_norm: 21.119545 [I 2022-10-11 19:03:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 1460/2304 [I 2022-10-11 19:03:27] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:11:51 [I 2022-10-11 19:03:27] ...trap/engines/engine.py.229: process: 0.83559 | load: 0.00295 [I 2022-10-11 19:03:27] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-11 19:03:27] ...optimizers/trijoint.py.151: total_norm: 10.492135 [I 2022-10-11 19:03:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1470/2304 [I 2022-10-11 19:03:35] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:11:45 [I 2022-10-11 19:03:35] ...trap/engines/engine.py.229: process: 0.85096 | load: 0.00295 [I 2022-10-11 19:03:35] ...trap/engines/engine.py.230: loss: 0.07881 [I 2022-10-11 19:03:35] ...optimizers/trijoint.py.151: total_norm: 16.411419 [I 2022-10-11 19:03:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 1480/2304 [I 2022-10-11 19:03:44] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:11:38 [I 2022-10-11 19:03:44] ...trap/engines/engine.py.229: process: 0.83709 | load: 0.00296 [I 2022-10-11 19:03:44] ...trap/engines/engine.py.230: loss: 0.08380 [I 2022-10-11 19:03:44] ...optimizers/trijoint.py.151: total_norm: 15.405519 [I 2022-10-11 19:03:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1490/2304 [I 2022-10-11 19:03:52] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:11:31 [I 2022-10-11 19:03:52] ...trap/engines/engine.py.229: process: 0.84819 | load: 0.00295 [I 2022-10-11 19:03:52] ...trap/engines/engine.py.230: loss: 0.08421 [I 2022-10-11 19:03:52] ...optimizers/trijoint.py.151: total_norm: 12.266989 [I 2022-10-11 19:04:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1500/2304 [I 2022-10-11 19:04:00] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:11:15 [I 2022-10-11 19:04:00] ...trap/engines/engine.py.229: process: 0.83613 | load: 0.00296 [I 2022-10-11 19:04:00] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-11 19:04:00] ...optimizers/trijoint.py.151: total_norm: 22.150749 [I 2022-10-11 19:04:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1510/2304 [I 2022-10-11 19:04:09] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:11:13 [I 2022-10-11 19:04:09] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00298 [I 2022-10-11 19:04:09] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-11 19:04:09] ...optimizers/trijoint.py.151: total_norm: 9.582795 [I 2022-10-11 19:04:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1520/2304 [I 2022-10-11 19:04:17] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:11:06 [I 2022-10-11 19:04:17] ...trap/engines/engine.py.229: process: 0.85398 | load: 0.00302 [I 2022-10-11 19:04:17] ...trap/engines/engine.py.230: loss: 0.09556 [I 2022-10-11 19:04:17] ...optimizers/trijoint.py.151: total_norm: 9.556710 [I 2022-10-11 19:04:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1530/2304 [I 2022-10-11 19:04:26] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:11:00 [I 2022-10-11 19:04:26] ...trap/engines/engine.py.229: process: 0.85466 | load: 0.00299 [I 2022-10-11 19:04:26] ...trap/engines/engine.py.230: loss: 0.17316 [I 2022-10-11 19:04:26] ...optimizers/trijoint.py.151: total_norm: 20.830282 [I 2022-10-11 19:04:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1540/2304 [I 2022-10-11 19:04:35] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:10:48 [I 2022-10-11 19:04:35] ...trap/engines/engine.py.229: process: 0.84460 | load: 0.00297 [I 2022-10-11 19:04:35] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-11 19:04:35] ...optimizers/trijoint.py.151: total_norm: 13.852625 [I 2022-10-11 19:04:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1550/2304 [I 2022-10-11 19:04:43] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:10:35 [I 2022-10-11 19:04:43] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00295 [I 2022-10-11 19:04:43] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-11 19:04:43] ...optimizers/trijoint.py.151: total_norm: 14.874518 [I 2022-10-11 19:04:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1560/2304 [I 2022-10-11 19:04:51] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:10:30 [I 2022-10-11 19:04:51] ...trap/engines/engine.py.229: process: 0.85054 | load: 0.00298 [I 2022-10-11 19:04:51] ...trap/engines/engine.py.230: loss: 0.10660 [I 2022-10-11 19:04:51] ...optimizers/trijoint.py.151: total_norm: 16.314585 [I 2022-10-11 19:05:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1570/2304 [I 2022-10-11 19:05:18] ...trap/engines/engine.py.228: elapsed: 0:22:44 | left: 0:28:21 [I 2022-10-11 19:05:18] ...trap/engines/engine.py.229: process: 0.83471 | load: 0.00299 [I 2022-10-11 19:05:18] ...trap/engines/engine.py.230: loss: 0.11463 [I 2022-10-11 19:05:18] ...optimizers/trijoint.py.151: total_norm: 13.686855 [I 2022-10-11 19:05:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1580/2304 [I 2022-10-11 19:05:26] ...trap/engines/engine.py.228: elapsed: 0:22:53 | left: 0:12:02 [I 2022-10-11 19:05:26] ...trap/engines/engine.py.229: process: 0.83213 | load: 0.00298 [I 2022-10-11 19:05:26] ...trap/engines/engine.py.230: loss: 0.07654 [I 2022-10-11 19:05:26] ...optimizers/trijoint.py.151: total_norm: 9.475589 [I 2022-10-11 19:05:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1590/2304 [I 2022-10-11 19:05:35] ...trap/engines/engine.py.228: elapsed: 0:23:01 | left: 0:10:16 [I 2022-10-11 19:05:35] ...trap/engines/engine.py.229: process: 0.84901 | load: 0.00300 [I 2022-10-11 19:05:35] ...trap/engines/engine.py.230: loss: 0.18997 [I 2022-10-11 19:05:35] ...optimizers/trijoint.py.151: total_norm: 26.079601 [I 2022-10-11 19:05:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1600/2304 [I 2022-10-11 19:05:43] ...trap/engines/engine.py.228: elapsed: 0:23:10 | left: 0:10:01 [I 2022-10-11 19:05:43] ...trap/engines/engine.py.229: process: 0.85688 | load: 0.00312 [I 2022-10-11 19:05:43] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-11 19:05:43] ...optimizers/trijoint.py.151: total_norm: 16.122271 [I 2022-10-11 19:05:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1610/2304 [I 2022-10-11 19:05:52] ...trap/engines/engine.py.228: elapsed: 0:23:18 | left: 0:09:50 [I 2022-10-11 19:05:52] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00313 [I 2022-10-11 19:05:52] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-11 19:05:52] ...optimizers/trijoint.py.151: total_norm: 12.330282 [I 2022-10-11 19:06:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1620/2304 [I 2022-10-11 19:06:00] ...trap/engines/engine.py.228: elapsed: 0:23:27 | left: 0:09:43 [I 2022-10-11 19:06:00] ...trap/engines/engine.py.229: process: 0.85331 | load: 0.00313 [I 2022-10-11 19:06:00] ...trap/engines/engine.py.230: loss: 0.12693 [I 2022-10-11 19:06:00] ...optimizers/trijoint.py.151: total_norm: 24.159784 [I 2022-10-11 19:06:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1630/2304 [I 2022-10-11 19:06:09] ...trap/engines/engine.py.228: elapsed: 0:23:35 | left: 0:09:33 [I 2022-10-11 19:06:09] ...trap/engines/engine.py.229: process: 0.83417 | load: 0.00314 [I 2022-10-11 19:06:09] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-11 19:06:09] ...optimizers/trijoint.py.151: total_norm: 16.688511 [I 2022-10-11 19:06:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1640/2304 [I 2022-10-11 19:06:18] ...trap/engines/engine.py.228: elapsed: 0:23:44 | left: 0:09:28 [I 2022-10-11 19:06:18] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00315 [I 2022-10-11 19:06:18] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-11 19:06:18] ...optimizers/trijoint.py.151: total_norm: 12.079154 [I 2022-10-11 19:06:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1650/2304 [I 2022-10-11 19:06:26] ...trap/engines/engine.py.228: elapsed: 0:23:52 | left: 0:09:13 [I 2022-10-11 19:06:26] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00333 [I 2022-10-11 19:06:26] ...trap/engines/engine.py.230: loss: 0.08285 [I 2022-10-11 19:06:26] ...optimizers/trijoint.py.151: total_norm: 19.208176 [I 2022-10-11 19:06:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1660/2304 [I 2022-10-11 19:06:35] ...trap/engines/engine.py.228: elapsed: 0:24:01 | left: 0:09:07 [I 2022-10-11 19:06:35] ...trap/engines/engine.py.229: process: 0.84377 | load: 0.00313 [I 2022-10-11 19:06:35] ...trap/engines/engine.py.230: loss: 0.14049 [I 2022-10-11 19:06:35] ...optimizers/trijoint.py.151: total_norm: 10.201998 [I 2022-10-11 19:06:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1670/2304 [I 2022-10-11 19:06:43] ...trap/engines/engine.py.228: elapsed: 0:24:09 | left: 0:09:00 [I 2022-10-11 19:06:43] ...trap/engines/engine.py.229: process: 0.85595 | load: 0.00315 [I 2022-10-11 19:06:43] ...trap/engines/engine.py.230: loss: 0.10305 [I 2022-10-11 19:06:43] ...optimizers/trijoint.py.151: total_norm: 10.108200 [I 2022-10-11 19:06:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1680/2304 [I 2022-10-11 19:06:52] ...trap/engines/engine.py.228: elapsed: 0:24:18 | left: 0:08:51 [I 2022-10-11 19:06:52] ...trap/engines/engine.py.229: process: 0.85107 | load: 0.00316 [I 2022-10-11 19:06:52] ...trap/engines/engine.py.230: loss: 0.09366 [I 2022-10-11 19:06:52] ...optimizers/trijoint.py.151: total_norm: 12.878017 [I 2022-10-11 19:07:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1690/2304 [I 2022-10-11 19:07:00] ...trap/engines/engine.py.228: elapsed: 0:24:26 | left: 0:08:43 [I 2022-10-11 19:07:00] ...trap/engines/engine.py.229: process: 0.85968 | load: 0.00313 [I 2022-10-11 19:07:00] ...trap/engines/engine.py.230: loss: 0.08683 [I 2022-10-11 19:07:00] ...optimizers/trijoint.py.151: total_norm: 14.995804 [I 2022-10-11 19:07:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1700/2304 [I 2022-10-11 19:07:09] ...trap/engines/engine.py.228: elapsed: 0:24:35 | left: 0:08:35 [I 2022-10-11 19:07:09] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00316 [I 2022-10-11 19:07:09] ...trap/engines/engine.py.230: loss: 0.11670 [I 2022-10-11 19:07:09] ...optimizers/trijoint.py.151: total_norm: 10.659638 [I 2022-10-11 19:07:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1710/2304 [I 2022-10-11 19:07:17] ...trap/engines/engine.py.228: elapsed: 0:24:44 | left: 0:08:29 [I 2022-10-11 19:07:17] ...trap/engines/engine.py.229: process: 0.85433 | load: 0.00317 [I 2022-10-11 19:07:17] ...trap/engines/engine.py.230: loss: 0.08761 [I 2022-10-11 19:07:17] ...optimizers/trijoint.py.151: total_norm: 11.104132 [I 2022-10-11 19:07:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1720/2304 [I 2022-10-11 19:07:26] ...trap/engines/engine.py.228: elapsed: 0:24:52 | left: 0:08:18 [I 2022-10-11 19:07:26] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00318 [I 2022-10-11 19:07:26] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-11 19:07:26] ...optimizers/trijoint.py.151: total_norm: 9.537021 [I 2022-10-11 19:07:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 1730/2304 [I 2022-10-11 19:07:34] ...trap/engines/engine.py.228: elapsed: 0:25:01 | left: 0:08:16 [I 2022-10-11 19:07:34] ...trap/engines/engine.py.229: process: 0.89048 | load: 0.00394 [I 2022-10-11 19:07:34] ...trap/engines/engine.py.230: loss: 0.08890 [I 2022-10-11 19:07:34] ...optimizers/trijoint.py.151: total_norm: 16.048059 [I 2022-10-11 19:07:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1740/2304 [I 2022-10-11 19:07:43] ...trap/engines/engine.py.228: elapsed: 0:25:09 | left: 0:08:09 [I 2022-10-11 19:07:43] ...trap/engines/engine.py.229: process: 0.85778 | load: 0.00313 [I 2022-10-11 19:07:43] ...trap/engines/engine.py.230: loss: 0.14904 [I 2022-10-11 19:07:43] ...optimizers/trijoint.py.151: total_norm: 18.765459 [I 2022-10-11 19:07:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1750/2304 [I 2022-10-11 19:07:52] ...trap/engines/engine.py.228: elapsed: 0:25:18 | left: 0:07:58 [I 2022-10-11 19:07:52] ...trap/engines/engine.py.229: process: 0.86908 | load: 0.00312 [I 2022-10-11 19:07:52] ...trap/engines/engine.py.230: loss: 0.14266 [I 2022-10-11 19:07:52] ...optimizers/trijoint.py.151: total_norm: 12.117390 [I 2022-10-11 19:08:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1760/2304 [I 2022-10-11 19:08:00] ...trap/engines/engine.py.228: elapsed: 0:25:27 | left: 0:07:44 [I 2022-10-11 19:08:00] ...trap/engines/engine.py.229: process: 0.84123 | load: 0.00318 [I 2022-10-11 19:08:00] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-11 19:08:00] ...optimizers/trijoint.py.151: total_norm: 11.565940 [I 2022-10-11 19:08:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1770/2304 [I 2022-10-11 19:08:09] ...trap/engines/engine.py.228: elapsed: 0:25:35 | left: 0:07:39 [I 2022-10-11 19:08:09] ...trap/engines/engine.py.229: process: 0.85877 | load: 0.00313 [I 2022-10-11 19:08:09] ...trap/engines/engine.py.230: loss: 0.15204 [I 2022-10-11 19:08:09] ...optimizers/trijoint.py.151: total_norm: 25.651218 [I 2022-10-11 19:08:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1780/2304 [I 2022-10-11 19:08:18] ...trap/engines/engine.py.228: elapsed: 0:25:44 | left: 0:07:28 [I 2022-10-11 19:08:18] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00312 [I 2022-10-11 19:08:18] ...trap/engines/engine.py.230: loss: 0.17766 [I 2022-10-11 19:08:18] ...optimizers/trijoint.py.151: total_norm: 14.706713 [I 2022-10-11 19:08:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1790/2304 [I 2022-10-11 19:08:26] ...trap/engines/engine.py.228: elapsed: 0:25:52 | left: 0:07:20 [I 2022-10-11 19:08:26] ...trap/engines/engine.py.229: process: 0.85552 | load: 0.00314 [I 2022-10-11 19:08:26] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-11 19:08:26] ...optimizers/trijoint.py.151: total_norm: 14.300713 [I 2022-10-11 19:08:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1800/2304 [I 2022-10-11 19:08:35] ...trap/engines/engine.py.228: elapsed: 0:26:01 | left: 0:07:14 [I 2022-10-11 19:08:35] ...trap/engines/engine.py.229: process: 0.87505 | load: 0.00317 [I 2022-10-11 19:08:35] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-11 19:08:35] ...optimizers/trijoint.py.151: total_norm: 17.811459 [I 2022-10-11 19:08:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1810/2304 [I 2022-10-11 19:08:43] ...trap/engines/engine.py.228: elapsed: 0:26:10 | left: 0:07:04 [I 2022-10-11 19:08:43] ...trap/engines/engine.py.229: process: 0.87577 | load: 0.00314 [I 2022-10-11 19:08:43] ...trap/engines/engine.py.230: loss: 0.09324 [I 2022-10-11 19:08:43] ...optimizers/trijoint.py.151: total_norm: 12.065177 [I 2022-10-11 19:08:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1820/2304 [I 2022-10-11 19:08:52] ...trap/engines/engine.py.228: elapsed: 0:26:18 | left: 0:06:53 [I 2022-10-11 19:08:52] ...trap/engines/engine.py.229: process: 0.85746 | load: 0.00313 [I 2022-10-11 19:08:52] ...trap/engines/engine.py.230: loss: 0.12799 [I 2022-10-11 19:08:52] ...optimizers/trijoint.py.151: total_norm: 11.470169 [I 2022-10-11 19:09:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1830/2304 [I 2022-10-11 19:09:01] ...trap/engines/engine.py.228: elapsed: 0:26:27 | left: 0:06:46 [I 2022-10-11 19:09:01] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00312 [I 2022-10-11 19:09:01] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-11 19:09:01] ...optimizers/trijoint.py.151: total_norm: 24.535538 [I 2022-10-11 19:09:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1840/2304 [I 2022-10-11 19:09:09] ...trap/engines/engine.py.228: elapsed: 0:26:36 | left: 0:06:38 [I 2022-10-11 19:09:09] ...trap/engines/engine.py.229: process: 0.84969 | load: 0.00315 [I 2022-10-11 19:09:09] ...trap/engines/engine.py.230: loss: 0.08773 [I 2022-10-11 19:09:09] ...optimizers/trijoint.py.151: total_norm: 33.061340 [I 2022-10-11 19:09:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1850/2304 [I 2022-10-11 19:09:18] ...trap/engines/engine.py.228: elapsed: 0:26:44 | left: 0:06:34 [I 2022-10-11 19:09:18] ...trap/engines/engine.py.229: process: 0.88985 | load: 0.00313 [I 2022-10-11 19:09:18] ...trap/engines/engine.py.230: loss: 0.08474 [I 2022-10-11 19:09:18] ...optimizers/trijoint.py.151: total_norm: 10.428697 [I 2022-10-11 19:09:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 1860/2304 [I 2022-10-11 19:09:27] ...trap/engines/engine.py.228: elapsed: 0:26:53 | left: 0:06:25 [I 2022-10-11 19:09:27] ...trap/engines/engine.py.229: process: 0.86151 | load: 0.00318 [I 2022-10-11 19:09:27] ...trap/engines/engine.py.230: loss: 0.13235 [I 2022-10-11 19:09:27] ...optimizers/trijoint.py.151: total_norm: 33.026493 [I 2022-10-11 19:09:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1870/2304 [I 2022-10-11 19:09:35] ...trap/engines/engine.py.228: elapsed: 0:27:02 | left: 0:06:14 [I 2022-10-11 19:09:35] ...trap/engines/engine.py.229: process: 0.87306 | load: 0.00310 [I 2022-10-11 19:09:35] ...trap/engines/engine.py.230: loss: 0.11414 [I 2022-10-11 19:09:35] ...optimizers/trijoint.py.151: total_norm: 15.581038 [I 2022-10-11 19:09:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 1880/2304 [I 2022-10-11 19:09:44] ...trap/engines/engine.py.228: elapsed: 0:27:10 | left: 0:05:58 [I 2022-10-11 19:09:44] ...trap/engines/engine.py.229: process: 0.83391 | load: 0.00295 [I 2022-10-11 19:09:44] ...trap/engines/engine.py.230: loss: 0.09890 [I 2022-10-11 19:09:44] ...optimizers/trijoint.py.151: total_norm: 12.475921 [I 2022-10-11 19:09:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1890/2304 [I 2022-10-11 19:09:52] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:05:48 [I 2022-10-11 19:09:52] ...trap/engines/engine.py.229: process: 0.84339 | load: 0.00299 [I 2022-10-11 19:09:52] ...trap/engines/engine.py.230: loss: 0.16570 [I 2022-10-11 19:09:52] ...optimizers/trijoint.py.151: total_norm: 23.585449 [I 2022-10-11 19:10:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1900/2304 [I 2022-10-11 19:10:01] ...trap/engines/engine.py.228: elapsed: 0:27:27 | left: 0:05:42 [I 2022-10-11 19:10:01] ...trap/engines/engine.py.229: process: 0.86429 | load: 0.00296 [I 2022-10-11 19:10:01] ...trap/engines/engine.py.230: loss: 0.16011 [I 2022-10-11 19:10:01] ...optimizers/trijoint.py.151: total_norm: 18.306805 [I 2022-10-11 19:10:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1910/2304 [I 2022-10-11 19:10:09] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:33 [I 2022-10-11 19:10:09] ...trap/engines/engine.py.229: process: 0.86337 | load: 0.00300 [I 2022-10-11 19:10:09] ...trap/engines/engine.py.230: loss: 0.14646 [I 2022-10-11 19:10:09] ...optimizers/trijoint.py.151: total_norm: 41.944134 [I 2022-10-11 19:10:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1920/2304 [I 2022-10-11 19:10:18] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:25 [I 2022-10-11 19:10:18] ...trap/engines/engine.py.229: process: 0.84380 | load: 0.00300 [I 2022-10-11 19:10:18] ...trap/engines/engine.py.230: loss: 0.09813 [I 2022-10-11 19:10:18] ...optimizers/trijoint.py.151: total_norm: 24.830849 [I 2022-10-11 19:10:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1930/2304 [I 2022-10-11 19:10:26] ...trap/engines/engine.py.228: elapsed: 0:27:52 | left: 0:05:15 [I 2022-10-11 19:10:26] ...trap/engines/engine.py.229: process: 0.84571 | load: 0.00298 [I 2022-10-11 19:10:26] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-11 19:10:26] ...optimizers/trijoint.py.151: total_norm: 17.682615 [I 2022-10-11 19:10:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1940/2304 [I 2022-10-11 19:10:35] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:10 [I 2022-10-11 19:10:35] ...trap/engines/engine.py.229: process: 0.84273 | load: 0.00299 [I 2022-10-11 19:10:35] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-11 19:10:35] ...optimizers/trijoint.py.151: total_norm: 12.733232 [I 2022-10-11 19:10:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1950/2304 [I 2022-10-11 19:10:43] ...trap/engines/engine.py.228: elapsed: 0:28:09 | left: 0:05:01 [I 2022-10-11 19:10:43] ...trap/engines/engine.py.229: process: 0.85570 | load: 0.00297 [I 2022-10-11 19:10:43] ...trap/engines/engine.py.230: loss: 0.09728 [I 2022-10-11 19:10:43] ...optimizers/trijoint.py.151: total_norm: 10.932827 [I 2022-10-11 19:10:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 1960/2304 [I 2022-10-11 19:10:52] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:04:51 [I 2022-10-11 19:10:52] ...trap/engines/engine.py.229: process: 0.82685 | load: 0.00300 [I 2022-10-11 19:10:52] ...trap/engines/engine.py.230: loss: 0.07902 [I 2022-10-11 19:10:52] ...optimizers/trijoint.py.151: total_norm: 11.317287 [I 2022-10-11 19:11:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1970/2304 [I 2022-10-11 19:11:00] ...trap/engines/engine.py.228: elapsed: 0:28:26 | left: 0:04:43 [I 2022-10-11 19:11:00] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00299 [I 2022-10-11 19:11:00] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-11 19:11:00] ...optimizers/trijoint.py.151: total_norm: 26.469746 [I 2022-10-11 19:11:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1980/2304 [I 2022-10-11 19:11:09] ...trap/engines/engine.py.228: elapsed: 0:28:35 | left: 0:04:34 [I 2022-10-11 19:11:09] ...trap/engines/engine.py.229: process: 0.83325 | load: 0.00300 [I 2022-10-11 19:11:09] ...trap/engines/engine.py.230: loss: 0.19616 [I 2022-10-11 19:11:09] ...optimizers/trijoint.py.151: total_norm: 17.740103 [I 2022-10-11 19:11:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1990/2304 [I 2022-10-11 19:11:17] ...trap/engines/engine.py.228: elapsed: 0:28:43 | left: 0:04:24 [I 2022-10-11 19:11:17] ...trap/engines/engine.py.229: process: 0.83700 | load: 0.00300 [I 2022-10-11 19:11:17] ...trap/engines/engine.py.230: loss: 0.08222 [I 2022-10-11 19:11:17] ...optimizers/trijoint.py.151: total_norm: 11.325066 [I 2022-10-11 19:11:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 2000/2304 [I 2022-10-11 19:11:26] ...trap/engines/engine.py.228: elapsed: 0:28:52 | left: 0:04:17 [I 2022-10-11 19:11:26] ...trap/engines/engine.py.229: process: 0.85410 | load: 0.00301 [I 2022-10-11 19:11:26] ...trap/engines/engine.py.230: loss: 0.14464 [I 2022-10-11 19:11:26] ...optimizers/trijoint.py.151: total_norm: 9.954405 [I 2022-10-11 19:11:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 2010/2304 [I 2022-10-11 19:11:34] ...trap/engines/engine.py.228: elapsed: 0:29:00 | left: 0:04:08 [I 2022-10-11 19:11:34] ...trap/engines/engine.py.229: process: 0.83598 | load: 0.00300 [I 2022-10-11 19:11:34] ...trap/engines/engine.py.230: loss: 0.16129 [I 2022-10-11 19:11:34] ...optimizers/trijoint.py.151: total_norm: 16.647190 [I 2022-10-11 19:11:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 2020/2304 [I 2022-10-11 19:11:43] ...trap/engines/engine.py.228: elapsed: 0:29:09 | left: 0:04:00 [I 2022-10-11 19:11:43] ...trap/engines/engine.py.229: process: 0.85193 | load: 0.00312 [I 2022-10-11 19:11:43] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-11 19:11:43] ...optimizers/trijoint.py.151: total_norm: 9.354807 [I 2022-10-11 19:11:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 2030/2304 [I 2022-10-11 19:11:51] ...trap/engines/engine.py.228: elapsed: 0:29:17 | left: 0:03:52 [I 2022-10-11 19:11:51] ...trap/engines/engine.py.229: process: 0.85603 | load: 0.00312 [I 2022-10-11 19:11:51] ...trap/engines/engine.py.230: loss: 0.08083 [I 2022-10-11 19:11:51] ...optimizers/trijoint.py.151: total_norm: 15.468213 [I 2022-10-11 19:12:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 2040/2304 [I 2022-10-11 19:12:00] ...trap/engines/engine.py.228: elapsed: 0:29:26 | left: 0:03:44 [I 2022-10-11 19:12:00] ...trap/engines/engine.py.229: process: 0.85795 | load: 0.00314 [I 2022-10-11 19:12:00] ...trap/engines/engine.py.230: loss: 0.09378 [I 2022-10-11 19:12:00] ...optimizers/trijoint.py.151: total_norm: 10.586136 [I 2022-10-11 19:12:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 2050/2304 [I 2022-10-11 19:12:08] ...trap/engines/engine.py.228: elapsed: 0:29:34 | left: 0:03:36 [I 2022-10-11 19:12:08] ...trap/engines/engine.py.229: process: 0.86937 | load: 0.00321 [I 2022-10-11 19:12:08] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-11 19:12:08] ...optimizers/trijoint.py.151: total_norm: 12.510013 [I 2022-10-11 19:12:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 2060/2304 [I 2022-10-11 19:12:17] ...trap/engines/engine.py.228: elapsed: 0:29:43 | left: 0:03:28 [I 2022-10-11 19:12:17] ...trap/engines/engine.py.229: process: 0.84865 | load: 0.00315 [I 2022-10-11 19:12:17] ...trap/engines/engine.py.230: loss: 0.11137 [I 2022-10-11 19:12:17] ...optimizers/trijoint.py.151: total_norm: 18.076712 [I 2022-10-11 19:12:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 2070/2304 [I 2022-10-11 19:12:25] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:18 [I 2022-10-11 19:12:25] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00312 [I 2022-10-11 19:12:25] ...trap/engines/engine.py.230: loss: 0.09997 [I 2022-10-11 19:12:25] ...optimizers/trijoint.py.151: total_norm: 12.494302 [I 2022-10-11 19:12:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 2080/2304 [I 2022-10-11 19:12:34] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:11 [I 2022-10-11 19:12:34] ...trap/engines/engine.py.229: process: 0.84150 | load: 0.00318 [I 2022-10-11 19:12:34] ...trap/engines/engine.py.230: loss: 0.09195 [I 2022-10-11 19:12:34] ...optimizers/trijoint.py.151: total_norm: 16.794798 [I 2022-10-11 19:12:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 2090/2304 [I 2022-10-11 19:12:42] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:01 [I 2022-10-11 19:12:42] ...trap/engines/engine.py.229: process: 0.84396 | load: 0.00313 [I 2022-10-11 19:12:42] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-11 19:12:42] ...optimizers/trijoint.py.151: total_norm: 22.160385 [I 2022-10-11 19:12:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 2100/2304 [I 2022-10-11 19:12:51] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:54 [I 2022-10-11 19:12:51] ...trap/engines/engine.py.229: process: 0.85427 | load: 0.00315 [I 2022-10-11 19:12:51] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-11 19:12:51] ...optimizers/trijoint.py.151: total_norm: 9.042383 [I 2022-10-11 19:12:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 2110/2304 [I 2022-10-11 19:12:59] ...trap/engines/engine.py.228: elapsed: 0:30:26 | left: 0:02:45 [I 2022-10-11 19:12:59] ...trap/engines/engine.py.229: process: 0.85762 | load: 0.00310 [I 2022-10-11 19:12:59] ...trap/engines/engine.py.230: loss: 0.09369 [I 2022-10-11 19:12:59] ...optimizers/trijoint.py.151: total_norm: 11.529198 [I 2022-10-11 19:13:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 2120/2304 [I 2022-10-11 19:13:08] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:36 [I 2022-10-11 19:13:08] ...trap/engines/engine.py.229: process: 0.85008 | load: 0.00317 [I 2022-10-11 19:13:08] ...trap/engines/engine.py.230: loss: 0.10823 [I 2022-10-11 19:13:08] ...optimizers/trijoint.py.151: total_norm: 20.997051 [I 2022-10-11 19:13:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 2130/2304 [I 2022-10-11 19:13:16] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:28 [I 2022-10-11 19:13:16] ...trap/engines/engine.py.229: process: 0.85561 | load: 0.00312 [I 2022-10-11 19:13:16] ...trap/engines/engine.py.230: loss: 0.08328 [I 2022-10-11 19:13:16] ...optimizers/trijoint.py.151: total_norm: 17.268486 [I 2022-10-11 19:13:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 2140/2304 [I 2022-10-11 19:13:25] ...trap/engines/engine.py.228: elapsed: 0:30:51 | left: 0:02:21 [I 2022-10-11 19:13:25] ...trap/engines/engine.py.229: process: 0.83713 | load: 0.00314 [I 2022-10-11 19:13:25] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-11 19:13:25] ...optimizers/trijoint.py.151: total_norm: 16.853645 [I 2022-10-11 19:13:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 2150/2304 [I 2022-10-11 19:13:34] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:11 [I 2022-10-11 19:13:34] ...trap/engines/engine.py.229: process: 0.85470 | load: 0.00316 [I 2022-10-11 19:13:34] ...trap/engines/engine.py.230: loss: 0.11621 [I 2022-10-11 19:13:34] ...optimizers/trijoint.py.151: total_norm: 8.382952 [I 2022-10-11 19:13:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 2160/2304 [I 2022-10-11 19:13:42] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:02:03 [I 2022-10-11 19:13:42] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00312 [I 2022-10-11 19:13:42] ...trap/engines/engine.py.230: loss: 0.11668 [I 2022-10-11 19:13:42] ...optimizers/trijoint.py.151: total_norm: 22.480383 [I 2022-10-11 19:13:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 2170/2304 [I 2022-10-11 19:13:51] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:54 [I 2022-10-11 19:13:51] ...trap/engines/engine.py.229: process: 0.86697 | load: 0.00313 [I 2022-10-11 19:13:51] ...trap/engines/engine.py.230: loss: 0.09528 [I 2022-10-11 19:13:51] ...optimizers/trijoint.py.151: total_norm: 11.456835 [I 2022-10-11 19:13:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 2180/2304 [I 2022-10-11 19:13:59] ...trap/engines/engine.py.228: elapsed: 0:31:26 | left: 0:01:46 [I 2022-10-11 19:13:59] ...trap/engines/engine.py.229: process: 0.85529 | load: 0.00315 [I 2022-10-11 19:13:59] ...trap/engines/engine.py.230: loss: 0.09069 [I 2022-10-11 19:13:59] ...optimizers/trijoint.py.151: total_norm: 28.201788 [I 2022-10-11 19:14:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 2190/2304 [I 2022-10-11 19:14:08] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:37 [I 2022-10-11 19:14:08] ...trap/engines/engine.py.229: process: 0.86399 | load: 0.00320 [I 2022-10-11 19:14:08] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-11 19:14:08] ...optimizers/trijoint.py.151: total_norm: 21.328766 [I 2022-10-11 19:14:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 2200/2304 [I 2022-10-11 19:14:16] ...trap/engines/engine.py.228: elapsed: 0:31:43 | left: 0:01:28 [I 2022-10-11 19:14:16] ...trap/engines/engine.py.229: process: 0.86548 | load: 0.00315 [I 2022-10-11 19:14:16] ...trap/engines/engine.py.230: loss: 0.17132 [I 2022-10-11 19:14:16] ...optimizers/trijoint.py.151: total_norm: 17.448650 [I 2022-10-11 19:14:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 2210/2304 [I 2022-10-11 19:14:25] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:20 [I 2022-10-11 19:14:25] ...trap/engines/engine.py.229: process: 0.87640 | load: 0.00312 [I 2022-10-11 19:14:25] ...trap/engines/engine.py.230: loss: 0.12061 [I 2022-10-11 19:14:25] ...optimizers/trijoint.py.151: total_norm: 26.237871 [I 2022-10-11 19:14:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 2220/2304 [I 2022-10-11 19:14:34] ...trap/engines/engine.py.228: elapsed: 0:32:00 | left: 0:01:12 [I 2022-10-11 19:14:34] ...trap/engines/engine.py.229: process: 0.86955 | load: 0.00313 [I 2022-10-11 19:14:34] ...trap/engines/engine.py.230: loss: 0.19162 [I 2022-10-11 19:14:34] ...optimizers/trijoint.py.151: total_norm: 11.642625 [I 2022-10-11 19:14:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 2230/2304 [I 2022-10-11 19:14:42] ...trap/engines/engine.py.228: elapsed: 0:32:08 | left: 0:01:03 [I 2022-10-11 19:14:42] ...trap/engines/engine.py.229: process: 0.86692 | load: 0.00313 [I 2022-10-11 19:14:42] ...trap/engines/engine.py.230: loss: 0.11501 [I 2022-10-11 19:14:42] ...optimizers/trijoint.py.151: total_norm: 12.494533 [I 2022-10-11 19:14:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 2240/2304 [I 2022-10-11 19:14:51] ...trap/engines/engine.py.228: elapsed: 0:32:17 | left: 0:00:54 [I 2022-10-11 19:14:51] ...trap/engines/engine.py.229: process: 0.82418 | load: 0.00315 [I 2022-10-11 19:14:51] ...trap/engines/engine.py.230: loss: 0.08748 [I 2022-10-11 19:14:51] ...optimizers/trijoint.py.151: total_norm: 8.696275 [I 2022-10-11 19:14:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 2250/2304 [I 2022-10-11 19:14:59] ...trap/engines/engine.py.228: elapsed: 0:32:26 | left: 0:00:46 [I 2022-10-11 19:14:59] ...trap/engines/engine.py.229: process: 0.85810 | load: 0.00315 [I 2022-10-11 19:14:59] ...trap/engines/engine.py.230: loss: 0.09663 [I 2022-10-11 19:14:59] ...optimizers/trijoint.py.151: total_norm: 26.682837 [I 2022-10-11 19:15:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 2260/2304 [I 2022-10-11 19:15:08] ...trap/engines/engine.py.228: elapsed: 0:32:34 | left: 0:00:37 [I 2022-10-11 19:15:08] ...trap/engines/engine.py.229: process: 0.86148 | load: 0.00312 [I 2022-10-11 19:15:08] ...trap/engines/engine.py.230: loss: 0.07219 [I 2022-10-11 19:15:08] ...optimizers/trijoint.py.151: total_norm: 15.979341 [I 2022-10-11 19:15:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 2270/2304 [I 2022-10-11 19:15:16] ...trap/engines/engine.py.228: elapsed: 0:32:43 | left: 0:00:29 [I 2022-10-11 19:15:16] ...trap/engines/engine.py.229: process: 0.87323 | load: 0.00312 [I 2022-10-11 19:15:16] ...trap/engines/engine.py.230: loss: 0.10226 [I 2022-10-11 19:15:16] ...optimizers/trijoint.py.151: total_norm: 12.173046 [I 2022-10-11 19:15:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 2280/2304 [I 2022-10-11 19:15:25] ...trap/engines/engine.py.228: elapsed: 0:32:51 | left: 0:00:20 [I 2022-10-11 19:15:25] ...trap/engines/engine.py.229: process: 0.84458 | load: 0.00312 [I 2022-10-11 19:15:25] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-11 19:15:25] ...optimizers/trijoint.py.151: total_norm: 12.457965 [I 2022-10-11 19:15:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 2290/2304 [I 2022-10-11 19:15:34] ...trap/engines/engine.py.228: elapsed: 0:33:00 | left: 0:00:12 [I 2022-10-11 19:15:34] ...trap/engines/engine.py.229: process: 0.88103 | load: 0.00311 [I 2022-10-11 19:15:34] ...trap/engines/engine.py.230: loss: 0.09755 [I 2022-10-11 19:15:34] ...optimizers/trijoint.py.151: total_norm: 9.361787 [I 2022-10-11 19:15:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 2300/2304 [I 2022-10-11 19:15:42] ...trap/engines/engine.py.228: elapsed: 0:33:08 | left: 0:00:03 [I 2022-10-11 19:15:42] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00016 [I 2022-10-11 19:15:42] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-11 19:15:42] ...optimizers/trijoint.py.151: total_norm: 18.377754 [I 2022-10-11 19:15:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 2304/2304 [I 2022-10-11 19:15:46] ...trap/engines/engine.py.228: elapsed: 0:33:12 | left: 0:00:00 [I 2022-10-11 19:15:46] ...trap/engines/engine.py.229: process: 0.84408 | load: 0.00017 [I 2022-10-11 19:15:46] ...trap/engines/engine.py.230: loss: 0.07855 [I 2022-10-11 19:15:46] ...optimizers/trijoint.py.151: total_norm: 15.588224 [S 2022-10-11 19:15:47] ...trap/engines/engine.py.129: train_epoch.epoch: 115 [S 2022-10-11 19:15:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.11862133727479654 [I 2022-10-11 19:15:47] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 19:15:49] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 115 [I 2022-10-11 19:15:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 19:15:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 19:15:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 19:15:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 19:15:55] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 19:15:55] ...trap/engines/engine.py.321: eval: epoch 115 | batch 0/510 [I 2022-10-11 19:15:55] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:54:59 [I 2022-10-11 19:15:55] ...trap/engines/engine.py.326: process: 6.47020 | load: 5.05828 [I 2022-10-11 19:15:59] ...trap/engines/engine.py.321: eval: epoch 115 | batch 10/510 [I 2022-10-11 19:15:59] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:08:54 [I 2022-10-11 19:15:59] ...trap/engines/engine.py.326: process: 0.41787 | load: 0.00018 [I 2022-10-11 19:16:04] ...trap/engines/engine.py.321: eval: epoch 115 | batch 20/510 [I 2022-10-11 19:16:04] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:18 [I 2022-10-11 19:16:04] ...trap/engines/engine.py.326: process: 0.53583 | load: 0.00018 [I 2022-10-11 19:16:09] ...trap/engines/engine.py.321: eval: epoch 115 | batch 30/510 [I 2022-10-11 19:16:09] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:04:13 [I 2022-10-11 19:16:09] ...trap/engines/engine.py.326: process: 0.54201 | load: 0.00018 [I 2022-10-11 19:16:14] ...trap/engines/engine.py.321: eval: epoch 115 | batch 40/510 [I 2022-10-11 19:16:14] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:03:57 [I 2022-10-11 19:16:14] ...trap/engines/engine.py.326: process: 0.45208 | load: 0.00018 [I 2022-10-11 19:16:19] ...trap/engines/engine.py.321: eval: epoch 115 | batch 50/510 [I 2022-10-11 19:16:19] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:03:45 [I 2022-10-11 19:16:19] ...trap/engines/engine.py.326: process: 0.49256 | load: 0.00020 [I 2022-10-11 19:16:23] ...trap/engines/engine.py.321: eval: epoch 115 | batch 60/510 [I 2022-10-11 19:16:23] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:19 [I 2022-10-11 19:16:23] ...trap/engines/engine.py.326: process: 0.42286 | load: 0.00018 [I 2022-10-11 19:16:27] ...trap/engines/engine.py.321: eval: epoch 115 | batch 70/510 [I 2022-10-11 19:16:27] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:14 [I 2022-10-11 19:16:27] ...trap/engines/engine.py.326: process: 0.47293 | load: 0.00019 [I 2022-10-11 19:16:33] ...trap/engines/engine.py.321: eval: epoch 115 | batch 80/510 [I 2022-10-11 19:16:33] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:03:39 [I 2022-10-11 19:16:33] ...trap/engines/engine.py.326: process: 0.52053 | load: 0.00018 [I 2022-10-11 19:16:37] ...trap/engines/engine.py.321: eval: epoch 115 | batch 90/510 [I 2022-10-11 19:16:37] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:01 [I 2022-10-11 19:16:37] ...trap/engines/engine.py.326: process: 0.42896 | load: 0.00018 [I 2022-10-11 19:16:41] ...trap/engines/engine.py.321: eval: epoch 115 | batch 100/510 [I 2022-10-11 19:16:41] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:02:45 [I 2022-10-11 19:16:41] ...trap/engines/engine.py.326: process: 0.31062 | load: 0.00018 [I 2022-10-11 19:16:44] ...trap/engines/engine.py.321: eval: epoch 115 | batch 110/510 [I 2022-10-11 19:16:44] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:10 [I 2022-10-11 19:16:44] ...trap/engines/engine.py.326: process: 0.31815 | load: 0.00019 [I 2022-10-11 19:16:47] ...trap/engines/engine.py.321: eval: epoch 115 | batch 120/510 [I 2022-10-11 19:16:47] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:02 [I 2022-10-11 19:16:47] ...trap/engines/engine.py.326: process: 0.30800 | load: 0.00019 [I 2022-10-11 19:16:50] ...trap/engines/engine.py.321: eval: epoch 115 | batch 130/510 [I 2022-10-11 19:16:50] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:59 [I 2022-10-11 19:16:50] ...trap/engines/engine.py.326: process: 0.31750 | load: 0.00019 [I 2022-10-11 19:16:54] ...trap/engines/engine.py.321: eval: epoch 115 | batch 140/510 [I 2022-10-11 19:16:54] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:57 [I 2022-10-11 19:16:54] ...trap/engines/engine.py.326: process: 0.31821 | load: 0.00020 [I 2022-10-11 19:16:57] ...trap/engines/engine.py.321: eval: epoch 115 | batch 150/510 [I 2022-10-11 19:16:57] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:53 [I 2022-10-11 19:16:57] ...trap/engines/engine.py.326: process: 0.31545 | load: 0.00019 [I 2022-10-11 19:17:00] ...trap/engines/engine.py.321: eval: epoch 115 | batch 160/510 [I 2022-10-11 19:17:00] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:49 [I 2022-10-11 19:17:00] ...trap/engines/engine.py.326: process: 0.30798 | load: 0.00020 [I 2022-10-11 19:17:03] ...trap/engines/engine.py.321: eval: epoch 115 | batch 170/510 [I 2022-10-11 19:17:03] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:47 [I 2022-10-11 19:17:03] ...trap/engines/engine.py.326: process: 0.31793 | load: 0.00019 [I 2022-10-11 19:17:06] ...trap/engines/engine.py.321: eval: epoch 115 | batch 180/510 [I 2022-10-11 19:17:06] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:44 [I 2022-10-11 19:17:06] ...trap/engines/engine.py.326: process: 0.31256 | load: 0.00020 [I 2022-10-11 19:17:09] ...trap/engines/engine.py.321: eval: epoch 115 | batch 190/510 [I 2022-10-11 19:17:09] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:40 [I 2022-10-11 19:17:09] ...trap/engines/engine.py.326: process: 0.31431 | load: 0.00020 [I 2022-10-11 19:17:13] ...trap/engines/engine.py.321: eval: epoch 115 | batch 200/510 [I 2022-10-11 19:17:13] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:38 [I 2022-10-11 19:17:13] ...trap/engines/engine.py.326: process: 0.31848 | load: 0.00019 [I 2022-10-11 19:17:16] ...trap/engines/engine.py.321: eval: epoch 115 | batch 210/510 [I 2022-10-11 19:17:16] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:34 [I 2022-10-11 19:17:16] ...trap/engines/engine.py.326: process: 0.31339 | load: 0.00020 [I 2022-10-11 19:17:19] ...trap/engines/engine.py.321: eval: epoch 115 | batch 220/510 [I 2022-10-11 19:17:19] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:31 [I 2022-10-11 19:17:19] ...trap/engines/engine.py.326: process: 0.31728 | load: 0.00020 [I 2022-10-11 19:17:22] ...trap/engines/engine.py.321: eval: epoch 115 | batch 230/510 [I 2022-10-11 19:17:22] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:28 [I 2022-10-11 19:17:22] ...trap/engines/engine.py.326: process: 0.31840 | load: 0.00019 [I 2022-10-11 19:17:25] ...trap/engines/engine.py.321: eval: epoch 115 | batch 240/510 [I 2022-10-11 19:17:25] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:25 [I 2022-10-11 19:17:25] ...trap/engines/engine.py.326: process: 0.31410 | load: 0.00020 [I 2022-10-11 19:17:29] ...trap/engines/engine.py.321: eval: epoch 115 | batch 250/510 [I 2022-10-11 19:17:29] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:22 [I 2022-10-11 19:17:29] ...trap/engines/engine.py.326: process: 0.32036 | load: 0.00021 [I 2022-10-11 19:17:32] ...trap/engines/engine.py.321: eval: epoch 115 | batch 260/510 [I 2022-10-11 19:17:32] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:01:20 [I 2022-10-11 19:17:32] ...trap/engines/engine.py.326: process: 0.32421 | load: 0.00021 [I 2022-10-11 19:17:35] ...trap/engines/engine.py.321: eval: epoch 115 | batch 270/510 [I 2022-10-11 19:17:35] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:16 [I 2022-10-11 19:17:35] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00021 [I 2022-10-11 19:17:38] ...trap/engines/engine.py.321: eval: epoch 115 | batch 280/510 [I 2022-10-11 19:17:38] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:13 [I 2022-10-11 19:17:38] ...trap/engines/engine.py.326: process: 0.31692 | load: 0.00022 [I 2022-10-11 19:17:41] ...trap/engines/engine.py.321: eval: epoch 115 | batch 290/510 [I 2022-10-11 19:17:41] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:12 [I 2022-10-11 19:17:41] ...trap/engines/engine.py.326: process: 0.34030 | load: 0.00021 [I 2022-10-11 19:17:45] ...trap/engines/engine.py.321: eval: epoch 115 | batch 300/510 [I 2022-10-11 19:17:45] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:07 [I 2022-10-11 19:17:45] ...trap/engines/engine.py.326: process: 0.32080 | load: 0.00021 [I 2022-10-11 19:17:48] ...trap/engines/engine.py.321: eval: epoch 115 | batch 310/510 [I 2022-10-11 19:17:48] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:05 [I 2022-10-11 19:17:48] ...trap/engines/engine.py.326: process: 0.33710 | load: 0.00021 [I 2022-10-11 19:17:51] ...trap/engines/engine.py.321: eval: epoch 115 | batch 320/510 [I 2022-10-11 19:17:51] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:01 [I 2022-10-11 19:17:51] ...trap/engines/engine.py.326: process: 0.31997 | load: 0.00024 [I 2022-10-11 19:17:54] ...trap/engines/engine.py.321: eval: epoch 115 | batch 330/510 [I 2022-10-11 19:17:54] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:58 [I 2022-10-11 19:17:54] ...trap/engines/engine.py.326: process: 0.32666 | load: 0.00021 [I 2022-10-11 19:17:58] ...trap/engines/engine.py.321: eval: epoch 115 | batch 340/510 [I 2022-10-11 19:17:58] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:54 [I 2022-10-11 19:17:58] ...trap/engines/engine.py.326: process: 0.32415 | load: 0.00022 [I 2022-10-11 19:18:01] ...trap/engines/engine.py.321: eval: epoch 115 | batch 350/510 [I 2022-10-11 19:18:01] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:51 [I 2022-10-11 19:18:01] ...trap/engines/engine.py.326: process: 0.32902 | load: 0.00022 [I 2022-10-11 19:18:04] ...trap/engines/engine.py.321: eval: epoch 115 | batch 360/510 [I 2022-10-11 19:18:04] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:48 [I 2022-10-11 19:18:04] ...trap/engines/engine.py.326: process: 0.32568 | load: 0.00022 [I 2022-10-11 19:18:07] ...trap/engines/engine.py.321: eval: epoch 115 | batch 370/510 [I 2022-10-11 19:18:07] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:45 [I 2022-10-11 19:18:07] ...trap/engines/engine.py.326: process: 0.32034 | load: 0.00020 [I 2022-10-11 19:18:11] ...trap/engines/engine.py.321: eval: epoch 115 | batch 380/510 [I 2022-10-11 19:18:11] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-11 19:18:11] ...trap/engines/engine.py.326: process: 0.32386 | load: 0.00023 [I 2022-10-11 19:18:14] ...trap/engines/engine.py.321: eval: epoch 115 | batch 390/510 [I 2022-10-11 19:18:14] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:38 [I 2022-10-11 19:18:14] ...trap/engines/engine.py.326: process: 0.32151 | load: 0.00021 [I 2022-10-11 19:18:17] ...trap/engines/engine.py.321: eval: epoch 115 | batch 400/510 [I 2022-10-11 19:18:17] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:35 [I 2022-10-11 19:18:17] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00021 [I 2022-10-11 19:18:20] ...trap/engines/engine.py.321: eval: epoch 115 | batch 410/510 [I 2022-10-11 19:18:20] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:32 [I 2022-10-11 19:18:20] ...trap/engines/engine.py.326: process: 0.33294 | load: 0.00024 [I 2022-10-11 19:18:24] ...trap/engines/engine.py.321: eval: epoch 115 | batch 420/510 [I 2022-10-11 19:18:24] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:29 [I 2022-10-11 19:18:24] ...trap/engines/engine.py.326: process: 0.32990 | load: 0.00023 [I 2022-10-11 19:18:27] ...trap/engines/engine.py.321: eval: epoch 115 | batch 430/510 [I 2022-10-11 19:18:27] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:26 [I 2022-10-11 19:18:27] ...trap/engines/engine.py.326: process: 0.33500 | load: 0.00025 [I 2022-10-11 19:18:30] ...trap/engines/engine.py.321: eval: epoch 115 | batch 440/510 [I 2022-10-11 19:18:30] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:23 [I 2022-10-11 19:18:30] ...trap/engines/engine.py.326: process: 0.33367 | load: 0.00022 [I 2022-10-11 19:18:34] ...trap/engines/engine.py.321: eval: epoch 115 | batch 450/510 [I 2022-10-11 19:18:34] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:20 [I 2022-10-11 19:18:34] ...trap/engines/engine.py.326: process: 0.33039 | load: 0.00024 [I 2022-10-11 19:18:37] ...trap/engines/engine.py.321: eval: epoch 115 | batch 460/510 [I 2022-10-11 19:18:37] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:16 [I 2022-10-11 19:18:37] ...trap/engines/engine.py.326: process: 0.33240 | load: 0.00023 [I 2022-10-11 19:18:40] ...trap/engines/engine.py.321: eval: epoch 115 | batch 470/510 [I 2022-10-11 19:18:40] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:13 [I 2022-10-11 19:18:40] ...trap/engines/engine.py.326: process: 0.33921 | load: 0.00025 [I 2022-10-11 19:18:44] ...trap/engines/engine.py.321: eval: epoch 115 | batch 480/510 [I 2022-10-11 19:18:44] ...trap/engines/engine.py.325: elapsed: 0:02:55 | left: 0:00:10 [I 2022-10-11 19:18:44] ...trap/engines/engine.py.326: process: 0.34098 | load: 0.00024 [I 2022-10-11 19:18:47] ...trap/engines/engine.py.321: eval: epoch 115 | batch 490/510 [I 2022-10-11 19:18:47] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:06 [I 2022-10-11 19:18:47] ...trap/engines/engine.py.326: process: 0.33655 | load: 0.00024 [I 2022-10-11 19:18:50] ...trap/engines/engine.py.321: eval: epoch 115 | batch 500/510 [I 2022-10-11 19:18:50] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:03 [I 2022-10-11 19:18:50] ...trap/engines/engine.py.326: process: 0.33816 | load: 0.00023 [I 2022-10-11 19:18:54] ...trap/engines/engine.py.321: eval: epoch 115 | batch 510/510 [I 2022-10-11 19:18:54] ...trap/engines/engine.py.325: elapsed: 0:03:05 | left: 0:00:00 [I 2022-10-11 19:18:54] ...trap/engines/engine.py.326: process: 0.32362 | load: 0.00019 [S 2022-10-11 19:18:55] ...trap/engines/engine.py.132: eval_epoch.epoch: 115 [I 2022-10-11 19:18:55] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.745900 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909400 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935700 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.744400 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907700 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014788 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009982 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007734 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017431 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007390 [S 2022-10-11 19:19:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008324 [I 2022-10-11 19:19:37] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 19:19:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 19:19:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 19:19:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 19:19:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 19:19:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 19:19:46] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 19:19:49] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 19:19:49] ...trap/engines/engine.py.163: Training model on trainset for epoch 116 [I 2022-10-11 19:19:49] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 19:19:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 0/2304 [I 2022-10-11 19:19:52] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:51 [I 2022-10-11 19:19:52] ...trap/engines/engine.py.229: process: 3.25147 | load: 2.31400 [I 2022-10-11 19:19:52] ...trap/engines/engine.py.230: loss: 0.10502 [I 2022-10-11 19:19:52] ...optimizers/trijoint.py.151: total_norm: 18.522978 [I 2022-10-11 19:20:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 10/2304 [I 2022-10-11 19:20:01] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:27 [I 2022-10-11 19:20:01] ...trap/engines/engine.py.229: process: 0.85289 | load: 0.00541 [I 2022-10-11 19:20:01] ...trap/engines/engine.py.230: loss: 0.10173 [I 2022-10-11 19:20:01] ...optimizers/trijoint.py.151: total_norm: 15.699224 [I 2022-10-11 19:20:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 20/2304 [I 2022-10-11 19:20:09] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:59 [I 2022-10-11 19:20:09] ...trap/engines/engine.py.229: process: 0.87007 | load: 0.00485 [I 2022-10-11 19:20:09] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-11 19:20:09] ...optimizers/trijoint.py.151: total_norm: 14.430120 [I 2022-10-11 19:20:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 30/2304 [I 2022-10-11 19:20:18] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:37 [I 2022-10-11 19:20:18] ...trap/engines/engine.py.229: process: 0.85609 | load: 0.00514 [I 2022-10-11 19:20:18] ...trap/engines/engine.py.230: loss: 0.13900 [I 2022-10-11 19:20:18] ...optimizers/trijoint.py.151: total_norm: 9.903500 [I 2022-10-11 19:20:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 40/2304 [I 2022-10-11 19:20:27] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:23 [I 2022-10-11 19:20:27] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00386 [I 2022-10-11 19:20:27] ...trap/engines/engine.py.230: loss: 0.13721 [I 2022-10-11 19:20:27] ...optimizers/trijoint.py.151: total_norm: 15.983243 [I 2022-10-11 19:20:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 50/2304 [I 2022-10-11 19:20:35] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:12 [I 2022-10-11 19:20:35] ...trap/engines/engine.py.229: process: 0.86035 | load: 0.00408 [I 2022-10-11 19:20:35] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-11 19:20:35] ...optimizers/trijoint.py.151: total_norm: 15.800387 [I 2022-10-11 19:20:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 60/2304 [I 2022-10-11 19:20:44] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:55 [I 2022-10-11 19:20:44] ...trap/engines/engine.py.229: process: 0.84909 | load: 0.00362 [I 2022-10-11 19:20:44] ...trap/engines/engine.py.230: loss: 0.09786 [I 2022-10-11 19:20:44] ...optimizers/trijoint.py.151: total_norm: 11.791819 [I 2022-10-11 19:20:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 70/2304 [I 2022-10-11 19:20:53] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:32:16 [I 2022-10-11 19:20:53] ...trap/engines/engine.py.229: process: 0.86981 | load: 0.00408 [I 2022-10-11 19:20:53] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-11 19:20:53] ...optimizers/trijoint.py.151: total_norm: 10.688256 [I 2022-10-11 19:21:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 80/2304 [I 2022-10-11 19:21:01] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:51 [I 2022-10-11 19:21:01] ...trap/engines/engine.py.229: process: 0.86232 | load: 0.00333 [I 2022-10-11 19:21:01] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-11 19:21:01] ...optimizers/trijoint.py.151: total_norm: 22.100224 [I 2022-10-11 19:21:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 90/2304 [I 2022-10-11 19:21:10] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:43 [I 2022-10-11 19:21:10] ...trap/engines/engine.py.229: process: 0.87388 | load: 0.00347 [I 2022-10-11 19:21:10] ...trap/engines/engine.py.230: loss: 0.08848 [I 2022-10-11 19:21:10] ...optimizers/trijoint.py.151: total_norm: 13.138365 [I 2022-10-11 19:21:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 100/2304 [I 2022-10-11 19:21:18] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:22 [I 2022-10-11 19:21:18] ...trap/engines/engine.py.229: process: 0.84031 | load: 0.00363 [I 2022-10-11 19:21:18] ...trap/engines/engine.py.230: loss: 0.08509 [I 2022-10-11 19:21:18] ...optimizers/trijoint.py.151: total_norm: 21.909845 [I 2022-10-11 19:21:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 110/2304 [I 2022-10-11 19:21:27] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:12 [I 2022-10-11 19:21:27] ...trap/engines/engine.py.229: process: 0.85456 | load: 0.00323 [I 2022-10-11 19:21:27] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-11 19:21:27] ...optimizers/trijoint.py.151: total_norm: 17.568794 [I 2022-10-11 19:21:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 120/2304 [I 2022-10-11 19:21:35] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:30:53 [I 2022-10-11 19:21:35] ...trap/engines/engine.py.229: process: 0.85709 | load: 0.00309 [I 2022-10-11 19:21:35] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-11 19:21:35] ...optimizers/trijoint.py.151: total_norm: 11.481809 [I 2022-10-11 19:21:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 130/2304 [I 2022-10-11 19:21:44] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:30:43 [I 2022-10-11 19:21:44] ...trap/engines/engine.py.229: process: 0.85646 | load: 0.00332 [I 2022-10-11 19:21:44] ...trap/engines/engine.py.230: loss: 0.08554 [I 2022-10-11 19:21:44] ...optimizers/trijoint.py.151: total_norm: 12.432377 [I 2022-10-11 19:21:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 140/2304 [I 2022-10-11 19:21:52] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:38 [I 2022-10-11 19:21:52] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00307 [I 2022-10-11 19:21:52] ...trap/engines/engine.py.230: loss: 0.09505 [I 2022-10-11 19:21:52] ...optimizers/trijoint.py.151: total_norm: 8.636547 [I 2022-10-11 19:22:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 150/2304 [I 2022-10-11 19:22:01] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:26 [I 2022-10-11 19:22:01] ...trap/engines/engine.py.229: process: 0.84565 | load: 0.00314 [I 2022-10-11 19:22:01] ...trap/engines/engine.py.230: loss: 0.14173 [I 2022-10-11 19:22:01] ...optimizers/trijoint.py.151: total_norm: 9.733123 [I 2022-10-11 19:22:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 160/2304 [I 2022-10-11 19:22:09] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:10 [I 2022-10-11 19:22:09] ...trap/engines/engine.py.229: process: 0.83842 | load: 0.00326 [I 2022-10-11 19:22:09] ...trap/engines/engine.py.230: loss: 0.10897 [I 2022-10-11 19:22:09] ...optimizers/trijoint.py.151: total_norm: 16.184984 [I 2022-10-11 19:22:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 170/2304 [I 2022-10-11 19:22:18] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:13 [I 2022-10-11 19:22:18] ...trap/engines/engine.py.229: process: 0.85821 | load: 0.00303 [I 2022-10-11 19:22:18] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-11 19:22:18] ...optimizers/trijoint.py.151: total_norm: 23.011314 [I 2022-10-11 19:22:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 180/2304 [I 2022-10-11 19:22:26] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:11 [I 2022-10-11 19:22:26] ...trap/engines/engine.py.229: process: 0.86305 | load: 0.00296 [I 2022-10-11 19:22:26] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-11 19:22:26] ...optimizers/trijoint.py.151: total_norm: 9.684629 [I 2022-10-11 19:22:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 190/2304 [I 2022-10-11 19:22:35] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:29:47 [I 2022-10-11 19:22:35] ...trap/engines/engine.py.229: process: 0.85484 | load: 0.00296 [I 2022-10-11 19:22:35] ...trap/engines/engine.py.230: loss: 0.09175 [I 2022-10-11 19:22:35] ...optimizers/trijoint.py.151: total_norm: 13.323124 [I 2022-10-11 19:22:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 200/2304 [I 2022-10-11 19:22:43] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:29:28 [I 2022-10-11 19:22:43] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00313 [I 2022-10-11 19:22:43] ...trap/engines/engine.py.230: loss: 0.17995 [I 2022-10-11 19:22:43] ...optimizers/trijoint.py.151: total_norm: 16.829292 [I 2022-10-11 19:22:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 210/2304 [I 2022-10-11 19:22:52] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:29:22 [I 2022-10-11 19:22:52] ...trap/engines/engine.py.229: process: 0.82580 | load: 0.00298 [I 2022-10-11 19:22:52] ...trap/engines/engine.py.230: loss: 0.08798 [I 2022-10-11 19:22:52] ...optimizers/trijoint.py.151: total_norm: 11.962703 [I 2022-10-11 19:23:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 220/2304 [I 2022-10-11 19:23:00] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:12 [I 2022-10-11 19:23:00] ...trap/engines/engine.py.229: process: 0.85273 | load: 0.00301 [I 2022-10-11 19:23:00] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-11 19:23:00] ...optimizers/trijoint.py.151: total_norm: 8.940181 [I 2022-10-11 19:23:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 230/2304 [I 2022-10-11 19:23:09] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:29:19 [I 2022-10-11 19:23:09] ...trap/engines/engine.py.229: process: 0.86261 | load: 0.00299 [I 2022-10-11 19:23:09] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-11 19:23:09] ...optimizers/trijoint.py.151: total_norm: 11.170968 [I 2022-10-11 19:23:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 240/2304 [I 2022-10-11 19:23:17] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:55 [I 2022-10-11 19:23:17] ...trap/engines/engine.py.229: process: 0.87495 | load: 0.00407 [I 2022-10-11 19:23:17] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-11 19:23:17] ...optimizers/trijoint.py.151: total_norm: 21.036093 [I 2022-10-11 19:23:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 250/2304 [I 2022-10-11 19:23:26] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:59 [I 2022-10-11 19:23:26] ...trap/engines/engine.py.229: process: 0.87956 | load: 0.00400 [I 2022-10-11 19:23:26] ...trap/engines/engine.py.230: loss: 0.10034 [I 2022-10-11 19:23:26] ...optimizers/trijoint.py.151: total_norm: 15.812260 [I 2022-10-11 19:23:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 260/2304 [I 2022-10-11 19:23:35] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:29:36 [I 2022-10-11 19:23:35] ...trap/engines/engine.py.229: process: 0.86769 | load: 0.00392 [I 2022-10-11 19:23:35] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-11 19:23:35] ...optimizers/trijoint.py.151: total_norm: 15.315974 [I 2022-10-11 19:23:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 270/2304 [I 2022-10-11 19:23:44] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:29:44 [I 2022-10-11 19:23:44] ...trap/engines/engine.py.229: process: 0.87930 | load: 0.00378 [I 2022-10-11 19:23:44] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-11 19:23:44] ...optimizers/trijoint.py.151: total_norm: 24.429434 [I 2022-10-11 19:23:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 280/2304 [I 2022-10-11 19:23:52] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:29:25 [I 2022-10-11 19:23:52] ...trap/engines/engine.py.229: process: 0.87538 | load: 0.00390 [I 2022-10-11 19:23:52] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-11 19:23:52] ...optimizers/trijoint.py.151: total_norm: 11.461804 [I 2022-10-11 19:24:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 290/2304 [I 2022-10-11 19:24:01] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:29:52 [I 2022-10-11 19:24:01] ...trap/engines/engine.py.229: process: 0.88801 | load: 0.00409 [I 2022-10-11 19:24:01] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-11 19:24:01] ...optimizers/trijoint.py.151: total_norm: 11.607430 [I 2022-10-11 19:24:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 300/2304 [I 2022-10-11 19:24:10] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:29:27 [I 2022-10-11 19:24:10] ...trap/engines/engine.py.229: process: 0.89048 | load: 0.00403 [I 2022-10-11 19:24:10] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-11 19:24:10] ...optimizers/trijoint.py.151: total_norm: 16.539066 [I 2022-10-11 19:24:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 310/2304 [I 2022-10-11 19:24:19] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:29:25 [I 2022-10-11 19:24:19] ...trap/engines/engine.py.229: process: 0.87681 | load: 0.00393 [I 2022-10-11 19:24:19] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-11 19:24:19] ...optimizers/trijoint.py.151: total_norm: 11.411946 [I 2022-10-11 19:24:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 320/2304 [I 2022-10-11 19:24:28] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:29:24 [I 2022-10-11 19:24:28] ...trap/engines/engine.py.229: process: 0.88952 | load: 0.00407 [I 2022-10-11 19:24:28] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-11 19:24:28] ...optimizers/trijoint.py.151: total_norm: 11.815798 [I 2022-10-11 19:24:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 330/2304 [I 2022-10-11 19:24:37] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:29:19 [I 2022-10-11 19:24:37] ...trap/engines/engine.py.229: process: 0.88904 | load: 0.00397 [I 2022-10-11 19:24:37] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-11 19:24:37] ...optimizers/trijoint.py.151: total_norm: 15.820830 [I 2022-10-11 19:24:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 340/2304 [I 2022-10-11 19:24:46] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:29:13 [I 2022-10-11 19:24:46] ...trap/engines/engine.py.229: process: 0.89866 | load: 0.00394 [I 2022-10-11 19:24:46] ...trap/engines/engine.py.230: loss: 0.18920 [I 2022-10-11 19:24:46] ...optimizers/trijoint.py.151: total_norm: 22.425232 [I 2022-10-11 19:24:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 350/2304 [I 2022-10-11 19:24:55] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:28:51 [I 2022-10-11 19:24:55] ...trap/engines/engine.py.229: process: 0.87790 | load: 0.00397 [I 2022-10-11 19:24:55] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-11 19:24:55] ...optimizers/trijoint.py.151: total_norm: 12.093654 [I 2022-10-11 19:25:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 360/2304 [I 2022-10-11 19:25:03] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:51 [I 2022-10-11 19:25:03] ...trap/engines/engine.py.229: process: 0.84917 | load: 0.00311 [I 2022-10-11 19:25:03] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-11 19:25:03] ...optimizers/trijoint.py.151: total_norm: 13.943562 [I 2022-10-11 19:25:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 370/2304 [I 2022-10-11 19:25:12] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:27:35 [I 2022-10-11 19:25:12] ...trap/engines/engine.py.229: process: 0.85169 | load: 0.00313 [I 2022-10-11 19:25:12] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-11 19:25:12] ...optimizers/trijoint.py.151: total_norm: 14.405217 [I 2022-10-11 19:25:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 380/2304 [I 2022-10-11 19:25:20] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:27:28 [I 2022-10-11 19:25:20] ...trap/engines/engine.py.229: process: 0.84404 | load: 0.00312 [I 2022-10-11 19:25:20] ...trap/engines/engine.py.230: loss: 0.09143 [I 2022-10-11 19:25:20] ...optimizers/trijoint.py.151: total_norm: 17.743975 [I 2022-10-11 19:25:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 390/2304 [I 2022-10-11 19:25:29] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:27:21 [I 2022-10-11 19:25:29] ...trap/engines/engine.py.229: process: 0.86076 | load: 0.00311 [I 2022-10-11 19:25:29] ...trap/engines/engine.py.230: loss: 0.07576 [I 2022-10-11 19:25:29] ...optimizers/trijoint.py.151: total_norm: 15.007595 [I 2022-10-11 19:25:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 400/2304 [I 2022-10-11 19:25:37] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:27:01 [I 2022-10-11 19:25:37] ...trap/engines/engine.py.229: process: 0.86169 | load: 0.00326 [I 2022-10-11 19:25:37] ...trap/engines/engine.py.230: loss: 0.07797 [I 2022-10-11 19:25:37] ...optimizers/trijoint.py.151: total_norm: 20.057322 [I 2022-10-11 19:25:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 410/2304 [I 2022-10-11 19:25:46] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:27:52 [I 2022-10-11 19:25:46] ...trap/engines/engine.py.229: process: 0.89288 | load: 0.00408 [I 2022-10-11 19:25:46] ...trap/engines/engine.py.230: loss: 0.09229 [I 2022-10-11 19:25:46] ...optimizers/trijoint.py.151: total_norm: 9.286686 [I 2022-10-11 19:25:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 420/2304 [I 2022-10-11 19:25:55] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:27:52 [I 2022-10-11 19:25:55] ...trap/engines/engine.py.229: process: 0.88442 | load: 0.00398 [I 2022-10-11 19:25:55] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-11 19:25:55] ...optimizers/trijoint.py.151: total_norm: 37.935329 [I 2022-10-11 19:26:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 430/2304 [I 2022-10-11 19:26:04] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:27:48 [I 2022-10-11 19:26:04] ...trap/engines/engine.py.229: process: 0.88637 | load: 0.00407 [I 2022-10-11 19:26:04] ...trap/engines/engine.py.230: loss: 0.12341 [I 2022-10-11 19:26:04] ...optimizers/trijoint.py.151: total_norm: 16.976088 [I 2022-10-11 19:26:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 440/2304 [I 2022-10-11 19:26:13] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:27:35 [I 2022-10-11 19:26:13] ...trap/engines/engine.py.229: process: 0.89376 | load: 0.00397 [I 2022-10-11 19:26:13] ...trap/engines/engine.py.230: loss: 0.15071 [I 2022-10-11 19:26:13] ...optimizers/trijoint.py.151: total_norm: 11.950440 [I 2022-10-11 19:26:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 450/2304 [I 2022-10-11 19:26:22] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:27:30 [I 2022-10-11 19:26:22] ...trap/engines/engine.py.229: process: 0.87615 | load: 0.00396 [I 2022-10-11 19:26:22] ...trap/engines/engine.py.230: loss: 0.10691 [I 2022-10-11 19:26:22] ...optimizers/trijoint.py.151: total_norm: 35.631092 [I 2022-10-11 19:26:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 460/2304 [I 2022-10-11 19:26:30] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:26:29 [I 2022-10-11 19:26:30] ...trap/engines/engine.py.229: process: 0.87328 | load: 0.00315 [I 2022-10-11 19:26:30] ...trap/engines/engine.py.230: loss: 0.09523 [I 2022-10-11 19:26:30] ...optimizers/trijoint.py.151: total_norm: 10.950471 [I 2022-10-11 19:26:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 470/2304 [I 2022-10-11 19:26:39] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:26:17 [I 2022-10-11 19:26:39] ...trap/engines/engine.py.229: process: 0.84965 | load: 0.00317 [I 2022-10-11 19:26:39] ...trap/engines/engine.py.230: loss: 0.08243 [I 2022-10-11 19:26:39] ...optimizers/trijoint.py.151: total_norm: 13.288659 [I 2022-10-11 19:26:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 480/2304 [I 2022-10-11 19:26:48] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:26:54 [I 2022-10-11 19:26:48] ...trap/engines/engine.py.229: process: 0.88581 | load: 0.00393 [I 2022-10-11 19:26:48] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-11 19:26:48] ...optimizers/trijoint.py.151: total_norm: 9.936185 [I 2022-10-11 19:26:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 490/2304 [I 2022-10-11 19:26:57] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:26:49 [I 2022-10-11 19:26:57] ...trap/engines/engine.py.229: process: 0.88405 | load: 0.00403 [I 2022-10-11 19:26:57] ...trap/engines/engine.py.230: loss: 0.14059 [I 2022-10-11 19:26:57] ...optimizers/trijoint.py.151: total_norm: 12.145042 [I 2022-10-11 19:27:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 500/2304 [I 2022-10-11 19:27:06] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:26:34 [I 2022-10-11 19:27:06] ...trap/engines/engine.py.229: process: 0.89083 | load: 0.00394 [I 2022-10-11 19:27:06] ...trap/engines/engine.py.230: loss: 0.07638 [I 2022-10-11 19:27:06] ...optimizers/trijoint.py.151: total_norm: 10.697621 [I 2022-10-11 19:27:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 510/2304 [I 2022-10-11 19:27:14] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:25:42 [I 2022-10-11 19:27:14] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00312 [I 2022-10-11 19:27:14] ...trap/engines/engine.py.230: loss: 0.09217 [I 2022-10-11 19:27:14] ...optimizers/trijoint.py.151: total_norm: 16.088989 [I 2022-10-11 19:27:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 520/2304 [I 2022-10-11 19:27:23] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:25:26 [I 2022-10-11 19:27:23] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00314 [I 2022-10-11 19:27:23] ...trap/engines/engine.py.230: loss: 0.11229 [I 2022-10-11 19:27:23] ...optimizers/trijoint.py.151: total_norm: 19.751280 [I 2022-10-11 19:27:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 530/2304 [I 2022-10-11 19:27:32] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:25:18 [I 2022-10-11 19:27:32] ...trap/engines/engine.py.229: process: 0.85283 | load: 0.00314 [I 2022-10-11 19:27:32] ...trap/engines/engine.py.230: loss: 0.16582 [I 2022-10-11 19:27:32] ...optimizers/trijoint.py.151: total_norm: 18.281656 [I 2022-10-11 19:27:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 540/2304 [I 2022-10-11 19:27:40] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:25:15 [I 2022-10-11 19:27:40] ...trap/engines/engine.py.229: process: 0.86637 | load: 0.00312 [I 2022-10-11 19:27:40] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-11 19:27:40] ...optimizers/trijoint.py.151: total_norm: 23.960169 [I 2022-10-11 19:27:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 550/2304 [I 2022-10-11 19:27:49] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:24:58 [I 2022-10-11 19:27:49] ...trap/engines/engine.py.229: process: 0.85618 | load: 0.00317 [I 2022-10-11 19:27:49] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-11 19:27:49] ...optimizers/trijoint.py.151: total_norm: 12.105129 [I 2022-10-11 19:27:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 560/2304 [I 2022-10-11 19:27:57] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:24:36 [I 2022-10-11 19:27:57] ...trap/engines/engine.py.229: process: 0.84228 | load: 0.00310 [I 2022-10-11 19:27:57] ...trap/engines/engine.py.230: loss: 0.10054 [I 2022-10-11 19:27:57] ...optimizers/trijoint.py.151: total_norm: 20.720463 [I 2022-10-11 19:28:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 570/2304 [I 2022-10-11 19:28:06] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:24:25 [I 2022-10-11 19:28:06] ...trap/engines/engine.py.229: process: 0.86214 | load: 0.00301 [I 2022-10-11 19:28:06] ...trap/engines/engine.py.230: loss: 0.09322 [I 2022-10-11 19:28:06] ...optimizers/trijoint.py.151: total_norm: 21.434626 [I 2022-10-11 19:28:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 580/2304 [I 2022-10-11 19:28:14] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:24:19 [I 2022-10-11 19:28:14] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00296 [I 2022-10-11 19:28:14] ...trap/engines/engine.py.230: loss: 0.12961 [I 2022-10-11 19:28:14] ...optimizers/trijoint.py.151: total_norm: 27.425417 [I 2022-10-11 19:28:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 590/2304 [I 2022-10-11 19:28:23] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:24:05 [I 2022-10-11 19:28:23] ...trap/engines/engine.py.229: process: 0.84316 | load: 0.00294 [I 2022-10-11 19:28:23] ...trap/engines/engine.py.230: loss: 0.11772 [I 2022-10-11 19:28:23] ...optimizers/trijoint.py.151: total_norm: 14.232426 [I 2022-10-11 19:28:31] ...trap/engines/engine.py.224: train: epoch 116 | batch 600/2304 [I 2022-10-11 19:28:31] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:24:06 [I 2022-10-11 19:28:31] ...trap/engines/engine.py.229: process: 0.85265 | load: 0.00300 [I 2022-10-11 19:28:31] ...trap/engines/engine.py.230: loss: 0.10227 [I 2022-10-11 19:28:31] ...optimizers/trijoint.py.151: total_norm: 18.195389 [I 2022-10-11 19:28:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 610/2304 [I 2022-10-11 19:28:40] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:24:03 [I 2022-10-11 19:28:40] ...trap/engines/engine.py.229: process: 0.85131 | load: 0.00295 [I 2022-10-11 19:28:40] ...trap/engines/engine.py.230: loss: 0.10594 [I 2022-10-11 19:28:40] ...optimizers/trijoint.py.151: total_norm: 9.162253 [I 2022-10-11 19:28:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 620/2304 [I 2022-10-11 19:28:48] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:23:56 [I 2022-10-11 19:28:48] ...trap/engines/engine.py.229: process: 0.84304 | load: 0.00297 [I 2022-10-11 19:28:48] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-11 19:28:48] ...optimizers/trijoint.py.151: total_norm: 19.609755 [I 2022-10-11 19:28:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 630/2304 [I 2022-10-11 19:28:57] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:23:47 [I 2022-10-11 19:28:57] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00294 [I 2022-10-11 19:28:57] ...trap/engines/engine.py.230: loss: 0.09819 [I 2022-10-11 19:28:57] ...optimizers/trijoint.py.151: total_norm: 9.265731 [I 2022-10-11 19:29:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 640/2304 [I 2022-10-11 19:29:05] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:23:35 [I 2022-10-11 19:29:05] ...trap/engines/engine.py.229: process: 0.84657 | load: 0.00297 [I 2022-10-11 19:29:05] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-11 19:29:05] ...optimizers/trijoint.py.151: total_norm: 13.062224 [I 2022-10-11 19:29:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 650/2304 [I 2022-10-11 19:29:14] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:34 [I 2022-10-11 19:29:14] ...trap/engines/engine.py.229: process: 0.85197 | load: 0.00299 [I 2022-10-11 19:29:14] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-11 19:29:14] ...optimizers/trijoint.py.151: total_norm: 14.969061 [I 2022-10-11 19:29:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 660/2304 [I 2022-10-11 19:29:22] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:23:18 [I 2022-10-11 19:29:22] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00299 [I 2022-10-11 19:29:22] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-11 19:29:22] ...optimizers/trijoint.py.151: total_norm: 16.632244 [I 2022-10-11 19:29:31] ...trap/engines/engine.py.224: train: epoch 116 | batch 670/2304 [I 2022-10-11 19:29:31] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:00 [I 2022-10-11 19:29:31] ...trap/engines/engine.py.229: process: 0.84483 | load: 0.00299 [I 2022-10-11 19:29:31] ...trap/engines/engine.py.230: loss: 0.14419 [I 2022-10-11 19:29:31] ...optimizers/trijoint.py.151: total_norm: 27.281057 [I 2022-10-11 19:29:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 680/2304 [I 2022-10-11 19:29:39] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:22:56 [I 2022-10-11 19:29:39] ...trap/engines/engine.py.229: process: 0.84462 | load: 0.00317 [I 2022-10-11 19:29:39] ...trap/engines/engine.py.230: loss: 0.07954 [I 2022-10-11 19:29:39] ...optimizers/trijoint.py.151: total_norm: 12.936293 [I 2022-10-11 19:29:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 690/2304 [I 2022-10-11 19:29:48] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:34 [I 2022-10-11 19:29:48] ...trap/engines/engine.py.229: process: 0.82763 | load: 0.00301 [I 2022-10-11 19:29:48] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-11 19:29:48] ...optimizers/trijoint.py.151: total_norm: 25.114531 [I 2022-10-11 19:29:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 700/2304 [I 2022-10-11 19:29:56] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:22:48 [I 2022-10-11 19:29:56] ...trap/engines/engine.py.229: process: 0.84885 | load: 0.00297 [I 2022-10-11 19:29:56] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-11 19:29:56] ...optimizers/trijoint.py.151: total_norm: 35.812309 [I 2022-10-11 19:30:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 710/2304 [I 2022-10-11 19:30:05] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:30 [I 2022-10-11 19:30:05] ...trap/engines/engine.py.229: process: 0.85205 | load: 0.00302 [I 2022-10-11 19:30:05] ...trap/engines/engine.py.230: loss: 0.08077 [I 2022-10-11 19:30:05] ...optimizers/trijoint.py.151: total_norm: 12.705035 [I 2022-10-11 19:30:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 720/2304 [I 2022-10-11 19:30:13] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:22:24 [I 2022-10-11 19:30:13] ...trap/engines/engine.py.229: process: 0.85690 | load: 0.00299 [I 2022-10-11 19:30:13] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-11 19:30:13] ...optimizers/trijoint.py.151: total_norm: 18.424850 [I 2022-10-11 19:30:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 730/2304 [I 2022-10-11 19:30:22] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:03 [I 2022-10-11 19:30:22] ...trap/engines/engine.py.229: process: 0.83427 | load: 0.00304 [I 2022-10-11 19:30:22] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-11 19:30:22] ...optimizers/trijoint.py.151: total_norm: 16.505444 [I 2022-10-11 19:30:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 740/2304 [I 2022-10-11 19:30:30] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:22:06 [I 2022-10-11 19:30:30] ...trap/engines/engine.py.229: process: 0.84892 | load: 0.00301 [I 2022-10-11 19:30:30] ...trap/engines/engine.py.230: loss: 0.09035 [I 2022-10-11 19:30:30] ...optimizers/trijoint.py.151: total_norm: 9.694239 [I 2022-10-11 19:30:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 750/2304 [I 2022-10-11 19:30:39] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:22:17 [I 2022-10-11 19:30:39] ...trap/engines/engine.py.229: process: 0.86343 | load: 0.00316 [I 2022-10-11 19:30:39] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-11 19:30:39] ...optimizers/trijoint.py.151: total_norm: 13.472230 [I 2022-10-11 19:30:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 760/2304 [I 2022-10-11 19:30:48] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:22:02 [I 2022-10-11 19:30:48] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00314 [I 2022-10-11 19:30:48] ...trap/engines/engine.py.230: loss: 0.12683 [I 2022-10-11 19:30:48] ...optimizers/trijoint.py.151: total_norm: 13.492589 [I 2022-10-11 19:30:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 770/2304 [I 2022-10-11 19:30:56] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:22:05 [I 2022-10-11 19:30:56] ...trap/engines/engine.py.229: process: 0.84006 | load: 0.00310 [I 2022-10-11 19:30:56] ...trap/engines/engine.py.230: loss: 0.10719 [I 2022-10-11 19:30:56] ...optimizers/trijoint.py.151: total_norm: 7.571833 [I 2022-10-11 19:31:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 780/2304 [I 2022-10-11 19:31:05] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:51 [I 2022-10-11 19:31:05] ...trap/engines/engine.py.229: process: 0.85793 | load: 0.00316 [I 2022-10-11 19:31:05] ...trap/engines/engine.py.230: loss: 0.12796 [I 2022-10-11 19:31:05] ...optimizers/trijoint.py.151: total_norm: 38.422340 [I 2022-10-11 19:31:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 790/2304 [I 2022-10-11 19:31:13] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:21:33 [I 2022-10-11 19:31:13] ...trap/engines/engine.py.229: process: 0.85099 | load: 0.00317 [I 2022-10-11 19:31:13] ...trap/engines/engine.py.230: loss: 0.10058 [I 2022-10-11 19:31:13] ...optimizers/trijoint.py.151: total_norm: 20.147964 [I 2022-10-11 19:31:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 800/2304 [I 2022-10-11 19:31:22] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:22:00 [I 2022-10-11 19:31:22] ...trap/engines/engine.py.229: process: 0.89502 | load: 0.00393 [I 2022-10-11 19:31:22] ...trap/engines/engine.py.230: loss: 0.11245 [I 2022-10-11 19:31:22] ...optimizers/trijoint.py.151: total_norm: 18.541735 [I 2022-10-11 19:31:31] ...trap/engines/engine.py.224: train: epoch 116 | batch 810/2304 [I 2022-10-11 19:31:31] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:21:59 [I 2022-10-11 19:31:31] ...trap/engines/engine.py.229: process: 0.88821 | load: 0.00395 [I 2022-10-11 19:31:31] ...trap/engines/engine.py.230: loss: 0.11334 [I 2022-10-11 19:31:31] ...optimizers/trijoint.py.151: total_norm: 14.538661 [I 2022-10-11 19:31:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 820/2304 [I 2022-10-11 19:31:40] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:21:59 [I 2022-10-11 19:31:40] ...trap/engines/engine.py.229: process: 0.90160 | load: 0.00399 [I 2022-10-11 19:31:40] ...trap/engines/engine.py.230: loss: 0.09120 [I 2022-10-11 19:31:40] ...optimizers/trijoint.py.151: total_norm: 19.042753 [I 2022-10-11 19:31:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 830/2304 [I 2022-10-11 19:31:49] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:21:53 [I 2022-10-11 19:31:49] ...trap/engines/engine.py.229: process: 0.88357 | load: 0.00410 [I 2022-10-11 19:31:49] ...trap/engines/engine.py.230: loss: 0.10761 [I 2022-10-11 19:31:49] ...optimizers/trijoint.py.151: total_norm: 18.249138 [I 2022-10-11 19:31:58] ...trap/engines/engine.py.224: train: epoch 116 | batch 840/2304 [I 2022-10-11 19:31:58] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:21:39 [I 2022-10-11 19:31:58] ...trap/engines/engine.py.229: process: 0.88121 | load: 0.00397 [I 2022-10-11 19:31:58] ...trap/engines/engine.py.230: loss: 0.13052 [I 2022-10-11 19:31:58] ...optimizers/trijoint.py.151: total_norm: 16.753262 [I 2022-10-11 19:32:07] ...trap/engines/engine.py.224: train: epoch 116 | batch 850/2304 [I 2022-10-11 19:32:07] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:21:33 [I 2022-10-11 19:32:07] ...trap/engines/engine.py.229: process: 0.89298 | load: 0.00398 [I 2022-10-11 19:32:07] ...trap/engines/engine.py.230: loss: 0.08539 [I 2022-10-11 19:32:07] ...optimizers/trijoint.py.151: total_norm: 8.580276 [I 2022-10-11 19:32:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 860/2304 [I 2022-10-11 19:32:16] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:21:22 [I 2022-10-11 19:32:16] ...trap/engines/engine.py.229: process: 0.86715 | load: 0.00393 [I 2022-10-11 19:32:16] ...trap/engines/engine.py.230: loss: 0.08505 [I 2022-10-11 19:32:16] ...optimizers/trijoint.py.151: total_norm: 12.030361 [I 2022-10-11 19:32:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 870/2304 [I 2022-10-11 19:32:25] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:21:27 [I 2022-10-11 19:32:25] ...trap/engines/engine.py.229: process: 0.92265 | load: 0.00397 [I 2022-10-11 19:32:25] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-11 19:32:25] ...optimizers/trijoint.py.151: total_norm: 15.927500 [I 2022-10-11 19:32:33] ...trap/engines/engine.py.224: train: epoch 116 | batch 880/2304 [I 2022-10-11 19:32:33] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:20:39 [I 2022-10-11 19:32:33] ...trap/engines/engine.py.229: process: 0.86212 | load: 0.00313 [I 2022-10-11 19:32:33] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-11 19:32:33] ...optimizers/trijoint.py.151: total_norm: 13.180698 [I 2022-10-11 19:32:42] ...trap/engines/engine.py.224: train: epoch 116 | batch 890/2304 [I 2022-10-11 19:32:42] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:20:21 [I 2022-10-11 19:32:42] ...trap/engines/engine.py.229: process: 0.85722 | load: 0.00316 [I 2022-10-11 19:32:42] ...trap/engines/engine.py.230: loss: 0.18782 [I 2022-10-11 19:32:42] ...optimizers/trijoint.py.151: total_norm: 17.635977 [I 2022-10-11 19:32:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 900/2304 [I 2022-10-11 19:32:51] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:20:08 [I 2022-10-11 19:32:51] ...trap/engines/engine.py.229: process: 0.85645 | load: 0.00312 [I 2022-10-11 19:32:51] ...trap/engines/engine.py.230: loss: 0.08791 [I 2022-10-11 19:32:51] ...optimizers/trijoint.py.151: total_norm: 12.598808 [I 2022-10-11 19:32:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 910/2304 [I 2022-10-11 19:32:59] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:19:55 [I 2022-10-11 19:32:59] ...trap/engines/engine.py.229: process: 0.84142 | load: 0.00313 [I 2022-10-11 19:32:59] ...trap/engines/engine.py.230: loss: 0.11525 [I 2022-10-11 19:32:59] ...optimizers/trijoint.py.151: total_norm: 16.006306 [I 2022-10-11 19:33:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 920/2304 [I 2022-10-11 19:33:08] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:19:41 [I 2022-10-11 19:33:08] ...trap/engines/engine.py.229: process: 0.86122 | load: 0.00312 [I 2022-10-11 19:33:08] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-11 19:33:08] ...optimizers/trijoint.py.151: total_norm: 15.870736 [I 2022-10-11 19:33:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 930/2304 [I 2022-10-11 19:33:16] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:19:32 [I 2022-10-11 19:33:16] ...trap/engines/engine.py.229: process: 0.85294 | load: 0.00316 [I 2022-10-11 19:33:16] ...trap/engines/engine.py.230: loss: 0.08815 [I 2022-10-11 19:33:16] ...optimizers/trijoint.py.151: total_norm: 13.494361 [I 2022-10-11 19:33:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 940/2304 [I 2022-10-11 19:33:25] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:19:35 [I 2022-10-11 19:33:25] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00317 [I 2022-10-11 19:33:25] ...trap/engines/engine.py.230: loss: 0.11242 [I 2022-10-11 19:33:25] ...optimizers/trijoint.py.151: total_norm: 17.320177 [I 2022-10-11 19:33:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 950/2304 [I 2022-10-11 19:33:34] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:19:47 [I 2022-10-11 19:33:34] ...trap/engines/engine.py.229: process: 0.88217 | load: 0.00397 [I 2022-10-11 19:33:34] ...trap/engines/engine.py.230: loss: 0.12830 [I 2022-10-11 19:33:34] ...optimizers/trijoint.py.151: total_norm: 11.979314 [I 2022-10-11 19:33:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 960/2304 [I 2022-10-11 19:33:43] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:19:55 [I 2022-10-11 19:33:43] ...trap/engines/engine.py.229: process: 0.89546 | load: 0.00394 [I 2022-10-11 19:33:43] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-11 19:33:43] ...optimizers/trijoint.py.151: total_norm: 16.285059 [I 2022-10-11 19:33:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 970/2304 [I 2022-10-11 19:33:52] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:19:47 [I 2022-10-11 19:33:52] ...trap/engines/engine.py.229: process: 0.89909 | load: 0.00395 [I 2022-10-11 19:33:52] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-11 19:33:52] ...optimizers/trijoint.py.151: total_norm: 16.159279 [I 2022-10-11 19:34:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 980/2304 [I 2022-10-11 19:34:01] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:19:39 [I 2022-10-11 19:34:01] ...trap/engines/engine.py.229: process: 0.88278 | load: 0.00395 [I 2022-10-11 19:34:01] ...trap/engines/engine.py.230: loss: 0.08241 [I 2022-10-11 19:34:01] ...optimizers/trijoint.py.151: total_norm: 16.197023 [I 2022-10-11 19:34:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 990/2304 [I 2022-10-11 19:34:09] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:19:35 [I 2022-10-11 19:34:09] ...trap/engines/engine.py.229: process: 0.90499 | load: 0.00398 [I 2022-10-11 19:34:09] ...trap/engines/engine.py.230: loss: 0.09084 [I 2022-10-11 19:34:09] ...optimizers/trijoint.py.151: total_norm: 12.261075 [I 2022-10-11 19:34:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 1000/2304 [I 2022-10-11 19:34:18] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:19:19 [I 2022-10-11 19:34:18] ...trap/engines/engine.py.229: process: 0.89546 | load: 0.00395 [I 2022-10-11 19:34:18] ...trap/engines/engine.py.230: loss: 0.21968 [I 2022-10-11 19:34:18] ...optimizers/trijoint.py.151: total_norm: 28.466646 [I 2022-10-11 19:34:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 1010/2304 [I 2022-10-11 19:34:27] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:18:39 [I 2022-10-11 19:34:27] ...trap/engines/engine.py.229: process: 0.86364 | load: 0.00312 [I 2022-10-11 19:34:27] ...trap/engines/engine.py.230: loss: 0.07720 [I 2022-10-11 19:34:27] ...optimizers/trijoint.py.151: total_norm: 15.307719 [I 2022-10-11 19:34:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 1020/2304 [I 2022-10-11 19:34:36] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:18:04 [I 2022-10-11 19:34:36] ...trap/engines/engine.py.229: process: 0.82848 | load: 0.00295 [I 2022-10-11 19:34:36] ...trap/engines/engine.py.230: loss: 0.11014 [I 2022-10-11 19:34:36] ...optimizers/trijoint.py.151: total_norm: 11.353673 [I 2022-10-11 19:34:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 1030/2304 [I 2022-10-11 19:34:44] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:18:06 [I 2022-10-11 19:34:44] ...trap/engines/engine.py.229: process: 0.85563 | load: 0.00304 [I 2022-10-11 19:34:44] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-11 19:34:44] ...optimizers/trijoint.py.151: total_norm: 13.489882 [I 2022-10-11 19:34:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 1040/2304 [I 2022-10-11 19:34:52] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:17:46 [I 2022-10-11 19:34:52] ...trap/engines/engine.py.229: process: 0.84105 | load: 0.00294 [I 2022-10-11 19:34:52] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-11 19:34:52] ...optimizers/trijoint.py.151: total_norm: 30.571001 [I 2022-10-11 19:35:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 1050/2304 [I 2022-10-11 19:35:01] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:17:40 [I 2022-10-11 19:35:01] ...trap/engines/engine.py.229: process: 0.85311 | load: 0.00295 [I 2022-10-11 19:35:01] ...trap/engines/engine.py.230: loss: 0.10293 [I 2022-10-11 19:35:01] ...optimizers/trijoint.py.151: total_norm: 10.921187 [I 2022-10-11 19:35:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 1060/2304 [I 2022-10-11 19:35:09] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:17:32 [I 2022-10-11 19:35:09] ...trap/engines/engine.py.229: process: 0.84355 | load: 0.00297 [I 2022-10-11 19:35:09] ...trap/engines/engine.py.230: loss: 0.07119 [I 2022-10-11 19:35:09] ...optimizers/trijoint.py.151: total_norm: 16.869530 [I 2022-10-11 19:35:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 1070/2304 [I 2022-10-11 19:35:18] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:17:31 [I 2022-10-11 19:35:18] ...trap/engines/engine.py.229: process: 0.86257 | load: 0.00295 [I 2022-10-11 19:35:18] ...trap/engines/engine.py.230: loss: 0.11006 [I 2022-10-11 19:35:18] ...optimizers/trijoint.py.151: total_norm: 15.392353 [I 2022-10-11 19:35:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1080/2304 [I 2022-10-11 19:35:26] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:17:17 [I 2022-10-11 19:35:26] ...trap/engines/engine.py.229: process: 0.85391 | load: 0.00297 [I 2022-10-11 19:35:26] ...trap/engines/engine.py.230: loss: 0.15601 [I 2022-10-11 19:35:26] ...optimizers/trijoint.py.151: total_norm: 21.191513 [I 2022-10-11 19:35:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 1090/2304 [I 2022-10-11 19:35:35] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:17:02 [I 2022-10-11 19:35:35] ...trap/engines/engine.py.229: process: 0.83352 | load: 0.00295 [I 2022-10-11 19:35:35] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-11 19:35:35] ...optimizers/trijoint.py.151: total_norm: 37.110977 [I 2022-10-11 19:35:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1100/2304 [I 2022-10-11 19:35:43] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:17:00 [I 2022-10-11 19:35:43] ...trap/engines/engine.py.229: process: 0.84324 | load: 0.00304 [I 2022-10-11 19:35:43] ...trap/engines/engine.py.230: loss: 0.17548 [I 2022-10-11 19:35:43] ...optimizers/trijoint.py.151: total_norm: 12.946677 [I 2022-10-11 19:35:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 1110/2304 [I 2022-10-11 19:35:52] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:16:53 [I 2022-10-11 19:35:52] ...trap/engines/engine.py.229: process: 0.85757 | load: 0.00297 [I 2022-10-11 19:35:52] ...trap/engines/engine.py.230: loss: 0.18771 [I 2022-10-11 19:35:52] ...optimizers/trijoint.py.151: total_norm: 12.621754 [I 2022-10-11 19:36:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 1120/2304 [I 2022-10-11 19:36:01] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:17:01 [I 2022-10-11 19:36:01] ...trap/engines/engine.py.229: process: 0.85827 | load: 0.00302 [I 2022-10-11 19:36:01] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-11 19:36:01] ...optimizers/trijoint.py.151: total_norm: 11.969640 [I 2022-10-11 19:36:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 1130/2304 [I 2022-10-11 19:36:09] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:16:49 [I 2022-10-11 19:36:09] ...trap/engines/engine.py.229: process: 0.86119 | load: 0.00299 [I 2022-10-11 19:36:09] ...trap/engines/engine.py.230: loss: 0.09694 [I 2022-10-11 19:36:09] ...optimizers/trijoint.py.151: total_norm: 9.200551 [I 2022-10-11 19:36:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 1140/2304 [I 2022-10-11 19:36:18] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:16:27 [I 2022-10-11 19:36:18] ...trap/engines/engine.py.229: process: 0.83597 | load: 0.00301 [I 2022-10-11 19:36:18] ...trap/engines/engine.py.230: loss: 0.10383 [I 2022-10-11 19:36:18] ...optimizers/trijoint.py.151: total_norm: 17.382290 [I 2022-10-11 19:36:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1150/2304 [I 2022-10-11 19:36:26] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:16:20 [I 2022-10-11 19:36:26] ...trap/engines/engine.py.229: process: 0.84718 | load: 0.00298 [I 2022-10-11 19:36:26] ...trap/engines/engine.py.230: loss: 0.08317 [I 2022-10-11 19:36:26] ...optimizers/trijoint.py.151: total_norm: 8.408229 [I 2022-10-11 19:36:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 1160/2304 [I 2022-10-11 19:36:35] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:16:14 [I 2022-10-11 19:36:35] ...trap/engines/engine.py.229: process: 0.84531 | load: 0.00299 [I 2022-10-11 19:36:35] ...trap/engines/engine.py.230: loss: 0.19295 [I 2022-10-11 19:36:35] ...optimizers/trijoint.py.151: total_norm: 11.797886 [I 2022-10-11 19:36:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1170/2304 [I 2022-10-11 19:36:43] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:16:10 [I 2022-10-11 19:36:43] ...trap/engines/engine.py.229: process: 0.85082 | load: 0.00295 [I 2022-10-11 19:36:43] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-11 19:36:43] ...optimizers/trijoint.py.151: total_norm: 16.173616 [I 2022-10-11 19:36:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 1180/2304 [I 2022-10-11 19:36:52] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:15:54 [I 2022-10-11 19:36:52] ...trap/engines/engine.py.229: process: 0.85601 | load: 0.00296 [I 2022-10-11 19:36:52] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-11 19:36:52] ...optimizers/trijoint.py.151: total_norm: 21.054050 [I 2022-10-11 19:37:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1190/2304 [I 2022-10-11 19:37:00] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:15:50 [I 2022-10-11 19:37:00] ...trap/engines/engine.py.229: process: 0.85508 | load: 0.00297 [I 2022-10-11 19:37:00] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-11 19:37:00] ...optimizers/trijoint.py.151: total_norm: 17.648285 [I 2022-10-11 19:37:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 1200/2304 [I 2022-10-11 19:37:09] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:15:43 [I 2022-10-11 19:37:09] ...trap/engines/engine.py.229: process: 0.85905 | load: 0.00295 [I 2022-10-11 19:37:09] ...trap/engines/engine.py.230: loss: 0.19811 [I 2022-10-11 19:37:09] ...optimizers/trijoint.py.151: total_norm: 29.572508 [I 2022-10-11 19:37:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1210/2304 [I 2022-10-11 19:37:17] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:15:31 [I 2022-10-11 19:37:17] ...trap/engines/engine.py.229: process: 0.83309 | load: 0.00296 [I 2022-10-11 19:37:17] ...trap/engines/engine.py.230: loss: 0.10895 [I 2022-10-11 19:37:17] ...optimizers/trijoint.py.151: total_norm: 16.871803 [I 2022-10-11 19:37:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1220/2304 [I 2022-10-11 19:37:26] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:15:21 [I 2022-10-11 19:37:26] ...trap/engines/engine.py.229: process: 0.86861 | load: 0.00295 [I 2022-10-11 19:37:26] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-11 19:37:26] ...optimizers/trijoint.py.151: total_norm: 12.489154 [I 2022-10-11 19:37:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 1230/2304 [I 2022-10-11 19:37:34] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:15:10 [I 2022-10-11 19:37:34] ...trap/engines/engine.py.229: process: 0.84862 | load: 0.00294 [I 2022-10-11 19:37:34] ...trap/engines/engine.py.230: loss: 0.09834 [I 2022-10-11 19:37:34] ...optimizers/trijoint.py.151: total_norm: 14.596773 [I 2022-10-11 19:37:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1240/2304 [I 2022-10-11 19:37:43] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:15:05 [I 2022-10-11 19:37:43] ...trap/engines/engine.py.229: process: 0.85858 | load: 0.00295 [I 2022-10-11 19:37:43] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-11 19:37:43] ...optimizers/trijoint.py.151: total_norm: 11.331238 [I 2022-10-11 19:37:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 1250/2304 [I 2022-10-11 19:37:51] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:15:02 [I 2022-10-11 19:37:51] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00295 [I 2022-10-11 19:37:51] ...trap/engines/engine.py.230: loss: 0.10579 [I 2022-10-11 19:37:51] ...optimizers/trijoint.py.151: total_norm: 22.460991 [I 2022-10-11 19:38:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1260/2304 [I 2022-10-11 19:38:00] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:14:53 [I 2022-10-11 19:38:00] ...trap/engines/engine.py.229: process: 0.86091 | load: 0.00296 [I 2022-10-11 19:38:00] ...trap/engines/engine.py.230: loss: 0.18926 [I 2022-10-11 19:38:00] ...optimizers/trijoint.py.151: total_norm: 45.855362 [I 2022-10-11 19:38:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 1270/2304 [I 2022-10-11 19:38:09] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:14:47 [I 2022-10-11 19:38:09] ...trap/engines/engine.py.229: process: 0.86009 | load: 0.00298 [I 2022-10-11 19:38:09] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-11 19:38:09] ...optimizers/trijoint.py.151: total_norm: 47.868786 [I 2022-10-11 19:38:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1280/2304 [I 2022-10-11 19:38:17] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:14:36 [I 2022-10-11 19:38:17] ...trap/engines/engine.py.229: process: 0.85308 | load: 0.00302 [I 2022-10-11 19:38:17] ...trap/engines/engine.py.230: loss: 0.12499 [I 2022-10-11 19:38:17] ...optimizers/trijoint.py.151: total_norm: 10.085826 [I 2022-10-11 19:38:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1290/2304 [I 2022-10-11 19:38:26] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:14:26 [I 2022-10-11 19:38:26] ...trap/engines/engine.py.229: process: 0.85263 | load: 0.00300 [I 2022-10-11 19:38:26] ...trap/engines/engine.py.230: loss: 0.12852 [I 2022-10-11 19:38:26] ...optimizers/trijoint.py.151: total_norm: 22.360811 [I 2022-10-11 19:38:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 1300/2304 [I 2022-10-11 19:38:35] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:14:48 [I 2022-10-11 19:38:35] ...trap/engines/engine.py.229: process: 0.89584 | load: 0.00403 [I 2022-10-11 19:38:35] ...trap/engines/engine.py.230: loss: 0.09313 [I 2022-10-11 19:38:35] ...optimizers/trijoint.py.151: total_norm: 15.365376 [I 2022-10-11 19:38:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 1310/2304 [I 2022-10-11 19:38:44] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:14:41 [I 2022-10-11 19:38:44] ...trap/engines/engine.py.229: process: 0.88841 | load: 0.00393 [I 2022-10-11 19:38:44] ...trap/engines/engine.py.230: loss: 0.20067 [I 2022-10-11 19:38:44] ...optimizers/trijoint.py.151: total_norm: 32.265644 [I 2022-10-11 19:38:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 1320/2304 [I 2022-10-11 19:38:52] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:14:38 [I 2022-10-11 19:38:52] ...trap/engines/engine.py.229: process: 0.90389 | load: 0.00393 [I 2022-10-11 19:38:52] ...trap/engines/engine.py.230: loss: 0.09764 [I 2022-10-11 19:38:52] ...optimizers/trijoint.py.151: total_norm: 17.378944 [I 2022-10-11 19:39:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 1330/2304 [I 2022-10-11 19:39:01] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:14:22 [I 2022-10-11 19:39:01] ...trap/engines/engine.py.229: process: 0.88165 | load: 0.00395 [I 2022-10-11 19:39:01] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-11 19:39:01] ...optimizers/trijoint.py.151: total_norm: 8.151092 [I 2022-10-11 19:39:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 1340/2304 [I 2022-10-11 19:39:10] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:14:21 [I 2022-10-11 19:39:10] ...trap/engines/engine.py.229: process: 0.89502 | load: 0.00397 [I 2022-10-11 19:39:10] ...trap/engines/engine.py.230: loss: 0.09027 [I 2022-10-11 19:39:10] ...optimizers/trijoint.py.151: total_norm: 13.159638 [I 2022-10-11 19:39:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 1350/2304 [I 2022-10-11 19:39:19] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:14:08 [I 2022-10-11 19:39:19] ...trap/engines/engine.py.229: process: 0.89728 | load: 0.00399 [I 2022-10-11 19:39:19] ...trap/engines/engine.py.230: loss: 0.12999 [I 2022-10-11 19:39:19] ...optimizers/trijoint.py.151: total_norm: 13.452485 [I 2022-10-11 19:39:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1360/2304 [I 2022-10-11 19:39:28] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:14:01 [I 2022-10-11 19:39:28] ...trap/engines/engine.py.229: process: 0.87885 | load: 0.00397 [I 2022-10-11 19:39:28] ...trap/engines/engine.py.230: loss: 0.11497 [I 2022-10-11 19:39:28] ...optimizers/trijoint.py.151: total_norm: 11.837848 [I 2022-10-11 19:39:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 1370/2304 [I 2022-10-11 19:39:37] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:13:21 [I 2022-10-11 19:39:37] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00315 [I 2022-10-11 19:39:37] ...trap/engines/engine.py.230: loss: 0.09367 [I 2022-10-11 19:39:37] ...optimizers/trijoint.py.151: total_norm: 15.110775 [I 2022-10-11 19:39:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 1380/2304 [I 2022-10-11 19:39:45] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:13:09 [I 2022-10-11 19:39:45] ...trap/engines/engine.py.229: process: 0.85845 | load: 0.00314 [I 2022-10-11 19:39:45] ...trap/engines/engine.py.230: loss: 0.10992 [I 2022-10-11 19:39:45] ...optimizers/trijoint.py.151: total_norm: 11.295341 [I 2022-10-11 19:39:54] ...trap/engines/engine.py.224: train: epoch 116 | batch 1390/2304 [I 2022-10-11 19:39:54] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:13:02 [I 2022-10-11 19:39:54] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00314 [I 2022-10-11 19:39:54] ...trap/engines/engine.py.230: loss: 0.18279 [I 2022-10-11 19:39:54] ...optimizers/trijoint.py.151: total_norm: 37.226269 [I 2022-10-11 19:40:02] ...trap/engines/engine.py.224: train: epoch 116 | batch 1400/2304 [I 2022-10-11 19:40:03] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:13:00 [I 2022-10-11 19:40:03] ...trap/engines/engine.py.229: process: 0.87052 | load: 0.00316 [I 2022-10-11 19:40:03] ...trap/engines/engine.py.230: loss: 0.08157 [I 2022-10-11 19:40:03] ...optimizers/trijoint.py.151: total_norm: 9.070821 [I 2022-10-11 19:40:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1410/2304 [I 2022-10-11 19:40:11] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:46 [I 2022-10-11 19:40:11] ...trap/engines/engine.py.229: process: 0.87039 | load: 0.00316 [I 2022-10-11 19:40:11] ...trap/engines/engine.py.230: loss: 0.08961 [I 2022-10-11 19:40:11] ...optimizers/trijoint.py.151: total_norm: 13.823965 [I 2022-10-11 19:40:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 1420/2304 [I 2022-10-11 19:40:20] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:38 [I 2022-10-11 19:40:20] ...trap/engines/engine.py.229: process: 0.85261 | load: 0.00311 [I 2022-10-11 19:40:20] ...trap/engines/engine.py.230: loss: 0.11076 [I 2022-10-11 19:40:20] ...optimizers/trijoint.py.151: total_norm: 17.368656 [I 2022-10-11 19:40:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1430/2304 [I 2022-10-11 19:40:28] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:30 [I 2022-10-11 19:40:28] ...trap/engines/engine.py.229: process: 0.85290 | load: 0.00311 [I 2022-10-11 19:40:28] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-11 19:40:28] ...optimizers/trijoint.py.151: total_norm: 21.033812 [I 2022-10-11 19:40:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 1440/2304 [I 2022-10-11 19:40:37] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:19 [I 2022-10-11 19:40:37] ...trap/engines/engine.py.229: process: 0.85629 | load: 0.00314 [I 2022-10-11 19:40:37] ...trap/engines/engine.py.230: loss: 0.10038 [I 2022-10-11 19:40:37] ...optimizers/trijoint.py.151: total_norm: 14.566207 [I 2022-10-11 19:40:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 1450/2304 [I 2022-10-11 19:40:45] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:07 [I 2022-10-11 19:40:45] ...trap/engines/engine.py.229: process: 0.84515 | load: 0.00314 [I 2022-10-11 19:40:45] ...trap/engines/engine.py.230: loss: 0.15728 [I 2022-10-11 19:40:45] ...optimizers/trijoint.py.151: total_norm: 25.267979 [I 2022-10-11 19:40:54] ...trap/engines/engine.py.224: train: epoch 116 | batch 1460/2304 [I 2022-10-11 19:40:54] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:12:06 [I 2022-10-11 19:40:54] ...trap/engines/engine.py.229: process: 0.87747 | load: 0.00317 [I 2022-10-11 19:40:54] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-11 19:40:54] ...optimizers/trijoint.py.151: total_norm: 10.586782 [I 2022-10-11 19:41:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 1470/2304 [I 2022-10-11 19:41:03] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:12:16 [I 2022-10-11 19:41:03] ...trap/engines/engine.py.229: process: 0.88198 | load: 0.00394 [I 2022-10-11 19:41:03] ...trap/engines/engine.py.230: loss: 0.16754 [I 2022-10-11 19:41:03] ...optimizers/trijoint.py.151: total_norm: 25.869549 [I 2022-10-11 19:41:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 1480/2304 [I 2022-10-11 19:41:12] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:12:09 [I 2022-10-11 19:41:12] ...trap/engines/engine.py.229: process: 0.87101 | load: 0.00396 [I 2022-10-11 19:41:12] ...trap/engines/engine.py.230: loss: 0.11801 [I 2022-10-11 19:41:12] ...optimizers/trijoint.py.151: total_norm: 8.817537 [I 2022-10-11 19:41:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1490/2304 [I 2022-10-11 19:41:21] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:12:00 [I 2022-10-11 19:41:21] ...trap/engines/engine.py.229: process: 0.88151 | load: 0.00395 [I 2022-10-11 19:41:21] ...trap/engines/engine.py.230: loss: 0.07438 [I 2022-10-11 19:41:21] ...optimizers/trijoint.py.151: total_norm: 11.768707 [I 2022-10-11 19:41:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1500/2304 [I 2022-10-11 19:41:30] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:11:51 [I 2022-10-11 19:41:30] ...trap/engines/engine.py.229: process: 0.87920 | load: 0.00397 [I 2022-10-11 19:41:30] ...trap/engines/engine.py.230: loss: 0.11137 [I 2022-10-11 19:41:30] ...optimizers/trijoint.py.151: total_norm: 11.027837 [I 2022-10-11 19:41:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1510/2304 [I 2022-10-11 19:41:38] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:11:42 [I 2022-10-11 19:41:38] ...trap/engines/engine.py.229: process: 0.89564 | load: 0.00393 [I 2022-10-11 19:41:38] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-11 19:41:38] ...optimizers/trijoint.py.151: total_norm: 13.155881 [I 2022-10-11 19:41:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 1520/2304 [I 2022-10-11 19:41:47] ...trap/engines/engine.py.228: elapsed: 0:21:58 | left: 0:11:16 [I 2022-10-11 19:41:47] ...trap/engines/engine.py.229: process: 0.86562 | load: 0.00314 [I 2022-10-11 19:41:47] ...trap/engines/engine.py.230: loss: 0.09482 [I 2022-10-11 19:41:47] ...optimizers/trijoint.py.151: total_norm: 11.867812 [I 2022-10-11 19:41:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 1530/2304 [I 2022-10-11 19:41:56] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:11:03 [I 2022-10-11 19:41:56] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00312 [I 2022-10-11 19:41:56] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-11 19:41:56] ...optimizers/trijoint.py.151: total_norm: 7.448250 [I 2022-10-11 19:42:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 1540/2304 [I 2022-10-11 19:42:04] ...trap/engines/engine.py.228: elapsed: 0:22:15 | left: 0:10:53 [I 2022-10-11 19:42:04] ...trap/engines/engine.py.229: process: 0.85843 | load: 0.00313 [I 2022-10-11 19:42:04] ...trap/engines/engine.py.230: loss: 0.11150 [I 2022-10-11 19:42:04] ...optimizers/trijoint.py.151: total_norm: 9.905645 [I 2022-10-11 19:42:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 1550/2304 [I 2022-10-11 19:42:13] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:10:41 [I 2022-10-11 19:42:13] ...trap/engines/engine.py.229: process: 0.85303 | load: 0.00317 [I 2022-10-11 19:42:13] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-11 19:42:13] ...optimizers/trijoint.py.151: total_norm: 13.405962 [I 2022-10-11 19:42:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1560/2304 [I 2022-10-11 19:42:21] ...trap/engines/engine.py.228: elapsed: 0:22:32 | left: 0:10:37 [I 2022-10-11 19:42:21] ...trap/engines/engine.py.229: process: 0.86779 | load: 0.00312 [I 2022-10-11 19:42:21] ...trap/engines/engine.py.230: loss: 0.22108 [I 2022-10-11 19:42:21] ...optimizers/trijoint.py.151: total_norm: 25.712975 [I 2022-10-11 19:42:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1570/2304 [I 2022-10-11 19:42:30] ...trap/engines/engine.py.228: elapsed: 0:22:40 | left: 0:10:21 [I 2022-10-11 19:42:30] ...trap/engines/engine.py.229: process: 0.84400 | load: 0.00297 [I 2022-10-11 19:42:30] ...trap/engines/engine.py.230: loss: 0.07946 [I 2022-10-11 19:42:30] ...optimizers/trijoint.py.151: total_norm: 10.974269 [I 2022-10-11 19:42:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1580/2304 [I 2022-10-11 19:42:38] ...trap/engines/engine.py.228: elapsed: 0:22:49 | left: 0:10:13 [I 2022-10-11 19:42:38] ...trap/engines/engine.py.229: process: 0.84815 | load: 0.00305 [I 2022-10-11 19:42:38] ...trap/engines/engine.py.230: loss: 0.12064 [I 2022-10-11 19:42:38] ...optimizers/trijoint.py.151: total_norm: 10.957632 [I 2022-10-11 19:42:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 1590/2304 [I 2022-10-11 19:42:47] ...trap/engines/engine.py.228: elapsed: 0:22:57 | left: 0:10:02 [I 2022-10-11 19:42:47] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00299 [I 2022-10-11 19:42:47] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-11 19:42:47] ...optimizers/trijoint.py.151: total_norm: 11.313102 [I 2022-10-11 19:42:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 1600/2304 [I 2022-10-11 19:42:55] ...trap/engines/engine.py.228: elapsed: 0:23:06 | left: 0:09:53 [I 2022-10-11 19:42:55] ...trap/engines/engine.py.229: process: 0.83825 | load: 0.00300 [I 2022-10-11 19:42:55] ...trap/engines/engine.py.230: loss: 0.09357 [I 2022-10-11 19:42:55] ...optimizers/trijoint.py.151: total_norm: 14.065820 [I 2022-10-11 19:43:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 1610/2304 [I 2022-10-11 19:43:04] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:46 [I 2022-10-11 19:43:04] ...trap/engines/engine.py.229: process: 0.83993 | load: 0.00298 [I 2022-10-11 19:43:04] ...trap/engines/engine.py.230: loss: 0.10952 [I 2022-10-11 19:43:04] ...optimizers/trijoint.py.151: total_norm: 14.091343 [I 2022-10-11 19:43:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 1620/2304 [I 2022-10-11 19:43:12] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:40 [I 2022-10-11 19:43:12] ...trap/engines/engine.py.229: process: 0.85009 | load: 0.00297 [I 2022-10-11 19:43:12] ...trap/engines/engine.py.230: loss: 0.11407 [I 2022-10-11 19:43:12] ...optimizers/trijoint.py.151: total_norm: 35.920475 [I 2022-10-11 19:43:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1630/2304 [I 2022-10-11 19:43:21] ...trap/engines/engine.py.228: elapsed: 0:23:31 | left: 0:09:27 [I 2022-10-11 19:43:21] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00299 [I 2022-10-11 19:43:21] ...trap/engines/engine.py.230: loss: 0.12903 [I 2022-10-11 19:43:21] ...optimizers/trijoint.py.151: total_norm: 18.090256 [I 2022-10-11 19:43:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 1640/2304 [I 2022-10-11 19:43:29] ...trap/engines/engine.py.228: elapsed: 0:23:40 | left: 0:09:21 [I 2022-10-11 19:43:29] ...trap/engines/engine.py.229: process: 0.84922 | load: 0.00300 [I 2022-10-11 19:43:29] ...trap/engines/engine.py.230: loss: 0.11646 [I 2022-10-11 19:43:29] ...optimizers/trijoint.py.151: total_norm: 10.074051 [I 2022-10-11 19:43:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1650/2304 [I 2022-10-11 19:43:38] ...trap/engines/engine.py.228: elapsed: 0:23:48 | left: 0:09:12 [I 2022-10-11 19:43:38] ...trap/engines/engine.py.229: process: 0.84124 | load: 0.00299 [I 2022-10-11 19:43:38] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-11 19:43:38] ...optimizers/trijoint.py.151: total_norm: 15.889179 [I 2022-10-11 19:43:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 1660/2304 [I 2022-10-11 19:43:46] ...trap/engines/engine.py.228: elapsed: 0:23:57 | left: 0:09:06 [I 2022-10-11 19:43:46] ...trap/engines/engine.py.229: process: 0.84733 | load: 0.00299 [I 2022-10-11 19:43:46] ...trap/engines/engine.py.230: loss: 0.09359 [I 2022-10-11 19:43:46] ...optimizers/trijoint.py.151: total_norm: 15.998496 [I 2022-10-11 19:43:54] ...trap/engines/engine.py.224: train: epoch 116 | batch 1670/2304 [I 2022-10-11 19:43:54] ...trap/engines/engine.py.228: elapsed: 0:24:05 | left: 0:08:50 [I 2022-10-11 19:43:54] ...trap/engines/engine.py.229: process: 0.83323 | load: 0.00302 [I 2022-10-11 19:43:54] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-11 19:43:54] ...optimizers/trijoint.py.151: total_norm: 15.034581 [I 2022-10-11 19:44:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 1680/2304 [I 2022-10-11 19:44:03] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:45 [I 2022-10-11 19:44:03] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00300 [I 2022-10-11 19:44:03] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-11 19:44:03] ...optimizers/trijoint.py.151: total_norm: 29.512150 [I 2022-10-11 19:44:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1690/2304 [I 2022-10-11 19:44:11] ...trap/engines/engine.py.228: elapsed: 0:24:22 | left: 0:08:36 [I 2022-10-11 19:44:11] ...trap/engines/engine.py.229: process: 0.83740 | load: 0.00299 [I 2022-10-11 19:44:11] ...trap/engines/engine.py.230: loss: 0.14280 [I 2022-10-11 19:44:11] ...optimizers/trijoint.py.151: total_norm: 13.991298 [I 2022-10-11 19:44:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 1700/2304 [I 2022-10-11 19:44:20] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:31 [I 2022-10-11 19:44:20] ...trap/engines/engine.py.229: process: 0.85504 | load: 0.00312 [I 2022-10-11 19:44:20] ...trap/engines/engine.py.230: loss: 0.12616 [I 2022-10-11 19:44:20] ...optimizers/trijoint.py.151: total_norm: 22.314116 [I 2022-10-11 19:44:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1710/2304 [I 2022-10-11 19:44:28] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:25 [I 2022-10-11 19:44:28] ...trap/engines/engine.py.229: process: 0.86149 | load: 0.00315 [I 2022-10-11 19:44:28] ...trap/engines/engine.py.230: loss: 0.15490 [I 2022-10-11 19:44:28] ...optimizers/trijoint.py.151: total_norm: 17.434578 [I 2022-10-11 19:44:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 1720/2304 [I 2022-10-11 19:44:37] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:29 [I 2022-10-11 19:44:37] ...trap/engines/engine.py.229: process: 0.89460 | load: 0.00395 [I 2022-10-11 19:44:37] ...trap/engines/engine.py.230: loss: 0.10269 [I 2022-10-11 19:44:37] ...optimizers/trijoint.py.151: total_norm: 9.150021 [I 2022-10-11 19:44:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 1730/2304 [I 2022-10-11 19:44:46] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:08 [I 2022-10-11 19:44:46] ...trap/engines/engine.py.229: process: 0.84012 | load: 0.00312 [I 2022-10-11 19:44:46] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-11 19:44:46] ...optimizers/trijoint.py.151: total_norm: 16.383865 [I 2022-10-11 19:44:54] ...trap/engines/engine.py.224: train: epoch 116 | batch 1740/2304 [I 2022-10-11 19:44:54] ...trap/engines/engine.py.228: elapsed: 0:25:05 | left: 0:08:00 [I 2022-10-11 19:44:54] ...trap/engines/engine.py.229: process: 0.84677 | load: 0.00312 [I 2022-10-11 19:44:54] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-11 19:44:54] ...optimizers/trijoint.py.151: total_norm: 17.143314 [I 2022-10-11 19:45:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 1750/2304 [I 2022-10-11 19:45:03] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:55 [I 2022-10-11 19:45:03] ...trap/engines/engine.py.229: process: 0.85524 | load: 0.00314 [I 2022-10-11 19:45:03] ...trap/engines/engine.py.230: loss: 0.09313 [I 2022-10-11 19:45:03] ...optimizers/trijoint.py.151: total_norm: 8.834836 [I 2022-10-11 19:45:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1760/2304 [I 2022-10-11 19:45:11] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:52 [I 2022-10-11 19:45:11] ...trap/engines/engine.py.229: process: 0.88110 | load: 0.00312 [I 2022-10-11 19:45:11] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-11 19:45:11] ...optimizers/trijoint.py.151: total_norm: 20.111813 [I 2022-10-11 19:45:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 1770/2304 [I 2022-10-11 19:45:20] ...trap/engines/engine.py.228: elapsed: 0:25:31 | left: 0:07:45 [I 2022-10-11 19:45:20] ...trap/engines/engine.py.229: process: 0.86271 | load: 0.00311 [I 2022-10-11 19:45:20] ...trap/engines/engine.py.230: loss: 0.09328 [I 2022-10-11 19:45:20] ...optimizers/trijoint.py.151: total_norm: 11.773687 [I 2022-10-11 19:45:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 1780/2304 [I 2022-10-11 19:45:29] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:36 [I 2022-10-11 19:45:29] ...trap/engines/engine.py.229: process: 0.87370 | load: 0.00314 [I 2022-10-11 19:45:29] ...trap/engines/engine.py.230: loss: 0.16724 [I 2022-10-11 19:45:29] ...optimizers/trijoint.py.151: total_norm: 16.775213 [I 2022-10-11 19:45:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1790/2304 [I 2022-10-11 19:45:38] ...trap/engines/engine.py.228: elapsed: 0:25:48 | left: 0:07:24 [I 2022-10-11 19:45:38] ...trap/engines/engine.py.229: process: 0.85582 | load: 0.00313 [I 2022-10-11 19:45:38] ...trap/engines/engine.py.230: loss: 0.08114 [I 2022-10-11 19:45:38] ...optimizers/trijoint.py.151: total_norm: 9.682240 [I 2022-10-11 19:45:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 1800/2304 [I 2022-10-11 19:45:46] ...trap/engines/engine.py.228: elapsed: 0:25:57 | left: 0:07:10 [I 2022-10-11 19:45:46] ...trap/engines/engine.py.229: process: 0.84652 | load: 0.00312 [I 2022-10-11 19:45:46] ...trap/engines/engine.py.230: loss: 0.08360 [I 2022-10-11 19:45:46] ...optimizers/trijoint.py.151: total_norm: 11.556756 [I 2022-10-11 19:45:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 1810/2304 [I 2022-10-11 19:45:55] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:07:07 [I 2022-10-11 19:45:55] ...trap/engines/engine.py.229: process: 0.86837 | load: 0.00316 [I 2022-10-11 19:45:55] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-11 19:45:55] ...optimizers/trijoint.py.151: total_norm: 8.413022 [I 2022-10-11 19:46:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 1820/2304 [I 2022-10-11 19:46:03] ...trap/engines/engine.py.228: elapsed: 0:26:14 | left: 0:06:59 [I 2022-10-11 19:46:03] ...trap/engines/engine.py.229: process: 0.88086 | load: 0.00316 [I 2022-10-11 19:46:03] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-11 19:46:03] ...optimizers/trijoint.py.151: total_norm: 23.636709 [I 2022-10-11 19:46:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 1830/2304 [I 2022-10-11 19:46:12] ...trap/engines/engine.py.228: elapsed: 0:26:23 | left: 0:06:51 [I 2022-10-11 19:46:12] ...trap/engines/engine.py.229: process: 0.87529 | load: 0.00313 [I 2022-10-11 19:46:12] ...trap/engines/engine.py.230: loss: 0.09562 [I 2022-10-11 19:46:12] ...optimizers/trijoint.py.151: total_norm: 15.597898 [I 2022-10-11 19:46:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1840/2304 [I 2022-10-11 19:46:21] ...trap/engines/engine.py.228: elapsed: 0:26:31 | left: 0:06:42 [I 2022-10-11 19:46:21] ...trap/engines/engine.py.229: process: 0.86759 | load: 0.00314 [I 2022-10-11 19:46:21] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-11 19:46:21] ...optimizers/trijoint.py.151: total_norm: 23.775354 [I 2022-10-11 19:46:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1850/2304 [I 2022-10-11 19:46:30] ...trap/engines/engine.py.228: elapsed: 0:26:40 | left: 0:06:32 [I 2022-10-11 19:46:30] ...trap/engines/engine.py.229: process: 0.85951 | load: 0.00312 [I 2022-10-11 19:46:30] ...trap/engines/engine.py.230: loss: 0.08512 [I 2022-10-11 19:46:30] ...optimizers/trijoint.py.151: total_norm: 11.764240 [I 2022-10-11 19:46:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1860/2304 [I 2022-10-11 19:46:38] ...trap/engines/engine.py.228: elapsed: 0:26:49 | left: 0:06:24 [I 2022-10-11 19:46:38] ...trap/engines/engine.py.229: process: 0.86900 | load: 0.00316 [I 2022-10-11 19:46:38] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-11 19:46:38] ...optimizers/trijoint.py.151: total_norm: 12.169396 [I 2022-10-11 19:46:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 1870/2304 [I 2022-10-11 19:46:47] ...trap/engines/engine.py.228: elapsed: 0:26:57 | left: 0:06:17 [I 2022-10-11 19:46:47] ...trap/engines/engine.py.229: process: 0.86999 | load: 0.00320 [I 2022-10-11 19:46:47] ...trap/engines/engine.py.230: loss: 0.15794 [I 2022-10-11 19:46:47] ...optimizers/trijoint.py.151: total_norm: 16.739325 [I 2022-10-11 19:46:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 1880/2304 [I 2022-10-11 19:46:56] ...trap/engines/engine.py.228: elapsed: 0:27:06 | left: 0:06:05 [I 2022-10-11 19:46:56] ...trap/engines/engine.py.229: process: 0.87533 | load: 0.00313 [I 2022-10-11 19:46:56] ...trap/engines/engine.py.230: loss: 0.11275 [I 2022-10-11 19:46:56] ...optimizers/trijoint.py.151: total_norm: 15.925050 [I 2022-10-11 19:47:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 1890/2304 [I 2022-10-11 19:47:04] ...trap/engines/engine.py.228: elapsed: 0:27:15 | left: 0:05:57 [I 2022-10-11 19:47:04] ...trap/engines/engine.py.229: process: 0.86440 | load: 0.00312 [I 2022-10-11 19:47:04] ...trap/engines/engine.py.230: loss: 0.11496 [I 2022-10-11 19:47:04] ...optimizers/trijoint.py.151: total_norm: 13.914303 [I 2022-10-11 19:47:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 1900/2304 [I 2022-10-11 19:47:13] ...trap/engines/engine.py.228: elapsed: 0:27:23 | left: 0:05:49 [I 2022-10-11 19:47:13] ...trap/engines/engine.py.229: process: 0.86646 | load: 0.00317 [I 2022-10-11 19:47:13] ...trap/engines/engine.py.230: loss: 0.17466 [I 2022-10-11 19:47:13] ...optimizers/trijoint.py.151: total_norm: 12.861035 [I 2022-10-11 19:47:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 1910/2304 [I 2022-10-11 19:47:22] ...trap/engines/engine.py.228: elapsed: 0:27:32 | left: 0:05:40 [I 2022-10-11 19:47:22] ...trap/engines/engine.py.229: process: 0.86233 | load: 0.00316 [I 2022-10-11 19:47:22] ...trap/engines/engine.py.230: loss: 0.09846 [I 2022-10-11 19:47:22] ...optimizers/trijoint.py.151: total_norm: 22.999357 [I 2022-10-11 19:47:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1920/2304 [I 2022-10-11 19:47:30] ...trap/engines/engine.py.228: elapsed: 0:27:41 | left: 0:05:33 [I 2022-10-11 19:47:30] ...trap/engines/engine.py.229: process: 0.86078 | load: 0.00315 [I 2022-10-11 19:47:30] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-11 19:47:30] ...optimizers/trijoint.py.151: total_norm: 25.209509 [I 2022-10-11 19:47:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 1930/2304 [I 2022-10-11 19:47:39] ...trap/engines/engine.py.228: elapsed: 0:27:49 | left: 0:05:22 [I 2022-10-11 19:47:39] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00313 [I 2022-10-11 19:47:39] ...trap/engines/engine.py.230: loss: 0.09916 [I 2022-10-11 19:47:39] ...optimizers/trijoint.py.151: total_norm: 14.546205 [I 2022-10-11 19:47:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 1940/2304 [I 2022-10-11 19:47:48] ...trap/engines/engine.py.228: elapsed: 0:27:58 | left: 0:05:14 [I 2022-10-11 19:47:48] ...trap/engines/engine.py.229: process: 0.87731 | load: 0.00312 [I 2022-10-11 19:47:48] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-11 19:47:48] ...optimizers/trijoint.py.151: total_norm: 15.636395 [I 2022-10-11 19:47:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 1950/2304 [I 2022-10-11 19:47:56] ...trap/engines/engine.py.228: elapsed: 0:28:07 | left: 0:05:05 [I 2022-10-11 19:47:56] ...trap/engines/engine.py.229: process: 0.85815 | load: 0.00311 [I 2022-10-11 19:47:56] ...trap/engines/engine.py.230: loss: 0.19519 [I 2022-10-11 19:47:56] ...optimizers/trijoint.py.151: total_norm: 23.286699 [I 2022-10-11 19:48:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 1960/2304 [I 2022-10-11 19:48:05] ...trap/engines/engine.py.228: elapsed: 0:28:15 | left: 0:04:54 [I 2022-10-11 19:48:05] ...trap/engines/engine.py.229: process: 0.85225 | load: 0.00312 [I 2022-10-11 19:48:05] ...trap/engines/engine.py.230: loss: 0.12961 [I 2022-10-11 19:48:05] ...optimizers/trijoint.py.151: total_norm: 24.672960 [I 2022-10-11 19:48:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 1970/2304 [I 2022-10-11 19:48:13] ...trap/engines/engine.py.228: elapsed: 0:28:24 | left: 0:04:45 [I 2022-10-11 19:48:13] ...trap/engines/engine.py.229: process: 0.83978 | load: 0.00306 [I 2022-10-11 19:48:13] ...trap/engines/engine.py.230: loss: 0.13495 [I 2022-10-11 19:48:13] ...optimizers/trijoint.py.151: total_norm: 13.936931 [I 2022-10-11 19:48:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 1980/2304 [I 2022-10-11 19:48:22] ...trap/engines/engine.py.228: elapsed: 0:28:32 | left: 0:04:32 [I 2022-10-11 19:48:22] ...trap/engines/engine.py.229: process: 0.83939 | load: 0.00296 [I 2022-10-11 19:48:22] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-11 19:48:22] ...optimizers/trijoint.py.151: total_norm: 53.619080 [I 2022-10-11 19:48:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1990/2304 [I 2022-10-11 19:48:30] ...trap/engines/engine.py.228: elapsed: 0:28:41 | left: 0:04:28 [I 2022-10-11 19:48:30] ...trap/engines/engine.py.229: process: 0.87076 | load: 0.00375 [I 2022-10-11 19:48:30] ...trap/engines/engine.py.230: loss: 0.06735 [I 2022-10-11 19:48:30] ...optimizers/trijoint.py.151: total_norm: 15.945851 [I 2022-10-11 19:48:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 2000/2304 [I 2022-10-11 19:48:39] ...trap/engines/engine.py.228: elapsed: 0:28:49 | left: 0:04:25 [I 2022-10-11 19:48:39] ...trap/engines/engine.py.229: process: 0.88074 | load: 0.00374 [I 2022-10-11 19:48:39] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-11 19:48:39] ...optimizers/trijoint.py.151: total_norm: 12.025202 [I 2022-10-11 19:48:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 2010/2304 [I 2022-10-11 19:48:48] ...trap/engines/engine.py.228: elapsed: 0:28:58 | left: 0:04:18 [I 2022-10-11 19:48:48] ...trap/engines/engine.py.229: process: 0.88712 | load: 0.00374 [I 2022-10-11 19:48:48] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-11 19:48:48] ...optimizers/trijoint.py.151: total_norm: 15.470582 [I 2022-10-11 19:48:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 2020/2304 [I 2022-10-11 19:48:57] ...trap/engines/engine.py.228: elapsed: 0:29:07 | left: 0:04:08 [I 2022-10-11 19:48:57] ...trap/engines/engine.py.229: process: 0.87649 | load: 0.00376 [I 2022-10-11 19:48:57] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-11 19:48:57] ...optimizers/trijoint.py.151: total_norm: 16.107178 [I 2022-10-11 19:49:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 2030/2304 [I 2022-10-11 19:49:05] ...trap/engines/engine.py.228: elapsed: 0:29:16 | left: 0:03:59 [I 2022-10-11 19:49:05] ...trap/engines/engine.py.229: process: 0.87818 | load: 0.00375 [I 2022-10-11 19:49:05] ...trap/engines/engine.py.230: loss: 0.08574 [I 2022-10-11 19:49:05] ...optimizers/trijoint.py.151: total_norm: 23.015301 [I 2022-10-11 19:49:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 2040/2304 [I 2022-10-11 19:49:14] ...trap/engines/engine.py.228: elapsed: 0:29:25 | left: 0:03:51 [I 2022-10-11 19:49:14] ...trap/engines/engine.py.229: process: 0.87991 | load: 0.00378 [I 2022-10-11 19:49:14] ...trap/engines/engine.py.230: loss: 0.13333 [I 2022-10-11 19:49:14] ...optimizers/trijoint.py.151: total_norm: 10.769785 [I 2022-10-11 19:49:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 2050/2304 [I 2022-10-11 19:49:23] ...trap/engines/engine.py.228: elapsed: 0:29:33 | left: 0:03:42 [I 2022-10-11 19:49:23] ...trap/engines/engine.py.229: process: 0.87298 | load: 0.00380 [I 2022-10-11 19:49:23] ...trap/engines/engine.py.230: loss: 0.06032 [I 2022-10-11 19:49:23] ...optimizers/trijoint.py.151: total_norm: 12.557673 [I 2022-10-11 19:49:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 2060/2304 [I 2022-10-11 19:49:32] ...trap/engines/engine.py.228: elapsed: 0:29:42 | left: 0:03:32 [I 2022-10-11 19:49:32] ...trap/engines/engine.py.229: process: 0.85838 | load: 0.00380 [I 2022-10-11 19:49:32] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-11 19:49:32] ...optimizers/trijoint.py.151: total_norm: 7.538896 [I 2022-10-11 19:49:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 2070/2304 [I 2022-10-11 19:49:40] ...trap/engines/engine.py.228: elapsed: 0:29:51 | left: 0:03:24 [I 2022-10-11 19:49:40] ...trap/engines/engine.py.229: process: 0.87027 | load: 0.00380 [I 2022-10-11 19:49:40] ...trap/engines/engine.py.230: loss: 0.22848 [I 2022-10-11 19:49:40] ...optimizers/trijoint.py.151: total_norm: 27.081934 [I 2022-10-11 19:49:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 2080/2304 [I 2022-10-11 19:49:49] ...trap/engines/engine.py.228: elapsed: 0:30:00 | left: 0:03:11 [I 2022-10-11 19:49:49] ...trap/engines/engine.py.229: process: 0.84170 | load: 0.00298 [I 2022-10-11 19:49:49] ...trap/engines/engine.py.230: loss: 0.07179 [I 2022-10-11 19:49:49] ...optimizers/trijoint.py.151: total_norm: 17.793638 [I 2022-10-11 19:49:58] ...trap/engines/engine.py.224: train: epoch 116 | batch 2090/2304 [I 2022-10-11 19:49:58] ...trap/engines/engine.py.228: elapsed: 0:30:08 | left: 0:03:02 [I 2022-10-11 19:49:58] ...trap/engines/engine.py.229: process: 0.86542 | load: 0.00301 [I 2022-10-11 19:49:58] ...trap/engines/engine.py.230: loss: 0.09302 [I 2022-10-11 19:49:58] ...optimizers/trijoint.py.151: total_norm: 17.004423 [I 2022-10-11 19:50:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 2100/2304 [I 2022-10-11 19:50:06] ...trap/engines/engine.py.228: elapsed: 0:30:17 | left: 0:02:53 [I 2022-10-11 19:50:06] ...trap/engines/engine.py.229: process: 0.85727 | load: 0.00296 [I 2022-10-11 19:50:06] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-11 19:50:06] ...optimizers/trijoint.py.151: total_norm: 14.135150 [I 2022-10-11 19:50:15] ...trap/engines/engine.py.224: train: epoch 116 | batch 2110/2304 [I 2022-10-11 19:50:15] ...trap/engines/engine.py.228: elapsed: 0:30:25 | left: 0:02:46 [I 2022-10-11 19:50:15] ...trap/engines/engine.py.229: process: 0.86828 | load: 0.00381 [I 2022-10-11 19:50:15] ...trap/engines/engine.py.230: loss: 0.12405 [I 2022-10-11 19:50:15] ...optimizers/trijoint.py.151: total_norm: 19.288595 [I 2022-10-11 19:50:24] ...trap/engines/engine.py.224: train: epoch 116 | batch 2120/2304 [I 2022-10-11 19:50:24] ...trap/engines/engine.py.228: elapsed: 0:30:34 | left: 0:02:41 [I 2022-10-11 19:50:24] ...trap/engines/engine.py.229: process: 0.87625 | load: 0.00383 [I 2022-10-11 19:50:24] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-11 19:50:24] ...optimizers/trijoint.py.151: total_norm: 16.483093 [I 2022-10-11 19:50:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 2130/2304 [I 2022-10-11 19:50:32] ...trap/engines/engine.py.228: elapsed: 0:30:43 | left: 0:02:31 [I 2022-10-11 19:50:32] ...trap/engines/engine.py.229: process: 0.86828 | load: 0.00380 [I 2022-10-11 19:50:32] ...trap/engines/engine.py.230: loss: 0.09666 [I 2022-10-11 19:50:32] ...optimizers/trijoint.py.151: total_norm: 14.479723 [I 2022-10-11 19:50:41] ...trap/engines/engine.py.224: train: epoch 116 | batch 2140/2304 [I 2022-10-11 19:50:41] ...trap/engines/engine.py.228: elapsed: 0:30:52 | left: 0:02:23 [I 2022-10-11 19:50:41] ...trap/engines/engine.py.229: process: 0.87548 | load: 0.00379 [I 2022-10-11 19:50:41] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-11 19:50:41] ...optimizers/trijoint.py.151: total_norm: 15.750662 [I 2022-10-11 19:50:50] ...trap/engines/engine.py.224: train: epoch 116 | batch 2150/2304 [I 2022-10-11 19:50:50] ...trap/engines/engine.py.228: elapsed: 0:31:00 | left: 0:02:11 [I 2022-10-11 19:50:50] ...trap/engines/engine.py.229: process: 0.85105 | load: 0.00304 [I 2022-10-11 19:50:50] ...trap/engines/engine.py.230: loss: 0.09144 [I 2022-10-11 19:50:50] ...optimizers/trijoint.py.151: total_norm: 10.355466 [I 2022-10-11 19:50:58] ...trap/engines/engine.py.224: train: epoch 116 | batch 2160/2304 [I 2022-10-11 19:50:58] ...trap/engines/engine.py.228: elapsed: 0:31:09 | left: 0:02:00 [I 2022-10-11 19:50:58] ...trap/engines/engine.py.229: process: 0.83094 | load: 0.00301 [I 2022-10-11 19:50:58] ...trap/engines/engine.py.230: loss: 0.11642 [I 2022-10-11 19:50:58] ...optimizers/trijoint.py.151: total_norm: 12.086278 [I 2022-10-11 19:51:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 2170/2304 [I 2022-10-11 19:51:07] ...trap/engines/engine.py.228: elapsed: 0:31:17 | left: 0:01:52 [I 2022-10-11 19:51:07] ...trap/engines/engine.py.229: process: 0.83327 | load: 0.00300 [I 2022-10-11 19:51:07] ...trap/engines/engine.py.230: loss: 0.12129 [I 2022-10-11 19:51:07] ...optimizers/trijoint.py.151: total_norm: 10.719635 [I 2022-10-11 19:51:15] ...trap/engines/engine.py.224: train: epoch 116 | batch 2180/2304 [I 2022-10-11 19:51:15] ...trap/engines/engine.py.228: elapsed: 0:31:25 | left: 0:01:44 [I 2022-10-11 19:51:15] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00306 [I 2022-10-11 19:51:15] ...trap/engines/engine.py.230: loss: 0.12149 [I 2022-10-11 19:51:15] ...optimizers/trijoint.py.151: total_norm: 8.448307 [I 2022-10-11 19:51:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 2190/2304 [I 2022-10-11 19:51:23] ...trap/engines/engine.py.228: elapsed: 0:31:34 | left: 0:01:36 [I 2022-10-11 19:51:23] ...trap/engines/engine.py.229: process: 0.84578 | load: 0.00296 [I 2022-10-11 19:51:23] ...trap/engines/engine.py.230: loss: 0.08297 [I 2022-10-11 19:51:23] ...optimizers/trijoint.py.151: total_norm: 11.377003 [I 2022-10-11 19:51:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 2200/2304 [I 2022-10-11 19:51:32] ...trap/engines/engine.py.228: elapsed: 0:31:42 | left: 0:01:27 [I 2022-10-11 19:51:32] ...trap/engines/engine.py.229: process: 0.84796 | load: 0.00298 [I 2022-10-11 19:51:32] ...trap/engines/engine.py.230: loss: 0.10618 [I 2022-10-11 19:51:32] ...optimizers/trijoint.py.151: total_norm: 12.424294 [I 2022-10-11 19:51:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 2210/2304 [I 2022-10-11 19:51:40] ...trap/engines/engine.py.228: elapsed: 0:31:51 | left: 0:01:18 [I 2022-10-11 19:51:40] ...trap/engines/engine.py.229: process: 0.84233 | load: 0.00300 [I 2022-10-11 19:51:40] ...trap/engines/engine.py.230: loss: 0.08989 [I 2022-10-11 19:51:40] ...optimizers/trijoint.py.151: total_norm: 7.911449 [I 2022-10-11 19:51:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 2220/2304 [I 2022-10-11 19:51:49] ...trap/engines/engine.py.228: elapsed: 0:31:59 | left: 0:01:11 [I 2022-10-11 19:51:49] ...trap/engines/engine.py.229: process: 0.85302 | load: 0.00300 [I 2022-10-11 19:51:49] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-11 19:51:49] ...optimizers/trijoint.py.151: total_norm: 12.477840 [I 2022-10-11 19:52:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 2230/2304 [I 2022-10-11 19:52:16] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:01:50 [I 2022-10-11 19:52:16] ...trap/engines/engine.py.229: process: 0.85812 | load: 0.00303 [I 2022-10-11 19:52:16] ...trap/engines/engine.py.230: loss: 0.16849 [I 2022-10-11 19:52:16] ...optimizers/trijoint.py.151: total_norm: 41.124619 [I 2022-10-11 19:52:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 2240/2304 [I 2022-10-11 19:52:25] ...trap/engines/engine.py.228: elapsed: 0:32:35 | left: 0:00:58 [I 2022-10-11 19:52:25] ...trap/engines/engine.py.229: process: 0.86732 | load: 0.00296 [I 2022-10-11 19:52:25] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-11 19:52:25] ...optimizers/trijoint.py.151: total_norm: 32.399689 [I 2022-10-11 19:52:33] ...trap/engines/engine.py.224: train: epoch 116 | batch 2250/2304 [I 2022-10-11 19:52:33] ...trap/engines/engine.py.228: elapsed: 0:32:44 | left: 0:00:46 [I 2022-10-11 19:52:33] ...trap/engines/engine.py.229: process: 0.83969 | load: 0.00304 [I 2022-10-11 19:52:33] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-11 19:52:33] ...optimizers/trijoint.py.151: total_norm: 13.938296 [I 2022-10-11 19:52:42] ...trap/engines/engine.py.224: train: epoch 116 | batch 2260/2304 [I 2022-10-11 19:52:42] ...trap/engines/engine.py.228: elapsed: 0:32:52 | left: 0:00:37 [I 2022-10-11 19:52:42] ...trap/engines/engine.py.229: process: 0.85542 | load: 0.00304 [I 2022-10-11 19:52:42] ...trap/engines/engine.py.230: loss: 0.10435 [I 2022-10-11 19:52:42] ...optimizers/trijoint.py.151: total_norm: 16.708557 [I 2022-10-11 19:52:50] ...trap/engines/engine.py.224: train: epoch 116 | batch 2270/2304 [I 2022-10-11 19:52:50] ...trap/engines/engine.py.228: elapsed: 0:33:01 | left: 0:00:29 [I 2022-10-11 19:52:50] ...trap/engines/engine.py.229: process: 0.87444 | load: 0.00317 [I 2022-10-11 19:52:50] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-11 19:52:50] ...optimizers/trijoint.py.151: total_norm: 16.028795 [I 2022-10-11 19:52:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 2280/2304 [I 2022-10-11 19:52:59] ...trap/engines/engine.py.228: elapsed: 0:33:10 | left: 0:00:20 [I 2022-10-11 19:52:59] ...trap/engines/engine.py.229: process: 0.89131 | load: 0.00395 [I 2022-10-11 19:52:59] ...trap/engines/engine.py.230: loss: 0.07663 [I 2022-10-11 19:52:59] ...optimizers/trijoint.py.151: total_norm: 14.910174 [I 2022-10-11 19:53:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 2290/2304 [I 2022-10-11 19:53:08] ...trap/engines/engine.py.228: elapsed: 0:33:18 | left: 0:00:12 [I 2022-10-11 19:53:08] ...trap/engines/engine.py.229: process: 0.86046 | load: 0.00392 [I 2022-10-11 19:53:08] ...trap/engines/engine.py.230: loss: 0.09174 [I 2022-10-11 19:53:08] ...optimizers/trijoint.py.151: total_norm: 10.849456 [I 2022-10-11 19:53:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 2300/2304 [I 2022-10-11 19:53:16] ...trap/engines/engine.py.228: elapsed: 0:33:27 | left: 0:00:03 [I 2022-10-11 19:53:16] ...trap/engines/engine.py.229: process: 0.84205 | load: 0.00020 [I 2022-10-11 19:53:16] ...trap/engines/engine.py.230: loss: 0.10347 [I 2022-10-11 19:53:16] ...optimizers/trijoint.py.151: total_norm: 12.754120 [I 2022-10-11 19:53:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 2304/2304 [I 2022-10-11 19:53:20] ...trap/engines/engine.py.228: elapsed: 0:33:30 | left: 0:00:00 [I 2022-10-11 19:53:20] ...trap/engines/engine.py.229: process: 0.83316 | load: 0.00020 [I 2022-10-11 19:53:20] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-11 19:53:20] ...optimizers/trijoint.py.151: total_norm: 10.988561 [S 2022-10-11 19:53:21] ...trap/engines/engine.py.129: train_epoch.epoch: 116 [S 2022-10-11 19:53:21] ...trap/engines/engine.py.129: train_epoch.loss: 0.11726118940496393 [I 2022-10-11 19:53:21] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 19:53:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 116 [I 2022-10-11 19:53:24] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 19:53:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 19:53:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 19:53:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 19:53:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 19:53:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/510 [I 2022-10-11 19:53:30] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:57:01 [I 2022-10-11 19:53:30] ...trap/engines/engine.py.326: process: 6.70933 | load: 4.56875 [I 2022-10-11 19:53:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/510 [I 2022-10-11 19:53:36] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:10:35 [I 2022-10-11 19:53:36] ...trap/engines/engine.py.326: process: 0.58953 | load: 0.00020 [I 2022-10-11 19:53:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/510 [I 2022-10-11 19:53:43] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:05:58 [I 2022-10-11 19:53:43] ...trap/engines/engine.py.326: process: 0.66916 | load: 0.00020 [I 2022-10-11 19:53:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/510 [I 2022-10-11 19:53:49] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:05:21 [I 2022-10-11 19:53:49] ...trap/engines/engine.py.326: process: 0.64491 | load: 0.00020 [I 2022-10-11 19:53:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/510 [I 2022-10-11 19:53:56] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:05:22 [I 2022-10-11 19:53:56] ...trap/engines/engine.py.326: process: 0.60561 | load: 0.00020 [I 2022-10-11 19:54:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/510 [I 2022-10-11 19:54:01] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:04:03 [I 2022-10-11 19:54:01] ...trap/engines/engine.py.326: process: 0.47264 | load: 0.00019 [I 2022-10-11 19:54:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/510 [I 2022-10-11 19:54:07] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:53 [I 2022-10-11 19:54:07] ...trap/engines/engine.py.326: process: 0.61278 | load: 0.00019 [I 2022-10-11 19:54:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/510 [I 2022-10-11 19:54:12] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:04:04 [I 2022-10-11 19:54:12] ...trap/engines/engine.py.326: process: 0.68042 | load: 0.00020 [I 2022-10-11 19:54:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/510 [I 2022-10-11 19:54:18] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:04:00 [I 2022-10-11 19:54:18] ...trap/engines/engine.py.326: process: 0.62541 | load: 0.00019 [I 2022-10-11 19:54:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/510 [I 2022-10-11 19:54:22] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:03:28 [I 2022-10-11 19:54:22] ...trap/engines/engine.py.326: process: 0.49455 | load: 0.00020 [I 2022-10-11 19:54:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/510 [I 2022-10-11 19:54:27] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:03:04 [I 2022-10-11 19:54:27] ...trap/engines/engine.py.326: process: 0.31177 | load: 0.00019 [I 2022-10-11 19:54:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/510 [I 2022-10-11 19:54:30] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:02:13 [I 2022-10-11 19:54:30] ...trap/engines/engine.py.326: process: 0.31312 | load: 0.00021 [I 2022-10-11 19:54:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/510 [I 2022-10-11 19:54:34] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:02:05 [I 2022-10-11 19:54:34] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00020 [I 2022-10-11 19:54:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/510 [I 2022-10-11 19:54:37] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:02:01 [I 2022-10-11 19:54:37] ...trap/engines/engine.py.326: process: 0.31990 | load: 0.00021 [I 2022-10-11 19:54:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/510 [I 2022-10-11 19:54:40] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:58 [I 2022-10-11 19:54:40] ...trap/engines/engine.py.326: process: 0.31984 | load: 0.00025 [I 2022-10-11 19:54:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/510 [I 2022-10-11 19:54:43] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:55 [I 2022-10-11 19:54:43] ...trap/engines/engine.py.326: process: 0.32160 | load: 0.00027 [I 2022-10-11 19:54:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/510 [I 2022-10-11 19:54:46] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:52 [I 2022-10-11 19:54:46] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00023 [I 2022-10-11 19:54:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/510 [I 2022-10-11 19:54:50] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:49 [I 2022-10-11 19:54:50] ...trap/engines/engine.py.326: process: 0.32470 | load: 0.00021 [I 2022-10-11 19:54:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/510 [I 2022-10-11 19:54:53] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:45 [I 2022-10-11 19:54:53] ...trap/engines/engine.py.326: process: 0.31961 | load: 0.00025 [I 2022-10-11 19:54:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/510 [I 2022-10-11 19:54:56] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:43 [I 2022-10-11 19:54:56] ...trap/engines/engine.py.326: process: 0.32484 | load: 0.00022 [I 2022-10-11 19:54:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/510 [I 2022-10-11 19:54:59] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:39 [I 2022-10-11 19:54:59] ...trap/engines/engine.py.326: process: 0.32477 | load: 0.00021 [I 2022-10-11 19:55:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/510 [I 2022-10-11 19:55:02] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:36 [I 2022-10-11 19:55:02] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-11 19:55:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/510 [I 2022-10-11 19:55:06] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:33 [I 2022-10-11 19:55:06] ...trap/engines/engine.py.326: process: 0.32002 | load: 0.00021 [I 2022-10-11 19:55:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/510 [I 2022-10-11 19:55:09] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:30 [I 2022-10-11 19:55:09] ...trap/engines/engine.py.326: process: 0.31934 | load: 0.00021 [I 2022-10-11 19:55:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/510 [I 2022-10-11 19:55:12] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:01:26 [I 2022-10-11 19:55:12] ...trap/engines/engine.py.326: process: 0.32467 | load: 0.00023 [I 2022-10-11 19:55:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/510 [I 2022-10-11 19:55:15] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:24 [I 2022-10-11 19:55:15] ...trap/engines/engine.py.326: process: 0.32155 | load: 0.00021 [I 2022-10-11 19:55:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/510 [I 2022-10-11 19:55:19] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:20 [I 2022-10-11 19:55:19] ...trap/engines/engine.py.326: process: 0.32413 | load: 0.00022 [I 2022-10-11 19:55:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/510 [I 2022-10-11 19:55:22] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:17 [I 2022-10-11 19:55:22] ...trap/engines/engine.py.326: process: 0.32352 | load: 0.00021 [I 2022-10-11 19:55:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/510 [I 2022-10-11 19:55:25] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:13 [I 2022-10-11 19:55:25] ...trap/engines/engine.py.326: process: 0.31788 | load: 0.00021 [I 2022-10-11 19:55:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/510 [I 2022-10-11 19:55:28] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:01:10 [I 2022-10-11 19:55:28] ...trap/engines/engine.py.326: process: 0.32005 | load: 0.00023 [I 2022-10-11 19:55:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/510 [I 2022-10-11 19:55:32] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:01:07 [I 2022-10-11 19:55:32] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00021 [I 2022-10-11 19:55:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/510 [I 2022-10-11 19:55:35] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:01:04 [I 2022-10-11 19:55:35] ...trap/engines/engine.py.326: process: 0.32374 | load: 0.00021 [I 2022-10-11 19:55:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/510 [I 2022-10-11 19:55:38] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:01:00 [I 2022-10-11 19:55:38] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00023 [I 2022-10-11 19:55:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/510 [I 2022-10-11 19:55:41] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:58 [I 2022-10-11 19:55:41] ...trap/engines/engine.py.326: process: 0.32811 | load: 0.00022 [I 2022-10-11 19:55:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/510 [I 2022-10-11 19:55:44] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:54 [I 2022-10-11 19:55:44] ...trap/engines/engine.py.326: process: 0.32313 | load: 0.00021 [I 2022-10-11 19:55:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/510 [I 2022-10-11 19:55:48] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:51 [I 2022-10-11 19:55:48] ...trap/engines/engine.py.326: process: 0.32046 | load: 0.00023 [I 2022-10-11 19:55:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/510 [I 2022-10-11 19:55:51] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:48 [I 2022-10-11 19:55:51] ...trap/engines/engine.py.326: process: 0.32524 | load: 0.00022 [I 2022-10-11 19:55:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/510 [I 2022-10-11 19:55:54] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:45 [I 2022-10-11 19:55:54] ...trap/engines/engine.py.326: process: 0.31701 | load: 0.00022 [I 2022-10-11 19:55:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/510 [I 2022-10-11 19:55:57] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:42 [I 2022-10-11 19:55:57] ...trap/engines/engine.py.326: process: 0.32798 | load: 0.00022 [I 2022-10-11 19:56:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/510 [I 2022-10-11 19:56:01] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:38 [I 2022-10-11 19:56:01] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00022 [I 2022-10-11 19:56:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/510 [I 2022-10-11 19:56:04] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:35 [I 2022-10-11 19:56:04] ...trap/engines/engine.py.326: process: 0.32117 | load: 0.00021 [I 2022-10-11 19:56:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/510 [I 2022-10-11 19:56:07] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:32 [I 2022-10-11 19:56:07] ...trap/engines/engine.py.326: process: 0.32565 | load: 0.00022 [I 2022-10-11 19:56:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/510 [I 2022-10-11 19:56:10] ...trap/engines/engine.py.325: elapsed: 0:02:47 | left: 0:00:29 [I 2022-10-11 19:56:10] ...trap/engines/engine.py.326: process: 0.32459 | load: 0.00021 [I 2022-10-11 19:56:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/510 [I 2022-10-11 19:56:13] ...trap/engines/engine.py.325: elapsed: 0:02:50 | left: 0:00:25 [I 2022-10-11 19:56:13] ...trap/engines/engine.py.326: process: 0.31973 | load: 0.00022 [I 2022-10-11 19:56:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/510 [I 2022-10-11 19:56:17] ...trap/engines/engine.py.325: elapsed: 0:02:53 | left: 0:00:22 [I 2022-10-11 19:56:17] ...trap/engines/engine.py.326: process: 0.31714 | load: 0.00021 [I 2022-10-11 19:56:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/510 [I 2022-10-11 19:56:20] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:19 [I 2022-10-11 19:56:20] ...trap/engines/engine.py.326: process: 0.31685 | load: 0.00023 [I 2022-10-11 19:56:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/510 [I 2022-10-11 19:56:23] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:16 [I 2022-10-11 19:56:23] ...trap/engines/engine.py.326: process: 0.33140 | load: 0.00021 [I 2022-10-11 19:56:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/510 [I 2022-10-11 19:56:26] ...trap/engines/engine.py.325: elapsed: 0:03:03 | left: 0:00:12 [I 2022-10-11 19:56:26] ...trap/engines/engine.py.326: process: 0.32126 | load: 0.00022 [I 2022-10-11 19:56:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/510 [I 2022-10-11 19:56:30] ...trap/engines/engine.py.325: elapsed: 0:03:06 | left: 0:00:09 [I 2022-10-11 19:56:30] ...trap/engines/engine.py.326: process: 0.32199 | load: 0.00021 [I 2022-10-11 19:56:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/510 [I 2022-10-11 19:56:33] ...trap/engines/engine.py.325: elapsed: 0:03:09 | left: 0:00:06 [I 2022-10-11 19:56:33] ...trap/engines/engine.py.326: process: 0.32196 | load: 0.00021 [I 2022-10-11 19:56:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/510 [I 2022-10-11 19:56:36] ...trap/engines/engine.py.325: elapsed: 0:03:12 | left: 0:00:03 [I 2022-10-11 19:56:36] ...trap/engines/engine.py.326: process: 0.31861 | load: 0.00022 [I 2022-10-11 19:56:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/510 [I 2022-10-11 19:56:39] ...trap/engines/engine.py.325: elapsed: 0:03:16 | left: 0:00:00 [I 2022-10-11 19:56:39] ...trap/engines/engine.py.326: process: 0.31715 | load: 0.00017 [S 2022-10-11 19:56:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 116 [I 2022-10-11 19:56:40] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746300 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909600 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935400 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.743900 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910100 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934700 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008331 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006815 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005257 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009638 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006107 [S 2022-10-11 19:57:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005041 [I 2022-10-11 19:57:24] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 19:57:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 19:57:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 19:57:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 19:57:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 19:57:29] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 19:57:33] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 19:57:37] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 19:57:37] ...trap/engines/engine.py.163: Training model on trainset for epoch 117 [I 2022-10-11 19:57:37] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 19:57:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 0/2304 [I 2022-10-11 19:57:40] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:05:43 [I 2022-10-11 19:57:40] ...trap/engines/engine.py.229: process: 3.27425 | load: 2.33604 [I 2022-10-11 19:57:40] ...trap/engines/engine.py.230: loss: 0.10952 [I 2022-10-11 19:57:40] ...optimizers/trijoint.py.151: total_norm: 15.878433 [I 2022-10-11 19:57:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 10/2304 [I 2022-10-11 19:57:48] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:46 [I 2022-10-11 19:57:48] ...trap/engines/engine.py.229: process: 0.87250 | load: 0.00655 [I 2022-10-11 19:57:48] ...trap/engines/engine.py.230: loss: 0.11197 [I 2022-10-11 19:57:48] ...optimizers/trijoint.py.151: total_norm: 22.655340 [I 2022-10-11 19:57:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 20/2304 [I 2022-10-11 19:57:57] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:43 [I 2022-10-11 19:57:57] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00498 [I 2022-10-11 19:57:57] ...trap/engines/engine.py.230: loss: 0.11032 [I 2022-10-11 19:57:57] ...optimizers/trijoint.py.151: total_norm: 18.557281 [I 2022-10-11 19:58:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 30/2304 [I 2022-10-11 19:58:06] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:17 [I 2022-10-11 19:58:06] ...trap/engines/engine.py.229: process: 0.84294 | load: 0.00423 [I 2022-10-11 19:58:06] ...trap/engines/engine.py.230: loss: 0.12069 [I 2022-10-11 19:58:06] ...optimizers/trijoint.py.151: total_norm: 25.445976 [I 2022-10-11 19:58:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 40/2304 [I 2022-10-11 19:58:14] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:02 [I 2022-10-11 19:58:14] ...trap/engines/engine.py.229: process: 0.85539 | load: 0.00412 [I 2022-10-11 19:58:14] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-11 19:58:14] ...optimizers/trijoint.py.151: total_norm: 15.410963 [I 2022-10-11 19:58:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 50/2304 [I 2022-10-11 19:58:23] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:10 [I 2022-10-11 19:58:23] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00381 [I 2022-10-11 19:58:23] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-11 19:58:23] ...optimizers/trijoint.py.151: total_norm: 13.413949 [I 2022-10-11 19:58:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 60/2304 [I 2022-10-11 19:58:31] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:53 [I 2022-10-11 19:58:31] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00346 [I 2022-10-11 19:58:31] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-11 19:58:31] ...optimizers/trijoint.py.151: total_norm: 23.268724 [I 2022-10-11 19:58:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 70/2304 [I 2022-10-11 19:58:40] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:40 [I 2022-10-11 19:58:40] ...trap/engines/engine.py.229: process: 0.83888 | load: 0.00325 [I 2022-10-11 19:58:40] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-11 19:58:40] ...optimizers/trijoint.py.151: total_norm: 12.053656 [I 2022-10-11 19:58:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 80/2304 [I 2022-10-11 19:58:48] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:38 [I 2022-10-11 19:58:48] ...trap/engines/engine.py.229: process: 0.85301 | load: 0.00373 [I 2022-10-11 19:58:48] ...trap/engines/engine.py.230: loss: 0.13835 [I 2022-10-11 19:58:48] ...optimizers/trijoint.py.151: total_norm: 22.221638 [I 2022-10-11 19:58:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 90/2304 [I 2022-10-11 19:58:57] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:24 [I 2022-10-11 19:58:57] ...trap/engines/engine.py.229: process: 0.84799 | load: 0.00336 [I 2022-10-11 19:58:57] ...trap/engines/engine.py.230: loss: 0.09133 [I 2022-10-11 19:58:57] ...optimizers/trijoint.py.151: total_norm: 23.760334 [I 2022-10-11 19:59:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 100/2304 [I 2022-10-11 19:59:05] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:48 [I 2022-10-11 19:59:05] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00351 [I 2022-10-11 19:59:05] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-11 19:59:05] ...optimizers/trijoint.py.151: total_norm: 13.179570 [I 2022-10-11 19:59:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 110/2304 [I 2022-10-11 19:59:14] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:12 [I 2022-10-11 19:59:14] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00368 [I 2022-10-11 19:59:14] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-11 19:59:14] ...optimizers/trijoint.py.151: total_norm: 26.896339 [I 2022-10-11 19:59:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 120/2304 [I 2022-10-11 19:59:23] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:31:10 [I 2022-10-11 19:59:23] ...trap/engines/engine.py.229: process: 0.85135 | load: 0.00327 [I 2022-10-11 19:59:23] ...trap/engines/engine.py.230: loss: 0.09728 [I 2022-10-11 19:59:23] ...optimizers/trijoint.py.151: total_norm: 16.283077 [I 2022-10-11 19:59:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 130/2304 [I 2022-10-11 19:59:31] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:00 [I 2022-10-11 19:59:31] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00313 [I 2022-10-11 19:59:31] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-11 19:59:31] ...optimizers/trijoint.py.151: total_norm: 9.927496 [I 2022-10-11 19:59:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 140/2304 [I 2022-10-11 19:59:40] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:30:40 [I 2022-10-11 19:59:40] ...trap/engines/engine.py.229: process: 0.85720 | load: 0.00348 [I 2022-10-11 19:59:40] ...trap/engines/engine.py.230: loss: 0.07873 [I 2022-10-11 19:59:40] ...optimizers/trijoint.py.151: total_norm: 31.459980 [I 2022-10-11 19:59:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 150/2304 [I 2022-10-11 19:59:48] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:30:44 [I 2022-10-11 19:59:48] ...trap/engines/engine.py.229: process: 0.85493 | load: 0.00329 [I 2022-10-11 19:59:48] ...trap/engines/engine.py.230: loss: 0.23932 [I 2022-10-11 19:59:48] ...optimizers/trijoint.py.151: total_norm: 20.480471 [I 2022-10-11 19:59:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 160/2304 [I 2022-10-11 19:59:57] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:24 [I 2022-10-11 19:59:57] ...trap/engines/engine.py.229: process: 0.84868 | load: 0.00327 [I 2022-10-11 19:59:57] ...trap/engines/engine.py.230: loss: 0.11508 [I 2022-10-11 19:59:57] ...optimizers/trijoint.py.151: total_norm: 14.156235 [I 2022-10-11 20:00:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 170/2304 [I 2022-10-11 20:00:05] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:30:26 [I 2022-10-11 20:00:05] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00327 [I 2022-10-11 20:00:05] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-11 20:00:05] ...optimizers/trijoint.py.151: total_norm: 10.446968 [I 2022-10-11 20:00:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 180/2304 [I 2022-10-11 20:00:14] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:30:17 [I 2022-10-11 20:00:14] ...trap/engines/engine.py.229: process: 0.85955 | load: 0.00313 [I 2022-10-11 20:00:14] ...trap/engines/engine.py.230: loss: 0.19501 [I 2022-10-11 20:00:14] ...optimizers/trijoint.py.151: total_norm: 38.500816 [I 2022-10-11 20:00:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 190/2304 [I 2022-10-11 20:00:23] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:45 [I 2022-10-11 20:00:23] ...trap/engines/engine.py.229: process: 0.86267 | load: 0.00397 [I 2022-10-11 20:00:23] ...trap/engines/engine.py.230: loss: 0.10948 [I 2022-10-11 20:00:23] ...optimizers/trijoint.py.151: total_norm: 17.446321 [I 2022-10-11 20:00:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 200/2304 [I 2022-10-11 20:00:31] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:30:08 [I 2022-10-11 20:00:31] ...trap/engines/engine.py.229: process: 0.86415 | load: 0.00323 [I 2022-10-11 20:00:31] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-11 20:00:31] ...optimizers/trijoint.py.151: total_norm: 15.159813 [I 2022-10-11 20:00:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 210/2304 [I 2022-10-11 20:00:40] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:29:48 [I 2022-10-11 20:00:40] ...trap/engines/engine.py.229: process: 0.85223 | load: 0.00316 [I 2022-10-11 20:00:40] ...trap/engines/engine.py.230: loss: 0.09192 [I 2022-10-11 20:00:40] ...optimizers/trijoint.py.151: total_norm: 13.271845 [I 2022-10-11 20:00:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 220/2304 [I 2022-10-11 20:00:48] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:29:36 [I 2022-10-11 20:00:48] ...trap/engines/engine.py.229: process: 0.83745 | load: 0.00337 [I 2022-10-11 20:00:48] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-11 20:00:48] ...optimizers/trijoint.py.151: total_norm: 27.381514 [I 2022-10-11 20:00:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 230/2304 [I 2022-10-11 20:00:57] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:29:25 [I 2022-10-11 20:00:57] ...trap/engines/engine.py.229: process: 0.84784 | load: 0.00336 [I 2022-10-11 20:00:57] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-11 20:00:57] ...optimizers/trijoint.py.151: total_norm: 11.997525 [I 2022-10-11 20:01:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 240/2304 [I 2022-10-11 20:01:05] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:29:29 [I 2022-10-11 20:01:05] ...trap/engines/engine.py.229: process: 0.86849 | load: 0.00311 [I 2022-10-11 20:01:05] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-11 20:01:05] ...optimizers/trijoint.py.151: total_norm: 10.244221 [I 2022-10-11 20:01:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 250/2304 [I 2022-10-11 20:01:14] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:29:37 [I 2022-10-11 20:01:14] ...trap/engines/engine.py.229: process: 0.87657 | load: 0.00344 [I 2022-10-11 20:01:14] ...trap/engines/engine.py.230: loss: 0.10550 [I 2022-10-11 20:01:14] ...optimizers/trijoint.py.151: total_norm: 31.476765 [I 2022-10-11 20:01:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 260/2304 [I 2022-10-11 20:01:23] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:28:55 [I 2022-10-11 20:01:23] ...trap/engines/engine.py.229: process: 0.83539 | load: 0.00351 [I 2022-10-11 20:01:23] ...trap/engines/engine.py.230: loss: 0.07815 [I 2022-10-11 20:01:23] ...optimizers/trijoint.py.151: total_norm: 17.844124 [I 2022-10-11 20:01:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 270/2304 [I 2022-10-11 20:01:31] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:28:58 [I 2022-10-11 20:01:31] ...trap/engines/engine.py.229: process: 0.84135 | load: 0.00312 [I 2022-10-11 20:01:31] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-11 20:01:31] ...optimizers/trijoint.py.151: total_norm: 12.124274 [I 2022-10-11 20:01:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 280/2304 [I 2022-10-11 20:01:40] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:28:47 [I 2022-10-11 20:01:40] ...trap/engines/engine.py.229: process: 0.85685 | load: 0.00329 [I 2022-10-11 20:01:40] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-11 20:01:40] ...optimizers/trijoint.py.151: total_norm: 16.441837 [I 2022-10-11 20:01:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 290/2304 [I 2022-10-11 20:01:48] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:28:42 [I 2022-10-11 20:01:48] ...trap/engines/engine.py.229: process: 0.86054 | load: 0.00333 [I 2022-10-11 20:01:48] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-11 20:01:48] ...optimizers/trijoint.py.151: total_norm: 8.818988 [I 2022-10-11 20:01:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 300/2304 [I 2022-10-11 20:01:57] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:28:24 [I 2022-10-11 20:01:57] ...trap/engines/engine.py.229: process: 0.83885 | load: 0.00315 [I 2022-10-11 20:01:57] ...trap/engines/engine.py.230: loss: 0.15026 [I 2022-10-11 20:01:57] ...optimizers/trijoint.py.151: total_norm: 44.329502 [I 2022-10-11 20:02:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 310/2304 [I 2022-10-11 20:02:05] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:28:12 [I 2022-10-11 20:02:05] ...trap/engines/engine.py.229: process: 0.84311 | load: 0.00315 [I 2022-10-11 20:02:05] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-11 20:02:05] ...optimizers/trijoint.py.151: total_norm: 10.096357 [I 2022-10-11 20:02:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 320/2304 [I 2022-10-11 20:02:14] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:28:24 [I 2022-10-11 20:02:14] ...trap/engines/engine.py.229: process: 0.85994 | load: 0.00315 [I 2022-10-11 20:02:14] ...trap/engines/engine.py.230: loss: 0.13015 [I 2022-10-11 20:02:14] ...optimizers/trijoint.py.151: total_norm: 11.405716 [I 2022-10-11 20:02:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 330/2304 [I 2022-10-11 20:02:22] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:28:07 [I 2022-10-11 20:02:22] ...trap/engines/engine.py.229: process: 0.84938 | load: 0.00324 [I 2022-10-11 20:02:22] ...trap/engines/engine.py.230: loss: 0.08787 [I 2022-10-11 20:02:22] ...optimizers/trijoint.py.151: total_norm: 16.675440 [I 2022-10-11 20:02:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 340/2304 [I 2022-10-11 20:02:31] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:28:29 [I 2022-10-11 20:02:31] ...trap/engines/engine.py.229: process: 0.86523 | load: 0.00318 [I 2022-10-11 20:02:31] ...trap/engines/engine.py.230: loss: 0.17502 [I 2022-10-11 20:02:31] ...optimizers/trijoint.py.151: total_norm: 11.933449 [I 2022-10-11 20:02:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 350/2304 [I 2022-10-11 20:02:40] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:27:34 [I 2022-10-11 20:02:40] ...trap/engines/engine.py.229: process: 0.82781 | load: 0.00309 [I 2022-10-11 20:02:40] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-11 20:02:40] ...optimizers/trijoint.py.151: total_norm: 17.985163 [I 2022-10-11 20:02:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 360/2304 [I 2022-10-11 20:02:48] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:27:24 [I 2022-10-11 20:02:48] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00316 [I 2022-10-11 20:02:48] ...trap/engines/engine.py.230: loss: 0.13975 [I 2022-10-11 20:02:48] ...optimizers/trijoint.py.151: total_norm: 35.574261 [I 2022-10-11 20:02:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 370/2304 [I 2022-10-11 20:02:57] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:27:33 [I 2022-10-11 20:02:57] ...trap/engines/engine.py.229: process: 0.86075 | load: 0.00313 [I 2022-10-11 20:02:57] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-11 20:02:57] ...optimizers/trijoint.py.151: total_norm: 13.288339 [I 2022-10-11 20:03:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 380/2304 [I 2022-10-11 20:03:05] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:27:20 [I 2022-10-11 20:03:05] ...trap/engines/engine.py.229: process: 0.85292 | load: 0.00318 [I 2022-10-11 20:03:05] ...trap/engines/engine.py.230: loss: 0.13757 [I 2022-10-11 20:03:05] ...optimizers/trijoint.py.151: total_norm: 13.254639 [I 2022-10-11 20:03:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 390/2304 [I 2022-10-11 20:03:14] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:27:08 [I 2022-10-11 20:03:14] ...trap/engines/engine.py.229: process: 0.86069 | load: 0.00312 [I 2022-10-11 20:03:14] ...trap/engines/engine.py.230: loss: 0.12010 [I 2022-10-11 20:03:14] ...optimizers/trijoint.py.151: total_norm: 12.836926 [I 2022-10-11 20:03:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 400/2304 [I 2022-10-11 20:03:22] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:27:01 [I 2022-10-11 20:03:22] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00314 [I 2022-10-11 20:03:22] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-11 20:03:22] ...optimizers/trijoint.py.151: total_norm: 13.612140 [I 2022-10-11 20:03:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 410/2304 [I 2022-10-11 20:03:31] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:27:10 [I 2022-10-11 20:03:31] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00315 [I 2022-10-11 20:03:31] ...trap/engines/engine.py.230: loss: 0.09753 [I 2022-10-11 20:03:31] ...optimizers/trijoint.py.151: total_norm: 11.724511 [I 2022-10-11 20:03:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 420/2304 [I 2022-10-11 20:03:39] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:26:58 [I 2022-10-11 20:03:39] ...trap/engines/engine.py.229: process: 0.86071 | load: 0.00315 [I 2022-10-11 20:03:39] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-11 20:03:39] ...optimizers/trijoint.py.151: total_norm: 13.604571 [I 2022-10-11 20:03:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 430/2304 [I 2022-10-11 20:03:48] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:40 [I 2022-10-11 20:03:48] ...trap/engines/engine.py.229: process: 0.85068 | load: 0.00320 [I 2022-10-11 20:03:48] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-11 20:03:48] ...optimizers/trijoint.py.151: total_norm: 10.986891 [I 2022-10-11 20:03:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 440/2304 [I 2022-10-11 20:03:57] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:49 [I 2022-10-11 20:03:57] ...trap/engines/engine.py.229: process: 0.84932 | load: 0.00313 [I 2022-10-11 20:03:57] ...trap/engines/engine.py.230: loss: 0.09202 [I 2022-10-11 20:03:57] ...optimizers/trijoint.py.151: total_norm: 10.230459 [I 2022-10-11 20:04:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 450/2304 [I 2022-10-11 20:04:05] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:26:25 [I 2022-10-11 20:04:05] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00313 [I 2022-10-11 20:04:05] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-11 20:04:05] ...optimizers/trijoint.py.151: total_norm: 18.721693 [I 2022-10-11 20:04:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 460/2304 [I 2022-10-11 20:04:14] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:27:01 [I 2022-10-11 20:04:14] ...trap/engines/engine.py.229: process: 0.88668 | load: 0.00400 [I 2022-10-11 20:04:14] ...trap/engines/engine.py.230: loss: 0.10702 [I 2022-10-11 20:04:14] ...optimizers/trijoint.py.151: total_norm: 17.872351 [I 2022-10-11 20:04:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 470/2304 [I 2022-10-11 20:04:23] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:27:05 [I 2022-10-11 20:04:23] ...trap/engines/engine.py.229: process: 0.88098 | load: 0.00392 [I 2022-10-11 20:04:23] ...trap/engines/engine.py.230: loss: 0.09417 [I 2022-10-11 20:04:23] ...optimizers/trijoint.py.151: total_norm: 15.108679 [I 2022-10-11 20:04:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 480/2304 [I 2022-10-11 20:04:32] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:26:57 [I 2022-10-11 20:04:32] ...trap/engines/engine.py.229: process: 0.88177 | load: 0.00395 [I 2022-10-11 20:04:32] ...trap/engines/engine.py.230: loss: 0.09616 [I 2022-10-11 20:04:32] ...optimizers/trijoint.py.151: total_norm: 17.987051 [I 2022-10-11 20:04:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 490/2304 [I 2022-10-11 20:04:41] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:26:45 [I 2022-10-11 20:04:41] ...trap/engines/engine.py.229: process: 0.88075 | load: 0.00414 [I 2022-10-11 20:04:41] ...trap/engines/engine.py.230: loss: 0.09090 [I 2022-10-11 20:04:41] ...optimizers/trijoint.py.151: total_norm: 9.638955 [I 2022-10-11 20:04:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 500/2304 [I 2022-10-11 20:04:50] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:26:42 [I 2022-10-11 20:04:50] ...trap/engines/engine.py.229: process: 0.88840 | load: 0.00396 [I 2022-10-11 20:04:50] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-11 20:04:50] ...optimizers/trijoint.py.151: total_norm: 30.710634 [I 2022-10-11 20:04:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 510/2304 [I 2022-10-11 20:04:59] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:26:37 [I 2022-10-11 20:04:59] ...trap/engines/engine.py.229: process: 0.88736 | load: 0.00393 [I 2022-10-11 20:04:59] ...trap/engines/engine.py.230: loss: 0.10444 [I 2022-10-11 20:04:59] ...optimizers/trijoint.py.151: total_norm: 11.355015 [I 2022-10-11 20:05:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 520/2304 [I 2022-10-11 20:05:07] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:26:09 [I 2022-10-11 20:05:07] ...trap/engines/engine.py.229: process: 0.87616 | load: 0.00397 [I 2022-10-11 20:05:07] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-11 20:05:07] ...optimizers/trijoint.py.151: total_norm: 26.041901 [I 2022-10-11 20:05:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 530/2304 [I 2022-10-11 20:05:16] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:26:09 [I 2022-10-11 20:05:16] ...trap/engines/engine.py.229: process: 0.86945 | load: 0.00393 [I 2022-10-11 20:05:16] ...trap/engines/engine.py.230: loss: 0.08244 [I 2022-10-11 20:05:16] ...optimizers/trijoint.py.151: total_norm: 7.778176 [I 2022-10-11 20:05:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 540/2304 [I 2022-10-11 20:05:25] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:25:53 [I 2022-10-11 20:05:25] ...trap/engines/engine.py.229: process: 0.88545 | load: 0.00394 [I 2022-10-11 20:05:25] ...trap/engines/engine.py.230: loss: 0.07971 [I 2022-10-11 20:05:25] ...optimizers/trijoint.py.151: total_norm: 16.426851 [I 2022-10-11 20:05:34] ...trap/engines/engine.py.224: train: epoch 117 | batch 550/2304 [I 2022-10-11 20:05:34] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:25:47 [I 2022-10-11 20:05:34] ...trap/engines/engine.py.229: process: 0.88703 | load: 0.00397 [I 2022-10-11 20:05:34] ...trap/engines/engine.py.230: loss: 0.12304 [I 2022-10-11 20:05:34] ...optimizers/trijoint.py.151: total_norm: 25.354620 [I 2022-10-11 20:05:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 560/2304 [I 2022-10-11 20:05:43] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:40 [I 2022-10-11 20:05:43] ...trap/engines/engine.py.229: process: 0.86254 | load: 0.00395 [I 2022-10-11 20:05:43] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-11 20:05:43] ...optimizers/trijoint.py.151: total_norm: 16.719959 [I 2022-10-11 20:05:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 570/2304 [I 2022-10-11 20:05:52] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:25:34 [I 2022-10-11 20:05:52] ...trap/engines/engine.py.229: process: 0.88175 | load: 0.00394 [I 2022-10-11 20:05:52] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-11 20:05:52] ...optimizers/trijoint.py.151: total_norm: 14.190261 [I 2022-10-11 20:06:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 580/2304 [I 2022-10-11 20:06:00] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:25:24 [I 2022-10-11 20:06:00] ...trap/engines/engine.py.229: process: 0.88181 | load: 0.00394 [I 2022-10-11 20:06:00] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-11 20:06:00] ...optimizers/trijoint.py.151: total_norm: 13.875004 [I 2022-10-11 20:06:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 590/2304 [I 2022-10-11 20:06:09] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:48 [I 2022-10-11 20:06:09] ...trap/engines/engine.py.229: process: 0.84609 | load: 0.00315 [I 2022-10-11 20:06:09] ...trap/engines/engine.py.230: loss: 0.08797 [I 2022-10-11 20:06:09] ...optimizers/trijoint.py.151: total_norm: 10.678580 [I 2022-10-11 20:06:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 600/2304 [I 2022-10-11 20:06:18] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:13 [I 2022-10-11 20:06:18] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00315 [I 2022-10-11 20:06:18] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-11 20:06:18] ...optimizers/trijoint.py.151: total_norm: 19.542953 [I 2022-10-11 20:06:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 610/2304 [I 2022-10-11 20:06:26] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:02 [I 2022-10-11 20:06:26] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00315 [I 2022-10-11 20:06:26] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-11 20:06:26] ...optimizers/trijoint.py.151: total_norm: 49.425060 [I 2022-10-11 20:06:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 620/2304 [I 2022-10-11 20:06:35] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:23:55 [I 2022-10-11 20:06:35] ...trap/engines/engine.py.229: process: 0.84620 | load: 0.00314 [I 2022-10-11 20:06:35] ...trap/engines/engine.py.230: loss: 0.09425 [I 2022-10-11 20:06:35] ...optimizers/trijoint.py.151: total_norm: 11.204992 [I 2022-10-11 20:06:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 630/2304 [I 2022-10-11 20:06:43] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:54 [I 2022-10-11 20:06:43] ...trap/engines/engine.py.229: process: 0.85548 | load: 0.00312 [I 2022-10-11 20:06:43] ...trap/engines/engine.py.230: loss: 0.09806 [I 2022-10-11 20:06:43] ...optimizers/trijoint.py.151: total_norm: 26.070435 [I 2022-10-11 20:06:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 640/2304 [I 2022-10-11 20:06:52] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:43 [I 2022-10-11 20:06:52] ...trap/engines/engine.py.229: process: 0.85396 | load: 0.00314 [I 2022-10-11 20:06:52] ...trap/engines/engine.py.230: loss: 0.14660 [I 2022-10-11 20:06:52] ...optimizers/trijoint.py.151: total_norm: 17.796253 [I 2022-10-11 20:07:01] ...trap/engines/engine.py.224: train: epoch 117 | batch 650/2304 [I 2022-10-11 20:07:01] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:23:23 [I 2022-10-11 20:07:01] ...trap/engines/engine.py.229: process: 0.84641 | load: 0.00313 [I 2022-10-11 20:07:01] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-11 20:07:01] ...optimizers/trijoint.py.151: total_norm: 12.887333 [I 2022-10-11 20:07:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 660/2304 [I 2022-10-11 20:07:09] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:14 [I 2022-10-11 20:07:09] ...trap/engines/engine.py.229: process: 0.85185 | load: 0.00312 [I 2022-10-11 20:07:09] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-11 20:07:09] ...optimizers/trijoint.py.151: total_norm: 16.381876 [I 2022-10-11 20:07:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 670/2304 [I 2022-10-11 20:07:18] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:23:06 [I 2022-10-11 20:07:18] ...trap/engines/engine.py.229: process: 0.85430 | load: 0.00310 [I 2022-10-11 20:07:18] ...trap/engines/engine.py.230: loss: 0.12138 [I 2022-10-11 20:07:18] ...optimizers/trijoint.py.151: total_norm: 15.012176 [I 2022-10-11 20:07:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 680/2304 [I 2022-10-11 20:07:26] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:06 [I 2022-10-11 20:07:26] ...trap/engines/engine.py.229: process: 0.85067 | load: 0.00313 [I 2022-10-11 20:07:26] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-11 20:07:26] ...optimizers/trijoint.py.151: total_norm: 9.594728 [I 2022-10-11 20:07:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 690/2304 [I 2022-10-11 20:07:35] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:56 [I 2022-10-11 20:07:35] ...trap/engines/engine.py.229: process: 0.86152 | load: 0.00315 [I 2022-10-11 20:07:35] ...trap/engines/engine.py.230: loss: 0.08706 [I 2022-10-11 20:07:35] ...optimizers/trijoint.py.151: total_norm: 25.804678 [I 2022-10-11 20:07:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 700/2304 [I 2022-10-11 20:07:43] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:39 [I 2022-10-11 20:07:43] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00310 [I 2022-10-11 20:07:43] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-11 20:07:43] ...optimizers/trijoint.py.151: total_norm: 12.759846 [I 2022-10-11 20:07:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 710/2304 [I 2022-10-11 20:07:52] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:27 [I 2022-10-11 20:07:52] ...trap/engines/engine.py.229: process: 0.84683 | load: 0.00307 [I 2022-10-11 20:07:52] ...trap/engines/engine.py.230: loss: 0.09608 [I 2022-10-11 20:07:52] ...optimizers/trijoint.py.151: total_norm: 13.118674 [I 2022-10-11 20:08:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 720/2304 [I 2022-10-11 20:08:00] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:11 [I 2022-10-11 20:08:00] ...trap/engines/engine.py.229: process: 0.82985 | load: 0.00306 [I 2022-10-11 20:08:00] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-11 20:08:00] ...optimizers/trijoint.py.151: total_norm: 24.184126 [I 2022-10-11 20:08:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 730/2304 [I 2022-10-11 20:08:08] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:22:02 [I 2022-10-11 20:08:08] ...trap/engines/engine.py.229: process: 0.83757 | load: 0.00300 [I 2022-10-11 20:08:08] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-11 20:08:08] ...optimizers/trijoint.py.151: total_norm: 16.849405 [I 2022-10-11 20:08:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 740/2304 [I 2022-10-11 20:08:17] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:21:48 [I 2022-10-11 20:08:17] ...trap/engines/engine.py.229: process: 0.84359 | load: 0.00301 [I 2022-10-11 20:08:17] ...trap/engines/engine.py.230: loss: 0.20193 [I 2022-10-11 20:08:17] ...optimizers/trijoint.py.151: total_norm: 41.560982 [I 2022-10-11 20:08:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 750/2304 [I 2022-10-11 20:08:25] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:21:46 [I 2022-10-11 20:08:25] ...trap/engines/engine.py.229: process: 0.84600 | load: 0.00301 [I 2022-10-11 20:08:25] ...trap/engines/engine.py.230: loss: 0.12374 [I 2022-10-11 20:08:25] ...optimizers/trijoint.py.151: total_norm: 25.482836 [I 2022-10-11 20:08:34] ...trap/engines/engine.py.224: train: epoch 117 | batch 760/2304 [I 2022-10-11 20:08:34] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:41 [I 2022-10-11 20:08:34] ...trap/engines/engine.py.229: process: 0.85238 | load: 0.00306 [I 2022-10-11 20:08:34] ...trap/engines/engine.py.230: loss: 0.07275 [I 2022-10-11 20:08:34] ...optimizers/trijoint.py.151: total_norm: 21.690416 [I 2022-10-11 20:08:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 770/2304 [I 2022-10-11 20:08:42] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:37 [I 2022-10-11 20:08:42] ...trap/engines/engine.py.229: process: 0.84250 | load: 0.00301 [I 2022-10-11 20:08:42] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-11 20:08:42] ...optimizers/trijoint.py.151: total_norm: 12.138235 [I 2022-10-11 20:08:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 780/2304 [I 2022-10-11 20:08:51] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:27 [I 2022-10-11 20:08:51] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00298 [I 2022-10-11 20:08:51] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-11 20:08:51] ...optimizers/trijoint.py.151: total_norm: 11.611728 [I 2022-10-11 20:08:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 790/2304 [I 2022-10-11 20:08:59] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:07 [I 2022-10-11 20:08:59] ...trap/engines/engine.py.229: process: 0.82279 | load: 0.00297 [I 2022-10-11 20:08:59] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-11 20:08:59] ...optimizers/trijoint.py.151: total_norm: 24.911024 [I 2022-10-11 20:09:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 800/2304 [I 2022-10-11 20:09:07] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:21:03 [I 2022-10-11 20:09:07] ...trap/engines/engine.py.229: process: 0.84203 | load: 0.00296 [I 2022-10-11 20:09:07] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-11 20:09:07] ...optimizers/trijoint.py.151: total_norm: 14.091863 [I 2022-10-11 20:09:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 810/2304 [I 2022-10-11 20:09:16] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:20:56 [I 2022-10-11 20:09:16] ...trap/engines/engine.py.229: process: 0.84179 | load: 0.00300 [I 2022-10-11 20:09:16] ...trap/engines/engine.py.230: loss: 0.07602 [I 2022-10-11 20:09:16] ...optimizers/trijoint.py.151: total_norm: 10.185349 [I 2022-10-11 20:09:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 820/2304 [I 2022-10-11 20:09:24] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:20:44 [I 2022-10-11 20:09:24] ...trap/engines/engine.py.229: process: 0.83009 | load: 0.00300 [I 2022-10-11 20:09:24] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-11 20:09:24] ...optimizers/trijoint.py.151: total_norm: 16.680159 [I 2022-10-11 20:09:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 830/2304 [I 2022-10-11 20:09:33] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:51 [I 2022-10-11 20:09:33] ...trap/engines/engine.py.229: process: 0.87821 | load: 0.00301 [I 2022-10-11 20:09:33] ...trap/engines/engine.py.230: loss: 0.10713 [I 2022-10-11 20:09:33] ...optimizers/trijoint.py.151: total_norm: 14.156262 [I 2022-10-11 20:09:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 840/2304 [I 2022-10-11 20:09:41] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:20:53 [I 2022-10-11 20:09:41] ...trap/engines/engine.py.229: process: 0.85397 | load: 0.00318 [I 2022-10-11 20:09:41] ...trap/engines/engine.py.230: loss: 0.11307 [I 2022-10-11 20:09:41] ...optimizers/trijoint.py.151: total_norm: 14.429698 [I 2022-10-11 20:09:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 850/2304 [I 2022-10-11 20:09:50] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:41 [I 2022-10-11 20:09:50] ...trap/engines/engine.py.229: process: 0.85771 | load: 0.00313 [I 2022-10-11 20:09:50] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-11 20:09:50] ...optimizers/trijoint.py.151: total_norm: 22.893927 [I 2022-10-11 20:09:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 860/2304 [I 2022-10-11 20:09:59] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:21:13 [I 2022-10-11 20:09:59] ...trap/engines/engine.py.229: process: 0.87471 | load: 0.00393 [I 2022-10-11 20:09:59] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-11 20:09:59] ...optimizers/trijoint.py.151: total_norm: 18.350388 [I 2022-10-11 20:10:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 870/2304 [I 2022-10-11 20:10:07] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:20:33 [I 2022-10-11 20:10:07] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00314 [I 2022-10-11 20:10:07] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-11 20:10:07] ...optimizers/trijoint.py.151: total_norm: 11.676394 [I 2022-10-11 20:10:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 880/2304 [I 2022-10-11 20:10:16] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:20:21 [I 2022-10-11 20:10:16] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00314 [I 2022-10-11 20:10:16] ...trap/engines/engine.py.230: loss: 0.11750 [I 2022-10-11 20:10:16] ...optimizers/trijoint.py.151: total_norm: 13.674693 [I 2022-10-11 20:10:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 890/2304 [I 2022-10-11 20:10:25] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:20:09 [I 2022-10-11 20:10:25] ...trap/engines/engine.py.229: process: 0.86375 | load: 0.00313 [I 2022-10-11 20:10:25] ...trap/engines/engine.py.230: loss: 0.12185 [I 2022-10-11 20:10:25] ...optimizers/trijoint.py.151: total_norm: 13.324088 [I 2022-10-11 20:10:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 900/2304 [I 2022-10-11 20:10:33] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:19:56 [I 2022-10-11 20:10:33] ...trap/engines/engine.py.229: process: 0.85159 | load: 0.00312 [I 2022-10-11 20:10:33] ...trap/engines/engine.py.230: loss: 0.16884 [I 2022-10-11 20:10:33] ...optimizers/trijoint.py.151: total_norm: 11.169497 [I 2022-10-11 20:10:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 910/2304 [I 2022-10-11 20:10:42] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:19:49 [I 2022-10-11 20:10:42] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00315 [I 2022-10-11 20:10:42] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-11 20:10:42] ...optimizers/trijoint.py.151: total_norm: 25.697367 [I 2022-10-11 20:10:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 920/2304 [I 2022-10-11 20:10:50] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:19:50 [I 2022-10-11 20:10:50] ...trap/engines/engine.py.229: process: 0.85353 | load: 0.00316 [I 2022-10-11 20:10:50] ...trap/engines/engine.py.230: loss: 0.18232 [I 2022-10-11 20:10:50] ...optimizers/trijoint.py.151: total_norm: 22.176569 [I 2022-10-11 20:10:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 930/2304 [I 2022-10-11 20:10:59] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:19:25 [I 2022-10-11 20:10:59] ...trap/engines/engine.py.229: process: 0.84388 | load: 0.00312 [I 2022-10-11 20:10:59] ...trap/engines/engine.py.230: loss: 0.16097 [I 2022-10-11 20:10:59] ...optimizers/trijoint.py.151: total_norm: 23.298922 [I 2022-10-11 20:11:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 940/2304 [I 2022-10-11 20:11:07] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:19:29 [I 2022-10-11 20:11:07] ...trap/engines/engine.py.229: process: 0.85908 | load: 0.00314 [I 2022-10-11 20:11:07] ...trap/engines/engine.py.230: loss: 0.12797 [I 2022-10-11 20:11:07] ...optimizers/trijoint.py.151: total_norm: 16.810543 [I 2022-10-11 20:11:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 950/2304 [I 2022-10-11 20:11:16] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:19:22 [I 2022-10-11 20:11:16] ...trap/engines/engine.py.229: process: 0.86334 | load: 0.00314 [I 2022-10-11 20:11:16] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-11 20:11:16] ...optimizers/trijoint.py.151: total_norm: 15.513946 [I 2022-10-11 20:11:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 960/2304 [I 2022-10-11 20:11:24] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:19:10 [I 2022-10-11 20:11:24] ...trap/engines/engine.py.229: process: 0.85380 | load: 0.00313 [I 2022-10-11 20:11:24] ...trap/engines/engine.py.230: loss: 0.10056 [I 2022-10-11 20:11:24] ...optimizers/trijoint.py.151: total_norm: 12.256430 [I 2022-10-11 20:11:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 970/2304 [I 2022-10-11 20:11:33] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:18:57 [I 2022-10-11 20:11:33] ...trap/engines/engine.py.229: process: 0.85678 | load: 0.00311 [I 2022-10-11 20:11:33] ...trap/engines/engine.py.230: loss: 0.07690 [I 2022-10-11 20:11:33] ...optimizers/trijoint.py.151: total_norm: 11.067151 [I 2022-10-11 20:11:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 980/2304 [I 2022-10-11 20:11:42] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:18:55 [I 2022-10-11 20:11:42] ...trap/engines/engine.py.229: process: 0.86628 | load: 0.00337 [I 2022-10-11 20:11:42] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-11 20:11:42] ...optimizers/trijoint.py.151: total_norm: 13.989732 [I 2022-10-11 20:11:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 990/2304 [I 2022-10-11 20:11:50] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:18:45 [I 2022-10-11 20:11:50] ...trap/engines/engine.py.229: process: 0.86677 | load: 0.00312 [I 2022-10-11 20:11:50] ...trap/engines/engine.py.230: loss: 0.08997 [I 2022-10-11 20:11:50] ...optimizers/trijoint.py.151: total_norm: 10.405060 [I 2022-10-11 20:11:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1000/2304 [I 2022-10-11 20:11:59] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:18:44 [I 2022-10-11 20:11:59] ...trap/engines/engine.py.229: process: 0.86770 | load: 0.00315 [I 2022-10-11 20:11:59] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-11 20:11:59] ...optimizers/trijoint.py.151: total_norm: 10.497060 [I 2022-10-11 20:12:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1010/2304 [I 2022-10-11 20:12:07] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:18:33 [I 2022-10-11 20:12:07] ...trap/engines/engine.py.229: process: 0.87004 | load: 0.00314 [I 2022-10-11 20:12:07] ...trap/engines/engine.py.230: loss: 0.21248 [I 2022-10-11 20:12:07] ...optimizers/trijoint.py.151: total_norm: 33.575096 [I 2022-10-11 20:12:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 1020/2304 [I 2022-10-11 20:12:16] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:18:20 [I 2022-10-11 20:12:16] ...trap/engines/engine.py.229: process: 0.87502 | load: 0.00312 [I 2022-10-11 20:12:16] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-11 20:12:16] ...optimizers/trijoint.py.151: total_norm: 10.812501 [I 2022-10-11 20:12:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 1030/2304 [I 2022-10-11 20:12:25] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:18:17 [I 2022-10-11 20:12:25] ...trap/engines/engine.py.229: process: 0.86814 | load: 0.00314 [I 2022-10-11 20:12:25] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 20:12:25] ...optimizers/trijoint.py.151: total_norm: 18.986900 [I 2022-10-11 20:12:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 1040/2304 [I 2022-10-11 20:12:33] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:18:07 [I 2022-10-11 20:12:33] ...trap/engines/engine.py.229: process: 0.85962 | load: 0.00313 [I 2022-10-11 20:12:33] ...trap/engines/engine.py.230: loss: 0.08434 [I 2022-10-11 20:12:33] ...optimizers/trijoint.py.151: total_norm: 13.296035 [I 2022-10-11 20:12:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 1050/2304 [I 2022-10-11 20:12:42] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:18:20 [I 2022-10-11 20:12:42] ...trap/engines/engine.py.229: process: 0.88165 | load: 0.00397 [I 2022-10-11 20:12:42] ...trap/engines/engine.py.230: loss: 0.09697 [I 2022-10-11 20:12:42] ...optimizers/trijoint.py.151: total_norm: 13.985068 [I 2022-10-11 20:12:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 1060/2304 [I 2022-10-11 20:12:51] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:18:12 [I 2022-10-11 20:12:51] ...trap/engines/engine.py.229: process: 0.87714 | load: 0.00401 [I 2022-10-11 20:12:51] ...trap/engines/engine.py.230: loss: 0.07501 [I 2022-10-11 20:12:51] ...optimizers/trijoint.py.151: total_norm: 12.758607 [I 2022-10-11 20:13:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 1070/2304 [I 2022-10-11 20:13:00] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:18:13 [I 2022-10-11 20:13:00] ...trap/engines/engine.py.229: process: 0.89326 | load: 0.00396 [I 2022-10-11 20:13:00] ...trap/engines/engine.py.230: loss: 0.08883 [I 2022-10-11 20:13:00] ...optimizers/trijoint.py.151: total_norm: 13.727884 [I 2022-10-11 20:13:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 1080/2304 [I 2022-10-11 20:13:08] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:18:06 [I 2022-10-11 20:13:08] ...trap/engines/engine.py.229: process: 0.88314 | load: 0.00403 [I 2022-10-11 20:13:08] ...trap/engines/engine.py.230: loss: 0.17694 [I 2022-10-11 20:13:08] ...optimizers/trijoint.py.151: total_norm: 20.710089 [I 2022-10-11 20:13:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 1090/2304 [I 2022-10-11 20:13:17] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:51 [I 2022-10-11 20:13:17] ...trap/engines/engine.py.229: process: 0.88155 | load: 0.00393 [I 2022-10-11 20:13:17] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-11 20:13:17] ...optimizers/trijoint.py.151: total_norm: 10.218385 [I 2022-10-11 20:13:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 1100/2304 [I 2022-10-11 20:13:26] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:29 [I 2022-10-11 20:13:26] ...trap/engines/engine.py.229: process: 0.87709 | load: 0.00312 [I 2022-10-11 20:13:26] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-11 20:13:26] ...optimizers/trijoint.py.151: total_norm: 10.540780 [I 2022-10-11 20:13:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 1110/2304 [I 2022-10-11 20:13:35] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:16:52 [I 2022-10-11 20:13:35] ...trap/engines/engine.py.229: process: 0.83217 | load: 0.00310 [I 2022-10-11 20:13:35] ...trap/engines/engine.py.230: loss: 0.10854 [I 2022-10-11 20:13:35] ...optimizers/trijoint.py.151: total_norm: 26.899855 [I 2022-10-11 20:13:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 1120/2304 [I 2022-10-11 20:13:43] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:38 [I 2022-10-11 20:13:43] ...trap/engines/engine.py.229: process: 0.84875 | load: 0.00302 [I 2022-10-11 20:13:43] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-11 20:13:43] ...optimizers/trijoint.py.151: total_norm: 25.002825 [I 2022-10-11 20:13:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 1130/2304 [I 2022-10-11 20:13:52] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:16:29 [I 2022-10-11 20:13:52] ...trap/engines/engine.py.229: process: 0.84182 | load: 0.00297 [I 2022-10-11 20:13:52] ...trap/engines/engine.py.230: loss: 0.12546 [I 2022-10-11 20:13:52] ...optimizers/trijoint.py.151: total_norm: 13.405358 [I 2022-10-11 20:14:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 1140/2304 [I 2022-10-11 20:14:00] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:16:25 [I 2022-10-11 20:14:00] ...trap/engines/engine.py.229: process: 0.85178 | load: 0.00294 [I 2022-10-11 20:14:00] ...trap/engines/engine.py.230: loss: 0.08774 [I 2022-10-11 20:14:00] ...optimizers/trijoint.py.151: total_norm: 19.778133 [I 2022-10-11 20:14:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 1150/2304 [I 2022-10-11 20:14:08] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:16:09 [I 2022-10-11 20:14:08] ...trap/engines/engine.py.229: process: 0.82967 | load: 0.00298 [I 2022-10-11 20:14:08] ...trap/engines/engine.py.230: loss: 0.11948 [I 2022-10-11 20:14:08] ...optimizers/trijoint.py.151: total_norm: 24.429070 [I 2022-10-11 20:14:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 1160/2304 [I 2022-10-11 20:14:17] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:16:03 [I 2022-10-11 20:14:17] ...trap/engines/engine.py.229: process: 0.83283 | load: 0.00295 [I 2022-10-11 20:14:17] ...trap/engines/engine.py.230: loss: 0.09656 [I 2022-10-11 20:14:17] ...optimizers/trijoint.py.151: total_norm: 13.418541 [I 2022-10-11 20:14:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 1170/2304 [I 2022-10-11 20:14:25] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:15:47 [I 2022-10-11 20:14:25] ...trap/engines/engine.py.229: process: 0.82868 | load: 0.00297 [I 2022-10-11 20:14:25] ...trap/engines/engine.py.230: loss: 0.21090 [I 2022-10-11 20:14:25] ...optimizers/trijoint.py.151: total_norm: 16.004269 [I 2022-10-11 20:14:34] ...trap/engines/engine.py.224: train: epoch 117 | batch 1180/2304 [I 2022-10-11 20:14:34] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:15:49 [I 2022-10-11 20:14:34] ...trap/engines/engine.py.229: process: 0.86472 | load: 0.00297 [I 2022-10-11 20:14:34] ...trap/engines/engine.py.230: loss: 0.09265 [I 2022-10-11 20:14:34] ...optimizers/trijoint.py.151: total_norm: 14.215952 [I 2022-10-11 20:14:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 1190/2304 [I 2022-10-11 20:14:42] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:15:31 [I 2022-10-11 20:14:42] ...trap/engines/engine.py.229: process: 0.82696 | load: 0.00299 [I 2022-10-11 20:14:42] ...trap/engines/engine.py.230: loss: 0.18756 [I 2022-10-11 20:14:42] ...optimizers/trijoint.py.151: total_norm: 12.501431 [I 2022-10-11 20:14:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 1200/2304 [I 2022-10-11 20:14:51] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:15:31 [I 2022-10-11 20:14:51] ...trap/engines/engine.py.229: process: 0.84840 | load: 0.00303 [I 2022-10-11 20:14:51] ...trap/engines/engine.py.230: loss: 0.10220 [I 2022-10-11 20:14:51] ...optimizers/trijoint.py.151: total_norm: 15.134736 [I 2022-10-11 20:14:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1210/2304 [I 2022-10-11 20:14:59] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:15:19 [I 2022-10-11 20:14:59] ...trap/engines/engine.py.229: process: 0.84270 | load: 0.00299 [I 2022-10-11 20:14:59] ...trap/engines/engine.py.230: loss: 0.08632 [I 2022-10-11 20:14:59] ...optimizers/trijoint.py.151: total_norm: 10.690429 [I 2022-10-11 20:15:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1220/2304 [I 2022-10-11 20:15:07] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:15:12 [I 2022-10-11 20:15:07] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00299 [I 2022-10-11 20:15:07] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-11 20:15:07] ...optimizers/trijoint.py.151: total_norm: 25.334911 [I 2022-10-11 20:15:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 1230/2304 [I 2022-10-11 20:15:16] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:15:00 [I 2022-10-11 20:15:16] ...trap/engines/engine.py.229: process: 0.83770 | load: 0.00301 [I 2022-10-11 20:15:16] ...trap/engines/engine.py.230: loss: 0.23795 [I 2022-10-11 20:15:16] ...optimizers/trijoint.py.151: total_norm: 23.937757 [I 2022-10-11 20:15:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 1240/2304 [I 2022-10-11 20:15:24] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:14:54 [I 2022-10-11 20:15:24] ...trap/engines/engine.py.229: process: 0.83332 | load: 0.00298 [I 2022-10-11 20:15:24] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-11 20:15:24] ...optimizers/trijoint.py.151: total_norm: 13.045090 [I 2022-10-11 20:15:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 1250/2304 [I 2022-10-11 20:15:33] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:14:47 [I 2022-10-11 20:15:33] ...trap/engines/engine.py.229: process: 0.84307 | load: 0.00299 [I 2022-10-11 20:15:33] ...trap/engines/engine.py.230: loss: 0.09905 [I 2022-10-11 20:15:33] ...optimizers/trijoint.py.151: total_norm: 14.521420 [I 2022-10-11 20:15:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 1260/2304 [I 2022-10-11 20:15:41] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:14:38 [I 2022-10-11 20:15:41] ...trap/engines/engine.py.229: process: 0.84597 | load: 0.00297 [I 2022-10-11 20:15:41] ...trap/engines/engine.py.230: loss: 0.09085 [I 2022-10-11 20:15:41] ...optimizers/trijoint.py.151: total_norm: 14.929724 [I 2022-10-11 20:15:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 1270/2304 [I 2022-10-11 20:15:50] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:14:28 [I 2022-10-11 20:15:50] ...trap/engines/engine.py.229: process: 0.83574 | load: 0.00304 [I 2022-10-11 20:15:50] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-11 20:15:50] ...optimizers/trijoint.py.151: total_norm: 15.151180 [I 2022-10-11 20:15:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 1280/2304 [I 2022-10-11 20:15:58] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:19 [I 2022-10-11 20:15:58] ...trap/engines/engine.py.229: process: 0.83520 | load: 0.00300 [I 2022-10-11 20:15:58] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-11 20:15:58] ...optimizers/trijoint.py.151: total_norm: 14.321009 [I 2022-10-11 20:16:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1290/2304 [I 2022-10-11 20:16:07] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:14:18 [I 2022-10-11 20:16:07] ...trap/engines/engine.py.229: process: 0.85153 | load: 0.00310 [I 2022-10-11 20:16:07] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-11 20:16:07] ...optimizers/trijoint.py.151: total_norm: 14.471535 [I 2022-10-11 20:16:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 1300/2304 [I 2022-10-11 20:16:15] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:14:15 [I 2022-10-11 20:16:15] ...trap/engines/engine.py.229: process: 0.86772 | load: 0.00312 [I 2022-10-11 20:16:15] ...trap/engines/engine.py.230: loss: 0.11458 [I 2022-10-11 20:16:15] ...optimizers/trijoint.py.151: total_norm: 26.065802 [I 2022-10-11 20:16:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 1310/2304 [I 2022-10-11 20:16:24] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:14:08 [I 2022-10-11 20:16:24] ...trap/engines/engine.py.229: process: 0.85375 | load: 0.00313 [I 2022-10-11 20:16:24] ...trap/engines/engine.py.230: loss: 0.10667 [I 2022-10-11 20:16:24] ...optimizers/trijoint.py.151: total_norm: 18.020983 [I 2022-10-11 20:16:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1320/2304 [I 2022-10-11 20:16:32] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:13:59 [I 2022-10-11 20:16:32] ...trap/engines/engine.py.229: process: 0.86240 | load: 0.00409 [I 2022-10-11 20:16:32] ...trap/engines/engine.py.230: loss: 0.23879 [I 2022-10-11 20:16:32] ...optimizers/trijoint.py.151: total_norm: 10.424189 [I 2022-10-11 20:16:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 1330/2304 [I 2022-10-11 20:16:41] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:13:49 [I 2022-10-11 20:16:41] ...trap/engines/engine.py.229: process: 0.85451 | load: 0.00317 [I 2022-10-11 20:16:41] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-11 20:16:41] ...optimizers/trijoint.py.151: total_norm: 21.476316 [I 2022-10-11 20:16:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1340/2304 [I 2022-10-11 20:16:49] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:13:44 [I 2022-10-11 20:16:49] ...trap/engines/engine.py.229: process: 0.86986 | load: 0.00311 [I 2022-10-11 20:16:49] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-11 20:16:49] ...optimizers/trijoint.py.151: total_norm: 10.174811 [I 2022-10-11 20:16:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 1350/2304 [I 2022-10-11 20:16:58] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:13:36 [I 2022-10-11 20:16:58] ...trap/engines/engine.py.229: process: 0.86520 | load: 0.00313 [I 2022-10-11 20:16:58] ...trap/engines/engine.py.230: loss: 0.08569 [I 2022-10-11 20:16:58] ...optimizers/trijoint.py.151: total_norm: 10.597878 [I 2022-10-11 20:17:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1360/2304 [I 2022-10-11 20:17:06] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:13:20 [I 2022-10-11 20:17:06] ...trap/engines/engine.py.229: process: 0.84071 | load: 0.00316 [I 2022-10-11 20:17:06] ...trap/engines/engine.py.230: loss: 0.11775 [I 2022-10-11 20:17:06] ...optimizers/trijoint.py.151: total_norm: 14.860122 [I 2022-10-11 20:17:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 1370/2304 [I 2022-10-11 20:17:15] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:13:20 [I 2022-10-11 20:17:15] ...trap/engines/engine.py.229: process: 0.86145 | load: 0.00313 [I 2022-10-11 20:17:15] ...trap/engines/engine.py.230: loss: 0.11774 [I 2022-10-11 20:17:15] ...optimizers/trijoint.py.151: total_norm: 17.853285 [I 2022-10-11 20:17:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1380/2304 [I 2022-10-11 20:17:23] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:13:10 [I 2022-10-11 20:17:23] ...trap/engines/engine.py.229: process: 0.85242 | load: 0.00315 [I 2022-10-11 20:17:23] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-11 20:17:23] ...optimizers/trijoint.py.151: total_norm: 12.515153 [I 2022-10-11 20:17:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1390/2304 [I 2022-10-11 20:17:32] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:13:03 [I 2022-10-11 20:17:32] ...trap/engines/engine.py.229: process: 0.85322 | load: 0.00313 [I 2022-10-11 20:17:32] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-11 20:17:32] ...optimizers/trijoint.py.151: total_norm: 14.470757 [I 2022-10-11 20:17:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1400/2304 [I 2022-10-11 20:17:40] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:12:48 [I 2022-10-11 20:17:40] ...trap/engines/engine.py.229: process: 0.84929 | load: 0.00314 [I 2022-10-11 20:17:40] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-11 20:17:40] ...optimizers/trijoint.py.151: total_norm: 26.109552 [I 2022-10-11 20:17:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1410/2304 [I 2022-10-11 20:17:49] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:12:37 [I 2022-10-11 20:17:49] ...trap/engines/engine.py.229: process: 0.84268 | load: 0.00315 [I 2022-10-11 20:17:49] ...trap/engines/engine.py.230: loss: 0.16382 [I 2022-10-11 20:17:49] ...optimizers/trijoint.py.151: total_norm: 17.550419 [I 2022-10-11 20:17:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1420/2304 [I 2022-10-11 20:17:57] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:12:36 [I 2022-10-11 20:17:57] ...trap/engines/engine.py.229: process: 0.86142 | load: 0.00315 [I 2022-10-11 20:17:57] ...trap/engines/engine.py.230: loss: 0.08679 [I 2022-10-11 20:17:57] ...optimizers/trijoint.py.151: total_norm: 12.129505 [I 2022-10-11 20:18:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1430/2304 [I 2022-10-11 20:18:06] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:12:22 [I 2022-10-11 20:18:06] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00311 [I 2022-10-11 20:18:06] ...trap/engines/engine.py.230: loss: 0.15336 [I 2022-10-11 20:18:06] ...optimizers/trijoint.py.151: total_norm: 18.512684 [I 2022-10-11 20:18:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1440/2304 [I 2022-10-11 20:18:14] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:12:10 [I 2022-10-11 20:18:14] ...trap/engines/engine.py.229: process: 0.83286 | load: 0.00313 [I 2022-10-11 20:18:14] ...trap/engines/engine.py.230: loss: 0.09236 [I 2022-10-11 20:18:14] ...optimizers/trijoint.py.151: total_norm: 20.431025 [I 2022-10-11 20:18:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1450/2304 [I 2022-10-11 20:18:23] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:12:02 [I 2022-10-11 20:18:23] ...trap/engines/engine.py.229: process: 0.83762 | load: 0.00312 [I 2022-10-11 20:18:23] ...trap/engines/engine.py.230: loss: 0.08959 [I 2022-10-11 20:18:23] ...optimizers/trijoint.py.151: total_norm: 14.128712 [I 2022-10-11 20:18:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1460/2304 [I 2022-10-11 20:18:31] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:11:55 [I 2022-10-11 20:18:31] ...trap/engines/engine.py.229: process: 0.85541 | load: 0.00316 [I 2022-10-11 20:18:31] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-11 20:18:31] ...optimizers/trijoint.py.151: total_norm: 16.667093 [I 2022-10-11 20:18:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1470/2304 [I 2022-10-11 20:18:40] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:11:49 [I 2022-10-11 20:18:40] ...trap/engines/engine.py.229: process: 0.83664 | load: 0.00312 [I 2022-10-11 20:18:40] ...trap/engines/engine.py.230: loss: 0.12999 [I 2022-10-11 20:18:40] ...optimizers/trijoint.py.151: total_norm: 20.256756 [I 2022-10-11 20:18:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1480/2304 [I 2022-10-11 20:18:49] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:11:48 [I 2022-10-11 20:18:49] ...trap/engines/engine.py.229: process: 0.85974 | load: 0.00313 [I 2022-10-11 20:18:49] ...trap/engines/engine.py.230: loss: 0.08816 [I 2022-10-11 20:18:49] ...optimizers/trijoint.py.151: total_norm: 12.594199 [I 2022-10-11 20:18:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1490/2304 [I 2022-10-11 20:18:57] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:11:31 [I 2022-10-11 20:18:57] ...trap/engines/engine.py.229: process: 0.83176 | load: 0.00313 [I 2022-10-11 20:18:57] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-11 20:18:57] ...optimizers/trijoint.py.151: total_norm: 46.191467 [I 2022-10-11 20:19:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1500/2304 [I 2022-10-11 20:19:06] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:11:30 [I 2022-10-11 20:19:06] ...trap/engines/engine.py.229: process: 0.85458 | load: 0.00311 [I 2022-10-11 20:19:06] ...trap/engines/engine.py.230: loss: 0.08804 [I 2022-10-11 20:19:06] ...optimizers/trijoint.py.151: total_norm: 11.885471 [I 2022-10-11 20:19:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1510/2304 [I 2022-10-11 20:19:14] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:11:21 [I 2022-10-11 20:19:14] ...trap/engines/engine.py.229: process: 0.85098 | load: 0.00314 [I 2022-10-11 20:19:14] ...trap/engines/engine.py.230: loss: 0.17916 [I 2022-10-11 20:19:14] ...optimizers/trijoint.py.151: total_norm: 17.288095 [I 2022-10-11 20:19:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1520/2304 [I 2022-10-11 20:19:23] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:11:16 [I 2022-10-11 20:19:23] ...trap/engines/engine.py.229: process: 0.87779 | load: 0.00524 [I 2022-10-11 20:19:23] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-11 20:19:23] ...optimizers/trijoint.py.151: total_norm: 26.662018 [I 2022-10-11 20:19:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1530/2304 [I 2022-10-11 20:19:32] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:11:02 [I 2022-10-11 20:19:32] ...trap/engines/engine.py.229: process: 0.85368 | load: 0.00313 [I 2022-10-11 20:19:32] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-11 20:19:32] ...optimizers/trijoint.py.151: total_norm: 18.824083 [I 2022-10-11 20:19:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1540/2304 [I 2022-10-11 20:19:40] ...trap/engines/engine.py.228: elapsed: 0:22:03 | left: 0:10:48 [I 2022-10-11 20:19:40] ...trap/engines/engine.py.229: process: 0.84172 | load: 0.00315 [I 2022-10-11 20:19:40] ...trap/engines/engine.py.230: loss: 0.10841 [I 2022-10-11 20:19:40] ...optimizers/trijoint.py.151: total_norm: 14.072359 [I 2022-10-11 20:19:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1550/2304 [I 2022-10-11 20:19:49] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:10:41 [I 2022-10-11 20:19:49] ...trap/engines/engine.py.229: process: 0.84933 | load: 0.00315 [I 2022-10-11 20:19:49] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-11 20:19:49] ...optimizers/trijoint.py.151: total_norm: 16.172325 [I 2022-10-11 20:19:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1560/2304 [I 2022-10-11 20:19:57] ...trap/engines/engine.py.228: elapsed: 0:22:20 | left: 0:10:35 [I 2022-10-11 20:19:57] ...trap/engines/engine.py.229: process: 0.83733 | load: 0.00304 [I 2022-10-11 20:19:57] ...trap/engines/engine.py.230: loss: 0.18616 [I 2022-10-11 20:19:57] ...optimizers/trijoint.py.151: total_norm: 13.402760 [I 2022-10-11 20:20:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1570/2304 [I 2022-10-11 20:20:06] ...trap/engines/engine.py.228: elapsed: 0:22:29 | left: 0:10:19 [I 2022-10-11 20:20:06] ...trap/engines/engine.py.229: process: 0.84333 | load: 0.00298 [I 2022-10-11 20:20:06] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-11 20:20:06] ...optimizers/trijoint.py.151: total_norm: 9.364367 [I 2022-10-11 20:20:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1580/2304 [I 2022-10-11 20:20:14] ...trap/engines/engine.py.228: elapsed: 0:22:37 | left: 0:10:09 [I 2022-10-11 20:20:14] ...trap/engines/engine.py.229: process: 0.83470 | load: 0.00294 [I 2022-10-11 20:20:14] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-11 20:20:14] ...optimizers/trijoint.py.151: total_norm: 20.454372 [I 2022-10-11 20:20:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1590/2304 [I 2022-10-11 20:20:23] ...trap/engines/engine.py.228: elapsed: 0:22:45 | left: 0:10:01 [I 2022-10-11 20:20:23] ...trap/engines/engine.py.229: process: 0.82921 | load: 0.00296 [I 2022-10-11 20:20:23] ...trap/engines/engine.py.230: loss: 0.11244 [I 2022-10-11 20:20:23] ...optimizers/trijoint.py.151: total_norm: 9.698239 [I 2022-10-11 20:20:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1600/2304 [I 2022-10-11 20:20:31] ...trap/engines/engine.py.228: elapsed: 0:22:54 | left: 0:09:52 [I 2022-10-11 20:20:31] ...trap/engines/engine.py.229: process: 0.83727 | load: 0.00299 [I 2022-10-11 20:20:31] ...trap/engines/engine.py.230: loss: 0.22243 [I 2022-10-11 20:20:31] ...optimizers/trijoint.py.151: total_norm: 29.948397 [I 2022-10-11 20:20:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 1610/2304 [I 2022-10-11 20:20:39] ...trap/engines/engine.py.228: elapsed: 0:23:02 | left: 0:09:48 [I 2022-10-11 20:20:39] ...trap/engines/engine.py.229: process: 0.84833 | load: 0.00298 [I 2022-10-11 20:20:39] ...trap/engines/engine.py.230: loss: 0.13939 [I 2022-10-11 20:20:39] ...optimizers/trijoint.py.151: total_norm: 16.732059 [I 2022-10-11 20:20:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 1620/2304 [I 2022-10-11 20:20:48] ...trap/engines/engine.py.228: elapsed: 0:23:11 | left: 0:09:59 [I 2022-10-11 20:20:48] ...trap/engines/engine.py.229: process: 0.87882 | load: 0.00378 [I 2022-10-11 20:20:48] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-11 20:20:48] ...optimizers/trijoint.py.151: total_norm: 14.846631 [I 2022-10-11 20:20:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1630/2304 [I 2022-10-11 20:20:57] ...trap/engines/engine.py.228: elapsed: 0:23:20 | left: 0:09:50 [I 2022-10-11 20:20:57] ...trap/engines/engine.py.229: process: 0.88694 | load: 0.00378 [I 2022-10-11 20:20:57] ...trap/engines/engine.py.230: loss: 0.19662 [I 2022-10-11 20:20:57] ...optimizers/trijoint.py.151: total_norm: 21.876474 [I 2022-10-11 20:21:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1640/2304 [I 2022-10-11 20:21:06] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:40 [I 2022-10-11 20:21:06] ...trap/engines/engine.py.229: process: 0.86781 | load: 0.00386 [I 2022-10-11 20:21:06] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-11 20:21:06] ...optimizers/trijoint.py.151: total_norm: 23.659761 [I 2022-10-11 20:21:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 1650/2304 [I 2022-10-11 20:21:15] ...trap/engines/engine.py.228: elapsed: 0:23:37 | left: 0:09:32 [I 2022-10-11 20:21:15] ...trap/engines/engine.py.229: process: 0.86795 | load: 0.00380 [I 2022-10-11 20:21:15] ...trap/engines/engine.py.230: loss: 0.07726 [I 2022-10-11 20:21:15] ...optimizers/trijoint.py.151: total_norm: 12.824639 [I 2022-10-11 20:21:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1660/2304 [I 2022-10-11 20:21:23] ...trap/engines/engine.py.228: elapsed: 0:23:46 | left: 0:09:23 [I 2022-10-11 20:21:23] ...trap/engines/engine.py.229: process: 0.87812 | load: 0.00385 [I 2022-10-11 20:21:23] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-11 20:21:23] ...optimizers/trijoint.py.151: total_norm: 71.752899 [I 2022-10-11 20:21:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1670/2304 [I 2022-10-11 20:21:32] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:08:53 [I 2022-10-11 20:21:32] ...trap/engines/engine.py.229: process: 0.83593 | load: 0.00300 [I 2022-10-11 20:21:32] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-11 20:21:32] ...optimizers/trijoint.py.151: total_norm: 12.803796 [I 2022-10-11 20:21:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1680/2304 [I 2022-10-11 20:21:40] ...trap/engines/engine.py.228: elapsed: 0:24:03 | left: 0:08:52 [I 2022-10-11 20:21:40] ...trap/engines/engine.py.229: process: 0.86785 | load: 0.00298 [I 2022-10-11 20:21:40] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-11 20:21:40] ...optimizers/trijoint.py.151: total_norm: 14.566605 [I 2022-10-11 20:21:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1690/2304 [I 2022-10-11 20:21:49] ...trap/engines/engine.py.228: elapsed: 0:24:12 | left: 0:08:41 [I 2022-10-11 20:21:49] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00300 [I 2022-10-11 20:21:49] ...trap/engines/engine.py.230: loss: 0.10402 [I 2022-10-11 20:21:49] ...optimizers/trijoint.py.151: total_norm: 25.463251 [I 2022-10-11 20:21:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1700/2304 [I 2022-10-11 20:21:57] ...trap/engines/engine.py.228: elapsed: 0:24:20 | left: 0:08:29 [I 2022-10-11 20:21:57] ...trap/engines/engine.py.229: process: 0.83814 | load: 0.00298 [I 2022-10-11 20:21:57] ...trap/engines/engine.py.230: loss: 0.23825 [I 2022-10-11 20:21:57] ...optimizers/trijoint.py.151: total_norm: 39.997334 [I 2022-10-11 20:22:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1710/2304 [I 2022-10-11 20:22:06] ...trap/engines/engine.py.228: elapsed: 0:24:29 | left: 0:08:22 [I 2022-10-11 20:22:06] ...trap/engines/engine.py.229: process: 0.85228 | load: 0.00304 [I 2022-10-11 20:22:06] ...trap/engines/engine.py.230: loss: 0.10174 [I 2022-10-11 20:22:06] ...optimizers/trijoint.py.151: total_norm: 24.860752 [I 2022-10-11 20:22:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1720/2304 [I 2022-10-11 20:22:14] ...trap/engines/engine.py.228: elapsed: 0:24:37 | left: 0:08:18 [I 2022-10-11 20:22:14] ...trap/engines/engine.py.229: process: 0.85627 | load: 0.00299 [I 2022-10-11 20:22:14] ...trap/engines/engine.py.230: loss: 0.10299 [I 2022-10-11 20:22:14] ...optimizers/trijoint.py.151: total_norm: 39.755054 [I 2022-10-11 20:22:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1730/2304 [I 2022-10-11 20:22:23] ...trap/engines/engine.py.228: elapsed: 0:24:46 | left: 0:08:11 [I 2022-10-11 20:22:23] ...trap/engines/engine.py.229: process: 0.85348 | load: 0.00311 [I 2022-10-11 20:22:23] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-11 20:22:23] ...optimizers/trijoint.py.151: total_norm: 17.513433 [I 2022-10-11 20:22:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1740/2304 [I 2022-10-11 20:22:32] ...trap/engines/engine.py.228: elapsed: 0:24:55 | left: 0:08:10 [I 2022-10-11 20:22:32] ...trap/engines/engine.py.229: process: 0.84065 | load: 0.00312 [I 2022-10-11 20:22:32] ...trap/engines/engine.py.230: loss: 0.11314 [I 2022-10-11 20:22:32] ...optimizers/trijoint.py.151: total_norm: 33.827274 [I 2022-10-11 20:22:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1750/2304 [I 2022-10-11 20:22:40] ...trap/engines/engine.py.228: elapsed: 0:25:03 | left: 0:07:54 [I 2022-10-11 20:22:40] ...trap/engines/engine.py.229: process: 0.87513 | load: 0.00314 [I 2022-10-11 20:22:40] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-11 20:22:40] ...optimizers/trijoint.py.151: total_norm: 28.144154 [I 2022-10-11 20:22:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1760/2304 [I 2022-10-11 20:22:49] ...trap/engines/engine.py.228: elapsed: 0:25:12 | left: 0:07:45 [I 2022-10-11 20:22:49] ...trap/engines/engine.py.229: process: 0.85862 | load: 0.00312 [I 2022-10-11 20:22:49] ...trap/engines/engine.py.230: loss: 0.15014 [I 2022-10-11 20:22:49] ...optimizers/trijoint.py.151: total_norm: 22.466454 [I 2022-10-11 20:22:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1770/2304 [I 2022-10-11 20:22:57] ...trap/engines/engine.py.228: elapsed: 0:25:20 | left: 0:07:36 [I 2022-10-11 20:22:57] ...trap/engines/engine.py.229: process: 0.84881 | load: 0.00313 [I 2022-10-11 20:22:57] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-11 20:22:57] ...optimizers/trijoint.py.151: total_norm: 8.823771 [I 2022-10-11 20:23:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1780/2304 [I 2022-10-11 20:23:06] ...trap/engines/engine.py.228: elapsed: 0:25:29 | left: 0:07:27 [I 2022-10-11 20:23:06] ...trap/engines/engine.py.229: process: 0.86613 | load: 0.00317 [I 2022-10-11 20:23:06] ...trap/engines/engine.py.230: loss: 0.13506 [I 2022-10-11 20:23:06] ...optimizers/trijoint.py.151: total_norm: 16.283051 [I 2022-10-11 20:23:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1790/2304 [I 2022-10-11 20:23:14] ...trap/engines/engine.py.228: elapsed: 0:25:37 | left: 0:07:20 [I 2022-10-11 20:23:14] ...trap/engines/engine.py.229: process: 0.85177 | load: 0.00313 [I 2022-10-11 20:23:14] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-11 20:23:14] ...optimizers/trijoint.py.151: total_norm: 17.622274 [I 2022-10-11 20:23:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1800/2304 [I 2022-10-11 20:23:23] ...trap/engines/engine.py.228: elapsed: 0:25:46 | left: 0:07:09 [I 2022-10-11 20:23:23] ...trap/engines/engine.py.229: process: 0.85706 | load: 0.00314 [I 2022-10-11 20:23:23] ...trap/engines/engine.py.230: loss: 0.10337 [I 2022-10-11 20:23:23] ...optimizers/trijoint.py.151: total_norm: 18.145967 [I 2022-10-11 20:23:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1810/2304 [I 2022-10-11 20:23:32] ...trap/engines/engine.py.228: elapsed: 0:25:55 | left: 0:07:04 [I 2022-10-11 20:23:32] ...trap/engines/engine.py.229: process: 0.87358 | load: 0.00859 [I 2022-10-11 20:23:32] ...trap/engines/engine.py.230: loss: 0.09300 [I 2022-10-11 20:23:32] ...optimizers/trijoint.py.151: total_norm: 26.102560 [I 2022-10-11 20:23:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1820/2304 [I 2022-10-11 20:23:40] ...trap/engines/engine.py.228: elapsed: 0:26:03 | left: 0:06:50 [I 2022-10-11 20:23:40] ...trap/engines/engine.py.229: process: 0.84384 | load: 0.00312 [I 2022-10-11 20:23:40] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-11 20:23:40] ...optimizers/trijoint.py.151: total_norm: 17.114786 [I 2022-10-11 20:23:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1830/2304 [I 2022-10-11 20:23:49] ...trap/engines/engine.py.228: elapsed: 0:26:12 | left: 0:06:43 [I 2022-10-11 20:23:49] ...trap/engines/engine.py.229: process: 0.84220 | load: 0.00314 [I 2022-10-11 20:23:49] ...trap/engines/engine.py.230: loss: 0.17983 [I 2022-10-11 20:23:49] ...optimizers/trijoint.py.151: total_norm: 26.573036 [I 2022-10-11 20:23:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1840/2304 [I 2022-10-11 20:23:57] ...trap/engines/engine.py.228: elapsed: 0:26:20 | left: 0:06:36 [I 2022-10-11 20:23:57] ...trap/engines/engine.py.229: process: 0.87084 | load: 0.00314 [I 2022-10-11 20:23:57] ...trap/engines/engine.py.230: loss: 0.14714 [I 2022-10-11 20:23:57] ...optimizers/trijoint.py.151: total_norm: 22.575912 [I 2022-10-11 20:24:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1850/2304 [I 2022-10-11 20:24:06] ...trap/engines/engine.py.228: elapsed: 0:26:29 | left: 0:06:29 [I 2022-10-11 20:24:06] ...trap/engines/engine.py.229: process: 0.85833 | load: 0.00313 [I 2022-10-11 20:24:06] ...trap/engines/engine.py.230: loss: 0.08347 [I 2022-10-11 20:24:06] ...optimizers/trijoint.py.151: total_norm: 11.288318 [I 2022-10-11 20:24:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1860/2304 [I 2022-10-11 20:24:14] ...trap/engines/engine.py.228: elapsed: 0:26:37 | left: 0:06:21 [I 2022-10-11 20:24:14] ...trap/engines/engine.py.229: process: 0.86445 | load: 0.00314 [I 2022-10-11 20:24:14] ...trap/engines/engine.py.230: loss: 0.11272 [I 2022-10-11 20:24:14] ...optimizers/trijoint.py.151: total_norm: 9.247458 [I 2022-10-11 20:24:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1870/2304 [I 2022-10-11 20:24:23] ...trap/engines/engine.py.228: elapsed: 0:26:46 | left: 0:06:13 [I 2022-10-11 20:24:23] ...trap/engines/engine.py.229: process: 0.86029 | load: 0.00311 [I 2022-10-11 20:24:23] ...trap/engines/engine.py.230: loss: 0.10016 [I 2022-10-11 20:24:23] ...optimizers/trijoint.py.151: total_norm: 19.823505 [I 2022-10-11 20:24:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1880/2304 [I 2022-10-11 20:24:32] ...trap/engines/engine.py.228: elapsed: 0:26:55 | left: 0:06:03 [I 2022-10-11 20:24:32] ...trap/engines/engine.py.229: process: 0.85041 | load: 0.00312 [I 2022-10-11 20:24:32] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-11 20:24:32] ...optimizers/trijoint.py.151: total_norm: 30.351492 [I 2022-10-11 20:24:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1890/2304 [I 2022-10-11 20:24:40] ...trap/engines/engine.py.228: elapsed: 0:27:03 | left: 0:06:05 [I 2022-10-11 20:24:40] ...trap/engines/engine.py.229: process: 0.89277 | load: 0.00397 [I 2022-10-11 20:24:40] ...trap/engines/engine.py.230: loss: 0.09643 [I 2022-10-11 20:24:40] ...optimizers/trijoint.py.151: total_norm: 11.110385 [I 2022-10-11 20:24:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1900/2304 [I 2022-10-11 20:24:49] ...trap/engines/engine.py.228: elapsed: 0:27:12 | left: 0:05:57 [I 2022-10-11 20:24:49] ...trap/engines/engine.py.229: process: 0.87551 | load: 0.00395 [I 2022-10-11 20:24:49] ...trap/engines/engine.py.230: loss: 0.14246 [I 2022-10-11 20:24:49] ...optimizers/trijoint.py.151: total_norm: 10.098889 [I 2022-10-11 20:24:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 1910/2304 [I 2022-10-11 20:24:58] ...trap/engines/engine.py.228: elapsed: 0:27:21 | left: 0:05:47 [I 2022-10-11 20:24:58] ...trap/engines/engine.py.229: process: 0.87687 | load: 0.00396 [I 2022-10-11 20:24:58] ...trap/engines/engine.py.230: loss: 0.08683 [I 2022-10-11 20:24:58] ...optimizers/trijoint.py.151: total_norm: 13.135111 [I 2022-10-11 20:25:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1920/2304 [I 2022-10-11 20:25:07] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:32 [I 2022-10-11 20:25:07] ...trap/engines/engine.py.229: process: 0.84874 | load: 0.00313 [I 2022-10-11 20:25:07] ...trap/engines/engine.py.230: loss: 0.19458 [I 2022-10-11 20:25:07] ...optimizers/trijoint.py.151: total_norm: 16.470985 [I 2022-10-11 20:25:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 1930/2304 [I 2022-10-11 20:25:16] ...trap/engines/engine.py.228: elapsed: 0:27:38 | left: 0:05:21 [I 2022-10-11 20:25:16] ...trap/engines/engine.py.229: process: 0.85012 | load: 0.00313 [I 2022-10-11 20:25:16] ...trap/engines/engine.py.230: loss: 0.16231 [I 2022-10-11 20:25:16] ...optimizers/trijoint.py.151: total_norm: 20.210741 [I 2022-10-11 20:25:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 1940/2304 [I 2022-10-11 20:25:24] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:11 [I 2022-10-11 20:25:24] ...trap/engines/engine.py.229: process: 0.84764 | load: 0.00313 [I 2022-10-11 20:25:24] ...trap/engines/engine.py.230: loss: 0.20297 [I 2022-10-11 20:25:24] ...optimizers/trijoint.py.151: total_norm: 15.973733 [I 2022-10-11 20:25:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 1950/2304 [I 2022-10-11 20:25:33] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:03 [I 2022-10-11 20:25:33] ...trap/engines/engine.py.229: process: 0.86423 | load: 0.00324 [I 2022-10-11 20:25:33] ...trap/engines/engine.py.230: loss: 0.10199 [I 2022-10-11 20:25:33] ...optimizers/trijoint.py.151: total_norm: 18.232336 [I 2022-10-11 20:25:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 1960/2304 [I 2022-10-11 20:25:41] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:54 [I 2022-10-11 20:25:41] ...trap/engines/engine.py.229: process: 0.86048 | load: 0.00312 [I 2022-10-11 20:25:41] ...trap/engines/engine.py.230: loss: 0.10056 [I 2022-10-11 20:25:41] ...optimizers/trijoint.py.151: total_norm: 32.258717 [I 2022-10-11 20:25:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 1970/2304 [I 2022-10-11 20:25:50] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:45 [I 2022-10-11 20:25:50] ...trap/engines/engine.py.229: process: 0.84647 | load: 0.00315 [I 2022-10-11 20:25:50] ...trap/engines/engine.py.230: loss: 0.13506 [I 2022-10-11 20:25:50] ...optimizers/trijoint.py.151: total_norm: 10.388631 [I 2022-10-11 20:25:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1980/2304 [I 2022-10-11 20:25:59] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:44 [I 2022-10-11 20:25:59] ...trap/engines/engine.py.229: process: 0.88995 | load: 0.00397 [I 2022-10-11 20:25:59] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-11 20:25:59] ...optimizers/trijoint.py.151: total_norm: 19.201725 [I 2022-10-11 20:26:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1990/2304 [I 2022-10-11 20:26:07] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:36 [I 2022-10-11 20:26:07] ...trap/engines/engine.py.229: process: 0.87704 | load: 0.00397 [I 2022-10-11 20:26:07] ...trap/engines/engine.py.230: loss: 0.13956 [I 2022-10-11 20:26:07] ...optimizers/trijoint.py.151: total_norm: 17.311209 [I 2022-10-11 20:26:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 2000/2304 [I 2022-10-11 20:26:16] ...trap/engines/engine.py.228: elapsed: 0:28:39 | left: 0:04:26 [I 2022-10-11 20:26:16] ...trap/engines/engine.py.229: process: 0.86115 | load: 0.00304 [I 2022-10-11 20:26:16] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-11 20:26:16] ...optimizers/trijoint.py.151: total_norm: 22.475386 [I 2022-10-11 20:26:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 2010/2304 [I 2022-10-11 20:26:25] ...trap/engines/engine.py.228: elapsed: 0:28:48 | left: 0:04:08 [I 2022-10-11 20:26:25] ...trap/engines/engine.py.229: process: 0.83219 | load: 0.00299 [I 2022-10-11 20:26:25] ...trap/engines/engine.py.230: loss: 0.12375 [I 2022-10-11 20:26:25] ...optimizers/trijoint.py.151: total_norm: 7.304337 [I 2022-10-11 20:26:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 2020/2304 [I 2022-10-11 20:26:33] ...trap/engines/engine.py.228: elapsed: 0:28:56 | left: 0:03:59 [I 2022-10-11 20:26:33] ...trap/engines/engine.py.229: process: 0.84577 | load: 0.00298 [I 2022-10-11 20:26:33] ...trap/engines/engine.py.230: loss: 0.10065 [I 2022-10-11 20:26:33] ...optimizers/trijoint.py.151: total_norm: 10.061187 [I 2022-10-11 20:26:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 2030/2304 [I 2022-10-11 20:26:42] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:51 [I 2022-10-11 20:26:42] ...trap/engines/engine.py.229: process: 0.86193 | load: 0.00297 [I 2022-10-11 20:26:42] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-11 20:26:42] ...optimizers/trijoint.py.151: total_norm: 19.438484 [I 2022-10-11 20:26:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 2040/2304 [I 2022-10-11 20:26:50] ...trap/engines/engine.py.228: elapsed: 0:29:13 | left: 0:03:43 [I 2022-10-11 20:26:50] ...trap/engines/engine.py.229: process: 0.85384 | load: 0.00295 [I 2022-10-11 20:26:50] ...trap/engines/engine.py.230: loss: 0.14087 [I 2022-10-11 20:26:50] ...optimizers/trijoint.py.151: total_norm: 18.870392 [I 2022-10-11 20:26:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 2050/2304 [I 2022-10-11 20:26:59] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:35 [I 2022-10-11 20:26:59] ...trap/engines/engine.py.229: process: 0.85264 | load: 0.00296 [I 2022-10-11 20:26:59] ...trap/engines/engine.py.230: loss: 0.15829 [I 2022-10-11 20:26:59] ...optimizers/trijoint.py.151: total_norm: 37.805237 [I 2022-10-11 20:27:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 2060/2304 [I 2022-10-11 20:27:07] ...trap/engines/engine.py.228: elapsed: 0:29:30 | left: 0:03:26 [I 2022-10-11 20:27:07] ...trap/engines/engine.py.229: process: 0.84358 | load: 0.00297 [I 2022-10-11 20:27:07] ...trap/engines/engine.py.230: loss: 0.10516 [I 2022-10-11 20:27:07] ...optimizers/trijoint.py.151: total_norm: 12.609662 [I 2022-10-11 20:27:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 2070/2304 [I 2022-10-11 20:27:15] ...trap/engines/engine.py.228: elapsed: 0:29:38 | left: 0:03:16 [I 2022-10-11 20:27:15] ...trap/engines/engine.py.229: process: 0.83529 | load: 0.00301 [I 2022-10-11 20:27:15] ...trap/engines/engine.py.230: loss: 0.14167 [I 2022-10-11 20:27:15] ...optimizers/trijoint.py.151: total_norm: 48.420193 [I 2022-10-11 20:27:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 2080/2304 [I 2022-10-11 20:27:24] ...trap/engines/engine.py.228: elapsed: 0:29:47 | left: 0:03:09 [I 2022-10-11 20:27:24] ...trap/engines/engine.py.229: process: 0.83953 | load: 0.00303 [I 2022-10-11 20:27:24] ...trap/engines/engine.py.230: loss: 0.07397 [I 2022-10-11 20:27:24] ...optimizers/trijoint.py.151: total_norm: 22.568426 [I 2022-10-11 20:27:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 2090/2304 [I 2022-10-11 20:27:32] ...trap/engines/engine.py.228: elapsed: 0:29:55 | left: 0:03:01 [I 2022-10-11 20:27:32] ...trap/engines/engine.py.229: process: 0.84993 | load: 0.00301 [I 2022-10-11 20:27:32] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-11 20:27:32] ...optimizers/trijoint.py.151: total_norm: 26.290384 [I 2022-10-11 20:27:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 2100/2304 [I 2022-10-11 20:27:41] ...trap/engines/engine.py.228: elapsed: 0:30:04 | left: 0:02:53 [I 2022-10-11 20:27:41] ...trap/engines/engine.py.229: process: 0.85390 | load: 0.00302 [I 2022-10-11 20:27:41] ...trap/engines/engine.py.230: loss: 0.12239 [I 2022-10-11 20:27:41] ...optimizers/trijoint.py.151: total_norm: 27.240589 [I 2022-10-11 20:27:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 2110/2304 [I 2022-10-11 20:27:49] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:45 [I 2022-10-11 20:27:49] ...trap/engines/engine.py.229: process: 0.85536 | load: 0.00304 [I 2022-10-11 20:27:49] ...trap/engines/engine.py.230: loss: 0.11445 [I 2022-10-11 20:27:49] ...optimizers/trijoint.py.151: total_norm: 26.287271 [I 2022-10-11 20:27:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 2120/2304 [I 2022-10-11 20:27:58] ...trap/engines/engine.py.228: elapsed: 0:30:21 | left: 0:02:34 [I 2022-10-11 20:27:58] ...trap/engines/engine.py.229: process: 0.82927 | load: 0.00304 [I 2022-10-11 20:27:58] ...trap/engines/engine.py.230: loss: 0.07702 [I 2022-10-11 20:27:58] ...optimizers/trijoint.py.151: total_norm: 9.249298 [I 2022-10-11 20:28:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 2130/2304 [I 2022-10-11 20:28:06] ...trap/engines/engine.py.228: elapsed: 0:30:29 | left: 0:02:26 [I 2022-10-11 20:28:06] ...trap/engines/engine.py.229: process: 0.84261 | load: 0.00297 [I 2022-10-11 20:28:06] ...trap/engines/engine.py.230: loss: 0.14134 [I 2022-10-11 20:28:06] ...optimizers/trijoint.py.151: total_norm: 30.097929 [I 2022-10-11 20:28:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 2140/2304 [I 2022-10-11 20:28:15] ...trap/engines/engine.py.228: elapsed: 0:30:38 | left: 0:02:18 [I 2022-10-11 20:28:15] ...trap/engines/engine.py.229: process: 0.84522 | load: 0.00300 [I 2022-10-11 20:28:15] ...trap/engines/engine.py.230: loss: 0.17331 [I 2022-10-11 20:28:15] ...optimizers/trijoint.py.151: total_norm: 22.858339 [I 2022-10-11 20:28:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 2150/2304 [I 2022-10-11 20:28:23] ...trap/engines/engine.py.228: elapsed: 0:30:46 | left: 0:02:10 [I 2022-10-11 20:28:23] ...trap/engines/engine.py.229: process: 0.85663 | load: 0.00313 [I 2022-10-11 20:28:23] ...trap/engines/engine.py.230: loss: 0.08832 [I 2022-10-11 20:28:23] ...optimizers/trijoint.py.151: total_norm: 16.219213 [I 2022-10-11 20:28:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 2160/2304 [I 2022-10-11 20:28:32] ...trap/engines/engine.py.228: elapsed: 0:30:55 | left: 0:02:03 [I 2022-10-11 20:28:32] ...trap/engines/engine.py.229: process: 0.87967 | load: 0.00314 [I 2022-10-11 20:28:32] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-11 20:28:32] ...optimizers/trijoint.py.151: total_norm: 30.851152 [I 2022-10-11 20:28:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 2170/2304 [I 2022-10-11 20:28:40] ...trap/engines/engine.py.228: elapsed: 0:31:03 | left: 0:01:53 [I 2022-10-11 20:28:40] ...trap/engines/engine.py.229: process: 0.85707 | load: 0.00314 [I 2022-10-11 20:28:40] ...trap/engines/engine.py.230: loss: 0.09011 [I 2022-10-11 20:28:40] ...optimizers/trijoint.py.151: total_norm: 10.085004 [I 2022-10-11 20:28:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 2180/2304 [I 2022-10-11 20:28:49] ...trap/engines/engine.py.228: elapsed: 0:31:12 | left: 0:01:45 [I 2022-10-11 20:28:49] ...trap/engines/engine.py.229: process: 0.85599 | load: 0.00314 [I 2022-10-11 20:28:49] ...trap/engines/engine.py.230: loss: 0.17347 [I 2022-10-11 20:28:49] ...optimizers/trijoint.py.151: total_norm: 18.447607 [I 2022-10-11 20:28:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 2190/2304 [I 2022-10-11 20:28:57] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:37 [I 2022-10-11 20:28:57] ...trap/engines/engine.py.229: process: 0.85704 | load: 0.00313 [I 2022-10-11 20:28:57] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-11 20:28:57] ...optimizers/trijoint.py.151: total_norm: 12.593843 [I 2022-10-11 20:29:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 2200/2304 [I 2022-10-11 20:29:06] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:28 [I 2022-10-11 20:29:06] ...trap/engines/engine.py.229: process: 0.84891 | load: 0.00313 [I 2022-10-11 20:29:06] ...trap/engines/engine.py.230: loss: 0.08463 [I 2022-10-11 20:29:06] ...optimizers/trijoint.py.151: total_norm: 14.970384 [I 2022-10-11 20:29:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 2210/2304 [I 2022-10-11 20:29:14] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:19 [I 2022-10-11 20:29:14] ...trap/engines/engine.py.229: process: 0.84886 | load: 0.00347 [I 2022-10-11 20:29:14] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-11 20:29:14] ...optimizers/trijoint.py.151: total_norm: 16.344648 [I 2022-10-11 20:29:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 2220/2304 [I 2022-10-11 20:29:23] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:11 [I 2022-10-11 20:29:23] ...trap/engines/engine.py.229: process: 0.86428 | load: 0.00315 [I 2022-10-11 20:29:23] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-11 20:29:23] ...optimizers/trijoint.py.151: total_norm: 11.993048 [I 2022-10-11 20:29:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 2230/2304 [I 2022-10-11 20:29:31] ...trap/engines/engine.py.228: elapsed: 0:31:54 | left: 0:01:03 [I 2022-10-11 20:29:31] ...trap/engines/engine.py.229: process: 0.85902 | load: 0.00313 [I 2022-10-11 20:29:31] ...trap/engines/engine.py.230: loss: 0.19972 [I 2022-10-11 20:29:31] ...optimizers/trijoint.py.151: total_norm: 30.265972 [I 2022-10-11 20:29:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 2240/2304 [I 2022-10-11 20:29:40] ...trap/engines/engine.py.228: elapsed: 0:32:03 | left: 0:00:54 [I 2022-10-11 20:29:40] ...trap/engines/engine.py.229: process: 0.86217 | load: 0.00311 [I 2022-10-11 20:29:40] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-11 20:29:40] ...optimizers/trijoint.py.151: total_norm: 27.010019 [I 2022-10-11 20:29:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 2250/2304 [I 2022-10-11 20:29:49] ...trap/engines/engine.py.228: elapsed: 0:32:12 | left: 0:00:46 [I 2022-10-11 20:29:49] ...trap/engines/engine.py.229: process: 0.86525 | load: 0.00315 [I 2022-10-11 20:29:49] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-11 20:29:49] ...optimizers/trijoint.py.151: total_norm: 12.434911 [I 2022-10-11 20:29:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 2260/2304 [I 2022-10-11 20:29:57] ...trap/engines/engine.py.228: elapsed: 0:32:20 | left: 0:00:37 [I 2022-10-11 20:29:57] ...trap/engines/engine.py.229: process: 0.86256 | load: 0.00314 [I 2022-10-11 20:29:57] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-11 20:29:57] ...optimizers/trijoint.py.151: total_norm: 10.702028 [I 2022-10-11 20:30:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 2270/2304 [I 2022-10-11 20:30:06] ...trap/engines/engine.py.228: elapsed: 0:32:29 | left: 0:00:29 [I 2022-10-11 20:30:06] ...trap/engines/engine.py.229: process: 0.85422 | load: 0.00311 [I 2022-10-11 20:30:06] ...trap/engines/engine.py.230: loss: 0.10521 [I 2022-10-11 20:30:06] ...optimizers/trijoint.py.151: total_norm: 29.046188 [I 2022-10-11 20:30:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 2280/2304 [I 2022-10-11 20:30:14] ...trap/engines/engine.py.228: elapsed: 0:32:37 | left: 0:00:20 [I 2022-10-11 20:30:14] ...trap/engines/engine.py.229: process: 0.87011 | load: 0.00315 [I 2022-10-11 20:30:14] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-11 20:30:14] ...optimizers/trijoint.py.151: total_norm: 11.259013 [I 2022-10-11 20:30:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 2290/2304 [I 2022-10-11 20:30:23] ...trap/engines/engine.py.228: elapsed: 0:32:46 | left: 0:00:11 [I 2022-10-11 20:30:23] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00313 [I 2022-10-11 20:30:23] ...trap/engines/engine.py.230: loss: 0.09555 [I 2022-10-11 20:30:23] ...optimizers/trijoint.py.151: total_norm: 7.770337 [I 2022-10-11 20:30:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 2300/2304 [I 2022-10-11 20:30:32] ...trap/engines/engine.py.228: elapsed: 0:32:55 | left: 0:00:03 [I 2022-10-11 20:30:32] ...trap/engines/engine.py.229: process: 0.84694 | load: 0.00019 [I 2022-10-11 20:30:32] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-11 20:30:32] ...optimizers/trijoint.py.151: total_norm: 24.372856 [I 2022-10-11 20:30:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 2304/2304 [I 2022-10-11 20:30:35] ...trap/engines/engine.py.228: elapsed: 0:32:58 | left: 0:00:00 [I 2022-10-11 20:30:35] ...trap/engines/engine.py.229: process: 0.86034 | load: 0.00022 [I 2022-10-11 20:30:35] ...trap/engines/engine.py.230: loss: 0.10290 [I 2022-10-11 20:30:35] ...optimizers/trijoint.py.151: total_norm: 25.477306 [S 2022-10-11 20:30:36] ...trap/engines/engine.py.129: train_epoch.epoch: 117 [S 2022-10-11 20:30:36] ...trap/engines/engine.py.129: train_epoch.loss: 0.11843753171975598 [I 2022-10-11 20:30:36] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 20:30:38] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 117 [I 2022-10-11 20:30:38] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 20:30:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 20:30:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 20:30:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 20:30:45] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 20:30:45] ...trap/engines/engine.py.321: eval: epoch 117 | batch 0/510 [I 2022-10-11 20:30:45] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:58:12 [I 2022-10-11 20:30:45] ...trap/engines/engine.py.326: process: 6.84824 | load: 4.26318 [I 2022-10-11 20:30:50] ...trap/engines/engine.py.321: eval: epoch 117 | batch 10/510 [I 2022-10-11 20:30:50] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:26 [I 2022-10-11 20:30:50] ...trap/engines/engine.py.326: process: 0.45539 | load: 0.00020 [I 2022-10-11 20:30:54] ...trap/engines/engine.py.321: eval: epoch 117 | batch 20/510 [I 2022-10-11 20:30:54] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:14 [I 2022-10-11 20:30:54] ...trap/engines/engine.py.326: process: 0.41902 | load: 0.00020 [I 2022-10-11 20:30:59] ...trap/engines/engine.py.321: eval: epoch 117 | batch 30/510 [I 2022-10-11 20:30:59] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:37 [I 2022-10-11 20:30:59] ...trap/engines/engine.py.326: process: 0.44900 | load: 0.00020 [I 2022-10-11 20:31:03] ...trap/engines/engine.py.321: eval: epoch 117 | batch 40/510 [I 2022-10-11 20:31:03] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:32 [I 2022-10-11 20:31:03] ...trap/engines/engine.py.326: process: 0.44435 | load: 0.00020 [I 2022-10-11 20:31:08] ...trap/engines/engine.py.321: eval: epoch 117 | batch 50/510 [I 2022-10-11 20:31:08] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:24 [I 2022-10-11 20:31:08] ...trap/engines/engine.py.326: process: 0.42377 | load: 0.00020 [I 2022-10-11 20:31:12] ...trap/engines/engine.py.321: eval: epoch 117 | batch 60/510 [I 2022-10-11 20:31:12] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:20 [I 2022-10-11 20:31:12] ...trap/engines/engine.py.326: process: 0.42546 | load: 0.00020 [I 2022-10-11 20:31:16] ...trap/engines/engine.py.321: eval: epoch 117 | batch 70/510 [I 2022-10-11 20:31:16] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:11 [I 2022-10-11 20:31:16] ...trap/engines/engine.py.326: process: 0.42495 | load: 0.00020 [I 2022-10-11 20:31:21] ...trap/engines/engine.py.321: eval: epoch 117 | batch 80/510 [I 2022-10-11 20:31:21] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:07 [I 2022-10-11 20:31:21] ...trap/engines/engine.py.326: process: 0.43387 | load: 0.00019 [I 2022-10-11 20:31:25] ...trap/engines/engine.py.321: eval: epoch 117 | batch 90/510 [I 2022-10-11 20:31:25] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:03:07 [I 2022-10-11 20:31:25] ...trap/engines/engine.py.326: process: 0.44076 | load: 0.00022 [I 2022-10-11 20:31:30] ...trap/engines/engine.py.321: eval: epoch 117 | batch 100/510 [I 2022-10-11 20:31:30] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:02:52 [I 2022-10-11 20:31:30] ...trap/engines/engine.py.326: process: 0.31302 | load: 0.00020 [I 2022-10-11 20:31:33] ...trap/engines/engine.py.321: eval: epoch 117 | batch 110/510 [I 2022-10-11 20:31:33] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:02:12 [I 2022-10-11 20:31:33] ...trap/engines/engine.py.326: process: 0.32280 | load: 0.00023 [I 2022-10-11 20:31:36] ...trap/engines/engine.py.321: eval: epoch 117 | batch 120/510 [I 2022-10-11 20:31:36] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:02:05 [I 2022-10-11 20:31:36] ...trap/engines/engine.py.326: process: 0.32100 | load: 0.00021 [I 2022-10-11 20:31:39] ...trap/engines/engine.py.321: eval: epoch 117 | batch 130/510 [I 2022-10-11 20:31:39] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:02:01 [I 2022-10-11 20:31:39] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00022 [I 2022-10-11 20:31:42] ...trap/engines/engine.py.321: eval: epoch 117 | batch 140/510 [I 2022-10-11 20:31:42] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:59 [I 2022-10-11 20:31:42] ...trap/engines/engine.py.326: process: 0.32479 | load: 0.00021 [I 2022-10-11 20:31:46] ...trap/engines/engine.py.321: eval: epoch 117 | batch 150/510 [I 2022-10-11 20:31:46] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:55 [I 2022-10-11 20:31:46] ...trap/engines/engine.py.326: process: 0.32249 | load: 0.00021 [I 2022-10-11 20:31:49] ...trap/engines/engine.py.321: eval: epoch 117 | batch 160/510 [I 2022-10-11 20:31:49] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:52 [I 2022-10-11 20:31:49] ...trap/engines/engine.py.326: process: 0.32221 | load: 0.00021 [I 2022-10-11 20:31:52] ...trap/engines/engine.py.321: eval: epoch 117 | batch 170/510 [I 2022-10-11 20:31:52] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:49 [I 2022-10-11 20:31:52] ...trap/engines/engine.py.326: process: 0.31789 | load: 0.00021 [I 2022-10-11 20:31:55] ...trap/engines/engine.py.321: eval: epoch 117 | batch 180/510 [I 2022-10-11 20:31:55] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:46 [I 2022-10-11 20:31:55] ...trap/engines/engine.py.326: process: 0.32441 | load: 0.00021 [I 2022-10-11 20:31:59] ...trap/engines/engine.py.321: eval: epoch 117 | batch 190/510 [I 2022-10-11 20:31:59] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:42 [I 2022-10-11 20:31:59] ...trap/engines/engine.py.326: process: 0.31197 | load: 0.00022 [I 2022-10-11 20:32:02] ...trap/engines/engine.py.321: eval: epoch 117 | batch 200/510 [I 2022-10-11 20:32:02] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:39 [I 2022-10-11 20:32:02] ...trap/engines/engine.py.326: process: 0.32781 | load: 0.00022 [I 2022-10-11 20:32:05] ...trap/engines/engine.py.321: eval: epoch 117 | batch 210/510 [I 2022-10-11 20:32:05] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:36 [I 2022-10-11 20:32:05] ...trap/engines/engine.py.326: process: 0.32143 | load: 0.00021 [I 2022-10-11 20:32:08] ...trap/engines/engine.py.321: eval: epoch 117 | batch 220/510 [I 2022-10-11 20:32:08] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:32 [I 2022-10-11 20:32:08] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00022 [I 2022-10-11 20:32:11] ...trap/engines/engine.py.321: eval: epoch 117 | batch 230/510 [I 2022-10-11 20:32:11] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:30 [I 2022-10-11 20:32:11] ...trap/engines/engine.py.326: process: 0.31871 | load: 0.00020 [I 2022-10-11 20:32:15] ...trap/engines/engine.py.321: eval: epoch 117 | batch 240/510 [I 2022-10-11 20:32:15] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:27 [I 2022-10-11 20:32:15] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00022 [I 2022-10-11 20:32:18] ...trap/engines/engine.py.321: eval: epoch 117 | batch 250/510 [I 2022-10-11 20:32:18] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:23 [I 2022-10-11 20:32:18] ...trap/engines/engine.py.326: process: 0.31615 | load: 0.00021 [I 2022-10-11 20:32:21] ...trap/engines/engine.py.321: eval: epoch 117 | batch 260/510 [I 2022-10-11 20:32:21] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:19 [I 2022-10-11 20:32:21] ...trap/engines/engine.py.326: process: 0.31605 | load: 0.00019 [I 2022-10-11 20:32:24] ...trap/engines/engine.py.321: eval: epoch 117 | batch 270/510 [I 2022-10-11 20:32:24] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:01:16 [I 2022-10-11 20:32:24] ...trap/engines/engine.py.326: process: 0.31936 | load: 0.00020 [I 2022-10-11 20:32:27] ...trap/engines/engine.py.321: eval: epoch 117 | batch 280/510 [I 2022-10-11 20:32:27] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:12 [I 2022-10-11 20:32:27] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00020 [I 2022-10-11 20:32:31] ...trap/engines/engine.py.321: eval: epoch 117 | batch 290/510 [I 2022-10-11 20:32:31] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:09 [I 2022-10-11 20:32:31] ...trap/engines/engine.py.326: process: 0.31461 | load: 0.00021 [I 2022-10-11 20:32:34] ...trap/engines/engine.py.321: eval: epoch 117 | batch 300/510 [I 2022-10-11 20:32:34] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:01:06 [I 2022-10-11 20:32:34] ...trap/engines/engine.py.326: process: 0.31295 | load: 0.00020 [I 2022-10-11 20:32:37] ...trap/engines/engine.py.321: eval: epoch 117 | batch 310/510 [I 2022-10-11 20:32:37] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:01:03 [I 2022-10-11 20:32:37] ...trap/engines/engine.py.326: process: 0.31807 | load: 0.00019 [I 2022-10-11 20:32:40] ...trap/engines/engine.py.321: eval: epoch 117 | batch 320/510 [I 2022-10-11 20:32:40] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:01:00 [I 2022-10-11 20:32:40] ...trap/engines/engine.py.326: process: 0.31560 | load: 0.00020 [I 2022-10-11 20:32:43] ...trap/engines/engine.py.321: eval: epoch 117 | batch 330/510 [I 2022-10-11 20:32:43] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:57 [I 2022-10-11 20:32:43] ...trap/engines/engine.py.326: process: 0.32859 | load: 0.00019 [I 2022-10-11 20:32:47] ...trap/engines/engine.py.321: eval: epoch 117 | batch 340/510 [I 2022-10-11 20:32:47] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:54 [I 2022-10-11 20:32:47] ...trap/engines/engine.py.326: process: 0.32146 | load: 0.00020 [I 2022-10-11 20:32:50] ...trap/engines/engine.py.321: eval: epoch 117 | batch 350/510 [I 2022-10-11 20:32:50] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:50 [I 2022-10-11 20:32:50] ...trap/engines/engine.py.326: process: 0.31980 | load: 0.00019 [I 2022-10-11 20:32:53] ...trap/engines/engine.py.321: eval: epoch 117 | batch 360/510 [I 2022-10-11 20:32:53] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:47 [I 2022-10-11 20:32:53] ...trap/engines/engine.py.326: process: 0.31958 | load: 0.00019 [I 2022-10-11 20:32:56] ...trap/engines/engine.py.321: eval: epoch 117 | batch 370/510 [I 2022-10-11 20:32:56] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:44 [I 2022-10-11 20:32:56] ...trap/engines/engine.py.326: process: 0.31666 | load: 0.00019 [I 2022-10-11 20:32:59] ...trap/engines/engine.py.321: eval: epoch 117 | batch 380/510 [I 2022-10-11 20:32:59] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:41 [I 2022-10-11 20:32:59] ...trap/engines/engine.py.326: process: 0.31687 | load: 0.00020 [I 2022-10-11 20:33:03] ...trap/engines/engine.py.321: eval: epoch 117 | batch 390/510 [I 2022-10-11 20:33:03] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:38 [I 2022-10-11 20:33:03] ...trap/engines/engine.py.326: process: 0.31990 | load: 0.00021 [I 2022-10-11 20:33:06] ...trap/engines/engine.py.321: eval: epoch 117 | batch 400/510 [I 2022-10-11 20:33:06] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:34 [I 2022-10-11 20:33:06] ...trap/engines/engine.py.326: process: 0.31338 | load: 0.00021 [I 2022-10-11 20:33:09] ...trap/engines/engine.py.321: eval: epoch 117 | batch 410/510 [I 2022-10-11 20:33:09] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:31 [I 2022-10-11 20:33:09] ...trap/engines/engine.py.326: process: 0.31533 | load: 0.00020 [I 2022-10-11 20:33:12] ...trap/engines/engine.py.321: eval: epoch 117 | batch 420/510 [I 2022-10-11 20:33:12] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:28 [I 2022-10-11 20:33:12] ...trap/engines/engine.py.326: process: 0.32186 | load: 0.00020 [I 2022-10-11 20:33:15] ...trap/engines/engine.py.321: eval: epoch 117 | batch 430/510 [I 2022-10-11 20:33:15] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:25 [I 2022-10-11 20:33:15] ...trap/engines/engine.py.326: process: 0.32633 | load: 0.00019 [I 2022-10-11 20:33:18] ...trap/engines/engine.py.321: eval: epoch 117 | batch 440/510 [I 2022-10-11 20:33:18] ...trap/engines/engine.py.325: elapsed: 0:02:40 | left: 0:00:22 [I 2022-10-11 20:33:18] ...trap/engines/engine.py.326: process: 0.31362 | load: 0.00020 [I 2022-10-11 20:33:22] ...trap/engines/engine.py.321: eval: epoch 117 | batch 450/510 [I 2022-10-11 20:33:22] ...trap/engines/engine.py.325: elapsed: 0:02:43 | left: 0:00:19 [I 2022-10-11 20:33:22] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00020 [I 2022-10-11 20:33:25] ...trap/engines/engine.py.321: eval: epoch 117 | batch 460/510 [I 2022-10-11 20:33:25] ...trap/engines/engine.py.325: elapsed: 0:02:46 | left: 0:00:15 [I 2022-10-11 20:33:25] ...trap/engines/engine.py.326: process: 0.32157 | load: 0.00019 [I 2022-10-11 20:33:28] ...trap/engines/engine.py.321: eval: epoch 117 | batch 470/510 [I 2022-10-11 20:33:28] ...trap/engines/engine.py.325: elapsed: 0:02:49 | left: 0:00:12 [I 2022-10-11 20:33:28] ...trap/engines/engine.py.326: process: 0.32370 | load: 0.00019 [I 2022-10-11 20:33:31] ...trap/engines/engine.py.321: eval: epoch 117 | batch 480/510 [I 2022-10-11 20:33:31] ...trap/engines/engine.py.325: elapsed: 0:02:52 | left: 0:00:09 [I 2022-10-11 20:33:31] ...trap/engines/engine.py.326: process: 0.32245 | load: 0.00020 [I 2022-10-11 20:33:34] ...trap/engines/engine.py.321: eval: epoch 117 | batch 490/510 [I 2022-10-11 20:33:34] ...trap/engines/engine.py.325: elapsed: 0:02:56 | left: 0:00:06 [I 2022-10-11 20:33:34] ...trap/engines/engine.py.326: process: 0.31695 | load: 0.00020 [I 2022-10-11 20:33:38] ...trap/engines/engine.py.321: eval: epoch 117 | batch 500/510 [I 2022-10-11 20:33:38] ...trap/engines/engine.py.325: elapsed: 0:02:59 | left: 0:00:03 [I 2022-10-11 20:33:38] ...trap/engines/engine.py.326: process: 0.32077 | load: 0.00020 [I 2022-10-11 20:33:41] ...trap/engines/engine.py.321: eval: epoch 117 | batch 510/510 [I 2022-10-11 20:33:41] ...trap/engines/engine.py.325: elapsed: 0:03:02 | left: 0:00:00 [I 2022-10-11 20:33:41] ...trap/engines/engine.py.326: process: 0.31292 | load: 0.00015 [S 2022-10-11 20:33:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 117 [I 2022-10-11 20:33:41] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746000 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909900 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937800 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.741300 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909400 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936200 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012736 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009731 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008875 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013871 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009687 [S 2022-10-11 20:34:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007373 [I 2022-10-11 20:34:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 20:34:16] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 20:34:17] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 20:34:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 20:34:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 20:34:21] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 20:34:25] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 20:34:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 20:34:29] ...trap/engines/engine.py.163: Training model on trainset for epoch 118 [I 2022-10-11 20:34:29] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 20:34:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 0/2304 [I 2022-10-11 20:34:32] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:40 [I 2022-10-11 20:34:32] ...trap/engines/engine.py.229: process: 3.24683 | load: 2.31424 [I 2022-10-11 20:34:32] ...trap/engines/engine.py.230: loss: 0.09323 [I 2022-10-11 20:34:32] ...optimizers/trijoint.py.151: total_norm: 9.229915 [I 2022-10-11 20:34:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 10/2304 [I 2022-10-11 20:34:41] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:42:21 [I 2022-10-11 20:34:41] ...trap/engines/engine.py.229: process: 0.84235 | load: 0.00449 [I 2022-10-11 20:34:41] ...trap/engines/engine.py.230: loss: 0.08573 [I 2022-10-11 20:34:41] ...optimizers/trijoint.py.151: total_norm: 9.457925 [I 2022-10-11 20:34:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 20/2304 [I 2022-10-11 20:34:49] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:53 [I 2022-10-11 20:34:49] ...trap/engines/engine.py.229: process: 0.87814 | load: 0.00563 [I 2022-10-11 20:34:49] ...trap/engines/engine.py.230: loss: 0.08746 [I 2022-10-11 20:34:49] ...optimizers/trijoint.py.151: total_norm: 19.086025 [I 2022-10-11 20:34:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 30/2304 [I 2022-10-11 20:34:58] ...trap/engines/engine.py.228: elapsed: 0:00:29 | left: 0:32:41 [I 2022-10-11 20:34:58] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00413 [I 2022-10-11 20:34:58] ...trap/engines/engine.py.230: loss: 0.08129 [I 2022-10-11 20:34:58] ...optimizers/trijoint.py.151: total_norm: 16.679392 [I 2022-10-11 20:35:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 40/2304 [I 2022-10-11 20:35:06] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:27 [I 2022-10-11 20:35:06] ...trap/engines/engine.py.229: process: 0.86930 | load: 0.00409 [I 2022-10-11 20:35:06] ...trap/engines/engine.py.230: loss: 0.09202 [I 2022-10-11 20:35:06] ...optimizers/trijoint.py.151: total_norm: 9.397086 [I 2022-10-11 20:35:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 50/2304 [I 2022-10-11 20:35:15] ...trap/engines/engine.py.228: elapsed: 0:00:46 | left: 0:32:20 [I 2022-10-11 20:35:15] ...trap/engines/engine.py.229: process: 0.85943 | load: 0.00378 [I 2022-10-11 20:35:15] ...trap/engines/engine.py.230: loss: 0.10076 [I 2022-10-11 20:35:15] ...optimizers/trijoint.py.151: total_norm: 13.209867 [I 2022-10-11 20:35:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 60/2304 [I 2022-10-11 20:35:24] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:32:12 [I 2022-10-11 20:35:24] ...trap/engines/engine.py.229: process: 0.87321 | load: 0.00388 [I 2022-10-11 20:35:24] ...trap/engines/engine.py.230: loss: 0.08975 [I 2022-10-11 20:35:24] ...optimizers/trijoint.py.151: total_norm: 9.851388 [I 2022-10-11 20:35:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 70/2304 [I 2022-10-11 20:35:32] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:31:57 [I 2022-10-11 20:35:32] ...trap/engines/engine.py.229: process: 0.85424 | load: 0.00397 [I 2022-10-11 20:35:32] ...trap/engines/engine.py.230: loss: 0.10449 [I 2022-10-11 20:35:32] ...optimizers/trijoint.py.151: total_norm: 10.414789 [I 2022-10-11 20:35:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 80/2304 [I 2022-10-11 20:35:41] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:31:52 [I 2022-10-11 20:35:41] ...trap/engines/engine.py.229: process: 0.85198 | load: 0.00343 [I 2022-10-11 20:35:41] ...trap/engines/engine.py.230: loss: 0.09878 [I 2022-10-11 20:35:41] ...optimizers/trijoint.py.151: total_norm: 11.295924 [I 2022-10-11 20:35:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 90/2304 [I 2022-10-11 20:35:49] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:29 [I 2022-10-11 20:35:49] ...trap/engines/engine.py.229: process: 0.85358 | load: 0.00389 [I 2022-10-11 20:35:49] ...trap/engines/engine.py.230: loss: 0.10729 [I 2022-10-11 20:35:49] ...optimizers/trijoint.py.151: total_norm: 14.723023 [I 2022-10-11 20:35:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 100/2304 [I 2022-10-11 20:35:58] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:31:26 [I 2022-10-11 20:35:58] ...trap/engines/engine.py.229: process: 0.86458 | load: 0.00333 [I 2022-10-11 20:35:58] ...trap/engines/engine.py.230: loss: 0.10557 [I 2022-10-11 20:35:58] ...optimizers/trijoint.py.151: total_norm: 23.093319 [I 2022-10-11 20:36:07] ...trap/engines/engine.py.224: train: epoch 118 | batch 110/2304 [I 2022-10-11 20:36:07] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:23 [I 2022-10-11 20:36:07] ...trap/engines/engine.py.229: process: 0.86503 | load: 0.00330 [I 2022-10-11 20:36:07] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-11 20:36:07] ...optimizers/trijoint.py.151: total_norm: 32.907730 [I 2022-10-11 20:36:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 120/2304 [I 2022-10-11 20:36:15] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:25 [I 2022-10-11 20:36:15] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00311 [I 2022-10-11 20:36:15] ...trap/engines/engine.py.230: loss: 0.09294 [I 2022-10-11 20:36:15] ...optimizers/trijoint.py.151: total_norm: 12.452006 [I 2022-10-11 20:36:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 130/2304 [I 2022-10-11 20:36:24] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:31:27 [I 2022-10-11 20:36:24] ...trap/engines/engine.py.229: process: 0.86465 | load: 0.00323 [I 2022-10-11 20:36:24] ...trap/engines/engine.py.230: loss: 0.07020 [I 2022-10-11 20:36:24] ...optimizers/trijoint.py.151: total_norm: 9.051710 [I 2022-10-11 20:36:33] ...trap/engines/engine.py.224: train: epoch 118 | batch 140/2304 [I 2022-10-11 20:36:33] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:31:04 [I 2022-10-11 20:36:33] ...trap/engines/engine.py.229: process: 0.86975 | load: 0.00362 [I 2022-10-11 20:36:33] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-11 20:36:33] ...optimizers/trijoint.py.151: total_norm: 20.508554 [I 2022-10-11 20:36:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 150/2304 [I 2022-10-11 20:36:41] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:40 [I 2022-10-11 20:36:41] ...trap/engines/engine.py.229: process: 0.84313 | load: 0.00314 [I 2022-10-11 20:36:41] ...trap/engines/engine.py.230: loss: 0.09758 [I 2022-10-11 20:36:41] ...optimizers/trijoint.py.151: total_norm: 10.337258 [I 2022-10-11 20:36:50] ...trap/engines/engine.py.224: train: epoch 118 | batch 160/2304 [I 2022-10-11 20:36:50] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:30:52 [I 2022-10-11 20:36:50] ...trap/engines/engine.py.229: process: 0.87508 | load: 0.00343 [I 2022-10-11 20:36:50] ...trap/engines/engine.py.230: loss: 0.09208 [I 2022-10-11 20:36:50] ...optimizers/trijoint.py.151: total_norm: 13.063787 [I 2022-10-11 20:36:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 170/2304 [I 2022-10-11 20:36:59] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:40 [I 2022-10-11 20:36:59] ...trap/engines/engine.py.229: process: 0.85659 | load: 0.00314 [I 2022-10-11 20:36:59] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-11 20:36:59] ...optimizers/trijoint.py.151: total_norm: 10.812306 [I 2022-10-11 20:37:07] ...trap/engines/engine.py.224: train: epoch 118 | batch 180/2304 [I 2022-10-11 20:37:07] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:26 [I 2022-10-11 20:37:07] ...trap/engines/engine.py.229: process: 0.85796 | load: 0.00326 [I 2022-10-11 20:37:07] ...trap/engines/engine.py.230: loss: 0.15126 [I 2022-10-11 20:37:07] ...optimizers/trijoint.py.151: total_norm: 20.323246 [I 2022-10-11 20:37:16] ...trap/engines/engine.py.224: train: epoch 118 | batch 190/2304 [I 2022-10-11 20:37:16] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:18 [I 2022-10-11 20:37:16] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00335 [I 2022-10-11 20:37:16] ...trap/engines/engine.py.230: loss: 0.10256 [I 2022-10-11 20:37:16] ...optimizers/trijoint.py.151: total_norm: 10.757129 [I 2022-10-11 20:37:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 200/2304 [I 2022-10-11 20:37:25] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:58 [I 2022-10-11 20:37:25] ...trap/engines/engine.py.229: process: 0.88411 | load: 0.00398 [I 2022-10-11 20:37:25] ...trap/engines/engine.py.230: loss: 0.09464 [I 2022-10-11 20:37:25] ...optimizers/trijoint.py.151: total_norm: 17.572590 [I 2022-10-11 20:37:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 210/2304 [I 2022-10-11 20:37:34] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:31:06 [I 2022-10-11 20:37:34] ...trap/engines/engine.py.229: process: 0.89516 | load: 0.00396 [I 2022-10-11 20:37:34] ...trap/engines/engine.py.230: loss: 0.09293 [I 2022-10-11 20:37:34] ...optimizers/trijoint.py.151: total_norm: 17.392113 [I 2022-10-11 20:37:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 220/2304 [I 2022-10-11 20:37:42] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:30:57 [I 2022-10-11 20:37:42] ...trap/engines/engine.py.229: process: 0.88659 | load: 0.00413 [I 2022-10-11 20:37:42] ...trap/engines/engine.py.230: loss: 0.13340 [I 2022-10-11 20:37:42] ...optimizers/trijoint.py.151: total_norm: 13.234958 [I 2022-10-11 20:37:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 230/2304 [I 2022-10-11 20:37:51] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:30:47 [I 2022-10-11 20:37:51] ...trap/engines/engine.py.229: process: 0.89817 | load: 0.00398 [I 2022-10-11 20:37:51] ...trap/engines/engine.py.230: loss: 0.16237 [I 2022-10-11 20:37:51] ...optimizers/trijoint.py.151: total_norm: 37.787762 [I 2022-10-11 20:38:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 240/2304 [I 2022-10-11 20:38:00] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:30:32 [I 2022-10-11 20:38:00] ...trap/engines/engine.py.229: process: 0.88841 | load: 0.00433 [I 2022-10-11 20:38:00] ...trap/engines/engine.py.230: loss: 0.25945 [I 2022-10-11 20:38:00] ...optimizers/trijoint.py.151: total_norm: 18.822611 [I 2022-10-11 20:38:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 250/2304 [I 2022-10-11 20:38:09] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:30:09 [I 2022-10-11 20:38:09] ...trap/engines/engine.py.229: process: 0.87513 | load: 0.00386 [I 2022-10-11 20:38:09] ...trap/engines/engine.py.230: loss: 0.09398 [I 2022-10-11 20:38:09] ...optimizers/trijoint.py.151: total_norm: 11.390114 [I 2022-10-11 20:38:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 260/2304 [I 2022-10-11 20:38:18] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:29:54 [I 2022-10-11 20:38:18] ...trap/engines/engine.py.229: process: 0.87749 | load: 0.00375 [I 2022-10-11 20:38:18] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-11 20:38:18] ...optimizers/trijoint.py.151: total_norm: 19.140055 [I 2022-10-11 20:38:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 270/2304 [I 2022-10-11 20:38:27] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:29:34 [I 2022-10-11 20:38:27] ...trap/engines/engine.py.229: process: 0.87451 | load: 0.00377 [I 2022-10-11 20:38:27] ...trap/engines/engine.py.230: loss: 0.08724 [I 2022-10-11 20:38:27] ...optimizers/trijoint.py.151: total_norm: 16.197845 [I 2022-10-11 20:38:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 280/2304 [I 2022-10-11 20:38:35] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:29:29 [I 2022-10-11 20:38:35] ...trap/engines/engine.py.229: process: 0.87238 | load: 0.00381 [I 2022-10-11 20:38:35] ...trap/engines/engine.py.230: loss: 0.08432 [I 2022-10-11 20:38:35] ...optimizers/trijoint.py.151: total_norm: 8.573053 [I 2022-10-11 20:38:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 290/2304 [I 2022-10-11 20:38:44] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:29:34 [I 2022-10-11 20:38:44] ...trap/engines/engine.py.229: process: 0.88556 | load: 0.00395 [I 2022-10-11 20:38:44] ...trap/engines/engine.py.230: loss: 0.15695 [I 2022-10-11 20:38:44] ...optimizers/trijoint.py.151: total_norm: 13.180506 [I 2022-10-11 20:38:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 300/2304 [I 2022-10-11 20:38:53] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:28:41 [I 2022-10-11 20:38:53] ...trap/engines/engine.py.229: process: 0.86164 | load: 0.00328 [I 2022-10-11 20:38:53] ...trap/engines/engine.py.230: loss: 0.08787 [I 2022-10-11 20:38:53] ...optimizers/trijoint.py.151: total_norm: 10.722173 [I 2022-10-11 20:39:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 310/2304 [I 2022-10-11 20:39:01] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:28:06 [I 2022-10-11 20:39:01] ...trap/engines/engine.py.229: process: 0.83497 | load: 0.00321 [I 2022-10-11 20:39:01] ...trap/engines/engine.py.230: loss: 0.08370 [I 2022-10-11 20:39:01] ...optimizers/trijoint.py.151: total_norm: 14.112900 [I 2022-10-11 20:39:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 320/2304 [I 2022-10-11 20:39:10] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:27:58 [I 2022-10-11 20:39:10] ...trap/engines/engine.py.229: process: 0.84822 | load: 0.00302 [I 2022-10-11 20:39:10] ...trap/engines/engine.py.230: loss: 0.10185 [I 2022-10-11 20:39:10] ...optimizers/trijoint.py.151: total_norm: 8.419690 [I 2022-10-11 20:39:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 330/2304 [I 2022-10-11 20:39:18] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:27:46 [I 2022-10-11 20:39:18] ...trap/engines/engine.py.229: process: 0.83687 | load: 0.00300 [I 2022-10-11 20:39:18] ...trap/engines/engine.py.230: loss: 0.09920 [I 2022-10-11 20:39:18] ...optimizers/trijoint.py.151: total_norm: 13.400641 [I 2022-10-11 20:39:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 340/2304 [I 2022-10-11 20:39:27] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:27:45 [I 2022-10-11 20:39:27] ...trap/engines/engine.py.229: process: 0.84058 | load: 0.00304 [I 2022-10-11 20:39:27] ...trap/engines/engine.py.230: loss: 0.12693 [I 2022-10-11 20:39:27] ...optimizers/trijoint.py.151: total_norm: 10.557929 [I 2022-10-11 20:39:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 350/2304 [I 2022-10-11 20:39:35] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:27:34 [I 2022-10-11 20:39:35] ...trap/engines/engine.py.229: process: 0.84327 | load: 0.00308 [I 2022-10-11 20:39:35] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-11 20:39:35] ...optimizers/trijoint.py.151: total_norm: 20.290920 [I 2022-10-11 20:39:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 360/2304 [I 2022-10-11 20:39:44] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:27:27 [I 2022-10-11 20:39:44] ...trap/engines/engine.py.229: process: 0.84884 | load: 0.00302 [I 2022-10-11 20:39:44] ...trap/engines/engine.py.230: loss: 0.08698 [I 2022-10-11 20:39:44] ...optimizers/trijoint.py.151: total_norm: 8.319062 [I 2022-10-11 20:39:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 370/2304 [I 2022-10-11 20:39:52] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:27:22 [I 2022-10-11 20:39:52] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00304 [I 2022-10-11 20:39:52] ...trap/engines/engine.py.230: loss: 0.08593 [I 2022-10-11 20:39:52] ...optimizers/trijoint.py.151: total_norm: 20.577868 [I 2022-10-11 20:40:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 380/2304 [I 2022-10-11 20:40:01] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:27:10 [I 2022-10-11 20:40:01] ...trap/engines/engine.py.229: process: 0.84289 | load: 0.00301 [I 2022-10-11 20:40:01] ...trap/engines/engine.py.230: loss: 0.09547 [I 2022-10-11 20:40:01] ...optimizers/trijoint.py.151: total_norm: 10.243683 [I 2022-10-11 20:40:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 390/2304 [I 2022-10-11 20:40:09] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:27:08 [I 2022-10-11 20:40:09] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00301 [I 2022-10-11 20:40:09] ...trap/engines/engine.py.230: loss: 0.12095 [I 2022-10-11 20:40:09] ...optimizers/trijoint.py.151: total_norm: 19.469004 [I 2022-10-11 20:40:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 400/2304 [I 2022-10-11 20:40:18] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:26:45 [I 2022-10-11 20:40:18] ...trap/engines/engine.py.229: process: 0.83569 | load: 0.00302 [I 2022-10-11 20:40:18] ...trap/engines/engine.py.230: loss: 0.15110 [I 2022-10-11 20:40:18] ...optimizers/trijoint.py.151: total_norm: 106.523285 [I 2022-10-11 20:40:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 410/2304 [I 2022-10-11 20:40:26] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:26:57 [I 2022-10-11 20:40:26] ...trap/engines/engine.py.229: process: 0.85834 | load: 0.00311 [I 2022-10-11 20:40:26] ...trap/engines/engine.py.230: loss: 0.12614 [I 2022-10-11 20:40:26] ...optimizers/trijoint.py.151: total_norm: 34.279671 [I 2022-10-11 20:40:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 420/2304 [I 2022-10-11 20:40:35] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:26:46 [I 2022-10-11 20:40:35] ...trap/engines/engine.py.229: process: 0.83605 | load: 0.00314 [I 2022-10-11 20:40:35] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-11 20:40:35] ...optimizers/trijoint.py.151: total_norm: 14.516752 [I 2022-10-11 20:40:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 430/2304 [I 2022-10-11 20:40:43] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:26:43 [I 2022-10-11 20:40:43] ...trap/engines/engine.py.229: process: 0.84044 | load: 0.00314 [I 2022-10-11 20:40:43] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-11 20:40:43] ...optimizers/trijoint.py.151: total_norm: 51.255009 [I 2022-10-11 20:40:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 440/2304 [I 2022-10-11 20:40:52] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:26:39 [I 2022-10-11 20:40:52] ...trap/engines/engine.py.229: process: 0.86790 | load: 0.00312 [I 2022-10-11 20:40:52] ...trap/engines/engine.py.230: loss: 0.18822 [I 2022-10-11 20:40:52] ...optimizers/trijoint.py.151: total_norm: 26.215479 [I 2022-10-11 20:41:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 450/2304 [I 2022-10-11 20:41:01] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:26:42 [I 2022-10-11 20:41:01] ...trap/engines/engine.py.229: process: 0.87308 | load: 0.00313 [I 2022-10-11 20:41:01] ...trap/engines/engine.py.230: loss: 0.08710 [I 2022-10-11 20:41:01] ...optimizers/trijoint.py.151: total_norm: 15.682888 [I 2022-10-11 20:41:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 460/2304 [I 2022-10-11 20:41:09] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:26:29 [I 2022-10-11 20:41:09] ...trap/engines/engine.py.229: process: 0.86181 | load: 0.00315 [I 2022-10-11 20:41:09] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-11 20:41:09] ...optimizers/trijoint.py.151: total_norm: 19.559544 [I 2022-10-11 20:41:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 470/2304 [I 2022-10-11 20:41:18] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:26:15 [I 2022-10-11 20:41:18] ...trap/engines/engine.py.229: process: 0.85401 | load: 0.00316 [I 2022-10-11 20:41:18] ...trap/engines/engine.py.230: loss: 0.09162 [I 2022-10-11 20:41:18] ...optimizers/trijoint.py.151: total_norm: 14.167423 [I 2022-10-11 20:41:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 480/2304 [I 2022-10-11 20:41:27] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:26:14 [I 2022-10-11 20:41:27] ...trap/engines/engine.py.229: process: 0.86967 | load: 0.00328 [I 2022-10-11 20:41:27] ...trap/engines/engine.py.230: loss: 0.09568 [I 2022-10-11 20:41:27] ...optimizers/trijoint.py.151: total_norm: 10.330379 [I 2022-10-11 20:41:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 490/2304 [I 2022-10-11 20:41:35] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:26:04 [I 2022-10-11 20:41:35] ...trap/engines/engine.py.229: process: 0.87269 | load: 0.00313 [I 2022-10-11 20:41:35] ...trap/engines/engine.py.230: loss: 0.09681 [I 2022-10-11 20:41:35] ...optimizers/trijoint.py.151: total_norm: 16.349827 [I 2022-10-11 20:41:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 500/2304 [I 2022-10-11 20:41:44] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:25:56 [I 2022-10-11 20:41:44] ...trap/engines/engine.py.229: process: 0.86750 | load: 0.00314 [I 2022-10-11 20:41:44] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-11 20:41:44] ...optimizers/trijoint.py.151: total_norm: 16.368629 [I 2022-10-11 20:41:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 510/2304 [I 2022-10-11 20:41:52] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:25:45 [I 2022-10-11 20:41:52] ...trap/engines/engine.py.229: process: 0.87018 | load: 0.00313 [I 2022-10-11 20:41:52] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-11 20:41:52] ...optimizers/trijoint.py.151: total_norm: 18.360895 [I 2022-10-11 20:42:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 520/2304 [I 2022-10-11 20:42:01] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:25:24 [I 2022-10-11 20:42:01] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00319 [I 2022-10-11 20:42:01] ...trap/engines/engine.py.230: loss: 0.08713 [I 2022-10-11 20:42:01] ...optimizers/trijoint.py.151: total_norm: 18.466902 [I 2022-10-11 20:42:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 530/2304 [I 2022-10-11 20:42:10] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:25:29 [I 2022-10-11 20:42:10] ...trap/engines/engine.py.229: process: 0.85254 | load: 0.00311 [I 2022-10-11 20:42:10] ...trap/engines/engine.py.230: loss: 0.10227 [I 2022-10-11 20:42:10] ...optimizers/trijoint.py.151: total_norm: 19.545090 [I 2022-10-11 20:42:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 540/2304 [I 2022-10-11 20:42:18] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:25:18 [I 2022-10-11 20:42:18] ...trap/engines/engine.py.229: process: 0.87074 | load: 0.00314 [I 2022-10-11 20:42:18] ...trap/engines/engine.py.230: loss: 0.10923 [I 2022-10-11 20:42:18] ...optimizers/trijoint.py.151: total_norm: 21.105812 [I 2022-10-11 20:42:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 550/2304 [I 2022-10-11 20:42:27] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:25:11 [I 2022-10-11 20:42:27] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00315 [I 2022-10-11 20:42:27] ...trap/engines/engine.py.230: loss: 0.10950 [I 2022-10-11 20:42:27] ...optimizers/trijoint.py.151: total_norm: 29.877554 [I 2022-10-11 20:42:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 560/2304 [I 2022-10-11 20:42:36] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:25:03 [I 2022-10-11 20:42:36] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00312 [I 2022-10-11 20:42:36] ...trap/engines/engine.py.230: loss: 0.12170 [I 2022-10-11 20:42:36] ...optimizers/trijoint.py.151: total_norm: 17.830114 [I 2022-10-11 20:42:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 570/2304 [I 2022-10-11 20:42:44] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:24:42 [I 2022-10-11 20:42:44] ...trap/engines/engine.py.229: process: 0.85631 | load: 0.00314 [I 2022-10-11 20:42:44] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-11 20:42:44] ...optimizers/trijoint.py.151: total_norm: 14.212222 [I 2022-10-11 20:42:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 580/2304 [I 2022-10-11 20:42:53] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:24:41 [I 2022-10-11 20:42:53] ...trap/engines/engine.py.229: process: 0.88213 | load: 0.00317 [I 2022-10-11 20:42:53] ...trap/engines/engine.py.230: loss: 0.12190 [I 2022-10-11 20:42:53] ...optimizers/trijoint.py.151: total_norm: 28.112583 [I 2022-10-11 20:43:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 590/2304 [I 2022-10-11 20:43:01] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:24:21 [I 2022-10-11 20:43:01] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00311 [I 2022-10-11 20:43:01] ...trap/engines/engine.py.230: loss: 0.07643 [I 2022-10-11 20:43:01] ...optimizers/trijoint.py.151: total_norm: 18.225533 [I 2022-10-11 20:43:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 600/2304 [I 2022-10-11 20:43:10] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:24:15 [I 2022-10-11 20:43:10] ...trap/engines/engine.py.229: process: 0.84590 | load: 0.00316 [I 2022-10-11 20:43:10] ...trap/engines/engine.py.230: loss: 0.08792 [I 2022-10-11 20:43:10] ...optimizers/trijoint.py.151: total_norm: 10.831571 [I 2022-10-11 20:43:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 610/2304 [I 2022-10-11 20:43:18] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:24:19 [I 2022-10-11 20:43:18] ...trap/engines/engine.py.229: process: 0.87475 | load: 0.00313 [I 2022-10-11 20:43:18] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-11 20:43:18] ...optimizers/trijoint.py.151: total_norm: 12.408577 [I 2022-10-11 20:43:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 620/2304 [I 2022-10-11 20:43:27] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:24:15 [I 2022-10-11 20:43:27] ...trap/engines/engine.py.229: process: 0.85477 | load: 0.00312 [I 2022-10-11 20:43:27] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-11 20:43:27] ...optimizers/trijoint.py.151: total_norm: 25.688614 [I 2022-10-11 20:43:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 630/2304 [I 2022-10-11 20:43:36] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:23:55 [I 2022-10-11 20:43:36] ...trap/engines/engine.py.229: process: 0.84674 | load: 0.00312 [I 2022-10-11 20:43:36] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-11 20:43:36] ...optimizers/trijoint.py.151: total_norm: 51.848469 [I 2022-10-11 20:43:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 640/2304 [I 2022-10-11 20:43:44] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:23:54 [I 2022-10-11 20:43:44] ...trap/engines/engine.py.229: process: 0.87180 | load: 0.00316 [I 2022-10-11 20:43:44] ...trap/engines/engine.py.230: loss: 0.08458 [I 2022-10-11 20:43:44] ...optimizers/trijoint.py.151: total_norm: 13.212562 [I 2022-10-11 20:43:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 650/2304 [I 2022-10-11 20:43:53] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:23:48 [I 2022-10-11 20:43:53] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00314 [I 2022-10-11 20:43:53] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-11 20:43:53] ...optimizers/trijoint.py.151: total_norm: 15.567617 [I 2022-10-11 20:44:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 660/2304 [I 2022-10-11 20:44:02] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:23:33 [I 2022-10-11 20:44:02] ...trap/engines/engine.py.229: process: 0.85513 | load: 0.00331 [I 2022-10-11 20:44:02] ...trap/engines/engine.py.230: loss: 0.10676 [I 2022-10-11 20:44:02] ...optimizers/trijoint.py.151: total_norm: 13.262152 [I 2022-10-11 20:44:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 670/2304 [I 2022-10-11 20:44:10] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:23:29 [I 2022-10-11 20:44:10] ...trap/engines/engine.py.229: process: 0.87512 | load: 0.00320 [I 2022-10-11 20:44:10] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-11 20:44:10] ...optimizers/trijoint.py.151: total_norm: 23.410133 [I 2022-10-11 20:44:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 680/2304 [I 2022-10-11 20:44:19] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:23:02 [I 2022-10-11 20:44:19] ...trap/engines/engine.py.229: process: 0.85483 | load: 0.00314 [I 2022-10-11 20:44:19] ...trap/engines/engine.py.230: loss: 0.09269 [I 2022-10-11 20:44:19] ...optimizers/trijoint.py.151: total_norm: 12.897229 [I 2022-10-11 20:44:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 690/2304 [I 2022-10-11 20:44:27] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:22:36 [I 2022-10-11 20:44:27] ...trap/engines/engine.py.229: process: 0.83351 | load: 0.00302 [I 2022-10-11 20:44:27] ...trap/engines/engine.py.230: loss: 0.14678 [I 2022-10-11 20:44:27] ...optimizers/trijoint.py.151: total_norm: 33.077141 [I 2022-10-11 20:44:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 700/2304 [I 2022-10-11 20:44:36] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:22:37 [I 2022-10-11 20:44:36] ...trap/engines/engine.py.229: process: 0.84842 | load: 0.00301 [I 2022-10-11 20:44:36] ...trap/engines/engine.py.230: loss: 0.11738 [I 2022-10-11 20:44:36] ...optimizers/trijoint.py.151: total_norm: 10.948896 [I 2022-10-11 20:44:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 710/2304 [I 2022-10-11 20:44:44] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:22:36 [I 2022-10-11 20:44:44] ...trap/engines/engine.py.229: process: 0.85329 | load: 0.00296 [I 2022-10-11 20:44:44] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-11 20:44:44] ...optimizers/trijoint.py.151: total_norm: 10.567502 [I 2022-10-11 20:44:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 720/2304 [I 2022-10-11 20:44:53] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:22:22 [I 2022-10-11 20:44:53] ...trap/engines/engine.py.229: process: 0.85365 | load: 0.00296 [I 2022-10-11 20:44:53] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-11 20:44:53] ...optimizers/trijoint.py.151: total_norm: 26.161564 [I 2022-10-11 20:45:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 730/2304 [I 2022-10-11 20:45:01] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:22:09 [I 2022-10-11 20:45:01] ...trap/engines/engine.py.229: process: 0.84846 | load: 0.00297 [I 2022-10-11 20:45:01] ...trap/engines/engine.py.230: loss: 0.09590 [I 2022-10-11 20:45:01] ...optimizers/trijoint.py.151: total_norm: 9.874285 [I 2022-10-11 20:45:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 740/2304 [I 2022-10-11 20:45:10] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:22:15 [I 2022-10-11 20:45:10] ...trap/engines/engine.py.229: process: 0.86627 | load: 0.00303 [I 2022-10-11 20:45:10] ...trap/engines/engine.py.230: loss: 0.15964 [I 2022-10-11 20:45:10] ...optimizers/trijoint.py.151: total_norm: 15.540462 [I 2022-10-11 20:45:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 750/2304 [I 2022-10-11 20:45:18] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:21:55 [I 2022-10-11 20:45:18] ...trap/engines/engine.py.229: process: 0.84358 | load: 0.00302 [I 2022-10-11 20:45:18] ...trap/engines/engine.py.230: loss: 0.09195 [I 2022-10-11 20:45:18] ...optimizers/trijoint.py.151: total_norm: 10.444822 [I 2022-10-11 20:45:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 760/2304 [I 2022-10-11 20:45:27] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:21:52 [I 2022-10-11 20:45:27] ...trap/engines/engine.py.229: process: 0.85058 | load: 0.00299 [I 2022-10-11 20:45:27] ...trap/engines/engine.py.230: loss: 0.17399 [I 2022-10-11 20:45:27] ...optimizers/trijoint.py.151: total_norm: 18.867241 [I 2022-10-11 20:45:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 770/2304 [I 2022-10-11 20:45:35] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:21:52 [I 2022-10-11 20:45:35] ...trap/engines/engine.py.229: process: 0.86247 | load: 0.00299 [I 2022-10-11 20:45:35] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-11 20:45:35] ...optimizers/trijoint.py.151: total_norm: 12.996928 [I 2022-10-11 20:45:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 780/2304 [I 2022-10-11 20:45:44] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:21:35 [I 2022-10-11 20:45:44] ...trap/engines/engine.py.229: process: 0.86083 | load: 0.00300 [I 2022-10-11 20:45:44] ...trap/engines/engine.py.230: loss: 0.09853 [I 2022-10-11 20:45:44] ...optimizers/trijoint.py.151: total_norm: 9.054523 [I 2022-10-11 20:45:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 790/2304 [I 2022-10-11 20:45:52] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:21:35 [I 2022-10-11 20:45:52] ...trap/engines/engine.py.229: process: 0.87799 | load: 0.00299 [I 2022-10-11 20:45:52] ...trap/engines/engine.py.230: loss: 0.08877 [I 2022-10-11 20:45:52] ...optimizers/trijoint.py.151: total_norm: 10.102448 [I 2022-10-11 20:46:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 800/2304 [I 2022-10-11 20:46:01] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:21:16 [I 2022-10-11 20:46:01] ...trap/engines/engine.py.229: process: 0.84843 | load: 0.00300 [I 2022-10-11 20:46:01] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-11 20:46:01] ...optimizers/trijoint.py.151: total_norm: 13.971630 [I 2022-10-11 20:46:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 810/2304 [I 2022-10-11 20:46:09] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:21:10 [I 2022-10-11 20:46:09] ...trap/engines/engine.py.229: process: 0.85133 | load: 0.00299 [I 2022-10-11 20:46:09] ...trap/engines/engine.py.230: loss: 0.09780 [I 2022-10-11 20:46:09] ...optimizers/trijoint.py.151: total_norm: 18.742638 [I 2022-10-11 20:46:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 820/2304 [I 2022-10-11 20:46:18] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:21:02 [I 2022-10-11 20:46:18] ...trap/engines/engine.py.229: process: 0.86303 | load: 0.00301 [I 2022-10-11 20:46:18] ...trap/engines/engine.py.230: loss: 0.08456 [I 2022-10-11 20:46:18] ...optimizers/trijoint.py.151: total_norm: 14.882535 [I 2022-10-11 20:46:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 830/2304 [I 2022-10-11 20:46:26] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:21:00 [I 2022-10-11 20:46:26] ...trap/engines/engine.py.229: process: 0.85715 | load: 0.00310 [I 2022-10-11 20:46:26] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-11 20:46:26] ...optimizers/trijoint.py.151: total_norm: 18.318411 [I 2022-10-11 20:46:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 840/2304 [I 2022-10-11 20:46:35] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:20:48 [I 2022-10-11 20:46:35] ...trap/engines/engine.py.229: process: 0.84992 | load: 0.00314 [I 2022-10-11 20:46:35] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-11 20:46:35] ...optimizers/trijoint.py.151: total_norm: 13.476525 [I 2022-10-11 20:46:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 850/2304 [I 2022-10-11 20:46:44] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:20:59 [I 2022-10-11 20:46:44] ...trap/engines/engine.py.229: process: 0.87518 | load: 0.00313 [I 2022-10-11 20:46:44] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-11 20:46:44] ...optimizers/trijoint.py.151: total_norm: 8.494245 [I 2022-10-11 20:46:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 860/2304 [I 2022-10-11 20:46:52] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:20:44 [I 2022-10-11 20:46:52] ...trap/engines/engine.py.229: process: 0.86070 | load: 0.00313 [I 2022-10-11 20:46:52] ...trap/engines/engine.py.230: loss: 0.11241 [I 2022-10-11 20:46:52] ...optimizers/trijoint.py.151: total_norm: 14.043689 [I 2022-10-11 20:47:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 870/2304 [I 2022-10-11 20:47:01] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:20:29 [I 2022-10-11 20:47:01] ...trap/engines/engine.py.229: process: 0.85235 | load: 0.00313 [I 2022-10-11 20:47:01] ...trap/engines/engine.py.230: loss: 0.08689 [I 2022-10-11 20:47:01] ...optimizers/trijoint.py.151: total_norm: 17.047066 [I 2022-10-11 20:47:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 880/2304 [I 2022-10-11 20:47:09] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:27 [I 2022-10-11 20:47:09] ...trap/engines/engine.py.229: process: 0.87883 | load: 0.00313 [I 2022-10-11 20:47:09] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-11 20:47:09] ...optimizers/trijoint.py.151: total_norm: 9.409470 [I 2022-10-11 20:47:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 890/2304 [I 2022-10-11 20:47:18] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:20:11 [I 2022-10-11 20:47:18] ...trap/engines/engine.py.229: process: 0.86125 | load: 0.00312 [I 2022-10-11 20:47:18] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-11 20:47:18] ...optimizers/trijoint.py.151: total_norm: 20.649368 [I 2022-10-11 20:47:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 900/2304 [I 2022-10-11 20:47:27] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:20:07 [I 2022-10-11 20:47:27] ...trap/engines/engine.py.229: process: 0.86972 | load: 0.00313 [I 2022-10-11 20:47:27] ...trap/engines/engine.py.230: loss: 0.11622 [I 2022-10-11 20:47:27] ...optimizers/trijoint.py.151: total_norm: 12.769780 [I 2022-10-11 20:47:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 910/2304 [I 2022-10-11 20:47:35] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:19:54 [I 2022-10-11 20:47:35] ...trap/engines/engine.py.229: process: 0.84854 | load: 0.00312 [I 2022-10-11 20:47:35] ...trap/engines/engine.py.230: loss: 0.08871 [I 2022-10-11 20:47:35] ...optimizers/trijoint.py.151: total_norm: 12.710321 [I 2022-10-11 20:47:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 920/2304 [I 2022-10-11 20:47:44] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:19:49 [I 2022-10-11 20:47:44] ...trap/engines/engine.py.229: process: 0.86401 | load: 0.00312 [I 2022-10-11 20:47:44] ...trap/engines/engine.py.230: loss: 0.15580 [I 2022-10-11 20:47:44] ...optimizers/trijoint.py.151: total_norm: 15.608151 [I 2022-10-11 20:47:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 930/2304 [I 2022-10-11 20:47:52] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:19:37 [I 2022-10-11 20:47:52] ...trap/engines/engine.py.229: process: 0.85386 | load: 0.00314 [I 2022-10-11 20:47:52] ...trap/engines/engine.py.230: loss: 0.07472 [I 2022-10-11 20:47:52] ...optimizers/trijoint.py.151: total_norm: 12.707025 [I 2022-10-11 20:48:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 940/2304 [I 2022-10-11 20:48:01] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:18 [I 2022-10-11 20:48:01] ...trap/engines/engine.py.229: process: 0.82633 | load: 0.00314 [I 2022-10-11 20:48:01] ...trap/engines/engine.py.230: loss: 0.08169 [I 2022-10-11 20:48:01] ...optimizers/trijoint.py.151: total_norm: 12.622643 [I 2022-10-11 20:48:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 950/2304 [I 2022-10-11 20:48:09] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:19:14 [I 2022-10-11 20:48:09] ...trap/engines/engine.py.229: process: 0.86035 | load: 0.00315 [I 2022-10-11 20:48:09] ...trap/engines/engine.py.230: loss: 0.17499 [I 2022-10-11 20:48:09] ...optimizers/trijoint.py.151: total_norm: 24.299194 [I 2022-10-11 20:48:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 960/2304 [I 2022-10-11 20:48:18] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:12 [I 2022-10-11 20:48:18] ...trap/engines/engine.py.229: process: 0.85448 | load: 0.00312 [I 2022-10-11 20:48:18] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-11 20:48:18] ...optimizers/trijoint.py.151: total_norm: 12.083092 [I 2022-10-11 20:48:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 970/2304 [I 2022-10-11 20:48:27] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:19:01 [I 2022-10-11 20:48:27] ...trap/engines/engine.py.229: process: 0.86278 | load: 0.00315 [I 2022-10-11 20:48:27] ...trap/engines/engine.py.230: loss: 0.08965 [I 2022-10-11 20:48:27] ...optimizers/trijoint.py.151: total_norm: 16.388046 [I 2022-10-11 20:48:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 980/2304 [I 2022-10-11 20:48:35] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:18:47 [I 2022-10-11 20:48:35] ...trap/engines/engine.py.229: process: 0.84306 | load: 0.00315 [I 2022-10-11 20:48:35] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-11 20:48:35] ...optimizers/trijoint.py.151: total_norm: 19.152309 [I 2022-10-11 20:48:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 990/2304 [I 2022-10-11 20:48:44] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:18:46 [I 2022-10-11 20:48:44] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00314 [I 2022-10-11 20:48:44] ...trap/engines/engine.py.230: loss: 0.09200 [I 2022-10-11 20:48:44] ...optimizers/trijoint.py.151: total_norm: 12.114744 [I 2022-10-11 20:48:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1000/2304 [I 2022-10-11 20:48:52] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:18:42 [I 2022-10-11 20:48:52] ...trap/engines/engine.py.229: process: 0.85963 | load: 0.00313 [I 2022-10-11 20:48:52] ...trap/engines/engine.py.230: loss: 0.14775 [I 2022-10-11 20:48:52] ...optimizers/trijoint.py.151: total_norm: 15.789982 [I 2022-10-11 20:49:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1010/2304 [I 2022-10-11 20:49:01] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:27 [I 2022-10-11 20:49:01] ...trap/engines/engine.py.229: process: 0.85783 | load: 0.00313 [I 2022-10-11 20:49:01] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-11 20:49:01] ...optimizers/trijoint.py.151: total_norm: 31.943907 [I 2022-10-11 20:49:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 1020/2304 [I 2022-10-11 20:49:09] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:18:20 [I 2022-10-11 20:49:09] ...trap/engines/engine.py.229: process: 0.85906 | load: 0.00313 [I 2022-10-11 20:49:09] ...trap/engines/engine.py.230: loss: 0.11007 [I 2022-10-11 20:49:09] ...optimizers/trijoint.py.151: total_norm: 17.291685 [I 2022-10-11 20:49:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1030/2304 [I 2022-10-11 20:49:18] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:18:08 [I 2022-10-11 20:49:18] ...trap/engines/engine.py.229: process: 0.85798 | load: 0.00312 [I 2022-10-11 20:49:18] ...trap/engines/engine.py.230: loss: 0.12781 [I 2022-10-11 20:49:18] ...optimizers/trijoint.py.151: total_norm: 16.415224 [I 2022-10-11 20:49:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1040/2304 [I 2022-10-11 20:49:27] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:18:04 [I 2022-10-11 20:49:27] ...trap/engines/engine.py.229: process: 0.86131 | load: 0.00314 [I 2022-10-11 20:49:27] ...trap/engines/engine.py.230: loss: 0.09049 [I 2022-10-11 20:49:27] ...optimizers/trijoint.py.151: total_norm: 13.837780 [I 2022-10-11 20:49:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1050/2304 [I 2022-10-11 20:49:35] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:17:55 [I 2022-10-11 20:49:35] ...trap/engines/engine.py.229: process: 0.85976 | load: 0.00316 [I 2022-10-11 20:49:35] ...trap/engines/engine.py.230: loss: 0.12770 [I 2022-10-11 20:49:35] ...optimizers/trijoint.py.151: total_norm: 7.429076 [I 2022-10-11 20:49:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1060/2304 [I 2022-10-11 20:49:44] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:17:53 [I 2022-10-11 20:49:44] ...trap/engines/engine.py.229: process: 0.85575 | load: 0.00314 [I 2022-10-11 20:49:44] ...trap/engines/engine.py.230: loss: 0.10969 [I 2022-10-11 20:49:44] ...optimizers/trijoint.py.151: total_norm: 8.524369 [I 2022-10-11 20:49:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1070/2304 [I 2022-10-11 20:49:52] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:17:43 [I 2022-10-11 20:49:52] ...trap/engines/engine.py.229: process: 0.86158 | load: 0.00311 [I 2022-10-11 20:49:52] ...trap/engines/engine.py.230: loss: 0.11209 [I 2022-10-11 20:49:52] ...optimizers/trijoint.py.151: total_norm: 10.284004 [I 2022-10-11 20:50:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1080/2304 [I 2022-10-11 20:50:01] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:17:33 [I 2022-10-11 20:50:01] ...trap/engines/engine.py.229: process: 0.85462 | load: 0.00314 [I 2022-10-11 20:50:01] ...trap/engines/engine.py.230: loss: 0.09305 [I 2022-10-11 20:50:01] ...optimizers/trijoint.py.151: total_norm: 21.014362 [I 2022-10-11 20:50:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1090/2304 [I 2022-10-11 20:50:10] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:17:12 [I 2022-10-11 20:50:10] ...trap/engines/engine.py.229: process: 0.84386 | load: 0.00312 [I 2022-10-11 20:50:10] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-11 20:50:10] ...optimizers/trijoint.py.151: total_norm: 13.955756 [I 2022-10-11 20:50:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1100/2304 [I 2022-10-11 20:50:18] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:17:06 [I 2022-10-11 20:50:18] ...trap/engines/engine.py.229: process: 0.85681 | load: 0.00311 [I 2022-10-11 20:50:18] ...trap/engines/engine.py.230: loss: 0.17039 [I 2022-10-11 20:50:18] ...optimizers/trijoint.py.151: total_norm: 8.880366 [I 2022-10-11 20:50:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1110/2304 [I 2022-10-11 20:50:27] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:16:57 [I 2022-10-11 20:50:27] ...trap/engines/engine.py.229: process: 0.85200 | load: 0.00306 [I 2022-10-11 20:50:27] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-11 20:50:27] ...optimizers/trijoint.py.151: total_norm: 12.839421 [I 2022-10-11 20:50:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1120/2304 [I 2022-10-11 20:50:35] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:16:37 [I 2022-10-11 20:50:35] ...trap/engines/engine.py.229: process: 0.85117 | load: 0.00310 [I 2022-10-11 20:50:35] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-11 20:50:35] ...optimizers/trijoint.py.151: total_norm: 10.659225 [I 2022-10-11 20:50:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1130/2304 [I 2022-10-11 20:50:44] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:17:13 [I 2022-10-11 20:50:44] ...trap/engines/engine.py.229: process: 0.87231 | load: 0.00384 [I 2022-10-11 20:50:44] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-11 20:50:44] ...optimizers/trijoint.py.151: total_norm: 19.223696 [I 2022-10-11 20:50:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 1140/2304 [I 2022-10-11 20:50:53] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:17:06 [I 2022-10-11 20:50:53] ...trap/engines/engine.py.229: process: 0.86932 | load: 0.00373 [I 2022-10-11 20:50:53] ...trap/engines/engine.py.230: loss: 0.09482 [I 2022-10-11 20:50:53] ...optimizers/trijoint.py.151: total_norm: 12.587678 [I 2022-10-11 20:51:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 1150/2304 [I 2022-10-11 20:51:02] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:16:57 [I 2022-10-11 20:51:02] ...trap/engines/engine.py.229: process: 0.87576 | load: 0.00373 [I 2022-10-11 20:51:02] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-11 20:51:02] ...optimizers/trijoint.py.151: total_norm: 12.248719 [I 2022-10-11 20:51:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1160/2304 [I 2022-10-11 20:51:10] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:16:17 [I 2022-10-11 20:51:10] ...trap/engines/engine.py.229: process: 0.84721 | load: 0.00301 [I 2022-10-11 20:51:10] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-11 20:51:10] ...optimizers/trijoint.py.151: total_norm: 13.046182 [I 2022-10-11 20:51:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 1170/2304 [I 2022-10-11 20:51:19] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:16:01 [I 2022-10-11 20:51:19] ...trap/engines/engine.py.229: process: 0.84464 | load: 0.00296 [I 2022-10-11 20:51:19] ...trap/engines/engine.py.230: loss: 0.10480 [I 2022-10-11 20:51:19] ...optimizers/trijoint.py.151: total_norm: 10.014616 [I 2022-10-11 20:51:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1180/2304 [I 2022-10-11 20:51:27] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:15:54 [I 2022-10-11 20:51:27] ...trap/engines/engine.py.229: process: 0.84483 | load: 0.00295 [I 2022-10-11 20:51:27] ...trap/engines/engine.py.230: loss: 0.07594 [I 2022-10-11 20:51:27] ...optimizers/trijoint.py.151: total_norm: 11.372377 [I 2022-10-11 20:51:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1190/2304 [I 2022-10-11 20:51:36] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:15:37 [I 2022-10-11 20:51:36] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00294 [I 2022-10-11 20:51:36] ...trap/engines/engine.py.230: loss: 0.08697 [I 2022-10-11 20:51:36] ...optimizers/trijoint.py.151: total_norm: 14.249128 [I 2022-10-11 20:51:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1200/2304 [I 2022-10-11 20:51:44] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:15:38 [I 2022-10-11 20:51:44] ...trap/engines/engine.py.229: process: 0.86097 | load: 0.00297 [I 2022-10-11 20:51:44] ...trap/engines/engine.py.230: loss: 0.09757 [I 2022-10-11 20:51:44] ...optimizers/trijoint.py.151: total_norm: 13.086441 [I 2022-10-11 20:51:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 1210/2304 [I 2022-10-11 20:51:53] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:15:25 [I 2022-10-11 20:51:53] ...trap/engines/engine.py.229: process: 0.84295 | load: 0.00295 [I 2022-10-11 20:51:53] ...trap/engines/engine.py.230: loss: 0.09370 [I 2022-10-11 20:51:53] ...optimizers/trijoint.py.151: total_norm: 18.277927 [I 2022-10-11 20:52:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1220/2304 [I 2022-10-11 20:52:01] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:15:18 [I 2022-10-11 20:52:01] ...trap/engines/engine.py.229: process: 0.85029 | load: 0.00295 [I 2022-10-11 20:52:01] ...trap/engines/engine.py.230: loss: 0.18671 [I 2022-10-11 20:52:01] ...optimizers/trijoint.py.151: total_norm: 29.583271 [I 2022-10-11 20:52:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1230/2304 [I 2022-10-11 20:52:10] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:15:18 [I 2022-10-11 20:52:10] ...trap/engines/engine.py.229: process: 0.85360 | load: 0.00297 [I 2022-10-11 20:52:10] ...trap/engines/engine.py.230: loss: 0.11338 [I 2022-10-11 20:52:10] ...optimizers/trijoint.py.151: total_norm: 10.406363 [I 2022-10-11 20:52:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1240/2304 [I 2022-10-11 20:52:18] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:15:03 [I 2022-10-11 20:52:18] ...trap/engines/engine.py.229: process: 0.84216 | load: 0.00296 [I 2022-10-11 20:52:18] ...trap/engines/engine.py.230: loss: 0.10141 [I 2022-10-11 20:52:18] ...optimizers/trijoint.py.151: total_norm: 17.979673 [I 2022-10-11 20:52:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1250/2304 [I 2022-10-11 20:52:27] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:14:52 [I 2022-10-11 20:52:27] ...trap/engines/engine.py.229: process: 0.83334 | load: 0.00296 [I 2022-10-11 20:52:27] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-11 20:52:27] ...optimizers/trijoint.py.151: total_norm: 12.361561 [I 2022-10-11 20:52:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1260/2304 [I 2022-10-11 20:52:35] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:14:44 [I 2022-10-11 20:52:35] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00295 [I 2022-10-11 20:52:35] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-11 20:52:35] ...optimizers/trijoint.py.151: total_norm: 14.522859 [I 2022-10-11 20:52:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1270/2304 [I 2022-10-11 20:52:44] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:14:33 [I 2022-10-11 20:52:44] ...trap/engines/engine.py.229: process: 0.84911 | load: 0.00296 [I 2022-10-11 20:52:44] ...trap/engines/engine.py.230: loss: 0.09347 [I 2022-10-11 20:52:44] ...optimizers/trijoint.py.151: total_norm: 17.738688 [I 2022-10-11 20:52:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1280/2304 [I 2022-10-11 20:52:52] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:14:18 [I 2022-10-11 20:52:52] ...trap/engines/engine.py.229: process: 0.84141 | load: 0.00299 [I 2022-10-11 20:52:52] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-11 20:52:52] ...optimizers/trijoint.py.151: total_norm: 13.738084 [I 2022-10-11 20:53:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1290/2304 [I 2022-10-11 20:53:00] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:14:12 [I 2022-10-11 20:53:00] ...trap/engines/engine.py.229: process: 0.83975 | load: 0.00302 [I 2022-10-11 20:53:00] ...trap/engines/engine.py.230: loss: 0.10979 [I 2022-10-11 20:53:00] ...optimizers/trijoint.py.151: total_norm: 28.470673 [I 2022-10-11 20:53:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 1300/2304 [I 2022-10-11 20:53:09] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:14:06 [I 2022-10-11 20:53:09] ...trap/engines/engine.py.229: process: 0.84162 | load: 0.00297 [I 2022-10-11 20:53:09] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-11 20:53:09] ...optimizers/trijoint.py.151: total_norm: 8.448956 [I 2022-10-11 20:53:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1310/2304 [I 2022-10-11 20:53:17] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:14:00 [I 2022-10-11 20:53:17] ...trap/engines/engine.py.229: process: 0.84719 | load: 0.00305 [I 2022-10-11 20:53:17] ...trap/engines/engine.py.230: loss: 0.17452 [I 2022-10-11 20:53:17] ...optimizers/trijoint.py.151: total_norm: 23.507494 [I 2022-10-11 20:53:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 1320/2304 [I 2022-10-11 20:53:26] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:13:47 [I 2022-10-11 20:53:26] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00299 [I 2022-10-11 20:53:26] ...trap/engines/engine.py.230: loss: 0.09163 [I 2022-10-11 20:53:26] ...optimizers/trijoint.py.151: total_norm: 9.382004 [I 2022-10-11 20:53:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 1330/2304 [I 2022-10-11 20:53:34] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:13:43 [I 2022-10-11 20:53:34] ...trap/engines/engine.py.229: process: 0.84923 | load: 0.00299 [I 2022-10-11 20:53:34] ...trap/engines/engine.py.230: loss: 0.15670 [I 2022-10-11 20:53:34] ...optimizers/trijoint.py.151: total_norm: 30.750769 [I 2022-10-11 20:53:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 1340/2304 [I 2022-10-11 20:53:43] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:13:30 [I 2022-10-11 20:53:43] ...trap/engines/engine.py.229: process: 0.83908 | load: 0.00302 [I 2022-10-11 20:53:43] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-11 20:53:43] ...optimizers/trijoint.py.151: total_norm: 18.954535 [I 2022-10-11 20:53:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 1350/2304 [I 2022-10-11 20:53:51] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:13:22 [I 2022-10-11 20:53:51] ...trap/engines/engine.py.229: process: 0.84514 | load: 0.00298 [I 2022-10-11 20:53:51] ...trap/engines/engine.py.230: loss: 0.14537 [I 2022-10-11 20:53:51] ...optimizers/trijoint.py.151: total_norm: 32.185417 [I 2022-10-11 20:54:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1360/2304 [I 2022-10-11 20:54:00] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:13:17 [I 2022-10-11 20:54:00] ...trap/engines/engine.py.229: process: 0.85118 | load: 0.00298 [I 2022-10-11 20:54:00] ...trap/engines/engine.py.230: loss: 0.08153 [I 2022-10-11 20:54:00] ...optimizers/trijoint.py.151: total_norm: 18.096060 [I 2022-10-11 20:54:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1370/2304 [I 2022-10-11 20:54:08] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:13:05 [I 2022-10-11 20:54:08] ...trap/engines/engine.py.229: process: 0.82283 | load: 0.00299 [I 2022-10-11 20:54:08] ...trap/engines/engine.py.230: loss: 0.13901 [I 2022-10-11 20:54:08] ...optimizers/trijoint.py.151: total_norm: 20.829861 [I 2022-10-11 20:54:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1380/2304 [I 2022-10-11 20:54:17] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:12:53 [I 2022-10-11 20:54:17] ...trap/engines/engine.py.229: process: 0.82433 | load: 0.00301 [I 2022-10-11 20:54:17] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-11 20:54:17] ...optimizers/trijoint.py.151: total_norm: 12.300014 [I 2022-10-11 20:54:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 1390/2304 [I 2022-10-11 20:54:25] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:12:58 [I 2022-10-11 20:54:25] ...trap/engines/engine.py.229: process: 0.86715 | load: 0.00312 [I 2022-10-11 20:54:25] ...trap/engines/engine.py.230: loss: 0.15586 [I 2022-10-11 20:54:25] ...optimizers/trijoint.py.151: total_norm: 11.955743 [I 2022-10-11 20:54:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 1400/2304 [I 2022-10-11 20:54:34] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:12:59 [I 2022-10-11 20:54:34] ...trap/engines/engine.py.229: process: 0.85652 | load: 0.00314 [I 2022-10-11 20:54:34] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-11 20:54:34] ...optimizers/trijoint.py.151: total_norm: 34.808960 [I 2022-10-11 20:54:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 1410/2304 [I 2022-10-11 20:54:42] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:12:47 [I 2022-10-11 20:54:42] ...trap/engines/engine.py.229: process: 0.85677 | load: 0.00312 [I 2022-10-11 20:54:42] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-11 20:54:42] ...optimizers/trijoint.py.151: total_norm: 16.448435 [I 2022-10-11 20:54:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 1420/2304 [I 2022-10-11 20:54:51] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:12:34 [I 2022-10-11 20:54:51] ...trap/engines/engine.py.229: process: 0.84729 | load: 0.00313 [I 2022-10-11 20:54:51] ...trap/engines/engine.py.230: loss: 0.17110 [I 2022-10-11 20:54:51] ...optimizers/trijoint.py.151: total_norm: 20.804983 [I 2022-10-11 20:54:59] ...trap/engines/engine.py.224: train: epoch 118 | batch 1430/2304 [I 2022-10-11 20:54:59] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:12:30 [I 2022-10-11 20:54:59] ...trap/engines/engine.py.229: process: 0.85964 | load: 0.00315 [I 2022-10-11 20:54:59] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-11 20:54:59] ...optimizers/trijoint.py.151: total_norm: 22.573408 [I 2022-10-11 20:55:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1440/2304 [I 2022-10-11 20:55:08] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:12:17 [I 2022-10-11 20:55:08] ...trap/engines/engine.py.229: process: 0.83683 | load: 0.00316 [I 2022-10-11 20:55:08] ...trap/engines/engine.py.230: loss: 0.11244 [I 2022-10-11 20:55:08] ...optimizers/trijoint.py.151: total_norm: 13.687822 [I 2022-10-11 20:55:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1450/2304 [I 2022-10-11 20:55:17] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:12:10 [I 2022-10-11 20:55:17] ...trap/engines/engine.py.229: process: 0.86110 | load: 0.00313 [I 2022-10-11 20:55:17] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-11 20:55:17] ...optimizers/trijoint.py.151: total_norm: 10.131157 [I 2022-10-11 20:55:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 1460/2304 [I 2022-10-11 20:55:25] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:12:02 [I 2022-10-11 20:55:25] ...trap/engines/engine.py.229: process: 0.84629 | load: 0.00318 [I 2022-10-11 20:55:25] ...trap/engines/engine.py.230: loss: 0.08156 [I 2022-10-11 20:55:25] ...optimizers/trijoint.py.151: total_norm: 10.089661 [I 2022-10-11 20:55:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 1470/2304 [I 2022-10-11 20:55:34] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:12:00 [I 2022-10-11 20:55:34] ...trap/engines/engine.py.229: process: 0.87403 | load: 0.00313 [I 2022-10-11 20:55:34] ...trap/engines/engine.py.230: loss: 0.12687 [I 2022-10-11 20:55:34] ...optimizers/trijoint.py.151: total_norm: 27.798452 [I 2022-10-11 20:55:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 1480/2304 [I 2022-10-11 20:55:42] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:48 [I 2022-10-11 20:55:42] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00319 [I 2022-10-11 20:55:42] ...trap/engines/engine.py.230: loss: 0.13712 [I 2022-10-11 20:55:42] ...optimizers/trijoint.py.151: total_norm: 9.847021 [I 2022-10-11 20:55:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 1490/2304 [I 2022-10-11 20:55:51] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:43 [I 2022-10-11 20:55:51] ...trap/engines/engine.py.229: process: 0.87655 | load: 0.00317 [I 2022-10-11 20:55:51] ...trap/engines/engine.py.230: loss: 0.13120 [I 2022-10-11 20:55:51] ...optimizers/trijoint.py.151: total_norm: 26.294220 [I 2022-10-11 20:56:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1500/2304 [I 2022-10-11 20:56:00] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:31 [I 2022-10-11 20:56:00] ...trap/engines/engine.py.229: process: 0.85887 | load: 0.00314 [I 2022-10-11 20:56:00] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-11 20:56:00] ...optimizers/trijoint.py.151: total_norm: 12.570669 [I 2022-10-11 20:56:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1510/2304 [I 2022-10-11 20:56:08] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:18 [I 2022-10-11 20:56:08] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00314 [I 2022-10-11 20:56:08] ...trap/engines/engine.py.230: loss: 0.11881 [I 2022-10-11 20:56:08] ...optimizers/trijoint.py.151: total_norm: 11.456703 [I 2022-10-11 20:56:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1520/2304 [I 2022-10-11 20:56:17] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:11:07 [I 2022-10-11 20:56:17] ...trap/engines/engine.py.229: process: 0.84491 | load: 0.00313 [I 2022-10-11 20:56:17] ...trap/engines/engine.py.230: loss: 0.08809 [I 2022-10-11 20:56:17] ...optimizers/trijoint.py.151: total_norm: 12.512626 [I 2022-10-11 20:56:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 1530/2304 [I 2022-10-11 20:56:25] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:11:06 [I 2022-10-11 20:56:25] ...trap/engines/engine.py.229: process: 0.86084 | load: 0.00314 [I 2022-10-11 20:56:25] ...trap/engines/engine.py.230: loss: 0.08938 [I 2022-10-11 20:56:25] ...optimizers/trijoint.py.151: total_norm: 11.112149 [I 2022-10-11 20:56:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 1540/2304 [I 2022-10-11 20:56:34] ...trap/engines/engine.py.228: elapsed: 0:22:05 | left: 0:10:56 [I 2022-10-11 20:56:34] ...trap/engines/engine.py.229: process: 0.84735 | load: 0.00315 [I 2022-10-11 20:56:34] ...trap/engines/engine.py.230: loss: 0.13346 [I 2022-10-11 20:56:34] ...optimizers/trijoint.py.151: total_norm: 14.199983 [I 2022-10-11 20:56:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 1550/2304 [I 2022-10-11 20:56:42] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:41 [I 2022-10-11 20:56:42] ...trap/engines/engine.py.229: process: 0.84523 | load: 0.00313 [I 2022-10-11 20:56:42] ...trap/engines/engine.py.230: loss: 0.15523 [I 2022-10-11 20:56:42] ...optimizers/trijoint.py.151: total_norm: 15.506974 [I 2022-10-11 20:56:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 1560/2304 [I 2022-10-11 20:56:51] ...trap/engines/engine.py.228: elapsed: 0:22:22 | left: 0:10:36 [I 2022-10-11 20:56:51] ...trap/engines/engine.py.229: process: 0.85938 | load: 0.00312 [I 2022-10-11 20:56:51] ...trap/engines/engine.py.230: loss: 0.21604 [I 2022-10-11 20:56:51] ...optimizers/trijoint.py.151: total_norm: 19.253132 [I 2022-10-11 20:57:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1570/2304 [I 2022-10-11 20:57:00] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:24 [I 2022-10-11 20:57:00] ...trap/engines/engine.py.229: process: 0.85280 | load: 0.00314 [I 2022-10-11 20:57:00] ...trap/engines/engine.py.230: loss: 0.10660 [I 2022-10-11 20:57:00] ...optimizers/trijoint.py.151: total_norm: 21.212181 [I 2022-10-11 20:57:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1580/2304 [I 2022-10-11 20:57:08] ...trap/engines/engine.py.228: elapsed: 0:22:39 | left: 0:10:21 [I 2022-10-11 20:57:08] ...trap/engines/engine.py.229: process: 0.85407 | load: 0.00311 [I 2022-10-11 20:57:08] ...trap/engines/engine.py.230: loss: 0.11980 [I 2022-10-11 20:57:08] ...optimizers/trijoint.py.151: total_norm: 30.263271 [I 2022-10-11 20:57:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1590/2304 [I 2022-10-11 20:57:17] ...trap/engines/engine.py.228: elapsed: 0:22:48 | left: 0:10:19 [I 2022-10-11 20:57:17] ...trap/engines/engine.py.229: process: 0.88559 | load: 0.00396 [I 2022-10-11 20:57:17] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-11 20:57:17] ...optimizers/trijoint.py.151: total_norm: 27.234787 [I 2022-10-11 20:57:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 1600/2304 [I 2022-10-11 20:57:26] ...trap/engines/engine.py.228: elapsed: 0:22:56 | left: 0:10:22 [I 2022-10-11 20:57:26] ...trap/engines/engine.py.229: process: 0.87529 | load: 0.00397 [I 2022-10-11 20:57:26] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-11 20:57:26] ...optimizers/trijoint.py.151: total_norm: 12.790235 [I 2022-10-11 20:57:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1610/2304 [I 2022-10-11 20:57:35] ...trap/engines/engine.py.228: elapsed: 0:23:05 | left: 0:10:14 [I 2022-10-11 20:57:35] ...trap/engines/engine.py.229: process: 0.88064 | load: 0.00392 [I 2022-10-11 20:57:35] ...trap/engines/engine.py.230: loss: 0.11750 [I 2022-10-11 20:57:35] ...optimizers/trijoint.py.151: total_norm: 24.070229 [I 2022-10-11 20:57:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 1620/2304 [I 2022-10-11 20:57:43] ...trap/engines/engine.py.228: elapsed: 0:23:14 | left: 0:09:58 [I 2022-10-11 20:57:43] ...trap/engines/engine.py.229: process: 0.87630 | load: 0.00312 [I 2022-10-11 20:57:43] ...trap/engines/engine.py.230: loss: 0.09609 [I 2022-10-11 20:57:43] ...optimizers/trijoint.py.151: total_norm: 29.003855 [I 2022-10-11 20:57:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1630/2304 [I 2022-10-11 20:57:52] ...trap/engines/engine.py.228: elapsed: 0:23:23 | left: 0:09:55 [I 2022-10-11 20:57:52] ...trap/engines/engine.py.229: process: 0.87939 | load: 0.00391 [I 2022-10-11 20:57:52] ...trap/engines/engine.py.230: loss: 0.16031 [I 2022-10-11 20:57:52] ...optimizers/trijoint.py.151: total_norm: 9.106911 [I 2022-10-11 20:58:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1640/2304 [I 2022-10-11 20:58:01] ...trap/engines/engine.py.228: elapsed: 0:23:32 | left: 0:09:55 [I 2022-10-11 20:58:01] ...trap/engines/engine.py.229: process: 0.89014 | load: 0.00398 [I 2022-10-11 20:58:01] ...trap/engines/engine.py.230: loss: 0.09053 [I 2022-10-11 20:58:01] ...optimizers/trijoint.py.151: total_norm: 10.301999 [I 2022-10-11 20:58:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1650/2304 [I 2022-10-11 20:58:10] ...trap/engines/engine.py.228: elapsed: 0:23:41 | left: 0:09:37 [I 2022-10-11 20:58:10] ...trap/engines/engine.py.229: process: 0.88663 | load: 0.00392 [I 2022-10-11 20:58:10] ...trap/engines/engine.py.230: loss: 0.10171 [I 2022-10-11 20:58:10] ...optimizers/trijoint.py.151: total_norm: 11.017613 [I 2022-10-11 20:58:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 1660/2304 [I 2022-10-11 20:58:19] ...trap/engines/engine.py.228: elapsed: 0:23:50 | left: 0:09:29 [I 2022-10-11 20:58:19] ...trap/engines/engine.py.229: process: 0.87909 | load: 0.00386 [I 2022-10-11 20:58:19] ...trap/engines/engine.py.230: loss: 0.10608 [I 2022-10-11 20:58:19] ...optimizers/trijoint.py.151: total_norm: 13.701792 [I 2022-10-11 20:58:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 1670/2304 [I 2022-10-11 20:58:28] ...trap/engines/engine.py.228: elapsed: 0:23:58 | left: 0:09:16 [I 2022-10-11 20:58:28] ...trap/engines/engine.py.229: process: 0.87687 | load: 0.00379 [I 2022-10-11 20:58:28] ...trap/engines/engine.py.230: loss: 0.09776 [I 2022-10-11 20:58:28] ...optimizers/trijoint.py.151: total_norm: 9.398272 [I 2022-10-11 20:58:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1680/2304 [I 2022-10-11 20:58:36] ...trap/engines/engine.py.228: elapsed: 0:24:07 | left: 0:09:06 [I 2022-10-11 20:58:36] ...trap/engines/engine.py.229: process: 0.87470 | load: 0.00378 [I 2022-10-11 20:58:36] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-11 20:58:36] ...optimizers/trijoint.py.151: total_norm: 14.452224 [I 2022-10-11 20:58:45] ...trap/engines/engine.py.224: train: epoch 118 | batch 1690/2304 [I 2022-10-11 20:58:45] ...trap/engines/engine.py.228: elapsed: 0:24:16 | left: 0:08:46 [I 2022-10-11 20:58:45] ...trap/engines/engine.py.229: process: 0.84513 | load: 0.00298 [I 2022-10-11 20:58:45] ...trap/engines/engine.py.230: loss: 0.08744 [I 2022-10-11 20:58:45] ...optimizers/trijoint.py.151: total_norm: 10.766389 [I 2022-10-11 20:58:54] ...trap/engines/engine.py.224: train: epoch 118 | batch 1700/2304 [I 2022-10-11 20:58:54] ...trap/engines/engine.py.228: elapsed: 0:24:24 | left: 0:08:29 [I 2022-10-11 20:58:54] ...trap/engines/engine.py.229: process: 0.83280 | load: 0.00299 [I 2022-10-11 20:58:54] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-11 20:58:54] ...optimizers/trijoint.py.151: total_norm: 7.732598 [I 2022-10-11 20:59:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 1710/2304 [I 2022-10-11 20:59:02] ...trap/engines/engine.py.228: elapsed: 0:24:33 | left: 0:08:22 [I 2022-10-11 20:59:02] ...trap/engines/engine.py.229: process: 0.85227 | load: 0.00298 [I 2022-10-11 20:59:02] ...trap/engines/engine.py.230: loss: 0.17020 [I 2022-10-11 20:59:02] ...optimizers/trijoint.py.151: total_norm: 22.765463 [I 2022-10-11 20:59:11] ...trap/engines/engine.py.224: train: epoch 118 | batch 1720/2304 [I 2022-10-11 20:59:11] ...trap/engines/engine.py.228: elapsed: 0:24:41 | left: 0:08:13 [I 2022-10-11 20:59:11] ...trap/engines/engine.py.229: process: 0.84861 | load: 0.00300 [I 2022-10-11 20:59:11] ...trap/engines/engine.py.230: loss: 0.23312 [I 2022-10-11 20:59:11] ...optimizers/trijoint.py.151: total_norm: 25.008169 [I 2022-10-11 20:59:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 1730/2304 [I 2022-10-11 20:59:19] ...trap/engines/engine.py.228: elapsed: 0:24:50 | left: 0:08:06 [I 2022-10-11 20:59:19] ...trap/engines/engine.py.229: process: 0.86298 | load: 0.00299 [I 2022-10-11 20:59:19] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-11 20:59:19] ...optimizers/trijoint.py.151: total_norm: 11.893776 [I 2022-10-11 20:59:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 1740/2304 [I 2022-10-11 20:59:28] ...trap/engines/engine.py.228: elapsed: 0:24:58 | left: 0:07:59 [I 2022-10-11 20:59:28] ...trap/engines/engine.py.229: process: 0.84727 | load: 0.00301 [I 2022-10-11 20:59:28] ...trap/engines/engine.py.230: loss: 0.10037 [I 2022-10-11 20:59:28] ...optimizers/trijoint.py.151: total_norm: 11.445225 [I 2022-10-11 20:59:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1750/2304 [I 2022-10-11 20:59:36] ...trap/engines/engine.py.228: elapsed: 0:25:07 | left: 0:07:47 [I 2022-10-11 20:59:36] ...trap/engines/engine.py.229: process: 0.82530 | load: 0.00300 [I 2022-10-11 20:59:36] ...trap/engines/engine.py.230: loss: 0.13302 [I 2022-10-11 20:59:36] ...optimizers/trijoint.py.151: total_norm: 11.103386 [I 2022-10-11 20:59:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1760/2304 [I 2022-10-11 20:59:44] ...trap/engines/engine.py.228: elapsed: 0:25:15 | left: 0:07:37 [I 2022-10-11 20:59:44] ...trap/engines/engine.py.229: process: 0.83480 | load: 0.00298 [I 2022-10-11 20:59:44] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-11 20:59:44] ...optimizers/trijoint.py.151: total_norm: 12.405419 [I 2022-10-11 20:59:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 1770/2304 [I 2022-10-11 20:59:53] ...trap/engines/engine.py.228: elapsed: 0:25:24 | left: 0:07:29 [I 2022-10-11 20:59:53] ...trap/engines/engine.py.229: process: 0.84791 | load: 0.00297 [I 2022-10-11 20:59:53] ...trap/engines/engine.py.230: loss: 0.14592 [I 2022-10-11 20:59:53] ...optimizers/trijoint.py.151: total_norm: 13.699074 [I 2022-10-11 21:00:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1780/2304 [I 2022-10-11 21:00:01] ...trap/engines/engine.py.228: elapsed: 0:25:32 | left: 0:07:22 [I 2022-10-11 21:00:01] ...trap/engines/engine.py.229: process: 0.84748 | load: 0.00301 [I 2022-10-11 21:00:01] ...trap/engines/engine.py.230: loss: 0.24293 [I 2022-10-11 21:00:01] ...optimizers/trijoint.py.151: total_norm: 32.869400 [I 2022-10-11 21:00:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1790/2304 [I 2022-10-11 21:00:10] ...trap/engines/engine.py.228: elapsed: 0:25:41 | left: 0:07:14 [I 2022-10-11 21:00:10] ...trap/engines/engine.py.229: process: 0.84645 | load: 0.00301 [I 2022-10-11 21:00:10] ...trap/engines/engine.py.230: loss: 0.10694 [I 2022-10-11 21:00:10] ...optimizers/trijoint.py.151: total_norm: 11.466564 [I 2022-10-11 21:00:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1800/2304 [I 2022-10-11 21:00:18] ...trap/engines/engine.py.228: elapsed: 0:25:49 | left: 0:07:08 [I 2022-10-11 21:00:18] ...trap/engines/engine.py.229: process: 0.84332 | load: 0.00312 [I 2022-10-11 21:00:18] ...trap/engines/engine.py.230: loss: 0.15429 [I 2022-10-11 21:00:18] ...optimizers/trijoint.py.151: total_norm: 18.610769 [I 2022-10-11 21:00:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1810/2304 [I 2022-10-11 21:00:27] ...trap/engines/engine.py.228: elapsed: 0:25:58 | left: 0:07:03 [I 2022-10-11 21:00:27] ...trap/engines/engine.py.229: process: 0.84936 | load: 0.00314 [I 2022-10-11 21:00:27] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-11 21:00:27] ...optimizers/trijoint.py.151: total_norm: 14.657511 [I 2022-10-11 21:00:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1820/2304 [I 2022-10-11 21:00:36] ...trap/engines/engine.py.228: elapsed: 0:26:06 | left: 0:06:53 [I 2022-10-11 21:00:36] ...trap/engines/engine.py.229: process: 0.84473 | load: 0.00311 [I 2022-10-11 21:00:36] ...trap/engines/engine.py.230: loss: 0.11016 [I 2022-10-11 21:00:36] ...optimizers/trijoint.py.151: total_norm: 13.833930 [I 2022-10-11 21:00:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1830/2304 [I 2022-10-11 21:00:44] ...trap/engines/engine.py.228: elapsed: 0:26:15 | left: 0:06:46 [I 2022-10-11 21:00:44] ...trap/engines/engine.py.229: process: 0.87262 | load: 0.00314 [I 2022-10-11 21:00:44] ...trap/engines/engine.py.230: loss: 0.19763 [I 2022-10-11 21:00:44] ...optimizers/trijoint.py.151: total_norm: 16.560482 [I 2022-10-11 21:01:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 1840/2304 [I 2022-10-11 21:01:13] ...trap/engines/engine.py.228: elapsed: 0:26:43 | left: 0:10:44 [I 2022-10-11 21:01:13] ...trap/engines/engine.py.229: process: 0.84072 | load: 0.00315 [I 2022-10-11 21:01:13] ...trap/engines/engine.py.230: loss: 0.14413 [I 2022-10-11 21:01:13] ...optimizers/trijoint.py.151: total_norm: 12.713755 [I 2022-10-11 21:01:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 1850/2304 [I 2022-10-11 21:01:21] ...trap/engines/engine.py.228: elapsed: 0:26:52 | left: 0:06:54 [I 2022-10-11 21:01:21] ...trap/engines/engine.py.229: process: 0.84188 | load: 0.00313 [I 2022-10-11 21:01:21] ...trap/engines/engine.py.230: loss: 0.11959 [I 2022-10-11 21:01:21] ...optimizers/trijoint.py.151: total_norm: 8.380679 [I 2022-10-11 21:01:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 1860/2304 [I 2022-10-11 21:01:30] ...trap/engines/engine.py.228: elapsed: 0:27:01 | left: 0:06:25 [I 2022-10-11 21:01:30] ...trap/engines/engine.py.229: process: 0.86634 | load: 0.00319 [I 2022-10-11 21:01:30] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-11 21:01:30] ...optimizers/trijoint.py.151: total_norm: 18.665251 [I 2022-10-11 21:01:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 1870/2304 [I 2022-10-11 21:01:38] ...trap/engines/engine.py.228: elapsed: 0:27:09 | left: 0:06:11 [I 2022-10-11 21:01:38] ...trap/engines/engine.py.229: process: 0.84814 | load: 0.00316 [I 2022-10-11 21:01:38] ...trap/engines/engine.py.230: loss: 0.17654 [I 2022-10-11 21:01:38] ...optimizers/trijoint.py.151: total_norm: 12.595483 [I 2022-10-11 21:01:47] ...trap/engines/engine.py.224: train: epoch 118 | batch 1880/2304 [I 2022-10-11 21:01:47] ...trap/engines/engine.py.228: elapsed: 0:27:18 | left: 0:06:06 [I 2022-10-11 21:01:47] ...trap/engines/engine.py.229: process: 0.86512 | load: 0.00311 [I 2022-10-11 21:01:47] ...trap/engines/engine.py.230: loss: 0.15075 [I 2022-10-11 21:01:47] ...optimizers/trijoint.py.151: total_norm: 10.626527 [I 2022-10-11 21:01:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 1890/2304 [I 2022-10-11 21:01:56] ...trap/engines/engine.py.228: elapsed: 0:27:26 | left: 0:05:57 [I 2022-10-11 21:01:56] ...trap/engines/engine.py.229: process: 0.85915 | load: 0.00321 [I 2022-10-11 21:01:56] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-11 21:01:56] ...optimizers/trijoint.py.151: total_norm: 12.378204 [I 2022-10-11 21:02:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 1900/2304 [I 2022-10-11 21:02:04] ...trap/engines/engine.py.228: elapsed: 0:27:35 | left: 0:05:49 [I 2022-10-11 21:02:04] ...trap/engines/engine.py.229: process: 0.86117 | load: 0.00313 [I 2022-10-11 21:02:04] ...trap/engines/engine.py.230: loss: 0.17929 [I 2022-10-11 21:02:04] ...optimizers/trijoint.py.151: total_norm: 14.560319 [I 2022-10-11 21:02:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 1910/2304 [I 2022-10-11 21:02:13] ...trap/engines/engine.py.228: elapsed: 0:27:44 | left: 0:05:42 [I 2022-10-11 21:02:13] ...trap/engines/engine.py.229: process: 0.86118 | load: 0.00313 [I 2022-10-11 21:02:13] ...trap/engines/engine.py.230: loss: 0.09142 [I 2022-10-11 21:02:13] ...optimizers/trijoint.py.151: total_norm: 13.789364 [I 2022-10-11 21:02:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 1920/2304 [I 2022-10-11 21:02:22] ...trap/engines/engine.py.228: elapsed: 0:27:53 | left: 0:05:29 [I 2022-10-11 21:02:22] ...trap/engines/engine.py.229: process: 0.85606 | load: 0.00312 [I 2022-10-11 21:02:22] ...trap/engines/engine.py.230: loss: 0.09750 [I 2022-10-11 21:02:22] ...optimizers/trijoint.py.151: total_norm: 13.502710 [I 2022-10-11 21:02:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 1930/2304 [I 2022-10-11 21:02:30] ...trap/engines/engine.py.228: elapsed: 0:28:01 | left: 0:05:20 [I 2022-10-11 21:02:30] ...trap/engines/engine.py.229: process: 0.84644 | load: 0.00311 [I 2022-10-11 21:02:30] ...trap/engines/engine.py.230: loss: 0.10162 [I 2022-10-11 21:02:30] ...optimizers/trijoint.py.151: total_norm: 12.972622 [I 2022-10-11 21:02:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 1940/2304 [I 2022-10-11 21:02:39] ...trap/engines/engine.py.228: elapsed: 0:28:10 | left: 0:05:11 [I 2022-10-11 21:02:39] ...trap/engines/engine.py.229: process: 0.85796 | load: 0.00311 [I 2022-10-11 21:02:39] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-11 21:02:39] ...optimizers/trijoint.py.151: total_norm: 13.769781 [I 2022-10-11 21:02:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 1950/2304 [I 2022-10-11 21:02:48] ...trap/engines/engine.py.228: elapsed: 0:28:18 | left: 0:05:02 [I 2022-10-11 21:02:48] ...trap/engines/engine.py.229: process: 0.84995 | load: 0.00314 [I 2022-10-11 21:02:48] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-11 21:02:48] ...optimizers/trijoint.py.151: total_norm: 10.054279 [I 2022-10-11 21:02:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 1960/2304 [I 2022-10-11 21:02:56] ...trap/engines/engine.py.228: elapsed: 0:28:27 | left: 0:04:53 [I 2022-10-11 21:02:56] ...trap/engines/engine.py.229: process: 0.85221 | load: 0.00317 [I 2022-10-11 21:02:56] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-11 21:02:56] ...optimizers/trijoint.py.151: total_norm: 42.280838 [I 2022-10-11 21:03:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 1970/2304 [I 2022-10-11 21:03:05] ...trap/engines/engine.py.228: elapsed: 0:28:36 | left: 0:04:57 [I 2022-10-11 21:03:05] ...trap/engines/engine.py.229: process: 0.91343 | load: 0.00399 [I 2022-10-11 21:03:05] ...trap/engines/engine.py.230: loss: 0.12765 [I 2022-10-11 21:03:05] ...optimizers/trijoint.py.151: total_norm: 16.297426 [I 2022-10-11 21:03:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 1980/2304 [I 2022-10-11 21:03:14] ...trap/engines/engine.py.228: elapsed: 0:28:45 | left: 0:04:48 [I 2022-10-11 21:03:14] ...trap/engines/engine.py.229: process: 0.89313 | load: 0.00396 [I 2022-10-11 21:03:14] ...trap/engines/engine.py.230: loss: 0.13570 [I 2022-10-11 21:03:14] ...optimizers/trijoint.py.151: total_norm: 27.770416 [I 2022-10-11 21:03:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 1990/2304 [I 2022-10-11 21:03:23] ...trap/engines/engine.py.228: elapsed: 0:28:53 | left: 0:04:39 [I 2022-10-11 21:03:23] ...trap/engines/engine.py.229: process: 0.89239 | load: 0.00397 [I 2022-10-11 21:03:23] ...trap/engines/engine.py.230: loss: 0.07579 [I 2022-10-11 21:03:23] ...optimizers/trijoint.py.151: total_norm: 9.165324 [I 2022-10-11 21:03:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 2000/2304 [I 2022-10-11 21:03:31] ...trap/engines/engine.py.228: elapsed: 0:29:02 | left: 0:04:24 [I 2022-10-11 21:03:31] ...trap/engines/engine.py.229: process: 0.84761 | load: 0.00317 [I 2022-10-11 21:03:31] ...trap/engines/engine.py.230: loss: 0.09611 [I 2022-10-11 21:03:31] ...optimizers/trijoint.py.151: total_norm: 9.946490 [I 2022-10-11 21:03:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 2010/2304 [I 2022-10-11 21:03:40] ...trap/engines/engine.py.228: elapsed: 0:29:11 | left: 0:04:11 [I 2022-10-11 21:03:40] ...trap/engines/engine.py.229: process: 0.86312 | load: 0.00312 [I 2022-10-11 21:03:40] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-11 21:03:40] ...optimizers/trijoint.py.151: total_norm: 34.061298 [I 2022-10-11 21:03:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 2020/2304 [I 2022-10-11 21:03:49] ...trap/engines/engine.py.228: elapsed: 0:29:19 | left: 0:04:04 [I 2022-10-11 21:03:49] ...trap/engines/engine.py.229: process: 0.87058 | load: 0.00319 [I 2022-10-11 21:03:49] ...trap/engines/engine.py.230: loss: 0.14793 [I 2022-10-11 21:03:49] ...optimizers/trijoint.py.151: total_norm: 18.749184 [I 2022-10-11 21:03:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 2030/2304 [I 2022-10-11 21:03:57] ...trap/engines/engine.py.228: elapsed: 0:29:28 | left: 0:03:52 [I 2022-10-11 21:03:57] ...trap/engines/engine.py.229: process: 0.85747 | load: 0.00312 [I 2022-10-11 21:03:57] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-11 21:03:57] ...optimizers/trijoint.py.151: total_norm: 12.195150 [I 2022-10-11 21:04:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 2040/2304 [I 2022-10-11 21:04:06] ...trap/engines/engine.py.228: elapsed: 0:29:36 | left: 0:03:44 [I 2022-10-11 21:04:06] ...trap/engines/engine.py.229: process: 0.84706 | load: 0.00311 [I 2022-10-11 21:04:06] ...trap/engines/engine.py.230: loss: 0.08510 [I 2022-10-11 21:04:06] ...optimizers/trijoint.py.151: total_norm: 19.137856 [I 2022-10-11 21:04:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 2050/2304 [I 2022-10-11 21:04:14] ...trap/engines/engine.py.228: elapsed: 0:29:45 | left: 0:03:34 [I 2022-10-11 21:04:14] ...trap/engines/engine.py.229: process: 0.85072 | load: 0.00296 [I 2022-10-11 21:04:14] ...trap/engines/engine.py.230: loss: 0.12326 [I 2022-10-11 21:04:14] ...optimizers/trijoint.py.151: total_norm: 17.980236 [I 2022-10-11 21:04:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 2060/2304 [I 2022-10-11 21:04:23] ...trap/engines/engine.py.228: elapsed: 0:29:53 | left: 0:03:26 [I 2022-10-11 21:04:23] ...trap/engines/engine.py.229: process: 0.85749 | load: 0.00300 [I 2022-10-11 21:04:23] ...trap/engines/engine.py.230: loss: 0.08653 [I 2022-10-11 21:04:23] ...optimizers/trijoint.py.151: total_norm: 10.511639 [I 2022-10-11 21:04:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 2070/2304 [I 2022-10-11 21:04:31] ...trap/engines/engine.py.228: elapsed: 0:30:02 | left: 0:03:19 [I 2022-10-11 21:04:31] ...trap/engines/engine.py.229: process: 0.86669 | load: 0.00300 [I 2022-10-11 21:04:31] ...trap/engines/engine.py.230: loss: 0.08951 [I 2022-10-11 21:04:31] ...optimizers/trijoint.py.151: total_norm: 15.710794 [I 2022-10-11 21:04:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 2080/2304 [I 2022-10-11 21:04:40] ...trap/engines/engine.py.228: elapsed: 0:30:10 | left: 0:03:09 [I 2022-10-11 21:04:40] ...trap/engines/engine.py.229: process: 0.84544 | load: 0.00305 [I 2022-10-11 21:04:40] ...trap/engines/engine.py.230: loss: 0.06614 [I 2022-10-11 21:04:40] ...optimizers/trijoint.py.151: total_norm: 17.184404 [I 2022-10-11 21:04:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 2090/2304 [I 2022-10-11 21:04:48] ...trap/engines/engine.py.228: elapsed: 0:30:19 | left: 0:03:01 [I 2022-10-11 21:04:48] ...trap/engines/engine.py.229: process: 0.85526 | load: 0.00303 [I 2022-10-11 21:04:48] ...trap/engines/engine.py.230: loss: 0.08015 [I 2022-10-11 21:04:48] ...optimizers/trijoint.py.151: total_norm: 17.002516 [I 2022-10-11 21:04:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 2100/2304 [I 2022-10-11 21:04:57] ...trap/engines/engine.py.228: elapsed: 0:30:27 | left: 0:02:52 [I 2022-10-11 21:04:57] ...trap/engines/engine.py.229: process: 0.83382 | load: 0.00299 [I 2022-10-11 21:04:57] ...trap/engines/engine.py.230: loss: 0.09473 [I 2022-10-11 21:04:57] ...optimizers/trijoint.py.151: total_norm: 17.730843 [I 2022-10-11 21:05:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 2110/2304 [I 2022-10-11 21:05:05] ...trap/engines/engine.py.228: elapsed: 0:30:36 | left: 0:02:44 [I 2022-10-11 21:05:05] ...trap/engines/engine.py.229: process: 0.84643 | load: 0.00298 [I 2022-10-11 21:05:05] ...trap/engines/engine.py.230: loss: 0.14525 [I 2022-10-11 21:05:05] ...optimizers/trijoint.py.151: total_norm: 27.190720 [I 2022-10-11 21:05:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 2120/2304 [I 2022-10-11 21:05:14] ...trap/engines/engine.py.228: elapsed: 0:30:44 | left: 0:02:36 [I 2022-10-11 21:05:14] ...trap/engines/engine.py.229: process: 0.84900 | load: 0.00304 [I 2022-10-11 21:05:14] ...trap/engines/engine.py.230: loss: 0.11801 [I 2022-10-11 21:05:14] ...optimizers/trijoint.py.151: total_norm: 13.915820 [I 2022-10-11 21:05:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 2130/2304 [I 2022-10-11 21:05:22] ...trap/engines/engine.py.228: elapsed: 0:30:53 | left: 0:02:28 [I 2022-10-11 21:05:22] ...trap/engines/engine.py.229: process: 0.85712 | load: 0.00305 [I 2022-10-11 21:05:22] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-11 21:05:22] ...optimizers/trijoint.py.151: total_norm: 21.226414 [I 2022-10-11 21:05:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 2140/2304 [I 2022-10-11 21:05:31] ...trap/engines/engine.py.228: elapsed: 0:31:01 | left: 0:02:19 [I 2022-10-11 21:05:31] ...trap/engines/engine.py.229: process: 0.84762 | load: 0.00298 [I 2022-10-11 21:05:31] ...trap/engines/engine.py.230: loss: 0.08690 [I 2022-10-11 21:05:31] ...optimizers/trijoint.py.151: total_norm: 13.063771 [I 2022-10-11 21:05:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 2150/2304 [I 2022-10-11 21:05:39] ...trap/engines/engine.py.228: elapsed: 0:31:10 | left: 0:02:11 [I 2022-10-11 21:05:39] ...trap/engines/engine.py.229: process: 0.84549 | load: 0.00300 [I 2022-10-11 21:05:39] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-11 21:05:39] ...optimizers/trijoint.py.151: total_norm: 29.496828 [I 2022-10-11 21:05:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 2160/2304 [I 2022-10-11 21:05:48] ...trap/engines/engine.py.228: elapsed: 0:31:18 | left: 0:02:01 [I 2022-10-11 21:05:48] ...trap/engines/engine.py.229: process: 0.83803 | load: 0.00299 [I 2022-10-11 21:05:48] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-11 21:05:48] ...optimizers/trijoint.py.151: total_norm: 9.869925 [I 2022-10-11 21:05:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 2170/2304 [I 2022-10-11 21:05:56] ...trap/engines/engine.py.228: elapsed: 0:31:27 | left: 0:01:53 [I 2022-10-11 21:05:56] ...trap/engines/engine.py.229: process: 0.83586 | load: 0.00302 [I 2022-10-11 21:05:56] ...trap/engines/engine.py.230: loss: 0.08159 [I 2022-10-11 21:05:56] ...optimizers/trijoint.py.151: total_norm: 10.351725 [I 2022-10-11 21:06:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 2180/2304 [I 2022-10-11 21:06:05] ...trap/engines/engine.py.228: elapsed: 0:31:35 | left: 0:01:44 [I 2022-10-11 21:06:05] ...trap/engines/engine.py.229: process: 0.84160 | load: 0.00300 [I 2022-10-11 21:06:05] ...trap/engines/engine.py.230: loss: 0.11336 [I 2022-10-11 21:06:05] ...optimizers/trijoint.py.151: total_norm: 29.766893 [I 2022-10-11 21:06:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 2190/2304 [I 2022-10-11 21:06:13] ...trap/engines/engine.py.228: elapsed: 0:31:44 | left: 0:01:36 [I 2022-10-11 21:06:13] ...trap/engines/engine.py.229: process: 0.84234 | load: 0.00302 [I 2022-10-11 21:06:13] ...trap/engines/engine.py.230: loss: 0.11924 [I 2022-10-11 21:06:13] ...optimizers/trijoint.py.151: total_norm: 34.445591 [I 2022-10-11 21:06:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 2200/2304 [I 2022-10-11 21:06:22] ...trap/engines/engine.py.228: elapsed: 0:31:52 | left: 0:01:28 [I 2022-10-11 21:06:22] ...trap/engines/engine.py.229: process: 0.85109 | load: 0.00318 [I 2022-10-11 21:06:22] ...trap/engines/engine.py.230: loss: 0.12114 [I 2022-10-11 21:06:22] ...optimizers/trijoint.py.151: total_norm: 13.687939 [I 2022-10-11 21:06:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 2210/2304 [I 2022-10-11 21:06:30] ...trap/engines/engine.py.228: elapsed: 0:32:01 | left: 0:01:20 [I 2022-10-11 21:06:30] ...trap/engines/engine.py.229: process: 0.85111 | load: 0.00312 [I 2022-10-11 21:06:30] ...trap/engines/engine.py.230: loss: 0.12713 [I 2022-10-11 21:06:30] ...optimizers/trijoint.py.151: total_norm: 19.677061 [I 2022-10-11 21:06:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 2220/2304 [I 2022-10-11 21:06:39] ...trap/engines/engine.py.228: elapsed: 0:32:10 | left: 0:01:12 [I 2022-10-11 21:06:39] ...trap/engines/engine.py.229: process: 0.86860 | load: 0.00315 [I 2022-10-11 21:06:39] ...trap/engines/engine.py.230: loss: 0.08354 [I 2022-10-11 21:06:39] ...optimizers/trijoint.py.151: total_norm: 10.057283 [I 2022-10-11 21:06:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 2230/2304 [I 2022-10-11 21:06:48] ...trap/engines/engine.py.228: elapsed: 0:32:18 | left: 0:01:05 [I 2022-10-11 21:06:48] ...trap/engines/engine.py.229: process: 0.88822 | load: 0.00399 [I 2022-10-11 21:06:48] ...trap/engines/engine.py.230: loss: 0.11918 [I 2022-10-11 21:06:48] ...optimizers/trijoint.py.151: total_norm: 10.939816 [I 2022-10-11 21:06:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 2240/2304 [I 2022-10-11 21:06:57] ...trap/engines/engine.py.228: elapsed: 0:32:27 | left: 0:00:56 [I 2022-10-11 21:06:57] ...trap/engines/engine.py.229: process: 0.88762 | load: 0.00402 [I 2022-10-11 21:06:57] ...trap/engines/engine.py.230: loss: 0.17976 [I 2022-10-11 21:06:57] ...optimizers/trijoint.py.151: total_norm: 14.826431 [I 2022-10-11 21:07:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 2250/2304 [I 2022-10-11 21:07:06] ...trap/engines/engine.py.228: elapsed: 0:32:36 | left: 0:00:47 [I 2022-10-11 21:07:06] ...trap/engines/engine.py.229: process: 0.88044 | load: 0.00395 [I 2022-10-11 21:07:06] ...trap/engines/engine.py.230: loss: 0.11726 [I 2022-10-11 21:07:06] ...optimizers/trijoint.py.151: total_norm: 20.795586 [I 2022-10-11 21:07:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 2260/2304 [I 2022-10-11 21:07:14] ...trap/engines/engine.py.228: elapsed: 0:32:45 | left: 0:00:38 [I 2022-10-11 21:07:14] ...trap/engines/engine.py.229: process: 0.87453 | load: 0.00316 [I 2022-10-11 21:07:14] ...trap/engines/engine.py.230: loss: 0.09095 [I 2022-10-11 21:07:14] ...optimizers/trijoint.py.151: total_norm: 15.450104 [I 2022-10-11 21:07:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 2270/2304 [I 2022-10-11 21:07:23] ...trap/engines/engine.py.228: elapsed: 0:32:54 | left: 0:00:29 [I 2022-10-11 21:07:23] ...trap/engines/engine.py.229: process: 0.83716 | load: 0.00314 [I 2022-10-11 21:07:23] ...trap/engines/engine.py.230: loss: 0.11145 [I 2022-10-11 21:07:23] ...optimizers/trijoint.py.151: total_norm: 9.925090 [I 2022-10-11 21:07:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 2280/2304 [I 2022-10-11 21:07:32] ...trap/engines/engine.py.228: elapsed: 0:33:02 | left: 0:00:20 [I 2022-10-11 21:07:32] ...trap/engines/engine.py.229: process: 0.86282 | load: 0.00314 [I 2022-10-11 21:07:32] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-11 21:07:32] ...optimizers/trijoint.py.151: total_norm: 16.349411 [I 2022-10-11 21:07:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 2290/2304 [I 2022-10-11 21:07:40] ...trap/engines/engine.py.228: elapsed: 0:33:11 | left: 0:00:12 [I 2022-10-11 21:07:40] ...trap/engines/engine.py.229: process: 0.86330 | load: 0.00312 [I 2022-10-11 21:07:40] ...trap/engines/engine.py.230: loss: 0.08742 [I 2022-10-11 21:07:40] ...optimizers/trijoint.py.151: total_norm: 26.211458 [I 2022-10-11 21:07:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 2300/2304 [I 2022-10-11 21:07:49] ...trap/engines/engine.py.228: elapsed: 0:33:20 | left: 0:00:03 [I 2022-10-11 21:07:49] ...trap/engines/engine.py.229: process: 0.87446 | load: 0.00019 [I 2022-10-11 21:07:49] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-11 21:07:49] ...optimizers/trijoint.py.151: total_norm: 37.869858 [I 2022-10-11 21:07:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 2304/2304 [I 2022-10-11 21:07:53] ...trap/engines/engine.py.228: elapsed: 0:33:23 | left: 0:00:00 [I 2022-10-11 21:07:53] ...trap/engines/engine.py.229: process: 0.87546 | load: 0.00021 [I 2022-10-11 21:07:53] ...trap/engines/engine.py.230: loss: 0.09634 [I 2022-10-11 21:07:53] ...optimizers/trijoint.py.151: total_norm: 13.156851 [S 2022-10-11 21:07:54] ...trap/engines/engine.py.129: train_epoch.epoch: 118 [S 2022-10-11 21:07:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.11737028596813663 [I 2022-10-11 21:07:54] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 21:07:56] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 118 [I 2022-10-11 21:07:56] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 21:07:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 21:07:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 21:07:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 21:08:02] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 21:08:03] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/510 [I 2022-10-11 21:08:03] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:59:07 [I 2022-10-11 21:08:03] ...trap/engines/engine.py.326: process: 6.95500 | load: 2.38223 [I 2022-10-11 21:08:07] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/510 [I 2022-10-11 21:08:07] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:28 [I 2022-10-11 21:08:07] ...trap/engines/engine.py.326: process: 0.43287 | load: 0.00021 [I 2022-10-11 21:08:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/510 [I 2022-10-11 21:08:12] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:14 [I 2022-10-11 21:08:12] ...trap/engines/engine.py.326: process: 0.46449 | load: 0.00022 [I 2022-10-11 21:08:16] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/510 [I 2022-10-11 21:08:16] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:37 [I 2022-10-11 21:08:16] ...trap/engines/engine.py.326: process: 0.44453 | load: 0.00021 [I 2022-10-11 21:08:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/510 [I 2022-10-11 21:08:20] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:25 [I 2022-10-11 21:08:20] ...trap/engines/engine.py.326: process: 0.44228 | load: 0.00021 [I 2022-10-11 21:08:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/510 [I 2022-10-11 21:08:25] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:29 [I 2022-10-11 21:08:25] ...trap/engines/engine.py.326: process: 0.45466 | load: 0.00023 [I 2022-10-11 21:08:29] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/510 [I 2022-10-11 21:08:29] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:03:18 [I 2022-10-11 21:08:29] ...trap/engines/engine.py.326: process: 0.42793 | load: 0.00021 [I 2022-10-11 21:08:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/510 [I 2022-10-11 21:08:34] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:26 [I 2022-10-11 21:08:34] ...trap/engines/engine.py.326: process: 0.45275 | load: 0.00021 [I 2022-10-11 21:08:39] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/510 [I 2022-10-11 21:08:39] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:03:15 [I 2022-10-11 21:08:39] ...trap/engines/engine.py.326: process: 0.42383 | load: 0.00021 [I 2022-10-11 21:08:43] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/510 [I 2022-10-11 21:08:43] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:03:11 [I 2022-10-11 21:08:43] ...trap/engines/engine.py.326: process: 0.45640 | load: 0.00024 [I 2022-10-11 21:08:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/510 [I 2022-10-11 21:08:48] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:00 [I 2022-10-11 21:08:48] ...trap/engines/engine.py.326: process: 0.32693 | load: 0.00022 [I 2022-10-11 21:08:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/510 [I 2022-10-11 21:08:51] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:02:19 [I 2022-10-11 21:08:51] ...trap/engines/engine.py.326: process: 0.33898 | load: 0.00024 [I 2022-10-11 21:08:55] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/510 [I 2022-10-11 21:08:55] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:02:12 [I 2022-10-11 21:08:55] ...trap/engines/engine.py.326: process: 0.34068 | load: 0.00024 [I 2022-10-11 21:08:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/510 [I 2022-10-11 21:08:58] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:07 [I 2022-10-11 21:08:58] ...trap/engines/engine.py.326: process: 0.32937 | load: 0.00023 [I 2022-10-11 21:09:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/510 [I 2022-10-11 21:09:01] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:02:05 [I 2022-10-11 21:09:01] ...trap/engines/engine.py.326: process: 0.34139 | load: 0.00023 [I 2022-10-11 21:09:05] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/510 [I 2022-10-11 21:09:05] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:02:00 [I 2022-10-11 21:09:05] ...trap/engines/engine.py.326: process: 0.33134 | load: 0.00023 [I 2022-10-11 21:09:08] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/510 [I 2022-10-11 21:09:08] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:58 [I 2022-10-11 21:09:08] ...trap/engines/engine.py.326: process: 0.33608 | load: 0.00024 [I 2022-10-11 21:09:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/510 [I 2022-10-11 21:09:12] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:54 [I 2022-10-11 21:09:12] ...trap/engines/engine.py.326: process: 0.34199 | load: 0.00023 [I 2022-10-11 21:09:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/510 [I 2022-10-11 21:09:15] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:51 [I 2022-10-11 21:09:15] ...trap/engines/engine.py.326: process: 0.34111 | load: 0.00025 [I 2022-10-11 21:09:18] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/510 [I 2022-10-11 21:09:18] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:48 [I 2022-10-11 21:09:18] ...trap/engines/engine.py.326: process: 0.33792 | load: 0.00025 [I 2022-10-11 21:09:22] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/510 [I 2022-10-11 21:09:22] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:44 [I 2022-10-11 21:09:22] ...trap/engines/engine.py.326: process: 0.33451 | load: 0.00025 [I 2022-10-11 21:09:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/510 [I 2022-10-11 21:09:25] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:40 [I 2022-10-11 21:09:25] ...trap/engines/engine.py.326: process: 0.34459 | load: 0.00024 [I 2022-10-11 21:09:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/510 [I 2022-10-11 21:09:28] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:38 [I 2022-10-11 21:09:28] ...trap/engines/engine.py.326: process: 0.34186 | load: 0.00024 [I 2022-10-11 21:09:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/510 [I 2022-10-11 21:09:32] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:34 [I 2022-10-11 21:09:32] ...trap/engines/engine.py.326: process: 0.33360 | load: 0.00025 [I 2022-10-11 21:09:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/510 [I 2022-10-11 21:09:35] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:01:31 [I 2022-10-11 21:09:35] ...trap/engines/engine.py.326: process: 0.34645 | load: 0.00025 [I 2022-10-11 21:09:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/510 [I 2022-10-11 21:09:38] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:01:27 [I 2022-10-11 21:09:38] ...trap/engines/engine.py.326: process: 0.33215 | load: 0.00023 [I 2022-10-11 21:09:42] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/510 [I 2022-10-11 21:09:42] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:01:24 [I 2022-10-11 21:09:42] ...trap/engines/engine.py.326: process: 0.33358 | load: 0.00024 [I 2022-10-11 21:09:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/510 [I 2022-10-11 21:09:45] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:01:20 [I 2022-10-11 21:09:45] ...trap/engines/engine.py.326: process: 0.33650 | load: 0.00024 [I 2022-10-11 21:09:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/510 [I 2022-10-11 21:09:49] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:01:17 [I 2022-10-11 21:09:49] ...trap/engines/engine.py.326: process: 0.34217 | load: 0.00024 [I 2022-10-11 21:09:52] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/510 [I 2022-10-11 21:09:52] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:01:14 [I 2022-10-11 21:09:52] ...trap/engines/engine.py.326: process: 0.33459 | load: 0.00023 [I 2022-10-11 21:09:55] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/510 [I 2022-10-11 21:09:55] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:01:10 [I 2022-10-11 21:09:55] ...trap/engines/engine.py.326: process: 0.32892 | load: 0.00023 [I 2022-10-11 21:09:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/510 [I 2022-10-11 21:09:59] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:01:07 [I 2022-10-11 21:09:59] ...trap/engines/engine.py.326: process: 0.33332 | load: 0.00024 [I 2022-10-11 21:10:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/510 [I 2022-10-11 21:10:02] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:01:04 [I 2022-10-11 21:10:02] ...trap/engines/engine.py.326: process: 0.33992 | load: 0.00024 [I 2022-10-11 21:10:05] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/510 [I 2022-10-11 21:10:05] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:01:00 [I 2022-10-11 21:10:05] ...trap/engines/engine.py.326: process: 0.33922 | load: 0.00023 [I 2022-10-11 21:10:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/510 [I 2022-10-11 21:10:09] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:56 [I 2022-10-11 21:10:09] ...trap/engines/engine.py.326: process: 0.33533 | load: 0.00023 [I 2022-10-11 21:10:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/510 [I 2022-10-11 21:10:12] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:53 [I 2022-10-11 21:10:12] ...trap/engines/engine.py.326: process: 0.32770 | load: 0.00023 [I 2022-10-11 21:10:16] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/510 [I 2022-10-11 21:10:16] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:50 [I 2022-10-11 21:10:16] ...trap/engines/engine.py.326: process: 0.33886 | load: 0.00022 [I 2022-10-11 21:10:19] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/510 [I 2022-10-11 21:10:19] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:45 [I 2022-10-11 21:10:19] ...trap/engines/engine.py.326: process: 0.31592 | load: 0.00020 [I 2022-10-11 21:10:22] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/510 [I 2022-10-11 21:10:22] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:41 [I 2022-10-11 21:10:22] ...trap/engines/engine.py.326: process: 0.31626 | load: 0.00020 [I 2022-10-11 21:10:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/510 [I 2022-10-11 21:10:25] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:38 [I 2022-10-11 21:10:25] ...trap/engines/engine.py.326: process: 0.32219 | load: 0.00019 [I 2022-10-11 21:10:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/510 [I 2022-10-11 21:10:28] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:35 [I 2022-10-11 21:10:28] ...trap/engines/engine.py.326: process: 0.32781 | load: 0.00020 [I 2022-10-11 21:10:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/510 [I 2022-10-11 21:10:32] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:32 [I 2022-10-11 21:10:32] ...trap/engines/engine.py.326: process: 0.32502 | load: 0.00019 [I 2022-10-11 21:10:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/510 [I 2022-10-11 21:10:35] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:28 [I 2022-10-11 21:10:35] ...trap/engines/engine.py.326: process: 0.31931 | load: 0.00019 [I 2022-10-11 21:10:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/510 [I 2022-10-11 21:10:38] ...trap/engines/engine.py.325: elapsed: 0:02:42 | left: 0:00:25 [I 2022-10-11 21:10:38] ...trap/engines/engine.py.326: process: 0.30769 | load: 0.00019 [I 2022-10-11 21:10:41] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/510 [I 2022-10-11 21:10:41] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:22 [I 2022-10-11 21:10:41] ...trap/engines/engine.py.326: process: 0.31951 | load: 0.00019 [I 2022-10-11 21:10:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/510 [I 2022-10-11 21:10:44] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:19 [I 2022-10-11 21:10:44] ...trap/engines/engine.py.326: process: 0.32225 | load: 0.00020 [I 2022-10-11 21:10:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/510 [I 2022-10-11 21:10:48] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:15 [I 2022-10-11 21:10:48] ...trap/engines/engine.py.326: process: 0.32107 | load: 0.00020 [I 2022-10-11 21:10:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/510 [I 2022-10-11 21:10:51] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:12 [I 2022-10-11 21:10:51] ...trap/engines/engine.py.326: process: 0.32627 | load: 0.00019 [I 2022-10-11 21:10:54] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/510 [I 2022-10-11 21:10:54] ...trap/engines/engine.py.325: elapsed: 0:02:58 | left: 0:00:09 [I 2022-10-11 21:10:54] ...trap/engines/engine.py.326: process: 0.31489 | load: 0.00020 [I 2022-10-11 21:10:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/510 [I 2022-10-11 21:10:57] ...trap/engines/engine.py.325: elapsed: 0:03:01 | left: 0:00:06 [I 2022-10-11 21:10:57] ...trap/engines/engine.py.326: process: 0.32277 | load: 0.00020 [I 2022-10-11 21:11:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/510 [I 2022-10-11 21:11:00] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:03 [I 2022-10-11 21:11:00] ...trap/engines/engine.py.326: process: 0.32191 | load: 0.00021 [I 2022-10-11 21:11:04] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/510 [I 2022-10-11 21:11:04] ...trap/engines/engine.py.325: elapsed: 0:03:07 | left: 0:00:00 [I 2022-10-11 21:11:04] ...trap/engines/engine.py.326: process: 0.31307 | load: 0.00015 [S 2022-10-11 21:11:04] ...trap/engines/engine.py.132: eval_epoch.epoch: 118 [I 2022-10-11 21:11:04] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.752200 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913000 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936500 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.746300 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909500 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936500 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012600 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006387 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005572 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011261 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005696 [S 2022-10-11 21:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006469 [I 2022-10-11 21:11:42] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-11 21:11:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 21:11:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 21:11:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 21:11:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 21:11:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 21:11:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 21:11:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 21:11:53] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 21:11:53] ...trap/engines/engine.py.470: Saving model... [I 2022-10-11 21:11:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 21:12:01] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 21:12:01] ...trap/engines/engine.py.163: Training model on trainset for epoch 119 [I 2022-10-11 21:12:01] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-11 21:12:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 0/2304 [I 2022-10-11 21:12:04] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 1:58:30 [I 2022-10-11 21:12:04] ...trap/engines/engine.py.229: process: 3.08614 | load: 2.16601 [I 2022-10-11 21:12:04] ...trap/engines/engine.py.230: loss: 0.07650 [I 2022-10-11 21:12:04] ...optimizers/trijoint.py.151: total_norm: 10.584153 [I 2022-10-11 21:12:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 10/2304 [I 2022-10-11 21:12:13] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:41:48 [I 2022-10-11 21:12:13] ...trap/engines/engine.py.229: process: 0.86767 | load: 0.00499 [I 2022-10-11 21:12:13] ...trap/engines/engine.py.230: loss: 0.16484 [I 2022-10-11 21:12:13] ...optimizers/trijoint.py.151: total_norm: 12.343625 [I 2022-10-11 21:12:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 20/2304 [I 2022-10-11 21:12:21] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:33:13 [I 2022-10-11 21:12:21] ...trap/engines/engine.py.229: process: 0.84208 | load: 0.00544 [I 2022-10-11 21:12:21] ...trap/engines/engine.py.230: loss: 0.11374 [I 2022-10-11 21:12:21] ...optimizers/trijoint.py.151: total_norm: 64.942307 [I 2022-10-11 21:12:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 30/2304 [I 2022-10-11 21:12:30] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:32:40 [I 2022-10-11 21:12:30] ...trap/engines/engine.py.229: process: 0.85975 | load: 0.00412 [I 2022-10-11 21:12:30] ...trap/engines/engine.py.230: loss: 0.09193 [I 2022-10-11 21:12:30] ...optimizers/trijoint.py.151: total_norm: 13.157810 [I 2022-10-11 21:12:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 40/2304 [I 2022-10-11 21:12:39] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:32:30 [I 2022-10-11 21:12:39] ...trap/engines/engine.py.229: process: 0.87215 | load: 0.00387 [I 2022-10-11 21:12:39] ...trap/engines/engine.py.230: loss: 0.11704 [I 2022-10-11 21:12:39] ...optimizers/trijoint.py.151: total_norm: 15.547584 [I 2022-10-11 21:12:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 50/2304 [I 2022-10-11 21:12:47] ...trap/engines/engine.py.228: elapsed: 0:00:45 | left: 0:31:59 [I 2022-10-11 21:12:47] ...trap/engines/engine.py.229: process: 0.85492 | load: 0.00402 [I 2022-10-11 21:12:47] ...trap/engines/engine.py.230: loss: 0.09710 [I 2022-10-11 21:12:47] ...optimizers/trijoint.py.151: total_norm: 8.518982 [I 2022-10-11 21:12:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 60/2304 [I 2022-10-11 21:12:56] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:31:52 [I 2022-10-11 21:12:56] ...trap/engines/engine.py.229: process: 0.85600 | load: 0.00418 [I 2022-10-11 21:12:56] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-11 21:12:56] ...optimizers/trijoint.py.151: total_norm: 14.877272 [I 2022-10-11 21:13:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 70/2304 [I 2022-10-11 21:13:04] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:31:57 [I 2022-10-11 21:13:04] ...trap/engines/engine.py.229: process: 0.85965 | load: 0.00471 [I 2022-10-11 21:13:04] ...trap/engines/engine.py.230: loss: 0.10457 [I 2022-10-11 21:13:04] ...optimizers/trijoint.py.151: total_norm: 13.741710 [I 2022-10-11 21:13:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 80/2304 [I 2022-10-11 21:13:13] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:31:49 [I 2022-10-11 21:13:13] ...trap/engines/engine.py.229: process: 0.86669 | load: 0.00357 [I 2022-10-11 21:13:13] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-11 21:13:13] ...optimizers/trijoint.py.151: total_norm: 25.740320 [I 2022-10-11 21:13:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 90/2304 [I 2022-10-11 21:13:21] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:31:47 [I 2022-10-11 21:13:21] ...trap/engines/engine.py.229: process: 0.85328 | load: 0.00350 [I 2022-10-11 21:13:21] ...trap/engines/engine.py.230: loss: 0.09380 [I 2022-10-11 21:13:21] ...optimizers/trijoint.py.151: total_norm: 12.865039 [I 2022-10-11 21:13:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 100/2304 [I 2022-10-11 21:13:30] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:31:47 [I 2022-10-11 21:13:30] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00346 [I 2022-10-11 21:13:30] ...trap/engines/engine.py.230: loss: 0.07622 [I 2022-10-11 21:13:30] ...optimizers/trijoint.py.151: total_norm: 10.996968 [I 2022-10-11 21:13:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 110/2304 [I 2022-10-11 21:13:39] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:31:40 [I 2022-10-11 21:13:39] ...trap/engines/engine.py.229: process: 0.87830 | load: 0.00379 [I 2022-10-11 21:13:39] ...trap/engines/engine.py.230: loss: 0.08186 [I 2022-10-11 21:13:39] ...optimizers/trijoint.py.151: total_norm: 8.840928 [I 2022-10-11 21:13:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 120/2304 [I 2022-10-11 21:13:48] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:31:37 [I 2022-10-11 21:13:48] ...trap/engines/engine.py.229: process: 0.87084 | load: 0.00335 [I 2022-10-11 21:13:48] ...trap/engines/engine.py.230: loss: 0.13770 [I 2022-10-11 21:13:48] ...optimizers/trijoint.py.151: total_norm: 12.240908 [I 2022-10-11 21:13:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 130/2304 [I 2022-10-11 21:13:56] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:31:31 [I 2022-10-11 21:13:56] ...trap/engines/engine.py.229: process: 0.87285 | load: 0.00349 [I 2022-10-11 21:13:56] ...trap/engines/engine.py.230: loss: 0.08334 [I 2022-10-11 21:13:56] ...optimizers/trijoint.py.151: total_norm: 10.284500 [I 2022-10-11 21:14:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 140/2304 [I 2022-10-11 21:14:05] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:31:02 [I 2022-10-11 21:14:05] ...trap/engines/engine.py.229: process: 0.84779 | load: 0.00347 [I 2022-10-11 21:14:05] ...trap/engines/engine.py.230: loss: 0.10872 [I 2022-10-11 21:14:05] ...optimizers/trijoint.py.151: total_norm: 27.679762 [I 2022-10-11 21:14:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 150/2304 [I 2022-10-11 21:14:14] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:30:57 [I 2022-10-11 21:14:14] ...trap/engines/engine.py.229: process: 0.85840 | load: 0.00313 [I 2022-10-11 21:14:14] ...trap/engines/engine.py.230: loss: 0.11811 [I 2022-10-11 21:14:14] ...optimizers/trijoint.py.151: total_norm: 15.287488 [I 2022-10-11 21:14:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 160/2304 [I 2022-10-11 21:14:22] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:30:46 [I 2022-10-11 21:14:22] ...trap/engines/engine.py.229: process: 0.85339 | load: 0.00329 [I 2022-10-11 21:14:22] ...trap/engines/engine.py.230: loss: 0.10714 [I 2022-10-11 21:14:22] ...optimizers/trijoint.py.151: total_norm: 21.038708 [I 2022-10-11 21:14:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 170/2304 [I 2022-10-11 21:14:31] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:30:50 [I 2022-10-11 21:14:31] ...trap/engines/engine.py.229: process: 0.86511 | load: 0.00326 [I 2022-10-11 21:14:31] ...trap/engines/engine.py.230: loss: 0.15921 [I 2022-10-11 21:14:31] ...optimizers/trijoint.py.151: total_norm: 24.627611 [I 2022-10-11 21:14:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 180/2304 [I 2022-10-11 21:14:40] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:30:46 [I 2022-10-11 21:14:40] ...trap/engines/engine.py.229: process: 0.87014 | load: 0.00345 [I 2022-10-11 21:14:40] ...trap/engines/engine.py.230: loss: 0.07131 [I 2022-10-11 21:14:40] ...optimizers/trijoint.py.151: total_norm: 14.388757 [I 2022-10-11 21:14:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 190/2304 [I 2022-10-11 21:14:48] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:30:42 [I 2022-10-11 21:14:48] ...trap/engines/engine.py.229: process: 0.87318 | load: 0.00337 [I 2022-10-11 21:14:48] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-11 21:14:48] ...optimizers/trijoint.py.151: total_norm: 25.782219 [I 2022-10-11 21:14:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 200/2304 [I 2022-10-11 21:14:57] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:30:12 [I 2022-10-11 21:14:57] ...trap/engines/engine.py.229: process: 0.85050 | load: 0.00313 [I 2022-10-11 21:14:57] ...trap/engines/engine.py.230: loss: 0.17270 [I 2022-10-11 21:14:57] ...optimizers/trijoint.py.151: total_norm: 45.865173 [I 2022-10-11 21:15:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 210/2304 [I 2022-10-11 21:15:05] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:30:04 [I 2022-10-11 21:15:05] ...trap/engines/engine.py.229: process: 0.86669 | load: 0.00327 [I 2022-10-11 21:15:05] ...trap/engines/engine.py.230: loss: 0.17007 [I 2022-10-11 21:15:05] ...optimizers/trijoint.py.151: total_norm: 11.580604 [I 2022-10-11 21:15:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 220/2304 [I 2022-10-11 21:15:14] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:29:59 [I 2022-10-11 21:15:14] ...trap/engines/engine.py.229: process: 0.86926 | load: 0.00310 [I 2022-10-11 21:15:14] ...trap/engines/engine.py.230: loss: 0.07402 [I 2022-10-11 21:15:14] ...optimizers/trijoint.py.151: total_norm: 10.051619 [I 2022-10-11 21:15:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 230/2304 [I 2022-10-11 21:15:23] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:30:04 [I 2022-10-11 21:15:23] ...trap/engines/engine.py.229: process: 0.86781 | load: 0.00352 [I 2022-10-11 21:15:23] ...trap/engines/engine.py.230: loss: 0.09560 [I 2022-10-11 21:15:23] ...optimizers/trijoint.py.151: total_norm: 11.084555 [I 2022-10-11 21:15:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 240/2304 [I 2022-10-11 21:15:32] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:29:54 [I 2022-10-11 21:15:32] ...trap/engines/engine.py.229: process: 0.87349 | load: 0.00314 [I 2022-10-11 21:15:32] ...trap/engines/engine.py.230: loss: 0.10603 [I 2022-10-11 21:15:32] ...optimizers/trijoint.py.151: total_norm: 13.649821 [I 2022-10-11 21:15:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 250/2304 [I 2022-10-11 21:15:40] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:29:44 [I 2022-10-11 21:15:40] ...trap/engines/engine.py.229: process: 0.85514 | load: 0.00329 [I 2022-10-11 21:15:40] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-11 21:15:40] ...optimizers/trijoint.py.151: total_norm: 20.245232 [I 2022-10-11 21:15:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 260/2304 [I 2022-10-11 21:15:49] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:29:29 [I 2022-10-11 21:15:49] ...trap/engines/engine.py.229: process: 0.86809 | load: 0.00315 [I 2022-10-11 21:15:49] ...trap/engines/engine.py.230: loss: 0.19625 [I 2022-10-11 21:15:49] ...optimizers/trijoint.py.151: total_norm: 16.963869 [I 2022-10-11 21:15:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 270/2304 [I 2022-10-11 21:15:58] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:29:22 [I 2022-10-11 21:15:58] ...trap/engines/engine.py.229: process: 0.86477 | load: 0.00314 [I 2022-10-11 21:15:58] ...trap/engines/engine.py.230: loss: 0.10320 [I 2022-10-11 21:15:58] ...optimizers/trijoint.py.151: total_norm: 20.466734 [I 2022-10-11 21:16:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 280/2304 [I 2022-10-11 21:16:06] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:29:17 [I 2022-10-11 21:16:06] ...trap/engines/engine.py.229: process: 0.87030 | load: 0.00315 [I 2022-10-11 21:16:06] ...trap/engines/engine.py.230: loss: 0.11640 [I 2022-10-11 21:16:06] ...optimizers/trijoint.py.151: total_norm: 10.054932 [I 2022-10-11 21:16:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 290/2304 [I 2022-10-11 21:16:15] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:29:05 [I 2022-10-11 21:16:15] ...trap/engines/engine.py.229: process: 0.86816 | load: 0.00326 [I 2022-10-11 21:16:15] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-11 21:16:15] ...optimizers/trijoint.py.151: total_norm: 18.001381 [I 2022-10-11 21:16:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 300/2304 [I 2022-10-11 21:16:24] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:28:56 [I 2022-10-11 21:16:24] ...trap/engines/engine.py.229: process: 0.86321 | load: 0.00314 [I 2022-10-11 21:16:24] ...trap/engines/engine.py.230: loss: 0.14976 [I 2022-10-11 21:16:24] ...optimizers/trijoint.py.151: total_norm: 11.290033 [I 2022-10-11 21:16:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 310/2304 [I 2022-10-11 21:16:32] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:28:05 [I 2022-10-11 21:16:32] ...trap/engines/engine.py.229: process: 0.83901 | load: 0.00317 [I 2022-10-11 21:16:32] ...trap/engines/engine.py.230: loss: 0.10284 [I 2022-10-11 21:16:32] ...optimizers/trijoint.py.151: total_norm: 24.038916 [I 2022-10-11 21:16:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 320/2304 [I 2022-10-11 21:16:41] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:27:55 [I 2022-10-11 21:16:41] ...trap/engines/engine.py.229: process: 0.84372 | load: 0.00305 [I 2022-10-11 21:16:41] ...trap/engines/engine.py.230: loss: 0.07347 [I 2022-10-11 21:16:41] ...optimizers/trijoint.py.151: total_norm: 13.829862 [I 2022-10-11 21:16:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 330/2304 [I 2022-10-11 21:16:49] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:27:44 [I 2022-10-11 21:16:49] ...trap/engines/engine.py.229: process: 0.84421 | load: 0.00294 [I 2022-10-11 21:16:49] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-11 21:16:49] ...optimizers/trijoint.py.151: total_norm: 9.668062 [I 2022-10-11 21:16:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 340/2304 [I 2022-10-11 21:16:57] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:27:23 [I 2022-10-11 21:16:57] ...trap/engines/engine.py.229: process: 0.82333 | load: 0.00293 [I 2022-10-11 21:16:57] ...trap/engines/engine.py.230: loss: 0.09057 [I 2022-10-11 21:16:57] ...optimizers/trijoint.py.151: total_norm: 13.116656 [I 2022-10-11 21:17:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 350/2304 [I 2022-10-11 21:17:06] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:27:16 [I 2022-10-11 21:17:06] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00294 [I 2022-10-11 21:17:06] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-11 21:17:06] ...optimizers/trijoint.py.151: total_norm: 45.304585 [I 2022-10-11 21:17:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 360/2304 [I 2022-10-11 21:17:14] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:27:10 [I 2022-10-11 21:17:14] ...trap/engines/engine.py.229: process: 0.83498 | load: 0.00293 [I 2022-10-11 21:17:14] ...trap/engines/engine.py.230: loss: 0.08193 [I 2022-10-11 21:17:14] ...optimizers/trijoint.py.151: total_norm: 10.945875 [I 2022-10-11 21:17:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 370/2304 [I 2022-10-11 21:17:23] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:27:01 [I 2022-10-11 21:17:23] ...trap/engines/engine.py.229: process: 0.83815 | load: 0.00299 [I 2022-10-11 21:17:23] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-11 21:17:23] ...optimizers/trijoint.py.151: total_norm: 39.229214 [I 2022-10-11 21:17:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 380/2304 [I 2022-10-11 21:17:31] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:26:55 [I 2022-10-11 21:17:31] ...trap/engines/engine.py.229: process: 0.84439 | load: 0.00293 [I 2022-10-11 21:17:31] ...trap/engines/engine.py.230: loss: 0.10711 [I 2022-10-11 21:17:31] ...optimizers/trijoint.py.151: total_norm: 10.311291 [I 2022-10-11 21:17:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 390/2304 [I 2022-10-11 21:17:39] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:26:52 [I 2022-10-11 21:17:39] ...trap/engines/engine.py.229: process: 0.83159 | load: 0.00295 [I 2022-10-11 21:17:39] ...trap/engines/engine.py.230: loss: 0.13705 [I 2022-10-11 21:17:39] ...optimizers/trijoint.py.151: total_norm: 14.230143 [I 2022-10-11 21:17:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 400/2304 [I 2022-10-11 21:17:48] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:26:43 [I 2022-10-11 21:17:48] ...trap/engines/engine.py.229: process: 0.84666 | load: 0.00305 [I 2022-10-11 21:17:48] ...trap/engines/engine.py.230: loss: 0.08521 [I 2022-10-11 21:17:48] ...optimizers/trijoint.py.151: total_norm: 11.413266 [I 2022-10-11 21:17:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 410/2304 [I 2022-10-11 21:17:56] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:26:42 [I 2022-10-11 21:17:56] ...trap/engines/engine.py.229: process: 0.84780 | load: 0.00299 [I 2022-10-11 21:17:56] ...trap/engines/engine.py.230: loss: 0.16070 [I 2022-10-11 21:17:56] ...optimizers/trijoint.py.151: total_norm: 15.450314 [I 2022-10-11 21:18:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 420/2304 [I 2022-10-11 21:18:05] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:26:32 [I 2022-10-11 21:18:05] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00300 [I 2022-10-11 21:18:05] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-11 21:18:05] ...optimizers/trijoint.py.151: total_norm: 12.113028 [I 2022-10-11 21:18:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 430/2304 [I 2022-10-11 21:18:13] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:26:25 [I 2022-10-11 21:18:13] ...trap/engines/engine.py.229: process: 0.84116 | load: 0.00295 [I 2022-10-11 21:18:13] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-11 21:18:13] ...optimizers/trijoint.py.151: total_norm: 17.254528 [I 2022-10-11 21:18:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 440/2304 [I 2022-10-11 21:18:22] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:26:09 [I 2022-10-11 21:18:22] ...trap/engines/engine.py.229: process: 0.84024 | load: 0.00297 [I 2022-10-11 21:18:22] ...trap/engines/engine.py.230: loss: 0.08511 [I 2022-10-11 21:18:22] ...optimizers/trijoint.py.151: total_norm: 11.010650 [I 2022-10-11 21:18:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 450/2304 [I 2022-10-11 21:18:30] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:25:59 [I 2022-10-11 21:18:30] ...trap/engines/engine.py.229: process: 0.84062 | load: 0.00300 [I 2022-10-11 21:18:30] ...trap/engines/engine.py.230: loss: 0.16924 [I 2022-10-11 21:18:30] ...optimizers/trijoint.py.151: total_norm: 17.300638 [I 2022-10-11 21:18:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 460/2304 [I 2022-10-11 21:18:39] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:26:00 [I 2022-10-11 21:18:39] ...trap/engines/engine.py.229: process: 0.85841 | load: 0.00298 [I 2022-10-11 21:18:39] ...trap/engines/engine.py.230: loss: 0.08948 [I 2022-10-11 21:18:39] ...optimizers/trijoint.py.151: total_norm: 15.268283 [I 2022-10-11 21:18:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 470/2304 [I 2022-10-11 21:18:47] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:25:54 [I 2022-10-11 21:18:47] ...trap/engines/engine.py.229: process: 0.86360 | load: 0.00298 [I 2022-10-11 21:18:47] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-11 21:18:47] ...optimizers/trijoint.py.151: total_norm: 21.137373 [I 2022-10-11 21:18:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 480/2304 [I 2022-10-11 21:18:56] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:25:37 [I 2022-10-11 21:18:56] ...trap/engines/engine.py.229: process: 0.84713 | load: 0.00295 [I 2022-10-11 21:18:56] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-11 21:18:56] ...optimizers/trijoint.py.151: total_norm: 16.852913 [I 2022-10-11 21:19:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 490/2304 [I 2022-10-11 21:19:04] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:25:30 [I 2022-10-11 21:19:04] ...trap/engines/engine.py.229: process: 0.83635 | load: 0.00301 [I 2022-10-11 21:19:04] ...trap/engines/engine.py.230: loss: 0.09340 [I 2022-10-11 21:19:04] ...optimizers/trijoint.py.151: total_norm: 12.287667 [I 2022-10-11 21:19:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 500/2304 [I 2022-10-11 21:19:13] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:25:22 [I 2022-10-11 21:19:13] ...trap/engines/engine.py.229: process: 0.84478 | load: 0.00308 [I 2022-10-11 21:19:13] ...trap/engines/engine.py.230: loss: 0.13837 [I 2022-10-11 21:19:13] ...optimizers/trijoint.py.151: total_norm: 13.049076 [I 2022-10-11 21:19:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 510/2304 [I 2022-10-11 21:19:21] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:25:23 [I 2022-10-11 21:19:21] ...trap/engines/engine.py.229: process: 0.85956 | load: 0.00297 [I 2022-10-11 21:19:21] ...trap/engines/engine.py.230: loss: 0.10773 [I 2022-10-11 21:19:21] ...optimizers/trijoint.py.151: total_norm: 60.674355 [I 2022-10-11 21:19:29] ...trap/engines/engine.py.224: train: epoch 119 | batch 520/2304 [I 2022-10-11 21:19:29] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:25:10 [I 2022-10-11 21:19:29] ...trap/engines/engine.py.229: process: 0.84817 | load: 0.00302 [I 2022-10-11 21:19:29] ...trap/engines/engine.py.230: loss: 0.11267 [I 2022-10-11 21:19:29] ...optimizers/trijoint.py.151: total_norm: 9.377243 [I 2022-10-11 21:19:38] ...trap/engines/engine.py.224: train: epoch 119 | batch 530/2304 [I 2022-10-11 21:19:38] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:24:59 [I 2022-10-11 21:19:38] ...trap/engines/engine.py.229: process: 0.82873 | load: 0.00300 [I 2022-10-11 21:19:38] ...trap/engines/engine.py.230: loss: 0.07726 [I 2022-10-11 21:19:38] ...optimizers/trijoint.py.151: total_norm: 19.087463 [I 2022-10-11 21:19:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 540/2304 [I 2022-10-11 21:19:47] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:25:04 [I 2022-10-11 21:19:47] ...trap/engines/engine.py.229: process: 0.84338 | load: 0.00313 [I 2022-10-11 21:19:47] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-11 21:19:47] ...optimizers/trijoint.py.151: total_norm: 13.042253 [I 2022-10-11 21:19:55] ...trap/engines/engine.py.224: train: epoch 119 | batch 550/2304 [I 2022-10-11 21:19:55] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:24:59 [I 2022-10-11 21:19:55] ...trap/engines/engine.py.229: process: 0.85919 | load: 0.00324 [I 2022-10-11 21:19:55] ...trap/engines/engine.py.230: loss: 0.10377 [I 2022-10-11 21:19:55] ...optimizers/trijoint.py.151: total_norm: 14.920579 [I 2022-10-11 21:20:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 560/2304 [I 2022-10-11 21:20:04] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:25:06 [I 2022-10-11 21:20:04] ...trap/engines/engine.py.229: process: 0.86393 | load: 0.00324 [I 2022-10-11 21:20:04] ...trap/engines/engine.py.230: loss: 0.07466 [I 2022-10-11 21:20:04] ...optimizers/trijoint.py.151: total_norm: 10.814584 [I 2022-10-11 21:20:12] ...trap/engines/engine.py.224: train: epoch 119 | batch 570/2304 [I 2022-10-11 21:20:12] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:24:50 [I 2022-10-11 21:20:12] ...trap/engines/engine.py.229: process: 0.85847 | load: 0.00314 [I 2022-10-11 21:20:12] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-11 21:20:12] ...optimizers/trijoint.py.151: total_norm: 10.573978 [I 2022-10-11 21:20:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 580/2304 [I 2022-10-11 21:20:21] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:25:13 [I 2022-10-11 21:20:21] ...trap/engines/engine.py.229: process: 0.89416 | load: 0.00396 [I 2022-10-11 21:20:21] ...trap/engines/engine.py.230: loss: 0.09428 [I 2022-10-11 21:20:21] ...optimizers/trijoint.py.151: total_norm: 18.823910 [I 2022-10-11 21:20:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 590/2304 [I 2022-10-11 21:20:30] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:25:28 [I 2022-10-11 21:20:30] ...trap/engines/engine.py.229: process: 0.89204 | load: 0.00407 [I 2022-10-11 21:20:30] ...trap/engines/engine.py.230: loss: 0.09543 [I 2022-10-11 21:20:30] ...optimizers/trijoint.py.151: total_norm: 10.459222 [I 2022-10-11 21:20:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 600/2304 [I 2022-10-11 21:20:39] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:25:20 [I 2022-10-11 21:20:39] ...trap/engines/engine.py.229: process: 0.89620 | load: 0.00396 [I 2022-10-11 21:20:39] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-11 21:20:39] ...optimizers/trijoint.py.151: total_norm: 28.091366 [I 2022-10-11 21:20:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 610/2304 [I 2022-10-11 21:20:48] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:25:06 [I 2022-10-11 21:20:48] ...trap/engines/engine.py.229: process: 0.88202 | load: 0.00396 [I 2022-10-11 21:20:48] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-11 21:20:48] ...optimizers/trijoint.py.151: total_norm: 10.717529 [I 2022-10-11 21:20:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 620/2304 [I 2022-10-11 21:20:57] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:25:06 [I 2022-10-11 21:20:57] ...trap/engines/engine.py.229: process: 0.90371 | load: 0.00399 [I 2022-10-11 21:20:57] ...trap/engines/engine.py.230: loss: 0.09354 [I 2022-10-11 21:20:57] ...optimizers/trijoint.py.151: total_norm: 15.365613 [I 2022-10-11 21:21:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 630/2304 [I 2022-10-11 21:21:06] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:24:58 [I 2022-10-11 21:21:06] ...trap/engines/engine.py.229: process: 0.88926 | load: 0.00395 [I 2022-10-11 21:21:06] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-11 21:21:06] ...optimizers/trijoint.py.151: total_norm: 17.532206 [I 2022-10-11 21:21:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 640/2304 [I 2022-10-11 21:21:15] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:24:39 [I 2022-10-11 21:21:15] ...trap/engines/engine.py.229: process: 0.88850 | load: 0.00394 [I 2022-10-11 21:21:15] ...trap/engines/engine.py.230: loss: 0.07621 [I 2022-10-11 21:21:15] ...optimizers/trijoint.py.151: total_norm: 14.024807 [I 2022-10-11 21:21:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 650/2304 [I 2022-10-11 21:21:23] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:23:52 [I 2022-10-11 21:21:23] ...trap/engines/engine.py.229: process: 0.85804 | load: 0.00318 [I 2022-10-11 21:21:23] ...trap/engines/engine.py.230: loss: 0.10766 [I 2022-10-11 21:21:23] ...optimizers/trijoint.py.151: total_norm: 13.743349 [I 2022-10-11 21:21:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 660/2304 [I 2022-10-11 21:21:32] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:23:27 [I 2022-10-11 21:21:32] ...trap/engines/engine.py.229: process: 0.84431 | load: 0.00312 [I 2022-10-11 21:21:32] ...trap/engines/engine.py.230: loss: 0.07277 [I 2022-10-11 21:21:32] ...optimizers/trijoint.py.151: total_norm: 13.751837 [I 2022-10-11 21:21:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 670/2304 [I 2022-10-11 21:21:41] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:23:20 [I 2022-10-11 21:21:41] ...trap/engines/engine.py.229: process: 0.85585 | load: 0.00314 [I 2022-10-11 21:21:41] ...trap/engines/engine.py.230: loss: 0.08813 [I 2022-10-11 21:21:41] ...optimizers/trijoint.py.151: total_norm: 24.788559 [I 2022-10-11 21:21:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 680/2304 [I 2022-10-11 21:21:49] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:23:14 [I 2022-10-11 21:21:49] ...trap/engines/engine.py.229: process: 0.85086 | load: 0.00313 [I 2022-10-11 21:21:49] ...trap/engines/engine.py.230: loss: 0.09324 [I 2022-10-11 21:21:49] ...optimizers/trijoint.py.151: total_norm: 41.177803 [I 2022-10-11 21:21:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 690/2304 [I 2022-10-11 21:21:58] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:23:15 [I 2022-10-11 21:21:58] ...trap/engines/engine.py.229: process: 0.87245 | load: 0.00314 [I 2022-10-11 21:21:58] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-11 21:21:58] ...optimizers/trijoint.py.151: total_norm: 42.079544 [I 2022-10-11 21:22:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 700/2304 [I 2022-10-11 21:22:07] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:22:58 [I 2022-10-11 21:22:07] ...trap/engines/engine.py.229: process: 0.84703 | load: 0.00312 [I 2022-10-11 21:22:07] ...trap/engines/engine.py.230: loss: 0.12640 [I 2022-10-11 21:22:07] ...optimizers/trijoint.py.151: total_norm: 13.584704 [I 2022-10-11 21:22:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 710/2304 [I 2022-10-11 21:22:15] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:22:45 [I 2022-10-11 21:22:15] ...trap/engines/engine.py.229: process: 0.85509 | load: 0.00313 [I 2022-10-11 21:22:15] ...trap/engines/engine.py.230: loss: 0.09661 [I 2022-10-11 21:22:15] ...optimizers/trijoint.py.151: total_norm: 10.864852 [I 2022-10-11 21:22:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 720/2304 [I 2022-10-11 21:22:24] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:22:42 [I 2022-10-11 21:22:24] ...trap/engines/engine.py.229: process: 0.85284 | load: 0.00317 [I 2022-10-11 21:22:24] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-11 21:22:24] ...optimizers/trijoint.py.151: total_norm: 13.893477 [I 2022-10-11 21:22:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 730/2304 [I 2022-10-11 21:22:32] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:22:23 [I 2022-10-11 21:22:32] ...trap/engines/engine.py.229: process: 0.85839 | load: 0.00315 [I 2022-10-11 21:22:32] ...trap/engines/engine.py.230: loss: 0.09380 [I 2022-10-11 21:22:32] ...optimizers/trijoint.py.151: total_norm: 20.137312 [I 2022-10-11 21:22:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 740/2304 [I 2022-10-11 21:22:41] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:22:26 [I 2022-10-11 21:22:41] ...trap/engines/engine.py.229: process: 0.86004 | load: 0.00316 [I 2022-10-11 21:22:41] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-11 21:22:41] ...optimizers/trijoint.py.151: total_norm: 12.988837 [I 2022-10-11 21:22:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 750/2304 [I 2022-10-11 21:22:50] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:22:13 [I 2022-10-11 21:22:50] ...trap/engines/engine.py.229: process: 0.86415 | load: 0.00315 [I 2022-10-11 21:22:50] ...trap/engines/engine.py.230: loss: 0.09761 [I 2022-10-11 21:22:50] ...optimizers/trijoint.py.151: total_norm: 20.838804 [I 2022-10-11 21:22:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 760/2304 [I 2022-10-11 21:22:58] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:22:07 [I 2022-10-11 21:22:58] ...trap/engines/engine.py.229: process: 0.85502 | load: 0.00314 [I 2022-10-11 21:22:58] ...trap/engines/engine.py.230: loss: 0.08297 [I 2022-10-11 21:22:58] ...optimizers/trijoint.py.151: total_norm: 10.213293 [I 2022-10-11 21:23:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 770/2304 [I 2022-10-11 21:23:07] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:21:50 [I 2022-10-11 21:23:07] ...trap/engines/engine.py.229: process: 0.84654 | load: 0.00312 [I 2022-10-11 21:23:07] ...trap/engines/engine.py.230: loss: 0.12366 [I 2022-10-11 21:23:07] ...optimizers/trijoint.py.151: total_norm: 15.930706 [I 2022-10-11 21:23:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 780/2304 [I 2022-10-11 21:23:15] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:21:52 [I 2022-10-11 21:23:15] ...trap/engines/engine.py.229: process: 0.86281 | load: 0.00312 [I 2022-10-11 21:23:15] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-11 21:23:15] ...optimizers/trijoint.py.151: total_norm: 11.891303 [I 2022-10-11 21:23:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 790/2304 [I 2022-10-11 21:23:24] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:21:42 [I 2022-10-11 21:23:24] ...trap/engines/engine.py.229: process: 0.85757 | load: 0.00318 [I 2022-10-11 21:23:24] ...trap/engines/engine.py.230: loss: 0.08521 [I 2022-10-11 21:23:24] ...optimizers/trijoint.py.151: total_norm: 13.055940 [I 2022-10-11 21:23:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 800/2304 [I 2022-10-11 21:23:33] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:21:20 [I 2022-10-11 21:23:33] ...trap/engines/engine.py.229: process: 0.84648 | load: 0.00325 [I 2022-10-11 21:23:33] ...trap/engines/engine.py.230: loss: 0.20021 [I 2022-10-11 21:23:33] ...optimizers/trijoint.py.151: total_norm: 36.592594 [I 2022-10-11 21:23:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 810/2304 [I 2022-10-11 21:23:41] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:21:08 [I 2022-10-11 21:23:41] ...trap/engines/engine.py.229: process: 0.84438 | load: 0.00295 [I 2022-10-11 21:23:41] ...trap/engines/engine.py.230: loss: 0.09819 [I 2022-10-11 21:23:41] ...optimizers/trijoint.py.151: total_norm: 23.768270 [I 2022-10-11 21:23:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 820/2304 [I 2022-10-11 21:23:50] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:20:56 [I 2022-10-11 21:23:50] ...trap/engines/engine.py.229: process: 0.83228 | load: 0.00304 [I 2022-10-11 21:23:50] ...trap/engines/engine.py.230: loss: 0.15126 [I 2022-10-11 21:23:50] ...optimizers/trijoint.py.151: total_norm: 36.146172 [I 2022-10-11 21:23:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 830/2304 [I 2022-10-11 21:23:58] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:20:58 [I 2022-10-11 21:23:58] ...trap/engines/engine.py.229: process: 0.85476 | load: 0.00296 [I 2022-10-11 21:23:58] ...trap/engines/engine.py.230: loss: 0.08462 [I 2022-10-11 21:23:58] ...optimizers/trijoint.py.151: total_norm: 12.019977 [I 2022-10-11 21:24:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 840/2304 [I 2022-10-11 21:24:07] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:20:54 [I 2022-10-11 21:24:07] ...trap/engines/engine.py.229: process: 0.84934 | load: 0.00295 [I 2022-10-11 21:24:07] ...trap/engines/engine.py.230: loss: 0.08410 [I 2022-10-11 21:24:07] ...optimizers/trijoint.py.151: total_norm: 18.546869 [I 2022-10-11 21:24:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 850/2304 [I 2022-10-11 21:24:15] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:20:45 [I 2022-10-11 21:24:15] ...trap/engines/engine.py.229: process: 0.86219 | load: 0.00298 [I 2022-10-11 21:24:15] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-11 21:24:15] ...optimizers/trijoint.py.151: total_norm: 13.622938 [I 2022-10-11 21:24:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 860/2304 [I 2022-10-11 21:24:24] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:20:42 [I 2022-10-11 21:24:24] ...trap/engines/engine.py.229: process: 0.85958 | load: 0.00378 [I 2022-10-11 21:24:24] ...trap/engines/engine.py.230: loss: 0.09410 [I 2022-10-11 21:24:24] ...optimizers/trijoint.py.151: total_norm: 12.160231 [I 2022-10-11 21:24:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 870/2304 [I 2022-10-11 21:24:33] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:20:52 [I 2022-10-11 21:24:33] ...trap/engines/engine.py.229: process: 0.87413 | load: 0.00378 [I 2022-10-11 21:24:33] ...trap/engines/engine.py.230: loss: 0.10452 [I 2022-10-11 21:24:33] ...optimizers/trijoint.py.151: total_norm: 9.119568 [I 2022-10-11 21:24:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 880/2304 [I 2022-10-11 21:24:41] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:20:47 [I 2022-10-11 21:24:41] ...trap/engines/engine.py.229: process: 0.88029 | load: 0.00379 [I 2022-10-11 21:24:41] ...trap/engines/engine.py.230: loss: 0.13341 [I 2022-10-11 21:24:41] ...optimizers/trijoint.py.151: total_norm: 12.421840 [I 2022-10-11 21:24:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 890/2304 [I 2022-10-11 21:24:50] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:20:34 [I 2022-10-11 21:24:50] ...trap/engines/engine.py.229: process: 0.86749 | load: 0.00378 [I 2022-10-11 21:24:50] ...trap/engines/engine.py.230: loss: 0.14077 [I 2022-10-11 21:24:50] ...optimizers/trijoint.py.151: total_norm: 12.815662 [I 2022-10-11 21:24:59] ...trap/engines/engine.py.224: train: epoch 119 | batch 900/2304 [I 2022-10-11 21:24:59] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:20:25 [I 2022-10-11 21:24:59] ...trap/engines/engine.py.229: process: 0.87025 | load: 0.00381 [I 2022-10-11 21:24:59] ...trap/engines/engine.py.230: loss: 0.09646 [I 2022-10-11 21:24:59] ...optimizers/trijoint.py.151: total_norm: 12.236735 [I 2022-10-11 21:25:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 910/2304 [I 2022-10-11 21:25:08] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:20:28 [I 2022-10-11 21:25:08] ...trap/engines/engine.py.229: process: 0.89451 | load: 0.00383 [I 2022-10-11 21:25:08] ...trap/engines/engine.py.230: loss: 0.08948 [I 2022-10-11 21:25:08] ...optimizers/trijoint.py.151: total_norm: 10.529161 [I 2022-10-11 21:25:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 920/2304 [I 2022-10-11 21:25:16] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:20:02 [I 2022-10-11 21:25:16] ...trap/engines/engine.py.229: process: 0.85413 | load: 0.00382 [I 2022-10-11 21:25:17] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-11 21:25:17] ...optimizers/trijoint.py.151: total_norm: 39.310436 [I 2022-10-11 21:25:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 930/2304 [I 2022-10-11 21:25:25] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:20:01 [I 2022-10-11 21:25:25] ...trap/engines/engine.py.229: process: 0.87608 | load: 0.00380 [I 2022-10-11 21:25:25] ...trap/engines/engine.py.230: loss: 0.24097 [I 2022-10-11 21:25:25] ...optimizers/trijoint.py.151: total_norm: 26.029675 [I 2022-10-11 21:25:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 940/2304 [I 2022-10-11 21:25:34] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:19:55 [I 2022-10-11 21:25:34] ...trap/engines/engine.py.229: process: 0.88157 | load: 0.00377 [I 2022-10-11 21:25:34] ...trap/engines/engine.py.230: loss: 0.08104 [I 2022-10-11 21:25:34] ...optimizers/trijoint.py.151: total_norm: 47.864052 [I 2022-10-11 21:25:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 950/2304 [I 2022-10-11 21:25:43] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:19:26 [I 2022-10-11 21:25:43] ...trap/engines/engine.py.229: process: 0.85655 | load: 0.00317 [I 2022-10-11 21:25:43] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-11 21:25:43] ...optimizers/trijoint.py.151: total_norm: 19.244150 [I 2022-10-11 21:25:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 960/2304 [I 2022-10-11 21:25:51] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:19:11 [I 2022-10-11 21:25:51] ...trap/engines/engine.py.229: process: 0.85969 | load: 0.00316 [I 2022-10-11 21:25:51] ...trap/engines/engine.py.230: loss: 0.12571 [I 2022-10-11 21:25:51] ...optimizers/trijoint.py.151: total_norm: 14.715795 [I 2022-10-11 21:26:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 970/2304 [I 2022-10-11 21:26:00] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:19:04 [I 2022-10-11 21:26:00] ...trap/engines/engine.py.229: process: 0.85675 | load: 0.00312 [I 2022-10-11 21:26:00] ...trap/engines/engine.py.230: loss: 0.14777 [I 2022-10-11 21:26:00] ...optimizers/trijoint.py.151: total_norm: 12.966694 [I 2022-10-11 21:26:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 980/2304 [I 2022-10-11 21:26:08] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:18:57 [I 2022-10-11 21:26:08] ...trap/engines/engine.py.229: process: 0.85785 | load: 0.00313 [I 2022-10-11 21:26:08] ...trap/engines/engine.py.230: loss: 0.11516 [I 2022-10-11 21:26:08] ...optimizers/trijoint.py.151: total_norm: 13.944408 [I 2022-10-11 21:26:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 990/2304 [I 2022-10-11 21:26:17] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:18:47 [I 2022-10-11 21:26:17] ...trap/engines/engine.py.229: process: 0.85831 | load: 0.00312 [I 2022-10-11 21:26:17] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-11 21:26:17] ...optimizers/trijoint.py.151: total_norm: 27.519979 [I 2022-10-11 21:26:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1000/2304 [I 2022-10-11 21:26:26] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:18:46 [I 2022-10-11 21:26:26] ...trap/engines/engine.py.229: process: 0.86552 | load: 0.00315 [I 2022-10-11 21:26:26] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-11 21:26:26] ...optimizers/trijoint.py.151: total_norm: 11.182028 [I 2022-10-11 21:26:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 1010/2304 [I 2022-10-11 21:26:34] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:18:25 [I 2022-10-11 21:26:34] ...trap/engines/engine.py.229: process: 0.85868 | load: 0.00315 [I 2022-10-11 21:26:34] ...trap/engines/engine.py.230: loss: 0.10253 [I 2022-10-11 21:26:34] ...optimizers/trijoint.py.151: total_norm: 13.300297 [I 2022-10-11 21:26:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1020/2304 [I 2022-10-11 21:26:43] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:18:24 [I 2022-10-11 21:26:43] ...trap/engines/engine.py.229: process: 0.85239 | load: 0.00317 [I 2022-10-11 21:26:43] ...trap/engines/engine.py.230: loss: 0.13831 [I 2022-10-11 21:26:43] ...optimizers/trijoint.py.151: total_norm: 15.945305 [I 2022-10-11 21:26:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 1030/2304 [I 2022-10-11 21:26:52] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:18:23 [I 2022-10-11 21:26:52] ...trap/engines/engine.py.229: process: 0.87842 | load: 0.00711 [I 2022-10-11 21:26:52] ...trap/engines/engine.py.230: loss: 0.06874 [I 2022-10-11 21:26:52] ...optimizers/trijoint.py.151: total_norm: 19.945591 [I 2022-10-11 21:27:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1040/2304 [I 2022-10-11 21:27:00] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:18:01 [I 2022-10-11 21:27:00] ...trap/engines/engine.py.229: process: 0.85606 | load: 0.00313 [I 2022-10-11 21:27:00] ...trap/engines/engine.py.230: loss: 0.10047 [I 2022-10-11 21:27:00] ...optimizers/trijoint.py.151: total_norm: 14.094128 [I 2022-10-11 21:27:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 1050/2304 [I 2022-10-11 21:27:09] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:17:56 [I 2022-10-11 21:27:09] ...trap/engines/engine.py.229: process: 0.87222 | load: 0.00313 [I 2022-10-11 21:27:09] ...trap/engines/engine.py.230: loss: 0.11924 [I 2022-10-11 21:27:09] ...optimizers/trijoint.py.151: total_norm: 15.591536 [I 2022-10-11 21:27:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 1060/2304 [I 2022-10-11 21:27:17] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:17:48 [I 2022-10-11 21:27:17] ...trap/engines/engine.py.229: process: 0.86662 | load: 0.00313 [I 2022-10-11 21:27:17] ...trap/engines/engine.py.230: loss: 0.07835 [I 2022-10-11 21:27:17] ...optimizers/trijoint.py.151: total_norm: 25.473642 [I 2022-10-11 21:27:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1070/2304 [I 2022-10-11 21:27:26] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:17:42 [I 2022-10-11 21:27:26] ...trap/engines/engine.py.229: process: 0.86508 | load: 0.00314 [I 2022-10-11 21:27:26] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-11 21:27:26] ...optimizers/trijoint.py.151: total_norm: 55.695747 [I 2022-10-11 21:27:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 1080/2304 [I 2022-10-11 21:27:35] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:17:40 [I 2022-10-11 21:27:35] ...trap/engines/engine.py.229: process: 0.87627 | load: 0.00314 [I 2022-10-11 21:27:35] ...trap/engines/engine.py.230: loss: 0.11782 [I 2022-10-11 21:27:35] ...optimizers/trijoint.py.151: total_norm: 22.108057 [I 2022-10-11 21:27:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1090/2304 [I 2022-10-11 21:27:43] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:17:32 [I 2022-10-11 21:27:43] ...trap/engines/engine.py.229: process: 0.86532 | load: 0.00321 [I 2022-10-11 21:27:43] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-11 21:27:43] ...optimizers/trijoint.py.151: total_norm: 13.344382 [I 2022-10-11 21:27:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 1100/2304 [I 2022-10-11 21:27:52] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:17:21 [I 2022-10-11 21:27:52] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00324 [I 2022-10-11 21:27:52] ...trap/engines/engine.py.230: loss: 0.11349 [I 2022-10-11 21:27:52] ...optimizers/trijoint.py.151: total_norm: 15.655870 [I 2022-10-11 21:28:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 1110/2304 [I 2022-10-11 21:28:01] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:17:14 [I 2022-10-11 21:28:01] ...trap/engines/engine.py.229: process: 0.86821 | load: 0.00312 [I 2022-10-11 21:28:01] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-11 21:28:01] ...optimizers/trijoint.py.151: total_norm: 19.891548 [I 2022-10-11 21:28:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 1120/2304 [I 2022-10-11 21:28:09] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:17:01 [I 2022-10-11 21:28:09] ...trap/engines/engine.py.229: process: 0.85031 | load: 0.00314 [I 2022-10-11 21:28:09] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-11 21:28:09] ...optimizers/trijoint.py.151: total_norm: 19.450445 [I 2022-10-11 21:28:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 1130/2304 [I 2022-10-11 21:28:18] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:16:52 [I 2022-10-11 21:28:18] ...trap/engines/engine.py.229: process: 0.86000 | load: 0.00312 [I 2022-10-11 21:28:18] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-11 21:28:18] ...optimizers/trijoint.py.151: total_norm: 15.135956 [I 2022-10-11 21:28:27] ...trap/engines/engine.py.224: train: epoch 119 | batch 1140/2304 [I 2022-10-11 21:28:27] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:16:35 [I 2022-10-11 21:28:27] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00377 [I 2022-10-11 21:28:27] ...trap/engines/engine.py.230: loss: 0.08691 [I 2022-10-11 21:28:27] ...optimizers/trijoint.py.151: total_norm: 12.498219 [I 2022-10-11 21:28:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 1150/2304 [I 2022-10-11 21:28:35] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:16:24 [I 2022-10-11 21:28:35] ...trap/engines/engine.py.229: process: 0.85689 | load: 0.00316 [I 2022-10-11 21:28:35] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-11 21:28:35] ...optimizers/trijoint.py.151: total_norm: 11.042928 [I 2022-10-11 21:28:44] ...trap/engines/engine.py.224: train: epoch 119 | batch 1160/2304 [I 2022-10-11 21:28:44] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:16:18 [I 2022-10-11 21:28:44] ...trap/engines/engine.py.229: process: 0.86194 | load: 0.00315 [I 2022-10-11 21:28:44] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-11 21:28:44] ...optimizers/trijoint.py.151: total_norm: 26.661005 [I 2022-10-11 21:28:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 1170/2304 [I 2022-10-11 21:28:52] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:16:11 [I 2022-10-11 21:28:52] ...trap/engines/engine.py.229: process: 0.85738 | load: 0.00313 [I 2022-10-11 21:28:52] ...trap/engines/engine.py.230: loss: 0.09282 [I 2022-10-11 21:28:52] ...optimizers/trijoint.py.151: total_norm: 14.372569 [I 2022-10-11 21:29:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 1180/2304 [I 2022-10-11 21:29:01] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:16:04 [I 2022-10-11 21:29:01] ...trap/engines/engine.py.229: process: 0.85657 | load: 0.00319 [I 2022-10-11 21:29:01] ...trap/engines/engine.py.230: loss: 0.12998 [I 2022-10-11 21:29:01] ...optimizers/trijoint.py.151: total_norm: 14.908324 [I 2022-10-11 21:29:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 1190/2304 [I 2022-10-11 21:29:09] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:15:57 [I 2022-10-11 21:29:09] ...trap/engines/engine.py.229: process: 0.85773 | load: 0.00315 [I 2022-10-11 21:29:09] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-11 21:29:09] ...optimizers/trijoint.py.151: total_norm: 14.157883 [I 2022-10-11 21:29:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 1200/2304 [I 2022-10-11 21:29:18] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:15:46 [I 2022-10-11 21:29:18] ...trap/engines/engine.py.229: process: 0.86448 | load: 0.00319 [I 2022-10-11 21:29:18] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-11 21:29:18] ...optimizers/trijoint.py.151: total_norm: 17.293890 [I 2022-10-11 21:29:27] ...trap/engines/engine.py.224: train: epoch 119 | batch 1210/2304 [I 2022-10-11 21:29:27] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:15:38 [I 2022-10-11 21:29:27] ...trap/engines/engine.py.229: process: 0.85958 | load: 0.00313 [I 2022-10-11 21:29:27] ...trap/engines/engine.py.230: loss: 0.08301 [I 2022-10-11 21:29:27] ...optimizers/trijoint.py.151: total_norm: 10.801221 [I 2022-10-11 21:29:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 1220/2304 [I 2022-10-11 21:29:35] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:15:30 [I 2022-10-11 21:29:35] ...trap/engines/engine.py.229: process: 0.85948 | load: 0.00313 [I 2022-10-11 21:29:35] ...trap/engines/engine.py.230: loss: 0.13683 [I 2022-10-11 21:29:35] ...optimizers/trijoint.py.151: total_norm: 22.209736 [I 2022-10-11 21:29:44] ...trap/engines/engine.py.224: train: epoch 119 | batch 1230/2304 [I 2022-10-11 21:29:44] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:15:07 [I 2022-10-11 21:29:44] ...trap/engines/engine.py.229: process: 0.84159 | load: 0.00296 [I 2022-10-11 21:29:44] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-11 21:29:44] ...optimizers/trijoint.py.151: total_norm: 12.045238 [I 2022-10-11 21:29:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 1240/2304 [I 2022-10-11 21:29:52] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:15:00 [I 2022-10-11 21:29:52] ...trap/engines/engine.py.229: process: 0.84920 | load: 0.00297 [I 2022-10-11 21:29:52] ...trap/engines/engine.py.230: loss: 0.10735 [I 2022-10-11 21:29:52] ...optimizers/trijoint.py.151: total_norm: 11.307689 [I 2022-10-11 21:30:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1250/2304 [I 2022-10-11 21:30:00] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:14:51 [I 2022-10-11 21:30:00] ...trap/engines/engine.py.229: process: 0.84532 | load: 0.00293 [I 2022-10-11 21:30:00] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-11 21:30:01] ...optimizers/trijoint.py.151: total_norm: 15.660848 [I 2022-10-11 21:30:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 1260/2304 [I 2022-10-11 21:30:09] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:14:34 [I 2022-10-11 21:30:09] ...trap/engines/engine.py.229: process: 0.83786 | load: 0.00296 [I 2022-10-11 21:30:09] ...trap/engines/engine.py.230: loss: 0.12167 [I 2022-10-11 21:30:09] ...optimizers/trijoint.py.151: total_norm: 11.988207 [I 2022-10-11 21:30:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 1270/2304 [I 2022-10-11 21:30:17] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:14:34 [I 2022-10-11 21:30:17] ...trap/engines/engine.py.229: process: 0.85376 | load: 0.00295 [I 2022-10-11 21:30:17] ...trap/engines/engine.py.230: loss: 0.13376 [I 2022-10-11 21:30:17] ...optimizers/trijoint.py.151: total_norm: 39.956715 [I 2022-10-11 21:30:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1280/2304 [I 2022-10-11 21:30:26] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:14:24 [I 2022-10-11 21:30:26] ...trap/engines/engine.py.229: process: 0.84029 | load: 0.00297 [I 2022-10-11 21:30:26] ...trap/engines/engine.py.230: loss: 0.10238 [I 2022-10-11 21:30:26] ...optimizers/trijoint.py.151: total_norm: 13.619346 [I 2022-10-11 21:30:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 1290/2304 [I 2022-10-11 21:30:34] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:14:17 [I 2022-10-11 21:30:34] ...trap/engines/engine.py.229: process: 0.84971 | load: 0.00299 [I 2022-10-11 21:30:34] ...trap/engines/engine.py.230: loss: 0.10042 [I 2022-10-11 21:30:34] ...optimizers/trijoint.py.151: total_norm: 15.730587 [I 2022-10-11 21:30:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1300/2304 [I 2022-10-11 21:30:43] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:14:05 [I 2022-10-11 21:30:43] ...trap/engines/engine.py.229: process: 0.84343 | load: 0.00296 [I 2022-10-11 21:30:43] ...trap/engines/engine.py.230: loss: 0.09971 [I 2022-10-11 21:30:43] ...optimizers/trijoint.py.151: total_norm: 12.023983 [I 2022-10-11 21:30:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 1310/2304 [I 2022-10-11 21:30:51] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:14:01 [I 2022-10-11 21:30:51] ...trap/engines/engine.py.229: process: 0.84361 | load: 0.00294 [I 2022-10-11 21:30:51] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-11 21:30:51] ...optimizers/trijoint.py.151: total_norm: 27.059198 [I 2022-10-11 21:31:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1320/2304 [I 2022-10-11 21:31:00] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:13:56 [I 2022-10-11 21:31:00] ...trap/engines/engine.py.229: process: 0.86351 | load: 0.00299 [I 2022-10-11 21:31:00] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-11 21:31:00] ...optimizers/trijoint.py.151: total_norm: 9.787883 [I 2022-10-11 21:31:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 1330/2304 [I 2022-10-11 21:31:08] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:13:46 [I 2022-10-11 21:31:08] ...trap/engines/engine.py.229: process: 0.85808 | load: 0.00294 [I 2022-10-11 21:31:08] ...trap/engines/engine.py.230: loss: 0.11006 [I 2022-10-11 21:31:08] ...optimizers/trijoint.py.151: total_norm: 9.940872 [I 2022-10-11 21:31:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 1340/2304 [I 2022-10-11 21:31:17] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:13:38 [I 2022-10-11 21:31:17] ...trap/engines/engine.py.229: process: 0.86622 | load: 0.00294 [I 2022-10-11 21:31:17] ...trap/engines/engine.py.230: loss: 0.09214 [I 2022-10-11 21:31:17] ...optimizers/trijoint.py.151: total_norm: 9.275702 [I 2022-10-11 21:31:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 1350/2304 [I 2022-10-11 21:31:25] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:13:29 [I 2022-10-11 21:31:25] ...trap/engines/engine.py.229: process: 0.84603 | load: 0.00296 [I 2022-10-11 21:31:25] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-11 21:31:25] ...optimizers/trijoint.py.151: total_norm: 13.409527 [I 2022-10-11 21:31:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 1360/2304 [I 2022-10-11 21:31:34] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:13:17 [I 2022-10-11 21:31:34] ...trap/engines/engine.py.229: process: 0.84095 | load: 0.00296 [I 2022-10-11 21:31:34] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-11 21:31:34] ...optimizers/trijoint.py.151: total_norm: 22.597935 [I 2022-10-11 21:31:42] ...trap/engines/engine.py.224: train: epoch 119 | batch 1370/2304 [I 2022-10-11 21:31:42] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:13:07 [I 2022-10-11 21:31:42] ...trap/engines/engine.py.229: process: 0.84829 | load: 0.00296 [I 2022-10-11 21:31:42] ...trap/engines/engine.py.230: loss: 0.11355 [I 2022-10-11 21:31:42] ...optimizers/trijoint.py.151: total_norm: 13.965360 [I 2022-10-11 21:31:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 1380/2304 [I 2022-10-11 21:31:51] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:12:57 [I 2022-10-11 21:31:51] ...trap/engines/engine.py.229: process: 0.82855 | load: 0.00298 [I 2022-10-11 21:31:51] ...trap/engines/engine.py.230: loss: 0.11302 [I 2022-10-11 21:31:51] ...optimizers/trijoint.py.151: total_norm: 10.695850 [I 2022-10-11 21:31:59] ...trap/engines/engine.py.224: train: epoch 119 | batch 1390/2304 [I 2022-10-11 21:31:59] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:12:57 [I 2022-10-11 21:31:59] ...trap/engines/engine.py.229: process: 0.86099 | load: 0.00297 [I 2022-10-11 21:31:59] ...trap/engines/engine.py.230: loss: 0.09352 [I 2022-10-11 21:31:59] ...optimizers/trijoint.py.151: total_norm: 9.254171 [I 2022-10-11 21:32:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 1400/2304 [I 2022-10-11 21:32:08] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:12:49 [I 2022-10-11 21:32:08] ...trap/engines/engine.py.229: process: 0.85236 | load: 0.00301 [I 2022-10-11 21:32:08] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-11 21:32:08] ...optimizers/trijoint.py.151: total_norm: 14.698100 [I 2022-10-11 21:32:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 1410/2304 [I 2022-10-11 21:32:16] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:12:32 [I 2022-10-11 21:32:16] ...trap/engines/engine.py.229: process: 0.83729 | load: 0.00296 [I 2022-10-11 21:32:16] ...trap/engines/engine.py.230: loss: 0.13750 [I 2022-10-11 21:32:16] ...optimizers/trijoint.py.151: total_norm: 19.190731 [I 2022-10-11 21:32:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 1420/2304 [I 2022-10-11 21:32:24] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:12:22 [I 2022-10-11 21:32:24] ...trap/engines/engine.py.229: process: 0.84277 | load: 0.00300 [I 2022-10-11 21:32:24] ...trap/engines/engine.py.230: loss: 0.09620 [I 2022-10-11 21:32:24] ...optimizers/trijoint.py.151: total_norm: 17.703056 [I 2022-10-11 21:32:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 1430/2304 [I 2022-10-11 21:32:33] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:12:17 [I 2022-10-11 21:32:33] ...trap/engines/engine.py.229: process: 0.84507 | load: 0.00296 [I 2022-10-11 21:32:33] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-11 21:32:33] ...optimizers/trijoint.py.151: total_norm: 14.439857 [I 2022-10-11 21:32:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 1440/2304 [I 2022-10-11 21:32:41] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:12:11 [I 2022-10-11 21:32:41] ...trap/engines/engine.py.229: process: 0.86053 | load: 0.00296 [I 2022-10-11 21:32:41] ...trap/engines/engine.py.230: loss: 0.13773 [I 2022-10-11 21:32:41] ...optimizers/trijoint.py.151: total_norm: 20.248926 [I 2022-10-11 21:32:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 1450/2304 [I 2022-10-11 21:32:50] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:12:01 [I 2022-10-11 21:32:50] ...trap/engines/engine.py.229: process: 0.84564 | load: 0.00294 [I 2022-10-11 21:32:50] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-11 21:32:50] ...optimizers/trijoint.py.151: total_norm: 19.356304 [I 2022-10-11 21:32:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1460/2304 [I 2022-10-11 21:32:58] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:11:50 [I 2022-10-11 21:32:58] ...trap/engines/engine.py.229: process: 0.83577 | load: 0.00295 [I 2022-10-11 21:32:58] ...trap/engines/engine.py.230: loss: 0.08803 [I 2022-10-11 21:32:58] ...optimizers/trijoint.py.151: total_norm: 12.347074 [I 2022-10-11 21:33:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 1470/2304 [I 2022-10-11 21:33:07] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:11:43 [I 2022-10-11 21:33:07] ...trap/engines/engine.py.229: process: 0.84783 | load: 0.00295 [I 2022-10-11 21:33:07] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-11 21:33:07] ...optimizers/trijoint.py.151: total_norm: 35.222878 [I 2022-10-11 21:33:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1480/2304 [I 2022-10-11 21:33:15] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:11:37 [I 2022-10-11 21:33:15] ...trap/engines/engine.py.229: process: 0.85441 | load: 0.00296 [I 2022-10-11 21:33:15] ...trap/engines/engine.py.230: loss: 0.09909 [I 2022-10-11 21:33:15] ...optimizers/trijoint.py.151: total_norm: 21.704309 [I 2022-10-11 21:33:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 1490/2304 [I 2022-10-11 21:33:24] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:11:30 [I 2022-10-11 21:33:24] ...trap/engines/engine.py.229: process: 0.84963 | load: 0.00294 [I 2022-10-11 21:33:24] ...trap/engines/engine.py.230: loss: 0.10041 [I 2022-10-11 21:33:24] ...optimizers/trijoint.py.151: total_norm: 12.554449 [I 2022-10-11 21:33:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1500/2304 [I 2022-10-11 21:33:32] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:11:17 [I 2022-10-11 21:33:32] ...trap/engines/engine.py.229: process: 0.84589 | load: 0.00301 [I 2022-10-11 21:33:32] ...trap/engines/engine.py.230: loss: 0.12013 [I 2022-10-11 21:33:32] ...optimizers/trijoint.py.151: total_norm: 14.413974 [I 2022-10-11 21:33:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 1510/2304 [I 2022-10-11 21:33:41] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:11:07 [I 2022-10-11 21:33:41] ...trap/engines/engine.py.229: process: 0.83371 | load: 0.00295 [I 2022-10-11 21:33:41] ...trap/engines/engine.py.230: loss: 0.12112 [I 2022-10-11 21:33:41] ...optimizers/trijoint.py.151: total_norm: 25.680511 [I 2022-10-11 21:33:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1520/2304 [I 2022-10-11 21:33:49] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:10:59 [I 2022-10-11 21:33:49] ...trap/engines/engine.py.229: process: 0.84635 | load: 0.00300 [I 2022-10-11 21:33:49] ...trap/engines/engine.py.230: loss: 0.09370 [I 2022-10-11 21:33:49] ...optimizers/trijoint.py.151: total_norm: 33.031952 [I 2022-10-11 21:33:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1530/2304 [I 2022-10-11 21:33:57] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:10:54 [I 2022-10-11 21:33:57] ...trap/engines/engine.py.229: process: 0.84328 | load: 0.00301 [I 2022-10-11 21:33:57] ...trap/engines/engine.py.230: loss: 0.10710 [I 2022-10-11 21:33:57] ...optimizers/trijoint.py.151: total_norm: 12.175015 [I 2022-10-11 21:34:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1540/2304 [I 2022-10-11 21:34:06] ...trap/engines/engine.py.228: elapsed: 0:22:04 | left: 0:10:44 [I 2022-10-11 21:34:06] ...trap/engines/engine.py.229: process: 0.84852 | load: 0.00296 [I 2022-10-11 21:34:06] ...trap/engines/engine.py.230: loss: 0.14524 [I 2022-10-11 21:34:06] ...optimizers/trijoint.py.151: total_norm: 21.268583 [I 2022-10-11 21:34:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1550/2304 [I 2022-10-11 21:34:14] ...trap/engines/engine.py.228: elapsed: 0:22:13 | left: 0:10:39 [I 2022-10-11 21:34:14] ...trap/engines/engine.py.229: process: 0.85429 | load: 0.00300 [I 2022-10-11 21:34:14] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-11 21:34:14] ...optimizers/trijoint.py.151: total_norm: 32.223278 [I 2022-10-11 21:34:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1560/2304 [I 2022-10-11 21:34:23] ...trap/engines/engine.py.228: elapsed: 0:22:21 | left: 0:10:27 [I 2022-10-11 21:34:23] ...trap/engines/engine.py.229: process: 0.85320 | load: 0.00299 [I 2022-10-11 21:34:23] ...trap/engines/engine.py.230: loss: 0.08024 [I 2022-10-11 21:34:23] ...optimizers/trijoint.py.151: total_norm: 14.420877 [I 2022-10-11 21:34:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 1570/2304 [I 2022-10-11 21:34:31] ...trap/engines/engine.py.228: elapsed: 0:22:30 | left: 0:10:20 [I 2022-10-11 21:34:31] ...trap/engines/engine.py.229: process: 0.84477 | load: 0.00302 [I 2022-10-11 21:34:31] ...trap/engines/engine.py.230: loss: 0.10686 [I 2022-10-11 21:34:31] ...optimizers/trijoint.py.151: total_norm: 16.153702 [I 2022-10-11 21:34:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1580/2304 [I 2022-10-11 21:34:40] ...trap/engines/engine.py.228: elapsed: 0:22:38 | left: 0:10:15 [I 2022-10-11 21:34:40] ...trap/engines/engine.py.229: process: 0.86072 | load: 0.00294 [I 2022-10-11 21:34:40] ...trap/engines/engine.py.230: loss: 0.12993 [I 2022-10-11 21:34:40] ...optimizers/trijoint.py.151: total_norm: 12.378951 [I 2022-10-11 21:34:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 1590/2304 [I 2022-10-11 21:34:48] ...trap/engines/engine.py.228: elapsed: 0:22:47 | left: 0:10:01 [I 2022-10-11 21:34:48] ...trap/engines/engine.py.229: process: 0.85521 | load: 0.00296 [I 2022-10-11 21:34:48] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-11 21:34:48] ...optimizers/trijoint.py.151: total_norm: 17.198643 [I 2022-10-11 21:34:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1600/2304 [I 2022-10-11 21:34:57] ...trap/engines/engine.py.228: elapsed: 0:22:55 | left: 0:09:57 [I 2022-10-11 21:34:57] ...trap/engines/engine.py.229: process: 0.85203 | load: 0.00297 [I 2022-10-11 21:34:57] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-11 21:34:57] ...optimizers/trijoint.py.151: total_norm: 18.467760 [I 2022-10-11 21:35:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 1610/2304 [I 2022-10-11 21:35:05] ...trap/engines/engine.py.228: elapsed: 0:23:04 | left: 0:09:52 [I 2022-10-11 21:35:05] ...trap/engines/engine.py.229: process: 0.85314 | load: 0.00295 [I 2022-10-11 21:35:05] ...trap/engines/engine.py.230: loss: 0.09646 [I 2022-10-11 21:35:05] ...optimizers/trijoint.py.151: total_norm: 17.695213 [I 2022-10-11 21:35:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1620/2304 [I 2022-10-11 21:35:14] ...trap/engines/engine.py.228: elapsed: 0:23:12 | left: 0:09:50 [I 2022-10-11 21:35:14] ...trap/engines/engine.py.229: process: 0.86519 | load: 0.00313 [I 2022-10-11 21:35:14] ...trap/engines/engine.py.230: loss: 0.13711 [I 2022-10-11 21:35:14] ...optimizers/trijoint.py.151: total_norm: 14.771238 [I 2022-10-11 21:35:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1630/2304 [I 2022-10-11 21:35:23] ...trap/engines/engine.py.228: elapsed: 0:23:21 | left: 0:09:41 [I 2022-10-11 21:35:23] ...trap/engines/engine.py.229: process: 0.86626 | load: 0.00315 [I 2022-10-11 21:35:23] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-11 21:35:23] ...optimizers/trijoint.py.151: total_norm: 46.351616 [I 2022-10-11 21:35:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 1640/2304 [I 2022-10-11 21:35:31] ...trap/engines/engine.py.228: elapsed: 0:23:29 | left: 0:09:29 [I 2022-10-11 21:35:31] ...trap/engines/engine.py.229: process: 0.85291 | load: 0.00317 [I 2022-10-11 21:35:31] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-11 21:35:31] ...optimizers/trijoint.py.151: total_norm: 16.654823 [I 2022-10-11 21:35:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1650/2304 [I 2022-10-11 21:35:40] ...trap/engines/engine.py.228: elapsed: 0:23:38 | left: 0:09:24 [I 2022-10-11 21:35:40] ...trap/engines/engine.py.229: process: 0.87332 | load: 0.00314 [I 2022-10-11 21:35:40] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-11 21:35:40] ...optimizers/trijoint.py.151: total_norm: 14.335340 [I 2022-10-11 21:35:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 1660/2304 [I 2022-10-11 21:35:48] ...trap/engines/engine.py.228: elapsed: 0:23:47 | left: 0:09:12 [I 2022-10-11 21:35:48] ...trap/engines/engine.py.229: process: 0.84621 | load: 0.00315 [I 2022-10-11 21:35:48] ...trap/engines/engine.py.230: loss: 0.09344 [I 2022-10-11 21:35:48] ...optimizers/trijoint.py.151: total_norm: 12.860737 [I 2022-10-11 21:35:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1670/2304 [I 2022-10-11 21:35:57] ...trap/engines/engine.py.228: elapsed: 0:23:55 | left: 0:09:07 [I 2022-10-11 21:35:57] ...trap/engines/engine.py.229: process: 0.86430 | load: 0.00316 [I 2022-10-11 21:35:57] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-11 21:35:57] ...optimizers/trijoint.py.151: total_norm: 15.287313 [I 2022-10-11 21:36:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1680/2304 [I 2022-10-11 21:36:06] ...trap/engines/engine.py.228: elapsed: 0:24:04 | left: 0:08:58 [I 2022-10-11 21:36:06] ...trap/engines/engine.py.229: process: 0.86734 | load: 0.00315 [I 2022-10-11 21:36:06] ...trap/engines/engine.py.230: loss: 0.08100 [I 2022-10-11 21:36:06] ...optimizers/trijoint.py.151: total_norm: 10.685309 [I 2022-10-11 21:36:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1690/2304 [I 2022-10-11 21:36:14] ...trap/engines/engine.py.228: elapsed: 0:24:13 | left: 0:08:53 [I 2022-10-11 21:36:14] ...trap/engines/engine.py.229: process: 0.88044 | load: 0.00317 [I 2022-10-11 21:36:14] ...trap/engines/engine.py.230: loss: 0.12640 [I 2022-10-11 21:36:14] ...optimizers/trijoint.py.151: total_norm: 13.724504 [I 2022-10-11 21:36:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1700/2304 [I 2022-10-11 21:36:23] ...trap/engines/engine.py.228: elapsed: 0:24:21 | left: 0:08:40 [I 2022-10-11 21:36:23] ...trap/engines/engine.py.229: process: 0.86799 | load: 0.00318 [I 2022-10-11 21:36:23] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-11 21:36:23] ...optimizers/trijoint.py.151: total_norm: 22.732725 [I 2022-10-11 21:36:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1710/2304 [I 2022-10-11 21:36:32] ...trap/engines/engine.py.228: elapsed: 0:24:30 | left: 0:08:36 [I 2022-10-11 21:36:32] ...trap/engines/engine.py.229: process: 0.86973 | load: 0.00314 [I 2022-10-11 21:36:32] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-11 21:36:32] ...optimizers/trijoint.py.151: total_norm: 21.502285 [I 2022-10-11 21:36:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1720/2304 [I 2022-10-11 21:36:40] ...trap/engines/engine.py.228: elapsed: 0:24:39 | left: 0:08:22 [I 2022-10-11 21:36:40] ...trap/engines/engine.py.229: process: 0.85130 | load: 0.00318 [I 2022-10-11 21:36:40] ...trap/engines/engine.py.230: loss: 0.12721 [I 2022-10-11 21:36:40] ...optimizers/trijoint.py.151: total_norm: 15.913986 [I 2022-10-11 21:36:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1730/2304 [I 2022-10-11 21:36:49] ...trap/engines/engine.py.228: elapsed: 0:24:47 | left: 0:08:10 [I 2022-10-11 21:36:49] ...trap/engines/engine.py.229: process: 0.83676 | load: 0.00317 [I 2022-10-11 21:36:49] ...trap/engines/engine.py.230: loss: 0.08009 [I 2022-10-11 21:36:49] ...optimizers/trijoint.py.151: total_norm: 15.690671 [I 2022-10-11 21:36:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1740/2304 [I 2022-10-11 21:36:58] ...trap/engines/engine.py.228: elapsed: 0:24:56 | left: 0:08:07 [I 2022-10-11 21:36:58] ...trap/engines/engine.py.229: process: 0.86016 | load: 0.00320 [I 2022-10-11 21:36:58] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-11 21:36:58] ...optimizers/trijoint.py.151: total_norm: 12.268122 [I 2022-10-11 21:37:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1750/2304 [I 2022-10-11 21:37:06] ...trap/engines/engine.py.228: elapsed: 0:25:04 | left: 0:07:52 [I 2022-10-11 21:37:06] ...trap/engines/engine.py.229: process: 0.85457 | load: 0.00316 [I 2022-10-11 21:37:06] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-11 21:37:06] ...optimizers/trijoint.py.151: total_norm: 12.283725 [I 2022-10-11 21:37:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1760/2304 [I 2022-10-11 21:37:15] ...trap/engines/engine.py.228: elapsed: 0:25:13 | left: 0:07:45 [I 2022-10-11 21:37:15] ...trap/engines/engine.py.229: process: 0.86650 | load: 0.00312 [I 2022-10-11 21:37:15] ...trap/engines/engine.py.230: loss: 0.08945 [I 2022-10-11 21:37:15] ...optimizers/trijoint.py.151: total_norm: 14.115325 [I 2022-10-11 21:37:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1770/2304 [I 2022-10-11 21:37:23] ...trap/engines/engine.py.228: elapsed: 0:25:22 | left: 0:07:37 [I 2022-10-11 21:37:23] ...trap/engines/engine.py.229: process: 0.85224 | load: 0.00313 [I 2022-10-11 21:37:23] ...trap/engines/engine.py.230: loss: 0.10753 [I 2022-10-11 21:37:23] ...optimizers/trijoint.py.151: total_norm: 12.308384 [I 2022-10-11 21:37:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1780/2304 [I 2022-10-11 21:37:32] ...trap/engines/engine.py.228: elapsed: 0:25:30 | left: 0:07:29 [I 2022-10-11 21:37:32] ...trap/engines/engine.py.229: process: 0.85860 | load: 0.00313 [I 2022-10-11 21:37:32] ...trap/engines/engine.py.230: loss: 0.11653 [I 2022-10-11 21:37:32] ...optimizers/trijoint.py.151: total_norm: 12.353763 [I 2022-10-11 21:37:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 1790/2304 [I 2022-10-11 21:37:41] ...trap/engines/engine.py.228: elapsed: 0:25:39 | left: 0:07:22 [I 2022-10-11 21:37:41] ...trap/engines/engine.py.229: process: 0.85711 | load: 0.00317 [I 2022-10-11 21:37:41] ...trap/engines/engine.py.230: loss: 0.09144 [I 2022-10-11 21:37:41] ...optimizers/trijoint.py.151: total_norm: 19.407549 [I 2022-10-11 21:37:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1800/2304 [I 2022-10-11 21:37:49] ...trap/engines/engine.py.228: elapsed: 0:25:47 | left: 0:07:11 [I 2022-10-11 21:37:49] ...trap/engines/engine.py.229: process: 0.84913 | load: 0.00313 [I 2022-10-11 21:37:49] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-11 21:37:49] ...optimizers/trijoint.py.151: total_norm: 16.455019 [I 2022-10-11 21:37:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1810/2304 [I 2022-10-11 21:37:58] ...trap/engines/engine.py.228: elapsed: 0:25:56 | left: 0:07:03 [I 2022-10-11 21:37:58] ...trap/engines/engine.py.229: process: 0.85979 | load: 0.00314 [I 2022-10-11 21:37:58] ...trap/engines/engine.py.230: loss: 0.14158 [I 2022-10-11 21:37:58] ...optimizers/trijoint.py.151: total_norm: 29.005701 [I 2022-10-11 21:38:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1820/2304 [I 2022-10-11 21:38:06] ...trap/engines/engine.py.228: elapsed: 0:26:05 | left: 0:06:55 [I 2022-10-11 21:38:06] ...trap/engines/engine.py.229: process: 0.86666 | load: 0.00314 [I 2022-10-11 21:38:06] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-11 21:38:06] ...optimizers/trijoint.py.151: total_norm: 14.955359 [I 2022-10-11 21:38:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1830/2304 [I 2022-10-11 21:38:15] ...trap/engines/engine.py.228: elapsed: 0:26:13 | left: 0:06:50 [I 2022-10-11 21:38:15] ...trap/engines/engine.py.229: process: 0.86717 | load: 0.00315 [I 2022-10-11 21:38:15] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-11 21:38:15] ...optimizers/trijoint.py.151: total_norm: 22.001869 [I 2022-10-11 21:38:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 1840/2304 [I 2022-10-11 21:38:24] ...trap/engines/engine.py.228: elapsed: 0:26:22 | left: 0:06:37 [I 2022-10-11 21:38:24] ...trap/engines/engine.py.229: process: 0.86215 | load: 0.00318 [I 2022-10-11 21:38:24] ...trap/engines/engine.py.230: loss: 0.07823 [I 2022-10-11 21:38:24] ...optimizers/trijoint.py.151: total_norm: 9.189655 [I 2022-10-11 21:38:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1850/2304 [I 2022-10-11 21:38:32] ...trap/engines/engine.py.228: elapsed: 0:26:30 | left: 0:06:30 [I 2022-10-11 21:38:32] ...trap/engines/engine.py.229: process: 0.85434 | load: 0.00313 [I 2022-10-11 21:38:32] ...trap/engines/engine.py.230: loss: 0.09553 [I 2022-10-11 21:38:32] ...optimizers/trijoint.py.151: total_norm: 14.529798 [I 2022-10-11 21:38:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 1860/2304 [I 2022-10-11 21:38:41] ...trap/engines/engine.py.228: elapsed: 0:26:39 | left: 0:06:23 [I 2022-10-11 21:38:41] ...trap/engines/engine.py.229: process: 0.86425 | load: 0.00316 [I 2022-10-11 21:38:41] ...trap/engines/engine.py.230: loss: 0.09460 [I 2022-10-11 21:38:41] ...optimizers/trijoint.py.151: total_norm: 9.682312 [I 2022-10-11 21:38:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1870/2304 [I 2022-10-11 21:38:49] ...trap/engines/engine.py.228: elapsed: 0:26:48 | left: 0:06:14 [I 2022-10-11 21:38:49] ...trap/engines/engine.py.229: process: 0.86762 | load: 0.00313 [I 2022-10-11 21:38:49] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-11 21:38:49] ...optimizers/trijoint.py.151: total_norm: 10.349879 [I 2022-10-11 21:38:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1880/2304 [I 2022-10-11 21:38:58] ...trap/engines/engine.py.228: elapsed: 0:26:56 | left: 0:06:04 [I 2022-10-11 21:38:58] ...trap/engines/engine.py.229: process: 0.85073 | load: 0.00314 [I 2022-10-11 21:38:58] ...trap/engines/engine.py.230: loss: 0.12388 [I 2022-10-11 21:38:58] ...optimizers/trijoint.py.151: total_norm: 9.627590 [I 2022-10-11 21:39:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 1890/2304 [I 2022-10-11 21:39:07] ...trap/engines/engine.py.228: elapsed: 0:27:05 | left: 0:05:53 [I 2022-10-11 21:39:07] ...trap/engines/engine.py.229: process: 0.85138 | load: 0.00296 [I 2022-10-11 21:39:07] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-11 21:39:07] ...optimizers/trijoint.py.151: total_norm: 11.576441 [I 2022-10-11 21:39:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1900/2304 [I 2022-10-11 21:39:15] ...trap/engines/engine.py.228: elapsed: 0:27:13 | left: 0:05:41 [I 2022-10-11 21:39:15] ...trap/engines/engine.py.229: process: 0.85403 | load: 0.00319 [I 2022-10-11 21:39:15] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-11 21:39:15] ...optimizers/trijoint.py.151: total_norm: 29.583014 [I 2022-10-11 21:39:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 1910/2304 [I 2022-10-11 21:39:24] ...trap/engines/engine.py.228: elapsed: 0:27:22 | left: 0:05:31 [I 2022-10-11 21:39:24] ...trap/engines/engine.py.229: process: 0.83524 | load: 0.00297 [I 2022-10-11 21:39:24] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-11 21:39:24] ...optimizers/trijoint.py.151: total_norm: 12.920982 [I 2022-10-11 21:39:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1920/2304 [I 2022-10-11 21:39:32] ...trap/engines/engine.py.228: elapsed: 0:27:30 | left: 0:05:24 [I 2022-10-11 21:39:32] ...trap/engines/engine.py.229: process: 0.85165 | load: 0.00296 [I 2022-10-11 21:39:32] ...trap/engines/engine.py.230: loss: 0.08957 [I 2022-10-11 21:39:32] ...optimizers/trijoint.py.151: total_norm: 14.984617 [I 2022-10-11 21:39:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1930/2304 [I 2022-10-11 21:39:40] ...trap/engines/engine.py.228: elapsed: 0:27:39 | left: 0:05:16 [I 2022-10-11 21:39:40] ...trap/engines/engine.py.229: process: 0.84483 | load: 0.00298 [I 2022-10-11 21:39:40] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-11 21:39:40] ...optimizers/trijoint.py.151: total_norm: 23.513201 [I 2022-10-11 21:39:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1940/2304 [I 2022-10-11 21:39:49] ...trap/engines/engine.py.228: elapsed: 0:27:47 | left: 0:05:07 [I 2022-10-11 21:39:49] ...trap/engines/engine.py.229: process: 0.83013 | load: 0.00295 [I 2022-10-11 21:39:49] ...trap/engines/engine.py.230: loss: 0.06942 [I 2022-10-11 21:39:49] ...optimizers/trijoint.py.151: total_norm: 12.379461 [I 2022-10-11 21:39:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1950/2304 [I 2022-10-11 21:39:57] ...trap/engines/engine.py.228: elapsed: 0:27:56 | left: 0:05:01 [I 2022-10-11 21:39:57] ...trap/engines/engine.py.229: process: 0.86388 | load: 0.00295 [I 2022-10-11 21:39:57] ...trap/engines/engine.py.230: loss: 0.11207 [I 2022-10-11 21:39:57] ...optimizers/trijoint.py.151: total_norm: 8.160953 [I 2022-10-11 21:40:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1960/2304 [I 2022-10-11 21:40:06] ...trap/engines/engine.py.228: elapsed: 0:28:04 | left: 0:04:51 [I 2022-10-11 21:40:06] ...trap/engines/engine.py.229: process: 0.84800 | load: 0.00298 [I 2022-10-11 21:40:06] ...trap/engines/engine.py.230: loss: 0.08794 [I 2022-10-11 21:40:06] ...optimizers/trijoint.py.151: total_norm: 13.227598 [I 2022-10-11 21:40:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1970/2304 [I 2022-10-11 21:40:14] ...trap/engines/engine.py.228: elapsed: 0:28:13 | left: 0:04:41 [I 2022-10-11 21:40:14] ...trap/engines/engine.py.229: process: 0.85468 | load: 0.00297 [I 2022-10-11 21:40:14] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-11 21:40:14] ...optimizers/trijoint.py.151: total_norm: 30.986927 [I 2022-10-11 21:40:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1980/2304 [I 2022-10-11 21:40:23] ...trap/engines/engine.py.228: elapsed: 0:28:21 | left: 0:04:33 [I 2022-10-11 21:40:23] ...trap/engines/engine.py.229: process: 0.84498 | load: 0.00296 [I 2022-10-11 21:40:23] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-11 21:40:23] ...optimizers/trijoint.py.151: total_norm: 33.761391 [I 2022-10-11 21:40:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 1990/2304 [I 2022-10-11 21:40:31] ...trap/engines/engine.py.228: elapsed: 0:28:30 | left: 0:04:24 [I 2022-10-11 21:40:31] ...trap/engines/engine.py.229: process: 0.85532 | load: 0.00295 [I 2022-10-11 21:40:31] ...trap/engines/engine.py.230: loss: 0.09822 [I 2022-10-11 21:40:31] ...optimizers/trijoint.py.151: total_norm: 11.799907 [I 2022-10-11 21:40:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 2000/2304 [I 2022-10-11 21:40:40] ...trap/engines/engine.py.228: elapsed: 0:28:38 | left: 0:04:18 [I 2022-10-11 21:40:40] ...trap/engines/engine.py.229: process: 0.85163 | load: 0.00296 [I 2022-10-11 21:40:40] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-11 21:40:40] ...optimizers/trijoint.py.151: total_norm: 18.031317 [I 2022-10-11 21:40:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 2010/2304 [I 2022-10-11 21:40:48] ...trap/engines/engine.py.228: elapsed: 0:28:47 | left: 0:04:10 [I 2022-10-11 21:40:48] ...trap/engines/engine.py.229: process: 0.87191 | load: 0.00298 [I 2022-10-11 21:40:48] ...trap/engines/engine.py.230: loss: 0.11180 [I 2022-10-11 21:40:48] ...optimizers/trijoint.py.151: total_norm: 10.889073 [I 2022-10-11 21:40:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 2020/2304 [I 2022-10-11 21:40:57] ...trap/engines/engine.py.228: elapsed: 0:28:55 | left: 0:04:01 [I 2022-10-11 21:40:57] ...trap/engines/engine.py.229: process: 0.85210 | load: 0.00298 [I 2022-10-11 21:40:57] ...trap/engines/engine.py.230: loss: 0.09395 [I 2022-10-11 21:40:57] ...optimizers/trijoint.py.151: total_norm: 19.493462 [I 2022-10-11 21:41:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 2030/2304 [I 2022-10-11 21:41:05] ...trap/engines/engine.py.228: elapsed: 0:29:04 | left: 0:03:51 [I 2022-10-11 21:41:05] ...trap/engines/engine.py.229: process: 0.85201 | load: 0.00298 [I 2022-10-11 21:41:05] ...trap/engines/engine.py.230: loss: 0.10687 [I 2022-10-11 21:41:05] ...optimizers/trijoint.py.151: total_norm: 20.889509 [I 2022-10-11 21:41:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 2040/2304 [I 2022-10-11 21:41:14] ...trap/engines/engine.py.228: elapsed: 0:29:12 | left: 0:03:43 [I 2022-10-11 21:41:14] ...trap/engines/engine.py.229: process: 0.84474 | load: 0.00302 [I 2022-10-11 21:41:14] ...trap/engines/engine.py.230: loss: 0.11800 [I 2022-10-11 21:41:14] ...optimizers/trijoint.py.151: total_norm: 24.390791 [I 2022-10-11 21:41:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 2050/2304 [I 2022-10-11 21:41:22] ...trap/engines/engine.py.228: elapsed: 0:29:21 | left: 0:03:33 [I 2022-10-11 21:41:22] ...trap/engines/engine.py.229: process: 0.82479 | load: 0.00294 [I 2022-10-11 21:41:22] ...trap/engines/engine.py.230: loss: 0.09305 [I 2022-10-11 21:41:22] ...optimizers/trijoint.py.151: total_norm: 15.849957 [I 2022-10-11 21:41:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 2060/2304 [I 2022-10-11 21:41:31] ...trap/engines/engine.py.228: elapsed: 0:29:29 | left: 0:03:26 [I 2022-10-11 21:41:31] ...trap/engines/engine.py.229: process: 0.85128 | load: 0.00292 [I 2022-10-11 21:41:31] ...trap/engines/engine.py.230: loss: 0.09914 [I 2022-10-11 21:41:31] ...optimizers/trijoint.py.151: total_norm: 25.232452 [I 2022-10-11 21:41:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 2070/2304 [I 2022-10-11 21:41:39] ...trap/engines/engine.py.228: elapsed: 0:29:37 | left: 0:03:17 [I 2022-10-11 21:41:39] ...trap/engines/engine.py.229: process: 0.84073 | load: 0.00296 [I 2022-10-11 21:41:39] ...trap/engines/engine.py.230: loss: 0.13070 [I 2022-10-11 21:41:39] ...optimizers/trijoint.py.151: total_norm: 14.273438 [I 2022-10-11 21:41:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 2080/2304 [I 2022-10-11 21:41:48] ...trap/engines/engine.py.228: elapsed: 0:29:46 | left: 0:03:09 [I 2022-10-11 21:41:48] ...trap/engines/engine.py.229: process: 0.84389 | load: 0.00305 [I 2022-10-11 21:41:48] ...trap/engines/engine.py.230: loss: 0.15874 [I 2022-10-11 21:41:48] ...optimizers/trijoint.py.151: total_norm: 19.812010 [I 2022-10-11 21:41:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 2090/2304 [I 2022-10-11 21:41:56] ...trap/engines/engine.py.228: elapsed: 0:29:54 | left: 0:03:01 [I 2022-10-11 21:41:56] ...trap/engines/engine.py.229: process: 0.84878 | load: 0.00300 [I 2022-10-11 21:41:56] ...trap/engines/engine.py.230: loss: 0.10812 [I 2022-10-11 21:41:56] ...optimizers/trijoint.py.151: total_norm: 12.325558 [I 2022-10-11 21:42:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 2100/2304 [I 2022-10-11 21:42:05] ...trap/engines/engine.py.228: elapsed: 0:30:03 | left: 0:02:53 [I 2022-10-11 21:42:05] ...trap/engines/engine.py.229: process: 0.84599 | load: 0.00298 [I 2022-10-11 21:42:05] ...trap/engines/engine.py.230: loss: 0.08582 [I 2022-10-11 21:42:05] ...optimizers/trijoint.py.151: total_norm: 8.687303 [I 2022-10-11 21:42:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 2110/2304 [I 2022-10-11 21:42:13] ...trap/engines/engine.py.228: elapsed: 0:30:12 | left: 0:02:45 [I 2022-10-11 21:42:13] ...trap/engines/engine.py.229: process: 0.85814 | load: 0.00296 [I 2022-10-11 21:42:13] ...trap/engines/engine.py.230: loss: 0.08071 [I 2022-10-11 21:42:13] ...optimizers/trijoint.py.151: total_norm: 12.169408 [I 2022-10-11 21:42:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 2120/2304 [I 2022-10-11 21:42:22] ...trap/engines/engine.py.228: elapsed: 0:30:20 | left: 0:02:34 [I 2022-10-11 21:42:22] ...trap/engines/engine.py.229: process: 0.83202 | load: 0.00300 [I 2022-10-11 21:42:22] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-11 21:42:22] ...optimizers/trijoint.py.151: total_norm: 14.359478 [I 2022-10-11 21:42:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 2130/2304 [I 2022-10-11 21:42:30] ...trap/engines/engine.py.228: elapsed: 0:30:28 | left: 0:02:26 [I 2022-10-11 21:42:30] ...trap/engines/engine.py.229: process: 0.82177 | load: 0.00297 [I 2022-10-11 21:42:30] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-11 21:42:30] ...optimizers/trijoint.py.151: total_norm: 14.976452 [I 2022-10-11 21:42:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 2140/2304 [I 2022-10-11 21:42:39] ...trap/engines/engine.py.228: elapsed: 0:30:37 | left: 0:02:19 [I 2022-10-11 21:42:39] ...trap/engines/engine.py.229: process: 0.84154 | load: 0.00307 [I 2022-10-11 21:42:39] ...trap/engines/engine.py.230: loss: 0.10828 [I 2022-10-11 21:42:39] ...optimizers/trijoint.py.151: total_norm: 10.393596 [I 2022-10-11 21:42:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 2150/2304 [I 2022-10-11 21:42:47] ...trap/engines/engine.py.228: elapsed: 0:30:45 | left: 0:02:09 [I 2022-10-11 21:42:47] ...trap/engines/engine.py.229: process: 0.84627 | load: 0.00297 [I 2022-10-11 21:42:47] ...trap/engines/engine.py.230: loss: 0.17368 [I 2022-10-11 21:42:47] ...optimizers/trijoint.py.151: total_norm: 23.869509 [I 2022-10-11 21:42:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 2160/2304 [I 2022-10-11 21:42:56] ...trap/engines/engine.py.228: elapsed: 0:30:54 | left: 0:02:00 [I 2022-10-11 21:42:56] ...trap/engines/engine.py.229: process: 0.84595 | load: 0.00299 [I 2022-10-11 21:42:56] ...trap/engines/engine.py.230: loss: 0.07348 [I 2022-10-11 21:42:56] ...optimizers/trijoint.py.151: total_norm: 11.498897 [I 2022-10-11 21:43:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 2170/2304 [I 2022-10-11 21:43:04] ...trap/engines/engine.py.228: elapsed: 0:31:02 | left: 0:01:52 [I 2022-10-11 21:43:04] ...trap/engines/engine.py.229: process: 0.84568 | load: 0.00300 [I 2022-10-11 21:43:04] ...trap/engines/engine.py.230: loss: 0.11722 [I 2022-10-11 21:43:04] ...optimizers/trijoint.py.151: total_norm: 13.265255 [I 2022-10-11 21:43:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 2180/2304 [I 2022-10-11 21:43:13] ...trap/engines/engine.py.228: elapsed: 0:31:11 | left: 0:01:45 [I 2022-10-11 21:43:13] ...trap/engines/engine.py.229: process: 0.86557 | load: 0.00312 [I 2022-10-11 21:43:13] ...trap/engines/engine.py.230: loss: 0.08717 [I 2022-10-11 21:43:13] ...optimizers/trijoint.py.151: total_norm: 21.945753 [I 2022-10-11 21:43:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 2190/2304 [I 2022-10-11 21:43:21] ...trap/engines/engine.py.228: elapsed: 0:31:20 | left: 0:01:40 [I 2022-10-11 21:43:21] ...trap/engines/engine.py.229: process: 0.89241 | load: 0.00392 [I 2022-10-11 21:43:21] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-11 21:43:21] ...optimizers/trijoint.py.151: total_norm: 12.581757 [I 2022-10-11 21:43:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 2200/2304 [I 2022-10-11 21:43:30] ...trap/engines/engine.py.228: elapsed: 0:31:29 | left: 0:01:32 [I 2022-10-11 21:43:30] ...trap/engines/engine.py.229: process: 0.88339 | load: 0.00402 [I 2022-10-11 21:43:30] ...trap/engines/engine.py.230: loss: 0.11041 [I 2022-10-11 21:43:30] ...optimizers/trijoint.py.151: total_norm: 9.618842 [I 2022-10-11 21:43:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 2210/2304 [I 2022-10-11 21:43:39] ...trap/engines/engine.py.228: elapsed: 0:31:37 | left: 0:01:23 [I 2022-10-11 21:43:39] ...trap/engines/engine.py.229: process: 0.88686 | load: 0.00396 [I 2022-10-11 21:43:39] ...trap/engines/engine.py.230: loss: 0.17959 [I 2022-10-11 21:43:39] ...optimizers/trijoint.py.151: total_norm: 11.137958 [I 2022-10-11 21:43:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 2220/2304 [I 2022-10-11 21:43:48] ...trap/engines/engine.py.228: elapsed: 0:31:46 | left: 0:01:14 [I 2022-10-11 21:43:48] ...trap/engines/engine.py.229: process: 0.89570 | load: 0.00397 [I 2022-10-11 21:43:48] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-11 21:43:48] ...optimizers/trijoint.py.151: total_norm: 10.416143 [I 2022-10-11 21:43:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 2230/2304 [I 2022-10-11 21:43:57] ...trap/engines/engine.py.228: elapsed: 0:31:55 | left: 0:01:05 [I 2022-10-11 21:43:57] ...trap/engines/engine.py.229: process: 0.88762 | load: 0.00398 [I 2022-10-11 21:43:57] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-11 21:43:57] ...optimizers/trijoint.py.151: total_norm: 15.044580 [I 2022-10-11 21:44:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 2240/2304 [I 2022-10-11 21:44:06] ...trap/engines/engine.py.228: elapsed: 0:32:04 | left: 0:00:56 [I 2022-10-11 21:44:06] ...trap/engines/engine.py.229: process: 0.88971 | load: 0.00402 [I 2022-10-11 21:44:06] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-11 21:44:06] ...optimizers/trijoint.py.151: total_norm: 13.865165 [I 2022-10-11 21:44:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 2250/2304 [I 2022-10-11 21:44:15] ...trap/engines/engine.py.228: elapsed: 0:32:13 | left: 0:00:47 [I 2022-10-11 21:44:15] ...trap/engines/engine.py.229: process: 0.86190 | load: 0.00317 [I 2022-10-11 21:44:15] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-11 21:44:15] ...optimizers/trijoint.py.151: total_norm: 12.754883 [I 2022-10-11 21:44:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 2260/2304 [I 2022-10-11 21:44:23] ...trap/engines/engine.py.228: elapsed: 0:32:21 | left: 0:00:37 [I 2022-10-11 21:44:23] ...trap/engines/engine.py.229: process: 0.85010 | load: 0.00313 [I 2022-10-11 21:44:23] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-11 21:44:23] ...optimizers/trijoint.py.151: total_norm: 24.020481 [I 2022-10-11 21:44:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 2270/2304 [I 2022-10-11 21:44:32] ...trap/engines/engine.py.228: elapsed: 0:32:30 | left: 0:00:29 [I 2022-10-11 21:44:32] ...trap/engines/engine.py.229: process: 0.85414 | load: 0.00312 [I 2022-10-11 21:44:32] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-11 21:44:32] ...optimizers/trijoint.py.151: total_norm: 12.843547 [I 2022-10-11 21:44:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 2280/2304 [I 2022-10-11 21:44:40] ...trap/engines/engine.py.228: elapsed: 0:32:39 | left: 0:00:20 [I 2022-10-11 21:44:40] ...trap/engines/engine.py.229: process: 0.86705 | load: 0.00312 [I 2022-10-11 21:44:40] ...trap/engines/engine.py.230: loss: 0.15574 [I 2022-10-11 21:44:40] ...optimizers/trijoint.py.151: total_norm: 11.701485 [I 2022-10-11 21:44:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 2290/2304 [I 2022-10-11 21:44:49] ...trap/engines/engine.py.228: elapsed: 0:32:47 | left: 0:00:12 [I 2022-10-11 21:44:49] ...trap/engines/engine.py.229: process: 0.86732 | load: 0.00320 [I 2022-10-11 21:44:49] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-11 21:44:49] ...optimizers/trijoint.py.151: total_norm: 12.337355 [I 2022-10-11 21:44:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 2300/2304 [I 2022-10-11 21:44:58] ...trap/engines/engine.py.228: elapsed: 0:32:56 | left: 0:00:03 [I 2022-10-11 21:44:58] ...trap/engines/engine.py.229: process: 0.84847 | load: 0.00019 [I 2022-10-11 21:44:58] ...trap/engines/engine.py.230: loss: 0.07063 [I 2022-10-11 21:44:58] ...optimizers/trijoint.py.151: total_norm: 8.487679 [I 2022-10-11 21:45:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 2304/2304 [I 2022-10-11 21:45:01] ...trap/engines/engine.py.228: elapsed: 0:32:59 | left: 0:00:00 [I 2022-10-11 21:45:01] ...trap/engines/engine.py.229: process: 0.84364 | load: 0.00019 [I 2022-10-11 21:45:01] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-11 21:45:01] ...optimizers/trijoint.py.151: total_norm: 17.742579 [S 2022-10-11 21:45:02] ...trap/engines/engine.py.129: train_epoch.epoch: 119 [S 2022-10-11 21:45:02] ...trap/engines/engine.py.129: train_epoch.loss: 0.11739174848878979 [I 2022-10-11 21:45:02] ...s/criterions/triplet.py.84: margin = 0.3 [I 2022-10-11 21:45:04] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 119 [I 2022-10-11 21:45:04] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-11 21:45:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 21:45:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 21:45:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 21:45:11] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 21:45:11] ...trap/engines/engine.py.321: eval: epoch 119 | batch 0/510 [I 2022-10-11 21:45:11] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:58:29 [I 2022-10-11 21:45:11] ...trap/engines/engine.py.326: process: 6.88163 | load: 5.70705 [I 2022-10-11 21:45:16] ...trap/engines/engine.py.321: eval: epoch 119 | batch 10/510 [I 2022-10-11 21:45:16] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:09:36 [I 2022-10-11 21:45:16] ...trap/engines/engine.py.326: process: 0.50967 | load: 0.00023 [I 2022-10-11 21:45:20] ...trap/engines/engine.py.321: eval: epoch 119 | batch 20/510 [I 2022-10-11 21:45:20] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:04:14 [I 2022-10-11 21:45:20] ...trap/engines/engine.py.326: process: 0.44117 | load: 0.00020 [I 2022-10-11 21:45:25] ...trap/engines/engine.py.321: eval: epoch 119 | batch 30/510 [I 2022-10-11 21:45:25] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:03:40 [I 2022-10-11 21:45:25] ...trap/engines/engine.py.326: process: 0.45788 | load: 0.00022 [I 2022-10-11 21:45:29] ...trap/engines/engine.py.321: eval: epoch 119 | batch 40/510 [I 2022-10-11 21:45:29] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:03:29 [I 2022-10-11 21:45:29] ...trap/engines/engine.py.326: process: 0.43308 | load: 0.00021 [I 2022-10-11 21:45:34] ...trap/engines/engine.py.321: eval: epoch 119 | batch 50/510 [I 2022-10-11 21:45:34] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:03:29 [I 2022-10-11 21:45:34] ...trap/engines/engine.py.326: process: 0.45107 | load: 0.00022 [I 2022-10-11 21:45:38] ...trap/engines/engine.py.321: eval: epoch 119 | batch 60/510 [I 2022-10-11 21:45:38] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:03:31 [I 2022-10-11 21:45:38] ...trap/engines/engine.py.326: process: 0.45159 | load: 0.00021 [I 2022-10-11 21:45:43] ...trap/engines/engine.py.321: eval: epoch 119 | batch 70/510 [I 2022-10-11 21:45:43] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:03:25 [I 2022-10-11 21:45:43] ...trap/engines/engine.py.326: process: 0.45919 | load: 0.00021 [I 2022-10-11 21:45:48] ...trap/engines/engine.py.321: eval: epoch 119 | batch 80/510 [I 2022-10-11 21:45:48] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:03:24 [I 2022-10-11 21:45:48] ...trap/engines/engine.py.326: process: 0.48585 | load: 0.00021 [I 2022-10-11 21:45:52] ...trap/engines/engine.py.321: eval: epoch 119 | batch 90/510 [I 2022-10-11 21:45:52] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:03:17 [I 2022-10-11 21:45:52] ...trap/engines/engine.py.326: process: 0.47978 | load: 0.00021 [I 2022-10-11 21:45:57] ...trap/engines/engine.py.321: eval: epoch 119 | batch 100/510 [I 2022-10-11 21:45:57] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:03:04 [I 2022-10-11 21:45:57] ...trap/engines/engine.py.326: process: 0.31076 | load: 0.00021 [I 2022-10-11 21:46:00] ...trap/engines/engine.py.321: eval: epoch 119 | batch 110/510 [I 2022-10-11 21:46:00] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:02:14 [I 2022-10-11 21:46:00] ...trap/engines/engine.py.326: process: 0.32389 | load: 0.00021 [I 2022-10-11 21:46:04] ...trap/engines/engine.py.321: eval: epoch 119 | batch 120/510 [I 2022-10-11 21:46:04] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:02:05 [I 2022-10-11 21:46:04] ...trap/engines/engine.py.326: process: 0.32057 | load: 0.00021 [I 2022-10-11 21:46:07] ...trap/engines/engine.py.321: eval: epoch 119 | batch 130/510 [I 2022-10-11 21:46:07] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:02:02 [I 2022-10-11 21:46:07] ...trap/engines/engine.py.326: process: 0.32155 | load: 0.00022 [I 2022-10-11 21:46:10] ...trap/engines/engine.py.321: eval: epoch 119 | batch 140/510 [I 2022-10-11 21:46:10] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:58 [I 2022-10-11 21:46:10] ...trap/engines/engine.py.326: process: 0.32245 | load: 0.00024 [I 2022-10-11 21:46:13] ...trap/engines/engine.py.321: eval: epoch 119 | batch 150/510 [I 2022-10-11 21:46:13] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:54 [I 2022-10-11 21:46:13] ...trap/engines/engine.py.326: process: 0.31386 | load: 0.00021 [I 2022-10-11 21:46:16] ...trap/engines/engine.py.321: eval: epoch 119 | batch 160/510 [I 2022-10-11 21:46:16] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:52 [I 2022-10-11 21:46:16] ...trap/engines/engine.py.326: process: 0.31741 | load: 0.00026 [I 2022-10-11 21:46:20] ...trap/engines/engine.py.321: eval: epoch 119 | batch 170/510 [I 2022-10-11 21:46:20] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:49 [I 2022-10-11 21:46:20] ...trap/engines/engine.py.326: process: 0.32094 | load: 0.00023 [I 2022-10-11 21:46:23] ...trap/engines/engine.py.321: eval: epoch 119 | batch 180/510 [I 2022-10-11 21:46:23] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:45 [I 2022-10-11 21:46:23] ...trap/engines/engine.py.326: process: 0.31947 | load: 0.00026 [I 2022-10-11 21:46:26] ...trap/engines/engine.py.321: eval: epoch 119 | batch 190/510 [I 2022-10-11 21:46:26] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:42 [I 2022-10-11 21:46:26] ...trap/engines/engine.py.326: process: 0.32454 | load: 0.00021 [I 2022-10-11 21:46:29] ...trap/engines/engine.py.321: eval: epoch 119 | batch 200/510 [I 2022-10-11 21:46:29] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:40 [I 2022-10-11 21:46:29] ...trap/engines/engine.py.326: process: 0.32195 | load: 0.00021 [I 2022-10-11 21:46:32] ...trap/engines/engine.py.321: eval: epoch 119 | batch 210/510 [I 2022-10-11 21:46:32] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:36 [I 2022-10-11 21:46:32] ...trap/engines/engine.py.326: process: 0.32462 | load: 0.00022 [I 2022-10-11 21:46:36] ...trap/engines/engine.py.321: eval: epoch 119 | batch 220/510 [I 2022-10-11 21:46:36] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:33 [I 2022-10-11 21:46:36] ...trap/engines/engine.py.326: process: 0.32072 | load: 0.00025 [I 2022-10-11 21:46:39] ...trap/engines/engine.py.321: eval: epoch 119 | batch 230/510 [I 2022-10-11 21:46:39] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:29 [I 2022-10-11 21:46:39] ...trap/engines/engine.py.326: process: 0.32346 | load: 0.00020 [I 2022-10-11 21:46:42] ...trap/engines/engine.py.321: eval: epoch 119 | batch 240/510 [I 2022-10-11 21:46:42] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:27 [I 2022-10-11 21:46:42] ...trap/engines/engine.py.326: process: 0.31962 | load: 0.00024 [I 2022-10-11 21:46:45] ...trap/engines/engine.py.321: eval: epoch 119 | batch 250/510 [I 2022-10-11 21:46:45] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:01:23 [I 2022-10-11 21:46:45] ...trap/engines/engine.py.326: process: 0.32758 | load: 0.00021 [I 2022-10-11 21:46:49] ...trap/engines/engine.py.321: eval: epoch 119 | batch 260/510 [I 2022-10-11 21:46:49] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:01:21 [I 2022-10-11 21:46:49] ...trap/engines/engine.py.326: process: 0.32800 | load: 0.00024 [I 2022-10-11 21:46:52] ...trap/engines/engine.py.321: eval: epoch 119 | batch 270/510 [I 2022-10-11 21:46:52] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:01:17 [I 2022-10-11 21:46:52] ...trap/engines/engine.py.326: process: 0.32096 | load: 0.00024 [I 2022-10-11 21:46:55] ...trap/engines/engine.py.321: eval: epoch 119 | batch 280/510 [I 2022-10-11 21:46:55] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:01:14 [I 2022-10-11 21:46:55] ...trap/engines/engine.py.326: process: 0.31962 | load: 0.00022 [I 2022-10-11 21:46:58] ...trap/engines/engine.py.321: eval: epoch 119 | batch 290/510 [I 2022-10-11 21:46:58] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:01:11 [I 2022-10-11 21:46:58] ...trap/engines/engine.py.326: process: 0.32557 | load: 0.00022 [I 2022-10-11 21:47:02] ...trap/engines/engine.py.321: eval: epoch 119 | batch 300/510 [I 2022-10-11 21:47:02] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:01:07 [I 2022-10-11 21:47:02] ...trap/engines/engine.py.326: process: 0.32035 | load: 0.00022 [I 2022-10-11 21:47:05] ...trap/engines/engine.py.321: eval: epoch 119 | batch 310/510 [I 2022-10-11 21:47:05] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:01:04 [I 2022-10-11 21:47:05] ...trap/engines/engine.py.326: process: 0.32134 | load: 0.00021 [I 2022-10-11 21:47:08] ...trap/engines/engine.py.321: eval: epoch 119 | batch 320/510 [I 2022-10-11 21:47:08] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:01:00 [I 2022-10-11 21:47:08] ...trap/engines/engine.py.326: process: 0.31682 | load: 0.00021 [I 2022-10-11 21:47:11] ...trap/engines/engine.py.321: eval: epoch 119 | batch 330/510 [I 2022-10-11 21:47:11] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:57 [I 2022-10-11 21:47:11] ...trap/engines/engine.py.326: process: 0.31794 | load: 0.00021 [I 2022-10-11 21:47:14] ...trap/engines/engine.py.321: eval: epoch 119 | batch 340/510 [I 2022-10-11 21:47:14] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:53 [I 2022-10-11 21:47:14] ...trap/engines/engine.py.326: process: 0.31978 | load: 0.00020 [I 2022-10-11 21:47:18] ...trap/engines/engine.py.321: eval: epoch 119 | batch 350/510 [I 2022-10-11 21:47:18] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:50 [I 2022-10-11 21:47:18] ...trap/engines/engine.py.326: process: 0.31847 | load: 0.00019 [I 2022-10-11 21:47:21] ...trap/engines/engine.py.321: eval: epoch 119 | batch 360/510 [I 2022-10-11 21:47:21] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:47 [I 2022-10-11 21:47:21] ...trap/engines/engine.py.326: process: 0.32151 | load: 0.00020 [I 2022-10-11 21:47:24] ...trap/engines/engine.py.321: eval: epoch 119 | batch 370/510 [I 2022-10-11 21:47:24] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:44 [I 2022-10-11 21:47:24] ...trap/engines/engine.py.326: process: 0.32123 | load: 0.00020 [I 2022-10-11 21:47:27] ...trap/engines/engine.py.321: eval: epoch 119 | batch 380/510 [I 2022-10-11 21:47:27] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:41 [I 2022-10-11 21:47:27] ...trap/engines/engine.py.326: process: 0.30917 | load: 0.00020 [I 2022-10-11 21:47:30] ...trap/engines/engine.py.321: eval: epoch 119 | batch 390/510 [I 2022-10-11 21:47:30] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:37 [I 2022-10-11 21:47:30] ...trap/engines/engine.py.326: process: 0.31440 | load: 0.00021 [I 2022-10-11 21:47:33] ...trap/engines/engine.py.321: eval: epoch 119 | batch 400/510 [I 2022-10-11 21:47:33] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:34 [I 2022-10-11 21:47:33] ...trap/engines/engine.py.326: process: 0.31478 | load: 0.00020 [I 2022-10-11 21:47:37] ...trap/engines/engine.py.321: eval: epoch 119 | batch 410/510 [I 2022-10-11 21:47:37] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:31 [I 2022-10-11 21:47:37] ...trap/engines/engine.py.326: process: 0.32226 | load: 0.00021 [I 2022-10-11 21:47:40] ...trap/engines/engine.py.321: eval: epoch 119 | batch 420/510 [I 2022-10-11 21:47:40] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:28 [I 2022-10-11 21:47:40] ...trap/engines/engine.py.326: process: 0.31147 | load: 0.00021 [I 2022-10-11 21:47:43] ...trap/engines/engine.py.321: eval: epoch 119 | batch 430/510 [I 2022-10-11 21:47:43] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:25 [I 2022-10-11 21:47:43] ...trap/engines/engine.py.326: process: 0.31350 | load: 0.00020 [I 2022-10-11 21:47:46] ...trap/engines/engine.py.321: eval: epoch 119 | batch 440/510 [I 2022-10-11 21:47:46] ...trap/engines/engine.py.325: elapsed: 0:02:41 | left: 0:00:22 [I 2022-10-11 21:47:46] ...trap/engines/engine.py.326: process: 0.32093 | load: 0.00020 [I 2022-10-11 21:47:49] ...trap/engines/engine.py.321: eval: epoch 119 | batch 450/510 [I 2022-10-11 21:47:49] ...trap/engines/engine.py.325: elapsed: 0:02:45 | left: 0:00:19 [I 2022-10-11 21:47:49] ...trap/engines/engine.py.326: process: 0.31433 | load: 0.00021 [I 2022-10-11 21:47:53] ...trap/engines/engine.py.321: eval: epoch 119 | batch 460/510 [I 2022-10-11 21:47:53] ...trap/engines/engine.py.325: elapsed: 0:02:48 | left: 0:00:15 [I 2022-10-11 21:47:53] ...trap/engines/engine.py.326: process: 0.31926 | load: 0.00019 [I 2022-10-11 21:47:56] ...trap/engines/engine.py.321: eval: epoch 119 | batch 470/510 [I 2022-10-11 21:47:56] ...trap/engines/engine.py.325: elapsed: 0:02:51 | left: 0:00:12 [I 2022-10-11 21:47:56] ...trap/engines/engine.py.326: process: 0.31492 | load: 0.00019 [I 2022-10-11 21:47:59] ...trap/engines/engine.py.321: eval: epoch 119 | batch 480/510 [I 2022-10-11 21:47:59] ...trap/engines/engine.py.325: elapsed: 0:02:54 | left: 0:00:09 [I 2022-10-11 21:47:59] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00019 [I 2022-10-11 21:48:02] ...trap/engines/engine.py.321: eval: epoch 119 | batch 490/510 [I 2022-10-11 21:48:02] ...trap/engines/engine.py.325: elapsed: 0:02:57 | left: 0:00:06 [I 2022-10-11 21:48:02] ...trap/engines/engine.py.326: process: 0.31826 | load: 0.00020 [I 2022-10-11 21:48:05] ...trap/engines/engine.py.321: eval: epoch 119 | batch 500/510 [I 2022-10-11 21:48:05] ...trap/engines/engine.py.325: elapsed: 0:03:00 | left: 0:00:03 [I 2022-10-11 21:48:05] ...trap/engines/engine.py.326: process: 0.32255 | load: 0.00019 [I 2022-10-11 21:48:08] ...trap/engines/engine.py.321: eval: epoch 119 | batch 510/510 [I 2022-10-11 21:48:08] ...trap/engines/engine.py.325: elapsed: 0:03:04 | left: 0:00:00 [I 2022-10-11 21:48:08] ...trap/engines/engine.py.326: process: 0.30967 | load: 0.00013 [S 2022-10-11 21:48:09] ...trap/engines/engine.py.132: eval_epoch.epoch: 119 [I 2022-10-11 21:48:09] ...els/metrics/trijoint.py.86: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.751200 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910100 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935100 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.741400 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909000 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935100 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012081 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008573 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009864 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013807 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008832 [S 2022-10-11 21:48:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010143 [I 2022-10-11 21:48:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-11 21:48:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-11 21:48:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-11 21:48:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-11 21:48:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-11 21:48:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-11 21:48:53] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html [I 2022-10-11 21:48:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-11 21:48:54] ...trap/engines/engine.py.145: Ending training procedures